2024-03-11T21:06:37.550 INFO:root:teuthology version: 0.0.1.dev262+ge691533 2024-03-11T21:06:37.550 DEBUG:teuthology.run:Teuthology command: teuthology --verbose --archive /home/teuthworker/archive/teuthology-2024-03-11_20:08:13-orch-main-distro-default-smithi/7591804 --name teuthology-2024-03-11_20:08:13-orch-main-distro-default-smithi --description orch/cephadm/thrash/{0-distro/centos_9.stream 1-start 2-thrash 3-tasks/radosbench fixed-2 msgr/async-v1only root} --owner scheduled_teuthology@teuthology -- /home/teuthworker/archive/teuthology-2024-03-11_20:08:13-orch-main-distro-default-smithi/7591804/orig.config.yaml 2024-03-11T21:06:37.583 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-03-11T21:06:37.665 INFO:teuthology.run:Config: archive_path: /home/teuthworker/archive/teuthology-2024-03-11_20:08:13-orch-main-distro-default-smithi/7591804 branch: main description: orch/cephadm/thrash/{0-distro/centos_9.stream 1-start 2-thrash 3-tasks/radosbench fixed-2 msgr/async-v1only root} email: ceph-qa@ceph.com first_in_suite: false job_id: '7591804' kernel: kdb: true sha1: distro last_in_suite: false machine_type: smithi name: teuthology-2024-03-11_20:08:13-orch-main-distro-default-smithi no_nested_subset: false nuke-on-error: true openstack: - volumes: count: 4 size: 10 os_type: centos os_version: 9.stream overrides: admin_socket: branch: main ceph: conf: client.0: debug ms: 1 debug objecter: 20 debug rados: 20 global: ms bind msgr2: false ms type: async mgr: debug mgr: 20 debug ms: 1 mon: debug mon: 20 debug ms: 1 debug paxos: 20 mon min osdmap epochs: 50 mon osdmap full prune interval: 2 mon osdmap full prune min: 15 mon osdmap full prune txsize: 2 paxos service trim min: 10 osd: debug ms: 1 debug osd: 20 osd debug reject backfill probability: 0.3 osd delete sleep: 1 osd max backfills: 3 osd scrub max interval: 120 osd scrub min interval: 60 osd shutdown pgref assert: true osd snap trim sleep: 2 flavor: default log-ignorelist: - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) - but it is still running - objects unfound and apparently lost - \(POOL_APP_NOT_ENABLED\) mon_bind_msgr2: false sha1: 2b0101eb74d60610d544cb8a473e23a35ac1cc0b ceph-deploy: conf: client: log file: /var/log/ceph/ceph-$name.$pid.log mon: {} cephadm: cephadm_mode: root install: ceph: flavor: default sha1: 2b0101eb74d60610d544cb8a473e23a35ac1cc0b selinux: allowlist: - scontext=system_u:system_r:logrotate_t:s0 workunit: branch: main sha1: 1c349451176cc5b4ebfb24b22eaaa754e05cff6c owner: scheduled_teuthology@teuthology priority: 200 repo: https://git.ceph.com/ceph.git roles: - - mon.a - mon.c - mgr.y - osd.0 - osd.1 - osd.2 - osd.3 - client.0 - ceph.rgw.foo.a - node-exporter.a - alertmanager.a - - mon.b - mgr.x - osd.4 - osd.5 - osd.6 - osd.7 - client.1 - prometheus.a - grafana.a - node-exporter.b - ceph.iscsi.iscsi.a seed: 6815 sha1: 2b0101eb74d60610d544cb8a473e23a35ac1cc0b sleep_before_teardown: 0 subset: 10/64 suite: orch suite_branch: main suite_path: /home/teuthworker/src/git.ceph.com_ceph_1c349451176cc5b4ebfb24b22eaaa754e05cff6c/qa suite_relpath: qa suite_repo: https://git.ceph.com/ceph.git suite_sha1: 1c349451176cc5b4ebfb24b22eaaa754e05cff6c targets: smithi092.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBO2eOnecSN6CH/yt9an69HTrIAcybKj1w6rVksVunYh88HFogz8kG7ra0xdBHXWEF9Ga340/EqBYclg4h7diz1w= smithi097.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBGoKhCCw7+ZfNtOLLnmD5QxlgNJ6jjDm1kgY07XdEwuq15OTBdIwGlcAaxlKl8z7Gx0OncR9d446T8YZC3lFVMw= tasks: - install: null - cephadm: conf: mgr: debug mgr: 20 debug ms: 1 - thrashosds: chance_pgnum_grow: 1 chance_pgnum_shrink: 1 chance_pgpnum_fix: 1 timeout: 1200 - full_sequential: - radosbench: clients: - client.0 time: 90 - radosbench: clients: - client.0 time: 90 - radosbench: clients: - client.0 time: 90 - radosbench: clients: - client.0 time: 90 - radosbench: clients: - client.0 time: 90 teuthology: fragments_dropped: [] meta: {} postmerge: [] teuthology_branch: main teuthology_sha1: e691533f9cbb33d85b2187bba20d7102f098636d timestamp: 2024-03-11_20:08:13 tube: smithi user: teuthology verbose: false worker_log: /home/teuthworker/archive/worker_logs/dispatcher.smithi.2226885 2024-03-11T21:06:37.665 INFO:teuthology.run:suite_path is set to /home/teuthworker/src/git.ceph.com_ceph_1c349451176cc5b4ebfb24b22eaaa754e05cff6c/qa; will attempt to use it 2024-03-11T21:06:37.666 INFO:teuthology.run:Found tasks at /home/teuthworker/src/git.ceph.com_ceph_1c349451176cc5b4ebfb24b22eaaa754e05cff6c/qa/tasks 2024-03-11T21:06:37.666 INFO:teuthology.run_tasks:Running task internal.check_packages... 2024-03-11T21:06:37.667 INFO:teuthology.task.internal:Checking packages... 2024-03-11T21:06:37.692 INFO:teuthology.task.internal:Checking packages for os_type 'centos', flavor 'default' and ceph hash '2b0101eb74d60610d544cb8a473e23a35ac1cc0b' 2024-03-11T21:06:37.692 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2024-03-11T21:06:37.692 INFO:teuthology.packaging:ref: None 2024-03-11T21:06:37.692 INFO:teuthology.packaging:tag: None 2024-03-11T21:06:37.692 INFO:teuthology.packaging:branch: main 2024-03-11T21:06:37.692 INFO:teuthology.packaging:sha1: 2b0101eb74d60610d544cb8a473e23a35ac1cc0b 2024-03-11T21:06:37.692 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&ref=main 2024-03-11T21:06:38.030 INFO:teuthology.task.internal:Found packages for ceph version 19.0.0-1998.g1c349451 2024-03-11T21:06:38.032 INFO:teuthology.run_tasks:Running task internal.buildpackages_prep... 2024-03-11T21:06:38.038 INFO:teuthology.task.internal:no buildpackages task found 2024-03-11T21:06:38.038 INFO:teuthology.run_tasks:Running task internal.save_config... 2024-03-11T21:06:38.052 INFO:teuthology.task.internal:Saving configuration 2024-03-11T21:06:38.065 INFO:teuthology.run_tasks:Running task internal.check_lock... 2024-03-11T21:06:38.070 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-03-11T21:06:38.098 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi092.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/teuthology-2024-03-11_20:08:13-orch-main-distro-default-smithi/7591804', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-03-11 20:59:37.405444', 'locked_by': 'scheduled_teuthology@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBO2eOnecSN6CH/yt9an69HTrIAcybKj1w6rVksVunYh88HFogz8kG7ra0xdBHXWEF9Ga340/EqBYclg4h7diz1w='} 2024-03-11T21:06:38.120 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi097.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/teuthology-2024-03-11_20:08:13-orch-main-distro-default-smithi/7591804', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-03-11 20:59:37.406977', 'locked_by': 'scheduled_teuthology@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBGoKhCCw7+ZfNtOLLnmD5QxlgNJ6jjDm1kgY07XdEwuq15OTBdIwGlcAaxlKl8z7Gx0OncR9d446T8YZC3lFVMw='} 2024-03-11T21:06:38.120 INFO:teuthology.run_tasks:Running task internal.add_remotes... 2024-03-11T21:06:38.127 INFO:teuthology.task.internal:roles: ubuntu@smithi092.front.sepia.ceph.com - ['mon.a', 'mon.c', 'mgr.y', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0', 'ceph.rgw.foo.a', 'node-exporter.a', 'alertmanager.a'] 2024-03-11T21:06:38.127 INFO:teuthology.task.internal:roles: ubuntu@smithi097.front.sepia.ceph.com - ['mon.b', 'mgr.x', 'osd.4', 'osd.5', 'osd.6', 'osd.7', 'client.1', 'prometheus.a', 'grafana.a', 'node-exporter.b', 'ceph.iscsi.iscsi.a'] 2024-03-11T21:06:38.127 INFO:teuthology.run_tasks:Running task console_log... 2024-03-11T21:06:38.202 DEBUG:teuthology.exit:Installing handler: Handler(exiter=, func=.kill_console_loggers at 0x7f5fddf49f70>, signals=[15]) 2024-03-11T21:06:38.202 INFO:teuthology.run_tasks:Running task internal.connect... 2024-03-11T21:06:38.207 INFO:teuthology.task.internal:Opening connections... 2024-03-11T21:06:38.207 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi092.front.sepia.ceph.com 2024-03-11T21:06:38.209 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi092.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-11T21:06:38.301 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi097.front.sepia.ceph.com 2024-03-11T21:06:38.302 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi097.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-11T21:06:38.390 INFO:teuthology.run_tasks:Running task internal.push_inventory... 2024-03-11T21:06:38.396 DEBUG:teuthology.orchestra.run.smithi092:> uname -m 2024-03-11T21:06:38.413 INFO:teuthology.orchestra.run.smithi092.stdout:x86_64 2024-03-11T21:06:38.414 DEBUG:teuthology.orchestra.run.smithi092:> cat /etc/os-release 2024-03-11T21:06:38.470 INFO:teuthology.orchestra.run.smithi092.stdout:NAME="CentOS Stream" 2024-03-11T21:06:38.470 INFO:teuthology.orchestra.run.smithi092.stdout:VERSION="9" 2024-03-11T21:06:38.470 INFO:teuthology.orchestra.run.smithi092.stdout:ID="centos" 2024-03-11T21:06:38.470 INFO:teuthology.orchestra.run.smithi092.stdout:ID_LIKE="rhel fedora" 2024-03-11T21:06:38.470 INFO:teuthology.orchestra.run.smithi092.stdout:VERSION_ID="9" 2024-03-11T21:06:38.470 INFO:teuthology.orchestra.run.smithi092.stdout:PLATFORM_ID="platform:el9" 2024-03-11T21:06:38.470 INFO:teuthology.orchestra.run.smithi092.stdout:PRETTY_NAME="CentOS Stream 9" 2024-03-11T21:06:38.470 INFO:teuthology.orchestra.run.smithi092.stdout:ANSI_COLOR="0;31" 2024-03-11T21:06:38.470 INFO:teuthology.orchestra.run.smithi092.stdout:LOGO="fedora-logo-icon" 2024-03-11T21:06:38.470 INFO:teuthology.orchestra.run.smithi092.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2024-03-11T21:06:38.471 INFO:teuthology.orchestra.run.smithi092.stdout:HOME_URL="https://centos.org/" 2024-03-11T21:06:38.471 INFO:teuthology.orchestra.run.smithi092.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2024-03-11T21:06:38.471 INFO:teuthology.orchestra.run.smithi092.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2024-03-11T21:06:38.471 INFO:teuthology.orchestra.run.smithi092.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-03-11T21:06:38.471 INFO:teuthology.lock.ops:Updating smithi092.front.sepia.ceph.com on lock server 2024-03-11T21:06:38.493 DEBUG:teuthology.orchestra.run.smithi097:> uname -m 2024-03-11T21:06:38.511 INFO:teuthology.orchestra.run.smithi097.stdout:x86_64 2024-03-11T21:06:38.511 DEBUG:teuthology.orchestra.run.smithi097:> cat /etc/os-release 2024-03-11T21:06:38.569 INFO:teuthology.orchestra.run.smithi097.stdout:NAME="CentOS Stream" 2024-03-11T21:06:38.569 INFO:teuthology.orchestra.run.smithi097.stdout:VERSION="9" 2024-03-11T21:06:38.569 INFO:teuthology.orchestra.run.smithi097.stdout:ID="centos" 2024-03-11T21:06:38.569 INFO:teuthology.orchestra.run.smithi097.stdout:ID_LIKE="rhel fedora" 2024-03-11T21:06:38.569 INFO:teuthology.orchestra.run.smithi097.stdout:VERSION_ID="9" 2024-03-11T21:06:38.569 INFO:teuthology.orchestra.run.smithi097.stdout:PLATFORM_ID="platform:el9" 2024-03-11T21:06:38.569 INFO:teuthology.orchestra.run.smithi097.stdout:PRETTY_NAME="CentOS Stream 9" 2024-03-11T21:06:38.569 INFO:teuthology.orchestra.run.smithi097.stdout:ANSI_COLOR="0;31" 2024-03-11T21:06:38.569 INFO:teuthology.orchestra.run.smithi097.stdout:LOGO="fedora-logo-icon" 2024-03-11T21:06:38.569 INFO:teuthology.orchestra.run.smithi097.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2024-03-11T21:06:38.569 INFO:teuthology.orchestra.run.smithi097.stdout:HOME_URL="https://centos.org/" 2024-03-11T21:06:38.570 INFO:teuthology.orchestra.run.smithi097.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2024-03-11T21:06:38.570 INFO:teuthology.orchestra.run.smithi097.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2024-03-11T21:06:38.570 INFO:teuthology.orchestra.run.smithi097.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-03-11T21:06:38.570 INFO:teuthology.lock.ops:Updating smithi097.front.sepia.ceph.com on lock server 2024-03-11T21:06:38.593 INFO:teuthology.run_tasks:Running task internal.serialize_remote_roles... 2024-03-11T21:06:38.601 INFO:teuthology.run_tasks:Running task internal.check_conflict... 2024-03-11T21:06:38.607 INFO:teuthology.task.internal:Checking for old test directory... 2024-03-11T21:06:38.607 DEBUG:teuthology.orchestra.run.smithi092:> test '!' -e /home/ubuntu/cephtest 2024-03-11T21:06:38.610 DEBUG:teuthology.orchestra.run.smithi097:> test '!' -e /home/ubuntu/cephtest 2024-03-11T21:06:38.625 INFO:teuthology.run_tasks:Running task internal.check_ceph_data... 2024-03-11T21:06:38.631 INFO:teuthology.task.internal:Checking for non-empty /var/lib/ceph... 2024-03-11T21:06:38.631 DEBUG:teuthology.orchestra.run.smithi092:> test -z $(ls -A /var/lib/ceph) 2024-03-11T21:06:38.666 DEBUG:teuthology.orchestra.run.smithi097:> test -z $(ls -A /var/lib/ceph) 2024-03-11T21:06:38.757 INFO:teuthology.run_tasks:Running task internal.vm_setup... 2024-03-11T21:06:38.811 INFO:teuthology.run_tasks:Running task kernel... 2024-03-11T21:06:38.827 INFO:teuthology.task.kernel:normalize config orig: {'kdb': True, 'sha1': 'distro'} 2024-03-11T21:06:38.827 INFO:teuthology.task.kernel:config {'mon.a': {'kdb': True, 'sha1': 'distro'}, 'mon.b': {'kdb': True, 'sha1': 'distro'}}, timeout 300 2024-03-11T21:06:38.827 DEBUG:teuthology.orchestra.run.smithi092:> test -f /run/.containerenv -o -f /.dockerenv 2024-03-11T21:06:38.828 DEBUG:teuthology.orchestra.run.smithi097:> test -f /run/.containerenv -o -f /.dockerenv 2024-03-11T21:06:38.843 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:06:38.844 DEBUG:teuthology.orchestra.run.smithi092:> uname -r 2024-03-11T21:06:38.845 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:06:38.845 DEBUG:teuthology.orchestra.run.smithi097:> uname -r 2024-03-11T21:06:38.901 INFO:teuthology.orchestra.run.smithi092.stdout:5.14.0-361.el9.x86_64 2024-03-11T21:06:38.901 INFO:teuthology.task.kernel:Running kernel on smithi092: 5.14.0-361.el9.x86_64 2024-03-11T21:06:38.901 DEBUG:teuthology.orchestra.run.smithi092:> sudo yum install -y kernel 2024-03-11T21:06:38.902 INFO:teuthology.orchestra.run.smithi097.stdout:5.14.0-361.el9.x86_64 2024-03-11T21:06:38.903 INFO:teuthology.task.kernel:Running kernel on smithi097: 5.14.0-361.el9.x86_64 2024-03-11T21:06:38.903 DEBUG:teuthology.orchestra.run.smithi097:> sudo yum install -y kernel 2024-03-11T21:06:41.010 INFO:teuthology.orchestra.run.smithi092.stdout:CentOS-9 - AppStream 14 MB/s | 6.7 MB 00:00 2024-03-11T21:06:41.105 INFO:teuthology.orchestra.run.smithi097.stdout:CentOS-9 - AppStream 14 MB/s | 6.7 MB 00:00 2024-03-11T21:06:42.563 INFO:teuthology.orchestra.run.smithi097.stdout:CentOS-9 - BaseOS 5.6 MB/s | 2.0 MB 00:00 2024-03-11T21:06:42.660 INFO:teuthology.orchestra.run.smithi092.stdout:CentOS-9 - BaseOS 3.7 MB/s | 2.0 MB 00:00 2024-03-11T21:06:44.921 INFO:teuthology.orchestra.run.smithi097.stdout:CentOS Stream 9 - BaseOS 4.0 MB/s | 8.0 MB 00:02 2024-03-11T21:06:44.926 INFO:teuthology.orchestra.run.smithi092.stdout:CentOS Stream 9 - BaseOS 4.1 MB/s | 8.0 MB 00:01 2024-03-11T21:06:48.985 INFO:teuthology.orchestra.run.smithi092.stdout:CentOS Stream 9 - AppStream 6.3 MB/s | 19 MB 00:02 2024-03-11T21:06:51.263 INFO:teuthology.orchestra.run.smithi097.stdout:CentOS Stream 9 - AppStream 3.6 MB/s | 19 MB 00:05 2024-03-11T21:06:55.532 INFO:teuthology.orchestra.run.smithi097.stdout:CentOS Stream 9 - CRB 7.5 MB/s | 6.1 MB 00:00 2024-03-11T21:06:56.914 INFO:teuthology.orchestra.run.smithi097.stdout:CentOS Stream 9 - Extras packages 63 kB/s | 15 kB 00:00 2024-03-11T21:06:58.274 INFO:teuthology.orchestra.run.smithi097.stdout:Extra Packages for Enterprise Linux 17 MB/s | 21 MB 00:01 2024-03-11T21:07:01.592 INFO:teuthology.orchestra.run.smithi092.stdout:CentOS Stream 9 - CRB 690 kB/s | 6.1 MB 00:09 2024-03-11T21:07:03.034 INFO:teuthology.orchestra.run.smithi092.stdout:CentOS Stream 9 - Extras packages 51 kB/s | 15 kB 00:00 2024-03-11T21:07:03.149 INFO:teuthology.orchestra.run.smithi097.stdout:lab-extras 20 kB/s | 1.7 kB 00:00 2024-03-11T21:07:04.910 INFO:teuthology.orchestra.run.smithi092.stdout:Extra Packages for Enterprise Linux 12 MB/s | 21 MB 00:01 2024-03-11T21:07:05.339 INFO:teuthology.orchestra.run.smithi097.stdout:Package kernel-5.14.0-239.el9.x86_64 is already installed. 2024-03-11T21:07:05.339 INFO:teuthology.orchestra.run.smithi097.stdout:Package kernel-5.14.0-361.el9.x86_64 is already installed. 2024-03-11T21:07:05.439 INFO:teuthology.orchestra.run.smithi097.stdout:Dependencies resolved. 2024-03-11T21:07:05.450 INFO:teuthology.orchestra.run.smithi097.stdout:================================================================================ 2024-03-11T21:07:05.450 INFO:teuthology.orchestra.run.smithi097.stdout: Package Architecture Version Repository Size 2024-03-11T21:07:05.450 INFO:teuthology.orchestra.run.smithi097.stdout:================================================================================ 2024-03-11T21:07:05.450 INFO:teuthology.orchestra.run.smithi097.stdout:Installing: 2024-03-11T21:07:05.450 INFO:teuthology.orchestra.run.smithi097.stdout: kernel x86_64 5.14.0-427.el9 baseos 6.0 M 2024-03-11T21:07:05.450 INFO:teuthology.orchestra.run.smithi097.stdout:Installing dependencies: 2024-03-11T21:07:05.451 INFO:teuthology.orchestra.run.smithi097.stdout: kernel-core x86_64 5.14.0-427.el9 baseos 21 M 2024-03-11T21:07:05.451 INFO:teuthology.orchestra.run.smithi097.stdout: kernel-modules x86_64 5.14.0-427.el9 baseos 39 M 2024-03-11T21:07:05.451 INFO:teuthology.orchestra.run.smithi097.stdout: kernel-modules-core x86_64 5.14.0-427.el9 baseos 33 M 2024-03-11T21:07:05.451 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-03-11T21:07:05.451 INFO:teuthology.orchestra.run.smithi097.stdout:Transaction Summary 2024-03-11T21:07:05.451 INFO:teuthology.orchestra.run.smithi097.stdout:================================================================================ 2024-03-11T21:07:05.451 INFO:teuthology.orchestra.run.smithi097.stdout:Install 4 Packages 2024-03-11T21:07:05.451 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-03-11T21:07:05.452 INFO:teuthology.orchestra.run.smithi097.stdout:Total download size: 100 M 2024-03-11T21:07:05.452 INFO:teuthology.orchestra.run.smithi097.stdout:Installed size: 123 M 2024-03-11T21:07:05.452 INFO:teuthology.orchestra.run.smithi097.stdout:Downloading Packages: 2024-03-11T21:07:05.925 INFO:teuthology.orchestra.run.smithi097.stdout:(1/4): kernel-5.14.0-427.el9.x86_64.rpm 14 MB/s | 6.0 MB 00:00 2024-03-11T21:07:06.458 INFO:teuthology.orchestra.run.smithi097.stdout:(2/4): kernel-core-5.14.0-427.el9.x86_64.rpm 22 MB/s | 21 MB 00:00 2024-03-11T21:07:07.050 INFO:teuthology.orchestra.run.smithi097.stdout:(3/4): kernel-modules-5.14.0-427.el9.x86_64.rpm 25 MB/s | 39 MB 00:01 2024-03-11T21:07:07.651 INFO:teuthology.orchestra.run.smithi097.stdout:(4/4): kernel-modules-core-5.14.0-427.el9.x86_6 19 MB/s | 33 MB 00:01 2024-03-11T21:07:07.651 INFO:teuthology.orchestra.run.smithi097.stdout:-------------------------------------------------------------------------------- 2024-03-11T21:07:07.651 INFO:teuthology.orchestra.run.smithi097.stdout:Total 45 MB/s | 100 MB 00:02 2024-03-11T21:07:08.536 INFO:teuthology.orchestra.run.smithi097.stdout:Running transaction check 2024-03-11T21:07:08.898 INFO:teuthology.orchestra.run.smithi097.stdout:Transaction check succeeded. 2024-03-11T21:07:08.898 INFO:teuthology.orchestra.run.smithi097.stdout:Running transaction test 2024-03-11T21:07:09.793 INFO:teuthology.orchestra.run.smithi092.stdout:lab-extras 23 kB/s | 1.7 kB 00:00 2024-03-11T21:07:10.245 INFO:teuthology.orchestra.run.smithi097.stdout:Transaction test succeeded. 2024-03-11T21:07:10.245 INFO:teuthology.orchestra.run.smithi097.stdout:Running transaction 2024-03-11T21:07:11.982 INFO:teuthology.orchestra.run.smithi092.stdout:Package kernel-5.14.0-239.el9.x86_64 is already installed. 2024-03-11T21:07:11.982 INFO:teuthology.orchestra.run.smithi092.stdout:Package kernel-5.14.0-361.el9.x86_64 is already installed. 2024-03-11T21:07:12.073 INFO:teuthology.orchestra.run.smithi092.stdout:Dependencies resolved. 2024-03-11T21:07:12.078 INFO:teuthology.orchestra.run.smithi092.stdout:================================================================================ 2024-03-11T21:07:12.078 INFO:teuthology.orchestra.run.smithi092.stdout: Package Architecture Version Repository Size 2024-03-11T21:07:12.078 INFO:teuthology.orchestra.run.smithi092.stdout:================================================================================ 2024-03-11T21:07:12.078 INFO:teuthology.orchestra.run.smithi092.stdout:Installing: 2024-03-11T21:07:12.078 INFO:teuthology.orchestra.run.smithi092.stdout: kernel x86_64 5.14.0-427.el9 baseos 6.0 M 2024-03-11T21:07:12.078 INFO:teuthology.orchestra.run.smithi092.stdout:Installing dependencies: 2024-03-11T21:07:12.078 INFO:teuthology.orchestra.run.smithi092.stdout: kernel-core x86_64 5.14.0-427.el9 baseos 21 M 2024-03-11T21:07:12.078 INFO:teuthology.orchestra.run.smithi092.stdout: kernel-modules x86_64 5.14.0-427.el9 baseos 39 M 2024-03-11T21:07:12.078 INFO:teuthology.orchestra.run.smithi092.stdout: kernel-modules-core x86_64 5.14.0-427.el9 baseos 33 M 2024-03-11T21:07:12.078 INFO:teuthology.orchestra.run.smithi092.stdout: 2024-03-11T21:07:12.078 INFO:teuthology.orchestra.run.smithi092.stdout:Transaction Summary 2024-03-11T21:07:12.079 INFO:teuthology.orchestra.run.smithi092.stdout:================================================================================ 2024-03-11T21:07:12.079 INFO:teuthology.orchestra.run.smithi092.stdout:Install 4 Packages 2024-03-11T21:07:12.079 INFO:teuthology.orchestra.run.smithi092.stdout: 2024-03-11T21:07:12.079 INFO:teuthology.orchestra.run.smithi092.stdout:Total download size: 100 M 2024-03-11T21:07:12.079 INFO:teuthology.orchestra.run.smithi092.stdout:Installed size: 123 M 2024-03-11T21:07:12.079 INFO:teuthology.orchestra.run.smithi092.stdout:Downloading Packages: 2024-03-11T21:07:12.685 INFO:teuthology.orchestra.run.smithi092.stdout:(1/4): kernel-5.14.0-427.el9.x86_64.rpm 14 MB/s | 6.0 MB 00:00 2024-03-11T21:07:13.169 INFO:teuthology.orchestra.run.smithi097.stdout: Preparing : 1/1 2024-03-11T21:07:13.661 INFO:teuthology.orchestra.run.smithi092.stdout:(2/4): kernel-core-5.14.0-427.el9.x86_64.rpm 15 MB/s | 21 MB 00:01 2024-03-11T21:07:13.823 INFO:teuthology.orchestra.run.smithi097.stdout: Installing : kernel-modules-core-5.14.0-427.el9.x86_64 1/4 2024-03-11T21:07:13.951 INFO:teuthology.orchestra.run.smithi097.stdout: Installing : kernel-core-5.14.0-427.el9.x86_64 2/4 2024-03-11T21:07:14.027 INFO:teuthology.orchestra.run.smithi092.stdout:(3/4): kernel-modules-core-5.14.0-427.el9.x86_6 25 MB/s | 33 MB 00:01 2024-03-11T21:07:14.703 INFO:teuthology.orchestra.run.smithi092.stdout:(4/4): kernel-modules-5.14.0-427.el9.x86_64.rpm 16 MB/s | 39 MB 00:02 2024-03-11T21:07:14.703 INFO:teuthology.orchestra.run.smithi092.stdout:-------------------------------------------------------------------------------- 2024-03-11T21:07:14.703 INFO:teuthology.orchestra.run.smithi092.stdout:Total 38 MB/s | 100 MB 00:02 2024-03-11T21:07:15.590 INFO:teuthology.orchestra.run.smithi092.stdout:Running transaction check 2024-03-11T21:07:15.958 INFO:teuthology.orchestra.run.smithi092.stdout:Transaction check succeeded. 2024-03-11T21:07:15.958 INFO:teuthology.orchestra.run.smithi092.stdout:Running transaction test 2024-03-11T21:07:16.751 INFO:teuthology.orchestra.run.smithi097.stdout: Running scriptlet: kernel-core-5.14.0-427.el9.x86_64 2/4 2024-03-11T21:07:16.937 INFO:teuthology.orchestra.run.smithi097.stdout: Installing : kernel-modules-5.14.0-427.el9.x86_64 3/4 2024-03-11T21:07:17.322 INFO:teuthology.orchestra.run.smithi092.stdout:Transaction test succeeded. 2024-03-11T21:07:17.323 INFO:teuthology.orchestra.run.smithi092.stdout:Running transaction 2024-03-11T21:07:20.259 INFO:teuthology.orchestra.run.smithi092.stdout: Preparing : 1/1 2024-03-11T21:07:21.023 INFO:teuthology.orchestra.run.smithi092.stdout: Installing : kernel-modules-core-5.14.0-427.el9.x86_64 1/4 2024-03-11T21:07:21.161 INFO:teuthology.orchestra.run.smithi092.stdout: Installing : kernel-core-5.14.0-427.el9.x86_64 2/4 2024-03-11T21:07:21.644 INFO:teuthology.orchestra.run.smithi097.stdout: Running scriptlet: kernel-modules-5.14.0-427.el9.x86_64 3/4 2024-03-11T21:07:21.781 INFO:teuthology.orchestra.run.smithi097.stdout: Installing : kernel-5.14.0-427.el9.x86_64 4/4 2024-03-11T21:07:23.950 INFO:teuthology.orchestra.run.smithi092.stdout: Running scriptlet: kernel-core-5.14.0-427.el9.x86_64 2/4 2024-03-11T21:07:24.138 INFO:teuthology.orchestra.run.smithi092.stdout: Installing : kernel-modules-5.14.0-427.el9.x86_64 3/4 2024-03-11T21:07:26.512 INFO:teuthology.orchestra.run.smithi097.stdout: Running scriptlet: kernel-modules-core-5.14.0-427.el9.x86_64 4/4 2024-03-11T21:07:28.873 INFO:teuthology.orchestra.run.smithi092.stdout: Running scriptlet: kernel-modules-5.14.0-427.el9.x86_64 3/4 2024-03-11T21:07:29.073 INFO:teuthology.orchestra.run.smithi092.stdout: Installing : kernel-5.14.0-427.el9.x86_64 4/4 2024-03-11T21:07:33.854 INFO:teuthology.orchestra.run.smithi092.stdout: Running scriptlet: kernel-modules-core-5.14.0-427.el9.x86_64 4/4 2024-03-11T21:07:55.611 INFO:teuthology.orchestra.run.smithi097.stdout: Running scriptlet: kernel-core-5.14.0-427.el9.x86_64 4/4 2024-03-11T21:07:55.611 INFO:teuthology.orchestra.run.smithi097.stdout:dracut: Disabling early microcode, because kernel does not support it. CONFIG_MICROCODE_[AMD|INTEL]!=y 2024-03-11T21:07:55.611 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-03-11T21:07:55.696 INFO:teuthology.orchestra.run.smithi097.stdout: Running scriptlet: kernel-modules-5.14.0-427.el9.x86_64 4/4 2024-03-11T21:07:56.148 INFO:teuthology.orchestra.run.smithi097.stdout: Running scriptlet: kernel-5.14.0-427.el9.x86_64 4/4 2024-03-11T21:07:56.148 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : kernel-5.14.0-427.el9.x86_64 1/4 2024-03-11T21:07:56.148 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : kernel-core-5.14.0-427.el9.x86_64 2/4 2024-03-11T21:07:56.148 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : kernel-modules-5.14.0-427.el9.x86_64 3/4 2024-03-11T21:07:56.595 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : kernel-modules-core-5.14.0-427.el9.x86_64 4/4 2024-03-11T21:07:56.595 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-03-11T21:07:56.595 INFO:teuthology.orchestra.run.smithi097.stdout:Installed: 2024-03-11T21:07:56.595 INFO:teuthology.orchestra.run.smithi097.stdout: kernel-5.14.0-427.el9.x86_64 2024-03-11T21:07:56.595 INFO:teuthology.orchestra.run.smithi097.stdout: kernel-core-5.14.0-427.el9.x86_64 2024-03-11T21:07:56.595 INFO:teuthology.orchestra.run.smithi097.stdout: kernel-modules-5.14.0-427.el9.x86_64 2024-03-11T21:07:56.595 INFO:teuthology.orchestra.run.smithi097.stdout: kernel-modules-core-5.14.0-427.el9.x86_64 2024-03-11T21:07:56.596 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-03-11T21:07:56.596 INFO:teuthology.orchestra.run.smithi097.stdout:Complete! 2024-03-11T21:07:57.102 DEBUG:teuthology.orchestra.run.smithi097:> rpm -q kernel | sort -rV | head -n 1 2024-03-11T21:07:57.241 INFO:teuthology.orchestra.run.smithi097.stdout:kernel-5.14.0-427.el9.x86_64 2024-03-11T21:07:57.242 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-427.el9.x86_64 2024-03-11T21:07:57.242 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 5.14.0-361.el9.x86_64 Expected: 5.14.0-427.el9.x86_64 2024-03-11T21:07:57.242 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2024-03-11T21:07:57.242 DEBUG:teuthology.task.kernel:src is distro, skipping download 2024-03-11T21:07:57.242 INFO:teuthology.task.kernel:Installing distro kernel on mon.b... 2024-03-11T21:07:57.242 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi097.front.sepia.ceph.com, path=None, version=distro) 2024-03-11T21:07:57.243 DEBUG:teuthology.orchestra.run.smithi097:> rpm -q kernel | sort -rV | head -n 1 2024-03-11T21:07:57.364 INFO:teuthology.orchestra.run.smithi097.stdout:kernel-5.14.0-427.el9.x86_64 2024-03-11T21:07:57.364 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-427.el9.x86_64 2024-03-11T21:07:57.365 DEBUG:teuthology.orchestra.run.smithi097:> sudo rpm -qi grub2-tools 2024-03-11T21:07:57.398 INFO:teuthology.orchestra.run.smithi097.stdout:Name : grub2-tools 2024-03-11T21:07:57.398 INFO:teuthology.orchestra.run.smithi097.stdout:Epoch : 1 2024-03-11T21:07:57.398 INFO:teuthology.orchestra.run.smithi097.stdout:Version : 2.06 2024-03-11T21:07:57.398 INFO:teuthology.orchestra.run.smithi097.stdout:Release : 68.el9 2024-03-11T21:07:57.398 INFO:teuthology.orchestra.run.smithi097.stdout:Architecture: x86_64 2024-03-11T21:07:57.398 INFO:teuthology.orchestra.run.smithi097.stdout:Install Date: Fri 01 Sep 2023 09:14:30 PM UTC 2024-03-11T21:07:57.398 INFO:teuthology.orchestra.run.smithi097.stdout:Group : Unspecified 2024-03-11T21:07:57.398 INFO:teuthology.orchestra.run.smithi097.stdout:Size : 8264254 2024-03-11T21:07:57.399 INFO:teuthology.orchestra.run.smithi097.stdout:License : GPLv3+ 2024-03-11T21:07:57.399 INFO:teuthology.orchestra.run.smithi097.stdout:Signature : RSA/SHA256, Mon 07 Aug 2023 01:57:16 PM UTC, Key ID 05b555b38483c65d 2024-03-11T21:07:57.399 INFO:teuthology.orchestra.run.smithi097.stdout:Source RPM : grub2-2.06-68.el9.src.rpm 2024-03-11T21:07:57.399 INFO:teuthology.orchestra.run.smithi097.stdout:Build Date : Wed 02 Aug 2023 03:15:48 PM UTC 2024-03-11T21:07:57.399 INFO:teuthology.orchestra.run.smithi097.stdout:Build Host : x86-05.stream.rdu2.redhat.com 2024-03-11T21:07:57.399 INFO:teuthology.orchestra.run.smithi097.stdout:Packager : builder@centos.org 2024-03-11T21:07:57.399 INFO:teuthology.orchestra.run.smithi097.stdout:Vendor : CentOS 2024-03-11T21:07:57.399 INFO:teuthology.orchestra.run.smithi097.stdout:URL : http://www.gnu.org/software/grub/ 2024-03-11T21:07:57.399 INFO:teuthology.orchestra.run.smithi097.stdout:Summary : Support tools for GRUB. 2024-03-11T21:07:57.399 INFO:teuthology.orchestra.run.smithi097.stdout:Description : 2024-03-11T21:07:57.399 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-03-11T21:07:57.399 INFO:teuthology.orchestra.run.smithi097.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2024-03-11T21:07:57.399 INFO:teuthology.orchestra.run.smithi097.stdout:customizable bootloader with modular architecture. It supports a rich 2024-03-11T21:07:57.399 INFO:teuthology.orchestra.run.smithi097.stdout:variety of kernel formats, file systems, computer architectures and 2024-03-11T21:07:57.399 INFO:teuthology.orchestra.run.smithi097.stdout:hardware devices. 2024-03-11T21:07:57.400 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-03-11T21:07:57.400 INFO:teuthology.orchestra.run.smithi097.stdout:This subpackage provides tools for support of all platforms. 2024-03-11T21:07:57.400 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2024-03-11T21:07:57.401 INFO:teuthology.task.kernel:Updating grub on smithi097 to boot 5.14.0-427.el9.x86_64 2024-03-11T21:07:57.401 DEBUG:teuthology.orchestra.run.smithi097:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2024-03-11T21:07:57.999 INFO:teuthology.orchestra.run.smithi097.stderr:Generating grub configuration file ... 2024-03-11T21:07:59.362 INFO:teuthology.orchestra.run.smithi097.stderr:Adding boot menu entry for UEFI Firmware Settings ... 2024-03-11T21:07:59.400 INFO:teuthology.orchestra.run.smithi097.stderr:done 2024-03-11T21:07:59.402 DEBUG:teuthology.orchestra.run.smithi097:> mktemp 2024-03-11T21:07:59.418 INFO:teuthology.orchestra.run.smithi097.stdout:/tmp/tmp.c4rrwbLJh5 2024-03-11T21:07:59.418 DEBUG:teuthology.orchestra.run.smithi097:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.c4rrwbLJh5 2024-03-11T21:07:59.486 DEBUG:teuthology.orchestra.run.smithi097:> sudo chmod 0666 /tmp/tmp.c4rrwbLJh5 2024-03-11T21:07:59.617 DEBUG:teuthology.orchestra.remote:smithi097:/tmp/tmp.c4rrwbLJh5 is 6KB 2024-03-11T21:07:59.668 DEBUG:teuthology.orchestra.run.smithi097:> rm -fr /tmp/tmp.c4rrwbLJh5 2024-03-11T21:07:59.682 DEBUG:teuthology.orchestra.run.smithi097:> sudo /bin/ls /boot/loader/entries || true 2024-03-11T21:07:59.746 INFO:teuthology.orchestra.run.smithi097.stdout:15949e3c88704300882482eddfd7baae-0-rescue.conf 2024-03-11T21:07:59.746 INFO:teuthology.orchestra.run.smithi097.stdout:15949e3c88704300882482eddfd7baae-5.14.0-239.el9.x86_64.conf 2024-03-11T21:07:59.746 INFO:teuthology.orchestra.run.smithi097.stdout:15949e3c88704300882482eddfd7baae-5.14.0-361.el9.x86_64.conf 2024-03-11T21:07:59.746 INFO:teuthology.orchestra.run.smithi097.stdout:15949e3c88704300882482eddfd7baae-5.14.0-427.el9.x86_64.conf 2024-03-11T21:07:59.747 DEBUG:teuthology.orchestra.run.smithi097:> sudo grub2-set-default 15949e3c88704300882482eddfd7baae-5.14.0-427.el9.x86_64 2024-03-11T21:07:59.900 DEBUG:teuthology.orchestra.run.smithi097:> sudo shutdown -r now 2024-03-11T21:08:03.044 INFO:teuthology.orchestra.run.smithi092.stdout: Running scriptlet: kernel-core-5.14.0-427.el9.x86_64 4/4 2024-03-11T21:08:03.044 INFO:teuthology.orchestra.run.smithi092.stdout:dracut: Disabling early microcode, because kernel does not support it. CONFIG_MICROCODE_[AMD|INTEL]!=y 2024-03-11T21:08:03.044 INFO:teuthology.orchestra.run.smithi092.stdout: 2024-03-11T21:08:03.128 INFO:teuthology.orchestra.run.smithi092.stdout: Running scriptlet: kernel-modules-5.14.0-427.el9.x86_64 4/4 2024-03-11T21:08:03.581 INFO:teuthology.orchestra.run.smithi092.stdout: Running scriptlet: kernel-5.14.0-427.el9.x86_64 4/4 2024-03-11T21:08:03.581 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : kernel-5.14.0-427.el9.x86_64 1/4 2024-03-11T21:08:03.581 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : kernel-core-5.14.0-427.el9.x86_64 2/4 2024-03-11T21:08:03.581 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : kernel-modules-5.14.0-427.el9.x86_64 3/4 2024-03-11T21:08:04.000 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : kernel-modules-core-5.14.0-427.el9.x86_64 4/4 2024-03-11T21:08:04.000 INFO:teuthology.orchestra.run.smithi092.stdout: 2024-03-11T21:08:04.000 INFO:teuthology.orchestra.run.smithi092.stdout:Installed: 2024-03-11T21:08:04.000 INFO:teuthology.orchestra.run.smithi092.stdout: kernel-5.14.0-427.el9.x86_64 2024-03-11T21:08:04.000 INFO:teuthology.orchestra.run.smithi092.stdout: kernel-core-5.14.0-427.el9.x86_64 2024-03-11T21:08:04.001 INFO:teuthology.orchestra.run.smithi092.stdout: kernel-modules-5.14.0-427.el9.x86_64 2024-03-11T21:08:04.001 INFO:teuthology.orchestra.run.smithi092.stdout: kernel-modules-core-5.14.0-427.el9.x86_64 2024-03-11T21:08:04.001 INFO:teuthology.orchestra.run.smithi092.stdout: 2024-03-11T21:08:04.001 INFO:teuthology.orchestra.run.smithi092.stdout:Complete! 2024-03-11T21:08:04.543 DEBUG:teuthology.orchestra.run.smithi092:> rpm -q kernel | sort -rV | head -n 1 2024-03-11T21:08:04.682 INFO:teuthology.orchestra.run.smithi092.stdout:kernel-5.14.0-427.el9.x86_64 2024-03-11T21:08:04.682 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-427.el9.x86_64 2024-03-11T21:08:04.682 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 5.14.0-361.el9.x86_64 Expected: 5.14.0-427.el9.x86_64 2024-03-11T21:08:04.682 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2024-03-11T21:08:04.682 DEBUG:teuthology.task.kernel:src is distro, skipping download 2024-03-11T21:08:04.682 INFO:teuthology.task.kernel:Installing distro kernel on mon.a... 2024-03-11T21:08:04.683 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi092.front.sepia.ceph.com, path=None, version=distro) 2024-03-11T21:08:04.683 DEBUG:teuthology.orchestra.run.smithi092:> rpm -q kernel | sort -rV | head -n 1 2024-03-11T21:08:04.803 INFO:teuthology.orchestra.run.smithi092.stdout:kernel-5.14.0-427.el9.x86_64 2024-03-11T21:08:04.803 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-427.el9.x86_64 2024-03-11T21:08:04.803 DEBUG:teuthology.orchestra.run.smithi092:> sudo rpm -qi grub2-tools 2024-03-11T21:08:04.837 INFO:teuthology.orchestra.run.smithi092.stdout:Name : grub2-tools 2024-03-11T21:08:04.837 INFO:teuthology.orchestra.run.smithi092.stdout:Epoch : 1 2024-03-11T21:08:04.837 INFO:teuthology.orchestra.run.smithi092.stdout:Version : 2.06 2024-03-11T21:08:04.837 INFO:teuthology.orchestra.run.smithi092.stdout:Release : 68.el9 2024-03-11T21:08:04.837 INFO:teuthology.orchestra.run.smithi092.stdout:Architecture: x86_64 2024-03-11T21:08:04.837 INFO:teuthology.orchestra.run.smithi092.stdout:Install Date: Fri 01 Sep 2023 09:14:30 PM UTC 2024-03-11T21:08:04.838 INFO:teuthology.orchestra.run.smithi092.stdout:Group : Unspecified 2024-03-11T21:08:04.838 INFO:teuthology.orchestra.run.smithi092.stdout:Size : 8264254 2024-03-11T21:08:04.838 INFO:teuthology.orchestra.run.smithi092.stdout:License : GPLv3+ 2024-03-11T21:08:04.838 INFO:teuthology.orchestra.run.smithi092.stdout:Signature : RSA/SHA256, Mon 07 Aug 2023 01:57:16 PM UTC, Key ID 05b555b38483c65d 2024-03-11T21:08:04.838 INFO:teuthology.orchestra.run.smithi092.stdout:Source RPM : grub2-2.06-68.el9.src.rpm 2024-03-11T21:08:04.838 INFO:teuthology.orchestra.run.smithi092.stdout:Build Date : Wed 02 Aug 2023 03:15:48 PM UTC 2024-03-11T21:08:04.838 INFO:teuthology.orchestra.run.smithi092.stdout:Build Host : x86-05.stream.rdu2.redhat.com 2024-03-11T21:08:04.838 INFO:teuthology.orchestra.run.smithi092.stdout:Packager : builder@centos.org 2024-03-11T21:08:04.838 INFO:teuthology.orchestra.run.smithi092.stdout:Vendor : CentOS 2024-03-11T21:08:04.838 INFO:teuthology.orchestra.run.smithi092.stdout:URL : http://www.gnu.org/software/grub/ 2024-03-11T21:08:04.838 INFO:teuthology.orchestra.run.smithi092.stdout:Summary : Support tools for GRUB. 2024-03-11T21:08:04.838 INFO:teuthology.orchestra.run.smithi092.stdout:Description : 2024-03-11T21:08:04.838 INFO:teuthology.orchestra.run.smithi092.stdout: 2024-03-11T21:08:04.838 INFO:teuthology.orchestra.run.smithi092.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2024-03-11T21:08:04.839 INFO:teuthology.orchestra.run.smithi092.stdout:customizable bootloader with modular architecture. It supports a rich 2024-03-11T21:08:04.839 INFO:teuthology.orchestra.run.smithi092.stdout:variety of kernel formats, file systems, computer architectures and 2024-03-11T21:08:04.839 INFO:teuthology.orchestra.run.smithi092.stdout:hardware devices. 2024-03-11T21:08:04.839 INFO:teuthology.orchestra.run.smithi092.stdout: 2024-03-11T21:08:04.839 INFO:teuthology.orchestra.run.smithi092.stdout:This subpackage provides tools for support of all platforms. 2024-03-11T21:08:04.840 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2024-03-11T21:08:04.840 INFO:teuthology.task.kernel:Updating grub on smithi092 to boot 5.14.0-427.el9.x86_64 2024-03-11T21:08:04.840 DEBUG:teuthology.orchestra.run.smithi092:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2024-03-11T21:08:05.410 INFO:teuthology.orchestra.run.smithi092.stderr:Generating grub configuration file ... 2024-03-11T21:08:06.774 INFO:teuthology.orchestra.run.smithi092.stderr:Adding boot menu entry for UEFI Firmware Settings ... 2024-03-11T21:08:06.817 INFO:teuthology.orchestra.run.smithi092.stderr:done 2024-03-11T21:08:06.819 DEBUG:teuthology.orchestra.run.smithi092:> mktemp 2024-03-11T21:08:06.861 INFO:teuthology.orchestra.run.smithi092.stdout:/tmp/tmp.tOgnbN281J 2024-03-11T21:08:06.861 DEBUG:teuthology.orchestra.run.smithi092:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.tOgnbN281J 2024-03-11T21:08:06.929 DEBUG:teuthology.orchestra.run.smithi092:> sudo chmod 0666 /tmp/tmp.tOgnbN281J 2024-03-11T21:08:07.067 DEBUG:teuthology.orchestra.remote:smithi092:/tmp/tmp.tOgnbN281J is 6KB 2024-03-11T21:08:07.115 DEBUG:teuthology.orchestra.run.smithi092:> rm -fr /tmp/tmp.tOgnbN281J 2024-03-11T21:08:07.129 DEBUG:teuthology.orchestra.run.smithi092:> sudo /bin/ls /boot/loader/entries || true 2024-03-11T21:08:07.194 INFO:teuthology.orchestra.run.smithi092.stdout:15949e3c88704300882482eddfd7baae-0-rescue.conf 2024-03-11T21:08:07.194 INFO:teuthology.orchestra.run.smithi092.stdout:15949e3c88704300882482eddfd7baae-5.14.0-239.el9.x86_64.conf 2024-03-11T21:08:07.194 INFO:teuthology.orchestra.run.smithi092.stdout:15949e3c88704300882482eddfd7baae-5.14.0-361.el9.x86_64.conf 2024-03-11T21:08:07.195 INFO:teuthology.orchestra.run.smithi092.stdout:15949e3c88704300882482eddfd7baae-5.14.0-427.el9.x86_64.conf 2024-03-11T21:08:07.195 DEBUG:teuthology.orchestra.run.smithi092:> sudo grub2-set-default 15949e3c88704300882482eddfd7baae-5.14.0-427.el9.x86_64 2024-03-11T21:08:07.349 DEBUG:teuthology.orchestra.run.smithi092:> sudo shutdown -r now 2024-03-11T21:08:29.922 INFO:teuthology.task.kernel:Checking client mon.b for new kernel version... 2024-03-11T21:08:29.923 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi097.front.sepia.ceph.com' 2024-03-11T21:08:29.925 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi097.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-11T21:08:37.358 INFO:teuthology.task.kernel:Checking client mon.a for new kernel version... 2024-03-11T21:08:37.359 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi092.front.sepia.ceph.com' 2024-03-11T21:08:37.360 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi092.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-11T21:09:11.843 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.92 2024-03-11T21:09:20.850 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi092.front.sepia.ceph.com' 2024-03-11T21:09:20.851 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi092.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-11T21:09:29.934 DEBUG:teuthology.orchestra.remote:timed out 2024-03-11T21:09:38.942 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi097.front.sepia.ceph.com' 2024-03-11T21:09:38.943 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi097.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-11T21:09:39.234 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.92 2024-03-11T21:09:39.342 DEBUG:teuthology.orchestra.run.smithi097:> true 2024-03-11T21:09:39.949 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi097.front.sepia.ceph.com' 2024-03-11T21:09:39.950 INFO:teuthology.task.kernel:Checking kernel version of mon.b, want "5.14.0-427.el9.x86_64"... 2024-03-11T21:09:39.950 DEBUG:teuthology.orchestra.run.smithi097:> uname -r 2024-03-11T21:09:39.964 INFO:teuthology.orchestra.run.smithi097.stdout:5.14.0-427.el9.x86_64 2024-03-11T21:09:39.964 DEBUG:teuthology.task.kernel:current kernel version is 5.14.0-427.el9.x86_64 vs 5.14.0-427.el9.x86_64 2024-03-11T21:09:39.964 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2024-03-11T21:09:39.965 DEBUG:teuthology.task.kernel:Distro of this test job: centos 2024-03-11T21:09:40.965 INFO:teuthology.task.kernel:Enabling kdb on mon.b... 2024-03-11T21:09:40.965 DEBUG:teuthology.orchestra.run.smithi097:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2024-03-11T21:09:41.043 INFO:teuthology.orchestra.run.smithi097.stdout:ttyS1 2024-03-11T21:09:41.087 DEBUG:teuthology.parallel:result is None 2024-03-11T21:09:51.235 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi092.front.sepia.ceph.com' 2024-03-11T21:09:51.236 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi092.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-11T21:09:51.648 DEBUG:teuthology.orchestra.run.smithi092:> true 2024-03-11T21:09:52.250 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi092.front.sepia.ceph.com' 2024-03-11T21:09:52.250 INFO:teuthology.task.kernel:Checking kernel version of mon.a, want "5.14.0-427.el9.x86_64"... 2024-03-11T21:09:52.250 DEBUG:teuthology.orchestra.run.smithi092:> uname -r 2024-03-11T21:09:52.264 INFO:teuthology.orchestra.run.smithi092.stdout:5.14.0-427.el9.x86_64 2024-03-11T21:09:52.264 DEBUG:teuthology.task.kernel:current kernel version is 5.14.0-427.el9.x86_64 vs 5.14.0-427.el9.x86_64 2024-03-11T21:09:52.264 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2024-03-11T21:09:52.264 DEBUG:teuthology.task.kernel:Distro of this test job: centos 2024-03-11T21:09:53.265 INFO:teuthology.task.kernel:Enabling kdb on mon.a... 2024-03-11T21:09:53.266 DEBUG:teuthology.orchestra.run.smithi092:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2024-03-11T21:09:53.349 INFO:teuthology.orchestra.run.smithi092.stdout:ttyS1 2024-03-11T21:09:53.384 DEBUG:teuthology.parallel:result is None 2024-03-11T21:09:53.385 INFO:teuthology.run_tasks:Running task internal.base... 2024-03-11T21:09:53.391 INFO:teuthology.task.internal:Creating test directory... 2024-03-11T21:09:53.392 DEBUG:teuthology.orchestra.run.smithi092:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-03-11T21:09:53.394 DEBUG:teuthology.orchestra.run.smithi097:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-03-11T21:09:53.412 INFO:teuthology.run_tasks:Running task internal.archive_upload... 2024-03-11T21:09:53.463 INFO:teuthology.run_tasks:Running task internal.archive... 2024-03-11T21:09:53.472 INFO:teuthology.task.internal:Creating archive directory... 2024-03-11T21:09:53.472 DEBUG:teuthology.orchestra.run.smithi092:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-03-11T21:09:53.478 DEBUG:teuthology.orchestra.run.smithi097:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-03-11T21:09:53.519 INFO:teuthology.run_tasks:Running task internal.coredump... 2024-03-11T21:09:53.526 INFO:teuthology.task.internal:Enabling coredump saving... 2024-03-11T21:09:53.526 DEBUG:teuthology.orchestra.run.smithi092:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-03-11T21:09:53.541 DEBUG:teuthology.orchestra.run.smithi097:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-03-11T21:09:53.576 INFO:teuthology.orchestra.run.smithi092.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-11T21:09:53.586 INFO:teuthology.orchestra.run.smithi097.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-11T21:09:53.589 INFO:teuthology.orchestra.run.smithi092.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-11T21:09:53.600 INFO:teuthology.orchestra.run.smithi097.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-11T21:09:53.603 INFO:teuthology.run_tasks:Running task internal.sudo... 2024-03-11T21:09:53.613 INFO:teuthology.task.internal:Configuring sudo... 2024-03-11T21:09:53.613 DEBUG:teuthology.orchestra.run.smithi092:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-03-11T21:09:53.634 DEBUG:teuthology.orchestra.run.smithi097:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-03-11T21:09:53.671 INFO:teuthology.run_tasks:Running task internal.syslog... 2024-03-11T21:09:53.681 INFO:teuthology.task.internal.syslog:Starting syslog monitoring... 2024-03-11T21:09:53.681 DEBUG:teuthology.orchestra.run.smithi092:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-03-11T21:09:53.700 DEBUG:teuthology.orchestra.run.smithi097:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-03-11T21:09:53.728 DEBUG:teuthology.orchestra.run.smithi092:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-03-11T21:09:53.794 DEBUG:teuthology.orchestra.run.smithi092:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-03-11T21:09:53.879 DEBUG:teuthology.orchestra.run.smithi092:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-03-11T21:09:53.921 DEBUG:teuthology.orchestra.run.smithi092:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-03-11T21:09:53.964 DEBUG:teuthology.orchestra.run.smithi092:> set -ex 2024-03-11T21:09:53.964 DEBUG:teuthology.orchestra.run.smithi092:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-03-11T21:09:54.041 DEBUG:teuthology.orchestra.run.smithi097:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-03-11T21:09:54.084 DEBUG:teuthology.orchestra.run.smithi097:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-03-11T21:09:54.137 DEBUG:teuthology.orchestra.run.smithi097:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-03-11T21:09:54.222 DEBUG:teuthology.orchestra.run.smithi097:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-03-11T21:09:54.264 DEBUG:teuthology.orchestra.run.smithi097:> set -ex 2024-03-11T21:09:54.264 DEBUG:teuthology.orchestra.run.smithi097:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-03-11T21:09:54.332 DEBUG:teuthology.orchestra.run.smithi092:> sudo service rsyslog restart 2024-03-11T21:09:54.334 DEBUG:teuthology.orchestra.run.smithi097:> sudo service rsyslog restart 2024-03-11T21:09:54.382 INFO:teuthology.orchestra.run.smithi092.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-03-11T21:09:54.426 INFO:teuthology.orchestra.run.smithi097.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-03-11T21:09:54.869 INFO:teuthology.run_tasks:Running task internal.timer... 2024-03-11T21:09:54.876 INFO:teuthology.task.internal:Starting timer... 2024-03-11T21:09:54.877 INFO:teuthology.run_tasks:Running task pcp... 2024-03-11T21:09:54.893 INFO:teuthology.run_tasks:Running task selinux... 2024-03-11T21:09:54.902 DEBUG:teuthology.task:Applying overrides for task selinux: {'allowlist': ['scontext=system_u:system_r:logrotate_t:s0']} 2024-03-11T21:09:54.902 DEBUG:teuthology.orchestra.run.smithi092:> sudo service auditd rotate 2024-03-11T21:09:54.946 INFO:teuthology.orchestra.run.smithi092.stdout:Rotating logs: 2024-03-11T21:09:54.948 DEBUG:teuthology.orchestra.run.smithi097:> sudo service auditd rotate 2024-03-11T21:09:55.011 INFO:teuthology.orchestra.run.smithi097.stdout:Rotating logs: 2024-03-11T21:09:55.012 DEBUG:teuthology.task.selinux:Getting current SELinux state 2024-03-11T21:09:55.013 DEBUG:teuthology.orchestra.run.smithi092:> /usr/sbin/getenforce 2024-03-11T21:09:55.030 INFO:teuthology.orchestra.run.smithi092.stdout:Permissive 2024-03-11T21:09:55.031 DEBUG:teuthology.orchestra.run.smithi097:> /usr/sbin/getenforce 2024-03-11T21:09:55.052 INFO:teuthology.orchestra.run.smithi097.stdout:Permissive 2024-03-11T21:09:55.052 DEBUG:teuthology.task.selinux:Existing SELinux modes: {'ubuntu@smithi092.front.sepia.ceph.com': 'permissive', 'ubuntu@smithi097.front.sepia.ceph.com': 'permissive'} 2024-03-11T21:09:55.052 DEBUG:teuthology.orchestra.run.smithi092:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e scontext=system_u:system_r:logrotate_t:s0 2024-03-11T21:09:55.098 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:09:55.098 DEBUG:teuthology.orchestra.run.smithi097:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e scontext=system_u:system_r:logrotate_t:s0 2024-03-11T21:09:55.126 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:09:55.127 INFO:teuthology.task.selinux:Putting SELinux into permissive mode 2024-03-11T21:09:55.127 DEBUG:teuthology.orchestra.run.smithi092:> sudo /usr/sbin/setenforce permissive 2024-03-11T21:09:55.182 DEBUG:teuthology.orchestra.run.smithi097:> sudo /usr/sbin/setenforce permissive 2024-03-11T21:09:55.221 INFO:teuthology.run_tasks:Running task ansible.cephlab... 2024-03-11T21:09:55.231 DEBUG:teuthology.repo_utils:Resetting repo at /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main to origin/main 2024-03-11T21:09:55.248 INFO:teuthology.task.ansible:Playbook: [{'import_playbook': 'ansible_managed.yml'}, {'import_playbook': 'teuthology.yml'}, {'hosts': 'testnodes', 'tasks': [{'set_fact': {'ran_from_cephlab_playbook': True}}]}, {'import_playbook': 'testnodes.yml'}, {'import_playbook': 'container-host.yml'}, {'import_playbook': 'cobbler.yml'}, {'import_playbook': 'paddles.yml'}, {'import_playbook': 'pulpito.yml'}, {'hosts': 'testnodes', 'become': True, 'tasks': [{'name': 'Touch /ceph-qa-ready', 'file': {'path': '/ceph-qa-ready', 'state': 'touch'}, 'when': 'ran_from_cephlab_playbook|bool'}]}] 2024-03-11T21:09:55.250 DEBUG:teuthology.task.ansible:Running ansible-playbook -v --extra-vars '{"ansible_ssh_user": "ubuntu"}' -i /etc/ansible/hosts --limit smithi092.front.sepia.ceph.com,smithi097.front.sepia.ceph.com /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main/cephlab.yml 2024-03-11T21:14:08.675 DEBUG:teuthology.task.ansible:Reconnecting to [Remote(name='ubuntu@smithi092.front.sepia.ceph.com'), Remote(name='ubuntu@smithi097.front.sepia.ceph.com')] 2024-03-11T21:14:08.676 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi092.front.sepia.ceph.com' 2024-03-11T21:14:08.677 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi092.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-11T21:14:08.768 DEBUG:teuthology.orchestra.run.smithi092:> true 2024-03-11T21:14:08.845 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi092.front.sepia.ceph.com' 2024-03-11T21:14:08.845 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi097.front.sepia.ceph.com' 2024-03-11T21:14:08.846 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi097.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-11T21:14:08.936 DEBUG:teuthology.orchestra.run.smithi097:> true 2024-03-11T21:14:09.016 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi097.front.sepia.ceph.com' 2024-03-11T21:14:09.017 INFO:teuthology.run_tasks:Running task clock... 2024-03-11T21:14:09.029 INFO:teuthology.task.clock:Syncing clocks and checking initial clock skew... 2024-03-11T21:14:09.029 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-03-11T21:14:09.029 DEBUG:teuthology.orchestra.run.smithi092:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-03-11T21:14:09.032 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-03-11T21:14:09.032 DEBUG:teuthology.orchestra.run.smithi097:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-03-11T21:14:09.061 INFO:teuthology.orchestra.run.smithi092.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-03-11T21:14:09.078 INFO:teuthology.orchestra.run.smithi092.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-03-11T21:14:09.086 INFO:teuthology.orchestra.run.smithi097.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-03-11T21:14:09.103 INFO:teuthology.orchestra.run.smithi097.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-03-11T21:14:09.107 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: ntpd: command not found 2024-03-11T21:14:09.121 INFO:teuthology.orchestra.run.smithi092.stdout:506 Cannot talk to daemon 2024-03-11T21:14:09.132 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: ntpd: command not found 2024-03-11T21:14:09.137 INFO:teuthology.orchestra.run.smithi092.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-03-11T21:14:09.145 INFO:teuthology.orchestra.run.smithi097.stdout:506 Cannot talk to daemon 2024-03-11T21:14:09.154 INFO:teuthology.orchestra.run.smithi092.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-03-11T21:14:09.162 INFO:teuthology.orchestra.run.smithi097.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-03-11T21:14:09.178 INFO:teuthology.orchestra.run.smithi097.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-03-11T21:14:09.205 INFO:teuthology.orchestra.run.smithi092.stderr:bash: line 1: ntpq: command not found 2024-03-11T21:14:09.208 INFO:teuthology.orchestra.run.smithi092.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-03-11T21:14:09.208 INFO:teuthology.orchestra.run.smithi092.stdout:=============================================================================== 2024-03-11T21:14:09.208 INFO:teuthology.orchestra.run.smithi092.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-11T21:14:09.208 INFO:teuthology.orchestra.run.smithi092.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-11T21:14:09.208 INFO:teuthology.orchestra.run.smithi092.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-11T21:14:09.208 INFO:teuthology.orchestra.run.smithi092.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-11T21:14:09.228 INFO:teuthology.orchestra.run.smithi097.stderr:bash: line 1: ntpq: command not found 2024-03-11T21:14:09.232 INFO:teuthology.orchestra.run.smithi097.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-03-11T21:14:09.232 INFO:teuthology.orchestra.run.smithi097.stdout:=============================================================================== 2024-03-11T21:14:09.232 INFO:teuthology.orchestra.run.smithi097.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-11T21:14:09.232 INFO:teuthology.orchestra.run.smithi097.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-11T21:14:09.232 INFO:teuthology.orchestra.run.smithi097.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-11T21:14:09.232 INFO:teuthology.orchestra.run.smithi097.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-11T21:14:09.233 INFO:teuthology.run_tasks:Running task install... 2024-03-11T21:14:09.241 DEBUG:teuthology.task.install:project ceph 2024-03-11T21:14:09.241 DEBUG:teuthology.task.install:INSTALL overrides: {'ceph': {'flavor': 'default', 'sha1': '2b0101eb74d60610d544cb8a473e23a35ac1cc0b'}} 2024-03-11T21:14:09.241 DEBUG:teuthology.task.install:config {'flavor': 'default', 'sha1': '2b0101eb74d60610d544cb8a473e23a35ac1cc0b'} 2024-03-11T21:14:09.242 INFO:teuthology.task.install:Using flavor: default 2024-03-11T21:14:09.247 DEBUG:teuthology.task.install:Package list is: {'deb': ['ceph', 'cephadm', 'ceph-mds', 'ceph-mgr', 'ceph-common', 'ceph-fuse', 'ceph-test', 'ceph-volume', 'radosgw', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'libcephfs2', 'libcephfs-dev', 'librados2', 'librbd1', 'rbd-fuse'], 'rpm': ['ceph-radosgw', 'ceph-test', 'ceph', 'ceph-base', 'cephadm', 'ceph-immutable-object-cache', 'ceph-mgr', 'ceph-mgr-dashboard', 'ceph-mgr-diskprediction-local', 'ceph-mgr-rook', 'ceph-mgr-cephadm', 'ceph-fuse', 'ceph-volume', 'librados-devel', 'libcephfs2', 'libcephfs-devel', 'librados2', 'librbd1', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'rbd-fuse', 'rbd-mirror', 'rbd-nbd']} 2024-03-11T21:14:09.247 INFO:teuthology.task.install:extra packages: [] 2024-03-11T21:14:09.248 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': [], 'extra_system_packages': [], 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': '2b0101eb74d60610d544cb8a473e23a35ac1cc0b', 'tag': None, 'wait_for_package': False} 2024-03-11T21:14:09.248 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=2b0101eb74d60610d544cb8a473e23a35ac1cc0b 2024-03-11T21:14:09.250 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': [], 'extra_system_packages': [], 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': '2b0101eb74d60610d544cb8a473e23a35ac1cc0b', 'tag': None, 'wait_for_package': False} 2024-03-11T21:14:09.251 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=2b0101eb74d60610d544cb8a473e23a35ac1cc0b 2024-03-11T21:14:09.391 INFO:teuthology.task.install.rpm:Pulling from https://3.chacra.ceph.com/r/ceph/main/2b0101eb74d60610d544cb8a473e23a35ac1cc0b/centos/9/flavors/default/ 2024-03-11T21:14:09.392 INFO:teuthology.task.install.rpm:Package version is 19.0.0-1979.g2b0101eb 2024-03-11T21:14:09.503 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://3.chacra.ceph.com/r/ceph/main/2b0101eb74d60610d544cb8a473e23a35ac1cc0b/centos/9/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://3.chacra.ceph.com/r/ceph/main/2b0101eb74d60610d544cb8a473e23a35ac1cc0b/centos/9/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://3.chacra.ceph.com/r/ceph/main/2b0101eb74d60610d544cb8a473e23a35ac1cc0b/centos/9/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2024-03-11T21:14:09.503 DEBUG:teuthology.orchestra.run.smithi097:> set -ex 2024-03-11T21:14:09.503 DEBUG:teuthology.orchestra.run.smithi097:> sudo dd of=/etc/yum.repos.d/ceph.repo 2024-03-11T21:14:09.530 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd on remote rpm x86_64 2024-03-11T21:14:09.530 DEBUG:teuthology.orchestra.run.smithi097:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/2b0101eb74d60610d544cb8a473e23a35ac1cc0b/;g' /etc/yum.repos.d/ceph.repo ; fi 2024-03-11T21:14:09.598 DEBUG:teuthology.orchestra.run.smithi097:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2024-03-11T21:14:09.619 INFO:teuthology.task.install.rpm:Pulling from https://3.chacra.ceph.com/r/ceph/main/2b0101eb74d60610d544cb8a473e23a35ac1cc0b/centos/9/flavors/default/ 2024-03-11T21:14:09.620 INFO:teuthology.task.install.rpm:Package version is 19.0.0-1979.g2b0101eb 2024-03-11T21:14:09.678 DEBUG:teuthology.orchestra.run.smithi097:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2024-03-11T21:14:09.745 INFO:teuthology.orchestra.run.smithi097.stdout:check_obsoletes = 1 2024-03-11T21:14:09.746 DEBUG:teuthology.orchestra.run.smithi097:> sudo yum clean all 2024-03-11T21:14:09.947 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://3.chacra.ceph.com/r/ceph/main/2b0101eb74d60610d544cb8a473e23a35ac1cc0b/centos/9/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://3.chacra.ceph.com/r/ceph/main/2b0101eb74d60610d544cb8a473e23a35ac1cc0b/centos/9/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://3.chacra.ceph.com/r/ceph/main/2b0101eb74d60610d544cb8a473e23a35ac1cc0b/centos/9/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2024-03-11T21:14:09.948 DEBUG:teuthology.orchestra.run.smithi092:> set -ex 2024-03-11T21:14:09.948 DEBUG:teuthology.orchestra.run.smithi092:> sudo dd of=/etc/yum.repos.d/ceph.repo 2024-03-11T21:14:09.978 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd on remote rpm x86_64 2024-03-11T21:14:09.978 DEBUG:teuthology.orchestra.run.smithi092:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/2b0101eb74d60610d544cb8a473e23a35ac1cc0b/;g' /etc/yum.repos.d/ceph.repo ; fi 2024-03-11T21:14:09.983 INFO:teuthology.orchestra.run.smithi097.stdout:74 files removed 2024-03-11T21:14:10.009 DEBUG:teuthology.orchestra.run.smithi097:> sudo yum -y install ceph-radosgw 2024-03-11T21:14:10.046 DEBUG:teuthology.orchestra.run.smithi092:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2024-03-11T21:14:10.127 DEBUG:teuthology.orchestra.run.smithi092:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2024-03-11T21:14:10.193 INFO:teuthology.orchestra.run.smithi092.stdout:check_obsoletes = 1 2024-03-11T21:14:10.194 DEBUG:teuthology.orchestra.run.smithi092:> sudo yum clean all 2024-03-11T21:14:10.433 INFO:teuthology.orchestra.run.smithi092.stdout:74 files removed 2024-03-11T21:14:10.460 DEBUG:teuthology.orchestra.run.smithi092:> sudo yum -y install ceph-radosgw 2024-03-11T21:14:11.021 INFO:teuthology.orchestra.run.smithi097.stdout:ceph packages for x86_64 118 kB/s | 82 kB 00:00 2024-03-11T21:14:11.343 INFO:teuthology.orchestra.run.smithi092.stdout:ceph packages for x86_64 145 kB/s | 82 kB 00:00 2024-03-11T21:14:11.646 INFO:teuthology.orchestra.run.smithi097.stdout:ceph noarch packages 17 kB/s | 10 kB 00:00 2024-03-11T21:14:11.652 INFO:teuthology.orchestra.run.smithi092.stdout:ceph noarch packages 39 kB/s | 10 kB 00:00 2024-03-11T21:14:11.969 INFO:teuthology.orchestra.run.smithi092.stdout:ceph source packages 6.8 kB/s | 1.9 kB 00:00 2024-03-11T21:14:12.113 INFO:teuthology.orchestra.run.smithi097.stdout:ceph source packages 4.4 kB/s | 1.9 kB 00:00 2024-03-11T21:14:12.413 INFO:teuthology.orchestra.run.smithi092.stdout:CentOS-9 - AppStream 16 MB/s | 6.7 MB 00:00 2024-03-11T21:14:12.666 INFO:teuthology.orchestra.run.smithi097.stdout:CentOS-9 - AppStream 13 MB/s | 6.7 MB 00:00 2024-03-11T21:14:13.830 INFO:teuthology.orchestra.run.smithi092.stdout:CentOS-9 - BaseOS 6.6 MB/s | 2.0 MB 00:00 2024-03-11T21:14:14.191 INFO:teuthology.orchestra.run.smithi097.stdout:CentOS-9 - BaseOS 4.8 MB/s | 2.0 MB 00:00 2024-03-11T21:14:15.840 INFO:teuthology.orchestra.run.smithi092.stdout:CentOS Stream 9 - BaseOS 4.8 MB/s | 8.0 MB 00:01 2024-03-11T21:14:16.782 INFO:teuthology.orchestra.run.smithi097.stdout:CentOS Stream 9 - BaseOS 3.6 MB/s | 8.0 MB 00:02 2024-03-11T21:14:33.119 INFO:teuthology.orchestra.run.smithi092.stdout:CentOS Stream 9 - AppStream 1.2 MB/s | 19 MB 00:16 2024-03-11T21:14:41.205 INFO:teuthology.orchestra.run.smithi097.stdout:CentOS Stream 9 - AppStream 826 kB/s | 19 MB 00:23 2024-03-11T21:14:47.825 INFO:teuthology.orchestra.run.smithi092.stdout:CentOS Stream 9 - CRB 559 kB/s | 6.1 MB 00:11 2024-03-11T21:14:49.768 INFO:teuthology.orchestra.run.smithi092.stdout:CentOS Stream 9 - Extras packages 28 kB/s | 15 kB 00:00 2024-03-11T21:14:51.928 INFO:teuthology.orchestra.run.smithi092.stdout:Extra Packages for Enterprise Linux 10 MB/s | 21 MB 00:02 2024-03-11T21:14:56.055 INFO:teuthology.orchestra.run.smithi097.stdout:CentOS Stream 9 - CRB 551 kB/s | 6.1 MB 00:11 2024-03-11T21:14:56.869 INFO:teuthology.orchestra.run.smithi092.stdout:lab-extras 14 kB/s | 1.7 kB 00:00 2024-03-11T21:14:57.731 INFO:teuthology.orchestra.run.smithi097.stdout:CentOS Stream 9 - Extras packages 50 kB/s | 15 kB 00:00 2024-03-11T21:14:59.353 INFO:teuthology.orchestra.run.smithi092.stdout:Dependencies resolved. 2024-03-11T21:14:59.360 INFO:teuthology.orchestra.run.smithi092.stdout:================================================================================ 2024-03-11T21:14:59.360 INFO:teuthology.orchestra.run.smithi092.stdout: Package Arch Version Repository Size 2024-03-11T21:14:59.360 INFO:teuthology.orchestra.run.smithi092.stdout:================================================================================ 2024-03-11T21:14:59.360 INFO:teuthology.orchestra.run.smithi092.stdout:Installing: 2024-03-11T21:14:59.361 INFO:teuthology.orchestra.run.smithi092.stdout: ceph-radosgw x86_64 2:19.0.0-1979.g2b0101eb.el9 ceph 9.8 M 2024-03-11T21:14:59.361 INFO:teuthology.orchestra.run.smithi092.stdout:Upgrading: 2024-03-11T21:14:59.361 INFO:teuthology.orchestra.run.smithi092.stdout: librados2 x86_64 2:19.0.0-1979.g2b0101eb.el9 ceph 3.4 M 2024-03-11T21:14:59.361 INFO:teuthology.orchestra.run.smithi092.stdout: librbd1 x86_64 2:19.0.0-1979.g2b0101eb.el9 ceph 3.1 M 2024-03-11T21:14:59.361 INFO:teuthology.orchestra.run.smithi092.stdout:Installing dependencies: 2024-03-11T21:14:59.361 INFO:teuthology.orchestra.run.smithi092.stdout: boost-program-options 2024-03-11T21:14:59.361 INFO:teuthology.orchestra.run.smithi092.stdout: x86_64 1.75.0-8.el9 CentOS-AppStream 106 k 2024-03-11T21:14:59.361 INFO:teuthology.orchestra.run.smithi092.stdout: ceph-base x86_64 2:19.0.0-1979.g2b0101eb.el9 ceph 5.3 M 2024-03-11T21:14:59.361 INFO:teuthology.orchestra.run.smithi092.stdout: ceph-common x86_64 2:19.0.0-1979.g2b0101eb.el9 ceph 20 M 2024-03-11T21:14:59.361 INFO:teuthology.orchestra.run.smithi092.stdout: ceph-selinux x86_64 2:19.0.0-1979.g2b0101eb.el9 ceph 25 k 2024-03-11T21:14:59.361 INFO:teuthology.orchestra.run.smithi092.stdout: gperftools-libs x86_64 2.9.1-2.el9 epel 309 k 2024-03-11T21:14:59.361 INFO:teuthology.orchestra.run.smithi092.stdout: libarrow x86_64 9.0.0-10.el9 epel 4.4 M 2024-03-11T21:14:59.361 INFO:teuthology.orchestra.run.smithi092.stdout: libarrow-doc noarch 9.0.0-10.el9 epel 26 k 2024-03-11T21:14:59.362 INFO:teuthology.orchestra.run.smithi092.stdout: libcephfs2 x86_64 2:19.0.0-1979.g2b0101eb.el9 ceph 745 k 2024-03-11T21:14:59.362 INFO:teuthology.orchestra.run.smithi092.stdout: liboath x86_64 2.6.7-2.el9 epel 49 k 2024-03-11T21:14:59.362 INFO:teuthology.orchestra.run.smithi092.stdout: libpmemobj x86_64 1.12.1-1.el9 CentOS-AppStream 159 k 2024-03-11T21:14:59.362 INFO:teuthology.orchestra.run.smithi092.stdout: librabbitmq x86_64 0.11.0-7.el9 CentOS-AppStream 44 k 2024-03-11T21:14:59.362 INFO:teuthology.orchestra.run.smithi092.stdout: libradosstriper1 x86_64 2:19.0.0-1979.g2b0101eb.el9 ceph 501 k 2024-03-11T21:14:59.362 INFO:teuthology.orchestra.run.smithi092.stdout: librdkafka x86_64 1.6.1-102.el9 CentOS-AppStream 661 k 2024-03-11T21:14:59.362 INFO:teuthology.orchestra.run.smithi092.stdout: librgw2 x86_64 2:19.0.0-1979.g2b0101eb.el9 ceph 5.2 M 2024-03-11T21:14:59.363 INFO:teuthology.orchestra.run.smithi092.stdout: libunwind x86_64 1.6.2-1.el9 epel 67 k 2024-03-11T21:14:59.363 INFO:teuthology.orchestra.run.smithi092.stdout: lttng-ust x86_64 2.12.0-6.el9 CentOS-AppStream 282 k 2024-03-11T21:14:59.363 INFO:teuthology.orchestra.run.smithi092.stdout: parquet-libs x86_64 9.0.0-10.el9 epel 838 k 2024-03-11T21:14:59.363 INFO:teuthology.orchestra.run.smithi092.stdout: python3-ceph-argparse 2024-03-11T21:14:59.363 INFO:teuthology.orchestra.run.smithi092.stdout: x86_64 2:19.0.0-1979.g2b0101eb.el9 ceph 45 k 2024-03-11T21:14:59.363 INFO:teuthology.orchestra.run.smithi092.stdout: python3-ceph-common x86_64 2:19.0.0-1979.g2b0101eb.el9 ceph 131 k 2024-03-11T21:14:59.363 INFO:teuthology.orchestra.run.smithi092.stdout: python3-cephfs x86_64 2:19.0.0-1979.g2b0101eb.el9 ceph 162 k 2024-03-11T21:14:59.363 INFO:teuthology.orchestra.run.smithi092.stdout: python3-prettytable noarch 0.7.2-27.el9 CentOS-AppStream 41 k 2024-03-11T21:14:59.363 INFO:teuthology.orchestra.run.smithi092.stdout: python3-rados x86_64 2:19.0.0-1979.g2b0101eb.el9 ceph 319 k 2024-03-11T21:14:59.363 INFO:teuthology.orchestra.run.smithi092.stdout: python3-rbd x86_64 2:19.0.0-1979.g2b0101eb.el9 ceph 298 k 2024-03-11T21:14:59.363 INFO:teuthology.orchestra.run.smithi092.stdout: python3-rgw x86_64 2:19.0.0-1979.g2b0101eb.el9 ceph 99 k 2024-03-11T21:14:59.363 INFO:teuthology.orchestra.run.smithi092.stdout: qatlib x86_64 23.11.0-1.el9 appstream 216 k 2024-03-11T21:14:59.363 INFO:teuthology.orchestra.run.smithi092.stdout: qatzip-libs x86_64 1.1.2-2.el9 appstream 44 k 2024-03-11T21:14:59.364 INFO:teuthology.orchestra.run.smithi092.stdout: re2 x86_64 1:20211101-3.el9 epel 192 k 2024-03-11T21:14:59.364 INFO:teuthology.orchestra.run.smithi092.stdout: thrift x86_64 0.15.0-2.el9 epel 1.6 M 2024-03-11T21:14:59.364 INFO:teuthology.orchestra.run.smithi092.stdout:Installing weak dependencies: 2024-03-11T21:14:59.364 INFO:teuthology.orchestra.run.smithi092.stdout: qatlib-service x86_64 23.11.0-1.el9 appstream 35 k 2024-03-11T21:14:59.364 INFO:teuthology.orchestra.run.smithi092.stdout: 2024-03-11T21:14:59.364 INFO:teuthology.orchestra.run.smithi092.stdout:Transaction Summary 2024-03-11T21:14:59.364 INFO:teuthology.orchestra.run.smithi092.stdout:================================================================================ 2024-03-11T21:14:59.364 INFO:teuthology.orchestra.run.smithi092.stdout:Install 30 Packages 2024-03-11T21:14:59.364 INFO:teuthology.orchestra.run.smithi092.stdout:Upgrade 2 Packages 2024-03-11T21:14:59.364 INFO:teuthology.orchestra.run.smithi092.stdout: 2024-03-11T21:14:59.365 INFO:teuthology.orchestra.run.smithi092.stdout:Total download size: 58 M 2024-03-11T21:14:59.365 INFO:teuthology.orchestra.run.smithi092.stdout:Downloading Packages: 2024-03-11T21:15:01.261 INFO:teuthology.orchestra.run.smithi092.stdout:(1/32): ceph-base-19.0.0-1979.g2b0101eb.el9.x86 4.1 MB/s | 5.3 MB 00:01 2024-03-11T21:15:01.311 INFO:teuthology.orchestra.run.smithi092.stdout:(2/32): ceph-selinux-19.0.0-1979.g2b0101eb.el9. 504 kB/s | 25 kB 00:00 2024-03-11T21:15:01.420 INFO:teuthology.orchestra.run.smithi092.stdout:(3/32): libcephfs2-19.0.0-1979.g2b0101eb.el9.x8 6.7 MB/s | 745 kB 00:00 2024-03-11T21:15:01.520 INFO:teuthology.orchestra.run.smithi092.stdout:(4/32): libradosstriper1-19.0.0-1979.g2b0101eb. 4.9 MB/s | 501 kB 00:00 2024-03-11T21:15:01.621 INFO:teuthology.orchestra.run.smithi092.stdout:(5/32): ceph-radosgw-19.0.0-1979.g2b0101eb.el9. 5.9 MB/s | 9.8 MB 00:01 2024-03-11T21:15:01.704 INFO:teuthology.orchestra.run.smithi092.stdout:(6/32): python3-ceph-argparse-19.0.0-1979.g2b01 543 kB/s | 45 kB 00:00 2024-03-11T21:15:01.779 INFO:teuthology.orchestra.run.smithi092.stdout:(7/32): python3-ceph-common-19.0.0-1979.g2b0101 1.7 MB/s | 131 kB 00:00 2024-03-11T21:15:01.846 INFO:teuthology.orchestra.run.smithi092.stdout:(8/32): python3-cephfs-19.0.0-1979.g2b0101eb.el 2.4 MB/s | 162 kB 00:00 2024-03-11T21:15:01.922 INFO:teuthology.orchestra.run.smithi092.stdout:(9/32): python3-rados-19.0.0-1979.g2b0101eb.el9 4.2 MB/s | 319 kB 00:00 2024-03-11T21:15:01.988 INFO:teuthology.orchestra.run.smithi092.stdout:(10/32): python3-rbd-19.0.0-1979.g2b0101eb.el9. 4.4 MB/s | 298 kB 00:00 2024-03-11T21:15:02.080 INFO:teuthology.orchestra.run.smithi092.stdout:(11/32): librgw2-19.0.0-1979.g2b0101eb.el9.x86_ 9.2 MB/s | 5.2 MB 00:00 2024-03-11T21:15:02.105 INFO:teuthology.orchestra.run.smithi092.stdout:(12/32): python3-rgw-19.0.0-1979.g2b0101eb.el9. 851 kB/s | 99 kB 00:00 2024-03-11T21:15:02.248 INFO:teuthology.orchestra.run.smithi092.stdout:(13/32): boost-program-options-1.75.0-8.el9.x86 635 kB/s | 106 kB 00:00 2024-03-11T21:15:02.273 INFO:teuthology.orchestra.run.smithi092.stdout:(14/32): libpmemobj-1.12.1-1.el9.x86_64.rpm 953 kB/s | 159 kB 00:00 2024-03-11T21:15:02.298 INFO:teuthology.orchestra.run.smithi092.stdout:(15/32): librabbitmq-0.11.0-7.el9.x86_64.rpm 883 kB/s | 44 kB 00:00 2024-03-11T21:15:02.365 INFO:teuthology.orchestra.run.smithi092.stdout:(16/32): librdkafka-1.6.1-102.el9.x86_64.rpm 7.0 MB/s | 661 kB 00:00 2024-03-11T21:15:02.391 INFO:teuthology.orchestra.run.smithi092.stdout:(17/32): lttng-ust-2.12.0-6.el9.x86_64.rpm 3.0 MB/s | 282 kB 00:00 2024-03-11T21:15:02.416 INFO:teuthology.orchestra.run.smithi092.stdout:(18/32): python3-prettytable-0.7.2-27.el9.noarc 825 kB/s | 41 kB 00:00 2024-03-11T21:15:02.716 INFO:teuthology.orchestra.run.smithi092.stdout:(19/32): qatlib-service-23.11.0-1.el9.x86_64.rp 116 kB/s | 35 kB 00:00 2024-03-11T21:15:02.858 INFO:teuthology.orchestra.run.smithi092.stdout:(20/32): qatzip-libs-1.1.2-2.el9.x86_64.rpm 315 kB/s | 44 kB 00:00 2024-03-11T21:15:03.217 INFO:teuthology.orchestra.run.smithi092.stdout:(21/32): qatlib-23.11.0-1.el9.x86_64.rpm 262 kB/s | 216 kB 00:00 2024-03-11T21:15:03.309 INFO:teuthology.orchestra.run.smithi092.stdout:(22/32): gperftools-libs-2.9.1-2.el9.x86_64.rpm 686 kB/s | 309 kB 00:00 2024-03-11T21:15:03.384 INFO:teuthology.orchestra.run.smithi092.stdout:(23/32): libarrow-doc-9.0.0-10.el9.noarch.rpm 347 kB/s | 26 kB 00:00 2024-03-11T21:15:03.459 INFO:teuthology.orchestra.run.smithi092.stdout:(24/32): liboath-2.6.7-2.el9.x86_64.rpm 648 kB/s | 49 kB 00:00 2024-03-11T21:15:03.560 INFO:teuthology.orchestra.run.smithi092.stdout:(25/32): libunwind-1.6.2-1.el9.x86_64.rpm 675 kB/s | 67 kB 00:00 2024-03-11T21:15:03.777 INFO:teuthology.orchestra.run.smithi092.stdout:(26/32): parquet-libs-9.0.0-10.el9.x86_64.rpm 3.8 MB/s | 838 kB 00:00 2024-03-11T21:15:03.868 INFO:teuthology.orchestra.run.smithi092.stdout:(27/32): re2-20211101-3.el9.x86_64.rpm 2.0 MB/s | 192 kB 00:00 2024-03-11T21:15:04.132 INFO:teuthology.orchestra.run.smithi092.stdout:(28/32): libarrow-9.0.0-10.el9.x86_64.rpm 4.8 MB/s | 4.4 MB 00:00 2024-03-11T21:15:04.327 INFO:teuthology.orchestra.run.smithi092.stdout:(29/32): ceph-common-19.0.0-1979.g2b0101eb.el9. 4.6 MB/s | 20 MB 00:04 2024-03-11T21:15:04.369 INFO:teuthology.orchestra.run.smithi092.stdout:(30/32): thrift-0.15.0-2.el9.x86_64.rpm 3.2 MB/s | 1.6 MB 00:00 2024-03-11T21:15:04.537 INFO:teuthology.orchestra.run.smithi092.stdout:(31/32): librados2-19.0.0-1979.g2b0101eb.el9.x8 8.4 MB/s | 3.4 MB 00:00 2024-03-11T21:15:05.053 INFO:teuthology.orchestra.run.smithi092.stdout:(32/32): librbd1-19.0.0-1979.g2b0101eb.el9.x86_ 4.3 MB/s | 3.1 MB 00:00 2024-03-11T21:15:05.056 INFO:teuthology.orchestra.run.smithi092.stdout:-------------------------------------------------------------------------------- 2024-03-11T21:15:05.056 INFO:teuthology.orchestra.run.smithi092.stdout:Total 10 MB/s | 58 MB 00:05 2024-03-11T21:15:05.088 INFO:teuthology.orchestra.run.smithi092.stdout:Running transaction check 2024-03-11T21:15:05.125 INFO:teuthology.orchestra.run.smithi092.stdout:Transaction check succeeded. 2024-03-11T21:15:05.125 INFO:teuthology.orchestra.run.smithi092.stdout:Running transaction test 2024-03-11T21:15:05.591 INFO:teuthology.orchestra.run.smithi092.stdout:Transaction test succeeded. 2024-03-11T21:15:05.592 INFO:teuthology.orchestra.run.smithi092.stdout:Running transaction 2024-03-11T21:15:07.038 INFO:teuthology.orchestra.run.smithi092.stdout: Preparing : 1/1 2024-03-11T21:15:07.131 INFO:teuthology.orchestra.run.smithi092.stdout: Installing : lttng-ust-2.12.0-6.el9.x86_64 1/34 2024-03-11T21:15:07.209 INFO:teuthology.orchestra.run.smithi092.stdout: Installing : librdkafka-1.6.1-102.el9.x86_64 2/34 2024-03-11T21:15:07.368 INFO:teuthology.orchestra.run.smithi092.stdout: Installing : librabbitmq-0.11.0-7.el9.x86_64 3/34 2024-03-11T21:15:07.397 INFO:teuthology.orchestra.run.smithi092.stdout: Installing : qatlib-service-23.11.0-1.el9.x86_64 4/34 2024-03-11T21:15:07.415 INFO:teuthology.orchestra.run.smithi092.stdout: Running scriptlet: qatlib-service-23.11.0-1.el9.x86_64 4/34 2024-03-11T21:15:07.615 INFO:teuthology.orchestra.run.smithi092.stdout: Running scriptlet: qatlib-23.11.0-1.el9.x86_64 5/34 2024-03-11T21:15:07.670 INFO:teuthology.orchestra.run.smithi092.stdout: Installing : qatlib-23.11.0-1.el9.x86_64 5/34 2024-03-11T21:15:08.004 INFO:teuthology.orchestra.run.smithi092.stdout: Installing : thrift-0.15.0-2.el9.x86_64 6/34 2024-03-11T21:15:08.068 INFO:teuthology.orchestra.run.smithi092.stdout: Upgrading : librados2-2:19.0.0-1979.g2b0101eb.el9.x86_64 7/34 2024-03-11T21:15:08.161 INFO:teuthology.orchestra.run.smithi092.stdout: Running scriptlet: librados2-2:19.0.0-1979.g2b0101eb.el9.x86_64 7/34 2024-03-11T21:15:08.250 INFO:teuthology.orchestra.run.smithi092.stdout: Installing : python3-rados-2:19.0.0-1979.g2b0101eb.el9.x86_64 8/34 2024-03-11T21:15:08.283 INFO:teuthology.orchestra.run.smithi092.stdout: Installing : libcephfs2-2:19.0.0-1979.g2b0101eb.el9.x86_64 9/34 2024-03-11T21:15:08.333 INFO:teuthology.orchestra.run.smithi092.stdout: Running scriptlet: libcephfs2-2:19.0.0-1979.g2b0101eb.el9.x86_64 9/34 2024-03-11T21:15:08.361 INFO:teuthology.orchestra.run.smithi092.stdout: Installing : liboath-2.6.7-2.el9.x86_64 10/34 2024-03-11T21:15:08.415 INFO:teuthology.orchestra.run.smithi092.stdout: Installing : python3-ceph-argparse-2:19.0.0-1979.g2b0101eb.el9. 11/34 2024-03-11T21:15:08.476 INFO:teuthology.orchestra.run.smithi092.stdout: Installing : python3-cephfs-2:19.0.0-1979.g2b0101eb.el9.x86_64 12/34 2024-03-11T21:15:08.505 INFO:teuthology.orchestra.run.smithi092.stdout: Installing : libradosstriper1-2:19.0.0-1979.g2b0101eb.el9.x86_6 13/34 2024-03-11T21:15:08.551 INFO:teuthology.orchestra.run.smithi092.stdout: Running scriptlet: libradosstriper1-2:19.0.0-1979.g2b0101eb.el9.x86_6 13/34 2024-03-11T21:15:08.587 INFO:teuthology.orchestra.run.smithi092.stdout: Installing : qatzip-libs-1.1.2-2.el9.x86_64 14/34 2024-03-11T21:15:08.698 INFO:teuthology.orchestra.run.smithi092.stdout: Installing : re2-1:20211101-3.el9.x86_64 15/34 2024-03-11T21:15:08.739 INFO:teuthology.orchestra.run.smithi092.stdout: Installing : libarrow-9.0.0-10.el9.x86_64 16/34 2024-03-11T21:15:08.779 INFO:teuthology.orchestra.run.smithi092.stdout: Installing : libunwind-1.6.2-1.el9.x86_64 17/34 2024-03-11T21:15:08.812 INFO:teuthology.orchestra.run.smithi092.stdout: Installing : gperftools-libs-2.9.1-2.el9.x86_64 18/34 2024-03-11T21:15:08.841 INFO:teuthology.orchestra.run.smithi092.stdout: Installing : libarrow-doc-9.0.0-10.el9.noarch 19/34 2024-03-11T21:15:08.873 INFO:teuthology.orchestra.run.smithi092.stdout: Installing : python3-prettytable-0.7.2-27.el9.noarch 20/34 2024-03-11T21:15:09.194 INFO:teuthology.orchestra.run.smithi092.stdout: Installing : libpmemobj-1.12.1-1.el9.x86_64 21/34 2024-03-11T21:15:09.220 INFO:teuthology.orchestra.run.smithi092.stdout: Upgrading : librbd1-2:19.0.0-1979.g2b0101eb.el9.x86_64 22/34 2024-03-11T21:15:09.294 INFO:teuthology.orchestra.run.smithi092.stdout: Running scriptlet: librbd1-2:19.0.0-1979.g2b0101eb.el9.x86_64 22/34 2024-03-11T21:15:09.324 INFO:teuthology.orchestra.run.smithi092.stdout: Installing : python3-rbd-2:19.0.0-1979.g2b0101eb.el9.x86_64 23/34 2024-03-11T21:15:09.366 INFO:teuthology.orchestra.run.smithi092.stdout: Installing : boost-program-options-1.75.0-8.el9.x86_64 24/34 2024-03-11T21:15:09.826 INFO:teuthology.orchestra.run.smithi092.stdout: Installing : parquet-libs-9.0.0-10.el9.x86_64 25/34 2024-03-11T21:15:09.856 INFO:teuthology.orchestra.run.smithi092.stdout: Installing : librgw2-2:19.0.0-1979.g2b0101eb.el9.x86_64 26/34 2024-03-11T21:15:09.919 INFO:teuthology.orchestra.run.smithi092.stdout: Running scriptlet: librgw2-2:19.0.0-1979.g2b0101eb.el9.x86_64 26/34 2024-03-11T21:15:09.987 INFO:teuthology.orchestra.run.smithi092.stdout: Installing : python3-rgw-2:19.0.0-1979.g2b0101eb.el9.x86_64 27/34 2024-03-11T21:15:10.017 INFO:teuthology.orchestra.run.smithi092.stdout: Installing : python3-ceph-common-2:19.0.0-1979.g2b0101eb.el9.x8 28/34 2024-03-11T21:15:12.186 INFO:teuthology.orchestra.run.smithi092.stdout: Running scriptlet: ceph-common-2:19.0.0-1979.g2b0101eb.el9.x86_64 29/34 2024-03-11T21:15:12.238 INFO:teuthology.orchestra.run.smithi092.stdout: Installing : ceph-common-2:19.0.0-1979.g2b0101eb.el9.x86_64 29/34 2024-03-11T21:15:12.786 INFO:teuthology.orchestra.run.smithi092.stdout: Running scriptlet: ceph-common-2:19.0.0-1979.g2b0101eb.el9.x86_64 29/34 2024-03-11T21:15:12.833 INFO:teuthology.orchestra.run.smithi092.stdout: Installing : ceph-base-2:19.0.0-1979.g2b0101eb.el9.x86_64 30/34 2024-03-11T21:15:12.924 INFO:teuthology.orchestra.run.smithi092.stdout: Running scriptlet: ceph-base-2:19.0.0-1979.g2b0101eb.el9.x86_64 30/34 2024-03-11T21:15:12.924 INFO:teuthology.orchestra.run.smithi092.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /usr/lib/systemd/system/ceph.target. 2024-03-11T21:15:12.924 INFO:teuthology.orchestra.run.smithi092.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service → /usr/lib/systemd/system/ceph-crash.service. 2024-03-11T21:15:12.924 INFO:teuthology.orchestra.run.smithi092.stdout: 2024-03-11T21:15:12.968 INFO:teuthology.orchestra.run.smithi092.stdout: Installing : ceph-selinux-2:19.0.0-1979.g2b0101eb.el9.x86_64 31/34 2024-03-11T21:15:22.382 INFO:teuthology.orchestra.run.smithi092.stdout: Running scriptlet: ceph-selinux-2:19.0.0-1979.g2b0101eb.el9.x86_64 31/34 2024-03-11T21:15:22.383 INFO:teuthology.orchestra.run.smithi092.stdout:skipping the directory /sys 2024-03-11T21:15:22.383 INFO:teuthology.orchestra.run.smithi092.stdout:skipping the directory /proc 2024-03-11T21:15:22.383 INFO:teuthology.orchestra.run.smithi092.stdout:skipping the directory /mnt 2024-03-11T21:15:22.383 INFO:teuthology.orchestra.run.smithi092.stdout:skipping the directory /var/tmp 2024-03-11T21:15:22.383 INFO:teuthology.orchestra.run.smithi092.stdout:skipping the directory /home 2024-03-11T21:15:22.383 INFO:teuthology.orchestra.run.smithi092.stdout:skipping the directory /root 2024-03-11T21:15:22.383 INFO:teuthology.orchestra.run.smithi092.stdout:skipping the directory /tmp 2024-03-11T21:15:22.383 INFO:teuthology.orchestra.run.smithi092.stdout: 2024-03-11T21:15:23.304 INFO:teuthology.orchestra.run.smithi092.stdout: Installing : ceph-radosgw-2:19.0.0-1979.g2b0101eb.el9.x86_64 32/34 2024-03-11T21:15:23.328 INFO:teuthology.orchestra.run.smithi092.stdout: Running scriptlet: ceph-radosgw-2:19.0.0-1979.g2b0101eb.el9.x86_64 32/34 2024-03-11T21:15:23.328 INFO:teuthology.orchestra.run.smithi092.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-11T21:15:23.329 INFO:teuthology.orchestra.run.smithi092.stdout:Invalid unit name "ceph-radosgw@*.service" escaped as "ceph-radosgw@\x2a.service". 2024-03-11T21:15:23.329 INFO:teuthology.orchestra.run.smithi092.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-03-11T21:15:23.329 INFO:teuthology.orchestra.run.smithi092.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-03-11T21:15:23.329 INFO:teuthology.orchestra.run.smithi092.stdout: 2024-03-11T21:15:23.331 INFO:teuthology.orchestra.run.smithi092.stdout: Cleanup : librbd1-2:16.2.4-5.el9.x86_64 33/34 2024-03-11T21:15:23.423 INFO:teuthology.orchestra.run.smithi092.stdout: Running scriptlet: librbd1-2:16.2.4-5.el9.x86_64 33/34 2024-03-11T21:15:23.423 INFO:teuthology.orchestra.run.smithi092.stdout: Cleanup : librados2-2:16.2.4-5.el9.x86_64 34/34 2024-03-11T21:15:24.563 INFO:teuthology.orchestra.run.smithi092.stdout: Running scriptlet: librados2-2:16.2.4-5.el9.x86_64 34/34 2024-03-11T21:15:24.564 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : ceph-base-2:19.0.0-1979.g2b0101eb.el9.x86_64 1/34 2024-03-11T21:15:24.564 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : ceph-common-2:19.0.0-1979.g2b0101eb.el9.x86_64 2/34 2024-03-11T21:15:24.564 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : ceph-radosgw-2:19.0.0-1979.g2b0101eb.el9.x86_64 3/34 2024-03-11T21:15:24.564 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : ceph-selinux-2:19.0.0-1979.g2b0101eb.el9.x86_64 4/34 2024-03-11T21:15:24.564 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : libcephfs2-2:19.0.0-1979.g2b0101eb.el9.x86_64 5/34 2024-03-11T21:15:24.564 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : libradosstriper1-2:19.0.0-1979.g2b0101eb.el9.x86_6 6/34 2024-03-11T21:15:24.564 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : librgw2-2:19.0.0-1979.g2b0101eb.el9.x86_64 7/34 2024-03-11T21:15:24.564 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : python3-ceph-argparse-2:19.0.0-1979.g2b0101eb.el9. 8/34 2024-03-11T21:15:24.564 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : python3-ceph-common-2:19.0.0-1979.g2b0101eb.el9.x8 9/34 2024-03-11T21:15:24.564 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : python3-cephfs-2:19.0.0-1979.g2b0101eb.el9.x86_64 10/34 2024-03-11T21:15:24.564 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : python3-rados-2:19.0.0-1979.g2b0101eb.el9.x86_64 11/34 2024-03-11T21:15:24.565 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : python3-rbd-2:19.0.0-1979.g2b0101eb.el9.x86_64 12/34 2024-03-11T21:15:24.565 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : python3-rgw-2:19.0.0-1979.g2b0101eb.el9.x86_64 13/34 2024-03-11T21:15:24.565 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : boost-program-options-1.75.0-8.el9.x86_64 14/34 2024-03-11T21:15:24.565 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : libpmemobj-1.12.1-1.el9.x86_64 15/34 2024-03-11T21:15:24.565 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : librabbitmq-0.11.0-7.el9.x86_64 16/34 2024-03-11T21:15:24.565 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : librdkafka-1.6.1-102.el9.x86_64 17/34 2024-03-11T21:15:24.565 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : lttng-ust-2.12.0-6.el9.x86_64 18/34 2024-03-11T21:15:24.565 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : python3-prettytable-0.7.2-27.el9.noarch 19/34 2024-03-11T21:15:24.565 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : qatlib-23.11.0-1.el9.x86_64 20/34 2024-03-11T21:15:24.565 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : qatlib-service-23.11.0-1.el9.x86_64 21/34 2024-03-11T21:15:24.565 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : qatzip-libs-1.1.2-2.el9.x86_64 22/34 2024-03-11T21:15:24.565 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : gperftools-libs-2.9.1-2.el9.x86_64 23/34 2024-03-11T21:15:24.565 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : libarrow-9.0.0-10.el9.x86_64 24/34 2024-03-11T21:15:24.565 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : libarrow-doc-9.0.0-10.el9.noarch 25/34 2024-03-11T21:15:24.565 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : liboath-2.6.7-2.el9.x86_64 26/34 2024-03-11T21:15:24.566 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : libunwind-1.6.2-1.el9.x86_64 27/34 2024-03-11T21:15:24.566 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : parquet-libs-9.0.0-10.el9.x86_64 28/34 2024-03-11T21:15:24.567 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : re2-1:20211101-3.el9.x86_64 29/34 2024-03-11T21:15:24.567 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : thrift-0.15.0-2.el9.x86_64 30/34 2024-03-11T21:15:24.567 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : librados2-2:19.0.0-1979.g2b0101eb.el9.x86_64 31/34 2024-03-11T21:15:24.567 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : librados2-2:16.2.4-5.el9.x86_64 32/34 2024-03-11T21:15:24.567 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : librbd1-2:19.0.0-1979.g2b0101eb.el9.x86_64 33/34 2024-03-11T21:15:25.684 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : librbd1-2:16.2.4-5.el9.x86_64 34/34 2024-03-11T21:15:25.684 INFO:teuthology.orchestra.run.smithi092.stdout: 2024-03-11T21:15:25.684 INFO:teuthology.orchestra.run.smithi092.stdout:Upgraded: 2024-03-11T21:15:25.684 INFO:teuthology.orchestra.run.smithi092.stdout: librados2-2:19.0.0-1979.g2b0101eb.el9.x86_64 2024-03-11T21:15:25.684 INFO:teuthology.orchestra.run.smithi092.stdout: librbd1-2:19.0.0-1979.g2b0101eb.el9.x86_64 2024-03-11T21:15:25.685 INFO:teuthology.orchestra.run.smithi092.stdout:Installed: 2024-03-11T21:15:25.685 INFO:teuthology.orchestra.run.smithi092.stdout: boost-program-options-1.75.0-8.el9.x86_64 2024-03-11T21:15:25.685 INFO:teuthology.orchestra.run.smithi092.stdout: ceph-base-2:19.0.0-1979.g2b0101eb.el9.x86_64 2024-03-11T21:15:25.685 INFO:teuthology.orchestra.run.smithi092.stdout: ceph-common-2:19.0.0-1979.g2b0101eb.el9.x86_64 2024-03-11T21:15:25.685 INFO:teuthology.orchestra.run.smithi092.stdout: ceph-radosgw-2:19.0.0-1979.g2b0101eb.el9.x86_64 2024-03-11T21:15:25.685 INFO:teuthology.orchestra.run.smithi092.stdout: ceph-selinux-2:19.0.0-1979.g2b0101eb.el9.x86_64 2024-03-11T21:15:25.685 INFO:teuthology.orchestra.run.smithi092.stdout: gperftools-libs-2.9.1-2.el9.x86_64 2024-03-11T21:15:25.685 INFO:teuthology.orchestra.run.smithi092.stdout: libarrow-9.0.0-10.el9.x86_64 2024-03-11T21:15:25.685 INFO:teuthology.orchestra.run.smithi092.stdout: libarrow-doc-9.0.0-10.el9.noarch 2024-03-11T21:15:25.685 INFO:teuthology.orchestra.run.smithi092.stdout: libcephfs2-2:19.0.0-1979.g2b0101eb.el9.x86_64 2024-03-11T21:15:25.685 INFO:teuthology.orchestra.run.smithi092.stdout: liboath-2.6.7-2.el9.x86_64 2024-03-11T21:15:25.685 INFO:teuthology.orchestra.run.smithi092.stdout: libpmemobj-1.12.1-1.el9.x86_64 2024-03-11T21:15:25.686 INFO:teuthology.orchestra.run.smithi092.stdout: librabbitmq-0.11.0-7.el9.x86_64 2024-03-11T21:15:25.686 INFO:teuthology.orchestra.run.smithi092.stdout: libradosstriper1-2:19.0.0-1979.g2b0101eb.el9.x86_64 2024-03-11T21:15:25.686 INFO:teuthology.orchestra.run.smithi092.stdout: librdkafka-1.6.1-102.el9.x86_64 2024-03-11T21:15:25.686 INFO:teuthology.orchestra.run.smithi092.stdout: librgw2-2:19.0.0-1979.g2b0101eb.el9.x86_64 2024-03-11T21:15:25.686 INFO:teuthology.orchestra.run.smithi092.stdout: libunwind-1.6.2-1.el9.x86_64 2024-03-11T21:15:25.686 INFO:teuthology.orchestra.run.smithi092.stdout: lttng-ust-2.12.0-6.el9.x86_64 2024-03-11T21:15:25.686 INFO:teuthology.orchestra.run.smithi092.stdout: parquet-libs-9.0.0-10.el9.x86_64 2024-03-11T21:15:25.686 INFO:teuthology.orchestra.run.smithi092.stdout: python3-ceph-argparse-2:19.0.0-1979.g2b0101eb.el9.x86_64 2024-03-11T21:15:25.686 INFO:teuthology.orchestra.run.smithi092.stdout: python3-ceph-common-2:19.0.0-1979.g2b0101eb.el9.x86_64 2024-03-11T21:15:25.686 INFO:teuthology.orchestra.run.smithi092.stdout: python3-cephfs-2:19.0.0-1979.g2b0101eb.el9.x86_64 2024-03-11T21:15:25.686 INFO:teuthology.orchestra.run.smithi092.stdout: python3-prettytable-0.7.2-27.el9.noarch 2024-03-11T21:15:25.687 INFO:teuthology.orchestra.run.smithi092.stdout: python3-rados-2:19.0.0-1979.g2b0101eb.el9.x86_64 2024-03-11T21:15:25.687 INFO:teuthology.orchestra.run.smithi092.stdout: python3-rbd-2:19.0.0-1979.g2b0101eb.el9.x86_64 2024-03-11T21:15:25.687 INFO:teuthology.orchestra.run.smithi092.stdout: python3-rgw-2:19.0.0-1979.g2b0101eb.el9.x86_64 2024-03-11T21:15:25.687 INFO:teuthology.orchestra.run.smithi092.stdout: qatlib-23.11.0-1.el9.x86_64 2024-03-11T21:15:25.687 INFO:teuthology.orchestra.run.smithi092.stdout: qatlib-service-23.11.0-1.el9.x86_64 2024-03-11T21:15:25.687 INFO:teuthology.orchestra.run.smithi092.stdout: qatzip-libs-1.1.2-2.el9.x86_64 2024-03-11T21:15:25.687 INFO:teuthology.orchestra.run.smithi092.stdout: re2-1:20211101-3.el9.x86_64 2024-03-11T21:15:25.687 INFO:teuthology.orchestra.run.smithi092.stdout: thrift-0.15.0-2.el9.x86_64 2024-03-11T21:15:25.687 INFO:teuthology.orchestra.run.smithi092.stdout: 2024-03-11T21:15:25.687 INFO:teuthology.orchestra.run.smithi092.stdout:Complete! 2024-03-11T21:15:26.013 DEBUG:teuthology.orchestra.run.smithi092:> sudo yum -y install ceph-test 2024-03-11T21:15:26.571 INFO:teuthology.orchestra.run.smithi092.stdout:Last metadata expiration check: 0:00:30 ago on Mon 11 Mar 2024 09:14:56 PM UTC. 2024-03-11T21:15:26.725 INFO:teuthology.orchestra.run.smithi092.stdout:Dependencies resolved. 2024-03-11T21:15:26.725 INFO:teuthology.orchestra.run.smithi092.stdout:================================================================================ 2024-03-11T21:15:26.725 INFO:teuthology.orchestra.run.smithi092.stdout: Package Arch Version Repository Size 2024-03-11T21:15:26.726 INFO:teuthology.orchestra.run.smithi092.stdout:================================================================================ 2024-03-11T21:15:26.726 INFO:teuthology.orchestra.run.smithi092.stdout:Installing: 2024-03-11T21:15:26.726 INFO:teuthology.orchestra.run.smithi092.stdout: ceph-test x86_64 2:19.0.0-1979.g2b0101eb.el9 ceph 48 M 2024-03-11T21:15:26.726 INFO:teuthology.orchestra.run.smithi092.stdout:Installing dependencies: 2024-03-11T21:15:26.726 INFO:teuthology.orchestra.run.smithi092.stdout: jq x86_64 1.6-16.el9 baseos 187 k 2024-03-11T21:15:26.726 INFO:teuthology.orchestra.run.smithi092.stdout: libcephsqlite x86_64 2:19.0.0-1979.g2b0101eb.el9 ceph 163 k 2024-03-11T21:15:26.726 INFO:teuthology.orchestra.run.smithi092.stdout: oniguruma x86_64 6.9.6-1.el9.5 CentOS-AppStream 217 k 2024-03-11T21:15:26.726 INFO:teuthology.orchestra.run.smithi092.stdout: socat x86_64 1.7.4.1-5.el9 CentOS-AppStream 300 k 2024-03-11T21:15:26.726 INFO:teuthology.orchestra.run.smithi092.stdout: xmlstarlet x86_64 1.6.1-20.el9 CentOS-AppStream 63 k 2024-03-11T21:15:26.726 INFO:teuthology.orchestra.run.smithi092.stdout: 2024-03-11T21:15:26.726 INFO:teuthology.orchestra.run.smithi092.stdout:Transaction Summary 2024-03-11T21:15:26.726 INFO:teuthology.orchestra.run.smithi092.stdout:================================================================================ 2024-03-11T21:15:26.726 INFO:teuthology.orchestra.run.smithi092.stdout:Install 6 Packages 2024-03-11T21:15:26.727 INFO:teuthology.orchestra.run.smithi092.stdout: 2024-03-11T21:15:26.727 INFO:teuthology.orchestra.run.smithi092.stdout:Total download size: 48 M 2024-03-11T21:15:26.727 INFO:teuthology.orchestra.run.smithi092.stdout:Installed size: 205 M 2024-03-11T21:15:26.727 INFO:teuthology.orchestra.run.smithi092.stdout:Downloading Packages: 2024-03-11T21:15:26.947 INFO:teuthology.orchestra.run.smithi092.stdout:(1/6): oniguruma-6.9.6-1.el9.5.x86_64.rpm 1.2 MB/s | 217 kB 00:00 2024-03-11T21:15:27.005 INFO:teuthology.orchestra.run.smithi092.stdout:(2/6): socat-1.7.4.1-5.el9.x86_64.rpm 5.0 MB/s | 300 kB 00:00 2024-03-11T21:15:27.048 INFO:teuthology.orchestra.run.smithi092.stdout:(3/6): xmlstarlet-1.6.1-20.el9.x86_64.rpm 1.5 MB/s | 63 kB 00:00 2024-03-11T21:15:27.273 INFO:teuthology.orchestra.run.smithi092.stdout:(4/6): libcephsqlite-19.0.0-1979.g2b0101eb.el9. 326 kB/s | 163 kB 00:00 2024-03-11T21:15:27.681 INFO:teuthology.orchestra.run.smithi092.stdout:(5/6): jq-1.6-16.el9.x86_64.rpm 295 kB/s | 187 kB 00:00 2024-03-11T21:15:29.682 INFO:teuthology.orchestra.run.smithi092.stdout:(6/6): ceph-test-19.0.0-1979.g2b0101eb.el9.x86_ 16 MB/s | 48 MB 00:02 2024-03-11T21:15:29.684 INFO:teuthology.orchestra.run.smithi092.stdout:-------------------------------------------------------------------------------- 2024-03-11T21:15:29.684 INFO:teuthology.orchestra.run.smithi092.stdout:Total 16 MB/s | 48 MB 00:02 2024-03-11T21:15:29.698 INFO:teuthology.orchestra.run.smithi092.stdout:Running transaction check 2024-03-11T21:15:29.717 INFO:teuthology.orchestra.run.smithi092.stdout:Transaction check succeeded. 2024-03-11T21:15:29.718 INFO:teuthology.orchestra.run.smithi092.stdout:Running transaction test 2024-03-11T21:15:30.103 INFO:teuthology.orchestra.run.smithi092.stdout:Transaction test succeeded. 2024-03-11T21:15:30.103 INFO:teuthology.orchestra.run.smithi092.stdout:Running transaction 2024-03-11T21:15:30.676 INFO:teuthology.orchestra.run.smithi092.stdout: Preparing : 1/1 2024-03-11T21:15:30.740 INFO:teuthology.orchestra.run.smithi092.stdout: Installing : xmlstarlet-1.6.1-20.el9.x86_64 1/6 2024-03-11T21:15:30.790 INFO:teuthology.orchestra.run.smithi092.stdout: Installing : socat-1.7.4.1-5.el9.x86_64 2/6 2024-03-11T21:15:30.831 INFO:teuthology.orchestra.run.smithi092.stdout: Installing : oniguruma-6.9.6-1.el9.5.x86_64 3/6 2024-03-11T21:15:30.873 INFO:teuthology.orchestra.run.smithi092.stdout: Installing : jq-1.6-16.el9.x86_64 4/6 2024-03-11T21:15:30.902 INFO:teuthology.orchestra.run.smithi092.stdout: Installing : libcephsqlite-2:19.0.0-1979.g2b0101eb.el9.x86_64 5/6 2024-03-11T21:15:35.412 INFO:teuthology.orchestra.run.smithi092.stdout: Running scriptlet: libcephsqlite-2:19.0.0-1979.g2b0101eb.el9.x86_64 5/6 2024-03-11T21:15:35.491 INFO:teuthology.orchestra.run.smithi092.stdout: Installing : ceph-test-2:19.0.0-1979.g2b0101eb.el9.x86_64 6/6 2024-03-11T21:15:37.917 INFO:teuthology.orchestra.run.smithi092.stdout: Running scriptlet: ceph-test-2:19.0.0-1979.g2b0101eb.el9.x86_64 6/6 2024-03-11T21:15:37.917 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : ceph-test-2:19.0.0-1979.g2b0101eb.el9.x86_64 1/6 2024-03-11T21:15:37.917 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : libcephsqlite-2:19.0.0-1979.g2b0101eb.el9.x86_64 2/6 2024-03-11T21:15:37.917 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : oniguruma-6.9.6-1.el9.5.x86_64 3/6 2024-03-11T21:15:37.917 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : socat-1.7.4.1-5.el9.x86_64 4/6 2024-03-11T21:15:37.917 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : xmlstarlet-1.6.1-20.el9.x86_64 5/6 2024-03-11T21:15:38.287 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : jq-1.6-16.el9.x86_64 6/6 2024-03-11T21:15:38.287 INFO:teuthology.orchestra.run.smithi092.stdout: 2024-03-11T21:15:38.287 INFO:teuthology.orchestra.run.smithi092.stdout:Installed: 2024-03-11T21:15:38.287 INFO:teuthology.orchestra.run.smithi092.stdout: ceph-test-2:19.0.0-1979.g2b0101eb.el9.x86_64 2024-03-11T21:15:38.287 INFO:teuthology.orchestra.run.smithi092.stdout: jq-1.6-16.el9.x86_64 2024-03-11T21:15:38.287 INFO:teuthology.orchestra.run.smithi092.stdout: libcephsqlite-2:19.0.0-1979.g2b0101eb.el9.x86_64 2024-03-11T21:15:38.287 INFO:teuthology.orchestra.run.smithi092.stdout: oniguruma-6.9.6-1.el9.5.x86_64 2024-03-11T21:15:38.287 INFO:teuthology.orchestra.run.smithi092.stdout: socat-1.7.4.1-5.el9.x86_64 2024-03-11T21:15:38.287 INFO:teuthology.orchestra.run.smithi092.stdout: xmlstarlet-1.6.1-20.el9.x86_64 2024-03-11T21:15:38.287 INFO:teuthology.orchestra.run.smithi092.stdout: 2024-03-11T21:15:38.287 INFO:teuthology.orchestra.run.smithi092.stdout:Complete! 2024-03-11T21:15:38.580 DEBUG:teuthology.orchestra.run.smithi092:> sudo yum -y install ceph 2024-03-11T21:15:39.139 INFO:teuthology.orchestra.run.smithi092.stdout:Last metadata expiration check: 0:00:43 ago on Mon 11 Mar 2024 09:14:56 PM UTC. 2024-03-11T21:15:39.290 INFO:teuthology.orchestra.run.smithi092.stdout:Dependencies resolved. 2024-03-11T21:15:39.292 INFO:teuthology.orchestra.run.smithi092.stdout:========================================================================================= 2024-03-11T21:15:39.292 INFO:teuthology.orchestra.run.smithi092.stdout: Package Arch Version Repository Size 2024-03-11T21:15:39.292 INFO:teuthology.orchestra.run.smithi092.stdout:========================================================================================= 2024-03-11T21:15:39.292 INFO:teuthology.orchestra.run.smithi092.stdout:Installing: 2024-03-11T21:15:39.292 INFO:teuthology.orchestra.run.smithi092.stdout: ceph x86_64 2:19.0.0-1979.g2b0101eb.el9 ceph 6.6 k 2024-03-11T21:15:39.293 INFO:teuthology.orchestra.run.smithi092.stdout:Installing dependencies: 2024-03-11T21:15:39.293 INFO:teuthology.orchestra.run.smithi092.stdout: ceph-mds x86_64 2:19.0.0-1979.g2b0101eb.el9 ceph 2.1 M 2024-03-11T21:15:39.293 INFO:teuthology.orchestra.run.smithi092.stdout: ceph-mgr x86_64 2:19.0.0-1979.g2b0101eb.el9 ceph 1.5 M 2024-03-11T21:15:39.293 INFO:teuthology.orchestra.run.smithi092.stdout: ceph-mgr-modules-core noarch 2:19.0.0-1979.g2b0101eb.el9 ceph-noarch 248 k 2024-03-11T21:15:39.293 INFO:teuthology.orchestra.run.smithi092.stdout: ceph-mon x86_64 2:19.0.0-1979.g2b0101eb.el9 ceph 4.7 M 2024-03-11T21:15:39.293 INFO:teuthology.orchestra.run.smithi092.stdout: ceph-osd x86_64 2:19.0.0-1979.g2b0101eb.el9 ceph 17 M 2024-03-11T21:15:39.293 INFO:teuthology.orchestra.run.smithi092.stdout: lua x86_64 5.4.4-4.el9 CentOS-AppStream 187 k 2024-03-11T21:15:39.293 INFO:teuthology.orchestra.run.smithi092.stdout: lua-devel x86_64 5.4.4-4.el9 crb 22 k 2024-03-11T21:15:39.293 INFO:teuthology.orchestra.run.smithi092.stdout: luarocks noarch 3.9.2-1.el9 epel 151 k 2024-03-11T21:15:39.293 INFO:teuthology.orchestra.run.smithi092.stdout: python3-bcrypt x86_64 3.2.2-1.el9 epel 43 k 2024-03-11T21:15:39.293 INFO:teuthology.orchestra.run.smithi092.stdout: python3-cffi x86_64 1.14.5-5.el9 CentOS-BaseOS 241 k 2024-03-11T21:15:39.293 INFO:teuthology.orchestra.run.smithi092.stdout: python3-cheroot noarch 8.6.0-4.el9 epel 172 k 2024-03-11T21:15:39.293 INFO:teuthology.orchestra.run.smithi092.stdout: python3-cherrypy noarch 18.6.1-2.el9 epel 358 k 2024-03-11T21:15:39.293 INFO:teuthology.orchestra.run.smithi092.stdout: python3-cryptography x86_64 36.0.1-4.el9 CentOS-BaseOS 1.1 M 2024-03-11T21:15:39.294 INFO:teuthology.orchestra.run.smithi092.stdout: python3-jaraco noarch 8.2.1-3.el9 epel 11 k 2024-03-11T21:15:39.294 INFO:teuthology.orchestra.run.smithi092.stdout: python3-jaraco-classes noarch 3.2.1-5.el9 epel 18 k 2024-03-11T21:15:39.294 INFO:teuthology.orchestra.run.smithi092.stdout: python3-jaraco-collections noarch 3.0.0-8.el9 epel 23 k 2024-03-11T21:15:39.294 INFO:teuthology.orchestra.run.smithi092.stdout: python3-jaraco-functools noarch 3.5.0-2.el9 epel 19 k 2024-03-11T21:15:39.294 INFO:teuthology.orchestra.run.smithi092.stdout: python3-jaraco-text noarch 3.2.0-6.el9 epel 20 k 2024-03-11T21:15:39.294 INFO:teuthology.orchestra.run.smithi092.stdout: python3-logutils noarch 0.3.5-21.el9 epel 46 k 2024-03-11T21:15:39.294 INFO:teuthology.orchestra.run.smithi092.stdout: python3-mako noarch 1.1.4-6.el9 CentOS-AppStream 144 k 2024-03-11T21:15:39.294 INFO:teuthology.orchestra.run.smithi092.stdout: python3-markupsafe x86_64 1.1.1-12.el9 CentOS-AppStream 32 k 2024-03-11T21:15:39.294 INFO:teuthology.orchestra.run.smithi092.stdout: python3-more-itertools noarch 8.12.0-2.el9 epel 79 k 2024-03-11T21:15:39.294 INFO:teuthology.orchestra.run.smithi092.stdout: python3-pecan noarch 1.4.2-3.el9 epel 272 k 2024-03-11T21:15:39.294 INFO:teuthology.orchestra.run.smithi092.stdout: python3-ply noarch 3.11-14.el9 CentOS-BaseOS 103 k 2024-03-11T21:15:39.294 INFO:teuthology.orchestra.run.smithi092.stdout: python3-portend noarch 3.1.0-2.el9 epel 16 k 2024-03-11T21:15:39.294 INFO:teuthology.orchestra.run.smithi092.stdout: python3-pyOpenSSL noarch 21.0.0-1.el9 epel 90 k 2024-03-11T21:15:39.295 INFO:teuthology.orchestra.run.smithi092.stdout: python3-pycparser noarch 2.20-6.el9 CentOS-BaseOS 124 k 2024-03-11T21:15:39.295 INFO:teuthology.orchestra.run.smithi092.stdout: python3-pytz noarch 2021.1-5.el9 CentOS-AppStream 47 k 2024-03-11T21:15:39.295 INFO:teuthology.orchestra.run.smithi092.stdout: python3-requests noarch 2.25.1-8.el9 baseos 125 k 2024-03-11T21:15:39.295 INFO:teuthology.orchestra.run.smithi092.stdout: python3-tempora noarch 5.0.0-2.el9 epel 36 k 2024-03-11T21:15:39.295 INFO:teuthology.orchestra.run.smithi092.stdout: python3-toml noarch 0.10.2-6.el9 CentOS-AppStream 39 k 2024-03-11T21:15:39.295 INFO:teuthology.orchestra.run.smithi092.stdout: python3-urllib3 noarch 1.26.5-5.el9 CentOS-BaseOS 187 k 2024-03-11T21:15:39.295 INFO:teuthology.orchestra.run.smithi092.stdout: python3-webob noarch 1.8.7-6.el9 epel 230 k 2024-03-11T21:15:39.295 INFO:teuthology.orchestra.run.smithi092.stdout: python3-werkzeug noarch 2.0.3-3.el9.1 epel 427 k 2024-03-11T21:15:39.295 INFO:teuthology.orchestra.run.smithi092.stdout: python3-zc-lockfile noarch 2.0-10.el9 epel 20 k 2024-03-11T21:15:39.295 INFO:teuthology.orchestra.run.smithi092.stdout: 2024-03-11T21:15:39.295 INFO:teuthology.orchestra.run.smithi092.stdout:Transaction Summary 2024-03-11T21:15:39.295 INFO:teuthology.orchestra.run.smithi092.stdout:========================================================================================= 2024-03-11T21:15:39.295 INFO:teuthology.orchestra.run.smithi092.stdout:Install 36 Packages 2024-03-11T21:15:39.295 INFO:teuthology.orchestra.run.smithi092.stdout: 2024-03-11T21:15:39.296 INFO:teuthology.orchestra.run.smithi092.stdout:Total download size: 30 M 2024-03-11T21:15:39.296 INFO:teuthology.orchestra.run.smithi092.stdout:Installed size: 107 M 2024-03-11T21:15:39.296 INFO:teuthology.orchestra.run.smithi092.stdout:Downloading Packages: 2024-03-11T21:15:40.034 INFO:teuthology.orchestra.run.smithi092.stdout:(1/36): ceph-19.0.0-1979.g2b0101eb.el9.x86_64.r 42 kB/s | 6.6 kB 00:00 2024-03-11T21:15:40.293 INFO:teuthology.orchestra.run.smithi092.stdout:(2/36): ceph-mds-19.0.0-1979.g2b0101eb.el9.x86_ 5.1 MB/s | 2.1 MB 00:00 2024-03-11T21:15:40.634 INFO:teuthology.orchestra.run.smithi092.stdout:(3/36): ceph-mon-19.0.0-1979.g2b0101eb.el9.x86_ 7.8 MB/s | 4.7 MB 00:00 2024-03-11T21:15:40.734 INFO:teuthology.orchestra.run.smithi092.stdout:(4/36): ceph-mgr-modules-core-19.0.0-1979.g2b01 2.4 MB/s | 248 kB 00:00 2024-03-11T21:15:40.893 INFO:teuthology.orchestra.run.smithi092.stdout:(5/36): lua-5.4.4-4.el9.x86_64.rpm 1.2 MB/s | 187 kB 00:00 2024-03-11T21:15:40.935 INFO:teuthology.orchestra.run.smithi092.stdout:(6/36): python3-mako-1.1.4-6.el9.noarch.rpm 3.4 MB/s | 144 kB 00:00 2024-03-11T21:15:40.977 INFO:teuthology.orchestra.run.smithi092.stdout:(7/36): python3-markupsafe-1.1.1-12.el9.x86_64. 760 kB/s | 32 kB 00:00 2024-03-11T21:15:41.010 INFO:teuthology.orchestra.run.smithi092.stdout:(8/36): python3-pytz-2021.1-5.el9.noarch.rpm 1.4 MB/s | 47 kB 00:00 2024-03-11T21:15:41.044 INFO:teuthology.orchestra.run.smithi092.stdout:(9/36): python3-toml-0.10.2-6.el9.noarch.rpm 1.1 MB/s | 39 kB 00:00 2024-03-11T21:15:41.220 INFO:teuthology.orchestra.run.smithi092.stdout:(10/36): ceph-osd-19.0.0-1979.g2b0101eb.el9.x86 18 MB/s | 17 MB 00:00 2024-03-11T21:15:41.245 INFO:teuthology.orchestra.run.smithi092.stdout:(11/36): python3-cffi-1.14.5-5.el9.x86_64.rpm 1.2 MB/s | 241 kB 00:00 2024-03-11T21:15:41.278 INFO:teuthology.orchestra.run.smithi092.stdout:(12/36): ceph-mgr-19.0.0-1979.g2b0101eb.el9.x86 1.1 MB/s | 1.5 MB 00:01 2024-03-11T21:15:41.337 INFO:teuthology.orchestra.run.smithi092.stdout:(13/36): python3-cryptography-36.0.1-4.el9.x86_ 9.8 MB/s | 1.1 MB 00:00 2024-03-11T21:15:41.362 INFO:teuthology.orchestra.run.smithi092.stdout:(14/36): python3-ply-3.11-14.el9.noarch.rpm 880 kB/s | 103 kB 00:00 2024-03-11T21:15:41.388 INFO:teuthology.orchestra.run.smithi092.stdout:(15/36): python3-pycparser-2.20-6.el9.noarch.rp 1.1 MB/s | 124 kB 00:00 2024-03-11T21:15:41.413 INFO:teuthology.orchestra.run.smithi092.stdout:(16/36): python3-urllib3-1.26.5-5.el9.noarch.rp 2.4 MB/s | 187 kB 00:00 2024-03-11T21:15:41.747 INFO:teuthology.orchestra.run.smithi092.stdout:(17/36): lua-devel-5.4.4-4.el9.x86_64.rpm 62 kB/s | 22 kB 00:00 2024-03-11T21:15:41.955 INFO:teuthology.orchestra.run.smithi092.stdout:(18/36): luarocks-3.9.2-1.el9.noarch.rpm 279 kB/s | 151 kB 00:00 2024-03-11T21:15:42.139 INFO:teuthology.orchestra.run.smithi092.stdout:(19/36): python3-cheroot-8.6.0-4.el9.noarch.rpm 938 kB/s | 172 kB 00:00 2024-03-11T21:15:42.164 INFO:teuthology.orchestra.run.smithi092.stdout:(20/36): python3-bcrypt-3.2.2-1.el9.x86_64.rpm 104 kB/s | 43 kB 00:00 2024-03-11T21:15:42.297 INFO:teuthology.orchestra.run.smithi092.stdout:(21/36): python3-jaraco-8.2.1-3.el9.noarch.rpm 80 kB/s | 11 kB 00:00 2024-03-11T21:15:42.339 INFO:teuthology.orchestra.run.smithi092.stdout:(22/36): python3-cherrypy-18.6.1-2.el9.noarch.r 1.7 MB/s | 358 kB 00:00 2024-03-11T21:15:42.417 INFO:teuthology.orchestra.run.smithi092.stdout:(23/36): python3-jaraco-classes-3.2.1-5.el9.noa 148 kB/s | 18 kB 00:00 2024-03-11T21:15:42.434 INFO:teuthology.orchestra.run.smithi092.stdout:(24/36): python3-jaraco-collections-3.0.0-8.el9 246 kB/s | 23 kB 00:00 2024-03-11T21:15:42.534 INFO:teuthology.orchestra.run.smithi092.stdout:(25/36): python3-jaraco-text-3.2.0-6.el9.noarch 196 kB/s | 20 kB 00:00 2024-03-11T21:15:42.559 INFO:teuthology.orchestra.run.smithi092.stdout:(26/36): python3-requests-2.25.1-8.el9.noarch.r 107 kB/s | 125 kB 00:01 2024-03-11T21:15:42.576 INFO:teuthology.orchestra.run.smithi092.stdout:(27/36): python3-jaraco-functools-3.5.0-2.el9.n 122 kB/s | 19 kB 00:00 2024-03-11T21:15:42.651 INFO:teuthology.orchestra.run.smithi092.stdout:(28/36): python3-logutils-0.3.5-21.el9.noarch.r 399 kB/s | 46 kB 00:00 2024-03-11T21:15:42.752 INFO:teuthology.orchestra.run.smithi092.stdout:(29/36): python3-portend-3.1.0-2.el9.noarch.rpm 164 kB/s | 16 kB 00:00 2024-03-11T21:15:42.785 INFO:teuthology.orchestra.run.smithi092.stdout:(30/36): python3-more-itertools-8.12.0-2.el9.no 378 kB/s | 79 kB 00:00 2024-03-11T21:15:42.869 INFO:teuthology.orchestra.run.smithi092.stdout:(31/36): python3-pyOpenSSL-21.0.0-1.el9.noarch. 777 kB/s | 90 kB 00:00 2024-03-11T21:15:42.911 INFO:teuthology.orchestra.run.smithi092.stdout:(32/36): python3-tempora-5.0.0-2.el9.noarch.rpm 285 kB/s | 36 kB 00:00 2024-03-11T21:15:43.002 INFO:teuthology.orchestra.run.smithi092.stdout:(33/36): python3-webob-1.8.7-6.el9.noarch.rpm 1.7 MB/s | 230 kB 00:00 2024-03-11T21:15:43.103 INFO:teuthology.orchestra.run.smithi092.stdout:(34/36): python3-zc-lockfile-2.0-10.el9.noarch. 200 kB/s | 20 kB 00:00 2024-03-11T21:15:43.161 INFO:teuthology.orchestra.run.smithi092.stdout:(35/36): python3-werkzeug-2.0.3-3.el9.1.noarch. 1.7 MB/s | 427 kB 00:00 2024-03-11T21:15:43.186 INFO:teuthology.orchestra.run.smithi092.stdout:(36/36): python3-pecan-1.4.2-3.el9.noarch.rpm 447 kB/s | 272 kB 00:00 2024-03-11T21:15:43.192 INFO:teuthology.orchestra.run.smithi092.stdout:-------------------------------------------------------------------------------- 2024-03-11T21:15:43.192 INFO:teuthology.orchestra.run.smithi092.stdout:Total 7.6 MB/s | 30 MB 00:03 2024-03-11T21:15:43.217 INFO:teuthology.orchestra.run.smithi092.stdout:Running transaction check 2024-03-11T21:15:43.260 INFO:teuthology.orchestra.run.smithi092.stdout:Transaction check succeeded. 2024-03-11T21:15:43.261 INFO:teuthology.orchestra.run.smithi092.stdout:Running transaction test 2024-03-11T21:15:43.680 INFO:teuthology.orchestra.run.smithi092.stdout:Transaction test succeeded. 2024-03-11T21:15:43.680 INFO:teuthology.orchestra.run.smithi092.stdout:Running transaction 2024-03-11T21:15:45.031 INFO:teuthology.orchestra.run.smithi092.stdout: Preparing : 1/1 2024-03-11T21:15:45.105 INFO:teuthology.orchestra.run.smithi092.stdout: Installing : python3-more-itertools-8.12.0-2.el9.noarch 1/36 2024-03-11T21:15:45.142 INFO:teuthology.orchestra.run.smithi092.stdout: Installing : python3-jaraco-8.2.1-3.el9.noarch 2/36 2024-03-11T21:15:45.183 INFO:teuthology.orchestra.run.smithi092.stdout: Installing : lua-5.4.4-4.el9.x86_64 3/36 2024-03-11T21:15:45.253 INFO:teuthology.orchestra.run.smithi092.stdout: Installing : lua-devel-5.4.4-4.el9.x86_64 4/36 2024-03-11T21:15:45.298 INFO:teuthology.orchestra.run.smithi092.stdout: Installing : luarocks-3.9.2-1.el9.noarch 5/36 2024-03-11T21:15:45.331 INFO:teuthology.orchestra.run.smithi092.stdout: Installing : python3-jaraco-classes-3.2.1-5.el9.noarch 6/36 2024-03-11T21:15:45.422 INFO:teuthology.orchestra.run.smithi092.stdout: Installing : python3-zc-lockfile-2.0-10.el9.noarch 7/36 2024-03-11T21:15:45.509 INFO:teuthology.orchestra.run.smithi092.stdout: Installing : python3-werkzeug-2.0.3-3.el9.1.noarch 8/36 2024-03-11T21:15:45.559 INFO:teuthology.orchestra.run.smithi092.stdout: Installing : python3-webob-1.8.7-6.el9.noarch 9/36 2024-03-11T21:15:45.631 INFO:teuthology.orchestra.run.smithi092.stdout: Installing : python3-logutils-0.3.5-21.el9.noarch 10/36 2024-03-11T21:15:45.685 INFO:teuthology.orchestra.run.smithi092.stdout: Installing : python3-urllib3-1.26.5-5.el9.noarch 11/36 2024-03-11T21:15:45.723 INFO:teuthology.orchestra.run.smithi092.stdout: Installing : python3-requests-2.25.1-8.el9.noarch 12/36 2024-03-11T21:15:45.771 INFO:teuthology.orchestra.run.smithi092.stdout: Installing : python3-ply-3.11-14.el9.noarch 13/36 2024-03-11T21:15:45.827 INFO:teuthology.orchestra.run.smithi092.stdout: Installing : python3-pycparser-2.20-6.el9.noarch 14/36 2024-03-11T21:15:46.018 INFO:teuthology.orchestra.run.smithi092.stdout: Installing : python3-cffi-1.14.5-5.el9.x86_64 15/36 2024-03-11T21:15:46.060 INFO:teuthology.orchestra.run.smithi092.stdout: Installing : python3-cryptography-36.0.1-4.el9.x86_64 16/36 2024-03-11T21:15:46.099 INFO:teuthology.orchestra.run.smithi092.stdout: Installing : python3-pyOpenSSL-21.0.0-1.el9.noarch 17/36 2024-03-11T21:15:46.133 INFO:teuthology.orchestra.run.smithi092.stdout: Installing : python3-bcrypt-3.2.2-1.el9.x86_64 18/36 2024-03-11T21:15:46.163 INFO:teuthology.orchestra.run.smithi092.stdout: Installing : python3-toml-0.10.2-6.el9.noarch 19/36 2024-03-11T21:15:46.240 INFO:teuthology.orchestra.run.smithi092.stdout: Installing : python3-jaraco-functools-3.5.0-2.el9.noarch 20/36 2024-03-11T21:15:46.274 INFO:teuthology.orchestra.run.smithi092.stdout: Installing : python3-cheroot-8.6.0-4.el9.noarch 21/36 2024-03-11T21:15:46.307 INFO:teuthology.orchestra.run.smithi092.stdout: Installing : python3-jaraco-text-3.2.0-6.el9.noarch 22/36 2024-03-11T21:15:46.347 INFO:teuthology.orchestra.run.smithi092.stdout: Installing : python3-jaraco-collections-3.0.0-8.el9.noarch 23/36 2024-03-11T21:15:46.382 INFO:teuthology.orchestra.run.smithi092.stdout: Installing : python3-pytz-2021.1-5.el9.noarch 24/36 2024-03-11T21:15:46.417 INFO:teuthology.orchestra.run.smithi092.stdout: Installing : python3-tempora-5.0.0-2.el9.noarch 25/36 2024-03-11T21:15:46.506 INFO:teuthology.orchestra.run.smithi092.stdout: Installing : python3-portend-3.1.0-2.el9.noarch 26/36 2024-03-11T21:15:46.547 INFO:teuthology.orchestra.run.smithi092.stdout: Installing : python3-cherrypy-18.6.1-2.el9.noarch 27/36 2024-03-11T21:15:46.619 INFO:teuthology.orchestra.run.smithi092.stdout: Installing : python3-markupsafe-1.1.1-12.el9.x86_64 28/36 2024-03-11T21:15:46.756 INFO:teuthology.orchestra.run.smithi092.stdout: Installing : python3-mako-1.1.4-6.el9.noarch 29/36 2024-03-11T21:15:46.908 INFO:teuthology.orchestra.run.smithi092.stdout: Installing : python3-pecan-1.4.2-3.el9.noarch 30/36 2024-03-11T21:15:47.072 INFO:teuthology.orchestra.run.smithi092.stdout: Installing : ceph-mgr-modules-core-2:19.0.0-1979.g2b0101eb.el9. 31/36 2024-03-11T21:15:47.102 INFO:teuthology.orchestra.run.smithi092.stdout: Installing : ceph-mgr-2:19.0.0-1979.g2b0101eb.el9.x86_64 32/36 2024-03-11T21:15:47.127 INFO:teuthology.orchestra.run.smithi092.stdout: Running scriptlet: ceph-mgr-2:19.0.0-1979.g2b0101eb.el9.x86_64 32/36 2024-03-11T21:15:47.127 INFO:teuthology.orchestra.run.smithi092.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-11T21:15:47.127 INFO:teuthology.orchestra.run.smithi092.stdout:Invalid unit name "ceph-mgr@*.service" escaped as "ceph-mgr@\x2a.service". 2024-03-11T21:15:47.127 INFO:teuthology.orchestra.run.smithi092.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-03-11T21:15:47.127 INFO:teuthology.orchestra.run.smithi092.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-03-11T21:15:47.127 INFO:teuthology.orchestra.run.smithi092.stdout: 2024-03-11T21:15:48.561 INFO:teuthology.orchestra.run.smithi092.stdout: Installing : ceph-osd-2:19.0.0-1979.g2b0101eb.el9.x86_64 33/36 2024-03-11T21:15:48.587 INFO:teuthology.orchestra.run.smithi092.stdout: Running scriptlet: ceph-osd-2:19.0.0-1979.g2b0101eb.el9.x86_64 33/36 2024-03-11T21:15:48.587 INFO:teuthology.orchestra.run.smithi092.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-11T21:15:48.587 INFO:teuthology.orchestra.run.smithi092.stdout:Invalid unit name "ceph-osd@*.service" escaped as "ceph-osd@\x2a.service". 2024-03-11T21:15:48.587 INFO:teuthology.orchestra.run.smithi092.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-03-11T21:15:48.587 INFO:teuthology.orchestra.run.smithi092.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-03-11T21:15:48.587 INFO:teuthology.orchestra.run.smithi092.stdout: 2024-03-11T21:15:49.029 INFO:teuthology.orchestra.run.smithi092.stdout: Installing : ceph-mon-2:19.0.0-1979.g2b0101eb.el9.x86_64 34/36 2024-03-11T21:15:49.051 INFO:teuthology.orchestra.run.smithi092.stdout: Running scriptlet: ceph-mon-2:19.0.0-1979.g2b0101eb.el9.x86_64 34/36 2024-03-11T21:15:49.052 INFO:teuthology.orchestra.run.smithi092.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-11T21:15:49.052 INFO:teuthology.orchestra.run.smithi092.stdout:Invalid unit name "ceph-mon@*.service" escaped as "ceph-mon@\x2a.service". 2024-03-11T21:15:49.052 INFO:teuthology.orchestra.run.smithi092.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-03-11T21:15:49.052 INFO:teuthology.orchestra.run.smithi092.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-03-11T21:15:49.052 INFO:teuthology.orchestra.run.smithi092.stdout: 2024-03-11T21:15:49.262 INFO:teuthology.orchestra.run.smithi092.stdout: Installing : ceph-mds-2:19.0.0-1979.g2b0101eb.el9.x86_64 35/36 2024-03-11T21:15:49.285 INFO:teuthology.orchestra.run.smithi092.stdout: Running scriptlet: ceph-mds-2:19.0.0-1979.g2b0101eb.el9.x86_64 35/36 2024-03-11T21:15:49.285 INFO:teuthology.orchestra.run.smithi092.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-11T21:15:49.285 INFO:teuthology.orchestra.run.smithi092.stdout:Invalid unit name "ceph-mds@*.service" escaped as "ceph-mds@\x2a.service". 2024-03-11T21:15:49.285 INFO:teuthology.orchestra.run.smithi092.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-03-11T21:15:49.285 INFO:teuthology.orchestra.run.smithi092.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-03-11T21:15:49.285 INFO:teuthology.orchestra.run.smithi092.stdout: 2024-03-11T21:15:49.352 INFO:teuthology.orchestra.run.smithi092.stdout: Installing : ceph-2:19.0.0-1979.g2b0101eb.el9.x86_64 36/36 2024-03-11T21:15:51.064 INFO:teuthology.orchestra.run.smithi092.stdout: Running scriptlet: ceph-2:19.0.0-1979.g2b0101eb.el9.x86_64 36/36 2024-03-11T21:15:51.064 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : ceph-2:19.0.0-1979.g2b0101eb.el9.x86_64 1/36 2024-03-11T21:15:51.064 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : ceph-mds-2:19.0.0-1979.g2b0101eb.el9.x86_64 2/36 2024-03-11T21:15:51.064 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : ceph-mgr-2:19.0.0-1979.g2b0101eb.el9.x86_64 3/36 2024-03-11T21:15:51.064 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : ceph-mon-2:19.0.0-1979.g2b0101eb.el9.x86_64 4/36 2024-03-11T21:15:51.064 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : ceph-osd-2:19.0.0-1979.g2b0101eb.el9.x86_64 5/36 2024-03-11T21:15:51.064 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : ceph-mgr-modules-core-2:19.0.0-1979.g2b0101eb.el9. 6/36 2024-03-11T21:15:51.064 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : lua-5.4.4-4.el9.x86_64 7/36 2024-03-11T21:15:51.064 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : python3-mako-1.1.4-6.el9.noarch 8/36 2024-03-11T21:15:51.064 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : python3-markupsafe-1.1.1-12.el9.x86_64 9/36 2024-03-11T21:15:51.065 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : python3-pytz-2021.1-5.el9.noarch 10/36 2024-03-11T21:15:51.065 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : python3-toml-0.10.2-6.el9.noarch 11/36 2024-03-11T21:15:51.065 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : python3-cffi-1.14.5-5.el9.x86_64 12/36 2024-03-11T21:15:51.065 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : python3-cryptography-36.0.1-4.el9.x86_64 13/36 2024-03-11T21:15:51.065 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : python3-ply-3.11-14.el9.noarch 14/36 2024-03-11T21:15:51.065 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : python3-pycparser-2.20-6.el9.noarch 15/36 2024-03-11T21:15:51.065 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : python3-urllib3-1.26.5-5.el9.noarch 16/36 2024-03-11T21:15:51.065 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : python3-requests-2.25.1-8.el9.noarch 17/36 2024-03-11T21:15:51.065 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : lua-devel-5.4.4-4.el9.x86_64 18/36 2024-03-11T21:15:51.065 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : luarocks-3.9.2-1.el9.noarch 19/36 2024-03-11T21:15:51.065 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : python3-bcrypt-3.2.2-1.el9.x86_64 20/36 2024-03-11T21:15:51.065 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : python3-cheroot-8.6.0-4.el9.noarch 21/36 2024-03-11T21:15:51.065 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : python3-cherrypy-18.6.1-2.el9.noarch 22/36 2024-03-11T21:15:51.065 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : python3-jaraco-8.2.1-3.el9.noarch 23/36 2024-03-11T21:15:51.066 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : python3-jaraco-classes-3.2.1-5.el9.noarch 24/36 2024-03-11T21:15:51.066 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : python3-jaraco-collections-3.0.0-8.el9.noarch 25/36 2024-03-11T21:15:51.067 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : python3-jaraco-functools-3.5.0-2.el9.noarch 26/36 2024-03-11T21:15:51.067 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : python3-jaraco-text-3.2.0-6.el9.noarch 27/36 2024-03-11T21:15:51.067 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : python3-logutils-0.3.5-21.el9.noarch 28/36 2024-03-11T21:15:51.067 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : python3-more-itertools-8.12.0-2.el9.noarch 29/36 2024-03-11T21:15:51.067 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : python3-pecan-1.4.2-3.el9.noarch 30/36 2024-03-11T21:15:51.067 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : python3-portend-3.1.0-2.el9.noarch 31/36 2024-03-11T21:15:51.067 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : python3-pyOpenSSL-21.0.0-1.el9.noarch 32/36 2024-03-11T21:15:51.067 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : python3-tempora-5.0.0-2.el9.noarch 33/36 2024-03-11T21:15:51.067 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : python3-webob-1.8.7-6.el9.noarch 34/36 2024-03-11T21:15:51.068 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : python3-werkzeug-2.0.3-3.el9.1.noarch 35/36 2024-03-11T21:15:51.914 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : python3-zc-lockfile-2.0-10.el9.noarch 36/36 2024-03-11T21:15:51.915 INFO:teuthology.orchestra.run.smithi092.stdout: 2024-03-11T21:15:51.915 INFO:teuthology.orchestra.run.smithi092.stdout:Installed: 2024-03-11T21:15:51.915 INFO:teuthology.orchestra.run.smithi092.stdout: ceph-2:19.0.0-1979.g2b0101eb.el9.x86_64 2024-03-11T21:15:51.915 INFO:teuthology.orchestra.run.smithi092.stdout: ceph-mds-2:19.0.0-1979.g2b0101eb.el9.x86_64 2024-03-11T21:15:51.915 INFO:teuthology.orchestra.run.smithi092.stdout: ceph-mgr-2:19.0.0-1979.g2b0101eb.el9.x86_64 2024-03-11T21:15:51.915 INFO:teuthology.orchestra.run.smithi092.stdout: ceph-mgr-modules-core-2:19.0.0-1979.g2b0101eb.el9.noarch 2024-03-11T21:15:51.915 INFO:teuthology.orchestra.run.smithi092.stdout: ceph-mon-2:19.0.0-1979.g2b0101eb.el9.x86_64 2024-03-11T21:15:51.915 INFO:teuthology.orchestra.run.smithi092.stdout: ceph-osd-2:19.0.0-1979.g2b0101eb.el9.x86_64 2024-03-11T21:15:51.915 INFO:teuthology.orchestra.run.smithi092.stdout: lua-5.4.4-4.el9.x86_64 2024-03-11T21:15:51.915 INFO:teuthology.orchestra.run.smithi092.stdout: lua-devel-5.4.4-4.el9.x86_64 2024-03-11T21:15:51.915 INFO:teuthology.orchestra.run.smithi092.stdout: luarocks-3.9.2-1.el9.noarch 2024-03-11T21:15:51.915 INFO:teuthology.orchestra.run.smithi092.stdout: python3-bcrypt-3.2.2-1.el9.x86_64 2024-03-11T21:15:51.916 INFO:teuthology.orchestra.run.smithi092.stdout: python3-cffi-1.14.5-5.el9.x86_64 2024-03-11T21:15:51.916 INFO:teuthology.orchestra.run.smithi092.stdout: python3-cheroot-8.6.0-4.el9.noarch 2024-03-11T21:15:51.916 INFO:teuthology.orchestra.run.smithi092.stdout: python3-cherrypy-18.6.1-2.el9.noarch 2024-03-11T21:15:51.916 INFO:teuthology.orchestra.run.smithi092.stdout: python3-cryptography-36.0.1-4.el9.x86_64 2024-03-11T21:15:51.916 INFO:teuthology.orchestra.run.smithi092.stdout: python3-jaraco-8.2.1-3.el9.noarch 2024-03-11T21:15:51.916 INFO:teuthology.orchestra.run.smithi092.stdout: python3-jaraco-classes-3.2.1-5.el9.noarch 2024-03-11T21:15:51.916 INFO:teuthology.orchestra.run.smithi092.stdout: python3-jaraco-collections-3.0.0-8.el9.noarch 2024-03-11T21:15:51.916 INFO:teuthology.orchestra.run.smithi092.stdout: python3-jaraco-functools-3.5.0-2.el9.noarch 2024-03-11T21:15:51.916 INFO:teuthology.orchestra.run.smithi092.stdout: python3-jaraco-text-3.2.0-6.el9.noarch 2024-03-11T21:15:51.916 INFO:teuthology.orchestra.run.smithi092.stdout: python3-logutils-0.3.5-21.el9.noarch 2024-03-11T21:15:51.916 INFO:teuthology.orchestra.run.smithi092.stdout: python3-mako-1.1.4-6.el9.noarch 2024-03-11T21:15:51.916 INFO:teuthology.orchestra.run.smithi092.stdout: python3-markupsafe-1.1.1-12.el9.x86_64 2024-03-11T21:15:51.916 INFO:teuthology.orchestra.run.smithi092.stdout: python3-more-itertools-8.12.0-2.el9.noarch 2024-03-11T21:15:51.917 INFO:teuthology.orchestra.run.smithi092.stdout: python3-pecan-1.4.2-3.el9.noarch 2024-03-11T21:15:51.917 INFO:teuthology.orchestra.run.smithi092.stdout: python3-ply-3.11-14.el9.noarch 2024-03-11T21:15:51.917 INFO:teuthology.orchestra.run.smithi092.stdout: python3-portend-3.1.0-2.el9.noarch 2024-03-11T21:15:51.917 INFO:teuthology.orchestra.run.smithi092.stdout: python3-pyOpenSSL-21.0.0-1.el9.noarch 2024-03-11T21:15:51.917 INFO:teuthology.orchestra.run.smithi092.stdout: python3-pycparser-2.20-6.el9.noarch 2024-03-11T21:15:51.917 INFO:teuthology.orchestra.run.smithi092.stdout: python3-pytz-2021.1-5.el9.noarch 2024-03-11T21:15:51.917 INFO:teuthology.orchestra.run.smithi092.stdout: python3-requests-2.25.1-8.el9.noarch 2024-03-11T21:15:51.917 INFO:teuthology.orchestra.run.smithi092.stdout: python3-tempora-5.0.0-2.el9.noarch 2024-03-11T21:15:51.917 INFO:teuthology.orchestra.run.smithi092.stdout: python3-toml-0.10.2-6.el9.noarch 2024-03-11T21:15:51.917 INFO:teuthology.orchestra.run.smithi092.stdout: python3-urllib3-1.26.5-5.el9.noarch 2024-03-11T21:15:51.917 INFO:teuthology.orchestra.run.smithi092.stdout: python3-webob-1.8.7-6.el9.noarch 2024-03-11T21:15:51.917 INFO:teuthology.orchestra.run.smithi092.stdout: python3-werkzeug-2.0.3-3.el9.1.noarch 2024-03-11T21:15:51.917 INFO:teuthology.orchestra.run.smithi092.stdout: python3-zc-lockfile-2.0-10.el9.noarch 2024-03-11T21:15:51.917 INFO:teuthology.orchestra.run.smithi092.stdout: 2024-03-11T21:15:51.918 INFO:teuthology.orchestra.run.smithi092.stdout:Complete! 2024-03-11T21:15:52.233 DEBUG:teuthology.orchestra.run.smithi092:> sudo yum -y install ceph-base 2024-03-11T21:15:52.792 INFO:teuthology.orchestra.run.smithi092.stdout:Last metadata expiration check: 0:00:56 ago on Mon 11 Mar 2024 09:14:56 PM UTC. 2024-03-11T21:15:52.889 INFO:teuthology.orchestra.run.smithi092.stdout:Package ceph-base-2:19.0.0-1979.g2b0101eb.el9.x86_64 is already installed. 2024-03-11T21:15:52.941 INFO:teuthology.orchestra.run.smithi092.stdout:Dependencies resolved. 2024-03-11T21:15:52.942 INFO:teuthology.orchestra.run.smithi092.stdout:Nothing to do. 2024-03-11T21:15:52.942 INFO:teuthology.orchestra.run.smithi092.stdout:Complete! 2024-03-11T21:15:53.006 DEBUG:teuthology.orchestra.run.smithi092:> sudo yum -y install cephadm 2024-03-11T21:15:53.566 INFO:teuthology.orchestra.run.smithi092.stdout:Last metadata expiration check: 0:00:57 ago on Mon 11 Mar 2024 09:14:56 PM UTC. 2024-03-11T21:15:53.714 INFO:teuthology.orchestra.run.smithi092.stdout:Dependencies resolved. 2024-03-11T21:15:53.715 INFO:teuthology.orchestra.run.smithi092.stdout:================================================================================ 2024-03-11T21:15:53.715 INFO:teuthology.orchestra.run.smithi092.stdout: Package Arch Version Repository Size 2024-03-11T21:15:53.715 INFO:teuthology.orchestra.run.smithi092.stdout:================================================================================ 2024-03-11T21:15:53.715 INFO:teuthology.orchestra.run.smithi092.stdout:Installing: 2024-03-11T21:15:53.715 INFO:teuthology.orchestra.run.smithi092.stdout: cephadm noarch 2:19.0.0-1979.g2b0101eb.el9 ceph-noarch 748 k 2024-03-11T21:15:53.715 INFO:teuthology.orchestra.run.smithi092.stdout: 2024-03-11T21:15:53.715 INFO:teuthology.orchestra.run.smithi092.stdout:Transaction Summary 2024-03-11T21:15:53.715 INFO:teuthology.orchestra.run.smithi092.stdout:================================================================================ 2024-03-11T21:15:53.715 INFO:teuthology.orchestra.run.smithi092.stdout:Install 1 Package 2024-03-11T21:15:53.715 INFO:teuthology.orchestra.run.smithi092.stdout: 2024-03-11T21:15:53.715 INFO:teuthology.orchestra.run.smithi092.stdout:Total download size: 748 k 2024-03-11T21:15:53.715 INFO:teuthology.orchestra.run.smithi092.stdout:Installed size: 755 k 2024-03-11T21:15:53.715 INFO:teuthology.orchestra.run.smithi092.stdout:Downloading Packages: 2024-03-11T21:15:54.028 INFO:teuthology.orchestra.run.smithi092.stdout:cephadm-19.0.0-1979.g2b0101eb.el9.noarch.rpm 2.3 MB/s | 748 kB 00:00 2024-03-11T21:15:54.028 INFO:teuthology.orchestra.run.smithi092.stdout:-------------------------------------------------------------------------------- 2024-03-11T21:15:54.028 INFO:teuthology.orchestra.run.smithi092.stdout:Total 2.3 MB/s | 748 kB 00:00 2024-03-11T21:15:54.028 INFO:teuthology.orchestra.run.smithi092.stdout:Running transaction check 2024-03-11T21:15:54.033 INFO:teuthology.orchestra.run.smithi092.stdout:Transaction check succeeded. 2024-03-11T21:15:54.033 INFO:teuthology.orchestra.run.smithi092.stdout:Running transaction test 2024-03-11T21:15:54.038 INFO:teuthology.orchestra.run.smithi092.stdout:Transaction test succeeded. 2024-03-11T21:15:54.039 INFO:teuthology.orchestra.run.smithi092.stdout:Running transaction 2024-03-11T21:15:54.113 INFO:teuthology.orchestra.run.smithi092.stdout: Preparing : 1/1 2024-03-11T21:15:54.517 INFO:teuthology.orchestra.run.smithi092.stdout: Running scriptlet: cephadm-2:19.0.0-1979.g2b0101eb.el9.noarch 1/1 2024-03-11T21:15:54.597 INFO:teuthology.orchestra.run.smithi092.stdout: Installing : cephadm-2:19.0.0-1979.g2b0101eb.el9.noarch 1/1 2024-03-11T21:15:54.980 INFO:teuthology.orchestra.run.smithi092.stdout: Running scriptlet: cephadm-2:19.0.0-1979.g2b0101eb.el9.noarch 1/1 2024-03-11T21:15:55.316 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : cephadm-2:19.0.0-1979.g2b0101eb.el9.noarch 1/1 2024-03-11T21:15:55.316 INFO:teuthology.orchestra.run.smithi092.stdout: 2024-03-11T21:15:55.316 INFO:teuthology.orchestra.run.smithi092.stdout:Installed: 2024-03-11T21:15:55.316 INFO:teuthology.orchestra.run.smithi092.stdout: cephadm-2:19.0.0-1979.g2b0101eb.el9.noarch 2024-03-11T21:15:55.316 INFO:teuthology.orchestra.run.smithi092.stdout: 2024-03-11T21:15:55.316 INFO:teuthology.orchestra.run.smithi092.stdout:Complete! 2024-03-11T21:15:55.526 DEBUG:teuthology.orchestra.run.smithi092:> sudo yum -y install ceph-immutable-object-cache 2024-03-11T21:15:56.098 INFO:teuthology.orchestra.run.smithi092.stdout:Last metadata expiration check: 0:01:00 ago on Mon 11 Mar 2024 09:14:56 PM UTC. 2024-03-11T21:15:56.251 INFO:teuthology.orchestra.run.smithi092.stdout:Dependencies resolved. 2024-03-11T21:15:56.252 INFO:teuthology.orchestra.run.smithi092.stdout:================================================================================ 2024-03-11T21:15:56.252 INFO:teuthology.orchestra.run.smithi092.stdout: Package Arch Version Repo Size 2024-03-11T21:15:56.252 INFO:teuthology.orchestra.run.smithi092.stdout:================================================================================ 2024-03-11T21:15:56.252 INFO:teuthology.orchestra.run.smithi092.stdout:Installing: 2024-03-11T21:15:56.252 INFO:teuthology.orchestra.run.smithi092.stdout: ceph-immutable-object-cache x86_64 2:19.0.0-1979.g2b0101eb.el9 ceph 145 k 2024-03-11T21:15:56.252 INFO:teuthology.orchestra.run.smithi092.stdout: 2024-03-11T21:15:56.252 INFO:teuthology.orchestra.run.smithi092.stdout:Transaction Summary 2024-03-11T21:15:56.252 INFO:teuthology.orchestra.run.smithi092.stdout:================================================================================ 2024-03-11T21:15:56.252 INFO:teuthology.orchestra.run.smithi092.stdout:Install 1 Package 2024-03-11T21:15:56.252 INFO:teuthology.orchestra.run.smithi092.stdout: 2024-03-11T21:15:56.252 INFO:teuthology.orchestra.run.smithi092.stdout:Total download size: 145 k 2024-03-11T21:15:56.253 INFO:teuthology.orchestra.run.smithi092.stdout:Installed size: 436 k 2024-03-11T21:15:56.253 INFO:teuthology.orchestra.run.smithi092.stdout:Downloading Packages: 2024-03-11T21:15:56.473 INFO:teuthology.orchestra.run.smithi092.stdout:ceph-immutable-object-cache-19.0.0-1979.g2b0101 659 kB/s | 145 kB 00:00 2024-03-11T21:15:56.474 INFO:teuthology.orchestra.run.smithi092.stdout:-------------------------------------------------------------------------------- 2024-03-11T21:15:56.474 INFO:teuthology.orchestra.run.smithi092.stdout:Total 655 kB/s | 145 kB 00:00 2024-03-11T21:15:56.474 INFO:teuthology.orchestra.run.smithi092.stdout:Running transaction check 2024-03-11T21:15:56.481 INFO:teuthology.orchestra.run.smithi092.stdout:Transaction check succeeded. 2024-03-11T21:15:56.481 INFO:teuthology.orchestra.run.smithi092.stdout:Running transaction test 2024-03-11T21:15:56.532 INFO:teuthology.orchestra.run.smithi092.stdout:Transaction test succeeded. 2024-03-11T21:15:56.533 INFO:teuthology.orchestra.run.smithi092.stdout:Running transaction 2024-03-11T21:15:56.669 INFO:teuthology.orchestra.run.smithi092.stdout: Preparing : 1/1 2024-03-11T21:15:56.740 INFO:teuthology.orchestra.run.smithi092.stdout: Installing : ceph-immutable-object-cache-2:19.0.0-1979.g2b0101eb. 1/1 2024-03-11T21:15:56.764 INFO:teuthology.orchestra.run.smithi092.stdout: Running scriptlet: ceph-immutable-object-cache-2:19.0.0-1979.g2b0101eb. 1/1 2024-03-11T21:15:56.765 INFO:teuthology.orchestra.run.smithi092.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-11T21:15:56.765 INFO:teuthology.orchestra.run.smithi092.stdout:Invalid unit name "ceph-immutable-object-cache@*.service" escaped as "ceph-immutable-object-cache@\x2a.service". 2024-03-11T21:15:56.765 INFO:teuthology.orchestra.run.smithi092.stdout: 2024-03-11T21:15:57.722 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : ceph-immutable-object-cache-2:19.0.0-1979.g2b0101eb. 1/1 2024-03-11T21:15:57.722 INFO:teuthology.orchestra.run.smithi092.stdout: 2024-03-11T21:15:57.722 INFO:teuthology.orchestra.run.smithi092.stdout:Installed: 2024-03-11T21:15:57.722 INFO:teuthology.orchestra.run.smithi092.stdout: ceph-immutable-object-cache-2:19.0.0-1979.g2b0101eb.el9.x86_64 2024-03-11T21:15:57.722 INFO:teuthology.orchestra.run.smithi092.stdout: 2024-03-11T21:15:57.722 INFO:teuthology.orchestra.run.smithi092.stdout:Complete! 2024-03-11T21:15:57.934 DEBUG:teuthology.orchestra.run.smithi092:> sudo yum -y install ceph-mgr 2024-03-11T21:15:58.500 INFO:teuthology.orchestra.run.smithi092.stdout:Last metadata expiration check: 0:01:02 ago on Mon 11 Mar 2024 09:14:56 PM UTC. 2024-03-11T21:15:58.596 INFO:teuthology.orchestra.run.smithi092.stdout:Package ceph-mgr-2:19.0.0-1979.g2b0101eb.el9.x86_64 is already installed. 2024-03-11T21:15:58.646 INFO:teuthology.orchestra.run.smithi092.stdout:Dependencies resolved. 2024-03-11T21:15:58.648 INFO:teuthology.orchestra.run.smithi092.stdout:Nothing to do. 2024-03-11T21:15:58.648 INFO:teuthology.orchestra.run.smithi092.stdout:Complete! 2024-03-11T21:15:58.710 DEBUG:teuthology.orchestra.run.smithi092:> sudo yum -y install ceph-mgr-dashboard 2024-03-11T21:15:59.262 INFO:teuthology.orchestra.run.smithi092.stdout:Last metadata expiration check: 0:01:03 ago on Mon 11 Mar 2024 09:14:56 PM UTC. 2024-03-11T21:15:59.413 INFO:teuthology.orchestra.run.smithi092.stdout:Dependencies resolved. 2024-03-11T21:15:59.414 INFO:teuthology.orchestra.run.smithi092.stdout:====================================================================================== 2024-03-11T21:15:59.414 INFO:teuthology.orchestra.run.smithi092.stdout: Package Arch Version Repository Size 2024-03-11T21:15:59.414 INFO:teuthology.orchestra.run.smithi092.stdout:====================================================================================== 2024-03-11T21:15:59.414 INFO:teuthology.orchestra.run.smithi092.stdout:Installing: 2024-03-11T21:15:59.414 INFO:teuthology.orchestra.run.smithi092.stdout: ceph-mgr-dashboard noarch 2:19.0.0-1979.g2b0101eb.el9 ceph-noarch 1.8 M 2024-03-11T21:15:59.414 INFO:teuthology.orchestra.run.smithi092.stdout:Installing dependencies: 2024-03-11T21:15:59.414 INFO:teuthology.orchestra.run.smithi092.stdout: abseil-cpp x86_64 20211102.0-3.el9 epel 548 k 2024-03-11T21:15:59.415 INFO:teuthology.orchestra.run.smithi092.stdout: ceph-grafana-dashboards noarch 2:19.0.0-1979.g2b0101eb.el9 ceph-noarch 34 k 2024-03-11T21:15:59.415 INFO:teuthology.orchestra.run.smithi092.stdout: ceph-prometheus-alerts noarch 2:19.0.0-1979.g2b0101eb.el9 ceph-noarch 16 k 2024-03-11T21:15:59.415 INFO:teuthology.orchestra.run.smithi092.stdout: grpc-data noarch 1.46.7-10.el9 epel 19 k 2024-03-11T21:15:59.415 INFO:teuthology.orchestra.run.smithi092.stdout: protobuf x86_64 3.14.0-13.el9 CentOS-AppStream 1.0 M 2024-03-11T21:15:59.415 INFO:teuthology.orchestra.run.smithi092.stdout: protobuf-compiler x86_64 3.14.0-13.el9 crb 863 k 2024-03-11T21:15:59.415 INFO:teuthology.orchestra.run.smithi092.stdout: python3-grpcio x86_64 1.46.7-10.el9 epel 2.0 M 2024-03-11T21:15:59.415 INFO:teuthology.orchestra.run.smithi092.stdout: python3-grpcio-tools x86_64 1.46.7-10.el9 epel 144 k 2024-03-11T21:15:59.415 INFO:teuthology.orchestra.run.smithi092.stdout: python3-protobuf noarch 3.14.0-13.el9 CentOS-AppStream 238 k 2024-03-11T21:15:59.415 INFO:teuthology.orchestra.run.smithi092.stdout: python3-repoze-lru noarch 0.7-16.el9 epel 31 k 2024-03-11T21:15:59.415 INFO:teuthology.orchestra.run.smithi092.stdout: python3-routes noarch 2.5.1-5.el9 epel 188 k 2024-03-11T21:15:59.415 INFO:teuthology.orchestra.run.smithi092.stdout: 2024-03-11T21:15:59.415 INFO:teuthology.orchestra.run.smithi092.stdout:Transaction Summary 2024-03-11T21:15:59.415 INFO:teuthology.orchestra.run.smithi092.stdout:====================================================================================== 2024-03-11T21:15:59.416 INFO:teuthology.orchestra.run.smithi092.stdout:Install 12 Packages 2024-03-11T21:15:59.416 INFO:teuthology.orchestra.run.smithi092.stdout: 2024-03-11T21:15:59.416 INFO:teuthology.orchestra.run.smithi092.stdout:Total download size: 6.9 M 2024-03-11T21:15:59.416 INFO:teuthology.orchestra.run.smithi092.stdout:Installed size: 26 M 2024-03-11T21:15:59.416 INFO:teuthology.orchestra.run.smithi092.stdout:Downloading Packages: 2024-03-11T21:15:59.860 INFO:teuthology.orchestra.run.smithi092.stdout:(1/12): ceph-grafana-dashboards-19.0.0-1979.g2b 144 kB/s | 34 kB 00:00 2024-03-11T21:15:59.977 INFO:teuthology.orchestra.run.smithi092.stdout:(2/12): ceph-prometheus-alerts-19.0.0-1979.g2b0 47 kB/s | 16 kB 00:00 2024-03-11T21:16:00.194 INFO:teuthology.orchestra.run.smithi092.stdout:(3/12): protobuf-3.14.0-13.el9.x86_64.rpm 3.0 MB/s | 1.0 MB 00:00 2024-03-11T21:16:00.219 INFO:teuthology.orchestra.run.smithi092.stdout:(4/12): python3-protobuf-3.14.0-13.el9.noarch.r 983 kB/s | 238 kB 00:00 2024-03-11T21:16:00.719 INFO:teuthology.orchestra.run.smithi092.stdout:(5/12): abseil-cpp-20211102.0-3.el9.x86_64.rpm 1.1 MB/s | 548 kB 00:00 2024-03-11T21:16:00.811 INFO:teuthology.orchestra.run.smithi092.stdout:(6/12): grpc-data-1.46.7-10.el9.noarch.rpm 212 kB/s | 19 kB 00:00 2024-03-11T21:16:01.094 INFO:teuthology.orchestra.run.smithi092.stdout:(7/12): python3-grpcio-1.46.7-10.el9.x86_64.rpm 7.2 MB/s | 2.0 MB 00:00 2024-03-11T21:16:01.195 INFO:teuthology.orchestra.run.smithi092.stdout:(8/12): python3-grpcio-tools-1.46.7-10.el9.x86_ 1.4 MB/s | 144 kB 00:00 2024-03-11T21:16:01.303 INFO:teuthology.orchestra.run.smithi092.stdout:(9/12): python3-repoze-lru-0.7-16.el9.noarch.rp 284 kB/s | 31 kB 00:00 2024-03-11T21:16:01.420 INFO:teuthology.orchestra.run.smithi092.stdout:(10/12): python3-routes-2.5.1-5.el9.noarch.rpm 1.6 MB/s | 188 kB 00:00 2024-03-11T21:16:02.121 INFO:teuthology.orchestra.run.smithi092.stdout:(11/12): ceph-mgr-dashboard-19.0.0-1979.g2b0101 736 kB/s | 1.8 MB 00:02 2024-03-11T21:16:02.537 INFO:teuthology.orchestra.run.smithi092.stdout:(12/12): protobuf-compiler-3.14.0-13.el9.x86_64 368 kB/s | 863 kB 00:02 2024-03-11T21:16:02.541 INFO:teuthology.orchestra.run.smithi092.stdout:-------------------------------------------------------------------------------- 2024-03-11T21:16:02.541 INFO:teuthology.orchestra.run.smithi092.stdout:Total 2.2 MB/s | 6.9 MB 00:03 2024-03-11T21:16:02.561 INFO:teuthology.orchestra.run.smithi092.stdout:Running transaction check 2024-03-11T21:16:02.579 INFO:teuthology.orchestra.run.smithi092.stdout:Transaction check succeeded. 2024-03-11T21:16:02.580 INFO:teuthology.orchestra.run.smithi092.stdout:Running transaction test 2024-03-11T21:16:02.760 INFO:teuthology.orchestra.run.smithi092.stdout:Transaction test succeeded. 2024-03-11T21:16:02.761 INFO:teuthology.orchestra.run.smithi092.stdout:Running transaction 2024-03-11T21:16:03.297 INFO:teuthology.orchestra.run.smithi092.stdout: Preparing : 1/1 2024-03-11T21:16:03.379 INFO:teuthology.orchestra.run.smithi092.stdout: Installing : protobuf-3.14.0-13.el9.x86_64 1/12 2024-03-11T21:16:03.414 INFO:teuthology.orchestra.run.smithi092.stdout: Installing : protobuf-compiler-3.14.0-13.el9.x86_64 2/12 2024-03-11T21:16:03.460 INFO:teuthology.orchestra.run.smithi092.stdout: Installing : python3-repoze-lru-0.7-16.el9.noarch 3/12 2024-03-11T21:16:03.491 INFO:teuthology.orchestra.run.smithi092.stdout: Installing : python3-routes-2.5.1-5.el9.noarch 4/12 2024-03-11T21:16:03.584 INFO:teuthology.orchestra.run.smithi092.stdout: Installing : grpc-data-1.46.7-10.el9.noarch 5/12 2024-03-11T21:16:03.709 INFO:teuthology.orchestra.run.smithi092.stdout: Installing : abseil-cpp-20211102.0-3.el9.x86_64 6/12 2024-03-11T21:16:03.812 INFO:teuthology.orchestra.run.smithi092.stdout: Installing : python3-grpcio-1.46.7-10.el9.x86_64 7/12 2024-03-11T21:16:03.863 INFO:teuthology.orchestra.run.smithi092.stdout: Installing : python3-protobuf-3.14.0-13.el9.noarch 8/12 2024-03-11T21:16:03.891 INFO:teuthology.orchestra.run.smithi092.stdout: Installing : python3-grpcio-tools-1.46.7-10.el9.x86_64 9/12 2024-03-11T21:16:03.923 INFO:teuthology.orchestra.run.smithi092.stdout: Installing : ceph-prometheus-alerts-2:19.0.0-1979.g2b0101eb.el9 10/12 2024-03-11T21:16:04.168 INFO:teuthology.orchestra.run.smithi092.stdout: Installing : ceph-grafana-dashboards-2:19.0.0-1979.g2b0101eb.el 11/12 2024-03-11T21:16:04.198 INFO:teuthology.orchestra.run.smithi092.stdout: Installing : ceph-mgr-dashboard-2:19.0.0-1979.g2b0101eb.el9.noa 12/12 2024-03-11T21:16:04.986 INFO:teuthology.orchestra.run.smithi092.stdout: Running scriptlet: ceph-mgr-dashboard-2:19.0.0-1979.g2b0101eb.el9.noa 12/12 2024-03-11T21:16:04.987 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : ceph-grafana-dashboards-2:19.0.0-1979.g2b0101eb.el 1/12 2024-03-11T21:16:04.987 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : ceph-mgr-dashboard-2:19.0.0-1979.g2b0101eb.el9.noa 2/12 2024-03-11T21:16:04.987 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : ceph-prometheus-alerts-2:19.0.0-1979.g2b0101eb.el9 3/12 2024-03-11T21:16:04.987 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : protobuf-3.14.0-13.el9.x86_64 4/12 2024-03-11T21:16:04.987 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : python3-protobuf-3.14.0-13.el9.noarch 5/12 2024-03-11T21:16:04.987 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : protobuf-compiler-3.14.0-13.el9.x86_64 6/12 2024-03-11T21:16:04.987 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : abseil-cpp-20211102.0-3.el9.x86_64 7/12 2024-03-11T21:16:04.987 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : grpc-data-1.46.7-10.el9.noarch 8/12 2024-03-11T21:16:04.987 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : python3-grpcio-1.46.7-10.el9.x86_64 9/12 2024-03-11T21:16:04.987 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : python3-grpcio-tools-1.46.7-10.el9.x86_64 10/12 2024-03-11T21:16:04.987 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : python3-repoze-lru-0.7-16.el9.noarch 11/12 2024-03-11T21:16:05.334 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : python3-routes-2.5.1-5.el9.noarch 12/12 2024-03-11T21:16:05.334 INFO:teuthology.orchestra.run.smithi092.stdout: 2024-03-11T21:16:05.334 INFO:teuthology.orchestra.run.smithi092.stdout:Installed: 2024-03-11T21:16:05.334 INFO:teuthology.orchestra.run.smithi092.stdout: abseil-cpp-20211102.0-3.el9.x86_64 2024-03-11T21:16:05.334 INFO:teuthology.orchestra.run.smithi092.stdout: ceph-grafana-dashboards-2:19.0.0-1979.g2b0101eb.el9.noarch 2024-03-11T21:16:05.334 INFO:teuthology.orchestra.run.smithi092.stdout: ceph-mgr-dashboard-2:19.0.0-1979.g2b0101eb.el9.noarch 2024-03-11T21:16:05.334 INFO:teuthology.orchestra.run.smithi092.stdout: ceph-prometheus-alerts-2:19.0.0-1979.g2b0101eb.el9.noarch 2024-03-11T21:16:05.334 INFO:teuthology.orchestra.run.smithi092.stdout: grpc-data-1.46.7-10.el9.noarch 2024-03-11T21:16:05.335 INFO:teuthology.orchestra.run.smithi092.stdout: protobuf-3.14.0-13.el9.x86_64 2024-03-11T21:16:05.335 INFO:teuthology.orchestra.run.smithi092.stdout: protobuf-compiler-3.14.0-13.el9.x86_64 2024-03-11T21:16:05.335 INFO:teuthology.orchestra.run.smithi092.stdout: python3-grpcio-1.46.7-10.el9.x86_64 2024-03-11T21:16:05.335 INFO:teuthology.orchestra.run.smithi092.stdout: python3-grpcio-tools-1.46.7-10.el9.x86_64 2024-03-11T21:16:05.335 INFO:teuthology.orchestra.run.smithi092.stdout: python3-protobuf-3.14.0-13.el9.noarch 2024-03-11T21:16:05.335 INFO:teuthology.orchestra.run.smithi092.stdout: python3-repoze-lru-0.7-16.el9.noarch 2024-03-11T21:16:05.335 INFO:teuthology.orchestra.run.smithi092.stdout: python3-routes-2.5.1-5.el9.noarch 2024-03-11T21:16:05.335 INFO:teuthology.orchestra.run.smithi092.stdout: 2024-03-11T21:16:05.335 INFO:teuthology.orchestra.run.smithi092.stdout:Complete! 2024-03-11T21:16:05.589 DEBUG:teuthology.orchestra.run.smithi092:> sudo yum -y install ceph-mgr-diskprediction-local 2024-03-11T21:16:06.146 INFO:teuthology.orchestra.run.smithi092.stdout:Last metadata expiration check: 0:01:10 ago on Mon 11 Mar 2024 09:14:56 PM UTC. 2024-03-11T21:16:06.299 INFO:teuthology.orchestra.run.smithi092.stdout:Dependencies resolved. 2024-03-11T21:16:06.300 INFO:teuthology.orchestra.run.smithi092.stdout:============================================================================================ 2024-03-11T21:16:06.300 INFO:teuthology.orchestra.run.smithi092.stdout: Package Arch Version Repository Size 2024-03-11T21:16:06.300 INFO:teuthology.orchestra.run.smithi092.stdout:============================================================================================ 2024-03-11T21:16:06.300 INFO:teuthology.orchestra.run.smithi092.stdout:Installing: 2024-03-11T21:16:06.300 INFO:teuthology.orchestra.run.smithi092.stdout: ceph-mgr-diskprediction-local noarch 2:19.0.0-1979.g2b0101eb.el9 ceph-noarch 7.4 M 2024-03-11T21:16:06.300 INFO:teuthology.orchestra.run.smithi092.stdout:Installing dependencies: 2024-03-11T21:16:06.300 INFO:teuthology.orchestra.run.smithi092.stdout: flexiblas x86_64 3.0.4-8.el9 CentOS-AppStream 32 k 2024-03-11T21:16:06.300 INFO:teuthology.orchestra.run.smithi092.stdout: flexiblas-netlib x86_64 3.0.4-8.el9 CentOS-AppStream 3.0 M 2024-03-11T21:16:06.300 INFO:teuthology.orchestra.run.smithi092.stdout: flexiblas-openblas-openmp x86_64 3.0.4-8.el9 CentOS-AppStream 17 k 2024-03-11T21:16:06.301 INFO:teuthology.orchestra.run.smithi092.stdout: libgfortran x86_64 11.4.1-3.el9 baseos 803 k 2024-03-11T21:16:06.301 INFO:teuthology.orchestra.run.smithi092.stdout: libquadmath x86_64 11.4.1-3.el9 baseos 195 k 2024-03-11T21:16:06.301 INFO:teuthology.orchestra.run.smithi092.stdout: openblas x86_64 0.3.21-2.el9 CentOS-AppStream 34 k 2024-03-11T21:16:06.301 INFO:teuthology.orchestra.run.smithi092.stdout: openblas-openmp x86_64 0.3.21-2.el9 CentOS-AppStream 4.7 M 2024-03-11T21:16:06.301 INFO:teuthology.orchestra.run.smithi092.stdout: python3-devel x86_64 3.9.17-2.el9 appstream 245 k 2024-03-11T21:16:06.301 INFO:teuthology.orchestra.run.smithi092.stdout: python3-numpy x86_64 1:1.20.1-5.el9 CentOS-AppStream 4.8 M 2024-03-11T21:16:06.301 INFO:teuthology.orchestra.run.smithi092.stdout: python3-numpy-f2py x86_64 1:1.20.1-5.el9 CentOS-AppStream 217 k 2024-03-11T21:16:06.301 INFO:teuthology.orchestra.run.smithi092.stdout: python3-scipy x86_64 1.6.2-8.el9 CentOS-AppStream 16 M 2024-03-11T21:16:06.301 INFO:teuthology.orchestra.run.smithi092.stdout: 2024-03-11T21:16:06.301 INFO:teuthology.orchestra.run.smithi092.stdout:Transaction Summary 2024-03-11T21:16:06.301 INFO:teuthology.orchestra.run.smithi092.stdout:============================================================================================ 2024-03-11T21:16:06.301 INFO:teuthology.orchestra.run.smithi092.stdout:Install 12 Packages 2024-03-11T21:16:06.301 INFO:teuthology.orchestra.run.smithi092.stdout: 2024-03-11T21:16:06.302 INFO:teuthology.orchestra.run.smithi092.stdout:Total download size: 37 M 2024-03-11T21:16:06.302 INFO:teuthology.orchestra.run.smithi092.stdout:Installed size: 209 M 2024-03-11T21:16:06.302 INFO:teuthology.orchestra.run.smithi092.stdout:Downloading Packages: 2024-03-11T21:16:06.609 INFO:teuthology.orchestra.run.smithi092.stdout:(1/12): flexiblas-3.0.4-8.el9.x86_64.rpm 204 kB/s | 32 kB 00:00 2024-03-11T21:16:06.643 INFO:teuthology.orchestra.run.smithi092.stdout:(2/12): flexiblas-openblas-openmp-3.0.4-8.el9.x 533 kB/s | 17 kB 00:00 2024-03-11T21:16:06.676 INFO:teuthology.orchestra.run.smithi092.stdout:(3/12): openblas-0.3.21-2.el9.x86_64.rpm 1.0 MB/s | 34 kB 00:00 2024-03-11T21:16:06.744 INFO:teuthology.orchestra.run.smithi092.stdout:(4/12): flexiblas-netlib-3.0.4-8.el9.x86_64.rpm 10 MB/s | 3.0 MB 00:00 2024-03-11T21:16:06.927 INFO:teuthology.orchestra.run.smithi092.stdout:(5/12): openblas-openmp-0.3.21-2.el9.x86_64.rpm 19 MB/s | 4.7 MB 00:00 2024-03-11T21:16:06.994 INFO:teuthology.orchestra.run.smithi092.stdout:(6/12): python3-numpy-1.20.1-5.el9.x86_64.rpm 19 MB/s | 4.8 MB 00:00 2024-03-11T21:16:07.019 INFO:teuthology.orchestra.run.smithi092.stdout:(7/12): python3-numpy-f2py-1.20.1-5.el9.x86_64. 2.4 MB/s | 217 kB 00:00 2024-03-11T21:16:07.236 INFO:teuthology.orchestra.run.smithi092.stdout:(8/12): python3-scipy-1.6.2-8.el9.x86_64.rpm 65 MB/s | 16 MB 00:00 2024-03-11T21:16:07.936 INFO:teuthology.orchestra.run.smithi092.stdout:(9/12): libquadmath-11.4.1-3.el9.x86_64.rpm 279 kB/s | 195 kB 00:00 2024-03-11T21:16:08.262 INFO:teuthology.orchestra.run.smithi092.stdout:(10/12): libgfortran-11.4.1-3.el9.x86_64.rpm 648 kB/s | 803 kB 00:01 2024-03-11T21:16:08.287 INFO:teuthology.orchestra.run.smithi092.stdout:(11/12): python3-devel-3.9.17-2.el9.x86_64.rpm 701 kB/s | 245 kB 00:00 2024-03-11T21:16:09.070 INFO:teuthology.orchestra.run.smithi092.stdout:(12/12): ceph-mgr-diskprediction-local-19.0.0-1 2.8 MB/s | 7.4 MB 00:02 2024-03-11T21:16:09.072 INFO:teuthology.orchestra.run.smithi092.stdout:-------------------------------------------------------------------------------- 2024-03-11T21:16:09.072 INFO:teuthology.orchestra.run.smithi092.stdout:Total 13 MB/s | 37 MB 00:02 2024-03-11T21:16:09.114 INFO:teuthology.orchestra.run.smithi092.stdout:Running transaction check 2024-03-11T21:16:09.139 INFO:teuthology.orchestra.run.smithi092.stdout:Transaction check succeeded. 2024-03-11T21:16:09.139 INFO:teuthology.orchestra.run.smithi092.stdout:Running transaction test 2024-03-11T21:16:09.497 INFO:teuthology.orchestra.run.smithi092.stdout:Transaction test succeeded. 2024-03-11T21:16:09.497 INFO:teuthology.orchestra.run.smithi092.stdout:Running transaction 2024-03-11T21:16:10.201 INFO:teuthology.orchestra.run.smithi092.stdout: Preparing : 1/1 2024-03-11T21:16:10.276 INFO:teuthology.orchestra.run.smithi092.stdout: Installing : libquadmath-11.4.1-3.el9.x86_64 1/12 2024-03-11T21:16:10.308 INFO:teuthology.orchestra.run.smithi092.stdout: Installing : libgfortran-11.4.1-3.el9.x86_64 2/12 2024-03-11T21:16:10.401 INFO:teuthology.orchestra.run.smithi092.stdout: Installing : flexiblas-3.0.4-8.el9.x86_64 3/12 2024-03-11T21:16:10.444 INFO:teuthology.orchestra.run.smithi092.stdout: Installing : python3-devel-3.9.17-2.el9.x86_64 4/12 2024-03-11T21:16:10.616 INFO:teuthology.orchestra.run.smithi092.stdout: Installing : openblas-0.3.21-2.el9.x86_64 5/12 2024-03-11T21:16:10.643 INFO:teuthology.orchestra.run.smithi092.stdout: Installing : openblas-openmp-0.3.21-2.el9.x86_64 6/12 2024-03-11T21:16:10.737 INFO:teuthology.orchestra.run.smithi092.stdout: Installing : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 7/12 2024-03-11T21:16:11.410 INFO:teuthology.orchestra.run.smithi092.stdout: Installing : flexiblas-netlib-3.0.4-8.el9.x86_64 8/12 2024-03-11T21:16:11.522 INFO:teuthology.orchestra.run.smithi092.stdout: Installing : python3-numpy-1:1.20.1-5.el9.x86_64 9/12 2024-03-11T21:16:12.745 INFO:teuthology.orchestra.run.smithi092.stdout: Installing : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 10/12 2024-03-11T21:16:13.779 INFO:teuthology.orchestra.run.smithi092.stdout: Installing : python3-scipy-1.6.2-8.el9.x86_64 11/12 2024-03-11T21:16:13.817 INFO:teuthology.orchestra.run.smithi092.stdout: Installing : ceph-mgr-diskprediction-local-2:19.0.0-1979.g2b010 12/12 2024-03-11T21:16:16.334 INFO:teuthology.orchestra.run.smithi092.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:19.0.0-1979.g2b010 12/12 2024-03-11T21:16:16.334 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : ceph-mgr-diskprediction-local-2:19.0.0-1979.g2b010 1/12 2024-03-11T21:16:16.334 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : flexiblas-3.0.4-8.el9.x86_64 2/12 2024-03-11T21:16:16.334 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : flexiblas-netlib-3.0.4-8.el9.x86_64 3/12 2024-03-11T21:16:16.334 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 4/12 2024-03-11T21:16:16.334 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : openblas-0.3.21-2.el9.x86_64 5/12 2024-03-11T21:16:16.334 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : openblas-openmp-0.3.21-2.el9.x86_64 6/12 2024-03-11T21:16:16.334 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : python3-numpy-1:1.20.1-5.el9.x86_64 7/12 2024-03-11T21:16:16.334 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 8/12 2024-03-11T21:16:16.334 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : python3-scipy-1.6.2-8.el9.x86_64 9/12 2024-03-11T21:16:16.334 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : libgfortran-11.4.1-3.el9.x86_64 10/12 2024-03-11T21:16:16.335 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : libquadmath-11.4.1-3.el9.x86_64 11/12 2024-03-11T21:16:16.736 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : python3-devel-3.9.17-2.el9.x86_64 12/12 2024-03-11T21:16:16.736 INFO:teuthology.orchestra.run.smithi092.stdout: 2024-03-11T21:16:16.736 INFO:teuthology.orchestra.run.smithi092.stdout:Installed: 2024-03-11T21:16:16.736 INFO:teuthology.orchestra.run.smithi092.stdout: ceph-mgr-diskprediction-local-2:19.0.0-1979.g2b0101eb.el9.noarch 2024-03-11T21:16:16.736 INFO:teuthology.orchestra.run.smithi092.stdout: flexiblas-3.0.4-8.el9.x86_64 2024-03-11T21:16:16.737 INFO:teuthology.orchestra.run.smithi092.stdout: flexiblas-netlib-3.0.4-8.el9.x86_64 2024-03-11T21:16:16.737 INFO:teuthology.orchestra.run.smithi092.stdout: flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 2024-03-11T21:16:16.737 INFO:teuthology.orchestra.run.smithi092.stdout: libgfortran-11.4.1-3.el9.x86_64 2024-03-11T21:16:16.737 INFO:teuthology.orchestra.run.smithi092.stdout: libquadmath-11.4.1-3.el9.x86_64 2024-03-11T21:16:16.737 INFO:teuthology.orchestra.run.smithi092.stdout: openblas-0.3.21-2.el9.x86_64 2024-03-11T21:16:16.737 INFO:teuthology.orchestra.run.smithi092.stdout: openblas-openmp-0.3.21-2.el9.x86_64 2024-03-11T21:16:16.737 INFO:teuthology.orchestra.run.smithi092.stdout: python3-devel-3.9.17-2.el9.x86_64 2024-03-11T21:16:16.737 INFO:teuthology.orchestra.run.smithi092.stdout: python3-numpy-1:1.20.1-5.el9.x86_64 2024-03-11T21:16:16.737 INFO:teuthology.orchestra.run.smithi092.stdout: python3-numpy-f2py-1:1.20.1-5.el9.x86_64 2024-03-11T21:16:16.737 INFO:teuthology.orchestra.run.smithi092.stdout: python3-scipy-1.6.2-8.el9.x86_64 2024-03-11T21:16:16.737 INFO:teuthology.orchestra.run.smithi092.stdout: 2024-03-11T21:16:16.737 INFO:teuthology.orchestra.run.smithi092.stdout:Complete! 2024-03-11T21:16:17.131 DEBUG:teuthology.orchestra.run.smithi092:> sudo yum -y install ceph-mgr-rook 2024-03-11T21:16:17.695 INFO:teuthology.orchestra.run.smithi092.stdout:Last metadata expiration check: 0:01:21 ago on Mon 11 Mar 2024 09:14:56 PM UTC. 2024-03-11T21:16:17.844 INFO:teuthology.orchestra.run.smithi092.stdout:Dependencies resolved. 2024-03-11T21:16:17.845 INFO:teuthology.orchestra.run.smithi092.stdout:======================================================================================== 2024-03-11T21:16:17.846 INFO:teuthology.orchestra.run.smithi092.stdout: Package Arch Version Repository Size 2024-03-11T21:16:17.846 INFO:teuthology.orchestra.run.smithi092.stdout:======================================================================================== 2024-03-11T21:16:17.846 INFO:teuthology.orchestra.run.smithi092.stdout:Installing: 2024-03-11T21:16:17.846 INFO:teuthology.orchestra.run.smithi092.stdout: ceph-mgr-rook noarch 2:19.0.0-1979.g2b0101eb.el9 ceph-noarch 49 k 2024-03-11T21:16:17.846 INFO:teuthology.orchestra.run.smithi092.stdout:Installing dependencies: 2024-03-11T21:16:17.846 INFO:teuthology.orchestra.run.smithi092.stdout: python3-cachetools noarch 4.2.4-1.el9 epel 32 k 2024-03-11T21:16:17.846 INFO:teuthology.orchestra.run.smithi092.stdout: python3-certifi noarch 2021.10.8-2.el9 epel 15 k 2024-03-11T21:16:17.846 INFO:teuthology.orchestra.run.smithi092.stdout: python3-google-auth noarch 1:2.28.0-1.el9 epel 203 k 2024-03-11T21:16:17.846 INFO:teuthology.orchestra.run.smithi092.stdout: python3-jsonpatch noarch 1.21-16.el9 CentOS-AppStream 24 k 2024-03-11T21:16:17.846 INFO:teuthology.orchestra.run.smithi092.stdout: python3-jsonpointer noarch 2.0-4.el9 CentOS-AppStream 18 k 2024-03-11T21:16:17.846 INFO:teuthology.orchestra.run.smithi092.stdout: python3-kubernetes noarch 1:26.1.0-2.el9 epel 1.0 M 2024-03-11T21:16:17.846 INFO:teuthology.orchestra.run.smithi092.stdout: python3-oauthlib noarch 3.1.1-5.el9 CentOS-AppStream 174 k 2024-03-11T21:16:17.846 INFO:teuthology.orchestra.run.smithi092.stdout: python3-pyasn1 noarch 0.4.8-6.el9 CentOS-AppStream 132 k 2024-03-11T21:16:17.847 INFO:teuthology.orchestra.run.smithi092.stdout: python3-pyasn1-modules noarch 0.4.8-6.el9 CentOS-AppStream 211 k 2024-03-11T21:16:17.847 INFO:teuthology.orchestra.run.smithi092.stdout: python3-requests-oauthlib noarch 1.3.0-12.el9 CentOS-AppStream 43 k 2024-03-11T21:16:17.847 INFO:teuthology.orchestra.run.smithi092.stdout: python3-rsa noarch 4.9-2.el9 epel 59 k 2024-03-11T21:16:17.847 INFO:teuthology.orchestra.run.smithi092.stdout: python3-websocket-client noarch 1.2.3-2.el9 epel 90 k 2024-03-11T21:16:17.847 INFO:teuthology.orchestra.run.smithi092.stdout: 2024-03-11T21:16:17.847 INFO:teuthology.orchestra.run.smithi092.stdout:Transaction Summary 2024-03-11T21:16:17.847 INFO:teuthology.orchestra.run.smithi092.stdout:======================================================================================== 2024-03-11T21:16:17.847 INFO:teuthology.orchestra.run.smithi092.stdout:Install 13 Packages 2024-03-11T21:16:17.847 INFO:teuthology.orchestra.run.smithi092.stdout: 2024-03-11T21:16:17.848 INFO:teuthology.orchestra.run.smithi092.stdout:Total download size: 2.0 M 2024-03-11T21:16:17.848 INFO:teuthology.orchestra.run.smithi092.stdout:Installed size: 26 M 2024-03-11T21:16:17.848 INFO:teuthology.orchestra.run.smithi092.stdout:Downloading Packages: 2024-03-11T21:16:18.217 INFO:teuthology.orchestra.run.smithi092.stdout:(1/13): python3-jsonpatch-1.21-16.el9.noarch.rp 216 kB/s | 24 kB 00:00 2024-03-11T21:16:18.242 INFO:teuthology.orchestra.run.smithi092.stdout:(2/13): python3-jsonpointer-2.0-4.el9.noarch.rp 131 kB/s | 18 kB 00:00 2024-03-11T21:16:18.301 INFO:teuthology.orchestra.run.smithi092.stdout:(3/13): python3-oauthlib-3.1.1-5.el9.noarch.rpm 2.0 MB/s | 174 kB 00:00 2024-03-11T21:16:18.326 INFO:teuthology.orchestra.run.smithi092.stdout:(4/13): python3-pyasn1-0.4.8-6.el9.noarch.rpm 1.5 MB/s | 132 kB 00:00 2024-03-11T21:16:18.351 INFO:teuthology.orchestra.run.smithi092.stdout:(5/13): python3-pyasn1-modules-0.4.8-6.el9.noar 4.1 MB/s | 211 kB 00:00 2024-03-11T21:16:18.377 INFO:teuthology.orchestra.run.smithi092.stdout:(6/13): python3-requests-oauthlib-1.3.0-12.el9. 860 kB/s | 43 kB 00:00 2024-03-11T21:16:18.518 INFO:teuthology.orchestra.run.smithi092.stdout:(7/13): ceph-mgr-rook-19.0.0-1979.g2b0101eb.el9 119 kB/s | 49 kB 00:00 2024-03-11T21:16:18.552 INFO:teuthology.orchestra.run.smithi092.stdout:(8/13): python3-certifi-2021.10.8-2.el9.noarch. 85 kB/s | 15 kB 00:00 2024-03-11T21:16:18.577 INFO:teuthology.orchestra.run.smithi092.stdout:(9/13): python3-cachetools-4.2.4-1.el9.noarch.r 143 kB/s | 32 kB 00:00 2024-03-11T21:16:18.778 INFO:teuthology.orchestra.run.smithi092.stdout:(10/13): python3-rsa-4.9-2.el9.noarch.rpm 295 kB/s | 59 kB 00:00 2024-03-11T21:16:18.928 INFO:teuthology.orchestra.run.smithi092.stdout:(11/13): python3-websocket-client-1.2.3-2.el9.n 598 kB/s | 90 kB 00:00 2024-03-11T21:16:18.944 INFO:teuthology.orchestra.run.smithi092.stdout:(12/13): python3-google-auth-2.28.0-1.el9.noarc 476 kB/s | 203 kB 00:00 2024-03-11T21:16:19.220 INFO:teuthology.orchestra.run.smithi092.stdout:(13/13): python3-kubernetes-26.1.0-2.el9.noarch 1.5 MB/s | 1.0 MB 00:00 2024-03-11T21:16:19.222 INFO:teuthology.orchestra.run.smithi092.stdout:-------------------------------------------------------------------------------- 2024-03-11T21:16:19.222 INFO:teuthology.orchestra.run.smithi092.stdout:Total 1.5 MB/s | 2.0 MB 00:01 2024-03-11T21:16:19.222 INFO:teuthology.orchestra.run.smithi092.stdout:Running transaction check 2024-03-11T21:16:19.236 INFO:teuthology.orchestra.run.smithi092.stdout:Transaction check succeeded. 2024-03-11T21:16:19.236 INFO:teuthology.orchestra.run.smithi092.stdout:Running transaction test 2024-03-11T21:16:19.339 INFO:teuthology.orchestra.run.smithi092.stdout:Transaction test succeeded. 2024-03-11T21:16:19.339 INFO:teuthology.orchestra.run.smithi092.stdout:Running transaction 2024-03-11T21:16:19.858 INFO:teuthology.orchestra.run.smithi092.stdout: Preparing : 1/1 2024-03-11T21:16:20.024 INFO:teuthology.orchestra.run.smithi092.stdout: Installing : python3-pyasn1-0.4.8-6.el9.noarch 1/13 2024-03-11T21:16:20.070 INFO:teuthology.orchestra.run.smithi092.stdout: Installing : python3-pyasn1-modules-0.4.8-6.el9.noarch 2/13 2024-03-11T21:16:20.134 INFO:teuthology.orchestra.run.smithi092.stdout: Installing : python3-rsa-4.9-2.el9.noarch 3/13 2024-03-11T21:16:20.163 INFO:teuthology.orchestra.run.smithi092.stdout: Installing : python3-websocket-client-1.2.3-2.el9.noarch 4/13 2024-03-11T21:16:20.205 INFO:teuthology.orchestra.run.smithi092.stdout: Installing : python3-certifi-2021.10.8-2.el9.noarch 5/13 2024-03-11T21:16:20.298 INFO:teuthology.orchestra.run.smithi092.stdout: Installing : python3-cachetools-4.2.4-1.el9.noarch 6/13 2024-03-11T21:16:20.414 INFO:teuthology.orchestra.run.smithi092.stdout: Installing : python3-google-auth-1:2.28.0-1.el9.noarch 7/13 2024-03-11T21:16:20.465 INFO:teuthology.orchestra.run.smithi092.stdout: Installing : python3-oauthlib-3.1.1-5.el9.noarch 8/13 2024-03-11T21:16:21.191 INFO:teuthology.orchestra.run.smithi092.stdout: Installing : python3-requests-oauthlib-1.3.0-12.el9.noarch 9/13 2024-03-11T21:16:21.244 INFO:teuthology.orchestra.run.smithi092.stdout: Installing : python3-kubernetes-1:26.1.0-2.el9.noarch 10/13 2024-03-11T21:16:21.278 INFO:teuthology.orchestra.run.smithi092.stdout: Installing : python3-jsonpointer-2.0-4.el9.noarch 11/13 2024-03-11T21:16:21.326 INFO:teuthology.orchestra.run.smithi092.stdout: Installing : python3-jsonpatch-1.21-16.el9.noarch 12/13 2024-03-11T21:16:21.348 INFO:teuthology.orchestra.run.smithi092.stdout: Installing : ceph-mgr-rook-2:19.0.0-1979.g2b0101eb.el9.noarch 13/13 2024-03-11T21:16:22.130 INFO:teuthology.orchestra.run.smithi092.stdout: Running scriptlet: ceph-mgr-rook-2:19.0.0-1979.g2b0101eb.el9.noarch 13/13 2024-03-11T21:16:22.130 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : ceph-mgr-rook-2:19.0.0-1979.g2b0101eb.el9.noarch 1/13 2024-03-11T21:16:22.130 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : python3-jsonpatch-1.21-16.el9.noarch 2/13 2024-03-11T21:16:22.130 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : python3-jsonpointer-2.0-4.el9.noarch 3/13 2024-03-11T21:16:22.130 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : python3-oauthlib-3.1.1-5.el9.noarch 4/13 2024-03-11T21:16:22.130 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : python3-pyasn1-0.4.8-6.el9.noarch 5/13 2024-03-11T21:16:22.130 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : python3-pyasn1-modules-0.4.8-6.el9.noarch 6/13 2024-03-11T21:16:22.131 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : python3-requests-oauthlib-1.3.0-12.el9.noarch 7/13 2024-03-11T21:16:22.131 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : python3-cachetools-4.2.4-1.el9.noarch 8/13 2024-03-11T21:16:22.131 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : python3-certifi-2021.10.8-2.el9.noarch 9/13 2024-03-11T21:16:22.131 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : python3-google-auth-1:2.28.0-1.el9.noarch 10/13 2024-03-11T21:16:22.131 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : python3-kubernetes-1:26.1.0-2.el9.noarch 11/13 2024-03-11T21:16:22.131 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : python3-rsa-4.9-2.el9.noarch 12/13 2024-03-11T21:16:22.486 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : python3-websocket-client-1.2.3-2.el9.noarch 13/13 2024-03-11T21:16:22.486 INFO:teuthology.orchestra.run.smithi092.stdout: 2024-03-11T21:16:22.486 INFO:teuthology.orchestra.run.smithi092.stdout:Installed: 2024-03-11T21:16:22.486 INFO:teuthology.orchestra.run.smithi092.stdout: ceph-mgr-rook-2:19.0.0-1979.g2b0101eb.el9.noarch 2024-03-11T21:16:22.486 INFO:teuthology.orchestra.run.smithi092.stdout: python3-cachetools-4.2.4-1.el9.noarch 2024-03-11T21:16:22.486 INFO:teuthology.orchestra.run.smithi092.stdout: python3-certifi-2021.10.8-2.el9.noarch 2024-03-11T21:16:22.487 INFO:teuthology.orchestra.run.smithi092.stdout: python3-google-auth-1:2.28.0-1.el9.noarch 2024-03-11T21:16:22.487 INFO:teuthology.orchestra.run.smithi092.stdout: python3-jsonpatch-1.21-16.el9.noarch 2024-03-11T21:16:22.487 INFO:teuthology.orchestra.run.smithi092.stdout: python3-jsonpointer-2.0-4.el9.noarch 2024-03-11T21:16:22.487 INFO:teuthology.orchestra.run.smithi092.stdout: python3-kubernetes-1:26.1.0-2.el9.noarch 2024-03-11T21:16:22.487 INFO:teuthology.orchestra.run.smithi092.stdout: python3-oauthlib-3.1.1-5.el9.noarch 2024-03-11T21:16:22.487 INFO:teuthology.orchestra.run.smithi092.stdout: python3-pyasn1-0.4.8-6.el9.noarch 2024-03-11T21:16:22.487 INFO:teuthology.orchestra.run.smithi092.stdout: python3-pyasn1-modules-0.4.8-6.el9.noarch 2024-03-11T21:16:22.487 INFO:teuthology.orchestra.run.smithi092.stdout: python3-requests-oauthlib-1.3.0-12.el9.noarch 2024-03-11T21:16:22.487 INFO:teuthology.orchestra.run.smithi092.stdout: python3-rsa-4.9-2.el9.noarch 2024-03-11T21:16:22.487 INFO:teuthology.orchestra.run.smithi092.stdout: python3-websocket-client-1.2.3-2.el9.noarch 2024-03-11T21:16:22.487 INFO:teuthology.orchestra.run.smithi092.stdout: 2024-03-11T21:16:22.487 INFO:teuthology.orchestra.run.smithi092.stdout:Complete! 2024-03-11T21:16:22.740 DEBUG:teuthology.orchestra.run.smithi092:> sudo yum -y install ceph-mgr-cephadm 2024-03-11T21:16:23.303 INFO:teuthology.orchestra.run.smithi092.stdout:Last metadata expiration check: 0:01:27 ago on Mon 11 Mar 2024 09:14:56 PM UTC. 2024-03-11T21:16:23.456 INFO:teuthology.orchestra.run.smithi092.stdout:Dependencies resolved. 2024-03-11T21:16:23.457 INFO:teuthology.orchestra.run.smithi092.stdout:======================================================================================== 2024-03-11T21:16:23.457 INFO:teuthology.orchestra.run.smithi092.stdout: Package Arch Version Repository Size 2024-03-11T21:16:23.457 INFO:teuthology.orchestra.run.smithi092.stdout:======================================================================================== 2024-03-11T21:16:23.457 INFO:teuthology.orchestra.run.smithi092.stdout:Installing: 2024-03-11T21:16:23.457 INFO:teuthology.orchestra.run.smithi092.stdout: ceph-mgr-cephadm noarch 2:19.0.0-1979.g2b0101eb.el9 ceph-noarch 139 k 2024-03-11T21:16:23.457 INFO:teuthology.orchestra.run.smithi092.stdout:Installing dependencies: 2024-03-11T21:16:23.457 INFO:teuthology.orchestra.run.smithi092.stdout: python3-asyncssh noarch 2.13.2-4.el9 epel 548 k 2024-03-11T21:16:23.458 INFO:teuthology.orchestra.run.smithi092.stdout: python3-babel noarch 2.9.1-2.el9 CentOS-AppStream 5.8 M 2024-03-11T21:16:23.458 INFO:teuthology.orchestra.run.smithi092.stdout: python3-jinja2 noarch 2.11.3-5.el9 appstream 248 k 2024-03-11T21:16:23.458 INFO:teuthology.orchestra.run.smithi092.stdout: python3-natsort noarch 7.1.1-5.el9 epel 58 k 2024-03-11T21:16:23.458 INFO:teuthology.orchestra.run.smithi092.stdout: python3-typing-extensions noarch 4.4.0-2.el9 epel 51 k 2024-03-11T21:16:23.458 INFO:teuthology.orchestra.run.smithi092.stdout: 2024-03-11T21:16:23.458 INFO:teuthology.orchestra.run.smithi092.stdout:Transaction Summary 2024-03-11T21:16:23.459 INFO:teuthology.orchestra.run.smithi092.stdout:======================================================================================== 2024-03-11T21:16:23.459 INFO:teuthology.orchestra.run.smithi092.stdout:Install 6 Packages 2024-03-11T21:16:23.459 INFO:teuthology.orchestra.run.smithi092.stdout: 2024-03-11T21:16:23.460 INFO:teuthology.orchestra.run.smithi092.stdout:Total download size: 6.8 M 2024-03-11T21:16:23.460 INFO:teuthology.orchestra.run.smithi092.stdout:Installed size: 33 M 2024-03-11T21:16:23.460 INFO:teuthology.orchestra.run.smithi092.stdout:Downloading Packages: 2024-03-11T21:16:23.796 INFO:teuthology.orchestra.run.smithi092.stdout:(1/6): ceph-mgr-cephadm-19.0.0-1979.g2b0101eb.e 614 kB/s | 139 kB 00:00 2024-03-11T21:16:23.874 INFO:teuthology.orchestra.run.smithi092.stdout:(2/6): python3-babel-2.9.1-2.el9.noarch.rpm 19 MB/s | 5.8 MB 00:00 2024-03-11T21:16:23.896 INFO:teuthology.orchestra.run.smithi092.stdout:(3/6): python3-jinja2-2.11.3-5.el9.noarch.rpm 760 kB/s | 248 kB 00:00 2024-03-11T21:16:24.097 INFO:teuthology.orchestra.run.smithi092.stdout:(4/6): python3-natsort-7.1.1-5.el9.noarch.rpm 256 kB/s | 58 kB 00:00 2024-03-11T21:16:24.122 INFO:teuthology.orchestra.run.smithi092.stdout:(5/6): python3-typing-extensions-4.4.0-2.el9.no 226 kB/s | 51 kB 00:00 2024-03-11T21:16:24.214 INFO:teuthology.orchestra.run.smithi092.stdout:(6/6): python3-asyncssh-2.13.2-4.el9.noarch.rpm 1.3 MB/s | 548 kB 00:00 2024-03-11T21:16:24.217 INFO:teuthology.orchestra.run.smithi092.stdout:-------------------------------------------------------------------------------- 2024-03-11T21:16:24.217 INFO:teuthology.orchestra.run.smithi092.stdout:Total 8.9 MB/s | 6.8 MB 00:00 2024-03-11T21:16:24.230 INFO:teuthology.orchestra.run.smithi092.stdout:Running transaction check 2024-03-11T21:16:24.240 INFO:teuthology.orchestra.run.smithi092.stdout:Transaction check succeeded. 2024-03-11T21:16:24.240 INFO:teuthology.orchestra.run.smithi092.stdout:Running transaction test 2024-03-11T21:16:24.333 INFO:teuthology.orchestra.run.smithi092.stdout:Transaction test succeeded. 2024-03-11T21:16:24.333 INFO:teuthology.orchestra.run.smithi092.stdout:Running transaction 2024-03-11T21:16:24.652 INFO:teuthology.orchestra.run.smithi092.stdout: Preparing : 1/1 2024-03-11T21:16:24.814 INFO:teuthology.orchestra.run.smithi092.stdout: Installing : python3-typing-extensions-4.4.0-2.el9.noarch 1/6 2024-03-11T21:16:24.874 INFO:teuthology.orchestra.run.smithi092.stdout: Installing : python3-asyncssh-2.13.2-4.el9.noarch 2/6 2024-03-11T21:16:25.395 INFO:teuthology.orchestra.run.smithi092.stdout: Installing : python3-natsort-7.1.1-5.el9.noarch 3/6 2024-03-11T21:16:25.485 INFO:teuthology.orchestra.run.smithi092.stdout: Installing : python3-babel-2.9.1-2.el9.noarch 4/6 2024-03-11T21:16:25.545 INFO:teuthology.orchestra.run.smithi092.stdout: Installing : python3-jinja2-2.11.3-5.el9.noarch 5/6 2024-03-11T21:16:25.569 INFO:teuthology.orchestra.run.smithi092.stdout: Installing : ceph-mgr-cephadm-2:19.0.0-1979.g2b0101eb.el9.noarch 6/6 2024-03-11T21:16:26.412 INFO:teuthology.orchestra.run.smithi092.stdout: Running scriptlet: ceph-mgr-cephadm-2:19.0.0-1979.g2b0101eb.el9.noarch 6/6 2024-03-11T21:16:26.412 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : ceph-mgr-cephadm-2:19.0.0-1979.g2b0101eb.el9.noarch 1/6 2024-03-11T21:16:26.412 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : python3-babel-2.9.1-2.el9.noarch 2/6 2024-03-11T21:16:26.412 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : python3-jinja2-2.11.3-5.el9.noarch 3/6 2024-03-11T21:16:26.412 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : python3-asyncssh-2.13.2-4.el9.noarch 4/6 2024-03-11T21:16:26.412 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : python3-natsort-7.1.1-5.el9.noarch 5/6 2024-03-11T21:16:26.716 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : python3-typing-extensions-4.4.0-2.el9.noarch 6/6 2024-03-11T21:16:26.716 INFO:teuthology.orchestra.run.smithi092.stdout: 2024-03-11T21:16:26.716 INFO:teuthology.orchestra.run.smithi092.stdout:Installed: 2024-03-11T21:16:26.716 INFO:teuthology.orchestra.run.smithi092.stdout: ceph-mgr-cephadm-2:19.0.0-1979.g2b0101eb.el9.noarch 2024-03-11T21:16:26.716 INFO:teuthology.orchestra.run.smithi092.stdout: python3-asyncssh-2.13.2-4.el9.noarch 2024-03-11T21:16:26.716 INFO:teuthology.orchestra.run.smithi092.stdout: python3-babel-2.9.1-2.el9.noarch 2024-03-11T21:16:26.716 INFO:teuthology.orchestra.run.smithi092.stdout: python3-jinja2-2.11.3-5.el9.noarch 2024-03-11T21:16:26.716 INFO:teuthology.orchestra.run.smithi092.stdout: python3-natsort-7.1.1-5.el9.noarch 2024-03-11T21:16:26.717 INFO:teuthology.orchestra.run.smithi092.stdout: python3-typing-extensions-4.4.0-2.el9.noarch 2024-03-11T21:16:26.717 INFO:teuthology.orchestra.run.smithi092.stdout: 2024-03-11T21:16:26.717 INFO:teuthology.orchestra.run.smithi092.stdout:Complete! 2024-03-11T21:16:26.980 DEBUG:teuthology.orchestra.run.smithi092:> sudo yum -y install ceph-fuse 2024-03-11T21:16:27.540 INFO:teuthology.orchestra.run.smithi092.stdout:Last metadata expiration check: 0:01:31 ago on Mon 11 Mar 2024 09:14:56 PM UTC. 2024-03-11T21:16:27.689 INFO:teuthology.orchestra.run.smithi092.stdout:Dependencies resolved. 2024-03-11T21:16:27.689 INFO:teuthology.orchestra.run.smithi092.stdout:================================================================================ 2024-03-11T21:16:27.689 INFO:teuthology.orchestra.run.smithi092.stdout: Package Arch Version Repository Size 2024-03-11T21:16:27.690 INFO:teuthology.orchestra.run.smithi092.stdout:================================================================================ 2024-03-11T21:16:27.690 INFO:teuthology.orchestra.run.smithi092.stdout:Installing: 2024-03-11T21:16:27.690 INFO:teuthology.orchestra.run.smithi092.stdout: ceph-fuse x86_64 2:19.0.0-1979.g2b0101eb.el9 ceph 888 k 2024-03-11T21:16:27.690 INFO:teuthology.orchestra.run.smithi092.stdout:Installing dependencies: 2024-03-11T21:16:27.690 INFO:teuthology.orchestra.run.smithi092.stdout: fuse x86_64 2.9.9-15.el9 CentOS-BaseOS 79 k 2024-03-11T21:16:27.690 INFO:teuthology.orchestra.run.smithi092.stdout: 2024-03-11T21:16:27.690 INFO:teuthology.orchestra.run.smithi092.stdout:Transaction Summary 2024-03-11T21:16:27.690 INFO:teuthology.orchestra.run.smithi092.stdout:================================================================================ 2024-03-11T21:16:27.690 INFO:teuthology.orchestra.run.smithi092.stdout:Install 2 Packages 2024-03-11T21:16:27.690 INFO:teuthology.orchestra.run.smithi092.stdout: 2024-03-11T21:16:27.691 INFO:teuthology.orchestra.run.smithi092.stdout:Total download size: 966 k 2024-03-11T21:16:27.691 INFO:teuthology.orchestra.run.smithi092.stdout:Installed size: 2.8 M 2024-03-11T21:16:27.691 INFO:teuthology.orchestra.run.smithi092.stdout:Downloading Packages: 2024-03-11T21:16:27.823 INFO:teuthology.orchestra.run.smithi092.stdout:(1/2): fuse-2.9.9-15.el9.x86_64.rpm 595 kB/s | 79 kB 00:00 2024-03-11T21:16:28.057 INFO:teuthology.orchestra.run.smithi092.stdout:(2/2): ceph-fuse-19.0.0-1979.g2b0101eb.el9.x86_ 2.4 MB/s | 888 kB 00:00 2024-03-11T21:16:28.058 INFO:teuthology.orchestra.run.smithi092.stdout:-------------------------------------------------------------------------------- 2024-03-11T21:16:28.059 INFO:teuthology.orchestra.run.smithi092.stdout:Total 2.6 MB/s | 966 kB 00:00 2024-03-11T21:16:28.059 INFO:teuthology.orchestra.run.smithi092.stdout:Running transaction check 2024-03-11T21:16:28.080 INFO:teuthology.orchestra.run.smithi092.stdout:Transaction check succeeded. 2024-03-11T21:16:28.080 INFO:teuthology.orchestra.run.smithi092.stdout:Running transaction test 2024-03-11T21:16:28.156 INFO:teuthology.orchestra.run.smithi092.stdout:Transaction test succeeded. 2024-03-11T21:16:28.157 INFO:teuthology.orchestra.run.smithi092.stdout:Running transaction 2024-03-11T21:16:28.349 INFO:teuthology.orchestra.run.smithi092.stdout: Preparing : 1/1 2024-03-11T21:16:28.488 INFO:teuthology.orchestra.run.smithi092.stdout: Installing : fuse-2.9.9-15.el9.x86_64 1/2 2024-03-11T21:16:28.556 INFO:teuthology.orchestra.run.smithi092.stdout: Installing : ceph-fuse-2:19.0.0-1979.g2b0101eb.el9.x86_64 2/2 2024-03-11T21:16:29.276 INFO:teuthology.orchestra.run.smithi092.stdout: Running scriptlet: ceph-fuse-2:19.0.0-1979.g2b0101eb.el9.x86_64 2/2 2024-03-11T21:16:29.276 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : ceph-fuse-2:19.0.0-1979.g2b0101eb.el9.x86_64 1/2 2024-03-11T21:16:29.596 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : fuse-2.9.9-15.el9.x86_64 2/2 2024-03-11T21:16:29.596 INFO:teuthology.orchestra.run.smithi092.stdout: 2024-03-11T21:16:29.596 INFO:teuthology.orchestra.run.smithi092.stdout:Installed: 2024-03-11T21:16:29.596 INFO:teuthology.orchestra.run.smithi092.stdout: ceph-fuse-2:19.0.0-1979.g2b0101eb.el9.x86_64 fuse-2.9.9-15.el9.x86_64 2024-03-11T21:16:29.596 INFO:teuthology.orchestra.run.smithi092.stdout: 2024-03-11T21:16:29.597 INFO:teuthology.orchestra.run.smithi092.stdout:Complete! 2024-03-11T21:16:29.865 DEBUG:teuthology.orchestra.run.smithi092:> sudo yum -y install ceph-volume 2024-03-11T21:16:30.437 INFO:teuthology.orchestra.run.smithi092.stdout:Last metadata expiration check: 0:01:34 ago on Mon 11 Mar 2024 09:14:56 PM UTC. 2024-03-11T21:16:30.587 INFO:teuthology.orchestra.run.smithi092.stdout:Dependencies resolved. 2024-03-11T21:16:30.588 INFO:teuthology.orchestra.run.smithi092.stdout:================================================================================ 2024-03-11T21:16:30.588 INFO:teuthology.orchestra.run.smithi092.stdout: Package Arch Version Repository Size 2024-03-11T21:16:30.588 INFO:teuthology.orchestra.run.smithi092.stdout:================================================================================ 2024-03-11T21:16:30.588 INFO:teuthology.orchestra.run.smithi092.stdout:Installing: 2024-03-11T21:16:30.588 INFO:teuthology.orchestra.run.smithi092.stdout: ceph-volume noarch 2:19.0.0-1979.g2b0101eb.el9 ceph-noarch 264 k 2024-03-11T21:16:30.588 INFO:teuthology.orchestra.run.smithi092.stdout:Installing dependencies: 2024-03-11T21:16:30.588 INFO:teuthology.orchestra.run.smithi092.stdout: python3-packaging noarch 20.9-5.el9 CentOS-AppStream 69 k 2024-03-11T21:16:30.588 INFO:teuthology.orchestra.run.smithi092.stdout: 2024-03-11T21:16:30.588 INFO:teuthology.orchestra.run.smithi092.stdout:Transaction Summary 2024-03-11T21:16:30.588 INFO:teuthology.orchestra.run.smithi092.stdout:================================================================================ 2024-03-11T21:16:30.588 INFO:teuthology.orchestra.run.smithi092.stdout:Install 2 Packages 2024-03-11T21:16:30.589 INFO:teuthology.orchestra.run.smithi092.stdout: 2024-03-11T21:16:30.589 INFO:teuthology.orchestra.run.smithi092.stdout:Total download size: 332 k 2024-03-11T21:16:30.589 INFO:teuthology.orchestra.run.smithi092.stdout:Installed size: 1.5 M 2024-03-11T21:16:30.589 INFO:teuthology.orchestra.run.smithi092.stdout:Downloading Packages: 2024-03-11T21:16:30.719 INFO:teuthology.orchestra.run.smithi092.stdout:(1/2): python3-packaging-20.9-5.el9.noarch.rpm 532 kB/s | 69 kB 00:00 2024-03-11T21:16:31.136 INFO:teuthology.orchestra.run.smithi092.stdout:(2/2): ceph-volume-19.0.0-1979.g2b0101eb.el9.no 483 kB/s | 264 kB 00:00 2024-03-11T21:16:31.137 INFO:teuthology.orchestra.run.smithi092.stdout:-------------------------------------------------------------------------------- 2024-03-11T21:16:31.137 INFO:teuthology.orchestra.run.smithi092.stdout:Total 606 kB/s | 332 kB 00:00 2024-03-11T21:16:31.137 INFO:teuthology.orchestra.run.smithi092.stdout:Running transaction check 2024-03-11T21:16:31.146 INFO:teuthology.orchestra.run.smithi092.stdout:Transaction check succeeded. 2024-03-11T21:16:31.146 INFO:teuthology.orchestra.run.smithi092.stdout:Running transaction test 2024-03-11T21:16:31.188 INFO:teuthology.orchestra.run.smithi092.stdout:Transaction test succeeded. 2024-03-11T21:16:31.188 INFO:teuthology.orchestra.run.smithi092.stdout:Running transaction 2024-03-11T21:16:31.341 INFO:teuthology.orchestra.run.smithi092.stdout: Preparing : 1/1 2024-03-11T21:16:31.516 INFO:teuthology.orchestra.run.smithi092.stdout: Installing : python3-packaging-20.9-5.el9.noarch 1/2 2024-03-11T21:16:31.555 INFO:teuthology.orchestra.run.smithi092.stdout: Installing : ceph-volume-2:19.0.0-1979.g2b0101eb.el9.noarch 2/2 2024-03-11T21:16:31.570 INFO:teuthology.orchestra.run.smithi092.stdout: Running scriptlet: ceph-volume-2:19.0.0-1979.g2b0101eb.el9.noarch 2/2 2024-03-11T21:16:31.571 INFO:teuthology.orchestra.run.smithi092.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-11T21:16:31.571 INFO:teuthology.orchestra.run.smithi092.stdout:Invalid unit name "ceph-volume@*.service" escaped as "ceph-volume@\x2a.service". 2024-03-11T21:16:31.571 INFO:teuthology.orchestra.run.smithi092.stdout: 2024-03-11T21:16:32.365 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : ceph-volume-2:19.0.0-1979.g2b0101eb.el9.noarch 1/2 2024-03-11T21:16:32.705 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : python3-packaging-20.9-5.el9.noarch 2/2 2024-03-11T21:16:32.705 INFO:teuthology.orchestra.run.smithi092.stdout: 2024-03-11T21:16:32.705 INFO:teuthology.orchestra.run.smithi092.stdout:Installed: 2024-03-11T21:16:32.705 INFO:teuthology.orchestra.run.smithi092.stdout: ceph-volume-2:19.0.0-1979.g2b0101eb.el9.noarch 2024-03-11T21:16:32.705 INFO:teuthology.orchestra.run.smithi092.stdout: python3-packaging-20.9-5.el9.noarch 2024-03-11T21:16:32.705 INFO:teuthology.orchestra.run.smithi092.stdout: 2024-03-11T21:16:32.705 INFO:teuthology.orchestra.run.smithi092.stdout:Complete! 2024-03-11T21:16:33.063 DEBUG:teuthology.orchestra.run.smithi092:> sudo yum -y install librados-devel 2024-03-11T21:16:33.629 INFO:teuthology.orchestra.run.smithi092.stdout:Last metadata expiration check: 0:01:37 ago on Mon 11 Mar 2024 09:14:56 PM UTC. 2024-03-11T21:16:33.780 INFO:teuthology.orchestra.run.smithi092.stdout:Dependencies resolved. 2024-03-11T21:16:33.781 INFO:teuthology.orchestra.run.smithi092.stdout:================================================================================ 2024-03-11T21:16:33.781 INFO:teuthology.orchestra.run.smithi092.stdout: Package Arch Version Repo Size 2024-03-11T21:16:33.781 INFO:teuthology.orchestra.run.smithi092.stdout:================================================================================ 2024-03-11T21:16:33.781 INFO:teuthology.orchestra.run.smithi092.stdout:Installing: 2024-03-11T21:16:33.781 INFO:teuthology.orchestra.run.smithi092.stdout: librados-devel x86_64 2:19.0.0-1979.g2b0101eb.el9 ceph 127 k 2024-03-11T21:16:33.781 INFO:teuthology.orchestra.run.smithi092.stdout: 2024-03-11T21:16:33.781 INFO:teuthology.orchestra.run.smithi092.stdout:Transaction Summary 2024-03-11T21:16:33.781 INFO:teuthology.orchestra.run.smithi092.stdout:================================================================================ 2024-03-11T21:16:33.781 INFO:teuthology.orchestra.run.smithi092.stdout:Install 1 Package 2024-03-11T21:16:33.782 INFO:teuthology.orchestra.run.smithi092.stdout: 2024-03-11T21:16:33.782 INFO:teuthology.orchestra.run.smithi092.stdout:Total download size: 127 k 2024-03-11T21:16:33.782 INFO:teuthology.orchestra.run.smithi092.stdout:Installed size: 456 k 2024-03-11T21:16:33.782 INFO:teuthology.orchestra.run.smithi092.stdout:Downloading Packages: 2024-03-11T21:16:34.009 INFO:teuthology.orchestra.run.smithi092.stdout:librados-devel-19.0.0-1979.g2b0101eb.el9.x86_64 557 kB/s | 127 kB 00:00 2024-03-11T21:16:34.010 INFO:teuthology.orchestra.run.smithi092.stdout:-------------------------------------------------------------------------------- 2024-03-11T21:16:34.010 INFO:teuthology.orchestra.run.smithi092.stdout:Total 553 kB/s | 127 kB 00:00 2024-03-11T21:16:34.010 INFO:teuthology.orchestra.run.smithi092.stdout:Running transaction check 2024-03-11T21:16:34.016 INFO:teuthology.orchestra.run.smithi092.stdout:Transaction check succeeded. 2024-03-11T21:16:34.016 INFO:teuthology.orchestra.run.smithi092.stdout:Running transaction test 2024-03-11T21:16:34.071 INFO:teuthology.orchestra.run.smithi092.stdout:Transaction test succeeded. 2024-03-11T21:16:34.071 INFO:teuthology.orchestra.run.smithi092.stdout:Running transaction 2024-03-11T21:16:34.216 INFO:teuthology.orchestra.run.smithi092.stdout: Preparing : 1/1 2024-03-11T21:16:34.334 INFO:teuthology.orchestra.run.smithi092.stdout: Installing : librados-devel-2:19.0.0-1979.g2b0101eb.el9.x86_64 1/1 2024-03-11T21:16:34.776 INFO:teuthology.orchestra.run.smithi092.stdout: Running scriptlet: librados-devel-2:19.0.0-1979.g2b0101eb.el9.x86_64 1/1 2024-03-11T21:16:35.062 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : librados-devel-2:19.0.0-1979.g2b0101eb.el9.x86_64 1/1 2024-03-11T21:16:35.062 INFO:teuthology.orchestra.run.smithi092.stdout: 2024-03-11T21:16:35.062 INFO:teuthology.orchestra.run.smithi092.stdout:Installed: 2024-03-11T21:16:35.062 INFO:teuthology.orchestra.run.smithi092.stdout: librados-devel-2:19.0.0-1979.g2b0101eb.el9.x86_64 2024-03-11T21:16:35.062 INFO:teuthology.orchestra.run.smithi092.stdout: 2024-03-11T21:16:35.062 INFO:teuthology.orchestra.run.smithi092.stdout:Complete! 2024-03-11T21:16:35.318 DEBUG:teuthology.orchestra.run.smithi092:> sudo yum -y install libcephfs2 2024-03-11T21:16:35.894 INFO:teuthology.orchestra.run.smithi092.stdout:Last metadata expiration check: 0:01:39 ago on Mon 11 Mar 2024 09:14:56 PM UTC. 2024-03-11T21:16:35.995 INFO:teuthology.orchestra.run.smithi092.stdout:Package libcephfs2-2:19.0.0-1979.g2b0101eb.el9.x86_64 is already installed. 2024-03-11T21:16:36.050 INFO:teuthology.orchestra.run.smithi092.stdout:Dependencies resolved. 2024-03-11T21:16:36.051 INFO:teuthology.orchestra.run.smithi092.stdout:Nothing to do. 2024-03-11T21:16:36.051 INFO:teuthology.orchestra.run.smithi092.stdout:Complete! 2024-03-11T21:16:36.118 DEBUG:teuthology.orchestra.run.smithi092:> sudo yum -y install libcephfs-devel 2024-03-11T21:16:36.679 INFO:teuthology.orchestra.run.smithi092.stdout:Last metadata expiration check: 0:01:40 ago on Mon 11 Mar 2024 09:14:56 PM UTC. 2024-03-11T21:16:36.829 INFO:teuthology.orchestra.run.smithi092.stdout:Dependencies resolved. 2024-03-11T21:16:36.830 INFO:teuthology.orchestra.run.smithi092.stdout:================================================================================ 2024-03-11T21:16:36.830 INFO:teuthology.orchestra.run.smithi092.stdout: Package Arch Version Repo Size 2024-03-11T21:16:36.830 INFO:teuthology.orchestra.run.smithi092.stdout:================================================================================ 2024-03-11T21:16:36.830 INFO:teuthology.orchestra.run.smithi092.stdout:Installing: 2024-03-11T21:16:36.831 INFO:teuthology.orchestra.run.smithi092.stdout: libcephfs-devel x86_64 2:19.0.0-1979.g2b0101eb.el9 ceph 31 k 2024-03-11T21:16:36.831 INFO:teuthology.orchestra.run.smithi092.stdout: 2024-03-11T21:16:36.831 INFO:teuthology.orchestra.run.smithi092.stdout:Transaction Summary 2024-03-11T21:16:36.831 INFO:teuthology.orchestra.run.smithi092.stdout:================================================================================ 2024-03-11T21:16:36.831 INFO:teuthology.orchestra.run.smithi092.stdout:Install 1 Package 2024-03-11T21:16:36.831 INFO:teuthology.orchestra.run.smithi092.stdout: 2024-03-11T21:16:36.831 INFO:teuthology.orchestra.run.smithi092.stdout:Total download size: 31 k 2024-03-11T21:16:36.831 INFO:teuthology.orchestra.run.smithi092.stdout:Installed size: 140 k 2024-03-11T21:16:36.831 INFO:teuthology.orchestra.run.smithi092.stdout:Downloading Packages: 2024-03-11T21:16:36.995 INFO:teuthology.orchestra.run.smithi092.stdout:libcephfs-devel-19.0.0-1979.g2b0101eb.el9.x86_6 190 kB/s | 31 kB 00:00 2024-03-11T21:16:36.996 INFO:teuthology.orchestra.run.smithi092.stdout:-------------------------------------------------------------------------------- 2024-03-11T21:16:36.996 INFO:teuthology.orchestra.run.smithi092.stdout:Total 188 kB/s | 31 kB 00:00 2024-03-11T21:16:36.996 INFO:teuthology.orchestra.run.smithi092.stdout:Running transaction check 2024-03-11T21:16:36.999 INFO:teuthology.orchestra.run.smithi092.stdout:Transaction check succeeded. 2024-03-11T21:16:36.999 INFO:teuthology.orchestra.run.smithi092.stdout:Running transaction test 2024-03-11T21:16:37.030 INFO:teuthology.orchestra.run.smithi092.stdout:Transaction test succeeded. 2024-03-11T21:16:37.031 INFO:teuthology.orchestra.run.smithi092.stdout:Running transaction 2024-03-11T21:16:37.157 INFO:teuthology.orchestra.run.smithi092.stdout: Preparing : 1/1 2024-03-11T21:16:37.252 INFO:teuthology.orchestra.run.smithi092.stdout: Installing : libcephfs-devel-2:19.0.0-1979.g2b0101eb.el9.x86_64 1/1 2024-03-11T21:16:37.684 INFO:teuthology.orchestra.run.smithi092.stdout: Running scriptlet: libcephfs-devel-2:19.0.0-1979.g2b0101eb.el9.x86_64 1/1 2024-03-11T21:16:37.962 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : libcephfs-devel-2:19.0.0-1979.g2b0101eb.el9.x86_64 1/1 2024-03-11T21:16:37.963 INFO:teuthology.orchestra.run.smithi092.stdout: 2024-03-11T21:16:37.963 INFO:teuthology.orchestra.run.smithi092.stdout:Installed: 2024-03-11T21:16:37.963 INFO:teuthology.orchestra.run.smithi092.stdout: libcephfs-devel-2:19.0.0-1979.g2b0101eb.el9.x86_64 2024-03-11T21:16:37.963 INFO:teuthology.orchestra.run.smithi092.stdout: 2024-03-11T21:16:37.963 INFO:teuthology.orchestra.run.smithi092.stdout:Complete! 2024-03-11T21:16:38.120 DEBUG:teuthology.orchestra.run.smithi092:> sudo yum -y install librados2 2024-03-11T21:16:38.675 INFO:teuthology.orchestra.run.smithi092.stdout:Last metadata expiration check: 0:01:42 ago on Mon 11 Mar 2024 09:14:56 PM UTC. 2024-03-11T21:16:38.771 INFO:teuthology.orchestra.run.smithi092.stdout:Package librados2-2:19.0.0-1979.g2b0101eb.el9.x86_64 is already installed. 2024-03-11T21:16:38.822 INFO:teuthology.orchestra.run.smithi092.stdout:Dependencies resolved. 2024-03-11T21:16:38.823 INFO:teuthology.orchestra.run.smithi092.stdout:Nothing to do. 2024-03-11T21:16:38.823 INFO:teuthology.orchestra.run.smithi092.stdout:Complete! 2024-03-11T21:16:38.865 DEBUG:teuthology.orchestra.run.smithi092:> sudo yum -y install librbd1 2024-03-11T21:16:39.426 INFO:teuthology.orchestra.run.smithi092.stdout:Last metadata expiration check: 0:01:43 ago on Mon 11 Mar 2024 09:14:56 PM UTC. 2024-03-11T21:16:39.524 INFO:teuthology.orchestra.run.smithi092.stdout:Package librbd1-2:19.0.0-1979.g2b0101eb.el9.x86_64 is already installed. 2024-03-11T21:16:39.575 INFO:teuthology.orchestra.run.smithi092.stdout:Dependencies resolved. 2024-03-11T21:16:39.576 INFO:teuthology.orchestra.run.smithi092.stdout:Nothing to do. 2024-03-11T21:16:39.577 INFO:teuthology.orchestra.run.smithi092.stdout:Complete! 2024-03-11T21:16:39.615 DEBUG:teuthology.orchestra.run.smithi092:> sudo yum -y install python3-rados 2024-03-11T21:16:40.184 INFO:teuthology.orchestra.run.smithi092.stdout:Last metadata expiration check: 0:01:44 ago on Mon 11 Mar 2024 09:14:56 PM UTC. 2024-03-11T21:16:40.281 INFO:teuthology.orchestra.run.smithi092.stdout:Package python3-rados-2:19.0.0-1979.g2b0101eb.el9.x86_64 is already installed. 2024-03-11T21:16:40.332 INFO:teuthology.orchestra.run.smithi092.stdout:Dependencies resolved. 2024-03-11T21:16:40.333 INFO:teuthology.orchestra.run.smithi092.stdout:Nothing to do. 2024-03-11T21:16:40.333 INFO:teuthology.orchestra.run.smithi092.stdout:Complete! 2024-03-11T21:16:40.378 DEBUG:teuthology.orchestra.run.smithi092:> sudo yum -y install python3-rgw 2024-03-11T21:16:40.943 INFO:teuthology.orchestra.run.smithi092.stdout:Last metadata expiration check: 0:01:44 ago on Mon 11 Mar 2024 09:14:56 PM UTC. 2024-03-11T21:16:41.039 INFO:teuthology.orchestra.run.smithi092.stdout:Package python3-rgw-2:19.0.0-1979.g2b0101eb.el9.x86_64 is already installed. 2024-03-11T21:16:41.090 INFO:teuthology.orchestra.run.smithi092.stdout:Dependencies resolved. 2024-03-11T21:16:41.091 INFO:teuthology.orchestra.run.smithi092.stdout:Nothing to do. 2024-03-11T21:16:41.091 INFO:teuthology.orchestra.run.smithi092.stdout:Complete! 2024-03-11T21:16:41.141 DEBUG:teuthology.orchestra.run.smithi092:> sudo yum -y install python3-cephfs 2024-03-11T21:16:41.698 INFO:teuthology.orchestra.run.smithi092.stdout:Last metadata expiration check: 0:01:45 ago on Mon 11 Mar 2024 09:14:56 PM UTC. 2024-03-11T21:16:41.795 INFO:teuthology.orchestra.run.smithi092.stdout:Package python3-cephfs-2:19.0.0-1979.g2b0101eb.el9.x86_64 is already installed. 2024-03-11T21:16:41.847 INFO:teuthology.orchestra.run.smithi092.stdout:Dependencies resolved. 2024-03-11T21:16:41.848 INFO:teuthology.orchestra.run.smithi092.stdout:Nothing to do. 2024-03-11T21:16:41.848 INFO:teuthology.orchestra.run.smithi092.stdout:Complete! 2024-03-11T21:16:41.895 DEBUG:teuthology.orchestra.run.smithi092:> sudo yum -y install python3-rbd 2024-03-11T21:16:42.461 INFO:teuthology.orchestra.run.smithi092.stdout:Last metadata expiration check: 0:01:46 ago on Mon 11 Mar 2024 09:14:56 PM UTC. 2024-03-11T21:16:42.558 INFO:teuthology.orchestra.run.smithi092.stdout:Package python3-rbd-2:19.0.0-1979.g2b0101eb.el9.x86_64 is already installed. 2024-03-11T21:16:42.608 INFO:teuthology.orchestra.run.smithi092.stdout:Dependencies resolved. 2024-03-11T21:16:42.608 INFO:teuthology.orchestra.run.smithi092.stdout:Nothing to do. 2024-03-11T21:16:42.609 INFO:teuthology.orchestra.run.smithi092.stdout:Complete! 2024-03-11T21:16:42.661 DEBUG:teuthology.orchestra.run.smithi092:> sudo yum -y install rbd-fuse 2024-03-11T21:16:43.227 INFO:teuthology.orchestra.run.smithi092.stdout:Last metadata expiration check: 0:01:47 ago on Mon 11 Mar 2024 09:14:56 PM UTC. 2024-03-11T21:16:43.383 INFO:teuthology.orchestra.run.smithi092.stdout:Dependencies resolved. 2024-03-11T21:16:43.384 INFO:teuthology.orchestra.run.smithi092.stdout:================================================================================ 2024-03-11T21:16:43.384 INFO:teuthology.orchestra.run.smithi092.stdout: Package Architecture Version Repository Size 2024-03-11T21:16:43.384 INFO:teuthology.orchestra.run.smithi092.stdout:================================================================================ 2024-03-11T21:16:43.384 INFO:teuthology.orchestra.run.smithi092.stdout:Installing: 2024-03-11T21:16:43.384 INFO:teuthology.orchestra.run.smithi092.stdout: rbd-fuse x86_64 2:19.0.0-1979.g2b0101eb.el9 ceph 85 k 2024-03-11T21:16:43.384 INFO:teuthology.orchestra.run.smithi092.stdout: 2024-03-11T21:16:43.384 INFO:teuthology.orchestra.run.smithi092.stdout:Transaction Summary 2024-03-11T21:16:43.384 INFO:teuthology.orchestra.run.smithi092.stdout:================================================================================ 2024-03-11T21:16:43.385 INFO:teuthology.orchestra.run.smithi092.stdout:Install 1 Package 2024-03-11T21:16:43.385 INFO:teuthology.orchestra.run.smithi092.stdout: 2024-03-11T21:16:43.385 INFO:teuthology.orchestra.run.smithi092.stdout:Total download size: 85 k 2024-03-11T21:16:43.385 INFO:teuthology.orchestra.run.smithi092.stdout:Installed size: 231 k 2024-03-11T21:16:43.385 INFO:teuthology.orchestra.run.smithi092.stdout:Downloading Packages: 2024-03-11T21:16:43.640 INFO:teuthology.orchestra.run.smithi092.stdout:rbd-fuse-19.0.0-1979.g2b0101eb.el9.x86_64.rpm 335 kB/s | 85 kB 00:00 2024-03-11T21:16:43.641 INFO:teuthology.orchestra.run.smithi092.stdout:-------------------------------------------------------------------------------- 2024-03-11T21:16:43.641 INFO:teuthology.orchestra.run.smithi092.stdout:Total 332 kB/s | 85 kB 00:00 2024-03-11T21:16:43.642 INFO:teuthology.orchestra.run.smithi092.stdout:Running transaction check 2024-03-11T21:16:43.654 INFO:teuthology.orchestra.run.smithi092.stdout:Transaction check succeeded. 2024-03-11T21:16:43.654 INFO:teuthology.orchestra.run.smithi092.stdout:Running transaction test 2024-03-11T21:16:43.719 INFO:teuthology.orchestra.run.smithi092.stdout:Transaction test succeeded. 2024-03-11T21:16:43.720 INFO:teuthology.orchestra.run.smithi092.stdout:Running transaction 2024-03-11T21:16:43.865 INFO:teuthology.orchestra.run.smithi092.stdout: Preparing : 1/1 2024-03-11T21:16:43.972 INFO:teuthology.orchestra.run.smithi092.stdout: Installing : rbd-fuse-2:19.0.0-1979.g2b0101eb.el9.x86_64 1/1 2024-03-11T21:16:44.402 INFO:teuthology.orchestra.run.smithi092.stdout: Running scriptlet: rbd-fuse-2:19.0.0-1979.g2b0101eb.el9.x86_64 1/1 2024-03-11T21:16:44.711 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : rbd-fuse-2:19.0.0-1979.g2b0101eb.el9.x86_64 1/1 2024-03-11T21:16:44.712 INFO:teuthology.orchestra.run.smithi092.stdout: 2024-03-11T21:16:44.712 INFO:teuthology.orchestra.run.smithi092.stdout:Installed: 2024-03-11T21:16:44.712 INFO:teuthology.orchestra.run.smithi092.stdout: rbd-fuse-2:19.0.0-1979.g2b0101eb.el9.x86_64 2024-03-11T21:16:44.712 INFO:teuthology.orchestra.run.smithi092.stdout: 2024-03-11T21:16:44.712 INFO:teuthology.orchestra.run.smithi092.stdout:Complete! 2024-03-11T21:16:44.907 DEBUG:teuthology.orchestra.run.smithi092:> sudo yum -y install rbd-mirror 2024-03-11T21:16:45.484 INFO:teuthology.orchestra.run.smithi092.stdout:Last metadata expiration check: 0:01:49 ago on Mon 11 Mar 2024 09:14:56 PM UTC. 2024-03-11T21:16:45.636 INFO:teuthology.orchestra.run.smithi092.stdout:Dependencies resolved. 2024-03-11T21:16:45.637 INFO:teuthology.orchestra.run.smithi092.stdout:================================================================================ 2024-03-11T21:16:45.637 INFO:teuthology.orchestra.run.smithi092.stdout: Package Arch Version Repo Size 2024-03-11T21:16:45.637 INFO:teuthology.orchestra.run.smithi092.stdout:================================================================================ 2024-03-11T21:16:45.637 INFO:teuthology.orchestra.run.smithi092.stdout:Installing: 2024-03-11T21:16:45.637 INFO:teuthology.orchestra.run.smithi092.stdout: rbd-mirror x86_64 2:19.0.0-1979.g2b0101eb.el9 ceph 3.1 M 2024-03-11T21:16:45.637 INFO:teuthology.orchestra.run.smithi092.stdout: 2024-03-11T21:16:45.637 INFO:teuthology.orchestra.run.smithi092.stdout:Transaction Summary 2024-03-11T21:16:45.637 INFO:teuthology.orchestra.run.smithi092.stdout:================================================================================ 2024-03-11T21:16:45.638 INFO:teuthology.orchestra.run.smithi092.stdout:Install 1 Package 2024-03-11T21:16:45.638 INFO:teuthology.orchestra.run.smithi092.stdout: 2024-03-11T21:16:45.638 INFO:teuthology.orchestra.run.smithi092.stdout:Total download size: 3.1 M 2024-03-11T21:16:45.638 INFO:teuthology.orchestra.run.smithi092.stdout:Installed size: 13 M 2024-03-11T21:16:45.638 INFO:teuthology.orchestra.run.smithi092.stdout:Downloading Packages: 2024-03-11T21:16:48.225 INFO:teuthology.orchestra.run.smithi092.stdout:rbd-mirror-19.0.0-1979.g2b0101eb.el9.x86_64.rpm 1.2 MB/s | 3.1 MB 00:02 2024-03-11T21:16:48.227 INFO:teuthology.orchestra.run.smithi092.stdout:-------------------------------------------------------------------------------- 2024-03-11T21:16:48.227 INFO:teuthology.orchestra.run.smithi092.stdout:Total 1.2 MB/s | 3.1 MB 00:02 2024-03-11T21:16:48.227 INFO:teuthology.orchestra.run.smithi092.stdout:Running transaction check 2024-03-11T21:16:48.237 INFO:teuthology.orchestra.run.smithi092.stdout:Transaction check succeeded. 2024-03-11T21:16:48.237 INFO:teuthology.orchestra.run.smithi092.stdout:Running transaction test 2024-03-11T21:16:48.313 INFO:teuthology.orchestra.run.smithi092.stdout:Transaction test succeeded. 2024-03-11T21:16:48.313 INFO:teuthology.orchestra.run.smithi092.stdout:Running transaction 2024-03-11T21:16:48.740 INFO:teuthology.orchestra.run.smithi092.stdout: Preparing : 1/1 2024-03-11T21:16:48.801 INFO:teuthology.orchestra.run.smithi092.stdout: Installing : rbd-mirror-2:19.0.0-1979.g2b0101eb.el9.x86_64 1/1 2024-03-11T21:16:48.826 INFO:teuthology.orchestra.run.smithi092.stdout: Running scriptlet: rbd-mirror-2:19.0.0-1979.g2b0101eb.el9.x86_64 1/1 2024-03-11T21:16:48.826 INFO:teuthology.orchestra.run.smithi092.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-11T21:16:48.826 INFO:teuthology.orchestra.run.smithi092.stdout:Invalid unit name "ceph-rbd-mirror@*.service" escaped as "ceph-rbd-mirror@\x2a.service". 2024-03-11T21:16:48.826 INFO:teuthology.orchestra.run.smithi092.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-03-11T21:16:48.827 INFO:teuthology.orchestra.run.smithi092.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-03-11T21:16:48.827 INFO:teuthology.orchestra.run.smithi092.stdout: 2024-03-11T21:16:49.963 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : rbd-mirror-2:19.0.0-1979.g2b0101eb.el9.x86_64 1/1 2024-03-11T21:16:49.964 INFO:teuthology.orchestra.run.smithi092.stdout: 2024-03-11T21:16:49.964 INFO:teuthology.orchestra.run.smithi092.stdout:Installed: 2024-03-11T21:16:49.964 INFO:teuthology.orchestra.run.smithi092.stdout: rbd-mirror-2:19.0.0-1979.g2b0101eb.el9.x86_64 2024-03-11T21:16:49.964 INFO:teuthology.orchestra.run.smithi092.stdout: 2024-03-11T21:16:49.964 INFO:teuthology.orchestra.run.smithi092.stdout:Complete! 2024-03-11T21:16:50.171 DEBUG:teuthology.orchestra.run.smithi092:> sudo yum -y install rbd-nbd 2024-03-11T21:16:50.735 INFO:teuthology.orchestra.run.smithi092.stdout:Last metadata expiration check: 0:01:54 ago on Mon 11 Mar 2024 09:14:56 PM UTC. 2024-03-11T21:16:50.883 INFO:teuthology.orchestra.run.smithi092.stdout:Dependencies resolved. 2024-03-11T21:16:50.884 INFO:teuthology.orchestra.run.smithi092.stdout:================================================================================ 2024-03-11T21:16:50.884 INFO:teuthology.orchestra.run.smithi092.stdout: Package Architecture Version Repository Size 2024-03-11T21:16:50.884 INFO:teuthology.orchestra.run.smithi092.stdout:================================================================================ 2024-03-11T21:16:50.884 INFO:teuthology.orchestra.run.smithi092.stdout:Installing: 2024-03-11T21:16:50.884 INFO:teuthology.orchestra.run.smithi092.stdout: rbd-nbd x86_64 2:19.0.0-1979.g2b0101eb.el9 ceph 170 k 2024-03-11T21:16:50.884 INFO:teuthology.orchestra.run.smithi092.stdout: 2024-03-11T21:16:50.884 INFO:teuthology.orchestra.run.smithi092.stdout:Transaction Summary 2024-03-11T21:16:50.885 INFO:teuthology.orchestra.run.smithi092.stdout:================================================================================ 2024-03-11T21:16:50.885 INFO:teuthology.orchestra.run.smithi092.stdout:Install 1 Package 2024-03-11T21:16:50.885 INFO:teuthology.orchestra.run.smithi092.stdout: 2024-03-11T21:16:50.885 INFO:teuthology.orchestra.run.smithi092.stdout:Total download size: 170 k 2024-03-11T21:16:50.885 INFO:teuthology.orchestra.run.smithi092.stdout:Installed size: 490 k 2024-03-11T21:16:50.885 INFO:teuthology.orchestra.run.smithi092.stdout:Downloading Packages: 2024-03-11T21:16:51.461 INFO:teuthology.orchestra.run.smithi092.stdout:rbd-nbd-19.0.0-1979.g2b0101eb.el9.x86_64.rpm 296 kB/s | 170 kB 00:00 2024-03-11T21:16:51.462 INFO:teuthology.orchestra.run.smithi092.stdout:-------------------------------------------------------------------------------- 2024-03-11T21:16:51.462 INFO:teuthology.orchestra.run.smithi092.stdout:Total 295 kB/s | 170 kB 00:00 2024-03-11T21:16:51.462 INFO:teuthology.orchestra.run.smithi092.stdout:Running transaction check 2024-03-11T21:16:51.470 INFO:teuthology.orchestra.run.smithi092.stdout:Transaction check succeeded. 2024-03-11T21:16:51.470 INFO:teuthology.orchestra.run.smithi092.stdout:Running transaction test 2024-03-11T21:16:51.527 INFO:teuthology.orchestra.run.smithi092.stdout:Transaction test succeeded. 2024-03-11T21:16:51.527 INFO:teuthology.orchestra.run.smithi092.stdout:Running transaction 2024-03-11T21:16:51.679 INFO:teuthology.orchestra.run.smithi092.stdout: Preparing : 1/1 2024-03-11T21:16:51.792 INFO:teuthology.orchestra.run.smithi092.stdout: Installing : rbd-nbd-2:19.0.0-1979.g2b0101eb.el9.x86_64 1/1 2024-03-11T21:16:52.245 INFO:teuthology.orchestra.run.smithi092.stdout: Running scriptlet: rbd-nbd-2:19.0.0-1979.g2b0101eb.el9.x86_64 1/1 2024-03-11T21:16:52.543 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : rbd-nbd-2:19.0.0-1979.g2b0101eb.el9.x86_64 1/1 2024-03-11T21:16:52.543 INFO:teuthology.orchestra.run.smithi092.stdout: 2024-03-11T21:16:52.543 INFO:teuthology.orchestra.run.smithi092.stdout:Installed: 2024-03-11T21:16:52.543 INFO:teuthology.orchestra.run.smithi092.stdout: rbd-nbd-2:19.0.0-1979.g2b0101eb.el9.x86_64 2024-03-11T21:16:52.543 INFO:teuthology.orchestra.run.smithi092.stdout: 2024-03-11T21:16:52.543 INFO:teuthology.orchestra.run.smithi092.stdout:Complete! 2024-03-11T21:16:52.765 DEBUG:teuthology.parallel:result is None 2024-03-11T21:17:11.083 INFO:teuthology.orchestra.run.smithi097.stdout:Extra Packages for Enterprise Linux 158 kB/s | 21 MB 02:13 2024-03-11T21:17:16.099 INFO:teuthology.orchestra.run.smithi097.stdout:lab-extras 27 kB/s | 1.7 kB 00:00 2024-03-11T21:17:18.643 INFO:teuthology.orchestra.run.smithi097.stdout:Dependencies resolved. 2024-03-11T21:17:18.654 INFO:teuthology.orchestra.run.smithi097.stdout:================================================================================ 2024-03-11T21:17:18.654 INFO:teuthology.orchestra.run.smithi097.stdout: Package Arch Version Repository Size 2024-03-11T21:17:18.655 INFO:teuthology.orchestra.run.smithi097.stdout:================================================================================ 2024-03-11T21:17:18.655 INFO:teuthology.orchestra.run.smithi097.stdout:Installing: 2024-03-11T21:17:18.655 INFO:teuthology.orchestra.run.smithi097.stdout: ceph-radosgw x86_64 2:19.0.0-1979.g2b0101eb.el9 ceph 9.8 M 2024-03-11T21:17:18.655 INFO:teuthology.orchestra.run.smithi097.stdout:Upgrading: 2024-03-11T21:17:18.655 INFO:teuthology.orchestra.run.smithi097.stdout: librados2 x86_64 2:19.0.0-1979.g2b0101eb.el9 ceph 3.4 M 2024-03-11T21:17:18.655 INFO:teuthology.orchestra.run.smithi097.stdout: librbd1 x86_64 2:19.0.0-1979.g2b0101eb.el9 ceph 3.1 M 2024-03-11T21:17:18.655 INFO:teuthology.orchestra.run.smithi097.stdout:Installing dependencies: 2024-03-11T21:17:18.655 INFO:teuthology.orchestra.run.smithi097.stdout: boost-program-options 2024-03-11T21:17:18.655 INFO:teuthology.orchestra.run.smithi097.stdout: x86_64 1.75.0-8.el9 CentOS-AppStream 106 k 2024-03-11T21:17:18.655 INFO:teuthology.orchestra.run.smithi097.stdout: ceph-base x86_64 2:19.0.0-1979.g2b0101eb.el9 ceph 5.3 M 2024-03-11T21:17:18.655 INFO:teuthology.orchestra.run.smithi097.stdout: ceph-common x86_64 2:19.0.0-1979.g2b0101eb.el9 ceph 20 M 2024-03-11T21:17:18.655 INFO:teuthology.orchestra.run.smithi097.stdout: ceph-selinux x86_64 2:19.0.0-1979.g2b0101eb.el9 ceph 25 k 2024-03-11T21:17:18.656 INFO:teuthology.orchestra.run.smithi097.stdout: gperftools-libs x86_64 2.9.1-2.el9 epel 309 k 2024-03-11T21:17:18.656 INFO:teuthology.orchestra.run.smithi097.stdout: libarrow x86_64 9.0.0-10.el9 epel 4.4 M 2024-03-11T21:17:18.656 INFO:teuthology.orchestra.run.smithi097.stdout: libarrow-doc noarch 9.0.0-10.el9 epel 26 k 2024-03-11T21:17:18.656 INFO:teuthology.orchestra.run.smithi097.stdout: libcephfs2 x86_64 2:19.0.0-1979.g2b0101eb.el9 ceph 745 k 2024-03-11T21:17:18.656 INFO:teuthology.orchestra.run.smithi097.stdout: liboath x86_64 2.6.7-2.el9 epel 49 k 2024-03-11T21:17:18.656 INFO:teuthology.orchestra.run.smithi097.stdout: libpmemobj x86_64 1.12.1-1.el9 CentOS-AppStream 159 k 2024-03-11T21:17:18.656 INFO:teuthology.orchestra.run.smithi097.stdout: librabbitmq x86_64 0.11.0-7.el9 CentOS-AppStream 44 k 2024-03-11T21:17:18.656 INFO:teuthology.orchestra.run.smithi097.stdout: libradosstriper1 x86_64 2:19.0.0-1979.g2b0101eb.el9 ceph 501 k 2024-03-11T21:17:18.656 INFO:teuthology.orchestra.run.smithi097.stdout: librdkafka x86_64 1.6.1-102.el9 CentOS-AppStream 661 k 2024-03-11T21:17:18.656 INFO:teuthology.orchestra.run.smithi097.stdout: librgw2 x86_64 2:19.0.0-1979.g2b0101eb.el9 ceph 5.2 M 2024-03-11T21:17:18.656 INFO:teuthology.orchestra.run.smithi097.stdout: libunwind x86_64 1.6.2-1.el9 epel 67 k 2024-03-11T21:17:18.657 INFO:teuthology.orchestra.run.smithi097.stdout: lttng-ust x86_64 2.12.0-6.el9 CentOS-AppStream 282 k 2024-03-11T21:17:18.657 INFO:teuthology.orchestra.run.smithi097.stdout: parquet-libs x86_64 9.0.0-10.el9 epel 838 k 2024-03-11T21:17:18.657 INFO:teuthology.orchestra.run.smithi097.stdout: python3-ceph-argparse 2024-03-11T21:17:18.657 INFO:teuthology.orchestra.run.smithi097.stdout: x86_64 2:19.0.0-1979.g2b0101eb.el9 ceph 45 k 2024-03-11T21:17:18.657 INFO:teuthology.orchestra.run.smithi097.stdout: python3-ceph-common x86_64 2:19.0.0-1979.g2b0101eb.el9 ceph 131 k 2024-03-11T21:17:18.657 INFO:teuthology.orchestra.run.smithi097.stdout: python3-cephfs x86_64 2:19.0.0-1979.g2b0101eb.el9 ceph 162 k 2024-03-11T21:17:18.657 INFO:teuthology.orchestra.run.smithi097.stdout: python3-prettytable noarch 0.7.2-27.el9 CentOS-AppStream 41 k 2024-03-11T21:17:18.657 INFO:teuthology.orchestra.run.smithi097.stdout: python3-rados x86_64 2:19.0.0-1979.g2b0101eb.el9 ceph 319 k 2024-03-11T21:17:18.657 INFO:teuthology.orchestra.run.smithi097.stdout: python3-rbd x86_64 2:19.0.0-1979.g2b0101eb.el9 ceph 298 k 2024-03-11T21:17:18.657 INFO:teuthology.orchestra.run.smithi097.stdout: python3-rgw x86_64 2:19.0.0-1979.g2b0101eb.el9 ceph 99 k 2024-03-11T21:17:18.657 INFO:teuthology.orchestra.run.smithi097.stdout: qatlib x86_64 23.11.0-1.el9 appstream 216 k 2024-03-11T21:17:18.657 INFO:teuthology.orchestra.run.smithi097.stdout: qatzip-libs x86_64 1.1.2-2.el9 appstream 44 k 2024-03-11T21:17:18.658 INFO:teuthology.orchestra.run.smithi097.stdout: re2 x86_64 1:20211101-3.el9 epel 192 k 2024-03-11T21:17:18.658 INFO:teuthology.orchestra.run.smithi097.stdout: thrift x86_64 0.15.0-2.el9 epel 1.6 M 2024-03-11T21:17:18.658 INFO:teuthology.orchestra.run.smithi097.stdout:Installing weak dependencies: 2024-03-11T21:17:18.658 INFO:teuthology.orchestra.run.smithi097.stdout: qatlib-service x86_64 23.11.0-1.el9 appstream 35 k 2024-03-11T21:17:18.658 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-03-11T21:17:18.658 INFO:teuthology.orchestra.run.smithi097.stdout:Transaction Summary 2024-03-11T21:17:18.658 INFO:teuthology.orchestra.run.smithi097.stdout:================================================================================ 2024-03-11T21:17:18.658 INFO:teuthology.orchestra.run.smithi097.stdout:Install 30 Packages 2024-03-11T21:17:18.658 INFO:teuthology.orchestra.run.smithi097.stdout:Upgrade 2 Packages 2024-03-11T21:17:18.658 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-03-11T21:17:18.659 INFO:teuthology.orchestra.run.smithi097.stdout:Total download size: 58 M 2024-03-11T21:17:18.659 INFO:teuthology.orchestra.run.smithi097.stdout:Downloading Packages: 2024-03-11T21:17:20.058 INFO:teuthology.orchestra.run.smithi097.stdout:(1/32): ceph-base-19.0.0-1979.g2b0101eb.el9.x86 4.7 MB/s | 5.3 MB 00:01 2024-03-11T21:17:20.092 INFO:teuthology.orchestra.run.smithi097.stdout:(2/32): ceph-selinux-19.0.0-1979.g2b0101eb.el9. 759 kB/s | 25 kB 00:00 2024-03-11T21:17:20.209 INFO:teuthology.orchestra.run.smithi097.stdout:(3/32): libcephfs2-19.0.0-1979.g2b0101eb.el9.x8 6.2 MB/s | 745 kB 00:00 2024-03-11T21:17:20.276 INFO:teuthology.orchestra.run.smithi097.stdout:(4/32): libradosstriper1-19.0.0-1979.g2b0101eb. 7.4 MB/s | 501 kB 00:00 2024-03-11T21:17:20.701 INFO:teuthology.orchestra.run.smithi097.stdout:(5/32): librgw2-19.0.0-1979.g2b0101eb.el9.x86_6 12 MB/s | 5.2 MB 00:00 2024-03-11T21:17:20.743 INFO:teuthology.orchestra.run.smithi097.stdout:(6/32): python3-ceph-argparse-19.0.0-1979.g2b01 1.1 MB/s | 45 kB 00:00 2024-03-11T21:17:20.801 INFO:teuthology.orchestra.run.smithi097.stdout:(7/32): python3-ceph-common-19.0.0-1979.g2b0101 2.2 MB/s | 131 kB 00:00 2024-03-11T21:17:20.860 INFO:teuthology.orchestra.run.smithi097.stdout:(8/32): python3-cephfs-19.0.0-1979.g2b0101eb.el 2.7 MB/s | 162 kB 00:00 2024-03-11T21:17:20.919 INFO:teuthology.orchestra.run.smithi097.stdout:(9/32): python3-rados-19.0.0-1979.g2b0101eb.el9 5.4 MB/s | 319 kB 00:00 2024-03-11T21:17:20.985 INFO:teuthology.orchestra.run.smithi097.stdout:(10/32): python3-rbd-19.0.0-1979.g2b0101eb.el9. 4.4 MB/s | 298 kB 00:00 2024-03-11T21:17:21.044 INFO:teuthology.orchestra.run.smithi097.stdout:(11/32): python3-rgw-19.0.0-1979.g2b0101eb.el9. 1.7 MB/s | 99 kB 00:00 2024-03-11T21:17:21.195 INFO:teuthology.orchestra.run.smithi097.stdout:(12/32): ceph-radosgw-19.0.0-1979.g2b0101eb.el9 4.3 MB/s | 9.8 MB 00:02 2024-03-11T21:17:21.336 INFO:teuthology.orchestra.run.smithi097.stdout:(13/32): boost-program-options-1.75.0-8.el9.x86 363 kB/s | 106 kB 00:00 2024-03-11T21:17:21.362 INFO:teuthology.orchestra.run.smithi097.stdout:(14/32): libpmemobj-1.12.1-1.el9.x86_64.rpm 1.1 MB/s | 159 kB 00:00 2024-03-11T21:17:21.387 INFO:teuthology.orchestra.run.smithi097.stdout:(15/32): librabbitmq-0.11.0-7.el9.x86_64.rpm 880 kB/s | 44 kB 00:00 2024-03-11T21:17:21.454 INFO:teuthology.orchestra.run.smithi097.stdout:(16/32): librdkafka-1.6.1-102.el9.x86_64.rpm 7.0 MB/s | 661 kB 00:00 2024-03-11T21:17:21.479 INFO:teuthology.orchestra.run.smithi097.stdout:(17/32): lttng-ust-2.12.0-6.el9.x86_64.rpm 3.0 MB/s | 282 kB 00:00 2024-03-11T21:17:21.505 INFO:teuthology.orchestra.run.smithi097.stdout:(18/32): python3-prettytable-0.7.2-27.el9.noarc 825 kB/s | 41 kB 00:00 2024-03-11T21:17:21.613 INFO:teuthology.orchestra.run.smithi097.stdout:(19/32): qatlib-service-23.11.0-1.el9.x86_64.rp 322 kB/s | 35 kB 00:00 2024-03-11T21:17:21.672 INFO:teuthology.orchestra.run.smithi097.stdout:(20/32): qatzip-libs-1.1.2-2.el9.x86_64.rpm 766 kB/s | 44 kB 00:00 2024-03-11T21:17:21.697 INFO:teuthology.orchestra.run.smithi097.stdout:(21/32): qatlib-23.11.0-1.el9.x86_64.rpm 997 kB/s | 216 kB 00:00 2024-03-11T21:17:22.022 INFO:teuthology.orchestra.run.smithi097.stdout:(22/32): ceph-common-19.0.0-1979.g2b0101eb.el9. 6.5 MB/s | 20 MB 00:03 2024-03-11T21:17:22.081 INFO:teuthology.orchestra.run.smithi097.stdout:(23/32): gperftools-libs-2.9.1-2.el9.x86_64.rpm 755 kB/s | 309 kB 00:00 2024-03-11T21:17:22.123 INFO:teuthology.orchestra.run.smithi097.stdout:(24/32): libarrow-doc-9.0.0-10.el9.noarch.rpm 260 kB/s | 26 kB 00:00 2024-03-11T21:17:22.156 INFO:teuthology.orchestra.run.smithi097.stdout:(25/32): liboath-2.6.7-2.el9.x86_64.rpm 647 kB/s | 49 kB 00:00 2024-03-11T21:17:22.190 INFO:teuthology.orchestra.run.smithi097.stdout:(26/32): libunwind-1.6.2-1.el9.x86_64.rpm 1.0 MB/s | 67 kB 00:00 2024-03-11T21:17:22.340 INFO:teuthology.orchestra.run.smithi097.stdout:(27/32): re2-20211101-3.el9.x86_64.rpm 1.2 MB/s | 192 kB 00:00 2024-03-11T21:17:22.565 INFO:teuthology.orchestra.run.smithi097.stdout:(28/32): parquet-libs-9.0.0-10.el9.x86_64.rpm 2.0 MB/s | 838 kB 00:00 2024-03-11T21:17:22.866 INFO:teuthology.orchestra.run.smithi097.stdout:(29/32): librados2-19.0.0-1979.g2b0101eb.el9.x8 11 MB/s | 3.4 MB 00:00 2024-03-11T21:17:23.049 INFO:teuthology.orchestra.run.smithi097.stdout:(30/32): librbd1-19.0.0-1979.g2b0101eb.el9.x86_ 17 MB/s | 3.1 MB 00:00 2024-03-11T21:17:23.091 INFO:teuthology.orchestra.run.smithi097.stdout:(31/32): thrift-0.15.0-2.el9.x86_64.rpm 2.1 MB/s | 1.6 MB 00:00 2024-03-11T21:17:23.350 INFO:teuthology.orchestra.run.smithi097.stdout:(32/32): libarrow-9.0.0-10.el9.x86_64.rpm 2.7 MB/s | 4.4 MB 00:01 2024-03-11T21:17:23.353 INFO:teuthology.orchestra.run.smithi097.stdout:-------------------------------------------------------------------------------- 2024-03-11T21:17:23.353 INFO:teuthology.orchestra.run.smithi097.stdout:Total 12 MB/s | 58 MB 00:04 2024-03-11T21:17:23.385 INFO:teuthology.orchestra.run.smithi097.stdout:Running transaction check 2024-03-11T21:17:23.420 INFO:teuthology.orchestra.run.smithi097.stdout:Transaction check succeeded. 2024-03-11T21:17:23.421 INFO:teuthology.orchestra.run.smithi097.stdout:Running transaction test 2024-03-11T21:17:23.894 INFO:teuthology.orchestra.run.smithi097.stdout:Transaction test succeeded. 2024-03-11T21:17:23.894 INFO:teuthology.orchestra.run.smithi097.stdout:Running transaction 2024-03-11T21:17:25.353 INFO:teuthology.orchestra.run.smithi097.stdout: Preparing : 1/1 2024-03-11T21:17:25.441 INFO:teuthology.orchestra.run.smithi097.stdout: Installing : lttng-ust-2.12.0-6.el9.x86_64 1/34 2024-03-11T21:17:25.471 INFO:teuthology.orchestra.run.smithi097.stdout: Installing : librdkafka-1.6.1-102.el9.x86_64 2/34 2024-03-11T21:17:25.497 INFO:teuthology.orchestra.run.smithi097.stdout: Installing : librabbitmq-0.11.0-7.el9.x86_64 3/34 2024-03-11T21:17:25.519 INFO:teuthology.orchestra.run.smithi097.stdout: Installing : qatlib-service-23.11.0-1.el9.x86_64 4/34 2024-03-11T21:17:25.535 INFO:teuthology.orchestra.run.smithi097.stdout: Running scriptlet: qatlib-service-23.11.0-1.el9.x86_64 4/34 2024-03-11T21:17:25.720 INFO:teuthology.orchestra.run.smithi097.stdout: Running scriptlet: qatlib-23.11.0-1.el9.x86_64 5/34 2024-03-11T21:17:25.774 INFO:teuthology.orchestra.run.smithi097.stdout: Installing : qatlib-23.11.0-1.el9.x86_64 5/34 2024-03-11T21:17:26.148 INFO:teuthology.orchestra.run.smithi097.stdout: Installing : thrift-0.15.0-2.el9.x86_64 6/34 2024-03-11T21:17:26.181 INFO:teuthology.orchestra.run.smithi097.stdout: Upgrading : librados2-2:19.0.0-1979.g2b0101eb.el9.x86_64 7/34 2024-03-11T21:17:26.256 INFO:teuthology.orchestra.run.smithi097.stdout: Running scriptlet: librados2-2:19.0.0-1979.g2b0101eb.el9.x86_64 7/34 2024-03-11T21:17:26.346 INFO:teuthology.orchestra.run.smithi097.stdout: Installing : python3-rados-2:19.0.0-1979.g2b0101eb.el9.x86_64 8/34 2024-03-11T21:17:26.375 INFO:teuthology.orchestra.run.smithi097.stdout: Installing : libcephfs2-2:19.0.0-1979.g2b0101eb.el9.x86_64 9/34 2024-03-11T21:17:26.423 INFO:teuthology.orchestra.run.smithi097.stdout: Running scriptlet: libcephfs2-2:19.0.0-1979.g2b0101eb.el9.x86_64 9/34 2024-03-11T21:17:26.459 INFO:teuthology.orchestra.run.smithi097.stdout: Installing : liboath-2.6.7-2.el9.x86_64 10/34 2024-03-11T21:17:26.495 INFO:teuthology.orchestra.run.smithi097.stdout: Installing : python3-ceph-argparse-2:19.0.0-1979.g2b0101eb.el9. 11/34 2024-03-11T21:17:26.563 INFO:teuthology.orchestra.run.smithi097.stdout: Installing : python3-cephfs-2:19.0.0-1979.g2b0101eb.el9.x86_64 12/34 2024-03-11T21:17:26.585 INFO:teuthology.orchestra.run.smithi097.stdout: Installing : libradosstriper1-2:19.0.0-1979.g2b0101eb.el9.x86_6 13/34 2024-03-11T21:17:26.647 INFO:teuthology.orchestra.run.smithi097.stdout: Running scriptlet: libradosstriper1-2:19.0.0-1979.g2b0101eb.el9.x86_6 13/34 2024-03-11T21:17:26.675 INFO:teuthology.orchestra.run.smithi097.stdout: Installing : qatzip-libs-1.1.2-2.el9.x86_64 14/34 2024-03-11T21:17:26.795 INFO:teuthology.orchestra.run.smithi097.stdout: Installing : re2-1:20211101-3.el9.x86_64 15/34 2024-03-11T21:17:26.826 INFO:teuthology.orchestra.run.smithi097.stdout: Installing : libarrow-9.0.0-10.el9.x86_64 16/34 2024-03-11T21:17:26.876 INFO:teuthology.orchestra.run.smithi097.stdout: Installing : libunwind-1.6.2-1.el9.x86_64 17/34 2024-03-11T21:17:26.908 INFO:teuthology.orchestra.run.smithi097.stdout: Installing : gperftools-libs-2.9.1-2.el9.x86_64 18/34 2024-03-11T21:17:26.936 INFO:teuthology.orchestra.run.smithi097.stdout: Installing : libarrow-doc-9.0.0-10.el9.noarch 19/34 2024-03-11T21:17:26.969 INFO:teuthology.orchestra.run.smithi097.stdout: Installing : python3-prettytable-0.7.2-27.el9.noarch 20/34 2024-03-11T21:17:27.304 INFO:teuthology.orchestra.run.smithi097.stdout: Installing : libpmemobj-1.12.1-1.el9.x86_64 21/34 2024-03-11T21:17:27.332 INFO:teuthology.orchestra.run.smithi097.stdout: Upgrading : librbd1-2:19.0.0-1979.g2b0101eb.el9.x86_64 22/34 2024-03-11T21:17:27.423 INFO:teuthology.orchestra.run.smithi097.stdout: Running scriptlet: librbd1-2:19.0.0-1979.g2b0101eb.el9.x86_64 22/34 2024-03-11T21:17:27.462 INFO:teuthology.orchestra.run.smithi097.stdout: Installing : python3-rbd-2:19.0.0-1979.g2b0101eb.el9.x86_64 23/34 2024-03-11T21:17:27.504 INFO:teuthology.orchestra.run.smithi097.stdout: Installing : boost-program-options-1.75.0-8.el9.x86_64 24/34 2024-03-11T21:17:27.965 INFO:teuthology.orchestra.run.smithi097.stdout: Installing : parquet-libs-9.0.0-10.el9.x86_64 25/34 2024-03-11T21:17:27.993 INFO:teuthology.orchestra.run.smithi097.stdout: Installing : librgw2-2:19.0.0-1979.g2b0101eb.el9.x86_64 26/34 2024-03-11T21:17:28.064 INFO:teuthology.orchestra.run.smithi097.stdout: Running scriptlet: librgw2-2:19.0.0-1979.g2b0101eb.el9.x86_64 26/34 2024-03-11T21:17:28.132 INFO:teuthology.orchestra.run.smithi097.stdout: Installing : python3-rgw-2:19.0.0-1979.g2b0101eb.el9.x86_64 27/34 2024-03-11T21:17:28.161 INFO:teuthology.orchestra.run.smithi097.stdout: Installing : python3-ceph-common-2:19.0.0-1979.g2b0101eb.el9.x8 28/34 2024-03-11T21:17:30.316 INFO:teuthology.orchestra.run.smithi097.stdout: Running scriptlet: ceph-common-2:19.0.0-1979.g2b0101eb.el9.x86_64 29/34 2024-03-11T21:17:30.358 INFO:teuthology.orchestra.run.smithi097.stdout: Installing : ceph-common-2:19.0.0-1979.g2b0101eb.el9.x86_64 29/34 2024-03-11T21:17:30.912 INFO:teuthology.orchestra.run.smithi097.stdout: Running scriptlet: ceph-common-2:19.0.0-1979.g2b0101eb.el9.x86_64 29/34 2024-03-11T21:17:30.945 INFO:teuthology.orchestra.run.smithi097.stdout: Installing : ceph-base-2:19.0.0-1979.g2b0101eb.el9.x86_64 30/34 2024-03-11T21:17:31.030 INFO:teuthology.orchestra.run.smithi097.stdout: Running scriptlet: ceph-base-2:19.0.0-1979.g2b0101eb.el9.x86_64 30/34 2024-03-11T21:17:31.031 INFO:teuthology.orchestra.run.smithi097.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /usr/lib/systemd/system/ceph.target. 2024-03-11T21:17:31.031 INFO:teuthology.orchestra.run.smithi097.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service → /usr/lib/systemd/system/ceph-crash.service. 2024-03-11T21:17:31.031 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-03-11T21:17:31.063 INFO:teuthology.orchestra.run.smithi097.stdout: Installing : ceph-selinux-2:19.0.0-1979.g2b0101eb.el9.x86_64 31/34 2024-03-11T21:17:40.519 INFO:teuthology.orchestra.run.smithi097.stdout: Running scriptlet: ceph-selinux-2:19.0.0-1979.g2b0101eb.el9.x86_64 31/34 2024-03-11T21:17:40.519 INFO:teuthology.orchestra.run.smithi097.stdout:skipping the directory /sys 2024-03-11T21:17:40.519 INFO:teuthology.orchestra.run.smithi097.stdout:skipping the directory /proc 2024-03-11T21:17:40.519 INFO:teuthology.orchestra.run.smithi097.stdout:skipping the directory /mnt 2024-03-11T21:17:40.519 INFO:teuthology.orchestra.run.smithi097.stdout:skipping the directory /var/tmp 2024-03-11T21:17:40.519 INFO:teuthology.orchestra.run.smithi097.stdout:skipping the directory /home 2024-03-11T21:17:40.520 INFO:teuthology.orchestra.run.smithi097.stdout:skipping the directory /root 2024-03-11T21:17:40.520 INFO:teuthology.orchestra.run.smithi097.stdout:skipping the directory /tmp 2024-03-11T21:17:40.520 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-03-11T21:17:41.441 INFO:teuthology.orchestra.run.smithi097.stdout: Installing : ceph-radosgw-2:19.0.0-1979.g2b0101eb.el9.x86_64 32/34 2024-03-11T21:17:41.466 INFO:teuthology.orchestra.run.smithi097.stdout: Running scriptlet: ceph-radosgw-2:19.0.0-1979.g2b0101eb.el9.x86_64 32/34 2024-03-11T21:17:41.466 INFO:teuthology.orchestra.run.smithi097.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-11T21:17:41.466 INFO:teuthology.orchestra.run.smithi097.stdout:Invalid unit name "ceph-radosgw@*.service" escaped as "ceph-radosgw@\x2a.service". 2024-03-11T21:17:41.466 INFO:teuthology.orchestra.run.smithi097.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-03-11T21:17:41.467 INFO:teuthology.orchestra.run.smithi097.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-03-11T21:17:41.467 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-03-11T21:17:41.468 INFO:teuthology.orchestra.run.smithi097.stdout: Cleanup : librbd1-2:16.2.4-5.el9.x86_64 33/34 2024-03-11T21:17:41.569 INFO:teuthology.orchestra.run.smithi097.stdout: Running scriptlet: librbd1-2:16.2.4-5.el9.x86_64 33/34 2024-03-11T21:17:41.569 INFO:teuthology.orchestra.run.smithi097.stdout: Cleanup : librados2-2:16.2.4-5.el9.x86_64 34/34 2024-03-11T21:17:42.671 INFO:teuthology.orchestra.run.smithi097.stdout: Running scriptlet: librados2-2:16.2.4-5.el9.x86_64 34/34 2024-03-11T21:17:42.671 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : ceph-base-2:19.0.0-1979.g2b0101eb.el9.x86_64 1/34 2024-03-11T21:17:42.672 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : ceph-common-2:19.0.0-1979.g2b0101eb.el9.x86_64 2/34 2024-03-11T21:17:42.672 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : ceph-radosgw-2:19.0.0-1979.g2b0101eb.el9.x86_64 3/34 2024-03-11T21:17:42.672 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : ceph-selinux-2:19.0.0-1979.g2b0101eb.el9.x86_64 4/34 2024-03-11T21:17:42.672 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : libcephfs2-2:19.0.0-1979.g2b0101eb.el9.x86_64 5/34 2024-03-11T21:17:42.672 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : libradosstriper1-2:19.0.0-1979.g2b0101eb.el9.x86_6 6/34 2024-03-11T21:17:42.672 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : librgw2-2:19.0.0-1979.g2b0101eb.el9.x86_64 7/34 2024-03-11T21:17:42.672 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : python3-ceph-argparse-2:19.0.0-1979.g2b0101eb.el9. 8/34 2024-03-11T21:17:42.672 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : python3-ceph-common-2:19.0.0-1979.g2b0101eb.el9.x8 9/34 2024-03-11T21:17:42.672 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : python3-cephfs-2:19.0.0-1979.g2b0101eb.el9.x86_64 10/34 2024-03-11T21:17:42.672 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : python3-rados-2:19.0.0-1979.g2b0101eb.el9.x86_64 11/34 2024-03-11T21:17:42.672 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : python3-rbd-2:19.0.0-1979.g2b0101eb.el9.x86_64 12/34 2024-03-11T21:17:42.672 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : python3-rgw-2:19.0.0-1979.g2b0101eb.el9.x86_64 13/34 2024-03-11T21:17:42.673 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : boost-program-options-1.75.0-8.el9.x86_64 14/34 2024-03-11T21:17:42.673 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : libpmemobj-1.12.1-1.el9.x86_64 15/34 2024-03-11T21:17:42.673 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : librabbitmq-0.11.0-7.el9.x86_64 16/34 2024-03-11T21:17:42.673 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : librdkafka-1.6.1-102.el9.x86_64 17/34 2024-03-11T21:17:42.673 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : lttng-ust-2.12.0-6.el9.x86_64 18/34 2024-03-11T21:17:42.673 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : python3-prettytable-0.7.2-27.el9.noarch 19/34 2024-03-11T21:17:42.673 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : qatlib-23.11.0-1.el9.x86_64 20/34 2024-03-11T21:17:42.673 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : qatlib-service-23.11.0-1.el9.x86_64 21/34 2024-03-11T21:17:42.673 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : qatzip-libs-1.1.2-2.el9.x86_64 22/34 2024-03-11T21:17:42.673 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : gperftools-libs-2.9.1-2.el9.x86_64 23/34 2024-03-11T21:17:42.673 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : libarrow-9.0.0-10.el9.x86_64 24/34 2024-03-11T21:17:42.673 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : libarrow-doc-9.0.0-10.el9.noarch 25/34 2024-03-11T21:17:42.673 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : liboath-2.6.7-2.el9.x86_64 26/34 2024-03-11T21:17:42.674 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : libunwind-1.6.2-1.el9.x86_64 27/34 2024-03-11T21:17:42.674 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : parquet-libs-9.0.0-10.el9.x86_64 28/34 2024-03-11T21:17:42.674 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : re2-1:20211101-3.el9.x86_64 29/34 2024-03-11T21:17:42.674 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : thrift-0.15.0-2.el9.x86_64 30/34 2024-03-11T21:17:42.674 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : librados2-2:19.0.0-1979.g2b0101eb.el9.x86_64 31/34 2024-03-11T21:17:42.674 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : librados2-2:16.2.4-5.el9.x86_64 32/34 2024-03-11T21:17:42.674 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : librbd1-2:19.0.0-1979.g2b0101eb.el9.x86_64 33/34 2024-03-11T21:17:43.712 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : librbd1-2:16.2.4-5.el9.x86_64 34/34 2024-03-11T21:17:43.713 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-03-11T21:17:43.713 INFO:teuthology.orchestra.run.smithi097.stdout:Upgraded: 2024-03-11T21:17:43.713 INFO:teuthology.orchestra.run.smithi097.stdout: librados2-2:19.0.0-1979.g2b0101eb.el9.x86_64 2024-03-11T21:17:43.713 INFO:teuthology.orchestra.run.smithi097.stdout: librbd1-2:19.0.0-1979.g2b0101eb.el9.x86_64 2024-03-11T21:17:43.713 INFO:teuthology.orchestra.run.smithi097.stdout:Installed: 2024-03-11T21:17:43.713 INFO:teuthology.orchestra.run.smithi097.stdout: boost-program-options-1.75.0-8.el9.x86_64 2024-03-11T21:17:43.713 INFO:teuthology.orchestra.run.smithi097.stdout: ceph-base-2:19.0.0-1979.g2b0101eb.el9.x86_64 2024-03-11T21:17:43.713 INFO:teuthology.orchestra.run.smithi097.stdout: ceph-common-2:19.0.0-1979.g2b0101eb.el9.x86_64 2024-03-11T21:17:43.713 INFO:teuthology.orchestra.run.smithi097.stdout: ceph-radosgw-2:19.0.0-1979.g2b0101eb.el9.x86_64 2024-03-11T21:17:43.713 INFO:teuthology.orchestra.run.smithi097.stdout: ceph-selinux-2:19.0.0-1979.g2b0101eb.el9.x86_64 2024-03-11T21:17:43.713 INFO:teuthology.orchestra.run.smithi097.stdout: gperftools-libs-2.9.1-2.el9.x86_64 2024-03-11T21:17:43.714 INFO:teuthology.orchestra.run.smithi097.stdout: libarrow-9.0.0-10.el9.x86_64 2024-03-11T21:17:43.714 INFO:teuthology.orchestra.run.smithi097.stdout: libarrow-doc-9.0.0-10.el9.noarch 2024-03-11T21:17:43.714 INFO:teuthology.orchestra.run.smithi097.stdout: libcephfs2-2:19.0.0-1979.g2b0101eb.el9.x86_64 2024-03-11T21:17:43.714 INFO:teuthology.orchestra.run.smithi097.stdout: liboath-2.6.7-2.el9.x86_64 2024-03-11T21:17:43.714 INFO:teuthology.orchestra.run.smithi097.stdout: libpmemobj-1.12.1-1.el9.x86_64 2024-03-11T21:17:43.714 INFO:teuthology.orchestra.run.smithi097.stdout: librabbitmq-0.11.0-7.el9.x86_64 2024-03-11T21:17:43.714 INFO:teuthology.orchestra.run.smithi097.stdout: libradosstriper1-2:19.0.0-1979.g2b0101eb.el9.x86_64 2024-03-11T21:17:43.714 INFO:teuthology.orchestra.run.smithi097.stdout: librdkafka-1.6.1-102.el9.x86_64 2024-03-11T21:17:43.714 INFO:teuthology.orchestra.run.smithi097.stdout: librgw2-2:19.0.0-1979.g2b0101eb.el9.x86_64 2024-03-11T21:17:43.714 INFO:teuthology.orchestra.run.smithi097.stdout: libunwind-1.6.2-1.el9.x86_64 2024-03-11T21:17:43.714 INFO:teuthology.orchestra.run.smithi097.stdout: lttng-ust-2.12.0-6.el9.x86_64 2024-03-11T21:17:43.714 INFO:teuthology.orchestra.run.smithi097.stdout: parquet-libs-9.0.0-10.el9.x86_64 2024-03-11T21:17:43.714 INFO:teuthology.orchestra.run.smithi097.stdout: python3-ceph-argparse-2:19.0.0-1979.g2b0101eb.el9.x86_64 2024-03-11T21:17:43.715 INFO:teuthology.orchestra.run.smithi097.stdout: python3-ceph-common-2:19.0.0-1979.g2b0101eb.el9.x86_64 2024-03-11T21:17:43.715 INFO:teuthology.orchestra.run.smithi097.stdout: python3-cephfs-2:19.0.0-1979.g2b0101eb.el9.x86_64 2024-03-11T21:17:43.715 INFO:teuthology.orchestra.run.smithi097.stdout: python3-prettytable-0.7.2-27.el9.noarch 2024-03-11T21:17:43.715 INFO:teuthology.orchestra.run.smithi097.stdout: python3-rados-2:19.0.0-1979.g2b0101eb.el9.x86_64 2024-03-11T21:17:43.715 INFO:teuthology.orchestra.run.smithi097.stdout: python3-rbd-2:19.0.0-1979.g2b0101eb.el9.x86_64 2024-03-11T21:17:43.715 INFO:teuthology.orchestra.run.smithi097.stdout: python3-rgw-2:19.0.0-1979.g2b0101eb.el9.x86_64 2024-03-11T21:17:43.715 INFO:teuthology.orchestra.run.smithi097.stdout: qatlib-23.11.0-1.el9.x86_64 2024-03-11T21:17:43.715 INFO:teuthology.orchestra.run.smithi097.stdout: qatlib-service-23.11.0-1.el9.x86_64 2024-03-11T21:17:43.715 INFO:teuthology.orchestra.run.smithi097.stdout: qatzip-libs-1.1.2-2.el9.x86_64 2024-03-11T21:17:43.715 INFO:teuthology.orchestra.run.smithi097.stdout: re2-1:20211101-3.el9.x86_64 2024-03-11T21:17:43.715 INFO:teuthology.orchestra.run.smithi097.stdout: thrift-0.15.0-2.el9.x86_64 2024-03-11T21:17:43.715 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-03-11T21:17:43.715 INFO:teuthology.orchestra.run.smithi097.stdout:Complete! 2024-03-11T21:17:44.083 DEBUG:teuthology.orchestra.run.smithi097:> sudo yum -y install ceph-test 2024-03-11T21:17:44.647 INFO:teuthology.orchestra.run.smithi097.stdout:Last metadata expiration check: 0:00:28 ago on Mon 11 Mar 2024 09:17:16 PM UTC. 2024-03-11T21:17:44.801 INFO:teuthology.orchestra.run.smithi097.stdout:Dependencies resolved. 2024-03-11T21:17:44.801 INFO:teuthology.orchestra.run.smithi097.stdout:================================================================================ 2024-03-11T21:17:44.802 INFO:teuthology.orchestra.run.smithi097.stdout: Package Arch Version Repository Size 2024-03-11T21:17:44.802 INFO:teuthology.orchestra.run.smithi097.stdout:================================================================================ 2024-03-11T21:17:44.802 INFO:teuthology.orchestra.run.smithi097.stdout:Installing: 2024-03-11T21:17:44.802 INFO:teuthology.orchestra.run.smithi097.stdout: ceph-test x86_64 2:19.0.0-1979.g2b0101eb.el9 ceph 48 M 2024-03-11T21:17:44.802 INFO:teuthology.orchestra.run.smithi097.stdout:Installing dependencies: 2024-03-11T21:17:44.802 INFO:teuthology.orchestra.run.smithi097.stdout: jq x86_64 1.6-16.el9 baseos 187 k 2024-03-11T21:17:44.802 INFO:teuthology.orchestra.run.smithi097.stdout: libcephsqlite x86_64 2:19.0.0-1979.g2b0101eb.el9 ceph 163 k 2024-03-11T21:17:44.802 INFO:teuthology.orchestra.run.smithi097.stdout: oniguruma x86_64 6.9.6-1.el9.5 CentOS-AppStream 217 k 2024-03-11T21:17:44.802 INFO:teuthology.orchestra.run.smithi097.stdout: socat x86_64 1.7.4.1-5.el9 CentOS-AppStream 300 k 2024-03-11T21:17:44.802 INFO:teuthology.orchestra.run.smithi097.stdout: xmlstarlet x86_64 1.6.1-20.el9 CentOS-AppStream 63 k 2024-03-11T21:17:44.802 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-03-11T21:17:44.802 INFO:teuthology.orchestra.run.smithi097.stdout:Transaction Summary 2024-03-11T21:17:44.802 INFO:teuthology.orchestra.run.smithi097.stdout:================================================================================ 2024-03-11T21:17:44.803 INFO:teuthology.orchestra.run.smithi097.stdout:Install 6 Packages 2024-03-11T21:17:44.803 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-03-11T21:17:44.803 INFO:teuthology.orchestra.run.smithi097.stdout:Total download size: 48 M 2024-03-11T21:17:44.803 INFO:teuthology.orchestra.run.smithi097.stdout:Installed size: 205 M 2024-03-11T21:17:44.803 INFO:teuthology.orchestra.run.smithi097.stdout:Downloading Packages: 2024-03-11T21:17:45.109 INFO:teuthology.orchestra.run.smithi097.stdout:(1/6): oniguruma-6.9.6-1.el9.5.x86_64.rpm 1.1 MB/s | 217 kB 00:00 2024-03-11T21:17:45.159 INFO:teuthology.orchestra.run.smithi097.stdout:(2/6): libcephsqlite-19.0.0-1979.g2b0101eb.el9. 687 kB/s | 163 kB 00:00 2024-03-11T21:17:45.193 INFO:teuthology.orchestra.run.smithi097.stdout:(3/6): socat-1.7.4.1-5.el9.x86_64.rpm 3.5 MB/s | 300 kB 00:00 2024-03-11T21:17:45.218 INFO:teuthology.orchestra.run.smithi097.stdout:(4/6): xmlstarlet-1.6.1-20.el9.x86_64.rpm 1.0 MB/s | 63 kB 00:00 2024-03-11T21:17:45.310 INFO:teuthology.orchestra.run.smithi097.stdout:(5/6): jq-1.6-16.el9.x86_64.rpm 1.6 MB/s | 187 kB 00:00 2024-03-11T21:17:48.610 INFO:teuthology.orchestra.run.smithi097.stdout:(6/6): ceph-test-19.0.0-1979.g2b0101eb.el9.x86_ 13 MB/s | 48 MB 00:03 2024-03-11T21:17:48.613 INFO:teuthology.orchestra.run.smithi097.stdout:-------------------------------------------------------------------------------- 2024-03-11T21:17:48.613 INFO:teuthology.orchestra.run.smithi097.stdout:Total 13 MB/s | 48 MB 00:03 2024-03-11T21:17:48.627 INFO:teuthology.orchestra.run.smithi097.stdout:Running transaction check 2024-03-11T21:17:48.649 INFO:teuthology.orchestra.run.smithi097.stdout:Transaction check succeeded. 2024-03-11T21:17:48.649 INFO:teuthology.orchestra.run.smithi097.stdout:Running transaction test 2024-03-11T21:17:49.044 INFO:teuthology.orchestra.run.smithi097.stdout:Transaction test succeeded. 2024-03-11T21:17:49.045 INFO:teuthology.orchestra.run.smithi097.stdout:Running transaction 2024-03-11T21:17:49.629 INFO:teuthology.orchestra.run.smithi097.stdout: Preparing : 1/1 2024-03-11T21:17:49.734 INFO:teuthology.orchestra.run.smithi097.stdout: Installing : xmlstarlet-1.6.1-20.el9.x86_64 1/6 2024-03-11T21:17:49.772 INFO:teuthology.orchestra.run.smithi097.stdout: Installing : socat-1.7.4.1-5.el9.x86_64 2/6 2024-03-11T21:17:49.804 INFO:teuthology.orchestra.run.smithi097.stdout: Installing : oniguruma-6.9.6-1.el9.5.x86_64 3/6 2024-03-11T21:17:49.843 INFO:teuthology.orchestra.run.smithi097.stdout: Installing : jq-1.6-16.el9.x86_64 4/6 2024-03-11T21:17:49.872 INFO:teuthology.orchestra.run.smithi097.stdout: Installing : libcephsqlite-2:19.0.0-1979.g2b0101eb.el9.x86_64 5/6 2024-03-11T21:17:54.407 INFO:teuthology.orchestra.run.smithi097.stdout: Running scriptlet: libcephsqlite-2:19.0.0-1979.g2b0101eb.el9.x86_64 5/6 2024-03-11T21:17:54.479 INFO:teuthology.orchestra.run.smithi097.stdout: Installing : ceph-test-2:19.0.0-1979.g2b0101eb.el9.x86_64 6/6 2024-03-11T21:17:56.764 INFO:teuthology.orchestra.run.smithi097.stdout: Running scriptlet: ceph-test-2:19.0.0-1979.g2b0101eb.el9.x86_64 6/6 2024-03-11T21:17:56.764 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : ceph-test-2:19.0.0-1979.g2b0101eb.el9.x86_64 1/6 2024-03-11T21:17:56.764 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : libcephsqlite-2:19.0.0-1979.g2b0101eb.el9.x86_64 2/6 2024-03-11T21:17:56.764 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : oniguruma-6.9.6-1.el9.5.x86_64 3/6 2024-03-11T21:17:56.764 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : socat-1.7.4.1-5.el9.x86_64 4/6 2024-03-11T21:17:56.765 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : xmlstarlet-1.6.1-20.el9.x86_64 5/6 2024-03-11T21:17:57.174 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : jq-1.6-16.el9.x86_64 6/6 2024-03-11T21:17:57.175 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-03-11T21:17:57.175 INFO:teuthology.orchestra.run.smithi097.stdout:Installed: 2024-03-11T21:17:57.175 INFO:teuthology.orchestra.run.smithi097.stdout: ceph-test-2:19.0.0-1979.g2b0101eb.el9.x86_64 2024-03-11T21:17:57.175 INFO:teuthology.orchestra.run.smithi097.stdout: jq-1.6-16.el9.x86_64 2024-03-11T21:17:57.175 INFO:teuthology.orchestra.run.smithi097.stdout: libcephsqlite-2:19.0.0-1979.g2b0101eb.el9.x86_64 2024-03-11T21:17:57.175 INFO:teuthology.orchestra.run.smithi097.stdout: oniguruma-6.9.6-1.el9.5.x86_64 2024-03-11T21:17:57.175 INFO:teuthology.orchestra.run.smithi097.stdout: socat-1.7.4.1-5.el9.x86_64 2024-03-11T21:17:57.175 INFO:teuthology.orchestra.run.smithi097.stdout: xmlstarlet-1.6.1-20.el9.x86_64 2024-03-11T21:17:57.175 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-03-11T21:17:57.175 INFO:teuthology.orchestra.run.smithi097.stdout:Complete! 2024-03-11T21:17:57.481 DEBUG:teuthology.orchestra.run.smithi097:> sudo yum -y install ceph 2024-03-11T21:17:58.066 INFO:teuthology.orchestra.run.smithi097.stdout:Last metadata expiration check: 0:00:42 ago on Mon 11 Mar 2024 09:17:16 PM UTC. 2024-03-11T21:17:58.236 INFO:teuthology.orchestra.run.smithi097.stdout:Dependencies resolved. 2024-03-11T21:17:58.238 INFO:teuthology.orchestra.run.smithi097.stdout:========================================================================================= 2024-03-11T21:17:58.238 INFO:teuthology.orchestra.run.smithi097.stdout: Package Arch Version Repository Size 2024-03-11T21:17:58.238 INFO:teuthology.orchestra.run.smithi097.stdout:========================================================================================= 2024-03-11T21:17:58.239 INFO:teuthology.orchestra.run.smithi097.stdout:Installing: 2024-03-11T21:17:58.239 INFO:teuthology.orchestra.run.smithi097.stdout: ceph x86_64 2:19.0.0-1979.g2b0101eb.el9 ceph 6.6 k 2024-03-11T21:17:58.239 INFO:teuthology.orchestra.run.smithi097.stdout:Installing dependencies: 2024-03-11T21:17:58.239 INFO:teuthology.orchestra.run.smithi097.stdout: ceph-mds x86_64 2:19.0.0-1979.g2b0101eb.el9 ceph 2.1 M 2024-03-11T21:17:58.239 INFO:teuthology.orchestra.run.smithi097.stdout: ceph-mgr x86_64 2:19.0.0-1979.g2b0101eb.el9 ceph 1.5 M 2024-03-11T21:17:58.239 INFO:teuthology.orchestra.run.smithi097.stdout: ceph-mgr-modules-core noarch 2:19.0.0-1979.g2b0101eb.el9 ceph-noarch 248 k 2024-03-11T21:17:58.239 INFO:teuthology.orchestra.run.smithi097.stdout: ceph-mon x86_64 2:19.0.0-1979.g2b0101eb.el9 ceph 4.7 M 2024-03-11T21:17:58.239 INFO:teuthology.orchestra.run.smithi097.stdout: ceph-osd x86_64 2:19.0.0-1979.g2b0101eb.el9 ceph 17 M 2024-03-11T21:17:58.239 INFO:teuthology.orchestra.run.smithi097.stdout: lua x86_64 5.4.4-4.el9 CentOS-AppStream 187 k 2024-03-11T21:17:58.239 INFO:teuthology.orchestra.run.smithi097.stdout: lua-devel x86_64 5.4.4-4.el9 crb 22 k 2024-03-11T21:17:58.239 INFO:teuthology.orchestra.run.smithi097.stdout: luarocks noarch 3.9.2-1.el9 epel 151 k 2024-03-11T21:17:58.239 INFO:teuthology.orchestra.run.smithi097.stdout: python3-bcrypt x86_64 3.2.2-1.el9 epel 43 k 2024-03-11T21:17:58.240 INFO:teuthology.orchestra.run.smithi097.stdout: python3-cffi x86_64 1.14.5-5.el9 CentOS-BaseOS 241 k 2024-03-11T21:17:58.240 INFO:teuthology.orchestra.run.smithi097.stdout: python3-cheroot noarch 8.6.0-4.el9 epel 172 k 2024-03-11T21:17:58.240 INFO:teuthology.orchestra.run.smithi097.stdout: python3-cherrypy noarch 18.6.1-2.el9 epel 358 k 2024-03-11T21:17:58.240 INFO:teuthology.orchestra.run.smithi097.stdout: python3-cryptography x86_64 36.0.1-4.el9 CentOS-BaseOS 1.1 M 2024-03-11T21:17:58.240 INFO:teuthology.orchestra.run.smithi097.stdout: python3-jaraco noarch 8.2.1-3.el9 epel 11 k 2024-03-11T21:17:58.240 INFO:teuthology.orchestra.run.smithi097.stdout: python3-jaraco-classes noarch 3.2.1-5.el9 epel 18 k 2024-03-11T21:17:58.240 INFO:teuthology.orchestra.run.smithi097.stdout: python3-jaraco-collections noarch 3.0.0-8.el9 epel 23 k 2024-03-11T21:17:58.240 INFO:teuthology.orchestra.run.smithi097.stdout: python3-jaraco-functools noarch 3.5.0-2.el9 epel 19 k 2024-03-11T21:17:58.240 INFO:teuthology.orchestra.run.smithi097.stdout: python3-jaraco-text noarch 3.2.0-6.el9 epel 20 k 2024-03-11T21:17:58.240 INFO:teuthology.orchestra.run.smithi097.stdout: python3-logutils noarch 0.3.5-21.el9 epel 46 k 2024-03-11T21:17:58.240 INFO:teuthology.orchestra.run.smithi097.stdout: python3-mako noarch 1.1.4-6.el9 CentOS-AppStream 144 k 2024-03-11T21:17:58.241 INFO:teuthology.orchestra.run.smithi097.stdout: python3-markupsafe x86_64 1.1.1-12.el9 CentOS-AppStream 32 k 2024-03-11T21:17:58.241 INFO:teuthology.orchestra.run.smithi097.stdout: python3-more-itertools noarch 8.12.0-2.el9 epel 79 k 2024-03-11T21:17:58.241 INFO:teuthology.orchestra.run.smithi097.stdout: python3-pecan noarch 1.4.2-3.el9 epel 272 k 2024-03-11T21:17:58.241 INFO:teuthology.orchestra.run.smithi097.stdout: python3-ply noarch 3.11-14.el9 CentOS-BaseOS 103 k 2024-03-11T21:17:58.241 INFO:teuthology.orchestra.run.smithi097.stdout: python3-portend noarch 3.1.0-2.el9 epel 16 k 2024-03-11T21:17:58.241 INFO:teuthology.orchestra.run.smithi097.stdout: python3-pyOpenSSL noarch 21.0.0-1.el9 epel 90 k 2024-03-11T21:17:58.241 INFO:teuthology.orchestra.run.smithi097.stdout: python3-pycparser noarch 2.20-6.el9 CentOS-BaseOS 124 k 2024-03-11T21:17:58.241 INFO:teuthology.orchestra.run.smithi097.stdout: python3-pytz noarch 2021.1-5.el9 CentOS-AppStream 47 k 2024-03-11T21:17:58.241 INFO:teuthology.orchestra.run.smithi097.stdout: python3-requests noarch 2.25.1-8.el9 baseos 125 k 2024-03-11T21:17:58.241 INFO:teuthology.orchestra.run.smithi097.stdout: python3-tempora noarch 5.0.0-2.el9 epel 36 k 2024-03-11T21:17:58.241 INFO:teuthology.orchestra.run.smithi097.stdout: python3-toml noarch 0.10.2-6.el9 CentOS-AppStream 39 k 2024-03-11T21:17:58.241 INFO:teuthology.orchestra.run.smithi097.stdout: python3-urllib3 noarch 1.26.5-5.el9 CentOS-BaseOS 187 k 2024-03-11T21:17:58.241 INFO:teuthology.orchestra.run.smithi097.stdout: python3-webob noarch 1.8.7-6.el9 epel 230 k 2024-03-11T21:17:58.241 INFO:teuthology.orchestra.run.smithi097.stdout: python3-werkzeug noarch 2.0.3-3.el9.1 epel 427 k 2024-03-11T21:17:58.242 INFO:teuthology.orchestra.run.smithi097.stdout: python3-zc-lockfile noarch 2.0-10.el9 epel 20 k 2024-03-11T21:17:58.242 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-03-11T21:17:58.242 INFO:teuthology.orchestra.run.smithi097.stdout:Transaction Summary 2024-03-11T21:17:58.242 INFO:teuthology.orchestra.run.smithi097.stdout:========================================================================================= 2024-03-11T21:17:58.242 INFO:teuthology.orchestra.run.smithi097.stdout:Install 36 Packages 2024-03-11T21:17:58.242 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-03-11T21:17:58.242 INFO:teuthology.orchestra.run.smithi097.stdout:Total download size: 30 M 2024-03-11T21:17:58.242 INFO:teuthology.orchestra.run.smithi097.stdout:Installed size: 107 M 2024-03-11T21:17:58.243 INFO:teuthology.orchestra.run.smithi097.stdout:Downloading Packages: 2024-03-11T21:17:58.613 INFO:teuthology.orchestra.run.smithi097.stdout:(1/36): ceph-19.0.0-1979.g2b0101eb.el9.x86_64.r 39 kB/s | 6.6 kB 00:00 2024-03-11T21:17:58.797 INFO:teuthology.orchestra.run.smithi097.stdout:(2/36): ceph-mds-19.0.0-1979.g2b0101eb.el9.x86_ 6.1 MB/s | 2.1 MB 00:00 2024-03-11T21:17:59.372 INFO:teuthology.orchestra.run.smithi097.stdout:(3/36): ceph-mon-19.0.0-1979.g2b0101eb.el9.x86_ 6.2 MB/s | 4.7 MB 00:00 2024-03-11T21:17:59.464 INFO:teuthology.orchestra.run.smithi097.stdout:(4/36): ceph-mgr-modules-core-19.0.0-1979.g2b01 2.6 MB/s | 248 kB 00:00 2024-03-11T21:17:59.706 INFO:teuthology.orchestra.run.smithi097.stdout:(5/36): ceph-osd-19.0.0-1979.g2b0101eb.el9.x86_ 18 MB/s | 17 MB 00:00 2024-03-11T21:17:59.731 INFO:teuthology.orchestra.run.smithi097.stdout:(6/36): lua-5.4.4-4.el9.x86_64.rpm 700 kB/s | 187 kB 00:00 2024-03-11T21:17:59.756 INFO:teuthology.orchestra.run.smithi097.stdout:(7/36): python3-mako-1.1.4-6.el9.noarch.rpm 2.8 MB/s | 144 kB 00:00 2024-03-11T21:17:59.782 INFO:teuthology.orchestra.run.smithi097.stdout:(8/36): python3-markupsafe-1.1.1-12.el9.x86_64. 630 kB/s | 32 kB 00:00 2024-03-11T21:17:59.807 INFO:teuthology.orchestra.run.smithi097.stdout:(9/36): python3-pytz-2021.1-5.el9.noarch.rpm 945 kB/s | 47 kB 00:00 2024-03-11T21:17:59.832 INFO:teuthology.orchestra.run.smithi097.stdout:(10/36): python3-toml-0.10.2-6.el9.noarch.rpm 772 kB/s | 39 kB 00:00 2024-03-11T21:17:59.857 INFO:teuthology.orchestra.run.smithi097.stdout:(11/36): python3-cffi-1.14.5-5.el9.x86_64.rpm 4.7 MB/s | 241 kB 00:00 2024-03-11T21:17:59.916 INFO:teuthology.orchestra.run.smithi097.stdout:(12/36): python3-cryptography-36.0.1-4.el9.x86_ 14 MB/s | 1.1 MB 00:00 2024-03-11T21:17:59.941 INFO:teuthology.orchestra.run.smithi097.stdout:(13/36): python3-ply-3.11-14.el9.noarch.rpm 1.2 MB/s | 103 kB 00:00 2024-03-11T21:17:59.967 INFO:teuthology.orchestra.run.smithi097.stdout:(14/36): python3-pycparser-2.20-6.el9.noarch.rp 2.4 MB/s | 124 kB 00:00 2024-03-11T21:17:59.992 INFO:teuthology.orchestra.run.smithi097.stdout:(15/36): python3-urllib3-1.26.5-5.el9.noarch.rp 3.7 MB/s | 187 kB 00:00 2024-03-11T21:18:00.176 INFO:teuthology.orchestra.run.smithi097.stdout:(16/36): ceph-mgr-19.0.0-1979.g2b0101eb.el9.x86 883 kB/s | 1.5 MB 00:01 2024-03-11T21:18:00.285 INFO:teuthology.orchestra.run.smithi097.stdout:(17/36): lua-devel-5.4.4-4.el9.x86_64.rpm 76 kB/s | 22 kB 00:00 2024-03-11T21:18:00.310 INFO:teuthology.orchestra.run.smithi097.stdout:(18/36): python3-requests-2.25.1-8.el9.noarch.r 365 kB/s | 125 kB 00:00 2024-03-11T21:18:00.493 INFO:teuthology.orchestra.run.smithi097.stdout:(19/36): luarocks-3.9.2-1.el9.noarch.rpm 478 kB/s | 151 kB 00:00 2024-03-11T21:18:00.519 INFO:teuthology.orchestra.run.smithi097.stdout:(20/36): python3-bcrypt-3.2.2-1.el9.x86_64.rpm 186 kB/s | 43 kB 00:00 2024-03-11T21:18:00.586 INFO:teuthology.orchestra.run.smithi097.stdout:(21/36): python3-jaraco-8.2.1-3.el9.noarch.rpm 160 kB/s | 11 kB 00:00 2024-03-11T21:18:00.611 INFO:teuthology.orchestra.run.smithi097.stdout:(22/36): python3-cheroot-8.6.0-4.el9.noarch.rpm 571 kB/s | 172 kB 00:00 2024-03-11T21:18:00.636 INFO:teuthology.orchestra.run.smithi097.stdout:(23/36): python3-cherrypy-18.6.1-2.el9.noarch.r 2.5 MB/s | 358 kB 00:00 2024-03-11T21:18:00.662 INFO:teuthology.orchestra.run.smithi097.stdout:(24/36): python3-jaraco-classes-3.2.1-5.el9.noa 234 kB/s | 18 kB 00:00 2024-03-11T21:18:00.687 INFO:teuthology.orchestra.run.smithi097.stdout:(25/36): python3-jaraco-collections-3.0.0-8.el9 307 kB/s | 23 kB 00:00 2024-03-11T21:18:00.712 INFO:teuthology.orchestra.run.smithi097.stdout:(26/36): python3-jaraco-functools-3.5.0-2.el9.n 258 kB/s | 19 kB 00:00 2024-03-11T21:18:00.737 INFO:teuthology.orchestra.run.smithi097.stdout:(27/36): python3-jaraco-text-3.2.0-6.el9.noarch 260 kB/s | 20 kB 00:00 2024-03-11T21:18:00.762 INFO:teuthology.orchestra.run.smithi097.stdout:(28/36): python3-logutils-0.3.5-21.el9.noarch.r 612 kB/s | 46 kB 00:00 2024-03-11T21:18:00.788 INFO:teuthology.orchestra.run.smithi097.stdout:(29/36): python3-more-itertools-8.12.0-2.el9.no 1.0 MB/s | 79 kB 00:00 2024-03-11T21:18:00.821 INFO:teuthology.orchestra.run.smithi097.stdout:(30/36): python3-portend-3.1.0-2.el9.noarch.rpm 281 kB/s | 16 kB 00:00 2024-03-11T21:18:00.863 INFO:teuthology.orchestra.run.smithi097.stdout:(31/36): python3-pyOpenSSL-21.0.0-1.el9.noarch. 1.2 MB/s | 90 kB 00:00 2024-03-11T21:18:00.897 INFO:teuthology.orchestra.run.smithi097.stdout:(32/36): python3-tempora-5.0.0-2.el9.noarch.rpm 477 kB/s | 36 kB 00:00 2024-03-11T21:18:00.922 INFO:teuthology.orchestra.run.smithi097.stdout:(33/36): python3-pecan-1.4.2-3.el9.noarch.rpm 1.4 MB/s | 272 kB 00:00 2024-03-11T21:18:00.956 INFO:teuthology.orchestra.run.smithi097.stdout:(34/36): python3-webob-1.8.7-6.el9.noarch.rpm 2.4 MB/s | 230 kB 00:00 2024-03-11T21:18:00.981 INFO:teuthology.orchestra.run.smithi097.stdout:(35/36): python3-zc-lockfile-2.0-10.el9.noarch. 341 kB/s | 20 kB 00:00 2024-03-11T21:18:01.056 INFO:teuthology.orchestra.run.smithi097.stdout:(36/36): python3-werkzeug-2.0.3-3.el9.1.noarch. 2.6 MB/s | 427 kB 00:00 2024-03-11T21:18:01.059 INFO:teuthology.orchestra.run.smithi097.stdout:-------------------------------------------------------------------------------- 2024-03-11T21:18:01.060 INFO:teuthology.orchestra.run.smithi097.stdout:Total 11 MB/s | 30 MB 00:02 2024-03-11T21:18:01.084 INFO:teuthology.orchestra.run.smithi097.stdout:Running transaction check 2024-03-11T21:18:01.127 INFO:teuthology.orchestra.run.smithi097.stdout:Transaction check succeeded. 2024-03-11T21:18:01.128 INFO:teuthology.orchestra.run.smithi097.stdout:Running transaction test 2024-03-11T21:18:01.523 INFO:teuthology.orchestra.run.smithi097.stdout:Transaction test succeeded. 2024-03-11T21:18:01.524 INFO:teuthology.orchestra.run.smithi097.stdout:Running transaction 2024-03-11T21:18:02.863 INFO:teuthology.orchestra.run.smithi097.stdout: Preparing : 1/1 2024-03-11T21:18:02.925 INFO:teuthology.orchestra.run.smithi097.stdout: Installing : python3-more-itertools-8.12.0-2.el9.noarch 1/36 2024-03-11T21:18:02.968 INFO:teuthology.orchestra.run.smithi097.stdout: Installing : python3-jaraco-8.2.1-3.el9.noarch 2/36 2024-03-11T21:18:03.011 INFO:teuthology.orchestra.run.smithi097.stdout: Installing : lua-5.4.4-4.el9.x86_64 3/36 2024-03-11T21:18:03.082 INFO:teuthology.orchestra.run.smithi097.stdout: Installing : lua-devel-5.4.4-4.el9.x86_64 4/36 2024-03-11T21:18:03.127 INFO:teuthology.orchestra.run.smithi097.stdout: Installing : luarocks-3.9.2-1.el9.noarch 5/36 2024-03-11T21:18:03.160 INFO:teuthology.orchestra.run.smithi097.stdout: Installing : python3-jaraco-classes-3.2.1-5.el9.noarch 6/36 2024-03-11T21:18:03.252 INFO:teuthology.orchestra.run.smithi097.stdout: Installing : python3-zc-lockfile-2.0-10.el9.noarch 7/36 2024-03-11T21:18:03.330 INFO:teuthology.orchestra.run.smithi097.stdout: Installing : python3-werkzeug-2.0.3-3.el9.1.noarch 8/36 2024-03-11T21:18:03.371 INFO:teuthology.orchestra.run.smithi097.stdout: Installing : python3-webob-1.8.7-6.el9.noarch 9/36 2024-03-11T21:18:03.443 INFO:teuthology.orchestra.run.smithi097.stdout: Installing : python3-logutils-0.3.5-21.el9.noarch 10/36 2024-03-11T21:18:03.497 INFO:teuthology.orchestra.run.smithi097.stdout: Installing : python3-urllib3-1.26.5-5.el9.noarch 11/36 2024-03-11T21:18:03.535 INFO:teuthology.orchestra.run.smithi097.stdout: Installing : python3-requests-2.25.1-8.el9.noarch 12/36 2024-03-11T21:18:03.584 INFO:teuthology.orchestra.run.smithi097.stdout: Installing : python3-ply-3.11-14.el9.noarch 13/36 2024-03-11T21:18:03.640 INFO:teuthology.orchestra.run.smithi097.stdout: Installing : python3-pycparser-2.20-6.el9.noarch 14/36 2024-03-11T21:18:03.839 INFO:teuthology.orchestra.run.smithi097.stdout: Installing : python3-cffi-1.14.5-5.el9.x86_64 15/36 2024-03-11T21:18:03.890 INFO:teuthology.orchestra.run.smithi097.stdout: Installing : python3-cryptography-36.0.1-4.el9.x86_64 16/36 2024-03-11T21:18:03.920 INFO:teuthology.orchestra.run.smithi097.stdout: Installing : python3-pyOpenSSL-21.0.0-1.el9.noarch 17/36 2024-03-11T21:18:03.954 INFO:teuthology.orchestra.run.smithi097.stdout: Installing : python3-bcrypt-3.2.2-1.el9.x86_64 18/36 2024-03-11T21:18:03.985 INFO:teuthology.orchestra.run.smithi097.stdout: Installing : python3-toml-0.10.2-6.el9.noarch 19/36 2024-03-11T21:18:04.061 INFO:teuthology.orchestra.run.smithi097.stdout: Installing : python3-jaraco-functools-3.5.0-2.el9.noarch 20/36 2024-03-11T21:18:04.095 INFO:teuthology.orchestra.run.smithi097.stdout: Installing : python3-cheroot-8.6.0-4.el9.noarch 21/36 2024-03-11T21:18:04.128 INFO:teuthology.orchestra.run.smithi097.stdout: Installing : python3-jaraco-text-3.2.0-6.el9.noarch 22/36 2024-03-11T21:18:04.168 INFO:teuthology.orchestra.run.smithi097.stdout: Installing : python3-jaraco-collections-3.0.0-8.el9.noarch 23/36 2024-03-11T21:18:04.204 INFO:teuthology.orchestra.run.smithi097.stdout: Installing : python3-pytz-2021.1-5.el9.noarch 24/36 2024-03-11T21:18:04.239 INFO:teuthology.orchestra.run.smithi097.stdout: Installing : python3-tempora-5.0.0-2.el9.noarch 25/36 2024-03-11T21:18:04.328 INFO:teuthology.orchestra.run.smithi097.stdout: Installing : python3-portend-3.1.0-2.el9.noarch 26/36 2024-03-11T21:18:04.369 INFO:teuthology.orchestra.run.smithi097.stdout: Installing : python3-cherrypy-18.6.1-2.el9.noarch 27/36 2024-03-11T21:18:04.442 INFO:teuthology.orchestra.run.smithi097.stdout: Installing : python3-markupsafe-1.1.1-12.el9.x86_64 28/36 2024-03-11T21:18:04.579 INFO:teuthology.orchestra.run.smithi097.stdout: Installing : python3-mako-1.1.4-6.el9.noarch 29/36 2024-03-11T21:18:04.704 INFO:teuthology.orchestra.run.smithi097.stdout: Installing : python3-pecan-1.4.2-3.el9.noarch 30/36 2024-03-11T21:18:04.870 INFO:teuthology.orchestra.run.smithi097.stdout: Installing : ceph-mgr-modules-core-2:19.0.0-1979.g2b0101eb.el9. 31/36 2024-03-11T21:18:04.899 INFO:teuthology.orchestra.run.smithi097.stdout: Installing : ceph-mgr-2:19.0.0-1979.g2b0101eb.el9.x86_64 32/36 2024-03-11T21:18:04.924 INFO:teuthology.orchestra.run.smithi097.stdout: Running scriptlet: ceph-mgr-2:19.0.0-1979.g2b0101eb.el9.x86_64 32/36 2024-03-11T21:18:04.925 INFO:teuthology.orchestra.run.smithi097.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-11T21:18:04.925 INFO:teuthology.orchestra.run.smithi097.stdout:Invalid unit name "ceph-mgr@*.service" escaped as "ceph-mgr@\x2a.service". 2024-03-11T21:18:04.925 INFO:teuthology.orchestra.run.smithi097.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-03-11T21:18:04.925 INFO:teuthology.orchestra.run.smithi097.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-03-11T21:18:04.925 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-03-11T21:18:06.367 INFO:teuthology.orchestra.run.smithi097.stdout: Installing : ceph-osd-2:19.0.0-1979.g2b0101eb.el9.x86_64 33/36 2024-03-11T21:18:06.393 INFO:teuthology.orchestra.run.smithi097.stdout: Running scriptlet: ceph-osd-2:19.0.0-1979.g2b0101eb.el9.x86_64 33/36 2024-03-11T21:18:06.393 INFO:teuthology.orchestra.run.smithi097.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-11T21:18:06.393 INFO:teuthology.orchestra.run.smithi097.stdout:Invalid unit name "ceph-osd@*.service" escaped as "ceph-osd@\x2a.service". 2024-03-11T21:18:06.393 INFO:teuthology.orchestra.run.smithi097.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-03-11T21:18:06.393 INFO:teuthology.orchestra.run.smithi097.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-03-11T21:18:06.393 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-03-11T21:18:06.834 INFO:teuthology.orchestra.run.smithi097.stdout: Installing : ceph-mon-2:19.0.0-1979.g2b0101eb.el9.x86_64 34/36 2024-03-11T21:18:06.857 INFO:teuthology.orchestra.run.smithi097.stdout: Running scriptlet: ceph-mon-2:19.0.0-1979.g2b0101eb.el9.x86_64 34/36 2024-03-11T21:18:06.857 INFO:teuthology.orchestra.run.smithi097.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-11T21:18:06.857 INFO:teuthology.orchestra.run.smithi097.stdout:Invalid unit name "ceph-mon@*.service" escaped as "ceph-mon@\x2a.service". 2024-03-11T21:18:06.857 INFO:teuthology.orchestra.run.smithi097.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-03-11T21:18:06.857 INFO:teuthology.orchestra.run.smithi097.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-03-11T21:18:06.857 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-03-11T21:18:07.060 INFO:teuthology.orchestra.run.smithi097.stdout: Installing : ceph-mds-2:19.0.0-1979.g2b0101eb.el9.x86_64 35/36 2024-03-11T21:18:07.082 INFO:teuthology.orchestra.run.smithi097.stdout: Running scriptlet: ceph-mds-2:19.0.0-1979.g2b0101eb.el9.x86_64 35/36 2024-03-11T21:18:07.083 INFO:teuthology.orchestra.run.smithi097.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-11T21:18:07.083 INFO:teuthology.orchestra.run.smithi097.stdout:Invalid unit name "ceph-mds@*.service" escaped as "ceph-mds@\x2a.service". 2024-03-11T21:18:07.083 INFO:teuthology.orchestra.run.smithi097.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-03-11T21:18:07.083 INFO:teuthology.orchestra.run.smithi097.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-03-11T21:18:07.083 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-03-11T21:18:07.150 INFO:teuthology.orchestra.run.smithi097.stdout: Installing : ceph-2:19.0.0-1979.g2b0101eb.el9.x86_64 36/36 2024-03-11T21:18:08.977 INFO:teuthology.orchestra.run.smithi097.stdout: Running scriptlet: ceph-2:19.0.0-1979.g2b0101eb.el9.x86_64 36/36 2024-03-11T21:18:08.977 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : ceph-2:19.0.0-1979.g2b0101eb.el9.x86_64 1/36 2024-03-11T21:18:08.977 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : ceph-mds-2:19.0.0-1979.g2b0101eb.el9.x86_64 2/36 2024-03-11T21:18:08.977 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : ceph-mgr-2:19.0.0-1979.g2b0101eb.el9.x86_64 3/36 2024-03-11T21:18:08.977 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : ceph-mon-2:19.0.0-1979.g2b0101eb.el9.x86_64 4/36 2024-03-11T21:18:08.977 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : ceph-osd-2:19.0.0-1979.g2b0101eb.el9.x86_64 5/36 2024-03-11T21:18:08.977 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : ceph-mgr-modules-core-2:19.0.0-1979.g2b0101eb.el9. 6/36 2024-03-11T21:18:08.977 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : lua-5.4.4-4.el9.x86_64 7/36 2024-03-11T21:18:08.978 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : python3-mako-1.1.4-6.el9.noarch 8/36 2024-03-11T21:18:08.978 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : python3-markupsafe-1.1.1-12.el9.x86_64 9/36 2024-03-11T21:18:08.978 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : python3-pytz-2021.1-5.el9.noarch 10/36 2024-03-11T21:18:08.978 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : python3-toml-0.10.2-6.el9.noarch 11/36 2024-03-11T21:18:08.978 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : python3-cffi-1.14.5-5.el9.x86_64 12/36 2024-03-11T21:18:08.978 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : python3-cryptography-36.0.1-4.el9.x86_64 13/36 2024-03-11T21:18:08.978 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : python3-ply-3.11-14.el9.noarch 14/36 2024-03-11T21:18:08.978 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : python3-pycparser-2.20-6.el9.noarch 15/36 2024-03-11T21:18:08.978 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : python3-urllib3-1.26.5-5.el9.noarch 16/36 2024-03-11T21:18:08.978 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : python3-requests-2.25.1-8.el9.noarch 17/36 2024-03-11T21:18:08.978 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : lua-devel-5.4.4-4.el9.x86_64 18/36 2024-03-11T21:18:08.978 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : luarocks-3.9.2-1.el9.noarch 19/36 2024-03-11T21:18:08.978 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : python3-bcrypt-3.2.2-1.el9.x86_64 20/36 2024-03-11T21:18:08.979 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : python3-cheroot-8.6.0-4.el9.noarch 21/36 2024-03-11T21:18:08.979 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : python3-cherrypy-18.6.1-2.el9.noarch 22/36 2024-03-11T21:18:08.979 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : python3-jaraco-8.2.1-3.el9.noarch 23/36 2024-03-11T21:18:08.979 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : python3-jaraco-classes-3.2.1-5.el9.noarch 24/36 2024-03-11T21:18:08.979 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : python3-jaraco-collections-3.0.0-8.el9.noarch 25/36 2024-03-11T21:18:08.979 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : python3-jaraco-functools-3.5.0-2.el9.noarch 26/36 2024-03-11T21:18:08.979 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : python3-jaraco-text-3.2.0-6.el9.noarch 27/36 2024-03-11T21:18:08.980 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : python3-logutils-0.3.5-21.el9.noarch 28/36 2024-03-11T21:18:08.980 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : python3-more-itertools-8.12.0-2.el9.noarch 29/36 2024-03-11T21:18:08.980 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : python3-pecan-1.4.2-3.el9.noarch 30/36 2024-03-11T21:18:08.980 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : python3-portend-3.1.0-2.el9.noarch 31/36 2024-03-11T21:18:08.980 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : python3-pyOpenSSL-21.0.0-1.el9.noarch 32/36 2024-03-11T21:18:08.980 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : python3-tempora-5.0.0-2.el9.noarch 33/36 2024-03-11T21:18:08.980 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : python3-webob-1.8.7-6.el9.noarch 34/36 2024-03-11T21:18:08.980 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : python3-werkzeug-2.0.3-3.el9.1.noarch 35/36 2024-03-11T21:18:09.953 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : python3-zc-lockfile-2.0-10.el9.noarch 36/36 2024-03-11T21:18:09.954 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-03-11T21:18:09.954 INFO:teuthology.orchestra.run.smithi097.stdout:Installed: 2024-03-11T21:18:09.954 INFO:teuthology.orchestra.run.smithi097.stdout: ceph-2:19.0.0-1979.g2b0101eb.el9.x86_64 2024-03-11T21:18:09.954 INFO:teuthology.orchestra.run.smithi097.stdout: ceph-mds-2:19.0.0-1979.g2b0101eb.el9.x86_64 2024-03-11T21:18:09.954 INFO:teuthology.orchestra.run.smithi097.stdout: ceph-mgr-2:19.0.0-1979.g2b0101eb.el9.x86_64 2024-03-11T21:18:09.954 INFO:teuthology.orchestra.run.smithi097.stdout: ceph-mgr-modules-core-2:19.0.0-1979.g2b0101eb.el9.noarch 2024-03-11T21:18:09.954 INFO:teuthology.orchestra.run.smithi097.stdout: ceph-mon-2:19.0.0-1979.g2b0101eb.el9.x86_64 2024-03-11T21:18:09.954 INFO:teuthology.orchestra.run.smithi097.stdout: ceph-osd-2:19.0.0-1979.g2b0101eb.el9.x86_64 2024-03-11T21:18:09.954 INFO:teuthology.orchestra.run.smithi097.stdout: lua-5.4.4-4.el9.x86_64 2024-03-11T21:18:09.954 INFO:teuthology.orchestra.run.smithi097.stdout: lua-devel-5.4.4-4.el9.x86_64 2024-03-11T21:18:09.955 INFO:teuthology.orchestra.run.smithi097.stdout: luarocks-3.9.2-1.el9.noarch 2024-03-11T21:18:09.955 INFO:teuthology.orchestra.run.smithi097.stdout: python3-bcrypt-3.2.2-1.el9.x86_64 2024-03-11T21:18:09.955 INFO:teuthology.orchestra.run.smithi097.stdout: python3-cffi-1.14.5-5.el9.x86_64 2024-03-11T21:18:09.955 INFO:teuthology.orchestra.run.smithi097.stdout: python3-cheroot-8.6.0-4.el9.noarch 2024-03-11T21:18:09.955 INFO:teuthology.orchestra.run.smithi097.stdout: python3-cherrypy-18.6.1-2.el9.noarch 2024-03-11T21:18:09.955 INFO:teuthology.orchestra.run.smithi097.stdout: python3-cryptography-36.0.1-4.el9.x86_64 2024-03-11T21:18:09.955 INFO:teuthology.orchestra.run.smithi097.stdout: python3-jaraco-8.2.1-3.el9.noarch 2024-03-11T21:18:09.955 INFO:teuthology.orchestra.run.smithi097.stdout: python3-jaraco-classes-3.2.1-5.el9.noarch 2024-03-11T21:18:09.955 INFO:teuthology.orchestra.run.smithi097.stdout: python3-jaraco-collections-3.0.0-8.el9.noarch 2024-03-11T21:18:09.955 INFO:teuthology.orchestra.run.smithi097.stdout: python3-jaraco-functools-3.5.0-2.el9.noarch 2024-03-11T21:18:09.955 INFO:teuthology.orchestra.run.smithi097.stdout: python3-jaraco-text-3.2.0-6.el9.noarch 2024-03-11T21:18:09.955 INFO:teuthology.orchestra.run.smithi097.stdout: python3-logutils-0.3.5-21.el9.noarch 2024-03-11T21:18:09.955 INFO:teuthology.orchestra.run.smithi097.stdout: python3-mako-1.1.4-6.el9.noarch 2024-03-11T21:18:09.955 INFO:teuthology.orchestra.run.smithi097.stdout: python3-markupsafe-1.1.1-12.el9.x86_64 2024-03-11T21:18:09.956 INFO:teuthology.orchestra.run.smithi097.stdout: python3-more-itertools-8.12.0-2.el9.noarch 2024-03-11T21:18:09.956 INFO:teuthology.orchestra.run.smithi097.stdout: python3-pecan-1.4.2-3.el9.noarch 2024-03-11T21:18:09.956 INFO:teuthology.orchestra.run.smithi097.stdout: python3-ply-3.11-14.el9.noarch 2024-03-11T21:18:09.956 INFO:teuthology.orchestra.run.smithi097.stdout: python3-portend-3.1.0-2.el9.noarch 2024-03-11T21:18:09.956 INFO:teuthology.orchestra.run.smithi097.stdout: python3-pyOpenSSL-21.0.0-1.el9.noarch 2024-03-11T21:18:09.956 INFO:teuthology.orchestra.run.smithi097.stdout: python3-pycparser-2.20-6.el9.noarch 2024-03-11T21:18:09.956 INFO:teuthology.orchestra.run.smithi097.stdout: python3-pytz-2021.1-5.el9.noarch 2024-03-11T21:18:09.956 INFO:teuthology.orchestra.run.smithi097.stdout: python3-requests-2.25.1-8.el9.noarch 2024-03-11T21:18:09.956 INFO:teuthology.orchestra.run.smithi097.stdout: python3-tempora-5.0.0-2.el9.noarch 2024-03-11T21:18:09.956 INFO:teuthology.orchestra.run.smithi097.stdout: python3-toml-0.10.2-6.el9.noarch 2024-03-11T21:18:09.956 INFO:teuthology.orchestra.run.smithi097.stdout: python3-urllib3-1.26.5-5.el9.noarch 2024-03-11T21:18:09.956 INFO:teuthology.orchestra.run.smithi097.stdout: python3-webob-1.8.7-6.el9.noarch 2024-03-11T21:18:09.956 INFO:teuthology.orchestra.run.smithi097.stdout: python3-werkzeug-2.0.3-3.el9.1.noarch 2024-03-11T21:18:09.956 INFO:teuthology.orchestra.run.smithi097.stdout: python3-zc-lockfile-2.0-10.el9.noarch 2024-03-11T21:18:09.957 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-03-11T21:18:09.957 INFO:teuthology.orchestra.run.smithi097.stdout:Complete! 2024-03-11T21:18:10.321 DEBUG:teuthology.orchestra.run.smithi097:> sudo yum -y install ceph-base 2024-03-11T21:18:10.881 INFO:teuthology.orchestra.run.smithi097.stdout:Last metadata expiration check: 0:00:54 ago on Mon 11 Mar 2024 09:17:16 PM UTC. 2024-03-11T21:18:10.976 INFO:teuthology.orchestra.run.smithi097.stdout:Package ceph-base-2:19.0.0-1979.g2b0101eb.el9.x86_64 is already installed. 2024-03-11T21:18:11.027 INFO:teuthology.orchestra.run.smithi097.stdout:Dependencies resolved. 2024-03-11T21:18:11.028 INFO:teuthology.orchestra.run.smithi097.stdout:Nothing to do. 2024-03-11T21:18:11.028 INFO:teuthology.orchestra.run.smithi097.stdout:Complete! 2024-03-11T21:18:11.091 DEBUG:teuthology.orchestra.run.smithi097:> sudo yum -y install cephadm 2024-03-11T21:18:11.646 INFO:teuthology.orchestra.run.smithi097.stdout:Last metadata expiration check: 0:00:55 ago on Mon 11 Mar 2024 09:17:16 PM UTC. 2024-03-11T21:18:11.792 INFO:teuthology.orchestra.run.smithi097.stdout:Dependencies resolved. 2024-03-11T21:18:11.793 INFO:teuthology.orchestra.run.smithi097.stdout:================================================================================ 2024-03-11T21:18:11.793 INFO:teuthology.orchestra.run.smithi097.stdout: Package Arch Version Repository Size 2024-03-11T21:18:11.793 INFO:teuthology.orchestra.run.smithi097.stdout:================================================================================ 2024-03-11T21:18:11.793 INFO:teuthology.orchestra.run.smithi097.stdout:Installing: 2024-03-11T21:18:11.793 INFO:teuthology.orchestra.run.smithi097.stdout: cephadm noarch 2:19.0.0-1979.g2b0101eb.el9 ceph-noarch 748 k 2024-03-11T21:18:11.793 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-03-11T21:18:11.793 INFO:teuthology.orchestra.run.smithi097.stdout:Transaction Summary 2024-03-11T21:18:11.793 INFO:teuthology.orchestra.run.smithi097.stdout:================================================================================ 2024-03-11T21:18:11.793 INFO:teuthology.orchestra.run.smithi097.stdout:Install 1 Package 2024-03-11T21:18:11.793 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-03-11T21:18:11.793 INFO:teuthology.orchestra.run.smithi097.stdout:Total download size: 748 k 2024-03-11T21:18:11.793 INFO:teuthology.orchestra.run.smithi097.stdout:Installed size: 755 k 2024-03-11T21:18:11.794 INFO:teuthology.orchestra.run.smithi097.stdout:Downloading Packages: 2024-03-11T21:18:12.088 INFO:teuthology.orchestra.run.smithi097.stdout:cephadm-19.0.0-1979.g2b0101eb.el9.noarch.rpm 2.5 MB/s | 748 kB 00:00 2024-03-11T21:18:12.088 INFO:teuthology.orchestra.run.smithi097.stdout:-------------------------------------------------------------------------------- 2024-03-11T21:18:12.089 INFO:teuthology.orchestra.run.smithi097.stdout:Total 2.5 MB/s | 748 kB 00:00 2024-03-11T21:18:12.089 INFO:teuthology.orchestra.run.smithi097.stdout:Running transaction check 2024-03-11T21:18:12.094 INFO:teuthology.orchestra.run.smithi097.stdout:Transaction check succeeded. 2024-03-11T21:18:12.094 INFO:teuthology.orchestra.run.smithi097.stdout:Running transaction test 2024-03-11T21:18:12.099 INFO:teuthology.orchestra.run.smithi097.stdout:Transaction test succeeded. 2024-03-11T21:18:12.100 INFO:teuthology.orchestra.run.smithi097.stdout:Running transaction 2024-03-11T21:18:12.175 INFO:teuthology.orchestra.run.smithi097.stdout: Preparing : 1/1 2024-03-11T21:18:12.613 INFO:teuthology.orchestra.run.smithi097.stdout: Running scriptlet: cephadm-2:19.0.0-1979.g2b0101eb.el9.noarch 1/1 2024-03-11T21:18:12.709 INFO:teuthology.orchestra.run.smithi097.stdout: Installing : cephadm-2:19.0.0-1979.g2b0101eb.el9.noarch 1/1 2024-03-11T21:18:13.101 INFO:teuthology.orchestra.run.smithi097.stdout: Running scriptlet: cephadm-2:19.0.0-1979.g2b0101eb.el9.noarch 1/1 2024-03-11T21:18:13.435 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : cephadm-2:19.0.0-1979.g2b0101eb.el9.noarch 1/1 2024-03-11T21:18:13.435 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-03-11T21:18:13.435 INFO:teuthology.orchestra.run.smithi097.stdout:Installed: 2024-03-11T21:18:13.435 INFO:teuthology.orchestra.run.smithi097.stdout: cephadm-2:19.0.0-1979.g2b0101eb.el9.noarch 2024-03-11T21:18:13.435 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-03-11T21:18:13.435 INFO:teuthology.orchestra.run.smithi097.stdout:Complete! 2024-03-11T21:18:13.663 DEBUG:teuthology.orchestra.run.smithi097:> sudo yum -y install ceph-immutable-object-cache 2024-03-11T21:18:14.233 INFO:teuthology.orchestra.run.smithi097.stdout:Last metadata expiration check: 0:00:58 ago on Mon 11 Mar 2024 09:17:16 PM UTC. 2024-03-11T21:18:14.380 INFO:teuthology.orchestra.run.smithi097.stdout:Dependencies resolved. 2024-03-11T21:18:14.380 INFO:teuthology.orchestra.run.smithi097.stdout:================================================================================ 2024-03-11T21:18:14.380 INFO:teuthology.orchestra.run.smithi097.stdout: Package Arch Version Repo Size 2024-03-11T21:18:14.380 INFO:teuthology.orchestra.run.smithi097.stdout:================================================================================ 2024-03-11T21:18:14.381 INFO:teuthology.orchestra.run.smithi097.stdout:Installing: 2024-03-11T21:18:14.381 INFO:teuthology.orchestra.run.smithi097.stdout: ceph-immutable-object-cache x86_64 2:19.0.0-1979.g2b0101eb.el9 ceph 145 k 2024-03-11T21:18:14.381 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-03-11T21:18:14.381 INFO:teuthology.orchestra.run.smithi097.stdout:Transaction Summary 2024-03-11T21:18:14.381 INFO:teuthology.orchestra.run.smithi097.stdout:================================================================================ 2024-03-11T21:18:14.381 INFO:teuthology.orchestra.run.smithi097.stdout:Install 1 Package 2024-03-11T21:18:14.381 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-03-11T21:18:14.381 INFO:teuthology.orchestra.run.smithi097.stdout:Total download size: 145 k 2024-03-11T21:18:14.381 INFO:teuthology.orchestra.run.smithi097.stdout:Installed size: 436 k 2024-03-11T21:18:14.381 INFO:teuthology.orchestra.run.smithi097.stdout:Downloading Packages: 2024-03-11T21:18:14.612 INFO:teuthology.orchestra.run.smithi097.stdout:ceph-immutable-object-cache-19.0.0-1979.g2b0101 630 kB/s | 145 kB 00:00 2024-03-11T21:18:14.612 INFO:teuthology.orchestra.run.smithi097.stdout:-------------------------------------------------------------------------------- 2024-03-11T21:18:14.612 INFO:teuthology.orchestra.run.smithi097.stdout:Total 627 kB/s | 145 kB 00:00 2024-03-11T21:18:14.612 INFO:teuthology.orchestra.run.smithi097.stdout:Running transaction check 2024-03-11T21:18:14.620 INFO:teuthology.orchestra.run.smithi097.stdout:Transaction check succeeded. 2024-03-11T21:18:14.620 INFO:teuthology.orchestra.run.smithi097.stdout:Running transaction test 2024-03-11T21:18:14.671 INFO:teuthology.orchestra.run.smithi097.stdout:Transaction test succeeded. 2024-03-11T21:18:14.672 INFO:teuthology.orchestra.run.smithi097.stdout:Running transaction 2024-03-11T21:18:14.813 INFO:teuthology.orchestra.run.smithi097.stdout: Preparing : 1/1 2024-03-11T21:18:14.879 INFO:teuthology.orchestra.run.smithi097.stdout: Installing : ceph-immutable-object-cache-2:19.0.0-1979.g2b0101eb. 1/1 2024-03-11T21:18:14.902 INFO:teuthology.orchestra.run.smithi097.stdout: Running scriptlet: ceph-immutable-object-cache-2:19.0.0-1979.g2b0101eb. 1/1 2024-03-11T21:18:14.903 INFO:teuthology.orchestra.run.smithi097.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-11T21:18:14.903 INFO:teuthology.orchestra.run.smithi097.stdout:Invalid unit name "ceph-immutable-object-cache@*.service" escaped as "ceph-immutable-object-cache@\x2a.service". 2024-03-11T21:18:14.903 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-03-11T21:18:15.878 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : ceph-immutable-object-cache-2:19.0.0-1979.g2b0101eb. 1/1 2024-03-11T21:18:15.879 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-03-11T21:18:15.879 INFO:teuthology.orchestra.run.smithi097.stdout:Installed: 2024-03-11T21:18:15.879 INFO:teuthology.orchestra.run.smithi097.stdout: ceph-immutable-object-cache-2:19.0.0-1979.g2b0101eb.el9.x86_64 2024-03-11T21:18:15.879 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-03-11T21:18:15.879 INFO:teuthology.orchestra.run.smithi097.stdout:Complete! 2024-03-11T21:18:16.126 DEBUG:teuthology.orchestra.run.smithi097:> sudo yum -y install ceph-mgr 2024-03-11T21:18:16.694 INFO:teuthology.orchestra.run.smithi097.stdout:Last metadata expiration check: 0:01:00 ago on Mon 11 Mar 2024 09:17:16 PM UTC. 2024-03-11T21:18:16.791 INFO:teuthology.orchestra.run.smithi097.stdout:Package ceph-mgr-2:19.0.0-1979.g2b0101eb.el9.x86_64 is already installed. 2024-03-11T21:18:16.843 INFO:teuthology.orchestra.run.smithi097.stdout:Dependencies resolved. 2024-03-11T21:18:16.844 INFO:teuthology.orchestra.run.smithi097.stdout:Nothing to do. 2024-03-11T21:18:16.844 INFO:teuthology.orchestra.run.smithi097.stdout:Complete! 2024-03-11T21:18:16.919 DEBUG:teuthology.orchestra.run.smithi097:> sudo yum -y install ceph-mgr-dashboard 2024-03-11T21:18:17.474 INFO:teuthology.orchestra.run.smithi097.stdout:Last metadata expiration check: 0:01:01 ago on Mon 11 Mar 2024 09:17:16 PM UTC. 2024-03-11T21:18:17.623 INFO:teuthology.orchestra.run.smithi097.stdout:Dependencies resolved. 2024-03-11T21:18:17.624 INFO:teuthology.orchestra.run.smithi097.stdout:====================================================================================== 2024-03-11T21:18:17.624 INFO:teuthology.orchestra.run.smithi097.stdout: Package Arch Version Repository Size 2024-03-11T21:18:17.624 INFO:teuthology.orchestra.run.smithi097.stdout:====================================================================================== 2024-03-11T21:18:17.625 INFO:teuthology.orchestra.run.smithi097.stdout:Installing: 2024-03-11T21:18:17.625 INFO:teuthology.orchestra.run.smithi097.stdout: ceph-mgr-dashboard noarch 2:19.0.0-1979.g2b0101eb.el9 ceph-noarch 1.8 M 2024-03-11T21:18:17.625 INFO:teuthology.orchestra.run.smithi097.stdout:Installing dependencies: 2024-03-11T21:18:17.625 INFO:teuthology.orchestra.run.smithi097.stdout: abseil-cpp x86_64 20211102.0-3.el9 epel 548 k 2024-03-11T21:18:17.625 INFO:teuthology.orchestra.run.smithi097.stdout: ceph-grafana-dashboards noarch 2:19.0.0-1979.g2b0101eb.el9 ceph-noarch 34 k 2024-03-11T21:18:17.625 INFO:teuthology.orchestra.run.smithi097.stdout: ceph-prometheus-alerts noarch 2:19.0.0-1979.g2b0101eb.el9 ceph-noarch 16 k 2024-03-11T21:18:17.625 INFO:teuthology.orchestra.run.smithi097.stdout: grpc-data noarch 1.46.7-10.el9 epel 19 k 2024-03-11T21:18:17.625 INFO:teuthology.orchestra.run.smithi097.stdout: protobuf x86_64 3.14.0-13.el9 CentOS-AppStream 1.0 M 2024-03-11T21:18:17.625 INFO:teuthology.orchestra.run.smithi097.stdout: protobuf-compiler x86_64 3.14.0-13.el9 crb 863 k 2024-03-11T21:18:17.625 INFO:teuthology.orchestra.run.smithi097.stdout: python3-grpcio x86_64 1.46.7-10.el9 epel 2.0 M 2024-03-11T21:18:17.625 INFO:teuthology.orchestra.run.smithi097.stdout: python3-grpcio-tools x86_64 1.46.7-10.el9 epel 144 k 2024-03-11T21:18:17.625 INFO:teuthology.orchestra.run.smithi097.stdout: python3-protobuf noarch 3.14.0-13.el9 CentOS-AppStream 238 k 2024-03-11T21:18:17.625 INFO:teuthology.orchestra.run.smithi097.stdout: python3-repoze-lru noarch 0.7-16.el9 epel 31 k 2024-03-11T21:18:17.625 INFO:teuthology.orchestra.run.smithi097.stdout: python3-routes noarch 2.5.1-5.el9 epel 188 k 2024-03-11T21:18:17.626 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-03-11T21:18:17.626 INFO:teuthology.orchestra.run.smithi097.stdout:Transaction Summary 2024-03-11T21:18:17.626 INFO:teuthology.orchestra.run.smithi097.stdout:====================================================================================== 2024-03-11T21:18:17.626 INFO:teuthology.orchestra.run.smithi097.stdout:Install 12 Packages 2024-03-11T21:18:17.626 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-03-11T21:18:17.626 INFO:teuthology.orchestra.run.smithi097.stdout:Total download size: 6.9 M 2024-03-11T21:18:17.626 INFO:teuthology.orchestra.run.smithi097.stdout:Installed size: 26 M 2024-03-11T21:18:17.627 INFO:teuthology.orchestra.run.smithi097.stdout:Downloading Packages: 2024-03-11T21:18:18.024 INFO:teuthology.orchestra.run.smithi097.stdout:(1/12): ceph-grafana-dashboards-19.0.0-1979.g2b 179 kB/s | 34 kB 00:00 2024-03-11T21:18:18.042 INFO:teuthology.orchestra.run.smithi097.stdout:(2/12): ceph-prometheus-alerts-19.0.0-1979.g2b0 79 kB/s | 16 kB 00:00 2024-03-11T21:18:18.208 INFO:teuthology.orchestra.run.smithi097.stdout:(3/12): python3-protobuf-3.14.0-13.el9.noarch.r 1.4 MB/s | 238 kB 00:00 2024-03-11T21:18:18.234 INFO:teuthology.orchestra.run.smithi097.stdout:(4/12): protobuf-3.14.0-13.el9.x86_64.rpm 4.8 MB/s | 1.0 MB 00:00 2024-03-11T21:18:18.550 INFO:teuthology.orchestra.run.smithi097.stdout:(5/12): abseil-cpp-20211102.0-3.el9.x86_64.rpm 1.7 MB/s | 548 kB 00:00 2024-03-11T21:18:18.593 INFO:teuthology.orchestra.run.smithi097.stdout:(6/12): grpc-data-1.46.7-10.el9.noarch.rpm 465 kB/s | 19 kB 00:00 2024-03-11T21:18:18.718 INFO:teuthology.orchestra.run.smithi097.stdout:(7/12): python3-grpcio-1.46.7-10.el9.x86_64.rpm 16 MB/s | 2.0 MB 00:00 2024-03-11T21:18:18.768 INFO:teuthology.orchestra.run.smithi097.stdout:(8/12): python3-grpcio-tools-1.46.7-10.el9.x86_ 2.8 MB/s | 144 kB 00:00 2024-03-11T21:18:18.810 INFO:teuthology.orchestra.run.smithi097.stdout:(9/12): python3-repoze-lru-0.7-16.el9.noarch.rp 735 kB/s | 31 kB 00:00 2024-03-11T21:18:18.861 INFO:teuthology.orchestra.run.smithi097.stdout:(10/12): python3-routes-2.5.1-5.el9.noarch.rpm 3.7 MB/s | 188 kB 00:00 2024-03-11T21:18:18.986 INFO:teuthology.orchestra.run.smithi097.stdout:(11/12): ceph-mgr-dashboard-19.0.0-1979.g2b0101 1.6 MB/s | 1.8 MB 00:01 2024-03-11T21:18:19.419 INFO:teuthology.orchestra.run.smithi097.stdout:(12/12): protobuf-compiler-3.14.0-13.el9.x86_64 713 kB/s | 863 kB 00:01 2024-03-11T21:18:19.423 INFO:teuthology.orchestra.run.smithi097.stdout:-------------------------------------------------------------------------------- 2024-03-11T21:18:19.423 INFO:teuthology.orchestra.run.smithi097.stdout:Total 3.8 MB/s | 6.9 MB 00:01 2024-03-11T21:18:19.441 INFO:teuthology.orchestra.run.smithi097.stdout:Running transaction check 2024-03-11T21:18:19.458 INFO:teuthology.orchestra.run.smithi097.stdout:Transaction check succeeded. 2024-03-11T21:18:19.458 INFO:teuthology.orchestra.run.smithi097.stdout:Running transaction test 2024-03-11T21:18:19.634 INFO:teuthology.orchestra.run.smithi097.stdout:Transaction test succeeded. 2024-03-11T21:18:19.634 INFO:teuthology.orchestra.run.smithi097.stdout:Running transaction 2024-03-11T21:18:20.141 INFO:teuthology.orchestra.run.smithi097.stdout: Preparing : 1/1 2024-03-11T21:18:20.203 INFO:teuthology.orchestra.run.smithi097.stdout: Installing : protobuf-3.14.0-13.el9.x86_64 1/12 2024-03-11T21:18:20.255 INFO:teuthology.orchestra.run.smithi097.stdout: Installing : protobuf-compiler-3.14.0-13.el9.x86_64 2/12 2024-03-11T21:18:20.301 INFO:teuthology.orchestra.run.smithi097.stdout: Installing : python3-repoze-lru-0.7-16.el9.noarch 3/12 2024-03-11T21:18:20.332 INFO:teuthology.orchestra.run.smithi097.stdout: Installing : python3-routes-2.5.1-5.el9.noarch 4/12 2024-03-11T21:18:20.429 INFO:teuthology.orchestra.run.smithi097.stdout: Installing : grpc-data-1.46.7-10.el9.noarch 5/12 2024-03-11T21:18:20.556 INFO:teuthology.orchestra.run.smithi097.stdout: Installing : abseil-cpp-20211102.0-3.el9.x86_64 6/12 2024-03-11T21:18:20.650 INFO:teuthology.orchestra.run.smithi097.stdout: Installing : python3-grpcio-1.46.7-10.el9.x86_64 7/12 2024-03-11T21:18:20.695 INFO:teuthology.orchestra.run.smithi097.stdout: Installing : python3-protobuf-3.14.0-13.el9.noarch 8/12 2024-03-11T21:18:20.732 INFO:teuthology.orchestra.run.smithi097.stdout: Installing : python3-grpcio-tools-1.46.7-10.el9.x86_64 9/12 2024-03-11T21:18:20.764 INFO:teuthology.orchestra.run.smithi097.stdout: Installing : ceph-prometheus-alerts-2:19.0.0-1979.g2b0101eb.el9 10/12 2024-03-11T21:18:21.008 INFO:teuthology.orchestra.run.smithi097.stdout: Installing : ceph-grafana-dashboards-2:19.0.0-1979.g2b0101eb.el 11/12 2024-03-11T21:18:21.039 INFO:teuthology.orchestra.run.smithi097.stdout: Installing : ceph-mgr-dashboard-2:19.0.0-1979.g2b0101eb.el9.noa 12/12 2024-03-11T21:18:21.863 INFO:teuthology.orchestra.run.smithi097.stdout: Running scriptlet: ceph-mgr-dashboard-2:19.0.0-1979.g2b0101eb.el9.noa 12/12 2024-03-11T21:18:21.863 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : ceph-grafana-dashboards-2:19.0.0-1979.g2b0101eb.el 1/12 2024-03-11T21:18:21.863 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : ceph-mgr-dashboard-2:19.0.0-1979.g2b0101eb.el9.noa 2/12 2024-03-11T21:18:21.863 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : ceph-prometheus-alerts-2:19.0.0-1979.g2b0101eb.el9 3/12 2024-03-11T21:18:21.863 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : protobuf-3.14.0-13.el9.x86_64 4/12 2024-03-11T21:18:21.863 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : python3-protobuf-3.14.0-13.el9.noarch 5/12 2024-03-11T21:18:21.864 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : protobuf-compiler-3.14.0-13.el9.x86_64 6/12 2024-03-11T21:18:21.864 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : abseil-cpp-20211102.0-3.el9.x86_64 7/12 2024-03-11T21:18:21.864 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : grpc-data-1.46.7-10.el9.noarch 8/12 2024-03-11T21:18:21.864 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : python3-grpcio-1.46.7-10.el9.x86_64 9/12 2024-03-11T21:18:21.864 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : python3-grpcio-tools-1.46.7-10.el9.x86_64 10/12 2024-03-11T21:18:21.864 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : python3-repoze-lru-0.7-16.el9.noarch 11/12 2024-03-11T21:18:22.224 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : python3-routes-2.5.1-5.el9.noarch 12/12 2024-03-11T21:18:22.224 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-03-11T21:18:22.224 INFO:teuthology.orchestra.run.smithi097.stdout:Installed: 2024-03-11T21:18:22.224 INFO:teuthology.orchestra.run.smithi097.stdout: abseil-cpp-20211102.0-3.el9.x86_64 2024-03-11T21:18:22.224 INFO:teuthology.orchestra.run.smithi097.stdout: ceph-grafana-dashboards-2:19.0.0-1979.g2b0101eb.el9.noarch 2024-03-11T21:18:22.224 INFO:teuthology.orchestra.run.smithi097.stdout: ceph-mgr-dashboard-2:19.0.0-1979.g2b0101eb.el9.noarch 2024-03-11T21:18:22.224 INFO:teuthology.orchestra.run.smithi097.stdout: ceph-prometheus-alerts-2:19.0.0-1979.g2b0101eb.el9.noarch 2024-03-11T21:18:22.224 INFO:teuthology.orchestra.run.smithi097.stdout: grpc-data-1.46.7-10.el9.noarch 2024-03-11T21:18:22.224 INFO:teuthology.orchestra.run.smithi097.stdout: protobuf-3.14.0-13.el9.x86_64 2024-03-11T21:18:22.224 INFO:teuthology.orchestra.run.smithi097.stdout: protobuf-compiler-3.14.0-13.el9.x86_64 2024-03-11T21:18:22.224 INFO:teuthology.orchestra.run.smithi097.stdout: python3-grpcio-1.46.7-10.el9.x86_64 2024-03-11T21:18:22.225 INFO:teuthology.orchestra.run.smithi097.stdout: python3-grpcio-tools-1.46.7-10.el9.x86_64 2024-03-11T21:18:22.225 INFO:teuthology.orchestra.run.smithi097.stdout: python3-protobuf-3.14.0-13.el9.noarch 2024-03-11T21:18:22.225 INFO:teuthology.orchestra.run.smithi097.stdout: python3-repoze-lru-0.7-16.el9.noarch 2024-03-11T21:18:22.225 INFO:teuthology.orchestra.run.smithi097.stdout: python3-routes-2.5.1-5.el9.noarch 2024-03-11T21:18:22.225 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-03-11T21:18:22.225 INFO:teuthology.orchestra.run.smithi097.stdout:Complete! 2024-03-11T21:18:22.512 DEBUG:teuthology.orchestra.run.smithi097:> sudo yum -y install ceph-mgr-diskprediction-local 2024-03-11T21:18:23.067 INFO:teuthology.orchestra.run.smithi097.stdout:Last metadata expiration check: 0:01:07 ago on Mon 11 Mar 2024 09:17:16 PM UTC. 2024-03-11T21:18:23.217 INFO:teuthology.orchestra.run.smithi097.stdout:Dependencies resolved. 2024-03-11T21:18:23.218 INFO:teuthology.orchestra.run.smithi097.stdout:============================================================================================ 2024-03-11T21:18:23.218 INFO:teuthology.orchestra.run.smithi097.stdout: Package Arch Version Repository Size 2024-03-11T21:18:23.218 INFO:teuthology.orchestra.run.smithi097.stdout:============================================================================================ 2024-03-11T21:18:23.218 INFO:teuthology.orchestra.run.smithi097.stdout:Installing: 2024-03-11T21:18:23.219 INFO:teuthology.orchestra.run.smithi097.stdout: ceph-mgr-diskprediction-local noarch 2:19.0.0-1979.g2b0101eb.el9 ceph-noarch 7.4 M 2024-03-11T21:18:23.219 INFO:teuthology.orchestra.run.smithi097.stdout:Installing dependencies: 2024-03-11T21:18:23.219 INFO:teuthology.orchestra.run.smithi097.stdout: flexiblas x86_64 3.0.4-8.el9 CentOS-AppStream 32 k 2024-03-11T21:18:23.219 INFO:teuthology.orchestra.run.smithi097.stdout: flexiblas-netlib x86_64 3.0.4-8.el9 CentOS-AppStream 3.0 M 2024-03-11T21:18:23.219 INFO:teuthology.orchestra.run.smithi097.stdout: flexiblas-openblas-openmp x86_64 3.0.4-8.el9 CentOS-AppStream 17 k 2024-03-11T21:18:23.219 INFO:teuthology.orchestra.run.smithi097.stdout: libgfortran x86_64 11.4.1-3.el9 baseos 803 k 2024-03-11T21:18:23.219 INFO:teuthology.orchestra.run.smithi097.stdout: libquadmath x86_64 11.4.1-3.el9 baseos 195 k 2024-03-11T21:18:23.219 INFO:teuthology.orchestra.run.smithi097.stdout: openblas x86_64 0.3.21-2.el9 CentOS-AppStream 34 k 2024-03-11T21:18:23.219 INFO:teuthology.orchestra.run.smithi097.stdout: openblas-openmp x86_64 0.3.21-2.el9 CentOS-AppStream 4.7 M 2024-03-11T21:18:23.219 INFO:teuthology.orchestra.run.smithi097.stdout: python3-devel x86_64 3.9.17-2.el9 appstream 245 k 2024-03-11T21:18:23.219 INFO:teuthology.orchestra.run.smithi097.stdout: python3-numpy x86_64 1:1.20.1-5.el9 CentOS-AppStream 4.8 M 2024-03-11T21:18:23.219 INFO:teuthology.orchestra.run.smithi097.stdout: python3-numpy-f2py x86_64 1:1.20.1-5.el9 CentOS-AppStream 217 k 2024-03-11T21:18:23.219 INFO:teuthology.orchestra.run.smithi097.stdout: python3-scipy x86_64 1.6.2-8.el9 CentOS-AppStream 16 M 2024-03-11T21:18:23.219 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-03-11T21:18:23.220 INFO:teuthology.orchestra.run.smithi097.stdout:Transaction Summary 2024-03-11T21:18:23.220 INFO:teuthology.orchestra.run.smithi097.stdout:============================================================================================ 2024-03-11T21:18:23.220 INFO:teuthology.orchestra.run.smithi097.stdout:Install 12 Packages 2024-03-11T21:18:23.220 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-03-11T21:18:23.220 INFO:teuthology.orchestra.run.smithi097.stdout:Total download size: 37 M 2024-03-11T21:18:23.220 INFO:teuthology.orchestra.run.smithi097.stdout:Installed size: 209 M 2024-03-11T21:18:23.220 INFO:teuthology.orchestra.run.smithi097.stdout:Downloading Packages: 2024-03-11T21:18:23.852 INFO:teuthology.orchestra.run.smithi097.stdout:(1/12): flexiblas-3.0.4-8.el9.x86_64.rpm 205 kB/s | 32 kB 00:00 2024-03-11T21:18:23.894 INFO:teuthology.orchestra.run.smithi097.stdout:(2/12): flexiblas-openblas-openmp-3.0.4-8.el9.x 413 kB/s | 17 kB 00:00 2024-03-11T21:18:23.969 INFO:teuthology.orchestra.run.smithi097.stdout:(3/12): flexiblas-netlib-3.0.4-8.el9.x86_64.rpm 11 MB/s | 3.0 MB 00:00 2024-03-11T21:18:23.994 INFO:teuthology.orchestra.run.smithi097.stdout:(4/12): openblas-0.3.21-2.el9.x86_64.rpm 346 kB/s | 34 kB 00:00 2024-03-11T21:18:24.145 INFO:teuthology.orchestra.run.smithi097.stdout:(5/12): openblas-openmp-0.3.21-2.el9.x86_64.rpm 27 MB/s | 4.7 MB 00:00 2024-03-11T21:18:24.211 INFO:teuthology.orchestra.run.smithi097.stdout:(6/12): python3-numpy-1.20.1-5.el9.x86_64.rpm 22 MB/s | 4.8 MB 00:00 2024-03-11T21:18:24.237 INFO:teuthology.orchestra.run.smithi097.stdout:(7/12): python3-numpy-f2py-1.20.1-5.el9.x86_64. 2.4 MB/s | 217 kB 00:00 2024-03-11T21:18:24.462 INFO:teuthology.orchestra.run.smithi097.stdout:(8/12): python3-scipy-1.6.2-8.el9.x86_64.rpm 63 MB/s | 16 MB 00:00 2024-03-11T21:18:24.562 INFO:teuthology.orchestra.run.smithi097.stdout:(9/12): ceph-mgr-diskprediction-local-19.0.0-19 8.5 MB/s | 7.4 MB 00:00 2024-03-11T21:18:24.671 INFO:teuthology.orchestra.run.smithi097.stdout:(10/12): libgfortran-11.4.1-3.el9.x86_64.rpm 1.8 MB/s | 803 kB 00:00 2024-03-11T21:18:24.713 INFO:teuthology.orchestra.run.smithi097.stdout:(11/12): libquadmath-11.4.1-3.el9.x86_64.rpm 781 kB/s | 195 kB 00:00 2024-03-11T21:18:25.130 INFO:teuthology.orchestra.run.smithi097.stdout:(12/12): python3-devel-3.9.17-2.el9.x86_64.rpm 433 kB/s | 245 kB 00:00 2024-03-11T21:18:25.131 INFO:teuthology.orchestra.run.smithi097.stdout:-------------------------------------------------------------------------------- 2024-03-11T21:18:25.132 INFO:teuthology.orchestra.run.smithi097.stdout:Total 19 MB/s | 37 MB 00:01 2024-03-11T21:18:25.171 INFO:teuthology.orchestra.run.smithi097.stdout:Running transaction check 2024-03-11T21:18:25.195 INFO:teuthology.orchestra.run.smithi097.stdout:Transaction check succeeded. 2024-03-11T21:18:25.195 INFO:teuthology.orchestra.run.smithi097.stdout:Running transaction test 2024-03-11T21:18:25.565 INFO:teuthology.orchestra.run.smithi097.stdout:Transaction test succeeded. 2024-03-11T21:18:25.565 INFO:teuthology.orchestra.run.smithi097.stdout:Running transaction 2024-03-11T21:18:26.265 INFO:teuthology.orchestra.run.smithi097.stdout: Preparing : 1/1 2024-03-11T21:18:26.336 INFO:teuthology.orchestra.run.smithi097.stdout: Installing : libquadmath-11.4.1-3.el9.x86_64 1/12 2024-03-11T21:18:26.375 INFO:teuthology.orchestra.run.smithi097.stdout: Installing : libgfortran-11.4.1-3.el9.x86_64 2/12 2024-03-11T21:18:26.458 INFO:teuthology.orchestra.run.smithi097.stdout: Installing : flexiblas-3.0.4-8.el9.x86_64 3/12 2024-03-11T21:18:26.495 INFO:teuthology.orchestra.run.smithi097.stdout: Installing : python3-devel-3.9.17-2.el9.x86_64 4/12 2024-03-11T21:18:26.664 INFO:teuthology.orchestra.run.smithi097.stdout: Installing : openblas-0.3.21-2.el9.x86_64 5/12 2024-03-11T21:18:26.711 INFO:teuthology.orchestra.run.smithi097.stdout: Installing : openblas-openmp-0.3.21-2.el9.x86_64 6/12 2024-03-11T21:18:26.787 INFO:teuthology.orchestra.run.smithi097.stdout: Installing : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 7/12 2024-03-11T21:18:27.427 INFO:teuthology.orchestra.run.smithi097.stdout: Installing : flexiblas-netlib-3.0.4-8.el9.x86_64 8/12 2024-03-11T21:18:27.548 INFO:teuthology.orchestra.run.smithi097.stdout: Installing : python3-numpy-1:1.20.1-5.el9.x86_64 9/12 2024-03-11T21:18:28.796 INFO:teuthology.orchestra.run.smithi097.stdout: Installing : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 10/12 2024-03-11T21:18:29.807 INFO:teuthology.orchestra.run.smithi097.stdout: Installing : python3-scipy-1.6.2-8.el9.x86_64 11/12 2024-03-11T21:18:29.837 INFO:teuthology.orchestra.run.smithi097.stdout: Installing : ceph-mgr-diskprediction-local-2:19.0.0-1979.g2b010 12/12 2024-03-11T21:18:32.379 INFO:teuthology.orchestra.run.smithi097.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:19.0.0-1979.g2b010 12/12 2024-03-11T21:18:32.379 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : ceph-mgr-diskprediction-local-2:19.0.0-1979.g2b010 1/12 2024-03-11T21:18:32.379 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : flexiblas-3.0.4-8.el9.x86_64 2/12 2024-03-11T21:18:32.380 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : flexiblas-netlib-3.0.4-8.el9.x86_64 3/12 2024-03-11T21:18:32.380 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 4/12 2024-03-11T21:18:32.380 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : openblas-0.3.21-2.el9.x86_64 5/12 2024-03-11T21:18:32.380 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : openblas-openmp-0.3.21-2.el9.x86_64 6/12 2024-03-11T21:18:32.380 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : python3-numpy-1:1.20.1-5.el9.x86_64 7/12 2024-03-11T21:18:32.380 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 8/12 2024-03-11T21:18:32.380 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : python3-scipy-1.6.2-8.el9.x86_64 9/12 2024-03-11T21:18:32.380 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : libgfortran-11.4.1-3.el9.x86_64 10/12 2024-03-11T21:18:32.380 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : libquadmath-11.4.1-3.el9.x86_64 11/12 2024-03-11T21:18:32.884 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : python3-devel-3.9.17-2.el9.x86_64 12/12 2024-03-11T21:18:32.884 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-03-11T21:18:32.884 INFO:teuthology.orchestra.run.smithi097.stdout:Installed: 2024-03-11T21:18:32.884 INFO:teuthology.orchestra.run.smithi097.stdout: ceph-mgr-diskprediction-local-2:19.0.0-1979.g2b0101eb.el9.noarch 2024-03-11T21:18:32.884 INFO:teuthology.orchestra.run.smithi097.stdout: flexiblas-3.0.4-8.el9.x86_64 2024-03-11T21:18:32.884 INFO:teuthology.orchestra.run.smithi097.stdout: flexiblas-netlib-3.0.4-8.el9.x86_64 2024-03-11T21:18:32.884 INFO:teuthology.orchestra.run.smithi097.stdout: flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 2024-03-11T21:18:32.884 INFO:teuthology.orchestra.run.smithi097.stdout: libgfortran-11.4.1-3.el9.x86_64 2024-03-11T21:18:32.885 INFO:teuthology.orchestra.run.smithi097.stdout: libquadmath-11.4.1-3.el9.x86_64 2024-03-11T21:18:32.885 INFO:teuthology.orchestra.run.smithi097.stdout: openblas-0.3.21-2.el9.x86_64 2024-03-11T21:18:32.885 INFO:teuthology.orchestra.run.smithi097.stdout: openblas-openmp-0.3.21-2.el9.x86_64 2024-03-11T21:18:32.885 INFO:teuthology.orchestra.run.smithi097.stdout: python3-devel-3.9.17-2.el9.x86_64 2024-03-11T21:18:32.885 INFO:teuthology.orchestra.run.smithi097.stdout: python3-numpy-1:1.20.1-5.el9.x86_64 2024-03-11T21:18:32.885 INFO:teuthology.orchestra.run.smithi097.stdout: python3-numpy-f2py-1:1.20.1-5.el9.x86_64 2024-03-11T21:18:32.885 INFO:teuthology.orchestra.run.smithi097.stdout: python3-scipy-1.6.2-8.el9.x86_64 2024-03-11T21:18:32.885 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-03-11T21:18:32.885 INFO:teuthology.orchestra.run.smithi097.stdout:Complete! 2024-03-11T21:18:33.249 DEBUG:teuthology.orchestra.run.smithi097:> sudo yum -y install ceph-mgr-rook 2024-03-11T21:18:33.821 INFO:teuthology.orchestra.run.smithi097.stdout:Last metadata expiration check: 0:01:17 ago on Mon 11 Mar 2024 09:17:16 PM UTC. 2024-03-11T21:18:33.974 INFO:teuthology.orchestra.run.smithi097.stdout:Dependencies resolved. 2024-03-11T21:18:33.975 INFO:teuthology.orchestra.run.smithi097.stdout:======================================================================================== 2024-03-11T21:18:33.975 INFO:teuthology.orchestra.run.smithi097.stdout: Package Arch Version Repository Size 2024-03-11T21:18:33.975 INFO:teuthology.orchestra.run.smithi097.stdout:======================================================================================== 2024-03-11T21:18:33.975 INFO:teuthology.orchestra.run.smithi097.stdout:Installing: 2024-03-11T21:18:33.975 INFO:teuthology.orchestra.run.smithi097.stdout: ceph-mgr-rook noarch 2:19.0.0-1979.g2b0101eb.el9 ceph-noarch 49 k 2024-03-11T21:18:33.975 INFO:teuthology.orchestra.run.smithi097.stdout:Installing dependencies: 2024-03-11T21:18:33.976 INFO:teuthology.orchestra.run.smithi097.stdout: python3-cachetools noarch 4.2.4-1.el9 epel 32 k 2024-03-11T21:18:33.976 INFO:teuthology.orchestra.run.smithi097.stdout: python3-certifi noarch 2021.10.8-2.el9 epel 15 k 2024-03-11T21:18:33.976 INFO:teuthology.orchestra.run.smithi097.stdout: python3-google-auth noarch 1:2.28.0-1.el9 epel 203 k 2024-03-11T21:18:33.976 INFO:teuthology.orchestra.run.smithi097.stdout: python3-jsonpatch noarch 1.21-16.el9 CentOS-AppStream 24 k 2024-03-11T21:18:33.976 INFO:teuthology.orchestra.run.smithi097.stdout: python3-jsonpointer noarch 2.0-4.el9 CentOS-AppStream 18 k 2024-03-11T21:18:33.976 INFO:teuthology.orchestra.run.smithi097.stdout: python3-kubernetes noarch 1:26.1.0-2.el9 epel 1.0 M 2024-03-11T21:18:33.976 INFO:teuthology.orchestra.run.smithi097.stdout: python3-oauthlib noarch 3.1.1-5.el9 CentOS-AppStream 174 k 2024-03-11T21:18:33.976 INFO:teuthology.orchestra.run.smithi097.stdout: python3-pyasn1 noarch 0.4.8-6.el9 CentOS-AppStream 132 k 2024-03-11T21:18:33.976 INFO:teuthology.orchestra.run.smithi097.stdout: python3-pyasn1-modules noarch 0.4.8-6.el9 CentOS-AppStream 211 k 2024-03-11T21:18:33.976 INFO:teuthology.orchestra.run.smithi097.stdout: python3-requests-oauthlib noarch 1.3.0-12.el9 CentOS-AppStream 43 k 2024-03-11T21:18:33.976 INFO:teuthology.orchestra.run.smithi097.stdout: python3-rsa noarch 4.9-2.el9 epel 59 k 2024-03-11T21:18:33.976 INFO:teuthology.orchestra.run.smithi097.stdout: python3-websocket-client noarch 1.2.3-2.el9 epel 90 k 2024-03-11T21:18:33.976 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-03-11T21:18:33.977 INFO:teuthology.orchestra.run.smithi097.stdout:Transaction Summary 2024-03-11T21:18:33.977 INFO:teuthology.orchestra.run.smithi097.stdout:======================================================================================== 2024-03-11T21:18:33.977 INFO:teuthology.orchestra.run.smithi097.stdout:Install 13 Packages 2024-03-11T21:18:33.977 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-03-11T21:18:33.977 INFO:teuthology.orchestra.run.smithi097.stdout:Total download size: 2.0 M 2024-03-11T21:18:33.977 INFO:teuthology.orchestra.run.smithi097.stdout:Installed size: 26 M 2024-03-11T21:18:33.977 INFO:teuthology.orchestra.run.smithi097.stdout:Downloading Packages: 2024-03-11T21:18:34.314 INFO:teuthology.orchestra.run.smithi097.stdout:(1/13): python3-jsonpatch-1.21-16.el9.noarch.rp 178 kB/s | 24 kB 00:00 2024-03-11T21:18:34.339 INFO:teuthology.orchestra.run.smithi097.stdout:(2/13): python3-jsonpointer-2.0-4.el9.noarch.rp 111 kB/s | 18 kB 00:00 2024-03-11T21:18:34.431 INFO:teuthology.orchestra.run.smithi097.stdout:(3/13): python3-oauthlib-3.1.1-5.el9.noarch.rpm 1.9 MB/s | 174 kB 00:00 2024-03-11T21:18:34.448 INFO:teuthology.orchestra.run.smithi097.stdout:(4/13): python3-pyasn1-0.4.8-6.el9.noarch.rpm 1.2 MB/s | 132 kB 00:00 2024-03-11T21:18:34.465 INFO:teuthology.orchestra.run.smithi097.stdout:(5/13): ceph-mgr-rook-19.0.0-1979.g2b0101eb.el9 171 kB/s | 49 kB 00:00 2024-03-11T21:18:34.490 INFO:teuthology.orchestra.run.smithi097.stdout:(6/13): python3-pyasn1-modules-0.4.8-6.el9.noar 3.5 MB/s | 211 kB 00:00 2024-03-11T21:18:34.507 INFO:teuthology.orchestra.run.smithi097.stdout:(7/13): python3-requests-oauthlib-1.3.0-12.el9. 727 kB/s | 43 kB 00:00 2024-03-11T21:18:34.582 INFO:teuthology.orchestra.run.smithi097.stdout:(8/13): python3-certifi-2021.10.8-2.el9.noarch. 200 kB/s | 15 kB 00:00 2024-03-11T21:18:34.599 INFO:teuthology.orchestra.run.smithi097.stdout:(9/13): python3-cachetools-4.2.4-1.el9.noarch.r 241 kB/s | 32 kB 00:00 2024-03-11T21:18:34.658 INFO:teuthology.orchestra.run.smithi097.stdout:(10/13): python3-google-auth-2.28.0-1.el9.noarc 1.3 MB/s | 203 kB 00:00 2024-03-11T21:18:34.674 INFO:teuthology.orchestra.run.smithi097.stdout:(11/13): python3-rsa-4.9-2.el9.noarch.rpm 784 kB/s | 59 kB 00:00 2024-03-11T21:18:34.708 INFO:teuthology.orchestra.run.smithi097.stdout:(12/13): python3-websocket-client-1.2.3-2.el9.n 1.8 MB/s | 90 kB 00:00 2024-03-11T21:18:34.758 INFO:teuthology.orchestra.run.smithi097.stdout:(13/13): python3-kubernetes-26.1.0-2.el9.noarch 5.8 MB/s | 1.0 MB 00:00 2024-03-11T21:18:34.760 INFO:teuthology.orchestra.run.smithi097.stdout:-------------------------------------------------------------------------------- 2024-03-11T21:18:34.760 INFO:teuthology.orchestra.run.smithi097.stdout:Total 2.6 MB/s | 2.0 MB 00:00 2024-03-11T21:18:34.761 INFO:teuthology.orchestra.run.smithi097.stdout:Running transaction check 2024-03-11T21:18:34.774 INFO:teuthology.orchestra.run.smithi097.stdout:Transaction check succeeded. 2024-03-11T21:18:34.774 INFO:teuthology.orchestra.run.smithi097.stdout:Running transaction test 2024-03-11T21:18:34.874 INFO:teuthology.orchestra.run.smithi097.stdout:Transaction test succeeded. 2024-03-11T21:18:34.874 INFO:teuthology.orchestra.run.smithi097.stdout:Running transaction 2024-03-11T21:18:35.409 INFO:teuthology.orchestra.run.smithi097.stdout: Preparing : 1/1 2024-03-11T21:18:35.574 INFO:teuthology.orchestra.run.smithi097.stdout: Installing : python3-pyasn1-0.4.8-6.el9.noarch 1/13 2024-03-11T21:18:35.635 INFO:teuthology.orchestra.run.smithi097.stdout: Installing : python3-pyasn1-modules-0.4.8-6.el9.noarch 2/13 2024-03-11T21:18:35.690 INFO:teuthology.orchestra.run.smithi097.stdout: Installing : python3-rsa-4.9-2.el9.noarch 3/13 2024-03-11T21:18:35.728 INFO:teuthology.orchestra.run.smithi097.stdout: Installing : python3-websocket-client-1.2.3-2.el9.noarch 4/13 2024-03-11T21:18:35.769 INFO:teuthology.orchestra.run.smithi097.stdout: Installing : python3-certifi-2021.10.8-2.el9.noarch 5/13 2024-03-11T21:18:35.862 INFO:teuthology.orchestra.run.smithi097.stdout: Installing : python3-cachetools-4.2.4-1.el9.noarch 6/13 2024-03-11T21:18:35.971 INFO:teuthology.orchestra.run.smithi097.stdout: Installing : python3-google-auth-1:2.28.0-1.el9.noarch 7/13 2024-03-11T21:18:36.021 INFO:teuthology.orchestra.run.smithi097.stdout: Installing : python3-oauthlib-3.1.1-5.el9.noarch 8/13 2024-03-11T21:18:36.756 INFO:teuthology.orchestra.run.smithi097.stdout: Installing : python3-requests-oauthlib-1.3.0-12.el9.noarch 9/13 2024-03-11T21:18:36.801 INFO:teuthology.orchestra.run.smithi097.stdout: Installing : python3-kubernetes-1:26.1.0-2.el9.noarch 10/13 2024-03-11T21:18:36.835 INFO:teuthology.orchestra.run.smithi097.stdout: Installing : python3-jsonpointer-2.0-4.el9.noarch 11/13 2024-03-11T21:18:36.883 INFO:teuthology.orchestra.run.smithi097.stdout: Installing : python3-jsonpatch-1.21-16.el9.noarch 12/13 2024-03-11T21:18:36.905 INFO:teuthology.orchestra.run.smithi097.stdout: Installing : ceph-mgr-rook-2:19.0.0-1979.g2b0101eb.el9.noarch 13/13 2024-03-11T21:18:37.711 INFO:teuthology.orchestra.run.smithi097.stdout: Running scriptlet: ceph-mgr-rook-2:19.0.0-1979.g2b0101eb.el9.noarch 13/13 2024-03-11T21:18:37.712 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : ceph-mgr-rook-2:19.0.0-1979.g2b0101eb.el9.noarch 1/13 2024-03-11T21:18:37.712 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : python3-jsonpatch-1.21-16.el9.noarch 2/13 2024-03-11T21:18:37.712 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : python3-jsonpointer-2.0-4.el9.noarch 3/13 2024-03-11T21:18:37.712 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : python3-oauthlib-3.1.1-5.el9.noarch 4/13 2024-03-11T21:18:37.712 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : python3-pyasn1-0.4.8-6.el9.noarch 5/13 2024-03-11T21:18:37.712 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : python3-pyasn1-modules-0.4.8-6.el9.noarch 6/13 2024-03-11T21:18:37.712 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : python3-requests-oauthlib-1.3.0-12.el9.noarch 7/13 2024-03-11T21:18:37.712 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : python3-cachetools-4.2.4-1.el9.noarch 8/13 2024-03-11T21:18:37.712 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : python3-certifi-2021.10.8-2.el9.noarch 9/13 2024-03-11T21:18:37.712 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : python3-google-auth-1:2.28.0-1.el9.noarch 10/13 2024-03-11T21:18:37.712 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : python3-kubernetes-1:26.1.0-2.el9.noarch 11/13 2024-03-11T21:18:37.712 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : python3-rsa-4.9-2.el9.noarch 12/13 2024-03-11T21:18:38.090 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : python3-websocket-client-1.2.3-2.el9.noarch 13/13 2024-03-11T21:18:38.090 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-03-11T21:18:38.090 INFO:teuthology.orchestra.run.smithi097.stdout:Installed: 2024-03-11T21:18:38.090 INFO:teuthology.orchestra.run.smithi097.stdout: ceph-mgr-rook-2:19.0.0-1979.g2b0101eb.el9.noarch 2024-03-11T21:18:38.090 INFO:teuthology.orchestra.run.smithi097.stdout: python3-cachetools-4.2.4-1.el9.noarch 2024-03-11T21:18:38.090 INFO:teuthology.orchestra.run.smithi097.stdout: python3-certifi-2021.10.8-2.el9.noarch 2024-03-11T21:18:38.090 INFO:teuthology.orchestra.run.smithi097.stdout: python3-google-auth-1:2.28.0-1.el9.noarch 2024-03-11T21:18:38.090 INFO:teuthology.orchestra.run.smithi097.stdout: python3-jsonpatch-1.21-16.el9.noarch 2024-03-11T21:18:38.090 INFO:teuthology.orchestra.run.smithi097.stdout: python3-jsonpointer-2.0-4.el9.noarch 2024-03-11T21:18:38.090 INFO:teuthology.orchestra.run.smithi097.stdout: python3-kubernetes-1:26.1.0-2.el9.noarch 2024-03-11T21:18:38.090 INFO:teuthology.orchestra.run.smithi097.stdout: python3-oauthlib-3.1.1-5.el9.noarch 2024-03-11T21:18:38.091 INFO:teuthology.orchestra.run.smithi097.stdout: python3-pyasn1-0.4.8-6.el9.noarch 2024-03-11T21:18:38.091 INFO:teuthology.orchestra.run.smithi097.stdout: python3-pyasn1-modules-0.4.8-6.el9.noarch 2024-03-11T21:18:38.091 INFO:teuthology.orchestra.run.smithi097.stdout: python3-requests-oauthlib-1.3.0-12.el9.noarch 2024-03-11T21:18:38.091 INFO:teuthology.orchestra.run.smithi097.stdout: python3-rsa-4.9-2.el9.noarch 2024-03-11T21:18:38.091 INFO:teuthology.orchestra.run.smithi097.stdout: python3-websocket-client-1.2.3-2.el9.noarch 2024-03-11T21:18:38.091 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-03-11T21:18:38.091 INFO:teuthology.orchestra.run.smithi097.stdout:Complete! 2024-03-11T21:18:38.353 DEBUG:teuthology.orchestra.run.smithi097:> sudo yum -y install ceph-mgr-cephadm 2024-03-11T21:18:38.912 INFO:teuthology.orchestra.run.smithi097.stdout:Last metadata expiration check: 0:01:22 ago on Mon 11 Mar 2024 09:17:16 PM UTC. 2024-03-11T21:18:39.064 INFO:teuthology.orchestra.run.smithi097.stdout:Dependencies resolved. 2024-03-11T21:18:39.065 INFO:teuthology.orchestra.run.smithi097.stdout:======================================================================================== 2024-03-11T21:18:39.065 INFO:teuthology.orchestra.run.smithi097.stdout: Package Arch Version Repository Size 2024-03-11T21:18:39.065 INFO:teuthology.orchestra.run.smithi097.stdout:======================================================================================== 2024-03-11T21:18:39.065 INFO:teuthology.orchestra.run.smithi097.stdout:Installing: 2024-03-11T21:18:39.065 INFO:teuthology.orchestra.run.smithi097.stdout: ceph-mgr-cephadm noarch 2:19.0.0-1979.g2b0101eb.el9 ceph-noarch 139 k 2024-03-11T21:18:39.065 INFO:teuthology.orchestra.run.smithi097.stdout:Installing dependencies: 2024-03-11T21:18:39.065 INFO:teuthology.orchestra.run.smithi097.stdout: python3-asyncssh noarch 2.13.2-4.el9 epel 548 k 2024-03-11T21:18:39.066 INFO:teuthology.orchestra.run.smithi097.stdout: python3-babel noarch 2.9.1-2.el9 CentOS-AppStream 5.8 M 2024-03-11T21:18:39.066 INFO:teuthology.orchestra.run.smithi097.stdout: python3-jinja2 noarch 2.11.3-5.el9 appstream 248 k 2024-03-11T21:18:39.066 INFO:teuthology.orchestra.run.smithi097.stdout: python3-natsort noarch 7.1.1-5.el9 epel 58 k 2024-03-11T21:18:39.066 INFO:teuthology.orchestra.run.smithi097.stdout: python3-typing-extensions noarch 4.4.0-2.el9 epel 51 k 2024-03-11T21:18:39.066 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-03-11T21:18:39.066 INFO:teuthology.orchestra.run.smithi097.stdout:Transaction Summary 2024-03-11T21:18:39.066 INFO:teuthology.orchestra.run.smithi097.stdout:======================================================================================== 2024-03-11T21:18:39.066 INFO:teuthology.orchestra.run.smithi097.stdout:Install 6 Packages 2024-03-11T21:18:39.066 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-03-11T21:18:39.067 INFO:teuthology.orchestra.run.smithi097.stdout:Total download size: 6.8 M 2024-03-11T21:18:39.067 INFO:teuthology.orchestra.run.smithi097.stdout:Installed size: 33 M 2024-03-11T21:18:39.067 INFO:teuthology.orchestra.run.smithi097.stdout:Downloading Packages: 2024-03-11T21:18:39.838 INFO:teuthology.orchestra.run.smithi097.stdout:(1/6): python3-jinja2-2.11.3-5.el9.noarch.rpm 1.2 MB/s | 248 kB 00:00 2024-03-11T21:18:39.863 INFO:teuthology.orchestra.run.smithi097.stdout:(2/6): ceph-mgr-cephadm-19.0.0-1979.g2b0101eb.e 595 kB/s | 139 kB 00:00 2024-03-11T21:18:39.930 INFO:teuthology.orchestra.run.smithi097.stdout:(3/6): python3-babel-2.9.1-2.el9.noarch.rpm 19 MB/s | 5.8 MB 00:00 2024-03-11T21:18:40.230 INFO:teuthology.orchestra.run.smithi097.stdout:(4/6): python3-natsort-7.1.1-5.el9.noarch.rpm 158 kB/s | 58 kB 00:00 2024-03-11T21:18:40.330 INFO:teuthology.orchestra.run.smithi097.stdout:(5/6): python3-typing-extensions-4.4.0-2.el9.no 144 kB/s | 51 kB 00:00 2024-03-11T21:18:40.423 INFO:teuthology.orchestra.run.smithi097.stdout:(6/6): python3-asyncssh-2.13.2-4.el9.noarch.rpm 941 kB/s | 548 kB 00:00 2024-03-11T21:18:40.425 INFO:teuthology.orchestra.run.smithi097.stdout:-------------------------------------------------------------------------------- 2024-03-11T21:18:40.425 INFO:teuthology.orchestra.run.smithi097.stdout:Total 5.0 MB/s | 6.8 MB 00:01 2024-03-11T21:18:40.438 INFO:teuthology.orchestra.run.smithi097.stdout:Running transaction check 2024-03-11T21:18:40.448 INFO:teuthology.orchestra.run.smithi097.stdout:Transaction check succeeded. 2024-03-11T21:18:40.448 INFO:teuthology.orchestra.run.smithi097.stdout:Running transaction test 2024-03-11T21:18:40.540 INFO:teuthology.orchestra.run.smithi097.stdout:Transaction test succeeded. 2024-03-11T21:18:40.541 INFO:teuthology.orchestra.run.smithi097.stdout:Running transaction 2024-03-11T21:18:40.853 INFO:teuthology.orchestra.run.smithi097.stdout: Preparing : 1/1 2024-03-11T21:18:41.030 INFO:teuthology.orchestra.run.smithi097.stdout: Installing : python3-typing-extensions-4.4.0-2.el9.noarch 1/6 2024-03-11T21:18:41.074 INFO:teuthology.orchestra.run.smithi097.stdout: Installing : python3-asyncssh-2.13.2-4.el9.noarch 2/6 2024-03-11T21:18:41.603 INFO:teuthology.orchestra.run.smithi097.stdout: Installing : python3-natsort-7.1.1-5.el9.noarch 3/6 2024-03-11T21:18:41.684 INFO:teuthology.orchestra.run.smithi097.stdout: Installing : python3-babel-2.9.1-2.el9.noarch 4/6 2024-03-11T21:18:41.746 INFO:teuthology.orchestra.run.smithi097.stdout: Installing : python3-jinja2-2.11.3-5.el9.noarch 5/6 2024-03-11T21:18:41.770 INFO:teuthology.orchestra.run.smithi097.stdout: Installing : ceph-mgr-cephadm-2:19.0.0-1979.g2b0101eb.el9.noarch 6/6 2024-03-11T21:18:42.573 INFO:teuthology.orchestra.run.smithi097.stdout: Running scriptlet: ceph-mgr-cephadm-2:19.0.0-1979.g2b0101eb.el9.noarch 6/6 2024-03-11T21:18:42.573 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : ceph-mgr-cephadm-2:19.0.0-1979.g2b0101eb.el9.noarch 1/6 2024-03-11T21:18:42.573 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : python3-babel-2.9.1-2.el9.noarch 2/6 2024-03-11T21:18:42.573 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : python3-jinja2-2.11.3-5.el9.noarch 3/6 2024-03-11T21:18:42.573 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : python3-asyncssh-2.13.2-4.el9.noarch 4/6 2024-03-11T21:18:42.574 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : python3-natsort-7.1.1-5.el9.noarch 5/6 2024-03-11T21:18:42.878 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : python3-typing-extensions-4.4.0-2.el9.noarch 6/6 2024-03-11T21:18:42.878 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-03-11T21:18:42.878 INFO:teuthology.orchestra.run.smithi097.stdout:Installed: 2024-03-11T21:18:42.878 INFO:teuthology.orchestra.run.smithi097.stdout: ceph-mgr-cephadm-2:19.0.0-1979.g2b0101eb.el9.noarch 2024-03-11T21:18:42.878 INFO:teuthology.orchestra.run.smithi097.stdout: python3-asyncssh-2.13.2-4.el9.noarch 2024-03-11T21:18:42.878 INFO:teuthology.orchestra.run.smithi097.stdout: python3-babel-2.9.1-2.el9.noarch 2024-03-11T21:18:42.879 INFO:teuthology.orchestra.run.smithi097.stdout: python3-jinja2-2.11.3-5.el9.noarch 2024-03-11T21:18:42.879 INFO:teuthology.orchestra.run.smithi097.stdout: python3-natsort-7.1.1-5.el9.noarch 2024-03-11T21:18:42.879 INFO:teuthology.orchestra.run.smithi097.stdout: python3-typing-extensions-4.4.0-2.el9.noarch 2024-03-11T21:18:42.879 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-03-11T21:18:42.879 INFO:teuthology.orchestra.run.smithi097.stdout:Complete! 2024-03-11T21:18:43.101 DEBUG:teuthology.orchestra.run.smithi097:> sudo yum -y install ceph-fuse 2024-03-11T21:18:43.662 INFO:teuthology.orchestra.run.smithi097.stdout:Last metadata expiration check: 0:01:27 ago on Mon 11 Mar 2024 09:17:16 PM UTC. 2024-03-11T21:18:43.812 INFO:teuthology.orchestra.run.smithi097.stdout:Dependencies resolved. 2024-03-11T21:18:43.813 INFO:teuthology.orchestra.run.smithi097.stdout:================================================================================ 2024-03-11T21:18:43.813 INFO:teuthology.orchestra.run.smithi097.stdout: Package Arch Version Repository Size 2024-03-11T21:18:43.813 INFO:teuthology.orchestra.run.smithi097.stdout:================================================================================ 2024-03-11T21:18:43.813 INFO:teuthology.orchestra.run.smithi097.stdout:Installing: 2024-03-11T21:18:43.813 INFO:teuthology.orchestra.run.smithi097.stdout: ceph-fuse x86_64 2:19.0.0-1979.g2b0101eb.el9 ceph 888 k 2024-03-11T21:18:43.813 INFO:teuthology.orchestra.run.smithi097.stdout:Installing dependencies: 2024-03-11T21:18:43.813 INFO:teuthology.orchestra.run.smithi097.stdout: fuse x86_64 2.9.9-15.el9 CentOS-BaseOS 79 k 2024-03-11T21:18:43.813 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-03-11T21:18:43.813 INFO:teuthology.orchestra.run.smithi097.stdout:Transaction Summary 2024-03-11T21:18:43.813 INFO:teuthology.orchestra.run.smithi097.stdout:================================================================================ 2024-03-11T21:18:43.813 INFO:teuthology.orchestra.run.smithi097.stdout:Install 2 Packages 2024-03-11T21:18:43.814 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-03-11T21:18:43.814 INFO:teuthology.orchestra.run.smithi097.stdout:Total download size: 966 k 2024-03-11T21:18:43.814 INFO:teuthology.orchestra.run.smithi097.stdout:Installed size: 2.8 M 2024-03-11T21:18:43.814 INFO:teuthology.orchestra.run.smithi097.stdout:Downloading Packages: 2024-03-11T21:18:43.949 INFO:teuthology.orchestra.run.smithi097.stdout:(1/2): fuse-2.9.9-15.el9.x86_64.rpm 582 kB/s | 79 kB 00:00 2024-03-11T21:18:45.274 INFO:teuthology.orchestra.run.smithi097.stdout:(2/2): ceph-fuse-19.0.0-1979.g2b0101eb.el9.x86_ 608 kB/s | 888 kB 00:01 2024-03-11T21:18:45.276 INFO:teuthology.orchestra.run.smithi097.stdout:-------------------------------------------------------------------------------- 2024-03-11T21:18:45.276 INFO:teuthology.orchestra.run.smithi097.stdout:Total 661 kB/s | 966 kB 00:01 2024-03-11T21:18:45.276 INFO:teuthology.orchestra.run.smithi097.stdout:Running transaction check 2024-03-11T21:18:45.300 INFO:teuthology.orchestra.run.smithi097.stdout:Transaction check succeeded. 2024-03-11T21:18:45.300 INFO:teuthology.orchestra.run.smithi097.stdout:Running transaction test 2024-03-11T21:18:45.377 INFO:teuthology.orchestra.run.smithi097.stdout:Transaction test succeeded. 2024-03-11T21:18:45.378 INFO:teuthology.orchestra.run.smithi097.stdout:Running transaction 2024-03-11T21:18:45.556 INFO:teuthology.orchestra.run.smithi097.stdout: Preparing : 1/1 2024-03-11T21:18:45.697 INFO:teuthology.orchestra.run.smithi097.stdout: Installing : fuse-2.9.9-15.el9.x86_64 1/2 2024-03-11T21:18:45.766 INFO:teuthology.orchestra.run.smithi097.stdout: Installing : ceph-fuse-2:19.0.0-1979.g2b0101eb.el9.x86_64 2/2 2024-03-11T21:18:46.501 INFO:teuthology.orchestra.run.smithi097.stdout: Running scriptlet: ceph-fuse-2:19.0.0-1979.g2b0101eb.el9.x86_64 2/2 2024-03-11T21:18:46.501 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : ceph-fuse-2:19.0.0-1979.g2b0101eb.el9.x86_64 1/2 2024-03-11T21:18:46.811 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : fuse-2.9.9-15.el9.x86_64 2/2 2024-03-11T21:18:46.811 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-03-11T21:18:46.811 INFO:teuthology.orchestra.run.smithi097.stdout:Installed: 2024-03-11T21:18:46.812 INFO:teuthology.orchestra.run.smithi097.stdout: ceph-fuse-2:19.0.0-1979.g2b0101eb.el9.x86_64 fuse-2.9.9-15.el9.x86_64 2024-03-11T21:18:46.812 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-03-11T21:18:46.812 INFO:teuthology.orchestra.run.smithi097.stdout:Complete! 2024-03-11T21:18:47.085 DEBUG:teuthology.orchestra.run.smithi097:> sudo yum -y install ceph-volume 2024-03-11T21:18:47.665 INFO:teuthology.orchestra.run.smithi097.stdout:Last metadata expiration check: 0:01:31 ago on Mon 11 Mar 2024 09:17:16 PM UTC. 2024-03-11T21:18:47.814 INFO:teuthology.orchestra.run.smithi097.stdout:Dependencies resolved. 2024-03-11T21:18:47.815 INFO:teuthology.orchestra.run.smithi097.stdout:================================================================================ 2024-03-11T21:18:47.815 INFO:teuthology.orchestra.run.smithi097.stdout: Package Arch Version Repository Size 2024-03-11T21:18:47.815 INFO:teuthology.orchestra.run.smithi097.stdout:================================================================================ 2024-03-11T21:18:47.815 INFO:teuthology.orchestra.run.smithi097.stdout:Installing: 2024-03-11T21:18:47.815 INFO:teuthology.orchestra.run.smithi097.stdout: ceph-volume noarch 2:19.0.0-1979.g2b0101eb.el9 ceph-noarch 264 k 2024-03-11T21:18:47.815 INFO:teuthology.orchestra.run.smithi097.stdout:Installing dependencies: 2024-03-11T21:18:47.815 INFO:teuthology.orchestra.run.smithi097.stdout: python3-packaging noarch 20.9-5.el9 CentOS-AppStream 69 k 2024-03-11T21:18:47.815 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-03-11T21:18:47.815 INFO:teuthology.orchestra.run.smithi097.stdout:Transaction Summary 2024-03-11T21:18:47.815 INFO:teuthology.orchestra.run.smithi097.stdout:================================================================================ 2024-03-11T21:18:47.815 INFO:teuthology.orchestra.run.smithi097.stdout:Install 2 Packages 2024-03-11T21:18:47.816 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-03-11T21:18:47.816 INFO:teuthology.orchestra.run.smithi097.stdout:Total download size: 332 k 2024-03-11T21:18:47.816 INFO:teuthology.orchestra.run.smithi097.stdout:Installed size: 1.5 M 2024-03-11T21:18:47.816 INFO:teuthology.orchestra.run.smithi097.stdout:Downloading Packages: 2024-03-11T21:18:47.946 INFO:teuthology.orchestra.run.smithi097.stdout:(1/2): python3-packaging-20.9-5.el9.noarch.rpm 526 kB/s | 69 kB 00:00 2024-03-11T21:18:48.613 INFO:teuthology.orchestra.run.smithi097.stdout:(2/2): ceph-volume-19.0.0-1979.g2b0101eb.el9.no 331 kB/s | 264 kB 00:00 2024-03-11T21:18:48.614 INFO:teuthology.orchestra.run.smithi097.stdout:-------------------------------------------------------------------------------- 2024-03-11T21:18:48.615 INFO:teuthology.orchestra.run.smithi097.stdout:Total 415 kB/s | 332 kB 00:00 2024-03-11T21:18:48.615 INFO:teuthology.orchestra.run.smithi097.stdout:Running transaction check 2024-03-11T21:18:48.625 INFO:teuthology.orchestra.run.smithi097.stdout:Transaction check succeeded. 2024-03-11T21:18:48.625 INFO:teuthology.orchestra.run.smithi097.stdout:Running transaction test 2024-03-11T21:18:48.672 INFO:teuthology.orchestra.run.smithi097.stdout:Transaction test succeeded. 2024-03-11T21:18:48.672 INFO:teuthology.orchestra.run.smithi097.stdout:Running transaction 2024-03-11T21:18:48.825 INFO:teuthology.orchestra.run.smithi097.stdout: Preparing : 1/1 2024-03-11T21:18:48.993 INFO:teuthology.orchestra.run.smithi097.stdout: Installing : python3-packaging-20.9-5.el9.noarch 1/2 2024-03-11T21:18:49.024 INFO:teuthology.orchestra.run.smithi097.stdout: Installing : ceph-volume-2:19.0.0-1979.g2b0101eb.el9.noarch 2/2 2024-03-11T21:18:49.039 INFO:teuthology.orchestra.run.smithi097.stdout: Running scriptlet: ceph-volume-2:19.0.0-1979.g2b0101eb.el9.noarch 2/2 2024-03-11T21:18:49.040 INFO:teuthology.orchestra.run.smithi097.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-11T21:18:49.040 INFO:teuthology.orchestra.run.smithi097.stdout:Invalid unit name "ceph-volume@*.service" escaped as "ceph-volume@\x2a.service". 2024-03-11T21:18:49.040 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-03-11T21:18:49.776 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : ceph-volume-2:19.0.0-1979.g2b0101eb.el9.noarch 1/2 2024-03-11T21:18:50.111 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : python3-packaging-20.9-5.el9.noarch 2/2 2024-03-11T21:18:50.112 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-03-11T21:18:50.112 INFO:teuthology.orchestra.run.smithi097.stdout:Installed: 2024-03-11T21:18:50.112 INFO:teuthology.orchestra.run.smithi097.stdout: ceph-volume-2:19.0.0-1979.g2b0101eb.el9.noarch 2024-03-11T21:18:50.112 INFO:teuthology.orchestra.run.smithi097.stdout: python3-packaging-20.9-5.el9.noarch 2024-03-11T21:18:50.112 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-03-11T21:18:50.112 INFO:teuthology.orchestra.run.smithi097.stdout:Complete! 2024-03-11T21:18:50.457 DEBUG:teuthology.orchestra.run.smithi097:> sudo yum -y install librados-devel 2024-03-11T21:18:51.032 INFO:teuthology.orchestra.run.smithi097.stdout:Last metadata expiration check: 0:01:35 ago on Mon 11 Mar 2024 09:17:16 PM UTC. 2024-03-11T21:18:51.183 INFO:teuthology.orchestra.run.smithi097.stdout:Dependencies resolved. 2024-03-11T21:18:51.183 INFO:teuthology.orchestra.run.smithi097.stdout:================================================================================ 2024-03-11T21:18:51.184 INFO:teuthology.orchestra.run.smithi097.stdout: Package Arch Version Repo Size 2024-03-11T21:18:51.184 INFO:teuthology.orchestra.run.smithi097.stdout:================================================================================ 2024-03-11T21:18:51.184 INFO:teuthology.orchestra.run.smithi097.stdout:Installing: 2024-03-11T21:18:51.184 INFO:teuthology.orchestra.run.smithi097.stdout: librados-devel x86_64 2:19.0.0-1979.g2b0101eb.el9 ceph 127 k 2024-03-11T21:18:51.184 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-03-11T21:18:51.184 INFO:teuthology.orchestra.run.smithi097.stdout:Transaction Summary 2024-03-11T21:18:51.184 INFO:teuthology.orchestra.run.smithi097.stdout:================================================================================ 2024-03-11T21:18:51.184 INFO:teuthology.orchestra.run.smithi097.stdout:Install 1 Package 2024-03-11T21:18:51.184 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-03-11T21:18:51.184 INFO:teuthology.orchestra.run.smithi097.stdout:Total download size: 127 k 2024-03-11T21:18:51.184 INFO:teuthology.orchestra.run.smithi097.stdout:Installed size: 456 k 2024-03-11T21:18:51.184 INFO:teuthology.orchestra.run.smithi097.stdout:Downloading Packages: 2024-03-11T21:18:51.586 INFO:teuthology.orchestra.run.smithi097.stdout:librados-devel-19.0.0-1979.g2b0101eb.el9.x86_64 315 kB/s | 127 kB 00:00 2024-03-11T21:18:51.587 INFO:teuthology.orchestra.run.smithi097.stdout:-------------------------------------------------------------------------------- 2024-03-11T21:18:51.587 INFO:teuthology.orchestra.run.smithi097.stdout:Total 314 kB/s | 127 kB 00:00 2024-03-11T21:18:51.588 INFO:teuthology.orchestra.run.smithi097.stdout:Running transaction check 2024-03-11T21:18:51.593 INFO:teuthology.orchestra.run.smithi097.stdout:Transaction check succeeded. 2024-03-11T21:18:51.593 INFO:teuthology.orchestra.run.smithi097.stdout:Running transaction test 2024-03-11T21:18:51.649 INFO:teuthology.orchestra.run.smithi097.stdout:Transaction test succeeded. 2024-03-11T21:18:51.650 INFO:teuthology.orchestra.run.smithi097.stdout:Running transaction 2024-03-11T21:18:51.793 INFO:teuthology.orchestra.run.smithi097.stdout: Preparing : 1/1 2024-03-11T21:18:51.894 INFO:teuthology.orchestra.run.smithi097.stdout: Installing : librados-devel-2:19.0.0-1979.g2b0101eb.el9.x86_64 1/1 2024-03-11T21:18:52.396 INFO:teuthology.orchestra.run.smithi097.stdout: Running scriptlet: librados-devel-2:19.0.0-1979.g2b0101eb.el9.x86_64 1/1 2024-03-11T21:18:52.687 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : librados-devel-2:19.0.0-1979.g2b0101eb.el9.x86_64 1/1 2024-03-11T21:18:52.687 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-03-11T21:18:52.687 INFO:teuthology.orchestra.run.smithi097.stdout:Installed: 2024-03-11T21:18:52.687 INFO:teuthology.orchestra.run.smithi097.stdout: librados-devel-2:19.0.0-1979.g2b0101eb.el9.x86_64 2024-03-11T21:18:52.687 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-03-11T21:18:52.687 INFO:teuthology.orchestra.run.smithi097.stdout:Complete! 2024-03-11T21:18:52.955 DEBUG:teuthology.orchestra.run.smithi097:> sudo yum -y install libcephfs2 2024-03-11T21:18:53.532 INFO:teuthology.orchestra.run.smithi097.stdout:Last metadata expiration check: 0:01:37 ago on Mon 11 Mar 2024 09:17:16 PM UTC. 2024-03-11T21:18:53.632 INFO:teuthology.orchestra.run.smithi097.stdout:Package libcephfs2-2:19.0.0-1979.g2b0101eb.el9.x86_64 is already installed. 2024-03-11T21:18:53.685 INFO:teuthology.orchestra.run.smithi097.stdout:Dependencies resolved. 2024-03-11T21:18:53.686 INFO:teuthology.orchestra.run.smithi097.stdout:Nothing to do. 2024-03-11T21:18:53.686 INFO:teuthology.orchestra.run.smithi097.stdout:Complete! 2024-03-11T21:18:53.746 DEBUG:teuthology.orchestra.run.smithi097:> sudo yum -y install libcephfs-devel 2024-03-11T21:18:54.305 INFO:teuthology.orchestra.run.smithi097.stdout:Last metadata expiration check: 0:01:38 ago on Mon 11 Mar 2024 09:17:16 PM UTC. 2024-03-11T21:18:54.457 INFO:teuthology.orchestra.run.smithi097.stdout:Dependencies resolved. 2024-03-11T21:18:54.458 INFO:teuthology.orchestra.run.smithi097.stdout:================================================================================ 2024-03-11T21:18:54.458 INFO:teuthology.orchestra.run.smithi097.stdout: Package Arch Version Repo Size 2024-03-11T21:18:54.458 INFO:teuthology.orchestra.run.smithi097.stdout:================================================================================ 2024-03-11T21:18:54.458 INFO:teuthology.orchestra.run.smithi097.stdout:Installing: 2024-03-11T21:18:54.458 INFO:teuthology.orchestra.run.smithi097.stdout: libcephfs-devel x86_64 2:19.0.0-1979.g2b0101eb.el9 ceph 31 k 2024-03-11T21:18:54.459 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-03-11T21:18:54.459 INFO:teuthology.orchestra.run.smithi097.stdout:Transaction Summary 2024-03-11T21:18:54.459 INFO:teuthology.orchestra.run.smithi097.stdout:================================================================================ 2024-03-11T21:18:54.459 INFO:teuthology.orchestra.run.smithi097.stdout:Install 1 Package 2024-03-11T21:18:54.459 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-03-11T21:18:54.459 INFO:teuthology.orchestra.run.smithi097.stdout:Total download size: 31 k 2024-03-11T21:18:54.459 INFO:teuthology.orchestra.run.smithi097.stdout:Installed size: 140 k 2024-03-11T21:18:54.459 INFO:teuthology.orchestra.run.smithi097.stdout:Downloading Packages: 2024-03-11T21:18:54.624 INFO:teuthology.orchestra.run.smithi097.stdout:libcephfs-devel-19.0.0-1979.g2b0101eb.el9.x86_6 189 kB/s | 31 kB 00:00 2024-03-11T21:18:54.624 INFO:teuthology.orchestra.run.smithi097.stdout:-------------------------------------------------------------------------------- 2024-03-11T21:18:54.625 INFO:teuthology.orchestra.run.smithi097.stdout:Total 188 kB/s | 31 kB 00:00 2024-03-11T21:18:54.625 INFO:teuthology.orchestra.run.smithi097.stdout:Running transaction check 2024-03-11T21:18:54.628 INFO:teuthology.orchestra.run.smithi097.stdout:Transaction check succeeded. 2024-03-11T21:18:54.628 INFO:teuthology.orchestra.run.smithi097.stdout:Running transaction test 2024-03-11T21:18:54.660 INFO:teuthology.orchestra.run.smithi097.stdout:Transaction test succeeded. 2024-03-11T21:18:54.661 INFO:teuthology.orchestra.run.smithi097.stdout:Running transaction 2024-03-11T21:18:54.779 INFO:teuthology.orchestra.run.smithi097.stdout: Preparing : 1/1 2024-03-11T21:18:54.880 INFO:teuthology.orchestra.run.smithi097.stdout: Installing : libcephfs-devel-2:19.0.0-1979.g2b0101eb.el9.x86_64 1/1 2024-03-11T21:18:55.280 INFO:teuthology.orchestra.run.smithi097.stdout: Running scriptlet: libcephfs-devel-2:19.0.0-1979.g2b0101eb.el9.x86_64 1/1 2024-03-11T21:18:55.563 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : libcephfs-devel-2:19.0.0-1979.g2b0101eb.el9.x86_64 1/1 2024-03-11T21:18:55.563 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-03-11T21:18:55.563 INFO:teuthology.orchestra.run.smithi097.stdout:Installed: 2024-03-11T21:18:55.563 INFO:teuthology.orchestra.run.smithi097.stdout: libcephfs-devel-2:19.0.0-1979.g2b0101eb.el9.x86_64 2024-03-11T21:18:55.563 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-03-11T21:18:55.563 INFO:teuthology.orchestra.run.smithi097.stdout:Complete! 2024-03-11T21:18:55.733 DEBUG:teuthology.orchestra.run.smithi097:> sudo yum -y install librados2 2024-03-11T21:18:56.298 INFO:teuthology.orchestra.run.smithi097.stdout:Last metadata expiration check: 0:01:40 ago on Mon 11 Mar 2024 09:17:16 PM UTC. 2024-03-11T21:18:56.395 INFO:teuthology.orchestra.run.smithi097.stdout:Package librados2-2:19.0.0-1979.g2b0101eb.el9.x86_64 is already installed. 2024-03-11T21:18:56.447 INFO:teuthology.orchestra.run.smithi097.stdout:Dependencies resolved. 2024-03-11T21:18:56.448 INFO:teuthology.orchestra.run.smithi097.stdout:Nothing to do. 2024-03-11T21:18:56.449 INFO:teuthology.orchestra.run.smithi097.stdout:Complete! 2024-03-11T21:18:56.492 DEBUG:teuthology.orchestra.run.smithi097:> sudo yum -y install librbd1 2024-03-11T21:18:57.049 INFO:teuthology.orchestra.run.smithi097.stdout:Last metadata expiration check: 0:01:41 ago on Mon 11 Mar 2024 09:17:16 PM UTC. 2024-03-11T21:18:57.147 INFO:teuthology.orchestra.run.smithi097.stdout:Package librbd1-2:19.0.0-1979.g2b0101eb.el9.x86_64 is already installed. 2024-03-11T21:18:57.199 INFO:teuthology.orchestra.run.smithi097.stdout:Dependencies resolved. 2024-03-11T21:18:57.199 INFO:teuthology.orchestra.run.smithi097.stdout:Nothing to do. 2024-03-11T21:18:57.200 INFO:teuthology.orchestra.run.smithi097.stdout:Complete! 2024-03-11T21:18:57.257 DEBUG:teuthology.orchestra.run.smithi097:> sudo yum -y install python3-rados 2024-03-11T21:18:57.827 INFO:teuthology.orchestra.run.smithi097.stdout:Last metadata expiration check: 0:01:41 ago on Mon 11 Mar 2024 09:17:16 PM UTC. 2024-03-11T21:18:57.924 INFO:teuthology.orchestra.run.smithi097.stdout:Package python3-rados-2:19.0.0-1979.g2b0101eb.el9.x86_64 is already installed. 2024-03-11T21:18:57.976 INFO:teuthology.orchestra.run.smithi097.stdout:Dependencies resolved. 2024-03-11T21:18:57.977 INFO:teuthology.orchestra.run.smithi097.stdout:Nothing to do. 2024-03-11T21:18:57.977 INFO:teuthology.orchestra.run.smithi097.stdout:Complete! 2024-03-11T21:18:58.032 DEBUG:teuthology.orchestra.run.smithi097:> sudo yum -y install python3-rgw 2024-03-11T21:18:58.594 INFO:teuthology.orchestra.run.smithi097.stdout:Last metadata expiration check: 0:01:42 ago on Mon 11 Mar 2024 09:17:16 PM UTC. 2024-03-11T21:18:58.692 INFO:teuthology.orchestra.run.smithi097.stdout:Package python3-rgw-2:19.0.0-1979.g2b0101eb.el9.x86_64 is already installed. 2024-03-11T21:18:58.742 INFO:teuthology.orchestra.run.smithi097.stdout:Dependencies resolved. 2024-03-11T21:18:58.743 INFO:teuthology.orchestra.run.smithi097.stdout:Nothing to do. 2024-03-11T21:18:58.744 INFO:teuthology.orchestra.run.smithi097.stdout:Complete! 2024-03-11T21:18:58.803 DEBUG:teuthology.orchestra.run.smithi097:> sudo yum -y install python3-cephfs 2024-03-11T21:18:59.361 INFO:teuthology.orchestra.run.smithi097.stdout:Last metadata expiration check: 0:01:43 ago on Mon 11 Mar 2024 09:17:16 PM UTC. 2024-03-11T21:18:59.460 INFO:teuthology.orchestra.run.smithi097.stdout:Package python3-cephfs-2:19.0.0-1979.g2b0101eb.el9.x86_64 is already installed. 2024-03-11T21:18:59.512 INFO:teuthology.orchestra.run.smithi097.stdout:Dependencies resolved. 2024-03-11T21:18:59.513 INFO:teuthology.orchestra.run.smithi097.stdout:Nothing to do. 2024-03-11T21:18:59.513 INFO:teuthology.orchestra.run.smithi097.stdout:Complete! 2024-03-11T21:18:59.574 DEBUG:teuthology.orchestra.run.smithi097:> sudo yum -y install python3-rbd 2024-03-11T21:19:00.140 INFO:teuthology.orchestra.run.smithi097.stdout:Last metadata expiration check: 0:01:44 ago on Mon 11 Mar 2024 09:17:16 PM UTC. 2024-03-11T21:19:00.237 INFO:teuthology.orchestra.run.smithi097.stdout:Package python3-rbd-2:19.0.0-1979.g2b0101eb.el9.x86_64 is already installed. 2024-03-11T21:19:00.288 INFO:teuthology.orchestra.run.smithi097.stdout:Dependencies resolved. 2024-03-11T21:19:00.289 INFO:teuthology.orchestra.run.smithi097.stdout:Nothing to do. 2024-03-11T21:19:00.289 INFO:teuthology.orchestra.run.smithi097.stdout:Complete! 2024-03-11T21:19:00.346 DEBUG:teuthology.orchestra.run.smithi097:> sudo yum -y install rbd-fuse 2024-03-11T21:19:00.905 INFO:teuthology.orchestra.run.smithi097.stdout:Last metadata expiration check: 0:01:44 ago on Mon 11 Mar 2024 09:17:16 PM UTC. 2024-03-11T21:19:01.058 INFO:teuthology.orchestra.run.smithi097.stdout:Dependencies resolved. 2024-03-11T21:19:01.059 INFO:teuthology.orchestra.run.smithi097.stdout:================================================================================ 2024-03-11T21:19:01.059 INFO:teuthology.orchestra.run.smithi097.stdout: Package Architecture Version Repository Size 2024-03-11T21:19:01.060 INFO:teuthology.orchestra.run.smithi097.stdout:================================================================================ 2024-03-11T21:19:01.060 INFO:teuthology.orchestra.run.smithi097.stdout:Installing: 2024-03-11T21:19:01.060 INFO:teuthology.orchestra.run.smithi097.stdout: rbd-fuse x86_64 2:19.0.0-1979.g2b0101eb.el9 ceph 85 k 2024-03-11T21:19:01.060 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-03-11T21:19:01.060 INFO:teuthology.orchestra.run.smithi097.stdout:Transaction Summary 2024-03-11T21:19:01.060 INFO:teuthology.orchestra.run.smithi097.stdout:================================================================================ 2024-03-11T21:19:01.060 INFO:teuthology.orchestra.run.smithi097.stdout:Install 1 Package 2024-03-11T21:19:01.060 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-03-11T21:19:01.060 INFO:teuthology.orchestra.run.smithi097.stdout:Total download size: 85 k 2024-03-11T21:19:01.060 INFO:teuthology.orchestra.run.smithi097.stdout:Installed size: 231 k 2024-03-11T21:19:01.060 INFO:teuthology.orchestra.run.smithi097.stdout:Downloading Packages: 2024-03-11T21:19:01.528 INFO:teuthology.orchestra.run.smithi097.stdout:rbd-fuse-19.0.0-1979.g2b0101eb.el9.x86_64.rpm 182 kB/s | 85 kB 00:00 2024-03-11T21:19:01.529 INFO:teuthology.orchestra.run.smithi097.stdout:-------------------------------------------------------------------------------- 2024-03-11T21:19:01.529 INFO:teuthology.orchestra.run.smithi097.stdout:Total 182 kB/s | 85 kB 00:00 2024-03-11T21:19:01.529 INFO:teuthology.orchestra.run.smithi097.stdout:Running transaction check 2024-03-11T21:19:01.536 INFO:teuthology.orchestra.run.smithi097.stdout:Transaction check succeeded. 2024-03-11T21:19:01.536 INFO:teuthology.orchestra.run.smithi097.stdout:Running transaction test 2024-03-11T21:19:01.591 INFO:teuthology.orchestra.run.smithi097.stdout:Transaction test succeeded. 2024-03-11T21:19:01.591 INFO:teuthology.orchestra.run.smithi097.stdout:Running transaction 2024-03-11T21:19:01.726 INFO:teuthology.orchestra.run.smithi097.stdout: Preparing : 1/1 2024-03-11T21:19:01.826 INFO:teuthology.orchestra.run.smithi097.stdout: Installing : rbd-fuse-2:19.0.0-1979.g2b0101eb.el9.x86_64 1/1 2024-03-11T21:19:02.222 INFO:teuthology.orchestra.run.smithi097.stdout: Running scriptlet: rbd-fuse-2:19.0.0-1979.g2b0101eb.el9.x86_64 1/1 2024-03-11T21:19:02.532 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : rbd-fuse-2:19.0.0-1979.g2b0101eb.el9.x86_64 1/1 2024-03-11T21:19:02.532 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-03-11T21:19:02.532 INFO:teuthology.orchestra.run.smithi097.stdout:Installed: 2024-03-11T21:19:02.532 INFO:teuthology.orchestra.run.smithi097.stdout: rbd-fuse-2:19.0.0-1979.g2b0101eb.el9.x86_64 2024-03-11T21:19:02.533 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-03-11T21:19:02.533 INFO:teuthology.orchestra.run.smithi097.stdout:Complete! 2024-03-11T21:19:02.753 DEBUG:teuthology.orchestra.run.smithi097:> sudo yum -y install rbd-mirror 2024-03-11T21:19:03.331 INFO:teuthology.orchestra.run.smithi097.stdout:Last metadata expiration check: 0:01:47 ago on Mon 11 Mar 2024 09:17:16 PM UTC. 2024-03-11T21:19:03.485 INFO:teuthology.orchestra.run.smithi097.stdout:Dependencies resolved. 2024-03-11T21:19:03.486 INFO:teuthology.orchestra.run.smithi097.stdout:================================================================================ 2024-03-11T21:19:03.486 INFO:teuthology.orchestra.run.smithi097.stdout: Package Arch Version Repo Size 2024-03-11T21:19:03.486 INFO:teuthology.orchestra.run.smithi097.stdout:================================================================================ 2024-03-11T21:19:03.486 INFO:teuthology.orchestra.run.smithi097.stdout:Installing: 2024-03-11T21:19:03.486 INFO:teuthology.orchestra.run.smithi097.stdout: rbd-mirror x86_64 2:19.0.0-1979.g2b0101eb.el9 ceph 3.1 M 2024-03-11T21:19:03.486 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-03-11T21:19:03.486 INFO:teuthology.orchestra.run.smithi097.stdout:Transaction Summary 2024-03-11T21:19:03.486 INFO:teuthology.orchestra.run.smithi097.stdout:================================================================================ 2024-03-11T21:19:03.486 INFO:teuthology.orchestra.run.smithi097.stdout:Install 1 Package 2024-03-11T21:19:03.486 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-03-11T21:19:03.487 INFO:teuthology.orchestra.run.smithi097.stdout:Total download size: 3.1 M 2024-03-11T21:19:03.487 INFO:teuthology.orchestra.run.smithi097.stdout:Installed size: 13 M 2024-03-11T21:19:03.487 INFO:teuthology.orchestra.run.smithi097.stdout:Downloading Packages: 2024-03-11T21:19:05.447 INFO:teuthology.orchestra.run.smithi097.stdout:rbd-mirror-19.0.0-1979.g2b0101eb.el9.x86_64.rpm 1.6 MB/s | 3.1 MB 00:01 2024-03-11T21:19:05.448 INFO:teuthology.orchestra.run.smithi097.stdout:-------------------------------------------------------------------------------- 2024-03-11T21:19:05.448 INFO:teuthology.orchestra.run.smithi097.stdout:Total 1.6 MB/s | 3.1 MB 00:01 2024-03-11T21:19:05.449 INFO:teuthology.orchestra.run.smithi097.stdout:Running transaction check 2024-03-11T21:19:05.460 INFO:teuthology.orchestra.run.smithi097.stdout:Transaction check succeeded. 2024-03-11T21:19:05.460 INFO:teuthology.orchestra.run.smithi097.stdout:Running transaction test 2024-03-11T21:19:05.536 INFO:teuthology.orchestra.run.smithi097.stdout:Transaction test succeeded. 2024-03-11T21:19:05.536 INFO:teuthology.orchestra.run.smithi097.stdout:Running transaction 2024-03-11T21:19:05.963 INFO:teuthology.orchestra.run.smithi097.stdout: Preparing : 1/1 2024-03-11T21:19:06.015 INFO:teuthology.orchestra.run.smithi097.stdout: Installing : rbd-mirror-2:19.0.0-1979.g2b0101eb.el9.x86_64 1/1 2024-03-11T21:19:06.039 INFO:teuthology.orchestra.run.smithi097.stdout: Running scriptlet: rbd-mirror-2:19.0.0-1979.g2b0101eb.el9.x86_64 1/1 2024-03-11T21:19:06.040 INFO:teuthology.orchestra.run.smithi097.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-11T21:19:06.040 INFO:teuthology.orchestra.run.smithi097.stdout:Invalid unit name "ceph-rbd-mirror@*.service" escaped as "ceph-rbd-mirror@\x2a.service". 2024-03-11T21:19:06.040 INFO:teuthology.orchestra.run.smithi097.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-03-11T21:19:06.040 INFO:teuthology.orchestra.run.smithi097.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-03-11T21:19:06.040 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-03-11T21:19:07.216 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : rbd-mirror-2:19.0.0-1979.g2b0101eb.el9.x86_64 1/1 2024-03-11T21:19:07.216 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-03-11T21:19:07.216 INFO:teuthology.orchestra.run.smithi097.stdout:Installed: 2024-03-11T21:19:07.216 INFO:teuthology.orchestra.run.smithi097.stdout: rbd-mirror-2:19.0.0-1979.g2b0101eb.el9.x86_64 2024-03-11T21:19:07.216 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-03-11T21:19:07.216 INFO:teuthology.orchestra.run.smithi097.stdout:Complete! 2024-03-11T21:19:07.462 DEBUG:teuthology.orchestra.run.smithi097:> sudo yum -y install rbd-nbd 2024-03-11T21:19:08.037 INFO:teuthology.orchestra.run.smithi097.stdout:Last metadata expiration check: 0:01:52 ago on Mon 11 Mar 2024 09:17:16 PM UTC. 2024-03-11T21:19:08.183 INFO:teuthology.orchestra.run.smithi097.stdout:Dependencies resolved. 2024-03-11T21:19:08.184 INFO:teuthology.orchestra.run.smithi097.stdout:================================================================================ 2024-03-11T21:19:08.184 INFO:teuthology.orchestra.run.smithi097.stdout: Package Architecture Version Repository Size 2024-03-11T21:19:08.184 INFO:teuthology.orchestra.run.smithi097.stdout:================================================================================ 2024-03-11T21:19:08.184 INFO:teuthology.orchestra.run.smithi097.stdout:Installing: 2024-03-11T21:19:08.184 INFO:teuthology.orchestra.run.smithi097.stdout: rbd-nbd x86_64 2:19.0.0-1979.g2b0101eb.el9 ceph 170 k 2024-03-11T21:19:08.184 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-03-11T21:19:08.184 INFO:teuthology.orchestra.run.smithi097.stdout:Transaction Summary 2024-03-11T21:19:08.184 INFO:teuthology.orchestra.run.smithi097.stdout:================================================================================ 2024-03-11T21:19:08.184 INFO:teuthology.orchestra.run.smithi097.stdout:Install 1 Package 2024-03-11T21:19:08.184 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-03-11T21:19:08.185 INFO:teuthology.orchestra.run.smithi097.stdout:Total download size: 170 k 2024-03-11T21:19:08.185 INFO:teuthology.orchestra.run.smithi097.stdout:Installed size: 490 k 2024-03-11T21:19:08.185 INFO:teuthology.orchestra.run.smithi097.stdout:Downloading Packages: 2024-03-11T21:19:08.401 INFO:teuthology.orchestra.run.smithi097.stdout:rbd-nbd-19.0.0-1979.g2b0101eb.el9.x86_64.rpm 786 kB/s | 170 kB 00:00 2024-03-11T21:19:08.402 INFO:teuthology.orchestra.run.smithi097.stdout:-------------------------------------------------------------------------------- 2024-03-11T21:19:08.402 INFO:teuthology.orchestra.run.smithi097.stdout:Total 782 kB/s | 170 kB 00:00 2024-03-11T21:19:08.402 INFO:teuthology.orchestra.run.smithi097.stdout:Running transaction check 2024-03-11T21:19:08.410 INFO:teuthology.orchestra.run.smithi097.stdout:Transaction check succeeded. 2024-03-11T21:19:08.410 INFO:teuthology.orchestra.run.smithi097.stdout:Running transaction test 2024-03-11T21:19:08.465 INFO:teuthology.orchestra.run.smithi097.stdout:Transaction test succeeded. 2024-03-11T21:19:08.466 INFO:teuthology.orchestra.run.smithi097.stdout:Running transaction 2024-03-11T21:19:08.613 INFO:teuthology.orchestra.run.smithi097.stdout: Preparing : 1/1 2024-03-11T21:19:08.724 INFO:teuthology.orchestra.run.smithi097.stdout: Installing : rbd-nbd-2:19.0.0-1979.g2b0101eb.el9.x86_64 1/1 2024-03-11T21:19:09.136 INFO:teuthology.orchestra.run.smithi097.stdout: Running scriptlet: rbd-nbd-2:19.0.0-1979.g2b0101eb.el9.x86_64 1/1 2024-03-11T21:19:09.440 INFO:teuthology.orchestra.run.smithi097.stdout: Verifying : rbd-nbd-2:19.0.0-1979.g2b0101eb.el9.x86_64 1/1 2024-03-11T21:19:09.441 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-03-11T21:19:09.441 INFO:teuthology.orchestra.run.smithi097.stdout:Installed: 2024-03-11T21:19:09.441 INFO:teuthology.orchestra.run.smithi097.stdout: rbd-nbd-2:19.0.0-1979.g2b0101eb.el9.x86_64 2024-03-11T21:19:09.441 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-03-11T21:19:09.441 INFO:teuthology.orchestra.run.smithi097.stdout:Complete! 2024-03-11T21:19:09.698 DEBUG:teuthology.parallel:result is None 2024-03-11T21:19:09.698 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=2b0101eb74d60610d544cb8a473e23a35ac1cc0b 2024-03-11T21:19:09.841 DEBUG:teuthology.orchestra.run.smithi092:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2024-03-11T21:19:09.867 INFO:teuthology.orchestra.run.smithi092.stdout:19.0.0-1979.g2b0101eb.el9 2024-03-11T21:19:09.867 INFO:teuthology.packaging:The installed version of ceph is 19.0.0-1979.g2b0101eb.el9 2024-03-11T21:19:09.868 INFO:teuthology.task.install:The correct ceph version 19.0.0-1979.g2b0101eb is installed. 2024-03-11T21:19:09.869 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=2b0101eb74d60610d544cb8a473e23a35ac1cc0b 2024-03-11T21:19:10.176 DEBUG:teuthology.orchestra.run.smithi097:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2024-03-11T21:19:10.199 INFO:teuthology.orchestra.run.smithi097.stdout:19.0.0-1979.g2b0101eb.el9 2024-03-11T21:19:10.199 INFO:teuthology.packaging:The installed version of ceph is 19.0.0-1979.g2b0101eb.el9 2024-03-11T21:19:10.199 INFO:teuthology.task.install:The correct ceph version 19.0.0-1979.g2b0101eb is installed. 2024-03-11T21:19:10.201 INFO:teuthology.task.install.util:Shipping valgrind.supp... 2024-03-11T21:19:10.201 DEBUG:teuthology.orchestra.run.smithi092:> set -ex 2024-03-11T21:19:10.201 DEBUG:teuthology.orchestra.run.smithi092:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-03-11T21:19:10.234 DEBUG:teuthology.orchestra.run.smithi097:> set -ex 2024-03-11T21:19:10.234 DEBUG:teuthology.orchestra.run.smithi097:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-03-11T21:19:10.269 INFO:teuthology.task.install.util:Shipping 'daemon-helper'... 2024-03-11T21:19:10.269 DEBUG:teuthology.orchestra.run.smithi092:> set -ex 2024-03-11T21:19:10.269 DEBUG:teuthology.orchestra.run.smithi092:> sudo dd of=/usr/bin/daemon-helper 2024-03-11T21:19:10.305 DEBUG:teuthology.orchestra.run.smithi092:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-03-11T21:19:10.375 DEBUG:teuthology.orchestra.run.smithi097:> set -ex 2024-03-11T21:19:10.375 DEBUG:teuthology.orchestra.run.smithi097:> sudo dd of=/usr/bin/daemon-helper 2024-03-11T21:19:10.403 DEBUG:teuthology.orchestra.run.smithi097:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-03-11T21:19:10.471 INFO:teuthology.task.install.util:Shipping 'adjust-ulimits'... 2024-03-11T21:19:10.471 DEBUG:teuthology.orchestra.run.smithi092:> set -ex 2024-03-11T21:19:10.472 DEBUG:teuthology.orchestra.run.smithi092:> sudo dd of=/usr/bin/adjust-ulimits 2024-03-11T21:19:10.502 DEBUG:teuthology.orchestra.run.smithi092:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-03-11T21:19:10.571 DEBUG:teuthology.orchestra.run.smithi097:> set -ex 2024-03-11T21:19:10.571 DEBUG:teuthology.orchestra.run.smithi097:> sudo dd of=/usr/bin/adjust-ulimits 2024-03-11T21:19:10.599 DEBUG:teuthology.orchestra.run.smithi097:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-03-11T21:19:10.667 INFO:teuthology.task.install.util:Shipping 'stdin-killer'... 2024-03-11T21:19:10.667 DEBUG:teuthology.orchestra.run.smithi092:> set -ex 2024-03-11T21:19:10.668 DEBUG:teuthology.orchestra.run.smithi092:> sudo dd of=/usr/bin/stdin-killer 2024-03-11T21:19:10.694 DEBUG:teuthology.orchestra.run.smithi092:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-03-11T21:19:10.761 DEBUG:teuthology.orchestra.run.smithi097:> set -ex 2024-03-11T21:19:10.761 DEBUG:teuthology.orchestra.run.smithi097:> sudo dd of=/usr/bin/stdin-killer 2024-03-11T21:19:10.789 DEBUG:teuthology.orchestra.run.smithi097:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-03-11T21:19:10.857 INFO:teuthology.run_tasks:Running task cephadm... 2024-03-11T21:19:10.965 INFO:tasks.cephadm:Config: {'conf': {'mgr': {'debug mgr': 20, 'debug ms': 1}, 'client.0': {'debug ms': 1, 'debug objecter': 20, 'debug rados': 20}, 'global': {'ms bind msgr2': False, 'ms type': 'async'}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon min osdmap epochs': 50, 'mon osdmap full prune interval': 2, 'mon osdmap full prune min': 15, 'mon osdmap full prune txsize': 2, 'paxos service trim min': 10}, 'osd': {'debug ms': 1, 'debug osd': 20, 'osd debug reject backfill probability': 0.3, 'osd delete sleep': 1, 'osd max backfills': 3, 'osd scrub max interval': 120, 'osd scrub min interval': 60, 'osd shutdown pgref assert': True, 'osd snap trim sleep': 2}}, 'flavor': 'default', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', 'but it is still running', 'objects unfound and apparently lost', '\\(POOL_APP_NOT_ENABLED\\)'], 'mon_bind_msgr2': False, 'sha1': '2b0101eb74d60610d544cb8a473e23a35ac1cc0b', 'cephadm_mode': 'root'} 2024-03-11T21:19:10.965 INFO:tasks.cephadm:Cluster image is quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b 2024-03-11T21:19:10.966 INFO:tasks.cephadm:Cluster fsid is 00db0016-dfed-11ee-95c8-87774f69a715 2024-03-11T21:19:10.966 INFO:tasks.cephadm:Choosing monitor IPs and ports... 2024-03-11T21:19:10.966 INFO:tasks.cephadm:Monitor IPs: {'mon.a': '[v1:172.21.15.92:6789]', 'mon.c': '[v1:172.21.15.92:6790]', 'mon.b': '[v1:172.21.15.97:6789]'} 2024-03-11T21:19:10.966 INFO:tasks.cephadm:First mon is mon.a on smithi092 2024-03-11T21:19:10.966 INFO:tasks.cephadm:First mgr is y 2024-03-11T21:19:10.966 INFO:tasks.cephadm:Normalizing hostnames... 2024-03-11T21:19:10.966 DEBUG:teuthology.orchestra.run.smithi092:> sudo hostname $(hostname -s) 2024-03-11T21:19:10.994 DEBUG:teuthology.orchestra.run.smithi097:> sudo hostname $(hostname -s) 2024-03-11T21:19:11.022 INFO:tasks.cephadm:Downloading "compiled" cephadm from cachra 2024-03-11T21:19:11.022 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=2b0101eb74d60610d544cb8a473e23a35ac1cc0b 2024-03-11T21:19:11.175 INFO:tasks.cephadm:builder_project result: [{'status': 'ready', 'sha1': '2b0101eb74d60610d544cb8a473e23a35ac1cc0b', 'extra': {'build_url': 'https://jenkins.ceph.com/job/ceph-dev-build/ARCH=x86_64,AVAILABLE_ARCH=x86_64,AVAILABLE_DIST=centos9,DIST=centos9,MACHINE_SIZE=gigantic/54565/', 'root_build_cause': 'SCMTRIGGER', 'version': '19.0.0-1979-g2b0101eb', 'node_name': '172.21.2.1+braggi01', 'job_name': 'ceph-dev-build/ARCH=x86_64,AVAILABLE_ARCH=x86_64,AVAILABLE_DIST=centos9,DIST=centos9,MACHINE_SIZE=gigantic', 'package_manager_version': '19.0.0-1979.g2b0101eb'}, 'url': 'https://3.chacra.ceph.com/r/ceph/main/2b0101eb74d60610d544cb8a473e23a35ac1cc0b/centos/9/flavors/default/', 'distro_codename': None, 'modified': '2024-03-11 00:47:00.272739', 'distro_version': '9', 'project': 'ceph', 'flavor': 'default', 'ref': 'main', 'chacra_url': 'https://3.chacra.ceph.com/repos/ceph/main/2b0101eb74d60610d544cb8a473e23a35ac1cc0b/centos/9/flavors/default/', 'archs': ['x86_64', 'source'], 'distro': 'centos'}] 2024-03-11T21:19:11.309 INFO:tasks.util.chacra:got chacra host 3.chacra.ceph.com, ref main, sha1 2b0101eb74d60610d544cb8a473e23a35ac1cc0b from https://shaman.ceph.com/api/search/?project=ceph&distros=centos%2F9%2Fx86_64&flavor=default&sha1=2b0101eb74d60610d544cb8a473e23a35ac1cc0b 2024-03-11T21:19:11.311 INFO:tasks.cephadm:Discovered cachra url: https://3.chacra.ceph.com/binaries/ceph/main/2b0101eb74d60610d544cb8a473e23a35ac1cc0b/centos/9/x86_64/flavors/default/cephadm 2024-03-11T21:19:11.311 DEBUG:teuthology.orchestra.run.smithi092:> curl --silent -L https://3.chacra.ceph.com/binaries/ceph/main/2b0101eb74d60610d544cb8a473e23a35ac1cc0b/centos/9/x86_64/flavors/default/cephadm > /home/ubuntu/cephtest/cephadm && ls -l /home/ubuntu/cephtest/cephadm 2024-03-11T21:19:11.794 INFO:teuthology.orchestra.run.smithi092.stdout:-rw-r--r--. 1 ubuntu ubuntu 767461 Mar 11 21:19 /home/ubuntu/cephtest/cephadm 2024-03-11T21:19:11.794 DEBUG:teuthology.orchestra.run.smithi097:> curl --silent -L https://3.chacra.ceph.com/binaries/ceph/main/2b0101eb74d60610d544cb8a473e23a35ac1cc0b/centos/9/x86_64/flavors/default/cephadm > /home/ubuntu/cephtest/cephadm && ls -l /home/ubuntu/cephtest/cephadm 2024-03-11T21:19:12.812 INFO:teuthology.orchestra.run.smithi097.stdout:-rw-r--r--. 1 ubuntu ubuntu 767461 Mar 11 21:19 /home/ubuntu/cephtest/cephadm 2024-03-11T21:19:12.812 DEBUG:teuthology.orchestra.run.smithi092:> test -s /home/ubuntu/cephtest/cephadm && test $(stat -c%s /home/ubuntu/cephtest/cephadm) -gt 1000 && chmod +x /home/ubuntu/cephtest/cephadm 2024-03-11T21:19:12.831 DEBUG:teuthology.orchestra.run.smithi097:> test -s /home/ubuntu/cephtest/cephadm && test $(stat -c%s /home/ubuntu/cephtest/cephadm) -gt 1000 && chmod +x /home/ubuntu/cephtest/cephadm 2024-03-11T21:19:12.862 INFO:tasks.cephadm:Pulling image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b on all hosts... 2024-03-11T21:19:12.862 DEBUG:teuthology.orchestra.run.smithi092:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b pull 2024-03-11T21:19:12.873 DEBUG:teuthology.orchestra.run.smithi097:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b pull 2024-03-11T21:19:13.649 INFO:teuthology.orchestra.run.smithi092.stderr:Pulling container image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b... 2024-03-11T21:19:13.837 INFO:teuthology.orchestra.run.smithi097.stderr:Pulling container image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b... 2024-03-11T21:19:55.367 INFO:teuthology.orchestra.run.smithi097.stdout:{ 2024-03-11T21:19:55.367 INFO:teuthology.orchestra.run.smithi097.stdout: "ceph_version": "ceph version 19.0.0-1979-g2b0101eb (2b0101eb74d60610d544cb8a473e23a35ac1cc0b) squid (dev)", 2024-03-11T21:19:55.367 INFO:teuthology.orchestra.run.smithi097.stdout: "image_id": "4133f8bc41bb201212a28168427a2edb1f6f3795e694bf61031ea9e2f3ff5e87", 2024-03-11T21:19:55.367 INFO:teuthology.orchestra.run.smithi097.stdout: "repo_digests": [ 2024-03-11T21:19:55.368 INFO:teuthology.orchestra.run.smithi097.stdout: "quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:8fe9635f3beb0247496d7e321f3068bd10c333c433ef4824d7e1c1a7d3d78379" 2024-03-11T21:19:55.368 INFO:teuthology.orchestra.run.smithi097.stdout: ] 2024-03-11T21:19:55.368 INFO:teuthology.orchestra.run.smithi097.stdout:} 2024-03-11T21:20:01.710 INFO:teuthology.orchestra.run.smithi092.stdout:{ 2024-03-11T21:20:01.710 INFO:teuthology.orchestra.run.smithi092.stdout: "ceph_version": "ceph version 19.0.0-1979-g2b0101eb (2b0101eb74d60610d544cb8a473e23a35ac1cc0b) squid (dev)", 2024-03-11T21:20:01.710 INFO:teuthology.orchestra.run.smithi092.stdout: "image_id": "4133f8bc41bb201212a28168427a2edb1f6f3795e694bf61031ea9e2f3ff5e87", 2024-03-11T21:20:01.710 INFO:teuthology.orchestra.run.smithi092.stdout: "repo_digests": [ 2024-03-11T21:20:01.710 INFO:teuthology.orchestra.run.smithi092.stdout: "quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:8fe9635f3beb0247496d7e321f3068bd10c333c433ef4824d7e1c1a7d3d78379" 2024-03-11T21:20:01.710 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-03-11T21:20:01.710 INFO:teuthology.orchestra.run.smithi092.stdout:} 2024-03-11T21:20:01.735 DEBUG:teuthology.orchestra.run.smithi092:> sudo mkdir -p /etc/ceph 2024-03-11T21:20:01.792 DEBUG:teuthology.orchestra.run.smithi097:> sudo mkdir -p /etc/ceph 2024-03-11T21:20:01.835 DEBUG:teuthology.orchestra.run.smithi092:> sudo chmod 777 /etc/ceph 2024-03-11T21:20:01.862 DEBUG:teuthology.orchestra.run.smithi097:> sudo chmod 777 /etc/ceph 2024-03-11T21:20:01.904 INFO:tasks.cephadm:Writing seed config... 2024-03-11T21:20:01.906 INFO:tasks.cephadm: override: [mgr] debug mgr = 20 2024-03-11T21:20:01.906 INFO:tasks.cephadm: override: [mgr] debug ms = 1 2024-03-11T21:20:01.906 INFO:tasks.cephadm: override: [client.0] debug ms = 1 2024-03-11T21:20:01.906 INFO:tasks.cephadm: override: [client.0] debug objecter = 20 2024-03-11T21:20:01.906 INFO:tasks.cephadm: override: [client.0] debug rados = 20 2024-03-11T21:20:01.906 INFO:tasks.cephadm: override: [global] ms bind msgr2 = False 2024-03-11T21:20:01.906 INFO:tasks.cephadm: override: [global] ms type = async 2024-03-11T21:20:01.906 INFO:tasks.cephadm: override: [mon] debug mon = 20 2024-03-11T21:20:01.906 INFO:tasks.cephadm: override: [mon] debug ms = 1 2024-03-11T21:20:01.906 INFO:tasks.cephadm: override: [mon] debug paxos = 20 2024-03-11T21:20:01.906 INFO:tasks.cephadm: override: [mon] mon min osdmap epochs = 50 2024-03-11T21:20:01.906 INFO:tasks.cephadm: override: [mon] mon osdmap full prune interval = 2 2024-03-11T21:20:01.907 INFO:tasks.cephadm: override: [mon] mon osdmap full prune min = 15 2024-03-11T21:20:01.907 INFO:tasks.cephadm: override: [mon] mon osdmap full prune txsize = 2 2024-03-11T21:20:01.907 INFO:tasks.cephadm: override: [mon] paxos service trim min = 10 2024-03-11T21:20:01.907 INFO:tasks.cephadm: override: [osd] debug ms = 1 2024-03-11T21:20:01.907 INFO:tasks.cephadm: override: [osd] debug osd = 20 2024-03-11T21:20:01.907 INFO:tasks.cephadm: override: [osd] osd debug reject backfill probability = 0.3 2024-03-11T21:20:01.907 INFO:tasks.cephadm: override: [osd] osd delete sleep = 1 2024-03-11T21:20:01.907 INFO:tasks.cephadm: override: [osd] osd max backfills = 3 2024-03-11T21:20:01.907 INFO:tasks.cephadm: override: [osd] osd scrub max interval = 120 2024-03-11T21:20:01.907 INFO:tasks.cephadm: override: [osd] osd scrub min interval = 60 2024-03-11T21:20:01.907 INFO:tasks.cephadm: override: [osd] osd shutdown pgref assert = True 2024-03-11T21:20:01.907 INFO:tasks.cephadm: override: [osd] osd snap trim sleep = 2 2024-03-11T21:20:01.908 DEBUG:teuthology.orchestra.run.smithi092:> set -ex 2024-03-11T21:20:01.908 DEBUG:teuthology.orchestra.run.smithi092:> dd of=/home/ubuntu/cephtest/seed.ceph.conf 2024-03-11T21:20:01.923 DEBUG:tasks.cephadm:Final config: [global] # make logging friendly to teuthology log_to_file = true log_to_stderr = false log to journald = false mon cluster log to file = true mon cluster log file level = debug mon clock drift allowed = 1.000 # replicate across OSDs, not hosts osd crush chooseleaf type = 0 #osd pool default size = 2 osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd # enable some debugging auth debug = true ms die on old message = true ms die on bug = true debug asserts on shutdown = true # adjust warnings mon max pg per osd = 10000# >= luminous mon pg warn max object skew = 0 mon osd allow primary affinity = true mon osd allow pg remap = true mon warn on legacy crush tunables = false mon warn on crush straw calc version zero = false mon warn on no sortbitwise = false mon warn on osd down out interval zero = false mon warn on too few osds = false mon_warn_on_pool_pg_num_not_power_of_two = false # disable pg_autoscaler by default for new pools osd_pool_default_pg_autoscale_mode = off # tests delete pools mon allow pool delete = true fsid = 00db0016-dfed-11ee-95c8-87774f69a715 ms bind msgr2 = False ms type = async [osd] osd scrub load threshold = 5.0 osd scrub max interval = 120 osd mclock profile = high_recovery_ops osd recover clone overlap = true osd recovery max chunk = 1048576 osd deep scrub update digest min age = 30 osd map max advance = 10 osd memory target autotune = true # debugging osd debug shutdown = true osd debug op order = true osd debug verify stray on activate = true osd debug pg log writeout = true osd debug verify cached snaps = true osd debug verify missing on start = true osd debug misdirected ops = true osd op queue = debug_random osd op queue cut off = debug_random osd shutdown pgref assert = True bdev debug aio = true osd sloppy crc = true debug ms = 1 debug osd = 20 osd debug reject backfill probability = 0.3 osd delete sleep = 1 osd max backfills = 3 osd scrub min interval = 60 osd snap trim sleep = 2 [mgr] mon reweight min pgs per osd = 4 mon reweight min bytes per osd = 10 mgr/telemetry/nag = false debug mgr = 20 debug ms = 1 [mon] mon data avail warn = 5 mon mgr mkfs grace = 240 mon reweight min pgs per osd = 4 mon osd reporter subtree level = osd mon osd prime pg temp = true mon reweight min bytes per osd = 10 # rotate auth tickets quickly to exercise renewal paths auth mon ticket ttl = 660# 11m auth service ticket ttl = 240# 4m # don't complain about global id reclaim mon_warn_on_insecure_global_id_reclaim = false mon_warn_on_insecure_global_id_reclaim_allowed = false debug mon = 20 debug ms = 1 debug paxos = 20 mon min osdmap epochs = 50 mon osdmap full prune interval = 2 mon osdmap full prune min = 15 mon osdmap full prune txsize = 2 paxos service trim min = 10 [client.rgw] rgw cache enabled = true rgw enable ops log = true rgw enable usage log = true [client.0] debug ms = 1 debug objecter = 20 debug rados = 20 2024-03-11T21:20:01.923 DEBUG:teuthology.orchestra.run.smithi092:mon.a> sudo journalctl -f -n 0 -u ceph-00db0016-dfed-11ee-95c8-87774f69a715@mon.a.service 2024-03-11T21:20:01.965 DEBUG:teuthology.orchestra.run.smithi092:mgr.y> sudo journalctl -f -n 0 -u ceph-00db0016-dfed-11ee-95c8-87774f69a715@mgr.y.service 2024-03-11T21:20:02.008 INFO:tasks.cephadm:Bootstrapping... 2024-03-11T21:20:02.008 DEBUG:teuthology.orchestra.run.smithi092:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b -v bootstrap --fsid 00db0016-dfed-11ee-95c8-87774f69a715 --config /home/ubuntu/cephtest/seed.ceph.conf --output-config /etc/ceph/ceph.conf --output-keyring /etc/ceph/ceph.client.admin.keyring --output-pub-ssh-key /home/ubuntu/cephtest/ceph.pub --mon-id a --mgr-id y --orphan-initial-daemons --skip-monitoring-stack --mon-addrv '[v1:172.21.15.92:6789]' --skip-admin-label && sudo chmod +r /etc/ceph/ceph.client.admin.keyring 2024-03-11T21:20:02.200 INFO:teuthology.orchestra.run.smithi092.stdout:-------------------------------------------------------------------------------- 2024-03-11T21:20:02.200 INFO:teuthology.orchestra.run.smithi092.stdout:cephadm ['--image', 'quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b', '-v', 'bootstrap', '--fsid', '00db0016-dfed-11ee-95c8-87774f69a715', '--config', '/home/ubuntu/cephtest/seed.ceph.conf', '--output-config', '/etc/ceph/ceph.conf', '--output-keyring', '/etc/ceph/ceph.client.admin.keyring', '--output-pub-ssh-key', '/home/ubuntu/cephtest/ceph.pub', '--mon-id', 'a', '--mgr-id', 'y', '--orphan-initial-daemons', '--skip-monitoring-stack', '--mon-addrv', '[v1:172.21.15.92:6789]', '--skip-admin-label'] 2024-03-11T21:20:02.224 INFO:teuthology.orchestra.run.smithi092.stdout:/bin/podman: stdout 4.9.3-dev 2024-03-11T21:20:02.224 INFO:teuthology.orchestra.run.smithi092.stderr:Specifying an fsid for your cluster offers no advantages and may increase the likelihood of fsid conflicts. 2024-03-11T21:20:02.225 INFO:teuthology.orchestra.run.smithi092.stdout:Verifying podman|docker is present... 2024-03-11T21:20:02.247 INFO:teuthology.orchestra.run.smithi092.stdout:/bin/podman: stdout 4.9.3-dev 2024-03-11T21:20:02.247 INFO:teuthology.orchestra.run.smithi092.stdout:Verifying lvm2 is present... 2024-03-11T21:20:02.247 INFO:teuthology.orchestra.run.smithi092.stdout:Verifying time synchronization is in place... 2024-03-11T21:20:02.255 INFO:teuthology.orchestra.run.smithi092.stdout:Non-zero exit code 1 from systemctl is-enabled chrony.service 2024-03-11T21:20:02.255 INFO:teuthology.orchestra.run.smithi092.stdout:systemctl: stderr Failed to get unit file state for chrony.service: No such file or directory 2024-03-11T21:20:02.261 INFO:teuthology.orchestra.run.smithi092.stdout:Non-zero exit code 3 from systemctl is-active chrony.service 2024-03-11T21:20:02.261 INFO:teuthology.orchestra.run.smithi092.stdout:systemctl: stdout inactive 2024-03-11T21:20:02.267 INFO:teuthology.orchestra.run.smithi092.stdout:systemctl: stdout enabled 2024-03-11T21:20:02.273 INFO:teuthology.orchestra.run.smithi092.stdout:systemctl: stdout active 2024-03-11T21:20:02.273 INFO:teuthology.orchestra.run.smithi092.stdout:Unit chronyd.service is enabled and running 2024-03-11T21:20:02.273 INFO:teuthology.orchestra.run.smithi092.stdout:Repeating the final host check... 2024-03-11T21:20:02.297 INFO:teuthology.orchestra.run.smithi092.stdout:/bin/podman: stdout 4.9.3-dev 2024-03-11T21:20:02.297 INFO:teuthology.orchestra.run.smithi092.stdout:podman (/bin/podman) version 4.9.3 is present 2024-03-11T21:20:02.297 INFO:teuthology.orchestra.run.smithi092.stdout:systemctl is present 2024-03-11T21:20:02.297 INFO:teuthology.orchestra.run.smithi092.stdout:lvcreate is present 2024-03-11T21:20:02.303 INFO:teuthology.orchestra.run.smithi092.stdout:Non-zero exit code 1 from systemctl is-enabled chrony.service 2024-03-11T21:20:02.303 INFO:teuthology.orchestra.run.smithi092.stdout:systemctl: stderr Failed to get unit file state for chrony.service: No such file or directory 2024-03-11T21:20:02.309 INFO:teuthology.orchestra.run.smithi092.stdout:Non-zero exit code 3 from systemctl is-active chrony.service 2024-03-11T21:20:02.309 INFO:teuthology.orchestra.run.smithi092.stdout:systemctl: stdout inactive 2024-03-11T21:20:02.315 INFO:teuthology.orchestra.run.smithi092.stdout:systemctl: stdout enabled 2024-03-11T21:20:02.321 INFO:teuthology.orchestra.run.smithi092.stdout:systemctl: stdout active 2024-03-11T21:20:02.321 INFO:teuthology.orchestra.run.smithi092.stdout:Unit chronyd.service is enabled and running 2024-03-11T21:20:02.321 INFO:teuthology.orchestra.run.smithi092.stdout:Host looks OK 2024-03-11T21:20:02.322 INFO:teuthology.orchestra.run.smithi092.stdout:Cluster fsid: 00db0016-dfed-11ee-95c8-87774f69a715 2024-03-11T21:20:02.322 INFO:teuthology.orchestra.run.smithi092.stdout:Acquiring lock 140555131248544 on /run/cephadm/00db0016-dfed-11ee-95c8-87774f69a715.lock 2024-03-11T21:20:02.322 INFO:teuthology.orchestra.run.smithi092.stdout:Lock 140555131248544 acquired on /run/cephadm/00db0016-dfed-11ee-95c8-87774f69a715.lock 2024-03-11T21:20:02.322 INFO:teuthology.orchestra.run.smithi092.stdout:Verifying IP 172.21.15.92 port 6789 ... 2024-03-11T21:20:02.322 INFO:teuthology.orchestra.run.smithi092.stdout:Base mon IP(s) is [172.21.15.92:6789], mon addrv is [v1:172.21.15.92:6789] 2024-03-11T21:20:02.325 INFO:teuthology.orchestra.run.smithi092.stdout:/sbin/ip: stdout default via 172.21.15.254 dev enp3s0f1 proto dhcp src 172.21.15.92 metric 100 2024-03-11T21:20:02.325 INFO:teuthology.orchestra.run.smithi092.stdout:/sbin/ip: stdout 172.21.0.0/20 dev enp3s0f1 proto kernel scope link src 172.21.15.92 metric 100 2024-03-11T21:20:02.328 INFO:teuthology.orchestra.run.smithi092.stdout:/sbin/ip: stdout ::1 dev lo proto kernel metric 256 pref medium 2024-03-11T21:20:02.328 INFO:teuthology.orchestra.run.smithi092.stdout:/sbin/ip: stdout fe80::/64 dev enp3s0f1 proto kernel metric 256 pref medium 2024-03-11T21:20:02.328 INFO:teuthology.orchestra.run.smithi092.stdout:/sbin/ip: stdout default via fe80::327c:5e00:6487:71e0 dev enp3s0f1 proto ra metric 1024 expires 1799sec hoplimit 64 pref medium 2024-03-11T21:20:02.330 INFO:teuthology.orchestra.run.smithi092.stdout:/sbin/ip: stdout 1: lo: mtu 65536 state UNKNOWN qlen 1000 2024-03-11T21:20:02.331 INFO:teuthology.orchestra.run.smithi092.stdout:/sbin/ip: stdout inet6 ::1/128 scope host 2024-03-11T21:20:02.331 INFO:teuthology.orchestra.run.smithi092.stdout:/sbin/ip: stdout valid_lft forever preferred_lft forever 2024-03-11T21:20:02.331 INFO:teuthology.orchestra.run.smithi092.stdout:/sbin/ip: stdout 5: enp3s0f1: mtu 1500 state UP qlen 1000 2024-03-11T21:20:02.331 INFO:teuthology.orchestra.run.smithi092.stdout:/sbin/ip: stdout inet6 fe80::ec4:7aff:fe8f:cd45/64 scope link 2024-03-11T21:20:02.331 INFO:teuthology.orchestra.run.smithi092.stdout:/sbin/ip: stdout valid_lft forever preferred_lft forever 2024-03-11T21:20:02.332 INFO:teuthology.orchestra.run.smithi092.stdout:Mon IP `172.21.15.92` is in CIDR network `172.21.0.0/20` 2024-03-11T21:20:02.332 INFO:teuthology.orchestra.run.smithi092.stdout:Inferred mon public CIDR from local network configuration ['172.21.0.0/20'] 2024-03-11T21:20:02.332 INFO:teuthology.orchestra.run.smithi092.stdout:Internal network (--cluster-network) has not been provided, OSD replication will default to the public_network 2024-03-11T21:20:02.332 INFO:teuthology.orchestra.run.smithi092.stdout:Pulling container image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b... 2024-03-11T21:20:02.797 INFO:teuthology.orchestra.run.smithi092.stdout:/bin/podman: stdout 4133f8bc41bb201212a28168427a2edb1f6f3795e694bf61031ea9e2f3ff5e87 2024-03-11T21:20:02.797 INFO:teuthology.orchestra.run.smithi092.stdout:/bin/podman: stderr Trying to pull quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b... 2024-03-11T21:20:02.797 INFO:teuthology.orchestra.run.smithi092.stdout:/bin/podman: stderr Getting image source signatures 2024-03-11T21:20:02.797 INFO:teuthology.orchestra.run.smithi092.stdout:/bin/podman: stderr Copying blob sha256:4392e0adce526b0883d65bbedc13478c05c0e708702f1c7015b93fe77c162cd5 2024-03-11T21:20:02.797 INFO:teuthology.orchestra.run.smithi092.stdout:/bin/podman: stderr Copying blob sha256:f58b4670ef4403d11d1c912aa26a4dbcf73447df5302e9dc66f6a7fc0d9d1e47 2024-03-11T21:20:02.797 INFO:teuthology.orchestra.run.smithi092.stdout:/bin/podman: stderr Copying config sha256:4133f8bc41bb201212a28168427a2edb1f6f3795e694bf61031ea9e2f3ff5e87 2024-03-11T21:20:02.797 INFO:teuthology.orchestra.run.smithi092.stdout:/bin/podman: stderr Writing manifest to image destination 2024-03-11T21:20:03.717 INFO:teuthology.orchestra.run.smithi092.stdout:ceph: stdout ceph version 19.0.0-1979-g2b0101eb (2b0101eb74d60610d544cb8a473e23a35ac1cc0b) squid (dev) 2024-03-11T21:20:03.718 INFO:teuthology.orchestra.run.smithi092.stdout:Ceph version: ceph version 19.0.0-1979-g2b0101eb (2b0101eb74d60610d544cb8a473e23a35ac1cc0b) squid (dev) 2024-03-11T21:20:03.718 INFO:teuthology.orchestra.run.smithi092.stdout:Extracting ceph user uid/gid from container image... 2024-03-11T21:20:04.556 INFO:teuthology.orchestra.run.smithi092.stdout:stat: stdout 167 167 2024-03-11T21:20:04.557 INFO:teuthology.orchestra.run.smithi092.stdout:Creating initial keys... 2024-03-11T21:20:05.445 INFO:teuthology.orchestra.run.smithi092.stdout:/usr/bin/ceph-authtool: stdout AQCEde9l2wOFNBAAKVv1koGBLBhwhbJdzBvvsQ== 2024-03-11T21:20:06.384 INFO:teuthology.orchestra.run.smithi092.stdout:/usr/bin/ceph-authtool: stdout AQCFde9l/vrmLhAAGfnuOmLfD0X+zmweT8Pn0Q== 2024-03-11T21:20:07.300 INFO:teuthology.orchestra.run.smithi092.stdout:/usr/bin/ceph-authtool: stdout AQCGde9lBfBmKhAA8g31H2iz7FgQKtY7LBTHzw== 2024-03-11T21:20:07.300 INFO:teuthology.orchestra.run.smithi092.stdout:Creating initial monmap... 2024-03-11T21:20:08.141 INFO:teuthology.orchestra.run.smithi092.stdout:/usr/bin/monmaptool: stdout /usr/bin/monmaptool: monmap file /tmp/monmap 2024-03-11T21:20:08.141 INFO:teuthology.orchestra.run.smithi092.stdout:/usr/bin/monmaptool: stdout setting min_mon_release = quincy 2024-03-11T21:20:08.141 INFO:teuthology.orchestra.run.smithi092.stdout:/usr/bin/monmaptool: stdout /usr/bin/monmaptool: set fsid to 00db0016-dfed-11ee-95c8-87774f69a715 2024-03-11T21:20:08.141 INFO:teuthology.orchestra.run.smithi092.stdout:/usr/bin/monmaptool: stdout /usr/bin/monmaptool: writing epoch 0 to /tmp/monmap (1 monitors) 2024-03-11T21:20:08.141 INFO:teuthology.orchestra.run.smithi092.stdout:monmaptool for a [v1:172.21.15.92:6789] on /usr/bin/monmaptool: monmap file /tmp/monmap 2024-03-11T21:20:08.142 INFO:teuthology.orchestra.run.smithi092.stdout:setting min_mon_release = quincy 2024-03-11T21:20:08.142 INFO:teuthology.orchestra.run.smithi092.stdout:/usr/bin/monmaptool: set fsid to 00db0016-dfed-11ee-95c8-87774f69a715 2024-03-11T21:20:08.142 INFO:teuthology.orchestra.run.smithi092.stdout:/usr/bin/monmaptool: writing epoch 0 to /tmp/monmap (1 monitors) 2024-03-11T21:20:08.142 INFO:teuthology.orchestra.run.smithi092.stdout: 2024-03-11T21:20:08.142 INFO:teuthology.orchestra.run.smithi092.stdout:Creating mon... 2024-03-11T21:20:09.063 INFO:teuthology.orchestra.run.smithi092.stdout:create mon.a on 2024-03-11T21:20:09.273 INFO:teuthology.orchestra.run.smithi092.stdout:systemctl: stderr Removed "/etc/systemd/system/multi-user.target.wants/ceph.target". 2024-03-11T21:20:09.414 INFO:teuthology.orchestra.run.smithi092.stdout:systemctl: stderr Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /etc/systemd/system/ceph.target. 2024-03-11T21:20:09.582 INFO:teuthology.orchestra.run.smithi092.stdout:systemctl: stderr Created symlink /etc/systemd/system/multi-user.target.wants/ceph-00db0016-dfed-11ee-95c8-87774f69a715.target → /etc/systemd/system/ceph-00db0016-dfed-11ee-95c8-87774f69a715.target. 2024-03-11T21:20:09.582 INFO:teuthology.orchestra.run.smithi092.stdout:systemctl: stderr Created symlink /etc/systemd/system/ceph.target.wants/ceph-00db0016-dfed-11ee-95c8-87774f69a715.target → /etc/systemd/system/ceph-00db0016-dfed-11ee-95c8-87774f69a715.target. 2024-03-11T21:20:09.792 INFO:teuthology.orchestra.run.smithi092.stdout:Non-zero exit code 1 from systemctl reset-failed ceph-00db0016-dfed-11ee-95c8-87774f69a715@mon.a 2024-03-11T21:20:09.792 INFO:teuthology.orchestra.run.smithi092.stdout:systemctl: stderr Failed to reset failed state of unit ceph-00db0016-dfed-11ee-95c8-87774f69a715@mon.a.service: Unit ceph-00db0016-dfed-11ee-95c8-87774f69a715@mon.a.service not loaded. 2024-03-11T21:20:09.955 INFO:teuthology.orchestra.run.smithi092.stdout:systemctl: stderr Created symlink /etc/systemd/system/ceph-00db0016-dfed-11ee-95c8-87774f69a715.target.wants/ceph-00db0016-dfed-11ee-95c8-87774f69a715@mon.a.service → /etc/systemd/system/ceph-00db0016-dfed-11ee-95c8-87774f69a715@.service. 2024-03-11T21:20:10.498 INFO:teuthology.orchestra.run.smithi092.stdout:Non-zero exit code 1 from systemctl is-enabled firewalld.service 2024-03-11T21:20:10.499 INFO:teuthology.orchestra.run.smithi092.stdout:systemctl: stdout disabled 2024-03-11T21:20:10.505 INFO:teuthology.orchestra.run.smithi092.stdout:Non-zero exit code 3 from systemctl is-active firewalld.service 2024-03-11T21:20:10.505 INFO:teuthology.orchestra.run.smithi092.stdout:systemctl: stdout inactive 2024-03-11T21:20:10.505 INFO:teuthology.orchestra.run.smithi092.stdout:firewalld.service is not enabled 2024-03-11T21:20:10.505 INFO:teuthology.orchestra.run.smithi092.stdout:Not possible to enable service . firewalld.service is not available 2024-03-11T21:20:10.506 INFO:teuthology.orchestra.run.smithi092.stdout:Waiting for mon to start... 2024-03-11T21:20:10.506 INFO:teuthology.orchestra.run.smithi092.stdout:Waiting for mon... 2024-03-11T21:20:11.517 INFO:teuthology.orchestra.run.smithi092.stdout:/usr/bin/ceph: stdout cluster: 2024-03-11T21:20:11.517 INFO:teuthology.orchestra.run.smithi092.stdout:/usr/bin/ceph: stdout id: 00db0016-dfed-11ee-95c8-87774f69a715 2024-03-11T21:20:11.517 INFO:teuthology.orchestra.run.smithi092.stdout:/usr/bin/ceph: stdout health: HEALTH_OK 2024-03-11T21:20:11.518 INFO:teuthology.orchestra.run.smithi092.stdout:/usr/bin/ceph: stdout 2024-03-11T21:20:11.518 INFO:teuthology.orchestra.run.smithi092.stdout:/usr/bin/ceph: stdout services: 2024-03-11T21:20:11.518 INFO:teuthology.orchestra.run.smithi092.stdout:/usr/bin/ceph: stdout mon: 1 daemons, quorum a (age 0.535954s) 2024-03-11T21:20:11.518 INFO:teuthology.orchestra.run.smithi092.stdout:/usr/bin/ceph: stdout mgr: no daemons active 2024-03-11T21:20:11.518 INFO:teuthology.orchestra.run.smithi092.stdout:/usr/bin/ceph: stdout osd: 0 osds: 0 up, 0 in 2024-03-11T21:20:11.518 INFO:teuthology.orchestra.run.smithi092.stdout:/usr/bin/ceph: stdout 2024-03-11T21:20:11.518 INFO:teuthology.orchestra.run.smithi092.stdout:/usr/bin/ceph: stdout data: 2024-03-11T21:20:11.518 INFO:teuthology.orchestra.run.smithi092.stdout:/usr/bin/ceph: stdout pools: 0 pools, 0 pgs 2024-03-11T21:20:11.518 INFO:teuthology.orchestra.run.smithi092.stdout:/usr/bin/ceph: stdout objects: 0 objects, 0 B 2024-03-11T21:20:11.518 INFO:teuthology.orchestra.run.smithi092.stdout:/usr/bin/ceph: stdout usage: 0 B used, 0 B / 0 B avail 2024-03-11T21:20:11.518 INFO:teuthology.orchestra.run.smithi092.stdout:/usr/bin/ceph: stdout pgs: 2024-03-11T21:20:11.518 INFO:teuthology.orchestra.run.smithi092.stdout:/usr/bin/ceph: stdout 2024-03-11T21:20:11.518 INFO:teuthology.orchestra.run.smithi092.stdout:mon is available 2024-03-11T21:20:11.519 INFO:teuthology.orchestra.run.smithi092.stdout:Assimilating anything we can from ceph.conf... 2024-03-11T21:20:12.566 INFO:teuthology.orchestra.run.smithi092.stdout:/usr/bin/ceph: stdout 2024-03-11T21:20:12.566 INFO:teuthology.orchestra.run.smithi092.stdout:/usr/bin/ceph: stdout [global] 2024-03-11T21:20:12.566 INFO:teuthology.orchestra.run.smithi092.stdout:/usr/bin/ceph: stdout fsid = 00db0016-dfed-11ee-95c8-87774f69a715 2024-03-11T21:20:12.566 INFO:teuthology.orchestra.run.smithi092.stdout:/usr/bin/ceph: stdout mon_host = [v1:172.21.15.92:6789] 2024-03-11T21:20:12.566 INFO:teuthology.orchestra.run.smithi092.stdout:/usr/bin/ceph: stdout mon_osd_allow_pg_remap = true 2024-03-11T21:20:12.566 INFO:teuthology.orchestra.run.smithi092.stdout:/usr/bin/ceph: stdout mon_osd_allow_primary_affinity = true 2024-03-11T21:20:12.566 INFO:teuthology.orchestra.run.smithi092.stdout:/usr/bin/ceph: stdout mon_warn_on_no_sortbitwise = false 2024-03-11T21:20:12.566 INFO:teuthology.orchestra.run.smithi092.stdout:/usr/bin/ceph: stdout osd_crush_chooseleaf_type = 0 2024-03-11T21:20:12.567 INFO:teuthology.orchestra.run.smithi092.stdout:/usr/bin/ceph: stdout 2024-03-11T21:20:12.567 INFO:teuthology.orchestra.run.smithi092.stdout:/usr/bin/ceph: stdout [mgr] 2024-03-11T21:20:12.567 INFO:teuthology.orchestra.run.smithi092.stdout:/usr/bin/ceph: stdout mgr/telemetry/nag = false 2024-03-11T21:20:12.567 INFO:teuthology.orchestra.run.smithi092.stdout:/usr/bin/ceph: stdout 2024-03-11T21:20:12.567 INFO:teuthology.orchestra.run.smithi092.stdout:/usr/bin/ceph: stdout [osd] 2024-03-11T21:20:12.567 INFO:teuthology.orchestra.run.smithi092.stdout:/usr/bin/ceph: stdout osd_map_max_advance = 10 2024-03-11T21:20:12.567 INFO:teuthology.orchestra.run.smithi092.stdout:/usr/bin/ceph: stdout osd_sloppy_crc = true 2024-03-11T21:20:12.567 INFO:teuthology.orchestra.run.smithi092.stdout:Generating new minimal ceph.conf... 2024-03-11T21:20:13.863 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:20:13 smithi092 ceph-mon[29201]: from='client.? v1:172.21.15.92:0/2724215462' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T21:20:16.419 INFO:teuthology.orchestra.run.smithi092.stdout:Restarting the monitor... 2024-03-11T21:20:16.862 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:20:16 smithi092 systemd[1]: Stopping Ceph mon.a for 00db0016-dfed-11ee-95c8-87774f69a715... 2024-03-11T21:20:17.878 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:20:17 smithi092 ceph-00db0016-dfed-11ee-95c8-87774f69a715-mon-a[29197]: 2024-03-11T21:20:17.636+0000 7f3517056700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.a -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2024-03-11T21:20:17.878 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:20:17 smithi092 ceph-00db0016-dfed-11ee-95c8-87774f69a715-mon-a[29197]: 2024-03-11T21:20:17.636+0000 7f3517056700 -1 mon.a@0(leader) e1 *** Got Signal Terminated *** 2024-03-11T21:20:17.878 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:20:17 smithi092 podman[29446]: 2024-03-11 21:20:17.893312194 +0000 UTC m=+1.287607441 container died c6b792aa77799cee9695afe28eb9eb112cc78ec2b73f1313183a9fdcf720f008 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b, name=ceph-00db0016-dfed-11ee-95c8-87774f69a715-mon-a, RELEASE=main-2b0101e, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, CEPH_POINT_RELEASE=, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, org.label-schema.build-date=20240307, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux ) 2024-03-11T21:20:22.112 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:20:21 smithi092 podman[29446]: 2024-03-11 21:20:21.75747671 +0000 UTC m=+5.151771956 container cleanup c6b792aa77799cee9695afe28eb9eb112cc78ec2b73f1313183a9fdcf720f008 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b, name=ceph-00db0016-dfed-11ee-95c8-87774f69a715-mon-a, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, ceph=True, GIT_CLEAN=True, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, RELEASE=main-2b0101e, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240307, GIT_BRANCH=HEAD) 2024-03-11T21:20:22.113 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:20:21 smithi092 bash[29446]: ceph-00db0016-dfed-11ee-95c8-87774f69a715-mon-a 2024-03-11T21:20:22.726 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:20:22 smithi092 podman[29458]: 2024-03-11 21:20:22.407859626 +0000 UTC m=+4.513843860 container remove c6b792aa77799cee9695afe28eb9eb112cc78ec2b73f1313183a9fdcf720f008 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b, name=ceph-00db0016-dfed-11ee-95c8-87774f69a715-mon-a, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240307, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, ceph=True, GIT_CLEAN=True, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, RELEASE=main-2b0101e) 2024-03-11T21:20:22.726 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:20:22 smithi092 systemd[1]: ceph-00db0016-dfed-11ee-95c8-87774f69a715@mon.a.service: Deactivated successfully. 2024-03-11T21:20:22.726 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:20:22 smithi092 systemd[1]: Stopped Ceph mon.a for 00db0016-dfed-11ee-95c8-87774f69a715. 2024-03-11T21:20:22.727 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:20:22 smithi092 systemd[1]: Starting Ceph mon.a for 00db0016-dfed-11ee-95c8-87774f69a715... 2024-03-11T21:20:22.727 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:20:22 smithi092 podman[29554]: 2024-03-11 21:20:22.742088532 +0000 UTC m=+0.108743219 container create 4cb262f5a41b5058d15402d9dda414ecb4d6e1da08df8a3f7b60b57c80e321a5 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b, name=ceph-00db0016-dfed-11ee-95c8-87774f69a715-mon-a, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, org.label-schema.license=GPLv2, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, GIT_CLEAN=True, GIT_BRANCH=HEAD, RELEASE=main-2b0101e, org.label-schema.build-date=20240307, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux ) 2024-03-11T21:20:22.965 INFO:teuthology.orchestra.run.smithi092.stdout:Setting public_network to 172.21.0.0/20 in mon config section 2024-03-11T21:20:23.021 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:20:22 smithi092 podman[29554]: 2024-03-11 21:20:22.651146147 +0000 UTC m=+0.017800812 image pull 4133f8bc41bb201212a28168427a2edb1f6f3795e694bf61031ea9e2f3ff5e87 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b 2024-03-11T21:20:23.021 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:20:22 smithi092 podman[29554]: 2024-03-11 21:20:22.919587424 +0000 UTC m=+0.286242076 container init 4cb262f5a41b5058d15402d9dda414ecb4d6e1da08df8a3f7b60b57c80e321a5 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b, name=ceph-00db0016-dfed-11ee-95c8-87774f69a715-mon-a, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, RELEASE=main-2b0101e, org.label-schema.build-date=20240307, GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image) 2024-03-11T21:20:23.021 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:20:22 smithi092 podman[29554]: 2024-03-11 21:20:22.922836924 +0000 UTC m=+0.289491570 container start 4cb262f5a41b5058d15402d9dda414ecb4d6e1da08df8a3f7b60b57c80e321a5 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b, name=ceph-00db0016-dfed-11ee-95c8-87774f69a715-mon-a, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240307, org.label-schema.license=GPLv2, RELEASE=main-2b0101e, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux ) 2024-03-11T21:20:23.021 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:20:22 smithi092 ceph-mon[29572]: set uid:gid to 167:167 (ceph:ceph) 2024-03-11T21:20:23.021 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:20:22 smithi092 ceph-mon[29572]: ceph version 19.0.0-1979-g2b0101eb (2b0101eb74d60610d544cb8a473e23a35ac1cc0b) squid (dev), process ceph-mon, pid 2 2024-03-11T21:20:23.021 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:20:22 smithi092 ceph-mon[29572]: pidfile_write: ignore empty --pid-file 2024-03-11T21:20:23.021 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:20:22 smithi092 ceph-mon[29572]: load: jerasure load: lrc 2024-03-11T21:20:23.021 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:20:22 smithi092 ceph-mon[29572]: rocksdb: RocksDB version: 7.9.2 2024-03-11T21:20:23.021 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:20:22 smithi092 ceph-mon[29572]: rocksdb: Git sha 0 2024-03-11T21:20:23.022 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:20:22 smithi092 ceph-mon[29572]: rocksdb: Compile date 2024-03-10 23:25:48 2024-03-11T21:20:23.022 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:20:22 smithi092 ceph-mon[29572]: rocksdb: DB SUMMARY 2024-03-11T21:20:23.022 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:20:22 smithi092 ceph-mon[29572]: rocksdb: DB Session ID: 4ULVSUF6S6DJRZ730KM9 2024-03-11T21:20:23.022 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:20:22 smithi092 ceph-mon[29572]: rocksdb: CURRENT file: CURRENT 2024-03-11T21:20:23.022 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:20:22 smithi092 ceph-mon[29572]: rocksdb: IDENTITY file: IDENTITY 2024-03-11T21:20:23.022 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:20:22 smithi092 ceph-mon[29572]: rocksdb: MANIFEST file: MANIFEST-000010 size: 179 Bytes 2024-03-11T21:20:23.022 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:20:22 smithi092 ceph-mon[29572]: rocksdb: SST files in /var/lib/ceph/mon/ceph-a/store.db dir, Total Num: 1, files: 000008.sst 2024-03-11T21:20:23.022 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:20:22 smithi092 ceph-mon[29572]: rocksdb: Write Ahead Log file in /var/lib/ceph/mon/ceph-a/store.db: 000009.log size: 89550 ; 2024-03-11T21:20:23.022 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:20:22 smithi092 ceph-mon[29572]: rocksdb: Options.error_if_exists: 0 2024-03-11T21:20:23.022 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:20:22 smithi092 ceph-mon[29572]: rocksdb: Options.create_if_missing: 0 2024-03-11T21:20:23.022 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:20:22 smithi092 ceph-mon[29572]: rocksdb: Options.paranoid_checks: 1 2024-03-11T21:20:23.023 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:20:22 smithi092 ceph-mon[29572]: rocksdb: Options.flush_verify_memtable_count: 1 2024-03-11T21:20:23.023 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:20:22 smithi092 ceph-mon[29572]: rocksdb: Options.track_and_verify_wals_in_manifest: 0 2024-03-11T21:20:23.023 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:20:22 smithi092 ceph-mon[29572]: rocksdb: Options.verify_sst_unique_id_in_manifest: 1 2024-03-11T21:20:23.023 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:20:22 smithi092 ceph-mon[29572]: rocksdb: Options.env: 0x56530d892b20 2024-03-11T21:20:23.023 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:20:22 smithi092 ceph-mon[29572]: rocksdb: Options.fs: PosixFileSystem 2024-03-11T21:20:23.024 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:20:22 smithi092 ceph-mon[29572]: rocksdb: Options.info_log: 0x56530efbf920 2024-03-11T21:20:23.024 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:20:22 smithi092 ceph-mon[29572]: rocksdb: Options.max_file_opening_threads: 16 2024-03-11T21:20:23.024 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:20:22 smithi092 ceph-mon[29572]: rocksdb: Options.statistics: (nil) 2024-03-11T21:20:23.025 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:20:22 smithi092 ceph-mon[29572]: rocksdb: Options.use_fsync: 0 2024-03-11T21:20:23.025 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:20:22 smithi092 ceph-mon[29572]: rocksdb: Options.max_log_file_size: 0 2024-03-11T21:20:23.025 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:20:22 smithi092 ceph-mon[29572]: rocksdb: Options.max_manifest_file_size: 1073741824 2024-03-11T21:20:23.025 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:20:22 smithi092 ceph-mon[29572]: rocksdb: Options.log_file_time_to_roll: 0 2024-03-11T21:20:23.025 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:20:22 smithi092 ceph-mon[29572]: rocksdb: Options.keep_log_file_num: 1000 2024-03-11T21:20:23.025 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:20:22 smithi092 ceph-mon[29572]: rocksdb: Options.recycle_log_file_num: 0 2024-03-11T21:20:23.025 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:20:22 smithi092 ceph-mon[29572]: rocksdb: Options.allow_fallocate: 1 2024-03-11T21:20:23.025 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:20:22 smithi092 ceph-mon[29572]: rocksdb: Options.allow_mmap_reads: 0 2024-03-11T21:20:23.025 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:20:22 smithi092 ceph-mon[29572]: rocksdb: Options.allow_mmap_writes: 0 2024-03-11T21:20:23.025 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:20:22 smithi092 ceph-mon[29572]: rocksdb: Options.use_direct_reads: 0 2024-03-11T21:20:23.025 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:20:22 smithi092 ceph-mon[29572]: rocksdb: Options.use_direct_io_for_flush_and_compaction: 0 2024-03-11T21:20:23.025 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:20:22 smithi092 ceph-mon[29572]: rocksdb: Options.create_missing_column_families: 0 2024-03-11T21:20:23.025 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:20:22 smithi092 ceph-mon[29572]: rocksdb: Options.db_log_dir: 2024-03-11T21:20:23.026 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:20:22 smithi092 ceph-mon[29572]: rocksdb: Options.wal_dir: 2024-03-11T21:20:23.026 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:20:22 smithi092 ceph-mon[29572]: rocksdb: Options.table_cache_numshardbits: 6 2024-03-11T21:20:23.026 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:20:22 smithi092 ceph-mon[29572]: rocksdb: Options.WAL_ttl_seconds: 0 2024-03-11T21:20:23.026 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:20:22 smithi092 ceph-mon[29572]: rocksdb: Options.WAL_size_limit_MB: 0 2024-03-11T21:20:23.026 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:20:22 smithi092 ceph-mon[29572]: rocksdb: Options.max_write_batch_group_size_bytes: 1048576 2024-03-11T21:20:23.026 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:20:22 smithi092 ceph-mon[29572]: rocksdb: Options.manifest_preallocation_size: 4194304 2024-03-11T21:20:23.026 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:20:22 smithi092 ceph-mon[29572]: rocksdb: Options.is_fd_close_on_exec: 1 2024-03-11T21:20:23.026 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:20:22 smithi092 ceph-mon[29572]: rocksdb: Options.advise_random_on_open: 1 2024-03-11T21:20:23.026 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:20:22 smithi092 ceph-mon[29572]: rocksdb: Options.db_write_buffer_size: 0 2024-03-11T21:20:23.026 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:20:22 smithi092 ceph-mon[29572]: rocksdb: Options.write_buffer_manager: 0x56530ef93400 2024-03-11T21:20:23.026 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:20:22 smithi092 ceph-mon[29572]: rocksdb: Options.access_hint_on_compaction_start: 1 2024-03-11T21:20:23.026 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:20:22 smithi092 ceph-mon[29572]: rocksdb: Options.random_access_max_buffer_size: 1048576 2024-03-11T21:20:23.026 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:20:22 smithi092 ceph-mon[29572]: rocksdb: Options.use_adaptive_mutex: 0 2024-03-11T21:20:23.026 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:20:22 smithi092 ceph-mon[29572]: rocksdb: Options.rate_limiter: (nil) 2024-03-11T21:20:23.027 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:20:22 smithi092 ceph-mon[29572]: rocksdb: Options.sst_file_manager.rate_bytes_per_sec: 0 2024-03-11T21:20:23.027 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:20:22 smithi092 ceph-mon[29572]: rocksdb: Options.wal_recovery_mode: 2 2024-03-11T21:20:23.027 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:20:22 smithi092 ceph-mon[29572]: rocksdb: Options.enable_thread_tracking: 0 2024-03-11T21:20:23.027 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:20:22 smithi092 ceph-mon[29572]: rocksdb: Options.enable_pipelined_write: 0 2024-03-11T21:20:23.027 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:20:22 smithi092 ceph-mon[29572]: rocksdb: Options.unordered_write: 0 2024-03-11T21:20:23.027 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:20:22 smithi092 ceph-mon[29572]: rocksdb: Options.allow_concurrent_memtable_write: 1 2024-03-11T21:20:23.027 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:20:22 smithi092 ceph-mon[29572]: rocksdb: Options.enable_write_thread_adaptive_yield: 1 2024-03-11T21:20:23.027 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:20:22 smithi092 ceph-mon[29572]: rocksdb: Options.write_thread_max_yield_usec: 100 2024-03-11T21:20:23.027 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:20:22 smithi092 ceph-mon[29572]: rocksdb: Options.write_thread_slow_yield_usec: 3 2024-03-11T21:20:23.027 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:20:22 smithi092 ceph-mon[29572]: rocksdb: Options.row_cache: None 2024-03-11T21:20:23.027 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:20:22 smithi092 ceph-mon[29572]: rocksdb: Options.wal_filter: None 2024-03-11T21:20:23.027 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:20:22 smithi092 ceph-mon[29572]: rocksdb: Options.avoid_flush_during_recovery: 0 2024-03-11T21:20:23.027 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:20:22 smithi092 ceph-mon[29572]: rocksdb: Options.allow_ingest_behind: 0 2024-03-11T21:20:23.027 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:20:22 smithi092 ceph-mon[29572]: rocksdb: Options.two_write_queues: 0 2024-03-11T21:20:23.028 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:20:22 smithi092 ceph-mon[29572]: rocksdb: Options.manual_wal_flush: 0 2024-03-11T21:20:23.028 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:20:22 smithi092 ceph-mon[29572]: rocksdb: Options.wal_compression: 0 2024-03-11T21:20:23.028 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:20:22 smithi092 ceph-mon[29572]: rocksdb: Options.atomic_flush: 0 2024-03-11T21:20:23.028 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:20:22 smithi092 ceph-mon[29572]: rocksdb: Options.avoid_unnecessary_blocking_io: 0 2024-03-11T21:20:23.028 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:20:22 smithi092 ceph-mon[29572]: rocksdb: Options.persist_stats_to_disk: 0 2024-03-11T21:20:23.028 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:20:22 smithi092 ceph-mon[29572]: rocksdb: Options.write_dbid_to_manifest: 0 2024-03-11T21:20:23.028 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:20:22 smithi092 ceph-mon[29572]: rocksdb: Options.log_readahead_size: 0 2024-03-11T21:20:23.028 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:20:22 smithi092 ceph-mon[29572]: rocksdb: Options.file_checksum_gen_factory: Unknown 2024-03-11T21:20:23.028 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:20:22 smithi092 ceph-mon[29572]: rocksdb: Options.best_efforts_recovery: 0 2024-03-11T21:20:23.028 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:20:22 smithi092 ceph-mon[29572]: rocksdb: Options.max_bgerror_resume_count: 2147483647 2024-03-11T21:20:23.028 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:20:22 smithi092 ceph-mon[29572]: rocksdb: Options.bgerror_resume_retry_interval: 1000000 2024-03-11T21:20:23.028 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:20:22 smithi092 ceph-mon[29572]: rocksdb: Options.allow_data_in_errors: 0 2024-03-11T21:20:23.028 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:20:22 smithi092 ceph-mon[29572]: rocksdb: Options.db_host_id: __hostname__ 2024-03-11T21:20:23.028 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:20:22 smithi092 ceph-mon[29572]: rocksdb: Options.enforce_single_del_contracts: true 2024-03-11T21:20:23.028 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:20:22 smithi092 ceph-mon[29572]: rocksdb: Options.max_background_jobs: 2 2024-03-11T21:20:23.029 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:20:22 smithi092 ceph-mon[29572]: rocksdb: Options.max_background_compactions: -1 2024-03-11T21:20:23.029 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:20:22 smithi092 ceph-mon[29572]: rocksdb: Options.max_subcompactions: 1 2024-03-11T21:20:23.029 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:20:22 smithi092 ceph-mon[29572]: rocksdb: Options.avoid_flush_during_shutdown: 0 2024-03-11T21:20:23.029 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:20:22 smithi092 ceph-mon[29572]: rocksdb: Options.writable_file_max_buffer_size: 1048576 2024-03-11T21:20:23.029 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:20:22 smithi092 ceph-mon[29572]: rocksdb: Options.delayed_write_rate : 16777216 2024-03-11T21:20:23.029 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:20:22 smithi092 ceph-mon[29572]: rocksdb: Options.max_total_wal_size: 0 2024-03-11T21:20:23.029 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:20:22 smithi092 ceph-mon[29572]: rocksdb: Options.delete_obsolete_files_period_micros: 21600000000 2024-03-11T21:20:23.029 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:20:22 smithi092 ceph-mon[29572]: rocksdb: Options.stats_dump_period_sec: 600 2024-03-11T21:20:23.030 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:20:22 smithi092 ceph-mon[29572]: rocksdb: Options.stats_persist_period_sec: 600 2024-03-11T21:20:23.030 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:20:22 smithi092 ceph-mon[29572]: rocksdb: Options.stats_history_buffer_size: 1048576 2024-03-11T21:20:23.030 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:20:22 smithi092 ceph-mon[29572]: rocksdb: Options.max_open_files: -1 2024-03-11T21:20:23.030 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:20:22 smithi092 ceph-mon[29572]: rocksdb: Options.bytes_per_sync: 0 2024-03-11T21:20:23.030 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:20:22 smithi092 ceph-mon[29572]: rocksdb: Options.wal_bytes_per_sync: 0 2024-03-11T21:20:23.030 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:20:22 smithi092 ceph-mon[29572]: rocksdb: Options.strict_bytes_per_sync: 0 2024-03-11T21:20:23.030 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:20:22 smithi092 ceph-mon[29572]: rocksdb: Options.compaction_readahead_size: 0 2024-03-11T21:20:23.030 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:20:22 smithi092 ceph-mon[29572]: rocksdb: Options.max_background_flushes: -1 2024-03-11T21:20:23.030 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:20:22 smithi092 ceph-mon[29572]: rocksdb: Compression algorithms supported: 2024-03-11T21:20:23.030 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:20:22 smithi092 ceph-mon[29572]: rocksdb: kZSTDNotFinalCompression supported: 0 2024-03-11T21:20:23.030 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:20:22 smithi092 ceph-mon[29572]: rocksdb: kZSTD supported: 0 2024-03-11T21:20:23.030 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:20:22 smithi092 ceph-mon[29572]: rocksdb: kXpressCompression supported: 0 2024-03-11T21:20:23.030 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:20:22 smithi092 ceph-mon[29572]: rocksdb: kLZ4HCCompression supported: 1 2024-03-11T21:20:23.031 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:20:22 smithi092 ceph-mon[29572]: rocksdb: kZlibCompression supported: 1 2024-03-11T21:20:23.031 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:20:22 smithi092 ceph-mon[29572]: rocksdb: kSnappyCompression supported: 1 2024-03-11T21:20:23.031 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:20:22 smithi092 ceph-mon[29572]: rocksdb: kLZ4Compression supported: 1 2024-03-11T21:20:23.031 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:20:22 smithi092 ceph-mon[29572]: rocksdb: kBZip2Compression supported: 0 2024-03-11T21:20:23.031 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:20:22 smithi092 ceph-mon[29572]: rocksdb: Fast CRC32 supported: Supported on x86 2024-03-11T21:20:23.031 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:20:22 smithi092 ceph-mon[29572]: rocksdb: DMutex implementation: pthread_mutex_t 2024-03-11T21:20:23.031 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:20:22 smithi092 ceph-mon[29572]: rocksdb: [db/version_set.cc:5527] Recovering from manifest file: /var/lib/ceph/mon/ceph-a/store.db/MANIFEST-000010 2024-03-11T21:20:23.031 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:20:22 smithi092 ceph-mon[29572]: rocksdb: [db/column_family.cc:630] --------------- Options for column family [default]: 2024-03-11T21:20:23.031 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:20:22 smithi092 ceph-mon[29572]: rocksdb: Options.comparator: leveldb.BytewiseComparator 2024-03-11T21:20:23.031 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:20:22 smithi092 ceph-mon[29572]: rocksdb: Options.merge_operator: 2024-03-11T21:20:23.032 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:20:22 smithi092 ceph-mon[29572]: rocksdb: Options.compaction_filter: None 2024-03-11T21:20:23.032 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:20:22 smithi092 ceph-mon[29572]: rocksdb: Options.compaction_filter_factory: None 2024-03-11T21:20:23.032 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:20:22 smithi092 ceph-mon[29572]: rocksdb: Options.sst_partitioner_factory: None 2024-03-11T21:20:23.032 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:20:22 smithi092 ceph-mon[29572]: rocksdb: Options.memtable_factory: SkipListFactory 2024-03-11T21:20:23.032 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:20:22 smithi092 ceph-mon[29572]: rocksdb: Options.table_factory: BlockBasedTable 2024-03-11T21:20:23.032 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:20:22 smithi092 ceph-mon[29572]: rocksdb: table_factory options: flush_block_policy_factory: FlushBlockBySizePolicyFactory (0x56530efbfa20) 2024-03-11T21:20:23.032 INFO:journalctl@ceph.mon.a.smithi092.stdout: cache_index_and_filter_blocks: 1 2024-03-11T21:20:23.032 INFO:journalctl@ceph.mon.a.smithi092.stdout: cache_index_and_filter_blocks_with_high_priority: 0 2024-03-11T21:20:23.032 INFO:journalctl@ceph.mon.a.smithi092.stdout: pin_l0_filter_and_index_blocks_in_cache: 0 2024-03-11T21:20:23.032 INFO:journalctl@ceph.mon.a.smithi092.stdout: pin_top_level_index_and_filter: 1 2024-03-11T21:20:23.032 INFO:journalctl@ceph.mon.a.smithi092.stdout: index_type: 0 2024-03-11T21:20:23.032 INFO:journalctl@ceph.mon.a.smithi092.stdout: data_block_index_type: 0 2024-03-11T21:20:23.032 INFO:journalctl@ceph.mon.a.smithi092.stdout: index_shortening: 1 2024-03-11T21:20:23.032 INFO:journalctl@ceph.mon.a.smithi092.stdout: data_block_hash_table_util_ratio: 0.750000 2024-03-11T21:20:23.033 INFO:journalctl@ceph.mon.a.smithi092.stdout: checksum: 4 2024-03-11T21:20:23.033 INFO:journalctl@ceph.mon.a.smithi092.stdout: no_block_cache: 0 2024-03-11T21:20:23.033 INFO:journalctl@ceph.mon.a.smithi092.stdout: block_cache: 0x56530efc5090 2024-03-11T21:20:23.033 INFO:journalctl@ceph.mon.a.smithi092.stdout: block_cache_name: BinnedLRUCache 2024-03-11T21:20:23.033 INFO:journalctl@ceph.mon.a.smithi092.stdout: block_cache_options: 2024-03-11T21:20:23.033 INFO:journalctl@ceph.mon.a.smithi092.stdout: capacity : 536870912 2024-03-11T21:20:23.033 INFO:journalctl@ceph.mon.a.smithi092.stdout: num_shard_bits : 4 2024-03-11T21:20:23.033 INFO:journalctl@ceph.mon.a.smithi092.stdout: strict_capacity_limit : 0 2024-03-11T21:20:23.033 INFO:journalctl@ceph.mon.a.smithi092.stdout: high_pri_pool_ratio: 0.000 2024-03-11T21:20:23.033 INFO:journalctl@ceph.mon.a.smithi092.stdout: block_cache_compressed: (nil) 2024-03-11T21:20:23.033 INFO:journalctl@ceph.mon.a.smithi092.stdout: persistent_cache: (nil) 2024-03-11T21:20:23.033 INFO:journalctl@ceph.mon.a.smithi092.stdout: block_size: 4096 2024-03-11T21:20:23.033 INFO:journalctl@ceph.mon.a.smithi092.stdout: block_size_deviation: 10 2024-03-11T21:20:23.033 INFO:journalctl@ceph.mon.a.smithi092.stdout: block_restart_interval: 16 2024-03-11T21:20:23.034 INFO:journalctl@ceph.mon.a.smithi092.stdout: index_block_restart_interval: 1 2024-03-11T21:20:23.034 INFO:journalctl@ceph.mon.a.smithi092.stdout: metadata_block_size: 4096 2024-03-11T21:20:23.034 INFO:journalctl@ceph.mon.a.smithi092.stdout: partition_filters: 0 2024-03-11T21:20:23.034 INFO:journalctl@ceph.mon.a.smithi092.stdout: use_delta_encoding: 1 2024-03-11T21:20:23.034 INFO:journalctl@ceph.mon.a.smithi092.stdout: filter_policy: bloomfilter 2024-03-11T21:20:23.034 INFO:journalctl@ceph.mon.a.smithi092.stdout: whole_key_filtering: 1 2024-03-11T21:20:23.034 INFO:journalctl@ceph.mon.a.smithi092.stdout: verify_compression: 0 2024-03-11T21:20:23.034 INFO:journalctl@ceph.mon.a.smithi092.stdout: read_amp_bytes_per_bit: 0 2024-03-11T21:20:23.034 INFO:journalctl@ceph.mon.a.smithi092.stdout: format_version: 5 2024-03-11T21:20:23.034 INFO:journalctl@ceph.mon.a.smithi092.stdout: enable_index_compression: 1 2024-03-11T21:20:23.034 INFO:journalctl@ceph.mon.a.smithi092.stdout: block_align: 0 2024-03-11T21:20:23.034 INFO:journalctl@ceph.mon.a.smithi092.stdout: max_auto_readahead_size: 262144 2024-03-11T21:20:23.034 INFO:journalctl@ceph.mon.a.smithi092.stdout: prepopulate_block_cache: 0 2024-03-11T21:20:23.034 INFO:journalctl@ceph.mon.a.smithi092.stdout: initial_auto_readahead_size: 8192 2024-03-11T21:20:23.035 INFO:journalctl@ceph.mon.a.smithi092.stdout: num_file_reads_for_auto_readahead: 2 2024-03-11T21:20:23.035 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:20:22 smithi092 ceph-mon[29572]: rocksdb: Options.write_buffer_size: 33554432 2024-03-11T21:20:23.035 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:20:22 smithi092 ceph-mon[29572]: rocksdb: Options.max_write_buffer_number: 2 2024-03-11T21:20:23.035 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:20:22 smithi092 ceph-mon[29572]: rocksdb: Options.compression: NoCompression 2024-03-11T21:20:23.035 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:20:22 smithi092 ceph-mon[29572]: rocksdb: Options.bottommost_compression: Disabled 2024-03-11T21:20:23.035 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:20:22 smithi092 ceph-mon[29572]: rocksdb: Options.prefix_extractor: nullptr 2024-03-11T21:20:23.035 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:20:22 smithi092 ceph-mon[29572]: rocksdb: Options.memtable_insert_with_hint_prefix_extractor: nullptr 2024-03-11T21:20:23.035 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:20:22 smithi092 ceph-mon[29572]: rocksdb: Options.num_levels: 7 2024-03-11T21:20:23.035 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:20:22 smithi092 ceph-mon[29572]: rocksdb: Options.min_write_buffer_number_to_merge: 1 2024-03-11T21:20:23.035 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:20:22 smithi092 ceph-mon[29572]: rocksdb: Options.max_write_buffer_number_to_maintain: 0 2024-03-11T21:20:23.035 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:20:22 smithi092 ceph-mon[29572]: rocksdb: Options.max_write_buffer_size_to_maintain: 0 2024-03-11T21:20:23.035 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:20:22 smithi092 ceph-mon[29572]: rocksdb: Options.bottommost_compression_opts.window_bits: -14 2024-03-11T21:20:23.035 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:20:22 smithi092 ceph-mon[29572]: rocksdb: Options.bottommost_compression_opts.level: 32767 2024-03-11T21:20:23.035 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:20:22 smithi092 ceph-mon[29572]: rocksdb: Options.bottommost_compression_opts.strategy: 0 2024-03-11T21:20:23.035 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:20:22 smithi092 ceph-mon[29572]: rocksdb: Options.bottommost_compression_opts.max_dict_bytes: 0 2024-03-11T21:20:23.036 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:20:22 smithi092 ceph-mon[29572]: rocksdb: Options.bottommost_compression_opts.zstd_max_train_bytes: 0 2024-03-11T21:20:23.036 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:20:22 smithi092 ceph-mon[29572]: rocksdb: Options.bottommost_compression_opts.parallel_threads: 1 2024-03-11T21:20:23.036 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:20:22 smithi092 ceph-mon[29572]: rocksdb: Options.bottommost_compression_opts.enabled: false 2024-03-11T21:20:23.036 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:20:22 smithi092 ceph-mon[29572]: rocksdb: Options.bottommost_compression_opts.max_dict_buffer_bytes: 0 2024-03-11T21:20:23.036 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:20:22 smithi092 ceph-mon[29572]: rocksdb: Options.bottommost_compression_opts.use_zstd_dict_trainer: true 2024-03-11T21:20:23.036 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:20:22 smithi092 ceph-mon[29572]: rocksdb: Options.compression_opts.window_bits: -14 2024-03-11T21:20:23.036 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:20:22 smithi092 ceph-mon[29572]: rocksdb: Options.compression_opts.level: 32767 2024-03-11T21:20:23.036 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:20:22 smithi092 ceph-mon[29572]: rocksdb: Options.compression_opts.strategy: 0 2024-03-11T21:20:23.036 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:20:22 smithi092 ceph-mon[29572]: rocksdb: Options.compression_opts.max_dict_bytes: 0 2024-03-11T21:20:23.036 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:20:22 smithi092 ceph-mon[29572]: rocksdb: Options.compression_opts.zstd_max_train_bytes: 0 2024-03-11T21:20:23.036 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:20:22 smithi092 ceph-mon[29572]: rocksdb: Options.compression_opts.use_zstd_dict_trainer: true 2024-03-11T21:20:23.036 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:20:22 smithi092 ceph-mon[29572]: rocksdb: Options.compression_opts.parallel_threads: 1 2024-03-11T21:20:23.036 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:20:22 smithi092 ceph-mon[29572]: rocksdb: Options.compression_opts.enabled: false 2024-03-11T21:20:23.036 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:20:22 smithi092 ceph-mon[29572]: rocksdb: Options.compression_opts.max_dict_buffer_bytes: 0 2024-03-11T21:20:23.037 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:20:22 smithi092 ceph-mon[29572]: rocksdb: Options.level0_file_num_compaction_trigger: 4 2024-03-11T21:20:23.037 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:20:22 smithi092 ceph-mon[29572]: rocksdb: Options.level0_slowdown_writes_trigger: 20 2024-03-11T21:20:23.037 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:20:22 smithi092 ceph-mon[29572]: rocksdb: Options.level0_stop_writes_trigger: 36 2024-03-11T21:20:23.037 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:20:22 smithi092 ceph-mon[29572]: rocksdb: Options.target_file_size_base: 67108864 2024-03-11T21:20:23.037 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:20:22 smithi092 ceph-mon[29572]: rocksdb: Options.target_file_size_multiplier: 1 2024-03-11T21:20:23.037 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:20:22 smithi092 ceph-mon[29572]: rocksdb: Options.max_bytes_for_level_base: 268435456 2024-03-11T21:20:23.037 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:20:22 smithi092 ceph-mon[29572]: rocksdb: Options.level_compaction_dynamic_level_bytes: 1 2024-03-11T21:20:23.037 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:20:22 smithi092 ceph-mon[29572]: rocksdb: Options.max_bytes_for_level_multiplier: 10.000000 2024-03-11T21:20:23.037 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:20:22 smithi092 ceph-mon[29572]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[0]: 1 2024-03-11T21:20:23.037 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:20:22 smithi092 ceph-mon[29572]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[1]: 1 2024-03-11T21:20:23.037 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:20:22 smithi092 ceph-mon[29572]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[2]: 1 2024-03-11T21:20:23.037 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:20:22 smithi092 ceph-mon[29572]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[3]: 1 2024-03-11T21:20:23.037 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:20:22 smithi092 ceph-mon[29572]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[4]: 1 2024-03-11T21:20:23.037 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:20:22 smithi092 ceph-mon[29572]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[5]: 1 2024-03-11T21:20:23.038 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:20:22 smithi092 ceph-mon[29572]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[6]: 1 2024-03-11T21:20:23.038 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:20:22 smithi092 ceph-mon[29572]: rocksdb: Options.max_sequential_skip_in_iterations: 8 2024-03-11T21:20:23.038 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:20:22 smithi092 ceph-mon[29572]: rocksdb: Options.max_compaction_bytes: 1677721600 2024-03-11T21:20:23.038 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:20:22 smithi092 ceph-mon[29572]: rocksdb: Options.ignore_max_compaction_bytes_for_input: true 2024-03-11T21:20:23.038 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:20:22 smithi092 ceph-mon[29572]: rocksdb: Options.arena_block_size: 1048576 2024-03-11T21:20:23.038 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:20:22 smithi092 ceph-mon[29572]: rocksdb: Options.soft_pending_compaction_bytes_limit: 68719476736 2024-03-11T21:20:23.038 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:20:22 smithi092 ceph-mon[29572]: rocksdb: Options.hard_pending_compaction_bytes_limit: 274877906944 2024-03-11T21:20:23.038 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:20:22 smithi092 ceph-mon[29572]: rocksdb: Options.disable_auto_compactions: 0 2024-03-11T21:20:23.038 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:20:22 smithi092 ceph-mon[29572]: rocksdb: Options.compaction_style: kCompactionStyleLevel 2024-03-11T21:20:23.038 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:20:22 smithi092 ceph-mon[29572]: rocksdb: Options.compaction_pri: kMinOverlappingRatio 2024-03-11T21:20:23.038 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:20:22 smithi092 ceph-mon[29572]: rocksdb: Options.compaction_options_universal.size_ratio: 1 2024-03-11T21:20:23.038 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:20:22 smithi092 ceph-mon[29572]: rocksdb: Options.compaction_options_universal.min_merge_width: 2 2024-03-11T21:20:23.038 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:20:22 smithi092 ceph-mon[29572]: rocksdb: Options.compaction_options_universal.max_merge_width: 4294967295 2024-03-11T21:20:23.038 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:20:22 smithi092 ceph-mon[29572]: rocksdb: Options.compaction_options_universal.max_size_amplification_percent: 200 2024-03-11T21:20:23.039 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:20:22 smithi092 ceph-mon[29572]: rocksdb: Options.compaction_options_universal.compression_size_percent: -1 2024-03-11T21:20:23.039 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:20:22 smithi092 ceph-mon[29572]: rocksdb: Options.compaction_options_universal.stop_style: kCompactionStopStyleTotalSize 2024-03-11T21:20:23.039 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:20:22 smithi092 ceph-mon[29572]: rocksdb: Options.compaction_options_fifo.max_table_files_size: 1073741824 2024-03-11T21:20:23.039 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:20:22 smithi092 ceph-mon[29572]: rocksdb: Options.compaction_options_fifo.allow_compaction: 0 2024-03-11T21:20:23.039 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:20:22 smithi092 ceph-mon[29572]: rocksdb: Options.table_properties_collectors: 2024-03-11T21:20:23.039 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:20:22 smithi092 ceph-mon[29572]: rocksdb: Options.inplace_update_support: 0 2024-03-11T21:20:23.039 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:20:22 smithi092 ceph-mon[29572]: rocksdb: Options.inplace_update_num_locks: 10000 2024-03-11T21:20:23.039 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:20:22 smithi092 ceph-mon[29572]: rocksdb: Options.memtable_prefix_bloom_size_ratio: 0.000000 2024-03-11T21:20:23.039 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:20:22 smithi092 ceph-mon[29572]: rocksdb: Options.memtable_whole_key_filtering: 0 2024-03-11T21:20:23.039 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:20:22 smithi092 ceph-mon[29572]: rocksdb: Options.memtable_huge_page_size: 0 2024-03-11T21:20:23.039 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:20:22 smithi092 ceph-mon[29572]: rocksdb: Options.bloom_locality: 0 2024-03-11T21:20:23.039 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:20:22 smithi092 ceph-mon[29572]: rocksdb: Options.max_successive_merges: 0 2024-03-11T21:20:23.039 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:20:22 smithi092 ceph-mon[29572]: rocksdb: Options.optimize_filters_for_hits: 0 2024-03-11T21:20:23.039 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:20:22 smithi092 ceph-mon[29572]: rocksdb: Options.paranoid_file_checks: 0 2024-03-11T21:20:23.039 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:20:22 smithi092 ceph-mon[29572]: rocksdb: Options.force_consistency_checks: 1 2024-03-11T21:20:23.040 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:20:22 smithi092 ceph-mon[29572]: rocksdb: Options.report_bg_io_stats: 0 2024-03-11T21:20:23.040 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:20:22 smithi092 ceph-mon[29572]: rocksdb: Options.ttl: 2592000 2024-03-11T21:20:23.040 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:20:22 smithi092 ceph-mon[29572]: rocksdb: Options.periodic_compaction_seconds: 0 2024-03-11T21:20:23.040 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:20:22 smithi092 ceph-mon[29572]: rocksdb: Options.preclude_last_level_data_seconds: 0 2024-03-11T21:20:23.040 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:20:22 smithi092 ceph-mon[29572]: rocksdb: Options.preserve_internal_time_seconds: 0 2024-03-11T21:20:23.040 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:20:22 smithi092 ceph-mon[29572]: rocksdb: Options.enable_blob_files: false 2024-03-11T21:20:23.040 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:20:22 smithi092 ceph-mon[29572]: rocksdb: Options.min_blob_size: 0 2024-03-11T21:20:23.040 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:20:22 smithi092 ceph-mon[29572]: rocksdb: Options.blob_file_size: 268435456 2024-03-11T21:20:23.040 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:20:22 smithi092 ceph-mon[29572]: rocksdb: Options.blob_compression_type: NoCompression 2024-03-11T21:20:23.040 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:20:22 smithi092 ceph-mon[29572]: rocksdb: Options.enable_blob_garbage_collection: false 2024-03-11T21:20:23.040 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:20:22 smithi092 ceph-mon[29572]: rocksdb: Options.blob_garbage_collection_age_cutoff: 0.250000 2024-03-11T21:20:23.040 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:20:22 smithi092 ceph-mon[29572]: rocksdb: Options.blob_garbage_collection_force_threshold: 1.000000 2024-03-11T21:20:23.040 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:20:22 smithi092 ceph-mon[29572]: rocksdb: Options.blob_compaction_readahead_size: 0 2024-03-11T21:20:23.040 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:20:22 smithi092 ceph-mon[29572]: rocksdb: Options.blob_file_starting_level: 0 2024-03-11T21:20:23.040 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:20:22 smithi092 ceph-mon[29572]: rocksdb: Options.experimental_mempurge_threshold: 0.000000 2024-03-11T21:20:23.041 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:20:22 smithi092 ceph-mon[29572]: rocksdb: [db/version_set.cc:5566] Recovered from manifest file:/var/lib/ceph/mon/ceph-a/store.db/MANIFEST-000010 succeeded,manifest_file_number is 10, next_file_number is 12, last_sequence is 5, log_number is 5,prev_log_number is 0,max_column_family is 0,min_log_number_to_keep is 5 2024-03-11T21:20:23.041 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:20:22 smithi092 ceph-mon[29572]: rocksdb: [db/version_set.cc:5581] Column family [default] (ID 0), log number is 5 2024-03-11T21:20:23.041 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:20:22 smithi092 ceph-mon[29572]: rocksdb: [db/db_impl/db_impl_open.cc:539] DB ID: 0f61929c-b440-42aa-be79-e841277cfb45 2024-03-11T21:20:23.041 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:20:22 smithi092 ceph-mon[29572]: rocksdb: EVENT_LOG_v1 {"time_micros": 1710192022950996, "job": 1, "event": "recovery_started", "wal_files": [9]} 2024-03-11T21:20:23.041 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:20:22 smithi092 ceph-mon[29572]: rocksdb: [db/db_impl/db_impl_open.cc:1043] Recovering log #9 mode 2 2024-03-11T21:20:23.041 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:20:22 smithi092 ceph-mon[29572]: rocksdb: EVENT_LOG_v1 {"time_micros": 1710192022951930, "cf_name": "default", "job": 1, "event": "table_file_creation", "file_number": 13, "file_size": 85487, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 8, "largest_seqno": 280, "table_properties": {"data_size": 83595, "index_size": 238, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 1, "index_value_is_delta_encoded": 1, "filter_size": 645, "raw_key_size": 11691, "raw_average_key_size": 48, "raw_value_size": 77150, "raw_average_value_size": 322, "num_data_blocks": 11, "num_entries": 239, "num_filter_entries": 239, "num_deletions": 3, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 0, "filter_policy": "bloomfilter", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "", "prefix_extractor_name": "nullptr", "property_collectors": "[]", "compression": "NoCompression", "compression_options": "window_bits=-14; level=32767; strategy=0; max_dict_bytes=0; zstd_max_train_bytes=0; enabled=0; max_dict_buffer_bytes=0; use_zstd_dict_trainer=1; ", "creation_time": 1710192022, "oldest_key_time": 0, "file_creation_time": 0, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "0f61929c-b440-42aa-be79-e841277cfb45", "db_session_id": "4ULVSUF6S6DJRZ730KM9", "orig_file_number": 13, "seqno_to_time_mapping": "N/A"}} 2024-03-11T21:20:23.041 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:20:22 smithi092 ceph-mon[29572]: rocksdb: EVENT_LOG_v1 {"time_micros": 1710192022952019, "job": 1, "event": "recovery_finished"} 2024-03-11T21:20:23.041 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:20:22 smithi092 ceph-mon[29572]: rocksdb: [db/version_set.cc:5047] Creating manifest 15 2024-03-11T21:20:23.041 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:20:22 smithi092 ceph-mon[29572]: rocksdb: [file/delete_scheduler.cc:74] Deleted file /var/lib/ceph/mon/ceph-a/store.db/000009.log immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024-03-11T21:20:23.041 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:20:22 smithi092 ceph-mon[29572]: rocksdb: [db/db_impl/db_impl_open.cc:1987] SstFileManager instance 0x56530f0b8000 2024-03-11T21:20:23.042 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:20:22 smithi092 ceph-mon[29572]: rocksdb: DB pointer 0x56530f0a2000 2024-03-11T21:20:23.042 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:20:22 smithi092 ceph-mon[29572]: rocksdb: [db/db_impl/db_impl.cc:1109] ------- DUMPING STATS ------- 2024-03-11T21:20:23.042 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:20:22 smithi092 ceph-mon[29572]: rocksdb: [db/db_impl/db_impl.cc:1111] 2024-03-11T21:20:23.042 INFO:journalctl@ceph.mon.a.smithi092.stdout: ** DB Stats ** 2024-03-11T21:20:23.042 INFO:journalctl@ceph.mon.a.smithi092.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-03-11T21:20:23.042 INFO:journalctl@ceph.mon.a.smithi092.stdout: Cumulative writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 GB, 0.00 MB/s 2024-03-11T21:20:23.042 INFO:journalctl@ceph.mon.a.smithi092.stdout: Cumulative WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-03-11T21:20:23.042 INFO:journalctl@ceph.mon.a.smithi092.stdout: Cumulative stall: 00:00:0.000 H:M:S, 0.0 percent 2024-03-11T21:20:23.042 INFO:journalctl@ceph.mon.a.smithi092.stdout: Interval writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 MB, 0.00 MB/s 2024-03-11T21:20:23.042 INFO:journalctl@ceph.mon.a.smithi092.stdout: Interval WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-03-11T21:20:23.042 INFO:journalctl@ceph.mon.a.smithi092.stdout: Interval stall: 00:00:0.000 H:M:S, 0.0 percent 2024-03-11T21:20:23.042 INFO:journalctl@ceph.mon.a.smithi092.stdout: 2024-03-11T21:20:23.042 INFO:journalctl@ceph.mon.a.smithi092.stdout: ** Compaction Stats [default] ** 2024-03-11T21:20:23.042 INFO:journalctl@ceph.mon.a.smithi092.stdout: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2024-03-11T21:20:23.043 INFO:journalctl@ceph.mon.a.smithi092.stdout: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ 2024-03-11T21:20:23.043 INFO:journalctl@ceph.mon.a.smithi092.stdout: L0 2/0 85.28 KB 0.5 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 131.9 0.00 0.00 1 0.001 0 0 0.0 0.0 2024-03-11T21:20:23.043 INFO:journalctl@ceph.mon.a.smithi092.stdout: Sum 2/0 85.28 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 131.9 0.00 0.00 1 0.001 0 0 0.0 0.0 2024-03-11T21:20:23.043 INFO:journalctl@ceph.mon.a.smithi092.stdout: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 131.9 0.00 0.00 1 0.001 0 0 0.0 0.0 2024-03-11T21:20:23.043 INFO:journalctl@ceph.mon.a.smithi092.stdout: 2024-03-11T21:20:23.043 INFO:journalctl@ceph.mon.a.smithi092.stdout: ** Compaction Stats [default] ** 2024-03-11T21:20:23.043 INFO:journalctl@ceph.mon.a.smithi092.stdout: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2024-03-11T21:20:23.043 INFO:journalctl@ceph.mon.a.smithi092.stdout: --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-03-11T21:20:23.043 INFO:journalctl@ceph.mon.a.smithi092.stdout: User 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 131.9 0.00 0.00 1 0.001 0 0 0.0 0.0 2024-03-11T21:20:23.043 INFO:journalctl@ceph.mon.a.smithi092.stdout: 2024-03-11T21:20:23.043 INFO:journalctl@ceph.mon.a.smithi092.stdout: Blob file count: 0, total size: 0.0 GB, garbage size: 0.0 GB, space amp: 0.0 2024-03-11T21:20:23.043 INFO:journalctl@ceph.mon.a.smithi092.stdout: 2024-03-11T21:20:23.043 INFO:journalctl@ceph.mon.a.smithi092.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-03-11T21:20:23.044 INFO:journalctl@ceph.mon.a.smithi092.stdout: Flush(GB): cumulative 0.000, interval 0.000 2024-03-11T21:20:23.044 INFO:journalctl@ceph.mon.a.smithi092.stdout: AddFile(GB): cumulative 0.000, interval 0.000 2024-03-11T21:20:23.044 INFO:journalctl@ceph.mon.a.smithi092.stdout: AddFile(Total Files): cumulative 0, interval 0 2024-03-11T21:20:23.044 INFO:journalctl@ceph.mon.a.smithi092.stdout: AddFile(L0 Files): cumulative 0, interval 0 2024-03-11T21:20:23.044 INFO:journalctl@ceph.mon.a.smithi092.stdout: AddFile(Keys): cumulative 0, interval 0 2024-03-11T21:20:23.044 INFO:journalctl@ceph.mon.a.smithi092.stdout: Cumulative compaction: 0.00 GB write, 28.28 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-03-11T21:20:23.044 INFO:journalctl@ceph.mon.a.smithi092.stdout: Interval compaction: 0.00 GB write, 28.28 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-03-11T21:20:23.044 INFO:journalctl@ceph.mon.a.smithi092.stdout: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2024-03-11T21:20:23.044 INFO:journalctl@ceph.mon.a.smithi092.stdout: Block cache BinnedLRUCache@0x56530efc5090#2 capacity: 512.00 MB usage: 1.25 KB table_size: 0 occupancy: 18446744073709551615 collections: 1 last_copies: 0 last_secs: 7e-06 secs_since: 0 2024-03-11T21:20:23.044 INFO:journalctl@ceph.mon.a.smithi092.stdout: Block cache entry stats(count,size,portion): FilterBlock(2,0.83 KB,0.000157952%) IndexBlock(2,0.42 KB,8.04663e-05%) Misc(1,0.00 KB,0%) 2024-03-11T21:20:23.044 INFO:journalctl@ceph.mon.a.smithi092.stdout: 2024-03-11T21:20:23.044 INFO:journalctl@ceph.mon.a.smithi092.stdout: ** File Read Latency Histogram By Level [default] ** 2024-03-11T21:20:23.044 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:20:22 smithi092 ceph-mon[29572]: starting mon.a rank 0 at public addrs v1:172.21.15.92:6789/0 at bind addrs v1:172.21.15.92:6789/0 mon_data /var/lib/ceph/mon/ceph-a fsid 00db0016-dfed-11ee-95c8-87774f69a715 2024-03-11T21:20:23.044 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:20:22 smithi092 ceph-mon[29572]: mon.a@-1(???) e1 preinit fsid 00db0016-dfed-11ee-95c8-87774f69a715 2024-03-11T21:20:23.044 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:20:22 smithi092 ceph-mon[29572]: mon.a@-1(???).mds e1 new map 2024-03-11T21:20:23.045 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:20:22 smithi092 ceph-mon[29572]: mon.a@-1(???).mds e1 print_map 2024-03-11T21:20:23.045 INFO:journalctl@ceph.mon.a.smithi092.stdout: e1 2024-03-11T21:20:23.045 INFO:journalctl@ceph.mon.a.smithi092.stdout: enable_multiple, ever_enabled_multiple: 1,1 2024-03-11T21:20:23.045 INFO:journalctl@ceph.mon.a.smithi092.stdout: default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2,11=minor log segments} 2024-03-11T21:20:23.045 INFO:journalctl@ceph.mon.a.smithi092.stdout: legacy client fscid: -1 2024-03-11T21:20:23.045 INFO:journalctl@ceph.mon.a.smithi092.stdout: 2024-03-11T21:20:23.045 INFO:journalctl@ceph.mon.a.smithi092.stdout: No filesystems configured 2024-03-11T21:20:23.045 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:20:22 smithi092 ceph-mon[29572]: mon.a@-1(???).osd e1 crush map has features 3314932999778484224, adjusting msgr requires 2024-03-11T21:20:23.045 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:20:22 smithi092 ceph-mon[29572]: mon.a@-1(???).osd e1 crush map has features 288514050185494528, adjusting msgr requires 2024-03-11T21:20:23.045 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:20:22 smithi092 ceph-mon[29572]: mon.a@-1(???).osd e1 crush map has features 288514050185494528, adjusting msgr requires 2024-03-11T21:20:23.045 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:20:22 smithi092 ceph-mon[29572]: mon.a@-1(???).osd e1 crush map has features 288514050185494528, adjusting msgr requires 2024-03-11T21:20:23.045 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:20:22 smithi092 ceph-mon[29572]: mon.a@-1(???).paxosservice(auth 1..2) refresh upgraded, format 0 -> 3 2024-03-11T21:20:23.045 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:20:22 smithi092 bash[29554]: 4cb262f5a41b5058d15402d9dda414ecb4d6e1da08df8a3f7b60b57c80e321a5 2024-03-11T21:20:23.046 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:20:22 smithi092 systemd[1]: Started Ceph mon.a for 00db0016-dfed-11ee-95c8-87774f69a715. 2024-03-11T21:20:23.046 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:20:23 smithi092 ceph-mon[29572]: mon.a is new leader, mons a in quorum (ranks 0) 2024-03-11T21:20:23.046 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:20:23 smithi092 ceph-mon[29572]: monmap e1: 1 mons at {a=v1:172.21.15.92:6789/0} removed_ranks: {} disallowed_leaders: {} 2024-03-11T21:20:23.046 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:20:23 smithi092 ceph-mon[29572]: fsmap 2024-03-11T21:20:23.046 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:20:23 smithi092 ceph-mon[29572]: osdmap e1: 0 total, 0 up, 0 in 2024-03-11T21:20:23.046 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:20:23 smithi092 ceph-mon[29572]: mgrmap e1: no daemons active 2024-03-11T21:20:24.097 INFO:teuthology.orchestra.run.smithi092.stdout:Wrote config to /etc/ceph/ceph.conf 2024-03-11T21:20:24.119 INFO:teuthology.orchestra.run.smithi092.stdout:Wrote keyring to /etc/ceph/ceph.client.admin.keyring 2024-03-11T21:20:24.119 INFO:teuthology.orchestra.run.smithi092.stdout:Creating mgr... 2024-03-11T21:20:24.119 INFO:teuthology.orchestra.run.smithi092.stdout:Verifying port 0.0.0.0:9283 ... 2024-03-11T21:20:24.119 INFO:teuthology.orchestra.run.smithi092.stdout:Verifying port 0.0.0.0:8765 ... 2024-03-11T21:20:24.366 INFO:teuthology.orchestra.run.smithi092.stdout:Non-zero exit code 1 from systemctl reset-failed ceph-00db0016-dfed-11ee-95c8-87774f69a715@mgr.y 2024-03-11T21:20:24.366 INFO:teuthology.orchestra.run.smithi092.stdout:systemctl: stderr Failed to reset failed state of unit ceph-00db0016-dfed-11ee-95c8-87774f69a715@mgr.y.service: Unit ceph-00db0016-dfed-11ee-95c8-87774f69a715@mgr.y.service not loaded. 2024-03-11T21:20:24.515 INFO:teuthology.orchestra.run.smithi092.stdout:systemctl: stderr Created symlink /etc/systemd/system/ceph-00db0016-dfed-11ee-95c8-87774f69a715.target.wants/ceph-00db0016-dfed-11ee-95c8-87774f69a715@mgr.y.service → /etc/systemd/system/ceph-00db0016-dfed-11ee-95c8-87774f69a715@.service. 2024-03-11T21:20:24.855 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:20:24 smithi092 ceph-mon[29572]: from='client.? v1:172.21.15.92:0/1198200651' entity='client.admin' 2024-03-11T21:20:25.044 INFO:teuthology.orchestra.run.smithi092.stdout:Non-zero exit code 1 from systemctl is-enabled firewalld.service 2024-03-11T21:20:25.044 INFO:teuthology.orchestra.run.smithi092.stdout:systemctl: stdout disabled 2024-03-11T21:20:25.051 INFO:teuthology.orchestra.run.smithi092.stdout:Non-zero exit code 3 from systemctl is-active firewalld.service 2024-03-11T21:20:25.051 INFO:teuthology.orchestra.run.smithi092.stdout:systemctl: stdout inactive 2024-03-11T21:20:25.051 INFO:teuthology.orchestra.run.smithi092.stdout:firewalld.service is not enabled 2024-03-11T21:20:25.051 INFO:teuthology.orchestra.run.smithi092.stdout:Not possible to enable service . firewalld.service is not available 2024-03-11T21:20:25.061 INFO:teuthology.orchestra.run.smithi092.stdout:Non-zero exit code 1 from systemctl is-enabled firewalld.service 2024-03-11T21:20:25.061 INFO:teuthology.orchestra.run.smithi092.stdout:systemctl: stdout disabled 2024-03-11T21:20:25.070 INFO:teuthology.orchestra.run.smithi092.stdout:Non-zero exit code 3 from systemctl is-active firewalld.service 2024-03-11T21:20:25.070 INFO:teuthology.orchestra.run.smithi092.stdout:systemctl: stdout inactive 2024-03-11T21:20:25.070 INFO:teuthology.orchestra.run.smithi092.stdout:firewalld.service is not enabled 2024-03-11T21:20:25.070 INFO:teuthology.orchestra.run.smithi092.stdout:Not possible to open ports <[9283, 8765]>. firewalld.service is not available 2024-03-11T21:20:25.070 INFO:teuthology.orchestra.run.smithi092.stdout:Waiting for mgr to start... 2024-03-11T21:20:25.070 INFO:teuthology.orchestra.run.smithi092.stdout:Waiting for mgr... 2024-03-11T21:20:25.113 INFO:journalctl@ceph.mgr.y.smithi092.stdout:Mar 11 21:20:25 smithi092 systemd[1]: Started Ceph mgr.y for 00db0016-dfed-11ee-95c8-87774f69a715. 2024-03-11T21:20:25.862 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:20:25 smithi092 ceph-mon[29572]: from='client.? v1:172.21.15.92:0/454045153' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2024-03-11T21:20:25.863 INFO:journalctl@ceph.mgr.y.smithi092.stdout:Mar 11 21:20:25 smithi092 ceph-00db0016-dfed-11ee-95c8-87774f69a715-mgr-y[29801]: 2024-03-11T21:20:25.534+0000 7f315c5af200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-03-11T21:20:25.863 INFO:journalctl@ceph.mgr.y.smithi092.stdout:Mar 11 21:20:25 smithi092 ceph-00db0016-dfed-11ee-95c8-87774f69a715-mgr-y[29801]: 2024-03-11T21:20:25.632+0000 7f315c5af200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-03-11T21:20:26.188 INFO:teuthology.orchestra.run.smithi092.stdout:/usr/bin/ceph: stdout 2024-03-11T21:20:26.189 INFO:teuthology.orchestra.run.smithi092.stdout:/usr/bin/ceph: stdout { 2024-03-11T21:20:26.189 INFO:teuthology.orchestra.run.smithi092.stdout:/usr/bin/ceph: stdout "fsid": "00db0016-dfed-11ee-95c8-87774f69a715", 2024-03-11T21:20:26.189 INFO:teuthology.orchestra.run.smithi092.stdout:/usr/bin/ceph: stdout "health": { 2024-03-11T21:20:26.189 INFO:teuthology.orchestra.run.smithi092.stdout:/usr/bin/ceph: stdout "status": "HEALTH_OK", 2024-03-11T21:20:26.189 INFO:teuthology.orchestra.run.smithi092.stdout:/usr/bin/ceph: stdout "checks": {}, 2024-03-11T21:20:26.189 INFO:teuthology.orchestra.run.smithi092.stdout:/usr/bin/ceph: stdout "mutes": [] 2024-03-11T21:20:26.189 INFO:teuthology.orchestra.run.smithi092.stdout:/usr/bin/ceph: stdout }, 2024-03-11T21:20:26.189 INFO:teuthology.orchestra.run.smithi092.stdout:/usr/bin/ceph: stdout "election_epoch": 5, 2024-03-11T21:20:26.189 INFO:teuthology.orchestra.run.smithi092.stdout:/usr/bin/ceph: stdout "quorum": [ 2024-03-11T21:20:26.189 INFO:teuthology.orchestra.run.smithi092.stdout:/usr/bin/ceph: stdout 0 2024-03-11T21:20:26.189 INFO:teuthology.orchestra.run.smithi092.stdout:/usr/bin/ceph: stdout ], 2024-03-11T21:20:26.190 INFO:teuthology.orchestra.run.smithi092.stdout:/usr/bin/ceph: stdout "quorum_names": [ 2024-03-11T21:20:26.190 INFO:teuthology.orchestra.run.smithi092.stdout:/usr/bin/ceph: stdout "a" 2024-03-11T21:20:26.190 INFO:teuthology.orchestra.run.smithi092.stdout:/usr/bin/ceph: stdout ], 2024-03-11T21:20:26.190 INFO:teuthology.orchestra.run.smithi092.stdout:/usr/bin/ceph: stdout "quorum_age": 2, 2024-03-11T21:20:26.190 INFO:teuthology.orchestra.run.smithi092.stdout:/usr/bin/ceph: stdout "monmap": { 2024-03-11T21:20:26.190 INFO:teuthology.orchestra.run.smithi092.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-11T21:20:26.190 INFO:teuthology.orchestra.run.smithi092.stdout:/usr/bin/ceph: stdout "min_mon_release_name": "squid", 2024-03-11T21:20:26.190 INFO:teuthology.orchestra.run.smithi092.stdout:/usr/bin/ceph: stdout "num_mons": 1 2024-03-11T21:20:26.190 INFO:teuthology.orchestra.run.smithi092.stdout:/usr/bin/ceph: stdout }, 2024-03-11T21:20:26.190 INFO:teuthology.orchestra.run.smithi092.stdout:/usr/bin/ceph: stdout "osdmap": { 2024-03-11T21:20:26.191 INFO:teuthology.orchestra.run.smithi092.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-11T21:20:26.191 INFO:teuthology.orchestra.run.smithi092.stdout:/usr/bin/ceph: stdout "num_osds": 0, 2024-03-11T21:20:26.191 INFO:teuthology.orchestra.run.smithi092.stdout:/usr/bin/ceph: stdout "num_up_osds": 0, 2024-03-11T21:20:26.191 INFO:teuthology.orchestra.run.smithi092.stdout:/usr/bin/ceph: stdout "osd_up_since": 0, 2024-03-11T21:20:26.191 INFO:teuthology.orchestra.run.smithi092.stdout:/usr/bin/ceph: stdout "num_in_osds": 0, 2024-03-11T21:20:26.191 INFO:teuthology.orchestra.run.smithi092.stdout:/usr/bin/ceph: stdout "osd_in_since": 0, 2024-03-11T21:20:26.191 INFO:teuthology.orchestra.run.smithi092.stdout:/usr/bin/ceph: stdout "num_remapped_pgs": 0 2024-03-11T21:20:26.191 INFO:teuthology.orchestra.run.smithi092.stdout:/usr/bin/ceph: stdout }, 2024-03-11T21:20:26.191 INFO:teuthology.orchestra.run.smithi092.stdout:/usr/bin/ceph: stdout "pgmap": { 2024-03-11T21:20:26.191 INFO:teuthology.orchestra.run.smithi092.stdout:/usr/bin/ceph: stdout "pgs_by_state": [], 2024-03-11T21:20:26.191 INFO:teuthology.orchestra.run.smithi092.stdout:/usr/bin/ceph: stdout "num_pgs": 0, 2024-03-11T21:20:26.191 INFO:teuthology.orchestra.run.smithi092.stdout:/usr/bin/ceph: stdout "num_pools": 0, 2024-03-11T21:20:26.191 INFO:teuthology.orchestra.run.smithi092.stdout:/usr/bin/ceph: stdout "num_objects": 0, 2024-03-11T21:20:26.191 INFO:teuthology.orchestra.run.smithi092.stdout:/usr/bin/ceph: stdout "data_bytes": 0, 2024-03-11T21:20:26.191 INFO:teuthology.orchestra.run.smithi092.stdout:/usr/bin/ceph: stdout "bytes_used": 0, 2024-03-11T21:20:26.192 INFO:teuthology.orchestra.run.smithi092.stdout:/usr/bin/ceph: stdout "bytes_avail": 0, 2024-03-11T21:20:26.192 INFO:teuthology.orchestra.run.smithi092.stdout:/usr/bin/ceph: stdout "bytes_total": 0 2024-03-11T21:20:26.192 INFO:teuthology.orchestra.run.smithi092.stdout:/usr/bin/ceph: stdout }, 2024-03-11T21:20:26.192 INFO:teuthology.orchestra.run.smithi092.stdout:/usr/bin/ceph: stdout "fsmap": { 2024-03-11T21:20:26.192 INFO:teuthology.orchestra.run.smithi092.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-11T21:20:26.192 INFO:teuthology.orchestra.run.smithi092.stdout:/usr/bin/ceph: stdout "by_rank": [], 2024-03-11T21:20:26.192 INFO:teuthology.orchestra.run.smithi092.stdout:/usr/bin/ceph: stdout "up:standby": 0 2024-03-11T21:20:26.192 INFO:teuthology.orchestra.run.smithi092.stdout:/usr/bin/ceph: stdout }, 2024-03-11T21:20:26.192 INFO:teuthology.orchestra.run.smithi092.stdout:/usr/bin/ceph: stdout "mgrmap": { 2024-03-11T21:20:26.192 INFO:teuthology.orchestra.run.smithi092.stdout:/usr/bin/ceph: stdout "available": false, 2024-03-11T21:20:26.192 INFO:teuthology.orchestra.run.smithi092.stdout:/usr/bin/ceph: stdout "num_standbys": 0, 2024-03-11T21:20:26.192 INFO:teuthology.orchestra.run.smithi092.stdout:/usr/bin/ceph: stdout "modules": [ 2024-03-11T21:20:26.192 INFO:teuthology.orchestra.run.smithi092.stdout:/usr/bin/ceph: stdout "iostat", 2024-03-11T21:20:26.192 INFO:teuthology.orchestra.run.smithi092.stdout:/usr/bin/ceph: stdout "nfs", 2024-03-11T21:20:26.192 INFO:teuthology.orchestra.run.smithi092.stdout:/usr/bin/ceph: stdout "restful" 2024-03-11T21:20:26.193 INFO:teuthology.orchestra.run.smithi092.stdout:/usr/bin/ceph: stdout ], 2024-03-11T21:20:26.193 INFO:teuthology.orchestra.run.smithi092.stdout:/usr/bin/ceph: stdout "services": {} 2024-03-11T21:20:26.193 INFO:teuthology.orchestra.run.smithi092.stdout:/usr/bin/ceph: stdout }, 2024-03-11T21:20:26.193 INFO:teuthology.orchestra.run.smithi092.stdout:/usr/bin/ceph: stdout "servicemap": { 2024-03-11T21:20:26.193 INFO:teuthology.orchestra.run.smithi092.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-11T21:20:26.193 INFO:teuthology.orchestra.run.smithi092.stdout:/usr/bin/ceph: stdout "modified": "2024-03-11T21:20:10.488509+0000", 2024-03-11T21:20:26.193 INFO:teuthology.orchestra.run.smithi092.stdout:/usr/bin/ceph: stdout "services": {} 2024-03-11T21:20:26.193 INFO:teuthology.orchestra.run.smithi092.stdout:/usr/bin/ceph: stdout }, 2024-03-11T21:20:26.193 INFO:teuthology.orchestra.run.smithi092.stdout:/usr/bin/ceph: stdout "progress_events": {} 2024-03-11T21:20:26.193 INFO:teuthology.orchestra.run.smithi092.stdout:/usr/bin/ceph: stdout } 2024-03-11T21:20:26.193 INFO:teuthology.orchestra.run.smithi092.stdout:mgr not available, waiting (1/15)... 2024-03-11T21:20:26.596 INFO:journalctl@ceph.mgr.y.smithi092.stdout:Mar 11 21:20:26 smithi092 ceph-00db0016-dfed-11ee-95c8-87774f69a715-mgr-y[29801]: 2024-03-11T21:20:26.288+0000 7f315c5af200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-03-11T21:20:26.596 INFO:journalctl@ceph.mgr.y.smithi092.stdout:Mar 11 21:20:26 smithi092 ceph-00db0016-dfed-11ee-95c8-87774f69a715-mgr-y[29801]: 2024-03-11T21:20:26.379+0000 7f315c5af200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-03-11T21:20:26.596 INFO:journalctl@ceph.mgr.y.smithi092.stdout:Mar 11 21:20:26 smithi092 ceph-00db0016-dfed-11ee-95c8-87774f69a715-mgr-y[29801]: 2024-03-11T21:20:26.518+0000 7f315c5af200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-03-11T21:20:26.862 INFO:journalctl@ceph.mgr.y.smithi092.stdout:Mar 11 21:20:26 smithi092 ceph-00db0016-dfed-11ee-95c8-87774f69a715-mgr-y[29801]: 2024-03-11T21:20:26.611+0000 7f315c5af200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-03-11T21:20:26.863 INFO:journalctl@ceph.mgr.y.smithi092.stdout:Mar 11 21:20:26 smithi092 ceph-00db0016-dfed-11ee-95c8-87774f69a715-mgr-y[29801]: 2024-03-11T21:20:26.692+0000 7f315c5af200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-03-11T21:20:27.220 INFO:journalctl@ceph.mgr.y.smithi092.stdout:Mar 11 21:20:26 smithi092 ceph-00db0016-dfed-11ee-95c8-87774f69a715-mgr-y[29801]: 2024-03-11T21:20:26.951+0000 7f315c5af200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-03-11T21:20:27.220 INFO:journalctl@ceph.mgr.y.smithi092.stdout:Mar 11 21:20:27 smithi092 ceph-00db0016-dfed-11ee-95c8-87774f69a715-mgr-y[29801]: 2024-03-11T21:20:27.058+0000 7f315c5af200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-03-11T21:20:27.220 INFO:journalctl@ceph.mgr.y.smithi092.stdout:Mar 11 21:20:27 smithi092 ceph-00db0016-dfed-11ee-95c8-87774f69a715-mgr-y[29801]: 2024-03-11T21:20:27.142+0000 7f315c5af200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-03-11T21:20:27.220 INFO:journalctl@ceph.mgr.y.smithi092.stdout:Mar 11 21:20:27 smithi092 ceph-00db0016-dfed-11ee-95c8-87774f69a715-mgr-y[29801]: 2024-03-11T21:20:27.235+0000 7f315c5af200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-03-11T21:20:27.612 INFO:journalctl@ceph.mgr.y.smithi092.stdout:Mar 11 21:20:27 smithi092 ceph-00db0016-dfed-11ee-95c8-87774f69a715-mgr-y[29801]: 2024-03-11T21:20:27.407+0000 7f315c5af200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-03-11T21:20:28.612 INFO:journalctl@ceph.mgr.y.smithi092.stdout:Mar 11 21:20:28 smithi092 ceph-00db0016-dfed-11ee-95c8-87774f69a715-mgr-y[29801]: 2024-03-11T21:20:28.379+0000 7f315c5af200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-03-11T21:20:28.612 INFO:journalctl@ceph.mgr.y.smithi092.stdout:Mar 11 21:20:28 smithi092 ceph-00db0016-dfed-11ee-95c8-87774f69a715-mgr-y[29801]: 2024-03-11T21:20:28.509+0000 7f315c5af200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-03-11T21:20:29.103 INFO:journalctl@ceph.mgr.y.smithi092.stdout:Mar 11 21:20:28 smithi092 ceph-00db0016-dfed-11ee-95c8-87774f69a715-mgr-y[29801]: 2024-03-11T21:20:28.837+0000 7f315c5af200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-03-11T21:20:29.103 INFO:journalctl@ceph.mgr.y.smithi092.stdout:Mar 11 21:20:28 smithi092 ceph-00db0016-dfed-11ee-95c8-87774f69a715-mgr-y[29801]: 2024-03-11T21:20:28.926+0000 7f315c5af200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-03-11T21:20:29.104 INFO:journalctl@ceph.mgr.y.smithi092.stdout:Mar 11 21:20:29 smithi092 ceph-00db0016-dfed-11ee-95c8-87774f69a715-mgr-y[29801]: 2024-03-11T21:20:29.118+0000 7f315c5af200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-03-11T21:20:29.362 INFO:journalctl@ceph.mgr.y.smithi092.stdout:Mar 11 21:20:29 smithi092 ceph-00db0016-dfed-11ee-95c8-87774f69a715-mgr-y[29801]: 2024-03-11T21:20:29.300+0000 7f315c5af200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-03-11T21:20:29.743 INFO:journalctl@ceph.mgr.y.smithi092.stdout:Mar 11 21:20:29 smithi092 ceph-00db0016-dfed-11ee-95c8-87774f69a715-mgr-y[29801]: 2024-03-11T21:20:29.586+0000 7f315c5af200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-03-11T21:20:30.085 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:20:29 smithi092 ceph-mon[29572]: from='client.? v1:172.21.15.92:0/1175239725' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2024-03-11T21:20:30.085 INFO:journalctl@ceph.mgr.y.smithi092.stdout:Mar 11 21:20:29 smithi092 ceph-00db0016-dfed-11ee-95c8-87774f69a715-mgr-y[29801]: 2024-03-11T21:20:29.849+0000 7f315c5af200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-03-11T21:20:30.362 INFO:journalctl@ceph.mgr.y.smithi092.stdout:Mar 11 21:20:30 smithi092 ceph-00db0016-dfed-11ee-95c8-87774f69a715-mgr-y[29801]: 2024-03-11T21:20:30.100+0000 7f315c5af200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-03-11T21:20:30.436 INFO:teuthology.orchestra.run.smithi092.stdout:/usr/bin/ceph: stdout 2024-03-11T21:20:30.436 INFO:teuthology.orchestra.run.smithi092.stdout:/usr/bin/ceph: stdout { 2024-03-11T21:20:30.436 INFO:teuthology.orchestra.run.smithi092.stdout:/usr/bin/ceph: stdout "fsid": "00db0016-dfed-11ee-95c8-87774f69a715", 2024-03-11T21:20:30.436 INFO:teuthology.orchestra.run.smithi092.stdout:/usr/bin/ceph: stdout "health": { 2024-03-11T21:20:30.436 INFO:teuthology.orchestra.run.smithi092.stdout:/usr/bin/ceph: stdout "status": "HEALTH_OK", 2024-03-11T21:20:30.436 INFO:teuthology.orchestra.run.smithi092.stdout:/usr/bin/ceph: stdout "checks": {}, 2024-03-11T21:20:30.436 INFO:teuthology.orchestra.run.smithi092.stdout:/usr/bin/ceph: stdout "mutes": [] 2024-03-11T21:20:30.436 INFO:teuthology.orchestra.run.smithi092.stdout:/usr/bin/ceph: stdout }, 2024-03-11T21:20:30.436 INFO:teuthology.orchestra.run.smithi092.stdout:/usr/bin/ceph: stdout "election_epoch": 5, 2024-03-11T21:20:30.436 INFO:teuthology.orchestra.run.smithi092.stdout:/usr/bin/ceph: stdout "quorum": [ 2024-03-11T21:20:30.436 INFO:teuthology.orchestra.run.smithi092.stdout:/usr/bin/ceph: stdout 0 2024-03-11T21:20:30.437 INFO:teuthology.orchestra.run.smithi092.stdout:/usr/bin/ceph: stdout ], 2024-03-11T21:20:30.437 INFO:teuthology.orchestra.run.smithi092.stdout:/usr/bin/ceph: stdout "quorum_names": [ 2024-03-11T21:20:30.437 INFO:teuthology.orchestra.run.smithi092.stdout:/usr/bin/ceph: stdout "a" 2024-03-11T21:20:30.437 INFO:teuthology.orchestra.run.smithi092.stdout:/usr/bin/ceph: stdout ], 2024-03-11T21:20:30.437 INFO:teuthology.orchestra.run.smithi092.stdout:/usr/bin/ceph: stdout "quorum_age": 6, 2024-03-11T21:20:30.437 INFO:teuthology.orchestra.run.smithi092.stdout:/usr/bin/ceph: stdout "monmap": { 2024-03-11T21:20:30.437 INFO:teuthology.orchestra.run.smithi092.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-11T21:20:30.437 INFO:teuthology.orchestra.run.smithi092.stdout:/usr/bin/ceph: stdout "min_mon_release_name": "squid", 2024-03-11T21:20:30.437 INFO:teuthology.orchestra.run.smithi092.stdout:/usr/bin/ceph: stdout "num_mons": 1 2024-03-11T21:20:30.437 INFO:teuthology.orchestra.run.smithi092.stdout:/usr/bin/ceph: stdout }, 2024-03-11T21:20:30.437 INFO:teuthology.orchestra.run.smithi092.stdout:/usr/bin/ceph: stdout "osdmap": { 2024-03-11T21:20:30.437 INFO:teuthology.orchestra.run.smithi092.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-11T21:20:30.437 INFO:teuthology.orchestra.run.smithi092.stdout:/usr/bin/ceph: stdout "num_osds": 0, 2024-03-11T21:20:30.437 INFO:teuthology.orchestra.run.smithi092.stdout:/usr/bin/ceph: stdout "num_up_osds": 0, 2024-03-11T21:20:30.438 INFO:teuthology.orchestra.run.smithi092.stdout:/usr/bin/ceph: stdout "osd_up_since": 0, 2024-03-11T21:20:30.438 INFO:teuthology.orchestra.run.smithi092.stdout:/usr/bin/ceph: stdout "num_in_osds": 0, 2024-03-11T21:20:30.438 INFO:teuthology.orchestra.run.smithi092.stdout:/usr/bin/ceph: stdout "osd_in_since": 0, 2024-03-11T21:20:30.438 INFO:teuthology.orchestra.run.smithi092.stdout:/usr/bin/ceph: stdout "num_remapped_pgs": 0 2024-03-11T21:20:30.441 INFO:teuthology.orchestra.run.smithi092.stdout:/usr/bin/ceph: stdout }, 2024-03-11T21:20:30.441 INFO:teuthology.orchestra.run.smithi092.stdout:/usr/bin/ceph: stdout "pgmap": { 2024-03-11T21:20:30.441 INFO:teuthology.orchestra.run.smithi092.stdout:/usr/bin/ceph: stdout "pgs_by_state": [], 2024-03-11T21:20:30.441 INFO:teuthology.orchestra.run.smithi092.stdout:/usr/bin/ceph: stdout "num_pgs": 0, 2024-03-11T21:20:30.441 INFO:teuthology.orchestra.run.smithi092.stdout:/usr/bin/ceph: stdout "num_pools": 0, 2024-03-11T21:20:30.441 INFO:teuthology.orchestra.run.smithi092.stdout:/usr/bin/ceph: stdout "num_objects": 0, 2024-03-11T21:20:30.441 INFO:teuthology.orchestra.run.smithi092.stdout:/usr/bin/ceph: stdout "data_bytes": 0, 2024-03-11T21:20:30.441 INFO:teuthology.orchestra.run.smithi092.stdout:/usr/bin/ceph: stdout "bytes_used": 0, 2024-03-11T21:20:30.441 INFO:teuthology.orchestra.run.smithi092.stdout:/usr/bin/ceph: stdout "bytes_avail": 0, 2024-03-11T21:20:30.441 INFO:teuthology.orchestra.run.smithi092.stdout:/usr/bin/ceph: stdout "bytes_total": 0 2024-03-11T21:20:30.442 INFO:teuthology.orchestra.run.smithi092.stdout:/usr/bin/ceph: stdout }, 2024-03-11T21:20:30.442 INFO:teuthology.orchestra.run.smithi092.stdout:/usr/bin/ceph: stdout "fsmap": { 2024-03-11T21:20:30.442 INFO:teuthology.orchestra.run.smithi092.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-11T21:20:30.442 INFO:teuthology.orchestra.run.smithi092.stdout:/usr/bin/ceph: stdout "by_rank": [], 2024-03-11T21:20:30.442 INFO:teuthology.orchestra.run.smithi092.stdout:/usr/bin/ceph: stdout "up:standby": 0 2024-03-11T21:20:30.442 INFO:teuthology.orchestra.run.smithi092.stdout:/usr/bin/ceph: stdout }, 2024-03-11T21:20:30.442 INFO:teuthology.orchestra.run.smithi092.stdout:/usr/bin/ceph: stdout "mgrmap": { 2024-03-11T21:20:30.442 INFO:teuthology.orchestra.run.smithi092.stdout:/usr/bin/ceph: stdout "available": false, 2024-03-11T21:20:30.442 INFO:teuthology.orchestra.run.smithi092.stdout:/usr/bin/ceph: stdout "num_standbys": 0, 2024-03-11T21:20:30.442 INFO:teuthology.orchestra.run.smithi092.stdout:/usr/bin/ceph: stdout "modules": [ 2024-03-11T21:20:30.442 INFO:teuthology.orchestra.run.smithi092.stdout:/usr/bin/ceph: stdout "iostat", 2024-03-11T21:20:30.442 INFO:teuthology.orchestra.run.smithi092.stdout:/usr/bin/ceph: stdout "nfs", 2024-03-11T21:20:30.442 INFO:teuthology.orchestra.run.smithi092.stdout:/usr/bin/ceph: stdout "restful" 2024-03-11T21:20:30.442 INFO:teuthology.orchestra.run.smithi092.stdout:/usr/bin/ceph: stdout ], 2024-03-11T21:20:30.442 INFO:teuthology.orchestra.run.smithi092.stdout:/usr/bin/ceph: stdout "services": {} 2024-03-11T21:20:30.443 INFO:teuthology.orchestra.run.smithi092.stdout:/usr/bin/ceph: stdout }, 2024-03-11T21:20:30.443 INFO:teuthology.orchestra.run.smithi092.stdout:/usr/bin/ceph: stdout "servicemap": { 2024-03-11T21:20:30.443 INFO:teuthology.orchestra.run.smithi092.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-11T21:20:30.443 INFO:teuthology.orchestra.run.smithi092.stdout:/usr/bin/ceph: stdout "modified": "2024-03-11T21:20:10.488509+0000", 2024-03-11T21:20:30.443 INFO:teuthology.orchestra.run.smithi092.stdout:/usr/bin/ceph: stdout "services": {} 2024-03-11T21:20:30.443 INFO:teuthology.orchestra.run.smithi092.stdout:/usr/bin/ceph: stdout }, 2024-03-11T21:20:30.443 INFO:teuthology.orchestra.run.smithi092.stdout:/usr/bin/ceph: stdout "progress_events": {} 2024-03-11T21:20:30.443 INFO:teuthology.orchestra.run.smithi092.stdout:/usr/bin/ceph: stdout } 2024-03-11T21:20:30.443 INFO:teuthology.orchestra.run.smithi092.stdout:mgr not available, waiting (2/15)... 2024-03-11T21:20:31.352 INFO:journalctl@ceph.mgr.y.smithi092.stdout:Mar 11 21:20:30 smithi092 ceph-00db0016-dfed-11ee-95c8-87774f69a715-mgr-y[29801]: 2024-03-11T21:20:30.899+0000 7f315c5af200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-03-11T21:20:31.612 INFO:journalctl@ceph.mgr.y.smithi092.stdout:Mar 11 21:20:31 smithi092 ceph-00db0016-dfed-11ee-95c8-87774f69a715-mgr-y[29801]: 2024-03-11T21:20:31.505+0000 7f315c5af200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-03-11T21:20:31.612 INFO:journalctl@ceph.mgr.y.smithi092.stdout:Mar 11 21:20:31 smithi092 ceph-00db0016-dfed-11ee-95c8-87774f69a715-mgr-y[29801]: 2024-03-11T21:20:31.608+0000 7f315c5af200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-03-11T21:20:32.112 INFO:journalctl@ceph.mgr.y.smithi092.stdout:Mar 11 21:20:31 smithi092 ceph-00db0016-dfed-11ee-95c8-87774f69a715-mgr-y[29801]: 2024-03-11T21:20:31.828+0000 7f315c5af200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-03-11T21:20:32.113 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:20:31 smithi092 ceph-mon[29572]: Activating manager daemon y 2024-03-11T21:20:32.113 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:20:31 smithi092 ceph-mon[29572]: mgrmap e2: y(active, starting, since 0.00295041s) 2024-03-11T21:20:32.113 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:20:31 smithi092 ceph-mon[29572]: from='mgr.14100 v1:172.21.15.92:0/2571648254' entity='mgr.y' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-03-11T21:20:32.113 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:20:31 smithi092 ceph-mon[29572]: from='mgr.14100 v1:172.21.15.92:0/2571648254' entity='mgr.y' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-03-11T21:20:32.113 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:20:31 smithi092 ceph-mon[29572]: from='mgr.14100 v1:172.21.15.92:0/2571648254' entity='mgr.y' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-03-11T21:20:32.113 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:20:31 smithi092 ceph-mon[29572]: from='mgr.14100 v1:172.21.15.92:0/2571648254' entity='mgr.y' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-03-11T21:20:32.113 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:20:31 smithi092 ceph-mon[29572]: from='mgr.14100 v1:172.21.15.92:0/2571648254' entity='mgr.y' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-03-11T21:20:32.113 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:20:31 smithi092 ceph-mon[29572]: from='mgr.14100 v1:172.21.15.92:0/2571648254' entity='mgr.y' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-03-11T21:20:32.113 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:20:31 smithi092 ceph-mon[29572]: from='mgr.14100 v1:172.21.15.92:0/2571648254' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-11T21:20:32.113 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:20:31 smithi092 ceph-mon[29572]: from='mgr.14100 v1:172.21.15.92:0/2571648254' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-03-11T21:20:32.114 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:20:31 smithi092 ceph-mon[29572]: Manager daemon y is now available 2024-03-11T21:20:32.114 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:20:31 smithi092 ceph-mon[29572]: from='mgr.14100 v1:172.21.15.92:0/2571648254' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-03-11T21:20:32.114 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:20:31 smithi092 ceph-mon[29572]: from='mgr.14100 v1:172.21.15.92:0/2571648254' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-03-11T21:20:32.114 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:20:31 smithi092 ceph-mon[29572]: from='mgr.14100 v1:172.21.15.92:0/2571648254' entity='mgr.y' 2024-03-11T21:20:32.114 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:20:31 smithi092 ceph-mon[29572]: from='mgr.14100 v1:172.21.15.92:0/2571648254' entity='mgr.y' 2024-03-11T21:20:32.114 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:20:31 smithi092 ceph-mon[29572]: from='mgr.14100 v1:172.21.15.92:0/2571648254' entity='mgr.y' 2024-03-11T21:20:33.661 INFO:teuthology.orchestra.run.smithi092.stdout:/usr/bin/ceph: stdout 2024-03-11T21:20:33.662 INFO:teuthology.orchestra.run.smithi092.stdout:/usr/bin/ceph: stdout { 2024-03-11T21:20:33.662 INFO:teuthology.orchestra.run.smithi092.stdout:/usr/bin/ceph: stdout "fsid": "00db0016-dfed-11ee-95c8-87774f69a715", 2024-03-11T21:20:33.662 INFO:teuthology.orchestra.run.smithi092.stdout:/usr/bin/ceph: stdout "health": { 2024-03-11T21:20:33.662 INFO:teuthology.orchestra.run.smithi092.stdout:/usr/bin/ceph: stdout "status": "HEALTH_OK", 2024-03-11T21:20:33.662 INFO:teuthology.orchestra.run.smithi092.stdout:/usr/bin/ceph: stdout "checks": {}, 2024-03-11T21:20:33.662 INFO:teuthology.orchestra.run.smithi092.stdout:/usr/bin/ceph: stdout "mutes": [] 2024-03-11T21:20:33.662 INFO:teuthology.orchestra.run.smithi092.stdout:/usr/bin/ceph: stdout }, 2024-03-11T21:20:33.662 INFO:teuthology.orchestra.run.smithi092.stdout:/usr/bin/ceph: stdout "election_epoch": 5, 2024-03-11T21:20:33.662 INFO:teuthology.orchestra.run.smithi092.stdout:/usr/bin/ceph: stdout "quorum": [ 2024-03-11T21:20:33.662 INFO:teuthology.orchestra.run.smithi092.stdout:/usr/bin/ceph: stdout 0 2024-03-11T21:20:33.662 INFO:teuthology.orchestra.run.smithi092.stdout:/usr/bin/ceph: stdout ], 2024-03-11T21:20:33.662 INFO:teuthology.orchestra.run.smithi092.stdout:/usr/bin/ceph: stdout "quorum_names": [ 2024-03-11T21:20:33.663 INFO:teuthology.orchestra.run.smithi092.stdout:/usr/bin/ceph: stdout "a" 2024-03-11T21:20:33.663 INFO:teuthology.orchestra.run.smithi092.stdout:/usr/bin/ceph: stdout ], 2024-03-11T21:20:33.663 INFO:teuthology.orchestra.run.smithi092.stdout:/usr/bin/ceph: stdout "quorum_age": 10, 2024-03-11T21:20:33.663 INFO:teuthology.orchestra.run.smithi092.stdout:/usr/bin/ceph: stdout "monmap": { 2024-03-11T21:20:33.663 INFO:teuthology.orchestra.run.smithi092.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-11T21:20:33.663 INFO:teuthology.orchestra.run.smithi092.stdout:/usr/bin/ceph: stdout "min_mon_release_name": "squid", 2024-03-11T21:20:33.663 INFO:teuthology.orchestra.run.smithi092.stdout:/usr/bin/ceph: stdout "num_mons": 1 2024-03-11T21:20:33.663 INFO:teuthology.orchestra.run.smithi092.stdout:/usr/bin/ceph: stdout }, 2024-03-11T21:20:33.663 INFO:teuthology.orchestra.run.smithi092.stdout:/usr/bin/ceph: stdout "osdmap": { 2024-03-11T21:20:33.663 INFO:teuthology.orchestra.run.smithi092.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-11T21:20:33.663 INFO:teuthology.orchestra.run.smithi092.stdout:/usr/bin/ceph: stdout "num_osds": 0, 2024-03-11T21:20:33.663 INFO:teuthology.orchestra.run.smithi092.stdout:/usr/bin/ceph: stdout "num_up_osds": 0, 2024-03-11T21:20:33.663 INFO:teuthology.orchestra.run.smithi092.stdout:/usr/bin/ceph: stdout "osd_up_since": 0, 2024-03-11T21:20:33.663 INFO:teuthology.orchestra.run.smithi092.stdout:/usr/bin/ceph: stdout "num_in_osds": 0, 2024-03-11T21:20:33.663 INFO:teuthology.orchestra.run.smithi092.stdout:/usr/bin/ceph: stdout "osd_in_since": 0, 2024-03-11T21:20:33.663 INFO:teuthology.orchestra.run.smithi092.stdout:/usr/bin/ceph: stdout "num_remapped_pgs": 0 2024-03-11T21:20:33.664 INFO:teuthology.orchestra.run.smithi092.stdout:/usr/bin/ceph: stdout }, 2024-03-11T21:20:33.664 INFO:teuthology.orchestra.run.smithi092.stdout:/usr/bin/ceph: stdout "pgmap": { 2024-03-11T21:20:33.664 INFO:teuthology.orchestra.run.smithi092.stdout:/usr/bin/ceph: stdout "pgs_by_state": [], 2024-03-11T21:20:33.664 INFO:teuthology.orchestra.run.smithi092.stdout:/usr/bin/ceph: stdout "num_pgs": 0, 2024-03-11T21:20:33.664 INFO:teuthology.orchestra.run.smithi092.stdout:/usr/bin/ceph: stdout "num_pools": 0, 2024-03-11T21:20:33.664 INFO:teuthology.orchestra.run.smithi092.stdout:/usr/bin/ceph: stdout "num_objects": 0, 2024-03-11T21:20:33.664 INFO:teuthology.orchestra.run.smithi092.stdout:/usr/bin/ceph: stdout "data_bytes": 0, 2024-03-11T21:20:33.664 INFO:teuthology.orchestra.run.smithi092.stdout:/usr/bin/ceph: stdout "bytes_used": 0, 2024-03-11T21:20:33.664 INFO:teuthology.orchestra.run.smithi092.stdout:/usr/bin/ceph: stdout "bytes_avail": 0, 2024-03-11T21:20:33.664 INFO:teuthology.orchestra.run.smithi092.stdout:/usr/bin/ceph: stdout "bytes_total": 0 2024-03-11T21:20:33.664 INFO:teuthology.orchestra.run.smithi092.stdout:/usr/bin/ceph: stdout }, 2024-03-11T21:20:33.664 INFO:teuthology.orchestra.run.smithi092.stdout:/usr/bin/ceph: stdout "fsmap": { 2024-03-11T21:20:33.664 INFO:teuthology.orchestra.run.smithi092.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-11T21:20:33.664 INFO:teuthology.orchestra.run.smithi092.stdout:/usr/bin/ceph: stdout "by_rank": [], 2024-03-11T21:20:33.665 INFO:teuthology.orchestra.run.smithi092.stdout:/usr/bin/ceph: stdout "up:standby": 0 2024-03-11T21:20:33.665 INFO:teuthology.orchestra.run.smithi092.stdout:/usr/bin/ceph: stdout }, 2024-03-11T21:20:33.665 INFO:teuthology.orchestra.run.smithi092.stdout:/usr/bin/ceph: stdout "mgrmap": { 2024-03-11T21:20:33.665 INFO:teuthology.orchestra.run.smithi092.stdout:/usr/bin/ceph: stdout "available": true, 2024-03-11T21:20:33.665 INFO:teuthology.orchestra.run.smithi092.stdout:/usr/bin/ceph: stdout "num_standbys": 0, 2024-03-11T21:20:33.665 INFO:teuthology.orchestra.run.smithi092.stdout:/usr/bin/ceph: stdout "modules": [ 2024-03-11T21:20:33.665 INFO:teuthology.orchestra.run.smithi092.stdout:/usr/bin/ceph: stdout "iostat", 2024-03-11T21:20:33.665 INFO:teuthology.orchestra.run.smithi092.stdout:/usr/bin/ceph: stdout "nfs", 2024-03-11T21:20:33.665 INFO:teuthology.orchestra.run.smithi092.stdout:/usr/bin/ceph: stdout "restful" 2024-03-11T21:20:33.665 INFO:teuthology.orchestra.run.smithi092.stdout:/usr/bin/ceph: stdout ], 2024-03-11T21:20:33.665 INFO:teuthology.orchestra.run.smithi092.stdout:/usr/bin/ceph: stdout "services": {} 2024-03-11T21:20:33.665 INFO:teuthology.orchestra.run.smithi092.stdout:/usr/bin/ceph: stdout }, 2024-03-11T21:20:33.665 INFO:teuthology.orchestra.run.smithi092.stdout:/usr/bin/ceph: stdout "servicemap": { 2024-03-11T21:20:33.665 INFO:teuthology.orchestra.run.smithi092.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-11T21:20:33.666 INFO:teuthology.orchestra.run.smithi092.stdout:/usr/bin/ceph: stdout "modified": "2024-03-11T21:20:10.488509+0000", 2024-03-11T21:20:33.666 INFO:teuthology.orchestra.run.smithi092.stdout:/usr/bin/ceph: stdout "services": {} 2024-03-11T21:20:33.666 INFO:teuthology.orchestra.run.smithi092.stdout:/usr/bin/ceph: stdout }, 2024-03-11T21:20:33.666 INFO:teuthology.orchestra.run.smithi092.stdout:/usr/bin/ceph: stdout "progress_events": {} 2024-03-11T21:20:33.666 INFO:teuthology.orchestra.run.smithi092.stdout:/usr/bin/ceph: stdout } 2024-03-11T21:20:33.666 INFO:teuthology.orchestra.run.smithi092.stdout:mgr is available 2024-03-11T21:20:33.973 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:20:33 smithi092 ceph-mon[29572]: mgrmap e3: y(active, since 1.00593s) 2024-03-11T21:20:33.973 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:20:33 smithi092 ceph-mon[29572]: from='client.? v1:172.21.15.92:0/357044903' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2024-03-11T21:20:34.773 INFO:teuthology.orchestra.run.smithi092.stdout:/usr/bin/ceph: stdout 2024-03-11T21:20:34.773 INFO:teuthology.orchestra.run.smithi092.stdout:/usr/bin/ceph: stdout [global] 2024-03-11T21:20:34.773 INFO:teuthology.orchestra.run.smithi092.stdout:/usr/bin/ceph: stdout fsid = 00db0016-dfed-11ee-95c8-87774f69a715 2024-03-11T21:20:34.773 INFO:teuthology.orchestra.run.smithi092.stdout:/usr/bin/ceph: stdout mon_osd_allow_pg_remap = true 2024-03-11T21:20:34.773 INFO:teuthology.orchestra.run.smithi092.stdout:/usr/bin/ceph: stdout mon_osd_allow_primary_affinity = true 2024-03-11T21:20:34.773 INFO:teuthology.orchestra.run.smithi092.stdout:/usr/bin/ceph: stdout mon_warn_on_no_sortbitwise = false 2024-03-11T21:20:34.773 INFO:teuthology.orchestra.run.smithi092.stdout:/usr/bin/ceph: stdout osd_crush_chooseleaf_type = 0 2024-03-11T21:20:34.773 INFO:teuthology.orchestra.run.smithi092.stdout:/usr/bin/ceph: stdout 2024-03-11T21:20:34.773 INFO:teuthology.orchestra.run.smithi092.stdout:/usr/bin/ceph: stdout [mgr] 2024-03-11T21:20:34.773 INFO:teuthology.orchestra.run.smithi092.stdout:/usr/bin/ceph: stdout mgr/telemetry/nag = false 2024-03-11T21:20:34.773 INFO:teuthology.orchestra.run.smithi092.stdout:/usr/bin/ceph: stdout 2024-03-11T21:20:34.773 INFO:teuthology.orchestra.run.smithi092.stdout:/usr/bin/ceph: stdout [osd] 2024-03-11T21:20:34.774 INFO:teuthology.orchestra.run.smithi092.stdout:/usr/bin/ceph: stdout osd_map_max_advance = 10 2024-03-11T21:20:34.774 INFO:teuthology.orchestra.run.smithi092.stdout:/usr/bin/ceph: stdout osd_sloppy_crc = true 2024-03-11T21:20:34.774 INFO:teuthology.orchestra.run.smithi092.stdout:Enabling cephadm module... 2024-03-11T21:20:35.112 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:20:34 smithi092 ceph-mon[29572]: from='client.? v1:172.21.15.92:0/3047179754' entity='client.admin' cmd=[{"prefix": "config assimilate-conf"}]: dispatch 2024-03-11T21:20:36.112 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:20:35 smithi092 ceph-mon[29572]: mgrmap e4: y(active, since 3s) 2024-03-11T21:20:37.112 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:20:36 smithi092 ceph-mon[29572]: from='client.? v1:172.21.15.92:0/11462280' entity='client.admin' cmd=[{"prefix": "mgr module enable", "module": "cephadm"}]: dispatch 2024-03-11T21:20:37.113 INFO:journalctl@ceph.mgr.y.smithi092.stdout:Mar 11 21:20:36 smithi092 ceph-00db0016-dfed-11ee-95c8-87774f69a715-mgr-y[29801]: ignoring --setuser ceph since I am not root 2024-03-11T21:20:37.113 INFO:journalctl@ceph.mgr.y.smithi092.stdout:Mar 11 21:20:36 smithi092 ceph-00db0016-dfed-11ee-95c8-87774f69a715-mgr-y[29801]: ignoring --setgroup ceph since I am not root 2024-03-11T21:20:37.728 INFO:journalctl@ceph.mgr.y.smithi092.stdout:Mar 11 21:20:37 smithi092 ceph-00db0016-dfed-11ee-95c8-87774f69a715-mgr-y[29801]: 2024-03-11T21:20:37.416+0000 7f03ef895200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-03-11T21:20:37.728 INFO:journalctl@ceph.mgr.y.smithi092.stdout:Mar 11 21:20:37 smithi092 ceph-00db0016-dfed-11ee-95c8-87774f69a715-mgr-y[29801]: 2024-03-11T21:20:37.512+0000 7f03ef895200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-03-11T21:20:38.099 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:20:37 smithi092 ceph-mon[29572]: from='client.? v1:172.21.15.92:0/11462280' entity='client.admin' cmd='[{"prefix": "mgr module enable", "module": "cephadm"}]': finished 2024-03-11T21:20:38.099 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:20:37 smithi092 ceph-mon[29572]: mgrmap e5: y(active, since 5s) 2024-03-11T21:20:38.362 INFO:journalctl@ceph.mgr.y.smithi092.stdout:Mar 11 21:20:38 smithi092 ceph-00db0016-dfed-11ee-95c8-87774f69a715-mgr-y[29801]: 2024-03-11T21:20:38.170+0000 7f03ef895200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-03-11T21:20:38.362 INFO:journalctl@ceph.mgr.y.smithi092.stdout:Mar 11 21:20:38 smithi092 ceph-00db0016-dfed-11ee-95c8-87774f69a715-mgr-y[29801]: 2024-03-11T21:20:38.260+0000 7f03ef895200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-03-11T21:20:38.666 INFO:journalctl@ceph.mgr.y.smithi092.stdout:Mar 11 21:20:38 smithi092 ceph-00db0016-dfed-11ee-95c8-87774f69a715-mgr-y[29801]: 2024-03-11T21:20:38.403+0000 7f03ef895200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-03-11T21:20:38.666 INFO:journalctl@ceph.mgr.y.smithi092.stdout:Mar 11 21:20:38 smithi092 ceph-00db0016-dfed-11ee-95c8-87774f69a715-mgr-y[29801]: 2024-03-11T21:20:38.499+0000 7f03ef895200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-03-11T21:20:38.666 INFO:journalctl@ceph.mgr.y.smithi092.stdout:Mar 11 21:20:38 smithi092 ceph-00db0016-dfed-11ee-95c8-87774f69a715-mgr-y[29801]: 2024-03-11T21:20:38.581+0000 7f03ef895200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-03-11T21:20:38.669 INFO:teuthology.orchestra.run.smithi092.stdout:/usr/bin/ceph: stdout { 2024-03-11T21:20:38.669 INFO:teuthology.orchestra.run.smithi092.stdout:/usr/bin/ceph: stdout "epoch": 5, 2024-03-11T21:20:38.669 INFO:teuthology.orchestra.run.smithi092.stdout:/usr/bin/ceph: stdout "available": true, 2024-03-11T21:20:38.669 INFO:teuthology.orchestra.run.smithi092.stdout:/usr/bin/ceph: stdout "active_name": "y", 2024-03-11T21:20:38.669 INFO:teuthology.orchestra.run.smithi092.stdout:/usr/bin/ceph: stdout "num_standby": 0 2024-03-11T21:20:38.669 INFO:teuthology.orchestra.run.smithi092.stdout:/usr/bin/ceph: stdout } 2024-03-11T21:20:38.669 INFO:teuthology.orchestra.run.smithi092.stdout:Waiting for the mgr to restart... 2024-03-11T21:20:38.669 INFO:teuthology.orchestra.run.smithi092.stdout:Waiting for mgr epoch 5... 2024-03-11T21:20:38.948 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:20:38 smithi092 ceph-mon[29572]: from='client.? v1:172.21.15.92:0/692279567' entity='client.admin' cmd=[{"prefix": "mgr stat"}]: dispatch 2024-03-11T21:20:38.948 INFO:journalctl@ceph.mgr.y.smithi092.stdout:Mar 11 21:20:38 smithi092 ceph-00db0016-dfed-11ee-95c8-87774f69a715-mgr-y[29801]: 2024-03-11T21:20:38.849+0000 7f03ef895200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-03-11T21:20:39.308 INFO:journalctl@ceph.mgr.y.smithi092.stdout:Mar 11 21:20:38 smithi092 ceph-00db0016-dfed-11ee-95c8-87774f69a715-mgr-y[29801]: 2024-03-11T21:20:38.963+0000 7f03ef895200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-03-11T21:20:39.308 INFO:journalctl@ceph.mgr.y.smithi092.stdout:Mar 11 21:20:39 smithi092 ceph-00db0016-dfed-11ee-95c8-87774f69a715-mgr-y[29801]: 2024-03-11T21:20:39.054+0000 7f03ef895200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-03-11T21:20:39.308 INFO:journalctl@ceph.mgr.y.smithi092.stdout:Mar 11 21:20:39 smithi092 ceph-00db0016-dfed-11ee-95c8-87774f69a715-mgr-y[29801]: 2024-03-11T21:20:39.151+0000 7f03ef895200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-03-11T21:20:39.612 INFO:journalctl@ceph.mgr.y.smithi092.stdout:Mar 11 21:20:39 smithi092 ceph-00db0016-dfed-11ee-95c8-87774f69a715-mgr-y[29801]: 2024-03-11T21:20:39.323+0000 7f03ef895200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-03-11T21:20:40.612 INFO:journalctl@ceph.mgr.y.smithi092.stdout:Mar 11 21:20:40 smithi092 ceph-00db0016-dfed-11ee-95c8-87774f69a715-mgr-y[29801]: 2024-03-11T21:20:40.270+0000 7f03ef895200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-03-11T21:20:40.613 INFO:journalctl@ceph.mgr.y.smithi092.stdout:Mar 11 21:20:40 smithi092 ceph-00db0016-dfed-11ee-95c8-87774f69a715-mgr-y[29801]: 2024-03-11T21:20:40.397+0000 7f03ef895200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-03-11T21:20:40.995 INFO:journalctl@ceph.mgr.y.smithi092.stdout:Mar 11 21:20:40 smithi092 ceph-00db0016-dfed-11ee-95c8-87774f69a715-mgr-y[29801]: 2024-03-11T21:20:40.725+0000 7f03ef895200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-03-11T21:20:40.995 INFO:journalctl@ceph.mgr.y.smithi092.stdout:Mar 11 21:20:40 smithi092 ceph-00db0016-dfed-11ee-95c8-87774f69a715-mgr-y[29801]: 2024-03-11T21:20:40.815+0000 7f03ef895200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-03-11T21:20:40.995 INFO:journalctl@ceph.mgr.y.smithi092.stdout:Mar 11 21:20:41 smithi092 ceph-00db0016-dfed-11ee-95c8-87774f69a715-mgr-y[29801]: 2024-03-11T21:20:41.010+0000 7f03ef895200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-03-11T21:20:41.362 INFO:journalctl@ceph.mgr.y.smithi092.stdout:Mar 11 21:20:41 smithi092 ceph-00db0016-dfed-11ee-95c8-87774f69a715-mgr-y[29801]: 2024-03-11T21:20:41.195+0000 7f03ef895200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-03-11T21:20:41.717 INFO:journalctl@ceph.mgr.y.smithi092.stdout:Mar 11 21:20:41 smithi092 ceph-00db0016-dfed-11ee-95c8-87774f69a715-mgr-y[29801]: 2024-03-11T21:20:41.478+0000 7f03ef895200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-03-11T21:20:41.717 INFO:journalctl@ceph.mgr.y.smithi092.stdout:Mar 11 21:20:41 smithi092 ceph-00db0016-dfed-11ee-95c8-87774f69a715-mgr-y[29801]: 2024-03-11T21:20:41.732+0000 7f03ef895200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-03-11T21:20:42.112 INFO:journalctl@ceph.mgr.y.smithi092.stdout:Mar 11 21:20:41 smithi092 ceph-00db0016-dfed-11ee-95c8-87774f69a715-mgr-y[29801]: 2024-03-11T21:20:41.969+0000 7f03ef895200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-03-11T21:20:43.112 INFO:journalctl@ceph.mgr.y.smithi092.stdout:Mar 11 21:20:42 smithi092 ceph-00db0016-dfed-11ee-95c8-87774f69a715-mgr-y[29801]: 2024-03-11T21:20:42.768+0000 7f03ef895200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-03-11T21:20:43.612 INFO:journalctl@ceph.mgr.y.smithi092.stdout:Mar 11 21:20:43 smithi092 ceph-00db0016-dfed-11ee-95c8-87774f69a715-mgr-y[29801]: 2024-03-11T21:20:43.346+0000 7f03ef895200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-03-11T21:20:43.612 INFO:journalctl@ceph.mgr.y.smithi092.stdout:Mar 11 21:20:43 smithi092 ceph-00db0016-dfed-11ee-95c8-87774f69a715-mgr-y[29801]: 2024-03-11T21:20:43.446+0000 7f03ef895200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-03-11T21:20:44.112 INFO:journalctl@ceph.mgr.y.smithi092.stdout:Mar 11 21:20:43 smithi092 ceph-00db0016-dfed-11ee-95c8-87774f69a715-mgr-y[29801]: 2024-03-11T21:20:43.662+0000 7f03ef895200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-03-11T21:20:44.112 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:20:43 smithi092 ceph-mon[29572]: Active manager daemon y restarted 2024-03-11T21:20:44.112 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:20:43 smithi092 ceph-mon[29572]: Activating manager daemon y 2024-03-11T21:20:44.112 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:20:43 smithi092 ceph-mon[29572]: osdmap e2: 0 total, 0 up, 0 in 2024-03-11T21:20:44.113 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:20:43 smithi092 ceph-mon[29572]: mgrmap e6: y(active, starting, since 0.00314042s) 2024-03-11T21:20:44.113 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:20:43 smithi092 ceph-mon[29572]: from='mgr.14118 v1:172.21.15.92:0/367308263' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-11T21:20:44.113 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:20:43 smithi092 ceph-mon[29572]: from='mgr.14118 v1:172.21.15.92:0/367308263' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-03-11T21:20:44.113 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:20:43 smithi092 ceph-mon[29572]: from='mgr.14118 v1:172.21.15.92:0/367308263' entity='mgr.y' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-03-11T21:20:44.113 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:20:43 smithi092 ceph-mon[29572]: from='mgr.14118 v1:172.21.15.92:0/367308263' entity='mgr.y' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-03-11T21:20:44.113 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:20:43 smithi092 ceph-mon[29572]: from='mgr.14118 v1:172.21.15.92:0/367308263' entity='mgr.y' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-03-11T21:20:44.113 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:20:43 smithi092 ceph-mon[29572]: Manager daemon y is now available 2024-03-11T21:20:44.113 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:20:43 smithi092 ceph-mon[29572]: from='mgr.14118 v1:172.21.15.92:0/367308263' entity='mgr.y' 2024-03-11T21:20:44.113 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:20:43 smithi092 ceph-mon[29572]: from='mgr.14118 v1:172.21.15.92:0/367308263' entity='mgr.y' 2024-03-11T21:20:44.113 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:20:43 smithi092 ceph-mon[29572]: from='mgr.14118 v1:172.21.15.92:0/367308263' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T21:20:44.113 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:20:43 smithi092 ceph-mon[29572]: from='mgr.14118 v1:172.21.15.92:0/367308263' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T21:20:44.996 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:20:44 smithi092 ceph-mon[29572]: Found migration_current of "None". Setting to last migration. 2024-03-11T21:20:44.996 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:20:44 smithi092 ceph-mon[29572]: from='mgr.14118 v1:172.21.15.92:0/367308263' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T21:20:44.996 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:20:44 smithi092 ceph-mon[29572]: from='mgr.14118 v1:172.21.15.92:0/367308263' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-03-11T21:20:44.996 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:20:44 smithi092 ceph-mon[29572]: from='mgr.14118 v1:172.21.15.92:0/367308263' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-03-11T21:20:44.996 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:20:44 smithi092 ceph-mon[29572]: from='mgr.14118 v1:172.21.15.92:0/367308263' entity='mgr.y' 2024-03-11T21:20:44.996 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:20:44 smithi092 ceph-mon[29572]: from='mgr.14118 v1:172.21.15.92:0/367308263' entity='mgr.y' 2024-03-11T21:20:44.996 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:20:44 smithi092 ceph-mon[29572]: mgrmap e7: y(active, since 1.00551s) 2024-03-11T21:20:45.266 INFO:teuthology.orchestra.run.smithi092.stdout:/usr/bin/ceph: stdout { 2024-03-11T21:20:45.266 INFO:teuthology.orchestra.run.smithi092.stdout:/usr/bin/ceph: stdout "mgrmap_epoch": 7, 2024-03-11T21:20:45.266 INFO:teuthology.orchestra.run.smithi092.stdout:/usr/bin/ceph: stdout "initialized": true 2024-03-11T21:20:45.267 INFO:teuthology.orchestra.run.smithi092.stdout:/usr/bin/ceph: stdout } 2024-03-11T21:20:45.267 INFO:teuthology.orchestra.run.smithi092.stdout:mgr epoch 5 is available 2024-03-11T21:20:45.267 INFO:teuthology.orchestra.run.smithi092.stdout:Setting orchestrator backend to cephadm... 2024-03-11T21:20:45.864 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:20:45 smithi092 ceph-mon[29572]: from='client.14122 v1:172.21.15.92:0/3556693895' entity='client.admin' cmd=[{"prefix": "get_command_descriptions"}]: dispatch 2024-03-11T21:20:45.864 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:20:45 smithi092 ceph-mon[29572]: from='client.14122 v1:172.21.15.92:0/3556693895' entity='client.admin' cmd=[{"prefix": "mgr_status"}]: dispatch 2024-03-11T21:20:45.864 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:20:45 smithi092 ceph-mon[29572]: from='mgr.14118 v1:172.21.15.92:0/367308263' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T21:20:46.862 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:20:46 smithi092 ceph-mon[29572]: [11/Mar/2024:21:20:45] ENGINE Bus STARTING 2024-03-11T21:20:46.862 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:20:46 smithi092 ceph-mon[29572]: [11/Mar/2024:21:20:45] ENGINE Serving on https://172.21.15.92:7150 2024-03-11T21:20:46.862 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:20:46 smithi092 ceph-mon[29572]: [11/Mar/2024:21:20:45] ENGINE Serving on http://172.21.15.92:8765 2024-03-11T21:20:46.863 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:20:46 smithi092 ceph-mon[29572]: [11/Mar/2024:21:20:45] ENGINE Bus STARTED 2024-03-11T21:20:46.863 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:20:46 smithi092 ceph-mon[29572]: mgrmap e8: y(active, since 2s) 2024-03-11T21:20:46.863 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:20:46 smithi092 ceph-mon[29572]: from='client.14130 v1:172.21.15.92:0/2436381021' entity='client.admin' cmd=[{"prefix": "orch set backend", "module_name": "cephadm", "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T21:20:46.863 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:20:46 smithi092 ceph-mon[29572]: from='mgr.14118 v1:172.21.15.92:0/367308263' entity='mgr.y' 2024-03-11T21:20:46.863 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:20:46 smithi092 ceph-mon[29572]: from='mgr.14118 v1:172.21.15.92:0/367308263' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T21:20:47.680 INFO:teuthology.orchestra.run.smithi092.stdout:/usr/bin/ceph: stdout value unchanged 2024-03-11T21:20:47.680 INFO:teuthology.orchestra.run.smithi092.stdout:Generating ssh key... 2024-03-11T21:20:48.362 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:20:47 smithi092 ceph-mon[29572]: from='client.14132 v1:172.21.15.92:0/690928600' entity='client.admin' cmd=[{"prefix": "cephadm set-user", "user": "root", "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T21:20:48.859 INFO:journalctl@ceph.mgr.y.smithi092.stdout:Mar 11 21:20:48 smithi092 ceph-00db0016-dfed-11ee-95c8-87774f69a715-mgr-y[29801]: Generating public/private rsa key pair. 2024-03-11T21:20:48.859 INFO:journalctl@ceph.mgr.y.smithi092.stdout:Mar 11 21:20:48 smithi092 ceph-00db0016-dfed-11ee-95c8-87774f69a715-mgr-y[29801]: Your identification has been saved in /tmp/tmp251aml1n/key. 2024-03-11T21:20:48.859 INFO:journalctl@ceph.mgr.y.smithi092.stdout:Mar 11 21:20:48 smithi092 ceph-00db0016-dfed-11ee-95c8-87774f69a715-mgr-y[29801]: Your public key has been saved in /tmp/tmp251aml1n/key.pub. 2024-03-11T21:20:48.859 INFO:journalctl@ceph.mgr.y.smithi092.stdout:Mar 11 21:20:48 smithi092 ceph-00db0016-dfed-11ee-95c8-87774f69a715-mgr-y[29801]: The key fingerprint is: 2024-03-11T21:20:48.859 INFO:journalctl@ceph.mgr.y.smithi092.stdout:Mar 11 21:20:48 smithi092 ceph-00db0016-dfed-11ee-95c8-87774f69a715-mgr-y[29801]: SHA256:zgO+evNbygTE0HdB2YSlmS/y55sLYUCyyi0qCo8iEn4 ceph-00db0016-dfed-11ee-95c8-87774f69a715 2024-03-11T21:20:48.859 INFO:journalctl@ceph.mgr.y.smithi092.stdout:Mar 11 21:20:48 smithi092 ceph-00db0016-dfed-11ee-95c8-87774f69a715-mgr-y[29801]: The key's randomart image is: 2024-03-11T21:20:48.859 INFO:journalctl@ceph.mgr.y.smithi092.stdout:Mar 11 21:20:48 smithi092 ceph-00db0016-dfed-11ee-95c8-87774f69a715-mgr-y[29801]: +---[RSA 3072]----+ 2024-03-11T21:20:48.859 INFO:journalctl@ceph.mgr.y.smithi092.stdout:Mar 11 21:20:48 smithi092 ceph-00db0016-dfed-11ee-95c8-87774f69a715-mgr-y[29801]: | ... ..oBo | 2024-03-11T21:20:48.859 INFO:journalctl@ceph.mgr.y.smithi092.stdout:Mar 11 21:20:48 smithi092 ceph-00db0016-dfed-11ee-95c8-87774f69a715-mgr-y[29801]: | o.+. ++. | 2024-03-11T21:20:48.859 INFO:journalctl@ceph.mgr.y.smithi092.stdout:Mar 11 21:20:48 smithi092 ceph-00db0016-dfed-11ee-95c8-87774f69a715-mgr-y[29801]: | +...+ | 2024-03-11T21:20:48.859 INFO:journalctl@ceph.mgr.y.smithi092.stdout:Mar 11 21:20:48 smithi092 ceph-00db0016-dfed-11ee-95c8-87774f69a715-mgr-y[29801]: | . + . . | 2024-03-11T21:20:48.859 INFO:journalctl@ceph.mgr.y.smithi092.stdout:Mar 11 21:20:48 smithi092 ceph-00db0016-dfed-11ee-95c8-87774f69a715-mgr-y[29801]: | + + S + . | 2024-03-11T21:20:48.859 INFO:journalctl@ceph.mgr.y.smithi092.stdout:Mar 11 21:20:48 smithi092 ceph-00db0016-dfed-11ee-95c8-87774f69a715-mgr-y[29801]: |. . o = + o | 2024-03-11T21:20:48.860 INFO:journalctl@ceph.mgr.y.smithi092.stdout:Mar 11 21:20:48 smithi092 ceph-00db0016-dfed-11ee-95c8-87774f69a715-mgr-y[29801]: |+.. . = + . | 2024-03-11T21:20:48.860 INFO:journalctl@ceph.mgr.y.smithi092.stdout:Mar 11 21:20:48 smithi092 ceph-00db0016-dfed-11ee-95c8-87774f69a715-mgr-y[29801]: |*= E o+ + + . | 2024-03-11T21:20:48.860 INFO:journalctl@ceph.mgr.y.smithi092.stdout:Mar 11 21:20:48 smithi092 ceph-00db0016-dfed-11ee-95c8-87774f69a715-mgr-y[29801]: |* o .o.o=. =o | 2024-03-11T21:20:48.860 INFO:journalctl@ceph.mgr.y.smithi092.stdout:Mar 11 21:20:48 smithi092 ceph-00db0016-dfed-11ee-95c8-87774f69a715-mgr-y[29801]: +----[SHA256]-----+ 2024-03-11T21:20:49.862 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:20:49 smithi092 ceph-mon[29572]: from='client.14134 v1:172.21.15.92:0/1136173722' entity='client.admin' cmd=[{"prefix": "cephadm generate-key", "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T21:20:49.863 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:20:49 smithi092 ceph-mon[29572]: Generating ssh key... 2024-03-11T21:20:49.863 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:20:49 smithi092 ceph-mon[29572]: from='mgr.14118 v1:172.21.15.92:0/367308263' entity='mgr.y' 2024-03-11T21:20:49.863 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:20:49 smithi092 ceph-mon[29572]: from='mgr.14118 v1:172.21.15.92:0/367308263' entity='mgr.y' 2024-03-11T21:20:50.185 INFO:teuthology.orchestra.run.smithi092.stdout:/usr/bin/ceph: stdout ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDkpHE7ktlULHi5ESUGy1cXP60OksFLGW22Y7uWtcg2azEvEP5iJAToo9YNeb1ZxqmWet5ge7s+1pPZlqbtfVns2YBRy28r/8oPsEjOTsrgEJ4/S3HIH2mbGDxnB8vW76ENKy3VFfZXwyPD9UMO32AXC2+f6j6j//51T0JUh5c6v3sbYquWiIW5jm90ZZH25fQFrkSF+h3w1hwUGGAQn01kIBkuDDQCJwK2MB/ay5vYzNruN+PJzRotxGG0r8R5EmNbZK2dXmAkJUl87+un4HZMIdzfVQYWsr9Sv62ieaVA63aKQAA7YGzNuBP2+43D2H+GOcBwI+/QAvSZZ9Ynvpoplqcc1yoAy5hkEujG7i6Oc6aWcbpqSyZQ8uq4ArveMO+vbjObxP7sTVEbZyOhQtY5X3MkjwKZXgk11JboA4s8Xea3n+2x7+A4YqrRpJxcY6v0SJs2xMap4E4i47Us+uo1OKaK/jwnRXHRa/B91g0+OAxJt9MqvUDmHodtxBhU5ek= ceph-00db0016-dfed-11ee-95c8-87774f69a715 2024-03-11T21:20:50.186 INFO:teuthology.orchestra.run.smithi092.stdout:Wrote public SSH key to /home/ubuntu/cephtest/ceph.pub 2024-03-11T21:20:50.186 INFO:teuthology.orchestra.run.smithi092.stdout:Adding key to root@localhost authorized_keys... 2024-03-11T21:20:50.186 INFO:teuthology.orchestra.run.smithi092.stdout:Adding host smithi092... 2024-03-11T21:20:50.779 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:20:50 smithi092 ceph-mon[29572]: from='client.14136 v1:172.21.15.92:0/2482780126' entity='client.admin' cmd=[{"prefix": "cephadm get-pub-key", "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T21:20:51.727 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:20:51 smithi092 ceph-mon[29572]: from='client.14138 v1:172.21.15.92:0/469248163' entity='client.admin' cmd=[{"prefix": "orch host add", "hostname": "smithi092", "addr": "172.21.15.92", "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T21:20:52.862 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:20:52 smithi092 ceph-mon[29572]: Deploying cephadm binary to smithi092 2024-03-11T21:20:55.450 INFO:teuthology.orchestra.run.smithi092.stdout:/usr/bin/ceph: stdout Added host 'smithi092' with addr '172.21.15.92' 2024-03-11T21:20:55.451 INFO:teuthology.orchestra.run.smithi092.stdout:Deploying unmanaged mon service... 2024-03-11T21:20:56.112 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:20:55 smithi092 ceph-mon[29572]: from='mgr.14118 v1:172.21.15.92:0/367308263' entity='mgr.y' 2024-03-11T21:20:56.112 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:20:55 smithi092 ceph-mon[29572]: Added host smithi092 2024-03-11T21:20:56.112 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:20:55 smithi092 ceph-mon[29572]: from='mgr.14118 v1:172.21.15.92:0/367308263' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T21:20:56.744 INFO:teuthology.orchestra.run.smithi092.stdout:/usr/bin/ceph: stdout Scheduled mon update... 2024-03-11T21:20:56.744 INFO:teuthology.orchestra.run.smithi092.stdout:Deploying unmanaged mgr service... 2024-03-11T21:20:57.362 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:20:57 smithi092 ceph-mon[29572]: from='client.14140 v1:172.21.15.92:0/360242922' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mon", "unmanaged": true, "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T21:20:57.362 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:20:57 smithi092 ceph-mon[29572]: Saving service mon spec with placement count:5 2024-03-11T21:20:57.362 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:20:57 smithi092 ceph-mon[29572]: from='mgr.14118 v1:172.21.15.92:0/367308263' entity='mgr.y' 2024-03-11T21:20:58.608 INFO:teuthology.orchestra.run.smithi092.stdout:/usr/bin/ceph: stdout Scheduled mgr update... 2024-03-11T21:20:58.729 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:20:58 smithi092 ceph-mon[29572]: from='client.14142 v1:172.21.15.92:0/4076013798' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mgr", "unmanaged": true, "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T21:20:58.729 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:20:58 smithi092 ceph-mon[29572]: Saving service mgr spec with placement count:2 2024-03-11T21:20:58.729 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:20:58 smithi092 ceph-mon[29572]: from='mgr.14118 v1:172.21.15.92:0/367308263' entity='mgr.y' 2024-03-11T21:20:58.729 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:20:58 smithi092 ceph-mon[29572]: from='mgr.14118 v1:172.21.15.92:0/367308263' entity='mgr.y' 2024-03-11T21:21:00.012 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:20:59 smithi092 ceph-mon[29572]: from='mgr.14118 v1:172.21.15.92:0/367308263' entity='mgr.y' 2024-03-11T21:21:00.013 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:20:59 smithi092 ceph-mon[29572]: from='client.? v1:172.21.15.92:0/1986043632' entity='client.admin' 2024-03-11T21:21:01.198 INFO:teuthology.orchestra.run.smithi092.stdout:Enabling the dashboard module... 2024-03-11T21:21:01.836 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:21:01 smithi092 ceph-mon[29572]: from='client.? v1:172.21.15.92:0/1478476186' entity='client.admin' 2024-03-11T21:21:02.854 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:21:02 smithi092 ceph-mon[29572]: from='client.? v1:172.21.15.92:0/3511135730' entity='client.admin' cmd=[{"prefix": "mgr module enable", "module": "dashboard"}]: dispatch 2024-03-11T21:21:02.854 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:21:02 smithi092 ceph-mon[29572]: from='mgr.14118 v1:172.21.15.92:0/367308263' entity='mgr.y' 2024-03-11T21:21:02.855 INFO:journalctl@ceph.mgr.y.smithi092.stdout:Mar 11 21:21:02 smithi092 ceph-00db0016-dfed-11ee-95c8-87774f69a715-mgr-y[29801]: ignoring --setuser ceph since I am not root 2024-03-11T21:21:02.855 INFO:journalctl@ceph.mgr.y.smithi092.stdout:Mar 11 21:21:02 smithi092 ceph-00db0016-dfed-11ee-95c8-87774f69a715-mgr-y[29801]: ignoring --setgroup ceph since I am not root 2024-03-11T21:21:03.365 INFO:journalctl@ceph.mgr.y.smithi092.stdout:Mar 11 21:21:03 smithi092 ceph-00db0016-dfed-11ee-95c8-87774f69a715-mgr-y[29801]: 2024-03-11T21:21:03.127+0000 7f7f31679200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-03-11T21:21:03.366 INFO:journalctl@ceph.mgr.y.smithi092.stdout:Mar 11 21:21:03 smithi092 ceph-00db0016-dfed-11ee-95c8-87774f69a715-mgr-y[29801]: 2024-03-11T21:21:03.228+0000 7f7f31679200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-03-11T21:21:03.862 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:21:03 smithi092 ceph-mon[29572]: from='client.? v1:172.21.15.92:0/3511135730' entity='client.admin' cmd='[{"prefix": "mgr module enable", "module": "dashboard"}]': finished 2024-03-11T21:21:03.863 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:21:03 smithi092 ceph-mon[29572]: mgrmap e9: y(active, since 18s) 2024-03-11T21:21:04.165 INFO:journalctl@ceph.mgr.y.smithi092.stdout:Mar 11 21:21:03 smithi092 ceph-00db0016-dfed-11ee-95c8-87774f69a715-mgr-y[29801]: 2024-03-11T21:21:03.944+0000 7f7f31679200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-03-11T21:21:04.165 INFO:journalctl@ceph.mgr.y.smithi092.stdout:Mar 11 21:21:04 smithi092 ceph-00db0016-dfed-11ee-95c8-87774f69a715-mgr-y[29801]: 2024-03-11T21:21:04.037+0000 7f7f31679200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-03-11T21:21:04.165 INFO:journalctl@ceph.mgr.y.smithi092.stdout:Mar 11 21:21:04 smithi092 ceph-00db0016-dfed-11ee-95c8-87774f69a715-mgr-y[29801]: 2024-03-11T21:21:04.180+0000 7f7f31679200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-03-11T21:21:04.444 INFO:journalctl@ceph.mgr.y.smithi092.stdout:Mar 11 21:21:04 smithi092 ceph-00db0016-dfed-11ee-95c8-87774f69a715-mgr-y[29801]: 2024-03-11T21:21:04.276+0000 7f7f31679200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-03-11T21:21:04.444 INFO:journalctl@ceph.mgr.y.smithi092.stdout:Mar 11 21:21:04 smithi092 ceph-00db0016-dfed-11ee-95c8-87774f69a715-mgr-y[29801]: 2024-03-11T21:21:04.359+0000 7f7f31679200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-03-11T21:21:04.446 INFO:teuthology.orchestra.run.smithi092.stdout:/usr/bin/ceph: stdout { 2024-03-11T21:21:04.447 INFO:teuthology.orchestra.run.smithi092.stdout:/usr/bin/ceph: stdout "epoch": 9, 2024-03-11T21:21:04.447 INFO:teuthology.orchestra.run.smithi092.stdout:/usr/bin/ceph: stdout "available": true, 2024-03-11T21:21:04.447 INFO:teuthology.orchestra.run.smithi092.stdout:/usr/bin/ceph: stdout "active_name": "y", 2024-03-11T21:21:04.447 INFO:teuthology.orchestra.run.smithi092.stdout:/usr/bin/ceph: stdout "num_standby": 0 2024-03-11T21:21:04.447 INFO:teuthology.orchestra.run.smithi092.stdout:/usr/bin/ceph: stdout } 2024-03-11T21:21:04.447 INFO:teuthology.orchestra.run.smithi092.stdout:Waiting for the mgr to restart... 2024-03-11T21:21:04.447 INFO:teuthology.orchestra.run.smithi092.stdout:Waiting for mgr epoch 9... 2024-03-11T21:21:04.729 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:21:04 smithi092 ceph-mon[29572]: from='client.? v1:172.21.15.92:0/4098796170' entity='client.admin' cmd=[{"prefix": "mgr stat"}]: dispatch 2024-03-11T21:21:04.729 INFO:journalctl@ceph.mgr.y.smithi092.stdout:Mar 11 21:21:04 smithi092 ceph-00db0016-dfed-11ee-95c8-87774f69a715-mgr-y[29801]: 2024-03-11T21:21:04.629+0000 7f7f31679200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-03-11T21:21:04.729 INFO:journalctl@ceph.mgr.y.smithi092.stdout:Mar 11 21:21:04 smithi092 ceph-00db0016-dfed-11ee-95c8-87774f69a715-mgr-y[29801]: 2024-03-11T21:21:04.744+0000 7f7f31679200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-03-11T21:21:05.091 INFO:journalctl@ceph.mgr.y.smithi092.stdout:Mar 11 21:21:04 smithi092 ceph-00db0016-dfed-11ee-95c8-87774f69a715-mgr-y[29801]: 2024-03-11T21:21:04.835+0000 7f7f31679200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-03-11T21:21:05.091 INFO:journalctl@ceph.mgr.y.smithi092.stdout:Mar 11 21:21:04 smithi092 ceph-00db0016-dfed-11ee-95c8-87774f69a715-mgr-y[29801]: 2024-03-11T21:21:04.931+0000 7f7f31679200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-03-11T21:21:05.362 INFO:journalctl@ceph.mgr.y.smithi092.stdout:Mar 11 21:21:05 smithi092 ceph-00db0016-dfed-11ee-95c8-87774f69a715-mgr-y[29801]: 2024-03-11T21:21:05.106+0000 7f7f31679200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-03-11T21:21:06.362 INFO:journalctl@ceph.mgr.y.smithi092.stdout:Mar 11 21:21:06 smithi092 ceph-00db0016-dfed-11ee-95c8-87774f69a715-mgr-y[29801]: 2024-03-11T21:21:06.061+0000 7f7f31679200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-03-11T21:21:06.362 INFO:journalctl@ceph.mgr.y.smithi092.stdout:Mar 11 21:21:06 smithi092 ceph-00db0016-dfed-11ee-95c8-87774f69a715-mgr-y[29801]: 2024-03-11T21:21:06.191+0000 7f7f31679200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-03-11T21:21:06.793 INFO:journalctl@ceph.mgr.y.smithi092.stdout:Mar 11 21:21:06 smithi092 ceph-00db0016-dfed-11ee-95c8-87774f69a715-mgr-y[29801]: 2024-03-11T21:21:06.526+0000 7f7f31679200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-03-11T21:21:06.794 INFO:journalctl@ceph.mgr.y.smithi092.stdout:Mar 11 21:21:06 smithi092 ceph-00db0016-dfed-11ee-95c8-87774f69a715-mgr-y[29801]: 2024-03-11T21:21:06.615+0000 7f7f31679200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-03-11T21:21:06.794 INFO:journalctl@ceph.mgr.y.smithi092.stdout:Mar 11 21:21:06 smithi092 ceph-00db0016-dfed-11ee-95c8-87774f69a715-mgr-y[29801]: 2024-03-11T21:21:06.808+0000 7f7f31679200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-03-11T21:21:07.112 INFO:journalctl@ceph.mgr.y.smithi092.stdout:Mar 11 21:21:06 smithi092 ceph-00db0016-dfed-11ee-95c8-87774f69a715-mgr-y[29801]: 2024-03-11T21:21:06.995+0000 7f7f31679200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-03-11T21:21:07.527 INFO:journalctl@ceph.mgr.y.smithi092.stdout:Mar 11 21:21:07 smithi092 ceph-00db0016-dfed-11ee-95c8-87774f69a715-mgr-y[29801]: 2024-03-11T21:21:07.281+0000 7f7f31679200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-03-11T21:21:07.528 INFO:journalctl@ceph.mgr.y.smithi092.stdout:Mar 11 21:21:07 smithi092 ceph-00db0016-dfed-11ee-95c8-87774f69a715-mgr-y[29801]: 2024-03-11T21:21:07.542+0000 7f7f31679200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-03-11T21:21:07.862 INFO:journalctl@ceph.mgr.y.smithi092.stdout:Mar 11 21:21:07 smithi092 ceph-00db0016-dfed-11ee-95c8-87774f69a715-mgr-y[29801]: 2024-03-11T21:21:07.777+0000 7f7f31679200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-03-11T21:21:08.862 INFO:journalctl@ceph.mgr.y.smithi092.stdout:Mar 11 21:21:08 smithi092 ceph-00db0016-dfed-11ee-95c8-87774f69a715-mgr-y[29801]: 2024-03-11T21:21:08.579+0000 7f7f31679200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-03-11T21:21:09.444 INFO:journalctl@ceph.mgr.y.smithi092.stdout:Mar 11 21:21:09 smithi092 ceph-00db0016-dfed-11ee-95c8-87774f69a715-mgr-y[29801]: 2024-03-11T21:21:09.143+0000 7f7f31679200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-03-11T21:21:09.444 INFO:journalctl@ceph.mgr.y.smithi092.stdout:Mar 11 21:21:09 smithi092 ceph-00db0016-dfed-11ee-95c8-87774f69a715-mgr-y[29801]: 2024-03-11T21:21:09.239+0000 7f7f31679200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-03-11T21:21:09.444 INFO:journalctl@ceph.mgr.y.smithi092.stdout:Mar 11 21:21:09 smithi092 ceph-00db0016-dfed-11ee-95c8-87774f69a715-mgr-y[29801]: 2024-03-11T21:21:09.459+0000 7f7f31679200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-03-11T21:21:09.862 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:21:09 smithi092 ceph-mon[29572]: Active manager daemon y restarted 2024-03-11T21:21:09.862 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:21:09 smithi092 ceph-mon[29572]: Activating manager daemon y 2024-03-11T21:21:09.863 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:21:09 smithi092 ceph-mon[29572]: osdmap e3: 0 total, 0 up, 0 in 2024-03-11T21:21:09.863 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:21:09 smithi092 ceph-mon[29572]: mgrmap e10: y(active, starting, since 0.00319268s) 2024-03-11T21:21:09.863 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:21:09 smithi092 ceph-mon[29572]: from='mgr.14150 v1:172.21.15.92:0/996057664' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-11T21:21:09.863 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:21:09 smithi092 ceph-mon[29572]: from='mgr.14150 v1:172.21.15.92:0/996057664' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-03-11T21:21:09.863 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:21:09 smithi092 ceph-mon[29572]: from='mgr.14150 v1:172.21.15.92:0/996057664' entity='mgr.y' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-03-11T21:21:09.863 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:21:09 smithi092 ceph-mon[29572]: from='mgr.14150 v1:172.21.15.92:0/996057664' entity='mgr.y' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-03-11T21:21:09.863 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:21:09 smithi092 ceph-mon[29572]: from='mgr.14150 v1:172.21.15.92:0/996057664' entity='mgr.y' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-03-11T21:21:09.863 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:21:09 smithi092 ceph-mon[29572]: Manager daemon y is now available 2024-03-11T21:21:09.863 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:21:09 smithi092 ceph-mon[29572]: from='mgr.14150 v1:172.21.15.92:0/996057664' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T21:21:09.863 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:21:09 smithi092 ceph-mon[29572]: from='mgr.14150 v1:172.21.15.92:0/996057664' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T21:21:10.692 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:21:10 smithi092 ceph-mon[29572]: from='mgr.14150 v1:172.21.15.92:0/996057664' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-03-11T21:21:10.692 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:21:10 smithi092 ceph-mon[29572]: from='mgr.14150 v1:172.21.15.92:0/996057664' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-03-11T21:21:10.692 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:21:10 smithi092 ceph-mon[29572]: mgrmap e11: y(active, since 1.0058s) 2024-03-11T21:21:11.391 INFO:teuthology.orchestra.run.smithi092.stdout:/usr/bin/ceph: stdout { 2024-03-11T21:21:11.391 INFO:teuthology.orchestra.run.smithi092.stdout:/usr/bin/ceph: stdout "mgrmap_epoch": 11, 2024-03-11T21:21:11.391 INFO:teuthology.orchestra.run.smithi092.stdout:/usr/bin/ceph: stdout "initialized": true 2024-03-11T21:21:11.391 INFO:teuthology.orchestra.run.smithi092.stdout:/usr/bin/ceph: stdout } 2024-03-11T21:21:11.391 INFO:teuthology.orchestra.run.smithi092.stdout:mgr epoch 9 is available 2024-03-11T21:21:11.391 INFO:teuthology.orchestra.run.smithi092.stdout:Generating a dashboard self-signed certificate... 2024-03-11T21:21:11.814 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:21:11 smithi092 ceph-mon[29572]: [11/Mar/2024:21:21:09] ENGINE Bus STARTING 2024-03-11T21:21:11.815 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:21:11 smithi092 ceph-mon[29572]: [11/Mar/2024:21:21:09] ENGINE Serving on http://172.21.15.92:8765 2024-03-11T21:21:11.815 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:21:11 smithi092 ceph-mon[29572]: [11/Mar/2024:21:21:10] ENGINE Serving on https://172.21.15.92:7150 2024-03-11T21:21:11.815 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:21:11 smithi092 ceph-mon[29572]: [11/Mar/2024:21:21:10] ENGINE Bus STARTED 2024-03-11T21:21:11.815 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:21:11 smithi092 ceph-mon[29572]: from='client.14154 v1:172.21.15.92:0/3468935522' entity='client.admin' cmd=[{"prefix": "get_command_descriptions"}]: dispatch 2024-03-11T21:21:11.815 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:21:11 smithi092 ceph-mon[29572]: from='client.14154 v1:172.21.15.92:0/3468935522' entity='client.admin' cmd=[{"prefix": "mgr_status"}]: dispatch 2024-03-11T21:21:11.816 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:21:11 smithi092 ceph-mon[29572]: from='mgr.14150 v1:172.21.15.92:0/996057664' entity='mgr.y' 2024-03-11T21:21:12.904 INFO:teuthology.orchestra.run.smithi092.stdout:/usr/bin/ceph: stdout Self-signed certificate created 2024-03-11T21:21:12.905 INFO:teuthology.orchestra.run.smithi092.stdout:Creating initial admin user... 2024-03-11T21:21:13.296 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:21:13 smithi092 ceph-mon[29572]: mgrmap e12: y(active, since 2s) 2024-03-11T21:21:13.297 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:21:13 smithi092 ceph-mon[29572]: from='client.14162 v1:172.21.15.92:0/2421277909' entity='client.admin' cmd=[{"prefix": "dashboard create-self-signed-cert", "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T21:21:13.297 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:21:13 smithi092 ceph-mon[29572]: from='mgr.14150 v1:172.21.15.92:0/996057664' entity='mgr.y' 2024-03-11T21:21:13.297 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:21:13 smithi092 ceph-mon[29572]: from='mgr.14150 v1:172.21.15.92:0/996057664' entity='mgr.y' 2024-03-11T21:21:13.297 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:21:13 smithi092 ceph-mon[29572]: from='mgr.14150 v1:172.21.15.92:0/996057664' entity='mgr.y' 2024-03-11T21:21:14.362 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:21:14 smithi092 ceph-mon[29572]: from='client.14164 v1:172.21.15.92:0/2674807151' entity='client.admin' cmd=[{"prefix": "dashboard ac-user-create", "username": "admin", "rolename": "administrator", "force_password": true, "pwd_update_required": true, "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T21:21:14.362 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:21:14 smithi092 ceph-mon[29572]: from='mgr.14150 v1:172.21.15.92:0/996057664' entity='mgr.y' 2024-03-11T21:21:14.465 INFO:teuthology.orchestra.run.smithi092.stdout:/usr/bin/ceph: stdout {"username": "admin", "password": "$2b$12$Z/OJegu3w6ng3aqQdqcRJ.cvEI1iP2G3OacWiKDW78417UxwizdV.", "roles": ["administrator"], "name": null, "email": null, "lastUpdate": 1710192073, "enabled": true, "pwdExpirationDate": null, "pwdUpdateRequired": true} 2024-03-11T21:21:14.465 INFO:teuthology.orchestra.run.smithi092.stdout:Fetching dashboard port number... 2024-03-11T21:21:15.862 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:21:15 smithi092 ceph-mon[29572]: from='client.? v1:172.21.15.92:0/1407059718' entity='client.admin' cmd=[{"prefix": "config get", "who": "mgr", "key": "mgr/dashboard/ssl_server_port"}]: dispatch 2024-03-11T21:21:15.863 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:21:15 smithi092 ceph-mon[29572]: mgrmap e13: y(active, since 6s) 2024-03-11T21:21:15.990 INFO:teuthology.orchestra.run.smithi092.stdout:/usr/bin/ceph: stdout 8443 2024-03-11T21:21:16.000 INFO:teuthology.orchestra.run.smithi092.stdout:Non-zero exit code 1 from systemctl is-enabled firewalld.service 2024-03-11T21:21:16.000 INFO:teuthology.orchestra.run.smithi092.stdout:systemctl: stdout disabled 2024-03-11T21:21:16.007 INFO:teuthology.orchestra.run.smithi092.stdout:Non-zero exit code 3 from systemctl is-active firewalld.service 2024-03-11T21:21:16.007 INFO:teuthology.orchestra.run.smithi092.stdout:systemctl: stdout inactive 2024-03-11T21:21:16.007 INFO:teuthology.orchestra.run.smithi092.stdout:firewalld.service is not enabled 2024-03-11T21:21:16.007 INFO:teuthology.orchestra.run.smithi092.stdout:Not possible to open ports <[8443]>. firewalld.service is not available 2024-03-11T21:21:16.008 INFO:teuthology.orchestra.run.smithi092.stdout:Ceph Dashboard is now available at: 2024-03-11T21:21:16.008 INFO:teuthology.orchestra.run.smithi092.stdout: 2024-03-11T21:21:16.008 INFO:teuthology.orchestra.run.smithi092.stdout: URL: https://smithi092.front.sepia.ceph.com:8443/ 2024-03-11T21:21:16.009 INFO:teuthology.orchestra.run.smithi092.stdout: User: admin 2024-03-11T21:21:16.009 INFO:teuthology.orchestra.run.smithi092.stdout: Password: s7r8vobosh 2024-03-11T21:21:16.009 INFO:teuthology.orchestra.run.smithi092.stdout: 2024-03-11T21:21:16.009 INFO:teuthology.orchestra.run.smithi092.stdout:Saving cluster configuration to /var/lib/ceph/00db0016-dfed-11ee-95c8-87774f69a715/config directory 2024-03-11T21:21:16.009 INFO:teuthology.orchestra.run.smithi092.stdout:Enabling autotune for osd_memory_target 2024-03-11T21:21:18.389 INFO:teuthology.orchestra.run.smithi092.stdout:/usr/bin/ceph: stderr set mgr/dashboard/cluster/status 2024-03-11T21:21:18.389 INFO:teuthology.orchestra.run.smithi092.stdout:You can access the Ceph CLI as following in case of multi-cluster or non-default config: 2024-03-11T21:21:18.389 INFO:teuthology.orchestra.run.smithi092.stdout: 2024-03-11T21:21:18.389 INFO:teuthology.orchestra.run.smithi092.stdout: sudo /home/ubuntu/cephtest/cephadm shell --fsid 00db0016-dfed-11ee-95c8-87774f69a715 -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring 2024-03-11T21:21:18.389 INFO:teuthology.orchestra.run.smithi092.stdout: 2024-03-11T21:21:18.390 INFO:teuthology.orchestra.run.smithi092.stdout:Or, if you are only running a single cluster on this host: 2024-03-11T21:21:18.390 INFO:teuthology.orchestra.run.smithi092.stdout: 2024-03-11T21:21:18.390 INFO:teuthology.orchestra.run.smithi092.stdout: sudo /home/ubuntu/cephtest/cephadm shell 2024-03-11T21:21:18.390 INFO:teuthology.orchestra.run.smithi092.stdout: 2024-03-11T21:21:18.390 INFO:teuthology.orchestra.run.smithi092.stdout:Please consider enabling telemetry to help improve Ceph: 2024-03-11T21:21:18.390 INFO:teuthology.orchestra.run.smithi092.stdout: 2024-03-11T21:21:18.390 INFO:teuthology.orchestra.run.smithi092.stdout: ceph telemetry on 2024-03-11T21:21:18.390 INFO:teuthology.orchestra.run.smithi092.stdout: 2024-03-11T21:21:18.390 INFO:teuthology.orchestra.run.smithi092.stdout:For more information see: 2024-03-11T21:21:18.391 INFO:teuthology.orchestra.run.smithi092.stdout: 2024-03-11T21:21:18.391 INFO:teuthology.orchestra.run.smithi092.stdout: https://docs.ceph.com/en/latest/mgr/telemetry/ 2024-03-11T21:21:18.391 INFO:teuthology.orchestra.run.smithi092.stdout: 2024-03-11T21:21:18.391 INFO:teuthology.orchestra.run.smithi092.stdout:Bootstrap complete. 2024-03-11T21:21:18.430 INFO:tasks.cephadm:Fetching config... 2024-03-11T21:21:18.431 DEBUG:teuthology.orchestra.run.smithi092:> set -ex 2024-03-11T21:21:18.431 DEBUG:teuthology.orchestra.run.smithi092:> dd if=/etc/ceph/ceph.conf of=/dev/stdout 2024-03-11T21:21:18.446 INFO:tasks.cephadm:Fetching client.admin keyring... 2024-03-11T21:21:18.446 DEBUG:teuthology.orchestra.run.smithi092:> set -ex 2024-03-11T21:21:18.446 DEBUG:teuthology.orchestra.run.smithi092:> dd if=/etc/ceph/ceph.client.admin.keyring of=/dev/stdout 2024-03-11T21:21:18.501 INFO:tasks.cephadm:Fetching mon keyring... 2024-03-11T21:21:18.501 DEBUG:teuthology.orchestra.run.smithi092:> set -ex 2024-03-11T21:21:18.501 DEBUG:teuthology.orchestra.run.smithi092:> sudo dd if=/var/lib/ceph/00db0016-dfed-11ee-95c8-87774f69a715/mon.a/keyring of=/dev/stdout 2024-03-11T21:21:18.568 INFO:tasks.cephadm:Fetching pub ssh key... 2024-03-11T21:21:18.569 DEBUG:teuthology.orchestra.run.smithi092:> set -ex 2024-03-11T21:21:18.569 DEBUG:teuthology.orchestra.run.smithi092:> dd if=/home/ubuntu/cephtest/ceph.pub of=/dev/stdout 2024-03-11T21:21:18.627 INFO:tasks.cephadm:Installing pub ssh key for root users... 2024-03-11T21:21:18.627 DEBUG:teuthology.orchestra.run.smithi092:> sudo install -d -m 0700 /root/.ssh && echo 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDkpHE7ktlULHi5ESUGy1cXP60OksFLGW22Y7uWtcg2azEvEP5iJAToo9YNeb1ZxqmWet5ge7s+1pPZlqbtfVns2YBRy28r/8oPsEjOTsrgEJ4/S3HIH2mbGDxnB8vW76ENKy3VFfZXwyPD9UMO32AXC2+f6j6j//51T0JUh5c6v3sbYquWiIW5jm90ZZH25fQFrkSF+h3w1hwUGGAQn01kIBkuDDQCJwK2MB/ay5vYzNruN+PJzRotxGG0r8R5EmNbZK2dXmAkJUl87+un4HZMIdzfVQYWsr9Sv62ieaVA63aKQAA7YGzNuBP2+43D2H+GOcBwI+/QAvSZZ9Ynvpoplqcc1yoAy5hkEujG7i6Oc6aWcbpqSyZQ8uq4ArveMO+vbjObxP7sTVEbZyOhQtY5X3MkjwKZXgk11JboA4s8Xea3n+2x7+A4YqrRpJxcY6v0SJs2xMap4E4i47Us+uo1OKaK/jwnRXHRa/B91g0+OAxJt9MqvUDmHodtxBhU5ek= ceph-00db0016-dfed-11ee-95c8-87774f69a715' | sudo tee -a /root/.ssh/authorized_keys && sudo chmod 0600 /root/.ssh/authorized_keys 2024-03-11T21:21:18.705 INFO:teuthology.orchestra.run.smithi092.stdout:ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDkpHE7ktlULHi5ESUGy1cXP60OksFLGW22Y7uWtcg2azEvEP5iJAToo9YNeb1ZxqmWet5ge7s+1pPZlqbtfVns2YBRy28r/8oPsEjOTsrgEJ4/S3HIH2mbGDxnB8vW76ENKy3VFfZXwyPD9UMO32AXC2+f6j6j//51T0JUh5c6v3sbYquWiIW5jm90ZZH25fQFrkSF+h3w1hwUGGAQn01kIBkuDDQCJwK2MB/ay5vYzNruN+PJzRotxGG0r8R5EmNbZK2dXmAkJUl87+un4HZMIdzfVQYWsr9Sv62ieaVA63aKQAA7YGzNuBP2+43D2H+GOcBwI+/QAvSZZ9Ynvpoplqcc1yoAy5hkEujG7i6Oc6aWcbpqSyZQ8uq4ArveMO+vbjObxP7sTVEbZyOhQtY5X3MkjwKZXgk11JboA4s8Xea3n+2x7+A4YqrRpJxcY6v0SJs2xMap4E4i47Us+uo1OKaK/jwnRXHRa/B91g0+OAxJt9MqvUDmHodtxBhU5ek= ceph-00db0016-dfed-11ee-95c8-87774f69a715 2024-03-11T21:21:18.719 DEBUG:teuthology.orchestra.run.smithi097:> sudo install -d -m 0700 /root/.ssh && echo 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDkpHE7ktlULHi5ESUGy1cXP60OksFLGW22Y7uWtcg2azEvEP5iJAToo9YNeb1ZxqmWet5ge7s+1pPZlqbtfVns2YBRy28r/8oPsEjOTsrgEJ4/S3HIH2mbGDxnB8vW76ENKy3VFfZXwyPD9UMO32AXC2+f6j6j//51T0JUh5c6v3sbYquWiIW5jm90ZZH25fQFrkSF+h3w1hwUGGAQn01kIBkuDDQCJwK2MB/ay5vYzNruN+PJzRotxGG0r8R5EmNbZK2dXmAkJUl87+un4HZMIdzfVQYWsr9Sv62ieaVA63aKQAA7YGzNuBP2+43D2H+GOcBwI+/QAvSZZ9Ynvpoplqcc1yoAy5hkEujG7i6Oc6aWcbpqSyZQ8uq4ArveMO+vbjObxP7sTVEbZyOhQtY5X3MkjwKZXgk11JboA4s8Xea3n+2x7+A4YqrRpJxcY6v0SJs2xMap4E4i47Us+uo1OKaK/jwnRXHRa/B91g0+OAxJt9MqvUDmHodtxBhU5ek= ceph-00db0016-dfed-11ee-95c8-87774f69a715' | sudo tee -a /root/.ssh/authorized_keys && sudo chmod 0600 /root/.ssh/authorized_keys 2024-03-11T21:21:18.763 INFO:teuthology.orchestra.run.smithi097.stdout:ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDkpHE7ktlULHi5ESUGy1cXP60OksFLGW22Y7uWtcg2azEvEP5iJAToo9YNeb1ZxqmWet5ge7s+1pPZlqbtfVns2YBRy28r/8oPsEjOTsrgEJ4/S3HIH2mbGDxnB8vW76ENKy3VFfZXwyPD9UMO32AXC2+f6j6j//51T0JUh5c6v3sbYquWiIW5jm90ZZH25fQFrkSF+h3w1hwUGGAQn01kIBkuDDQCJwK2MB/ay5vYzNruN+PJzRotxGG0r8R5EmNbZK2dXmAkJUl87+un4HZMIdzfVQYWsr9Sv62ieaVA63aKQAA7YGzNuBP2+43D2H+GOcBwI+/QAvSZZ9Ynvpoplqcc1yoAy5hkEujG7i6Oc6aWcbpqSyZQ8uq4ArveMO+vbjObxP7sTVEbZyOhQtY5X3MkjwKZXgk11JboA4s8Xea3n+2x7+A4YqrRpJxcY6v0SJs2xMap4E4i47Us+uo1OKaK/jwnRXHRa/B91g0+OAxJt9MqvUDmHodtxBhU5ek= ceph-00db0016-dfed-11ee-95c8-87774f69a715 2024-03-11T21:21:18.779 DEBUG:teuthology.orchestra.run.smithi092:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 00db0016-dfed-11ee-95c8-87774f69a715 -- ceph config set mgr mgr/cephadm/allow_ptrace true 2024-03-11T21:21:18.956 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:21:18 smithi092 ceph-mon[29572]: from='client.? v1:172.21.15.92:0/46115348' entity='client.admin' 2024-03-11T21:21:19.020 INFO:teuthology.orchestra.run.smithi092.stderr:Inferring config /var/lib/ceph/00db0016-dfed-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T21:21:21.851 INFO:tasks.cephadm:Distributing conf and client.admin keyring to all hosts + 0755 2024-03-11T21:21:21.851 DEBUG:teuthology.orchestra.run.smithi092:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 00db0016-dfed-11ee-95c8-87774f69a715 -- ceph orch client-keyring set client.admin '*' --mode 0755 2024-03-11T21:21:22.085 INFO:teuthology.orchestra.run.smithi092.stderr:Inferring config /var/lib/ceph/00db0016-dfed-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T21:21:22.112 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:21:21 smithi092 ceph-mon[29572]: from='mgr.14150 v1:172.21.15.92:0/996057664' entity='mgr.y' 2024-03-11T21:21:22.112 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:21:21 smithi092 ceph-mon[29572]: from='mgr.14150 v1:172.21.15.92:0/996057664' entity='mgr.y' 2024-03-11T21:21:22.112 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:21:21 smithi092 ceph-mon[29572]: from='mgr.14150 v1:172.21.15.92:0/996057664' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd/host:smithi092", "name": "osd_memory_target"}]: dispatch 2024-03-11T21:21:22.113 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:21:21 smithi092 ceph-mon[29572]: from='mgr.14150 v1:172.21.15.92:0/996057664' entity='mgr.y' 2024-03-11T21:21:22.113 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:21:21 smithi092 ceph-mon[29572]: from='mgr.14150 v1:172.21.15.92:0/996057664' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T21:21:22.113 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:21:21 smithi092 ceph-mon[29572]: from='mgr.14150 v1:172.21.15.92:0/996057664' entity='mgr.y' 2024-03-11T21:21:22.113 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:21:21 smithi092 ceph-mon[29572]: from='client.? v1:172.21.15.92:0/529721870' entity='client.admin' 2024-03-11T21:21:22.113 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:21:21 smithi092 ceph-mon[29572]: from='mgr.14150 v1:172.21.15.92:0/996057664' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T21:21:22.113 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:21:21 smithi092 ceph-mon[29572]: from='mgr.14150 v1:172.21.15.92:0/996057664' entity='mgr.y' 2024-03-11T21:21:24.291 INFO:tasks.cephadm:Writing (initial) conf and keyring to smithi097 2024-03-11T21:21:24.292 DEBUG:teuthology.orchestra.run.smithi097:> set -ex 2024-03-11T21:21:24.292 DEBUG:teuthology.orchestra.run.smithi097:> dd of=/etc/ceph/ceph.conf 2024-03-11T21:21:24.312 DEBUG:teuthology.orchestra.run.smithi097:> set -ex 2024-03-11T21:21:24.312 DEBUG:teuthology.orchestra.run.smithi097:> dd of=/etc/ceph/ceph.client.admin.keyring 2024-03-11T21:21:24.370 INFO:tasks.cephadm:Adding host smithi097 to orchestrator... 2024-03-11T21:21:24.370 DEBUG:teuthology.orchestra.run.smithi092:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 00db0016-dfed-11ee-95c8-87774f69a715 -- ceph orch host add smithi097 2024-03-11T21:21:24.614 INFO:teuthology.orchestra.run.smithi092.stderr:Inferring config /var/lib/ceph/00db0016-dfed-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T21:21:24.824 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:21:24 smithi092 ceph-mon[29572]: from='client.14174 v1:172.21.15.92:0/3061173963' entity='client.admin' cmd=[{"prefix": "orch client-keyring set", "entity": "client.admin", "placement": "*", "mode": "0755", "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T21:21:24.824 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:21:24 smithi092 ceph-mon[29572]: from='mgr.14150 v1:172.21.15.92:0/996057664' entity='mgr.y' 2024-03-11T21:21:24.825 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:21:24 smithi092 ceph-mon[29572]: from='mgr.14150 v1:172.21.15.92:0/996057664' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T21:21:24.825 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:21:24 smithi092 ceph-mon[29572]: from='mgr.14150 v1:172.21.15.92:0/996057664' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T21:21:24.825 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:21:24 smithi092 ceph-mon[29572]: from='mgr.14150 v1:172.21.15.92:0/996057664' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T21:21:24.825 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:21:24 smithi092 ceph-mon[29572]: Updating smithi092:/etc/ceph/ceph.conf 2024-03-11T21:21:24.825 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:21:24 smithi092 ceph-mon[29572]: from='mgr.14150 v1:172.21.15.92:0/996057664' entity='mgr.y' 2024-03-11T21:21:24.825 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:21:24 smithi092 ceph-mon[29572]: from='mgr.14150 v1:172.21.15.92:0/996057664' entity='mgr.y' 2024-03-11T21:21:24.825 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:21:24 smithi092 ceph-mon[29572]: from='mgr.14150 v1:172.21.15.92:0/996057664' entity='mgr.y' 2024-03-11T21:21:25.822 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:21:25 smithi092 ceph-mon[29572]: Updating smithi092:/var/lib/ceph/00db0016-dfed-11ee-95c8-87774f69a715/config/ceph.conf 2024-03-11T21:21:25.822 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:21:25 smithi092 ceph-mon[29572]: Updating smithi092:/etc/ceph/ceph.client.admin.keyring 2024-03-11T21:21:25.822 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:21:25 smithi092 ceph-mon[29572]: Updating smithi092:/var/lib/ceph/00db0016-dfed-11ee-95c8-87774f69a715/config/ceph.client.admin.keyring 2024-03-11T21:21:27.112 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:21:26 smithi092 ceph-mon[29572]: from='client.14176 v1:172.21.15.92:0/3689780264' entity='client.admin' cmd=[{"prefix": "orch host add", "hostname": "smithi097", "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T21:21:28.112 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:21:27 smithi092 ceph-mon[29572]: Deploying cephadm binary to smithi097 2024-03-11T21:21:30.840 INFO:teuthology.orchestra.run.smithi092.stdout:Added host 'smithi097' with addr '172.21.15.97' 2024-03-11T21:21:30.840 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:21:30 smithi092 ceph-mon[29572]: pgmap v3: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T21:21:31.434 DEBUG:teuthology.orchestra.run.smithi092:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 00db0016-dfed-11ee-95c8-87774f69a715 -- ceph orch host ls --format=json 2024-03-11T21:21:31.671 INFO:teuthology.orchestra.run.smithi092.stderr:Inferring config /var/lib/ceph/00db0016-dfed-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T21:21:32.053 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:21:31 smithi092 ceph-mon[29572]: from='mgr.14150 v1:172.21.15.92:0/996057664' entity='mgr.y' 2024-03-11T21:21:32.053 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:21:31 smithi092 ceph-mon[29572]: Added host smithi097 2024-03-11T21:21:32.053 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:21:31 smithi092 ceph-mon[29572]: from='mgr.14150 v1:172.21.15.92:0/996057664' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T21:21:32.054 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:21:31 smithi092 ceph-mon[29572]: from='mgr.14150 v1:172.21.15.92:0/996057664' entity='mgr.y' 2024-03-11T21:21:32.054 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:21:31 smithi092 ceph-mon[29572]: pgmap v4: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T21:21:32.054 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:21:31 smithi092 ceph-mon[29572]: from='mgr.14150 v1:172.21.15.92:0/996057664' entity='mgr.y' 2024-03-11T21:21:33.317 INFO:teuthology.orchestra.run.smithi092.stdout: 2024-03-11T21:21:33.317 INFO:teuthology.orchestra.run.smithi092.stdout:[{"addr": "172.21.15.92", "hostname": "smithi092", "labels": [], "status": ""}, {"addr": "172.21.15.97", "hostname": "smithi097", "labels": [], "status": ""}] 2024-03-11T21:21:33.863 INFO:tasks.cephadm:Setting crush tunables to default 2024-03-11T21:21:33.863 DEBUG:teuthology.orchestra.run.smithi092:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 00db0016-dfed-11ee-95c8-87774f69a715 -- ceph osd crush tunables default 2024-03-11T21:21:34.095 INFO:teuthology.orchestra.run.smithi092.stderr:Inferring config /var/lib/ceph/00db0016-dfed-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T21:21:34.112 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:21:33 smithi092 ceph-mon[29572]: from='mgr.14150 v1:172.21.15.92:0/996057664' entity='mgr.y' 2024-03-11T21:21:34.112 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:21:33 smithi092 ceph-mon[29572]: from='client.14178 v1:172.21.15.92:0/2405818028' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-11T21:21:34.112 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:21:33 smithi092 ceph-mon[29572]: pgmap v5: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T21:21:36.524 INFO:teuthology.orchestra.run.smithi092.stderr:adjusted tunables profile to default 2024-03-11T21:21:36.841 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:21:36 smithi092 ceph-mon[29572]: pgmap v6: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T21:21:36.841 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:21:36 smithi092 ceph-mon[29572]: from='client.? v1:172.21.15.92:0/2169605665' entity='client.admin' cmd=[{"prefix": "osd crush tunables", "profile": "default"}]: dispatch 2024-03-11T21:21:36.841 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:21:36 smithi092 ceph-mon[29572]: from='mgr.14150 v1:172.21.15.92:0/996057664' entity='mgr.y' 2024-03-11T21:21:36.841 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:21:36 smithi092 ceph-mon[29572]: from='mgr.14150 v1:172.21.15.92:0/996057664' entity='mgr.y' 2024-03-11T21:21:36.842 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:21:36 smithi092 ceph-mon[29572]: from='mgr.14150 v1:172.21.15.92:0/996057664' entity='mgr.y' 2024-03-11T21:21:36.842 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:21:36 smithi092 ceph-mon[29572]: from='mgr.14150 v1:172.21.15.92:0/996057664' entity='mgr.y' 2024-03-11T21:21:36.842 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:21:36 smithi092 ceph-mon[29572]: from='mgr.14150 v1:172.21.15.92:0/996057664' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd/host:smithi097", "name": "osd_memory_target"}]: dispatch 2024-03-11T21:21:36.842 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:21:36 smithi092 ceph-mon[29572]: from='mgr.14150 v1:172.21.15.92:0/996057664' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T21:21:36.842 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:21:36 smithi092 ceph-mon[29572]: from='mgr.14150 v1:172.21.15.92:0/996057664' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T21:21:37.091 INFO:tasks.cephadm:Adding mon.a on smithi092 2024-03-11T21:21:37.092 INFO:tasks.cephadm:Adding mon.c on smithi092 2024-03-11T21:21:37.092 INFO:tasks.cephadm:Adding mon.b on smithi097 2024-03-11T21:21:37.092 DEBUG:teuthology.orchestra.run.smithi097:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 00db0016-dfed-11ee-95c8-87774f69a715 -- ceph orch apply mon '3;smithi092:[v1:172.21.15.92:6789]=a;smithi092:[v1:172.21.15.92:6790]=c;smithi097:[v1:172.21.15.97:6789]=b' 2024-03-11T21:21:37.321 INFO:teuthology.orchestra.run.smithi097.stderr:Inferring config /var/lib/ceph/00db0016-dfed-11ee-95c8-87774f69a715/config/ceph.conf 2024-03-11T21:21:37.862 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:21:37 smithi092 ceph-mon[29572]: Updating smithi097:/etc/ceph/ceph.conf 2024-03-11T21:21:37.862 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:21:37 smithi092 ceph-mon[29572]: Updating smithi097:/var/lib/ceph/00db0016-dfed-11ee-95c8-87774f69a715/config/ceph.conf 2024-03-11T21:21:37.862 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:21:37 smithi092 ceph-mon[29572]: from='client.? v1:172.21.15.92:0/2169605665' entity='client.admin' cmd='[{"prefix": "osd crush tunables", "profile": "default"}]': finished 2024-03-11T21:21:37.862 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:21:37 smithi092 ceph-mon[29572]: osdmap e4: 0 total, 0 up, 0 in 2024-03-11T21:21:37.863 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:21:37 smithi092 ceph-mon[29572]: from='mgr.14150 v1:172.21.15.92:0/996057664' entity='mgr.y' 2024-03-11T21:21:37.863 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:21:37 smithi092 ceph-mon[29572]: from='mgr.14150 v1:172.21.15.92:0/996057664' entity='mgr.y' 2024-03-11T21:21:37.863 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:21:37 smithi092 ceph-mon[29572]: from='mgr.14150 v1:172.21.15.92:0/996057664' entity='mgr.y' 2024-03-11T21:21:38.862 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:21:38 smithi092 ceph-mon[29572]: Updating smithi097:/etc/ceph/ceph.client.admin.keyring 2024-03-11T21:21:38.862 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:21:38 smithi092 ceph-mon[29572]: Updating smithi097:/var/lib/ceph/00db0016-dfed-11ee-95c8-87774f69a715/config/ceph.client.admin.keyring 2024-03-11T21:21:38.862 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:21:38 smithi092 ceph-mon[29572]: pgmap v8: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T21:21:38.893 INFO:teuthology.orchestra.run.smithi097.stdout:Scheduled mon update... 2024-03-11T21:21:39.518 DEBUG:teuthology.orchestra.run.smithi092:mon.c> sudo journalctl -f -n 0 -u ceph-00db0016-dfed-11ee-95c8-87774f69a715@mon.c.service 2024-03-11T21:21:39.520 DEBUG:teuthology.orchestra.run.smithi097:mon.b> sudo journalctl -f -n 0 -u ceph-00db0016-dfed-11ee-95c8-87774f69a715@mon.b.service 2024-03-11T21:21:39.523 INFO:tasks.cephadm:Waiting for 3 mons in monmap... 2024-03-11T21:21:39.523 DEBUG:teuthology.orchestra.run.smithi097:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 00db0016-dfed-11ee-95c8-87774f69a715 -- ceph mon dump -f json 2024-03-11T21:21:40.362 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:21:39 smithi092 ceph-mon[29572]: from='client.14182 v1:172.21.15.97:0/812993916' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mon", "placement": "3;smithi092:[v1:172.21.15.92:6789]=a;smithi092:[v1:172.21.15.92:6790]=c;smithi097:[v1:172.21.15.97:6789]=b", "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T21:21:40.362 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:21:39 smithi092 ceph-mon[29572]: Saving service mon spec with placement smithi092:[v1:172.21.15.92:6789]=a;smithi092:[v1:172.21.15.92:6790]=c;smithi097:[v1:172.21.15.97:6789]=b;count:3 2024-03-11T21:21:40.362 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:21:39 smithi092 ceph-mon[29572]: from='mgr.14150 v1:172.21.15.92:0/996057664' entity='mgr.y' 2024-03-11T21:21:40.362 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:21:39 smithi092 ceph-mon[29572]: from='mgr.14150 v1:172.21.15.92:0/996057664' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T21:21:40.363 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:21:39 smithi092 ceph-mon[29572]: from='mgr.14150 v1:172.21.15.92:0/996057664' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T21:21:40.363 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:21:39 smithi092 ceph-mon[29572]: from='mgr.14150 v1:172.21.15.92:0/996057664' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T21:21:40.363 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:21:39 smithi092 ceph-mon[29572]: from='mgr.14150 v1:172.21.15.92:0/996057664' entity='mgr.y' 2024-03-11T21:21:40.363 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:21:39 smithi092 ceph-mon[29572]: from='mgr.14150 v1:172.21.15.92:0/996057664' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-11T21:21:40.363 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:21:39 smithi092 ceph-mon[29572]: from='mgr.14150 v1:172.21.15.92:0/996057664' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T21:21:40.363 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:21:39 smithi092 ceph-mon[29572]: Deploying daemon mon.b on smithi097 2024-03-11T21:21:40.363 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:21:39 smithi092 ceph-mon[29572]: pgmap v9: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T21:21:41.547 INFO:teuthology.orchestra.run.smithi097.stderr:Inferring config /var/lib/ceph/00db0016-dfed-11ee-95c8-87774f69a715/mon.b/config 2024-03-11T21:21:42.862 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:21:42 smithi092 ceph-mon[29572]: pgmap v10: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T21:21:43.737 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-03-11T21:21:43.737 INFO:teuthology.orchestra.run.smithi097.stdout:{"epoch":1,"fsid":"00db0016-dfed-11ee-95c8-87774f69a715","modified":"2024-03-11T21:20:07.689951Z","created":"2024-03-11T21:20:07.689951Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.92:6789","nonce":0}]},"addr":"172.21.15.92:6789/0","public_addr":"172.21.15.92:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-03-11T21:21:43.739 INFO:teuthology.orchestra.run.smithi097.stderr:dumped monmap epoch 1 2024-03-11T21:21:44.078 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:21:43 smithi097 systemd[1]: Started Ceph mon.b for 00db0016-dfed-11ee-95c8-87774f69a715. 2024-03-11T21:21:45.446 INFO:tasks.cephadm:Waiting for 3 mons in monmap... 2024-03-11T21:21:45.446 DEBUG:teuthology.orchestra.run.smithi097:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 00db0016-dfed-11ee-95c8-87774f69a715 -- ceph mon dump -f json 2024-03-11T21:21:45.692 INFO:teuthology.orchestra.run.smithi097.stderr:Inferring config /var/lib/ceph/00db0016-dfed-11ee-95c8-87774f69a715/mon.b/config 2024-03-11T21:21:49.232 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-03-11T21:21:49.232 INFO:teuthology.orchestra.run.smithi097.stdout:{"epoch":2,"fsid":"00db0016-dfed-11ee-95c8-87774f69a715","modified":"2024-03-11T21:21:43.892511Z","created":"2024-03-11T21:20:07.689951Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.92:6789","nonce":0}]},"addr":"172.21.15.92:6789/0","public_addr":"172.21.15.92:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.97:6789","nonce":0}]},"addr":"172.21.15.97:6789/0","public_addr":"172.21.15.97:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0,1]} 2024-03-11T21:21:49.233 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:21:48 smithi097 ceph-mon[33317]: from='mgr.14150 v1:172.21.15.92:0/996057664' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-11T21:21:49.233 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:21:48 smithi097 ceph-mon[33317]: mon.a calling monitor election 2024-03-11T21:21:49.233 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:21:48 smithi097 ceph-mon[33317]: from='mgr.14150 v1:172.21.15.92:0/996057664' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-11T21:21:49.233 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:21:48 smithi097 ceph-mon[33317]: from='mgr.14150 v1:172.21.15.92:0/996057664' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-11T21:21:49.233 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:21:48 smithi097 ceph-mon[33317]: pgmap v12: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T21:21:49.233 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:21:48 smithi097 ceph-mon[33317]: from='mgr.14150 v1:172.21.15.92:0/996057664' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-11T21:21:49.233 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:21:48 smithi097 ceph-mon[33317]: mon.b calling monitor election 2024-03-11T21:21:49.233 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:21:48 smithi097 ceph-mon[33317]: from='mgr.14150 v1:172.21.15.92:0/996057664' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-11T21:21:49.233 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:21:48 smithi097 ceph-mon[33317]: pgmap v13: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T21:21:49.233 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:21:48 smithi097 ceph-mon[33317]: from='mgr.14150 v1:172.21.15.92:0/996057664' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-11T21:21:49.233 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:21:48 smithi097 ceph-mon[33317]: from='mgr.14150 v1:172.21.15.92:0/996057664' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-11T21:21:49.234 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:21:48 smithi097 ceph-mon[33317]: mon.a is new leader, mons a,b in quorum (ranks 0,1) 2024-03-11T21:21:49.234 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:21:48 smithi097 ceph-mon[33317]: monmap e2: 2 mons at {a=v1:172.21.15.92:6789/0,b=v1:172.21.15.97:6789/0} removed_ranks: {} disallowed_leaders: {} 2024-03-11T21:21:49.234 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:21:48 smithi097 ceph-mon[33317]: fsmap 2024-03-11T21:21:49.234 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:21:48 smithi097 ceph-mon[33317]: osdmap e4: 0 total, 0 up, 0 in 2024-03-11T21:21:49.234 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:21:48 smithi097 ceph-mon[33317]: mgrmap e13: y(active, since 39s) 2024-03-11T21:21:49.234 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:21:48 smithi097 ceph-mon[33317]: overall HEALTH_OK 2024-03-11T21:21:49.234 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:21:48 smithi097 ceph-mon[33317]: from='mgr.14150 v1:172.21.15.92:0/996057664' entity='mgr.y' 2024-03-11T21:21:49.234 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:21:48 smithi097 ceph-mon[33317]: from='mgr.14150 v1:172.21.15.92:0/996057664' entity='mgr.y' 2024-03-11T21:21:49.234 INFO:teuthology.orchestra.run.smithi097.stderr:dumped monmap epoch 2 2024-03-11T21:21:49.362 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:21:48 smithi092 ceph-mon[29572]: from='mgr.14150 v1:172.21.15.92:0/996057664' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-11T21:21:49.362 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:21:48 smithi092 ceph-mon[29572]: mon.a calling monitor election 2024-03-11T21:21:49.362 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:21:48 smithi092 ceph-mon[29572]: from='mgr.14150 v1:172.21.15.92:0/996057664' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-11T21:21:49.362 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:21:48 smithi092 ceph-mon[29572]: from='mgr.14150 v1:172.21.15.92:0/996057664' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-11T21:21:49.363 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:21:48 smithi092 ceph-mon[29572]: pgmap v12: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T21:21:49.363 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:21:48 smithi092 ceph-mon[29572]: from='mgr.14150 v1:172.21.15.92:0/996057664' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-11T21:21:49.363 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:21:48 smithi092 ceph-mon[29572]: mon.b calling monitor election 2024-03-11T21:21:49.363 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:21:48 smithi092 ceph-mon[29572]: from='mgr.14150 v1:172.21.15.92:0/996057664' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-11T21:21:49.363 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:21:48 smithi092 ceph-mon[29572]: pgmap v13: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T21:21:49.363 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:21:48 smithi092 ceph-mon[29572]: from='mgr.14150 v1:172.21.15.92:0/996057664' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-11T21:21:49.363 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:21:48 smithi092 ceph-mon[29572]: from='mgr.14150 v1:172.21.15.92:0/996057664' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-11T21:21:49.363 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:21:48 smithi092 ceph-mon[29572]: mon.a is new leader, mons a,b in quorum (ranks 0,1) 2024-03-11T21:21:49.363 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:21:48 smithi092 ceph-mon[29572]: monmap e2: 2 mons at {a=v1:172.21.15.92:6789/0,b=v1:172.21.15.97:6789/0} removed_ranks: {} disallowed_leaders: {} 2024-03-11T21:21:49.363 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:21:48 smithi092 ceph-mon[29572]: fsmap 2024-03-11T21:21:49.363 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:21:48 smithi092 ceph-mon[29572]: osdmap e4: 0 total, 0 up, 0 in 2024-03-11T21:21:49.363 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:21:48 smithi092 ceph-mon[29572]: mgrmap e13: y(active, since 39s) 2024-03-11T21:21:49.363 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:21:48 smithi092 ceph-mon[29572]: overall HEALTH_OK 2024-03-11T21:21:49.363 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:21:48 smithi092 ceph-mon[29572]: from='mgr.14150 v1:172.21.15.92:0/996057664' entity='mgr.y' 2024-03-11T21:21:49.364 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:21:48 smithi092 ceph-mon[29572]: from='mgr.14150 v1:172.21.15.92:0/996057664' entity='mgr.y' 2024-03-11T21:21:50.078 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:21:50 smithi097 ceph-mon[33317]: from='mgr.14150 v1:172.21.15.92:0/996057664' entity='mgr.y' 2024-03-11T21:21:50.078 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:21:50 smithi097 ceph-mon[33317]: from='mgr.14150 v1:172.21.15.92:0/996057664' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-11T21:21:50.078 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:21:50 smithi097 ceph-mon[33317]: from='mgr.14150 v1:172.21.15.92:0/996057664' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T21:21:50.078 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:21:50 smithi097 ceph-mon[33317]: Deploying daemon mon.c on smithi092 2024-03-11T21:21:50.078 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:21:50 smithi097 ceph-mon[33317]: from='client.? v1:172.21.15.97:0/2488983162' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-03-11T21:21:50.078 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:21:50 smithi097 ceph-mon[33317]: pgmap v14: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T21:21:50.078 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:21:50 smithi097 ceph-mon[33317]: from='mgr.14150 v1:172.21.15.92:0/996057664' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-11T21:21:50.362 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:21:50 smithi092 ceph-mon[29572]: from='mgr.14150 v1:172.21.15.92:0/996057664' entity='mgr.y' 2024-03-11T21:21:50.362 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:21:50 smithi092 ceph-mon[29572]: from='mgr.14150 v1:172.21.15.92:0/996057664' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-11T21:21:50.362 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:21:50 smithi092 ceph-mon[29572]: from='mgr.14150 v1:172.21.15.92:0/996057664' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T21:21:50.362 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:21:50 smithi092 ceph-mon[29572]: Deploying daemon mon.c on smithi092 2024-03-11T21:21:50.362 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:21:50 smithi092 ceph-mon[29572]: from='client.? v1:172.21.15.97:0/2488983162' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-03-11T21:21:50.363 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:21:50 smithi092 ceph-mon[29572]: pgmap v14: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T21:21:50.363 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:21:50 smithi092 ceph-mon[29572]: from='mgr.14150 v1:172.21.15.92:0/996057664' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-11T21:21:50.818 INFO:tasks.cephadm:Waiting for 3 mons in monmap... 2024-03-11T21:21:50.819 DEBUG:teuthology.orchestra.run.smithi097:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 00db0016-dfed-11ee-95c8-87774f69a715 -- ceph mon dump -f json 2024-03-11T21:21:51.064 INFO:teuthology.orchestra.run.smithi097.stderr:Inferring config /var/lib/ceph/00db0016-dfed-11ee-95c8-87774f69a715/mon.b/config 2024-03-11T21:21:52.287 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:21:52 smithi092 ceph-mon[37563]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[2]: 1 2024-03-11T21:21:52.287 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:21:52 smithi092 ceph-mon[37563]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[3]: 1 2024-03-11T21:21:52.287 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:21:52 smithi092 ceph-mon[37563]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[4]: 1 2024-03-11T21:21:52.287 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:21:52 smithi092 ceph-mon[37563]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[5]: 1 2024-03-11T21:21:52.287 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:21:52 smithi092 ceph-mon[37563]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[6]: 1 2024-03-11T21:21:52.287 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:21:52 smithi092 ceph-mon[37563]: rocksdb: Options.max_sequential_skip_in_iterations: 8 2024-03-11T21:21:52.287 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:21:52 smithi092 ceph-mon[37563]: rocksdb: Options.max_compaction_bytes: 1677721600 2024-03-11T21:21:52.287 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:21:52 smithi092 ceph-mon[37563]: rocksdb: Options.ignore_max_compaction_bytes_for_input: true 2024-03-11T21:21:52.288 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:21:52 smithi092 ceph-mon[37563]: rocksdb: Options.arena_block_size: 1048576 2024-03-11T21:21:52.288 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:21:52 smithi092 ceph-mon[37563]: rocksdb: Options.soft_pending_compaction_bytes_limit: 68719476736 2024-03-11T21:21:52.288 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:21:52 smithi092 ceph-mon[37563]: rocksdb: Options.hard_pending_compaction_bytes_limit: 274877906944 2024-03-11T21:21:52.288 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:21:52 smithi092 ceph-mon[37563]: rocksdb: Options.disable_auto_compactions: 0 2024-03-11T21:21:52.288 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:21:52 smithi092 ceph-mon[37563]: rocksdb: Options.compaction_style: kCompactionStyleLevel 2024-03-11T21:21:52.288 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:21:52 smithi092 ceph-mon[37563]: rocksdb: Options.compaction_pri: kMinOverlappingRatio 2024-03-11T21:21:52.288 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:21:52 smithi092 ceph-mon[37563]: rocksdb: Options.compaction_options_universal.size_ratio: 1 2024-03-11T21:21:52.288 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:21:52 smithi092 ceph-mon[37563]: rocksdb: Options.compaction_options_universal.min_merge_width: 2 2024-03-11T21:21:52.288 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:21:52 smithi092 ceph-mon[37563]: rocksdb: Options.compaction_options_universal.max_merge_width: 4294967295 2024-03-11T21:21:52.288 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:21:52 smithi092 ceph-mon[37563]: rocksdb: Options.compaction_options_universal.max_size_amplification_percent: 200 2024-03-11T21:21:52.288 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:21:52 smithi092 ceph-mon[37563]: rocksdb: Options.compaction_options_universal.compression_size_percent: -1 2024-03-11T21:21:52.288 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:21:52 smithi092 ceph-mon[37563]: rocksdb: Options.compaction_options_universal.stop_style: kCompactionStopStyleTotalSize 2024-03-11T21:21:52.288 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:21:52 smithi092 ceph-mon[37563]: rocksdb: Options.compaction_options_fifo.max_table_files_size: 1073741824 2024-03-11T21:21:52.288 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:21:52 smithi092 ceph-mon[37563]: rocksdb: Options.compaction_options_fifo.allow_compaction: 0 2024-03-11T21:21:52.613 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:21:52 smithi092 ceph-mon[37563]: rocksdb: Options.table_properties_collectors: 2024-03-11T21:21:52.613 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:21:52 smithi092 ceph-mon[37563]: rocksdb: Options.inplace_update_support: 0 2024-03-11T21:21:52.613 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:21:52 smithi092 ceph-mon[37563]: rocksdb: Options.inplace_update_num_locks: 10000 2024-03-11T21:21:52.613 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:21:52 smithi092 ceph-mon[37563]: rocksdb: Options.memtable_prefix_bloom_size_ratio: 0.000000 2024-03-11T21:21:52.613 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:21:52 smithi092 ceph-mon[37563]: rocksdb: Options.memtable_whole_key_filtering: 0 2024-03-11T21:21:52.614 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:21:52 smithi092 ceph-mon[37563]: rocksdb: Options.memtable_huge_page_size: 0 2024-03-11T21:21:52.614 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:21:52 smithi092 ceph-mon[37563]: rocksdb: Options.bloom_locality: 0 2024-03-11T21:21:52.614 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:21:52 smithi092 ceph-mon[37563]: rocksdb: Options.max_successive_merges: 0 2024-03-11T21:21:52.614 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:21:52 smithi092 ceph-mon[37563]: rocksdb: Options.optimize_filters_for_hits: 0 2024-03-11T21:21:52.614 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:21:52 smithi092 ceph-mon[37563]: rocksdb: Options.paranoid_file_checks: 0 2024-03-11T21:21:52.614 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:21:52 smithi092 ceph-mon[37563]: rocksdb: Options.force_consistency_checks: 1 2024-03-11T21:21:52.614 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:21:52 smithi092 ceph-mon[37563]: rocksdb: Options.report_bg_io_stats: 0 2024-03-11T21:21:52.614 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:21:52 smithi092 ceph-mon[37563]: rocksdb: Options.ttl: 2592000 2024-03-11T21:21:52.614 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:21:52 smithi092 ceph-mon[37563]: rocksdb: Options.periodic_compaction_seconds: 0 2024-03-11T21:21:52.614 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:21:52 smithi092 ceph-mon[37563]: rocksdb: Options.preclude_last_level_data_seconds: 0 2024-03-11T21:21:52.614 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:21:52 smithi092 ceph-mon[37563]: rocksdb: Options.preserve_internal_time_seconds: 0 2024-03-11T21:21:52.614 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:21:52 smithi092 ceph-mon[37563]: rocksdb: Options.enable_blob_files: false 2024-03-11T21:21:52.614 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:21:52 smithi092 ceph-mon[37563]: rocksdb: Options.min_blob_size: 0 2024-03-11T21:21:52.614 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:21:52 smithi092 ceph-mon[37563]: rocksdb: Options.blob_file_size: 268435456 2024-03-11T21:21:52.615 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:21:52 smithi092 ceph-mon[37563]: rocksdb: Options.blob_compression_type: NoCompression 2024-03-11T21:21:52.615 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:21:52 smithi092 ceph-mon[37563]: rocksdb: Options.enable_blob_garbage_collection: false 2024-03-11T21:21:52.615 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:21:52 smithi092 ceph-mon[37563]: rocksdb: Options.blob_garbage_collection_age_cutoff: 0.250000 2024-03-11T21:21:52.615 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:21:52 smithi092 ceph-mon[37563]: rocksdb: Options.blob_garbage_collection_force_threshold: 1.000000 2024-03-11T21:21:52.615 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:21:52 smithi092 ceph-mon[37563]: rocksdb: Options.blob_compaction_readahead_size: 0 2024-03-11T21:21:52.615 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:21:52 smithi092 ceph-mon[37563]: rocksdb: Options.blob_file_starting_level: 0 2024-03-11T21:21:52.615 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:21:52 smithi092 ceph-mon[37563]: rocksdb: Options.experimental_mempurge_threshold: 0.000000 2024-03-11T21:21:52.615 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:21:52 smithi092 ceph-mon[37563]: rocksdb: [db/version_set.cc:5566] Recovered from manifest file:/var/lib/ceph/mon/ceph-c/store.db/MANIFEST-000005 succeeded,manifest_file_number is 5, next_file_number is 7, last_sequence is 0, log_number is 0,prev_log_number is 0,max_column_family is 0,min_log_number_to_keep is 0 2024-03-11T21:21:52.615 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:21:52 smithi092 ceph-mon[37563]: rocksdb: [db/version_set.cc:5581] Column family [default] (ID 0), log number is 0 2024-03-11T21:21:52.615 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:21:52 smithi092 ceph-mon[37563]: rocksdb: [db/db_impl/db_impl_open.cc:539] DB ID: d894163b-a579-409c-a5c5-5336f54f3fca 2024-03-11T21:21:52.615 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:21:52 smithi092 ceph-mon[37563]: rocksdb: EVENT_LOG_v1 {"time_micros": 1710192112298933, "job": 1, "event": "recovery_started", "wal_files": [4]} 2024-03-11T21:21:52.615 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:21:52 smithi092 ceph-mon[37563]: rocksdb: [db/db_impl/db_impl_open.cc:1043] Recovering log #4 mode 2 2024-03-11T21:21:52.615 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:21:52 smithi092 ceph-mon[37563]: rocksdb: EVENT_LOG_v1 {"time_micros": 1710192112299396, "cf_name": "default", "job": 1, "event": "table_file_creation", "file_number": 8, "file_size": 1677, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 1, "largest_seqno": 5, "table_properties": {"data_size": 578, "index_size": 31, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 1, "index_value_is_delta_encoded": 1, "filter_size": 69, "raw_key_size": 115, "raw_average_key_size": 23, "raw_value_size": 456, "raw_average_value_size": 91, "num_data_blocks": 1, "num_entries": 5, "num_filter_entries": 5, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 0, "filter_policy": "bloomfilter", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "", "prefix_extractor_name": "nullptr", "property_collectors": "[]", "compression": "NoCompression", "compression_options": "window_bits=-14; level=32767; strategy=0; max_dict_bytes=0; zstd_max_train_bytes=0; enabled=0; max_dict_buffer_bytes=0; use_zstd_dict_trainer=1; ", "creation_time": 1710192112, "oldest_key_time": 0, "file_creation_time": 0, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "d894163b-a579-409c-a5c5-5336f54f3fca", "db_session_id": "EAI3YZ4SJ71KSQQUVOGQ", "orig_file_number": 8, "seqno_to_time_mapping": "N/A"}} 2024-03-11T21:21:52.616 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:21:52 smithi092 ceph-mon[37563]: rocksdb: EVENT_LOG_v1 {"time_micros": 1710192112299477, "job": 1, "event": "recovery_finished"} 2024-03-11T21:21:52.616 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:21:52 smithi092 ceph-mon[37563]: rocksdb: [db/version_set.cc:5047] Creating manifest 10 2024-03-11T21:21:52.616 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:21:52 smithi092 ceph-mon[37563]: rocksdb: [file/delete_scheduler.cc:74] Deleted file /var/lib/ceph/mon/ceph-c/store.db/000004.log immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024-03-11T21:21:52.616 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:21:52 smithi092 ceph-mon[37563]: rocksdb: [db/db_impl/db_impl_open.cc:1987] SstFileManager instance 0x561c41b9c000 2024-03-11T21:21:52.616 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:21:52 smithi092 ceph-mon[37563]: rocksdb: DB pointer 0x561c41b86000 2024-03-11T21:21:52.616 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:21:52 smithi092 ceph-mon[37563]: rocksdb: [db/db_impl/db_impl.cc:1109] ------- DUMPING STATS ------- 2024-03-11T21:21:52.616 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:21:52 smithi092 ceph-mon[37563]: rocksdb: [db/db_impl/db_impl.cc:1111] 2024-03-11T21:21:52.616 INFO:journalctl@ceph.mon.c.smithi092.stdout: ** DB Stats ** 2024-03-11T21:21:52.616 INFO:journalctl@ceph.mon.c.smithi092.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-03-11T21:21:52.616 INFO:journalctl@ceph.mon.c.smithi092.stdout: Cumulative writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 GB, 0.00 MB/s 2024-03-11T21:21:52.616 INFO:journalctl@ceph.mon.c.smithi092.stdout: Cumulative WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-03-11T21:21:52.616 INFO:journalctl@ceph.mon.c.smithi092.stdout: Cumulative stall: 00:00:0.000 H:M:S, 0.0 percent 2024-03-11T21:21:52.616 INFO:journalctl@ceph.mon.c.smithi092.stdout: Interval writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 MB, 0.00 MB/s 2024-03-11T21:21:52.617 INFO:journalctl@ceph.mon.c.smithi092.stdout: Interval WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-03-11T21:21:52.617 INFO:journalctl@ceph.mon.c.smithi092.stdout: Interval stall: 00:00:0.000 H:M:S, 0.0 percent 2024-03-11T21:21:52.617 INFO:journalctl@ceph.mon.c.smithi092.stdout: 2024-03-11T21:21:52.617 INFO:journalctl@ceph.mon.c.smithi092.stdout: ** Compaction Stats [default] ** 2024-03-11T21:21:52.617 INFO:journalctl@ceph.mon.c.smithi092.stdout: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2024-03-11T21:21:52.617 INFO:journalctl@ceph.mon.c.smithi092.stdout: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ 2024-03-11T21:21:52.617 INFO:journalctl@ceph.mon.c.smithi092.stdout: L0 1/0 1.64 KB 0.2 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 3.8 0.00 0.00 1 0.000 0 0 0.0 0.0 2024-03-11T21:21:52.617 INFO:journalctl@ceph.mon.c.smithi092.stdout: Sum 1/0 1.64 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 3.8 0.00 0.00 1 0.000 0 0 0.0 0.0 2024-03-11T21:21:52.617 INFO:journalctl@ceph.mon.c.smithi092.stdout: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 3.8 0.00 0.00 1 0.000 0 0 0.0 0.0 2024-03-11T21:21:52.617 INFO:journalctl@ceph.mon.c.smithi092.stdout: 2024-03-11T21:21:52.617 INFO:journalctl@ceph.mon.c.smithi092.stdout: ** Compaction Stats [default] ** 2024-03-11T21:21:52.617 INFO:journalctl@ceph.mon.c.smithi092.stdout: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2024-03-11T21:21:52.617 INFO:journalctl@ceph.mon.c.smithi092.stdout: --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-03-11T21:21:52.617 INFO:journalctl@ceph.mon.c.smithi092.stdout: User 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 3.8 0.00 0.00 1 0.000 0 0 0.0 0.0 2024-03-11T21:21:52.618 INFO:journalctl@ceph.mon.c.smithi092.stdout: 2024-03-11T21:21:52.618 INFO:journalctl@ceph.mon.c.smithi092.stdout: Blob file count: 0, total size: 0.0 GB, garbage size: 0.0 GB, space amp: 0.0 2024-03-11T21:21:52.618 INFO:journalctl@ceph.mon.c.smithi092.stdout: 2024-03-11T21:21:52.618 INFO:journalctl@ceph.mon.c.smithi092.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-03-11T21:21:52.618 INFO:journalctl@ceph.mon.c.smithi092.stdout: Flush(GB): cumulative 0.000, interval 0.000 2024-03-11T21:21:52.618 INFO:journalctl@ceph.mon.c.smithi092.stdout: AddFile(GB): cumulative 0.000, interval 0.000 2024-03-11T21:21:52.618 INFO:journalctl@ceph.mon.c.smithi092.stdout: AddFile(Total Files): cumulative 0, interval 0 2024-03-11T21:21:52.618 INFO:journalctl@ceph.mon.c.smithi092.stdout: AddFile(L0 Files): cumulative 0, interval 0 2024-03-11T21:21:52.618 INFO:journalctl@ceph.mon.c.smithi092.stdout: AddFile(Keys): cumulative 0, interval 0 2024-03-11T21:21:52.618 INFO:journalctl@ceph.mon.c.smithi092.stdout: Cumulative compaction: 0.00 GB write, 0.66 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-03-11T21:21:52.618 INFO:journalctl@ceph.mon.c.smithi092.stdout: Interval compaction: 0.00 GB write, 0.66 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-03-11T21:21:52.618 INFO:journalctl@ceph.mon.c.smithi092.stdout: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2024-03-11T21:21:52.618 INFO:journalctl@ceph.mon.c.smithi092.stdout: Block cache BinnedLRUCache@0x561c41aa9090#2 capacity: 512.00 MB usage: 0.22 KB table_size: 0 occupancy: 18446744073709551615 collections: 1 last_copies: 0 last_secs: 9e-06 secs_since: 0 2024-03-11T21:21:52.619 INFO:journalctl@ceph.mon.c.smithi092.stdout: Block cache entry stats(count,size,portion): FilterBlock(1,0.11 KB,2.08616e-05%) IndexBlock(1,0.11 KB,2.08616e-05%) Misc(1,0.00 KB,0%) 2024-03-11T21:21:52.619 INFO:journalctl@ceph.mon.c.smithi092.stdout: 2024-03-11T21:21:52.619 INFO:journalctl@ceph.mon.c.smithi092.stdout: ** File Read Latency Histogram By Level [default] ** 2024-03-11T21:21:52.619 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:21:52 smithi092 ceph-mon[37563]: mon.c does not exist in monmap, will attempt to join an existing cluster 2024-03-11T21:21:52.619 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:21:52 smithi092 ceph-mon[37563]: using public_addrv v1:172.21.15.92:6790/0 2024-03-11T21:21:52.619 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:21:52 smithi092 ceph-mon[37563]: starting mon.c rank -1 at public addrs v1:172.21.15.92:6790/0 at bind addrs v1:172.21.15.92:6790/0 mon_data /var/lib/ceph/mon/ceph-c fsid 00db0016-dfed-11ee-95c8-87774f69a715 2024-03-11T21:21:52.619 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:21:52 smithi092 ceph-mon[37563]: mon.c@-1(???) e0 preinit fsid 00db0016-dfed-11ee-95c8-87774f69a715 2024-03-11T21:21:52.619 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:21:52 smithi092 ceph-mon[37563]: mon.c@-1(synchronizing).mds e1 new map 2024-03-11T21:21:52.619 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:21:52 smithi092 ceph-mon[37563]: mon.c@-1(synchronizing).mds e1 print_map 2024-03-11T21:21:52.619 INFO:journalctl@ceph.mon.c.smithi092.stdout: e1 2024-03-11T21:21:52.619 INFO:journalctl@ceph.mon.c.smithi092.stdout: enable_multiple, ever_enabled_multiple: 1,1 2024-03-11T21:21:52.619 INFO:journalctl@ceph.mon.c.smithi092.stdout: default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2,11=minor log segments} 2024-03-11T21:21:52.619 INFO:journalctl@ceph.mon.c.smithi092.stdout: legacy client fscid: -1 2024-03-11T21:21:52.619 INFO:journalctl@ceph.mon.c.smithi092.stdout: 2024-03-11T21:21:52.619 INFO:journalctl@ceph.mon.c.smithi092.stdout: No filesystems configured 2024-03-11T21:21:52.620 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:21:52 smithi092 ceph-mon[37563]: mon.c@-1(synchronizing).osd e0 _set_cache_ratios kv ratio 0.25 inc ratio 0.375 full ratio 0.375 2024-03-11T21:21:52.620 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:21:52 smithi092 ceph-mon[37563]: mon.c@-1(synchronizing).osd e0 register_cache_with_pcm pcm target: 2147483648 pcm max: 1020054732 pcm min: 134217728 inc_osd_cache size: 1 2024-03-11T21:21:52.620 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:21:52 smithi092 ceph-mon[37563]: mon.c@-1(synchronizing).osd e1 e1: 0 total, 0 up, 0 in 2024-03-11T21:21:52.620 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:21:52 smithi092 ceph-mon[37563]: mon.c@-1(synchronizing).osd e2 e2: 0 total, 0 up, 0 in 2024-03-11T21:21:52.620 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:21:52 smithi092 ceph-mon[37563]: mon.c@-1(synchronizing).osd e3 e3: 0 total, 0 up, 0 in 2024-03-11T21:21:52.620 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:21:52 smithi092 ceph-mon[37563]: mon.c@-1(synchronizing).osd e4 e4: 0 total, 0 up, 0 in 2024-03-11T21:21:52.620 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:21:52 smithi092 ceph-mon[37563]: mon.c@-1(synchronizing).osd e4 crush map has features 3314932999778484224, adjusting msgr requires 2024-03-11T21:21:52.620 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:21:52 smithi092 ceph-mon[37563]: mon.c@-1(synchronizing).osd e4 crush map has features 288514050185494528, adjusting msgr requires 2024-03-11T21:21:52.620 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:21:52 smithi092 ceph-mon[37563]: mon.c@-1(synchronizing).osd e4 crush map has features 288514050185494528, adjusting msgr requires 2024-03-11T21:21:52.620 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:21:52 smithi092 ceph-mon[37563]: mon.c@-1(synchronizing).osd e4 crush map has features 288514050185494528, adjusting msgr requires 2024-03-11T21:21:52.620 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:21:52 smithi092 ceph-mon[37563]: from='client.14182 v1:172.21.15.97:0/812993916' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mon", "placement": "3;smithi092:[v1:172.21.15.92:6789]=a;smithi092:[v1:172.21.15.92:6790]=c;smithi097:[v1:172.21.15.97:6789]=b", "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T21:21:52.620 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:21:52 smithi092 ceph-mon[37563]: Saving service mon spec with placement smithi092:[v1:172.21.15.92:6789]=a;smithi092:[v1:172.21.15.92:6790]=c;smithi097:[v1:172.21.15.97:6789]=b;count:3 2024-03-11T21:21:52.620 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:21:52 smithi092 ceph-mon[37563]: from='mgr.14150 v1:172.21.15.92:0/996057664' entity='mgr.y' 2024-03-11T21:21:52.620 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:21:52 smithi092 ceph-mon[37563]: from='mgr.14150 v1:172.21.15.92:0/996057664' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T21:21:52.621 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:21:52 smithi092 ceph-mon[37563]: from='mgr.14150 v1:172.21.15.92:0/996057664' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T21:21:52.621 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:21:52 smithi092 ceph-mon[37563]: from='mgr.14150 v1:172.21.15.92:0/996057664' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T21:21:52.621 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:21:52 smithi092 ceph-mon[37563]: from='mgr.14150 v1:172.21.15.92:0/996057664' entity='mgr.y' 2024-03-11T21:21:52.621 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:21:52 smithi092 ceph-mon[37563]: from='mgr.14150 v1:172.21.15.92:0/996057664' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-11T21:21:52.621 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:21:52 smithi092 ceph-mon[37563]: from='mgr.14150 v1:172.21.15.92:0/996057664' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T21:21:52.621 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:21:52 smithi092 ceph-mon[37563]: Deploying daemon mon.b on smithi097 2024-03-11T21:21:52.621 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:21:52 smithi092 ceph-mon[37563]: pgmap v9: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T21:21:52.621 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:21:52 smithi092 ceph-mon[37563]: pgmap v10: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T21:21:52.621 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:21:52 smithi092 ceph-mon[37563]: from='mgr.14150 v1:172.21.15.92:0/996057664' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-11T21:21:52.621 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:21:52 smithi092 ceph-mon[37563]: mon.a calling monitor election 2024-03-11T21:21:52.621 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:21:52 smithi092 ceph-mon[37563]: from='mgr.14150 v1:172.21.15.92:0/996057664' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-11T21:21:52.621 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:21:52 smithi092 ceph-mon[37563]: from='mgr.14150 v1:172.21.15.92:0/996057664' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-11T21:21:52.621 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:21:52 smithi092 ceph-mon[37563]: pgmap v12: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T21:21:52.621 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:21:52 smithi092 ceph-mon[37563]: from='mgr.14150 v1:172.21.15.92:0/996057664' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-11T21:21:52.621 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:21:52 smithi092 ceph-mon[37563]: mon.b calling monitor election 2024-03-11T21:21:52.622 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:21:52 smithi092 ceph-mon[37563]: from='mgr.14150 v1:172.21.15.92:0/996057664' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-11T21:21:52.622 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:21:52 smithi092 ceph-mon[37563]: pgmap v13: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T21:21:52.622 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:21:52 smithi092 ceph-mon[37563]: from='mgr.14150 v1:172.21.15.92:0/996057664' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-11T21:21:52.622 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:21:52 smithi092 ceph-mon[37563]: from='mgr.14150 v1:172.21.15.92:0/996057664' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-11T21:21:52.622 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:21:52 smithi092 ceph-mon[37563]: mon.a is new leader, mons a,b in quorum (ranks 0,1) 2024-03-11T21:21:52.622 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:21:52 smithi092 ceph-mon[37563]: monmap e2: 2 mons at {a=v1:172.21.15.92:6789/0,b=v1:172.21.15.97:6789/0} removed_ranks: {} disallowed_leaders: {} 2024-03-11T21:21:52.622 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:21:52 smithi092 ceph-mon[37563]: fsmap 2024-03-11T21:21:52.622 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:21:52 smithi092 ceph-mon[37563]: osdmap e4: 0 total, 0 up, 0 in 2024-03-11T21:21:52.622 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:21:52 smithi092 ceph-mon[37563]: mgrmap e13: y(active, since 39s) 2024-03-11T21:21:52.622 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:21:52 smithi092 ceph-mon[37563]: overall HEALTH_OK 2024-03-11T21:21:52.622 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:21:52 smithi092 ceph-mon[37563]: from='mgr.14150 v1:172.21.15.92:0/996057664' entity='mgr.y' 2024-03-11T21:21:52.622 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:21:52 smithi092 ceph-mon[37563]: from='mgr.14150 v1:172.21.15.92:0/996057664' entity='mgr.y' 2024-03-11T21:21:52.622 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:21:52 smithi092 ceph-mon[37563]: from='mgr.14150 v1:172.21.15.92:0/996057664' entity='mgr.y' 2024-03-11T21:21:52.622 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:21:52 smithi092 ceph-mon[37563]: from='mgr.14150 v1:172.21.15.92:0/996057664' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-11T21:21:52.622 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:21:52 smithi092 ceph-mon[37563]: from='mgr.14150 v1:172.21.15.92:0/996057664' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T21:21:52.623 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:21:52 smithi092 ceph-mon[37563]: Deploying daemon mon.c on smithi092 2024-03-11T21:21:52.623 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:21:52 smithi092 ceph-mon[37563]: from='client.? v1:172.21.15.97:0/2488983162' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-03-11T21:21:52.623 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:21:52 smithi092 ceph-mon[37563]: pgmap v14: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T21:21:52.623 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:21:52 smithi092 ceph-mon[37563]: from='mgr.14150 v1:172.21.15.92:0/996057664' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-11T21:21:52.623 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:21:52 smithi092 ceph-mon[37563]: mon.c@-1(synchronizing).paxosservice(auth 1..4) refresh upgraded, format 0 -> 3 2024-03-11T21:21:52.623 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:21:52 smithi092 bash[37536]: f3c58c2e4c882f276ebfab3f9f184559400cdd4a5ebca9bfeb2cdaee68940465 2024-03-11T21:21:52.623 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:21:52 smithi092 systemd[1]: Started Ceph mon.c for 00db0016-dfed-11ee-95c8-87774f69a715. 2024-03-11T21:21:57.349 INFO:teuthology.orchestra.run.smithi097.stdout: 2024-03-11T21:21:57.349 INFO:teuthology.orchestra.run.smithi097.stdout:{"epoch":3,"fsid":"00db0016-dfed-11ee-95c8-87774f69a715","modified":"2024-03-11T21:21:52.320521Z","created":"2024-03-11T21:20:07.689951Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.92:6789","nonce":0}]},"addr":"172.21.15.92:6789/0","public_addr":"172.21.15.92:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.97:6789","nonce":0}]},"addr":"172.21.15.97:6789/0","public_addr":"172.21.15.97:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.92:6790","nonce":0}]},"addr":"172.21.15.92:6790/0","public_addr":"172.21.15.92:6790/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0,1,2]} 2024-03-11T21:21:57.353 INFO:teuthology.orchestra.run.smithi097.stderr:dumped monmap epoch 3 2024-03-11T21:21:57.626 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:21:57 smithi097 ceph-mon[33317]: from='mgr.14150 v1:172.21.15.92:0/996057664' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-11T21:21:57.626 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:21:57 smithi097 ceph-mon[33317]: mon.a calling monitor election 2024-03-11T21:21:57.626 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:21:57 smithi097 ceph-mon[33317]: mon.b calling monitor election 2024-03-11T21:21:57.626 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:21:57 smithi097 ceph-mon[33317]: from='mgr.14150 v1:172.21.15.92:0/996057664' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-11T21:21:57.626 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:21:57 smithi097 ceph-mon[33317]: from='mgr.14150 v1:172.21.15.92:0/996057664' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-11T21:21:57.626 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:21:57 smithi097 ceph-mon[33317]: from='client.? v1:172.21.15.97:0/2087245699' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-03-11T21:21:57.626 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:21:57 smithi097 ceph-mon[33317]: from='mgr.14150 v1:172.21.15.92:0/996057664' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-11T21:21:57.626 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:21:57 smithi097 ceph-mon[33317]: pgmap v16: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T21:21:57.627 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:21:57 smithi097 ceph-mon[33317]: from='mgr.14150 v1:172.21.15.92:0/996057664' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-11T21:21:57.627 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:21:57 smithi097 ceph-mon[33317]: mon.c calling monitor election 2024-03-11T21:21:57.627 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:21:57 smithi097 ceph-mon[33317]: from='mgr.14150 v1:172.21.15.92:0/996057664' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-11T21:21:57.627 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:21:57 smithi097 ceph-mon[33317]: pgmap v17: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T21:21:57.627 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:21:57 smithi097 ceph-mon[33317]: from='mgr.14150 v1:172.21.15.92:0/996057664' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-11T21:21:57.627 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:21:57 smithi097 ceph-mon[33317]: from='mgr.14150 v1:172.21.15.92:0/996057664' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-11T21:21:57.627 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:21:57 smithi097 ceph-mon[33317]: mon.a is new leader, mons a,b,c in quorum (ranks 0,1,2) 2024-03-11T21:21:57.627 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:21:57 smithi097 ceph-mon[33317]: monmap e3: 3 mons at {a=v1:172.21.15.92:6789/0,b=v1:172.21.15.97:6789/0,c=v1:172.21.15.92:6790/0} removed_ranks: {} disallowed_leaders: {} 2024-03-11T21:21:57.627 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:21:57 smithi097 ceph-mon[33317]: fsmap 2024-03-11T21:21:57.627 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:21:57 smithi097 ceph-mon[33317]: osdmap e4: 0 total, 0 up, 0 in 2024-03-11T21:21:57.627 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:21:57 smithi097 ceph-mon[33317]: mgrmap e13: y(active, since 47s) 2024-03-11T21:21:57.627 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:21:57 smithi097 ceph-mon[33317]: overall HEALTH_OK 2024-03-11T21:21:57.627 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:21:57 smithi097 ceph-mon[33317]: from='mgr.14150 v1:172.21.15.92:0/996057664' entity='mgr.y' 2024-03-11T21:21:57.627 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:21:57 smithi097 ceph-mon[33317]: from='mgr.14150 v1:172.21.15.92:0/996057664' entity='mgr.y' 2024-03-11T21:21:57.627 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:21:57 smithi097 ceph-mon[33317]: from='mgr.14150 v1:172.21.15.92:0/996057664' entity='mgr.y' 2024-03-11T21:21:57.628 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:21:57 smithi097 ceph-mon[33317]: from='mgr.14150 v1:172.21.15.92:0/996057664' entity='mgr.y' 2024-03-11T21:21:57.862 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:21:57 smithi092 ceph-mon[29572]: from='mgr.14150 v1:172.21.15.92:0/996057664' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-11T21:21:57.862 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:21:57 smithi092 ceph-mon[29572]: mon.a calling monitor election 2024-03-11T21:21:57.862 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:21:57 smithi092 ceph-mon[29572]: mon.b calling monitor election 2024-03-11T21:21:57.863 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:21:57 smithi092 ceph-mon[29572]: from='mgr.14150 v1:172.21.15.92:0/996057664' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-11T21:21:57.863 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:21:57 smithi092 ceph-mon[29572]: from='mgr.14150 v1:172.21.15.92:0/996057664' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-11T21:21:57.863 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:21:57 smithi092 ceph-mon[29572]: from='client.? v1:172.21.15.97:0/2087245699' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-03-11T21:21:57.863 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:21:57 smithi092 ceph-mon[29572]: from='mgr.14150 v1:172.21.15.92:0/996057664' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-11T21:21:57.863 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:21:57 smithi092 ceph-mon[29572]: pgmap v16: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T21:21:57.863 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:21:57 smithi092 ceph-mon[29572]: from='mgr.14150 v1:172.21.15.92:0/996057664' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-11T21:21:57.863 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:21:57 smithi092 ceph-mon[29572]: mon.c calling monitor election 2024-03-11T21:21:57.863 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:21:57 smithi092 ceph-mon[29572]: from='mgr.14150 v1:172.21.15.92:0/996057664' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-11T21:21:57.863 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:21:57 smithi092 ceph-mon[29572]: pgmap v17: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T21:21:57.863 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:21:57 smithi092 ceph-mon[29572]: from='mgr.14150 v1:172.21.15.92:0/996057664' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-11T21:21:57.863 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:21:57 smithi092 ceph-mon[29572]: from='mgr.14150 v1:172.21.15.92:0/996057664' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-11T21:21:57.863 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:21:57 smithi092 ceph-mon[29572]: mon.a is new leader, mons a,b,c in quorum (ranks 0,1,2) 2024-03-11T21:21:57.863 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:21:57 smithi092 ceph-mon[29572]: monmap e3: 3 mons at {a=v1:172.21.15.92:6789/0,b=v1:172.21.15.97:6789/0,c=v1:172.21.15.92:6790/0} removed_ranks: {} disallowed_leaders: {} 2024-03-11T21:21:57.863 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:21:57 smithi092 ceph-mon[29572]: fsmap 2024-03-11T21:21:57.863 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:21:57 smithi092 ceph-mon[29572]: osdmap e4: 0 total, 0 up, 0 in 2024-03-11T21:21:57.864 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:21:57 smithi092 ceph-mon[29572]: mgrmap e13: y(active, since 47s) 2024-03-11T21:21:57.864 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:21:57 smithi092 ceph-mon[29572]: overall HEALTH_OK 2024-03-11T21:21:57.864 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:21:57 smithi092 ceph-mon[29572]: from='mgr.14150 v1:172.21.15.92:0/996057664' entity='mgr.y' 2024-03-11T21:21:57.864 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:21:57 smithi092 ceph-mon[29572]: from='mgr.14150 v1:172.21.15.92:0/996057664' entity='mgr.y' 2024-03-11T21:21:57.864 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:21:57 smithi092 ceph-mon[29572]: from='mgr.14150 v1:172.21.15.92:0/996057664' entity='mgr.y' 2024-03-11T21:21:57.864 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:21:57 smithi092 ceph-mon[29572]: from='mgr.14150 v1:172.21.15.92:0/996057664' entity='mgr.y' 2024-03-11T21:21:57.864 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:21:57 smithi092 ceph-mon[37563]: from='mgr.14150 v1:172.21.15.92:0/996057664' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-11T21:21:57.864 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:21:57 smithi092 ceph-mon[37563]: mon.a calling monitor election 2024-03-11T21:21:57.864 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:21:57 smithi092 ceph-mon[37563]: mon.b calling monitor election 2024-03-11T21:21:57.865 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:21:57 smithi092 ceph-mon[37563]: from='mgr.14150 v1:172.21.15.92:0/996057664' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-11T21:21:57.865 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:21:57 smithi092 ceph-mon[37563]: from='mgr.14150 v1:172.21.15.92:0/996057664' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-11T21:21:57.865 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:21:57 smithi092 ceph-mon[37563]: from='client.? v1:172.21.15.97:0/2087245699' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-03-11T21:21:57.865 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:21:57 smithi092 ceph-mon[37563]: from='mgr.14150 v1:172.21.15.92:0/996057664' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-11T21:21:57.865 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:21:57 smithi092 ceph-mon[37563]: pgmap v16: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T21:21:57.865 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:21:57 smithi092 ceph-mon[37563]: from='mgr.14150 v1:172.21.15.92:0/996057664' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-11T21:21:57.865 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:21:57 smithi092 ceph-mon[37563]: mon.c calling monitor election 2024-03-11T21:21:57.865 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:21:57 smithi092 ceph-mon[37563]: from='mgr.14150 v1:172.21.15.92:0/996057664' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-11T21:21:57.865 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:21:57 smithi092 ceph-mon[37563]: pgmap v17: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T21:21:57.865 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:21:57 smithi092 ceph-mon[37563]: from='mgr.14150 v1:172.21.15.92:0/996057664' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-11T21:21:57.865 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:21:57 smithi092 ceph-mon[37563]: from='mgr.14150 v1:172.21.15.92:0/996057664' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-11T21:21:57.865 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:21:57 smithi092 ceph-mon[37563]: mon.a is new leader, mons a,b,c in quorum (ranks 0,1,2) 2024-03-11T21:21:57.865 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:21:57 smithi092 ceph-mon[37563]: monmap e3: 3 mons at {a=v1:172.21.15.92:6789/0,b=v1:172.21.15.97:6789/0,c=v1:172.21.15.92:6790/0} removed_ranks: {} disallowed_leaders: {} 2024-03-11T21:21:57.865 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:21:57 smithi092 ceph-mon[37563]: fsmap 2024-03-11T21:21:57.865 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:21:57 smithi092 ceph-mon[37563]: osdmap e4: 0 total, 0 up, 0 in 2024-03-11T21:21:57.866 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:21:57 smithi092 ceph-mon[37563]: mgrmap e13: y(active, since 47s) 2024-03-11T21:21:57.866 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:21:57 smithi092 ceph-mon[37563]: overall HEALTH_OK 2024-03-11T21:21:57.866 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:21:57 smithi092 ceph-mon[37563]: from='mgr.14150 v1:172.21.15.92:0/996057664' entity='mgr.y' 2024-03-11T21:21:57.866 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:21:57 smithi092 ceph-mon[37563]: from='mgr.14150 v1:172.21.15.92:0/996057664' entity='mgr.y' 2024-03-11T21:21:57.866 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:21:57 smithi092 ceph-mon[37563]: from='mgr.14150 v1:172.21.15.92:0/996057664' entity='mgr.y' 2024-03-11T21:21:57.866 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:21:57 smithi092 ceph-mon[37563]: from='mgr.14150 v1:172.21.15.92:0/996057664' entity='mgr.y' 2024-03-11T21:21:57.937 INFO:tasks.cephadm:Generating final ceph.conf file... 2024-03-11T21:21:57.938 DEBUG:teuthology.orchestra.run.smithi092:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 00db0016-dfed-11ee-95c8-87774f69a715 -- ceph config generate-minimal-conf 2024-03-11T21:21:58.578 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:21:58 smithi097 ceph-mon[33317]: pgmap v18: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T21:21:58.578 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:21:58 smithi097 ceph-mon[33317]: from='mgr.14150 v1:172.21.15.92:0/996057664' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T21:21:58.578 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:21:58 smithi097 ceph-mon[33317]: from='mgr.14150 v1:172.21.15.92:0/996057664' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-11T21:21:58.612 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:21:58 smithi092 ceph-mon[29572]: pgmap v18: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T21:21:58.612 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:21:58 smithi092 ceph-mon[29572]: from='mgr.14150 v1:172.21.15.92:0/996057664' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T21:21:58.612 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:21:58 smithi092 ceph-mon[29572]: from='mgr.14150 v1:172.21.15.92:0/996057664' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-11T21:21:58.613 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:21:58 smithi092 ceph-mon[37563]: pgmap v18: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T21:21:58.613 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:21:58 smithi092 ceph-mon[37563]: from='mgr.14150 v1:172.21.15.92:0/996057664' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T21:21:58.613 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:21:58 smithi092 ceph-mon[37563]: from='mgr.14150 v1:172.21.15.92:0/996057664' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-11T21:21:58.829 INFO:teuthology.orchestra.run.smithi092.stderr:Inferring config /var/lib/ceph/00db0016-dfed-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T21:21:59.970 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:21:59 smithi092 ceph-mon[29572]: from='mgr.14150 v1:172.21.15.92:0/996057664' entity='mgr.y' 2024-03-11T21:21:59.970 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:21:59 smithi092 ceph-mon[29572]: from='mgr.14150 v1:172.21.15.92:0/996057664' entity='mgr.y' 2024-03-11T21:21:59.970 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:21:59 smithi092 ceph-mon[29572]: from='mgr.14150 v1:172.21.15.92:0/996057664' entity='mgr.y' 2024-03-11T21:21:59.970 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:21:59 smithi092 ceph-mon[29572]: from='mgr.14150 v1:172.21.15.92:0/996057664' entity='mgr.y' 2024-03-11T21:21:59.971 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:21:59 smithi092 ceph-mon[29572]: from='mgr.14150 v1:172.21.15.92:0/996057664' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T21:21:59.971 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:21:59 smithi092 ceph-mon[29572]: from='mgr.14150 v1:172.21.15.92:0/996057664' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T21:21:59.971 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:21:59 smithi092 ceph-mon[29572]: Updating smithi092:/etc/ceph/ceph.conf 2024-03-11T21:21:59.971 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:21:59 smithi092 ceph-mon[29572]: Updating smithi097:/etc/ceph/ceph.conf 2024-03-11T21:21:59.971 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:21:59 smithi092 ceph-mon[29572]: Updating smithi097:/var/lib/ceph/00db0016-dfed-11ee-95c8-87774f69a715/config/ceph.conf 2024-03-11T21:21:59.971 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:21:59 smithi092 ceph-mon[29572]: Updating smithi092:/var/lib/ceph/00db0016-dfed-11ee-95c8-87774f69a715/config/ceph.conf 2024-03-11T21:21:59.971 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:21:59 smithi092 ceph-mon[29572]: from='mgr.14150 v1:172.21.15.92:0/996057664' entity='mgr.y' 2024-03-11T21:21:59.971 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:21:59 smithi092 ceph-mon[29572]: from='mgr.14150 v1:172.21.15.92:0/996057664' entity='mgr.y' 2024-03-11T21:21:59.971 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:21:59 smithi092 ceph-mon[29572]: from='mgr.14150 v1:172.21.15.92:0/996057664' entity='mgr.y' 2024-03-11T21:21:59.971 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:21:59 smithi092 ceph-mon[29572]: from='mgr.14150 v1:172.21.15.92:0/996057664' entity='mgr.y' 2024-03-11T21:21:59.971 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:21:59 smithi092 ceph-mon[29572]: from='mgr.14150 v1:172.21.15.92:0/996057664' entity='mgr.y' 2024-03-11T21:21:59.972 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:21:59 smithi092 ceph-mon[29572]: from='mgr.14150 v1:172.21.15.92:0/996057664' entity='mgr.y' 2024-03-11T21:21:59.972 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:21:59 smithi092 ceph-mon[29572]: from='mgr.14150 v1:172.21.15.92:0/996057664' entity='mgr.y' 2024-03-11T21:21:59.972 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:21:59 smithi092 ceph-mon[29572]: from='mgr.14150 v1:172.21.15.92:0/996057664' entity='mgr.y' 2024-03-11T21:21:59.972 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:21:59 smithi092 ceph-mon[29572]: from='mgr.14150 v1:172.21.15.92:0/996057664' entity='mgr.y' 2024-03-11T21:21:59.972 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:21:59 smithi092 ceph-mon[29572]: Reconfiguring mon.a (unknown last config time)... 2024-03-11T21:21:59.972 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:21:59 smithi092 ceph-mon[29572]: from='mgr.14150 v1:172.21.15.92:0/996057664' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-11T21:21:59.972 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:21:59 smithi092 ceph-mon[29572]: from='mgr.14150 v1:172.21.15.92:0/996057664' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-03-11T21:21:59.972 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:21:59 smithi092 ceph-mon[29572]: from='mgr.14150 v1:172.21.15.92:0/996057664' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T21:21:59.972 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:21:59 smithi092 ceph-mon[29572]: Reconfiguring daemon mon.a on smithi092 2024-03-11T21:21:59.972 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:21:59 smithi092 ceph-mon[29572]: pgmap v19: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T21:21:59.972 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:21:59 smithi092 ceph-mon[29572]: from='mgr.14150 v1:172.21.15.92:0/996057664' entity='mgr.y' 2024-03-11T21:21:59.973 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:21:59 smithi092 ceph-mon[37563]: from='mgr.14150 v1:172.21.15.92:0/996057664' entity='mgr.y' 2024-03-11T21:21:59.973 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:21:59 smithi092 ceph-mon[37563]: from='mgr.14150 v1:172.21.15.92:0/996057664' entity='mgr.y' 2024-03-11T21:21:59.973 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:21:59 smithi092 ceph-mon[37563]: from='mgr.14150 v1:172.21.15.92:0/996057664' entity='mgr.y' 2024-03-11T21:21:59.973 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:21:59 smithi092 ceph-mon[37563]: from='mgr.14150 v1:172.21.15.92:0/996057664' entity='mgr.y' 2024-03-11T21:21:59.973 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:21:59 smithi092 ceph-mon[37563]: from='mgr.14150 v1:172.21.15.92:0/996057664' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T21:21:59.973 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:21:59 smithi092 ceph-mon[37563]: from='mgr.14150 v1:172.21.15.92:0/996057664' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T21:21:59.974 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:21:59 smithi092 ceph-mon[37563]: Updating smithi092:/etc/ceph/ceph.conf 2024-03-11T21:21:59.974 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:21:59 smithi092 ceph-mon[37563]: Updating smithi097:/etc/ceph/ceph.conf 2024-03-11T21:21:59.974 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:21:59 smithi092 ceph-mon[37563]: Updating smithi097:/var/lib/ceph/00db0016-dfed-11ee-95c8-87774f69a715/config/ceph.conf 2024-03-11T21:21:59.974 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:21:59 smithi092 ceph-mon[37563]: Updating smithi092:/var/lib/ceph/00db0016-dfed-11ee-95c8-87774f69a715/config/ceph.conf 2024-03-11T21:21:59.974 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:21:59 smithi092 ceph-mon[37563]: from='mgr.14150 v1:172.21.15.92:0/996057664' entity='mgr.y' 2024-03-11T21:21:59.974 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:21:59 smithi092 ceph-mon[37563]: from='mgr.14150 v1:172.21.15.92:0/996057664' entity='mgr.y' 2024-03-11T21:21:59.974 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:21:59 smithi092 ceph-mon[37563]: from='mgr.14150 v1:172.21.15.92:0/996057664' entity='mgr.y' 2024-03-11T21:21:59.974 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:21:59 smithi092 ceph-mon[37563]: from='mgr.14150 v1:172.21.15.92:0/996057664' entity='mgr.y' 2024-03-11T21:21:59.974 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:21:59 smithi092 ceph-mon[37563]: from='mgr.14150 v1:172.21.15.92:0/996057664' entity='mgr.y' 2024-03-11T21:21:59.974 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:21:59 smithi092 ceph-mon[37563]: from='mgr.14150 v1:172.21.15.92:0/996057664' entity='mgr.y' 2024-03-11T21:21:59.974 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:21:59 smithi092 ceph-mon[37563]: from='mgr.14150 v1:172.21.15.92:0/996057664' entity='mgr.y' 2024-03-11T21:21:59.974 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:21:59 smithi092 ceph-mon[37563]: from='mgr.14150 v1:172.21.15.92:0/996057664' entity='mgr.y' 2024-03-11T21:21:59.975 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:21:59 smithi092 ceph-mon[37563]: from='mgr.14150 v1:172.21.15.92:0/996057664' entity='mgr.y' 2024-03-11T21:21:59.975 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:21:59 smithi092 ceph-mon[37563]: Reconfiguring mon.a (unknown last config time)... 2024-03-11T21:21:59.975 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:21:59 smithi092 ceph-mon[37563]: from='mgr.14150 v1:172.21.15.92:0/996057664' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-11T21:21:59.975 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:21:59 smithi092 ceph-mon[37563]: from='mgr.14150 v1:172.21.15.92:0/996057664' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-03-11T21:21:59.975 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:21:59 smithi092 ceph-mon[37563]: from='mgr.14150 v1:172.21.15.92:0/996057664' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T21:21:59.975 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:21:59 smithi092 ceph-mon[37563]: Reconfiguring daemon mon.a on smithi092 2024-03-11T21:21:59.975 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:21:59 smithi092 ceph-mon[37563]: pgmap v19: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T21:21:59.975 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:21:59 smithi092 ceph-mon[37563]: from='mgr.14150 v1:172.21.15.92:0/996057664' entity='mgr.y' 2024-03-11T21:22:00.078 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:21:59 smithi097 ceph-mon[33317]: from='mgr.14150 v1:172.21.15.92:0/996057664' entity='mgr.y' 2024-03-11T21:22:00.078 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:21:59 smithi097 ceph-mon[33317]: from='mgr.14150 v1:172.21.15.92:0/996057664' entity='mgr.y' 2024-03-11T21:22:00.078 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:21:59 smithi097 ceph-mon[33317]: from='mgr.14150 v1:172.21.15.92:0/996057664' entity='mgr.y' 2024-03-11T21:22:00.078 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:21:59 smithi097 ceph-mon[33317]: from='mgr.14150 v1:172.21.15.92:0/996057664' entity='mgr.y' 2024-03-11T21:22:00.078 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:21:59 smithi097 ceph-mon[33317]: from='mgr.14150 v1:172.21.15.92:0/996057664' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T21:22:00.078 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:21:59 smithi097 ceph-mon[33317]: from='mgr.14150 v1:172.21.15.92:0/996057664' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T21:22:00.079 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:21:59 smithi097 ceph-mon[33317]: Updating smithi092:/etc/ceph/ceph.conf 2024-03-11T21:22:00.079 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:21:59 smithi097 ceph-mon[33317]: Updating smithi097:/etc/ceph/ceph.conf 2024-03-11T21:22:00.079 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:21:59 smithi097 ceph-mon[33317]: Updating smithi097:/var/lib/ceph/00db0016-dfed-11ee-95c8-87774f69a715/config/ceph.conf 2024-03-11T21:22:00.079 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:21:59 smithi097 ceph-mon[33317]: Updating smithi092:/var/lib/ceph/00db0016-dfed-11ee-95c8-87774f69a715/config/ceph.conf 2024-03-11T21:22:00.079 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:21:59 smithi097 ceph-mon[33317]: from='mgr.14150 v1:172.21.15.92:0/996057664' entity='mgr.y' 2024-03-11T21:22:00.079 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:21:59 smithi097 ceph-mon[33317]: from='mgr.14150 v1:172.21.15.92:0/996057664' entity='mgr.y' 2024-03-11T21:22:00.079 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:21:59 smithi097 ceph-mon[33317]: from='mgr.14150 v1:172.21.15.92:0/996057664' entity='mgr.y' 2024-03-11T21:22:00.079 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:21:59 smithi097 ceph-mon[33317]: from='mgr.14150 v1:172.21.15.92:0/996057664' entity='mgr.y' 2024-03-11T21:22:00.079 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:21:59 smithi097 ceph-mon[33317]: from='mgr.14150 v1:172.21.15.92:0/996057664' entity='mgr.y' 2024-03-11T21:22:00.079 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:21:59 smithi097 ceph-mon[33317]: from='mgr.14150 v1:172.21.15.92:0/996057664' entity='mgr.y' 2024-03-11T21:22:00.079 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:21:59 smithi097 ceph-mon[33317]: from='mgr.14150 v1:172.21.15.92:0/996057664' entity='mgr.y' 2024-03-11T21:22:00.079 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:21:59 smithi097 ceph-mon[33317]: from='mgr.14150 v1:172.21.15.92:0/996057664' entity='mgr.y' 2024-03-11T21:22:00.079 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:21:59 smithi097 ceph-mon[33317]: from='mgr.14150 v1:172.21.15.92:0/996057664' entity='mgr.y' 2024-03-11T21:22:00.080 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:21:59 smithi097 ceph-mon[33317]: Reconfiguring mon.a (unknown last config time)... 2024-03-11T21:22:00.080 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:21:59 smithi097 ceph-mon[33317]: from='mgr.14150 v1:172.21.15.92:0/996057664' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-11T21:22:00.080 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:21:59 smithi097 ceph-mon[33317]: from='mgr.14150 v1:172.21.15.92:0/996057664' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-03-11T21:22:00.080 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:21:59 smithi097 ceph-mon[33317]: from='mgr.14150 v1:172.21.15.92:0/996057664' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T21:22:00.080 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:21:59 smithi097 ceph-mon[33317]: Reconfiguring daemon mon.a on smithi092 2024-03-11T21:22:00.080 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:21:59 smithi097 ceph-mon[33317]: pgmap v19: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T21:22:00.080 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:21:59 smithi097 ceph-mon[33317]: from='mgr.14150 v1:172.21.15.92:0/996057664' entity='mgr.y' 2024-03-11T21:22:01.153 INFO:teuthology.orchestra.run.smithi092.stdout:# minimal ceph.conf for 00db0016-dfed-11ee-95c8-87774f69a715 2024-03-11T21:22:01.153 INFO:teuthology.orchestra.run.smithi092.stdout:[global] 2024-03-11T21:22:01.153 INFO:teuthology.orchestra.run.smithi092.stdout: fsid = 00db0016-dfed-11ee-95c8-87774f69a715 2024-03-11T21:22:01.153 INFO:teuthology.orchestra.run.smithi092.stdout: mon_host = 172.21.15.92:6789/0 172.21.15.97:6789/0 v1:172.21.15.92:6790/0 2024-03-11T21:22:01.872 INFO:tasks.cephadm:Distributing (final) config and client.admin keyring... 2024-03-11T21:22:01.872 DEBUG:teuthology.orchestra.run.smithi092:> set -ex 2024-03-11T21:22:01.872 DEBUG:teuthology.orchestra.run.smithi092:> sudo dd of=/etc/ceph/ceph.conf 2024-03-11T21:22:01.903 DEBUG:teuthology.orchestra.run.smithi092:> set -ex 2024-03-11T21:22:01.903 DEBUG:teuthology.orchestra.run.smithi092:> sudo dd of=/etc/ceph/ceph.client.admin.keyring 2024-03-11T21:22:01.972 DEBUG:teuthology.orchestra.run.smithi097:> set -ex 2024-03-11T21:22:01.972 DEBUG:teuthology.orchestra.run.smithi097:> sudo dd of=/etc/ceph/ceph.conf 2024-03-11T21:22:02.007 DEBUG:teuthology.orchestra.run.smithi097:> set -ex 2024-03-11T21:22:02.008 DEBUG:teuthology.orchestra.run.smithi097:> sudo dd of=/etc/ceph/ceph.client.admin.keyring 2024-03-11T21:22:02.078 INFO:tasks.cephadm:Adding mgr.y on smithi092 2024-03-11T21:22:02.078 INFO:tasks.cephadm:Adding mgr.x on smithi097 2024-03-11T21:22:02.078 DEBUG:teuthology.orchestra.run.smithi097:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 00db0016-dfed-11ee-95c8-87774f69a715 -- ceph orch apply mgr '2;smithi092=y;smithi097=x' 2024-03-11T21:22:02.217 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:22:01 smithi092 ceph-mon[29572]: from='client.? v1:172.21.15.92:0/3777807278' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T21:22:02.217 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:22:01 smithi092 ceph-mon[29572]: from='mgr.14150 v1:172.21.15.92:0/996057664' entity='mgr.y' 2024-03-11T21:22:02.217 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:22:01 smithi092 ceph-mon[29572]: from='mgr.14150 v1:172.21.15.92:0/996057664' entity='mgr.y' 2024-03-11T21:22:02.217 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:22:01 smithi092 ceph-mon[29572]: Reconfiguring mon.c (monmap changed)... 2024-03-11T21:22:02.218 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:22:01 smithi092 ceph-mon[29572]: from='mgr.14150 v1:172.21.15.92:0/996057664' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-11T21:22:02.218 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:22:01 smithi092 ceph-mon[29572]: from='mgr.14150 v1:172.21.15.92:0/996057664' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-03-11T21:22:02.218 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:22:01 smithi092 ceph-mon[29572]: from='mgr.14150 v1:172.21.15.92:0/996057664' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T21:22:02.218 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:22:01 smithi092 ceph-mon[29572]: Reconfiguring daemon mon.c on smithi092 2024-03-11T21:22:02.218 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:22:01 smithi092 ceph-mon[29572]: pgmap v20: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T21:22:02.218 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:22:01 smithi092 ceph-mon[37563]: from='client.? v1:172.21.15.92:0/3777807278' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T21:22:02.218 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:22:01 smithi092 ceph-mon[37563]: from='mgr.14150 v1:172.21.15.92:0/996057664' entity='mgr.y' 2024-03-11T21:22:02.218 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:22:01 smithi092 ceph-mon[37563]: from='mgr.14150 v1:172.21.15.92:0/996057664' entity='mgr.y' 2024-03-11T21:22:02.218 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:22:01 smithi092 ceph-mon[37563]: Reconfiguring mon.c (monmap changed)... 2024-03-11T21:22:02.218 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:22:01 smithi092 ceph-mon[37563]: from='mgr.14150 v1:172.21.15.92:0/996057664' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-11T21:22:02.219 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:22:01 smithi092 ceph-mon[37563]: from='mgr.14150 v1:172.21.15.92:0/996057664' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-03-11T21:22:02.219 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:22:01 smithi092 ceph-mon[37563]: from='mgr.14150 v1:172.21.15.92:0/996057664' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T21:22:02.219 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:22:01 smithi092 ceph-mon[37563]: Reconfiguring daemon mon.c on smithi092 2024-03-11T21:22:02.219 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:22:01 smithi092 ceph-mon[37563]: pgmap v20: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T21:22:02.328 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:22:01 smithi097 ceph-mon[33317]: from='client.? v1:172.21.15.92:0/3777807278' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T21:22:02.328 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:22:01 smithi097 ceph-mon[33317]: from='mgr.14150 v1:172.21.15.92:0/996057664' entity='mgr.y' 2024-03-11T21:22:02.328 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:22:01 smithi097 ceph-mon[33317]: from='mgr.14150 v1:172.21.15.92:0/996057664' entity='mgr.y' 2024-03-11T21:22:02.328 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:22:01 smithi097 ceph-mon[33317]: Reconfiguring mon.c (monmap changed)... 2024-03-11T21:22:02.328 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:22:01 smithi097 ceph-mon[33317]: from='mgr.14150 v1:172.21.15.92:0/996057664' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-11T21:22:02.328 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:22:01 smithi097 ceph-mon[33317]: from='mgr.14150 v1:172.21.15.92:0/996057664' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-03-11T21:22:02.328 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:22:01 smithi097 ceph-mon[33317]: from='mgr.14150 v1:172.21.15.92:0/996057664' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T21:22:02.328 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:22:01 smithi097 ceph-mon[33317]: Reconfiguring daemon mon.c on smithi092 2024-03-11T21:22:02.329 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:22:01 smithi097 ceph-mon[33317]: pgmap v20: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T21:22:02.353 INFO:teuthology.orchestra.run.smithi097.stderr:Inferring config /var/lib/ceph/00db0016-dfed-11ee-95c8-87774f69a715/mon.b/config 2024-03-11T21:22:04.210 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:22:03 smithi097 ceph-mon[33317]: from='mgr.14150 v1:172.21.15.92:0/996057664' entity='mgr.y' 2024-03-11T21:22:04.210 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:22:03 smithi097 ceph-mon[33317]: from='mgr.14150 v1:172.21.15.92:0/996057664' entity='mgr.y' 2024-03-11T21:22:04.210 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:22:03 smithi097 ceph-mon[33317]: Reconfiguring mon.b (monmap changed)... 2024-03-11T21:22:04.210 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:22:03 smithi097 ceph-mon[33317]: from='mgr.14150 v1:172.21.15.92:0/996057664' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-11T21:22:04.210 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:22:03 smithi097 ceph-mon[33317]: from='mgr.14150 v1:172.21.15.92:0/996057664' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-03-11T21:22:04.211 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:22:03 smithi097 ceph-mon[33317]: from='mgr.14150 v1:172.21.15.92:0/996057664' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T21:22:04.211 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:22:03 smithi097 ceph-mon[33317]: Reconfiguring daemon mon.b on smithi097 2024-03-11T21:22:04.211 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:22:03 smithi097 ceph-mon[33317]: pgmap v21: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T21:22:04.362 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:22:03 smithi092 ceph-mon[29572]: from='mgr.14150 v1:172.21.15.92:0/996057664' entity='mgr.y' 2024-03-11T21:22:04.363 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:22:03 smithi092 ceph-mon[29572]: from='mgr.14150 v1:172.21.15.92:0/996057664' entity='mgr.y' 2024-03-11T21:22:04.363 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:22:03 smithi092 ceph-mon[29572]: Reconfiguring mon.b (monmap changed)... 2024-03-11T21:22:04.363 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:22:03 smithi092 ceph-mon[29572]: from='mgr.14150 v1:172.21.15.92:0/996057664' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-11T21:22:04.363 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:22:03 smithi092 ceph-mon[29572]: from='mgr.14150 v1:172.21.15.92:0/996057664' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-03-11T21:22:04.363 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:22:03 smithi092 ceph-mon[29572]: from='mgr.14150 v1:172.21.15.92:0/996057664' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T21:22:04.363 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:22:03 smithi092 ceph-mon[29572]: Reconfiguring daemon mon.b on smithi097 2024-03-11T21:22:04.363 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:22:03 smithi092 ceph-mon[29572]: pgmap v21: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T21:22:04.363 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:22:03 smithi092 ceph-mon[37563]: from='mgr.14150 v1:172.21.15.92:0/996057664' entity='mgr.y' 2024-03-11T21:22:04.363 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:22:03 smithi092 ceph-mon[37563]: from='mgr.14150 v1:172.21.15.92:0/996057664' entity='mgr.y' 2024-03-11T21:22:04.363 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:22:03 smithi092 ceph-mon[37563]: Reconfiguring mon.b (monmap changed)... 2024-03-11T21:22:04.363 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:22:03 smithi092 ceph-mon[37563]: from='mgr.14150 v1:172.21.15.92:0/996057664' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-11T21:22:04.363 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:22:03 smithi092 ceph-mon[37563]: from='mgr.14150 v1:172.21.15.92:0/996057664' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-03-11T21:22:04.364 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:22:03 smithi092 ceph-mon[37563]: from='mgr.14150 v1:172.21.15.92:0/996057664' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T21:22:04.364 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:22:03 smithi092 ceph-mon[37563]: Reconfiguring daemon mon.b on smithi097 2024-03-11T21:22:04.364 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:22:03 smithi092 ceph-mon[37563]: pgmap v21: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T21:22:04.539 INFO:teuthology.orchestra.run.smithi097.stdout:Scheduled mgr update... 2024-03-11T21:22:05.377 DEBUG:teuthology.orchestra.run.smithi097:mgr.x> sudo journalctl -f -n 0 -u ceph-00db0016-dfed-11ee-95c8-87774f69a715@mgr.x.service 2024-03-11T21:22:05.379 INFO:tasks.cephadm:Deploying OSDs... 2024-03-11T21:22:05.379 DEBUG:teuthology.orchestra.run.smithi092:> set -ex 2024-03-11T21:22:05.379 DEBUG:teuthology.orchestra.run.smithi092:> dd if=/scratch_devs of=/dev/stdout 2024-03-11T21:22:05.398 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-03-11T21:22:05.398 DEBUG:teuthology.orchestra.run.smithi092:> stat /dev/vg_nvme/lv_1 2024-03-11T21:22:05.456 INFO:teuthology.orchestra.run.smithi092.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-03-11T21:22:05.456 INFO:teuthology.orchestra.run.smithi092.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-11T21:22:05.456 INFO:teuthology.orchestra.run.smithi092.stdout:Device: 5h/5d Inode: 668 Links: 1 2024-03-11T21:22:05.456 INFO:teuthology.orchestra.run.smithi092.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-11T21:22:05.456 INFO:teuthology.orchestra.run.smithi092.stdout:Context: system_u:object_r:device_t:s0 2024-03-11T21:22:05.456 INFO:teuthology.orchestra.run.smithi092.stdout:Access: 2024-03-11 21:21:51.770137276 +0000 2024-03-11T21:22:05.457 INFO:teuthology.orchestra.run.smithi092.stdout:Modify: 2024-03-11 21:13:45.785303231 +0000 2024-03-11T21:22:05.457 INFO:teuthology.orchestra.run.smithi092.stdout:Change: 2024-03-11 21:13:45.785303231 +0000 2024-03-11T21:22:05.457 INFO:teuthology.orchestra.run.smithi092.stdout: Birth: 2024-03-11 21:13:45.785303231 +0000 2024-03-11T21:22:05.457 DEBUG:teuthology.orchestra.run.smithi092:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-03-11T21:22:05.527 INFO:teuthology.orchestra.run.smithi092.stderr:1+0 records in 2024-03-11T21:22:05.528 INFO:teuthology.orchestra.run.smithi092.stderr:1+0 records out 2024-03-11T21:22:05.528 INFO:teuthology.orchestra.run.smithi092.stderr:512 bytes copied, 0.000238163 s, 2.1 MB/s 2024-03-11T21:22:05.529 DEBUG:teuthology.orchestra.run.smithi092:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-03-11T21:22:05.588 DEBUG:teuthology.orchestra.run.smithi092:> stat /dev/vg_nvme/lv_2 2024-03-11T21:22:05.646 INFO:teuthology.orchestra.run.smithi092.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-03-11T21:22:05.647 INFO:teuthology.orchestra.run.smithi092.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-11T21:22:05.647 INFO:teuthology.orchestra.run.smithi092.stdout:Device: 5h/5d Inode: 687 Links: 1 2024-03-11T21:22:05.647 INFO:teuthology.orchestra.run.smithi092.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-11T21:22:05.647 INFO:teuthology.orchestra.run.smithi092.stdout:Context: system_u:object_r:device_t:s0 2024-03-11T21:22:05.647 INFO:teuthology.orchestra.run.smithi092.stdout:Access: 2024-03-11 21:21:51.771137247 +0000 2024-03-11T21:22:05.647 INFO:teuthology.orchestra.run.smithi092.stdout:Modify: 2024-03-11 21:13:46.129293403 +0000 2024-03-11T21:22:05.647 INFO:teuthology.orchestra.run.smithi092.stdout:Change: 2024-03-11 21:13:46.129293403 +0000 2024-03-11T21:22:05.647 INFO:teuthology.orchestra.run.smithi092.stdout: Birth: 2024-03-11 21:13:46.129293403 +0000 2024-03-11T21:22:05.647 DEBUG:teuthology.orchestra.run.smithi092:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-03-11T21:22:05.656 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:22:05 smithi097 ceph-mon[33317]: from='client.24106 v1:172.21.15.97:0/1748244579' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mgr", "placement": "2;smithi092=y;smithi097=x", "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T21:22:05.656 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:22:05 smithi097 ceph-mon[33317]: Saving service mgr spec with placement smithi092=y;smithi097=x;count:2 2024-03-11T21:22:05.656 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:22:05 smithi097 ceph-mon[33317]: from='mgr.14150 v1:172.21.15.92:0/996057664' entity='mgr.y' 2024-03-11T21:22:05.656 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:22:05 smithi097 ceph-mon[33317]: from='mgr.14150 v1:172.21.15.92:0/996057664' entity='mgr.y' 2024-03-11T21:22:05.656 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:22:05 smithi097 ceph-mon[33317]: from='mgr.14150 v1:172.21.15.92:0/996057664' entity='mgr.y' 2024-03-11T21:22:05.656 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:22:05 smithi097 ceph-mon[33317]: from='mgr.14150 v1:172.21.15.92:0/996057664' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T21:22:05.656 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:22:05 smithi097 ceph-mon[33317]: from='mgr.14150 v1:172.21.15.92:0/996057664' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T21:22:05.656 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:22:05 smithi097 ceph-mon[33317]: from='mgr.14150 v1:172.21.15.92:0/996057664' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T21:22:05.656 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:22:05 smithi097 ceph-mon[33317]: from='mgr.14150 v1:172.21.15.92:0/996057664' entity='mgr.y' 2024-03-11T21:22:05.656 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:22:05 smithi097 ceph-mon[33317]: from='mgr.14150 v1:172.21.15.92:0/996057664' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-03-11T21:22:05.656 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:22:05 smithi097 ceph-mon[33317]: from='mgr.14150 v1:172.21.15.92:0/996057664' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2024-03-11T21:22:05.656 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:22:05 smithi097 ceph-mon[33317]: from='mgr.14150 v1:172.21.15.92:0/996057664' entity='mgr.y' cmd=[{"prefix": "mgr services"}]: dispatch 2024-03-11T21:22:05.657 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:22:05 smithi097 ceph-mon[33317]: from='mgr.14150 v1:172.21.15.92:0/996057664' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T21:22:05.716 INFO:teuthology.orchestra.run.smithi092.stderr:1+0 records in 2024-03-11T21:22:05.716 INFO:teuthology.orchestra.run.smithi092.stderr:1+0 records out 2024-03-11T21:22:05.716 INFO:teuthology.orchestra.run.smithi092.stderr:512 bytes copied, 0.000261888 s, 2.0 MB/s 2024-03-11T21:22:05.717 DEBUG:teuthology.orchestra.run.smithi092:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-03-11T21:22:05.775 DEBUG:teuthology.orchestra.run.smithi092:> stat /dev/vg_nvme/lv_3 2024-03-11T21:22:05.832 INFO:teuthology.orchestra.run.smithi092.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-03-11T21:22:05.832 INFO:teuthology.orchestra.run.smithi092.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-11T21:22:05.832 INFO:teuthology.orchestra.run.smithi092.stdout:Device: 5h/5d Inode: 703 Links: 1 2024-03-11T21:22:05.832 INFO:teuthology.orchestra.run.smithi092.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-11T21:22:05.832 INFO:teuthology.orchestra.run.smithi092.stdout:Context: system_u:object_r:device_t:s0 2024-03-11T21:22:05.832 INFO:teuthology.orchestra.run.smithi092.stdout:Access: 2024-03-11 21:21:51.771137247 +0000 2024-03-11T21:22:05.833 INFO:teuthology.orchestra.run.smithi092.stdout:Modify: 2024-03-11 21:13:46.470283659 +0000 2024-03-11T21:22:05.833 INFO:teuthology.orchestra.run.smithi092.stdout:Change: 2024-03-11 21:13:46.470283659 +0000 2024-03-11T21:22:05.833 INFO:teuthology.orchestra.run.smithi092.stdout: Birth: 2024-03-11 21:13:46.470283659 +0000 2024-03-11T21:22:05.833 DEBUG:teuthology.orchestra.run.smithi092:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-03-11T21:22:05.862 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:22:05 smithi092 ceph-mon[29572]: from='client.24106 v1:172.21.15.97:0/1748244579' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mgr", "placement": "2;smithi092=y;smithi097=x", "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T21:22:05.863 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:22:05 smithi092 ceph-mon[29572]: Saving service mgr spec with placement smithi092=y;smithi097=x;count:2 2024-03-11T21:22:05.863 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:22:05 smithi092 ceph-mon[29572]: from='mgr.14150 v1:172.21.15.92:0/996057664' entity='mgr.y' 2024-03-11T21:22:05.863 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:22:05 smithi092 ceph-mon[29572]: from='mgr.14150 v1:172.21.15.92:0/996057664' entity='mgr.y' 2024-03-11T21:22:05.863 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:22:05 smithi092 ceph-mon[29572]: from='mgr.14150 v1:172.21.15.92:0/996057664' entity='mgr.y' 2024-03-11T21:22:05.863 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:22:05 smithi092 ceph-mon[29572]: from='mgr.14150 v1:172.21.15.92:0/996057664' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T21:22:05.863 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:22:05 smithi092 ceph-mon[29572]: from='mgr.14150 v1:172.21.15.92:0/996057664' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T21:22:05.863 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:22:05 smithi092 ceph-mon[29572]: from='mgr.14150 v1:172.21.15.92:0/996057664' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T21:22:05.863 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:22:05 smithi092 ceph-mon[29572]: from='mgr.14150 v1:172.21.15.92:0/996057664' entity='mgr.y' 2024-03-11T21:22:05.863 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:22:05 smithi092 ceph-mon[29572]: from='mgr.14150 v1:172.21.15.92:0/996057664' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-03-11T21:22:05.863 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:22:05 smithi092 ceph-mon[29572]: from='mgr.14150 v1:172.21.15.92:0/996057664' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2024-03-11T21:22:05.863 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:22:05 smithi092 ceph-mon[29572]: from='mgr.14150 v1:172.21.15.92:0/996057664' entity='mgr.y' cmd=[{"prefix": "mgr services"}]: dispatch 2024-03-11T21:22:05.863 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:22:05 smithi092 ceph-mon[29572]: from='mgr.14150 v1:172.21.15.92:0/996057664' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T21:22:05.864 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:22:05 smithi092 ceph-mon[37563]: from='client.24106 v1:172.21.15.97:0/1748244579' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mgr", "placement": "2;smithi092=y;smithi097=x", "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T21:22:05.864 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:22:05 smithi092 ceph-mon[37563]: Saving service mgr spec with placement smithi092=y;smithi097=x;count:2 2024-03-11T21:22:05.864 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:22:05 smithi092 ceph-mon[37563]: from='mgr.14150 v1:172.21.15.92:0/996057664' entity='mgr.y' 2024-03-11T21:22:05.864 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:22:05 smithi092 ceph-mon[37563]: from='mgr.14150 v1:172.21.15.92:0/996057664' entity='mgr.y' 2024-03-11T21:22:05.864 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:22:05 smithi092 ceph-mon[37563]: from='mgr.14150 v1:172.21.15.92:0/996057664' entity='mgr.y' 2024-03-11T21:22:05.864 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:22:05 smithi092 ceph-mon[37563]: from='mgr.14150 v1:172.21.15.92:0/996057664' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T21:22:05.864 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:22:05 smithi092 ceph-mon[37563]: from='mgr.14150 v1:172.21.15.92:0/996057664' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T21:22:05.864 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:22:05 smithi092 ceph-mon[37563]: from='mgr.14150 v1:172.21.15.92:0/996057664' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T21:22:05.864 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:22:05 smithi092 ceph-mon[37563]: from='mgr.14150 v1:172.21.15.92:0/996057664' entity='mgr.y' 2024-03-11T21:22:05.865 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:22:05 smithi092 ceph-mon[37563]: from='mgr.14150 v1:172.21.15.92:0/996057664' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-03-11T21:22:05.865 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:22:05 smithi092 ceph-mon[37563]: from='mgr.14150 v1:172.21.15.92:0/996057664' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2024-03-11T21:22:05.865 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:22:05 smithi092 ceph-mon[37563]: from='mgr.14150 v1:172.21.15.92:0/996057664' entity='mgr.y' cmd=[{"prefix": "mgr services"}]: dispatch 2024-03-11T21:22:05.865 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:22:05 smithi092 ceph-mon[37563]: from='mgr.14150 v1:172.21.15.92:0/996057664' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T21:22:05.892 INFO:teuthology.orchestra.run.smithi092.stderr:1+0 records in 2024-03-11T21:22:05.892 INFO:teuthology.orchestra.run.smithi092.stderr:1+0 records out 2024-03-11T21:22:05.892 INFO:teuthology.orchestra.run.smithi092.stderr:512 bytes copied, 0.000204194 s, 2.5 MB/s 2024-03-11T21:22:05.893 DEBUG:teuthology.orchestra.run.smithi092:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-03-11T21:22:05.950 DEBUG:teuthology.orchestra.run.smithi092:> stat /dev/vg_nvme/lv_4 2024-03-11T21:22:06.006 INFO:teuthology.orchestra.run.smithi092.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-03-11T21:22:06.006 INFO:teuthology.orchestra.run.smithi092.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-11T21:22:06.006 INFO:teuthology.orchestra.run.smithi092.stdout:Device: 5h/5d Inode: 722 Links: 1 2024-03-11T21:22:06.006 INFO:teuthology.orchestra.run.smithi092.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-11T21:22:06.006 INFO:teuthology.orchestra.run.smithi092.stdout:Context: system_u:object_r:device_t:s0 2024-03-11T21:22:06.006 INFO:teuthology.orchestra.run.smithi092.stdout:Access: 2024-03-11 21:21:51.771137247 +0000 2024-03-11T21:22:06.007 INFO:teuthology.orchestra.run.smithi092.stdout:Modify: 2024-03-11 21:13:46.816273773 +0000 2024-03-11T21:22:06.007 INFO:teuthology.orchestra.run.smithi092.stdout:Change: 2024-03-11 21:13:46.816273773 +0000 2024-03-11T21:22:06.007 INFO:teuthology.orchestra.run.smithi092.stdout: Birth: 2024-03-11 21:13:46.816273773 +0000 2024-03-11T21:22:06.007 DEBUG:teuthology.orchestra.run.smithi092:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-03-11T21:22:06.074 INFO:teuthology.orchestra.run.smithi092.stderr:1+0 records in 2024-03-11T21:22:06.074 INFO:teuthology.orchestra.run.smithi092.stderr:1+0 records out 2024-03-11T21:22:06.074 INFO:teuthology.orchestra.run.smithi092.stderr:512 bytes copied, 0.000216485 s, 2.4 MB/s 2024-03-11T21:22:06.075 DEBUG:teuthology.orchestra.run.smithi092:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-03-11T21:22:06.132 DEBUG:teuthology.orchestra.run.smithi097:> set -ex 2024-03-11T21:22:06.132 DEBUG:teuthology.orchestra.run.smithi097:> dd if=/scratch_devs of=/dev/stdout 2024-03-11T21:22:06.149 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-03-11T21:22:06.149 DEBUG:teuthology.orchestra.run.smithi097:> stat /dev/vg_nvme/lv_1 2024-03-11T21:22:06.205 INFO:teuthology.orchestra.run.smithi097.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-03-11T21:22:06.206 INFO:teuthology.orchestra.run.smithi097.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-11T21:22:06.206 INFO:teuthology.orchestra.run.smithi097.stdout:Device: 5h/5d Inode: 667 Links: 1 2024-03-11T21:22:06.206 INFO:teuthology.orchestra.run.smithi097.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-11T21:22:06.206 INFO:teuthology.orchestra.run.smithi097.stdout:Context: system_u:object_r:device_t:s0 2024-03-11T21:22:06.206 INFO:teuthology.orchestra.run.smithi097.stdout:Access: 2024-03-11 21:21:42.603731740 +0000 2024-03-11T21:22:06.206 INFO:teuthology.orchestra.run.smithi097.stdout:Modify: 2024-03-11 21:13:47.005424929 +0000 2024-03-11T21:22:06.206 INFO:teuthology.orchestra.run.smithi097.stdout:Change: 2024-03-11 21:13:47.005424929 +0000 2024-03-11T21:22:06.206 INFO:teuthology.orchestra.run.smithi097.stdout: Birth: 2024-03-11 21:13:47.005424929 +0000 2024-03-11T21:22:06.206 DEBUG:teuthology.orchestra.run.smithi097:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-03-11T21:22:06.275 INFO:teuthology.orchestra.run.smithi097.stderr:1+0 records in 2024-03-11T21:22:06.275 INFO:teuthology.orchestra.run.smithi097.stderr:1+0 records out 2024-03-11T21:22:06.275 INFO:teuthology.orchestra.run.smithi097.stderr:512 bytes copied, 0.000277966 s, 1.8 MB/s 2024-03-11T21:22:06.276 DEBUG:teuthology.orchestra.run.smithi097:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-03-11T21:22:06.333 DEBUG:teuthology.orchestra.run.smithi097:> stat /dev/vg_nvme/lv_2 2024-03-11T21:22:06.389 INFO:teuthology.orchestra.run.smithi097.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-03-11T21:22:06.389 INFO:teuthology.orchestra.run.smithi097.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-11T21:22:06.389 INFO:teuthology.orchestra.run.smithi097.stdout:Device: 5h/5d Inode: 684 Links: 1 2024-03-11T21:22:06.389 INFO:teuthology.orchestra.run.smithi097.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-11T21:22:06.389 INFO:teuthology.orchestra.run.smithi097.stdout:Context: system_u:object_r:device_t:s0 2024-03-11T21:22:06.389 INFO:teuthology.orchestra.run.smithi097.stdout:Access: 2024-03-11 21:21:42.603731740 +0000 2024-03-11T21:22:06.389 INFO:teuthology.orchestra.run.smithi097.stdout:Modify: 2024-03-11 21:13:47.341419482 +0000 2024-03-11T21:22:06.389 INFO:teuthology.orchestra.run.smithi097.stdout:Change: 2024-03-11 21:13:47.341419482 +0000 2024-03-11T21:22:06.389 INFO:teuthology.orchestra.run.smithi097.stdout: Birth: 2024-03-11 21:13:47.341419482 +0000 2024-03-11T21:22:06.390 DEBUG:teuthology.orchestra.run.smithi097:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-03-11T21:22:06.456 INFO:teuthology.orchestra.run.smithi097.stderr:1+0 records in 2024-03-11T21:22:06.456 INFO:teuthology.orchestra.run.smithi097.stderr:1+0 records out 2024-03-11T21:22:06.456 INFO:teuthology.orchestra.run.smithi097.stderr:512 bytes copied, 0.000217703 s, 2.4 MB/s 2024-03-11T21:22:06.457 DEBUG:teuthology.orchestra.run.smithi097:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-03-11T21:22:06.513 DEBUG:teuthology.orchestra.run.smithi097:> stat /dev/vg_nvme/lv_3 2024-03-11T21:22:06.569 INFO:teuthology.orchestra.run.smithi097.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-03-11T21:22:06.569 INFO:teuthology.orchestra.run.smithi097.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-11T21:22:06.569 INFO:teuthology.orchestra.run.smithi097.stdout:Device: 5h/5d Inode: 703 Links: 1 2024-03-11T21:22:06.569 INFO:teuthology.orchestra.run.smithi097.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-11T21:22:06.569 INFO:teuthology.orchestra.run.smithi097.stdout:Context: system_u:object_r:device_t:s0 2024-03-11T21:22:06.569 INFO:teuthology.orchestra.run.smithi097.stdout:Access: 2024-03-11 21:21:42.604731724 +0000 2024-03-11T21:22:06.569 INFO:teuthology.orchestra.run.smithi097.stdout:Modify: 2024-03-11 21:13:47.677414035 +0000 2024-03-11T21:22:06.569 INFO:teuthology.orchestra.run.smithi097.stdout:Change: 2024-03-11 21:13:47.677414035 +0000 2024-03-11T21:22:06.570 INFO:teuthology.orchestra.run.smithi097.stdout: Birth: 2024-03-11 21:13:47.677414035 +0000 2024-03-11T21:22:06.570 DEBUG:teuthology.orchestra.run.smithi097:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-03-11T21:22:06.731 INFO:teuthology.orchestra.run.smithi097.stderr:1+0 records in 2024-03-11T21:22:06.731 INFO:teuthology.orchestra.run.smithi097.stderr:1+0 records out 2024-03-11T21:22:06.731 INFO:teuthology.orchestra.run.smithi097.stderr:512 bytes copied, 0.000219102 s, 2.3 MB/s 2024-03-11T21:22:06.732 DEBUG:teuthology.orchestra.run.smithi097:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-03-11T21:22:06.747 DEBUG:teuthology.orchestra.run.smithi097:> stat /dev/vg_nvme/lv_4 2024-03-11T21:22:06.807 INFO:teuthology.orchestra.run.smithi097.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-03-11T21:22:06.807 INFO:teuthology.orchestra.run.smithi097.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-11T21:22:06.807 INFO:teuthology.orchestra.run.smithi097.stdout:Device: 5h/5d Inode: 722 Links: 1 2024-03-11T21:22:06.807 INFO:teuthology.orchestra.run.smithi097.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-11T21:22:06.807 INFO:teuthology.orchestra.run.smithi097.stdout:Context: system_u:object_r:device_t:s0 2024-03-11T21:22:06.807 INFO:teuthology.orchestra.run.smithi097.stdout:Access: 2024-03-11 21:22:06.703346565 +0000 2024-03-11T21:22:06.807 INFO:teuthology.orchestra.run.smithi097.stdout:Modify: 2024-03-11 21:13:48.011408621 +0000 2024-03-11T21:22:06.807 INFO:teuthology.orchestra.run.smithi097.stdout:Change: 2024-03-11 21:13:48.011408621 +0000 2024-03-11T21:22:06.807 INFO:teuthology.orchestra.run.smithi097.stdout: Birth: 2024-03-11 21:13:48.011408621 +0000 2024-03-11T21:22:06.808 DEBUG:teuthology.orchestra.run.smithi097:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-03-11T21:22:06.828 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:22:06 smithi097 ceph-mon[33317]: Deploying daemon mgr.x on smithi097 2024-03-11T21:22:06.828 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:22:06 smithi097 ceph-mon[33317]: pgmap v22: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T21:22:06.862 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:22:06 smithi092 ceph-mon[29572]: Deploying daemon mgr.x on smithi097 2024-03-11T21:22:06.862 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:22:06 smithi092 ceph-mon[29572]: pgmap v22: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T21:22:06.862 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:22:06 smithi092 ceph-mon[37563]: Deploying daemon mgr.x on smithi097 2024-03-11T21:22:06.862 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:22:06 smithi092 ceph-mon[37563]: pgmap v22: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T21:22:06.907 INFO:teuthology.orchestra.run.smithi097.stderr:1+0 records in 2024-03-11T21:22:06.908 INFO:teuthology.orchestra.run.smithi097.stderr:1+0 records out 2024-03-11T21:22:06.908 INFO:teuthology.orchestra.run.smithi097.stderr:512 bytes copied, 0.000194648 s, 2.6 MB/s 2024-03-11T21:22:06.909 DEBUG:teuthology.orchestra.run.smithi097:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-03-11T21:22:06.924 INFO:tasks.cephadm:Deploying osd.0 on smithi092 with /dev/vg_nvme/lv_4... 2024-03-11T21:22:06.924 DEBUG:teuthology.orchestra.run.smithi092:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 00db0016-dfed-11ee-95c8-87774f69a715 -- ceph-volume lvm zap /dev/vg_nvme/lv_4 2024-03-11T21:22:07.143 INFO:journalctl@ceph.mgr.x.smithi097.stdout:Mar 11 21:22:06 smithi097 systemd[1]: Starting Ceph mgr.x for 00db0016-dfed-11ee-95c8-87774f69a715... 2024-03-11T21:22:07.168 INFO:teuthology.orchestra.run.smithi092.stderr:Inferring config /var/lib/ceph/00db0016-dfed-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T21:22:07.401 INFO:journalctl@ceph.mgr.x.smithi097.stdout:Mar 11 21:22:07 smithi097 podman[35154]: 2024-03-11 21:22:07.061958982 +0000 UTC m=+0.017165863 image pull 4133f8bc41bb201212a28168427a2edb1f6f3795e694bf61031ea9e2f3ff5e87 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:8fe9635f3beb0247496d7e321f3068bd10c333c433ef4824d7e1c1a7d3d78379 2024-03-11T21:22:07.401 INFO:journalctl@ceph.mgr.x.smithi097.stdout:Mar 11 21:22:07 smithi097 podman[35154]: 2024-03-11 21:22:07.164926901 +0000 UTC m=+0.120133771 container create b3c990ab09b5f1119217e21ef3929d7b4a26c4a3930ba40c7eb2a64b60cc69f2 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:8fe9635f3beb0247496d7e321f3068bd10c333c433ef4824d7e1c1a7d3d78379, name=ceph-00db0016-dfed-11ee-95c8-87774f69a715-mgr-x, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, GIT_CLEAN=True, CEPH_POINT_RELEASE=, RELEASE=main-2b0101e, io.buildah.version=1.33.5, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.build-date=20240307, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS) 2024-03-11T21:22:07.401 INFO:journalctl@ceph.mgr.x.smithi097.stdout:Mar 11 21:22:07 smithi097 podman[35154]: 2024-03-11 21:22:07.342711122 +0000 UTC m=+0.297917989 container init b3c990ab09b5f1119217e21ef3929d7b4a26c4a3930ba40c7eb2a64b60cc69f2 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:8fe9635f3beb0247496d7e321f3068bd10c333c433ef4824d7e1c1a7d3d78379, name=ceph-00db0016-dfed-11ee-95c8-87774f69a715-mgr-x, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240307, RELEASE=main-2b0101e, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=) 2024-03-11T21:22:07.401 INFO:journalctl@ceph.mgr.x.smithi097.stdout:Mar 11 21:22:07 smithi097 podman[35154]: 2024-03-11 21:22:07.34530952 +0000 UTC m=+0.300516386 container start b3c990ab09b5f1119217e21ef3929d7b4a26c4a3930ba40c7eb2a64b60cc69f2 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:8fe9635f3beb0247496d7e321f3068bd10c333c433ef4824d7e1c1a7d3d78379, name=ceph-00db0016-dfed-11ee-95c8-87774f69a715-mgr-x, ceph=True, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=main-2b0101e, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240307, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=) 2024-03-11T21:22:07.401 INFO:journalctl@ceph.mgr.x.smithi097.stdout:Mar 11 21:22:07 smithi097 bash[35154]: b3c990ab09b5f1119217e21ef3929d7b4a26c4a3930ba40c7eb2a64b60cc69f2 2024-03-11T21:22:07.828 INFO:journalctl@ceph.mgr.x.smithi097.stdout:Mar 11 21:22:07 smithi097 systemd[1]: Started Ceph mgr.x for 00db0016-dfed-11ee-95c8-87774f69a715. 2024-03-11T21:22:08.328 INFO:journalctl@ceph.mgr.x.smithi097.stdout:Mar 11 21:22:07 smithi097 ceph-00db0016-dfed-11ee-95c8-87774f69a715-mgr-x[35169]: 2024-03-11T21:22:07.904+0000 7f41ff20c200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-03-11T21:22:08.328 INFO:journalctl@ceph.mgr.x.smithi097.stdout:Mar 11 21:22:08 smithi097 ceph-00db0016-dfed-11ee-95c8-87774f69a715-mgr-x[35169]: 2024-03-11T21:22:08.001+0000 7f41ff20c200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-03-11T21:22:08.504 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:22:08 smithi092 ceph-mon[37563]: from='mgr.14150 v1:172.21.15.92:0/996057664' entity='mgr.y' 2024-03-11T21:22:08.504 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:22:08 smithi092 ceph-mon[37563]: pgmap v23: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T21:22:08.504 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:22:08 smithi092 ceph-mon[37563]: from='mgr.14150 v1:172.21.15.92:0/996057664' entity='mgr.y' 2024-03-11T21:22:08.504 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:22:08 smithi092 ceph-mon[37563]: from='mgr.14150 v1:172.21.15.92:0/996057664' entity='mgr.y' 2024-03-11T21:22:08.504 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:22:08 smithi092 ceph-mon[37563]: from='mgr.14150 v1:172.21.15.92:0/996057664' entity='mgr.y' 2024-03-11T21:22:08.504 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:22:08 smithi092 ceph-mon[37563]: from='mgr.14150 v1:172.21.15.92:0/996057664' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T21:22:08.504 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:22:08 smithi092 ceph-mon[29572]: from='mgr.14150 v1:172.21.15.92:0/996057664' entity='mgr.y' 2024-03-11T21:22:08.505 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:22:08 smithi092 ceph-mon[29572]: pgmap v23: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T21:22:08.505 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:22:08 smithi092 ceph-mon[29572]: from='mgr.14150 v1:172.21.15.92:0/996057664' entity='mgr.y' 2024-03-11T21:22:08.505 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:22:08 smithi092 ceph-mon[29572]: from='mgr.14150 v1:172.21.15.92:0/996057664' entity='mgr.y' 2024-03-11T21:22:08.505 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:22:08 smithi092 ceph-mon[29572]: from='mgr.14150 v1:172.21.15.92:0/996057664' entity='mgr.y' 2024-03-11T21:22:08.505 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:22:08 smithi092 ceph-mon[29572]: from='mgr.14150 v1:172.21.15.92:0/996057664' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T21:22:08.755 INFO:journalctl@ceph.mgr.x.smithi097.stdout:Mar 11 21:22:08 smithi097 ceph-00db0016-dfed-11ee-95c8-87774f69a715-mgr-x[35169]: 2024-03-11T21:22:08.678+0000 7f41ff20c200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-03-11T21:22:08.756 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:22:08 smithi097 ceph-mon[33317]: from='mgr.14150 v1:172.21.15.92:0/996057664' entity='mgr.y' 2024-03-11T21:22:08.756 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:22:08 smithi097 ceph-mon[33317]: pgmap v23: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T21:22:08.756 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:22:08 smithi097 ceph-mon[33317]: from='mgr.14150 v1:172.21.15.92:0/996057664' entity='mgr.y' 2024-03-11T21:22:08.756 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:22:08 smithi097 ceph-mon[33317]: from='mgr.14150 v1:172.21.15.92:0/996057664' entity='mgr.y' 2024-03-11T21:22:08.756 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:22:08 smithi097 ceph-mon[33317]: from='mgr.14150 v1:172.21.15.92:0/996057664' entity='mgr.y' 2024-03-11T21:22:08.756 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:22:08 smithi097 ceph-mon[33317]: from='mgr.14150 v1:172.21.15.92:0/996057664' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T21:22:09.011 INFO:journalctl@ceph.mgr.x.smithi097.stdout:Mar 11 21:22:08 smithi097 ceph-00db0016-dfed-11ee-95c8-87774f69a715-mgr-x[35169]: 2024-03-11T21:22:08.771+0000 7f41ff20c200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-03-11T21:22:09.011 INFO:journalctl@ceph.mgr.x.smithi097.stdout:Mar 11 21:22:08 smithi097 ceph-00db0016-dfed-11ee-95c8-87774f69a715-mgr-x[35169]: 2024-03-11T21:22:08.930+0000 7f41ff20c200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-03-11T21:22:09.011 INFO:journalctl@ceph.mgr.x.smithi097.stdout:Mar 11 21:22:09 smithi097 ceph-00db0016-dfed-11ee-95c8-87774f69a715-mgr-x[35169]: 2024-03-11T21:22:09.026+0000 7f41ff20c200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-03-11T21:22:09.073 INFO:teuthology.orchestra.run.smithi092.stderr:--> Zapping: /dev/vg_nvme/lv_4 2024-03-11T21:22:09.073 INFO:teuthology.orchestra.run.smithi092.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_4 bs=1M count=10 conv=fsync 2024-03-11T21:22:09.073 INFO:teuthology.orchestra.run.smithi092.stderr: stderr: 10+0 records in 2024-03-11T21:22:09.073 INFO:teuthology.orchestra.run.smithi092.stderr:10+0 records out 2024-03-11T21:22:09.073 INFO:teuthology.orchestra.run.smithi092.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.014325 s, 732 MB/s 2024-03-11T21:22:09.073 INFO:teuthology.orchestra.run.smithi092.stderr:--> Zapping successful for: 2024-03-11T21:22:09.330 INFO:journalctl@ceph.mgr.x.smithi097.stdout:Mar 11 21:22:09 smithi097 ceph-00db0016-dfed-11ee-95c8-87774f69a715-mgr-x[35169]: 2024-03-11T21:22:09.109+0000 7f41ff20c200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-03-11T21:22:09.649 INFO:journalctl@ceph.mgr.x.smithi097.stdout:Mar 11 21:22:09 smithi097 ceph-00db0016-dfed-11ee-95c8-87774f69a715-mgr-x[35169]: 2024-03-11T21:22:09.378+0000 7f41ff20c200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-03-11T21:22:09.650 INFO:journalctl@ceph.mgr.x.smithi097.stdout:Mar 11 21:22:09 smithi097 ceph-00db0016-dfed-11ee-95c8-87774f69a715-mgr-x[35169]: 2024-03-11T21:22:09.486+0000 7f41ff20c200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-03-11T21:22:09.650 INFO:journalctl@ceph.mgr.x.smithi097.stdout:Mar 11 21:22:09 smithi097 ceph-00db0016-dfed-11ee-95c8-87774f69a715-mgr-x[35169]: 2024-03-11T21:22:09.571+0000 7f41ff20c200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-03-11T21:22:09.650 INFO:journalctl@ceph.mgr.x.smithi097.stdout:Mar 11 21:22:09 smithi097 ceph-00db0016-dfed-11ee-95c8-87774f69a715-mgr-x[35169]: 2024-03-11T21:22:09.665+0000 7f41ff20c200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-03-11T21:22:09.708 DEBUG:teuthology.orchestra.run.smithi092:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 00db0016-dfed-11ee-95c8-87774f69a715 -- ceph orch daemon add osd smithi092:vg_nvme/lv_4 2024-03-11T21:22:10.004 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:22:09 smithi092 ceph-mon[29572]: from='mgr.14150 v1:172.21.15.92:0/996057664' entity='mgr.y' 2024-03-11T21:22:10.004 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:22:09 smithi092 ceph-mon[29572]: from='mgr.14150 v1:172.21.15.92:0/996057664' entity='mgr.y' 2024-03-11T21:22:10.004 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:22:09 smithi092 ceph-mon[29572]: from='mgr.14150 v1:172.21.15.92:0/996057664' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T21:22:10.004 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:22:09 smithi092 ceph-mon[29572]: from='mgr.14150 v1:172.21.15.92:0/996057664' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T21:22:10.004 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:22:09 smithi092 ceph-mon[29572]: from='mgr.14150 v1:172.21.15.92:0/996057664' entity='mgr.y' 2024-03-11T21:22:10.004 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:22:09 smithi092 ceph-mon[29572]: Reconfiguring mgr.y (unknown last config time)... 2024-03-11T21:22:10.004 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:22:09 smithi092 ceph-mon[29572]: from='mgr.14150 v1:172.21.15.92:0/996057664' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-03-11T21:22:10.005 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:22:09 smithi092 ceph-mon[29572]: from='mgr.14150 v1:172.21.15.92:0/996057664' entity='mgr.y' cmd=[{"prefix": "mgr services"}]: dispatch 2024-03-11T21:22:10.005 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:22:09 smithi092 ceph-mon[29572]: from='mgr.14150 v1:172.21.15.92:0/996057664' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T21:22:10.005 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:22:09 smithi092 ceph-mon[29572]: Reconfiguring daemon mgr.y on smithi092 2024-03-11T21:22:10.005 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:22:09 smithi092 ceph-mon[29572]: pgmap v24: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T21:22:10.005 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:22:09 smithi092 ceph-mon[29572]: from='mgr.14150 v1:172.21.15.92:0/996057664' entity='mgr.y' 2024-03-11T21:22:10.005 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:22:09 smithi092 ceph-mon[37563]: from='mgr.14150 v1:172.21.15.92:0/996057664' entity='mgr.y' 2024-03-11T21:22:10.005 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:22:09 smithi092 ceph-mon[37563]: from='mgr.14150 v1:172.21.15.92:0/996057664' entity='mgr.y' 2024-03-11T21:22:10.005 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:22:09 smithi092 ceph-mon[37563]: from='mgr.14150 v1:172.21.15.92:0/996057664' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T21:22:10.005 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:22:09 smithi092 ceph-mon[37563]: from='mgr.14150 v1:172.21.15.92:0/996057664' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T21:22:10.005 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:22:09 smithi092 ceph-mon[37563]: from='mgr.14150 v1:172.21.15.92:0/996057664' entity='mgr.y' 2024-03-11T21:22:10.006 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:22:09 smithi092 ceph-mon[37563]: Reconfiguring mgr.y (unknown last config time)... 2024-03-11T21:22:10.006 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:22:09 smithi092 ceph-mon[37563]: from='mgr.14150 v1:172.21.15.92:0/996057664' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-03-11T21:22:10.006 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:22:09 smithi092 ceph-mon[37563]: from='mgr.14150 v1:172.21.15.92:0/996057664' entity='mgr.y' cmd=[{"prefix": "mgr services"}]: dispatch 2024-03-11T21:22:10.006 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:22:09 smithi092 ceph-mon[37563]: from='mgr.14150 v1:172.21.15.92:0/996057664' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T21:22:10.006 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:22:09 smithi092 ceph-mon[37563]: Reconfiguring daemon mgr.y on smithi092 2024-03-11T21:22:10.006 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:22:09 smithi092 ceph-mon[37563]: pgmap v24: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T21:22:10.006 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:22:09 smithi092 ceph-mon[37563]: from='mgr.14150 v1:172.21.15.92:0/996057664' entity='mgr.y' 2024-03-11T21:22:10.078 INFO:journalctl@ceph.mgr.x.smithi097.stdout:Mar 11 21:22:09 smithi097 ceph-00db0016-dfed-11ee-95c8-87774f69a715-mgr-x[35169]: 2024-03-11T21:22:09.838+0000 7f41ff20c200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-03-11T21:22:10.078 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:22:09 smithi097 ceph-mon[33317]: from='mgr.14150 v1:172.21.15.92:0/996057664' entity='mgr.y' 2024-03-11T21:22:10.078 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:22:09 smithi097 ceph-mon[33317]: from='mgr.14150 v1:172.21.15.92:0/996057664' entity='mgr.y' 2024-03-11T21:22:10.078 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:22:09 smithi097 ceph-mon[33317]: from='mgr.14150 v1:172.21.15.92:0/996057664' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T21:22:10.078 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:22:09 smithi097 ceph-mon[33317]: from='mgr.14150 v1:172.21.15.92:0/996057664' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T21:22:10.078 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:22:09 smithi097 ceph-mon[33317]: from='mgr.14150 v1:172.21.15.92:0/996057664' entity='mgr.y' 2024-03-11T21:22:10.078 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:22:09 smithi097 ceph-mon[33317]: Reconfiguring mgr.y (unknown last config time)... 2024-03-11T21:22:10.078 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:22:09 smithi097 ceph-mon[33317]: from='mgr.14150 v1:172.21.15.92:0/996057664' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-03-11T21:22:10.079 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:22:09 smithi097 ceph-mon[33317]: from='mgr.14150 v1:172.21.15.92:0/996057664' entity='mgr.y' cmd=[{"prefix": "mgr services"}]: dispatch 2024-03-11T21:22:10.079 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:22:09 smithi097 ceph-mon[33317]: from='mgr.14150 v1:172.21.15.92:0/996057664' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T21:22:10.079 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:22:09 smithi097 ceph-mon[33317]: Reconfiguring daemon mgr.y on smithi092 2024-03-11T21:22:10.079 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:22:09 smithi097 ceph-mon[33317]: pgmap v24: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T21:22:10.079 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:22:09 smithi097 ceph-mon[33317]: from='mgr.14150 v1:172.21.15.92:0/996057664' entity='mgr.y' 2024-03-11T21:22:10.792 INFO:teuthology.orchestra.run.smithi092.stderr:Inferring config /var/lib/ceph/00db0016-dfed-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T21:22:11.078 INFO:journalctl@ceph.mgr.x.smithi097.stdout:Mar 11 21:22:10 smithi097 ceph-00db0016-dfed-11ee-95c8-87774f69a715-mgr-x[35169]: 2024-03-11T21:22:10.794+0000 7f41ff20c200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-03-11T21:22:11.078 INFO:journalctl@ceph.mgr.x.smithi097.stdout:Mar 11 21:22:10 smithi097 ceph-00db0016-dfed-11ee-95c8-87774f69a715-mgr-x[35169]: 2024-03-11T21:22:10.927+0000 7f41ff20c200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-03-11T21:22:11.541 INFO:journalctl@ceph.mgr.x.smithi097.stdout:Mar 11 21:22:11 smithi097 ceph-00db0016-dfed-11ee-95c8-87774f69a715-mgr-x[35169]: 2024-03-11T21:22:11.274+0000 7f41ff20c200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-03-11T21:22:11.541 INFO:journalctl@ceph.mgr.x.smithi097.stdout:Mar 11 21:22:11 smithi097 ceph-00db0016-dfed-11ee-95c8-87774f69a715-mgr-x[35169]: 2024-03-11T21:22:11.364+0000 7f41ff20c200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-03-11T21:22:11.541 INFO:journalctl@ceph.mgr.x.smithi097.stdout:Mar 11 21:22:11 smithi097 ceph-00db0016-dfed-11ee-95c8-87774f69a715-mgr-x[35169]: 2024-03-11T21:22:11.556+0000 7f41ff20c200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-03-11T21:22:11.828 INFO:journalctl@ceph.mgr.x.smithi097.stdout:Mar 11 21:22:11 smithi097 ceph-00db0016-dfed-11ee-95c8-87774f69a715-mgr-x[35169]: 2024-03-11T21:22:11.740+0000 7f41ff20c200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-03-11T21:22:11.828 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:22:11 smithi097 ceph-mon[33317]: from='mgr.14150 v1:172.21.15.92:0/996057664' entity='mgr.y' 2024-03-11T21:22:11.828 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:22:11 smithi097 ceph-mon[33317]: from='mgr.14150 v1:172.21.15.92:0/996057664' entity='mgr.y' 2024-03-11T21:22:11.828 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:22:11 smithi097 ceph-mon[33317]: from='mgr.14150 v1:172.21.15.92:0/996057664' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T21:22:11.828 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:22:11 smithi097 ceph-mon[33317]: from='mgr.14150 v1:172.21.15.92:0/996057664' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T21:22:11.828 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:22:11 smithi097 ceph-mon[33317]: from='mgr.14150 v1:172.21.15.92:0/996057664' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T21:22:11.828 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:22:11 smithi097 ceph-mon[33317]: from='mgr.14150 v1:172.21.15.92:0/996057664' entity='mgr.y' 2024-03-11T21:22:11.829 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:22:11 smithi097 ceph-mon[33317]: pgmap v25: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T21:22:12.020 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:22:11 smithi092 ceph-mon[29572]: from='mgr.14150 v1:172.21.15.92:0/996057664' entity='mgr.y' 2024-03-11T21:22:12.021 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:22:11 smithi092 ceph-mon[29572]: from='mgr.14150 v1:172.21.15.92:0/996057664' entity='mgr.y' 2024-03-11T21:22:12.021 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:22:11 smithi092 ceph-mon[29572]: from='mgr.14150 v1:172.21.15.92:0/996057664' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T21:22:12.021 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:22:11 smithi092 ceph-mon[29572]: from='mgr.14150 v1:172.21.15.92:0/996057664' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T21:22:12.021 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:22:11 smithi092 ceph-mon[29572]: from='mgr.14150 v1:172.21.15.92:0/996057664' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T21:22:12.021 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:22:11 smithi092 ceph-mon[29572]: from='mgr.14150 v1:172.21.15.92:0/996057664' entity='mgr.y' 2024-03-11T21:22:12.021 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:22:11 smithi092 ceph-mon[29572]: pgmap v25: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T21:22:12.021 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:22:11 smithi092 ceph-mon[37563]: from='mgr.14150 v1:172.21.15.92:0/996057664' entity='mgr.y' 2024-03-11T21:22:12.021 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:22:11 smithi092 ceph-mon[37563]: from='mgr.14150 v1:172.21.15.92:0/996057664' entity='mgr.y' 2024-03-11T21:22:12.021 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:22:11 smithi092 ceph-mon[37563]: from='mgr.14150 v1:172.21.15.92:0/996057664' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T21:22:12.021 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:22:11 smithi092 ceph-mon[37563]: from='mgr.14150 v1:172.21.15.92:0/996057664' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T21:22:12.022 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:22:11 smithi092 ceph-mon[37563]: from='mgr.14150 v1:172.21.15.92:0/996057664' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T21:22:12.022 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:22:11 smithi092 ceph-mon[37563]: from='mgr.14150 v1:172.21.15.92:0/996057664' entity='mgr.y' 2024-03-11T21:22:12.022 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:22:11 smithi092 ceph-mon[37563]: pgmap v25: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T21:22:12.287 INFO:journalctl@ceph.mgr.x.smithi097.stdout:Mar 11 21:22:12 smithi097 ceph-00db0016-dfed-11ee-95c8-87774f69a715-mgr-x[35169]: 2024-03-11T21:22:12.022+0000 7f41ff20c200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-03-11T21:22:12.287 INFO:journalctl@ceph.mgr.x.smithi097.stdout:Mar 11 21:22:12 smithi097 ceph-00db0016-dfed-11ee-95c8-87774f69a715-mgr-x[35169]: 2024-03-11T21:22:12.302+0000 7f41ff20c200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-03-11T21:22:12.578 INFO:journalctl@ceph.mgr.x.smithi097.stdout:Mar 11 21:22:12 smithi097 ceph-00db0016-dfed-11ee-95c8-87774f69a715-mgr-x[35169]: 2024-03-11T21:22:12.535+0000 7f41ff20c200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-03-11T21:22:13.112 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:22:12 smithi092 ceph-mon[29572]: from='client.14214 v1:172.21.15.92:0/1898311' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi092:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T21:22:13.112 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:22:12 smithi092 ceph-mon[29572]: from='mgr.14150 v1:172.21.15.92:0/996057664' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-11T21:22:13.112 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:22:12 smithi092 ceph-mon[29572]: from='mgr.14150 v1:172.21.15.92:0/996057664' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-11T21:22:13.112 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:22:12 smithi092 ceph-mon[29572]: from='mgr.14150 v1:172.21.15.92:0/996057664' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T21:22:13.112 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:22:12 smithi092 ceph-mon[37563]: from='client.14214 v1:172.21.15.92:0/1898311' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi092:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T21:22:13.113 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:22:12 smithi092 ceph-mon[37563]: from='mgr.14150 v1:172.21.15.92:0/996057664' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-11T21:22:13.113 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:22:12 smithi092 ceph-mon[37563]: from='mgr.14150 v1:172.21.15.92:0/996057664' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-11T21:22:13.113 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:22:12 smithi092 ceph-mon[37563]: from='mgr.14150 v1:172.21.15.92:0/996057664' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T21:22:13.311 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:22:12 smithi097 ceph-mon[33317]: from='client.14214 v1:172.21.15.92:0/1898311' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi092:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T21:22:13.311 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:22:12 smithi097 ceph-mon[33317]: from='mgr.14150 v1:172.21.15.92:0/996057664' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-11T21:22:13.311 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:22:12 smithi097 ceph-mon[33317]: from='mgr.14150 v1:172.21.15.92:0/996057664' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-11T21:22:13.311 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:22:12 smithi097 ceph-mon[33317]: from='mgr.14150 v1:172.21.15.92:0/996057664' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T21:22:13.578 INFO:journalctl@ceph.mgr.x.smithi097.stdout:Mar 11 21:22:13 smithi097 ceph-00db0016-dfed-11ee-95c8-87774f69a715-mgr-x[35169]: 2024-03-11T21:22:13.326+0000 7f41ff20c200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-03-11T21:22:14.107 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:22:13 smithi092 ceph-mon[29572]: pgmap v26: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T21:22:14.108 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:22:13 smithi092 ceph-mon[37563]: pgmap v26: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T21:22:14.185 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:22:13 smithi097 ceph-mon[33317]: pgmap v26: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T21:22:14.186 INFO:journalctl@ceph.mgr.x.smithi097.stdout:Mar 11 21:22:13 smithi097 ceph-00db0016-dfed-11ee-95c8-87774f69a715-mgr-x[35169]: 2024-03-11T21:22:13.892+0000 7f41ff20c200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-03-11T21:22:14.186 INFO:journalctl@ceph.mgr.x.smithi097.stdout:Mar 11 21:22:13 smithi097 ceph-00db0016-dfed-11ee-95c8-87774f69a715-mgr-x[35169]: 2024-03-11T21:22:13.990+0000 7f41ff20c200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-03-11T21:22:14.186 INFO:journalctl@ceph.mgr.x.smithi097.stdout:Mar 11 21:22:14 smithi097 ceph-00db0016-dfed-11ee-95c8-87774f69a715-mgr-x[35169]: 2024-03-11T21:22:14.201+0000 7f41ff20c200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-03-11T21:22:15.328 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:22:14 smithi097 ceph-mon[33317]: Standby manager daemon x started 2024-03-11T21:22:15.328 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:22:14 smithi097 ceph-mon[33317]: from='mgr.? v1:172.21.15.97:0/2207523917' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/crt"}]: dispatch 2024-03-11T21:22:15.328 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:22:14 smithi097 ceph-mon[33317]: from='mgr.? v1:172.21.15.97:0/2207523917' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-03-11T21:22:15.328 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:22:14 smithi097 ceph-mon[33317]: from='mgr.? v1:172.21.15.97:0/2207523917' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/key"}]: dispatch 2024-03-11T21:22:15.328 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:22:14 smithi097 ceph-mon[33317]: from='mgr.? v1:172.21.15.97:0/2207523917' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-03-11T21:22:15.328 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:22:14 smithi097 ceph-mon[33317]: from='client.? v1:172.21.15.92:0/3788479471' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "79f15843-f0ec-44aa-931e-4e07a74d3768"}]: dispatch 2024-03-11T21:22:15.328 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:22:14 smithi097 ceph-mon[33317]: from='client.34118 ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "79f15843-f0ec-44aa-931e-4e07a74d3768"}]: dispatch 2024-03-11T21:22:15.329 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:22:14 smithi097 ceph-mon[33317]: from='client.34118 ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "79f15843-f0ec-44aa-931e-4e07a74d3768"}]': finished 2024-03-11T21:22:15.329 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:22:14 smithi097 ceph-mon[33317]: osdmap e5: 1 total, 0 up, 1 in 2024-03-11T21:22:15.329 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:22:14 smithi097 ceph-mon[33317]: from='mgr.14150 v1:172.21.15.92:0/996057664' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-11T21:22:15.362 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:22:14 smithi092 ceph-mon[29572]: Standby manager daemon x started 2024-03-11T21:22:15.363 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:22:14 smithi092 ceph-mon[29572]: from='mgr.? v1:172.21.15.97:0/2207523917' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/crt"}]: dispatch 2024-03-11T21:22:15.363 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:22:14 smithi092 ceph-mon[29572]: from='mgr.? v1:172.21.15.97:0/2207523917' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-03-11T21:22:15.363 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:22:14 smithi092 ceph-mon[29572]: from='mgr.? v1:172.21.15.97:0/2207523917' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/key"}]: dispatch 2024-03-11T21:22:15.363 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:22:14 smithi092 ceph-mon[29572]: from='mgr.? v1:172.21.15.97:0/2207523917' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-03-11T21:22:15.363 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:22:14 smithi092 ceph-mon[29572]: from='client.? v1:172.21.15.92:0/3788479471' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "79f15843-f0ec-44aa-931e-4e07a74d3768"}]: dispatch 2024-03-11T21:22:15.363 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:22:14 smithi092 ceph-mon[29572]: from='client.34118 ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "79f15843-f0ec-44aa-931e-4e07a74d3768"}]: dispatch 2024-03-11T21:22:15.363 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:22:14 smithi092 ceph-mon[29572]: from='client.34118 ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "79f15843-f0ec-44aa-931e-4e07a74d3768"}]': finished 2024-03-11T21:22:15.363 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:22:14 smithi092 ceph-mon[29572]: osdmap e5: 1 total, 0 up, 1 in 2024-03-11T21:22:15.363 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:22:14 smithi092 ceph-mon[29572]: from='mgr.14150 v1:172.21.15.92:0/996057664' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-11T21:22:15.363 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:22:14 smithi092 ceph-mon[37563]: Standby manager daemon x started 2024-03-11T21:22:15.364 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:22:14 smithi092 ceph-mon[37563]: from='mgr.? v1:172.21.15.97:0/2207523917' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/crt"}]: dispatch 2024-03-11T21:22:15.364 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:22:14 smithi092 ceph-mon[37563]: from='mgr.? v1:172.21.15.97:0/2207523917' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-03-11T21:22:15.364 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:22:14 smithi092 ceph-mon[37563]: from='mgr.? v1:172.21.15.97:0/2207523917' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/key"}]: dispatch 2024-03-11T21:22:15.364 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:22:14 smithi092 ceph-mon[37563]: from='mgr.? v1:172.21.15.97:0/2207523917' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-03-11T21:22:15.364 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:22:14 smithi092 ceph-mon[37563]: from='client.? v1:172.21.15.92:0/3788479471' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "79f15843-f0ec-44aa-931e-4e07a74d3768"}]: dispatch 2024-03-11T21:22:15.364 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:22:14 smithi092 ceph-mon[37563]: from='client.34118 ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "79f15843-f0ec-44aa-931e-4e07a74d3768"}]: dispatch 2024-03-11T21:22:15.364 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:22:14 smithi092 ceph-mon[37563]: from='client.34118 ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "79f15843-f0ec-44aa-931e-4e07a74d3768"}]': finished 2024-03-11T21:22:15.364 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:22:14 smithi092 ceph-mon[37563]: osdmap e5: 1 total, 0 up, 1 in 2024-03-11T21:22:15.364 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:22:14 smithi092 ceph-mon[37563]: from='mgr.14150 v1:172.21.15.92:0/996057664' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-11T21:22:16.328 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:22:15 smithi097 ceph-mon[33317]: mgrmap e14: y(active, since 65s), standbys: x 2024-03-11T21:22:16.328 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:22:15 smithi097 ceph-mon[33317]: from='mgr.14150 v1:172.21.15.92:0/996057664' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-03-11T21:22:16.328 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:22:15 smithi097 ceph-mon[33317]: pgmap v28: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T21:22:16.328 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:22:15 smithi097 ceph-mon[33317]: from='client.? v1:172.21.15.92:0/1672845722' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-11T21:22:16.362 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:22:15 smithi092 ceph-mon[37563]: mgrmap e14: y(active, since 65s), standbys: x 2024-03-11T21:22:16.362 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:22:15 smithi092 ceph-mon[37563]: from='mgr.14150 v1:172.21.15.92:0/996057664' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-03-11T21:22:16.362 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:22:15 smithi092 ceph-mon[37563]: pgmap v28: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T21:22:16.362 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:22:15 smithi092 ceph-mon[37563]: from='client.? v1:172.21.15.92:0/1672845722' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-11T21:22:16.362 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:22:15 smithi092 ceph-mon[29572]: mgrmap e14: y(active, since 65s), standbys: x 2024-03-11T21:22:16.363 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:22:15 smithi092 ceph-mon[29572]: from='mgr.14150 v1:172.21.15.92:0/996057664' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-03-11T21:22:16.363 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:22:15 smithi092 ceph-mon[29572]: pgmap v28: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T21:22:16.363 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:22:15 smithi092 ceph-mon[29572]: from='client.? v1:172.21.15.92:0/1672845722' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-11T21:22:18.828 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:22:18 smithi097 ceph-mon[33317]: pgmap v29: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T21:22:18.862 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:22:18 smithi092 ceph-mon[29572]: pgmap v29: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T21:22:18.862 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:22:18 smithi092 ceph-mon[37563]: pgmap v29: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T21:22:20.828 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:22:20 smithi097 ceph-mon[33317]: pgmap v30: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T21:22:20.862 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:22:20 smithi092 ceph-mon[29572]: pgmap v30: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T21:22:20.863 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:22:20 smithi092 ceph-mon[37563]: pgmap v30: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T21:22:21.799 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:22:21 smithi097 ceph-mon[33317]: from='mgr.14150 v1:172.21.15.92:0/996057664' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.0"}]: dispatch 2024-03-11T21:22:21.799 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:22:21 smithi097 ceph-mon[33317]: from='mgr.14150 v1:172.21.15.92:0/996057664' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T21:22:21.862 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:22:21 smithi092 ceph-mon[29572]: from='mgr.14150 v1:172.21.15.92:0/996057664' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.0"}]: dispatch 2024-03-11T21:22:21.862 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:22:21 smithi092 ceph-mon[29572]: from='mgr.14150 v1:172.21.15.92:0/996057664' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T21:22:21.862 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:22:21 smithi092 ceph-mon[37563]: from='mgr.14150 v1:172.21.15.92:0/996057664' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.0"}]: dispatch 2024-03-11T21:22:21.862 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:22:21 smithi092 ceph-mon[37563]: from='mgr.14150 v1:172.21.15.92:0/996057664' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T21:22:22.774 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:22:22 smithi092 ceph-mon[29572]: Deploying daemon osd.0 on smithi092 2024-03-11T21:22:22.774 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:22:22 smithi092 ceph-mon[29572]: pgmap v31: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T21:22:22.775 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:22:22 smithi092 ceph-mon[37563]: Deploying daemon osd.0 on smithi092 2024-03-11T21:22:22.775 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:22:22 smithi092 ceph-mon[37563]: pgmap v31: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T21:22:22.828 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:22:22 smithi097 ceph-mon[33317]: Deploying daemon osd.0 on smithi092 2024-03-11T21:22:22.828 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:22:22 smithi097 ceph-mon[33317]: pgmap v31: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T21:22:24.328 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:22:23 smithi097 ceph-mon[33317]: pgmap v32: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T21:22:24.362 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:22:23 smithi092 ceph-mon[29572]: pgmap v32: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T21:22:24.363 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:22:23 smithi092 ceph-mon[37563]: pgmap v32: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T21:22:26.828 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:22:26 smithi097 ceph-mon[33317]: pgmap v33: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T21:22:26.828 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:22:26 smithi097 ceph-mon[33317]: from='mgr.14150 v1:172.21.15.92:0/996057664' entity='mgr.y' 2024-03-11T21:22:26.828 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:22:26 smithi097 ceph-mon[33317]: from='mgr.14150 v1:172.21.15.92:0/996057664' entity='mgr.y' 2024-03-11T21:22:26.828 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:22:26 smithi097 ceph-mon[33317]: from='mgr.14150 v1:172.21.15.92:0/996057664' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T21:22:26.862 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:22:26 smithi092 ceph-mon[29572]: pgmap v33: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T21:22:26.862 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:22:26 smithi092 ceph-mon[29572]: from='mgr.14150 v1:172.21.15.92:0/996057664' entity='mgr.y' 2024-03-11T21:22:26.862 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:22:26 smithi092 ceph-mon[29572]: from='mgr.14150 v1:172.21.15.92:0/996057664' entity='mgr.y' 2024-03-11T21:22:26.862 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:22:26 smithi092 ceph-mon[29572]: from='mgr.14150 v1:172.21.15.92:0/996057664' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T21:22:26.863 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:22:26 smithi092 ceph-mon[37563]: pgmap v33: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T21:22:26.863 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:22:26 smithi092 ceph-mon[37563]: from='mgr.14150 v1:172.21.15.92:0/996057664' entity='mgr.y' 2024-03-11T21:22:26.863 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:22:26 smithi092 ceph-mon[37563]: from='mgr.14150 v1:172.21.15.92:0/996057664' entity='mgr.y' 2024-03-11T21:22:26.863 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:22:26 smithi092 ceph-mon[37563]: from='mgr.14150 v1:172.21.15.92:0/996057664' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T21:22:28.828 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:22:28 smithi097 ceph-mon[33317]: pgmap v34: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T21:22:28.861 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:22:28 smithi092 ceph-mon[29572]: pgmap v34: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T21:22:28.861 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:22:28 smithi092 ceph-mon[37563]: pgmap v34: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T21:22:30.058 INFO:teuthology.orchestra.run.smithi092.stdout:Created osd(s) 0 on host 'smithi092' 2024-03-11T21:22:30.328 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:22:30 smithi097 ceph-mon[33317]: from='mgr.14150 v1:172.21.15.92:0/996057664' entity='mgr.y' 2024-03-11T21:22:30.328 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:22:30 smithi097 ceph-mon[33317]: from='mgr.14150 v1:172.21.15.92:0/996057664' entity='mgr.y' 2024-03-11T21:22:30.328 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:22:30 smithi097 ceph-mon[33317]: pgmap v35: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T21:22:30.328 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:22:30 smithi097 ceph-mon[33317]: from='osd.0 v1:172.21.15.92:6801/2844100201' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2024-03-11T21:22:30.328 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:22:30 smithi097 ceph-mon[33317]: from='mgr.14150 v1:172.21.15.92:0/996057664' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T21:22:30.328 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:22:30 smithi097 ceph-mon[33317]: from='mgr.14150 v1:172.21.15.92:0/996057664' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T21:22:30.328 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:22:30 smithi097 ceph-mon[33317]: from='mgr.14150 v1:172.21.15.92:0/996057664' entity='mgr.y' 2024-03-11T21:22:30.362 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:22:30 smithi092 ceph-mon[29572]: from='mgr.14150 v1:172.21.15.92:0/996057664' entity='mgr.y' 2024-03-11T21:22:30.362 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:22:30 smithi092 ceph-mon[29572]: from='mgr.14150 v1:172.21.15.92:0/996057664' entity='mgr.y' 2024-03-11T21:22:30.362 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:22:30 smithi092 ceph-mon[29572]: pgmap v35: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T21:22:30.362 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:22:30 smithi092 ceph-mon[29572]: from='osd.0 v1:172.21.15.92:6801/2844100201' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2024-03-11T21:22:30.362 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:22:30 smithi092 ceph-mon[29572]: from='mgr.14150 v1:172.21.15.92:0/996057664' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T21:22:30.362 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:22:30 smithi092 ceph-mon[29572]: from='mgr.14150 v1:172.21.15.92:0/996057664' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T21:22:30.362 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:22:30 smithi092 ceph-mon[29572]: from='mgr.14150 v1:172.21.15.92:0/996057664' entity='mgr.y' 2024-03-11T21:22:30.362 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:22:30 smithi092 ceph-mon[37563]: from='mgr.14150 v1:172.21.15.92:0/996057664' entity='mgr.y' 2024-03-11T21:22:30.363 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:22:30 smithi092 ceph-mon[37563]: from='mgr.14150 v1:172.21.15.92:0/996057664' entity='mgr.y' 2024-03-11T21:22:30.363 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:22:30 smithi092 ceph-mon[37563]: pgmap v35: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T21:22:30.363 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:22:30 smithi092 ceph-mon[37563]: from='osd.0 v1:172.21.15.92:6801/2844100201' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2024-03-11T21:22:30.363 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:22:30 smithi092 ceph-mon[37563]: from='mgr.14150 v1:172.21.15.92:0/996057664' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T21:22:30.363 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:22:30 smithi092 ceph-mon[37563]: from='mgr.14150 v1:172.21.15.92:0/996057664' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T21:22:30.363 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:22:30 smithi092 ceph-mon[37563]: from='mgr.14150 v1:172.21.15.92:0/996057664' entity='mgr.y' 2024-03-11T21:22:30.674 DEBUG:teuthology.orchestra.run.smithi092:osd.0> sudo journalctl -f -n 0 -u ceph-00db0016-dfed-11ee-95c8-87774f69a715@osd.0.service 2024-03-11T21:22:30.676 INFO:tasks.cephadm:Deploying osd.1 on smithi092 with /dev/vg_nvme/lv_3... 2024-03-11T21:22:30.677 DEBUG:teuthology.orchestra.run.smithi092:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 00db0016-dfed-11ee-95c8-87774f69a715 -- ceph-volume lvm zap /dev/vg_nvme/lv_3 2024-03-11T21:22:31.310 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:22:31 smithi097 ceph-mon[33317]: from='osd.0 v1:172.21.15.92:6801/2844100201' entity='osd.0' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]': finished 2024-03-11T21:22:31.310 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:22:31 smithi097 ceph-mon[33317]: osdmap e6: 1 total, 0 up, 1 in 2024-03-11T21:22:31.310 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:22:31 smithi097 ceph-mon[33317]: from='osd.0 v1:172.21.15.92:6801/2844100201' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi092", "root=default"]}]: dispatch 2024-03-11T21:22:31.310 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:22:31 smithi097 ceph-mon[33317]: from='mgr.14150 v1:172.21.15.92:0/996057664' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-11T21:22:31.311 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:22:31 smithi097 ceph-mon[33317]: from='mgr.14150 v1:172.21.15.92:0/996057664' entity='mgr.y' 2024-03-11T21:22:31.311 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:22:31 smithi097 ceph-mon[33317]: from='mgr.14150 v1:172.21.15.92:0/996057664' entity='mgr.y' 2024-03-11T21:22:31.311 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:22:31 smithi097 ceph-mon[33317]: from='mgr.14150 v1:172.21.15.92:0/996057664' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T21:22:31.362 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:22:31 smithi092 ceph-mon[37563]: from='osd.0 v1:172.21.15.92:6801/2844100201' entity='osd.0' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]': finished 2024-03-11T21:22:31.362 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:22:31 smithi092 ceph-mon[37563]: osdmap e6: 1 total, 0 up, 1 in 2024-03-11T21:22:31.362 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:22:31 smithi092 ceph-mon[37563]: from='osd.0 v1:172.21.15.92:6801/2844100201' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi092", "root=default"]}]: dispatch 2024-03-11T21:22:31.363 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:22:31 smithi092 ceph-mon[37563]: from='mgr.14150 v1:172.21.15.92:0/996057664' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-11T21:22:31.363 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:22:31 smithi092 ceph-mon[37563]: from='mgr.14150 v1:172.21.15.92:0/996057664' entity='mgr.y' 2024-03-11T21:22:31.363 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:22:31 smithi092 ceph-mon[37563]: from='mgr.14150 v1:172.21.15.92:0/996057664' entity='mgr.y' 2024-03-11T21:22:31.363 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:22:31 smithi092 ceph-mon[37563]: from='mgr.14150 v1:172.21.15.92:0/996057664' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T21:22:31.363 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:22:31 smithi092 ceph-mon[29572]: from='osd.0 v1:172.21.15.92:6801/2844100201' entity='osd.0' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]': finished 2024-03-11T21:22:31.363 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:22:31 smithi092 ceph-mon[29572]: osdmap e6: 1 total, 0 up, 1 in 2024-03-11T21:22:31.363 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:22:31 smithi092 ceph-mon[29572]: from='osd.0 v1:172.21.15.92:6801/2844100201' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi092", "root=default"]}]: dispatch 2024-03-11T21:22:31.363 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:22:31 smithi092 ceph-mon[29572]: from='mgr.14150 v1:172.21.15.92:0/996057664' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-11T21:22:31.363 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:22:31 smithi092 ceph-mon[29572]: from='mgr.14150 v1:172.21.15.92:0/996057664' entity='mgr.y' 2024-03-11T21:22:31.363 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:22:31 smithi092 ceph-mon[29572]: from='mgr.14150 v1:172.21.15.92:0/996057664' entity='mgr.y' 2024-03-11T21:22:31.363 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:22:31 smithi092 ceph-mon[29572]: from='mgr.14150 v1:172.21.15.92:0/996057664' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T21:22:31.364 INFO:journalctl@ceph.osd.0.smithi092.stdout:Mar 11 21:22:31 smithi092 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd-0[41435]: 2024-03-11T21:22:31.043+0000 7fa414f4b700 -1 osd.0 0 waiting for initial osdmap 2024-03-11T21:22:31.364 INFO:journalctl@ceph.osd.0.smithi092.stdout:Mar 11 21:22:31 smithi092 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd-0[41435]: 2024-03-11T21:22:31.046+0000 7fa410d3e700 -1 osd.0 7 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-11T21:22:31.653 INFO:teuthology.orchestra.run.smithi092.stderr:Inferring config /var/lib/ceph/00db0016-dfed-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T21:22:32.299 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:22:32 smithi092 ceph-mon[29572]: from='osd.0 v1:172.21.15.92:6801/2844100201' entity='osd.0' cmd='[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi092", "root=default"]}]': finished 2024-03-11T21:22:32.300 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:22:32 smithi092 ceph-mon[29572]: osdmap e7: 1 total, 0 up, 1 in 2024-03-11T21:22:32.300 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:22:32 smithi092 ceph-mon[29572]: from='mgr.14150 v1:172.21.15.92:0/996057664' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-11T21:22:32.300 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:22:32 smithi092 ceph-mon[29572]: from='mgr.14150 v1:172.21.15.92:0/996057664' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-11T21:22:32.300 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:22:32 smithi092 ceph-mon[29572]: pgmap v38: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T21:22:32.300 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:22:32 smithi092 ceph-mon[37563]: from='osd.0 v1:172.21.15.92:6801/2844100201' entity='osd.0' cmd='[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi092", "root=default"]}]': finished 2024-03-11T21:22:32.300 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:22:32 smithi092 ceph-mon[37563]: osdmap e7: 1 total, 0 up, 1 in 2024-03-11T21:22:32.300 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:22:32 smithi092 ceph-mon[37563]: from='mgr.14150 v1:172.21.15.92:0/996057664' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-11T21:22:32.300 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:22:32 smithi092 ceph-mon[37563]: from='mgr.14150 v1:172.21.15.92:0/996057664' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-11T21:22:32.300 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:22:32 smithi092 ceph-mon[37563]: pgmap v38: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T21:22:32.328 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:22:32 smithi097 ceph-mon[33317]: from='osd.0 v1:172.21.15.92:6801/2844100201' entity='osd.0' cmd='[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi092", "root=default"]}]': finished 2024-03-11T21:22:32.328 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:22:32 smithi097 ceph-mon[33317]: osdmap e7: 1 total, 0 up, 1 in 2024-03-11T21:22:32.328 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:22:32 smithi097 ceph-mon[33317]: from='mgr.14150 v1:172.21.15.92:0/996057664' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-11T21:22:32.328 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:22:32 smithi097 ceph-mon[33317]: from='mgr.14150 v1:172.21.15.92:0/996057664' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-11T21:22:32.328 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:22:32 smithi097 ceph-mon[33317]: pgmap v38: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-11T21:22:33.112 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:22:33 smithi092 ceph-mon[29572]: purged_snaps scrub starts 2024-03-11T21:22:33.112 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:22:33 smithi092 ceph-mon[29572]: purged_snaps scrub ok 2024-03-11T21:22:33.112 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:22:33 smithi092 ceph-mon[29572]: osd.0 v1:172.21.15.92:6801/2844100201 boot 2024-03-11T21:22:33.112 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:22:33 smithi092 ceph-mon[29572]: osdmap e8: 1 total, 1 up, 1 in 2024-03-11T21:22:33.113 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:22:33 smithi092 ceph-mon[29572]: from='mgr.14150 v1:172.21.15.92:0/996057664' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-11T21:22:33.113 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:22:33 smithi092 ceph-mon[37563]: purged_snaps scrub starts 2024-03-11T21:22:33.113 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:22:33 smithi092 ceph-mon[37563]: purged_snaps scrub ok 2024-03-11T21:22:33.113 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:22:33 smithi092 ceph-mon[37563]: osd.0 v1:172.21.15.92:6801/2844100201 boot 2024-03-11T21:22:33.113 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:22:33 smithi092 ceph-mon[37563]: osdmap e8: 1 total, 1 up, 1 in 2024-03-11T21:22:33.113 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:22:33 smithi092 ceph-mon[37563]: from='mgr.14150 v1:172.21.15.92:0/996057664' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-11T21:22:33.328 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:22:33 smithi097 ceph-mon[33317]: purged_snaps scrub starts 2024-03-11T21:22:33.328 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:22:33 smithi097 ceph-mon[33317]: purged_snaps scrub ok 2024-03-11T21:22:33.328 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:22:33 smithi097 ceph-mon[33317]: osd.0 v1:172.21.15.92:6801/2844100201 boot 2024-03-11T21:22:33.328 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:22:33 smithi097 ceph-mon[33317]: osdmap e8: 1 total, 1 up, 1 in 2024-03-11T21:22:33.328 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:22:33 smithi097 ceph-mon[33317]: from='mgr.14150 v1:172.21.15.92:0/996057664' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-11T21:22:33.960 INFO:teuthology.orchestra.run.smithi092.stderr:--> Zapping: /dev/vg_nvme/lv_3 2024-03-11T21:22:33.960 INFO:teuthology.orchestra.run.smithi092.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_3 bs=1M count=10 conv=fsync 2024-03-11T21:22:33.960 INFO:teuthology.orchestra.run.smithi092.stderr: stderr: 10+0 records in 2024-03-11T21:22:33.960 INFO:teuthology.orchestra.run.smithi092.stderr:10+0 records out 2024-03-11T21:22:33.960 INFO:teuthology.orchestra.run.smithi092.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.012567 s, 834 MB/s 2024-03-11T21:22:33.960 INFO:teuthology.orchestra.run.smithi092.stderr:--> Zapping successful for: 2024-03-11T21:22:34.112 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:22:34 smithi092 ceph-mon[29572]: osdmap e9: 1 total, 1 up, 1 in 2024-03-11T21:22:34.112 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:22:34 smithi092 ceph-mon[29572]: pgmap v41: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-11T21:22:34.112 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:22:34 smithi092 ceph-mon[37563]: osdmap e9: 1 total, 1 up, 1 in 2024-03-11T21:22:34.112 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:22:34 smithi092 ceph-mon[37563]: pgmap v41: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-11T21:22:34.327 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:22:34 smithi097 ceph-mon[33317]: osdmap e9: 1 total, 1 up, 1 in 2024-03-11T21:22:34.328 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:22:34 smithi097 ceph-mon[33317]: pgmap v41: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-11T21:22:34.541 DEBUG:teuthology.orchestra.run.smithi092:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 00db0016-dfed-11ee-95c8-87774f69a715 -- ceph orch daemon add osd smithi092:vg_nvme/lv_3 2024-03-11T21:22:35.017 INFO:teuthology.orchestra.run.smithi092.stderr:Inferring config /var/lib/ceph/00db0016-dfed-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T21:22:36.328 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:22:36 smithi097 ceph-mon[33317]: Detected new or changed devices on smithi092 2024-03-11T21:22:36.328 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:22:36 smithi097 ceph-mon[33317]: from='mgr.14150 v1:172.21.15.92:0/996057664' entity='mgr.y' 2024-03-11T21:22:36.328 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:22:36 smithi097 ceph-mon[33317]: from='mgr.14150 v1:172.21.15.92:0/996057664' entity='mgr.y' 2024-03-11T21:22:36.328 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:22:36 smithi097 ceph-mon[33317]: from='mgr.14150 v1:172.21.15.92:0/996057664' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-11T21:22:36.328 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:22:36 smithi097 ceph-mon[33317]: Adjusting osd_memory_target on smithi092 to 16064M 2024-03-11T21:22:36.328 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:22:36 smithi097 ceph-mon[33317]: from='mgr.14150 v1:172.21.15.92:0/996057664' entity='mgr.y' 2024-03-11T21:22:36.328 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:22:36 smithi097 ceph-mon[33317]: from='mgr.14150 v1:172.21.15.92:0/996057664' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T21:22:36.328 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:22:36 smithi097 ceph-mon[33317]: from='mgr.14150 v1:172.21.15.92:0/996057664' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T21:22:36.328 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:22:36 smithi097 ceph-mon[33317]: from='mgr.14150 v1:172.21.15.92:0/996057664' entity='mgr.y' 2024-03-11T21:22:36.328 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:22:36 smithi097 ceph-mon[33317]: pgmap v42: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-11T21:22:36.356 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:22:36 smithi092 ceph-mon[29572]: Detected new or changed devices on smithi092 2024-03-11T21:22:36.356 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:22:36 smithi092 ceph-mon[29572]: from='mgr.14150 v1:172.21.15.92:0/996057664' entity='mgr.y' 2024-03-11T21:22:36.356 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:22:36 smithi092 ceph-mon[29572]: from='mgr.14150 v1:172.21.15.92:0/996057664' entity='mgr.y' 2024-03-11T21:22:36.356 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:22:36 smithi092 ceph-mon[29572]: from='mgr.14150 v1:172.21.15.92:0/996057664' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-11T21:22:36.357 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:22:36 smithi092 ceph-mon[29572]: Adjusting osd_memory_target on smithi092 to 16064M 2024-03-11T21:22:36.357 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:22:36 smithi092 ceph-mon[29572]: from='mgr.14150 v1:172.21.15.92:0/996057664' entity='mgr.y' 2024-03-11T21:22:36.357 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:22:36 smithi092 ceph-mon[29572]: from='mgr.14150 v1:172.21.15.92:0/996057664' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T21:22:36.357 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:22:36 smithi092 ceph-mon[29572]: from='mgr.14150 v1:172.21.15.92:0/996057664' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T21:22:36.357 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:22:36 smithi092 ceph-mon[29572]: from='mgr.14150 v1:172.21.15.92:0/996057664' entity='mgr.y' 2024-03-11T21:22:36.357 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:22:36 smithi092 ceph-mon[29572]: pgmap v42: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-11T21:22:36.357 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:22:36 smithi092 ceph-mon[37563]: Detected new or changed devices on smithi092 2024-03-11T21:22:36.357 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:22:36 smithi092 ceph-mon[37563]: from='mgr.14150 v1:172.21.15.92:0/996057664' entity='mgr.y' 2024-03-11T21:22:36.357 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:22:36 smithi092 ceph-mon[37563]: from='mgr.14150 v1:172.21.15.92:0/996057664' entity='mgr.y' 2024-03-11T21:22:36.357 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:22:36 smithi092 ceph-mon[37563]: from='mgr.14150 v1:172.21.15.92:0/996057664' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-11T21:22:36.357 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:22:36 smithi092 ceph-mon[37563]: Adjusting osd_memory_target on smithi092 to 16064M 2024-03-11T21:22:36.358 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:22:36 smithi092 ceph-mon[37563]: from='mgr.14150 v1:172.21.15.92:0/996057664' entity='mgr.y' 2024-03-11T21:22:36.358 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:22:36 smithi092 ceph-mon[37563]: from='mgr.14150 v1:172.21.15.92:0/996057664' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T21:22:36.358 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:22:36 smithi092 ceph-mon[37563]: from='mgr.14150 v1:172.21.15.92:0/996057664' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T21:22:36.358 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:22:36 smithi092 ceph-mon[37563]: from='mgr.14150 v1:172.21.15.92:0/996057664' entity='mgr.y' 2024-03-11T21:22:36.358 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:22:36 smithi092 ceph-mon[37563]: pgmap v42: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-11T21:22:37.328 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:22:37 smithi097 ceph-mon[33317]: from='mgr.14150 v1:172.21.15.92:0/996057664' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-11T21:22:37.328 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:22:37 smithi097 ceph-mon[33317]: from='mgr.14150 v1:172.21.15.92:0/996057664' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-11T21:22:37.328 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:22:37 smithi097 ceph-mon[33317]: from='mgr.14150 v1:172.21.15.92:0/996057664' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T21:22:37.359 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:22:37 smithi092 ceph-mon[29572]: from='mgr.14150 v1:172.21.15.92:0/996057664' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-11T21:22:37.359 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:22:37 smithi092 ceph-mon[29572]: from='mgr.14150 v1:172.21.15.92:0/996057664' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-11T21:22:37.359 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:22:37 smithi092 ceph-mon[29572]: from='mgr.14150 v1:172.21.15.92:0/996057664' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T21:22:37.359 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:22:37 smithi092 ceph-mon[37563]: from='mgr.14150 v1:172.21.15.92:0/996057664' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-11T21:22:37.360 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:22:37 smithi092 ceph-mon[37563]: from='mgr.14150 v1:172.21.15.92:0/996057664' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-11T21:22:37.360 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:22:37 smithi092 ceph-mon[37563]: from='mgr.14150 v1:172.21.15.92:0/996057664' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T21:22:38.328 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:22:38 smithi097 ceph-mon[33317]: from='client.34139 v1:172.21.15.92:0/588427462' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi092:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T21:22:38.328 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:22:38 smithi097 ceph-mon[33317]: pgmap v43: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-11T21:22:38.332 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:22:38 smithi092 ceph-mon[29572]: from='client.34139 v1:172.21.15.92:0/588427462' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi092:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T21:22:38.332 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:22:38 smithi092 ceph-mon[29572]: pgmap v43: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-11T21:22:38.332 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:22:38 smithi092 ceph-mon[37563]: from='client.34139 v1:172.21.15.92:0/588427462' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi092:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T21:22:38.332 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:22:38 smithi092 ceph-mon[37563]: pgmap v43: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-11T21:22:39.612 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:22:39 smithi092 ceph-mon[29572]: from='client.? v1:172.21.15.92:0/884986282' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "3a5a0e78-1293-4ace-a93f-69156098571d"}]: dispatch 2024-03-11T21:22:39.612 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:22:39 smithi092 ceph-mon[29572]: from='client.? v1:172.21.15.92:0/884986282' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "3a5a0e78-1293-4ace-a93f-69156098571d"}]': finished 2024-03-11T21:22:39.612 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:22:39 smithi092 ceph-mon[29572]: osdmap e10: 2 total, 1 up, 2 in 2024-03-11T21:22:39.613 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:22:39 smithi092 ceph-mon[29572]: from='mgr.14150 v1:172.21.15.92:0/996057664' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-11T21:22:39.613 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:22:39 smithi092 ceph-mon[37563]: from='client.? v1:172.21.15.92:0/884986282' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "3a5a0e78-1293-4ace-a93f-69156098571d"}]: dispatch 2024-03-11T21:22:39.613 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:22:39 smithi092 ceph-mon[37563]: from='client.? v1:172.21.15.92:0/884986282' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "3a5a0e78-1293-4ace-a93f-69156098571d"}]': finished 2024-03-11T21:22:39.613 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:22:39 smithi092 ceph-mon[37563]: osdmap e10: 2 total, 1 up, 2 in 2024-03-11T21:22:39.613 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:22:39 smithi092 ceph-mon[37563]: from='mgr.14150 v1:172.21.15.92:0/996057664' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-11T21:22:39.828 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:22:39 smithi097 ceph-mon[33317]: from='client.? v1:172.21.15.92:0/884986282' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "3a5a0e78-1293-4ace-a93f-69156098571d"}]: dispatch 2024-03-11T21:22:39.828 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:22:39 smithi097 ceph-mon[33317]: from='client.? v1:172.21.15.92:0/884986282' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "3a5a0e78-1293-4ace-a93f-69156098571d"}]': finished 2024-03-11T21:22:39.828 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:22:39 smithi097 ceph-mon[33317]: osdmap e10: 2 total, 1 up, 2 in 2024-03-11T21:22:39.828 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:22:39 smithi097 ceph-mon[33317]: from='mgr.14150 v1:172.21.15.92:0/996057664' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-11T21:22:40.612 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:22:40 smithi092 ceph-mon[29572]: pgmap v45: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-11T21:22:40.612 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:22:40 smithi092 ceph-mon[29572]: from='client.? v1:172.21.15.92:0/3377555336' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-11T21:22:40.612 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:22:40 smithi092 ceph-mon[37563]: pgmap v45: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-11T21:22:40.612 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:22:40 smithi092 ceph-mon[37563]: from='client.? v1:172.21.15.92:0/3377555336' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-11T21:22:40.828 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:22:40 smithi097 ceph-mon[33317]: pgmap v45: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-11T21:22:40.828 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:22:40 smithi097 ceph-mon[33317]: from='client.? v1:172.21.15.92:0/3377555336' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-11T21:22:42.773 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:22:42 smithi092 ceph-mon[29572]: pgmap v46: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-11T21:22:42.773 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:22:42 smithi092 ceph-mon[37563]: pgmap v46: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-11T21:22:42.828 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:22:42 smithi097 ceph-mon[33317]: pgmap v46: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-11T21:22:44.328 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:22:43 smithi097 ceph-mon[33317]: pgmap v47: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-11T21:22:44.362 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:22:43 smithi092 ceph-mon[37563]: pgmap v47: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-11T21:22:44.365 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:22:43 smithi092 ceph-mon[29572]: pgmap v47: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-11T21:22:46.722 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:22:46 smithi092 ceph-mon[29572]: pgmap v48: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-11T21:22:46.723 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:22:46 smithi092 ceph-mon[29572]: from='mgr.14150 v1:172.21.15.92:0/996057664' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.1"}]: dispatch 2024-03-11T21:22:46.723 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:22:46 smithi092 ceph-mon[29572]: from='mgr.14150 v1:172.21.15.92:0/996057664' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T21:22:46.723 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:22:46 smithi092 ceph-mon[37563]: pgmap v48: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-11T21:22:46.723 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:22:46 smithi092 ceph-mon[37563]: from='mgr.14150 v1:172.21.15.92:0/996057664' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.1"}]: dispatch 2024-03-11T21:22:46.723 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:22:46 smithi092 ceph-mon[37563]: from='mgr.14150 v1:172.21.15.92:0/996057664' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T21:22:46.827 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:22:46 smithi097 ceph-mon[33317]: pgmap v48: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-11T21:22:46.828 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:22:46 smithi097 ceph-mon[33317]: from='mgr.14150 v1:172.21.15.92:0/996057664' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.1"}]: dispatch 2024-03-11T21:22:46.828 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:22:46 smithi097 ceph-mon[33317]: from='mgr.14150 v1:172.21.15.92:0/996057664' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T21:22:47.812 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:22:47 smithi092 ceph-mon[29572]: Deploying daemon osd.1 on smithi092 2024-03-11T21:22:47.813 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:22:47 smithi092 ceph-mon[37563]: Deploying daemon osd.1 on smithi092 2024-03-11T21:22:47.827 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:22:47 smithi097 ceph-mon[33317]: Deploying daemon osd.1 on smithi092 2024-03-11T21:22:48.828 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:22:48 smithi097 ceph-mon[33317]: pgmap v49: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-11T21:22:48.862 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:22:48 smithi092 ceph-mon[29572]: pgmap v49: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-11T21:22:48.862 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:22:48 smithi092 ceph-mon[37563]: pgmap v49: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-11T21:22:50.803 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:22:50 smithi092 ceph-mon[29572]: pgmap v50: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-11T21:22:50.804 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:22:50 smithi092 ceph-mon[37563]: pgmap v50: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-11T21:22:50.828 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:22:50 smithi097 ceph-mon[33317]: pgmap v50: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-11T21:22:52.349 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:22:52 smithi092 ceph-mon[29572]: from='mgr.14150 v1:172.21.15.92:0/996057664' entity='mgr.y' 2024-03-11T21:22:52.349 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:22:52 smithi092 ceph-mon[29572]: from='mgr.14150 v1:172.21.15.92:0/996057664' entity='mgr.y' 2024-03-11T21:22:52.349 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:22:52 smithi092 ceph-mon[29572]: from='mgr.14150 v1:172.21.15.92:0/996057664' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T21:22:52.349 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:22:52 smithi092 ceph-mon[29572]: pgmap v51: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-11T21:22:52.350 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:22:52 smithi092 ceph-mon[37563]: from='mgr.14150 v1:172.21.15.92:0/996057664' entity='mgr.y' 2024-03-11T21:22:52.350 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:22:52 smithi092 ceph-mon[37563]: from='mgr.14150 v1:172.21.15.92:0/996057664' entity='mgr.y' 2024-03-11T21:22:52.350 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:22:52 smithi092 ceph-mon[37563]: from='mgr.14150 v1:172.21.15.92:0/996057664' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T21:22:52.350 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:22:52 smithi092 ceph-mon[37563]: pgmap v51: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-11T21:22:52.578 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:22:52 smithi097 ceph-mon[33317]: from='mgr.14150 v1:172.21.15.92:0/996057664' entity='mgr.y' 2024-03-11T21:22:52.578 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:22:52 smithi097 ceph-mon[33317]: from='mgr.14150 v1:172.21.15.92:0/996057664' entity='mgr.y' 2024-03-11T21:22:52.578 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:22:52 smithi097 ceph-mon[33317]: from='mgr.14150 v1:172.21.15.92:0/996057664' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T21:22:52.578 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:22:52 smithi097 ceph-mon[33317]: pgmap v51: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-11T21:22:54.703 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:22:54 smithi092 ceph-mon[29572]: pgmap v52: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-11T21:22:54.703 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:22:54 smithi092 ceph-mon[29572]: from='mgr.14150 v1:172.21.15.92:0/996057664' entity='mgr.y' 2024-03-11T21:22:54.703 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:22:54 smithi092 ceph-mon[29572]: from='mgr.14150 v1:172.21.15.92:0/996057664' entity='mgr.y' 2024-03-11T21:22:54.703 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:22:54 smithi092 ceph-mon[29572]: from='mgr.14150 v1:172.21.15.92:0/996057664' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T21:22:54.703 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:22:54 smithi092 ceph-mon[29572]: from='mgr.14150 v1:172.21.15.92:0/996057664' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T21:22:54.703 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:22:54 smithi092 ceph-mon[29572]: from='mgr.14150 v1:172.21.15.92:0/996057664' entity='mgr.y' 2024-03-11T21:22:54.703 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:22:54 smithi092 ceph-mon[29572]: from='osd.1 v1:172.21.15.92:6805/3577871157' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-03-11T21:22:54.703 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:22:54 smithi092 ceph-mon[29572]: from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-03-11T21:22:54.703 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:22:54 smithi092 ceph-mon[37563]: pgmap v52: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-11T21:22:54.703 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:22:54 smithi092 ceph-mon[37563]: from='mgr.14150 v1:172.21.15.92:0/996057664' entity='mgr.y' 2024-03-11T21:22:54.704 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:22:54 smithi092 ceph-mon[37563]: from='mgr.14150 v1:172.21.15.92:0/996057664' entity='mgr.y' 2024-03-11T21:22:54.704 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:22:54 smithi092 ceph-mon[37563]: from='mgr.14150 v1:172.21.15.92:0/996057664' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T21:22:54.704 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:22:54 smithi092 ceph-mon[37563]: from='mgr.14150 v1:172.21.15.92:0/996057664' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T21:22:54.704 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:22:54 smithi092 ceph-mon[37563]: from='mgr.14150 v1:172.21.15.92:0/996057664' entity='mgr.y' 2024-03-11T21:22:54.704 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:22:54 smithi092 ceph-mon[37563]: from='osd.1 v1:172.21.15.92:6805/3577871157' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-03-11T21:22:54.704 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:22:54 smithi092 ceph-mon[37563]: from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-03-11T21:22:54.778 INFO:teuthology.orchestra.run.smithi092.stdout:Created osd(s) 1 on host 'smithi092' 2024-03-11T21:22:55.078 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:22:54 smithi097 ceph-mon[33317]: pgmap v52: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-11T21:22:55.078 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:22:54 smithi097 ceph-mon[33317]: from='mgr.14150 v1:172.21.15.92:0/996057664' entity='mgr.y' 2024-03-11T21:22:55.078 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:22:54 smithi097 ceph-mon[33317]: from='mgr.14150 v1:172.21.15.92:0/996057664' entity='mgr.y' 2024-03-11T21:22:55.078 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:22:54 smithi097 ceph-mon[33317]: from='mgr.14150 v1:172.21.15.92:0/996057664' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T21:22:55.078 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:22:54 smithi097 ceph-mon[33317]: from='mgr.14150 v1:172.21.15.92:0/996057664' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T21:22:55.078 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:22:54 smithi097 ceph-mon[33317]: from='mgr.14150 v1:172.21.15.92:0/996057664' entity='mgr.y' 2024-03-11T21:22:55.078 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:22:54 smithi097 ceph-mon[33317]: from='osd.1 v1:172.21.15.92:6805/3577871157' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-03-11T21:22:55.078 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:22:54 smithi097 ceph-mon[33317]: from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-03-11T21:22:55.372 DEBUG:teuthology.orchestra.run.smithi092:osd.1> sudo journalctl -f -n 0 -u ceph-00db0016-dfed-11ee-95c8-87774f69a715@osd.1.service 2024-03-11T21:22:55.374 INFO:tasks.cephadm:Deploying osd.2 on smithi092 with /dev/vg_nvme/lv_2... 2024-03-11T21:22:55.374 DEBUG:teuthology.orchestra.run.smithi092:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 00db0016-dfed-11ee-95c8-87774f69a715 -- ceph-volume lvm zap /dev/vg_nvme/lv_2 2024-03-11T21:22:55.862 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:22:55 smithi092 ceph-mon[29572]: from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]': finished 2024-03-11T21:22:55.862 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:22:55 smithi092 ceph-mon[29572]: osdmap e11: 2 total, 1 up, 2 in 2024-03-11T21:22:55.862 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:22:55 smithi092 ceph-mon[29572]: from='mgr.14150 v1:172.21.15.92:0/996057664' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-11T21:22:55.862 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:22:55 smithi092 ceph-mon[29572]: from='osd.1 v1:172.21.15.92:6805/3577871157' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi092", "root=default"]}]: dispatch 2024-03-11T21:22:55.862 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:22:55 smithi092 ceph-mon[29572]: from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi092", "root=default"]}]: dispatch 2024-03-11T21:22:55.863 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:22:55 smithi092 ceph-mon[29572]: from='mgr.14150 v1:172.21.15.92:0/996057664' entity='mgr.y' 2024-03-11T21:22:55.863 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:22:55 smithi092 ceph-mon[29572]: from='mgr.14150 v1:172.21.15.92:0/996057664' entity='mgr.y' 2024-03-11T21:22:55.863 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:22:55 smithi092 ceph-mon[29572]: from='mgr.14150 v1:172.21.15.92:0/996057664' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T21:22:55.863 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:22:55 smithi092 ceph-mon[37563]: from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]': finished 2024-03-11T21:22:55.863 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:22:55 smithi092 ceph-mon[37563]: osdmap e11: 2 total, 1 up, 2 in 2024-03-11T21:22:55.863 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:22:55 smithi092 ceph-mon[37563]: from='mgr.14150 v1:172.21.15.92:0/996057664' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-11T21:22:55.863 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:22:55 smithi092 ceph-mon[37563]: from='osd.1 v1:172.21.15.92:6805/3577871157' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi092", "root=default"]}]: dispatch 2024-03-11T21:22:55.863 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:22:55 smithi092 ceph-mon[37563]: from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi092", "root=default"]}]: dispatch 2024-03-11T21:22:55.863 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:22:55 smithi092 ceph-mon[37563]: from='mgr.14150 v1:172.21.15.92:0/996057664' entity='mgr.y' 2024-03-11T21:22:55.863 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:22:55 smithi092 ceph-mon[37563]: from='mgr.14150 v1:172.21.15.92:0/996057664' entity='mgr.y' 2024-03-11T21:22:55.863 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:22:55 smithi092 ceph-mon[37563]: from='mgr.14150 v1:172.21.15.92:0/996057664' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T21:22:55.864 INFO:journalctl@ceph.osd.1.smithi092.stdout:Mar 11 21:22:55 smithi092 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd-1[47398]: 2024-03-11T21:22:55.625+0000 7f704c94f700 -1 osd.1 0 waiting for initial osdmap 2024-03-11T21:22:55.864 INFO:journalctl@ceph.osd.1.smithi092.stdout:Mar 11 21:22:55 smithi092 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd-1[47398]: 2024-03-11T21:22:55.627+0000 7f7048742700 -1 osd.1 12 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-11T21:22:56.077 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:22:55 smithi097 ceph-mon[33317]: from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]': finished 2024-03-11T21:22:56.078 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:22:55 smithi097 ceph-mon[33317]: osdmap e11: 2 total, 1 up, 2 in 2024-03-11T21:22:56.078 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:22:55 smithi097 ceph-mon[33317]: from='mgr.14150 v1:172.21.15.92:0/996057664' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-11T21:22:56.078 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:22:55 smithi097 ceph-mon[33317]: from='osd.1 v1:172.21.15.92:6805/3577871157' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi092", "root=default"]}]: dispatch 2024-03-11T21:22:56.078 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:22:55 smithi097 ceph-mon[33317]: from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi092", "root=default"]}]: dispatch 2024-03-11T21:22:56.078 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:22:55 smithi097 ceph-mon[33317]: from='mgr.14150 v1:172.21.15.92:0/996057664' entity='mgr.y' 2024-03-11T21:22:56.078 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:22:55 smithi097 ceph-mon[33317]: from='mgr.14150 v1:172.21.15.92:0/996057664' entity='mgr.y' 2024-03-11T21:22:56.078 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:22:55 smithi097 ceph-mon[33317]: from='mgr.14150 v1:172.21.15.92:0/996057664' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T21:22:56.621 INFO:teuthology.orchestra.run.smithi092.stderr:Inferring config /var/lib/ceph/00db0016-dfed-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T21:22:56.821 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:22:56 smithi092 ceph-mon[29572]: pgmap v54: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-11T21:22:56.822 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:22:56 smithi092 ceph-mon[29572]: from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi092", "root=default"]}]': finished 2024-03-11T21:22:56.822 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:22:56 smithi092 ceph-mon[29572]: osdmap e12: 2 total, 1 up, 2 in 2024-03-11T21:22:56.822 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:22:56 smithi092 ceph-mon[29572]: from='mgr.14150 v1:172.21.15.92:0/996057664' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-11T21:22:56.822 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:22:56 smithi092 ceph-mon[29572]: from='mgr.14150 v1:172.21.15.92:0/996057664' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-11T21:22:56.822 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:22:56 smithi092 ceph-mon[29572]: from='mgr.14150 v1:172.21.15.92:0/996057664' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-11T21:22:56.822 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:22:56 smithi092 ceph-mon[37563]: pgmap v54: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-11T21:22:56.822 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:22:56 smithi092 ceph-mon[37563]: from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi092", "root=default"]}]': finished 2024-03-11T21:22:56.822 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:22:56 smithi092 ceph-mon[37563]: osdmap e12: 2 total, 1 up, 2 in 2024-03-11T21:22:56.822 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:22:56 smithi092 ceph-mon[37563]: from='mgr.14150 v1:172.21.15.92:0/996057664' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-11T21:22:56.822 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:22:56 smithi092 ceph-mon[37563]: from='mgr.14150 v1:172.21.15.92:0/996057664' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-11T21:22:56.823 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:22:56 smithi092 ceph-mon[37563]: from='mgr.14150 v1:172.21.15.92:0/996057664' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-11T21:22:57.078 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:22:56 smithi097 ceph-mon[33317]: pgmap v54: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-11T21:22:57.078 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:22:56 smithi097 ceph-mon[33317]: from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi092", "root=default"]}]': finished 2024-03-11T21:22:57.078 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:22:56 smithi097 ceph-mon[33317]: osdmap e12: 2 total, 1 up, 2 in 2024-03-11T21:22:57.078 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:22:56 smithi097 ceph-mon[33317]: from='mgr.14150 v1:172.21.15.92:0/996057664' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-11T21:22:57.078 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:22:56 smithi097 ceph-mon[33317]: from='mgr.14150 v1:172.21.15.92:0/996057664' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-11T21:22:57.078 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:22:56 smithi097 ceph-mon[33317]: from='mgr.14150 v1:172.21.15.92:0/996057664' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-11T21:22:57.862 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:22:57 smithi092 ceph-mon[29572]: purged_snaps scrub starts 2024-03-11T21:22:57.862 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:22:57 smithi092 ceph-mon[29572]: purged_snaps scrub ok 2024-03-11T21:22:57.862 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:22:57 smithi092 ceph-mon[29572]: osd.1 v1:172.21.15.92:6805/3577871157 boot 2024-03-11T21:22:57.863 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:22:57 smithi092 ceph-mon[29572]: osdmap e13: 2 total, 2 up, 2 in 2024-03-11T21:22:57.863 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:22:57 smithi092 ceph-mon[29572]: from='mgr.14150 v1:172.21.15.92:0/996057664' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-11T21:22:57.863 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:22:57 smithi092 ceph-mon[29572]: pgmap v57: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-11T21:22:57.863 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:22:57 smithi092 ceph-mon[37563]: purged_snaps scrub starts 2024-03-11T21:22:57.863 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:22:57 smithi092 ceph-mon[37563]: purged_snaps scrub ok 2024-03-11T21:22:57.863 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:22:57 smithi092 ceph-mon[37563]: osd.1 v1:172.21.15.92:6805/3577871157 boot 2024-03-11T21:22:57.863 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:22:57 smithi092 ceph-mon[37563]: osdmap e13: 2 total, 2 up, 2 in 2024-03-11T21:22:57.863 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:22:57 smithi092 ceph-mon[37563]: from='mgr.14150 v1:172.21.15.92:0/996057664' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-11T21:22:57.863 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:22:57 smithi092 ceph-mon[37563]: pgmap v57: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-11T21:22:58.078 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:22:57 smithi097 ceph-mon[33317]: purged_snaps scrub starts 2024-03-11T21:22:58.078 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:22:57 smithi097 ceph-mon[33317]: purged_snaps scrub ok 2024-03-11T21:22:58.078 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:22:57 smithi097 ceph-mon[33317]: osd.1 v1:172.21.15.92:6805/3577871157 boot 2024-03-11T21:22:58.078 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:22:57 smithi097 ceph-mon[33317]: osdmap e13: 2 total, 2 up, 2 in 2024-03-11T21:22:58.078 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:22:57 smithi097 ceph-mon[33317]: from='mgr.14150 v1:172.21.15.92:0/996057664' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-11T21:22:58.078 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:22:57 smithi097 ceph-mon[33317]: pgmap v57: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-11T21:22:58.684 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:22:58 smithi092 ceph-mon[29572]: osdmap e14: 2 total, 2 up, 2 in 2024-03-11T21:22:58.684 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:22:58 smithi092 ceph-mon[37563]: osdmap e14: 2 total, 2 up, 2 in 2024-03-11T21:22:58.942 INFO:teuthology.orchestra.run.smithi092.stderr:--> Zapping: /dev/vg_nvme/lv_2 2024-03-11T21:22:58.943 INFO:teuthology.orchestra.run.smithi092.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_2 bs=1M count=10 conv=fsync 2024-03-11T21:22:58.943 INFO:teuthology.orchestra.run.smithi092.stderr: stderr: 10+0 records in 2024-03-11T21:22:58.943 INFO:teuthology.orchestra.run.smithi092.stderr:10+0 records out 2024-03-11T21:22:58.943 INFO:teuthology.orchestra.run.smithi092.stderr:10485760 bytes (10 MB, 10 MiB) copied, 0.0114504 s, 916 MB/s 2024-03-11T21:22:58.943 INFO:teuthology.orchestra.run.smithi092.stderr: stderr: 2024-03-11T21:22:58.943 INFO:teuthology.orchestra.run.smithi092.stderr:--> Zapping successful for: 2024-03-11T21:22:59.078 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:22:58 smithi097 ceph-mon[33317]: osdmap e14: 2 total, 2 up, 2 in 2024-03-11T21:23:00.028 DEBUG:teuthology.orchestra.run.smithi092:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 00db0016-dfed-11ee-95c8-87774f69a715 -- ceph orch daemon add osd smithi092:vg_nvme/lv_2 2024-03-11T21:23:00.267 INFO:teuthology.orchestra.run.smithi092.stderr:Inferring config /var/lib/ceph/00db0016-dfed-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T21:23:00.694 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:23:00 smithi092 ceph-mon[37563]: Detected new or changed devices on smithi092 2024-03-11T21:23:00.694 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:23:00 smithi092 ceph-mon[37563]: from='mgr.14150 v1:172.21.15.92:0/996057664' entity='mgr.y' 2024-03-11T21:23:00.694 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:23:00 smithi092 ceph-mon[37563]: from='mgr.14150 v1:172.21.15.92:0/996057664' entity='mgr.y' 2024-03-11T21:23:00.694 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:23:00 smithi092 ceph-mon[37563]: from='mgr.14150 v1:172.21.15.92:0/996057664' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-11T21:23:00.695 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:23:00 smithi092 ceph-mon[37563]: from='mgr.14150 v1:172.21.15.92:0/996057664' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-11T21:23:00.695 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:23:00 smithi092 ceph-mon[37563]: Adjusting osd_memory_target on smithi092 to 8032M 2024-03-11T21:23:00.695 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:23:00 smithi092 ceph-mon[37563]: from='mgr.14150 v1:172.21.15.92:0/996057664' entity='mgr.y' 2024-03-11T21:23:00.695 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:23:00 smithi092 ceph-mon[37563]: from='mgr.14150 v1:172.21.15.92:0/996057664' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T21:23:00.695 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:23:00 smithi092 ceph-mon[37563]: from='mgr.14150 v1:172.21.15.92:0/996057664' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T21:23:00.695 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:23:00 smithi092 ceph-mon[37563]: from='mgr.14150 v1:172.21.15.92:0/996057664' entity='mgr.y' 2024-03-11T21:23:00.695 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:23:00 smithi092 ceph-mon[37563]: pgmap v59: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-11T21:23:00.695 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:23:00 smithi092 ceph-mon[29572]: Detected new or changed devices on smithi092 2024-03-11T21:23:00.695 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:23:00 smithi092 ceph-mon[29572]: from='mgr.14150 v1:172.21.15.92:0/996057664' entity='mgr.y' 2024-03-11T21:23:00.695 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:23:00 smithi092 ceph-mon[29572]: from='mgr.14150 v1:172.21.15.92:0/996057664' entity='mgr.y' 2024-03-11T21:23:00.695 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:23:00 smithi092 ceph-mon[29572]: from='mgr.14150 v1:172.21.15.92:0/996057664' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-11T21:23:00.695 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:23:00 smithi092 ceph-mon[29572]: from='mgr.14150 v1:172.21.15.92:0/996057664' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-11T21:23:00.695 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:23:00 smithi092 ceph-mon[29572]: Adjusting osd_memory_target on smithi092 to 8032M 2024-03-11T21:23:00.696 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:23:00 smithi092 ceph-mon[29572]: from='mgr.14150 v1:172.21.15.92:0/996057664' entity='mgr.y' 2024-03-11T21:23:00.696 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:23:00 smithi092 ceph-mon[29572]: from='mgr.14150 v1:172.21.15.92:0/996057664' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T21:23:00.696 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:23:00 smithi092 ceph-mon[29572]: from='mgr.14150 v1:172.21.15.92:0/996057664' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T21:23:00.696 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:23:00 smithi092 ceph-mon[29572]: from='mgr.14150 v1:172.21.15.92:0/996057664' entity='mgr.y' 2024-03-11T21:23:00.696 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:23:00 smithi092 ceph-mon[29572]: pgmap v59: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-11T21:23:00.828 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:23:00 smithi097 ceph-mon[33317]: Detected new or changed devices on smithi092 2024-03-11T21:23:00.828 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:23:00 smithi097 ceph-mon[33317]: from='mgr.14150 v1:172.21.15.92:0/996057664' entity='mgr.y' 2024-03-11T21:23:00.828 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:23:00 smithi097 ceph-mon[33317]: from='mgr.14150 v1:172.21.15.92:0/996057664' entity='mgr.y' 2024-03-11T21:23:00.828 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:23:00 smithi097 ceph-mon[33317]: from='mgr.14150 v1:172.21.15.92:0/996057664' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-11T21:23:00.828 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:23:00 smithi097 ceph-mon[33317]: from='mgr.14150 v1:172.21.15.92:0/996057664' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-11T21:23:00.828 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:23:00 smithi097 ceph-mon[33317]: Adjusting osd_memory_target on smithi092 to 8032M 2024-03-11T21:23:00.828 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:23:00 smithi097 ceph-mon[33317]: from='mgr.14150 v1:172.21.15.92:0/996057664' entity='mgr.y' 2024-03-11T21:23:00.828 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:23:00 smithi097 ceph-mon[33317]: from='mgr.14150 v1:172.21.15.92:0/996057664' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T21:23:00.828 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:23:00 smithi097 ceph-mon[33317]: from='mgr.14150 v1:172.21.15.92:0/996057664' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T21:23:00.828 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:23:00 smithi097 ceph-mon[33317]: from='mgr.14150 v1:172.21.15.92:0/996057664' entity='mgr.y' 2024-03-11T21:23:00.828 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:23:00 smithi097 ceph-mon[33317]: pgmap v59: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-11T21:23:02.828 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:23:02 smithi097 ceph-mon[33317]: pgmap v60: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-11T21:23:02.828 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:23:02 smithi097 ceph-mon[33317]: from='mgr.14150 v1:172.21.15.92:0/996057664' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-11T21:23:02.828 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:23:02 smithi097 ceph-mon[33317]: from='mgr.14150 v1:172.21.15.92:0/996057664' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-11T21:23:02.828 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:23:02 smithi097 ceph-mon[33317]: from='mgr.14150 v1:172.21.15.92:0/996057664' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T21:23:02.853 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:23:02 smithi092 ceph-mon[29572]: pgmap v60: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-11T21:23:02.853 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:23:02 smithi092 ceph-mon[29572]: from='mgr.14150 v1:172.21.15.92:0/996057664' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-11T21:23:02.853 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:23:02 smithi092 ceph-mon[29572]: from='mgr.14150 v1:172.21.15.92:0/996057664' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-11T21:23:02.854 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:23:02 smithi092 ceph-mon[29572]: from='mgr.14150 v1:172.21.15.92:0/996057664' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T21:23:02.854 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:23:02 smithi092 ceph-mon[37563]: pgmap v60: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-11T21:23:02.854 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:23:02 smithi092 ceph-mon[37563]: from='mgr.14150 v1:172.21.15.92:0/996057664' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-11T21:23:02.854 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:23:02 smithi092 ceph-mon[37563]: from='mgr.14150 v1:172.21.15.92:0/996057664' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-11T21:23:02.854 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:23:02 smithi092 ceph-mon[37563]: from='mgr.14150 v1:172.21.15.92:0/996057664' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T21:23:03.704 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:23:03 smithi092 ceph-mon[29572]: from='client.14268 v1:172.21.15.92:0/2658816931' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi092:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T21:23:03.705 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:23:03 smithi092 ceph-mon[37563]: from='client.14268 v1:172.21.15.92:0/2658816931' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi092:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T21:23:03.828 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:23:03 smithi097 ceph-mon[33317]: from='client.14268 v1:172.21.15.92:0/2658816931' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi092:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T21:23:04.828 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:23:04 smithi097 ceph-mon[33317]: pgmap v61: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-11T21:23:04.828 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:23:04 smithi097 ceph-mon[33317]: from='client.? v1:172.21.15.92:0/3602074246' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "f207b743-0df4-40ed-a735-950e272fea60"}]: dispatch 2024-03-11T21:23:04.828 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:23:04 smithi097 ceph-mon[33317]: from='client.34172 ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "f207b743-0df4-40ed-a735-950e272fea60"}]: dispatch 2024-03-11T21:23:04.828 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:23:04 smithi097 ceph-mon[33317]: from='client.34172 ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "f207b743-0df4-40ed-a735-950e272fea60"}]': finished 2024-03-11T21:23:04.862 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:23:04 smithi092 ceph-mon[29572]: pgmap v61: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-11T21:23:04.862 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:23:04 smithi092 ceph-mon[29572]: from='client.? v1:172.21.15.92:0/3602074246' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "f207b743-0df4-40ed-a735-950e272fea60"}]: dispatch 2024-03-11T21:23:04.862 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:23:04 smithi092 ceph-mon[29572]: from='client.34172 ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "f207b743-0df4-40ed-a735-950e272fea60"}]: dispatch 2024-03-11T21:23:04.862 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:23:04 smithi092 ceph-mon[29572]: from='client.34172 ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "f207b743-0df4-40ed-a735-950e272fea60"}]': finished 2024-03-11T21:23:04.862 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:23:04 smithi092 ceph-mon[37563]: pgmap v61: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-11T21:23:04.862 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:23:04 smithi092 ceph-mon[37563]: from='client.? v1:172.21.15.92:0/3602074246' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "f207b743-0df4-40ed-a735-950e272fea60"}]: dispatch 2024-03-11T21:23:04.862 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:23:04 smithi092 ceph-mon[37563]: from='client.34172 ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "f207b743-0df4-40ed-a735-950e272fea60"}]: dispatch 2024-03-11T21:23:04.862 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:23:04 smithi092 ceph-mon[37563]: from='client.34172 ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "f207b743-0df4-40ed-a735-950e272fea60"}]': finished 2024-03-11T21:23:05.827 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:23:05 smithi097 ceph-mon[33317]: osdmap e15: 3 total, 2 up, 3 in 2024-03-11T21:23:05.828 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:23:05 smithi097 ceph-mon[33317]: from='mgr.14150 v1:172.21.15.92:0/996057664' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-11T21:23:05.828 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:23:05 smithi097 ceph-mon[33317]: from='client.? v1:172.21.15.92:0/4256352380' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-11T21:23:05.862 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:23:05 smithi092 ceph-mon[29572]: osdmap e15: 3 total, 2 up, 3 in 2024-03-11T21:23:05.862 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:23:05 smithi092 ceph-mon[29572]: from='mgr.14150 v1:172.21.15.92:0/996057664' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-11T21:23:05.862 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:23:05 smithi092 ceph-mon[29572]: from='client.? v1:172.21.15.92:0/4256352380' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-11T21:23:05.862 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:23:05 smithi092 ceph-mon[37563]: osdmap e15: 3 total, 2 up, 3 in 2024-03-11T21:23:05.862 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:23:05 smithi092 ceph-mon[37563]: from='mgr.14150 v1:172.21.15.92:0/996057664' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-11T21:23:05.862 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:23:05 smithi092 ceph-mon[37563]: from='client.? v1:172.21.15.92:0/4256352380' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-11T21:23:06.828 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:23:06 smithi097 ceph-mon[33317]: pgmap v63: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-11T21:23:06.862 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:23:06 smithi092 ceph-mon[29572]: pgmap v63: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-11T21:23:06.862 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:23:06 smithi092 ceph-mon[37563]: pgmap v63: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-11T21:23:08.828 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:23:08 smithi097 ceph-mon[33317]: pgmap v64: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-11T21:23:08.859 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:23:08 smithi092 ceph-mon[29572]: pgmap v64: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-11T21:23:08.859 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:23:08 smithi092 ceph-mon[37563]: pgmap v64: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-11T21:23:10.828 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:23:10 smithi097 ceph-mon[33317]: pgmap v65: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-11T21:23:10.862 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:23:10 smithi092 ceph-mon[29572]: pgmap v65: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-11T21:23:10.862 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:23:10 smithi092 ceph-mon[37563]: pgmap v65: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-11T21:23:11.822 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:23:11 smithi092 ceph-mon[29572]: from='mgr.14150 v1:172.21.15.92:0/996057664' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.2"}]: dispatch 2024-03-11T21:23:11.822 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:23:11 smithi092 ceph-mon[29572]: from='mgr.14150 v1:172.21.15.92:0/996057664' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T21:23:11.822 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:23:11 smithi092 ceph-mon[37563]: from='mgr.14150 v1:172.21.15.92:0/996057664' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.2"}]: dispatch 2024-03-11T21:23:11.823 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:23:11 smithi092 ceph-mon[37563]: from='mgr.14150 v1:172.21.15.92:0/996057664' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T21:23:11.827 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:23:11 smithi097 ceph-mon[33317]: from='mgr.14150 v1:172.21.15.92:0/996057664' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.2"}]: dispatch 2024-03-11T21:23:11.827 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:23:11 smithi097 ceph-mon[33317]: from='mgr.14150 v1:172.21.15.92:0/996057664' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T21:23:12.828 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:23:12 smithi097 ceph-mon[33317]: Deploying daemon osd.2 on smithi092 2024-03-11T21:23:12.828 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:23:12 smithi097 ceph-mon[33317]: pgmap v66: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-11T21:23:12.861 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:23:12 smithi092 ceph-mon[29572]: Deploying daemon osd.2 on smithi092 2024-03-11T21:23:12.862 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:23:12 smithi092 ceph-mon[29572]: pgmap v66: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-11T21:23:12.862 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:23:12 smithi092 ceph-mon[37563]: Deploying daemon osd.2 on smithi092 2024-03-11T21:23:12.862 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:23:12 smithi092 ceph-mon[37563]: pgmap v66: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-11T21:23:14.612 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:23:14 smithi092 ceph-mon[37563]: pgmap v67: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-11T21:23:14.612 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:23:14 smithi092 ceph-mon[29572]: pgmap v67: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-11T21:23:14.827 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:23:14 smithi097 ceph-mon[33317]: pgmap v67: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-11T21:23:16.831 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:23:16 smithi097 ceph-mon[33317]: pgmap v68: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-11T21:23:16.831 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:23:16 smithi097 ceph-mon[33317]: from='mgr.14150 v1:172.21.15.92:0/996057664' entity='mgr.y' 2024-03-11T21:23:16.831 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:23:16 smithi097 ceph-mon[33317]: from='mgr.14150 v1:172.21.15.92:0/996057664' entity='mgr.y' 2024-03-11T21:23:16.831 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:23:16 smithi097 ceph-mon[33317]: from='mgr.14150 v1:172.21.15.92:0/996057664' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T21:23:16.862 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:23:16 smithi092 ceph-mon[29572]: pgmap v68: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-11T21:23:16.862 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:23:16 smithi092 ceph-mon[29572]: from='mgr.14150 v1:172.21.15.92:0/996057664' entity='mgr.y' 2024-03-11T21:23:16.862 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:23:16 smithi092 ceph-mon[29572]: from='mgr.14150 v1:172.21.15.92:0/996057664' entity='mgr.y' 2024-03-11T21:23:16.862 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:23:16 smithi092 ceph-mon[29572]: from='mgr.14150 v1:172.21.15.92:0/996057664' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T21:23:16.863 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:23:16 smithi092 ceph-mon[37563]: pgmap v68: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-11T21:23:16.863 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:23:16 smithi092 ceph-mon[37563]: from='mgr.14150 v1:172.21.15.92:0/996057664' entity='mgr.y' 2024-03-11T21:23:16.863 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:23:16 smithi092 ceph-mon[37563]: from='mgr.14150 v1:172.21.15.92:0/996057664' entity='mgr.y' 2024-03-11T21:23:16.863 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:23:16 smithi092 ceph-mon[37563]: from='mgr.14150 v1:172.21.15.92:0/996057664' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T21:23:18.827 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:23:18 smithi097 ceph-mon[33317]: pgmap v69: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-11T21:23:18.862 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:23:18 smithi092 ceph-mon[29572]: pgmap v69: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-11T21:23:18.862 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:23:18 smithi092 ceph-mon[37563]: pgmap v69: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-11T21:23:20.129 INFO:teuthology.orchestra.run.smithi092.stdout:Created osd(s) 2 on host 'smithi092' 2024-03-11T21:23:20.362 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:23:20 smithi092 ceph-mon[37563]: from='mgr.14150 v1:172.21.15.92:0/996057664' entity='mgr.y' 2024-03-11T21:23:20.362 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:23:20 smithi092 ceph-mon[37563]: from='mgr.14150 v1:172.21.15.92:0/996057664' entity='mgr.y' 2024-03-11T21:23:20.362 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:23:20 smithi092 ceph-mon[37563]: from='mgr.14150 v1:172.21.15.92:0/996057664' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T21:23:20.362 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:23:20 smithi092 ceph-mon[37563]: from='mgr.14150 v1:172.21.15.92:0/996057664' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T21:23:20.362 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:23:20 smithi092 ceph-mon[37563]: from='mgr.14150 v1:172.21.15.92:0/996057664' entity='mgr.y' 2024-03-11T21:23:20.362 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:23:20 smithi092 ceph-mon[37563]: from='osd.2 v1:172.21.15.92:6809/760825152' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-03-11T21:23:20.362 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:23:20 smithi092 ceph-mon[37563]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-03-11T21:23:20.362 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:23:20 smithi092 ceph-mon[37563]: pgmap v70: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-11T21:23:20.363 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:23:20 smithi092 ceph-mon[37563]: from='mgr.14150 v1:172.21.15.92:0/996057664' entity='mgr.y' 2024-03-11T21:23:20.363 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:23:20 smithi092 ceph-mon[37563]: from='mgr.14150 v1:172.21.15.92:0/996057664' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T21:23:20.363 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:23:20 smithi092 ceph-mon[37563]: from='mgr.14150 v1:172.21.15.92:0/996057664' entity='mgr.y' 2024-03-11T21:23:20.363 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:23:20 smithi092 ceph-mon[29572]: from='mgr.14150 v1:172.21.15.92:0/996057664' entity='mgr.y' 2024-03-11T21:23:20.363 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:23:20 smithi092 ceph-mon[29572]: from='mgr.14150 v1:172.21.15.92:0/996057664' entity='mgr.y' 2024-03-11T21:23:20.363 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:23:20 smithi092 ceph-mon[29572]: from='mgr.14150 v1:172.21.15.92:0/996057664' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T21:23:20.363 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:23:20 smithi092 ceph-mon[29572]: from='mgr.14150 v1:172.21.15.92:0/996057664' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T21:23:20.363 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:23:20 smithi092 ceph-mon[29572]: from='mgr.14150 v1:172.21.15.92:0/996057664' entity='mgr.y' 2024-03-11T21:23:20.363 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:23:20 smithi092 ceph-mon[29572]: from='osd.2 v1:172.21.15.92:6809/760825152' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-03-11T21:23:20.363 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:23:20 smithi092 ceph-mon[29572]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-03-11T21:23:20.363 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:23:20 smithi092 ceph-mon[29572]: pgmap v70: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-11T21:23:20.364 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:23:20 smithi092 ceph-mon[29572]: from='mgr.14150 v1:172.21.15.92:0/996057664' entity='mgr.y' 2024-03-11T21:23:20.364 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:23:20 smithi092 ceph-mon[29572]: from='mgr.14150 v1:172.21.15.92:0/996057664' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T21:23:20.364 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:23:20 smithi092 ceph-mon[29572]: from='mgr.14150 v1:172.21.15.92:0/996057664' entity='mgr.y' 2024-03-11T21:23:20.578 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:23:20 smithi097 ceph-mon[33317]: from='mgr.14150 v1:172.21.15.92:0/996057664' entity='mgr.y' 2024-03-11T21:23:20.578 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:23:20 smithi097 ceph-mon[33317]: from='mgr.14150 v1:172.21.15.92:0/996057664' entity='mgr.y' 2024-03-11T21:23:20.578 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:23:20 smithi097 ceph-mon[33317]: from='mgr.14150 v1:172.21.15.92:0/996057664' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T21:23:20.578 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:23:20 smithi097 ceph-mon[33317]: from='mgr.14150 v1:172.21.15.92:0/996057664' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T21:23:20.578 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:23:20 smithi097 ceph-mon[33317]: from='mgr.14150 v1:172.21.15.92:0/996057664' entity='mgr.y' 2024-03-11T21:23:20.578 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:23:20 smithi097 ceph-mon[33317]: from='osd.2 v1:172.21.15.92:6809/760825152' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-03-11T21:23:20.578 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:23:20 smithi097 ceph-mon[33317]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-03-11T21:23:20.578 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:23:20 smithi097 ceph-mon[33317]: pgmap v70: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-11T21:23:20.578 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:23:20 smithi097 ceph-mon[33317]: from='mgr.14150 v1:172.21.15.92:0/996057664' entity='mgr.y' 2024-03-11T21:23:20.578 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:23:20 smithi097 ceph-mon[33317]: from='mgr.14150 v1:172.21.15.92:0/996057664' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T21:23:20.578 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:23:20 smithi097 ceph-mon[33317]: from='mgr.14150 v1:172.21.15.92:0/996057664' entity='mgr.y' 2024-03-11T21:23:20.705 DEBUG:teuthology.orchestra.run.smithi092:osd.2> sudo journalctl -f -n 0 -u ceph-00db0016-dfed-11ee-95c8-87774f69a715@osd.2.service 2024-03-11T21:23:20.707 INFO:tasks.cephadm:Deploying osd.3 on smithi092 with /dev/vg_nvme/lv_1... 2024-03-11T21:23:20.707 DEBUG:teuthology.orchestra.run.smithi092:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 00db0016-dfed-11ee-95c8-87774f69a715 -- ceph-volume lvm zap /dev/vg_nvme/lv_1 2024-03-11T21:23:21.267 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:23:21 smithi092 ceph-mon[29572]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]': finished 2024-03-11T21:23:21.267 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:23:21 smithi092 ceph-mon[29572]: osdmap e16: 3 total, 2 up, 3 in 2024-03-11T21:23:21.267 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:23:21 smithi092 ceph-mon[29572]: from='mgr.14150 v1:172.21.15.92:0/996057664' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-11T21:23:21.267 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:23:21 smithi092 ceph-mon[29572]: from='osd.2 v1:172.21.15.92:6809/760825152' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi092", "root=default"]}]: dispatch 2024-03-11T21:23:21.267 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:23:21 smithi092 ceph-mon[29572]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi092", "root=default"]}]: dispatch 2024-03-11T21:23:21.268 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:23:21 smithi092 ceph-mon[37563]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]': finished 2024-03-11T21:23:21.268 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:23:21 smithi092 ceph-mon[37563]: osdmap e16: 3 total, 2 up, 3 in 2024-03-11T21:23:21.268 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:23:21 smithi092 ceph-mon[37563]: from='mgr.14150 v1:172.21.15.92:0/996057664' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-11T21:23:21.268 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:23:21 smithi092 ceph-mon[37563]: from='osd.2 v1:172.21.15.92:6809/760825152' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi092", "root=default"]}]: dispatch 2024-03-11T21:23:21.268 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:23:21 smithi092 ceph-mon[37563]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi092", "root=default"]}]: dispatch 2024-03-11T21:23:21.268 INFO:journalctl@ceph.osd.2.smithi092.stdout:Mar 11 21:23:21 smithi092 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd-2[53445]: 2024-03-11T21:23:21.252+0000 7f80ca4ab700 -1 osd.2 0 waiting for initial osdmap 2024-03-11T21:23:21.268 INFO:journalctl@ceph.osd.2.smithi092.stdout:Mar 11 21:23:21 smithi092 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd-2[53445]: 2024-03-11T21:23:21.255+0000 7f80c629e700 -1 osd.2 17 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-11T21:23:21.578 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:23:21 smithi097 ceph-mon[33317]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]': finished 2024-03-11T21:23:21.578 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:23:21 smithi097 ceph-mon[33317]: osdmap e16: 3 total, 2 up, 3 in 2024-03-11T21:23:21.578 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:23:21 smithi097 ceph-mon[33317]: from='mgr.14150 v1:172.21.15.92:0/996057664' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-11T21:23:21.578 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:23:21 smithi097 ceph-mon[33317]: from='osd.2 v1:172.21.15.92:6809/760825152' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi092", "root=default"]}]: dispatch 2024-03-11T21:23:21.578 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:23:21 smithi097 ceph-mon[33317]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi092", "root=default"]}]: dispatch 2024-03-11T21:23:22.307 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:23:22 smithi092 ceph-mon[37563]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi092", "root=default"]}]': finished 2024-03-11T21:23:22.307 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:23:22 smithi092 ceph-mon[37563]: osdmap e17: 3 total, 2 up, 3 in 2024-03-11T21:23:22.307 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:23:22 smithi092 ceph-mon[37563]: from='mgr.14150 v1:172.21.15.92:0/996057664' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-11T21:23:22.307 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:23:22 smithi092 ceph-mon[37563]: from='mgr.14150 v1:172.21.15.92:0/996057664' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-11T21:23:22.307 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:23:22 smithi092 ceph-mon[37563]: pgmap v73: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-11T21:23:22.308 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:23:22 smithi092 ceph-mon[29572]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi092", "root=default"]}]': finished 2024-03-11T21:23:22.308 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:23:22 smithi092 ceph-mon[29572]: osdmap e17: 3 total, 2 up, 3 in 2024-03-11T21:23:22.308 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:23:22 smithi092 ceph-mon[29572]: from='mgr.14150 v1:172.21.15.92:0/996057664' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-11T21:23:22.308 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:23:22 smithi092 ceph-mon[29572]: from='mgr.14150 v1:172.21.15.92:0/996057664' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-11T21:23:22.308 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:23:22 smithi092 ceph-mon[29572]: pgmap v73: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-11T21:23:22.412 INFO:teuthology.orchestra.run.smithi092.stderr:Inferring config /var/lib/ceph/00db0016-dfed-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T21:23:22.580 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:23:22 smithi097 ceph-mon[33317]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi092", "root=default"]}]': finished 2024-03-11T21:23:22.580 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:23:22 smithi097 ceph-mon[33317]: osdmap e17: 3 total, 2 up, 3 in 2024-03-11T21:23:22.580 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:23:22 smithi097 ceph-mon[33317]: from='mgr.14150 v1:172.21.15.92:0/996057664' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-11T21:23:22.580 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:23:22 smithi097 ceph-mon[33317]: from='mgr.14150 v1:172.21.15.92:0/996057664' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-11T21:23:22.580 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:23:22 smithi097 ceph-mon[33317]: pgmap v73: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-11T21:23:23.573 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:23:23 smithi092 ceph-mon[29572]: purged_snaps scrub starts 2024-03-11T21:23:23.574 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:23:23 smithi092 ceph-mon[29572]: purged_snaps scrub ok 2024-03-11T21:23:23.574 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:23:23 smithi092 ceph-mon[29572]: from='mgr.14150 v1:172.21.15.92:0/996057664' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-11T21:23:23.574 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:23:23 smithi092 ceph-mon[29572]: osd.2 v1:172.21.15.92:6809/760825152 boot 2024-03-11T21:23:23.574 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:23:23 smithi092 ceph-mon[29572]: osdmap e18: 3 total, 3 up, 3 in 2024-03-11T21:23:23.574 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:23:23 smithi092 ceph-mon[29572]: from='mgr.14150 v1:172.21.15.92:0/996057664' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-11T21:23:23.578 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:23:23 smithi092 ceph-mon[37563]: purged_snaps scrub starts 2024-03-11T21:23:23.578 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:23:23 smithi092 ceph-mon[37563]: purged_snaps scrub ok 2024-03-11T21:23:23.578 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:23:23 smithi092 ceph-mon[37563]: from='mgr.14150 v1:172.21.15.92:0/996057664' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-11T21:23:23.578 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:23:23 smithi092 ceph-mon[37563]: osd.2 v1:172.21.15.92:6809/760825152 boot 2024-03-11T21:23:23.578 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:23:23 smithi092 ceph-mon[37563]: osdmap e18: 3 total, 3 up, 3 in 2024-03-11T21:23:23.578 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:23:23 smithi092 ceph-mon[37563]: from='mgr.14150 v1:172.21.15.92:0/996057664' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-11T21:23:23.579 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:23:23 smithi097 ceph-mon[33317]: purged_snaps scrub starts 2024-03-11T21:23:23.579 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:23:23 smithi097 ceph-mon[33317]: purged_snaps scrub ok 2024-03-11T21:23:23.579 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:23:23 smithi097 ceph-mon[33317]: from='mgr.14150 v1:172.21.15.92:0/996057664' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-11T21:23:23.579 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:23:23 smithi097 ceph-mon[33317]: osd.2 v1:172.21.15.92:6809/760825152 boot 2024-03-11T21:23:23.579 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:23:23 smithi097 ceph-mon[33317]: osdmap e18: 3 total, 3 up, 3 in 2024-03-11T21:23:23.579 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:23:23 smithi097 ceph-mon[33317]: from='mgr.14150 v1:172.21.15.92:0/996057664' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-11T21:23:24.578 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:23:24 smithi097 ceph-mon[33317]: osdmap e19: 3 total, 3 up, 3 in 2024-03-11T21:23:24.578 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:23:24 smithi097 ceph-mon[33317]: pgmap v76: 0 pgs: ; 0 B data, 80 MiB used, 268 GiB / 268 GiB avail 2024-03-11T21:23:24.578 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:23:24 smithi097 ceph-mon[33317]: from='mgr.14150 v1:172.21.15.92:0/996057664' entity='mgr.y' cmd=[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]: dispatch 2024-03-11T21:23:24.578 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:23:24 smithi097 ceph-mon[33317]: from='mgr.14150 v1:172.21.15.92:0/996057664' entity='mgr.y' 2024-03-11T21:23:24.578 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:23:24 smithi097 ceph-mon[33317]: from='mgr.14150 v1:172.21.15.92:0/996057664' entity='mgr.y' 2024-03-11T21:23:24.578 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:23:24 smithi097 ceph-mon[33317]: from='mgr.14150 v1:172.21.15.92:0/996057664' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-11T21:23:24.578 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:23:24 smithi097 ceph-mon[33317]: from='mgr.14150 v1:172.21.15.92:0/996057664' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-11T21:23:24.578 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:23:24 smithi097 ceph-mon[33317]: from='mgr.14150 v1:172.21.15.92:0/996057664' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-11T21:23:24.579 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:23:24 smithi097 ceph-mon[33317]: from='mgr.14150 v1:172.21.15.92:0/996057664' entity='mgr.y' 2024-03-11T21:23:24.579 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:23:24 smithi097 ceph-mon[33317]: from='mgr.14150 v1:172.21.15.92:0/996057664' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T21:23:24.579 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:23:24 smithi097 ceph-mon[33317]: from='mgr.14150 v1:172.21.15.92:0/996057664' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T21:23:24.579 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:23:24 smithi097 ceph-mon[33317]: from='mgr.14150 v1:172.21.15.92:0/996057664' entity='mgr.y' 2024-03-11T21:23:24.612 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:23:24 smithi092 ceph-mon[37563]: osdmap e19: 3 total, 3 up, 3 in 2024-03-11T21:23:24.612 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:23:24 smithi092 ceph-mon[37563]: pgmap v76: 0 pgs: ; 0 B data, 80 MiB used, 268 GiB / 268 GiB avail 2024-03-11T21:23:24.612 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:23:24 smithi092 ceph-mon[37563]: from='mgr.14150 v1:172.21.15.92:0/996057664' entity='mgr.y' cmd=[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]: dispatch 2024-03-11T21:23:24.612 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:23:24 smithi092 ceph-mon[37563]: from='mgr.14150 v1:172.21.15.92:0/996057664' entity='mgr.y' 2024-03-11T21:23:24.612 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:23:24 smithi092 ceph-mon[37563]: from='mgr.14150 v1:172.21.15.92:0/996057664' entity='mgr.y' 2024-03-11T21:23:24.612 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:23:24 smithi092 ceph-mon[37563]: from='mgr.14150 v1:172.21.15.92:0/996057664' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-11T21:23:24.612 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:23:24 smithi092 ceph-mon[37563]: from='mgr.14150 v1:172.21.15.92:0/996057664' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-11T21:23:24.612 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:23:24 smithi092 ceph-mon[37563]: from='mgr.14150 v1:172.21.15.92:0/996057664' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-11T21:23:24.612 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:23:24 smithi092 ceph-mon[37563]: from='mgr.14150 v1:172.21.15.92:0/996057664' entity='mgr.y' 2024-03-11T21:23:24.612 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:23:24 smithi092 ceph-mon[37563]: from='mgr.14150 v1:172.21.15.92:0/996057664' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T21:23:24.612 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:23:24 smithi092 ceph-mon[37563]: from='mgr.14150 v1:172.21.15.92:0/996057664' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T21:23:24.612 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:23:24 smithi092 ceph-mon[37563]: from='mgr.14150 v1:172.21.15.92:0/996057664' entity='mgr.y' 2024-03-11T21:23:24.613 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:23:24 smithi092 ceph-mon[29572]: osdmap e19: 3 total, 3 up, 3 in 2024-03-11T21:23:24.613 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:23:24 smithi092 ceph-mon[29572]: pgmap v76: 0 pgs: ; 0 B data, 80 MiB used, 268 GiB / 268 GiB avail 2024-03-11T21:23:24.613 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:23:24 smithi092 ceph-mon[29572]: from='mgr.14150 v1:172.21.15.92:0/996057664' entity='mgr.y' cmd=[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]: dispatch 2024-03-11T21:23:24.613 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:23:24 smithi092 ceph-mon[29572]: from='mgr.14150 v1:172.21.15.92:0/996057664' entity='mgr.y' 2024-03-11T21:23:24.613 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:23:24 smithi092 ceph-mon[29572]: from='mgr.14150 v1:172.21.15.92:0/996057664' entity='mgr.y' 2024-03-11T21:23:24.613 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:23:24 smithi092 ceph-mon[29572]: from='mgr.14150 v1:172.21.15.92:0/996057664' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-11T21:23:24.613 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:23:24 smithi092 ceph-mon[29572]: from='mgr.14150 v1:172.21.15.92:0/996057664' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-11T21:23:24.613 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:23:24 smithi092 ceph-mon[29572]: from='mgr.14150 v1:172.21.15.92:0/996057664' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-11T21:23:24.613 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:23:24 smithi092 ceph-mon[29572]: from='mgr.14150 v1:172.21.15.92:0/996057664' entity='mgr.y' 2024-03-11T21:23:24.613 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:23:24 smithi092 ceph-mon[29572]: from='mgr.14150 v1:172.21.15.92:0/996057664' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T21:23:24.613 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:23:24 smithi092 ceph-mon[29572]: from='mgr.14150 v1:172.21.15.92:0/996057664' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T21:23:24.613 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:23:24 smithi092 ceph-mon[29572]: from='mgr.14150 v1:172.21.15.92:0/996057664' entity='mgr.y' 2024-03-11T21:23:24.815 INFO:teuthology.orchestra.run.smithi092.stderr:--> Zapping: /dev/vg_nvme/lv_1 2024-03-11T21:23:24.816 INFO:teuthology.orchestra.run.smithi092.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_1 bs=1M count=10 conv=fsync 2024-03-11T21:23:24.816 INFO:teuthology.orchestra.run.smithi092.stderr: stderr: 10+0 records in 2024-03-11T21:23:24.816 INFO:teuthology.orchestra.run.smithi092.stderr:10+0 records out 2024-03-11T21:23:24.816 INFO:teuthology.orchestra.run.smithi092.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0124344 s, 843 MB/s 2024-03-11T21:23:24.816 INFO:teuthology.orchestra.run.smithi092.stderr:--> Zapping successful for: 2024-03-11T21:23:25.410 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:23:25 smithi092 ceph-mon[37563]: Detected new or changed devices on smithi092 2024-03-11T21:23:25.410 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:23:25 smithi092 ceph-mon[37563]: Adjusting osd_memory_target on smithi092 to 5354M 2024-03-11T21:23:25.411 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:23:25 smithi092 ceph-mon[37563]: from='mgr.14150 v1:172.21.15.92:0/996057664' entity='mgr.y' cmd='[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]': finished 2024-03-11T21:23:25.411 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:23:25 smithi092 ceph-mon[37563]: osdmap e20: 3 total, 3 up, 3 in 2024-03-11T21:23:25.411 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:23:25 smithi092 ceph-mon[37563]: from='mgr.14150 v1:172.21.15.92:0/996057664' entity='mgr.y' cmd=[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]: dispatch 2024-03-11T21:23:25.411 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:23:25 smithi092 ceph-mon[29572]: Detected new or changed devices on smithi092 2024-03-11T21:23:25.411 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:23:25 smithi092 ceph-mon[29572]: Adjusting osd_memory_target on smithi092 to 5354M 2024-03-11T21:23:25.411 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:23:25 smithi092 ceph-mon[29572]: from='mgr.14150 v1:172.21.15.92:0/996057664' entity='mgr.y' cmd='[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]': finished 2024-03-11T21:23:25.411 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:23:25 smithi092 ceph-mon[29572]: osdmap e20: 3 total, 3 up, 3 in 2024-03-11T21:23:25.411 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:23:25 smithi092 ceph-mon[29572]: from='mgr.14150 v1:172.21.15.92:0/996057664' entity='mgr.y' cmd=[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]: dispatch 2024-03-11T21:23:25.436 DEBUG:teuthology.orchestra.run.smithi092:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 00db0016-dfed-11ee-95c8-87774f69a715 -- ceph orch daemon add osd smithi092:vg_nvme/lv_1 2024-03-11T21:23:25.577 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:23:25 smithi097 ceph-mon[33317]: Detected new or changed devices on smithi092 2024-03-11T21:23:25.578 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:23:25 smithi097 ceph-mon[33317]: Adjusting osd_memory_target on smithi092 to 5354M 2024-03-11T21:23:25.578 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:23:25 smithi097 ceph-mon[33317]: from='mgr.14150 v1:172.21.15.92:0/996057664' entity='mgr.y' cmd='[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]': finished 2024-03-11T21:23:25.578 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:23:25 smithi097 ceph-mon[33317]: osdmap e20: 3 total, 3 up, 3 in 2024-03-11T21:23:25.578 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:23:25 smithi097 ceph-mon[33317]: from='mgr.14150 v1:172.21.15.92:0/996057664' entity='mgr.y' cmd=[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]: dispatch 2024-03-11T21:23:25.681 INFO:teuthology.orchestra.run.smithi092.stderr:Inferring config /var/lib/ceph/00db0016-dfed-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T21:23:25.803 INFO:journalctl@ceph.osd.0.smithi092.stdout:Mar 11 21:23:25 smithi092 sudo[57543]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-03-11T21:23:25.803 INFO:journalctl@ceph.osd.0.smithi092.stdout:Mar 11 21:23:25 smithi092 sudo[57543]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-03-11T21:23:25.803 INFO:journalctl@ceph.osd.0.smithi092.stdout:Mar 11 21:23:25 smithi092 sudo[57543]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-03-11T21:23:25.803 INFO:journalctl@ceph.osd.0.smithi092.stdout:Mar 11 21:23:25 smithi092 sudo[57543]: pam_unix(sudo:session): session closed for user root 2024-03-11T21:23:26.112 INFO:journalctl@ceph.osd.0.smithi092.stdout:Mar 11 21:23:25 smithi092 sudo[57602]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-03-11T21:23:26.112 INFO:journalctl@ceph.osd.0.smithi092.stdout:Mar 11 21:23:25 smithi092 sudo[57602]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-03-11T21:23:26.112 INFO:journalctl@ceph.osd.0.smithi092.stdout:Mar 11 21:23:25 smithi092 sudo[57602]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-03-11T21:23:26.112 INFO:journalctl@ceph.osd.0.smithi092.stdout:Mar 11 21:23:25 smithi092 sudo[57602]: pam_unix(sudo:session): session closed for user root 2024-03-11T21:23:26.392 INFO:journalctl@ceph.osd.1.smithi092.stdout:Mar 11 21:23:26 smithi092 sudo[57623]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-03-11T21:23:26.392 INFO:journalctl@ceph.osd.1.smithi092.stdout:Mar 11 21:23:26 smithi092 sudo[57623]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-03-11T21:23:26.392 INFO:journalctl@ceph.osd.1.smithi092.stdout:Mar 11 21:23:26 smithi092 sudo[57623]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-03-11T21:23:26.392 INFO:journalctl@ceph.osd.1.smithi092.stdout:Mar 11 21:23:26 smithi092 sudo[57623]: pam_unix(sudo:session): session closed for user root 2024-03-11T21:23:26.392 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:23:26 smithi092 ceph-mon[29572]: from='mgr.14150 v1:172.21.15.92:0/996057664' entity='mgr.y' cmd='[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]': finished 2024-03-11T21:23:26.392 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:23:26 smithi092 ceph-mon[29572]: osdmap e21: 3 total, 3 up, 3 in 2024-03-11T21:23:26.392 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:23:26 smithi092 ceph-mon[29572]: pgmap v79: 1 pgs: 1 unknown; 0 B data, 80 MiB used, 268 GiB / 268 GiB avail 2024-03-11T21:23:26.392 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:23:26 smithi092 ceph-mon[29572]: from='mgr.14150 v1:172.21.15.92:0/996057664' entity='mgr.y' 2024-03-11T21:23:26.393 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:23:26 smithi092 ceph-mon[37563]: from='mgr.14150 v1:172.21.15.92:0/996057664' entity='mgr.y' cmd='[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]': finished 2024-03-11T21:23:26.393 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:23:26 smithi092 ceph-mon[37563]: osdmap e21: 3 total, 3 up, 3 in 2024-03-11T21:23:26.393 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:23:26 smithi092 ceph-mon[37563]: pgmap v79: 1 pgs: 1 unknown; 0 B data, 80 MiB used, 268 GiB / 268 GiB avail 2024-03-11T21:23:26.393 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:23:26 smithi092 ceph-mon[37563]: from='mgr.14150 v1:172.21.15.92:0/996057664' entity='mgr.y' 2024-03-11T21:23:26.578 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:23:26 smithi097 ceph-mon[33317]: from='mgr.14150 v1:172.21.15.92:0/996057664' entity='mgr.y' cmd='[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]': finished 2024-03-11T21:23:26.578 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:23:26 smithi097 ceph-mon[33317]: osdmap e21: 3 total, 3 up, 3 in 2024-03-11T21:23:26.578 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:23:26 smithi097 ceph-mon[33317]: pgmap v79: 1 pgs: 1 unknown; 0 B data, 80 MiB used, 268 GiB / 268 GiB avail 2024-03-11T21:23:26.578 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:23:26 smithi097 ceph-mon[33317]: from='mgr.14150 v1:172.21.15.92:0/996057664' entity='mgr.y' 2024-03-11T21:23:26.653 INFO:journalctl@ceph.osd.1.smithi092.stdout:Mar 11 21:23:26 smithi092 sudo[57629]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-03-11T21:23:26.654 INFO:journalctl@ceph.osd.1.smithi092.stdout:Mar 11 21:23:26 smithi092 sudo[57629]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-03-11T21:23:26.654 INFO:journalctl@ceph.osd.1.smithi092.stdout:Mar 11 21:23:26 smithi092 sudo[57629]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-03-11T21:23:26.654 INFO:journalctl@ceph.osd.1.smithi092.stdout:Mar 11 21:23:26 smithi092 sudo[57629]: pam_unix(sudo:session): session closed for user root 2024-03-11T21:23:26.654 INFO:journalctl@ceph.osd.2.smithi092.stdout:Mar 11 21:23:26 smithi092 sudo[57632]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-03-11T21:23:26.654 INFO:journalctl@ceph.osd.2.smithi092.stdout:Mar 11 21:23:26 smithi092 sudo[57632]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-03-11T21:23:26.654 INFO:journalctl@ceph.osd.2.smithi092.stdout:Mar 11 21:23:26 smithi092 sudo[57632]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-03-11T21:23:26.654 INFO:journalctl@ceph.osd.2.smithi092.stdout:Mar 11 21:23:26 smithi092 sudo[57632]: pam_unix(sudo:session): session closed for user root 2024-03-11T21:23:26.916 INFO:journalctl@ceph.osd.2.smithi092.stdout:Mar 11 21:23:26 smithi092 sudo[57655]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-03-11T21:23:26.916 INFO:journalctl@ceph.osd.2.smithi092.stdout:Mar 11 21:23:26 smithi092 sudo[57655]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-03-11T21:23:26.916 INFO:journalctl@ceph.osd.2.smithi092.stdout:Mar 11 21:23:26 smithi092 sudo[57655]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-03-11T21:23:26.916 INFO:journalctl@ceph.osd.2.smithi092.stdout:Mar 11 21:23:26 smithi092 sudo[57655]: pam_unix(sudo:session): session closed for user root 2024-03-11T21:23:27.167 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:23:27 smithi092 sudo[57666]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-03-11T21:23:27.168 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:23:27 smithi092 sudo[57666]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-03-11T21:23:27.168 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:23:27 smithi092 sudo[57666]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-03-11T21:23:27.168 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:23:27 smithi092 sudo[57666]: pam_unix(sudo:session): session closed for user root 2024-03-11T21:23:27.417 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:23:27 smithi092 sudo[57689]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-03-11T21:23:27.687 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:23:27 smithi097 ceph-mon[33317]: osdmap e22: 3 total, 3 up, 3 in 2024-03-11T21:23:27.687 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:23:27 smithi097 ceph-mon[33317]: from='mgr.14150 v1:172.21.15.92:0/996057664' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-11T21:23:27.687 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:23:27 smithi097 ceph-mon[33317]: from='mgr.14150 v1:172.21.15.92:0/996057664' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-11T21:23:27.687 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:23:27 smithi097 ceph-mon[33317]: from='mgr.14150 v1:172.21.15.92:0/996057664' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-11T21:23:27.687 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:23:27 smithi097 sudo[35876]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-03-11T21:23:27.688 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:23:27 smithi097 sudo[35876]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-03-11T21:23:27.688 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:23:27 smithi097 sudo[35876]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-03-11T21:23:27.688 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:23:27 smithi097 sudo[35876]: pam_unix(sudo:session): session closed for user root 2024-03-11T21:23:27.862 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:23:27 smithi092 sudo[57689]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-03-11T21:23:27.862 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:23:27 smithi092 sudo[57689]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-03-11T21:23:27.862 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:23:27 smithi092 sudo[57689]: pam_unix(sudo:session): session closed for user root 2024-03-11T21:23:27.862 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:23:27 smithi092 ceph-mon[29572]: osdmap e22: 3 total, 3 up, 3 in 2024-03-11T21:23:27.862 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:23:27 smithi092 ceph-mon[29572]: from='mgr.14150 v1:172.21.15.92:0/996057664' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-11T21:23:27.862 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:23:27 smithi092 ceph-mon[29572]: from='mgr.14150 v1:172.21.15.92:0/996057664' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-11T21:23:27.862 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:23:27 smithi092 ceph-mon[29572]: from='mgr.14150 v1:172.21.15.92:0/996057664' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-11T21:23:27.863 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:23:27 smithi092 ceph-mon[37563]: osdmap e22: 3 total, 3 up, 3 in 2024-03-11T21:23:27.863 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:23:27 smithi092 ceph-mon[37563]: from='mgr.14150 v1:172.21.15.92:0/996057664' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-11T21:23:27.863 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:23:27 smithi092 ceph-mon[37563]: from='mgr.14150 v1:172.21.15.92:0/996057664' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-11T21:23:27.863 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:23:27 smithi092 ceph-mon[37563]: from='mgr.14150 v1:172.21.15.92:0/996057664' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-11T21:23:28.078 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:23:27 smithi097 sudo[35879]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-03-11T21:23:28.078 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:23:27 smithi097 sudo[35879]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-03-11T21:23:28.078 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:23:27 smithi097 sudo[35879]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-03-11T21:23:28.078 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:23:27 smithi097 sudo[35879]: pam_unix(sudo:session): session closed for user root 2024-03-11T21:23:28.360 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:23:28 smithi092 sudo[57819]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-03-11T21:23:28.361 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:23:28 smithi092 sudo[57819]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-03-11T21:23:28.361 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:23:28 smithi092 sudo[57819]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-03-11T21:23:28.361 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:23:28 smithi092 sudo[57819]: pam_unix(sudo:session): session closed for user root 2024-03-11T21:23:28.612 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:23:28 smithi092 ceph-mon[29572]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-03-11T21:23:28.612 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:23:28 smithi092 ceph-mon[29572]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-03-11T21:23:28.612 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:23:28 smithi092 ceph-mon[29572]: mgrmap e15: y(active, since 2m), standbys: x 2024-03-11T21:23:28.612 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:23:28 smithi092 ceph-mon[29572]: from='mgr.14150 v1:172.21.15.92:0/996057664' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-11T21:23:28.612 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:23:28 smithi092 ceph-mon[29572]: from='mgr.14150 v1:172.21.15.92:0/996057664' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-11T21:23:28.612 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:23:28 smithi092 ceph-mon[29572]: from='mgr.14150 v1:172.21.15.92:0/996057664' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-11T21:23:28.612 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:23:28 smithi092 ceph-mon[29572]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-03-11T21:23:28.612 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:23:28 smithi092 ceph-mon[29572]: from='client.34199 v1:172.21.15.92:0/3620294921' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi092:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T21:23:28.613 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:23:28 smithi092 ceph-mon[29572]: from='mgr.14150 v1:172.21.15.92:0/996057664' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-11T21:23:28.613 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:23:28 smithi092 ceph-mon[29572]: from='mgr.14150 v1:172.21.15.92:0/996057664' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-11T21:23:28.613 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:23:28 smithi092 ceph-mon[29572]: from='mgr.14150 v1:172.21.15.92:0/996057664' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T21:23:28.613 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:23:28 smithi092 ceph-mon[29572]: pgmap v81: 1 pgs: 1 unknown; 0 B data, 80 MiB used, 268 GiB / 268 GiB avail 2024-03-11T21:23:28.613 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:23:28 smithi092 ceph-mon[29572]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-03-11T21:23:28.613 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:23:28 smithi092 ceph-mon[29572]: from='mgr.14150 v1:172.21.15.92:0/996057664' entity='mgr.y' 2024-03-11T21:23:28.613 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:23:28 smithi092 ceph-mon[29572]: from='mgr.14150 v1:172.21.15.92:0/996057664' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-11T21:23:28.613 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:23:28 smithi092 ceph-mon[29572]: from='mgr.14150 v1:172.21.15.92:0/996057664' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-11T21:23:28.613 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:23:28 smithi092 ceph-mon[29572]: from='mgr.14150 v1:172.21.15.92:0/996057664' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-11T21:23:28.613 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:23:28 smithi092 sudo[57834]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-03-11T21:23:28.613 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:23:28 smithi092 sudo[57834]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-03-11T21:23:28.613 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:23:28 smithi092 sudo[57834]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-03-11T21:23:28.614 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:23:28 smithi092 sudo[57834]: pam_unix(sudo:session): session closed for user root 2024-03-11T21:23:28.614 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:23:28 smithi092 ceph-mon[37563]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-03-11T21:23:28.614 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:23:28 smithi092 ceph-mon[37563]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-03-11T21:23:28.614 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:23:28 smithi092 ceph-mon[37563]: mgrmap e15: y(active, since 2m), standbys: x 2024-03-11T21:23:28.614 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:23:28 smithi092 ceph-mon[37563]: from='mgr.14150 v1:172.21.15.92:0/996057664' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-11T21:23:28.614 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:23:28 smithi092 ceph-mon[37563]: from='mgr.14150 v1:172.21.15.92:0/996057664' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-11T21:23:28.614 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:23:28 smithi092 ceph-mon[37563]: from='mgr.14150 v1:172.21.15.92:0/996057664' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-11T21:23:28.614 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:23:28 smithi092 ceph-mon[37563]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-03-11T21:23:28.614 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:23:28 smithi092 ceph-mon[37563]: from='client.34199 v1:172.21.15.92:0/3620294921' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi092:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T21:23:28.614 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:23:28 smithi092 ceph-mon[37563]: from='mgr.14150 v1:172.21.15.92:0/996057664' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-11T21:23:28.614 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:23:28 smithi092 ceph-mon[37563]: from='mgr.14150 v1:172.21.15.92:0/996057664' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-11T21:23:28.614 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:23:28 smithi092 ceph-mon[37563]: from='mgr.14150 v1:172.21.15.92:0/996057664' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T21:23:28.614 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:23:28 smithi092 ceph-mon[37563]: pgmap v81: 1 pgs: 1 unknown; 0 B data, 80 MiB used, 268 GiB / 268 GiB avail 2024-03-11T21:23:28.614 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:23:28 smithi092 ceph-mon[37563]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-03-11T21:23:28.615 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:23:28 smithi092 ceph-mon[37563]: from='mgr.14150 v1:172.21.15.92:0/996057664' entity='mgr.y' 2024-03-11T21:23:28.615 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:23:28 smithi092 ceph-mon[37563]: from='mgr.14150 v1:172.21.15.92:0/996057664' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-11T21:23:28.615 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:23:28 smithi092 ceph-mon[37563]: from='mgr.14150 v1:172.21.15.92:0/996057664' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-11T21:23:28.615 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:23:28 smithi092 ceph-mon[37563]: from='mgr.14150 v1:172.21.15.92:0/996057664' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-11T21:23:28.828 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:23:28 smithi097 ceph-mon[33317]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-03-11T21:23:28.828 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:23:28 smithi097 ceph-mon[33317]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-03-11T21:23:28.828 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:23:28 smithi097 ceph-mon[33317]: mgrmap e15: y(active, since 2m), standbys: x 2024-03-11T21:23:28.828 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:23:28 smithi097 ceph-mon[33317]: from='mgr.14150 v1:172.21.15.92:0/996057664' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-11T21:23:28.828 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:23:28 smithi097 ceph-mon[33317]: from='mgr.14150 v1:172.21.15.92:0/996057664' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-11T21:23:28.828 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:23:28 smithi097 ceph-mon[33317]: from='mgr.14150 v1:172.21.15.92:0/996057664' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-11T21:23:28.828 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:23:28 smithi097 ceph-mon[33317]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-03-11T21:23:28.828 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:23:28 smithi097 ceph-mon[33317]: from='client.34199 v1:172.21.15.92:0/3620294921' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi092:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T21:23:28.828 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:23:28 smithi097 ceph-mon[33317]: from='mgr.14150 v1:172.21.15.92:0/996057664' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-11T21:23:28.829 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:23:28 smithi097 ceph-mon[33317]: from='mgr.14150 v1:172.21.15.92:0/996057664' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-11T21:23:28.829 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:23:28 smithi097 ceph-mon[33317]: from='mgr.14150 v1:172.21.15.92:0/996057664' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T21:23:28.829 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:23:28 smithi097 ceph-mon[33317]: pgmap v81: 1 pgs: 1 unknown; 0 B data, 80 MiB used, 268 GiB / 268 GiB avail 2024-03-11T21:23:28.829 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:23:28 smithi097 ceph-mon[33317]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-03-11T21:23:28.829 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:23:28 smithi097 ceph-mon[33317]: from='mgr.14150 v1:172.21.15.92:0/996057664' entity='mgr.y' 2024-03-11T21:23:28.829 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:23:28 smithi097 ceph-mon[33317]: from='mgr.14150 v1:172.21.15.92:0/996057664' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-11T21:23:28.829 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:23:28 smithi097 ceph-mon[33317]: from='mgr.14150 v1:172.21.15.92:0/996057664' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-11T21:23:28.829 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:23:28 smithi097 ceph-mon[33317]: from='mgr.14150 v1:172.21.15.92:0/996057664' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-11T21:23:29.611 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:23:29 smithi092 ceph-mon[37563]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-03-11T21:23:29.612 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:23:29 smithi092 ceph-mon[37563]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-03-11T21:23:29.612 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:23:29 smithi092 ceph-mon[29572]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-03-11T21:23:29.612 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:23:29 smithi092 ceph-mon[29572]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-03-11T21:23:29.827 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:23:29 smithi097 ceph-mon[33317]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-03-11T21:23:29.828 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:23:29 smithi097 ceph-mon[33317]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-03-11T21:23:30.827 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:23:30 smithi097 ceph-mon[33317]: pgmap v82: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail 2024-03-11T21:23:30.828 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:23:30 smithi097 ceph-mon[33317]: from='client.? v1:172.21.15.92:0/2938335937' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "bbb6f69a-8f0f-47e6-937a-062e1902ae13"}]: dispatch 2024-03-11T21:23:30.828 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:23:30 smithi097 ceph-mon[33317]: from='client.? v1:172.21.15.92:0/2938335937' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "bbb6f69a-8f0f-47e6-937a-062e1902ae13"}]': finished 2024-03-11T21:23:30.828 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:23:30 smithi097 ceph-mon[33317]: osdmap e23: 4 total, 3 up, 4 in 2024-03-11T21:23:30.828 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:23:30 smithi097 ceph-mon[33317]: from='mgr.14150 v1:172.21.15.92:0/996057664' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-11T21:23:30.862 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:23:30 smithi092 ceph-mon[37563]: pgmap v82: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail 2024-03-11T21:23:30.862 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:23:30 smithi092 ceph-mon[37563]: from='client.? v1:172.21.15.92:0/2938335937' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "bbb6f69a-8f0f-47e6-937a-062e1902ae13"}]: dispatch 2024-03-11T21:23:30.862 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:23:30 smithi092 ceph-mon[37563]: from='client.? v1:172.21.15.92:0/2938335937' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "bbb6f69a-8f0f-47e6-937a-062e1902ae13"}]': finished 2024-03-11T21:23:30.862 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:23:30 smithi092 ceph-mon[37563]: osdmap e23: 4 total, 3 up, 4 in 2024-03-11T21:23:30.862 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:23:30 smithi092 ceph-mon[37563]: from='mgr.14150 v1:172.21.15.92:0/996057664' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-11T21:23:30.863 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:23:30 smithi092 ceph-mon[29572]: pgmap v82: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail 2024-03-11T21:23:30.863 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:23:30 smithi092 ceph-mon[29572]: from='client.? v1:172.21.15.92:0/2938335937' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "bbb6f69a-8f0f-47e6-937a-062e1902ae13"}]: dispatch 2024-03-11T21:23:30.863 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:23:30 smithi092 ceph-mon[29572]: from='client.? v1:172.21.15.92:0/2938335937' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "bbb6f69a-8f0f-47e6-937a-062e1902ae13"}]': finished 2024-03-11T21:23:30.863 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:23:30 smithi092 ceph-mon[29572]: osdmap e23: 4 total, 3 up, 4 in 2024-03-11T21:23:30.863 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:23:30 smithi092 ceph-mon[29572]: from='mgr.14150 v1:172.21.15.92:0/996057664' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-11T21:23:31.827 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:23:31 smithi097 ceph-mon[33317]: from='client.? v1:172.21.15.92:0/2322025984' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-11T21:23:31.861 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:23:31 smithi092 ceph-mon[29572]: from='client.? v1:172.21.15.92:0/2322025984' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-11T21:23:31.862 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:23:31 smithi092 ceph-mon[37563]: from='client.? v1:172.21.15.92:0/2322025984' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-11T21:23:32.827 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:23:32 smithi097 ceph-mon[33317]: pgmap v84: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail 2024-03-11T21:23:32.861 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:23:32 smithi092 ceph-mon[29572]: pgmap v84: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail 2024-03-11T21:23:32.862 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:23:32 smithi092 ceph-mon[37563]: pgmap v84: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail 2024-03-11T21:23:34.550 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:23:34 smithi092 ceph-mon[29572]: pgmap v85: 1 pgs: 1 active+clean; 577 KiB data, 81 MiB used, 268 GiB / 268 GiB avail; 4.5 KiB/s rd, 196 KiB/s wr, 13 op/s 2024-03-11T21:23:34.550 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:23:34 smithi092 ceph-mon[37563]: pgmap v85: 1 pgs: 1 active+clean; 577 KiB data, 81 MiB used, 268 GiB / 268 GiB avail; 4.5 KiB/s rd, 196 KiB/s wr, 13 op/s 2024-03-11T21:23:34.827 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:23:34 smithi097 ceph-mon[33317]: pgmap v85: 1 pgs: 1 active+clean; 577 KiB data, 81 MiB used, 268 GiB / 268 GiB avail; 4.5 KiB/s rd, 196 KiB/s wr, 13 op/s 2024-03-11T21:23:36.612 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:23:36 smithi092 ceph-mon[29572]: pgmap v86: 1 pgs: 1 active+clean; 577 KiB data, 81 MiB used, 268 GiB / 268 GiB avail; 3.0 KiB/s rd, 131 KiB/s wr, 9 op/s 2024-03-11T21:23:36.612 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:23:36 smithi092 ceph-mon[29572]: from='mgr.14150 v1:172.21.15.92:0/996057664' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.3"}]: dispatch 2024-03-11T21:23:36.612 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:23:36 smithi092 ceph-mon[29572]: from='mgr.14150 v1:172.21.15.92:0/996057664' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T21:23:36.612 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:23:36 smithi092 ceph-mon[37563]: pgmap v86: 1 pgs: 1 active+clean; 577 KiB data, 81 MiB used, 268 GiB / 268 GiB avail; 3.0 KiB/s rd, 131 KiB/s wr, 9 op/s 2024-03-11T21:23:36.612 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:23:36 smithi092 ceph-mon[37563]: from='mgr.14150 v1:172.21.15.92:0/996057664' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.3"}]: dispatch 2024-03-11T21:23:36.612 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:23:36 smithi092 ceph-mon[37563]: from='mgr.14150 v1:172.21.15.92:0/996057664' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T21:23:36.827 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:23:36 smithi097 ceph-mon[33317]: pgmap v86: 1 pgs: 1 active+clean; 577 KiB data, 81 MiB used, 268 GiB / 268 GiB avail; 3.0 KiB/s rd, 131 KiB/s wr, 9 op/s 2024-03-11T21:23:36.828 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:23:36 smithi097 ceph-mon[33317]: from='mgr.14150 v1:172.21.15.92:0/996057664' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.3"}]: dispatch 2024-03-11T21:23:36.828 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:23:36 smithi097 ceph-mon[33317]: from='mgr.14150 v1:172.21.15.92:0/996057664' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T21:23:37.612 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:23:37 smithi092 ceph-mon[29572]: Deploying daemon osd.3 on smithi092 2024-03-11T21:23:37.612 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:23:37 smithi092 ceph-mon[37563]: Deploying daemon osd.3 on smithi092 2024-03-11T21:23:37.827 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:23:37 smithi097 ceph-mon[33317]: Deploying daemon osd.3 on smithi092 2024-03-11T21:23:38.827 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:23:38 smithi097 ceph-mon[33317]: pgmap v87: 1 pgs: 1 active+clean; 577 KiB data, 81 MiB used, 268 GiB / 268 GiB avail; 2.2 KiB/s rd, 98 KiB/s wr, 6 op/s 2024-03-11T21:23:38.861 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:23:38 smithi092 ceph-mon[29572]: pgmap v87: 1 pgs: 1 active+clean; 577 KiB data, 81 MiB used, 268 GiB / 268 GiB avail; 2.2 KiB/s rd, 98 KiB/s wr, 6 op/s 2024-03-11T21:23:38.862 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:23:38 smithi092 ceph-mon[37563]: pgmap v87: 1 pgs: 1 active+clean; 577 KiB data, 81 MiB used, 268 GiB / 268 GiB avail; 2.2 KiB/s rd, 98 KiB/s wr, 6 op/s 2024-03-11T21:23:40.566 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:23:40 smithi092 ceph-mon[29572]: pgmap v88: 1 pgs: 1 active+clean; 577 KiB data, 81 MiB used, 268 GiB / 268 GiB avail; 1.8 KiB/s rd, 78 KiB/s wr, 5 op/s 2024-03-11T21:23:40.566 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:23:40 smithi092 ceph-mon[37563]: pgmap v88: 1 pgs: 1 active+clean; 577 KiB data, 81 MiB used, 268 GiB / 268 GiB avail; 1.8 KiB/s rd, 78 KiB/s wr, 5 op/s 2024-03-11T21:23:40.827 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:23:40 smithi097 ceph-mon[33317]: pgmap v88: 1 pgs: 1 active+clean; 577 KiB data, 81 MiB used, 268 GiB / 268 GiB avail; 1.8 KiB/s rd, 78 KiB/s wr, 5 op/s 2024-03-11T21:23:42.827 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:23:42 smithi097 ceph-mon[33317]: pgmap v89: 1 pgs: 1 active+clean; 577 KiB data, 81 MiB used, 268 GiB / 268 GiB avail; 1.6 KiB/s rd, 69 KiB/s wr, 4 op/s 2024-03-11T21:23:42.828 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:23:42 smithi097 ceph-mon[33317]: from='mgr.14150 v1:172.21.15.92:0/996057664' entity='mgr.y' 2024-03-11T21:23:42.828 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:23:42 smithi097 ceph-mon[33317]: from='mgr.14150 v1:172.21.15.92:0/996057664' entity='mgr.y' 2024-03-11T21:23:42.828 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:23:42 smithi097 ceph-mon[33317]: from='mgr.14150 v1:172.21.15.92:0/996057664' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T21:23:42.861 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:23:42 smithi092 ceph-mon[37563]: pgmap v89: 1 pgs: 1 active+clean; 577 KiB data, 81 MiB used, 268 GiB / 268 GiB avail; 1.6 KiB/s rd, 69 KiB/s wr, 4 op/s 2024-03-11T21:23:42.862 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:23:42 smithi092 ceph-mon[37563]: from='mgr.14150 v1:172.21.15.92:0/996057664' entity='mgr.y' 2024-03-11T21:23:42.862 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:23:42 smithi092 ceph-mon[37563]: from='mgr.14150 v1:172.21.15.92:0/996057664' entity='mgr.y' 2024-03-11T21:23:42.862 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:23:42 smithi092 ceph-mon[37563]: from='mgr.14150 v1:172.21.15.92:0/996057664' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T21:23:42.862 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:23:42 smithi092 ceph-mon[29572]: pgmap v89: 1 pgs: 1 active+clean; 577 KiB data, 81 MiB used, 268 GiB / 268 GiB avail; 1.6 KiB/s rd, 69 KiB/s wr, 4 op/s 2024-03-11T21:23:42.862 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:23:42 smithi092 ceph-mon[29572]: from='mgr.14150 v1:172.21.15.92:0/996057664' entity='mgr.y' 2024-03-11T21:23:42.862 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:23:42 smithi092 ceph-mon[29572]: from='mgr.14150 v1:172.21.15.92:0/996057664' entity='mgr.y' 2024-03-11T21:23:42.862 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:23:42 smithi092 ceph-mon[29572]: from='mgr.14150 v1:172.21.15.92:0/996057664' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T21:23:44.317 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:23:43 smithi092 ceph-mon[29572]: pgmap v90: 1 pgs: 1 active+clean; 577 KiB data, 81 MiB used, 268 GiB / 268 GiB avail; 1.5 KiB/s rd, 65 KiB/s wr, 4 op/s 2024-03-11T21:23:44.317 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:23:43 smithi092 ceph-mon[37563]: pgmap v90: 1 pgs: 1 active+clean; 577 KiB data, 81 MiB used, 268 GiB / 268 GiB avail; 1.5 KiB/s rd, 65 KiB/s wr, 4 op/s 2024-03-11T21:23:44.327 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:23:43 smithi097 ceph-mon[33317]: pgmap v90: 1 pgs: 1 active+clean; 577 KiB data, 81 MiB used, 268 GiB / 268 GiB avail; 1.5 KiB/s rd, 65 KiB/s wr, 4 op/s 2024-03-11T21:23:45.111 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:23:44 smithi092 ceph-mon[37563]: purged_snaps scrub starts 2024-03-11T21:23:45.112 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:23:44 smithi092 ceph-mon[37563]: purged_snaps scrub ok 2024-03-11T21:23:45.112 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:23:44 smithi092 ceph-mon[37563]: from='mgr.14150 v1:172.21.15.92:0/996057664' entity='mgr.y' 2024-03-11T21:23:45.112 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:23:44 smithi092 ceph-mon[37563]: from='mgr.14150 v1:172.21.15.92:0/996057664' entity='mgr.y' 2024-03-11T21:23:45.112 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:23:44 smithi092 ceph-mon[37563]: from='osd.3 v1:172.21.15.92:6813/670596264' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-03-11T21:23:45.112 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:23:44 smithi092 ceph-mon[37563]: from='osd.3 ' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-03-11T21:23:45.112 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:23:44 smithi092 ceph-mon[29572]: purged_snaps scrub starts 2024-03-11T21:23:45.112 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:23:44 smithi092 ceph-mon[29572]: purged_snaps scrub ok 2024-03-11T21:23:45.112 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:23:44 smithi092 ceph-mon[29572]: from='mgr.14150 v1:172.21.15.92:0/996057664' entity='mgr.y' 2024-03-11T21:23:45.112 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:23:44 smithi092 ceph-mon[29572]: from='mgr.14150 v1:172.21.15.92:0/996057664' entity='mgr.y' 2024-03-11T21:23:45.112 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:23:44 smithi092 ceph-mon[29572]: from='osd.3 v1:172.21.15.92:6813/670596264' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-03-11T21:23:45.113 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:23:44 smithi092 ceph-mon[29572]: from='osd.3 ' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-03-11T21:23:45.328 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:23:44 smithi097 ceph-mon[33317]: purged_snaps scrub starts 2024-03-11T21:23:45.328 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:23:44 smithi097 ceph-mon[33317]: purged_snaps scrub ok 2024-03-11T21:23:45.328 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:23:44 smithi097 ceph-mon[33317]: from='mgr.14150 v1:172.21.15.92:0/996057664' entity='mgr.y' 2024-03-11T21:23:45.328 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:23:44 smithi097 ceph-mon[33317]: from='mgr.14150 v1:172.21.15.92:0/996057664' entity='mgr.y' 2024-03-11T21:23:45.328 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:23:44 smithi097 ceph-mon[33317]: from='osd.3 v1:172.21.15.92:6813/670596264' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-03-11T21:23:45.328 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:23:44 smithi097 ceph-mon[33317]: from='osd.3 ' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-03-11T21:23:45.658 INFO:teuthology.orchestra.run.smithi092.stdout:Created osd(s) 3 on host 'smithi092' 2024-03-11T21:23:46.268 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:23:46 smithi092 ceph-mon[29572]: from='osd.3 ' entity='osd.3' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]': finished 2024-03-11T21:23:46.268 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:23:46 smithi092 ceph-mon[29572]: from='osd.3 v1:172.21.15.92:6813/670596264' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi092", "root=default"]}]: dispatch 2024-03-11T21:23:46.268 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:23:46 smithi092 ceph-mon[29572]: osdmap e24: 4 total, 3 up, 4 in 2024-03-11T21:23:46.268 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:23:46 smithi092 ceph-mon[29572]: from='mgr.14150 v1:172.21.15.92:0/996057664' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-11T21:23:46.268 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:23:46 smithi092 ceph-mon[29572]: from='osd.3 ' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi092", "root=default"]}]: dispatch 2024-03-11T21:23:46.268 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:23:46 smithi092 ceph-mon[29572]: pgmap v92: 1 pgs: 1 active+clean; 577 KiB data, 81 MiB used, 268 GiB / 268 GiB avail 2024-03-11T21:23:46.268 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:23:46 smithi092 ceph-mon[29572]: from='mgr.14150 v1:172.21.15.92:0/996057664' entity='mgr.y' 2024-03-11T21:23:46.269 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:23:46 smithi092 ceph-mon[29572]: from='mgr.14150 v1:172.21.15.92:0/996057664' entity='mgr.y' 2024-03-11T21:23:46.269 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:23:46 smithi092 ceph-mon[29572]: from='osd.3 ' entity='osd.3' cmd='[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi092", "root=default"]}]': finished 2024-03-11T21:23:46.269 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:23:46 smithi092 ceph-mon[29572]: osdmap e25: 4 total, 3 up, 4 in 2024-03-11T21:23:46.269 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:23:46 smithi092 ceph-mon[29572]: from='mgr.14150 v1:172.21.15.92:0/996057664' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-11T21:23:46.269 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:23:46 smithi092 ceph-mon[37563]: from='osd.3 ' entity='osd.3' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]': finished 2024-03-11T21:23:46.269 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:23:46 smithi092 ceph-mon[37563]: from='osd.3 v1:172.21.15.92:6813/670596264' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi092", "root=default"]}]: dispatch 2024-03-11T21:23:46.269 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:23:46 smithi092 ceph-mon[37563]: osdmap e24: 4 total, 3 up, 4 in 2024-03-11T21:23:46.270 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:23:46 smithi092 ceph-mon[37563]: from='mgr.14150 v1:172.21.15.92:0/996057664' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-11T21:23:46.270 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:23:46 smithi092 ceph-mon[37563]: from='osd.3 ' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi092", "root=default"]}]: dispatch 2024-03-11T21:23:46.270 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:23:46 smithi092 ceph-mon[37563]: pgmap v92: 1 pgs: 1 active+clean; 577 KiB data, 81 MiB used, 268 GiB / 268 GiB avail 2024-03-11T21:23:46.270 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:23:46 smithi092 ceph-mon[37563]: from='mgr.14150 v1:172.21.15.92:0/996057664' entity='mgr.y' 2024-03-11T21:23:46.270 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:23:46 smithi092 ceph-mon[37563]: from='mgr.14150 v1:172.21.15.92:0/996057664' entity='mgr.y' 2024-03-11T21:23:46.270 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:23:46 smithi092 ceph-mon[37563]: from='osd.3 ' entity='osd.3' cmd='[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi092", "root=default"]}]': finished 2024-03-11T21:23:46.270 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:23:46 smithi092 ceph-mon[37563]: osdmap e25: 4 total, 3 up, 4 in 2024-03-11T21:23:46.270 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:23:46 smithi092 ceph-mon[37563]: from='mgr.14150 v1:172.21.15.92:0/996057664' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-11T21:23:46.298 DEBUG:teuthology.orchestra.run.smithi092:osd.3> sudo journalctl -f -n 0 -u ceph-00db0016-dfed-11ee-95c8-87774f69a715@osd.3.service 2024-03-11T21:23:46.301 INFO:tasks.cephadm:Deploying osd.4 on smithi097 with /dev/vg_nvme/lv_4... 2024-03-11T21:23:46.301 DEBUG:teuthology.orchestra.run.smithi097:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 00db0016-dfed-11ee-95c8-87774f69a715 -- ceph-volume lvm zap /dev/vg_nvme/lv_4 2024-03-11T21:23:46.327 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:23:46 smithi097 ceph-mon[33317]: from='osd.3 ' entity='osd.3' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]': finished 2024-03-11T21:23:46.327 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:23:46 smithi097 ceph-mon[33317]: from='osd.3 v1:172.21.15.92:6813/670596264' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi092", "root=default"]}]: dispatch 2024-03-11T21:23:46.328 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:23:46 smithi097 ceph-mon[33317]: osdmap e24: 4 total, 3 up, 4 in 2024-03-11T21:23:46.328 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:23:46 smithi097 ceph-mon[33317]: from='mgr.14150 v1:172.21.15.92:0/996057664' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-11T21:23:46.328 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:23:46 smithi097 ceph-mon[33317]: from='osd.3 ' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi092", "root=default"]}]: dispatch 2024-03-11T21:23:46.328 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:23:46 smithi097 ceph-mon[33317]: pgmap v92: 1 pgs: 1 active+clean; 577 KiB data, 81 MiB used, 268 GiB / 268 GiB avail 2024-03-11T21:23:46.328 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:23:46 smithi097 ceph-mon[33317]: from='mgr.14150 v1:172.21.15.92:0/996057664' entity='mgr.y' 2024-03-11T21:23:46.328 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:23:46 smithi097 ceph-mon[33317]: from='mgr.14150 v1:172.21.15.92:0/996057664' entity='mgr.y' 2024-03-11T21:23:46.328 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:23:46 smithi097 ceph-mon[33317]: from='osd.3 ' entity='osd.3' cmd='[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi092", "root=default"]}]': finished 2024-03-11T21:23:46.328 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:23:46 smithi097 ceph-mon[33317]: osdmap e25: 4 total, 3 up, 4 in 2024-03-11T21:23:46.328 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:23:46 smithi097 ceph-mon[33317]: from='mgr.14150 v1:172.21.15.92:0/996057664' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-11T21:23:46.544 INFO:teuthology.orchestra.run.smithi097.stderr:Inferring config /var/lib/ceph/00db0016-dfed-11ee-95c8-87774f69a715/mon.b/config 2024-03-11T21:23:46.612 INFO:journalctl@ceph.osd.3.smithi092.stdout:Mar 11 21:23:46 smithi092 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd-3[59566]: 2024-03-11T21:23:46.480+0000 7f484745d700 -1 osd.3 0 waiting for initial osdmap 2024-03-11T21:23:46.612 INFO:journalctl@ceph.osd.3.smithi092.stdout:Mar 11 21:23:46 smithi092 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd-3[59566]: 2024-03-11T21:23:46.484+0000 7f483f248700 -1 osd.3 25 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-11T21:23:47.328 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:23:47 smithi097 ceph-mon[33317]: from='mgr.14150 v1:172.21.15.92:0/996057664' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-11T21:23:47.328 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:23:47 smithi097 ceph-mon[33317]: from='osd.3 ' entity='osd.3' 2024-03-11T21:23:47.328 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:23:47 smithi097 ceph-mon[33317]: from='mgr.14150 v1:172.21.15.92:0/996057664' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-11T21:23:47.337 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:23:47 smithi092 ceph-mon[29572]: from='mgr.14150 v1:172.21.15.92:0/996057664' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-11T21:23:47.337 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:23:47 smithi092 ceph-mon[29572]: from='osd.3 ' entity='osd.3' 2024-03-11T21:23:47.337 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:23:47 smithi092 ceph-mon[29572]: from='mgr.14150 v1:172.21.15.92:0/996057664' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-11T21:23:47.338 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:23:47 smithi092 ceph-mon[37563]: from='mgr.14150 v1:172.21.15.92:0/996057664' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-11T21:23:47.338 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:23:47 smithi092 ceph-mon[37563]: from='osd.3 ' entity='osd.3' 2024-03-11T21:23:47.338 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:23:47 smithi092 ceph-mon[37563]: from='mgr.14150 v1:172.21.15.92:0/996057664' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-11T21:23:48.322 INFO:teuthology.orchestra.run.smithi097.stderr:--> Zapping: /dev/vg_nvme/lv_4 2024-03-11T21:23:48.322 INFO:teuthology.orchestra.run.smithi097.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_4 bs=1M count=10 conv=fsync 2024-03-11T21:23:48.322 INFO:teuthology.orchestra.run.smithi097.stderr: stderr: 10+0 records in 2024-03-11T21:23:48.322 INFO:teuthology.orchestra.run.smithi097.stderr:10+0 records out 2024-03-11T21:23:48.322 INFO:teuthology.orchestra.run.smithi097.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0114168 s, 918 MB/s 2024-03-11T21:23:48.323 INFO:teuthology.orchestra.run.smithi097.stderr:--> Zapping successful for: 2024-03-11T21:23:48.627 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:23:48 smithi097 ceph-mon[33317]: purged_snaps scrub starts 2024-03-11T21:23:48.627 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:23:48 smithi097 ceph-mon[33317]: purged_snaps scrub ok 2024-03-11T21:23:48.627 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:23:48 smithi097 ceph-mon[33317]: osd.3 v1:172.21.15.92:6813/670596264 boot 2024-03-11T21:23:48.628 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:23:48 smithi097 ceph-mon[33317]: osdmap e26: 4 total, 4 up, 4 in 2024-03-11T21:23:48.628 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:23:48 smithi097 ceph-mon[33317]: from='mgr.14150 v1:172.21.15.92:0/996057664' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-11T21:23:48.628 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:23:48 smithi097 ceph-mon[33317]: pgmap v95: 1 pgs: 1 active+clean; 577 KiB data, 81 MiB used, 268 GiB / 268 GiB avail 2024-03-11T21:23:48.862 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:23:48 smithi092 ceph-mon[37563]: purged_snaps scrub starts 2024-03-11T21:23:48.862 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:23:48 smithi092 ceph-mon[37563]: purged_snaps scrub ok 2024-03-11T21:23:48.862 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:23:48 smithi092 ceph-mon[37563]: osd.3 v1:172.21.15.92:6813/670596264 boot 2024-03-11T21:23:48.862 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:23:48 smithi092 ceph-mon[37563]: osdmap e26: 4 total, 4 up, 4 in 2024-03-11T21:23:48.862 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:23:48 smithi092 ceph-mon[37563]: from='mgr.14150 v1:172.21.15.92:0/996057664' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-11T21:23:48.862 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:23:48 smithi092 ceph-mon[37563]: pgmap v95: 1 pgs: 1 active+clean; 577 KiB data, 81 MiB used, 268 GiB / 268 GiB avail 2024-03-11T21:23:48.862 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:23:48 smithi092 ceph-mon[29572]: purged_snaps scrub starts 2024-03-11T21:23:48.862 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:23:48 smithi092 ceph-mon[29572]: purged_snaps scrub ok 2024-03-11T21:23:48.862 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:23:48 smithi092 ceph-mon[29572]: osd.3 v1:172.21.15.92:6813/670596264 boot 2024-03-11T21:23:48.863 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:23:48 smithi092 ceph-mon[29572]: osdmap e26: 4 total, 4 up, 4 in 2024-03-11T21:23:48.863 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:23:48 smithi092 ceph-mon[29572]: from='mgr.14150 v1:172.21.15.92:0/996057664' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-11T21:23:48.863 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:23:48 smithi092 ceph-mon[29572]: pgmap v95: 1 pgs: 1 active+clean; 577 KiB data, 81 MiB used, 268 GiB / 268 GiB avail 2024-03-11T21:23:49.021 DEBUG:teuthology.orchestra.run.smithi097:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 00db0016-dfed-11ee-95c8-87774f69a715 -- ceph orch daemon add osd smithi097:vg_nvme/lv_4 2024-03-11T21:23:49.305 INFO:teuthology.orchestra.run.smithi097.stderr:Inferring config /var/lib/ceph/00db0016-dfed-11ee-95c8-87774f69a715/mon.b/config 2024-03-11T21:23:49.649 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:23:49 smithi092 ceph-mon[29572]: osdmap e27: 4 total, 4 up, 4 in 2024-03-11T21:23:49.649 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:23:49 smithi092 ceph-mon[37563]: osdmap e27: 4 total, 4 up, 4 in 2024-03-11T21:23:49.688 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:23:49 smithi097 ceph-mon[33317]: osdmap e27: 4 total, 4 up, 4 in 2024-03-11T21:23:50.634 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:23:50 smithi097 ceph-mon[33317]: pgmap v97: 1 pgs: 1 active+clean; 577 KiB data, 112 MiB used, 357 GiB / 358 GiB avail 2024-03-11T21:23:50.634 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:23:50 smithi097 ceph-mon[33317]: from='mgr.14150 v1:172.21.15.92:0/996057664' entity='mgr.y' 2024-03-11T21:23:50.634 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:23:50 smithi097 ceph-mon[33317]: from='mgr.14150 v1:172.21.15.92:0/996057664' entity='mgr.y' 2024-03-11T21:23:50.634 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:23:50 smithi097 ceph-mon[33317]: from='mgr.14150 v1:172.21.15.92:0/996057664' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-11T21:23:50.635 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:23:50 smithi097 ceph-mon[33317]: from='mgr.14150 v1:172.21.15.92:0/996057664' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-11T21:23:50.635 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:23:50 smithi097 ceph-mon[33317]: from='mgr.14150 v1:172.21.15.92:0/996057664' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-11T21:23:50.635 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:23:50 smithi097 ceph-mon[33317]: from='mgr.14150 v1:172.21.15.92:0/996057664' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-03-11T21:23:50.635 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:23:50 smithi097 ceph-mon[33317]: from='mgr.14150 v1:172.21.15.92:0/996057664' entity='mgr.y' 2024-03-11T21:23:50.635 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:23:50 smithi097 ceph-mon[33317]: from='mgr.14150 v1:172.21.15.92:0/996057664' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T21:23:50.635 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:23:50 smithi097 ceph-mon[33317]: from='mgr.14150 v1:172.21.15.92:0/996057664' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T21:23:50.635 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:23:50 smithi097 ceph-mon[33317]: from='mgr.14150 v1:172.21.15.92:0/996057664' entity='mgr.y' 2024-03-11T21:23:50.635 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:23:50 smithi097 ceph-mon[33317]: from='mgr.14150 v1:172.21.15.92:0/996057664' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T21:23:50.635 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:23:50 smithi097 ceph-mon[33317]: from='mgr.14150 v1:172.21.15.92:0/996057664' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T21:23:50.635 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:23:50 smithi097 ceph-mon[33317]: from='mgr.14150 v1:172.21.15.92:0/996057664' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T21:23:50.635 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:23:50 smithi097 ceph-mon[33317]: from='mgr.14150 v1:172.21.15.92:0/996057664' entity='mgr.y' 2024-03-11T21:23:50.862 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:23:50 smithi092 ceph-mon[29572]: pgmap v97: 1 pgs: 1 active+clean; 577 KiB data, 112 MiB used, 357 GiB / 358 GiB avail 2024-03-11T21:23:50.862 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:23:50 smithi092 ceph-mon[29572]: from='mgr.14150 v1:172.21.15.92:0/996057664' entity='mgr.y' 2024-03-11T21:23:50.862 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:23:50 smithi092 ceph-mon[29572]: from='mgr.14150 v1:172.21.15.92:0/996057664' entity='mgr.y' 2024-03-11T21:23:50.862 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:23:50 smithi092 ceph-mon[29572]: from='mgr.14150 v1:172.21.15.92:0/996057664' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-11T21:23:50.862 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:23:50 smithi092 ceph-mon[29572]: from='mgr.14150 v1:172.21.15.92:0/996057664' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-11T21:23:50.862 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:23:50 smithi092 ceph-mon[29572]: from='mgr.14150 v1:172.21.15.92:0/996057664' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-11T21:23:50.862 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:23:50 smithi092 ceph-mon[29572]: from='mgr.14150 v1:172.21.15.92:0/996057664' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-03-11T21:23:50.862 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:23:50 smithi092 ceph-mon[29572]: from='mgr.14150 v1:172.21.15.92:0/996057664' entity='mgr.y' 2024-03-11T21:23:50.862 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:23:50 smithi092 ceph-mon[29572]: from='mgr.14150 v1:172.21.15.92:0/996057664' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T21:23:50.862 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:23:50 smithi092 ceph-mon[29572]: from='mgr.14150 v1:172.21.15.92:0/996057664' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T21:23:50.863 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:23:50 smithi092 ceph-mon[29572]: from='mgr.14150 v1:172.21.15.92:0/996057664' entity='mgr.y' 2024-03-11T21:23:50.863 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:23:50 smithi092 ceph-mon[29572]: from='mgr.14150 v1:172.21.15.92:0/996057664' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T21:23:50.863 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:23:50 smithi092 ceph-mon[29572]: from='mgr.14150 v1:172.21.15.92:0/996057664' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T21:23:50.863 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:23:50 smithi092 ceph-mon[29572]: from='mgr.14150 v1:172.21.15.92:0/996057664' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T21:23:50.863 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:23:50 smithi092 ceph-mon[29572]: from='mgr.14150 v1:172.21.15.92:0/996057664' entity='mgr.y' 2024-03-11T21:23:50.863 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:23:50 smithi092 ceph-mon[37563]: pgmap v97: 1 pgs: 1 active+clean; 577 KiB data, 112 MiB used, 357 GiB / 358 GiB avail 2024-03-11T21:23:50.863 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:23:50 smithi092 ceph-mon[37563]: from='mgr.14150 v1:172.21.15.92:0/996057664' entity='mgr.y' 2024-03-11T21:23:50.863 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:23:50 smithi092 ceph-mon[37563]: from='mgr.14150 v1:172.21.15.92:0/996057664' entity='mgr.y' 2024-03-11T21:23:50.863 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:23:50 smithi092 ceph-mon[37563]: from='mgr.14150 v1:172.21.15.92:0/996057664' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-11T21:23:50.863 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:23:50 smithi092 ceph-mon[37563]: from='mgr.14150 v1:172.21.15.92:0/996057664' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-11T21:23:50.864 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:23:50 smithi092 ceph-mon[37563]: from='mgr.14150 v1:172.21.15.92:0/996057664' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-11T21:23:50.864 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:23:50 smithi092 ceph-mon[37563]: from='mgr.14150 v1:172.21.15.92:0/996057664' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-03-11T21:23:50.864 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:23:50 smithi092 ceph-mon[37563]: from='mgr.14150 v1:172.21.15.92:0/996057664' entity='mgr.y' 2024-03-11T21:23:50.864 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:23:50 smithi092 ceph-mon[37563]: from='mgr.14150 v1:172.21.15.92:0/996057664' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T21:23:50.864 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:23:50 smithi092 ceph-mon[37563]: from='mgr.14150 v1:172.21.15.92:0/996057664' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T21:23:50.864 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:23:50 smithi092 ceph-mon[37563]: from='mgr.14150 v1:172.21.15.92:0/996057664' entity='mgr.y' 2024-03-11T21:23:50.864 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:23:50 smithi092 ceph-mon[37563]: from='mgr.14150 v1:172.21.15.92:0/996057664' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T21:23:50.864 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:23:50 smithi092 ceph-mon[37563]: from='mgr.14150 v1:172.21.15.92:0/996057664' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T21:23:50.864 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:23:50 smithi092 ceph-mon[37563]: from='mgr.14150 v1:172.21.15.92:0/996057664' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T21:23:50.864 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:23:50 smithi092 ceph-mon[37563]: from='mgr.14150 v1:172.21.15.92:0/996057664' entity='mgr.y' 2024-03-11T21:23:51.794 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:23:51 smithi097 ceph-mon[33317]: Detected new or changed devices on smithi092 2024-03-11T21:23:51.794 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:23:51 smithi097 ceph-mon[33317]: Adjusting osd_memory_target on smithi092 to 4016M 2024-03-11T21:23:51.794 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:23:51 smithi097 ceph-mon[33317]: from='mgr.14150 v1:172.21.15.92:0/996057664' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-11T21:23:51.794 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:23:51 smithi097 ceph-mon[33317]: from='mgr.14150 v1:172.21.15.92:0/996057664' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-11T21:23:51.794 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:23:51 smithi097 ceph-mon[33317]: from='mgr.14150 v1:172.21.15.92:0/996057664' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T21:23:51.862 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:23:51 smithi092 ceph-mon[29572]: Detected new or changed devices on smithi092 2024-03-11T21:23:51.862 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:23:51 smithi092 ceph-mon[29572]: Adjusting osd_memory_target on smithi092 to 4016M 2024-03-11T21:23:51.862 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:23:51 smithi092 ceph-mon[29572]: from='mgr.14150 v1:172.21.15.92:0/996057664' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-11T21:23:51.862 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:23:51 smithi092 ceph-mon[29572]: from='mgr.14150 v1:172.21.15.92:0/996057664' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-11T21:23:51.862 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:23:51 smithi092 ceph-mon[29572]: from='mgr.14150 v1:172.21.15.92:0/996057664' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T21:23:51.862 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:23:51 smithi092 ceph-mon[37563]: Detected new or changed devices on smithi092 2024-03-11T21:23:51.862 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:23:51 smithi092 ceph-mon[37563]: Adjusting osd_memory_target on smithi092 to 4016M 2024-03-11T21:23:51.862 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:23:51 smithi092 ceph-mon[37563]: from='mgr.14150 v1:172.21.15.92:0/996057664' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-11T21:23:51.862 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:23:51 smithi092 ceph-mon[37563]: from='mgr.14150 v1:172.21.15.92:0/996057664' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-11T21:23:51.863 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:23:51 smithi092 ceph-mon[37563]: from='mgr.14150 v1:172.21.15.92:0/996057664' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T21:23:52.805 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:23:52 smithi097 ceph-mon[33317]: from='client.24232 v1:172.21.15.97:0/183106418' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi097:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T21:23:52.805 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:23:52 smithi097 ceph-mon[33317]: pgmap v98: 1 pgs: 1 active+clean; 577 KiB data, 112 MiB used, 357 GiB / 358 GiB avail 2024-03-11T21:23:52.862 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:23:52 smithi092 ceph-mon[29572]: from='client.24232 v1:172.21.15.97:0/183106418' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi097:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T21:23:52.862 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:23:52 smithi092 ceph-mon[29572]: pgmap v98: 1 pgs: 1 active+clean; 577 KiB data, 112 MiB used, 357 GiB / 358 GiB avail 2024-03-11T21:23:52.862 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:23:52 smithi092 ceph-mon[37563]: from='client.24232 v1:172.21.15.97:0/183106418' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi097:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T21:23:52.862 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:23:52 smithi092 ceph-mon[37563]: pgmap v98: 1 pgs: 1 active+clean; 577 KiB data, 112 MiB used, 357 GiB / 358 GiB avail 2024-03-11T21:23:54.327 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:23:53 smithi097 ceph-mon[33317]: pgmap v99: 1 pgs: 1 active+clean; 577 KiB data, 112 MiB used, 357 GiB / 358 GiB avail 2024-03-11T21:23:54.328 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:23:53 smithi097 ceph-mon[33317]: from='client.? v1:172.21.15.97:0/2901462292' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "00266cbc-d2cf-4dee-8a96-3753ebc95ca8"}]: dispatch 2024-03-11T21:23:54.328 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:23:53 smithi097 ceph-mon[33317]: from='client.24238 ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "00266cbc-d2cf-4dee-8a96-3753ebc95ca8"}]: dispatch 2024-03-11T21:23:54.328 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:23:53 smithi097 ceph-mon[33317]: from='client.24238 ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "00266cbc-d2cf-4dee-8a96-3753ebc95ca8"}]': finished 2024-03-11T21:23:54.328 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:23:53 smithi097 ceph-mon[33317]: osdmap e28: 5 total, 4 up, 5 in 2024-03-11T21:23:54.328 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:23:53 smithi097 ceph-mon[33317]: from='mgr.14150 v1:172.21.15.92:0/996057664' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-11T21:23:54.362 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:23:53 smithi092 ceph-mon[29572]: pgmap v99: 1 pgs: 1 active+clean; 577 KiB data, 112 MiB used, 357 GiB / 358 GiB avail 2024-03-11T21:23:54.362 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:23:53 smithi092 ceph-mon[29572]: from='client.? v1:172.21.15.97:0/2901462292' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "00266cbc-d2cf-4dee-8a96-3753ebc95ca8"}]: dispatch 2024-03-11T21:23:54.362 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:23:53 smithi092 ceph-mon[29572]: from='client.24238 ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "00266cbc-d2cf-4dee-8a96-3753ebc95ca8"}]: dispatch 2024-03-11T21:23:54.362 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:23:53 smithi092 ceph-mon[29572]: from='client.24238 ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "00266cbc-d2cf-4dee-8a96-3753ebc95ca8"}]': finished 2024-03-11T21:23:54.362 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:23:53 smithi092 ceph-mon[29572]: osdmap e28: 5 total, 4 up, 5 in 2024-03-11T21:23:54.362 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:23:53 smithi092 ceph-mon[29572]: from='mgr.14150 v1:172.21.15.92:0/996057664' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-11T21:23:54.363 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:23:53 smithi092 ceph-mon[37563]: pgmap v99: 1 pgs: 1 active+clean; 577 KiB data, 112 MiB used, 357 GiB / 358 GiB avail 2024-03-11T21:23:54.363 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:23:53 smithi092 ceph-mon[37563]: from='client.? v1:172.21.15.97:0/2901462292' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "00266cbc-d2cf-4dee-8a96-3753ebc95ca8"}]: dispatch 2024-03-11T21:23:54.363 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:23:53 smithi092 ceph-mon[37563]: from='client.24238 ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "00266cbc-d2cf-4dee-8a96-3753ebc95ca8"}]: dispatch 2024-03-11T21:23:54.363 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:23:53 smithi092 ceph-mon[37563]: from='client.24238 ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "00266cbc-d2cf-4dee-8a96-3753ebc95ca8"}]': finished 2024-03-11T21:23:54.363 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:23:53 smithi092 ceph-mon[37563]: osdmap e28: 5 total, 4 up, 5 in 2024-03-11T21:23:54.363 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:23:53 smithi092 ceph-mon[37563]: from='mgr.14150 v1:172.21.15.92:0/996057664' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-11T21:23:55.327 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:23:55 smithi097 ceph-mon[33317]: from='client.? v1:172.21.15.97:0/3221698736' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-11T21:23:55.362 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:23:55 smithi092 ceph-mon[37563]: from='client.? v1:172.21.15.97:0/3221698736' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-11T21:23:55.362 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:23:55 smithi092 ceph-mon[29572]: from='client.? v1:172.21.15.97:0/3221698736' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-11T21:23:56.327 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:23:56 smithi097 ceph-mon[33317]: pgmap v101: 1 pgs: 1 active+clean; 577 KiB data, 112 MiB used, 357 GiB / 358 GiB avail 2024-03-11T21:23:56.361 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:23:56 smithi092 ceph-mon[29572]: pgmap v101: 1 pgs: 1 active+clean; 577 KiB data, 112 MiB used, 357 GiB / 358 GiB avail 2024-03-11T21:23:56.362 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:23:56 smithi092 ceph-mon[37563]: pgmap v101: 1 pgs: 1 active+clean; 577 KiB data, 112 MiB used, 357 GiB / 358 GiB avail 2024-03-11T21:23:58.823 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:23:58 smithi097 ceph-mon[33317]: pgmap v102: 1 pgs: 1 active+clean; 577 KiB data, 112 MiB used, 357 GiB / 358 GiB avail 2024-03-11T21:23:58.862 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:23:58 smithi092 ceph-mon[37563]: pgmap v102: 1 pgs: 1 active+clean; 577 KiB data, 112 MiB used, 357 GiB / 358 GiB avail 2024-03-11T21:23:58.862 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:23:58 smithi092 ceph-mon[29572]: pgmap v102: 1 pgs: 1 active+clean; 577 KiB data, 112 MiB used, 357 GiB / 358 GiB avail 2024-03-11T21:24:00.767 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:24:00 smithi097 ceph-mon[33317]: pgmap v103: 1 pgs: 1 active+clean; 577 KiB data, 112 MiB used, 357 GiB / 358 GiB avail 2024-03-11T21:24:00.767 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:24:00 smithi097 ceph-mon[33317]: from='mgr.14150 v1:172.21.15.92:0/996057664' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.4"}]: dispatch 2024-03-11T21:24:00.767 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:24:00 smithi097 ceph-mon[33317]: from='mgr.14150 v1:172.21.15.92:0/996057664' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T21:24:00.861 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:24:00 smithi092 ceph-mon[29572]: pgmap v103: 1 pgs: 1 active+clean; 577 KiB data, 112 MiB used, 357 GiB / 358 GiB avail 2024-03-11T21:24:00.862 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:24:00 smithi092 ceph-mon[29572]: from='mgr.14150 v1:172.21.15.92:0/996057664' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.4"}]: dispatch 2024-03-11T21:24:00.862 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:24:00 smithi092 ceph-mon[29572]: from='mgr.14150 v1:172.21.15.92:0/996057664' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T21:24:00.862 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:24:00 smithi092 ceph-mon[37563]: pgmap v103: 1 pgs: 1 active+clean; 577 KiB data, 112 MiB used, 357 GiB / 358 GiB avail 2024-03-11T21:24:00.862 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:24:00 smithi092 ceph-mon[37563]: from='mgr.14150 v1:172.21.15.92:0/996057664' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.4"}]: dispatch 2024-03-11T21:24:00.862 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:24:00 smithi092 ceph-mon[37563]: from='mgr.14150 v1:172.21.15.92:0/996057664' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T21:24:01.731 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:24:01 smithi097 ceph-mon[33317]: Deploying daemon osd.4 on smithi097 2024-03-11T21:24:01.861 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:24:01 smithi092 ceph-mon[29572]: Deploying daemon osd.4 on smithi097 2024-03-11T21:24:01.862 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:24:01 smithi092 ceph-mon[37563]: Deploying daemon osd.4 on smithi097 2024-03-11T21:24:02.827 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:24:02 smithi097 ceph-mon[33317]: pgmap v104: 1 pgs: 1 active+clean; 577 KiB data, 112 MiB used, 357 GiB / 358 GiB avail 2024-03-11T21:24:02.861 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:24:02 smithi092 ceph-mon[29572]: pgmap v104: 1 pgs: 1 active+clean; 577 KiB data, 112 MiB used, 357 GiB / 358 GiB avail 2024-03-11T21:24:02.862 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:24:02 smithi092 ceph-mon[37563]: pgmap v104: 1 pgs: 1 active+clean; 577 KiB data, 112 MiB used, 357 GiB / 358 GiB avail 2024-03-11T21:24:04.297 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:24:03 smithi097 ceph-mon[33317]: pgmap v105: 1 pgs: 1 active+clean; 577 KiB data, 112 MiB used, 357 GiB / 358 GiB avail 2024-03-11T21:24:04.361 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:24:03 smithi092 ceph-mon[29572]: pgmap v105: 1 pgs: 1 active+clean; 577 KiB data, 112 MiB used, 357 GiB / 358 GiB avail 2024-03-11T21:24:04.362 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:24:03 smithi092 ceph-mon[37563]: pgmap v105: 1 pgs: 1 active+clean; 577 KiB data, 112 MiB used, 357 GiB / 358 GiB avail 2024-03-11T21:24:05.070 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:24:04 smithi097 ceph-mon[33317]: purged_snaps scrub starts 2024-03-11T21:24:05.071 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:24:04 smithi097 ceph-mon[33317]: purged_snaps scrub ok 2024-03-11T21:24:05.071 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:24:04 smithi097 ceph-mon[33317]: osdmap e29: 5 total, 4 up, 5 in 2024-03-11T21:24:05.071 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:24:04 smithi097 ceph-mon[33317]: from='mgr.14150 v1:172.21.15.92:0/996057664' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-11T21:24:05.361 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:24:04 smithi092 ceph-mon[37563]: purged_snaps scrub starts 2024-03-11T21:24:05.362 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:24:04 smithi092 ceph-mon[37563]: purged_snaps scrub ok 2024-03-11T21:24:05.362 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:24:04 smithi092 ceph-mon[37563]: osdmap e29: 5 total, 4 up, 5 in 2024-03-11T21:24:05.362 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:24:04 smithi092 ceph-mon[37563]: from='mgr.14150 v1:172.21.15.92:0/996057664' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-11T21:24:05.362 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:24:04 smithi092 ceph-mon[29572]: purged_snaps scrub starts 2024-03-11T21:24:05.362 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:24:04 smithi092 ceph-mon[29572]: purged_snaps scrub ok 2024-03-11T21:24:05.362 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:24:04 smithi092 ceph-mon[29572]: osdmap e29: 5 total, 4 up, 5 in 2024-03-11T21:24:05.362 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:24:04 smithi092 ceph-mon[29572]: from='mgr.14150 v1:172.21.15.92:0/996057664' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-11T21:24:06.328 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:24:06 smithi097 ceph-mon[33317]: from='mgr.14150 v1:172.21.15.92:0/996057664' entity='mgr.y' 2024-03-11T21:24:06.328 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:24:06 smithi097 ceph-mon[33317]: from='mgr.14150 v1:172.21.15.92:0/996057664' entity='mgr.y' 2024-03-11T21:24:06.328 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:24:06 smithi097 ceph-mon[33317]: from='mgr.14150 v1:172.21.15.92:0/996057664' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T21:24:06.328 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:24:06 smithi097 ceph-mon[33317]: pgmap v107: 1 pgs: 1 active+clean; 577 KiB data, 112 MiB used, 357 GiB / 358 GiB avail 2024-03-11T21:24:06.612 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:24:06 smithi092 ceph-mon[29572]: from='mgr.14150 v1:172.21.15.92:0/996057664' entity='mgr.y' 2024-03-11T21:24:06.612 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:24:06 smithi092 ceph-mon[29572]: from='mgr.14150 v1:172.21.15.92:0/996057664' entity='mgr.y' 2024-03-11T21:24:06.612 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:24:06 smithi092 ceph-mon[29572]: from='mgr.14150 v1:172.21.15.92:0/996057664' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T21:24:06.612 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:24:06 smithi092 ceph-mon[29572]: pgmap v107: 1 pgs: 1 active+clean; 577 KiB data, 112 MiB used, 357 GiB / 358 GiB avail 2024-03-11T21:24:06.612 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:24:06 smithi092 ceph-mon[37563]: from='mgr.14150 v1:172.21.15.92:0/996057664' entity='mgr.y' 2024-03-11T21:24:06.613 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:24:06 smithi092 ceph-mon[37563]: from='mgr.14150 v1:172.21.15.92:0/996057664' entity='mgr.y' 2024-03-11T21:24:06.613 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:24:06 smithi092 ceph-mon[37563]: from='mgr.14150 v1:172.21.15.92:0/996057664' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T21:24:06.613 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:24:06 smithi092 ceph-mon[37563]: pgmap v107: 1 pgs: 1 active+clean; 577 KiB data, 112 MiB used, 357 GiB / 358 GiB avail 2024-03-11T21:24:08.281 INFO:teuthology.orchestra.run.smithi097.stdout:Created osd(s) 4 on host 'smithi097' 2024-03-11T21:24:08.827 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:24:08 smithi097 ceph-mon[33317]: pgmap v108: 1 pgs: 1 active+clean; 577 KiB data, 112 MiB used, 357 GiB / 358 GiB avail 2024-03-11T21:24:08.827 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:24:08 smithi097 ceph-mon[33317]: from='osd.4 v1:172.21.15.97:6800/420733620' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-03-11T21:24:08.828 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:24:08 smithi097 ceph-mon[33317]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-03-11T21:24:08.828 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:24:08 smithi097 ceph-mon[33317]: from='mgr.14150 v1:172.21.15.92:0/996057664' entity='mgr.y' 2024-03-11T21:24:08.828 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:24:08 smithi097 ceph-mon[33317]: from='mgr.14150 v1:172.21.15.92:0/996057664' entity='mgr.y' 2024-03-11T21:24:08.862 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:24:08 smithi092 ceph-mon[29572]: pgmap v108: 1 pgs: 1 active+clean; 577 KiB data, 112 MiB used, 357 GiB / 358 GiB avail 2024-03-11T21:24:08.862 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:24:08 smithi092 ceph-mon[29572]: from='osd.4 v1:172.21.15.97:6800/420733620' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-03-11T21:24:08.862 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:24:08 smithi092 ceph-mon[29572]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-03-11T21:24:08.862 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:24:08 smithi092 ceph-mon[29572]: from='mgr.14150 v1:172.21.15.92:0/996057664' entity='mgr.y' 2024-03-11T21:24:08.862 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:24:08 smithi092 ceph-mon[29572]: from='mgr.14150 v1:172.21.15.92:0/996057664' entity='mgr.y' 2024-03-11T21:24:08.862 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:24:08 smithi092 ceph-mon[37563]: pgmap v108: 1 pgs: 1 active+clean; 577 KiB data, 112 MiB used, 357 GiB / 358 GiB avail 2024-03-11T21:24:08.862 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:24:08 smithi092 ceph-mon[37563]: from='osd.4 v1:172.21.15.97:6800/420733620' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-03-11T21:24:08.862 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:24:08 smithi092 ceph-mon[37563]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-03-11T21:24:08.863 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:24:08 smithi092 ceph-mon[37563]: from='mgr.14150 v1:172.21.15.92:0/996057664' entity='mgr.y' 2024-03-11T21:24:08.863 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:24:08 smithi092 ceph-mon[37563]: from='mgr.14150 v1:172.21.15.92:0/996057664' entity='mgr.y' 2024-03-11T21:24:09.151 DEBUG:teuthology.orchestra.run.smithi097:osd.4> sudo journalctl -f -n 0 -u ceph-00db0016-dfed-11ee-95c8-87774f69a715@osd.4.service 2024-03-11T21:24:09.153 INFO:tasks.cephadm:Deploying osd.5 on smithi097 with /dev/vg_nvme/lv_3... 2024-03-11T21:24:09.154 DEBUG:teuthology.orchestra.run.smithi097:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 00db0016-dfed-11ee-95c8-87774f69a715 -- ceph-volume lvm zap /dev/vg_nvme/lv_3 2024-03-11T21:24:09.434 INFO:teuthology.orchestra.run.smithi097.stderr:Inferring config /var/lib/ceph/00db0016-dfed-11ee-95c8-87774f69a715/mon.b/config 2024-03-11T21:24:09.828 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:24:09 smithi097 ceph-mon[33317]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]': finished 2024-03-11T21:24:09.828 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:24:09 smithi097 ceph-mon[33317]: from='osd.4 v1:172.21.15.97:6800/420733620' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi097", "root=default"]}]: dispatch 2024-03-11T21:24:09.828 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:24:09 smithi097 ceph-mon[33317]: osdmap e30: 5 total, 4 up, 5 in 2024-03-11T21:24:09.828 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:24:09 smithi097 ceph-mon[33317]: from='mgr.14150 v1:172.21.15.92:0/996057664' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-11T21:24:09.828 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:24:09 smithi097 ceph-mon[33317]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi097", "root=default"]}]: dispatch 2024-03-11T21:24:09.828 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:24:09 smithi097 ceph-mon[33317]: from='mgr.14150 v1:172.21.15.92:0/996057664' entity='mgr.y' 2024-03-11T21:24:09.828 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:24:09 smithi097 ceph-mon[33317]: from='mgr.14150 v1:172.21.15.92:0/996057664' entity='mgr.y' 2024-03-11T21:24:09.828 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:24:09 smithi097 ceph-mon[33317]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi097", "root=default"]}]': finished 2024-03-11T21:24:09.828 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:24:09 smithi097 ceph-mon[33317]: osdmap e31: 5 total, 4 up, 5 in 2024-03-11T21:24:09.828 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:24:09 smithi097 ceph-mon[33317]: from='mgr.14150 v1:172.21.15.92:0/996057664' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-11T21:24:09.829 INFO:journalctl@ceph.osd.4.smithi097.stdout:Mar 11 21:24:09 smithi097 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd-4[38221]: 2024-03-11T21:24:09.568+0000 7f9fac174700 -1 osd.4 0 waiting for initial osdmap 2024-03-11T21:24:09.829 INFO:journalctl@ceph.osd.4.smithi097.stdout:Mar 11 21:24:09 smithi097 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd-4[38221]: 2024-03-11T21:24:09.573+0000 7f9fa7f67700 -1 osd.4 31 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-11T21:24:09.862 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:24:09 smithi092 ceph-mon[29572]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]': finished 2024-03-11T21:24:09.862 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:24:09 smithi092 ceph-mon[29572]: from='osd.4 v1:172.21.15.97:6800/420733620' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi097", "root=default"]}]: dispatch 2024-03-11T21:24:09.862 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:24:09 smithi092 ceph-mon[29572]: osdmap e30: 5 total, 4 up, 5 in 2024-03-11T21:24:09.862 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:24:09 smithi092 ceph-mon[29572]: from='mgr.14150 v1:172.21.15.92:0/996057664' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-11T21:24:09.862 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:24:09 smithi092 ceph-mon[29572]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi097", "root=default"]}]: dispatch 2024-03-11T21:24:09.862 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:24:09 smithi092 ceph-mon[29572]: from='mgr.14150 v1:172.21.15.92:0/996057664' entity='mgr.y' 2024-03-11T21:24:09.862 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:24:09 smithi092 ceph-mon[29572]: from='mgr.14150 v1:172.21.15.92:0/996057664' entity='mgr.y' 2024-03-11T21:24:09.862 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:24:09 smithi092 ceph-mon[29572]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi097", "root=default"]}]': finished 2024-03-11T21:24:09.862 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:24:09 smithi092 ceph-mon[29572]: osdmap e31: 5 total, 4 up, 5 in 2024-03-11T21:24:09.863 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:24:09 smithi092 ceph-mon[29572]: from='mgr.14150 v1:172.21.15.92:0/996057664' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-11T21:24:09.863 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:24:09 smithi092 ceph-mon[37563]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]': finished 2024-03-11T21:24:09.863 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:24:09 smithi092 ceph-mon[37563]: from='osd.4 v1:172.21.15.97:6800/420733620' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi097", "root=default"]}]: dispatch 2024-03-11T21:24:09.863 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:24:09 smithi092 ceph-mon[37563]: osdmap e30: 5 total, 4 up, 5 in 2024-03-11T21:24:09.863 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:24:09 smithi092 ceph-mon[37563]: from='mgr.14150 v1:172.21.15.92:0/996057664' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-11T21:24:09.863 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:24:09 smithi092 ceph-mon[37563]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi097", "root=default"]}]: dispatch 2024-03-11T21:24:09.863 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:24:09 smithi092 ceph-mon[37563]: from='mgr.14150 v1:172.21.15.92:0/996057664' entity='mgr.y' 2024-03-11T21:24:09.863 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:24:09 smithi092 ceph-mon[37563]: from='mgr.14150 v1:172.21.15.92:0/996057664' entity='mgr.y' 2024-03-11T21:24:09.863 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:24:09 smithi092 ceph-mon[37563]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi097", "root=default"]}]': finished 2024-03-11T21:24:09.863 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:24:09 smithi092 ceph-mon[37563]: osdmap e31: 5 total, 4 up, 5 in 2024-03-11T21:24:09.863 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:24:09 smithi092 ceph-mon[37563]: from='mgr.14150 v1:172.21.15.92:0/996057664' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-11T21:24:10.804 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:24:10 smithi097 ceph-mon[33317]: pgmap v110: 1 pgs: 1 active+clean; 577 KiB data, 112 MiB used, 357 GiB / 358 GiB avail 2024-03-11T21:24:10.804 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:24:10 smithi097 ceph-mon[33317]: osd.4 v1:172.21.15.97:6800/420733620 boot 2024-03-11T21:24:10.804 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:24:10 smithi097 ceph-mon[33317]: osdmap e32: 5 total, 5 up, 5 in 2024-03-11T21:24:10.804 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:24:10 smithi097 ceph-mon[33317]: from='mgr.14150 v1:172.21.15.92:0/996057664' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-11T21:24:10.861 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:24:10 smithi092 ceph-mon[29572]: pgmap v110: 1 pgs: 1 active+clean; 577 KiB data, 112 MiB used, 357 GiB / 358 GiB avail 2024-03-11T21:24:10.862 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:24:10 smithi092 ceph-mon[29572]: osd.4 v1:172.21.15.97:6800/420733620 boot 2024-03-11T21:24:10.862 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:24:10 smithi092 ceph-mon[29572]: osdmap e32: 5 total, 5 up, 5 in 2024-03-11T21:24:10.862 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:24:10 smithi092 ceph-mon[29572]: from='mgr.14150 v1:172.21.15.92:0/996057664' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-11T21:24:10.862 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:24:10 smithi092 ceph-mon[37563]: pgmap v110: 1 pgs: 1 active+clean; 577 KiB data, 112 MiB used, 357 GiB / 358 GiB avail 2024-03-11T21:24:10.862 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:24:10 smithi092 ceph-mon[37563]: osd.4 v1:172.21.15.97:6800/420733620 boot 2024-03-11T21:24:10.862 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:24:10 smithi092 ceph-mon[37563]: osdmap e32: 5 total, 5 up, 5 in 2024-03-11T21:24:10.862 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:24:10 smithi092 ceph-mon[37563]: from='mgr.14150 v1:172.21.15.92:0/996057664' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-11T21:24:11.827 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:24:11 smithi097 ceph-mon[33317]: purged_snaps scrub starts 2024-03-11T21:24:11.828 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:24:11 smithi097 ceph-mon[33317]: purged_snaps scrub ok 2024-03-11T21:24:11.828 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:24:11 smithi097 ceph-mon[33317]: osdmap e33: 5 total, 5 up, 5 in 2024-03-11T21:24:11.861 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:24:11 smithi092 ceph-mon[29572]: purged_snaps scrub starts 2024-03-11T21:24:11.862 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:24:11 smithi092 ceph-mon[29572]: purged_snaps scrub ok 2024-03-11T21:24:11.862 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:24:11 smithi092 ceph-mon[29572]: osdmap e33: 5 total, 5 up, 5 in 2024-03-11T21:24:11.862 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:24:11 smithi092 ceph-mon[37563]: purged_snaps scrub starts 2024-03-11T21:24:11.862 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:24:11 smithi092 ceph-mon[37563]: purged_snaps scrub ok 2024-03-11T21:24:11.862 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:24:11 smithi092 ceph-mon[37563]: osdmap e33: 5 total, 5 up, 5 in 2024-03-11T21:24:11.875 INFO:teuthology.orchestra.run.smithi097.stderr:--> Zapping: /dev/vg_nvme/lv_3 2024-03-11T21:24:11.875 INFO:teuthology.orchestra.run.smithi097.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_3 bs=1M count=10 conv=fsync 2024-03-11T21:24:11.875 INFO:teuthology.orchestra.run.smithi097.stderr: stderr: 10+0 records in 2024-03-11T21:24:11.875 INFO:teuthology.orchestra.run.smithi097.stderr:10+0 records out 2024-03-11T21:24:11.875 INFO:teuthology.orchestra.run.smithi097.stderr:10485760 bytes (10 MB, 10 MiB) copied, 0.0122217 s, 858 MB/s 2024-03-11T21:24:11.875 INFO:teuthology.orchestra.run.smithi097.stderr:--> Zapping successful for: 2024-03-11T21:24:12.782 DEBUG:teuthology.orchestra.run.smithi097:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 00db0016-dfed-11ee-95c8-87774f69a715 -- ceph orch daemon add osd smithi097:vg_nvme/lv_3 2024-03-11T21:24:12.827 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:24:12 smithi097 ceph-mon[33317]: pgmap v113: 1 pgs: 1 active+clean; 577 KiB data, 139 MiB used, 447 GiB / 447 GiB avail 2024-03-11T21:24:12.861 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:24:12 smithi092 ceph-mon[29572]: pgmap v113: 1 pgs: 1 active+clean; 577 KiB data, 139 MiB used, 447 GiB / 447 GiB avail 2024-03-11T21:24:12.862 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:24:12 smithi092 ceph-mon[37563]: pgmap v113: 1 pgs: 1 active+clean; 577 KiB data, 139 MiB used, 447 GiB / 447 GiB avail 2024-03-11T21:24:13.043 INFO:teuthology.orchestra.run.smithi097.stderr:Inferring config /var/lib/ceph/00db0016-dfed-11ee-95c8-87774f69a715/mon.b/config 2024-03-11T21:24:13.734 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:24:13 smithi097 ceph-mon[33317]: osdmap e34: 5 total, 5 up, 5 in 2024-03-11T21:24:13.735 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:24:13 smithi097 ceph-mon[33317]: osdmap e35: 5 total, 5 up, 5 in 2024-03-11T21:24:13.861 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:24:13 smithi092 ceph-mon[29572]: osdmap e34: 5 total, 5 up, 5 in 2024-03-11T21:24:13.862 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:24:13 smithi092 ceph-mon[29572]: osdmap e35: 5 total, 5 up, 5 in 2024-03-11T21:24:13.862 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:24:13 smithi092 ceph-mon[37563]: osdmap e34: 5 total, 5 up, 5 in 2024-03-11T21:24:13.862 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:24:13 smithi092 ceph-mon[37563]: osdmap e35: 5 total, 5 up, 5 in 2024-03-11T21:24:14.828 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:24:14 smithi097 ceph-mon[33317]: pgmap v117: 1 pgs: 1 active+undersized+remapped; 577 KiB data, 139 MiB used, 447 GiB / 447 GiB avail; 146 KiB/s, 0 objects/s recovering 2024-03-11T21:24:14.828 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:24:14 smithi097 ceph-mon[33317]: osdmap e36: 5 total, 5 up, 5 in 2024-03-11T21:24:14.828 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:24:14 smithi097 ceph-mon[33317]: from='mgr.14150 v1:172.21.15.92:0/996057664' entity='mgr.y' 2024-03-11T21:24:14.828 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:24:14 smithi097 ceph-mon[33317]: from='mgr.14150 v1:172.21.15.92:0/996057664' entity='mgr.y' 2024-03-11T21:24:14.828 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:24:14 smithi097 ceph-mon[33317]: from='mgr.14150 v1:172.21.15.92:0/996057664' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-11T21:24:14.828 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:24:14 smithi097 ceph-mon[33317]: from='mgr.14150 v1:172.21.15.92:0/996057664' entity='mgr.y' 2024-03-11T21:24:14.828 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:24:14 smithi097 ceph-mon[33317]: from='mgr.14150 v1:172.21.15.92:0/996057664' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T21:24:14.828 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:24:14 smithi097 ceph-mon[33317]: from='mgr.14150 v1:172.21.15.92:0/996057664' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T21:24:14.828 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:24:14 smithi097 ceph-mon[33317]: from='mgr.14150 v1:172.21.15.92:0/996057664' entity='mgr.y' 2024-03-11T21:24:14.828 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:24:14 smithi097 ceph-mon[33317]: from='mgr.14150 v1:172.21.15.92:0/996057664' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T21:24:14.828 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:24:14 smithi097 ceph-mon[33317]: from='mgr.14150 v1:172.21.15.92:0/996057664' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T21:24:14.828 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:24:14 smithi097 ceph-mon[33317]: from='mgr.14150 v1:172.21.15.92:0/996057664' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T21:24:14.829 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:24:14 smithi097 ceph-mon[33317]: from='mgr.14150 v1:172.21.15.92:0/996057664' entity='mgr.y' 2024-03-11T21:24:14.862 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:24:14 smithi092 ceph-mon[29572]: pgmap v117: 1 pgs: 1 active+undersized+remapped; 577 KiB data, 139 MiB used, 447 GiB / 447 GiB avail; 146 KiB/s, 0 objects/s recovering 2024-03-11T21:24:14.862 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:24:14 smithi092 ceph-mon[29572]: osdmap e36: 5 total, 5 up, 5 in 2024-03-11T21:24:14.862 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:24:14 smithi092 ceph-mon[29572]: from='mgr.14150 v1:172.21.15.92:0/996057664' entity='mgr.y' 2024-03-11T21:24:14.862 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:24:14 smithi092 ceph-mon[29572]: from='mgr.14150 v1:172.21.15.92:0/996057664' entity='mgr.y' 2024-03-11T21:24:14.862 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:24:14 smithi092 ceph-mon[29572]: from='mgr.14150 v1:172.21.15.92:0/996057664' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-11T21:24:14.862 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:24:14 smithi092 ceph-mon[29572]: from='mgr.14150 v1:172.21.15.92:0/996057664' entity='mgr.y' 2024-03-11T21:24:14.862 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:24:14 smithi092 ceph-mon[29572]: from='mgr.14150 v1:172.21.15.92:0/996057664' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T21:24:14.863 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:24:14 smithi092 ceph-mon[29572]: from='mgr.14150 v1:172.21.15.92:0/996057664' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T21:24:14.863 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:24:14 smithi092 ceph-mon[29572]: from='mgr.14150 v1:172.21.15.92:0/996057664' entity='mgr.y' 2024-03-11T21:24:14.863 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:24:14 smithi092 ceph-mon[29572]: from='mgr.14150 v1:172.21.15.92:0/996057664' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T21:24:14.863 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:24:14 smithi092 ceph-mon[29572]: from='mgr.14150 v1:172.21.15.92:0/996057664' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T21:24:14.863 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:24:14 smithi092 ceph-mon[29572]: from='mgr.14150 v1:172.21.15.92:0/996057664' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T21:24:14.863 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:24:14 smithi092 ceph-mon[29572]: from='mgr.14150 v1:172.21.15.92:0/996057664' entity='mgr.y' 2024-03-11T21:24:14.863 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:24:14 smithi092 ceph-mon[37563]: pgmap v117: 1 pgs: 1 active+undersized+remapped; 577 KiB data, 139 MiB used, 447 GiB / 447 GiB avail; 146 KiB/s, 0 objects/s recovering 2024-03-11T21:24:14.863 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:24:14 smithi092 ceph-mon[37563]: osdmap e36: 5 total, 5 up, 5 in 2024-03-11T21:24:14.863 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:24:14 smithi092 ceph-mon[37563]: from='mgr.14150 v1:172.21.15.92:0/996057664' entity='mgr.y' 2024-03-11T21:24:14.863 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:24:14 smithi092 ceph-mon[37563]: from='mgr.14150 v1:172.21.15.92:0/996057664' entity='mgr.y' 2024-03-11T21:24:14.863 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:24:14 smithi092 ceph-mon[37563]: from='mgr.14150 v1:172.21.15.92:0/996057664' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-11T21:24:14.864 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:24:14 smithi092 ceph-mon[37563]: from='mgr.14150 v1:172.21.15.92:0/996057664' entity='mgr.y' 2024-03-11T21:24:14.864 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:24:14 smithi092 ceph-mon[37563]: from='mgr.14150 v1:172.21.15.92:0/996057664' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T21:24:14.864 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:24:14 smithi092 ceph-mon[37563]: from='mgr.14150 v1:172.21.15.92:0/996057664' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T21:24:14.864 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:24:14 smithi092 ceph-mon[37563]: from='mgr.14150 v1:172.21.15.92:0/996057664' entity='mgr.y' 2024-03-11T21:24:14.864 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:24:14 smithi092 ceph-mon[37563]: from='mgr.14150 v1:172.21.15.92:0/996057664' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T21:24:14.864 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:24:14 smithi092 ceph-mon[37563]: from='mgr.14150 v1:172.21.15.92:0/996057664' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T21:24:14.864 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:24:14 smithi092 ceph-mon[37563]: from='mgr.14150 v1:172.21.15.92:0/996057664' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T21:24:14.864 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:24:14 smithi092 ceph-mon[37563]: from='mgr.14150 v1:172.21.15.92:0/996057664' entity='mgr.y' 2024-03-11T21:24:15.827 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:24:15 smithi097 ceph-mon[33317]: Detected new or changed devices on smithi097 2024-03-11T21:24:15.827 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:24:15 smithi097 ceph-mon[33317]: Adjusting osd_memory_target on smithi097 to 17088M 2024-03-11T21:24:15.827 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:24:15 smithi097 ceph-mon[33317]: from='mgr.14150 v1:172.21.15.92:0/996057664' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-11T21:24:15.828 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:24:15 smithi097 ceph-mon[33317]: from='mgr.14150 v1:172.21.15.92:0/996057664' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-11T21:24:15.828 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:24:15 smithi097 ceph-mon[33317]: from='mgr.14150 v1:172.21.15.92:0/996057664' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T21:24:15.862 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:24:15 smithi092 ceph-mon[29572]: Detected new or changed devices on smithi097 2024-03-11T21:24:15.862 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:24:15 smithi092 ceph-mon[29572]: Adjusting osd_memory_target on smithi097 to 17088M 2024-03-11T21:24:15.862 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:24:15 smithi092 ceph-mon[29572]: from='mgr.14150 v1:172.21.15.92:0/996057664' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-11T21:24:15.862 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:24:15 smithi092 ceph-mon[29572]: from='mgr.14150 v1:172.21.15.92:0/996057664' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-11T21:24:15.862 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:24:15 smithi092 ceph-mon[29572]: from='mgr.14150 v1:172.21.15.92:0/996057664' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T21:24:15.862 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:24:15 smithi092 ceph-mon[37563]: Detected new or changed devices on smithi097 2024-03-11T21:24:15.862 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:24:15 smithi092 ceph-mon[37563]: Adjusting osd_memory_target on smithi097 to 17088M 2024-03-11T21:24:15.862 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:24:15 smithi092 ceph-mon[37563]: from='mgr.14150 v1:172.21.15.92:0/996057664' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-11T21:24:15.863 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:24:15 smithi092 ceph-mon[37563]: from='mgr.14150 v1:172.21.15.92:0/996057664' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-11T21:24:15.863 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:24:15 smithi092 ceph-mon[37563]: from='mgr.14150 v1:172.21.15.92:0/996057664' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T21:24:16.827 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:24:16 smithi097 ceph-mon[33317]: from='client.24259 v1:172.21.15.97:0/658516127' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi097:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T21:24:16.828 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:24:16 smithi097 ceph-mon[33317]: pgmap v119: 1 pgs: 1 active+undersized+remapped; 577 KiB data, 139 MiB used, 447 GiB / 447 GiB avail; 144 KiB/s, 0 objects/s recovering 2024-03-11T21:24:16.861 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:24:16 smithi092 ceph-mon[29572]: from='client.24259 v1:172.21.15.97:0/658516127' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi097:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T21:24:16.862 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:24:16 smithi092 ceph-mon[29572]: pgmap v119: 1 pgs: 1 active+undersized+remapped; 577 KiB data, 139 MiB used, 447 GiB / 447 GiB avail; 144 KiB/s, 0 objects/s recovering 2024-03-11T21:24:16.862 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:24:16 smithi092 ceph-mon[37563]: from='client.24259 v1:172.21.15.97:0/658516127' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi097:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T21:24:16.862 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:24:16 smithi092 ceph-mon[37563]: pgmap v119: 1 pgs: 1 active+undersized+remapped; 577 KiB data, 139 MiB used, 447 GiB / 447 GiB avail; 144 KiB/s, 0 objects/s recovering 2024-03-11T21:24:18.828 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:24:18 smithi097 ceph-mon[33317]: pgmap v120: 1 pgs: 1 active+undersized+remapped; 577 KiB data, 140 MiB used, 447 GiB / 447 GiB avail; 97 KiB/s, 0 objects/s recovering 2024-03-11T21:24:18.828 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:24:18 smithi097 ceph-mon[33317]: from='client.? v1:172.21.15.97:0/4205571460' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "593a6f6b-8874-40ae-8848-d51c5723a5e4"}]: dispatch 2024-03-11T21:24:18.828 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:24:18 smithi097 ceph-mon[33317]: from='client.24265 ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "593a6f6b-8874-40ae-8848-d51c5723a5e4"}]: dispatch 2024-03-11T21:24:18.828 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:24:18 smithi097 ceph-mon[33317]: from='client.24265 ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "593a6f6b-8874-40ae-8848-d51c5723a5e4"}]': finished 2024-03-11T21:24:18.828 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:24:18 smithi097 ceph-mon[33317]: osdmap e37: 6 total, 5 up, 6 in 2024-03-11T21:24:18.828 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:24:18 smithi097 ceph-mon[33317]: from='mgr.14150 v1:172.21.15.92:0/996057664' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-11T21:24:18.828 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:24:18 smithi097 ceph-mon[33317]: from='client.? v1:172.21.15.97:0/3717401712' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-11T21:24:18.863 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:24:18 smithi092 ceph-mon[29572]: pgmap v120: 1 pgs: 1 active+undersized+remapped; 577 KiB data, 140 MiB used, 447 GiB / 447 GiB avail; 97 KiB/s, 0 objects/s recovering 2024-03-11T21:24:18.863 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:24:18 smithi092 ceph-mon[29572]: from='client.? v1:172.21.15.97:0/4205571460' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "593a6f6b-8874-40ae-8848-d51c5723a5e4"}]: dispatch 2024-03-11T21:24:18.863 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:24:18 smithi092 ceph-mon[29572]: from='client.24265 ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "593a6f6b-8874-40ae-8848-d51c5723a5e4"}]: dispatch 2024-03-11T21:24:18.863 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:24:18 smithi092 ceph-mon[29572]: from='client.24265 ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "593a6f6b-8874-40ae-8848-d51c5723a5e4"}]': finished 2024-03-11T21:24:18.863 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:24:18 smithi092 ceph-mon[29572]: osdmap e37: 6 total, 5 up, 6 in 2024-03-11T21:24:18.863 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:24:18 smithi092 ceph-mon[29572]: from='mgr.14150 v1:172.21.15.92:0/996057664' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-11T21:24:18.864 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:24:18 smithi092 ceph-mon[29572]: from='client.? v1:172.21.15.97:0/3717401712' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-11T21:24:18.864 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:24:18 smithi092 ceph-mon[37563]: pgmap v120: 1 pgs: 1 active+undersized+remapped; 577 KiB data, 140 MiB used, 447 GiB / 447 GiB avail; 97 KiB/s, 0 objects/s recovering 2024-03-11T21:24:18.864 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:24:18 smithi092 ceph-mon[37563]: from='client.? v1:172.21.15.97:0/4205571460' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "593a6f6b-8874-40ae-8848-d51c5723a5e4"}]: dispatch 2024-03-11T21:24:18.864 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:24:18 smithi092 ceph-mon[37563]: from='client.24265 ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "593a6f6b-8874-40ae-8848-d51c5723a5e4"}]: dispatch 2024-03-11T21:24:18.864 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:24:18 smithi092 ceph-mon[37563]: from='client.24265 ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "593a6f6b-8874-40ae-8848-d51c5723a5e4"}]': finished 2024-03-11T21:24:18.864 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:24:18 smithi092 ceph-mon[37563]: osdmap e37: 6 total, 5 up, 6 in 2024-03-11T21:24:18.864 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:24:18 smithi092 ceph-mon[37563]: from='mgr.14150 v1:172.21.15.92:0/996057664' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-11T21:24:18.864 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:24:18 smithi092 ceph-mon[37563]: from='client.? v1:172.21.15.97:0/3717401712' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-11T21:24:20.827 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:24:20 smithi097 ceph-mon[33317]: pgmap v122: 1 pgs: 1 active+clean; 577 KiB data, 139 MiB used, 447 GiB / 447 GiB avail 2024-03-11T21:24:20.861 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:24:20 smithi092 ceph-mon[29572]: pgmap v122: 1 pgs: 1 active+clean; 577 KiB data, 139 MiB used, 447 GiB / 447 GiB avail 2024-03-11T21:24:20.862 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:24:20 smithi092 ceph-mon[37563]: pgmap v122: 1 pgs: 1 active+clean; 577 KiB data, 139 MiB used, 447 GiB / 447 GiB avail 2024-03-11T21:24:22.828 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:24:22 smithi097 ceph-mon[33317]: pgmap v123: 1 pgs: 1 active+clean; 577 KiB data, 139 MiB used, 447 GiB / 447 GiB avail 2024-03-11T21:24:22.862 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:24:22 smithi092 ceph-mon[29572]: pgmap v123: 1 pgs: 1 active+clean; 577 KiB data, 139 MiB used, 447 GiB / 447 GiB avail 2024-03-11T21:24:22.862 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:24:22 smithi092 ceph-mon[37563]: pgmap v123: 1 pgs: 1 active+clean; 577 KiB data, 139 MiB used, 447 GiB / 447 GiB avail 2024-03-11T21:24:24.321 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:24:24 smithi097 ceph-mon[33317]: pgmap v124: 1 pgs: 1 active+clean; 577 KiB data, 139 MiB used, 447 GiB / 447 GiB avail 2024-03-11T21:24:24.361 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:24:24 smithi092 ceph-mon[29572]: pgmap v124: 1 pgs: 1 active+clean; 577 KiB data, 139 MiB used, 447 GiB / 447 GiB avail 2024-03-11T21:24:24.362 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:24:24 smithi092 ceph-mon[37563]: pgmap v124: 1 pgs: 1 active+clean; 577 KiB data, 139 MiB used, 447 GiB / 447 GiB avail 2024-03-11T21:24:25.291 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:24:25 smithi097 ceph-mon[33317]: from='mgr.14150 v1:172.21.15.92:0/996057664' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.5"}]: dispatch 2024-03-11T21:24:25.291 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:24:25 smithi097 ceph-mon[33317]: from='mgr.14150 v1:172.21.15.92:0/996057664' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T21:24:25.291 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:24:25 smithi097 ceph-mon[33317]: Deploying daemon osd.5 on smithi097 2024-03-11T21:24:25.361 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:24:25 smithi092 ceph-mon[29572]: from='mgr.14150 v1:172.21.15.92:0/996057664' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.5"}]: dispatch 2024-03-11T21:24:25.362 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:24:25 smithi092 ceph-mon[29572]: from='mgr.14150 v1:172.21.15.92:0/996057664' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T21:24:25.362 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:24:25 smithi092 ceph-mon[29572]: Deploying daemon osd.5 on smithi097 2024-03-11T21:24:25.362 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:24:25 smithi092 ceph-mon[37563]: from='mgr.14150 v1:172.21.15.92:0/996057664' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.5"}]: dispatch 2024-03-11T21:24:25.362 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:24:25 smithi092 ceph-mon[37563]: from='mgr.14150 v1:172.21.15.92:0/996057664' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T21:24:25.362 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:24:25 smithi092 ceph-mon[37563]: Deploying daemon osd.5 on smithi097 2024-03-11T21:24:26.185 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:24:26 smithi097 ceph-mon[33317]: pgmap v125: 1 pgs: 1 active+clean; 577 KiB data, 139 MiB used, 447 GiB / 447 GiB avail 2024-03-11T21:24:26.361 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:24:26 smithi092 ceph-mon[29572]: pgmap v125: 1 pgs: 1 active+clean; 577 KiB data, 139 MiB used, 447 GiB / 447 GiB avail 2024-03-11T21:24:26.362 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:24:26 smithi092 ceph-mon[37563]: pgmap v125: 1 pgs: 1 active+clean; 577 KiB data, 139 MiB used, 447 GiB / 447 GiB avail 2024-03-11T21:24:28.828 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:24:28 smithi097 ceph-mon[33317]: pgmap v126: 1 pgs: 1 active+clean; 577 KiB data, 139 MiB used, 447 GiB / 447 GiB avail 2024-03-11T21:24:28.861 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:24:28 smithi092 ceph-mon[29572]: pgmap v126: 1 pgs: 1 active+clean; 577 KiB data, 139 MiB used, 447 GiB / 447 GiB avail 2024-03-11T21:24:28.862 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:24:28 smithi092 ceph-mon[37563]: pgmap v126: 1 pgs: 1 active+clean; 577 KiB data, 139 MiB used, 447 GiB / 447 GiB avail 2024-03-11T21:24:29.736 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:24:29 smithi097 ceph-mon[33317]: purged_snaps scrub starts 2024-03-11T21:24:29.736 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:24:29 smithi097 ceph-mon[33317]: purged_snaps scrub ok 2024-03-11T21:24:29.863 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:24:29 smithi092 ceph-mon[29572]: purged_snaps scrub starts 2024-03-11T21:24:29.863 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:24:29 smithi092 ceph-mon[29572]: purged_snaps scrub ok 2024-03-11T21:24:29.863 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:24:29 smithi092 ceph-mon[37563]: purged_snaps scrub starts 2024-03-11T21:24:29.863 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:24:29 smithi092 ceph-mon[37563]: purged_snaps scrub ok 2024-03-11T21:24:30.827 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:24:30 smithi097 ceph-mon[33317]: pgmap v127: 1 pgs: 1 active+clean; 577 KiB data, 139 MiB used, 447 GiB / 447 GiB avail 2024-03-11T21:24:30.827 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:24:30 smithi097 ceph-mon[33317]: osdmap e38: 6 total, 5 up, 6 in 2024-03-11T21:24:30.828 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:24:30 smithi097 ceph-mon[33317]: from='mgr.14150 v1:172.21.15.92:0/996057664' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-11T21:24:30.828 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:24:30 smithi097 ceph-mon[33317]: from='mgr.14150 v1:172.21.15.92:0/996057664' entity='mgr.y' 2024-03-11T21:24:30.828 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:24:30 smithi097 ceph-mon[33317]: from='mgr.14150 v1:172.21.15.92:0/996057664' entity='mgr.y' 2024-03-11T21:24:30.828 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:24:30 smithi097 ceph-mon[33317]: from='mgr.14150 v1:172.21.15.92:0/996057664' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T21:24:30.862 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:24:30 smithi092 ceph-mon[37563]: pgmap v127: 1 pgs: 1 active+clean; 577 KiB data, 139 MiB used, 447 GiB / 447 GiB avail 2024-03-11T21:24:30.862 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:24:30 smithi092 ceph-mon[37563]: osdmap e38: 6 total, 5 up, 6 in 2024-03-11T21:24:30.862 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:24:30 smithi092 ceph-mon[37563]: from='mgr.14150 v1:172.21.15.92:0/996057664' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-11T21:24:30.862 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:24:30 smithi092 ceph-mon[37563]: from='mgr.14150 v1:172.21.15.92:0/996057664' entity='mgr.y' 2024-03-11T21:24:30.862 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:24:30 smithi092 ceph-mon[37563]: from='mgr.14150 v1:172.21.15.92:0/996057664' entity='mgr.y' 2024-03-11T21:24:30.862 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:24:30 smithi092 ceph-mon[37563]: from='mgr.14150 v1:172.21.15.92:0/996057664' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T21:24:30.862 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:24:30 smithi092 ceph-mon[29572]: pgmap v127: 1 pgs: 1 active+clean; 577 KiB data, 139 MiB used, 447 GiB / 447 GiB avail 2024-03-11T21:24:30.862 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:24:30 smithi092 ceph-mon[29572]: osdmap e38: 6 total, 5 up, 6 in 2024-03-11T21:24:30.863 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:24:30 smithi092 ceph-mon[29572]: from='mgr.14150 v1:172.21.15.92:0/996057664' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-11T21:24:30.863 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:24:30 smithi092 ceph-mon[29572]: from='mgr.14150 v1:172.21.15.92:0/996057664' entity='mgr.y' 2024-03-11T21:24:30.863 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:24:30 smithi092 ceph-mon[29572]: from='mgr.14150 v1:172.21.15.92:0/996057664' entity='mgr.y' 2024-03-11T21:24:30.863 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:24:30 smithi092 ceph-mon[29572]: from='mgr.14150 v1:172.21.15.92:0/996057664' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T21:24:32.828 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:24:32 smithi097 ceph-mon[33317]: pgmap v129: 1 pgs: 1 active+clean; 577 KiB data, 139 MiB used, 447 GiB / 447 GiB avail 2024-03-11T21:24:32.861 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:24:32 smithi092 ceph-mon[29572]: pgmap v129: 1 pgs: 1 active+clean; 577 KiB data, 139 MiB used, 447 GiB / 447 GiB avail 2024-03-11T21:24:32.862 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:24:32 smithi092 ceph-mon[37563]: pgmap v129: 1 pgs: 1 active+clean; 577 KiB data, 139 MiB used, 447 GiB / 447 GiB avail 2024-03-11T21:24:33.040 INFO:teuthology.orchestra.run.smithi097.stdout:Created osd(s) 5 on host 'smithi097' 2024-03-11T21:24:33.634 DEBUG:teuthology.orchestra.run.smithi097:osd.5> sudo journalctl -f -n 0 -u ceph-00db0016-dfed-11ee-95c8-87774f69a715@osd.5.service 2024-03-11T21:24:33.636 INFO:tasks.cephadm:Deploying osd.6 on smithi097 with /dev/vg_nvme/lv_2... 2024-03-11T21:24:33.636 DEBUG:teuthology.orchestra.run.smithi097:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 00db0016-dfed-11ee-95c8-87774f69a715 -- ceph-volume lvm zap /dev/vg_nvme/lv_2 2024-03-11T21:24:33.702 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:24:33 smithi097 ceph-mon[33317]: from='osd.5 v1:172.21.15.97:6804/312056041' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-03-11T21:24:33.702 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:24:33 smithi097 ceph-mon[33317]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-03-11T21:24:33.702 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:24:33 smithi097 ceph-mon[33317]: from='mgr.14150 v1:172.21.15.92:0/996057664' entity='mgr.y' 2024-03-11T21:24:33.702 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:24:33 smithi097 ceph-mon[33317]: from='mgr.14150 v1:172.21.15.92:0/996057664' entity='mgr.y' 2024-03-11T21:24:33.702 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:24:33 smithi097 ceph-mon[33317]: from='mgr.14150 v1:172.21.15.92:0/996057664' entity='mgr.y' 2024-03-11T21:24:33.702 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:24:33 smithi097 ceph-mon[33317]: from='mgr.14150 v1:172.21.15.92:0/996057664' entity='mgr.y' 2024-03-11T21:24:33.862 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:24:33 smithi092 ceph-mon[29572]: from='osd.5 v1:172.21.15.97:6804/312056041' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-03-11T21:24:33.862 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:24:33 smithi092 ceph-mon[29572]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-03-11T21:24:33.862 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:24:33 smithi092 ceph-mon[29572]: from='mgr.14150 v1:172.21.15.92:0/996057664' entity='mgr.y' 2024-03-11T21:24:33.862 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:24:33 smithi092 ceph-mon[29572]: from='mgr.14150 v1:172.21.15.92:0/996057664' entity='mgr.y' 2024-03-11T21:24:33.862 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:24:33 smithi092 ceph-mon[29572]: from='mgr.14150 v1:172.21.15.92:0/996057664' entity='mgr.y' 2024-03-11T21:24:33.862 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:24:33 smithi092 ceph-mon[29572]: from='mgr.14150 v1:172.21.15.92:0/996057664' entity='mgr.y' 2024-03-11T21:24:33.862 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:24:33 smithi092 ceph-mon[37563]: from='osd.5 v1:172.21.15.97:6804/312056041' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-03-11T21:24:33.863 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:24:33 smithi092 ceph-mon[37563]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-03-11T21:24:33.863 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:24:33 smithi092 ceph-mon[37563]: from='mgr.14150 v1:172.21.15.92:0/996057664' entity='mgr.y' 2024-03-11T21:24:33.863 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:24:33 smithi092 ceph-mon[37563]: from='mgr.14150 v1:172.21.15.92:0/996057664' entity='mgr.y' 2024-03-11T21:24:33.863 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:24:33 smithi092 ceph-mon[37563]: from='mgr.14150 v1:172.21.15.92:0/996057664' entity='mgr.y' 2024-03-11T21:24:33.863 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:24:33 smithi092 ceph-mon[37563]: from='mgr.14150 v1:172.21.15.92:0/996057664' entity='mgr.y' 2024-03-11T21:24:34.704 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:24:34 smithi097 ceph-mon[33317]: pgmap v130: 1 pgs: 1 active+clean; 577 KiB data, 139 MiB used, 447 GiB / 447 GiB avail 2024-03-11T21:24:34.704 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:24:34 smithi097 ceph-mon[33317]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]': finished 2024-03-11T21:24:34.704 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:24:34 smithi097 ceph-mon[33317]: from='osd.5 v1:172.21.15.97:6804/312056041' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi097", "root=default"]}]: dispatch 2024-03-11T21:24:34.704 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:24:34 smithi097 ceph-mon[33317]: osdmap e39: 6 total, 5 up, 6 in 2024-03-11T21:24:34.704 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:24:34 smithi097 ceph-mon[33317]: from='mgr.14150 v1:172.21.15.92:0/996057664' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-11T21:24:34.704 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:24:34 smithi097 ceph-mon[33317]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi097", "root=default"]}]: dispatch 2024-03-11T21:24:34.781 INFO:teuthology.orchestra.run.smithi097.stderr:Inferring config /var/lib/ceph/00db0016-dfed-11ee-95c8-87774f69a715/mon.b/config 2024-03-11T21:24:34.862 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:24:34 smithi092 ceph-mon[29572]: pgmap v130: 1 pgs: 1 active+clean; 577 KiB data, 139 MiB used, 447 GiB / 447 GiB avail 2024-03-11T21:24:34.862 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:24:34 smithi092 ceph-mon[29572]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]': finished 2024-03-11T21:24:34.862 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:24:34 smithi092 ceph-mon[29572]: from='osd.5 v1:172.21.15.97:6804/312056041' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi097", "root=default"]}]: dispatch 2024-03-11T21:24:34.862 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:24:34 smithi092 ceph-mon[29572]: osdmap e39: 6 total, 5 up, 6 in 2024-03-11T21:24:34.862 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:24:34 smithi092 ceph-mon[29572]: from='mgr.14150 v1:172.21.15.92:0/996057664' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-11T21:24:34.862 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:24:34 smithi092 ceph-mon[29572]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi097", "root=default"]}]: dispatch 2024-03-11T21:24:34.863 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:24:34 smithi092 ceph-mon[37563]: pgmap v130: 1 pgs: 1 active+clean; 577 KiB data, 139 MiB used, 447 GiB / 447 GiB avail 2024-03-11T21:24:34.863 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:24:34 smithi092 ceph-mon[37563]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]': finished 2024-03-11T21:24:34.863 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:24:34 smithi092 ceph-mon[37563]: from='osd.5 v1:172.21.15.97:6804/312056041' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi097", "root=default"]}]: dispatch 2024-03-11T21:24:34.863 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:24:34 smithi092 ceph-mon[37563]: osdmap e39: 6 total, 5 up, 6 in 2024-03-11T21:24:34.863 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:24:34 smithi092 ceph-mon[37563]: from='mgr.14150 v1:172.21.15.92:0/996057664' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-11T21:24:34.863 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:24:34 smithi092 ceph-mon[37563]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi097", "root=default"]}]: dispatch 2024-03-11T21:24:35.292 INFO:journalctl@ceph.osd.5.smithi097.stdout:Mar 11 21:24:35 smithi097 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd-5[44269]: 2024-03-11T21:24:35.046+0000 7f6c54bf8700 -1 osd.5 0 waiting for initial osdmap 2024-03-11T21:24:35.292 INFO:journalctl@ceph.osd.5.smithi097.stdout:Mar 11 21:24:35 smithi097 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd-5[44269]: 2024-03-11T21:24:35.050+0000 7f6c4c9e3700 -1 osd.5 40 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-11T21:24:35.827 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:24:35 smithi097 ceph-mon[33317]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi097", "root=default"]}]': finished 2024-03-11T21:24:35.827 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:24:35 smithi097 ceph-mon[33317]: osdmap e40: 6 total, 5 up, 6 in 2024-03-11T21:24:35.827 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:24:35 smithi097 ceph-mon[33317]: from='mgr.14150 v1:172.21.15.92:0/996057664' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-11T21:24:35.827 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:24:35 smithi097 ceph-mon[33317]: from='osd.5 ' entity='osd.5' 2024-03-11T21:24:35.828 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:24:35 smithi097 ceph-mon[33317]: osd.5 v1:172.21.15.97:6804/312056041 boot 2024-03-11T21:24:35.828 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:24:35 smithi097 ceph-mon[33317]: osdmap e41: 6 total, 6 up, 6 in 2024-03-11T21:24:35.828 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:24:35 smithi097 ceph-mon[33317]: from='mgr.14150 v1:172.21.15.92:0/996057664' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-11T21:24:35.862 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:24:35 smithi092 ceph-mon[29572]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi097", "root=default"]}]': finished 2024-03-11T21:24:35.862 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:24:35 smithi092 ceph-mon[29572]: osdmap e40: 6 total, 5 up, 6 in 2024-03-11T21:24:35.862 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:24:35 smithi092 ceph-mon[29572]: from='mgr.14150 v1:172.21.15.92:0/996057664' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-11T21:24:35.862 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:24:35 smithi092 ceph-mon[29572]: from='osd.5 ' entity='osd.5' 2024-03-11T21:24:35.862 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:24:35 smithi092 ceph-mon[29572]: osd.5 v1:172.21.15.97:6804/312056041 boot 2024-03-11T21:24:35.862 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:24:35 smithi092 ceph-mon[29572]: osdmap e41: 6 total, 6 up, 6 in 2024-03-11T21:24:35.862 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:24:35 smithi092 ceph-mon[29572]: from='mgr.14150 v1:172.21.15.92:0/996057664' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-11T21:24:35.862 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:24:35 smithi092 ceph-mon[37563]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi097", "root=default"]}]': finished 2024-03-11T21:24:35.863 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:24:35 smithi092 ceph-mon[37563]: osdmap e40: 6 total, 5 up, 6 in 2024-03-11T21:24:35.863 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:24:35 smithi092 ceph-mon[37563]: from='mgr.14150 v1:172.21.15.92:0/996057664' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-11T21:24:35.863 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:24:35 smithi092 ceph-mon[37563]: from='osd.5 ' entity='osd.5' 2024-03-11T21:24:35.863 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:24:35 smithi092 ceph-mon[37563]: osd.5 v1:172.21.15.97:6804/312056041 boot 2024-03-11T21:24:35.863 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:24:35 smithi092 ceph-mon[37563]: osdmap e41: 6 total, 6 up, 6 in 2024-03-11T21:24:35.863 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:24:35 smithi092 ceph-mon[37563]: from='mgr.14150 v1:172.21.15.92:0/996057664' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-11T21:24:36.827 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:24:36 smithi097 ceph-mon[33317]: purged_snaps scrub starts 2024-03-11T21:24:36.827 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:24:36 smithi097 ceph-mon[33317]: purged_snaps scrub ok 2024-03-11T21:24:36.827 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:24:36 smithi097 ceph-mon[33317]: pgmap v133: 1 pgs: 1 active+clean; 577 KiB data, 139 MiB used, 447 GiB / 447 GiB avail 2024-03-11T21:24:36.828 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:24:36 smithi097 ceph-mon[33317]: osdmap e42: 6 total, 6 up, 6 in 2024-03-11T21:24:36.862 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:24:36 smithi092 ceph-mon[29572]: purged_snaps scrub starts 2024-03-11T21:24:36.862 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:24:36 smithi092 ceph-mon[29572]: purged_snaps scrub ok 2024-03-11T21:24:36.862 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:24:36 smithi092 ceph-mon[29572]: pgmap v133: 1 pgs: 1 active+clean; 577 KiB data, 139 MiB used, 447 GiB / 447 GiB avail 2024-03-11T21:24:36.862 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:24:36 smithi092 ceph-mon[29572]: osdmap e42: 6 total, 6 up, 6 in 2024-03-11T21:24:36.862 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:24:36 smithi092 ceph-mon[37563]: purged_snaps scrub starts 2024-03-11T21:24:36.862 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:24:36 smithi092 ceph-mon[37563]: purged_snaps scrub ok 2024-03-11T21:24:36.863 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:24:36 smithi092 ceph-mon[37563]: pgmap v133: 1 pgs: 1 active+clean; 577 KiB data, 139 MiB used, 447 GiB / 447 GiB avail 2024-03-11T21:24:36.863 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:24:36 smithi092 ceph-mon[37563]: osdmap e42: 6 total, 6 up, 6 in 2024-03-11T21:24:37.687 INFO:teuthology.orchestra.run.smithi097.stderr:--> Zapping: /dev/vg_nvme/lv_2 2024-03-11T21:24:37.687 INFO:teuthology.orchestra.run.smithi097.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_2 bs=1M count=10 conv=fsync 2024-03-11T21:24:37.687 INFO:teuthology.orchestra.run.smithi097.stderr: stderr: 10+0 records in 2024-03-11T21:24:37.687 INFO:teuthology.orchestra.run.smithi097.stderr:10+0 records out 2024-03-11T21:24:37.688 INFO:teuthology.orchestra.run.smithi097.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.01235 s, 849 MB/s 2024-03-11T21:24:37.688 INFO:teuthology.orchestra.run.smithi097.stderr:--> Zapping successful for: 2024-03-11T21:24:38.300 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:24:38 smithi097 ceph-mon[33317]: Detected new or changed devices on smithi097 2024-03-11T21:24:38.300 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:24:38 smithi097 ceph-mon[33317]: from='mgr.14150 v1:172.21.15.92:0/996057664' entity='mgr.y' 2024-03-11T21:24:38.300 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:24:38 smithi097 ceph-mon[33317]: from='mgr.14150 v1:172.21.15.92:0/996057664' entity='mgr.y' 2024-03-11T21:24:38.300 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:24:38 smithi097 ceph-mon[33317]: from='mgr.14150 v1:172.21.15.92:0/996057664' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-11T21:24:38.300 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:24:38 smithi097 ceph-mon[33317]: from='mgr.14150 v1:172.21.15.92:0/996057664' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-11T21:24:38.300 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:24:38 smithi097 ceph-mon[33317]: Adjusting osd_memory_target on smithi097 to 8544M 2024-03-11T21:24:38.301 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:24:38 smithi097 ceph-mon[33317]: from='mgr.14150 v1:172.21.15.92:0/996057664' entity='mgr.y' 2024-03-11T21:24:38.301 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:24:38 smithi097 ceph-mon[33317]: from='mgr.14150 v1:172.21.15.92:0/996057664' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T21:24:38.301 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:24:38 smithi097 ceph-mon[33317]: from='mgr.14150 v1:172.21.15.92:0/996057664' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T21:24:38.301 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:24:38 smithi097 ceph-mon[33317]: from='mgr.14150 v1:172.21.15.92:0/996057664' entity='mgr.y' 2024-03-11T21:24:38.301 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:24:38 smithi097 ceph-mon[33317]: from='mgr.14150 v1:172.21.15.92:0/996057664' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T21:24:38.301 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:24:38 smithi097 ceph-mon[33317]: from='mgr.14150 v1:172.21.15.92:0/996057664' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T21:24:38.301 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:24:38 smithi097 ceph-mon[33317]: from='mgr.14150 v1:172.21.15.92:0/996057664' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T21:24:38.301 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:24:38 smithi097 ceph-mon[33317]: from='mgr.14150 v1:172.21.15.92:0/996057664' entity='mgr.y' 2024-03-11T21:24:38.301 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:24:38 smithi097 ceph-mon[33317]: pgmap v136: 1 pgs: 1 active+clean; 577 KiB data, 550 MiB used, 536 GiB / 536 GiB avail 2024-03-11T21:24:38.301 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:24:38 smithi097 ceph-mon[33317]: osdmap e43: 6 total, 6 up, 6 in 2024-03-11T21:24:38.301 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:24:38 smithi097 ceph-mon[33317]: osdmap e44: 6 total, 6 up, 6 in 2024-03-11T21:24:38.327 DEBUG:teuthology.orchestra.run.smithi097:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 00db0016-dfed-11ee-95c8-87774f69a715 -- ceph orch daemon add osd smithi097:vg_nvme/lv_2 2024-03-11T21:24:38.362 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:24:38 smithi092 ceph-mon[37563]: Detected new or changed devices on smithi097 2024-03-11T21:24:38.362 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:24:38 smithi092 ceph-mon[37563]: from='mgr.14150 v1:172.21.15.92:0/996057664' entity='mgr.y' 2024-03-11T21:24:38.362 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:24:38 smithi092 ceph-mon[37563]: from='mgr.14150 v1:172.21.15.92:0/996057664' entity='mgr.y' 2024-03-11T21:24:38.362 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:24:38 smithi092 ceph-mon[37563]: from='mgr.14150 v1:172.21.15.92:0/996057664' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-11T21:24:38.362 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:24:38 smithi092 ceph-mon[37563]: from='mgr.14150 v1:172.21.15.92:0/996057664' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-11T21:24:38.362 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:24:38 smithi092 ceph-mon[37563]: Adjusting osd_memory_target on smithi097 to 8544M 2024-03-11T21:24:38.362 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:24:38 smithi092 ceph-mon[37563]: from='mgr.14150 v1:172.21.15.92:0/996057664' entity='mgr.y' 2024-03-11T21:24:38.363 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:24:38 smithi092 ceph-mon[37563]: from='mgr.14150 v1:172.21.15.92:0/996057664' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T21:24:38.363 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:24:38 smithi092 ceph-mon[37563]: from='mgr.14150 v1:172.21.15.92:0/996057664' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T21:24:38.363 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:24:38 smithi092 ceph-mon[37563]: from='mgr.14150 v1:172.21.15.92:0/996057664' entity='mgr.y' 2024-03-11T21:24:38.363 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:24:38 smithi092 ceph-mon[37563]: from='mgr.14150 v1:172.21.15.92:0/996057664' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T21:24:38.363 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:24:38 smithi092 ceph-mon[37563]: from='mgr.14150 v1:172.21.15.92:0/996057664' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T21:24:38.363 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:24:38 smithi092 ceph-mon[37563]: from='mgr.14150 v1:172.21.15.92:0/996057664' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T21:24:38.363 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:24:38 smithi092 ceph-mon[37563]: from='mgr.14150 v1:172.21.15.92:0/996057664' entity='mgr.y' 2024-03-11T21:24:38.363 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:24:38 smithi092 ceph-mon[37563]: pgmap v136: 1 pgs: 1 active+clean; 577 KiB data, 550 MiB used, 536 GiB / 536 GiB avail 2024-03-11T21:24:38.363 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:24:38 smithi092 ceph-mon[37563]: osdmap e43: 6 total, 6 up, 6 in 2024-03-11T21:24:38.363 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:24:38 smithi092 ceph-mon[37563]: osdmap e44: 6 total, 6 up, 6 in 2024-03-11T21:24:38.363 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:24:38 smithi092 ceph-mon[29572]: Detected new or changed devices on smithi097 2024-03-11T21:24:38.363 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:24:38 smithi092 ceph-mon[29572]: from='mgr.14150 v1:172.21.15.92:0/996057664' entity='mgr.y' 2024-03-11T21:24:38.364 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:24:38 smithi092 ceph-mon[29572]: from='mgr.14150 v1:172.21.15.92:0/996057664' entity='mgr.y' 2024-03-11T21:24:38.364 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:24:38 smithi092 ceph-mon[29572]: from='mgr.14150 v1:172.21.15.92:0/996057664' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-11T21:24:38.364 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:24:38 smithi092 ceph-mon[29572]: from='mgr.14150 v1:172.21.15.92:0/996057664' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-11T21:24:38.364 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:24:38 smithi092 ceph-mon[29572]: Adjusting osd_memory_target on smithi097 to 8544M 2024-03-11T21:24:38.364 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:24:38 smithi092 ceph-mon[29572]: from='mgr.14150 v1:172.21.15.92:0/996057664' entity='mgr.y' 2024-03-11T21:24:38.364 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:24:38 smithi092 ceph-mon[29572]: from='mgr.14150 v1:172.21.15.92:0/996057664' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T21:24:38.364 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:24:38 smithi092 ceph-mon[29572]: from='mgr.14150 v1:172.21.15.92:0/996057664' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T21:24:38.364 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:24:38 smithi092 ceph-mon[29572]: from='mgr.14150 v1:172.21.15.92:0/996057664' entity='mgr.y' 2024-03-11T21:24:38.364 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:24:38 smithi092 ceph-mon[29572]: from='mgr.14150 v1:172.21.15.92:0/996057664' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T21:24:38.364 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:24:38 smithi092 ceph-mon[29572]: from='mgr.14150 v1:172.21.15.92:0/996057664' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T21:24:38.364 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:24:38 smithi092 ceph-mon[29572]: from='mgr.14150 v1:172.21.15.92:0/996057664' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T21:24:38.364 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:24:38 smithi092 ceph-mon[29572]: from='mgr.14150 v1:172.21.15.92:0/996057664' entity='mgr.y' 2024-03-11T21:24:38.364 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:24:38 smithi092 ceph-mon[29572]: pgmap v136: 1 pgs: 1 active+clean; 577 KiB data, 550 MiB used, 536 GiB / 536 GiB avail 2024-03-11T21:24:38.364 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:24:38 smithi092 ceph-mon[29572]: osdmap e43: 6 total, 6 up, 6 in 2024-03-11T21:24:38.364 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:24:38 smithi092 ceph-mon[29572]: osdmap e44: 6 total, 6 up, 6 in 2024-03-11T21:24:38.602 INFO:teuthology.orchestra.run.smithi097.stderr:Inferring config /var/lib/ceph/00db0016-dfed-11ee-95c8-87774f69a715/mon.b/config 2024-03-11T21:24:40.329 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:24:40 smithi097 ceph-mon[33317]: osdmap e45: 6 total, 6 up, 6 in 2024-03-11T21:24:40.329 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:24:40 smithi097 ceph-mon[33317]: pgmap v140: 1 pgs: 1 active+clean; 577 KiB data, 550 MiB used, 536 GiB / 536 GiB avail 2024-03-11T21:24:40.361 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:24:40 smithi092 ceph-mon[29572]: osdmap e45: 6 total, 6 up, 6 in 2024-03-11T21:24:40.362 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:24:40 smithi092 ceph-mon[29572]: pgmap v140: 1 pgs: 1 active+clean; 577 KiB data, 550 MiB used, 536 GiB / 536 GiB avail 2024-03-11T21:24:40.362 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:24:40 smithi092 ceph-mon[37563]: osdmap e45: 6 total, 6 up, 6 in 2024-03-11T21:24:40.362 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:24:40 smithi092 ceph-mon[37563]: pgmap v140: 1 pgs: 1 active+clean; 577 KiB data, 550 MiB used, 536 GiB / 536 GiB avail 2024-03-11T21:24:41.224 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:24:41 smithi097 ceph-mon[33317]: from='client.24286 v1:172.21.15.97:0/2194983662' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi097:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T21:24:41.225 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:24:41 smithi097 ceph-mon[33317]: from='mgr.14150 v1:172.21.15.92:0/996057664' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-11T21:24:41.225 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:24:41 smithi097 ceph-mon[33317]: from='mgr.14150 v1:172.21.15.92:0/996057664' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-11T21:24:41.225 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:24:41 smithi097 ceph-mon[33317]: from='mgr.14150 v1:172.21.15.92:0/996057664' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T21:24:41.362 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:24:41 smithi092 ceph-mon[29572]: from='client.24286 v1:172.21.15.97:0/2194983662' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi097:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T21:24:41.362 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:24:41 smithi092 ceph-mon[29572]: from='mgr.14150 v1:172.21.15.92:0/996057664' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-11T21:24:41.362 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:24:41 smithi092 ceph-mon[29572]: from='mgr.14150 v1:172.21.15.92:0/996057664' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-11T21:24:41.362 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:24:41 smithi092 ceph-mon[29572]: from='mgr.14150 v1:172.21.15.92:0/996057664' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T21:24:41.362 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:24:41 smithi092 ceph-mon[37563]: from='client.24286 v1:172.21.15.97:0/2194983662' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi097:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T21:24:41.362 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:24:41 smithi092 ceph-mon[37563]: from='mgr.14150 v1:172.21.15.92:0/996057664' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-11T21:24:41.362 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:24:41 smithi092 ceph-mon[37563]: from='mgr.14150 v1:172.21.15.92:0/996057664' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-11T21:24:41.362 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:24:41 smithi092 ceph-mon[37563]: from='mgr.14150 v1:172.21.15.92:0/996057664' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T21:24:42.269 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:24:42 smithi097 ceph-mon[33317]: pgmap v141: 1 pgs: 1 active+clean; 577 KiB data, 166 MiB used, 536 GiB / 536 GiB avail 2024-03-11T21:24:42.269 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:24:42 smithi097 ceph-mon[33317]: 1.0 scrub starts 2024-03-11T21:24:42.269 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:24:42 smithi097 ceph-mon[33317]: 1.0 scrub ok 2024-03-11T21:24:42.361 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:24:42 smithi092 ceph-mon[37563]: pgmap v141: 1 pgs: 1 active+clean; 577 KiB data, 166 MiB used, 536 GiB / 536 GiB avail 2024-03-11T21:24:42.362 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:24:42 smithi092 ceph-mon[37563]: 1.0 scrub starts 2024-03-11T21:24:42.362 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:24:42 smithi092 ceph-mon[37563]: 1.0 scrub ok 2024-03-11T21:24:42.362 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:24:42 smithi092 ceph-mon[29572]: pgmap v141: 1 pgs: 1 active+clean; 577 KiB data, 166 MiB used, 536 GiB / 536 GiB avail 2024-03-11T21:24:42.362 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:24:42 smithi092 ceph-mon[29572]: 1.0 scrub starts 2024-03-11T21:24:42.362 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:24:42 smithi092 ceph-mon[29572]: 1.0 scrub ok 2024-03-11T21:24:44.328 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:24:44 smithi097 ceph-mon[33317]: from='client.? v1:172.21.15.97:0/4019932922' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "91035ecc-6f1a-4cec-bda6-090af9d53eff"}]: dispatch 2024-03-11T21:24:44.328 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:24:44 smithi097 ceph-mon[33317]: from='client.24292 ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "91035ecc-6f1a-4cec-bda6-090af9d53eff"}]: dispatch 2024-03-11T21:24:44.328 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:24:44 smithi097 ceph-mon[33317]: from='client.24292 ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "91035ecc-6f1a-4cec-bda6-090af9d53eff"}]': finished 2024-03-11T21:24:44.328 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:24:44 smithi097 ceph-mon[33317]: osdmap e46: 7 total, 6 up, 7 in 2024-03-11T21:24:44.328 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:24:44 smithi097 ceph-mon[33317]: from='mgr.14150 v1:172.21.15.92:0/996057664' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-11T21:24:44.328 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:24:44 smithi097 ceph-mon[33317]: pgmap v143: 1 pgs: 1 active+clean; 577 KiB data, 166 MiB used, 536 GiB / 536 GiB avail 2024-03-11T21:24:44.328 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:24:44 smithi097 ceph-mon[33317]: from='client.? v1:172.21.15.97:0/3769522512' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-11T21:24:44.362 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:24:44 smithi092 ceph-mon[29572]: from='client.? v1:172.21.15.97:0/4019932922' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "91035ecc-6f1a-4cec-bda6-090af9d53eff"}]: dispatch 2024-03-11T21:24:44.362 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:24:44 smithi092 ceph-mon[29572]: from='client.24292 ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "91035ecc-6f1a-4cec-bda6-090af9d53eff"}]: dispatch 2024-03-11T21:24:44.362 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:24:44 smithi092 ceph-mon[29572]: from='client.24292 ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "91035ecc-6f1a-4cec-bda6-090af9d53eff"}]': finished 2024-03-11T21:24:44.362 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:24:44 smithi092 ceph-mon[29572]: osdmap e46: 7 total, 6 up, 7 in 2024-03-11T21:24:44.362 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:24:44 smithi092 ceph-mon[29572]: from='mgr.14150 v1:172.21.15.92:0/996057664' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-11T21:24:44.362 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:24:44 smithi092 ceph-mon[29572]: pgmap v143: 1 pgs: 1 active+clean; 577 KiB data, 166 MiB used, 536 GiB / 536 GiB avail 2024-03-11T21:24:44.362 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:24:44 smithi092 ceph-mon[29572]: from='client.? v1:172.21.15.97:0/3769522512' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-11T21:24:44.362 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:24:44 smithi092 ceph-mon[37563]: from='client.? v1:172.21.15.97:0/4019932922' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "91035ecc-6f1a-4cec-bda6-090af9d53eff"}]: dispatch 2024-03-11T21:24:44.362 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:24:44 smithi092 ceph-mon[37563]: from='client.24292 ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "91035ecc-6f1a-4cec-bda6-090af9d53eff"}]: dispatch 2024-03-11T21:24:44.363 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:24:44 smithi092 ceph-mon[37563]: from='client.24292 ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "91035ecc-6f1a-4cec-bda6-090af9d53eff"}]': finished 2024-03-11T21:24:44.363 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:24:44 smithi092 ceph-mon[37563]: osdmap e46: 7 total, 6 up, 7 in 2024-03-11T21:24:44.363 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:24:44 smithi092 ceph-mon[37563]: from='mgr.14150 v1:172.21.15.92:0/996057664' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-11T21:24:44.363 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:24:44 smithi092 ceph-mon[37563]: pgmap v143: 1 pgs: 1 active+clean; 577 KiB data, 166 MiB used, 536 GiB / 536 GiB avail 2024-03-11T21:24:44.363 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:24:44 smithi092 ceph-mon[37563]: from='client.? v1:172.21.15.97:0/3769522512' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-11T21:24:46.827 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:24:46 smithi097 ceph-mon[33317]: pgmap v144: 1 pgs: 1 active+clean; 577 KiB data, 166 MiB used, 536 GiB / 536 GiB avail 2024-03-11T21:24:46.861 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:24:46 smithi092 ceph-mon[29572]: pgmap v144: 1 pgs: 1 active+clean; 577 KiB data, 166 MiB used, 536 GiB / 536 GiB avail 2024-03-11T21:24:46.862 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:24:46 smithi092 ceph-mon[37563]: pgmap v144: 1 pgs: 1 active+clean; 577 KiB data, 166 MiB used, 536 GiB / 536 GiB avail 2024-03-11T21:24:48.827 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:24:48 smithi097 ceph-mon[33317]: pgmap v145: 1 pgs: 1 active+clean; 577 KiB data, 166 MiB used, 536 GiB / 536 GiB avail 2024-03-11T21:24:48.861 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:24:48 smithi092 ceph-mon[29572]: pgmap v145: 1 pgs: 1 active+clean; 577 KiB data, 166 MiB used, 536 GiB / 536 GiB avail 2024-03-11T21:24:48.862 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:24:48 smithi092 ceph-mon[37563]: pgmap v145: 1 pgs: 1 active+clean; 577 KiB data, 166 MiB used, 536 GiB / 536 GiB avail 2024-03-11T21:24:50.681 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:24:50 smithi097 ceph-mon[33317]: purged_snaps scrub starts 2024-03-11T21:24:50.681 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:24:50 smithi097 ceph-mon[33317]: purged_snaps scrub ok 2024-03-11T21:24:50.681 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:24:50 smithi097 ceph-mon[33317]: pgmap v146: 1 pgs: 1 active+clean; 577 KiB data, 166 MiB used, 536 GiB / 536 GiB avail 2024-03-11T21:24:50.681 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:24:50 smithi097 ceph-mon[33317]: osdmap e47: 7 total, 6 up, 7 in 2024-03-11T21:24:50.681 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:24:50 smithi097 ceph-mon[33317]: from='mgr.14150 v1:172.21.15.92:0/996057664' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-11T21:24:50.682 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:24:50 smithi097 ceph-mon[33317]: from='mgr.14150 v1:172.21.15.92:0/996057664' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.6"}]: dispatch 2024-03-11T21:24:50.682 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:24:50 smithi097 ceph-mon[33317]: from='mgr.14150 v1:172.21.15.92:0/996057664' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T21:24:50.861 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:24:50 smithi092 ceph-mon[37563]: purged_snaps scrub starts 2024-03-11T21:24:50.862 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:24:50 smithi092 ceph-mon[37563]: purged_snaps scrub ok 2024-03-11T21:24:50.862 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:24:50 smithi092 ceph-mon[37563]: pgmap v146: 1 pgs: 1 active+clean; 577 KiB data, 166 MiB used, 536 GiB / 536 GiB avail 2024-03-11T21:24:50.862 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:24:50 smithi092 ceph-mon[37563]: osdmap e47: 7 total, 6 up, 7 in 2024-03-11T21:24:50.862 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:24:50 smithi092 ceph-mon[37563]: from='mgr.14150 v1:172.21.15.92:0/996057664' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-11T21:24:50.862 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:24:50 smithi092 ceph-mon[37563]: from='mgr.14150 v1:172.21.15.92:0/996057664' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.6"}]: dispatch 2024-03-11T21:24:50.862 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:24:50 smithi092 ceph-mon[37563]: from='mgr.14150 v1:172.21.15.92:0/996057664' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T21:24:50.862 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:24:50 smithi092 ceph-mon[29572]: purged_snaps scrub starts 2024-03-11T21:24:50.862 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:24:50 smithi092 ceph-mon[29572]: purged_snaps scrub ok 2024-03-11T21:24:50.862 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:24:50 smithi092 ceph-mon[29572]: pgmap v146: 1 pgs: 1 active+clean; 577 KiB data, 166 MiB used, 536 GiB / 536 GiB avail 2024-03-11T21:24:50.862 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:24:50 smithi092 ceph-mon[29572]: osdmap e47: 7 total, 6 up, 7 in 2024-03-11T21:24:50.863 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:24:50 smithi092 ceph-mon[29572]: from='mgr.14150 v1:172.21.15.92:0/996057664' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-11T21:24:50.863 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:24:50 smithi092 ceph-mon[29572]: from='mgr.14150 v1:172.21.15.92:0/996057664' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.6"}]: dispatch 2024-03-11T21:24:50.863 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:24:50 smithi092 ceph-mon[29572]: from='mgr.14150 v1:172.21.15.92:0/996057664' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T21:24:51.827 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:24:51 smithi097 ceph-mon[33317]: Deploying daemon osd.6 on smithi097 2024-03-11T21:24:51.861 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:24:51 smithi092 ceph-mon[29572]: Deploying daemon osd.6 on smithi097 2024-03-11T21:24:51.862 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:24:51 smithi092 ceph-mon[37563]: Deploying daemon osd.6 on smithi097 2024-03-11T21:24:52.827 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:24:52 smithi097 ceph-mon[33317]: pgmap v148: 1 pgs: 1 active+clean; 577 KiB data, 166 MiB used, 536 GiB / 536 GiB avail 2024-03-11T21:24:52.861 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:24:52 smithi092 ceph-mon[29572]: pgmap v148: 1 pgs: 1 active+clean; 577 KiB data, 166 MiB used, 536 GiB / 536 GiB avail 2024-03-11T21:24:52.862 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:24:52 smithi092 ceph-mon[37563]: pgmap v148: 1 pgs: 1 active+clean; 577 KiB data, 166 MiB used, 536 GiB / 536 GiB avail 2024-03-11T21:24:54.828 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:24:54 smithi097 ceph-mon[33317]: pgmap v149: 1 pgs: 1 active+clean; 577 KiB data, 166 MiB used, 536 GiB / 536 GiB avail 2024-03-11T21:24:54.861 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:24:54 smithi092 ceph-mon[29572]: pgmap v149: 1 pgs: 1 active+clean; 577 KiB data, 166 MiB used, 536 GiB / 536 GiB avail 2024-03-11T21:24:54.862 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:24:54 smithi092 ceph-mon[37563]: pgmap v149: 1 pgs: 1 active+clean; 577 KiB data, 166 MiB used, 536 GiB / 536 GiB avail 2024-03-11T21:24:56.327 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:24:56 smithi097 ceph-mon[33317]: from='mgr.14150 v1:172.21.15.92:0/996057664' entity='mgr.y' 2024-03-11T21:24:56.327 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:24:56 smithi097 ceph-mon[33317]: from='mgr.14150 v1:172.21.15.92:0/996057664' entity='mgr.y' 2024-03-11T21:24:56.327 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:24:56 smithi097 ceph-mon[33317]: from='mgr.14150 v1:172.21.15.92:0/996057664' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T21:24:56.327 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:24:56 smithi097 ceph-mon[33317]: pgmap v150: 1 pgs: 1 active+clean; 577 KiB data, 166 MiB used, 536 GiB / 536 GiB avail 2024-03-11T21:24:56.361 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:24:56 smithi092 ceph-mon[29572]: from='mgr.14150 v1:172.21.15.92:0/996057664' entity='mgr.y' 2024-03-11T21:24:56.361 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:24:56 smithi092 ceph-mon[29572]: from='mgr.14150 v1:172.21.15.92:0/996057664' entity='mgr.y' 2024-03-11T21:24:56.361 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:24:56 smithi092 ceph-mon[29572]: from='mgr.14150 v1:172.21.15.92:0/996057664' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T21:24:56.362 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:24:56 smithi092 ceph-mon[29572]: pgmap v150: 1 pgs: 1 active+clean; 577 KiB data, 166 MiB used, 536 GiB / 536 GiB avail 2024-03-11T21:24:56.362 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:24:56 smithi092 ceph-mon[37563]: from='mgr.14150 v1:172.21.15.92:0/996057664' entity='mgr.y' 2024-03-11T21:24:56.362 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:24:56 smithi092 ceph-mon[37563]: from='mgr.14150 v1:172.21.15.92:0/996057664' entity='mgr.y' 2024-03-11T21:24:56.362 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:24:56 smithi092 ceph-mon[37563]: from='mgr.14150 v1:172.21.15.92:0/996057664' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T21:24:56.362 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:24:56 smithi092 ceph-mon[37563]: pgmap v150: 1 pgs: 1 active+clean; 577 KiB data, 166 MiB used, 536 GiB / 536 GiB avail 2024-03-11T21:24:58.691 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:24:58 smithi097 ceph-mon[33317]: pgmap v151: 1 pgs: 1 active+clean; 577 KiB data, 166 MiB used, 536 GiB / 536 GiB avail 2024-03-11T21:24:58.691 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:24:58 smithi097 ceph-mon[33317]: from='mgr.14150 v1:172.21.15.92:0/996057664' entity='mgr.y' 2024-03-11T21:24:58.691 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:24:58 smithi097 ceph-mon[33317]: from='mgr.14150 v1:172.21.15.92:0/996057664' entity='mgr.y' 2024-03-11T21:24:58.692 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:24:58 smithi097 ceph-mon[33317]: from='mgr.14150 v1:172.21.15.92:0/996057664' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T21:24:58.692 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:24:58 smithi097 ceph-mon[33317]: from='mgr.14150 v1:172.21.15.92:0/996057664' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T21:24:58.692 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:24:58 smithi097 ceph-mon[33317]: from='mgr.14150 v1:172.21.15.92:0/996057664' entity='mgr.y' 2024-03-11T21:24:58.692 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:24:58 smithi097 ceph-mon[33317]: from='osd.6 v1:172.21.15.97:6808/178994999' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2024-03-11T21:24:58.692 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:24:58 smithi097 ceph-mon[33317]: from='osd.6 ' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2024-03-11T21:24:58.692 INFO:teuthology.orchestra.run.smithi097.stdout:Created osd(s) 6 on host 'smithi097' 2024-03-11T21:24:58.861 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:24:58 smithi092 ceph-mon[29572]: pgmap v151: 1 pgs: 1 active+clean; 577 KiB data, 166 MiB used, 536 GiB / 536 GiB avail 2024-03-11T21:24:58.862 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:24:58 smithi092 ceph-mon[29572]: from='mgr.14150 v1:172.21.15.92:0/996057664' entity='mgr.y' 2024-03-11T21:24:58.862 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:24:58 smithi092 ceph-mon[29572]: from='mgr.14150 v1:172.21.15.92:0/996057664' entity='mgr.y' 2024-03-11T21:24:58.862 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:24:58 smithi092 ceph-mon[29572]: from='mgr.14150 v1:172.21.15.92:0/996057664' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T21:24:58.862 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:24:58 smithi092 ceph-mon[29572]: from='mgr.14150 v1:172.21.15.92:0/996057664' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T21:24:58.862 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:24:58 smithi092 ceph-mon[29572]: from='mgr.14150 v1:172.21.15.92:0/996057664' entity='mgr.y' 2024-03-11T21:24:58.862 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:24:58 smithi092 ceph-mon[29572]: from='osd.6 v1:172.21.15.97:6808/178994999' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2024-03-11T21:24:58.862 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:24:58 smithi092 ceph-mon[29572]: from='osd.6 ' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2024-03-11T21:24:58.862 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:24:58 smithi092 ceph-mon[37563]: pgmap v151: 1 pgs: 1 active+clean; 577 KiB data, 166 MiB used, 536 GiB / 536 GiB avail 2024-03-11T21:24:58.862 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:24:58 smithi092 ceph-mon[37563]: from='mgr.14150 v1:172.21.15.92:0/996057664' entity='mgr.y' 2024-03-11T21:24:58.863 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:24:58 smithi092 ceph-mon[37563]: from='mgr.14150 v1:172.21.15.92:0/996057664' entity='mgr.y' 2024-03-11T21:24:58.863 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:24:58 smithi092 ceph-mon[37563]: from='mgr.14150 v1:172.21.15.92:0/996057664' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T21:24:58.863 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:24:58 smithi092 ceph-mon[37563]: from='mgr.14150 v1:172.21.15.92:0/996057664' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T21:24:58.863 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:24:58 smithi092 ceph-mon[37563]: from='mgr.14150 v1:172.21.15.92:0/996057664' entity='mgr.y' 2024-03-11T21:24:58.863 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:24:58 smithi092 ceph-mon[37563]: from='osd.6 v1:172.21.15.97:6808/178994999' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2024-03-11T21:24:58.863 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:24:58 smithi092 ceph-mon[37563]: from='osd.6 ' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2024-03-11T21:24:59.273 DEBUG:teuthology.orchestra.run.smithi097:osd.6> sudo journalctl -f -n 0 -u ceph-00db0016-dfed-11ee-95c8-87774f69a715@osd.6.service 2024-03-11T21:24:59.276 INFO:tasks.cephadm:Deploying osd.7 on smithi097 with /dev/vg_nvme/lv_1... 2024-03-11T21:24:59.276 DEBUG:teuthology.orchestra.run.smithi097:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 00db0016-dfed-11ee-95c8-87774f69a715 -- ceph-volume lvm zap /dev/vg_nvme/lv_1 2024-03-11T21:24:59.925 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:24:59 smithi097 ceph-mon[33317]: from='mgr.14150 v1:172.21.15.92:0/996057664' entity='mgr.y' 2024-03-11T21:24:59.925 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:24:59 smithi097 ceph-mon[33317]: from='mgr.14150 v1:172.21.15.92:0/996057664' entity='mgr.y' 2024-03-11T21:24:59.925 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:24:59 smithi097 ceph-mon[33317]: from='mgr.14150 v1:172.21.15.92:0/996057664' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T21:24:59.925 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:24:59 smithi097 ceph-mon[33317]: from='osd.6 ' entity='osd.6' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]': finished 2024-03-11T21:24:59.925 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:24:59 smithi097 ceph-mon[33317]: from='osd.6 v1:172.21.15.97:6808/178994999' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi097", "root=default"]}]: dispatch 2024-03-11T21:24:59.925 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:24:59 smithi097 ceph-mon[33317]: osdmap e48: 7 total, 6 up, 7 in 2024-03-11T21:24:59.925 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:24:59 smithi097 ceph-mon[33317]: from='mgr.14150 v1:172.21.15.92:0/996057664' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-11T21:24:59.925 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:24:59 smithi097 ceph-mon[33317]: from='osd.6 ' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi097", "root=default"]}]: dispatch 2024-03-11T21:24:59.926 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:24:59 smithi097 ceph-mon[33317]: pgmap v153: 1 pgs: 1 active+clean; 577 KiB data, 166 MiB used, 536 GiB / 536 GiB avail 2024-03-11T21:25:00.111 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:24:59 smithi092 ceph-mon[29572]: from='mgr.14150 v1:172.21.15.92:0/996057664' entity='mgr.y' 2024-03-11T21:25:00.112 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:24:59 smithi092 ceph-mon[29572]: from='mgr.14150 v1:172.21.15.92:0/996057664' entity='mgr.y' 2024-03-11T21:25:00.112 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:24:59 smithi092 ceph-mon[29572]: from='mgr.14150 v1:172.21.15.92:0/996057664' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T21:25:00.112 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:24:59 smithi092 ceph-mon[29572]: from='osd.6 ' entity='osd.6' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]': finished 2024-03-11T21:25:00.112 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:24:59 smithi092 ceph-mon[29572]: from='osd.6 v1:172.21.15.97:6808/178994999' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi097", "root=default"]}]: dispatch 2024-03-11T21:25:00.112 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:24:59 smithi092 ceph-mon[29572]: osdmap e48: 7 total, 6 up, 7 in 2024-03-11T21:25:00.112 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:24:59 smithi092 ceph-mon[29572]: from='mgr.14150 v1:172.21.15.92:0/996057664' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-11T21:25:00.112 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:24:59 smithi092 ceph-mon[29572]: from='osd.6 ' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi097", "root=default"]}]: dispatch 2024-03-11T21:25:00.112 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:24:59 smithi092 ceph-mon[29572]: pgmap v153: 1 pgs: 1 active+clean; 577 KiB data, 166 MiB used, 536 GiB / 536 GiB avail 2024-03-11T21:25:00.112 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:24:59 smithi092 ceph-mon[37563]: from='mgr.14150 v1:172.21.15.92:0/996057664' entity='mgr.y' 2024-03-11T21:25:00.112 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:24:59 smithi092 ceph-mon[37563]: from='mgr.14150 v1:172.21.15.92:0/996057664' entity='mgr.y' 2024-03-11T21:25:00.113 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:24:59 smithi092 ceph-mon[37563]: from='mgr.14150 v1:172.21.15.92:0/996057664' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T21:25:00.113 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:24:59 smithi092 ceph-mon[37563]: from='osd.6 ' entity='osd.6' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]': finished 2024-03-11T21:25:00.113 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:24:59 smithi092 ceph-mon[37563]: from='osd.6 v1:172.21.15.97:6808/178994999' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi097", "root=default"]}]: dispatch 2024-03-11T21:25:00.113 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:24:59 smithi092 ceph-mon[37563]: osdmap e48: 7 total, 6 up, 7 in 2024-03-11T21:25:00.113 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:24:59 smithi092 ceph-mon[37563]: from='mgr.14150 v1:172.21.15.92:0/996057664' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-11T21:25:00.113 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:24:59 smithi092 ceph-mon[37563]: from='osd.6 ' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi097", "root=default"]}]: dispatch 2024-03-11T21:25:00.113 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:24:59 smithi092 ceph-mon[37563]: pgmap v153: 1 pgs: 1 active+clean; 577 KiB data, 166 MiB used, 536 GiB / 536 GiB avail 2024-03-11T21:25:00.328 INFO:journalctl@ceph.osd.6.smithi097.stdout:Mar 11 21:24:59 smithi097 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd-6[50273]: 2024-03-11T21:24:59.940+0000 7f51abd32700 -1 osd.6 0 waiting for initial osdmap 2024-03-11T21:25:00.328 INFO:journalctl@ceph.osd.6.smithi097.stdout:Mar 11 21:24:59 smithi097 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd-6[50273]: 2024-03-11T21:24:59.949+0000 7f51a7b25700 -1 osd.6 49 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-11T21:25:00.472 INFO:teuthology.orchestra.run.smithi097.stderr:Inferring config /var/lib/ceph/00db0016-dfed-11ee-95c8-87774f69a715/mon.b/config 2024-03-11T21:25:01.077 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:25:00 smithi097 ceph-mon[33317]: from='osd.6 ' entity='osd.6' cmd='[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi097", "root=default"]}]': finished 2024-03-11T21:25:01.077 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:25:00 smithi097 ceph-mon[33317]: osdmap e49: 7 total, 6 up, 7 in 2024-03-11T21:25:01.078 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:25:00 smithi097 ceph-mon[33317]: from='mgr.14150 v1:172.21.15.92:0/996057664' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-11T21:25:01.078 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:25:00 smithi097 ceph-mon[33317]: from='mgr.14150 v1:172.21.15.92:0/996057664' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-11T21:25:01.362 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:25:00 smithi092 ceph-mon[29572]: from='osd.6 ' entity='osd.6' cmd='[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi097", "root=default"]}]': finished 2024-03-11T21:25:01.362 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:25:00 smithi092 ceph-mon[29572]: osdmap e49: 7 total, 6 up, 7 in 2024-03-11T21:25:01.362 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:25:00 smithi092 ceph-mon[29572]: from='mgr.14150 v1:172.21.15.92:0/996057664' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-11T21:25:01.362 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:25:00 smithi092 ceph-mon[29572]: from='mgr.14150 v1:172.21.15.92:0/996057664' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-11T21:25:01.363 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:25:00 smithi092 ceph-mon[37563]: from='osd.6 ' entity='osd.6' cmd='[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi097", "root=default"]}]': finished 2024-03-11T21:25:01.363 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:25:00 smithi092 ceph-mon[37563]: osdmap e49: 7 total, 6 up, 7 in 2024-03-11T21:25:01.363 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:25:00 smithi092 ceph-mon[37563]: from='mgr.14150 v1:172.21.15.92:0/996057664' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-11T21:25:01.363 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:25:00 smithi092 ceph-mon[37563]: from='mgr.14150 v1:172.21.15.92:0/996057664' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-11T21:25:02.033 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:25:01 smithi097 ceph-mon[33317]: purged_snaps scrub starts 2024-03-11T21:25:02.034 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:25:01 smithi097 ceph-mon[33317]: purged_snaps scrub ok 2024-03-11T21:25:02.034 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:25:01 smithi097 ceph-mon[33317]: osd.6 v1:172.21.15.97:6808/178994999 boot 2024-03-11T21:25:02.034 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:25:01 smithi097 ceph-mon[33317]: osdmap e50: 7 total, 7 up, 7 in 2024-03-11T21:25:02.034 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:25:01 smithi097 ceph-mon[33317]: from='mgr.14150 v1:172.21.15.92:0/996057664' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-11T21:25:02.034 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:25:01 smithi097 ceph-mon[33317]: pgmap v156: 1 pgs: 1 active+clean; 577 KiB data, 166 MiB used, 536 GiB / 536 GiB avail 2024-03-11T21:25:02.034 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:25:01 smithi097 ceph-mon[33317]: osdmap e51: 7 total, 7 up, 7 in 2024-03-11T21:25:02.361 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:25:01 smithi092 ceph-mon[29572]: purged_snaps scrub starts 2024-03-11T21:25:02.362 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:25:01 smithi092 ceph-mon[29572]: purged_snaps scrub ok 2024-03-11T21:25:02.362 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:25:01 smithi092 ceph-mon[29572]: osd.6 v1:172.21.15.97:6808/178994999 boot 2024-03-11T21:25:02.362 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:25:01 smithi092 ceph-mon[29572]: osdmap e50: 7 total, 7 up, 7 in 2024-03-11T21:25:02.362 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:25:01 smithi092 ceph-mon[29572]: from='mgr.14150 v1:172.21.15.92:0/996057664' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-11T21:25:02.362 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:25:01 smithi092 ceph-mon[29572]: pgmap v156: 1 pgs: 1 active+clean; 577 KiB data, 166 MiB used, 536 GiB / 536 GiB avail 2024-03-11T21:25:02.362 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:25:01 smithi092 ceph-mon[29572]: osdmap e51: 7 total, 7 up, 7 in 2024-03-11T21:25:02.362 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:25:01 smithi092 ceph-mon[37563]: purged_snaps scrub starts 2024-03-11T21:25:02.362 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:25:01 smithi092 ceph-mon[37563]: purged_snaps scrub ok 2024-03-11T21:25:02.362 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:25:01 smithi092 ceph-mon[37563]: osd.6 v1:172.21.15.97:6808/178994999 boot 2024-03-11T21:25:02.362 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:25:01 smithi092 ceph-mon[37563]: osdmap e50: 7 total, 7 up, 7 in 2024-03-11T21:25:02.362 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:25:01 smithi092 ceph-mon[37563]: from='mgr.14150 v1:172.21.15.92:0/996057664' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-11T21:25:02.363 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:25:01 smithi092 ceph-mon[37563]: pgmap v156: 1 pgs: 1 active+clean; 577 KiB data, 166 MiB used, 536 GiB / 536 GiB avail 2024-03-11T21:25:02.363 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:25:01 smithi092 ceph-mon[37563]: osdmap e51: 7 total, 7 up, 7 in 2024-03-11T21:25:02.615 INFO:teuthology.orchestra.run.smithi097.stderr:--> Zapping: /dev/vg_nvme/lv_1 2024-03-11T21:25:02.615 INFO:teuthology.orchestra.run.smithi097.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_1 bs=1M count=10 conv=fsync 2024-03-11T21:25:02.615 INFO:teuthology.orchestra.run.smithi097.stderr: stderr: 10+0 records in 2024-03-11T21:25:02.615 INFO:teuthology.orchestra.run.smithi097.stderr:10+0 records out 2024-03-11T21:25:02.615 INFO:teuthology.orchestra.run.smithi097.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0135028 s, 777 MB/s 2024-03-11T21:25:02.615 INFO:teuthology.orchestra.run.smithi097.stderr:--> Zapping successful for: 2024-03-11T21:25:03.200 DEBUG:teuthology.orchestra.run.smithi097:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 00db0016-dfed-11ee-95c8-87774f69a715 -- ceph orch daemon add osd smithi097:vg_nvme/lv_1 2024-03-11T21:25:03.674 INFO:teuthology.orchestra.run.smithi097.stderr:Inferring config /var/lib/ceph/00db0016-dfed-11ee-95c8-87774f69a715/mon.b/config 2024-03-11T21:25:04.327 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:25:03 smithi097 ceph-mon[33317]: osdmap e52: 7 total, 7 up, 7 in 2024-03-11T21:25:04.328 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:25:03 smithi097 ceph-mon[33317]: pgmap v159: 1 pgs: 1 activating+undersized+degraded+remapped; 577 KiB data, 193 MiB used, 626 GiB / 626 GiB avail; 2/6 objects degraded (33.333%) 2024-03-11T21:25:04.328 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:25:03 smithi097 ceph-mon[33317]: from='mgr.14150 v1:172.21.15.92:0/996057664' entity='mgr.y' 2024-03-11T21:25:04.328 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:25:03 smithi097 ceph-mon[33317]: from='mgr.14150 v1:172.21.15.92:0/996057664' entity='mgr.y' 2024-03-11T21:25:04.328 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:25:03 smithi097 ceph-mon[33317]: from='mgr.14150 v1:172.21.15.92:0/996057664' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-11T21:25:04.328 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:25:03 smithi097 ceph-mon[33317]: from='mgr.14150 v1:172.21.15.92:0/996057664' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-11T21:25:04.328 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:25:03 smithi097 ceph-mon[33317]: from='mgr.14150 v1:172.21.15.92:0/996057664' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-03-11T21:25:04.328 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:25:03 smithi097 ceph-mon[33317]: from='mgr.14150 v1:172.21.15.92:0/996057664' entity='mgr.y' 2024-03-11T21:25:04.328 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:25:03 smithi097 ceph-mon[33317]: from='mgr.14150 v1:172.21.15.92:0/996057664' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T21:25:04.328 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:25:03 smithi097 ceph-mon[33317]: from='mgr.14150 v1:172.21.15.92:0/996057664' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T21:25:04.328 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:25:03 smithi097 ceph-mon[33317]: from='mgr.14150 v1:172.21.15.92:0/996057664' entity='mgr.y' 2024-03-11T21:25:04.362 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:25:03 smithi092 ceph-mon[29572]: osdmap e52: 7 total, 7 up, 7 in 2024-03-11T21:25:04.362 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:25:03 smithi092 ceph-mon[29572]: pgmap v159: 1 pgs: 1 activating+undersized+degraded+remapped; 577 KiB data, 193 MiB used, 626 GiB / 626 GiB avail; 2/6 objects degraded (33.333%) 2024-03-11T21:25:04.362 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:25:03 smithi092 ceph-mon[29572]: from='mgr.14150 v1:172.21.15.92:0/996057664' entity='mgr.y' 2024-03-11T21:25:04.362 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:25:03 smithi092 ceph-mon[29572]: from='mgr.14150 v1:172.21.15.92:0/996057664' entity='mgr.y' 2024-03-11T21:25:04.362 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:25:03 smithi092 ceph-mon[29572]: from='mgr.14150 v1:172.21.15.92:0/996057664' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-11T21:25:04.362 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:25:03 smithi092 ceph-mon[29572]: from='mgr.14150 v1:172.21.15.92:0/996057664' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-11T21:25:04.362 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:25:03 smithi092 ceph-mon[29572]: from='mgr.14150 v1:172.21.15.92:0/996057664' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-03-11T21:25:04.362 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:25:03 smithi092 ceph-mon[29572]: from='mgr.14150 v1:172.21.15.92:0/996057664' entity='mgr.y' 2024-03-11T21:25:04.362 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:25:03 smithi092 ceph-mon[29572]: from='mgr.14150 v1:172.21.15.92:0/996057664' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T21:25:04.362 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:25:03 smithi092 ceph-mon[29572]: from='mgr.14150 v1:172.21.15.92:0/996057664' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T21:25:04.362 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:25:03 smithi092 ceph-mon[29572]: from='mgr.14150 v1:172.21.15.92:0/996057664' entity='mgr.y' 2024-03-11T21:25:04.363 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:25:03 smithi092 ceph-mon[37563]: osdmap e52: 7 total, 7 up, 7 in 2024-03-11T21:25:04.363 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:25:03 smithi092 ceph-mon[37563]: pgmap v159: 1 pgs: 1 activating+undersized+degraded+remapped; 577 KiB data, 193 MiB used, 626 GiB / 626 GiB avail; 2/6 objects degraded (33.333%) 2024-03-11T21:25:04.363 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:25:03 smithi092 ceph-mon[37563]: from='mgr.14150 v1:172.21.15.92:0/996057664' entity='mgr.y' 2024-03-11T21:25:04.363 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:25:03 smithi092 ceph-mon[37563]: from='mgr.14150 v1:172.21.15.92:0/996057664' entity='mgr.y' 2024-03-11T21:25:04.363 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:25:03 smithi092 ceph-mon[37563]: from='mgr.14150 v1:172.21.15.92:0/996057664' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-11T21:25:04.363 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:25:03 smithi092 ceph-mon[37563]: from='mgr.14150 v1:172.21.15.92:0/996057664' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-11T21:25:04.363 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:25:03 smithi092 ceph-mon[37563]: from='mgr.14150 v1:172.21.15.92:0/996057664' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-03-11T21:25:04.363 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:25:03 smithi092 ceph-mon[37563]: from='mgr.14150 v1:172.21.15.92:0/996057664' entity='mgr.y' 2024-03-11T21:25:04.363 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:25:03 smithi092 ceph-mon[37563]: from='mgr.14150 v1:172.21.15.92:0/996057664' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T21:25:04.363 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:25:03 smithi092 ceph-mon[37563]: from='mgr.14150 v1:172.21.15.92:0/996057664' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T21:25:04.363 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:25:03 smithi092 ceph-mon[37563]: from='mgr.14150 v1:172.21.15.92:0/996057664' entity='mgr.y' 2024-03-11T21:25:05.066 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:25:04 smithi097 ceph-mon[33317]: Detected new or changed devices on smithi097 2024-03-11T21:25:05.066 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:25:04 smithi097 ceph-mon[33317]: Adjusting osd_memory_target on smithi097 to 5696M 2024-03-11T21:25:05.067 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:25:04 smithi097 ceph-mon[33317]: Health check failed: Degraded data redundancy: 2/6 objects degraded (33.333%), 1 pg degraded (PG_DEGRADED) 2024-03-11T21:25:05.067 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:25:04 smithi097 ceph-mon[33317]: osdmap e53: 7 total, 7 up, 7 in 2024-03-11T21:25:05.362 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:25:04 smithi092 ceph-mon[29572]: Detected new or changed devices on smithi097 2024-03-11T21:25:05.362 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:25:04 smithi092 ceph-mon[29572]: Adjusting osd_memory_target on smithi097 to 5696M 2024-03-11T21:25:05.362 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:25:04 smithi092 ceph-mon[29572]: Health check failed: Degraded data redundancy: 2/6 objects degraded (33.333%), 1 pg degraded (PG_DEGRADED) 2024-03-11T21:25:05.362 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:25:04 smithi092 ceph-mon[29572]: osdmap e53: 7 total, 7 up, 7 in 2024-03-11T21:25:05.363 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:25:04 smithi092 ceph-mon[37563]: Detected new or changed devices on smithi097 2024-03-11T21:25:05.363 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:25:04 smithi092 ceph-mon[37563]: Adjusting osd_memory_target on smithi097 to 5696M 2024-03-11T21:25:05.363 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:25:04 smithi092 ceph-mon[37563]: Health check failed: Degraded data redundancy: 2/6 objects degraded (33.333%), 1 pg degraded (PG_DEGRADED) 2024-03-11T21:25:05.363 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:25:04 smithi092 ceph-mon[37563]: osdmap e53: 7 total, 7 up, 7 in 2024-03-11T21:25:06.327 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:25:06 smithi097 ceph-mon[33317]: osdmap e54: 7 total, 7 up, 7 in 2024-03-11T21:25:06.327 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:25:06 smithi097 ceph-mon[33317]: purged_snaps scrub starts 2024-03-11T21:25:06.327 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:25:06 smithi097 ceph-mon[33317]: purged_snaps scrub ok 2024-03-11T21:25:06.327 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:25:06 smithi097 ceph-mon[33317]: pgmap v162: 1 pgs: 1 activating+undersized+degraded+remapped; 577 KiB data, 193 MiB used, 626 GiB / 626 GiB avail; 2/6 objects degraded (33.333%) 2024-03-11T21:25:06.327 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:25:06 smithi097 ceph-mon[33317]: from='client.24313 v1:172.21.15.97:0/1921248701' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi097:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T21:25:06.327 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:25:06 smithi097 ceph-mon[33317]: from='mgr.14150 v1:172.21.15.92:0/996057664' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-11T21:25:06.328 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:25:06 smithi097 ceph-mon[33317]: from='mgr.14150 v1:172.21.15.92:0/996057664' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-11T21:25:06.328 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:25:06 smithi097 ceph-mon[33317]: from='mgr.14150 v1:172.21.15.92:0/996057664' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T21:25:06.361 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:25:06 smithi092 ceph-mon[29572]: osdmap e54: 7 total, 7 up, 7 in 2024-03-11T21:25:06.361 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:25:06 smithi092 ceph-mon[29572]: purged_snaps scrub starts 2024-03-11T21:25:06.361 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:25:06 smithi092 ceph-mon[29572]: purged_snaps scrub ok 2024-03-11T21:25:06.361 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:25:06 smithi092 ceph-mon[29572]: pgmap v162: 1 pgs: 1 activating+undersized+degraded+remapped; 577 KiB data, 193 MiB used, 626 GiB / 626 GiB avail; 2/6 objects degraded (33.333%) 2024-03-11T21:25:06.361 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:25:06 smithi092 ceph-mon[29572]: from='client.24313 v1:172.21.15.97:0/1921248701' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi097:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T21:25:06.362 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:25:06 smithi092 ceph-mon[29572]: from='mgr.14150 v1:172.21.15.92:0/996057664' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-11T21:25:06.362 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:25:06 smithi092 ceph-mon[29572]: from='mgr.14150 v1:172.21.15.92:0/996057664' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-11T21:25:06.362 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:25:06 smithi092 ceph-mon[29572]: from='mgr.14150 v1:172.21.15.92:0/996057664' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T21:25:06.362 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:25:06 smithi092 ceph-mon[37563]: osdmap e54: 7 total, 7 up, 7 in 2024-03-11T21:25:06.362 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:25:06 smithi092 ceph-mon[37563]: purged_snaps scrub starts 2024-03-11T21:25:06.362 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:25:06 smithi092 ceph-mon[37563]: purged_snaps scrub ok 2024-03-11T21:25:06.362 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:25:06 smithi092 ceph-mon[37563]: pgmap v162: 1 pgs: 1 activating+undersized+degraded+remapped; 577 KiB data, 193 MiB used, 626 GiB / 626 GiB avail; 2/6 objects degraded (33.333%) 2024-03-11T21:25:06.362 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:25:06 smithi092 ceph-mon[37563]: from='client.24313 v1:172.21.15.97:0/1921248701' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi097:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T21:25:06.362 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:25:06 smithi092 ceph-mon[37563]: from='mgr.14150 v1:172.21.15.92:0/996057664' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-11T21:25:06.362 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:25:06 smithi092 ceph-mon[37563]: from='mgr.14150 v1:172.21.15.92:0/996057664' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-11T21:25:06.362 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:25:06 smithi092 ceph-mon[37563]: from='mgr.14150 v1:172.21.15.92:0/996057664' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T21:25:07.327 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:25:07 smithi097 ceph-mon[33317]: osdmap e55: 7 total, 7 up, 7 in 2024-03-11T21:25:07.361 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:25:07 smithi092 ceph-mon[29572]: osdmap e55: 7 total, 7 up, 7 in 2024-03-11T21:25:07.362 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:25:07 smithi092 ceph-mon[37563]: osdmap e55: 7 total, 7 up, 7 in 2024-03-11T21:25:08.327 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:25:08 smithi097 ceph-mon[33317]: pgmap v164: 1 pgs: 1 active+clean; 577 KiB data, 194 MiB used, 626 GiB / 626 GiB avail; 126 KiB/s, 0 objects/s recovering 2024-03-11T21:25:08.361 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:25:08 smithi092 ceph-mon[29572]: pgmap v164: 1 pgs: 1 active+clean; 577 KiB data, 194 MiB used, 626 GiB / 626 GiB avail; 126 KiB/s, 0 objects/s recovering 2024-03-11T21:25:08.362 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:25:08 smithi092 ceph-mon[37563]: pgmap v164: 1 pgs: 1 active+clean; 577 KiB data, 194 MiB used, 626 GiB / 626 GiB avail; 126 KiB/s, 0 objects/s recovering 2024-03-11T21:25:09.327 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:25:09 smithi097 ceph-mon[33317]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 2/6 objects degraded (33.333%), 1 pg degraded) 2024-03-11T21:25:09.327 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:25:09 smithi097 ceph-mon[33317]: Cluster is now healthy 2024-03-11T21:25:09.328 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:25:09 smithi097 ceph-mon[33317]: from='client.? v1:172.21.15.97:0/2168343274' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "3eb28c24-664a-4bd6-869d-6bf5db98f52f"}]: dispatch 2024-03-11T21:25:09.328 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:25:09 smithi097 ceph-mon[33317]: from='client.24319 ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "3eb28c24-664a-4bd6-869d-6bf5db98f52f"}]: dispatch 2024-03-11T21:25:09.328 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:25:09 smithi097 ceph-mon[33317]: from='client.24319 ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "3eb28c24-664a-4bd6-869d-6bf5db98f52f"}]': finished 2024-03-11T21:25:09.328 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:25:09 smithi097 ceph-mon[33317]: osdmap e56: 8 total, 7 up, 8 in 2024-03-11T21:25:09.328 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:25:09 smithi097 ceph-mon[33317]: from='mgr.14150 v1:172.21.15.92:0/996057664' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-11T21:25:09.328 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:25:09 smithi097 ceph-mon[33317]: from='client.? v1:172.21.15.97:0/1486321225' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-11T21:25:09.361 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:25:09 smithi092 ceph-mon[29572]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 2/6 objects degraded (33.333%), 1 pg degraded) 2024-03-11T21:25:09.361 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:25:09 smithi092 ceph-mon[29572]: Cluster is now healthy 2024-03-11T21:25:09.361 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:25:09 smithi092 ceph-mon[29572]: from='client.? v1:172.21.15.97:0/2168343274' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "3eb28c24-664a-4bd6-869d-6bf5db98f52f"}]: dispatch 2024-03-11T21:25:09.362 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:25:09 smithi092 ceph-mon[29572]: from='client.24319 ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "3eb28c24-664a-4bd6-869d-6bf5db98f52f"}]: dispatch 2024-03-11T21:25:09.362 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:25:09 smithi092 ceph-mon[29572]: from='client.24319 ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "3eb28c24-664a-4bd6-869d-6bf5db98f52f"}]': finished 2024-03-11T21:25:09.362 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:25:09 smithi092 ceph-mon[29572]: osdmap e56: 8 total, 7 up, 8 in 2024-03-11T21:25:09.362 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:25:09 smithi092 ceph-mon[29572]: from='mgr.14150 v1:172.21.15.92:0/996057664' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-11T21:25:09.362 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:25:09 smithi092 ceph-mon[29572]: from='client.? v1:172.21.15.97:0/1486321225' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-11T21:25:09.362 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:25:09 smithi092 ceph-mon[37563]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 2/6 objects degraded (33.333%), 1 pg degraded) 2024-03-11T21:25:09.362 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:25:09 smithi092 ceph-mon[37563]: Cluster is now healthy 2024-03-11T21:25:09.362 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:25:09 smithi092 ceph-mon[37563]: from='client.? v1:172.21.15.97:0/2168343274' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "3eb28c24-664a-4bd6-869d-6bf5db98f52f"}]: dispatch 2024-03-11T21:25:09.362 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:25:09 smithi092 ceph-mon[37563]: from='client.24319 ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "3eb28c24-664a-4bd6-869d-6bf5db98f52f"}]: dispatch 2024-03-11T21:25:09.362 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:25:09 smithi092 ceph-mon[37563]: from='client.24319 ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "3eb28c24-664a-4bd6-869d-6bf5db98f52f"}]': finished 2024-03-11T21:25:09.362 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:25:09 smithi092 ceph-mon[37563]: osdmap e56: 8 total, 7 up, 8 in 2024-03-11T21:25:09.362 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:25:09 smithi092 ceph-mon[37563]: from='mgr.14150 v1:172.21.15.92:0/996057664' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-11T21:25:09.363 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:25:09 smithi092 ceph-mon[37563]: from='client.? v1:172.21.15.97:0/1486321225' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-11T21:25:10.327 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:25:10 smithi097 ceph-mon[33317]: pgmap v166: 1 pgs: 1 active+clean; 577 KiB data, 194 MiB used, 626 GiB / 626 GiB avail; 105 KiB/s, 0 objects/s recovering 2024-03-11T21:25:10.361 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:25:10 smithi092 ceph-mon[29572]: pgmap v166: 1 pgs: 1 active+clean; 577 KiB data, 194 MiB used, 626 GiB / 626 GiB avail; 105 KiB/s, 0 objects/s recovering 2024-03-11T21:25:10.362 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:25:10 smithi092 ceph-mon[37563]: pgmap v166: 1 pgs: 1 active+clean; 577 KiB data, 194 MiB used, 626 GiB / 626 GiB avail; 105 KiB/s, 0 objects/s recovering 2024-03-11T21:25:12.827 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:25:12 smithi097 ceph-mon[33317]: pgmap v167: 1 pgs: 1 active+clean; 577 KiB data, 194 MiB used, 626 GiB / 626 GiB avail; 89 KiB/s, 0 objects/s recovering 2024-03-11T21:25:12.827 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:25:12 smithi097 ceph-mon[33317]: purged_snaps scrub starts 2024-03-11T21:25:12.827 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:25:12 smithi097 ceph-mon[33317]: purged_snaps scrub ok 2024-03-11T21:25:12.861 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:25:12 smithi092 ceph-mon[37563]: pgmap v167: 1 pgs: 1 active+clean; 577 KiB data, 194 MiB used, 626 GiB / 626 GiB avail; 89 KiB/s, 0 objects/s recovering 2024-03-11T21:25:12.861 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:25:12 smithi092 ceph-mon[37563]: purged_snaps scrub starts 2024-03-11T21:25:12.861 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:25:12 smithi092 ceph-mon[37563]: purged_snaps scrub ok 2024-03-11T21:25:12.862 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:25:12 smithi092 ceph-mon[29572]: pgmap v167: 1 pgs: 1 active+clean; 577 KiB data, 194 MiB used, 626 GiB / 626 GiB avail; 89 KiB/s, 0 objects/s recovering 2024-03-11T21:25:12.862 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:25:12 smithi092 ceph-mon[29572]: purged_snaps scrub starts 2024-03-11T21:25:12.862 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:25:12 smithi092 ceph-mon[29572]: purged_snaps scrub ok 2024-03-11T21:25:13.793 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:25:13 smithi097 ceph-mon[33317]: osdmap e57: 8 total, 7 up, 8 in 2024-03-11T21:25:13.794 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:25:13 smithi097 ceph-mon[33317]: from='mgr.14150 v1:172.21.15.92:0/996057664' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-11T21:25:13.861 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:25:13 smithi092 ceph-mon[29572]: osdmap e57: 8 total, 7 up, 8 in 2024-03-11T21:25:13.861 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:25:13 smithi092 ceph-mon[29572]: from='mgr.14150 v1:172.21.15.92:0/996057664' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-11T21:25:13.861 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:25:13 smithi092 ceph-mon[37563]: osdmap e57: 8 total, 7 up, 8 in 2024-03-11T21:25:13.862 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:25:13 smithi092 ceph-mon[37563]: from='mgr.14150 v1:172.21.15.92:0/996057664' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-11T21:25:14.772 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:25:14 smithi097 ceph-mon[33317]: pgmap v169: 1 pgs: 1 active+clean; 577 KiB data, 194 MiB used, 626 GiB / 626 GiB avail; 77 KiB/s, 0 objects/s recovering 2024-03-11T21:25:14.861 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:25:14 smithi092 ceph-mon[29572]: pgmap v169: 1 pgs: 1 active+clean; 577 KiB data, 194 MiB used, 626 GiB / 626 GiB avail; 77 KiB/s, 0 objects/s recovering 2024-03-11T21:25:14.861 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:25:14 smithi092 ceph-mon[37563]: pgmap v169: 1 pgs: 1 active+clean; 577 KiB data, 194 MiB used, 626 GiB / 626 GiB avail; 77 KiB/s, 0 objects/s recovering 2024-03-11T21:25:15.700 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:25:15 smithi097 ceph-mon[33317]: from='mgr.14150 v1:172.21.15.92:0/996057664' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.7"}]: dispatch 2024-03-11T21:25:15.700 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:25:15 smithi097 ceph-mon[33317]: from='mgr.14150 v1:172.21.15.92:0/996057664' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T21:25:15.861 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:25:15 smithi092 ceph-mon[37563]: from='mgr.14150 v1:172.21.15.92:0/996057664' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.7"}]: dispatch 2024-03-11T21:25:15.862 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:25:15 smithi092 ceph-mon[37563]: from='mgr.14150 v1:172.21.15.92:0/996057664' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T21:25:15.862 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:25:15 smithi092 ceph-mon[29572]: from='mgr.14150 v1:172.21.15.92:0/996057664' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.7"}]: dispatch 2024-03-11T21:25:15.862 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:25:15 smithi092 ceph-mon[29572]: from='mgr.14150 v1:172.21.15.92:0/996057664' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T21:25:16.693 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:25:16 smithi097 ceph-mon[33317]: Deploying daemon osd.7 on smithi097 2024-03-11T21:25:16.693 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:25:16 smithi097 ceph-mon[33317]: pgmap v170: 1 pgs: 1 active+clean; 577 KiB data, 194 MiB used, 626 GiB / 626 GiB avail 2024-03-11T21:25:16.861 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:25:16 smithi092 ceph-mon[29572]: Deploying daemon osd.7 on smithi097 2024-03-11T21:25:16.861 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:25:16 smithi092 ceph-mon[29572]: pgmap v170: 1 pgs: 1 active+clean; 577 KiB data, 194 MiB used, 626 GiB / 626 GiB avail 2024-03-11T21:25:16.862 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:25:16 smithi092 ceph-mon[37563]: Deploying daemon osd.7 on smithi097 2024-03-11T21:25:16.862 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:25:16 smithi092 ceph-mon[37563]: pgmap v170: 1 pgs: 1 active+clean; 577 KiB data, 194 MiB used, 626 GiB / 626 GiB avail 2024-03-11T21:25:18.656 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:25:18 smithi097 ceph-mon[33317]: pgmap v171: 1 pgs: 1 active+clean; 577 KiB data, 194 MiB used, 626 GiB / 626 GiB avail 2024-03-11T21:25:18.861 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:25:18 smithi092 ceph-mon[37563]: pgmap v171: 1 pgs: 1 active+clean; 577 KiB data, 194 MiB used, 626 GiB / 626 GiB avail 2024-03-11T21:25:18.861 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:25:18 smithi092 ceph-mon[29572]: pgmap v171: 1 pgs: 1 active+clean; 577 KiB data, 194 MiB used, 626 GiB / 626 GiB avail 2024-03-11T21:25:20.861 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:25:20 smithi092 ceph-mon[29572]: pgmap v172: 1 pgs: 1 active+clean; 577 KiB data, 194 MiB used, 626 GiB / 626 GiB avail 2024-03-11T21:25:20.862 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:25:20 smithi092 ceph-mon[29572]: from='mgr.14150 v1:172.21.15.92:0/996057664' entity='mgr.y' 2024-03-11T21:25:20.862 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:25:20 smithi092 ceph-mon[29572]: from='mgr.14150 v1:172.21.15.92:0/996057664' entity='mgr.y' 2024-03-11T21:25:20.862 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:25:20 smithi092 ceph-mon[29572]: from='mgr.14150 v1:172.21.15.92:0/996057664' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T21:25:20.862 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:25:20 smithi092 ceph-mon[37563]: pgmap v172: 1 pgs: 1 active+clean; 577 KiB data, 194 MiB used, 626 GiB / 626 GiB avail 2024-03-11T21:25:20.862 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:25:20 smithi092 ceph-mon[37563]: from='mgr.14150 v1:172.21.15.92:0/996057664' entity='mgr.y' 2024-03-11T21:25:20.862 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:25:20 smithi092 ceph-mon[37563]: from='mgr.14150 v1:172.21.15.92:0/996057664' entity='mgr.y' 2024-03-11T21:25:20.862 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:25:20 smithi092 ceph-mon[37563]: from='mgr.14150 v1:172.21.15.92:0/996057664' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T21:25:20.898 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:25:20 smithi097 ceph-mon[33317]: pgmap v172: 1 pgs: 1 active+clean; 577 KiB data, 194 MiB used, 626 GiB / 626 GiB avail 2024-03-11T21:25:20.898 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:25:20 smithi097 ceph-mon[33317]: from='mgr.14150 v1:172.21.15.92:0/996057664' entity='mgr.y' 2024-03-11T21:25:20.898 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:25:20 smithi097 ceph-mon[33317]: from='mgr.14150 v1:172.21.15.92:0/996057664' entity='mgr.y' 2024-03-11T21:25:20.898 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:25:20 smithi097 ceph-mon[33317]: from='mgr.14150 v1:172.21.15.92:0/996057664' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T21:25:22.827 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:25:22 smithi097 ceph-mon[33317]: pgmap v173: 1 pgs: 1 active+clean; 577 KiB data, 194 MiB used, 626 GiB / 626 GiB avail 2024-03-11T21:25:22.861 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:25:22 smithi092 ceph-mon[29572]: pgmap v173: 1 pgs: 1 active+clean; 577 KiB data, 194 MiB used, 626 GiB / 626 GiB avail 2024-03-11T21:25:22.862 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:25:22 smithi092 ceph-mon[37563]: pgmap v173: 1 pgs: 1 active+clean; 577 KiB data, 194 MiB used, 626 GiB / 626 GiB avail 2024-03-11T21:25:23.670 INFO:teuthology.orchestra.run.smithi097.stdout:Created osd(s) 7 on host 'smithi097' 2024-03-11T21:25:24.243 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:25:24 smithi097 ceph-mon[33317]: from='osd.7 v1:172.21.15.97:6812/3241560509' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-03-11T21:25:24.243 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:25:24 smithi097 ceph-mon[33317]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-03-11T21:25:24.243 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:25:24 smithi097 ceph-mon[33317]: pgmap v174: 1 pgs: 1 active+clean; 577 KiB data, 194 MiB used, 626 GiB / 626 GiB avail 2024-03-11T21:25:24.243 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:25:24 smithi097 ceph-mon[33317]: from='mgr.14150 v1:172.21.15.92:0/996057664' entity='mgr.y' 2024-03-11T21:25:24.243 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:25:24 smithi097 ceph-mon[33317]: from='mgr.14150 v1:172.21.15.92:0/996057664' entity='mgr.y' 2024-03-11T21:25:24.271 DEBUG:teuthology.orchestra.run.smithi097:osd.7> sudo journalctl -f -n 0 -u ceph-00db0016-dfed-11ee-95c8-87774f69a715@osd.7.service 2024-03-11T21:25:24.273 INFO:tasks.cephadm:Waiting for 8 OSDs to come up... 2024-03-11T21:25:24.274 DEBUG:teuthology.orchestra.run.smithi092:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 00db0016-dfed-11ee-95c8-87774f69a715 -- ceph osd stat -f json 2024-03-11T21:25:24.303 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:25:24 smithi092 ceph-mon[29572]: from='osd.7 v1:172.21.15.97:6812/3241560509' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-03-11T21:25:24.303 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:25:24 smithi092 ceph-mon[29572]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-03-11T21:25:24.304 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:25:24 smithi092 ceph-mon[29572]: pgmap v174: 1 pgs: 1 active+clean; 577 KiB data, 194 MiB used, 626 GiB / 626 GiB avail 2024-03-11T21:25:24.304 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:25:24 smithi092 ceph-mon[29572]: from='mgr.14150 v1:172.21.15.92:0/996057664' entity='mgr.y' 2024-03-11T21:25:24.304 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:25:24 smithi092 ceph-mon[29572]: from='mgr.14150 v1:172.21.15.92:0/996057664' entity='mgr.y' 2024-03-11T21:25:24.304 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:25:24 smithi092 ceph-mon[37563]: from='osd.7 v1:172.21.15.97:6812/3241560509' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-03-11T21:25:24.304 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:25:24 smithi092 ceph-mon[37563]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-03-11T21:25:24.304 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:25:24 smithi092 ceph-mon[37563]: pgmap v174: 1 pgs: 1 active+clean; 577 KiB data, 194 MiB used, 626 GiB / 626 GiB avail 2024-03-11T21:25:24.304 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:25:24 smithi092 ceph-mon[37563]: from='mgr.14150 v1:172.21.15.92:0/996057664' entity='mgr.y' 2024-03-11T21:25:24.304 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:25:24 smithi092 ceph-mon[37563]: from='mgr.14150 v1:172.21.15.92:0/996057664' entity='mgr.y' 2024-03-11T21:25:24.523 INFO:teuthology.orchestra.run.smithi092.stderr:Inferring config /var/lib/ceph/00db0016-dfed-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T21:25:25.327 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:25:25 smithi097 ceph-mon[33317]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]': finished 2024-03-11T21:25:25.328 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:25:25 smithi097 ceph-mon[33317]: from='osd.7 v1:172.21.15.97:6812/3241560509' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi097", "root=default"]}]: dispatch 2024-03-11T21:25:25.328 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:25:25 smithi097 ceph-mon[33317]: osdmap e58: 8 total, 7 up, 8 in 2024-03-11T21:25:25.328 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:25:25 smithi097 ceph-mon[33317]: from='mgr.14150 v1:172.21.15.92:0/996057664' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-11T21:25:25.328 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:25:25 smithi097 ceph-mon[33317]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi097", "root=default"]}]: dispatch 2024-03-11T21:25:25.328 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:25:25 smithi097 ceph-mon[33317]: from='mgr.14150 v1:172.21.15.92:0/996057664' entity='mgr.y' 2024-03-11T21:25:25.329 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:25:25 smithi097 ceph-mon[33317]: from='mgr.14150 v1:172.21.15.92:0/996057664' entity='mgr.y' 2024-03-11T21:25:25.329 INFO:journalctl@ceph.osd.7.smithi097.stdout:Mar 11 21:25:25 smithi097 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd-7[56319]: 2024-03-11T21:25:25.022+0000 7ffbb6fec700 -1 osd.7 0 waiting for initial osdmap 2024-03-11T21:25:25.329 INFO:journalctl@ceph.osd.7.smithi097.stdout:Mar 11 21:25:25 smithi097 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd-7[56319]: 2024-03-11T21:25:25.030+0000 7ffbb2ddf700 -1 osd.7 59 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-11T21:25:25.362 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:25:25 smithi092 ceph-mon[37563]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]': finished 2024-03-11T21:25:25.362 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:25:25 smithi092 ceph-mon[37563]: from='osd.7 v1:172.21.15.97:6812/3241560509' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi097", "root=default"]}]: dispatch 2024-03-11T21:25:25.362 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:25:25 smithi092 ceph-mon[37563]: osdmap e58: 8 total, 7 up, 8 in 2024-03-11T21:25:25.362 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:25:25 smithi092 ceph-mon[37563]: from='mgr.14150 v1:172.21.15.92:0/996057664' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-11T21:25:25.362 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:25:25 smithi092 ceph-mon[37563]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi097", "root=default"]}]: dispatch 2024-03-11T21:25:25.362 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:25:25 smithi092 ceph-mon[37563]: from='mgr.14150 v1:172.21.15.92:0/996057664' entity='mgr.y' 2024-03-11T21:25:25.362 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:25:25 smithi092 ceph-mon[37563]: from='mgr.14150 v1:172.21.15.92:0/996057664' entity='mgr.y' 2024-03-11T21:25:25.362 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:25:25 smithi092 ceph-mon[29572]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]': finished 2024-03-11T21:25:25.363 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:25:25 smithi092 ceph-mon[29572]: from='osd.7 v1:172.21.15.97:6812/3241560509' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi097", "root=default"]}]: dispatch 2024-03-11T21:25:25.363 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:25:25 smithi092 ceph-mon[29572]: osdmap e58: 8 total, 7 up, 8 in 2024-03-11T21:25:25.363 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:25:25 smithi092 ceph-mon[29572]: from='mgr.14150 v1:172.21.15.92:0/996057664' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-11T21:25:25.363 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:25:25 smithi092 ceph-mon[29572]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi097", "root=default"]}]: dispatch 2024-03-11T21:25:25.363 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:25:25 smithi092 ceph-mon[29572]: from='mgr.14150 v1:172.21.15.92:0/996057664' entity='mgr.y' 2024-03-11T21:25:25.363 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:25:25 smithi092 ceph-mon[29572]: from='mgr.14150 v1:172.21.15.92:0/996057664' entity='mgr.y' 2024-03-11T21:25:26.327 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:25:26 smithi097 ceph-mon[33317]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi097", "root=default"]}]': finished 2024-03-11T21:25:26.327 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:25:26 smithi097 ceph-mon[33317]: osdmap e59: 8 total, 7 up, 8 in 2024-03-11T21:25:26.327 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:25:26 smithi097 ceph-mon[33317]: from='mgr.14150 v1:172.21.15.92:0/996057664' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-11T21:25:26.327 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:25:26 smithi097 ceph-mon[33317]: from='mgr.14150 v1:172.21.15.92:0/996057664' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-11T21:25:26.327 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:25:26 smithi097 ceph-mon[33317]: pgmap v177: 1 pgs: 1 active+clean; 577 KiB data, 194 MiB used, 626 GiB / 626 GiB avail 2024-03-11T21:25:26.354 INFO:teuthology.orchestra.run.smithi092.stdout: 2024-03-11T21:25:26.354 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:25:26 smithi092 ceph-mon[29572]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi097", "root=default"]}]': finished 2024-03-11T21:25:26.354 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:25:26 smithi092 ceph-mon[29572]: osdmap e59: 8 total, 7 up, 8 in 2024-03-11T21:25:26.354 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:25:26 smithi092 ceph-mon[29572]: from='mgr.14150 v1:172.21.15.92:0/996057664' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-11T21:25:26.354 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:25:26 smithi092 ceph-mon[29572]: from='mgr.14150 v1:172.21.15.92:0/996057664' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-11T21:25:26.354 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:25:26 smithi092 ceph-mon[29572]: pgmap v177: 1 pgs: 1 active+clean; 577 KiB data, 194 MiB used, 626 GiB / 626 GiB avail 2024-03-11T21:25:26.355 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:25:26 smithi092 ceph-mon[37563]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi097", "root=default"]}]': finished 2024-03-11T21:25:26.355 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:25:26 smithi092 ceph-mon[37563]: osdmap e59: 8 total, 7 up, 8 in 2024-03-11T21:25:26.355 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:25:26 smithi092 ceph-mon[37563]: from='mgr.14150 v1:172.21.15.92:0/996057664' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-11T21:25:26.355 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:25:26 smithi092 ceph-mon[37563]: from='mgr.14150 v1:172.21.15.92:0/996057664' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-11T21:25:26.355 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:25:26 smithi092 ceph-mon[37563]: pgmap v177: 1 pgs: 1 active+clean; 577 KiB data, 194 MiB used, 626 GiB / 626 GiB avail 2024-03-11T21:25:26.938 INFO:teuthology.orchestra.run.smithi092.stdout:{"epoch":60,"num_osds":8,"num_up_osds":8,"osd_up_since":1710192326,"num_in_osds":8,"osd_in_since":1710192308,"num_remapped_pgs":1} 2024-03-11T21:25:26.939 DEBUG:teuthology.orchestra.run.smithi092:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid 00db0016-dfed-11ee-95c8-87774f69a715 -- ceph osd dump --format=json 2024-03-11T21:25:27.224 INFO:teuthology.orchestra.run.smithi092.stderr:Inferring config /var/lib/ceph/00db0016-dfed-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T21:25:27.327 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:25:27 smithi097 ceph-mon[33317]: purged_snaps scrub starts 2024-03-11T21:25:27.327 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:25:27 smithi097 ceph-mon[33317]: purged_snaps scrub ok 2024-03-11T21:25:27.327 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:25:27 smithi097 ceph-mon[33317]: osd.7 v1:172.21.15.97:6812/3241560509 boot 2024-03-11T21:25:27.328 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:25:27 smithi097 ceph-mon[33317]: osdmap e60: 8 total, 8 up, 8 in 2024-03-11T21:25:27.328 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:25:27 smithi097 ceph-mon[33317]: from='mgr.14150 v1:172.21.15.92:0/996057664' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-11T21:25:27.328 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:25:27 smithi097 ceph-mon[33317]: from='client.? v1:172.21.15.92:0/2594519714' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-03-11T21:25:27.328 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:25:27 smithi097 ceph-mon[33317]: osdmap e61: 8 total, 8 up, 8 in 2024-03-11T21:25:27.345 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:25:27 smithi092 ceph-mon[29572]: purged_snaps scrub starts 2024-03-11T21:25:27.345 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:25:27 smithi092 ceph-mon[29572]: purged_snaps scrub ok 2024-03-11T21:25:27.345 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:25:27 smithi092 ceph-mon[29572]: osd.7 v1:172.21.15.97:6812/3241560509 boot 2024-03-11T21:25:27.345 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:25:27 smithi092 ceph-mon[29572]: osdmap e60: 8 total, 8 up, 8 in 2024-03-11T21:25:27.345 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:25:27 smithi092 ceph-mon[29572]: from='mgr.14150 v1:172.21.15.92:0/996057664' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-11T21:25:27.345 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:25:27 smithi092 ceph-mon[29572]: from='client.? v1:172.21.15.92:0/2594519714' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-03-11T21:25:27.345 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:25:27 smithi092 ceph-mon[29572]: osdmap e61: 8 total, 8 up, 8 in 2024-03-11T21:25:27.346 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:25:27 smithi092 ceph-mon[37563]: purged_snaps scrub starts 2024-03-11T21:25:27.346 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:25:27 smithi092 ceph-mon[37563]: purged_snaps scrub ok 2024-03-11T21:25:27.346 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:25:27 smithi092 ceph-mon[37563]: osd.7 v1:172.21.15.97:6812/3241560509 boot 2024-03-11T21:25:27.346 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:25:27 smithi092 ceph-mon[37563]: osdmap e60: 8 total, 8 up, 8 in 2024-03-11T21:25:27.346 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:25:27 smithi092 ceph-mon[37563]: from='mgr.14150 v1:172.21.15.92:0/996057664' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-11T21:25:27.346 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:25:27 smithi092 ceph-mon[37563]: from='client.? v1:172.21.15.92:0/2594519714' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-03-11T21:25:27.346 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:25:27 smithi092 ceph-mon[37563]: osdmap e61: 8 total, 8 up, 8 in 2024-03-11T21:25:28.300 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:25:28 smithi092 ceph-mon[29572]: pgmap v180: 1 pgs: 1 active+clean; 577 KiB data, 221 MiB used, 715 GiB / 715 GiB avail 2024-03-11T21:25:28.301 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:25:28 smithi092 ceph-mon[37563]: pgmap v180: 1 pgs: 1 active+clean; 577 KiB data, 221 MiB used, 715 GiB / 715 GiB avail 2024-03-11T21:25:28.327 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:25:28 smithi097 ceph-mon[33317]: pgmap v180: 1 pgs: 1 active+clean; 577 KiB data, 221 MiB used, 715 GiB / 715 GiB avail 2024-03-11T21:25:28.856 INFO:teuthology.orchestra.run.smithi092.stdout: 2024-03-11T21:25:28.856 INFO:teuthology.orchestra.run.smithi092.stdout:{"epoch":62,"fsid":"00db0016-dfed-11ee-95c8-87774f69a715","created":"2024-03-11T21:20:10.488059+0000","modified":"2024-03-11T21:25:28.021189+0000","last_up_change":"2024-03-11T21:25:26.018427+0000","last_in_change":"2024-03-11T21:25:08.189557+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":18,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-11T21:23:23.805029+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":7.8899998664855957,"score_stable":7.8899998664855957,"optimal_score":0.37999999523162842,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"79f15843-f0ec-44aa-931e-4e07a74d3768","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":61,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.92:6801","nonce":2844100201}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.92:6802","nonce":2844100201}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.92:6804","nonce":2844100201}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.92:6803","nonce":2844100201}]},"public_addr":"172.21.15.92:6801/2844100201","cluster_addr":"172.21.15.92:6802/2844100201","heartbeat_back_addr":"172.21.15.92:6804/2844100201","heartbeat_front_addr":"172.21.15.92:6803/2844100201","state":["exists","up"]},{"osd":1,"uuid":"3a5a0e78-1293-4ace-a93f-69156098571d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":35,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.92:6805","nonce":3577871157}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.92:6806","nonce":3577871157}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.92:6808","nonce":3577871157}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.92:6807","nonce":3577871157}]},"public_addr":"172.21.15.92:6805/3577871157","cluster_addr":"172.21.15.92:6806/3577871157","heartbeat_back_addr":"172.21.15.92:6808/3577871157","heartbeat_front_addr":"172.21.15.92:6807/3577871157","state":["exists","up"]},{"osd":2,"uuid":"f207b743-0df4-40ed-a735-950e272fea60","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.92:6809","nonce":760825152}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.92:6810","nonce":760825152}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.92:6812","nonce":760825152}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.92:6811","nonce":760825152}]},"public_addr":"172.21.15.92:6809/760825152","cluster_addr":"172.21.15.92:6810/760825152","heartbeat_back_addr":"172.21.15.92:6812/760825152","heartbeat_front_addr":"172.21.15.92:6811/760825152","state":["exists","up"]},{"osd":3,"uuid":"bbb6f69a-8f0f-47e6-937a-062e1902ae13","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":26,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.92:6813","nonce":670596264}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.92:6814","nonce":670596264}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.92:6816","nonce":670596264}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.92:6815","nonce":670596264}]},"public_addr":"172.21.15.92:6813/670596264","cluster_addr":"172.21.15.92:6814/670596264","heartbeat_back_addr":"172.21.15.92:6816/670596264","heartbeat_front_addr":"172.21.15.92:6815/670596264","state":["exists","up"]},{"osd":4,"uuid":"00266cbc-d2cf-4dee-8a96-3753ebc95ca8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":32,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.97:6800","nonce":420733620}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.97:6801","nonce":420733620}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.97:6803","nonce":420733620}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.97:6802","nonce":420733620}]},"public_addr":"172.21.15.97:6800/420733620","cluster_addr":"172.21.15.97:6801/420733620","heartbeat_back_addr":"172.21.15.97:6803/420733620","heartbeat_front_addr":"172.21.15.97:6802/420733620","state":["exists","up"]},{"osd":5,"uuid":"593a6f6b-8874-40ae-8848-d51c5723a5e4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":41,"up_thru":44,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.97:6804","nonce":312056041}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.97:6805","nonce":312056041}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.97:6807","nonce":312056041}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.97:6806","nonce":312056041}]},"public_addr":"172.21.15.97:6804/312056041","cluster_addr":"172.21.15.97:6805/312056041","heartbeat_back_addr":"172.21.15.97:6807/312056041","heartbeat_front_addr":"172.21.15.97:6806/312056041","state":["exists","up"]},{"osd":6,"uuid":"91035ecc-6f1a-4cec-bda6-090af9d53eff","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":50,"up_thru":53,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.97:6808","nonce":178994999}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.97:6809","nonce":178994999}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.97:6811","nonce":178994999}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.97:6810","nonce":178994999}]},"public_addr":"172.21.15.97:6808/178994999","cluster_addr":"172.21.15.97:6809/178994999","heartbeat_back_addr":"172.21.15.97:6811/178994999","heartbeat_front_addr":"172.21.15.97:6810/178994999","state":["exists","up"]},{"osd":7,"uuid":"3eb28c24-664a-4bd6-869d-6bf5db98f52f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":60,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.97:6812","nonce":3241560509}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.97:6813","nonce":3241560509}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.97:6815","nonce":3241560509}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.97:6814","nonce":3241560509}]},"public_addr":"172.21.15.97:6812/3241560509","cluster_addr":"172.21.15.97:6813/3241560509","heartbeat_back_addr":"172.21.15.97:6815/3241560509","heartbeat_front_addr":"172.21.15.97:6814/3241560509","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-11T21:24:49.411360+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-11T21:25:11.578251+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-11T21:24:29.342159+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-11T21:25:05.154676+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-11T21:24:09.222591+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-11T21:24:33.930223+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-11T21:24:59.295496+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-11T21:25:24.387220+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[{"pgid":"1.0","osds":[0,6]}],"primary_temp":[],"blocklist":{"172.21.15.92:0/3817549754":"2024-03-12T21:21:09.473916+0000","172.21.15.92:0/2211594569":"2024-03-12T21:21:09.473916+0000","172.21.15.92:0/3569900179":"2024-03-12T21:20:43.677016+0000","172.21.15.92:0/3380359534":"2024-03-12T21:20:43.677016+0000","172.21.15.92:0/2509373130":"2024-03-12T21:20:43.677016+0000","172.21.15.92:6800/1559007462":"2024-03-12T21:20:43.677016+0000","172.21.15.92:6800/1270902819":"2024-03-12T21:21:09.473916+0000","172.21.15.92:0/3562645553":"2024-03-12T21:21:09.473916+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-11T21:25:29.112 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:25:29 smithi092 ceph-mon[37563]: osdmap e62: 8 total, 8 up, 8 in 2024-03-11T21:25:29.112 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:25:29 smithi092 ceph-mon[37563]: Detected new or changed devices on smithi097 2024-03-11T21:25:29.112 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:25:29 smithi092 ceph-mon[37563]: from='mgr.14150 v1:172.21.15.92:0/996057664' entity='mgr.y' 2024-03-11T21:25:29.112 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:25:29 smithi092 ceph-mon[37563]: from='mgr.14150 v1:172.21.15.92:0/996057664' entity='mgr.y' 2024-03-11T21:25:29.112 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:25:29 smithi092 ceph-mon[37563]: from='mgr.14150 v1:172.21.15.92:0/996057664' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-11T21:25:29.112 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:25:29 smithi092 ceph-mon[37563]: from='mgr.14150 v1:172.21.15.92:0/996057664' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-11T21:25:29.112 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:25:29 smithi092 ceph-mon[37563]: from='mgr.14150 v1:172.21.15.92:0/996057664' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-03-11T21:25:29.112 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:25:29 smithi092 ceph-mon[37563]: from='mgr.14150 v1:172.21.15.92:0/996057664' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-03-11T21:25:29.112 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:25:29 smithi092 ceph-mon[37563]: Adjusting osd_memory_target on smithi097 to 4272M 2024-03-11T21:25:29.112 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:25:29 smithi092 ceph-mon[37563]: from='mgr.14150 v1:172.21.15.92:0/996057664' entity='mgr.y' 2024-03-11T21:25:29.112 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:25:29 smithi092 ceph-mon[37563]: from='mgr.14150 v1:172.21.15.92:0/996057664' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T21:25:29.113 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:25:29 smithi092 ceph-mon[37563]: from='mgr.14150 v1:172.21.15.92:0/996057664' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T21:25:29.113 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:25:29 smithi092 ceph-mon[37563]: from='mgr.14150 v1:172.21.15.92:0/996057664' entity='mgr.y' 2024-03-11T21:25:29.113 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:25:29 smithi092 ceph-mon[37563]: from='mgr.14150 v1:172.21.15.92:0/996057664' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T21:25:29.113 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:25:29 smithi092 ceph-mon[37563]: from='mgr.14150 v1:172.21.15.92:0/996057664' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T21:25:29.113 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:25:29 smithi092 ceph-mon[37563]: from='mgr.14150 v1:172.21.15.92:0/996057664' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T21:25:29.113 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:25:29 smithi092 ceph-mon[37563]: from='mgr.14150 v1:172.21.15.92:0/996057664' entity='mgr.y' 2024-03-11T21:25:29.113 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:25:29 smithi092 ceph-mon[37563]: from='client.? v1:172.21.15.92:0/342231441' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-11T21:25:29.113 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:25:29 smithi092 ceph-mon[29572]: osdmap e62: 8 total, 8 up, 8 in 2024-03-11T21:25:29.113 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:25:29 smithi092 ceph-mon[29572]: Detected new or changed devices on smithi097 2024-03-11T21:25:29.113 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:25:29 smithi092 ceph-mon[29572]: from='mgr.14150 v1:172.21.15.92:0/996057664' entity='mgr.y' 2024-03-11T21:25:29.113 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:25:29 smithi092 ceph-mon[29572]: from='mgr.14150 v1:172.21.15.92:0/996057664' entity='mgr.y' 2024-03-11T21:25:29.113 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:25:29 smithi092 ceph-mon[29572]: from='mgr.14150 v1:172.21.15.92:0/996057664' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-11T21:25:29.114 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:25:29 smithi092 ceph-mon[29572]: from='mgr.14150 v1:172.21.15.92:0/996057664' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-11T21:25:29.114 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:25:29 smithi092 ceph-mon[29572]: from='mgr.14150 v1:172.21.15.92:0/996057664' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-03-11T21:25:29.114 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:25:29 smithi092 ceph-mon[29572]: from='mgr.14150 v1:172.21.15.92:0/996057664' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-03-11T21:25:29.114 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:25:29 smithi092 ceph-mon[29572]: Adjusting osd_memory_target on smithi097 to 4272M 2024-03-11T21:25:29.114 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:25:29 smithi092 ceph-mon[29572]: from='mgr.14150 v1:172.21.15.92:0/996057664' entity='mgr.y' 2024-03-11T21:25:29.114 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:25:29 smithi092 ceph-mon[29572]: from='mgr.14150 v1:172.21.15.92:0/996057664' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T21:25:29.114 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:25:29 smithi092 ceph-mon[29572]: from='mgr.14150 v1:172.21.15.92:0/996057664' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T21:25:29.114 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:25:29 smithi092 ceph-mon[29572]: from='mgr.14150 v1:172.21.15.92:0/996057664' entity='mgr.y' 2024-03-11T21:25:29.114 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:25:29 smithi092 ceph-mon[29572]: from='mgr.14150 v1:172.21.15.92:0/996057664' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T21:25:29.114 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:25:29 smithi092 ceph-mon[29572]: from='mgr.14150 v1:172.21.15.92:0/996057664' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T21:25:29.114 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:25:29 smithi092 ceph-mon[29572]: from='mgr.14150 v1:172.21.15.92:0/996057664' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T21:25:29.115 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:25:29 smithi092 ceph-mon[29572]: from='mgr.14150 v1:172.21.15.92:0/996057664' entity='mgr.y' 2024-03-11T21:25:29.115 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:25:29 smithi092 ceph-mon[29572]: from='client.? v1:172.21.15.92:0/342231441' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-11T21:25:29.327 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:25:29 smithi097 ceph-mon[33317]: osdmap e62: 8 total, 8 up, 8 in 2024-03-11T21:25:29.327 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:25:29 smithi097 ceph-mon[33317]: Detected new or changed devices on smithi097 2024-03-11T21:25:29.327 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:25:29 smithi097 ceph-mon[33317]: from='mgr.14150 v1:172.21.15.92:0/996057664' entity='mgr.y' 2024-03-11T21:25:29.327 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:25:29 smithi097 ceph-mon[33317]: from='mgr.14150 v1:172.21.15.92:0/996057664' entity='mgr.y' 2024-03-11T21:25:29.327 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:25:29 smithi097 ceph-mon[33317]: from='mgr.14150 v1:172.21.15.92:0/996057664' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-11T21:25:29.328 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:25:29 smithi097 ceph-mon[33317]: from='mgr.14150 v1:172.21.15.92:0/996057664' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-11T21:25:29.328 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:25:29 smithi097 ceph-mon[33317]: from='mgr.14150 v1:172.21.15.92:0/996057664' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-03-11T21:25:29.328 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:25:29 smithi097 ceph-mon[33317]: from='mgr.14150 v1:172.21.15.92:0/996057664' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-03-11T21:25:29.328 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:25:29 smithi097 ceph-mon[33317]: Adjusting osd_memory_target on smithi097 to 4272M 2024-03-11T21:25:29.328 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:25:29 smithi097 ceph-mon[33317]: from='mgr.14150 v1:172.21.15.92:0/996057664' entity='mgr.y' 2024-03-11T21:25:29.328 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:25:29 smithi097 ceph-mon[33317]: from='mgr.14150 v1:172.21.15.92:0/996057664' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T21:25:29.328 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:25:29 smithi097 ceph-mon[33317]: from='mgr.14150 v1:172.21.15.92:0/996057664' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T21:25:29.328 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:25:29 smithi097 ceph-mon[33317]: from='mgr.14150 v1:172.21.15.92:0/996057664' entity='mgr.y' 2024-03-11T21:25:29.328 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:25:29 smithi097 ceph-mon[33317]: from='mgr.14150 v1:172.21.15.92:0/996057664' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T21:25:29.328 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:25:29 smithi097 ceph-mon[33317]: from='mgr.14150 v1:172.21.15.92:0/996057664' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T21:25:29.328 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:25:29 smithi097 ceph-mon[33317]: from='mgr.14150 v1:172.21.15.92:0/996057664' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T21:25:29.328 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:25:29 smithi097 ceph-mon[33317]: from='mgr.14150 v1:172.21.15.92:0/996057664' entity='mgr.y' 2024-03-11T21:25:29.328 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:25:29 smithi097 ceph-mon[33317]: from='client.? v1:172.21.15.92:0/342231441' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-11T21:25:29.508 INFO:tasks.cephadm.ceph_manager.ceph:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-11T21:23:23.805029+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '22', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 7.889999866485596, 'score_stable': 7.889999866485596, 'optimal_score': 0.3799999952316284, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-11T21:25:29.508 DEBUG:teuthology.orchestra.run.smithi092:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid 00db0016-dfed-11ee-95c8-87774f69a715 -- ceph osd pool get .mgr pg_num 2024-03-11T21:25:29.748 INFO:teuthology.orchestra.run.smithi092.stderr:Inferring config /var/lib/ceph/00db0016-dfed-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T21:25:30.827 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:25:30 smithi097 ceph-mon[33317]: osdmap e63: 8 total, 8 up, 8 in 2024-03-11T21:25:30.827 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:25:30 smithi097 ceph-mon[33317]: pgmap v183: 1 pgs: 1 activating+undersized+degraded+remapped; 577 KiB data, 221 MiB used, 715 GiB / 715 GiB avail; 2/6 objects degraded (33.333%) 2024-03-11T21:25:30.838 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:25:30 smithi092 ceph-mon[29572]: osdmap e63: 8 total, 8 up, 8 in 2024-03-11T21:25:30.838 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:25:30 smithi092 ceph-mon[29572]: pgmap v183: 1 pgs: 1 activating+undersized+degraded+remapped; 577 KiB data, 221 MiB used, 715 GiB / 715 GiB avail; 2/6 objects degraded (33.333%) 2024-03-11T21:25:30.838 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:25:30 smithi092 ceph-mon[37563]: osdmap e63: 8 total, 8 up, 8 in 2024-03-11T21:25:30.838 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:25:30 smithi092 ceph-mon[37563]: pgmap v183: 1 pgs: 1 activating+undersized+degraded+remapped; 577 KiB data, 221 MiB used, 715 GiB / 715 GiB avail; 2/6 objects degraded (33.333%) 2024-03-11T21:25:31.539 INFO:teuthology.orchestra.run.smithi092.stdout:pg_num: 1 2024-03-11T21:25:31.829 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:25:31 smithi097 ceph-mon[33317]: Health check failed: Degraded data redundancy: 2/6 objects degraded (33.333%), 1 pg degraded (PG_DEGRADED) 2024-03-11T21:25:31.829 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:25:31 smithi097 ceph-mon[33317]: osdmap e64: 8 total, 8 up, 8 in 2024-03-11T21:25:31.852 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:25:31 smithi092 ceph-mon[29572]: Health check failed: Degraded data redundancy: 2/6 objects degraded (33.333%), 1 pg degraded (PG_DEGRADED) 2024-03-11T21:25:31.853 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:25:31 smithi092 ceph-mon[29572]: osdmap e64: 8 total, 8 up, 8 in 2024-03-11T21:25:31.853 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:25:31 smithi092 ceph-mon[37563]: Health check failed: Degraded data redundancy: 2/6 objects degraded (33.333%), 1 pg degraded (PG_DEGRADED) 2024-03-11T21:25:31.853 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:25:31 smithi092 ceph-mon[37563]: osdmap e64: 8 total, 8 up, 8 in 2024-03-11T21:25:32.141 INFO:tasks.cephadm:Adding ceph.rgw.foo.a on smithi092 2024-03-11T21:25:32.142 DEBUG:teuthology.orchestra.run.smithi097:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 00db0016-dfed-11ee-95c8-87774f69a715 -- ceph orch apply rgw foo.a --placement '1;smithi092=foo.a' 2024-03-11T21:25:32.390 INFO:teuthology.orchestra.run.smithi097.stderr:Inferring config /var/lib/ceph/00db0016-dfed-11ee-95c8-87774f69a715/mon.b/config 2024-03-11T21:25:32.801 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:25:32 smithi097 ceph-mon[33317]: pgmap v185: 1 pgs: 1 active+clean; 577 KiB data, 221 MiB used, 715 GiB / 715 GiB avail; 128 KiB/s, 0 objects/s recovering 2024-03-11T21:25:32.801 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:25:32 smithi097 ceph-mon[33317]: from='client.? v1:172.21.15.92:0/2635413490' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-11T21:25:32.801 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:25:32 smithi097 ceph-mon[33317]: purged_snaps scrub starts 2024-03-11T21:25:32.801 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:25:32 smithi097 ceph-mon[33317]: purged_snaps scrub ok 2024-03-11T21:25:32.861 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:25:32 smithi092 ceph-mon[29572]: pgmap v185: 1 pgs: 1 active+clean; 577 KiB data, 221 MiB used, 715 GiB / 715 GiB avail; 128 KiB/s, 0 objects/s recovering 2024-03-11T21:25:32.861 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:25:32 smithi092 ceph-mon[29572]: from='client.? v1:172.21.15.92:0/2635413490' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-11T21:25:32.861 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:25:32 smithi092 ceph-mon[29572]: purged_snaps scrub starts 2024-03-11T21:25:32.862 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:25:32 smithi092 ceph-mon[29572]: purged_snaps scrub ok 2024-03-11T21:25:32.862 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:25:32 smithi092 ceph-mon[37563]: pgmap v185: 1 pgs: 1 active+clean; 577 KiB data, 221 MiB used, 715 GiB / 715 GiB avail; 128 KiB/s, 0 objects/s recovering 2024-03-11T21:25:32.862 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:25:32 smithi092 ceph-mon[37563]: from='client.? v1:172.21.15.92:0/2635413490' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-11T21:25:32.862 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:25:32 smithi092 ceph-mon[37563]: purged_snaps scrub starts 2024-03-11T21:25:32.862 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:25:32 smithi092 ceph-mon[37563]: purged_snaps scrub ok 2024-03-11T21:25:33.670 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:25:33 smithi097 ceph-mon[33317]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 2/6 objects degraded (33.333%), 1 pg degraded) 2024-03-11T21:25:33.670 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:25:33 smithi097 ceph-mon[33317]: Cluster is now healthy 2024-03-11T21:25:33.670 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:25:33 smithi097 ceph-mon[33317]: osdmap e65: 8 total, 8 up, 8 in 2024-03-11T21:25:33.861 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:25:33 smithi092 ceph-mon[29572]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 2/6 objects degraded (33.333%), 1 pg degraded) 2024-03-11T21:25:33.862 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:25:33 smithi092 ceph-mon[29572]: Cluster is now healthy 2024-03-11T21:25:33.862 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:25:33 smithi092 ceph-mon[29572]: osdmap e65: 8 total, 8 up, 8 in 2024-03-11T21:25:33.862 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:25:33 smithi092 ceph-mon[37563]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 2/6 objects degraded (33.333%), 1 pg degraded) 2024-03-11T21:25:33.862 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:25:33 smithi092 ceph-mon[37563]: Cluster is now healthy 2024-03-11T21:25:33.862 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:25:33 smithi092 ceph-mon[37563]: osdmap e65: 8 total, 8 up, 8 in 2024-03-11T21:25:34.126 INFO:teuthology.orchestra.run.smithi097.stdout:Scheduled rgw.foo.a update... 2024-03-11T21:25:34.760 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:25:34 smithi097 ceph-mon[33317]: pgmap v187: 1 pgs: 1 active+clean; 577 KiB data, 220 MiB used, 715 GiB / 715 GiB avail; 105 KiB/s, 0 objects/s recovering 2024-03-11T21:25:34.761 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:25:34 smithi097 ceph-mon[33317]: from='mgr.14150 v1:172.21.15.92:0/996057664' entity='mgr.y' 2024-03-11T21:25:34.761 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:25:34 smithi097 ceph-mon[33317]: from='mgr.14150 v1:172.21.15.92:0/996057664' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T21:25:34.761 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:25:34 smithi097 ceph-mon[33317]: from='mgr.14150 v1:172.21.15.92:0/996057664' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T21:25:34.761 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:25:34 smithi097 ceph-mon[33317]: from='mgr.14150 v1:172.21.15.92:0/996057664' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T21:25:34.761 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:25:34 smithi097 ceph-mon[33317]: from='mgr.14150 v1:172.21.15.92:0/996057664' entity='mgr.y' 2024-03-11T21:25:34.761 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:25:34 smithi097 ceph-mon[33317]: from='mgr.14150 v1:172.21.15.92:0/996057664' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.rgw.foo.a", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2024-03-11T21:25:34.761 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:25:34 smithi097 ceph-mon[33317]: from='mgr.14150 v1:172.21.15.92:0/996057664' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "client.rgw.foo.a", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]': finished 2024-03-11T21:25:34.761 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:25:34 smithi097 ceph-mon[33317]: from='mgr.14150 v1:172.21.15.92:0/996057664' entity='mgr.y' 2024-03-11T21:25:34.761 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:25:34 smithi097 ceph-mon[33317]: from='mgr.14150 v1:172.21.15.92:0/996057664' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T21:25:34.762 DEBUG:teuthology.orchestra.run.smithi092:rgw.foo.a> sudo journalctl -f -n 0 -u ceph-00db0016-dfed-11ee-95c8-87774f69a715@rgw.foo.a.service 2024-03-11T21:25:34.765 INFO:tasks.cephadm:Adding ceph.iscsi.iscsi.a on smithi097 2024-03-11T21:25:34.765 DEBUG:teuthology.orchestra.run.smithi097:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 00db0016-dfed-11ee-95c8-87774f69a715 -- ceph osd pool create datapool 3 3 replicated 2024-03-11T21:25:34.787 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:25:34 smithi092 ceph-mon[29572]: pgmap v187: 1 pgs: 1 active+clean; 577 KiB data, 220 MiB used, 715 GiB / 715 GiB avail; 105 KiB/s, 0 objects/s recovering 2024-03-11T21:25:34.787 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:25:34 smithi092 ceph-mon[29572]: from='mgr.14150 v1:172.21.15.92:0/996057664' entity='mgr.y' 2024-03-11T21:25:34.787 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:25:34 smithi092 ceph-mon[29572]: from='mgr.14150 v1:172.21.15.92:0/996057664' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T21:25:34.787 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:25:34 smithi092 ceph-mon[29572]: from='mgr.14150 v1:172.21.15.92:0/996057664' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T21:25:34.787 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:25:34 smithi092 ceph-mon[29572]: from='mgr.14150 v1:172.21.15.92:0/996057664' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T21:25:34.788 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:25:34 smithi092 ceph-mon[29572]: from='mgr.14150 v1:172.21.15.92:0/996057664' entity='mgr.y' 2024-03-11T21:25:34.788 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:25:34 smithi092 ceph-mon[29572]: from='mgr.14150 v1:172.21.15.92:0/996057664' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.rgw.foo.a", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2024-03-11T21:25:34.788 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:25:34 smithi092 ceph-mon[29572]: from='mgr.14150 v1:172.21.15.92:0/996057664' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "client.rgw.foo.a", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]': finished 2024-03-11T21:25:34.788 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:25:34 smithi092 ceph-mon[29572]: from='mgr.14150 v1:172.21.15.92:0/996057664' entity='mgr.y' 2024-03-11T21:25:34.788 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:25:34 smithi092 ceph-mon[29572]: from='mgr.14150 v1:172.21.15.92:0/996057664' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T21:25:34.788 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:25:34 smithi092 ceph-mon[37563]: pgmap v187: 1 pgs: 1 active+clean; 577 KiB data, 220 MiB used, 715 GiB / 715 GiB avail; 105 KiB/s, 0 objects/s recovering 2024-03-11T21:25:34.788 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:25:34 smithi092 ceph-mon[37563]: from='mgr.14150 v1:172.21.15.92:0/996057664' entity='mgr.y' 2024-03-11T21:25:34.788 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:25:34 smithi092 ceph-mon[37563]: from='mgr.14150 v1:172.21.15.92:0/996057664' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T21:25:34.789 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:25:34 smithi092 ceph-mon[37563]: from='mgr.14150 v1:172.21.15.92:0/996057664' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T21:25:34.789 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:25:34 smithi092 ceph-mon[37563]: from='mgr.14150 v1:172.21.15.92:0/996057664' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T21:25:34.789 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:25:34 smithi092 ceph-mon[37563]: from='mgr.14150 v1:172.21.15.92:0/996057664' entity='mgr.y' 2024-03-11T21:25:34.789 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:25:34 smithi092 ceph-mon[37563]: from='mgr.14150 v1:172.21.15.92:0/996057664' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.rgw.foo.a", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2024-03-11T21:25:34.789 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:25:34 smithi092 ceph-mon[37563]: from='mgr.14150 v1:172.21.15.92:0/996057664' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "client.rgw.foo.a", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]': finished 2024-03-11T21:25:34.789 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:25:34 smithi092 ceph-mon[37563]: from='mgr.14150 v1:172.21.15.92:0/996057664' entity='mgr.y' 2024-03-11T21:25:34.789 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:25:34 smithi092 ceph-mon[37563]: from='mgr.14150 v1:172.21.15.92:0/996057664' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T21:25:35.000 INFO:teuthology.orchestra.run.smithi097.stderr:Inferring config /var/lib/ceph/00db0016-dfed-11ee-95c8-87774f69a715/mon.b/config 2024-03-11T21:25:35.793 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:25:35 smithi097 ceph-mon[33317]: from='client.24358 v1:172.21.15.97:0/3588582428' entity='client.admin' cmd=[{"prefix": "orch apply rgw", "svc_id": "foo.a", "placement": "1;smithi092=foo.a", "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T21:25:35.793 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:25:35 smithi097 ceph-mon[33317]: Saving service rgw.foo.a spec with placement smithi092=foo.a;count:1 2024-03-11T21:25:35.793 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:25:35 smithi097 ceph-mon[33317]: Deploying daemon rgw.foo.a on smithi092 2024-03-11T21:25:35.810 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:25:35 smithi092 ceph-mon[29572]: from='client.24358 v1:172.21.15.97:0/3588582428' entity='client.admin' cmd=[{"prefix": "orch apply rgw", "svc_id": "foo.a", "placement": "1;smithi092=foo.a", "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T21:25:35.810 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:25:35 smithi092 ceph-mon[29572]: Saving service rgw.foo.a spec with placement smithi092=foo.a;count:1 2024-03-11T21:25:35.810 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:25:35 smithi092 ceph-mon[29572]: Deploying daemon rgw.foo.a on smithi092 2024-03-11T21:25:35.810 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:25:35 smithi092 ceph-mon[37563]: from='client.24358 v1:172.21.15.97:0/3588582428' entity='client.admin' cmd=[{"prefix": "orch apply rgw", "svc_id": "foo.a", "placement": "1;smithi092=foo.a", "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T21:25:35.811 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:25:35 smithi092 ceph-mon[37563]: Saving service rgw.foo.a spec with placement smithi092=foo.a;count:1 2024-03-11T21:25:35.811 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:25:35 smithi092 ceph-mon[37563]: Deploying daemon rgw.foo.a on smithi092 2024-03-11T21:25:36.361 INFO:journalctl@ceph.rgw.foo.a.smithi092.stdout:Mar 11 21:25:36 smithi092 systemd[1]: Starting Ceph rgw.foo.a for 00db0016-dfed-11ee-95c8-87774f69a715... 2024-03-11T21:25:36.752 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:25:36 smithi092 ceph-mon[29572]: pgmap v188: 1 pgs: 1 active+clean; 577 KiB data, 220 MiB used, 715 GiB / 715 GiB avail; 96 KiB/s, 0 objects/s recovering 2024-03-11T21:25:36.753 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:25:36 smithi092 ceph-mon[37563]: pgmap v188: 1 pgs: 1 active+clean; 577 KiB data, 220 MiB used, 715 GiB / 715 GiB avail; 96 KiB/s, 0 objects/s recovering 2024-03-11T21:25:36.753 INFO:journalctl@ceph.rgw.foo.a.smithi092.stdout:Mar 11 21:25:36 smithi092 podman[64248]: 2024-03-11 21:25:36.410080898 +0000 UTC m=+0.018350616 image pull 4133f8bc41bb201212a28168427a2edb1f6f3795e694bf61031ea9e2f3ff5e87 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:8fe9635f3beb0247496d7e321f3068bd10c333c433ef4824d7e1c1a7d3d78379 2024-03-11T21:25:36.753 INFO:journalctl@ceph.rgw.foo.a.smithi092.stdout:Mar 11 21:25:36 smithi092 podman[64248]: 2024-03-11 21:25:36.550758878 +0000 UTC m=+0.159028584 container create 81ad59000ca20f444683eed7ed2e88d14839dfff4afe044b028e922cdbe68b1f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:8fe9635f3beb0247496d7e321f3068bd10c333c433ef4824d7e1c1a7d3d78379, name=ceph-00db0016-dfed-11ee-95c8-87774f69a715-rgw-foo-a, ceph=True, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , RELEASE=main-2b0101e, GIT_CLEAN=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, org.label-schema.build-date=20240307) 2024-03-11T21:25:36.753 INFO:journalctl@ceph.rgw.foo.a.smithi092.stdout:Mar 11 21:25:36 smithi092 podman[64248]: 2024-03-11 21:25:36.770059393 +0000 UTC m=+0.378329113 container init 81ad59000ca20f444683eed7ed2e88d14839dfff4afe044b028e922cdbe68b1f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:8fe9635f3beb0247496d7e321f3068bd10c333c433ef4824d7e1c1a7d3d78379, name=ceph-00db0016-dfed-11ee-95c8-87774f69a715-rgw-foo-a, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, ceph=True, RELEASE=main-2b0101e, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, org.label-schema.build-date=20240307, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image) 2024-03-11T21:25:36.827 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:25:36 smithi097 ceph-mon[33317]: pgmap v188: 1 pgs: 1 active+clean; 577 KiB data, 220 MiB used, 715 GiB / 715 GiB avail; 96 KiB/s, 0 objects/s recovering 2024-03-11T21:25:37.111 INFO:journalctl@ceph.rgw.foo.a.smithi092.stdout:Mar 11 21:25:36 smithi092 podman[64248]: 2024-03-11 21:25:36.773005772 +0000 UTC m=+0.381275485 container start 81ad59000ca20f444683eed7ed2e88d14839dfff4afe044b028e922cdbe68b1f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:8fe9635f3beb0247496d7e321f3068bd10c333c433ef4824d7e1c1a7d3d78379, name=ceph-00db0016-dfed-11ee-95c8-87774f69a715-rgw-foo-a, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, RELEASE=main-2b0101e, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, org.label-schema.build-date=20240307, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux ) 2024-03-11T21:25:37.111 INFO:journalctl@ceph.rgw.foo.a.smithi092.stdout:Mar 11 21:25:36 smithi092 bash[64248]: 81ad59000ca20f444683eed7ed2e88d14839dfff4afe044b028e922cdbe68b1f 2024-03-11T21:25:37.111 INFO:journalctl@ceph.rgw.foo.a.smithi092.stdout:Mar 11 21:25:36 smithi092 systemd[1]: Started Ceph rgw.foo.a for 00db0016-dfed-11ee-95c8-87774f69a715. 2024-03-11T21:25:37.514 INFO:teuthology.orchestra.run.smithi097.stderr:pool 'datapool' created 2024-03-11T21:25:37.803 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:25:37 smithi097 ceph-mon[33317]: from='client.? v1:172.21.15.97:0/791702814' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "datapool", "pg_num": 3, "pgp_num": 3, "pool_type": "replicated"}]: dispatch 2024-03-11T21:25:37.804 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:25:37 smithi097 ceph-mon[33317]: from='client.24364 ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "datapool", "pg_num": 3, "pgp_num": 3, "pool_type": "replicated"}]: dispatch 2024-03-11T21:25:37.804 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:25:37 smithi097 ceph-mon[33317]: from='mgr.14150 v1:172.21.15.92:0/996057664' entity='mgr.y' 2024-03-11T21:25:37.804 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:25:37 smithi097 ceph-mon[33317]: from='mgr.14150 v1:172.21.15.92:0/996057664' entity='mgr.y' 2024-03-11T21:25:37.804 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:25:37 smithi097 ceph-mon[33317]: from='mgr.14150 v1:172.21.15.92:0/996057664' entity='mgr.y' 2024-03-11T21:25:37.804 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:25:37 smithi097 ceph-mon[33317]: from='mgr.14150 v1:172.21.15.92:0/996057664' entity='mgr.y' 2024-03-11T21:25:37.804 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:25:37 smithi097 ceph-mon[33317]: from='mgr.14150 v1:172.21.15.92:0/996057664' entity='mgr.y' 2024-03-11T21:25:37.804 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:25:37 smithi097 ceph-mon[33317]: from='mgr.14150 v1:172.21.15.92:0/996057664' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T21:25:37.861 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:25:37 smithi092 ceph-mon[37563]: from='client.? v1:172.21.15.97:0/791702814' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "datapool", "pg_num": 3, "pgp_num": 3, "pool_type": "replicated"}]: dispatch 2024-03-11T21:25:37.861 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:25:37 smithi092 ceph-mon[37563]: from='client.24364 ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "datapool", "pg_num": 3, "pgp_num": 3, "pool_type": "replicated"}]: dispatch 2024-03-11T21:25:37.861 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:25:37 smithi092 ceph-mon[37563]: from='mgr.14150 v1:172.21.15.92:0/996057664' entity='mgr.y' 2024-03-11T21:25:37.862 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:25:37 smithi092 ceph-mon[37563]: from='mgr.14150 v1:172.21.15.92:0/996057664' entity='mgr.y' 2024-03-11T21:25:37.862 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:25:37 smithi092 ceph-mon[37563]: from='mgr.14150 v1:172.21.15.92:0/996057664' entity='mgr.y' 2024-03-11T21:25:37.862 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:25:37 smithi092 ceph-mon[37563]: from='mgr.14150 v1:172.21.15.92:0/996057664' entity='mgr.y' 2024-03-11T21:25:37.862 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:25:37 smithi092 ceph-mon[37563]: from='mgr.14150 v1:172.21.15.92:0/996057664' entity='mgr.y' 2024-03-11T21:25:37.862 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:25:37 smithi092 ceph-mon[37563]: from='mgr.14150 v1:172.21.15.92:0/996057664' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T21:25:37.862 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:25:37 smithi092 ceph-mon[29572]: from='client.? v1:172.21.15.97:0/791702814' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "datapool", "pg_num": 3, "pgp_num": 3, "pool_type": "replicated"}]: dispatch 2024-03-11T21:25:37.862 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:25:37 smithi092 ceph-mon[29572]: from='client.24364 ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "datapool", "pg_num": 3, "pgp_num": 3, "pool_type": "replicated"}]: dispatch 2024-03-11T21:25:37.862 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:25:37 smithi092 ceph-mon[29572]: from='mgr.14150 v1:172.21.15.92:0/996057664' entity='mgr.y' 2024-03-11T21:25:37.862 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:25:37 smithi092 ceph-mon[29572]: from='mgr.14150 v1:172.21.15.92:0/996057664' entity='mgr.y' 2024-03-11T21:25:37.863 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:25:37 smithi092 ceph-mon[29572]: from='mgr.14150 v1:172.21.15.92:0/996057664' entity='mgr.y' 2024-03-11T21:25:37.863 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:25:37 smithi092 ceph-mon[29572]: from='mgr.14150 v1:172.21.15.92:0/996057664' entity='mgr.y' 2024-03-11T21:25:37.863 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:25:37 smithi092 ceph-mon[29572]: from='mgr.14150 v1:172.21.15.92:0/996057664' entity='mgr.y' 2024-03-11T21:25:37.863 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:25:37 smithi092 ceph-mon[29572]: from='mgr.14150 v1:172.21.15.92:0/996057664' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T21:25:38.142 DEBUG:teuthology.orchestra.run.smithi097:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 00db0016-dfed-11ee-95c8-87774f69a715 -- rbd pool init datapool 2024-03-11T21:25:38.377 INFO:teuthology.orchestra.run.smithi097.stderr:Inferring config /var/lib/ceph/00db0016-dfed-11ee-95c8-87774f69a715/mon.b/config 2024-03-11T21:25:38.745 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:25:38 smithi097 ceph-mon[33317]: Saving service rgw.foo.a spec with placement smithi092=foo.a;count:1 2024-03-11T21:25:38.745 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:25:38 smithi097 ceph-mon[33317]: from='client.24364 ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "datapool", "pg_num": 3, "pgp_num": 3, "pool_type": "replicated"}]': finished 2024-03-11T21:25:38.745 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:25:38 smithi097 ceph-mon[33317]: pgmap v190: 36 pgs: 35 unknown, 1 active+clean; 577 KiB data, 220 MiB used, 715 GiB / 715 GiB avail; 82 KiB/s, 0 objects/s recovering 2024-03-11T21:25:38.745 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:25:38 smithi097 ceph-mon[33317]: osdmap e66: 8 total, 8 up, 8 in 2024-03-11T21:25:38.745 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:25:38 smithi097 ceph-mon[33317]: from='client.? v1:172.21.15.92:0/3010846684' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool application enable","pool": ".rgw.root","app": "rgw"}]: dispatch 2024-03-11T21:25:38.745 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:25:38 smithi097 ceph-mon[33317]: from='client.34343 ' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool application enable","pool": ".rgw.root","app": "rgw"}]: dispatch 2024-03-11T21:25:38.861 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:25:38 smithi092 ceph-mon[37563]: Saving service rgw.foo.a spec with placement smithi092=foo.a;count:1 2024-03-11T21:25:38.861 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:25:38 smithi092 ceph-mon[37563]: from='client.24364 ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "datapool", "pg_num": 3, "pgp_num": 3, "pool_type": "replicated"}]': finished 2024-03-11T21:25:38.861 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:25:38 smithi092 ceph-mon[37563]: pgmap v190: 36 pgs: 35 unknown, 1 active+clean; 577 KiB data, 220 MiB used, 715 GiB / 715 GiB avail; 82 KiB/s, 0 objects/s recovering 2024-03-11T21:25:38.862 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:25:38 smithi092 ceph-mon[37563]: osdmap e66: 8 total, 8 up, 8 in 2024-03-11T21:25:38.862 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:25:38 smithi092 ceph-mon[37563]: from='client.? v1:172.21.15.92:0/3010846684' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool application enable","pool": ".rgw.root","app": "rgw"}]: dispatch 2024-03-11T21:25:38.862 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:25:38 smithi092 ceph-mon[37563]: from='client.34343 ' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool application enable","pool": ".rgw.root","app": "rgw"}]: dispatch 2024-03-11T21:25:38.862 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:25:38 smithi092 ceph-mon[29572]: Saving service rgw.foo.a spec with placement smithi092=foo.a;count:1 2024-03-11T21:25:38.862 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:25:38 smithi092 ceph-mon[29572]: from='client.24364 ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "datapool", "pg_num": 3, "pgp_num": 3, "pool_type": "replicated"}]': finished 2024-03-11T21:25:38.862 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:25:38 smithi092 ceph-mon[29572]: pgmap v190: 36 pgs: 35 unknown, 1 active+clean; 577 KiB data, 220 MiB used, 715 GiB / 715 GiB avail; 82 KiB/s, 0 objects/s recovering 2024-03-11T21:25:38.862 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:25:38 smithi092 ceph-mon[29572]: osdmap e66: 8 total, 8 up, 8 in 2024-03-11T21:25:38.862 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:25:38 smithi092 ceph-mon[29572]: from='client.? v1:172.21.15.92:0/3010846684' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool application enable","pool": ".rgw.root","app": "rgw"}]: dispatch 2024-03-11T21:25:38.862 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:25:38 smithi092 ceph-mon[29572]: from='client.34343 ' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool application enable","pool": ".rgw.root","app": "rgw"}]: dispatch 2024-03-11T21:25:39.574 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:25:39 smithi097 ceph-mon[33317]: Health check failed: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-11T21:25:39.574 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:25:39 smithi097 ceph-mon[33317]: from='client.34343 ' entity='client.rgw.foo.a' cmd='[{"prefix": "osd pool application enable","pool": ".rgw.root","app": "rgw"}]': finished 2024-03-11T21:25:39.574 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:25:39 smithi097 ceph-mon[33317]: osdmap e67: 8 total, 8 up, 8 in 2024-03-11T21:25:39.574 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:25:39 smithi097 ceph-mon[33317]: from='mgr.14150 v1:172.21.15.92:0/996057664' entity='mgr.y' 2024-03-11T21:25:39.574 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:25:39 smithi097 ceph-mon[33317]: from='mgr.14150 v1:172.21.15.92:0/996057664' entity='mgr.y' 2024-03-11T21:25:39.574 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:25:39 smithi097 ceph-mon[33317]: from='mgr.14150 v1:172.21.15.92:0/996057664' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T21:25:39.574 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:25:39 smithi097 ceph-mon[33317]: from='mgr.14150 v1:172.21.15.92:0/996057664' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T21:25:39.861 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:25:39 smithi092 ceph-mon[37563]: Health check failed: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-11T21:25:39.862 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:25:39 smithi092 ceph-mon[37563]: from='client.34343 ' entity='client.rgw.foo.a' cmd='[{"prefix": "osd pool application enable","pool": ".rgw.root","app": "rgw"}]': finished 2024-03-11T21:25:39.862 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:25:39 smithi092 ceph-mon[37563]: osdmap e67: 8 total, 8 up, 8 in 2024-03-11T21:25:39.862 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:25:39 smithi092 ceph-mon[37563]: from='mgr.14150 v1:172.21.15.92:0/996057664' entity='mgr.y' 2024-03-11T21:25:39.862 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:25:39 smithi092 ceph-mon[37563]: from='mgr.14150 v1:172.21.15.92:0/996057664' entity='mgr.y' 2024-03-11T21:25:39.862 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:25:39 smithi092 ceph-mon[37563]: from='mgr.14150 v1:172.21.15.92:0/996057664' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T21:25:39.862 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:25:39 smithi092 ceph-mon[37563]: from='mgr.14150 v1:172.21.15.92:0/996057664' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T21:25:39.862 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:25:39 smithi092 ceph-mon[29572]: Health check failed: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-11T21:25:39.862 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:25:39 smithi092 ceph-mon[29572]: from='client.34343 ' entity='client.rgw.foo.a' cmd='[{"prefix": "osd pool application enable","pool": ".rgw.root","app": "rgw"}]': finished 2024-03-11T21:25:39.862 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:25:39 smithi092 ceph-mon[29572]: osdmap e67: 8 total, 8 up, 8 in 2024-03-11T21:25:39.862 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:25:39 smithi092 ceph-mon[29572]: from='mgr.14150 v1:172.21.15.92:0/996057664' entity='mgr.y' 2024-03-11T21:25:39.862 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:25:39 smithi092 ceph-mon[29572]: from='mgr.14150 v1:172.21.15.92:0/996057664' entity='mgr.y' 2024-03-11T21:25:39.863 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:25:39 smithi092 ceph-mon[29572]: from='mgr.14150 v1:172.21.15.92:0/996057664' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T21:25:39.863 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:25:39 smithi092 ceph-mon[29572]: from='mgr.14150 v1:172.21.15.92:0/996057664' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T21:25:40.827 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:25:40 smithi097 ceph-mon[33317]: Checking dashboard <-> RGW credentials 2024-03-11T21:25:40.828 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:25:40 smithi097 ceph-mon[33317]: pgmap v192: 36 pgs: 18 creating+peering, 1 active+clean, 17 unknown; 577 KiB data, 220 MiB used, 715 GiB / 715 GiB avail 2024-03-11T21:25:40.828 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:25:40 smithi097 ceph-mon[33317]: osdmap e68: 8 total, 8 up, 8 in 2024-03-11T21:25:40.828 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:25:40 smithi097 ceph-mon[33317]: from='client.? v1:172.21.15.92:0/372197479' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.log","app": "rgw"}]: dispatch 2024-03-11T21:25:40.828 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:25:40 smithi097 ceph-mon[33317]: from='client.34352 ' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.log","app": "rgw"}]: dispatch 2024-03-11T21:25:40.828 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:25:40 smithi097 ceph-mon[33317]: from='client.? v1:172.21.15.92:0/2960078881' entity='mgr.y' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.log","app": "rgw"}]: dispatch 2024-03-11T21:25:40.828 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:25:40 smithi097 ceph-mon[33317]: from='mgr.14150 v1:172.21.15.92:0/996057664' entity='mgr.y' 2024-03-11T21:25:40.828 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:25:40 smithi097 ceph-mon[33317]: from='client.? v1:172.21.15.97:0/3954111469' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "datapool","app": "rbd"}]: dispatch 2024-03-11T21:25:40.828 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:25:40 smithi097 ceph-mon[33317]: from='client.24418 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "datapool","app": "rbd"}]: dispatch 2024-03-11T21:25:40.861 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:25:40 smithi092 ceph-mon[29572]: Checking dashboard <-> RGW credentials 2024-03-11T21:25:40.861 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:25:40 smithi092 ceph-mon[29572]: pgmap v192: 36 pgs: 18 creating+peering, 1 active+clean, 17 unknown; 577 KiB data, 220 MiB used, 715 GiB / 715 GiB avail 2024-03-11T21:25:40.861 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:25:40 smithi092 ceph-mon[29572]: osdmap e68: 8 total, 8 up, 8 in 2024-03-11T21:25:40.861 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:25:40 smithi092 ceph-mon[29572]: from='client.? v1:172.21.15.92:0/372197479' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.log","app": "rgw"}]: dispatch 2024-03-11T21:25:40.862 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:25:40 smithi092 ceph-mon[29572]: from='client.34352 ' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.log","app": "rgw"}]: dispatch 2024-03-11T21:25:40.862 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:25:40 smithi092 ceph-mon[29572]: from='client.? v1:172.21.15.92:0/2960078881' entity='mgr.y' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.log","app": "rgw"}]: dispatch 2024-03-11T21:25:40.862 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:25:40 smithi092 ceph-mon[29572]: from='mgr.14150 v1:172.21.15.92:0/996057664' entity='mgr.y' 2024-03-11T21:25:40.862 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:25:40 smithi092 ceph-mon[29572]: from='client.? v1:172.21.15.97:0/3954111469' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "datapool","app": "rbd"}]: dispatch 2024-03-11T21:25:40.862 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:25:40 smithi092 ceph-mon[29572]: from='client.24418 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "datapool","app": "rbd"}]: dispatch 2024-03-11T21:25:40.862 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:25:40 smithi092 ceph-mon[37563]: Checking dashboard <-> RGW credentials 2024-03-11T21:25:40.862 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:25:40 smithi092 ceph-mon[37563]: pgmap v192: 36 pgs: 18 creating+peering, 1 active+clean, 17 unknown; 577 KiB data, 220 MiB used, 715 GiB / 715 GiB avail 2024-03-11T21:25:40.862 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:25:40 smithi092 ceph-mon[37563]: osdmap e68: 8 total, 8 up, 8 in 2024-03-11T21:25:40.862 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:25:40 smithi092 ceph-mon[37563]: from='client.? v1:172.21.15.92:0/372197479' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.log","app": "rgw"}]: dispatch 2024-03-11T21:25:40.862 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:25:40 smithi092 ceph-mon[37563]: from='client.34352 ' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.log","app": "rgw"}]: dispatch 2024-03-11T21:25:40.862 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:25:40 smithi092 ceph-mon[37563]: from='client.? v1:172.21.15.92:0/2960078881' entity='mgr.y' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.log","app": "rgw"}]: dispatch 2024-03-11T21:25:40.862 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:25:40 smithi092 ceph-mon[37563]: from='mgr.14150 v1:172.21.15.92:0/996057664' entity='mgr.y' 2024-03-11T21:25:40.863 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:25:40 smithi092 ceph-mon[37563]: from='client.? v1:172.21.15.97:0/3954111469' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "datapool","app": "rbd"}]: dispatch 2024-03-11T21:25:40.863 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:25:40 smithi092 ceph-mon[37563]: from='client.24418 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "datapool","app": "rbd"}]: dispatch 2024-03-11T21:25:41.827 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:25:41 smithi097 ceph-mon[33317]: from='client.34352 ' entity='client.rgw.foo.a' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.log","app": "rgw"}]': finished 2024-03-11T21:25:41.827 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:25:41 smithi097 ceph-mon[33317]: from='client.? v1:172.21.15.92:0/2960078881' entity='mgr.y' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.log","app": "rgw"}]': finished 2024-03-11T21:25:41.827 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:25:41 smithi097 ceph-mon[33317]: from='client.24418 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "datapool","app": "rbd"}]': finished 2024-03-11T21:25:41.827 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:25:41 smithi097 ceph-mon[33317]: osdmap e69: 8 total, 8 up, 8 in 2024-03-11T21:25:41.861 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:25:41 smithi092 ceph-mon[29572]: from='client.34352 ' entity='client.rgw.foo.a' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.log","app": "rgw"}]': finished 2024-03-11T21:25:41.861 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:25:41 smithi092 ceph-mon[29572]: from='client.? v1:172.21.15.92:0/2960078881' entity='mgr.y' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.log","app": "rgw"}]': finished 2024-03-11T21:25:41.861 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:25:41 smithi092 ceph-mon[29572]: from='client.24418 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "datapool","app": "rbd"}]': finished 2024-03-11T21:25:41.861 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:25:41 smithi092 ceph-mon[29572]: osdmap e69: 8 total, 8 up, 8 in 2024-03-11T21:25:41.862 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:25:41 smithi092 ceph-mon[37563]: from='client.34352 ' entity='client.rgw.foo.a' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.log","app": "rgw"}]': finished 2024-03-11T21:25:41.862 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:25:41 smithi092 ceph-mon[37563]: from='client.? v1:172.21.15.92:0/2960078881' entity='mgr.y' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.log","app": "rgw"}]': finished 2024-03-11T21:25:41.862 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:25:41 smithi092 ceph-mon[37563]: from='client.24418 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "datapool","app": "rbd"}]': finished 2024-03-11T21:25:41.862 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:25:41 smithi092 ceph-mon[37563]: osdmap e69: 8 total, 8 up, 8 in 2024-03-11T21:25:42.827 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:25:42 smithi097 ceph-mon[33317]: pgmap v195: 68 pgs: 22 active+clean, 2 creating+activating, 18 creating+peering, 26 unknown; 577 KiB data, 221 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 767 B/s wr, 3 op/s 2024-03-11T21:25:42.827 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:25:42 smithi097 ceph-mon[33317]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-03-11T21:25:42.827 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:25:42 smithi097 ceph-mon[33317]: Cluster is now healthy 2024-03-11T21:25:42.827 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:25:42 smithi097 ceph-mon[33317]: osdmap e70: 8 total, 8 up, 8 in 2024-03-11T21:25:42.828 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:25:42 smithi097 ceph-mon[33317]: from='client.? v1:172.21.15.92:0/372197479' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.control","app": "rgw"}]: dispatch 2024-03-11T21:25:42.828 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:25:42 smithi097 ceph-mon[33317]: from='client.? v1:172.21.15.92:0/2960078881' entity='mgr.y' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.control","app": "rgw"}]: dispatch 2024-03-11T21:25:42.828 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:25:42 smithi097 ceph-mon[33317]: from='client.34352 ' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.control","app": "rgw"}]: dispatch 2024-03-11T21:25:42.861 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:25:42 smithi092 ceph-mon[29572]: pgmap v195: 68 pgs: 22 active+clean, 2 creating+activating, 18 creating+peering, 26 unknown; 577 KiB data, 221 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 767 B/s wr, 3 op/s 2024-03-11T21:25:42.861 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:25:42 smithi092 ceph-mon[29572]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-03-11T21:25:42.861 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:25:42 smithi092 ceph-mon[29572]: Cluster is now healthy 2024-03-11T21:25:42.861 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:25:42 smithi092 ceph-mon[29572]: osdmap e70: 8 total, 8 up, 8 in 2024-03-11T21:25:42.861 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:25:42 smithi092 ceph-mon[29572]: from='client.? v1:172.21.15.92:0/372197479' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.control","app": "rgw"}]: dispatch 2024-03-11T21:25:42.861 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:25:42 smithi092 ceph-mon[29572]: from='client.? v1:172.21.15.92:0/2960078881' entity='mgr.y' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.control","app": "rgw"}]: dispatch 2024-03-11T21:25:42.861 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:25:42 smithi092 ceph-mon[29572]: from='client.34352 ' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.control","app": "rgw"}]: dispatch 2024-03-11T21:25:42.862 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:25:42 smithi092 ceph-mon[37563]: pgmap v195: 68 pgs: 22 active+clean, 2 creating+activating, 18 creating+peering, 26 unknown; 577 KiB data, 221 MiB used, 715 GiB / 715 GiB avail; 2.5 KiB/s rd, 767 B/s wr, 3 op/s 2024-03-11T21:25:42.862 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:25:42 smithi092 ceph-mon[37563]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-03-11T21:25:42.862 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:25:42 smithi092 ceph-mon[37563]: Cluster is now healthy 2024-03-11T21:25:42.862 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:25:42 smithi092 ceph-mon[37563]: osdmap e70: 8 total, 8 up, 8 in 2024-03-11T21:25:42.862 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:25:42 smithi092 ceph-mon[37563]: from='client.? v1:172.21.15.92:0/372197479' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.control","app": "rgw"}]: dispatch 2024-03-11T21:25:42.862 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:25:42 smithi092 ceph-mon[37563]: from='client.? v1:172.21.15.92:0/2960078881' entity='mgr.y' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.control","app": "rgw"}]: dispatch 2024-03-11T21:25:42.862 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:25:42 smithi092 ceph-mon[37563]: from='client.34352 ' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.control","app": "rgw"}]: dispatch 2024-03-11T21:25:43.255 DEBUG:teuthology.orchestra.run.smithi097:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 00db0016-dfed-11ee-95c8-87774f69a715 -- ceph orch apply iscsi datapool admin admin --trusted_ip_list 172.21.15.97 --placement '1;smithi097=iscsi.a' 2024-03-11T21:25:43.497 INFO:teuthology.orchestra.run.smithi097.stderr:Inferring config /var/lib/ceph/00db0016-dfed-11ee-95c8-87774f69a715/mon.b/config 2024-03-11T21:25:43.827 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:25:43 smithi097 ceph-mon[33317]: from='client.? v1:172.21.15.92:0/2960078881' entity='mgr.y' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.control","app": "rgw"}]': finished 2024-03-11T21:25:43.827 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:25:43 smithi097 ceph-mon[33317]: from='client.34352 ' entity='client.rgw.foo.a' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.control","app": "rgw"}]': finished 2024-03-11T21:25:43.827 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:25:43 smithi097 ceph-mon[33317]: osdmap e71: 8 total, 8 up, 8 in 2024-03-11T21:25:43.861 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:25:43 smithi092 ceph-mon[29572]: from='client.? v1:172.21.15.92:0/2960078881' entity='mgr.y' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.control","app": "rgw"}]': finished 2024-03-11T21:25:43.861 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:25:43 smithi092 ceph-mon[29572]: from='client.34352 ' entity='client.rgw.foo.a' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.control","app": "rgw"}]': finished 2024-03-11T21:25:43.861 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:25:43 smithi092 ceph-mon[29572]: osdmap e71: 8 total, 8 up, 8 in 2024-03-11T21:25:43.862 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:25:43 smithi092 ceph-mon[37563]: from='client.? v1:172.21.15.92:0/2960078881' entity='mgr.y' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.control","app": "rgw"}]': finished 2024-03-11T21:25:43.862 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:25:43 smithi092 ceph-mon[37563]: from='client.34352 ' entity='client.rgw.foo.a' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.control","app": "rgw"}]': finished 2024-03-11T21:25:43.862 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:25:43 smithi092 ceph-mon[37563]: osdmap e71: 8 total, 8 up, 8 in 2024-03-11T21:25:44.792 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:25:44 smithi097 ceph-mon[33317]: pgmap v198: 100 pgs: 82 active+clean, 2 creating+activating, 3 creating+peering, 13 unknown; 578 KiB data, 221 MiB used, 715 GiB / 715 GiB avail; 4.2 KiB/s rd, 2.2 KiB/s wr, 7 op/s 2024-03-11T21:25:44.792 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:25:44 smithi097 ceph-mon[33317]: osdmap e72: 8 total, 8 up, 8 in 2024-03-11T21:25:44.792 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:25:44 smithi097 ceph-mon[33317]: from='client.? v1:172.21.15.92:0/2960078881' entity='mgr.y' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.meta","app": "rgw"}]: dispatch 2024-03-11T21:25:44.792 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:25:44 smithi097 ceph-mon[33317]: from='client.? v1:172.21.15.92:0/372197479' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.meta","app": "rgw"}]: dispatch 2024-03-11T21:25:44.792 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:25:44 smithi097 ceph-mon[33317]: from='client.34352 ' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.meta","app": "rgw"}]: dispatch 2024-03-11T21:25:44.861 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:25:44 smithi092 ceph-mon[29572]: pgmap v198: 100 pgs: 82 active+clean, 2 creating+activating, 3 creating+peering, 13 unknown; 578 KiB data, 221 MiB used, 715 GiB / 715 GiB avail; 4.2 KiB/s rd, 2.2 KiB/s wr, 7 op/s 2024-03-11T21:25:44.862 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:25:44 smithi092 ceph-mon[29572]: osdmap e72: 8 total, 8 up, 8 in 2024-03-11T21:25:44.862 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:25:44 smithi092 ceph-mon[29572]: from='client.? v1:172.21.15.92:0/2960078881' entity='mgr.y' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.meta","app": "rgw"}]: dispatch 2024-03-11T21:25:44.862 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:25:44 smithi092 ceph-mon[29572]: from='client.? v1:172.21.15.92:0/372197479' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.meta","app": "rgw"}]: dispatch 2024-03-11T21:25:44.862 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:25:44 smithi092 ceph-mon[29572]: from='client.34352 ' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.meta","app": "rgw"}]: dispatch 2024-03-11T21:25:44.862 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:25:44 smithi092 ceph-mon[37563]: pgmap v198: 100 pgs: 82 active+clean, 2 creating+activating, 3 creating+peering, 13 unknown; 578 KiB data, 221 MiB used, 715 GiB / 715 GiB avail; 4.2 KiB/s rd, 2.2 KiB/s wr, 7 op/s 2024-03-11T21:25:44.862 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:25:44 smithi092 ceph-mon[37563]: osdmap e72: 8 total, 8 up, 8 in 2024-03-11T21:25:44.862 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:25:44 smithi092 ceph-mon[37563]: from='client.? v1:172.21.15.92:0/2960078881' entity='mgr.y' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.meta","app": "rgw"}]: dispatch 2024-03-11T21:25:44.862 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:25:44 smithi092 ceph-mon[37563]: from='client.? v1:172.21.15.92:0/372197479' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.meta","app": "rgw"}]: dispatch 2024-03-11T21:25:44.862 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:25:44 smithi092 ceph-mon[37563]: from='client.34352 ' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.meta","app": "rgw"}]: dispatch 2024-03-11T21:25:45.400 INFO:teuthology.orchestra.run.smithi097.stdout:Scheduled iscsi.datapool update... 2024-03-11T21:25:45.719 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:25:45 smithi097 ceph-mon[33317]: from='client.? v1:172.21.15.92:0/2960078881' entity='mgr.y' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.meta","app": "rgw"}]': finished 2024-03-11T21:25:45.720 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:25:45 smithi097 ceph-mon[33317]: from='client.34352 ' entity='client.rgw.foo.a' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.meta","app": "rgw"}]': finished 2024-03-11T21:25:45.720 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:25:45 smithi097 ceph-mon[33317]: osdmap e73: 8 total, 8 up, 8 in 2024-03-11T21:25:45.720 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:25:45 smithi097 ceph-mon[33317]: from='client.? v1:172.21.15.92:0/372197479' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]: dispatch 2024-03-11T21:25:45.720 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:25:45 smithi097 ceph-mon[33317]: from='client.34352 ' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]: dispatch 2024-03-11T21:25:45.720 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:25:45 smithi097 ceph-mon[33317]: from='client.? v1:172.21.15.92:0/2960078881' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]: dispatch 2024-03-11T21:25:45.720 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:25:45 smithi097 ceph-mon[33317]: from='mgr.14150 v1:172.21.15.92:0/996057664' entity='mgr.y' 2024-03-11T21:25:45.720 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:25:45 smithi097 ceph-mon[33317]: from='client.34352 ' entity='client.rgw.foo.a' cmd='[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]': finished 2024-03-11T21:25:45.720 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:25:45 smithi097 ceph-mon[33317]: from='client.? v1:172.21.15.92:0/2960078881' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]': finished 2024-03-11T21:25:45.720 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:25:45 smithi097 ceph-mon[33317]: osdmap e74: 8 total, 8 up, 8 in 2024-03-11T21:25:45.862 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:25:45 smithi092 ceph-mon[29572]: from='client.? v1:172.21.15.92:0/2960078881' entity='mgr.y' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.meta","app": "rgw"}]': finished 2024-03-11T21:25:45.862 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:25:45 smithi092 ceph-mon[29572]: from='client.34352 ' entity='client.rgw.foo.a' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.meta","app": "rgw"}]': finished 2024-03-11T21:25:45.862 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:25:45 smithi092 ceph-mon[29572]: osdmap e73: 8 total, 8 up, 8 in 2024-03-11T21:25:45.862 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:25:45 smithi092 ceph-mon[29572]: from='client.? v1:172.21.15.92:0/372197479' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]: dispatch 2024-03-11T21:25:45.862 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:25:45 smithi092 ceph-mon[29572]: from='client.34352 ' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]: dispatch 2024-03-11T21:25:45.862 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:25:45 smithi092 ceph-mon[29572]: from='client.? v1:172.21.15.92:0/2960078881' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]: dispatch 2024-03-11T21:25:45.862 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:25:45 smithi092 ceph-mon[29572]: from='mgr.14150 v1:172.21.15.92:0/996057664' entity='mgr.y' 2024-03-11T21:25:45.862 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:25:45 smithi092 ceph-mon[29572]: from='client.34352 ' entity='client.rgw.foo.a' cmd='[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]': finished 2024-03-11T21:25:45.863 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:25:45 smithi092 ceph-mon[29572]: from='client.? v1:172.21.15.92:0/2960078881' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]': finished 2024-03-11T21:25:45.863 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:25:45 smithi092 ceph-mon[29572]: osdmap e74: 8 total, 8 up, 8 in 2024-03-11T21:25:45.863 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:25:45 smithi092 ceph-mon[37563]: from='client.? v1:172.21.15.92:0/2960078881' entity='mgr.y' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.meta","app": "rgw"}]': finished 2024-03-11T21:25:45.863 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:25:45 smithi092 ceph-mon[37563]: from='client.34352 ' entity='client.rgw.foo.a' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.meta","app": "rgw"}]': finished 2024-03-11T21:25:45.863 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:25:45 smithi092 ceph-mon[37563]: osdmap e73: 8 total, 8 up, 8 in 2024-03-11T21:25:45.863 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:25:45 smithi092 ceph-mon[37563]: from='client.? v1:172.21.15.92:0/372197479' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]: dispatch 2024-03-11T21:25:45.863 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:25:45 smithi092 ceph-mon[37563]: from='client.34352 ' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]: dispatch 2024-03-11T21:25:45.863 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:25:45 smithi092 ceph-mon[37563]: from='client.? v1:172.21.15.92:0/2960078881' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]: dispatch 2024-03-11T21:25:45.863 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:25:45 smithi092 ceph-mon[37563]: from='mgr.14150 v1:172.21.15.92:0/996057664' entity='mgr.y' 2024-03-11T21:25:45.864 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:25:45 smithi092 ceph-mon[37563]: from='client.34352 ' entity='client.rgw.foo.a' cmd='[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]': finished 2024-03-11T21:25:45.864 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:25:45 smithi092 ceph-mon[37563]: from='client.? v1:172.21.15.92:0/2960078881' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]': finished 2024-03-11T21:25:45.864 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:25:45 smithi092 ceph-mon[37563]: osdmap e74: 8 total, 8 up, 8 in 2024-03-11T21:25:46.049 INFO:tasks.cephadm:Distributing iscsi-gateway.cfg... 2024-03-11T21:25:46.049 DEBUG:teuthology.orchestra.run.smithi092:> set -ex 2024-03-11T21:25:46.049 DEBUG:teuthology.orchestra.run.smithi092:> sudo dd of=/etc/ceph/iscsi-gateway.cfg 2024-03-11T21:25:46.079 DEBUG:teuthology.orchestra.run.smithi097:> set -ex 2024-03-11T21:25:46.079 DEBUG:teuthology.orchestra.run.smithi097:> sudo dd of=/etc/ceph/iscsi-gateway.cfg 2024-03-11T21:25:46.109 DEBUG:teuthology.orchestra.run.smithi097:iscsi.iscsi.a> sudo journalctl -f -n 0 -u ceph-00db0016-dfed-11ee-95c8-87774f69a715@iscsi.iscsi.a.service 2024-03-11T21:25:46.151 INFO:tasks.cephadm:Adding prometheus.a on smithi097 2024-03-11T21:25:46.151 DEBUG:teuthology.orchestra.run.smithi097:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 00db0016-dfed-11ee-95c8-87774f69a715 -- ceph orch apply prometheus '1;smithi097=a' 2024-03-11T21:25:46.708 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:25:46 smithi097 ceph-mon[33317]: from='client.24424 v1:172.21.15.97:0/3040030617' entity='client.admin' cmd=[{"prefix": "orch apply iscsi", "pool": "datapool", "api_user": "admin", "api_password": "admin", "trusted_ip_list": "172.21.15.97", "placement": "1;smithi097=iscsi.a", "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T21:25:46.708 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:25:46 smithi097 ceph-mon[33317]: Saving service iscsi.datapool spec with placement smithi097=iscsi.a;count:1 2024-03-11T21:25:46.709 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:25:46 smithi097 ceph-mon[33317]: pgmap v201: 132 pgs: 82 active+clean, 2 creating+activating, 3 creating+peering, 45 unknown; 578 KiB data, 221 MiB used, 715 GiB / 715 GiB avail; 1.7 KiB/s rd, 1.5 KiB/s wr, 3 op/s 2024-03-11T21:25:46.709 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:25:46 smithi097 ceph-mon[33317]: purged_snaps scrub starts 2024-03-11T21:25:46.709 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:25:46 smithi097 ceph-mon[33317]: purged_snaps scrub ok 2024-03-11T21:25:46.709 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:25:46 smithi097 ceph-mon[33317]: from='mgr.14150 v1:172.21.15.92:0/996057664' entity='mgr.y' 2024-03-11T21:25:46.709 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:25:46 smithi097 ceph-mon[33317]: from='mgr.14150 v1:172.21.15.92:0/996057664' entity='mgr.y' 2024-03-11T21:25:46.709 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:25:46 smithi097 ceph-mon[33317]: from='mgr.14150 v1:172.21.15.92:0/996057664' entity='mgr.y' 2024-03-11T21:25:46.709 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:25:46 smithi097 ceph-mon[33317]: from='mgr.14150 v1:172.21.15.92:0/996057664' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.iscsi.a", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-03-11T21:25:46.709 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:25:46 smithi097 ceph-mon[33317]: from='mgr.14150 v1:172.21.15.92:0/996057664' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "client.iscsi.iscsi.a", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]': finished 2024-03-11T21:25:46.709 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:25:46 smithi097 ceph-mon[33317]: from='mgr.14150 v1:172.21.15.92:0/996057664' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T21:25:46.861 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:25:46 smithi092 ceph-mon[29572]: from='client.24424 v1:172.21.15.97:0/3040030617' entity='client.admin' cmd=[{"prefix": "orch apply iscsi", "pool": "datapool", "api_user": "admin", "api_password": "admin", "trusted_ip_list": "172.21.15.97", "placement": "1;smithi097=iscsi.a", "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T21:25:46.862 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:25:46 smithi092 ceph-mon[29572]: Saving service iscsi.datapool spec with placement smithi097=iscsi.a;count:1 2024-03-11T21:25:46.862 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:25:46 smithi092 ceph-mon[29572]: pgmap v201: 132 pgs: 82 active+clean, 2 creating+activating, 3 creating+peering, 45 unknown; 578 KiB data, 221 MiB used, 715 GiB / 715 GiB avail; 1.7 KiB/s rd, 1.5 KiB/s wr, 3 op/s 2024-03-11T21:25:46.862 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:25:46 smithi092 ceph-mon[29572]: purged_snaps scrub starts 2024-03-11T21:25:46.862 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:25:46 smithi092 ceph-mon[29572]: purged_snaps scrub ok 2024-03-11T21:25:46.862 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:25:46 smithi092 ceph-mon[29572]: from='mgr.14150 v1:172.21.15.92:0/996057664' entity='mgr.y' 2024-03-11T21:25:46.862 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:25:46 smithi092 ceph-mon[29572]: from='mgr.14150 v1:172.21.15.92:0/996057664' entity='mgr.y' 2024-03-11T21:25:46.862 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:25:46 smithi092 ceph-mon[29572]: from='mgr.14150 v1:172.21.15.92:0/996057664' entity='mgr.y' 2024-03-11T21:25:46.862 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:25:46 smithi092 ceph-mon[29572]: from='mgr.14150 v1:172.21.15.92:0/996057664' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.iscsi.a", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-03-11T21:25:46.862 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:25:46 smithi092 ceph-mon[29572]: from='mgr.14150 v1:172.21.15.92:0/996057664' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "client.iscsi.iscsi.a", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]': finished 2024-03-11T21:25:46.862 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:25:46 smithi092 ceph-mon[29572]: from='mgr.14150 v1:172.21.15.92:0/996057664' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T21:25:46.863 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:25:46 smithi092 ceph-mon[37563]: from='client.24424 v1:172.21.15.97:0/3040030617' entity='client.admin' cmd=[{"prefix": "orch apply iscsi", "pool": "datapool", "api_user": "admin", "api_password": "admin", "trusted_ip_list": "172.21.15.97", "placement": "1;smithi097=iscsi.a", "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T21:25:46.863 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:25:46 smithi092 ceph-mon[37563]: Saving service iscsi.datapool spec with placement smithi097=iscsi.a;count:1 2024-03-11T21:25:46.863 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:25:46 smithi092 ceph-mon[37563]: pgmap v201: 132 pgs: 82 active+clean, 2 creating+activating, 3 creating+peering, 45 unknown; 578 KiB data, 221 MiB used, 715 GiB / 715 GiB avail; 1.7 KiB/s rd, 1.5 KiB/s wr, 3 op/s 2024-03-11T21:25:46.863 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:25:46 smithi092 ceph-mon[37563]: purged_snaps scrub starts 2024-03-11T21:25:46.863 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:25:46 smithi092 ceph-mon[37563]: purged_snaps scrub ok 2024-03-11T21:25:46.863 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:25:46 smithi092 ceph-mon[37563]: from='mgr.14150 v1:172.21.15.92:0/996057664' entity='mgr.y' 2024-03-11T21:25:46.863 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:25:46 smithi092 ceph-mon[37563]: from='mgr.14150 v1:172.21.15.92:0/996057664' entity='mgr.y' 2024-03-11T21:25:46.863 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:25:46 smithi092 ceph-mon[37563]: from='mgr.14150 v1:172.21.15.92:0/996057664' entity='mgr.y' 2024-03-11T21:25:46.863 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:25:46 smithi092 ceph-mon[37563]: from='mgr.14150 v1:172.21.15.92:0/996057664' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.iscsi.a", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-03-11T21:25:46.863 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:25:46 smithi092 ceph-mon[37563]: from='mgr.14150 v1:172.21.15.92:0/996057664' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "client.iscsi.iscsi.a", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]': finished 2024-03-11T21:25:46.863 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:25:46 smithi092 ceph-mon[37563]: from='mgr.14150 v1:172.21.15.92:0/996057664' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T21:25:47.338 INFO:teuthology.orchestra.run.smithi097.stderr:Inferring config /var/lib/ceph/00db0016-dfed-11ee-95c8-87774f69a715/mon.b/config 2024-03-11T21:25:47.814 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:25:47 smithi097 ceph-mon[33317]: Deploying daemon iscsi.iscsi.a on smithi097 2024-03-11T21:25:47.814 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:25:47 smithi097 ceph-mon[33317]: osdmap e75: 8 total, 8 up, 8 in 2024-03-11T21:25:47.861 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:25:47 smithi092 ceph-mon[37563]: Deploying daemon iscsi.iscsi.a on smithi097 2024-03-11T21:25:47.861 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:25:47 smithi092 ceph-mon[37563]: osdmap e75: 8 total, 8 up, 8 in 2024-03-11T21:25:47.862 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:25:47 smithi092 ceph-mon[29572]: Deploying daemon iscsi.iscsi.a on smithi097 2024-03-11T21:25:47.862 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:25:47 smithi092 ceph-mon[29572]: osdmap e75: 8 total, 8 up, 8 in 2024-03-11T21:25:48.077 INFO:journalctl@ceph.iscsi.iscsi.a.smithi097.stdout:Mar 11 21:25:47 smithi097 systemd[1]: Starting Ceph iscsi.iscsi.a for 00db0016-dfed-11ee-95c8-87774f69a715... 2024-03-11T21:25:48.669 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:25:48 smithi097 ceph-mon[33317]: pgmap v204: 132 pgs: 115 active+clean, 17 unknown; 579 KiB data, 225 MiB used, 715 GiB / 715 GiB avail; 12 KiB/s rd, 1.5 KiB/s wr, 32 op/s 2024-03-11T21:25:48.669 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:25:48 smithi097 ceph-mon[33317]: osdmap e76: 8 total, 8 up, 8 in 2024-03-11T21:25:48.861 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:25:48 smithi092 ceph-mon[37563]: pgmap v204: 132 pgs: 115 active+clean, 17 unknown; 579 KiB data, 225 MiB used, 715 GiB / 715 GiB avail; 12 KiB/s rd, 1.5 KiB/s wr, 32 op/s 2024-03-11T21:25:48.861 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:25:48 smithi092 ceph-mon[37563]: osdmap e76: 8 total, 8 up, 8 in 2024-03-11T21:25:48.862 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:25:48 smithi092 ceph-mon[29572]: pgmap v204: 132 pgs: 115 active+clean, 17 unknown; 579 KiB data, 225 MiB used, 715 GiB / 715 GiB avail; 12 KiB/s rd, 1.5 KiB/s wr, 32 op/s 2024-03-11T21:25:48.862 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:25:48 smithi092 ceph-mon[29572]: osdmap e76: 8 total, 8 up, 8 in 2024-03-11T21:25:48.998 INFO:journalctl@ceph.iscsi.iscsi.a.smithi097.stdout:Mar 11 21:25:48 smithi097 podman[60899]: 2024-03-11 21:25:48.697035726 +0000 UTC m=+0.250238086 image pull 4133f8bc41bb201212a28168427a2edb1f6f3795e694bf61031ea9e2f3ff5e87 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:8fe9635f3beb0247496d7e321f3068bd10c333c433ef4824d7e1c1a7d3d78379 2024-03-11T21:25:49.111 INFO:teuthology.orchestra.run.smithi097.stdout:Scheduled prometheus update... 2024-03-11T21:25:49.284 INFO:journalctl@ceph.iscsi.iscsi.a.smithi097.stdout:Mar 11 21:25:49 smithi097 podman[60899]: 2024-03-11 21:25:49.015552919 +0000 UTC m=+0.568755261 container create 8bd740122036d3a69d3780a0f0a02517a39b42ae3bde406eefdaf62d1aec85be (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:8fe9635f3beb0247496d7e321f3068bd10c333c433ef4824d7e1c1a7d3d78379, name=ceph-00db0016-dfed-11ee-95c8-87774f69a715-iscsi-iscsi-a, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, RELEASE=main-2b0101e, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , ceph=True, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, GIT_CLEAN=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.build-date=20240307) 2024-03-11T21:25:49.284 INFO:journalctl@ceph.iscsi.iscsi.a.smithi097.stdout:Mar 11 21:25:49 smithi097 podman[60899]: 2024-03-11 21:25:49.209713421 +0000 UTC m=+0.762915764 container init 8bd740122036d3a69d3780a0f0a02517a39b42ae3bde406eefdaf62d1aec85be (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:8fe9635f3beb0247496d7e321f3068bd10c333c433ef4824d7e1c1a7d3d78379, name=ceph-00db0016-dfed-11ee-95c8-87774f69a715-iscsi-iscsi-a, ceph=True, org.label-schema.build-date=20240307, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, RELEASE=main-2b0101e, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD) 2024-03-11T21:25:49.284 INFO:journalctl@ceph.iscsi.iscsi.a.smithi097.stdout:Mar 11 21:25:49 smithi097 podman[60899]: 2024-03-11 21:25:49.213222612 +0000 UTC m=+0.766424956 container start 8bd740122036d3a69d3780a0f0a02517a39b42ae3bde406eefdaf62d1aec85be (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:8fe9635f3beb0247496d7e321f3068bd10c333c433ef4824d7e1c1a7d3d78379, name=ceph-00db0016-dfed-11ee-95c8-87774f69a715-iscsi-iscsi-a, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=main-2b0101e, GIT_BRANCH=HEAD, org.label-schema.build-date=20240307, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.vendor=CentOS, ceph=True, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True) 2024-03-11T21:25:49.577 INFO:journalctl@ceph.iscsi.iscsi.a.smithi097.stdout:Mar 11 21:25:49 smithi097 bash[60899]: 8bd740122036d3a69d3780a0f0a02517a39b42ae3bde406eefdaf62d1aec85be 2024-03-11T21:25:49.577 INFO:journalctl@ceph.iscsi.iscsi.a.smithi097.stdout:Mar 11 21:25:49 smithi097 systemd[1]: Started Ceph iscsi.iscsi.a for 00db0016-dfed-11ee-95c8-87774f69a715. 2024-03-11T21:25:49.577 INFO:journalctl@ceph.iscsi.iscsi.a.smithi097.stdout:Mar 11 21:25:49 smithi097 ceph-00db0016-dfed-11ee-95c8-87774f69a715-iscsi-iscsi-a[60943]: debug Started the configuration object watcher 2024-03-11T21:25:49.577 INFO:journalctl@ceph.iscsi.iscsi.a.smithi097.stdout:Mar 11 21:25:49 smithi097 ceph-00db0016-dfed-11ee-95c8-87774f69a715-iscsi-iscsi-a[60943]: debug Checking for config object changes every 1s 2024-03-11T21:25:49.577 INFO:journalctl@ceph.iscsi.iscsi.a.smithi097.stdout:Mar 11 21:25:49 smithi097 ceph-00db0016-dfed-11ee-95c8-87774f69a715-iscsi-iscsi-a[60943]: debug Processing osd blocklist entries for this node 2024-03-11T21:25:49.880 INFO:journalctl@ceph.iscsi.iscsi.a.smithi097.stdout:Mar 11 21:25:49 smithi097 ceph-00db0016-dfed-11ee-95c8-87774f69a715-iscsi-iscsi-a[60943]: debug Reading the configuration object to update local LIO configuration 2024-03-11T21:25:49.881 INFO:journalctl@ceph.iscsi.iscsi.a.smithi097.stdout:Mar 11 21:25:49 smithi097 ceph-00db0016-dfed-11ee-95c8-87774f69a715-iscsi-iscsi-a[60943]: debug Configuration does not have an entry for this host(smithi097.front.sepia.ceph.com) - nothing to define to LIO 2024-03-11T21:25:49.881 INFO:journalctl@ceph.iscsi.iscsi.a.smithi097.stdout:Mar 11 21:25:49 smithi097 ceph-00db0016-dfed-11ee-95c8-87774f69a715-iscsi-iscsi-a[60943]: debug * Running on http://[::]:5000/ (Press CTRL+C to quit) 2024-03-11T21:25:49.908 DEBUG:teuthology.orchestra.run.smithi097:prometheus.a> sudo journalctl -f -n 0 -u ceph-00db0016-dfed-11ee-95c8-87774f69a715@prometheus.a.service 2024-03-11T21:25:49.910 INFO:tasks.cephadm:Adding node-exporter.a on smithi092 2024-03-11T21:25:49.910 INFO:tasks.cephadm:Adding node-exporter.b on smithi097 2024-03-11T21:25:49.910 DEBUG:teuthology.orchestra.run.smithi097:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 00db0016-dfed-11ee-95c8-87774f69a715 -- ceph orch apply node-exporter '2;smithi092=a;smithi097=b' 2024-03-11T21:25:50.259 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:25:50 smithi097 ceph-mon[33317]: from='client.34415 v1:172.21.15.97:0/2914000750' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "prometheus", "placement": "1;smithi097=a", "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T21:25:50.259 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:25:50 smithi097 ceph-mon[33317]: Saving service prometheus spec with placement smithi097=a;count:1 2024-03-11T21:25:50.259 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:25:50 smithi097 ceph-mon[33317]: from='mgr.14150 v1:172.21.15.92:0/996057664' entity='mgr.y' 2024-03-11T21:25:50.259 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:25:50 smithi097 ceph-mon[33317]: from='mgr.14150 v1:172.21.15.92:0/996057664' entity='mgr.y' 2024-03-11T21:25:50.259 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:25:50 smithi097 ceph-mon[33317]: from='mgr.14150 v1:172.21.15.92:0/996057664' entity='mgr.y' 2024-03-11T21:25:50.259 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:25:50 smithi097 ceph-mon[33317]: from='mgr.14150 v1:172.21.15.92:0/996057664' entity='mgr.y' 2024-03-11T21:25:50.259 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:25:50 smithi097 ceph-mon[33317]: Checking pool "datapool" exists for service iscsi.datapool 2024-03-11T21:25:50.259 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:25:50 smithi097 ceph-mon[33317]: from='mgr.14150 v1:172.21.15.92:0/996057664' entity='mgr.y' 2024-03-11T21:25:50.260 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:25:50 smithi097 ceph-mon[33317]: pgmap v206: 132 pgs: 132 active+clean; 581 KiB data, 226 MiB used, 715 GiB / 715 GiB avail; 63 KiB/s rd, 5.2 KiB/s wr, 156 op/s 2024-03-11T21:25:50.260 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:25:50 smithi097 ceph-mon[33317]: from='mgr.14150 v1:172.21.15.92:0/996057664' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T21:25:50.260 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:25:50 smithi097 ceph-mon[33317]: from='mgr.14150 v1:172.21.15.92:0/996057664' entity='mgr.y' 2024-03-11T21:25:50.260 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:25:50 smithi097 ceph-mon[33317]: from='client.? v1:172.21.15.97:0/2771533369' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-03-11T21:25:50.361 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:25:50 smithi092 ceph-mon[29572]: from='client.34415 v1:172.21.15.97:0/2914000750' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "prometheus", "placement": "1;smithi097=a", "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T21:25:50.362 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:25:50 smithi092 ceph-mon[29572]: Saving service prometheus spec with placement smithi097=a;count:1 2024-03-11T21:25:50.362 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:25:50 smithi092 ceph-mon[29572]: from='mgr.14150 v1:172.21.15.92:0/996057664' entity='mgr.y' 2024-03-11T21:25:50.362 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:25:50 smithi092 ceph-mon[29572]: from='mgr.14150 v1:172.21.15.92:0/996057664' entity='mgr.y' 2024-03-11T21:25:50.362 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:25:50 smithi092 ceph-mon[29572]: from='mgr.14150 v1:172.21.15.92:0/996057664' entity='mgr.y' 2024-03-11T21:25:50.362 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:25:50 smithi092 ceph-mon[29572]: from='mgr.14150 v1:172.21.15.92:0/996057664' entity='mgr.y' 2024-03-11T21:25:50.362 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:25:50 smithi092 ceph-mon[29572]: Checking pool "datapool" exists for service iscsi.datapool 2024-03-11T21:25:50.362 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:25:50 smithi092 ceph-mon[29572]: from='mgr.14150 v1:172.21.15.92:0/996057664' entity='mgr.y' 2024-03-11T21:25:50.362 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:25:50 smithi092 ceph-mon[29572]: pgmap v206: 132 pgs: 132 active+clean; 581 KiB data, 226 MiB used, 715 GiB / 715 GiB avail; 63 KiB/s rd, 5.2 KiB/s wr, 156 op/s 2024-03-11T21:25:50.362 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:25:50 smithi092 ceph-mon[29572]: from='mgr.14150 v1:172.21.15.92:0/996057664' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T21:25:50.362 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:25:50 smithi092 ceph-mon[29572]: from='mgr.14150 v1:172.21.15.92:0/996057664' entity='mgr.y' 2024-03-11T21:25:50.362 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:25:50 smithi092 ceph-mon[29572]: from='client.? v1:172.21.15.97:0/2771533369' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-03-11T21:25:50.363 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:25:50 smithi092 ceph-mon[37563]: from='client.34415 v1:172.21.15.97:0/2914000750' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "prometheus", "placement": "1;smithi097=a", "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T21:25:50.363 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:25:50 smithi092 ceph-mon[37563]: Saving service prometheus spec with placement smithi097=a;count:1 2024-03-11T21:25:50.363 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:25:50 smithi092 ceph-mon[37563]: from='mgr.14150 v1:172.21.15.92:0/996057664' entity='mgr.y' 2024-03-11T21:25:50.363 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:25:50 smithi092 ceph-mon[37563]: from='mgr.14150 v1:172.21.15.92:0/996057664' entity='mgr.y' 2024-03-11T21:25:50.363 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:25:50 smithi092 ceph-mon[37563]: from='mgr.14150 v1:172.21.15.92:0/996057664' entity='mgr.y' 2024-03-11T21:25:50.363 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:25:50 smithi092 ceph-mon[37563]: from='mgr.14150 v1:172.21.15.92:0/996057664' entity='mgr.y' 2024-03-11T21:25:50.363 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:25:50 smithi092 ceph-mon[37563]: Checking pool "datapool" exists for service iscsi.datapool 2024-03-11T21:25:50.363 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:25:50 smithi092 ceph-mon[37563]: from='mgr.14150 v1:172.21.15.92:0/996057664' entity='mgr.y' 2024-03-11T21:25:50.363 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:25:50 smithi092 ceph-mon[37563]: pgmap v206: 132 pgs: 132 active+clean; 581 KiB data, 226 MiB used, 715 GiB / 715 GiB avail; 63 KiB/s rd, 5.2 KiB/s wr, 156 op/s 2024-03-11T21:25:50.363 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:25:50 smithi092 ceph-mon[37563]: from='mgr.14150 v1:172.21.15.92:0/996057664' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T21:25:50.363 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:25:50 smithi092 ceph-mon[37563]: from='mgr.14150 v1:172.21.15.92:0/996057664' entity='mgr.y' 2024-03-11T21:25:50.363 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:25:50 smithi092 ceph-mon[37563]: from='client.? v1:172.21.15.97:0/2771533369' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-03-11T21:25:50.551 INFO:teuthology.orchestra.run.smithi097.stderr:Inferring config /var/lib/ceph/00db0016-dfed-11ee-95c8-87774f69a715/mon.b/config 2024-03-11T21:25:52.827 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:25:52 smithi097 ceph-mon[33317]: mgrmap e16: y(active, since 4m), standbys: x 2024-03-11T21:25:52.828 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:25:52 smithi097 ceph-mon[33317]: pgmap v207: 132 pgs: 132 active+clean; 582 KiB data, 226 MiB used, 715 GiB / 715 GiB avail; 77 KiB/s rd, 5.8 KiB/s wr, 187 op/s 2024-03-11T21:25:52.861 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:25:52 smithi092 ceph-mon[29572]: mgrmap e16: y(active, since 4m), standbys: x 2024-03-11T21:25:52.861 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:25:52 smithi092 ceph-mon[29572]: pgmap v207: 132 pgs: 132 active+clean; 582 KiB data, 226 MiB used, 715 GiB / 715 GiB avail; 77 KiB/s rd, 5.8 KiB/s wr, 187 op/s 2024-03-11T21:25:52.862 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:25:52 smithi092 ceph-mon[37563]: mgrmap e16: y(active, since 4m), standbys: x 2024-03-11T21:25:52.862 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:25:52 smithi092 ceph-mon[37563]: pgmap v207: 132 pgs: 132 active+clean; 582 KiB data, 226 MiB used, 715 GiB / 715 GiB avail; 77 KiB/s rd, 5.8 KiB/s wr, 187 op/s 2024-03-11T21:25:52.935 INFO:teuthology.orchestra.run.smithi097.stdout:Scheduled node-exporter update... 2024-03-11T21:25:54.070 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:25:53 smithi097 ceph-mon[33317]: from='client.24481 v1:172.21.15.97:0/1534751627' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "node-exporter", "placement": "2;smithi092=a;smithi097=b", "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T21:25:54.070 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:25:53 smithi097 ceph-mon[33317]: Saving service node-exporter spec with placement smithi092=a;smithi097=b;count:2 2024-03-11T21:25:54.070 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:25:53 smithi097 ceph-mon[33317]: from='mgr.14150 v1:172.21.15.92:0/996057664' entity='mgr.y' 2024-03-11T21:25:54.070 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:25:53 smithi097 ceph-mon[33317]: osdmap e77: 8 total, 8 up, 8 in 2024-03-11T21:25:54.070 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:25:53 smithi097 ceph-mon[33317]: pgmap v209: 132 pgs: 132 active+clean; 582 KiB data, 226 MiB used, 715 GiB / 715 GiB avail; 67 KiB/s rd, 4.8 KiB/s wr, 162 op/s 2024-03-11T21:25:54.070 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:25:53 smithi097 ceph-mon[33317]: from='mgr.14150 v1:172.21.15.92:0/996057664' entity='mgr.y' 2024-03-11T21:25:54.070 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:25:53 smithi097 ceph-mon[33317]: from='mgr.14150 v1:172.21.15.92:0/996057664' entity='mgr.y' 2024-03-11T21:25:54.070 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:25:53 smithi097 ceph-mon[33317]: from='mgr.14150 v1:172.21.15.92:0/996057664' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T21:25:54.070 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:25:53 smithi097 ceph-mon[33317]: from='mgr.14150 v1:172.21.15.92:0/996057664' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T21:25:54.070 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:25:53 smithi097 ceph-mon[33317]: from='mgr.14150 v1:172.21.15.92:0/996057664' entity='mgr.y' 2024-03-11T21:25:54.098 DEBUG:teuthology.orchestra.run.smithi092:node-exporter.a> sudo journalctl -f -n 0 -u ceph-00db0016-dfed-11ee-95c8-87774f69a715@node-exporter.a.service 2024-03-11T21:25:54.100 DEBUG:teuthology.orchestra.run.smithi097:node-exporter.b> sudo journalctl -f -n 0 -u ceph-00db0016-dfed-11ee-95c8-87774f69a715@node-exporter.b.service 2024-03-11T21:25:54.103 INFO:tasks.cephadm:Adding alertmanager.a on smithi092 2024-03-11T21:25:54.103 DEBUG:teuthology.orchestra.run.smithi097:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 00db0016-dfed-11ee-95c8-87774f69a715 -- ceph orch apply alertmanager '1;smithi092=a' 2024-03-11T21:25:54.361 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:25:53 smithi092 ceph-mon[29572]: from='client.24481 v1:172.21.15.97:0/1534751627' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "node-exporter", "placement": "2;smithi092=a;smithi097=b", "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T21:25:54.362 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:25:53 smithi092 ceph-mon[29572]: Saving service node-exporter spec with placement smithi092=a;smithi097=b;count:2 2024-03-11T21:25:54.362 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:25:53 smithi092 ceph-mon[29572]: from='mgr.14150 v1:172.21.15.92:0/996057664' entity='mgr.y' 2024-03-11T21:25:54.362 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:25:53 smithi092 ceph-mon[29572]: osdmap e77: 8 total, 8 up, 8 in 2024-03-11T21:25:54.362 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:25:53 smithi092 ceph-mon[29572]: pgmap v209: 132 pgs: 132 active+clean; 582 KiB data, 226 MiB used, 715 GiB / 715 GiB avail; 67 KiB/s rd, 4.8 KiB/s wr, 162 op/s 2024-03-11T21:25:54.362 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:25:53 smithi092 ceph-mon[29572]: from='mgr.14150 v1:172.21.15.92:0/996057664' entity='mgr.y' 2024-03-11T21:25:54.362 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:25:53 smithi092 ceph-mon[29572]: from='mgr.14150 v1:172.21.15.92:0/996057664' entity='mgr.y' 2024-03-11T21:25:54.362 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:25:53 smithi092 ceph-mon[29572]: from='mgr.14150 v1:172.21.15.92:0/996057664' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T21:25:54.362 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:25:53 smithi092 ceph-mon[29572]: from='mgr.14150 v1:172.21.15.92:0/996057664' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T21:25:54.362 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:25:53 smithi092 ceph-mon[29572]: from='mgr.14150 v1:172.21.15.92:0/996057664' entity='mgr.y' 2024-03-11T21:25:54.362 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:25:53 smithi092 ceph-mon[37563]: from='client.24481 v1:172.21.15.97:0/1534751627' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "node-exporter", "placement": "2;smithi092=a;smithi097=b", "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T21:25:54.363 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:25:53 smithi092 ceph-mon[37563]: Saving service node-exporter spec with placement smithi092=a;smithi097=b;count:2 2024-03-11T21:25:54.363 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:25:53 smithi092 ceph-mon[37563]: from='mgr.14150 v1:172.21.15.92:0/996057664' entity='mgr.y' 2024-03-11T21:25:54.363 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:25:53 smithi092 ceph-mon[37563]: osdmap e77: 8 total, 8 up, 8 in 2024-03-11T21:25:54.363 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:25:53 smithi092 ceph-mon[37563]: pgmap v209: 132 pgs: 132 active+clean; 582 KiB data, 226 MiB used, 715 GiB / 715 GiB avail; 67 KiB/s rd, 4.8 KiB/s wr, 162 op/s 2024-03-11T21:25:54.363 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:25:53 smithi092 ceph-mon[37563]: from='mgr.14150 v1:172.21.15.92:0/996057664' entity='mgr.y' 2024-03-11T21:25:54.363 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:25:53 smithi092 ceph-mon[37563]: from='mgr.14150 v1:172.21.15.92:0/996057664' entity='mgr.y' 2024-03-11T21:25:54.363 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:25:53 smithi092 ceph-mon[37563]: from='mgr.14150 v1:172.21.15.92:0/996057664' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T21:25:54.363 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:25:53 smithi092 ceph-mon[37563]: from='mgr.14150 v1:172.21.15.92:0/996057664' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T21:25:54.363 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:25:53 smithi092 ceph-mon[37563]: from='mgr.14150 v1:172.21.15.92:0/996057664' entity='mgr.y' 2024-03-11T21:25:54.406 INFO:teuthology.orchestra.run.smithi097.stderr:Inferring config /var/lib/ceph/00db0016-dfed-11ee-95c8-87774f69a715/mon.b/config 2024-03-11T21:25:55.184 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:25:55 smithi097 ceph-mon[33317]: Deploying daemon prometheus.a on smithi097 2024-03-11T21:25:55.361 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:25:55 smithi092 ceph-mon[29572]: Deploying daemon prometheus.a on smithi097 2024-03-11T21:25:55.362 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:25:55 smithi092 ceph-mon[37563]: Deploying daemon prometheus.a on smithi097 2024-03-11T21:25:56.179 INFO:teuthology.orchestra.run.smithi097.stdout:Scheduled alertmanager update... 2024-03-11T21:25:56.361 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:25:56 smithi092 ceph-mon[29572]: pgmap v210: 132 pgs: 132 active+clean; 582 KiB data, 226 MiB used, 715 GiB / 715 GiB avail; 52 KiB/s rd, 3.7 KiB/s wr, 125 op/s 2024-03-11T21:25:56.362 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:25:56 smithi092 ceph-mon[37563]: pgmap v210: 132 pgs: 132 active+clean; 582 KiB data, 226 MiB used, 715 GiB / 715 GiB avail; 52 KiB/s rd, 3.7 KiB/s wr, 125 op/s 2024-03-11T21:25:56.482 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:25:56 smithi097 ceph-mon[33317]: pgmap v210: 132 pgs: 132 active+clean; 582 KiB data, 226 MiB used, 715 GiB / 715 GiB avail; 52 KiB/s rd, 3.7 KiB/s wr, 125 op/s 2024-03-11T21:25:56.861 DEBUG:teuthology.orchestra.run.smithi092:alertmanager.a> sudo journalctl -f -n 0 -u ceph-00db0016-dfed-11ee-95c8-87774f69a715@alertmanager.a.service 2024-03-11T21:25:56.864 INFO:tasks.cephadm:Adding grafana.a on smithi097 2024-03-11T21:25:56.864 DEBUG:teuthology.orchestra.run.smithi097:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 00db0016-dfed-11ee-95c8-87774f69a715 -- ceph orch apply grafana '1;smithi097=a' 2024-03-11T21:25:57.103 INFO:teuthology.orchestra.run.smithi097.stderr:Inferring config /var/lib/ceph/00db0016-dfed-11ee-95c8-87774f69a715/mon.b/config 2024-03-11T21:25:57.441 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:25:57 smithi097 ceph-mon[33317]: from='client.24487 v1:172.21.15.97:0/2843941174' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "alertmanager", "placement": "1;smithi092=a", "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T21:25:57.450 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:25:57 smithi097 ceph-mon[33317]: Saving service alertmanager spec with placement smithi092=a;count:1 2024-03-11T21:25:57.450 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:25:57 smithi097 ceph-mon[33317]: from='mgr.14150 v1:172.21.15.92:0/996057664' entity='mgr.y' 2024-03-11T21:25:57.611 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:25:57 smithi092 ceph-mon[29572]: from='client.24487 v1:172.21.15.97:0/2843941174' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "alertmanager", "placement": "1;smithi092=a", "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T21:25:57.612 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:25:57 smithi092 ceph-mon[29572]: Saving service alertmanager spec with placement smithi092=a;count:1 2024-03-11T21:25:57.612 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:25:57 smithi092 ceph-mon[29572]: from='mgr.14150 v1:172.21.15.92:0/996057664' entity='mgr.y' 2024-03-11T21:25:57.612 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:25:57 smithi092 ceph-mon[37563]: from='client.24487 v1:172.21.15.97:0/2843941174' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "alertmanager", "placement": "1;smithi092=a", "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T21:25:57.612 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:25:57 smithi092 ceph-mon[37563]: Saving service alertmanager spec with placement smithi092=a;count:1 2024-03-11T21:25:57.612 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:25:57 smithi092 ceph-mon[37563]: from='mgr.14150 v1:172.21.15.92:0/996057664' entity='mgr.y' 2024-03-11T21:25:58.577 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:25:58 smithi097 ceph-mon[33317]: pgmap v211: 132 pgs: 132 active+clean; 582 KiB data, 226 MiB used, 715 GiB / 715 GiB avail; 18 KiB/s rd, 1.2 KiB/s wr, 42 op/s 2024-03-11T21:25:58.577 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:25:58 smithi097 ceph-mon[33317]: osdmap e78: 8 total, 8 up, 8 in 2024-03-11T21:25:58.611 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:25:58 smithi092 ceph-mon[29572]: pgmap v211: 132 pgs: 132 active+clean; 582 KiB data, 226 MiB used, 715 GiB / 715 GiB avail; 18 KiB/s rd, 1.2 KiB/s wr, 42 op/s 2024-03-11T21:25:58.611 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:25:58 smithi092 ceph-mon[29572]: osdmap e78: 8 total, 8 up, 8 in 2024-03-11T21:25:58.611 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:25:58 smithi092 ceph-mon[37563]: pgmap v211: 132 pgs: 132 active+clean; 582 KiB data, 226 MiB used, 715 GiB / 715 GiB avail; 18 KiB/s rd, 1.2 KiB/s wr, 42 op/s 2024-03-11T21:25:58.611 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:25:58 smithi092 ceph-mon[37563]: osdmap e78: 8 total, 8 up, 8 in 2024-03-11T21:25:59.262 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:25:59 smithi097 ceph-mon[33317]: purged_snaps scrub starts 2024-03-11T21:25:59.263 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:25:59 smithi097 ceph-mon[33317]: purged_snaps scrub ok 2024-03-11T21:25:59.577 INFO:journalctl@ceph.iscsi.iscsi.a.smithi097.stdout:Mar 11 21:25:59 smithi097 ceph-00db0016-dfed-11ee-95c8-87774f69a715-iscsi-iscsi-a[60943]: debug there is no tcmu-runner data available 2024-03-11T21:25:59.611 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:25:59 smithi092 ceph-mon[29572]: purged_snaps scrub starts 2024-03-11T21:25:59.611 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:25:59 smithi092 ceph-mon[29572]: purged_snaps scrub ok 2024-03-11T21:25:59.611 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:25:59 smithi092 ceph-mon[37563]: purged_snaps scrub starts 2024-03-11T21:25:59.611 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:25:59 smithi092 ceph-mon[37563]: purged_snaps scrub ok 2024-03-11T21:25:59.624 INFO:teuthology.orchestra.run.smithi097.stdout:Scheduled grafana update... 2024-03-11T21:26:00.538 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:26:00 smithi097 ceph-mon[33317]: osdmap e79: 8 total, 8 up, 8 in 2024-03-11T21:26:00.538 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:26:00 smithi097 ceph-mon[33317]: from='client.24469 v1:172.21.15.97:0/404208310' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:26:00.538 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:26:00 smithi097 ceph-mon[33317]: pgmap v214: 132 pgs: 132 active+clean; 582 KiB data, 226 MiB used, 715 GiB / 715 GiB avail; 797 B/s rd, 0 op/s 2024-03-11T21:26:00.539 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:26:00 smithi097 ceph-mon[33317]: from='client.24493 v1:172.21.15.97:0/2440232012' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "grafana", "placement": "1;smithi097=a", "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T21:26:00.539 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:26:00 smithi097 ceph-mon[33317]: Saving service grafana spec with placement smithi097=a;count:1 2024-03-11T21:26:00.539 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:26:00 smithi097 ceph-mon[33317]: from='mgr.14150 v1:172.21.15.92:0/996057664' entity='mgr.y' 2024-03-11T21:26:00.564 DEBUG:teuthology.orchestra.run.smithi097:grafana.a> sudo journalctl -f -n 0 -u ceph-00db0016-dfed-11ee-95c8-87774f69a715@grafana.a.service 2024-03-11T21:26:00.566 INFO:tasks.cephadm:Setting up client nodes... 2024-03-11T21:26:00.566 DEBUG:teuthology.orchestra.run.smithi092:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 00db0016-dfed-11ee-95c8-87774f69a715 -- ceph auth get-or-create client.0 mon 'allow *' osd 'allow *' mds 'allow *' mgr 'allow *' 2024-03-11T21:26:00.595 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:26:00 smithi092 ceph-mon[29572]: osdmap e79: 8 total, 8 up, 8 in 2024-03-11T21:26:00.595 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:26:00 smithi092 ceph-mon[29572]: from='client.24469 v1:172.21.15.97:0/404208310' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:26:00.595 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:26:00 smithi092 ceph-mon[29572]: pgmap v214: 132 pgs: 132 active+clean; 582 KiB data, 226 MiB used, 715 GiB / 715 GiB avail; 797 B/s rd, 0 op/s 2024-03-11T21:26:00.595 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:26:00 smithi092 ceph-mon[29572]: from='client.24493 v1:172.21.15.97:0/2440232012' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "grafana", "placement": "1;smithi097=a", "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T21:26:00.595 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:26:00 smithi092 ceph-mon[29572]: Saving service grafana spec with placement smithi097=a;count:1 2024-03-11T21:26:00.595 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:26:00 smithi092 ceph-mon[29572]: from='mgr.14150 v1:172.21.15.92:0/996057664' entity='mgr.y' 2024-03-11T21:26:00.595 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:26:00 smithi092 ceph-mon[37563]: osdmap e79: 8 total, 8 up, 8 in 2024-03-11T21:26:00.595 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:26:00 smithi092 ceph-mon[37563]: from='client.24469 v1:172.21.15.97:0/404208310' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:26:00.595 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:26:00 smithi092 ceph-mon[37563]: pgmap v214: 132 pgs: 132 active+clean; 582 KiB data, 226 MiB used, 715 GiB / 715 GiB avail; 797 B/s rd, 0 op/s 2024-03-11T21:26:00.596 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:26:00 smithi092 ceph-mon[37563]: from='client.24493 v1:172.21.15.97:0/2440232012' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "grafana", "placement": "1;smithi097=a", "target": ["mon-mgr", ""]}]: dispatch 2024-03-11T21:26:00.596 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:26:00 smithi092 ceph-mon[37563]: Saving service grafana spec with placement smithi097=a;count:1 2024-03-11T21:26:00.596 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:26:00 smithi092 ceph-mon[37563]: from='mgr.14150 v1:172.21.15.92:0/996057664' entity='mgr.y' 2024-03-11T21:26:00.822 INFO:teuthology.orchestra.run.smithi092.stderr:Inferring config /var/lib/ceph/00db0016-dfed-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T21:26:02.827 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:26:02 smithi097 ceph-mon[33317]: purged_snaps scrub starts 2024-03-11T21:26:02.827 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:26:02 smithi097 ceph-mon[33317]: purged_snaps scrub ok 2024-03-11T21:26:02.827 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:26:02 smithi097 ceph-mon[33317]: pgmap v215: 132 pgs: 132 active+clean; 582 KiB data, 226 MiB used, 715 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-03-11T21:26:02.827 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:26:02 smithi097 ceph-mon[33317]: osdmap e80: 8 total, 8 up, 8 in 2024-03-11T21:26:02.861 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:26:02 smithi092 ceph-mon[29572]: purged_snaps scrub starts 2024-03-11T21:26:02.862 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:26:02 smithi092 ceph-mon[29572]: purged_snaps scrub ok 2024-03-11T21:26:02.862 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:26:02 smithi092 ceph-mon[29572]: pgmap v215: 132 pgs: 132 active+clean; 582 KiB data, 226 MiB used, 715 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-03-11T21:26:02.862 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:26:02 smithi092 ceph-mon[29572]: osdmap e80: 8 total, 8 up, 8 in 2024-03-11T21:26:02.862 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:26:02 smithi092 ceph-mon[37563]: purged_snaps scrub starts 2024-03-11T21:26:02.862 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:26:02 smithi092 ceph-mon[37563]: purged_snaps scrub ok 2024-03-11T21:26:02.862 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:26:02 smithi092 ceph-mon[37563]: pgmap v215: 132 pgs: 132 active+clean; 582 KiB data, 226 MiB used, 715 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-03-11T21:26:02.862 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:26:02 smithi092 ceph-mon[37563]: osdmap e80: 8 total, 8 up, 8 in 2024-03-11T21:26:03.057 INFO:teuthology.orchestra.run.smithi092.stdout:[client.0] 2024-03-11T21:26:03.057 INFO:teuthology.orchestra.run.smithi092.stdout: key = AQDrdu9laahXBBAAOJnBZ+qizMtZjx3gcYt6Xg== 2024-03-11T21:26:03.611 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:26:03 smithi092 ceph-mon[37563]: from='client.? v1:172.21.15.92:0/256000404' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-03-11T21:26:03.612 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:26:03 smithi092 ceph-mon[37563]: from='client.? v1:172.21.15.92:0/256000404' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-03-11T21:26:03.612 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:26:03 smithi092 ceph-mon[37563]: osdmap e81: 8 total, 8 up, 8 in 2024-03-11T21:26:03.612 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:26:03 smithi092 ceph-mon[29572]: from='client.? v1:172.21.15.92:0/256000404' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-03-11T21:26:03.612 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:26:03 smithi092 ceph-mon[29572]: from='client.? v1:172.21.15.92:0/256000404' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-03-11T21:26:03.612 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:26:03 smithi092 ceph-mon[29572]: osdmap e81: 8 total, 8 up, 8 in 2024-03-11T21:26:03.807 DEBUG:teuthology.orchestra.run.smithi092:> set -ex 2024-03-11T21:26:03.807 DEBUG:teuthology.orchestra.run.smithi092:> sudo dd of=/etc/ceph/ceph.client.0.keyring 2024-03-11T21:26:03.807 DEBUG:teuthology.orchestra.run.smithi092:> sudo chmod 0644 /etc/ceph/ceph.client.0.keyring 2024-03-11T21:26:03.827 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:26:03 smithi097 ceph-mon[33317]: from='client.? v1:172.21.15.92:0/256000404' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-03-11T21:26:03.827 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:26:03 smithi097 ceph-mon[33317]: from='client.? v1:172.21.15.92:0/256000404' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-03-11T21:26:03.827 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:26:03 smithi097 ceph-mon[33317]: osdmap e81: 8 total, 8 up, 8 in 2024-03-11T21:26:03.893 DEBUG:teuthology.orchestra.run.smithi097:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 00db0016-dfed-11ee-95c8-87774f69a715 -- ceph auth get-or-create client.1 mon 'allow *' osd 'allow *' mds 'allow *' mgr 'allow *' 2024-03-11T21:26:04.133 INFO:teuthology.orchestra.run.smithi097.stderr:Inferring config /var/lib/ceph/00db0016-dfed-11ee-95c8-87774f69a715/mon.b/config 2024-03-11T21:26:04.827 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:26:04 smithi097 ceph-mon[33317]: pgmap v218: 132 pgs: 132 active+clean; 582 KiB data, 226 MiB used, 715 GiB / 715 GiB avail; 944 B/s rd, 0 op/s 2024-03-11T21:26:04.861 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:26:04 smithi092 ceph-mon[29572]: pgmap v218: 132 pgs: 132 active+clean; 582 KiB data, 226 MiB used, 715 GiB / 715 GiB avail; 944 B/s rd, 0 op/s 2024-03-11T21:26:04.861 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:26:04 smithi092 ceph-mon[37563]: pgmap v218: 132 pgs: 132 active+clean; 582 KiB data, 226 MiB used, 715 GiB / 715 GiB avail; 944 B/s rd, 0 op/s 2024-03-11T21:26:06.827 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:26:06 smithi097 ceph-mon[33317]: pgmap v219: 132 pgs: 132 active+clean; 582 KiB data, 226 MiB used, 715 GiB / 715 GiB avail; 810 B/s rd, 0 op/s 2024-03-11T21:26:06.861 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:26:06 smithi092 ceph-mon[29572]: pgmap v219: 132 pgs: 132 active+clean; 582 KiB data, 226 MiB used, 715 GiB / 715 GiB avail; 810 B/s rd, 0 op/s 2024-03-11T21:26:06.861 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:26:06 smithi092 ceph-mon[37563]: pgmap v219: 132 pgs: 132 active+clean; 582 KiB data, 226 MiB used, 715 GiB / 715 GiB avail; 810 B/s rd, 0 op/s 2024-03-11T21:26:08.778 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:26:08 smithi097 ceph-mon[33317]: pgmap v220: 132 pgs: 132 active+clean; 582 KiB data, 226 MiB used, 715 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-03-11T21:26:08.778 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:26:08 smithi097 ceph-mon[33317]: osdmap e82: 8 total, 8 up, 8 in 2024-03-11T21:26:08.861 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:26:08 smithi092 ceph-mon[29572]: pgmap v220: 132 pgs: 132 active+clean; 582 KiB data, 226 MiB used, 715 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-03-11T21:26:08.861 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:26:08 smithi092 ceph-mon[29572]: osdmap e82: 8 total, 8 up, 8 in 2024-03-11T21:26:08.862 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:26:08 smithi092 ceph-mon[37563]: pgmap v220: 132 pgs: 132 active+clean; 582 KiB data, 226 MiB used, 715 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-03-11T21:26:08.862 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:26:08 smithi092 ceph-mon[37563]: osdmap e82: 8 total, 8 up, 8 in 2024-03-11T21:26:09.183 INFO:teuthology.orchestra.run.smithi097.stdout:[client.1] 2024-03-11T21:26:09.183 INFO:teuthology.orchestra.run.smithi097.stdout: key = AQDxdu9l3+TACxAAGL0yjh97sN1SN4BfYPacGA== 2024-03-11T21:26:09.692 INFO:journalctl@ceph.iscsi.iscsi.a.smithi097.stdout:Mar 11 21:26:09 smithi097 ceph-00db0016-dfed-11ee-95c8-87774f69a715-iscsi-iscsi-a[60943]: debug there is no tcmu-runner data available 2024-03-11T21:26:09.692 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:26:09 smithi097 ceph-mon[33317]: from='client.? v1:172.21.15.97:0/4028856187' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-03-11T21:26:09.692 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:26:09 smithi097 ceph-mon[33317]: from='client.24505 ' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-03-11T21:26:09.692 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:26:09 smithi097 ceph-mon[33317]: from='client.24505 ' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-03-11T21:26:09.692 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:26:09 smithi097 ceph-mon[33317]: from='mgr.14150 v1:172.21.15.92:0/996057664' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.3", "id": [7, 4]}]: dispatch 2024-03-11T21:26:09.692 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:26:09 smithi097 ceph-mon[33317]: from='mgr.14150 v1:172.21.15.92:0/996057664' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.f", "id": [1, 2]}]: dispatch 2024-03-11T21:26:09.692 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:26:09 smithi097 ceph-mon[33317]: from='mgr.14150 v1:172.21.15.92:0/996057664' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.13", "id": [1, 2]}]: dispatch 2024-03-11T21:26:09.692 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:26:09 smithi097 ceph-mon[33317]: from='mgr.14150 v1:172.21.15.92:0/996057664' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.17", "id": [1, 5]}]: dispatch 2024-03-11T21:26:09.861 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:26:09 smithi092 ceph-mon[29572]: from='client.? v1:172.21.15.97:0/4028856187' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-03-11T21:26:09.862 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:26:09 smithi092 ceph-mon[29572]: from='client.24505 ' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-03-11T21:26:09.862 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:26:09 smithi092 ceph-mon[29572]: from='client.24505 ' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-03-11T21:26:09.862 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:26:09 smithi092 ceph-mon[29572]: from='mgr.14150 v1:172.21.15.92:0/996057664' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.3", "id": [7, 4]}]: dispatch 2024-03-11T21:26:09.862 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:26:09 smithi092 ceph-mon[29572]: from='mgr.14150 v1:172.21.15.92:0/996057664' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.f", "id": [1, 2]}]: dispatch 2024-03-11T21:26:09.862 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:26:09 smithi092 ceph-mon[29572]: from='mgr.14150 v1:172.21.15.92:0/996057664' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.13", "id": [1, 2]}]: dispatch 2024-03-11T21:26:09.862 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:26:09 smithi092 ceph-mon[29572]: from='mgr.14150 v1:172.21.15.92:0/996057664' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.17", "id": [1, 5]}]: dispatch 2024-03-11T21:26:09.862 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:26:09 smithi092 ceph-mon[37563]: from='client.? v1:172.21.15.97:0/4028856187' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-03-11T21:26:09.862 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:26:09 smithi092 ceph-mon[37563]: from='client.24505 ' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-03-11T21:26:09.862 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:26:09 smithi092 ceph-mon[37563]: from='client.24505 ' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-03-11T21:26:09.862 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:26:09 smithi092 ceph-mon[37563]: from='mgr.14150 v1:172.21.15.92:0/996057664' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.3", "id": [7, 4]}]: dispatch 2024-03-11T21:26:09.863 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:26:09 smithi092 ceph-mon[37563]: from='mgr.14150 v1:172.21.15.92:0/996057664' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.f", "id": [1, 2]}]: dispatch 2024-03-11T21:26:09.863 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:26:09 smithi092 ceph-mon[37563]: from='mgr.14150 v1:172.21.15.92:0/996057664' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.13", "id": [1, 2]}]: dispatch 2024-03-11T21:26:09.863 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:26:09 smithi092 ceph-mon[37563]: from='mgr.14150 v1:172.21.15.92:0/996057664' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.17", "id": [1, 5]}]: dispatch 2024-03-11T21:26:10.827 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:26:10 smithi097 ceph-mon[33317]: from='client.24469 v1:172.21.15.97:0/404208310' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:26:10.827 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:26:10 smithi097 ceph-mon[33317]: pgmap v222: 132 pgs: 132 active+clean; 582 KiB data, 226 MiB used, 715 GiB / 715 GiB avail; 1.3 KiB/s rd, 1 op/s 2024-03-11T21:26:10.827 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:26:10 smithi097 ceph-mon[33317]: from='mgr.14150 v1:172.21.15.92:0/996057664' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.3", "id": [7, 4]}]': finished 2024-03-11T21:26:10.827 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:26:10 smithi097 ceph-mon[33317]: from='mgr.14150 v1:172.21.15.92:0/996057664' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.f", "id": [1, 2]}]': finished 2024-03-11T21:26:10.828 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:26:10 smithi097 ceph-mon[33317]: from='mgr.14150 v1:172.21.15.92:0/996057664' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.13", "id": [1, 2]}]': finished 2024-03-11T21:26:10.828 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:26:10 smithi097 ceph-mon[33317]: from='mgr.14150 v1:172.21.15.92:0/996057664' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.17", "id": [1, 5]}]': finished 2024-03-11T21:26:10.828 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:26:10 smithi097 ceph-mon[33317]: osdmap e83: 8 total, 8 up, 8 in 2024-03-11T21:26:10.861 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:26:10 smithi092 ceph-mon[29572]: from='client.24469 v1:172.21.15.97:0/404208310' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:26:10.861 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:26:10 smithi092 ceph-mon[29572]: pgmap v222: 132 pgs: 132 active+clean; 582 KiB data, 226 MiB used, 715 GiB / 715 GiB avail; 1.3 KiB/s rd, 1 op/s 2024-03-11T21:26:10.861 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:26:10 smithi092 ceph-mon[29572]: from='mgr.14150 v1:172.21.15.92:0/996057664' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.3", "id": [7, 4]}]': finished 2024-03-11T21:26:10.862 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:26:10 smithi092 ceph-mon[29572]: from='mgr.14150 v1:172.21.15.92:0/996057664' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.f", "id": [1, 2]}]': finished 2024-03-11T21:26:10.862 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:26:10 smithi092 ceph-mon[29572]: from='mgr.14150 v1:172.21.15.92:0/996057664' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.13", "id": [1, 2]}]': finished 2024-03-11T21:26:10.862 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:26:10 smithi092 ceph-mon[29572]: from='mgr.14150 v1:172.21.15.92:0/996057664' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.17", "id": [1, 5]}]': finished 2024-03-11T21:26:10.862 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:26:10 smithi092 ceph-mon[29572]: osdmap e83: 8 total, 8 up, 8 in 2024-03-11T21:26:10.862 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:26:10 smithi092 ceph-mon[37563]: from='client.24469 v1:172.21.15.97:0/404208310' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:26:10.862 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:26:10 smithi092 ceph-mon[37563]: pgmap v222: 132 pgs: 132 active+clean; 582 KiB data, 226 MiB used, 715 GiB / 715 GiB avail; 1.3 KiB/s rd, 1 op/s 2024-03-11T21:26:10.862 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:26:10 smithi092 ceph-mon[37563]: from='mgr.14150 v1:172.21.15.92:0/996057664' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.3", "id": [7, 4]}]': finished 2024-03-11T21:26:10.862 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:26:10 smithi092 ceph-mon[37563]: from='mgr.14150 v1:172.21.15.92:0/996057664' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.f", "id": [1, 2]}]': finished 2024-03-11T21:26:10.862 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:26:10 smithi092 ceph-mon[37563]: from='mgr.14150 v1:172.21.15.92:0/996057664' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.13", "id": [1, 2]}]': finished 2024-03-11T21:26:10.862 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:26:10 smithi092 ceph-mon[37563]: from='mgr.14150 v1:172.21.15.92:0/996057664' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.17", "id": [1, 5]}]': finished 2024-03-11T21:26:10.862 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:26:10 smithi092 ceph-mon[37563]: osdmap e83: 8 total, 8 up, 8 in 2024-03-11T21:26:12.113 DEBUG:teuthology.orchestra.run.smithi097:> set -ex 2024-03-11T21:26:12.113 DEBUG:teuthology.orchestra.run.smithi097:> sudo dd of=/etc/ceph/ceph.client.1.keyring 2024-03-11T21:26:12.113 DEBUG:teuthology.orchestra.run.smithi097:> sudo chmod 0644 /etc/ceph/ceph.client.1.keyring 2024-03-11T21:26:12.238 INFO:tasks.ceph:Waiting until ceph daemons up and pgs clean... 2024-03-11T21:26:12.238 INFO:tasks.cephadm.ceph_manager.ceph:waiting for mgr available 2024-03-11T21:26:12.239 DEBUG:teuthology.orchestra.run.smithi092:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid 00db0016-dfed-11ee-95c8-87774f69a715 -- ceph mgr dump --format=json 2024-03-11T21:26:12.451 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:26:12 smithi097 ceph-mon[33317]: osdmap e84: 8 total, 8 up, 8 in 2024-03-11T21:26:12.451 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:26:12 smithi097 ceph-mon[33317]: 1.0 scrub starts 2024-03-11T21:26:12.452 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:26:12 smithi097 ceph-mon[33317]: 1.0 scrub ok 2024-03-11T21:26:12.452 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:26:12 smithi097 ceph-mon[33317]: pgmap v225: 132 pgs: 2 peering, 130 active+clean; 582 KiB data, 226 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-11T21:26:12.494 INFO:teuthology.orchestra.run.smithi092.stderr:Inferring config /var/lib/ceph/00db0016-dfed-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T21:26:12.611 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:26:12 smithi092 ceph-mon[37563]: osdmap e84: 8 total, 8 up, 8 in 2024-03-11T21:26:12.611 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:26:12 smithi092 ceph-mon[37563]: 1.0 scrub starts 2024-03-11T21:26:12.611 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:26:12 smithi092 ceph-mon[37563]: 1.0 scrub ok 2024-03-11T21:26:12.611 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:26:12 smithi092 ceph-mon[37563]: pgmap v225: 132 pgs: 2 peering, 130 active+clean; 582 KiB data, 226 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-11T21:26:12.612 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:26:12 smithi092 ceph-mon[29572]: osdmap e84: 8 total, 8 up, 8 in 2024-03-11T21:26:12.612 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:26:12 smithi092 ceph-mon[29572]: 1.0 scrub starts 2024-03-11T21:26:12.612 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:26:12 smithi092 ceph-mon[29572]: 1.0 scrub ok 2024-03-11T21:26:12.612 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:26:12 smithi092 ceph-mon[29572]: pgmap v225: 132 pgs: 2 peering, 130 active+clean; 582 KiB data, 226 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-11T21:26:12.716 INFO:journalctl@ceph.prometheus.a.smithi097.stdout:Mar 11 21:26:12 smithi097 ceph-00db0016-dfed-11ee-95c8-87774f69a715-prometheus-a[62608]: ts=2024-03-11T21:26:12.714Z caller=main.go:1209 level=info msg="Loading configuration file" filename=/etc/prometheus/prometheus.yml 2024-03-11T21:26:12.717 INFO:journalctl@ceph.prometheus.a.smithi097.stdout:Mar 11 21:26:12 smithi097 ceph-00db0016-dfed-11ee-95c8-87774f69a715-prometheus-a[62608]: ts=2024-03-11T21:26:12.734Z caller=main.go:1246 level=info msg="Completed loading of configuration file" filename=/etc/prometheus/prometheus.yml totalDuration=19.781958ms db_storage=762ns remote_storage=1.51µs web_handler=281ns query_engine=570ns scrape=380.789µs scrape_sd=70.631µs notify=693ns notify_sd=1.397µs rules=19.074491ms tracing=5.504µs 2024-03-11T21:26:12.717 INFO:journalctl@ceph.prometheus.a.smithi097.stdout:Mar 11 21:26:12 smithi097 ceph-00db0016-dfed-11ee-95c8-87774f69a715-prometheus-a[62608]: ts=2024-03-11T21:26:12.734Z caller=main.go:990 level=info msg="Server is ready to receive web requests." 2024-03-11T21:26:12.717 INFO:journalctl@ceph.prometheus.a.smithi097.stdout:Mar 11 21:26:12 smithi097 ceph-00db0016-dfed-11ee-95c8-87774f69a715-prometheus-a[62608]: ts=2024-03-11T21:26:12.734Z caller=manager.go:974 level=info component="rule manager" msg="Starting rule manager..." 2024-03-11T21:26:13.077 INFO:journalctl@ceph.prometheus.a.smithi097.stdout:Mar 11 21:26:12 smithi097 bash[62594]: 905dc8aea8e89f027807d67743354a08f5c0189061939f389bc0a2c4c5ce4207 2024-03-11T21:26:13.077 INFO:journalctl@ceph.prometheus.a.smithi097.stdout:Mar 11 21:26:12 smithi097 systemd[1]: Started Ceph prometheus.a for 00db0016-dfed-11ee-95c8-87774f69a715. 2024-03-11T21:26:13.577 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:26:13 smithi097 ceph-mon[33317]: purged_snaps scrub starts 2024-03-11T21:26:13.577 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:26:13 smithi097 ceph-mon[33317]: purged_snaps scrub ok 2024-03-11T21:26:13.577 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:26:13 smithi097 ceph-mon[33317]: from='mgr.14150 v1:172.21.15.92:0/996057664' entity='mgr.y' 2024-03-11T21:26:13.577 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:26:13 smithi097 ceph-mon[33317]: from='mgr.14150 v1:172.21.15.92:0/996057664' entity='mgr.y' 2024-03-11T21:26:13.578 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:26:13 smithi097 ceph-mon[33317]: from='mgr.14150 v1:172.21.15.92:0/996057664' entity='mgr.y' 2024-03-11T21:26:13.578 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:26:13 smithi097 ceph-mon[33317]: from='mgr.14150 v1:172.21.15.92:0/996057664' entity='mgr.y' cmd=[{"prefix": "mgr module enable", "module": "prometheus"}]: dispatch 2024-03-11T21:26:13.611 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:26:13 smithi092 ceph-mon[29572]: purged_snaps scrub starts 2024-03-11T21:26:13.612 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:26:13 smithi092 ceph-mon[29572]: purged_snaps scrub ok 2024-03-11T21:26:13.612 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:26:13 smithi092 ceph-mon[29572]: from='mgr.14150 v1:172.21.15.92:0/996057664' entity='mgr.y' 2024-03-11T21:26:13.612 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:26:13 smithi092 ceph-mon[29572]: from='mgr.14150 v1:172.21.15.92:0/996057664' entity='mgr.y' 2024-03-11T21:26:13.612 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:26:13 smithi092 ceph-mon[29572]: from='mgr.14150 v1:172.21.15.92:0/996057664' entity='mgr.y' 2024-03-11T21:26:13.612 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:26:13 smithi092 ceph-mon[29572]: from='mgr.14150 v1:172.21.15.92:0/996057664' entity='mgr.y' cmd=[{"prefix": "mgr module enable", "module": "prometheus"}]: dispatch 2024-03-11T21:26:13.612 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:26:13 smithi092 ceph-mon[37563]: purged_snaps scrub starts 2024-03-11T21:26:13.612 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:26:13 smithi092 ceph-mon[37563]: purged_snaps scrub ok 2024-03-11T21:26:13.612 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:26:13 smithi092 ceph-mon[37563]: from='mgr.14150 v1:172.21.15.92:0/996057664' entity='mgr.y' 2024-03-11T21:26:13.612 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:26:13 smithi092 ceph-mon[37563]: from='mgr.14150 v1:172.21.15.92:0/996057664' entity='mgr.y' 2024-03-11T21:26:13.613 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:26:13 smithi092 ceph-mon[37563]: from='mgr.14150 v1:172.21.15.92:0/996057664' entity='mgr.y' 2024-03-11T21:26:13.613 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:26:13 smithi092 ceph-mon[37563]: from='mgr.14150 v1:172.21.15.92:0/996057664' entity='mgr.y' cmd=[{"prefix": "mgr module enable", "module": "prometheus"}]: dispatch 2024-03-11T21:26:13.979 INFO:journalctl@ceph.mgr.y.smithi092.stdout:Mar 11 21:26:13 smithi092 ceph-00db0016-dfed-11ee-95c8-87774f69a715-mgr-y[29801]: ignoring --setuser ceph since I am not root 2024-03-11T21:26:13.979 INFO:journalctl@ceph.mgr.y.smithi092.stdout:Mar 11 21:26:13 smithi092 ceph-00db0016-dfed-11ee-95c8-87774f69a715-mgr-y[29801]: ignoring --setgroup ceph since I am not root 2024-03-11T21:26:14.204 INFO:journalctl@ceph.mgr.x.smithi097.stdout:Mar 11 21:26:13 smithi097 ceph-00db0016-dfed-11ee-95c8-87774f69a715-mgr-x[35169]: ignoring --setuser ceph since I am not root 2024-03-11T21:26:14.204 INFO:journalctl@ceph.mgr.x.smithi097.stdout:Mar 11 21:26:13 smithi097 ceph-00db0016-dfed-11ee-95c8-87774f69a715-mgr-x[35169]: ignoring --setgroup ceph since I am not root 2024-03-11T21:26:14.340 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:26:14 smithi092 ceph-mon[29572]: osdmap e85: 8 total, 8 up, 8 in 2024-03-11T21:26:14.340 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:26:14 smithi092 ceph-mon[29572]: pgmap v227: 132 pgs: 2 peering, 130 active+clean; 582 KiB data, 227 MiB used, 715 GiB / 715 GiB avail; 944 B/s rd, 0 op/s; 20 B/s, 1 objects/s recovering 2024-03-11T21:26:14.340 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:26:14 smithi092 ceph-mon[29572]: from='mgr.14150 v1:172.21.15.92:0/996057664' entity='mgr.y' cmd='[{"prefix": "mgr module enable", "module": "prometheus"}]': finished 2024-03-11T21:26:14.340 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:26:14 smithi092 ceph-mon[29572]: mgrmap e17: y(active, since 5m), standbys: x 2024-03-11T21:26:14.341 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:26:14 smithi092 ceph-mon[37563]: osdmap e85: 8 total, 8 up, 8 in 2024-03-11T21:26:14.341 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:26:14 smithi092 ceph-mon[37563]: pgmap v227: 132 pgs: 2 peering, 130 active+clean; 582 KiB data, 227 MiB used, 715 GiB / 715 GiB avail; 944 B/s rd, 0 op/s; 20 B/s, 1 objects/s recovering 2024-03-11T21:26:14.341 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:26:14 smithi092 ceph-mon[37563]: from='mgr.14150 v1:172.21.15.92:0/996057664' entity='mgr.y' cmd='[{"prefix": "mgr module enable", "module": "prometheus"}]': finished 2024-03-11T21:26:14.341 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:26:14 smithi092 ceph-mon[37563]: mgrmap e17: y(active, since 5m), standbys: x 2024-03-11T21:26:14.483 INFO:teuthology.orchestra.run.smithi092.stdout: 2024-03-11T21:26:14.512 INFO:journalctl@ceph.mgr.x.smithi097.stdout:Mar 11 21:26:14 smithi097 ceph-00db0016-dfed-11ee-95c8-87774f69a715-mgr-x[35169]: 2024-03-11T21:26:14.435+0000 7f07987a2200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-03-11T21:26:14.513 INFO:journalctl@ceph.mgr.x.smithi097.stdout:Mar 11 21:26:14 smithi097 ceph-00db0016-dfed-11ee-95c8-87774f69a715-mgr-x[35169]: 2024-03-11T21:26:14.529+0000 7f07987a2200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-03-11T21:26:14.513 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:26:14 smithi097 ceph-mon[33317]: osdmap e85: 8 total, 8 up, 8 in 2024-03-11T21:26:14.513 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:26:14 smithi097 ceph-mon[33317]: pgmap v227: 132 pgs: 2 peering, 130 active+clean; 582 KiB data, 227 MiB used, 715 GiB / 715 GiB avail; 944 B/s rd, 0 op/s; 20 B/s, 1 objects/s recovering 2024-03-11T21:26:14.513 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:26:14 smithi097 ceph-mon[33317]: from='mgr.14150 v1:172.21.15.92:0/996057664' entity='mgr.y' cmd='[{"prefix": "mgr module enable", "module": "prometheus"}]': finished 2024-03-11T21:26:14.513 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:26:14 smithi097 ceph-mon[33317]: mgrmap e17: y(active, since 5m), standbys: x 2024-03-11T21:26:14.611 INFO:journalctl@ceph.mgr.y.smithi092.stdout:Mar 11 21:26:14 smithi092 ceph-00db0016-dfed-11ee-95c8-87774f69a715-mgr-y[29801]: 2024-03-11T21:26:14.482+0000 7f2bd61b5200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-03-11T21:26:14.611 INFO:journalctl@ceph.mgr.y.smithi092.stdout:Mar 11 21:26:14 smithi092 ceph-00db0016-dfed-11ee-95c8-87774f69a715-mgr-y[29801]: 2024-03-11T21:26:14.582+0000 7f2bd61b5200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-03-11T21:26:15.124 INFO:teuthology.orchestra.run.smithi092.stdout:{"epoch":17,"flags":0,"active_gid":14150,"active_name":"y","active_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.92:6800","nonce":2343383852}]},"active_addr":"172.21.15.92:6800/2343383852","active_change":"2024-03-11T21:21:09.474037+0000","active_mgr_features":4540701547738038271,"available":true,"standbys":[{"gid":24112,"name":"x","mgr_features":4540701547738038271,"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2359","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","read","upmap","upmap-read"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cephadm_log_destination":{"name":"cephadm_log_destination","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":["file","file,syslog","syslog"],"desc":"Destination for cephadm command's persistent logging","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.25.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_elasticsearch":{"name":"container_image_elasticsearch","type":"str","level":"advanced","flags":0,"default_value":"quay.io/omrizeneva/elasticsearch:6.8.23","min":"","max":"","enum_allowed":[],"desc":"elasticsearch container image","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/ceph-grafana:9.4.12","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_agent":{"name":"container_image_jaeger_agent","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-agent:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger agent container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_collector":{"name":"container_image_jaeger_collector","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-collector:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger collector container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_query":{"name":"container_image_jaeger_query","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-query:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger query container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.2.4","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.5.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_nvmeof":{"name":"container_image_nvmeof","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nvmeof:1.0.0","min":"","max":"","enum_allowed":[],"desc":"Nvme-of container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.43.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"int","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"hw_monitoring":{"name":"hw_monitoring","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Deploy hw monitoring daemon on every host.","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"oob_default_addr":{"name":"oob_default_addr","type":"str","level":"advanced","flags":0,"default_value":"169.254.1.1","min":"","max":"","enum_allowed":[],"desc":"Default address for RedFish API (oob management).","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all the monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"service_discovery_port":{"name":"service_discovery_port","type":"int","level":"advanced","flags":0,"default_value":"8765","min":"","max":"","enum_allowed":[],"desc":"cephadm service discovery port","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"MULTICLUSTER_CONFIG":{"name":"MULTICLUSTER_CONFIG","type":"str","level":"advanced","flags":0,"default_value":"{}","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"UNSAFE_TLS_v1_2":{"name":"UNSAFE_TLS_v1_2","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"k8sevents","can_run":true,"error_string":"","module_options":{"ceph_event_retention_days":{"name":"ceph_event_retention_days","type":"int","level":"advanced","flags":0,"default_value":"7","min":"","max":"","enum_allowed":[],"desc":"Days to hold ceph event information within local cache","long_desc":"","tags":[],"see_also":[]},"config_check_secs":{"name":"config_check_secs","type":"int","level":"advanced","flags":0,"default_value":"10","min":"10","max":"","enum_allowed":[],"desc":"interval (secs) to check for cluster configuration changes","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"fail_fs":{"name":"fail_fs","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Fail filesystem for rapid multi-rank mds upgrade","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"exclude_perf_counters":{"name":"exclude_perf_counters","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Do not include perf-counters in the metrics output","long_desc":"Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rgw","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard_description":{"name":"leaderboard_description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"periodic_async_work":{"name":"periodic_async_work","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Periodically check for async work","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_no_wait":{"name":"snapshot_clone_no_wait","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Reject subvolume clone request when cloner threads are busy","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}]}],"modules":["cephadm","dashboard","iostat","nfs","prometheus","restful"],"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2359","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","read","upmap","upmap-read"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cephadm_log_destination":{"name":"cephadm_log_destination","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":["file","file,syslog","syslog"],"desc":"Destination for cephadm command's persistent logging","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.25.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_elasticsearch":{"name":"container_image_elasticsearch","type":"str","level":"advanced","flags":0,"default_value":"quay.io/omrizeneva/elasticsearch:6.8.23","min":"","max":"","enum_allowed":[],"desc":"elasticsearch container image","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/ceph-grafana:9.4.12","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_agent":{"name":"container_image_jaeger_agent","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-agent:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger agent container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_collector":{"name":"container_image_jaeger_collector","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-collector:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger collector container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_query":{"name":"container_image_jaeger_query","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-query:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger query container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.2.4","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.5.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_nvmeof":{"name":"container_image_nvmeof","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nvmeof:1.0.0","min":"","max":"","enum_allowed":[],"desc":"Nvme-of container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.43.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"int","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"hw_monitoring":{"name":"hw_monitoring","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Deploy hw monitoring daemon on every host.","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"oob_default_addr":{"name":"oob_default_addr","type":"str","level":"advanced","flags":0,"default_value":"169.254.1.1","min":"","max":"","enum_allowed":[],"desc":"Default address for RedFish API (oob management).","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all the monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"service_discovery_port":{"name":"service_discovery_port","type":"int","level":"advanced","flags":0,"default_value":"8765","min":"","max":"","enum_allowed":[],"desc":"cephadm service discovery port","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"MULTICLUSTER_CONFIG":{"name":"MULTICLUSTER_CONFIG","type":"str","level":"advanced","flags":0,"default_value":"{}","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"UNSAFE_TLS_v1_2":{"name":"UNSAFE_TLS_v1_2","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"k8sevents","can_run":true,"error_string":"","module_options":{"ceph_event_retention_days":{"name":"ceph_event_retention_days","type":"int","level":"advanced","flags":0,"default_value":"7","min":"","max":"","enum_allowed":[],"desc":"Days to hold ceph event information within local cache","long_desc":"","tags":[],"see_also":[]},"config_check_secs":{"name":"config_check_secs","type":"int","level":"advanced","flags":0,"default_value":"10","min":"10","max":"","enum_allowed":[],"desc":"interval (secs) to check for cluster configuration changes","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"fail_fs":{"name":"fail_fs","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Fail filesystem for rapid multi-rank mds upgrade","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"exclude_perf_counters":{"name":"exclude_perf_counters","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Do not include perf-counters in the metrics output","long_desc":"Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rgw","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard_description":{"name":"leaderboard_description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"periodic_async_work":{"name":"periodic_async_work","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Periodically check for async work","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_no_wait":{"name":"snapshot_clone_no_wait","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Reject subvolume clone request when cloner threads are busy","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}],"services":{"dashboard":"https://172.21.15.92:8443/"},"always_on_modules":{"octopus":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"pacific":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"quincy":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"reef":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"squid":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"]},"last_failure_osd_epoch":3,"active_clients":[{"name":"cephadm","addrvec":[{"type":"v2","addr":"172.21.15.92:0","nonce":2706821241}]},{"name":"devicehealth","addrvec":[{"type":"v2","addr":"172.21.15.92:0","nonce":2912406656}]},{"name":"libcephsqlite","addrvec":[{"type":"v2","addr":"172.21.15.92:0","nonce":437911229}]},{"name":"rbd_support","addrvec":[{"type":"v2","addr":"172.21.15.92:0","nonce":4246054050}]},{"name":"volumes","addrvec":[{"type":"v2","addr":"172.21.15.92:0","nonce":303695351}]}]} 2024-03-11T21:26:15.128 INFO:tasks.cephadm.ceph_manager.ceph:mgr available! 2024-03-11T21:26:15.128 INFO:tasks.cephadm.ceph_manager.ceph:waiting for all up 2024-03-11T21:26:15.128 DEBUG:teuthology.orchestra.run.smithi092:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid 00db0016-dfed-11ee-95c8-87774f69a715 -- ceph osd dump --format=json 2024-03-11T21:26:15.361 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:26:15 smithi092 ceph-mon[29572]: from='client.? v1:172.21.15.92:0/1277572167' entity='client.admin' cmd=[{"prefix": "mgr dump", "format": "json"}]: dispatch 2024-03-11T21:26:15.362 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:26:15 smithi092 ceph-mon[37563]: from='client.? v1:172.21.15.92:0/1277572167' entity='client.admin' cmd=[{"prefix": "mgr dump", "format": "json"}]: dispatch 2024-03-11T21:26:15.362 INFO:journalctl@ceph.mgr.y.smithi092.stdout:Mar 11 21:26:15 smithi092 ceph-00db0016-dfed-11ee-95c8-87774f69a715-mgr-y[29801]: 2024-03-11T21:26:15.235+0000 7f2bd61b5200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-03-11T21:26:15.362 INFO:journalctl@ceph.mgr.y.smithi092.stdout:Mar 11 21:26:15 smithi092 ceph-00db0016-dfed-11ee-95c8-87774f69a715-mgr-y[29801]: 2024-03-11T21:26:15.333+0000 7f2bd61b5200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-03-11T21:26:15.388 INFO:teuthology.orchestra.run.smithi092.stderr:Inferring config /var/lib/ceph/00db0016-dfed-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T21:26:15.473 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:26:15 smithi097 ceph-mon[33317]: from='client.? v1:172.21.15.92:0/1277572167' entity='client.admin' cmd=[{"prefix": "mgr dump", "format": "json"}]: dispatch 2024-03-11T21:26:15.473 INFO:journalctl@ceph.mgr.x.smithi097.stdout:Mar 11 21:26:15 smithi097 ceph-00db0016-dfed-11ee-95c8-87774f69a715-mgr-x[35169]: 2024-03-11T21:26:15.160+0000 7f07987a2200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-03-11T21:26:15.474 INFO:journalctl@ceph.mgr.x.smithi097.stdout:Mar 11 21:26:15 smithi097 ceph-00db0016-dfed-11ee-95c8-87774f69a715-mgr-x[35169]: 2024-03-11T21:26:15.253+0000 7f07987a2200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-03-11T21:26:15.474 INFO:journalctl@ceph.mgr.x.smithi097.stdout:Mar 11 21:26:15 smithi097 ceph-00db0016-dfed-11ee-95c8-87774f69a715-mgr-x[35169]: 2024-03-11T21:26:15.394+0000 7f07987a2200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-03-11T21:26:15.474 INFO:journalctl@ceph.mgr.x.smithi097.stdout:Mar 11 21:26:15 smithi097 ceph-00db0016-dfed-11ee-95c8-87774f69a715-mgr-x[35169]: 2024-03-11T21:26:15.489+0000 7f07987a2200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-03-11T21:26:15.749 INFO:journalctl@ceph.mgr.y.smithi092.stdout:Mar 11 21:26:15 smithi092 ceph-00db0016-dfed-11ee-95c8-87774f69a715-mgr-y[29801]: 2024-03-11T21:26:15.495+0000 7f2bd61b5200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-03-11T21:26:15.750 INFO:journalctl@ceph.mgr.y.smithi092.stdout:Mar 11 21:26:15 smithi092 ceph-00db0016-dfed-11ee-95c8-87774f69a715-mgr-y[29801]: 2024-03-11T21:26:15.592+0000 7f2bd61b5200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-03-11T21:26:15.750 INFO:journalctl@ceph.mgr.y.smithi092.stdout:Mar 11 21:26:15 smithi092 ceph-00db0016-dfed-11ee-95c8-87774f69a715-mgr-y[29801]: 2024-03-11T21:26:15.680+0000 7f2bd61b5200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-03-11T21:26:15.820 INFO:journalctl@ceph.mgr.x.smithi097.stdout:Mar 11 21:26:15 smithi097 ceph-00db0016-dfed-11ee-95c8-87774f69a715-mgr-x[35169]: 2024-03-11T21:26:15.573+0000 7f07987a2200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-03-11T21:26:16.043 INFO:journalctl@ceph.mgr.y.smithi092.stdout:Mar 11 21:26:15 smithi092 ceph-00db0016-dfed-11ee-95c8-87774f69a715-mgr-y[29801]: 2024-03-11T21:26:15.949+0000 7f2bd61b5200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-03-11T21:26:16.077 INFO:journalctl@ceph.mgr.x.smithi097.stdout:Mar 11 21:26:15 smithi097 ceph-00db0016-dfed-11ee-95c8-87774f69a715-mgr-x[35169]: 2024-03-11T21:26:15.836+0000 7f07987a2200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-03-11T21:26:16.077 INFO:journalctl@ceph.mgr.x.smithi097.stdout:Mar 11 21:26:15 smithi097 ceph-00db0016-dfed-11ee-95c8-87774f69a715-mgr-x[35169]: 2024-03-11T21:26:15.945+0000 7f07987a2200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-03-11T21:26:16.077 INFO:journalctl@ceph.mgr.x.smithi097.stdout:Mar 11 21:26:16 smithi097 ceph-00db0016-dfed-11ee-95c8-87774f69a715-mgr-x[35169]: 2024-03-11T21:26:16.030+0000 7f07987a2200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-03-11T21:26:16.361 INFO:journalctl@ceph.mgr.y.smithi092.stdout:Mar 11 21:26:16 smithi092 ceph-00db0016-dfed-11ee-95c8-87774f69a715-mgr-y[29801]: 2024-03-11T21:26:16.059+0000 7f2bd61b5200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-03-11T21:26:16.361 INFO:journalctl@ceph.mgr.y.smithi092.stdout:Mar 11 21:26:16 smithi092 ceph-00db0016-dfed-11ee-95c8-87774f69a715-mgr-y[29801]: 2024-03-11T21:26:16.150+0000 7f2bd61b5200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-03-11T21:26:16.361 INFO:journalctl@ceph.mgr.y.smithi092.stdout:Mar 11 21:26:16 smithi092 ceph-00db0016-dfed-11ee-95c8-87774f69a715-mgr-y[29801]: 2024-03-11T21:26:16.246+0000 7f2bd61b5200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-03-11T21:26:16.577 INFO:journalctl@ceph.mgr.x.smithi097.stdout:Mar 11 21:26:16 smithi097 ceph-00db0016-dfed-11ee-95c8-87774f69a715-mgr-x[35169]: 2024-03-11T21:26:16.124+0000 7f07987a2200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-03-11T21:26:16.577 INFO:journalctl@ceph.mgr.x.smithi097.stdout:Mar 11 21:26:16 smithi097 ceph-00db0016-dfed-11ee-95c8-87774f69a715-mgr-x[35169]: 2024-03-11T21:26:16.298+0000 7f07987a2200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-03-11T21:26:16.768 INFO:journalctl@ceph.mgr.y.smithi092.stdout:Mar 11 21:26:16 smithi092 ceph-00db0016-dfed-11ee-95c8-87774f69a715-mgr-y[29801]: 2024-03-11T21:26:16.425+0000 7f2bd61b5200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-03-11T21:26:17.319 INFO:teuthology.orchestra.run.smithi092.stdout: 2024-03-11T21:26:17.319 INFO:teuthology.orchestra.run.smithi092.stdout:{"epoch":85,"fsid":"00db0016-dfed-11ee-95c8-87774f69a715","created":"2024-03-11T21:20:10.488059+0000","modified":"2024-03-11T21:26:13.212133+0000","last_up_change":"2024-03-11T21:25:26.018427+0000","last_in_change":"2024-03-11T21:25:08.189557+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":18,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":6,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-11T21:23:23.805029+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":7.8899998664855957,"score_stable":7.8899998664855957,"optimal_score":0.37999999523162842,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":".rgw.root","create_time":"2024-03-11T21:25:37.044504+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"68","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rgw":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"datapool","create_time":"2024-03-11T21:25:37.280820+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":3,"pg_placement_num":3,"pg_placement_num_target":3,"pg_num_target":3,"pg_num_pending":3,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"71","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":2,"snap_epoch":71,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_type":"Fair distribution","score_acting":2.6500000953674316,"score_stable":2.6500000953674316,"optimal_score":0.87999999523162842,"raw_score_acting":2.3299999237060547,"raw_score_stable":2.3299999237060547,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"default.rgw.log","create_time":"2024-03-11T21:25:38.850207+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"70","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rgw":{}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"default.rgw.control","create_time":"2024-03-11T21:25:40.794686+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"72","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rgw":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.25,"score_stable":1.25,"optimal_score":1,"raw_score_acting":1.25,"raw_score_stable":1.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":6,"pool_name":"default.rgw.meta","create_time":"2024-03-11T21:25:42.872077+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"74","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4},"application_metadata":{"rgw":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"79f15843-f0ec-44aa-931e-4e07a74d3768","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":72,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.92:6801","nonce":2844100201}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.92:6802","nonce":2844100201}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.92:6804","nonce":2844100201}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.92:6803","nonce":2844100201}]},"public_addr":"172.21.15.92:6801/2844100201","cluster_addr":"172.21.15.92:6802/2844100201","heartbeat_back_addr":"172.21.15.92:6804/2844100201","heartbeat_front_addr":"172.21.15.92:6803/2844100201","state":["exists","up"]},{"osd":1,"uuid":"3a5a0e78-1293-4ace-a93f-69156098571d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":72,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.92:6805","nonce":3577871157}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.92:6806","nonce":3577871157}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.92:6808","nonce":3577871157}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.92:6807","nonce":3577871157}]},"public_addr":"172.21.15.92:6805/3577871157","cluster_addr":"172.21.15.92:6806/3577871157","heartbeat_back_addr":"172.21.15.92:6808/3577871157","heartbeat_front_addr":"172.21.15.92:6807/3577871157","state":["exists","up"]},{"osd":2,"uuid":"f207b743-0df4-40ed-a735-950e272fea60","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.92:6809","nonce":760825152}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.92:6810","nonce":760825152}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.92:6812","nonce":760825152}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.92:6811","nonce":760825152}]},"public_addr":"172.21.15.92:6809/760825152","cluster_addr":"172.21.15.92:6810/760825152","heartbeat_back_addr":"172.21.15.92:6812/760825152","heartbeat_front_addr":"172.21.15.92:6811/760825152","state":["exists","up"]},{"osd":3,"uuid":"bbb6f69a-8f0f-47e6-937a-062e1902ae13","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":26,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.92:6813","nonce":670596264}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.92:6814","nonce":670596264}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.92:6816","nonce":670596264}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.92:6815","nonce":670596264}]},"public_addr":"172.21.15.92:6813/670596264","cluster_addr":"172.21.15.92:6814/670596264","heartbeat_back_addr":"172.21.15.92:6816/670596264","heartbeat_front_addr":"172.21.15.92:6815/670596264","state":["exists","up"]},{"osd":4,"uuid":"00266cbc-d2cf-4dee-8a96-3753ebc95ca8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":32,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.97:6800","nonce":420733620}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.97:6801","nonce":420733620}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.97:6803","nonce":420733620}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.97:6802","nonce":420733620}]},"public_addr":"172.21.15.97:6800/420733620","cluster_addr":"172.21.15.97:6801/420733620","heartbeat_back_addr":"172.21.15.97:6803/420733620","heartbeat_front_addr":"172.21.15.97:6802/420733620","state":["exists","up"]},{"osd":5,"uuid":"593a6f6b-8874-40ae-8848-d51c5723a5e4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":41,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.97:6804","nonce":312056041}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.97:6805","nonce":312056041}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.97:6807","nonce":312056041}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.97:6806","nonce":312056041}]},"public_addr":"172.21.15.97:6804/312056041","cluster_addr":"172.21.15.97:6805/312056041","heartbeat_back_addr":"172.21.15.97:6807/312056041","heartbeat_front_addr":"172.21.15.97:6806/312056041","state":["exists","up"]},{"osd":6,"uuid":"91035ecc-6f1a-4cec-bda6-090af9d53eff","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":50,"up_thru":70,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.97:6808","nonce":178994999}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.97:6809","nonce":178994999}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.97:6811","nonce":178994999}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.97:6810","nonce":178994999}]},"public_addr":"172.21.15.97:6808/178994999","cluster_addr":"172.21.15.97:6809/178994999","heartbeat_back_addr":"172.21.15.97:6811/178994999","heartbeat_front_addr":"172.21.15.97:6810/178994999","state":["exists","up"]},{"osd":7,"uuid":"3eb28c24-664a-4bd6-869d-6bf5db98f52f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":60,"up_thru":72,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.97:6812","nonce":3241560509}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.97:6813","nonce":3241560509}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.97:6815","nonce":3241560509}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.97:6814","nonce":3241560509}]},"public_addr":"172.21.15.97:6812/3241560509","cluster_addr":"172.21.15.97:6813/3241560509","heartbeat_back_addr":"172.21.15.97:6815/3241560509","heartbeat_front_addr":"172.21.15.97:6814/3241560509","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-11T21:26:12.296018+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-11T21:25:11.578251+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-11T21:25:45.729567+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-11T21:25:05.154676+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-11T21:25:31.699231+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-11T21:25:58.278204+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-11T21:26:01.515615+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-11T21:25:24.387220+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.3","mappings":[{"from":7,"to":4}]},{"pgid":"4.f","mappings":[{"from":1,"to":2}]},{"pgid":"4.13","mappings":[{"from":1,"to":2}]},{"pgid":"4.17","mappings":[{"from":1,"to":5}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.92:0/3817549754":"2024-03-12T21:21:09.473916+0000","172.21.15.92:0/2211594569":"2024-03-12T21:21:09.473916+0000","172.21.15.92:0/3569900179":"2024-03-12T21:20:43.677016+0000","172.21.15.92:0/3380359534":"2024-03-12T21:20:43.677016+0000","172.21.15.92:0/2509373130":"2024-03-12T21:20:43.677016+0000","172.21.15.92:6800/1559007462":"2024-03-12T21:20:43.677016+0000","172.21.15.92:6800/1270902819":"2024-03-12T21:21:09.473916+0000","172.21.15.92:0/3562645553":"2024-03-12T21:21:09.473916+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-11T21:26:17.577 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:26:17 smithi097 ceph-mon[33317]: from='client.? v1:172.21.15.92:0/1864420962' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-11T21:26:17.577 INFO:journalctl@ceph.mgr.x.smithi097.stdout:Mar 11 21:26:17 smithi097 ceph-00db0016-dfed-11ee-95c8-87774f69a715-mgr-x[35169]: 2024-03-11T21:26:17.250+0000 7f07987a2200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-03-11T21:26:17.577 INFO:journalctl@ceph.mgr.x.smithi097.stdout:Mar 11 21:26:17 smithi097 ceph-00db0016-dfed-11ee-95c8-87774f69a715-mgr-x[35169]: 2024-03-11T21:26:17.381+0000 7f07987a2200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-03-11T21:26:17.611 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:26:17 smithi092 ceph-mon[29572]: from='client.? v1:172.21.15.92:0/1864420962' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-11T21:26:17.611 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:26:17 smithi092 ceph-mon[37563]: from='client.? v1:172.21.15.92:0/1864420962' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-11T21:26:17.611 INFO:journalctl@ceph.mgr.y.smithi092.stdout:Mar 11 21:26:17 smithi092 ceph-00db0016-dfed-11ee-95c8-87774f69a715-mgr-y[29801]: 2024-03-11T21:26:17.421+0000 7f2bd61b5200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-03-11T21:26:17.612 INFO:journalctl@ceph.mgr.y.smithi092.stdout:Mar 11 21:26:17 smithi092 ceph-00db0016-dfed-11ee-95c8-87774f69a715-mgr-y[29801]: 2024-03-11T21:26:17.553+0000 7f2bd61b5200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-03-11T21:26:17.980 INFO:journalctl@ceph.mgr.x.smithi097.stdout:Mar 11 21:26:17 smithi097 ceph-00db0016-dfed-11ee-95c8-87774f69a715-mgr-x[35169]: 2024-03-11T21:26:17.712+0000 7f07987a2200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-03-11T21:26:17.980 INFO:journalctl@ceph.mgr.x.smithi097.stdout:Mar 11 21:26:17 smithi097 ceph-00db0016-dfed-11ee-95c8-87774f69a715-mgr-x[35169]: 2024-03-11T21:26:17.801+0000 7f07987a2200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-03-11T21:26:17.980 INFO:journalctl@ceph.mgr.x.smithi097.stdout:Mar 11 21:26:17 smithi097 ceph-00db0016-dfed-11ee-95c8-87774f69a715-mgr-x[35169]: 2024-03-11T21:26:17.996+0000 7f07987a2200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-03-11T21:26:18.079 INFO:journalctl@ceph.mgr.y.smithi092.stdout:Mar 11 21:26:17 smithi092 ceph-00db0016-dfed-11ee-95c8-87774f69a715-mgr-y[29801]: 2024-03-11T21:26:17.888+0000 7f2bd61b5200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-03-11T21:26:18.079 INFO:journalctl@ceph.mgr.y.smithi092.stdout:Mar 11 21:26:17 smithi092 ceph-00db0016-dfed-11ee-95c8-87774f69a715-mgr-y[29801]: 2024-03-11T21:26:17.980+0000 7f2bd61b5200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-03-11T21:26:18.109 INFO:tasks.cephadm.ceph_manager.ceph:all up! 2024-03-11T21:26:18.109 DEBUG:teuthology.orchestra.run.smithi092:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid 00db0016-dfed-11ee-95c8-87774f69a715 -- ceph osd dump --format=json 2024-03-11T21:26:18.327 INFO:journalctl@ceph.mgr.x.smithi097.stdout:Mar 11 21:26:18 smithi097 ceph-00db0016-dfed-11ee-95c8-87774f69a715-mgr-x[35169]: 2024-03-11T21:26:18.182+0000 7f07987a2200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-03-11T21:26:18.359 INFO:teuthology.orchestra.run.smithi092.stderr:Inferring config /var/lib/ceph/00db0016-dfed-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T21:26:18.361 INFO:journalctl@ceph.mgr.y.smithi092.stdout:Mar 11 21:26:18 smithi092 ceph-00db0016-dfed-11ee-95c8-87774f69a715-mgr-y[29801]: 2024-03-11T21:26:18.186+0000 7f2bd61b5200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-03-11T21:26:18.675 INFO:journalctl@ceph.mgr.y.smithi092.stdout:Mar 11 21:26:18 smithi092 ceph-00db0016-dfed-11ee-95c8-87774f69a715-mgr-y[29801]: 2024-03-11T21:26:18.393+0000 7f2bd61b5200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-03-11T21:26:18.709 INFO:journalctl@ceph.mgr.x.smithi097.stdout:Mar 11 21:26:18 smithi097 ceph-00db0016-dfed-11ee-95c8-87774f69a715-mgr-x[35169]: 2024-03-11T21:26:18.465+0000 7f07987a2200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-03-11T21:26:18.940 INFO:journalctl@ceph.mgr.y.smithi092.stdout:Mar 11 21:26:18 smithi092 ceph-00db0016-dfed-11ee-95c8-87774f69a715-mgr-y[29801]: 2024-03-11T21:26:18.691+0000 7f2bd61b5200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-03-11T21:26:18.940 INFO:journalctl@ceph.mgr.y.smithi092.stdout:Mar 11 21:26:18 smithi092 ceph-00db0016-dfed-11ee-95c8-87774f69a715-mgr-y[29801]: 2024-03-11T21:26:18.956+0000 7f2bd61b5200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-03-11T21:26:19.077 INFO:journalctl@ceph.mgr.x.smithi097.stdout:Mar 11 21:26:18 smithi097 ceph-00db0016-dfed-11ee-95c8-87774f69a715-mgr-x[35169]: 2024-03-11T21:26:18.726+0000 7f07987a2200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-03-11T21:26:19.077 INFO:journalctl@ceph.mgr.x.smithi097.stdout:Mar 11 21:26:18 smithi097 ceph-00db0016-dfed-11ee-95c8-87774f69a715-mgr-x[35169]: 2024-03-11T21:26:18.964+0000 7f07987a2200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-03-11T21:26:19.211 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:26:19 smithi092 ceph-mon[29572]: osdmap e86: 8 total, 8 up, 8 in 2024-03-11T21:26:19.212 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:26:19 smithi092 ceph-mon[37563]: osdmap e86: 8 total, 8 up, 8 in 2024-03-11T21:26:19.212 INFO:journalctl@ceph.mgr.y.smithi092.stdout:Mar 11 21:26:19 smithi092 ceph-00db0016-dfed-11ee-95c8-87774f69a715-mgr-y[29801]: 2024-03-11T21:26:19.199+0000 7f2bd61b5200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-03-11T21:26:19.577 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:26:19 smithi097 ceph-mon[33317]: osdmap e86: 8 total, 8 up, 8 in 2024-03-11T21:26:20.077 INFO:journalctl@ceph.mgr.x.smithi097.stdout:Mar 11 21:26:19 smithi097 ceph-00db0016-dfed-11ee-95c8-87774f69a715-mgr-x[35169]: 2024-03-11T21:26:19.764+0000 7f07987a2200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-03-11T21:26:20.183 INFO:journalctl@ceph.mgr.y.smithi092.stdout:Mar 11 21:26:20 smithi092 ceph-00db0016-dfed-11ee-95c8-87774f69a715-mgr-y[29801]: 2024-03-11T21:26:20.019+0000 7f2bd61b5200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-03-11T21:26:20.339 INFO:journalctl@ceph.mgr.x.smithi097.stdout:Mar 11 21:26:20 smithi097 ceph-00db0016-dfed-11ee-95c8-87774f69a715-mgr-x[35169]: 2024-03-11T21:26:20.338+0000 7f07987a2200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-03-11T21:26:20.402 INFO:teuthology.orchestra.run.smithi092.stdout: 2024-03-11T21:26:20.403 INFO:teuthology.orchestra.run.smithi092.stdout:{"epoch":86,"fsid":"00db0016-dfed-11ee-95c8-87774f69a715","created":"2024-03-11T21:20:10.488059+0000","modified":"2024-03-11T21:26:18.113878+0000","last_up_change":"2024-03-11T21:25:26.018427+0000","last_in_change":"2024-03-11T21:25:08.189557+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":18,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":6,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-11T21:23:23.805029+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":7.8899998664855957,"score_stable":7.8899998664855957,"optimal_score":0.37999999523162842,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":".rgw.root","create_time":"2024-03-11T21:25:37.044504+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"68","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rgw":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"datapool","create_time":"2024-03-11T21:25:37.280820+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":3,"pg_placement_num":3,"pg_placement_num_target":3,"pg_num_target":3,"pg_num_pending":3,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"71","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":2,"snap_epoch":71,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_type":"Fair distribution","score_acting":2.6500000953674316,"score_stable":2.6500000953674316,"optimal_score":0.87999999523162842,"raw_score_acting":2.3299999237060547,"raw_score_stable":2.3299999237060547,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"default.rgw.log","create_time":"2024-03-11T21:25:38.850207+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"70","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rgw":{}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"default.rgw.control","create_time":"2024-03-11T21:25:40.794686+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"72","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rgw":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.25,"score_stable":1.25,"optimal_score":1,"raw_score_acting":1.25,"raw_score_stable":1.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":6,"pool_name":"default.rgw.meta","create_time":"2024-03-11T21:25:42.872077+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"74","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4},"application_metadata":{"rgw":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"79f15843-f0ec-44aa-931e-4e07a74d3768","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":72,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.92:6801","nonce":2844100201}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.92:6802","nonce":2844100201}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.92:6804","nonce":2844100201}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.92:6803","nonce":2844100201}]},"public_addr":"172.21.15.92:6801/2844100201","cluster_addr":"172.21.15.92:6802/2844100201","heartbeat_back_addr":"172.21.15.92:6804/2844100201","heartbeat_front_addr":"172.21.15.92:6803/2844100201","state":["exists","up"]},{"osd":1,"uuid":"3a5a0e78-1293-4ace-a93f-69156098571d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":72,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.92:6805","nonce":3577871157}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.92:6806","nonce":3577871157}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.92:6808","nonce":3577871157}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.92:6807","nonce":3577871157}]},"public_addr":"172.21.15.92:6805/3577871157","cluster_addr":"172.21.15.92:6806/3577871157","heartbeat_back_addr":"172.21.15.92:6808/3577871157","heartbeat_front_addr":"172.21.15.92:6807/3577871157","state":["exists","up"]},{"osd":2,"uuid":"f207b743-0df4-40ed-a735-950e272fea60","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.92:6809","nonce":760825152}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.92:6810","nonce":760825152}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.92:6812","nonce":760825152}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.92:6811","nonce":760825152}]},"public_addr":"172.21.15.92:6809/760825152","cluster_addr":"172.21.15.92:6810/760825152","heartbeat_back_addr":"172.21.15.92:6812/760825152","heartbeat_front_addr":"172.21.15.92:6811/760825152","state":["exists","up"]},{"osd":3,"uuid":"bbb6f69a-8f0f-47e6-937a-062e1902ae13","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":26,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.92:6813","nonce":670596264}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.92:6814","nonce":670596264}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.92:6816","nonce":670596264}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.92:6815","nonce":670596264}]},"public_addr":"172.21.15.92:6813/670596264","cluster_addr":"172.21.15.92:6814/670596264","heartbeat_back_addr":"172.21.15.92:6816/670596264","heartbeat_front_addr":"172.21.15.92:6815/670596264","state":["exists","up"]},{"osd":4,"uuid":"00266cbc-d2cf-4dee-8a96-3753ebc95ca8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":32,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.97:6800","nonce":420733620}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.97:6801","nonce":420733620}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.97:6803","nonce":420733620}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.97:6802","nonce":420733620}]},"public_addr":"172.21.15.97:6800/420733620","cluster_addr":"172.21.15.97:6801/420733620","heartbeat_back_addr":"172.21.15.97:6803/420733620","heartbeat_front_addr":"172.21.15.97:6802/420733620","state":["exists","up"]},{"osd":5,"uuid":"593a6f6b-8874-40ae-8848-d51c5723a5e4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":41,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.97:6804","nonce":312056041}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.97:6805","nonce":312056041}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.97:6807","nonce":312056041}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.97:6806","nonce":312056041}]},"public_addr":"172.21.15.97:6804/312056041","cluster_addr":"172.21.15.97:6805/312056041","heartbeat_back_addr":"172.21.15.97:6807/312056041","heartbeat_front_addr":"172.21.15.97:6806/312056041","state":["exists","up"]},{"osd":6,"uuid":"91035ecc-6f1a-4cec-bda6-090af9d53eff","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":50,"up_thru":70,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.97:6808","nonce":178994999}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.97:6809","nonce":178994999}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.97:6811","nonce":178994999}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.97:6810","nonce":178994999}]},"public_addr":"172.21.15.97:6808/178994999","cluster_addr":"172.21.15.97:6809/178994999","heartbeat_back_addr":"172.21.15.97:6811/178994999","heartbeat_front_addr":"172.21.15.97:6810/178994999","state":["exists","up"]},{"osd":7,"uuid":"3eb28c24-664a-4bd6-869d-6bf5db98f52f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":60,"up_thru":72,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.97:6812","nonce":3241560509}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.97:6813","nonce":3241560509}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.97:6815","nonce":3241560509}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.97:6814","nonce":3241560509}]},"public_addr":"172.21.15.97:6812/3241560509","cluster_addr":"172.21.15.97:6813/3241560509","heartbeat_back_addr":"172.21.15.97:6815/3241560509","heartbeat_front_addr":"172.21.15.97:6814/3241560509","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-11T21:26:12.296018+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-11T21:25:11.578251+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-11T21:25:45.729567+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-11T21:25:05.154676+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-11T21:25:31.699231+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-11T21:25:58.278204+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-11T21:26:01.515615+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-11T21:25:24.387220+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.3","mappings":[{"from":7,"to":4}]},{"pgid":"4.f","mappings":[{"from":1,"to":2}]},{"pgid":"4.13","mappings":[{"from":1,"to":2}]},{"pgid":"4.17","mappings":[{"from":1,"to":5}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.92:0/3817549754":"2024-03-12T21:21:09.473916+0000","172.21.15.92:0/2211594569":"2024-03-12T21:21:09.473916+0000","172.21.15.92:0/3569900179":"2024-03-12T21:20:43.677016+0000","172.21.15.92:0/3380359534":"2024-03-12T21:20:43.677016+0000","172.21.15.92:0/2509373130":"2024-03-12T21:20:43.677016+0000","172.21.15.92:6800/1559007462":"2024-03-12T21:20:43.677016+0000","172.21.15.92:6800/1270902819":"2024-03-12T21:21:09.473916+0000","172.21.15.92:0/3562645553":"2024-03-12T21:21:09.473916+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-11T21:26:20.654 INFO:journalctl@ceph.mgr.x.smithi097.stdout:Mar 11 21:26:20 smithi097 ceph-00db0016-dfed-11ee-95c8-87774f69a715-mgr-x[35169]: 2024-03-11T21:26:20.442+0000 7f07987a2200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-03-11T21:26:20.655 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:26:20 smithi097 ceph-mon[33317]: from='client.? v1:172.21.15.92:0/2371711224' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-11T21:26:20.783 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:26:20 smithi092 ceph-mon[29572]: from='client.? v1:172.21.15.92:0/2371711224' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-11T21:26:20.783 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:26:20 smithi092 ceph-mon[37563]: from='client.? v1:172.21.15.92:0/2371711224' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-11T21:26:20.783 INFO:journalctl@ceph.mgr.y.smithi092.stdout:Mar 11 21:26:20 smithi092 ceph-00db0016-dfed-11ee-95c8-87774f69a715-mgr-y[29801]: 2024-03-11T21:26:20.609+0000 7f2bd61b5200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-03-11T21:26:20.784 INFO:journalctl@ceph.mgr.y.smithi092.stdout:Mar 11 21:26:20 smithi092 ceph-00db0016-dfed-11ee-95c8-87774f69a715-mgr-y[29801]: 2024-03-11T21:26:20.709+0000 7f2bd61b5200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-03-11T21:26:21.061 INFO:journalctl@ceph.mgr.x.smithi097.stdout:Mar 11 21:26:20 smithi097 ceph-00db0016-dfed-11ee-95c8-87774f69a715-mgr-x[35169]: 2024-03-11T21:26:20.670+0000 7f07987a2200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-03-11T21:26:21.061 INFO:journalctl@ceph.mgr.x.smithi097.stdout:Mar 11 21:26:20 smithi097 ceph-00db0016-dfed-11ee-95c8-87774f69a715-mgr-x[35169]: [11/Mar/2024:21:26:20] ENGINE Bus STARTING 2024-03-11T21:26:21.061 INFO:journalctl@ceph.mgr.x.smithi097.stdout:Mar 11 21:26:20 smithi097 ceph-00db0016-dfed-11ee-95c8-87774f69a715-mgr-x[35169]: CherryPy Checker: 2024-03-11T21:26:21.061 INFO:journalctl@ceph.mgr.x.smithi097.stdout:Mar 11 21:26:20 smithi097 ceph-00db0016-dfed-11ee-95c8-87774f69a715-mgr-x[35169]: The Application mounted at '' has an empty config. 2024-03-11T21:26:21.062 INFO:journalctl@ceph.mgr.x.smithi097.stdout:Mar 11 21:26:20 smithi097 ceph-00db0016-dfed-11ee-95c8-87774f69a715-mgr-x[35169]: 2024-03-11T21:26:21.062 INFO:journalctl@ceph.mgr.x.smithi097.stdout:Mar 11 21:26:20 smithi097 ceph-00db0016-dfed-11ee-95c8-87774f69a715-mgr-x[35169]: [11/Mar/2024:21:26:20] ENGINE Serving on http://:::9283 2024-03-11T21:26:21.062 INFO:journalctl@ceph.mgr.x.smithi097.stdout:Mar 11 21:26:20 smithi097 ceph-00db0016-dfed-11ee-95c8-87774f69a715-mgr-x[35169]: [11/Mar/2024:21:26:20] ENGINE Bus STARTED 2024-03-11T21:26:21.062 INFO:journalctl@ceph.mgr.y.smithi092.stdout:Mar 11 21:26:20 smithi092 ceph-00db0016-dfed-11ee-95c8-87774f69a715-mgr-y[29801]: 2024-03-11T21:26:20.929+0000 7f2bd61b5200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-03-11T21:26:21.062 INFO:journalctl@ceph.mgr.y.smithi092.stdout:Mar 11 21:26:21 smithi092 ceph-00db0016-dfed-11ee-95c8-87774f69a715-mgr-y[29801]: [11/Mar/2024:21:26:21] ENGINE Bus STARTING 2024-03-11T21:26:21.068 DEBUG:teuthology.orchestra.run.smithi092:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid 00db0016-dfed-11ee-95c8-87774f69a715 -- ceph tell osd.0 flush_pg_stats 2024-03-11T21:26:21.069 DEBUG:teuthology.orchestra.run.smithi092:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid 00db0016-dfed-11ee-95c8-87774f69a715 -- ceph tell osd.1 flush_pg_stats 2024-03-11T21:26:21.069 DEBUG:teuthology.orchestra.run.smithi092:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid 00db0016-dfed-11ee-95c8-87774f69a715 -- ceph tell osd.2 flush_pg_stats 2024-03-11T21:26:21.070 DEBUG:teuthology.orchestra.run.smithi092:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid 00db0016-dfed-11ee-95c8-87774f69a715 -- ceph tell osd.3 flush_pg_stats 2024-03-11T21:26:21.070 DEBUG:teuthology.orchestra.run.smithi092:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid 00db0016-dfed-11ee-95c8-87774f69a715 -- ceph tell osd.4 flush_pg_stats 2024-03-11T21:26:21.071 DEBUG:teuthology.orchestra.run.smithi092:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid 00db0016-dfed-11ee-95c8-87774f69a715 -- ceph tell osd.5 flush_pg_stats 2024-03-11T21:26:21.071 DEBUG:teuthology.orchestra.run.smithi092:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid 00db0016-dfed-11ee-95c8-87774f69a715 -- ceph tell osd.6 flush_pg_stats 2024-03-11T21:26:21.071 DEBUG:teuthology.orchestra.run.smithi092:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid 00db0016-dfed-11ee-95c8-87774f69a715 -- ceph tell osd.7 flush_pg_stats 2024-03-11T21:26:21.361 INFO:journalctl@ceph.mgr.y.smithi092.stdout:Mar 11 21:26:21 smithi092 ceph-00db0016-dfed-11ee-95c8-87774f69a715-mgr-y[29801]: CherryPy Checker: 2024-03-11T21:26:21.361 INFO:journalctl@ceph.mgr.y.smithi092.stdout:Mar 11 21:26:21 smithi092 ceph-00db0016-dfed-11ee-95c8-87774f69a715-mgr-y[29801]: The Application mounted at '' has an empty config. 2024-03-11T21:26:21.361 INFO:journalctl@ceph.mgr.y.smithi092.stdout:Mar 11 21:26:21 smithi092 ceph-00db0016-dfed-11ee-95c8-87774f69a715-mgr-y[29801]: 2024-03-11T21:26:21.362 INFO:journalctl@ceph.mgr.y.smithi092.stdout:Mar 11 21:26:21 smithi092 ceph-00db0016-dfed-11ee-95c8-87774f69a715-mgr-y[29801]: [11/Mar/2024:21:26:21] ENGINE Serving on http://:::9283 2024-03-11T21:26:21.362 INFO:journalctl@ceph.mgr.y.smithi092.stdout:Mar 11 21:26:21 smithi092 ceph-00db0016-dfed-11ee-95c8-87774f69a715-mgr-y[29801]: [11/Mar/2024:21:26:21] ENGINE Bus STARTED 2024-03-11T21:26:21.625 INFO:teuthology.orchestra.run.smithi092.stderr:Inferring config /var/lib/ceph/00db0016-dfed-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T21:26:21.626 INFO:teuthology.orchestra.run.smithi092.stderr:Inferring config /var/lib/ceph/00db0016-dfed-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T21:26:21.628 INFO:teuthology.orchestra.run.smithi092.stderr:Inferring config /var/lib/ceph/00db0016-dfed-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T21:26:21.701 INFO:teuthology.orchestra.run.smithi092.stderr:Inferring config /var/lib/ceph/00db0016-dfed-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T21:26:21.719 INFO:teuthology.orchestra.run.smithi092.stderr:Inferring config /var/lib/ceph/00db0016-dfed-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T21:26:21.719 INFO:teuthology.orchestra.run.smithi092.stderr:Inferring config /var/lib/ceph/00db0016-dfed-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T21:26:21.722 INFO:teuthology.orchestra.run.smithi092.stderr:Inferring config /var/lib/ceph/00db0016-dfed-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T21:26:21.795 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:26:21 smithi092 ceph-mon[29572]: Standby manager daemon x restarted 2024-03-11T21:26:21.795 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:26:21 smithi092 ceph-mon[29572]: Standby manager daemon x started 2024-03-11T21:26:21.795 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:26:21 smithi092 ceph-mon[29572]: from='mgr.? v1:172.21.15.97:0/3034862845' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/crt"}]: dispatch 2024-03-11T21:26:21.795 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:26:21 smithi092 ceph-mon[29572]: from='mgr.? v1:172.21.15.97:0/3034862845' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-03-11T21:26:21.796 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:26:21 smithi092 ceph-mon[29572]: from='mgr.? v1:172.21.15.97:0/3034862845' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/key"}]: dispatch 2024-03-11T21:26:21.796 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:26:21 smithi092 ceph-mon[29572]: from='mgr.? v1:172.21.15.97:0/3034862845' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-03-11T21:26:21.796 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:26:21 smithi092 ceph-mon[29572]: Active manager daemon y restarted 2024-03-11T21:26:21.796 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:26:21 smithi092 ceph-mon[29572]: Activating manager daemon y 2024-03-11T21:26:21.796 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:26:21 smithi092 ceph-mon[29572]: osdmap e87: 8 total, 8 up, 8 in 2024-03-11T21:26:21.796 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:26:21 smithi092 ceph-mon[29572]: mgrmap e18: y(active, starting, since 0.00843865s), standbys: x 2024-03-11T21:26:21.796 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:26:21 smithi092 ceph-mon[29572]: from='mgr.14607 v1:172.21.15.92:0/1702717597' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-11T21:26:21.796 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:26:21 smithi092 ceph-mon[29572]: from='mgr.14607 v1:172.21.15.92:0/1702717597' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-11T21:26:21.796 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:26:21 smithi092 ceph-mon[29572]: from='mgr.14607 v1:172.21.15.92:0/1702717597' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-11T21:26:21.796 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:26:21 smithi092 ceph-mon[29572]: from='mgr.14607 v1:172.21.15.92:0/1702717597' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-03-11T21:26:21.796 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:26:21 smithi092 ceph-mon[29572]: from='mgr.14607 v1:172.21.15.92:0/1702717597' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-03-11T21:26:21.796 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:26:21 smithi092 ceph-mon[29572]: from='mgr.14607 v1:172.21.15.92:0/1702717597' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-11T21:26:21.796 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:26:21 smithi092 ceph-mon[29572]: from='mgr.14607 v1:172.21.15.92:0/1702717597' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-11T21:26:21.796 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:26:21 smithi092 ceph-mon[29572]: from='mgr.14607 v1:172.21.15.92:0/1702717597' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-11T21:26:21.797 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:26:21 smithi092 ceph-mon[29572]: from='mgr.14607 v1:172.21.15.92:0/1702717597' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-11T21:26:21.797 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:26:21 smithi092 ceph-mon[29572]: from='mgr.14607 v1:172.21.15.92:0/1702717597' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-11T21:26:21.797 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:26:21 smithi092 ceph-mon[29572]: from='mgr.14607 v1:172.21.15.92:0/1702717597' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-11T21:26:21.797 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:26:21 smithi092 ceph-mon[29572]: from='mgr.14607 v1:172.21.15.92:0/1702717597' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-11T21:26:21.797 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:26:21 smithi092 ceph-mon[29572]: from='mgr.14607 v1:172.21.15.92:0/1702717597' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-11T21:26:21.797 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:26:21 smithi092 ceph-mon[29572]: from='mgr.14607 v1:172.21.15.92:0/1702717597' entity='mgr.y' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-03-11T21:26:21.797 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:26:21 smithi092 ceph-mon[29572]: from='mgr.14607 v1:172.21.15.92:0/1702717597' entity='mgr.y' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-03-11T21:26:21.797 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:26:21 smithi092 ceph-mon[29572]: from='mgr.14607 v1:172.21.15.92:0/1702717597' entity='mgr.y' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-03-11T21:26:21.797 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:26:21 smithi092 ceph-mon[29572]: Manager daemon y is now available 2024-03-11T21:26:21.797 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:26:21 smithi092 ceph-mon[29572]: from='mgr.14607 v1:172.21.15.92:0/1702717597' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T21:26:21.797 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:26:21 smithi092 ceph-mon[29572]: from='mgr.14607 v1:172.21.15.92:0/1702717597' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T21:26:21.797 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:26:21 smithi092 ceph-mon[29572]: from='mgr.14607 ' entity='mgr.y' 2024-03-11T21:26:21.797 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:26:21 smithi092 ceph-mon[29572]: from='mgr.14607 v1:172.21.15.92:0/1702717597' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T21:26:21.797 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:26:21 smithi092 ceph-mon[29572]: from='mgr.14607 v1:172.21.15.92:0/1702717597' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-03-11T21:26:21.798 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:26:21 smithi092 ceph-mon[29572]: from='mgr.14607 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-03-11T21:26:21.798 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:26:21 smithi092 ceph-mon[29572]: from='mgr.14607 v1:172.21.15.92:0/1702717597' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-03-11T21:26:21.798 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:26:21 smithi092 ceph-mon[29572]: from='mgr.14607 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-03-11T21:26:21.798 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:26:21 smithi092 ceph-mon[37563]: Standby manager daemon x restarted 2024-03-11T21:26:21.798 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:26:21 smithi092 ceph-mon[37563]: Standby manager daemon x started 2024-03-11T21:26:21.798 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:26:21 smithi092 ceph-mon[37563]: from='mgr.? v1:172.21.15.97:0/3034862845' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/crt"}]: dispatch 2024-03-11T21:26:21.798 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:26:21 smithi092 ceph-mon[37563]: from='mgr.? v1:172.21.15.97:0/3034862845' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-03-11T21:26:21.799 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:26:21 smithi092 ceph-mon[37563]: from='mgr.? v1:172.21.15.97:0/3034862845' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/key"}]: dispatch 2024-03-11T21:26:21.799 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:26:21 smithi092 ceph-mon[37563]: from='mgr.? v1:172.21.15.97:0/3034862845' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-03-11T21:26:21.799 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:26:21 smithi092 ceph-mon[37563]: Active manager daemon y restarted 2024-03-11T21:26:21.799 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:26:21 smithi092 ceph-mon[37563]: Activating manager daemon y 2024-03-11T21:26:21.799 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:26:21 smithi092 ceph-mon[37563]: osdmap e87: 8 total, 8 up, 8 in 2024-03-11T21:26:21.799 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:26:21 smithi092 ceph-mon[37563]: mgrmap e18: y(active, starting, since 0.00843865s), standbys: x 2024-03-11T21:26:21.799 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:26:21 smithi092 ceph-mon[37563]: from='mgr.14607 v1:172.21.15.92:0/1702717597' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-11T21:26:21.799 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:26:21 smithi092 ceph-mon[37563]: from='mgr.14607 v1:172.21.15.92:0/1702717597' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-11T21:26:21.799 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:26:21 smithi092 ceph-mon[37563]: from='mgr.14607 v1:172.21.15.92:0/1702717597' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-11T21:26:21.799 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:26:21 smithi092 ceph-mon[37563]: from='mgr.14607 v1:172.21.15.92:0/1702717597' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-03-11T21:26:21.799 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:26:21 smithi092 ceph-mon[37563]: from='mgr.14607 v1:172.21.15.92:0/1702717597' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-03-11T21:26:21.799 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:26:21 smithi092 ceph-mon[37563]: from='mgr.14607 v1:172.21.15.92:0/1702717597' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-11T21:26:21.799 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:26:21 smithi092 ceph-mon[37563]: from='mgr.14607 v1:172.21.15.92:0/1702717597' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-11T21:26:21.799 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:26:21 smithi092 ceph-mon[37563]: from='mgr.14607 v1:172.21.15.92:0/1702717597' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-11T21:26:21.800 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:26:21 smithi092 ceph-mon[37563]: from='mgr.14607 v1:172.21.15.92:0/1702717597' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-11T21:26:21.800 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:26:21 smithi092 ceph-mon[37563]: from='mgr.14607 v1:172.21.15.92:0/1702717597' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-11T21:26:21.800 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:26:21 smithi092 ceph-mon[37563]: from='mgr.14607 v1:172.21.15.92:0/1702717597' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-11T21:26:21.800 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:26:21 smithi092 ceph-mon[37563]: from='mgr.14607 v1:172.21.15.92:0/1702717597' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-11T21:26:21.800 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:26:21 smithi092 ceph-mon[37563]: from='mgr.14607 v1:172.21.15.92:0/1702717597' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-11T21:26:21.800 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:26:21 smithi092 ceph-mon[37563]: from='mgr.14607 v1:172.21.15.92:0/1702717597' entity='mgr.y' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-03-11T21:26:21.800 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:26:21 smithi092 ceph-mon[37563]: from='mgr.14607 v1:172.21.15.92:0/1702717597' entity='mgr.y' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-03-11T21:26:21.800 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:26:21 smithi092 ceph-mon[37563]: from='mgr.14607 v1:172.21.15.92:0/1702717597' entity='mgr.y' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-03-11T21:26:21.800 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:26:21 smithi092 ceph-mon[37563]: Manager daemon y is now available 2024-03-11T21:26:21.800 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:26:21 smithi092 ceph-mon[37563]: from='mgr.14607 v1:172.21.15.92:0/1702717597' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T21:26:21.800 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:26:21 smithi092 ceph-mon[37563]: from='mgr.14607 v1:172.21.15.92:0/1702717597' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T21:26:21.800 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:26:21 smithi092 ceph-mon[37563]: from='mgr.14607 ' entity='mgr.y' 2024-03-11T21:26:21.800 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:26:21 smithi092 ceph-mon[37563]: from='mgr.14607 v1:172.21.15.92:0/1702717597' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T21:26:21.800 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:26:21 smithi092 ceph-mon[37563]: from='mgr.14607 v1:172.21.15.92:0/1702717597' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-03-11T21:26:21.801 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:26:21 smithi092 ceph-mon[37563]: from='mgr.14607 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-03-11T21:26:21.801 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:26:21 smithi092 ceph-mon[37563]: from='mgr.14607 v1:172.21.15.92:0/1702717597' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-03-11T21:26:21.801 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:26:21 smithi092 ceph-mon[37563]: from='mgr.14607 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-03-11T21:26:21.828 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:26:21 smithi097 ceph-mon[33317]: Standby manager daemon x restarted 2024-03-11T21:26:21.828 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:26:21 smithi097 ceph-mon[33317]: Standby manager daemon x started 2024-03-11T21:26:21.828 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:26:21 smithi097 ceph-mon[33317]: from='mgr.? v1:172.21.15.97:0/3034862845' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/crt"}]: dispatch 2024-03-11T21:26:21.828 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:26:21 smithi097 ceph-mon[33317]: from='mgr.? v1:172.21.15.97:0/3034862845' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-03-11T21:26:21.828 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:26:21 smithi097 ceph-mon[33317]: from='mgr.? v1:172.21.15.97:0/3034862845' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/key"}]: dispatch 2024-03-11T21:26:21.828 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:26:21 smithi097 ceph-mon[33317]: from='mgr.? v1:172.21.15.97:0/3034862845' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-03-11T21:26:21.828 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:26:21 smithi097 ceph-mon[33317]: Active manager daemon y restarted 2024-03-11T21:26:21.828 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:26:21 smithi097 ceph-mon[33317]: Activating manager daemon y 2024-03-11T21:26:21.828 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:26:21 smithi097 ceph-mon[33317]: osdmap e87: 8 total, 8 up, 8 in 2024-03-11T21:26:21.829 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:26:21 smithi097 ceph-mon[33317]: mgrmap e18: y(active, starting, since 0.00843865s), standbys: x 2024-03-11T21:26:21.829 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:26:21 smithi097 ceph-mon[33317]: from='mgr.14607 v1:172.21.15.92:0/1702717597' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-11T21:26:21.829 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:26:21 smithi097 ceph-mon[33317]: from='mgr.14607 v1:172.21.15.92:0/1702717597' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-11T21:26:21.829 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:26:21 smithi097 ceph-mon[33317]: from='mgr.14607 v1:172.21.15.92:0/1702717597' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-11T21:26:21.829 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:26:21 smithi097 ceph-mon[33317]: from='mgr.14607 v1:172.21.15.92:0/1702717597' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-03-11T21:26:21.829 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:26:21 smithi097 ceph-mon[33317]: from='mgr.14607 v1:172.21.15.92:0/1702717597' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-03-11T21:26:21.829 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:26:21 smithi097 ceph-mon[33317]: from='mgr.14607 v1:172.21.15.92:0/1702717597' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-11T21:26:21.829 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:26:21 smithi097 ceph-mon[33317]: from='mgr.14607 v1:172.21.15.92:0/1702717597' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-11T21:26:21.829 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:26:21 smithi097 ceph-mon[33317]: from='mgr.14607 v1:172.21.15.92:0/1702717597' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-11T21:26:21.829 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:26:21 smithi097 ceph-mon[33317]: from='mgr.14607 v1:172.21.15.92:0/1702717597' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-11T21:26:21.829 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:26:21 smithi097 ceph-mon[33317]: from='mgr.14607 v1:172.21.15.92:0/1702717597' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-11T21:26:21.829 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:26:21 smithi097 ceph-mon[33317]: from='mgr.14607 v1:172.21.15.92:0/1702717597' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-11T21:26:21.829 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:26:21 smithi097 ceph-mon[33317]: from='mgr.14607 v1:172.21.15.92:0/1702717597' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-11T21:26:21.829 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:26:21 smithi097 ceph-mon[33317]: from='mgr.14607 v1:172.21.15.92:0/1702717597' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-11T21:26:21.830 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:26:21 smithi097 ceph-mon[33317]: from='mgr.14607 v1:172.21.15.92:0/1702717597' entity='mgr.y' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-03-11T21:26:21.830 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:26:21 smithi097 ceph-mon[33317]: from='mgr.14607 v1:172.21.15.92:0/1702717597' entity='mgr.y' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-03-11T21:26:21.830 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:26:21 smithi097 ceph-mon[33317]: from='mgr.14607 v1:172.21.15.92:0/1702717597' entity='mgr.y' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-03-11T21:26:21.830 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:26:21 smithi097 ceph-mon[33317]: Manager daemon y is now available 2024-03-11T21:26:21.830 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:26:21 smithi097 ceph-mon[33317]: from='mgr.14607 v1:172.21.15.92:0/1702717597' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T21:26:21.830 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:26:21 smithi097 ceph-mon[33317]: from='mgr.14607 v1:172.21.15.92:0/1702717597' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T21:26:21.830 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:26:21 smithi097 ceph-mon[33317]: from='mgr.14607 ' entity='mgr.y' 2024-03-11T21:26:21.830 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:26:21 smithi097 ceph-mon[33317]: from='mgr.14607 v1:172.21.15.92:0/1702717597' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T21:26:21.830 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:26:21 smithi097 ceph-mon[33317]: from='mgr.14607 v1:172.21.15.92:0/1702717597' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-03-11T21:26:21.830 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:26:21 smithi097 ceph-mon[33317]: from='mgr.14607 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-03-11T21:26:21.830 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:26:21 smithi097 ceph-mon[33317]: from='mgr.14607 v1:172.21.15.92:0/1702717597' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-03-11T21:26:21.830 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:26:21 smithi097 ceph-mon[33317]: from='mgr.14607 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-03-11T21:26:21.949 INFO:teuthology.orchestra.run.smithi092.stderr:Inferring config /var/lib/ceph/00db0016-dfed-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T21:26:22.305 INFO:journalctl@ceph.iscsi.iscsi.a.smithi097.stdout:Mar 11 21:26:21 smithi097 ceph-00db0016-dfed-11ee-95c8-87774f69a715-iscsi-iscsi-a[60943]: debug there is no tcmu-runner data available 2024-03-11T21:26:22.965 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:26:22 smithi092 ceph-mon[29572]: mgrmap e19: y(active, since 1.01317s), standbys: x 2024-03-11T21:26:22.966 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:26:22 smithi092 ceph-mon[29572]: [11/Mar/2024:21:26:22] ENGINE Bus STARTING 2024-03-11T21:26:22.966 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:26:22 smithi092 ceph-mon[29572]: [11/Mar/2024:21:26:22] ENGINE Serving on https://172.21.15.92:7150 2024-03-11T21:26:22.966 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:26:22 smithi092 ceph-mon[29572]: [11/Mar/2024:21:26:22] ENGINE Serving on http://172.21.15.92:8765 2024-03-11T21:26:22.966 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:26:22 smithi092 ceph-mon[29572]: [11/Mar/2024:21:26:22] ENGINE Bus STARTED 2024-03-11T21:26:22.966 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:26:22 smithi092 ceph-mon[37563]: mgrmap e19: y(active, since 1.01317s), standbys: x 2024-03-11T21:26:22.966 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:26:22 smithi092 ceph-mon[37563]: [11/Mar/2024:21:26:22] ENGINE Bus STARTING 2024-03-11T21:26:22.967 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:26:22 smithi092 ceph-mon[37563]: [11/Mar/2024:21:26:22] ENGINE Serving on https://172.21.15.92:7150 2024-03-11T21:26:22.967 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:26:22 smithi092 ceph-mon[37563]: [11/Mar/2024:21:26:22] ENGINE Serving on http://172.21.15.92:8765 2024-03-11T21:26:22.967 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:26:22 smithi092 ceph-mon[37563]: [11/Mar/2024:21:26:22] ENGINE Bus STARTED 2024-03-11T21:26:23.199 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:26:22 smithi097 ceph-mon[33317]: mgrmap e19: y(active, since 1.01317s), standbys: x 2024-03-11T21:26:23.199 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:26:22 smithi097 ceph-mon[33317]: [11/Mar/2024:21:26:22] ENGINE Bus STARTING 2024-03-11T21:26:23.199 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:26:22 smithi097 ceph-mon[33317]: [11/Mar/2024:21:26:22] ENGINE Serving on https://172.21.15.92:7150 2024-03-11T21:26:23.199 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:26:22 smithi097 ceph-mon[33317]: [11/Mar/2024:21:26:22] ENGINE Serving on http://172.21.15.92:8765 2024-03-11T21:26:23.199 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:26:22 smithi097 ceph-mon[33317]: [11/Mar/2024:21:26:22] ENGINE Bus STARTED 2024-03-11T21:26:24.269 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:26:24 smithi092 ceph-mon[29572]: pgmap v4: 132 pgs: 132 active+clean; 582 KiB data, 227 MiB used, 715 GiB / 715 GiB avail 2024-03-11T21:26:24.269 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:26:24 smithi092 ceph-mon[29572]: osdmap e88: 8 total, 8 up, 8 in 2024-03-11T21:26:24.269 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:26:24 smithi092 ceph-mon[29572]: mgrmap e20: y(active, since 3s), standbys: x 2024-03-11T21:26:24.270 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:26:24 smithi092 ceph-mon[37563]: pgmap v4: 132 pgs: 132 active+clean; 582 KiB data, 227 MiB used, 715 GiB / 715 GiB avail 2024-03-11T21:26:24.270 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:26:24 smithi092 ceph-mon[37563]: osdmap e88: 8 total, 8 up, 8 in 2024-03-11T21:26:24.270 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:26:24 smithi092 ceph-mon[37563]: mgrmap e20: y(active, since 3s), standbys: x 2024-03-11T21:26:24.577 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:26:24 smithi097 ceph-mon[33317]: pgmap v4: 132 pgs: 132 active+clean; 582 KiB data, 227 MiB used, 715 GiB / 715 GiB avail 2024-03-11T21:26:24.577 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:26:24 smithi097 ceph-mon[33317]: osdmap e88: 8 total, 8 up, 8 in 2024-03-11T21:26:24.577 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:26:24 smithi097 ceph-mon[33317]: mgrmap e20: y(active, since 3s), standbys: x 2024-03-11T21:26:25.577 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:26:25 smithi097 ceph-mon[33317]: from='mgr.14607 ' entity='mgr.y' 2024-03-11T21:26:25.577 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:26:25 smithi097 ceph-mon[33317]: from='mgr.14607 ' entity='mgr.y' 2024-03-11T21:26:25.611 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:26:25 smithi092 ceph-mon[37563]: from='mgr.14607 ' entity='mgr.y' 2024-03-11T21:26:25.611 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:26:25 smithi092 ceph-mon[37563]: from='mgr.14607 ' entity='mgr.y' 2024-03-11T21:26:25.611 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:26:25 smithi092 ceph-mon[29572]: from='mgr.14607 ' entity='mgr.y' 2024-03-11T21:26:25.611 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:26:25 smithi092 ceph-mon[29572]: from='mgr.14607 ' entity='mgr.y' 2024-03-11T21:26:26.612 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:26:26 smithi092 ceph-mon[29572]: pgmap v6: 132 pgs: 132 active+clean; 582 KiB data, 227 MiB used, 715 GiB / 715 GiB avail 2024-03-11T21:26:26.612 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:26:26 smithi092 ceph-mon[29572]: mgrmap e21: y(active, since 4s), standbys: x 2024-03-11T21:26:26.612 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:26:26 smithi092 ceph-mon[29572]: from='mgr.14607 ' entity='mgr.y' 2024-03-11T21:26:26.612 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:26:26 smithi092 ceph-mon[29572]: from='mgr.14607 ' entity='mgr.y' 2024-03-11T21:26:26.612 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:26:26 smithi092 ceph-mon[29572]: from='mgr.14607 v1:172.21.15.92:0/1702717597' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-11T21:26:26.612 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:26:26 smithi092 ceph-mon[29572]: from='mgr.14607 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-11T21:26:26.612 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:26:26 smithi092 ceph-mon[29572]: from='mgr.14607 v1:172.21.15.92:0/1702717597' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-11T21:26:26.612 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:26:26 smithi092 ceph-mon[29572]: from='mgr.14607 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-11T21:26:26.612 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:26:26 smithi092 ceph-mon[29572]: from='mgr.14607 v1:172.21.15.92:0/1702717597' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-03-11T21:26:26.612 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:26:26 smithi092 ceph-mon[29572]: from='mgr.14607 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-03-11T21:26:26.612 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:26:26 smithi092 ceph-mon[29572]: from='mgr.14607 v1:172.21.15.92:0/1702717597' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-03-11T21:26:26.612 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:26:26 smithi092 ceph-mon[29572]: from='mgr.14607 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-03-11T21:26:26.612 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:26:26 smithi092 ceph-mon[29572]: Adjusting osd_memory_target on smithi097 to 3760M 2024-03-11T21:26:26.613 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:26:26 smithi092 ceph-mon[29572]: from='mgr.14607 ' entity='mgr.y' 2024-03-11T21:26:26.613 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:26:26 smithi092 ceph-mon[37563]: pgmap v6: 132 pgs: 132 active+clean; 582 KiB data, 227 MiB used, 715 GiB / 715 GiB avail 2024-03-11T21:26:26.613 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:26:26 smithi092 ceph-mon[37563]: mgrmap e21: y(active, since 4s), standbys: x 2024-03-11T21:26:26.613 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:26:26 smithi092 ceph-mon[37563]: from='mgr.14607 ' entity='mgr.y' 2024-03-11T21:26:26.613 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:26:26 smithi092 ceph-mon[37563]: from='mgr.14607 ' entity='mgr.y' 2024-03-11T21:26:26.613 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:26:26 smithi092 ceph-mon[37563]: from='mgr.14607 v1:172.21.15.92:0/1702717597' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-11T21:26:26.613 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:26:26 smithi092 ceph-mon[37563]: from='mgr.14607 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-11T21:26:26.613 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:26:26 smithi092 ceph-mon[37563]: from='mgr.14607 v1:172.21.15.92:0/1702717597' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-11T21:26:26.613 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:26:26 smithi092 ceph-mon[37563]: from='mgr.14607 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-11T21:26:26.613 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:26:26 smithi092 ceph-mon[37563]: from='mgr.14607 v1:172.21.15.92:0/1702717597' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-03-11T21:26:26.613 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:26:26 smithi092 ceph-mon[37563]: from='mgr.14607 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-03-11T21:26:26.613 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:26:26 smithi092 ceph-mon[37563]: from='mgr.14607 v1:172.21.15.92:0/1702717597' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-03-11T21:26:26.613 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:26:26 smithi092 ceph-mon[37563]: from='mgr.14607 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-03-11T21:26:26.613 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:26:26 smithi092 ceph-mon[37563]: Adjusting osd_memory_target on smithi097 to 3760M 2024-03-11T21:26:26.614 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:26:26 smithi092 ceph-mon[37563]: from='mgr.14607 ' entity='mgr.y' 2024-03-11T21:26:26.827 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:26:26 smithi097 ceph-mon[33317]: pgmap v6: 132 pgs: 132 active+clean; 582 KiB data, 227 MiB used, 715 GiB / 715 GiB avail 2024-03-11T21:26:26.827 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:26:26 smithi097 ceph-mon[33317]: mgrmap e21: y(active, since 4s), standbys: x 2024-03-11T21:26:26.827 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:26:26 smithi097 ceph-mon[33317]: from='mgr.14607 ' entity='mgr.y' 2024-03-11T21:26:26.827 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:26:26 smithi097 ceph-mon[33317]: from='mgr.14607 ' entity='mgr.y' 2024-03-11T21:26:26.827 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:26:26 smithi097 ceph-mon[33317]: from='mgr.14607 v1:172.21.15.92:0/1702717597' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-11T21:26:26.828 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:26:26 smithi097 ceph-mon[33317]: from='mgr.14607 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-11T21:26:26.828 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:26:26 smithi097 ceph-mon[33317]: from='mgr.14607 v1:172.21.15.92:0/1702717597' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-11T21:26:26.828 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:26:26 smithi097 ceph-mon[33317]: from='mgr.14607 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-11T21:26:26.828 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:26:26 smithi097 ceph-mon[33317]: from='mgr.14607 v1:172.21.15.92:0/1702717597' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-03-11T21:26:26.828 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:26:26 smithi097 ceph-mon[33317]: from='mgr.14607 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-03-11T21:26:26.828 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:26:26 smithi097 ceph-mon[33317]: from='mgr.14607 v1:172.21.15.92:0/1702717597' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-03-11T21:26:26.828 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:26:26 smithi097 ceph-mon[33317]: from='mgr.14607 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-03-11T21:26:26.828 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:26:26 smithi097 ceph-mon[33317]: Adjusting osd_memory_target on smithi097 to 3760M 2024-03-11T21:26:26.828 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:26:26 smithi097 ceph-mon[33317]: from='mgr.14607 ' entity='mgr.y' 2024-03-11T21:26:28.111 INFO:journalctl@ceph.mgr.y.smithi092.stdout:Mar 11 21:26:27 smithi092 ceph-00db0016-dfed-11ee-95c8-87774f69a715-mgr-y[29801]: ::ffff:172.21.15.97 - - [11/Mar/2024:21:26:27] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-11T21:26:28.611 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:26:28 smithi092 ceph-mon[29572]: pgmap v7: 132 pgs: 132 active+clean; 582 KiB data, 227 MiB used, 715 GiB / 715 GiB avail; 41 KiB/s rd, 0 B/s wr, 23 op/s 2024-03-11T21:26:28.611 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:26:28 smithi092 ceph-mon[37563]: pgmap v7: 132 pgs: 132 active+clean; 582 KiB data, 227 MiB used, 715 GiB / 715 GiB avail; 41 KiB/s rd, 0 B/s wr, 23 op/s 2024-03-11T21:26:28.787 INFO:teuthology.orchestra.run.smithi092.stdout:34359738417 2024-03-11T21:26:28.788 DEBUG:teuthology.orchestra.run.smithi092:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid 00db0016-dfed-11ee-95c8-87774f69a715 -- ceph osd last-stat-seq osd.0 2024-03-11T21:26:28.827 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:26:28 smithi097 ceph-mon[33317]: pgmap v7: 132 pgs: 132 active+clean; 582 KiB data, 227 MiB used, 715 GiB / 715 GiB avail; 41 KiB/s rd, 0 B/s wr, 23 op/s 2024-03-11T21:26:30.435 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:26:30 smithi092 ceph-mon[29572]: pgmap v8: 132 pgs: 132 active+clean; 582 KiB data, 227 MiB used, 715 GiB / 715 GiB avail; 30 KiB/s rd, 0 B/s wr, 17 op/s 2024-03-11T21:26:30.435 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:26:30 smithi092 ceph-mon[37563]: pgmap v8: 132 pgs: 132 active+clean; 582 KiB data, 227 MiB used, 715 GiB / 715 GiB avail; 30 KiB/s rd, 0 B/s wr, 17 op/s 2024-03-11T21:26:30.779 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:26:30 smithi097 ceph-mon[33317]: pgmap v8: 132 pgs: 132 active+clean; 582 KiB data, 227 MiB used, 715 GiB / 715 GiB avail; 30 KiB/s rd, 0 B/s wr, 17 op/s 2024-03-11T21:26:31.262 INFO:teuthology.orchestra.run.smithi092.stdout:257698037772 2024-03-11T21:26:31.262 DEBUG:teuthology.orchestra.run.smithi092:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid 00db0016-dfed-11ee-95c8-87774f69a715 -- ceph osd last-stat-seq osd.7 2024-03-11T21:26:31.611 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:26:31 smithi092 ceph-mon[29572]: purged_snaps scrub starts 2024-03-11T21:26:31.611 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:26:31 smithi092 ceph-mon[29572]: purged_snaps scrub ok 2024-03-11T21:26:31.612 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:26:31 smithi092 ceph-mon[37563]: purged_snaps scrub starts 2024-03-11T21:26:31.612 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:26:31 smithi092 ceph-mon[37563]: purged_snaps scrub ok 2024-03-11T21:26:31.827 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:26:31 smithi097 ceph-mon[33317]: purged_snaps scrub starts 2024-03-11T21:26:31.827 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:26:31 smithi097 ceph-mon[33317]: purged_snaps scrub ok 2024-03-11T21:26:32.327 INFO:journalctl@ceph.iscsi.iscsi.a.smithi097.stdout:Mar 11 21:26:31 smithi097 ceph-00db0016-dfed-11ee-95c8-87774f69a715-iscsi-iscsi-a[60943]: debug there is no tcmu-runner data available 2024-03-11T21:26:32.611 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:26:32 smithi092 ceph-mon[29572]: pgmap v9: 132 pgs: 132 active+clean; 582 KiB data, 227 MiB used, 715 GiB / 715 GiB avail; 23 KiB/s rd, 0 B/s wr, 13 op/s 2024-03-11T21:26:32.611 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:26:32 smithi092 ceph-mon[29572]: osdmap e89: 8 total, 8 up, 8 in 2024-03-11T21:26:32.611 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:26:32 smithi092 ceph-mon[37563]: pgmap v9: 132 pgs: 132 active+clean; 582 KiB data, 227 MiB used, 715 GiB / 715 GiB avail; 23 KiB/s rd, 0 B/s wr, 13 op/s 2024-03-11T21:26:32.612 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:26:32 smithi092 ceph-mon[37563]: osdmap e89: 8 total, 8 up, 8 in 2024-03-11T21:26:32.827 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:26:32 smithi097 ceph-mon[33317]: pgmap v9: 132 pgs: 132 active+clean; 582 KiB data, 227 MiB used, 715 GiB / 715 GiB avail; 23 KiB/s rd, 0 B/s wr, 13 op/s 2024-03-11T21:26:32.827 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:26:32 smithi097 ceph-mon[33317]: osdmap e89: 8 total, 8 up, 8 in 2024-03-11T21:26:33.611 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:26:33 smithi092 ceph-mon[29572]: from='client.24469 v1:172.21.15.97:0/404208310' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:26:33.611 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:26:33 smithi092 ceph-mon[29572]: osdmap e90: 8 total, 8 up, 8 in 2024-03-11T21:26:33.611 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:26:33 smithi092 ceph-mon[37563]: from='client.24469 v1:172.21.15.97:0/404208310' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:26:33.611 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:26:33 smithi092 ceph-mon[37563]: osdmap e90: 8 total, 8 up, 8 in 2024-03-11T21:26:33.827 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:26:33 smithi097 ceph-mon[33317]: from='client.24469 v1:172.21.15.97:0/404208310' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:26:33.827 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:26:33 smithi097 ceph-mon[33317]: osdmap e90: 8 total, 8 up, 8 in 2024-03-11T21:26:34.273 INFO:teuthology.orchestra.run.smithi092.stdout:77309411367 2024-03-11T21:26:34.274 DEBUG:teuthology.orchestra.run.smithi092:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid 00db0016-dfed-11ee-95c8-87774f69a715 -- ceph osd last-stat-seq osd.2 2024-03-11T21:26:34.582 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:26:34 smithi092 ceph-mon[29572]: pgmap v11: 132 pgs: 132 active+clean; 582 KiB data, 227 MiB used, 715 GiB / 715 GiB avail; 22 KiB/s rd, 0 B/s wr, 12 op/s 2024-03-11T21:26:34.583 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:26:34 smithi092 ceph-mon[37563]: pgmap v11: 132 pgs: 132 active+clean; 582 KiB data, 227 MiB used, 715 GiB / 715 GiB avail; 22 KiB/s rd, 0 B/s wr, 12 op/s 2024-03-11T21:26:34.827 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:26:34 smithi097 ceph-mon[33317]: pgmap v11: 132 pgs: 132 active+clean; 582 KiB data, 227 MiB used, 715 GiB / 715 GiB avail; 22 KiB/s rd, 0 B/s wr, 12 op/s 2024-03-11T21:26:34.942 INFO:teuthology.orchestra.run.smithi092.stdout:176093659160 2024-03-11T21:26:34.943 DEBUG:teuthology.orchestra.run.smithi092:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid 00db0016-dfed-11ee-95c8-87774f69a715 -- ceph osd last-stat-seq osd.5 2024-03-11T21:26:35.437 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:26:35 smithi092 ceph-mon[29572]: purged_snaps scrub starts 2024-03-11T21:26:35.438 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:26:35 smithi092 ceph-mon[29572]: purged_snaps scrub ok 2024-03-11T21:26:35.438 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:26:35 smithi092 ceph-mon[29572]: purged_snaps scrub starts 2024-03-11T21:26:35.438 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:26:35 smithi092 ceph-mon[29572]: purged_snaps scrub ok 2024-03-11T21:26:35.438 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:26:35 smithi092 ceph-mon[37563]: purged_snaps scrub starts 2024-03-11T21:26:35.438 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:26:35 smithi092 ceph-mon[37563]: purged_snaps scrub ok 2024-03-11T21:26:35.438 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:26:35 smithi092 ceph-mon[37563]: purged_snaps scrub starts 2024-03-11T21:26:35.438 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:26:35 smithi092 ceph-mon[37563]: purged_snaps scrub ok 2024-03-11T21:26:35.827 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:26:35 smithi097 ceph-mon[33317]: purged_snaps scrub starts 2024-03-11T21:26:35.827 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:26:35 smithi097 ceph-mon[33317]: purged_snaps scrub ok 2024-03-11T21:26:35.827 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:26:35 smithi097 ceph-mon[33317]: purged_snaps scrub starts 2024-03-11T21:26:35.827 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:26:35 smithi097 ceph-mon[33317]: purged_snaps scrub ok 2024-03-11T21:26:36.597 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:26:36 smithi092 ceph-mon[29572]: pgmap v13: 132 pgs: 132 active+clean; 582 KiB data, 227 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T21:26:36.597 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:26:36 smithi092 ceph-mon[29572]: osdmap e91: 8 total, 8 up, 8 in 2024-03-11T21:26:36.597 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:26:36 smithi092 ceph-mon[29572]: from='mgr.14607 v1:172.21.15.92:0/1702717597' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T21:26:36.597 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:26:36 smithi092 ceph-mon[37563]: pgmap v13: 132 pgs: 132 active+clean; 582 KiB data, 227 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T21:26:36.597 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:26:36 smithi092 ceph-mon[37563]: osdmap e91: 8 total, 8 up, 8 in 2024-03-11T21:26:36.598 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:26:36 smithi092 ceph-mon[37563]: from='mgr.14607 v1:172.21.15.92:0/1702717597' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T21:26:36.827 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:26:36 smithi097 ceph-mon[33317]: pgmap v13: 132 pgs: 132 active+clean; 582 KiB data, 227 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T21:26:36.827 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:26:36 smithi097 ceph-mon[33317]: osdmap e91: 8 total, 8 up, 8 in 2024-03-11T21:26:36.827 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:26:36 smithi097 ceph-mon[33317]: from='mgr.14607 v1:172.21.15.92:0/1702717597' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T21:26:37.423 INFO:teuthology.orchestra.run.smithi092.stderr:Inferring config /var/lib/ceph/00db0016-dfed-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T21:26:37.776 INFO:teuthology.orchestra.run.smithi092.stdout:111669149732 2024-03-11T21:26:37.777 DEBUG:teuthology.orchestra.run.smithi092:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid 00db0016-dfed-11ee-95c8-87774f69a715 -- ceph osd last-stat-seq osd.3 2024-03-11T21:26:38.063 INFO:journalctl@ceph.mgr.y.smithi092.stdout:Mar 11 21:26:37 smithi092 ceph-00db0016-dfed-11ee-95c8-87774f69a715-mgr-y[29801]: ::ffff:172.21.15.97 - - [11/Mar/2024:21:26:37] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-11T21:26:38.071 INFO:teuthology.orchestra.run.smithi092.stderr:Inferring config /var/lib/ceph/00db0016-dfed-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T21:26:38.073 INFO:teuthology.orchestra.run.smithi092.stderr:Inferring config /var/lib/ceph/00db0016-dfed-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T21:26:38.103 INFO:teuthology.orchestra.run.smithi092.stdout:137438953501 2024-03-11T21:26:38.103 DEBUG:teuthology.orchestra.run.smithi092:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid 00db0016-dfed-11ee-95c8-87774f69a715 -- ceph osd last-stat-seq osd.4 2024-03-11T21:26:38.611 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:26:38 smithi092 ceph-mon[29572]: pgmap v15: 132 pgs: 132 active+clean; 582 KiB data, 227 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-11T21:26:38.611 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:26:38 smithi092 ceph-mon[29572]: osdmap e92: 8 total, 8 up, 8 in 2024-03-11T21:26:38.611 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:26:38 smithi092 ceph-mon[37563]: pgmap v15: 132 pgs: 132 active+clean; 582 KiB data, 227 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-11T21:26:38.611 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:26:38 smithi092 ceph-mon[37563]: osdmap e92: 8 total, 8 up, 8 in 2024-03-11T21:26:38.827 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:26:38 smithi097 ceph-mon[33317]: pgmap v15: 132 pgs: 132 active+clean; 582 KiB data, 227 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-11T21:26:38.827 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:26:38 smithi097 ceph-mon[33317]: osdmap e92: 8 total, 8 up, 8 in 2024-03-11T21:26:39.271 INFO:teuthology.orchestra.run.smithi092.stdout:55834574895 2024-03-11T21:26:39.272 DEBUG:teuthology.orchestra.run.smithi092:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid 00db0016-dfed-11ee-95c8-87774f69a715 -- ceph osd last-stat-seq osd.1 2024-03-11T21:26:39.611 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:26:39 smithi092 ceph-mon[29572]: 3.1 scrub starts 2024-03-11T21:26:39.611 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:26:39 smithi092 ceph-mon[29572]: 3.1 scrub ok 2024-03-11T21:26:39.611 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:26:39 smithi092 ceph-mon[37563]: 3.1 scrub starts 2024-03-11T21:26:39.612 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:26:39 smithi092 ceph-mon[37563]: 3.1 scrub ok 2024-03-11T21:26:39.827 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:26:39 smithi097 ceph-mon[33317]: 3.1 scrub starts 2024-03-11T21:26:39.827 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:26:39 smithi097 ceph-mon[33317]: 3.1 scrub ok 2024-03-11T21:26:40.827 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:26:40 smithi097 ceph-mon[33317]: purged_snaps scrub starts 2024-03-11T21:26:40.827 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:26:40 smithi097 ceph-mon[33317]: purged_snaps scrub ok 2024-03-11T21:26:40.827 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:26:40 smithi097 ceph-mon[33317]: pgmap v17: 132 pgs: 132 active+clean; 582 KiB data, 227 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-11T21:26:40.827 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:26:40 smithi097 ceph-mon[33317]: osdmap e93: 8 total, 8 up, 8 in 2024-03-11T21:26:40.855 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:26:40 smithi092 ceph-mon[37563]: purged_snaps scrub starts 2024-03-11T21:26:40.855 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:26:40 smithi092 ceph-mon[37563]: purged_snaps scrub ok 2024-03-11T21:26:40.855 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:26:40 smithi092 ceph-mon[37563]: pgmap v17: 132 pgs: 132 active+clean; 582 KiB data, 227 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-11T21:26:40.855 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:26:40 smithi092 ceph-mon[37563]: osdmap e93: 8 total, 8 up, 8 in 2024-03-11T21:26:40.855 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:26:40 smithi092 ceph-mon[29572]: purged_snaps scrub starts 2024-03-11T21:26:40.856 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:26:40 smithi092 ceph-mon[29572]: purged_snaps scrub ok 2024-03-11T21:26:40.856 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:26:40 smithi092 ceph-mon[29572]: pgmap v17: 132 pgs: 132 active+clean; 582 KiB data, 227 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-11T21:26:40.856 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:26:40 smithi092 ceph-mon[29572]: osdmap e93: 8 total, 8 up, 8 in 2024-03-11T21:26:41.611 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:26:41 smithi092 ceph-mon[29572]: 4.0 deep-scrub starts 2024-03-11T21:26:41.611 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:26:41 smithi092 ceph-mon[29572]: 4.0 deep-scrub ok 2024-03-11T21:26:41.611 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:26:41 smithi092 ceph-mon[37563]: 4.0 deep-scrub starts 2024-03-11T21:26:41.612 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:26:41 smithi092 ceph-mon[37563]: 4.0 deep-scrub ok 2024-03-11T21:26:41.827 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:26:41 smithi097 ceph-mon[33317]: 4.0 deep-scrub starts 2024-03-11T21:26:41.827 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:26:41 smithi097 ceph-mon[33317]: 4.0 deep-scrub ok 2024-03-11T21:26:42.274 INFO:teuthology.orchestra.run.smithi092.stdout:34359738420 2024-03-11T21:26:42.327 INFO:journalctl@ceph.iscsi.iscsi.a.smithi097.stdout:Mar 11 21:26:41 smithi097 ceph-00db0016-dfed-11ee-95c8-87774f69a715-iscsi-iscsi-a[60943]: debug there is no tcmu-runner data available 2024-03-11T21:26:42.353 INFO:teuthology.orchestra.run.smithi092.stdout:214748364820 2024-03-11T21:26:42.353 DEBUG:teuthology.orchestra.run.smithi092:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid 00db0016-dfed-11ee-95c8-87774f69a715 -- ceph osd last-stat-seq osd.6 2024-03-11T21:26:42.611 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:26:42 smithi092 ceph-mon[29572]: pgmap v19: 132 pgs: 132 active+clean; 582 KiB data, 227 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-11T21:26:42.611 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:26:42 smithi092 ceph-mon[29572]: from='client.? v1:172.21.15.92:0/2383948361' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 0}]: dispatch 2024-03-11T21:26:42.611 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:26:42 smithi092 ceph-mon[37563]: pgmap v19: 132 pgs: 132 active+clean; 582 KiB data, 227 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-11T21:26:42.611 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:26:42 smithi092 ceph-mon[37563]: from='client.? v1:172.21.15.92:0/2383948361' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 0}]: dispatch 2024-03-11T21:26:42.827 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:26:42 smithi097 ceph-mon[33317]: pgmap v19: 132 pgs: 132 active+clean; 582 KiB data, 227 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-11T21:26:42.827 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:26:42 smithi097 ceph-mon[33317]: from='client.? v1:172.21.15.92:0/2383948361' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 0}]: dispatch 2024-03-11T21:26:42.966 INFO:tasks.cephadm.ceph_manager.ceph:need seq 34359738417 got 34359738420 for osd.0 2024-03-11T21:26:42.966 DEBUG:teuthology.parallel:result is None 2024-03-11T21:26:43.528 INFO:teuthology.orchestra.run.smithi092.stderr:Inferring config /var/lib/ceph/00db0016-dfed-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T21:26:43.611 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:26:43 smithi092 ceph-mon[29572]: from='client.24469 v1:172.21.15.97:0/404208310' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:26:43.611 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:26:43 smithi092 ceph-mon[29572]: 2.3 scrub starts 2024-03-11T21:26:43.611 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:26:43 smithi092 ceph-mon[29572]: 2.3 scrub ok 2024-03-11T21:26:43.611 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:26:43 smithi092 ceph-mon[29572]: 4.1e scrub starts 2024-03-11T21:26:43.611 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:26:43 smithi092 ceph-mon[29572]: 4.1e scrub ok 2024-03-11T21:26:43.612 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:26:43 smithi092 ceph-mon[29572]: 3.2 scrub starts 2024-03-11T21:26:43.612 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:26:43 smithi092 ceph-mon[29572]: 3.2 scrub ok 2024-03-11T21:26:43.612 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:26:43 smithi092 ceph-mon[29572]: osdmap e94: 8 total, 8 up, 8 in 2024-03-11T21:26:43.612 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:26:43 smithi092 ceph-mon[37563]: from='client.24469 v1:172.21.15.97:0/404208310' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:26:43.612 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:26:43 smithi092 ceph-mon[37563]: 2.3 scrub starts 2024-03-11T21:26:43.612 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:26:43 smithi092 ceph-mon[37563]: 2.3 scrub ok 2024-03-11T21:26:43.612 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:26:43 smithi092 ceph-mon[37563]: 4.1e scrub starts 2024-03-11T21:26:43.612 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:26:43 smithi092 ceph-mon[37563]: 4.1e scrub ok 2024-03-11T21:26:43.612 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:26:43 smithi092 ceph-mon[37563]: 3.2 scrub starts 2024-03-11T21:26:43.612 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:26:43 smithi092 ceph-mon[37563]: 3.2 scrub ok 2024-03-11T21:26:43.612 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:26:43 smithi092 ceph-mon[37563]: osdmap e94: 8 total, 8 up, 8 in 2024-03-11T21:26:43.827 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:26:43 smithi097 ceph-mon[33317]: from='client.24469 v1:172.21.15.97:0/404208310' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:26:43.827 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:26:43 smithi097 ceph-mon[33317]: 2.3 scrub starts 2024-03-11T21:26:43.827 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:26:43 smithi097 ceph-mon[33317]: 2.3 scrub ok 2024-03-11T21:26:43.827 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:26:43 smithi097 ceph-mon[33317]: 4.1e scrub starts 2024-03-11T21:26:43.827 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:26:43 smithi097 ceph-mon[33317]: 4.1e scrub ok 2024-03-11T21:26:43.828 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:26:43 smithi097 ceph-mon[33317]: 3.2 scrub starts 2024-03-11T21:26:43.828 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:26:43 smithi097 ceph-mon[33317]: 3.2 scrub ok 2024-03-11T21:26:43.828 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:26:43 smithi097 ceph-mon[33317]: osdmap e94: 8 total, 8 up, 8 in 2024-03-11T21:26:44.542 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:26:44 smithi092 ceph-mon[29572]: pgmap v20: 132 pgs: 132 active+clean; 582 KiB data, 228 MiB used, 715 GiB / 715 GiB avail; 1.3 KiB/s rd, 1 op/s 2024-03-11T21:26:44.543 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:26:44 smithi092 ceph-mon[37563]: pgmap v20: 132 pgs: 132 active+clean; 582 KiB data, 228 MiB used, 715 GiB / 715 GiB avail; 1.3 KiB/s rd, 1 op/s 2024-03-11T21:26:44.827 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:26:44 smithi097 ceph-mon[33317]: pgmap v20: 132 pgs: 132 active+clean; 582 KiB data, 228 MiB used, 715 GiB / 715 GiB avail; 1.3 KiB/s rd, 1 op/s 2024-03-11T21:26:45.193 INFO:teuthology.orchestra.run.smithi092.stdout:257698037776 2024-03-11T21:26:45.396 INFO:teuthology.orchestra.run.smithi092.stdout:176093659162 2024-03-11T21:26:45.747 INFO:teuthology.orchestra.run.smithi092.stderr:Inferring config /var/lib/ceph/00db0016-dfed-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T21:26:45.827 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:26:45 smithi097 ceph-mon[33317]: 4.1a scrub starts 2024-03-11T21:26:45.827 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:26:45 smithi097 ceph-mon[33317]: 4.1a scrub ok 2024-03-11T21:26:45.827 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:26:45 smithi097 ceph-mon[33317]: 2.15 deep-scrub starts 2024-03-11T21:26:45.827 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:26:45 smithi097 ceph-mon[33317]: 2.15 deep-scrub ok 2024-03-11T21:26:45.827 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:26:45 smithi097 ceph-mon[33317]: from='client.? v1:172.21.15.92:0/3262540176' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 7}]: dispatch 2024-03-11T21:26:45.839 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:26:45 smithi092 ceph-mon[37563]: 4.1a scrub starts 2024-03-11T21:26:45.839 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:26:45 smithi092 ceph-mon[37563]: 4.1a scrub ok 2024-03-11T21:26:45.839 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:26:45 smithi092 ceph-mon[37563]: 2.15 deep-scrub starts 2024-03-11T21:26:45.839 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:26:45 smithi092 ceph-mon[37563]: 2.15 deep-scrub ok 2024-03-11T21:26:45.839 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:26:45 smithi092 ceph-mon[37563]: from='client.? v1:172.21.15.92:0/3262540176' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 7}]: dispatch 2024-03-11T21:26:45.840 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:26:45 smithi092 ceph-mon[29572]: 4.1a scrub starts 2024-03-11T21:26:45.840 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:26:45 smithi092 ceph-mon[29572]: 4.1a scrub ok 2024-03-11T21:26:45.840 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:26:45 smithi092 ceph-mon[29572]: 2.15 deep-scrub starts 2024-03-11T21:26:45.840 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:26:45 smithi092 ceph-mon[29572]: 2.15 deep-scrub ok 2024-03-11T21:26:45.840 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:26:45 smithi092 ceph-mon[29572]: from='client.? v1:172.21.15.92:0/3262540176' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 7}]: dispatch 2024-03-11T21:26:45.989 INFO:teuthology.orchestra.run.smithi092.stderr:Inferring config /var/lib/ceph/00db0016-dfed-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T21:26:46.486 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:26:46 smithi092 ceph-mon[29572]: pgmap v22: 132 pgs: 132 active+clean; 582 KiB data, 228 MiB used, 715 GiB / 715 GiB avail; 748 B/s rd, 0 op/s 2024-03-11T21:26:46.486 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:26:46 smithi092 ceph-mon[29572]: 2.2 scrub starts 2024-03-11T21:26:46.486 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:26:46 smithi092 ceph-mon[29572]: 2.2 scrub ok 2024-03-11T21:26:46.486 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:26:46 smithi092 ceph-mon[29572]: from='client.? v1:172.21.15.92:0/2966154383' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 5}]: dispatch 2024-03-11T21:26:46.486 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:26:46 smithi092 ceph-mon[29572]: 4.18 scrub starts 2024-03-11T21:26:46.486 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:26:46 smithi092 ceph-mon[29572]: 4.18 scrub ok 2024-03-11T21:26:46.486 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:26:46 smithi092 ceph-mon[29572]: 2.d scrub starts 2024-03-11T21:26:46.487 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:26:46 smithi092 ceph-mon[29572]: 2.d scrub ok 2024-03-11T21:26:46.487 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:26:46 smithi092 ceph-mon[37563]: pgmap v22: 132 pgs: 132 active+clean; 582 KiB data, 228 MiB used, 715 GiB / 715 GiB avail; 748 B/s rd, 0 op/s 2024-03-11T21:26:46.487 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:26:46 smithi092 ceph-mon[37563]: 2.2 scrub starts 2024-03-11T21:26:46.487 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:26:46 smithi092 ceph-mon[37563]: 2.2 scrub ok 2024-03-11T21:26:46.487 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:26:46 smithi092 ceph-mon[37563]: from='client.? v1:172.21.15.92:0/2966154383' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 5}]: dispatch 2024-03-11T21:26:46.487 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:26:46 smithi092 ceph-mon[37563]: 4.18 scrub starts 2024-03-11T21:26:46.487 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:26:46 smithi092 ceph-mon[37563]: 4.18 scrub ok 2024-03-11T21:26:46.487 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:26:46 smithi092 ceph-mon[37563]: 2.d scrub starts 2024-03-11T21:26:46.487 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:26:46 smithi092 ceph-mon[37563]: 2.d scrub ok 2024-03-11T21:26:46.827 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:26:46 smithi097 ceph-mon[33317]: pgmap v22: 132 pgs: 132 active+clean; 582 KiB data, 228 MiB used, 715 GiB / 715 GiB avail; 748 B/s rd, 0 op/s 2024-03-11T21:26:46.827 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:26:46 smithi097 ceph-mon[33317]: 2.2 scrub starts 2024-03-11T21:26:46.827 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:26:46 smithi097 ceph-mon[33317]: 2.2 scrub ok 2024-03-11T21:26:46.827 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:26:46 smithi097 ceph-mon[33317]: from='client.? v1:172.21.15.92:0/2966154383' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 5}]: dispatch 2024-03-11T21:26:46.827 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:26:46 smithi097 ceph-mon[33317]: 4.18 scrub starts 2024-03-11T21:26:46.827 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:26:46 smithi097 ceph-mon[33317]: 4.18 scrub ok 2024-03-11T21:26:46.827 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:26:46 smithi097 ceph-mon[33317]: 2.d scrub starts 2024-03-11T21:26:46.827 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:26:46 smithi097 ceph-mon[33317]: 2.d scrub ok 2024-03-11T21:26:47.417 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:26:47 smithi092 ceph-mon[29572]: 2.1f scrub starts 2024-03-11T21:26:47.417 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:26:47 smithi092 ceph-mon[29572]: 2.1f scrub ok 2024-03-11T21:26:47.417 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:26:47 smithi092 ceph-mon[37563]: 2.1f scrub starts 2024-03-11T21:26:47.417 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:26:47 smithi092 ceph-mon[37563]: 2.1f scrub ok 2024-03-11T21:26:47.827 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:26:47 smithi097 ceph-mon[33317]: 2.1f scrub starts 2024-03-11T21:26:47.827 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:26:47 smithi097 ceph-mon[33317]: 2.1f scrub ok 2024-03-11T21:26:48.082 INFO:tasks.cephadm.ceph_manager.ceph:need seq 176093659160 got 176093659162 for osd.5 2024-03-11T21:26:48.082 DEBUG:teuthology.parallel:result is None 2024-03-11T21:26:48.111 INFO:journalctl@ceph.mgr.y.smithi092.stdout:Mar 11 21:26:47 smithi092 ceph-00db0016-dfed-11ee-95c8-87774f69a715-mgr-y[29801]: ::ffff:172.21.15.97 - - [11/Mar/2024:21:26:47] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-11T21:26:48.465 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:26:48 smithi092 ceph-mon[29572]: pgmap v23: 132 pgs: 132 active+clean; 582 KiB data, 228 MiB used, 715 GiB / 715 GiB avail; 511 B/s rd, 0 op/s 2024-03-11T21:26:48.465 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:26:48 smithi092 ceph-mon[29572]: 2.18 scrub starts 2024-03-11T21:26:48.465 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:26:48 smithi092 ceph-mon[29572]: 2.18 scrub ok 2024-03-11T21:26:48.465 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:26:48 smithi092 ceph-mon[29572]: 2.7 scrub starts 2024-03-11T21:26:48.465 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:26:48 smithi092 ceph-mon[29572]: 2.7 scrub ok 2024-03-11T21:26:48.465 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:26:48 smithi092 ceph-mon[37563]: pgmap v23: 132 pgs: 132 active+clean; 582 KiB data, 228 MiB used, 715 GiB / 715 GiB avail; 511 B/s rd, 0 op/s 2024-03-11T21:26:48.465 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:26:48 smithi092 ceph-mon[37563]: 2.18 scrub starts 2024-03-11T21:26:48.465 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:26:48 smithi092 ceph-mon[37563]: 2.18 scrub ok 2024-03-11T21:26:48.465 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:26:48 smithi092 ceph-mon[37563]: 2.7 scrub starts 2024-03-11T21:26:48.466 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:26:48 smithi092 ceph-mon[37563]: 2.7 scrub ok 2024-03-11T21:26:48.827 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:26:48 smithi097 ceph-mon[33317]: pgmap v23: 132 pgs: 132 active+clean; 582 KiB data, 228 MiB used, 715 GiB / 715 GiB avail; 511 B/s rd, 0 op/s 2024-03-11T21:26:48.827 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:26:48 smithi097 ceph-mon[33317]: 2.18 scrub starts 2024-03-11T21:26:48.827 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:26:48 smithi097 ceph-mon[33317]: 2.18 scrub ok 2024-03-11T21:26:48.827 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:26:48 smithi097 ceph-mon[33317]: 2.7 scrub starts 2024-03-11T21:26:48.827 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:26:48 smithi097 ceph-mon[33317]: 2.7 scrub ok 2024-03-11T21:26:48.850 INFO:teuthology.orchestra.run.smithi092.stderr:Inferring config /var/lib/ceph/00db0016-dfed-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T21:26:48.850 INFO:teuthology.orchestra.run.smithi092.stderr:Inferring config /var/lib/ceph/00db0016-dfed-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T21:26:48.886 INFO:tasks.cephadm.ceph_manager.ceph:need seq 257698037772 got 257698037776 for osd.7 2024-03-11T21:26:48.887 DEBUG:teuthology.parallel:result is None 2024-03-11T21:26:49.604 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:26:49 smithi092 ceph-mon[29572]: 2.12 scrub starts 2024-03-11T21:26:49.604 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:26:49 smithi092 ceph-mon[29572]: 2.12 scrub ok 2024-03-11T21:26:49.605 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:26:49 smithi092 ceph-mon[29572]: 2.8 scrub starts 2024-03-11T21:26:49.605 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:26:49 smithi092 ceph-mon[29572]: 2.8 scrub ok 2024-03-11T21:26:49.605 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:26:49 smithi092 ceph-mon[29572]: 5.1f scrub starts 2024-03-11T21:26:49.605 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:26:49 smithi092 ceph-mon[29572]: 5.1f scrub ok 2024-03-11T21:26:49.605 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:26:49 smithi092 ceph-mon[37563]: 2.12 scrub starts 2024-03-11T21:26:49.605 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:26:49 smithi092 ceph-mon[37563]: 2.12 scrub ok 2024-03-11T21:26:49.605 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:26:49 smithi092 ceph-mon[37563]: 2.8 scrub starts 2024-03-11T21:26:49.605 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:26:49 smithi092 ceph-mon[37563]: 2.8 scrub ok 2024-03-11T21:26:49.605 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:26:49 smithi092 ceph-mon[37563]: 5.1f scrub starts 2024-03-11T21:26:49.605 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:26:49 smithi092 ceph-mon[37563]: 5.1f scrub ok 2024-03-11T21:26:49.827 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:26:49 smithi097 ceph-mon[33317]: 2.12 scrub starts 2024-03-11T21:26:49.827 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:26:49 smithi097 ceph-mon[33317]: 2.12 scrub ok 2024-03-11T21:26:49.827 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:26:49 smithi097 ceph-mon[33317]: 2.8 scrub starts 2024-03-11T21:26:49.827 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:26:49 smithi097 ceph-mon[33317]: 2.8 scrub ok 2024-03-11T21:26:49.827 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:26:49 smithi097 ceph-mon[33317]: 5.1f scrub starts 2024-03-11T21:26:49.827 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:26:49 smithi097 ceph-mon[33317]: 5.1f scrub ok 2024-03-11T21:26:50.208 INFO:teuthology.orchestra.run.smithi092.stdout:77309411371 2024-03-11T21:26:50.388 INFO:teuthology.orchestra.run.smithi092.stdout:111669149735 2024-03-11T21:26:50.611 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:26:50 smithi092 ceph-mon[29572]: pgmap v24: 132 pgs: 132 active+clean; 582 KiB data, 228 MiB used, 715 GiB / 715 GiB avail; 961 B/s rd, 0 op/s 2024-03-11T21:26:50.611 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:26:50 smithi092 ceph-mon[29572]: 2.1 deep-scrub starts 2024-03-11T21:26:50.611 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:26:50 smithi092 ceph-mon[29572]: 2.1 deep-scrub ok 2024-03-11T21:26:50.611 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:26:50 smithi092 ceph-mon[29572]: 2.1e scrub starts 2024-03-11T21:26:50.611 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:26:50 smithi092 ceph-mon[29572]: 2.1e scrub ok 2024-03-11T21:26:50.611 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:26:50 smithi092 ceph-mon[29572]: 4.2 deep-scrub starts 2024-03-11T21:26:50.611 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:26:50 smithi092 ceph-mon[29572]: 4.2 deep-scrub ok 2024-03-11T21:26:50.611 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:26:50 smithi092 ceph-mon[29572]: from='client.? v1:172.21.15.92:0/2077432684' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 2}]: dispatch 2024-03-11T21:26:50.612 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:26:50 smithi092 ceph-mon[29572]: from='client.? v1:172.21.15.92:0/1631220128' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 3}]: dispatch 2024-03-11T21:26:50.612 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:26:50 smithi092 ceph-mon[37563]: pgmap v24: 132 pgs: 132 active+clean; 582 KiB data, 228 MiB used, 715 GiB / 715 GiB avail; 961 B/s rd, 0 op/s 2024-03-11T21:26:50.612 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:26:50 smithi092 ceph-mon[37563]: 2.1 deep-scrub starts 2024-03-11T21:26:50.612 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:26:50 smithi092 ceph-mon[37563]: 2.1 deep-scrub ok 2024-03-11T21:26:50.612 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:26:50 smithi092 ceph-mon[37563]: 2.1e scrub starts 2024-03-11T21:26:50.612 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:26:50 smithi092 ceph-mon[37563]: 2.1e scrub ok 2024-03-11T21:26:50.612 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:26:50 smithi092 ceph-mon[37563]: 4.2 deep-scrub starts 2024-03-11T21:26:50.612 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:26:50 smithi092 ceph-mon[37563]: 4.2 deep-scrub ok 2024-03-11T21:26:50.612 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:26:50 smithi092 ceph-mon[37563]: from='client.? v1:172.21.15.92:0/2077432684' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 2}]: dispatch 2024-03-11T21:26:50.612 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:26:50 smithi092 ceph-mon[37563]: from='client.? v1:172.21.15.92:0/1631220128' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 3}]: dispatch 2024-03-11T21:26:50.728 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:26:50 smithi097 ceph-mon[33317]: pgmap v24: 132 pgs: 132 active+clean; 582 KiB data, 228 MiB used, 715 GiB / 715 GiB avail; 961 B/s rd, 0 op/s 2024-03-11T21:26:50.729 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:26:50 smithi097 ceph-mon[33317]: 2.1 deep-scrub starts 2024-03-11T21:26:50.729 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:26:50 smithi097 ceph-mon[33317]: 2.1 deep-scrub ok 2024-03-11T21:26:50.729 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:26:50 smithi097 ceph-mon[33317]: 2.1e scrub starts 2024-03-11T21:26:50.729 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:26:50 smithi097 ceph-mon[33317]: 2.1e scrub ok 2024-03-11T21:26:50.729 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:26:50 smithi097 ceph-mon[33317]: 4.2 deep-scrub starts 2024-03-11T21:26:50.729 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:26:50 smithi097 ceph-mon[33317]: 4.2 deep-scrub ok 2024-03-11T21:26:50.729 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:26:50 smithi097 ceph-mon[33317]: from='client.? v1:172.21.15.92:0/2077432684' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 2}]: dispatch 2024-03-11T21:26:50.729 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:26:50 smithi097 ceph-mon[33317]: from='client.? v1:172.21.15.92:0/1631220128' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 3}]: dispatch 2024-03-11T21:26:51.752 INFO:tasks.cephadm.ceph_manager.ceph:need seq 77309411367 got 77309411371 for osd.2 2024-03-11T21:26:51.752 DEBUG:teuthology.parallel:result is None 2024-03-11T21:26:52.077 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:26:51 smithi092 ceph-mon[29572]: 6.2 scrub starts 2024-03-11T21:26:52.077 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:26:51 smithi092 ceph-mon[29572]: 6.2 scrub ok 2024-03-11T21:26:52.077 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:26:51 smithi092 ceph-mon[29572]: 4.8 scrub starts 2024-03-11T21:26:52.077 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:26:51 smithi092 ceph-mon[29572]: 4.8 scrub ok 2024-03-11T21:26:52.078 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:26:51 smithi092 ceph-mon[29572]: pgmap v25: 132 pgs: 132 active+clean; 582 KiB data, 228 MiB used, 715 GiB / 715 GiB avail; 921 B/s rd, 0 op/s 2024-03-11T21:26:52.078 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:26:51 smithi092 ceph-mon[29572]: from='mgr.14607 ' entity='mgr.y' 2024-03-11T21:26:52.078 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:26:51 smithi092 ceph-mon[29572]: from='mgr.14607 ' entity='mgr.y' 2024-03-11T21:26:52.078 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:26:51 smithi092 ceph-mon[29572]: 2.9 deep-scrub starts 2024-03-11T21:26:52.078 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:26:51 smithi092 ceph-mon[29572]: 2.9 deep-scrub ok 2024-03-11T21:26:52.078 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:26:51 smithi092 ceph-mon[29572]: from='mgr.14607 v1:172.21.15.92:0/1702717597' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T21:26:52.078 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:26:51 smithi092 ceph-mon[37563]: 6.2 scrub starts 2024-03-11T21:26:52.078 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:26:51 smithi092 ceph-mon[37563]: 6.2 scrub ok 2024-03-11T21:26:52.078 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:26:51 smithi092 ceph-mon[37563]: 4.8 scrub starts 2024-03-11T21:26:52.078 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:26:51 smithi092 ceph-mon[37563]: 4.8 scrub ok 2024-03-11T21:26:52.078 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:26:51 smithi092 ceph-mon[37563]: pgmap v25: 132 pgs: 132 active+clean; 582 KiB data, 228 MiB used, 715 GiB / 715 GiB avail; 921 B/s rd, 0 op/s 2024-03-11T21:26:52.079 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:26:51 smithi092 ceph-mon[37563]: from='mgr.14607 ' entity='mgr.y' 2024-03-11T21:26:52.079 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:26:51 smithi092 ceph-mon[37563]: from='mgr.14607 ' entity='mgr.y' 2024-03-11T21:26:52.079 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:26:51 smithi092 ceph-mon[37563]: 2.9 deep-scrub starts 2024-03-11T21:26:52.079 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:26:51 smithi092 ceph-mon[37563]: 2.9 deep-scrub ok 2024-03-11T21:26:52.079 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:26:51 smithi092 ceph-mon[37563]: from='mgr.14607 v1:172.21.15.92:0/1702717597' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T21:26:52.327 INFO:journalctl@ceph.iscsi.iscsi.a.smithi097.stdout:Mar 11 21:26:51 smithi097 ceph-00db0016-dfed-11ee-95c8-87774f69a715-iscsi-iscsi-a[60943]: debug there is no tcmu-runner data available 2024-03-11T21:26:52.327 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:26:51 smithi097 ceph-mon[33317]: 6.2 scrub starts 2024-03-11T21:26:52.327 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:26:51 smithi097 ceph-mon[33317]: 6.2 scrub ok 2024-03-11T21:26:52.328 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:26:51 smithi097 ceph-mon[33317]: 4.8 scrub starts 2024-03-11T21:26:52.328 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:26:51 smithi097 ceph-mon[33317]: 4.8 scrub ok 2024-03-11T21:26:52.328 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:26:51 smithi097 ceph-mon[33317]: pgmap v25: 132 pgs: 132 active+clean; 582 KiB data, 228 MiB used, 715 GiB / 715 GiB avail; 921 B/s rd, 0 op/s 2024-03-11T21:26:52.328 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:26:51 smithi097 ceph-mon[33317]: from='mgr.14607 ' entity='mgr.y' 2024-03-11T21:26:52.328 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:26:51 smithi097 ceph-mon[33317]: from='mgr.14607 ' entity='mgr.y' 2024-03-11T21:26:52.328 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:26:51 smithi097 ceph-mon[33317]: 2.9 deep-scrub starts 2024-03-11T21:26:52.328 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:26:51 smithi097 ceph-mon[33317]: 2.9 deep-scrub ok 2024-03-11T21:26:52.328 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:26:51 smithi097 ceph-mon[33317]: from='mgr.14607 v1:172.21.15.92:0/1702717597' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T21:26:52.400 INFO:teuthology.orchestra.run.smithi092.stdout:137438953504 2024-03-11T21:26:53.041 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:26:52 smithi092 ceph-mon[29572]: 2.0 scrub starts 2024-03-11T21:26:53.041 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:26:52 smithi092 ceph-mon[29572]: 2.0 scrub ok 2024-03-11T21:26:53.042 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:26:52 smithi092 ceph-mon[29572]: 5.1 deep-scrub starts 2024-03-11T21:26:53.042 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:26:52 smithi092 ceph-mon[29572]: 5.1 deep-scrub ok 2024-03-11T21:26:53.042 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:26:52 smithi092 ceph-mon[29572]: 5.f scrub starts 2024-03-11T21:26:53.042 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:26:52 smithi092 ceph-mon[29572]: 5.f scrub ok 2024-03-11T21:26:53.042 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:26:52 smithi092 ceph-mon[29572]: from='client.24469 v1:172.21.15.97:0/404208310' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:26:53.042 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:26:52 smithi092 ceph-mon[29572]: 2.a scrub starts 2024-03-11T21:26:53.042 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:26:52 smithi092 ceph-mon[29572]: 2.a scrub ok 2024-03-11T21:26:53.042 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:26:52 smithi092 ceph-mon[29572]: from='client.? v1:172.21.15.92:0/2723434999' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 4}]: dispatch 2024-03-11T21:26:53.042 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:26:52 smithi092 ceph-mon[37563]: 2.0 scrub starts 2024-03-11T21:26:53.042 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:26:52 smithi092 ceph-mon[37563]: 2.0 scrub ok 2024-03-11T21:26:53.042 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:26:52 smithi092 ceph-mon[37563]: 5.1 deep-scrub starts 2024-03-11T21:26:53.043 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:26:52 smithi092 ceph-mon[37563]: 5.1 deep-scrub ok 2024-03-11T21:26:53.043 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:26:52 smithi092 ceph-mon[37563]: 5.f scrub starts 2024-03-11T21:26:53.043 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:26:52 smithi092 ceph-mon[37563]: 5.f scrub ok 2024-03-11T21:26:53.043 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:26:52 smithi092 ceph-mon[37563]: from='client.24469 v1:172.21.15.97:0/404208310' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:26:53.043 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:26:52 smithi092 ceph-mon[37563]: 2.a scrub starts 2024-03-11T21:26:53.043 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:26:52 smithi092 ceph-mon[37563]: 2.a scrub ok 2024-03-11T21:26:53.043 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:26:52 smithi092 ceph-mon[37563]: from='client.? v1:172.21.15.92:0/2723434999' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 4}]: dispatch 2024-03-11T21:26:53.327 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:26:52 smithi097 ceph-mon[33317]: 2.0 scrub starts 2024-03-11T21:26:53.327 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:26:52 smithi097 ceph-mon[33317]: 2.0 scrub ok 2024-03-11T21:26:53.327 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:26:52 smithi097 ceph-mon[33317]: 5.1 deep-scrub starts 2024-03-11T21:26:53.327 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:26:52 smithi097 ceph-mon[33317]: 5.1 deep-scrub ok 2024-03-11T21:26:53.327 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:26:52 smithi097 ceph-mon[33317]: 5.f scrub starts 2024-03-11T21:26:53.327 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:26:52 smithi097 ceph-mon[33317]: 5.f scrub ok 2024-03-11T21:26:53.327 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:26:52 smithi097 ceph-mon[33317]: from='client.24469 v1:172.21.15.97:0/404208310' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:26:53.328 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:26:52 smithi097 ceph-mon[33317]: 2.a scrub starts 2024-03-11T21:26:53.328 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:26:52 smithi097 ceph-mon[33317]: 2.a scrub ok 2024-03-11T21:26:53.328 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:26:52 smithi097 ceph-mon[33317]: from='client.? v1:172.21.15.92:0/2723434999' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 4}]: dispatch 2024-03-11T21:26:54.111 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:26:53 smithi092 ceph-mon[37563]: pgmap v26: 132 pgs: 132 active+clean; 582 KiB data, 228 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-11T21:26:54.111 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:26:53 smithi092 ceph-mon[37563]: 5.9 deep-scrub starts 2024-03-11T21:26:54.112 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:26:53 smithi092 ceph-mon[37563]: 5.9 deep-scrub ok 2024-03-11T21:26:54.112 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:26:53 smithi092 ceph-mon[37563]: from='mgr.14607 ' entity='mgr.y' 2024-03-11T21:26:54.112 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:26:53 smithi092 ceph-mon[37563]: from='mgr.14607 ' entity='mgr.y' 2024-03-11T21:26:54.112 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:26:53 smithi092 ceph-mon[37563]: 4.16 scrub starts 2024-03-11T21:26:54.112 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:26:53 smithi092 ceph-mon[37563]: from='mgr.14607 v1:172.21.15.92:0/1702717597' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-11T21:26:54.112 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:26:53 smithi092 ceph-mon[37563]: from='mgr.14607 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-11T21:26:54.112 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:26:53 smithi092 ceph-mon[37563]: 4.16 scrub ok 2024-03-11T21:26:54.112 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:26:53 smithi092 ceph-mon[37563]: from='mgr.14607 v1:172.21.15.92:0/1702717597' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-11T21:26:54.112 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:26:53 smithi092 ceph-mon[37563]: from='mgr.14607 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-11T21:26:54.112 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:26:53 smithi092 ceph-mon[37563]: from='mgr.14607 v1:172.21.15.92:0/1702717597' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-11T21:26:54.112 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:26:53 smithi092 ceph-mon[37563]: from='mgr.14607 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-11T21:26:54.112 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:26:53 smithi092 ceph-mon[37563]: from='mgr.14607 v1:172.21.15.92:0/1702717597' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-03-11T21:26:54.112 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:26:53 smithi092 ceph-mon[37563]: from='mgr.14607 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-03-11T21:26:54.113 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:26:53 smithi092 ceph-mon[37563]: Adjusting osd_memory_target on smithi092 to 3760M 2024-03-11T21:26:54.113 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:26:53 smithi092 ceph-mon[37563]: from='mgr.14607 ' entity='mgr.y' 2024-03-11T21:26:54.113 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:26:53 smithi092 ceph-mon[37563]: from='mgr.14607 v1:172.21.15.92:0/1702717597' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T21:26:54.113 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:26:53 smithi092 ceph-mon[37563]: from='mgr.14607 v1:172.21.15.92:0/1702717597' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T21:26:54.113 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:26:53 smithi092 ceph-mon[37563]: Updating smithi092:/etc/ceph/ceph.conf 2024-03-11T21:26:54.113 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:26:53 smithi092 ceph-mon[37563]: Updating smithi097:/etc/ceph/ceph.conf 2024-03-11T21:26:54.113 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:26:53 smithi092 ceph-mon[37563]: 6.1b scrub starts 2024-03-11T21:26:54.113 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:26:53 smithi092 ceph-mon[37563]: 6.1b scrub ok 2024-03-11T21:26:54.113 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:26:53 smithi092 ceph-mon[37563]: Updating smithi092:/var/lib/ceph/00db0016-dfed-11ee-95c8-87774f69a715/config/ceph.conf 2024-03-11T21:26:54.113 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:26:53 smithi092 ceph-mon[37563]: Updating smithi097:/var/lib/ceph/00db0016-dfed-11ee-95c8-87774f69a715/config/ceph.conf 2024-03-11T21:26:54.113 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:26:53 smithi092 ceph-mon[37563]: Updating smithi092:/etc/ceph/ceph.client.admin.keyring 2024-03-11T21:26:54.113 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:26:53 smithi092 ceph-mon[29572]: pgmap v26: 132 pgs: 132 active+clean; 582 KiB data, 228 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-11T21:26:54.114 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:26:53 smithi092 ceph-mon[29572]: 5.9 deep-scrub starts 2024-03-11T21:26:54.114 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:26:53 smithi092 ceph-mon[29572]: 5.9 deep-scrub ok 2024-03-11T21:26:54.114 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:26:53 smithi092 ceph-mon[29572]: from='mgr.14607 ' entity='mgr.y' 2024-03-11T21:26:54.114 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:26:53 smithi092 ceph-mon[29572]: from='mgr.14607 ' entity='mgr.y' 2024-03-11T21:26:54.114 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:26:53 smithi092 ceph-mon[29572]: 4.16 scrub starts 2024-03-11T21:26:54.114 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:26:53 smithi092 ceph-mon[29572]: from='mgr.14607 v1:172.21.15.92:0/1702717597' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-11T21:26:54.114 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:26:53 smithi092 ceph-mon[29572]: from='mgr.14607 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-11T21:26:54.114 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:26:53 smithi092 ceph-mon[29572]: 4.16 scrub ok 2024-03-11T21:26:54.114 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:26:53 smithi092 ceph-mon[29572]: from='mgr.14607 v1:172.21.15.92:0/1702717597' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-11T21:26:54.114 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:26:53 smithi092 ceph-mon[29572]: from='mgr.14607 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-11T21:26:54.114 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:26:53 smithi092 ceph-mon[29572]: from='mgr.14607 v1:172.21.15.92:0/1702717597' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-11T21:26:54.114 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:26:53 smithi092 ceph-mon[29572]: from='mgr.14607 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-11T21:26:54.114 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:26:53 smithi092 ceph-mon[29572]: from='mgr.14607 v1:172.21.15.92:0/1702717597' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-03-11T21:26:54.115 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:26:53 smithi092 ceph-mon[29572]: from='mgr.14607 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-03-11T21:26:54.115 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:26:53 smithi092 ceph-mon[29572]: Adjusting osd_memory_target on smithi092 to 3760M 2024-03-11T21:26:54.115 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:26:53 smithi092 ceph-mon[29572]: from='mgr.14607 ' entity='mgr.y' 2024-03-11T21:26:54.115 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:26:53 smithi092 ceph-mon[29572]: from='mgr.14607 v1:172.21.15.92:0/1702717597' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T21:26:54.115 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:26:53 smithi092 ceph-mon[29572]: from='mgr.14607 v1:172.21.15.92:0/1702717597' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T21:26:54.115 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:26:53 smithi092 ceph-mon[29572]: Updating smithi092:/etc/ceph/ceph.conf 2024-03-11T21:26:54.115 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:26:53 smithi092 ceph-mon[29572]: Updating smithi097:/etc/ceph/ceph.conf 2024-03-11T21:26:54.115 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:26:53 smithi092 ceph-mon[29572]: 6.1b scrub starts 2024-03-11T21:26:54.115 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:26:53 smithi092 ceph-mon[29572]: 6.1b scrub ok 2024-03-11T21:26:54.115 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:26:53 smithi092 ceph-mon[29572]: Updating smithi092:/var/lib/ceph/00db0016-dfed-11ee-95c8-87774f69a715/config/ceph.conf 2024-03-11T21:26:54.115 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:26:53 smithi092 ceph-mon[29572]: Updating smithi097:/var/lib/ceph/00db0016-dfed-11ee-95c8-87774f69a715/config/ceph.conf 2024-03-11T21:26:54.115 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:26:53 smithi092 ceph-mon[29572]: Updating smithi092:/etc/ceph/ceph.client.admin.keyring 2024-03-11T21:26:54.327 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:26:53 smithi097 ceph-mon[33317]: pgmap v26: 132 pgs: 132 active+clean; 582 KiB data, 228 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-11T21:26:54.327 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:26:53 smithi097 ceph-mon[33317]: 5.9 deep-scrub starts 2024-03-11T21:26:54.327 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:26:53 smithi097 ceph-mon[33317]: 5.9 deep-scrub ok 2024-03-11T21:26:54.327 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:26:53 smithi097 ceph-mon[33317]: from='mgr.14607 ' entity='mgr.y' 2024-03-11T21:26:54.327 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:26:53 smithi097 ceph-mon[33317]: from='mgr.14607 ' entity='mgr.y' 2024-03-11T21:26:54.327 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:26:53 smithi097 ceph-mon[33317]: 4.16 scrub starts 2024-03-11T21:26:54.327 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:26:53 smithi097 ceph-mon[33317]: from='mgr.14607 v1:172.21.15.92:0/1702717597' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-11T21:26:54.327 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:26:53 smithi097 ceph-mon[33317]: from='mgr.14607 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-11T21:26:54.328 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:26:53 smithi097 ceph-mon[33317]: 4.16 scrub ok 2024-03-11T21:26:54.328 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:26:53 smithi097 ceph-mon[33317]: from='mgr.14607 v1:172.21.15.92:0/1702717597' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-11T21:26:54.328 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:26:53 smithi097 ceph-mon[33317]: from='mgr.14607 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-11T21:26:54.328 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:26:53 smithi097 ceph-mon[33317]: from='mgr.14607 v1:172.21.15.92:0/1702717597' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-11T21:26:54.328 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:26:53 smithi097 ceph-mon[33317]: from='mgr.14607 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-11T21:26:54.328 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:26:53 smithi097 ceph-mon[33317]: from='mgr.14607 v1:172.21.15.92:0/1702717597' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-03-11T21:26:54.328 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:26:53 smithi097 ceph-mon[33317]: from='mgr.14607 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-03-11T21:26:54.328 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:26:53 smithi097 ceph-mon[33317]: Adjusting osd_memory_target on smithi092 to 3760M 2024-03-11T21:26:54.328 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:26:53 smithi097 ceph-mon[33317]: from='mgr.14607 ' entity='mgr.y' 2024-03-11T21:26:54.328 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:26:53 smithi097 ceph-mon[33317]: from='mgr.14607 v1:172.21.15.92:0/1702717597' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T21:26:54.328 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:26:53 smithi097 ceph-mon[33317]: from='mgr.14607 v1:172.21.15.92:0/1702717597' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T21:26:54.328 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:26:53 smithi097 ceph-mon[33317]: Updating smithi092:/etc/ceph/ceph.conf 2024-03-11T21:26:54.328 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:26:53 smithi097 ceph-mon[33317]: Updating smithi097:/etc/ceph/ceph.conf 2024-03-11T21:26:54.328 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:26:53 smithi097 ceph-mon[33317]: 6.1b scrub starts 2024-03-11T21:26:54.329 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:26:53 smithi097 ceph-mon[33317]: 6.1b scrub ok 2024-03-11T21:26:54.329 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:26:53 smithi097 ceph-mon[33317]: Updating smithi092:/var/lib/ceph/00db0016-dfed-11ee-95c8-87774f69a715/config/ceph.conf 2024-03-11T21:26:54.329 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:26:53 smithi097 ceph-mon[33317]: Updating smithi097:/var/lib/ceph/00db0016-dfed-11ee-95c8-87774f69a715/config/ceph.conf 2024-03-11T21:26:54.329 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:26:53 smithi097 ceph-mon[33317]: Updating smithi092:/etc/ceph/ceph.client.admin.keyring 2024-03-11T21:26:54.367 INFO:tasks.cephadm.ceph_manager.ceph:need seq 111669149732 got 111669149735 for osd.3 2024-03-11T21:26:54.367 DEBUG:teuthology.parallel:result is None 2024-03-11T21:26:55.111 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:26:54 smithi092 ceph-mon[29572]: Updating smithi097:/etc/ceph/ceph.client.admin.keyring 2024-03-11T21:26:55.111 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:26:54 smithi092 ceph-mon[29572]: 3.0 scrub starts 2024-03-11T21:26:55.111 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:26:54 smithi092 ceph-mon[29572]: 3.0 scrub ok 2024-03-11T21:26:55.111 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:26:54 smithi092 ceph-mon[29572]: Updating smithi092:/var/lib/ceph/00db0016-dfed-11ee-95c8-87774f69a715/config/ceph.client.admin.keyring 2024-03-11T21:26:55.112 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:26:54 smithi092 ceph-mon[29572]: Updating smithi097:/var/lib/ceph/00db0016-dfed-11ee-95c8-87774f69a715/config/ceph.client.admin.keyring 2024-03-11T21:26:55.112 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:26:54 smithi092 ceph-mon[29572]: from='mgr.14607 ' entity='mgr.y' 2024-03-11T21:26:55.112 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:26:54 smithi092 ceph-mon[29572]: from='mgr.14607 ' entity='mgr.y' 2024-03-11T21:26:55.112 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:26:54 smithi092 ceph-mon[29572]: 6.3 scrub starts 2024-03-11T21:26:55.112 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:26:54 smithi092 ceph-mon[29572]: 6.3 scrub ok 2024-03-11T21:26:55.112 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:26:54 smithi092 ceph-mon[29572]: from='mgr.14607 ' entity='mgr.y' 2024-03-11T21:26:55.112 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:26:54 smithi092 ceph-mon[29572]: from='mgr.14607 ' entity='mgr.y' 2024-03-11T21:26:55.112 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:26:54 smithi092 ceph-mon[29572]: from='mgr.14607 ' entity='mgr.y' 2024-03-11T21:26:55.112 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:26:54 smithi092 ceph-mon[29572]: Deploying daemon alertmanager.a on smithi092 2024-03-11T21:26:55.112 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:26:54 smithi092 ceph-mon[37563]: Updating smithi097:/etc/ceph/ceph.client.admin.keyring 2024-03-11T21:26:55.112 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:26:54 smithi092 ceph-mon[37563]: 3.0 scrub starts 2024-03-11T21:26:55.112 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:26:54 smithi092 ceph-mon[37563]: 3.0 scrub ok 2024-03-11T21:26:55.112 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:26:54 smithi092 ceph-mon[37563]: Updating smithi092:/var/lib/ceph/00db0016-dfed-11ee-95c8-87774f69a715/config/ceph.client.admin.keyring 2024-03-11T21:26:55.113 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:26:54 smithi092 ceph-mon[37563]: Updating smithi097:/var/lib/ceph/00db0016-dfed-11ee-95c8-87774f69a715/config/ceph.client.admin.keyring 2024-03-11T21:26:55.113 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:26:54 smithi092 ceph-mon[37563]: from='mgr.14607 ' entity='mgr.y' 2024-03-11T21:26:55.113 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:26:54 smithi092 ceph-mon[37563]: from='mgr.14607 ' entity='mgr.y' 2024-03-11T21:26:55.113 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:26:54 smithi092 ceph-mon[37563]: 6.3 scrub starts 2024-03-11T21:26:55.113 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:26:54 smithi092 ceph-mon[37563]: 6.3 scrub ok 2024-03-11T21:26:55.113 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:26:54 smithi092 ceph-mon[37563]: from='mgr.14607 ' entity='mgr.y' 2024-03-11T21:26:55.113 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:26:54 smithi092 ceph-mon[37563]: from='mgr.14607 ' entity='mgr.y' 2024-03-11T21:26:55.113 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:26:54 smithi092 ceph-mon[37563]: from='mgr.14607 ' entity='mgr.y' 2024-03-11T21:26:55.113 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:26:54 smithi092 ceph-mon[37563]: Deploying daemon alertmanager.a on smithi092 2024-03-11T21:26:55.257 INFO:tasks.cephadm.ceph_manager.ceph:need seq 137438953501 got 137438953504 for osd.4 2024-03-11T21:26:55.258 DEBUG:teuthology.parallel:result is None 2024-03-11T21:26:55.327 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:26:54 smithi097 ceph-mon[33317]: Updating smithi097:/etc/ceph/ceph.client.admin.keyring 2024-03-11T21:26:55.327 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:26:54 smithi097 ceph-mon[33317]: 3.0 scrub starts 2024-03-11T21:26:55.327 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:26:54 smithi097 ceph-mon[33317]: 3.0 scrub ok 2024-03-11T21:26:55.327 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:26:54 smithi097 ceph-mon[33317]: Updating smithi092:/var/lib/ceph/00db0016-dfed-11ee-95c8-87774f69a715/config/ceph.client.admin.keyring 2024-03-11T21:26:55.327 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:26:54 smithi097 ceph-mon[33317]: Updating smithi097:/var/lib/ceph/00db0016-dfed-11ee-95c8-87774f69a715/config/ceph.client.admin.keyring 2024-03-11T21:26:55.327 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:26:54 smithi097 ceph-mon[33317]: from='mgr.14607 ' entity='mgr.y' 2024-03-11T21:26:55.327 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:26:54 smithi097 ceph-mon[33317]: from='mgr.14607 ' entity='mgr.y' 2024-03-11T21:26:55.328 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:26:54 smithi097 ceph-mon[33317]: 6.3 scrub starts 2024-03-11T21:26:55.328 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:26:54 smithi097 ceph-mon[33317]: 6.3 scrub ok 2024-03-11T21:26:55.328 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:26:54 smithi097 ceph-mon[33317]: from='mgr.14607 ' entity='mgr.y' 2024-03-11T21:26:55.328 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:26:54 smithi097 ceph-mon[33317]: from='mgr.14607 ' entity='mgr.y' 2024-03-11T21:26:55.328 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:26:54 smithi097 ceph-mon[33317]: from='mgr.14607 ' entity='mgr.y' 2024-03-11T21:26:55.328 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:26:54 smithi097 ceph-mon[33317]: Deploying daemon alertmanager.a on smithi092 2024-03-11T21:26:55.934 INFO:teuthology.orchestra.run.smithi092.stdout:55834574898 2024-03-11T21:26:56.309 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:26:55 smithi092 ceph-mon[29572]: 6.d scrub starts 2024-03-11T21:26:56.309 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:26:55 smithi092 ceph-mon[29572]: 6.d scrub ok 2024-03-11T21:26:56.309 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:26:55 smithi092 ceph-mon[29572]: pgmap v27: 132 pgs: 132 active+clean; 582 KiB data, 228 MiB used, 715 GiB / 715 GiB avail; 864 B/s rd, 0 op/s 2024-03-11T21:26:56.309 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:26:55 smithi092 ceph-mon[29572]: 4.1f scrub starts 2024-03-11T21:26:56.309 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:26:55 smithi092 ceph-mon[29572]: 4.1f scrub ok 2024-03-11T21:26:56.309 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:26:55 smithi092 ceph-mon[29572]: 2.1b scrub starts 2024-03-11T21:26:56.309 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:26:55 smithi092 ceph-mon[29572]: 2.1b scrub ok 2024-03-11T21:26:56.309 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:26:55 smithi092 ceph-mon[29572]: from='client.? v1:172.21.15.92:0/1916542695' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 1}]: dispatch 2024-03-11T21:26:56.310 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:26:55 smithi092 ceph-mon[37563]: 6.d scrub starts 2024-03-11T21:26:56.310 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:26:55 smithi092 ceph-mon[37563]: 6.d scrub ok 2024-03-11T21:26:56.310 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:26:55 smithi092 ceph-mon[37563]: pgmap v27: 132 pgs: 132 active+clean; 582 KiB data, 228 MiB used, 715 GiB / 715 GiB avail; 864 B/s rd, 0 op/s 2024-03-11T21:26:56.310 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:26:55 smithi092 ceph-mon[37563]: 4.1f scrub starts 2024-03-11T21:26:56.310 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:26:55 smithi092 ceph-mon[37563]: 4.1f scrub ok 2024-03-11T21:26:56.310 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:26:55 smithi092 ceph-mon[37563]: 2.1b scrub starts 2024-03-11T21:26:56.310 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:26:55 smithi092 ceph-mon[37563]: 2.1b scrub ok 2024-03-11T21:26:56.310 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:26:55 smithi092 ceph-mon[37563]: from='client.? v1:172.21.15.92:0/1916542695' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 1}]: dispatch 2024-03-11T21:26:56.327 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:26:55 smithi097 ceph-mon[33317]: 6.d scrub starts 2024-03-11T21:26:56.327 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:26:55 smithi097 ceph-mon[33317]: 6.d scrub ok 2024-03-11T21:26:56.327 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:26:55 smithi097 ceph-mon[33317]: pgmap v27: 132 pgs: 132 active+clean; 582 KiB data, 228 MiB used, 715 GiB / 715 GiB avail; 864 B/s rd, 0 op/s 2024-03-11T21:26:56.327 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:26:55 smithi097 ceph-mon[33317]: 4.1f scrub starts 2024-03-11T21:26:56.327 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:26:55 smithi097 ceph-mon[33317]: 4.1f scrub ok 2024-03-11T21:26:56.327 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:26:55 smithi097 ceph-mon[33317]: 2.1b scrub starts 2024-03-11T21:26:56.327 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:26:55 smithi097 ceph-mon[33317]: 2.1b scrub ok 2024-03-11T21:26:56.327 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:26:55 smithi097 ceph-mon[33317]: from='client.? v1:172.21.15.92:0/1916542695' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 1}]: dispatch 2024-03-11T21:26:56.618 INFO:teuthology.orchestra.run.smithi092.stdout:214748364824 2024-03-11T21:26:56.912 INFO:tasks.cephadm.ceph_manager.ceph:need seq 55834574895 got 55834574898 for osd.1 2024-03-11T21:26:56.913 DEBUG:teuthology.parallel:result is None 2024-03-11T21:26:57.327 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:26:56 smithi097 ceph-mon[33317]: 6.a scrub starts 2024-03-11T21:26:57.327 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:26:57 smithi097 ceph-mon[33317]: 6.a scrub ok 2024-03-11T21:26:57.327 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:26:57 smithi097 ceph-mon[33317]: 2.17 scrub starts 2024-03-11T21:26:57.327 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:26:57 smithi097 ceph-mon[33317]: 2.17 scrub ok 2024-03-11T21:26:57.327 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:26:57 smithi097 ceph-mon[33317]: 2.1d scrub starts 2024-03-11T21:26:57.327 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:26:57 smithi097 ceph-mon[33317]: 2.1d scrub ok 2024-03-11T21:26:57.327 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:26:57 smithi097 ceph-mon[33317]: from='client.? v1:172.21.15.92:0/2082339182' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 6}]: dispatch 2024-03-11T21:26:57.331 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:26:56 smithi092 ceph-mon[29572]: 6.a scrub starts 2024-03-11T21:26:57.332 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:26:56 smithi092 ceph-mon[29572]: 6.a scrub ok 2024-03-11T21:26:57.332 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:26:56 smithi092 ceph-mon[29572]: 2.17 scrub starts 2024-03-11T21:26:57.332 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:26:56 smithi092 ceph-mon[29572]: 2.17 scrub ok 2024-03-11T21:26:57.332 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:26:56 smithi092 ceph-mon[29572]: 2.1d scrub starts 2024-03-11T21:26:57.332 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:26:56 smithi092 ceph-mon[29572]: 2.1d scrub ok 2024-03-11T21:26:57.332 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:26:56 smithi092 ceph-mon[29572]: from='client.? v1:172.21.15.92:0/2082339182' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 6}]: dispatch 2024-03-11T21:26:57.332 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:26:56 smithi092 ceph-mon[37563]: 6.a scrub starts 2024-03-11T21:26:57.332 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:26:56 smithi092 ceph-mon[37563]: 6.a scrub ok 2024-03-11T21:26:57.332 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:26:56 smithi092 ceph-mon[37563]: 2.17 scrub starts 2024-03-11T21:26:57.332 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:26:56 smithi092 ceph-mon[37563]: 2.17 scrub ok 2024-03-11T21:26:57.333 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:26:56 smithi092 ceph-mon[37563]: 2.1d scrub starts 2024-03-11T21:26:57.333 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:26:56 smithi092 ceph-mon[37563]: 2.1d scrub ok 2024-03-11T21:26:57.333 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:26:56 smithi092 ceph-mon[37563]: from='client.? v1:172.21.15.92:0/2082339182' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 6}]: dispatch 2024-03-11T21:26:57.828 INFO:tasks.cephadm.ceph_manager.ceph:need seq 214748364820 got 214748364824 for osd.6 2024-03-11T21:26:57.829 DEBUG:teuthology.parallel:result is None 2024-03-11T21:26:57.829 INFO:tasks.cephadm.ceph_manager.ceph:waiting for clean 2024-03-11T21:26:57.829 DEBUG:teuthology.orchestra.run.smithi092:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid 00db0016-dfed-11ee-95c8-87774f69a715 -- ceph pg dump --format=json 2024-03-11T21:26:58.111 INFO:journalctl@ceph.mgr.y.smithi092.stdout:Mar 11 21:26:57 smithi092 ceph-00db0016-dfed-11ee-95c8-87774f69a715-mgr-y[29801]: ::ffff:172.21.15.97 - - [11/Mar/2024:21:26:57] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-11T21:26:58.112 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:26:58 smithi092 ceph-mon[29572]: pgmap v28: 132 pgs: 132 active+clean; 582 KiB data, 228 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-11T21:26:58.112 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:26:58 smithi092 ceph-mon[29572]: 2.5 deep-scrub starts 2024-03-11T21:26:58.112 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:26:58 smithi092 ceph-mon[29572]: 2.5 deep-scrub ok 2024-03-11T21:26:58.112 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:26:58 smithi092 ceph-mon[29572]: 6.16 scrub starts 2024-03-11T21:26:58.112 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:26:58 smithi092 ceph-mon[29572]: 6.16 scrub ok 2024-03-11T21:26:58.113 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:26:58 smithi092 ceph-mon[37563]: pgmap v28: 132 pgs: 132 active+clean; 582 KiB data, 228 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-11T21:26:58.113 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:26:58 smithi092 ceph-mon[37563]: 2.5 deep-scrub starts 2024-03-11T21:26:58.113 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:26:58 smithi092 ceph-mon[37563]: 2.5 deep-scrub ok 2024-03-11T21:26:58.113 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:26:58 smithi092 ceph-mon[37563]: 6.16 scrub starts 2024-03-11T21:26:58.113 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:26:58 smithi092 ceph-mon[37563]: 6.16 scrub ok 2024-03-11T21:26:58.152 INFO:teuthology.orchestra.run.smithi092.stderr:Inferring config /var/lib/ceph/00db0016-dfed-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T21:26:58.327 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:26:58 smithi097 ceph-mon[33317]: pgmap v28: 132 pgs: 132 active+clean; 582 KiB data, 228 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-11T21:26:58.327 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:26:58 smithi097 ceph-mon[33317]: 2.5 deep-scrub starts 2024-03-11T21:26:58.327 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:26:58 smithi097 ceph-mon[33317]: 2.5 deep-scrub ok 2024-03-11T21:26:58.327 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:26:58 smithi097 ceph-mon[33317]: 6.16 scrub starts 2024-03-11T21:26:58.327 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:26:58 smithi097 ceph-mon[33317]: 6.16 scrub ok 2024-03-11T21:26:59.361 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:26:59 smithi092 ceph-mon[37563]: 4.e scrub starts 2024-03-11T21:26:59.361 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:26:59 smithi092 ceph-mon[37563]: 4.e scrub ok 2024-03-11T21:26:59.361 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:26:59 smithi092 ceph-mon[37563]: 2.e scrub starts 2024-03-11T21:26:59.361 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:26:59 smithi092 ceph-mon[37563]: 2.e scrub ok 2024-03-11T21:26:59.361 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:26:59 smithi092 ceph-mon[37563]: 5.1e deep-scrub starts 2024-03-11T21:26:59.362 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:26:59 smithi092 ceph-mon[37563]: 5.1e deep-scrub ok 2024-03-11T21:26:59.362 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:26:59 smithi092 ceph-mon[37563]: 5.4 scrub starts 2024-03-11T21:26:59.362 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:26:59 smithi092 ceph-mon[37563]: 5.4 scrub ok 2024-03-11T21:26:59.362 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:26:59 smithi092 ceph-mon[37563]: 4.10 deep-scrub starts 2024-03-11T21:26:59.362 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:26:59 smithi092 ceph-mon[37563]: 4.10 deep-scrub ok 2024-03-11T21:26:59.362 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:26:59 smithi092 ceph-mon[29572]: 4.e scrub starts 2024-03-11T21:26:59.362 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:26:59 smithi092 ceph-mon[29572]: 4.e scrub ok 2024-03-11T21:26:59.362 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:26:59 smithi092 ceph-mon[29572]: 2.e scrub starts 2024-03-11T21:26:59.362 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:26:59 smithi092 ceph-mon[29572]: 2.e scrub ok 2024-03-11T21:26:59.362 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:26:59 smithi092 ceph-mon[29572]: 5.1e deep-scrub starts 2024-03-11T21:26:59.362 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:26:59 smithi092 ceph-mon[29572]: 5.1e deep-scrub ok 2024-03-11T21:26:59.363 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:26:59 smithi092 ceph-mon[29572]: 5.4 scrub starts 2024-03-11T21:26:59.363 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:26:59 smithi092 ceph-mon[29572]: 5.4 scrub ok 2024-03-11T21:26:59.363 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:26:59 smithi092 ceph-mon[29572]: 4.10 deep-scrub starts 2024-03-11T21:26:59.363 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:26:59 smithi092 ceph-mon[29572]: 4.10 deep-scrub ok 2024-03-11T21:26:59.577 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:26:59 smithi097 ceph-mon[33317]: 4.e scrub starts 2024-03-11T21:26:59.577 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:26:59 smithi097 ceph-mon[33317]: 4.e scrub ok 2024-03-11T21:26:59.577 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:26:59 smithi097 ceph-mon[33317]: 2.e scrub starts 2024-03-11T21:26:59.577 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:26:59 smithi097 ceph-mon[33317]: 2.e scrub ok 2024-03-11T21:26:59.577 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:26:59 smithi097 ceph-mon[33317]: 5.1e deep-scrub starts 2024-03-11T21:26:59.578 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:26:59 smithi097 ceph-mon[33317]: 5.1e deep-scrub ok 2024-03-11T21:26:59.578 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:26:59 smithi097 ceph-mon[33317]: 5.4 scrub starts 2024-03-11T21:26:59.578 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:26:59 smithi097 ceph-mon[33317]: 5.4 scrub ok 2024-03-11T21:26:59.578 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:26:59 smithi097 ceph-mon[33317]: 4.10 deep-scrub starts 2024-03-11T21:26:59.578 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:26:59 smithi097 ceph-mon[33317]: 4.10 deep-scrub ok 2024-03-11T21:27:00.377 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:27:00 smithi092 ceph-mon[29572]: pgmap v29: 132 pgs: 132 active+clean; 582 KiB data, 228 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T21:27:00.377 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:27:00 smithi092 ceph-mon[29572]: 4.15 deep-scrub starts 2024-03-11T21:27:00.377 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:27:00 smithi092 ceph-mon[29572]: 4.15 deep-scrub ok 2024-03-11T21:27:00.377 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:27:00 smithi092 ceph-mon[29572]: 5.d scrub starts 2024-03-11T21:27:00.377 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:27:00 smithi092 ceph-mon[29572]: 5.d scrub ok 2024-03-11T21:27:00.377 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:27:00 smithi092 ceph-mon[29572]: 2.14 scrub starts 2024-03-11T21:27:00.377 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:27:00 smithi092 ceph-mon[29572]: 2.14 scrub ok 2024-03-11T21:27:00.377 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:27:00 smithi092 ceph-mon[29572]: 6.1 scrub starts 2024-03-11T21:27:00.377 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:27:00 smithi092 ceph-mon[29572]: 6.1 scrub ok 2024-03-11T21:27:00.378 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:27:00 smithi092 ceph-mon[37563]: pgmap v29: 132 pgs: 132 active+clean; 582 KiB data, 228 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T21:27:00.378 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:27:00 smithi092 ceph-mon[37563]: 4.15 deep-scrub starts 2024-03-11T21:27:00.378 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:27:00 smithi092 ceph-mon[37563]: 4.15 deep-scrub ok 2024-03-11T21:27:00.378 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:27:00 smithi092 ceph-mon[37563]: 5.d scrub starts 2024-03-11T21:27:00.378 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:27:00 smithi092 ceph-mon[37563]: 5.d scrub ok 2024-03-11T21:27:00.378 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:27:00 smithi092 ceph-mon[37563]: 2.14 scrub starts 2024-03-11T21:27:00.378 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:27:00 smithi092 ceph-mon[37563]: 2.14 scrub ok 2024-03-11T21:27:00.378 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:27:00 smithi092 ceph-mon[37563]: 6.1 scrub starts 2024-03-11T21:27:00.378 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:27:00 smithi092 ceph-mon[37563]: 6.1 scrub ok 2024-03-11T21:27:00.577 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:27:00 smithi097 ceph-mon[33317]: pgmap v29: 132 pgs: 132 active+clean; 582 KiB data, 228 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T21:27:00.577 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:27:00 smithi097 ceph-mon[33317]: 4.15 deep-scrub starts 2024-03-11T21:27:00.577 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:27:00 smithi097 ceph-mon[33317]: 4.15 deep-scrub ok 2024-03-11T21:27:00.577 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:27:00 smithi097 ceph-mon[33317]: 5.d scrub starts 2024-03-11T21:27:00.577 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:27:00 smithi097 ceph-mon[33317]: 5.d scrub ok 2024-03-11T21:27:00.577 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:27:00 smithi097 ceph-mon[33317]: 2.14 scrub starts 2024-03-11T21:27:00.577 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:27:00 smithi097 ceph-mon[33317]: 2.14 scrub ok 2024-03-11T21:27:00.577 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:27:00 smithi097 ceph-mon[33317]: 6.1 scrub starts 2024-03-11T21:27:00.577 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:27:00 smithi097 ceph-mon[33317]: 6.1 scrub ok 2024-03-11T21:27:00.630 INFO:teuthology.orchestra.run.smithi092.stdout: 2024-03-11T21:27:00.633 INFO:teuthology.orchestra.run.smithi092.stderr:dumped all 2024-03-11T21:27:01.361 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:27:01 smithi092 ceph-mon[37563]: 6.4 scrub starts 2024-03-11T21:27:01.361 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:27:01 smithi092 ceph-mon[37563]: 6.4 scrub ok 2024-03-11T21:27:01.362 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:27:01 smithi092 ceph-mon[37563]: 6.15 scrub starts 2024-03-11T21:27:01.362 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:27:01 smithi092 ceph-mon[37563]: 6.15 scrub ok 2024-03-11T21:27:01.362 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:27:01 smithi092 ceph-mon[37563]: from='client.14751 v1:172.21.15.92:0/3561928629' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-11T21:27:01.363 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:27:01 smithi092 ceph-mon[29572]: 6.4 scrub starts 2024-03-11T21:27:01.363 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:27:01 smithi092 ceph-mon[29572]: 6.4 scrub ok 2024-03-11T21:27:01.363 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:27:01 smithi092 ceph-mon[29572]: 6.15 scrub starts 2024-03-11T21:27:01.363 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:27:01 smithi092 ceph-mon[29572]: 6.15 scrub ok 2024-03-11T21:27:01.363 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:27:01 smithi092 ceph-mon[29572]: from='client.14751 v1:172.21.15.92:0/3561928629' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-11T21:27:01.392 INFO:teuthology.orchestra.run.smithi092.stdout:{"pg_ready":true,"pg_map":{"version":29,"stamp":"2024-03-11T21:26:58.964072+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":596392,"num_objects":198,"num_object_clones":0,"num_object_copies":594,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":198,"num_whiteouts":0,"num_read":1007,"num_read_kb":853,"num_write":615,"num_write_kb":2461,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":21,"num_bytes_recovered":1771324,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":614,"ondisk_log_size":614,"up":396,"acting":396,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":396,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":749961216,"kb_used":233084,"kb_used_data":8636,"kb_used_omap":12,"kb_used_meta":224371,"kb_avail":749728132,"statfs":{"total":767960285184,"available":767721607168,"internally_reserved":0,"allocated":8843264,"data_stored":4867456,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12752,"internal_metadata":229756464},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":15,"num_read_kb":15,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003250"},"pg_stats":[{"pgid":"6.1b","version":"0'0","reported_seq":61,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-11T21:26:53.584377+0000","last_change":"2024-03-11T21:26:53.584399+0000","last_active":"2024-03-11T21:26:53.584377+0000","last_peered":"2024-03-11T21:26:53.584377+0000","last_clean":"2024-03-11T21:26:53.584377+0000","last_became_active":"2024-03-11T21:25:45.302931+0000","last_became_peered":"2024-03-11T21:25:45.302931+0000","last_unstale":"2024-03-11T21:26:53.584377+0000","last_undegraded":"2024-03-11T21:26:53.584377+0000","last_fullsized":"2024-03-11T21:26:53.584377+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:26:53.584307+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:43.559452+0000","last_clean_scrub_stamp":"2024-03-11T21:26:53.584307+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:27:55.466757+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":57,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-11T21:26:43.123284+0000","last_change":"2024-03-11T21:25:42.579254+0000","last_active":"2024-03-11T21:26:43.123284+0000","last_peered":"2024-03-11T21:26:43.123284+0000","last_clean":"2024-03-11T21:26:43.123284+0000","last_became_active":"2024-03-11T21:25:42.578893+0000","last_became_peered":"2024-03-11T21:25:42.578893+0000","last_unstale":"2024-03-11T21:26:43.123284+0000","last_undegraded":"2024-03-11T21:26:43.123284+0000","last_fullsized":"2024-03-11T21:26:43.123284+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:25:41.544550+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:41.544550+0000","last_clean_scrub_stamp":"2024-03-11T21:25:41.544550+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:27:10.983445+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1f","version":"0'0","reported_seq":73,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-11T21:26:46.588443+0000","last_change":"2024-03-11T21:26:46.588463+0000","last_active":"2024-03-11T21:26:46.588443+0000","last_peered":"2024-03-11T21:26:46.588443+0000","last_clean":"2024-03-11T21:26:46.588443+0000","last_became_active":"2024-03-11T21:25:38.540510+0000","last_became_peered":"2024-03-11T21:25:38.540510+0000","last_unstale":"2024-03-11T21:26:46.588443+0000","last_undegraded":"2024-03-11T21:26:46.588443+0000","last_fullsized":"2024-03-11T21:26:46.588443+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:26:46.588380+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:37.512018+0000","last_clean_scrub_stamp":"2024-03-11T21:26:46.588380+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:28:06.851002+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,4],"acting":[0,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.19","version":"74'15","reported_seq":86,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-11T21:26:43.661037+0000","last_change":"2024-03-11T21:25:40.554584+0000","last_active":"2024-03-11T21:26:43.661037+0000","last_peered":"2024-03-11T21:26:43.661037+0000","last_clean":"2024-03-11T21:26:43.661037+0000","last_became_active":"2024-03-11T21:25:40.554207+0000","last_became_peered":"2024-03-11T21:25:40.554207+0000","last_unstale":"2024-03-11T21:26:43.661037+0000","last_undegraded":"2024-03-11T21:26:43.661037+0000","last_fullsized":"2024-03-11T21:26:43.661037+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:25:39.525554+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:39.525554+0000","last_clean_scrub_stamp":"2024-03-11T21:25:39.525554+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:26:54.679300+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.1a","version":"0'0","reported_seq":53,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-11T21:26:43.123537+0000","last_change":"2024-03-11T21:25:44.591548+0000","last_active":"2024-03-11T21:26:43.123537+0000","last_peered":"2024-03-11T21:26:43.123537+0000","last_clean":"2024-03-11T21:26:43.123537+0000","last_became_active":"2024-03-11T21:25:44.590923+0000","last_became_peered":"2024-03-11T21:25:44.590923+0000","last_unstale":"2024-03-11T21:26:43.123537+0000","last_undegraded":"2024-03-11T21:26:43.123537+0000","last_fullsized":"2024-03-11T21:26:43.123537+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:25:43.559452+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:43.559452+0000","last_clean_scrub_stamp":"2024-03-11T21:25:43.559452+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:27:00.390815+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,1],"acting":[4,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":57,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-11T21:26:43.122994+0000","last_change":"2024-03-11T21:25:42.578761+0000","last_active":"2024-03-11T21:26:43.122994+0000","last_peered":"2024-03-11T21:26:43.122994+0000","last_clean":"2024-03-11T21:26:43.122994+0000","last_became_active":"2024-03-11T21:25:42.578424+0000","last_became_peered":"2024-03-11T21:25:42.578424+0000","last_unstale":"2024-03-11T21:26:43.122994+0000","last_undegraded":"2024-03-11T21:26:43.122994+0000","last_fullsized":"2024-03-11T21:26:43.122994+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:25:41.544550+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:41.544550+0000","last_clean_scrub_stamp":"2024-03-11T21:25:41.544550+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:27:05.977939+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,7],"acting":[1,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1e","version":"0'0","reported_seq":73,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-11T21:26:49.595438+0000","last_change":"2024-03-11T21:26:49.595465+0000","last_active":"2024-03-11T21:26:49.595438+0000","last_peered":"2024-03-11T21:26:49.595438+0000","last_clean":"2024-03-11T21:26:49.595438+0000","last_became_active":"2024-03-11T21:25:38.537283+0000","last_became_peered":"2024-03-11T21:25:38.537283+0000","last_unstale":"2024-03-11T21:26:49.595438+0000","last_undegraded":"2024-03-11T21:26:49.595438+0000","last_fullsized":"2024-03-11T21:26:49.595438+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:26:49.595363+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:37.512018+0000","last_clean_scrub_stamp":"2024-03-11T21:26:49.595363+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:27:52.155430+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.18","version":"74'9","reported_seq":85,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-11T21:26:45.870036+0000","last_change":"2024-03-11T21:26:45.870099+0000","last_active":"2024-03-11T21:26:45.870036+0000","last_peered":"2024-03-11T21:26:45.870036+0000","last_clean":"2024-03-11T21:26:45.870036+0000","last_became_active":"2024-03-11T21:25:40.552216+0000","last_became_peered":"2024-03-11T21:25:40.552216+0000","last_unstale":"2024-03-11T21:26:45.870036+0000","last_undegraded":"2024-03-11T21:26:45.870036+0000","last_fullsized":"2024-03-11T21:26:45.870036+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'9","last_scrub_stamp":"2024-03-11T21:26:45.869943+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:39.525554+0000","last_clean_scrub_stamp":"2024-03-11T21:26:45.869943+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:27:50.731066+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":53,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-11T21:26:44.103957+0000","last_change":"2024-03-11T21:25:44.590541+0000","last_active":"2024-03-11T21:26:44.103957+0000","last_peered":"2024-03-11T21:26:44.103957+0000","last_clean":"2024-03-11T21:26:44.103957+0000","last_became_active":"2024-03-11T21:25:44.590234+0000","last_became_peered":"2024-03-11T21:25:44.590234+0000","last_unstale":"2024-03-11T21:26:44.103957+0000","last_undegraded":"2024-03-11T21:26:44.103957+0000","last_fullsized":"2024-03-11T21:26:44.103957+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:25:43.559452+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:43.559452+0000","last_clean_scrub_stamp":"2024-03-11T21:25:43.559452+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:26:54.344730+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3],"acting":[5,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":57,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-11T21:26:45.417397+0000","last_change":"2024-03-11T21:25:42.580998+0000","last_active":"2024-03-11T21:26:45.417397+0000","last_peered":"2024-03-11T21:26:45.417397+0000","last_clean":"2024-03-11T21:26:45.417397+0000","last_became_active":"2024-03-11T21:25:42.580533+0000","last_became_peered":"2024-03-11T21:25:42.580533+0000","last_unstale":"2024-03-11T21:26:45.417397+0000","last_undegraded":"2024-03-11T21:26:45.417397+0000","last_fullsized":"2024-03-11T21:26:45.417397+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:25:41.544550+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:41.544550+0000","last_clean_scrub_stamp":"2024-03-11T21:25:41.544550+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:27:06.038044+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.1d","version":"0'0","reported_seq":73,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-11T21:26:56.370884+0000","last_change":"2024-03-11T21:26:56.370911+0000","last_active":"2024-03-11T21:26:56.370884+0000","last_peered":"2024-03-11T21:26:56.370884+0000","last_clean":"2024-03-11T21:26:56.370884+0000","last_became_active":"2024-03-11T21:25:38.542628+0000","last_became_peered":"2024-03-11T21:25:38.542628+0000","last_unstale":"2024-03-11T21:26:56.370884+0000","last_undegraded":"2024-03-11T21:26:56.370884+0000","last_fullsized":"2024-03-11T21:26:56.370884+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:26:56.370811+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:37.512018+0000","last_clean_scrub_stamp":"2024-03-11T21:26:56.370811+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:28:03.069431+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,0],"acting":[7,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.1b","version":"74'5","reported_seq":71,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-11T21:26:43.123759+0000","last_change":"2024-03-11T21:25:40.556136+0000","last_active":"2024-03-11T21:26:43.123759+0000","last_peered":"2024-03-11T21:26:43.123759+0000","last_clean":"2024-03-11T21:26:43.123759+0000","last_became_active":"2024-03-11T21:25:40.555826+0000","last_became_peered":"2024-03-11T21:25:40.555826+0000","last_unstale":"2024-03-11T21:26:43.123759+0000","last_undegraded":"2024-03-11T21:26:43.123759+0000","last_fullsized":"2024-03-11T21:26:43.123759+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:25:39.525554+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:39.525554+0000","last_clean_scrub_stamp":"2024-03-11T21:25:39.525554+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:27:08.490979+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":11,"num_read_kb":7,"num_write":6,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,1],"acting":[4,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":53,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-11T21:26:43.122899+0000","last_change":"2024-03-11T21:25:44.590228+0000","last_active":"2024-03-11T21:26:43.122899+0000","last_peered":"2024-03-11T21:26:43.122899+0000","last_clean":"2024-03-11T21:26:43.122899+0000","last_became_active":"2024-03-11T21:25:44.589786+0000","last_became_peered":"2024-03-11T21:25:44.589786+0000","last_unstale":"2024-03-11T21:26:43.122899+0000","last_undegraded":"2024-03-11T21:26:43.122899+0000","last_fullsized":"2024-03-11T21:26:43.122899+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:25:43.559452+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:43.559452+0000","last_clean_scrub_stamp":"2024-03-11T21:25:43.559452+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:27:12.142655+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,7],"acting":[0,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":57,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-11T21:26:44.104432+0000","last_change":"2024-03-11T21:25:42.574391+0000","last_active":"2024-03-11T21:26:44.104432+0000","last_peered":"2024-03-11T21:26:44.104432+0000","last_clean":"2024-03-11T21:26:44.104432+0000","last_became_active":"2024-03-11T21:25:42.573831+0000","last_became_peered":"2024-03-11T21:25:42.573831+0000","last_unstale":"2024-03-11T21:26:44.104432+0000","last_undegraded":"2024-03-11T21:26:44.104432+0000","last_fullsized":"2024-03-11T21:26:44.104432+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:25:41.544550+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:41.544550+0000","last_clean_scrub_stamp":"2024-03-11T21:25:41.544550+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:26:53.725038+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,7],"acting":[5,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1c","version":"0'0","reported_seq":65,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-11T21:26:45.417664+0000","last_change":"2024-03-11T21:25:38.542244+0000","last_active":"2024-03-11T21:26:45.417664+0000","last_peered":"2024-03-11T21:26:45.417664+0000","last_clean":"2024-03-11T21:26:45.417664+0000","last_became_active":"2024-03-11T21:25:38.541364+0000","last_became_peered":"2024-03-11T21:25:38.541364+0000","last_unstale":"2024-03-11T21:26:45.417664+0000","last_undegraded":"2024-03-11T21:26:45.417664+0000","last_fullsized":"2024-03-11T21:26:45.417664+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:25:37.512018+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:37.512018+0000","last_clean_scrub_stamp":"2024-03-11T21:25:37.512018+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:27:03.620571+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.1a","version":"74'9","reported_seq":85,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-11T21:26:44.851036+0000","last_change":"2024-03-11T21:26:44.851070+0000","last_active":"2024-03-11T21:26:44.851036+0000","last_peered":"2024-03-11T21:26:44.851036+0000","last_clean":"2024-03-11T21:26:44.851036+0000","last_became_active":"2024-03-11T21:25:40.556629+0000","last_became_peered":"2024-03-11T21:25:40.556629+0000","last_unstale":"2024-03-11T21:26:44.851036+0000","last_undegraded":"2024-03-11T21:26:44.851036+0000","last_fullsized":"2024-03-11T21:26:44.851036+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'9","last_scrub_stamp":"2024-03-11T21:26:44.850930+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:39.525554+0000","last_clean_scrub_stamp":"2024-03-11T21:26:44.850930+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:27:47.017552+0000","scrub_duration":24,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0],"acting":[4,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1f","version":"0'0","reported_seq":53,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-11T21:26:43.661614+0000","last_change":"2024-03-11T21:25:45.303017+0000","last_active":"2024-03-11T21:26:43.661614+0000","last_peered":"2024-03-11T21:26:43.661614+0000","last_clean":"2024-03-11T21:26:43.661614+0000","last_became_active":"2024-03-11T21:25:45.302637+0000","last_became_peered":"2024-03-11T21:25:45.302637+0000","last_unstale":"2024-03-11T21:26:43.661614+0000","last_undegraded":"2024-03-11T21:26:43.661614+0000","last_fullsized":"2024-03-11T21:26:43.661614+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:25:43.559452+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:43.559452+0000","last_clean_scrub_stamp":"2024-03-11T21:25:43.559452+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:26:53.655260+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1c","version":"0'0","reported_seq":57,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-11T21:26:43.123925+0000","last_change":"2024-03-11T21:25:42.576745+0000","last_active":"2024-03-11T21:26:43.123925+0000","last_peered":"2024-03-11T21:26:43.123925+0000","last_clean":"2024-03-11T21:26:43.123925+0000","last_became_active":"2024-03-11T21:25:42.576064+0000","last_became_peered":"2024-03-11T21:25:42.576064+0000","last_unstale":"2024-03-11T21:26:43.123925+0000","last_undegraded":"2024-03-11T21:26:43.123925+0000","last_fullsized":"2024-03-11T21:26:43.123925+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:25:41.544550+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:41.544550+0000","last_clean_scrub_stamp":"2024-03-11T21:25:41.544550+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:26:53.587425+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1b","version":"0'0","reported_seq":73,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-11T21:26:55.579961+0000","last_change":"2024-03-11T21:26:55.579982+0000","last_active":"2024-03-11T21:26:55.579961+0000","last_peered":"2024-03-11T21:26:55.579961+0000","last_clean":"2024-03-11T21:26:55.579961+0000","last_became_active":"2024-03-11T21:25:38.537520+0000","last_became_peered":"2024-03-11T21:25:38.537520+0000","last_unstale":"2024-03-11T21:26:55.579961+0000","last_undegraded":"2024-03-11T21:26:55.579961+0000","last_fullsized":"2024-03-11T21:26:55.579961+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:26:55.579901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:37.512018+0000","last_clean_scrub_stamp":"2024-03-11T21:26:55.579901+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:28:17.884453+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1d","version":"74'12","reported_seq":84,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-11T21:26:43.661355+0000","last_change":"2024-03-11T21:25:40.556651+0000","last_active":"2024-03-11T21:26:43.661355+0000","last_peered":"2024-03-11T21:26:43.661355+0000","last_clean":"2024-03-11T21:26:43.661355+0000","last_became_active":"2024-03-11T21:25:40.556330+0000","last_became_peered":"2024-03-11T21:25:40.556330+0000","last_unstale":"2024-03-11T21:26:43.661355+0000","last_undegraded":"2024-03-11T21:26:43.661355+0000","last_fullsized":"2024-03-11T21:26:43.661355+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:25:39.525554+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:39.525554+0000","last_clean_scrub_stamp":"2024-03-11T21:25:39.525554+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:26:52.042075+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":25,"num_read_kb":16,"num_write":14,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.1e","version":"0'0","reported_seq":53,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-11T21:26:43.124283+0000","last_change":"2024-03-11T21:25:45.300689+0000","last_active":"2024-03-11T21:26:43.124283+0000","last_peered":"2024-03-11T21:26:43.124283+0000","last_clean":"2024-03-11T21:26:43.124283+0000","last_became_active":"2024-03-11T21:25:45.300355+0000","last_became_peered":"2024-03-11T21:25:45.300355+0000","last_unstale":"2024-03-11T21:26:43.124283+0000","last_undegraded":"2024-03-11T21:26:43.124283+0000","last_fullsized":"2024-03-11T21:26:43.124283+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:25:43.559452+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:43.559452+0000","last_clean_scrub_stamp":"2024-03-11T21:25:43.559452+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:27:05.215620+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5],"acting":[4,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1d","version":"0'0","reported_seq":57,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-11T21:26:43.124268+0000","last_change":"2024-03-11T21:25:42.579792+0000","last_active":"2024-03-11T21:26:43.124268+0000","last_peered":"2024-03-11T21:26:43.124268+0000","last_clean":"2024-03-11T21:26:43.124268+0000","last_became_active":"2024-03-11T21:25:42.579407+0000","last_became_peered":"2024-03-11T21:25:42.579407+0000","last_unstale":"2024-03-11T21:26:43.124268+0000","last_undegraded":"2024-03-11T21:26:43.124268+0000","last_fullsized":"2024-03-11T21:26:43.124268+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:25:41.544550+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:41.544550+0000","last_clean_scrub_stamp":"2024-03-11T21:25:41.544550+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:27:03.100488+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1a","version":"0'0","reported_seq":65,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-11T21:26:44.037494+0000","last_change":"2024-03-11T21:25:38.540007+0000","last_active":"2024-03-11T21:26:44.037494+0000","last_peered":"2024-03-11T21:26:44.037494+0000","last_clean":"2024-03-11T21:26:44.037494+0000","last_became_active":"2024-03-11T21:25:38.539725+0000","last_became_peered":"2024-03-11T21:25:38.539725+0000","last_unstale":"2024-03-11T21:26:44.037494+0000","last_undegraded":"2024-03-11T21:26:44.037494+0000","last_fullsized":"2024-03-11T21:26:44.037494+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:25:37.512018+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:37.512018+0000","last_clean_scrub_stamp":"2024-03-11T21:25:37.512018+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:26:56.284995+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.1c","version":"74'15","reported_seq":86,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-11T21:26:43.291216+0000","last_change":"2024-03-11T21:25:40.551757+0000","last_active":"2024-03-11T21:26:43.291216+0000","last_peered":"2024-03-11T21:26:43.291216+0000","last_clean":"2024-03-11T21:26:43.291216+0000","last_became_active":"2024-03-11T21:25:40.551453+0000","last_became_peered":"2024-03-11T21:25:40.551453+0000","last_unstale":"2024-03-11T21:26:43.291216+0000","last_undegraded":"2024-03-11T21:26:43.291216+0000","last_fullsized":"2024-03-11T21:26:43.291216+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:25:39.525554+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:39.525554+0000","last_clean_scrub_stamp":"2024-03-11T21:25:39.525554+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:26:56.664246+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.1d","version":"0'0","reported_seq":53,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-11T21:26:43.123803+0000","last_change":"2024-03-11T21:25:44.589198+0000","last_active":"2024-03-11T21:26:43.123803+0000","last_peered":"2024-03-11T21:26:43.123803+0000","last_clean":"2024-03-11T21:26:43.123803+0000","last_became_active":"2024-03-11T21:25:44.588845+0000","last_became_peered":"2024-03-11T21:25:44.588845+0000","last_unstale":"2024-03-11T21:26:43.123803+0000","last_undegraded":"2024-03-11T21:26:43.123803+0000","last_fullsized":"2024-03-11T21:26:43.123803+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:25:43.559452+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:43.559452+0000","last_clean_scrub_stamp":"2024-03-11T21:25:43.559452+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:27:08.822378+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.1e","version":"0'0","reported_seq":57,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-11T21:26:43.123437+0000","last_change":"2024-03-11T21:25:42.583314+0000","last_active":"2024-03-11T21:26:43.123437+0000","last_peered":"2024-03-11T21:26:43.123437+0000","last_clean":"2024-03-11T21:26:43.123437+0000","last_became_active":"2024-03-11T21:25:42.582959+0000","last_became_peered":"2024-03-11T21:25:42.582959+0000","last_unstale":"2024-03-11T21:26:43.123437+0000","last_undegraded":"2024-03-11T21:26:43.123437+0000","last_fullsized":"2024-03-11T21:26:43.123437+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:25:41.544550+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:41.544550+0000","last_clean_scrub_stamp":"2024-03-11T21:25:41.544550+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:26:51.321555+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.19","version":"67'2","reported_seq":72,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-11T21:26:43.661117+0000","last_change":"2024-03-11T21:25:38.539600+0000","last_active":"2024-03-11T21:26:43.661117+0000","last_peered":"2024-03-11T21:26:43.661117+0000","last_clean":"2024-03-11T21:26:43.661117+0000","last_became_active":"2024-03-11T21:25:38.539243+0000","last_became_peered":"2024-03-11T21:25:38.539243+0000","last_unstale":"2024-03-11T21:26:43.661117+0000","last_undegraded":"2024-03-11T21:26:43.661117+0000","last_fullsized":"2024-03-11T21:26:43.661117+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:25:37.512018+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:37.512018+0000","last_clean_scrub_stamp":"2024-03-11T21:25:37.512018+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:26:59.905535+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":957,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":7,"num_read_kb":7,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,0],"acting":[3,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1f","version":"74'11","reported_seq":88,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-11T21:26:55.050937+0000","last_change":"2024-03-11T21:26:55.050980+0000","last_active":"2024-03-11T21:26:55.050937+0000","last_peered":"2024-03-11T21:26:55.050937+0000","last_clean":"2024-03-11T21:26:55.050937+0000","last_became_active":"2024-03-11T21:25:40.554686+0000","last_became_peered":"2024-03-11T21:25:40.554686+0000","last_unstale":"2024-03-11T21:26:55.050937+0000","last_undegraded":"2024-03-11T21:26:55.050937+0000","last_fullsized":"2024-03-11T21:26:55.050937+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'11","last_scrub_stamp":"2024-03-11T21:26:55.050872+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:39.525554+0000","last_clean_scrub_stamp":"2024-03-11T21:26:55.050872+0000","objects_scrubbed":5,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:28:13.716793+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,1],"acting":[6,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.1c","version":"74'1","reported_seq":54,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-11T21:26:45.418002+0000","last_change":"2024-03-11T21:25:44.593623+0000","last_active":"2024-03-11T21:26:45.418002+0000","last_peered":"2024-03-11T21:26:45.418002+0000","last_clean":"2024-03-11T21:26:45.418002+0000","last_became_active":"2024-03-11T21:25:44.593326+0000","last_became_peered":"2024-03-11T21:25:44.593326+0000","last_unstale":"2024-03-11T21:26:45.418002+0000","last_undegraded":"2024-03-11T21:26:45.418002+0000","last_fullsized":"2024-03-11T21:26:45.418002+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:25:43.559452+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:43.559452+0000","last_clean_scrub_stamp":"2024-03-11T21:25:43.559452+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:27:07.557915+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":370,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.1f","version":"74'8","reported_seq":92,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-11T21:26:48.955663+0000","last_change":"2024-03-11T21:26:48.955663+0000","last_active":"2024-03-11T21:26:48.955663+0000","last_peered":"2024-03-11T21:26:48.955663+0000","last_clean":"2024-03-11T21:26:48.955663+0000","last_became_active":"2024-03-11T21:25:42.577401+0000","last_became_peered":"2024-03-11T21:25:42.577401+0000","last_unstale":"2024-03-11T21:26:48.955663+0000","last_undegraded":"2024-03-11T21:26:48.955663+0000","last_fullsized":"2024-03-11T21:26:48.955663+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'8","last_scrub_stamp":"2024-03-11T21:26:48.955484+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:41.544550+0000","last_clean_scrub_stamp":"2024-03-11T21:26:48.955484+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:27:51.651922+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.18","version":"67'1","reported_seq":79,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-11T21:26:46.995320+0000","last_change":"2024-03-11T21:26:46.995375+0000","last_active":"2024-03-11T21:26:46.995320+0000","last_peered":"2024-03-11T21:26:46.995320+0000","last_clean":"2024-03-11T21:26:46.995320+0000","last_became_active":"2024-03-11T21:25:38.538321+0000","last_became_peered":"2024-03-11T21:25:38.538321+0000","last_unstale":"2024-03-11T21:26:46.995320+0000","last_undegraded":"2024-03-11T21:26:46.995320+0000","last_fullsized":"2024-03-11T21:26:46.995320+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'1","last_scrub_stamp":"2024-03-11T21:26:46.995228+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:37.512018+0000","last_clean_scrub_stamp":"2024-03-11T21:26:46.995228+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:27:58.922326+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":436,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":7,"num_read_kb":7,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,7],"acting":[5,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1e","version":"74'10","reported_seq":84,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-11T21:26:43.123360+0000","last_change":"2024-03-11T21:26:42.521121+0000","last_active":"2024-03-11T21:26:43.123360+0000","last_peered":"2024-03-11T21:26:43.123360+0000","last_clean":"2024-03-11T21:26:43.123360+0000","last_became_active":"2024-03-11T21:25:40.763438+0000","last_became_peered":"2024-03-11T21:25:40.763438+0000","last_unstale":"2024-03-11T21:26:43.123360+0000","last_undegraded":"2024-03-11T21:26:43.123360+0000","last_fullsized":"2024-03-11T21:26:43.123360+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'10","last_scrub_stamp":"2024-03-11T21:26:42.521016+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:39.525554+0000","last_clean_scrub_stamp":"2024-03-11T21:26:42.521016+0000","objects_scrubbed":5,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:28:12.491436+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.13","version":"0'0","reported_seq":53,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-11T21:26:43.661347+0000","last_change":"2024-03-11T21:25:45.302505+0000","last_active":"2024-03-11T21:26:43.661347+0000","last_peered":"2024-03-11T21:26:43.661347+0000","last_clean":"2024-03-11T21:26:43.661347+0000","last_became_active":"2024-03-11T21:25:45.302102+0000","last_became_peered":"2024-03-11T21:25:45.302102+0000","last_unstale":"2024-03-11T21:26:43.661347+0000","last_undegraded":"2024-03-11T21:26:43.661347+0000","last_fullsized":"2024-03-11T21:26:43.661347+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:25:43.559452+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:43.559452+0000","last_clean_scrub_stamp":"2024-03-11T21:25:43.559452+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:26:53.208128+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":57,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-11T21:26:45.417478+0000","last_change":"2024-03-11T21:25:42.580085+0000","last_active":"2024-03-11T21:26:45.417478+0000","last_peered":"2024-03-11T21:26:45.417478+0000","last_clean":"2024-03-11T21:26:45.417478+0000","last_became_active":"2024-03-11T21:25:42.579781+0000","last_became_peered":"2024-03-11T21:25:42.579781+0000","last_unstale":"2024-03-11T21:26:45.417478+0000","last_undegraded":"2024-03-11T21:26:45.417478+0000","last_fullsized":"2024-03-11T21:26:45.417478+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:25:41.544550+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:41.544550+0000","last_clean_scrub_stamp":"2024-03-11T21:25:41.544550+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:27:05.067119+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.17","version":"0'0","reported_seq":73,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-11T21:26:56.020287+0000","last_change":"2024-03-11T21:26:56.020325+0000","last_active":"2024-03-11T21:26:56.020287+0000","last_peered":"2024-03-11T21:26:56.020287+0000","last_clean":"2024-03-11T21:26:56.020287+0000","last_became_active":"2024-03-11T21:25:38.540349+0000","last_became_peered":"2024-03-11T21:25:38.540349+0000","last_unstale":"2024-03-11T21:26:56.020287+0000","last_undegraded":"2024-03-11T21:26:56.020287+0000","last_fullsized":"2024-03-11T21:26:56.020287+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:26:56.020191+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:37.512018+0000","last_clean_scrub_stamp":"2024-03-11T21:26:56.020191+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:28:25.054065+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,2],"acting":[6,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.11","version":"74'11","reported_seq":80,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-11T21:26:43.661547+0000","last_change":"2024-03-11T21:25:40.761012+0000","last_active":"2024-03-11T21:26:43.661547+0000","last_peered":"2024-03-11T21:26:43.661547+0000","last_clean":"2024-03-11T21:26:43.661547+0000","last_became_active":"2024-03-11T21:25:40.760664+0000","last_became_peered":"2024-03-11T21:25:40.760664+0000","last_unstale":"2024-03-11T21:26:43.661547+0000","last_undegraded":"2024-03-11T21:26:43.661547+0000","last_fullsized":"2024-03-11T21:26:43.661547+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:25:39.525554+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:39.525554+0000","last_clean_scrub_stamp":"2024-03-11T21:25:39.525554+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:26:57.124438+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":53,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-11T21:26:45.417330+0000","last_change":"2024-03-11T21:25:44.591375+0000","last_active":"2024-03-11T21:26:45.417330+0000","last_peered":"2024-03-11T21:26:45.417330+0000","last_clean":"2024-03-11T21:26:45.417330+0000","last_became_active":"2024-03-11T21:25:44.591068+0000","last_became_peered":"2024-03-11T21:25:44.591068+0000","last_unstale":"2024-03-11T21:26:45.417330+0000","last_undegraded":"2024-03-11T21:26:45.417330+0000","last_fullsized":"2024-03-11T21:26:45.417330+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:25:43.559452+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:43.559452+0000","last_clean_scrub_stamp":"2024-03-11T21:25:43.559452+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:27:04.984773+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,4],"acting":[7,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":57,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-11T21:26:44.037422+0000","last_change":"2024-03-11T21:25:42.584581+0000","last_active":"2024-03-11T21:26:44.037422+0000","last_peered":"2024-03-11T21:26:44.037422+0000","last_clean":"2024-03-11T21:26:44.037422+0000","last_became_active":"2024-03-11T21:25:42.584148+0000","last_became_peered":"2024-03-11T21:25:42.584148+0000","last_unstale":"2024-03-11T21:26:44.037422+0000","last_undegraded":"2024-03-11T21:26:44.037422+0000","last_fullsized":"2024-03-11T21:26:44.037422+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:25:41.544550+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:41.544550+0000","last_clean_scrub_stamp":"2024-03-11T21:25:41.544550+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:27:00.403188+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.16","version":"0'0","reported_seq":65,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-11T21:26:44.104846+0000","last_change":"2024-03-11T21:25:38.539399+0000","last_active":"2024-03-11T21:26:44.104846+0000","last_peered":"2024-03-11T21:26:44.104846+0000","last_clean":"2024-03-11T21:26:44.104846+0000","last_became_active":"2024-03-11T21:25:38.539103+0000","last_became_peered":"2024-03-11T21:25:38.539103+0000","last_unstale":"2024-03-11T21:26:44.104846+0000","last_undegraded":"2024-03-11T21:26:44.104846+0000","last_fullsized":"2024-03-11T21:26:44.104846+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:25:37.512018+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:37.512018+0000","last_clean_scrub_stamp":"2024-03-11T21:25:37.512018+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:26:51.659287+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,2],"acting":[5,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.10","version":"74'4","reported_seq":67,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-11T21:26:43.661019+0000","last_change":"2024-03-11T21:25:40.553542+0000","last_active":"2024-03-11T21:26:43.661019+0000","last_peered":"2024-03-11T21:26:43.661019+0000","last_clean":"2024-03-11T21:26:43.661019+0000","last_became_active":"2024-03-11T21:25:40.552720+0000","last_became_peered":"2024-03-11T21:25:40.552720+0000","last_unstale":"2024-03-11T21:26:43.661019+0000","last_undegraded":"2024-03-11T21:26:43.661019+0000","last_fullsized":"2024-03-11T21:26:43.661019+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:25:39.525554+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:39.525554+0000","last_clean_scrub_stamp":"2024-03-11T21:25:39.525554+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:26:48.131413+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":6,"num_read_kb":4,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6],"acting":[3,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":53,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-11T21:26:43.660898+0000","last_change":"2024-03-11T21:25:44.583516+0000","last_active":"2024-03-11T21:26:43.660898+0000","last_peered":"2024-03-11T21:26:43.660898+0000","last_clean":"2024-03-11T21:26:43.660898+0000","last_became_active":"2024-03-11T21:25:44.583181+0000","last_became_peered":"2024-03-11T21:25:44.583181+0000","last_unstale":"2024-03-11T21:26:43.660898+0000","last_undegraded":"2024-03-11T21:26:43.660898+0000","last_fullsized":"2024-03-11T21:26:43.660898+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:25:43.559452+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:43.559452+0000","last_clean_scrub_stamp":"2024-03-11T21:25:43.559452+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:26:53.958265+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":57,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-11T21:26:43.124642+0000","last_change":"2024-03-11T21:25:42.573497+0000","last_active":"2024-03-11T21:26:43.124642+0000","last_peered":"2024-03-11T21:26:43.124642+0000","last_clean":"2024-03-11T21:26:43.124642+0000","last_became_active":"2024-03-11T21:25:42.573158+0000","last_became_peered":"2024-03-11T21:25:42.573158+0000","last_unstale":"2024-03-11T21:26:43.124642+0000","last_undegraded":"2024-03-11T21:26:43.124642+0000","last_fullsized":"2024-03-11T21:26:43.124642+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:25:41.544550+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:41.544550+0000","last_clean_scrub_stamp":"2024-03-11T21:25:41.544550+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:27:10.181203+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.15","version":"0'0","reported_seq":73,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-11T21:26:44.860570+0000","last_change":"2024-03-11T21:26:44.860603+0000","last_active":"2024-03-11T21:26:44.860570+0000","last_peered":"2024-03-11T21:26:44.860570+0000","last_clean":"2024-03-11T21:26:44.860570+0000","last_became_active":"2024-03-11T21:25:38.541705+0000","last_became_peered":"2024-03-11T21:25:38.541705+0000","last_unstale":"2024-03-11T21:26:44.860570+0000","last_undegraded":"2024-03-11T21:26:44.860570+0000","last_fullsized":"2024-03-11T21:26:44.860570+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:26:44.860499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:26:44.860499+0000","last_clean_scrub_stamp":"2024-03-11T21:26:44.860499+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:28:11.867124+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.13","version":"74'11","reported_seq":98,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-11T21:26:43.123914+0000","last_change":"2024-03-11T21:26:11.223911+0000","last_active":"2024-03-11T21:26:43.123914+0000","last_peered":"2024-03-11T21:26:43.123914+0000","last_clean":"2024-03-11T21:26:43.123914+0000","last_became_active":"2024-03-11T21:26:11.214879+0000","last_became_peered":"2024-03-11T21:26:11.214879+0000","last_unstale":"2024-03-11T21:26:43.123914+0000","last_undegraded":"2024-03-11T21:26:43.123914+0000","last_fullsized":"2024-03-11T21:26:43.123914+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:25:39.525554+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:39.525554+0000","last_clean_scrub_stamp":"2024-03-11T21:25:39.525554+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:26:50.821337+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,2],"acting":[4,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.10","version":"0'0","reported_seq":53,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-11T21:26:43.122385+0000","last_change":"2024-03-11T21:25:44.591915+0000","last_active":"2024-03-11T21:26:43.122385+0000","last_peered":"2024-03-11T21:26:43.122385+0000","last_clean":"2024-03-11T21:26:43.122385+0000","last_became_active":"2024-03-11T21:25:44.591189+0000","last_became_peered":"2024-03-11T21:25:44.591189+0000","last_unstale":"2024-03-11T21:26:43.122385+0000","last_undegraded":"2024-03-11T21:26:43.122385+0000","last_fullsized":"2024-03-11T21:26:43.122385+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:25:43.559452+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:43.559452+0000","last_clean_scrub_stamp":"2024-03-11T21:25:43.559452+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:27:08.613618+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,1],"acting":[0,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":57,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-11T21:26:43.661240+0000","last_change":"2024-03-11T21:25:42.576742+0000","last_active":"2024-03-11T21:26:43.661240+0000","last_peered":"2024-03-11T21:26:43.661240+0000","last_clean":"2024-03-11T21:26:43.661240+0000","last_became_active":"2024-03-11T21:25:42.576425+0000","last_became_peered":"2024-03-11T21:25:42.576425+0000","last_unstale":"2024-03-11T21:26:43.661240+0000","last_undegraded":"2024-03-11T21:26:43.661240+0000","last_fullsized":"2024-03-11T21:26:43.661240+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:25:41.544550+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:41.544550+0000","last_clean_scrub_stamp":"2024-03-11T21:25:41.544550+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:26:54.144762+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.14","version":"0'0","reported_seq":65,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-11T21:26:44.037749+0000","last_change":"2024-03-11T21:25:38.542192+0000","last_active":"2024-03-11T21:26:44.037749+0000","last_peered":"2024-03-11T21:26:44.037749+0000","last_clean":"2024-03-11T21:26:44.037749+0000","last_became_active":"2024-03-11T21:25:38.541852+0000","last_became_peered":"2024-03-11T21:25:38.541852+0000","last_unstale":"2024-03-11T21:26:44.037749+0000","last_undegraded":"2024-03-11T21:26:44.037749+0000","last_fullsized":"2024-03-11T21:26:44.037749+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:25:37.512018+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:37.512018+0000","last_clean_scrub_stamp":"2024-03-11T21:25:37.512018+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:26:53.950926+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,5],"acting":[6,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.12","version":"74'9","reported_seq":77,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-11T21:26:43.124565+0000","last_change":"2024-03-11T21:25:40.549164+0000","last_active":"2024-03-11T21:26:43.124565+0000","last_peered":"2024-03-11T21:26:43.124565+0000","last_clean":"2024-03-11T21:26:43.124565+0000","last_became_active":"2024-03-11T21:25:40.547082+0000","last_became_peered":"2024-03-11T21:25:40.547082+0000","last_unstale":"2024-03-11T21:26:43.124565+0000","last_undegraded":"2024-03-11T21:26:43.124565+0000","last_fullsized":"2024-03-11T21:26:43.124565+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:25:39.525554+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:39.525554+0000","last_clean_scrub_stamp":"2024-03-11T21:25:39.525554+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:27:04.348158+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":53,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-11T21:26:43.124523+0000","last_change":"2024-03-11T21:25:44.589956+0000","last_active":"2024-03-11T21:26:43.124523+0000","last_peered":"2024-03-11T21:26:43.124523+0000","last_clean":"2024-03-11T21:26:43.124523+0000","last_became_active":"2024-03-11T21:25:44.589586+0000","last_became_peered":"2024-03-11T21:25:44.589586+0000","last_unstale":"2024-03-11T21:26:43.124523+0000","last_undegraded":"2024-03-11T21:26:43.124523+0000","last_fullsized":"2024-03-11T21:26:43.124523+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:25:43.559452+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:43.559452+0000","last_clean_scrub_stamp":"2024-03-11T21:25:43.559452+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:26:51.679819+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,5],"acting":[4,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.14","version":"74'8","reported_seq":81,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-11T21:26:48.591698+0000","last_change":"2024-03-11T21:25:42.583567+0000","last_active":"2024-03-11T21:26:48.591698+0000","last_peered":"2024-03-11T21:26:48.591698+0000","last_clean":"2024-03-11T21:26:48.591698+0000","last_became_active":"2024-03-11T21:25:42.583266+0000","last_became_peered":"2024-03-11T21:25:42.583266+0000","last_unstale":"2024-03-11T21:26:48.591698+0000","last_undegraded":"2024-03-11T21:26:48.591698+0000","last_fullsized":"2024-03-11T21:26:48.591698+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:25:41.544550+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:41.544550+0000","last_clean_scrub_stamp":"2024-03-11T21:25:41.544550+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:26:57.374841+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,2],"acting":[3,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.13","version":"0'0","reported_seq":65,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-11T21:26:43.123835+0000","last_change":"2024-03-11T21:25:38.539722+0000","last_active":"2024-03-11T21:26:43.123835+0000","last_peered":"2024-03-11T21:26:43.123835+0000","last_clean":"2024-03-11T21:26:43.123835+0000","last_became_active":"2024-03-11T21:25:38.539467+0000","last_became_peered":"2024-03-11T21:25:38.539467+0000","last_unstale":"2024-03-11T21:26:43.123835+0000","last_undegraded":"2024-03-11T21:26:43.123835+0000","last_fullsized":"2024-03-11T21:26:43.123835+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:25:37.512018+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:37.512018+0000","last_clean_scrub_stamp":"2024-03-11T21:25:37.512018+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:26:56.667567+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.15","version":"74'9","reported_seq":77,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-11T21:26:44.104778+0000","last_change":"2024-03-11T21:25:40.761259+0000","last_active":"2024-03-11T21:26:44.104778+0000","last_peered":"2024-03-11T21:26:44.104778+0000","last_clean":"2024-03-11T21:26:44.104778+0000","last_became_active":"2024-03-11T21:25:40.760907+0000","last_became_peered":"2024-03-11T21:25:40.760907+0000","last_unstale":"2024-03-11T21:26:44.104778+0000","last_undegraded":"2024-03-11T21:26:44.104778+0000","last_fullsized":"2024-03-11T21:26:44.104778+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:25:39.525554+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:39.525554+0000","last_clean_scrub_stamp":"2024-03-11T21:25:39.525554+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:26:51.152546+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.16","version":"0'0","reported_seq":53,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-11T21:26:43.123132+0000","last_change":"2024-03-11T21:25:44.583741+0000","last_active":"2024-03-11T21:26:43.123132+0000","last_peered":"2024-03-11T21:26:43.123132+0000","last_clean":"2024-03-11T21:26:43.123132+0000","last_became_active":"2024-03-11T21:25:44.583393+0000","last_became_peered":"2024-03-11T21:25:44.583393+0000","last_unstale":"2024-03-11T21:26:43.123132+0000","last_undegraded":"2024-03-11T21:26:43.123132+0000","last_fullsized":"2024-03-11T21:26:43.123132+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:25:43.559452+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:43.559452+0000","last_clean_scrub_stamp":"2024-03-11T21:25:43.559452+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:26:50.059408+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.15","version":"74'8","reported_seq":81,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-11T21:26:48.591761+0000","last_change":"2024-03-11T21:25:42.572973+0000","last_active":"2024-03-11T21:26:48.591761+0000","last_peered":"2024-03-11T21:26:48.591761+0000","last_clean":"2024-03-11T21:26:48.591761+0000","last_became_active":"2024-03-11T21:25:42.572555+0000","last_became_peered":"2024-03-11T21:25:42.572555+0000","last_unstale":"2024-03-11T21:26:48.591761+0000","last_undegraded":"2024-03-11T21:26:48.591761+0000","last_fullsized":"2024-03-11T21:26:48.591761+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:25:41.544550+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:41.544550+0000","last_clean_scrub_stamp":"2024-03-11T21:25:41.544550+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:26:58.902319+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.12","version":"0'0","reported_seq":73,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-11T21:26:47.960329+0000","last_change":"2024-03-11T21:26:47.960353+0000","last_active":"2024-03-11T21:26:47.960329+0000","last_peered":"2024-03-11T21:26:47.960329+0000","last_clean":"2024-03-11T21:26:47.960329+0000","last_became_active":"2024-03-11T21:25:38.543962+0000","last_became_peered":"2024-03-11T21:25:38.543962+0000","last_unstale":"2024-03-11T21:26:47.960329+0000","last_undegraded":"2024-03-11T21:26:47.960329+0000","last_fullsized":"2024-03-11T21:26:47.960329+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:26:47.960265+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:37.512018+0000","last_clean_scrub_stamp":"2024-03-11T21:26:47.960265+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:28:15.602449+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.14","version":"74'10","reported_seq":76,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-11T21:26:43.661342+0000","last_change":"2024-03-11T21:25:40.760925+0000","last_active":"2024-03-11T21:26:43.661342+0000","last_peered":"2024-03-11T21:26:43.661342+0000","last_clean":"2024-03-11T21:26:43.661342+0000","last_became_active":"2024-03-11T21:25:40.760620+0000","last_became_peered":"2024-03-11T21:25:40.760620+0000","last_unstale":"2024-03-11T21:26:43.661342+0000","last_undegraded":"2024-03-11T21:26:43.661342+0000","last_fullsized":"2024-03-11T21:26:43.661342+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:25:39.525554+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:39.525554+0000","last_clean_scrub_stamp":"2024-03-11T21:25:39.525554+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:26:56.858432+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.15","version":"0'0","reported_seq":53,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-11T21:26:45.417736+0000","last_change":"2024-03-11T21:25:45.303093+0000","last_active":"2024-03-11T21:26:45.417736+0000","last_peered":"2024-03-11T21:26:45.417736+0000","last_clean":"2024-03-11T21:26:45.417736+0000","last_became_active":"2024-03-11T21:25:45.302501+0000","last_became_peered":"2024-03-11T21:25:45.302501+0000","last_unstale":"2024-03-11T21:26:45.417736+0000","last_undegraded":"2024-03-11T21:26:45.417736+0000","last_fullsized":"2024-03-11T21:26:45.417736+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:25:43.559452+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:43.559452+0000","last_clean_scrub_stamp":"2024-03-11T21:25:43.559452+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:26:53.821356+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":57,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-11T21:26:44.104781+0000","last_change":"2024-03-11T21:25:42.579268+0000","last_active":"2024-03-11T21:26:44.104781+0000","last_peered":"2024-03-11T21:26:44.104781+0000","last_clean":"2024-03-11T21:26:44.104781+0000","last_became_active":"2024-03-11T21:25:42.577351+0000","last_became_peered":"2024-03-11T21:25:42.577351+0000","last_unstale":"2024-03-11T21:26:44.104781+0000","last_undegraded":"2024-03-11T21:26:44.104781+0000","last_fullsized":"2024-03-11T21:26:44.104781+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:25:41.544550+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:41.544550+0000","last_clean_scrub_stamp":"2024-03-11T21:25:41.544550+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:26:56.437117+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,1],"acting":[5,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.11","version":"0'0","reported_seq":65,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-11T21:26:44.037422+0000","last_change":"2024-03-11T21:25:38.538379+0000","last_active":"2024-03-11T21:26:44.037422+0000","last_peered":"2024-03-11T21:26:44.037422+0000","last_clean":"2024-03-11T21:26:44.037422+0000","last_became_active":"2024-03-11T21:25:38.537415+0000","last_became_peered":"2024-03-11T21:25:38.537415+0000","last_unstale":"2024-03-11T21:26:44.037422+0000","last_undegraded":"2024-03-11T21:26:44.037422+0000","last_fullsized":"2024-03-11T21:26:44.037422+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:25:37.512018+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:37.512018+0000","last_clean_scrub_stamp":"2024-03-11T21:25:37.512018+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:27:06.885921+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.17","version":"74'6","reported_seq":84,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-11T21:26:43.661607+0000","last_change":"2024-03-11T21:26:11.224229+0000","last_active":"2024-03-11T21:26:43.661607+0000","last_peered":"2024-03-11T21:26:43.661607+0000","last_clean":"2024-03-11T21:26:43.661607+0000","last_became_active":"2024-03-11T21:26:11.219020+0000","last_became_peered":"2024-03-11T21:26:11.219020+0000","last_unstale":"2024-03-11T21:26:43.661607+0000","last_undegraded":"2024-03-11T21:26:43.661607+0000","last_fullsized":"2024-03-11T21:26:43.661607+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:25:39.525554+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:39.525554+0000","last_clean_scrub_stamp":"2024-03-11T21:25:39.525554+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:26:59.858426+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":9,"num_read_kb":6,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,5],"acting":[3,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.14","version":"0'0","reported_seq":53,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-11T21:26:43.291057+0000","last_change":"2024-03-11T21:25:44.587755+0000","last_active":"2024-03-11T21:26:43.291057+0000","last_peered":"2024-03-11T21:26:43.291057+0000","last_clean":"2024-03-11T21:26:43.291057+0000","last_became_active":"2024-03-11T21:25:44.587439+0000","last_became_peered":"2024-03-11T21:25:44.587439+0000","last_unstale":"2024-03-11T21:26:43.291057+0000","last_undegraded":"2024-03-11T21:26:43.291057+0000","last_fullsized":"2024-03-11T21:26:43.291057+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:25:43.559452+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:43.559452+0000","last_clean_scrub_stamp":"2024-03-11T21:25:43.559452+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:27:03.851108+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,7],"acting":[2,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":57,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-11T21:26:43.661670+0000","last_change":"2024-03-11T21:25:42.583802+0000","last_active":"2024-03-11T21:26:43.661670+0000","last_peered":"2024-03-11T21:26:43.661670+0000","last_clean":"2024-03-11T21:26:43.661670+0000","last_became_active":"2024-03-11T21:25:42.583514+0000","last_became_peered":"2024-03-11T21:25:42.583514+0000","last_unstale":"2024-03-11T21:26:43.661670+0000","last_undegraded":"2024-03-11T21:26:43.661670+0000","last_fullsized":"2024-03-11T21:26:43.661670+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:25:41.544550+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:41.544550+0000","last_clean_scrub_stamp":"2024-03-11T21:25:41.544550+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:26:54.765803+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.10","version":"0'0","reported_seq":65,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-11T21:26:43.290854+0000","last_change":"2024-03-11T21:25:38.533523+0000","last_active":"2024-03-11T21:26:43.290854+0000","last_peered":"2024-03-11T21:26:43.290854+0000","last_clean":"2024-03-11T21:26:43.290854+0000","last_became_active":"2024-03-11T21:25:38.533138+0000","last_became_peered":"2024-03-11T21:25:38.533138+0000","last_unstale":"2024-03-11T21:26:43.290854+0000","last_undegraded":"2024-03-11T21:26:43.290854+0000","last_fullsized":"2024-03-11T21:26:43.290854+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:25:37.512018+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:37.512018+0000","last_clean_scrub_stamp":"2024-03-11T21:25:37.512018+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:27:02.027595+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,0],"acting":[2,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.16","version":"74'9","reported_seq":85,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-11T21:26:53.490036+0000","last_change":"2024-03-11T21:26:53.490093+0000","last_active":"2024-03-11T21:26:53.490036+0000","last_peered":"2024-03-11T21:26:53.490036+0000","last_clean":"2024-03-11T21:26:53.490036+0000","last_became_active":"2024-03-11T21:25:40.763434+0000","last_became_peered":"2024-03-11T21:25:40.763434+0000","last_unstale":"2024-03-11T21:26:53.490036+0000","last_undegraded":"2024-03-11T21:26:53.490036+0000","last_fullsized":"2024-03-11T21:26:53.490036+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'9","last_scrub_stamp":"2024-03-11T21:26:53.489953+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:39.525554+0000","last_clean_scrub_stamp":"2024-03-11T21:26:53.489953+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:27:55.665353+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.b","version":"0'0","reported_seq":53,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-11T21:26:43.661344+0000","last_change":"2024-03-11T21:25:44.590403+0000","last_active":"2024-03-11T21:26:43.661344+0000","last_peered":"2024-03-11T21:26:43.661344+0000","last_clean":"2024-03-11T21:26:43.661344+0000","last_became_active":"2024-03-11T21:25:44.590071+0000","last_became_peered":"2024-03-11T21:25:44.590071+0000","last_unstale":"2024-03-11T21:26:43.661344+0000","last_undegraded":"2024-03-11T21:26:43.661344+0000","last_fullsized":"2024-03-11T21:26:43.661344+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:25:43.559452+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:43.559452+0000","last_clean_scrub_stamp":"2024-03-11T21:25:43.559452+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:27:05.309857+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.f","version":"67'2","reported_seq":77,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-11T21:26:43.124372+0000","last_change":"2024-03-11T21:25:38.537988+0000","last_active":"2024-03-11T21:26:43.124372+0000","last_peered":"2024-03-11T21:26:43.124372+0000","last_clean":"2024-03-11T21:26:43.124372+0000","last_became_active":"2024-03-11T21:25:38.537711+0000","last_became_peered":"2024-03-11T21:25:38.537711+0000","last_unstale":"2024-03-11T21:26:43.124372+0000","last_undegraded":"2024-03-11T21:26:43.124372+0000","last_fullsized":"2024-03-11T21:26:43.124372+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:25:37.512018+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:37.512018+0000","last_clean_scrub_stamp":"2024-03-11T21:25:37.512018+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:26:57.338014+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":92,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":10,"num_read_kb":10,"num_write":4,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,7],"acting":[4,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.9","version":"74'12","reported_seq":84,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-11T21:26:43.123650+0000","last_change":"2024-03-11T21:25:40.552685+0000","last_active":"2024-03-11T21:26:43.123650+0000","last_peered":"2024-03-11T21:26:43.123650+0000","last_clean":"2024-03-11T21:26:43.123650+0000","last_became_active":"2024-03-11T21:25:40.552267+0000","last_became_peered":"2024-03-11T21:25:40.552267+0000","last_unstale":"2024-03-11T21:26:43.123650+0000","last_undegraded":"2024-03-11T21:26:43.123650+0000","last_fullsized":"2024-03-11T21:26:43.123650+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:25:39.525554+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:39.525554+0000","last_clean_scrub_stamp":"2024-03-11T21:25:39.525554+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:27:00.249786+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":25,"num_read_kb":16,"num_write":14,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,3],"acting":[4,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":57,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-11T21:26:43.290838+0000","last_change":"2024-03-11T21:25:42.574312+0000","last_active":"2024-03-11T21:26:43.290838+0000","last_peered":"2024-03-11T21:26:43.290838+0000","last_clean":"2024-03-11T21:26:43.290838+0000","last_became_active":"2024-03-11T21:25:42.573987+0000","last_became_peered":"2024-03-11T21:25:42.573987+0000","last_unstale":"2024-03-11T21:26:43.290838+0000","last_undegraded":"2024-03-11T21:26:43.290838+0000","last_fullsized":"2024-03-11T21:26:43.290838+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:25:41.544550+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:41.544550+0000","last_clean_scrub_stamp":"2024-03-11T21:25:41.544550+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:27:01.129396+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.0","version":"0'0","reported_seq":53,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-11T21:26:43.122501+0000","last_change":"2024-03-11T21:25:44.585477+0000","last_active":"2024-03-11T21:26:43.122501+0000","last_peered":"2024-03-11T21:26:43.122501+0000","last_clean":"2024-03-11T21:26:43.122501+0000","last_became_active":"2024-03-11T21:25:44.584565+0000","last_became_peered":"2024-03-11T21:25:44.584565+0000","last_unstale":"2024-03-11T21:26:43.122501+0000","last_undegraded":"2024-03-11T21:26:43.122501+0000","last_fullsized":"2024-03-11T21:26:43.122501+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:25:43.559452+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:43.559452+0000","last_clean_scrub_stamp":"2024-03-11T21:25:43.559452+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:27:04.318172+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":65,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-11T21:26:43.123446+0000","last_change":"2024-03-11T21:25:38.542056+0000","last_active":"2024-03-11T21:26:43.123446+0000","last_peered":"2024-03-11T21:26:43.123446+0000","last_clean":"2024-03-11T21:26:43.123446+0000","last_became_active":"2024-03-11T21:25:38.541760+0000","last_became_peered":"2024-03-11T21:25:38.541760+0000","last_unstale":"2024-03-11T21:26:43.123446+0000","last_undegraded":"2024-03-11T21:26:43.123446+0000","last_fullsized":"2024-03-11T21:26:43.123446+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:25:37.512018+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:37.512018+0000","last_clean_scrub_stamp":"2024-03-11T21:25:37.512018+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:26:51.218550+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,7],"acting":[1,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.2","version":"94'15","reported_seq":89,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-11T21:26:49.993148+0000","last_change":"2024-03-11T21:26:49.993206+0000","last_active":"2024-03-11T21:26:49.993148+0000","last_peered":"2024-03-11T21:26:49.993148+0000","last_clean":"2024-03-11T21:26:49.993148+0000","last_became_active":"2024-03-11T21:25:40.548833+0000","last_became_peered":"2024-03-11T21:25:40.548833+0000","last_unstale":"2024-03-11T21:26:49.993148+0000","last_undegraded":"2024-03-11T21:26:49.993148+0000","last_fullsized":"2024-03-11T21:26:49.993148+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"94'15","last_scrub_stamp":"2024-03-11T21:26:49.993060+0000","last_deep_scrub":"94'15","last_deep_scrub_stamp":"2024-03-11T21:26:49.993060+0000","last_clean_scrub_stamp":"2024-03-11T21:26:49.993060+0000","objects_scrubbed":5,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:27:59.041594+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.3","version":"74'8","reported_seq":81,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-11T21:26:48.591523+0000","last_change":"2024-03-11T21:25:42.578555+0000","last_active":"2024-03-11T21:26:48.591523+0000","last_peered":"2024-03-11T21:26:48.591523+0000","last_clean":"2024-03-11T21:26:48.591523+0000","last_became_active":"2024-03-11T21:25:42.578161+0000","last_became_peered":"2024-03-11T21:25:42.578161+0000","last_unstale":"2024-03-11T21:26:48.591523+0000","last_undegraded":"2024-03-11T21:26:48.591523+0000","last_fullsized":"2024-03-11T21:26:48.591523+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:25:41.544550+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:41.544550+0000","last_clean_scrub_stamp":"2024-03-11T21:25:41.544550+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:26:55.305524+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,5],"acting":[0,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":53,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-11T21:26:43.661632+0000","last_change":"2024-03-11T21:25:44.585641+0000","last_active":"2024-03-11T21:26:43.661632+0000","last_peered":"2024-03-11T21:26:43.661632+0000","last_clean":"2024-03-11T21:26:43.661632+0000","last_became_active":"2024-03-11T21:25:44.585223+0000","last_became_peered":"2024-03-11T21:25:44.585223+0000","last_unstale":"2024-03-11T21:26:43.661632+0000","last_undegraded":"2024-03-11T21:26:43.661632+0000","last_fullsized":"2024-03-11T21:26:43.661632+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:25:43.559452+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:43.559452+0000","last_clean_scrub_stamp":"2024-03-11T21:25:43.559452+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:26:54.535741+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,7],"acting":[3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":73,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-11T21:26:45.072135+0000","last_change":"2024-03-11T21:26:45.072161+0000","last_active":"2024-03-11T21:26:45.072135+0000","last_peered":"2024-03-11T21:26:45.072135+0000","last_clean":"2024-03-11T21:26:45.072135+0000","last_became_active":"2024-03-11T21:25:38.538540+0000","last_became_peered":"2024-03-11T21:25:38.538540+0000","last_unstale":"2024-03-11T21:26:45.072135+0000","last_undegraded":"2024-03-11T21:26:45.072135+0000","last_fullsized":"2024-03-11T21:26:45.072135+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:26:45.072058+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:37.512018+0000","last_clean_scrub_stamp":"2024-03-11T21:26:45.072058+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:27:49.711890+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.4","version":"74'28","reported_seq":125,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-11T21:26:48.591583+0000","last_change":"2024-03-11T21:25:40.553218+0000","last_active":"2024-03-11T21:26:48.591583+0000","last_peered":"2024-03-11T21:26:48.591583+0000","last_clean":"2024-03-11T21:26:48.591583+0000","last_became_active":"2024-03-11T21:25:40.552902+0000","last_became_peered":"2024-03-11T21:25:40.552902+0000","last_unstale":"2024-03-11T21:26:48.591583+0000","last_undegraded":"2024-03-11T21:26:48.591583+0000","last_fullsized":"2024-03-11T21:26:48.591583+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:25:39.525554+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:39.525554+0000","last_clean_scrub_stamp":"2024-03-11T21:25:39.525554+0000","objects_scrubbed":0,"log_size":28,"log_dups_size":0,"ondisk_log_size":28,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:26:58.308755+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":358,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":48,"num_read_kb":33,"num_write":26,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3],"acting":[1,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":57,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-11T21:26:43.122939+0000","last_change":"2024-03-11T21:25:42.573353+0000","last_active":"2024-03-11T21:26:43.122939+0000","last_peered":"2024-03-11T21:26:43.122939+0000","last_clean":"2024-03-11T21:26:43.122939+0000","last_became_active":"2024-03-11T21:25:42.572673+0000","last_became_peered":"2024-03-11T21:25:42.572673+0000","last_unstale":"2024-03-11T21:26:43.122939+0000","last_undegraded":"2024-03-11T21:26:43.122939+0000","last_fullsized":"2024-03-11T21:26:43.122939+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:25:41.544550+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:41.544550+0000","last_clean_scrub_stamp":"2024-03-11T21:25:41.544550+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:27:02.832892+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.5","version":"0'0","reported_seq":53,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-11T21:26:45.417895+0000","last_change":"2024-03-11T21:25:45.302725+0000","last_active":"2024-03-11T21:26:45.417895+0000","last_peered":"2024-03-11T21:26:45.417895+0000","last_clean":"2024-03-11T21:26:45.417895+0000","last_became_active":"2024-03-11T21:25:45.302120+0000","last_became_peered":"2024-03-11T21:25:45.302120+0000","last_unstale":"2024-03-11T21:26:45.417895+0000","last_undegraded":"2024-03-11T21:26:45.417895+0000","last_fullsized":"2024-03-11T21:26:45.417895+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:25:43.559452+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:43.559452+0000","last_clean_scrub_stamp":"2024-03-11T21:25:43.559452+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:27:03.884398+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,3],"acting":[7,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.0","version":"76'5","reported_seq":156,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-11T21:26:55.585521+0000","last_change":"2024-03-11T21:26:54.088445+0000","last_active":"2024-03-11T21:26:55.585521+0000","last_peered":"2024-03-11T21:26:55.585521+0000","last_clean":"2024-03-11T21:26:55.585521+0000","last_became_active":"2024-03-11T21:25:38.536890+0000","last_became_peered":"2024-03-11T21:25:38.536890+0000","last_unstale":"2024-03-11T21:26:55.585521+0000","last_undegraded":"2024-03-11T21:26:55.585521+0000","last_fullsized":"2024-03-11T21:26:55.585521+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"76'5","last_scrub_stamp":"2024-03-11T21:26:54.088356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:37.512018+0000","last_clean_scrub_stamp":"2024-03-11T21:26:54.088356+0000","objects_scrubbed":1,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:28:21.305217+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.000753559,"stat_sum":{"num_bytes":389,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":73,"num_read_kb":68,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,6],"acting":[1,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":73,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-11T21:26:48.997525+0000","last_change":"2024-03-11T21:26:48.997561+0000","last_active":"2024-03-11T21:26:48.997525+0000","last_peered":"2024-03-11T21:26:48.997525+0000","last_clean":"2024-03-11T21:26:48.997525+0000","last_became_active":"2024-03-11T21:25:38.532984+0000","last_became_peered":"2024-03-11T21:25:38.532984+0000","last_unstale":"2024-03-11T21:26:48.997525+0000","last_undegraded":"2024-03-11T21:26:48.997525+0000","last_fullsized":"2024-03-11T21:26:48.997525+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:26:48.997454+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:26:48.997454+0000","last_clean_scrub_stamp":"2024-03-11T21:26:48.997454+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:28:14.373168+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.7","version":"74'13","reported_seq":88,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-11T21:26:43.123321+0000","last_change":"2024-03-11T21:25:40.542117+0000","last_active":"2024-03-11T21:26:43.123321+0000","last_peered":"2024-03-11T21:26:43.123321+0000","last_clean":"2024-03-11T21:26:43.123321+0000","last_became_active":"2024-03-11T21:25:40.541769+0000","last_became_peered":"2024-03-11T21:25:40.541769+0000","last_unstale":"2024-03-11T21:26:43.123321+0000","last_undegraded":"2024-03-11T21:26:43.123321+0000","last_fullsized":"2024-03-11T21:26:43.123321+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:25:39.525554+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:39.525554+0000","last_clean_scrub_stamp":"2024-03-11T21:25:39.525554+0000","objects_scrubbed":0,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:27:07.035253+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":19,"num_write":16,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":57,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-11T21:26:43.291237+0000","last_change":"2024-03-11T21:25:42.582034+0000","last_active":"2024-03-11T21:26:43.291237+0000","last_peered":"2024-03-11T21:26:43.291237+0000","last_clean":"2024-03-11T21:26:43.291237+0000","last_became_active":"2024-03-11T21:25:42.581710+0000","last_became_peered":"2024-03-11T21:25:42.581710+0000","last_unstale":"2024-03-11T21:26:43.291237+0000","last_undegraded":"2024-03-11T21:26:43.291237+0000","last_fullsized":"2024-03-11T21:26:43.291237+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:25:41.544550+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:41.544550+0000","last_clean_scrub_stamp":"2024-03-11T21:25:41.544550+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:26:46.362146+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7],"acting":[2,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.7","version":"74'1","reported_seq":54,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-11T21:26:44.105021+0000","last_change":"2024-03-11T21:25:44.591645+0000","last_active":"2024-03-11T21:26:44.105021+0000","last_peered":"2024-03-11T21:26:44.105021+0000","last_clean":"2024-03-11T21:26:44.105021+0000","last_became_active":"2024-03-11T21:25:44.591376+0000","last_became_peered":"2024-03-11T21:25:44.591376+0000","last_unstale":"2024-03-11T21:26:44.105021+0000","last_undegraded":"2024-03-11T21:26:44.105021+0000","last_fullsized":"2024-03-11T21:26:44.105021+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:25:43.559452+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:43.559452+0000","last_clean_scrub_stamp":"2024-03-11T21:25:43.559452+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:27:09.252622+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,4],"acting":[5,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2","version":"71'2","reported_seq":82,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-11T21:26:43.660984+0000","last_change":"2024-03-11T21:26:42.663119+0000","last_active":"2024-03-11T21:26:43.660984+0000","last_peered":"2024-03-11T21:26:43.660984+0000","last_clean":"2024-03-11T21:26:43.660984+0000","last_became_active":"2024-03-11T21:25:38.539413+0000","last_became_peered":"2024-03-11T21:25:38.539413+0000","last_unstale":"2024-03-11T21:26:43.660984+0000","last_undegraded":"2024-03-11T21:26:43.660984+0000","last_fullsized":"2024-03-11T21:26:43.660984+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"71'2","last_scrub_stamp":"2024-03-11T21:26:42.662979+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:37.512018+0000","last_clean_scrub_stamp":"2024-03-11T21:26:42.662979+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:27:54.085660+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.0014206449999999999,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":79,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-11T21:26:44.103841+0000","last_change":"2024-03-11T21:26:42.171361+0000","last_active":"2024-03-11T21:26:44.103841+0000","last_peered":"2024-03-11T21:26:44.103841+0000","last_clean":"2024-03-11T21:26:44.103841+0000","last_became_active":"2024-03-11T21:26:11.216188+0000","last_became_peered":"2024-03-11T21:26:11.216188+0000","last_unstale":"2024-03-11T21:26:44.103841+0000","last_undegraded":"2024-03-11T21:26:44.103841+0000","last_fullsized":"2024-03-11T21:26:44.103841+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:26:42.171256+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:37.512018+0000","last_clean_scrub_stamp":"2024-03-11T21:26:42.171256+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:27:58.632937+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,4],"acting":[5,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"87'128","reported_seq":318,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-11T21:26:45.417265+0000","last_change":"2024-03-11T21:26:11.340374+0000","last_active":"2024-03-11T21:26:45.417265+0000","last_peered":"2024-03-11T21:26:45.417265+0000","last_clean":"2024-03-11T21:26:45.417265+0000","last_became_active":"2024-03-11T21:25:30.502597+0000","last_became_peered":"2024-03-11T21:25:30.502597+0000","last_unstale":"2024-03-11T21:26:45.417265+0000","last_undegraded":"2024-03-11T21:26:45.417265+0000","last_fullsized":"2024-03-11T21:26:45.417265+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'117","last_scrub_stamp":"2024-03-11T21:26:11.340228+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:24.261974+0000","last_clean_scrub_stamp":"2024-03-11T21:26:11.340228+0000","objects_scrubbed":2,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:27:32.864192+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":212,"num_read_kb":306,"num_write":179,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1771104,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.5","version":"74'16","reported_seq":99,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-11T21:26:48.591834+0000","last_change":"2024-03-11T21:25:40.761955+0000","last_active":"2024-03-11T21:26:48.591834+0000","last_peered":"2024-03-11T21:26:48.591834+0000","last_clean":"2024-03-11T21:26:48.591834+0000","last_became_active":"2024-03-11T21:25:40.761457+0000","last_became_peered":"2024-03-11T21:25:40.761457+0000","last_unstale":"2024-03-11T21:26:48.591834+0000","last_undegraded":"2024-03-11T21:26:48.591834+0000","last_fullsized":"2024-03-11T21:26:48.591834+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:25:39.525554+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:39.525554+0000","last_clean_scrub_stamp":"2024-03-11T21:25:39.525554+0000","objects_scrubbed":0,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:27:06.180463+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":154,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":25,"num_read_kb":15,"num_write":13,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":57,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-11T21:26:45.417785+0000","last_change":"2024-03-11T21:25:42.586604+0000","last_active":"2024-03-11T21:26:45.417785+0000","last_peered":"2024-03-11T21:26:45.417785+0000","last_clean":"2024-03-11T21:26:45.417785+0000","last_became_active":"2024-03-11T21:25:42.586226+0000","last_became_peered":"2024-03-11T21:25:42.586226+0000","last_unstale":"2024-03-11T21:26:45.417785+0000","last_undegraded":"2024-03-11T21:26:45.417785+0000","last_fullsized":"2024-03-11T21:26:45.417785+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:25:41.544550+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:41.544550+0000","last_clean_scrub_stamp":"2024-03-11T21:25:41.544550+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:26:53.601105+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,5],"acting":[7,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.4","version":"0'0","reported_seq":53,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-11T21:26:43.123201+0000","last_change":"2024-03-11T21:25:44.594544+0000","last_active":"2024-03-11T21:26:43.123201+0000","last_peered":"2024-03-11T21:26:43.123201+0000","last_clean":"2024-03-11T21:26:43.123201+0000","last_became_active":"2024-03-11T21:25:44.594188+0000","last_became_peered":"2024-03-11T21:25:44.594188+0000","last_unstale":"2024-03-11T21:26:43.123201+0000","last_undegraded":"2024-03-11T21:26:43.123201+0000","last_fullsized":"2024-03-11T21:26:43.123201+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:25:43.559452+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:43.559452+0000","last_clean_scrub_stamp":"2024-03-11T21:25:43.559452+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:26:50.855383+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":80,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-11T21:26:43.122732+0000","last_change":"2024-03-11T21:26:38.589294+0000","last_active":"2024-03-11T21:26:43.122732+0000","last_peered":"2024-03-11T21:26:43.122732+0000","last_clean":"2024-03-11T21:26:43.122732+0000","last_became_active":"2024-03-11T21:25:38.543834+0000","last_became_peered":"2024-03-11T21:25:38.543834+0000","last_unstale":"2024-03-11T21:26:43.122732+0000","last_undegraded":"2024-03-11T21:26:43.122732+0000","last_fullsized":"2024-03-11T21:26:43.122732+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:26:38.589187+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:37.512018+0000","last_clean_scrub_stamp":"2024-03-11T21:26:38.589187+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:27:54.798633+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0.00089342499999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":73,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-11T21:26:51.473023+0000","last_change":"2024-03-11T21:26:51.473044+0000","last_active":"2024-03-11T21:26:51.473023+0000","last_peered":"2024-03-11T21:26:51.473023+0000","last_clean":"2024-03-11T21:26:51.473023+0000","last_became_active":"2024-03-11T21:25:38.541617+0000","last_became_peered":"2024-03-11T21:25:38.541617+0000","last_unstale":"2024-03-11T21:26:51.473023+0000","last_undegraded":"2024-03-11T21:26:51.473023+0000","last_fullsized":"2024-03-11T21:26:51.473023+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:26:51.472965+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:37.512018+0000","last_clean_scrub_stamp":"2024-03-11T21:26:51.472965+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:28:11.852536+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,0],"acting":[7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.6","version":"74'12","reported_seq":79,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-11T21:26:43.123194+0000","last_change":"2024-03-11T21:25:40.550728+0000","last_active":"2024-03-11T21:26:43.123194+0000","last_peered":"2024-03-11T21:26:43.123194+0000","last_clean":"2024-03-11T21:26:43.123194+0000","last_became_active":"2024-03-11T21:25:40.550388+0000","last_became_peered":"2024-03-11T21:25:40.550388+0000","last_unstale":"2024-03-11T21:26:43.123194+0000","last_undegraded":"2024-03-11T21:26:43.123194+0000","last_fullsized":"2024-03-11T21:26:43.123194+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:25:39.525554+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:39.525554+0000","last_clean_scrub_stamp":"2024-03-11T21:25:39.525554+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:26:51.725602+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":18,"num_read_kb":12,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":57,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-11T21:26:44.104985+0000","last_change":"2024-03-11T21:25:42.572970+0000","last_active":"2024-03-11T21:26:44.104985+0000","last_peered":"2024-03-11T21:26:44.104985+0000","last_clean":"2024-03-11T21:26:44.104985+0000","last_became_active":"2024-03-11T21:25:42.572324+0000","last_became_peered":"2024-03-11T21:25:42.572324+0000","last_unstale":"2024-03-11T21:26:44.104985+0000","last_undegraded":"2024-03-11T21:26:44.104985+0000","last_fullsized":"2024-03-11T21:26:44.104985+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:25:41.544550+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:41.544550+0000","last_clean_scrub_stamp":"2024-03-11T21:25:41.544550+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:26:54.642546+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":53,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-11T21:26:43.122936+0000","last_change":"2024-03-11T21:25:45.301814+0000","last_active":"2024-03-11T21:26:43.122936+0000","last_peered":"2024-03-11T21:26:43.122936+0000","last_clean":"2024-03-11T21:26:43.122936+0000","last_became_active":"2024-03-11T21:25:45.301416+0000","last_became_peered":"2024-03-11T21:25:45.301416+0000","last_unstale":"2024-03-11T21:26:43.122936+0000","last_undegraded":"2024-03-11T21:26:43.122936+0000","last_fullsized":"2024-03-11T21:26:43.122936+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:25:43.559452+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:43.559452+0000","last_clean_scrub_stamp":"2024-03-11T21:25:43.559452+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:26:49.637431+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":65,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-11T21:26:45.417982+0000","last_change":"2024-03-11T21:25:38.545457+0000","last_active":"2024-03-11T21:26:45.417982+0000","last_peered":"2024-03-11T21:26:45.417982+0000","last_clean":"2024-03-11T21:26:45.417982+0000","last_became_active":"2024-03-11T21:25:38.545148+0000","last_became_peered":"2024-03-11T21:25:38.545148+0000","last_unstale":"2024-03-11T21:26:45.417982+0000","last_undegraded":"2024-03-11T21:26:45.417982+0000","last_fullsized":"2024-03-11T21:26:45.417982+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:25:37.512018+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:37.512018+0000","last_clean_scrub_stamp":"2024-03-11T21:25:37.512018+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:26:53.496284+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.3","version":"74'19","reported_seq":97,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-11T21:26:43.123750+0000","last_change":"2024-03-11T21:25:40.760898+0000","last_active":"2024-03-11T21:26:43.123750+0000","last_peered":"2024-03-11T21:26:43.123750+0000","last_clean":"2024-03-11T21:26:43.123750+0000","last_became_active":"2024-03-11T21:25:40.760594+0000","last_became_peered":"2024-03-11T21:25:40.760594+0000","last_unstale":"2024-03-11T21:26:43.123750+0000","last_undegraded":"2024-03-11T21:26:43.123750+0000","last_fullsized":"2024-03-11T21:26:43.123750+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:25:39.525554+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:39.525554+0000","last_clean_scrub_stamp":"2024-03-11T21:25:39.525554+0000","objects_scrubbed":0,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:27:01.689572+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":39,"num_read_kb":25,"num_write":22,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":57,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-11T21:26:44.037634+0000","last_change":"2024-03-11T21:25:42.579400+0000","last_active":"2024-03-11T21:26:44.037634+0000","last_peered":"2024-03-11T21:26:44.037634+0000","last_clean":"2024-03-11T21:26:44.037634+0000","last_became_active":"2024-03-11T21:25:42.579045+0000","last_became_peered":"2024-03-11T21:25:42.579045+0000","last_unstale":"2024-03-11T21:26:44.037634+0000","last_undegraded":"2024-03-11T21:26:44.037634+0000","last_fullsized":"2024-03-11T21:26:44.037634+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:25:41.544550+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:41.544550+0000","last_clean_scrub_stamp":"2024-03-11T21:25:41.544550+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:26:57.514032+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":61,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-11T21:26:50.836544+0000","last_change":"2024-03-11T21:26:50.836566+0000","last_active":"2024-03-11T21:26:50.836544+0000","last_peered":"2024-03-11T21:26:50.836544+0000","last_clean":"2024-03-11T21:26:50.836544+0000","last_became_active":"2024-03-11T21:25:44.583638+0000","last_became_peered":"2024-03-11T21:25:44.583638+0000","last_unstale":"2024-03-11T21:26:50.836544+0000","last_undegraded":"2024-03-11T21:26:50.836544+0000","last_fullsized":"2024-03-11T21:26:50.836544+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:26:50.836479+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:43.559452+0000","last_clean_scrub_stamp":"2024-03-11T21:26:50.836479+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:28:03.817969+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.6","version":"67'1","reported_seq":66,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-11T21:26:43.123657+0000","last_change":"2024-03-11T21:25:38.536530+0000","last_active":"2024-03-11T21:26:43.123657+0000","last_peered":"2024-03-11T21:26:43.123657+0000","last_clean":"2024-03-11T21:26:43.123657+0000","last_became_active":"2024-03-11T21:25:38.536142+0000","last_became_peered":"2024-03-11T21:25:38.536142+0000","last_unstale":"2024-03-11T21:26:43.123657+0000","last_undegraded":"2024-03-11T21:26:43.123657+0000","last_fullsized":"2024-03-11T21:26:43.123657+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:25:37.512018+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:37.512018+0000","last_clean_scrub_stamp":"2024-03-11T21:25:37.512018+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:26:57.512473+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,4],"acting":[1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.0","version":"93'26","reported_seq":109,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-11T21:26:43.660907+0000","last_change":"2024-03-11T21:26:40.680446+0000","last_active":"2024-03-11T21:26:43.660907+0000","last_peered":"2024-03-11T21:26:43.660907+0000","last_clean":"2024-03-11T21:26:43.660907+0000","last_became_active":"2024-03-11T21:25:40.761716+0000","last_became_peered":"2024-03-11T21:25:40.761716+0000","last_unstale":"2024-03-11T21:26:43.660907+0000","last_undegraded":"2024-03-11T21:26:43.660907+0000","last_fullsized":"2024-03-11T21:26:43.660907+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"93'26","last_scrub_stamp":"2024-03-11T21:26:40.680324+0000","last_deep_scrub":"93'26","last_deep_scrub_stamp":"2024-03-11T21:26:40.680324+0000","last_clean_scrub_stamp":"2024-03-11T21:26:40.680324+0000","objects_scrubbed":8,"log_size":26,"log_dups_size":0,"ondisk_log_size":26,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:28:00.090900+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":34,"num_read_kb":22,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":65,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-11T21:26:51.863681+0000","last_change":"2024-03-11T21:26:51.863702+0000","last_active":"2024-03-11T21:26:51.863681+0000","last_peered":"2024-03-11T21:26:51.863681+0000","last_clean":"2024-03-11T21:26:51.863681+0000","last_became_active":"2024-03-11T21:25:42.583647+0000","last_became_peered":"2024-03-11T21:25:42.583647+0000","last_unstale":"2024-03-11T21:26:51.863681+0000","last_undegraded":"2024-03-11T21:26:51.863681+0000","last_fullsized":"2024-03-11T21:26:51.863681+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:26:51.863621+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:26:51.863621+0000","last_clean_scrub_stamp":"2024-03-11T21:26:51.863621+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:27:53.832869+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,7],"acting":[4,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.3","version":"0'0","reported_seq":61,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-11T21:26:54.421255+0000","last_change":"2024-03-11T21:26:54.421277+0000","last_active":"2024-03-11T21:26:54.421255+0000","last_peered":"2024-03-11T21:26:54.421255+0000","last_clean":"2024-03-11T21:26:54.421255+0000","last_became_active":"2024-03-11T21:25:45.302719+0000","last_became_peered":"2024-03-11T21:25:45.302719+0000","last_unstale":"2024-03-11T21:26:54.421255+0000","last_undegraded":"2024-03-11T21:26:54.421255+0000","last_fullsized":"2024-03-11T21:26:54.421255+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:26:54.421196+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:43.559452+0000","last_clean_scrub_stamp":"2024-03-11T21:26:54.421196+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:28:15.162446+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,2],"acting":[7,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":73,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-11T21:26:47.927473+0000","last_change":"2024-03-11T21:26:47.927498+0000","last_active":"2024-03-11T21:26:47.927473+0000","last_peered":"2024-03-11T21:26:47.927473+0000","last_clean":"2024-03-11T21:26:47.927473+0000","last_became_active":"2024-03-11T21:25:38.540013+0000","last_became_peered":"2024-03-11T21:25:38.540013+0000","last_unstale":"2024-03-11T21:26:47.927473+0000","last_undegraded":"2024-03-11T21:26:47.927473+0000","last_fullsized":"2024-03-11T21:26:47.927473+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:26:47.927394+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:37.512018+0000","last_clean_scrub_stamp":"2024-03-11T21:26:47.927394+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:28:02.205612+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,2],"acting":[6,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.1","version":"74'14","reported_seq":82,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-11T21:26:43.123471+0000","last_change":"2024-03-11T21:25:40.546609+0000","last_active":"2024-03-11T21:26:43.123471+0000","last_peered":"2024-03-11T21:26:43.123471+0000","last_clean":"2024-03-11T21:26:43.123471+0000","last_became_active":"2024-03-11T21:25:40.546219+0000","last_became_peered":"2024-03-11T21:25:40.546219+0000","last_unstale":"2024-03-11T21:26:43.123471+0000","last_undegraded":"2024-03-11T21:26:43.123471+0000","last_fullsized":"2024-03-11T21:26:43.123471+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:25:39.525554+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:39.525554+0000","last_clean_scrub_stamp":"2024-03-11T21:25:39.525554+0000","objects_scrubbed":0,"log_size":14,"log_dups_size":0,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:26:59.941670+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":21,"num_read_kb":14,"num_write":14,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6],"acting":[4,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.0","version":"74'8","reported_seq":81,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-11T21:26:48.591692+0000","last_change":"2024-03-11T21:25:42.577103+0000","last_active":"2024-03-11T21:26:48.591692+0000","last_peered":"2024-03-11T21:26:48.591692+0000","last_clean":"2024-03-11T21:26:48.591692+0000","last_became_active":"2024-03-11T21:25:42.576126+0000","last_became_peered":"2024-03-11T21:25:42.576126+0000","last_unstale":"2024-03-11T21:26:48.591692+0000","last_undegraded":"2024-03-11T21:26:48.591692+0000","last_fullsized":"2024-03-11T21:26:48.591692+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:25:41.544550+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:41.544550+0000","last_clean_scrub_stamp":"2024-03-11T21:25:41.544550+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:27:01.064639+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.c","version":"0'0","reported_seq":53,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-11T21:26:43.661225+0000","last_change":"2024-03-11T21:25:45.303235+0000","last_active":"2024-03-11T21:26:43.661225+0000","last_peered":"2024-03-11T21:26:43.661225+0000","last_clean":"2024-03-11T21:26:43.661225+0000","last_became_active":"2024-03-11T21:25:45.302207+0000","last_became_peered":"2024-03-11T21:25:45.302207+0000","last_unstale":"2024-03-11T21:26:43.661225+0000","last_undegraded":"2024-03-11T21:26:43.661225+0000","last_fullsized":"2024-03-11T21:26:43.661225+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:25:43.559452+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:43.559452+0000","last_clean_scrub_stamp":"2024-03-11T21:25:43.559452+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:27:05.058318+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.f","version":"0'0","reported_seq":65,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-11T21:26:51.925445+0000","last_change":"2024-03-11T21:26:51.925489+0000","last_active":"2024-03-11T21:26:51.925445+0000","last_peered":"2024-03-11T21:26:51.925445+0000","last_clean":"2024-03-11T21:26:51.925445+0000","last_became_active":"2024-03-11T21:25:42.575935+0000","last_became_peered":"2024-03-11T21:25:42.575935+0000","last_unstale":"2024-03-11T21:26:51.925445+0000","last_undegraded":"2024-03-11T21:26:51.925445+0000","last_fullsized":"2024-03-11T21:26:51.925445+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:26:51.925329+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:41.544550+0000","last_clean_scrub_stamp":"2024-03-11T21:26:51.925329+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:28:02.478228+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.8","version":"0'0","reported_seq":73,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-11T21:26:48.423986+0000","last_change":"2024-03-11T21:26:48.424013+0000","last_active":"2024-03-11T21:26:48.423986+0000","last_peered":"2024-03-11T21:26:48.423986+0000","last_clean":"2024-03-11T21:26:48.423986+0000","last_became_active":"2024-03-11T21:25:38.540910+0000","last_became_peered":"2024-03-11T21:25:38.540910+0000","last_unstale":"2024-03-11T21:26:48.423986+0000","last_undegraded":"2024-03-11T21:26:48.423986+0000","last_fullsized":"2024-03-11T21:26:48.423986+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:26:48.423910+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:37.512018+0000","last_clean_scrub_stamp":"2024-03-11T21:26:48.423910+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:28:17.378160+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,1],"acting":[7,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.e","version":"74'11","reported_seq":80,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-11T21:26:43.124311+0000","last_change":"2024-03-11T21:25:40.552530+0000","last_active":"2024-03-11T21:26:43.124311+0000","last_peered":"2024-03-11T21:26:43.124311+0000","last_clean":"2024-03-11T21:26:43.124311+0000","last_became_active":"2024-03-11T21:25:40.552178+0000","last_became_peered":"2024-03-11T21:25:40.552178+0000","last_unstale":"2024-03-11T21:26:43.124311+0000","last_undegraded":"2024-03-11T21:26:43.124311+0000","last_fullsized":"2024-03-11T21:26:43.124311+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:25:39.525554+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:39.525554+0000","last_clean_scrub_stamp":"2024-03-11T21:25:39.525554+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:26:47.457663+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.d","version":"0'0","reported_seq":61,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-11T21:26:54.957646+0000","last_change":"2024-03-11T21:26:54.957670+0000","last_active":"2024-03-11T21:26:54.957646+0000","last_peered":"2024-03-11T21:26:54.957646+0000","last_clean":"2024-03-11T21:26:54.957646+0000","last_became_active":"2024-03-11T21:25:44.590333+0000","last_became_peered":"2024-03-11T21:25:44.590333+0000","last_unstale":"2024-03-11T21:26:54.957646+0000","last_undegraded":"2024-03-11T21:26:54.957646+0000","last_fullsized":"2024-03-11T21:26:54.957646+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:26:54.957577+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:43.559452+0000","last_clean_scrub_stamp":"2024-03-11T21:26:54.957577+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:28:05.595894+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.e","version":"74'8","reported_seq":81,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-11T21:26:48.592206+0000","last_change":"2024-03-11T21:25:42.577257+0000","last_active":"2024-03-11T21:26:48.592206+0000","last_peered":"2024-03-11T21:26:48.592206+0000","last_clean":"2024-03-11T21:26:48.592206+0000","last_became_active":"2024-03-11T21:25:42.576742+0000","last_became_peered":"2024-03-11T21:25:42.576742+0000","last_unstale":"2024-03-11T21:26:48.592206+0000","last_undegraded":"2024-03-11T21:26:48.592206+0000","last_fullsized":"2024-03-11T21:26:48.592206+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:25:41.544550+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:41.544550+0000","last_clean_scrub_stamp":"2024-03-11T21:25:41.544550+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:26:54.995521+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,0],"acting":[4,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.9","version":"0'0","reported_seq":73,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-11T21:26:50.991796+0000","last_change":"2024-03-11T21:26:50.991832+0000","last_active":"2024-03-11T21:26:50.991796+0000","last_peered":"2024-03-11T21:26:50.991796+0000","last_clean":"2024-03-11T21:26:50.991796+0000","last_became_active":"2024-03-11T21:25:38.540976+0000","last_became_peered":"2024-03-11T21:25:38.540976+0000","last_unstale":"2024-03-11T21:26:50.991796+0000","last_undegraded":"2024-03-11T21:26:50.991796+0000","last_fullsized":"2024-03-11T21:26:50.991796+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:26:50.991712+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:26:50.991712+0000","last_clean_scrub_stamp":"2024-03-11T21:26:50.991712+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:27:58.101352+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,3],"acting":[1,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.f","version":"74'15","reported_seq":47,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-11T21:26:43.290730+0000","last_change":"2024-03-11T21:26:11.226642+0000","last_active":"2024-03-11T21:26:43.290730+0000","last_peered":"2024-03-11T21:26:43.290730+0000","last_clean":"2024-03-11T21:26:43.290730+0000","last_became_active":"2024-03-11T21:26:11.216224+0000","last_became_peered":"2024-03-11T21:26:11.216224+0000","last_unstale":"2024-03-11T21:26:43.290730+0000","last_undegraded":"2024-03-11T21:26:43.290730+0000","last_fullsized":"2024-03-11T21:26:43.290730+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:25:39.525554+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:39.525554+0000","last_clean_scrub_stamp":"2024-03-11T21:25:39.525554+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:27:08.139224+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,4],"acting":[2,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.e","version":"0'0","reported_seq":53,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-11T21:26:43.123979+0000","last_change":"2024-03-11T21:25:44.590476+0000","last_active":"2024-03-11T21:26:43.123979+0000","last_peered":"2024-03-11T21:26:43.123979+0000","last_clean":"2024-03-11T21:26:43.123979+0000","last_became_active":"2024-03-11T21:25:44.590198+0000","last_became_peered":"2024-03-11T21:25:44.590198+0000","last_unstale":"2024-03-11T21:26:43.123979+0000","last_undegraded":"2024-03-11T21:26:43.123979+0000","last_fullsized":"2024-03-11T21:26:43.123979+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:25:43.559452+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:43.559452+0000","last_clean_scrub_stamp":"2024-03-11T21:25:43.559452+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:27:00.005970+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.d","version":"74'8","reported_seq":81,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-11T21:26:48.591573+0000","last_change":"2024-03-11T21:25:42.575905+0000","last_active":"2024-03-11T21:26:48.591573+0000","last_peered":"2024-03-11T21:26:48.591573+0000","last_clean":"2024-03-11T21:26:48.591573+0000","last_became_active":"2024-03-11T21:25:42.575583+0000","last_became_peered":"2024-03-11T21:25:42.575583+0000","last_unstale":"2024-03-11T21:26:48.591573+0000","last_undegraded":"2024-03-11T21:26:48.591573+0000","last_fullsized":"2024-03-11T21:26:48.591573+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:25:41.544550+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:41.544550+0000","last_clean_scrub_stamp":"2024-03-11T21:25:41.544550+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,5],"acting":[2,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.a","version":"0'0","reported_seq":73,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-11T21:26:52.008073+0000","last_change":"2024-03-11T21:26:52.008094+0000","last_active":"2024-03-11T21:26:52.008073+0000","last_peered":"2024-03-11T21:26:52.008073+0000","last_clean":"2024-03-11T21:26:52.008073+0000","last_became_active":"2024-03-11T21:25:38.541045+0000","last_became_peered":"2024-03-11T21:25:38.541045+0000","last_unstale":"2024-03-11T21:26:52.008073+0000","last_undegraded":"2024-03-11T21:26:52.008073+0000","last_fullsized":"2024-03-11T21:26:52.008073+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:26:52.007996+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:37.512018+0000","last_clean_scrub_stamp":"2024-03-11T21:26:52.007996+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:27:52.862240+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,7],"acting":[1,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.c","version":"74'10","reported_seq":76,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-11T21:26:43.123721+0000","last_change":"2024-03-11T21:25:40.551223+0000","last_active":"2024-03-11T21:26:43.123721+0000","last_peered":"2024-03-11T21:26:43.123721+0000","last_clean":"2024-03-11T21:26:43.123721+0000","last_became_active":"2024-03-11T21:25:40.550885+0000","last_became_peered":"2024-03-11T21:25:40.550885+0000","last_unstale":"2024-03-11T21:26:43.123721+0000","last_undegraded":"2024-03-11T21:26:43.123721+0000","last_fullsized":"2024-03-11T21:26:43.123721+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:25:39.525554+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:39.525554+0000","last_clean_scrub_stamp":"2024-03-11T21:25:39.525554+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:26:48.279848+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.f","version":"0'0","reported_seq":53,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-11T21:26:43.291327+0000","last_change":"2024-03-11T21:25:44.593783+0000","last_active":"2024-03-11T21:26:43.291327+0000","last_peered":"2024-03-11T21:26:43.291327+0000","last_clean":"2024-03-11T21:26:43.291327+0000","last_became_active":"2024-03-11T21:25:44.593414+0000","last_became_peered":"2024-03-11T21:25:44.593414+0000","last_unstale":"2024-03-11T21:26:43.291327+0000","last_undegraded":"2024-03-11T21:26:43.291327+0000","last_fullsized":"2024-03-11T21:26:43.291327+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:25:43.559452+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:43.559452+0000","last_clean_scrub_stamp":"2024-03-11T21:25:43.559452+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:26:47.810287+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,4],"acting":[2,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":57,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-11T21:26:43.123297+0000","last_change":"2024-03-11T21:25:42.579815+0000","last_active":"2024-03-11T21:26:43.123297+0000","last_peered":"2024-03-11T21:26:43.123297+0000","last_clean":"2024-03-11T21:26:43.123297+0000","last_became_active":"2024-03-11T21:25:42.579409+0000","last_became_peered":"2024-03-11T21:25:42.579409+0000","last_unstale":"2024-03-11T21:26:43.123297+0000","last_undegraded":"2024-03-11T21:26:43.123297+0000","last_fullsized":"2024-03-11T21:26:43.123297+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:25:41.544550+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:41.544550+0000","last_clean_scrub_stamp":"2024-03-11T21:25:41.544550+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:27:11.166706+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.b","version":"0'0","reported_seq":65,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-11T21:26:45.417597+0000","last_change":"2024-03-11T21:25:38.544246+0000","last_active":"2024-03-11T21:26:45.417597+0000","last_peered":"2024-03-11T21:26:45.417597+0000","last_clean":"2024-03-11T21:26:45.417597+0000","last_became_active":"2024-03-11T21:25:38.543377+0000","last_became_peered":"2024-03-11T21:25:38.543377+0000","last_unstale":"2024-03-11T21:26:45.417597+0000","last_undegraded":"2024-03-11T21:26:45.417597+0000","last_fullsized":"2024-03-11T21:26:45.417597+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:25:37.512018+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:37.512018+0000","last_clean_scrub_stamp":"2024-03-11T21:25:37.512018+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:27:05.980046+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,5],"acting":[7,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.d","version":"74'17","reported_seq":89,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-11T21:26:43.124238+0000","last_change":"2024-03-11T21:25:40.553693+0000","last_active":"2024-03-11T21:26:43.124238+0000","last_peered":"2024-03-11T21:26:43.124238+0000","last_clean":"2024-03-11T21:26:43.124238+0000","last_became_active":"2024-03-11T21:25:40.553454+0000","last_became_peered":"2024-03-11T21:25:40.553454+0000","last_unstale":"2024-03-11T21:26:43.124238+0000","last_undegraded":"2024-03-11T21:26:43.124238+0000","last_fullsized":"2024-03-11T21:26:43.124238+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:25:39.525554+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:39.525554+0000","last_clean_scrub_stamp":"2024-03-11T21:25:39.525554+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:26:49.114384+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":29,"num_read_kb":19,"num_write":18,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,1],"acting":[4,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":53,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-11T21:26:45.417101+0000","last_change":"2024-03-11T21:25:44.590258+0000","last_active":"2024-03-11T21:26:45.417101+0000","last_peered":"2024-03-11T21:26:45.417101+0000","last_clean":"2024-03-11T21:26:45.417101+0000","last_became_active":"2024-03-11T21:25:44.589973+0000","last_became_peered":"2024-03-11T21:25:44.589973+0000","last_unstale":"2024-03-11T21:26:45.417101+0000","last_undegraded":"2024-03-11T21:26:45.417101+0000","last_fullsized":"2024-03-11T21:26:45.417101+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:25:43.559452+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:43.559452+0000","last_clean_scrub_stamp":"2024-03-11T21:25:43.559452+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:27:03.886246+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,3],"acting":[7,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":57,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-11T21:26:43.291053+0000","last_change":"2024-03-11T21:25:42.574877+0000","last_active":"2024-03-11T21:26:43.291053+0000","last_peered":"2024-03-11T21:26:43.291053+0000","last_clean":"2024-03-11T21:26:43.291053+0000","last_became_active":"2024-03-11T21:25:42.574515+0000","last_became_peered":"2024-03-11T21:25:42.574515+0000","last_unstale":"2024-03-11T21:26:43.291053+0000","last_undegraded":"2024-03-11T21:26:43.291053+0000","last_fullsized":"2024-03-11T21:26:43.291053+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:25:41.544550+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:41.544550+0000","last_clean_scrub_stamp":"2024-03-11T21:25:41.544550+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:27:09.339666+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,5],"acting":[2,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.c","version":"0'0","reported_seq":65,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-11T21:26:43.291201+0000","last_change":"2024-03-11T21:25:38.533719+0000","last_active":"2024-03-11T21:26:43.291201+0000","last_peered":"2024-03-11T21:26:43.291201+0000","last_clean":"2024-03-11T21:26:43.291201+0000","last_became_active":"2024-03-11T21:25:38.533240+0000","last_became_peered":"2024-03-11T21:25:38.533240+0000","last_unstale":"2024-03-11T21:26:43.291201+0000","last_undegraded":"2024-03-11T21:26:43.291201+0000","last_fullsized":"2024-03-11T21:26:43.291201+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:25:37.512018+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:37.512018+0000","last_clean_scrub_stamp":"2024-03-11T21:25:37.512018+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:27:06.014847+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,0],"acting":[2,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.a","version":"74'19","reported_seq":92,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-11T21:26:44.037495+0000","last_change":"2024-03-11T21:25:40.761035+0000","last_active":"2024-03-11T21:26:44.037495+0000","last_peered":"2024-03-11T21:26:44.037495+0000","last_clean":"2024-03-11T21:26:44.037495+0000","last_became_active":"2024-03-11T21:25:40.760698+0000","last_became_peered":"2024-03-11T21:25:40.760698+0000","last_unstale":"2024-03-11T21:26:44.037495+0000","last_undegraded":"2024-03-11T21:26:44.037495+0000","last_fullsized":"2024-03-11T21:26:44.037495+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:25:39.525554+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:39.525554+0000","last_clean_scrub_stamp":"2024-03-11T21:25:39.525554+0000","objects_scrubbed":0,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:26:59.237690+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":32,"num_read_kb":21,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,7],"acting":[6,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":53,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-11T21:26:43.122417+0000","last_change":"2024-03-11T21:25:44.585252+0000","last_active":"2024-03-11T21:26:43.122417+0000","last_peered":"2024-03-11T21:26:43.122417+0000","last_clean":"2024-03-11T21:26:43.122417+0000","last_became_active":"2024-03-11T21:25:44.584154+0000","last_became_peered":"2024-03-11T21:25:44.584154+0000","last_unstale":"2024-03-11T21:26:43.122417+0000","last_undegraded":"2024-03-11T21:26:43.122417+0000","last_fullsized":"2024-03-11T21:26:43.122417+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:25:43.559452+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:43.559452+0000","last_clean_scrub_stamp":"2024-03-11T21:25:43.559452+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:26:55.472231+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":57,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-11T21:26:43.290954+0000","last_change":"2024-03-11T21:25:42.577027+0000","last_active":"2024-03-11T21:26:43.290954+0000","last_peered":"2024-03-11T21:26:43.290954+0000","last_clean":"2024-03-11T21:26:43.290954+0000","last_became_active":"2024-03-11T21:25:42.576679+0000","last_became_peered":"2024-03-11T21:25:42.576679+0000","last_unstale":"2024-03-11T21:26:43.290954+0000","last_undegraded":"2024-03-11T21:26:43.290954+0000","last_fullsized":"2024-03-11T21:26:43.290954+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:25:41.544550+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:41.544550+0000","last_clean_scrub_stamp":"2024-03-11T21:25:41.544550+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:27:03.346770+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,3],"acting":[2,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.d","version":"0'0","reported_seq":73,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-11T21:26:45.885199+0000","last_change":"2024-03-11T21:26:45.885219+0000","last_active":"2024-03-11T21:26:45.885199+0000","last_peered":"2024-03-11T21:26:45.885199+0000","last_clean":"2024-03-11T21:26:45.885199+0000","last_became_active":"2024-03-11T21:25:38.539681+0000","last_became_peered":"2024-03-11T21:25:38.539681+0000","last_unstale":"2024-03-11T21:26:45.885199+0000","last_undegraded":"2024-03-11T21:26:45.885199+0000","last_fullsized":"2024-03-11T21:26:45.885199+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:26:45.885144+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:37.512018+0000","last_clean_scrub_stamp":"2024-03-11T21:26:45.885144+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:27:47.637748+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,3],"acting":[1,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.b","version":"74'9","reported_seq":77,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-11T21:26:43.123916+0000","last_change":"2024-03-11T21:25:40.549925+0000","last_active":"2024-03-11T21:26:43.123916+0000","last_peered":"2024-03-11T21:26:43.123916+0000","last_clean":"2024-03-11T21:26:43.123916+0000","last_became_active":"2024-03-11T21:25:40.549612+0000","last_became_peered":"2024-03-11T21:25:40.549612+0000","last_unstale":"2024-03-11T21:26:43.123916+0000","last_undegraded":"2024-03-11T21:26:43.123916+0000","last_fullsized":"2024-03-11T21:26:43.123916+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:25:39.525554+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:39.525554+0000","last_clean_scrub_stamp":"2024-03-11T21:25:39.525554+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:27:08.607529+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":61,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-11T21:26:55.966049+0000","last_change":"2024-03-11T21:26:55.966077+0000","last_active":"2024-03-11T21:26:55.966049+0000","last_peered":"2024-03-11T21:26:55.966049+0000","last_clean":"2024-03-11T21:26:55.966049+0000","last_became_active":"2024-03-11T21:25:45.300333+0000","last_became_peered":"2024-03-11T21:25:45.300333+0000","last_unstale":"2024-03-11T21:26:55.966049+0000","last_undegraded":"2024-03-11T21:26:55.966049+0000","last_fullsized":"2024-03-11T21:26:55.966049+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:26:55.965980+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:43.559452+0000","last_clean_scrub_stamp":"2024-03-11T21:26:55.965980+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:28:02.706121+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,0],"acting":[5,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.e","version":"0'0","reported_seq":65,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-11T21:26:43.291306+0000","last_change":"2024-03-11T21:25:38.540232+0000","last_active":"2024-03-11T21:26:43.291306+0000","last_peered":"2024-03-11T21:26:43.291306+0000","last_clean":"2024-03-11T21:26:43.291306+0000","last_became_active":"2024-03-11T21:25:38.539954+0000","last_became_peered":"2024-03-11T21:25:38.539954+0000","last_unstale":"2024-03-11T21:26:43.291306+0000","last_undegraded":"2024-03-11T21:26:43.291306+0000","last_fullsized":"2024-03-11T21:26:43.291306+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:25:37.512018+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:37.512018+0000","last_clean_scrub_stamp":"2024-03-11T21:25:37.512018+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:26:43.546973+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,7],"acting":[2,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.8","version":"74'15","reported_seq":94,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-11T21:26:50.934555+0000","last_change":"2024-03-11T21:26:50.934584+0000","last_active":"2024-03-11T21:26:50.934555+0000","last_peered":"2024-03-11T21:26:50.934555+0000","last_clean":"2024-03-11T21:26:50.934555+0000","last_became_active":"2024-03-11T21:25:40.761938+0000","last_became_peered":"2024-03-11T21:25:40.761938+0000","last_unstale":"2024-03-11T21:26:50.934555+0000","last_undegraded":"2024-03-11T21:26:50.934555+0000","last_fullsized":"2024-03-11T21:26:50.934555+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'15","last_scrub_stamp":"2024-03-11T21:26:50.934489+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:39.525554+0000","last_clean_scrub_stamp":"2024-03-11T21:26:50.934489+0000","objects_scrubbed":7,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:28:08.719408+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.9","version":"94'9","reported_seq":93,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-11T21:26:53.422673+0000","last_change":"2024-03-11T21:26:53.422673+0000","last_active":"2024-03-11T21:26:53.422673+0000","last_peered":"2024-03-11T21:26:53.422673+0000","last_clean":"2024-03-11T21:26:53.422673+0000","last_became_active":"2024-03-11T21:25:42.584461+0000","last_became_peered":"2024-03-11T21:25:42.584461+0000","last_unstale":"2024-03-11T21:26:53.422673+0000","last_undegraded":"2024-03-11T21:26:53.422673+0000","last_fullsized":"2024-03-11T21:26:53.422673+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"94'9","last_scrub_stamp":"2024-03-11T21:26:53.422552+0000","last_deep_scrub":"94'9","last_deep_scrub_stamp":"2024-03-11T21:26:53.422552+0000","last_clean_scrub_stamp":"2024-03-11T21:26:53.422552+0000","objects_scrubbed":1,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:28:07.835997+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":6,"num_pg":32,"stat_sum":{"num_bytes":383,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1149,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2,"ondisk_log_size":2,"up":96,"acting":96,"num_store_stats":8},{"poolid":5,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":65,"ondisk_log_size":65,"up":96,"acting":96,"num_store_stats":8},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":3702,"num_objects":178,"num_object_clones":0,"num_object_copies":534,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":178,"num_whiteouts":0,"num_read":698,"num_read_kb":455,"num_write":417,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":417792,"data_stored":11106,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":406,"ondisk_log_size":406,"up":96,"acting":96,"num_store_stats":8},{"poolid":3,"num_pg":3,"stat_sum":{"num_bytes":408,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":73,"num_read_kb":68,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1224,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":9,"acting":9,"num_store_stats":7},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":212,"num_read_kb":306,"num_write":179,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1771104,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1781760,"data_stored":1771104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":128,"ondisk_log_size":128,"up":3,"acting":3,"num_store_stats":3},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":1531,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":24,"num_read_kb":24,"num_write":10,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":4593,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":6,"ondisk_log_size":6,"up":96,"acting":96,"num_store_stats":8}],"osd_stats":[{"osd":7,"up_from":60,"seq":257698037779,"num_pgs":56,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28452,"kb_used_data":1436,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93716700,"statfs":{"total":95995035648,"available":95965900800,"internally_reserved":0,"allocated":1470464,"data_stored":977305,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":50,"seq":214748364825,"num_pgs":44,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28448,"kb_used_data":1432,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93716704,"statfs":{"total":95995035648,"available":95965904896,"internally_reserved":0,"allocated":1466368,"data_stored":977598,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574899,"num_pgs":53,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27900,"kb_used_data":884,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717252,"statfs":{"total":95995035648,"available":95966466048,"internally_reserved":0,"allocated":905216,"data_stored":387338,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738424,"num_pgs":47,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28456,"kb_used_data":1440,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93716696,"statfs":{"total":95995035648,"available":95965896704,"internally_reserved":0,"allocated":1474560,"data_stored":977522,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411373,"num_pgs":39,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27860,"kb_used_data":844,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717292,"statfs":{"total":95995035648,"available":95966507008,"internally_reserved":0,"allocated":864256,"data_stored":386562,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":26,"seq":111669149737,"num_pgs":55,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32060,"kb_used_data":884,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713092,"statfs":{"total":95995035648,"available":95962206208,"internally_reserved":0,"allocated":905216,"data_stored":387672,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":32,"seq":137438953506,"num_pgs":53,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27884,"kb_used_data":868,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717268,"statfs":{"total":95995035648,"available":95966482432,"internally_reserved":0,"allocated":888832,"data_stored":386846,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":41,"seq":176093659165,"num_pgs":49,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32024,"kb_used_data":848,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713128,"statfs":{"total":95995035648,"available":95962243072,"internally_reserved":0,"allocated":868352,"data_stored":386613,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1049,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":46,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":957,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":574,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":436,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1003,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":528,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":1320,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":81920,"data_stored":2118,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":1018,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":1898,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":1474,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":990,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":1034,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":1254,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":370,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":383,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":370,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-11T21:27:01.398 DEBUG:teuthology.orchestra.run.smithi092:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid 00db0016-dfed-11ee-95c8-87774f69a715 -- ceph pg dump --format=json 2024-03-11T21:27:01.432 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:27:01 smithi097 ceph-mon[33317]: 6.4 scrub starts 2024-03-11T21:27:01.432 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:27:01 smithi097 ceph-mon[33317]: 6.4 scrub ok 2024-03-11T21:27:01.432 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:27:01 smithi097 ceph-mon[33317]: 6.15 scrub starts 2024-03-11T21:27:01.433 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:27:01 smithi097 ceph-mon[33317]: 6.15 scrub ok 2024-03-11T21:27:01.433 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:27:01 smithi097 ceph-mon[33317]: from='client.14751 v1:172.21.15.92:0/3561928629' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-11T21:27:01.684 INFO:teuthology.orchestra.run.smithi092.stderr:Inferring config /var/lib/ceph/00db0016-dfed-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T21:27:02.327 INFO:journalctl@ceph.iscsi.iscsi.a.smithi097.stdout:Mar 11 21:27:01 smithi097 ceph-00db0016-dfed-11ee-95c8-87774f69a715-iscsi-iscsi-a[60943]: debug there is no tcmu-runner data available 2024-03-11T21:27:02.327 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:27:02 smithi097 ceph-mon[33317]: 4.c scrub starts 2024-03-11T21:27:02.327 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:27:02 smithi097 ceph-mon[33317]: 4.c scrub ok 2024-03-11T21:27:02.327 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:27:02 smithi097 ceph-mon[33317]: pgmap v30: 132 pgs: 132 active+clean; 582 KiB data, 228 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-11T21:27:02.327 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:27:02 smithi097 ceph-mon[33317]: 4.6 scrub starts 2024-03-11T21:27:02.328 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:27:02 smithi097 ceph-mon[33317]: 4.6 scrub ok 2024-03-11T21:27:02.328 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:27:02 smithi097 ceph-mon[33317]: 4.1d scrub starts 2024-03-11T21:27:02.328 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:27:02 smithi097 ceph-mon[33317]: 4.1d scrub ok 2024-03-11T21:27:02.481 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:27:02 smithi092 ceph-mon[29572]: 4.c scrub starts 2024-03-11T21:27:02.481 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:27:02 smithi092 ceph-mon[29572]: 4.c scrub ok 2024-03-11T21:27:02.481 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:27:02 smithi092 ceph-mon[29572]: pgmap v30: 132 pgs: 132 active+clean; 582 KiB data, 228 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-11T21:27:02.481 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:27:02 smithi092 ceph-mon[29572]: 4.6 scrub starts 2024-03-11T21:27:02.481 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:27:02 smithi092 ceph-mon[29572]: 4.6 scrub ok 2024-03-11T21:27:02.481 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:27:02 smithi092 ceph-mon[29572]: 4.1d scrub starts 2024-03-11T21:27:02.481 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:27:02 smithi092 ceph-mon[29572]: 4.1d scrub ok 2024-03-11T21:27:02.481 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:27:02 smithi092 ceph-mon[37563]: 4.c scrub starts 2024-03-11T21:27:02.481 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:27:02 smithi092 ceph-mon[37563]: 4.c scrub ok 2024-03-11T21:27:02.482 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:27:02 smithi092 ceph-mon[37563]: pgmap v30: 132 pgs: 132 active+clean; 582 KiB data, 228 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-11T21:27:02.482 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:27:02 smithi092 ceph-mon[37563]: 4.6 scrub starts 2024-03-11T21:27:02.482 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:27:02 smithi092 ceph-mon[37563]: 4.6 scrub ok 2024-03-11T21:27:02.482 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:27:02 smithi092 ceph-mon[37563]: 4.1d scrub starts 2024-03-11T21:27:02.482 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:27:02 smithi092 ceph-mon[37563]: 4.1d scrub ok 2024-03-11T21:27:03.379 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:27:03 smithi092 ceph-mon[29572]: from='client.24469 v1:172.21.15.97:0/404208310' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:27:03.380 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:27:03 smithi092 ceph-mon[29572]: 5.6 scrub starts 2024-03-11T21:27:03.380 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:27:03 smithi092 ceph-mon[29572]: 5.6 scrub ok 2024-03-11T21:27:03.380 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:27:03 smithi092 ceph-mon[37563]: from='client.24469 v1:172.21.15.97:0/404208310' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:27:03.380 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:27:03 smithi092 ceph-mon[37563]: 5.6 scrub starts 2024-03-11T21:27:03.380 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:27:03 smithi092 ceph-mon[37563]: 5.6 scrub ok 2024-03-11T21:27:03.577 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:27:03 smithi097 ceph-mon[33317]: from='client.24469 v1:172.21.15.97:0/404208310' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:27:03.577 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:27:03 smithi097 ceph-mon[33317]: 5.6 scrub starts 2024-03-11T21:27:03.577 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:27:03 smithi097 ceph-mon[33317]: 5.6 scrub ok 2024-03-11T21:27:03.886 INFO:teuthology.orchestra.run.smithi092.stdout: 2024-03-11T21:27:03.888 INFO:teuthology.orchestra.run.smithi092.stderr:dumped all 2024-03-11T21:27:04.260 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:27:04 smithi092 ceph-mon[37563]: 4.d scrub starts 2024-03-11T21:27:04.260 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:27:04 smithi092 ceph-mon[37563]: 4.d scrub ok 2024-03-11T21:27:04.260 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:27:04 smithi092 ceph-mon[37563]: pgmap v31: 132 pgs: 132 active+clean; 582 KiB data, 228 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T21:27:04.260 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:27:04 smithi092 ceph-mon[37563]: 6.f deep-scrub starts 2024-03-11T21:27:04.261 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:27:04 smithi092 ceph-mon[37563]: 6.f deep-scrub ok 2024-03-11T21:27:04.261 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:27:04 smithi092 ceph-mon[37563]: 6.13 scrub starts 2024-03-11T21:27:04.261 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:27:04 smithi092 ceph-mon[37563]: 6.13 scrub ok 2024-03-11T21:27:04.261 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:27:04 smithi092 ceph-mon[37563]: from='client.14757 v1:172.21.15.92:0/2255883365' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-11T21:27:04.261 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:27:04 smithi092 ceph-mon[29572]: 4.d scrub starts 2024-03-11T21:27:04.262 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:27:04 smithi092 ceph-mon[29572]: 4.d scrub ok 2024-03-11T21:27:04.262 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:27:04 smithi092 ceph-mon[29572]: pgmap v31: 132 pgs: 132 active+clean; 582 KiB data, 228 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T21:27:04.262 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:27:04 smithi092 ceph-mon[29572]: 6.f deep-scrub starts 2024-03-11T21:27:04.262 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:27:04 smithi092 ceph-mon[29572]: 6.f deep-scrub ok 2024-03-11T21:27:04.262 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:27:04 smithi092 ceph-mon[29572]: 6.13 scrub starts 2024-03-11T21:27:04.262 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:27:04 smithi092 ceph-mon[29572]: 6.13 scrub ok 2024-03-11T21:27:04.262 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:27:04 smithi092 ceph-mon[29572]: from='client.14757 v1:172.21.15.92:0/2255883365' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-11T21:27:04.577 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:27:04 smithi097 ceph-mon[33317]: 4.d scrub starts 2024-03-11T21:27:04.577 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:27:04 smithi097 ceph-mon[33317]: 4.d scrub ok 2024-03-11T21:27:04.577 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:27:04 smithi097 ceph-mon[33317]: pgmap v31: 132 pgs: 132 active+clean; 582 KiB data, 228 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T21:27:04.577 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:27:04 smithi097 ceph-mon[33317]: 6.f deep-scrub starts 2024-03-11T21:27:04.577 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:27:04 smithi097 ceph-mon[33317]: 6.f deep-scrub ok 2024-03-11T21:27:04.577 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:27:04 smithi097 ceph-mon[33317]: 6.13 scrub starts 2024-03-11T21:27:04.577 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:27:04 smithi097 ceph-mon[33317]: 6.13 scrub ok 2024-03-11T21:27:04.577 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:27:04 smithi097 ceph-mon[33317]: from='client.14757 v1:172.21.15.92:0/2255883365' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-11T21:27:04.896 INFO:teuthology.orchestra.run.smithi092.stdout:{"pg_ready":true,"pg_map":{"version":31,"stamp":"2024-03-11T21:27:02.964830+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":596392,"num_objects":198,"num_object_clones":0,"num_object_copies":594,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":198,"num_whiteouts":0,"num_read":1012,"num_read_kb":858,"num_write":615,"num_write_kb":2461,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":21,"num_bytes_recovered":1771324,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":620,"ondisk_log_size":620,"up":396,"acting":396,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":396,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":749961216,"kb_used":233084,"kb_used_data":8636,"kb_used_omap":12,"kb_used_meta":224371,"kb_avail":749728132,"statfs":{"total":767960285184,"available":767721607168,"internally_reserved":0,"allocated":8843264,"data_stored":4867456,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12752,"internal_metadata":229756464},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":15,"num_read_kb":15,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002705"},"pg_stats":[{"pgid":"6.1b","version":"0'0","reported_seq":61,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-11T21:26:53.584377+0000","last_change":"2024-03-11T21:26:53.584399+0000","last_active":"2024-03-11T21:26:53.584377+0000","last_peered":"2024-03-11T21:26:53.584377+0000","last_clean":"2024-03-11T21:26:53.584377+0000","last_became_active":"2024-03-11T21:25:45.302931+0000","last_became_peered":"2024-03-11T21:25:45.302931+0000","last_unstale":"2024-03-11T21:26:53.584377+0000","last_undegraded":"2024-03-11T21:26:53.584377+0000","last_fullsized":"2024-03-11T21:26:53.584377+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:26:53.584307+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:43.559452+0000","last_clean_scrub_stamp":"2024-03-11T21:26:53.584307+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:27:55.466757+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":57,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-11T21:26:43.123284+0000","last_change":"2024-03-11T21:25:42.579254+0000","last_active":"2024-03-11T21:26:43.123284+0000","last_peered":"2024-03-11T21:26:43.123284+0000","last_clean":"2024-03-11T21:26:43.123284+0000","last_became_active":"2024-03-11T21:25:42.578893+0000","last_became_peered":"2024-03-11T21:25:42.578893+0000","last_unstale":"2024-03-11T21:26:43.123284+0000","last_undegraded":"2024-03-11T21:26:43.123284+0000","last_fullsized":"2024-03-11T21:26:43.123284+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:25:41.544550+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:41.544550+0000","last_clean_scrub_stamp":"2024-03-11T21:25:41.544550+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:27:10.983445+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1f","version":"0'0","reported_seq":73,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-11T21:26:46.588443+0000","last_change":"2024-03-11T21:26:46.588463+0000","last_active":"2024-03-11T21:26:46.588443+0000","last_peered":"2024-03-11T21:26:46.588443+0000","last_clean":"2024-03-11T21:26:46.588443+0000","last_became_active":"2024-03-11T21:25:38.540510+0000","last_became_peered":"2024-03-11T21:25:38.540510+0000","last_unstale":"2024-03-11T21:26:46.588443+0000","last_undegraded":"2024-03-11T21:26:46.588443+0000","last_fullsized":"2024-03-11T21:26:46.588443+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:26:46.588380+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:37.512018+0000","last_clean_scrub_stamp":"2024-03-11T21:26:46.588380+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:28:06.851002+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,4],"acting":[0,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.19","version":"74'15","reported_seq":86,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-11T21:26:43.661037+0000","last_change":"2024-03-11T21:25:40.554584+0000","last_active":"2024-03-11T21:26:43.661037+0000","last_peered":"2024-03-11T21:26:43.661037+0000","last_clean":"2024-03-11T21:26:43.661037+0000","last_became_active":"2024-03-11T21:25:40.554207+0000","last_became_peered":"2024-03-11T21:25:40.554207+0000","last_unstale":"2024-03-11T21:26:43.661037+0000","last_undegraded":"2024-03-11T21:26:43.661037+0000","last_fullsized":"2024-03-11T21:26:43.661037+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:25:39.525554+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:39.525554+0000","last_clean_scrub_stamp":"2024-03-11T21:25:39.525554+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:26:54.679300+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.1a","version":"0'0","reported_seq":53,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-11T21:26:43.123537+0000","last_change":"2024-03-11T21:25:44.591548+0000","last_active":"2024-03-11T21:26:43.123537+0000","last_peered":"2024-03-11T21:26:43.123537+0000","last_clean":"2024-03-11T21:26:43.123537+0000","last_became_active":"2024-03-11T21:25:44.590923+0000","last_became_peered":"2024-03-11T21:25:44.590923+0000","last_unstale":"2024-03-11T21:26:43.123537+0000","last_undegraded":"2024-03-11T21:26:43.123537+0000","last_fullsized":"2024-03-11T21:26:43.123537+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:25:43.559452+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:43.559452+0000","last_clean_scrub_stamp":"2024-03-11T21:25:43.559452+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:27:00.390815+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,1],"acting":[4,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":57,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-11T21:26:43.122994+0000","last_change":"2024-03-11T21:25:42.578761+0000","last_active":"2024-03-11T21:26:43.122994+0000","last_peered":"2024-03-11T21:26:43.122994+0000","last_clean":"2024-03-11T21:26:43.122994+0000","last_became_active":"2024-03-11T21:25:42.578424+0000","last_became_peered":"2024-03-11T21:25:42.578424+0000","last_unstale":"2024-03-11T21:26:43.122994+0000","last_undegraded":"2024-03-11T21:26:43.122994+0000","last_fullsized":"2024-03-11T21:26:43.122994+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:25:41.544550+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:41.544550+0000","last_clean_scrub_stamp":"2024-03-11T21:25:41.544550+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:27:05.977939+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,7],"acting":[1,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1e","version":"0'0","reported_seq":73,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-11T21:26:49.595438+0000","last_change":"2024-03-11T21:26:49.595465+0000","last_active":"2024-03-11T21:26:49.595438+0000","last_peered":"2024-03-11T21:26:49.595438+0000","last_clean":"2024-03-11T21:26:49.595438+0000","last_became_active":"2024-03-11T21:25:38.537283+0000","last_became_peered":"2024-03-11T21:25:38.537283+0000","last_unstale":"2024-03-11T21:26:49.595438+0000","last_undegraded":"2024-03-11T21:26:49.595438+0000","last_fullsized":"2024-03-11T21:26:49.595438+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:26:49.595363+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:37.512018+0000","last_clean_scrub_stamp":"2024-03-11T21:26:49.595363+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:27:52.155430+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.18","version":"74'9","reported_seq":85,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-11T21:26:45.870036+0000","last_change":"2024-03-11T21:26:45.870099+0000","last_active":"2024-03-11T21:26:45.870036+0000","last_peered":"2024-03-11T21:26:45.870036+0000","last_clean":"2024-03-11T21:26:45.870036+0000","last_became_active":"2024-03-11T21:25:40.552216+0000","last_became_peered":"2024-03-11T21:25:40.552216+0000","last_unstale":"2024-03-11T21:26:45.870036+0000","last_undegraded":"2024-03-11T21:26:45.870036+0000","last_fullsized":"2024-03-11T21:26:45.870036+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'9","last_scrub_stamp":"2024-03-11T21:26:45.869943+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:39.525554+0000","last_clean_scrub_stamp":"2024-03-11T21:26:45.869943+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:27:50.731066+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":53,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-11T21:26:44.103957+0000","last_change":"2024-03-11T21:25:44.590541+0000","last_active":"2024-03-11T21:26:44.103957+0000","last_peered":"2024-03-11T21:26:44.103957+0000","last_clean":"2024-03-11T21:26:44.103957+0000","last_became_active":"2024-03-11T21:25:44.590234+0000","last_became_peered":"2024-03-11T21:25:44.590234+0000","last_unstale":"2024-03-11T21:26:44.103957+0000","last_undegraded":"2024-03-11T21:26:44.103957+0000","last_fullsized":"2024-03-11T21:26:44.103957+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:25:43.559452+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:43.559452+0000","last_clean_scrub_stamp":"2024-03-11T21:25:43.559452+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:26:54.344730+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3],"acting":[5,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":57,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-11T21:26:45.417397+0000","last_change":"2024-03-11T21:25:42.580998+0000","last_active":"2024-03-11T21:26:45.417397+0000","last_peered":"2024-03-11T21:26:45.417397+0000","last_clean":"2024-03-11T21:26:45.417397+0000","last_became_active":"2024-03-11T21:25:42.580533+0000","last_became_peered":"2024-03-11T21:25:42.580533+0000","last_unstale":"2024-03-11T21:26:45.417397+0000","last_undegraded":"2024-03-11T21:26:45.417397+0000","last_fullsized":"2024-03-11T21:26:45.417397+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:25:41.544550+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:41.544550+0000","last_clean_scrub_stamp":"2024-03-11T21:25:41.544550+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:27:06.038044+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.1d","version":"0'0","reported_seq":73,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-11T21:26:56.370884+0000","last_change":"2024-03-11T21:26:56.370911+0000","last_active":"2024-03-11T21:26:56.370884+0000","last_peered":"2024-03-11T21:26:56.370884+0000","last_clean":"2024-03-11T21:26:56.370884+0000","last_became_active":"2024-03-11T21:25:38.542628+0000","last_became_peered":"2024-03-11T21:25:38.542628+0000","last_unstale":"2024-03-11T21:26:56.370884+0000","last_undegraded":"2024-03-11T21:26:56.370884+0000","last_fullsized":"2024-03-11T21:26:56.370884+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:26:56.370811+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:37.512018+0000","last_clean_scrub_stamp":"2024-03-11T21:26:56.370811+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:28:03.069431+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,0],"acting":[7,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.1b","version":"74'5","reported_seq":71,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-11T21:26:43.123759+0000","last_change":"2024-03-11T21:25:40.556136+0000","last_active":"2024-03-11T21:26:43.123759+0000","last_peered":"2024-03-11T21:26:43.123759+0000","last_clean":"2024-03-11T21:26:43.123759+0000","last_became_active":"2024-03-11T21:25:40.555826+0000","last_became_peered":"2024-03-11T21:25:40.555826+0000","last_unstale":"2024-03-11T21:26:43.123759+0000","last_undegraded":"2024-03-11T21:26:43.123759+0000","last_fullsized":"2024-03-11T21:26:43.123759+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:25:39.525554+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:39.525554+0000","last_clean_scrub_stamp":"2024-03-11T21:25:39.525554+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:27:08.490979+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":11,"num_read_kb":7,"num_write":6,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,1],"acting":[4,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":53,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-11T21:26:43.122899+0000","last_change":"2024-03-11T21:25:44.590228+0000","last_active":"2024-03-11T21:26:43.122899+0000","last_peered":"2024-03-11T21:26:43.122899+0000","last_clean":"2024-03-11T21:26:43.122899+0000","last_became_active":"2024-03-11T21:25:44.589786+0000","last_became_peered":"2024-03-11T21:25:44.589786+0000","last_unstale":"2024-03-11T21:26:43.122899+0000","last_undegraded":"2024-03-11T21:26:43.122899+0000","last_fullsized":"2024-03-11T21:26:43.122899+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:25:43.559452+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:43.559452+0000","last_clean_scrub_stamp":"2024-03-11T21:25:43.559452+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:27:12.142655+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,7],"acting":[0,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":57,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-11T21:26:44.104432+0000","last_change":"2024-03-11T21:25:42.574391+0000","last_active":"2024-03-11T21:26:44.104432+0000","last_peered":"2024-03-11T21:26:44.104432+0000","last_clean":"2024-03-11T21:26:44.104432+0000","last_became_active":"2024-03-11T21:25:42.573831+0000","last_became_peered":"2024-03-11T21:25:42.573831+0000","last_unstale":"2024-03-11T21:26:44.104432+0000","last_undegraded":"2024-03-11T21:26:44.104432+0000","last_fullsized":"2024-03-11T21:26:44.104432+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:25:41.544550+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:41.544550+0000","last_clean_scrub_stamp":"2024-03-11T21:25:41.544550+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:26:53.725038+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,7],"acting":[5,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1c","version":"0'0","reported_seq":65,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-11T21:26:45.417664+0000","last_change":"2024-03-11T21:25:38.542244+0000","last_active":"2024-03-11T21:26:45.417664+0000","last_peered":"2024-03-11T21:26:45.417664+0000","last_clean":"2024-03-11T21:26:45.417664+0000","last_became_active":"2024-03-11T21:25:38.541364+0000","last_became_peered":"2024-03-11T21:25:38.541364+0000","last_unstale":"2024-03-11T21:26:45.417664+0000","last_undegraded":"2024-03-11T21:26:45.417664+0000","last_fullsized":"2024-03-11T21:26:45.417664+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:25:37.512018+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:37.512018+0000","last_clean_scrub_stamp":"2024-03-11T21:25:37.512018+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:27:03.620571+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.1a","version":"74'9","reported_seq":85,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-11T21:26:44.851036+0000","last_change":"2024-03-11T21:26:44.851070+0000","last_active":"2024-03-11T21:26:44.851036+0000","last_peered":"2024-03-11T21:26:44.851036+0000","last_clean":"2024-03-11T21:26:44.851036+0000","last_became_active":"2024-03-11T21:25:40.556629+0000","last_became_peered":"2024-03-11T21:25:40.556629+0000","last_unstale":"2024-03-11T21:26:44.851036+0000","last_undegraded":"2024-03-11T21:26:44.851036+0000","last_fullsized":"2024-03-11T21:26:44.851036+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'9","last_scrub_stamp":"2024-03-11T21:26:44.850930+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:39.525554+0000","last_clean_scrub_stamp":"2024-03-11T21:26:44.850930+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:27:47.017552+0000","scrub_duration":24,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0],"acting":[4,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1f","version":"0'0","reported_seq":53,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-11T21:26:43.661614+0000","last_change":"2024-03-11T21:25:45.303017+0000","last_active":"2024-03-11T21:26:43.661614+0000","last_peered":"2024-03-11T21:26:43.661614+0000","last_clean":"2024-03-11T21:26:43.661614+0000","last_became_active":"2024-03-11T21:25:45.302637+0000","last_became_peered":"2024-03-11T21:25:45.302637+0000","last_unstale":"2024-03-11T21:26:43.661614+0000","last_undegraded":"2024-03-11T21:26:43.661614+0000","last_fullsized":"2024-03-11T21:26:43.661614+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:25:43.559452+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:43.559452+0000","last_clean_scrub_stamp":"2024-03-11T21:25:43.559452+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:26:53.655260+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1c","version":"0'0","reported_seq":57,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-11T21:26:43.123925+0000","last_change":"2024-03-11T21:25:42.576745+0000","last_active":"2024-03-11T21:26:43.123925+0000","last_peered":"2024-03-11T21:26:43.123925+0000","last_clean":"2024-03-11T21:26:43.123925+0000","last_became_active":"2024-03-11T21:25:42.576064+0000","last_became_peered":"2024-03-11T21:25:42.576064+0000","last_unstale":"2024-03-11T21:26:43.123925+0000","last_undegraded":"2024-03-11T21:26:43.123925+0000","last_fullsized":"2024-03-11T21:26:43.123925+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:25:41.544550+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:41.544550+0000","last_clean_scrub_stamp":"2024-03-11T21:25:41.544550+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:26:53.587425+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1b","version":"0'0","reported_seq":73,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-11T21:26:55.579961+0000","last_change":"2024-03-11T21:26:55.579982+0000","last_active":"2024-03-11T21:26:55.579961+0000","last_peered":"2024-03-11T21:26:55.579961+0000","last_clean":"2024-03-11T21:26:55.579961+0000","last_became_active":"2024-03-11T21:25:38.537520+0000","last_became_peered":"2024-03-11T21:25:38.537520+0000","last_unstale":"2024-03-11T21:26:55.579961+0000","last_undegraded":"2024-03-11T21:26:55.579961+0000","last_fullsized":"2024-03-11T21:26:55.579961+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:26:55.579901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:37.512018+0000","last_clean_scrub_stamp":"2024-03-11T21:26:55.579901+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:28:17.884453+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1d","version":"74'12","reported_seq":92,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-11T21:27:01.629540+0000","last_change":"2024-03-11T21:27:01.629574+0000","last_active":"2024-03-11T21:27:01.629540+0000","last_peered":"2024-03-11T21:27:01.629540+0000","last_clean":"2024-03-11T21:27:01.629540+0000","last_became_active":"2024-03-11T21:25:40.556330+0000","last_became_peered":"2024-03-11T21:25:40.556330+0000","last_unstale":"2024-03-11T21:27:01.629540+0000","last_undegraded":"2024-03-11T21:27:01.629540+0000","last_fullsized":"2024-03-11T21:27:01.629540+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'12","last_scrub_stamp":"2024-03-11T21:27:01.629471+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:39.525554+0000","last_clean_scrub_stamp":"2024-03-11T21:27:01.629471+0000","objects_scrubbed":5,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:28:26.432819+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":25,"num_read_kb":16,"num_write":14,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.1e","version":"0'0","reported_seq":53,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-11T21:26:43.124283+0000","last_change":"2024-03-11T21:25:45.300689+0000","last_active":"2024-03-11T21:26:43.124283+0000","last_peered":"2024-03-11T21:26:43.124283+0000","last_clean":"2024-03-11T21:26:43.124283+0000","last_became_active":"2024-03-11T21:25:45.300355+0000","last_became_peered":"2024-03-11T21:25:45.300355+0000","last_unstale":"2024-03-11T21:26:43.124283+0000","last_undegraded":"2024-03-11T21:26:43.124283+0000","last_fullsized":"2024-03-11T21:26:43.124283+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:25:43.559452+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:43.559452+0000","last_clean_scrub_stamp":"2024-03-11T21:25:43.559452+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:27:05.215620+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5],"acting":[4,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1d","version":"0'0","reported_seq":57,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-11T21:26:43.124268+0000","last_change":"2024-03-11T21:25:42.579792+0000","last_active":"2024-03-11T21:26:43.124268+0000","last_peered":"2024-03-11T21:26:43.124268+0000","last_clean":"2024-03-11T21:26:43.124268+0000","last_became_active":"2024-03-11T21:25:42.579407+0000","last_became_peered":"2024-03-11T21:25:42.579407+0000","last_unstale":"2024-03-11T21:26:43.124268+0000","last_undegraded":"2024-03-11T21:26:43.124268+0000","last_fullsized":"2024-03-11T21:26:43.124268+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:25:41.544550+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:41.544550+0000","last_clean_scrub_stamp":"2024-03-11T21:25:41.544550+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:27:03.100488+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1a","version":"0'0","reported_seq":65,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-11T21:26:44.037494+0000","last_change":"2024-03-11T21:25:38.540007+0000","last_active":"2024-03-11T21:26:44.037494+0000","last_peered":"2024-03-11T21:26:44.037494+0000","last_clean":"2024-03-11T21:26:44.037494+0000","last_became_active":"2024-03-11T21:25:38.539725+0000","last_became_peered":"2024-03-11T21:25:38.539725+0000","last_unstale":"2024-03-11T21:26:44.037494+0000","last_undegraded":"2024-03-11T21:26:44.037494+0000","last_fullsized":"2024-03-11T21:26:44.037494+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:25:37.512018+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:37.512018+0000","last_clean_scrub_stamp":"2024-03-11T21:25:37.512018+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:26:56.284995+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.1c","version":"74'15","reported_seq":86,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-11T21:26:43.291216+0000","last_change":"2024-03-11T21:25:40.551757+0000","last_active":"2024-03-11T21:26:43.291216+0000","last_peered":"2024-03-11T21:26:43.291216+0000","last_clean":"2024-03-11T21:26:43.291216+0000","last_became_active":"2024-03-11T21:25:40.551453+0000","last_became_peered":"2024-03-11T21:25:40.551453+0000","last_unstale":"2024-03-11T21:26:43.291216+0000","last_undegraded":"2024-03-11T21:26:43.291216+0000","last_fullsized":"2024-03-11T21:26:43.291216+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:25:39.525554+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:39.525554+0000","last_clean_scrub_stamp":"2024-03-11T21:25:39.525554+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:26:56.664246+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.1d","version":"0'0","reported_seq":53,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-11T21:26:43.123803+0000","last_change":"2024-03-11T21:25:44.589198+0000","last_active":"2024-03-11T21:26:43.123803+0000","last_peered":"2024-03-11T21:26:43.123803+0000","last_clean":"2024-03-11T21:26:43.123803+0000","last_became_active":"2024-03-11T21:25:44.588845+0000","last_became_peered":"2024-03-11T21:25:44.588845+0000","last_unstale":"2024-03-11T21:26:43.123803+0000","last_undegraded":"2024-03-11T21:26:43.123803+0000","last_fullsized":"2024-03-11T21:26:43.123803+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:25:43.559452+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:43.559452+0000","last_clean_scrub_stamp":"2024-03-11T21:25:43.559452+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:27:08.822378+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.1e","version":"0'0","reported_seq":65,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-11T21:26:58.335493+0000","last_change":"2024-03-11T21:26:58.335515+0000","last_active":"2024-03-11T21:26:58.335493+0000","last_peered":"2024-03-11T21:26:58.335493+0000","last_clean":"2024-03-11T21:26:58.335493+0000","last_became_active":"2024-03-11T21:25:42.582959+0000","last_became_peered":"2024-03-11T21:25:42.582959+0000","last_unstale":"2024-03-11T21:26:58.335493+0000","last_undegraded":"2024-03-11T21:26:58.335493+0000","last_fullsized":"2024-03-11T21:26:58.335493+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:26:58.335431+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:26:58.335431+0000","last_clean_scrub_stamp":"2024-03-11T21:26:58.335431+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:28:08.893453+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.19","version":"67'2","reported_seq":72,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-11T21:26:43.661117+0000","last_change":"2024-03-11T21:25:38.539600+0000","last_active":"2024-03-11T21:26:43.661117+0000","last_peered":"2024-03-11T21:26:43.661117+0000","last_clean":"2024-03-11T21:26:43.661117+0000","last_became_active":"2024-03-11T21:25:38.539243+0000","last_became_peered":"2024-03-11T21:25:38.539243+0000","last_unstale":"2024-03-11T21:26:43.661117+0000","last_undegraded":"2024-03-11T21:26:43.661117+0000","last_fullsized":"2024-03-11T21:26:43.661117+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:25:37.512018+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:37.512018+0000","last_clean_scrub_stamp":"2024-03-11T21:25:37.512018+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:26:59.905535+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":957,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":7,"num_read_kb":7,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,0],"acting":[3,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1f","version":"74'11","reported_seq":88,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-11T21:26:55.050937+0000","last_change":"2024-03-11T21:26:55.050980+0000","last_active":"2024-03-11T21:26:55.050937+0000","last_peered":"2024-03-11T21:26:55.050937+0000","last_clean":"2024-03-11T21:26:55.050937+0000","last_became_active":"2024-03-11T21:25:40.554686+0000","last_became_peered":"2024-03-11T21:25:40.554686+0000","last_unstale":"2024-03-11T21:26:55.050937+0000","last_undegraded":"2024-03-11T21:26:55.050937+0000","last_fullsized":"2024-03-11T21:26:55.050937+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'11","last_scrub_stamp":"2024-03-11T21:26:55.050872+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:39.525554+0000","last_clean_scrub_stamp":"2024-03-11T21:26:55.050872+0000","objects_scrubbed":5,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:28:13.716793+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,1],"acting":[6,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.1c","version":"74'1","reported_seq":54,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-11T21:26:45.418002+0000","last_change":"2024-03-11T21:25:44.593623+0000","last_active":"2024-03-11T21:26:45.418002+0000","last_peered":"2024-03-11T21:26:45.418002+0000","last_clean":"2024-03-11T21:26:45.418002+0000","last_became_active":"2024-03-11T21:25:44.593326+0000","last_became_peered":"2024-03-11T21:25:44.593326+0000","last_unstale":"2024-03-11T21:26:45.418002+0000","last_undegraded":"2024-03-11T21:26:45.418002+0000","last_fullsized":"2024-03-11T21:26:45.418002+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:25:43.559452+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:43.559452+0000","last_clean_scrub_stamp":"2024-03-11T21:25:43.559452+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:27:07.557915+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":370,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.1f","version":"74'8","reported_seq":93,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-11T21:26:53.592679+0000","last_change":"2024-03-11T21:26:48.955663+0000","last_active":"2024-03-11T21:26:53.592679+0000","last_peered":"2024-03-11T21:26:53.592679+0000","last_clean":"2024-03-11T21:26:53.592679+0000","last_became_active":"2024-03-11T21:25:42.577401+0000","last_became_peered":"2024-03-11T21:25:42.577401+0000","last_unstale":"2024-03-11T21:26:53.592679+0000","last_undegraded":"2024-03-11T21:26:53.592679+0000","last_fullsized":"2024-03-11T21:26:53.592679+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'8","last_scrub_stamp":"2024-03-11T21:26:48.955484+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:41.544550+0000","last_clean_scrub_stamp":"2024-03-11T21:26:48.955484+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:27:51.651922+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.18","version":"67'1","reported_seq":79,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-11T21:26:46.995320+0000","last_change":"2024-03-11T21:26:46.995375+0000","last_active":"2024-03-11T21:26:46.995320+0000","last_peered":"2024-03-11T21:26:46.995320+0000","last_clean":"2024-03-11T21:26:46.995320+0000","last_became_active":"2024-03-11T21:25:38.538321+0000","last_became_peered":"2024-03-11T21:25:38.538321+0000","last_unstale":"2024-03-11T21:26:46.995320+0000","last_undegraded":"2024-03-11T21:26:46.995320+0000","last_fullsized":"2024-03-11T21:26:46.995320+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'1","last_scrub_stamp":"2024-03-11T21:26:46.995228+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:37.512018+0000","last_clean_scrub_stamp":"2024-03-11T21:26:46.995228+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:27:58.922326+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":436,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":7,"num_read_kb":7,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,7],"acting":[5,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1e","version":"74'10","reported_seq":84,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-11T21:26:43.123360+0000","last_change":"2024-03-11T21:26:42.521121+0000","last_active":"2024-03-11T21:26:43.123360+0000","last_peered":"2024-03-11T21:26:43.123360+0000","last_clean":"2024-03-11T21:26:43.123360+0000","last_became_active":"2024-03-11T21:25:40.763438+0000","last_became_peered":"2024-03-11T21:25:40.763438+0000","last_unstale":"2024-03-11T21:26:43.123360+0000","last_undegraded":"2024-03-11T21:26:43.123360+0000","last_fullsized":"2024-03-11T21:26:43.123360+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'10","last_scrub_stamp":"2024-03-11T21:26:42.521016+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:39.525554+0000","last_clean_scrub_stamp":"2024-03-11T21:26:42.521016+0000","objects_scrubbed":5,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:28:12.491436+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.13","version":"0'0","reported_seq":53,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-11T21:26:43.661347+0000","last_change":"2024-03-11T21:25:45.302505+0000","last_active":"2024-03-11T21:26:43.661347+0000","last_peered":"2024-03-11T21:26:43.661347+0000","last_clean":"2024-03-11T21:26:43.661347+0000","last_became_active":"2024-03-11T21:25:45.302102+0000","last_became_peered":"2024-03-11T21:25:45.302102+0000","last_unstale":"2024-03-11T21:26:43.661347+0000","last_undegraded":"2024-03-11T21:26:43.661347+0000","last_fullsized":"2024-03-11T21:26:43.661347+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:25:43.559452+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:43.559452+0000","last_clean_scrub_stamp":"2024-03-11T21:25:43.559452+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:26:53.208128+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":57,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-11T21:26:45.417478+0000","last_change":"2024-03-11T21:25:42.580085+0000","last_active":"2024-03-11T21:26:45.417478+0000","last_peered":"2024-03-11T21:26:45.417478+0000","last_clean":"2024-03-11T21:26:45.417478+0000","last_became_active":"2024-03-11T21:25:42.579781+0000","last_became_peered":"2024-03-11T21:25:42.579781+0000","last_unstale":"2024-03-11T21:26:45.417478+0000","last_undegraded":"2024-03-11T21:26:45.417478+0000","last_fullsized":"2024-03-11T21:26:45.417478+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:25:41.544550+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:41.544550+0000","last_clean_scrub_stamp":"2024-03-11T21:25:41.544550+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:27:05.067119+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.17","version":"0'0","reported_seq":73,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-11T21:26:56.020287+0000","last_change":"2024-03-11T21:26:56.020325+0000","last_active":"2024-03-11T21:26:56.020287+0000","last_peered":"2024-03-11T21:26:56.020287+0000","last_clean":"2024-03-11T21:26:56.020287+0000","last_became_active":"2024-03-11T21:25:38.540349+0000","last_became_peered":"2024-03-11T21:25:38.540349+0000","last_unstale":"2024-03-11T21:26:56.020287+0000","last_undegraded":"2024-03-11T21:26:56.020287+0000","last_fullsized":"2024-03-11T21:26:56.020287+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:26:56.020191+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:37.512018+0000","last_clean_scrub_stamp":"2024-03-11T21:26:56.020191+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:28:25.054065+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,2],"acting":[6,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.11","version":"74'11","reported_seq":80,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-11T21:26:43.661547+0000","last_change":"2024-03-11T21:25:40.761012+0000","last_active":"2024-03-11T21:26:43.661547+0000","last_peered":"2024-03-11T21:26:43.661547+0000","last_clean":"2024-03-11T21:26:43.661547+0000","last_became_active":"2024-03-11T21:25:40.760664+0000","last_became_peered":"2024-03-11T21:25:40.760664+0000","last_unstale":"2024-03-11T21:26:43.661547+0000","last_undegraded":"2024-03-11T21:26:43.661547+0000","last_fullsized":"2024-03-11T21:26:43.661547+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:25:39.525554+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:39.525554+0000","last_clean_scrub_stamp":"2024-03-11T21:25:39.525554+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:26:57.124438+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":53,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-11T21:26:45.417330+0000","last_change":"2024-03-11T21:25:44.591375+0000","last_active":"2024-03-11T21:26:45.417330+0000","last_peered":"2024-03-11T21:26:45.417330+0000","last_clean":"2024-03-11T21:26:45.417330+0000","last_became_active":"2024-03-11T21:25:44.591068+0000","last_became_peered":"2024-03-11T21:25:44.591068+0000","last_unstale":"2024-03-11T21:26:45.417330+0000","last_undegraded":"2024-03-11T21:26:45.417330+0000","last_fullsized":"2024-03-11T21:26:45.417330+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:25:43.559452+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:43.559452+0000","last_clean_scrub_stamp":"2024-03-11T21:25:43.559452+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:27:04.984773+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,4],"acting":[7,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":57,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-11T21:26:44.037422+0000","last_change":"2024-03-11T21:25:42.584581+0000","last_active":"2024-03-11T21:26:44.037422+0000","last_peered":"2024-03-11T21:26:44.037422+0000","last_clean":"2024-03-11T21:26:44.037422+0000","last_became_active":"2024-03-11T21:25:42.584148+0000","last_became_peered":"2024-03-11T21:25:42.584148+0000","last_unstale":"2024-03-11T21:26:44.037422+0000","last_undegraded":"2024-03-11T21:26:44.037422+0000","last_fullsized":"2024-03-11T21:26:44.037422+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:25:41.544550+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:41.544550+0000","last_clean_scrub_stamp":"2024-03-11T21:25:41.544550+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:27:00.403188+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.16","version":"0'0","reported_seq":65,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-11T21:26:44.104846+0000","last_change":"2024-03-11T21:25:38.539399+0000","last_active":"2024-03-11T21:26:44.104846+0000","last_peered":"2024-03-11T21:26:44.104846+0000","last_clean":"2024-03-11T21:26:44.104846+0000","last_became_active":"2024-03-11T21:25:38.539103+0000","last_became_peered":"2024-03-11T21:25:38.539103+0000","last_unstale":"2024-03-11T21:26:44.104846+0000","last_undegraded":"2024-03-11T21:26:44.104846+0000","last_fullsized":"2024-03-11T21:26:44.104846+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:25:37.512018+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:37.512018+0000","last_clean_scrub_stamp":"2024-03-11T21:25:37.512018+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:26:51.659287+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,2],"acting":[5,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.10","version":"94'6","reported_seq":77,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-11T21:26:58.601888+0000","last_change":"2024-03-11T21:26:58.601920+0000","last_active":"2024-03-11T21:26:58.601888+0000","last_peered":"2024-03-11T21:26:58.601888+0000","last_clean":"2024-03-11T21:26:58.601888+0000","last_became_active":"2024-03-11T21:25:40.552720+0000","last_became_peered":"2024-03-11T21:25:40.552720+0000","last_unstale":"2024-03-11T21:26:58.601888+0000","last_undegraded":"2024-03-11T21:26:58.601888+0000","last_fullsized":"2024-03-11T21:26:58.601888+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"94'6","last_scrub_stamp":"2024-03-11T21:26:58.601831+0000","last_deep_scrub":"94'6","last_deep_scrub_stamp":"2024-03-11T21:26:58.601831+0000","last_clean_scrub_stamp":"2024-03-11T21:26:58.601831+0000","objects_scrubbed":2,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:28:12.313486+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":6,"num_read_kb":4,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6],"acting":[3,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":53,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-11T21:26:43.660898+0000","last_change":"2024-03-11T21:25:44.583516+0000","last_active":"2024-03-11T21:26:43.660898+0000","last_peered":"2024-03-11T21:26:43.660898+0000","last_clean":"2024-03-11T21:26:43.660898+0000","last_became_active":"2024-03-11T21:25:44.583181+0000","last_became_peered":"2024-03-11T21:25:44.583181+0000","last_unstale":"2024-03-11T21:26:43.660898+0000","last_undegraded":"2024-03-11T21:26:43.660898+0000","last_fullsized":"2024-03-11T21:26:43.660898+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:25:43.559452+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:43.559452+0000","last_clean_scrub_stamp":"2024-03-11T21:25:43.559452+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:26:53.958265+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":57,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-11T21:26:43.124642+0000","last_change":"2024-03-11T21:25:42.573497+0000","last_active":"2024-03-11T21:26:43.124642+0000","last_peered":"2024-03-11T21:26:43.124642+0000","last_clean":"2024-03-11T21:26:43.124642+0000","last_became_active":"2024-03-11T21:25:42.573158+0000","last_became_peered":"2024-03-11T21:25:42.573158+0000","last_unstale":"2024-03-11T21:26:43.124642+0000","last_undegraded":"2024-03-11T21:26:43.124642+0000","last_fullsized":"2024-03-11T21:26:43.124642+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:25:41.544550+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:41.544550+0000","last_clean_scrub_stamp":"2024-03-11T21:25:41.544550+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:27:10.181203+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.15","version":"0'0","reported_seq":73,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-11T21:26:44.860570+0000","last_change":"2024-03-11T21:26:44.860603+0000","last_active":"2024-03-11T21:26:44.860570+0000","last_peered":"2024-03-11T21:26:44.860570+0000","last_clean":"2024-03-11T21:26:44.860570+0000","last_became_active":"2024-03-11T21:25:38.541705+0000","last_became_peered":"2024-03-11T21:25:38.541705+0000","last_unstale":"2024-03-11T21:26:44.860570+0000","last_undegraded":"2024-03-11T21:26:44.860570+0000","last_fullsized":"2024-03-11T21:26:44.860570+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:26:44.860499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:26:44.860499+0000","last_clean_scrub_stamp":"2024-03-11T21:26:44.860499+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:28:11.867124+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.13","version":"74'11","reported_seq":98,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-11T21:26:43.123914+0000","last_change":"2024-03-11T21:26:11.223911+0000","last_active":"2024-03-11T21:26:43.123914+0000","last_peered":"2024-03-11T21:26:43.123914+0000","last_clean":"2024-03-11T21:26:43.123914+0000","last_became_active":"2024-03-11T21:26:11.214879+0000","last_became_peered":"2024-03-11T21:26:11.214879+0000","last_unstale":"2024-03-11T21:26:43.123914+0000","last_undegraded":"2024-03-11T21:26:43.123914+0000","last_fullsized":"2024-03-11T21:26:43.123914+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:25:39.525554+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:39.525554+0000","last_clean_scrub_stamp":"2024-03-11T21:25:39.525554+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:26:50.821337+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,2],"acting":[4,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.10","version":"0'0","reported_seq":53,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-11T21:26:43.122385+0000","last_change":"2024-03-11T21:25:44.591915+0000","last_active":"2024-03-11T21:26:43.122385+0000","last_peered":"2024-03-11T21:26:43.122385+0000","last_clean":"2024-03-11T21:26:43.122385+0000","last_became_active":"2024-03-11T21:25:44.591189+0000","last_became_peered":"2024-03-11T21:25:44.591189+0000","last_unstale":"2024-03-11T21:26:43.122385+0000","last_undegraded":"2024-03-11T21:26:43.122385+0000","last_fullsized":"2024-03-11T21:26:43.122385+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:25:43.559452+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:43.559452+0000","last_clean_scrub_stamp":"2024-03-11T21:25:43.559452+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:27:08.613618+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,1],"acting":[0,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":57,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-11T21:26:43.661240+0000","last_change":"2024-03-11T21:25:42.576742+0000","last_active":"2024-03-11T21:26:43.661240+0000","last_peered":"2024-03-11T21:26:43.661240+0000","last_clean":"2024-03-11T21:26:43.661240+0000","last_became_active":"2024-03-11T21:25:42.576425+0000","last_became_peered":"2024-03-11T21:25:42.576425+0000","last_unstale":"2024-03-11T21:26:43.661240+0000","last_undegraded":"2024-03-11T21:26:43.661240+0000","last_fullsized":"2024-03-11T21:26:43.661240+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:25:41.544550+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:41.544550+0000","last_clean_scrub_stamp":"2024-03-11T21:25:41.544550+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:26:54.144762+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.14","version":"0'0","reported_seq":73,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-11T21:26:59.050037+0000","last_change":"2024-03-11T21:26:59.050063+0000","last_active":"2024-03-11T21:26:59.050037+0000","last_peered":"2024-03-11T21:26:59.050037+0000","last_clean":"2024-03-11T21:26:59.050037+0000","last_became_active":"2024-03-11T21:25:38.541852+0000","last_became_peered":"2024-03-11T21:25:38.541852+0000","last_unstale":"2024-03-11T21:26:59.050037+0000","last_undegraded":"2024-03-11T21:26:59.050037+0000","last_fullsized":"2024-03-11T21:26:59.050037+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:26:59.049960+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:37.512018+0000","last_clean_scrub_stamp":"2024-03-11T21:26:59.049960+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:28:08.381102+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,5],"acting":[6,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.12","version":"74'9","reported_seq":77,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-11T21:26:43.124565+0000","last_change":"2024-03-11T21:25:40.549164+0000","last_active":"2024-03-11T21:26:43.124565+0000","last_peered":"2024-03-11T21:26:43.124565+0000","last_clean":"2024-03-11T21:26:43.124565+0000","last_became_active":"2024-03-11T21:25:40.547082+0000","last_became_peered":"2024-03-11T21:25:40.547082+0000","last_unstale":"2024-03-11T21:26:43.124565+0000","last_undegraded":"2024-03-11T21:26:43.124565+0000","last_fullsized":"2024-03-11T21:26:43.124565+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:25:39.525554+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:39.525554+0000","last_clean_scrub_stamp":"2024-03-11T21:25:39.525554+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:27:04.348158+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":53,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-11T21:26:43.124523+0000","last_change":"2024-03-11T21:25:44.589956+0000","last_active":"2024-03-11T21:26:43.124523+0000","last_peered":"2024-03-11T21:26:43.124523+0000","last_clean":"2024-03-11T21:26:43.124523+0000","last_became_active":"2024-03-11T21:25:44.589586+0000","last_became_peered":"2024-03-11T21:25:44.589586+0000","last_unstale":"2024-03-11T21:26:43.124523+0000","last_undegraded":"2024-03-11T21:26:43.124523+0000","last_fullsized":"2024-03-11T21:26:43.124523+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:25:43.559452+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:43.559452+0000","last_clean_scrub_stamp":"2024-03-11T21:25:43.559452+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:26:51.679819+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,5],"acting":[4,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.14","version":"74'8","reported_seq":82,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-11T21:26:53.592775+0000","last_change":"2024-03-11T21:25:42.583567+0000","last_active":"2024-03-11T21:26:53.592775+0000","last_peered":"2024-03-11T21:26:53.592775+0000","last_clean":"2024-03-11T21:26:53.592775+0000","last_became_active":"2024-03-11T21:25:42.583266+0000","last_became_peered":"2024-03-11T21:25:42.583266+0000","last_unstale":"2024-03-11T21:26:53.592775+0000","last_undegraded":"2024-03-11T21:26:53.592775+0000","last_fullsized":"2024-03-11T21:26:53.592775+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:25:41.544550+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:41.544550+0000","last_clean_scrub_stamp":"2024-03-11T21:25:41.544550+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,2],"acting":[3,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.13","version":"0'0","reported_seq":65,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-11T21:26:43.123835+0000","last_change":"2024-03-11T21:25:38.539722+0000","last_active":"2024-03-11T21:26:43.123835+0000","last_peered":"2024-03-11T21:26:43.123835+0000","last_clean":"2024-03-11T21:26:43.123835+0000","last_became_active":"2024-03-11T21:25:38.539467+0000","last_became_peered":"2024-03-11T21:25:38.539467+0000","last_unstale":"2024-03-11T21:26:43.123835+0000","last_undegraded":"2024-03-11T21:26:43.123835+0000","last_fullsized":"2024-03-11T21:26:43.123835+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:25:37.512018+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:37.512018+0000","last_clean_scrub_stamp":"2024-03-11T21:25:37.512018+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:26:56.667567+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.15","version":"94'13","reported_seq":89,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-11T21:26:59.013895+0000","last_change":"2024-03-11T21:26:59.013939+0000","last_active":"2024-03-11T21:26:59.013895+0000","last_peered":"2024-03-11T21:26:59.013895+0000","last_clean":"2024-03-11T21:26:59.013895+0000","last_became_active":"2024-03-11T21:25:40.760907+0000","last_became_peered":"2024-03-11T21:25:40.760907+0000","last_unstale":"2024-03-11T21:26:59.013895+0000","last_undegraded":"2024-03-11T21:26:59.013895+0000","last_fullsized":"2024-03-11T21:26:59.013895+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"94'13","last_scrub_stamp":"2024-03-11T21:26:59.013823+0000","last_deep_scrub":"94'13","last_deep_scrub_stamp":"2024-03-11T21:26:59.013823+0000","last_clean_scrub_stamp":"2024-03-11T21:26:59.013823+0000","objects_scrubbed":4,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:28:25.819110+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.16","version":"0'0","reported_seq":61,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-11T21:26:57.367213+0000","last_change":"2024-03-11T21:26:57.367233+0000","last_active":"2024-03-11T21:26:57.367213+0000","last_peered":"2024-03-11T21:26:57.367213+0000","last_clean":"2024-03-11T21:26:57.367213+0000","last_became_active":"2024-03-11T21:25:44.583393+0000","last_became_peered":"2024-03-11T21:25:44.583393+0000","last_unstale":"2024-03-11T21:26:57.367213+0000","last_undegraded":"2024-03-11T21:26:57.367213+0000","last_fullsized":"2024-03-11T21:26:57.367213+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:26:57.367155+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:43.559452+0000","last_clean_scrub_stamp":"2024-03-11T21:26:57.367155+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:28:24.056062+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.15","version":"74'8","reported_seq":82,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-11T21:26:53.592658+0000","last_change":"2024-03-11T21:25:42.572973+0000","last_active":"2024-03-11T21:26:53.592658+0000","last_peered":"2024-03-11T21:26:53.592658+0000","last_clean":"2024-03-11T21:26:53.592658+0000","last_became_active":"2024-03-11T21:25:42.572555+0000","last_became_peered":"2024-03-11T21:25:42.572555+0000","last_unstale":"2024-03-11T21:26:53.592658+0000","last_undegraded":"2024-03-11T21:26:53.592658+0000","last_fullsized":"2024-03-11T21:26:53.592658+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:25:41.544550+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:41.544550+0000","last_clean_scrub_stamp":"2024-03-11T21:25:41.544550+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:26:58.902319+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.12","version":"0'0","reported_seq":73,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-11T21:26:47.960329+0000","last_change":"2024-03-11T21:26:47.960353+0000","last_active":"2024-03-11T21:26:47.960329+0000","last_peered":"2024-03-11T21:26:47.960329+0000","last_clean":"2024-03-11T21:26:47.960329+0000","last_became_active":"2024-03-11T21:25:38.543962+0000","last_became_peered":"2024-03-11T21:25:38.543962+0000","last_unstale":"2024-03-11T21:26:47.960329+0000","last_undegraded":"2024-03-11T21:26:47.960329+0000","last_fullsized":"2024-03-11T21:26:47.960329+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:26:47.960265+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:37.512018+0000","last_clean_scrub_stamp":"2024-03-11T21:26:47.960265+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:28:15.602449+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.14","version":"74'10","reported_seq":76,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-11T21:26:43.661342+0000","last_change":"2024-03-11T21:25:40.760925+0000","last_active":"2024-03-11T21:26:43.661342+0000","last_peered":"2024-03-11T21:26:43.661342+0000","last_clean":"2024-03-11T21:26:43.661342+0000","last_became_active":"2024-03-11T21:25:40.760620+0000","last_became_peered":"2024-03-11T21:25:40.760620+0000","last_unstale":"2024-03-11T21:26:43.661342+0000","last_undegraded":"2024-03-11T21:26:43.661342+0000","last_fullsized":"2024-03-11T21:26:43.661342+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:25:39.525554+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:39.525554+0000","last_clean_scrub_stamp":"2024-03-11T21:25:39.525554+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:26:56.858432+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.15","version":"0'0","reported_seq":61,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-11T21:27:00.378078+0000","last_change":"2024-03-11T21:27:00.378136+0000","last_active":"2024-03-11T21:27:00.378078+0000","last_peered":"2024-03-11T21:27:00.378078+0000","last_clean":"2024-03-11T21:27:00.378078+0000","last_became_active":"2024-03-11T21:25:45.302501+0000","last_became_peered":"2024-03-11T21:25:45.302501+0000","last_unstale":"2024-03-11T21:27:00.378078+0000","last_undegraded":"2024-03-11T21:27:00.378078+0000","last_fullsized":"2024-03-11T21:27:00.378078+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:27:00.377987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:43.559452+0000","last_clean_scrub_stamp":"2024-03-11T21:27:00.377987+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:28:21.911670+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":57,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-11T21:26:44.104781+0000","last_change":"2024-03-11T21:25:42.579268+0000","last_active":"2024-03-11T21:26:44.104781+0000","last_peered":"2024-03-11T21:26:44.104781+0000","last_clean":"2024-03-11T21:26:44.104781+0000","last_became_active":"2024-03-11T21:25:42.577351+0000","last_became_peered":"2024-03-11T21:25:42.577351+0000","last_unstale":"2024-03-11T21:26:44.104781+0000","last_undegraded":"2024-03-11T21:26:44.104781+0000","last_fullsized":"2024-03-11T21:26:44.104781+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:25:41.544550+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:41.544550+0000","last_clean_scrub_stamp":"2024-03-11T21:25:41.544550+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:26:56.437117+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,1],"acting":[5,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.11","version":"0'0","reported_seq":65,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-11T21:26:44.037422+0000","last_change":"2024-03-11T21:25:38.538379+0000","last_active":"2024-03-11T21:26:44.037422+0000","last_peered":"2024-03-11T21:26:44.037422+0000","last_clean":"2024-03-11T21:26:44.037422+0000","last_became_active":"2024-03-11T21:25:38.537415+0000","last_became_peered":"2024-03-11T21:25:38.537415+0000","last_unstale":"2024-03-11T21:26:44.037422+0000","last_undegraded":"2024-03-11T21:26:44.037422+0000","last_fullsized":"2024-03-11T21:26:44.037422+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:25:37.512018+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:37.512018+0000","last_clean_scrub_stamp":"2024-03-11T21:25:37.512018+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:27:06.885921+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.17","version":"74'6","reported_seq":84,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-11T21:26:43.661607+0000","last_change":"2024-03-11T21:26:11.224229+0000","last_active":"2024-03-11T21:26:43.661607+0000","last_peered":"2024-03-11T21:26:43.661607+0000","last_clean":"2024-03-11T21:26:43.661607+0000","last_became_active":"2024-03-11T21:26:11.219020+0000","last_became_peered":"2024-03-11T21:26:11.219020+0000","last_unstale":"2024-03-11T21:26:43.661607+0000","last_undegraded":"2024-03-11T21:26:43.661607+0000","last_fullsized":"2024-03-11T21:26:43.661607+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:25:39.525554+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:39.525554+0000","last_clean_scrub_stamp":"2024-03-11T21:25:39.525554+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:26:59.858426+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":9,"num_read_kb":6,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,5],"acting":[3,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.14","version":"0'0","reported_seq":53,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-11T21:26:43.291057+0000","last_change":"2024-03-11T21:25:44.587755+0000","last_active":"2024-03-11T21:26:43.291057+0000","last_peered":"2024-03-11T21:26:43.291057+0000","last_clean":"2024-03-11T21:26:43.291057+0000","last_became_active":"2024-03-11T21:25:44.587439+0000","last_became_peered":"2024-03-11T21:25:44.587439+0000","last_unstale":"2024-03-11T21:26:43.291057+0000","last_undegraded":"2024-03-11T21:26:43.291057+0000","last_fullsized":"2024-03-11T21:26:43.291057+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:25:43.559452+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:43.559452+0000","last_clean_scrub_stamp":"2024-03-11T21:25:43.559452+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:27:03.851108+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,7],"acting":[2,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":57,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-11T21:26:43.661670+0000","last_change":"2024-03-11T21:25:42.583802+0000","last_active":"2024-03-11T21:26:43.661670+0000","last_peered":"2024-03-11T21:26:43.661670+0000","last_clean":"2024-03-11T21:26:43.661670+0000","last_became_active":"2024-03-11T21:25:42.583514+0000","last_became_peered":"2024-03-11T21:25:42.583514+0000","last_unstale":"2024-03-11T21:26:43.661670+0000","last_undegraded":"2024-03-11T21:26:43.661670+0000","last_fullsized":"2024-03-11T21:26:43.661670+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:25:41.544550+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:41.544550+0000","last_clean_scrub_stamp":"2024-03-11T21:25:41.544550+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:26:54.765803+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.10","version":"0'0","reported_seq":65,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-11T21:26:43.290854+0000","last_change":"2024-03-11T21:25:38.533523+0000","last_active":"2024-03-11T21:26:43.290854+0000","last_peered":"2024-03-11T21:26:43.290854+0000","last_clean":"2024-03-11T21:26:43.290854+0000","last_became_active":"2024-03-11T21:25:38.533138+0000","last_became_peered":"2024-03-11T21:25:38.533138+0000","last_unstale":"2024-03-11T21:26:43.290854+0000","last_undegraded":"2024-03-11T21:26:43.290854+0000","last_fullsized":"2024-03-11T21:26:43.290854+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:25:37.512018+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:37.512018+0000","last_clean_scrub_stamp":"2024-03-11T21:25:37.512018+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:27:02.027595+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,0],"acting":[2,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.16","version":"74'9","reported_seq":85,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-11T21:26:53.490036+0000","last_change":"2024-03-11T21:26:53.490093+0000","last_active":"2024-03-11T21:26:53.490036+0000","last_peered":"2024-03-11T21:26:53.490036+0000","last_clean":"2024-03-11T21:26:53.490036+0000","last_became_active":"2024-03-11T21:25:40.763434+0000","last_became_peered":"2024-03-11T21:25:40.763434+0000","last_unstale":"2024-03-11T21:26:53.490036+0000","last_undegraded":"2024-03-11T21:26:53.490036+0000","last_fullsized":"2024-03-11T21:26:53.490036+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'9","last_scrub_stamp":"2024-03-11T21:26:53.489953+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:39.525554+0000","last_clean_scrub_stamp":"2024-03-11T21:26:53.489953+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:27:55.665353+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.b","version":"0'0","reported_seq":53,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-11T21:26:43.661344+0000","last_change":"2024-03-11T21:25:44.590403+0000","last_active":"2024-03-11T21:26:43.661344+0000","last_peered":"2024-03-11T21:26:43.661344+0000","last_clean":"2024-03-11T21:26:43.661344+0000","last_became_active":"2024-03-11T21:25:44.590071+0000","last_became_peered":"2024-03-11T21:25:44.590071+0000","last_unstale":"2024-03-11T21:26:43.661344+0000","last_undegraded":"2024-03-11T21:26:43.661344+0000","last_fullsized":"2024-03-11T21:26:43.661344+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:25:43.559452+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:43.559452+0000","last_clean_scrub_stamp":"2024-03-11T21:25:43.559452+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:27:05.309857+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.f","version":"67'2","reported_seq":77,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-11T21:26:43.124372+0000","last_change":"2024-03-11T21:25:38.537988+0000","last_active":"2024-03-11T21:26:43.124372+0000","last_peered":"2024-03-11T21:26:43.124372+0000","last_clean":"2024-03-11T21:26:43.124372+0000","last_became_active":"2024-03-11T21:25:38.537711+0000","last_became_peered":"2024-03-11T21:25:38.537711+0000","last_unstale":"2024-03-11T21:26:43.124372+0000","last_undegraded":"2024-03-11T21:26:43.124372+0000","last_fullsized":"2024-03-11T21:26:43.124372+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:25:37.512018+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:37.512018+0000","last_clean_scrub_stamp":"2024-03-11T21:25:37.512018+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:26:57.338014+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":92,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":10,"num_read_kb":10,"num_write":4,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,7],"acting":[4,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.9","version":"74'12","reported_seq":84,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-11T21:26:43.123650+0000","last_change":"2024-03-11T21:25:40.552685+0000","last_active":"2024-03-11T21:26:43.123650+0000","last_peered":"2024-03-11T21:26:43.123650+0000","last_clean":"2024-03-11T21:26:43.123650+0000","last_became_active":"2024-03-11T21:25:40.552267+0000","last_became_peered":"2024-03-11T21:25:40.552267+0000","last_unstale":"2024-03-11T21:26:43.123650+0000","last_undegraded":"2024-03-11T21:26:43.123650+0000","last_fullsized":"2024-03-11T21:26:43.123650+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:25:39.525554+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:39.525554+0000","last_clean_scrub_stamp":"2024-03-11T21:25:39.525554+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:27:00.249786+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":25,"num_read_kb":16,"num_write":14,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,3],"acting":[4,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":57,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-11T21:26:43.290838+0000","last_change":"2024-03-11T21:25:42.574312+0000","last_active":"2024-03-11T21:26:43.290838+0000","last_peered":"2024-03-11T21:26:43.290838+0000","last_clean":"2024-03-11T21:26:43.290838+0000","last_became_active":"2024-03-11T21:25:42.573987+0000","last_became_peered":"2024-03-11T21:25:42.573987+0000","last_unstale":"2024-03-11T21:26:43.290838+0000","last_undegraded":"2024-03-11T21:26:43.290838+0000","last_fullsized":"2024-03-11T21:26:43.290838+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:25:41.544550+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:41.544550+0000","last_clean_scrub_stamp":"2024-03-11T21:25:41.544550+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:27:01.129396+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.0","version":"0'0","reported_seq":53,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-11T21:26:43.122501+0000","last_change":"2024-03-11T21:25:44.585477+0000","last_active":"2024-03-11T21:26:43.122501+0000","last_peered":"2024-03-11T21:26:43.122501+0000","last_clean":"2024-03-11T21:26:43.122501+0000","last_became_active":"2024-03-11T21:25:44.584565+0000","last_became_peered":"2024-03-11T21:25:44.584565+0000","last_unstale":"2024-03-11T21:26:43.122501+0000","last_undegraded":"2024-03-11T21:26:43.122501+0000","last_fullsized":"2024-03-11T21:26:43.122501+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:25:43.559452+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:43.559452+0000","last_clean_scrub_stamp":"2024-03-11T21:25:43.559452+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:27:04.318172+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":65,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-11T21:26:43.123446+0000","last_change":"2024-03-11T21:25:38.542056+0000","last_active":"2024-03-11T21:26:43.123446+0000","last_peered":"2024-03-11T21:26:43.123446+0000","last_clean":"2024-03-11T21:26:43.123446+0000","last_became_active":"2024-03-11T21:25:38.541760+0000","last_became_peered":"2024-03-11T21:25:38.541760+0000","last_unstale":"2024-03-11T21:26:43.123446+0000","last_undegraded":"2024-03-11T21:26:43.123446+0000","last_fullsized":"2024-03-11T21:26:43.123446+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:25:37.512018+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:37.512018+0000","last_clean_scrub_stamp":"2024-03-11T21:25:37.512018+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:26:51.218550+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,7],"acting":[1,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.2","version":"94'15","reported_seq":89,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-11T21:26:49.993148+0000","last_change":"2024-03-11T21:26:49.993206+0000","last_active":"2024-03-11T21:26:49.993148+0000","last_peered":"2024-03-11T21:26:49.993148+0000","last_clean":"2024-03-11T21:26:49.993148+0000","last_became_active":"2024-03-11T21:25:40.548833+0000","last_became_peered":"2024-03-11T21:25:40.548833+0000","last_unstale":"2024-03-11T21:26:49.993148+0000","last_undegraded":"2024-03-11T21:26:49.993148+0000","last_fullsized":"2024-03-11T21:26:49.993148+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"94'15","last_scrub_stamp":"2024-03-11T21:26:49.993060+0000","last_deep_scrub":"94'15","last_deep_scrub_stamp":"2024-03-11T21:26:49.993060+0000","last_clean_scrub_stamp":"2024-03-11T21:26:49.993060+0000","objects_scrubbed":5,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:27:59.041594+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.3","version":"74'8","reported_seq":82,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-11T21:26:53.592487+0000","last_change":"2024-03-11T21:25:42.578555+0000","last_active":"2024-03-11T21:26:53.592487+0000","last_peered":"2024-03-11T21:26:53.592487+0000","last_clean":"2024-03-11T21:26:53.592487+0000","last_became_active":"2024-03-11T21:25:42.578161+0000","last_became_peered":"2024-03-11T21:25:42.578161+0000","last_unstale":"2024-03-11T21:26:53.592487+0000","last_undegraded":"2024-03-11T21:26:53.592487+0000","last_fullsized":"2024-03-11T21:26:53.592487+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:25:41.544550+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:41.544550+0000","last_clean_scrub_stamp":"2024-03-11T21:25:41.544550+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,5],"acting":[0,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":53,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-11T21:26:43.661632+0000","last_change":"2024-03-11T21:25:44.585641+0000","last_active":"2024-03-11T21:26:43.661632+0000","last_peered":"2024-03-11T21:26:43.661632+0000","last_clean":"2024-03-11T21:26:43.661632+0000","last_became_active":"2024-03-11T21:25:44.585223+0000","last_became_peered":"2024-03-11T21:25:44.585223+0000","last_unstale":"2024-03-11T21:26:43.661632+0000","last_undegraded":"2024-03-11T21:26:43.661632+0000","last_fullsized":"2024-03-11T21:26:43.661632+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:25:43.559452+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:43.559452+0000","last_clean_scrub_stamp":"2024-03-11T21:25:43.559452+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:26:54.535741+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,7],"acting":[3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":73,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-11T21:26:45.072135+0000","last_change":"2024-03-11T21:26:45.072161+0000","last_active":"2024-03-11T21:26:45.072135+0000","last_peered":"2024-03-11T21:26:45.072135+0000","last_clean":"2024-03-11T21:26:45.072135+0000","last_became_active":"2024-03-11T21:25:38.538540+0000","last_became_peered":"2024-03-11T21:25:38.538540+0000","last_unstale":"2024-03-11T21:26:45.072135+0000","last_undegraded":"2024-03-11T21:26:45.072135+0000","last_fullsized":"2024-03-11T21:26:45.072135+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:26:45.072058+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:37.512018+0000","last_clean_scrub_stamp":"2024-03-11T21:26:45.072058+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:27:49.711890+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.4","version":"74'28","reported_seq":126,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-11T21:26:53.592475+0000","last_change":"2024-03-11T21:25:40.553218+0000","last_active":"2024-03-11T21:26:53.592475+0000","last_peered":"2024-03-11T21:26:53.592475+0000","last_clean":"2024-03-11T21:26:53.592475+0000","last_became_active":"2024-03-11T21:25:40.552902+0000","last_became_peered":"2024-03-11T21:25:40.552902+0000","last_unstale":"2024-03-11T21:26:53.592475+0000","last_undegraded":"2024-03-11T21:26:53.592475+0000","last_fullsized":"2024-03-11T21:26:53.592475+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:25:39.525554+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:39.525554+0000","last_clean_scrub_stamp":"2024-03-11T21:25:39.525554+0000","objects_scrubbed":0,"log_size":28,"log_dups_size":0,"ondisk_log_size":28,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":358,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":48,"num_read_kb":33,"num_write":26,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3],"acting":[1,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":57,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-11T21:26:43.122939+0000","last_change":"2024-03-11T21:25:42.573353+0000","last_active":"2024-03-11T21:26:43.122939+0000","last_peered":"2024-03-11T21:26:43.122939+0000","last_clean":"2024-03-11T21:26:43.122939+0000","last_became_active":"2024-03-11T21:25:42.572673+0000","last_became_peered":"2024-03-11T21:25:42.572673+0000","last_unstale":"2024-03-11T21:26:43.122939+0000","last_undegraded":"2024-03-11T21:26:43.122939+0000","last_fullsized":"2024-03-11T21:26:43.122939+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:25:41.544550+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:41.544550+0000","last_clean_scrub_stamp":"2024-03-11T21:25:41.544550+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:27:02.832892+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.5","version":"0'0","reported_seq":53,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-11T21:26:45.417895+0000","last_change":"2024-03-11T21:25:45.302725+0000","last_active":"2024-03-11T21:26:45.417895+0000","last_peered":"2024-03-11T21:26:45.417895+0000","last_clean":"2024-03-11T21:26:45.417895+0000","last_became_active":"2024-03-11T21:25:45.302120+0000","last_became_peered":"2024-03-11T21:25:45.302120+0000","last_unstale":"2024-03-11T21:26:45.417895+0000","last_undegraded":"2024-03-11T21:26:45.417895+0000","last_fullsized":"2024-03-11T21:26:45.417895+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:25:43.559452+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:43.559452+0000","last_clean_scrub_stamp":"2024-03-11T21:25:43.559452+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:27:03.884398+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,3],"acting":[7,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.0","version":"76'5","reported_seq":161,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-11T21:27:00.593318+0000","last_change":"2024-03-11T21:26:54.088445+0000","last_active":"2024-03-11T21:27:00.593318+0000","last_peered":"2024-03-11T21:27:00.593318+0000","last_clean":"2024-03-11T21:27:00.593318+0000","last_became_active":"2024-03-11T21:25:38.536890+0000","last_became_peered":"2024-03-11T21:25:38.536890+0000","last_unstale":"2024-03-11T21:27:00.593318+0000","last_undegraded":"2024-03-11T21:27:00.593318+0000","last_fullsized":"2024-03-11T21:27:00.593318+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"76'5","last_scrub_stamp":"2024-03-11T21:26:54.088356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:37.512018+0000","last_clean_scrub_stamp":"2024-03-11T21:26:54.088356+0000","objects_scrubbed":1,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:28:21.305217+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.000753559,"stat_sum":{"num_bytes":389,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":78,"num_read_kb":73,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,6],"acting":[1,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":73,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-11T21:26:48.997525+0000","last_change":"2024-03-11T21:26:48.997561+0000","last_active":"2024-03-11T21:26:48.997525+0000","last_peered":"2024-03-11T21:26:48.997525+0000","last_clean":"2024-03-11T21:26:48.997525+0000","last_became_active":"2024-03-11T21:25:38.532984+0000","last_became_peered":"2024-03-11T21:25:38.532984+0000","last_unstale":"2024-03-11T21:26:48.997525+0000","last_undegraded":"2024-03-11T21:26:48.997525+0000","last_fullsized":"2024-03-11T21:26:48.997525+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:26:48.997454+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:26:48.997454+0000","last_clean_scrub_stamp":"2024-03-11T21:26:48.997454+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:28:14.373168+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.7","version":"74'13","reported_seq":88,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-11T21:26:43.123321+0000","last_change":"2024-03-11T21:25:40.542117+0000","last_active":"2024-03-11T21:26:43.123321+0000","last_peered":"2024-03-11T21:26:43.123321+0000","last_clean":"2024-03-11T21:26:43.123321+0000","last_became_active":"2024-03-11T21:25:40.541769+0000","last_became_peered":"2024-03-11T21:25:40.541769+0000","last_unstale":"2024-03-11T21:26:43.123321+0000","last_undegraded":"2024-03-11T21:26:43.123321+0000","last_fullsized":"2024-03-11T21:26:43.123321+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:25:39.525554+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:39.525554+0000","last_clean_scrub_stamp":"2024-03-11T21:25:39.525554+0000","objects_scrubbed":0,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:27:07.035253+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":19,"num_write":16,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":57,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-11T21:26:43.291237+0000","last_change":"2024-03-11T21:25:42.582034+0000","last_active":"2024-03-11T21:26:43.291237+0000","last_peered":"2024-03-11T21:26:43.291237+0000","last_clean":"2024-03-11T21:26:43.291237+0000","last_became_active":"2024-03-11T21:25:42.581710+0000","last_became_peered":"2024-03-11T21:25:42.581710+0000","last_unstale":"2024-03-11T21:26:43.291237+0000","last_undegraded":"2024-03-11T21:26:43.291237+0000","last_fullsized":"2024-03-11T21:26:43.291237+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:25:41.544550+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:41.544550+0000","last_clean_scrub_stamp":"2024-03-11T21:25:41.544550+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:26:46.362146+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7],"acting":[2,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.7","version":"74'1","reported_seq":54,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-11T21:26:44.105021+0000","last_change":"2024-03-11T21:25:44.591645+0000","last_active":"2024-03-11T21:26:44.105021+0000","last_peered":"2024-03-11T21:26:44.105021+0000","last_clean":"2024-03-11T21:26:44.105021+0000","last_became_active":"2024-03-11T21:25:44.591376+0000","last_became_peered":"2024-03-11T21:25:44.591376+0000","last_unstale":"2024-03-11T21:26:44.105021+0000","last_undegraded":"2024-03-11T21:26:44.105021+0000","last_fullsized":"2024-03-11T21:26:44.105021+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:25:43.559452+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:43.559452+0000","last_clean_scrub_stamp":"2024-03-11T21:25:43.559452+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:27:09.252622+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,4],"acting":[5,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2","version":"71'2","reported_seq":82,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-11T21:26:43.660984+0000","last_change":"2024-03-11T21:26:42.663119+0000","last_active":"2024-03-11T21:26:43.660984+0000","last_peered":"2024-03-11T21:26:43.660984+0000","last_clean":"2024-03-11T21:26:43.660984+0000","last_became_active":"2024-03-11T21:25:38.539413+0000","last_became_peered":"2024-03-11T21:25:38.539413+0000","last_unstale":"2024-03-11T21:26:43.660984+0000","last_undegraded":"2024-03-11T21:26:43.660984+0000","last_fullsized":"2024-03-11T21:26:43.660984+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"71'2","last_scrub_stamp":"2024-03-11T21:26:42.662979+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:37.512018+0000","last_clean_scrub_stamp":"2024-03-11T21:26:42.662979+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:27:54.085660+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.0014206449999999999,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":79,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-11T21:26:44.103841+0000","last_change":"2024-03-11T21:26:42.171361+0000","last_active":"2024-03-11T21:26:44.103841+0000","last_peered":"2024-03-11T21:26:44.103841+0000","last_clean":"2024-03-11T21:26:44.103841+0000","last_became_active":"2024-03-11T21:26:11.216188+0000","last_became_peered":"2024-03-11T21:26:11.216188+0000","last_unstale":"2024-03-11T21:26:44.103841+0000","last_undegraded":"2024-03-11T21:26:44.103841+0000","last_fullsized":"2024-03-11T21:26:44.103841+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:26:42.171256+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:37.512018+0000","last_clean_scrub_stamp":"2024-03-11T21:26:42.171256+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:27:58.632937+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,4],"acting":[5,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"87'128","reported_seq":318,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-11T21:26:45.417265+0000","last_change":"2024-03-11T21:26:11.340374+0000","last_active":"2024-03-11T21:26:45.417265+0000","last_peered":"2024-03-11T21:26:45.417265+0000","last_clean":"2024-03-11T21:26:45.417265+0000","last_became_active":"2024-03-11T21:25:30.502597+0000","last_became_peered":"2024-03-11T21:25:30.502597+0000","last_unstale":"2024-03-11T21:26:45.417265+0000","last_undegraded":"2024-03-11T21:26:45.417265+0000","last_fullsized":"2024-03-11T21:26:45.417265+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'117","last_scrub_stamp":"2024-03-11T21:26:11.340228+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:24.261974+0000","last_clean_scrub_stamp":"2024-03-11T21:26:11.340228+0000","objects_scrubbed":2,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:27:32.864192+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":212,"num_read_kb":306,"num_write":179,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1771104,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.5","version":"74'16","reported_seq":100,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-11T21:26:53.592663+0000","last_change":"2024-03-11T21:25:40.761955+0000","last_active":"2024-03-11T21:26:53.592663+0000","last_peered":"2024-03-11T21:26:53.592663+0000","last_clean":"2024-03-11T21:26:53.592663+0000","last_became_active":"2024-03-11T21:25:40.761457+0000","last_became_peered":"2024-03-11T21:25:40.761457+0000","last_unstale":"2024-03-11T21:26:53.592663+0000","last_undegraded":"2024-03-11T21:26:53.592663+0000","last_fullsized":"2024-03-11T21:26:53.592663+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:25:39.525554+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:39.525554+0000","last_clean_scrub_stamp":"2024-03-11T21:25:39.525554+0000","objects_scrubbed":0,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:27:06.180463+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":154,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":25,"num_read_kb":15,"num_write":13,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":65,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-11T21:26:58.350357+0000","last_change":"2024-03-11T21:26:58.350384+0000","last_active":"2024-03-11T21:26:58.350357+0000","last_peered":"2024-03-11T21:26:58.350357+0000","last_clean":"2024-03-11T21:26:58.350357+0000","last_became_active":"2024-03-11T21:25:42.586226+0000","last_became_peered":"2024-03-11T21:25:42.586226+0000","last_unstale":"2024-03-11T21:26:58.350357+0000","last_undegraded":"2024-03-11T21:26:58.350357+0000","last_fullsized":"2024-03-11T21:26:58.350357+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:26:58.350270+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:41.544550+0000","last_clean_scrub_stamp":"2024-03-11T21:26:58.350270+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:28:06.391624+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,5],"acting":[7,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.4","version":"0'0","reported_seq":61,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-11T21:27:00.119565+0000","last_change":"2024-03-11T21:27:00.119584+0000","last_active":"2024-03-11T21:27:00.119565+0000","last_peered":"2024-03-11T21:27:00.119565+0000","last_clean":"2024-03-11T21:27:00.119565+0000","last_became_active":"2024-03-11T21:25:44.594188+0000","last_became_peered":"2024-03-11T21:25:44.594188+0000","last_unstale":"2024-03-11T21:27:00.119565+0000","last_undegraded":"2024-03-11T21:27:00.119565+0000","last_fullsized":"2024-03-11T21:27:00.119565+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:27:00.119508+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:43.559452+0000","last_clean_scrub_stamp":"2024-03-11T21:27:00.119508+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:28:23.003619+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":80,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-11T21:26:43.122732+0000","last_change":"2024-03-11T21:26:38.589294+0000","last_active":"2024-03-11T21:26:43.122732+0000","last_peered":"2024-03-11T21:26:43.122732+0000","last_clean":"2024-03-11T21:26:43.122732+0000","last_became_active":"2024-03-11T21:25:38.543834+0000","last_became_peered":"2024-03-11T21:25:38.543834+0000","last_unstale":"2024-03-11T21:26:43.122732+0000","last_undegraded":"2024-03-11T21:26:43.122732+0000","last_fullsized":"2024-03-11T21:26:43.122732+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:26:38.589187+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:37.512018+0000","last_clean_scrub_stamp":"2024-03-11T21:26:38.589187+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:27:54.798633+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0.00089342499999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":73,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-11T21:26:51.473023+0000","last_change":"2024-03-11T21:26:51.473044+0000","last_active":"2024-03-11T21:26:51.473023+0000","last_peered":"2024-03-11T21:26:51.473023+0000","last_clean":"2024-03-11T21:26:51.473023+0000","last_became_active":"2024-03-11T21:25:38.541617+0000","last_became_peered":"2024-03-11T21:25:38.541617+0000","last_unstale":"2024-03-11T21:26:51.473023+0000","last_undegraded":"2024-03-11T21:26:51.473023+0000","last_fullsized":"2024-03-11T21:26:51.473023+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:26:51.472965+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:37.512018+0000","last_clean_scrub_stamp":"2024-03-11T21:26:51.472965+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:28:11.852536+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,0],"acting":[7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.6","version":"74'12","reported_seq":87,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-11T21:27:01.353417+0000","last_change":"2024-03-11T21:27:01.353450+0000","last_active":"2024-03-11T21:27:01.353417+0000","last_peered":"2024-03-11T21:27:01.353417+0000","last_clean":"2024-03-11T21:27:01.353417+0000","last_became_active":"2024-03-11T21:25:40.550388+0000","last_became_peered":"2024-03-11T21:25:40.550388+0000","last_unstale":"2024-03-11T21:27:01.353417+0000","last_undegraded":"2024-03-11T21:27:01.353417+0000","last_fullsized":"2024-03-11T21:27:01.353417+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'12","last_scrub_stamp":"2024-03-11T21:27:01.353347+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:39.525554+0000","last_clean_scrub_stamp":"2024-03-11T21:27:01.353347+0000","objects_scrubbed":6,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:28:25.846115+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":18,"num_read_kb":12,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":57,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-11T21:26:44.104985+0000","last_change":"2024-03-11T21:25:42.572970+0000","last_active":"2024-03-11T21:26:44.104985+0000","last_peered":"2024-03-11T21:26:44.104985+0000","last_clean":"2024-03-11T21:26:44.104985+0000","last_became_active":"2024-03-11T21:25:42.572324+0000","last_became_peered":"2024-03-11T21:25:42.572324+0000","last_unstale":"2024-03-11T21:26:44.104985+0000","last_undegraded":"2024-03-11T21:26:44.104985+0000","last_fullsized":"2024-03-11T21:26:44.104985+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:25:41.544550+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:41.544550+0000","last_clean_scrub_stamp":"2024-03-11T21:25:41.544550+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:26:54.642546+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":61,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-11T21:26:59.108181+0000","last_change":"2024-03-11T21:26:59.108203+0000","last_active":"2024-03-11T21:26:59.108181+0000","last_peered":"2024-03-11T21:26:59.108181+0000","last_clean":"2024-03-11T21:26:59.108181+0000","last_became_active":"2024-03-11T21:25:45.301416+0000","last_became_peered":"2024-03-11T21:25:45.301416+0000","last_unstale":"2024-03-11T21:26:59.108181+0000","last_undegraded":"2024-03-11T21:26:59.108181+0000","last_fullsized":"2024-03-11T21:26:59.108181+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:26:59.108111+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:43.559452+0000","last_clean_scrub_stamp":"2024-03-11T21:26:59.108111+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:28:03.778211+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":73,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-11T21:26:57.333742+0000","last_change":"2024-03-11T21:26:57.333768+0000","last_active":"2024-03-11T21:26:57.333742+0000","last_peered":"2024-03-11T21:26:57.333742+0000","last_clean":"2024-03-11T21:26:57.333742+0000","last_became_active":"2024-03-11T21:25:38.545148+0000","last_became_peered":"2024-03-11T21:25:38.545148+0000","last_unstale":"2024-03-11T21:26:57.333742+0000","last_undegraded":"2024-03-11T21:26:57.333742+0000","last_fullsized":"2024-03-11T21:26:57.333742+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:26:57.333669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:26:57.333669+0000","last_clean_scrub_stamp":"2024-03-11T21:26:57.333669+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:28:23.923732+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.3","version":"74'19","reported_seq":97,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-11T21:26:43.123750+0000","last_change":"2024-03-11T21:25:40.760898+0000","last_active":"2024-03-11T21:26:43.123750+0000","last_peered":"2024-03-11T21:26:43.123750+0000","last_clean":"2024-03-11T21:26:43.123750+0000","last_became_active":"2024-03-11T21:25:40.760594+0000","last_became_peered":"2024-03-11T21:25:40.760594+0000","last_unstale":"2024-03-11T21:26:43.123750+0000","last_undegraded":"2024-03-11T21:26:43.123750+0000","last_fullsized":"2024-03-11T21:26:43.123750+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:25:39.525554+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:39.525554+0000","last_clean_scrub_stamp":"2024-03-11T21:25:39.525554+0000","objects_scrubbed":0,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:27:01.689572+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":39,"num_read_kb":25,"num_write":22,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":57,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-11T21:26:44.037634+0000","last_change":"2024-03-11T21:25:42.579400+0000","last_active":"2024-03-11T21:26:44.037634+0000","last_peered":"2024-03-11T21:26:44.037634+0000","last_clean":"2024-03-11T21:26:44.037634+0000","last_became_active":"2024-03-11T21:25:42.579045+0000","last_became_peered":"2024-03-11T21:25:42.579045+0000","last_unstale":"2024-03-11T21:26:44.037634+0000","last_undegraded":"2024-03-11T21:26:44.037634+0000","last_fullsized":"2024-03-11T21:26:44.037634+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:25:41.544550+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:41.544550+0000","last_clean_scrub_stamp":"2024-03-11T21:25:41.544550+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:26:57.514032+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":61,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-11T21:26:50.836544+0000","last_change":"2024-03-11T21:26:50.836566+0000","last_active":"2024-03-11T21:26:50.836544+0000","last_peered":"2024-03-11T21:26:50.836544+0000","last_clean":"2024-03-11T21:26:50.836544+0000","last_became_active":"2024-03-11T21:25:44.583638+0000","last_became_peered":"2024-03-11T21:25:44.583638+0000","last_unstale":"2024-03-11T21:26:50.836544+0000","last_undegraded":"2024-03-11T21:26:50.836544+0000","last_fullsized":"2024-03-11T21:26:50.836544+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:26:50.836479+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:43.559452+0000","last_clean_scrub_stamp":"2024-03-11T21:26:50.836479+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:28:03.817969+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.6","version":"67'1","reported_seq":66,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-11T21:26:43.123657+0000","last_change":"2024-03-11T21:25:38.536530+0000","last_active":"2024-03-11T21:26:43.123657+0000","last_peered":"2024-03-11T21:26:43.123657+0000","last_clean":"2024-03-11T21:26:43.123657+0000","last_became_active":"2024-03-11T21:25:38.536142+0000","last_became_peered":"2024-03-11T21:25:38.536142+0000","last_unstale":"2024-03-11T21:26:43.123657+0000","last_undegraded":"2024-03-11T21:26:43.123657+0000","last_fullsized":"2024-03-11T21:26:43.123657+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:25:37.512018+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:37.512018+0000","last_clean_scrub_stamp":"2024-03-11T21:25:37.512018+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:26:57.512473+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,4],"acting":[1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.0","version":"93'26","reported_seq":109,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-11T21:26:43.660907+0000","last_change":"2024-03-11T21:26:40.680446+0000","last_active":"2024-03-11T21:26:43.660907+0000","last_peered":"2024-03-11T21:26:43.660907+0000","last_clean":"2024-03-11T21:26:43.660907+0000","last_became_active":"2024-03-11T21:25:40.761716+0000","last_became_peered":"2024-03-11T21:25:40.761716+0000","last_unstale":"2024-03-11T21:26:43.660907+0000","last_undegraded":"2024-03-11T21:26:43.660907+0000","last_fullsized":"2024-03-11T21:26:43.660907+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"93'26","last_scrub_stamp":"2024-03-11T21:26:40.680324+0000","last_deep_scrub":"93'26","last_deep_scrub_stamp":"2024-03-11T21:26:40.680324+0000","last_clean_scrub_stamp":"2024-03-11T21:26:40.680324+0000","objects_scrubbed":8,"log_size":26,"log_dups_size":0,"ondisk_log_size":26,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:28:00.090900+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":34,"num_read_kb":22,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":65,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-11T21:26:51.863681+0000","last_change":"2024-03-11T21:26:51.863702+0000","last_active":"2024-03-11T21:26:51.863681+0000","last_peered":"2024-03-11T21:26:51.863681+0000","last_clean":"2024-03-11T21:26:51.863681+0000","last_became_active":"2024-03-11T21:25:42.583647+0000","last_became_peered":"2024-03-11T21:25:42.583647+0000","last_unstale":"2024-03-11T21:26:51.863681+0000","last_undegraded":"2024-03-11T21:26:51.863681+0000","last_fullsized":"2024-03-11T21:26:51.863681+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:26:51.863621+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:26:51.863621+0000","last_clean_scrub_stamp":"2024-03-11T21:26:51.863621+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:27:53.832869+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,7],"acting":[4,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.3","version":"0'0","reported_seq":61,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-11T21:26:54.421255+0000","last_change":"2024-03-11T21:26:54.421277+0000","last_active":"2024-03-11T21:26:54.421255+0000","last_peered":"2024-03-11T21:26:54.421255+0000","last_clean":"2024-03-11T21:26:54.421255+0000","last_became_active":"2024-03-11T21:25:45.302719+0000","last_became_peered":"2024-03-11T21:25:45.302719+0000","last_unstale":"2024-03-11T21:26:54.421255+0000","last_undegraded":"2024-03-11T21:26:54.421255+0000","last_fullsized":"2024-03-11T21:26:54.421255+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:26:54.421196+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:43.559452+0000","last_clean_scrub_stamp":"2024-03-11T21:26:54.421196+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:28:15.162446+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,2],"acting":[7,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":73,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-11T21:26:47.927473+0000","last_change":"2024-03-11T21:26:47.927498+0000","last_active":"2024-03-11T21:26:47.927473+0000","last_peered":"2024-03-11T21:26:47.927473+0000","last_clean":"2024-03-11T21:26:47.927473+0000","last_became_active":"2024-03-11T21:25:38.540013+0000","last_became_peered":"2024-03-11T21:25:38.540013+0000","last_unstale":"2024-03-11T21:26:47.927473+0000","last_undegraded":"2024-03-11T21:26:47.927473+0000","last_fullsized":"2024-03-11T21:26:47.927473+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:26:47.927394+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:37.512018+0000","last_clean_scrub_stamp":"2024-03-11T21:26:47.927394+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:28:02.205612+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,2],"acting":[6,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.1","version":"74'14","reported_seq":82,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-11T21:26:43.123471+0000","last_change":"2024-03-11T21:25:40.546609+0000","last_active":"2024-03-11T21:26:43.123471+0000","last_peered":"2024-03-11T21:26:43.123471+0000","last_clean":"2024-03-11T21:26:43.123471+0000","last_became_active":"2024-03-11T21:25:40.546219+0000","last_became_peered":"2024-03-11T21:25:40.546219+0000","last_unstale":"2024-03-11T21:26:43.123471+0000","last_undegraded":"2024-03-11T21:26:43.123471+0000","last_fullsized":"2024-03-11T21:26:43.123471+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:25:39.525554+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:39.525554+0000","last_clean_scrub_stamp":"2024-03-11T21:25:39.525554+0000","objects_scrubbed":0,"log_size":14,"log_dups_size":0,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:26:59.941670+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":21,"num_read_kb":14,"num_write":14,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6],"acting":[4,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.0","version":"74'8","reported_seq":82,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-11T21:26:53.592474+0000","last_change":"2024-03-11T21:25:42.577103+0000","last_active":"2024-03-11T21:26:53.592474+0000","last_peered":"2024-03-11T21:26:53.592474+0000","last_clean":"2024-03-11T21:26:53.592474+0000","last_became_active":"2024-03-11T21:25:42.576126+0000","last_became_peered":"2024-03-11T21:25:42.576126+0000","last_unstale":"2024-03-11T21:26:53.592474+0000","last_undegraded":"2024-03-11T21:26:53.592474+0000","last_fullsized":"2024-03-11T21:26:53.592474+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:25:41.544550+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:41.544550+0000","last_clean_scrub_stamp":"2024-03-11T21:25:41.544550+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:27:01.064639+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.c","version":"0'0","reported_seq":53,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-11T21:26:43.661225+0000","last_change":"2024-03-11T21:25:45.303235+0000","last_active":"2024-03-11T21:26:43.661225+0000","last_peered":"2024-03-11T21:26:43.661225+0000","last_clean":"2024-03-11T21:26:43.661225+0000","last_became_active":"2024-03-11T21:25:45.302207+0000","last_became_peered":"2024-03-11T21:25:45.302207+0000","last_unstale":"2024-03-11T21:26:43.661225+0000","last_undegraded":"2024-03-11T21:26:43.661225+0000","last_fullsized":"2024-03-11T21:26:43.661225+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:25:43.559452+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:43.559452+0000","last_clean_scrub_stamp":"2024-03-11T21:25:43.559452+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:27:05.058318+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.f","version":"0'0","reported_seq":65,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-11T21:26:51.925445+0000","last_change":"2024-03-11T21:26:51.925489+0000","last_active":"2024-03-11T21:26:51.925445+0000","last_peered":"2024-03-11T21:26:51.925445+0000","last_clean":"2024-03-11T21:26:51.925445+0000","last_became_active":"2024-03-11T21:25:42.575935+0000","last_became_peered":"2024-03-11T21:25:42.575935+0000","last_unstale":"2024-03-11T21:26:51.925445+0000","last_undegraded":"2024-03-11T21:26:51.925445+0000","last_fullsized":"2024-03-11T21:26:51.925445+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:26:51.925329+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:41.544550+0000","last_clean_scrub_stamp":"2024-03-11T21:26:51.925329+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:28:02.478228+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.8","version":"0'0","reported_seq":73,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-11T21:26:48.423986+0000","last_change":"2024-03-11T21:26:48.424013+0000","last_active":"2024-03-11T21:26:48.423986+0000","last_peered":"2024-03-11T21:26:48.423986+0000","last_clean":"2024-03-11T21:26:48.423986+0000","last_became_active":"2024-03-11T21:25:38.540910+0000","last_became_peered":"2024-03-11T21:25:38.540910+0000","last_unstale":"2024-03-11T21:26:48.423986+0000","last_undegraded":"2024-03-11T21:26:48.423986+0000","last_fullsized":"2024-03-11T21:26:48.423986+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:26:48.423910+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:37.512018+0000","last_clean_scrub_stamp":"2024-03-11T21:26:48.423910+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:28:17.378160+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,1],"acting":[7,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.e","version":"74'11","reported_seq":88,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-11T21:26:57.843702+0000","last_change":"2024-03-11T21:26:57.843770+0000","last_active":"2024-03-11T21:26:57.843702+0000","last_peered":"2024-03-11T21:26:57.843702+0000","last_clean":"2024-03-11T21:26:57.843702+0000","last_became_active":"2024-03-11T21:25:40.552178+0000","last_became_peered":"2024-03-11T21:25:40.552178+0000","last_unstale":"2024-03-11T21:26:57.843702+0000","last_undegraded":"2024-03-11T21:26:57.843702+0000","last_fullsized":"2024-03-11T21:26:57.843702+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'11","last_scrub_stamp":"2024-03-11T21:26:57.843615+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:39.525554+0000","last_clean_scrub_stamp":"2024-03-11T21:26:57.843615+0000","objects_scrubbed":5,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:28:03.562699+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.d","version":"0'0","reported_seq":61,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-11T21:26:54.957646+0000","last_change":"2024-03-11T21:26:54.957670+0000","last_active":"2024-03-11T21:26:54.957646+0000","last_peered":"2024-03-11T21:26:54.957646+0000","last_clean":"2024-03-11T21:26:54.957646+0000","last_became_active":"2024-03-11T21:25:44.590333+0000","last_became_peered":"2024-03-11T21:25:44.590333+0000","last_unstale":"2024-03-11T21:26:54.957646+0000","last_undegraded":"2024-03-11T21:26:54.957646+0000","last_fullsized":"2024-03-11T21:26:54.957646+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:26:54.957577+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:43.559452+0000","last_clean_scrub_stamp":"2024-03-11T21:26:54.957577+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:28:05.595894+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.e","version":"74'8","reported_seq":82,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-11T21:26:53.592655+0000","last_change":"2024-03-11T21:25:42.577257+0000","last_active":"2024-03-11T21:26:53.592655+0000","last_peered":"2024-03-11T21:26:53.592655+0000","last_clean":"2024-03-11T21:26:53.592655+0000","last_became_active":"2024-03-11T21:25:42.576742+0000","last_became_peered":"2024-03-11T21:25:42.576742+0000","last_unstale":"2024-03-11T21:26:53.592655+0000","last_undegraded":"2024-03-11T21:26:53.592655+0000","last_fullsized":"2024-03-11T21:26:53.592655+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:25:41.544550+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:41.544550+0000","last_clean_scrub_stamp":"2024-03-11T21:25:41.544550+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,0],"acting":[4,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.9","version":"0'0","reported_seq":73,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-11T21:26:50.991796+0000","last_change":"2024-03-11T21:26:50.991832+0000","last_active":"2024-03-11T21:26:50.991796+0000","last_peered":"2024-03-11T21:26:50.991796+0000","last_clean":"2024-03-11T21:26:50.991796+0000","last_became_active":"2024-03-11T21:25:38.540976+0000","last_became_peered":"2024-03-11T21:25:38.540976+0000","last_unstale":"2024-03-11T21:26:50.991796+0000","last_undegraded":"2024-03-11T21:26:50.991796+0000","last_fullsized":"2024-03-11T21:26:50.991796+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:26:50.991712+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:26:50.991712+0000","last_clean_scrub_stamp":"2024-03-11T21:26:50.991712+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:27:58.101352+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,3],"acting":[1,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.f","version":"74'15","reported_seq":47,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-11T21:26:43.290730+0000","last_change":"2024-03-11T21:26:11.226642+0000","last_active":"2024-03-11T21:26:43.290730+0000","last_peered":"2024-03-11T21:26:43.290730+0000","last_clean":"2024-03-11T21:26:43.290730+0000","last_became_active":"2024-03-11T21:26:11.216224+0000","last_became_peered":"2024-03-11T21:26:11.216224+0000","last_unstale":"2024-03-11T21:26:43.290730+0000","last_undegraded":"2024-03-11T21:26:43.290730+0000","last_fullsized":"2024-03-11T21:26:43.290730+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:25:39.525554+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:39.525554+0000","last_clean_scrub_stamp":"2024-03-11T21:25:39.525554+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:27:08.139224+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,4],"acting":[2,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.e","version":"0'0","reported_seq":53,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-11T21:26:43.123979+0000","last_change":"2024-03-11T21:25:44.590476+0000","last_active":"2024-03-11T21:26:43.123979+0000","last_peered":"2024-03-11T21:26:43.123979+0000","last_clean":"2024-03-11T21:26:43.123979+0000","last_became_active":"2024-03-11T21:25:44.590198+0000","last_became_peered":"2024-03-11T21:25:44.590198+0000","last_unstale":"2024-03-11T21:26:43.123979+0000","last_undegraded":"2024-03-11T21:26:43.123979+0000","last_fullsized":"2024-03-11T21:26:43.123979+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:25:43.559452+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:43.559452+0000","last_clean_scrub_stamp":"2024-03-11T21:25:43.559452+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:27:00.005970+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.d","version":"74'8","reported_seq":90,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-11T21:26:59.021578+0000","last_change":"2024-03-11T21:26:59.021603+0000","last_active":"2024-03-11T21:26:59.021578+0000","last_peered":"2024-03-11T21:26:59.021578+0000","last_clean":"2024-03-11T21:26:59.021578+0000","last_became_active":"2024-03-11T21:25:42.575583+0000","last_became_peered":"2024-03-11T21:25:42.575583+0000","last_unstale":"2024-03-11T21:26:59.021578+0000","last_undegraded":"2024-03-11T21:26:59.021578+0000","last_fullsized":"2024-03-11T21:26:59.021578+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'8","last_scrub_stamp":"2024-03-11T21:26:59.021523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:41.544550+0000","last_clean_scrub_stamp":"2024-03-11T21:26:59.021523+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:28:21.121669+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,5],"acting":[2,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.a","version":"0'0","reported_seq":73,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-11T21:26:52.008073+0000","last_change":"2024-03-11T21:26:52.008094+0000","last_active":"2024-03-11T21:26:52.008073+0000","last_peered":"2024-03-11T21:26:52.008073+0000","last_clean":"2024-03-11T21:26:52.008073+0000","last_became_active":"2024-03-11T21:25:38.541045+0000","last_became_peered":"2024-03-11T21:25:38.541045+0000","last_unstale":"2024-03-11T21:26:52.008073+0000","last_undegraded":"2024-03-11T21:26:52.008073+0000","last_fullsized":"2024-03-11T21:26:52.008073+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:26:52.007996+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:37.512018+0000","last_clean_scrub_stamp":"2024-03-11T21:26:52.007996+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:27:52.862240+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,7],"acting":[1,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.c","version":"74'10","reported_seq":84,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-11T21:27:00.807919+0000","last_change":"2024-03-11T21:27:00.807955+0000","last_active":"2024-03-11T21:27:00.807919+0000","last_peered":"2024-03-11T21:27:00.807919+0000","last_clean":"2024-03-11T21:27:00.807919+0000","last_became_active":"2024-03-11T21:25:40.550885+0000","last_became_peered":"2024-03-11T21:25:40.550885+0000","last_unstale":"2024-03-11T21:27:00.807919+0000","last_undegraded":"2024-03-11T21:27:00.807919+0000","last_fullsized":"2024-03-11T21:27:00.807919+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'10","last_scrub_stamp":"2024-03-11T21:27:00.807849+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:39.525554+0000","last_clean_scrub_stamp":"2024-03-11T21:27:00.807849+0000","objects_scrubbed":5,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:28:27.818066+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.f","version":"0'0","reported_seq":53,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-11T21:26:43.291327+0000","last_change":"2024-03-11T21:25:44.593783+0000","last_active":"2024-03-11T21:26:43.291327+0000","last_peered":"2024-03-11T21:26:43.291327+0000","last_clean":"2024-03-11T21:26:43.291327+0000","last_became_active":"2024-03-11T21:25:44.593414+0000","last_became_peered":"2024-03-11T21:25:44.593414+0000","last_unstale":"2024-03-11T21:26:43.291327+0000","last_undegraded":"2024-03-11T21:26:43.291327+0000","last_fullsized":"2024-03-11T21:26:43.291327+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:25:43.559452+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:43.559452+0000","last_clean_scrub_stamp":"2024-03-11T21:25:43.559452+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:26:47.810287+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,4],"acting":[2,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":57,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-11T21:26:43.123297+0000","last_change":"2024-03-11T21:25:42.579815+0000","last_active":"2024-03-11T21:26:43.123297+0000","last_peered":"2024-03-11T21:26:43.123297+0000","last_clean":"2024-03-11T21:26:43.123297+0000","last_became_active":"2024-03-11T21:25:42.579409+0000","last_became_peered":"2024-03-11T21:25:42.579409+0000","last_unstale":"2024-03-11T21:26:43.123297+0000","last_undegraded":"2024-03-11T21:26:43.123297+0000","last_fullsized":"2024-03-11T21:26:43.123297+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:25:41.544550+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:41.544550+0000","last_clean_scrub_stamp":"2024-03-11T21:25:41.544550+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:27:11.166706+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.b","version":"0'0","reported_seq":65,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-11T21:26:45.417597+0000","last_change":"2024-03-11T21:25:38.544246+0000","last_active":"2024-03-11T21:26:45.417597+0000","last_peered":"2024-03-11T21:26:45.417597+0000","last_clean":"2024-03-11T21:26:45.417597+0000","last_became_active":"2024-03-11T21:25:38.543377+0000","last_became_peered":"2024-03-11T21:25:38.543377+0000","last_unstale":"2024-03-11T21:26:45.417597+0000","last_undegraded":"2024-03-11T21:26:45.417597+0000","last_fullsized":"2024-03-11T21:26:45.417597+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:25:37.512018+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:37.512018+0000","last_clean_scrub_stamp":"2024-03-11T21:25:37.512018+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:27:05.980046+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,5],"acting":[7,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.d","version":"74'17","reported_seq":89,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-11T21:26:43.124238+0000","last_change":"2024-03-11T21:25:40.553693+0000","last_active":"2024-03-11T21:26:43.124238+0000","last_peered":"2024-03-11T21:26:43.124238+0000","last_clean":"2024-03-11T21:26:43.124238+0000","last_became_active":"2024-03-11T21:25:40.553454+0000","last_became_peered":"2024-03-11T21:25:40.553454+0000","last_unstale":"2024-03-11T21:26:43.124238+0000","last_undegraded":"2024-03-11T21:26:43.124238+0000","last_fullsized":"2024-03-11T21:26:43.124238+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:25:39.525554+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:39.525554+0000","last_clean_scrub_stamp":"2024-03-11T21:25:39.525554+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:26:49.114384+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":29,"num_read_kb":19,"num_write":18,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,1],"acting":[4,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":53,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-11T21:26:45.417101+0000","last_change":"2024-03-11T21:25:44.590258+0000","last_active":"2024-03-11T21:26:45.417101+0000","last_peered":"2024-03-11T21:26:45.417101+0000","last_clean":"2024-03-11T21:26:45.417101+0000","last_became_active":"2024-03-11T21:25:44.589973+0000","last_became_peered":"2024-03-11T21:25:44.589973+0000","last_unstale":"2024-03-11T21:26:45.417101+0000","last_undegraded":"2024-03-11T21:26:45.417101+0000","last_fullsized":"2024-03-11T21:26:45.417101+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:25:43.559452+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:43.559452+0000","last_clean_scrub_stamp":"2024-03-11T21:25:43.559452+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:27:03.886246+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,3],"acting":[7,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":57,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-11T21:26:43.291053+0000","last_change":"2024-03-11T21:25:42.574877+0000","last_active":"2024-03-11T21:26:43.291053+0000","last_peered":"2024-03-11T21:26:43.291053+0000","last_clean":"2024-03-11T21:26:43.291053+0000","last_became_active":"2024-03-11T21:25:42.574515+0000","last_became_peered":"2024-03-11T21:25:42.574515+0000","last_unstale":"2024-03-11T21:26:43.291053+0000","last_undegraded":"2024-03-11T21:26:43.291053+0000","last_fullsized":"2024-03-11T21:26:43.291053+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:25:41.544550+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:41.544550+0000","last_clean_scrub_stamp":"2024-03-11T21:25:41.544550+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:27:09.339666+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,5],"acting":[2,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.c","version":"0'0","reported_seq":65,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-11T21:26:43.291201+0000","last_change":"2024-03-11T21:25:38.533719+0000","last_active":"2024-03-11T21:26:43.291201+0000","last_peered":"2024-03-11T21:26:43.291201+0000","last_clean":"2024-03-11T21:26:43.291201+0000","last_became_active":"2024-03-11T21:25:38.533240+0000","last_became_peered":"2024-03-11T21:25:38.533240+0000","last_unstale":"2024-03-11T21:26:43.291201+0000","last_undegraded":"2024-03-11T21:26:43.291201+0000","last_fullsized":"2024-03-11T21:26:43.291201+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:25:37.512018+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:37.512018+0000","last_clean_scrub_stamp":"2024-03-11T21:25:37.512018+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:27:06.014847+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,0],"acting":[2,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.a","version":"74'19","reported_seq":92,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-11T21:26:44.037495+0000","last_change":"2024-03-11T21:25:40.761035+0000","last_active":"2024-03-11T21:26:44.037495+0000","last_peered":"2024-03-11T21:26:44.037495+0000","last_clean":"2024-03-11T21:26:44.037495+0000","last_became_active":"2024-03-11T21:25:40.760698+0000","last_became_peered":"2024-03-11T21:25:40.760698+0000","last_unstale":"2024-03-11T21:26:44.037495+0000","last_undegraded":"2024-03-11T21:26:44.037495+0000","last_fullsized":"2024-03-11T21:26:44.037495+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:25:39.525554+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:39.525554+0000","last_clean_scrub_stamp":"2024-03-11T21:25:39.525554+0000","objects_scrubbed":0,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:26:59.237690+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":32,"num_read_kb":21,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,7],"acting":[6,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":53,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-11T21:26:43.122417+0000","last_change":"2024-03-11T21:25:44.585252+0000","last_active":"2024-03-11T21:26:43.122417+0000","last_peered":"2024-03-11T21:26:43.122417+0000","last_clean":"2024-03-11T21:26:43.122417+0000","last_became_active":"2024-03-11T21:25:44.584154+0000","last_became_peered":"2024-03-11T21:25:44.584154+0000","last_unstale":"2024-03-11T21:26:43.122417+0000","last_undegraded":"2024-03-11T21:26:43.122417+0000","last_fullsized":"2024-03-11T21:26:43.122417+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:25:43.559452+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:43.559452+0000","last_clean_scrub_stamp":"2024-03-11T21:25:43.559452+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:26:55.472231+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":57,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-11T21:26:43.290954+0000","last_change":"2024-03-11T21:25:42.577027+0000","last_active":"2024-03-11T21:26:43.290954+0000","last_peered":"2024-03-11T21:26:43.290954+0000","last_clean":"2024-03-11T21:26:43.290954+0000","last_became_active":"2024-03-11T21:25:42.576679+0000","last_became_peered":"2024-03-11T21:25:42.576679+0000","last_unstale":"2024-03-11T21:26:43.290954+0000","last_undegraded":"2024-03-11T21:26:43.290954+0000","last_fullsized":"2024-03-11T21:26:43.290954+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:25:41.544550+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:41.544550+0000","last_clean_scrub_stamp":"2024-03-11T21:25:41.544550+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:27:03.346770+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,3],"acting":[2,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.d","version":"0'0","reported_seq":73,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-11T21:26:45.885199+0000","last_change":"2024-03-11T21:26:45.885219+0000","last_active":"2024-03-11T21:26:45.885199+0000","last_peered":"2024-03-11T21:26:45.885199+0000","last_clean":"2024-03-11T21:26:45.885199+0000","last_became_active":"2024-03-11T21:25:38.539681+0000","last_became_peered":"2024-03-11T21:25:38.539681+0000","last_unstale":"2024-03-11T21:26:45.885199+0000","last_undegraded":"2024-03-11T21:26:45.885199+0000","last_fullsized":"2024-03-11T21:26:45.885199+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:26:45.885144+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:37.512018+0000","last_clean_scrub_stamp":"2024-03-11T21:26:45.885144+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:27:47.637748+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,3],"acting":[1,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.b","version":"74'9","reported_seq":77,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-11T21:26:43.123916+0000","last_change":"2024-03-11T21:25:40.549925+0000","last_active":"2024-03-11T21:26:43.123916+0000","last_peered":"2024-03-11T21:26:43.123916+0000","last_clean":"2024-03-11T21:26:43.123916+0000","last_became_active":"2024-03-11T21:25:40.549612+0000","last_became_peered":"2024-03-11T21:25:40.549612+0000","last_unstale":"2024-03-11T21:26:43.123916+0000","last_undegraded":"2024-03-11T21:26:43.123916+0000","last_fullsized":"2024-03-11T21:26:43.123916+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:25:39.525554+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:39.525554+0000","last_clean_scrub_stamp":"2024-03-11T21:25:39.525554+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:27:08.607529+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":61,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-11T21:26:55.966049+0000","last_change":"2024-03-11T21:26:55.966077+0000","last_active":"2024-03-11T21:26:55.966049+0000","last_peered":"2024-03-11T21:26:55.966049+0000","last_clean":"2024-03-11T21:26:55.966049+0000","last_became_active":"2024-03-11T21:25:45.300333+0000","last_became_peered":"2024-03-11T21:25:45.300333+0000","last_unstale":"2024-03-11T21:26:55.966049+0000","last_undegraded":"2024-03-11T21:26:55.966049+0000","last_fullsized":"2024-03-11T21:26:55.966049+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:26:55.965980+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:43.559452+0000","last_clean_scrub_stamp":"2024-03-11T21:26:55.965980+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:28:02.706121+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,0],"acting":[5,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.e","version":"0'0","reported_seq":73,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-11T21:26:57.974895+0000","last_change":"2024-03-11T21:26:57.974916+0000","last_active":"2024-03-11T21:26:57.974895+0000","last_peered":"2024-03-11T21:26:57.974895+0000","last_clean":"2024-03-11T21:26:57.974895+0000","last_became_active":"2024-03-11T21:25:38.539954+0000","last_became_peered":"2024-03-11T21:25:38.539954+0000","last_unstale":"2024-03-11T21:26:57.974895+0000","last_undegraded":"2024-03-11T21:26:57.974895+0000","last_fullsized":"2024-03-11T21:26:57.974895+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:26:57.974827+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:37.512018+0000","last_clean_scrub_stamp":"2024-03-11T21:26:57.974827+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:28:10.300007+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,7],"acting":[2,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.8","version":"74'15","reported_seq":94,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-11T21:26:50.934555+0000","last_change":"2024-03-11T21:26:50.934584+0000","last_active":"2024-03-11T21:26:50.934555+0000","last_peered":"2024-03-11T21:26:50.934555+0000","last_clean":"2024-03-11T21:26:50.934555+0000","last_became_active":"2024-03-11T21:25:40.761938+0000","last_became_peered":"2024-03-11T21:25:40.761938+0000","last_unstale":"2024-03-11T21:26:50.934555+0000","last_undegraded":"2024-03-11T21:26:50.934555+0000","last_fullsized":"2024-03-11T21:26:50.934555+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'15","last_scrub_stamp":"2024-03-11T21:26:50.934489+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:39.525554+0000","last_clean_scrub_stamp":"2024-03-11T21:26:50.934489+0000","objects_scrubbed":7,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:28:08.719408+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.9","version":"94'9","reported_seq":94,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-03-11T21:26:53.592655+0000","last_change":"2024-03-11T21:26:53.422673+0000","last_active":"2024-03-11T21:26:53.592655+0000","last_peered":"2024-03-11T21:26:53.592655+0000","last_clean":"2024-03-11T21:26:53.592655+0000","last_became_active":"2024-03-11T21:25:42.584461+0000","last_became_peered":"2024-03-11T21:25:42.584461+0000","last_unstale":"2024-03-11T21:26:53.592655+0000","last_undegraded":"2024-03-11T21:26:53.592655+0000","last_fullsized":"2024-03-11T21:26:53.592655+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"94'9","last_scrub_stamp":"2024-03-11T21:26:53.422552+0000","last_deep_scrub":"94'9","last_deep_scrub_stamp":"2024-03-11T21:26:53.422552+0000","last_clean_scrub_stamp":"2024-03-11T21:26:53.422552+0000","objects_scrubbed":1,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:28:07.835997+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":6,"num_pg":32,"stat_sum":{"num_bytes":383,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1149,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2,"ondisk_log_size":2,"up":96,"acting":96,"num_store_stats":8},{"poolid":5,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":65,"ondisk_log_size":65,"up":96,"acting":96,"num_store_stats":8},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":3702,"num_objects":178,"num_object_clones":0,"num_object_copies":534,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":178,"num_whiteouts":0,"num_read":698,"num_read_kb":455,"num_write":417,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":417792,"data_stored":11106,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":412,"ondisk_log_size":412,"up":96,"acting":96,"num_store_stats":8},{"poolid":3,"num_pg":3,"stat_sum":{"num_bytes":408,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":78,"num_read_kb":73,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1224,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":9,"acting":9,"num_store_stats":7},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":212,"num_read_kb":306,"num_write":179,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1771104,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1781760,"data_stored":1771104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":128,"ondisk_log_size":128,"up":3,"acting":3,"num_store_stats":3},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":1531,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":24,"num_read_kb":24,"num_write":10,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":4593,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":6,"ondisk_log_size":6,"up":96,"acting":96,"num_store_stats":8}],"osd_stats":[{"osd":7,"up_from":60,"seq":257698037780,"num_pgs":56,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28452,"kb_used_data":1436,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93716700,"statfs":{"total":95995035648,"available":95965900800,"internally_reserved":0,"allocated":1470464,"data_stored":977305,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":50,"seq":214748364826,"num_pgs":44,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28448,"kb_used_data":1432,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93716704,"statfs":{"total":95995035648,"available":95965904896,"internally_reserved":0,"allocated":1466368,"data_stored":977598,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574900,"num_pgs":53,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27900,"kb_used_data":884,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717252,"statfs":{"total":95995035648,"available":95966466048,"internally_reserved":0,"allocated":905216,"data_stored":387338,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738425,"num_pgs":47,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28456,"kb_used_data":1440,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93716696,"statfs":{"total":95995035648,"available":95965896704,"internally_reserved":0,"allocated":1474560,"data_stored":977522,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411374,"num_pgs":39,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27860,"kb_used_data":844,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717292,"statfs":{"total":95995035648,"available":95966507008,"internally_reserved":0,"allocated":864256,"data_stored":386562,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":26,"seq":111669149738,"num_pgs":55,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32060,"kb_used_data":884,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713092,"statfs":{"total":95995035648,"available":95962206208,"internally_reserved":0,"allocated":905216,"data_stored":387672,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":32,"seq":137438953507,"num_pgs":53,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27884,"kb_used_data":868,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717268,"statfs":{"total":95995035648,"available":95966482432,"internally_reserved":0,"allocated":888832,"data_stored":386846,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":41,"seq":176093659166,"num_pgs":49,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32024,"kb_used_data":848,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713128,"statfs":{"total":95995035648,"available":95962243072,"internally_reserved":0,"allocated":868352,"data_stored":386613,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1049,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":46,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":957,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":574,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":436,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1003,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":528,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":1320,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":81920,"data_stored":2118,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":1018,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":1898,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":1474,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":990,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":1034,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":1254,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":370,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":383,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":370,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-11T21:27:04.902 INFO:tasks.cephadm.ceph_manager.ceph:clean! 2024-03-11T21:27:04.902 INFO:tasks.ceph:Waiting until ceph cluster ceph is healthy... 2024-03-11T21:27:04.902 INFO:tasks.cephadm.ceph_manager.ceph:wait_until_healthy 2024-03-11T21:27:04.902 DEBUG:teuthology.orchestra.run.smithi092:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid 00db0016-dfed-11ee-95c8-87774f69a715 -- ceph health --format=json 2024-03-11T21:27:05.203 INFO:teuthology.orchestra.run.smithi092.stderr:Inferring config /var/lib/ceph/00db0016-dfed-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T21:27:05.324 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:27:05 smithi092 ceph-mon[37563]: 4.13 scrub starts 2024-03-11T21:27:05.324 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:27:05 smithi092 ceph-mon[37563]: 4.13 scrub ok 2024-03-11T21:27:05.324 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:27:05 smithi092 ceph-mon[37563]: 2.1a scrub starts 2024-03-11T21:27:05.324 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:27:05 smithi092 ceph-mon[37563]: 2.1a scrub ok 2024-03-11T21:27:05.325 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:27:05 smithi092 ceph-mon[37563]: 4.1c scrub starts 2024-03-11T21:27:05.325 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:27:05 smithi092 ceph-mon[37563]: 4.1c scrub ok 2024-03-11T21:27:05.325 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:27:05 smithi092 ceph-mon[37563]: 2.1c scrub starts 2024-03-11T21:27:05.325 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:27:05 smithi092 ceph-mon[37563]: 2.1c scrub ok 2024-03-11T21:27:05.325 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:27:05 smithi092 ceph-mon[37563]: 6.1f scrub starts 2024-03-11T21:27:05.325 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:27:05 smithi092 ceph-mon[37563]: 6.1f scrub ok 2024-03-11T21:27:05.325 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:27:05 smithi092 ceph-mon[29572]: 4.13 scrub starts 2024-03-11T21:27:05.325 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:27:05 smithi092 ceph-mon[29572]: 4.13 scrub ok 2024-03-11T21:27:05.325 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:27:05 smithi092 ceph-mon[29572]: 2.1a scrub starts 2024-03-11T21:27:05.325 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:27:05 smithi092 ceph-mon[29572]: 2.1a scrub ok 2024-03-11T21:27:05.325 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:27:05 smithi092 ceph-mon[29572]: 4.1c scrub starts 2024-03-11T21:27:05.326 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:27:05 smithi092 ceph-mon[29572]: 4.1c scrub ok 2024-03-11T21:27:05.326 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:27:05 smithi092 ceph-mon[29572]: 2.1c scrub starts 2024-03-11T21:27:05.326 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:27:05 smithi092 ceph-mon[29572]: 2.1c scrub ok 2024-03-11T21:27:05.326 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:27:05 smithi092 ceph-mon[29572]: 6.1f scrub starts 2024-03-11T21:27:05.326 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:27:05 smithi092 ceph-mon[29572]: 6.1f scrub ok 2024-03-11T21:27:05.577 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:27:05 smithi097 ceph-mon[33317]: 4.13 scrub starts 2024-03-11T21:27:05.577 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:27:05 smithi097 ceph-mon[33317]: 4.13 scrub ok 2024-03-11T21:27:05.577 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:27:05 smithi097 ceph-mon[33317]: 2.1a scrub starts 2024-03-11T21:27:05.577 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:27:05 smithi097 ceph-mon[33317]: 2.1a scrub ok 2024-03-11T21:27:05.577 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:27:05 smithi097 ceph-mon[33317]: 4.1c scrub starts 2024-03-11T21:27:05.577 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:27:05 smithi097 ceph-mon[33317]: 4.1c scrub ok 2024-03-11T21:27:05.577 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:27:05 smithi097 ceph-mon[33317]: 2.1c scrub starts 2024-03-11T21:27:05.577 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:27:05 smithi097 ceph-mon[33317]: 2.1c scrub ok 2024-03-11T21:27:05.577 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:27:05 smithi097 ceph-mon[33317]: 6.1f scrub starts 2024-03-11T21:27:05.578 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:27:05 smithi097 ceph-mon[33317]: 6.1f scrub ok 2024-03-11T21:27:06.310 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:27:06 smithi092 ceph-mon[29572]: pgmap v32: 132 pgs: 132 active+clean; 582 KiB data, 228 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-11T21:27:06.310 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:27:06 smithi092 ceph-mon[29572]: 5.2 scrub starts 2024-03-11T21:27:06.310 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:27:06 smithi092 ceph-mon[29572]: 5.2 scrub ok 2024-03-11T21:27:06.310 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:27:06 smithi092 ceph-mon[29572]: purged_snaps scrub starts 2024-03-11T21:27:06.310 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:27:06 smithi092 ceph-mon[29572]: purged_snaps scrub ok 2024-03-11T21:27:06.310 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:27:06 smithi092 ceph-mon[29572]: osdmap e95: 8 total, 8 up, 8 in 2024-03-11T21:27:06.310 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:27:06 smithi092 ceph-mon[29572]: 6.5 scrub starts 2024-03-11T21:27:06.311 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:27:06 smithi092 ceph-mon[29572]: 6.5 scrub ok 2024-03-11T21:27:06.311 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:27:06 smithi092 ceph-mon[29572]: 6.11 scrub starts 2024-03-11T21:27:06.311 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:27:06 smithi092 ceph-mon[29572]: 6.11 scrub ok 2024-03-11T21:27:06.311 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:27:06 smithi092 ceph-mon[29572]: from='mgr.14607 v1:172.21.15.92:0/1702717597' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T21:27:06.311 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:27:06 smithi092 ceph-mon[37563]: pgmap v32: 132 pgs: 132 active+clean; 582 KiB data, 228 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-11T21:27:06.311 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:27:06 smithi092 ceph-mon[37563]: 5.2 scrub starts 2024-03-11T21:27:06.311 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:27:06 smithi092 ceph-mon[37563]: 5.2 scrub ok 2024-03-11T21:27:06.311 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:27:06 smithi092 ceph-mon[37563]: purged_snaps scrub starts 2024-03-11T21:27:06.311 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:27:06 smithi092 ceph-mon[37563]: purged_snaps scrub ok 2024-03-11T21:27:06.311 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:27:06 smithi092 ceph-mon[37563]: osdmap e95: 8 total, 8 up, 8 in 2024-03-11T21:27:06.311 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:27:06 smithi092 ceph-mon[37563]: 6.5 scrub starts 2024-03-11T21:27:06.312 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:27:06 smithi092 ceph-mon[37563]: 6.5 scrub ok 2024-03-11T21:27:06.312 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:27:06 smithi092 ceph-mon[37563]: 6.11 scrub starts 2024-03-11T21:27:06.312 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:27:06 smithi092 ceph-mon[37563]: 6.11 scrub ok 2024-03-11T21:27:06.312 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:27:06 smithi092 ceph-mon[37563]: from='mgr.14607 v1:172.21.15.92:0/1702717597' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T21:27:06.577 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:27:06 smithi097 ceph-mon[33317]: pgmap v32: 132 pgs: 132 active+clean; 582 KiB data, 228 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-11T21:27:06.577 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:27:06 smithi097 ceph-mon[33317]: 5.2 scrub starts 2024-03-11T21:27:06.577 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:27:06 smithi097 ceph-mon[33317]: 5.2 scrub ok 2024-03-11T21:27:06.577 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:27:06 smithi097 ceph-mon[33317]: purged_snaps scrub starts 2024-03-11T21:27:06.577 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:27:06 smithi097 ceph-mon[33317]: purged_snaps scrub ok 2024-03-11T21:27:06.577 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:27:06 smithi097 ceph-mon[33317]: osdmap e95: 8 total, 8 up, 8 in 2024-03-11T21:27:06.578 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:27:06 smithi097 ceph-mon[33317]: 6.5 scrub starts 2024-03-11T21:27:06.578 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:27:06 smithi097 ceph-mon[33317]: 6.5 scrub ok 2024-03-11T21:27:06.578 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:27:06 smithi097 ceph-mon[33317]: 6.11 scrub starts 2024-03-11T21:27:06.578 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:27:06 smithi097 ceph-mon[33317]: 6.11 scrub ok 2024-03-11T21:27:06.578 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:27:06 smithi097 ceph-mon[33317]: from='mgr.14607 v1:172.21.15.92:0/1702717597' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T21:27:07.361 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:27:07 smithi092 ceph-mon[37563]: 2.4 scrub starts 2024-03-11T21:27:07.361 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:27:07 smithi092 ceph-mon[37563]: 2.4 scrub ok 2024-03-11T21:27:07.361 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:27:07 smithi092 ceph-mon[37563]: 5.3 scrub starts 2024-03-11T21:27:07.361 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:27:07 smithi092 ceph-mon[37563]: 5.3 scrub ok 2024-03-11T21:27:07.362 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:27:07 smithi092 ceph-mon[37563]: 5.13 scrub starts 2024-03-11T21:27:07.362 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:27:07 smithi092 ceph-mon[37563]: 5.13 scrub ok 2024-03-11T21:27:07.362 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:27:07 smithi092 ceph-mon[29572]: 2.4 scrub starts 2024-03-11T21:27:07.362 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:27:07 smithi092 ceph-mon[29572]: 2.4 scrub ok 2024-03-11T21:27:07.362 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:27:07 smithi092 ceph-mon[29572]: 5.3 scrub starts 2024-03-11T21:27:07.362 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:27:07 smithi092 ceph-mon[29572]: 5.3 scrub ok 2024-03-11T21:27:07.362 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:27:07 smithi092 ceph-mon[29572]: 5.13 scrub starts 2024-03-11T21:27:07.362 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:27:07 smithi092 ceph-mon[29572]: 5.13 scrub ok 2024-03-11T21:27:07.577 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:27:07 smithi097 ceph-mon[33317]: 2.4 scrub starts 2024-03-11T21:27:07.577 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:27:07 smithi097 ceph-mon[33317]: 2.4 scrub ok 2024-03-11T21:27:07.577 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:27:07 smithi097 ceph-mon[33317]: 5.3 scrub starts 2024-03-11T21:27:07.577 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:27:07 smithi097 ceph-mon[33317]: 5.3 scrub ok 2024-03-11T21:27:07.577 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:27:07 smithi097 ceph-mon[33317]: 5.13 scrub starts 2024-03-11T21:27:07.577 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:27:07 smithi097 ceph-mon[33317]: 5.13 scrub ok 2024-03-11T21:27:08.111 INFO:journalctl@ceph.mgr.y.smithi092.stdout:Mar 11 21:27:07 smithi092 ceph-00db0016-dfed-11ee-95c8-87774f69a715-mgr-y[29801]: ::ffff:172.21.15.97 - - [11/Mar/2024:21:27:07] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-11T21:27:08.403 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:27:08 smithi092 ceph-mon[29572]: pgmap v34: 132 pgs: 132 active+clean; 582 KiB data, 228 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-11T21:27:08.404 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:27:08 smithi092 ceph-mon[29572]: 4.a scrub starts 2024-03-11T21:27:08.404 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:27:08 smithi092 ceph-mon[29572]: 4.a scrub ok 2024-03-11T21:27:08.404 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:27:08 smithi092 ceph-mon[29572]: 2.6 scrub starts 2024-03-11T21:27:08.404 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:27:08 smithi092 ceph-mon[29572]: 2.6 scrub ok 2024-03-11T21:27:08.404 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:27:08 smithi092 ceph-mon[29572]: 6.8 scrub starts 2024-03-11T21:27:08.404 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:27:08 smithi092 ceph-mon[29572]: 6.8 scrub ok 2024-03-11T21:27:08.404 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:27:08 smithi092 ceph-mon[29572]: 6.6 scrub starts 2024-03-11T21:27:08.404 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:27:08 smithi092 ceph-mon[29572]: 6.6 scrub ok 2024-03-11T21:27:08.404 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:27:08 smithi092 ceph-mon[29572]: osdmap e96: 8 total, 8 up, 8 in 2024-03-11T21:27:08.404 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:27:08 smithi092 ceph-mon[37563]: pgmap v34: 132 pgs: 132 active+clean; 582 KiB data, 228 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-11T21:27:08.404 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:27:08 smithi092 ceph-mon[37563]: 4.a scrub starts 2024-03-11T21:27:08.405 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:27:08 smithi092 ceph-mon[37563]: 4.a scrub ok 2024-03-11T21:27:08.405 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:27:08 smithi092 ceph-mon[37563]: 2.6 scrub starts 2024-03-11T21:27:08.405 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:27:08 smithi092 ceph-mon[37563]: 2.6 scrub ok 2024-03-11T21:27:08.405 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:27:08 smithi092 ceph-mon[37563]: 6.8 scrub starts 2024-03-11T21:27:08.405 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:27:08 smithi092 ceph-mon[37563]: 6.8 scrub ok 2024-03-11T21:27:08.405 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:27:08 smithi092 ceph-mon[37563]: 6.6 scrub starts 2024-03-11T21:27:08.405 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:27:08 smithi092 ceph-mon[37563]: 6.6 scrub ok 2024-03-11T21:27:08.405 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:27:08 smithi092 ceph-mon[37563]: osdmap e96: 8 total, 8 up, 8 in 2024-03-11T21:27:08.577 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:27:08 smithi097 ceph-mon[33317]: pgmap v34: 132 pgs: 132 active+clean; 582 KiB data, 228 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-11T21:27:08.577 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:27:08 smithi097 ceph-mon[33317]: 4.a scrub starts 2024-03-11T21:27:08.577 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:27:08 smithi097 ceph-mon[33317]: 4.a scrub ok 2024-03-11T21:27:08.578 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:27:08 smithi097 ceph-mon[33317]: 2.6 scrub starts 2024-03-11T21:27:08.578 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:27:08 smithi097 ceph-mon[33317]: 2.6 scrub ok 2024-03-11T21:27:08.578 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:27:08 smithi097 ceph-mon[33317]: 6.8 scrub starts 2024-03-11T21:27:08.578 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:27:08 smithi097 ceph-mon[33317]: 6.8 scrub ok 2024-03-11T21:27:08.578 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:27:08 smithi097 ceph-mon[33317]: 6.6 scrub starts 2024-03-11T21:27:08.578 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:27:08 smithi097 ceph-mon[33317]: 6.6 scrub ok 2024-03-11T21:27:08.578 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:27:08 smithi097 ceph-mon[33317]: osdmap e96: 8 total, 8 up, 8 in 2024-03-11T21:27:09.456 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:27:09 smithi092 ceph-mon[29572]: 4.4 scrub starts 2024-03-11T21:27:09.456 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:27:09 smithi092 ceph-mon[29572]: 4.4 scrub ok 2024-03-11T21:27:09.456 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:27:09 smithi092 ceph-mon[37563]: 4.4 scrub starts 2024-03-11T21:27:09.456 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:27:09 smithi092 ceph-mon[37563]: 4.4 scrub ok 2024-03-11T21:27:09.577 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:27:09 smithi097 ceph-mon[33317]: 4.4 scrub starts 2024-03-11T21:27:09.577 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:27:09 smithi097 ceph-mon[33317]: 4.4 scrub ok 2024-03-11T21:27:09.729 INFO:teuthology.orchestra.run.smithi092.stdout: 2024-03-11T21:27:09.729 INFO:teuthology.orchestra.run.smithi092.stdout:{"status":"HEALTH_OK","checks":{},"mutes":[]} 2024-03-11T21:27:10.361 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:27:10 smithi092 ceph-mon[29572]: 5.8 scrub starts 2024-03-11T21:27:10.361 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:27:10 smithi092 ceph-mon[29572]: 5.8 scrub ok 2024-03-11T21:27:10.361 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:27:10 smithi092 ceph-mon[29572]: pgmap v36: 132 pgs: 132 active+clean; 582 KiB data, 228 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T21:27:10.361 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:27:10 smithi092 ceph-mon[29572]: 5.1d deep-scrub starts 2024-03-11T21:27:10.362 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:27:10 smithi092 ceph-mon[29572]: 5.1d deep-scrub ok 2024-03-11T21:27:10.362 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:27:10 smithi092 ceph-mon[29572]: from='client.? v1:172.21.15.92:0/3639943355' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-03-11T21:27:10.362 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:27:10 smithi092 ceph-mon[37563]: 5.8 scrub starts 2024-03-11T21:27:10.362 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:27:10 smithi092 ceph-mon[37563]: 5.8 scrub ok 2024-03-11T21:27:10.362 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:27:10 smithi092 ceph-mon[37563]: pgmap v36: 132 pgs: 132 active+clean; 582 KiB data, 228 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T21:27:10.362 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:27:10 smithi092 ceph-mon[37563]: 5.1d deep-scrub starts 2024-03-11T21:27:10.362 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:27:10 smithi092 ceph-mon[37563]: 5.1d deep-scrub ok 2024-03-11T21:27:10.362 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:27:10 smithi092 ceph-mon[37563]: from='client.? v1:172.21.15.92:0/3639943355' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-03-11T21:27:10.577 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:27:10 smithi097 ceph-mon[33317]: 5.8 scrub starts 2024-03-11T21:27:10.577 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:27:10 smithi097 ceph-mon[33317]: 5.8 scrub ok 2024-03-11T21:27:10.577 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:27:10 smithi097 ceph-mon[33317]: pgmap v36: 132 pgs: 132 active+clean; 582 KiB data, 228 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T21:27:10.577 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:27:10 smithi097 ceph-mon[33317]: 5.1d deep-scrub starts 2024-03-11T21:27:10.577 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:27:10 smithi097 ceph-mon[33317]: 5.1d deep-scrub ok 2024-03-11T21:27:10.577 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:27:10 smithi097 ceph-mon[33317]: from='client.? v1:172.21.15.92:0/3639943355' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-03-11T21:27:10.832 INFO:journalctl@ceph.alertmanager.a.smithi092.stdout:Mar 11 21:27:10 smithi092 systemd[1]: Starting Ceph alertmanager.a for 00db0016-dfed-11ee-95c8-87774f69a715... 2024-03-11T21:27:10.860 INFO:tasks.cephadm.ceph_manager.ceph:wait_until_healthy done 2024-03-11T21:27:10.860 INFO:tasks.cephadm:Setup complete, yielding 2024-03-11T21:27:10.860 INFO:teuthology.run_tasks:Running task thrashosds... 2024-03-11T21:27:10.871 INFO:tasks.thrashosds:config is {'chance_pgnum_grow': 1, 'chance_pgnum_shrink': 1, 'chance_pgpnum_fix': 1, 'timeout': 1200, 'sighup_delay': 0.1, 'optrack_toggle_delay': 2.0, 'dump_ops_enable': 'true', 'noscrub_toggle_delay': 2.0, 'random_eio': 0.0} 2024-03-11T21:27:10.872 INFO:tasks.thrashosds:overrides is {'admin_socket': {'branch': 'main'}, 'ceph': {'conf': {'client.0': {'debug ms': 1, 'debug objecter': 20, 'debug rados': 20}, 'global': {'ms bind msgr2': False, 'ms type': 'async'}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon min osdmap epochs': 50, 'mon osdmap full prune interval': 2, 'mon osdmap full prune min': 15, 'mon osdmap full prune txsize': 2, 'paxos service trim min': 10}, 'osd': {'debug ms': 1, 'debug osd': 20, 'osd debug reject backfill probability': 0.3, 'osd delete sleep': 1, 'osd max backfills': 3, 'osd scrub max interval': 120, 'osd scrub min interval': 60, 'osd shutdown pgref assert': True, 'osd snap trim sleep': 2}}, 'flavor': 'default', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', 'but it is still running', 'objects unfound and apparently lost', '\\(POOL_APP_NOT_ENABLED\\)'], 'mon_bind_msgr2': False, 'sha1': '2b0101eb74d60610d544cb8a473e23a35ac1cc0b'}, 'ceph-deploy': {'conf': {'client': {'log file': '/var/log/ceph/ceph-$name.$pid.log'}, 'mon': {}}}, 'cephadm': {'cephadm_mode': 'root'}, 'install': {'ceph': {'flavor': 'default', 'sha1': '2b0101eb74d60610d544cb8a473e23a35ac1cc0b'}}, 'selinux': {'allowlist': ['scontext=system_u:system_r:logrotate_t:s0']}, 'workunit': {'branch': 'main', 'sha1': '1c349451176cc5b4ebfb24b22eaaa754e05cff6c'}} 2024-03-11T21:27:10.872 INFO:tasks.thrashosds:config is {'chance_pgnum_grow': 1, 'chance_pgnum_shrink': 1, 'chance_pgpnum_fix': 1, 'timeout': 1200, 'sighup_delay': 0.1, 'optrack_toggle_delay': 2.0, 'dump_ops_enable': 'true', 'noscrub_toggle_delay': 2.0, 'random_eio': 0.0} 2024-03-11T21:27:10.872 DEBUG:teuthology.orchestra.run.smithi092:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid 00db0016-dfed-11ee-95c8-87774f69a715 -- ceph config set mgr mgr_debug_aggressive_pg_num_changes true 2024-03-11T21:27:11.111 INFO:journalctl@ceph.alertmanager.a.smithi092.stdout:Mar 11 21:27:11 smithi092 podman[72662]: 2024-03-11 21:27:10.965517729 +0000 UTC m=+0.020180165 image pull c8568f914cd25b2062c44e9f79f9c18da6e3b85fe0c47a12a2191c61426c2b19 quay.io/prometheus/alertmanager:v0.25.0 2024-03-11T21:27:11.111 INFO:journalctl@ceph.alertmanager.a.smithi092.stdout:Mar 11 21:27:11 smithi092 podman[72662]: 2024-03-11 21:27:11.083559043 +0000 UTC m=+0.138221492 volume create 44b788cd7107f4e9f36d15ed43b805a5c6ef32d26dd6e7cd4bc7b825d0bb58f9 2024-03-11T21:27:11.452 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:27:11 smithi092 ceph-mon[37563]: 6.17 deep-scrub starts 2024-03-11T21:27:11.452 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:27:11 smithi092 ceph-mon[37563]: 6.17 deep-scrub ok 2024-03-11T21:27:11.453 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:27:11 smithi092 ceph-mon[37563]: 2.10 scrub starts 2024-03-11T21:27:11.453 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:27:11 smithi092 ceph-mon[37563]: 2.10 scrub ok 2024-03-11T21:27:11.453 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:27:11 smithi092 ceph-mon[37563]: 5.11 deep-scrub starts 2024-03-11T21:27:11.453 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:27:11 smithi092 ceph-mon[37563]: 5.11 deep-scrub ok 2024-03-11T21:27:11.453 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:27:11 smithi092 ceph-mon[29572]: 6.17 deep-scrub starts 2024-03-11T21:27:11.453 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:27:11 smithi092 ceph-mon[29572]: 6.17 deep-scrub ok 2024-03-11T21:27:11.453 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:27:11 smithi092 ceph-mon[29572]: 2.10 scrub starts 2024-03-11T21:27:11.453 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:27:11 smithi092 ceph-mon[29572]: 2.10 scrub ok 2024-03-11T21:27:11.453 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:27:11 smithi092 ceph-mon[29572]: 5.11 deep-scrub starts 2024-03-11T21:27:11.453 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:27:11 smithi092 ceph-mon[29572]: 5.11 deep-scrub ok 2024-03-11T21:27:11.454 INFO:journalctl@ceph.alertmanager.a.smithi092.stdout:Mar 11 21:27:11 smithi092 podman[72662]: 2024-03-11 21:27:11.175735221 +0000 UTC m=+0.230397624 container create bea3b94d892511f16d09aff1f908794e43c75beb055ed367ef24c96249cb49df (image=quay.io/prometheus/alertmanager:v0.25.0, name=ceph-00db0016-dfed-11ee-95c8-87774f69a715-alertmanager-a, maintainer=The Prometheus Authors ) 2024-03-11T21:27:11.454 INFO:journalctl@ceph.alertmanager.a.smithi092.stdout:Mar 11 21:27:11 smithi092 podman[72662]: 2024-03-11 21:27:11.470185766 +0000 UTC m=+0.524848184 container init bea3b94d892511f16d09aff1f908794e43c75beb055ed367ef24c96249cb49df (image=quay.io/prometheus/alertmanager:v0.25.0, name=ceph-00db0016-dfed-11ee-95c8-87774f69a715-alertmanager-a, maintainer=The Prometheus Authors ) 2024-03-11T21:27:11.577 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:27:11 smithi097 ceph-mon[33317]: 6.17 deep-scrub starts 2024-03-11T21:27:11.577 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:27:11 smithi097 ceph-mon[33317]: 6.17 deep-scrub ok 2024-03-11T21:27:11.577 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:27:11 smithi097 ceph-mon[33317]: 2.10 scrub starts 2024-03-11T21:27:11.577 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:27:11 smithi097 ceph-mon[33317]: 2.10 scrub ok 2024-03-11T21:27:11.577 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:27:11 smithi097 ceph-mon[33317]: 5.11 deep-scrub starts 2024-03-11T21:27:11.577 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:27:11 smithi097 ceph-mon[33317]: 5.11 deep-scrub ok 2024-03-11T21:27:11.666 INFO:teuthology.orchestra.run.smithi092.stderr:Inferring config /var/lib/ceph/00db0016-dfed-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T21:27:11.733 INFO:journalctl@ceph.alertmanager.a.smithi092.stdout:Mar 11 21:27:11 smithi092 podman[72662]: 2024-03-11 21:27:11.472865739 +0000 UTC m=+0.527528145 container start bea3b94d892511f16d09aff1f908794e43c75beb055ed367ef24c96249cb49df (image=quay.io/prometheus/alertmanager:v0.25.0, name=ceph-00db0016-dfed-11ee-95c8-87774f69a715-alertmanager-a, maintainer=The Prometheus Authors ) 2024-03-11T21:27:11.733 INFO:journalctl@ceph.alertmanager.a.smithi092.stdout:Mar 11 21:27:11 smithi092 ceph-00db0016-dfed-11ee-95c8-87774f69a715-alertmanager-a[72702]: ts=2024-03-11T21:27:11.485Z caller=main.go:240 level=info msg="Starting Alertmanager" version="(version=0.25.0, branch=HEAD, revision=258fab7cdd551f2cf251ed0348f0ad7289aee789)" 2024-03-11T21:27:11.733 INFO:journalctl@ceph.alertmanager.a.smithi092.stdout:Mar 11 21:27:11 smithi092 ceph-00db0016-dfed-11ee-95c8-87774f69a715-alertmanager-a[72702]: ts=2024-03-11T21:27:11.485Z caller=main.go:241 level=info build_context="(go=go1.19.4, user=root@abe866dd5717, date=20221222-14:51:36)" 2024-03-11T21:27:11.733 INFO:journalctl@ceph.alertmanager.a.smithi092.stdout:Mar 11 21:27:11 smithi092 ceph-00db0016-dfed-11ee-95c8-87774f69a715-alertmanager-a[72702]: ts=2024-03-11T21:27:11.486Z caller=cluster.go:185 level=info component=cluster msg="setting advertise address explicitly" addr=172.21.15.92 port=9094 2024-03-11T21:27:11.733 INFO:journalctl@ceph.alertmanager.a.smithi092.stdout:Mar 11 21:27:11 smithi092 ceph-00db0016-dfed-11ee-95c8-87774f69a715-alertmanager-a[72702]: ts=2024-03-11T21:27:11.487Z caller=cluster.go:681 level=info component=cluster msg="Waiting for gossip to settle..." interval=2s 2024-03-11T21:27:11.733 INFO:journalctl@ceph.alertmanager.a.smithi092.stdout:Mar 11 21:27:11 smithi092 ceph-00db0016-dfed-11ee-95c8-87774f69a715-alertmanager-a[72702]: ts=2024-03-11T21:27:11.510Z caller=coordinator.go:113 level=info component=configuration msg="Loading configuration file" file=/etc/alertmanager/alertmanager.yml 2024-03-11T21:27:11.733 INFO:journalctl@ceph.alertmanager.a.smithi092.stdout:Mar 11 21:27:11 smithi092 ceph-00db0016-dfed-11ee-95c8-87774f69a715-alertmanager-a[72702]: ts=2024-03-11T21:27:11.510Z caller=coordinator.go:126 level=info component=configuration msg="Completed loading of configuration file" file=/etc/alertmanager/alertmanager.yml 2024-03-11T21:27:11.733 INFO:journalctl@ceph.alertmanager.a.smithi092.stdout:Mar 11 21:27:11 smithi092 ceph-00db0016-dfed-11ee-95c8-87774f69a715-alertmanager-a[72702]: ts=2024-03-11T21:27:11.512Z caller=tls_config.go:232 level=info msg="Listening on" address=[::]:9093 2024-03-11T21:27:11.734 INFO:journalctl@ceph.alertmanager.a.smithi092.stdout:Mar 11 21:27:11 smithi092 ceph-00db0016-dfed-11ee-95c8-87774f69a715-alertmanager-a[72702]: ts=2024-03-11T21:27:11.512Z caller=tls_config.go:235 level=info msg="TLS is disabled." http2=false address=[::]:9093 2024-03-11T21:27:11.734 INFO:journalctl@ceph.alertmanager.a.smithi092.stdout:Mar 11 21:27:11 smithi092 bash[72662]: bea3b94d892511f16d09aff1f908794e43c75beb055ed367ef24c96249cb49df 2024-03-11T21:27:11.734 INFO:journalctl@ceph.alertmanager.a.smithi092.stdout:Mar 11 21:27:11 smithi092 systemd[1]: Started Ceph alertmanager.a for 00db0016-dfed-11ee-95c8-87774f69a715. 2024-03-11T21:27:12.015 INFO:journalctl@ceph.mgr.y.smithi092.stdout:Mar 11 21:27:11 smithi092 ceph-00db0016-dfed-11ee-95c8-87774f69a715-mgr-y[29801]: [11/Mar/2024:21:27:11] ENGINE Bus STOPPING 2024-03-11T21:27:12.015 INFO:journalctl@ceph.mgr.y.smithi092.stdout:Mar 11 21:27:11 smithi092 ceph-00db0016-dfed-11ee-95c8-87774f69a715-mgr-y[29801]: [11/Mar/2024:21:27:11] ENGINE HTTP Server cherrypy._cpwsgi_server.CPWSGIServer(('::', 9283)) shut down 2024-03-11T21:27:12.015 INFO:journalctl@ceph.mgr.y.smithi092.stdout:Mar 11 21:27:11 smithi092 ceph-00db0016-dfed-11ee-95c8-87774f69a715-mgr-y[29801]: [11/Mar/2024:21:27:11] ENGINE Bus STOPPED 2024-03-11T21:27:12.015 INFO:journalctl@ceph.mgr.y.smithi092.stdout:Mar 11 21:27:11 smithi092 ceph-00db0016-dfed-11ee-95c8-87774f69a715-mgr-y[29801]: [11/Mar/2024:21:27:11] ENGINE Bus STARTING 2024-03-11T21:27:12.015 INFO:journalctl@ceph.mgr.y.smithi092.stdout:Mar 11 21:27:11 smithi092 ceph-00db0016-dfed-11ee-95c8-87774f69a715-mgr-y[29801]: [11/Mar/2024:21:27:11] ENGINE Serving on http://:::9283 2024-03-11T21:27:12.015 INFO:journalctl@ceph.mgr.y.smithi092.stdout:Mar 11 21:27:11 smithi092 ceph-00db0016-dfed-11ee-95c8-87774f69a715-mgr-y[29801]: [11/Mar/2024:21:27:11] ENGINE Bus STARTED 2024-03-11T21:27:12.327 INFO:journalctl@ceph.iscsi.iscsi.a.smithi097.stdout:Mar 11 21:27:11 smithi097 ceph-00db0016-dfed-11ee-95c8-87774f69a715-iscsi-iscsi-a[60943]: debug there is no tcmu-runner data available 2024-03-11T21:27:12.327 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:27:12 smithi097 ceph-mon[33317]: 5.1c scrub starts 2024-03-11T21:27:12.327 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:27:12 smithi097 ceph-mon[33317]: 5.1c scrub ok 2024-03-11T21:27:12.327 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:27:12 smithi097 ceph-mon[33317]: 5.a scrub starts 2024-03-11T21:27:12.327 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:27:12 smithi097 ceph-mon[33317]: 5.a scrub ok 2024-03-11T21:27:12.327 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:27:12 smithi097 ceph-mon[33317]: pgmap v37: 132 pgs: 132 active+clean; 582 KiB data, 228 MiB used, 715 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-03-11T21:27:12.327 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:27:12 smithi097 ceph-mon[33317]: 6.9 deep-scrub starts 2024-03-11T21:27:12.327 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:27:12 smithi097 ceph-mon[33317]: 6.9 deep-scrub ok 2024-03-11T21:27:12.327 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:27:12 smithi097 ceph-mon[33317]: from='mgr.14607 ' entity='mgr.y' 2024-03-11T21:27:12.327 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:27:12 smithi097 ceph-mon[33317]: from='mgr.14607 ' entity='mgr.y' 2024-03-11T21:27:12.327 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:27:12 smithi097 ceph-mon[33317]: from='mgr.14607 ' entity='mgr.y' 2024-03-11T21:27:12.328 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:27:12 smithi097 ceph-mon[33317]: from='mgr.14607 ' entity='mgr.y' 2024-03-11T21:27:12.328 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:27:12 smithi097 ceph-mon[33317]: Regenerating cephadm self-signed grafana TLS certificates 2024-03-11T21:27:12.328 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:27:12 smithi097 ceph-mon[33317]: from='mgr.14607 ' entity='mgr.y' 2024-03-11T21:27:12.328 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:27:12 smithi097 ceph-mon[33317]: from='mgr.14607 ' entity='mgr.y' 2024-03-11T21:27:12.328 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:27:12 smithi097 ceph-mon[33317]: from='mgr.14607 v1:172.21.15.92:0/1702717597' entity='mgr.y' cmd=[{"prefix": "dashboard set-grafana-api-ssl-verify", "value": "false"}]: dispatch 2024-03-11T21:27:12.328 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:27:12 smithi097 ceph-mon[33317]: from='mon.? v1:172.21.15.97:0/414987216' entity='mon.' cmd=[{"prefix": "dashboard set-grafana-api-ssl-verify", "value": "false"}]: dispatch 2024-03-11T21:27:12.328 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:27:12 smithi097 ceph-mon[33317]: from='mgr.14607 ' entity='mgr.y' 2024-03-11T21:27:12.328 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:27:12 smithi097 ceph-mon[33317]: Deploying daemon grafana.a on smithi097 2024-03-11T21:27:12.361 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:27:12 smithi092 ceph-mon[29572]: 5.1c scrub starts 2024-03-11T21:27:12.361 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:27:12 smithi092 ceph-mon[29572]: 5.1c scrub ok 2024-03-11T21:27:12.361 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:27:12 smithi092 ceph-mon[29572]: 5.a scrub starts 2024-03-11T21:27:12.362 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:27:12 smithi092 ceph-mon[29572]: 5.a scrub ok 2024-03-11T21:27:12.362 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:27:12 smithi092 ceph-mon[29572]: pgmap v37: 132 pgs: 132 active+clean; 582 KiB data, 228 MiB used, 715 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-03-11T21:27:12.362 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:27:12 smithi092 ceph-mon[29572]: 6.9 deep-scrub starts 2024-03-11T21:27:12.362 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:27:12 smithi092 ceph-mon[29572]: 6.9 deep-scrub ok 2024-03-11T21:27:12.362 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:27:12 smithi092 ceph-mon[29572]: from='mgr.14607 ' entity='mgr.y' 2024-03-11T21:27:12.362 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:27:12 smithi092 ceph-mon[29572]: from='mgr.14607 ' entity='mgr.y' 2024-03-11T21:27:12.362 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:27:12 smithi092 ceph-mon[29572]: from='mgr.14607 ' entity='mgr.y' 2024-03-11T21:27:12.362 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:27:12 smithi092 ceph-mon[29572]: from='mgr.14607 ' entity='mgr.y' 2024-03-11T21:27:12.362 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:27:12 smithi092 ceph-mon[29572]: Regenerating cephadm self-signed grafana TLS certificates 2024-03-11T21:27:12.362 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:27:12 smithi092 ceph-mon[29572]: from='mgr.14607 ' entity='mgr.y' 2024-03-11T21:27:12.362 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:27:12 smithi092 ceph-mon[29572]: from='mgr.14607 ' entity='mgr.y' 2024-03-11T21:27:12.362 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:27:12 smithi092 ceph-mon[29572]: from='mgr.14607 v1:172.21.15.92:0/1702717597' entity='mgr.y' cmd=[{"prefix": "dashboard set-grafana-api-ssl-verify", "value": "false"}]: dispatch 2024-03-11T21:27:12.362 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:27:12 smithi092 ceph-mon[29572]: from='mon.? v1:172.21.15.97:0/414987216' entity='mon.' cmd=[{"prefix": "dashboard set-grafana-api-ssl-verify", "value": "false"}]: dispatch 2024-03-11T21:27:12.363 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:27:12 smithi092 ceph-mon[29572]: from='mgr.14607 ' entity='mgr.y' 2024-03-11T21:27:12.363 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:27:12 smithi092 ceph-mon[29572]: Deploying daemon grafana.a on smithi097 2024-03-11T21:27:12.363 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:27:12 smithi092 ceph-mon[37563]: 5.1c scrub starts 2024-03-11T21:27:12.363 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:27:12 smithi092 ceph-mon[37563]: 5.1c scrub ok 2024-03-11T21:27:12.363 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:27:12 smithi092 ceph-mon[37563]: 5.a scrub starts 2024-03-11T21:27:12.363 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:27:12 smithi092 ceph-mon[37563]: 5.a scrub ok 2024-03-11T21:27:12.363 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:27:12 smithi092 ceph-mon[37563]: pgmap v37: 132 pgs: 132 active+clean; 582 KiB data, 228 MiB used, 715 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-03-11T21:27:12.363 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:27:12 smithi092 ceph-mon[37563]: 6.9 deep-scrub starts 2024-03-11T21:27:12.363 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:27:12 smithi092 ceph-mon[37563]: 6.9 deep-scrub ok 2024-03-11T21:27:12.363 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:27:12 smithi092 ceph-mon[37563]: from='mgr.14607 ' entity='mgr.y' 2024-03-11T21:27:12.363 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:27:12 smithi092 ceph-mon[37563]: from='mgr.14607 ' entity='mgr.y' 2024-03-11T21:27:12.363 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:27:12 smithi092 ceph-mon[37563]: from='mgr.14607 ' entity='mgr.y' 2024-03-11T21:27:12.363 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:27:12 smithi092 ceph-mon[37563]: from='mgr.14607 ' entity='mgr.y' 2024-03-11T21:27:12.364 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:27:12 smithi092 ceph-mon[37563]: Regenerating cephadm self-signed grafana TLS certificates 2024-03-11T21:27:12.364 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:27:12 smithi092 ceph-mon[37563]: from='mgr.14607 ' entity='mgr.y' 2024-03-11T21:27:12.364 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:27:12 smithi092 ceph-mon[37563]: from='mgr.14607 ' entity='mgr.y' 2024-03-11T21:27:12.364 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:27:12 smithi092 ceph-mon[37563]: from='mgr.14607 v1:172.21.15.92:0/1702717597' entity='mgr.y' cmd=[{"prefix": "dashboard set-grafana-api-ssl-verify", "value": "false"}]: dispatch 2024-03-11T21:27:12.364 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:27:12 smithi092 ceph-mon[37563]: from='mon.? v1:172.21.15.97:0/414987216' entity='mon.' cmd=[{"prefix": "dashboard set-grafana-api-ssl-verify", "value": "false"}]: dispatch 2024-03-11T21:27:12.364 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:27:12 smithi092 ceph-mon[37563]: from='mgr.14607 ' entity='mgr.y' 2024-03-11T21:27:12.364 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:27:12 smithi092 ceph-mon[37563]: Deploying daemon grafana.a on smithi097 2024-03-11T21:27:13.470 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:27:13 smithi092 ceph-mon[29572]: 2.16 deep-scrub starts 2024-03-11T21:27:13.470 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:27:13 smithi092 ceph-mon[29572]: 2.16 deep-scrub ok 2024-03-11T21:27:13.470 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:27:13 smithi092 ceph-mon[29572]: from='client.24469 v1:172.21.15.97:0/404208310' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:27:13.470 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:27:13 smithi092 ceph-mon[29572]: 2.13 scrub starts 2024-03-11T21:27:13.471 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:27:13 smithi092 ceph-mon[29572]: 2.13 scrub ok 2024-03-11T21:27:13.471 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:27:13 smithi092 ceph-mon[29572]: 6.12 scrub starts 2024-03-11T21:27:13.471 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:27:13 smithi092 ceph-mon[29572]: 6.12 scrub ok 2024-03-11T21:27:13.471 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:27:13 smithi092 ceph-mon[29572]: 4.19 scrub starts 2024-03-11T21:27:13.471 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:27:13 smithi092 ceph-mon[29572]: 4.19 scrub ok 2024-03-11T21:27:13.471 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:27:13 smithi092 ceph-mon[37563]: 2.16 deep-scrub starts 2024-03-11T21:27:13.471 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:27:13 smithi092 ceph-mon[37563]: 2.16 deep-scrub ok 2024-03-11T21:27:13.471 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:27:13 smithi092 ceph-mon[37563]: from='client.24469 v1:172.21.15.97:0/404208310' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:27:13.471 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:27:13 smithi092 ceph-mon[37563]: 2.13 scrub starts 2024-03-11T21:27:13.471 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:27:13 smithi092 ceph-mon[37563]: 2.13 scrub ok 2024-03-11T21:27:13.471 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:27:13 smithi092 ceph-mon[37563]: 6.12 scrub starts 2024-03-11T21:27:13.471 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:27:13 smithi092 ceph-mon[37563]: 6.12 scrub ok 2024-03-11T21:27:13.471 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:27:13 smithi092 ceph-mon[37563]: 4.19 scrub starts 2024-03-11T21:27:13.472 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:27:13 smithi092 ceph-mon[37563]: 4.19 scrub ok 2024-03-11T21:27:13.577 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:27:13 smithi097 ceph-mon[33317]: 2.16 deep-scrub starts 2024-03-11T21:27:13.577 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:27:13 smithi097 ceph-mon[33317]: 2.16 deep-scrub ok 2024-03-11T21:27:13.577 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:27:13 smithi097 ceph-mon[33317]: from='client.24469 v1:172.21.15.97:0/404208310' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:27:13.577 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:27:13 smithi097 ceph-mon[33317]: 2.13 scrub starts 2024-03-11T21:27:13.577 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:27:13 smithi097 ceph-mon[33317]: 2.13 scrub ok 2024-03-11T21:27:13.577 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:27:13 smithi097 ceph-mon[33317]: 6.12 scrub starts 2024-03-11T21:27:13.577 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:27:13 smithi097 ceph-mon[33317]: 6.12 scrub ok 2024-03-11T21:27:13.577 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:27:13 smithi097 ceph-mon[33317]: 4.19 scrub starts 2024-03-11T21:27:13.577 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:27:13 smithi097 ceph-mon[33317]: 4.19 scrub ok 2024-03-11T21:27:13.861 INFO:journalctl@ceph.alertmanager.a.smithi092.stdout:Mar 11 21:27:13 smithi092 ceph-00db0016-dfed-11ee-95c8-87774f69a715-alertmanager-a[72702]: ts=2024-03-11T21:27:13.487Z caller=cluster.go:706 level=info component=cluster msg="gossip not settled" polls=0 before=0 now=1 elapsed=2.000146069s 2024-03-11T21:27:13.861 INFO:journalctl@ceph.mgr.y.smithi092.stdout:Mar 11 21:27:13 smithi092 ceph-00db0016-dfed-11ee-95c8-87774f69a715-mgr-y[29801]: [11/Mar/2024:21:27:13] ENGINE Bus STOPPING 2024-03-11T21:27:13.861 INFO:journalctl@ceph.mgr.y.smithi092.stdout:Mar 11 21:27:13 smithi092 ceph-00db0016-dfed-11ee-95c8-87774f69a715-mgr-y[29801]: [11/Mar/2024:21:27:13] ENGINE HTTP Server cherrypy._cpwsgi_server.CPWSGIServer(('::', 9283)) shut down 2024-03-11T21:27:13.861 INFO:journalctl@ceph.mgr.y.smithi092.stdout:Mar 11 21:27:13 smithi092 ceph-00db0016-dfed-11ee-95c8-87774f69a715-mgr-y[29801]: [11/Mar/2024:21:27:13] ENGINE Bus STOPPED 2024-03-11T21:27:13.861 INFO:journalctl@ceph.mgr.y.smithi092.stdout:Mar 11 21:27:13 smithi092 ceph-00db0016-dfed-11ee-95c8-87774f69a715-mgr-y[29801]: [11/Mar/2024:21:27:13] ENGINE Bus STARTING 2024-03-11T21:27:13.862 INFO:journalctl@ceph.mgr.y.smithi092.stdout:Mar 11 21:27:13 smithi092 ceph-00db0016-dfed-11ee-95c8-87774f69a715-mgr-y[29801]: [11/Mar/2024:21:27:13] ENGINE Serving on http://:::9283 2024-03-11T21:27:13.862 INFO:journalctl@ceph.mgr.y.smithi092.stdout:Mar 11 21:27:13 smithi092 ceph-00db0016-dfed-11ee-95c8-87774f69a715-mgr-y[29801]: [11/Mar/2024:21:27:13] ENGINE Bus STARTED 2024-03-11T21:27:14.263 INFO:tasks.thrashosds:Beginning thrashosds... 2024-03-11T21:27:14.263 INFO:tasks.cephadm.ceph_manager.ceph:waiting for clean 2024-03-11T21:27:14.263 DEBUG:teuthology.orchestra.run.smithi092:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid 00db0016-dfed-11ee-95c8-87774f69a715 -- ceph pg dump --format=json 2024-03-11T21:27:14.509 INFO:teuthology.orchestra.run.smithi092.stderr:Inferring config /var/lib/ceph/00db0016-dfed-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T21:27:14.844 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:27:14 smithi092 ceph-mon[29572]: pgmap v38: 132 pgs: 132 active+clean; 582 KiB data, 228 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T21:27:14.844 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:27:14 smithi092 ceph-mon[29572]: purged_snaps scrub starts 2024-03-11T21:27:14.845 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:27:14 smithi092 ceph-mon[29572]: purged_snaps scrub ok 2024-03-11T21:27:14.845 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:27:14 smithi092 ceph-mon[29572]: from='client.? v1:172.21.15.92:0/178891884' entity='client.admin' 2024-03-11T21:27:14.845 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:27:14 smithi092 ceph-mon[29572]: 4.5 scrub starts 2024-03-11T21:27:14.845 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:27:14 smithi092 ceph-mon[29572]: 4.5 scrub ok 2024-03-11T21:27:14.845 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:27:14 smithi092 ceph-mon[29572]: osdmap e97: 8 total, 8 up, 8 in 2024-03-11T21:27:14.845 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:27:14 smithi092 ceph-mon[37563]: pgmap v38: 132 pgs: 132 active+clean; 582 KiB data, 228 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T21:27:14.845 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:27:14 smithi092 ceph-mon[37563]: purged_snaps scrub starts 2024-03-11T21:27:14.845 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:27:14 smithi092 ceph-mon[37563]: purged_snaps scrub ok 2024-03-11T21:27:14.845 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:27:14 smithi092 ceph-mon[37563]: from='client.? v1:172.21.15.92:0/178891884' entity='client.admin' 2024-03-11T21:27:14.845 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:27:14 smithi092 ceph-mon[37563]: 4.5 scrub starts 2024-03-11T21:27:14.845 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:27:14 smithi092 ceph-mon[37563]: 4.5 scrub ok 2024-03-11T21:27:14.846 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:27:14 smithi092 ceph-mon[37563]: osdmap e97: 8 total, 8 up, 8 in 2024-03-11T21:27:15.077 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:27:14 smithi097 ceph-mon[33317]: pgmap v38: 132 pgs: 132 active+clean; 582 KiB data, 228 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T21:27:15.077 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:27:14 smithi097 ceph-mon[33317]: purged_snaps scrub starts 2024-03-11T21:27:15.077 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:27:14 smithi097 ceph-mon[33317]: purged_snaps scrub ok 2024-03-11T21:27:15.077 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:27:14 smithi097 ceph-mon[33317]: from='client.? v1:172.21.15.92:0/178891884' entity='client.admin' 2024-03-11T21:27:15.077 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:27:14 smithi097 ceph-mon[33317]: 4.5 scrub starts 2024-03-11T21:27:15.077 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:27:14 smithi097 ceph-mon[33317]: 4.5 scrub ok 2024-03-11T21:27:15.077 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:27:14 smithi097 ceph-mon[33317]: osdmap e97: 8 total, 8 up, 8 in 2024-03-11T21:27:15.841 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:27:15 smithi092 ceph-mon[29572]: 4.3 scrub starts 2024-03-11T21:27:15.841 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:27:15 smithi092 ceph-mon[29572]: 4.3 scrub ok 2024-03-11T21:27:15.841 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:27:15 smithi092 ceph-mon[29572]: 5.10 scrub starts 2024-03-11T21:27:15.841 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:27:15 smithi092 ceph-mon[29572]: 5.10 scrub ok 2024-03-11T21:27:15.841 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:27:15 smithi092 ceph-mon[29572]: 4.12 scrub starts 2024-03-11T21:27:15.842 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:27:15 smithi092 ceph-mon[37563]: 4.3 scrub starts 2024-03-11T21:27:15.842 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:27:15 smithi092 ceph-mon[37563]: 4.3 scrub ok 2024-03-11T21:27:15.842 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:27:15 smithi092 ceph-mon[37563]: 5.10 scrub starts 2024-03-11T21:27:15.842 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:27:15 smithi092 ceph-mon[37563]: 5.10 scrub ok 2024-03-11T21:27:15.842 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:27:15 smithi092 ceph-mon[37563]: 4.12 scrub starts 2024-03-11T21:27:16.077 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:27:15 smithi097 ceph-mon[33317]: 4.3 scrub starts 2024-03-11T21:27:16.077 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:27:15 smithi097 ceph-mon[33317]: 4.3 scrub ok 2024-03-11T21:27:16.077 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:27:15 smithi097 ceph-mon[33317]: 5.10 scrub starts 2024-03-11T21:27:16.077 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:27:15 smithi097 ceph-mon[33317]: 5.10 scrub ok 2024-03-11T21:27:16.077 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:27:15 smithi097 ceph-mon[33317]: 4.12 scrub starts 2024-03-11T21:27:16.584 INFO:teuthology.orchestra.run.smithi092.stdout: 2024-03-11T21:27:16.586 INFO:teuthology.orchestra.run.smithi092.stderr:dumped all 2024-03-11T21:27:16.861 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:27:16 smithi092 ceph-mon[29572]: pgmap v40: 132 pgs: 132 active+clean; 582 KiB data, 228 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T21:27:16.861 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:27:16 smithi092 ceph-mon[29572]: 4.12 scrub ok 2024-03-11T21:27:16.861 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:27:16 smithi092 ceph-mon[29572]: 5.17 scrub starts 2024-03-11T21:27:16.861 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:27:16 smithi092 ceph-mon[29572]: 5.17 scrub ok 2024-03-11T21:27:16.861 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:27:16 smithi092 ceph-mon[29572]: from='mgr.14607 ' entity='mgr.y' 2024-03-11T21:27:16.861 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:27:16 smithi092 ceph-mon[37563]: pgmap v40: 132 pgs: 132 active+clean; 582 KiB data, 228 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T21:27:16.862 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:27:16 smithi092 ceph-mon[37563]: 4.12 scrub ok 2024-03-11T21:27:16.862 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:27:16 smithi092 ceph-mon[37563]: 5.17 scrub starts 2024-03-11T21:27:16.862 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:27:16 smithi092 ceph-mon[37563]: 5.17 scrub ok 2024-03-11T21:27:16.862 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:27:16 smithi092 ceph-mon[37563]: from='mgr.14607 ' entity='mgr.y' 2024-03-11T21:27:17.077 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:27:16 smithi097 ceph-mon[33317]: pgmap v40: 132 pgs: 132 active+clean; 582 KiB data, 228 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T21:27:17.077 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:27:16 smithi097 ceph-mon[33317]: 4.12 scrub ok 2024-03-11T21:27:17.077 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:27:16 smithi097 ceph-mon[33317]: 5.17 scrub starts 2024-03-11T21:27:17.077 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:27:16 smithi097 ceph-mon[33317]: 5.17 scrub ok 2024-03-11T21:27:17.077 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:27:16 smithi097 ceph-mon[33317]: from='mgr.14607 ' entity='mgr.y' 2024-03-11T21:27:17.372 INFO:teuthology.orchestra.run.smithi092.stdout:{"pg_ready":true,"pg_map":{"version":40,"stamp":"2024-03-11T21:27:14.967662+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":596392,"num_objects":198,"num_object_clones":0,"num_object_copies":594,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":198,"num_whiteouts":0,"num_read":1022,"num_read_kb":868,"num_write":615,"num_write_kb":2461,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":21,"num_bytes_recovered":1771324,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":620,"ondisk_log_size":620,"up":396,"acting":396,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":396,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":749961216,"kb_used":233288,"kb_used_data":8828,"kb_used_omap":12,"kb_used_meta":224371,"kb_avail":749727928,"statfs":{"total":767960285184,"available":767721398272,"internally_reserved":0,"allocated":9039872,"data_stored":4971664,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12752,"internal_metadata":229756464},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":10,"num_read_kb":10,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.001993"},"pg_stats":[{"pgid":"6.1b","version":"0'0","reported_seq":65,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-03-11T21:27:08.131567+0000","last_change":"2024-03-11T21:26:53.584399+0000","last_active":"2024-03-11T21:27:08.131567+0000","last_peered":"2024-03-11T21:27:08.131567+0000","last_clean":"2024-03-11T21:27:08.131567+0000","last_became_active":"2024-03-11T21:25:45.302931+0000","last_became_peered":"2024-03-11T21:25:45.302931+0000","last_unstale":"2024-03-11T21:27:08.131567+0000","last_undegraded":"2024-03-11T21:27:08.131567+0000","last_fullsized":"2024-03-11T21:27:08.131567+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:26:53.584307+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:43.559452+0000","last_clean_scrub_stamp":"2024-03-11T21:26:53.584307+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:27:55.466757+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":61,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-03-11T21:27:08.128237+0000","last_change":"2024-03-11T21:25:42.579254+0000","last_active":"2024-03-11T21:27:08.128237+0000","last_peered":"2024-03-11T21:27:08.128237+0000","last_clean":"2024-03-11T21:27:08.128237+0000","last_became_active":"2024-03-11T21:25:42.578893+0000","last_became_peered":"2024-03-11T21:25:42.578893+0000","last_unstale":"2024-03-11T21:27:08.128237+0000","last_undegraded":"2024-03-11T21:27:08.128237+0000","last_fullsized":"2024-03-11T21:27:08.128237+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:25:41.544550+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:41.544550+0000","last_clean_scrub_stamp":"2024-03-11T21:25:41.544550+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:27:10.983445+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1f","version":"0'0","reported_seq":77,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-03-11T21:27:08.130819+0000","last_change":"2024-03-11T21:26:46.588463+0000","last_active":"2024-03-11T21:27:08.130819+0000","last_peered":"2024-03-11T21:27:08.130819+0000","last_clean":"2024-03-11T21:27:08.130819+0000","last_became_active":"2024-03-11T21:25:38.540510+0000","last_became_peered":"2024-03-11T21:25:38.540510+0000","last_unstale":"2024-03-11T21:27:08.130819+0000","last_undegraded":"2024-03-11T21:27:08.130819+0000","last_fullsized":"2024-03-11T21:27:08.130819+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:26:46.588380+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:37.512018+0000","last_clean_scrub_stamp":"2024-03-11T21:26:46.588380+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:28:06.851002+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,4],"acting":[0,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.19","version":"74'15","reported_seq":90,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-03-11T21:27:08.131480+0000","last_change":"2024-03-11T21:25:40.554584+0000","last_active":"2024-03-11T21:27:08.131480+0000","last_peered":"2024-03-11T21:27:08.131480+0000","last_clean":"2024-03-11T21:27:08.131480+0000","last_became_active":"2024-03-11T21:25:40.554207+0000","last_became_peered":"2024-03-11T21:25:40.554207+0000","last_unstale":"2024-03-11T21:27:08.131480+0000","last_undegraded":"2024-03-11T21:27:08.131480+0000","last_fullsized":"2024-03-11T21:27:08.131480+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:25:39.525554+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:39.525554+0000","last_clean_scrub_stamp":"2024-03-11T21:25:39.525554+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.1a","version":"0'0","reported_seq":57,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-03-11T21:27:08.128436+0000","last_change":"2024-03-11T21:25:44.591548+0000","last_active":"2024-03-11T21:27:08.128436+0000","last_peered":"2024-03-11T21:27:08.128436+0000","last_clean":"2024-03-11T21:27:08.128436+0000","last_became_active":"2024-03-11T21:25:44.590923+0000","last_became_peered":"2024-03-11T21:25:44.590923+0000","last_unstale":"2024-03-11T21:27:08.128436+0000","last_undegraded":"2024-03-11T21:27:08.128436+0000","last_fullsized":"2024-03-11T21:27:08.128436+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:25:43.559452+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:43.559452+0000","last_clean_scrub_stamp":"2024-03-11T21:25:43.559452+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,1],"acting":[4,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":61,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-03-11T21:27:08.171256+0000","last_change":"2024-03-11T21:25:42.578761+0000","last_active":"2024-03-11T21:27:08.171256+0000","last_peered":"2024-03-11T21:27:08.171256+0000","last_clean":"2024-03-11T21:27:08.171256+0000","last_became_active":"2024-03-11T21:25:42.578424+0000","last_became_peered":"2024-03-11T21:25:42.578424+0000","last_unstale":"2024-03-11T21:27:08.171256+0000","last_undegraded":"2024-03-11T21:27:08.171256+0000","last_fullsized":"2024-03-11T21:27:08.171256+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:25:41.544550+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:41.544550+0000","last_clean_scrub_stamp":"2024-03-11T21:25:41.544550+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,7],"acting":[1,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1e","version":"0'0","reported_seq":77,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-03-11T21:27:08.132070+0000","last_change":"2024-03-11T21:26:49.595465+0000","last_active":"2024-03-11T21:27:08.132070+0000","last_peered":"2024-03-11T21:27:08.132070+0000","last_clean":"2024-03-11T21:27:08.132070+0000","last_became_active":"2024-03-11T21:25:38.537283+0000","last_became_peered":"2024-03-11T21:25:38.537283+0000","last_unstale":"2024-03-11T21:27:08.132070+0000","last_undegraded":"2024-03-11T21:27:08.132070+0000","last_fullsized":"2024-03-11T21:27:08.132070+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:26:49.595363+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:37.512018+0000","last_clean_scrub_stamp":"2024-03-11T21:26:49.595363+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:27:52.155430+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.18","version":"74'9","reported_seq":89,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-03-11T21:27:08.128402+0000","last_change":"2024-03-11T21:26:45.870099+0000","last_active":"2024-03-11T21:27:08.128402+0000","last_peered":"2024-03-11T21:27:08.128402+0000","last_clean":"2024-03-11T21:27:08.128402+0000","last_became_active":"2024-03-11T21:25:40.552216+0000","last_became_peered":"2024-03-11T21:25:40.552216+0000","last_unstale":"2024-03-11T21:27:08.128402+0000","last_undegraded":"2024-03-11T21:27:08.128402+0000","last_fullsized":"2024-03-11T21:27:08.128402+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'9","last_scrub_stamp":"2024-03-11T21:26:45.869943+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:39.525554+0000","last_clean_scrub_stamp":"2024-03-11T21:26:45.869943+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:27:50.731066+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":57,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-03-11T21:27:08.324572+0000","last_change":"2024-03-11T21:25:44.590541+0000","last_active":"2024-03-11T21:27:08.324572+0000","last_peered":"2024-03-11T21:27:08.324572+0000","last_clean":"2024-03-11T21:27:08.324572+0000","last_became_active":"2024-03-11T21:25:44.590234+0000","last_became_peered":"2024-03-11T21:25:44.590234+0000","last_unstale":"2024-03-11T21:27:08.324572+0000","last_undegraded":"2024-03-11T21:27:08.324572+0000","last_fullsized":"2024-03-11T21:27:08.324572+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:25:43.559452+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:43.559452+0000","last_clean_scrub_stamp":"2024-03-11T21:25:43.559452+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3],"acting":[5,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":61,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-03-11T21:27:08.323954+0000","last_change":"2024-03-11T21:25:42.580998+0000","last_active":"2024-03-11T21:27:08.323954+0000","last_peered":"2024-03-11T21:27:08.323954+0000","last_clean":"2024-03-11T21:27:08.323954+0000","last_became_active":"2024-03-11T21:25:42.580533+0000","last_became_peered":"2024-03-11T21:25:42.580533+0000","last_unstale":"2024-03-11T21:27:08.323954+0000","last_undegraded":"2024-03-11T21:27:08.323954+0000","last_fullsized":"2024-03-11T21:27:08.323954+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:25:41.544550+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:41.544550+0000","last_clean_scrub_stamp":"2024-03-11T21:25:41.544550+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.1d","version":"0'0","reported_seq":77,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-03-11T21:27:08.325188+0000","last_change":"2024-03-11T21:26:56.370911+0000","last_active":"2024-03-11T21:27:08.325188+0000","last_peered":"2024-03-11T21:27:08.325188+0000","last_clean":"2024-03-11T21:27:08.325188+0000","last_became_active":"2024-03-11T21:25:38.542628+0000","last_became_peered":"2024-03-11T21:25:38.542628+0000","last_unstale":"2024-03-11T21:27:08.325188+0000","last_undegraded":"2024-03-11T21:27:08.325188+0000","last_fullsized":"2024-03-11T21:27:08.325188+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:26:56.370811+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:37.512018+0000","last_clean_scrub_stamp":"2024-03-11T21:26:56.370811+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:28:03.069431+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,0],"acting":[7,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.1b","version":"74'5","reported_seq":75,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-03-11T21:27:08.128857+0000","last_change":"2024-03-11T21:25:40.556136+0000","last_active":"2024-03-11T21:27:08.128857+0000","last_peered":"2024-03-11T21:27:08.128857+0000","last_clean":"2024-03-11T21:27:08.128857+0000","last_became_active":"2024-03-11T21:25:40.555826+0000","last_became_peered":"2024-03-11T21:25:40.555826+0000","last_unstale":"2024-03-11T21:27:08.128857+0000","last_undegraded":"2024-03-11T21:27:08.128857+0000","last_fullsized":"2024-03-11T21:27:08.128857+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:25:39.525554+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:39.525554+0000","last_clean_scrub_stamp":"2024-03-11T21:25:39.525554+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:27:08.490979+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":11,"num_read_kb":7,"num_write":6,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,1],"acting":[4,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":57,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-03-11T21:27:08.130225+0000","last_change":"2024-03-11T21:25:44.590228+0000","last_active":"2024-03-11T21:27:08.130225+0000","last_peered":"2024-03-11T21:27:08.130225+0000","last_clean":"2024-03-11T21:27:08.130225+0000","last_became_active":"2024-03-11T21:25:44.589786+0000","last_became_peered":"2024-03-11T21:25:44.589786+0000","last_unstale":"2024-03-11T21:27:08.130225+0000","last_undegraded":"2024-03-11T21:27:08.130225+0000","last_fullsized":"2024-03-11T21:27:08.130225+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:25:43.559452+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:43.559452+0000","last_clean_scrub_stamp":"2024-03-11T21:25:43.559452+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:27:12.142655+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,7],"acting":[0,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":61,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-03-11T21:27:08.324760+0000","last_change":"2024-03-11T21:25:42.574391+0000","last_active":"2024-03-11T21:27:08.324760+0000","last_peered":"2024-03-11T21:27:08.324760+0000","last_clean":"2024-03-11T21:27:08.324760+0000","last_became_active":"2024-03-11T21:25:42.573831+0000","last_became_peered":"2024-03-11T21:25:42.573831+0000","last_unstale":"2024-03-11T21:27:08.324760+0000","last_undegraded":"2024-03-11T21:27:08.324760+0000","last_fullsized":"2024-03-11T21:27:08.324760+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:25:41.544550+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:41.544550+0000","last_clean_scrub_stamp":"2024-03-11T21:25:41.544550+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,7],"acting":[5,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1c","version":"0'0","reported_seq":77,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-03-11T21:27:08.324632+0000","last_change":"2024-03-11T21:27:04.375295+0000","last_active":"2024-03-11T21:27:08.324632+0000","last_peered":"2024-03-11T21:27:08.324632+0000","last_clean":"2024-03-11T21:27:08.324632+0000","last_became_active":"2024-03-11T21:25:38.541364+0000","last_became_peered":"2024-03-11T21:25:38.541364+0000","last_unstale":"2024-03-11T21:27:08.324632+0000","last_undegraded":"2024-03-11T21:27:08.324632+0000","last_fullsized":"2024-03-11T21:27:08.324632+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:27:04.375194+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:37.512018+0000","last_clean_scrub_stamp":"2024-03-11T21:27:04.375194+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:28:17.432840+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.1a","version":"74'9","reported_seq":89,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-03-11T21:27:08.128522+0000","last_change":"2024-03-11T21:26:44.851070+0000","last_active":"2024-03-11T21:27:08.128522+0000","last_peered":"2024-03-11T21:27:08.128522+0000","last_clean":"2024-03-11T21:27:08.128522+0000","last_became_active":"2024-03-11T21:25:40.556629+0000","last_became_peered":"2024-03-11T21:25:40.556629+0000","last_unstale":"2024-03-11T21:27:08.128522+0000","last_undegraded":"2024-03-11T21:27:08.128522+0000","last_fullsized":"2024-03-11T21:27:08.128522+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'9","last_scrub_stamp":"2024-03-11T21:26:44.850930+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:39.525554+0000","last_clean_scrub_stamp":"2024-03-11T21:26:44.850930+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:27:47.017552+0000","scrub_duration":24,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0],"acting":[4,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1f","version":"0'0","reported_seq":65,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-03-11T21:27:08.132058+0000","last_change":"2024-03-11T21:27:04.620314+0000","last_active":"2024-03-11T21:27:08.132058+0000","last_peered":"2024-03-11T21:27:08.132058+0000","last_clean":"2024-03-11T21:27:08.132058+0000","last_became_active":"2024-03-11T21:25:45.302637+0000","last_became_peered":"2024-03-11T21:25:45.302637+0000","last_unstale":"2024-03-11T21:27:08.132058+0000","last_undegraded":"2024-03-11T21:27:08.132058+0000","last_fullsized":"2024-03-11T21:27:08.132058+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:27:04.620217+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:43.559452+0000","last_clean_scrub_stamp":"2024-03-11T21:27:04.620217+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:28:12.483514+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1c","version":"0'0","reported_seq":69,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-03-11T21:27:10.787918+0000","last_change":"2024-03-11T21:27:10.787962+0000","last_active":"2024-03-11T21:27:10.787918+0000","last_peered":"2024-03-11T21:27:10.787918+0000","last_clean":"2024-03-11T21:27:10.787918+0000","last_became_active":"2024-03-11T21:25:42.576064+0000","last_became_peered":"2024-03-11T21:25:42.576064+0000","last_unstale":"2024-03-11T21:27:10.787918+0000","last_undegraded":"2024-03-11T21:27:10.787918+0000","last_fullsized":"2024-03-11T21:27:10.787918+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:27:10.787826+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:41.544550+0000","last_clean_scrub_stamp":"2024-03-11T21:27:10.787826+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:28:32.419330+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1b","version":"0'0","reported_seq":77,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-03-11T21:27:08.131797+0000","last_change":"2024-03-11T21:26:55.579982+0000","last_active":"2024-03-11T21:27:08.131797+0000","last_peered":"2024-03-11T21:27:08.131797+0000","last_clean":"2024-03-11T21:27:08.131797+0000","last_became_active":"2024-03-11T21:25:38.537520+0000","last_became_peered":"2024-03-11T21:25:38.537520+0000","last_unstale":"2024-03-11T21:27:08.131797+0000","last_undegraded":"2024-03-11T21:27:08.131797+0000","last_fullsized":"2024-03-11T21:27:08.131797+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:26:55.579901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:37.512018+0000","last_clean_scrub_stamp":"2024-03-11T21:26:55.579901+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:28:17.884453+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1d","version":"74'12","reported_seq":96,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-03-11T21:27:08.130937+0000","last_change":"2024-03-11T21:27:01.629574+0000","last_active":"2024-03-11T21:27:08.130937+0000","last_peered":"2024-03-11T21:27:08.130937+0000","last_clean":"2024-03-11T21:27:08.130937+0000","last_became_active":"2024-03-11T21:25:40.556330+0000","last_became_peered":"2024-03-11T21:25:40.556330+0000","last_unstale":"2024-03-11T21:27:08.130937+0000","last_undegraded":"2024-03-11T21:27:08.130937+0000","last_fullsized":"2024-03-11T21:27:08.130937+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'12","last_scrub_stamp":"2024-03-11T21:27:01.629471+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:39.525554+0000","last_clean_scrub_stamp":"2024-03-11T21:27:01.629471+0000","objects_scrubbed":5,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:28:26.432819+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":25,"num_read_kb":16,"num_write":14,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.1e","version":"0'0","reported_seq":57,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-03-11T21:27:08.129202+0000","last_change":"2024-03-11T21:25:45.300689+0000","last_active":"2024-03-11T21:27:08.129202+0000","last_peered":"2024-03-11T21:27:08.129202+0000","last_clean":"2024-03-11T21:27:08.129202+0000","last_became_active":"2024-03-11T21:25:45.300355+0000","last_became_peered":"2024-03-11T21:25:45.300355+0000","last_unstale":"2024-03-11T21:27:08.129202+0000","last_undegraded":"2024-03-11T21:27:08.129202+0000","last_fullsized":"2024-03-11T21:27:08.129202+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:25:43.559452+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:43.559452+0000","last_clean_scrub_stamp":"2024-03-11T21:25:43.559452+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5],"acting":[4,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1d","version":"0'0","reported_seq":69,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-03-11T21:27:09.199570+0000","last_change":"2024-03-11T21:27:09.199590+0000","last_active":"2024-03-11T21:27:09.199570+0000","last_peered":"2024-03-11T21:27:09.199570+0000","last_clean":"2024-03-11T21:27:09.199570+0000","last_became_active":"2024-03-11T21:25:42.579407+0000","last_became_peered":"2024-03-11T21:25:42.579407+0000","last_unstale":"2024-03-11T21:27:09.199570+0000","last_undegraded":"2024-03-11T21:27:09.199570+0000","last_fullsized":"2024-03-11T21:27:09.199570+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:27:09.199513+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:27:09.199513+0000","last_clean_scrub_stamp":"2024-03-11T21:27:09.199513+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:28:28.109684+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1a","version":"0'0","reported_seq":77,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-03-11T21:27:08.323937+0000","last_change":"2024-03-11T21:27:04.037030+0000","last_active":"2024-03-11T21:27:08.323937+0000","last_peered":"2024-03-11T21:27:08.323937+0000","last_clean":"2024-03-11T21:27:08.323937+0000","last_became_active":"2024-03-11T21:25:38.539725+0000","last_became_peered":"2024-03-11T21:25:38.539725+0000","last_unstale":"2024-03-11T21:27:08.323937+0000","last_undegraded":"2024-03-11T21:27:08.323937+0000","last_fullsized":"2024-03-11T21:27:08.323937+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:27:04.036903+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:37.512018+0000","last_clean_scrub_stamp":"2024-03-11T21:27:04.036903+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:28:11.356799+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.1c","version":"74'15","reported_seq":98,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-03-11T21:27:08.172490+0000","last_change":"2024-03-11T21:27:04.045043+0000","last_active":"2024-03-11T21:27:08.172490+0000","last_peered":"2024-03-11T21:27:08.172490+0000","last_clean":"2024-03-11T21:27:08.172490+0000","last_became_active":"2024-03-11T21:25:40.551453+0000","last_became_peered":"2024-03-11T21:25:40.551453+0000","last_unstale":"2024-03-11T21:27:08.172490+0000","last_undegraded":"2024-03-11T21:27:08.172490+0000","last_fullsized":"2024-03-11T21:27:08.172490+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'15","last_scrub_stamp":"2024-03-11T21:27:04.044946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:39.525554+0000","last_clean_scrub_stamp":"2024-03-11T21:27:04.044946+0000","objects_scrubbed":7,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:28:23.430891+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.1d","version":"0'0","reported_seq":57,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-03-11T21:27:08.171181+0000","last_change":"2024-03-11T21:25:44.589198+0000","last_active":"2024-03-11T21:27:08.171181+0000","last_peered":"2024-03-11T21:27:08.171181+0000","last_clean":"2024-03-11T21:27:08.171181+0000","last_became_active":"2024-03-11T21:25:44.588845+0000","last_became_peered":"2024-03-11T21:25:44.588845+0000","last_unstale":"2024-03-11T21:27:08.171181+0000","last_undegraded":"2024-03-11T21:27:08.171181+0000","last_fullsized":"2024-03-11T21:27:08.171181+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:25:43.559452+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:43.559452+0000","last_clean_scrub_stamp":"2024-03-11T21:25:43.559452+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:27:08.822378+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.1e","version":"0'0","reported_seq":69,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-03-11T21:27:08.131012+0000","last_change":"2024-03-11T21:26:58.335515+0000","last_active":"2024-03-11T21:27:08.131012+0000","last_peered":"2024-03-11T21:27:08.131012+0000","last_clean":"2024-03-11T21:27:08.131012+0000","last_became_active":"2024-03-11T21:25:42.582959+0000","last_became_peered":"2024-03-11T21:25:42.582959+0000","last_unstale":"2024-03-11T21:27:08.131012+0000","last_undegraded":"2024-03-11T21:27:08.131012+0000","last_fullsized":"2024-03-11T21:27:08.131012+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:26:58.335431+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:26:58.335431+0000","last_clean_scrub_stamp":"2024-03-11T21:26:58.335431+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:28:08.893453+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.19","version":"67'2","reported_seq":76,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-03-11T21:27:08.131626+0000","last_change":"2024-03-11T21:25:38.539600+0000","last_active":"2024-03-11T21:27:08.131626+0000","last_peered":"2024-03-11T21:27:08.131626+0000","last_clean":"2024-03-11T21:27:08.131626+0000","last_became_active":"2024-03-11T21:25:38.539243+0000","last_became_peered":"2024-03-11T21:25:38.539243+0000","last_unstale":"2024-03-11T21:27:08.131626+0000","last_undegraded":"2024-03-11T21:27:08.131626+0000","last_fullsized":"2024-03-11T21:27:08.131626+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:25:37.512018+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:37.512018+0000","last_clean_scrub_stamp":"2024-03-11T21:25:37.512018+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":957,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":7,"num_read_kb":7,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,0],"acting":[3,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1f","version":"74'11","reported_seq":92,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-03-11T21:27:08.324475+0000","last_change":"2024-03-11T21:26:55.050980+0000","last_active":"2024-03-11T21:27:08.324475+0000","last_peered":"2024-03-11T21:27:08.324475+0000","last_clean":"2024-03-11T21:27:08.324475+0000","last_became_active":"2024-03-11T21:25:40.554686+0000","last_became_peered":"2024-03-11T21:25:40.554686+0000","last_unstale":"2024-03-11T21:27:08.324475+0000","last_undegraded":"2024-03-11T21:27:08.324475+0000","last_fullsized":"2024-03-11T21:27:08.324475+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'11","last_scrub_stamp":"2024-03-11T21:26:55.050872+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:39.525554+0000","last_clean_scrub_stamp":"2024-03-11T21:26:55.050872+0000","objects_scrubbed":5,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:28:13.716793+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,1],"acting":[6,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.1c","version":"74'1","reported_seq":58,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-03-11T21:27:08.325020+0000","last_change":"2024-03-11T21:25:44.593623+0000","last_active":"2024-03-11T21:27:08.325020+0000","last_peered":"2024-03-11T21:27:08.325020+0000","last_clean":"2024-03-11T21:27:08.325020+0000","last_became_active":"2024-03-11T21:25:44.593326+0000","last_became_peered":"2024-03-11T21:25:44.593326+0000","last_unstale":"2024-03-11T21:27:08.325020+0000","last_undegraded":"2024-03-11T21:27:08.325020+0000","last_fullsized":"2024-03-11T21:27:08.325020+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:25:43.559452+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:43.559452+0000","last_clean_scrub_stamp":"2024-03-11T21:25:43.559452+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":370,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.1f","version":"74'8","reported_seq":99,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-03-11T21:27:08.324628+0000","last_change":"2024-03-11T21:26:48.955663+0000","last_active":"2024-03-11T21:27:08.324628+0000","last_peered":"2024-03-11T21:27:08.324628+0000","last_clean":"2024-03-11T21:27:08.324628+0000","last_became_active":"2024-03-11T21:25:42.577401+0000","last_became_peered":"2024-03-11T21:25:42.577401+0000","last_unstale":"2024-03-11T21:27:08.324628+0000","last_undegraded":"2024-03-11T21:27:08.324628+0000","last_fullsized":"2024-03-11T21:27:08.324628+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'8","last_scrub_stamp":"2024-03-11T21:26:48.955484+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:41.544550+0000","last_clean_scrub_stamp":"2024-03-11T21:26:48.955484+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:27:51.651922+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.18","version":"67'1","reported_seq":83,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-03-11T21:27:08.324447+0000","last_change":"2024-03-11T21:26:46.995375+0000","last_active":"2024-03-11T21:27:08.324447+0000","last_peered":"2024-03-11T21:27:08.324447+0000","last_clean":"2024-03-11T21:27:08.324447+0000","last_became_active":"2024-03-11T21:25:38.538321+0000","last_became_peered":"2024-03-11T21:25:38.538321+0000","last_unstale":"2024-03-11T21:27:08.324447+0000","last_undegraded":"2024-03-11T21:27:08.324447+0000","last_fullsized":"2024-03-11T21:27:08.324447+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'1","last_scrub_stamp":"2024-03-11T21:26:46.995228+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:37.512018+0000","last_clean_scrub_stamp":"2024-03-11T21:26:46.995228+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:27:58.922326+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":436,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":7,"num_read_kb":7,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,7],"acting":[5,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1e","version":"74'10","reported_seq":88,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-03-11T21:27:08.130910+0000","last_change":"2024-03-11T21:26:42.521121+0000","last_active":"2024-03-11T21:27:08.130910+0000","last_peered":"2024-03-11T21:27:08.130910+0000","last_clean":"2024-03-11T21:27:08.130910+0000","last_became_active":"2024-03-11T21:25:40.763438+0000","last_became_peered":"2024-03-11T21:25:40.763438+0000","last_unstale":"2024-03-11T21:27:08.130910+0000","last_undegraded":"2024-03-11T21:27:08.130910+0000","last_fullsized":"2024-03-11T21:27:08.130910+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'10","last_scrub_stamp":"2024-03-11T21:26:42.521016+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:39.525554+0000","last_clean_scrub_stamp":"2024-03-11T21:26:42.521016+0000","objects_scrubbed":5,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:28:12.491436+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.13","version":"0'0","reported_seq":65,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-03-11T21:27:08.131956+0000","last_change":"2024-03-11T21:27:03.623032+0000","last_active":"2024-03-11T21:27:08.131956+0000","last_peered":"2024-03-11T21:27:08.131956+0000","last_clean":"2024-03-11T21:27:08.131956+0000","last_became_active":"2024-03-11T21:25:45.302102+0000","last_became_peered":"2024-03-11T21:25:45.302102+0000","last_unstale":"2024-03-11T21:27:08.131956+0000","last_undegraded":"2024-03-11T21:27:08.131956+0000","last_fullsized":"2024-03-11T21:27:08.131956+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:27:03.622909+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:43.559452+0000","last_clean_scrub_stamp":"2024-03-11T21:27:03.622909+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:28:12.458954+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":61,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-03-11T21:27:08.323644+0000","last_change":"2024-03-11T21:25:42.580085+0000","last_active":"2024-03-11T21:27:08.323644+0000","last_peered":"2024-03-11T21:27:08.323644+0000","last_clean":"2024-03-11T21:27:08.323644+0000","last_became_active":"2024-03-11T21:25:42.579781+0000","last_became_peered":"2024-03-11T21:25:42.579781+0000","last_unstale":"2024-03-11T21:27:08.323644+0000","last_undegraded":"2024-03-11T21:27:08.323644+0000","last_fullsized":"2024-03-11T21:27:08.323644+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:25:41.544550+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:41.544550+0000","last_clean_scrub_stamp":"2024-03-11T21:25:41.544550+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.17","version":"0'0","reported_seq":77,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-03-11T21:27:08.324772+0000","last_change":"2024-03-11T21:26:56.020325+0000","last_active":"2024-03-11T21:27:08.324772+0000","last_peered":"2024-03-11T21:27:08.324772+0000","last_clean":"2024-03-11T21:27:08.324772+0000","last_became_active":"2024-03-11T21:25:38.540349+0000","last_became_peered":"2024-03-11T21:25:38.540349+0000","last_unstale":"2024-03-11T21:27:08.324772+0000","last_undegraded":"2024-03-11T21:27:08.324772+0000","last_fullsized":"2024-03-11T21:27:08.324772+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:26:56.020191+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:37.512018+0000","last_clean_scrub_stamp":"2024-03-11T21:26:56.020191+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:28:25.054065+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,2],"acting":[6,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.11","version":"74'11","reported_seq":84,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-03-11T21:27:08.132033+0000","last_change":"2024-03-11T21:25:40.761012+0000","last_active":"2024-03-11T21:27:08.132033+0000","last_peered":"2024-03-11T21:27:08.132033+0000","last_clean":"2024-03-11T21:27:08.132033+0000","last_became_active":"2024-03-11T21:25:40.760664+0000","last_became_peered":"2024-03-11T21:25:40.760664+0000","last_unstale":"2024-03-11T21:27:08.132033+0000","last_undegraded":"2024-03-11T21:27:08.132033+0000","last_fullsized":"2024-03-11T21:27:08.132033+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:25:39.525554+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:39.525554+0000","last_clean_scrub_stamp":"2024-03-11T21:25:39.525554+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":57,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-03-11T21:27:08.323436+0000","last_change":"2024-03-11T21:25:44.591375+0000","last_active":"2024-03-11T21:27:08.323436+0000","last_peered":"2024-03-11T21:27:08.323436+0000","last_clean":"2024-03-11T21:27:08.323436+0000","last_became_active":"2024-03-11T21:25:44.591068+0000","last_became_peered":"2024-03-11T21:25:44.591068+0000","last_unstale":"2024-03-11T21:27:08.323436+0000","last_undegraded":"2024-03-11T21:27:08.323436+0000","last_fullsized":"2024-03-11T21:27:08.323436+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:25:43.559452+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:43.559452+0000","last_clean_scrub_stamp":"2024-03-11T21:25:43.559452+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,4],"acting":[7,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":69,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-03-11T21:27:09.976834+0000","last_change":"2024-03-11T21:27:09.976879+0000","last_active":"2024-03-11T21:27:09.976834+0000","last_peered":"2024-03-11T21:27:09.976834+0000","last_clean":"2024-03-11T21:27:09.976834+0000","last_became_active":"2024-03-11T21:25:42.584148+0000","last_became_peered":"2024-03-11T21:25:42.584148+0000","last_unstale":"2024-03-11T21:27:09.976834+0000","last_undegraded":"2024-03-11T21:27:09.976834+0000","last_fullsized":"2024-03-11T21:27:09.976834+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:27:09.976747+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:27:09.976747+0000","last_clean_scrub_stamp":"2024-03-11T21:27:09.976747+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:28:32.267721+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.16","version":"0'0","reported_seq":77,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-03-11T21:27:11.938990+0000","last_change":"2024-03-11T21:27:11.939012+0000","last_active":"2024-03-11T21:27:11.938990+0000","last_peered":"2024-03-11T21:27:11.938990+0000","last_clean":"2024-03-11T21:27:11.938990+0000","last_became_active":"2024-03-11T21:25:38.539103+0000","last_became_peered":"2024-03-11T21:25:38.539103+0000","last_unstale":"2024-03-11T21:27:11.938990+0000","last_undegraded":"2024-03-11T21:27:11.938990+0000","last_fullsized":"2024-03-11T21:27:11.938990+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:27:11.938909+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:27:11.938909+0000","last_clean_scrub_stamp":"2024-03-11T21:27:11.938909+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:28:32.608220+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,2],"acting":[5,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.10","version":"94'6","reported_seq":81,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-03-11T21:27:08.130686+0000","last_change":"2024-03-11T21:26:58.601920+0000","last_active":"2024-03-11T21:27:08.130686+0000","last_peered":"2024-03-11T21:27:08.130686+0000","last_clean":"2024-03-11T21:27:08.130686+0000","last_became_active":"2024-03-11T21:25:40.552720+0000","last_became_peered":"2024-03-11T21:25:40.552720+0000","last_unstale":"2024-03-11T21:27:08.130686+0000","last_undegraded":"2024-03-11T21:27:08.130686+0000","last_fullsized":"2024-03-11T21:27:08.130686+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"94'6","last_scrub_stamp":"2024-03-11T21:26:58.601831+0000","last_deep_scrub":"94'6","last_deep_scrub_stamp":"2024-03-11T21:26:58.601831+0000","last_clean_scrub_stamp":"2024-03-11T21:26:58.601831+0000","objects_scrubbed":2,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:28:12.313486+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":6,"num_read_kb":4,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6],"acting":[3,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":65,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-03-11T21:27:08.131324+0000","last_change":"2024-03-11T21:27:05.669068+0000","last_active":"2024-03-11T21:27:08.131324+0000","last_peered":"2024-03-11T21:27:08.131324+0000","last_clean":"2024-03-11T21:27:08.131324+0000","last_became_active":"2024-03-11T21:25:44.583181+0000","last_became_peered":"2024-03-11T21:25:44.583181+0000","last_unstale":"2024-03-11T21:27:08.131324+0000","last_undegraded":"2024-03-11T21:27:08.131324+0000","last_fullsized":"2024-03-11T21:27:08.131324+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:27:05.668971+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:43.559452+0000","last_clean_scrub_stamp":"2024-03-11T21:27:05.668971+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:28:34.193069+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":61,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-03-11T21:27:08.171777+0000","last_change":"2024-03-11T21:25:42.573497+0000","last_active":"2024-03-11T21:27:08.171777+0000","last_peered":"2024-03-11T21:27:08.171777+0000","last_clean":"2024-03-11T21:27:08.171777+0000","last_became_active":"2024-03-11T21:25:42.573158+0000","last_became_peered":"2024-03-11T21:25:42.573158+0000","last_unstale":"2024-03-11T21:27:08.171777+0000","last_undegraded":"2024-03-11T21:27:08.171777+0000","last_fullsized":"2024-03-11T21:27:08.171777+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:25:41.544550+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:41.544550+0000","last_clean_scrub_stamp":"2024-03-11T21:25:41.544550+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:27:10.181203+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.15","version":"0'0","reported_seq":77,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-03-11T21:27:08.171389+0000","last_change":"2024-03-11T21:26:44.860603+0000","last_active":"2024-03-11T21:27:08.171389+0000","last_peered":"2024-03-11T21:27:08.171389+0000","last_clean":"2024-03-11T21:27:08.171389+0000","last_became_active":"2024-03-11T21:25:38.541705+0000","last_became_peered":"2024-03-11T21:25:38.541705+0000","last_unstale":"2024-03-11T21:27:08.171389+0000","last_undegraded":"2024-03-11T21:27:08.171389+0000","last_fullsized":"2024-03-11T21:27:08.171389+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:26:44.860499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:26:44.860499+0000","last_clean_scrub_stamp":"2024-03-11T21:26:44.860499+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:28:11.867124+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.13","version":"74'11","reported_seq":110,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-03-11T21:27:08.128976+0000","last_change":"2024-03-11T21:27:03.784968+0000","last_active":"2024-03-11T21:27:08.128976+0000","last_peered":"2024-03-11T21:27:08.128976+0000","last_clean":"2024-03-11T21:27:08.128976+0000","last_became_active":"2024-03-11T21:26:11.214879+0000","last_became_peered":"2024-03-11T21:26:11.214879+0000","last_unstale":"2024-03-11T21:27:08.128976+0000","last_undegraded":"2024-03-11T21:27:08.128976+0000","last_fullsized":"2024-03-11T21:27:08.128976+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'11","last_scrub_stamp":"2024-03-11T21:27:03.784851+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:39.525554+0000","last_clean_scrub_stamp":"2024-03-11T21:27:03.784851+0000","objects_scrubbed":5,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:28:14.160633+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,2],"acting":[4,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.10","version":"0'0","reported_seq":57,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-03-11T21:27:08.129839+0000","last_change":"2024-03-11T21:25:44.591915+0000","last_active":"2024-03-11T21:27:08.129839+0000","last_peered":"2024-03-11T21:27:08.129839+0000","last_clean":"2024-03-11T21:27:08.129839+0000","last_became_active":"2024-03-11T21:25:44.591189+0000","last_became_peered":"2024-03-11T21:25:44.591189+0000","last_unstale":"2024-03-11T21:27:08.129839+0000","last_undegraded":"2024-03-11T21:27:08.129839+0000","last_fullsized":"2024-03-11T21:27:08.129839+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:25:43.559452+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:43.559452+0000","last_clean_scrub_stamp":"2024-03-11T21:25:43.559452+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:27:08.613618+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,1],"acting":[0,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":69,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-03-11T21:27:08.131799+0000","last_change":"2024-03-11T21:27:06.637829+0000","last_active":"2024-03-11T21:27:08.131799+0000","last_peered":"2024-03-11T21:27:08.131799+0000","last_clean":"2024-03-11T21:27:08.131799+0000","last_became_active":"2024-03-11T21:25:42.576425+0000","last_became_peered":"2024-03-11T21:25:42.576425+0000","last_unstale":"2024-03-11T21:27:08.131799+0000","last_undegraded":"2024-03-11T21:27:08.131799+0000","last_fullsized":"2024-03-11T21:27:08.131799+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:27:06.637752+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:41.544550+0000","last_clean_scrub_stamp":"2024-03-11T21:27:06.637752+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:28:24.953840+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.14","version":"0'0","reported_seq":77,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-03-11T21:27:08.324139+0000","last_change":"2024-03-11T21:26:59.050063+0000","last_active":"2024-03-11T21:27:08.324139+0000","last_peered":"2024-03-11T21:27:08.324139+0000","last_clean":"2024-03-11T21:27:08.324139+0000","last_became_active":"2024-03-11T21:25:38.541852+0000","last_became_peered":"2024-03-11T21:25:38.541852+0000","last_unstale":"2024-03-11T21:27:08.324139+0000","last_undegraded":"2024-03-11T21:27:08.324139+0000","last_fullsized":"2024-03-11T21:27:08.324139+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:26:59.049960+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:37.512018+0000","last_clean_scrub_stamp":"2024-03-11T21:26:59.049960+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:28:08.381102+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,5],"acting":[6,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.12","version":"74'9","reported_seq":81,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-03-11T21:27:08.171676+0000","last_change":"2024-03-11T21:25:40.549164+0000","last_active":"2024-03-11T21:27:08.171676+0000","last_peered":"2024-03-11T21:27:08.171676+0000","last_clean":"2024-03-11T21:27:08.171676+0000","last_became_active":"2024-03-11T21:25:40.547082+0000","last_became_peered":"2024-03-11T21:25:40.547082+0000","last_unstale":"2024-03-11T21:27:08.171676+0000","last_undegraded":"2024-03-11T21:27:08.171676+0000","last_fullsized":"2024-03-11T21:27:08.171676+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:25:39.525554+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:39.525554+0000","last_clean_scrub_stamp":"2024-03-11T21:25:39.525554+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":65,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-03-11T21:27:09.801214+0000","last_change":"2024-03-11T21:27:09.801239+0000","last_active":"2024-03-11T21:27:09.801214+0000","last_peered":"2024-03-11T21:27:09.801214+0000","last_clean":"2024-03-11T21:27:09.801214+0000","last_became_active":"2024-03-11T21:25:44.589586+0000","last_became_peered":"2024-03-11T21:25:44.589586+0000","last_unstale":"2024-03-11T21:27:09.801214+0000","last_undegraded":"2024-03-11T21:27:09.801214+0000","last_fullsized":"2024-03-11T21:27:09.801214+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:27:09.801138+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:27:09.801138+0000","last_clean_scrub_stamp":"2024-03-11T21:27:09.801138+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:28:37.698118+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,5],"acting":[4,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.14","version":"74'8","reported_seq":88,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-03-11T21:27:08.131488+0000","last_change":"2024-03-11T21:25:42.583567+0000","last_active":"2024-03-11T21:27:08.131488+0000","last_peered":"2024-03-11T21:27:08.131488+0000","last_clean":"2024-03-11T21:27:08.131488+0000","last_became_active":"2024-03-11T21:25:42.583266+0000","last_became_peered":"2024-03-11T21:25:42.583266+0000","last_unstale":"2024-03-11T21:27:08.131488+0000","last_undegraded":"2024-03-11T21:27:08.131488+0000","last_fullsized":"2024-03-11T21:27:08.131488+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:25:41.544550+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:41.544550+0000","last_clean_scrub_stamp":"2024-03-11T21:25:41.544550+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,2],"acting":[3,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.13","version":"0'0","reported_seq":69,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-03-11T21:27:08.130432+0000","last_change":"2024-03-11T21:25:38.539722+0000","last_active":"2024-03-11T21:27:08.130432+0000","last_peered":"2024-03-11T21:27:08.130432+0000","last_clean":"2024-03-11T21:27:08.130432+0000","last_became_active":"2024-03-11T21:25:38.539467+0000","last_became_peered":"2024-03-11T21:25:38.539467+0000","last_unstale":"2024-03-11T21:27:08.130432+0000","last_undegraded":"2024-03-11T21:27:08.130432+0000","last_fullsized":"2024-03-11T21:27:08.130432+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:25:37.512018+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:37.512018+0000","last_clean_scrub_stamp":"2024-03-11T21:25:37.512018+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.15","version":"94'13","reported_seq":93,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-03-11T21:27:08.325393+0000","last_change":"2024-03-11T21:26:59.013939+0000","last_active":"2024-03-11T21:27:08.325393+0000","last_peered":"2024-03-11T21:27:08.325393+0000","last_clean":"2024-03-11T21:27:08.325393+0000","last_became_active":"2024-03-11T21:25:40.760907+0000","last_became_peered":"2024-03-11T21:25:40.760907+0000","last_unstale":"2024-03-11T21:27:08.325393+0000","last_undegraded":"2024-03-11T21:27:08.325393+0000","last_fullsized":"2024-03-11T21:27:08.325393+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"94'13","last_scrub_stamp":"2024-03-11T21:26:59.013823+0000","last_deep_scrub":"94'13","last_deep_scrub_stamp":"2024-03-11T21:26:59.013823+0000","last_clean_scrub_stamp":"2024-03-11T21:26:59.013823+0000","objects_scrubbed":4,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:28:25.819110+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.16","version":"0'0","reported_seq":65,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-03-11T21:27:08.130646+0000","last_change":"2024-03-11T21:26:57.367233+0000","last_active":"2024-03-11T21:27:08.130646+0000","last_peered":"2024-03-11T21:27:08.130646+0000","last_clean":"2024-03-11T21:27:08.130646+0000","last_became_active":"2024-03-11T21:25:44.583393+0000","last_became_peered":"2024-03-11T21:25:44.583393+0000","last_unstale":"2024-03-11T21:27:08.130646+0000","last_undegraded":"2024-03-11T21:27:08.130646+0000","last_fullsized":"2024-03-11T21:27:08.130646+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:26:57.367155+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:43.559452+0000","last_clean_scrub_stamp":"2024-03-11T21:26:57.367155+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:28:24.056062+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.15","version":"74'8","reported_seq":88,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-03-11T21:27:08.325298+0000","last_change":"2024-03-11T21:25:42.572973+0000","last_active":"2024-03-11T21:27:08.325298+0000","last_peered":"2024-03-11T21:27:08.325298+0000","last_clean":"2024-03-11T21:27:08.325298+0000","last_became_active":"2024-03-11T21:25:42.572555+0000","last_became_peered":"2024-03-11T21:25:42.572555+0000","last_unstale":"2024-03-11T21:27:08.325298+0000","last_undegraded":"2024-03-11T21:27:08.325298+0000","last_fullsized":"2024-03-11T21:27:08.325298+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:25:41.544550+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:41.544550+0000","last_clean_scrub_stamp":"2024-03-11T21:25:41.544550+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.12","version":"0'0","reported_seq":77,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-03-11T21:27:08.325254+0000","last_change":"2024-03-11T21:26:47.960353+0000","last_active":"2024-03-11T21:27:08.325254+0000","last_peered":"2024-03-11T21:27:08.325254+0000","last_clean":"2024-03-11T21:27:08.325254+0000","last_became_active":"2024-03-11T21:25:38.543962+0000","last_became_peered":"2024-03-11T21:25:38.543962+0000","last_unstale":"2024-03-11T21:27:08.325254+0000","last_undegraded":"2024-03-11T21:27:08.325254+0000","last_fullsized":"2024-03-11T21:27:08.325254+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:26:47.960265+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:37.512018+0000","last_clean_scrub_stamp":"2024-03-11T21:26:47.960265+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:28:15.602449+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.14","version":"74'10","reported_seq":80,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-03-11T21:27:08.131210+0000","last_change":"2024-03-11T21:25:40.760925+0000","last_active":"2024-03-11T21:27:08.131210+0000","last_peered":"2024-03-11T21:27:08.131210+0000","last_clean":"2024-03-11T21:27:08.131210+0000","last_became_active":"2024-03-11T21:25:40.760620+0000","last_became_peered":"2024-03-11T21:25:40.760620+0000","last_unstale":"2024-03-11T21:27:08.131210+0000","last_undegraded":"2024-03-11T21:27:08.131210+0000","last_fullsized":"2024-03-11T21:27:08.131210+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:25:39.525554+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:39.525554+0000","last_clean_scrub_stamp":"2024-03-11T21:25:39.525554+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.15","version":"0'0","reported_seq":65,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-03-11T21:27:08.325179+0000","last_change":"2024-03-11T21:27:00.378136+0000","last_active":"2024-03-11T21:27:08.325179+0000","last_peered":"2024-03-11T21:27:08.325179+0000","last_clean":"2024-03-11T21:27:08.325179+0000","last_became_active":"2024-03-11T21:25:45.302501+0000","last_became_peered":"2024-03-11T21:25:45.302501+0000","last_unstale":"2024-03-11T21:27:08.325179+0000","last_undegraded":"2024-03-11T21:27:08.325179+0000","last_fullsized":"2024-03-11T21:27:08.325179+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:27:00.377987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:43.559452+0000","last_clean_scrub_stamp":"2024-03-11T21:27:00.377987+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:28:21.911670+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":61,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-03-11T21:27:08.325522+0000","last_change":"2024-03-11T21:25:42.579268+0000","last_active":"2024-03-11T21:27:08.325522+0000","last_peered":"2024-03-11T21:27:08.325522+0000","last_clean":"2024-03-11T21:27:08.325522+0000","last_became_active":"2024-03-11T21:25:42.577351+0000","last_became_peered":"2024-03-11T21:25:42.577351+0000","last_unstale":"2024-03-11T21:27:08.325522+0000","last_undegraded":"2024-03-11T21:27:08.325522+0000","last_fullsized":"2024-03-11T21:27:08.325522+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:25:41.544550+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:41.544550+0000","last_clean_scrub_stamp":"2024-03-11T21:25:41.544550+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,1],"acting":[5,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.11","version":"0'0","reported_seq":69,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-03-11T21:27:08.323812+0000","last_change":"2024-03-11T21:25:38.538379+0000","last_active":"2024-03-11T21:27:08.323812+0000","last_peered":"2024-03-11T21:27:08.323812+0000","last_clean":"2024-03-11T21:27:08.323812+0000","last_became_active":"2024-03-11T21:25:38.537415+0000","last_became_peered":"2024-03-11T21:25:38.537415+0000","last_unstale":"2024-03-11T21:27:08.323812+0000","last_undegraded":"2024-03-11T21:27:08.323812+0000","last_fullsized":"2024-03-11T21:27:08.323812+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:25:37.512018+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:37.512018+0000","last_clean_scrub_stamp":"2024-03-11T21:25:37.512018+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.17","version":"74'6","reported_seq":88,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-03-11T21:27:08.131944+0000","last_change":"2024-03-11T21:26:11.224229+0000","last_active":"2024-03-11T21:27:08.131944+0000","last_peered":"2024-03-11T21:27:08.131944+0000","last_clean":"2024-03-11T21:27:08.131944+0000","last_became_active":"2024-03-11T21:26:11.219020+0000","last_became_peered":"2024-03-11T21:26:11.219020+0000","last_unstale":"2024-03-11T21:27:08.131944+0000","last_undegraded":"2024-03-11T21:27:08.131944+0000","last_fullsized":"2024-03-11T21:27:08.131944+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:25:39.525554+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:39.525554+0000","last_clean_scrub_stamp":"2024-03-11T21:25:39.525554+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":9,"num_read_kb":6,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,5],"acting":[3,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.14","version":"0'0","reported_seq":57,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-03-11T21:27:08.172280+0000","last_change":"2024-03-11T21:25:44.587755+0000","last_active":"2024-03-11T21:27:08.172280+0000","last_peered":"2024-03-11T21:27:08.172280+0000","last_clean":"2024-03-11T21:27:08.172280+0000","last_became_active":"2024-03-11T21:25:44.587439+0000","last_became_peered":"2024-03-11T21:25:44.587439+0000","last_unstale":"2024-03-11T21:27:08.172280+0000","last_undegraded":"2024-03-11T21:27:08.172280+0000","last_fullsized":"2024-03-11T21:27:08.172280+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:25:43.559452+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:43.559452+0000","last_clean_scrub_stamp":"2024-03-11T21:25:43.559452+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,7],"acting":[2,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":61,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-03-11T21:27:08.132061+0000","last_change":"2024-03-11T21:25:42.583802+0000","last_active":"2024-03-11T21:27:08.132061+0000","last_peered":"2024-03-11T21:27:08.132061+0000","last_clean":"2024-03-11T21:27:08.132061+0000","last_became_active":"2024-03-11T21:25:42.583514+0000","last_became_peered":"2024-03-11T21:25:42.583514+0000","last_unstale":"2024-03-11T21:27:08.132061+0000","last_undegraded":"2024-03-11T21:27:08.132061+0000","last_fullsized":"2024-03-11T21:27:08.132061+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:25:41.544550+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:41.544550+0000","last_clean_scrub_stamp":"2024-03-11T21:25:41.544550+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.10","version":"0'0","reported_seq":77,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-03-11T21:27:09.937488+0000","last_change":"2024-03-11T21:27:09.937508+0000","last_active":"2024-03-11T21:27:09.937488+0000","last_peered":"2024-03-11T21:27:09.937488+0000","last_clean":"2024-03-11T21:27:09.937488+0000","last_became_active":"2024-03-11T21:25:38.533138+0000","last_became_peered":"2024-03-11T21:25:38.533138+0000","last_unstale":"2024-03-11T21:27:09.937488+0000","last_undegraded":"2024-03-11T21:27:09.937488+0000","last_fullsized":"2024-03-11T21:27:09.937488+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:27:09.937426+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:37.512018+0000","last_clean_scrub_stamp":"2024-03-11T21:27:09.937426+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:28:23.648986+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,0],"acting":[2,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.16","version":"74'9","reported_seq":89,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-03-11T21:27:08.130746+0000","last_change":"2024-03-11T21:26:53.490093+0000","last_active":"2024-03-11T21:27:08.130746+0000","last_peered":"2024-03-11T21:27:08.130746+0000","last_clean":"2024-03-11T21:27:08.130746+0000","last_became_active":"2024-03-11T21:25:40.763434+0000","last_became_peered":"2024-03-11T21:25:40.763434+0000","last_unstale":"2024-03-11T21:27:08.130746+0000","last_undegraded":"2024-03-11T21:27:08.130746+0000","last_fullsized":"2024-03-11T21:27:08.130746+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'9","last_scrub_stamp":"2024-03-11T21:26:53.489953+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:39.525554+0000","last_clean_scrub_stamp":"2024-03-11T21:26:53.489953+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:27:55.665353+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.b","version":"0'0","reported_seq":57,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-03-11T21:27:08.131910+0000","last_change":"2024-03-11T21:25:44.590403+0000","last_active":"2024-03-11T21:27:08.131910+0000","last_peered":"2024-03-11T21:27:08.131910+0000","last_clean":"2024-03-11T21:27:08.131910+0000","last_became_active":"2024-03-11T21:25:44.590071+0000","last_became_peered":"2024-03-11T21:25:44.590071+0000","last_unstale":"2024-03-11T21:27:08.131910+0000","last_undegraded":"2024-03-11T21:27:08.131910+0000","last_fullsized":"2024-03-11T21:27:08.131910+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:25:43.559452+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:43.559452+0000","last_clean_scrub_stamp":"2024-03-11T21:25:43.559452+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.f","version":"67'2","reported_seq":81,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-03-11T21:27:08.129358+0000","last_change":"2024-03-11T21:25:38.537988+0000","last_active":"2024-03-11T21:27:08.129358+0000","last_peered":"2024-03-11T21:27:08.129358+0000","last_clean":"2024-03-11T21:27:08.129358+0000","last_became_active":"2024-03-11T21:25:38.537711+0000","last_became_peered":"2024-03-11T21:25:38.537711+0000","last_unstale":"2024-03-11T21:27:08.129358+0000","last_undegraded":"2024-03-11T21:27:08.129358+0000","last_fullsized":"2024-03-11T21:27:08.129358+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:25:37.512018+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:37.512018+0000","last_clean_scrub_stamp":"2024-03-11T21:25:37.512018+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":92,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":10,"num_read_kb":10,"num_write":4,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,7],"acting":[4,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.9","version":"74'12","reported_seq":88,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-03-11T21:27:08.128558+0000","last_change":"2024-03-11T21:25:40.552685+0000","last_active":"2024-03-11T21:27:08.128558+0000","last_peered":"2024-03-11T21:27:08.128558+0000","last_clean":"2024-03-11T21:27:08.128558+0000","last_became_active":"2024-03-11T21:25:40.552267+0000","last_became_peered":"2024-03-11T21:25:40.552267+0000","last_unstale":"2024-03-11T21:27:08.128558+0000","last_undegraded":"2024-03-11T21:27:08.128558+0000","last_fullsized":"2024-03-11T21:27:08.128558+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:25:39.525554+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:39.525554+0000","last_clean_scrub_stamp":"2024-03-11T21:25:39.525554+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":25,"num_read_kb":16,"num_write":14,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,3],"acting":[4,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":69,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-03-11T21:27:08.954218+0000","last_change":"2024-03-11T21:27:08.954258+0000","last_active":"2024-03-11T21:27:08.954218+0000","last_peered":"2024-03-11T21:27:08.954218+0000","last_clean":"2024-03-11T21:27:08.954218+0000","last_became_active":"2024-03-11T21:25:42.573987+0000","last_became_peered":"2024-03-11T21:25:42.573987+0000","last_unstale":"2024-03-11T21:27:08.954218+0000","last_undegraded":"2024-03-11T21:27:08.954218+0000","last_fullsized":"2024-03-11T21:27:08.954218+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:27:08.954143+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:41.544550+0000","last_clean_scrub_stamp":"2024-03-11T21:27:08.954143+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:28:34.087042+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.0","version":"0'0","reported_seq":57,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-03-11T21:27:08.129847+0000","last_change":"2024-03-11T21:25:44.585477+0000","last_active":"2024-03-11T21:27:08.129847+0000","last_peered":"2024-03-11T21:27:08.129847+0000","last_clean":"2024-03-11T21:27:08.129847+0000","last_became_active":"2024-03-11T21:25:44.584565+0000","last_became_peered":"2024-03-11T21:25:44.584565+0000","last_unstale":"2024-03-11T21:27:08.129847+0000","last_undegraded":"2024-03-11T21:27:08.129847+0000","last_fullsized":"2024-03-11T21:27:08.129847+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:25:43.559452+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:43.559452+0000","last_clean_scrub_stamp":"2024-03-11T21:25:43.559452+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":77,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-03-11T21:27:08.171345+0000","last_change":"2024-03-11T21:27:06.092300+0000","last_active":"2024-03-11T21:27:08.171345+0000","last_peered":"2024-03-11T21:27:08.171345+0000","last_clean":"2024-03-11T21:27:08.171345+0000","last_became_active":"2024-03-11T21:25:38.541760+0000","last_became_peered":"2024-03-11T21:25:38.541760+0000","last_unstale":"2024-03-11T21:27:08.171345+0000","last_undegraded":"2024-03-11T21:27:08.171345+0000","last_fullsized":"2024-03-11T21:27:08.171345+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:27:06.092220+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:37.512018+0000","last_clean_scrub_stamp":"2024-03-11T21:27:06.092220+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:28:22.259535+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,7],"acting":[1,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.2","version":"94'15","reported_seq":93,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-03-11T21:27:08.171743+0000","last_change":"2024-03-11T21:26:49.993206+0000","last_active":"2024-03-11T21:27:08.171743+0000","last_peered":"2024-03-11T21:27:08.171743+0000","last_clean":"2024-03-11T21:27:08.171743+0000","last_became_active":"2024-03-11T21:25:40.548833+0000","last_became_peered":"2024-03-11T21:25:40.548833+0000","last_unstale":"2024-03-11T21:27:08.171743+0000","last_undegraded":"2024-03-11T21:27:08.171743+0000","last_fullsized":"2024-03-11T21:27:08.171743+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"94'15","last_scrub_stamp":"2024-03-11T21:26:49.993060+0000","last_deep_scrub":"94'15","last_deep_scrub_stamp":"2024-03-11T21:26:49.993060+0000","last_clean_scrub_stamp":"2024-03-11T21:26:49.993060+0000","objects_scrubbed":5,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:27:59.041594+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.3","version":"74'8","reported_seq":96,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-03-11T21:27:08.130573+0000","last_change":"2024-03-11T21:27:06.321926+0000","last_active":"2024-03-11T21:27:08.130573+0000","last_peered":"2024-03-11T21:27:08.130573+0000","last_clean":"2024-03-11T21:27:08.130573+0000","last_became_active":"2024-03-11T21:25:42.578161+0000","last_became_peered":"2024-03-11T21:25:42.578161+0000","last_unstale":"2024-03-11T21:27:08.130573+0000","last_undegraded":"2024-03-11T21:27:08.130573+0000","last_fullsized":"2024-03-11T21:27:08.130573+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'8","last_scrub_stamp":"2024-03-11T21:27:06.321832+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:41.544550+0000","last_clean_scrub_stamp":"2024-03-11T21:27:06.321832+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:28:27.084331+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,5],"acting":[0,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":65,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-03-11T21:27:08.132122+0000","last_change":"2024-03-11T21:27:07.651113+0000","last_active":"2024-03-11T21:27:08.132122+0000","last_peered":"2024-03-11T21:27:08.132122+0000","last_clean":"2024-03-11T21:27:08.132122+0000","last_became_active":"2024-03-11T21:25:44.585223+0000","last_became_peered":"2024-03-11T21:25:44.585223+0000","last_unstale":"2024-03-11T21:27:08.132122+0000","last_undegraded":"2024-03-11T21:27:08.132122+0000","last_fullsized":"2024-03-11T21:27:08.132122+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:27:07.651016+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:43.559452+0000","last_clean_scrub_stamp":"2024-03-11T21:27:07.651016+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:28:14.928745+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,7],"acting":[3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":77,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-03-11T21:27:08.325557+0000","last_change":"2024-03-11T21:26:45.072161+0000","last_active":"2024-03-11T21:27:08.325557+0000","last_peered":"2024-03-11T21:27:08.325557+0000","last_clean":"2024-03-11T21:27:08.325557+0000","last_became_active":"2024-03-11T21:25:38.538540+0000","last_became_peered":"2024-03-11T21:25:38.538540+0000","last_unstale":"2024-03-11T21:27:08.325557+0000","last_undegraded":"2024-03-11T21:27:08.325557+0000","last_fullsized":"2024-03-11T21:27:08.325557+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:26:45.072058+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:37.512018+0000","last_clean_scrub_stamp":"2024-03-11T21:26:45.072058+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:27:49.711890+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.4","version":"74'28","reported_seq":140,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-03-11T21:27:08.173608+0000","last_change":"2024-03-11T21:27:08.173608+0000","last_active":"2024-03-11T21:27:08.173608+0000","last_peered":"2024-03-11T21:27:08.173608+0000","last_clean":"2024-03-11T21:27:08.173608+0000","last_became_active":"2024-03-11T21:25:40.552902+0000","last_became_peered":"2024-03-11T21:25:40.552902+0000","last_unstale":"2024-03-11T21:27:08.173608+0000","last_undegraded":"2024-03-11T21:27:08.173608+0000","last_fullsized":"2024-03-11T21:27:08.173608+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'28","last_scrub_stamp":"2024-03-11T21:27:08.173498+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:39.525554+0000","last_clean_scrub_stamp":"2024-03-11T21:27:08.173498+0000","objects_scrubbed":10,"log_size":28,"log_dups_size":0,"ondisk_log_size":28,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:28:21.301768+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":358,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":48,"num_read_kb":33,"num_write":26,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3],"acting":[1,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":61,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-03-11T21:27:08.130644+0000","last_change":"2024-03-11T21:25:42.573353+0000","last_active":"2024-03-11T21:27:08.130644+0000","last_peered":"2024-03-11T21:27:08.130644+0000","last_clean":"2024-03-11T21:27:08.130644+0000","last_became_active":"2024-03-11T21:25:42.572673+0000","last_became_peered":"2024-03-11T21:25:42.572673+0000","last_unstale":"2024-03-11T21:27:08.130644+0000","last_undegraded":"2024-03-11T21:27:08.130644+0000","last_fullsized":"2024-03-11T21:27:08.130644+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:25:41.544550+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:41.544550+0000","last_clean_scrub_stamp":"2024-03-11T21:25:41.544550+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.5","version":"0'0","reported_seq":65,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-03-11T21:27:08.325379+0000","last_change":"2024-03-11T21:27:05.338089+0000","last_active":"2024-03-11T21:27:08.325379+0000","last_peered":"2024-03-11T21:27:08.325379+0000","last_clean":"2024-03-11T21:27:08.325379+0000","last_became_active":"2024-03-11T21:25:45.302120+0000","last_became_peered":"2024-03-11T21:25:45.302120+0000","last_unstale":"2024-03-11T21:27:08.325379+0000","last_undegraded":"2024-03-11T21:27:08.325379+0000","last_fullsized":"2024-03-11T21:27:08.325379+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:27:05.337991+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:43.559452+0000","last_clean_scrub_stamp":"2024-03-11T21:27:05.337991+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:28:33.442171+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,3],"acting":[7,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.0","version":"76'5","reported_seq":175,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-03-11T21:27:10.608262+0000","last_change":"2024-03-11T21:26:54.088445+0000","last_active":"2024-03-11T21:27:10.608262+0000","last_peered":"2024-03-11T21:27:10.608262+0000","last_clean":"2024-03-11T21:27:10.608262+0000","last_became_active":"2024-03-11T21:25:38.536890+0000","last_became_peered":"2024-03-11T21:25:38.536890+0000","last_unstale":"2024-03-11T21:27:10.608262+0000","last_undegraded":"2024-03-11T21:27:10.608262+0000","last_fullsized":"2024-03-11T21:27:10.608262+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"76'5","last_scrub_stamp":"2024-03-11T21:26:54.088356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:37.512018+0000","last_clean_scrub_stamp":"2024-03-11T21:26:54.088356+0000","objects_scrubbed":1,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:28:21.305217+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.000753559,"stat_sum":{"num_bytes":389,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":88,"num_read_kb":83,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,6],"acting":[1,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":77,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-03-11T21:27:08.172058+0000","last_change":"2024-03-11T21:26:48.997561+0000","last_active":"2024-03-11T21:27:08.172058+0000","last_peered":"2024-03-11T21:27:08.172058+0000","last_clean":"2024-03-11T21:27:08.172058+0000","last_became_active":"2024-03-11T21:25:38.532984+0000","last_became_peered":"2024-03-11T21:25:38.532984+0000","last_unstale":"2024-03-11T21:27:08.172058+0000","last_undegraded":"2024-03-11T21:27:08.172058+0000","last_fullsized":"2024-03-11T21:27:08.172058+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:26:48.997454+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:26:48.997454+0000","last_clean_scrub_stamp":"2024-03-11T21:26:48.997454+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:28:14.373168+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.7","version":"74'13","reported_seq":92,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-03-11T21:27:08.171441+0000","last_change":"2024-03-11T21:25:40.542117+0000","last_active":"2024-03-11T21:27:08.171441+0000","last_peered":"2024-03-11T21:27:08.171441+0000","last_clean":"2024-03-11T21:27:08.171441+0000","last_became_active":"2024-03-11T21:25:40.541769+0000","last_became_peered":"2024-03-11T21:25:40.541769+0000","last_unstale":"2024-03-11T21:27:08.171441+0000","last_undegraded":"2024-03-11T21:27:08.171441+0000","last_fullsized":"2024-03-11T21:27:08.171441+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:25:39.525554+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:39.525554+0000","last_clean_scrub_stamp":"2024-03-11T21:25:39.525554+0000","objects_scrubbed":0,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":19,"num_write":16,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":69,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-03-11T21:27:08.172410+0000","last_change":"2024-03-11T21:27:02.039134+0000","last_active":"2024-03-11T21:27:08.172410+0000","last_peered":"2024-03-11T21:27:08.172410+0000","last_clean":"2024-03-11T21:27:08.172410+0000","last_became_active":"2024-03-11T21:25:42.581710+0000","last_became_peered":"2024-03-11T21:25:42.581710+0000","last_unstale":"2024-03-11T21:27:08.172410+0000","last_undegraded":"2024-03-11T21:27:08.172410+0000","last_fullsized":"2024-03-11T21:27:08.172410+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:27:02.039041+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:41.544550+0000","last_clean_scrub_stamp":"2024-03-11T21:27:02.039041+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:28:26.912856+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7],"acting":[2,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.7","version":"74'1","reported_seq":58,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-03-11T21:27:08.325779+0000","last_change":"2024-03-11T21:25:44.591645+0000","last_active":"2024-03-11T21:27:08.325779+0000","last_peered":"2024-03-11T21:27:08.325779+0000","last_clean":"2024-03-11T21:27:08.325779+0000","last_became_active":"2024-03-11T21:25:44.591376+0000","last_became_peered":"2024-03-11T21:25:44.591376+0000","last_unstale":"2024-03-11T21:27:08.325779+0000","last_undegraded":"2024-03-11T21:27:08.325779+0000","last_fullsized":"2024-03-11T21:27:08.325779+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:25:43.559452+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:43.559452+0000","last_clean_scrub_stamp":"2024-03-11T21:25:43.559452+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:27:09.252622+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,4],"acting":[5,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2","version":"71'2","reported_seq":86,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-03-11T21:27:08.130695+0000","last_change":"2024-03-11T21:26:42.663119+0000","last_active":"2024-03-11T21:27:08.130695+0000","last_peered":"2024-03-11T21:27:08.130695+0000","last_clean":"2024-03-11T21:27:08.130695+0000","last_became_active":"2024-03-11T21:25:38.539413+0000","last_became_peered":"2024-03-11T21:25:38.539413+0000","last_unstale":"2024-03-11T21:27:08.130695+0000","last_undegraded":"2024-03-11T21:27:08.130695+0000","last_fullsized":"2024-03-11T21:27:08.130695+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"71'2","last_scrub_stamp":"2024-03-11T21:26:42.662979+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:37.512018+0000","last_clean_scrub_stamp":"2024-03-11T21:26:42.662979+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:27:54.085660+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.0014206449999999999,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":83,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-03-11T21:27:08.324643+0000","last_change":"2024-03-11T21:26:42.171361+0000","last_active":"2024-03-11T21:27:08.324643+0000","last_peered":"2024-03-11T21:27:08.324643+0000","last_clean":"2024-03-11T21:27:08.324643+0000","last_became_active":"2024-03-11T21:26:11.216188+0000","last_became_peered":"2024-03-11T21:26:11.216188+0000","last_unstale":"2024-03-11T21:27:08.324643+0000","last_undegraded":"2024-03-11T21:27:08.324643+0000","last_fullsized":"2024-03-11T21:27:08.324643+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:26:42.171256+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:37.512018+0000","last_clean_scrub_stamp":"2024-03-11T21:26:42.171256+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:27:58.632937+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,4],"acting":[5,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"87'128","reported_seq":322,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-03-11T21:27:08.323323+0000","last_change":"2024-03-11T21:26:11.340374+0000","last_active":"2024-03-11T21:27:08.323323+0000","last_peered":"2024-03-11T21:27:08.323323+0000","last_clean":"2024-03-11T21:27:08.323323+0000","last_became_active":"2024-03-11T21:25:30.502597+0000","last_became_peered":"2024-03-11T21:25:30.502597+0000","last_unstale":"2024-03-11T21:27:08.323323+0000","last_undegraded":"2024-03-11T21:27:08.323323+0000","last_fullsized":"2024-03-11T21:27:08.323323+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'117","last_scrub_stamp":"2024-03-11T21:26:11.340228+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:24.261974+0000","last_clean_scrub_stamp":"2024-03-11T21:26:11.340228+0000","objects_scrubbed":2,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:27:32.864192+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":212,"num_read_kb":306,"num_write":179,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1771104,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.5","version":"74'16","reported_seq":106,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-03-11T21:27:08.325056+0000","last_change":"2024-03-11T21:25:40.761955+0000","last_active":"2024-03-11T21:27:08.325056+0000","last_peered":"2024-03-11T21:27:08.325056+0000","last_clean":"2024-03-11T21:27:08.325056+0000","last_became_active":"2024-03-11T21:25:40.761457+0000","last_became_peered":"2024-03-11T21:25:40.761457+0000","last_unstale":"2024-03-11T21:27:08.325056+0000","last_undegraded":"2024-03-11T21:27:08.325056+0000","last_fullsized":"2024-03-11T21:27:08.325056+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:25:39.525554+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:39.525554+0000","last_clean_scrub_stamp":"2024-03-11T21:25:39.525554+0000","objects_scrubbed":0,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":154,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":25,"num_read_kb":15,"num_write":13,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":69,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-03-11T21:27:08.324854+0000","last_change":"2024-03-11T21:26:58.350384+0000","last_active":"2024-03-11T21:27:08.324854+0000","last_peered":"2024-03-11T21:27:08.324854+0000","last_clean":"2024-03-11T21:27:08.324854+0000","last_became_active":"2024-03-11T21:25:42.586226+0000","last_became_peered":"2024-03-11T21:25:42.586226+0000","last_unstale":"2024-03-11T21:27:08.324854+0000","last_undegraded":"2024-03-11T21:27:08.324854+0000","last_fullsized":"2024-03-11T21:27:08.324854+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:26:58.350270+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:41.544550+0000","last_clean_scrub_stamp":"2024-03-11T21:26:58.350270+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:28:06.391624+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,5],"acting":[7,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.4","version":"0'0","reported_seq":65,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-03-11T21:27:08.171063+0000","last_change":"2024-03-11T21:27:00.119584+0000","last_active":"2024-03-11T21:27:08.171063+0000","last_peered":"2024-03-11T21:27:08.171063+0000","last_clean":"2024-03-11T21:27:08.171063+0000","last_became_active":"2024-03-11T21:25:44.594188+0000","last_became_peered":"2024-03-11T21:25:44.594188+0000","last_unstale":"2024-03-11T21:27:08.171063+0000","last_undegraded":"2024-03-11T21:27:08.171063+0000","last_fullsized":"2024-03-11T21:27:08.171063+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:27:00.119508+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:43.559452+0000","last_clean_scrub_stamp":"2024-03-11T21:27:00.119508+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:28:23.003619+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":84,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-03-11T21:27:08.130097+0000","last_change":"2024-03-11T21:26:38.589294+0000","last_active":"2024-03-11T21:27:08.130097+0000","last_peered":"2024-03-11T21:27:08.130097+0000","last_clean":"2024-03-11T21:27:08.130097+0000","last_became_active":"2024-03-11T21:25:38.543834+0000","last_became_peered":"2024-03-11T21:25:38.543834+0000","last_unstale":"2024-03-11T21:27:08.130097+0000","last_undegraded":"2024-03-11T21:27:08.130097+0000","last_fullsized":"2024-03-11T21:27:08.130097+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:26:38.589187+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:37.512018+0000","last_clean_scrub_stamp":"2024-03-11T21:26:38.589187+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:27:54.798633+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0.00089342499999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":77,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-03-11T21:27:08.323214+0000","last_change":"2024-03-11T21:26:51.473044+0000","last_active":"2024-03-11T21:27:08.323214+0000","last_peered":"2024-03-11T21:27:08.323214+0000","last_clean":"2024-03-11T21:27:08.323214+0000","last_became_active":"2024-03-11T21:25:38.541617+0000","last_became_peered":"2024-03-11T21:25:38.541617+0000","last_unstale":"2024-03-11T21:27:08.323214+0000","last_undegraded":"2024-03-11T21:27:08.323214+0000","last_fullsized":"2024-03-11T21:27:08.323214+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:26:51.472965+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:37.512018+0000","last_clean_scrub_stamp":"2024-03-11T21:26:51.472965+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:28:11.852536+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,0],"acting":[7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.6","version":"74'12","reported_seq":91,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-03-11T21:27:08.130695+0000","last_change":"2024-03-11T21:27:01.353450+0000","last_active":"2024-03-11T21:27:08.130695+0000","last_peered":"2024-03-11T21:27:08.130695+0000","last_clean":"2024-03-11T21:27:08.130695+0000","last_became_active":"2024-03-11T21:25:40.550388+0000","last_became_peered":"2024-03-11T21:25:40.550388+0000","last_unstale":"2024-03-11T21:27:08.130695+0000","last_undegraded":"2024-03-11T21:27:08.130695+0000","last_fullsized":"2024-03-11T21:27:08.130695+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'12","last_scrub_stamp":"2024-03-11T21:27:01.353347+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:39.525554+0000","last_clean_scrub_stamp":"2024-03-11T21:27:01.353347+0000","objects_scrubbed":6,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:28:25.846115+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":18,"num_read_kb":12,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":61,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-03-11T21:27:08.325755+0000","last_change":"2024-03-11T21:25:42.572970+0000","last_active":"2024-03-11T21:27:08.325755+0000","last_peered":"2024-03-11T21:27:08.325755+0000","last_clean":"2024-03-11T21:27:08.325755+0000","last_became_active":"2024-03-11T21:25:42.572324+0000","last_became_peered":"2024-03-11T21:25:42.572324+0000","last_unstale":"2024-03-11T21:27:08.325755+0000","last_undegraded":"2024-03-11T21:27:08.325755+0000","last_fullsized":"2024-03-11T21:27:08.325755+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:25:41.544550+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:41.544550+0000","last_clean_scrub_stamp":"2024-03-11T21:25:41.544550+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":65,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-03-11T21:27:08.171154+0000","last_change":"2024-03-11T21:26:59.108203+0000","last_active":"2024-03-11T21:27:08.171154+0000","last_peered":"2024-03-11T21:27:08.171154+0000","last_clean":"2024-03-11T21:27:08.171154+0000","last_became_active":"2024-03-11T21:25:45.301416+0000","last_became_peered":"2024-03-11T21:25:45.301416+0000","last_unstale":"2024-03-11T21:27:08.171154+0000","last_undegraded":"2024-03-11T21:27:08.171154+0000","last_fullsized":"2024-03-11T21:27:08.171154+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:26:59.108111+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:43.559452+0000","last_clean_scrub_stamp":"2024-03-11T21:26:59.108111+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:28:03.778211+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":77,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-03-11T21:27:08.325481+0000","last_change":"2024-03-11T21:26:57.333768+0000","last_active":"2024-03-11T21:27:08.325481+0000","last_peered":"2024-03-11T21:27:08.325481+0000","last_clean":"2024-03-11T21:27:08.325481+0000","last_became_active":"2024-03-11T21:25:38.545148+0000","last_became_peered":"2024-03-11T21:25:38.545148+0000","last_unstale":"2024-03-11T21:27:08.325481+0000","last_undegraded":"2024-03-11T21:27:08.325481+0000","last_fullsized":"2024-03-11T21:27:08.325481+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:26:57.333669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:26:57.333669+0000","last_clean_scrub_stamp":"2024-03-11T21:26:57.333669+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:28:23.923732+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.3","version":"74'19","reported_seq":101,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-03-11T21:27:08.130419+0000","last_change":"2024-03-11T21:25:40.760898+0000","last_active":"2024-03-11T21:27:08.130419+0000","last_peered":"2024-03-11T21:27:08.130419+0000","last_clean":"2024-03-11T21:27:08.130419+0000","last_became_active":"2024-03-11T21:25:40.760594+0000","last_became_peered":"2024-03-11T21:25:40.760594+0000","last_unstale":"2024-03-11T21:27:08.130419+0000","last_undegraded":"2024-03-11T21:27:08.130419+0000","last_fullsized":"2024-03-11T21:27:08.130419+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:25:39.525554+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:39.525554+0000","last_clean_scrub_stamp":"2024-03-11T21:25:39.525554+0000","objects_scrubbed":0,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":39,"num_read_kb":25,"num_write":22,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":69,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-03-11T21:27:08.323903+0000","last_change":"2024-03-11T21:27:05.035735+0000","last_active":"2024-03-11T21:27:08.323903+0000","last_peered":"2024-03-11T21:27:08.323903+0000","last_clean":"2024-03-11T21:27:08.323903+0000","last_became_active":"2024-03-11T21:25:42.579045+0000","last_became_peered":"2024-03-11T21:25:42.579045+0000","last_unstale":"2024-03-11T21:27:08.323903+0000","last_undegraded":"2024-03-11T21:27:08.323903+0000","last_fullsized":"2024-03-11T21:27:08.323903+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:27:05.035647+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:41.544550+0000","last_clean_scrub_stamp":"2024-03-11T21:27:05.035647+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:28:32.345929+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":65,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-03-11T21:27:08.128755+0000","last_change":"2024-03-11T21:26:50.836566+0000","last_active":"2024-03-11T21:27:08.128755+0000","last_peered":"2024-03-11T21:27:08.128755+0000","last_clean":"2024-03-11T21:27:08.128755+0000","last_became_active":"2024-03-11T21:25:44.583638+0000","last_became_peered":"2024-03-11T21:25:44.583638+0000","last_unstale":"2024-03-11T21:27:08.128755+0000","last_undegraded":"2024-03-11T21:27:08.128755+0000","last_fullsized":"2024-03-11T21:27:08.128755+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:26:50.836479+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:43.559452+0000","last_clean_scrub_stamp":"2024-03-11T21:26:50.836479+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:28:03.817969+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.6","version":"67'1","reported_seq":78,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-03-11T21:27:08.171533+0000","last_change":"2024-03-11T21:27:07.129584+0000","last_active":"2024-03-11T21:27:08.171533+0000","last_peered":"2024-03-11T21:27:08.171533+0000","last_clean":"2024-03-11T21:27:08.171533+0000","last_became_active":"2024-03-11T21:25:38.536142+0000","last_became_peered":"2024-03-11T21:25:38.536142+0000","last_unstale":"2024-03-11T21:27:08.171533+0000","last_undegraded":"2024-03-11T21:27:08.171533+0000","last_fullsized":"2024-03-11T21:27:08.171533+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'1","last_scrub_stamp":"2024-03-11T21:27:07.129494+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:37.512018+0000","last_clean_scrub_stamp":"2024-03-11T21:27:07.129494+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:28:12.668804+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,4],"acting":[1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.0","version":"93'26","reported_seq":113,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-03-11T21:27:08.130527+0000","last_change":"2024-03-11T21:26:40.680446+0000","last_active":"2024-03-11T21:27:08.130527+0000","last_peered":"2024-03-11T21:27:08.130527+0000","last_clean":"2024-03-11T21:27:08.130527+0000","last_became_active":"2024-03-11T21:25:40.761716+0000","last_became_peered":"2024-03-11T21:25:40.761716+0000","last_unstale":"2024-03-11T21:27:08.130527+0000","last_undegraded":"2024-03-11T21:27:08.130527+0000","last_fullsized":"2024-03-11T21:27:08.130527+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"93'26","last_scrub_stamp":"2024-03-11T21:26:40.680324+0000","last_deep_scrub":"93'26","last_deep_scrub_stamp":"2024-03-11T21:26:40.680324+0000","last_clean_scrub_stamp":"2024-03-11T21:26:40.680324+0000","objects_scrubbed":8,"log_size":26,"log_dups_size":0,"ondisk_log_size":26,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:28:00.090900+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":34,"num_read_kb":22,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":69,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-03-11T21:27:08.128602+0000","last_change":"2024-03-11T21:26:51.863702+0000","last_active":"2024-03-11T21:27:08.128602+0000","last_peered":"2024-03-11T21:27:08.128602+0000","last_clean":"2024-03-11T21:27:08.128602+0000","last_became_active":"2024-03-11T21:25:42.583647+0000","last_became_peered":"2024-03-11T21:25:42.583647+0000","last_unstale":"2024-03-11T21:27:08.128602+0000","last_undegraded":"2024-03-11T21:27:08.128602+0000","last_fullsized":"2024-03-11T21:27:08.128602+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:26:51.863621+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:26:51.863621+0000","last_clean_scrub_stamp":"2024-03-11T21:26:51.863621+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:27:53.832869+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,7],"acting":[4,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.3","version":"0'0","reported_seq":65,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-03-11T21:27:08.324258+0000","last_change":"2024-03-11T21:26:54.421277+0000","last_active":"2024-03-11T21:27:08.324258+0000","last_peered":"2024-03-11T21:27:08.324258+0000","last_clean":"2024-03-11T21:27:08.324258+0000","last_became_active":"2024-03-11T21:25:45.302719+0000","last_became_peered":"2024-03-11T21:25:45.302719+0000","last_unstale":"2024-03-11T21:27:08.324258+0000","last_undegraded":"2024-03-11T21:27:08.324258+0000","last_fullsized":"2024-03-11T21:27:08.324258+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:26:54.421196+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:43.559452+0000","last_clean_scrub_stamp":"2024-03-11T21:26:54.421196+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:28:15.162446+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,2],"acting":[7,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":77,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-03-11T21:27:08.324682+0000","last_change":"2024-03-11T21:26:47.927498+0000","last_active":"2024-03-11T21:27:08.324682+0000","last_peered":"2024-03-11T21:27:08.324682+0000","last_clean":"2024-03-11T21:27:08.324682+0000","last_became_active":"2024-03-11T21:25:38.540013+0000","last_became_peered":"2024-03-11T21:25:38.540013+0000","last_unstale":"2024-03-11T21:27:08.324682+0000","last_undegraded":"2024-03-11T21:27:08.324682+0000","last_fullsized":"2024-03-11T21:27:08.324682+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:26:47.927394+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:37.512018+0000","last_clean_scrub_stamp":"2024-03-11T21:26:47.927394+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:28:02.205612+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,2],"acting":[6,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.1","version":"74'14","reported_seq":86,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-03-11T21:27:08.128436+0000","last_change":"2024-03-11T21:25:40.546609+0000","last_active":"2024-03-11T21:27:08.128436+0000","last_peered":"2024-03-11T21:27:08.128436+0000","last_clean":"2024-03-11T21:27:08.128436+0000","last_became_active":"2024-03-11T21:25:40.546219+0000","last_became_peered":"2024-03-11T21:25:40.546219+0000","last_unstale":"2024-03-11T21:27:08.128436+0000","last_undegraded":"2024-03-11T21:27:08.128436+0000","last_fullsized":"2024-03-11T21:27:08.128436+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:25:39.525554+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:39.525554+0000","last_clean_scrub_stamp":"2024-03-11T21:25:39.525554+0000","objects_scrubbed":0,"log_size":14,"log_dups_size":0,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":21,"num_read_kb":14,"num_write":14,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6],"acting":[4,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.0","version":"74'8","reported_seq":88,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-03-11T21:27:08.130471+0000","last_change":"2024-03-11T21:25:42.577103+0000","last_active":"2024-03-11T21:27:08.130471+0000","last_peered":"2024-03-11T21:27:08.130471+0000","last_clean":"2024-03-11T21:27:08.130471+0000","last_became_active":"2024-03-11T21:25:42.576126+0000","last_became_peered":"2024-03-11T21:25:42.576126+0000","last_unstale":"2024-03-11T21:27:08.130471+0000","last_undegraded":"2024-03-11T21:27:08.130471+0000","last_fullsized":"2024-03-11T21:27:08.130471+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:25:41.544550+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:41.544550+0000","last_clean_scrub_stamp":"2024-03-11T21:25:41.544550+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.c","version":"0'0","reported_seq":57,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-03-11T21:27:08.130926+0000","last_change":"2024-03-11T21:25:45.303235+0000","last_active":"2024-03-11T21:27:08.130926+0000","last_peered":"2024-03-11T21:27:08.130926+0000","last_clean":"2024-03-11T21:27:08.130926+0000","last_became_active":"2024-03-11T21:25:45.302207+0000","last_became_peered":"2024-03-11T21:25:45.302207+0000","last_unstale":"2024-03-11T21:27:08.130926+0000","last_undegraded":"2024-03-11T21:27:08.130926+0000","last_fullsized":"2024-03-11T21:27:08.130926+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:25:43.559452+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:43.559452+0000","last_clean_scrub_stamp":"2024-03-11T21:25:43.559452+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.f","version":"0'0","reported_seq":69,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-03-11T21:27:08.325567+0000","last_change":"2024-03-11T21:26:51.925489+0000","last_active":"2024-03-11T21:27:08.325567+0000","last_peered":"2024-03-11T21:27:08.325567+0000","last_clean":"2024-03-11T21:27:08.325567+0000","last_became_active":"2024-03-11T21:25:42.575935+0000","last_became_peered":"2024-03-11T21:25:42.575935+0000","last_unstale":"2024-03-11T21:27:08.325567+0000","last_undegraded":"2024-03-11T21:27:08.325567+0000","last_fullsized":"2024-03-11T21:27:08.325567+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:26:51.925329+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:41.544550+0000","last_clean_scrub_stamp":"2024-03-11T21:26:51.925329+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:28:02.478228+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.8","version":"0'0","reported_seq":77,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-03-11T21:27:08.323325+0000","last_change":"2024-03-11T21:26:48.424013+0000","last_active":"2024-03-11T21:27:08.323325+0000","last_peered":"2024-03-11T21:27:08.323325+0000","last_clean":"2024-03-11T21:27:08.323325+0000","last_became_active":"2024-03-11T21:25:38.540910+0000","last_became_peered":"2024-03-11T21:25:38.540910+0000","last_unstale":"2024-03-11T21:27:08.323325+0000","last_undegraded":"2024-03-11T21:27:08.323325+0000","last_fullsized":"2024-03-11T21:27:08.323325+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:26:48.423910+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:37.512018+0000","last_clean_scrub_stamp":"2024-03-11T21:26:48.423910+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:28:17.378160+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,1],"acting":[7,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.e","version":"74'11","reported_seq":92,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-03-11T21:27:08.129256+0000","last_change":"2024-03-11T21:26:57.843770+0000","last_active":"2024-03-11T21:27:08.129256+0000","last_peered":"2024-03-11T21:27:08.129256+0000","last_clean":"2024-03-11T21:27:08.129256+0000","last_became_active":"2024-03-11T21:25:40.552178+0000","last_became_peered":"2024-03-11T21:25:40.552178+0000","last_unstale":"2024-03-11T21:27:08.129256+0000","last_undegraded":"2024-03-11T21:27:08.129256+0000","last_fullsized":"2024-03-11T21:27:08.129256+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'11","last_scrub_stamp":"2024-03-11T21:26:57.843615+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:39.525554+0000","last_clean_scrub_stamp":"2024-03-11T21:26:57.843615+0000","objects_scrubbed":5,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:28:03.562699+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.d","version":"0'0","reported_seq":65,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-03-11T21:27:08.325115+0000","last_change":"2024-03-11T21:26:54.957670+0000","last_active":"2024-03-11T21:27:08.325115+0000","last_peered":"2024-03-11T21:27:08.325115+0000","last_clean":"2024-03-11T21:27:08.325115+0000","last_became_active":"2024-03-11T21:25:44.590333+0000","last_became_peered":"2024-03-11T21:25:44.590333+0000","last_unstale":"2024-03-11T21:27:08.325115+0000","last_undegraded":"2024-03-11T21:27:08.325115+0000","last_fullsized":"2024-03-11T21:27:08.325115+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:26:54.957577+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:43.559452+0000","last_clean_scrub_stamp":"2024-03-11T21:26:54.957577+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:28:05.595894+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.e","version":"74'8","reported_seq":88,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-03-11T21:27:08.129126+0000","last_change":"2024-03-11T21:25:42.577257+0000","last_active":"2024-03-11T21:27:08.129126+0000","last_peered":"2024-03-11T21:27:08.129126+0000","last_clean":"2024-03-11T21:27:08.129126+0000","last_became_active":"2024-03-11T21:25:42.576742+0000","last_became_peered":"2024-03-11T21:25:42.576742+0000","last_unstale":"2024-03-11T21:27:08.129126+0000","last_undegraded":"2024-03-11T21:27:08.129126+0000","last_fullsized":"2024-03-11T21:27:08.129126+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:25:41.544550+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:41.544550+0000","last_clean_scrub_stamp":"2024-03-11T21:25:41.544550+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,0],"acting":[4,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.9","version":"0'0","reported_seq":77,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-03-11T21:27:08.171881+0000","last_change":"2024-03-11T21:26:50.991832+0000","last_active":"2024-03-11T21:27:08.171881+0000","last_peered":"2024-03-11T21:27:08.171881+0000","last_clean":"2024-03-11T21:27:08.171881+0000","last_became_active":"2024-03-11T21:25:38.540976+0000","last_became_peered":"2024-03-11T21:25:38.540976+0000","last_unstale":"2024-03-11T21:27:08.171881+0000","last_undegraded":"2024-03-11T21:27:08.171881+0000","last_fullsized":"2024-03-11T21:27:08.171881+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:26:50.991712+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:26:50.991712+0000","last_clean_scrub_stamp":"2024-03-11T21:26:50.991712+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:27:58.101352+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,3],"acting":[1,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.f","version":"74'15","reported_seq":51,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-03-11T21:27:08.172595+0000","last_change":"2024-03-11T21:26:11.226642+0000","last_active":"2024-03-11T21:27:08.172595+0000","last_peered":"2024-03-11T21:27:08.172595+0000","last_clean":"2024-03-11T21:27:08.172595+0000","last_became_active":"2024-03-11T21:26:11.216224+0000","last_became_peered":"2024-03-11T21:26:11.216224+0000","last_unstale":"2024-03-11T21:27:08.172595+0000","last_undegraded":"2024-03-11T21:27:08.172595+0000","last_fullsized":"2024-03-11T21:27:08.172595+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:25:39.525554+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:39.525554+0000","last_clean_scrub_stamp":"2024-03-11T21:25:39.525554+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,4],"acting":[2,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.e","version":"0'0","reported_seq":57,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-03-11T21:27:08.129032+0000","last_change":"2024-03-11T21:25:44.590476+0000","last_active":"2024-03-11T21:27:08.129032+0000","last_peered":"2024-03-11T21:27:08.129032+0000","last_clean":"2024-03-11T21:27:08.129032+0000","last_became_active":"2024-03-11T21:25:44.590198+0000","last_became_peered":"2024-03-11T21:25:44.590198+0000","last_unstale":"2024-03-11T21:27:08.129032+0000","last_undegraded":"2024-03-11T21:27:08.129032+0000","last_fullsized":"2024-03-11T21:27:08.129032+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:25:43.559452+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:43.559452+0000","last_clean_scrub_stamp":"2024-03-11T21:25:43.559452+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.d","version":"74'8","reported_seq":96,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-03-11T21:27:08.172324+0000","last_change":"2024-03-11T21:26:59.021603+0000","last_active":"2024-03-11T21:27:08.172324+0000","last_peered":"2024-03-11T21:27:08.172324+0000","last_clean":"2024-03-11T21:27:08.172324+0000","last_became_active":"2024-03-11T21:25:42.575583+0000","last_became_peered":"2024-03-11T21:25:42.575583+0000","last_unstale":"2024-03-11T21:27:08.172324+0000","last_undegraded":"2024-03-11T21:27:08.172324+0000","last_fullsized":"2024-03-11T21:27:08.172324+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'8","last_scrub_stamp":"2024-03-11T21:26:59.021523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:41.544550+0000","last_clean_scrub_stamp":"2024-03-11T21:26:59.021523+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:28:21.121669+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,5],"acting":[2,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.a","version":"0'0","reported_seq":77,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-03-11T21:27:08.171644+0000","last_change":"2024-03-11T21:26:52.008094+0000","last_active":"2024-03-11T21:27:08.171644+0000","last_peered":"2024-03-11T21:27:08.171644+0000","last_clean":"2024-03-11T21:27:08.171644+0000","last_became_active":"2024-03-11T21:25:38.541045+0000","last_became_peered":"2024-03-11T21:25:38.541045+0000","last_unstale":"2024-03-11T21:27:08.171644+0000","last_undegraded":"2024-03-11T21:27:08.171644+0000","last_fullsized":"2024-03-11T21:27:08.171644+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:26:52.007996+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:37.512018+0000","last_clean_scrub_stamp":"2024-03-11T21:26:52.007996+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:27:52.862240+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,7],"acting":[1,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.c","version":"74'10","reported_seq":88,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-03-11T21:27:08.128861+0000","last_change":"2024-03-11T21:27:00.807955+0000","last_active":"2024-03-11T21:27:08.128861+0000","last_peered":"2024-03-11T21:27:08.128861+0000","last_clean":"2024-03-11T21:27:08.128861+0000","last_became_active":"2024-03-11T21:25:40.550885+0000","last_became_peered":"2024-03-11T21:25:40.550885+0000","last_unstale":"2024-03-11T21:27:08.128861+0000","last_undegraded":"2024-03-11T21:27:08.128861+0000","last_fullsized":"2024-03-11T21:27:08.128861+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'10","last_scrub_stamp":"2024-03-11T21:27:00.807849+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:39.525554+0000","last_clean_scrub_stamp":"2024-03-11T21:27:00.807849+0000","objects_scrubbed":5,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:28:27.818066+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.f","version":"0'0","reported_seq":65,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-03-11T21:27:08.172539+0000","last_change":"2024-03-11T21:27:03.011122+0000","last_active":"2024-03-11T21:27:08.172539+0000","last_peered":"2024-03-11T21:27:08.172539+0000","last_clean":"2024-03-11T21:27:08.172539+0000","last_became_active":"2024-03-11T21:25:44.593414+0000","last_became_peered":"2024-03-11T21:25:44.593414+0000","last_unstale":"2024-03-11T21:27:08.172539+0000","last_undegraded":"2024-03-11T21:27:08.172539+0000","last_fullsized":"2024-03-11T21:27:08.172539+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:27:03.010969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:27:03.010969+0000","last_clean_scrub_stamp":"2024-03-11T21:27:03.010969+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:28:22.636838+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,4],"acting":[2,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":61,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-03-11T21:27:08.171172+0000","last_change":"2024-03-11T21:25:42.579815+0000","last_active":"2024-03-11T21:27:08.171172+0000","last_peered":"2024-03-11T21:27:08.171172+0000","last_clean":"2024-03-11T21:27:08.171172+0000","last_became_active":"2024-03-11T21:25:42.579409+0000","last_became_peered":"2024-03-11T21:25:42.579409+0000","last_unstale":"2024-03-11T21:27:08.171172+0000","last_undegraded":"2024-03-11T21:27:08.171172+0000","last_fullsized":"2024-03-11T21:27:08.171172+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:25:41.544550+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:41.544550+0000","last_clean_scrub_stamp":"2024-03-11T21:25:41.544550+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:27:11.166706+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.b","version":"0'0","reported_seq":69,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-03-11T21:27:08.323669+0000","last_change":"2024-03-11T21:25:38.544246+0000","last_active":"2024-03-11T21:27:08.323669+0000","last_peered":"2024-03-11T21:27:08.323669+0000","last_clean":"2024-03-11T21:27:08.323669+0000","last_became_active":"2024-03-11T21:25:38.543377+0000","last_became_peered":"2024-03-11T21:25:38.543377+0000","last_unstale":"2024-03-11T21:27:08.323669+0000","last_undegraded":"2024-03-11T21:27:08.323669+0000","last_fullsized":"2024-03-11T21:27:08.323669+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:25:37.512018+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:37.512018+0000","last_clean_scrub_stamp":"2024-03-11T21:25:37.512018+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,5],"acting":[7,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.d","version":"74'17","reported_seq":101,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-03-11T21:27:08.129005+0000","last_change":"2024-03-11T21:27:02.816136+0000","last_active":"2024-03-11T21:27:08.129005+0000","last_peered":"2024-03-11T21:27:08.129005+0000","last_clean":"2024-03-11T21:27:08.129005+0000","last_became_active":"2024-03-11T21:25:40.553454+0000","last_became_peered":"2024-03-11T21:25:40.553454+0000","last_unstale":"2024-03-11T21:27:08.129005+0000","last_undegraded":"2024-03-11T21:27:08.129005+0000","last_fullsized":"2024-03-11T21:27:08.129005+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'17","last_scrub_stamp":"2024-03-11T21:27:02.815980+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:39.525554+0000","last_clean_scrub_stamp":"2024-03-11T21:27:02.815980+0000","objects_scrubbed":8,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:28:06.520165+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":29,"num_read_kb":19,"num_write":18,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,1],"acting":[4,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":65,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-03-11T21:27:08.323131+0000","last_change":"2024-03-11T21:27:07.304361+0000","last_active":"2024-03-11T21:27:08.323131+0000","last_peered":"2024-03-11T21:27:08.323131+0000","last_clean":"2024-03-11T21:27:08.323131+0000","last_became_active":"2024-03-11T21:25:44.589973+0000","last_became_peered":"2024-03-11T21:25:44.589973+0000","last_unstale":"2024-03-11T21:27:08.323131+0000","last_undegraded":"2024-03-11T21:27:08.323131+0000","last_fullsized":"2024-03-11T21:27:08.323131+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:27:07.304264+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:43.559452+0000","last_clean_scrub_stamp":"2024-03-11T21:27:07.304264+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:28:32.511531+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,3],"acting":[7,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":61,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-03-11T21:27:08.172239+0000","last_change":"2024-03-11T21:25:42.574877+0000","last_active":"2024-03-11T21:27:08.172239+0000","last_peered":"2024-03-11T21:27:08.172239+0000","last_clean":"2024-03-11T21:27:08.172239+0000","last_became_active":"2024-03-11T21:25:42.574515+0000","last_became_peered":"2024-03-11T21:25:42.574515+0000","last_unstale":"2024-03-11T21:27:08.172239+0000","last_undegraded":"2024-03-11T21:27:08.172239+0000","last_fullsized":"2024-03-11T21:27:08.172239+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:25:41.544550+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:41.544550+0000","last_clean_scrub_stamp":"2024-03-11T21:25:41.544550+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:27:09.339666+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,5],"acting":[2,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.c","version":"0'0","reported_seq":69,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-03-11T21:27:08.172463+0000","last_change":"2024-03-11T21:25:38.533719+0000","last_active":"2024-03-11T21:27:08.172463+0000","last_peered":"2024-03-11T21:27:08.172463+0000","last_clean":"2024-03-11T21:27:08.172463+0000","last_became_active":"2024-03-11T21:25:38.533240+0000","last_became_peered":"2024-03-11T21:25:38.533240+0000","last_unstale":"2024-03-11T21:27:08.172463+0000","last_undegraded":"2024-03-11T21:27:08.172463+0000","last_fullsized":"2024-03-11T21:27:08.172463+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:25:37.512018+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:37.512018+0000","last_clean_scrub_stamp":"2024-03-11T21:25:37.512018+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,0],"acting":[2,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.a","version":"74'19","reported_seq":104,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-03-11T21:27:08.323762+0000","last_change":"2024-03-11T21:27:07.010787+0000","last_active":"2024-03-11T21:27:08.323762+0000","last_peered":"2024-03-11T21:27:08.323762+0000","last_clean":"2024-03-11T21:27:08.323762+0000","last_became_active":"2024-03-11T21:25:40.760698+0000","last_became_peered":"2024-03-11T21:25:40.760698+0000","last_unstale":"2024-03-11T21:27:08.323762+0000","last_undegraded":"2024-03-11T21:27:08.323762+0000","last_fullsized":"2024-03-11T21:27:08.323762+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'19","last_scrub_stamp":"2024-03-11T21:27:07.010662+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:39.525554+0000","last_clean_scrub_stamp":"2024-03-11T21:27:07.010662+0000","objects_scrubbed":9,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:28:22.036849+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":32,"num_read_kb":21,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,7],"acting":[6,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":65,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-03-11T21:27:11.361984+0000","last_change":"2024-03-11T21:27:11.362017+0000","last_active":"2024-03-11T21:27:11.361984+0000","last_peered":"2024-03-11T21:27:11.361984+0000","last_clean":"2024-03-11T21:27:11.361984+0000","last_became_active":"2024-03-11T21:25:44.584154+0000","last_became_peered":"2024-03-11T21:25:44.584154+0000","last_unstale":"2024-03-11T21:27:11.361984+0000","last_undegraded":"2024-03-11T21:27:11.361984+0000","last_fullsized":"2024-03-11T21:27:11.361984+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:27:11.361913+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:27:11.361913+0000","last_clean_scrub_stamp":"2024-03-11T21:27:11.361913+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:28:25.991645+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":69,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-03-11T21:27:10.909660+0000","last_change":"2024-03-11T21:27:10.909699+0000","last_active":"2024-03-11T21:27:10.909660+0000","last_peered":"2024-03-11T21:27:10.909660+0000","last_clean":"2024-03-11T21:27:10.909660+0000","last_became_active":"2024-03-11T21:25:42.576679+0000","last_became_peered":"2024-03-11T21:25:42.576679+0000","last_unstale":"2024-03-11T21:27:10.909660+0000","last_undegraded":"2024-03-11T21:27:10.909660+0000","last_fullsized":"2024-03-11T21:27:10.909660+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:27:10.909543+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:41.544550+0000","last_clean_scrub_stamp":"2024-03-11T21:27:10.909543+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:28:29.725395+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,3],"acting":[2,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.d","version":"0'0","reported_seq":77,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-03-11T21:27:08.171571+0000","last_change":"2024-03-11T21:26:45.885219+0000","last_active":"2024-03-11T21:27:08.171571+0000","last_peered":"2024-03-11T21:27:08.171571+0000","last_clean":"2024-03-11T21:27:08.171571+0000","last_became_active":"2024-03-11T21:25:38.539681+0000","last_became_peered":"2024-03-11T21:25:38.539681+0000","last_unstale":"2024-03-11T21:27:08.171571+0000","last_undegraded":"2024-03-11T21:27:08.171571+0000","last_fullsized":"2024-03-11T21:27:08.171571+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:26:45.885144+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:37.512018+0000","last_clean_scrub_stamp":"2024-03-11T21:26:45.885144+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:27:47.637748+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,3],"acting":[1,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.b","version":"74'9","reported_seq":81,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-03-11T21:27:08.130522+0000","last_change":"2024-03-11T21:25:40.549925+0000","last_active":"2024-03-11T21:27:08.130522+0000","last_peered":"2024-03-11T21:27:08.130522+0000","last_clean":"2024-03-11T21:27:08.130522+0000","last_became_active":"2024-03-11T21:25:40.549612+0000","last_became_peered":"2024-03-11T21:25:40.549612+0000","last_unstale":"2024-03-11T21:27:08.130522+0000","last_undegraded":"2024-03-11T21:27:08.130522+0000","last_fullsized":"2024-03-11T21:27:08.130522+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:25:39.525554+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:39.525554+0000","last_clean_scrub_stamp":"2024-03-11T21:25:39.525554+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:27:08.607529+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":65,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-03-11T21:27:08.325145+0000","last_change":"2024-03-11T21:26:55.966077+0000","last_active":"2024-03-11T21:27:08.325145+0000","last_peered":"2024-03-11T21:27:08.325145+0000","last_clean":"2024-03-11T21:27:08.325145+0000","last_became_active":"2024-03-11T21:25:45.300333+0000","last_became_peered":"2024-03-11T21:25:45.300333+0000","last_unstale":"2024-03-11T21:27:08.325145+0000","last_undegraded":"2024-03-11T21:27:08.325145+0000","last_fullsized":"2024-03-11T21:27:08.325145+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:26:55.965980+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:43.559452+0000","last_clean_scrub_stamp":"2024-03-11T21:26:55.965980+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:28:02.706121+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,0],"acting":[5,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.e","version":"0'0","reported_seq":77,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-03-11T21:27:08.172516+0000","last_change":"2024-03-11T21:26:57.974916+0000","last_active":"2024-03-11T21:27:08.172516+0000","last_peered":"2024-03-11T21:27:08.172516+0000","last_clean":"2024-03-11T21:27:08.172516+0000","last_became_active":"2024-03-11T21:25:38.539954+0000","last_became_peered":"2024-03-11T21:25:38.539954+0000","last_unstale":"2024-03-11T21:27:08.172516+0000","last_undegraded":"2024-03-11T21:27:08.172516+0000","last_fullsized":"2024-03-11T21:27:08.172516+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:26:57.974827+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:37.512018+0000","last_clean_scrub_stamp":"2024-03-11T21:26:57.974827+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:28:10.300007+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,7],"acting":[2,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.8","version":"74'15","reported_seq":98,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-03-11T21:27:08.324956+0000","last_change":"2024-03-11T21:26:50.934584+0000","last_active":"2024-03-11T21:27:08.324956+0000","last_peered":"2024-03-11T21:27:08.324956+0000","last_clean":"2024-03-11T21:27:08.324956+0000","last_became_active":"2024-03-11T21:25:40.761938+0000","last_became_peered":"2024-03-11T21:25:40.761938+0000","last_unstale":"2024-03-11T21:27:08.324956+0000","last_undegraded":"2024-03-11T21:27:08.324956+0000","last_fullsized":"2024-03-11T21:27:08.324956+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'15","last_scrub_stamp":"2024-03-11T21:26:50.934489+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:39.525554+0000","last_clean_scrub_stamp":"2024-03-11T21:26:50.934489+0000","objects_scrubbed":7,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:28:08.719408+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.9","version":"94'9","reported_seq":100,"reported_epoch":96,"state":"active+clean","last_fresh":"2024-03-11T21:27:08.323656+0000","last_change":"2024-03-11T21:26:53.422673+0000","last_active":"2024-03-11T21:27:08.323656+0000","last_peered":"2024-03-11T21:27:08.323656+0000","last_clean":"2024-03-11T21:27:08.323656+0000","last_became_active":"2024-03-11T21:25:42.584461+0000","last_became_peered":"2024-03-11T21:25:42.584461+0000","last_unstale":"2024-03-11T21:27:08.323656+0000","last_undegraded":"2024-03-11T21:27:08.323656+0000","last_fullsized":"2024-03-11T21:27:08.323656+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"94'9","last_scrub_stamp":"2024-03-11T21:26:53.422552+0000","last_deep_scrub":"94'9","last_deep_scrub_stamp":"2024-03-11T21:26:53.422552+0000","last_clean_scrub_stamp":"2024-03-11T21:26:53.422552+0000","objects_scrubbed":1,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:28:07.835997+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":6,"num_pg":32,"stat_sum":{"num_bytes":383,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1149,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2,"ondisk_log_size":2,"up":96,"acting":96,"num_store_stats":8},{"poolid":5,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":65,"ondisk_log_size":65,"up":96,"acting":96,"num_store_stats":8},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":3702,"num_objects":178,"num_object_clones":0,"num_object_copies":534,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":178,"num_whiteouts":0,"num_read":698,"num_read_kb":455,"num_write":417,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":417792,"data_stored":11106,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":412,"ondisk_log_size":412,"up":96,"acting":96,"num_store_stats":8},{"poolid":3,"num_pg":3,"stat_sum":{"num_bytes":408,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":88,"num_read_kb":83,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1224,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":9,"acting":9,"num_store_stats":7},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":212,"num_read_kb":306,"num_write":179,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1771104,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1781760,"data_stored":1771104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":128,"ondisk_log_size":128,"up":3,"acting":3,"num_store_stats":3},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":1531,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":24,"num_read_kb":24,"num_write":10,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":4593,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":6,"ondisk_log_size":6,"up":96,"acting":96,"num_store_stats":8}],"osd_stats":[{"osd":7,"up_from":60,"seq":257698037782,"num_pgs":56,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28480,"kb_used_data":1460,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93716672,"statfs":{"total":95995035648,"available":95965872128,"internally_reserved":0,"allocated":1495040,"data_stored":990331,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":50,"seq":214748364828,"num_pgs":44,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28476,"kb_used_data":1456,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93716676,"statfs":{"total":95995035648,"available":95965876224,"internally_reserved":0,"allocated":1490944,"data_stored":990624,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574902,"num_pgs":53,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27924,"kb_used_data":908,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717228,"statfs":{"total":95995035648,"available":95966441472,"internally_reserved":0,"allocated":929792,"data_stored":400364,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738427,"num_pgs":47,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28480,"kb_used_data":1464,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93716672,"statfs":{"total":95995035648,"available":95965872128,"internally_reserved":0,"allocated":1499136,"data_stored":990548,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411376,"num_pgs":39,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27884,"kb_used_data":868,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717268,"statfs":{"total":95995035648,"available":95966482432,"internally_reserved":0,"allocated":888832,"data_stored":399588,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":26,"seq":111669149740,"num_pgs":55,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32084,"kb_used_data":908,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713068,"statfs":{"total":95995035648,"available":95962181632,"internally_reserved":0,"allocated":929792,"data_stored":400698,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":32,"seq":137438953509,"num_pgs":53,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27908,"kb_used_data":892,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717244,"statfs":{"total":95995035648,"available":95966457856,"internally_reserved":0,"allocated":913408,"data_stored":399872,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":41,"seq":176093659168,"num_pgs":49,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32052,"kb_used_data":872,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713100,"statfs":{"total":95995035648,"available":95962214400,"internally_reserved":0,"allocated":892928,"data_stored":399639,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1049,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":46,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":957,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":574,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":436,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1003,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":528,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":1320,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":81920,"data_stored":2118,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":1018,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":1898,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":1474,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":990,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":1034,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":1254,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":370,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":383,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":370,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-11T21:27:17.378 DEBUG:teuthology.orchestra.run.smithi092:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid 00db0016-dfed-11ee-95c8-87774f69a715 -- ceph pg dump --format=json 2024-03-11T21:27:17.625 INFO:teuthology.orchestra.run.smithi092.stderr:Inferring config /var/lib/ceph/00db0016-dfed-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T21:27:17.854 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:27:17 smithi092 ceph-mon[29572]: from='client.34652 v1:172.21.15.92:0/2847064402' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-11T21:27:17.855 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:27:17 smithi092 ceph-mon[29572]: 2.11 scrub starts 2024-03-11T21:27:17.855 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:27:17 smithi092 ceph-mon[29572]: 2.11 scrub ok 2024-03-11T21:27:17.855 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:27:17 smithi092 ceph-mon[29572]: 5.19 scrub starts 2024-03-11T21:27:17.855 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:27:17 smithi092 ceph-mon[29572]: 5.19 scrub ok 2024-03-11T21:27:17.855 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:27:17 smithi092 ceph-mon[37563]: from='client.34652 v1:172.21.15.92:0/2847064402' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-11T21:27:17.855 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:27:17 smithi092 ceph-mon[37563]: 2.11 scrub starts 2024-03-11T21:27:17.855 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:27:17 smithi092 ceph-mon[37563]: 2.11 scrub ok 2024-03-11T21:27:17.855 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:27:17 smithi092 ceph-mon[37563]: 5.19 scrub starts 2024-03-11T21:27:17.855 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:27:17 smithi092 ceph-mon[37563]: 5.19 scrub ok 2024-03-11T21:27:18.077 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:27:17 smithi097 ceph-mon[33317]: from='client.34652 v1:172.21.15.92:0/2847064402' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-11T21:27:18.077 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:27:17 smithi097 ceph-mon[33317]: 2.11 scrub starts 2024-03-11T21:27:18.077 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:27:17 smithi097 ceph-mon[33317]: 2.11 scrub ok 2024-03-11T21:27:18.077 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:27:17 smithi097 ceph-mon[33317]: 5.19 scrub starts 2024-03-11T21:27:18.077 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:27:17 smithi097 ceph-mon[33317]: 5.19 scrub ok 2024-03-11T21:27:18.111 INFO:journalctl@ceph.mgr.y.smithi092.stdout:Mar 11 21:27:17 smithi092 ceph-00db0016-dfed-11ee-95c8-87774f69a715-mgr-y[29801]: ::ffff:172.21.15.97 - - [11/Mar/2024:21:27:17] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-11T21:27:18.861 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:27:18 smithi092 ceph-mon[29572]: pgmap v41: 132 pgs: 132 active+clean; 582 KiB data, 228 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-03-11T21:27:18.861 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:27:18 smithi092 ceph-mon[29572]: 2.b scrub starts 2024-03-11T21:27:18.861 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:27:18 smithi092 ceph-mon[29572]: 2.b scrub ok 2024-03-11T21:27:18.862 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:27:18 smithi092 ceph-mon[37563]: pgmap v41: 132 pgs: 132 active+clean; 582 KiB data, 228 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-03-11T21:27:18.862 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:27:18 smithi092 ceph-mon[37563]: 2.b scrub starts 2024-03-11T21:27:18.862 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:27:18 smithi092 ceph-mon[37563]: 2.b scrub ok 2024-03-11T21:27:19.078 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:27:18 smithi097 ceph-mon[33317]: pgmap v41: 132 pgs: 132 active+clean; 582 KiB data, 228 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-03-11T21:27:19.078 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:27:18 smithi097 ceph-mon[33317]: 2.b scrub starts 2024-03-11T21:27:19.079 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:27:18 smithi097 ceph-mon[33317]: 2.b scrub ok 2024-03-11T21:27:19.504 INFO:teuthology.orchestra.run.smithi092.stdout: 2024-03-11T21:27:19.507 INFO:teuthology.orchestra.run.smithi092.stderr:dumped all 2024-03-11T21:27:19.811 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:27:19 smithi092 ceph-mon[29572]: 5.1a scrub starts 2024-03-11T21:27:19.811 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:27:19 smithi092 ceph-mon[29572]: 5.1a scrub ok 2024-03-11T21:27:19.811 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:27:19 smithi092 ceph-mon[29572]: 4.7 deep-scrub starts 2024-03-11T21:27:19.811 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:27:19 smithi092 ceph-mon[29572]: 4.7 deep-scrub ok 2024-03-11T21:27:19.812 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:27:19 smithi092 ceph-mon[37563]: 5.1a scrub starts 2024-03-11T21:27:19.812 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:27:19 smithi092 ceph-mon[37563]: 5.1a scrub ok 2024-03-11T21:27:19.812 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:27:19 smithi092 ceph-mon[37563]: 4.7 deep-scrub starts 2024-03-11T21:27:19.812 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:27:19 smithi092 ceph-mon[37563]: 4.7 deep-scrub ok 2024-03-11T21:27:20.077 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:27:19 smithi097 ceph-mon[33317]: 5.1a scrub starts 2024-03-11T21:27:20.077 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:27:19 smithi097 ceph-mon[33317]: 5.1a scrub ok 2024-03-11T21:27:20.077 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:27:19 smithi097 ceph-mon[33317]: 4.7 deep-scrub starts 2024-03-11T21:27:20.077 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:27:19 smithi097 ceph-mon[33317]: 4.7 deep-scrub ok 2024-03-11T21:27:20.284 INFO:teuthology.orchestra.run.smithi092.stdout:{"pg_ready":true,"pg_map":{"version":42,"stamp":"2024-03-11T21:27:18.968827+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":596392,"num_objects":198,"num_object_clones":0,"num_object_copies":594,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":198,"num_whiteouts":0,"num_read":1027,"num_read_kb":873,"num_write":615,"num_write_kb":2461,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":21,"num_bytes_recovered":1771324,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":620,"ondisk_log_size":620,"up":396,"acting":396,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":396,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":749961216,"kb_used":233156,"kb_used_data":7964,"kb_used_omap":12,"kb_used_meta":224371,"kb_avail":749728060,"statfs":{"total":767960285184,"available":767721533440,"internally_reserved":0,"allocated":8155136,"data_stored":4693432,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12752,"internal_metadata":229756464},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":10,"num_read_kb":10,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"10.002624"},"pg_stats":[{"pgid":"6.1b","version":"0'0","reported_seq":67,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-03-11T21:27:14.205703+0000","last_change":"2024-03-11T21:26:53.584399+0000","last_active":"2024-03-11T21:27:14.205703+0000","last_peered":"2024-03-11T21:27:14.205703+0000","last_clean":"2024-03-11T21:27:14.205703+0000","last_became_active":"2024-03-11T21:25:45.302931+0000","last_became_peered":"2024-03-11T21:25:45.302931+0000","last_unstale":"2024-03-11T21:27:14.205703+0000","last_undegraded":"2024-03-11T21:27:14.205703+0000","last_fullsized":"2024-03-11T21:27:14.205703+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:26:53.584307+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:43.559452+0000","last_clean_scrub_stamp":"2024-03-11T21:26:53.584307+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:27:55.466757+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":63,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-03-11T21:27:14.200683+0000","last_change":"2024-03-11T21:25:42.579254+0000","last_active":"2024-03-11T21:27:14.200683+0000","last_peered":"2024-03-11T21:27:14.200683+0000","last_clean":"2024-03-11T21:27:14.200683+0000","last_became_active":"2024-03-11T21:25:42.578893+0000","last_became_peered":"2024-03-11T21:25:42.578893+0000","last_unstale":"2024-03-11T21:27:14.200683+0000","last_undegraded":"2024-03-11T21:27:14.200683+0000","last_fullsized":"2024-03-11T21:27:14.200683+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:25:41.544550+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:41.544550+0000","last_clean_scrub_stamp":"2024-03-11T21:25:41.544550+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1f","version":"0'0","reported_seq":79,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-03-11T21:27:14.206564+0000","last_change":"2024-03-11T21:26:46.588463+0000","last_active":"2024-03-11T21:27:14.206564+0000","last_peered":"2024-03-11T21:27:14.206564+0000","last_clean":"2024-03-11T21:27:14.206564+0000","last_became_active":"2024-03-11T21:25:38.540510+0000","last_became_peered":"2024-03-11T21:25:38.540510+0000","last_unstale":"2024-03-11T21:27:14.206564+0000","last_undegraded":"2024-03-11T21:27:14.206564+0000","last_fullsized":"2024-03-11T21:27:14.206564+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:26:46.588380+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:37.512018+0000","last_clean_scrub_stamp":"2024-03-11T21:26:46.588380+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:28:06.851002+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,4],"acting":[0,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.19","version":"74'15","reported_seq":100,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-03-11T21:27:14.206330+0000","last_change":"2024-03-11T21:27:12.751287+0000","last_active":"2024-03-11T21:27:14.206330+0000","last_peered":"2024-03-11T21:27:14.206330+0000","last_clean":"2024-03-11T21:27:14.206330+0000","last_became_active":"2024-03-11T21:25:40.554207+0000","last_became_peered":"2024-03-11T21:25:40.554207+0000","last_unstale":"2024-03-11T21:27:14.206330+0000","last_undegraded":"2024-03-11T21:27:14.206330+0000","last_fullsized":"2024-03-11T21:27:14.206330+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'15","last_scrub_stamp":"2024-03-11T21:27:12.751184+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:39.525554+0000","last_clean_scrub_stamp":"2024-03-11T21:27:12.751184+0000","objects_scrubbed":7,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:28:24.326457+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.1a","version":"0'0","reported_seq":59,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-03-11T21:27:14.200890+0000","last_change":"2024-03-11T21:25:44.591548+0000","last_active":"2024-03-11T21:27:14.200890+0000","last_peered":"2024-03-11T21:27:14.200890+0000","last_clean":"2024-03-11T21:27:14.200890+0000","last_became_active":"2024-03-11T21:25:44.590923+0000","last_became_peered":"2024-03-11T21:25:44.590923+0000","last_unstale":"2024-03-11T21:27:14.200890+0000","last_undegraded":"2024-03-11T21:27:14.200890+0000","last_fullsized":"2024-03-11T21:27:14.200890+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:25:43.559452+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:43.559452+0000","last_clean_scrub_stamp":"2024-03-11T21:25:43.559452+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,1],"acting":[4,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":63,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-03-11T21:27:14.471875+0000","last_change":"2024-03-11T21:25:42.578761+0000","last_active":"2024-03-11T21:27:14.471875+0000","last_peered":"2024-03-11T21:27:14.471875+0000","last_clean":"2024-03-11T21:27:14.471875+0000","last_became_active":"2024-03-11T21:25:42.578424+0000","last_became_peered":"2024-03-11T21:25:42.578424+0000","last_unstale":"2024-03-11T21:27:14.471875+0000","last_undegraded":"2024-03-11T21:27:14.471875+0000","last_fullsized":"2024-03-11T21:27:14.471875+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:25:41.544550+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:41.544550+0000","last_clean_scrub_stamp":"2024-03-11T21:25:41.544550+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,7],"acting":[1,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1e","version":"0'0","reported_seq":79,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-03-11T21:27:14.207226+0000","last_change":"2024-03-11T21:26:49.595465+0000","last_active":"2024-03-11T21:27:14.207226+0000","last_peered":"2024-03-11T21:27:14.207226+0000","last_clean":"2024-03-11T21:27:14.207226+0000","last_became_active":"2024-03-11T21:25:38.537283+0000","last_became_peered":"2024-03-11T21:25:38.537283+0000","last_unstale":"2024-03-11T21:27:14.207226+0000","last_undegraded":"2024-03-11T21:27:14.207226+0000","last_fullsized":"2024-03-11T21:27:14.207226+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:26:49.595363+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:37.512018+0000","last_clean_scrub_stamp":"2024-03-11T21:26:49.595363+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:27:52.155430+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.18","version":"74'9","reported_seq":91,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-03-11T21:27:14.200889+0000","last_change":"2024-03-11T21:26:45.870099+0000","last_active":"2024-03-11T21:27:14.200889+0000","last_peered":"2024-03-11T21:27:14.200889+0000","last_clean":"2024-03-11T21:27:14.200889+0000","last_became_active":"2024-03-11T21:25:40.552216+0000","last_became_peered":"2024-03-11T21:25:40.552216+0000","last_unstale":"2024-03-11T21:27:14.200889+0000","last_undegraded":"2024-03-11T21:27:14.200889+0000","last_fullsized":"2024-03-11T21:27:14.200889+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'9","last_scrub_stamp":"2024-03-11T21:26:45.869943+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:39.525554+0000","last_clean_scrub_stamp":"2024-03-11T21:26:45.869943+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:27:50.731066+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":59,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-03-11T21:27:14.316872+0000","last_change":"2024-03-11T21:25:44.590541+0000","last_active":"2024-03-11T21:27:14.316872+0000","last_peered":"2024-03-11T21:27:14.316872+0000","last_clean":"2024-03-11T21:27:14.316872+0000","last_became_active":"2024-03-11T21:25:44.590234+0000","last_became_peered":"2024-03-11T21:25:44.590234+0000","last_unstale":"2024-03-11T21:27:14.316872+0000","last_undegraded":"2024-03-11T21:27:14.316872+0000","last_fullsized":"2024-03-11T21:27:14.316872+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:25:43.559452+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:43.559452+0000","last_clean_scrub_stamp":"2024-03-11T21:25:43.559452+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3],"acting":[5,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":63,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-03-11T21:27:14.319055+0000","last_change":"2024-03-11T21:25:42.580998+0000","last_active":"2024-03-11T21:27:14.319055+0000","last_peered":"2024-03-11T21:27:14.319055+0000","last_clean":"2024-03-11T21:27:14.319055+0000","last_became_active":"2024-03-11T21:25:42.580533+0000","last_became_peered":"2024-03-11T21:25:42.580533+0000","last_unstale":"2024-03-11T21:27:14.319055+0000","last_undegraded":"2024-03-11T21:27:14.319055+0000","last_fullsized":"2024-03-11T21:27:14.319055+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:25:41.544550+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:41.544550+0000","last_clean_scrub_stamp":"2024-03-11T21:25:41.544550+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.1d","version":"0'0","reported_seq":79,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-03-11T21:27:14.319357+0000","last_change":"2024-03-11T21:26:56.370911+0000","last_active":"2024-03-11T21:27:14.319357+0000","last_peered":"2024-03-11T21:27:14.319357+0000","last_clean":"2024-03-11T21:27:14.319357+0000","last_became_active":"2024-03-11T21:25:38.542628+0000","last_became_peered":"2024-03-11T21:25:38.542628+0000","last_unstale":"2024-03-11T21:27:14.319357+0000","last_undegraded":"2024-03-11T21:27:14.319357+0000","last_fullsized":"2024-03-11T21:27:14.319357+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:26:56.370811+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:37.512018+0000","last_clean_scrub_stamp":"2024-03-11T21:26:56.370811+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:28:03.069431+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,0],"acting":[7,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.1b","version":"74'5","reported_seq":77,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-03-11T21:27:14.201180+0000","last_change":"2024-03-11T21:25:40.556136+0000","last_active":"2024-03-11T21:27:14.201180+0000","last_peered":"2024-03-11T21:27:14.201180+0000","last_clean":"2024-03-11T21:27:14.201180+0000","last_became_active":"2024-03-11T21:25:40.555826+0000","last_became_peered":"2024-03-11T21:25:40.555826+0000","last_unstale":"2024-03-11T21:27:14.201180+0000","last_undegraded":"2024-03-11T21:27:14.201180+0000","last_fullsized":"2024-03-11T21:27:14.201180+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:25:39.525554+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:39.525554+0000","last_clean_scrub_stamp":"2024-03-11T21:25:39.525554+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":11,"num_read_kb":7,"num_write":6,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,1],"acting":[4,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":59,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-03-11T21:27:14.205556+0000","last_change":"2024-03-11T21:25:44.590228+0000","last_active":"2024-03-11T21:27:14.205556+0000","last_peered":"2024-03-11T21:27:14.205556+0000","last_clean":"2024-03-11T21:27:14.205556+0000","last_became_active":"2024-03-11T21:25:44.589786+0000","last_became_peered":"2024-03-11T21:25:44.589786+0000","last_unstale":"2024-03-11T21:27:14.205556+0000","last_undegraded":"2024-03-11T21:27:14.205556+0000","last_fullsized":"2024-03-11T21:27:14.205556+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:25:43.559452+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:43.559452+0000","last_clean_scrub_stamp":"2024-03-11T21:25:43.559452+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,7],"acting":[0,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":63,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-03-11T21:27:14.317411+0000","last_change":"2024-03-11T21:25:42.574391+0000","last_active":"2024-03-11T21:27:14.317411+0000","last_peered":"2024-03-11T21:27:14.317411+0000","last_clean":"2024-03-11T21:27:14.317411+0000","last_became_active":"2024-03-11T21:25:42.573831+0000","last_became_peered":"2024-03-11T21:25:42.573831+0000","last_unstale":"2024-03-11T21:27:14.317411+0000","last_undegraded":"2024-03-11T21:27:14.317411+0000","last_fullsized":"2024-03-11T21:27:14.317411+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:25:41.544550+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:41.544550+0000","last_clean_scrub_stamp":"2024-03-11T21:25:41.544550+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,7],"acting":[5,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1c","version":"0'0","reported_seq":79,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-03-11T21:27:14.319117+0000","last_change":"2024-03-11T21:27:04.375295+0000","last_active":"2024-03-11T21:27:14.319117+0000","last_peered":"2024-03-11T21:27:14.319117+0000","last_clean":"2024-03-11T21:27:14.319117+0000","last_became_active":"2024-03-11T21:25:38.541364+0000","last_became_peered":"2024-03-11T21:25:38.541364+0000","last_unstale":"2024-03-11T21:27:14.319117+0000","last_undegraded":"2024-03-11T21:27:14.319117+0000","last_fullsized":"2024-03-11T21:27:14.319117+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:27:04.375194+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:37.512018+0000","last_clean_scrub_stamp":"2024-03-11T21:27:04.375194+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:28:17.432840+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.1a","version":"74'9","reported_seq":91,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-03-11T21:27:14.200990+0000","last_change":"2024-03-11T21:26:44.851070+0000","last_active":"2024-03-11T21:27:14.200990+0000","last_peered":"2024-03-11T21:27:14.200990+0000","last_clean":"2024-03-11T21:27:14.200990+0000","last_became_active":"2024-03-11T21:25:40.556629+0000","last_became_peered":"2024-03-11T21:25:40.556629+0000","last_unstale":"2024-03-11T21:27:14.200990+0000","last_undegraded":"2024-03-11T21:27:14.200990+0000","last_fullsized":"2024-03-11T21:27:14.200990+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'9","last_scrub_stamp":"2024-03-11T21:26:44.850930+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:39.525554+0000","last_clean_scrub_stamp":"2024-03-11T21:26:44.850930+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:27:47.017552+0000","scrub_duration":24,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0],"acting":[4,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1f","version":"0'0","reported_seq":67,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-03-11T21:27:14.207589+0000","last_change":"2024-03-11T21:27:04.620314+0000","last_active":"2024-03-11T21:27:14.207589+0000","last_peered":"2024-03-11T21:27:14.207589+0000","last_clean":"2024-03-11T21:27:14.207589+0000","last_became_active":"2024-03-11T21:25:45.302637+0000","last_became_peered":"2024-03-11T21:25:45.302637+0000","last_unstale":"2024-03-11T21:27:14.207589+0000","last_undegraded":"2024-03-11T21:27:14.207589+0000","last_fullsized":"2024-03-11T21:27:14.207589+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:27:04.620217+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:43.559452+0000","last_clean_scrub_stamp":"2024-03-11T21:27:04.620217+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:28:12.483514+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1c","version":"0'0","reported_seq":71,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-03-11T21:27:14.201435+0000","last_change":"2024-03-11T21:27:10.787962+0000","last_active":"2024-03-11T21:27:14.201435+0000","last_peered":"2024-03-11T21:27:14.201435+0000","last_clean":"2024-03-11T21:27:14.201435+0000","last_became_active":"2024-03-11T21:25:42.576064+0000","last_became_peered":"2024-03-11T21:25:42.576064+0000","last_unstale":"2024-03-11T21:27:14.201435+0000","last_undegraded":"2024-03-11T21:27:14.201435+0000","last_fullsized":"2024-03-11T21:27:14.201435+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:27:10.787826+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:41.544550+0000","last_clean_scrub_stamp":"2024-03-11T21:27:10.787826+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:28:32.419330+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1b","version":"0'0","reported_seq":79,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-03-11T21:27:14.205923+0000","last_change":"2024-03-11T21:26:55.579982+0000","last_active":"2024-03-11T21:27:14.205923+0000","last_peered":"2024-03-11T21:27:14.205923+0000","last_clean":"2024-03-11T21:27:14.205923+0000","last_became_active":"2024-03-11T21:25:38.537520+0000","last_became_peered":"2024-03-11T21:25:38.537520+0000","last_unstale":"2024-03-11T21:27:14.205923+0000","last_undegraded":"2024-03-11T21:27:14.205923+0000","last_fullsized":"2024-03-11T21:27:14.205923+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:26:55.579901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:37.512018+0000","last_clean_scrub_stamp":"2024-03-11T21:26:55.579901+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:28:17.884453+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1d","version":"74'12","reported_seq":98,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-03-11T21:27:14.206908+0000","last_change":"2024-03-11T21:27:01.629574+0000","last_active":"2024-03-11T21:27:14.206908+0000","last_peered":"2024-03-11T21:27:14.206908+0000","last_clean":"2024-03-11T21:27:14.206908+0000","last_became_active":"2024-03-11T21:25:40.556330+0000","last_became_peered":"2024-03-11T21:25:40.556330+0000","last_unstale":"2024-03-11T21:27:14.206908+0000","last_undegraded":"2024-03-11T21:27:14.206908+0000","last_fullsized":"2024-03-11T21:27:14.206908+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'12","last_scrub_stamp":"2024-03-11T21:27:01.629471+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:39.525554+0000","last_clean_scrub_stamp":"2024-03-11T21:27:01.629471+0000","objects_scrubbed":5,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:28:26.432819+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":25,"num_read_kb":16,"num_write":14,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.1e","version":"0'0","reported_seq":59,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-03-11T21:27:14.201810+0000","last_change":"2024-03-11T21:25:45.300689+0000","last_active":"2024-03-11T21:27:14.201810+0000","last_peered":"2024-03-11T21:27:14.201810+0000","last_clean":"2024-03-11T21:27:14.201810+0000","last_became_active":"2024-03-11T21:25:45.300355+0000","last_became_peered":"2024-03-11T21:25:45.300355+0000","last_unstale":"2024-03-11T21:27:14.201810+0000","last_undegraded":"2024-03-11T21:27:14.201810+0000","last_fullsized":"2024-03-11T21:27:14.201810+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:25:43.559452+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:43.559452+0000","last_clean_scrub_stamp":"2024-03-11T21:25:43.559452+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5],"acting":[4,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1d","version":"0'0","reported_seq":71,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-03-11T21:27:14.472525+0000","last_change":"2024-03-11T21:27:09.199590+0000","last_active":"2024-03-11T21:27:14.472525+0000","last_peered":"2024-03-11T21:27:14.472525+0000","last_clean":"2024-03-11T21:27:14.472525+0000","last_became_active":"2024-03-11T21:25:42.579407+0000","last_became_peered":"2024-03-11T21:25:42.579407+0000","last_unstale":"2024-03-11T21:27:14.472525+0000","last_undegraded":"2024-03-11T21:27:14.472525+0000","last_fullsized":"2024-03-11T21:27:14.472525+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:27:09.199513+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:27:09.199513+0000","last_clean_scrub_stamp":"2024-03-11T21:27:09.199513+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:28:28.109684+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1a","version":"0'0","reported_seq":79,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-03-11T21:27:14.461851+0000","last_change":"2024-03-11T21:27:04.037030+0000","last_active":"2024-03-11T21:27:14.461851+0000","last_peered":"2024-03-11T21:27:14.461851+0000","last_clean":"2024-03-11T21:27:14.461851+0000","last_became_active":"2024-03-11T21:25:38.539725+0000","last_became_peered":"2024-03-11T21:25:38.539725+0000","last_unstale":"2024-03-11T21:27:14.461851+0000","last_undegraded":"2024-03-11T21:27:14.461851+0000","last_fullsized":"2024-03-11T21:27:14.461851+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:27:04.036903+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:37.512018+0000","last_clean_scrub_stamp":"2024-03-11T21:27:04.036903+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:28:11.356799+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.1c","version":"74'15","reported_seq":100,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-03-11T21:27:14.473051+0000","last_change":"2024-03-11T21:27:04.045043+0000","last_active":"2024-03-11T21:27:14.473051+0000","last_peered":"2024-03-11T21:27:14.473051+0000","last_clean":"2024-03-11T21:27:14.473051+0000","last_became_active":"2024-03-11T21:25:40.551453+0000","last_became_peered":"2024-03-11T21:25:40.551453+0000","last_unstale":"2024-03-11T21:27:14.473051+0000","last_undegraded":"2024-03-11T21:27:14.473051+0000","last_fullsized":"2024-03-11T21:27:14.473051+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'15","last_scrub_stamp":"2024-03-11T21:27:04.044946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:39.525554+0000","last_clean_scrub_stamp":"2024-03-11T21:27:04.044946+0000","objects_scrubbed":7,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:28:23.430891+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.1d","version":"0'0","reported_seq":59,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-03-11T21:27:14.472272+0000","last_change":"2024-03-11T21:25:44.589198+0000","last_active":"2024-03-11T21:27:14.472272+0000","last_peered":"2024-03-11T21:27:14.472272+0000","last_clean":"2024-03-11T21:27:14.472272+0000","last_became_active":"2024-03-11T21:25:44.588845+0000","last_became_peered":"2024-03-11T21:25:44.588845+0000","last_unstale":"2024-03-11T21:27:14.472272+0000","last_undegraded":"2024-03-11T21:27:14.472272+0000","last_fullsized":"2024-03-11T21:27:14.472272+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:25:43.559452+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:43.559452+0000","last_clean_scrub_stamp":"2024-03-11T21:25:43.559452+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.1e","version":"0'0","reported_seq":71,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-03-11T21:27:14.206648+0000","last_change":"2024-03-11T21:26:58.335515+0000","last_active":"2024-03-11T21:27:14.206648+0000","last_peered":"2024-03-11T21:27:14.206648+0000","last_clean":"2024-03-11T21:27:14.206648+0000","last_became_active":"2024-03-11T21:25:42.582959+0000","last_became_peered":"2024-03-11T21:25:42.582959+0000","last_unstale":"2024-03-11T21:27:14.206648+0000","last_undegraded":"2024-03-11T21:27:14.206648+0000","last_fullsized":"2024-03-11T21:27:14.206648+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:26:58.335431+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:26:58.335431+0000","last_clean_scrub_stamp":"2024-03-11T21:26:58.335431+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:28:08.893453+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.19","version":"67'2","reported_seq":78,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-03-11T21:27:14.206399+0000","last_change":"2024-03-11T21:25:38.539600+0000","last_active":"2024-03-11T21:27:14.206399+0000","last_peered":"2024-03-11T21:27:14.206399+0000","last_clean":"2024-03-11T21:27:14.206399+0000","last_became_active":"2024-03-11T21:25:38.539243+0000","last_became_peered":"2024-03-11T21:25:38.539243+0000","last_unstale":"2024-03-11T21:27:14.206399+0000","last_undegraded":"2024-03-11T21:27:14.206399+0000","last_fullsized":"2024-03-11T21:27:14.206399+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:25:37.512018+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:37.512018+0000","last_clean_scrub_stamp":"2024-03-11T21:25:37.512018+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":957,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":7,"num_read_kb":7,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,0],"acting":[3,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1f","version":"74'11","reported_seq":94,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-03-11T21:27:14.462472+0000","last_change":"2024-03-11T21:26:55.050980+0000","last_active":"2024-03-11T21:27:14.462472+0000","last_peered":"2024-03-11T21:27:14.462472+0000","last_clean":"2024-03-11T21:27:14.462472+0000","last_became_active":"2024-03-11T21:25:40.554686+0000","last_became_peered":"2024-03-11T21:25:40.554686+0000","last_unstale":"2024-03-11T21:27:14.462472+0000","last_undegraded":"2024-03-11T21:27:14.462472+0000","last_fullsized":"2024-03-11T21:27:14.462472+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'11","last_scrub_stamp":"2024-03-11T21:26:55.050872+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:39.525554+0000","last_clean_scrub_stamp":"2024-03-11T21:26:55.050872+0000","objects_scrubbed":5,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:28:13.716793+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,1],"acting":[6,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.1c","version":"74'1","reported_seq":60,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-03-11T21:27:14.319549+0000","last_change":"2024-03-11T21:25:44.593623+0000","last_active":"2024-03-11T21:27:14.319549+0000","last_peered":"2024-03-11T21:27:14.319549+0000","last_clean":"2024-03-11T21:27:14.319549+0000","last_became_active":"2024-03-11T21:25:44.593326+0000","last_became_peered":"2024-03-11T21:25:44.593326+0000","last_unstale":"2024-03-11T21:27:14.319549+0000","last_undegraded":"2024-03-11T21:27:14.319549+0000","last_fullsized":"2024-03-11T21:27:14.319549+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:25:43.559452+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:43.559452+0000","last_clean_scrub_stamp":"2024-03-11T21:25:43.559452+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":370,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.1f","version":"74'8","reported_seq":102,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-03-11T21:27:14.462489+0000","last_change":"2024-03-11T21:26:48.955663+0000","last_active":"2024-03-11T21:27:14.462489+0000","last_peered":"2024-03-11T21:27:14.462489+0000","last_clean":"2024-03-11T21:27:14.462489+0000","last_became_active":"2024-03-11T21:25:42.577401+0000","last_became_peered":"2024-03-11T21:25:42.577401+0000","last_unstale":"2024-03-11T21:27:14.462489+0000","last_undegraded":"2024-03-11T21:27:14.462489+0000","last_fullsized":"2024-03-11T21:27:14.462489+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'8","last_scrub_stamp":"2024-03-11T21:26:48.955484+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:41.544550+0000","last_clean_scrub_stamp":"2024-03-11T21:26:48.955484+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:27:51.651922+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.18","version":"67'1","reported_seq":85,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-03-11T21:27:14.316810+0000","last_change":"2024-03-11T21:26:46.995375+0000","last_active":"2024-03-11T21:27:14.316810+0000","last_peered":"2024-03-11T21:27:14.316810+0000","last_clean":"2024-03-11T21:27:14.316810+0000","last_became_active":"2024-03-11T21:25:38.538321+0000","last_became_peered":"2024-03-11T21:25:38.538321+0000","last_unstale":"2024-03-11T21:27:14.316810+0000","last_undegraded":"2024-03-11T21:27:14.316810+0000","last_fullsized":"2024-03-11T21:27:14.316810+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'1","last_scrub_stamp":"2024-03-11T21:26:46.995228+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:37.512018+0000","last_clean_scrub_stamp":"2024-03-11T21:26:46.995228+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:27:58.922326+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":436,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":7,"num_read_kb":7,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,7],"acting":[5,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1e","version":"74'10","reported_seq":90,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-03-11T21:27:14.206554+0000","last_change":"2024-03-11T21:26:42.521121+0000","last_active":"2024-03-11T21:27:14.206554+0000","last_peered":"2024-03-11T21:27:14.206554+0000","last_clean":"2024-03-11T21:27:14.206554+0000","last_became_active":"2024-03-11T21:25:40.763438+0000","last_became_peered":"2024-03-11T21:25:40.763438+0000","last_unstale":"2024-03-11T21:27:14.206554+0000","last_undegraded":"2024-03-11T21:27:14.206554+0000","last_fullsized":"2024-03-11T21:27:14.206554+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'10","last_scrub_stamp":"2024-03-11T21:26:42.521016+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:39.525554+0000","last_clean_scrub_stamp":"2024-03-11T21:26:42.521016+0000","objects_scrubbed":5,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:28:12.491436+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.13","version":"0'0","reported_seq":67,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-03-11T21:27:14.206639+0000","last_change":"2024-03-11T21:27:03.623032+0000","last_active":"2024-03-11T21:27:14.206639+0000","last_peered":"2024-03-11T21:27:14.206639+0000","last_clean":"2024-03-11T21:27:14.206639+0000","last_became_active":"2024-03-11T21:25:45.302102+0000","last_became_peered":"2024-03-11T21:25:45.302102+0000","last_unstale":"2024-03-11T21:27:14.206639+0000","last_undegraded":"2024-03-11T21:27:14.206639+0000","last_fullsized":"2024-03-11T21:27:14.206639+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:27:03.622909+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:43.559452+0000","last_clean_scrub_stamp":"2024-03-11T21:27:03.622909+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:28:12.458954+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":71,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-03-11T21:27:14.463389+0000","last_change":"2024-03-11T21:27:14.463412+0000","last_active":"2024-03-11T21:27:14.463389+0000","last_peered":"2024-03-11T21:27:14.463389+0000","last_clean":"2024-03-11T21:27:14.463389+0000","last_became_active":"2024-03-11T21:25:42.579781+0000","last_became_peered":"2024-03-11T21:25:42.579781+0000","last_unstale":"2024-03-11T21:27:14.463389+0000","last_undegraded":"2024-03-11T21:27:14.463389+0000","last_fullsized":"2024-03-11T21:27:14.463389+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:27:14.463307+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:41.544550+0000","last_clean_scrub_stamp":"2024-03-11T21:27:14.463307+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:28:15.427900+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.17","version":"0'0","reported_seq":79,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-03-11T21:27:14.462607+0000","last_change":"2024-03-11T21:26:56.020325+0000","last_active":"2024-03-11T21:27:14.462607+0000","last_peered":"2024-03-11T21:27:14.462607+0000","last_clean":"2024-03-11T21:27:14.462607+0000","last_became_active":"2024-03-11T21:25:38.540349+0000","last_became_peered":"2024-03-11T21:25:38.540349+0000","last_unstale":"2024-03-11T21:27:14.462607+0000","last_undegraded":"2024-03-11T21:27:14.462607+0000","last_fullsized":"2024-03-11T21:27:14.462607+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:26:56.020191+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:37.512018+0000","last_clean_scrub_stamp":"2024-03-11T21:26:56.020191+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:28:25.054065+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,2],"acting":[6,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.11","version":"74'11","reported_seq":86,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-03-11T21:27:14.206526+0000","last_change":"2024-03-11T21:25:40.761012+0000","last_active":"2024-03-11T21:27:14.206526+0000","last_peered":"2024-03-11T21:27:14.206526+0000","last_clean":"2024-03-11T21:27:14.206526+0000","last_became_active":"2024-03-11T21:25:40.760664+0000","last_became_peered":"2024-03-11T21:25:40.760664+0000","last_unstale":"2024-03-11T21:27:14.206526+0000","last_undegraded":"2024-03-11T21:27:14.206526+0000","last_fullsized":"2024-03-11T21:27:14.206526+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:25:39.525554+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:39.525554+0000","last_clean_scrub_stamp":"2024-03-11T21:25:39.525554+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":67,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-03-11T21:27:14.318920+0000","last_change":"2024-03-11T21:27:12.403265+0000","last_active":"2024-03-11T21:27:14.318920+0000","last_peered":"2024-03-11T21:27:14.318920+0000","last_clean":"2024-03-11T21:27:14.318920+0000","last_became_active":"2024-03-11T21:25:44.591068+0000","last_became_peered":"2024-03-11T21:25:44.591068+0000","last_unstale":"2024-03-11T21:27:14.318920+0000","last_undegraded":"2024-03-11T21:27:14.318920+0000","last_fullsized":"2024-03-11T21:27:14.318920+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:27:12.403188+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:43.559452+0000","last_clean_scrub_stamp":"2024-03-11T21:27:12.403188+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:28:14.413531+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,4],"acting":[7,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":71,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-03-11T21:27:14.461738+0000","last_change":"2024-03-11T21:27:09.976879+0000","last_active":"2024-03-11T21:27:14.461738+0000","last_peered":"2024-03-11T21:27:14.461738+0000","last_clean":"2024-03-11T21:27:14.461738+0000","last_became_active":"2024-03-11T21:25:42.584148+0000","last_became_peered":"2024-03-11T21:25:42.584148+0000","last_unstale":"2024-03-11T21:27:14.461738+0000","last_undegraded":"2024-03-11T21:27:14.461738+0000","last_fullsized":"2024-03-11T21:27:14.461738+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:27:09.976747+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:27:09.976747+0000","last_clean_scrub_stamp":"2024-03-11T21:27:09.976747+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:28:32.267721+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.16","version":"0'0","reported_seq":79,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-03-11T21:27:14.317585+0000","last_change":"2024-03-11T21:27:11.939012+0000","last_active":"2024-03-11T21:27:14.317585+0000","last_peered":"2024-03-11T21:27:14.317585+0000","last_clean":"2024-03-11T21:27:14.317585+0000","last_became_active":"2024-03-11T21:25:38.539103+0000","last_became_peered":"2024-03-11T21:25:38.539103+0000","last_unstale":"2024-03-11T21:27:14.317585+0000","last_undegraded":"2024-03-11T21:27:14.317585+0000","last_fullsized":"2024-03-11T21:27:14.317585+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:27:11.938909+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:27:11.938909+0000","last_clean_scrub_stamp":"2024-03-11T21:27:11.938909+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:28:32.608220+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,2],"acting":[5,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.10","version":"94'6","reported_seq":83,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-03-11T21:27:14.205468+0000","last_change":"2024-03-11T21:26:58.601920+0000","last_active":"2024-03-11T21:27:14.205468+0000","last_peered":"2024-03-11T21:27:14.205468+0000","last_clean":"2024-03-11T21:27:14.205468+0000","last_became_active":"2024-03-11T21:25:40.552720+0000","last_became_peered":"2024-03-11T21:25:40.552720+0000","last_unstale":"2024-03-11T21:27:14.205468+0000","last_undegraded":"2024-03-11T21:27:14.205468+0000","last_fullsized":"2024-03-11T21:27:14.205468+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"94'6","last_scrub_stamp":"2024-03-11T21:26:58.601831+0000","last_deep_scrub":"94'6","last_deep_scrub_stamp":"2024-03-11T21:26:58.601831+0000","last_clean_scrub_stamp":"2024-03-11T21:26:58.601831+0000","objects_scrubbed":2,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:28:12.313486+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":6,"num_read_kb":4,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6],"acting":[3,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":67,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-03-11T21:27:14.206255+0000","last_change":"2024-03-11T21:27:05.669068+0000","last_active":"2024-03-11T21:27:14.206255+0000","last_peered":"2024-03-11T21:27:14.206255+0000","last_clean":"2024-03-11T21:27:14.206255+0000","last_became_active":"2024-03-11T21:25:44.583181+0000","last_became_peered":"2024-03-11T21:25:44.583181+0000","last_unstale":"2024-03-11T21:27:14.206255+0000","last_undegraded":"2024-03-11T21:27:14.206255+0000","last_fullsized":"2024-03-11T21:27:14.206255+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:27:05.668971+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:43.559452+0000","last_clean_scrub_stamp":"2024-03-11T21:27:05.668971+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:28:34.193069+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":63,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-03-11T21:27:14.472751+0000","last_change":"2024-03-11T21:25:42.573497+0000","last_active":"2024-03-11T21:27:14.472751+0000","last_peered":"2024-03-11T21:27:14.472751+0000","last_clean":"2024-03-11T21:27:14.472751+0000","last_became_active":"2024-03-11T21:25:42.573158+0000","last_became_peered":"2024-03-11T21:25:42.573158+0000","last_unstale":"2024-03-11T21:27:14.472751+0000","last_undegraded":"2024-03-11T21:27:14.472751+0000","last_fullsized":"2024-03-11T21:27:14.472751+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:25:41.544550+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:41.544550+0000","last_clean_scrub_stamp":"2024-03-11T21:25:41.544550+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.15","version":"0'0","reported_seq":79,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-03-11T21:27:14.472366+0000","last_change":"2024-03-11T21:26:44.860603+0000","last_active":"2024-03-11T21:27:14.472366+0000","last_peered":"2024-03-11T21:27:14.472366+0000","last_clean":"2024-03-11T21:27:14.472366+0000","last_became_active":"2024-03-11T21:25:38.541705+0000","last_became_peered":"2024-03-11T21:25:38.541705+0000","last_unstale":"2024-03-11T21:27:14.472366+0000","last_undegraded":"2024-03-11T21:27:14.472366+0000","last_fullsized":"2024-03-11T21:27:14.472366+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:26:44.860499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:26:44.860499+0000","last_clean_scrub_stamp":"2024-03-11T21:26:44.860499+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:28:11.867124+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.13","version":"74'11","reported_seq":112,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-03-11T21:27:14.201293+0000","last_change":"2024-03-11T21:27:03.784968+0000","last_active":"2024-03-11T21:27:14.201293+0000","last_peered":"2024-03-11T21:27:14.201293+0000","last_clean":"2024-03-11T21:27:14.201293+0000","last_became_active":"2024-03-11T21:26:11.214879+0000","last_became_peered":"2024-03-11T21:26:11.214879+0000","last_unstale":"2024-03-11T21:27:14.201293+0000","last_undegraded":"2024-03-11T21:27:14.201293+0000","last_fullsized":"2024-03-11T21:27:14.201293+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'11","last_scrub_stamp":"2024-03-11T21:27:03.784851+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:39.525554+0000","last_clean_scrub_stamp":"2024-03-11T21:27:03.784851+0000","objects_scrubbed":5,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:28:14.160633+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,2],"acting":[4,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.10","version":"0'0","reported_seq":59,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-03-11T21:27:14.205245+0000","last_change":"2024-03-11T21:25:44.591915+0000","last_active":"2024-03-11T21:27:14.205245+0000","last_peered":"2024-03-11T21:27:14.205245+0000","last_clean":"2024-03-11T21:27:14.205245+0000","last_became_active":"2024-03-11T21:25:44.591189+0000","last_became_peered":"2024-03-11T21:25:44.591189+0000","last_unstale":"2024-03-11T21:27:14.205245+0000","last_undegraded":"2024-03-11T21:27:14.205245+0000","last_fullsized":"2024-03-11T21:27:14.205245+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:25:43.559452+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:43.559452+0000","last_clean_scrub_stamp":"2024-03-11T21:25:43.559452+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,1],"acting":[0,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":71,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-03-11T21:27:14.206026+0000","last_change":"2024-03-11T21:27:06.637829+0000","last_active":"2024-03-11T21:27:14.206026+0000","last_peered":"2024-03-11T21:27:14.206026+0000","last_clean":"2024-03-11T21:27:14.206026+0000","last_became_active":"2024-03-11T21:25:42.576425+0000","last_became_peered":"2024-03-11T21:25:42.576425+0000","last_unstale":"2024-03-11T21:27:14.206026+0000","last_undegraded":"2024-03-11T21:27:14.206026+0000","last_fullsized":"2024-03-11T21:27:14.206026+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:27:06.637752+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:41.544550+0000","last_clean_scrub_stamp":"2024-03-11T21:27:06.637752+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:28:24.953840+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.14","version":"0'0","reported_seq":79,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-03-11T21:27:14.462162+0000","last_change":"2024-03-11T21:26:59.050063+0000","last_active":"2024-03-11T21:27:14.462162+0000","last_peered":"2024-03-11T21:27:14.462162+0000","last_clean":"2024-03-11T21:27:14.462162+0000","last_became_active":"2024-03-11T21:25:38.541852+0000","last_became_peered":"2024-03-11T21:25:38.541852+0000","last_unstale":"2024-03-11T21:27:14.462162+0000","last_undegraded":"2024-03-11T21:27:14.462162+0000","last_fullsized":"2024-03-11T21:27:14.462162+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:26:59.049960+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:37.512018+0000","last_clean_scrub_stamp":"2024-03-11T21:26:59.049960+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:28:08.381102+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,5],"acting":[6,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.12","version":"74'9","reported_seq":91,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-03-11T21:27:15.286121+0000","last_change":"2024-03-11T21:27:15.286166+0000","last_active":"2024-03-11T21:27:15.286121+0000","last_peered":"2024-03-11T21:27:15.286121+0000","last_clean":"2024-03-11T21:27:15.286121+0000","last_became_active":"2024-03-11T21:25:40.547082+0000","last_became_peered":"2024-03-11T21:25:40.547082+0000","last_unstale":"2024-03-11T21:27:15.286121+0000","last_undegraded":"2024-03-11T21:27:15.286121+0000","last_fullsized":"2024-03-11T21:27:15.286121+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'9","last_scrub_stamp":"2024-03-11T21:27:15.286044+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:39.525554+0000","last_clean_scrub_stamp":"2024-03-11T21:27:15.286044+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:28:29.151259+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":67,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-03-11T21:27:14.201890+0000","last_change":"2024-03-11T21:27:09.801239+0000","last_active":"2024-03-11T21:27:14.201890+0000","last_peered":"2024-03-11T21:27:14.201890+0000","last_clean":"2024-03-11T21:27:14.201890+0000","last_became_active":"2024-03-11T21:25:44.589586+0000","last_became_peered":"2024-03-11T21:25:44.589586+0000","last_unstale":"2024-03-11T21:27:14.201890+0000","last_undegraded":"2024-03-11T21:27:14.201890+0000","last_fullsized":"2024-03-11T21:27:14.201890+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:27:09.801138+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:27:09.801138+0000","last_clean_scrub_stamp":"2024-03-11T21:27:09.801138+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:28:37.698118+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,5],"acting":[4,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.14","version":"74'8","reported_seq":91,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-03-11T21:27:14.207126+0000","last_change":"2024-03-11T21:25:42.583567+0000","last_active":"2024-03-11T21:27:14.207126+0000","last_peered":"2024-03-11T21:27:14.207126+0000","last_clean":"2024-03-11T21:27:14.207126+0000","last_became_active":"2024-03-11T21:25:42.583266+0000","last_became_peered":"2024-03-11T21:25:42.583266+0000","last_unstale":"2024-03-11T21:27:14.207126+0000","last_undegraded":"2024-03-11T21:27:14.207126+0000","last_fullsized":"2024-03-11T21:27:14.207126+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:25:41.544550+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:41.544550+0000","last_clean_scrub_stamp":"2024-03-11T21:25:41.544550+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,2],"acting":[3,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.13","version":"0'0","reported_seq":79,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-03-11T21:27:14.207100+0000","last_change":"2024-03-11T21:27:12.315278+0000","last_active":"2024-03-11T21:27:14.207100+0000","last_peered":"2024-03-11T21:27:14.207100+0000","last_clean":"2024-03-11T21:27:14.207100+0000","last_became_active":"2024-03-11T21:25:38.539467+0000","last_became_peered":"2024-03-11T21:25:38.539467+0000","last_unstale":"2024-03-11T21:27:14.207100+0000","last_undegraded":"2024-03-11T21:27:14.207100+0000","last_fullsized":"2024-03-11T21:27:14.207100+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:27:12.315200+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:37.512018+0000","last_clean_scrub_stamp":"2024-03-11T21:27:12.315200+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:28:29.228183+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.15","version":"94'13","reported_seq":95,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-03-11T21:27:14.317564+0000","last_change":"2024-03-11T21:26:59.013939+0000","last_active":"2024-03-11T21:27:14.317564+0000","last_peered":"2024-03-11T21:27:14.317564+0000","last_clean":"2024-03-11T21:27:14.317564+0000","last_became_active":"2024-03-11T21:25:40.760907+0000","last_became_peered":"2024-03-11T21:25:40.760907+0000","last_unstale":"2024-03-11T21:27:14.317564+0000","last_undegraded":"2024-03-11T21:27:14.317564+0000","last_fullsized":"2024-03-11T21:27:14.317564+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"94'13","last_scrub_stamp":"2024-03-11T21:26:59.013823+0000","last_deep_scrub":"94'13","last_deep_scrub_stamp":"2024-03-11T21:26:59.013823+0000","last_clean_scrub_stamp":"2024-03-11T21:26:59.013823+0000","objects_scrubbed":4,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:28:25.819110+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.16","version":"0'0","reported_seq":67,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-03-11T21:27:14.206153+0000","last_change":"2024-03-11T21:26:57.367233+0000","last_active":"2024-03-11T21:27:14.206153+0000","last_peered":"2024-03-11T21:27:14.206153+0000","last_clean":"2024-03-11T21:27:14.206153+0000","last_became_active":"2024-03-11T21:25:44.583393+0000","last_became_peered":"2024-03-11T21:25:44.583393+0000","last_unstale":"2024-03-11T21:27:14.206153+0000","last_undegraded":"2024-03-11T21:27:14.206153+0000","last_fullsized":"2024-03-11T21:27:14.206153+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:26:57.367155+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:43.559452+0000","last_clean_scrub_stamp":"2024-03-11T21:26:57.367155+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:28:24.056062+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.15","version":"74'8","reported_seq":91,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-03-11T21:27:14.317451+0000","last_change":"2024-03-11T21:25:42.572973+0000","last_active":"2024-03-11T21:27:14.317451+0000","last_peered":"2024-03-11T21:27:14.317451+0000","last_clean":"2024-03-11T21:27:14.317451+0000","last_became_active":"2024-03-11T21:25:42.572555+0000","last_became_peered":"2024-03-11T21:25:42.572555+0000","last_unstale":"2024-03-11T21:27:14.317451+0000","last_undegraded":"2024-03-11T21:27:14.317451+0000","last_fullsized":"2024-03-11T21:27:14.317451+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:25:41.544550+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:41.544550+0000","last_clean_scrub_stamp":"2024-03-11T21:25:41.544550+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.12","version":"0'0","reported_seq":79,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-03-11T21:27:14.317059+0000","last_change":"2024-03-11T21:26:47.960353+0000","last_active":"2024-03-11T21:27:14.317059+0000","last_peered":"2024-03-11T21:27:14.317059+0000","last_clean":"2024-03-11T21:27:14.317059+0000","last_became_active":"2024-03-11T21:25:38.543962+0000","last_became_peered":"2024-03-11T21:25:38.543962+0000","last_unstale":"2024-03-11T21:27:14.317059+0000","last_undegraded":"2024-03-11T21:27:14.317059+0000","last_fullsized":"2024-03-11T21:27:14.317059+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:26:47.960265+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:37.512018+0000","last_clean_scrub_stamp":"2024-03-11T21:26:47.960265+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:28:15.602449+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.14","version":"74'10","reported_seq":82,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-03-11T21:27:14.206922+0000","last_change":"2024-03-11T21:25:40.760925+0000","last_active":"2024-03-11T21:27:14.206922+0000","last_peered":"2024-03-11T21:27:14.206922+0000","last_clean":"2024-03-11T21:27:14.206922+0000","last_became_active":"2024-03-11T21:25:40.760620+0000","last_became_peered":"2024-03-11T21:25:40.760620+0000","last_unstale":"2024-03-11T21:27:14.206922+0000","last_undegraded":"2024-03-11T21:27:14.206922+0000","last_fullsized":"2024-03-11T21:27:14.206922+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:25:39.525554+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:39.525554+0000","last_clean_scrub_stamp":"2024-03-11T21:25:39.525554+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.15","version":"0'0","reported_seq":67,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-03-11T21:27:14.319317+0000","last_change":"2024-03-11T21:27:00.378136+0000","last_active":"2024-03-11T21:27:14.319317+0000","last_peered":"2024-03-11T21:27:14.319317+0000","last_clean":"2024-03-11T21:27:14.319317+0000","last_became_active":"2024-03-11T21:25:45.302501+0000","last_became_peered":"2024-03-11T21:25:45.302501+0000","last_unstale":"2024-03-11T21:27:14.319317+0000","last_undegraded":"2024-03-11T21:27:14.319317+0000","last_fullsized":"2024-03-11T21:27:14.319317+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:27:00.377987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:43.559452+0000","last_clean_scrub_stamp":"2024-03-11T21:27:00.377987+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:28:21.911670+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":63,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-03-11T21:27:14.317583+0000","last_change":"2024-03-11T21:25:42.579268+0000","last_active":"2024-03-11T21:27:14.317583+0000","last_peered":"2024-03-11T21:27:14.317583+0000","last_clean":"2024-03-11T21:27:14.317583+0000","last_became_active":"2024-03-11T21:25:42.577351+0000","last_became_peered":"2024-03-11T21:25:42.577351+0000","last_unstale":"2024-03-11T21:27:14.317583+0000","last_undegraded":"2024-03-11T21:27:14.317583+0000","last_fullsized":"2024-03-11T21:27:14.317583+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:25:41.544550+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:41.544550+0000","last_clean_scrub_stamp":"2024-03-11T21:25:41.544550+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,1],"acting":[5,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.11","version":"0'0","reported_seq":71,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-03-11T21:27:14.461749+0000","last_change":"2024-03-11T21:25:38.538379+0000","last_active":"2024-03-11T21:27:14.461749+0000","last_peered":"2024-03-11T21:27:14.461749+0000","last_clean":"2024-03-11T21:27:14.461749+0000","last_became_active":"2024-03-11T21:25:38.537415+0000","last_became_peered":"2024-03-11T21:25:38.537415+0000","last_unstale":"2024-03-11T21:27:14.461749+0000","last_undegraded":"2024-03-11T21:27:14.461749+0000","last_fullsized":"2024-03-11T21:27:14.461749+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:25:37.512018+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:37.512018+0000","last_clean_scrub_stamp":"2024-03-11T21:25:37.512018+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.17","version":"74'6","reported_seq":90,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-03-11T21:27:14.207574+0000","last_change":"2024-03-11T21:26:11.224229+0000","last_active":"2024-03-11T21:27:14.207574+0000","last_peered":"2024-03-11T21:27:14.207574+0000","last_clean":"2024-03-11T21:27:14.207574+0000","last_became_active":"2024-03-11T21:26:11.219020+0000","last_became_peered":"2024-03-11T21:26:11.219020+0000","last_unstale":"2024-03-11T21:27:14.207574+0000","last_undegraded":"2024-03-11T21:27:14.207574+0000","last_fullsized":"2024-03-11T21:27:14.207574+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:25:39.525554+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:39.525554+0000","last_clean_scrub_stamp":"2024-03-11T21:25:39.525554+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":9,"num_read_kb":6,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,5],"acting":[3,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.14","version":"0'0","reported_seq":59,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-03-11T21:27:14.472858+0000","last_change":"2024-03-11T21:25:44.587755+0000","last_active":"2024-03-11T21:27:14.472858+0000","last_peered":"2024-03-11T21:27:14.472858+0000","last_clean":"2024-03-11T21:27:14.472858+0000","last_became_active":"2024-03-11T21:25:44.587439+0000","last_became_peered":"2024-03-11T21:25:44.587439+0000","last_unstale":"2024-03-11T21:27:14.472858+0000","last_undegraded":"2024-03-11T21:27:14.472858+0000","last_fullsized":"2024-03-11T21:27:14.472858+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:25:43.559452+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:43.559452+0000","last_clean_scrub_stamp":"2024-03-11T21:25:43.559452+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,7],"acting":[2,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":71,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-03-11T21:27:15.766260+0000","last_change":"2024-03-11T21:27:15.766288+0000","last_active":"2024-03-11T21:27:15.766260+0000","last_peered":"2024-03-11T21:27:15.766260+0000","last_clean":"2024-03-11T21:27:15.766260+0000","last_became_active":"2024-03-11T21:25:42.583514+0000","last_became_peered":"2024-03-11T21:25:42.583514+0000","last_unstale":"2024-03-11T21:27:15.766260+0000","last_undegraded":"2024-03-11T21:27:15.766260+0000","last_fullsized":"2024-03-11T21:27:15.766260+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:27:15.766188+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:41.544550+0000","last_clean_scrub_stamp":"2024-03-11T21:27:15.766188+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:28:22.407684+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.10","version":"0'0","reported_seq":79,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-03-11T21:27:14.471861+0000","last_change":"2024-03-11T21:27:09.937508+0000","last_active":"2024-03-11T21:27:14.471861+0000","last_peered":"2024-03-11T21:27:14.471861+0000","last_clean":"2024-03-11T21:27:14.471861+0000","last_became_active":"2024-03-11T21:25:38.533138+0000","last_became_peered":"2024-03-11T21:25:38.533138+0000","last_unstale":"2024-03-11T21:27:14.471861+0000","last_undegraded":"2024-03-11T21:27:14.471861+0000","last_fullsized":"2024-03-11T21:27:14.471861+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:27:09.937426+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:37.512018+0000","last_clean_scrub_stamp":"2024-03-11T21:27:09.937426+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:28:23.648986+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,0],"acting":[2,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.16","version":"74'9","reported_seq":91,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-03-11T21:27:14.206345+0000","last_change":"2024-03-11T21:26:53.490093+0000","last_active":"2024-03-11T21:27:14.206345+0000","last_peered":"2024-03-11T21:27:14.206345+0000","last_clean":"2024-03-11T21:27:14.206345+0000","last_became_active":"2024-03-11T21:25:40.763434+0000","last_became_peered":"2024-03-11T21:25:40.763434+0000","last_unstale":"2024-03-11T21:27:14.206345+0000","last_undegraded":"2024-03-11T21:27:14.206345+0000","last_fullsized":"2024-03-11T21:27:14.206345+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'9","last_scrub_stamp":"2024-03-11T21:26:53.489953+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:39.525554+0000","last_clean_scrub_stamp":"2024-03-11T21:26:53.489953+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:27:55.665353+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.b","version":"0'0","reported_seq":59,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-03-11T21:27:14.206117+0000","last_change":"2024-03-11T21:25:44.590403+0000","last_active":"2024-03-11T21:27:14.206117+0000","last_peered":"2024-03-11T21:27:14.206117+0000","last_clean":"2024-03-11T21:27:14.206117+0000","last_became_active":"2024-03-11T21:25:44.590071+0000","last_became_peered":"2024-03-11T21:25:44.590071+0000","last_unstale":"2024-03-11T21:27:14.206117+0000","last_undegraded":"2024-03-11T21:27:14.206117+0000","last_fullsized":"2024-03-11T21:27:14.206117+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:25:43.559452+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:43.559452+0000","last_clean_scrub_stamp":"2024-03-11T21:25:43.559452+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.f","version":"67'2","reported_seq":83,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-03-11T21:27:14.201721+0000","last_change":"2024-03-11T21:25:38.537988+0000","last_active":"2024-03-11T21:27:14.201721+0000","last_peered":"2024-03-11T21:27:14.201721+0000","last_clean":"2024-03-11T21:27:14.201721+0000","last_became_active":"2024-03-11T21:25:38.537711+0000","last_became_peered":"2024-03-11T21:25:38.537711+0000","last_unstale":"2024-03-11T21:27:14.201721+0000","last_undegraded":"2024-03-11T21:27:14.201721+0000","last_fullsized":"2024-03-11T21:27:14.201721+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:25:37.512018+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:37.512018+0000","last_clean_scrub_stamp":"2024-03-11T21:25:37.512018+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":92,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":10,"num_read_kb":10,"num_write":4,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,7],"acting":[4,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.9","version":"74'12","reported_seq":90,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-03-11T21:27:14.201048+0000","last_change":"2024-03-11T21:25:40.552685+0000","last_active":"2024-03-11T21:27:14.201048+0000","last_peered":"2024-03-11T21:27:14.201048+0000","last_clean":"2024-03-11T21:27:14.201048+0000","last_became_active":"2024-03-11T21:25:40.552267+0000","last_became_peered":"2024-03-11T21:25:40.552267+0000","last_unstale":"2024-03-11T21:27:14.201048+0000","last_undegraded":"2024-03-11T21:27:14.201048+0000","last_fullsized":"2024-03-11T21:27:14.201048+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:25:39.525554+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:39.525554+0000","last_clean_scrub_stamp":"2024-03-11T21:25:39.525554+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":25,"num_read_kb":16,"num_write":14,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,3],"acting":[4,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":71,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-03-11T21:27:14.471760+0000","last_change":"2024-03-11T21:27:08.954258+0000","last_active":"2024-03-11T21:27:14.471760+0000","last_peered":"2024-03-11T21:27:14.471760+0000","last_clean":"2024-03-11T21:27:14.471760+0000","last_became_active":"2024-03-11T21:25:42.573987+0000","last_became_peered":"2024-03-11T21:25:42.573987+0000","last_unstale":"2024-03-11T21:27:14.471760+0000","last_undegraded":"2024-03-11T21:27:14.471760+0000","last_fullsized":"2024-03-11T21:27:14.471760+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:27:08.954143+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:41.544550+0000","last_clean_scrub_stamp":"2024-03-11T21:27:08.954143+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:28:34.087042+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.0","version":"0'0","reported_seq":59,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-03-11T21:27:14.205245+0000","last_change":"2024-03-11T21:25:44.585477+0000","last_active":"2024-03-11T21:27:14.205245+0000","last_peered":"2024-03-11T21:27:14.205245+0000","last_clean":"2024-03-11T21:27:14.205245+0000","last_became_active":"2024-03-11T21:25:44.584565+0000","last_became_peered":"2024-03-11T21:25:44.584565+0000","last_unstale":"2024-03-11T21:27:14.205245+0000","last_undegraded":"2024-03-11T21:27:14.205245+0000","last_fullsized":"2024-03-11T21:27:14.205245+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:25:43.559452+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:43.559452+0000","last_clean_scrub_stamp":"2024-03-11T21:25:43.559452+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":79,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-03-11T21:27:14.472529+0000","last_change":"2024-03-11T21:27:06.092300+0000","last_active":"2024-03-11T21:27:14.472529+0000","last_peered":"2024-03-11T21:27:14.472529+0000","last_clean":"2024-03-11T21:27:14.472529+0000","last_became_active":"2024-03-11T21:25:38.541760+0000","last_became_peered":"2024-03-11T21:25:38.541760+0000","last_unstale":"2024-03-11T21:27:14.472529+0000","last_undegraded":"2024-03-11T21:27:14.472529+0000","last_fullsized":"2024-03-11T21:27:14.472529+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:27:06.092220+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:37.512018+0000","last_clean_scrub_stamp":"2024-03-11T21:27:06.092220+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:28:22.259535+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,7],"acting":[1,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.2","version":"94'15","reported_seq":95,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-03-11T21:27:14.472708+0000","last_change":"2024-03-11T21:26:49.993206+0000","last_active":"2024-03-11T21:27:14.472708+0000","last_peered":"2024-03-11T21:27:14.472708+0000","last_clean":"2024-03-11T21:27:14.472708+0000","last_became_active":"2024-03-11T21:25:40.548833+0000","last_became_peered":"2024-03-11T21:25:40.548833+0000","last_unstale":"2024-03-11T21:27:14.472708+0000","last_undegraded":"2024-03-11T21:27:14.472708+0000","last_fullsized":"2024-03-11T21:27:14.472708+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"94'15","last_scrub_stamp":"2024-03-11T21:26:49.993060+0000","last_deep_scrub":"94'15","last_deep_scrub_stamp":"2024-03-11T21:26:49.993060+0000","last_clean_scrub_stamp":"2024-03-11T21:26:49.993060+0000","objects_scrubbed":5,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:27:59.041594+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.3","version":"74'8","reported_seq":99,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-03-11T21:27:14.207276+0000","last_change":"2024-03-11T21:27:06.321926+0000","last_active":"2024-03-11T21:27:14.207276+0000","last_peered":"2024-03-11T21:27:14.207276+0000","last_clean":"2024-03-11T21:27:14.207276+0000","last_became_active":"2024-03-11T21:25:42.578161+0000","last_became_peered":"2024-03-11T21:25:42.578161+0000","last_unstale":"2024-03-11T21:27:14.207276+0000","last_undegraded":"2024-03-11T21:27:14.207276+0000","last_fullsized":"2024-03-11T21:27:14.207276+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'8","last_scrub_stamp":"2024-03-11T21:27:06.321832+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:41.544550+0000","last_clean_scrub_stamp":"2024-03-11T21:27:06.321832+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:28:27.084331+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,5],"acting":[0,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":67,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-03-11T21:27:14.207312+0000","last_change":"2024-03-11T21:27:07.651113+0000","last_active":"2024-03-11T21:27:14.207312+0000","last_peered":"2024-03-11T21:27:14.207312+0000","last_clean":"2024-03-11T21:27:14.207312+0000","last_became_active":"2024-03-11T21:25:44.585223+0000","last_became_peered":"2024-03-11T21:25:44.585223+0000","last_unstale":"2024-03-11T21:27:14.207312+0000","last_undegraded":"2024-03-11T21:27:14.207312+0000","last_fullsized":"2024-03-11T21:27:14.207312+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:27:07.651016+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:43.559452+0000","last_clean_scrub_stamp":"2024-03-11T21:27:07.651016+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:28:14.928745+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,7],"acting":[3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":79,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-03-11T21:27:14.317151+0000","last_change":"2024-03-11T21:26:45.072161+0000","last_active":"2024-03-11T21:27:14.317151+0000","last_peered":"2024-03-11T21:27:14.317151+0000","last_clean":"2024-03-11T21:27:14.317151+0000","last_became_active":"2024-03-11T21:25:38.538540+0000","last_became_peered":"2024-03-11T21:25:38.538540+0000","last_unstale":"2024-03-11T21:27:14.317151+0000","last_undegraded":"2024-03-11T21:27:14.317151+0000","last_fullsized":"2024-03-11T21:27:14.317151+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:26:45.072058+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:37.512018+0000","last_clean_scrub_stamp":"2024-03-11T21:26:45.072058+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:27:49.711890+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.4","version":"74'28","reported_seq":143,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-03-11T21:27:14.472616+0000","last_change":"2024-03-11T21:27:08.173608+0000","last_active":"2024-03-11T21:27:14.472616+0000","last_peered":"2024-03-11T21:27:14.472616+0000","last_clean":"2024-03-11T21:27:14.472616+0000","last_became_active":"2024-03-11T21:25:40.552902+0000","last_became_peered":"2024-03-11T21:25:40.552902+0000","last_unstale":"2024-03-11T21:27:14.472616+0000","last_undegraded":"2024-03-11T21:27:14.472616+0000","last_fullsized":"2024-03-11T21:27:14.472616+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'28","last_scrub_stamp":"2024-03-11T21:27:08.173498+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:39.525554+0000","last_clean_scrub_stamp":"2024-03-11T21:27:08.173498+0000","objects_scrubbed":10,"log_size":28,"log_dups_size":0,"ondisk_log_size":28,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:28:21.301768+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":358,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":48,"num_read_kb":33,"num_write":26,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3],"acting":[1,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":63,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-03-11T21:27:14.206354+0000","last_change":"2024-03-11T21:25:42.573353+0000","last_active":"2024-03-11T21:27:14.206354+0000","last_peered":"2024-03-11T21:27:14.206354+0000","last_clean":"2024-03-11T21:27:14.206354+0000","last_became_active":"2024-03-11T21:25:42.572673+0000","last_became_peered":"2024-03-11T21:25:42.572673+0000","last_unstale":"2024-03-11T21:27:14.206354+0000","last_undegraded":"2024-03-11T21:27:14.206354+0000","last_fullsized":"2024-03-11T21:27:14.206354+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:25:41.544550+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:41.544550+0000","last_clean_scrub_stamp":"2024-03-11T21:25:41.544550+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.5","version":"0'0","reported_seq":67,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-03-11T21:27:14.319547+0000","last_change":"2024-03-11T21:27:05.338089+0000","last_active":"2024-03-11T21:27:14.319547+0000","last_peered":"2024-03-11T21:27:14.319547+0000","last_clean":"2024-03-11T21:27:14.319547+0000","last_became_active":"2024-03-11T21:25:45.302120+0000","last_became_peered":"2024-03-11T21:25:45.302120+0000","last_unstale":"2024-03-11T21:27:14.319547+0000","last_undegraded":"2024-03-11T21:27:14.319547+0000","last_fullsized":"2024-03-11T21:27:14.319547+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:27:05.337991+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:43.559452+0000","last_clean_scrub_stamp":"2024-03-11T21:27:05.337991+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:28:33.442171+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,3],"acting":[7,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.0","version":"76'5","reported_seq":182,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-03-11T21:27:15.616619+0000","last_change":"2024-03-11T21:26:54.088445+0000","last_active":"2024-03-11T21:27:15.616619+0000","last_peered":"2024-03-11T21:27:15.616619+0000","last_clean":"2024-03-11T21:27:15.616619+0000","last_became_active":"2024-03-11T21:25:38.536890+0000","last_became_peered":"2024-03-11T21:25:38.536890+0000","last_unstale":"2024-03-11T21:27:15.616619+0000","last_undegraded":"2024-03-11T21:27:15.616619+0000","last_fullsized":"2024-03-11T21:27:15.616619+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"76'5","last_scrub_stamp":"2024-03-11T21:26:54.088356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:37.512018+0000","last_clean_scrub_stamp":"2024-03-11T21:26:54.088356+0000","objects_scrubbed":1,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:28:21.305217+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.000753559,"stat_sum":{"num_bytes":389,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":93,"num_read_kb":88,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,6],"acting":[1,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":79,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-03-11T21:27:14.471913+0000","last_change":"2024-03-11T21:26:48.997561+0000","last_active":"2024-03-11T21:27:14.471913+0000","last_peered":"2024-03-11T21:27:14.471913+0000","last_clean":"2024-03-11T21:27:14.471913+0000","last_became_active":"2024-03-11T21:25:38.532984+0000","last_became_peered":"2024-03-11T21:25:38.532984+0000","last_unstale":"2024-03-11T21:27:14.471913+0000","last_undegraded":"2024-03-11T21:27:14.471913+0000","last_fullsized":"2024-03-11T21:27:14.471913+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:26:48.997454+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:26:48.997454+0000","last_clean_scrub_stamp":"2024-03-11T21:26:48.997454+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:28:14.373168+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.7","version":"74'13","reported_seq":94,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-03-11T21:27:14.473404+0000","last_change":"2024-03-11T21:25:40.542117+0000","last_active":"2024-03-11T21:27:14.473404+0000","last_peered":"2024-03-11T21:27:14.473404+0000","last_clean":"2024-03-11T21:27:14.473404+0000","last_became_active":"2024-03-11T21:25:40.541769+0000","last_became_peered":"2024-03-11T21:25:40.541769+0000","last_unstale":"2024-03-11T21:27:14.473404+0000","last_undegraded":"2024-03-11T21:27:14.473404+0000","last_fullsized":"2024-03-11T21:27:14.473404+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:25:39.525554+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:39.525554+0000","last_clean_scrub_stamp":"2024-03-11T21:25:39.525554+0000","objects_scrubbed":0,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":19,"num_write":16,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":71,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-03-11T21:27:14.473094+0000","last_change":"2024-03-11T21:27:02.039134+0000","last_active":"2024-03-11T21:27:14.473094+0000","last_peered":"2024-03-11T21:27:14.473094+0000","last_clean":"2024-03-11T21:27:14.473094+0000","last_became_active":"2024-03-11T21:25:42.581710+0000","last_became_peered":"2024-03-11T21:25:42.581710+0000","last_unstale":"2024-03-11T21:27:14.473094+0000","last_undegraded":"2024-03-11T21:27:14.473094+0000","last_fullsized":"2024-03-11T21:27:14.473094+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:27:02.039041+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:41.544550+0000","last_clean_scrub_stamp":"2024-03-11T21:27:02.039041+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:28:26.912856+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7],"acting":[2,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.7","version":"74'1","reported_seq":60,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-03-11T21:27:14.317860+0000","last_change":"2024-03-11T21:25:44.591645+0000","last_active":"2024-03-11T21:27:14.317860+0000","last_peered":"2024-03-11T21:27:14.317860+0000","last_clean":"2024-03-11T21:27:14.317860+0000","last_became_active":"2024-03-11T21:25:44.591376+0000","last_became_peered":"2024-03-11T21:25:44.591376+0000","last_unstale":"2024-03-11T21:27:14.317860+0000","last_undegraded":"2024-03-11T21:27:14.317860+0000","last_fullsized":"2024-03-11T21:27:14.317860+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:25:43.559452+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:43.559452+0000","last_clean_scrub_stamp":"2024-03-11T21:25:43.559452+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,4],"acting":[5,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2","version":"71'2","reported_seq":88,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-03-11T21:27:14.205761+0000","last_change":"2024-03-11T21:26:42.663119+0000","last_active":"2024-03-11T21:27:14.205761+0000","last_peered":"2024-03-11T21:27:14.205761+0000","last_clean":"2024-03-11T21:27:14.205761+0000","last_became_active":"2024-03-11T21:25:38.539413+0000","last_became_peered":"2024-03-11T21:25:38.539413+0000","last_unstale":"2024-03-11T21:27:14.205761+0000","last_undegraded":"2024-03-11T21:27:14.205761+0000","last_fullsized":"2024-03-11T21:27:14.205761+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"71'2","last_scrub_stamp":"2024-03-11T21:26:42.662979+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:37.512018+0000","last_clean_scrub_stamp":"2024-03-11T21:26:42.662979+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:27:54.085660+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.0014206449999999999,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":85,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-03-11T21:27:14.317489+0000","last_change":"2024-03-11T21:26:42.171361+0000","last_active":"2024-03-11T21:27:14.317489+0000","last_peered":"2024-03-11T21:27:14.317489+0000","last_clean":"2024-03-11T21:27:14.317489+0000","last_became_active":"2024-03-11T21:26:11.216188+0000","last_became_peered":"2024-03-11T21:26:11.216188+0000","last_unstale":"2024-03-11T21:27:14.317489+0000","last_undegraded":"2024-03-11T21:27:14.317489+0000","last_fullsized":"2024-03-11T21:27:14.317489+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:26:42.171256+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:37.512018+0000","last_clean_scrub_stamp":"2024-03-11T21:26:42.171256+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:27:58.632937+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,4],"acting":[5,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"87'128","reported_seq":324,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-03-11T21:27:14.318969+0000","last_change":"2024-03-11T21:26:11.340374+0000","last_active":"2024-03-11T21:27:14.318969+0000","last_peered":"2024-03-11T21:27:14.318969+0000","last_clean":"2024-03-11T21:27:14.318969+0000","last_became_active":"2024-03-11T21:25:30.502597+0000","last_became_peered":"2024-03-11T21:25:30.502597+0000","last_unstale":"2024-03-11T21:27:14.318969+0000","last_undegraded":"2024-03-11T21:27:14.318969+0000","last_fullsized":"2024-03-11T21:27:14.318969+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'117","last_scrub_stamp":"2024-03-11T21:26:11.340228+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:24.261974+0000","last_clean_scrub_stamp":"2024-03-11T21:26:11.340228+0000","objects_scrubbed":2,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:27:32.864192+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":212,"num_read_kb":306,"num_write":179,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1771104,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.5","version":"74'16","reported_seq":117,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-03-11T21:27:14.462266+0000","last_change":"2024-03-11T21:27:13.982835+0000","last_active":"2024-03-11T21:27:14.462266+0000","last_peered":"2024-03-11T21:27:14.462266+0000","last_clean":"2024-03-11T21:27:14.462266+0000","last_became_active":"2024-03-11T21:25:40.761457+0000","last_became_peered":"2024-03-11T21:25:40.761457+0000","last_unstale":"2024-03-11T21:27:14.462266+0000","last_undegraded":"2024-03-11T21:27:14.462266+0000","last_fullsized":"2024-03-11T21:27:14.462266+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'16","last_scrub_stamp":"2024-03-11T21:27:13.982681+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:39.525554+0000","last_clean_scrub_stamp":"2024-03-11T21:27:13.982681+0000","objects_scrubbed":8,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:28:28.254017+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":154,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":25,"num_read_kb":15,"num_write":13,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":71,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-03-11T21:27:14.319404+0000","last_change":"2024-03-11T21:26:58.350384+0000","last_active":"2024-03-11T21:27:14.319404+0000","last_peered":"2024-03-11T21:27:14.319404+0000","last_clean":"2024-03-11T21:27:14.319404+0000","last_became_active":"2024-03-11T21:25:42.586226+0000","last_became_peered":"2024-03-11T21:25:42.586226+0000","last_unstale":"2024-03-11T21:27:14.319404+0000","last_undegraded":"2024-03-11T21:27:14.319404+0000","last_fullsized":"2024-03-11T21:27:14.319404+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:26:58.350270+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:41.544550+0000","last_clean_scrub_stamp":"2024-03-11T21:26:58.350270+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:28:06.391624+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,5],"acting":[7,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.4","version":"0'0","reported_seq":67,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-03-11T21:27:14.472281+0000","last_change":"2024-03-11T21:27:00.119584+0000","last_active":"2024-03-11T21:27:14.472281+0000","last_peered":"2024-03-11T21:27:14.472281+0000","last_clean":"2024-03-11T21:27:14.472281+0000","last_became_active":"2024-03-11T21:25:44.594188+0000","last_became_peered":"2024-03-11T21:25:44.594188+0000","last_unstale":"2024-03-11T21:27:14.472281+0000","last_undegraded":"2024-03-11T21:27:14.472281+0000","last_fullsized":"2024-03-11T21:27:14.472281+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:27:00.119508+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:43.559452+0000","last_clean_scrub_stamp":"2024-03-11T21:27:00.119508+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:28:23.003619+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":86,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-03-11T21:27:14.205856+0000","last_change":"2024-03-11T21:26:38.589294+0000","last_active":"2024-03-11T21:27:14.205856+0000","last_peered":"2024-03-11T21:27:14.205856+0000","last_clean":"2024-03-11T21:27:14.205856+0000","last_became_active":"2024-03-11T21:25:38.543834+0000","last_became_peered":"2024-03-11T21:25:38.543834+0000","last_unstale":"2024-03-11T21:27:14.205856+0000","last_undegraded":"2024-03-11T21:27:14.205856+0000","last_fullsized":"2024-03-11T21:27:14.205856+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:26:38.589187+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:37.512018+0000","last_clean_scrub_stamp":"2024-03-11T21:26:38.589187+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:27:54.798633+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0.00089342499999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":79,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-03-11T21:27:14.318857+0000","last_change":"2024-03-11T21:26:51.473044+0000","last_active":"2024-03-11T21:27:14.318857+0000","last_peered":"2024-03-11T21:27:14.318857+0000","last_clean":"2024-03-11T21:27:14.318857+0000","last_became_active":"2024-03-11T21:25:38.541617+0000","last_became_peered":"2024-03-11T21:25:38.541617+0000","last_unstale":"2024-03-11T21:27:14.318857+0000","last_undegraded":"2024-03-11T21:27:14.318857+0000","last_fullsized":"2024-03-11T21:27:14.318857+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:26:51.472965+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:37.512018+0000","last_clean_scrub_stamp":"2024-03-11T21:26:51.472965+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:28:11.852536+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,0],"acting":[7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.6","version":"74'12","reported_seq":93,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-03-11T21:27:14.206256+0000","last_change":"2024-03-11T21:27:01.353450+0000","last_active":"2024-03-11T21:27:14.206256+0000","last_peered":"2024-03-11T21:27:14.206256+0000","last_clean":"2024-03-11T21:27:14.206256+0000","last_became_active":"2024-03-11T21:25:40.550388+0000","last_became_peered":"2024-03-11T21:25:40.550388+0000","last_unstale":"2024-03-11T21:27:14.206256+0000","last_undegraded":"2024-03-11T21:27:14.206256+0000","last_fullsized":"2024-03-11T21:27:14.206256+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'12","last_scrub_stamp":"2024-03-11T21:27:01.353347+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:39.525554+0000","last_clean_scrub_stamp":"2024-03-11T21:27:01.353347+0000","objects_scrubbed":6,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:28:25.846115+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":18,"num_read_kb":12,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":63,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-03-11T21:27:14.317862+0000","last_change":"2024-03-11T21:25:42.572970+0000","last_active":"2024-03-11T21:27:14.317862+0000","last_peered":"2024-03-11T21:27:14.317862+0000","last_clean":"2024-03-11T21:27:14.317862+0000","last_became_active":"2024-03-11T21:25:42.572324+0000","last_became_peered":"2024-03-11T21:25:42.572324+0000","last_unstale":"2024-03-11T21:27:14.317862+0000","last_undegraded":"2024-03-11T21:27:14.317862+0000","last_fullsized":"2024-03-11T21:27:14.317862+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:25:41.544550+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:41.544550+0000","last_clean_scrub_stamp":"2024-03-11T21:25:41.544550+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":67,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-03-11T21:27:14.471780+0000","last_change":"2024-03-11T21:26:59.108203+0000","last_active":"2024-03-11T21:27:14.471780+0000","last_peered":"2024-03-11T21:27:14.471780+0000","last_clean":"2024-03-11T21:27:14.471780+0000","last_became_active":"2024-03-11T21:25:45.301416+0000","last_became_peered":"2024-03-11T21:25:45.301416+0000","last_unstale":"2024-03-11T21:27:14.471780+0000","last_undegraded":"2024-03-11T21:27:14.471780+0000","last_fullsized":"2024-03-11T21:27:14.471780+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:26:59.108111+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:43.559452+0000","last_clean_scrub_stamp":"2024-03-11T21:26:59.108111+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:28:03.778211+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":79,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-03-11T21:27:14.319553+0000","last_change":"2024-03-11T21:26:57.333768+0000","last_active":"2024-03-11T21:27:14.319553+0000","last_peered":"2024-03-11T21:27:14.319553+0000","last_clean":"2024-03-11T21:27:14.319553+0000","last_became_active":"2024-03-11T21:25:38.545148+0000","last_became_peered":"2024-03-11T21:25:38.545148+0000","last_unstale":"2024-03-11T21:27:14.319553+0000","last_undegraded":"2024-03-11T21:27:14.319553+0000","last_fullsized":"2024-03-11T21:27:14.319553+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:26:57.333669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:26:57.333669+0000","last_clean_scrub_stamp":"2024-03-11T21:26:57.333669+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:28:23.923732+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.3","version":"74'19","reported_seq":111,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-03-11T21:27:14.321636+0000","last_change":"2024-03-11T21:27:14.321667+0000","last_active":"2024-03-11T21:27:14.321636+0000","last_peered":"2024-03-11T21:27:14.321636+0000","last_clean":"2024-03-11T21:27:14.321636+0000","last_became_active":"2024-03-11T21:25:40.760594+0000","last_became_peered":"2024-03-11T21:25:40.760594+0000","last_unstale":"2024-03-11T21:27:14.321636+0000","last_undegraded":"2024-03-11T21:27:14.321636+0000","last_fullsized":"2024-03-11T21:27:14.321636+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'19","last_scrub_stamp":"2024-03-11T21:27:14.321574+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:39.525554+0000","last_clean_scrub_stamp":"2024-03-11T21:27:14.321574+0000","objects_scrubbed":8,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:28:21.292097+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":39,"num_read_kb":25,"num_write":22,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":71,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-03-11T21:27:14.461979+0000","last_change":"2024-03-11T21:27:05.035735+0000","last_active":"2024-03-11T21:27:14.461979+0000","last_peered":"2024-03-11T21:27:14.461979+0000","last_clean":"2024-03-11T21:27:14.461979+0000","last_became_active":"2024-03-11T21:25:42.579045+0000","last_became_peered":"2024-03-11T21:25:42.579045+0000","last_unstale":"2024-03-11T21:27:14.461979+0000","last_undegraded":"2024-03-11T21:27:14.461979+0000","last_fullsized":"2024-03-11T21:27:14.461979+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:27:05.035647+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:41.544550+0000","last_clean_scrub_stamp":"2024-03-11T21:27:05.035647+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:28:32.345929+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":67,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-03-11T21:27:14.201234+0000","last_change":"2024-03-11T21:26:50.836566+0000","last_active":"2024-03-11T21:27:14.201234+0000","last_peered":"2024-03-11T21:27:14.201234+0000","last_clean":"2024-03-11T21:27:14.201234+0000","last_became_active":"2024-03-11T21:25:44.583638+0000","last_became_peered":"2024-03-11T21:25:44.583638+0000","last_unstale":"2024-03-11T21:27:14.201234+0000","last_undegraded":"2024-03-11T21:27:14.201234+0000","last_fullsized":"2024-03-11T21:27:14.201234+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:26:50.836479+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:43.559452+0000","last_clean_scrub_stamp":"2024-03-11T21:26:50.836479+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:28:03.817969+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.6","version":"67'1","reported_seq":80,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-03-11T21:27:14.473323+0000","last_change":"2024-03-11T21:27:07.129584+0000","last_active":"2024-03-11T21:27:14.473323+0000","last_peered":"2024-03-11T21:27:14.473323+0000","last_clean":"2024-03-11T21:27:14.473323+0000","last_became_active":"2024-03-11T21:25:38.536142+0000","last_became_peered":"2024-03-11T21:25:38.536142+0000","last_unstale":"2024-03-11T21:27:14.473323+0000","last_undegraded":"2024-03-11T21:27:14.473323+0000","last_fullsized":"2024-03-11T21:27:14.473323+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'1","last_scrub_stamp":"2024-03-11T21:27:07.129494+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:37.512018+0000","last_clean_scrub_stamp":"2024-03-11T21:27:07.129494+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:28:12.668804+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,4],"acting":[1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.0","version":"93'26","reported_seq":115,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-03-11T21:27:14.205362+0000","last_change":"2024-03-11T21:26:40.680446+0000","last_active":"2024-03-11T21:27:14.205362+0000","last_peered":"2024-03-11T21:27:14.205362+0000","last_clean":"2024-03-11T21:27:14.205362+0000","last_became_active":"2024-03-11T21:25:40.761716+0000","last_became_peered":"2024-03-11T21:25:40.761716+0000","last_unstale":"2024-03-11T21:27:14.205362+0000","last_undegraded":"2024-03-11T21:27:14.205362+0000","last_fullsized":"2024-03-11T21:27:14.205362+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"93'26","last_scrub_stamp":"2024-03-11T21:26:40.680324+0000","last_deep_scrub":"93'26","last_deep_scrub_stamp":"2024-03-11T21:26:40.680324+0000","last_clean_scrub_stamp":"2024-03-11T21:26:40.680324+0000","objects_scrubbed":8,"log_size":26,"log_dups_size":0,"ondisk_log_size":26,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:28:00.090900+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":34,"num_read_kb":22,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":71,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-03-11T21:27:14.201040+0000","last_change":"2024-03-11T21:26:51.863702+0000","last_active":"2024-03-11T21:27:14.201040+0000","last_peered":"2024-03-11T21:27:14.201040+0000","last_clean":"2024-03-11T21:27:14.201040+0000","last_became_active":"2024-03-11T21:25:42.583647+0000","last_became_peered":"2024-03-11T21:25:42.583647+0000","last_unstale":"2024-03-11T21:27:14.201040+0000","last_undegraded":"2024-03-11T21:27:14.201040+0000","last_fullsized":"2024-03-11T21:27:14.201040+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:26:51.863621+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:26:51.863621+0000","last_clean_scrub_stamp":"2024-03-11T21:26:51.863621+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:27:53.832869+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,7],"acting":[4,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.3","version":"0'0","reported_seq":67,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-03-11T21:27:14.319559+0000","last_change":"2024-03-11T21:26:54.421277+0000","last_active":"2024-03-11T21:27:14.319559+0000","last_peered":"2024-03-11T21:27:14.319559+0000","last_clean":"2024-03-11T21:27:14.319559+0000","last_became_active":"2024-03-11T21:25:45.302719+0000","last_became_peered":"2024-03-11T21:25:45.302719+0000","last_unstale":"2024-03-11T21:27:14.319559+0000","last_undegraded":"2024-03-11T21:27:14.319559+0000","last_fullsized":"2024-03-11T21:27:14.319559+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:26:54.421196+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:43.559452+0000","last_clean_scrub_stamp":"2024-03-11T21:26:54.421196+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:28:15.162446+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,2],"acting":[7,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":79,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-03-11T21:27:14.462549+0000","last_change":"2024-03-11T21:26:47.927498+0000","last_active":"2024-03-11T21:27:14.462549+0000","last_peered":"2024-03-11T21:27:14.462549+0000","last_clean":"2024-03-11T21:27:14.462549+0000","last_became_active":"2024-03-11T21:25:38.540013+0000","last_became_peered":"2024-03-11T21:25:38.540013+0000","last_unstale":"2024-03-11T21:27:14.462549+0000","last_undegraded":"2024-03-11T21:27:14.462549+0000","last_fullsized":"2024-03-11T21:27:14.462549+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:26:47.927394+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:37.512018+0000","last_clean_scrub_stamp":"2024-03-11T21:26:47.927394+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:28:02.205612+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,2],"acting":[6,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.1","version":"74'14","reported_seq":88,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-03-11T21:27:14.200818+0000","last_change":"2024-03-11T21:25:40.546609+0000","last_active":"2024-03-11T21:27:14.200818+0000","last_peered":"2024-03-11T21:27:14.200818+0000","last_clean":"2024-03-11T21:27:14.200818+0000","last_became_active":"2024-03-11T21:25:40.546219+0000","last_became_peered":"2024-03-11T21:25:40.546219+0000","last_unstale":"2024-03-11T21:27:14.200818+0000","last_undegraded":"2024-03-11T21:27:14.200818+0000","last_fullsized":"2024-03-11T21:27:14.200818+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:25:39.525554+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:39.525554+0000","last_clean_scrub_stamp":"2024-03-11T21:25:39.525554+0000","objects_scrubbed":0,"log_size":14,"log_dups_size":0,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":21,"num_read_kb":14,"num_write":14,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6],"acting":[4,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.0","version":"74'8","reported_seq":91,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-03-11T21:27:14.205292+0000","last_change":"2024-03-11T21:25:42.577103+0000","last_active":"2024-03-11T21:27:14.205292+0000","last_peered":"2024-03-11T21:27:14.205292+0000","last_clean":"2024-03-11T21:27:14.205292+0000","last_became_active":"2024-03-11T21:25:42.576126+0000","last_became_peered":"2024-03-11T21:25:42.576126+0000","last_unstale":"2024-03-11T21:27:14.205292+0000","last_undegraded":"2024-03-11T21:27:14.205292+0000","last_fullsized":"2024-03-11T21:27:14.205292+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:25:41.544550+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:41.544550+0000","last_clean_scrub_stamp":"2024-03-11T21:25:41.544550+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.c","version":"0'0","reported_seq":59,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-03-11T21:27:14.206765+0000","last_change":"2024-03-11T21:25:45.303235+0000","last_active":"2024-03-11T21:27:14.206765+0000","last_peered":"2024-03-11T21:27:14.206765+0000","last_clean":"2024-03-11T21:27:14.206765+0000","last_became_active":"2024-03-11T21:25:45.302207+0000","last_became_peered":"2024-03-11T21:25:45.302207+0000","last_unstale":"2024-03-11T21:27:14.206765+0000","last_undegraded":"2024-03-11T21:27:14.206765+0000","last_fullsized":"2024-03-11T21:27:14.206765+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:25:43.559452+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:43.559452+0000","last_clean_scrub_stamp":"2024-03-11T21:25:43.559452+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.f","version":"0'0","reported_seq":71,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-03-11T21:27:14.317762+0000","last_change":"2024-03-11T21:26:51.925489+0000","last_active":"2024-03-11T21:27:14.317762+0000","last_peered":"2024-03-11T21:27:14.317762+0000","last_clean":"2024-03-11T21:27:14.317762+0000","last_became_active":"2024-03-11T21:25:42.575935+0000","last_became_peered":"2024-03-11T21:25:42.575935+0000","last_unstale":"2024-03-11T21:27:14.317762+0000","last_undegraded":"2024-03-11T21:27:14.317762+0000","last_fullsized":"2024-03-11T21:27:14.317762+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:26:51.925329+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:41.544550+0000","last_clean_scrub_stamp":"2024-03-11T21:26:51.925329+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:28:02.478228+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.8","version":"0'0","reported_seq":79,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-03-11T21:27:14.318972+0000","last_change":"2024-03-11T21:26:48.424013+0000","last_active":"2024-03-11T21:27:14.318972+0000","last_peered":"2024-03-11T21:27:14.318972+0000","last_clean":"2024-03-11T21:27:14.318972+0000","last_became_active":"2024-03-11T21:25:38.540910+0000","last_became_peered":"2024-03-11T21:25:38.540910+0000","last_unstale":"2024-03-11T21:27:14.318972+0000","last_undegraded":"2024-03-11T21:27:14.318972+0000","last_fullsized":"2024-03-11T21:27:14.318972+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:26:48.423910+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:37.512018+0000","last_clean_scrub_stamp":"2024-03-11T21:26:48.423910+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:28:17.378160+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,1],"acting":[7,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.e","version":"74'11","reported_seq":94,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-03-11T21:27:14.201800+0000","last_change":"2024-03-11T21:26:57.843770+0000","last_active":"2024-03-11T21:27:14.201800+0000","last_peered":"2024-03-11T21:27:14.201800+0000","last_clean":"2024-03-11T21:27:14.201800+0000","last_became_active":"2024-03-11T21:25:40.552178+0000","last_became_peered":"2024-03-11T21:25:40.552178+0000","last_unstale":"2024-03-11T21:27:14.201800+0000","last_undegraded":"2024-03-11T21:27:14.201800+0000","last_fullsized":"2024-03-11T21:27:14.201800+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'11","last_scrub_stamp":"2024-03-11T21:26:57.843615+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:39.525554+0000","last_clean_scrub_stamp":"2024-03-11T21:26:57.843615+0000","objects_scrubbed":5,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:28:03.562699+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.d","version":"0'0","reported_seq":67,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-03-11T21:27:14.317230+0000","last_change":"2024-03-11T21:26:54.957670+0000","last_active":"2024-03-11T21:27:14.317230+0000","last_peered":"2024-03-11T21:27:14.317230+0000","last_clean":"2024-03-11T21:27:14.317230+0000","last_became_active":"2024-03-11T21:25:44.590333+0000","last_became_peered":"2024-03-11T21:25:44.590333+0000","last_unstale":"2024-03-11T21:27:14.317230+0000","last_undegraded":"2024-03-11T21:27:14.317230+0000","last_fullsized":"2024-03-11T21:27:14.317230+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:26:54.957577+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:43.559452+0000","last_clean_scrub_stamp":"2024-03-11T21:26:54.957577+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:28:05.595894+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.e","version":"74'8","reported_seq":91,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-03-11T21:27:14.201637+0000","last_change":"2024-03-11T21:25:42.577257+0000","last_active":"2024-03-11T21:27:14.201637+0000","last_peered":"2024-03-11T21:27:14.201637+0000","last_clean":"2024-03-11T21:27:14.201637+0000","last_became_active":"2024-03-11T21:25:42.576742+0000","last_became_peered":"2024-03-11T21:25:42.576742+0000","last_unstale":"2024-03-11T21:27:14.201637+0000","last_undegraded":"2024-03-11T21:27:14.201637+0000","last_fullsized":"2024-03-11T21:27:14.201637+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:25:41.544550+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:41.544550+0000","last_clean_scrub_stamp":"2024-03-11T21:25:41.544550+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,0],"acting":[4,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.9","version":"0'0","reported_seq":79,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-03-11T21:27:14.472098+0000","last_change":"2024-03-11T21:26:50.991832+0000","last_active":"2024-03-11T21:27:14.472098+0000","last_peered":"2024-03-11T21:27:14.472098+0000","last_clean":"2024-03-11T21:27:14.472098+0000","last_became_active":"2024-03-11T21:25:38.540976+0000","last_became_peered":"2024-03-11T21:25:38.540976+0000","last_unstale":"2024-03-11T21:27:14.472098+0000","last_undegraded":"2024-03-11T21:27:14.472098+0000","last_fullsized":"2024-03-11T21:27:14.472098+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:26:50.991712+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:26:50.991712+0000","last_clean_scrub_stamp":"2024-03-11T21:26:50.991712+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:27:58.101352+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,3],"acting":[1,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.f","version":"74'15","reported_seq":53,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-03-11T21:27:14.472856+0000","last_change":"2024-03-11T21:26:11.226642+0000","last_active":"2024-03-11T21:27:14.472856+0000","last_peered":"2024-03-11T21:27:14.472856+0000","last_clean":"2024-03-11T21:27:14.472856+0000","last_became_active":"2024-03-11T21:26:11.216224+0000","last_became_peered":"2024-03-11T21:26:11.216224+0000","last_unstale":"2024-03-11T21:27:14.472856+0000","last_undegraded":"2024-03-11T21:27:14.472856+0000","last_fullsized":"2024-03-11T21:27:14.472856+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:25:39.525554+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:39.525554+0000","last_clean_scrub_stamp":"2024-03-11T21:25:39.525554+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,4],"acting":[2,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.e","version":"0'0","reported_seq":59,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-03-11T21:27:14.201575+0000","last_change":"2024-03-11T21:25:44.590476+0000","last_active":"2024-03-11T21:27:14.201575+0000","last_peered":"2024-03-11T21:27:14.201575+0000","last_clean":"2024-03-11T21:27:14.201575+0000","last_became_active":"2024-03-11T21:25:44.590198+0000","last_became_peered":"2024-03-11T21:25:44.590198+0000","last_unstale":"2024-03-11T21:27:14.201575+0000","last_undegraded":"2024-03-11T21:27:14.201575+0000","last_fullsized":"2024-03-11T21:27:14.201575+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:25:43.559452+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:43.559452+0000","last_clean_scrub_stamp":"2024-03-11T21:25:43.559452+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.d","version":"74'8","reported_seq":99,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-03-11T21:27:14.472857+0000","last_change":"2024-03-11T21:26:59.021603+0000","last_active":"2024-03-11T21:27:14.472857+0000","last_peered":"2024-03-11T21:27:14.472857+0000","last_clean":"2024-03-11T21:27:14.472857+0000","last_became_active":"2024-03-11T21:25:42.575583+0000","last_became_peered":"2024-03-11T21:25:42.575583+0000","last_unstale":"2024-03-11T21:27:14.472857+0000","last_undegraded":"2024-03-11T21:27:14.472857+0000","last_fullsized":"2024-03-11T21:27:14.472857+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'8","last_scrub_stamp":"2024-03-11T21:26:59.021523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:41.544550+0000","last_clean_scrub_stamp":"2024-03-11T21:26:59.021523+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:28:21.121669+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,5],"acting":[2,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.a","version":"0'0","reported_seq":79,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-03-11T21:27:14.472540+0000","last_change":"2024-03-11T21:26:52.008094+0000","last_active":"2024-03-11T21:27:14.472540+0000","last_peered":"2024-03-11T21:27:14.472540+0000","last_clean":"2024-03-11T21:27:14.472540+0000","last_became_active":"2024-03-11T21:25:38.541045+0000","last_became_peered":"2024-03-11T21:25:38.541045+0000","last_unstale":"2024-03-11T21:27:14.472540+0000","last_undegraded":"2024-03-11T21:27:14.472540+0000","last_fullsized":"2024-03-11T21:27:14.472540+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:26:52.007996+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:37.512018+0000","last_clean_scrub_stamp":"2024-03-11T21:26:52.007996+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:27:52.862240+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,7],"acting":[1,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.c","version":"74'10","reported_seq":90,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-03-11T21:27:14.201351+0000","last_change":"2024-03-11T21:27:00.807955+0000","last_active":"2024-03-11T21:27:14.201351+0000","last_peered":"2024-03-11T21:27:14.201351+0000","last_clean":"2024-03-11T21:27:14.201351+0000","last_became_active":"2024-03-11T21:25:40.550885+0000","last_became_peered":"2024-03-11T21:25:40.550885+0000","last_unstale":"2024-03-11T21:27:14.201351+0000","last_undegraded":"2024-03-11T21:27:14.201351+0000","last_fullsized":"2024-03-11T21:27:14.201351+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'10","last_scrub_stamp":"2024-03-11T21:27:00.807849+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:39.525554+0000","last_clean_scrub_stamp":"2024-03-11T21:27:00.807849+0000","objects_scrubbed":5,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:28:27.818066+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.f","version":"0'0","reported_seq":67,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-03-11T21:27:14.472963+0000","last_change":"2024-03-11T21:27:03.011122+0000","last_active":"2024-03-11T21:27:14.472963+0000","last_peered":"2024-03-11T21:27:14.472963+0000","last_clean":"2024-03-11T21:27:14.472963+0000","last_became_active":"2024-03-11T21:25:44.593414+0000","last_became_peered":"2024-03-11T21:25:44.593414+0000","last_unstale":"2024-03-11T21:27:14.472963+0000","last_undegraded":"2024-03-11T21:27:14.472963+0000","last_fullsized":"2024-03-11T21:27:14.472963+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:27:03.010969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:27:03.010969+0000","last_clean_scrub_stamp":"2024-03-11T21:27:03.010969+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:28:22.636838+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,4],"acting":[2,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":63,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-03-11T21:27:14.472377+0000","last_change":"2024-03-11T21:25:42.579815+0000","last_active":"2024-03-11T21:27:14.472377+0000","last_peered":"2024-03-11T21:27:14.472377+0000","last_clean":"2024-03-11T21:27:14.472377+0000","last_became_active":"2024-03-11T21:25:42.579409+0000","last_became_peered":"2024-03-11T21:25:42.579409+0000","last_unstale":"2024-03-11T21:27:14.472377+0000","last_undegraded":"2024-03-11T21:27:14.472377+0000","last_fullsized":"2024-03-11T21:27:14.472377+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:25:41.544550+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:41.544550+0000","last_clean_scrub_stamp":"2024-03-11T21:25:41.544550+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.b","version":"0'0","reported_seq":71,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-03-11T21:27:14.319158+0000","last_change":"2024-03-11T21:25:38.544246+0000","last_active":"2024-03-11T21:27:14.319158+0000","last_peered":"2024-03-11T21:27:14.319158+0000","last_clean":"2024-03-11T21:27:14.319158+0000","last_became_active":"2024-03-11T21:25:38.543377+0000","last_became_peered":"2024-03-11T21:25:38.543377+0000","last_unstale":"2024-03-11T21:27:14.319158+0000","last_undegraded":"2024-03-11T21:27:14.319158+0000","last_fullsized":"2024-03-11T21:27:14.319158+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:25:37.512018+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:37.512018+0000","last_clean_scrub_stamp":"2024-03-11T21:25:37.512018+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,5],"acting":[7,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.d","version":"74'17","reported_seq":103,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-03-11T21:27:14.201528+0000","last_change":"2024-03-11T21:27:02.816136+0000","last_active":"2024-03-11T21:27:14.201528+0000","last_peered":"2024-03-11T21:27:14.201528+0000","last_clean":"2024-03-11T21:27:14.201528+0000","last_became_active":"2024-03-11T21:25:40.553454+0000","last_became_peered":"2024-03-11T21:25:40.553454+0000","last_unstale":"2024-03-11T21:27:14.201528+0000","last_undegraded":"2024-03-11T21:27:14.201528+0000","last_fullsized":"2024-03-11T21:27:14.201528+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'17","last_scrub_stamp":"2024-03-11T21:27:02.815980+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:39.525554+0000","last_clean_scrub_stamp":"2024-03-11T21:27:02.815980+0000","objects_scrubbed":8,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:28:06.520165+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":29,"num_read_kb":19,"num_write":18,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,1],"acting":[4,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":67,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-03-11T21:27:14.318810+0000","last_change":"2024-03-11T21:27:07.304361+0000","last_active":"2024-03-11T21:27:14.318810+0000","last_peered":"2024-03-11T21:27:14.318810+0000","last_clean":"2024-03-11T21:27:14.318810+0000","last_became_active":"2024-03-11T21:25:44.589973+0000","last_became_peered":"2024-03-11T21:25:44.589973+0000","last_unstale":"2024-03-11T21:27:14.318810+0000","last_undegraded":"2024-03-11T21:27:14.318810+0000","last_fullsized":"2024-03-11T21:27:14.318810+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:27:07.304264+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:43.559452+0000","last_clean_scrub_stamp":"2024-03-11T21:27:07.304264+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:28:32.511531+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,3],"acting":[7,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":63,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-03-11T21:27:14.472632+0000","last_change":"2024-03-11T21:25:42.574877+0000","last_active":"2024-03-11T21:27:14.472632+0000","last_peered":"2024-03-11T21:27:14.472632+0000","last_clean":"2024-03-11T21:27:14.472632+0000","last_became_active":"2024-03-11T21:25:42.574515+0000","last_became_peered":"2024-03-11T21:25:42.574515+0000","last_unstale":"2024-03-11T21:27:14.472632+0000","last_undegraded":"2024-03-11T21:27:14.472632+0000","last_fullsized":"2024-03-11T21:27:14.472632+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:25:41.544550+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:41.544550+0000","last_clean_scrub_stamp":"2024-03-11T21:25:41.544550+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,5],"acting":[2,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.c","version":"0'0","reported_seq":71,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-03-11T21:27:14.473014+0000","last_change":"2024-03-11T21:25:38.533719+0000","last_active":"2024-03-11T21:27:14.473014+0000","last_peered":"2024-03-11T21:27:14.473014+0000","last_clean":"2024-03-11T21:27:14.473014+0000","last_became_active":"2024-03-11T21:25:38.533240+0000","last_became_peered":"2024-03-11T21:25:38.533240+0000","last_unstale":"2024-03-11T21:27:14.473014+0000","last_undegraded":"2024-03-11T21:27:14.473014+0000","last_fullsized":"2024-03-11T21:27:14.473014+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:25:37.512018+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:37.512018+0000","last_clean_scrub_stamp":"2024-03-11T21:25:37.512018+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,0],"acting":[2,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.a","version":"74'19","reported_seq":106,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-03-11T21:27:14.461847+0000","last_change":"2024-03-11T21:27:07.010787+0000","last_active":"2024-03-11T21:27:14.461847+0000","last_peered":"2024-03-11T21:27:14.461847+0000","last_clean":"2024-03-11T21:27:14.461847+0000","last_became_active":"2024-03-11T21:25:40.760698+0000","last_became_peered":"2024-03-11T21:25:40.760698+0000","last_unstale":"2024-03-11T21:27:14.461847+0000","last_undegraded":"2024-03-11T21:27:14.461847+0000","last_fullsized":"2024-03-11T21:27:14.461847+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'19","last_scrub_stamp":"2024-03-11T21:27:07.010662+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:39.525554+0000","last_clean_scrub_stamp":"2024-03-11T21:27:07.010662+0000","objects_scrubbed":9,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:28:22.036849+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":32,"num_read_kb":21,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,7],"acting":[6,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":67,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-03-11T21:27:14.205555+0000","last_change":"2024-03-11T21:27:11.362017+0000","last_active":"2024-03-11T21:27:14.205555+0000","last_peered":"2024-03-11T21:27:14.205555+0000","last_clean":"2024-03-11T21:27:14.205555+0000","last_became_active":"2024-03-11T21:25:44.584154+0000","last_became_peered":"2024-03-11T21:25:44.584154+0000","last_unstale":"2024-03-11T21:27:14.205555+0000","last_undegraded":"2024-03-11T21:27:14.205555+0000","last_fullsized":"2024-03-11T21:27:14.205555+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:27:11.361913+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:27:11.361913+0000","last_clean_scrub_stamp":"2024-03-11T21:27:11.361913+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:28:25.991645+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":71,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-03-11T21:27:14.472335+0000","last_change":"2024-03-11T21:27:10.909699+0000","last_active":"2024-03-11T21:27:14.472335+0000","last_peered":"2024-03-11T21:27:14.472335+0000","last_clean":"2024-03-11T21:27:14.472335+0000","last_became_active":"2024-03-11T21:25:42.576679+0000","last_became_peered":"2024-03-11T21:25:42.576679+0000","last_unstale":"2024-03-11T21:27:14.472335+0000","last_undegraded":"2024-03-11T21:27:14.472335+0000","last_fullsized":"2024-03-11T21:27:14.472335+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:27:10.909543+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:41.544550+0000","last_clean_scrub_stamp":"2024-03-11T21:27:10.909543+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:28:29.725395+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,3],"acting":[2,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.d","version":"0'0","reported_seq":79,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-03-11T21:27:14.472447+0000","last_change":"2024-03-11T21:26:45.885219+0000","last_active":"2024-03-11T21:27:14.472447+0000","last_peered":"2024-03-11T21:27:14.472447+0000","last_clean":"2024-03-11T21:27:14.472447+0000","last_became_active":"2024-03-11T21:25:38.539681+0000","last_became_peered":"2024-03-11T21:25:38.539681+0000","last_unstale":"2024-03-11T21:27:14.472447+0000","last_undegraded":"2024-03-11T21:27:14.472447+0000","last_fullsized":"2024-03-11T21:27:14.472447+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:26:45.885144+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:37.512018+0000","last_clean_scrub_stamp":"2024-03-11T21:26:45.885144+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:27:47.637748+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,3],"acting":[1,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.b","version":"74'9","reported_seq":83,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-03-11T21:27:14.207189+0000","last_change":"2024-03-11T21:25:40.549925+0000","last_active":"2024-03-11T21:27:14.207189+0000","last_peered":"2024-03-11T21:27:14.207189+0000","last_clean":"2024-03-11T21:27:14.207189+0000","last_became_active":"2024-03-11T21:25:40.549612+0000","last_became_peered":"2024-03-11T21:25:40.549612+0000","last_unstale":"2024-03-11T21:27:14.207189+0000","last_undegraded":"2024-03-11T21:27:14.207189+0000","last_fullsized":"2024-03-11T21:27:14.207189+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:25:39.525554+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:39.525554+0000","last_clean_scrub_stamp":"2024-03-11T21:25:39.525554+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":67,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-03-11T21:27:14.317001+0000","last_change":"2024-03-11T21:26:55.966077+0000","last_active":"2024-03-11T21:27:14.317001+0000","last_peered":"2024-03-11T21:27:14.317001+0000","last_clean":"2024-03-11T21:27:14.317001+0000","last_became_active":"2024-03-11T21:25:45.300333+0000","last_became_peered":"2024-03-11T21:25:45.300333+0000","last_unstale":"2024-03-11T21:27:14.317001+0000","last_undegraded":"2024-03-11T21:27:14.317001+0000","last_fullsized":"2024-03-11T21:27:14.317001+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:26:55.965980+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:43.559452+0000","last_clean_scrub_stamp":"2024-03-11T21:26:55.965980+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:28:02.706121+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,0],"acting":[5,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.e","version":"0'0","reported_seq":79,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-03-11T21:27:14.473183+0000","last_change":"2024-03-11T21:26:57.974916+0000","last_active":"2024-03-11T21:27:14.473183+0000","last_peered":"2024-03-11T21:27:14.473183+0000","last_clean":"2024-03-11T21:27:14.473183+0000","last_became_active":"2024-03-11T21:25:38.539954+0000","last_became_peered":"2024-03-11T21:25:38.539954+0000","last_unstale":"2024-03-11T21:27:14.473183+0000","last_undegraded":"2024-03-11T21:27:14.473183+0000","last_fullsized":"2024-03-11T21:27:14.473183+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:26:57.974827+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:37.512018+0000","last_clean_scrub_stamp":"2024-03-11T21:26:57.974827+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:28:10.300007+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,7],"acting":[2,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.8","version":"74'15","reported_seq":100,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-03-11T21:27:14.316858+0000","last_change":"2024-03-11T21:26:50.934584+0000","last_active":"2024-03-11T21:27:14.316858+0000","last_peered":"2024-03-11T21:27:14.316858+0000","last_clean":"2024-03-11T21:27:14.316858+0000","last_became_active":"2024-03-11T21:25:40.761938+0000","last_became_peered":"2024-03-11T21:25:40.761938+0000","last_unstale":"2024-03-11T21:27:14.316858+0000","last_undegraded":"2024-03-11T21:27:14.316858+0000","last_fullsized":"2024-03-11T21:27:14.316858+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'15","last_scrub_stamp":"2024-03-11T21:26:50.934489+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:39.525554+0000","last_clean_scrub_stamp":"2024-03-11T21:26:50.934489+0000","objects_scrubbed":7,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:28:08.719408+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.9","version":"94'9","reported_seq":103,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-03-11T21:27:14.319095+0000","last_change":"2024-03-11T21:26:53.422673+0000","last_active":"2024-03-11T21:27:14.319095+0000","last_peered":"2024-03-11T21:27:14.319095+0000","last_clean":"2024-03-11T21:27:14.319095+0000","last_became_active":"2024-03-11T21:25:42.584461+0000","last_became_peered":"2024-03-11T21:25:42.584461+0000","last_unstale":"2024-03-11T21:27:14.319095+0000","last_undegraded":"2024-03-11T21:27:14.319095+0000","last_fullsized":"2024-03-11T21:27:14.319095+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"94'9","last_scrub_stamp":"2024-03-11T21:26:53.422552+0000","last_deep_scrub":"94'9","last_deep_scrub_stamp":"2024-03-11T21:26:53.422552+0000","last_clean_scrub_stamp":"2024-03-11T21:26:53.422552+0000","objects_scrubbed":1,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:28:07.835997+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":6,"num_pg":32,"stat_sum":{"num_bytes":383,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1149,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2,"ondisk_log_size":2,"up":96,"acting":96,"num_store_stats":8},{"poolid":5,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":65,"ondisk_log_size":65,"up":96,"acting":96,"num_store_stats":8},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":3702,"num_objects":178,"num_object_clones":0,"num_object_copies":534,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":178,"num_whiteouts":0,"num_read":698,"num_read_kb":455,"num_write":417,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":417792,"data_stored":11106,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":412,"ondisk_log_size":412,"up":96,"acting":96,"num_store_stats":8},{"poolid":3,"num_pg":3,"stat_sum":{"num_bytes":408,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":93,"num_read_kb":88,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1224,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":9,"acting":9,"num_store_stats":7},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":212,"num_read_kb":306,"num_write":179,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1771104,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1781760,"data_stored":1771104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":128,"ondisk_log_size":128,"up":3,"acting":3,"num_store_stats":3},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":1531,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":24,"num_read_kb":24,"num_write":10,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":4593,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":6,"ondisk_log_size":6,"up":96,"acting":96,"num_store_stats":8}],"osd_stats":[{"osd":7,"up_from":60,"seq":257698037783,"num_pgs":56,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28492,"kb_used_data":1352,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93716660,"statfs":{"total":95995035648,"available":95965859840,"internally_reserved":0,"allocated":1384448,"data_stored":955552,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":50,"seq":214748364829,"num_pgs":44,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28488,"kb_used_data":1348,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93716664,"statfs":{"total":95995035648,"available":95965863936,"internally_reserved":0,"allocated":1380352,"data_stored":955845,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574903,"num_pgs":53,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27816,"kb_used_data":800,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717336,"statfs":{"total":95995035648,"available":95966552064,"internally_reserved":0,"allocated":819200,"data_stored":365585,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738428,"num_pgs":47,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28496,"kb_used_data":1356,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93716656,"statfs":{"total":95995035648,"available":95965855744,"internally_reserved":0,"allocated":1388544,"data_stored":955769,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411377,"num_pgs":39,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27900,"kb_used_data":760,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717252,"statfs":{"total":95995035648,"available":95966466048,"internally_reserved":0,"allocated":778240,"data_stored":364809,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":26,"seq":111669149741,"num_pgs":55,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":31976,"kb_used_data":800,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713176,"statfs":{"total":95995035648,"available":95962292224,"internally_reserved":0,"allocated":819200,"data_stored":365919,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":32,"seq":137438953510,"num_pgs":53,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27924,"kb_used_data":784,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717228,"statfs":{"total":95995035648,"available":95966441472,"internally_reserved":0,"allocated":802816,"data_stored":365093,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":41,"seq":176093659169,"num_pgs":49,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32064,"kb_used_data":764,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713088,"statfs":{"total":95995035648,"available":95962202112,"internally_reserved":0,"allocated":782336,"data_stored":364860,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1049,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":46,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":957,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":574,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":436,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1003,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":528,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":1320,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":81920,"data_stored":2118,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":1018,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":1898,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":1474,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":990,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":1034,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":1254,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":370,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":383,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":370,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-11T21:27:20.289 INFO:tasks.cephadm.ceph_manager.ceph:clean! 2024-03-11T21:27:20.290 DEBUG:teuthology.orchestra.run.smithi092:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid 00db0016-dfed-11ee-95c8-87774f69a715 -- ceph osd dump 2024-03-11T21:27:20.533 INFO:teuthology.orchestra.run.smithi092.stderr:Inferring config /var/lib/ceph/00db0016-dfed-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T21:27:20.861 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:27:20 smithi092 ceph-mon[37563]: 5.1b scrub starts 2024-03-11T21:27:20.861 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:27:20 smithi092 ceph-mon[37563]: 5.1b scrub ok 2024-03-11T21:27:20.861 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:27:20 smithi092 ceph-mon[37563]: pgmap v42: 132 pgs: 132 active+clean; 582 KiB data, 228 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-11T21:27:20.861 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:27:20 smithi092 ceph-mon[37563]: from='client.34658 v1:172.21.15.92:0/1248674856' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-11T21:27:20.861 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:27:20 smithi092 ceph-mon[37563]: 4.14 scrub starts 2024-03-11T21:27:20.861 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:27:20 smithi092 ceph-mon[37563]: 4.14 scrub ok 2024-03-11T21:27:20.861 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:27:20 smithi092 ceph-mon[37563]: 5.e scrub starts 2024-03-11T21:27:20.861 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:27:20 smithi092 ceph-mon[37563]: 5.e scrub ok 2024-03-11T21:27:20.862 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:27:20 smithi092 ceph-mon[37563]: 6.1d scrub starts 2024-03-11T21:27:20.862 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:27:20 smithi092 ceph-mon[37563]: 6.1d scrub ok 2024-03-11T21:27:20.862 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:27:20 smithi092 ceph-mon[29572]: 5.1b scrub starts 2024-03-11T21:27:20.862 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:27:20 smithi092 ceph-mon[29572]: 5.1b scrub ok 2024-03-11T21:27:20.862 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:27:20 smithi092 ceph-mon[29572]: pgmap v42: 132 pgs: 132 active+clean; 582 KiB data, 228 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-11T21:27:20.862 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:27:20 smithi092 ceph-mon[29572]: from='client.34658 v1:172.21.15.92:0/1248674856' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-11T21:27:20.862 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:27:20 smithi092 ceph-mon[29572]: 4.14 scrub starts 2024-03-11T21:27:20.862 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:27:20 smithi092 ceph-mon[29572]: 4.14 scrub ok 2024-03-11T21:27:20.862 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:27:20 smithi092 ceph-mon[29572]: 5.e scrub starts 2024-03-11T21:27:20.862 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:27:20 smithi092 ceph-mon[29572]: 5.e scrub ok 2024-03-11T21:27:20.862 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:27:20 smithi092 ceph-mon[29572]: 6.1d scrub starts 2024-03-11T21:27:20.863 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:27:20 smithi092 ceph-mon[29572]: 6.1d scrub ok 2024-03-11T21:27:20.977 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:27:20 smithi097 ceph-mon[33317]: 5.1b scrub starts 2024-03-11T21:27:20.977 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:27:20 smithi097 ceph-mon[33317]: 5.1b scrub ok 2024-03-11T21:27:20.978 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:27:20 smithi097 ceph-mon[33317]: pgmap v42: 132 pgs: 132 active+clean; 582 KiB data, 228 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-11T21:27:20.978 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:27:20 smithi097 ceph-mon[33317]: from='client.34658 v1:172.21.15.92:0/1248674856' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-11T21:27:20.978 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:27:20 smithi097 ceph-mon[33317]: 4.14 scrub starts 2024-03-11T21:27:20.978 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:27:20 smithi097 ceph-mon[33317]: 4.14 scrub ok 2024-03-11T21:27:20.978 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:27:20 smithi097 ceph-mon[33317]: 5.e scrub starts 2024-03-11T21:27:20.978 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:27:20 smithi097 ceph-mon[33317]: 5.e scrub ok 2024-03-11T21:27:20.978 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:27:20 smithi097 ceph-mon[33317]: 6.1d scrub starts 2024-03-11T21:27:20.978 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:27:20 smithi097 ceph-mon[33317]: 6.1d scrub ok 2024-03-11T21:27:21.723 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:27:21 smithi092 ceph-mon[29572]: 5.5 scrub starts 2024-03-11T21:27:21.723 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:27:21 smithi092 ceph-mon[29572]: 5.5 scrub ok 2024-03-11T21:27:21.723 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:27:21 smithi092 ceph-mon[29572]: 6.14 scrub starts 2024-03-11T21:27:21.723 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:27:21 smithi092 ceph-mon[29572]: 6.14 scrub ok 2024-03-11T21:27:21.723 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:27:21 smithi092 ceph-mon[29572]: from='mgr.14607 v1:172.21.15.92:0/1702717597' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T21:27:21.723 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:27:21 smithi092 ceph-mon[29572]: 5.12 deep-scrub starts 2024-03-11T21:27:21.723 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:27:21 smithi092 ceph-mon[29572]: 5.12 deep-scrub ok 2024-03-11T21:27:21.723 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:27:21 smithi092 ceph-mon[37563]: 5.5 scrub starts 2024-03-11T21:27:21.723 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:27:21 smithi092 ceph-mon[37563]: 5.5 scrub ok 2024-03-11T21:27:21.723 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:27:21 smithi092 ceph-mon[37563]: 6.14 scrub starts 2024-03-11T21:27:21.724 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:27:21 smithi092 ceph-mon[37563]: 6.14 scrub ok 2024-03-11T21:27:21.724 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:27:21 smithi092 ceph-mon[37563]: from='mgr.14607 v1:172.21.15.92:0/1702717597' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T21:27:21.724 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:27:21 smithi092 ceph-mon[37563]: 5.12 deep-scrub starts 2024-03-11T21:27:21.724 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:27:21 smithi092 ceph-mon[37563]: 5.12 deep-scrub ok 2024-03-11T21:27:21.724 INFO:journalctl@ceph.alertmanager.a.smithi092.stdout:Mar 11 21:27:21 smithi092 ceph-00db0016-dfed-11ee-95c8-87774f69a715-alertmanager-a[72702]: ts=2024-03-11T21:27:21.488Z caller=cluster.go:698 level=info component=cluster msg="gossip settled; proceeding" elapsed=10.001357713s 2024-03-11T21:27:21.989 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:27:21 smithi097 ceph-mon[33317]: 5.5 scrub starts 2024-03-11T21:27:21.989 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:27:21 smithi097 ceph-mon[33317]: 5.5 scrub ok 2024-03-11T21:27:21.989 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:27:21 smithi097 ceph-mon[33317]: 6.14 scrub starts 2024-03-11T21:27:21.989 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:27:21 smithi097 ceph-mon[33317]: 6.14 scrub ok 2024-03-11T21:27:21.989 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:27:21 smithi097 ceph-mon[33317]: from='mgr.14607 v1:172.21.15.92:0/1702717597' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T21:27:21.989 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:27:21 smithi097 ceph-mon[33317]: 5.12 deep-scrub starts 2024-03-11T21:27:21.989 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:27:21 smithi097 ceph-mon[33317]: 5.12 deep-scrub ok 2024-03-11T21:27:22.254 INFO:journalctl@ceph.iscsi.iscsi.a.smithi097.stdout:Mar 11 21:27:22 smithi097 ceph-00db0016-dfed-11ee-95c8-87774f69a715-iscsi-iscsi-a[60943]: debug there is no tcmu-runner data available 2024-03-11T21:27:22.507 INFO:teuthology.orchestra.run.smithi092.stdout:epoch 97 2024-03-11T21:27:22.507 INFO:teuthology.orchestra.run.smithi092.stdout:fsid 00db0016-dfed-11ee-95c8-87774f69a715 2024-03-11T21:27:22.507 INFO:teuthology.orchestra.run.smithi092.stdout:created 2024-03-11T21:20:10.488059+0000 2024-03-11T21:27:22.508 INFO:teuthology.orchestra.run.smithi092.stdout:modified 2024-03-11T21:27:14.197238+0000 2024-03-11T21:27:22.508 INFO:teuthology.orchestra.run.smithi092.stdout:flags sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit 2024-03-11T21:27:22.508 INFO:teuthology.orchestra.run.smithi092.stdout:crush_version 18 2024-03-11T21:27:22.508 INFO:teuthology.orchestra.run.smithi092.stdout:full_ratio 0.95 2024-03-11T21:27:22.508 INFO:teuthology.orchestra.run.smithi092.stdout:backfillfull_ratio 0.9 2024-03-11T21:27:22.508 INFO:teuthology.orchestra.run.smithi092.stdout:nearfull_ratio 0.85 2024-03-11T21:27:22.508 INFO:teuthology.orchestra.run.smithi092.stdout:require_min_compat_client luminous 2024-03-11T21:27:22.508 INFO:teuthology.orchestra.run.smithi092.stdout:min_compat_client luminous 2024-03-11T21:27:22.508 INFO:teuthology.orchestra.run.smithi092.stdout:require_osd_release squid 2024-03-11T21:27:22.508 INFO:teuthology.orchestra.run.smithi092.stdout:stretch_mode_enabled false 2024-03-11T21:27:22.508 INFO:teuthology.orchestra.run.smithi092.stdout:pool 1 '.mgr' replicated size 3 min_size 2 crush_rule 0 object_hash rjenkins pg_num 1 pgp_num 1 autoscale_mode off last_change 22 flags hashpspool stripe_width 0 pg_num_max 32 pg_num_min 1 application mgr read_balance_score 7.89 2024-03-11T21:27:22.508 INFO:teuthology.orchestra.run.smithi092.stdout:pool 2 '.rgw.root' replicated size 3 min_size 2 crush_rule 0 object_hash rjenkins pg_num 32 pgp_num 32 autoscale_mode off last_change 68 flags hashpspool stripe_width 0 application rgw read_balance_score 1.50 2024-03-11T21:27:22.509 INFO:teuthology.orchestra.run.smithi092.stdout:pool 3 'datapool' replicated size 3 min_size 2 crush_rule 0 object_hash rjenkins pg_num 3 pgp_num 3 autoscale_mode off last_change 71 flags hashpspool,selfmanaged_snaps stripe_width 0 application rbd read_balance_score 2.65 2024-03-11T21:27:22.509 INFO:teuthology.orchestra.run.smithi092.stdout:pool 4 'default.rgw.log' replicated size 3 min_size 2 crush_rule 0 object_hash rjenkins pg_num 32 pgp_num 32 autoscale_mode off last_change 70 flags hashpspool stripe_width 0 application rgw read_balance_score 2.25 2024-03-11T21:27:22.509 INFO:teuthology.orchestra.run.smithi092.stdout:pool 5 'default.rgw.control' replicated size 3 min_size 2 crush_rule 0 object_hash rjenkins pg_num 32 pgp_num 32 autoscale_mode off last_change 72 flags hashpspool stripe_width 0 application rgw read_balance_score 1.25 2024-03-11T21:27:22.509 INFO:teuthology.orchestra.run.smithi092.stdout:pool 6 'default.rgw.meta' replicated size 3 min_size 2 crush_rule 0 object_hash rjenkins pg_num 32 pgp_num 32 autoscale_mode off last_change 74 flags hashpspool stripe_width 0 pg_autoscale_bias 4 application rgw read_balance_score 1.75 2024-03-11T21:27:22.509 INFO:teuthology.orchestra.run.smithi092.stdout:max_osd 8 2024-03-11T21:27:22.509 INFO:teuthology.orchestra.run.smithi092.stdout:osd.0 up in weight 1 up_from 8 up_thru 72 down_at 0 last_clean_interval [0,0) v1:172.21.15.92:6801/2844100201 v1:172.21.15.92:6802/2844100201 exists,up 79f15843-f0ec-44aa-931e-4e07a74d3768 2024-03-11T21:27:22.509 INFO:teuthology.orchestra.run.smithi092.stdout:osd.1 up in weight 1 up_from 13 up_thru 72 down_at 0 last_clean_interval [0,0) v1:172.21.15.92:6805/3577871157 v1:172.21.15.92:6806/3577871157 exists,up 3a5a0e78-1293-4ace-a93f-69156098571d 2024-03-11T21:27:22.509 INFO:teuthology.orchestra.run.smithi092.stdout:osd.2 up in weight 1 up_from 18 up_thru 83 down_at 0 last_clean_interval [0,0) v1:172.21.15.92:6809/760825152 v1:172.21.15.92:6810/760825152 exists,up f207b743-0df4-40ed-a735-950e272fea60 2024-03-11T21:27:22.509 INFO:teuthology.orchestra.run.smithi092.stdout:osd.3 up in weight 1 up_from 26 up_thru 83 down_at 0 last_clean_interval [0,0) v1:172.21.15.92:6813/670596264 v1:172.21.15.92:6814/670596264 exists,up bbb6f69a-8f0f-47e6-937a-062e1902ae13 2024-03-11T21:27:22.509 INFO:teuthology.orchestra.run.smithi092.stdout:osd.4 up in weight 1 up_from 32 up_thru 83 down_at 0 last_clean_interval [0,0) v1:172.21.15.97:6800/420733620 v1:172.21.15.97:6801/420733620 exists,up 00266cbc-d2cf-4dee-8a96-3753ebc95ca8 2024-03-11T21:27:22.509 INFO:teuthology.orchestra.run.smithi092.stdout:osd.5 up in weight 1 up_from 41 up_thru 83 down_at 0 last_clean_interval [0,0) v1:172.21.15.97:6804/312056041 v1:172.21.15.97:6805/312056041 exists,up 593a6f6b-8874-40ae-8848-d51c5723a5e4 2024-03-11T21:27:22.509 INFO:teuthology.orchestra.run.smithi092.stdout:osd.6 up in weight 1 up_from 50 up_thru 70 down_at 0 last_clean_interval [0,0) v1:172.21.15.97:6808/178994999 v1:172.21.15.97:6809/178994999 exists,up 91035ecc-6f1a-4cec-bda6-090af9d53eff 2024-03-11T21:27:22.509 INFO:teuthology.orchestra.run.smithi092.stdout:osd.7 up in weight 1 up_from 60 up_thru 72 down_at 0 last_clean_interval [0,0) v1:172.21.15.97:6812/3241560509 v1:172.21.15.97:6813/3241560509 exists,up 3eb28c24-664a-4bd6-869d-6bf5db98f52f 2024-03-11T21:27:22.510 INFO:teuthology.orchestra.run.smithi092.stdout:pg_upmap_items 2.3 [7,4] 2024-03-11T21:27:22.510 INFO:teuthology.orchestra.run.smithi092.stdout:pg_upmap_items 4.f [1,2] 2024-03-11T21:27:22.510 INFO:teuthology.orchestra.run.smithi092.stdout:pg_upmap_items 4.13 [1,2] 2024-03-11T21:27:22.510 INFO:teuthology.orchestra.run.smithi092.stdout:pg_upmap_items 4.17 [1,5] 2024-03-11T21:27:22.510 INFO:teuthology.orchestra.run.smithi092.stdout:blocklist 172.21.15.92:0/303695351 expires 2024-03-12T21:26:20.943013+0000 2024-03-11T21:27:22.510 INFO:teuthology.orchestra.run.smithi092.stdout:blocklist 172.21.15.92:0/437911229 expires 2024-03-12T21:26:20.943013+0000 2024-03-11T21:27:22.510 INFO:teuthology.orchestra.run.smithi092.stdout:blocklist 172.21.15.92:0/4246054050 expires 2024-03-12T21:26:20.943013+0000 2024-03-11T21:27:22.510 INFO:teuthology.orchestra.run.smithi092.stdout:blocklist 172.21.15.92:0/2912406656 expires 2024-03-12T21:26:20.943013+0000 2024-03-11T21:27:22.510 INFO:teuthology.orchestra.run.smithi092.stdout:blocklist 172.21.15.92:0/2706821241 expires 2024-03-12T21:26:20.943013+0000 2024-03-11T21:27:22.510 INFO:teuthology.orchestra.run.smithi092.stdout:blocklist 172.21.15.92:0/3817549754 expires 2024-03-12T21:21:09.473916+0000 2024-03-11T21:27:22.510 INFO:teuthology.orchestra.run.smithi092.stdout:blocklist 172.21.15.92:0/2211594569 expires 2024-03-12T21:21:09.473916+0000 2024-03-11T21:27:22.510 INFO:teuthology.orchestra.run.smithi092.stdout:blocklist 172.21.15.92:0/3569900179 expires 2024-03-12T21:20:43.677016+0000 2024-03-11T21:27:22.510 INFO:teuthology.orchestra.run.smithi092.stdout:blocklist 172.21.15.92:0/3380359534 expires 2024-03-12T21:20:43.677016+0000 2024-03-11T21:27:22.510 INFO:teuthology.orchestra.run.smithi092.stdout:blocklist 172.21.15.92:0/2509373130 expires 2024-03-12T21:20:43.677016+0000 2024-03-11T21:27:22.511 INFO:teuthology.orchestra.run.smithi092.stdout:blocklist 172.21.15.92:6800/2343383852 expires 2024-03-12T21:26:20.943013+0000 2024-03-11T21:27:22.511 INFO:teuthology.orchestra.run.smithi092.stdout:blocklist 172.21.15.92:6800/1559007462 expires 2024-03-12T21:20:43.677016+0000 2024-03-11T21:27:22.511 INFO:teuthology.orchestra.run.smithi092.stdout:blocklist 172.21.15.92:6800/1270902819 expires 2024-03-12T21:21:09.473916+0000 2024-03-11T21:27:22.511 INFO:teuthology.orchestra.run.smithi092.stdout:blocklist 172.21.15.92:0/3562645553 expires 2024-03-12T21:21:09.473916+0000 2024-03-11T21:27:22.802 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:27:22 smithi092 ceph-mon[29572]: pgmap v43: 132 pgs: 132 active+clean; 582 KiB data, 228 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-11T21:27:22.802 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:27:22 smithi092 ceph-mon[29572]: 2.f scrub starts 2024-03-11T21:27:22.802 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:27:22 smithi092 ceph-mon[29572]: 2.f scrub ok 2024-03-11T21:27:22.803 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:27:22 smithi092 ceph-mon[29572]: 5.c scrub starts 2024-03-11T21:27:22.803 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:27:22 smithi092 ceph-mon[29572]: 5.c scrub ok 2024-03-11T21:27:22.803 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:27:22 smithi092 ceph-mon[29572]: from='client.? v1:172.21.15.92:0/761818585' entity='client.admin' cmd=[{"prefix": "osd dump"}]: dispatch 2024-03-11T21:27:22.803 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:27:22 smithi092 ceph-mon[37563]: pgmap v43: 132 pgs: 132 active+clean; 582 KiB data, 228 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-11T21:27:22.803 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:27:22 smithi092 ceph-mon[37563]: 2.f scrub starts 2024-03-11T21:27:22.803 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:27:22 smithi092 ceph-mon[37563]: 2.f scrub ok 2024-03-11T21:27:22.803 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:27:22 smithi092 ceph-mon[37563]: 5.c scrub starts 2024-03-11T21:27:22.804 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:27:22 smithi092 ceph-mon[37563]: 5.c scrub ok 2024-03-11T21:27:22.804 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:27:22 smithi092 ceph-mon[37563]: from='client.? v1:172.21.15.92:0/761818585' entity='client.admin' cmd=[{"prefix": "osd dump"}]: dispatch 2024-03-11T21:27:23.077 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:27:22 smithi097 ceph-mon[33317]: pgmap v43: 132 pgs: 132 active+clean; 582 KiB data, 228 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-11T21:27:23.077 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:27:22 smithi097 ceph-mon[33317]: 2.f scrub starts 2024-03-11T21:27:23.077 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:27:22 smithi097 ceph-mon[33317]: 2.f scrub ok 2024-03-11T21:27:23.077 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:27:22 smithi097 ceph-mon[33317]: 5.c scrub starts 2024-03-11T21:27:23.077 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:27:22 smithi097 ceph-mon[33317]: 5.c scrub ok 2024-03-11T21:27:23.077 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:27:22 smithi097 ceph-mon[33317]: from='client.? v1:172.21.15.92:0/761818585' entity='client.admin' cmd=[{"prefix": "osd dump"}]: dispatch 2024-03-11T21:27:23.265 INFO:tasks.cephadm.ceph_manager.ceph:['osd.0 up in weight 1 up_from 8 up_thru 72 down_at 0 last_clean_interval [0,0) v1:172.21.15.92:6801/2844100201 v1:172.21.15.92:6802/2844100201 exists,up 79f15843-f0ec-44aa-931e-4e07a74d3768', 'osd.1 up in weight 1 up_from 13 up_thru 72 down_at 0 last_clean_interval [0,0) v1:172.21.15.92:6805/3577871157 v1:172.21.15.92:6806/3577871157 exists,up 3a5a0e78-1293-4ace-a93f-69156098571d', 'osd.2 up in weight 1 up_from 18 up_thru 83 down_at 0 last_clean_interval [0,0) v1:172.21.15.92:6809/760825152 v1:172.21.15.92:6810/760825152 exists,up f207b743-0df4-40ed-a735-950e272fea60', 'osd.3 up in weight 1 up_from 26 up_thru 83 down_at 0 last_clean_interval [0,0) v1:172.21.15.92:6813/670596264 v1:172.21.15.92:6814/670596264 exists,up bbb6f69a-8f0f-47e6-937a-062e1902ae13', 'osd.4 up in weight 1 up_from 32 up_thru 83 down_at 0 last_clean_interval [0,0) v1:172.21.15.97:6800/420733620 v1:172.21.15.97:6801/420733620 exists,up 00266cbc-d2cf-4dee-8a96-3753ebc95ca8', 'osd.5 up in weight 1 up_from 41 up_thru 83 down_at 0 last_clean_interval [0,0) v1:172.21.15.97:6804/312056041 v1:172.21.15.97:6805/312056041 exists,up 593a6f6b-8874-40ae-8848-d51c5723a5e4', 'osd.6 up in weight 1 up_from 50 up_thru 70 down_at 0 last_clean_interval [0,0) v1:172.21.15.97:6808/178994999 v1:172.21.15.97:6809/178994999 exists,up 91035ecc-6f1a-4cec-bda6-090af9d53eff', 'osd.7 up in weight 1 up_from 60 up_thru 72 down_at 0 last_clean_interval [0,0) v1:172.21.15.97:6812/3241560509 v1:172.21.15.97:6813/3241560509 exists,up 3eb28c24-664a-4bd6-869d-6bf5db98f52f'] 2024-03-11T21:27:23.265 DEBUG:teuthology.orchestra.run.smithi092:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid 00db0016-dfed-11ee-95c8-87774f69a715 -- ceph daemon mon.a config show 2024-03-11T21:27:23.507 INFO:teuthology.orchestra.run.smithi092.stderr:Inferring config /var/lib/ceph/00db0016-dfed-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T21:27:23.932 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:27:23 smithi092 ceph-mon[29572]: from='client.24469 v1:172.21.15.97:0/404208310' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:27:23.932 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:27:23 smithi092 ceph-mon[29572]: 6.1c scrub starts 2024-03-11T21:27:23.932 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:27:23 smithi092 ceph-mon[29572]: 6.1c scrub ok 2024-03-11T21:27:23.932 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:27:23 smithi092 ceph-mon[29572]: 4.11 scrub starts 2024-03-11T21:27:23.932 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:27:23 smithi092 ceph-mon[29572]: 4.11 scrub ok 2024-03-11T21:27:23.932 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:27:23 smithi092 ceph-mon[29572]: 4.1 scrub starts 2024-03-11T21:27:23.932 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:27:23 smithi092 ceph-mon[29572]: 4.1 scrub ok 2024-03-11T21:27:23.933 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:27:23 smithi092 ceph-mon[37563]: from='client.24469 v1:172.21.15.97:0/404208310' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:27:23.933 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:27:23 smithi092 ceph-mon[37563]: 6.1c scrub starts 2024-03-11T21:27:23.933 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:27:23 smithi092 ceph-mon[37563]: 6.1c scrub ok 2024-03-11T21:27:23.933 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:27:23 smithi092 ceph-mon[37563]: 4.11 scrub starts 2024-03-11T21:27:23.933 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:27:23 smithi092 ceph-mon[37563]: 4.11 scrub ok 2024-03-11T21:27:23.933 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:27:23 smithi092 ceph-mon[37563]: 4.1 scrub starts 2024-03-11T21:27:23.933 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:27:23 smithi092 ceph-mon[37563]: 4.1 scrub ok 2024-03-11T21:27:24.077 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:27:23 smithi097 ceph-mon[33317]: from='client.24469 v1:172.21.15.97:0/404208310' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:27:24.077 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:27:23 smithi097 ceph-mon[33317]: 6.1c scrub starts 2024-03-11T21:27:24.077 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:27:23 smithi097 ceph-mon[33317]: 6.1c scrub ok 2024-03-11T21:27:24.077 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:27:23 smithi097 ceph-mon[33317]: 4.11 scrub starts 2024-03-11T21:27:24.077 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:27:23 smithi097 ceph-mon[33317]: 4.11 scrub ok 2024-03-11T21:27:24.077 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:27:23 smithi097 ceph-mon[33317]: 4.1 scrub starts 2024-03-11T21:27:24.077 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:27:23 smithi097 ceph-mon[33317]: 4.1 scrub ok 2024-03-11T21:27:24.878 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:27:24 smithi092 ceph-mon[29572]: pgmap v44: 132 pgs: 132 active+clean; 582 KiB data, 227 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-11T21:27:24.879 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:27:24 smithi092 ceph-mon[29572]: 6.0 scrub starts 2024-03-11T21:27:24.879 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:27:24 smithi092 ceph-mon[29572]: 6.0 scrub ok 2024-03-11T21:27:24.879 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:27:24 smithi092 ceph-mon[29572]: 2.c scrub starts 2024-03-11T21:27:24.879 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:27:24 smithi092 ceph-mon[29572]: 2.c scrub ok 2024-03-11T21:27:24.879 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:27:24 smithi092 ceph-mon[37563]: pgmap v44: 132 pgs: 132 active+clean; 582 KiB data, 227 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-11T21:27:24.879 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:27:24 smithi092 ceph-mon[37563]: 6.0 scrub starts 2024-03-11T21:27:24.879 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:27:24 smithi092 ceph-mon[37563]: 6.0 scrub ok 2024-03-11T21:27:24.879 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:27:24 smithi092 ceph-mon[37563]: 2.c scrub starts 2024-03-11T21:27:24.879 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:27:24 smithi092 ceph-mon[37563]: 2.c scrub ok 2024-03-11T21:27:25.077 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:27:24 smithi097 ceph-mon[33317]: pgmap v44: 132 pgs: 132 active+clean; 582 KiB data, 227 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-11T21:27:25.077 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:27:24 smithi097 ceph-mon[33317]: 6.0 scrub starts 2024-03-11T21:27:25.077 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:27:24 smithi097 ceph-mon[33317]: 6.0 scrub ok 2024-03-11T21:27:25.077 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:27:24 smithi097 ceph-mon[33317]: 2.c scrub starts 2024-03-11T21:27:25.077 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:27:24 smithi097 ceph-mon[33317]: 2.c scrub ok 2024-03-11T21:27:25.214 INFO:teuthology.orchestra.run.smithi092.stdout:{ 2024-03-11T21:27:25.214 INFO:teuthology.orchestra.run.smithi092.stdout: "name": "mon.a", 2024-03-11T21:27:25.214 INFO:teuthology.orchestra.run.smithi092.stdout: "cluster": "ceph", 2024-03-11T21:27:25.214 INFO:teuthology.orchestra.run.smithi092.stdout: "admin_socket": "/var/run/ceph/ceph-mon.a.asok", 2024-03-11T21:27:25.214 INFO:teuthology.orchestra.run.smithi092.stdout: "admin_socket_mode": "", 2024-03-11T21:27:25.215 INFO:teuthology.orchestra.run.smithi092.stdout: "auth_allow_insecure_global_id_reclaim": "false", 2024-03-11T21:27:25.215 INFO:teuthology.orchestra.run.smithi092.stdout: "auth_client_required": "cephx, none", 2024-03-11T21:27:25.215 INFO:teuthology.orchestra.run.smithi092.stdout: "auth_cluster_required": "cephx", 2024-03-11T21:27:25.215 INFO:teuthology.orchestra.run.smithi092.stdout: "auth_debug": "true", 2024-03-11T21:27:25.215 INFO:teuthology.orchestra.run.smithi092.stdout: "auth_expose_insecure_global_id_reclaim": "true", 2024-03-11T21:27:25.215 INFO:teuthology.orchestra.run.smithi092.stdout: "auth_mon_ticket_ttl": "660.000000", 2024-03-11T21:27:25.215 INFO:teuthology.orchestra.run.smithi092.stdout: "auth_service_required": "cephx", 2024-03-11T21:27:25.215 INFO:teuthology.orchestra.run.smithi092.stdout: "auth_service_ticket_ttl": "240.000000", 2024-03-11T21:27:25.215 INFO:teuthology.orchestra.run.smithi092.stdout: "auth_supported": "", 2024-03-11T21:27:25.215 INFO:teuthology.orchestra.run.smithi092.stdout: "bdev_aio": "true", 2024-03-11T21:27:25.215 INFO:teuthology.orchestra.run.smithi092.stdout: "bdev_aio_max_queue_depth": "1024", 2024-03-11T21:27:25.215 INFO:teuthology.orchestra.run.smithi092.stdout: "bdev_aio_poll_ms": "250", 2024-03-11T21:27:25.215 INFO:teuthology.orchestra.run.smithi092.stdout: "bdev_aio_reap_max": "16", 2024-03-11T21:27:25.215 INFO:teuthology.orchestra.run.smithi092.stdout: "bdev_async_discard": "false", 2024-03-11T21:27:25.216 INFO:teuthology.orchestra.run.smithi092.stdout: "bdev_block_size": "4096", 2024-03-11T21:27:25.216 INFO:teuthology.orchestra.run.smithi092.stdout: "bdev_debug_aio": "false", 2024-03-11T21:27:25.216 INFO:teuthology.orchestra.run.smithi092.stdout: "bdev_debug_aio_log_age": "5.000000", 2024-03-11T21:27:25.216 INFO:teuthology.orchestra.run.smithi092.stdout: "bdev_debug_aio_suicide_timeout": "60.000000", 2024-03-11T21:27:25.216 INFO:teuthology.orchestra.run.smithi092.stdout: "bdev_debug_inflight_ios": "false", 2024-03-11T21:27:25.216 INFO:teuthology.orchestra.run.smithi092.stdout: "bdev_enable_discard": "false", 2024-03-11T21:27:25.216 INFO:teuthology.orchestra.run.smithi092.stdout: "bdev_flock_retry": "3", 2024-03-11T21:27:25.216 INFO:teuthology.orchestra.run.smithi092.stdout: "bdev_flock_retry_interval": "0.100000", 2024-03-11T21:27:25.216 INFO:teuthology.orchestra.run.smithi092.stdout: "bdev_inject_crash": "0", 2024-03-11T21:27:25.216 INFO:teuthology.orchestra.run.smithi092.stdout: "bdev_inject_crash_flush_delay": "2", 2024-03-11T21:27:25.216 INFO:teuthology.orchestra.run.smithi092.stdout: "bdev_ioring": "false", 2024-03-11T21:27:25.216 INFO:teuthology.orchestra.run.smithi092.stdout: "bdev_ioring_hipri": "false", 2024-03-11T21:27:25.216 INFO:teuthology.orchestra.run.smithi092.stdout: "bdev_ioring_sqthread_poll": "false", 2024-03-11T21:27:25.217 INFO:teuthology.orchestra.run.smithi092.stdout: "bdev_nvme_unbind_from_kernel": "false", 2024-03-11T21:27:25.217 INFO:teuthology.orchestra.run.smithi092.stdout: "bdev_read_buffer_alignment": "4096", 2024-03-11T21:27:25.217 INFO:teuthology.orchestra.run.smithi092.stdout: "bdev_read_preallocated_huge_buffers": "", 2024-03-11T21:27:25.217 INFO:teuthology.orchestra.run.smithi092.stdout: "bdev_type": "", 2024-03-11T21:27:25.217 INFO:teuthology.orchestra.run.smithi092.stdout: "bluefs_alloc_size": "1048576", 2024-03-11T21:27:25.217 INFO:teuthology.orchestra.run.smithi092.stdout: "bluefs_allocator": "hybrid", 2024-03-11T21:27:25.217 INFO:teuthology.orchestra.run.smithi092.stdout: "bluefs_buffered_io": "true", 2024-03-11T21:27:25.217 INFO:teuthology.orchestra.run.smithi092.stdout: "bluefs_check_for_zeros": "false", 2024-03-11T21:27:25.217 INFO:teuthology.orchestra.run.smithi092.stdout: "bluefs_check_volume_selector_often": "false", 2024-03-11T21:27:25.217 INFO:teuthology.orchestra.run.smithi092.stdout: "bluefs_check_volume_selector_on_umount": "false", 2024-03-11T21:27:25.217 INFO:teuthology.orchestra.run.smithi092.stdout: "bluefs_compact_log_sync": "false", 2024-03-11T21:27:25.217 INFO:teuthology.orchestra.run.smithi092.stdout: "bluefs_failed_shared_alloc_cooldown": "600.000000", 2024-03-11T21:27:25.217 INFO:teuthology.orchestra.run.smithi092.stdout: "bluefs_log_compact_min_ratio": "5.000000", 2024-03-11T21:27:25.217 INFO:teuthology.orchestra.run.smithi092.stdout: "bluefs_log_compact_min_size": "16777216", 2024-03-11T21:27:25.218 INFO:teuthology.orchestra.run.smithi092.stdout: "bluefs_log_replay_check_allocations": "true", 2024-03-11T21:27:25.218 INFO:teuthology.orchestra.run.smithi092.stdout: "bluefs_max_log_runway": "4194304", 2024-03-11T21:27:25.218 INFO:teuthology.orchestra.run.smithi092.stdout: "bluefs_max_prefetch": "1048576", 2024-03-11T21:27:25.218 INFO:teuthology.orchestra.run.smithi092.stdout: "bluefs_min_flush_size": "524288", 2024-03-11T21:27:25.218 INFO:teuthology.orchestra.run.smithi092.stdout: "bluefs_min_log_runway": "1048576", 2024-03-11T21:27:25.218 INFO:teuthology.orchestra.run.smithi092.stdout: "bluefs_replay_recovery": "false", 2024-03-11T21:27:25.218 INFO:teuthology.orchestra.run.smithi092.stdout: "bluefs_replay_recovery_disable_compact": "false", 2024-03-11T21:27:25.218 INFO:teuthology.orchestra.run.smithi092.stdout: "bluefs_shared_alloc_size": "65536", 2024-03-11T21:27:25.218 INFO:teuthology.orchestra.run.smithi092.stdout: "bluefs_sync_write": "false", 2024-03-11T21:27:25.218 INFO:teuthology.orchestra.run.smithi092.stdout: "bluestore_2q_cache_kin_ratio": "0.500000", 2024-03-11T21:27:25.218 INFO:teuthology.orchestra.run.smithi092.stdout: "bluestore_2q_cache_kout_ratio": "0.500000", 2024-03-11T21:27:25.218 INFO:teuthology.orchestra.run.smithi092.stdout: "bluestore_alloc_stats_dump_interval": "86400.000000", 2024-03-11T21:27:25.219 INFO:teuthology.orchestra.run.smithi092.stdout: "bluestore_allocation_from_file": "true", 2024-03-11T21:27:25.219 INFO:teuthology.orchestra.run.smithi092.stdout: "bluestore_allocator": "hybrid", 2024-03-11T21:27:25.219 INFO:teuthology.orchestra.run.smithi092.stdout: "bluestore_avl_alloc_bf_free_pct": "4", 2024-03-11T21:27:25.219 INFO:teuthology.orchestra.run.smithi092.stdout: "bluestore_avl_alloc_bf_threshold": "131072", 2024-03-11T21:27:25.219 INFO:teuthology.orchestra.run.smithi092.stdout: "bluestore_avl_alloc_ff_max_search_bytes": "16777216", 2024-03-11T21:27:25.219 INFO:teuthology.orchestra.run.smithi092.stdout: "bluestore_avl_alloc_ff_max_search_count": "100", 2024-03-11T21:27:25.219 INFO:teuthology.orchestra.run.smithi092.stdout: "bluestore_bitmapallocator_blocks_per_zone": "1024", 2024-03-11T21:27:25.219 INFO:teuthology.orchestra.run.smithi092.stdout: "bluestore_bitmapallocator_span_size": "1024", 2024-03-11T21:27:25.219 INFO:teuthology.orchestra.run.smithi092.stdout: "bluestore_blobid_prealloc": "10240", 2024-03-11T21:27:25.219 INFO:teuthology.orchestra.run.smithi092.stdout: "bluestore_block_create": "true", 2024-03-11T21:27:25.219 INFO:teuthology.orchestra.run.smithi092.stdout: "bluestore_block_db_create": "false", 2024-03-11T21:27:25.219 INFO:teuthology.orchestra.run.smithi092.stdout: "bluestore_block_db_path": "", 2024-03-11T21:27:25.219 INFO:teuthology.orchestra.run.smithi092.stdout: "bluestore_block_db_size": "0", 2024-03-11T21:27:25.219 INFO:teuthology.orchestra.run.smithi092.stdout: "bluestore_block_path": "", 2024-03-11T21:27:25.219 INFO:teuthology.orchestra.run.smithi092.stdout: "bluestore_block_preallocate_file": "false", 2024-03-11T21:27:25.220 INFO:teuthology.orchestra.run.smithi092.stdout: "bluestore_block_size": "107374182400", 2024-03-11T21:27:25.220 INFO:teuthology.orchestra.run.smithi092.stdout: "bluestore_block_wal_create": "false", 2024-03-11T21:27:25.220 INFO:teuthology.orchestra.run.smithi092.stdout: "bluestore_block_wal_path": "", 2024-03-11T21:27:25.220 INFO:teuthology.orchestra.run.smithi092.stdout: "bluestore_block_wal_size": "100663296", 2024-03-11T21:27:25.220 INFO:teuthology.orchestra.run.smithi092.stdout: "bluestore_bluefs": "true", 2024-03-11T21:27:25.220 INFO:teuthology.orchestra.run.smithi092.stdout: "bluestore_bluefs_alloc_failure_dump_interval": "0.000000", 2024-03-11T21:27:25.220 INFO:teuthology.orchestra.run.smithi092.stdout: "bluestore_bluefs_env_mirror": "false", 2024-03-11T21:27:25.220 INFO:teuthology.orchestra.run.smithi092.stdout: "bluestore_bluefs_max_free": "10737418240", 2024-03-11T21:27:25.220 INFO:teuthology.orchestra.run.smithi092.stdout: "bluestore_cache_age_bin_interval": "1.000000", 2024-03-11T21:27:25.220 INFO:teuthology.orchestra.run.smithi092.stdout: "bluestore_cache_age_bins_data": "1 2 6 24 120 720 0 0 0 0", 2024-03-11T21:27:25.220 INFO:teuthology.orchestra.run.smithi092.stdout: "bluestore_cache_age_bins_kv": "1 2 6 24 120 720 0 0 0 0", 2024-03-11T21:27:25.220 INFO:teuthology.orchestra.run.smithi092.stdout: "bluestore_cache_age_bins_kv_onode": "0 0 0 0 0 0 0 0 0 720", 2024-03-11T21:27:25.220 INFO:teuthology.orchestra.run.smithi092.stdout: "bluestore_cache_age_bins_meta": "1 2 6 24 120 720 0 0 0 0", 2024-03-11T21:27:25.221 INFO:teuthology.orchestra.run.smithi092.stdout: "bluestore_cache_autotune": "true", 2024-03-11T21:27:25.221 INFO:teuthology.orchestra.run.smithi092.stdout: "bluestore_cache_autotune_interval": "5.000000", 2024-03-11T21:27:25.221 INFO:teuthology.orchestra.run.smithi092.stdout: "bluestore_cache_kv_onode_ratio": "0.040000", 2024-03-11T21:27:25.221 INFO:teuthology.orchestra.run.smithi092.stdout: "bluestore_cache_kv_ratio": "0.450000", 2024-03-11T21:27:25.221 INFO:teuthology.orchestra.run.smithi092.stdout: "bluestore_cache_meta_ratio": "0.450000", 2024-03-11T21:27:25.221 INFO:teuthology.orchestra.run.smithi092.stdout: "bluestore_cache_size": "0", 2024-03-11T21:27:25.221 INFO:teuthology.orchestra.run.smithi092.stdout: "bluestore_cache_size_hdd": "1073741824", 2024-03-11T21:27:25.221 INFO:teuthology.orchestra.run.smithi092.stdout: "bluestore_cache_size_ssd": "3221225472", 2024-03-11T21:27:25.221 INFO:teuthology.orchestra.run.smithi092.stdout: "bluestore_cache_trim_interval": "0.050000", 2024-03-11T21:27:25.221 INFO:teuthology.orchestra.run.smithi092.stdout: "bluestore_cache_trim_max_skip_pinned": "1000", 2024-03-11T21:27:25.221 INFO:teuthology.orchestra.run.smithi092.stdout: "bluestore_cache_type": "2q", 2024-03-11T21:27:25.221 INFO:teuthology.orchestra.run.smithi092.stdout: "bluestore_cleaner_sleep_interval": "5.000000", 2024-03-11T21:27:25.222 INFO:teuthology.orchestra.run.smithi092.stdout: "bluestore_clone_cow": "true", 2024-03-11T21:27:25.222 INFO:teuthology.orchestra.run.smithi092.stdout: "bluestore_compression_algorithm": "snappy", 2024-03-11T21:27:25.222 INFO:teuthology.orchestra.run.smithi092.stdout: "bluestore_compression_max_blob_size": "0", 2024-03-11T21:27:25.222 INFO:teuthology.orchestra.run.smithi092.stdout: "bluestore_compression_max_blob_size_hdd": "65536", 2024-03-11T21:27:25.222 INFO:teuthology.orchestra.run.smithi092.stdout: "bluestore_compression_max_blob_size_ssd": "65536", 2024-03-11T21:27:25.222 INFO:teuthology.orchestra.run.smithi092.stdout: "bluestore_compression_min_blob_size": "0", 2024-03-11T21:27:25.222 INFO:teuthology.orchestra.run.smithi092.stdout: "bluestore_compression_min_blob_size_hdd": "8192", 2024-03-11T21:27:25.222 INFO:teuthology.orchestra.run.smithi092.stdout: "bluestore_compression_min_blob_size_ssd": "65536", 2024-03-11T21:27:25.222 INFO:teuthology.orchestra.run.smithi092.stdout: "bluestore_compression_mode": "none", 2024-03-11T21:27:25.222 INFO:teuthology.orchestra.run.smithi092.stdout: "bluestore_compression_required_ratio": "0.875000", 2024-03-11T21:27:25.222 INFO:teuthology.orchestra.run.smithi092.stdout: "bluestore_csum_type": "crc32c", 2024-03-11T21:27:25.222 INFO:teuthology.orchestra.run.smithi092.stdout: "bluestore_debug_enforce_min_alloc_size": "0", 2024-03-11T21:27:25.222 INFO:teuthology.orchestra.run.smithi092.stdout: "bluestore_debug_enforce_settings": "default", 2024-03-11T21:27:25.222 INFO:teuthology.orchestra.run.smithi092.stdout: "bluestore_debug_freelist": "false", 2024-03-11T21:27:25.223 INFO:teuthology.orchestra.run.smithi092.stdout: "bluestore_debug_fsck_abort": "false", 2024-03-11T21:27:25.223 INFO:teuthology.orchestra.run.smithi092.stdout: "bluestore_debug_inject_allocation_from_file_failure": "0.000000", 2024-03-11T21:27:25.223 INFO:teuthology.orchestra.run.smithi092.stdout: "bluestore_debug_inject_csum_err_probability": "0.000000", 2024-03-11T21:27:25.223 INFO:teuthology.orchestra.run.smithi092.stdout: "bluestore_debug_inject_read_err": "false", 2024-03-11T21:27:25.223 INFO:teuthology.orchestra.run.smithi092.stdout: "bluestore_debug_legacy_omap": "false", 2024-03-11T21:27:25.223 INFO:teuthology.orchestra.run.smithi092.stdout: "bluestore_debug_no_reuse_blocks": "false", 2024-03-11T21:27:25.223 INFO:teuthology.orchestra.run.smithi092.stdout: "bluestore_debug_omit_block_device_write": "false", 2024-03-11T21:27:25.223 INFO:teuthology.orchestra.run.smithi092.stdout: "bluestore_debug_omit_kv_commit": "false", 2024-03-11T21:27:25.223 INFO:teuthology.orchestra.run.smithi092.stdout: "bluestore_debug_permit_any_bdev_label": "false", 2024-03-11T21:27:25.223 INFO:teuthology.orchestra.run.smithi092.stdout: "bluestore_debug_prefill": "0.000000", 2024-03-11T21:27:25.223 INFO:teuthology.orchestra.run.smithi092.stdout: "bluestore_debug_prefragment_max": "1048576", 2024-03-11T21:27:25.223 INFO:teuthology.orchestra.run.smithi092.stdout: "bluestore_debug_random_read_err": "0.000000", 2024-03-11T21:27:25.223 INFO:teuthology.orchestra.run.smithi092.stdout: "bluestore_debug_randomize_serial_transaction": "0", 2024-03-11T21:27:25.224 INFO:teuthology.orchestra.run.smithi092.stdout: "bluestore_debug_small_allocations": "0", 2024-03-11T21:27:25.224 INFO:teuthology.orchestra.run.smithi092.stdout: "bluestore_debug_too_many_blobs_threshold": "24576", 2024-03-11T21:27:25.224 INFO:teuthology.orchestra.run.smithi092.stdout: "bluestore_default_buffered_read": "true", 2024-03-11T21:27:25.224 INFO:teuthology.orchestra.run.smithi092.stdout: "bluestore_default_buffered_write": "false", 2024-03-11T21:27:25.224 INFO:teuthology.orchestra.run.smithi092.stdout: "bluestore_deferred_batch_ops": "0", 2024-03-11T21:27:25.224 INFO:teuthology.orchestra.run.smithi092.stdout: "bluestore_deferred_batch_ops_hdd": "64", 2024-03-11T21:27:25.224 INFO:teuthology.orchestra.run.smithi092.stdout: "bluestore_deferred_batch_ops_ssd": "16", 2024-03-11T21:27:25.224 INFO:teuthology.orchestra.run.smithi092.stdout: "bluestore_elastic_shared_blobs": "true", 2024-03-11T21:27:25.224 INFO:teuthology.orchestra.run.smithi092.stdout: "bluestore_extent_map_inline_shard_prealloc_size": "256", 2024-03-11T21:27:25.224 INFO:teuthology.orchestra.run.smithi092.stdout: "bluestore_extent_map_shard_max_size": "1200", 2024-03-11T21:27:25.224 INFO:teuthology.orchestra.run.smithi092.stdout: "bluestore_extent_map_shard_min_size": "150", 2024-03-11T21:27:25.224 INFO:teuthology.orchestra.run.smithi092.stdout: "bluestore_extent_map_shard_target_size": "500", 2024-03-11T21:27:25.224 INFO:teuthology.orchestra.run.smithi092.stdout: "bluestore_extent_map_shard_target_size_slop": "0.200000", 2024-03-11T21:27:25.224 INFO:teuthology.orchestra.run.smithi092.stdout: "bluestore_fail_eio": "false", 2024-03-11T21:27:25.225 INFO:teuthology.orchestra.run.smithi092.stdout: "bluestore_freelist_blocks_per_key": "128", 2024-03-11T21:27:25.225 INFO:teuthology.orchestra.run.smithi092.stdout: "bluestore_fsck_error_on_no_per_pg_omap": "false", 2024-03-11T21:27:25.225 INFO:teuthology.orchestra.run.smithi092.stdout: "bluestore_fsck_error_on_no_per_pool_omap": "false", 2024-03-11T21:27:25.225 INFO:teuthology.orchestra.run.smithi092.stdout: "bluestore_fsck_error_on_no_per_pool_stats": "false", 2024-03-11T21:27:25.225 INFO:teuthology.orchestra.run.smithi092.stdout: "bluestore_fsck_on_mkfs": "true", 2024-03-11T21:27:25.225 INFO:teuthology.orchestra.run.smithi092.stdout: "bluestore_fsck_on_mkfs_deep": "false", 2024-03-11T21:27:25.225 INFO:teuthology.orchestra.run.smithi092.stdout: "bluestore_fsck_on_mount": "false", 2024-03-11T21:27:25.225 INFO:teuthology.orchestra.run.smithi092.stdout: "bluestore_fsck_on_mount_deep": "false", 2024-03-11T21:27:25.225 INFO:teuthology.orchestra.run.smithi092.stdout: "bluestore_fsck_on_umount": "false", 2024-03-11T21:27:25.225 INFO:teuthology.orchestra.run.smithi092.stdout: "bluestore_fsck_on_umount_deep": "false", 2024-03-11T21:27:25.225 INFO:teuthology.orchestra.run.smithi092.stdout: "bluestore_fsck_quick_fix_on_mount": "false", 2024-03-11T21:27:25.225 INFO:teuthology.orchestra.run.smithi092.stdout: "bluestore_fsck_quick_fix_threads": "2", 2024-03-11T21:27:25.225 INFO:teuthology.orchestra.run.smithi092.stdout: "bluestore_fsck_read_bytes_cap": "67108864", 2024-03-11T21:27:25.225 INFO:teuthology.orchestra.run.smithi092.stdout: "bluestore_fsck_shared_blob_tracker_size": "0.031250", 2024-03-11T21:27:25.226 INFO:teuthology.orchestra.run.smithi092.stdout: "bluestore_gc_enable_blob_threshold": "0", 2024-03-11T21:27:25.226 INFO:teuthology.orchestra.run.smithi092.stdout: "bluestore_gc_enable_total_threshold": "0", 2024-03-11T21:27:25.226 INFO:teuthology.orchestra.run.smithi092.stdout: "bluestore_hybrid_alloc_mem_cap": "67108864", 2024-03-11T21:27:25.226 INFO:teuthology.orchestra.run.smithi092.stdout: "bluestore_ignore_data_csum": "false", 2024-03-11T21:27:25.226 INFO:teuthology.orchestra.run.smithi092.stdout: "bluestore_kv_sync_util_logging_s": "10.000000", 2024-03-11T21:27:25.226 INFO:teuthology.orchestra.run.smithi092.stdout: "bluestore_kvbackend": "rocksdb", 2024-03-11T21:27:25.226 INFO:teuthology.orchestra.run.smithi092.stdout: "bluestore_log_collection_list_age": "60.000000", 2024-03-11T21:27:25.226 INFO:teuthology.orchestra.run.smithi092.stdout: "bluestore_log_omap_iterator_age": "5.000000", 2024-03-11T21:27:25.226 INFO:teuthology.orchestra.run.smithi092.stdout: "bluestore_log_op_age": "5.000000", 2024-03-11T21:27:25.226 INFO:teuthology.orchestra.run.smithi092.stdout: "bluestore_max_alloc_size": "0", 2024-03-11T21:27:25.226 INFO:teuthology.orchestra.run.smithi092.stdout: "bluestore_max_blob_size": "0", 2024-03-11T21:27:25.226 INFO:teuthology.orchestra.run.smithi092.stdout: "bluestore_max_blob_size_hdd": "65536", 2024-03-11T21:27:25.226 INFO:teuthology.orchestra.run.smithi092.stdout: "bluestore_max_blob_size_ssd": "65536", 2024-03-11T21:27:25.226 INFO:teuthology.orchestra.run.smithi092.stdout: "bluestore_max_defer_interval": "3.000000", 2024-03-11T21:27:25.227 INFO:teuthology.orchestra.run.smithi092.stdout: "bluestore_max_deferred_txc": "32", 2024-03-11T21:27:25.227 INFO:teuthology.orchestra.run.smithi092.stdout: "bluestore_min_alloc_size": "0", 2024-03-11T21:27:25.227 INFO:teuthology.orchestra.run.smithi092.stdout: "bluestore_min_alloc_size_hdd": "4096", 2024-03-11T21:27:25.227 INFO:teuthology.orchestra.run.smithi092.stdout: "bluestore_min_alloc_size_ssd": "4096", 2024-03-11T21:27:25.227 INFO:teuthology.orchestra.run.smithi092.stdout: "bluestore_nid_prealloc": "1024", 2024-03-11T21:27:25.227 INFO:teuthology.orchestra.run.smithi092.stdout: "bluestore_prefer_deferred_size": "0", 2024-03-11T21:27:25.227 INFO:teuthology.orchestra.run.smithi092.stdout: "bluestore_prefer_deferred_size_hdd": "65536", 2024-03-11T21:27:25.227 INFO:teuthology.orchestra.run.smithi092.stdout: "bluestore_prefer_deferred_size_ssd": "0", 2024-03-11T21:27:25.227 INFO:teuthology.orchestra.run.smithi092.stdout: "bluestore_qfsck_on_mount": "true", 2024-03-11T21:27:25.227 INFO:teuthology.orchestra.run.smithi092.stdout: "bluestore_retry_disk_reads": "3", 2024-03-11T21:27:25.227 INFO:teuthology.orchestra.run.smithi092.stdout: "bluestore_rocksdb_cf": "true", 2024-03-11T21:27:25.227 INFO:teuthology.orchestra.run.smithi092.stdout: "bluestore_rocksdb_cfs": "m(3) p(3,0-12) O(3,0-13)=block_cache={type=binned_lru} L=min_write_buffer_number_to_merge=32 P=min_write_buffer_number_to_merge=32", 2024-03-11T21:27:25.228 INFO:teuthology.orchestra.run.smithi092.stdout: "bluestore_rocksdb_options": "compression=kLZ4Compression,max_write_buffer_number=64,min_write_buffer_number_to_merge=6,compaction_style=kCompactionStyleLevel,write_buffer_size=16777216,max_background_jobs=4,level0_file_num_compaction_trigger=8,max_bytes_for_level_base=1073741824,max_bytes_for_level_multiplier=8,compaction_readahead_size=2MB,max_total_wal_size=1073741824,writable_file_max_buffer_size=0", 2024-03-11T21:27:25.228 INFO:teuthology.orchestra.run.smithi092.stdout: "bluestore_rocksdb_options_annex": "", 2024-03-11T21:27:25.228 INFO:teuthology.orchestra.run.smithi092.stdout: "bluestore_spdk_coremask": "0x1", 2024-03-11T21:27:25.228 INFO:teuthology.orchestra.run.smithi092.stdout: "bluestore_spdk_io_sleep": "5", 2024-03-11T21:27:25.228 INFO:teuthology.orchestra.run.smithi092.stdout: "bluestore_spdk_max_io_completion": "0", 2024-03-11T21:27:25.229 INFO:teuthology.orchestra.run.smithi092.stdout: "bluestore_spdk_mem": "512", 2024-03-11T21:27:25.229 INFO:teuthology.orchestra.run.smithi092.stdout: "bluestore_sync_submit_transaction": "false", 2024-03-11T21:27:25.229 INFO:teuthology.orchestra.run.smithi092.stdout: "bluestore_throttle_bytes": "67108864", 2024-03-11T21:27:25.229 INFO:teuthology.orchestra.run.smithi092.stdout: "bluestore_throttle_cost_per_io": "0", 2024-03-11T21:27:25.229 INFO:teuthology.orchestra.run.smithi092.stdout: "bluestore_throttle_cost_per_io_hdd": "670000", 2024-03-11T21:27:25.229 INFO:teuthology.orchestra.run.smithi092.stdout: "bluestore_throttle_cost_per_io_ssd": "4000", 2024-03-11T21:27:25.229 INFO:teuthology.orchestra.run.smithi092.stdout: "bluestore_throttle_deferred_bytes": "134217728", 2024-03-11T21:27:25.229 INFO:teuthology.orchestra.run.smithi092.stdout: "bluestore_throttle_trace_rate": "0.000000", 2024-03-11T21:27:25.229 INFO:teuthology.orchestra.run.smithi092.stdout: "bluestore_tracing": "false", 2024-03-11T21:27:25.229 INFO:teuthology.orchestra.run.smithi092.stdout: "bluestore_use_optimal_io_size_for_min_alloc_size": "false", 2024-03-11T21:27:25.229 INFO:teuthology.orchestra.run.smithi092.stdout: "bluestore_volume_selection_policy": "use_some_extra", 2024-03-11T21:27:25.229 INFO:teuthology.orchestra.run.smithi092.stdout: "bluestore_volume_selection_reserved": "0", 2024-03-11T21:27:25.229 INFO:teuthology.orchestra.run.smithi092.stdout: "bluestore_volume_selection_reserved_factor": "2.000000", 2024-03-11T21:27:25.229 INFO:teuthology.orchestra.run.smithi092.stdout: "bluestore_warn_on_bluefs_spillover": "true", 2024-03-11T21:27:25.230 INFO:teuthology.orchestra.run.smithi092.stdout: "bluestore_warn_on_legacy_statfs": "true", 2024-03-11T21:27:25.230 INFO:teuthology.orchestra.run.smithi092.stdout: "bluestore_warn_on_no_per_pg_omap": "false", 2024-03-11T21:27:25.230 INFO:teuthology.orchestra.run.smithi092.stdout: "bluestore_warn_on_no_per_pool_omap": "true", 2024-03-11T21:27:25.230 INFO:teuthology.orchestra.run.smithi092.stdout: "bluestore_warn_on_spurious_read_errors": "true", 2024-03-11T21:27:25.230 INFO:teuthology.orchestra.run.smithi092.stdout: "bluestore_zero_block_detection": "false", 2024-03-11T21:27:25.230 INFO:teuthology.orchestra.run.smithi092.stdout: "cephadm_path": "/usr/sbin/cephadm", 2024-03-11T21:27:25.230 INFO:teuthology.orchestra.run.smithi092.stdout: "cephfs_mirror_action_update_interval": "2", 2024-03-11T21:27:25.230 INFO:teuthology.orchestra.run.smithi092.stdout: "cephfs_mirror_directory_scan_interval": "10", 2024-03-11T21:27:25.230 INFO:teuthology.orchestra.run.smithi092.stdout: "cephfs_mirror_max_concurrent_directory_syncs": "3", 2024-03-11T21:27:25.230 INFO:teuthology.orchestra.run.smithi092.stdout: "cephfs_mirror_max_consecutive_failures_per_directory": "10", 2024-03-11T21:27:25.230 INFO:teuthology.orchestra.run.smithi092.stdout: "cephfs_mirror_max_snapshot_sync_per_cycle": "3", 2024-03-11T21:27:25.230 INFO:teuthology.orchestra.run.smithi092.stdout: "cephfs_mirror_mount_timeout": "10", 2024-03-11T21:27:25.230 INFO:teuthology.orchestra.run.smithi092.stdout: "cephfs_mirror_perf_stats_prio": "5", 2024-03-11T21:27:25.231 INFO:teuthology.orchestra.run.smithi092.stdout: "cephfs_mirror_restart_mirror_on_blocklist_interval": "30", 2024-03-11T21:27:25.231 INFO:teuthology.orchestra.run.smithi092.stdout: "cephfs_mirror_restart_mirror_on_failure_interval": "20", 2024-03-11T21:27:25.231 INFO:teuthology.orchestra.run.smithi092.stdout: "cephfs_mirror_retry_failed_directories_interval": "60", 2024-03-11T21:27:25.231 INFO:teuthology.orchestra.run.smithi092.stdout: "cephsqlite_blocklist_dead_locker": "true", 2024-03-11T21:27:25.231 INFO:teuthology.orchestra.run.smithi092.stdout: "cephsqlite_lock_renewal_interval": "2000", 2024-03-11T21:27:25.231 INFO:teuthology.orchestra.run.smithi092.stdout: "cephsqlite_lock_renewal_timeout": "30000", 2024-03-11T21:27:25.231 INFO:teuthology.orchestra.run.smithi092.stdout: "cephx_cluster_require_signatures": "false", 2024-03-11T21:27:25.231 INFO:teuthology.orchestra.run.smithi092.stdout: "cephx_cluster_require_version": "2", 2024-03-11T21:27:25.231 INFO:teuthology.orchestra.run.smithi092.stdout: "cephx_require_signatures": "false", 2024-03-11T21:27:25.231 INFO:teuthology.orchestra.run.smithi092.stdout: "cephx_require_version": "2", 2024-03-11T21:27:25.231 INFO:teuthology.orchestra.run.smithi092.stdout: "cephx_service_require_signatures": "false", 2024-03-11T21:27:25.231 INFO:teuthology.orchestra.run.smithi092.stdout: "cephx_service_require_version": "2", 2024-03-11T21:27:25.231 INFO:teuthology.orchestra.run.smithi092.stdout: "cephx_sign_messages": "true", 2024-03-11T21:27:25.231 INFO:teuthology.orchestra.run.smithi092.stdout: "chdir": "", 2024-03-11T21:27:25.232 INFO:teuthology.orchestra.run.smithi092.stdout: "client_acl_type": "", 2024-03-11T21:27:25.232 INFO:teuthology.orchestra.run.smithi092.stdout: "client_asio_thread_count": "2", 2024-03-11T21:27:25.232 INFO:teuthology.orchestra.run.smithi092.stdout: "client_cache_mid": "0.750000", 2024-03-11T21:27:25.232 INFO:teuthology.orchestra.run.smithi092.stdout: "client_cache_size": "16384", 2024-03-11T21:27:25.232 INFO:teuthology.orchestra.run.smithi092.stdout: "client_caps_release_delay": "5", 2024-03-11T21:27:25.232 INFO:teuthology.orchestra.run.smithi092.stdout: "client_check_pool_perm": "true", 2024-03-11T21:27:25.232 INFO:teuthology.orchestra.run.smithi092.stdout: "client_collect_and_send_global_metrics": "false", 2024-03-11T21:27:25.232 INFO:teuthology.orchestra.run.smithi092.stdout: "client_debug_force_sync_read": "false", 2024-03-11T21:27:25.232 INFO:teuthology.orchestra.run.smithi092.stdout: "client_debug_getattr_caps": "false", 2024-03-11T21:27:25.232 INFO:teuthology.orchestra.run.smithi092.stdout: "client_debug_inject_tick_delay": "0", 2024-03-11T21:27:25.232 INFO:teuthology.orchestra.run.smithi092.stdout: "client_die_on_failed_dentry_invalidate": "true", 2024-03-11T21:27:25.232 INFO:teuthology.orchestra.run.smithi092.stdout: "client_die_on_failed_remount": "false", 2024-03-11T21:27:25.232 INFO:teuthology.orchestra.run.smithi092.stdout: "client_dirsize_rbytes": "true", 2024-03-11T21:27:25.232 INFO:teuthology.orchestra.run.smithi092.stdout: "client_force_lazyio": "false", 2024-03-11T21:27:25.232 INFO:teuthology.orchestra.run.smithi092.stdout: "client_fs": "", 2024-03-11T21:27:25.233 INFO:teuthology.orchestra.run.smithi092.stdout: "client_inject_fixed_oldest_tid": "false", 2024-03-11T21:27:25.233 INFO:teuthology.orchestra.run.smithi092.stdout: "client_inject_release_failure": "false", 2024-03-11T21:27:25.233 INFO:teuthology.orchestra.run.smithi092.stdout: "client_max_inline_size": "4096", 2024-03-11T21:27:25.233 INFO:teuthology.orchestra.run.smithi092.stdout: "client_max_retries_on_remount_failure": "5", 2024-03-11T21:27:25.233 INFO:teuthology.orchestra.run.smithi092.stdout: "client_mds_namespace": "", 2024-03-11T21:27:25.233 INFO:teuthology.orchestra.run.smithi092.stdout: "client_metadata": "", 2024-03-11T21:27:25.233 INFO:teuthology.orchestra.run.smithi092.stdout: "client_mount_gid": "-1", 2024-03-11T21:27:25.233 INFO:teuthology.orchestra.run.smithi092.stdout: "client_mount_timeout": "300", 2024-03-11T21:27:25.233 INFO:teuthology.orchestra.run.smithi092.stdout: "client_mount_uid": "-1", 2024-03-11T21:27:25.233 INFO:teuthology.orchestra.run.smithi092.stdout: "client_mountpoint": "/", 2024-03-11T21:27:25.233 INFO:teuthology.orchestra.run.smithi092.stdout: "client_notify_timeout": "10", 2024-03-11T21:27:25.233 INFO:teuthology.orchestra.run.smithi092.stdout: "client_oc": "true", 2024-03-11T21:27:25.233 INFO:teuthology.orchestra.run.smithi092.stdout: "client_oc_max_dirty": "104857600", 2024-03-11T21:27:25.234 INFO:teuthology.orchestra.run.smithi092.stdout: "client_oc_max_dirty_age": "5.000000", 2024-03-11T21:27:25.234 INFO:teuthology.orchestra.run.smithi092.stdout: "client_oc_max_objects": "1000", 2024-03-11T21:27:25.234 INFO:teuthology.orchestra.run.smithi092.stdout: "client_oc_size": "209715200", 2024-03-11T21:27:25.234 INFO:teuthology.orchestra.run.smithi092.stdout: "client_oc_target_dirty": "8388608", 2024-03-11T21:27:25.234 INFO:teuthology.orchestra.run.smithi092.stdout: "client_permissions": "true", 2024-03-11T21:27:25.235 INFO:teuthology.orchestra.run.smithi092.stdout: "client_quota": "true", 2024-03-11T21:27:25.235 INFO:teuthology.orchestra.run.smithi092.stdout: "client_quota_df": "true", 2024-03-11T21:27:25.235 INFO:teuthology.orchestra.run.smithi092.stdout: "client_readahead_max_bytes": "0", 2024-03-11T21:27:25.235 INFO:teuthology.orchestra.run.smithi092.stdout: "client_readahead_max_periods": "4", 2024-03-11T21:27:25.235 INFO:teuthology.orchestra.run.smithi092.stdout: "client_readahead_min": "131072", 2024-03-11T21:27:25.235 INFO:teuthology.orchestra.run.smithi092.stdout: "client_reconnect_stale": "false", 2024-03-11T21:27:25.235 INFO:teuthology.orchestra.run.smithi092.stdout: "client_shutdown_timeout": "30", 2024-03-11T21:27:25.235 INFO:teuthology.orchestra.run.smithi092.stdout: "client_snapdir": ".snap", 2024-03-11T21:27:25.236 INFO:teuthology.orchestra.run.smithi092.stdout: "client_tick_interval": "1", 2024-03-11T21:27:25.236 INFO:teuthology.orchestra.run.smithi092.stdout: "client_trace": "", 2024-03-11T21:27:25.236 INFO:teuthology.orchestra.run.smithi092.stdout: "client_try_dentry_invalidate": "false", 2024-03-11T21:27:25.236 INFO:teuthology.orchestra.run.smithi092.stdout: "client_use_faked_inos": "false", 2024-03-11T21:27:25.236 INFO:teuthology.orchestra.run.smithi092.stdout: "client_use_random_mds": "false", 2024-03-11T21:27:25.236 INFO:teuthology.orchestra.run.smithi092.stdout: "clog_to_graylog": "false", 2024-03-11T21:27:25.236 INFO:teuthology.orchestra.run.smithi092.stdout: "clog_to_graylog_host": "127.0.0.1", 2024-03-11T21:27:25.236 INFO:teuthology.orchestra.run.smithi092.stdout: "clog_to_graylog_port": "12201", 2024-03-11T21:27:25.236 INFO:teuthology.orchestra.run.smithi092.stdout: "clog_to_monitors": "default=true", 2024-03-11T21:27:25.236 INFO:teuthology.orchestra.run.smithi092.stdout: "clog_to_syslog": "false", 2024-03-11T21:27:25.236 INFO:teuthology.orchestra.run.smithi092.stdout: "clog_to_syslog_facility": "default=daemon audit=local0", 2024-03-11T21:27:25.236 INFO:teuthology.orchestra.run.smithi092.stdout: "clog_to_syslog_level": "info", 2024-03-11T21:27:25.236 INFO:teuthology.orchestra.run.smithi092.stdout: "cluster_addr": "-", 2024-03-11T21:27:25.236 INFO:teuthology.orchestra.run.smithi092.stdout: "cluster_network": "", 2024-03-11T21:27:25.237 INFO:teuthology.orchestra.run.smithi092.stdout: "cluster_network_interface": "", 2024-03-11T21:27:25.237 INFO:teuthology.orchestra.run.smithi092.stdout: "compressor_zlib_isal": "false", 2024-03-11T21:27:25.237 INFO:teuthology.orchestra.run.smithi092.stdout: "compressor_zlib_level": "5", 2024-03-11T21:27:25.237 INFO:teuthology.orchestra.run.smithi092.stdout: "compressor_zlib_winsize": "-15", 2024-03-11T21:27:25.237 INFO:teuthology.orchestra.run.smithi092.stdout: "compressor_zstd_level": "1", 2024-03-11T21:27:25.237 INFO:teuthology.orchestra.run.smithi092.stdout: "container_image": "quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:8fe9635f3beb0247496d7e321f3068bd10c333c433ef4824d7e1c1a7d3d78379", 2024-03-11T21:27:25.237 INFO:teuthology.orchestra.run.smithi092.stdout: "crash_dir": "/var/lib/ceph/crash", 2024-03-11T21:27:25.237 INFO:teuthology.orchestra.run.smithi092.stdout: "crimson_alien_op_num_threads": "6", 2024-03-11T21:27:25.237 INFO:teuthology.orchestra.run.smithi092.stdout: "crimson_alien_thread_cpu_cores": "", 2024-03-11T21:27:25.237 INFO:teuthology.orchestra.run.smithi092.stdout: "crimson_osd_obc_lru_size": "512", 2024-03-11T21:27:25.237 INFO:teuthology.orchestra.run.smithi092.stdout: "crimson_osd_scheduler_concurrency": "0", 2024-03-11T21:27:25.237 INFO:teuthology.orchestra.run.smithi092.stdout: "crimson_seastar_cpu_cores": "", 2024-03-11T21:27:25.237 INFO:teuthology.orchestra.run.smithi092.stdout: "crush_location": "", 2024-03-11T21:27:25.237 INFO:teuthology.orchestra.run.smithi092.stdout: "crush_location_hook": "", 2024-03-11T21:27:25.238 INFO:teuthology.orchestra.run.smithi092.stdout: "crush_location_hook_timeout": "10", 2024-03-11T21:27:25.238 INFO:teuthology.orchestra.run.smithi092.stdout: "daemonize": "false", 2024-03-11T21:27:25.238 INFO:teuthology.orchestra.run.smithi092.stdout: "daos_pool": "tank", 2024-03-11T21:27:25.238 INFO:teuthology.orchestra.run.smithi092.stdout: "dbstore_config_uri": "file:/var/lib/ceph/radosgw/dbstore-config.db", 2024-03-11T21:27:25.238 INFO:teuthology.orchestra.run.smithi092.stdout: "dbstore_db_dir": "/var/lib/ceph/radosgw", 2024-03-11T21:27:25.238 INFO:teuthology.orchestra.run.smithi092.stdout: "dbstore_db_name_prefix": "dbstore", 2024-03-11T21:27:25.238 INFO:teuthology.orchestra.run.smithi092.stdout: "debug_alienstore": "0/5", 2024-03-11T21:27:25.238 INFO:teuthology.orchestra.run.smithi092.stdout: "debug_allow_any_pool_priority": "false", 2024-03-11T21:27:25.238 INFO:teuthology.orchestra.run.smithi092.stdout: "debug_asok": "1/5", 2024-03-11T21:27:25.238 INFO:teuthology.orchestra.run.smithi092.stdout: "debug_asok_assert_abort": "false", 2024-03-11T21:27:25.238 INFO:teuthology.orchestra.run.smithi092.stdout: "debug_asserts_on_shutdown": "true", 2024-03-11T21:27:25.238 INFO:teuthology.orchestra.run.smithi092.stdout: "debug_auth": "1/5", 2024-03-11T21:27:25.238 INFO:teuthology.orchestra.run.smithi092.stdout: "debug_bdev": "1/3", 2024-03-11T21:27:25.239 INFO:teuthology.orchestra.run.smithi092.stdout: "debug_bluefs": "1/5", 2024-03-11T21:27:25.239 INFO:teuthology.orchestra.run.smithi092.stdout: "debug_bluestore": "1/5", 2024-03-11T21:27:25.239 INFO:teuthology.orchestra.run.smithi092.stdout: "debug_buffer": "0/1", 2024-03-11T21:27:25.239 INFO:teuthology.orchestra.run.smithi092.stdout: "debug_ceph_exporter": "1/5", 2024-03-11T21:27:25.239 INFO:teuthology.orchestra.run.smithi092.stdout: "debug_cephfs_mirror": "0/5", 2024-03-11T21:27:25.239 INFO:teuthology.orchestra.run.smithi092.stdout: "debug_cephsqlite": "0/5", 2024-03-11T21:27:25.239 INFO:teuthology.orchestra.run.smithi092.stdout: "debug_client": "0/5", 2024-03-11T21:27:25.239 INFO:teuthology.orchestra.run.smithi092.stdout: "debug_compressor": "1/5", 2024-03-11T21:27:25.239 INFO:teuthology.orchestra.run.smithi092.stdout: "debug_context": "0/1", 2024-03-11T21:27:25.239 INFO:teuthology.orchestra.run.smithi092.stdout: "debug_crimson_interrupt": "0/5", 2024-03-11T21:27:25.239 INFO:teuthology.orchestra.run.smithi092.stdout: "debug_crush": "1/1", 2024-03-11T21:27:25.239 INFO:teuthology.orchestra.run.smithi092.stdout: "debug_crypto": "1/5", 2024-03-11T21:27:25.239 INFO:teuthology.orchestra.run.smithi092.stdout: "debug_cyanstore": "0/5", 2024-03-11T21:27:25.239 INFO:teuthology.orchestra.run.smithi092.stdout: "debug_deliberately_leak_memory": "false", 2024-03-11T21:27:25.239 INFO:teuthology.orchestra.run.smithi092.stdout: "debug_disable_randomized_ping": "false", 2024-03-11T21:27:25.240 INFO:teuthology.orchestra.run.smithi092.stdout: "debug_dpdk": "1/5", 2024-03-11T21:27:25.240 INFO:teuthology.orchestra.run.smithi092.stdout: "debug_eventtrace": "1/5", 2024-03-11T21:27:25.240 INFO:teuthology.orchestra.run.smithi092.stdout: "debug_filer": "0/1", 2024-03-11T21:27:25.240 INFO:teuthology.orchestra.run.smithi092.stdout: "debug_filestore": "1/3", 2024-03-11T21:27:25.240 INFO:teuthology.orchestra.run.smithi092.stdout: "debug_finisher": "1/1", 2024-03-11T21:27:25.240 INFO:teuthology.orchestra.run.smithi092.stdout: "debug_fuse": "1/5", 2024-03-11T21:27:25.240 INFO:teuthology.orchestra.run.smithi092.stdout: "debug_heartbeat_testing_span": "0", 2024-03-11T21:27:25.240 INFO:teuthology.orchestra.run.smithi092.stdout: "debug_heartbeatmap": "1/5", 2024-03-11T21:27:25.240 INFO:teuthology.orchestra.run.smithi092.stdout: "debug_immutable_obj_cache": "0/5", 2024-03-11T21:27:25.240 INFO:teuthology.orchestra.run.smithi092.stdout: "debug_javaclient": "1/5", 2024-03-11T21:27:25.240 INFO:teuthology.orchestra.run.smithi092.stdout: "debug_journal": "1/3", 2024-03-11T21:27:25.240 INFO:teuthology.orchestra.run.smithi092.stdout: "debug_journaler": "0/5", 2024-03-11T21:27:25.240 INFO:teuthology.orchestra.run.smithi092.stdout: "debug_kstore": "1/5", 2024-03-11T21:27:25.240 INFO:teuthology.orchestra.run.smithi092.stdout: "debug_lockdep": "0/1", 2024-03-11T21:27:25.241 INFO:teuthology.orchestra.run.smithi092.stdout: "debug_mclock": "1/5", 2024-03-11T21:27:25.241 INFO:teuthology.orchestra.run.smithi092.stdout: "debug_mds": "1/5", 2024-03-11T21:27:25.241 INFO:teuthology.orchestra.run.smithi092.stdout: "debug_mds_balancer": "1/5", 2024-03-11T21:27:25.241 INFO:teuthology.orchestra.run.smithi092.stdout: "debug_mds_locker": "1/5", 2024-03-11T21:27:25.241 INFO:teuthology.orchestra.run.smithi092.stdout: "debug_mds_log": "1/5", 2024-03-11T21:27:25.241 INFO:teuthology.orchestra.run.smithi092.stdout: "debug_mds_log_expire": "1/5", 2024-03-11T21:27:25.241 INFO:teuthology.orchestra.run.smithi092.stdout: "debug_mds_migrator": "1/5", 2024-03-11T21:27:25.241 INFO:teuthology.orchestra.run.smithi092.stdout: "debug_memstore": "1/5", 2024-03-11T21:27:25.241 INFO:teuthology.orchestra.run.smithi092.stdout: "debug_mgr": "2/5", 2024-03-11T21:27:25.241 INFO:teuthology.orchestra.run.smithi092.stdout: "debug_mgrc": "1/5", 2024-03-11T21:27:25.241 INFO:teuthology.orchestra.run.smithi092.stdout: "debug_mon": "20/20", 2024-03-11T21:27:25.241 INFO:teuthology.orchestra.run.smithi092.stdout: "debug_monc": "0/10", 2024-03-11T21:27:25.241 INFO:teuthology.orchestra.run.smithi092.stdout: "debug_ms": "1/1", 2024-03-11T21:27:25.241 INFO:teuthology.orchestra.run.smithi092.stdout: "debug_none": "0/5", 2024-03-11T21:27:25.242 INFO:teuthology.orchestra.run.smithi092.stdout: "debug_objclass": "0/5", 2024-03-11T21:27:25.242 INFO:teuthology.orchestra.run.smithi092.stdout: "debug_objectcacher": "0/5", 2024-03-11T21:27:25.242 INFO:teuthology.orchestra.run.smithi092.stdout: "debug_objecter": "0/1", 2024-03-11T21:27:25.242 INFO:teuthology.orchestra.run.smithi092.stdout: "debug_optracker": "0/5", 2024-03-11T21:27:25.242 INFO:teuthology.orchestra.run.smithi092.stdout: "debug_osd": "1/5", 2024-03-11T21:27:25.242 INFO:teuthology.orchestra.run.smithi092.stdout: "debug_paxos": "20/20", 2024-03-11T21:27:25.242 INFO:teuthology.orchestra.run.smithi092.stdout: "debug_perfcounter": "1/5", 2024-03-11T21:27:25.242 INFO:teuthology.orchestra.run.smithi092.stdout: "debug_prioritycache": "1/5", 2024-03-11T21:27:25.242 INFO:teuthology.orchestra.run.smithi092.stdout: "debug_rados": "0/5", 2024-03-11T21:27:25.242 INFO:teuthology.orchestra.run.smithi092.stdout: "debug_rbd": "0/5", 2024-03-11T21:27:25.242 INFO:teuthology.orchestra.run.smithi092.stdout: "debug_rbd_mirror": "0/5", 2024-03-11T21:27:25.242 INFO:teuthology.orchestra.run.smithi092.stdout: "debug_rbd_pwl": "0/5", 2024-03-11T21:27:25.242 INFO:teuthology.orchestra.run.smithi092.stdout: "debug_rbd_replay": "0/5", 2024-03-11T21:27:25.242 INFO:teuthology.orchestra.run.smithi092.stdout: "debug_refs": "0/0", 2024-03-11T21:27:25.243 INFO:teuthology.orchestra.run.smithi092.stdout: "debug_reserver": "1/1", 2024-03-11T21:27:25.243 INFO:teuthology.orchestra.run.smithi092.stdout: "debug_rgw": "1/5", 2024-03-11T21:27:25.243 INFO:teuthology.orchestra.run.smithi092.stdout: "debug_rgw_access": "1/5", 2024-03-11T21:27:25.243 INFO:teuthology.orchestra.run.smithi092.stdout: "debug_rgw_datacache": "1/5", 2024-03-11T21:27:25.243 INFO:teuthology.orchestra.run.smithi092.stdout: "debug_rgw_dbstore": "1/5", 2024-03-11T21:27:25.243 INFO:teuthology.orchestra.run.smithi092.stdout: "debug_rgw_flight": "1/5", 2024-03-11T21:27:25.243 INFO:teuthology.orchestra.run.smithi092.stdout: "debug_rgw_lifecycle": "1/5", 2024-03-11T21:27:25.243 INFO:teuthology.orchestra.run.smithi092.stdout: "debug_rgw_sync": "1/5", 2024-03-11T21:27:25.243 INFO:teuthology.orchestra.run.smithi092.stdout: "debug_rocksdb": "4/5", 2024-03-11T21:27:25.243 INFO:teuthology.orchestra.run.smithi092.stdout: "debug_seastore": "0/5", 2024-03-11T21:27:25.243 INFO:teuthology.orchestra.run.smithi092.stdout: "debug_seastore_backref": "0/5", 2024-03-11T21:27:25.243 INFO:teuthology.orchestra.run.smithi092.stdout: "debug_seastore_cache": "0/5", 2024-03-11T21:27:25.243 INFO:teuthology.orchestra.run.smithi092.stdout: "debug_seastore_cleaner": "0/5", 2024-03-11T21:27:25.243 INFO:teuthology.orchestra.run.smithi092.stdout: "debug_seastore_device": "0/5", 2024-03-11T21:27:25.244 INFO:teuthology.orchestra.run.smithi092.stdout: "debug_seastore_epm": "0/5", 2024-03-11T21:27:25.244 INFO:teuthology.orchestra.run.smithi092.stdout: "debug_seastore_fixedkv_tree": "0/5", 2024-03-11T21:27:25.244 INFO:teuthology.orchestra.run.smithi092.stdout: "debug_seastore_journal": "0/5", 2024-03-11T21:27:25.244 INFO:teuthology.orchestra.run.smithi092.stdout: "debug_seastore_lba": "0/5", 2024-03-11T21:27:25.244 INFO:teuthology.orchestra.run.smithi092.stdout: "debug_seastore_odata": "0/5", 2024-03-11T21:27:25.244 INFO:teuthology.orchestra.run.smithi092.stdout: "debug_seastore_omap": "0/5", 2024-03-11T21:27:25.244 INFO:teuthology.orchestra.run.smithi092.stdout: "debug_seastore_onode": "0/5", 2024-03-11T21:27:25.244 INFO:teuthology.orchestra.run.smithi092.stdout: "debug_seastore_t": "0/5", 2024-03-11T21:27:25.244 INFO:teuthology.orchestra.run.smithi092.stdout: "debug_seastore_tm": "0/5", 2024-03-11T21:27:25.244 INFO:teuthology.orchestra.run.smithi092.stdout: "debug_striper": "0/1", 2024-03-11T21:27:25.244 INFO:teuthology.orchestra.run.smithi092.stdout: "debug_test": "0/5", 2024-03-11T21:27:25.245 INFO:teuthology.orchestra.run.smithi092.stdout: "debug_throttle": "1/1", 2024-03-11T21:27:25.245 INFO:teuthology.orchestra.run.smithi092.stdout: "debug_timer": "0/1", 2024-03-11T21:27:25.245 INFO:teuthology.orchestra.run.smithi092.stdout: "debug_tp": "0/5", 2024-03-11T21:27:25.245 INFO:teuthology.orchestra.run.smithi092.stdout: "debug_trace": "1/5", 2024-03-11T21:27:25.245 INFO:teuthology.orchestra.run.smithi092.stdout: "defer_client_eviction_on_laggy_osds": "true", 2024-03-11T21:27:25.245 INFO:teuthology.orchestra.run.smithi092.stdout: "device_failure_prediction_mode": "none", 2024-03-11T21:27:25.245 INFO:teuthology.orchestra.run.smithi092.stdout: "enable_experimental_unrecoverable_data_corrupting_features": "", 2024-03-11T21:27:25.245 INFO:teuthology.orchestra.run.smithi092.stdout: "erasure_code_dir": "/usr/lib64/ceph/erasure-code", 2024-03-11T21:27:25.246 INFO:teuthology.orchestra.run.smithi092.stdout: "err_to_graylog": "false", 2024-03-11T21:27:25.246 INFO:teuthology.orchestra.run.smithi092.stdout: "err_to_journald": "false", 2024-03-11T21:27:25.246 INFO:teuthology.orchestra.run.smithi092.stdout: "err_to_stderr": "true", 2024-03-11T21:27:25.246 INFO:teuthology.orchestra.run.smithi092.stdout: "err_to_syslog": "false", 2024-03-11T21:27:25.246 INFO:teuthology.orchestra.run.smithi092.stdout: "event_tracing": "false", 2024-03-11T21:27:25.246 INFO:teuthology.orchestra.run.smithi092.stdout: "exporter_addr": "0.0.0.0", 2024-03-11T21:27:25.246 INFO:teuthology.orchestra.run.smithi092.stdout: "exporter_http_port": "9926", 2024-03-11T21:27:25.246 INFO:teuthology.orchestra.run.smithi092.stdout: "exporter_prio_limit": "5", 2024-03-11T21:27:25.246 INFO:teuthology.orchestra.run.smithi092.stdout: "exporter_sock_dir": "/var/run/ceph/", 2024-03-11T21:27:25.246 INFO:teuthology.orchestra.run.smithi092.stdout: "exporter_sort_metrics": "true", 2024-03-11T21:27:25.246 INFO:teuthology.orchestra.run.smithi092.stdout: "exporter_stats_period": "5", 2024-03-11T21:27:25.246 INFO:teuthology.orchestra.run.smithi092.stdout: "fake_statfs_for_testing": "0", 2024-03-11T21:27:25.246 INFO:teuthology.orchestra.run.smithi092.stdout: "fatal_signal_handlers": "true", 2024-03-11T21:27:25.247 INFO:teuthology.orchestra.run.smithi092.stdout: "filer_max_purge_ops": "10", 2024-03-11T21:27:25.247 INFO:teuthology.orchestra.run.smithi092.stdout: "filer_max_truncate_ops": "128", 2024-03-11T21:27:25.247 INFO:teuthology.orchestra.run.smithi092.stdout: "filestore_apply_finisher_threads": "1", 2024-03-11T21:27:25.247 INFO:teuthology.orchestra.run.smithi092.stdout: "filestore_blackhole": "false", 2024-03-11T21:27:25.247 INFO:teuthology.orchestra.run.smithi092.stdout: "filestore_btrfs_clone_range": "true", 2024-03-11T21:27:25.247 INFO:teuthology.orchestra.run.smithi092.stdout: "filestore_btrfs_snap": "true", 2024-03-11T21:27:25.247 INFO:teuthology.orchestra.run.smithi092.stdout: "filestore_caller_concurrency": "10", 2024-03-11T21:27:25.247 INFO:teuthology.orchestra.run.smithi092.stdout: "filestore_collect_device_partition_information": "true", 2024-03-11T21:27:25.247 INFO:teuthology.orchestra.run.smithi092.stdout: "filestore_commit_timeout": "600.000000", 2024-03-11T21:27:25.247 INFO:teuthology.orchestra.run.smithi092.stdout: "filestore_debug_inject_read_err": "false", 2024-03-11T21:27:25.247 INFO:teuthology.orchestra.run.smithi092.stdout: "filestore_debug_omap_check": "false", 2024-03-11T21:27:25.247 INFO:teuthology.orchestra.run.smithi092.stdout: "filestore_debug_random_read_err": "0.000000", 2024-03-11T21:27:25.247 INFO:teuthology.orchestra.run.smithi092.stdout: "filestore_debug_verify_split": "false", 2024-03-11T21:27:25.248 INFO:teuthology.orchestra.run.smithi092.stdout: "filestore_dump_file": "", 2024-03-11T21:27:25.248 INFO:teuthology.orchestra.run.smithi092.stdout: "filestore_expected_throughput_bytes": "209715200.000000", 2024-03-11T21:27:25.248 INFO:teuthology.orchestra.run.smithi092.stdout: "filestore_expected_throughput_ops": "200.000000", 2024-03-11T21:27:25.248 INFO:teuthology.orchestra.run.smithi092.stdout: "filestore_fadvise": "true", 2024-03-11T21:27:25.248 INFO:teuthology.orchestra.run.smithi092.stdout: "filestore_fail_eio": "true", 2024-03-11T21:27:25.248 INFO:teuthology.orchestra.run.smithi092.stdout: "filestore_fd_cache_shards": "16", 2024-03-11T21:27:25.248 INFO:teuthology.orchestra.run.smithi092.stdout: "filestore_fd_cache_size": "128", 2024-03-11T21:27:25.248 INFO:teuthology.orchestra.run.smithi092.stdout: "filestore_fiemap": "false", 2024-03-11T21:27:25.248 INFO:teuthology.orchestra.run.smithi092.stdout: "filestore_fiemap_threshold": "4096", 2024-03-11T21:27:25.248 INFO:teuthology.orchestra.run.smithi092.stdout: "filestore_fsync_flushes_journal_data": "false", 2024-03-11T21:27:25.248 INFO:teuthology.orchestra.run.smithi092.stdout: "filestore_index_retry_probability": "0.000000", 2024-03-11T21:27:25.248 INFO:teuthology.orchestra.run.smithi092.stdout: "filestore_inject_stall": "0", 2024-03-11T21:27:25.248 INFO:teuthology.orchestra.run.smithi092.stdout: "filestore_journal_parallel": "false", 2024-03-11T21:27:25.249 INFO:teuthology.orchestra.run.smithi092.stdout: "filestore_journal_trailing": "false", 2024-03-11T21:27:25.249 INFO:teuthology.orchestra.run.smithi092.stdout: "filestore_journal_writeahead": "false", 2024-03-11T21:27:25.249 INFO:teuthology.orchestra.run.smithi092.stdout: "filestore_kill_at": "0", 2024-03-11T21:27:25.249 INFO:teuthology.orchestra.run.smithi092.stdout: "filestore_max_alloc_hint_size": "1048576", 2024-03-11T21:27:25.249 INFO:teuthology.orchestra.run.smithi092.stdout: "filestore_max_inline_xattr_size": "0", 2024-03-11T21:27:25.249 INFO:teuthology.orchestra.run.smithi092.stdout: "filestore_max_inline_xattr_size_btrfs": "2048", 2024-03-11T21:27:25.249 INFO:teuthology.orchestra.run.smithi092.stdout: "filestore_max_inline_xattr_size_other": "512", 2024-03-11T21:27:25.249 INFO:teuthology.orchestra.run.smithi092.stdout: "filestore_max_inline_xattr_size_xfs": "65536", 2024-03-11T21:27:25.249 INFO:teuthology.orchestra.run.smithi092.stdout: "filestore_max_inline_xattrs": "0", 2024-03-11T21:27:25.249 INFO:teuthology.orchestra.run.smithi092.stdout: "filestore_max_inline_xattrs_btrfs": "10", 2024-03-11T21:27:25.249 INFO:teuthology.orchestra.run.smithi092.stdout: "filestore_max_inline_xattrs_other": "2", 2024-03-11T21:27:25.249 INFO:teuthology.orchestra.run.smithi092.stdout: "filestore_max_inline_xattrs_xfs": "10", 2024-03-11T21:27:25.249 INFO:teuthology.orchestra.run.smithi092.stdout: "filestore_max_sync_interval": "5.000000", 2024-03-11T21:27:25.249 INFO:teuthology.orchestra.run.smithi092.stdout: "filestore_max_xattr_value_size": "0", 2024-03-11T21:27:25.250 INFO:teuthology.orchestra.run.smithi092.stdout: "filestore_max_xattr_value_size_btrfs": "65536", 2024-03-11T21:27:25.250 INFO:teuthology.orchestra.run.smithi092.stdout: "filestore_max_xattr_value_size_other": "1024", 2024-03-11T21:27:25.250 INFO:teuthology.orchestra.run.smithi092.stdout: "filestore_max_xattr_value_size_xfs": "65536", 2024-03-11T21:27:25.250 INFO:teuthology.orchestra.run.smithi092.stdout: "filestore_merge_threshold": "-10", 2024-03-11T21:27:25.250 INFO:teuthology.orchestra.run.smithi092.stdout: "filestore_min_sync_interval": "0.010000", 2024-03-11T21:27:25.250 INFO:teuthology.orchestra.run.smithi092.stdout: "filestore_odsync_write": "false", 2024-03-11T21:27:25.250 INFO:teuthology.orchestra.run.smithi092.stdout: "filestore_omap_backend": "rocksdb", 2024-03-11T21:27:25.250 INFO:teuthology.orchestra.run.smithi092.stdout: "filestore_omap_backend_path": "", 2024-03-11T21:27:25.250 INFO:teuthology.orchestra.run.smithi092.stdout: "filestore_omap_header_cache_size": "1024", 2024-03-11T21:27:25.250 INFO:teuthology.orchestra.run.smithi092.stdout: "filestore_ondisk_finisher_threads": "1", 2024-03-11T21:27:25.250 INFO:teuthology.orchestra.run.smithi092.stdout: "filestore_op_thread_suicide_timeout": "180", 2024-03-11T21:27:25.251 INFO:teuthology.orchestra.run.smithi092.stdout: "filestore_op_thread_timeout": "60", 2024-03-11T21:27:25.251 INFO:teuthology.orchestra.run.smithi092.stdout: "filestore_op_threads": "2", 2024-03-11T21:27:25.251 INFO:teuthology.orchestra.run.smithi092.stdout: "filestore_punch_hole": "false", 2024-03-11T21:27:25.251 INFO:teuthology.orchestra.run.smithi092.stdout: "filestore_queue_high_delay_multiple": "0.000000", 2024-03-11T21:27:25.251 INFO:teuthology.orchestra.run.smithi092.stdout: "filestore_queue_high_delay_multiple_bytes": "0.000000", 2024-03-11T21:27:25.251 INFO:teuthology.orchestra.run.smithi092.stdout: "filestore_queue_high_delay_multiple_ops": "0.000000", 2024-03-11T21:27:25.251 INFO:teuthology.orchestra.run.smithi092.stdout: "filestore_queue_high_threshhold": "0.900000", 2024-03-11T21:27:25.252 INFO:teuthology.orchestra.run.smithi092.stdout: "filestore_queue_low_threshhold": "0.300000", 2024-03-11T21:27:25.252 INFO:teuthology.orchestra.run.smithi092.stdout: "filestore_queue_max_bytes": "104857600", 2024-03-11T21:27:25.252 INFO:teuthology.orchestra.run.smithi092.stdout: "filestore_queue_max_delay_multiple": "0.000000", 2024-03-11T21:27:25.252 INFO:teuthology.orchestra.run.smithi092.stdout: "filestore_queue_max_delay_multiple_bytes": "0.000000", 2024-03-11T21:27:25.252 INFO:teuthology.orchestra.run.smithi092.stdout: "filestore_queue_max_delay_multiple_ops": "0.000000", 2024-03-11T21:27:25.252 INFO:teuthology.orchestra.run.smithi092.stdout: "filestore_queue_max_ops": "50", 2024-03-11T21:27:25.252 INFO:teuthology.orchestra.run.smithi092.stdout: "filestore_rocksdb_options": "max_background_jobs=10,compaction_readahead_size=2097152,compression=kNoCompression", 2024-03-11T21:27:25.252 INFO:teuthology.orchestra.run.smithi092.stdout: "filestore_seek_data_hole": "false", 2024-03-11T21:27:25.252 INFO:teuthology.orchestra.run.smithi092.stdout: "filestore_sloppy_crc": "false", 2024-03-11T21:27:25.253 INFO:teuthology.orchestra.run.smithi092.stdout: "filestore_sloppy_crc_block_size": "65536", 2024-03-11T21:27:25.253 INFO:teuthology.orchestra.run.smithi092.stdout: "filestore_splice": "false", 2024-03-11T21:27:25.253 INFO:teuthology.orchestra.run.smithi092.stdout: "filestore_split_multiple": "2", 2024-03-11T21:27:25.253 INFO:teuthology.orchestra.run.smithi092.stdout: "filestore_split_rand_factor": "20", 2024-03-11T21:27:25.253 INFO:teuthology.orchestra.run.smithi092.stdout: "filestore_update_to": "1000", 2024-03-11T21:27:25.253 INFO:teuthology.orchestra.run.smithi092.stdout: "filestore_wbthrottle_btrfs_bytes_hard_limit": "419430400", 2024-03-11T21:27:25.253 INFO:teuthology.orchestra.run.smithi092.stdout: "filestore_wbthrottle_btrfs_bytes_start_flusher": "41943040", 2024-03-11T21:27:25.253 INFO:teuthology.orchestra.run.smithi092.stdout: "filestore_wbthrottle_btrfs_inodes_hard_limit": "5000", 2024-03-11T21:27:25.253 INFO:teuthology.orchestra.run.smithi092.stdout: "filestore_wbthrottle_btrfs_inodes_start_flusher": "500", 2024-03-11T21:27:25.253 INFO:teuthology.orchestra.run.smithi092.stdout: "filestore_wbthrottle_btrfs_ios_hard_limit": "5000", 2024-03-11T21:27:25.253 INFO:teuthology.orchestra.run.smithi092.stdout: "filestore_wbthrottle_btrfs_ios_start_flusher": "500", 2024-03-11T21:27:25.253 INFO:teuthology.orchestra.run.smithi092.stdout: "filestore_wbthrottle_enable": "true", 2024-03-11T21:27:25.253 INFO:teuthology.orchestra.run.smithi092.stdout: "filestore_wbthrottle_xfs_bytes_hard_limit": "419430400", 2024-03-11T21:27:25.254 INFO:teuthology.orchestra.run.smithi092.stdout: "filestore_wbthrottle_xfs_bytes_start_flusher": "41943040", 2024-03-11T21:27:25.254 INFO:teuthology.orchestra.run.smithi092.stdout: "filestore_wbthrottle_xfs_inodes_hard_limit": "5000", 2024-03-11T21:27:25.254 INFO:teuthology.orchestra.run.smithi092.stdout: "filestore_wbthrottle_xfs_inodes_start_flusher": "500", 2024-03-11T21:27:25.254 INFO:teuthology.orchestra.run.smithi092.stdout: "filestore_wbthrottle_xfs_ios_hard_limit": "5000", 2024-03-11T21:27:25.254 INFO:teuthology.orchestra.run.smithi092.stdout: "filestore_wbthrottle_xfs_ios_start_flusher": "500", 2024-03-11T21:27:25.254 INFO:teuthology.orchestra.run.smithi092.stdout: "filestore_xfs_extsize": "false", 2024-03-11T21:27:25.254 INFO:teuthology.orchestra.run.smithi092.stdout: "filestore_zfs_snap": "false", 2024-03-11T21:27:25.254 INFO:teuthology.orchestra.run.smithi092.stdout: "fio_dir": "/tmp/fio", 2024-03-11T21:27:25.254 INFO:teuthology.orchestra.run.smithi092.stdout: "fsid": "00db0016-dfed-11ee-95c8-87774f69a715", 2024-03-11T21:27:25.254 INFO:teuthology.orchestra.run.smithi092.stdout: "fuse_allow_other": "true", 2024-03-11T21:27:25.254 INFO:teuthology.orchestra.run.smithi092.stdout: "fuse_atomic_o_trunc": "true", 2024-03-11T21:27:25.255 INFO:teuthology.orchestra.run.smithi092.stdout: "fuse_big_writes": "true", 2024-03-11T21:27:25.255 INFO:teuthology.orchestra.run.smithi092.stdout: "fuse_debug": "false", 2024-03-11T21:27:25.255 INFO:teuthology.orchestra.run.smithi092.stdout: "fuse_default_permissions": "false", 2024-03-11T21:27:25.255 INFO:teuthology.orchestra.run.smithi092.stdout: "fuse_disable_pagecache": "false", 2024-03-11T21:27:25.255 INFO:teuthology.orchestra.run.smithi092.stdout: "fuse_max_write": "0", 2024-03-11T21:27:25.255 INFO:teuthology.orchestra.run.smithi092.stdout: "fuse_multithreaded": "true", 2024-03-11T21:27:25.255 INFO:teuthology.orchestra.run.smithi092.stdout: "fuse_require_active_mds": "true", 2024-03-11T21:27:25.255 INFO:teuthology.orchestra.run.smithi092.stdout: "fuse_set_user_groups": "true", 2024-03-11T21:27:25.255 INFO:teuthology.orchestra.run.smithi092.stdout: "fuse_splice_move": "true", 2024-03-11T21:27:25.256 INFO:teuthology.orchestra.run.smithi092.stdout: "fuse_splice_read": "true", 2024-03-11T21:27:25.256 INFO:teuthology.orchestra.run.smithi092.stdout: "fuse_splice_write": "true", 2024-03-11T21:27:25.256 INFO:teuthology.orchestra.run.smithi092.stdout: "fuse_syncfs_on_mksnap": "true", 2024-03-11T21:27:25.256 INFO:teuthology.orchestra.run.smithi092.stdout: "fuse_use_invalidate_cb": "true", 2024-03-11T21:27:25.256 INFO:teuthology.orchestra.run.smithi092.stdout: "gss_ktab_client_file": "/var/lib/ceph/mon.a/gss_client_mon.a.ktab", 2024-03-11T21:27:25.256 INFO:teuthology.orchestra.run.smithi092.stdout: "gss_target_name": "ceph", 2024-03-11T21:27:25.256 INFO:teuthology.orchestra.run.smithi092.stdout: "heartbeat_file": "", 2024-03-11T21:27:25.256 INFO:teuthology.orchestra.run.smithi092.stdout: "heartbeat_inject_failure": "0", 2024-03-11T21:27:25.257 INFO:teuthology.orchestra.run.smithi092.stdout: "heartbeat_interval": "5", 2024-03-11T21:27:25.257 INFO:teuthology.orchestra.run.smithi092.stdout: "host": "smithi092", 2024-03-11T21:27:25.257 INFO:teuthology.orchestra.run.smithi092.stdout: "immutable_object_cache_client_dedicated_thread_num": "2", 2024-03-11T21:27:25.257 INFO:teuthology.orchestra.run.smithi092.stdout: "immutable_object_cache_max_inflight_ops": "128", 2024-03-11T21:27:25.257 INFO:teuthology.orchestra.run.smithi092.stdout: "immutable_object_cache_max_size": "1073741824", 2024-03-11T21:27:25.257 INFO:teuthology.orchestra.run.smithi092.stdout: "immutable_object_cache_path": "/tmp/ceph_immutable_object_cache", 2024-03-11T21:27:25.257 INFO:teuthology.orchestra.run.smithi092.stdout: "immutable_object_cache_qos_bps_burst": "0", 2024-03-11T21:27:25.257 INFO:teuthology.orchestra.run.smithi092.stdout: "immutable_object_cache_qos_bps_burst_seconds": "1", 2024-03-11T21:27:25.257 INFO:teuthology.orchestra.run.smithi092.stdout: "immutable_object_cache_qos_bps_limit": "0", 2024-03-11T21:27:25.258 INFO:teuthology.orchestra.run.smithi092.stdout: "immutable_object_cache_qos_iops_burst": "0", 2024-03-11T21:27:25.258 INFO:teuthology.orchestra.run.smithi092.stdout: "immutable_object_cache_qos_iops_burst_seconds": "1", 2024-03-11T21:27:25.258 INFO:teuthology.orchestra.run.smithi092.stdout: "immutable_object_cache_qos_iops_limit": "0", 2024-03-11T21:27:25.258 INFO:teuthology.orchestra.run.smithi092.stdout: "immutable_object_cache_qos_schedule_tick_min": "50", 2024-03-11T21:27:25.258 INFO:teuthology.orchestra.run.smithi092.stdout: "immutable_object_cache_sock": "/var/run/ceph/immutable_object_cache_sock", 2024-03-11T21:27:25.258 INFO:teuthology.orchestra.run.smithi092.stdout: "immutable_object_cache_watermark": "0.900000", 2024-03-11T21:27:25.258 INFO:teuthology.orchestra.run.smithi092.stdout: "inject_early_sigterm": "false", 2024-03-11T21:27:25.258 INFO:teuthology.orchestra.run.smithi092.stdout: "jaeger_agent_port": "6799", 2024-03-11T21:27:25.258 INFO:teuthology.orchestra.run.smithi092.stdout: "jaeger_tracing_enable": "false", 2024-03-11T21:27:25.258 INFO:teuthology.orchestra.run.smithi092.stdout: "journal_aio": "true", 2024-03-11T21:27:25.258 INFO:teuthology.orchestra.run.smithi092.stdout: "journal_align_min_size": "65536", 2024-03-11T21:27:25.258 INFO:teuthology.orchestra.run.smithi092.stdout: "journal_block_align": "true", 2024-03-11T21:27:25.258 INFO:teuthology.orchestra.run.smithi092.stdout: "journal_block_size": "4096", 2024-03-11T21:27:25.258 INFO:teuthology.orchestra.run.smithi092.stdout: "journal_dio": "true", 2024-03-11T21:27:25.259 INFO:teuthology.orchestra.run.smithi092.stdout: "journal_discard": "false", 2024-03-11T21:27:25.259 INFO:teuthology.orchestra.run.smithi092.stdout: "journal_force_aio": "false", 2024-03-11T21:27:25.259 INFO:teuthology.orchestra.run.smithi092.stdout: "journal_ignore_corruption": "false", 2024-03-11T21:27:25.259 INFO:teuthology.orchestra.run.smithi092.stdout: "journal_max_write_bytes": "10485760", 2024-03-11T21:27:25.259 INFO:teuthology.orchestra.run.smithi092.stdout: "journal_max_write_entries": "100", 2024-03-11T21:27:25.259 INFO:teuthology.orchestra.run.smithi092.stdout: "journal_replay_from": "0", 2024-03-11T21:27:25.259 INFO:teuthology.orchestra.run.smithi092.stdout: "journal_throttle_high_multiple": "0.000000", 2024-03-11T21:27:25.259 INFO:teuthology.orchestra.run.smithi092.stdout: "journal_throttle_high_threshhold": "0.900000", 2024-03-11T21:27:25.259 INFO:teuthology.orchestra.run.smithi092.stdout: "journal_throttle_low_threshhold": "0.600000", 2024-03-11T21:27:25.259 INFO:teuthology.orchestra.run.smithi092.stdout: "journal_throttle_max_multiple": "0.000000", 2024-03-11T21:27:25.259 INFO:teuthology.orchestra.run.smithi092.stdout: "journal_write_header_frequency": "0", 2024-03-11T21:27:25.259 INFO:teuthology.orchestra.run.smithi092.stdout: "journal_zero_on_create": "false", 2024-03-11T21:27:25.259 INFO:teuthology.orchestra.run.smithi092.stdout: "journaler_prefetch_periods": "10", 2024-03-11T21:27:25.259 INFO:teuthology.orchestra.run.smithi092.stdout: "journaler_prezero_periods": "5", 2024-03-11T21:27:25.260 INFO:teuthology.orchestra.run.smithi092.stdout: "journaler_write_head_interval": "15", 2024-03-11T21:27:25.260 INFO:teuthology.orchestra.run.smithi092.stdout: "key": "", 2024-03-11T21:27:25.260 INFO:teuthology.orchestra.run.smithi092.stdout: "keyfile": "", 2024-03-11T21:27:25.260 INFO:teuthology.orchestra.run.smithi092.stdout: "keyring": "/var/lib/ceph/mon/ceph-a/keyring", 2024-03-11T21:27:25.260 INFO:teuthology.orchestra.run.smithi092.stdout: "kstore_backend": "rocksdb", 2024-03-11T21:27:25.260 INFO:teuthology.orchestra.run.smithi092.stdout: "kstore_default_stripe_size": "65536", 2024-03-11T21:27:25.260 INFO:teuthology.orchestra.run.smithi092.stdout: "kstore_fsck_on_mount": "false", 2024-03-11T21:27:25.260 INFO:teuthology.orchestra.run.smithi092.stdout: "kstore_fsck_on_mount_deep": "true", 2024-03-11T21:27:25.260 INFO:teuthology.orchestra.run.smithi092.stdout: "kstore_max_bytes": "67108864", 2024-03-11T21:27:25.260 INFO:teuthology.orchestra.run.smithi092.stdout: "kstore_max_ops": "512", 2024-03-11T21:27:25.260 INFO:teuthology.orchestra.run.smithi092.stdout: "kstore_nid_prealloc": "1024", 2024-03-11T21:27:25.260 INFO:teuthology.orchestra.run.smithi092.stdout: "kstore_onode_map_size": "1024", 2024-03-11T21:27:25.260 INFO:teuthology.orchestra.run.smithi092.stdout: "kstore_rocksdb_options": "compression=kNoCompression", 2024-03-11T21:27:25.260 INFO:teuthology.orchestra.run.smithi092.stdout: "kstore_sync_submit_transaction": "false", 2024-03-11T21:27:25.261 INFO:teuthology.orchestra.run.smithi092.stdout: "kstore_sync_transaction": "false", 2024-03-11T21:27:25.261 INFO:teuthology.orchestra.run.smithi092.stdout: "librados_thread_count": "2", 2024-03-11T21:27:25.261 INFO:teuthology.orchestra.run.smithi092.stdout: "lockdep": "false", 2024-03-11T21:27:25.261 INFO:teuthology.orchestra.run.smithi092.stdout: "lockdep_force_backtrace": "false", 2024-03-11T21:27:25.261 INFO:teuthology.orchestra.run.smithi092.stdout: "log_coarse_timestamps": "true", 2024-03-11T21:27:25.261 INFO:teuthology.orchestra.run.smithi092.stdout: "log_file": "/var/log/ceph/ceph-mon.a.log", 2024-03-11T21:27:25.261 INFO:teuthology.orchestra.run.smithi092.stdout: "log_flush_on_exit": "false", 2024-03-11T21:27:25.261 INFO:teuthology.orchestra.run.smithi092.stdout: "log_graylog_host": "127.0.0.1", 2024-03-11T21:27:25.261 INFO:teuthology.orchestra.run.smithi092.stdout: "log_graylog_port": "12201", 2024-03-11T21:27:25.261 INFO:teuthology.orchestra.run.smithi092.stdout: "log_max_new": "1000", 2024-03-11T21:27:25.261 INFO:teuthology.orchestra.run.smithi092.stdout: "log_max_recent": "10000", 2024-03-11T21:27:25.261 INFO:teuthology.orchestra.run.smithi092.stdout: "log_stderr_prefix": "", 2024-03-11T21:27:25.261 INFO:teuthology.orchestra.run.smithi092.stdout: "log_stop_at_utilization": "0.970000", 2024-03-11T21:27:25.261 INFO:teuthology.orchestra.run.smithi092.stdout: "log_to_file": "true", 2024-03-11T21:27:25.262 INFO:teuthology.orchestra.run.smithi092.stdout: "log_to_graylog": "false", 2024-03-11T21:27:25.262 INFO:teuthology.orchestra.run.smithi092.stdout: "log_to_journald": "false", 2024-03-11T21:27:25.262 INFO:teuthology.orchestra.run.smithi092.stdout: "log_to_stderr": "false", 2024-03-11T21:27:25.262 INFO:teuthology.orchestra.run.smithi092.stdout: "log_to_syslog": "false", 2024-03-11T21:27:25.262 INFO:teuthology.orchestra.run.smithi092.stdout: "max_rotating_auth_attempts": "10", 2024-03-11T21:27:25.262 INFO:teuthology.orchestra.run.smithi092.stdout: "mds_abort_on_newly_corrupt_dentry": "true", 2024-03-11T21:27:25.262 INFO:teuthology.orchestra.run.smithi092.stdout: "mds_action_on_write_error": "1", 2024-03-11T21:27:25.262 INFO:teuthology.orchestra.run.smithi092.stdout: "mds_alternate_name_max": "8192", 2024-03-11T21:27:25.262 INFO:teuthology.orchestra.run.smithi092.stdout: "mds_asio_thread_count": "2", 2024-03-11T21:27:25.262 INFO:teuthology.orchestra.run.smithi092.stdout: "mds_bal_export_pin": "true", 2024-03-11T21:27:25.262 INFO:teuthology.orchestra.run.smithi092.stdout: "mds_bal_fragment_dirs": "true", 2024-03-11T21:27:25.262 INFO:teuthology.orchestra.run.smithi092.stdout: "mds_bal_fragment_fast_factor": "1.500000", 2024-03-11T21:27:25.262 INFO:teuthology.orchestra.run.smithi092.stdout: "mds_bal_fragment_interval": "5", 2024-03-11T21:27:25.263 INFO:teuthology.orchestra.run.smithi092.stdout: "mds_bal_fragment_size_max": "100000", 2024-03-11T21:27:25.263 INFO:teuthology.orchestra.run.smithi092.stdout: "mds_bal_idle_threshold": "0.000000", 2024-03-11T21:27:25.263 INFO:teuthology.orchestra.run.smithi092.stdout: "mds_bal_interval": "10", 2024-03-11T21:27:25.263 INFO:teuthology.orchestra.run.smithi092.stdout: "mds_bal_max": "-1", 2024-03-11T21:27:25.263 INFO:teuthology.orchestra.run.smithi092.stdout: "mds_bal_max_until": "-1", 2024-03-11T21:27:25.263 INFO:teuthology.orchestra.run.smithi092.stdout: "mds_bal_merge_size": "50", 2024-03-11T21:27:25.263 INFO:teuthology.orchestra.run.smithi092.stdout: "mds_bal_midchunk": "0.300000", 2024-03-11T21:27:25.263 INFO:teuthology.orchestra.run.smithi092.stdout: "mds_bal_min_rebalance": "0.100000", 2024-03-11T21:27:25.263 INFO:teuthology.orchestra.run.smithi092.stdout: "mds_bal_min_start": "0.200000", 2024-03-11T21:27:25.263 INFO:teuthology.orchestra.run.smithi092.stdout: "mds_bal_minchunk": "0.001000", 2024-03-11T21:27:25.263 INFO:teuthology.orchestra.run.smithi092.stdout: "mds_bal_mode": "0", 2024-03-11T21:27:25.263 INFO:teuthology.orchestra.run.smithi092.stdout: "mds_bal_need_max": "1.200000", 2024-03-11T21:27:25.263 INFO:teuthology.orchestra.run.smithi092.stdout: "mds_bal_need_min": "0.800000", 2024-03-11T21:27:25.264 INFO:teuthology.orchestra.run.smithi092.stdout: "mds_bal_overload_epochs": "2", 2024-03-11T21:27:25.264 INFO:teuthology.orchestra.run.smithi092.stdout: "mds_bal_replicate_threshold": "8000.000000", 2024-03-11T21:27:25.264 INFO:teuthology.orchestra.run.smithi092.stdout: "mds_bal_sample_interval": "3.000000", 2024-03-11T21:27:25.264 INFO:teuthology.orchestra.run.smithi092.stdout: "mds_bal_split_bits": "3", 2024-03-11T21:27:25.264 INFO:teuthology.orchestra.run.smithi092.stdout: "mds_bal_split_rd": "25000.000000", 2024-03-11T21:27:25.264 INFO:teuthology.orchestra.run.smithi092.stdout: "mds_bal_split_size": "10000", 2024-03-11T21:27:25.264 INFO:teuthology.orchestra.run.smithi092.stdout: "mds_bal_split_wr": "10000.000000", 2024-03-11T21:27:25.264 INFO:teuthology.orchestra.run.smithi092.stdout: "mds_bal_target_decay": "10.000000", 2024-03-11T21:27:25.264 INFO:teuthology.orchestra.run.smithi092.stdout: "mds_bal_unreplicate_threshold": "0.000000", 2024-03-11T21:27:25.264 INFO:teuthology.orchestra.run.smithi092.stdout: "mds_beacon_grace": "15.000000", 2024-03-11T21:27:25.264 INFO:teuthology.orchestra.run.smithi092.stdout: "mds_beacon_interval": "4.000000", 2024-03-11T21:27:25.264 INFO:teuthology.orchestra.run.smithi092.stdout: "mds_beacon_mon_down_grace": "60", 2024-03-11T21:27:25.264 INFO:teuthology.orchestra.run.smithi092.stdout: "mds_cache_memory_limit": "4294967296", 2024-03-11T21:27:25.264 INFO:teuthology.orchestra.run.smithi092.stdout: "mds_cache_mid": "0.700000", 2024-03-11T21:27:25.265 INFO:teuthology.orchestra.run.smithi092.stdout: "mds_cache_release_free_interval": "10", 2024-03-11T21:27:25.265 INFO:teuthology.orchestra.run.smithi092.stdout: "mds_cache_reservation": "0.050000", 2024-03-11T21:27:25.265 INFO:teuthology.orchestra.run.smithi092.stdout: "mds_cache_trim_decay_rate": "1.000000", 2024-03-11T21:27:25.265 INFO:teuthology.orchestra.run.smithi092.stdout: "mds_cache_trim_interval": "1", 2024-03-11T21:27:25.265 INFO:teuthology.orchestra.run.smithi092.stdout: "mds_cache_trim_threshold": "262144", 2024-03-11T21:27:25.265 INFO:teuthology.orchestra.run.smithi092.stdout: "mds_cap_acquisition_throttle_retry_request_timeout": "0.500000", 2024-03-11T21:27:25.265 INFO:teuthology.orchestra.run.smithi092.stdout: "mds_cap_revoke_eviction_timeout": "0.000000", 2024-03-11T21:27:25.265 INFO:teuthology.orchestra.run.smithi092.stdout: "mds_client_delegate_inos_pct": "50", 2024-03-11T21:27:25.265 INFO:teuthology.orchestra.run.smithi092.stdout: "mds_client_prealloc_inos": "1000", 2024-03-11T21:27:25.265 INFO:teuthology.orchestra.run.smithi092.stdout: "mds_client_writeable_range_max_inc_objs": "1024", 2024-03-11T21:27:25.265 INFO:teuthology.orchestra.run.smithi092.stdout: "mds_connect_bootstrapping": "false", 2024-03-11T21:27:25.265 INFO:teuthology.orchestra.run.smithi092.stdout: "mds_damage_table_max_entries": "10000", 2024-03-11T21:27:25.265 INFO:teuthology.orchestra.run.smithi092.stdout: "mds_data": "/var/lib/ceph/mds/ceph-a", 2024-03-11T21:27:25.266 INFO:teuthology.orchestra.run.smithi092.stdout: "mds_debug_auth_pins": "false", 2024-03-11T21:27:25.266 INFO:teuthology.orchestra.run.smithi092.stdout: "mds_debug_frag": "false", 2024-03-11T21:27:25.266 INFO:teuthology.orchestra.run.smithi092.stdout: "mds_debug_scatterstat": "false", 2024-03-11T21:27:25.266 INFO:teuthology.orchestra.run.smithi092.stdout: "mds_debug_subtrees": "false", 2024-03-11T21:27:25.266 INFO:teuthology.orchestra.run.smithi092.stdout: "mds_decay_halflife": "5.000000", 2024-03-11T21:27:25.266 INFO:teuthology.orchestra.run.smithi092.stdout: "mds_default_dir_hash": "2", 2024-03-11T21:27:25.266 INFO:teuthology.orchestra.run.smithi092.stdout: "mds_defer_session_stale": "true", 2024-03-11T21:27:25.266 INFO:teuthology.orchestra.run.smithi092.stdout: "mds_deny_all_reconnect": "false", 2024-03-11T21:27:25.266 INFO:teuthology.orchestra.run.smithi092.stdout: "mds_dir_keys_per_op": "16384", 2024-03-11T21:27:25.266 INFO:teuthology.orchestra.run.smithi092.stdout: "mds_dir_max_commit_size": "10", 2024-03-11T21:27:25.266 INFO:teuthology.orchestra.run.smithi092.stdout: "mds_dir_max_entries": "0", 2024-03-11T21:27:25.266 INFO:teuthology.orchestra.run.smithi092.stdout: "mds_dir_prefetch": "true", 2024-03-11T21:27:25.266 INFO:teuthology.orchestra.run.smithi092.stdout: "mds_dirstat_min_interval": "1.000000", 2024-03-11T21:27:25.267 INFO:teuthology.orchestra.run.smithi092.stdout: "mds_dump_cache_after_rejoin": "false", 2024-03-11T21:27:25.267 INFO:teuthology.orchestra.run.smithi092.stdout: "mds_dump_cache_on_map": "false", 2024-03-11T21:27:25.267 INFO:teuthology.orchestra.run.smithi092.stdout: "mds_dump_cache_threshold_file": "0", 2024-03-11T21:27:25.267 INFO:teuthology.orchestra.run.smithi092.stdout: "mds_dump_cache_threshold_formatter": "1073741824", 2024-03-11T21:27:25.267 INFO:teuthology.orchestra.run.smithi092.stdout: "mds_early_reply": "true", 2024-03-11T21:27:25.267 INFO:teuthology.orchestra.run.smithi092.stdout: "mds_enable_op_tracker": "true", 2024-03-11T21:27:25.267 INFO:teuthology.orchestra.run.smithi092.stdout: "mds_enforce_unique_name": "true", 2024-03-11T21:27:25.267 INFO:teuthology.orchestra.run.smithi092.stdout: "mds_export_ephemeral_distributed": "true", 2024-03-11T21:27:25.267 INFO:teuthology.orchestra.run.smithi092.stdout: "mds_export_ephemeral_distributed_factor": "2.000000", 2024-03-11T21:27:25.267 INFO:teuthology.orchestra.run.smithi092.stdout: "mds_export_ephemeral_random": "true", 2024-03-11T21:27:25.267 INFO:teuthology.orchestra.run.smithi092.stdout: "mds_export_ephemeral_random_max": "0.010000", 2024-03-11T21:27:25.267 INFO:teuthology.orchestra.run.smithi092.stdout: "mds_extraordinary_events_dump_interval": "0", 2024-03-11T21:27:25.267 INFO:teuthology.orchestra.run.smithi092.stdout: "mds_forward_all_requests_to_auth": "false", 2024-03-11T21:27:25.267 INFO:teuthology.orchestra.run.smithi092.stdout: "mds_freeze_tree_timeout": "30.000000", 2024-03-11T21:27:25.268 INFO:teuthology.orchestra.run.smithi092.stdout: "mds_fscrypt_last_block_max_size": "4096", 2024-03-11T21:27:25.268 INFO:teuthology.orchestra.run.smithi092.stdout: "mds_go_bad_corrupt_dentry": "true", 2024-03-11T21:27:25.268 INFO:teuthology.orchestra.run.smithi092.stdout: "mds_hack_allow_loading_invalid_metadata": "false", 2024-03-11T21:27:25.268 INFO:teuthology.orchestra.run.smithi092.stdout: "mds_health_cache_threshold": "1.500000", 2024-03-11T21:27:25.268 INFO:teuthology.orchestra.run.smithi092.stdout: "mds_health_summarize_threshold": "10", 2024-03-11T21:27:25.268 INFO:teuthology.orchestra.run.smithi092.stdout: "mds_heartbeat_grace": "15.000000", 2024-03-11T21:27:25.268 INFO:teuthology.orchestra.run.smithi092.stdout: "mds_heartbeat_reset_grace": "1000", 2024-03-11T21:27:25.268 INFO:teuthology.orchestra.run.smithi092.stdout: "mds_inject_health_dummy": "false", 2024-03-11T21:27:25.268 INFO:teuthology.orchestra.run.smithi092.stdout: "mds_inject_journal_corrupt_dentry_first": "0.000000", 2024-03-11T21:27:25.268 INFO:teuthology.orchestra.run.smithi092.stdout: "mds_inject_migrator_session_race": "false", 2024-03-11T21:27:25.268 INFO:teuthology.orchestra.run.smithi092.stdout: "mds_inject_rename_corrupt_dentry_first": "0.000000", 2024-03-11T21:27:25.268 INFO:teuthology.orchestra.run.smithi092.stdout: "mds_inject_skip_replaying_inotable": "false", 2024-03-11T21:27:25.268 INFO:teuthology.orchestra.run.smithi092.stdout: "mds_inject_traceless_reply_probability": "0.000000", 2024-03-11T21:27:25.268 INFO:teuthology.orchestra.run.smithi092.stdout: "mds_join_fs": "", 2024-03-11T21:27:25.269 INFO:teuthology.orchestra.run.smithi092.stdout: "mds_journal_format": "1", 2024-03-11T21:27:25.269 INFO:teuthology.orchestra.run.smithi092.stdout: "mds_kill_create_at": "0", 2024-03-11T21:27:25.269 INFO:teuthology.orchestra.run.smithi092.stdout: "mds_kill_export_at": "0", 2024-03-11T21:27:25.269 INFO:teuthology.orchestra.run.smithi092.stdout: "mds_kill_import_at": "0", 2024-03-11T21:27:25.269 INFO:teuthology.orchestra.run.smithi092.stdout: "mds_kill_journal_at": "0", 2024-03-11T21:27:25.269 INFO:teuthology.orchestra.run.smithi092.stdout: "mds_kill_journal_expire_at": "0", 2024-03-11T21:27:25.269 INFO:teuthology.orchestra.run.smithi092.stdout: "mds_kill_journal_replay_at": "0", 2024-03-11T21:27:25.269 INFO:teuthology.orchestra.run.smithi092.stdout: "mds_kill_link_at": "0", 2024-03-11T21:27:25.269 INFO:teuthology.orchestra.run.smithi092.stdout: "mds_kill_mdstable_at": "0", 2024-03-11T21:27:25.269 INFO:teuthology.orchestra.run.smithi092.stdout: "mds_kill_openc_at": "0", 2024-03-11T21:27:25.269 INFO:teuthology.orchestra.run.smithi092.stdout: "mds_kill_rename_at": "0", 2024-03-11T21:27:25.269 INFO:teuthology.orchestra.run.smithi092.stdout: "mds_kill_shutdown_at": "0", 2024-03-11T21:27:25.269 INFO:teuthology.orchestra.run.smithi092.stdout: "mds_kill_skip_replaying_inotable": "false", 2024-03-11T21:27:25.269 INFO:teuthology.orchestra.run.smithi092.stdout: "mds_log_event_large_threshold": "524288", 2024-03-11T21:27:25.269 INFO:teuthology.orchestra.run.smithi092.stdout: "mds_log_events_per_segment": "1024", 2024-03-11T21:27:25.270 INFO:teuthology.orchestra.run.smithi092.stdout: "mds_log_major_segment_event_ratio": "12", 2024-03-11T21:27:25.270 INFO:teuthology.orchestra.run.smithi092.stdout: "mds_log_max_events": "-1", 2024-03-11T21:27:25.270 INFO:teuthology.orchestra.run.smithi092.stdout: "mds_log_max_segments": "128", 2024-03-11T21:27:25.270 INFO:teuthology.orchestra.run.smithi092.stdout: "mds_log_pause": "false", 2024-03-11T21:27:25.270 INFO:teuthology.orchestra.run.smithi092.stdout: "mds_log_segment_size": "0", 2024-03-11T21:27:25.270 INFO:teuthology.orchestra.run.smithi092.stdout: "mds_log_skip_corrupt_events": "false", 2024-03-11T21:27:25.270 INFO:teuthology.orchestra.run.smithi092.stdout: "mds_log_skip_unbounded_events": "false", 2024-03-11T21:27:25.270 INFO:teuthology.orchestra.run.smithi092.stdout: "mds_log_trim_decay_rate": "1.000000", 2024-03-11T21:27:25.270 INFO:teuthology.orchestra.run.smithi092.stdout: "mds_log_trim_threshold": "128", 2024-03-11T21:27:25.270 INFO:teuthology.orchestra.run.smithi092.stdout: "mds_log_trim_upkeep_interval": "1000", 2024-03-11T21:27:25.270 INFO:teuthology.orchestra.run.smithi092.stdout: "mds_log_warn_factor": "2.000000", 2024-03-11T21:27:25.270 INFO:teuthology.orchestra.run.smithi092.stdout: "mds_max_caps_per_client": "1048576", 2024-03-11T21:27:25.270 INFO:teuthology.orchestra.run.smithi092.stdout: "mds_max_completed_flushes": "100000", 2024-03-11T21:27:25.271 INFO:teuthology.orchestra.run.smithi092.stdout: "mds_max_completed_requests": "100000", 2024-03-11T21:27:25.271 INFO:teuthology.orchestra.run.smithi092.stdout: "mds_max_export_size": "20971520", 2024-03-11T21:27:25.271 INFO:teuthology.orchestra.run.smithi092.stdout: "mds_max_file_recover": "32", 2024-03-11T21:27:25.271 INFO:teuthology.orchestra.run.smithi092.stdout: "mds_max_purge_files": "64", 2024-03-11T21:27:25.271 INFO:teuthology.orchestra.run.smithi092.stdout: "mds_max_purge_ops": "8192", 2024-03-11T21:27:25.271 INFO:teuthology.orchestra.run.smithi092.stdout: "mds_max_purge_ops_per_pg": "0.500000", 2024-03-11T21:27:25.271 INFO:teuthology.orchestra.run.smithi092.stdout: "mds_max_scrub_ops_in_progress": "5", 2024-03-11T21:27:25.271 INFO:teuthology.orchestra.run.smithi092.stdout: "mds_max_snaps_per_dir": "100", 2024-03-11T21:27:25.271 INFO:teuthology.orchestra.run.smithi092.stdout: "mds_metrics_update_interval": "2", 2024-03-11T21:27:25.271 INFO:teuthology.orchestra.run.smithi092.stdout: "mds_min_caps_per_client": "100", 2024-03-11T21:27:25.271 INFO:teuthology.orchestra.run.smithi092.stdout: "mds_min_caps_working_set": "10000", 2024-03-11T21:27:25.271 INFO:teuthology.orchestra.run.smithi092.stdout: "mds_mon_shutdown_timeout": "5.000000", 2024-03-11T21:27:25.271 INFO:teuthology.orchestra.run.smithi092.stdout: "mds_numa_node": "-1", 2024-03-11T21:27:25.272 INFO:teuthology.orchestra.run.smithi092.stdout: "mds_oft_prefetch_dirfrags": "false", 2024-03-11T21:27:25.272 INFO:teuthology.orchestra.run.smithi092.stdout: "mds_op_complaint_time": "30.000000", 2024-03-11T21:27:25.272 INFO:teuthology.orchestra.run.smithi092.stdout: "mds_op_history_duration": "600", 2024-03-11T21:27:25.272 INFO:teuthology.orchestra.run.smithi092.stdout: "mds_op_history_size": "20", 2024-03-11T21:27:25.272 INFO:teuthology.orchestra.run.smithi092.stdout: "mds_op_history_slow_op_size": "20", 2024-03-11T21:27:25.272 INFO:teuthology.orchestra.run.smithi092.stdout: "mds_op_history_slow_op_threshold": "10", 2024-03-11T21:27:25.272 INFO:teuthology.orchestra.run.smithi092.stdout: "mds_op_log_threshold": "5", 2024-03-11T21:27:25.272 INFO:teuthology.orchestra.run.smithi092.stdout: "mds_ping_grace": "15", 2024-03-11T21:27:25.272 INFO:teuthology.orchestra.run.smithi092.stdout: "mds_ping_interval": "5", 2024-03-11T21:27:25.272 INFO:teuthology.orchestra.run.smithi092.stdout: "mds_purge_queue_busy_flush_period": "1.000000", 2024-03-11T21:27:25.272 INFO:teuthology.orchestra.run.smithi092.stdout: "mds_recall_global_max_decay_threshold": "131072", 2024-03-11T21:27:25.272 INFO:teuthology.orchestra.run.smithi092.stdout: "mds_recall_max_caps": "30000", 2024-03-11T21:27:25.272 INFO:teuthology.orchestra.run.smithi092.stdout: "mds_recall_max_decay_rate": "1.500000", 2024-03-11T21:27:25.272 INFO:teuthology.orchestra.run.smithi092.stdout: "mds_recall_max_decay_threshold": "131072", 2024-03-11T21:27:25.273 INFO:teuthology.orchestra.run.smithi092.stdout: "mds_recall_warning_decay_rate": "60.000000", 2024-03-11T21:27:25.273 INFO:teuthology.orchestra.run.smithi092.stdout: "mds_recall_warning_threshold": "262144", 2024-03-11T21:27:25.273 INFO:teuthology.orchestra.run.smithi092.stdout: "mds_reconnect_timeout": "45.000000", 2024-03-11T21:27:25.273 INFO:teuthology.orchestra.run.smithi092.stdout: "mds_replay_interval": "1.000000", 2024-03-11T21:27:25.273 INFO:teuthology.orchestra.run.smithi092.stdout: "mds_replay_unsafe_with_closed_session": "false", 2024-03-11T21:27:25.273 INFO:teuthology.orchestra.run.smithi092.stdout: "mds_request_load_average_decay_rate": "60.000000", 2024-03-11T21:27:25.273 INFO:teuthology.orchestra.run.smithi092.stdout: "mds_root_ino_gid": "0", 2024-03-11T21:27:25.273 INFO:teuthology.orchestra.run.smithi092.stdout: "mds_root_ino_uid": "0", 2024-03-11T21:27:25.273 INFO:teuthology.orchestra.run.smithi092.stdout: "mds_scatter_nudge_interval": "5.000000", 2024-03-11T21:27:25.273 INFO:teuthology.orchestra.run.smithi092.stdout: "mds_session_blocklist_on_evict": "true", 2024-03-11T21:27:25.273 INFO:teuthology.orchestra.run.smithi092.stdout: "mds_session_blocklist_on_timeout": "true", 2024-03-11T21:27:25.273 INFO:teuthology.orchestra.run.smithi092.stdout: "mds_session_cache_liveness_decay_rate": "300.000000", 2024-03-11T21:27:25.273 INFO:teuthology.orchestra.run.smithi092.stdout: "mds_session_cache_liveness_magnitude": "10", 2024-03-11T21:27:25.273 INFO:teuthology.orchestra.run.smithi092.stdout: "mds_session_cap_acquisition_decay_rate": "30.000000", 2024-03-11T21:27:25.273 INFO:teuthology.orchestra.run.smithi092.stdout: "mds_session_cap_acquisition_throttle": "100000", 2024-03-11T21:27:25.274 INFO:teuthology.orchestra.run.smithi092.stdout: "mds_session_max_caps_throttle_ratio": "1.100000", 2024-03-11T21:27:25.274 INFO:teuthology.orchestra.run.smithi092.stdout: "mds_session_metadata_threshold": "16777216", 2024-03-11T21:27:25.274 INFO:teuthology.orchestra.run.smithi092.stdout: "mds_sessionmap_keys_per_op": "1024", 2024-03-11T21:27:25.274 INFO:teuthology.orchestra.run.smithi092.stdout: "mds_shutdown_check": "0", 2024-03-11T21:27:25.274 INFO:teuthology.orchestra.run.smithi092.stdout: "mds_skip_ino": "0", 2024-03-11T21:27:25.274 INFO:teuthology.orchestra.run.smithi092.stdout: "mds_sleep_rank_change": "0.000000", 2024-03-11T21:27:25.274 INFO:teuthology.orchestra.run.smithi092.stdout: "mds_snap_max_uid": "4294967294", 2024-03-11T21:27:25.274 INFO:teuthology.orchestra.run.smithi092.stdout: "mds_snap_min_uid": "0", 2024-03-11T21:27:25.274 INFO:teuthology.orchestra.run.smithi092.stdout: "mds_snap_rstat": "false", 2024-03-11T21:27:25.274 INFO:teuthology.orchestra.run.smithi092.stdout: "mds_standby_replay_damaged": "false", 2024-03-11T21:27:25.274 INFO:teuthology.orchestra.run.smithi092.stdout: "mds_symlink_recovery": "true", 2024-03-11T21:27:25.274 INFO:teuthology.orchestra.run.smithi092.stdout: "mds_task_status_update_interval": "2.000000", 2024-03-11T21:27:25.274 INFO:teuthology.orchestra.run.smithi092.stdout: "mds_thrash_exports": "0", 2024-03-11T21:27:25.274 INFO:teuthology.orchestra.run.smithi092.stdout: "mds_thrash_fragments": "0", 2024-03-11T21:27:25.275 INFO:teuthology.orchestra.run.smithi092.stdout: "mds_tick_interval": "5.000000", 2024-03-11T21:27:25.275 INFO:teuthology.orchestra.run.smithi092.stdout: "mds_valgrind_exit": "false", 2024-03-11T21:27:25.275 INFO:teuthology.orchestra.run.smithi092.stdout: "mds_verify_backtrace": "1", 2024-03-11T21:27:25.275 INFO:teuthology.orchestra.run.smithi092.stdout: "mds_verify_scatter": "false", 2024-03-11T21:27:25.275 INFO:teuthology.orchestra.run.smithi092.stdout: "mds_wipe_ino_prealloc": "false", 2024-03-11T21:27:25.275 INFO:teuthology.orchestra.run.smithi092.stdout: "mds_wipe_sessions": "false", 2024-03-11T21:27:25.275 INFO:teuthology.orchestra.run.smithi092.stdout: "mempool_debug": "false", 2024-03-11T21:27:25.275 INFO:teuthology.orchestra.run.smithi092.stdout: "memstore_debug_omit_block_device_write": "false", 2024-03-11T21:27:25.275 INFO:teuthology.orchestra.run.smithi092.stdout: "memstore_device_bytes": "1073741824", 2024-03-11T21:27:25.275 INFO:teuthology.orchestra.run.smithi092.stdout: "memstore_page_set": "false", 2024-03-11T21:27:25.275 INFO:teuthology.orchestra.run.smithi092.stdout: "memstore_page_size": "65536", 2024-03-11T21:27:25.275 INFO:teuthology.orchestra.run.smithi092.stdout: "mgr_client_bytes": "134217728", 2024-03-11T21:27:25.275 INFO:teuthology.orchestra.run.smithi092.stdout: "mgr_client_messages": "512", 2024-03-11T21:27:25.275 INFO:teuthology.orchestra.run.smithi092.stdout: "mgr_client_service_daemon_unregister_timeout": "1.000000", 2024-03-11T21:27:25.276 INFO:teuthology.orchestra.run.smithi092.stdout: "mgr_connect_retry_interval": "1.000000", 2024-03-11T21:27:25.276 INFO:teuthology.orchestra.run.smithi092.stdout: "mgr_data": "/var/lib/ceph/mgr/ceph-a", 2024-03-11T21:27:25.276 INFO:teuthology.orchestra.run.smithi092.stdout: "mgr_debug_aggressive_pg_num_changes": "false", 2024-03-11T21:27:25.276 INFO:teuthology.orchestra.run.smithi092.stdout: "mgr_disabled_modules": "", 2024-03-11T21:27:25.276 INFO:teuthology.orchestra.run.smithi092.stdout: "mgr_initial_modules": "restful iostat nfs", 2024-03-11T21:27:25.276 INFO:teuthology.orchestra.run.smithi092.stdout: "mgr_max_pg_num_change": "128", 2024-03-11T21:27:25.276 INFO:teuthology.orchestra.run.smithi092.stdout: "mgr_mds_bytes": "134217728", 2024-03-11T21:27:25.276 INFO:teuthology.orchestra.run.smithi092.stdout: "mgr_mds_messages": "128", 2024-03-11T21:27:25.277 INFO:teuthology.orchestra.run.smithi092.stdout: "mgr_module_path": "/usr/share/ceph/mgr", 2024-03-11T21:27:25.277 INFO:teuthology.orchestra.run.smithi092.stdout: "mgr_mon_bytes": "134217728", 2024-03-11T21:27:25.278 INFO:teuthology.orchestra.run.smithi092.stdout: "mgr_mon_messages": "128", 2024-03-11T21:27:25.278 INFO:teuthology.orchestra.run.smithi092.stdout: "mgr_osd_bytes": "536870912", 2024-03-11T21:27:25.278 INFO:teuthology.orchestra.run.smithi092.stdout: "mgr_osd_messages": "8192", 2024-03-11T21:27:25.278 INFO:teuthology.orchestra.run.smithi092.stdout: "mgr_pool": "true", 2024-03-11T21:27:25.278 INFO:teuthology.orchestra.run.smithi092.stdout: "mgr_service_beacon_grace": "60.000000", 2024-03-11T21:27:25.278 INFO:teuthology.orchestra.run.smithi092.stdout: "mgr_standby_modules": "true", 2024-03-11T21:27:25.278 INFO:teuthology.orchestra.run.smithi092.stdout: "mgr_stats_period": "5", 2024-03-11T21:27:25.278 INFO:teuthology.orchestra.run.smithi092.stdout: "mgr_stats_threshold": "5", 2024-03-11T21:27:25.278 INFO:teuthology.orchestra.run.smithi092.stdout: "mgr_tick_period": "2", 2024-03-11T21:27:25.278 INFO:teuthology.orchestra.run.smithi092.stdout: "mgr_ttl_cache_expire_seconds": "0", 2024-03-11T21:27:25.278 INFO:teuthology.orchestra.run.smithi092.stdout: "mon_accept_timeout_factor": "2.000000", 2024-03-11T21:27:25.278 INFO:teuthology.orchestra.run.smithi092.stdout: "mon_allow_pool_delete": "true", 2024-03-11T21:27:25.278 INFO:teuthology.orchestra.run.smithi092.stdout: "mon_allow_pool_size_one": "false", 2024-03-11T21:27:25.279 INFO:teuthology.orchestra.run.smithi092.stdout: "mon_auth_validate_all_caps": "true", 2024-03-11T21:27:25.279 INFO:teuthology.orchestra.run.smithi092.stdout: "mon_cache_target_full_warn_ratio": "0.660000", 2024-03-11T21:27:25.279 INFO:teuthology.orchestra.run.smithi092.stdout: "mon_clean_pg_upmaps_per_chunk": "256", 2024-03-11T21:27:25.279 INFO:teuthology.orchestra.run.smithi092.stdout: "mon_client_bytes": "104857600", 2024-03-11T21:27:25.279 INFO:teuthology.orchestra.run.smithi092.stdout: "mon_client_directed_command_retry": "2", 2024-03-11T21:27:25.279 INFO:teuthology.orchestra.run.smithi092.stdout: "mon_client_hunt_interval": "3.000000", 2024-03-11T21:27:25.279 INFO:teuthology.orchestra.run.smithi092.stdout: "mon_client_hunt_interval_backoff": "1.500000", 2024-03-11T21:27:25.279 INFO:teuthology.orchestra.run.smithi092.stdout: "mon_client_hunt_interval_max_multiple": "10.000000", 2024-03-11T21:27:25.279 INFO:teuthology.orchestra.run.smithi092.stdout: "mon_client_hunt_interval_min_multiple": "1.000000", 2024-03-11T21:27:25.279 INFO:teuthology.orchestra.run.smithi092.stdout: "mon_client_hunt_parallel": "3", 2024-03-11T21:27:25.279 INFO:teuthology.orchestra.run.smithi092.stdout: "mon_client_log_interval": "1.000000", 2024-03-11T21:27:25.279 INFO:teuthology.orchestra.run.smithi092.stdout: "mon_client_max_log_entries_per_message": "1000", 2024-03-11T21:27:25.279 INFO:teuthology.orchestra.run.smithi092.stdout: "mon_client_ping_interval": "10.000000", 2024-03-11T21:27:25.280 INFO:teuthology.orchestra.run.smithi092.stdout: "mon_client_ping_timeout": "30.000000", 2024-03-11T21:27:25.280 INFO:teuthology.orchestra.run.smithi092.stdout: "mon_client_target_rank": "-1", 2024-03-11T21:27:25.280 INFO:teuthology.orchestra.run.smithi092.stdout: "mon_clock_drift_allowed": "1.000000", 2024-03-11T21:27:25.280 INFO:teuthology.orchestra.run.smithi092.stdout: "mon_clock_drift_warn_backoff": "5.000000", 2024-03-11T21:27:25.280 INFO:teuthology.orchestra.run.smithi092.stdout: "mon_cluster_log_file": "default=/var/log/ceph/ceph.$channel.log cluster=/var/log/ceph/ceph.log", 2024-03-11T21:27:25.280 INFO:teuthology.orchestra.run.smithi092.stdout: "mon_cluster_log_file_level": "debug", 2024-03-11T21:27:25.280 INFO:teuthology.orchestra.run.smithi092.stdout: "mon_cluster_log_to_file": "true", 2024-03-11T21:27:25.280 INFO:teuthology.orchestra.run.smithi092.stdout: "mon_cluster_log_to_graylog": "false", 2024-03-11T21:27:25.280 INFO:teuthology.orchestra.run.smithi092.stdout: "mon_cluster_log_to_graylog_host": "127.0.0.1", 2024-03-11T21:27:25.280 INFO:teuthology.orchestra.run.smithi092.stdout: "mon_cluster_log_to_graylog_port": "12201", 2024-03-11T21:27:25.280 INFO:teuthology.orchestra.run.smithi092.stdout: "mon_cluster_log_to_journald": "true", 2024-03-11T21:27:25.280 INFO:teuthology.orchestra.run.smithi092.stdout: "mon_cluster_log_to_stderr": "false", 2024-03-11T21:27:25.280 INFO:teuthology.orchestra.run.smithi092.stdout: "mon_cluster_log_to_syslog": "default=false", 2024-03-11T21:27:25.280 INFO:teuthology.orchestra.run.smithi092.stdout: "mon_cluster_log_to_syslog_facility": "daemon", 2024-03-11T21:27:25.281 INFO:teuthology.orchestra.run.smithi092.stdout: "mon_cluster_log_to_syslog_level": "info", 2024-03-11T21:27:25.281 INFO:teuthology.orchestra.run.smithi092.stdout: "mon_compact_on_bootstrap": "false", 2024-03-11T21:27:25.281 INFO:teuthology.orchestra.run.smithi092.stdout: "mon_compact_on_start": "false", 2024-03-11T21:27:25.281 INFO:teuthology.orchestra.run.smithi092.stdout: "mon_compact_on_trim": "true", 2024-03-11T21:27:25.281 INFO:teuthology.orchestra.run.smithi092.stdout: "mon_con_tracker_persist_interval": "10", 2024-03-11T21:27:25.281 INFO:teuthology.orchestra.run.smithi092.stdout: "mon_con_tracker_score_halflife": "43200", 2024-03-11T21:27:25.281 INFO:teuthology.orchestra.run.smithi092.stdout: "mon_config_key_max_entry_size": "65536", 2024-03-11T21:27:25.281 INFO:teuthology.orchestra.run.smithi092.stdout: "mon_cpu_threads": "4", 2024-03-11T21:27:25.281 INFO:teuthology.orchestra.run.smithi092.stdout: "mon_crush_min_required_version": "hammer", 2024-03-11T21:27:25.281 INFO:teuthology.orchestra.run.smithi092.stdout: "mon_daemon_bytes": "419430400", 2024-03-11T21:27:25.281 INFO:teuthology.orchestra.run.smithi092.stdout: "mon_data": "/var/lib/ceph/mon/ceph-a", 2024-03-11T21:27:25.281 INFO:teuthology.orchestra.run.smithi092.stdout: "mon_data_avail_crit": "5", 2024-03-11T21:27:25.281 INFO:teuthology.orchestra.run.smithi092.stdout: "mon_data_avail_warn": "5", 2024-03-11T21:27:25.282 INFO:teuthology.orchestra.run.smithi092.stdout: "mon_data_size_warn": "16106127360", 2024-03-11T21:27:25.282 INFO:teuthology.orchestra.run.smithi092.stdout: "mon_debug_block_osdmap_trim": "false", 2024-03-11T21:27:25.282 INFO:teuthology.orchestra.run.smithi092.stdout: "mon_debug_deprecated_as_obsolete": "false", 2024-03-11T21:27:25.282 INFO:teuthology.orchestra.run.smithi092.stdout: "mon_debug_dump_json": "false", 2024-03-11T21:27:25.282 INFO:teuthology.orchestra.run.smithi092.stdout: "mon_debug_dump_location": "/var/log/ceph/ceph-mon.a.tdump", 2024-03-11T21:27:25.282 INFO:teuthology.orchestra.run.smithi092.stdout: "mon_debug_dump_transactions": "false", 2024-03-11T21:27:25.282 INFO:teuthology.orchestra.run.smithi092.stdout: "mon_debug_extra_checks": "false", 2024-03-11T21:27:25.282 INFO:teuthology.orchestra.run.smithi092.stdout: "mon_debug_no_initial_persistent_features": "false", 2024-03-11T21:27:25.282 INFO:teuthology.orchestra.run.smithi092.stdout: "mon_debug_no_require_bluestore_for_ec_overwrites": "false", 2024-03-11T21:27:25.282 INFO:teuthology.orchestra.run.smithi092.stdout: "mon_debug_no_require_reef": "false", 2024-03-11T21:27:25.282 INFO:teuthology.orchestra.run.smithi092.stdout: "mon_debug_no_require_squid": "false", 2024-03-11T21:27:25.282 INFO:teuthology.orchestra.run.smithi092.stdout: "mon_debug_unsafe_allow_tier_with_nonempty_snaps": "false", 2024-03-11T21:27:25.282 INFO:teuthology.orchestra.run.smithi092.stdout: "mon_delta_reset_interval": "10.000000", 2024-03-11T21:27:25.283 INFO:teuthology.orchestra.run.smithi092.stdout: "mon_dns_srv_name": "ceph-mon", 2024-03-11T21:27:25.283 INFO:teuthology.orchestra.run.smithi092.stdout: "mon_down_mkfs_grace": "60", 2024-03-11T21:27:25.283 INFO:teuthology.orchestra.run.smithi092.stdout: "mon_election_default_strategy": "1", 2024-03-11T21:27:25.283 INFO:teuthology.orchestra.run.smithi092.stdout: "mon_election_timeout": "5.000000", 2024-03-11T21:27:25.283 INFO:teuthology.orchestra.run.smithi092.stdout: "mon_elector_ignore_propose_margin": "0.000500", 2024-03-11T21:27:25.283 INFO:teuthology.orchestra.run.smithi092.stdout: "mon_elector_ping_divisor": "2", 2024-03-11T21:27:25.283 INFO:teuthology.orchestra.run.smithi092.stdout: "mon_elector_ping_timeout": "2.000000", 2024-03-11T21:27:25.283 INFO:teuthology.orchestra.run.smithi092.stdout: "mon_enable_op_tracker": "true", 2024-03-11T21:27:25.283 INFO:teuthology.orchestra.run.smithi092.stdout: "mon_fake_pool_delete": "false", 2024-03-11T21:27:25.283 INFO:teuthology.orchestra.run.smithi092.stdout: "mon_force_quorum_join": "false", 2024-03-11T21:27:25.283 INFO:teuthology.orchestra.run.smithi092.stdout: "mon_globalid_prealloc": "10000", 2024-03-11T21:27:25.283 INFO:teuthology.orchestra.run.smithi092.stdout: "mon_health_detail_to_clog": "true", 2024-03-11T21:27:25.283 INFO:teuthology.orchestra.run.smithi092.stdout: "mon_health_log_update_period": "5", 2024-03-11T21:27:25.284 INFO:teuthology.orchestra.run.smithi092.stdout: "mon_health_max_detail": "50", 2024-03-11T21:27:25.284 INFO:teuthology.orchestra.run.smithi092.stdout: "mon_health_to_clog": "true", 2024-03-11T21:27:25.284 INFO:teuthology.orchestra.run.smithi092.stdout: "mon_health_to_clog_interval": "600", 2024-03-11T21:27:25.284 INFO:teuthology.orchestra.run.smithi092.stdout: "mon_health_to_clog_tick_interval": "60.000000", 2024-03-11T21:27:25.284 INFO:teuthology.orchestra.run.smithi092.stdout: "mon_host": "172.21.15.92:6789/0", 2024-03-11T21:27:25.284 INFO:teuthology.orchestra.run.smithi092.stdout: "mon_host_override": "", 2024-03-11T21:27:25.284 INFO:teuthology.orchestra.run.smithi092.stdout: "mon_initial_members": "", 2024-03-11T21:27:25.284 INFO:teuthology.orchestra.run.smithi092.stdout: "mon_inject_pg_merge_bounce_probability": "0.000000", 2024-03-11T21:27:25.284 INFO:teuthology.orchestra.run.smithi092.stdout: "mon_inject_sync_get_chunk_delay": "0.000000", 2024-03-11T21:27:25.284 INFO:teuthology.orchestra.run.smithi092.stdout: "mon_inject_transaction_delay_max": "10.000000", 2024-03-11T21:27:25.284 INFO:teuthology.orchestra.run.smithi092.stdout: "mon_inject_transaction_delay_probability": "0.000000", 2024-03-11T21:27:25.284 INFO:teuthology.orchestra.run.smithi092.stdout: "mon_keyvaluedb": "rocksdb", 2024-03-11T21:27:25.284 INFO:teuthology.orchestra.run.smithi092.stdout: "mon_lease": "5.000000", 2024-03-11T21:27:25.284 INFO:teuthology.orchestra.run.smithi092.stdout: "mon_lease_ack_timeout_factor": "2.000000", 2024-03-11T21:27:25.285 INFO:teuthology.orchestra.run.smithi092.stdout: "mon_lease_renew_interval_factor": "0.600000", 2024-03-11T21:27:25.285 INFO:teuthology.orchestra.run.smithi092.stdout: "mon_log_full_interval": "50", 2024-03-11T21:27:25.285 INFO:teuthology.orchestra.run.smithi092.stdout: "mon_log_max": "10000", 2024-03-11T21:27:25.285 INFO:teuthology.orchestra.run.smithi092.stdout: "mon_log_max_summary": "50", 2024-03-11T21:27:25.285 INFO:teuthology.orchestra.run.smithi092.stdout: "mon_max_log_entries_per_event": "4096", 2024-03-11T21:27:25.285 INFO:teuthology.orchestra.run.smithi092.stdout: "mon_max_log_epochs": "500", 2024-03-11T21:27:25.285 INFO:teuthology.orchestra.run.smithi092.stdout: "mon_max_mdsmap_epochs": "500", 2024-03-11T21:27:25.285 INFO:teuthology.orchestra.run.smithi092.stdout: "mon_max_mgrmap_epochs": "500", 2024-03-11T21:27:25.285 INFO:teuthology.orchestra.run.smithi092.stdout: "mon_max_osd": "10000", 2024-03-11T21:27:25.285 INFO:teuthology.orchestra.run.smithi092.stdout: "mon_max_pg_per_osd": "10000", 2024-03-11T21:27:25.285 INFO:teuthology.orchestra.run.smithi092.stdout: "mon_max_pool_pg_num": "65536", 2024-03-11T21:27:25.285 INFO:teuthology.orchestra.run.smithi092.stdout: "mon_max_snap_prune_per_epoch": "100", 2024-03-11T21:27:25.285 INFO:teuthology.orchestra.run.smithi092.stdout: "mon_mds_blocklist_interval": "86400.000000", 2024-03-11T21:27:25.286 INFO:teuthology.orchestra.run.smithi092.stdout: "mon_mds_force_trim_to": "0", 2024-03-11T21:27:25.286 INFO:teuthology.orchestra.run.smithi092.stdout: "mon_mds_skip_sanity": "false", 2024-03-11T21:27:25.286 INFO:teuthology.orchestra.run.smithi092.stdout: "mon_memory_autotune": "true", 2024-03-11T21:27:25.286 INFO:teuthology.orchestra.run.smithi092.stdout: "mon_memory_target": "2147483648", 2024-03-11T21:27:25.286 INFO:teuthology.orchestra.run.smithi092.stdout: "mon_mgr_beacon_grace": "30", 2024-03-11T21:27:25.286 INFO:teuthology.orchestra.run.smithi092.stdout: "mon_mgr_blocklist_interval": "86400.000000", 2024-03-11T21:27:25.286 INFO:teuthology.orchestra.run.smithi092.stdout: "mon_mgr_digest_period": "5", 2024-03-11T21:27:25.286 INFO:teuthology.orchestra.run.smithi092.stdout: "mon_mgr_inactive_grace": "60", 2024-03-11T21:27:25.286 INFO:teuthology.orchestra.run.smithi092.stdout: "mon_mgr_mkfs_grace": "240", 2024-03-11T21:27:25.286 INFO:teuthology.orchestra.run.smithi092.stdout: "mon_mgr_proxy_client_bytes_ratio": "0.300000", 2024-03-11T21:27:25.286 INFO:teuthology.orchestra.run.smithi092.stdout: "mon_min_osdmap_epochs": "50", 2024-03-11T21:27:25.286 INFO:teuthology.orchestra.run.smithi092.stdout: "mon_op_complaint_time": "30", 2024-03-11T21:27:25.286 INFO:teuthology.orchestra.run.smithi092.stdout: "mon_op_history_duration": "600", 2024-03-11T21:27:25.287 INFO:teuthology.orchestra.run.smithi092.stdout: "mon_op_history_size": "20", 2024-03-11T21:27:25.287 INFO:teuthology.orchestra.run.smithi092.stdout: "mon_op_history_slow_op_size": "20", 2024-03-11T21:27:25.287 INFO:teuthology.orchestra.run.smithi092.stdout: "mon_op_history_slow_op_threshold": "10", 2024-03-11T21:27:25.287 INFO:teuthology.orchestra.run.smithi092.stdout: "mon_op_log_threshold": "5", 2024-03-11T21:27:25.287 INFO:teuthology.orchestra.run.smithi092.stdout: "mon_osd_adjust_down_out_interval": "true", 2024-03-11T21:27:25.287 INFO:teuthology.orchestra.run.smithi092.stdout: "mon_osd_adjust_heartbeat_grace": "true", 2024-03-11T21:27:25.287 INFO:teuthology.orchestra.run.smithi092.stdout: "mon_osd_auto_mark_auto_out_in": "true", 2024-03-11T21:27:25.287 INFO:teuthology.orchestra.run.smithi092.stdout: "mon_osd_auto_mark_in": "false", 2024-03-11T21:27:25.287 INFO:teuthology.orchestra.run.smithi092.stdout: "mon_osd_auto_mark_new_in": "true", 2024-03-11T21:27:25.287 INFO:teuthology.orchestra.run.smithi092.stdout: "mon_osd_backfillfull_ratio": "0.900000", 2024-03-11T21:27:25.287 INFO:teuthology.orchestra.run.smithi092.stdout: "mon_osd_blocklist_default_expire": "3600.000000", 2024-03-11T21:27:25.287 INFO:teuthology.orchestra.run.smithi092.stdout: "mon_osd_cache_size": "500", 2024-03-11T21:27:25.287 INFO:teuthology.orchestra.run.smithi092.stdout: "mon_osd_cache_size_min": "134217728", 2024-03-11T21:27:25.287 INFO:teuthology.orchestra.run.smithi092.stdout: "mon_osd_crush_smoke_test": "true", 2024-03-11T21:27:25.287 INFO:teuthology.orchestra.run.smithi092.stdout: "mon_osd_destroyed_out_interval": "600", 2024-03-11T21:27:25.288 INFO:teuthology.orchestra.run.smithi092.stdout: "mon_osd_down_out_interval": "600", 2024-03-11T21:27:25.288 INFO:teuthology.orchestra.run.smithi092.stdout: "mon_osd_down_out_subtree_limit": "rack", 2024-03-11T21:27:25.288 INFO:teuthology.orchestra.run.smithi092.stdout: "mon_osd_err_op_age_ratio": "128.000000", 2024-03-11T21:27:25.288 INFO:teuthology.orchestra.run.smithi092.stdout: "mon_osd_force_trim_to": "0", 2024-03-11T21:27:25.288 INFO:teuthology.orchestra.run.smithi092.stdout: "mon_osd_full_ratio": "0.950000", 2024-03-11T21:27:25.288 INFO:teuthology.orchestra.run.smithi092.stdout: "mon_osd_initial_require_min_compat_client": "luminous", 2024-03-11T21:27:25.288 INFO:teuthology.orchestra.run.smithi092.stdout: "mon_osd_laggy_halflife": "3600", 2024-03-11T21:27:25.288 INFO:teuthology.orchestra.run.smithi092.stdout: "mon_osd_laggy_max_interval": "300", 2024-03-11T21:27:25.288 INFO:teuthology.orchestra.run.smithi092.stdout: "mon_osd_laggy_weight": "0.300000", 2024-03-11T21:27:25.288 INFO:teuthology.orchestra.run.smithi092.stdout: "mon_osd_mapping_pgs_per_chunk": "4096", 2024-03-11T21:27:25.288 INFO:teuthology.orchestra.run.smithi092.stdout: "mon_osd_max_creating_pgs": "1024", 2024-03-11T21:27:25.288 INFO:teuthology.orchestra.run.smithi092.stdout: "mon_osd_max_initial_pgs": "1024", 2024-03-11T21:27:25.288 INFO:teuthology.orchestra.run.smithi092.stdout: "mon_osd_min_down_reporters": "2", 2024-03-11T21:27:25.288 INFO:teuthology.orchestra.run.smithi092.stdout: "mon_osd_min_in_ratio": "0.750000", 2024-03-11T21:27:25.289 INFO:teuthology.orchestra.run.smithi092.stdout: "mon_osd_min_up_ratio": "0.300000", 2024-03-11T21:27:25.289 INFO:teuthology.orchestra.run.smithi092.stdout: "mon_osd_nearfull_ratio": "0.850000", 2024-03-11T21:27:25.289 INFO:teuthology.orchestra.run.smithi092.stdout: "mon_osd_prime_pg_temp": "true", 2024-03-11T21:27:25.289 INFO:teuthology.orchestra.run.smithi092.stdout: "mon_osd_prime_pg_temp_max_estimate": "0.250000", 2024-03-11T21:27:25.289 INFO:teuthology.orchestra.run.smithi092.stdout: "mon_osd_prime_pg_temp_max_time": "0.500000", 2024-03-11T21:27:25.289 INFO:teuthology.orchestra.run.smithi092.stdout: "mon_osd_report_timeout": "900", 2024-03-11T21:27:25.289 INFO:teuthology.orchestra.run.smithi092.stdout: "mon_osd_reporter_subtree_level": "osd", 2024-03-11T21:27:25.289 INFO:teuthology.orchestra.run.smithi092.stdout: "mon_osd_snap_trim_queue_warn_on": "32768", 2024-03-11T21:27:25.289 INFO:teuthology.orchestra.run.smithi092.stdout: "mon_osd_warn_num_repaired": "10", 2024-03-11T21:27:25.289 INFO:teuthology.orchestra.run.smithi092.stdout: "mon_osd_warn_op_age": "32.000000", 2024-03-11T21:27:25.289 INFO:teuthology.orchestra.run.smithi092.stdout: "mon_osdmap_full_prune_enabled": "true", 2024-03-11T21:27:25.289 INFO:teuthology.orchestra.run.smithi092.stdout: "mon_osdmap_full_prune_interval": "2", 2024-03-11T21:27:25.289 INFO:teuthology.orchestra.run.smithi092.stdout: "mon_osdmap_full_prune_min": "15", 2024-03-11T21:27:25.289 INFO:teuthology.orchestra.run.smithi092.stdout: "mon_osdmap_full_prune_txsize": "2", 2024-03-11T21:27:25.290 INFO:teuthology.orchestra.run.smithi092.stdout: "mon_pg_check_down_all_threshold": "0.500000", 2024-03-11T21:27:25.290 INFO:teuthology.orchestra.run.smithi092.stdout: "mon_pg_stuck_threshold": "60", 2024-03-11T21:27:25.290 INFO:teuthology.orchestra.run.smithi092.stdout: "mon_pg_warn_max_object_skew": "0.000000", 2024-03-11T21:27:25.290 INFO:teuthology.orchestra.run.smithi092.stdout: "mon_pg_warn_min_objects": "10000", 2024-03-11T21:27:25.290 INFO:teuthology.orchestra.run.smithi092.stdout: "mon_pg_warn_min_per_osd": "0", 2024-03-11T21:27:25.290 INFO:teuthology.orchestra.run.smithi092.stdout: "mon_pg_warn_min_pool_objects": "1000", 2024-03-11T21:27:25.290 INFO:teuthology.orchestra.run.smithi092.stdout: "mon_pool_quota_crit_threshold": "0", 2024-03-11T21:27:25.290 INFO:teuthology.orchestra.run.smithi092.stdout: "mon_pool_quota_warn_threshold": "0", 2024-03-11T21:27:25.290 INFO:teuthology.orchestra.run.smithi092.stdout: "mon_probe_timeout": "2.000000", 2024-03-11T21:27:25.290 INFO:teuthology.orchestra.run.smithi092.stdout: "mon_reweight_max_change": "0.050000", 2024-03-11T21:27:25.290 INFO:teuthology.orchestra.run.smithi092.stdout: "mon_reweight_max_osds": "4", 2024-03-11T21:27:25.290 INFO:teuthology.orchestra.run.smithi092.stdout: "mon_reweight_min_bytes_per_osd": "10", 2024-03-11T21:27:25.290 INFO:teuthology.orchestra.run.smithi092.stdout: "mon_reweight_min_pgs_per_osd": "4", 2024-03-11T21:27:25.290 INFO:teuthology.orchestra.run.smithi092.stdout: "mon_rocksdb_options": "write_buffer_size=33554432,compression=kNoCompression,level_compaction_dynamic_level_bytes=true", 2024-03-11T21:27:25.291 INFO:teuthology.orchestra.run.smithi092.stdout: "mon_scrub_inject_crc_mismatch": "0.000000", 2024-03-11T21:27:25.291 INFO:teuthology.orchestra.run.smithi092.stdout: "mon_scrub_inject_missing_keys": "0.000000", 2024-03-11T21:27:25.291 INFO:teuthology.orchestra.run.smithi092.stdout: "mon_scrub_interval": "86400", 2024-03-11T21:27:25.291 INFO:teuthology.orchestra.run.smithi092.stdout: "mon_scrub_max_keys": "100", 2024-03-11T21:27:25.291 INFO:teuthology.orchestra.run.smithi092.stdout: "mon_scrub_timeout": "300", 2024-03-11T21:27:25.291 INFO:teuthology.orchestra.run.smithi092.stdout: "mon_session_timeout": "300", 2024-03-11T21:27:25.291 INFO:teuthology.orchestra.run.smithi092.stdout: "mon_smart_report_timeout": "5", 2024-03-11T21:27:25.291 INFO:teuthology.orchestra.run.smithi092.stdout: "mon_stat_smooth_intervals": "6", 2024-03-11T21:27:25.291 INFO:teuthology.orchestra.run.smithi092.stdout: "mon_stretch_cluster_recovery_ratio": "0.600000", 2024-03-11T21:27:25.291 INFO:teuthology.orchestra.run.smithi092.stdout: "mon_stretch_pool_min_size": "2", 2024-03-11T21:27:25.291 INFO:teuthology.orchestra.run.smithi092.stdout: "mon_stretch_pool_size": "4", 2024-03-11T21:27:25.291 INFO:teuthology.orchestra.run.smithi092.stdout: "mon_stretch_recovery_min_wait": "15.000000", 2024-03-11T21:27:25.291 INFO:teuthology.orchestra.run.smithi092.stdout: "mon_subscribe_interval": "86400.000000", 2024-03-11T21:27:25.291 INFO:teuthology.orchestra.run.smithi092.stdout: "mon_sync_debug": "false", 2024-03-11T21:27:25.291 INFO:teuthology.orchestra.run.smithi092.stdout: "mon_sync_max_payload_keys": "2000", 2024-03-11T21:27:25.292 INFO:teuthology.orchestra.run.smithi092.stdout: "mon_sync_max_payload_size": "1048576", 2024-03-11T21:27:25.292 INFO:teuthology.orchestra.run.smithi092.stdout: "mon_sync_provider_kill_at": "0", 2024-03-11T21:27:25.292 INFO:teuthology.orchestra.run.smithi092.stdout: "mon_sync_requester_kill_at": "0", 2024-03-11T21:27:25.292 INFO:teuthology.orchestra.run.smithi092.stdout: "mon_sync_timeout": "60.000000", 2024-03-11T21:27:25.292 INFO:teuthology.orchestra.run.smithi092.stdout: "mon_target_pg_per_osd": "100", 2024-03-11T21:27:25.292 INFO:teuthology.orchestra.run.smithi092.stdout: "mon_tick_interval": "5", 2024-03-11T21:27:25.292 INFO:teuthology.orchestra.run.smithi092.stdout: "mon_timecheck_interval": "300.000000", 2024-03-11T21:27:25.292 INFO:teuthology.orchestra.run.smithi092.stdout: "mon_timecheck_skew_interval": "30.000000", 2024-03-11T21:27:25.292 INFO:teuthology.orchestra.run.smithi092.stdout: "mon_use_min_delay_socket": "false", 2024-03-11T21:27:25.292 INFO:teuthology.orchestra.run.smithi092.stdout: "mon_warn_older_version_delay": "604800", 2024-03-11T21:27:25.292 INFO:teuthology.orchestra.run.smithi092.stdout: "mon_warn_on_cache_pools_without_hit_sets": "true", 2024-03-11T21:27:25.292 INFO:teuthology.orchestra.run.smithi092.stdout: "mon_warn_on_crush_straw_calc_version_zero": "false", 2024-03-11T21:27:25.292 INFO:teuthology.orchestra.run.smithi092.stdout: "mon_warn_on_degraded_stretch_mode": "true", 2024-03-11T21:27:25.292 INFO:teuthology.orchestra.run.smithi092.stdout: "mon_warn_on_filestore_osds": "true", 2024-03-11T21:27:25.292 INFO:teuthology.orchestra.run.smithi092.stdout: "mon_warn_on_insecure_global_id_reclaim": "false", 2024-03-11T21:27:25.293 INFO:teuthology.orchestra.run.smithi092.stdout: "mon_warn_on_insecure_global_id_reclaim_allowed": "false", 2024-03-11T21:27:25.293 INFO:teuthology.orchestra.run.smithi092.stdout: "mon_warn_on_legacy_crush_tunables": "false", 2024-03-11T21:27:25.293 INFO:teuthology.orchestra.run.smithi092.stdout: "mon_warn_on_misplaced": "false", 2024-03-11T21:27:25.293 INFO:teuthology.orchestra.run.smithi092.stdout: "mon_warn_on_msgr2_not_enabled": "true", 2024-03-11T21:27:25.293 INFO:teuthology.orchestra.run.smithi092.stdout: "mon_warn_on_older_version": "true", 2024-03-11T21:27:25.293 INFO:teuthology.orchestra.run.smithi092.stdout: "mon_warn_on_osd_down_out_interval_zero": "false", 2024-03-11T21:27:25.293 INFO:teuthology.orchestra.run.smithi092.stdout: "mon_warn_on_pool_no_app": "true", 2024-03-11T21:27:25.293 INFO:teuthology.orchestra.run.smithi092.stdout: "mon_warn_on_pool_no_redundancy": "true", 2024-03-11T21:27:25.293 INFO:teuthology.orchestra.run.smithi092.stdout: "mon_warn_on_pool_pg_num_not_power_of_two": "false", 2024-03-11T21:27:25.293 INFO:teuthology.orchestra.run.smithi092.stdout: "mon_warn_on_slow_ping_ratio": "0.050000", 2024-03-11T21:27:25.294 INFO:teuthology.orchestra.run.smithi092.stdout: "mon_warn_on_slow_ping_time": "0.000000", 2024-03-11T21:27:25.294 INFO:teuthology.orchestra.run.smithi092.stdout: "mon_warn_on_too_few_osds": "false", 2024-03-11T21:27:25.294 INFO:teuthology.orchestra.run.smithi092.stdout: "mon_warn_pg_not_deep_scrubbed_ratio": "0.750000", 2024-03-11T21:27:25.294 INFO:teuthology.orchestra.run.smithi092.stdout: "mon_warn_pg_not_scrubbed_ratio": "0.500000", 2024-03-11T21:27:25.294 INFO:teuthology.orchestra.run.smithi092.stdout: "monmap": "", 2024-03-11T21:27:25.294 INFO:teuthology.orchestra.run.smithi092.stdout: "motr_admin_endpoint": "192.168.180.182@tcp:12345:4:1", 2024-03-11T21:27:25.294 INFO:teuthology.orchestra.run.smithi092.stdout: "motr_admin_fid": "0x7200000000000001:0x0", 2024-03-11T21:27:25.294 INFO:teuthology.orchestra.run.smithi092.stdout: "motr_ha_endpoint": "192.168.180.182@tcp:12345:1:1", 2024-03-11T21:27:25.294 INFO:teuthology.orchestra.run.smithi092.stdout: "motr_my_endpoint": "192.168.180.182@tcp:12345:4:1", 2024-03-11T21:27:25.294 INFO:teuthology.orchestra.run.smithi092.stdout: "motr_my_fid": "0x7200000000000001:0x0", 2024-03-11T21:27:25.294 INFO:teuthology.orchestra.run.smithi092.stdout: "motr_profile_fid": "0x7000000000000001:0x0", 2024-03-11T21:27:25.294 INFO:teuthology.orchestra.run.smithi092.stdout: "motr_tracing_enabled": "false", 2024-03-11T21:27:25.294 INFO:teuthology.orchestra.run.smithi092.stdout: "ms_async_op_threads": "3", 2024-03-11T21:27:25.295 INFO:teuthology.orchestra.run.smithi092.stdout: "ms_async_rdma_buffer_size": "131072", 2024-03-11T21:27:25.295 INFO:teuthology.orchestra.run.smithi092.stdout: "ms_async_rdma_cm": "false", 2024-03-11T21:27:25.295 INFO:teuthology.orchestra.run.smithi092.stdout: "ms_async_rdma_device_name": "", 2024-03-11T21:27:25.295 INFO:teuthology.orchestra.run.smithi092.stdout: "ms_async_rdma_dscp": "96", 2024-03-11T21:27:25.295 INFO:teuthology.orchestra.run.smithi092.stdout: "ms_async_rdma_enable_hugepage": "false", 2024-03-11T21:27:25.295 INFO:teuthology.orchestra.run.smithi092.stdout: "ms_async_rdma_gid_idx": "0", 2024-03-11T21:27:25.295 INFO:teuthology.orchestra.run.smithi092.stdout: "ms_async_rdma_local_gid": "", 2024-03-11T21:27:25.295 INFO:teuthology.orchestra.run.smithi092.stdout: "ms_async_rdma_polling_us": "1000", 2024-03-11T21:27:25.295 INFO:teuthology.orchestra.run.smithi092.stdout: "ms_async_rdma_port_num": "1", 2024-03-11T21:27:25.295 INFO:teuthology.orchestra.run.smithi092.stdout: "ms_async_rdma_receive_buffers": "32768", 2024-03-11T21:27:25.295 INFO:teuthology.orchestra.run.smithi092.stdout: "ms_async_rdma_receive_queue_len": "4096", 2024-03-11T21:27:25.295 INFO:teuthology.orchestra.run.smithi092.stdout: "ms_async_rdma_roce_ver": "1", 2024-03-11T21:27:25.295 INFO:teuthology.orchestra.run.smithi092.stdout: "ms_async_rdma_send_buffers": "1024", 2024-03-11T21:27:25.295 INFO:teuthology.orchestra.run.smithi092.stdout: "ms_async_rdma_sl": "3", 2024-03-11T21:27:25.296 INFO:teuthology.orchestra.run.smithi092.stdout: "ms_async_rdma_support_srq": "true", 2024-03-11T21:27:25.296 INFO:teuthology.orchestra.run.smithi092.stdout: "ms_async_rdma_type": "ib", 2024-03-11T21:27:25.296 INFO:teuthology.orchestra.run.smithi092.stdout: "ms_async_reap_threshold": "5", 2024-03-11T21:27:25.296 INFO:teuthology.orchestra.run.smithi092.stdout: "ms_bind_before_connect": "false", 2024-03-11T21:27:25.296 INFO:teuthology.orchestra.run.smithi092.stdout: "ms_bind_ipv4": "true", 2024-03-11T21:27:25.296 INFO:teuthology.orchestra.run.smithi092.stdout: "ms_bind_ipv6": "false", 2024-03-11T21:27:25.296 INFO:teuthology.orchestra.run.smithi092.stdout: "ms_bind_msgr1": "true", 2024-03-11T21:27:25.296 INFO:teuthology.orchestra.run.smithi092.stdout: "ms_bind_msgr2": "false", 2024-03-11T21:27:25.296 INFO:teuthology.orchestra.run.smithi092.stdout: "ms_bind_port_max": "7568", 2024-03-11T21:27:25.296 INFO:teuthology.orchestra.run.smithi092.stdout: "ms_bind_port_min": "6800", 2024-03-11T21:27:25.296 INFO:teuthology.orchestra.run.smithi092.stdout: "ms_bind_prefer_ipv4": "false", 2024-03-11T21:27:25.296 INFO:teuthology.orchestra.run.smithi092.stdout: "ms_bind_retry_count": "3", 2024-03-11T21:27:25.296 INFO:teuthology.orchestra.run.smithi092.stdout: "ms_bind_retry_delay": "5", 2024-03-11T21:27:25.297 INFO:teuthology.orchestra.run.smithi092.stdout: "ms_blackhole_client": "false", 2024-03-11T21:27:25.297 INFO:teuthology.orchestra.run.smithi092.stdout: "ms_blackhole_mds": "false", 2024-03-11T21:27:25.297 INFO:teuthology.orchestra.run.smithi092.stdout: "ms_blackhole_mgr": "false", 2024-03-11T21:27:25.297 INFO:teuthology.orchestra.run.smithi092.stdout: "ms_blackhole_mon": "false", 2024-03-11T21:27:25.297 INFO:teuthology.orchestra.run.smithi092.stdout: "ms_blackhole_osd": "false", 2024-03-11T21:27:25.297 INFO:teuthology.orchestra.run.smithi092.stdout: "ms_client_mode": "crc secure", 2024-03-11T21:27:25.297 INFO:teuthology.orchestra.run.smithi092.stdout: "ms_cluster_mode": "crc secure", 2024-03-11T21:27:25.297 INFO:teuthology.orchestra.run.smithi092.stdout: "ms_cluster_type": "", 2024-03-11T21:27:25.297 INFO:teuthology.orchestra.run.smithi092.stdout: "ms_compress_secure": "false", 2024-03-11T21:27:25.297 INFO:teuthology.orchestra.run.smithi092.stdout: "ms_connection_idle_timeout": "900", 2024-03-11T21:27:25.297 INFO:teuthology.orchestra.run.smithi092.stdout: "ms_connection_ready_timeout": "10", 2024-03-11T21:27:25.297 INFO:teuthology.orchestra.run.smithi092.stdout: "ms_crc_data": "true", 2024-03-11T21:27:25.297 INFO:teuthology.orchestra.run.smithi092.stdout: "ms_crc_header": "true", 2024-03-11T21:27:25.297 INFO:teuthology.orchestra.run.smithi092.stdout: "ms_die_on_bad_msg": "false", 2024-03-11T21:27:25.297 INFO:teuthology.orchestra.run.smithi092.stdout: "ms_die_on_bug": "true", 2024-03-11T21:27:25.298 INFO:teuthology.orchestra.run.smithi092.stdout: "ms_die_on_old_message": "true", 2024-03-11T21:27:25.298 INFO:teuthology.orchestra.run.smithi092.stdout: "ms_die_on_skipped_message": "false", 2024-03-11T21:27:25.298 INFO:teuthology.orchestra.run.smithi092.stdout: "ms_die_on_unhandled_msg": "false", 2024-03-11T21:27:25.298 INFO:teuthology.orchestra.run.smithi092.stdout: "ms_dispatch_throttle_bytes": "104857600", 2024-03-11T21:27:25.298 INFO:teuthology.orchestra.run.smithi092.stdout: "ms_dpdk_coremask": "0xF", 2024-03-11T21:27:25.298 INFO:teuthology.orchestra.run.smithi092.stdout: "ms_dpdk_debug_allow_loopback": "false", 2024-03-11T21:27:25.298 INFO:teuthology.orchestra.run.smithi092.stdout: "ms_dpdk_devs_allowlist": "", 2024-03-11T21:27:25.298 INFO:teuthology.orchestra.run.smithi092.stdout: "ms_dpdk_enable_tso": "true", 2024-03-11T21:27:25.298 INFO:teuthology.orchestra.run.smithi092.stdout: "ms_dpdk_gateway_ipv4_addr": "", 2024-03-11T21:27:25.298 INFO:teuthology.orchestra.run.smithi092.stdout: "ms_dpdk_host_ipv4_addr": "", 2024-03-11T21:27:25.298 INFO:teuthology.orchestra.run.smithi092.stdout: "ms_dpdk_hugepages": "", 2024-03-11T21:27:25.298 INFO:teuthology.orchestra.run.smithi092.stdout: "ms_dpdk_hw_flow_control": "true", 2024-03-11T21:27:25.298 INFO:teuthology.orchestra.run.smithi092.stdout: "ms_dpdk_hw_queue_weight": "1.000000", 2024-03-11T21:27:25.299 INFO:teuthology.orchestra.run.smithi092.stdout: "ms_dpdk_lro": "true", 2024-03-11T21:27:25.299 INFO:teuthology.orchestra.run.smithi092.stdout: "ms_dpdk_memory_channel": "4", 2024-03-11T21:27:25.299 INFO:teuthology.orchestra.run.smithi092.stdout: "ms_dpdk_netmask_ipv4_addr": "", 2024-03-11T21:27:25.299 INFO:teuthology.orchestra.run.smithi092.stdout: "ms_dpdk_pmd": "", 2024-03-11T21:27:25.299 INFO:teuthology.orchestra.run.smithi092.stdout: "ms_dpdk_port_id": "0", 2024-03-11T21:27:25.299 INFO:teuthology.orchestra.run.smithi092.stdout: "ms_dpdk_rx_buffer_count_per_core": "8192", 2024-03-11T21:27:25.299 INFO:teuthology.orchestra.run.smithi092.stdout: "ms_dump_corrupt_message_level": "1", 2024-03-11T21:27:25.299 INFO:teuthology.orchestra.run.smithi092.stdout: "ms_dump_on_send": "false", 2024-03-11T21:27:25.299 INFO:teuthology.orchestra.run.smithi092.stdout: "ms_initial_backoff": "0.200000", 2024-03-11T21:27:25.299 INFO:teuthology.orchestra.run.smithi092.stdout: "ms_inject_delay_max": "1.000000", 2024-03-11T21:27:25.299 INFO:teuthology.orchestra.run.smithi092.stdout: "ms_inject_delay_probability": "0.000000", 2024-03-11T21:27:25.299 INFO:teuthology.orchestra.run.smithi092.stdout: "ms_inject_delay_type": "", 2024-03-11T21:27:25.299 INFO:teuthology.orchestra.run.smithi092.stdout: "ms_inject_internal_delays": "0.000000", 2024-03-11T21:27:25.299 INFO:teuthology.orchestra.run.smithi092.stdout: "ms_inject_network_congestion": "0", 2024-03-11T21:27:25.299 INFO:teuthology.orchestra.run.smithi092.stdout: "ms_inject_socket_failures": "0", 2024-03-11T21:27:25.300 INFO:teuthology.orchestra.run.smithi092.stdout: "ms_learn_addr_from_peer": "true", 2024-03-11T21:27:25.300 INFO:teuthology.orchestra.run.smithi092.stdout: "ms_max_accept_failures": "4", 2024-03-11T21:27:25.300 INFO:teuthology.orchestra.run.smithi092.stdout: "ms_max_backoff": "15.000000", 2024-03-11T21:27:25.300 INFO:teuthology.orchestra.run.smithi092.stdout: "ms_mon_client_mode": "secure crc", 2024-03-11T21:27:25.300 INFO:teuthology.orchestra.run.smithi092.stdout: "ms_mon_cluster_mode": "secure crc", 2024-03-11T21:27:25.300 INFO:teuthology.orchestra.run.smithi092.stdout: "ms_mon_service_mode": "secure crc", 2024-03-11T21:27:25.300 INFO:teuthology.orchestra.run.smithi092.stdout: "ms_osd_compress_min_size": "1024", 2024-03-11T21:27:25.300 INFO:teuthology.orchestra.run.smithi092.stdout: "ms_osd_compress_mode": "none", 2024-03-11T21:27:25.300 INFO:teuthology.orchestra.run.smithi092.stdout: "ms_osd_compression_algorithm": "snappy", 2024-03-11T21:27:25.300 INFO:teuthology.orchestra.run.smithi092.stdout: "ms_pq_max_tokens_per_priority": "16777216", 2024-03-11T21:27:25.300 INFO:teuthology.orchestra.run.smithi092.stdout: "ms_pq_min_cost": "65536", 2024-03-11T21:27:25.300 INFO:teuthology.orchestra.run.smithi092.stdout: "ms_public_type": "", 2024-03-11T21:27:25.300 INFO:teuthology.orchestra.run.smithi092.stdout: "ms_service_mode": "crc secure", 2024-03-11T21:27:25.300 INFO:teuthology.orchestra.run.smithi092.stdout: "ms_tcp_listen_backlog": "512", 2024-03-11T21:27:25.300 INFO:teuthology.orchestra.run.smithi092.stdout: "ms_tcp_nodelay": "true", 2024-03-11T21:27:25.301 INFO:teuthology.orchestra.run.smithi092.stdout: "ms_tcp_prefetch_max_size": "4096", 2024-03-11T21:27:25.301 INFO:teuthology.orchestra.run.smithi092.stdout: "ms_tcp_rcvbuf": "0", 2024-03-11T21:27:25.301 INFO:teuthology.orchestra.run.smithi092.stdout: "ms_type": "async+posix", 2024-03-11T21:27:25.301 INFO:teuthology.orchestra.run.smithi092.stdout: "no_config_file": "false", 2024-03-11T21:27:25.301 INFO:teuthology.orchestra.run.smithi092.stdout: "objecter_completion_locks_per_session": "32", 2024-03-11T21:27:25.301 INFO:teuthology.orchestra.run.smithi092.stdout: "objecter_debug_inject_relock_delay": "false", 2024-03-11T21:27:25.301 INFO:teuthology.orchestra.run.smithi092.stdout: "objecter_inflight_op_bytes": "104857600", 2024-03-11T21:27:25.301 INFO:teuthology.orchestra.run.smithi092.stdout: "objecter_inflight_ops": "1024", 2024-03-11T21:27:25.301 INFO:teuthology.orchestra.run.smithi092.stdout: "objecter_inject_no_watch_ping": "false", 2024-03-11T21:27:25.301 INFO:teuthology.orchestra.run.smithi092.stdout: "objecter_retry_writes_after_first_reply": "false", 2024-03-11T21:27:25.301 INFO:teuthology.orchestra.run.smithi092.stdout: "objecter_tick_interval": "5.000000", 2024-03-11T21:27:25.301 INFO:teuthology.orchestra.run.smithi092.stdout: "objecter_timeout": "10.000000", 2024-03-11T21:27:25.301 INFO:teuthology.orchestra.run.smithi092.stdout: "objectstore_blackhole": "false", 2024-03-11T21:27:25.301 INFO:teuthology.orchestra.run.smithi092.stdout: "objectstore_debug_throw_on_failed_txc": "false", 2024-03-11T21:27:25.301 INFO:teuthology.orchestra.run.smithi092.stdout: "openssl_engine_opts": "", 2024-03-11T21:27:25.302 INFO:teuthology.orchestra.run.smithi092.stdout: "osd_agent_delay_time": "5.000000", 2024-03-11T21:27:25.302 INFO:teuthology.orchestra.run.smithi092.stdout: "osd_agent_hist_halflife": "1000", 2024-03-11T21:27:25.302 INFO:teuthology.orchestra.run.smithi092.stdout: "osd_agent_max_low_ops": "2", 2024-03-11T21:27:25.302 INFO:teuthology.orchestra.run.smithi092.stdout: "osd_agent_max_ops": "4", 2024-03-11T21:27:25.302 INFO:teuthology.orchestra.run.smithi092.stdout: "osd_agent_min_evict_effort": "0.100000", 2024-03-11T21:27:25.302 INFO:teuthology.orchestra.run.smithi092.stdout: "osd_agent_quantize_effort": "0.100000", 2024-03-11T21:27:25.302 INFO:teuthology.orchestra.run.smithi092.stdout: "osd_agent_slop": "0.020000", 2024-03-11T21:27:25.302 INFO:teuthology.orchestra.run.smithi092.stdout: "osd_aggregated_slow_ops_logging": "true", 2024-03-11T21:27:25.302 INFO:teuthology.orchestra.run.smithi092.stdout: "osd_allow_recovery_below_min_size": "true", 2024-03-11T21:27:25.302 INFO:teuthology.orchestra.run.smithi092.stdout: "osd_asio_thread_count": "2", 2024-03-11T21:27:25.302 INFO:teuthology.orchestra.run.smithi092.stdout: "osd_async_recovery_min_cost": "100", 2024-03-11T21:27:25.302 INFO:teuthology.orchestra.run.smithi092.stdout: "osd_auto_mark_unfound_lost": "false", 2024-03-11T21:27:25.302 INFO:teuthology.orchestra.run.smithi092.stdout: "osd_backfill_retry_interval": "30.000000", 2024-03-11T21:27:25.303 INFO:teuthology.orchestra.run.smithi092.stdout: "osd_backfill_scan_max": "512", 2024-03-11T21:27:25.303 INFO:teuthology.orchestra.run.smithi092.stdout: "osd_backfill_scan_min": "64", 2024-03-11T21:27:25.303 INFO:teuthology.orchestra.run.smithi092.stdout: "osd_backoff_on_degraded": "false", 2024-03-11T21:27:25.303 INFO:teuthology.orchestra.run.smithi092.stdout: "osd_backoff_on_peering": "false", 2024-03-11T21:27:25.303 INFO:teuthology.orchestra.run.smithi092.stdout: "osd_backoff_on_unfound": "true", 2024-03-11T21:27:25.303 INFO:teuthology.orchestra.run.smithi092.stdout: "osd_beacon_report_interval": "300", 2024-03-11T21:27:25.303 INFO:teuthology.orchestra.run.smithi092.stdout: "osd_bench_duration": "30", 2024-03-11T21:27:25.303 INFO:teuthology.orchestra.run.smithi092.stdout: "osd_bench_large_size_max_throughput": "104857600", 2024-03-11T21:27:25.303 INFO:teuthology.orchestra.run.smithi092.stdout: "osd_bench_max_block_size": "67108864", 2024-03-11T21:27:25.303 INFO:teuthology.orchestra.run.smithi092.stdout: "osd_bench_small_size_max_iops": "100", 2024-03-11T21:27:25.303 INFO:teuthology.orchestra.run.smithi092.stdout: "osd_blkin_trace_all": "false", 2024-03-11T21:27:25.303 INFO:teuthology.orchestra.run.smithi092.stdout: "osd_blocked_scrub_grace_period": "120", 2024-03-11T21:27:25.303 INFO:teuthology.orchestra.run.smithi092.stdout: "osd_calc_pg_upmaps_aggressively": "true", 2024-03-11T21:27:25.304 INFO:teuthology.orchestra.run.smithi092.stdout: "osd_calc_pg_upmaps_aggressively_fast": "true", 2024-03-11T21:27:25.304 INFO:teuthology.orchestra.run.smithi092.stdout: "osd_calc_pg_upmaps_local_fallback_retries": "100", 2024-03-11T21:27:25.304 INFO:teuthology.orchestra.run.smithi092.stdout: "osd_check_for_log_corruption": "false", 2024-03-11T21:27:25.304 INFO:teuthology.orchestra.run.smithi092.stdout: "osd_check_max_object_name_len_on_startup": "true", 2024-03-11T21:27:25.304 INFO:teuthology.orchestra.run.smithi092.stdout: "osd_class_default_list": "cephfs hello journal lock log numops otp rbd refcount rgw rgw_gc timeindex user version cas cmpomap queue 2pc_queue fifo", 2024-03-11T21:27:25.304 INFO:teuthology.orchestra.run.smithi092.stdout: "osd_class_dir": "/usr/lib64/rados-classes", 2024-03-11T21:27:25.304 INFO:teuthology.orchestra.run.smithi092.stdout: "osd_class_load_list": "cephfs hello journal lock log numops otp rbd refcount rgw rgw_gc timeindex user version cas cmpomap queue 2pc_queue fifo", 2024-03-11T21:27:25.304 INFO:teuthology.orchestra.run.smithi092.stdout: "osd_class_update_on_start": "true", 2024-03-11T21:27:25.304 INFO:teuthology.orchestra.run.smithi092.stdout: "osd_client_message_cap": "256", 2024-03-11T21:27:25.304 INFO:teuthology.orchestra.run.smithi092.stdout: "osd_client_message_size_cap": "524288000", 2024-03-11T21:27:25.304 INFO:teuthology.orchestra.run.smithi092.stdout: "osd_client_op_priority": "63", 2024-03-11T21:27:25.304 INFO:teuthology.orchestra.run.smithi092.stdout: "osd_client_watch_timeout": "30", 2024-03-11T21:27:25.304 INFO:teuthology.orchestra.run.smithi092.stdout: "osd_command_max_records": "256", 2024-03-11T21:27:25.305 INFO:teuthology.orchestra.run.smithi092.stdout: "osd_compact_on_start": "false", 2024-03-11T21:27:25.305 INFO:teuthology.orchestra.run.smithi092.stdout: "osd_copyfrom_max_chunk": "8388608", 2024-03-11T21:27:25.305 INFO:teuthology.orchestra.run.smithi092.stdout: "osd_crush_chooseleaf_type": "1", 2024-03-11T21:27:25.305 INFO:teuthology.orchestra.run.smithi092.stdout: "osd_crush_initial_weight": "-1.000000", 2024-03-11T21:27:25.305 INFO:teuthology.orchestra.run.smithi092.stdout: "osd_crush_update_on_start": "true", 2024-03-11T21:27:25.305 INFO:teuthology.orchestra.run.smithi092.stdout: "osd_crush_update_weight_set": "true", 2024-03-11T21:27:25.305 INFO:teuthology.orchestra.run.smithi092.stdout: "osd_data": "/var/lib/ceph/osd/ceph-a", 2024-03-11T21:27:25.305 INFO:teuthology.orchestra.run.smithi092.stdout: "osd_debug_crash_on_ignored_backoff": "false", 2024-03-11T21:27:25.305 INFO:teuthology.orchestra.run.smithi092.stdout: "osd_debug_deep_scrub_sleep": "0.000000", 2024-03-11T21:27:25.305 INFO:teuthology.orchestra.run.smithi092.stdout: "osd_debug_drop_ping_duration": "0", 2024-03-11T21:27:25.305 INFO:teuthology.orchestra.run.smithi092.stdout: "osd_debug_drop_ping_probability": "0.000000", 2024-03-11T21:27:25.305 INFO:teuthology.orchestra.run.smithi092.stdout: "osd_debug_feed_pullee": "-1", 2024-03-11T21:27:25.305 INFO:teuthology.orchestra.run.smithi092.stdout: "osd_debug_inject_copyfrom_error": "false", 2024-03-11T21:27:25.306 INFO:teuthology.orchestra.run.smithi092.stdout: "osd_debug_inject_dispatch_delay_duration": "0.100000", 2024-03-11T21:27:25.306 INFO:teuthology.orchestra.run.smithi092.stdout: "osd_debug_inject_dispatch_delay_probability": "0.000000", 2024-03-11T21:27:25.306 INFO:teuthology.orchestra.run.smithi092.stdout: "osd_debug_misdirected_ops": "false", 2024-03-11T21:27:25.306 INFO:teuthology.orchestra.run.smithi092.stdout: "osd_debug_no_acting_change": "false", 2024-03-11T21:27:25.306 INFO:teuthology.orchestra.run.smithi092.stdout: "osd_debug_no_purge_strays": "false", 2024-03-11T21:27:25.306 INFO:teuthology.orchestra.run.smithi092.stdout: "osd_debug_op_order": "false", 2024-03-11T21:27:25.306 INFO:teuthology.orchestra.run.smithi092.stdout: "osd_debug_pg_log_writeout": "false", 2024-03-11T21:27:25.306 INFO:teuthology.orchestra.run.smithi092.stdout: "osd_debug_pretend_recovery_active": "false", 2024-03-11T21:27:25.306 INFO:teuthology.orchestra.run.smithi092.stdout: "osd_debug_random_push_read_error": "0.000000", 2024-03-11T21:27:25.306 INFO:teuthology.orchestra.run.smithi092.stdout: "osd_debug_reject_backfill_probability": "0.000000", 2024-03-11T21:27:25.306 INFO:teuthology.orchestra.run.smithi092.stdout: "osd_debug_shutdown": "false", 2024-03-11T21:27:25.306 INFO:teuthology.orchestra.run.smithi092.stdout: "osd_debug_skip_full_check_in_backfill_reservation": "false", 2024-03-11T21:27:25.306 INFO:teuthology.orchestra.run.smithi092.stdout: "osd_debug_skip_full_check_in_recovery": "false", 2024-03-11T21:27:25.307 INFO:teuthology.orchestra.run.smithi092.stdout: "osd_debug_trim_objects": "false", 2024-03-11T21:27:25.307 INFO:teuthology.orchestra.run.smithi092.stdout: "osd_debug_verify_cached_snaps": "false", 2024-03-11T21:27:25.307 INFO:teuthology.orchestra.run.smithi092.stdout: "osd_debug_verify_missing_on_start": "false", 2024-03-11T21:27:25.307 INFO:teuthology.orchestra.run.smithi092.stdout: "osd_debug_verify_snaps": "false", 2024-03-11T21:27:25.307 INFO:teuthology.orchestra.run.smithi092.stdout: "osd_debug_verify_stray_on_activate": "false", 2024-03-11T21:27:25.307 INFO:teuthology.orchestra.run.smithi092.stdout: "osd_deep_scrub_interval": "604800.000000", 2024-03-11T21:27:25.307 INFO:teuthology.orchestra.run.smithi092.stdout: "osd_deep_scrub_keys": "1024", 2024-03-11T21:27:25.307 INFO:teuthology.orchestra.run.smithi092.stdout: "osd_deep_scrub_large_omap_object_key_threshold": "200000", 2024-03-11T21:27:25.307 INFO:teuthology.orchestra.run.smithi092.stdout: "osd_deep_scrub_large_omap_object_value_sum_threshold": "1073741824", 2024-03-11T21:27:25.307 INFO:teuthology.orchestra.run.smithi092.stdout: "osd_deep_scrub_randomize_ratio": "0.150000", 2024-03-11T21:27:25.307 INFO:teuthology.orchestra.run.smithi092.stdout: "osd_deep_scrub_stride": "524288", 2024-03-11T21:27:25.307 INFO:teuthology.orchestra.run.smithi092.stdout: "osd_deep_scrub_update_digest_min_age": "7200", 2024-03-11T21:27:25.307 INFO:teuthology.orchestra.run.smithi092.stdout: "osd_default_data_pool_replay_window": "45", 2024-03-11T21:27:25.307 INFO:teuthology.orchestra.run.smithi092.stdout: "osd_default_notify_timeout": "30", 2024-03-11T21:27:25.307 INFO:teuthology.orchestra.run.smithi092.stdout: "osd_delete_sleep": "0.000000", 2024-03-11T21:27:25.308 INFO:teuthology.orchestra.run.smithi092.stdout: "osd_delete_sleep_hdd": "5.000000", 2024-03-11T21:27:25.308 INFO:teuthology.orchestra.run.smithi092.stdout: "osd_delete_sleep_hybrid": "1.000000", 2024-03-11T21:27:25.308 INFO:teuthology.orchestra.run.smithi092.stdout: "osd_delete_sleep_ssd": "1.000000", 2024-03-11T21:27:25.308 INFO:teuthology.orchestra.run.smithi092.stdout: "osd_discard_disconnected_ops": "true", 2024-03-11T21:27:25.308 INFO:teuthology.orchestra.run.smithi092.stdout: "osd_enable_op_tracker": "true", 2024-03-11T21:27:25.308 INFO:teuthology.orchestra.run.smithi092.stdout: "osd_erasure_code_plugins": "jerasure lrc", 2024-03-11T21:27:25.308 INFO:teuthology.orchestra.run.smithi092.stdout: "osd_extblkdev_plugins": "vdo", 2024-03-11T21:27:25.308 INFO:teuthology.orchestra.run.smithi092.stdout: "osd_failsafe_full_ratio": "0.970000", 2024-03-11T21:27:25.308 INFO:teuthology.orchestra.run.smithi092.stdout: "osd_fast_fail_on_connection_refused": "true", 2024-03-11T21:27:25.308 INFO:teuthology.orchestra.run.smithi092.stdout: "osd_fast_info": "true", 2024-03-11T21:27:25.308 INFO:teuthology.orchestra.run.smithi092.stdout: "osd_fast_shutdown": "true", 2024-03-11T21:27:25.308 INFO:teuthology.orchestra.run.smithi092.stdout: "osd_fast_shutdown_notify_mon": "true", 2024-03-11T21:27:25.308 INFO:teuthology.orchestra.run.smithi092.stdout: "osd_fast_shutdown_timeout": "15", 2024-03-11T21:27:25.308 INFO:teuthology.orchestra.run.smithi092.stdout: "osd_find_best_info_ignore_history_les": "false", 2024-03-11T21:27:25.309 INFO:teuthology.orchestra.run.smithi092.stdout: "osd_force_auth_primary_missing_objects": "100", 2024-03-11T21:27:25.309 INFO:teuthology.orchestra.run.smithi092.stdout: "osd_force_recovery_pg_log_entries_factor": "1.300000", 2024-03-11T21:27:25.309 INFO:teuthology.orchestra.run.smithi092.stdout: "osd_function_tracing": "false", 2024-03-11T21:27:25.309 INFO:teuthology.orchestra.run.smithi092.stdout: "osd_heartbeat_grace": "20", 2024-03-11T21:27:25.309 INFO:teuthology.orchestra.run.smithi092.stdout: "osd_heartbeat_interval": "6", 2024-03-11T21:27:25.309 INFO:teuthology.orchestra.run.smithi092.stdout: "osd_heartbeat_min_healthy_ratio": "0.330000", 2024-03-11T21:27:25.309 INFO:teuthology.orchestra.run.smithi092.stdout: "osd_heartbeat_min_peers": "10", 2024-03-11T21:27:25.309 INFO:teuthology.orchestra.run.smithi092.stdout: "osd_heartbeat_min_size": "2000", 2024-03-11T21:27:25.309 INFO:teuthology.orchestra.run.smithi092.stdout: "osd_heartbeat_stale": "600", 2024-03-11T21:27:25.309 INFO:teuthology.orchestra.run.smithi092.stdout: "osd_heartbeat_use_min_delay_socket": "false", 2024-03-11T21:27:25.309 INFO:teuthology.orchestra.run.smithi092.stdout: "osd_hit_set_max_size": "100000", 2024-03-11T21:27:25.309 INFO:teuthology.orchestra.run.smithi092.stdout: "osd_hit_set_min_size": "1000", 2024-03-11T21:27:25.309 INFO:teuthology.orchestra.run.smithi092.stdout: "osd_hit_set_namespace": ".ceph-internal", 2024-03-11T21:27:25.310 INFO:teuthology.orchestra.run.smithi092.stdout: "osd_ignore_stale_divergent_priors": "false", 2024-03-11T21:27:25.310 INFO:teuthology.orchestra.run.smithi092.stdout: "osd_inject_bad_map_crc_probability": "0.000000", 2024-03-11T21:27:25.310 INFO:teuthology.orchestra.run.smithi092.stdout: "osd_inject_failure_on_pg_removal": "false", 2024-03-11T21:27:25.310 INFO:teuthology.orchestra.run.smithi092.stdout: "osd_journal": "/var/lib/ceph/osd/ceph-a/journal", 2024-03-11T21:27:25.310 INFO:teuthology.orchestra.run.smithi092.stdout: "osd_journal_flush_on_shutdown": "true", 2024-03-11T21:27:25.310 INFO:teuthology.orchestra.run.smithi092.stdout: "osd_journal_size": "5120", 2024-03-11T21:27:25.310 INFO:teuthology.orchestra.run.smithi092.stdout: "osd_kill_backfill_at": "0", 2024-03-11T21:27:25.310 INFO:teuthology.orchestra.run.smithi092.stdout: "osd_loop_before_reset_tphandle": "64", 2024-03-11T21:27:25.310 INFO:teuthology.orchestra.run.smithi092.stdout: "osd_map_cache_size": "50", 2024-03-11T21:27:25.310 INFO:teuthology.orchestra.run.smithi092.stdout: "osd_map_dedup": "true", 2024-03-11T21:27:25.310 INFO:teuthology.orchestra.run.smithi092.stdout: "osd_map_message_max": "40", 2024-03-11T21:27:25.310 INFO:teuthology.orchestra.run.smithi092.stdout: "osd_map_message_max_bytes": "10485760", 2024-03-11T21:27:25.310 INFO:teuthology.orchestra.run.smithi092.stdout: "osd_map_share_max_epochs": "40", 2024-03-11T21:27:25.310 INFO:teuthology.orchestra.run.smithi092.stdout: "osd_max_attr_name_len": "100", 2024-03-11T21:27:25.311 INFO:teuthology.orchestra.run.smithi092.stdout: "osd_max_attr_size": "0", 2024-03-11T21:27:25.311 INFO:teuthology.orchestra.run.smithi092.stdout: "osd_max_backfills": "1", 2024-03-11T21:27:25.311 INFO:teuthology.orchestra.run.smithi092.stdout: "osd_max_markdown_count": "5", 2024-03-11T21:27:25.311 INFO:teuthology.orchestra.run.smithi092.stdout: "osd_max_markdown_period": "600", 2024-03-11T21:27:25.311 INFO:teuthology.orchestra.run.smithi092.stdout: "osd_max_object_name_len": "2048", 2024-03-11T21:27:25.311 INFO:teuthology.orchestra.run.smithi092.stdout: "osd_max_object_namespace_len": "256", 2024-03-11T21:27:25.311 INFO:teuthology.orchestra.run.smithi092.stdout: "osd_max_object_size": "134217728", 2024-03-11T21:27:25.311 INFO:teuthology.orchestra.run.smithi092.stdout: "osd_max_omap_bytes_per_request": "1073741824", 2024-03-11T21:27:25.311 INFO:teuthology.orchestra.run.smithi092.stdout: "osd_max_omap_entries_per_request": "1024", 2024-03-11T21:27:25.311 INFO:teuthology.orchestra.run.smithi092.stdout: "osd_max_pg_blocked_by": "16", 2024-03-11T21:27:25.311 INFO:teuthology.orchestra.run.smithi092.stdout: "osd_max_pg_log_entries": "10000", 2024-03-11T21:27:25.311 INFO:teuthology.orchestra.run.smithi092.stdout: "osd_max_pg_per_osd_hard_ratio": "3.000000", 2024-03-11T21:27:25.311 INFO:teuthology.orchestra.run.smithi092.stdout: "osd_max_pgls": "1024", 2024-03-11T21:27:25.311 INFO:teuthology.orchestra.run.smithi092.stdout: "osd_max_push_cost": "8388608", 2024-03-11T21:27:25.312 INFO:teuthology.orchestra.run.smithi092.stdout: "osd_max_push_objects": "10", 2024-03-11T21:27:25.312 INFO:teuthology.orchestra.run.smithi092.stdout: "osd_max_scrubs": "3", 2024-03-11T21:27:25.312 INFO:teuthology.orchestra.run.smithi092.stdout: "osd_max_snap_prune_intervals_per_epoch": "512", 2024-03-11T21:27:25.312 INFO:teuthology.orchestra.run.smithi092.stdout: "osd_max_trimming_pgs": "2", 2024-03-11T21:27:25.312 INFO:teuthology.orchestra.run.smithi092.stdout: "osd_max_write_op_reply_len": "64", 2024-03-11T21:27:25.312 INFO:teuthology.orchestra.run.smithi092.stdout: "osd_max_write_size": "90", 2024-03-11T21:27:25.312 INFO:teuthology.orchestra.run.smithi092.stdout: "osd_mclock_force_run_benchmark_on_init": "false", 2024-03-11T21:27:25.312 INFO:teuthology.orchestra.run.smithi092.stdout: "osd_mclock_iops_capacity_threshold_hdd": "500.000000", 2024-03-11T21:27:25.312 INFO:teuthology.orchestra.run.smithi092.stdout: "osd_mclock_iops_capacity_threshold_ssd": "80000.000000", 2024-03-11T21:27:25.312 INFO:teuthology.orchestra.run.smithi092.stdout: "osd_mclock_max_capacity_iops_hdd": "315.000000", 2024-03-11T21:27:25.312 INFO:teuthology.orchestra.run.smithi092.stdout: "osd_mclock_max_capacity_iops_ssd": "21500.000000", 2024-03-11T21:27:25.312 INFO:teuthology.orchestra.run.smithi092.stdout: "osd_mclock_max_sequential_bandwidth_hdd": "157286400", 2024-03-11T21:27:25.312 INFO:teuthology.orchestra.run.smithi092.stdout: "osd_mclock_max_sequential_bandwidth_ssd": "1258291200", 2024-03-11T21:27:25.312 INFO:teuthology.orchestra.run.smithi092.stdout: "osd_mclock_override_recovery_settings": "false", 2024-03-11T21:27:25.312 INFO:teuthology.orchestra.run.smithi092.stdout: "osd_mclock_profile": "balanced", 2024-03-11T21:27:25.313 INFO:teuthology.orchestra.run.smithi092.stdout: "osd_mclock_scheduler_anticipation_timeout": "0.000000", 2024-03-11T21:27:25.313 INFO:teuthology.orchestra.run.smithi092.stdout: "osd_mclock_scheduler_background_best_effort_lim": "0.000000", 2024-03-11T21:27:25.313 INFO:teuthology.orchestra.run.smithi092.stdout: "osd_mclock_scheduler_background_best_effort_res": "0.000000", 2024-03-11T21:27:25.313 INFO:teuthology.orchestra.run.smithi092.stdout: "osd_mclock_scheduler_background_best_effort_wgt": "1", 2024-03-11T21:27:25.313 INFO:teuthology.orchestra.run.smithi092.stdout: "osd_mclock_scheduler_background_recovery_lim": "0.000000", 2024-03-11T21:27:25.313 INFO:teuthology.orchestra.run.smithi092.stdout: "osd_mclock_scheduler_background_recovery_res": "0.000000", 2024-03-11T21:27:25.313 INFO:teuthology.orchestra.run.smithi092.stdout: "osd_mclock_scheduler_background_recovery_wgt": "1", 2024-03-11T21:27:25.313 INFO:teuthology.orchestra.run.smithi092.stdout: "osd_mclock_scheduler_client_lim": "0.000000", 2024-03-11T21:27:25.313 INFO:teuthology.orchestra.run.smithi092.stdout: "osd_mclock_scheduler_client_res": "0.000000", 2024-03-11T21:27:25.313 INFO:teuthology.orchestra.run.smithi092.stdout: "osd_mclock_scheduler_client_wgt": "1", 2024-03-11T21:27:25.313 INFO:teuthology.orchestra.run.smithi092.stdout: "osd_mclock_skip_benchmark": "false", 2024-03-11T21:27:25.313 INFO:teuthology.orchestra.run.smithi092.stdout: "osd_memory_base": "805306368", 2024-03-11T21:27:25.313 INFO:teuthology.orchestra.run.smithi092.stdout: "osd_memory_cache_min": "134217728", 2024-03-11T21:27:25.313 INFO:teuthology.orchestra.run.smithi092.stdout: "osd_memory_cache_resize_interval": "1.000000", 2024-03-11T21:27:25.314 INFO:teuthology.orchestra.run.smithi092.stdout: "osd_memory_expected_fragmentation": "0.150000", 2024-03-11T21:27:25.314 INFO:teuthology.orchestra.run.smithi092.stdout: "osd_memory_target": "4294967296", 2024-03-11T21:27:25.314 INFO:teuthology.orchestra.run.smithi092.stdout: "osd_memory_target_autotune": "false", 2024-03-11T21:27:25.314 INFO:teuthology.orchestra.run.smithi092.stdout: "osd_memory_target_cgroup_limit_ratio": "0.800000", 2024-03-11T21:27:25.314 INFO:teuthology.orchestra.run.smithi092.stdout: "osd_min_pg_log_entries": "250", 2024-03-11T21:27:25.314 INFO:teuthology.orchestra.run.smithi092.stdout: "osd_min_recovery_priority": "0", 2024-03-11T21:27:25.314 INFO:teuthology.orchestra.run.smithi092.stdout: "osd_mon_heartbeat_interval": "30", 2024-03-11T21:27:25.314 INFO:teuthology.orchestra.run.smithi092.stdout: "osd_mon_heartbeat_stat_stale": "3600", 2024-03-11T21:27:25.314 INFO:teuthology.orchestra.run.smithi092.stdout: "osd_mon_report_interval": "5", 2024-03-11T21:27:25.314 INFO:teuthology.orchestra.run.smithi092.stdout: "osd_mon_report_max_in_flight": "2", 2024-03-11T21:27:25.314 INFO:teuthology.orchestra.run.smithi092.stdout: "osd_mon_shutdown_timeout": "5.000000", 2024-03-11T21:27:25.314 INFO:teuthology.orchestra.run.smithi092.stdout: "osd_num_cache_shards": "32", 2024-03-11T21:27:25.314 INFO:teuthology.orchestra.run.smithi092.stdout: "osd_num_op_tracker_shard": "32", 2024-03-11T21:27:25.315 INFO:teuthology.orchestra.run.smithi092.stdout: "osd_numa_auto_affinity": "true", 2024-03-11T21:27:25.315 INFO:teuthology.orchestra.run.smithi092.stdout: "osd_numa_node": "-1", 2024-03-11T21:27:25.315 INFO:teuthology.orchestra.run.smithi092.stdout: "osd_numa_prefer_iface": "true", 2024-03-11T21:27:25.315 INFO:teuthology.orchestra.run.smithi092.stdout: "osd_object_clean_region_max_num_intervals": "10", 2024-03-11T21:27:25.315 INFO:teuthology.orchestra.run.smithi092.stdout: "osd_objecter_finishers": "1", 2024-03-11T21:27:25.315 INFO:teuthology.orchestra.run.smithi092.stdout: "osd_objectstore": "bluestore", 2024-03-11T21:27:25.315 INFO:teuthology.orchestra.run.smithi092.stdout: "osd_objectstore_fuse": "false", 2024-03-11T21:27:25.315 INFO:teuthology.orchestra.run.smithi092.stdout: "osd_objectstore_tracing": "false", 2024-03-11T21:27:25.315 INFO:teuthology.orchestra.run.smithi092.stdout: "osd_op_complaint_time": "30.000000", 2024-03-11T21:27:25.315 INFO:teuthology.orchestra.run.smithi092.stdout: "osd_op_history_duration": "600", 2024-03-11T21:27:25.315 INFO:teuthology.orchestra.run.smithi092.stdout: "osd_op_history_size": "20", 2024-03-11T21:27:25.315 INFO:teuthology.orchestra.run.smithi092.stdout: "osd_op_history_slow_op_size": "20", 2024-03-11T21:27:25.315 INFO:teuthology.orchestra.run.smithi092.stdout: "osd_op_history_slow_op_threshold": "10.000000", 2024-03-11T21:27:25.315 INFO:teuthology.orchestra.run.smithi092.stdout: "osd_op_log_threshold": "5", 2024-03-11T21:27:25.316 INFO:teuthology.orchestra.run.smithi092.stdout: "osd_op_num_shards": "0", 2024-03-11T21:27:25.316 INFO:teuthology.orchestra.run.smithi092.stdout: "osd_op_num_shards_hdd": "5", 2024-03-11T21:27:25.316 INFO:teuthology.orchestra.run.smithi092.stdout: "osd_op_num_shards_ssd": "8", 2024-03-11T21:27:25.316 INFO:teuthology.orchestra.run.smithi092.stdout: "osd_op_num_threads_per_shard": "0", 2024-03-11T21:27:25.316 INFO:teuthology.orchestra.run.smithi092.stdout: "osd_op_num_threads_per_shard_hdd": "1", 2024-03-11T21:27:25.316 INFO:teuthology.orchestra.run.smithi092.stdout: "osd_op_num_threads_per_shard_ssd": "2", 2024-03-11T21:27:25.316 INFO:teuthology.orchestra.run.smithi092.stdout: "osd_op_pq_max_tokens_per_priority": "4194304", 2024-03-11T21:27:25.316 INFO:teuthology.orchestra.run.smithi092.stdout: "osd_op_pq_min_cost": "65536", 2024-03-11T21:27:25.316 INFO:teuthology.orchestra.run.smithi092.stdout: "osd_op_queue": "mclock_scheduler", 2024-03-11T21:27:25.316 INFO:teuthology.orchestra.run.smithi092.stdout: "osd_op_queue_cut_off": "high", 2024-03-11T21:27:25.316 INFO:teuthology.orchestra.run.smithi092.stdout: "osd_op_thread_suicide_timeout": "150", 2024-03-11T21:27:25.316 INFO:teuthology.orchestra.run.smithi092.stdout: "osd_op_thread_timeout": "15", 2024-03-11T21:27:25.316 INFO:teuthology.orchestra.run.smithi092.stdout: "osd_open_classes_on_start": "true", 2024-03-11T21:27:25.316 INFO:teuthology.orchestra.run.smithi092.stdout: "osd_os_flags": "0", 2024-03-11T21:27:25.317 INFO:teuthology.orchestra.run.smithi092.stdout: "osd_peering_op_priority": "255", 2024-03-11T21:27:25.317 INFO:teuthology.orchestra.run.smithi092.stdout: "osd_pg_delete_cost": "1048576", 2024-03-11T21:27:25.317 INFO:teuthology.orchestra.run.smithi092.stdout: "osd_pg_delete_priority": "5", 2024-03-11T21:27:25.317 INFO:teuthology.orchestra.run.smithi092.stdout: "osd_pg_epoch_max_lag_factor": "2.000000", 2024-03-11T21:27:25.317 INFO:teuthology.orchestra.run.smithi092.stdout: "osd_pg_epoch_persisted_max_stale": "40", 2024-03-11T21:27:25.317 INFO:teuthology.orchestra.run.smithi092.stdout: "osd_pg_log_dups_tracked": "3000", 2024-03-11T21:27:25.317 INFO:teuthology.orchestra.run.smithi092.stdout: "osd_pg_log_trim_max": "10000", 2024-03-11T21:27:25.317 INFO:teuthology.orchestra.run.smithi092.stdout: "osd_pg_log_trim_min": "100", 2024-03-11T21:27:25.317 INFO:teuthology.orchestra.run.smithi092.stdout: "osd_pg_max_concurrent_snap_trims": "2", 2024-03-11T21:27:25.317 INFO:teuthology.orchestra.run.smithi092.stdout: "osd_pg_object_context_cache_count": "64", 2024-03-11T21:27:25.317 INFO:teuthology.orchestra.run.smithi092.stdout: "osd_pg_stat_report_interval_max_epochs": "500", 2024-03-11T21:27:25.317 INFO:teuthology.orchestra.run.smithi092.stdout: "osd_pg_stat_report_interval_max_seconds": "5", 2024-03-11T21:27:25.317 INFO:teuthology.orchestra.run.smithi092.stdout: "osd_pool_default_cache_max_evict_check_size": "10", 2024-03-11T21:27:25.317 INFO:teuthology.orchestra.run.smithi092.stdout: "osd_pool_default_cache_min_evict_age": "0", 2024-03-11T21:27:25.317 INFO:teuthology.orchestra.run.smithi092.stdout: "osd_pool_default_cache_min_flush_age": "0", 2024-03-11T21:27:25.317 INFO:teuthology.orchestra.run.smithi092.stdout: "osd_pool_default_cache_target_dirty_high_ratio": "0.600000", 2024-03-11T21:27:25.318 INFO:teuthology.orchestra.run.smithi092.stdout: "osd_pool_default_cache_target_dirty_ratio": "0.400000", 2024-03-11T21:27:25.318 INFO:teuthology.orchestra.run.smithi092.stdout: "osd_pool_default_cache_target_full_ratio": "0.800000", 2024-03-11T21:27:25.318 INFO:teuthology.orchestra.run.smithi092.stdout: "osd_pool_default_crimson": "false", 2024-03-11T21:27:25.318 INFO:teuthology.orchestra.run.smithi092.stdout: "osd_pool_default_crush_rule": "-1", 2024-03-11T21:27:25.318 INFO:teuthology.orchestra.run.smithi092.stdout: "osd_pool_default_ec_fast_read": "false", 2024-03-11T21:27:25.318 INFO:teuthology.orchestra.run.smithi092.stdout: "osd_pool_default_erasure_code_profile": "plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd", 2024-03-11T21:27:25.318 INFO:teuthology.orchestra.run.smithi092.stdout: "osd_pool_default_flag_bulk": "false", 2024-03-11T21:27:25.318 INFO:teuthology.orchestra.run.smithi092.stdout: "osd_pool_default_flag_hashpspool": "true", 2024-03-11T21:27:25.318 INFO:teuthology.orchestra.run.smithi092.stdout: "osd_pool_default_flag_nodelete": "false", 2024-03-11T21:27:25.318 INFO:teuthology.orchestra.run.smithi092.stdout: "osd_pool_default_flag_nopgchange": "false", 2024-03-11T21:27:25.318 INFO:teuthology.orchestra.run.smithi092.stdout: "osd_pool_default_flag_nosizechange": "false", 2024-03-11T21:27:25.318 INFO:teuthology.orchestra.run.smithi092.stdout: "osd_pool_default_flags": "0", 2024-03-11T21:27:25.318 INFO:teuthology.orchestra.run.smithi092.stdout: "osd_pool_default_hit_set_bloom_fpp": "0.050000", 2024-03-11T21:27:25.319 INFO:teuthology.orchestra.run.smithi092.stdout: "osd_pool_default_min_size": "0", 2024-03-11T21:27:25.319 INFO:teuthology.orchestra.run.smithi092.stdout: "osd_pool_default_pg_autoscale_mode": "off", 2024-03-11T21:27:25.319 INFO:teuthology.orchestra.run.smithi092.stdout: "osd_pool_default_pg_num": "32", 2024-03-11T21:27:25.319 INFO:teuthology.orchestra.run.smithi092.stdout: "osd_pool_default_pgp_num": "0", 2024-03-11T21:27:25.319 INFO:teuthology.orchestra.run.smithi092.stdout: "osd_pool_default_read_lease_ratio": "0.800000", 2024-03-11T21:27:25.319 INFO:teuthology.orchestra.run.smithi092.stdout: "osd_pool_default_read_ratio": "70", 2024-03-11T21:27:25.319 INFO:teuthology.orchestra.run.smithi092.stdout: "osd_pool_default_size": "3", 2024-03-11T21:27:25.319 INFO:teuthology.orchestra.run.smithi092.stdout: "osd_pool_default_type": "replicated", 2024-03-11T21:27:25.319 INFO:teuthology.orchestra.run.smithi092.stdout: "osd_pool_erasure_code_stripe_unit": "4096", 2024-03-11T21:27:25.320 INFO:teuthology.orchestra.run.smithi092.stdout: "osd_pool_use_gmt_hitset": "true", 2024-03-11T21:27:25.320 INFO:teuthology.orchestra.run.smithi092.stdout: "osd_push_per_object_cost": "1000", 2024-03-11T21:27:25.320 INFO:teuthology.orchestra.run.smithi092.stdout: "osd_read_ec_check_for_errors": "false", 2024-03-11T21:27:25.320 INFO:teuthology.orchestra.run.smithi092.stdout: "osd_recover_clone_overlap": "true", 2024-03-11T21:27:25.320 INFO:teuthology.orchestra.run.smithi092.stdout: "osd_recover_clone_overlap_limit": "10", 2024-03-11T21:27:25.320 INFO:teuthology.orchestra.run.smithi092.stdout: "osd_recovery_cost": "20971520", 2024-03-11T21:27:25.320 INFO:teuthology.orchestra.run.smithi092.stdout: "osd_recovery_delay_start": "0.000000", 2024-03-11T21:27:25.320 INFO:teuthology.orchestra.run.smithi092.stdout: "osd_recovery_max_active": "0", 2024-03-11T21:27:25.320 INFO:teuthology.orchestra.run.smithi092.stdout: "osd_recovery_max_active_hdd": "3", 2024-03-11T21:27:25.320 INFO:teuthology.orchestra.run.smithi092.stdout: "osd_recovery_max_active_ssd": "10", 2024-03-11T21:27:25.320 INFO:teuthology.orchestra.run.smithi092.stdout: "osd_recovery_max_chunk": "8388608", 2024-03-11T21:27:25.320 INFO:teuthology.orchestra.run.smithi092.stdout: "osd_recovery_max_omap_entries_per_chunk": "8096", 2024-03-11T21:27:25.320 INFO:teuthology.orchestra.run.smithi092.stdout: "osd_recovery_max_single_start": "1", 2024-03-11T21:27:25.320 INFO:teuthology.orchestra.run.smithi092.stdout: "osd_recovery_op_priority": "3", 2024-03-11T21:27:25.321 INFO:teuthology.orchestra.run.smithi092.stdout: "osd_recovery_op_warn_multiple": "16", 2024-03-11T21:27:25.321 INFO:teuthology.orchestra.run.smithi092.stdout: "osd_recovery_priority": "5", 2024-03-11T21:27:25.321 INFO:teuthology.orchestra.run.smithi092.stdout: "osd_recovery_retry_interval": "30.000000", 2024-03-11T21:27:25.321 INFO:teuthology.orchestra.run.smithi092.stdout: "osd_recovery_sleep": "0.000000", 2024-03-11T21:27:25.321 INFO:teuthology.orchestra.run.smithi092.stdout: "osd_recovery_sleep_hdd": "0.100000", 2024-03-11T21:27:25.321 INFO:teuthology.orchestra.run.smithi092.stdout: "osd_recovery_sleep_hybrid": "0.025000", 2024-03-11T21:27:25.321 INFO:teuthology.orchestra.run.smithi092.stdout: "osd_recovery_sleep_ssd": "0.000000", 2024-03-11T21:27:25.321 INFO:teuthology.orchestra.run.smithi092.stdout: "osd_repair_during_recovery": "false", 2024-03-11T21:27:25.321 INFO:teuthology.orchestra.run.smithi092.stdout: "osd_requested_scrub_priority": "120", 2024-03-11T21:27:25.321 INFO:teuthology.orchestra.run.smithi092.stdout: "osd_rocksdb_iterator_bounds_enabled": "true", 2024-03-11T21:27:25.321 INFO:teuthology.orchestra.run.smithi092.stdout: "osd_rollback_to_cluster_snap": "", 2024-03-11T21:27:25.321 INFO:teuthology.orchestra.run.smithi092.stdout: "osd_scrub_auto_repair": "false", 2024-03-11T21:27:25.321 INFO:teuthology.orchestra.run.smithi092.stdout: "osd_scrub_auto_repair_num_errors": "5", 2024-03-11T21:27:25.322 INFO:teuthology.orchestra.run.smithi092.stdout: "osd_scrub_backoff_ratio": "0.660000", 2024-03-11T21:27:25.322 INFO:teuthology.orchestra.run.smithi092.stdout: "osd_scrub_begin_hour": "0", 2024-03-11T21:27:25.322 INFO:teuthology.orchestra.run.smithi092.stdout: "osd_scrub_begin_week_day": "0", 2024-03-11T21:27:25.322 INFO:teuthology.orchestra.run.smithi092.stdout: "osd_scrub_chunk_max": "25", 2024-03-11T21:27:25.322 INFO:teuthology.orchestra.run.smithi092.stdout: "osd_scrub_chunk_min": "5", 2024-03-11T21:27:25.322 INFO:teuthology.orchestra.run.smithi092.stdout: "osd_scrub_cost": "52428800", 2024-03-11T21:27:25.322 INFO:teuthology.orchestra.run.smithi092.stdout: "osd_scrub_disable_reservation_queuing": "false", 2024-03-11T21:27:25.322 INFO:teuthology.orchestra.run.smithi092.stdout: "osd_scrub_during_recovery": "false", 2024-03-11T21:27:25.322 INFO:teuthology.orchestra.run.smithi092.stdout: "osd_scrub_end_hour": "0", 2024-03-11T21:27:25.322 INFO:teuthology.orchestra.run.smithi092.stdout: "osd_scrub_end_week_day": "0", 2024-03-11T21:27:25.322 INFO:teuthology.orchestra.run.smithi092.stdout: "osd_scrub_event_cost": "4096", 2024-03-11T21:27:25.322 INFO:teuthology.orchestra.run.smithi092.stdout: "osd_scrub_extended_sleep": "0.000000", 2024-03-11T21:27:25.322 INFO:teuthology.orchestra.run.smithi092.stdout: "osd_scrub_interval_randomize_ratio": "0.500000", 2024-03-11T21:27:25.323 INFO:teuthology.orchestra.run.smithi092.stdout: "osd_scrub_invalid_stats": "true", 2024-03-11T21:27:25.323 INFO:teuthology.orchestra.run.smithi092.stdout: "osd_scrub_load_threshold": "0.500000", 2024-03-11T21:27:25.323 INFO:teuthology.orchestra.run.smithi092.stdout: "osd_scrub_max_interval": "604800.000000", 2024-03-11T21:27:25.323 INFO:teuthology.orchestra.run.smithi092.stdout: "osd_scrub_max_preemptions": "5", 2024-03-11T21:27:25.323 INFO:teuthology.orchestra.run.smithi092.stdout: "osd_scrub_min_interval": "86400.000000", 2024-03-11T21:27:25.323 INFO:teuthology.orchestra.run.smithi092.stdout: "osd_scrub_priority": "5", 2024-03-11T21:27:25.323 INFO:teuthology.orchestra.run.smithi092.stdout: "osd_scrub_reservation_timeout": "300000", 2024-03-11T21:27:25.323 INFO:teuthology.orchestra.run.smithi092.stdout: "osd_scrub_sleep": "0.000000", 2024-03-11T21:27:25.323 INFO:teuthology.orchestra.run.smithi092.stdout: "osd_scrub_slow_reservation_response": "30000", 2024-03-11T21:27:25.323 INFO:teuthology.orchestra.run.smithi092.stdout: "osd_shallow_scrub_chunk_max": "100", 2024-03-11T21:27:25.323 INFO:teuthology.orchestra.run.smithi092.stdout: "osd_shallow_scrub_chunk_min": "50", 2024-03-11T21:27:25.323 INFO:teuthology.orchestra.run.smithi092.stdout: "osd_shutdown_pgref_assert": "false", 2024-03-11T21:27:25.323 INFO:teuthology.orchestra.run.smithi092.stdout: "osd_skip_check_past_interval_bounds": "false", 2024-03-11T21:27:25.323 INFO:teuthology.orchestra.run.smithi092.stdout: "osd_skip_data_digest": "false", 2024-03-11T21:27:25.323 INFO:teuthology.orchestra.run.smithi092.stdout: "osd_smart_report_timeout": "5", 2024-03-11T21:27:25.324 INFO:teuthology.orchestra.run.smithi092.stdout: "osd_snap_trim_cost": "1048576", 2024-03-11T21:27:25.324 INFO:teuthology.orchestra.run.smithi092.stdout: "osd_snap_trim_priority": "5", 2024-03-11T21:27:25.324 INFO:teuthology.orchestra.run.smithi092.stdout: "osd_snap_trim_sleep": "0.000000", 2024-03-11T21:27:25.324 INFO:teuthology.orchestra.run.smithi092.stdout: "osd_snap_trim_sleep_hdd": "5.000000", 2024-03-11T21:27:25.324 INFO:teuthology.orchestra.run.smithi092.stdout: "osd_snap_trim_sleep_hybrid": "2.000000", 2024-03-11T21:27:25.324 INFO:teuthology.orchestra.run.smithi092.stdout: "osd_snap_trim_sleep_ssd": "0.000000", 2024-03-11T21:27:25.324 INFO:teuthology.orchestra.run.smithi092.stdout: "osd_stats_update_period_not_scrubbing": "120", 2024-03-11T21:27:25.324 INFO:teuthology.orchestra.run.smithi092.stdout: "osd_stats_update_period_scrubbing": "15", 2024-03-11T21:27:25.324 INFO:teuthology.orchestra.run.smithi092.stdout: "osd_target_pg_log_entries_per_osd": "300000", 2024-03-11T21:27:25.324 INFO:teuthology.orchestra.run.smithi092.stdout: "osd_target_transaction_size": "30", 2024-03-11T21:27:25.324 INFO:teuthology.orchestra.run.smithi092.stdout: "osd_tier_default_cache_hit_set_count": "4", 2024-03-11T21:27:25.324 INFO:teuthology.orchestra.run.smithi092.stdout: "osd_tier_default_cache_hit_set_grade_decay_rate": "20", 2024-03-11T21:27:25.324 INFO:teuthology.orchestra.run.smithi092.stdout: "osd_tier_default_cache_hit_set_period": "1200", 2024-03-11T21:27:25.324 INFO:teuthology.orchestra.run.smithi092.stdout: "osd_tier_default_cache_hit_set_search_last_n": "1", 2024-03-11T21:27:25.324 INFO:teuthology.orchestra.run.smithi092.stdout: "osd_tier_default_cache_hit_set_type": "bloom", 2024-03-11T21:27:25.325 INFO:teuthology.orchestra.run.smithi092.stdout: "osd_tier_default_cache_min_read_recency_for_promote": "1", 2024-03-11T21:27:25.325 INFO:teuthology.orchestra.run.smithi092.stdout: "osd_tier_default_cache_min_write_recency_for_promote": "1", 2024-03-11T21:27:25.325 INFO:teuthology.orchestra.run.smithi092.stdout: "osd_tier_default_cache_mode": "writeback", 2024-03-11T21:27:25.325 INFO:teuthology.orchestra.run.smithi092.stdout: "osd_tier_promote_max_bytes_sec": "5242880", 2024-03-11T21:27:25.325 INFO:teuthology.orchestra.run.smithi092.stdout: "osd_tier_promote_max_objects_sec": "25", 2024-03-11T21:27:25.325 INFO:teuthology.orchestra.run.smithi092.stdout: "osd_tracing": "false", 2024-03-11T21:27:25.325 INFO:teuthology.orchestra.run.smithi092.stdout: "osd_use_stale_snap": "false", 2024-03-11T21:27:25.325 INFO:teuthology.orchestra.run.smithi092.stdout: "osd_uuid": "00000000-0000-0000-0000-000000000000", 2024-03-11T21:27:25.325 INFO:teuthology.orchestra.run.smithi092.stdout: "osdc_blkin_trace_all": "false", 2024-03-11T21:27:25.325 INFO:teuthology.orchestra.run.smithi092.stdout: "paxos_kill_at": "0", 2024-03-11T21:27:25.325 INFO:teuthology.orchestra.run.smithi092.stdout: "paxos_max_join_drift": "10", 2024-03-11T21:27:25.325 INFO:teuthology.orchestra.run.smithi092.stdout: "paxos_min": "500", 2024-03-11T21:27:25.325 INFO:teuthology.orchestra.run.smithi092.stdout: "paxos_min_wait": "0.050000", 2024-03-11T21:27:25.326 INFO:teuthology.orchestra.run.smithi092.stdout: "paxos_propose_interval": "1.000000", 2024-03-11T21:27:25.326 INFO:teuthology.orchestra.run.smithi092.stdout: "paxos_service_trim_max": "500", 2024-03-11T21:27:25.326 INFO:teuthology.orchestra.run.smithi092.stdout: "paxos_service_trim_max_multiplier": "20", 2024-03-11T21:27:25.326 INFO:teuthology.orchestra.run.smithi092.stdout: "paxos_service_trim_min": "10", 2024-03-11T21:27:25.326 INFO:teuthology.orchestra.run.smithi092.stdout: "paxos_stash_full_interval": "25", 2024-03-11T21:27:25.326 INFO:teuthology.orchestra.run.smithi092.stdout: "paxos_trim_max": "500", 2024-03-11T21:27:25.326 INFO:teuthology.orchestra.run.smithi092.stdout: "paxos_trim_min": "250", 2024-03-11T21:27:25.326 INFO:teuthology.orchestra.run.smithi092.stdout: "perf": "true", 2024-03-11T21:27:25.326 INFO:teuthology.orchestra.run.smithi092.stdout: "pid_file": "", 2024-03-11T21:27:25.326 INFO:teuthology.orchestra.run.smithi092.stdout: "plugin_crypto_accelerator": "crypto_isal", 2024-03-11T21:27:25.326 INFO:teuthology.orchestra.run.smithi092.stdout: "plugin_dir": "/usr/lib64/ceph", 2024-03-11T21:27:25.326 INFO:teuthology.orchestra.run.smithi092.stdout: "public_addr": "-", 2024-03-11T21:27:25.326 INFO:teuthology.orchestra.run.smithi092.stdout: "public_addrv": "", 2024-03-11T21:27:25.326 INFO:teuthology.orchestra.run.smithi092.stdout: "public_bind_addr": "-", 2024-03-11T21:27:25.327 INFO:teuthology.orchestra.run.smithi092.stdout: "public_network": "", 2024-03-11T21:27:25.327 INFO:teuthology.orchestra.run.smithi092.stdout: "public_network_interface": "", 2024-03-11T21:27:25.327 INFO:teuthology.orchestra.run.smithi092.stdout: "qat_compressor_enabled": "false", 2024-03-11T21:27:25.327 INFO:teuthology.orchestra.run.smithi092.stdout: "qat_compressor_session_max_number": "256", 2024-03-11T21:27:25.327 INFO:teuthology.orchestra.run.smithi092.stdout: "rados_mon_op_timeout": "0", 2024-03-11T21:27:25.327 INFO:teuthology.orchestra.run.smithi092.stdout: "rados_osd_op_timeout": "0", 2024-03-11T21:27:25.327 INFO:teuthology.orchestra.run.smithi092.stdout: "rados_tracing": "false", 2024-03-11T21:27:25.327 INFO:teuthology.orchestra.run.smithi092.stdout: "rbd_atime_update_interval": "60", 2024-03-11T21:27:25.327 INFO:teuthology.orchestra.run.smithi092.stdout: "rbd_auto_exclusive_lock_until_manual_request": "true", 2024-03-11T21:27:25.327 INFO:teuthology.orchestra.run.smithi092.stdout: "rbd_balance_parent_reads": "false", 2024-03-11T21:27:25.327 INFO:teuthology.orchestra.run.smithi092.stdout: "rbd_balance_snap_reads": "false", 2024-03-11T21:27:25.327 INFO:teuthology.orchestra.run.smithi092.stdout: "rbd_blkin_trace_all": "false", 2024-03-11T21:27:25.327 INFO:teuthology.orchestra.run.smithi092.stdout: "rbd_blocklist_expire_seconds": "0", 2024-03-11T21:27:25.327 INFO:teuthology.orchestra.run.smithi092.stdout: "rbd_blocklist_on_break_lock": "true", 2024-03-11T21:27:25.327 INFO:teuthology.orchestra.run.smithi092.stdout: "rbd_cache": "true", 2024-03-11T21:27:25.327 INFO:teuthology.orchestra.run.smithi092.stdout: "rbd_cache_block_writes_upfront": "false", 2024-03-11T21:27:25.328 INFO:teuthology.orchestra.run.smithi092.stdout: "rbd_cache_max_dirty": "25165824", 2024-03-11T21:27:25.328 INFO:teuthology.orchestra.run.smithi092.stdout: "rbd_cache_max_dirty_age": "1.000000", 2024-03-11T21:27:25.328 INFO:teuthology.orchestra.run.smithi092.stdout: "rbd_cache_max_dirty_object": "0", 2024-03-11T21:27:25.328 INFO:teuthology.orchestra.run.smithi092.stdout: "rbd_cache_policy": "writearound", 2024-03-11T21:27:25.328 INFO:teuthology.orchestra.run.smithi092.stdout: "rbd_cache_size": "33554432", 2024-03-11T21:27:25.328 INFO:teuthology.orchestra.run.smithi092.stdout: "rbd_cache_target_dirty": "16777216", 2024-03-11T21:27:25.328 INFO:teuthology.orchestra.run.smithi092.stdout: "rbd_cache_writethrough_until_flush": "true", 2024-03-11T21:27:25.328 INFO:teuthology.orchestra.run.smithi092.stdout: "rbd_clone_copy_on_read": "false", 2024-03-11T21:27:25.328 INFO:teuthology.orchestra.run.smithi092.stdout: "rbd_compression_hint": "none", 2024-03-11T21:27:25.328 INFO:teuthology.orchestra.run.smithi092.stdout: "rbd_concurrent_management_ops": "10", 2024-03-11T21:27:25.328 INFO:teuthology.orchestra.run.smithi092.stdout: "rbd_config_pool_override_update_timestamp": "0", 2024-03-11T21:27:25.328 INFO:teuthology.orchestra.run.smithi092.stdout: "rbd_default_clone_format": "auto", 2024-03-11T21:27:25.328 INFO:teuthology.orchestra.run.smithi092.stdout: "rbd_default_data_pool": "", 2024-03-11T21:27:25.328 INFO:teuthology.orchestra.run.smithi092.stdout: "rbd_default_features": "61", 2024-03-11T21:27:25.328 INFO:teuthology.orchestra.run.smithi092.stdout: "rbd_default_format": "2", 2024-03-11T21:27:25.329 INFO:teuthology.orchestra.run.smithi092.stdout: "rbd_default_map_options": "", 2024-03-11T21:27:25.329 INFO:teuthology.orchestra.run.smithi092.stdout: "rbd_default_order": "22", 2024-03-11T21:27:25.329 INFO:teuthology.orchestra.run.smithi092.stdout: "rbd_default_pool": "rbd", 2024-03-11T21:27:25.329 INFO:teuthology.orchestra.run.smithi092.stdout: "rbd_default_snapshot_quiesce_mode": "required", 2024-03-11T21:27:25.329 INFO:teuthology.orchestra.run.smithi092.stdout: "rbd_default_stripe_count": "0", 2024-03-11T21:27:25.329 INFO:teuthology.orchestra.run.smithi092.stdout: "rbd_default_stripe_unit": "0", 2024-03-11T21:27:25.329 INFO:teuthology.orchestra.run.smithi092.stdout: "rbd_disable_zero_copy_writes": "true", 2024-03-11T21:27:25.329 INFO:teuthology.orchestra.run.smithi092.stdout: "rbd_discard_granularity_bytes": "65536", 2024-03-11T21:27:25.329 INFO:teuthology.orchestra.run.smithi092.stdout: "rbd_discard_on_zeroed_write_same": "true", 2024-03-11T21:27:25.329 INFO:teuthology.orchestra.run.smithi092.stdout: "rbd_enable_alloc_hint": "true", 2024-03-11T21:27:25.329 INFO:teuthology.orchestra.run.smithi092.stdout: "rbd_invalidate_object_map_on_timeout": "true", 2024-03-11T21:27:25.329 INFO:teuthology.orchestra.run.smithi092.stdout: "rbd_io_scheduler": "simple", 2024-03-11T21:27:25.329 INFO:teuthology.orchestra.run.smithi092.stdout: "rbd_io_scheduler_simple_max_delay": "0", 2024-03-11T21:27:25.330 INFO:teuthology.orchestra.run.smithi092.stdout: "rbd_journal_commit_age": "5.000000", 2024-03-11T21:27:25.330 INFO:teuthology.orchestra.run.smithi092.stdout: "rbd_journal_max_concurrent_object_sets": "0", 2024-03-11T21:27:25.330 INFO:teuthology.orchestra.run.smithi092.stdout: "rbd_journal_max_payload_bytes": "16384", 2024-03-11T21:27:25.330 INFO:teuthology.orchestra.run.smithi092.stdout: "rbd_journal_object_flush_age": "0.000000", 2024-03-11T21:27:25.330 INFO:teuthology.orchestra.run.smithi092.stdout: "rbd_journal_object_flush_bytes": "1048576", 2024-03-11T21:27:25.330 INFO:teuthology.orchestra.run.smithi092.stdout: "rbd_journal_object_flush_interval": "0", 2024-03-11T21:27:25.330 INFO:teuthology.orchestra.run.smithi092.stdout: "rbd_journal_object_max_in_flight_appends": "0", 2024-03-11T21:27:25.330 INFO:teuthology.orchestra.run.smithi092.stdout: "rbd_journal_object_writethrough_until_flush": "true", 2024-03-11T21:27:25.330 INFO:teuthology.orchestra.run.smithi092.stdout: "rbd_journal_order": "24", 2024-03-11T21:27:25.330 INFO:teuthology.orchestra.run.smithi092.stdout: "rbd_journal_pool": "", 2024-03-11T21:27:25.330 INFO:teuthology.orchestra.run.smithi092.stdout: "rbd_journal_splay_width": "4", 2024-03-11T21:27:25.330 INFO:teuthology.orchestra.run.smithi092.stdout: "rbd_localize_parent_reads": "false", 2024-03-11T21:27:25.330 INFO:teuthology.orchestra.run.smithi092.stdout: "rbd_localize_snap_reads": "false", 2024-03-11T21:27:25.331 INFO:teuthology.orchestra.run.smithi092.stdout: "rbd_mirror_concurrent_image_deletions": "1", 2024-03-11T21:27:25.331 INFO:teuthology.orchestra.run.smithi092.stdout: "rbd_mirror_concurrent_image_syncs": "5", 2024-03-11T21:27:25.331 INFO:teuthology.orchestra.run.smithi092.stdout: "rbd_mirror_delete_retry_interval": "30.000000", 2024-03-11T21:27:25.331 INFO:teuthology.orchestra.run.smithi092.stdout: "rbd_mirror_image_perf_stats_prio": "5", 2024-03-11T21:27:25.331 INFO:teuthology.orchestra.run.smithi092.stdout: "rbd_mirror_image_policy_migration_throttle": "300", 2024-03-11T21:27:25.331 INFO:teuthology.orchestra.run.smithi092.stdout: "rbd_mirror_image_policy_rebalance_timeout": "0.000000", 2024-03-11T21:27:25.331 INFO:teuthology.orchestra.run.smithi092.stdout: "rbd_mirror_image_policy_type": "simple", 2024-03-11T21:27:25.331 INFO:teuthology.orchestra.run.smithi092.stdout: "rbd_mirror_image_policy_update_throttle_interval": "1.000000", 2024-03-11T21:27:25.331 INFO:teuthology.orchestra.run.smithi092.stdout: "rbd_mirror_image_state_check_interval": "30", 2024-03-11T21:27:25.331 INFO:teuthology.orchestra.run.smithi092.stdout: "rbd_mirror_journal_commit_age": "5.000000", 2024-03-11T21:27:25.331 INFO:teuthology.orchestra.run.smithi092.stdout: "rbd_mirror_journal_poll_age": "5.000000", 2024-03-11T21:27:25.331 INFO:teuthology.orchestra.run.smithi092.stdout: "rbd_mirror_leader_heartbeat_interval": "5", 2024-03-11T21:27:25.331 INFO:teuthology.orchestra.run.smithi092.stdout: "rbd_mirror_leader_max_acquire_attempts_before_break": "3", 2024-03-11T21:27:25.331 INFO:teuthology.orchestra.run.smithi092.stdout: "rbd_mirror_leader_max_missed_heartbeats": "2", 2024-03-11T21:27:25.332 INFO:teuthology.orchestra.run.smithi092.stdout: "rbd_mirror_memory_autotune": "true", 2024-03-11T21:27:25.332 INFO:teuthology.orchestra.run.smithi092.stdout: "rbd_mirror_memory_base": "805306368", 2024-03-11T21:27:25.332 INFO:teuthology.orchestra.run.smithi092.stdout: "rbd_mirror_memory_cache_autotune_interval": "30.000000", 2024-03-11T21:27:25.332 INFO:teuthology.orchestra.run.smithi092.stdout: "rbd_mirror_memory_cache_min": "134217728", 2024-03-11T21:27:25.332 INFO:teuthology.orchestra.run.smithi092.stdout: "rbd_mirror_memory_cache_resize_interval": "5.000000", 2024-03-11T21:27:25.332 INFO:teuthology.orchestra.run.smithi092.stdout: "rbd_mirror_memory_expected_fragmentation": "0.150000", 2024-03-11T21:27:25.332 INFO:teuthology.orchestra.run.smithi092.stdout: "rbd_mirror_memory_target": "4294967296", 2024-03-11T21:27:25.332 INFO:teuthology.orchestra.run.smithi092.stdout: "rbd_mirror_perf_stats_prio": "5", 2024-03-11T21:27:25.332 INFO:teuthology.orchestra.run.smithi092.stdout: "rbd_mirror_pool_replayers_refresh_interval": "30", 2024-03-11T21:27:25.332 INFO:teuthology.orchestra.run.smithi092.stdout: "rbd_mirror_sync_point_update_age": "30.000000", 2024-03-11T21:27:25.332 INFO:teuthology.orchestra.run.smithi092.stdout: "rbd_mirroring_delete_delay": "0", 2024-03-11T21:27:25.332 INFO:teuthology.orchestra.run.smithi092.stdout: "rbd_mirroring_max_mirroring_snapshots": "5", 2024-03-11T21:27:25.332 INFO:teuthology.orchestra.run.smithi092.stdout: "rbd_mirroring_replay_delay": "0", 2024-03-11T21:27:25.333 INFO:teuthology.orchestra.run.smithi092.stdout: "rbd_mirroring_resync_after_disconnect": "false", 2024-03-11T21:27:25.333 INFO:teuthology.orchestra.run.smithi092.stdout: "rbd_move_parent_to_trash_on_remove": "false", 2024-03-11T21:27:25.333 INFO:teuthology.orchestra.run.smithi092.stdout: "rbd_move_to_trash_on_remove": "false", 2024-03-11T21:27:25.333 INFO:teuthology.orchestra.run.smithi092.stdout: "rbd_move_to_trash_on_remove_expire_seconds": "0", 2024-03-11T21:27:25.333 INFO:teuthology.orchestra.run.smithi092.stdout: "rbd_mtime_update_interval": "60", 2024-03-11T21:27:25.333 INFO:teuthology.orchestra.run.smithi092.stdout: "rbd_non_blocking_aio": "true", 2024-03-11T21:27:25.333 INFO:teuthology.orchestra.run.smithi092.stdout: "rbd_op_thread_timeout": "60", 2024-03-11T21:27:25.333 INFO:teuthology.orchestra.run.smithi092.stdout: "rbd_op_threads": "1", 2024-03-11T21:27:25.333 INFO:teuthology.orchestra.run.smithi092.stdout: "rbd_parent_cache_enabled": "false", 2024-03-11T21:27:25.333 INFO:teuthology.orchestra.run.smithi092.stdout: "rbd_persistent_cache_mode": "disabled", 2024-03-11T21:27:25.333 INFO:teuthology.orchestra.run.smithi092.stdout: "rbd_persistent_cache_path": "/tmp", 2024-03-11T21:27:25.333 INFO:teuthology.orchestra.run.smithi092.stdout: "rbd_persistent_cache_size": "1073741824", 2024-03-11T21:27:25.333 INFO:teuthology.orchestra.run.smithi092.stdout: "rbd_plugins": "", 2024-03-11T21:27:25.333 INFO:teuthology.orchestra.run.smithi092.stdout: "rbd_qos_bps_burst": "0", 2024-03-11T21:27:25.333 INFO:teuthology.orchestra.run.smithi092.stdout: "rbd_qos_bps_burst_seconds": "1", 2024-03-11T21:27:25.334 INFO:teuthology.orchestra.run.smithi092.stdout: "rbd_qos_bps_limit": "0", 2024-03-11T21:27:25.334 INFO:teuthology.orchestra.run.smithi092.stdout: "rbd_qos_exclude_ops": "0", 2024-03-11T21:27:25.334 INFO:teuthology.orchestra.run.smithi092.stdout: "rbd_qos_iops_burst": "0", 2024-03-11T21:27:25.334 INFO:teuthology.orchestra.run.smithi092.stdout: "rbd_qos_iops_burst_seconds": "1", 2024-03-11T21:27:25.334 INFO:teuthology.orchestra.run.smithi092.stdout: "rbd_qos_iops_limit": "0", 2024-03-11T21:27:25.334 INFO:teuthology.orchestra.run.smithi092.stdout: "rbd_qos_read_bps_burst": "0", 2024-03-11T21:27:25.334 INFO:teuthology.orchestra.run.smithi092.stdout: "rbd_qos_read_bps_burst_seconds": "1", 2024-03-11T21:27:25.334 INFO:teuthology.orchestra.run.smithi092.stdout: "rbd_qos_read_bps_limit": "0", 2024-03-11T21:27:25.334 INFO:teuthology.orchestra.run.smithi092.stdout: "rbd_qos_read_iops_burst": "0", 2024-03-11T21:27:25.334 INFO:teuthology.orchestra.run.smithi092.stdout: "rbd_qos_read_iops_burst_seconds": "1", 2024-03-11T21:27:25.334 INFO:teuthology.orchestra.run.smithi092.stdout: "rbd_qos_read_iops_limit": "0", 2024-03-11T21:27:25.334 INFO:teuthology.orchestra.run.smithi092.stdout: "rbd_qos_schedule_tick_min": "50", 2024-03-11T21:27:25.334 INFO:teuthology.orchestra.run.smithi092.stdout: "rbd_qos_write_bps_burst": "0", 2024-03-11T21:27:25.335 INFO:teuthology.orchestra.run.smithi092.stdout: "rbd_qos_write_bps_burst_seconds": "1", 2024-03-11T21:27:25.335 INFO:teuthology.orchestra.run.smithi092.stdout: "rbd_qos_write_bps_limit": "0", 2024-03-11T21:27:25.335 INFO:teuthology.orchestra.run.smithi092.stdout: "rbd_qos_write_iops_burst": "0", 2024-03-11T21:27:25.335 INFO:teuthology.orchestra.run.smithi092.stdout: "rbd_qos_write_iops_burst_seconds": "1", 2024-03-11T21:27:25.335 INFO:teuthology.orchestra.run.smithi092.stdout: "rbd_qos_write_iops_limit": "0", 2024-03-11T21:27:25.335 INFO:teuthology.orchestra.run.smithi092.stdout: "rbd_quiesce_notification_attempts": "10", 2024-03-11T21:27:25.335 INFO:teuthology.orchestra.run.smithi092.stdout: "rbd_read_from_replica_policy": "default", 2024-03-11T21:27:25.335 INFO:teuthology.orchestra.run.smithi092.stdout: "rbd_readahead_disable_after_bytes": "52428800", 2024-03-11T21:27:25.335 INFO:teuthology.orchestra.run.smithi092.stdout: "rbd_readahead_max_bytes": "524288", 2024-03-11T21:27:25.335 INFO:teuthology.orchestra.run.smithi092.stdout: "rbd_readahead_trigger_requests": "10", 2024-03-11T21:27:25.335 INFO:teuthology.orchestra.run.smithi092.stdout: "rbd_request_timed_out_seconds": "30", 2024-03-11T21:27:25.335 INFO:teuthology.orchestra.run.smithi092.stdout: "rbd_skip_partial_discard": "true", 2024-03-11T21:27:25.335 INFO:teuthology.orchestra.run.smithi092.stdout: "rbd_sparse_read_threshold_bytes": "65536", 2024-03-11T21:27:25.336 INFO:teuthology.orchestra.run.smithi092.stdout: "rbd_tracing": "false", 2024-03-11T21:27:25.336 INFO:teuthology.orchestra.run.smithi092.stdout: "rbd_validate_names": "true", 2024-03-11T21:27:25.336 INFO:teuthology.orchestra.run.smithi092.stdout: "rbd_validate_pool": "true", 2024-03-11T21:27:25.336 INFO:teuthology.orchestra.run.smithi092.stdout: "restapi_base_url": "", 2024-03-11T21:27:25.336 INFO:teuthology.orchestra.run.smithi092.stdout: "restapi_log_level": "", 2024-03-11T21:27:25.336 INFO:teuthology.orchestra.run.smithi092.stdout: "rgw_acl_grants_max_num": "100", 2024-03-11T21:27:25.336 INFO:teuthology.orchestra.run.smithi092.stdout: "rgw_admin_entry": "admin", 2024-03-11T21:27:25.336 INFO:teuthology.orchestra.run.smithi092.stdout: "rgw_allow_notification_secrets_in_cleartext": "false", 2024-03-11T21:27:25.336 INFO:teuthology.orchestra.run.smithi092.stdout: "rgw_backend_store": "rados", 2024-03-11T21:27:25.336 INFO:teuthology.orchestra.run.smithi092.stdout: "rgw_barbican_url": "", 2024-03-11T21:27:25.336 INFO:teuthology.orchestra.run.smithi092.stdout: "rgw_beast_enable_async": "true", 2024-03-11T21:27:25.336 INFO:teuthology.orchestra.run.smithi092.stdout: "rgw_bucket_counters_cache": "false", 2024-03-11T21:27:25.336 INFO:teuthology.orchestra.run.smithi092.stdout: "rgw_bucket_counters_cache_size": "10000", 2024-03-11T21:27:25.336 INFO:teuthology.orchestra.run.smithi092.stdout: "rgw_bucket_default_quota_max_objects": "-1", 2024-03-11T21:27:25.337 INFO:teuthology.orchestra.run.smithi092.stdout: "rgw_bucket_default_quota_max_size": "-1", 2024-03-11T21:27:25.337 INFO:teuthology.orchestra.run.smithi092.stdout: "rgw_bucket_index_max_aio": "128", 2024-03-11T21:27:25.337 INFO:teuthology.orchestra.run.smithi092.stdout: "rgw_bucket_index_transaction_instrumentation": "false", 2024-03-11T21:27:25.337 INFO:teuthology.orchestra.run.smithi092.stdout: "rgw_bucket_quota_cache_size": "10000", 2024-03-11T21:27:25.337 INFO:teuthology.orchestra.run.smithi092.stdout: "rgw_bucket_quota_ttl": "600", 2024-03-11T21:27:25.337 INFO:teuthology.orchestra.run.smithi092.stdout: "rgw_bucket_sync_spawn_window": "20", 2024-03-11T21:27:25.337 INFO:teuthology.orchestra.run.smithi092.stdout: "rgw_cache_enabled": "true", 2024-03-11T21:27:25.337 INFO:teuthology.orchestra.run.smithi092.stdout: "rgw_cache_expiry_interval": "900", 2024-03-11T21:27:25.337 INFO:teuthology.orchestra.run.smithi092.stdout: "rgw_cache_lru_size": "10000", 2024-03-11T21:27:25.337 INFO:teuthology.orchestra.run.smithi092.stdout: "rgw_config_store": "rados", 2024-03-11T21:27:25.337 INFO:teuthology.orchestra.run.smithi092.stdout: "rgw_content_length_compat": "false", 2024-03-11T21:27:25.337 INFO:teuthology.orchestra.run.smithi092.stdout: "rgw_copy_obj_progress": "true", 2024-03-11T21:27:25.337 INFO:teuthology.orchestra.run.smithi092.stdout: "rgw_copy_obj_progress_every_bytes": "1048576", 2024-03-11T21:27:25.337 INFO:teuthology.orchestra.run.smithi092.stdout: "rgw_cors_rules_max_num": "100", 2024-03-11T21:27:25.338 INFO:teuthology.orchestra.run.smithi092.stdout: "rgw_cross_domain_policy": "", 2024-03-11T21:27:25.338 INFO:teuthology.orchestra.run.smithi092.stdout: "rgw_crypt_default_encryption_key": "", 2024-03-11T21:27:25.338 INFO:teuthology.orchestra.run.smithi092.stdout: "rgw_crypt_kmip_addr": "", 2024-03-11T21:27:25.338 INFO:teuthology.orchestra.run.smithi092.stdout: "rgw_crypt_kmip_ca_path": "", 2024-03-11T21:27:25.338 INFO:teuthology.orchestra.run.smithi092.stdout: "rgw_crypt_kmip_client_cert": "", 2024-03-11T21:27:25.338 INFO:teuthology.orchestra.run.smithi092.stdout: "rgw_crypt_kmip_client_key": "", 2024-03-11T21:27:25.338 INFO:teuthology.orchestra.run.smithi092.stdout: "rgw_crypt_kmip_kms_key_template": "", 2024-03-11T21:27:25.338 INFO:teuthology.orchestra.run.smithi092.stdout: "rgw_crypt_kmip_password": "", 2024-03-11T21:27:25.338 INFO:teuthology.orchestra.run.smithi092.stdout: "rgw_crypt_kmip_s3_key_template": "$keyid", 2024-03-11T21:27:25.338 INFO:teuthology.orchestra.run.smithi092.stdout: "rgw_crypt_kmip_username": "", 2024-03-11T21:27:25.338 INFO:teuthology.orchestra.run.smithi092.stdout: "rgw_crypt_require_ssl": "true", 2024-03-11T21:27:25.338 INFO:teuthology.orchestra.run.smithi092.stdout: "rgw_crypt_s3_kms_backend": "barbican", 2024-03-11T21:27:25.338 INFO:teuthology.orchestra.run.smithi092.stdout: "rgw_crypt_s3_kms_encryption_keys": "", 2024-03-11T21:27:25.339 INFO:teuthology.orchestra.run.smithi092.stdout: "rgw_crypt_sse_s3_backend": "vault", 2024-03-11T21:27:25.339 INFO:teuthology.orchestra.run.smithi092.stdout: "rgw_crypt_sse_s3_key_template": "%bucket_id", 2024-03-11T21:27:25.339 INFO:teuthology.orchestra.run.smithi092.stdout: "rgw_crypt_sse_s3_vault_addr": "", 2024-03-11T21:27:25.339 INFO:teuthology.orchestra.run.smithi092.stdout: "rgw_crypt_sse_s3_vault_auth": "token", 2024-03-11T21:27:25.339 INFO:teuthology.orchestra.run.smithi092.stdout: "rgw_crypt_sse_s3_vault_namespace": "", 2024-03-11T21:27:25.339 INFO:teuthology.orchestra.run.smithi092.stdout: "rgw_crypt_sse_s3_vault_prefix": "", 2024-03-11T21:27:25.339 INFO:teuthology.orchestra.run.smithi092.stdout: "rgw_crypt_sse_s3_vault_secret_engine": "transit", 2024-03-11T21:27:25.339 INFO:teuthology.orchestra.run.smithi092.stdout: "rgw_crypt_sse_s3_vault_ssl_cacert": "", 2024-03-11T21:27:25.339 INFO:teuthology.orchestra.run.smithi092.stdout: "rgw_crypt_sse_s3_vault_ssl_clientcert": "", 2024-03-11T21:27:25.339 INFO:teuthology.orchestra.run.smithi092.stdout: "rgw_crypt_sse_s3_vault_ssl_clientkey": "", 2024-03-11T21:27:25.339 INFO:teuthology.orchestra.run.smithi092.stdout: "rgw_crypt_sse_s3_vault_token_file": "", 2024-03-11T21:27:25.339 INFO:teuthology.orchestra.run.smithi092.stdout: "rgw_crypt_sse_s3_vault_verify_ssl": "true", 2024-03-11T21:27:25.339 INFO:teuthology.orchestra.run.smithi092.stdout: "rgw_crypt_suppress_logs": "true", 2024-03-11T21:27:25.339 INFO:teuthology.orchestra.run.smithi092.stdout: "rgw_crypt_vault_addr": "", 2024-03-11T21:27:25.339 INFO:teuthology.orchestra.run.smithi092.stdout: "rgw_crypt_vault_auth": "token", 2024-03-11T21:27:25.340 INFO:teuthology.orchestra.run.smithi092.stdout: "rgw_crypt_vault_namespace": "", 2024-03-11T21:27:25.340 INFO:teuthology.orchestra.run.smithi092.stdout: "rgw_crypt_vault_prefix": "", 2024-03-11T21:27:25.340 INFO:teuthology.orchestra.run.smithi092.stdout: "rgw_crypt_vault_secret_engine": "transit", 2024-03-11T21:27:25.340 INFO:teuthology.orchestra.run.smithi092.stdout: "rgw_crypt_vault_ssl_cacert": "", 2024-03-11T21:27:25.340 INFO:teuthology.orchestra.run.smithi092.stdout: "rgw_crypt_vault_ssl_clientcert": "", 2024-03-11T21:27:25.340 INFO:teuthology.orchestra.run.smithi092.stdout: "rgw_crypt_vault_ssl_clientkey": "", 2024-03-11T21:27:25.340 INFO:teuthology.orchestra.run.smithi092.stdout: "rgw_crypt_vault_token_file": "", 2024-03-11T21:27:25.340 INFO:teuthology.orchestra.run.smithi092.stdout: "rgw_crypt_vault_verify_ssl": "true", 2024-03-11T21:27:25.340 INFO:teuthology.orchestra.run.smithi092.stdout: "rgw_curl_buffersize": "524288", 2024-03-11T21:27:25.340 INFO:teuthology.orchestra.run.smithi092.stdout: "rgw_curl_low_speed_limit": "1024", 2024-03-11T21:27:25.340 INFO:teuthology.orchestra.run.smithi092.stdout: "rgw_curl_low_speed_time": "300", 2024-03-11T21:27:25.340 INFO:teuthology.orchestra.run.smithi092.stdout: "rgw_curl_tcp_keepalive": "0", 2024-03-11T21:27:25.340 INFO:teuthology.orchestra.run.smithi092.stdout: "rgw_curl_wait_timeout_ms": "1000", 2024-03-11T21:27:25.340 INFO:teuthology.orchestra.run.smithi092.stdout: "rgw_d3n_l1_datacache_persistent_path": "/tmp/rgw_datacache/", 2024-03-11T21:27:25.341 INFO:teuthology.orchestra.run.smithi092.stdout: "rgw_d3n_l1_datacache_size": "1073741824", 2024-03-11T21:27:25.341 INFO:teuthology.orchestra.run.smithi092.stdout: "rgw_d3n_l1_evict_cache_on_start": "true", 2024-03-11T21:27:25.341 INFO:teuthology.orchestra.run.smithi092.stdout: "rgw_d3n_l1_eviction_policy": "lru", 2024-03-11T21:27:25.341 INFO:teuthology.orchestra.run.smithi092.stdout: "rgw_d3n_l1_fadvise": "4", 2024-03-11T21:27:25.341 INFO:teuthology.orchestra.run.smithi092.stdout: "rgw_d3n_l1_local_datacache_enabled": "false", 2024-03-11T21:27:25.341 INFO:teuthology.orchestra.run.smithi092.stdout: "rgw_d3n_libaio_aio_num": "64", 2024-03-11T21:27:25.341 INFO:teuthology.orchestra.run.smithi092.stdout: "rgw_d3n_libaio_aio_threads": "20", 2024-03-11T21:27:25.341 INFO:teuthology.orchestra.run.smithi092.stdout: "rgw_d4n_host": "127.0.0.1", 2024-03-11T21:27:25.341 INFO:teuthology.orchestra.run.smithi092.stdout: "rgw_d4n_port": "6379", 2024-03-11T21:27:25.341 INFO:teuthology.orchestra.run.smithi092.stdout: "rgw_data": "/var/lib/ceph/radosgw/ceph-a", 2024-03-11T21:27:25.341 INFO:teuthology.orchestra.run.smithi092.stdout: "rgw_data_log_changes_size": "1000", 2024-03-11T21:27:25.341 INFO:teuthology.orchestra.run.smithi092.stdout: "rgw_data_log_num_shards": "128", 2024-03-11T21:27:25.341 INFO:teuthology.orchestra.run.smithi092.stdout: "rgw_data_log_obj_prefix": "data_log", 2024-03-11T21:27:25.341 INFO:teuthology.orchestra.run.smithi092.stdout: "rgw_data_log_window": "30", 2024-03-11T21:27:25.342 INFO:teuthology.orchestra.run.smithi092.stdout: "rgw_data_notify_interval_msec": "0", 2024-03-11T21:27:25.342 INFO:teuthology.orchestra.run.smithi092.stdout: "rgw_data_sync_poll_interval": "20", 2024-03-11T21:27:25.342 INFO:teuthology.orchestra.run.smithi092.stdout: "rgw_data_sync_spawn_window": "20", 2024-03-11T21:27:25.342 INFO:teuthology.orchestra.run.smithi092.stdout: "rgw_debug_inject_latency_bi_unlink": "0", 2024-03-11T21:27:25.342 INFO:teuthology.orchestra.run.smithi092.stdout: "rgw_debug_inject_olh_cancel_modification_err": "false", 2024-03-11T21:27:25.342 INFO:teuthology.orchestra.run.smithi092.stdout: "rgw_debug_inject_set_olh_err": "0", 2024-03-11T21:27:25.342 INFO:teuthology.orchestra.run.smithi092.stdout: "rgw_default_data_log_backing": "fifo", 2024-03-11T21:27:25.342 INFO:teuthology.orchestra.run.smithi092.stdout: "rgw_default_realm_info_oid": "default.realm", 2024-03-11T21:27:25.342 INFO:teuthology.orchestra.run.smithi092.stdout: "rgw_default_region_info_oid": "default.region", 2024-03-11T21:27:25.342 INFO:teuthology.orchestra.run.smithi092.stdout: "rgw_default_zone_info_oid": "default.zone", 2024-03-11T21:27:25.342 INFO:teuthology.orchestra.run.smithi092.stdout: "rgw_default_zonegroup_info_oid": "default.zonegroup", 2024-03-11T21:27:25.342 INFO:teuthology.orchestra.run.smithi092.stdout: "rgw_defer_to_bucket_acls": "", 2024-03-11T21:27:25.342 INFO:teuthology.orchestra.run.smithi092.stdout: "rgw_delete_multi_obj_max_num": "1000", 2024-03-11T21:27:25.343 INFO:teuthology.orchestra.run.smithi092.stdout: "rgw_dmclock_admin_lim": "0.000000", 2024-03-11T21:27:25.343 INFO:teuthology.orchestra.run.smithi092.stdout: "rgw_dmclock_admin_res": "100.000000", 2024-03-11T21:27:25.343 INFO:teuthology.orchestra.run.smithi092.stdout: "rgw_dmclock_admin_wgt": "100.000000", 2024-03-11T21:27:25.343 INFO:teuthology.orchestra.run.smithi092.stdout: "rgw_dmclock_auth_lim": "0.000000", 2024-03-11T21:27:25.343 INFO:teuthology.orchestra.run.smithi092.stdout: "rgw_dmclock_auth_res": "200.000000", 2024-03-11T21:27:25.343 INFO:teuthology.orchestra.run.smithi092.stdout: "rgw_dmclock_auth_wgt": "100.000000", 2024-03-11T21:27:25.343 INFO:teuthology.orchestra.run.smithi092.stdout: "rgw_dmclock_data_lim": "0.000000", 2024-03-11T21:27:25.343 INFO:teuthology.orchestra.run.smithi092.stdout: "rgw_dmclock_data_res": "500.000000", 2024-03-11T21:27:25.343 INFO:teuthology.orchestra.run.smithi092.stdout: "rgw_dmclock_data_wgt": "500.000000", 2024-03-11T21:27:25.343 INFO:teuthology.orchestra.run.smithi092.stdout: "rgw_dmclock_metadata_lim": "0.000000", 2024-03-11T21:27:25.343 INFO:teuthology.orchestra.run.smithi092.stdout: "rgw_dmclock_metadata_res": "500.000000", 2024-03-11T21:27:25.343 INFO:teuthology.orchestra.run.smithi092.stdout: "rgw_dmclock_metadata_wgt": "500.000000", 2024-03-11T21:27:25.343 INFO:teuthology.orchestra.run.smithi092.stdout: "rgw_dns_name": "", 2024-03-11T21:27:25.343 INFO:teuthology.orchestra.run.smithi092.stdout: "rgw_dns_s3website_name": "", 2024-03-11T21:27:25.343 INFO:teuthology.orchestra.run.smithi092.stdout: "rgw_dynamic_resharding": "true", 2024-03-11T21:27:25.344 INFO:teuthology.orchestra.run.smithi092.stdout: "rgw_enable_apis": "s3, s3website, swift, swift_auth, admin, sts, iam, notifications", 2024-03-11T21:27:25.344 INFO:teuthology.orchestra.run.smithi092.stdout: "rgw_enable_gc_threads": "true", 2024-03-11T21:27:25.344 INFO:teuthology.orchestra.run.smithi092.stdout: "rgw_enable_lc_threads": "true", 2024-03-11T21:27:25.344 INFO:teuthology.orchestra.run.smithi092.stdout: "rgw_enable_ops_log": "false", 2024-03-11T21:27:25.344 INFO:teuthology.orchestra.run.smithi092.stdout: "rgw_enable_quota_threads": "true", 2024-03-11T21:27:25.344 INFO:teuthology.orchestra.run.smithi092.stdout: "rgw_enable_static_website": "false", 2024-03-11T21:27:25.344 INFO:teuthology.orchestra.run.smithi092.stdout: "rgw_enable_usage_log": "false", 2024-03-11T21:27:25.344 INFO:teuthology.orchestra.run.smithi092.stdout: "rgw_enforce_swift_acls": "true", 2024-03-11T21:27:25.344 INFO:teuthology.orchestra.run.smithi092.stdout: "rgw_exit_timeout_secs": "120", 2024-03-11T21:27:25.344 INFO:teuthology.orchestra.run.smithi092.stdout: "rgw_expose_bucket": "false", 2024-03-11T21:27:25.344 INFO:teuthology.orchestra.run.smithi092.stdout: "rgw_extended_http_attrs": "", 2024-03-11T21:27:25.344 INFO:teuthology.orchestra.run.smithi092.stdout: "rgw_filter": "none", 2024-03-11T21:27:25.344 INFO:teuthology.orchestra.run.smithi092.stdout: "rgw_frontend_defaults": "beast ssl_certificate=config://rgw/cert/$realm/$zone.crt ssl_private_key=config://rgw/cert/$realm/$zone.key", 2024-03-11T21:27:25.344 INFO:teuthology.orchestra.run.smithi092.stdout: "rgw_frontends": "beast port=7480", 2024-03-11T21:27:25.345 INFO:teuthology.orchestra.run.smithi092.stdout: "rgw_gc_max_concurrent_io": "10", 2024-03-11T21:27:25.345 INFO:teuthology.orchestra.run.smithi092.stdout: "rgw_gc_max_deferred": "50", 2024-03-11T21:27:25.345 INFO:teuthology.orchestra.run.smithi092.stdout: "rgw_gc_max_deferred_entries_size": "3072", 2024-03-11T21:27:25.345 INFO:teuthology.orchestra.run.smithi092.stdout: "rgw_gc_max_objs": "32", 2024-03-11T21:27:25.345 INFO:teuthology.orchestra.run.smithi092.stdout: "rgw_gc_max_queue_size": "134213632", 2024-03-11T21:27:25.345 INFO:teuthology.orchestra.run.smithi092.stdout: "rgw_gc_max_trim_chunk": "16", 2024-03-11T21:27:25.345 INFO:teuthology.orchestra.run.smithi092.stdout: "rgw_gc_obj_min_wait": "7200", 2024-03-11T21:27:25.345 INFO:teuthology.orchestra.run.smithi092.stdout: "rgw_gc_processor_max_time": "3600", 2024-03-11T21:27:25.345 INFO:teuthology.orchestra.run.smithi092.stdout: "rgw_gc_processor_period": "3600", 2024-03-11T21:27:25.345 INFO:teuthology.orchestra.run.smithi092.stdout: "rgw_get_obj_max_req_size": "4194304", 2024-03-11T21:27:25.345 INFO:teuthology.orchestra.run.smithi092.stdout: "rgw_get_obj_window_size": "16777216", 2024-03-11T21:27:25.345 INFO:teuthology.orchestra.run.smithi092.stdout: "rgw_healthcheck_disabling_path": "", 2024-03-11T21:27:25.345 INFO:teuthology.orchestra.run.smithi092.stdout: "rgw_ignore_get_invalid_range": "false", 2024-03-11T21:27:25.345 INFO:teuthology.orchestra.run.smithi092.stdout: "rgw_init_timeout": "300", 2024-03-11T21:27:25.346 INFO:teuthology.orchestra.run.smithi092.stdout: "rgw_inject_notify_timeout_probability": "0.000000", 2024-03-11T21:27:25.346 INFO:teuthology.orchestra.run.smithi092.stdout: "rgw_json_config": "/var/lib/ceph/radosgw/config.json", 2024-03-11T21:27:25.346 INFO:teuthology.orchestra.run.smithi092.stdout: "rgw_kafka_connection_idle": "30", 2024-03-11T21:27:25.346 INFO:teuthology.orchestra.run.smithi092.stdout: "rgw_kafka_sleep_timeout": "10", 2024-03-11T21:27:25.346 INFO:teuthology.orchestra.run.smithi092.stdout: "rgw_keystone_accepted_admin_roles": "", 2024-03-11T21:27:25.346 INFO:teuthology.orchestra.run.smithi092.stdout: "rgw_keystone_accepted_reader_roles": "", 2024-03-11T21:27:25.346 INFO:teuthology.orchestra.run.smithi092.stdout: "rgw_keystone_accepted_roles": "Member, admin", 2024-03-11T21:27:25.346 INFO:teuthology.orchestra.run.smithi092.stdout: "rgw_keystone_admin_domain": "", 2024-03-11T21:27:25.346 INFO:teuthology.orchestra.run.smithi092.stdout: "rgw_keystone_admin_password": "", 2024-03-11T21:27:25.346 INFO:teuthology.orchestra.run.smithi092.stdout: "rgw_keystone_admin_password_path": "", 2024-03-11T21:27:25.346 INFO:teuthology.orchestra.run.smithi092.stdout: "rgw_keystone_admin_project": "", 2024-03-11T21:27:25.346 INFO:teuthology.orchestra.run.smithi092.stdout: "rgw_keystone_admin_tenant": "", 2024-03-11T21:27:25.346 INFO:teuthology.orchestra.run.smithi092.stdout: "rgw_keystone_admin_token": "", 2024-03-11T21:27:25.347 INFO:teuthology.orchestra.run.smithi092.stdout: "rgw_keystone_admin_token_path": "", 2024-03-11T21:27:25.347 INFO:teuthology.orchestra.run.smithi092.stdout: "rgw_keystone_admin_user": "", 2024-03-11T21:27:25.347 INFO:teuthology.orchestra.run.smithi092.stdout: "rgw_keystone_api_version": "2", 2024-03-11T21:27:25.347 INFO:teuthology.orchestra.run.smithi092.stdout: "rgw_keystone_barbican_domain": "", 2024-03-11T21:27:25.347 INFO:teuthology.orchestra.run.smithi092.stdout: "rgw_keystone_barbican_password": "", 2024-03-11T21:27:25.347 INFO:teuthology.orchestra.run.smithi092.stdout: "rgw_keystone_barbican_project": "", 2024-03-11T21:27:25.347 INFO:teuthology.orchestra.run.smithi092.stdout: "rgw_keystone_barbican_tenant": "", 2024-03-11T21:27:25.347 INFO:teuthology.orchestra.run.smithi092.stdout: "rgw_keystone_barbican_user": "", 2024-03-11T21:27:25.347 INFO:teuthology.orchestra.run.smithi092.stdout: "rgw_keystone_expired_token_cache_expiration": "3600", 2024-03-11T21:27:25.347 INFO:teuthology.orchestra.run.smithi092.stdout: "rgw_keystone_implicit_tenants": "false", 2024-03-11T21:27:25.347 INFO:teuthology.orchestra.run.smithi092.stdout: "rgw_keystone_service_token_accepted_roles": "admin", 2024-03-11T21:27:25.347 INFO:teuthology.orchestra.run.smithi092.stdout: "rgw_keystone_service_token_enabled": "false", 2024-03-11T21:27:25.347 INFO:teuthology.orchestra.run.smithi092.stdout: "rgw_keystone_token_cache_size": "10000", 2024-03-11T21:27:25.347 INFO:teuthology.orchestra.run.smithi092.stdout: "rgw_keystone_url": "", 2024-03-11T21:27:25.347 INFO:teuthology.orchestra.run.smithi092.stdout: "rgw_keystone_verify_ssl": "true", 2024-03-11T21:27:25.348 INFO:teuthology.orchestra.run.smithi092.stdout: "rgw_lc_debug_interval": "-1", 2024-03-11T21:27:25.348 INFO:teuthology.orchestra.run.smithi092.stdout: "rgw_lc_lock_max_time": "90", 2024-03-11T21:27:25.348 INFO:teuthology.orchestra.run.smithi092.stdout: "rgw_lc_max_objs": "32", 2024-03-11T21:27:25.348 INFO:teuthology.orchestra.run.smithi092.stdout: "rgw_lc_max_rules": "1000", 2024-03-11T21:27:25.348 INFO:teuthology.orchestra.run.smithi092.stdout: "rgw_lc_max_worker": "3", 2024-03-11T21:27:25.348 INFO:teuthology.orchestra.run.smithi092.stdout: "rgw_lc_max_wp_worker": "3", 2024-03-11T21:27:25.348 INFO:teuthology.orchestra.run.smithi092.stdout: "rgw_lc_thread_delay": "0", 2024-03-11T21:27:25.348 INFO:teuthology.orchestra.run.smithi092.stdout: "rgw_ldap_binddn": "uid=admin,cn=users,dc=example,dc=com", 2024-03-11T21:27:25.348 INFO:teuthology.orchestra.run.smithi092.stdout: "rgw_ldap_dnattr": "uid", 2024-03-11T21:27:25.348 INFO:teuthology.orchestra.run.smithi092.stdout: "rgw_ldap_searchdn": "cn=users,cn=accounts,dc=example,dc=com", 2024-03-11T21:27:25.348 INFO:teuthology.orchestra.run.smithi092.stdout: "rgw_ldap_searchfilter": "", 2024-03-11T21:27:25.348 INFO:teuthology.orchestra.run.smithi092.stdout: "rgw_ldap_secret": "/etc/openldap/secret", 2024-03-11T21:27:25.348 INFO:teuthology.orchestra.run.smithi092.stdout: "rgw_ldap_uri": "ldaps://", 2024-03-11T21:27:25.348 INFO:teuthology.orchestra.run.smithi092.stdout: "rgw_lifecycle_work_time": "00:00-06:00", 2024-03-11T21:27:25.349 INFO:teuthology.orchestra.run.smithi092.stdout: "rgw_list_bucket_min_readahead": "1000", 2024-03-11T21:27:25.349 INFO:teuthology.orchestra.run.smithi092.stdout: "rgw_list_buckets_max_chunk": "1000", 2024-03-11T21:27:25.349 INFO:teuthology.orchestra.run.smithi092.stdout: "rgw_log_http_headers": "", 2024-03-11T21:27:25.349 INFO:teuthology.orchestra.run.smithi092.stdout: "rgw_log_nonexistent_bucket": "false", 2024-03-11T21:27:25.349 INFO:teuthology.orchestra.run.smithi092.stdout: "rgw_log_object_name": "%Y-%m-%d-%H-%i-%n", 2024-03-11T21:27:25.349 INFO:teuthology.orchestra.run.smithi092.stdout: "rgw_log_object_name_utc": "false", 2024-03-11T21:27:25.349 INFO:teuthology.orchestra.run.smithi092.stdout: "rgw_lua_max_memory_per_state": "128000", 2024-03-11T21:27:25.349 INFO:teuthology.orchestra.run.smithi092.stdout: "rgw_luarocks_location": "/tmp/rgw_luarocks/mon.a", 2024-03-11T21:27:25.349 INFO:teuthology.orchestra.run.smithi092.stdout: "rgw_max_attr_name_len": "0", 2024-03-11T21:27:25.349 INFO:teuthology.orchestra.run.smithi092.stdout: "rgw_max_attr_size": "0", 2024-03-11T21:27:25.349 INFO:teuthology.orchestra.run.smithi092.stdout: "rgw_max_attrs_num_in_req": "0", 2024-03-11T21:27:25.349 INFO:teuthology.orchestra.run.smithi092.stdout: "rgw_max_chunk_size": "4194304", 2024-03-11T21:27:25.349 INFO:teuthology.orchestra.run.smithi092.stdout: "rgw_max_concurrent_requests": "1024", 2024-03-11T21:27:25.349 INFO:teuthology.orchestra.run.smithi092.stdout: "rgw_max_copy_obj_concurrent_io": "10", 2024-03-11T21:27:25.349 INFO:teuthology.orchestra.run.smithi092.stdout: "rgw_max_dynamic_shards": "1999", 2024-03-11T21:27:25.350 INFO:teuthology.orchestra.run.smithi092.stdout: "rgw_max_listing_results": "1000", 2024-03-11T21:27:25.350 INFO:teuthology.orchestra.run.smithi092.stdout: "rgw_max_notify_retries": "10", 2024-03-11T21:27:25.350 INFO:teuthology.orchestra.run.smithi092.stdout: "rgw_max_objs_per_shard": "100000", 2024-03-11T21:27:25.350 INFO:teuthology.orchestra.run.smithi092.stdout: "rgw_max_put_param_size": "1048576", 2024-03-11T21:27:25.350 INFO:teuthology.orchestra.run.smithi092.stdout: "rgw_max_put_size": "5368709120", 2024-03-11T21:27:25.350 INFO:teuthology.orchestra.run.smithi092.stdout: "rgw_max_slo_entries": "1000", 2024-03-11T21:27:25.350 INFO:teuthology.orchestra.run.smithi092.stdout: "rgw_md_log_max_shards": "64", 2024-03-11T21:27:25.350 INFO:teuthology.orchestra.run.smithi092.stdout: "rgw_md_notify_interval_msec": "200", 2024-03-11T21:27:25.350 INFO:teuthology.orchestra.run.smithi092.stdout: "rgw_meta_sync_poll_interval": "20", 2024-03-11T21:27:25.350 INFO:teuthology.orchestra.run.smithi092.stdout: "rgw_meta_sync_spawn_window": "20", 2024-03-11T21:27:25.350 INFO:teuthology.orchestra.run.smithi092.stdout: "rgw_mime_types_file": "/etc/mime.types", 2024-03-11T21:27:25.350 INFO:teuthology.orchestra.run.smithi092.stdout: "rgw_mp_lock_max_time": "600", 2024-03-11T21:27:25.350 INFO:teuthology.orchestra.run.smithi092.stdout: "rgw_multi_obj_del_max_aio": "16", 2024-03-11T21:27:25.351 INFO:teuthology.orchestra.run.smithi092.stdout: "rgw_multipart_min_part_size": "5242880", 2024-03-11T21:27:25.351 INFO:teuthology.orchestra.run.smithi092.stdout: "rgw_multipart_part_upload_limit": "10000", 2024-03-11T21:27:25.351 INFO:teuthology.orchestra.run.smithi092.stdout: "rgw_nfs_fhcache_partitions": "3", 2024-03-11T21:27:25.351 INFO:teuthology.orchestra.run.smithi092.stdout: "rgw_nfs_fhcache_size": "2017", 2024-03-11T21:27:25.351 INFO:teuthology.orchestra.run.smithi092.stdout: "rgw_nfs_frontends": "rgw-nfs", 2024-03-11T21:27:25.351 INFO:teuthology.orchestra.run.smithi092.stdout: "rgw_nfs_lru_lane_hiwat": "911", 2024-03-11T21:27:25.351 INFO:teuthology.orchestra.run.smithi092.stdout: "rgw_nfs_lru_lanes": "5", 2024-03-11T21:27:25.351 INFO:teuthology.orchestra.run.smithi092.stdout: "rgw_nfs_max_gc": "300", 2024-03-11T21:27:25.351 INFO:teuthology.orchestra.run.smithi092.stdout: "rgw_nfs_namespace_expire_secs": "300", 2024-03-11T21:27:25.351 INFO:teuthology.orchestra.run.smithi092.stdout: "rgw_nfs_run_gc_threads": "false", 2024-03-11T21:27:25.351 INFO:teuthology.orchestra.run.smithi092.stdout: "rgw_nfs_run_lc_threads": "false", 2024-03-11T21:27:25.351 INFO:teuthology.orchestra.run.smithi092.stdout: "rgw_nfs_run_quota_threads": "false", 2024-03-11T21:27:25.351 INFO:teuthology.orchestra.run.smithi092.stdout: "rgw_nfs_run_sync_thread": "false", 2024-03-11T21:27:25.352 INFO:teuthology.orchestra.run.smithi092.stdout: "rgw_nfs_s3_fast_attrs": "false", 2024-03-11T21:27:25.352 INFO:teuthology.orchestra.run.smithi092.stdout: "rgw_nfs_write_completion_interval_s": "10", 2024-03-11T21:27:25.352 INFO:teuthology.orchestra.run.smithi092.stdout: "rgw_num_async_rados_threads": "32", 2024-03-11T21:27:25.352 INFO:teuthology.orchestra.run.smithi092.stdout: "rgw_num_control_oids": "8", 2024-03-11T21:27:25.352 INFO:teuthology.orchestra.run.smithi092.stdout: "rgw_numa_node": "-1", 2024-03-11T21:27:25.352 INFO:teuthology.orchestra.run.smithi092.stdout: "rgw_obj_stripe_size": "4194304", 2024-03-11T21:27:25.352 INFO:teuthology.orchestra.run.smithi092.stdout: "rgw_obj_tombstone_cache_size": "1000", 2024-03-11T21:27:25.352 INFO:teuthology.orchestra.run.smithi092.stdout: "rgw_objexp_chunk_size": "100", 2024-03-11T21:27:25.352 INFO:teuthology.orchestra.run.smithi092.stdout: "rgw_objexp_gc_interval": "600", 2024-03-11T21:27:25.352 INFO:teuthology.orchestra.run.smithi092.stdout: "rgw_objexp_hints_num_shards": "127", 2024-03-11T21:27:25.352 INFO:teuthology.orchestra.run.smithi092.stdout: "rgw_olh_pending_timeout_sec": "3600", 2024-03-11T21:27:25.352 INFO:teuthology.orchestra.run.smithi092.stdout: "rgw_op_thread_suicide_timeout": "0", 2024-03-11T21:27:25.352 INFO:teuthology.orchestra.run.smithi092.stdout: "rgw_op_thread_timeout": "600", 2024-03-11T21:27:25.352 INFO:teuthology.orchestra.run.smithi092.stdout: "rgw_op_tracing": "false", 2024-03-11T21:27:25.352 INFO:teuthology.orchestra.run.smithi092.stdout: "rgw_opa_token": "", 2024-03-11T21:27:25.353 INFO:teuthology.orchestra.run.smithi092.stdout: "rgw_opa_url": "", 2024-03-11T21:27:25.353 INFO:teuthology.orchestra.run.smithi092.stdout: "rgw_opa_verify_ssl": "true", 2024-03-11T21:27:25.353 INFO:teuthology.orchestra.run.smithi092.stdout: "rgw_ops_log_data_backlog": "5242880", 2024-03-11T21:27:25.353 INFO:teuthology.orchestra.run.smithi092.stdout: "rgw_ops_log_file_path": "/var/log/ceph/ops-log-ceph-mon.a.log", 2024-03-11T21:27:25.353 INFO:teuthology.orchestra.run.smithi092.stdout: "rgw_ops_log_rados": "false", 2024-03-11T21:27:25.353 INFO:teuthology.orchestra.run.smithi092.stdout: "rgw_ops_log_socket_path": "", 2024-03-11T21:27:25.353 INFO:teuthology.orchestra.run.smithi092.stdout: "rgw_override_bucket_index_max_shards": "0", 2024-03-11T21:27:25.353 INFO:teuthology.orchestra.run.smithi092.stdout: "rgw_pending_bucket_index_op_expiration": "120", 2024-03-11T21:27:25.353 INFO:teuthology.orchestra.run.smithi092.stdout: "rgw_period_latest_epoch_info_oid": ".latest_epoch", 2024-03-11T21:27:25.353 INFO:teuthology.orchestra.run.smithi092.stdout: "rgw_period_push_interval": "2.000000", 2024-03-11T21:27:25.353 INFO:teuthology.orchestra.run.smithi092.stdout: "rgw_period_push_interval_max": "30.000000", 2024-03-11T21:27:25.353 INFO:teuthology.orchestra.run.smithi092.stdout: "rgw_period_root_pool": ".rgw.root", 2024-03-11T21:27:25.353 INFO:teuthology.orchestra.run.smithi092.stdout: "rgw_policy_reject_invalid_principals": "true", 2024-03-11T21:27:25.353 INFO:teuthology.orchestra.run.smithi092.stdout: "rgw_posix_base_path": "/tmp/rgw_posix_driver", 2024-03-11T21:27:25.354 INFO:teuthology.orchestra.run.smithi092.stdout: "rgw_posix_cache_lanes": "3", 2024-03-11T21:27:25.354 INFO:teuthology.orchestra.run.smithi092.stdout: "rgw_posix_cache_lmdb_count": "3", 2024-03-11T21:27:25.354 INFO:teuthology.orchestra.run.smithi092.stdout: "rgw_posix_cache_max_buckets": "100", 2024-03-11T21:27:25.354 INFO:teuthology.orchestra.run.smithi092.stdout: "rgw_posix_cache_partitions": "3", 2024-03-11T21:27:25.354 INFO:teuthology.orchestra.run.smithi092.stdout: "rgw_posix_database_root": "/var/lib/ceph/radosgw", 2024-03-11T21:27:25.354 INFO:teuthology.orchestra.run.smithi092.stdout: "rgw_print_continue": "true", 2024-03-11T21:27:25.354 INFO:teuthology.orchestra.run.smithi092.stdout: "rgw_print_prohibited_content_length": "false", 2024-03-11T21:27:25.354 INFO:teuthology.orchestra.run.smithi092.stdout: "rgw_put_obj_max_window_size": "67108864", 2024-03-11T21:27:25.354 INFO:teuthology.orchestra.run.smithi092.stdout: "rgw_put_obj_min_window_size": "16777216", 2024-03-11T21:27:25.354 INFO:teuthology.orchestra.run.smithi092.stdout: "rgw_rados_pool_autoscale_bias": "4.000000", 2024-03-11T21:27:25.354 INFO:teuthology.orchestra.run.smithi092.stdout: "rgw_rados_pool_recovery_priority": "5", 2024-03-11T21:27:25.354 INFO:teuthology.orchestra.run.smithi092.stdout: "rgw_rados_tracing": "false", 2024-03-11T21:27:25.354 INFO:teuthology.orchestra.run.smithi092.stdout: "rgw_realm": "", 2024-03-11T21:27:25.355 INFO:teuthology.orchestra.run.smithi092.stdout: "rgw_realm_id": "", 2024-03-11T21:27:25.355 INFO:teuthology.orchestra.run.smithi092.stdout: "rgw_realm_root_pool": ".rgw.root", 2024-03-11T21:27:25.355 INFO:teuthology.orchestra.run.smithi092.stdout: "rgw_region": "", 2024-03-11T21:27:25.355 INFO:teuthology.orchestra.run.smithi092.stdout: "rgw_region_root_pool": ".rgw.root", 2024-03-11T21:27:25.355 INFO:teuthology.orchestra.run.smithi092.stdout: "rgw_relaxed_region_enforcement": "false", 2024-03-11T21:27:25.355 INFO:teuthology.orchestra.run.smithi092.stdout: "rgw_relaxed_s3_bucket_names": "false", 2024-03-11T21:27:25.355 INFO:teuthology.orchestra.run.smithi092.stdout: "rgw_remote_addr_param": "REMOTE_ADDR", 2024-03-11T21:27:25.355 INFO:teuthology.orchestra.run.smithi092.stdout: "rgw_request_uri": "", 2024-03-11T21:27:25.355 INFO:teuthology.orchestra.run.smithi092.stdout: "rgw_reshard_batch_size": "64", 2024-03-11T21:27:25.355 INFO:teuthology.orchestra.run.smithi092.stdout: "rgw_reshard_bucket_lock_duration": "360", 2024-03-11T21:27:25.355 INFO:teuthology.orchestra.run.smithi092.stdout: "rgw_reshard_max_aio": "128", 2024-03-11T21:27:25.355 INFO:teuthology.orchestra.run.smithi092.stdout: "rgw_reshard_num_logs": "16", 2024-03-11T21:27:25.355 INFO:teuthology.orchestra.run.smithi092.stdout: "rgw_reshard_thread_interval": "600", 2024-03-11T21:27:25.355 INFO:teuthology.orchestra.run.smithi092.stdout: "rgw_resolve_cname": "false", 2024-03-11T21:27:25.355 INFO:teuthology.orchestra.run.smithi092.stdout: "rgw_rest_getusage_op_compat": "false", 2024-03-11T21:27:25.356 INFO:teuthology.orchestra.run.smithi092.stdout: "rgw_run_sync_thread": "true", 2024-03-11T21:27:25.356 INFO:teuthology.orchestra.run.smithi092.stdout: "rgw_s3_auth_order": "sts, external, local", 2024-03-11T21:27:25.356 INFO:teuthology.orchestra.run.smithi092.stdout: "rgw_s3_auth_use_keystone": "false", 2024-03-11T21:27:25.356 INFO:teuthology.orchestra.run.smithi092.stdout: "rgw_s3_auth_use_ldap": "false", 2024-03-11T21:27:25.356 INFO:teuthology.orchestra.run.smithi092.stdout: "rgw_s3_auth_use_rados": "true", 2024-03-11T21:27:25.356 INFO:teuthology.orchestra.run.smithi092.stdout: "rgw_s3_auth_use_sts": "false", 2024-03-11T21:27:25.356 INFO:teuthology.orchestra.run.smithi092.stdout: "rgw_s3_client_max_sig_ver": "-1", 2024-03-11T21:27:25.356 INFO:teuthology.orchestra.run.smithi092.stdout: "rgw_s3_success_create_obj_status": "0", 2024-03-11T21:27:25.356 INFO:teuthology.orchestra.run.smithi092.stdout: "rgw_safe_max_objects_per_shard": "102400", 2024-03-11T21:27:25.356 INFO:teuthology.orchestra.run.smithi092.stdout: "rgw_scheduler_type": "throttler", 2024-03-11T21:27:25.356 INFO:teuthology.orchestra.run.smithi092.stdout: "rgw_script_uri": "", 2024-03-11T21:27:25.356 INFO:teuthology.orchestra.run.smithi092.stdout: "rgw_service_provider_name": "", 2024-03-11T21:27:25.356 INFO:teuthology.orchestra.run.smithi092.stdout: "rgw_shard_warning_threshold": "90.000000", 2024-03-11T21:27:25.356 INFO:teuthology.orchestra.run.smithi092.stdout: "rgw_sts_client_id": "", 2024-03-11T21:27:25.357 INFO:teuthology.orchestra.run.smithi092.stdout: "rgw_sts_client_secret": "", 2024-03-11T21:27:25.357 INFO:teuthology.orchestra.run.smithi092.stdout: "rgw_sts_entry": "sts", 2024-03-11T21:27:25.357 INFO:teuthology.orchestra.run.smithi092.stdout: "rgw_sts_key": "sts", 2024-03-11T21:27:25.357 INFO:teuthology.orchestra.run.smithi092.stdout: "rgw_sts_max_session_duration": "43200", 2024-03-11T21:27:25.357 INFO:teuthology.orchestra.run.smithi092.stdout: "rgw_sts_min_session_duration": "900", 2024-03-11T21:27:25.357 INFO:teuthology.orchestra.run.smithi092.stdout: "rgw_sts_token_introspection_url": "", 2024-03-11T21:27:25.357 INFO:teuthology.orchestra.run.smithi092.stdout: "rgw_swift_account_in_url": "false", 2024-03-11T21:27:25.357 INFO:teuthology.orchestra.run.smithi092.stdout: "rgw_swift_auth_entry": "auth", 2024-03-11T21:27:25.357 INFO:teuthology.orchestra.run.smithi092.stdout: "rgw_swift_auth_url": "", 2024-03-11T21:27:25.357 INFO:teuthology.orchestra.run.smithi092.stdout: "rgw_swift_custom_header": "", 2024-03-11T21:27:25.357 INFO:teuthology.orchestra.run.smithi092.stdout: "rgw_swift_enforce_content_length": "false", 2024-03-11T21:27:25.357 INFO:teuthology.orchestra.run.smithi092.stdout: "rgw_swift_need_stats": "true", 2024-03-11T21:27:25.357 INFO:teuthology.orchestra.run.smithi092.stdout: "rgw_swift_tenant_name": "", 2024-03-11T21:27:25.357 INFO:teuthology.orchestra.run.smithi092.stdout: "rgw_swift_token_expiration": "86400", 2024-03-11T21:27:25.358 INFO:teuthology.orchestra.run.smithi092.stdout: "rgw_swift_url": "", 2024-03-11T21:27:25.358 INFO:teuthology.orchestra.run.smithi092.stdout: "rgw_swift_url_prefix": "swift", 2024-03-11T21:27:25.358 INFO:teuthology.orchestra.run.smithi092.stdout: "rgw_swift_versioning_enabled": "false", 2024-03-11T21:27:25.358 INFO:teuthology.orchestra.run.smithi092.stdout: "rgw_sync_data_full_inject_err_probability": "0.000000", 2024-03-11T21:27:25.358 INFO:teuthology.orchestra.run.smithi092.stdout: "rgw_sync_data_inject_err_probability": "0.000000", 2024-03-11T21:27:25.358 INFO:teuthology.orchestra.run.smithi092.stdout: "rgw_sync_lease_period": "120", 2024-03-11T21:27:25.358 INFO:teuthology.orchestra.run.smithi092.stdout: "rgw_sync_log_trim_concurrent_buckets": "4", 2024-03-11T21:27:25.358 INFO:teuthology.orchestra.run.smithi092.stdout: "rgw_sync_log_trim_interval": "1200", 2024-03-11T21:27:25.358 INFO:teuthology.orchestra.run.smithi092.stdout: "rgw_sync_log_trim_max_buckets": "16", 2024-03-11T21:27:25.358 INFO:teuthology.orchestra.run.smithi092.stdout: "rgw_sync_log_trim_min_cold_buckets": "4", 2024-03-11T21:27:25.358 INFO:teuthology.orchestra.run.smithi092.stdout: "rgw_sync_meta_inject_err_probability": "0.000000", 2024-03-11T21:27:25.358 INFO:teuthology.orchestra.run.smithi092.stdout: "rgw_sync_obj_etag_verify": "false", 2024-03-11T21:27:25.358 INFO:teuthology.orchestra.run.smithi092.stdout: "rgw_sync_trace_history_size": "4096", 2024-03-11T21:27:25.359 INFO:teuthology.orchestra.run.smithi092.stdout: "rgw_sync_trace_per_node_log_size": "32", 2024-03-11T21:27:25.359 INFO:teuthology.orchestra.run.smithi092.stdout: "rgw_sync_trace_servicemap_update_interval": "10", 2024-03-11T21:27:25.359 INFO:teuthology.orchestra.run.smithi092.stdout: "rgw_thread_pool_size": "512", 2024-03-11T21:27:25.359 INFO:teuthology.orchestra.run.smithi092.stdout: "rgw_topic_persistency_max_retries": "0", 2024-03-11T21:27:25.359 INFO:teuthology.orchestra.run.smithi092.stdout: "rgw_topic_persistency_sleep_duration": "0", 2024-03-11T21:27:25.359 INFO:teuthology.orchestra.run.smithi092.stdout: "rgw_topic_persistency_time_to_live": "0", 2024-03-11T21:27:25.359 INFO:teuthology.orchestra.run.smithi092.stdout: "rgw_topic_require_publish_policy": "false", 2024-03-11T21:27:25.359 INFO:teuthology.orchestra.run.smithi092.stdout: "rgw_torrent_comment": "", 2024-03-11T21:27:25.359 INFO:teuthology.orchestra.run.smithi092.stdout: "rgw_torrent_createby": "", 2024-03-11T21:27:25.359 INFO:teuthology.orchestra.run.smithi092.stdout: "rgw_torrent_encoding": "", 2024-03-11T21:27:25.359 INFO:teuthology.orchestra.run.smithi092.stdout: "rgw_torrent_flag": "false", 2024-03-11T21:27:25.359 INFO:teuthology.orchestra.run.smithi092.stdout: "rgw_torrent_max_size": "5368709120", 2024-03-11T21:27:25.359 INFO:teuthology.orchestra.run.smithi092.stdout: "rgw_torrent_origin": "", 2024-03-11T21:27:25.359 INFO:teuthology.orchestra.run.smithi092.stdout: "rgw_torrent_sha_unit": "524288", 2024-03-11T21:27:25.360 INFO:teuthology.orchestra.run.smithi092.stdout: "rgw_torrent_tracker": "", 2024-03-11T21:27:25.360 INFO:teuthology.orchestra.run.smithi092.stdout: "rgw_trust_forwarded_https": "false", 2024-03-11T21:27:25.360 INFO:teuthology.orchestra.run.smithi092.stdout: "rgw_usage_log_flush_threshold": "1024", 2024-03-11T21:27:25.360 INFO:teuthology.orchestra.run.smithi092.stdout: "rgw_usage_log_tick_interval": "30", 2024-03-11T21:27:25.360 INFO:teuthology.orchestra.run.smithi092.stdout: "rgw_usage_max_shards": "32", 2024-03-11T21:27:25.360 INFO:teuthology.orchestra.run.smithi092.stdout: "rgw_usage_max_user_shards": "1", 2024-03-11T21:27:25.360 INFO:teuthology.orchestra.run.smithi092.stdout: "rgw_use_opa_authz": "false", 2024-03-11T21:27:25.360 INFO:teuthology.orchestra.run.smithi092.stdout: "rgw_user_counters_cache": "false", 2024-03-11T21:27:25.360 INFO:teuthology.orchestra.run.smithi092.stdout: "rgw_user_counters_cache_size": "10000", 2024-03-11T21:27:25.360 INFO:teuthology.orchestra.run.smithi092.stdout: "rgw_user_default_quota_max_objects": "-1", 2024-03-11T21:27:25.360 INFO:teuthology.orchestra.run.smithi092.stdout: "rgw_user_default_quota_max_size": "-1", 2024-03-11T21:27:25.360 INFO:teuthology.orchestra.run.smithi092.stdout: "rgw_user_max_buckets": "1000", 2024-03-11T21:27:25.360 INFO:teuthology.orchestra.run.smithi092.stdout: "rgw_user_policies_max_num": "100", 2024-03-11T21:27:25.361 INFO:teuthology.orchestra.run.smithi092.stdout: "rgw_user_quota_bucket_sync_interval": "180", 2024-03-11T21:27:25.361 INFO:teuthology.orchestra.run.smithi092.stdout: "rgw_user_quota_sync_idle_users": "false", 2024-03-11T21:27:25.361 INFO:teuthology.orchestra.run.smithi092.stdout: "rgw_user_quota_sync_interval": "86400", 2024-03-11T21:27:25.361 INFO:teuthology.orchestra.run.smithi092.stdout: "rgw_user_quota_sync_wait_time": "86400", 2024-03-11T21:27:25.361 INFO:teuthology.orchestra.run.smithi092.stdout: "rgw_user_unique_email": "true", 2024-03-11T21:27:25.361 INFO:teuthology.orchestra.run.smithi092.stdout: "rgw_verify_ssl": "true", 2024-03-11T21:27:25.361 INFO:teuthology.orchestra.run.smithi092.stdout: "rgw_website_routing_rules_max_num": "50", 2024-03-11T21:27:25.361 INFO:teuthology.orchestra.run.smithi092.stdout: "rgw_zone": "", 2024-03-11T21:27:25.361 INFO:teuthology.orchestra.run.smithi092.stdout: "rgw_zone_id": "", 2024-03-11T21:27:25.361 INFO:teuthology.orchestra.run.smithi092.stdout: "rgw_zone_root_pool": ".rgw.root", 2024-03-11T21:27:25.361 INFO:teuthology.orchestra.run.smithi092.stdout: "rgw_zonegroup": "", 2024-03-11T21:27:25.361 INFO:teuthology.orchestra.run.smithi092.stdout: "rgw_zonegroup_id": "", 2024-03-11T21:27:25.361 INFO:teuthology.orchestra.run.smithi092.stdout: "rgw_zonegroup_root_pool": ".rgw.root", 2024-03-11T21:27:25.361 INFO:teuthology.orchestra.run.smithi092.stdout: "rgwlc_auto_session_clear": "true", 2024-03-11T21:27:25.362 INFO:teuthology.orchestra.run.smithi092.stdout: "rgwlc_skip_bucket_step": "false", 2024-03-11T21:27:25.362 INFO:teuthology.orchestra.run.smithi092.stdout: "rocksdb_block_size": "4096", 2024-03-11T21:27:25.362 INFO:teuthology.orchestra.run.smithi092.stdout: "rocksdb_bloom_bits_per_key": "20", 2024-03-11T21:27:25.362 INFO:teuthology.orchestra.run.smithi092.stdout: "rocksdb_cache_index_and_filter_blocks": "true", 2024-03-11T21:27:25.362 INFO:teuthology.orchestra.run.smithi092.stdout: "rocksdb_cache_index_and_filter_blocks_with_high_priority": "false", 2024-03-11T21:27:25.362 INFO:teuthology.orchestra.run.smithi092.stdout: "rocksdb_cache_row_ratio": "0.000000", 2024-03-11T21:27:25.362 INFO:teuthology.orchestra.run.smithi092.stdout: "rocksdb_cache_shard_bits": "4", 2024-03-11T21:27:25.362 INFO:teuthology.orchestra.run.smithi092.stdout: "rocksdb_cache_size": "536870912", 2024-03-11T21:27:25.362 INFO:teuthology.orchestra.run.smithi092.stdout: "rocksdb_cache_type": "binned_lru", 2024-03-11T21:27:25.362 INFO:teuthology.orchestra.run.smithi092.stdout: "rocksdb_cf_compact_on_deletion": "true", 2024-03-11T21:27:25.362 INFO:teuthology.orchestra.run.smithi092.stdout: "rocksdb_cf_compact_on_deletion_sliding_window": "32768", 2024-03-11T21:27:25.362 INFO:teuthology.orchestra.run.smithi092.stdout: "rocksdb_cf_compact_on_deletion_trigger": "16384", 2024-03-11T21:27:25.362 INFO:teuthology.orchestra.run.smithi092.stdout: "rocksdb_collect_compaction_stats": "false", 2024-03-11T21:27:25.363 INFO:teuthology.orchestra.run.smithi092.stdout: "rocksdb_collect_extended_stats": "false", 2024-03-11T21:27:25.363 INFO:teuthology.orchestra.run.smithi092.stdout: "rocksdb_collect_memory_stats": "false", 2024-03-11T21:27:25.363 INFO:teuthology.orchestra.run.smithi092.stdout: "rocksdb_delete_range_threshold": "1048576", 2024-03-11T21:27:25.363 INFO:teuthology.orchestra.run.smithi092.stdout: "rocksdb_index_type": "binary_search", 2024-03-11T21:27:25.363 INFO:teuthology.orchestra.run.smithi092.stdout: "rocksdb_log_to_ceph_log": "true", 2024-03-11T21:27:25.363 INFO:teuthology.orchestra.run.smithi092.stdout: "rocksdb_metadata_block_size": "4096", 2024-03-11T21:27:25.363 INFO:teuthology.orchestra.run.smithi092.stdout: "rocksdb_partition_filters": "false", 2024-03-11T21:27:25.363 INFO:teuthology.orchestra.run.smithi092.stdout: "rocksdb_perf": "false", 2024-03-11T21:27:25.363 INFO:teuthology.orchestra.run.smithi092.stdout: "rocksdb_pin_l0_filter_and_index_blocks_in_cache": "false", 2024-03-11T21:27:25.363 INFO:teuthology.orchestra.run.smithi092.stdout: "rotating_keys_bootstrap_timeout": "30", 2024-03-11T21:27:25.363 INFO:teuthology.orchestra.run.smithi092.stdout: "rotating_keys_renewal_timeout": "10", 2024-03-11T21:27:25.363 INFO:teuthology.orchestra.run.smithi092.stdout: "run_dir": "/var/run/ceph", 2024-03-11T21:27:25.363 INFO:teuthology.orchestra.run.smithi092.stdout: "seastore_block_create": "true", 2024-03-11T21:27:25.363 INFO:teuthology.orchestra.run.smithi092.stdout: "seastore_cache_lru_size": "67108864", 2024-03-11T21:27:25.363 INFO:teuthology.orchestra.run.smithi092.stdout: "seastore_cbjournal_size": "5368709120", 2024-03-11T21:27:25.364 INFO:teuthology.orchestra.run.smithi092.stdout: "seastore_data_delta_based_overwrite": "0", 2024-03-11T21:27:25.364 INFO:teuthology.orchestra.run.smithi092.stdout: "seastore_default_max_object_size": "16777216", 2024-03-11T21:27:25.364 INFO:teuthology.orchestra.run.smithi092.stdout: "seastore_default_object_metadata_reservation": "16777216", 2024-03-11T21:27:25.364 INFO:teuthology.orchestra.run.smithi092.stdout: "seastore_device_size": "53687091200", 2024-03-11T21:27:25.364 INFO:teuthology.orchestra.run.smithi092.stdout: "seastore_journal_batch_capacity": "16", 2024-03-11T21:27:25.364 INFO:teuthology.orchestra.run.smithi092.stdout: "seastore_journal_batch_flush_size": "16777216", 2024-03-11T21:27:25.364 INFO:teuthology.orchestra.run.smithi092.stdout: "seastore_journal_batch_preferred_fullness": "0.950000", 2024-03-11T21:27:25.364 INFO:teuthology.orchestra.run.smithi092.stdout: "seastore_journal_iodepth_limit": "5", 2024-03-11T21:27:25.364 INFO:teuthology.orchestra.run.smithi092.stdout: "seastore_main_device_type": "SSD", 2024-03-11T21:27:25.364 INFO:teuthology.orchestra.run.smithi092.stdout: "seastore_max_concurrent_transactions": "8", 2024-03-11T21:27:25.364 INFO:teuthology.orchestra.run.smithi092.stdout: "seastore_multiple_tiers_default_evict_ratio": "0.600000", 2024-03-11T21:27:25.364 INFO:teuthology.orchestra.run.smithi092.stdout: "seastore_multiple_tiers_fast_evict_ratio": "0.700000", 2024-03-11T21:27:25.364 INFO:teuthology.orchestra.run.smithi092.stdout: "seastore_multiple_tiers_stop_evict_ratio": "0.500000", 2024-03-11T21:27:25.364 INFO:teuthology.orchestra.run.smithi092.stdout: "seastore_obj_data_write_amplification": "1.250000", 2024-03-11T21:27:25.365 INFO:teuthology.orchestra.run.smithi092.stdout: "seastore_segment_size": "67108864", 2024-03-11T21:27:25.365 INFO:teuthology.orchestra.run.smithi092.stdout: "set_keepcaps": "false", 2024-03-11T21:27:25.365 INFO:teuthology.orchestra.run.smithi092.stdout: "setgroup": "ceph", 2024-03-11T21:27:25.365 INFO:teuthology.orchestra.run.smithi092.stdout: "setuser": "ceph", 2024-03-11T21:27:25.365 INFO:teuthology.orchestra.run.smithi092.stdout: "setuser_match_path": "", 2024-03-11T21:27:25.365 INFO:teuthology.orchestra.run.smithi092.stdout: "target_max_misplaced_ratio": "0.050000", 2024-03-11T21:27:25.365 INFO:teuthology.orchestra.run.smithi092.stdout: "thp": "false", 2024-03-11T21:27:25.365 INFO:teuthology.orchestra.run.smithi092.stdout: "threadpool_default_timeout": "60", 2024-03-11T21:27:25.365 INFO:teuthology.orchestra.run.smithi092.stdout: "threadpool_empty_queue_max_wait": "2", 2024-03-11T21:27:25.365 INFO:teuthology.orchestra.run.smithi092.stdout: "throttler_perf_counter": "true" 2024-03-11T21:27:25.365 INFO:teuthology.orchestra.run.smithi092.stdout:} 2024-03-11T21:27:25.881 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:27:25 smithi092 ceph-mon[29572]: 4.b scrub starts 2024-03-11T21:27:25.882 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:27:25 smithi092 ceph-mon[29572]: 4.b scrub ok 2024-03-11T21:27:25.882 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:27:25 smithi092 ceph-mon[29572]: 5.14 scrub starts 2024-03-11T21:27:25.882 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:27:25 smithi092 ceph-mon[29572]: 5.14 scrub ok 2024-03-11T21:27:25.882 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:27:25 smithi092 ceph-mon[37563]: 4.b scrub starts 2024-03-11T21:27:25.882 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:27:25 smithi092 ceph-mon[37563]: 4.b scrub ok 2024-03-11T21:27:25.882 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:27:25 smithi092 ceph-mon[37563]: 5.14 scrub starts 2024-03-11T21:27:25.882 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:27:25 smithi092 ceph-mon[37563]: 5.14 scrub ok 2024-03-11T21:27:26.063 DEBUG:teuthology.orchestra.run.smithi092:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid 00db0016-dfed-11ee-95c8-87774f69a715 -- ceph -- tell 'mon.*' injectargs --mon_osd_down_out_interval=0 2024-03-11T21:27:26.076 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:27:25 smithi097 ceph-mon[33317]: 4.b scrub starts 2024-03-11T21:27:26.077 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:27:25 smithi097 ceph-mon[33317]: 4.b scrub ok 2024-03-11T21:27:26.077 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:27:25 smithi097 ceph-mon[33317]: 5.14 scrub starts 2024-03-11T21:27:26.077 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:27:25 smithi097 ceph-mon[33317]: 5.14 scrub ok 2024-03-11T21:27:26.304 INFO:teuthology.orchestra.run.smithi092.stderr:Inferring config /var/lib/ceph/00db0016-dfed-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T21:27:26.759 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:27:26 smithi092 ceph-mon[29572]: pgmap v45: 132 pgs: 132 active+clean; 582 KiB data, 227 MiB used, 715 GiB / 715 GiB avail; 950 B/s rd, 0 op/s 2024-03-11T21:27:26.760 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:27:26 smithi092 ceph-mon[29572]: 6.10 deep-scrub starts 2024-03-11T21:27:26.760 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:27:26 smithi092 ceph-mon[29572]: 6.10 deep-scrub ok 2024-03-11T21:27:26.760 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:27:26 smithi092 ceph-mon[29572]: 4.17 deep-scrub starts 2024-03-11T21:27:26.760 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:27:26 smithi092 ceph-mon[29572]: 4.17 deep-scrub ok 2024-03-11T21:27:26.760 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:27:26 smithi092 ceph-mon[37563]: pgmap v45: 132 pgs: 132 active+clean; 582 KiB data, 227 MiB used, 715 GiB / 715 GiB avail; 950 B/s rd, 0 op/s 2024-03-11T21:27:26.760 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:27:26 smithi092 ceph-mon[37563]: 6.10 deep-scrub starts 2024-03-11T21:27:26.760 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:27:26 smithi092 ceph-mon[37563]: 6.10 deep-scrub ok 2024-03-11T21:27:26.760 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:27:26 smithi092 ceph-mon[37563]: 4.17 deep-scrub starts 2024-03-11T21:27:26.760 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:27:26 smithi092 ceph-mon[37563]: 4.17 deep-scrub ok 2024-03-11T21:27:27.077 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:27:26 smithi097 ceph-mon[33317]: pgmap v45: 132 pgs: 132 active+clean; 582 KiB data, 227 MiB used, 715 GiB / 715 GiB avail; 950 B/s rd, 0 op/s 2024-03-11T21:27:27.077 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:27:26 smithi097 ceph-mon[33317]: 6.10 deep-scrub starts 2024-03-11T21:27:27.077 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:27:26 smithi097 ceph-mon[33317]: 6.10 deep-scrub ok 2024-03-11T21:27:27.077 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:27:26 smithi097 ceph-mon[33317]: 4.17 deep-scrub starts 2024-03-11T21:27:27.077 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:27:26 smithi097 ceph-mon[33317]: 4.17 deep-scrub ok 2024-03-11T21:27:27.737 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:27:27 smithi092 ceph-mon[29572]: purged_snaps scrub starts 2024-03-11T21:27:27.738 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:27:27 smithi092 ceph-mon[29572]: purged_snaps scrub ok 2024-03-11T21:27:27.738 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:27:27 smithi092 ceph-mon[29572]: osdmap e98: 8 total, 8 up, 8 in 2024-03-11T21:27:27.738 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:27:27 smithi092 ceph-mon[37563]: purged_snaps scrub starts 2024-03-11T21:27:27.738 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:27:27 smithi092 ceph-mon[37563]: purged_snaps scrub ok 2024-03-11T21:27:27.738 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:27:27 smithi092 ceph-mon[37563]: osdmap e98: 8 total, 8 up, 8 in 2024-03-11T21:27:27.972 INFO:teuthology.orchestra.run.smithi092.stdout:mon.a: {} 2024-03-11T21:27:27.982 INFO:teuthology.orchestra.run.smithi092.stdout:mon.b: {} 2024-03-11T21:27:27.992 INFO:teuthology.orchestra.run.smithi092.stdout:mon.c: {} 2024-03-11T21:27:27.992 INFO:journalctl@ceph.mgr.y.smithi092.stdout:Mar 11 21:27:27 smithi092 ceph-00db0016-dfed-11ee-95c8-87774f69a715-mgr-y[29801]: ::ffff:172.21.15.97 - - [11/Mar/2024:21:27:27] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-11T21:27:27.993 INFO:teuthology.orchestra.run.smithi092.stderr:mon.a: mon_osd_down_out_interval = '' (not observed, change may require restart) 2024-03-11T21:27:27.993 INFO:teuthology.orchestra.run.smithi092.stderr:mon.b: mon_osd_down_out_interval = '' (not observed, change may require restart) 2024-03-11T21:27:27.993 INFO:teuthology.orchestra.run.smithi092.stderr:mon.c: mon_osd_down_out_interval = '' (not observed, change may require restart) 2024-03-11T21:27:28.077 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:27:27 smithi097 ceph-mon[33317]: purged_snaps scrub starts 2024-03-11T21:27:28.077 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:27:27 smithi097 ceph-mon[33317]: purged_snaps scrub ok 2024-03-11T21:27:28.077 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:27:27 smithi097 ceph-mon[33317]: osdmap e98: 8 total, 8 up, 8 in 2024-03-11T21:27:28.673 INFO:teuthology.run_tasks:Running task full_sequential... 2024-03-11T21:27:28.703 INFO:teuthology.task.full_sequential:In full_sequential, running task radosbench... 2024-03-11T21:27:28.705 INFO:tasks.radosbench:Beginning radosbench... 2024-03-11T21:27:28.706 INFO:tasks.cephadm.ceph_manager.ceph:creating pool_name unique_pool_0 2024-03-11T21:27:28.706 DEBUG:teuthology.orchestra.run.smithi092:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid 00db0016-dfed-11ee-95c8-87774f69a715 -- ceph osd pool create unique_pool_0 16 2024-03-11T21:27:28.706 INFO:tasks.thrashosds.thrasher:starting do_thrash 2024-03-11T21:27:28.706 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5, 6, 7] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7] 2024-03-11T21:27:28.707 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-03-11T21:27:28.707 INFO:tasks.thrashosds.thrasher:starting do_sighup with a delay of 0.1 2024-03-11T21:27:28.707 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:27:28.707 INFO:tasks.thrashosds.thrasher:starting do_optrack_toggle with a delay of 2.0 2024-03-11T21:27:28.708 DEBUG:teuthology.orchestra.run.smithi092:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid 00db0016-dfed-11ee-95c8-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-03-11T21:27:28.708 INFO:tasks.thrashosds.thrasher:starting do_dump_ops 2024-03-11T21:27:28.708 DEBUG:teuthology.orchestra.run.smithi092:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid 00db0016-dfed-11ee-95c8-87774f69a715 -- ceph daemon osd.0 dump_ops_in_flight 2024-03-11T21:27:28.709 INFO:tasks.thrashosds.thrasher:starting do_noscrub_toggle with a delay of 2.0 2024-03-11T21:27:28.709 DEBUG:teuthology.orchestra.run.smithi092:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid 00db0016-dfed-11ee-95c8-87774f69a715 -- ceph osd set noscrub 2024-03-11T21:27:28.735 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:27:28.736 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:27:28.736 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:27:28.837 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:27:28.871 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:27:28.872 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:27:28.872 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:27:28.973 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:27:28.979 INFO:teuthology.orchestra.run.smithi092.stderr:Inferring config /var/lib/ceph/00db0016-dfed-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T21:27:28.987 INFO:teuthology.orchestra.run.smithi092.stderr:Inferring config /var/lib/ceph/00db0016-dfed-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T21:27:29.004 INFO:teuthology.orchestra.run.smithi092.stderr:Inferring config /var/lib/ceph/00db0016-dfed-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T21:27:29.005 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:27:29.006 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:27:29.006 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:27:29.011 INFO:teuthology.orchestra.run.smithi092.stderr:Inferring config /var/lib/ceph/00db0016-dfed-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T21:27:29.076 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:27:28 smithi097 ceph-mon[33317]: pgmap v47: 132 pgs: 132 active+clean; 582 KiB data, 227 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-11T21:27:29.076 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:27:28 smithi097 ceph-mon[33317]: 2.19 scrub starts 2024-03-11T21:27:29.076 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:27:28 smithi097 ceph-mon[33317]: 2.19 scrub ok 2024-03-11T21:27:29.077 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:27:28 smithi097 ceph-mon[33317]: from='client.? v1:172.21.15.92:0/3271329014' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-03-11T21:27:29.077 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:27:28 smithi097 ceph-mon[33317]: 4.f scrub starts 2024-03-11T21:27:29.077 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:27:28 smithi097 ceph-mon[33317]: 4.f scrub ok 2024-03-11T21:27:29.107 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:27:29.110 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:27:28 smithi092 ceph-mon[29572]: pgmap v47: 132 pgs: 132 active+clean; 582 KiB data, 227 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-11T21:27:29.110 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:27:28 smithi092 ceph-mon[29572]: 2.19 scrub starts 2024-03-11T21:27:29.110 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:27:28 smithi092 ceph-mon[29572]: 2.19 scrub ok 2024-03-11T21:27:29.110 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:27:28 smithi092 ceph-mon[29572]: from='client.? v1:172.21.15.92:0/3271329014' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-03-11T21:27:29.110 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:27:28 smithi092 ceph-mon[29572]: 4.f scrub starts 2024-03-11T21:27:29.110 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:27:28 smithi092 ceph-mon[29572]: 4.f scrub ok 2024-03-11T21:27:29.110 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:27:28 smithi092 ceph-mon[37563]: pgmap v47: 132 pgs: 132 active+clean; 582 KiB data, 227 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-11T21:27:29.110 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:27:28 smithi092 ceph-mon[37563]: 2.19 scrub starts 2024-03-11T21:27:29.110 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:27:28 smithi092 ceph-mon[37563]: 2.19 scrub ok 2024-03-11T21:27:29.110 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:27:28 smithi092 ceph-mon[37563]: from='client.? v1:172.21.15.92:0/3271329014' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-03-11T21:27:29.111 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:27:28 smithi092 ceph-mon[37563]: 4.f scrub starts 2024-03-11T21:27:29.111 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:27:28 smithi092 ceph-mon[37563]: 4.f scrub ok 2024-03-11T21:27:29.131 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:27:29.132 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:27:29.133 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:27:29.233 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:27:29.260 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:27:29.260 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:27:29.261 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:27:29.362 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:27:29.385 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:27:29.386 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:27:29.387 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:27:29.487 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:27:29.512 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:27:29.513 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:27:29.513 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:27:29.614 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:27:29.638 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:27:29.639 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:27:29.639 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:27:29.740 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:27:29.764 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:27:29 smithi092 ceph-mon[29572]: 5.b scrub starts 2024-03-11T21:27:29.764 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:27:29 smithi092 ceph-mon[29572]: 5.b scrub ok 2024-03-11T21:27:29.764 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:27:29 smithi092 ceph-mon[37563]: 5.b scrub starts 2024-03-11T21:27:29.764 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:27:29 smithi092 ceph-mon[37563]: 5.b scrub ok 2024-03-11T21:27:29.765 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:27:29.766 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:27:29.766 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:27:29.867 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:27:29.890 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:27:29.891 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:27:29.891 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:27:29.992 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:27:30.017 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:27:30.018 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:27:30.018 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:27:30.077 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:27:29 smithi097 ceph-mon[33317]: 5.b scrub starts 2024-03-11T21:27:30.077 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:27:29 smithi097 ceph-mon[33317]: 5.b scrub ok 2024-03-11T21:27:30.119 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:27:30.143 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:27:30.144 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:27:30.144 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:27:30.245 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:27:30.270 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:27:30.271 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:27:30.271 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:27:30.372 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:27:30.397 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:27:30.398 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:27:30.399 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:27:30.499 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:27:30.523 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:27:30.524 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:27:30.524 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:27:30.625 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:27:30.649 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:27:30.650 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:27:30.651 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:27:30.751 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:27:30.777 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:27:30.777 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:27:30.777 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:27:30.878 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:27:30.902 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:27:30.903 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:27:30.903 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:27:31.004 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:27:31.028 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:27:31.029 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:27:31.029 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:27:31.036 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:27:30 smithi092 ceph-mon[29572]: pgmap v48: 132 pgs: 132 active+clean; 582 KiB data, 227 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-11T21:27:31.036 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:27:30 smithi092 ceph-mon[29572]: 6.18 scrub starts 2024-03-11T21:27:31.036 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:27:30 smithi092 ceph-mon[29572]: 6.18 scrub ok 2024-03-11T21:27:31.036 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:27:30 smithi092 ceph-mon[29572]: 5.0 deep-scrub starts 2024-03-11T21:27:31.036 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:27:30 smithi092 ceph-mon[29572]: 5.0 deep-scrub ok 2024-03-11T21:27:31.037 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:27:30 smithi092 ceph-mon[37563]: pgmap v48: 132 pgs: 132 active+clean; 582 KiB data, 227 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-11T21:27:31.037 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:27:30 smithi092 ceph-mon[37563]: 6.18 scrub starts 2024-03-11T21:27:31.037 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:27:30 smithi092 ceph-mon[37563]: 6.18 scrub ok 2024-03-11T21:27:31.037 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:27:30 smithi092 ceph-mon[37563]: 5.0 deep-scrub starts 2024-03-11T21:27:31.037 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:27:30 smithi092 ceph-mon[37563]: 5.0 deep-scrub ok 2024-03-11T21:27:31.076 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:27:30 smithi097 ceph-mon[33317]: pgmap v48: 132 pgs: 132 active+clean; 582 KiB data, 227 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-11T21:27:31.077 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:27:30 smithi097 ceph-mon[33317]: 6.18 scrub starts 2024-03-11T21:27:31.077 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:27:30 smithi097 ceph-mon[33317]: 6.18 scrub ok 2024-03-11T21:27:31.077 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:27:30 smithi097 ceph-mon[33317]: 5.0 deep-scrub starts 2024-03-11T21:27:31.077 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:27:30 smithi097 ceph-mon[33317]: 5.0 deep-scrub ok 2024-03-11T21:27:31.130 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:27:31.155 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:27:31.156 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:27:31.156 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:27:31.257 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:27:31.282 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:27:31.282 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:27:31.283 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:27:31.384 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:27:31.408 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:27:31.409 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:27:31.409 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:27:31.510 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:27:31.533 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:27:31.534 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:27:31.534 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:27:31.635 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:27:31.658 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:27:31.659 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:27:31.659 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:27:31.760 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:27:31.784 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:27:31.785 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:27:31.785 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:27:31.886 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:27:31.911 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:27:31.912 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:27:31.912 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:27:32.000 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:27:31 smithi097 ceph-mon[33317]: 6.19 scrub starts 2024-03-11T21:27:32.000 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:27:31 smithi097 ceph-mon[33317]: 6.19 scrub ok 2024-03-11T21:27:32.013 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:27:32.038 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:27:32.038 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:27:32.039 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:27:32.078 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:27:31 smithi092 ceph-mon[29572]: 6.19 scrub starts 2024-03-11T21:27:32.078 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:27:31 smithi092 ceph-mon[29572]: 6.19 scrub ok 2024-03-11T21:27:32.078 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:27:31 smithi092 ceph-mon[37563]: 6.19 scrub starts 2024-03-11T21:27:32.078 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:27:31 smithi092 ceph-mon[37563]: 6.19 scrub ok 2024-03-11T21:27:32.140 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:27:32.165 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:27:32.166 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:27:32.167 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:27:32.268 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:27:32.293 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:27:32.294 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:27:32.294 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:27:32.327 INFO:journalctl@ceph.iscsi.iscsi.a.smithi097.stdout:Mar 11 21:27:32 smithi097 ceph-00db0016-dfed-11ee-95c8-87774f69a715-iscsi-iscsi-a[60943]: debug there is no tcmu-runner data available 2024-03-11T21:27:32.396 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:27:32.422 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:27:32.423 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:27:32.423 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:27:32.524 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:27:32.549 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:27:32.549 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:27:32.550 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:27:32.651 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:27:32.674 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:27:32.675 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:27:32.675 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:27:32.767 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:27:32 smithi092 ceph-mon[29572]: pgmap v49: 132 pgs: 132 active+clean; 582 KiB data, 227 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-11T21:27:32.767 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:27:32 smithi092 ceph-mon[29572]: 6.e scrub starts 2024-03-11T21:27:32.767 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:27:32 smithi092 ceph-mon[29572]: 6.e scrub ok 2024-03-11T21:27:32.767 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:27:32 smithi092 ceph-mon[37563]: pgmap v49: 132 pgs: 132 active+clean; 582 KiB data, 227 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-11T21:27:32.767 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:27:32 smithi092 ceph-mon[37563]: 6.e scrub starts 2024-03-11T21:27:32.767 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:27:32 smithi092 ceph-mon[37563]: 6.e scrub ok 2024-03-11T21:27:32.777 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:27:32.800 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:27:32.801 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:27:32.802 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:27:32.902 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:27:32.926 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:27:32.926 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:27:32.927 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:27:33.027 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:27:33.051 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:27:33.052 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:27:33.052 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:27:33.076 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:27:32 smithi097 ceph-mon[33317]: pgmap v49: 132 pgs: 132 active+clean; 582 KiB data, 227 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-11T21:27:33.076 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:27:32 smithi097 ceph-mon[33317]: 6.e scrub starts 2024-03-11T21:27:33.077 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:27:32 smithi097 ceph-mon[33317]: 6.e scrub ok 2024-03-11T21:27:33.153 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:27:33.177 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:27:33.178 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:27:33.178 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:27:33.279 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:27:33.303 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:27:33.303 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:27:33.303 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:27:33.404 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:27:33.428 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:27:33.429 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:27:33.429 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:27:33.530 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:27:33.553 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:27:33.554 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:27:33.554 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:27:33.655 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:27:33.679 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:27:33.680 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:27:33.680 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:27:33.781 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:27:33.807 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:27:33.808 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:27:33.808 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:27:33.909 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:27:33.932 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:27:33 smithi092 ceph-mon[29572]: from='client.24469 v1:172.21.15.97:0/404208310' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:27:33.932 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:27:33 smithi092 ceph-mon[37563]: from='client.24469 v1:172.21.15.97:0/404208310' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:27:33.934 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:27:33.935 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:27:33.935 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:27:34.036 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:27:34.062 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:27:34.062 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:27:34.063 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:27:34.076 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:27:33 smithi097 ceph-mon[33317]: from='client.24469 v1:172.21.15.97:0/404208310' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:27:34.163 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:27:34.186 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:27:34.187 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:27:34.187 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:27:34.288 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:27:34.313 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:27:34.314 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:27:34.314 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:27:34.415 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:27:34.439 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:27:34.439 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:27:34.440 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:27:34.541 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:27:34.565 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:27:34.565 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:27:34.566 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:27:34.666 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:27:34.690 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:27:34.818 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:27:34 smithi097 ceph-mon[33317]: pgmap v50: 132 pgs: 132 active+clean; 582 KiB data, 226 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-11T21:27:34.818 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:27:34 smithi097 ceph-mon[33317]: 5.7 scrub starts 2024-03-11T21:27:34.818 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:27:34 smithi097 ceph-mon[33317]: 5.7 scrub ok 2024-03-11T21:27:34.818 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:27:34.819 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:27:34.819 INFO:teuthology.orchestra.run.smithi092.stdout:{ 2024-03-11T21:27:34.819 INFO:teuthology.orchestra.run.smithi092.stdout: "ops": [], 2024-03-11T21:27:34.819 INFO:teuthology.orchestra.run.smithi092.stdout: "num_ops": 0 2024-03-11T21:27:34.819 INFO:teuthology.orchestra.run.smithi092.stdout:} 2024-03-11T21:27:34.819 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:27:34 smithi092 ceph-mon[29572]: pgmap v50: 132 pgs: 132 active+clean; 582 KiB data, 226 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-11T21:27:34.820 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:27:34 smithi092 ceph-mon[29572]: 5.7 scrub starts 2024-03-11T21:27:34.820 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:27:34 smithi092 ceph-mon[29572]: 5.7 scrub ok 2024-03-11T21:27:34.820 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:27:34 smithi092 ceph-mon[37563]: pgmap v50: 132 pgs: 132 active+clean; 582 KiB data, 226 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-11T21:27:34.820 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:27:34 smithi092 ceph-mon[37563]: 5.7 scrub starts 2024-03-11T21:27:34.820 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:27:34 smithi092 ceph-mon[37563]: 5.7 scrub ok 2024-03-11T21:27:34.920 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:27:34.946 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:27:34.946 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:27:34.946 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:27:35.047 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:27:35.072 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:27:35.072 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:27:35.072 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:27:35.173 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:27:35.198 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:27:35.199 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:27:35.199 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:27:35.299 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:27:35.322 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:27:35.323 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:27:35.323 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:27:35.424 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:27:35.446 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:27:35.447 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:27:35.447 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:27:35.548 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:27:35.572 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:27:35.573 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:27:35.573 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:27:35.664 INFO:teuthology.orchestra.run.smithi092.stderr:pool 'unique_pool_0' created 2024-03-11T21:27:35.664 INFO:teuthology.orchestra.run.smithi092.stderr:noscrub is set 2024-03-11T21:27:35.674 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:27:35.701 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:27:35.702 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:27:35.702 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:27:35.803 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:27:35.828 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:27:35.829 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:27:35.829 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:27:35.861 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:27:35 smithi092 ceph-mon[29572]: 5.16 scrub starts 2024-03-11T21:27:35.861 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:27:35 smithi092 ceph-mon[29572]: 5.16 scrub ok 2024-03-11T21:27:35.861 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:27:35 smithi092 ceph-mon[29572]: from='client.? v1:172.21.15.92:0/3521337632' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "unique_pool_0", "pg_num": 16}]: dispatch 2024-03-11T21:27:35.861 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:27:35 smithi092 ceph-mon[29572]: from='client.? v1:172.21.15.92:0/2345074901' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-11T21:27:35.861 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:27:35 smithi092 ceph-mon[29572]: from='client.34688 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-11T21:27:35.861 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:27:35 smithi092 ceph-mon[37563]: 5.16 scrub starts 2024-03-11T21:27:35.861 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:27:35 smithi092 ceph-mon[37563]: 5.16 scrub ok 2024-03-11T21:27:35.861 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:27:35 smithi092 ceph-mon[37563]: from='client.? v1:172.21.15.92:0/3521337632' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "unique_pool_0", "pg_num": 16}]: dispatch 2024-03-11T21:27:35.861 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:27:35 smithi092 ceph-mon[37563]: from='client.? v1:172.21.15.92:0/2345074901' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-11T21:27:35.861 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:27:35 smithi092 ceph-mon[37563]: from='client.34688 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-11T21:27:35.930 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:27:35.957 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:27:35.957 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:27:35.958 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:27:35.967 DEBUG:teuthology.orchestra.run.smithi092:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid 00db0016-dfed-11ee-95c8-87774f69a715 -- ceph daemon osd.0 dump_blocked_ops 2024-03-11T21:27:36.059 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:27:36.077 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:27:35 smithi097 ceph-mon[33317]: 5.16 scrub starts 2024-03-11T21:27:36.077 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:27:35 smithi097 ceph-mon[33317]: 5.16 scrub ok 2024-03-11T21:27:36.077 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:27:35 smithi097 ceph-mon[33317]: from='client.? v1:172.21.15.92:0/3521337632' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "unique_pool_0", "pg_num": 16}]: dispatch 2024-03-11T21:27:36.077 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:27:35 smithi097 ceph-mon[33317]: from='client.? v1:172.21.15.92:0/2345074901' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-11T21:27:36.077 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:27:35 smithi097 ceph-mon[33317]: from='client.34688 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-11T21:27:36.082 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:27:36.083 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:27:36.084 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:27:36.169 INFO:teuthology.orchestra.run.smithi092.stdout:osd.0: {} 2024-03-11T21:27:36.185 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:27:36.190 INFO:teuthology.orchestra.run.smithi092.stdout:osd.1: {} 2024-03-11T21:27:36.209 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:27:36.209 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:27:36.209 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:27:36.210 INFO:teuthology.orchestra.run.smithi092.stdout:osd.2: {} 2024-03-11T21:27:36.230 INFO:teuthology.orchestra.run.smithi092.stdout:osd.3: {} 2024-03-11T21:27:36.250 INFO:teuthology.orchestra.run.smithi092.stdout:osd.4: {} 2024-03-11T21:27:36.270 INFO:teuthology.orchestra.run.smithi092.stdout:osd.5: {} 2024-03-11T21:27:36.290 INFO:teuthology.orchestra.run.smithi092.stdout:osd.6: {} 2024-03-11T21:27:36.310 INFO:teuthology.orchestra.run.smithi092.stdout:osd.7: {} 2024-03-11T21:27:36.310 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:27:36.311 INFO:teuthology.orchestra.run.smithi092.stderr:osd.0: osd_enable_op_tracker = '' 2024-03-11T21:27:36.311 INFO:teuthology.orchestra.run.smithi092.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-11T21:27:36.311 INFO:teuthology.orchestra.run.smithi092.stderr:osd.2: osd_enable_op_tracker = '' 2024-03-11T21:27:36.311 INFO:teuthology.orchestra.run.smithi092.stderr:osd.3: osd_enable_op_tracker = '' 2024-03-11T21:27:36.311 INFO:teuthology.orchestra.run.smithi092.stderr:osd.4: osd_enable_op_tracker = '' 2024-03-11T21:27:36.311 INFO:teuthology.orchestra.run.smithi092.stderr:osd.5: osd_enable_op_tracker = '' 2024-03-11T21:27:36.311 INFO:teuthology.orchestra.run.smithi092.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-11T21:27:36.312 INFO:teuthology.orchestra.run.smithi092.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-11T21:27:36.334 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:27:36.335 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:27:36.335 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:27:36.436 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:27:36.466 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:27:36.467 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:27:36.467 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:27:36.568 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:27:36.595 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:27:36.596 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:27:36.596 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:27:36.697 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:27:36.723 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:27:36.724 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:27:36.724 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:27:36.825 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:27:36.851 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:27:36.851 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:27:36.852 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:27:36.947 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:27:36 smithi092 ceph-mon[29572]: 5.15 deep-scrub starts 2024-03-11T21:27:36.947 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:27:36 smithi092 ceph-mon[29572]: 5.15 deep-scrub ok 2024-03-11T21:27:36.947 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:27:36 smithi092 ceph-mon[29572]: pgmap v51: 132 pgs: 132 active+clean; 582 KiB data, 226 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-11T21:27:36.948 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:27:36 smithi092 ceph-mon[29572]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-03-11T21:27:36.948 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:27:36 smithi092 ceph-mon[29572]: from='client.? v1:172.21.15.92:0/3521337632' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "unique_pool_0", "pg_num": 16}]': finished 2024-03-11T21:27:36.948 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:27:36 smithi092 ceph-mon[29572]: from='client.34688 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-03-11T21:27:36.948 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:27:36 smithi092 ceph-mon[29572]: osdmap e99: 8 total, 8 up, 8 in 2024-03-11T21:27:36.948 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:27:36 smithi092 ceph-mon[29572]: from='mgr.14607 ' entity='mgr.y' 2024-03-11T21:27:36.948 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:27:36 smithi092 ceph-mon[29572]: from='mgr.14607 v1:172.21.15.92:0/1702717597' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T21:27:36.948 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:27:36 smithi092 ceph-mon[29572]: from='client.? v1:172.21.15.92:0/875523632' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-11T21:27:36.948 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:27:36 smithi092 ceph-mon[37563]: 5.15 deep-scrub starts 2024-03-11T21:27:36.948 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:27:36 smithi092 ceph-mon[37563]: 5.15 deep-scrub ok 2024-03-11T21:27:36.948 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:27:36 smithi092 ceph-mon[37563]: pgmap v51: 132 pgs: 132 active+clean; 582 KiB data, 226 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-11T21:27:36.948 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:27:36 smithi092 ceph-mon[37563]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-03-11T21:27:36.948 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:27:36 smithi092 ceph-mon[37563]: from='client.? v1:172.21.15.92:0/3521337632' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "unique_pool_0", "pg_num": 16}]': finished 2024-03-11T21:27:36.949 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:27:36 smithi092 ceph-mon[37563]: from='client.34688 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-03-11T21:27:36.949 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:27:36 smithi092 ceph-mon[37563]: osdmap e99: 8 total, 8 up, 8 in 2024-03-11T21:27:36.949 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:27:36 smithi092 ceph-mon[37563]: from='mgr.14607 ' entity='mgr.y' 2024-03-11T21:27:36.949 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:27:36 smithi092 ceph-mon[37563]: from='mgr.14607 v1:172.21.15.92:0/1702717597' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T21:27:36.949 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:27:36 smithi092 ceph-mon[37563]: from='client.? v1:172.21.15.92:0/875523632' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-11T21:27:36.952 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:27:36.973 DEBUG:teuthology.orchestra.run.smithi092:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid 00db0016-dfed-11ee-95c8-87774f69a715 -- ceph osd pool application enable unique_pool_0 rados --yes-i-really-mean-it || true 2024-03-11T21:27:36.977 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:27:36.978 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:27:36.978 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:27:36.994 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:27:36 smithi097 ceph-mon[33317]: 5.15 deep-scrub starts 2024-03-11T21:27:36.994 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:27:36 smithi097 ceph-mon[33317]: 5.15 deep-scrub ok 2024-03-11T21:27:36.995 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:27:36 smithi097 ceph-mon[33317]: pgmap v51: 132 pgs: 132 active+clean; 582 KiB data, 226 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-11T21:27:36.995 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:27:36 smithi097 ceph-mon[33317]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-03-11T21:27:36.995 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:27:36 smithi097 ceph-mon[33317]: from='client.? v1:172.21.15.92:0/3521337632' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "unique_pool_0", "pg_num": 16}]': finished 2024-03-11T21:27:36.995 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:27:36 smithi097 ceph-mon[33317]: from='client.34688 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-03-11T21:27:36.995 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:27:36 smithi097 ceph-mon[33317]: osdmap e99: 8 total, 8 up, 8 in 2024-03-11T21:27:36.995 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:27:36 smithi097 ceph-mon[33317]: from='mgr.14607 ' entity='mgr.y' 2024-03-11T21:27:36.995 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:27:36 smithi097 ceph-mon[33317]: from='mgr.14607 v1:172.21.15.92:0/1702717597' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T21:27:36.995 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:27:36 smithi097 ceph-mon[33317]: from='client.? v1:172.21.15.92:0/875523632' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-11T21:27:37.080 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:27:37.105 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:27:37.105 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:27:37.106 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:27:37.207 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:27:37.231 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:27:37.232 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:27:37.232 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:27:37.333 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:27:37.357 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:27:37.357 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:27:37.357 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:27:37.458 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:27:37.485 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:27:37.486 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:27:37.486 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:27:37.577 INFO:journalctl@ceph.grafana.a.smithi097.stdout:Mar 11 21:27:37 smithi097 systemd[1]: Starting Ceph grafana.a for 00db0016-dfed-11ee-95c8-87774f69a715... 2024-03-11T21:27:37.587 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:27:37.611 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:27:37.611 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:27:37.611 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:27:37.692 INFO:teuthology.orchestra.run.smithi092.stderr:Inferring config /var/lib/ceph/00db0016-dfed-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T21:27:37.712 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:27:37.739 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:27:37.740 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:27:37.740 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:27:37.841 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:27:37.866 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:27:37.866 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:27:37.866 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:27:37.924 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:27:37 smithi092 ceph-mon[29572]: osdmap e100: 8 total, 8 up, 8 in 2024-03-11T21:27:37.924 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:27:37 smithi092 ceph-mon[37563]: osdmap e100: 8 total, 8 up, 8 in 2024-03-11T21:27:37.925 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:27:37 smithi097 ceph-mon[33317]: osdmap e100: 8 total, 8 up, 8 in 2024-03-11T21:27:37.925 INFO:journalctl@ceph.grafana.a.smithi097.stdout:Mar 11 21:27:37 smithi097 podman[65094]: 2024-03-11 21:27:37.601875447 +0000 UTC m=+0.017094414 image pull a0ea5e3c46d9434bd173d585b7c2638c7451eaf2543d1d22f9e78442615aaffb quay.io/ceph/ceph-grafana:9.4.12 2024-03-11T21:27:37.926 INFO:journalctl@ceph.grafana.a.smithi097.stdout:Mar 11 21:27:37 smithi097 podman[65094]: 2024-03-11 21:27:37.715298957 +0000 UTC m=+0.130517955 container create 2bc78bd29e96e10effd38c8e2bda8c99c1d139e375736df4ef16defdbd141aeb (image=quay.io/ceph/ceph-grafana:9.4.12, name=ceph-00db0016-dfed-11ee-95c8-87774f69a715-grafana-a, vcs-type=git, com.redhat.license_terms=https://www.redhat.com/en/about/red-hat-end-user-license-agreements#UBI, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., description=Ceph Grafana Container, build-date=2022-04-29T11:48:11.055897, summary=Grafana Container configured for Ceph mgr/dashboard integration, io.buildah.version=1.32.0, vendor=Red Hat, Inc., version=8.5, release=239.1651231664, vcs-ref=3aadd00326f3dd6cfe65ee31017ab98915fddb56, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.5-239.1651231664, name=ubi8, maintainer=Paul Cuzner , io.openshift.tags=base rhel8, io.openshift.expose-services=, com.redhat.build-host=cpt-1007.osbs.prod.upshift.rdu2.redhat.com, com.redhat.component=ubi8-container, architecture=x86_64, distribution-scope=public, io.k8s.display-name=Red Hat Universal Base Image 8) 2024-03-11T21:27:37.926 INFO:journalctl@ceph.grafana.a.smithi097.stdout:Mar 11 21:27:37 smithi097 podman[65094]: 2024-03-11 21:27:37.943394994 +0000 UTC m=+0.358613958 container init 2bc78bd29e96e10effd38c8e2bda8c99c1d139e375736df4ef16defdbd141aeb (image=quay.io/ceph/ceph-grafana:9.4.12, name=ceph-00db0016-dfed-11ee-95c8-87774f69a715-grafana-a, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., maintainer=Paul Cuzner , io.openshift.tags=base rhel8, summary=Grafana Container configured for Ceph mgr/dashboard integration, com.redhat.license_terms=https://www.redhat.com/en/about/red-hat-end-user-license-agreements#UBI, io.buildah.version=1.32.0, architecture=x86_64, vcs-type=git, name=ubi8, com.redhat.component=ubi8-container, release=239.1651231664, description=Ceph Grafana Container, vcs-ref=3aadd00326f3dd6cfe65ee31017ab98915fddb56, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.5-239.1651231664, vendor=Red Hat, Inc., com.redhat.build-host=cpt-1007.osbs.prod.upshift.rdu2.redhat.com, io.openshift.expose-services=, distribution-scope=public, build-date=2022-04-29T11:48:11.055897, version=8.5, io.k8s.display-name=Red Hat Universal Base Image 8) 2024-03-11T21:27:37.967 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:27:37.992 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:27:37.992 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:27:37.993 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:27:38.094 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:27:38.118 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:27:38.119 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:27:38.120 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:27:38.176 INFO:journalctl@ceph.grafana.a.smithi097.stdout:Mar 11 21:27:37 smithi097 podman[65094]: 2024-03-11 21:27:37.946239365 +0000 UTC m=+0.361458330 container start 2bc78bd29e96e10effd38c8e2bda8c99c1d139e375736df4ef16defdbd141aeb (image=quay.io/ceph/ceph-grafana:9.4.12, name=ceph-00db0016-dfed-11ee-95c8-87774f69a715-grafana-a, version=8.5, maintainer=Paul Cuzner , vcs-ref=3aadd00326f3dd6cfe65ee31017ab98915fddb56, com.redhat.license_terms=https://www.redhat.com/en/about/red-hat-end-user-license-agreements#UBI, vendor=Red Hat, Inc., description=Ceph Grafana Container, build-date=2022-04-29T11:48:11.055897, io.openshift.expose-services=, release=239.1651231664, architecture=x86_64, vcs-type=git, io.buildah.version=1.32.0, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., com.redhat.build-host=cpt-1007.osbs.prod.upshift.rdu2.redhat.com, io.openshift.tags=base rhel8, distribution-scope=public, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.5-239.1651231664, summary=Grafana Container configured for Ceph mgr/dashboard integration, name=ubi8, com.redhat.component=ubi8-container, io.k8s.display-name=Red Hat Universal Base Image 8) 2024-03-11T21:27:38.176 INFO:journalctl@ceph.grafana.a.smithi097.stdout:Mar 11 21:27:38 smithi097 ceph-00db0016-dfed-11ee-95c8-87774f69a715-grafana-a[65126]: logger=settings t=2024-03-11T21:27:38.096513636Z level=info msg="Starting Grafana" version=9.4.12 commit=37ee25ba46 branch=HEAD compiled=2023-05-22T15:58:05Z 2024-03-11T21:27:38.177 INFO:journalctl@ceph.grafana.a.smithi097.stdout:Mar 11 21:27:38 smithi097 ceph-00db0016-dfed-11ee-95c8-87774f69a715-grafana-a[65126]: logger=settings t=2024-03-11T21:27:38.096707931Z level=warn msg="\"sentry\" frontend logging provider is deprecated and will be removed in the next major version. Use \"grafana\" provider instead." 2024-03-11T21:27:38.177 INFO:journalctl@ceph.grafana.a.smithi097.stdout:Mar 11 21:27:38 smithi097 ceph-00db0016-dfed-11ee-95c8-87774f69a715-grafana-a[65126]: logger=settings t=2024-03-11T21:27:38.096726261Z level=info msg="Config loaded from" file=/usr/share/grafana/conf/defaults.ini 2024-03-11T21:27:38.177 INFO:journalctl@ceph.grafana.a.smithi097.stdout:Mar 11 21:27:38 smithi097 ceph-00db0016-dfed-11ee-95c8-87774f69a715-grafana-a[65126]: logger=settings t=2024-03-11T21:27:38.096733871Z level=info msg="Config loaded from" file=/etc/grafana/grafana.ini 2024-03-11T21:27:38.177 INFO:journalctl@ceph.grafana.a.smithi097.stdout:Mar 11 21:27:38 smithi097 ceph-00db0016-dfed-11ee-95c8-87774f69a715-grafana-a[65126]: logger=settings t=2024-03-11T21:27:38.096739822Z level=info msg="Config overridden from Environment variable" var="GF_PATHS_DATA=/var/lib/grafana" 2024-03-11T21:27:38.177 INFO:journalctl@ceph.grafana.a.smithi097.stdout:Mar 11 21:27:38 smithi097 ceph-00db0016-dfed-11ee-95c8-87774f69a715-grafana-a[65126]: logger=settings t=2024-03-11T21:27:38.096745721Z level=info msg="Config overridden from Environment variable" var="GF_PATHS_LOGS=/var/log/grafana" 2024-03-11T21:27:38.177 INFO:journalctl@ceph.grafana.a.smithi097.stdout:Mar 11 21:27:38 smithi097 ceph-00db0016-dfed-11ee-95c8-87774f69a715-grafana-a[65126]: logger=settings t=2024-03-11T21:27:38.09675139Z level=info msg="Config overridden from Environment variable" var="GF_PATHS_PLUGINS=/var/lib/grafana/plugins" 2024-03-11T21:27:38.177 INFO:journalctl@ceph.grafana.a.smithi097.stdout:Mar 11 21:27:38 smithi097 ceph-00db0016-dfed-11ee-95c8-87774f69a715-grafana-a[65126]: logger=settings t=2024-03-11T21:27:38.09675721Z level=info msg="Config overridden from Environment variable" var="GF_PATHS_PROVISIONING=/etc/grafana/provisioning" 2024-03-11T21:27:38.177 INFO:journalctl@ceph.grafana.a.smithi097.stdout:Mar 11 21:27:38 smithi097 ceph-00db0016-dfed-11ee-95c8-87774f69a715-grafana-a[65126]: logger=settings t=2024-03-11T21:27:38.096763061Z level=info msg="Path Home" path=/usr/share/grafana 2024-03-11T21:27:38.177 INFO:journalctl@ceph.grafana.a.smithi097.stdout:Mar 11 21:27:38 smithi097 ceph-00db0016-dfed-11ee-95c8-87774f69a715-grafana-a[65126]: logger=settings t=2024-03-11T21:27:38.096768759Z level=info msg="Path Data" path=/var/lib/grafana 2024-03-11T21:27:38.177 INFO:journalctl@ceph.grafana.a.smithi097.stdout:Mar 11 21:27:38 smithi097 ceph-00db0016-dfed-11ee-95c8-87774f69a715-grafana-a[65126]: logger=settings t=2024-03-11T21:27:38.096774988Z level=info msg="Path Logs" path=/var/log/grafana 2024-03-11T21:27:38.177 INFO:journalctl@ceph.grafana.a.smithi097.stdout:Mar 11 21:27:38 smithi097 ceph-00db0016-dfed-11ee-95c8-87774f69a715-grafana-a[65126]: logger=settings t=2024-03-11T21:27:38.096780511Z level=info msg="Path Plugins" path=/var/lib/grafana/plugins 2024-03-11T21:27:38.177 INFO:journalctl@ceph.grafana.a.smithi097.stdout:Mar 11 21:27:38 smithi097 ceph-00db0016-dfed-11ee-95c8-87774f69a715-grafana-a[65126]: logger=settings t=2024-03-11T21:27:38.096787278Z level=info msg="Path Provisioning" path=/etc/grafana/provisioning 2024-03-11T21:27:38.177 INFO:journalctl@ceph.grafana.a.smithi097.stdout:Mar 11 21:27:38 smithi097 ceph-00db0016-dfed-11ee-95c8-87774f69a715-grafana-a[65126]: logger=settings t=2024-03-11T21:27:38.096793289Z level=info msg="App mode production" 2024-03-11T21:27:38.177 INFO:journalctl@ceph.grafana.a.smithi097.stdout:Mar 11 21:27:38 smithi097 ceph-00db0016-dfed-11ee-95c8-87774f69a715-grafana-a[65126]: logger=sqlstore t=2024-03-11T21:27:38.096840968Z level=info msg="Connecting to DB" dbtype=sqlite3 2024-03-11T21:27:38.178 INFO:journalctl@ceph.grafana.a.smithi097.stdout:Mar 11 21:27:38 smithi097 ceph-00db0016-dfed-11ee-95c8-87774f69a715-grafana-a[65126]: logger=sqlstore t=2024-03-11T21:27:38.096857725Z level=warn msg="SQLite database file has broader permissions than it should" path=/var/lib/grafana/grafana.db mode=-rw-r--r-- expected=-rw-r----- 2024-03-11T21:27:38.178 INFO:journalctl@ceph.grafana.a.smithi097.stdout:Mar 11 21:27:38 smithi097 ceph-00db0016-dfed-11ee-95c8-87774f69a715-grafana-a[65126]: logger=migrator t=2024-03-11T21:27:38.097730924Z level=info msg="Starting DB migrations" 2024-03-11T21:27:38.178 INFO:journalctl@ceph.grafana.a.smithi097.stdout:Mar 11 21:27:38 smithi097 ceph-00db0016-dfed-11ee-95c8-87774f69a715-grafana-a[65126]: logger=migrator t=2024-03-11T21:27:38.097806612Z level=info msg="Executing migration" id="create migration_log table" 2024-03-11T21:27:38.178 INFO:journalctl@ceph.grafana.a.smithi097.stdout:Mar 11 21:27:38 smithi097 ceph-00db0016-dfed-11ee-95c8-87774f69a715-grafana-a[65126]: logger=migrator t=2024-03-11T21:27:38.098344119Z level=info msg="Executing migration" id="create user table" 2024-03-11T21:27:38.178 INFO:journalctl@ceph.grafana.a.smithi097.stdout:Mar 11 21:27:38 smithi097 ceph-00db0016-dfed-11ee-95c8-87774f69a715-grafana-a[65126]: logger=migrator t=2024-03-11T21:27:38.098789713Z level=info msg="Executing migration" id="add unique index user.login" 2024-03-11T21:27:38.178 INFO:journalctl@ceph.grafana.a.smithi097.stdout:Mar 11 21:27:38 smithi097 ceph-00db0016-dfed-11ee-95c8-87774f69a715-grafana-a[65126]: logger=migrator t=2024-03-11T21:27:38.099582517Z level=info msg="Executing migration" id="add unique index user.email" 2024-03-11T21:27:38.178 INFO:journalctl@ceph.grafana.a.smithi097.stdout:Mar 11 21:27:38 smithi097 ceph-00db0016-dfed-11ee-95c8-87774f69a715-grafana-a[65126]: logger=migrator t=2024-03-11T21:27:38.100604953Z level=info msg="Executing migration" id="drop index UQE_user_login - v1" 2024-03-11T21:27:38.178 INFO:journalctl@ceph.grafana.a.smithi097.stdout:Mar 11 21:27:38 smithi097 ceph-00db0016-dfed-11ee-95c8-87774f69a715-grafana-a[65126]: logger=migrator t=2024-03-11T21:27:38.101259358Z level=info msg="Executing migration" id="drop index UQE_user_email - v1" 2024-03-11T21:27:38.178 INFO:journalctl@ceph.grafana.a.smithi097.stdout:Mar 11 21:27:38 smithi097 ceph-00db0016-dfed-11ee-95c8-87774f69a715-grafana-a[65126]: logger=migrator t=2024-03-11T21:27:38.102113224Z level=info msg="Executing migration" id="Rename table user to user_v1 - v1" 2024-03-11T21:27:38.178 INFO:journalctl@ceph.grafana.a.smithi097.stdout:Mar 11 21:27:38 smithi097 bash[65094]: 2bc78bd29e96e10effd38c8e2bda8c99c1d139e375736df4ef16defdbd141aeb 2024-03-11T21:27:38.178 INFO:journalctl@ceph.grafana.a.smithi097.stdout:Mar 11 21:27:38 smithi097 ceph-00db0016-dfed-11ee-95c8-87774f69a715-grafana-a[65126]: logger=migrator t=2024-03-11T21:27:38.102941291Z level=info msg="Executing migration" id="create user table v2" 2024-03-11T21:27:38.178 INFO:journalctl@ceph.grafana.a.smithi097.stdout:Mar 11 21:27:38 smithi097 ceph-00db0016-dfed-11ee-95c8-87774f69a715-grafana-a[65126]: logger=migrator t=2024-03-11T21:27:38.103787096Z level=info msg="Executing migration" id="create index UQE_user_login - v2" 2024-03-11T21:27:38.178 INFO:journalctl@ceph.grafana.a.smithi097.stdout:Mar 11 21:27:38 smithi097 ceph-00db0016-dfed-11ee-95c8-87774f69a715-grafana-a[65126]: logger=migrator t=2024-03-11T21:27:38.104652536Z level=info msg="Executing migration" id="create index UQE_user_email - v2" 2024-03-11T21:27:38.179 INFO:journalctl@ceph.grafana.a.smithi097.stdout:Mar 11 21:27:38 smithi097 systemd[1]: Started Ceph grafana.a for 00db0016-dfed-11ee-95c8-87774f69a715. 2024-03-11T21:27:38.179 INFO:journalctl@ceph.grafana.a.smithi097.stdout:Mar 11 21:27:38 smithi097 ceph-00db0016-dfed-11ee-95c8-87774f69a715-grafana-a[65126]: logger=migrator t=2024-03-11T21:27:38.105560641Z level=info msg="Executing migration" id="copy data_source v1 to v2" 2024-03-11T21:27:38.179 INFO:journalctl@ceph.grafana.a.smithi097.stdout:Mar 11 21:27:38 smithi097 ceph-00db0016-dfed-11ee-95c8-87774f69a715-grafana-a[65126]: logger=migrator t=2024-03-11T21:27:38.106286788Z level=info msg="Executing migration" id="Drop old table user_v1" 2024-03-11T21:27:38.179 INFO:journalctl@ceph.grafana.a.smithi097.stdout:Mar 11 21:27:38 smithi097 ceph-00db0016-dfed-11ee-95c8-87774f69a715-grafana-a[65126]: logger=migrator t=2024-03-11T21:27:38.107099805Z level=info msg="Executing migration" id="Add column help_flags1 to user table" 2024-03-11T21:27:38.180 INFO:journalctl@ceph.grafana.a.smithi097.stdout:Mar 11 21:27:38 smithi097 ceph-00db0016-dfed-11ee-95c8-87774f69a715-grafana-a[65126]: logger=migrator t=2024-03-11T21:27:38.107533908Z level=info msg="Executing migration" id="Update user table charset" 2024-03-11T21:27:38.180 INFO:journalctl@ceph.grafana.a.smithi097.stdout:Mar 11 21:27:38 smithi097 ceph-00db0016-dfed-11ee-95c8-87774f69a715-grafana-a[65126]: logger=migrator t=2024-03-11T21:27:38.107817843Z level=info msg="Executing migration" id="Add last_seen_at column to user" 2024-03-11T21:27:38.180 INFO:journalctl@ceph.grafana.a.smithi097.stdout:Mar 11 21:27:38 smithi097 ceph-00db0016-dfed-11ee-95c8-87774f69a715-grafana-a[65126]: logger=migrator t=2024-03-11T21:27:38.108217156Z level=info msg="Executing migration" id="Add missing user data" 2024-03-11T21:27:38.180 INFO:journalctl@ceph.grafana.a.smithi097.stdout:Mar 11 21:27:38 smithi097 ceph-00db0016-dfed-11ee-95c8-87774f69a715-grafana-a[65126]: logger=migrator t=2024-03-11T21:27:38.108604429Z level=info msg="Executing migration" id="Add is_disabled column to user" 2024-03-11T21:27:38.180 INFO:journalctl@ceph.grafana.a.smithi097.stdout:Mar 11 21:27:38 smithi097 ceph-00db0016-dfed-11ee-95c8-87774f69a715-grafana-a[65126]: logger=migrator t=2024-03-11T21:27:38.109001921Z level=info msg="Executing migration" id="Add index user.login/user.email" 2024-03-11T21:27:38.180 INFO:journalctl@ceph.grafana.a.smithi097.stdout:Mar 11 21:27:38 smithi097 ceph-00db0016-dfed-11ee-95c8-87774f69a715-grafana-a[65126]: logger=migrator t=2024-03-11T21:27:38.109419876Z level=info msg="Executing migration" id="Add is_service_account column to user" 2024-03-11T21:27:38.180 INFO:journalctl@ceph.grafana.a.smithi097.stdout:Mar 11 21:27:38 smithi097 ceph-00db0016-dfed-11ee-95c8-87774f69a715-grafana-a[65126]: logger=migrator t=2024-03-11T21:27:38.109848477Z level=info msg="Executing migration" id="Update is_service_account column to nullable" 2024-03-11T21:27:38.180 INFO:journalctl@ceph.grafana.a.smithi097.stdout:Mar 11 21:27:38 smithi097 ceph-00db0016-dfed-11ee-95c8-87774f69a715-grafana-a[65126]: logger=migrator t=2024-03-11T21:27:38.110997034Z level=info msg="Executing migration" id="create temp user table v1-7" 2024-03-11T21:27:38.180 INFO:journalctl@ceph.grafana.a.smithi097.stdout:Mar 11 21:27:38 smithi097 ceph-00db0016-dfed-11ee-95c8-87774f69a715-grafana-a[65126]: logger=migrator t=2024-03-11T21:27:38.111477945Z level=info msg="Executing migration" id="create index IDX_temp_user_email - v1-7" 2024-03-11T21:27:38.180 INFO:journalctl@ceph.grafana.a.smithi097.stdout:Mar 11 21:27:38 smithi097 ceph-00db0016-dfed-11ee-95c8-87774f69a715-grafana-a[65126]: logger=migrator t=2024-03-11T21:27:38.11193929Z level=info msg="Executing migration" id="create index IDX_temp_user_org_id - v1-7" 2024-03-11T21:27:38.180 INFO:journalctl@ceph.grafana.a.smithi097.stdout:Mar 11 21:27:38 smithi097 ceph-00db0016-dfed-11ee-95c8-87774f69a715-grafana-a[65126]: logger=migrator t=2024-03-11T21:27:38.112360566Z level=info msg="Executing migration" id="create index IDX_temp_user_code - v1-7" 2024-03-11T21:27:38.180 INFO:journalctl@ceph.grafana.a.smithi097.stdout:Mar 11 21:27:38 smithi097 ceph-00db0016-dfed-11ee-95c8-87774f69a715-grafana-a[65126]: logger=migrator t=2024-03-11T21:27:38.112759279Z level=info msg="Executing migration" id="create index IDX_temp_user_status - v1-7" 2024-03-11T21:27:38.180 INFO:journalctl@ceph.grafana.a.smithi097.stdout:Mar 11 21:27:38 smithi097 ceph-00db0016-dfed-11ee-95c8-87774f69a715-grafana-a[65126]: logger=migrator t=2024-03-11T21:27:38.113182222Z level=info msg="Executing migration" id="Update temp_user table charset" 2024-03-11T21:27:38.181 INFO:journalctl@ceph.grafana.a.smithi097.stdout:Mar 11 21:27:38 smithi097 ceph-00db0016-dfed-11ee-95c8-87774f69a715-grafana-a[65126]: logger=migrator t=2024-03-11T21:27:38.113490171Z level=info msg="Executing migration" id="drop index IDX_temp_user_email - v1" 2024-03-11T21:27:38.181 INFO:journalctl@ceph.grafana.a.smithi097.stdout:Mar 11 21:27:38 smithi097 ceph-00db0016-dfed-11ee-95c8-87774f69a715-grafana-a[65126]: logger=migrator t=2024-03-11T21:27:38.113846477Z level=info msg="Executing migration" id="drop index IDX_temp_user_org_id - v1" 2024-03-11T21:27:38.181 INFO:journalctl@ceph.grafana.a.smithi097.stdout:Mar 11 21:27:38 smithi097 ceph-00db0016-dfed-11ee-95c8-87774f69a715-grafana-a[65126]: logger=migrator t=2024-03-11T21:27:38.114292689Z level=info msg="Executing migration" id="drop index IDX_temp_user_code - v1" 2024-03-11T21:27:38.181 INFO:journalctl@ceph.grafana.a.smithi097.stdout:Mar 11 21:27:38 smithi097 ceph-00db0016-dfed-11ee-95c8-87774f69a715-grafana-a[65126]: logger=migrator t=2024-03-11T21:27:38.114672333Z level=info msg="Executing migration" id="drop index IDX_temp_user_status - v1" 2024-03-11T21:27:38.181 INFO:journalctl@ceph.grafana.a.smithi097.stdout:Mar 11 21:27:38 smithi097 ceph-00db0016-dfed-11ee-95c8-87774f69a715-grafana-a[65126]: logger=migrator t=2024-03-11T21:27:38.115052225Z level=info msg="Executing migration" id="Rename table temp_user to temp_user_tmp_qwerty - v1" 2024-03-11T21:27:38.181 INFO:journalctl@ceph.grafana.a.smithi097.stdout:Mar 11 21:27:38 smithi097 ceph-00db0016-dfed-11ee-95c8-87774f69a715-grafana-a[65126]: logger=migrator t=2024-03-11T21:27:38.115749127Z level=info msg="Executing migration" id="create temp_user v2" 2024-03-11T21:27:38.181 INFO:journalctl@ceph.grafana.a.smithi097.stdout:Mar 11 21:27:38 smithi097 ceph-00db0016-dfed-11ee-95c8-87774f69a715-grafana-a[65126]: logger=migrator t=2024-03-11T21:27:38.116200056Z level=info msg="Executing migration" id="create index IDX_temp_user_email - v2" 2024-03-11T21:27:38.181 INFO:journalctl@ceph.grafana.a.smithi097.stdout:Mar 11 21:27:38 smithi097 ceph-00db0016-dfed-11ee-95c8-87774f69a715-grafana-a[65126]: logger=migrator t=2024-03-11T21:27:38.116745752Z level=info msg="Executing migration" id="create index IDX_temp_user_org_id - v2" 2024-03-11T21:27:38.181 INFO:journalctl@ceph.grafana.a.smithi097.stdout:Mar 11 21:27:38 smithi097 ceph-00db0016-dfed-11ee-95c8-87774f69a715-grafana-a[65126]: logger=migrator t=2024-03-11T21:27:38.117281374Z level=info msg="Executing migration" id="create index IDX_temp_user_code - v2" 2024-03-11T21:27:38.181 INFO:journalctl@ceph.grafana.a.smithi097.stdout:Mar 11 21:27:38 smithi097 ceph-00db0016-dfed-11ee-95c8-87774f69a715-grafana-a[65126]: logger=migrator t=2024-03-11T21:27:38.117780379Z level=info msg="Executing migration" id="create index IDX_temp_user_status - v2" 2024-03-11T21:27:38.181 INFO:journalctl@ceph.grafana.a.smithi097.stdout:Mar 11 21:27:38 smithi097 ceph-00db0016-dfed-11ee-95c8-87774f69a715-grafana-a[65126]: logger=migrator t=2024-03-11T21:27:38.118316432Z level=info msg="Executing migration" id="copy temp_user v1 to v2" 2024-03-11T21:27:38.181 INFO:journalctl@ceph.grafana.a.smithi097.stdout:Mar 11 21:27:38 smithi097 ceph-00db0016-dfed-11ee-95c8-87774f69a715-grafana-a[65126]: logger=migrator t=2024-03-11T21:27:38.118668906Z level=info msg="Executing migration" id="drop temp_user_tmp_qwerty" 2024-03-11T21:27:38.181 INFO:journalctl@ceph.grafana.a.smithi097.stdout:Mar 11 21:27:38 smithi097 ceph-00db0016-dfed-11ee-95c8-87774f69a715-grafana-a[65126]: logger=migrator t=2024-03-11T21:27:38.119091301Z level=info msg="Executing migration" id="Set created for temp users that will otherwise prematurely expire" 2024-03-11T21:27:38.181 INFO:journalctl@ceph.grafana.a.smithi097.stdout:Mar 11 21:27:38 smithi097 ceph-00db0016-dfed-11ee-95c8-87774f69a715-grafana-a[65126]: logger=migrator t=2024-03-11T21:27:38.119586939Z level=info msg="Executing migration" id="create star table" 2024-03-11T21:27:38.181 INFO:journalctl@ceph.grafana.a.smithi097.stdout:Mar 11 21:27:38 smithi097 ceph-00db0016-dfed-11ee-95c8-87774f69a715-grafana-a[65126]: logger=migrator t=2024-03-11T21:27:38.120017945Z level=info msg="Executing migration" id="add unique index star.user_id_dashboard_id" 2024-03-11T21:27:38.182 INFO:journalctl@ceph.grafana.a.smithi097.stdout:Mar 11 21:27:38 smithi097 ceph-00db0016-dfed-11ee-95c8-87774f69a715-grafana-a[65126]: logger=migrator t=2024-03-11T21:27:38.120457113Z level=info msg="Executing migration" id="create org table v1" 2024-03-11T21:27:38.182 INFO:journalctl@ceph.grafana.a.smithi097.stdout:Mar 11 21:27:38 smithi097 ceph-00db0016-dfed-11ee-95c8-87774f69a715-grafana-a[65126]: logger=migrator t=2024-03-11T21:27:38.120856402Z level=info msg="Executing migration" id="create index UQE_org_name - v1" 2024-03-11T21:27:38.182 INFO:journalctl@ceph.grafana.a.smithi097.stdout:Mar 11 21:27:38 smithi097 ceph-00db0016-dfed-11ee-95c8-87774f69a715-grafana-a[65126]: logger=migrator t=2024-03-11T21:27:38.121274554Z level=info msg="Executing migration" id="create org_user table v1" 2024-03-11T21:27:38.182 INFO:journalctl@ceph.grafana.a.smithi097.stdout:Mar 11 21:27:38 smithi097 ceph-00db0016-dfed-11ee-95c8-87774f69a715-grafana-a[65126]: logger=migrator t=2024-03-11T21:27:38.121662377Z level=info msg="Executing migration" id="create index IDX_org_user_org_id - v1" 2024-03-11T21:27:38.182 INFO:journalctl@ceph.grafana.a.smithi097.stdout:Mar 11 21:27:38 smithi097 ceph-00db0016-dfed-11ee-95c8-87774f69a715-grafana-a[65126]: logger=migrator t=2024-03-11T21:27:38.122066759Z level=info msg="Executing migration" id="create index UQE_org_user_org_id_user_id - v1" 2024-03-11T21:27:38.182 INFO:journalctl@ceph.grafana.a.smithi097.stdout:Mar 11 21:27:38 smithi097 ceph-00db0016-dfed-11ee-95c8-87774f69a715-grafana-a[65126]: logger=migrator t=2024-03-11T21:27:38.122600174Z level=info msg="Executing migration" id="create index IDX_org_user_user_id - v1" 2024-03-11T21:27:38.182 INFO:journalctl@ceph.grafana.a.smithi097.stdout:Mar 11 21:27:38 smithi097 ceph-00db0016-dfed-11ee-95c8-87774f69a715-grafana-a[65126]: logger=migrator t=2024-03-11T21:27:38.123064902Z level=info msg="Executing migration" id="Update org table charset" 2024-03-11T21:27:38.182 INFO:journalctl@ceph.grafana.a.smithi097.stdout:Mar 11 21:27:38 smithi097 ceph-00db0016-dfed-11ee-95c8-87774f69a715-grafana-a[65126]: logger=migrator t=2024-03-11T21:27:38.123431216Z level=info msg="Executing migration" id="Update org_user table charset" 2024-03-11T21:27:38.182 INFO:journalctl@ceph.grafana.a.smithi097.stdout:Mar 11 21:27:38 smithi097 ceph-00db0016-dfed-11ee-95c8-87774f69a715-grafana-a[65126]: logger=migrator t=2024-03-11T21:27:38.12372481Z level=info msg="Executing migration" id="Migrate all Read Only Viewers to Viewers" 2024-03-11T21:27:38.182 INFO:journalctl@ceph.grafana.a.smithi097.stdout:Mar 11 21:27:38 smithi097 ceph-00db0016-dfed-11ee-95c8-87774f69a715-grafana-a[65126]: logger=migrator t=2024-03-11T21:27:38.124037699Z level=info msg="Executing migration" id="create dashboard table" 2024-03-11T21:27:38.182 INFO:journalctl@ceph.grafana.a.smithi097.stdout:Mar 11 21:27:38 smithi097 ceph-00db0016-dfed-11ee-95c8-87774f69a715-grafana-a[65126]: logger=migrator t=2024-03-11T21:27:38.124491972Z level=info msg="Executing migration" id="add index dashboard.account_id" 2024-03-11T21:27:38.182 INFO:journalctl@ceph.grafana.a.smithi097.stdout:Mar 11 21:27:38 smithi097 ceph-00db0016-dfed-11ee-95c8-87774f69a715-grafana-a[65126]: logger=migrator t=2024-03-11T21:27:38.124907569Z level=info msg="Executing migration" id="add unique index dashboard_account_id_slug" 2024-03-11T21:27:38.182 INFO:journalctl@ceph.grafana.a.smithi097.stdout:Mar 11 21:27:38 smithi097 ceph-00db0016-dfed-11ee-95c8-87774f69a715-grafana-a[65126]: logger=migrator t=2024-03-11T21:27:38.12534764Z level=info msg="Executing migration" id="create dashboard_tag table" 2024-03-11T21:27:38.182 INFO:journalctl@ceph.grafana.a.smithi097.stdout:Mar 11 21:27:38 smithi097 ceph-00db0016-dfed-11ee-95c8-87774f69a715-grafana-a[65126]: logger=migrator t=2024-03-11T21:27:38.12576576Z level=info msg="Executing migration" id="add unique index dashboard_tag.dasboard_id_term" 2024-03-11T21:27:38.183 INFO:journalctl@ceph.grafana.a.smithi097.stdout:Mar 11 21:27:38 smithi097 ceph-00db0016-dfed-11ee-95c8-87774f69a715-grafana-a[65126]: logger=migrator t=2024-03-11T21:27:38.126644052Z level=info msg="Executing migration" id="drop index UQE_dashboard_tag_dashboard_id_term - v1" 2024-03-11T21:27:38.183 INFO:journalctl@ceph.grafana.a.smithi097.stdout:Mar 11 21:27:38 smithi097 ceph-00db0016-dfed-11ee-95c8-87774f69a715-grafana-a[65126]: logger=migrator t=2024-03-11T21:27:38.127525326Z level=info msg="Executing migration" id="Rename table dashboard to dashboard_v1 - v1" 2024-03-11T21:27:38.183 INFO:journalctl@ceph.grafana.a.smithi097.stdout:Mar 11 21:27:38 smithi097 ceph-00db0016-dfed-11ee-95c8-87774f69a715-grafana-a[65126]: logger=migrator t=2024-03-11T21:27:38.128432834Z level=info msg="Executing migration" id="create dashboard v2" 2024-03-11T21:27:38.183 INFO:journalctl@ceph.grafana.a.smithi097.stdout:Mar 11 21:27:38 smithi097 ceph-00db0016-dfed-11ee-95c8-87774f69a715-grafana-a[65126]: logger=migrator t=2024-03-11T21:27:38.129278344Z level=info msg="Executing migration" id="create index IDX_dashboard_org_id - v2" 2024-03-11T21:27:38.183 INFO:journalctl@ceph.grafana.a.smithi097.stdout:Mar 11 21:27:38 smithi097 ceph-00db0016-dfed-11ee-95c8-87774f69a715-grafana-a[65126]: logger=migrator t=2024-03-11T21:27:38.130218451Z level=info msg="Executing migration" id="create index UQE_dashboard_org_id_slug - v2" 2024-03-11T21:27:38.183 INFO:journalctl@ceph.grafana.a.smithi097.stdout:Mar 11 21:27:38 smithi097 ceph-00db0016-dfed-11ee-95c8-87774f69a715-grafana-a[65126]: logger=migrator t=2024-03-11T21:27:38.133035512Z level=info msg="Executing migration" id="copy dashboard v1 to v2" 2024-03-11T21:27:38.183 INFO:journalctl@ceph.grafana.a.smithi097.stdout:Mar 11 21:27:38 smithi097 ceph-00db0016-dfed-11ee-95c8-87774f69a715-grafana-a[65126]: logger=migrator t=2024-03-11T21:27:38.133485618Z level=info msg="Executing migration" id="drop table dashboard_v1" 2024-03-11T21:27:38.183 INFO:journalctl@ceph.grafana.a.smithi097.stdout:Mar 11 21:27:38 smithi097 ceph-00db0016-dfed-11ee-95c8-87774f69a715-grafana-a[65126]: logger=migrator t=2024-03-11T21:27:38.13583671Z level=info msg="Executing migration" id="alter dashboard.data to mediumtext v1" 2024-03-11T21:27:38.183 INFO:journalctl@ceph.grafana.a.smithi097.stdout:Mar 11 21:27:38 smithi097 ceph-00db0016-dfed-11ee-95c8-87774f69a715-grafana-a[65126]: logger=migrator t=2024-03-11T21:27:38.138716854Z level=info msg="Executing migration" id="Add column updated_by in dashboard - v2" 2024-03-11T21:27:38.183 INFO:journalctl@ceph.grafana.a.smithi097.stdout:Mar 11 21:27:38 smithi097 ceph-00db0016-dfed-11ee-95c8-87774f69a715-grafana-a[65126]: logger=migrator t=2024-03-11T21:27:38.141554057Z level=info msg="Executing migration" id="Add column created_by in dashboard - v2" 2024-03-11T21:27:38.183 INFO:journalctl@ceph.grafana.a.smithi097.stdout:Mar 11 21:27:38 smithi097 ceph-00db0016-dfed-11ee-95c8-87774f69a715-grafana-a[65126]: logger=migrator t=2024-03-11T21:27:38.142134511Z level=info msg="Executing migration" id="Add column gnetId in dashboard" 2024-03-11T21:27:38.183 INFO:journalctl@ceph.grafana.a.smithi097.stdout:Mar 11 21:27:38 smithi097 ceph-00db0016-dfed-11ee-95c8-87774f69a715-grafana-a[65126]: logger=migrator t=2024-03-11T21:27:38.144919333Z level=info msg="Executing migration" id="Add index for gnetId in dashboard" 2024-03-11T21:27:38.183 INFO:journalctl@ceph.grafana.a.smithi097.stdout:Mar 11 21:27:38 smithi097 ceph-00db0016-dfed-11ee-95c8-87774f69a715-grafana-a[65126]: logger=migrator t=2024-03-11T21:27:38.147719874Z level=info msg="Executing migration" id="Add column plugin_id in dashboard" 2024-03-11T21:27:38.183 INFO:journalctl@ceph.grafana.a.smithi097.stdout:Mar 11 21:27:38 smithi097 ceph-00db0016-dfed-11ee-95c8-87774f69a715-grafana-a[65126]: logger=migrator t=2024-03-11T21:27:38.156055065Z level=info msg="Executing migration" id="Add index for plugin_id in dashboard" 2024-03-11T21:27:38.184 INFO:journalctl@ceph.grafana.a.smithi097.stdout:Mar 11 21:27:38 smithi097 ceph-00db0016-dfed-11ee-95c8-87774f69a715-grafana-a[65126]: logger=migrator t=2024-03-11T21:27:38.156660536Z level=info msg="Executing migration" id="Add index for dashboard_id in dashboard_tag" 2024-03-11T21:27:38.184 INFO:journalctl@ceph.grafana.a.smithi097.stdout:Mar 11 21:27:38 smithi097 ceph-00db0016-dfed-11ee-95c8-87774f69a715-grafana-a[65126]: logger=migrator t=2024-03-11T21:27:38.157055859Z level=info msg="Executing migration" id="Update dashboard table charset" 2024-03-11T21:27:38.184 INFO:journalctl@ceph.grafana.a.smithi097.stdout:Mar 11 21:27:38 smithi097 ceph-00db0016-dfed-11ee-95c8-87774f69a715-grafana-a[65126]: logger=migrator t=2024-03-11T21:27:38.157370772Z level=info msg="Executing migration" id="Update dashboard_tag table charset" 2024-03-11T21:27:38.184 INFO:journalctl@ceph.grafana.a.smithi097.stdout:Mar 11 21:27:38 smithi097 ceph-00db0016-dfed-11ee-95c8-87774f69a715-grafana-a[65126]: logger=migrator t=2024-03-11T21:27:38.157647954Z level=info msg="Executing migration" id="Add column folder_id in dashboard" 2024-03-11T21:27:38.184 INFO:journalctl@ceph.grafana.a.smithi097.stdout:Mar 11 21:27:38 smithi097 ceph-00db0016-dfed-11ee-95c8-87774f69a715-grafana-a[65126]: logger=migrator t=2024-03-11T21:27:38.158201517Z level=info msg="Executing migration" id="Add column isFolder in dashboard" 2024-03-11T21:27:38.184 INFO:journalctl@ceph.grafana.a.smithi097.stdout:Mar 11 21:27:38 smithi097 ceph-00db0016-dfed-11ee-95c8-87774f69a715-grafana-a[65126]: logger=migrator t=2024-03-11T21:27:38.159006271Z level=info msg="Executing migration" id="Add column has_acl in dashboard" 2024-03-11T21:27:38.184 INFO:journalctl@ceph.grafana.a.smithi097.stdout:Mar 11 21:27:38 smithi097 ceph-00db0016-dfed-11ee-95c8-87774f69a715-grafana-a[65126]: logger=migrator t=2024-03-11T21:27:38.159868471Z level=info msg="Executing migration" id="Add column uid in dashboard" 2024-03-11T21:27:38.184 INFO:journalctl@ceph.grafana.a.smithi097.stdout:Mar 11 21:27:38 smithi097 ceph-00db0016-dfed-11ee-95c8-87774f69a715-grafana-a[65126]: logger=migrator t=2024-03-11T21:27:38.160753844Z level=info msg="Executing migration" id="Update uid column values in dashboard" 2024-03-11T21:27:38.184 INFO:journalctl@ceph.grafana.a.smithi097.stdout:Mar 11 21:27:38 smithi097 ceph-00db0016-dfed-11ee-95c8-87774f69a715-grafana-a[65126]: logger=migrator t=2024-03-11T21:27:38.16161963Z level=info msg="Executing migration" id="Add unique index dashboard_org_id_uid" 2024-03-11T21:27:38.184 INFO:journalctl@ceph.grafana.a.smithi097.stdout:Mar 11 21:27:38 smithi097 ceph-00db0016-dfed-11ee-95c8-87774f69a715-grafana-a[65126]: logger=migrator t=2024-03-11T21:27:38.162483253Z level=info msg="Executing migration" id="Remove unique index org_id_slug" 2024-03-11T21:27:38.184 INFO:journalctl@ceph.grafana.a.smithi097.stdout:Mar 11 21:27:38 smithi097 ceph-00db0016-dfed-11ee-95c8-87774f69a715-grafana-a[65126]: logger=migrator t=2024-03-11T21:27:38.162906064Z level=info msg="Executing migration" id="Update dashboard title length" 2024-03-11T21:27:38.184 INFO:journalctl@ceph.grafana.a.smithi097.stdout:Mar 11 21:27:38 smithi097 ceph-00db0016-dfed-11ee-95c8-87774f69a715-grafana-a[65126]: logger=migrator t=2024-03-11T21:27:38.163766692Z level=info msg="Executing migration" id="Add unique index for dashboard_org_id_title_folder_id" 2024-03-11T21:27:38.184 INFO:journalctl@ceph.grafana.a.smithi097.stdout:Mar 11 21:27:38 smithi097 ceph-00db0016-dfed-11ee-95c8-87774f69a715-grafana-a[65126]: logger=migrator t=2024-03-11T21:27:38.164231435Z level=info msg="Executing migration" id="create dashboard_provisioning" 2024-03-11T21:27:38.184 INFO:journalctl@ceph.grafana.a.smithi097.stdout:Mar 11 21:27:38 smithi097 ceph-00db0016-dfed-11ee-95c8-87774f69a715-grafana-a[65126]: logger=migrator t=2024-03-11T21:27:38.164686797Z level=info msg="Executing migration" id="Rename table dashboard_provisioning to dashboard_provisioning_tmp_qwerty - v1" 2024-03-11T21:27:38.185 INFO:journalctl@ceph.grafana.a.smithi097.stdout:Mar 11 21:27:38 smithi097 ceph-00db0016-dfed-11ee-95c8-87774f69a715-grafana-a[65126]: logger=migrator t=2024-03-11T21:27:38.165621015Z level=info msg="Executing migration" id="create dashboard_provisioning v2" 2024-03-11T21:27:38.185 INFO:journalctl@ceph.grafana.a.smithi097.stdout:Mar 11 21:27:38 smithi097 ceph-00db0016-dfed-11ee-95c8-87774f69a715-grafana-a[65126]: logger=migrator t=2024-03-11T21:27:38.166517865Z level=info msg="Executing migration" id="create index IDX_dashboard_provisioning_dashboard_id - v2" 2024-03-11T21:27:38.185 INFO:journalctl@ceph.grafana.a.smithi097.stdout:Mar 11 21:27:38 smithi097 ceph-00db0016-dfed-11ee-95c8-87774f69a715-grafana-a[65126]: logger=migrator t=2024-03-11T21:27:38.167347603Z level=info msg="Executing migration" id="create index IDX_dashboard_provisioning_dashboard_id_name - v2" 2024-03-11T21:27:38.185 INFO:journalctl@ceph.grafana.a.smithi097.stdout:Mar 11 21:27:38 smithi097 ceph-00db0016-dfed-11ee-95c8-87774f69a715-grafana-a[65126]: logger=migrator t=2024-03-11T21:27:38.168367366Z level=info msg="Executing migration" id="copy dashboard_provisioning v1 to v2" 2024-03-11T21:27:38.185 INFO:journalctl@ceph.grafana.a.smithi097.stdout:Mar 11 21:27:38 smithi097 ceph-00db0016-dfed-11ee-95c8-87774f69a715-grafana-a[65126]: logger=migrator t=2024-03-11T21:27:38.17117372Z level=info msg="Executing migration" id="drop dashboard_provisioning_tmp_qwerty" 2024-03-11T21:27:38.185 INFO:journalctl@ceph.grafana.a.smithi097.stdout:Mar 11 21:27:38 smithi097 ceph-00db0016-dfed-11ee-95c8-87774f69a715-grafana-a[65126]: logger=migrator t=2024-03-11T21:27:38.17395583Z level=info msg="Executing migration" id="Add check_sum column" 2024-03-11T21:27:38.185 INFO:journalctl@ceph.grafana.a.smithi097.stdout:Mar 11 21:27:38 smithi097 ceph-00db0016-dfed-11ee-95c8-87774f69a715-grafana-a[65126]: logger=migrator t=2024-03-11T21:27:38.176850996Z level=info msg="Executing migration" id="Add index for dashboard_title" 2024-03-11T21:27:38.185 INFO:journalctl@ceph.grafana.a.smithi097.stdout:Mar 11 21:27:38 smithi097 ceph-00db0016-dfed-11ee-95c8-87774f69a715-grafana-a[65126]: logger=migrator t=2024-03-11T21:27:38.179563325Z level=info msg="Executing migration" id="delete tags for deleted dashboards" 2024-03-11T21:27:38.185 INFO:journalctl@ceph.grafana.a.smithi097.stdout:Mar 11 21:27:38 smithi097 ceph-00db0016-dfed-11ee-95c8-87774f69a715-grafana-a[65126]: logger=migrator t=2024-03-11T21:27:38.182362162Z level=info msg="Executing migration" id="delete stars for deleted dashboards" 2024-03-11T21:27:38.185 INFO:journalctl@ceph.grafana.a.smithi097.stdout:Mar 11 21:27:38 smithi097 ceph-00db0016-dfed-11ee-95c8-87774f69a715-grafana-a[65126]: logger=migrator t=2024-03-11T21:27:38.185181131Z level=info msg="Executing migration" id="Add index for dashboard_is_folder" 2024-03-11T21:27:38.185 INFO:journalctl@ceph.grafana.a.smithi097.stdout:Mar 11 21:27:38 smithi097 ceph-00db0016-dfed-11ee-95c8-87774f69a715-grafana-a[65126]: logger=migrator t=2024-03-11T21:27:38.187988184Z level=info msg="Executing migration" id="Add isPublic for dashboard" 2024-03-11T21:27:38.185 INFO:journalctl@ceph.grafana.a.smithi097.stdout:Mar 11 21:27:38 smithi097 ceph-00db0016-dfed-11ee-95c8-87774f69a715-grafana-a[65126]: logger=migrator t=2024-03-11T21:27:38.190905311Z level=info msg="Executing migration" id="create data_source table" 2024-03-11T21:27:38.185 INFO:journalctl@ceph.grafana.a.smithi097.stdout:Mar 11 21:27:38 smithi097 ceph-00db0016-dfed-11ee-95c8-87774f69a715-grafana-a[65126]: logger=migrator t=2024-03-11T21:27:38.193761633Z level=info msg="Executing migration" id="add index data_source.account_id" 2024-03-11T21:27:38.185 INFO:journalctl@ceph.grafana.a.smithi097.stdout:Mar 11 21:27:38 smithi097 ceph-00db0016-dfed-11ee-95c8-87774f69a715-grafana-a[65126]: logger=migrator t=2024-03-11T21:27:38.19634732Z level=info msg="Executing migration" id="add unique index data_source.account_id_name" 2024-03-11T21:27:38.221 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:27:38.245 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:27:38.245 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:27:38.245 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:27:38.346 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:27:38.361 INFO:journalctl@ceph.mgr.y.smithi092.stdout:Mar 11 21:27:37 smithi092 ceph-00db0016-dfed-11ee-95c8-87774f69a715-mgr-y[29801]: ::ffff:172.21.15.97 - - [11/Mar/2024:21:27:37] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-11T21:27:38.369 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:27:38.370 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:27:38.370 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:27:38.396 INFO:teuthology.orchestra.run.smithi092.stderr:Inferring config /var/lib/ceph/00db0016-dfed-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T21:27:38.427 INFO:journalctl@ceph.grafana.a.smithi097.stdout:Mar 11 21:27:38 smithi097 ceph-00db0016-dfed-11ee-95c8-87774f69a715-grafana-a[65126]: logger=migrator t=2024-03-11T21:27:38.196914875Z level=info msg="Executing migration" id="drop index IDX_data_source_account_id - v1" 2024-03-11T21:27:38.427 INFO:journalctl@ceph.grafana.a.smithi097.stdout:Mar 11 21:27:38 smithi097 ceph-00db0016-dfed-11ee-95c8-87774f69a715-grafana-a[65126]: logger=migrator t=2024-03-11T21:27:38.199948139Z level=info msg="Executing migration" id="drop index UQE_data_source_account_id_name - v1" 2024-03-11T21:27:38.428 INFO:journalctl@ceph.grafana.a.smithi097.stdout:Mar 11 21:27:38 smithi097 ceph-00db0016-dfed-11ee-95c8-87774f69a715-grafana-a[65126]: logger=migrator t=2024-03-11T21:27:38.202924589Z level=info msg="Executing migration" id="Rename table data_source to data_source_v1 - v1" 2024-03-11T21:27:38.428 INFO:journalctl@ceph.grafana.a.smithi097.stdout:Mar 11 21:27:38 smithi097 ceph-00db0016-dfed-11ee-95c8-87774f69a715-grafana-a[65126]: logger=migrator t=2024-03-11T21:27:38.205984965Z level=info msg="Executing migration" id="create data_source table v2" 2024-03-11T21:27:38.428 INFO:journalctl@ceph.grafana.a.smithi097.stdout:Mar 11 21:27:38 smithi097 ceph-00db0016-dfed-11ee-95c8-87774f69a715-grafana-a[65126]: logger=migrator t=2024-03-11T21:27:38.208635618Z level=info msg="Executing migration" id="create index IDX_data_source_org_id - v2" 2024-03-11T21:27:38.428 INFO:journalctl@ceph.grafana.a.smithi097.stdout:Mar 11 21:27:38 smithi097 ceph-00db0016-dfed-11ee-95c8-87774f69a715-grafana-a[65126]: logger=migrator t=2024-03-11T21:27:38.211582711Z level=info msg="Executing migration" id="create index UQE_data_source_org_id_name - v2" 2024-03-11T21:27:38.428 INFO:journalctl@ceph.grafana.a.smithi097.stdout:Mar 11 21:27:38 smithi097 ceph-00db0016-dfed-11ee-95c8-87774f69a715-grafana-a[65126]: logger=migrator t=2024-03-11T21:27:38.21434551Z level=info msg="Executing migration" id="Drop old table data_source_v1 #2" 2024-03-11T21:27:38.428 INFO:journalctl@ceph.grafana.a.smithi097.stdout:Mar 11 21:27:38 smithi097 ceph-00db0016-dfed-11ee-95c8-87774f69a715-grafana-a[65126]: logger=migrator t=2024-03-11T21:27:38.217442222Z level=info msg="Executing migration" id="Add column with_credentials" 2024-03-11T21:27:38.428 INFO:journalctl@ceph.grafana.a.smithi097.stdout:Mar 11 21:27:38 smithi097 ceph-00db0016-dfed-11ee-95c8-87774f69a715-grafana-a[65126]: logger=migrator t=2024-03-11T21:27:38.22021172Z level=info msg="Executing migration" id="Add secure json data column" 2024-03-11T21:27:38.428 INFO:journalctl@ceph.grafana.a.smithi097.stdout:Mar 11 21:27:38 smithi097 ceph-00db0016-dfed-11ee-95c8-87774f69a715-grafana-a[65126]: logger=migrator t=2024-03-11T21:27:38.220817727Z level=info msg="Executing migration" id="Update data_source table charset" 2024-03-11T21:27:38.428 INFO:journalctl@ceph.grafana.a.smithi097.stdout:Mar 11 21:27:38 smithi097 ceph-00db0016-dfed-11ee-95c8-87774f69a715-grafana-a[65126]: logger=migrator t=2024-03-11T21:27:38.223422097Z level=info msg="Executing migration" id="Update initial version to 1" 2024-03-11T21:27:38.428 INFO:journalctl@ceph.grafana.a.smithi097.stdout:Mar 11 21:27:38 smithi097 ceph-00db0016-dfed-11ee-95c8-87774f69a715-grafana-a[65126]: logger=migrator t=2024-03-11T21:27:38.226052522Z level=info msg="Executing migration" id="Add read_only data column" 2024-03-11T21:27:38.428 INFO:journalctl@ceph.grafana.a.smithi097.stdout:Mar 11 21:27:38 smithi097 ceph-00db0016-dfed-11ee-95c8-87774f69a715-grafana-a[65126]: logger=migrator t=2024-03-11T21:27:38.226642755Z level=info msg="Executing migration" id="Migrate logging ds to loki ds" 2024-03-11T21:27:38.428 INFO:journalctl@ceph.grafana.a.smithi097.stdout:Mar 11 21:27:38 smithi097 ceph-00db0016-dfed-11ee-95c8-87774f69a715-grafana-a[65126]: logger=migrator t=2024-03-11T21:27:38.229350689Z level=info msg="Executing migration" id="Update json_data with nulls" 2024-03-11T21:27:38.428 INFO:journalctl@ceph.grafana.a.smithi097.stdout:Mar 11 21:27:38 smithi097 ceph-00db0016-dfed-11ee-95c8-87774f69a715-grafana-a[65126]: logger=migrator t=2024-03-11T21:27:38.229682606Z level=info msg="Executing migration" id="Add uid column" 2024-03-11T21:27:38.429 INFO:journalctl@ceph.grafana.a.smithi097.stdout:Mar 11 21:27:38 smithi097 ceph-00db0016-dfed-11ee-95c8-87774f69a715-grafana-a[65126]: logger=migrator t=2024-03-11T21:27:38.232095028Z level=info msg="Executing migration" id="Update uid value" 2024-03-11T21:27:38.429 INFO:journalctl@ceph.grafana.a.smithi097.stdout:Mar 11 21:27:38 smithi097 ceph-00db0016-dfed-11ee-95c8-87774f69a715-grafana-a[65126]: logger=migrator t=2024-03-11T21:27:38.232448037Z level=info msg="Executing migration" id="Add unique index datasource_org_id_uid" 2024-03-11T21:27:38.429 INFO:journalctl@ceph.grafana.a.smithi097.stdout:Mar 11 21:27:38 smithi097 ceph-00db0016-dfed-11ee-95c8-87774f69a715-grafana-a[65126]: logger=migrator t=2024-03-11T21:27:38.235259504Z level=info msg="Executing migration" id="add unique index datasource_org_id_is_default" 2024-03-11T21:27:38.429 INFO:journalctl@ceph.grafana.a.smithi097.stdout:Mar 11 21:27:38 smithi097 ceph-00db0016-dfed-11ee-95c8-87774f69a715-grafana-a[65126]: logger=migrator t=2024-03-11T21:27:38.235725865Z level=info msg="Executing migration" id="create api_key table" 2024-03-11T21:27:38.429 INFO:journalctl@ceph.grafana.a.smithi097.stdout:Mar 11 21:27:38 smithi097 ceph-00db0016-dfed-11ee-95c8-87774f69a715-grafana-a[65126]: logger=migrator t=2024-03-11T21:27:38.237777294Z level=info msg="Executing migration" id="add index api_key.account_id" 2024-03-11T21:27:38.429 INFO:journalctl@ceph.grafana.a.smithi097.stdout:Mar 11 21:27:38 smithi097 ceph-00db0016-dfed-11ee-95c8-87774f69a715-grafana-a[65126]: logger=migrator t=2024-03-11T21:27:38.24042044Z level=info msg="Executing migration" id="add index api_key.key" 2024-03-11T21:27:38.429 INFO:journalctl@ceph.grafana.a.smithi097.stdout:Mar 11 21:27:38 smithi097 ceph-00db0016-dfed-11ee-95c8-87774f69a715-grafana-a[65126]: logger=migrator t=2024-03-11T21:27:38.243215405Z level=info msg="Executing migration" id="add index api_key.account_id_name" 2024-03-11T21:27:38.429 INFO:journalctl@ceph.grafana.a.smithi097.stdout:Mar 11 21:27:38 smithi097 ceph-00db0016-dfed-11ee-95c8-87774f69a715-grafana-a[65126]: logger=migrator t=2024-03-11T21:27:38.245816815Z level=info msg="Executing migration" id="drop index IDX_api_key_account_id - v1" 2024-03-11T21:27:38.429 INFO:journalctl@ceph.grafana.a.smithi097.stdout:Mar 11 21:27:38 smithi097 ceph-00db0016-dfed-11ee-95c8-87774f69a715-grafana-a[65126]: logger=migrator t=2024-03-11T21:27:38.24662862Z level=info msg="Executing migration" id="drop index UQE_api_key_key - v1" 2024-03-11T21:27:38.429 INFO:journalctl@ceph.grafana.a.smithi097.stdout:Mar 11 21:27:38 smithi097 ceph-00db0016-dfed-11ee-95c8-87774f69a715-grafana-a[65126]: logger=migrator t=2024-03-11T21:27:38.249286126Z level=info msg="Executing migration" id="drop index UQE_api_key_account_id_name - v1" 2024-03-11T21:27:38.429 INFO:journalctl@ceph.grafana.a.smithi097.stdout:Mar 11 21:27:38 smithi097 ceph-00db0016-dfed-11ee-95c8-87774f69a715-grafana-a[65126]: logger=migrator t=2024-03-11T21:27:38.251888771Z level=info msg="Executing migration" id="Rename table api_key to api_key_v1 - v1" 2024-03-11T21:27:38.429 INFO:journalctl@ceph.grafana.a.smithi097.stdout:Mar 11 21:27:38 smithi097 ceph-00db0016-dfed-11ee-95c8-87774f69a715-grafana-a[65126]: logger=migrator t=2024-03-11T21:27:38.252951106Z level=info msg="Executing migration" id="create api_key table v2" 2024-03-11T21:27:38.429 INFO:journalctl@ceph.grafana.a.smithi097.stdout:Mar 11 21:27:38 smithi097 ceph-00db0016-dfed-11ee-95c8-87774f69a715-grafana-a[65126]: logger=migrator t=2024-03-11T21:27:38.255817775Z level=info msg="Executing migration" id="create index IDX_api_key_org_id - v2" 2024-03-11T21:27:38.429 INFO:journalctl@ceph.grafana.a.smithi097.stdout:Mar 11 21:27:38 smithi097 ceph-00db0016-dfed-11ee-95c8-87774f69a715-grafana-a[65126]: logger=migrator t=2024-03-11T21:27:38.258629664Z level=info msg="Executing migration" id="create index UQE_api_key_key - v2" 2024-03-11T21:27:38.429 INFO:journalctl@ceph.grafana.a.smithi097.stdout:Mar 11 21:27:38 smithi097 ceph-00db0016-dfed-11ee-95c8-87774f69a715-grafana-a[65126]: logger=migrator t=2024-03-11T21:27:38.261511126Z level=info msg="Executing migration" id="create index UQE_api_key_org_id_name - v2" 2024-03-11T21:27:38.430 INFO:journalctl@ceph.grafana.a.smithi097.stdout:Mar 11 21:27:38 smithi097 ceph-00db0016-dfed-11ee-95c8-87774f69a715-grafana-a[65126]: logger=migrator t=2024-03-11T21:27:38.26437983Z level=info msg="Executing migration" id="copy api_key v1 to v2" 2024-03-11T21:27:38.430 INFO:journalctl@ceph.grafana.a.smithi097.stdout:Mar 11 21:27:38 smithi097 ceph-00db0016-dfed-11ee-95c8-87774f69a715-grafana-a[65126]: logger=migrator t=2024-03-11T21:27:38.267022512Z level=info msg="Executing migration" id="Drop old table api_key_v1" 2024-03-11T21:27:38.430 INFO:journalctl@ceph.grafana.a.smithi097.stdout:Mar 11 21:27:38 smithi097 ceph-00db0016-dfed-11ee-95c8-87774f69a715-grafana-a[65126]: logger=migrator t=2024-03-11T21:27:38.270040967Z level=info msg="Executing migration" id="Update api_key table charset" 2024-03-11T21:27:38.430 INFO:journalctl@ceph.grafana.a.smithi097.stdout:Mar 11 21:27:38 smithi097 ceph-00db0016-dfed-11ee-95c8-87774f69a715-grafana-a[65126]: logger=migrator t=2024-03-11T21:27:38.272793824Z level=info msg="Executing migration" id="Add expires to api_key table" 2024-03-11T21:27:38.430 INFO:journalctl@ceph.grafana.a.smithi097.stdout:Mar 11 21:27:38 smithi097 ceph-00db0016-dfed-11ee-95c8-87774f69a715-grafana-a[65126]: logger=migrator t=2024-03-11T21:27:38.273378531Z level=info msg="Executing migration" id="Add service account foreign key" 2024-03-11T21:27:38.430 INFO:journalctl@ceph.grafana.a.smithi097.stdout:Mar 11 21:27:38 smithi097 ceph-00db0016-dfed-11ee-95c8-87774f69a715-grafana-a[65126]: logger=migrator t=2024-03-11T21:27:38.276009266Z level=info msg="Executing migration" id="set service account foreign key to nil if 0" 2024-03-11T21:27:38.430 INFO:journalctl@ceph.grafana.a.smithi097.stdout:Mar 11 21:27:38 smithi097 ceph-00db0016-dfed-11ee-95c8-87774f69a715-grafana-a[65126]: logger=migrator t=2024-03-11T21:27:38.278900272Z level=info msg="Executing migration" id="Add last_used_at to api_key table" 2024-03-11T21:27:38.430 INFO:journalctl@ceph.grafana.a.smithi097.stdout:Mar 11 21:27:38 smithi097 ceph-00db0016-dfed-11ee-95c8-87774f69a715-grafana-a[65126]: logger=migrator t=2024-03-11T21:27:38.282033495Z level=info msg="Executing migration" id="Add is_revoked column to api_key table" 2024-03-11T21:27:38.430 INFO:journalctl@ceph.grafana.a.smithi097.stdout:Mar 11 21:27:38 smithi097 ceph-00db0016-dfed-11ee-95c8-87774f69a715-grafana-a[65126]: logger=migrator t=2024-03-11T21:27:38.2826611Z level=info msg="Executing migration" id="create dashboard_snapshot table v4" 2024-03-11T21:27:38.431 INFO:journalctl@ceph.grafana.a.smithi097.stdout:Mar 11 21:27:38 smithi097 ceph-00db0016-dfed-11ee-95c8-87774f69a715-grafana-a[65126]: logger=migrator t=2024-03-11T21:27:38.285334475Z level=info msg="Executing migration" id="drop table dashboard_snapshot_v4 #1" 2024-03-11T21:27:38.432 INFO:journalctl@ceph.grafana.a.smithi097.stdout:Mar 11 21:27:38 smithi097 ceph-00db0016-dfed-11ee-95c8-87774f69a715-grafana-a[65126]: logger=migrator t=2024-03-11T21:27:38.285731467Z level=info msg="Executing migration" id="create dashboard_snapshot table v5 #2" 2024-03-11T21:27:38.432 INFO:journalctl@ceph.grafana.a.smithi097.stdout:Mar 11 21:27:38 smithi097 ceph-00db0016-dfed-11ee-95c8-87774f69a715-grafana-a[65126]: logger=migrator t=2024-03-11T21:27:38.28795619Z level=info msg="Executing migration" id="create index UQE_dashboard_snapshot_key - v5" 2024-03-11T21:27:38.432 INFO:journalctl@ceph.grafana.a.smithi097.stdout:Mar 11 21:27:38 smithi097 ceph-00db0016-dfed-11ee-95c8-87774f69a715-grafana-a[65126]: logger=migrator t=2024-03-11T21:27:38.288416215Z level=info msg="Executing migration" id="create index UQE_dashboard_snapshot_delete_key - v5" 2024-03-11T21:27:38.432 INFO:journalctl@ceph.grafana.a.smithi097.stdout:Mar 11 21:27:38 smithi097 ceph-00db0016-dfed-11ee-95c8-87774f69a715-grafana-a[65126]: logger=migrator t=2024-03-11T21:27:38.29087787Z level=info msg="Executing migration" id="create index IDX_dashboard_snapshot_user_id - v5" 2024-03-11T21:27:38.432 INFO:journalctl@ceph.grafana.a.smithi097.stdout:Mar 11 21:27:38 smithi097 ceph-00db0016-dfed-11ee-95c8-87774f69a715-grafana-a[65126]: logger=migrator t=2024-03-11T21:27:38.291314936Z level=info msg="Executing migration" id="alter dashboard_snapshot to mediumtext v2" 2024-03-11T21:27:38.432 INFO:journalctl@ceph.grafana.a.smithi097.stdout:Mar 11 21:27:38 smithi097 ceph-00db0016-dfed-11ee-95c8-87774f69a715-grafana-a[65126]: logger=migrator t=2024-03-11T21:27:38.294003565Z level=info msg="Executing migration" id="Update dashboard_snapshot table charset" 2024-03-11T21:27:38.432 INFO:journalctl@ceph.grafana.a.smithi097.stdout:Mar 11 21:27:38 smithi097 ceph-00db0016-dfed-11ee-95c8-87774f69a715-grafana-a[65126]: logger=migrator t=2024-03-11T21:27:38.296690846Z level=info msg="Executing migration" id="Add column external_delete_url to dashboard_snapshots table" 2024-03-11T21:27:38.432 INFO:journalctl@ceph.grafana.a.smithi097.stdout:Mar 11 21:27:38 smithi097 ceph-00db0016-dfed-11ee-95c8-87774f69a715-grafana-a[65126]: logger=migrator t=2024-03-11T21:27:38.299120075Z level=info msg="Executing migration" id="Add encrypted dashboard json column" 2024-03-11T21:27:38.432 INFO:journalctl@ceph.grafana.a.smithi097.stdout:Mar 11 21:27:38 smithi097 ceph-00db0016-dfed-11ee-95c8-87774f69a715-grafana-a[65126]: logger=migrator t=2024-03-11T21:27:38.301761867Z level=info msg="Executing migration" id="Change dashboard_encrypted column to MEDIUMBLOB" 2024-03-11T21:27:38.432 INFO:journalctl@ceph.grafana.a.smithi097.stdout:Mar 11 21:27:38 smithi097 ceph-00db0016-dfed-11ee-95c8-87774f69a715-grafana-a[65126]: logger=migrator t=2024-03-11T21:27:38.304374591Z level=info msg="Executing migration" id="create quota table v1" 2024-03-11T21:27:38.432 INFO:journalctl@ceph.grafana.a.smithi097.stdout:Mar 11 21:27:38 smithi097 ceph-00db0016-dfed-11ee-95c8-87774f69a715-grafana-a[65126]: logger=migrator t=2024-03-11T21:27:38.307283977Z level=info msg="Executing migration" id="create index UQE_quota_org_id_user_id_target - v1" 2024-03-11T21:27:38.432 INFO:journalctl@ceph.grafana.a.smithi097.stdout:Mar 11 21:27:38 smithi097 ceph-00db0016-dfed-11ee-95c8-87774f69a715-grafana-a[65126]: logger=migrator t=2024-03-11T21:27:38.309962022Z level=info msg="Executing migration" id="Update quota table charset" 2024-03-11T21:27:38.432 INFO:journalctl@ceph.grafana.a.smithi097.stdout:Mar 11 21:27:38 smithi097 ceph-00db0016-dfed-11ee-95c8-87774f69a715-grafana-a[65126]: logger=migrator t=2024-03-11T21:27:38.310283894Z level=info msg="Executing migration" id="create plugin_setting table" 2024-03-11T21:27:38.432 INFO:journalctl@ceph.grafana.a.smithi097.stdout:Mar 11 21:27:38 smithi097 ceph-00db0016-dfed-11ee-95c8-87774f69a715-grafana-a[65126]: logger=migrator t=2024-03-11T21:27:38.313086632Z level=info msg="Executing migration" id="create index UQE_plugin_setting_org_id_plugin_id - v1" 2024-03-11T21:27:38.433 INFO:journalctl@ceph.grafana.a.smithi097.stdout:Mar 11 21:27:38 smithi097 ceph-00db0016-dfed-11ee-95c8-87774f69a715-grafana-a[65126]: logger=migrator t=2024-03-11T21:27:38.316046609Z level=info msg="Executing migration" id="Add column plugin_version to plugin_settings" 2024-03-11T21:27:38.433 INFO:journalctl@ceph.grafana.a.smithi097.stdout:Mar 11 21:27:38 smithi097 ceph-00db0016-dfed-11ee-95c8-87774f69a715-grafana-a[65126]: logger=migrator t=2024-03-11T21:27:38.318758053Z level=info msg="Executing migration" id="Update plugin_setting table charset" 2024-03-11T21:27:38.433 INFO:journalctl@ceph.grafana.a.smithi097.stdout:Mar 11 21:27:38 smithi097 ceph-00db0016-dfed-11ee-95c8-87774f69a715-grafana-a[65126]: logger=migrator t=2024-03-11T21:27:38.319181363Z level=info msg="Executing migration" id="create session table" 2024-03-11T21:27:38.433 INFO:journalctl@ceph.grafana.a.smithi097.stdout:Mar 11 21:27:38 smithi097 ceph-00db0016-dfed-11ee-95c8-87774f69a715-grafana-a[65126]: logger=migrator t=2024-03-11T21:27:38.320045978Z level=info msg="Executing migration" id="Drop old table playlist table" 2024-03-11T21:27:38.433 INFO:journalctl@ceph.grafana.a.smithi097.stdout:Mar 11 21:27:38 smithi097 ceph-00db0016-dfed-11ee-95c8-87774f69a715-grafana-a[65126]: logger=migrator t=2024-03-11T21:27:38.321022264Z level=info msg="Executing migration" id="Drop old table playlist_item table" 2024-03-11T21:27:38.433 INFO:journalctl@ceph.grafana.a.smithi097.stdout:Mar 11 21:27:38 smithi097 ceph-00db0016-dfed-11ee-95c8-87774f69a715-grafana-a[65126]: logger=migrator t=2024-03-11T21:27:38.321784196Z level=info msg="Executing migration" id="create playlist table v2" 2024-03-11T21:27:38.433 INFO:journalctl@ceph.grafana.a.smithi097.stdout:Mar 11 21:27:38 smithi097 ceph-00db0016-dfed-11ee-95c8-87774f69a715-grafana-a[65126]: logger=migrator t=2024-03-11T21:27:38.322170044Z level=info msg="Executing migration" id="create playlist item table v2" 2024-03-11T21:27:38.433 INFO:journalctl@ceph.grafana.a.smithi097.stdout:Mar 11 21:27:38 smithi097 ceph-00db0016-dfed-11ee-95c8-87774f69a715-grafana-a[65126]: logger=migrator t=2024-03-11T21:27:38.322990931Z level=info msg="Executing migration" id="Update playlist table charset" 2024-03-11T21:27:38.433 INFO:journalctl@ceph.grafana.a.smithi097.stdout:Mar 11 21:27:38 smithi097 ceph-00db0016-dfed-11ee-95c8-87774f69a715-grafana-a[65126]: logger=migrator t=2024-03-11T21:27:38.323827175Z level=info msg="Executing migration" id="Update playlist_item table charset" 2024-03-11T21:27:38.433 INFO:journalctl@ceph.grafana.a.smithi097.stdout:Mar 11 21:27:38 smithi097 ceph-00db0016-dfed-11ee-95c8-87774f69a715-grafana-a[65126]: logger=migrator t=2024-03-11T21:27:38.324669387Z level=info msg="Executing migration" id="drop preferences table v2" 2024-03-11T21:27:38.433 INFO:journalctl@ceph.grafana.a.smithi097.stdout:Mar 11 21:27:38 smithi097 ceph-00db0016-dfed-11ee-95c8-87774f69a715-grafana-a[65126]: logger=migrator t=2024-03-11T21:27:38.324955473Z level=info msg="Executing migration" id="drop preferences table v3" 2024-03-11T21:27:38.433 INFO:journalctl@ceph.grafana.a.smithi097.stdout:Mar 11 21:27:38 smithi097 ceph-00db0016-dfed-11ee-95c8-87774f69a715-grafana-a[65126]: logger=migrator t=2024-03-11T21:27:38.325821008Z level=info msg="Executing migration" id="create preferences table v3" 2024-03-11T21:27:38.433 INFO:journalctl@ceph.grafana.a.smithi097.stdout:Mar 11 21:27:38 smithi097 ceph-00db0016-dfed-11ee-95c8-87774f69a715-grafana-a[65126]: logger=migrator t=2024-03-11T21:27:38.326764753Z level=info msg="Executing migration" id="Update preferences table charset" 2024-03-11T21:27:38.433 INFO:journalctl@ceph.grafana.a.smithi097.stdout:Mar 11 21:27:38 smithi097 ceph-00db0016-dfed-11ee-95c8-87774f69a715-grafana-a[65126]: logger=migrator t=2024-03-11T21:27:38.327553807Z level=info msg="Executing migration" id="Add column team_id in preferences" 2024-03-11T21:27:38.433 INFO:journalctl@ceph.grafana.a.smithi097.stdout:Mar 11 21:27:38 smithi097 ceph-00db0016-dfed-11ee-95c8-87774f69a715-grafana-a[65126]: logger=migrator t=2024-03-11T21:27:38.328355843Z level=info msg="Executing migration" id="Update team_id column values in preferences" 2024-03-11T21:27:38.434 INFO:journalctl@ceph.grafana.a.smithi097.stdout:Mar 11 21:27:38 smithi097 ceph-00db0016-dfed-11ee-95c8-87774f69a715-grafana-a[65126]: logger=migrator t=2024-03-11T21:27:38.329281925Z level=info msg="Executing migration" id="Add column week_start in preferences" 2024-03-11T21:27:38.434 INFO:journalctl@ceph.grafana.a.smithi097.stdout:Mar 11 21:27:38 smithi097 ceph-00db0016-dfed-11ee-95c8-87774f69a715-grafana-a[65126]: logger=migrator t=2024-03-11T21:27:38.330076855Z level=info msg="Executing migration" id="Add column preferences.json_data" 2024-03-11T21:27:38.434 INFO:journalctl@ceph.grafana.a.smithi097.stdout:Mar 11 21:27:38 smithi097 ceph-00db0016-dfed-11ee-95c8-87774f69a715-grafana-a[65126]: logger=migrator t=2024-03-11T21:27:38.330979919Z level=info msg="Executing migration" id="alter preferences.json_data to mediumtext v1" 2024-03-11T21:27:38.434 INFO:journalctl@ceph.grafana.a.smithi097.stdout:Mar 11 21:27:38 smithi097 ceph-00db0016-dfed-11ee-95c8-87774f69a715-grafana-a[65126]: logger=migrator t=2024-03-11T21:27:38.331282011Z level=info msg="Executing migration" id="Add preferences index org_id" 2024-03-11T21:27:38.434 INFO:journalctl@ceph.grafana.a.smithi097.stdout:Mar 11 21:27:38 smithi097 ceph-00db0016-dfed-11ee-95c8-87774f69a715-grafana-a[65126]: logger=migrator t=2024-03-11T21:27:38.33230459Z level=info msg="Executing migration" id="Add preferences index user_id" 2024-03-11T21:27:38.434 INFO:journalctl@ceph.grafana.a.smithi097.stdout:Mar 11 21:27:38 smithi097 ceph-00db0016-dfed-11ee-95c8-87774f69a715-grafana-a[65126]: logger=migrator t=2024-03-11T21:27:38.333200032Z level=info msg="Executing migration" id="create alert table v1" 2024-03-11T21:27:38.434 INFO:journalctl@ceph.grafana.a.smithi097.stdout:Mar 11 21:27:38 smithi097 ceph-00db0016-dfed-11ee-95c8-87774f69a715-grafana-a[65126]: logger=migrator t=2024-03-11T21:27:38.333869536Z level=info msg="Executing migration" id="add index alert org_id & id " 2024-03-11T21:27:38.434 INFO:journalctl@ceph.grafana.a.smithi097.stdout:Mar 11 21:27:38 smithi097 ceph-00db0016-dfed-11ee-95c8-87774f69a715-grafana-a[65126]: logger=migrator t=2024-03-11T21:27:38.334321559Z level=info msg="Executing migration" id="add index alert state" 2024-03-11T21:27:38.434 INFO:journalctl@ceph.grafana.a.smithi097.stdout:Mar 11 21:27:38 smithi097 ceph-00db0016-dfed-11ee-95c8-87774f69a715-grafana-a[65126]: logger=migrator t=2024-03-11T21:27:38.335097722Z level=info msg="Executing migration" id="add index alert dashboard_id" 2024-03-11T21:27:38.434 INFO:journalctl@ceph.grafana.a.smithi097.stdout:Mar 11 21:27:38 smithi097 ceph-00db0016-dfed-11ee-95c8-87774f69a715-grafana-a[65126]: logger=migrator t=2024-03-11T21:27:38.336001394Z level=info msg="Executing migration" id="Create alert_rule_tag table v1" 2024-03-11T21:27:38.434 INFO:journalctl@ceph.grafana.a.smithi097.stdout:Mar 11 21:27:38 smithi097 ceph-00db0016-dfed-11ee-95c8-87774f69a715-grafana-a[65126]: logger=migrator t=2024-03-11T21:27:38.336725948Z level=info msg="Executing migration" id="Add unique index alert_rule_tag.alert_id_tag_id" 2024-03-11T21:27:38.434 INFO:journalctl@ceph.grafana.a.smithi097.stdout:Mar 11 21:27:38 smithi097 ceph-00db0016-dfed-11ee-95c8-87774f69a715-grafana-a[65126]: logger=migrator t=2024-03-11T21:27:38.337135738Z level=info msg="Executing migration" id="drop index UQE_alert_rule_tag_alert_id_tag_id - v1" 2024-03-11T21:27:38.434 INFO:journalctl@ceph.grafana.a.smithi097.stdout:Mar 11 21:27:38 smithi097 ceph-00db0016-dfed-11ee-95c8-87774f69a715-grafana-a[65126]: logger=migrator t=2024-03-11T21:27:38.337999732Z level=info msg="Executing migration" id="Rename table alert_rule_tag to alert_rule_tag_v1 - v1" 2024-03-11T21:27:38.435 INFO:journalctl@ceph.grafana.a.smithi097.stdout:Mar 11 21:27:38 smithi097 ceph-00db0016-dfed-11ee-95c8-87774f69a715-grafana-a[65126]: logger=migrator t=2024-03-11T21:27:38.339383648Z level=info msg="Executing migration" id="Create alert_rule_tag table v2" 2024-03-11T21:27:38.435 INFO:journalctl@ceph.grafana.a.smithi097.stdout:Mar 11 21:27:38 smithi097 ceph-00db0016-dfed-11ee-95c8-87774f69a715-grafana-a[65126]: logger=migrator t=2024-03-11T21:27:38.340237184Z level=info msg="Executing migration" id="create index UQE_alert_rule_tag_alert_id_tag_id - Add unique index alert_rule_tag.alert_id_tag_id V2" 2024-03-11T21:27:38.435 INFO:journalctl@ceph.grafana.a.smithi097.stdout:Mar 11 21:27:38 smithi097 ceph-00db0016-dfed-11ee-95c8-87774f69a715-grafana-a[65126]: logger=migrator t=2024-03-11T21:27:38.341185363Z level=info msg="Executing migration" id="copy alert_rule_tag v1 to v2" 2024-03-11T21:27:38.435 INFO:journalctl@ceph.grafana.a.smithi097.stdout:Mar 11 21:27:38 smithi097 ceph-00db0016-dfed-11ee-95c8-87774f69a715-grafana-a[65126]: logger=migrator t=2024-03-11T21:27:38.342015573Z level=info msg="Executing migration" id="drop table alert_rule_tag_v1" 2024-03-11T21:27:38.435 INFO:journalctl@ceph.grafana.a.smithi097.stdout:Mar 11 21:27:38 smithi097 ceph-00db0016-dfed-11ee-95c8-87774f69a715-grafana-a[65126]: logger=migrator t=2024-03-11T21:27:38.342705884Z level=info msg="Executing migration" id="create alert_notification table v1" 2024-03-11T21:27:38.435 INFO:journalctl@ceph.grafana.a.smithi097.stdout:Mar 11 21:27:38 smithi097 ceph-00db0016-dfed-11ee-95c8-87774f69a715-grafana-a[65126]: logger=migrator t=2024-03-11T21:27:38.343483332Z level=info msg="Executing migration" id="Add column is_default" 2024-03-11T21:27:38.435 INFO:journalctl@ceph.grafana.a.smithi097.stdout:Mar 11 21:27:38 smithi097 ceph-00db0016-dfed-11ee-95c8-87774f69a715-grafana-a[65126]: logger=migrator t=2024-03-11T21:27:38.344156016Z level=info msg="Executing migration" id="Add column frequency" 2024-03-11T21:27:38.435 INFO:journalctl@ceph.grafana.a.smithi097.stdout:Mar 11 21:27:38 smithi097 ceph-00db0016-dfed-11ee-95c8-87774f69a715-grafana-a[65126]: logger=migrator t=2024-03-11T21:27:38.344784609Z level=info msg="Executing migration" id="Add column send_reminder" 2024-03-11T21:27:38.435 INFO:journalctl@ceph.grafana.a.smithi097.stdout:Mar 11 21:27:38 smithi097 ceph-00db0016-dfed-11ee-95c8-87774f69a715-grafana-a[65126]: logger=migrator t=2024-03-11T21:27:38.345695384Z level=info msg="Executing migration" id="Add column disable_resolve_message" 2024-03-11T21:27:38.435 INFO:journalctl@ceph.grafana.a.smithi097.stdout:Mar 11 21:27:38 smithi097 ceph-00db0016-dfed-11ee-95c8-87774f69a715-grafana-a[65126]: logger=migrator t=2024-03-11T21:27:38.346568416Z level=info msg="Executing migration" id="add index alert_notification org_id & name" 2024-03-11T21:27:38.435 INFO:journalctl@ceph.grafana.a.smithi097.stdout:Mar 11 21:27:38 smithi097 ceph-00db0016-dfed-11ee-95c8-87774f69a715-grafana-a[65126]: logger=migrator t=2024-03-11T21:27:38.349263192Z level=info msg="Executing migration" id="Update alert table charset" 2024-03-11T21:27:38.435 INFO:journalctl@ceph.grafana.a.smithi097.stdout:Mar 11 21:27:38 smithi097 ceph-00db0016-dfed-11ee-95c8-87774f69a715-grafana-a[65126]: logger=migrator t=2024-03-11T21:27:38.34955867Z level=info msg="Executing migration" id="Update alert_notification table charset" 2024-03-11T21:27:38.435 INFO:journalctl@ceph.grafana.a.smithi097.stdout:Mar 11 21:27:38 smithi097 ceph-00db0016-dfed-11ee-95c8-87774f69a715-grafana-a[65126]: logger=migrator t=2024-03-11T21:27:38.352472039Z level=info msg="Executing migration" id="create notification_journal table v1" 2024-03-11T21:27:38.435 INFO:journalctl@ceph.grafana.a.smithi097.stdout:Mar 11 21:27:38 smithi097 ceph-00db0016-dfed-11ee-95c8-87774f69a715-grafana-a[65126]: logger=migrator t=2024-03-11T21:27:38.353145391Z level=info msg="Executing migration" id="add index notification_journal org_id & alert_id & notifier_id" 2024-03-11T21:27:38.435 INFO:journalctl@ceph.grafana.a.smithi097.stdout:Mar 11 21:27:38 smithi097 ceph-00db0016-dfed-11ee-95c8-87774f69a715-grafana-a[65126]: logger=migrator t=2024-03-11T21:27:38.355100126Z level=info msg="Executing migration" id="drop alert_notification_journal" 2024-03-11T21:27:38.436 INFO:journalctl@ceph.grafana.a.smithi097.stdout:Mar 11 21:27:38 smithi097 ceph-00db0016-dfed-11ee-95c8-87774f69a715-grafana-a[65126]: logger=migrator t=2024-03-11T21:27:38.357979596Z level=info msg="Executing migration" id="create alert_notification_state table v1" 2024-03-11T21:27:38.436 INFO:journalctl@ceph.grafana.a.smithi097.stdout:Mar 11 21:27:38 smithi097 ceph-00db0016-dfed-11ee-95c8-87774f69a715-grafana-a[65126]: logger=migrator t=2024-03-11T21:27:38.358374358Z level=info msg="Executing migration" id="add index alert_notification_state org_id & alert_id & notifier_id" 2024-03-11T21:27:38.436 INFO:journalctl@ceph.grafana.a.smithi097.stdout:Mar 11 21:27:38 smithi097 ceph-00db0016-dfed-11ee-95c8-87774f69a715-grafana-a[65126]: logger=migrator t=2024-03-11T21:27:38.361003385Z level=info msg="Executing migration" id="Add for to alert table" 2024-03-11T21:27:38.436 INFO:journalctl@ceph.grafana.a.smithi097.stdout:Mar 11 21:27:38 smithi097 ceph-00db0016-dfed-11ee-95c8-87774f69a715-grafana-a[65126]: logger=migrator t=2024-03-11T21:27:38.363794216Z level=info msg="Executing migration" id="Add column uid in alert_notification" 2024-03-11T21:27:38.436 INFO:journalctl@ceph.grafana.a.smithi097.stdout:Mar 11 21:27:38 smithi097 ceph-00db0016-dfed-11ee-95c8-87774f69a715-grafana-a[65126]: logger=migrator t=2024-03-11T21:27:38.366577336Z level=info msg="Executing migration" id="Update uid column values in alert_notification" 2024-03-11T21:27:38.436 INFO:journalctl@ceph.grafana.a.smithi097.stdout:Mar 11 21:27:38 smithi097 ceph-00db0016-dfed-11ee-95c8-87774f69a715-grafana-a[65126]: logger=migrator t=2024-03-11T21:27:38.36692473Z level=info msg="Executing migration" id="Add unique index alert_notification_org_id_uid" 2024-03-11T21:27:38.436 INFO:journalctl@ceph.grafana.a.smithi097.stdout:Mar 11 21:27:38 smithi097 ceph-00db0016-dfed-11ee-95c8-87774f69a715-grafana-a[65126]: logger=migrator t=2024-03-11T21:27:38.369683928Z level=info msg="Executing migration" id="Remove unique index org_id_name" 2024-03-11T21:27:38.436 INFO:journalctl@ceph.grafana.a.smithi097.stdout:Mar 11 21:27:38 smithi097 ceph-00db0016-dfed-11ee-95c8-87774f69a715-grafana-a[65126]: logger=migrator t=2024-03-11T21:27:38.372348701Z level=info msg="Executing migration" id="Add column secure_settings in alert_notification" 2024-03-11T21:27:38.436 INFO:journalctl@ceph.grafana.a.smithi097.stdout:Mar 11 21:27:38 smithi097 ceph-00db0016-dfed-11ee-95c8-87774f69a715-grafana-a[65126]: logger=migrator t=2024-03-11T21:27:38.375183366Z level=info msg="Executing migration" id="alter alert.settings to mediumtext" 2024-03-11T21:27:38.436 INFO:journalctl@ceph.grafana.a.smithi097.stdout:Mar 11 21:27:38 smithi097 ceph-00db0016-dfed-11ee-95c8-87774f69a715-grafana-a[65126]: logger=migrator t=2024-03-11T21:27:38.375520361Z level=info msg="Executing migration" id="Add non-unique index alert_notification_state_alert_id" 2024-03-11T21:27:38.436 INFO:journalctl@ceph.grafana.a.smithi097.stdout:Mar 11 21:27:38 smithi097 ceph-00db0016-dfed-11ee-95c8-87774f69a715-grafana-a[65126]: logger=migrator t=2024-03-11T21:27:38.378247455Z level=info msg="Executing migration" id="Add non-unique index alert_rule_tag_alert_id" 2024-03-11T21:27:38.436 INFO:journalctl@ceph.grafana.a.smithi097.stdout:Mar 11 21:27:38 smithi097 ceph-00db0016-dfed-11ee-95c8-87774f69a715-grafana-a[65126]: logger=migrator t=2024-03-11T21:27:38.378708668Z level=info msg="Executing migration" id="Drop old annotation table v4" 2024-03-11T21:27:38.436 INFO:journalctl@ceph.grafana.a.smithi097.stdout:Mar 11 21:27:38 smithi097 ceph-00db0016-dfed-11ee-95c8-87774f69a715-grafana-a[65126]: logger=migrator t=2024-03-11T21:27:38.381142743Z level=info msg="Executing migration" id="create annotation table v5" 2024-03-11T21:27:38.436 INFO:journalctl@ceph.grafana.a.smithi097.stdout:Mar 11 21:27:38 smithi097 ceph-00db0016-dfed-11ee-95c8-87774f69a715-grafana-a[65126]: logger=migrator t=2024-03-11T21:27:38.381652822Z level=info msg="Executing migration" id="add index annotation 0 v3" 2024-03-11T21:27:38.437 INFO:journalctl@ceph.grafana.a.smithi097.stdout:Mar 11 21:27:38 smithi097 ceph-00db0016-dfed-11ee-95c8-87774f69a715-grafana-a[65126]: logger=migrator t=2024-03-11T21:27:38.383860394Z level=info msg="Executing migration" id="add index annotation 1 v3" 2024-03-11T21:27:38.437 INFO:journalctl@ceph.grafana.a.smithi097.stdout:Mar 11 21:27:38 smithi097 ceph-00db0016-dfed-11ee-95c8-87774f69a715-grafana-a[65126]: logger=migrator t=2024-03-11T21:27:38.38429584Z level=info msg="Executing migration" id="add index annotation 2 v3" 2024-03-11T21:27:38.437 INFO:journalctl@ceph.grafana.a.smithi097.stdout:Mar 11 21:27:38 smithi097 ceph-00db0016-dfed-11ee-95c8-87774f69a715-grafana-a[65126]: logger=migrator t=2024-03-11T21:27:38.3868015Z level=info msg="Executing migration" id="add index annotation 3 v3" 2024-03-11T21:27:38.437 INFO:journalctl@ceph.grafana.a.smithi097.stdout:Mar 11 21:27:38 smithi097 ceph-00db0016-dfed-11ee-95c8-87774f69a715-grafana-a[65126]: logger=migrator t=2024-03-11T21:27:38.38946114Z level=info msg="Executing migration" id="add index annotation 4 v3" 2024-03-11T21:27:38.437 INFO:journalctl@ceph.grafana.a.smithi097.stdout:Mar 11 21:27:38 smithi097 ceph-00db0016-dfed-11ee-95c8-87774f69a715-grafana-a[65126]: logger=migrator t=2024-03-11T21:27:38.392205969Z level=info msg="Executing migration" id="Update annotation table charset" 2024-03-11T21:27:38.437 INFO:journalctl@ceph.grafana.a.smithi097.stdout:Mar 11 21:27:38 smithi097 ceph-00db0016-dfed-11ee-95c8-87774f69a715-grafana-a[65126]: logger=migrator t=2024-03-11T21:27:38.392512741Z level=info msg="Executing migration" id="Add column region_id to annotation table" 2024-03-11T21:27:38.437 INFO:journalctl@ceph.grafana.a.smithi097.stdout:Mar 11 21:27:38 smithi097 ceph-00db0016-dfed-11ee-95c8-87774f69a715-grafana-a[65126]: logger=migrator t=2024-03-11T21:27:38.395082278Z level=info msg="Executing migration" id="Drop category_id index" 2024-03-11T21:27:38.437 INFO:journalctl@ceph.grafana.a.smithi097.stdout:Mar 11 21:27:38 smithi097 ceph-00db0016-dfed-11ee-95c8-87774f69a715-grafana-a[65126]: logger=migrator t=2024-03-11T21:27:38.395533078Z level=info msg="Executing migration" id="Add column tags to annotation table" 2024-03-11T21:27:38.437 INFO:journalctl@ceph.grafana.a.smithi097.stdout:Mar 11 21:27:38 smithi097 ceph-00db0016-dfed-11ee-95c8-87774f69a715-grafana-a[65126]: logger=migrator t=2024-03-11T21:27:38.397817801Z level=info msg="Executing migration" id="Create annotation_tag table v2" 2024-03-11T21:27:38.437 INFO:journalctl@ceph.grafana.a.smithi097.stdout:Mar 11 21:27:38 smithi097 ceph-00db0016-dfed-11ee-95c8-87774f69a715-grafana-a[65126]: logger=migrator t=2024-03-11T21:27:38.398282111Z level=info msg="Executing migration" id="Add unique index annotation_tag.annotation_id_tag_id" 2024-03-11T21:27:38.437 INFO:journalctl@ceph.grafana.a.smithi097.stdout:Mar 11 21:27:38 smithi097 ceph-00db0016-dfed-11ee-95c8-87774f69a715-grafana-a[65126]: logger=migrator t=2024-03-11T21:27:38.400611966Z level=info msg="Executing migration" id="drop index UQE_annotation_tag_annotation_id_tag_id - v2" 2024-03-11T21:27:38.437 INFO:journalctl@ceph.grafana.a.smithi097.stdout:Mar 11 21:27:38 smithi097 ceph-00db0016-dfed-11ee-95c8-87774f69a715-grafana-a[65126]: logger=migrator t=2024-03-11T21:27:38.401039658Z level=info msg="Executing migration" id="Rename table annotation_tag to annotation_tag_v2 - v2" 2024-03-11T21:27:38.437 INFO:journalctl@ceph.grafana.a.smithi097.stdout:Mar 11 21:27:38 smithi097 ceph-00db0016-dfed-11ee-95c8-87774f69a715-grafana-a[65126]: logger=migrator t=2024-03-11T21:27:38.403892885Z level=info msg="Executing migration" id="Create annotation_tag table v3" 2024-03-11T21:27:38.437 INFO:journalctl@ceph.grafana.a.smithi097.stdout:Mar 11 21:27:38 smithi097 ceph-00db0016-dfed-11ee-95c8-87774f69a715-grafana-a[65126]: logger=migrator t=2024-03-11T21:27:38.406495433Z level=info msg="Executing migration" id="create index UQE_annotation_tag_annotation_id_tag_id - Add unique index annotation_tag.annotation_id_tag_id V3" 2024-03-11T21:27:38.437 INFO:journalctl@ceph.grafana.a.smithi097.stdout:Mar 11 21:27:38 smithi097 ceph-00db0016-dfed-11ee-95c8-87774f69a715-grafana-a[65126]: logger=migrator t=2024-03-11T21:27:38.409172709Z level=info msg="Executing migration" id="copy annotation_tag v2 to v3" 2024-03-11T21:27:38.438 INFO:journalctl@ceph.grafana.a.smithi097.stdout:Mar 11 21:27:38 smithi097 ceph-00db0016-dfed-11ee-95c8-87774f69a715-grafana-a[65126]: logger=migrator t=2024-03-11T21:27:38.409499424Z level=info msg="Executing migration" id="drop table annotation_tag_v2" 2024-03-11T21:27:38.438 INFO:journalctl@ceph.grafana.a.smithi097.stdout:Mar 11 21:27:38 smithi097 ceph-00db0016-dfed-11ee-95c8-87774f69a715-grafana-a[65126]: logger=migrator t=2024-03-11T21:27:38.410322612Z level=info msg="Executing migration" id="Update alert annotations and set TEXT to empty" 2024-03-11T21:27:38.438 INFO:journalctl@ceph.grafana.a.smithi097.stdout:Mar 11 21:27:38 smithi097 ceph-00db0016-dfed-11ee-95c8-87774f69a715-grafana-a[65126]: logger=migrator t=2024-03-11T21:27:38.410633253Z level=info msg="Executing migration" id="Add created time to annotation table" 2024-03-11T21:27:38.438 INFO:journalctl@ceph.grafana.a.smithi097.stdout:Mar 11 21:27:38 smithi097 ceph-00db0016-dfed-11ee-95c8-87774f69a715-grafana-a[65126]: logger=migrator t=2024-03-11T21:27:38.411326772Z level=info msg="Executing migration" id="Add updated time to annotation table" 2024-03-11T21:27:38.438 INFO:journalctl@ceph.grafana.a.smithi097.stdout:Mar 11 21:27:38 smithi097 ceph-00db0016-dfed-11ee-95c8-87774f69a715-grafana-a[65126]: logger=migrator t=2024-03-11T21:27:38.412137251Z level=info msg="Executing migration" id="Add index for created in annotation table" 2024-03-11T21:27:38.438 INFO:journalctl@ceph.grafana.a.smithi097.stdout:Mar 11 21:27:38 smithi097 ceph-00db0016-dfed-11ee-95c8-87774f69a715-grafana-a[65126]: logger=migrator t=2024-03-11T21:27:38.412566044Z level=info msg="Executing migration" id="Add index for updated in annotation table" 2024-03-11T21:27:38.438 INFO:journalctl@ceph.grafana.a.smithi097.stdout:Mar 11 21:27:38 smithi097 ceph-00db0016-dfed-11ee-95c8-87774f69a715-grafana-a[65126]: logger=migrator t=2024-03-11T21:27:38.413372334Z level=info msg="Executing migration" id="Convert existing annotations from seconds to milliseconds" 2024-03-11T21:27:38.438 INFO:journalctl@ceph.grafana.a.smithi097.stdout:Mar 11 21:27:38 smithi097 ceph-00db0016-dfed-11ee-95c8-87774f69a715-grafana-a[65126]: logger=migrator t=2024-03-11T21:27:38.4141926Z level=info msg="Executing migration" id="Add epoch_end column" 2024-03-11T21:27:38.438 INFO:journalctl@ceph.grafana.a.smithi097.stdout:Mar 11 21:27:38 smithi097 ceph-00db0016-dfed-11ee-95c8-87774f69a715-grafana-a[65126]: logger=migrator t=2024-03-11T21:27:38.415053668Z level=info msg="Executing migration" id="Add index for epoch_end" 2024-03-11T21:27:38.438 INFO:journalctl@ceph.grafana.a.smithi097.stdout:Mar 11 21:27:38 smithi097 ceph-00db0016-dfed-11ee-95c8-87774f69a715-grafana-a[65126]: logger=migrator t=2024-03-11T21:27:38.415515275Z level=info msg="Executing migration" id="Make epoch_end the same as epoch" 2024-03-11T21:27:38.439 INFO:journalctl@ceph.grafana.a.smithi097.stdout:Mar 11 21:27:38 smithi097 ceph-00db0016-dfed-11ee-95c8-87774f69a715-grafana-a[65126]: logger=migrator t=2024-03-11T21:27:38.416227456Z level=info msg="Executing migration" id="Move region to single row" 2024-03-11T21:27:38.439 INFO:journalctl@ceph.grafana.a.smithi097.stdout:Mar 11 21:27:38 smithi097 ceph-00db0016-dfed-11ee-95c8-87774f69a715-grafana-a[65126]: logger=migrator t=2024-03-11T21:27:38.416583795Z level=info msg="Executing migration" id="Remove index org_id_epoch from annotation table" 2024-03-11T21:27:38.439 INFO:journalctl@ceph.grafana.a.smithi097.stdout:Mar 11 21:27:38 smithi097 ceph-00db0016-dfed-11ee-95c8-87774f69a715-grafana-a[65126]: logger=migrator t=2024-03-11T21:27:38.417080851Z level=info msg="Executing migration" id="Remove index org_id_dashboard_id_panel_id_epoch from annotation table" 2024-03-11T21:27:38.439 INFO:journalctl@ceph.grafana.a.smithi097.stdout:Mar 11 21:27:38 smithi097 ceph-00db0016-dfed-11ee-95c8-87774f69a715-grafana-a[65126]: logger=migrator t=2024-03-11T21:27:38.417525669Z level=info msg="Executing migration" id="Add index for org_id_dashboard_id_epoch_end_epoch on annotation table" 2024-03-11T21:27:38.439 INFO:journalctl@ceph.grafana.a.smithi097.stdout:Mar 11 21:27:38 smithi097 ceph-00db0016-dfed-11ee-95c8-87774f69a715-grafana-a[65126]: logger=migrator t=2024-03-11T21:27:38.417914893Z level=info msg="Executing migration" id="Add index for org_id_epoch_end_epoch on annotation table" 2024-03-11T21:27:38.439 INFO:journalctl@ceph.grafana.a.smithi097.stdout:Mar 11 21:27:38 smithi097 ceph-00db0016-dfed-11ee-95c8-87774f69a715-grafana-a[65126]: logger=migrator t=2024-03-11T21:27:38.418340831Z level=info msg="Executing migration" id="Remove index org_id_epoch_epoch_end from annotation table" 2024-03-11T21:27:38.439 INFO:journalctl@ceph.grafana.a.smithi097.stdout:Mar 11 21:27:38 smithi097 ceph-00db0016-dfed-11ee-95c8-87774f69a715-grafana-a[65126]: logger=migrator t=2024-03-11T21:27:38.418738495Z level=info msg="Executing migration" id="Add index for alert_id on annotation table" 2024-03-11T21:27:38.439 INFO:journalctl@ceph.grafana.a.smithi097.stdout:Mar 11 21:27:38 smithi097 ceph-00db0016-dfed-11ee-95c8-87774f69a715-grafana-a[65126]: logger=migrator t=2024-03-11T21:27:38.419547849Z level=info msg="Executing migration" id="Increase tags column to length 4096" 2024-03-11T21:27:38.439 INFO:journalctl@ceph.grafana.a.smithi097.stdout:Mar 11 21:27:38 smithi097 ceph-00db0016-dfed-11ee-95c8-87774f69a715-grafana-a[65126]: logger=migrator t=2024-03-11T21:27:38.419875378Z level=info msg="Executing migration" id="create test_data table" 2024-03-11T21:27:38.439 INFO:journalctl@ceph.grafana.a.smithi097.stdout:Mar 11 21:27:38 smithi097 ceph-00db0016-dfed-11ee-95c8-87774f69a715-grafana-a[65126]: logger=migrator t=2024-03-11T21:27:38.420364278Z level=info msg="Executing migration" id="create dashboard_version table v1" 2024-03-11T21:27:38.439 INFO:journalctl@ceph.grafana.a.smithi097.stdout:Mar 11 21:27:38 smithi097 ceph-00db0016-dfed-11ee-95c8-87774f69a715-grafana-a[65126]: logger=migrator t=2024-03-11T21:27:38.421252342Z level=info msg="Executing migration" id="add index dashboard_version.dashboard_id" 2024-03-11T21:27:38.439 INFO:journalctl@ceph.grafana.a.smithi097.stdout:Mar 11 21:27:38 smithi097 ceph-00db0016-dfed-11ee-95c8-87774f69a715-grafana-a[65126]: logger=migrator t=2024-03-11T21:27:38.421669987Z level=info msg="Executing migration" id="add unique index dashboard_version.dashboard_id and dashboard_version.version" 2024-03-11T21:27:38.439 INFO:journalctl@ceph.grafana.a.smithi097.stdout:Mar 11 21:27:38 smithi097 ceph-00db0016-dfed-11ee-95c8-87774f69a715-grafana-a[65126]: logger=migrator t=2024-03-11T21:27:38.422496132Z level=info msg="Executing migration" id="Set dashboard version to 1 where 0" 2024-03-11T21:27:38.439 INFO:journalctl@ceph.grafana.a.smithi097.stdout:Mar 11 21:27:38 smithi097 ceph-00db0016-dfed-11ee-95c8-87774f69a715-grafana-a[65126]: logger=migrator t=2024-03-11T21:27:38.42335435Z level=info msg="Executing migration" id="save existing dashboard data in dashboard_version table v1" 2024-03-11T21:27:38.439 INFO:journalctl@ceph.grafana.a.smithi097.stdout:Mar 11 21:27:38 smithi097 ceph-00db0016-dfed-11ee-95c8-87774f69a715-grafana-a[65126]: logger=migrator t=2024-03-11T21:27:38.42420894Z level=info msg="Executing migration" id="alter dashboard_version.data to mediumtext v1" 2024-03-11T21:27:38.440 INFO:journalctl@ceph.grafana.a.smithi097.stdout:Mar 11 21:27:38 smithi097 ceph-00db0016-dfed-11ee-95c8-87774f69a715-grafana-a[65126]: logger=migrator t=2024-03-11T21:27:38.424504051Z level=info msg="Executing migration" id="create team table" 2024-03-11T21:27:38.440 INFO:journalctl@ceph.grafana.a.smithi097.stdout:Mar 11 21:27:38 smithi097 ceph-00db0016-dfed-11ee-95c8-87774f69a715-grafana-a[65126]: logger=migrator t=2024-03-11T21:27:38.425385524Z level=info msg="Executing migration" id="add index team.org_id" 2024-03-11T21:27:38.440 INFO:journalctl@ceph.grafana.a.smithi097.stdout:Mar 11 21:27:38 smithi097 ceph-00db0016-dfed-11ee-95c8-87774f69a715-grafana-a[65126]: logger=migrator t=2024-03-11T21:27:38.426136493Z level=info msg="Executing migration" id="add unique index team_org_id_name" 2024-03-11T21:27:38.440 INFO:journalctl@ceph.grafana.a.smithi097.stdout:Mar 11 21:27:38 smithi097 ceph-00db0016-dfed-11ee-95c8-87774f69a715-grafana-a[65126]: logger=migrator t=2024-03-11T21:27:38.427089702Z level=info msg="Executing migration" id="create team member table" 2024-03-11T21:27:38.440 INFO:journalctl@ceph.grafana.a.smithi097.stdout:Mar 11 21:27:38 smithi097 ceph-00db0016-dfed-11ee-95c8-87774f69a715-grafana-a[65126]: logger=migrator t=2024-03-11T21:27:38.427923878Z level=info msg="Executing migration" id="add index team_member.org_id" 2024-03-11T21:27:38.440 INFO:journalctl@ceph.grafana.a.smithi097.stdout:Mar 11 21:27:38 smithi097 ceph-00db0016-dfed-11ee-95c8-87774f69a715-grafana-a[65126]: logger=migrator t=2024-03-11T21:27:38.42869604Z level=info msg="Executing migration" id="add unique index team_member_org_id_team_id_user_id" 2024-03-11T21:27:38.440 INFO:journalctl@ceph.grafana.a.smithi097.stdout:Mar 11 21:27:38 smithi097 ceph-00db0016-dfed-11ee-95c8-87774f69a715-grafana-a[65126]: logger=migrator t=2024-03-11T21:27:38.429621379Z level=info msg="Executing migration" id="add index team_member.team_id" 2024-03-11T21:27:38.440 INFO:journalctl@ceph.grafana.a.smithi097.stdout:Mar 11 21:27:38 smithi097 ceph-00db0016-dfed-11ee-95c8-87774f69a715-grafana-a[65126]: logger=migrator t=2024-03-11T21:27:38.430334517Z level=info msg="Executing migration" id="Add column email to team table" 2024-03-11T21:27:38.440 INFO:journalctl@ceph.grafana.a.smithi097.stdout:Mar 11 21:27:38 smithi097 ceph-00db0016-dfed-11ee-95c8-87774f69a715-grafana-a[65126]: logger=migrator t=2024-03-11T21:27:38.431046665Z level=info msg="Executing migration" id="Add column external to team_member table" 2024-03-11T21:27:38.440 INFO:journalctl@ceph.grafana.a.smithi097.stdout:Mar 11 21:27:38 smithi097 ceph-00db0016-dfed-11ee-95c8-87774f69a715-grafana-a[65126]: logger=migrator t=2024-03-11T21:27:38.431928642Z level=info msg="Executing migration" id="Add column permission to team_member table" 2024-03-11T21:27:38.440 INFO:journalctl@ceph.grafana.a.smithi097.stdout:Mar 11 21:27:38 smithi097 ceph-00db0016-dfed-11ee-95c8-87774f69a715-grafana-a[65126]: logger=migrator t=2024-03-11T21:27:38.432717678Z level=info msg="Executing migration" id="create dashboard acl table" 2024-03-11T21:27:38.440 INFO:journalctl@ceph.grafana.a.smithi097.stdout:Mar 11 21:27:38 smithi097 ceph-00db0016-dfed-11ee-95c8-87774f69a715-grafana-a[65126]: logger=migrator t=2024-03-11T21:27:38.435641618Z level=info msg="Executing migration" id="add index dashboard_acl_dashboard_id" 2024-03-11T21:27:38.440 INFO:journalctl@ceph.grafana.a.smithi097.stdout:Mar 11 21:27:38 smithi097 ceph-00db0016-dfed-11ee-95c8-87774f69a715-grafana-a[65126]: logger=migrator t=2024-03-11T21:27:38.436084794Z level=info msg="Executing migration" id="add unique index dashboard_acl_dashboard_id_user_id" 2024-03-11T21:27:38.440 INFO:journalctl@ceph.grafana.a.smithi097.stdout:Mar 11 21:27:38 smithi097 ceph-00db0016-dfed-11ee-95c8-87774f69a715-grafana-a[65126]: logger=migrator t=2024-03-11T21:27:38.436894798Z level=info msg="Executing migration" id="add unique index dashboard_acl_dashboard_id_team_id" 2024-03-11T21:27:38.441 INFO:journalctl@ceph.grafana.a.smithi097.stdout:Mar 11 21:27:38 smithi097 ceph-00db0016-dfed-11ee-95c8-87774f69a715-grafana-a[65126]: logger=migrator t=2024-03-11T21:27:38.437316155Z level=info msg="Executing migration" id="add index dashboard_acl_user_id" 2024-03-11T21:27:38.441 INFO:journalctl@ceph.grafana.a.smithi097.stdout:Mar 11 21:27:38 smithi097 ceph-00db0016-dfed-11ee-95c8-87774f69a715-grafana-a[65126]: logger=migrator t=2024-03-11T21:27:38.438304987Z level=info msg="Executing migration" id="add index dashboard_acl_team_id" 2024-03-11T21:27:38.441 INFO:journalctl@ceph.grafana.a.smithi097.stdout:Mar 11 21:27:38 smithi097 ceph-00db0016-dfed-11ee-95c8-87774f69a715-grafana-a[65126]: logger=migrator t=2024-03-11T21:27:38.438719868Z level=info msg="Executing migration" id="add index dashboard_acl_org_id_role" 2024-03-11T21:27:38.441 INFO:journalctl@ceph.grafana.a.smithi097.stdout:Mar 11 21:27:38 smithi097 ceph-00db0016-dfed-11ee-95c8-87774f69a715-grafana-a[65126]: logger=migrator t=2024-03-11T21:27:38.439712749Z level=info msg="Executing migration" id="add index dashboard_permission" 2024-03-11T21:27:38.441 INFO:journalctl@ceph.grafana.a.smithi097.stdout:Mar 11 21:27:38 smithi097 ceph-00db0016-dfed-11ee-95c8-87774f69a715-grafana-a[65126]: logger=migrator t=2024-03-11T21:27:38.440414547Z level=info msg="Executing migration" id="save default acl rules in dashboard_acl table" 2024-03-11T21:27:38.441 INFO:journalctl@ceph.grafana.a.smithi097.stdout:Mar 11 21:27:38 smithi097 ceph-00db0016-dfed-11ee-95c8-87774f69a715-grafana-a[65126]: logger=migrator t=2024-03-11T21:27:38.441365671Z level=info msg="Executing migration" id="delete acl rules for deleted dashboards and folders" 2024-03-11T21:27:38.441 INFO:journalctl@ceph.grafana.a.smithi097.stdout:Mar 11 21:27:38 smithi097 ceph-00db0016-dfed-11ee-95c8-87774f69a715-grafana-a[65126]: logger=migrator t=2024-03-11T21:27:38.442221639Z level=info msg="Executing migration" id="create tag table" 2024-03-11T21:27:38.441 INFO:journalctl@ceph.grafana.a.smithi097.stdout:Mar 11 21:27:38 smithi097 ceph-00db0016-dfed-11ee-95c8-87774f69a715-grafana-a[65126]: logger=migrator t=2024-03-11T21:27:38.442980918Z level=info msg="Executing migration" id="add index tag.key_value" 2024-03-11T21:27:38.441 INFO:journalctl@ceph.grafana.a.smithi097.stdout:Mar 11 21:27:38 smithi097 ceph-00db0016-dfed-11ee-95c8-87774f69a715-grafana-a[65126]: logger=migrator t=2024-03-11T21:27:38.443856973Z level=info msg="Executing migration" id="create login attempt table" 2024-03-11T21:27:38.441 INFO:journalctl@ceph.grafana.a.smithi097.stdout:Mar 11 21:27:38 smithi097 ceph-00db0016-dfed-11ee-95c8-87774f69a715-grafana-a[65126]: logger=migrator t=2024-03-11T21:27:38.4448461Z level=info msg="Executing migration" id="add index login_attempt.username" 2024-03-11T21:27:38.441 INFO:journalctl@ceph.grafana.a.smithi097.stdout:Mar 11 21:27:38 smithi097 ceph-00db0016-dfed-11ee-95c8-87774f69a715-grafana-a[65126]: logger=migrator t=2024-03-11T21:27:38.445657512Z level=info msg="Executing migration" id="drop index IDX_login_attempt_username - v1" 2024-03-11T21:27:38.441 INFO:journalctl@ceph.grafana.a.smithi097.stdout:Mar 11 21:27:38 smithi097 ceph-00db0016-dfed-11ee-95c8-87774f69a715-grafana-a[65126]: logger=migrator t=2024-03-11T21:27:38.446466675Z level=info msg="Executing migration" id="Rename table login_attempt to login_attempt_tmp_qwerty - v1" 2024-03-11T21:27:38.441 INFO:journalctl@ceph.grafana.a.smithi097.stdout:Mar 11 21:27:38 smithi097 ceph-00db0016-dfed-11ee-95c8-87774f69a715-grafana-a[65126]: logger=migrator t=2024-03-11T21:27:38.448450504Z level=info msg="Executing migration" id="create login_attempt v2" 2024-03-11T21:27:38.472 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:27:38.496 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:27:38.497 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:27:38.497 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:27:38.598 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:27:38.621 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:27:38.622 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:27:38.622 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:27:38.677 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:27:38 smithi097 ceph-mon[33317]: pgmap v54: 148 pgs: 148 active+clean; 582 KiB data, 226 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T21:27:38.677 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:27:38 smithi097 ceph-mon[33317]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-11T21:27:38.677 INFO:journalctl@ceph.grafana.a.smithi097.stdout:Mar 11 21:27:38 smithi097 ceph-00db0016-dfed-11ee-95c8-87774f69a715-grafana-a[65126]: logger=migrator t=2024-03-11T21:27:38.449279091Z level=info msg="Executing migration" id="create index IDX_login_attempt_username - v2" 2024-03-11T21:27:38.678 INFO:journalctl@ceph.grafana.a.smithi097.stdout:Mar 11 21:27:38 smithi097 ceph-00db0016-dfed-11ee-95c8-87774f69a715-grafana-a[65126]: logger=migrator t=2024-03-11T21:27:38.450128986Z level=info msg="Executing migration" id="copy login_attempt v1 to v2" 2024-03-11T21:27:38.678 INFO:journalctl@ceph.grafana.a.smithi097.stdout:Mar 11 21:27:38 smithi097 ceph-00db0016-dfed-11ee-95c8-87774f69a715-grafana-a[65126]: logger=migrator t=2024-03-11T21:27:38.450978991Z level=info msg="Executing migration" id="drop login_attempt_tmp_qwerty" 2024-03-11T21:27:38.678 INFO:journalctl@ceph.grafana.a.smithi097.stdout:Mar 11 21:27:38 smithi097 ceph-00db0016-dfed-11ee-95c8-87774f69a715-grafana-a[65126]: logger=migrator t=2024-03-11T21:27:38.451850598Z level=info msg="Executing migration" id="create user auth table" 2024-03-11T21:27:38.678 INFO:journalctl@ceph.grafana.a.smithi097.stdout:Mar 11 21:27:38 smithi097 ceph-00db0016-dfed-11ee-95c8-87774f69a715-grafana-a[65126]: logger=migrator t=2024-03-11T21:27:38.454663098Z level=info msg="Executing migration" id="create index IDX_user_auth_auth_module_auth_id - v1" 2024-03-11T21:27:38.678 INFO:journalctl@ceph.grafana.a.smithi097.stdout:Mar 11 21:27:38 smithi097 ceph-00db0016-dfed-11ee-95c8-87774f69a715-grafana-a[65126]: logger=migrator t=2024-03-11T21:27:38.457632726Z level=info msg="Executing migration" id="alter user_auth.auth_id to length 190" 2024-03-11T21:27:38.678 INFO:journalctl@ceph.grafana.a.smithi097.stdout:Mar 11 21:27:38 smithi097 ceph-00db0016-dfed-11ee-95c8-87774f69a715-grafana-a[65126]: logger=migrator t=2024-03-11T21:27:38.460462921Z level=info msg="Executing migration" id="Add OAuth access token to user_auth" 2024-03-11T21:27:38.678 INFO:journalctl@ceph.grafana.a.smithi097.stdout:Mar 11 21:27:38 smithi097 ceph-00db0016-dfed-11ee-95c8-87774f69a715-grafana-a[65126]: logger=migrator t=2024-03-11T21:27:38.463254605Z level=info msg="Executing migration" id="Add OAuth refresh token to user_auth" 2024-03-11T21:27:38.678 INFO:journalctl@ceph.grafana.a.smithi097.stdout:Mar 11 21:27:38 smithi097 ceph-00db0016-dfed-11ee-95c8-87774f69a715-grafana-a[65126]: logger=migrator t=2024-03-11T21:27:38.465893039Z level=info msg="Executing migration" id="Add OAuth token type to user_auth" 2024-03-11T21:27:38.678 INFO:journalctl@ceph.grafana.a.smithi097.stdout:Mar 11 21:27:38 smithi097 ceph-00db0016-dfed-11ee-95c8-87774f69a715-grafana-a[65126]: logger=migrator t=2024-03-11T21:27:38.46873019Z level=info msg="Executing migration" id="Add OAuth expiry to user_auth" 2024-03-11T21:27:38.678 INFO:journalctl@ceph.grafana.a.smithi097.stdout:Mar 11 21:27:38 smithi097 ceph-00db0016-dfed-11ee-95c8-87774f69a715-grafana-a[65126]: logger=migrator t=2024-03-11T21:27:38.471715507Z level=info msg="Executing migration" id="Add index to user_id column in user_auth" 2024-03-11T21:27:38.678 INFO:journalctl@ceph.grafana.a.smithi097.stdout:Mar 11 21:27:38 smithi097 ceph-00db0016-dfed-11ee-95c8-87774f69a715-grafana-a[65126]: logger=migrator t=2024-03-11T21:27:38.474438067Z level=info msg="Executing migration" id="Add OAuth ID token to user_auth" 2024-03-11T21:27:38.678 INFO:journalctl@ceph.grafana.a.smithi097.stdout:Mar 11 21:27:38 smithi097 ceph-00db0016-dfed-11ee-95c8-87774f69a715-grafana-a[65126]: logger=migrator t=2024-03-11T21:27:38.477296532Z level=info msg="Executing migration" id="create server_lock table" 2024-03-11T21:27:38.678 INFO:journalctl@ceph.grafana.a.smithi097.stdout:Mar 11 21:27:38 smithi097 ceph-00db0016-dfed-11ee-95c8-87774f69a715-grafana-a[65126]: logger=migrator t=2024-03-11T21:27:38.480001108Z level=info msg="Executing migration" id="add index server_lock.operation_uid" 2024-03-11T21:27:38.679 INFO:journalctl@ceph.grafana.a.smithi097.stdout:Mar 11 21:27:38 smithi097 ceph-00db0016-dfed-11ee-95c8-87774f69a715-grafana-a[65126]: logger=migrator t=2024-03-11T21:27:38.482915641Z level=info msg="Executing migration" id="create user auth token table" 2024-03-11T21:27:38.679 INFO:journalctl@ceph.grafana.a.smithi097.stdout:Mar 11 21:27:38 smithi097 ceph-00db0016-dfed-11ee-95c8-87774f69a715-grafana-a[65126]: logger=migrator t=2024-03-11T21:27:38.485835638Z level=info msg="Executing migration" id="add unique index user_auth_token.auth_token" 2024-03-11T21:27:38.679 INFO:journalctl@ceph.grafana.a.smithi097.stdout:Mar 11 21:27:38 smithi097 ceph-00db0016-dfed-11ee-95c8-87774f69a715-grafana-a[65126]: logger=migrator t=2024-03-11T21:27:38.4888486Z level=info msg="Executing migration" id="add unique index user_auth_token.prev_auth_token" 2024-03-11T21:27:38.679 INFO:journalctl@ceph.grafana.a.smithi097.stdout:Mar 11 21:27:38 smithi097 ceph-00db0016-dfed-11ee-95c8-87774f69a715-grafana-a[65126]: logger=migrator t=2024-03-11T21:27:38.491980894Z level=info msg="Executing migration" id="add index user_auth_token.user_id" 2024-03-11T21:27:38.679 INFO:journalctl@ceph.grafana.a.smithi097.stdout:Mar 11 21:27:38 smithi097 ceph-00db0016-dfed-11ee-95c8-87774f69a715-grafana-a[65126]: logger=migrator t=2024-03-11T21:27:38.494682959Z level=info msg="Executing migration" id="Add revoked_at to the user auth token" 2024-03-11T21:27:38.679 INFO:journalctl@ceph.grafana.a.smithi097.stdout:Mar 11 21:27:38 smithi097 ceph-00db0016-dfed-11ee-95c8-87774f69a715-grafana-a[65126]: logger=migrator t=2024-03-11T21:27:38.497593969Z level=info msg="Executing migration" id="create cache_data table" 2024-03-11T21:27:38.679 INFO:journalctl@ceph.grafana.a.smithi097.stdout:Mar 11 21:27:38 smithi097 ceph-00db0016-dfed-11ee-95c8-87774f69a715-grafana-a[65126]: logger=migrator t=2024-03-11T21:27:38.500586521Z level=info msg="Executing migration" id="add unique index cache_data.cache_key" 2024-03-11T21:27:38.679 INFO:journalctl@ceph.grafana.a.smithi097.stdout:Mar 11 21:27:38 smithi097 ceph-00db0016-dfed-11ee-95c8-87774f69a715-grafana-a[65126]: logger=migrator t=2024-03-11T21:27:38.503498185Z level=info msg="Executing migration" id="create short_url table v1" 2024-03-11T21:27:38.679 INFO:journalctl@ceph.grafana.a.smithi097.stdout:Mar 11 21:27:38 smithi097 ceph-00db0016-dfed-11ee-95c8-87774f69a715-grafana-a[65126]: logger=migrator t=2024-03-11T21:27:38.506030105Z level=info msg="Executing migration" id="add index short_url.org_id-uid" 2024-03-11T21:27:38.679 INFO:journalctl@ceph.grafana.a.smithi097.stdout:Mar 11 21:27:38 smithi097 ceph-00db0016-dfed-11ee-95c8-87774f69a715-grafana-a[65126]: logger=migrator t=2024-03-11T21:27:38.508845161Z level=info msg="Executing migration" id="alter table short_url alter column created_by type to bigint" 2024-03-11T21:27:38.679 INFO:journalctl@ceph.grafana.a.smithi097.stdout:Mar 11 21:27:38 smithi097 ceph-00db0016-dfed-11ee-95c8-87774f69a715-grafana-a[65126]: logger=migrator t=2024-03-11T21:27:38.509219865Z level=info msg="Executing migration" id="delete alert_definition table" 2024-03-11T21:27:38.679 INFO:journalctl@ceph.grafana.a.smithi097.stdout:Mar 11 21:27:38 smithi097 ceph-00db0016-dfed-11ee-95c8-87774f69a715-grafana-a[65126]: logger=migrator t=2024-03-11T21:27:38.509553696Z level=info msg="Executing migration" id="recreate alert_definition table" 2024-03-11T21:27:38.679 INFO:journalctl@ceph.grafana.a.smithi097.stdout:Mar 11 21:27:38 smithi097 ceph-00db0016-dfed-11ee-95c8-87774f69a715-grafana-a[65126]: logger=migrator t=2024-03-11T21:27:38.511205464Z level=info msg="Executing migration" id="add index in alert_definition on org_id and title columns" 2024-03-11T21:27:38.679 INFO:journalctl@ceph.grafana.a.smithi097.stdout:Mar 11 21:27:38 smithi097 ceph-00db0016-dfed-11ee-95c8-87774f69a715-grafana-a[65126]: logger=migrator t=2024-03-11T21:27:38.513742812Z level=info msg="Executing migration" id="add index in alert_definition on org_id and uid columns" 2024-03-11T21:27:38.680 INFO:journalctl@ceph.grafana.a.smithi097.stdout:Mar 11 21:27:38 smithi097 ceph-00db0016-dfed-11ee-95c8-87774f69a715-grafana-a[65126]: logger=migrator t=2024-03-11T21:27:38.514197877Z level=info msg="Executing migration" id="alter alert_definition table data column to mediumtext in mysql" 2024-03-11T21:27:38.680 INFO:journalctl@ceph.grafana.a.smithi097.stdout:Mar 11 21:27:38 smithi097 ceph-00db0016-dfed-11ee-95c8-87774f69a715-grafana-a[65126]: logger=migrator t=2024-03-11T21:27:38.514500054Z level=info msg="Executing migration" id="drop index in alert_definition on org_id and title columns" 2024-03-11T21:27:38.680 INFO:journalctl@ceph.grafana.a.smithi097.stdout:Mar 11 21:27:38 smithi097 ceph-00db0016-dfed-11ee-95c8-87774f69a715-grafana-a[65126]: logger=migrator t=2024-03-11T21:27:38.51488875Z level=info msg="Executing migration" id="drop index in alert_definition on org_id and uid columns" 2024-03-11T21:27:38.680 INFO:journalctl@ceph.grafana.a.smithi097.stdout:Mar 11 21:27:38 smithi097 ceph-00db0016-dfed-11ee-95c8-87774f69a715-grafana-a[65126]: logger=migrator t=2024-03-11T21:27:38.515313164Z level=info msg="Executing migration" id="add unique index in alert_definition on org_id and title columns" 2024-03-11T21:27:38.680 INFO:journalctl@ceph.grafana.a.smithi097.stdout:Mar 11 21:27:38 smithi097 ceph-00db0016-dfed-11ee-95c8-87774f69a715-grafana-a[65126]: logger=migrator t=2024-03-11T21:27:38.515743977Z level=info msg="Executing migration" id="add unique index in alert_definition on org_id and uid columns" 2024-03-11T21:27:38.680 INFO:journalctl@ceph.grafana.a.smithi097.stdout:Mar 11 21:27:38 smithi097 ceph-00db0016-dfed-11ee-95c8-87774f69a715-grafana-a[65126]: logger=migrator t=2024-03-11T21:27:38.51616422Z level=info msg="Executing migration" id="Add column paused in alert_definition" 2024-03-11T21:27:38.680 INFO:journalctl@ceph.grafana.a.smithi097.stdout:Mar 11 21:27:38 smithi097 ceph-00db0016-dfed-11ee-95c8-87774f69a715-grafana-a[65126]: logger=migrator t=2024-03-11T21:27:38.516958786Z level=info msg="Executing migration" id="drop alert_definition table" 2024-03-11T21:27:38.680 INFO:journalctl@ceph.grafana.a.smithi097.stdout:Mar 11 21:27:38 smithi097 ceph-00db0016-dfed-11ee-95c8-87774f69a715-grafana-a[65126]: logger=migrator t=2024-03-11T21:27:38.517409284Z level=info msg="Executing migration" id="delete alert_definition_version table" 2024-03-11T21:27:38.680 INFO:journalctl@ceph.grafana.a.smithi097.stdout:Mar 11 21:27:38 smithi097 ceph-00db0016-dfed-11ee-95c8-87774f69a715-grafana-a[65126]: logger=migrator t=2024-03-11T21:27:38.517717358Z level=info msg="Executing migration" id="recreate alert_definition_version table" 2024-03-11T21:27:38.681 INFO:journalctl@ceph.grafana.a.smithi097.stdout:Mar 11 21:27:38 smithi097 ceph-00db0016-dfed-11ee-95c8-87774f69a715-grafana-a[65126]: logger=migrator t=2024-03-11T21:27:38.518117523Z level=info msg="Executing migration" id="add index in alert_definition_version table on alert_definition_id and version columns" 2024-03-11T21:27:38.681 INFO:journalctl@ceph.grafana.a.smithi097.stdout:Mar 11 21:27:38 smithi097 ceph-00db0016-dfed-11ee-95c8-87774f69a715-grafana-a[65126]: logger=migrator t=2024-03-11T21:27:38.518554288Z level=info msg="Executing migration" id="add index in alert_definition_version table on alert_definition_uid and version columns" 2024-03-11T21:27:38.681 INFO:journalctl@ceph.grafana.a.smithi097.stdout:Mar 11 21:27:38 smithi097 ceph-00db0016-dfed-11ee-95c8-87774f69a715-grafana-a[65126]: logger=migrator t=2024-03-11T21:27:38.518951617Z level=info msg="Executing migration" id="alter alert_definition_version table data column to mediumtext in mysql" 2024-03-11T21:27:38.681 INFO:journalctl@ceph.grafana.a.smithi097.stdout:Mar 11 21:27:38 smithi097 ceph-00db0016-dfed-11ee-95c8-87774f69a715-grafana-a[65126]: logger=migrator t=2024-03-11T21:27:38.519268302Z level=info msg="Executing migration" id="drop alert_definition_version table" 2024-03-11T21:27:38.682 INFO:journalctl@ceph.grafana.a.smithi097.stdout:Mar 11 21:27:38 smithi097 ceph-00db0016-dfed-11ee-95c8-87774f69a715-grafana-a[65126]: logger=migrator t=2024-03-11T21:27:38.519711252Z level=info msg="Executing migration" id="create alert_instance table" 2024-03-11T21:27:38.682 INFO:journalctl@ceph.grafana.a.smithi097.stdout:Mar 11 21:27:38 smithi097 ceph-00db0016-dfed-11ee-95c8-87774f69a715-grafana-a[65126]: logger=migrator t=2024-03-11T21:27:38.520089823Z level=info msg="Executing migration" id="add index in alert_instance table on def_org_id, def_uid and current_state columns" 2024-03-11T21:27:38.682 INFO:journalctl@ceph.grafana.a.smithi097.stdout:Mar 11 21:27:38 smithi097 ceph-00db0016-dfed-11ee-95c8-87774f69a715-grafana-a[65126]: logger=migrator t=2024-03-11T21:27:38.520548655Z level=info msg="Executing migration" id="add index in alert_instance table on def_org_id, current_state columns" 2024-03-11T21:27:38.682 INFO:journalctl@ceph.grafana.a.smithi097.stdout:Mar 11 21:27:38 smithi097 ceph-00db0016-dfed-11ee-95c8-87774f69a715-grafana-a[65126]: logger=migrator t=2024-03-11T21:27:38.520966737Z level=info msg="Executing migration" id="add column current_state_end to alert_instance" 2024-03-11T21:27:38.682 INFO:journalctl@ceph.grafana.a.smithi097.stdout:Mar 11 21:27:38 smithi097 ceph-00db0016-dfed-11ee-95c8-87774f69a715-grafana-a[65126]: logger=migrator t=2024-03-11T21:27:38.521757332Z level=info msg="Executing migration" id="remove index def_org_id, def_uid, current_state on alert_instance" 2024-03-11T21:27:38.682 INFO:journalctl@ceph.grafana.a.smithi097.stdout:Mar 11 21:27:38 smithi097 ceph-00db0016-dfed-11ee-95c8-87774f69a715-grafana-a[65126]: logger=migrator t=2024-03-11T21:27:38.522179645Z level=info msg="Executing migration" id="remove index def_org_id, current_state on alert_instance" 2024-03-11T21:27:38.682 INFO:journalctl@ceph.grafana.a.smithi097.stdout:Mar 11 21:27:38 smithi097 ceph-00db0016-dfed-11ee-95c8-87774f69a715-grafana-a[65126]: logger=migrator t=2024-03-11T21:27:38.522598819Z level=info msg="Executing migration" id="rename def_org_id to rule_org_id in alert_instance" 2024-03-11T21:27:38.682 INFO:journalctl@ceph.grafana.a.smithi097.stdout:Mar 11 21:27:38 smithi097 ceph-00db0016-dfed-11ee-95c8-87774f69a715-grafana-a[65126]: logger=migrator t=2024-03-11T21:27:38.525964474Z level=info msg="Executing migration" id="rename def_uid to rule_uid in alert_instance" 2024-03-11T21:27:38.682 INFO:journalctl@ceph.grafana.a.smithi097.stdout:Mar 11 21:27:38 smithi097 ceph-00db0016-dfed-11ee-95c8-87774f69a715-grafana-a[65126]: logger=migrator t=2024-03-11T21:27:38.529282126Z level=info msg="Executing migration" id="add index rule_org_id, rule_uid, current_state on alert_instance" 2024-03-11T21:27:38.682 INFO:journalctl@ceph.grafana.a.smithi097.stdout:Mar 11 21:27:38 smithi097 ceph-00db0016-dfed-11ee-95c8-87774f69a715-grafana-a[65126]: logger=migrator t=2024-03-11T21:27:38.529731562Z level=info msg="Executing migration" id="add index rule_org_id, current_state on alert_instance" 2024-03-11T21:27:38.682 INFO:journalctl@ceph.grafana.a.smithi097.stdout:Mar 11 21:27:38 smithi097 ceph-00db0016-dfed-11ee-95c8-87774f69a715-grafana-a[65126]: logger=migrator t=2024-03-11T21:27:38.53016198Z level=info msg="Executing migration" id="add current_reason column related to current_state" 2024-03-11T21:27:38.682 INFO:journalctl@ceph.grafana.a.smithi097.stdout:Mar 11 21:27:38 smithi097 ceph-00db0016-dfed-11ee-95c8-87774f69a715-grafana-a[65126]: logger=migrator t=2024-03-11T21:27:38.530968963Z level=info msg="Executing migration" id="create alert_rule table" 2024-03-11T21:27:38.682 INFO:journalctl@ceph.grafana.a.smithi097.stdout:Mar 11 21:27:38 smithi097 ceph-00db0016-dfed-11ee-95c8-87774f69a715-grafana-a[65126]: logger=migrator t=2024-03-11T21:27:38.531460755Z level=info msg="Executing migration" id="add index in alert_rule on org_id and title columns" 2024-03-11T21:27:38.682 INFO:journalctl@ceph.grafana.a.smithi097.stdout:Mar 11 21:27:38 smithi097 ceph-00db0016-dfed-11ee-95c8-87774f69a715-grafana-a[65126]: logger=migrator t=2024-03-11T21:27:38.531916362Z level=info msg="Executing migration" id="add index in alert_rule on org_id and uid columns" 2024-03-11T21:27:38.683 INFO:journalctl@ceph.grafana.a.smithi097.stdout:Mar 11 21:27:38 smithi097 ceph-00db0016-dfed-11ee-95c8-87774f69a715-grafana-a[65126]: logger=migrator t=2024-03-11T21:27:38.532365236Z level=info msg="Executing migration" id="add index in alert_rule on org_id, namespace_uid, group_uid columns" 2024-03-11T21:27:38.683 INFO:journalctl@ceph.grafana.a.smithi097.stdout:Mar 11 21:27:38 smithi097 ceph-00db0016-dfed-11ee-95c8-87774f69a715-grafana-a[65126]: logger=migrator t=2024-03-11T21:27:38.532832836Z level=info msg="Executing migration" id="alter alert_rule table data column to mediumtext in mysql" 2024-03-11T21:27:38.683 INFO:journalctl@ceph.grafana.a.smithi097.stdout:Mar 11 21:27:38 smithi097 ceph-00db0016-dfed-11ee-95c8-87774f69a715-grafana-a[65126]: logger=migrator t=2024-03-11T21:27:38.533114407Z level=info msg="Executing migration" id="add column for to alert_rule" 2024-03-11T21:27:38.683 INFO:journalctl@ceph.grafana.a.smithi097.stdout:Mar 11 21:27:38 smithi097 ceph-00db0016-dfed-11ee-95c8-87774f69a715-grafana-a[65126]: logger=migrator t=2024-03-11T21:27:38.533918988Z level=info msg="Executing migration" id="add column annotations to alert_rule" 2024-03-11T21:27:38.683 INFO:journalctl@ceph.grafana.a.smithi097.stdout:Mar 11 21:27:38 smithi097 ceph-00db0016-dfed-11ee-95c8-87774f69a715-grafana-a[65126]: logger=migrator t=2024-03-11T21:27:38.534752076Z level=info msg="Executing migration" id="add column labels to alert_rule" 2024-03-11T21:27:38.683 INFO:journalctl@ceph.grafana.a.smithi097.stdout:Mar 11 21:27:38 smithi097 ceph-00db0016-dfed-11ee-95c8-87774f69a715-grafana-a[65126]: logger=migrator t=2024-03-11T21:27:38.535616484Z level=info msg="Executing migration" id="remove unique index from alert_rule on org_id, title columns" 2024-03-11T21:27:38.683 INFO:journalctl@ceph.grafana.a.smithi097.stdout:Mar 11 21:27:38 smithi097 ceph-00db0016-dfed-11ee-95c8-87774f69a715-grafana-a[65126]: logger=migrator t=2024-03-11T21:27:38.536059119Z level=info msg="Executing migration" id="add index in alert_rule on org_id, namespase_uid and title columns" 2024-03-11T21:27:38.683 INFO:journalctl@ceph.grafana.a.smithi097.stdout:Mar 11 21:27:38 smithi097 ceph-00db0016-dfed-11ee-95c8-87774f69a715-grafana-a[65126]: logger=migrator t=2024-03-11T21:27:38.536504225Z level=info msg="Executing migration" id="add dashboard_uid column to alert_rule" 2024-03-11T21:27:38.683 INFO:journalctl@ceph.grafana.a.smithi097.stdout:Mar 11 21:27:38 smithi097 ceph-00db0016-dfed-11ee-95c8-87774f69a715-grafana-a[65126]: logger=migrator t=2024-03-11T21:27:38.537347286Z level=info msg="Executing migration" id="add panel_id column to alert_rule" 2024-03-11T21:27:38.683 INFO:journalctl@ceph.grafana.a.smithi097.stdout:Mar 11 21:27:38 smithi097 ceph-00db0016-dfed-11ee-95c8-87774f69a715-grafana-a[65126]: logger=migrator t=2024-03-11T21:27:38.538147844Z level=info msg="Executing migration" id="add index in alert_rule on org_id, dashboard_uid and panel_id columns" 2024-03-11T21:27:38.683 INFO:journalctl@ceph.grafana.a.smithi097.stdout:Mar 11 21:27:38 smithi097 ceph-00db0016-dfed-11ee-95c8-87774f69a715-grafana-a[65126]: logger=migrator t=2024-03-11T21:27:38.538596798Z level=info msg="Executing migration" id="add rule_group_idx column to alert_rule" 2024-03-11T21:27:38.683 INFO:journalctl@ceph.grafana.a.smithi097.stdout:Mar 11 21:27:38 smithi097 ceph-00db0016-dfed-11ee-95c8-87774f69a715-grafana-a[65126]: logger=migrator t=2024-03-11T21:27:38.539522999Z level=info msg="Executing migration" id="add is_paused column to alert_rule table" 2024-03-11T21:27:38.683 INFO:journalctl@ceph.grafana.a.smithi097.stdout:Mar 11 21:27:38 smithi097 ceph-00db0016-dfed-11ee-95c8-87774f69a715-grafana-a[65126]: logger=migrator t=2024-03-11T21:27:38.540541101Z level=info msg="Executing migration" id="fix is_paused column for alert_rule table" 2024-03-11T21:27:38.683 INFO:journalctl@ceph.grafana.a.smithi097.stdout:Mar 11 21:27:38 smithi097 ceph-00db0016-dfed-11ee-95c8-87774f69a715-grafana-a[65126]: logger=migrator t=2024-03-11T21:27:38.540960698Z level=info msg="Executing migration" id="create alert_rule_version table" 2024-03-11T21:27:38.684 INFO:journalctl@ceph.grafana.a.smithi097.stdout:Mar 11 21:27:38 smithi097 ceph-00db0016-dfed-11ee-95c8-87774f69a715-grafana-a[65126]: logger=migrator t=2024-03-11T21:27:38.541553751Z level=info msg="Executing migration" id="add index in alert_rule_version table on rule_org_id, rule_uid and version columns" 2024-03-11T21:27:38.684 INFO:journalctl@ceph.grafana.a.smithi097.stdout:Mar 11 21:27:38 smithi097 ceph-00db0016-dfed-11ee-95c8-87774f69a715-grafana-a[65126]: logger=migrator t=2024-03-11T21:27:38.542055687Z level=info msg="Executing migration" id="add index in alert_rule_version table on rule_org_id, rule_namespace_uid and rule_group columns" 2024-03-11T21:27:38.684 INFO:journalctl@ceph.grafana.a.smithi097.stdout:Mar 11 21:27:38 smithi097 ceph-00db0016-dfed-11ee-95c8-87774f69a715-grafana-a[65126]: logger=migrator t=2024-03-11T21:27:38.542715684Z level=info msg="Executing migration" id="alter alert_rule_version table data column to mediumtext in mysql" 2024-03-11T21:27:38.684 INFO:journalctl@ceph.grafana.a.smithi097.stdout:Mar 11 21:27:38 smithi097 ceph-00db0016-dfed-11ee-95c8-87774f69a715-grafana-a[65126]: logger=migrator t=2024-03-11T21:27:38.543192062Z level=info msg="Executing migration" id="add column for to alert_rule_version" 2024-03-11T21:27:38.684 INFO:journalctl@ceph.grafana.a.smithi097.stdout:Mar 11 21:27:38 smithi097 ceph-00db0016-dfed-11ee-95c8-87774f69a715-grafana-a[65126]: logger=migrator t=2024-03-11T21:27:38.54433823Z level=info msg="Executing migration" id="add column annotations to alert_rule_version" 2024-03-11T21:27:38.684 INFO:journalctl@ceph.grafana.a.smithi097.stdout:Mar 11 21:27:38 smithi097 ceph-00db0016-dfed-11ee-95c8-87774f69a715-grafana-a[65126]: logger=migrator t=2024-03-11T21:27:38.546998812Z level=info msg="Executing migration" id="add column labels to alert_rule_version" 2024-03-11T21:27:38.684 INFO:journalctl@ceph.grafana.a.smithi097.stdout:Mar 11 21:27:38 smithi097 ceph-00db0016-dfed-11ee-95c8-87774f69a715-grafana-a[65126]: logger=migrator t=2024-03-11T21:27:38.547912289Z level=info msg="Executing migration" id="add rule_group_idx column to alert_rule_version" 2024-03-11T21:27:38.684 INFO:journalctl@ceph.grafana.a.smithi097.stdout:Mar 11 21:27:38 smithi097 ceph-00db0016-dfed-11ee-95c8-87774f69a715-grafana-a[65126]: logger=migrator t=2024-03-11T21:27:38.548758303Z level=info msg="Executing migration" id="add is_paused column to alert_rule_versions table" 2024-03-11T21:27:38.684 INFO:journalctl@ceph.grafana.a.smithi097.stdout:Mar 11 21:27:38 smithi097 ceph-00db0016-dfed-11ee-95c8-87774f69a715-grafana-a[65126]: logger=migrator t=2024-03-11T21:27:38.549638251Z level=info msg="Executing migration" id="fix is_paused column for alert_rule_version table" 2024-03-11T21:27:38.684 INFO:journalctl@ceph.grafana.a.smithi097.stdout:Mar 11 21:27:38 smithi097 ceph-00db0016-dfed-11ee-95c8-87774f69a715-grafana-a[65126]: logger=migrator t=2024-03-11T21:27:38.549945737Z level=info msg="Executing migration" id=create_alert_configuration_table 2024-03-11T21:27:38.684 INFO:journalctl@ceph.grafana.a.smithi097.stdout:Mar 11 21:27:38 smithi097 ceph-00db0016-dfed-11ee-95c8-87774f69a715-grafana-a[65126]: logger=migrator t=2024-03-11T21:27:38.550407619Z level=info msg="Executing migration" id="Add column default in alert_configuration" 2024-03-11T21:27:38.684 INFO:journalctl@ceph.grafana.a.smithi097.stdout:Mar 11 21:27:38 smithi097 ceph-00db0016-dfed-11ee-95c8-87774f69a715-grafana-a[65126]: logger=migrator t=2024-03-11T21:27:38.551245453Z level=info msg="Executing migration" id="alert alert_configuration alertmanager_configuration column from TEXT to MEDIUMTEXT if mysql" 2024-03-11T21:27:38.684 INFO:journalctl@ceph.grafana.a.smithi097.stdout:Mar 11 21:27:38 smithi097 ceph-00db0016-dfed-11ee-95c8-87774f69a715-grafana-a[65126]: logger=migrator t=2024-03-11T21:27:38.551551049Z level=info msg="Executing migration" id="add column org_id in alert_configuration" 2024-03-11T21:27:38.684 INFO:journalctl@ceph.grafana.a.smithi097.stdout:Mar 11 21:27:38 smithi097 ceph-00db0016-dfed-11ee-95c8-87774f69a715-grafana-a[65126]: logger=migrator t=2024-03-11T21:27:38.552438385Z level=info msg="Executing migration" id="add index in alert_configuration table on org_id column" 2024-03-11T21:27:38.685 INFO:journalctl@ceph.grafana.a.smithi097.stdout:Mar 11 21:27:38 smithi097 ceph-00db0016-dfed-11ee-95c8-87774f69a715-grafana-a[65126]: logger=migrator t=2024-03-11T21:27:38.555285632Z level=info msg="Executing migration" id="add configuration_hash column to alert_configuration" 2024-03-11T21:27:38.685 INFO:journalctl@ceph.grafana.a.smithi097.stdout:Mar 11 21:27:38 smithi097 ceph-00db0016-dfed-11ee-95c8-87774f69a715-grafana-a[65126]: logger=migrator t=2024-03-11T21:27:38.557879491Z level=info msg="Executing migration" id=create_ngalert_configuration_table 2024-03-11T21:27:38.685 INFO:journalctl@ceph.grafana.a.smithi097.stdout:Mar 11 21:27:38 smithi097 ceph-00db0016-dfed-11ee-95c8-87774f69a715-grafana-a[65126]: logger=migrator t=2024-03-11T21:27:38.56044768Z level=info msg="Executing migration" id="add index in ngalert_configuration on org_id column" 2024-03-11T21:27:38.685 INFO:journalctl@ceph.grafana.a.smithi097.stdout:Mar 11 21:27:38 smithi097 ceph-00db0016-dfed-11ee-95c8-87774f69a715-grafana-a[65126]: logger=migrator t=2024-03-11T21:27:38.563100255Z level=info msg="Executing migration" id="add column send_alerts_to in ngalert_configuration" 2024-03-11T21:27:38.685 INFO:journalctl@ceph.grafana.a.smithi097.stdout:Mar 11 21:27:38 smithi097 ceph-00db0016-dfed-11ee-95c8-87774f69a715-grafana-a[65126]: logger=migrator t=2024-03-11T21:27:38.565829033Z level=info msg="Executing migration" id="create provenance_type table" 2024-03-11T21:27:38.685 INFO:journalctl@ceph.grafana.a.smithi097.stdout:Mar 11 21:27:38 smithi097 ceph-00db0016-dfed-11ee-95c8-87774f69a715-grafana-a[65126]: logger=migrator t=2024-03-11T21:27:38.568441325Z level=info msg="Executing migration" id="add index to uniquify (record_key, record_type, org_id) columns" 2024-03-11T21:27:38.685 INFO:journalctl@ceph.grafana.a.smithi097.stdout:Mar 11 21:27:38 smithi097 ceph-00db0016-dfed-11ee-95c8-87774f69a715-grafana-a[65126]: logger=migrator t=2024-03-11T21:27:38.571366926Z level=info msg="Executing migration" id="create alert_image table" 2024-03-11T21:27:38.685 INFO:journalctl@ceph.grafana.a.smithi097.stdout:Mar 11 21:27:38 smithi097 ceph-00db0016-dfed-11ee-95c8-87774f69a715-grafana-a[65126]: logger=migrator t=2024-03-11T21:27:38.573972468Z level=info msg="Executing migration" id="add unique index on token to alert_image table" 2024-03-11T21:27:38.685 INFO:journalctl@ceph.grafana.a.smithi097.stdout:Mar 11 21:27:38 smithi097 ceph-00db0016-dfed-11ee-95c8-87774f69a715-grafana-a[65126]: logger=migrator t=2024-03-11T21:27:38.576723892Z level=info msg="Executing migration" id="support longer URLs in alert_image table" 2024-03-11T21:27:38.685 INFO:journalctl@ceph.grafana.a.smithi097.stdout:Mar 11 21:27:38 smithi097 ceph-00db0016-dfed-11ee-95c8-87774f69a715-grafana-a[65126]: logger=migrator t=2024-03-11T21:27:38.579287562Z level=info msg="Executing migration" id=create_alert_configuration_history_table 2024-03-11T21:27:38.686 INFO:journalctl@ceph.grafana.a.smithi097.stdout:Mar 11 21:27:38 smithi097 ceph-00db0016-dfed-11ee-95c8-87774f69a715-grafana-a[65126]: logger=migrator t=2024-03-11T21:27:38.579710458Z level=info msg="Executing migration" id="drop non-unique orgID index on alert_configuration" 2024-03-11T21:27:38.686 INFO:journalctl@ceph.grafana.a.smithi097.stdout:Mar 11 21:27:38 smithi097 ceph-00db0016-dfed-11ee-95c8-87774f69a715-grafana-a[65126]: logger=migrator t=2024-03-11T21:27:38.582366837Z level=info msg="Executing migration" id="drop unique orgID index on alert_configuration if exists" 2024-03-11T21:27:38.686 INFO:journalctl@ceph.grafana.a.smithi097.stdout:Mar 11 21:27:38 smithi097 ceph-00db0016-dfed-11ee-95c8-87774f69a715-grafana-a[65126]: logger=migrator t=2024-03-11T21:27:38.582442206Z level=warn msg="Skipping migration: Already executed, but not recorded in migration log" id="drop unique orgID index on alert_configuration if exists" 2024-03-11T21:27:38.686 INFO:journalctl@ceph.grafana.a.smithi097.stdout:Mar 11 21:27:38 smithi097 ceph-00db0016-dfed-11ee-95c8-87774f69a715-grafana-a[65126]: logger=migrator t=2024-03-11T21:27:38.582717158Z level=info msg="Executing migration" id="extract alertmanager configuration history to separate table" 2024-03-11T21:27:38.686 INFO:journalctl@ceph.grafana.a.smithi097.stdout:Mar 11 21:27:38 smithi097 ceph-00db0016-dfed-11ee-95c8-87774f69a715-grafana-a[65126]: logger=migrator t=2024-03-11T21:27:38.583205504Z level=info msg="Executing migration" id="add unique index on orgID to alert_configuration" 2024-03-11T21:27:38.686 INFO:journalctl@ceph.grafana.a.smithi097.stdout:Mar 11 21:27:38 smithi097 ceph-00db0016-dfed-11ee-95c8-87774f69a715-grafana-a[65126]: logger=migrator t=2024-03-11T21:27:38.583630351Z level=info msg="Executing migration" id="clear migration entry \"remove unified alerting data\"" 2024-03-11T21:27:38.686 INFO:journalctl@ceph.grafana.a.smithi097.stdout:Mar 11 21:27:38 smithi097 ceph-00db0016-dfed-11ee-95c8-87774f69a715-grafana-a[65126]: logger=migrator t=2024-03-11T21:27:38.583745612Z level=info msg="Executing migration" id="move dashboard alerts to unified alerting" 2024-03-11T21:27:38.686 INFO:journalctl@ceph.grafana.a.smithi097.stdout:Mar 11 21:27:38 smithi097 ceph-00db0016-dfed-11ee-95c8-87774f69a715-grafana-a[65126]: logger=migrator t=2024-03-11T21:27:38.583854318Z level=info msg="alerts found to migrate" alerts=0 2024-03-11T21:27:38.686 INFO:journalctl@ceph.grafana.a.smithi097.stdout:Mar 11 21:27:38 smithi097 ceph-00db0016-dfed-11ee-95c8-87774f69a715-grafana-a[65126]: logger=migrator t=2024-03-11T21:27:38.584210608Z level=info msg="Executing migration" id="create library_element table v1" 2024-03-11T21:27:38.686 INFO:journalctl@ceph.grafana.a.smithi097.stdout:Mar 11 21:27:38 smithi097 ceph-00db0016-dfed-11ee-95c8-87774f69a715-grafana-a[65126]: logger=migrator t=2024-03-11T21:27:38.584653879Z level=info msg="Executing migration" id="add index library_element org_id-folder_id-name-kind" 2024-03-11T21:27:38.686 INFO:journalctl@ceph.grafana.a.smithi097.stdout:Mar 11 21:27:38 smithi097 ceph-00db0016-dfed-11ee-95c8-87774f69a715-grafana-a[65126]: logger=migrator t=2024-03-11T21:27:38.585090057Z level=info msg="Executing migration" id="create library_element_connection table v1" 2024-03-11T21:27:38.686 INFO:journalctl@ceph.grafana.a.smithi097.stdout:Mar 11 21:27:38 smithi097 ceph-00db0016-dfed-11ee-95c8-87774f69a715-grafana-a[65126]: logger=migrator t=2024-03-11T21:27:38.58612748Z level=info msg="Executing migration" id="add index library_element_connection element_id-kind-connection_id" 2024-03-11T21:27:38.686 INFO:journalctl@ceph.grafana.a.smithi097.stdout:Mar 11 21:27:38 smithi097 ceph-00db0016-dfed-11ee-95c8-87774f69a715-grafana-a[65126]: logger=migrator t=2024-03-11T21:27:38.586834299Z level=info msg="Executing migration" id="add unique index library_element org_id_uid" 2024-03-11T21:27:38.686 INFO:journalctl@ceph.grafana.a.smithi097.stdout:Mar 11 21:27:38 smithi097 ceph-00db0016-dfed-11ee-95c8-87774f69a715-grafana-a[65126]: logger=migrator t=2024-03-11T21:27:38.587800774Z level=info msg="Executing migration" id="increase max description length to 2048" 2024-03-11T21:27:38.687 INFO:journalctl@ceph.grafana.a.smithi097.stdout:Mar 11 21:27:38 smithi097 ceph-00db0016-dfed-11ee-95c8-87774f69a715-grafana-a[65126]: logger=migrator t=2024-03-11T21:27:38.588520987Z level=info msg="Executing migration" id="clone move dashboard alerts to unified alerting" 2024-03-11T21:27:38.687 INFO:journalctl@ceph.grafana.a.smithi097.stdout:Mar 11 21:27:38 smithi097 ceph-00db0016-dfed-11ee-95c8-87774f69a715-grafana-a[65126]: logger=migrator t=2024-03-11T21:27:38.589399398Z level=info msg="Executing migration" id="create data_keys table" 2024-03-11T21:27:38.687 INFO:journalctl@ceph.grafana.a.smithi097.stdout:Mar 11 21:27:38 smithi097 ceph-00db0016-dfed-11ee-95c8-87774f69a715-grafana-a[65126]: logger=migrator t=2024-03-11T21:27:38.590323592Z level=info msg="Executing migration" id="create secrets table" 2024-03-11T21:27:38.687 INFO:journalctl@ceph.grafana.a.smithi097.stdout:Mar 11 21:27:38 smithi097 ceph-00db0016-dfed-11ee-95c8-87774f69a715-grafana-a[65126]: logger=migrator t=2024-03-11T21:27:38.591231513Z level=info msg="Executing migration" id="rename data_keys name column to id" 2024-03-11T21:27:38.687 INFO:journalctl@ceph.grafana.a.smithi097.stdout:Mar 11 21:27:38 smithi097 ceph-00db0016-dfed-11ee-95c8-87774f69a715-grafana-a[65126]: logger=migrator t=2024-03-11T21:27:38.595441492Z level=info msg="Executing migration" id="add name column into data_keys" 2024-03-11T21:27:38.687 INFO:journalctl@ceph.grafana.a.smithi097.stdout:Mar 11 21:27:38 smithi097 ceph-00db0016-dfed-11ee-95c8-87774f69a715-grafana-a[65126]: logger=migrator t=2024-03-11T21:27:38.596368506Z level=info msg="Executing migration" id="copy data_keys id column values into name" 2024-03-11T21:27:38.687 INFO:journalctl@ceph.grafana.a.smithi097.stdout:Mar 11 21:27:38 smithi097 ceph-00db0016-dfed-11ee-95c8-87774f69a715-grafana-a[65126]: logger=migrator t=2024-03-11T21:27:38.597297705Z level=info msg="Executing migration" id="rename data_keys name column to label" 2024-03-11T21:27:38.687 INFO:journalctl@ceph.grafana.a.smithi097.stdout:Mar 11 21:27:38 smithi097 ceph-00db0016-dfed-11ee-95c8-87774f69a715-grafana-a[65126]: logger=migrator t=2024-03-11T21:27:38.60179059Z level=info msg="Executing migration" id="rename data_keys id column back to name" 2024-03-11T21:27:38.687 INFO:journalctl@ceph.grafana.a.smithi097.stdout:Mar 11 21:27:38 smithi097 ceph-00db0016-dfed-11ee-95c8-87774f69a715-grafana-a[65126]: logger=migrator t=2024-03-11T21:27:38.605847655Z level=info msg="Executing migration" id="create kv_store table v1" 2024-03-11T21:27:38.687 INFO:journalctl@ceph.grafana.a.smithi097.stdout:Mar 11 21:27:38 smithi097 ceph-00db0016-dfed-11ee-95c8-87774f69a715-grafana-a[65126]: logger=migrator t=2024-03-11T21:27:38.606733544Z level=info msg="Executing migration" id="add index kv_store.org_id-namespace-key" 2024-03-11T21:27:38.687 INFO:journalctl@ceph.grafana.a.smithi097.stdout:Mar 11 21:27:38 smithi097 ceph-00db0016-dfed-11ee-95c8-87774f69a715-grafana-a[65126]: logger=migrator t=2024-03-11T21:27:38.607617628Z level=info msg="Executing migration" id="update dashboard_uid and panel_id from existing annotations" 2024-03-11T21:27:38.687 INFO:journalctl@ceph.grafana.a.smithi097.stdout:Mar 11 21:27:38 smithi097 ceph-00db0016-dfed-11ee-95c8-87774f69a715-grafana-a[65126]: logger=migrator t=2024-03-11T21:27:38.608478532Z level=info msg="Executing migration" id="create permission table" 2024-03-11T21:27:38.687 INFO:journalctl@ceph.grafana.a.smithi097.stdout:Mar 11 21:27:38 smithi097 ceph-00db0016-dfed-11ee-95c8-87774f69a715-grafana-a[65126]: logger=migrator t=2024-03-11T21:27:38.609288163Z level=info msg="Executing migration" id="add unique index permission.role_id" 2024-03-11T21:27:38.687 INFO:journalctl@ceph.grafana.a.smithi097.stdout:Mar 11 21:27:38 smithi097 ceph-00db0016-dfed-11ee-95c8-87774f69a715-grafana-a[65126]: logger=migrator t=2024-03-11T21:27:38.610163348Z level=info msg="Executing migration" id="add unique index role_id_action_scope" 2024-03-11T21:27:38.688 INFO:journalctl@ceph.grafana.a.smithi097.stdout:Mar 11 21:27:38 smithi097 ceph-00db0016-dfed-11ee-95c8-87774f69a715-grafana-a[65126]: logger=migrator t=2024-03-11T21:27:38.611018711Z level=info msg="Executing migration" id="create role table" 2024-03-11T21:27:38.688 INFO:journalctl@ceph.grafana.a.smithi097.stdout:Mar 11 21:27:38 smithi097 ceph-00db0016-dfed-11ee-95c8-87774f69a715-grafana-a[65126]: logger=migrator t=2024-03-11T21:27:38.611907476Z level=info msg="Executing migration" id="add column display_name" 2024-03-11T21:27:38.688 INFO:journalctl@ceph.grafana.a.smithi097.stdout:Mar 11 21:27:38 smithi097 ceph-00db0016-dfed-11ee-95c8-87774f69a715-grafana-a[65126]: logger=migrator t=2024-03-11T21:27:38.612906827Z level=info msg="Executing migration" id="add column group_name" 2024-03-11T21:27:38.688 INFO:journalctl@ceph.grafana.a.smithi097.stdout:Mar 11 21:27:38 smithi097 ceph-00db0016-dfed-11ee-95c8-87774f69a715-grafana-a[65126]: logger=migrator t=2024-03-11T21:27:38.613910593Z level=info msg="Executing migration" id="add index role.org_id" 2024-03-11T21:27:38.688 INFO:journalctl@ceph.grafana.a.smithi097.stdout:Mar 11 21:27:38 smithi097 ceph-00db0016-dfed-11ee-95c8-87774f69a715-grafana-a[65126]: logger=migrator t=2024-03-11T21:27:38.614831117Z level=info msg="Executing migration" id="add unique index role_org_id_name" 2024-03-11T21:27:38.688 INFO:journalctl@ceph.grafana.a.smithi097.stdout:Mar 11 21:27:38 smithi097 ceph-00db0016-dfed-11ee-95c8-87774f69a715-grafana-a[65126]: logger=migrator t=2024-03-11T21:27:38.61533207Z level=info msg="Executing migration" id="add index role_org_id_uid" 2024-03-11T21:27:38.688 INFO:journalctl@ceph.grafana.a.smithi097.stdout:Mar 11 21:27:38 smithi097 ceph-00db0016-dfed-11ee-95c8-87774f69a715-grafana-a[65126]: logger=migrator t=2024-03-11T21:27:38.618232359Z level=info msg="Executing migration" id="create team role table" 2024-03-11T21:27:38.688 INFO:journalctl@ceph.grafana.a.smithi097.stdout:Mar 11 21:27:38 smithi097 ceph-00db0016-dfed-11ee-95c8-87774f69a715-grafana-a[65126]: logger=migrator t=2024-03-11T21:27:38.62114761Z level=info msg="Executing migration" id="add index team_role.org_id" 2024-03-11T21:27:38.688 INFO:journalctl@ceph.grafana.a.smithi097.stdout:Mar 11 21:27:38 smithi097 ceph-00db0016-dfed-11ee-95c8-87774f69a715-grafana-a[65126]: logger=migrator t=2024-03-11T21:27:38.623770159Z level=info msg="Executing migration" id="add unique index team_role_org_id_team_id_role_id" 2024-03-11T21:27:38.688 INFO:journalctl@ceph.grafana.a.smithi097.stdout:Mar 11 21:27:38 smithi097 ceph-00db0016-dfed-11ee-95c8-87774f69a715-grafana-a[65126]: logger=migrator t=2024-03-11T21:27:38.626407638Z level=info msg="Executing migration" id="add index team_role.team_id" 2024-03-11T21:27:38.688 INFO:journalctl@ceph.grafana.a.smithi097.stdout:Mar 11 21:27:38 smithi097 ceph-00db0016-dfed-11ee-95c8-87774f69a715-grafana-a[65126]: logger=migrator t=2024-03-11T21:27:38.629040518Z level=info msg="Executing migration" id="create user role table" 2024-03-11T21:27:38.688 INFO:journalctl@ceph.grafana.a.smithi097.stdout:Mar 11 21:27:38 smithi097 ceph-00db0016-dfed-11ee-95c8-87774f69a715-grafana-a[65126]: logger=migrator t=2024-03-11T21:27:38.631939127Z level=info msg="Executing migration" id="add index user_role.org_id" 2024-03-11T21:27:38.688 INFO:journalctl@ceph.grafana.a.smithi097.stdout:Mar 11 21:27:38 smithi097 ceph-00db0016-dfed-11ee-95c8-87774f69a715-grafana-a[65126]: logger=migrator t=2024-03-11T21:27:38.6347455Z level=info msg="Executing migration" id="add unique index user_role_org_id_user_id_role_id" 2024-03-11T21:27:38.688 INFO:journalctl@ceph.grafana.a.smithi097.stdout:Mar 11 21:27:38 smithi097 ceph-00db0016-dfed-11ee-95c8-87774f69a715-grafana-a[65126]: logger=migrator t=2024-03-11T21:27:38.637344376Z level=info msg="Executing migration" id="add index user_role.user_id" 2024-03-11T21:27:38.688 INFO:journalctl@ceph.grafana.a.smithi097.stdout:Mar 11 21:27:38 smithi097 ceph-00db0016-dfed-11ee-95c8-87774f69a715-grafana-a[65126]: logger=migrator t=2024-03-11T21:27:38.6400435Z level=info msg="Executing migration" id="create builtin role table" 2024-03-11T21:27:38.689 INFO:journalctl@ceph.grafana.a.smithi097.stdout:Mar 11 21:27:38 smithi097 ceph-00db0016-dfed-11ee-95c8-87774f69a715-grafana-a[65126]: logger=migrator t=2024-03-11T21:27:38.64277397Z level=info msg="Executing migration" id="add index builtin_role.role_id" 2024-03-11T21:27:38.689 INFO:journalctl@ceph.grafana.a.smithi097.stdout:Mar 11 21:27:38 smithi097 ceph-00db0016-dfed-11ee-95c8-87774f69a715-grafana-a[65126]: logger=migrator t=2024-03-11T21:27:38.645835405Z level=info msg="Executing migration" id="add index builtin_role.name" 2024-03-11T21:27:38.689 INFO:journalctl@ceph.grafana.a.smithi097.stdout:Mar 11 21:27:38 smithi097 ceph-00db0016-dfed-11ee-95c8-87774f69a715-grafana-a[65126]: logger=migrator t=2024-03-11T21:27:38.648497257Z level=info msg="Executing migration" id="Add column org_id to builtin_role table" 2024-03-11T21:27:38.689 INFO:journalctl@ceph.grafana.a.smithi097.stdout:Mar 11 21:27:38 smithi097 ceph-00db0016-dfed-11ee-95c8-87774f69a715-grafana-a[65126]: logger=migrator t=2024-03-11T21:27:38.651378961Z level=info msg="Executing migration" id="add index builtin_role.org_id" 2024-03-11T21:27:38.689 INFO:journalctl@ceph.grafana.a.smithi097.stdout:Mar 11 21:27:38 smithi097 ceph-00db0016-dfed-11ee-95c8-87774f69a715-grafana-a[65126]: logger=migrator t=2024-03-11T21:27:38.654092214Z level=info msg="Executing migration" id="add unique index builtin_role_org_id_role_id_role" 2024-03-11T21:27:38.689 INFO:journalctl@ceph.grafana.a.smithi097.stdout:Mar 11 21:27:38 smithi097 ceph-00db0016-dfed-11ee-95c8-87774f69a715-grafana-a[65126]: logger=migrator t=2024-03-11T21:27:38.656794234Z level=info msg="Executing migration" id="Remove unique index role_org_id_uid" 2024-03-11T21:27:38.689 INFO:journalctl@ceph.grafana.a.smithi097.stdout:Mar 11 21:27:38 smithi097 ceph-00db0016-dfed-11ee-95c8-87774f69a715-grafana-a[65126]: logger=migrator t=2024-03-11T21:27:38.659757809Z level=info msg="Executing migration" id="add unique index role.uid" 2024-03-11T21:27:38.689 INFO:journalctl@ceph.grafana.a.smithi097.stdout:Mar 11 21:27:38 smithi097 ceph-00db0016-dfed-11ee-95c8-87774f69a715-grafana-a[65126]: logger=migrator t=2024-03-11T21:27:38.662287903Z level=info msg="Executing migration" id="create seed assignment table" 2024-03-11T21:27:38.689 INFO:journalctl@ceph.grafana.a.smithi097.stdout:Mar 11 21:27:38 smithi097 ceph-00db0016-dfed-11ee-95c8-87774f69a715-grafana-a[65126]: logger=migrator t=2024-03-11T21:27:38.664928137Z level=info msg="Executing migration" id="add unique index builtin_role_role_name" 2024-03-11T21:27:38.689 INFO:journalctl@ceph.grafana.a.smithi097.stdout:Mar 11 21:27:38 smithi097 ceph-00db0016-dfed-11ee-95c8-87774f69a715-grafana-a[65126]: logger=migrator t=2024-03-11T21:27:38.6676875Z level=info msg="Executing migration" id="add column hidden to role table" 2024-03-11T21:27:38.689 INFO:journalctl@ceph.grafana.a.smithi097.stdout:Mar 11 21:27:38 smithi097 ceph-00db0016-dfed-11ee-95c8-87774f69a715-grafana-a[65126]: logger=migrator t=2024-03-11T21:27:38.670311287Z level=info msg="Executing migration" id="create query_history table v1" 2024-03-11T21:27:38.689 INFO:journalctl@ceph.grafana.a.smithi097.stdout:Mar 11 21:27:38 smithi097 ceph-00db0016-dfed-11ee-95c8-87774f69a715-grafana-a[65126]: logger=migrator t=2024-03-11T21:27:38.672656683Z level=info msg="Executing migration" id="add index query_history.org_id-created_by-datasource_uid" 2024-03-11T21:27:38.689 INFO:journalctl@ceph.grafana.a.smithi097.stdout:Mar 11 21:27:38 smithi097 ceph-00db0016-dfed-11ee-95c8-87774f69a715-grafana-a[65126]: logger=migrator t=2024-03-11T21:27:38.675247967Z level=info msg="Executing migration" id="alter table query_history alter column created_by type to bigint" 2024-03-11T21:27:38.689 INFO:journalctl@ceph.grafana.a.smithi097.stdout:Mar 11 21:27:38 smithi097 ceph-00db0016-dfed-11ee-95c8-87774f69a715-grafana-a[65126]: logger=migrator t=2024-03-11T21:27:38.677899992Z level=info msg="Executing migration" id="teams permissions migration" 2024-03-11T21:27:38.689 INFO:journalctl@ceph.grafana.a.smithi097.stdout:Mar 11 21:27:38 smithi097 ceph-00db0016-dfed-11ee-95c8-87774f69a715-grafana-a[65126]: logger=migrator t=2024-03-11T21:27:38.680518083Z level=info msg="Executing migration" id="dashboard permissions" 2024-03-11T21:27:38.690 INFO:journalctl@ceph.grafana.a.smithi097.stdout:Mar 11 21:27:38 smithi097 ceph-00db0016-dfed-11ee-95c8-87774f69a715-grafana-a[65126]: logger=migrator t=2024-03-11T21:27:38.683353157Z level=info msg="Executing migration" id="dashboard permissions uid scopes" 2024-03-11T21:27:38.690 INFO:journalctl@ceph.grafana.a.smithi097.stdout:Mar 11 21:27:38 smithi097 ceph-00db0016-dfed-11ee-95c8-87774f69a715-grafana-a[65126]: logger=migrator t=2024-03-11T21:27:38.685832594Z level=info msg="Executing migration" id="drop managed folder create actions" 2024-03-11T21:27:38.690 INFO:journalctl@ceph.grafana.a.smithi097.stdout:Mar 11 21:27:38 smithi097 ceph-00db0016-dfed-11ee-95c8-87774f69a715-grafana-a[65126]: logger=migrator t=2024-03-11T21:27:38.688476313Z level=info msg="Executing migration" id="alerting notification permissions" 2024-03-11T21:27:38.690 INFO:journalctl@ceph.grafana.a.smithi097.stdout:Mar 11 21:27:38 smithi097 ceph-00db0016-dfed-11ee-95c8-87774f69a715-grafana-a[65126]: logger=migrator t=2024-03-11T21:27:38.691282073Z level=info msg="Executing migration" id="create query_history_star table v1" 2024-03-11T21:27:38.690 INFO:journalctl@ceph.grafana.a.smithi097.stdout:Mar 11 21:27:38 smithi097 ceph-00db0016-dfed-11ee-95c8-87774f69a715-grafana-a[65126]: logger=migrator t=2024-03-11T21:27:38.694224296Z level=info msg="Executing migration" id="add index query_history.user_id-query_uid" 2024-03-11T21:27:38.690 INFO:journalctl@ceph.grafana.a.smithi097.stdout:Mar 11 21:27:38 smithi097 ceph-00db0016-dfed-11ee-95c8-87774f69a715-grafana-a[65126]: logger=migrator t=2024-03-11T21:27:38.697165772Z level=info msg="Executing migration" id="add column org_id in query_history_star" 2024-03-11T21:27:38.723 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:27:38.746 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:27:38.747 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:27:38.747 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:27:38.848 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:27:38.872 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:27:38.872 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:27:38.872 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:27:38.929 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:27:38 smithi092 ceph-mon[37563]: pgmap v54: 148 pgs: 148 active+clean; 582 KiB data, 226 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T21:27:38.930 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:27:38 smithi092 ceph-mon[37563]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-11T21:27:38.930 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:27:38 smithi092 ceph-mon[37563]: osdmap e101: 8 total, 8 up, 8 in 2024-03-11T21:27:38.930 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:27:38 smithi092 ceph-mon[37563]: from='mgr.14607 ' entity='mgr.y' 2024-03-11T21:27:38.930 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:27:38 smithi092 ceph-mon[37563]: from='mgr.14607 ' entity='mgr.y' 2024-03-11T21:27:38.930 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:27:38 smithi092 ceph-mon[37563]: from='mgr.14607 ' entity='mgr.y' 2024-03-11T21:27:38.930 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:27:38 smithi092 ceph-mon[37563]: from='mgr.14607 ' entity='mgr.y' 2024-03-11T21:27:38.930 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:27:38 smithi092 ceph-mon[29572]: pgmap v54: 148 pgs: 148 active+clean; 582 KiB data, 226 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T21:27:38.930 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:27:38 smithi092 ceph-mon[29572]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-11T21:27:38.930 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:27:38 smithi092 ceph-mon[29572]: osdmap e101: 8 total, 8 up, 8 in 2024-03-11T21:27:38.931 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:27:38 smithi092 ceph-mon[29572]: from='mgr.14607 ' entity='mgr.y' 2024-03-11T21:27:38.931 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:27:38 smithi092 ceph-mon[29572]: from='mgr.14607 ' entity='mgr.y' 2024-03-11T21:27:38.931 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:27:38 smithi092 ceph-mon[29572]: from='mgr.14607 ' entity='mgr.y' 2024-03-11T21:27:38.931 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:27:38 smithi092 ceph-mon[29572]: from='mgr.14607 ' entity='mgr.y' 2024-03-11T21:27:38.973 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:27:38.998 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:27:38.998 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:27:38.999 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:27:39.010 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:27:38 smithi097 ceph-mon[33317]: osdmap e101: 8 total, 8 up, 8 in 2024-03-11T21:27:39.010 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:27:38 smithi097 ceph-mon[33317]: from='mgr.14607 ' entity='mgr.y' 2024-03-11T21:27:39.010 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:27:38 smithi097 ceph-mon[33317]: from='mgr.14607 ' entity='mgr.y' 2024-03-11T21:27:39.010 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:27:38 smithi097 ceph-mon[33317]: from='mgr.14607 ' entity='mgr.y' 2024-03-11T21:27:39.010 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:27:38 smithi097 ceph-mon[33317]: from='mgr.14607 ' entity='mgr.y' 2024-03-11T21:27:39.011 INFO:journalctl@ceph.grafana.a.smithi097.stdout:Mar 11 21:27:38 smithi097 ceph-00db0016-dfed-11ee-95c8-87774f69a715-grafana-a[65126]: logger=migrator t=2024-03-11T21:27:38.698440062Z level=info msg="Executing migration" id="alter table query_history_star_mig column user_id type to bigint" 2024-03-11T21:27:39.011 INFO:journalctl@ceph.grafana.a.smithi097.stdout:Mar 11 21:27:38 smithi097 ceph-00db0016-dfed-11ee-95c8-87774f69a715-grafana-a[65126]: logger=migrator t=2024-03-11T21:27:38.700865829Z level=info msg="Executing migration" id="create correlation table v1" 2024-03-11T21:27:39.011 INFO:journalctl@ceph.grafana.a.smithi097.stdout:Mar 11 21:27:38 smithi097 ceph-00db0016-dfed-11ee-95c8-87774f69a715-grafana-a[65126]: logger=migrator t=2024-03-11T21:27:38.703748923Z level=info msg="Executing migration" id="add index correlations.uid" 2024-03-11T21:27:39.011 INFO:journalctl@ceph.grafana.a.smithi097.stdout:Mar 11 21:27:38 smithi097 ceph-00db0016-dfed-11ee-95c8-87774f69a715-grafana-a[65126]: logger=migrator t=2024-03-11T21:27:38.704208891Z level=info msg="Executing migration" id="add index correlations.source_uid" 2024-03-11T21:27:39.011 INFO:journalctl@ceph.grafana.a.smithi097.stdout:Mar 11 21:27:38 smithi097 ceph-00db0016-dfed-11ee-95c8-87774f69a715-grafana-a[65126]: logger=migrator t=2024-03-11T21:27:38.706834922Z level=info msg="Executing migration" id="add correlation config column" 2024-03-11T21:27:39.011 INFO:journalctl@ceph.grafana.a.smithi097.stdout:Mar 11 21:27:38 smithi097 ceph-00db0016-dfed-11ee-95c8-87774f69a715-grafana-a[65126]: logger=migrator t=2024-03-11T21:27:38.707984363Z level=info msg="Executing migration" id="create entity_events table" 2024-03-11T21:27:39.011 INFO:journalctl@ceph.grafana.a.smithi097.stdout:Mar 11 21:27:38 smithi097 ceph-00db0016-dfed-11ee-95c8-87774f69a715-grafana-a[65126]: logger=migrator t=2024-03-11T21:27:38.708866669Z level=info msg="Executing migration" id="create dashboard public config v1" 2024-03-11T21:27:39.011 INFO:journalctl@ceph.grafana.a.smithi097.stdout:Mar 11 21:27:38 smithi097 ceph-00db0016-dfed-11ee-95c8-87774f69a715-grafana-a[65126]: logger=migrator t=2024-03-11T21:27:38.709324054Z level=info msg="Executing migration" id="drop index UQE_dashboard_public_config_uid - v1" 2024-03-11T21:27:39.011 INFO:journalctl@ceph.grafana.a.smithi097.stdout:Mar 11 21:27:38 smithi097 ceph-00db0016-dfed-11ee-95c8-87774f69a715-grafana-a[65126]: logger=migrator t=2024-03-11T21:27:38.709416854Z level=warn msg="Skipping migration: Already executed, but not recorded in migration log" id="drop index UQE_dashboard_public_config_uid - v1" 2024-03-11T21:27:39.011 INFO:journalctl@ceph.grafana.a.smithi097.stdout:Mar 11 21:27:38 smithi097 ceph-00db0016-dfed-11ee-95c8-87774f69a715-grafana-a[65126]: logger=migrator t=2024-03-11T21:27:38.71030086Z level=info msg="Executing migration" id="drop index IDX_dashboard_public_config_org_id_dashboard_uid - v1" 2024-03-11T21:27:39.011 INFO:journalctl@ceph.grafana.a.smithi097.stdout:Mar 11 21:27:38 smithi097 ceph-00db0016-dfed-11ee-95c8-87774f69a715-grafana-a[65126]: logger=migrator t=2024-03-11T21:27:38.710402Z level=warn msg="Skipping migration: Already executed, but not recorded in migration log" id="drop index IDX_dashboard_public_config_org_id_dashboard_uid - v1" 2024-03-11T21:27:39.012 INFO:journalctl@ceph.grafana.a.smithi097.stdout:Mar 11 21:27:38 smithi097 ceph-00db0016-dfed-11ee-95c8-87774f69a715-grafana-a[65126]: logger=migrator t=2024-03-11T21:27:38.710962295Z level=info msg="Executing migration" id="Drop old dashboard public config table" 2024-03-11T21:27:39.012 INFO:journalctl@ceph.grafana.a.smithi097.stdout:Mar 11 21:27:38 smithi097 ceph-00db0016-dfed-11ee-95c8-87774f69a715-grafana-a[65126]: logger=migrator t=2024-03-11T21:27:38.711818454Z level=info msg="Executing migration" id="recreate dashboard public config v1" 2024-03-11T21:27:39.012 INFO:journalctl@ceph.grafana.a.smithi097.stdout:Mar 11 21:27:38 smithi097 ceph-00db0016-dfed-11ee-95c8-87774f69a715-grafana-a[65126]: logger=migrator t=2024-03-11T21:27:38.71224302Z level=info msg="Executing migration" id="create index UQE_dashboard_public_config_uid - v1" 2024-03-11T21:27:39.012 INFO:journalctl@ceph.grafana.a.smithi097.stdout:Mar 11 21:27:38 smithi097 ceph-00db0016-dfed-11ee-95c8-87774f69a715-grafana-a[65126]: logger=migrator t=2024-03-11T21:27:38.71302698Z level=info msg="Executing migration" id="create index IDX_dashboard_public_config_org_id_dashboard_uid - v1" 2024-03-11T21:27:39.012 INFO:journalctl@ceph.grafana.a.smithi097.stdout:Mar 11 21:27:38 smithi097 ceph-00db0016-dfed-11ee-95c8-87774f69a715-grafana-a[65126]: logger=migrator t=2024-03-11T21:27:38.713982121Z level=info msg="Executing migration" id="drop index UQE_dashboard_public_config_uid - v2" 2024-03-11T21:27:39.012 INFO:journalctl@ceph.grafana.a.smithi097.stdout:Mar 11 21:27:38 smithi097 ceph-00db0016-dfed-11ee-95c8-87774f69a715-grafana-a[65126]: logger=migrator t=2024-03-11T21:27:38.714647839Z level=info msg="Executing migration" id="drop index IDX_dashboard_public_config_org_id_dashboard_uid - v2" 2024-03-11T21:27:39.012 INFO:journalctl@ceph.grafana.a.smithi097.stdout:Mar 11 21:27:38 smithi097 ceph-00db0016-dfed-11ee-95c8-87774f69a715-grafana-a[65126]: logger=migrator t=2024-03-11T21:27:38.715084302Z level=info msg="Executing migration" id="Drop public config table" 2024-03-11T21:27:39.013 INFO:journalctl@ceph.grafana.a.smithi097.stdout:Mar 11 21:27:38 smithi097 ceph-00db0016-dfed-11ee-95c8-87774f69a715-grafana-a[65126]: logger=migrator t=2024-03-11T21:27:38.715498845Z level=info msg="Executing migration" id="Recreate dashboard public config v2" 2024-03-11T21:27:39.013 INFO:journalctl@ceph.grafana.a.smithi097.stdout:Mar 11 21:27:38 smithi097 ceph-00db0016-dfed-11ee-95c8-87774f69a715-grafana-a[65126]: logger=migrator t=2024-03-11T21:27:38.715915942Z level=info msg="Executing migration" id="create index UQE_dashboard_public_config_uid - v2" 2024-03-11T21:27:39.013 INFO:journalctl@ceph.grafana.a.smithi097.stdout:Mar 11 21:27:38 smithi097 ceph-00db0016-dfed-11ee-95c8-87774f69a715-grafana-a[65126]: logger=migrator t=2024-03-11T21:27:38.716393535Z level=info msg="Executing migration" id="create index IDX_dashboard_public_config_org_id_dashboard_uid - v2" 2024-03-11T21:27:39.013 INFO:journalctl@ceph.grafana.a.smithi097.stdout:Mar 11 21:27:38 smithi097 ceph-00db0016-dfed-11ee-95c8-87774f69a715-grafana-a[65126]: logger=migrator t=2024-03-11T21:27:38.716884741Z level=info msg="Executing migration" id="create index UQE_dashboard_public_config_access_token - v2" 2024-03-11T21:27:39.013 INFO:journalctl@ceph.grafana.a.smithi097.stdout:Mar 11 21:27:38 smithi097 ceph-00db0016-dfed-11ee-95c8-87774f69a715-grafana-a[65126]: logger=migrator t=2024-03-11T21:27:38.717767985Z level=info msg="Executing migration" id="Rename table dashboard_public_config to dashboard_public - v2" 2024-03-11T21:27:39.013 INFO:journalctl@ceph.grafana.a.smithi097.stdout:Mar 11 21:27:38 smithi097 ceph-00db0016-dfed-11ee-95c8-87774f69a715-grafana-a[65126]: logger=migrator t=2024-03-11T21:27:38.720791121Z level=info msg="Executing migration" id="add annotations_enabled column" 2024-03-11T21:27:39.013 INFO:journalctl@ceph.grafana.a.smithi097.stdout:Mar 11 21:27:38 smithi097 ceph-00db0016-dfed-11ee-95c8-87774f69a715-grafana-a[65126]: logger=migrator t=2024-03-11T21:27:38.721805896Z level=info msg="Executing migration" id="add time_selection_enabled column" 2024-03-11T21:27:39.013 INFO:journalctl@ceph.grafana.a.smithi097.stdout:Mar 11 21:27:38 smithi097 ceph-00db0016-dfed-11ee-95c8-87774f69a715-grafana-a[65126]: logger=migrator t=2024-03-11T21:27:38.722832998Z level=info msg="Executing migration" id="delete orphaned public dashboards" 2024-03-11T21:27:39.013 INFO:journalctl@ceph.grafana.a.smithi097.stdout:Mar 11 21:27:38 smithi097 ceph-00db0016-dfed-11ee-95c8-87774f69a715-grafana-a[65126]: logger=migrator t=2024-03-11T21:27:38.723702916Z level=info msg="Executing migration" id="add share column" 2024-03-11T21:27:39.013 INFO:journalctl@ceph.grafana.a.smithi097.stdout:Mar 11 21:27:38 smithi097 ceph-00db0016-dfed-11ee-95c8-87774f69a715-grafana-a[65126]: logger=migrator t=2024-03-11T21:27:38.724830288Z level=info msg="Executing migration" id="create default alerting folders" 2024-03-11T21:27:39.013 INFO:journalctl@ceph.grafana.a.smithi097.stdout:Mar 11 21:27:38 smithi097 ceph-00db0016-dfed-11ee-95c8-87774f69a715-grafana-a[65126]: logger=migrator t=2024-03-11T21:27:38.725745122Z level=info msg="Executing migration" id="create file table" 2024-03-11T21:27:39.013 INFO:journalctl@ceph.grafana.a.smithi097.stdout:Mar 11 21:27:38 smithi097 ceph-00db0016-dfed-11ee-95c8-87774f69a715-grafana-a[65126]: logger=migrator t=2024-03-11T21:27:38.726514952Z level=info msg="Executing migration" id="file table idx: path natural pk" 2024-03-11T21:27:39.013 INFO:journalctl@ceph.grafana.a.smithi097.stdout:Mar 11 21:27:38 smithi097 ceph-00db0016-dfed-11ee-95c8-87774f69a715-grafana-a[65126]: logger=migrator t=2024-03-11T21:27:38.727479909Z level=info msg="Executing migration" id="file table idx: parent_folder_path_hash fast folder retrieval" 2024-03-11T21:27:39.013 INFO:journalctl@ceph.grafana.a.smithi097.stdout:Mar 11 21:27:38 smithi097 ceph-00db0016-dfed-11ee-95c8-87774f69a715-grafana-a[65126]: logger=migrator t=2024-03-11T21:27:38.728005253Z level=info msg="Executing migration" id="create file_meta table" 2024-03-11T21:27:39.014 INFO:journalctl@ceph.grafana.a.smithi097.stdout:Mar 11 21:27:38 smithi097 ceph-00db0016-dfed-11ee-95c8-87774f69a715-grafana-a[65126]: logger=migrator t=2024-03-11T21:27:38.728937988Z level=info msg="Executing migration" id="file table idx: path key" 2024-03-11T21:27:39.014 INFO:journalctl@ceph.grafana.a.smithi097.stdout:Mar 11 21:27:38 smithi097 ceph-00db0016-dfed-11ee-95c8-87774f69a715-grafana-a[65126]: logger=migrator t=2024-03-11T21:27:38.729639181Z level=info msg="Executing migration" id="set path collation in file table" 2024-03-11T21:27:39.014 INFO:journalctl@ceph.grafana.a.smithi097.stdout:Mar 11 21:27:38 smithi097 ceph-00db0016-dfed-11ee-95c8-87774f69a715-grafana-a[65126]: logger=migrator t=2024-03-11T21:27:38.730626883Z level=info msg="Executing migration" id="managed permissions migration" 2024-03-11T21:27:39.014 INFO:journalctl@ceph.grafana.a.smithi097.stdout:Mar 11 21:27:38 smithi097 ceph-00db0016-dfed-11ee-95c8-87774f69a715-grafana-a[65126]: logger=migrator t=2024-03-11T21:27:38.731117689Z level=info msg="Executing migration" id="managed folder permissions alert actions migration" 2024-03-11T21:27:39.014 INFO:journalctl@ceph.grafana.a.smithi097.stdout:Mar 11 21:27:38 smithi097 ceph-00db0016-dfed-11ee-95c8-87774f69a715-grafana-a[65126]: logger=migrator t=2024-03-11T21:27:38.731962715Z level=info msg="Executing migration" id="RBAC action name migrator" 2024-03-11T21:27:39.014 INFO:journalctl@ceph.grafana.a.smithi097.stdout:Mar 11 21:27:38 smithi097 ceph-00db0016-dfed-11ee-95c8-87774f69a715-grafana-a[65126]: logger=migrator t=2024-03-11T21:27:38.732795566Z level=info msg="Executing migration" id="Add UID column to playlist" 2024-03-11T21:27:39.014 INFO:journalctl@ceph.grafana.a.smithi097.stdout:Mar 11 21:27:38 smithi097 ceph-00db0016-dfed-11ee-95c8-87774f69a715-grafana-a[65126]: logger=migrator t=2024-03-11T21:27:38.734080094Z level=info msg="Executing migration" id="Update uid column values in playlist" 2024-03-11T21:27:39.014 INFO:journalctl@ceph.grafana.a.smithi097.stdout:Mar 11 21:27:38 smithi097 ceph-00db0016-dfed-11ee-95c8-87774f69a715-grafana-a[65126]: logger=migrator t=2024-03-11T21:27:38.734422993Z level=info msg="Executing migration" id="Add index for uid in playlist" 2024-03-11T21:27:39.014 INFO:journalctl@ceph.grafana.a.smithi097.stdout:Mar 11 21:27:38 smithi097 ceph-00db0016-dfed-11ee-95c8-87774f69a715-grafana-a[65126]: logger=migrator t=2024-03-11T21:27:38.735274775Z level=info msg="Executing migration" id="update group index for alert rules" 2024-03-11T21:27:39.014 INFO:journalctl@ceph.grafana.a.smithi097.stdout:Mar 11 21:27:38 smithi097 ceph-00db0016-dfed-11ee-95c8-87774f69a715-grafana-a[65126]: logger=migrator t=2024-03-11T21:27:38.736081163Z level=info msg="Executing migration" id="managed folder permissions alert actions repeated migration" 2024-03-11T21:27:39.014 INFO:journalctl@ceph.grafana.a.smithi097.stdout:Mar 11 21:27:38 smithi097 ceph-00db0016-dfed-11ee-95c8-87774f69a715-grafana-a[65126]: logger=migrator t=2024-03-11T21:27:38.739085563Z level=info msg="Executing migration" id="admin only folder/dashboard permission" 2024-03-11T21:27:39.014 INFO:journalctl@ceph.grafana.a.smithi097.stdout:Mar 11 21:27:38 smithi097 ceph-00db0016-dfed-11ee-95c8-87774f69a715-grafana-a[65126]: logger=migrator t=2024-03-11T21:27:38.739708802Z level=info msg="Executing migration" id="add action column to seed_assignment" 2024-03-11T21:27:39.014 INFO:journalctl@ceph.grafana.a.smithi097.stdout:Mar 11 21:27:38 smithi097 ceph-00db0016-dfed-11ee-95c8-87774f69a715-grafana-a[65126]: logger=migrator t=2024-03-11T21:27:38.742584221Z level=info msg="Executing migration" id="add scope column to seed_assignment" 2024-03-11T21:27:39.014 INFO:journalctl@ceph.grafana.a.smithi097.stdout:Mar 11 21:27:38 smithi097 ceph-00db0016-dfed-11ee-95c8-87774f69a715-grafana-a[65126]: logger=migrator t=2024-03-11T21:27:38.745023604Z level=info msg="Executing migration" id="remove unique index builtin_role_role_name before nullable update" 2024-03-11T21:27:39.015 INFO:journalctl@ceph.grafana.a.smithi097.stdout:Mar 11 21:27:38 smithi097 ceph-00db0016-dfed-11ee-95c8-87774f69a715-grafana-a[65126]: logger=migrator t=2024-03-11T21:27:38.747623497Z level=info msg="Executing migration" id="update seed_assignment role_name column to nullable" 2024-03-11T21:27:39.015 INFO:journalctl@ceph.grafana.a.smithi097.stdout:Mar 11 21:27:38 smithi097 ceph-00db0016-dfed-11ee-95c8-87774f69a715-grafana-a[65126]: logger=migrator t=2024-03-11T21:27:38.75751768Z level=info msg="Executing migration" id="add unique index builtin_role_name back" 2024-03-11T21:27:39.015 INFO:journalctl@ceph.grafana.a.smithi097.stdout:Mar 11 21:27:38 smithi097 ceph-00db0016-dfed-11ee-95c8-87774f69a715-grafana-a[65126]: logger=migrator t=2024-03-11T21:27:38.760284855Z level=info msg="Executing migration" id="add unique index builtin_role_action_scope" 2024-03-11T21:27:39.015 INFO:journalctl@ceph.grafana.a.smithi097.stdout:Mar 11 21:27:38 smithi097 ceph-00db0016-dfed-11ee-95c8-87774f69a715-grafana-a[65126]: logger=migrator t=2024-03-11T21:27:38.763132057Z level=info msg="Executing migration" id="add primary key to seed_assigment" 2024-03-11T21:27:39.015 INFO:journalctl@ceph.grafana.a.smithi097.stdout:Mar 11 21:27:38 smithi097 ceph-00db0016-dfed-11ee-95c8-87774f69a715-grafana-a[65126]: logger=migrator t=2024-03-11T21:27:38.766681504Z level=info msg="Executing migration" id="managed folder permissions alert actions repeated fixed migration" 2024-03-11T21:27:39.015 INFO:journalctl@ceph.grafana.a.smithi097.stdout:Mar 11 21:27:38 smithi097 ceph-00db0016-dfed-11ee-95c8-87774f69a715-grafana-a[65126]: logger=migrator t=2024-03-11T21:27:38.76701825Z level=info msg="Executing migration" id="migrate external alertmanagers to datsourcse" 2024-03-11T21:27:39.015 INFO:journalctl@ceph.grafana.a.smithi097.stdout:Mar 11 21:27:38 smithi097 ceph-00db0016-dfed-11ee-95c8-87774f69a715-grafana-a[65126]: logger=migrator t=2024-03-11T21:27:38.770122551Z level=info msg="Executing migration" id="create folder table" 2024-03-11T21:27:39.015 INFO:journalctl@ceph.grafana.a.smithi097.stdout:Mar 11 21:27:38 smithi097 ceph-00db0016-dfed-11ee-95c8-87774f69a715-grafana-a[65126]: logger=migrator t=2024-03-11T21:27:38.772941253Z level=info msg="Executing migration" id="Add index for parent_uid" 2024-03-11T21:27:39.015 INFO:journalctl@ceph.grafana.a.smithi097.stdout:Mar 11 21:27:38 smithi097 ceph-00db0016-dfed-11ee-95c8-87774f69a715-grafana-a[65126]: logger=migrator t=2024-03-11T21:27:38.775438851Z level=info msg="Executing migration" id="Add unique index for folder.uid and folder.org_id" 2024-03-11T21:27:39.015 INFO:journalctl@ceph.grafana.a.smithi097.stdout:Mar 11 21:27:38 smithi097 ceph-00db0016-dfed-11ee-95c8-87774f69a715-grafana-a[65126]: logger=migrator t=2024-03-11T21:27:38.775949841Z level=info msg="Executing migration" id="Update folder title length" 2024-03-11T21:27:39.015 INFO:journalctl@ceph.grafana.a.smithi097.stdout:Mar 11 21:27:38 smithi097 ceph-00db0016-dfed-11ee-95c8-87774f69a715-grafana-a[65126]: logger=migrator t=2024-03-11T21:27:38.77876733Z level=info msg="Executing migration" id="Add unique index for folder.title and folder.parent_uid" 2024-03-11T21:27:39.015 INFO:journalctl@ceph.grafana.a.smithi097.stdout:Mar 11 21:27:38 smithi097 ceph-00db0016-dfed-11ee-95c8-87774f69a715-grafana-a[65126]: logger=migrator t=2024-03-11T21:27:38.781666963Z level=info msg="migrations completed" performed=485 skipped=0 duration=683.88291ms 2024-03-11T21:27:39.015 INFO:journalctl@ceph.grafana.a.smithi097.stdout:Mar 11 21:27:38 smithi097 ceph-00db0016-dfed-11ee-95c8-87774f69a715-grafana-a[65126]: logger=sqlstore t=2024-03-11T21:27:38.782057259Z level=info msg="Created default organization" 2024-03-11T21:27:39.015 INFO:journalctl@ceph.grafana.a.smithi097.stdout:Mar 11 21:27:38 smithi097 ceph-00db0016-dfed-11ee-95c8-87774f69a715-grafana-a[65126]: logger=plugin.loader t=2024-03-11T21:27:38.8392046Z level=info msg="Plugin registered" pluginID=input 2024-03-11T21:27:39.015 INFO:journalctl@ceph.grafana.a.smithi097.stdout:Mar 11 21:27:38 smithi097 ceph-00db0016-dfed-11ee-95c8-87774f69a715-grafana-a[65126]: logger=plugin.loader t=2024-03-11T21:27:38.865884604Z level=info msg="Plugin registered" pluginID=grafana-piechart-panel 2024-03-11T21:27:39.016 INFO:journalctl@ceph.grafana.a.smithi097.stdout:Mar 11 21:27:38 smithi097 ceph-00db0016-dfed-11ee-95c8-87774f69a715-grafana-a[65126]: logger=plugin.loader t=2024-03-11T21:27:38.865908026Z level=info msg="Plugin registered" pluginID=vonage-status-panel 2024-03-11T21:27:39.016 INFO:journalctl@ceph.grafana.a.smithi097.stdout:Mar 11 21:27:38 smithi097 ceph-00db0016-dfed-11ee-95c8-87774f69a715-grafana-a[65126]: logger=secrets t=2024-03-11T21:27:38.865950446Z level=info msg="Envelope encryption state" enabled=true currentprovider=secretKey.v1 2024-03-11T21:27:39.016 INFO:journalctl@ceph.grafana.a.smithi097.stdout:Mar 11 21:27:38 smithi097 ceph-00db0016-dfed-11ee-95c8-87774f69a715-grafana-a[65126]: logger=query_data t=2024-03-11T21:27:38.867928465Z level=info msg="Query Service initialization" 2024-03-11T21:27:39.016 INFO:journalctl@ceph.grafana.a.smithi097.stdout:Mar 11 21:27:38 smithi097 ceph-00db0016-dfed-11ee-95c8-87774f69a715-grafana-a[65126]: logger=live.push_http t=2024-03-11T21:27:38.870241721Z level=info msg="Live Push Gateway initialization" 2024-03-11T21:27:39.099 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:27:39.126 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:27:39.126 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:27:39.126 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:27:39.227 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:27:39.253 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:27:39.254 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:27:39.254 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:27:39.355 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:27:39.381 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:27:39.382 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:27:39.382 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:27:39.483 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:27:39.508 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:27:39.508 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:27:39.509 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:27:39.526 INFO:journalctl@ceph.grafana.a.smithi097.stdout:Mar 11 21:27:39 smithi097 ceph-00db0016-dfed-11ee-95c8-87774f69a715-grafana-a[65126]: logger=infra.usagestats.collector t=2024-03-11T21:27:39.499900572Z level=info msg="registering usage stat providers" usageStatsProvidersLen=2 2024-03-11T21:27:39.526 INFO:journalctl@ceph.grafana.a.smithi097.stdout:Mar 11 21:27:39 smithi097 ceph-00db0016-dfed-11ee-95c8-87774f69a715-grafana-a[65126]: logger=provisioning.datasources t=2024-03-11T21:27:39.500868355Z level=info msg="inserting datasource from configuration " name=Dashboard1 uid=P43CA22E17D0F9596 2024-03-11T21:27:39.526 INFO:journalctl@ceph.grafana.a.smithi097.stdout:Mar 11 21:27:39 smithi097 ceph-00db0016-dfed-11ee-95c8-87774f69a715-grafana-a[65126]: logger=provisioning.datasources t=2024-03-11T21:27:39.511865652Z level=info msg="inserting datasource from configuration " name=Loki uid=P8E80F9AEF21F6940 2024-03-11T21:27:39.526 INFO:journalctl@ceph.grafana.a.smithi097.stdout:Mar 11 21:27:39 smithi097 ceph-00db0016-dfed-11ee-95c8-87774f69a715-grafana-a[65126]: logger=provisioning.alerting t=2024-03-11T21:27:39.52284873Z level=info msg="starting to provision alerting" 2024-03-11T21:27:39.526 INFO:journalctl@ceph.grafana.a.smithi097.stdout:Mar 11 21:27:39 smithi097 ceph-00db0016-dfed-11ee-95c8-87774f69a715-grafana-a[65126]: logger=provisioning.alerting t=2024-03-11T21:27:39.522887223Z level=info msg="finished to provision alerting" 2024-03-11T21:27:39.526 INFO:journalctl@ceph.grafana.a.smithi097.stdout:Mar 11 21:27:39 smithi097 ceph-00db0016-dfed-11ee-95c8-87774f69a715-grafana-a[65126]: logger=grafanaStorageLogger t=2024-03-11T21:27:39.522966938Z level=info msg="storage starting" 2024-03-11T21:27:39.527 INFO:journalctl@ceph.grafana.a.smithi097.stdout:Mar 11 21:27:39 smithi097 ceph-00db0016-dfed-11ee-95c8-87774f69a715-grafana-a[65126]: logger=ngalert.state.manager t=2024-03-11T21:27:39.523258555Z level=info msg="Warming state cache for startup" 2024-03-11T21:27:39.527 INFO:journalctl@ceph.grafana.a.smithi097.stdout:Mar 11 21:27:39 smithi097 ceph-00db0016-dfed-11ee-95c8-87774f69a715-grafana-a[65126]: logger=http.server t=2024-03-11T21:27:39.527192879Z level=info msg="HTTP Server Listen" address=[::]:3000 protocol=https subUrl= socket= 2024-03-11T21:27:39.527 INFO:journalctl@ceph.grafana.a.smithi097.stdout:Mar 11 21:27:39 smithi097 ceph-00db0016-dfed-11ee-95c8-87774f69a715-grafana-a[65126]: logger=ngalert.state.manager t=2024-03-11T21:27:39.535708094Z level=info msg="State cache has been initialized" states=0 duration=12.447325ms 2024-03-11T21:27:39.527 INFO:journalctl@ceph.grafana.a.smithi097.stdout:Mar 11 21:27:39 smithi097 ceph-00db0016-dfed-11ee-95c8-87774f69a715-grafana-a[65126]: logger=ticker t=2024-03-11T21:27:39.535768927Z level=info msg=starting first_tick=2024-03-11T21:27:40Z 2024-03-11T21:27:39.527 INFO:journalctl@ceph.grafana.a.smithi097.stdout:Mar 11 21:27:39 smithi097 ceph-00db0016-dfed-11ee-95c8-87774f69a715-grafana-a[65126]: logger=ngalert.multiorg.alertmanager t=2024-03-11T21:27:39.535780497Z level=info msg="starting MultiOrg Alertmanager" 2024-03-11T21:27:39.527 INFO:journalctl@ceph.grafana.a.smithi097.stdout:Mar 11 21:27:39 smithi097 ceph-00db0016-dfed-11ee-95c8-87774f69a715-grafana-a[65126]: logger=sqlstore.transactions t=2024-03-11T21:27:39.543718008Z level=info msg="Database locked, sleeping then retrying" error="database is locked" retry=0 code="database is locked" 2024-03-11T21:27:39.609 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:27:39.633 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:27:39.634 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:27:39.634 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:27:39.719 DEBUG:teuthology.orchestra.run.smithi092:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid 00db0016-dfed-11ee-95c8-87774f69a715 -- ceph osd set nodeep-scrub 2024-03-11T21:27:39.735 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:27:39.758 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:27:39.759 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:27:39.759 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:27:39.826 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:27:39 smithi097 ceph-mon[33317]: Deploying daemon node-exporter.a on smithi092 2024-03-11T21:27:39.827 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:27:39 smithi097 ceph-mon[33317]: purged_snaps scrub starts 2024-03-11T21:27:39.827 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:27:39 smithi097 ceph-mon[33317]: purged_snaps scrub ok 2024-03-11T21:27:39.827 INFO:journalctl@ceph.grafana.a.smithi097.stdout:Mar 11 21:27:39 smithi097 ceph-00db0016-dfed-11ee-95c8-87774f69a715-grafana-a[65126]: logger=sqlstore.transactions t=2024-03-11T21:27:39.546659982Z level=info msg="Database locked, sleeping then retrying" error="database is locked" retry=0 code="database is locked" 2024-03-11T21:27:39.827 INFO:journalctl@ceph.grafana.a.smithi097.stdout:Mar 11 21:27:39 smithi097 ceph-00db0016-dfed-11ee-95c8-87774f69a715-grafana-a[65126]: logger=sqlstore.transactions t=2024-03-11T21:27:39.555251296Z level=info msg="Database locked, sleeping then retrying" error="database is locked" retry=1 code="database is locked" 2024-03-11T21:27:39.860 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:27:39.861 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:27:39 smithi092 ceph-mon[29572]: Deploying daemon node-exporter.a on smithi092 2024-03-11T21:27:39.861 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:27:39 smithi092 ceph-mon[29572]: purged_snaps scrub starts 2024-03-11T21:27:39.861 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:27:39 smithi092 ceph-mon[29572]: purged_snaps scrub ok 2024-03-11T21:27:39.862 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:27:39 smithi092 ceph-mon[37563]: Deploying daemon node-exporter.a on smithi092 2024-03-11T21:27:39.862 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:27:39 smithi092 ceph-mon[37563]: purged_snaps scrub starts 2024-03-11T21:27:39.862 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:27:39 smithi092 ceph-mon[37563]: purged_snaps scrub ok 2024-03-11T21:27:39.884 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:27:39.885 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:27:39.885 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:27:39.986 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:27:39.994 INFO:teuthology.orchestra.run.smithi092.stderr:Inferring config /var/lib/ceph/00db0016-dfed-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T21:27:40.013 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:27:40.013 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:27:40.013 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:27:40.114 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:27:40.138 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:27:40.138 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:27:40.139 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:27:40.240 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:27:40.264 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:27:40.264 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:27:40.265 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:27:40.365 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:27:40.390 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:27:40.390 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:27:40.391 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:27:40.492 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:27:40.515 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:27:40.515 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:27:40.516 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:27:40.617 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:27:40.641 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:27:40.641 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:27:40.642 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:27:40.742 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:27:40.767 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:27:40.767 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:27:40.768 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:27:40.797 DEBUG:teuthology.orchestra.run.smithi092:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid 00db0016-dfed-11ee-95c8-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-03-11T21:27:40.869 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:27:40.894 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:27:40.894 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:27:40.894 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:27:40.951 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:27:40 smithi092 ceph-mon[29572]: pgmap v56: 148 pgs: 148 active+clean; 582 KiB data, 226 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-11T21:27:40.951 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:27:40 smithi092 ceph-mon[29572]: purged_snaps scrub starts 2024-03-11T21:27:40.951 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:27:40 smithi092 ceph-mon[29572]: purged_snaps scrub ok 2024-03-11T21:27:40.951 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:27:40 smithi092 ceph-mon[29572]: osdmap e102: 8 total, 8 up, 8 in 2024-03-11T21:27:40.952 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:27:40 smithi092 ceph-mon[37563]: pgmap v56: 148 pgs: 148 active+clean; 582 KiB data, 226 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-11T21:27:40.952 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:27:40 smithi092 ceph-mon[37563]: purged_snaps scrub starts 2024-03-11T21:27:40.952 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:27:40 smithi092 ceph-mon[37563]: purged_snaps scrub ok 2024-03-11T21:27:40.952 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:27:40 smithi092 ceph-mon[37563]: osdmap e102: 8 total, 8 up, 8 in 2024-03-11T21:27:40.996 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:27:41.022 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:27:41.023 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:27:41.023 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:27:41.076 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:27:40 smithi097 ceph-mon[33317]: pgmap v56: 148 pgs: 148 active+clean; 582 KiB data, 226 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-11T21:27:41.077 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:27:40 smithi097 ceph-mon[33317]: purged_snaps scrub starts 2024-03-11T21:27:41.077 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:27:40 smithi097 ceph-mon[33317]: purged_snaps scrub ok 2024-03-11T21:27:41.077 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:27:40 smithi097 ceph-mon[33317]: osdmap e102: 8 total, 8 up, 8 in 2024-03-11T21:27:41.124 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:27:41.149 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:27:41.150 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:27:41.151 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:27:41.251 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:27:41.276 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:27:41.276 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:27:41.277 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:27:41.378 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:27:41.403 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:27:41.403 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:27:41.403 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:27:41.504 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:27:41.530 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:27:41.531 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:27:41.531 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:27:41.632 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:27:41.642 INFO:teuthology.orchestra.run.smithi092.stdout:ERROR: (22) Invalid argument 2024-03-11T21:27:41.642 INFO:teuthology.orchestra.run.smithi092.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-11T21:27:41.657 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:27:41.658 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:27:41.658 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:27:41.759 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:27:41.783 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:27:41.785 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:27:41.785 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:27:41.876 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:27:41 smithi092 ceph-mon[29572]: purged_snaps scrub starts 2024-03-11T21:27:41.877 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:27:41 smithi092 ceph-mon[29572]: purged_snaps scrub ok 2024-03-11T21:27:41.877 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:27:41 smithi092 ceph-mon[29572]: osdmap e103: 8 total, 8 up, 8 in 2024-03-11T21:27:41.877 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:27:41 smithi092 ceph-mon[29572]: from='mgr.14607 ' entity='mgr.y' 2024-03-11T21:27:41.877 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:27:41 smithi092 ceph-mon[37563]: purged_snaps scrub starts 2024-03-11T21:27:41.877 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:27:41 smithi092 ceph-mon[37563]: purged_snaps scrub ok 2024-03-11T21:27:41.877 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:27:41 smithi092 ceph-mon[37563]: osdmap e103: 8 total, 8 up, 8 in 2024-03-11T21:27:41.877 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:27:41 smithi092 ceph-mon[37563]: from='mgr.14607 ' entity='mgr.y' 2024-03-11T21:27:41.886 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:27:41.917 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:27:41.918 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:27:41.918 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:27:42.010 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:27:41 smithi097 ceph-mon[33317]: purged_snaps scrub starts 2024-03-11T21:27:42.010 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:27:41 smithi097 ceph-mon[33317]: purged_snaps scrub ok 2024-03-11T21:27:42.010 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:27:41 smithi097 ceph-mon[33317]: osdmap e103: 8 total, 8 up, 8 in 2024-03-11T21:27:42.010 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:27:41 smithi097 ceph-mon[33317]: from='mgr.14607 ' entity='mgr.y' 2024-03-11T21:27:42.019 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:27:42.047 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:27:42.048 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:27:42.048 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:27:42.149 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:27:42.172 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:27:42.173 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:27:42.173 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:27:42.274 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:27:42.284 INFO:teuthology.orchestra.run.smithi092.stderr:Inferring config /var/lib/ceph/00db0016-dfed-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T21:27:42.303 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:27:42.303 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:27:42.304 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:27:42.327 INFO:journalctl@ceph.iscsi.iscsi.a.smithi097.stdout:Mar 11 21:27:42 smithi097 ceph-00db0016-dfed-11ee-95c8-87774f69a715-iscsi-iscsi-a[60943]: debug there is no tcmu-runner data available 2024-03-11T21:27:42.404 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:27:42.429 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:27:42.429 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:27:42.430 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:27:42.531 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:27:42.554 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:27:42.555 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:27:42.555 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:27:42.611 INFO:journalctl@ceph.node-exporter.a.smithi092.stdout:Mar 11 21:27:42 smithi092 bash[75806]: Trying to pull quay.io/prometheus/node-exporter:v1.5.0... 2024-03-11T21:27:42.657 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:27:42.681 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:27:42.681 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:27:42.682 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:27:42.701 INFO:teuthology.orchestra.run.smithi092.stderr:enabled application 'rados' on pool 'unique_pool_0' 2024-03-11T21:27:42.783 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:27:42.810 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:27:42.811 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:27:42.812 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:27:42.913 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:27:42.940 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:27:42.940 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:27:42.941 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:27:43.042 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:27:43.065 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:27:42 smithi092 ceph-mon[29572]: pgmap v59: 148 pgs: 148 active+clean; 582 KiB data, 226 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T21:27:43.065 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:27:42 smithi092 ceph-mon[29572]: from='client.? v1:172.21.15.92:0/1335768857' entity='client.admin' cmd=[{"prefix": "osd pool application enable", "pool": "unique_pool_0", "app": "rados", "yes_i_really_mean_it": true}]: dispatch 2024-03-11T21:27:43.065 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:27:42 smithi092 ceph-mon[29572]: from='client.34700 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable", "pool": "unique_pool_0", "app": "rados", "yes_i_really_mean_it": true}]: dispatch 2024-03-11T21:27:43.065 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:27:42 smithi092 ceph-mon[37563]: pgmap v59: 148 pgs: 148 active+clean; 582 KiB data, 226 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T21:27:43.066 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:27:42 smithi092 ceph-mon[37563]: from='client.? v1:172.21.15.92:0/1335768857' entity='client.admin' cmd=[{"prefix": "osd pool application enable", "pool": "unique_pool_0", "app": "rados", "yes_i_really_mean_it": true}]: dispatch 2024-03-11T21:27:43.066 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:27:42 smithi092 ceph-mon[37563]: from='client.34700 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable", "pool": "unique_pool_0", "app": "rados", "yes_i_really_mean_it": true}]: dispatch 2024-03-11T21:27:43.067 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:27:43.068 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:27:43.068 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:27:43.077 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:27:42 smithi097 ceph-mon[33317]: pgmap v59: 148 pgs: 148 active+clean; 582 KiB data, 226 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T21:27:43.077 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:27:42 smithi097 ceph-mon[33317]: from='client.? v1:172.21.15.92:0/1335768857' entity='client.admin' cmd=[{"prefix": "osd pool application enable", "pool": "unique_pool_0", "app": "rados", "yes_i_really_mean_it": true}]: dispatch 2024-03-11T21:27:43.077 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:27:42 smithi097 ceph-mon[33317]: from='client.34700 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable", "pool": "unique_pool_0", "app": "rados", "yes_i_really_mean_it": true}]: dispatch 2024-03-11T21:27:43.169 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:27:43.192 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:27:43.192 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:27:43.193 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:27:43.293 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:27:43.322 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:27:43.322 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:27:43.323 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:27:43.361 INFO:journalctl@ceph.node-exporter.a.smithi092.stdout:Mar 11 21:27:43 smithi092 bash[75806]: Getting image source signatures 2024-03-11T21:27:43.361 INFO:journalctl@ceph.node-exporter.a.smithi092.stdout:Mar 11 21:27:43 smithi092 bash[75806]: Copying blob sha256:c0e87333d3805a7b7e1c25783dd25f27da0fe70a8e262eee7ed190c8dc843fa7 2024-03-11T21:27:43.361 INFO:journalctl@ceph.node-exporter.a.smithi092.stdout:Mar 11 21:27:43 smithi092 bash[75806]: Copying blob sha256:22b70bddd3acadc892fca4c2af4260629bfda5dfd11ebc106a93ce24e752b5ed 2024-03-11T21:27:43.361 INFO:journalctl@ceph.node-exporter.a.smithi092.stdout:Mar 11 21:27:43 smithi092 bash[75806]: Copying blob sha256:5c12815fee558b157dee7f7509dedbaba0a8379098858a65ec869e1f1526ea0c 2024-03-11T21:27:43.423 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:27:43.451 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:27:43.452 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:27:43.452 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:27:43.553 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:27:43.580 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:27:43.581 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:27:43.581 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:27:43.682 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:27:43.709 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:27:43.711 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:27:43.711 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:27:43.812 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:27:43.841 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:27:43.841 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:27:43.841 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:27:43.859 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:27:43 smithi092 ceph-mon[29572]: from='client.24469 v1:172.21.15.97:0/404208310' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:27:43.859 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:27:43 smithi092 ceph-mon[29572]: from='client.34700 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable", "pool": "unique_pool_0", "app": "rados", "yes_i_really_mean_it": true}]': finished 2024-03-11T21:27:43.859 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:27:43 smithi092 ceph-mon[29572]: osdmap e104: 8 total, 8 up, 8 in 2024-03-11T21:27:43.860 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:27:43 smithi092 ceph-mon[37563]: from='client.24469 v1:172.21.15.97:0/404208310' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:27:43.860 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:27:43 smithi092 ceph-mon[37563]: from='client.34700 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable", "pool": "unique_pool_0", "app": "rados", "yes_i_really_mean_it": true}]': finished 2024-03-11T21:27:43.860 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:27:43 smithi092 ceph-mon[37563]: osdmap e104: 8 total, 8 up, 8 in 2024-03-11T21:27:43.942 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:27:43.969 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:27:43.970 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:27:43.970 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:27:44.071 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:27:44.076 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:27:43 smithi097 ceph-mon[33317]: from='client.24469 v1:172.21.15.97:0/404208310' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:27:44.076 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:27:43 smithi097 ceph-mon[33317]: from='client.34700 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable", "pool": "unique_pool_0", "app": "rados", "yes_i_really_mean_it": true}]': finished 2024-03-11T21:27:44.077 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:27:43 smithi097 ceph-mon[33317]: osdmap e104: 8 total, 8 up, 8 in 2024-03-11T21:27:44.099 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:27:44.099 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:27:44.099 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:27:44.200 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:27:44.228 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:27:44.229 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:27:44.229 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:27:44.330 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:27:44.356 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:27:44.356 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:27:44.357 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:27:44.458 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:27:44.482 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:27:44.482 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:27:44.483 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:27:44.507 DEBUG:teuthology.orchestra.run.smithi092:> /bin/sh -c 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados --no-log-to-stderr --name client.0 -b 65536 --object-size 65536 -p unique_pool_0 bench 90 write' 2024-03-11T21:27:44.508 INFO:tasks.thrashosds.thrasher:Shrinking pool .mgr 2024-03-11T21:27:44.508 INFO:tasks.cephadm.ceph_manager.ceph:contract_pool .mgr by 10 min 8 2024-03-11T21:27:44.508 DEBUG:teuthology.orchestra.run.smithi092:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid 00db0016-dfed-11ee-95c8-87774f69a715 -- ceph pg dump --format=json 2024-03-11T21:27:44.525 INFO:tasks.radosbench:joining radosbench (timing out after 3000s) 2024-03-11T21:27:44.525 INFO:teuthology.orchestra.run:waiting for 3000 2024-03-11T21:27:44.584 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:27:44.612 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:27:44.613 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:27:44.613 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:27:44.618 INFO:tasks.radosbench.radosbench.0.smithi092.stdout:hints = 1 2024-03-11T21:27:44.618 INFO:tasks.radosbench.radosbench.0.smithi092.stdout:Maintaining 16 concurrent writes of 65536 bytes to objects of size 65536 for up to 90 seconds or 0 objects 2024-03-11T21:27:44.618 INFO:tasks.radosbench.radosbench.0.smithi092.stdout:Object prefix: benchmark_data_smithi092_76040 2024-03-11T21:27:44.619 INFO:tasks.radosbench.radosbench.0.smithi092.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-03-11T21:27:44.619 INFO:tasks.radosbench.radosbench.0.smithi092.stdout: 0 0 0 0 0 0 - 0 2024-03-11T21:27:44.713 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:27:44.729 INFO:teuthology.orchestra.run.smithi092.stderr:nodeep-scrub is set 2024-03-11T21:27:44.735 DEBUG:teuthology.orchestra.run.smithi092:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid 00db0016-dfed-11ee-95c8-87774f69a715 -- ceph daemon osd.0 dump_historic_ops 2024-03-11T21:27:44.782 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:27:44.783 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:27:44.783 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:27:44.809 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:27:44 smithi092 ceph-mon[29572]: pgmap v61: 148 pgs: 148 active+clean; 582 KiB data, 230 MiB used, 715 GiB / 715 GiB avail; 968 B/s rd, 0 op/s 2024-03-11T21:27:44.809 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:27:44 smithi092 ceph-mon[29572]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-03-11T21:27:44.809 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:27:44 smithi092 ceph-mon[29572]: from='client.? v1:172.21.15.92:0/3772879116' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-11T21:27:44.809 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:27:44 smithi092 ceph-mon[29572]: from='client.34706 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-11T21:27:44.809 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:27:44 smithi092 ceph-mon[37563]: pgmap v61: 148 pgs: 148 active+clean; 582 KiB data, 230 MiB used, 715 GiB / 715 GiB avail; 968 B/s rd, 0 op/s 2024-03-11T21:27:44.809 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:27:44 smithi092 ceph-mon[37563]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-03-11T21:27:44.810 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:27:44 smithi092 ceph-mon[37563]: from='client.? v1:172.21.15.92:0/3772879116' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-11T21:27:44.810 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:27:44 smithi092 ceph-mon[37563]: from='client.34706 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-11T21:27:44.884 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:27:44.914 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:27:44.915 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:27:44.915 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:27:45.016 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:27:45.077 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:27:44 smithi097 ceph-mon[33317]: pgmap v61: 148 pgs: 148 active+clean; 582 KiB data, 230 MiB used, 715 GiB / 715 GiB avail; 968 B/s rd, 0 op/s 2024-03-11T21:27:45.077 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:27:44 smithi097 ceph-mon[33317]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-03-11T21:27:45.077 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:27:44 smithi097 ceph-mon[33317]: from='client.? v1:172.21.15.92:0/3772879116' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-11T21:27:45.077 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:27:44 smithi097 ceph-mon[33317]: from='client.34706 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-11T21:27:45.087 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:27:45.087 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:27:45.088 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:27:45.189 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:27:45.223 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:27:45.224 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:27:45.224 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:27:45.325 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:27:45.355 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:27:45.356 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:27:45.357 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:27:45.457 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:27:45.498 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:27:45.499 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:27:45.499 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:27:45.600 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:27:45.618 INFO:tasks.radosbench.radosbench.0.smithi092.stdout: 1 15 3084 3069 191.802 191.812 0.00315184 0.00519752 2024-03-11T21:27:45.633 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:27:45.634 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:27:45.634 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:27:45.735 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:27:45.782 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:27:45.783 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:27:45.783 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:27:45.861 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:27:45 smithi092 ceph-mon[29572]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-11T21:27:45.862 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:27:45 smithi092 ceph-mon[29572]: from='client.34706 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-03-11T21:27:45.862 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:27:45 smithi092 ceph-mon[29572]: osdmap e105: 8 total, 8 up, 8 in 2024-03-11T21:27:45.862 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:27:45 smithi092 ceph-mon[37563]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-11T21:27:45.862 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:27:45 smithi092 ceph-mon[37563]: from='client.34706 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-03-11T21:27:45.862 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:27:45 smithi092 ceph-mon[37563]: osdmap e105: 8 total, 8 up, 8 in 2024-03-11T21:27:45.884 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:27:45.914 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:27:45.914 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:27:45.915 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:27:46.016 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:27:46.048 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:27:46.049 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:27:46.049 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:27:46.077 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:27:45 smithi097 ceph-mon[33317]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-11T21:27:46.077 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:27:45 smithi097 ceph-mon[33317]: from='client.34706 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-03-11T21:27:46.077 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:27:45 smithi097 ceph-mon[33317]: osdmap e105: 8 total, 8 up, 8 in 2024-03-11T21:27:46.150 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:27:46.190 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:27:46.190 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:27:46.191 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:27:46.291 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:27:46.324 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:27:46.325 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:27:46.326 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:27:46.427 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:27:46.459 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:27:46.459 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:27:46.460 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:27:46.560 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:27:46.593 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:27:46.594 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:27:46.594 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:27:46.618 INFO:tasks.radosbench.radosbench.0.smithi092.stdout: 2 16 6367 6351 198.454 205.125 0.00474879 0.00502714 2024-03-11T21:27:46.695 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:27:46.737 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:27:46.738 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:27:46.738 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:27:46.839 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:27:46.861 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:27:46 smithi092 ceph-mon[29572]: pgmap v63: 148 pgs: 148 active+clean; 582 KiB data, 230 MiB used, 715 GiB / 715 GiB avail; 970 B/s rd, 0 op/s 2024-03-11T21:27:46.861 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:27:46 smithi092 ceph-mon[29572]: from='mgr.14607 ' entity='mgr.y' 2024-03-11T21:27:46.861 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:27:46 smithi092 ceph-mon[37563]: pgmap v63: 148 pgs: 148 active+clean; 582 KiB data, 230 MiB used, 715 GiB / 715 GiB avail; 970 B/s rd, 0 op/s 2024-03-11T21:27:46.862 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:27:46 smithi092 ceph-mon[37563]: from='mgr.14607 ' entity='mgr.y' 2024-03-11T21:27:46.862 INFO:journalctl@ceph.node-exporter.a.smithi092.stdout:Mar 11 21:27:46 smithi092 bash[75806]: Copying config sha256:0da6a335fe1356545476b749c68f022c897de3a2139e8f0054f6937349ee2b83 2024-03-11T21:27:46.878 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:27:46.879 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:27:46.879 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:27:46.980 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:27:47.013 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:27:47.013 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:27:47.014 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:27:47.076 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:27:46 smithi097 ceph-mon[33317]: pgmap v63: 148 pgs: 148 active+clean; 582 KiB data, 230 MiB used, 715 GiB / 715 GiB avail; 970 B/s rd, 0 op/s 2024-03-11T21:27:47.077 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:27:46 smithi097 ceph-mon[33317]: from='mgr.14607 ' entity='mgr.y' 2024-03-11T21:27:47.114 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:27:47.146 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:27:47.147 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:27:47.147 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:27:47.214 INFO:journalctl@ceph.node-exporter.a.smithi092.stdout:Mar 11 21:27:46 smithi092 bash[75806]: Writing manifest to image destination 2024-03-11T21:27:47.248 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:27:47.276 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:27:47.277 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:27:47.277 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:27:47.378 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:27:47.408 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:27:47.409 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:27:47.409 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:27:47.510 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:27:47.548 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:27:47.549 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:27:47.549 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:27:47.611 INFO:journalctl@ceph.node-exporter.a.smithi092.stdout:Mar 11 21:27:47 smithi092 podman[75806]: 2024-03-11 21:27:47.272600241 +0000 UTC m=+5.004282541 image pull 0da6a335fe1356545476b749c68f022c897de3a2139e8f0054f6937349ee2b83 quay.io/prometheus/node-exporter:v1.5.0 2024-03-11T21:27:47.619 INFO:tasks.radosbench.radosbench.0.smithi092.stdout: 3 15 9529 9514 198.193 197.688 0.00429223 0.00503743 2024-03-11T21:27:47.650 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:27:47.691 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:27:47.691 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:27:47.692 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:27:47.779 INFO:teuthology.orchestra.run.smithi092.stderr:Inferring config /var/lib/ceph/00db0016-dfed-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T21:27:47.793 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:27:47.794 INFO:teuthology.orchestra.run.smithi092.stderr:Inferring config /var/lib/ceph/00db0016-dfed-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T21:27:47.821 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:27:47.822 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:27:47.822 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:27:47.923 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:27:47.955 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:27:47.955 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:27:47.956 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:27:48.048 INFO:journalctl@ceph.node-exporter.a.smithi092.stdout:Mar 11 21:27:47 smithi092 podman[75806]: 2024-03-11 21:27:47.941127843 +0000 UTC m=+5.672810189 container create b3a2398e1da535b9ba3b93fe29546cc745fbf1b243495aa59563654260e010ed (image=quay.io/prometheus/node-exporter:v1.5.0, name=ceph-00db0016-dfed-11ee-95c8-87774f69a715-node-exporter-a, maintainer=The Prometheus Authors ) 2024-03-11T21:27:48.048 INFO:journalctl@ceph.mgr.y.smithi092.stdout:Mar 11 21:27:47 smithi092 ceph-00db0016-dfed-11ee-95c8-87774f69a715-mgr-y[29801]: ::ffff:172.21.15.97 - - [11/Mar/2024:21:27:47] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-11T21:27:48.057 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:27:48.085 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:27:48.086 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:27:48.086 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:27:48.187 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:27:48.219 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:27:48.220 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:27:48.220 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:27:48.321 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:27:48.365 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:27:48.365 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:27:48.366 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:27:48.467 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:27:48.499 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:27:48.500 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:27:48.501 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:27:48.601 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:27:48.619 INFO:tasks.radosbench.radosbench.0.smithi092.stdout: 4 16 12240 12224 190.983 169.375 0.00604541 0.00521034 2024-03-11T21:27:48.631 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:27:48.632 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:27:48.632 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:27:48.733 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:27:48.764 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:27:48.765 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:27:48.765 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:27:48.866 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:27:48.907 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:27:48.908 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:27:48.909 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:27:49.003 INFO:teuthology.orchestra.run.smithi092.stdout:osd.0: {} 2024-03-11T21:27:49.004 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:27:48 smithi092 ceph-mon[29572]: pgmap v64: 148 pgs: 148 active+clean; 465 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 1.6 KiB/s rd, 74 MiB/s wr, 1.19k op/s 2024-03-11T21:27:49.004 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:27:48 smithi092 ceph-mon[37563]: pgmap v64: 148 pgs: 148 active+clean; 465 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 1.6 KiB/s rd, 74 MiB/s wr, 1.19k op/s 2024-03-11T21:27:49.009 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:27:49.054 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:27:49.054 INFO:teuthology.orchestra.run.smithi092.stdout:osd.1: {} 2024-03-11T21:27:49.054 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:27:49.055 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:27:49.076 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:27:48 smithi097 ceph-mon[33317]: pgmap v64: 148 pgs: 148 active+clean; 465 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 1.6 KiB/s rd, 74 MiB/s wr, 1.19k op/s 2024-03-11T21:27:49.096 INFO:teuthology.orchestra.run.smithi092.stdout:osd.2: {} 2024-03-11T21:27:49.140 INFO:teuthology.orchestra.run.smithi092.stdout:osd.3: {} 2024-03-11T21:27:49.156 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:27:49.178 INFO:teuthology.orchestra.run.smithi092.stdout:osd.4: {} 2024-03-11T21:27:49.199 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:27:49.200 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:27:49.200 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:27:49.219 INFO:teuthology.orchestra.run.smithi092.stdout:osd.5: {} 2024-03-11T21:27:49.263 DEBUG:teuthology.orchestra.run.smithi092:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid 00db0016-dfed-11ee-95c8-87774f69a715 -- ceph osd unset noscrub 2024-03-11T21:27:49.263 INFO:teuthology.orchestra.run.smithi092.stdout:osd.6: {} 2024-03-11T21:27:49.301 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:27:49.305 INFO:teuthology.orchestra.run.smithi092.stdout:osd.7: {} 2024-03-11T21:27:49.309 INFO:teuthology.orchestra.run.smithi092.stderr:osd.0: osd_enable_op_tracker = '' 2024-03-11T21:27:49.309 INFO:teuthology.orchestra.run.smithi092.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-11T21:27:49.309 INFO:teuthology.orchestra.run.smithi092.stderr:osd.2: osd_enable_op_tracker = '' 2024-03-11T21:27:49.309 INFO:teuthology.orchestra.run.smithi092.stderr:osd.3: osd_enable_op_tracker = '' 2024-03-11T21:27:49.309 INFO:teuthology.orchestra.run.smithi092.stderr:osd.4: osd_enable_op_tracker = '' 2024-03-11T21:27:49.309 INFO:teuthology.orchestra.run.smithi092.stderr:osd.5: osd_enable_op_tracker = '' 2024-03-11T21:27:49.309 INFO:teuthology.orchestra.run.smithi092.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-11T21:27:49.309 INFO:teuthology.orchestra.run.smithi092.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-11T21:27:49.329 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:27:49.329 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:27:49.329 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:27:49.430 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:27:49.477 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:27:49.478 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:27:49.479 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:27:49.579 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:27:49.611 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:27:49.611 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:27:49.612 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:27:49.619 INFO:tasks.radosbench.radosbench.0.smithi092.stdout: 5 16 14736 14720 183.969 156 0.00350637 0.00542801 2024-03-11T21:27:49.713 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:27:49.746 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:27:49.747 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:27:49.747 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:27:49.848 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:27:49.880 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:27:49.880 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:27:49.881 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:27:49.981 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:27:50.020 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:27:50.022 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:27:50.022 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:27:50.042 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:27:49 smithi092 ceph-mon[29572]: from='client.? v1:172.21.15.92:0/3592709778' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-11T21:27:50.043 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:27:49 smithi092 ceph-mon[37563]: from='client.? v1:172.21.15.92:0/3592709778' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-11T21:27:50.076 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:27:49 smithi097 ceph-mon[33317]: from='client.? v1:172.21.15.92:0/3592709778' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-11T21:27:50.123 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:27:50.155 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:27:50.156 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:27:50.156 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:27:50.257 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:27:50.292 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:27:50.293 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:27:50.293 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:27:50.394 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:27:50.434 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:27:50.435 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:27:50.435 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:27:50.536 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:27:50.577 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:27:50.577 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:27:50.578 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:27:50.619 INFO:tasks.radosbench.radosbench.0.smithi092.stdout: 6 16 17099 17083 177.92 147.688 0.0084828 0.0056132 2024-03-11T21:27:50.679 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:27:50.720 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:27:50.720 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:27:50.721 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:27:50.822 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:27:50.861 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:27:50.861 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:27:50.862 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:27:50.962 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:27:50.994 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:27:50.995 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:27:50.995 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:27:51.076 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:27:50 smithi097 ceph-mon[33317]: pgmap v65: 148 pgs: 148 active+clean; 465 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 58 MiB/s wr, 929 op/s 2024-03-11T21:27:51.096 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:27:51.111 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:27:50 smithi092 ceph-mon[37563]: pgmap v65: 148 pgs: 148 active+clean; 465 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 58 MiB/s wr, 929 op/s 2024-03-11T21:27:51.112 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:27:50 smithi092 ceph-mon[29572]: pgmap v65: 148 pgs: 148 active+clean; 465 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 58 MiB/s wr, 929 op/s 2024-03-11T21:27:51.134 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:27:51.135 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:27:51.135 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:27:51.236 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:27:51.273 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:27:51.273 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:27:51.273 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:27:51.374 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:27:51.405 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:27:51.405 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:27:51.406 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:27:51.507 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:27:51.538 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:27:51.538 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:27:51.539 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:27:51.620 INFO:tasks.radosbench.radosbench.0.smithi092.stdout: 7 16 19539 19523 174.285 152.5 0.00706721 0.00571008 2024-03-11T21:27:51.639 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:27:51.683 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:27:51.683 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:27:51.684 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:27:51.785 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:27:51.825 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:27:51.826 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:27:51.826 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:27:51.861 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:27:51 smithi092 ceph-mon[29572]: from='mgr.14607 v1:172.21.15.92:0/1702717597' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T21:27:51.861 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:27:51 smithi092 ceph-mon[37563]: from='mgr.14607 v1:172.21.15.92:0/1702717597' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T21:27:51.927 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:27:51.959 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:27:51.959 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:27:51.959 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:27:52.021 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:27:51 smithi097 ceph-mon[33317]: from='mgr.14607 v1:172.21.15.92:0/1702717597' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T21:27:52.060 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:27:52.101 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:27:52.101 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:27:52.102 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:27:52.203 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:27:52.243 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:27:52.244 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:27:52.244 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:27:52.326 INFO:journalctl@ceph.iscsi.iscsi.a.smithi097.stdout:Mar 11 21:27:52 smithi097 ceph-00db0016-dfed-11ee-95c8-87774f69a715-iscsi-iscsi-a[60943]: debug there is no tcmu-runner data available 2024-03-11T21:27:52.345 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:27:52.376 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:27:52.377 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:27:52.377 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:27:52.478 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:27:52.515 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:27:52.516 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:27:52.517 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:27:52.618 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:27:52.619 INFO:tasks.radosbench.radosbench.0.smithi092.stdout: 8 16 21643 21627 168.935 131.5 0.00198672 0.00587463 2024-03-11T21:27:52.650 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:27:52.651 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:27:52.651 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:27:52.752 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:27:52.791 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:27:52.793 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:27:52.793 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:27:52.894 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:27:52.927 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:27:52.927 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:27:52.928 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:27:53.028 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:27:53.068 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:27:53.068 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:27:53.069 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:27:53.076 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:27:52 smithi097 ceph-mon[33317]: pgmap v66: 148 pgs: 148 active+clean; 465 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 619 B/s rd, 56 MiB/s wr, 899 op/s 2024-03-11T21:27:53.111 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:27:52 smithi092 ceph-mon[29572]: pgmap v66: 148 pgs: 148 active+clean; 465 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 619 B/s rd, 56 MiB/s wr, 899 op/s 2024-03-11T21:27:53.111 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:27:52 smithi092 ceph-mon[37563]: pgmap v66: 148 pgs: 148 active+clean; 465 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 619 B/s rd, 56 MiB/s wr, 899 op/s 2024-03-11T21:27:53.170 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:27:53.210 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:27:53.211 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:27:53.211 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:27:53.312 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:27:53.372 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:27:53.372 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:27:53.373 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:27:53.460 INFO:journalctl@ceph.node-exporter.a.smithi092.stdout:Mar 11 21:27:53 smithi092 podman[75806]: 2024-03-11 21:27:53.222352514 +0000 UTC m=+10.954034812 container init b3a2398e1da535b9ba3b93fe29546cc745fbf1b243495aa59563654260e010ed (image=quay.io/prometheus/node-exporter:v1.5.0, name=ceph-00db0016-dfed-11ee-95c8-87774f69a715-node-exporter-a, maintainer=The Prometheus Authors ) 2024-03-11T21:27:53.460 INFO:journalctl@ceph.node-exporter.a.smithi092.stdout:Mar 11 21:27:53 smithi092 podman[75806]: 2024-03-11 21:27:53.230884173 +0000 UTC m=+10.962566463 container start b3a2398e1da535b9ba3b93fe29546cc745fbf1b243495aa59563654260e010ed (image=quay.io/prometheus/node-exporter:v1.5.0, name=ceph-00db0016-dfed-11ee-95c8-87774f69a715-node-exporter-a, maintainer=The Prometheus Authors ) 2024-03-11T21:27:53.460 INFO:journalctl@ceph.node-exporter.a.smithi092.stdout:Mar 11 21:27:53 smithi092 ceph-00db0016-dfed-11ee-95c8-87774f69a715-node-exporter-a[76431]: ts=2024-03-11T21:27:53.239Z caller=node_exporter.go:180 level=info msg="Starting node_exporter" version="(version=1.5.0, branch=HEAD, revision=1b48970ffcf5630534fb00bb0687d73c66d1c959)" 2024-03-11T21:27:53.460 INFO:journalctl@ceph.node-exporter.a.smithi092.stdout:Mar 11 21:27:53 smithi092 ceph-00db0016-dfed-11ee-95c8-87774f69a715-node-exporter-a[76431]: ts=2024-03-11T21:27:53.239Z caller=node_exporter.go:181 level=info msg="Build context" build_context="(go=go1.19.3, user=root@6e7732a7b81b, date=20221129-18:59:09)" 2024-03-11T21:27:53.461 INFO:journalctl@ceph.node-exporter.a.smithi092.stdout:Mar 11 21:27:53 smithi092 ceph-00db0016-dfed-11ee-95c8-87774f69a715-node-exporter-a[76431]: ts=2024-03-11T21:27:53.240Z caller=diskstats_common.go:111 level=info collector=diskstats msg="Parsed flag --collector.diskstats.device-exclude" flag=^(ram|loop|fd|(h|s|v|xv)d[a-z]|nvme\d+n\d+p)\d+$ 2024-03-11T21:27:53.461 INFO:journalctl@ceph.node-exporter.a.smithi092.stdout:Mar 11 21:27:53 smithi092 ceph-00db0016-dfed-11ee-95c8-87774f69a715-node-exporter-a[76431]: ts=2024-03-11T21:27:53.240Z caller=diskstats_linux.go:264 level=error collector=diskstats msg="Failed to open directory, disabling udev device properties" path=/run/udev/data 2024-03-11T21:27:53.461 INFO:journalctl@ceph.node-exporter.a.smithi092.stdout:Mar 11 21:27:53 smithi092 ceph-00db0016-dfed-11ee-95c8-87774f69a715-node-exporter-a[76431]: ts=2024-03-11T21:27:53.241Z caller=filesystem_common.go:111 level=info collector=filesystem msg="Parsed flag --collector.filesystem.mount-points-exclude" flag=^/(dev|proc|run/credentials/.+|sys|var/lib/docker/.+|var/lib/containers/storage/.+)($|/) 2024-03-11T21:27:53.461 INFO:journalctl@ceph.node-exporter.a.smithi092.stdout:Mar 11 21:27:53 smithi092 ceph-00db0016-dfed-11ee-95c8-87774f69a715-node-exporter-a[76431]: ts=2024-03-11T21:27:53.241Z caller=filesystem_common.go:113 level=info collector=filesystem msg="Parsed flag --collector.filesystem.fs-types-exclude" flag=^(autofs|binfmt_misc|bpf|cgroup2?|configfs|debugfs|devpts|devtmpfs|fusectl|hugetlbfs|iso9660|mqueue|nsfs|overlay|proc|procfs|pstore|rpc_pipefs|securityfs|selinuxfs|squashfs|sysfs|tracefs)$ 2024-03-11T21:27:53.461 INFO:journalctl@ceph.node-exporter.a.smithi092.stdout:Mar 11 21:27:53 smithi092 ceph-00db0016-dfed-11ee-95c8-87774f69a715-node-exporter-a[76431]: ts=2024-03-11T21:27:53.241Z caller=node_exporter.go:110 level=info msg="Enabled collectors" 2024-03-11T21:27:53.461 INFO:journalctl@ceph.node-exporter.a.smithi092.stdout:Mar 11 21:27:53 smithi092 ceph-00db0016-dfed-11ee-95c8-87774f69a715-node-exporter-a[76431]: ts=2024-03-11T21:27:53.241Z caller=node_exporter.go:117 level=info collector=arp 2024-03-11T21:27:53.461 INFO:journalctl@ceph.node-exporter.a.smithi092.stdout:Mar 11 21:27:53 smithi092 ceph-00db0016-dfed-11ee-95c8-87774f69a715-node-exporter-a[76431]: ts=2024-03-11T21:27:53.241Z caller=node_exporter.go:117 level=info collector=bcache 2024-03-11T21:27:53.461 INFO:journalctl@ceph.node-exporter.a.smithi092.stdout:Mar 11 21:27:53 smithi092 ceph-00db0016-dfed-11ee-95c8-87774f69a715-node-exporter-a[76431]: ts=2024-03-11T21:27:53.241Z caller=node_exporter.go:117 level=info collector=bonding 2024-03-11T21:27:53.461 INFO:journalctl@ceph.node-exporter.a.smithi092.stdout:Mar 11 21:27:53 smithi092 ceph-00db0016-dfed-11ee-95c8-87774f69a715-node-exporter-a[76431]: ts=2024-03-11T21:27:53.241Z caller=node_exporter.go:117 level=info collector=btrfs 2024-03-11T21:27:53.461 INFO:journalctl@ceph.node-exporter.a.smithi092.stdout:Mar 11 21:27:53 smithi092 ceph-00db0016-dfed-11ee-95c8-87774f69a715-node-exporter-a[76431]: ts=2024-03-11T21:27:53.241Z caller=node_exporter.go:117 level=info collector=conntrack 2024-03-11T21:27:53.461 INFO:journalctl@ceph.node-exporter.a.smithi092.stdout:Mar 11 21:27:53 smithi092 ceph-00db0016-dfed-11ee-95c8-87774f69a715-node-exporter-a[76431]: ts=2024-03-11T21:27:53.241Z caller=node_exporter.go:117 level=info collector=cpu 2024-03-11T21:27:53.461 INFO:journalctl@ceph.node-exporter.a.smithi092.stdout:Mar 11 21:27:53 smithi092 ceph-00db0016-dfed-11ee-95c8-87774f69a715-node-exporter-a[76431]: ts=2024-03-11T21:27:53.241Z caller=node_exporter.go:117 level=info collector=cpufreq 2024-03-11T21:27:53.462 INFO:journalctl@ceph.node-exporter.a.smithi092.stdout:Mar 11 21:27:53 smithi092 ceph-00db0016-dfed-11ee-95c8-87774f69a715-node-exporter-a[76431]: ts=2024-03-11T21:27:53.241Z caller=node_exporter.go:117 level=info collector=diskstats 2024-03-11T21:27:53.462 INFO:journalctl@ceph.node-exporter.a.smithi092.stdout:Mar 11 21:27:53 smithi092 ceph-00db0016-dfed-11ee-95c8-87774f69a715-node-exporter-a[76431]: ts=2024-03-11T21:27:53.241Z caller=node_exporter.go:117 level=info collector=dmi 2024-03-11T21:27:53.462 INFO:journalctl@ceph.node-exporter.a.smithi092.stdout:Mar 11 21:27:53 smithi092 ceph-00db0016-dfed-11ee-95c8-87774f69a715-node-exporter-a[76431]: ts=2024-03-11T21:27:53.241Z caller=node_exporter.go:117 level=info collector=edac 2024-03-11T21:27:53.462 INFO:journalctl@ceph.node-exporter.a.smithi092.stdout:Mar 11 21:27:53 smithi092 ceph-00db0016-dfed-11ee-95c8-87774f69a715-node-exporter-a[76431]: ts=2024-03-11T21:27:53.241Z caller=node_exporter.go:117 level=info collector=entropy 2024-03-11T21:27:53.462 INFO:journalctl@ceph.node-exporter.a.smithi092.stdout:Mar 11 21:27:53 smithi092 ceph-00db0016-dfed-11ee-95c8-87774f69a715-node-exporter-a[76431]: ts=2024-03-11T21:27:53.241Z caller=node_exporter.go:117 level=info collector=fibrechannel 2024-03-11T21:27:53.462 INFO:journalctl@ceph.node-exporter.a.smithi092.stdout:Mar 11 21:27:53 smithi092 ceph-00db0016-dfed-11ee-95c8-87774f69a715-node-exporter-a[76431]: ts=2024-03-11T21:27:53.241Z caller=node_exporter.go:117 level=info collector=filefd 2024-03-11T21:27:53.462 INFO:journalctl@ceph.node-exporter.a.smithi092.stdout:Mar 11 21:27:53 smithi092 ceph-00db0016-dfed-11ee-95c8-87774f69a715-node-exporter-a[76431]: ts=2024-03-11T21:27:53.241Z caller=node_exporter.go:117 level=info collector=filesystem 2024-03-11T21:27:53.463 INFO:journalctl@ceph.node-exporter.a.smithi092.stdout:Mar 11 21:27:53 smithi092 ceph-00db0016-dfed-11ee-95c8-87774f69a715-node-exporter-a[76431]: ts=2024-03-11T21:27:53.241Z caller=node_exporter.go:117 level=info collector=hwmon 2024-03-11T21:27:53.463 INFO:journalctl@ceph.node-exporter.a.smithi092.stdout:Mar 11 21:27:53 smithi092 ceph-00db0016-dfed-11ee-95c8-87774f69a715-node-exporter-a[76431]: ts=2024-03-11T21:27:53.241Z caller=node_exporter.go:117 level=info collector=infiniband 2024-03-11T21:27:53.463 INFO:journalctl@ceph.node-exporter.a.smithi092.stdout:Mar 11 21:27:53 smithi092 ceph-00db0016-dfed-11ee-95c8-87774f69a715-node-exporter-a[76431]: ts=2024-03-11T21:27:53.241Z caller=node_exporter.go:117 level=info collector=ipvs 2024-03-11T21:27:53.463 INFO:journalctl@ceph.node-exporter.a.smithi092.stdout:Mar 11 21:27:53 smithi092 ceph-00db0016-dfed-11ee-95c8-87774f69a715-node-exporter-a[76431]: ts=2024-03-11T21:27:53.241Z caller=node_exporter.go:117 level=info collector=loadavg 2024-03-11T21:27:53.463 INFO:journalctl@ceph.node-exporter.a.smithi092.stdout:Mar 11 21:27:53 smithi092 ceph-00db0016-dfed-11ee-95c8-87774f69a715-node-exporter-a[76431]: ts=2024-03-11T21:27:53.241Z caller=node_exporter.go:117 level=info collector=mdadm 2024-03-11T21:27:53.463 INFO:journalctl@ceph.node-exporter.a.smithi092.stdout:Mar 11 21:27:53 smithi092 ceph-00db0016-dfed-11ee-95c8-87774f69a715-node-exporter-a[76431]: ts=2024-03-11T21:27:53.241Z caller=node_exporter.go:117 level=info collector=meminfo 2024-03-11T21:27:53.463 INFO:journalctl@ceph.node-exporter.a.smithi092.stdout:Mar 11 21:27:53 smithi092 ceph-00db0016-dfed-11ee-95c8-87774f69a715-node-exporter-a[76431]: ts=2024-03-11T21:27:53.241Z caller=node_exporter.go:117 level=info collector=netclass 2024-03-11T21:27:53.463 INFO:journalctl@ceph.node-exporter.a.smithi092.stdout:Mar 11 21:27:53 smithi092 ceph-00db0016-dfed-11ee-95c8-87774f69a715-node-exporter-a[76431]: ts=2024-03-11T21:27:53.241Z caller=node_exporter.go:117 level=info collector=netdev 2024-03-11T21:27:53.463 INFO:journalctl@ceph.node-exporter.a.smithi092.stdout:Mar 11 21:27:53 smithi092 ceph-00db0016-dfed-11ee-95c8-87774f69a715-node-exporter-a[76431]: ts=2024-03-11T21:27:53.241Z caller=node_exporter.go:117 level=info collector=netstat 2024-03-11T21:27:53.463 INFO:journalctl@ceph.node-exporter.a.smithi092.stdout:Mar 11 21:27:53 smithi092 ceph-00db0016-dfed-11ee-95c8-87774f69a715-node-exporter-a[76431]: ts=2024-03-11T21:27:53.241Z caller=node_exporter.go:117 level=info collector=nfs 2024-03-11T21:27:53.463 INFO:journalctl@ceph.node-exporter.a.smithi092.stdout:Mar 11 21:27:53 smithi092 ceph-00db0016-dfed-11ee-95c8-87774f69a715-node-exporter-a[76431]: ts=2024-03-11T21:27:53.241Z caller=node_exporter.go:117 level=info collector=nfsd 2024-03-11T21:27:53.463 INFO:journalctl@ceph.node-exporter.a.smithi092.stdout:Mar 11 21:27:53 smithi092 ceph-00db0016-dfed-11ee-95c8-87774f69a715-node-exporter-a[76431]: ts=2024-03-11T21:27:53.241Z caller=node_exporter.go:117 level=info collector=nvme 2024-03-11T21:27:53.463 INFO:journalctl@ceph.node-exporter.a.smithi092.stdout:Mar 11 21:27:53 smithi092 ceph-00db0016-dfed-11ee-95c8-87774f69a715-node-exporter-a[76431]: ts=2024-03-11T21:27:53.241Z caller=node_exporter.go:117 level=info collector=os 2024-03-11T21:27:53.464 INFO:journalctl@ceph.node-exporter.a.smithi092.stdout:Mar 11 21:27:53 smithi092 ceph-00db0016-dfed-11ee-95c8-87774f69a715-node-exporter-a[76431]: ts=2024-03-11T21:27:53.241Z caller=node_exporter.go:117 level=info collector=powersupplyclass 2024-03-11T21:27:53.464 INFO:journalctl@ceph.node-exporter.a.smithi092.stdout:Mar 11 21:27:53 smithi092 ceph-00db0016-dfed-11ee-95c8-87774f69a715-node-exporter-a[76431]: ts=2024-03-11T21:27:53.241Z caller=node_exporter.go:117 level=info collector=pressure 2024-03-11T21:27:53.464 INFO:journalctl@ceph.node-exporter.a.smithi092.stdout:Mar 11 21:27:53 smithi092 ceph-00db0016-dfed-11ee-95c8-87774f69a715-node-exporter-a[76431]: ts=2024-03-11T21:27:53.241Z caller=node_exporter.go:117 level=info collector=rapl 2024-03-11T21:27:53.464 INFO:journalctl@ceph.node-exporter.a.smithi092.stdout:Mar 11 21:27:53 smithi092 ceph-00db0016-dfed-11ee-95c8-87774f69a715-node-exporter-a[76431]: ts=2024-03-11T21:27:53.241Z caller=node_exporter.go:117 level=info collector=schedstat 2024-03-11T21:27:53.464 INFO:journalctl@ceph.node-exporter.a.smithi092.stdout:Mar 11 21:27:53 smithi092 ceph-00db0016-dfed-11ee-95c8-87774f69a715-node-exporter-a[76431]: ts=2024-03-11T21:27:53.241Z caller=node_exporter.go:117 level=info collector=selinux 2024-03-11T21:27:53.464 INFO:journalctl@ceph.node-exporter.a.smithi092.stdout:Mar 11 21:27:53 smithi092 ceph-00db0016-dfed-11ee-95c8-87774f69a715-node-exporter-a[76431]: ts=2024-03-11T21:27:53.241Z caller=node_exporter.go:117 level=info collector=sockstat 2024-03-11T21:27:53.464 INFO:journalctl@ceph.node-exporter.a.smithi092.stdout:Mar 11 21:27:53 smithi092 ceph-00db0016-dfed-11ee-95c8-87774f69a715-node-exporter-a[76431]: ts=2024-03-11T21:27:53.241Z caller=node_exporter.go:117 level=info collector=softnet 2024-03-11T21:27:53.464 INFO:journalctl@ceph.node-exporter.a.smithi092.stdout:Mar 11 21:27:53 smithi092 ceph-00db0016-dfed-11ee-95c8-87774f69a715-node-exporter-a[76431]: ts=2024-03-11T21:27:53.241Z caller=node_exporter.go:117 level=info collector=stat 2024-03-11T21:27:53.464 INFO:journalctl@ceph.node-exporter.a.smithi092.stdout:Mar 11 21:27:53 smithi092 ceph-00db0016-dfed-11ee-95c8-87774f69a715-node-exporter-a[76431]: ts=2024-03-11T21:27:53.241Z caller=node_exporter.go:117 level=info collector=tapestats 2024-03-11T21:27:53.464 INFO:journalctl@ceph.node-exporter.a.smithi092.stdout:Mar 11 21:27:53 smithi092 ceph-00db0016-dfed-11ee-95c8-87774f69a715-node-exporter-a[76431]: ts=2024-03-11T21:27:53.241Z caller=node_exporter.go:117 level=info collector=textfile 2024-03-11T21:27:53.464 INFO:journalctl@ceph.node-exporter.a.smithi092.stdout:Mar 11 21:27:53 smithi092 ceph-00db0016-dfed-11ee-95c8-87774f69a715-node-exporter-a[76431]: ts=2024-03-11T21:27:53.241Z caller=node_exporter.go:117 level=info collector=thermal_zone 2024-03-11T21:27:53.464 INFO:journalctl@ceph.node-exporter.a.smithi092.stdout:Mar 11 21:27:53 smithi092 ceph-00db0016-dfed-11ee-95c8-87774f69a715-node-exporter-a[76431]: ts=2024-03-11T21:27:53.241Z caller=node_exporter.go:117 level=info collector=time 2024-03-11T21:27:53.464 INFO:journalctl@ceph.node-exporter.a.smithi092.stdout:Mar 11 21:27:53 smithi092 ceph-00db0016-dfed-11ee-95c8-87774f69a715-node-exporter-a[76431]: ts=2024-03-11T21:27:53.241Z caller=node_exporter.go:117 level=info collector=udp_queues 2024-03-11T21:27:53.464 INFO:journalctl@ceph.node-exporter.a.smithi092.stdout:Mar 11 21:27:53 smithi092 ceph-00db0016-dfed-11ee-95c8-87774f69a715-node-exporter-a[76431]: ts=2024-03-11T21:27:53.241Z caller=node_exporter.go:117 level=info collector=uname 2024-03-11T21:27:53.465 INFO:journalctl@ceph.node-exporter.a.smithi092.stdout:Mar 11 21:27:53 smithi092 ceph-00db0016-dfed-11ee-95c8-87774f69a715-node-exporter-a[76431]: ts=2024-03-11T21:27:53.241Z caller=node_exporter.go:117 level=info collector=vmstat 2024-03-11T21:27:53.465 INFO:journalctl@ceph.node-exporter.a.smithi092.stdout:Mar 11 21:27:53 smithi092 ceph-00db0016-dfed-11ee-95c8-87774f69a715-node-exporter-a[76431]: ts=2024-03-11T21:27:53.241Z caller=node_exporter.go:117 level=info collector=xfs 2024-03-11T21:27:53.465 INFO:journalctl@ceph.node-exporter.a.smithi092.stdout:Mar 11 21:27:53 smithi092 ceph-00db0016-dfed-11ee-95c8-87774f69a715-node-exporter-a[76431]: ts=2024-03-11T21:27:53.241Z caller=node_exporter.go:117 level=info collector=zfs 2024-03-11T21:27:53.465 INFO:journalctl@ceph.node-exporter.a.smithi092.stdout:Mar 11 21:27:53 smithi092 ceph-00db0016-dfed-11ee-95c8-87774f69a715-node-exporter-a[76431]: ts=2024-03-11T21:27:53.242Z caller=tls_config.go:232 level=info msg="Listening on" address=[::]:9100 2024-03-11T21:27:53.465 INFO:journalctl@ceph.node-exporter.a.smithi092.stdout:Mar 11 21:27:53 smithi092 ceph-00db0016-dfed-11ee-95c8-87774f69a715-node-exporter-a[76431]: ts=2024-03-11T21:27:53.242Z caller=tls_config.go:235 level=info msg="TLS is disabled." http2=false address=[::]:9100 2024-03-11T21:27:53.465 INFO:journalctl@ceph.node-exporter.a.smithi092.stdout:Mar 11 21:27:53 smithi092 bash[75806]: b3a2398e1da535b9ba3b93fe29546cc745fbf1b243495aa59563654260e010ed 2024-03-11T21:27:53.465 INFO:journalctl@ceph.node-exporter.a.smithi092.stdout:Mar 11 21:27:53 smithi092 systemd[1]: Started Ceph node-exporter.a for 00db0016-dfed-11ee-95c8-87774f69a715. 2024-03-11T21:27:53.473 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:27:53.512 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:27:53.513 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:27:53.514 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:27:53.614 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:27:53.619 INFO:tasks.radosbench.radosbench.0.smithi092.stdout: 9 16 23953 23937 166.205 144.375 0.00645766 0.00601108 2024-03-11T21:27:53.650 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:27:53.651 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:27:53.651 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:27:53.752 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:27:53.781 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:27:53.782 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:27:53.783 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:27:53.883 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:27:53.915 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:27:53.916 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:27:53.916 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:27:53.965 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:27:53 smithi092 ceph-mon[29572]: from='client.24469 v1:172.21.15.97:0/404208310' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:27:53.966 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:27:53 smithi092 ceph-mon[29572]: from='mgr.14607 ' entity='mgr.y' 2024-03-11T21:27:53.966 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:27:53 smithi092 ceph-mon[29572]: from='mgr.14607 ' entity='mgr.y' 2024-03-11T21:27:53.966 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:27:53 smithi092 ceph-mon[29572]: from='mgr.14607 ' entity='mgr.y' 2024-03-11T21:27:53.966 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:27:53 smithi092 ceph-mon[37563]: from='client.24469 v1:172.21.15.97:0/404208310' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:27:53.966 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:27:53 smithi092 ceph-mon[37563]: from='mgr.14607 ' entity='mgr.y' 2024-03-11T21:27:53.966 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:27:53 smithi092 ceph-mon[37563]: from='mgr.14607 ' entity='mgr.y' 2024-03-11T21:27:53.966 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:27:53 smithi092 ceph-mon[37563]: from='mgr.14607 ' entity='mgr.y' 2024-03-11T21:27:54.017 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:27:54.050 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:27:54.051 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:27:54.051 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:27:54.076 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:27:53 smithi097 ceph-mon[33317]: from='client.24469 v1:172.21.15.97:0/404208310' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:27:54.077 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:27:53 smithi097 ceph-mon[33317]: from='mgr.14607 ' entity='mgr.y' 2024-03-11T21:27:54.077 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:27:53 smithi097 ceph-mon[33317]: from='mgr.14607 ' entity='mgr.y' 2024-03-11T21:27:54.077 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:27:53 smithi097 ceph-mon[33317]: from='mgr.14607 ' entity='mgr.y' 2024-03-11T21:27:54.152 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:27:54.187 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:27:54.187 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:27:54.188 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:27:54.289 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:27:54.321 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:27:54.322 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:27:54.322 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:27:54.423 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:27:54.454 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:27:54.455 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:27:54.456 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:27:54.556 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:27:54.599 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:27:54.600 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:27:54.601 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:27:54.620 INFO:tasks.radosbench.radosbench.0.smithi092.stdout: 10 15 26193 26178 163.59 140.062 0.00466132 0.00610724 2024-03-11T21:27:54.702 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:27:54.743 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:27:54.743 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:27:54.744 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:27:54.845 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:27:54.890 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:27:54.893 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:27:54.894 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:27:54.995 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:27:55.037 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:27:55.037 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:27:55.038 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:27:55.077 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:27:54 smithi097 ceph-mon[33317]: pgmap v67: 148 pgs: 148 active+clean; 1.2 GiB data, 3.8 GiB used, 711 GiB / 715 GiB avail; 1023 B/s rd, 127 MiB/s wr, 2.03k op/s 2024-03-11T21:27:55.077 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:27:54 smithi097 ceph-mon[33317]: Deploying daemon node-exporter.b on smithi097 2024-03-11T21:27:55.111 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:27:54 smithi092 ceph-mon[29572]: pgmap v67: 148 pgs: 148 active+clean; 1.2 GiB data, 3.8 GiB used, 711 GiB / 715 GiB avail; 1023 B/s rd, 127 MiB/s wr, 2.03k op/s 2024-03-11T21:27:55.111 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:27:54 smithi092 ceph-mon[29572]: Deploying daemon node-exporter.b on smithi097 2024-03-11T21:27:55.111 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:27:54 smithi092 ceph-mon[37563]: pgmap v67: 148 pgs: 148 active+clean; 1.2 GiB data, 3.8 GiB used, 711 GiB / 715 GiB avail; 1023 B/s rd, 127 MiB/s wr, 2.03k op/s 2024-03-11T21:27:55.111 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:27:54 smithi092 ceph-mon[37563]: Deploying daemon node-exporter.b on smithi097 2024-03-11T21:27:55.139 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:27:55.178 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:27:55.296 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:27:55.296 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:27:55.397 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:27:55.430 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:27:55.431 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:27:55.431 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:27:55.532 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:27:55.567 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:27:55.568 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:27:55.568 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:27:55.620 INFO:tasks.radosbench.radosbench.0.smithi092.stdout: 11 16 28048 28032 159.251 115.875 0.00549106 0.0062685 2024-03-11T21:27:55.669 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:27:55.705 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:27:55.705 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:27:55.706 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:27:55.737 INFO:journalctl@ceph.node-exporter.b.smithi097.stdout:Mar 11 21:27:55 smithi097 systemd[1]: Starting Ceph node-exporter.b for 00db0016-dfed-11ee-95c8-87774f69a715... 2024-03-11T21:27:55.807 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:27:55.844 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:27:55.845 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:27:55.845 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:27:55.946 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:27:55.977 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:27:55.978 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:27:55.978 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:27:56.077 INFO:journalctl@ceph.node-exporter.b.smithi097.stdout:Mar 11 21:27:55 smithi097 bash[66565]: Trying to pull quay.io/prometheus/node-exporter:v1.5.0... 2024-03-11T21:27:56.079 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:27:56.104 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:27:56.104 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:27:56.105 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:27:56.150 DEBUG:teuthology.orchestra.run.smithi092:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid 00db0016-dfed-11ee-95c8-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-03-11T21:27:56.205 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:27:56.257 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:27:56.258 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:27:56.258 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:27:56.359 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:27:56.382 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:27:56.383 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:27:56.383 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:27:56.484 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:27:56.524 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:27:56.525 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:27:56.526 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:27:56.620 INFO:tasks.radosbench.radosbench.0.smithi092.stdout: 12 16 29728 29712 154.73 105 0.00929253 0.00645704 2024-03-11T21:27:56.626 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:27:56.655 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:27:56.656 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:27:56.656 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:27:56.757 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:27:56.788 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:27:56.789 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:27:56.790 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:27:56.818 INFO:teuthology.orchestra.run.smithi092.stdout: 2024-03-11T21:27:56.824 INFO:teuthology.orchestra.run.smithi092.stderr:dumped all 2024-03-11T21:27:56.890 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:27:56.921 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:27:56.921 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:27:56.922 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:27:56.931 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:27:56 smithi092 ceph-mon[29572]: pgmap v68: 148 pgs: 148 active+clean; 1.2 GiB data, 3.8 GiB used, 711 GiB / 715 GiB avail; 999 B/s rd, 124 MiB/s wr, 1.98k op/s 2024-03-11T21:27:56.931 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:27:56 smithi092 ceph-mon[37563]: pgmap v68: 148 pgs: 148 active+clean; 1.2 GiB data, 3.8 GiB used, 711 GiB / 715 GiB avail; 999 B/s rd, 124 MiB/s wr, 1.98k op/s 2024-03-11T21:27:57.023 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:27:57.054 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:27:57.055 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:27:57.056 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:27:57.077 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:27:56 smithi097 ceph-mon[33317]: pgmap v68: 148 pgs: 148 active+clean; 1.2 GiB data, 3.8 GiB used, 711 GiB / 715 GiB avail; 999 B/s rd, 124 MiB/s wr, 1.98k op/s 2024-03-11T21:27:57.103 INFO:teuthology.orchestra.run.smithi092.stderr:Inferring config /var/lib/ceph/00db0016-dfed-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T21:27:57.157 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:27:57.187 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:27:57.188 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:27:57.188 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:27:57.289 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:27:57.318 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:27:57.319 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:27:57.320 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:27:57.421 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:27:57.452 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:27:57.453 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:27:57.453 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:27:57.554 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:27:57.586 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:27:57.587 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:27:57.587 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:27:57.620 INFO:tasks.radosbench.radosbench.0.smithi092.stdout: 13 16 31736 31720 152.48 125.5 0.00755835 0.00655247 2024-03-11T21:27:57.687 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:27:57.719 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:27:57.720 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:27:57.721 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:27:57.821 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:27:57.845 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:27:57.846 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:27:57.847 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:27:57.911 INFO:teuthology.orchestra.run.smithi092.stdout:{"pg_ready":true,"pg_map":{"version":68,"stamp":"2024-03-11T21:27:54.977915+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":1329928616,"num_objects":20482,"num_object_clones":0,"num_object_copies":61446,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":20482,"num_whiteouts":0,"num_read":1062,"num_read_kb":908,"num_write":20899,"num_write_kb":1300637,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":21,"num_bytes_recovered":1771324,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":20914,"ondisk_log_size":20914,"up":444,"acting":444,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":444,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":749961216,"kb_used":3967132,"kb_used_data":3688604,"kb_used_omap":12,"kb_used_meta":278451,"kb_avail":745994084,"statfs":{"total":767960285184,"available":763897942016,"internally_reserved":0,"allocated":3777130496,"data_stored":3774054176,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12752,"internal_metadata":285134384},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[8,4,3,2],"upper_bound":16},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":1329332224,"num_objects":20284,"num_object_clones":0,"num_object_copies":60852,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":20284,"num_whiteouts":0,"num_read":10,"num_read_kb":10,"num_write":20284,"num_write_kb":1298176,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"10.249491"},"pg_stats":[{"pgid":"6.1b","version":"0'0","reported_seq":82,"reported_epoch":105,"state":"active+clean","last_fresh":"2024-03-11T21:27:44.735428+0000","last_change":"2024-03-11T21:26:53.584399+0000","last_active":"2024-03-11T21:27:44.735428+0000","last_peered":"2024-03-11T21:27:44.735428+0000","last_clean":"2024-03-11T21:27:44.735428+0000","last_became_active":"2024-03-11T21:25:45.302931+0000","last_became_peered":"2024-03-11T21:25:45.302931+0000","last_unstale":"2024-03-11T21:27:44.735428+0000","last_undegraded":"2024-03-11T21:27:44.735428+0000","last_fullsized":"2024-03-11T21:27:44.735428+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:26:53.584307+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:43.559452+0000","last_clean_scrub_stamp":"2024-03-11T21:26:53.584307+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:27:55.466757+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":79,"reported_epoch":105,"state":"active+clean","last_fresh":"2024-03-11T21:27:44.729288+0000","last_change":"2024-03-11T21:25:42.579254+0000","last_active":"2024-03-11T21:27:44.729288+0000","last_peered":"2024-03-11T21:27:44.729288+0000","last_clean":"2024-03-11T21:27:44.729288+0000","last_became_active":"2024-03-11T21:25:42.578893+0000","last_became_peered":"2024-03-11T21:25:42.578893+0000","last_unstale":"2024-03-11T21:27:44.729288+0000","last_undegraded":"2024-03-11T21:27:44.729288+0000","last_fullsized":"2024-03-11T21:27:44.729288+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:25:41.544550+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:41.544550+0000","last_clean_scrub_stamp":"2024-03-11T21:25:41.544550+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:27:47.808430+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1f","version":"0'0","reported_seq":96,"reported_epoch":105,"state":"active+clean","last_fresh":"2024-03-11T21:27:44.736696+0000","last_change":"2024-03-11T21:26:46.588463+0000","last_active":"2024-03-11T21:27:44.736696+0000","last_peered":"2024-03-11T21:27:44.736696+0000","last_clean":"2024-03-11T21:27:44.736696+0000","last_became_active":"2024-03-11T21:25:38.540510+0000","last_became_peered":"2024-03-11T21:25:38.540510+0000","last_unstale":"2024-03-11T21:27:44.736696+0000","last_undegraded":"2024-03-11T21:27:44.736696+0000","last_fullsized":"2024-03-11T21:27:44.736696+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:26:46.588380+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:37.512018+0000","last_clean_scrub_stamp":"2024-03-11T21:26:46.588380+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:28:06.851002+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,4],"acting":[0,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.19","version":"74'15","reported_seq":115,"reported_epoch":105,"state":"active+clean","last_fresh":"2024-03-11T21:27:44.733509+0000","last_change":"2024-03-11T21:27:12.751287+0000","last_active":"2024-03-11T21:27:44.733509+0000","last_peered":"2024-03-11T21:27:44.733509+0000","last_clean":"2024-03-11T21:27:44.733509+0000","last_became_active":"2024-03-11T21:25:40.554207+0000","last_became_peered":"2024-03-11T21:25:40.554207+0000","last_unstale":"2024-03-11T21:27:44.733509+0000","last_undegraded":"2024-03-11T21:27:44.733509+0000","last_fullsized":"2024-03-11T21:27:44.733509+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'15","last_scrub_stamp":"2024-03-11T21:27:12.751184+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:39.525554+0000","last_clean_scrub_stamp":"2024-03-11T21:27:12.751184+0000","objects_scrubbed":7,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:28:24.326457+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.1a","version":"0'0","reported_seq":75,"reported_epoch":105,"state":"active+clean","last_fresh":"2024-03-11T21:27:44.729720+0000","last_change":"2024-03-11T21:25:44.591548+0000","last_active":"2024-03-11T21:27:44.729720+0000","last_peered":"2024-03-11T21:27:44.729720+0000","last_clean":"2024-03-11T21:27:44.729720+0000","last_became_active":"2024-03-11T21:25:44.590923+0000","last_became_peered":"2024-03-11T21:25:44.590923+0000","last_unstale":"2024-03-11T21:27:44.729720+0000","last_undegraded":"2024-03-11T21:27:44.729720+0000","last_fullsized":"2024-03-11T21:27:44.729720+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:25:43.559452+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:43.559452+0000","last_clean_scrub_stamp":"2024-03-11T21:25:43.559452+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:27:47.808377+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,1],"acting":[4,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":87,"reported_epoch":105,"state":"active+clean","last_fresh":"2024-03-11T21:27:44.732975+0000","last_change":"2024-03-11T21:27:17.264407+0000","last_active":"2024-03-11T21:27:44.732975+0000","last_peered":"2024-03-11T21:27:44.732975+0000","last_clean":"2024-03-11T21:27:44.732975+0000","last_became_active":"2024-03-11T21:25:42.578424+0000","last_became_peered":"2024-03-11T21:25:42.578424+0000","last_unstale":"2024-03-11T21:27:44.732975+0000","last_undegraded":"2024-03-11T21:27:44.732975+0000","last_fullsized":"2024-03-11T21:27:44.732975+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:27:17.264317+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:41.544550+0000","last_clean_scrub_stamp":"2024-03-11T21:27:17.264317+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:28:32.538444+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,7],"acting":[1,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1e","version":"0'0","reported_seq":95,"reported_epoch":105,"state":"active+clean","last_fresh":"2024-03-11T21:27:44.736989+0000","last_change":"2024-03-11T21:26:49.595465+0000","last_active":"2024-03-11T21:27:44.736989+0000","last_peered":"2024-03-11T21:27:44.736989+0000","last_clean":"2024-03-11T21:27:44.736989+0000","last_became_active":"2024-03-11T21:25:38.537283+0000","last_became_peered":"2024-03-11T21:25:38.537283+0000","last_unstale":"2024-03-11T21:27:44.736989+0000","last_undegraded":"2024-03-11T21:27:44.736989+0000","last_fullsized":"2024-03-11T21:27:44.736989+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:26:49.595363+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:37.512018+0000","last_clean_scrub_stamp":"2024-03-11T21:26:49.595363+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:27:52.155430+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.18","version":"74'9","reported_seq":107,"reported_epoch":105,"state":"active+clean","last_fresh":"2024-03-11T21:27:44.729567+0000","last_change":"2024-03-11T21:26:45.870099+0000","last_active":"2024-03-11T21:27:44.729567+0000","last_peered":"2024-03-11T21:27:44.729567+0000","last_clean":"2024-03-11T21:27:44.729567+0000","last_became_active":"2024-03-11T21:25:40.552216+0000","last_became_peered":"2024-03-11T21:25:40.552216+0000","last_unstale":"2024-03-11T21:27:44.729567+0000","last_undegraded":"2024-03-11T21:27:44.729567+0000","last_fullsized":"2024-03-11T21:27:44.729567+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'9","last_scrub_stamp":"2024-03-11T21:26:45.869943+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:39.525554+0000","last_clean_scrub_stamp":"2024-03-11T21:26:45.869943+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:27:50.731066+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":82,"reported_epoch":105,"state":"active+clean","last_fresh":"2024-03-11T21:27:44.742913+0000","last_change":"2024-03-11T21:27:29.953758+0000","last_active":"2024-03-11T21:27:44.742913+0000","last_peered":"2024-03-11T21:27:44.742913+0000","last_clean":"2024-03-11T21:27:44.742913+0000","last_became_active":"2024-03-11T21:25:44.590234+0000","last_became_peered":"2024-03-11T21:25:44.590234+0000","last_unstale":"2024-03-11T21:27:44.742913+0000","last_undegraded":"2024-03-11T21:27:44.742913+0000","last_fullsized":"2024-03-11T21:27:44.742913+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:27:29.953654+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:43.559452+0000","last_clean_scrub_stamp":"2024-03-11T21:27:29.953654+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:28:30.681967+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3],"acting":[5,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":86,"reported_epoch":105,"state":"active+clean","last_fresh":"2024-03-11T21:27:44.738627+0000","last_change":"2024-03-11T21:27:18.434331+0000","last_active":"2024-03-11T21:27:44.738627+0000","last_peered":"2024-03-11T21:27:44.738627+0000","last_clean":"2024-03-11T21:27:44.738627+0000","last_became_active":"2024-03-11T21:25:42.580533+0000","last_became_peered":"2024-03-11T21:25:42.580533+0000","last_unstale":"2024-03-11T21:27:44.738627+0000","last_undegraded":"2024-03-11T21:27:44.738627+0000","last_fullsized":"2024-03-11T21:27:44.738627+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:27:18.434247+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:41.544550+0000","last_clean_scrub_stamp":"2024-03-11T21:27:18.434247+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:28:39.839411+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.1d","version":"0'0","reported_seq":94,"reported_epoch":105,"state":"active+clean","last_fresh":"2024-03-11T21:27:44.739168+0000","last_change":"2024-03-11T21:26:56.370911+0000","last_active":"2024-03-11T21:27:44.739168+0000","last_peered":"2024-03-11T21:27:44.739168+0000","last_clean":"2024-03-11T21:27:44.739168+0000","last_became_active":"2024-03-11T21:25:38.542628+0000","last_became_peered":"2024-03-11T21:25:38.542628+0000","last_unstale":"2024-03-11T21:27:44.739168+0000","last_undegraded":"2024-03-11T21:27:44.739168+0000","last_fullsized":"2024-03-11T21:27:44.739168+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:26:56.370811+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:37.512018+0000","last_clean_scrub_stamp":"2024-03-11T21:26:56.370811+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:28:03.069431+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,0],"acting":[7,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.1b","version":"74'5","reported_seq":93,"reported_epoch":105,"state":"active+clean","last_fresh":"2024-03-11T21:27:44.730671+0000","last_change":"2024-03-11T21:25:40.556136+0000","last_active":"2024-03-11T21:27:44.730671+0000","last_peered":"2024-03-11T21:27:44.730671+0000","last_clean":"2024-03-11T21:27:44.730671+0000","last_became_active":"2024-03-11T21:25:40.555826+0000","last_became_peered":"2024-03-11T21:25:40.555826+0000","last_unstale":"2024-03-11T21:27:44.730671+0000","last_undegraded":"2024-03-11T21:27:44.730671+0000","last_fullsized":"2024-03-11T21:27:44.730671+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:25:39.525554+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:39.525554+0000","last_clean_scrub_stamp":"2024-03-11T21:25:39.525554+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:27:47.808413+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":11,"num_read_kb":7,"num_write":6,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,1],"acting":[4,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":83,"reported_epoch":105,"state":"active+clean","last_fresh":"2024-03-11T21:27:44.733121+0000","last_change":"2024-03-11T21:27:29.476225+0000","last_active":"2024-03-11T21:27:44.733121+0000","last_peered":"2024-03-11T21:27:44.733121+0000","last_clean":"2024-03-11T21:27:44.733121+0000","last_became_active":"2024-03-11T21:25:44.589786+0000","last_became_peered":"2024-03-11T21:25:44.589786+0000","last_unstale":"2024-03-11T21:27:44.733121+0000","last_undegraded":"2024-03-11T21:27:44.733121+0000","last_fullsized":"2024-03-11T21:27:44.733121+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:27:29.476144+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:43.559452+0000","last_clean_scrub_stamp":"2024-03-11T21:27:29.476144+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:28:53.520269+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,7],"acting":[0,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":86,"reported_epoch":105,"state":"active+clean","last_fresh":"2024-03-11T21:27:44.743173+0000","last_change":"2024-03-11T21:27:18.957913+0000","last_active":"2024-03-11T21:27:44.743173+0000","last_peered":"2024-03-11T21:27:44.743173+0000","last_clean":"2024-03-11T21:27:44.743173+0000","last_became_active":"2024-03-11T21:25:42.573831+0000","last_became_peered":"2024-03-11T21:25:42.573831+0000","last_unstale":"2024-03-11T21:27:44.743173+0000","last_undegraded":"2024-03-11T21:27:44.743173+0000","last_fullsized":"2024-03-11T21:27:44.743173+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:27:18.957808+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:41.544550+0000","last_clean_scrub_stamp":"2024-03-11T21:27:18.957808+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:28:37.225502+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,7],"acting":[5,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1c","version":"0'0","reported_seq":94,"reported_epoch":105,"state":"active+clean","last_fresh":"2024-03-11T21:27:44.738891+0000","last_change":"2024-03-11T21:27:04.375295+0000","last_active":"2024-03-11T21:27:44.738891+0000","last_peered":"2024-03-11T21:27:44.738891+0000","last_clean":"2024-03-11T21:27:44.738891+0000","last_became_active":"2024-03-11T21:25:38.541364+0000","last_became_peered":"2024-03-11T21:25:38.541364+0000","last_unstale":"2024-03-11T21:27:44.738891+0000","last_undegraded":"2024-03-11T21:27:44.738891+0000","last_fullsized":"2024-03-11T21:27:44.738891+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:27:04.375194+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:37.512018+0000","last_clean_scrub_stamp":"2024-03-11T21:27:04.375194+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:28:17.432840+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.1a","version":"74'9","reported_seq":107,"reported_epoch":105,"state":"active+clean","last_fresh":"2024-03-11T21:27:44.729884+0000","last_change":"2024-03-11T21:26:44.851070+0000","last_active":"2024-03-11T21:27:44.729884+0000","last_peered":"2024-03-11T21:27:44.729884+0000","last_clean":"2024-03-11T21:27:44.729884+0000","last_became_active":"2024-03-11T21:25:40.556629+0000","last_became_peered":"2024-03-11T21:25:40.556629+0000","last_unstale":"2024-03-11T21:27:44.729884+0000","last_undegraded":"2024-03-11T21:27:44.729884+0000","last_fullsized":"2024-03-11T21:27:44.729884+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'9","last_scrub_stamp":"2024-03-11T21:26:44.850930+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:39.525554+0000","last_clean_scrub_stamp":"2024-03-11T21:26:44.850930+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:27:47.017552+0000","scrub_duration":24,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0],"acting":[4,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1f","version":"0'0","reported_seq":83,"reported_epoch":105,"state":"active+clean","last_fresh":"2024-03-11T21:27:44.736668+0000","last_change":"2024-03-11T21:27:04.620314+0000","last_active":"2024-03-11T21:27:44.736668+0000","last_peered":"2024-03-11T21:27:44.736668+0000","last_clean":"2024-03-11T21:27:44.736668+0000","last_became_active":"2024-03-11T21:25:45.302637+0000","last_became_peered":"2024-03-11T21:25:45.302637+0000","last_unstale":"2024-03-11T21:27:44.736668+0000","last_undegraded":"2024-03-11T21:27:44.736668+0000","last_fullsized":"2024-03-11T21:27:44.736668+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:27:04.620217+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:43.559452+0000","last_clean_scrub_stamp":"2024-03-11T21:27:04.620217+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:28:12.483514+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1c","version":"0'0","reported_seq":87,"reported_epoch":105,"state":"active+clean","last_fresh":"2024-03-11T21:27:44.730927+0000","last_change":"2024-03-11T21:27:10.787962+0000","last_active":"2024-03-11T21:27:44.730927+0000","last_peered":"2024-03-11T21:27:44.730927+0000","last_clean":"2024-03-11T21:27:44.730927+0000","last_became_active":"2024-03-11T21:25:42.576064+0000","last_became_peered":"2024-03-11T21:25:42.576064+0000","last_unstale":"2024-03-11T21:27:44.730927+0000","last_undegraded":"2024-03-11T21:27:44.730927+0000","last_fullsized":"2024-03-11T21:27:44.730927+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:27:10.787826+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:41.544550+0000","last_clean_scrub_stamp":"2024-03-11T21:27:10.787826+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:28:32.419330+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1b","version":"0'0","reported_seq":94,"reported_epoch":105,"state":"active+clean","last_fresh":"2024-03-11T21:27:44.735643+0000","last_change":"2024-03-11T21:26:55.579982+0000","last_active":"2024-03-11T21:27:44.735643+0000","last_peered":"2024-03-11T21:27:44.735643+0000","last_clean":"2024-03-11T21:27:44.735643+0000","last_became_active":"2024-03-11T21:25:38.537520+0000","last_became_peered":"2024-03-11T21:25:38.537520+0000","last_unstale":"2024-03-11T21:27:44.735643+0000","last_undegraded":"2024-03-11T21:27:44.735643+0000","last_fullsized":"2024-03-11T21:27:44.735643+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:26:55.579901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:37.512018+0000","last_clean_scrub_stamp":"2024-03-11T21:26:55.579901+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:28:17.884453+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1d","version":"74'12","reported_seq":113,"reported_epoch":105,"state":"active+clean","last_fresh":"2024-03-11T21:27:44.736441+0000","last_change":"2024-03-11T21:27:01.629574+0000","last_active":"2024-03-11T21:27:44.736441+0000","last_peered":"2024-03-11T21:27:44.736441+0000","last_clean":"2024-03-11T21:27:44.736441+0000","last_became_active":"2024-03-11T21:25:40.556330+0000","last_became_peered":"2024-03-11T21:25:40.556330+0000","last_unstale":"2024-03-11T21:27:44.736441+0000","last_undegraded":"2024-03-11T21:27:44.736441+0000","last_fullsized":"2024-03-11T21:27:44.736441+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'12","last_scrub_stamp":"2024-03-11T21:27:01.629471+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:39.525554+0000","last_clean_scrub_stamp":"2024-03-11T21:27:01.629471+0000","objects_scrubbed":5,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:28:26.432819+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":25,"num_read_kb":16,"num_write":14,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.1e","version":"0'0","reported_seq":76,"reported_epoch":105,"state":"active+clean","last_fresh":"2024-03-11T21:27:44.730418+0000","last_change":"2024-03-11T21:25:45.300689+0000","last_active":"2024-03-11T21:27:44.730418+0000","last_peered":"2024-03-11T21:27:44.730418+0000","last_clean":"2024-03-11T21:27:44.730418+0000","last_became_active":"2024-03-11T21:25:45.300355+0000","last_became_peered":"2024-03-11T21:25:45.300355+0000","last_unstale":"2024-03-11T21:27:44.730418+0000","last_undegraded":"2024-03-11T21:27:44.730418+0000","last_fullsized":"2024-03-11T21:27:44.730418+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:25:43.559452+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:43.559452+0000","last_clean_scrub_stamp":"2024-03-11T21:25:43.559452+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:27:47.808394+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5],"acting":[4,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1d","version":"0'0","reported_seq":87,"reported_epoch":105,"state":"active+clean","last_fresh":"2024-03-11T21:27:44.738223+0000","last_change":"2024-03-11T21:27:09.199590+0000","last_active":"2024-03-11T21:27:44.738223+0000","last_peered":"2024-03-11T21:27:44.738223+0000","last_clean":"2024-03-11T21:27:44.738223+0000","last_became_active":"2024-03-11T21:25:42.579407+0000","last_became_peered":"2024-03-11T21:25:42.579407+0000","last_unstale":"2024-03-11T21:27:44.738223+0000","last_undegraded":"2024-03-11T21:27:44.738223+0000","last_fullsized":"2024-03-11T21:27:44.738223+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:27:09.199513+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:27:09.199513+0000","last_clean_scrub_stamp":"2024-03-11T21:27:09.199513+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:28:28.109684+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1a","version":"0'0","reported_seq":95,"reported_epoch":105,"state":"active+clean","last_fresh":"2024-03-11T21:27:44.733085+0000","last_change":"2024-03-11T21:27:04.037030+0000","last_active":"2024-03-11T21:27:44.733085+0000","last_peered":"2024-03-11T21:27:44.733085+0000","last_clean":"2024-03-11T21:27:44.733085+0000","last_became_active":"2024-03-11T21:25:38.539725+0000","last_became_peered":"2024-03-11T21:25:38.539725+0000","last_unstale":"2024-03-11T21:27:44.733085+0000","last_undegraded":"2024-03-11T21:27:44.733085+0000","last_fullsized":"2024-03-11T21:27:44.733085+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:27:04.036903+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:37.512018+0000","last_clean_scrub_stamp":"2024-03-11T21:27:04.036903+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:28:11.356799+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.1c","version":"74'15","reported_seq":115,"reported_epoch":105,"state":"active+clean","last_fresh":"2024-03-11T21:27:44.746646+0000","last_change":"2024-03-11T21:27:04.045043+0000","last_active":"2024-03-11T21:27:44.746646+0000","last_peered":"2024-03-11T21:27:44.746646+0000","last_clean":"2024-03-11T21:27:44.746646+0000","last_became_active":"2024-03-11T21:25:40.551453+0000","last_became_peered":"2024-03-11T21:25:40.551453+0000","last_unstale":"2024-03-11T21:27:44.746646+0000","last_undegraded":"2024-03-11T21:27:44.746646+0000","last_fullsized":"2024-03-11T21:27:44.746646+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'15","last_scrub_stamp":"2024-03-11T21:27:04.044946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:39.525554+0000","last_clean_scrub_stamp":"2024-03-11T21:27:04.044946+0000","objects_scrubbed":7,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:28:23.430891+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.1d","version":"0'0","reported_seq":83,"reported_epoch":105,"state":"active+clean","last_fresh":"2024-03-11T21:27:44.734325+0000","last_change":"2024-03-11T21:27:20.249317+0000","last_active":"2024-03-11T21:27:44.734325+0000","last_peered":"2024-03-11T21:27:44.734325+0000","last_clean":"2024-03-11T21:27:44.734325+0000","last_became_active":"2024-03-11T21:25:44.588845+0000","last_became_peered":"2024-03-11T21:25:44.588845+0000","last_unstale":"2024-03-11T21:27:44.734325+0000","last_undegraded":"2024-03-11T21:27:44.734325+0000","last_fullsized":"2024-03-11T21:27:44.734325+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:27:20.249231+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:43.559452+0000","last_clean_scrub_stamp":"2024-03-11T21:27:20.249231+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:28:48.831204+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.1e","version":"0'0","reported_seq":88,"reported_epoch":105,"state":"active+clean","last_fresh":"2024-03-11T21:27:44.737291+0000","last_change":"2024-03-11T21:26:58.335515+0000","last_active":"2024-03-11T21:27:44.737291+0000","last_peered":"2024-03-11T21:27:44.737291+0000","last_clean":"2024-03-11T21:27:44.737291+0000","last_became_active":"2024-03-11T21:25:42.582959+0000","last_became_peered":"2024-03-11T21:25:42.582959+0000","last_unstale":"2024-03-11T21:27:44.737291+0000","last_undegraded":"2024-03-11T21:27:44.737291+0000","last_fullsized":"2024-03-11T21:27:44.737291+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:26:58.335431+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:26:58.335431+0000","last_clean_scrub_stamp":"2024-03-11T21:26:58.335431+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:28:08.893453+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.19","version":"67'2","reported_seq":101,"reported_epoch":105,"state":"active+clean","last_fresh":"2024-03-11T21:27:44.733716+0000","last_change":"2024-03-11T21:27:27.823258+0000","last_active":"2024-03-11T21:27:44.733716+0000","last_peered":"2024-03-11T21:27:44.733716+0000","last_clean":"2024-03-11T21:27:44.733716+0000","last_became_active":"2024-03-11T21:25:38.539243+0000","last_became_peered":"2024-03-11T21:25:38.539243+0000","last_unstale":"2024-03-11T21:27:44.733716+0000","last_undegraded":"2024-03-11T21:27:44.733716+0000","last_fullsized":"2024-03-11T21:27:44.733716+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'2","last_scrub_stamp":"2024-03-11T21:27:27.823167+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:37.512018+0000","last_clean_scrub_stamp":"2024-03-11T21:27:27.823167+0000","objects_scrubbed":2,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:28:51.277114+0000","scrub_duration":19,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":957,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":7,"num_read_kb":7,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,0],"acting":[3,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1f","version":"74'11","reported_seq":111,"reported_epoch":105,"state":"active+clean","last_fresh":"2024-03-11T21:27:44.733474+0000","last_change":"2024-03-11T21:26:55.050980+0000","last_active":"2024-03-11T21:27:44.733474+0000","last_peered":"2024-03-11T21:27:44.733474+0000","last_clean":"2024-03-11T21:27:44.733474+0000","last_became_active":"2024-03-11T21:25:40.554686+0000","last_became_peered":"2024-03-11T21:25:40.554686+0000","last_unstale":"2024-03-11T21:27:44.733474+0000","last_undegraded":"2024-03-11T21:27:44.733474+0000","last_fullsized":"2024-03-11T21:27:44.733474+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'11","last_scrub_stamp":"2024-03-11T21:26:55.050872+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:39.525554+0000","last_clean_scrub_stamp":"2024-03-11T21:26:55.050872+0000","objects_scrubbed":5,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:28:13.716793+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,1],"acting":[6,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.1c","version":"74'1","reported_seq":83,"reported_epoch":105,"state":"active+clean","last_fresh":"2024-03-11T21:27:44.739147+0000","last_change":"2024-03-11T21:27:22.466670+0000","last_active":"2024-03-11T21:27:44.739147+0000","last_peered":"2024-03-11T21:27:44.739147+0000","last_clean":"2024-03-11T21:27:44.739147+0000","last_became_active":"2024-03-11T21:25:44.593326+0000","last_became_peered":"2024-03-11T21:25:44.593326+0000","last_unstale":"2024-03-11T21:27:44.739147+0000","last_undegraded":"2024-03-11T21:27:44.739147+0000","last_fullsized":"2024-03-11T21:27:44.739147+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'1","last_scrub_stamp":"2024-03-11T21:27:22.466585+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:43.559452+0000","last_clean_scrub_stamp":"2024-03-11T21:27:22.466585+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:28:38.188343+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":370,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.1f","version":"74'8","reported_seq":126,"reported_epoch":105,"state":"active+clean","last_fresh":"2024-03-11T21:27:46.972655+0000","last_change":"2024-03-11T21:26:48.955663+0000","last_active":"2024-03-11T21:27:46.972655+0000","last_peered":"2024-03-11T21:27:46.972655+0000","last_clean":"2024-03-11T21:27:46.972655+0000","last_became_active":"2024-03-11T21:25:42.577401+0000","last_became_peered":"2024-03-11T21:25:42.577401+0000","last_unstale":"2024-03-11T21:27:46.972655+0000","last_undegraded":"2024-03-11T21:27:46.972655+0000","last_fullsized":"2024-03-11T21:27:46.972655+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'8","last_scrub_stamp":"2024-03-11T21:26:48.955484+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:41.544550+0000","last_clean_scrub_stamp":"2024-03-11T21:26:48.955484+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:27:51.651922+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.18","version":"67'1","reported_seq":100,"reported_epoch":105,"state":"active+clean","last_fresh":"2024-03-11T21:27:44.742920+0000","last_change":"2024-03-11T21:26:46.995375+0000","last_active":"2024-03-11T21:27:44.742920+0000","last_peered":"2024-03-11T21:27:44.742920+0000","last_clean":"2024-03-11T21:27:44.742920+0000","last_became_active":"2024-03-11T21:25:38.538321+0000","last_became_peered":"2024-03-11T21:25:38.538321+0000","last_unstale":"2024-03-11T21:27:44.742920+0000","last_undegraded":"2024-03-11T21:27:44.742920+0000","last_fullsized":"2024-03-11T21:27:44.742920+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'1","last_scrub_stamp":"2024-03-11T21:26:46.995228+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:37.512018+0000","last_clean_scrub_stamp":"2024-03-11T21:26:46.995228+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:27:58.922326+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":436,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":7,"num_read_kb":7,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,7],"acting":[5,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1e","version":"74'10","reported_seq":107,"reported_epoch":105,"state":"active+clean","last_fresh":"2024-03-11T21:27:44.737232+0000","last_change":"2024-03-11T21:26:42.521121+0000","last_active":"2024-03-11T21:27:44.737232+0000","last_peered":"2024-03-11T21:27:44.737232+0000","last_clean":"2024-03-11T21:27:44.737232+0000","last_became_active":"2024-03-11T21:25:40.763438+0000","last_became_peered":"2024-03-11T21:25:40.763438+0000","last_unstale":"2024-03-11T21:27:44.737232+0000","last_undegraded":"2024-03-11T21:27:44.737232+0000","last_fullsized":"2024-03-11T21:27:44.737232+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'10","last_scrub_stamp":"2024-03-11T21:26:42.521016+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:39.525554+0000","last_clean_scrub_stamp":"2024-03-11T21:26:42.521016+0000","objects_scrubbed":5,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:28:12.491436+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.13","version":"0'0","reported_seq":82,"reported_epoch":105,"state":"active+clean","last_fresh":"2024-03-11T21:27:44.735990+0000","last_change":"2024-03-11T21:27:03.623032+0000","last_active":"2024-03-11T21:27:44.735990+0000","last_peered":"2024-03-11T21:27:44.735990+0000","last_clean":"2024-03-11T21:27:44.735990+0000","last_became_active":"2024-03-11T21:25:45.302102+0000","last_became_peered":"2024-03-11T21:25:45.302102+0000","last_unstale":"2024-03-11T21:27:44.735990+0000","last_undegraded":"2024-03-11T21:27:44.735990+0000","last_fullsized":"2024-03-11T21:27:44.735990+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:27:03.622909+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:43.559452+0000","last_clean_scrub_stamp":"2024-03-11T21:27:03.622909+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:28:12.458954+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":86,"reported_epoch":105,"state":"active+clean","last_fresh":"2024-03-11T21:27:44.738759+0000","last_change":"2024-03-11T21:27:14.463412+0000","last_active":"2024-03-11T21:27:44.738759+0000","last_peered":"2024-03-11T21:27:44.738759+0000","last_clean":"2024-03-11T21:27:44.738759+0000","last_became_active":"2024-03-11T21:25:42.579781+0000","last_became_peered":"2024-03-11T21:25:42.579781+0000","last_unstale":"2024-03-11T21:27:44.738759+0000","last_undegraded":"2024-03-11T21:27:44.738759+0000","last_fullsized":"2024-03-11T21:27:44.738759+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:27:14.463307+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:41.544550+0000","last_clean_scrub_stamp":"2024-03-11T21:27:14.463307+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:28:15.427900+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.17","version":"0'0","reported_seq":95,"reported_epoch":105,"state":"active+clean","last_fresh":"2024-03-11T21:27:44.733346+0000","last_change":"2024-03-11T21:26:56.020325+0000","last_active":"2024-03-11T21:27:44.733346+0000","last_peered":"2024-03-11T21:27:44.733346+0000","last_clean":"2024-03-11T21:27:44.733346+0000","last_became_active":"2024-03-11T21:25:38.540349+0000","last_became_peered":"2024-03-11T21:25:38.540349+0000","last_unstale":"2024-03-11T21:27:44.733346+0000","last_undegraded":"2024-03-11T21:27:44.733346+0000","last_fullsized":"2024-03-11T21:27:44.733346+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:26:56.020191+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:37.512018+0000","last_clean_scrub_stamp":"2024-03-11T21:26:56.020191+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:28:25.054065+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,2],"acting":[6,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.11","version":"74'11","reported_seq":109,"reported_epoch":105,"state":"active+clean","last_fresh":"2024-03-11T21:27:44.733962+0000","last_change":"2024-03-11T21:27:22.793832+0000","last_active":"2024-03-11T21:27:44.733962+0000","last_peered":"2024-03-11T21:27:44.733962+0000","last_clean":"2024-03-11T21:27:44.733962+0000","last_became_active":"2024-03-11T21:25:40.760664+0000","last_became_peered":"2024-03-11T21:25:40.760664+0000","last_unstale":"2024-03-11T21:27:44.733962+0000","last_undegraded":"2024-03-11T21:27:44.733962+0000","last_fullsized":"2024-03-11T21:27:44.733962+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'11","last_scrub_stamp":"2024-03-11T21:27:22.793725+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:39.525554+0000","last_clean_scrub_stamp":"2024-03-11T21:27:22.793725+0000","objects_scrubbed":5,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:28:28.896623+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":82,"reported_epoch":105,"state":"active+clean","last_fresh":"2024-03-11T21:27:44.738492+0000","last_change":"2024-03-11T21:27:12.403265+0000","last_active":"2024-03-11T21:27:44.738492+0000","last_peered":"2024-03-11T21:27:44.738492+0000","last_clean":"2024-03-11T21:27:44.738492+0000","last_became_active":"2024-03-11T21:25:44.591068+0000","last_became_peered":"2024-03-11T21:25:44.591068+0000","last_unstale":"2024-03-11T21:27:44.738492+0000","last_undegraded":"2024-03-11T21:27:44.738492+0000","last_fullsized":"2024-03-11T21:27:44.738492+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:27:12.403188+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:43.559452+0000","last_clean_scrub_stamp":"2024-03-11T21:27:12.403188+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:28:14.413531+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,4],"acting":[7,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":87,"reported_epoch":105,"state":"active+clean","last_fresh":"2024-03-11T21:27:44.732868+0000","last_change":"2024-03-11T21:27:09.976879+0000","last_active":"2024-03-11T21:27:44.732868+0000","last_peered":"2024-03-11T21:27:44.732868+0000","last_clean":"2024-03-11T21:27:44.732868+0000","last_became_active":"2024-03-11T21:25:42.584148+0000","last_became_peered":"2024-03-11T21:25:42.584148+0000","last_unstale":"2024-03-11T21:27:44.732868+0000","last_undegraded":"2024-03-11T21:27:44.732868+0000","last_fullsized":"2024-03-11T21:27:44.732868+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:27:09.976747+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:27:09.976747+0000","last_clean_scrub_stamp":"2024-03-11T21:27:09.976747+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:28:32.267721+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.16","version":"0'0","reported_seq":94,"reported_epoch":105,"state":"active+clean","last_fresh":"2024-03-11T21:27:44.743461+0000","last_change":"2024-03-11T21:27:11.939012+0000","last_active":"2024-03-11T21:27:44.743461+0000","last_peered":"2024-03-11T21:27:44.743461+0000","last_clean":"2024-03-11T21:27:44.743461+0000","last_became_active":"2024-03-11T21:25:38.539103+0000","last_became_peered":"2024-03-11T21:25:38.539103+0000","last_unstale":"2024-03-11T21:27:44.743461+0000","last_undegraded":"2024-03-11T21:27:44.743461+0000","last_fullsized":"2024-03-11T21:27:44.743461+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:27:11.938909+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:27:11.938909+0000","last_clean_scrub_stamp":"2024-03-11T21:27:11.938909+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:28:32.608220+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,2],"acting":[5,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.10","version":"94'6","reported_seq":98,"reported_epoch":105,"state":"active+clean","last_fresh":"2024-03-11T21:27:44.733869+0000","last_change":"2024-03-11T21:26:58.601920+0000","last_active":"2024-03-11T21:27:44.733869+0000","last_peered":"2024-03-11T21:27:44.733869+0000","last_clean":"2024-03-11T21:27:44.733869+0000","last_became_active":"2024-03-11T21:25:40.552720+0000","last_became_peered":"2024-03-11T21:25:40.552720+0000","last_unstale":"2024-03-11T21:27:44.733869+0000","last_undegraded":"2024-03-11T21:27:44.733869+0000","last_fullsized":"2024-03-11T21:27:44.733869+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"94'6","last_scrub_stamp":"2024-03-11T21:26:58.601831+0000","last_deep_scrub":"94'6","last_deep_scrub_stamp":"2024-03-11T21:26:58.601831+0000","last_clean_scrub_stamp":"2024-03-11T21:26:58.601831+0000","objects_scrubbed":2,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:28:12.313486+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":6,"num_read_kb":4,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6],"acting":[3,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":82,"reported_epoch":105,"state":"active+clean","last_fresh":"2024-03-11T21:27:44.733384+0000","last_change":"2024-03-11T21:27:05.669068+0000","last_active":"2024-03-11T21:27:44.733384+0000","last_peered":"2024-03-11T21:27:44.733384+0000","last_clean":"2024-03-11T21:27:44.733384+0000","last_became_active":"2024-03-11T21:25:44.583181+0000","last_became_peered":"2024-03-11T21:25:44.583181+0000","last_unstale":"2024-03-11T21:27:44.733384+0000","last_undegraded":"2024-03-11T21:27:44.733384+0000","last_fullsized":"2024-03-11T21:27:44.733384+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:27:05.668971+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:43.559452+0000","last_clean_scrub_stamp":"2024-03-11T21:27:05.668971+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:28:34.193069+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":87,"reported_epoch":105,"state":"active+clean","last_fresh":"2024-03-11T21:27:44.733552+0000","last_change":"2024-03-11T21:27:21.207387+0000","last_active":"2024-03-11T21:27:44.733552+0000","last_peered":"2024-03-11T21:27:44.733552+0000","last_clean":"2024-03-11T21:27:44.733552+0000","last_became_active":"2024-03-11T21:25:42.573158+0000","last_became_peered":"2024-03-11T21:25:42.573158+0000","last_unstale":"2024-03-11T21:27:44.733552+0000","last_undegraded":"2024-03-11T21:27:44.733552+0000","last_fullsized":"2024-03-11T21:27:44.733552+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:27:21.207303+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:27:21.207303+0000","last_clean_scrub_stamp":"2024-03-11T21:27:21.207303+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:28:31.619875+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.15","version":"0'0","reported_seq":95,"reported_epoch":105,"state":"active+clean","last_fresh":"2024-03-11T21:27:44.734723+0000","last_change":"2024-03-11T21:26:44.860603+0000","last_active":"2024-03-11T21:27:44.734723+0000","last_peered":"2024-03-11T21:27:44.734723+0000","last_clean":"2024-03-11T21:27:44.734723+0000","last_became_active":"2024-03-11T21:25:38.541705+0000","last_became_peered":"2024-03-11T21:25:38.541705+0000","last_unstale":"2024-03-11T21:27:44.734723+0000","last_undegraded":"2024-03-11T21:27:44.734723+0000","last_fullsized":"2024-03-11T21:27:44.734723+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:26:44.860499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:26:44.860499+0000","last_clean_scrub_stamp":"2024-03-11T21:26:44.860499+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:28:11.867124+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.13","version":"74'11","reported_seq":128,"reported_epoch":105,"state":"active+clean","last_fresh":"2024-03-11T21:27:44.730634+0000","last_change":"2024-03-11T21:27:03.784968+0000","last_active":"2024-03-11T21:27:44.730634+0000","last_peered":"2024-03-11T21:27:44.730634+0000","last_clean":"2024-03-11T21:27:44.730634+0000","last_became_active":"2024-03-11T21:26:11.214879+0000","last_became_peered":"2024-03-11T21:26:11.214879+0000","last_unstale":"2024-03-11T21:27:44.730634+0000","last_undegraded":"2024-03-11T21:27:44.730634+0000","last_fullsized":"2024-03-11T21:27:44.730634+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'11","last_scrub_stamp":"2024-03-11T21:27:03.784851+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:39.525554+0000","last_clean_scrub_stamp":"2024-03-11T21:27:03.784851+0000","objects_scrubbed":5,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:28:14.160633+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,2],"acting":[4,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.10","version":"0'0","reported_seq":83,"reported_epoch":105,"state":"active+clean","last_fresh":"2024-03-11T21:27:44.732602+0000","last_change":"2024-03-11T21:27:25.417528+0000","last_active":"2024-03-11T21:27:44.732602+0000","last_peered":"2024-03-11T21:27:44.732602+0000","last_clean":"2024-03-11T21:27:44.732602+0000","last_became_active":"2024-03-11T21:25:44.591189+0000","last_became_peered":"2024-03-11T21:25:44.591189+0000","last_unstale":"2024-03-11T21:27:44.732602+0000","last_undegraded":"2024-03-11T21:27:44.732602+0000","last_fullsized":"2024-03-11T21:27:44.732602+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:27:25.417450+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:27:25.417450+0000","last_clean_scrub_stamp":"2024-03-11T21:27:25.417450+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:28:52.772668+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,1],"acting":[0,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":86,"reported_epoch":105,"state":"active+clean","last_fresh":"2024-03-11T21:27:44.735723+0000","last_change":"2024-03-11T21:27:06.637829+0000","last_active":"2024-03-11T21:27:44.735723+0000","last_peered":"2024-03-11T21:27:44.735723+0000","last_clean":"2024-03-11T21:27:44.735723+0000","last_became_active":"2024-03-11T21:25:42.576425+0000","last_became_peered":"2024-03-11T21:25:42.576425+0000","last_unstale":"2024-03-11T21:27:44.735723+0000","last_undegraded":"2024-03-11T21:27:44.735723+0000","last_fullsized":"2024-03-11T21:27:44.735723+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:27:06.637752+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:41.544550+0000","last_clean_scrub_stamp":"2024-03-11T21:27:06.637752+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:28:24.953840+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.14","version":"0'0","reported_seq":95,"reported_epoch":105,"state":"active+clean","last_fresh":"2024-03-11T21:27:44.733087+0000","last_change":"2024-03-11T21:26:59.050063+0000","last_active":"2024-03-11T21:27:44.733087+0000","last_peered":"2024-03-11T21:27:44.733087+0000","last_clean":"2024-03-11T21:27:44.733087+0000","last_became_active":"2024-03-11T21:25:38.541852+0000","last_became_peered":"2024-03-11T21:25:38.541852+0000","last_unstale":"2024-03-11T21:27:44.733087+0000","last_undegraded":"2024-03-11T21:27:44.733087+0000","last_fullsized":"2024-03-11T21:27:44.733087+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:26:59.049960+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:37.512018+0000","last_clean_scrub_stamp":"2024-03-11T21:26:59.049960+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:28:08.381102+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,5],"acting":[6,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.12","version":"74'9","reported_seq":107,"reported_epoch":105,"state":"active+clean","last_fresh":"2024-03-11T21:27:44.733360+0000","last_change":"2024-03-11T21:27:15.286166+0000","last_active":"2024-03-11T21:27:44.733360+0000","last_peered":"2024-03-11T21:27:44.733360+0000","last_clean":"2024-03-11T21:27:44.733360+0000","last_became_active":"2024-03-11T21:25:40.547082+0000","last_became_peered":"2024-03-11T21:25:40.547082+0000","last_unstale":"2024-03-11T21:27:44.733360+0000","last_undegraded":"2024-03-11T21:27:44.733360+0000","last_fullsized":"2024-03-11T21:27:44.733360+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'9","last_scrub_stamp":"2024-03-11T21:27:15.286044+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:39.525554+0000","last_clean_scrub_stamp":"2024-03-11T21:27:15.286044+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:28:29.151259+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":83,"reported_epoch":105,"state":"active+clean","last_fresh":"2024-03-11T21:27:44.731174+0000","last_change":"2024-03-11T21:27:09.801239+0000","last_active":"2024-03-11T21:27:44.731174+0000","last_peered":"2024-03-11T21:27:44.731174+0000","last_clean":"2024-03-11T21:27:44.731174+0000","last_became_active":"2024-03-11T21:25:44.589586+0000","last_became_peered":"2024-03-11T21:25:44.589586+0000","last_unstale":"2024-03-11T21:27:44.731174+0000","last_undegraded":"2024-03-11T21:27:44.731174+0000","last_fullsized":"2024-03-11T21:27:44.731174+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:27:09.801138+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:27:09.801138+0000","last_clean_scrub_stamp":"2024-03-11T21:27:09.801138+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:28:37.698118+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,5],"acting":[4,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.14","version":"74'8","reported_seq":121,"reported_epoch":105,"state":"active+clean","last_fresh":"2024-03-11T21:27:44.738132+0000","last_change":"2024-03-11T21:27:24.803037+0000","last_active":"2024-03-11T21:27:44.738132+0000","last_peered":"2024-03-11T21:27:44.738132+0000","last_clean":"2024-03-11T21:27:44.738132+0000","last_became_active":"2024-03-11T21:25:42.583266+0000","last_became_peered":"2024-03-11T21:25:42.583266+0000","last_unstale":"2024-03-11T21:27:44.738132+0000","last_undegraded":"2024-03-11T21:27:44.738132+0000","last_fullsized":"2024-03-11T21:27:44.738132+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'8","last_scrub_stamp":"2024-03-11T21:27:24.802932+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:41.544550+0000","last_clean_scrub_stamp":"2024-03-11T21:27:24.802932+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:28:44.617485+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,2],"acting":[3,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.13","version":"0'0","reported_seq":95,"reported_epoch":105,"state":"active+clean","last_fresh":"2024-03-11T21:27:44.733736+0000","last_change":"2024-03-11T21:27:12.315278+0000","last_active":"2024-03-11T21:27:44.733736+0000","last_peered":"2024-03-11T21:27:44.733736+0000","last_clean":"2024-03-11T21:27:44.733736+0000","last_became_active":"2024-03-11T21:25:38.539467+0000","last_became_peered":"2024-03-11T21:25:38.539467+0000","last_unstale":"2024-03-11T21:27:44.733736+0000","last_undegraded":"2024-03-11T21:27:44.733736+0000","last_fullsized":"2024-03-11T21:27:44.733736+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:27:12.315200+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:37.512018+0000","last_clean_scrub_stamp":"2024-03-11T21:27:12.315200+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:28:29.228183+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.15","version":"94'13","reported_seq":110,"reported_epoch":105,"state":"active+clean","last_fresh":"2024-03-11T21:27:44.743607+0000","last_change":"2024-03-11T21:26:59.013939+0000","last_active":"2024-03-11T21:27:44.743607+0000","last_peered":"2024-03-11T21:27:44.743607+0000","last_clean":"2024-03-11T21:27:44.743607+0000","last_became_active":"2024-03-11T21:25:40.760907+0000","last_became_peered":"2024-03-11T21:25:40.760907+0000","last_unstale":"2024-03-11T21:27:44.743607+0000","last_undegraded":"2024-03-11T21:27:44.743607+0000","last_fullsized":"2024-03-11T21:27:44.743607+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"94'13","last_scrub_stamp":"2024-03-11T21:26:59.013823+0000","last_deep_scrub":"94'13","last_deep_scrub_stamp":"2024-03-11T21:26:59.013823+0000","last_clean_scrub_stamp":"2024-03-11T21:26:59.013823+0000","objects_scrubbed":4,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:28:25.819110+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.16","version":"0'0","reported_seq":83,"reported_epoch":105,"state":"active+clean","last_fresh":"2024-03-11T21:27:44.735820+0000","last_change":"2024-03-11T21:26:57.367233+0000","last_active":"2024-03-11T21:27:44.735820+0000","last_peered":"2024-03-11T21:27:44.735820+0000","last_clean":"2024-03-11T21:27:44.735820+0000","last_became_active":"2024-03-11T21:25:44.583393+0000","last_became_peered":"2024-03-11T21:25:44.583393+0000","last_unstale":"2024-03-11T21:27:44.735820+0000","last_undegraded":"2024-03-11T21:27:44.735820+0000","last_fullsized":"2024-03-11T21:27:44.735820+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:26:57.367155+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:43.559452+0000","last_clean_scrub_stamp":"2024-03-11T21:26:57.367155+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:28:24.056062+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.15","version":"98'9","reported_seq":122,"reported_epoch":105,"state":"active+clean","last_fresh":"2024-03-11T21:27:44.743548+0000","last_change":"2024-03-11T21:27:34.948935+0000","last_active":"2024-03-11T21:27:44.743548+0000","last_peered":"2024-03-11T21:27:44.743548+0000","last_clean":"2024-03-11T21:27:44.743548+0000","last_became_active":"2024-03-11T21:25:42.572555+0000","last_became_peered":"2024-03-11T21:25:42.572555+0000","last_unstale":"2024-03-11T21:27:44.743548+0000","last_undegraded":"2024-03-11T21:27:44.743548+0000","last_fullsized":"2024-03-11T21:27:44.743548+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"98'9","last_scrub_stamp":"2024-03-11T21:27:34.948855+0000","last_deep_scrub":"98'9","last_deep_scrub_stamp":"2024-03-11T21:27:34.948855+0000","last_clean_scrub_stamp":"2024-03-11T21:27:34.948855+0000","objects_scrubbed":1,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:28:45.397760+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.12","version":"0'0","reported_seq":94,"reported_epoch":105,"state":"active+clean","last_fresh":"2024-03-11T21:27:44.743031+0000","last_change":"2024-03-11T21:26:47.960353+0000","last_active":"2024-03-11T21:27:44.743031+0000","last_peered":"2024-03-11T21:27:44.743031+0000","last_clean":"2024-03-11T21:27:44.743031+0000","last_became_active":"2024-03-11T21:25:38.543962+0000","last_became_peered":"2024-03-11T21:25:38.543962+0000","last_unstale":"2024-03-11T21:27:44.743031+0000","last_undegraded":"2024-03-11T21:27:44.743031+0000","last_fullsized":"2024-03-11T21:27:44.743031+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:26:47.960265+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:37.512018+0000","last_clean_scrub_stamp":"2024-03-11T21:26:47.960265+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:28:15.602449+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.14","version":"74'10","reported_seq":105,"reported_epoch":105,"state":"active+clean","last_fresh":"2024-03-11T21:27:44.737856+0000","last_change":"2024-03-11T21:27:19.754590+0000","last_active":"2024-03-11T21:27:44.737856+0000","last_peered":"2024-03-11T21:27:44.737856+0000","last_clean":"2024-03-11T21:27:44.737856+0000","last_became_active":"2024-03-11T21:25:40.760620+0000","last_became_peered":"2024-03-11T21:25:40.760620+0000","last_unstale":"2024-03-11T21:27:44.737856+0000","last_undegraded":"2024-03-11T21:27:44.737856+0000","last_fullsized":"2024-03-11T21:27:44.737856+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'10","last_scrub_stamp":"2024-03-11T21:27:19.754499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:39.525554+0000","last_clean_scrub_stamp":"2024-03-11T21:27:19.754499+0000","objects_scrubbed":5,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:28:40.603422+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.15","version":"0'0","reported_seq":82,"reported_epoch":105,"state":"active+clean","last_fresh":"2024-03-11T21:27:44.739061+0000","last_change":"2024-03-11T21:27:00.378136+0000","last_active":"2024-03-11T21:27:44.739061+0000","last_peered":"2024-03-11T21:27:44.739061+0000","last_clean":"2024-03-11T21:27:44.739061+0000","last_became_active":"2024-03-11T21:25:45.302501+0000","last_became_peered":"2024-03-11T21:25:45.302501+0000","last_unstale":"2024-03-11T21:27:44.739061+0000","last_undegraded":"2024-03-11T21:27:44.739061+0000","last_fullsized":"2024-03-11T21:27:44.739061+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:27:00.377987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:43.559452+0000","last_clean_scrub_stamp":"2024-03-11T21:27:00.377987+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:28:21.911670+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":86,"reported_epoch":105,"state":"active+clean","last_fresh":"2024-03-11T21:27:44.743383+0000","last_change":"2024-03-11T21:27:33.987450+0000","last_active":"2024-03-11T21:27:44.743383+0000","last_peered":"2024-03-11T21:27:44.743383+0000","last_clean":"2024-03-11T21:27:44.743383+0000","last_became_active":"2024-03-11T21:25:42.577351+0000","last_became_peered":"2024-03-11T21:25:42.577351+0000","last_unstale":"2024-03-11T21:27:44.743383+0000","last_undegraded":"2024-03-11T21:27:44.743383+0000","last_fullsized":"2024-03-11T21:27:44.743383+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:27:33.987367+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:41.544550+0000","last_clean_scrub_stamp":"2024-03-11T21:27:33.987367+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:28:41.819337+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,1],"acting":[5,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.11","version":"0'0","reported_seq":95,"reported_epoch":105,"state":"active+clean","last_fresh":"2024-03-11T21:27:44.732891+0000","last_change":"2024-03-11T21:27:16.985709+0000","last_active":"2024-03-11T21:27:44.732891+0000","last_peered":"2024-03-11T21:27:44.732891+0000","last_clean":"2024-03-11T21:27:44.732891+0000","last_became_active":"2024-03-11T21:25:38.537415+0000","last_became_peered":"2024-03-11T21:25:38.537415+0000","last_unstale":"2024-03-11T21:27:44.732891+0000","last_undegraded":"2024-03-11T21:27:44.732891+0000","last_fullsized":"2024-03-11T21:27:44.732891+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:27:16.985601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:37.512018+0000","last_clean_scrub_stamp":"2024-03-11T21:27:16.985601+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:28:27.096179+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.17","version":"97'9","reported_seq":116,"reported_epoch":105,"state":"active+clean","last_fresh":"2024-03-11T21:27:44.736718+0000","last_change":"2024-03-11T21:27:25.803221+0000","last_active":"2024-03-11T21:27:44.736718+0000","last_peered":"2024-03-11T21:27:44.736718+0000","last_clean":"2024-03-11T21:27:44.736718+0000","last_became_active":"2024-03-11T21:26:11.219020+0000","last_became_peered":"2024-03-11T21:26:11.219020+0000","last_unstale":"2024-03-11T21:27:44.736718+0000","last_undegraded":"2024-03-11T21:27:44.736718+0000","last_fullsized":"2024-03-11T21:27:44.736718+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"97'9","last_scrub_stamp":"2024-03-11T21:27:25.803132+0000","last_deep_scrub":"97'9","last_deep_scrub_stamp":"2024-03-11T21:27:25.803132+0000","last_clean_scrub_stamp":"2024-03-11T21:27:25.803132+0000","objects_scrubbed":3,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:28:40.421034+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":9,"num_read_kb":6,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,5],"acting":[3,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.14","version":"0'0","reported_seq":82,"reported_epoch":105,"state":"active+clean","last_fresh":"2024-03-11T21:27:44.746351+0000","last_change":"2024-03-11T21:27:21.063894+0000","last_active":"2024-03-11T21:27:44.746351+0000","last_peered":"2024-03-11T21:27:44.746351+0000","last_clean":"2024-03-11T21:27:44.746351+0000","last_became_active":"2024-03-11T21:25:44.587439+0000","last_became_peered":"2024-03-11T21:25:44.587439+0000","last_unstale":"2024-03-11T21:27:44.746351+0000","last_undegraded":"2024-03-11T21:27:44.746351+0000","last_fullsized":"2024-03-11T21:27:44.746351+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:27:21.063817+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:43.559452+0000","last_clean_scrub_stamp":"2024-03-11T21:27:21.063817+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:28:29.835843+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,7],"acting":[2,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":86,"reported_epoch":105,"state":"active+clean","last_fresh":"2024-03-11T21:27:44.736810+0000","last_change":"2024-03-11T21:27:15.766288+0000","last_active":"2024-03-11T21:27:44.736810+0000","last_peered":"2024-03-11T21:27:44.736810+0000","last_clean":"2024-03-11T21:27:44.736810+0000","last_became_active":"2024-03-11T21:25:42.583514+0000","last_became_peered":"2024-03-11T21:25:42.583514+0000","last_unstale":"2024-03-11T21:27:44.736810+0000","last_undegraded":"2024-03-11T21:27:44.736810+0000","last_fullsized":"2024-03-11T21:27:44.736810+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:27:15.766188+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:41.544550+0000","last_clean_scrub_stamp":"2024-03-11T21:27:15.766188+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:28:22.407684+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.10","version":"0'0","reported_seq":94,"reported_epoch":105,"state":"active+clean","last_fresh":"2024-03-11T21:27:44.745767+0000","last_change":"2024-03-11T21:27:09.937508+0000","last_active":"2024-03-11T21:27:44.745767+0000","last_peered":"2024-03-11T21:27:44.745767+0000","last_clean":"2024-03-11T21:27:44.745767+0000","last_became_active":"2024-03-11T21:25:38.533138+0000","last_became_peered":"2024-03-11T21:25:38.533138+0000","last_unstale":"2024-03-11T21:27:44.745767+0000","last_undegraded":"2024-03-11T21:27:44.745767+0000","last_fullsized":"2024-03-11T21:27:44.745767+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:27:09.937426+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:37.512018+0000","last_clean_scrub_stamp":"2024-03-11T21:27:09.937426+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:28:23.648986+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,0],"acting":[2,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.16","version":"74'9","reported_seq":107,"reported_epoch":105,"state":"active+clean","last_fresh":"2024-03-11T21:27:44.737031+0000","last_change":"2024-03-11T21:26:53.490093+0000","last_active":"2024-03-11T21:27:44.737031+0000","last_peered":"2024-03-11T21:27:44.737031+0000","last_clean":"2024-03-11T21:27:44.737031+0000","last_became_active":"2024-03-11T21:25:40.763434+0000","last_became_peered":"2024-03-11T21:25:40.763434+0000","last_unstale":"2024-03-11T21:27:44.737031+0000","last_undegraded":"2024-03-11T21:27:44.737031+0000","last_fullsized":"2024-03-11T21:27:44.737031+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'9","last_scrub_stamp":"2024-03-11T21:26:53.489953+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:39.525554+0000","last_clean_scrub_stamp":"2024-03-11T21:26:53.489953+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:27:55.665353+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.a","version":"105'1220","reported_seq":1240,"reported_epoch":105,"state":"active+clean","last_fresh":"2024-03-11T21:27:51.976761+0000","last_change":"2024-03-11T21:27:36.693277+0000","last_active":"2024-03-11T21:27:51.976761+0000","last_peered":"2024-03-11T21:27:51.976761+0000","last_clean":"2024-03-11T21:27:51.976761+0000","last_became_active":"2024-03-11T21:27:36.692988+0000","last_became_peered":"2024-03-11T21:27:36.692988+0000","last_unstale":"2024-03-11T21:27:51.976761+0000","last_undegraded":"2024-03-11T21:27:51.976761+0000","last_fullsized":"2024-03-11T21:27:51.976761+0000","mapping_epoch":99,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:27:35.667522+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:27:35.667522+0000","last_clean_scrub_stamp":"2024-03-11T21:27:35.667522+0000","objects_scrubbed":0,"log_size":1220,"log_dups_size":0,"ondisk_log_size":1220,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:28:43.758763+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":79953920,"num_objects":1220,"num_object_clones":0,"num_object_copies":3660,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1220,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1220,"num_write_kb":78080,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5],"acting":[3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.b","version":"0'0","reported_seq":74,"reported_epoch":105,"state":"active+clean","last_fresh":"2024-03-11T21:27:44.735797+0000","last_change":"2024-03-11T21:25:44.590403+0000","last_active":"2024-03-11T21:27:44.735797+0000","last_peered":"2024-03-11T21:27:44.735797+0000","last_clean":"2024-03-11T21:27:44.735797+0000","last_became_active":"2024-03-11T21:25:44.590071+0000","last_became_peered":"2024-03-11T21:25:44.590071+0000","last_unstale":"2024-03-11T21:27:44.735797+0000","last_undegraded":"2024-03-11T21:27:44.735797+0000","last_fullsized":"2024-03-11T21:27:44.735797+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:25:43.559452+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:43.559452+0000","last_clean_scrub_stamp":"2024-03-11T21:25:43.559452+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:27:46.744766+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.f","version":"67'2","reported_seq":107,"reported_epoch":105,"state":"active+clean","last_fresh":"2024-03-11T21:27:44.731027+0000","last_change":"2024-03-11T21:27:21.959075+0000","last_active":"2024-03-11T21:27:44.731027+0000","last_peered":"2024-03-11T21:27:44.731027+0000","last_clean":"2024-03-11T21:27:44.731027+0000","last_became_active":"2024-03-11T21:25:38.537711+0000","last_became_peered":"2024-03-11T21:25:38.537711+0000","last_unstale":"2024-03-11T21:27:44.731027+0000","last_undegraded":"2024-03-11T21:27:44.731027+0000","last_fullsized":"2024-03-11T21:27:44.731027+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'2","last_scrub_stamp":"2024-03-11T21:27:21.958970+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:37.512018+0000","last_clean_scrub_stamp":"2024-03-11T21:27:21.958970+0000","objects_scrubbed":2,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:28:34.591168+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":92,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":10,"num_read_kb":10,"num_write":4,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,7],"acting":[4,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.9","version":"74'12","reported_seq":107,"reported_epoch":105,"state":"active+clean","last_fresh":"2024-03-11T21:27:44.729948+0000","last_change":"2024-03-11T21:25:40.552685+0000","last_active":"2024-03-11T21:27:44.729948+0000","last_peered":"2024-03-11T21:27:44.729948+0000","last_clean":"2024-03-11T21:27:44.729948+0000","last_became_active":"2024-03-11T21:25:40.552267+0000","last_became_peered":"2024-03-11T21:25:40.552267+0000","last_unstale":"2024-03-11T21:27:44.729948+0000","last_undegraded":"2024-03-11T21:27:44.729948+0000","last_fullsized":"2024-03-11T21:27:44.729948+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:25:39.525554+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:39.525554+0000","last_clean_scrub_stamp":"2024-03-11T21:25:39.525554+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:27:52.810441+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":25,"num_read_kb":16,"num_write":14,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,3],"acting":[4,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":87,"reported_epoch":105,"state":"active+clean","last_fresh":"2024-03-11T21:27:44.745713+0000","last_change":"2024-03-11T21:27:08.954258+0000","last_active":"2024-03-11T21:27:44.745713+0000","last_peered":"2024-03-11T21:27:44.745713+0000","last_clean":"2024-03-11T21:27:44.745713+0000","last_became_active":"2024-03-11T21:25:42.573987+0000","last_became_peered":"2024-03-11T21:25:42.573987+0000","last_unstale":"2024-03-11T21:27:44.745713+0000","last_undegraded":"2024-03-11T21:27:44.745713+0000","last_fullsized":"2024-03-11T21:27:44.745713+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:27:08.954143+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:41.544550+0000","last_clean_scrub_stamp":"2024-03-11T21:27:08.954143+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:28:34.087042+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.1","version":"105'1236","reported_seq":1257,"reported_epoch":105,"state":"active+clean","last_fresh":"2024-03-11T21:27:51.963238+0000","last_change":"2024-03-11T21:27:36.697329+0000","last_active":"2024-03-11T21:27:51.963238+0000","last_peered":"2024-03-11T21:27:51.963238+0000","last_clean":"2024-03-11T21:27:51.963238+0000","last_became_active":"2024-03-11T21:27:36.696990+0000","last_became_peered":"2024-03-11T21:27:36.696990+0000","last_unstale":"2024-03-11T21:27:51.963238+0000","last_undegraded":"2024-03-11T21:27:51.963238+0000","last_fullsized":"2024-03-11T21:27:51.963238+0000","mapping_epoch":99,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:27:35.667522+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:27:35.667522+0000","last_clean_scrub_stamp":"2024-03-11T21:27:35.667522+0000","objects_scrubbed":0,"log_size":1236,"log_dups_size":0,"ondisk_log_size":1236,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:28:46.268780+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":81002496,"num_objects":1236,"num_object_clones":0,"num_object_copies":3708,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1236,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1236,"num_write_kb":79104,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.0","version":"0'0","reported_seq":84,"reported_epoch":105,"state":"active+clean","last_fresh":"2024-03-11T21:27:44.732810+0000","last_change":"2024-03-11T21:27:23.389441+0000","last_active":"2024-03-11T21:27:44.732810+0000","last_peered":"2024-03-11T21:27:44.732810+0000","last_clean":"2024-03-11T21:27:44.732810+0000","last_became_active":"2024-03-11T21:25:44.584565+0000","last_became_peered":"2024-03-11T21:25:44.584565+0000","last_unstale":"2024-03-11T21:27:44.732810+0000","last_undegraded":"2024-03-11T21:27:44.732810+0000","last_fullsized":"2024-03-11T21:27:44.732810+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:27:23.389362+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:43.559452+0000","last_clean_scrub_stamp":"2024-03-11T21:27:23.389362+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:28:49.772995+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":96,"reported_epoch":105,"state":"active+clean","last_fresh":"2024-03-11T21:27:44.733877+0000","last_change":"2024-03-11T21:27:06.092300+0000","last_active":"2024-03-11T21:27:44.733877+0000","last_peered":"2024-03-11T21:27:44.733877+0000","last_clean":"2024-03-11T21:27:44.733877+0000","last_became_active":"2024-03-11T21:25:38.541760+0000","last_became_peered":"2024-03-11T21:25:38.541760+0000","last_unstale":"2024-03-11T21:27:44.733877+0000","last_undegraded":"2024-03-11T21:27:44.733877+0000","last_fullsized":"2024-03-11T21:27:44.733877+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:27:06.092220+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:37.512018+0000","last_clean_scrub_stamp":"2024-03-11T21:27:06.092220+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:28:22.259535+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,7],"acting":[1,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.2","version":"94'15","reported_seq":112,"reported_epoch":105,"state":"active+clean","last_fresh":"2024-03-11T21:27:44.733504+0000","last_change":"2024-03-11T21:26:49.993206+0000","last_active":"2024-03-11T21:27:44.733504+0000","last_peered":"2024-03-11T21:27:44.733504+0000","last_clean":"2024-03-11T21:27:44.733504+0000","last_became_active":"2024-03-11T21:25:40.548833+0000","last_became_peered":"2024-03-11T21:25:40.548833+0000","last_unstale":"2024-03-11T21:27:44.733504+0000","last_undegraded":"2024-03-11T21:27:44.733504+0000","last_fullsized":"2024-03-11T21:27:44.733504+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"94'15","last_scrub_stamp":"2024-03-11T21:26:49.993060+0000","last_deep_scrub":"94'15","last_deep_scrub_stamp":"2024-03-11T21:26:49.993060+0000","last_clean_scrub_stamp":"2024-03-11T21:26:49.993060+0000","objects_scrubbed":5,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:27:59.041594+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.3","version":"74'8","reported_seq":123,"reported_epoch":105,"state":"active+clean","last_fresh":"2024-03-11T21:27:46.964803+0000","last_change":"2024-03-11T21:27:06.321926+0000","last_active":"2024-03-11T21:27:46.964803+0000","last_peered":"2024-03-11T21:27:46.964803+0000","last_clean":"2024-03-11T21:27:46.964803+0000","last_became_active":"2024-03-11T21:25:42.578161+0000","last_became_peered":"2024-03-11T21:25:42.578161+0000","last_unstale":"2024-03-11T21:27:46.964803+0000","last_undegraded":"2024-03-11T21:27:46.964803+0000","last_fullsized":"2024-03-11T21:27:46.964803+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'8","last_scrub_stamp":"2024-03-11T21:27:06.321832+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:41.544550+0000","last_clean_scrub_stamp":"2024-03-11T21:27:06.321832+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:28:27.084331+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,5],"acting":[0,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.7","version":"105'1296","reported_seq":1318,"reported_epoch":105,"state":"active+clean","last_fresh":"2024-03-11T21:27:51.962942+0000","last_change":"2024-03-11T21:27:36.695426+0000","last_active":"2024-03-11T21:27:51.962942+0000","last_peered":"2024-03-11T21:27:51.962942+0000","last_clean":"2024-03-11T21:27:51.962942+0000","last_became_active":"2024-03-11T21:27:36.695055+0000","last_became_peered":"2024-03-11T21:27:36.695055+0000","last_unstale":"2024-03-11T21:27:51.962942+0000","last_undegraded":"2024-03-11T21:27:51.962942+0000","last_fullsized":"2024-03-11T21:27:51.962942+0000","mapping_epoch":99,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:27:35.667522+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:27:35.667522+0000","last_clean_scrub_stamp":"2024-03-11T21:27:35.667522+0000","objects_scrubbed":0,"log_size":1296,"log_dups_size":0,"ondisk_log_size":1296,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:29:04.544436+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":84934656,"num_objects":1296,"num_object_clones":0,"num_object_copies":3888,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1296,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1296,"num_write_kb":82944,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,3],"acting":[6,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":82,"reported_epoch":105,"state":"active+clean","last_fresh":"2024-03-11T21:27:44.737102+0000","last_change":"2024-03-11T21:27:07.651113+0000","last_active":"2024-03-11T21:27:44.737102+0000","last_peered":"2024-03-11T21:27:44.737102+0000","last_clean":"2024-03-11T21:27:44.737102+0000","last_became_active":"2024-03-11T21:25:44.585223+0000","last_became_peered":"2024-03-11T21:25:44.585223+0000","last_unstale":"2024-03-11T21:27:44.737102+0000","last_undegraded":"2024-03-11T21:27:44.737102+0000","last_fullsized":"2024-03-11T21:27:44.737102+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:27:07.651016+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:43.559452+0000","last_clean_scrub_stamp":"2024-03-11T21:27:07.651016+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:28:14.928745+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,7],"acting":[3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":95,"reported_epoch":105,"state":"active+clean","last_fresh":"2024-03-11T21:27:44.743164+0000","last_change":"2024-03-11T21:26:45.072161+0000","last_active":"2024-03-11T21:27:44.743164+0000","last_peered":"2024-03-11T21:27:44.743164+0000","last_clean":"2024-03-11T21:27:44.743164+0000","last_became_active":"2024-03-11T21:25:38.538540+0000","last_became_peered":"2024-03-11T21:25:38.538540+0000","last_unstale":"2024-03-11T21:27:44.743164+0000","last_undegraded":"2024-03-11T21:27:44.743164+0000","last_fullsized":"2024-03-11T21:27:44.743164+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:26:45.072058+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:37.512018+0000","last_clean_scrub_stamp":"2024-03-11T21:26:45.072058+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:27:49.711890+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.4","version":"74'28","reported_seq":167,"reported_epoch":105,"state":"active+clean","last_fresh":"2024-03-11T21:27:46.969532+0000","last_change":"2024-03-11T21:27:08.173608+0000","last_active":"2024-03-11T21:27:46.969532+0000","last_peered":"2024-03-11T21:27:46.969532+0000","last_clean":"2024-03-11T21:27:46.969532+0000","last_became_active":"2024-03-11T21:25:40.552902+0000","last_became_peered":"2024-03-11T21:25:40.552902+0000","last_unstale":"2024-03-11T21:27:46.969532+0000","last_undegraded":"2024-03-11T21:27:46.969532+0000","last_fullsized":"2024-03-11T21:27:46.969532+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'28","last_scrub_stamp":"2024-03-11T21:27:08.173498+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:39.525554+0000","last_clean_scrub_stamp":"2024-03-11T21:27:08.173498+0000","objects_scrubbed":10,"log_size":28,"log_dups_size":0,"ondisk_log_size":28,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:28:21.301768+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":358,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":48,"num_read_kb":33,"num_write":26,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3],"acting":[1,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":88,"reported_epoch":105,"state":"active+clean","last_fresh":"2024-03-11T21:27:44.734612+0000","last_change":"2024-03-11T21:27:20.393288+0000","last_active":"2024-03-11T21:27:44.734612+0000","last_peered":"2024-03-11T21:27:44.734612+0000","last_clean":"2024-03-11T21:27:44.734612+0000","last_became_active":"2024-03-11T21:25:42.572673+0000","last_became_peered":"2024-03-11T21:25:42.572673+0000","last_unstale":"2024-03-11T21:27:44.734612+0000","last_undegraded":"2024-03-11T21:27:44.734612+0000","last_fullsized":"2024-03-11T21:27:44.734612+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:27:20.393205+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:41.544550+0000","last_clean_scrub_stamp":"2024-03-11T21:27:20.393205+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:28:49.759997+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.4","version":"105'1323","reported_seq":1345,"reported_epoch":105,"state":"active+clean","last_fresh":"2024-03-11T21:27:51.949175+0000","last_change":"2024-03-11T21:27:36.691053+0000","last_active":"2024-03-11T21:27:51.949175+0000","last_peered":"2024-03-11T21:27:51.949175+0000","last_clean":"2024-03-11T21:27:51.949175+0000","last_became_active":"2024-03-11T21:27:36.690751+0000","last_became_peered":"2024-03-11T21:27:36.690751+0000","last_unstale":"2024-03-11T21:27:51.949175+0000","last_undegraded":"2024-03-11T21:27:51.949175+0000","last_fullsized":"2024-03-11T21:27:51.949175+0000","mapping_epoch":99,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:27:35.667522+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:27:35.667522+0000","last_clean_scrub_stamp":"2024-03-11T21:27:35.667522+0000","objects_scrubbed":0,"log_size":1323,"log_dups_size":0,"ondisk_log_size":1323,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:28:49.540586+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":86704128,"num_objects":1323,"num_object_clones":0,"num_object_copies":3969,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1323,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1323,"num_write_kb":84672,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,1],"acting":[0,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.5","version":"0'0","reported_seq":83,"reported_epoch":105,"state":"active+clean","last_fresh":"2024-03-11T21:27:44.739290+0000","last_change":"2024-03-11T21:27:05.338089+0000","last_active":"2024-03-11T21:27:44.739290+0000","last_peered":"2024-03-11T21:27:44.739290+0000","last_clean":"2024-03-11T21:27:44.739290+0000","last_became_active":"2024-03-11T21:25:45.302120+0000","last_became_peered":"2024-03-11T21:25:45.302120+0000","last_unstale":"2024-03-11T21:27:44.739290+0000","last_undegraded":"2024-03-11T21:27:44.739290+0000","last_fullsized":"2024-03-11T21:27:44.739290+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:27:05.337991+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:43.559452+0000","last_clean_scrub_stamp":"2024-03-11T21:27:05.337991+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:28:33.442171+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,3],"acting":[7,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.0","version":"76'5","reported_seq":234,"reported_epoch":105,"state":"active+clean","last_fresh":"2024-03-11T21:27:50.658360+0000","last_change":"2024-03-11T21:26:54.088445+0000","last_active":"2024-03-11T21:27:50.658360+0000","last_peered":"2024-03-11T21:27:50.658360+0000","last_clean":"2024-03-11T21:27:50.658360+0000","last_became_active":"2024-03-11T21:25:38.536890+0000","last_became_peered":"2024-03-11T21:25:38.536890+0000","last_unstale":"2024-03-11T21:27:50.658360+0000","last_undegraded":"2024-03-11T21:27:50.658360+0000","last_fullsized":"2024-03-11T21:27:50.658360+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"76'5","last_scrub_stamp":"2024-03-11T21:26:54.088356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:37.512018+0000","last_clean_scrub_stamp":"2024-03-11T21:26:54.088356+0000","objects_scrubbed":1,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:28:21.305217+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.000753559,"stat_sum":{"num_bytes":389,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":128,"num_read_kb":123,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,6],"acting":[1,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":95,"reported_epoch":105,"state":"active+clean","last_fresh":"2024-03-11T21:27:44.745618+0000","last_change":"2024-03-11T21:26:48.997561+0000","last_active":"2024-03-11T21:27:44.745618+0000","last_peered":"2024-03-11T21:27:44.745618+0000","last_clean":"2024-03-11T21:27:44.745618+0000","last_became_active":"2024-03-11T21:25:38.532984+0000","last_became_peered":"2024-03-11T21:25:38.532984+0000","last_unstale":"2024-03-11T21:27:44.745618+0000","last_undegraded":"2024-03-11T21:27:44.745618+0000","last_fullsized":"2024-03-11T21:27:44.745618+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:26:48.997454+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:26:48.997454+0000","last_clean_scrub_stamp":"2024-03-11T21:26:48.997454+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:28:14.373168+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.7","version":"97'18","reported_seq":124,"reported_epoch":105,"state":"active+clean","last_fresh":"2024-03-11T21:27:44.738893+0000","last_change":"2024-03-11T21:27:19.239524+0000","last_active":"2024-03-11T21:27:44.738893+0000","last_peered":"2024-03-11T21:27:44.738893+0000","last_clean":"2024-03-11T21:27:44.738893+0000","last_became_active":"2024-03-11T21:25:40.541769+0000","last_became_peered":"2024-03-11T21:25:40.541769+0000","last_unstale":"2024-03-11T21:27:44.738893+0000","last_undegraded":"2024-03-11T21:27:44.738893+0000","last_fullsized":"2024-03-11T21:27:44.738893+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"97'18","last_scrub_stamp":"2024-03-11T21:27:19.239404+0000","last_deep_scrub":"97'18","last_deep_scrub_stamp":"2024-03-11T21:27:19.239404+0000","last_clean_scrub_stamp":"2024-03-11T21:27:19.239404+0000","objects_scrubbed":5,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:28:31.185368+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":19,"num_write":16,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":87,"reported_epoch":105,"state":"active+clean","last_fresh":"2024-03-11T21:27:44.747622+0000","last_change":"2024-03-11T21:27:02.039134+0000","last_active":"2024-03-11T21:27:44.747622+0000","last_peered":"2024-03-11T21:27:44.747622+0000","last_clean":"2024-03-11T21:27:44.747622+0000","last_became_active":"2024-03-11T21:25:42.581710+0000","last_became_peered":"2024-03-11T21:25:42.581710+0000","last_unstale":"2024-03-11T21:27:44.747622+0000","last_undegraded":"2024-03-11T21:27:44.747622+0000","last_fullsized":"2024-03-11T21:27:44.747622+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:27:02.039041+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:41.544550+0000","last_clean_scrub_stamp":"2024-03-11T21:27:02.039041+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:28:26.912856+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7],"acting":[2,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.6","version":"105'1287","reported_seq":1309,"reported_epoch":105,"state":"active+clean","last_fresh":"2024-03-11T21:27:51.966839+0000","last_change":"2024-03-11T21:27:36.695308+0000","last_active":"2024-03-11T21:27:51.966839+0000","last_peered":"2024-03-11T21:27:51.966839+0000","last_clean":"2024-03-11T21:27:51.966839+0000","last_became_active":"2024-03-11T21:27:36.695018+0000","last_became_peered":"2024-03-11T21:27:36.695018+0000","last_unstale":"2024-03-11T21:27:51.966839+0000","last_undegraded":"2024-03-11T21:27:51.966839+0000","last_fullsized":"2024-03-11T21:27:51.966839+0000","mapping_epoch":99,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:27:35.667522+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:27:35.667522+0000","last_clean_scrub_stamp":"2024-03-11T21:27:35.667522+0000","objects_scrubbed":0,"log_size":1287,"log_dups_size":0,"ondisk_log_size":1287,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:28:37.233229+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":84344832,"num_objects":1287,"num_object_clones":0,"num_object_copies":3861,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1287,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1287,"num_write_kb":82368,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.7","version":"74'1","reported_seq":75,"reported_epoch":105,"state":"active+clean","last_fresh":"2024-03-11T21:27:44.743839+0000","last_change":"2024-03-11T21:25:44.591645+0000","last_active":"2024-03-11T21:27:44.743839+0000","last_peered":"2024-03-11T21:27:44.743839+0000","last_clean":"2024-03-11T21:27:44.743839+0000","last_became_active":"2024-03-11T21:25:44.591376+0000","last_became_peered":"2024-03-11T21:25:44.591376+0000","last_unstale":"2024-03-11T21:27:44.743839+0000","last_undegraded":"2024-03-11T21:27:44.743839+0000","last_fullsized":"2024-03-11T21:27:44.743839+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:25:43.559452+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:43.559452+0000","last_clean_scrub_stamp":"2024-03-11T21:25:43.559452+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,4],"acting":[5,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2","version":"71'2","reported_seq":104,"reported_epoch":105,"state":"active+clean","last_fresh":"2024-03-11T21:27:44.733532+0000","last_change":"2024-03-11T21:26:42.663119+0000","last_active":"2024-03-11T21:27:44.733532+0000","last_peered":"2024-03-11T21:27:44.733532+0000","last_clean":"2024-03-11T21:27:44.733532+0000","last_became_active":"2024-03-11T21:25:38.539413+0000","last_became_peered":"2024-03-11T21:25:38.539413+0000","last_unstale":"2024-03-11T21:27:44.733532+0000","last_undegraded":"2024-03-11T21:27:44.733532+0000","last_fullsized":"2024-03-11T21:27:44.733532+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"71'2","last_scrub_stamp":"2024-03-11T21:26:42.662979+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:37.512018+0000","last_clean_scrub_stamp":"2024-03-11T21:26:42.662979+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:27:54.085660+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.0014206449999999999,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":101,"reported_epoch":105,"state":"active+clean","last_fresh":"2024-03-11T21:27:44.743316+0000","last_change":"2024-03-11T21:26:42.171361+0000","last_active":"2024-03-11T21:27:44.743316+0000","last_peered":"2024-03-11T21:27:44.743316+0000","last_clean":"2024-03-11T21:27:44.743316+0000","last_became_active":"2024-03-11T21:26:11.216188+0000","last_became_peered":"2024-03-11T21:26:11.216188+0000","last_unstale":"2024-03-11T21:27:44.743316+0000","last_undegraded":"2024-03-11T21:27:44.743316+0000","last_fullsized":"2024-03-11T21:27:44.743316+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:26:42.171256+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:37.512018+0000","last_clean_scrub_stamp":"2024-03-11T21:26:42.171256+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:27:58.632937+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,4],"acting":[5,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"87'128","reported_seq":340,"reported_epoch":105,"state":"active+clean","last_fresh":"2024-03-11T21:27:44.738454+0000","last_change":"2024-03-11T21:26:11.340374+0000","last_active":"2024-03-11T21:27:44.738454+0000","last_peered":"2024-03-11T21:27:44.738454+0000","last_clean":"2024-03-11T21:27:44.738454+0000","last_became_active":"2024-03-11T21:25:30.502597+0000","last_became_peered":"2024-03-11T21:25:30.502597+0000","last_unstale":"2024-03-11T21:27:44.738454+0000","last_undegraded":"2024-03-11T21:27:44.738454+0000","last_fullsized":"2024-03-11T21:27:44.738454+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'117","last_scrub_stamp":"2024-03-11T21:26:11.340228+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:24.261974+0000","last_clean_scrub_stamp":"2024-03-11T21:26:11.340228+0000","objects_scrubbed":2,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":212,"num_read_kb":306,"num_write":179,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1771104,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.5","version":"74'16","reported_seq":141,"reported_epoch":105,"state":"active+clean","last_fresh":"2024-03-11T21:27:46.972640+0000","last_change":"2024-03-11T21:27:13.982835+0000","last_active":"2024-03-11T21:27:46.972640+0000","last_peered":"2024-03-11T21:27:46.972640+0000","last_clean":"2024-03-11T21:27:46.972640+0000","last_became_active":"2024-03-11T21:25:40.761457+0000","last_became_peered":"2024-03-11T21:25:40.761457+0000","last_unstale":"2024-03-11T21:27:46.972640+0000","last_undegraded":"2024-03-11T21:27:46.972640+0000","last_fullsized":"2024-03-11T21:27:46.972640+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'16","last_scrub_stamp":"2024-03-11T21:27:13.982681+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:39.525554+0000","last_clean_scrub_stamp":"2024-03-11T21:27:13.982681+0000","objects_scrubbed":8,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:28:28.254017+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":154,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":25,"num_read_kb":15,"num_write":13,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":87,"reported_epoch":105,"state":"active+clean","last_fresh":"2024-03-11T21:27:44.739098+0000","last_change":"2024-03-11T21:26:58.350384+0000","last_active":"2024-03-11T21:27:44.739098+0000","last_peered":"2024-03-11T21:27:44.739098+0000","last_clean":"2024-03-11T21:27:44.739098+0000","last_became_active":"2024-03-11T21:25:42.586226+0000","last_became_peered":"2024-03-11T21:25:42.586226+0000","last_unstale":"2024-03-11T21:27:44.739098+0000","last_undegraded":"2024-03-11T21:27:44.739098+0000","last_fullsized":"2024-03-11T21:27:44.739098+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:26:58.350270+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:41.544550+0000","last_clean_scrub_stamp":"2024-03-11T21:26:58.350270+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:28:06.391624+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,5],"acting":[7,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.5","version":"105'1250","reported_seq":1271,"reported_epoch":105,"state":"active+clean","last_fresh":"2024-03-11T21:27:51.957741+0000","last_change":"2024-03-11T21:27:36.692195+0000","last_active":"2024-03-11T21:27:51.957741+0000","last_peered":"2024-03-11T21:27:51.957741+0000","last_clean":"2024-03-11T21:27:51.957741+0000","last_became_active":"2024-03-11T21:27:36.691428+0000","last_became_peered":"2024-03-11T21:27:36.691428+0000","last_unstale":"2024-03-11T21:27:51.957741+0000","last_undegraded":"2024-03-11T21:27:51.957741+0000","last_fullsized":"2024-03-11T21:27:51.957741+0000","mapping_epoch":99,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:27:35.667522+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:27:35.667522+0000","last_clean_scrub_stamp":"2024-03-11T21:27:35.667522+0000","objects_scrubbed":0,"log_size":1250,"log_dups_size":0,"ondisk_log_size":1250,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:29:01.981820+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":81920000,"num_objects":1250,"num_object_clones":0,"num_object_copies":3750,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1250,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1250,"num_write_kb":80000,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,0],"acting":[7,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.4","version":"0'0","reported_seq":84,"reported_epoch":105,"state":"active+clean","last_fresh":"2024-03-11T21:27:44.734182+0000","last_change":"2024-03-11T21:27:00.119584+0000","last_active":"2024-03-11T21:27:44.734182+0000","last_peered":"2024-03-11T21:27:44.734182+0000","last_clean":"2024-03-11T21:27:44.734182+0000","last_became_active":"2024-03-11T21:25:44.594188+0000","last_became_peered":"2024-03-11T21:25:44.594188+0000","last_unstale":"2024-03-11T21:27:44.734182+0000","last_undegraded":"2024-03-11T21:27:44.734182+0000","last_fullsized":"2024-03-11T21:27:44.734182+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:27:00.119508+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:43.559452+0000","last_clean_scrub_stamp":"2024-03-11T21:27:00.119508+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:28:23.003619+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":103,"reported_epoch":105,"state":"active+clean","last_fresh":"2024-03-11T21:27:44.735113+0000","last_change":"2024-03-11T21:26:38.589294+0000","last_active":"2024-03-11T21:27:44.735113+0000","last_peered":"2024-03-11T21:27:44.735113+0000","last_clean":"2024-03-11T21:27:44.735113+0000","last_became_active":"2024-03-11T21:25:38.543834+0000","last_became_peered":"2024-03-11T21:25:38.543834+0000","last_unstale":"2024-03-11T21:27:44.735113+0000","last_undegraded":"2024-03-11T21:27:44.735113+0000","last_fullsized":"2024-03-11T21:27:44.735113+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:26:38.589187+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:37.512018+0000","last_clean_scrub_stamp":"2024-03-11T21:26:38.589187+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:27:54.798633+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0.00089342499999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":95,"reported_epoch":105,"state":"active+clean","last_fresh":"2024-03-11T21:27:44.738386+0000","last_change":"2024-03-11T21:26:51.473044+0000","last_active":"2024-03-11T21:27:44.738386+0000","last_peered":"2024-03-11T21:27:44.738386+0000","last_clean":"2024-03-11T21:27:44.738386+0000","last_became_active":"2024-03-11T21:25:38.541617+0000","last_became_peered":"2024-03-11T21:25:38.541617+0000","last_unstale":"2024-03-11T21:27:44.738386+0000","last_undegraded":"2024-03-11T21:27:44.738386+0000","last_fullsized":"2024-03-11T21:27:44.738386+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:26:51.472965+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:37.512018+0000","last_clean_scrub_stamp":"2024-03-11T21:26:51.472965+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:28:11.852536+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,0],"acting":[7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.6","version":"74'12","reported_seq":110,"reported_epoch":105,"state":"active+clean","last_fresh":"2024-03-11T21:27:44.737031+0000","last_change":"2024-03-11T21:27:01.353450+0000","last_active":"2024-03-11T21:27:44.737031+0000","last_peered":"2024-03-11T21:27:44.737031+0000","last_clean":"2024-03-11T21:27:44.737031+0000","last_became_active":"2024-03-11T21:25:40.550388+0000","last_became_peered":"2024-03-11T21:25:40.550388+0000","last_unstale":"2024-03-11T21:27:44.737031+0000","last_undegraded":"2024-03-11T21:27:44.737031+0000","last_fullsized":"2024-03-11T21:27:44.737031+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'12","last_scrub_stamp":"2024-03-11T21:27:01.353347+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:39.525554+0000","last_clean_scrub_stamp":"2024-03-11T21:27:01.353347+0000","objects_scrubbed":6,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:28:25.846115+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":18,"num_read_kb":12,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":86,"reported_epoch":105,"state":"active+clean","last_fresh":"2024-03-11T21:27:44.743781+0000","last_change":"2024-03-11T21:27:32.991020+0000","last_active":"2024-03-11T21:27:44.743781+0000","last_peered":"2024-03-11T21:27:44.743781+0000","last_clean":"2024-03-11T21:27:44.743781+0000","last_became_active":"2024-03-11T21:25:42.572324+0000","last_became_peered":"2024-03-11T21:25:42.572324+0000","last_unstale":"2024-03-11T21:27:44.743781+0000","last_undegraded":"2024-03-11T21:27:44.743781+0000","last_fullsized":"2024-03-11T21:27:44.743781+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:27:32.990943+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:41.544550+0000","last_clean_scrub_stamp":"2024-03-11T21:27:32.990943+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:28:42.753007+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.0","version":"105'1249","reported_seq":1270,"reported_epoch":105,"state":"active+clean","last_fresh":"2024-03-11T21:27:51.948670+0000","last_change":"2024-03-11T21:27:36.694516+0000","last_active":"2024-03-11T21:27:51.948670+0000","last_peered":"2024-03-11T21:27:51.948670+0000","last_clean":"2024-03-11T21:27:51.948670+0000","last_became_active":"2024-03-11T21:27:36.694232+0000","last_became_peered":"2024-03-11T21:27:36.694232+0000","last_unstale":"2024-03-11T21:27:51.948670+0000","last_undegraded":"2024-03-11T21:27:51.948670+0000","last_fullsized":"2024-03-11T21:27:51.948670+0000","mapping_epoch":99,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:27:35.667522+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:27:35.667522+0000","last_clean_scrub_stamp":"2024-03-11T21:27:35.667522+0000","objects_scrubbed":0,"log_size":1249,"log_dups_size":0,"ondisk_log_size":1249,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:28:44.939811+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":81854464,"num_objects":1249,"num_object_clones":0,"num_object_copies":3747,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1249,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1249,"num_write_kb":79936,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3],"acting":[5,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":84,"reported_epoch":105,"state":"active+clean","last_fresh":"2024-03-11T21:27:44.733102+0000","last_change":"2024-03-11T21:26:59.108203+0000","last_active":"2024-03-11T21:27:44.733102+0000","last_peered":"2024-03-11T21:27:44.733102+0000","last_clean":"2024-03-11T21:27:44.733102+0000","last_became_active":"2024-03-11T21:25:45.301416+0000","last_became_peered":"2024-03-11T21:25:45.301416+0000","last_unstale":"2024-03-11T21:27:44.733102+0000","last_undegraded":"2024-03-11T21:27:44.733102+0000","last_fullsized":"2024-03-11T21:27:44.733102+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:26:59.108111+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:43.559452+0000","last_clean_scrub_stamp":"2024-03-11T21:26:59.108111+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:28:03.778211+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":95,"reported_epoch":105,"state":"active+clean","last_fresh":"2024-03-11T21:27:44.739297+0000","last_change":"2024-03-11T21:26:57.333768+0000","last_active":"2024-03-11T21:27:44.739297+0000","last_peered":"2024-03-11T21:27:44.739297+0000","last_clean":"2024-03-11T21:27:44.739297+0000","last_became_active":"2024-03-11T21:25:38.545148+0000","last_became_peered":"2024-03-11T21:25:38.545148+0000","last_unstale":"2024-03-11T21:27:44.739297+0000","last_undegraded":"2024-03-11T21:27:44.739297+0000","last_fullsized":"2024-03-11T21:27:44.739297+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:26:57.333669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:26:57.333669+0000","last_clean_scrub_stamp":"2024-03-11T21:26:57.333669+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:28:23.923732+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.3","version":"74'19","reported_seq":128,"reported_epoch":105,"state":"active+clean","last_fresh":"2024-03-11T21:27:44.733686+0000","last_change":"2024-03-11T21:27:14.321667+0000","last_active":"2024-03-11T21:27:44.733686+0000","last_peered":"2024-03-11T21:27:44.733686+0000","last_clean":"2024-03-11T21:27:44.733686+0000","last_became_active":"2024-03-11T21:25:40.760594+0000","last_became_peered":"2024-03-11T21:25:40.760594+0000","last_unstale":"2024-03-11T21:27:44.733686+0000","last_undegraded":"2024-03-11T21:27:44.733686+0000","last_fullsized":"2024-03-11T21:27:44.733686+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'19","last_scrub_stamp":"2024-03-11T21:27:14.321574+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:39.525554+0000","last_clean_scrub_stamp":"2024-03-11T21:27:14.321574+0000","objects_scrubbed":8,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:28:21.292097+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":39,"num_read_kb":25,"num_write":22,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":88,"reported_epoch":105,"state":"active+clean","last_fresh":"2024-03-11T21:27:44.733229+0000","last_change":"2024-03-11T21:27:05.035735+0000","last_active":"2024-03-11T21:27:44.733229+0000","last_peered":"2024-03-11T21:27:44.733229+0000","last_clean":"2024-03-11T21:27:44.733229+0000","last_became_active":"2024-03-11T21:25:42.579045+0000","last_became_peered":"2024-03-11T21:25:42.579045+0000","last_unstale":"2024-03-11T21:27:44.733229+0000","last_undegraded":"2024-03-11T21:27:44.733229+0000","last_fullsized":"2024-03-11T21:27:44.733229+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:27:05.035647+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:41.544550+0000","last_clean_scrub_stamp":"2024-03-11T21:27:05.035647+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:28:32.345929+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.3","version":"105'1293","reported_seq":1314,"reported_epoch":105,"state":"active+clean","last_fresh":"2024-03-11T21:27:51.976285+0000","last_change":"2024-03-11T21:27:36.693897+0000","last_active":"2024-03-11T21:27:51.976285+0000","last_peered":"2024-03-11T21:27:51.976285+0000","last_clean":"2024-03-11T21:27:51.976285+0000","last_became_active":"2024-03-11T21:27:36.693506+0000","last_became_peered":"2024-03-11T21:27:36.693506+0000","last_unstale":"2024-03-11T21:27:51.976285+0000","last_undegraded":"2024-03-11T21:27:51.976285+0000","last_fullsized":"2024-03-11T21:27:51.976285+0000","mapping_epoch":99,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:27:35.667522+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:27:35.667522+0000","last_clean_scrub_stamp":"2024-03-11T21:27:35.667522+0000","objects_scrubbed":0,"log_size":1293,"log_dups_size":0,"ondisk_log_size":1293,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:28:54.029141+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":84738048,"num_objects":1293,"num_object_clones":0,"num_object_copies":3879,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1293,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1293,"num_write_kb":82752,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,4],"acting":[3,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":84,"reported_epoch":105,"state":"active+clean","last_fresh":"2024-03-11T21:27:44.730631+0000","last_change":"2024-03-11T21:26:50.836566+0000","last_active":"2024-03-11T21:27:44.730631+0000","last_peered":"2024-03-11T21:27:44.730631+0000","last_clean":"2024-03-11T21:27:44.730631+0000","last_became_active":"2024-03-11T21:25:44.583638+0000","last_became_peered":"2024-03-11T21:25:44.583638+0000","last_unstale":"2024-03-11T21:27:44.730631+0000","last_undegraded":"2024-03-11T21:27:44.730631+0000","last_fullsized":"2024-03-11T21:27:44.730631+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:26:50.836479+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:43.559452+0000","last_clean_scrub_stamp":"2024-03-11T21:26:50.836479+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:28:03.817969+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.6","version":"67'1","reported_seq":97,"reported_epoch":105,"state":"active+clean","last_fresh":"2024-03-11T21:27:44.737951+0000","last_change":"2024-03-11T21:27:07.129584+0000","last_active":"2024-03-11T21:27:44.737951+0000","last_peered":"2024-03-11T21:27:44.737951+0000","last_clean":"2024-03-11T21:27:44.737951+0000","last_became_active":"2024-03-11T21:25:38.536142+0000","last_became_peered":"2024-03-11T21:25:38.536142+0000","last_unstale":"2024-03-11T21:27:44.737951+0000","last_undegraded":"2024-03-11T21:27:44.737951+0000","last_fullsized":"2024-03-11T21:27:44.737951+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'1","last_scrub_stamp":"2024-03-11T21:27:07.129494+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:37.512018+0000","last_clean_scrub_stamp":"2024-03-11T21:27:07.129494+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:28:12.668804+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,4],"acting":[1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.0","version":"93'26","reported_seq":131,"reported_epoch":105,"state":"active+clean","last_fresh":"2024-03-11T21:27:44.733768+0000","last_change":"2024-03-11T21:26:40.680446+0000","last_active":"2024-03-11T21:27:44.733768+0000","last_peered":"2024-03-11T21:27:44.733768+0000","last_clean":"2024-03-11T21:27:44.733768+0000","last_became_active":"2024-03-11T21:25:40.761716+0000","last_became_peered":"2024-03-11T21:25:40.761716+0000","last_unstale":"2024-03-11T21:27:44.733768+0000","last_undegraded":"2024-03-11T21:27:44.733768+0000","last_fullsized":"2024-03-11T21:27:44.733768+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"93'26","last_scrub_stamp":"2024-03-11T21:26:40.680324+0000","last_deep_scrub":"93'26","last_deep_scrub_stamp":"2024-03-11T21:26:40.680324+0000","last_clean_scrub_stamp":"2024-03-11T21:26:40.680324+0000","objects_scrubbed":8,"log_size":26,"log_dups_size":0,"ondisk_log_size":26,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:28:00.090900+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":34,"num_read_kb":22,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":88,"reported_epoch":105,"state":"active+clean","last_fresh":"2024-03-11T21:27:44.730153+0000","last_change":"2024-03-11T21:26:51.863702+0000","last_active":"2024-03-11T21:27:44.730153+0000","last_peered":"2024-03-11T21:27:44.730153+0000","last_clean":"2024-03-11T21:27:44.730153+0000","last_became_active":"2024-03-11T21:25:42.583647+0000","last_became_peered":"2024-03-11T21:25:42.583647+0000","last_unstale":"2024-03-11T21:27:44.730153+0000","last_undegraded":"2024-03-11T21:27:44.730153+0000","last_fullsized":"2024-03-11T21:27:44.730153+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:26:51.863621+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:26:51.863621+0000","last_clean_scrub_stamp":"2024-03-11T21:26:51.863621+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:27:53.832869+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,7],"acting":[4,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.2","version":"105'1276","reported_seq":1297,"reported_epoch":105,"state":"active+clean","last_fresh":"2024-03-11T21:27:51.969502+0000","last_change":"2024-03-11T21:27:36.692676+0000","last_active":"2024-03-11T21:27:51.969502+0000","last_peered":"2024-03-11T21:27:51.969502+0000","last_clean":"2024-03-11T21:27:51.969502+0000","last_became_active":"2024-03-11T21:27:36.692405+0000","last_became_peered":"2024-03-11T21:27:36.692405+0000","last_unstale":"2024-03-11T21:27:51.969502+0000","last_undegraded":"2024-03-11T21:27:51.969502+0000","last_fullsized":"2024-03-11T21:27:51.969502+0000","mapping_epoch":99,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:27:35.667522+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:27:35.667522+0000","last_clean_scrub_stamp":"2024-03-11T21:27:35.667522+0000","objects_scrubbed":0,"log_size":1276,"log_dups_size":0,"ondisk_log_size":1276,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:28:46.439683+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":83623936,"num_objects":1276,"num_object_clones":0,"num_object_copies":3828,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1276,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1276,"num_write_kb":81664,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.3","version":"0'0","reported_seq":83,"reported_epoch":105,"state":"active+clean","last_fresh":"2024-03-11T21:27:44.738968+0000","last_change":"2024-03-11T21:26:54.421277+0000","last_active":"2024-03-11T21:27:44.738968+0000","last_peered":"2024-03-11T21:27:44.738968+0000","last_clean":"2024-03-11T21:27:44.738968+0000","last_became_active":"2024-03-11T21:25:45.302719+0000","last_became_peered":"2024-03-11T21:25:45.302719+0000","last_unstale":"2024-03-11T21:27:44.738968+0000","last_undegraded":"2024-03-11T21:27:44.738968+0000","last_fullsized":"2024-03-11T21:27:44.738968+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:26:54.421196+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:43.559452+0000","last_clean_scrub_stamp":"2024-03-11T21:26:54.421196+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:28:15.162446+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,2],"acting":[7,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":96,"reported_epoch":105,"state":"active+clean","last_fresh":"2024-03-11T21:27:44.733402+0000","last_change":"2024-03-11T21:26:47.927498+0000","last_active":"2024-03-11T21:27:44.733402+0000","last_peered":"2024-03-11T21:27:44.733402+0000","last_clean":"2024-03-11T21:27:44.733402+0000","last_became_active":"2024-03-11T21:25:38.540013+0000","last_became_peered":"2024-03-11T21:25:38.540013+0000","last_unstale":"2024-03-11T21:27:44.733402+0000","last_undegraded":"2024-03-11T21:27:44.733402+0000","last_fullsized":"2024-03-11T21:27:44.733402+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:26:47.927394+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:37.512018+0000","last_clean_scrub_stamp":"2024-03-11T21:26:47.927394+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:28:02.205612+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,2],"acting":[6,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.1","version":"74'14","reported_seq":113,"reported_epoch":105,"state":"active+clean","last_fresh":"2024-03-11T21:27:44.729748+0000","last_change":"2024-03-11T21:27:22.932743+0000","last_active":"2024-03-11T21:27:44.729748+0000","last_peered":"2024-03-11T21:27:44.729748+0000","last_clean":"2024-03-11T21:27:44.729748+0000","last_became_active":"2024-03-11T21:25:40.546219+0000","last_became_peered":"2024-03-11T21:25:40.546219+0000","last_unstale":"2024-03-11T21:27:44.729748+0000","last_undegraded":"2024-03-11T21:27:44.729748+0000","last_fullsized":"2024-03-11T21:27:44.729748+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'14","last_scrub_stamp":"2024-03-11T21:27:22.932642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:39.525554+0000","last_clean_scrub_stamp":"2024-03-11T21:27:22.932642+0000","objects_scrubbed":7,"log_size":14,"log_dups_size":0,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:28:36.086651+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":21,"num_read_kb":14,"num_write":14,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6],"acting":[4,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.0","version":"98'9","reported_seq":123,"reported_epoch":105,"state":"active+clean","last_fresh":"2024-03-11T21:27:46.974215+0000","last_change":"2024-03-11T21:27:29.745799+0000","last_active":"2024-03-11T21:27:46.974215+0000","last_peered":"2024-03-11T21:27:46.974215+0000","last_clean":"2024-03-11T21:27:46.974215+0000","last_became_active":"2024-03-11T21:25:42.576126+0000","last_became_peered":"2024-03-11T21:25:42.576126+0000","last_unstale":"2024-03-11T21:27:46.974215+0000","last_undegraded":"2024-03-11T21:27:46.974215+0000","last_fullsized":"2024-03-11T21:27:46.974215+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"98'9","last_scrub_stamp":"2024-03-11T21:27:29.745711+0000","last_deep_scrub":"98'9","last_deep_scrub_stamp":"2024-03-11T21:27:29.745711+0000","last_clean_scrub_stamp":"2024-03-11T21:27:29.745711+0000","objects_scrubbed":1,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:28:34.303318+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.d","version":"105'1300","reported_seq":1321,"reported_epoch":105,"state":"active+clean","last_fresh":"2024-03-11T21:27:51.959574+0000","last_change":"2024-03-11T21:27:36.695471+0000","last_active":"2024-03-11T21:27:51.959574+0000","last_peered":"2024-03-11T21:27:51.959574+0000","last_clean":"2024-03-11T21:27:51.959574+0000","last_became_active":"2024-03-11T21:27:36.695186+0000","last_became_peered":"2024-03-11T21:27:36.695186+0000","last_unstale":"2024-03-11T21:27:51.959574+0000","last_undegraded":"2024-03-11T21:27:51.959574+0000","last_fullsized":"2024-03-11T21:27:51.959574+0000","mapping_epoch":99,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:27:35.667522+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:27:35.667522+0000","last_clean_scrub_stamp":"2024-03-11T21:27:35.667522+0000","objects_scrubbed":0,"log_size":1300,"log_dups_size":0,"ondisk_log_size":1300,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:28:55.480832+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":85196800,"num_objects":1300,"num_object_clones":0,"num_object_copies":3900,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1300,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1300,"num_write_kb":83200,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,0],"acting":[6,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.c","version":"0'0","reported_seq":74,"reported_epoch":105,"state":"active+clean","last_fresh":"2024-03-11T21:27:44.737585+0000","last_change":"2024-03-11T21:25:45.303235+0000","last_active":"2024-03-11T21:27:44.737585+0000","last_peered":"2024-03-11T21:27:44.737585+0000","last_clean":"2024-03-11T21:27:44.737585+0000","last_became_active":"2024-03-11T21:25:45.302207+0000","last_became_peered":"2024-03-11T21:25:45.302207+0000","last_unstale":"2024-03-11T21:27:44.737585+0000","last_undegraded":"2024-03-11T21:27:44.737585+0000","last_fullsized":"2024-03-11T21:27:44.737585+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:25:43.559452+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:43.559452+0000","last_clean_scrub_stamp":"2024-03-11T21:25:43.559452+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:27:46.744748+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.f","version":"0'0","reported_seq":86,"reported_epoch":105,"state":"active+clean","last_fresh":"2024-03-11T21:27:44.743678+0000","last_change":"2024-03-11T21:26:51.925489+0000","last_active":"2024-03-11T21:27:44.743678+0000","last_peered":"2024-03-11T21:27:44.743678+0000","last_clean":"2024-03-11T21:27:44.743678+0000","last_became_active":"2024-03-11T21:25:42.575935+0000","last_became_peered":"2024-03-11T21:25:42.575935+0000","last_unstale":"2024-03-11T21:27:44.743678+0000","last_undegraded":"2024-03-11T21:27:44.743678+0000","last_fullsized":"2024-03-11T21:27:44.743678+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:26:51.925329+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:41.544550+0000","last_clean_scrub_stamp":"2024-03-11T21:26:51.925329+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:28:02.478228+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.8","version":"0'0","reported_seq":95,"reported_epoch":105,"state":"active+clean","last_fresh":"2024-03-11T21:27:44.738742+0000","last_change":"2024-03-11T21:26:48.424013+0000","last_active":"2024-03-11T21:27:44.738742+0000","last_peered":"2024-03-11T21:27:44.738742+0000","last_clean":"2024-03-11T21:27:44.738742+0000","last_became_active":"2024-03-11T21:25:38.540910+0000","last_became_peered":"2024-03-11T21:25:38.540910+0000","last_unstale":"2024-03-11T21:27:44.738742+0000","last_undegraded":"2024-03-11T21:27:44.738742+0000","last_fullsized":"2024-03-11T21:27:44.738742+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:26:48.423910+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:37.512018+0000","last_clean_scrub_stamp":"2024-03-11T21:26:48.423910+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:28:17.378160+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,1],"acting":[7,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.e","version":"74'11","reported_seq":110,"reported_epoch":105,"state":"active+clean","last_fresh":"2024-03-11T21:27:44.730399+0000","last_change":"2024-03-11T21:26:57.843770+0000","last_active":"2024-03-11T21:27:44.730399+0000","last_peered":"2024-03-11T21:27:44.730399+0000","last_clean":"2024-03-11T21:27:44.730399+0000","last_became_active":"2024-03-11T21:25:40.552178+0000","last_became_peered":"2024-03-11T21:25:40.552178+0000","last_unstale":"2024-03-11T21:27:44.730399+0000","last_undegraded":"2024-03-11T21:27:44.730399+0000","last_fullsized":"2024-03-11T21:27:44.730399+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'11","last_scrub_stamp":"2024-03-11T21:26:57.843615+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:39.525554+0000","last_clean_scrub_stamp":"2024-03-11T21:26:57.843615+0000","objects_scrubbed":5,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:28:03.562699+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.c","version":"105'1248","reported_seq":1270,"reported_epoch":105,"state":"active+clean","last_fresh":"2024-03-11T21:27:51.953395+0000","last_change":"2024-03-11T21:27:36.693712+0000","last_active":"2024-03-11T21:27:51.953395+0000","last_peered":"2024-03-11T21:27:51.953395+0000","last_clean":"2024-03-11T21:27:51.953395+0000","last_became_active":"2024-03-11T21:27:36.693325+0000","last_became_peered":"2024-03-11T21:27:36.693325+0000","last_unstale":"2024-03-11T21:27:51.953395+0000","last_undegraded":"2024-03-11T21:27:51.953395+0000","last_fullsized":"2024-03-11T21:27:51.953395+0000","mapping_epoch":99,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:27:35.667522+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:27:35.667522+0000","last_clean_scrub_stamp":"2024-03-11T21:27:35.667522+0000","objects_scrubbed":0,"log_size":1248,"log_dups_size":0,"ondisk_log_size":1248,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:28:41.510375+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":81788928,"num_objects":1248,"num_object_clones":0,"num_object_copies":3744,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1248,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1248,"num_write_kb":79872,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,7],"acting":[6,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.d","version":"0'0","reported_seq":82,"reported_epoch":105,"state":"active+clean","last_fresh":"2024-03-11T21:27:44.743423+0000","last_change":"2024-03-11T21:26:54.957670+0000","last_active":"2024-03-11T21:27:44.743423+0000","last_peered":"2024-03-11T21:27:44.743423+0000","last_clean":"2024-03-11T21:27:44.743423+0000","last_became_active":"2024-03-11T21:25:44.590333+0000","last_became_peered":"2024-03-11T21:25:44.590333+0000","last_unstale":"2024-03-11T21:27:44.743423+0000","last_undegraded":"2024-03-11T21:27:44.743423+0000","last_fullsized":"2024-03-11T21:27:44.743423+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:26:54.957577+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:43.559452+0000","last_clean_scrub_stamp":"2024-03-11T21:26:54.957577+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:28:05.595894+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.e","version":"74'8","reported_seq":122,"reported_epoch":105,"state":"active+clean","last_fresh":"2024-03-11T21:27:44.730331+0000","last_change":"2024-03-11T21:27:19.970228+0000","last_active":"2024-03-11T21:27:44.730331+0000","last_peered":"2024-03-11T21:27:44.730331+0000","last_clean":"2024-03-11T21:27:44.730331+0000","last_became_active":"2024-03-11T21:25:42.576742+0000","last_became_peered":"2024-03-11T21:25:42.576742+0000","last_unstale":"2024-03-11T21:27:44.730331+0000","last_undegraded":"2024-03-11T21:27:44.730331+0000","last_fullsized":"2024-03-11T21:27:44.730331+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'8","last_scrub_stamp":"2024-03-11T21:27:19.970141+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:41.544550+0000","last_clean_scrub_stamp":"2024-03-11T21:27:19.970141+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:28:48.877453+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,0],"acting":[4,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.9","version":"0'0","reported_seq":96,"reported_epoch":105,"state":"active+clean","last_fresh":"2024-03-11T21:27:44.732926+0000","last_change":"2024-03-11T21:26:50.991832+0000","last_active":"2024-03-11T21:27:44.732926+0000","last_peered":"2024-03-11T21:27:44.732926+0000","last_clean":"2024-03-11T21:27:44.732926+0000","last_became_active":"2024-03-11T21:25:38.540976+0000","last_became_peered":"2024-03-11T21:25:38.540976+0000","last_unstale":"2024-03-11T21:27:44.732926+0000","last_undegraded":"2024-03-11T21:27:44.732926+0000","last_fullsized":"2024-03-11T21:27:44.732926+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:26:50.991712+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:26:50.991712+0000","last_clean_scrub_stamp":"2024-03-11T21:26:50.991712+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:27:58.101352+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,3],"acting":[1,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.f","version":"74'15","reported_seq":76,"reported_epoch":105,"state":"active+clean","last_fresh":"2024-03-11T21:27:44.746779+0000","last_change":"2024-03-11T21:27:28.081459+0000","last_active":"2024-03-11T21:27:44.746779+0000","last_peered":"2024-03-11T21:27:44.746779+0000","last_clean":"2024-03-11T21:27:44.746779+0000","last_became_active":"2024-03-11T21:26:11.216224+0000","last_became_peered":"2024-03-11T21:26:11.216224+0000","last_unstale":"2024-03-11T21:27:44.746779+0000","last_undegraded":"2024-03-11T21:27:44.746779+0000","last_fullsized":"2024-03-11T21:27:44.746779+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'15","last_scrub_stamp":"2024-03-11T21:27:28.081356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:39.525554+0000","last_clean_scrub_stamp":"2024-03-11T21:27:28.081356+0000","objects_scrubbed":7,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:28:57.158997+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,4],"acting":[2,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.f","version":"105'1308","reported_seq":1330,"reported_epoch":105,"state":"active+clean","last_fresh":"2024-03-11T21:27:51.964322+0000","last_change":"2024-03-11T21:27:36.694341+0000","last_active":"2024-03-11T21:27:51.964322+0000","last_peered":"2024-03-11T21:27:51.964322+0000","last_clean":"2024-03-11T21:27:51.964322+0000","last_became_active":"2024-03-11T21:27:36.694032+0000","last_became_peered":"2024-03-11T21:27:36.694032+0000","last_unstale":"2024-03-11T21:27:51.964322+0000","last_undegraded":"2024-03-11T21:27:51.964322+0000","last_fullsized":"2024-03-11T21:27:51.964322+0000","mapping_epoch":99,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:27:35.667522+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:27:35.667522+0000","last_clean_scrub_stamp":"2024-03-11T21:27:35.667522+0000","objects_scrubbed":0,"log_size":1308,"log_dups_size":0,"ondisk_log_size":1308,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:28:41.584726+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":85721088,"num_objects":1308,"num_object_clones":0,"num_object_copies":3924,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1308,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1308,"num_write_kb":83712,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.e","version":"0'0","reported_seq":83,"reported_epoch":105,"state":"active+clean","last_fresh":"2024-03-11T21:27:44.730163+0000","last_change":"2024-03-11T21:27:31.792816+0000","last_active":"2024-03-11T21:27:44.730163+0000","last_peered":"2024-03-11T21:27:44.730163+0000","last_clean":"2024-03-11T21:27:44.730163+0000","last_became_active":"2024-03-11T21:25:44.590198+0000","last_became_peered":"2024-03-11T21:25:44.590198+0000","last_unstale":"2024-03-11T21:27:44.730163+0000","last_undegraded":"2024-03-11T21:27:44.730163+0000","last_fullsized":"2024-03-11T21:27:44.730163+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:27:31.792707+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:43.559452+0000","last_clean_scrub_stamp":"2024-03-11T21:27:31.792707+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:28:51.775206+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.d","version":"74'8","reported_seq":121,"reported_epoch":105,"state":"active+clean","last_fresh":"2024-03-11T21:27:44.747252+0000","last_change":"2024-03-11T21:26:59.021603+0000","last_active":"2024-03-11T21:27:44.747252+0000","last_peered":"2024-03-11T21:27:44.747252+0000","last_clean":"2024-03-11T21:27:44.747252+0000","last_became_active":"2024-03-11T21:25:42.575583+0000","last_became_peered":"2024-03-11T21:25:42.575583+0000","last_unstale":"2024-03-11T21:27:44.747252+0000","last_undegraded":"2024-03-11T21:27:44.747252+0000","last_fullsized":"2024-03-11T21:27:44.747252+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'8","last_scrub_stamp":"2024-03-11T21:26:59.021523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:41.544550+0000","last_clean_scrub_stamp":"2024-03-11T21:26:59.021523+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:28:21.121669+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,5],"acting":[2,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.a","version":"0'0","reported_seq":96,"reported_epoch":105,"state":"active+clean","last_fresh":"2024-03-11T21:27:44.733086+0000","last_change":"2024-03-11T21:26:52.008094+0000","last_active":"2024-03-11T21:27:44.733086+0000","last_peered":"2024-03-11T21:27:44.733086+0000","last_clean":"2024-03-11T21:27:44.733086+0000","last_became_active":"2024-03-11T21:25:38.541045+0000","last_became_peered":"2024-03-11T21:25:38.541045+0000","last_unstale":"2024-03-11T21:27:44.733086+0000","last_undegraded":"2024-03-11T21:27:44.733086+0000","last_fullsized":"2024-03-11T21:27:44.733086+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:26:52.007996+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:37.512018+0000","last_clean_scrub_stamp":"2024-03-11T21:26:52.007996+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:27:52.862240+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,7],"acting":[1,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.c","version":"74'10","reported_seq":106,"reported_epoch":105,"state":"active+clean","last_fresh":"2024-03-11T21:27:44.730818+0000","last_change":"2024-03-11T21:27:00.807955+0000","last_active":"2024-03-11T21:27:44.730818+0000","last_peered":"2024-03-11T21:27:44.730818+0000","last_clean":"2024-03-11T21:27:44.730818+0000","last_became_active":"2024-03-11T21:25:40.550885+0000","last_became_peered":"2024-03-11T21:25:40.550885+0000","last_unstale":"2024-03-11T21:27:44.730818+0000","last_undegraded":"2024-03-11T21:27:44.730818+0000","last_fullsized":"2024-03-11T21:27:44.730818+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'10","last_scrub_stamp":"2024-03-11T21:27:00.807849+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:39.525554+0000","last_clean_scrub_stamp":"2024-03-11T21:27:00.807849+0000","objects_scrubbed":5,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:28:27.818066+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.e","version":"105'1233","reported_seq":1254,"reported_epoch":105,"state":"active+clean","last_fresh":"2024-03-11T21:27:51.958337+0000","last_change":"2024-03-11T21:27:36.692095+0000","last_active":"2024-03-11T21:27:51.958337+0000","last_peered":"2024-03-11T21:27:51.958337+0000","last_clean":"2024-03-11T21:27:51.958337+0000","last_became_active":"2024-03-11T21:27:36.691800+0000","last_became_peered":"2024-03-11T21:27:36.691800+0000","last_unstale":"2024-03-11T21:27:51.958337+0000","last_undegraded":"2024-03-11T21:27:51.958337+0000","last_fullsized":"2024-03-11T21:27:51.958337+0000","mapping_epoch":99,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:27:35.667522+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:27:35.667522+0000","last_clean_scrub_stamp":"2024-03-11T21:27:35.667522+0000","objects_scrubbed":0,"log_size":1233,"log_dups_size":0,"ondisk_log_size":1233,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:28:54.001153+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":80805888,"num_objects":1233,"num_object_clones":0,"num_object_copies":3699,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1233,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1233,"num_write_kb":78912,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,0],"acting":[2,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.f","version":"0'0","reported_seq":82,"reported_epoch":105,"state":"active+clean","last_fresh":"2024-03-11T21:27:44.746918+0000","last_change":"2024-03-11T21:27:03.011122+0000","last_active":"2024-03-11T21:27:44.746918+0000","last_peered":"2024-03-11T21:27:44.746918+0000","last_clean":"2024-03-11T21:27:44.746918+0000","last_became_active":"2024-03-11T21:25:44.593414+0000","last_became_peered":"2024-03-11T21:25:44.593414+0000","last_unstale":"2024-03-11T21:27:44.746918+0000","last_undegraded":"2024-03-11T21:27:44.746918+0000","last_fullsized":"2024-03-11T21:27:44.746918+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:27:03.010969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:27:03.010969+0000","last_clean_scrub_stamp":"2024-03-11T21:27:03.010969+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:28:22.636838+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,4],"acting":[2,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":87,"reported_epoch":105,"state":"active+clean","last_fresh":"2024-03-11T21:27:44.734035+0000","last_change":"2024-03-11T21:27:22.232224+0000","last_active":"2024-03-11T21:27:44.734035+0000","last_peered":"2024-03-11T21:27:44.734035+0000","last_clean":"2024-03-11T21:27:44.734035+0000","last_became_active":"2024-03-11T21:25:42.579409+0000","last_became_peered":"2024-03-11T21:25:42.579409+0000","last_unstale":"2024-03-11T21:27:44.734035+0000","last_undegraded":"2024-03-11T21:27:44.734035+0000","last_fullsized":"2024-03-11T21:27:44.734035+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:27:22.232114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:41.544550+0000","last_clean_scrub_stamp":"2024-03-11T21:27:22.232114+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:28:27.887527+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.b","version":"0'0","reported_seq":94,"reported_epoch":105,"state":"active+clean","last_fresh":"2024-03-11T21:27:44.738723+0000","last_change":"2024-03-11T21:27:17.475508+0000","last_active":"2024-03-11T21:27:44.738723+0000","last_peered":"2024-03-11T21:27:44.738723+0000","last_clean":"2024-03-11T21:27:44.738723+0000","last_became_active":"2024-03-11T21:25:38.543377+0000","last_became_peered":"2024-03-11T21:25:38.543377+0000","last_unstale":"2024-03-11T21:27:44.738723+0000","last_undegraded":"2024-03-11T21:27:44.738723+0000","last_fullsized":"2024-03-11T21:27:44.738723+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:27:17.475421+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:37.512018+0000","last_clean_scrub_stamp":"2024-03-11T21:27:17.475421+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:28:32.112149+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,5],"acting":[7,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.d","version":"74'17","reported_seq":119,"reported_epoch":105,"state":"active+clean","last_fresh":"2024-03-11T21:27:44.730467+0000","last_change":"2024-03-11T21:27:02.816136+0000","last_active":"2024-03-11T21:27:44.730467+0000","last_peered":"2024-03-11T21:27:44.730467+0000","last_clean":"2024-03-11T21:27:44.730467+0000","last_became_active":"2024-03-11T21:25:40.553454+0000","last_became_peered":"2024-03-11T21:25:40.553454+0000","last_unstale":"2024-03-11T21:27:44.730467+0000","last_undegraded":"2024-03-11T21:27:44.730467+0000","last_fullsized":"2024-03-11T21:27:44.730467+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'17","last_scrub_stamp":"2024-03-11T21:27:02.815980+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:39.525554+0000","last_clean_scrub_stamp":"2024-03-11T21:27:02.815980+0000","objects_scrubbed":8,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:28:06.520165+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":29,"num_read_kb":19,"num_write":18,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,1],"acting":[4,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.9","version":"105'1260","reported_seq":1282,"reported_epoch":105,"state":"active+clean","last_fresh":"2024-03-11T21:27:51.963509+0000","last_change":"2024-03-11T21:27:36.693276+0000","last_active":"2024-03-11T21:27:51.963509+0000","last_peered":"2024-03-11T21:27:51.963509+0000","last_clean":"2024-03-11T21:27:51.963509+0000","last_became_active":"2024-03-11T21:27:36.692978+0000","last_became_peered":"2024-03-11T21:27:36.692978+0000","last_unstale":"2024-03-11T21:27:51.963509+0000","last_undegraded":"2024-03-11T21:27:51.963509+0000","last_fullsized":"2024-03-11T21:27:51.963509+0000","mapping_epoch":99,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:27:35.667522+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:27:35.667522+0000","last_clean_scrub_stamp":"2024-03-11T21:27:35.667522+0000","objects_scrubbed":0,"log_size":1260,"log_dups_size":0,"ondisk_log_size":1260,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:29:00.262706+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":82575360,"num_objects":1260,"num_object_clones":0,"num_object_copies":3780,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1260,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1260,"num_write_kb":80640,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":82,"reported_epoch":105,"state":"active+clean","last_fresh":"2024-03-11T21:27:44.738389+0000","last_change":"2024-03-11T21:27:07.304361+0000","last_active":"2024-03-11T21:27:44.738389+0000","last_peered":"2024-03-11T21:27:44.738389+0000","last_clean":"2024-03-11T21:27:44.738389+0000","last_became_active":"2024-03-11T21:25:44.589973+0000","last_became_peered":"2024-03-11T21:25:44.589973+0000","last_unstale":"2024-03-11T21:27:44.738389+0000","last_undegraded":"2024-03-11T21:27:44.738389+0000","last_fullsized":"2024-03-11T21:27:44.738389+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:27:07.304264+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:43.559452+0000","last_clean_scrub_stamp":"2024-03-11T21:27:07.304264+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:28:32.511531+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,3],"acting":[7,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":86,"reported_epoch":105,"state":"active+clean","last_fresh":"2024-03-11T21:27:44.746189+0000","last_change":"2024-03-11T21:27:29.075249+0000","last_active":"2024-03-11T21:27:44.746189+0000","last_peered":"2024-03-11T21:27:44.746189+0000","last_clean":"2024-03-11T21:27:44.746189+0000","last_became_active":"2024-03-11T21:25:42.574515+0000","last_became_peered":"2024-03-11T21:25:42.574515+0000","last_unstale":"2024-03-11T21:27:44.746189+0000","last_undegraded":"2024-03-11T21:27:44.746189+0000","last_fullsized":"2024-03-11T21:27:44.746189+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:27:29.075166+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:41.544550+0000","last_clean_scrub_stamp":"2024-03-11T21:27:29.075166+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:28:52.194040+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,5],"acting":[2,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.c","version":"0'0","reported_seq":95,"reported_epoch":105,"state":"active+clean","last_fresh":"2024-03-11T21:27:44.746602+0000","last_change":"2024-03-11T21:27:24.105818+0000","last_active":"2024-03-11T21:27:44.746602+0000","last_peered":"2024-03-11T21:27:44.746602+0000","last_clean":"2024-03-11T21:27:44.746602+0000","last_became_active":"2024-03-11T21:25:38.533240+0000","last_became_peered":"2024-03-11T21:25:38.533240+0000","last_unstale":"2024-03-11T21:27:44.746602+0000","last_undegraded":"2024-03-11T21:27:44.746602+0000","last_fullsized":"2024-03-11T21:27:44.746602+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:27:24.105731+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:37.512018+0000","last_clean_scrub_stamp":"2024-03-11T21:27:24.105731+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:28:44.133006+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,0],"acting":[2,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.a","version":"74'19","reported_seq":123,"reported_epoch":105,"state":"active+clean","last_fresh":"2024-03-11T21:27:44.732925+0000","last_change":"2024-03-11T21:27:07.010787+0000","last_active":"2024-03-11T21:27:44.732925+0000","last_peered":"2024-03-11T21:27:44.732925+0000","last_clean":"2024-03-11T21:27:44.732925+0000","last_became_active":"2024-03-11T21:25:40.760698+0000","last_became_peered":"2024-03-11T21:25:40.760698+0000","last_unstale":"2024-03-11T21:27:44.732925+0000","last_undegraded":"2024-03-11T21:27:44.732925+0000","last_fullsized":"2024-03-11T21:27:44.732925+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'19","last_scrub_stamp":"2024-03-11T21:27:07.010662+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:39.525554+0000","last_clean_scrub_stamp":"2024-03-11T21:27:07.010662+0000","objects_scrubbed":9,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:28:22.036849+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":32,"num_read_kb":21,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,7],"acting":[6,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.8","version":"105'1250","reported_seq":1271,"reported_epoch":105,"state":"active+clean","last_fresh":"2024-03-11T21:27:51.962654+0000","last_change":"2024-03-11T21:27:36.693400+0000","last_active":"2024-03-11T21:27:51.962654+0000","last_peered":"2024-03-11T21:27:51.962654+0000","last_clean":"2024-03-11T21:27:51.962654+0000","last_became_active":"2024-03-11T21:27:36.693060+0000","last_became_peered":"2024-03-11T21:27:36.693060+0000","last_unstale":"2024-03-11T21:27:51.962654+0000","last_undegraded":"2024-03-11T21:27:51.962654+0000","last_fullsized":"2024-03-11T21:27:51.962654+0000","mapping_epoch":99,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:27:35.667522+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:27:35.667522+0000","last_clean_scrub_stamp":"2024-03-11T21:27:35.667522+0000","objects_scrubbed":0,"log_size":1250,"log_dups_size":0,"ondisk_log_size":1250,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:28:37.218739+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":81920000,"num_objects":1250,"num_object_clones":0,"num_object_copies":3750,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1250,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1250,"num_write_kb":80000,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,4],"acting":[2,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":83,"reported_epoch":105,"state":"active+clean","last_fresh":"2024-03-11T21:27:44.735585+0000","last_change":"2024-03-11T21:27:11.362017+0000","last_active":"2024-03-11T21:27:44.735585+0000","last_peered":"2024-03-11T21:27:44.735585+0000","last_clean":"2024-03-11T21:27:44.735585+0000","last_became_active":"2024-03-11T21:25:44.584154+0000","last_became_peered":"2024-03-11T21:25:44.584154+0000","last_unstale":"2024-03-11T21:27:44.735585+0000","last_undegraded":"2024-03-11T21:27:44.735585+0000","last_fullsized":"2024-03-11T21:27:44.735585+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:27:11.361913+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:27:11.361913+0000","last_clean_scrub_stamp":"2024-03-11T21:27:11.361913+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:28:25.991645+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":86,"reported_epoch":105,"state":"active+clean","last_fresh":"2024-03-11T21:27:44.746883+0000","last_change":"2024-03-11T21:27:10.909699+0000","last_active":"2024-03-11T21:27:44.746883+0000","last_peered":"2024-03-11T21:27:44.746883+0000","last_clean":"2024-03-11T21:27:44.746883+0000","last_became_active":"2024-03-11T21:25:42.576679+0000","last_became_peered":"2024-03-11T21:25:42.576679+0000","last_unstale":"2024-03-11T21:27:44.746883+0000","last_undegraded":"2024-03-11T21:27:44.746883+0000","last_fullsized":"2024-03-11T21:27:44.746883+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:27:10.909543+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:41.544550+0000","last_clean_scrub_stamp":"2024-03-11T21:27:10.909543+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:28:29.725395+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,3],"acting":[2,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.d","version":"0'0","reported_seq":96,"reported_epoch":105,"state":"active+clean","last_fresh":"2024-03-11T21:27:44.738110+0000","last_change":"2024-03-11T21:26:45.885219+0000","last_active":"2024-03-11T21:27:44.738110+0000","last_peered":"2024-03-11T21:27:44.738110+0000","last_clean":"2024-03-11T21:27:44.738110+0000","last_became_active":"2024-03-11T21:25:38.539681+0000","last_became_peered":"2024-03-11T21:25:38.539681+0000","last_unstale":"2024-03-11T21:27:44.738110+0000","last_undegraded":"2024-03-11T21:27:44.738110+0000","last_fullsized":"2024-03-11T21:27:44.738110+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:26:45.885144+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:37.512018+0000","last_clean_scrub_stamp":"2024-03-11T21:26:45.885144+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,3],"acting":[1,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.b","version":"74'9","reported_seq":107,"reported_epoch":105,"state":"active+clean","last_fresh":"2024-03-11T21:27:44.733845+0000","last_change":"2024-03-11T21:27:24.392581+0000","last_active":"2024-03-11T21:27:44.733845+0000","last_peered":"2024-03-11T21:27:44.733845+0000","last_clean":"2024-03-11T21:27:44.733845+0000","last_became_active":"2024-03-11T21:25:40.549612+0000","last_became_peered":"2024-03-11T21:25:40.549612+0000","last_unstale":"2024-03-11T21:27:44.733845+0000","last_undegraded":"2024-03-11T21:27:44.733845+0000","last_fullsized":"2024-03-11T21:27:44.733845+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'9","last_scrub_stamp":"2024-03-11T21:27:24.392483+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:39.525554+0000","last_clean_scrub_stamp":"2024-03-11T21:27:24.392483+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:28:37.805493+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.b","version":"105'1255","reported_seq":1277,"reported_epoch":105,"state":"active+clean","last_fresh":"2024-03-11T21:27:51.940747+0000","last_change":"2024-03-11T21:27:36.691639+0000","last_active":"2024-03-11T21:27:51.940747+0000","last_peered":"2024-03-11T21:27:51.940747+0000","last_clean":"2024-03-11T21:27:51.940747+0000","last_became_active":"2024-03-11T21:27:36.691322+0000","last_became_peered":"2024-03-11T21:27:36.691322+0000","last_unstale":"2024-03-11T21:27:51.940747+0000","last_undegraded":"2024-03-11T21:27:51.940747+0000","last_fullsized":"2024-03-11T21:27:51.940747+0000","mapping_epoch":99,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:27:35.667522+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:27:35.667522+0000","last_clean_scrub_stamp":"2024-03-11T21:27:35.667522+0000","objects_scrubbed":0,"log_size":1255,"log_dups_size":0,"ondisk_log_size":1255,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:29:04.800550+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":82247680,"num_objects":1255,"num_object_clones":0,"num_object_copies":3765,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1255,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1255,"num_write_kb":80320,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,6],"acting":[1,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":82,"reported_epoch":105,"state":"active+clean","last_fresh":"2024-03-11T21:27:44.743007+0000","last_change":"2024-03-11T21:26:55.966077+0000","last_active":"2024-03-11T21:27:44.743007+0000","last_peered":"2024-03-11T21:27:44.743007+0000","last_clean":"2024-03-11T21:27:44.743007+0000","last_became_active":"2024-03-11T21:25:45.300333+0000","last_became_peered":"2024-03-11T21:25:45.300333+0000","last_unstale":"2024-03-11T21:27:44.743007+0000","last_undegraded":"2024-03-11T21:27:44.743007+0000","last_fullsized":"2024-03-11T21:27:44.743007+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:26:55.965980+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:43.559452+0000","last_clean_scrub_stamp":"2024-03-11T21:26:55.965980+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:28:02.706121+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,0],"acting":[5,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.e","version":"0'0","reported_seq":94,"reported_epoch":105,"state":"active+clean","last_fresh":"2024-03-11T21:27:44.747481+0000","last_change":"2024-03-11T21:26:57.974916+0000","last_active":"2024-03-11T21:27:44.747481+0000","last_peered":"2024-03-11T21:27:44.747481+0000","last_clean":"2024-03-11T21:27:44.747481+0000","last_became_active":"2024-03-11T21:25:38.539954+0000","last_became_peered":"2024-03-11T21:25:38.539954+0000","last_unstale":"2024-03-11T21:27:44.747481+0000","last_undegraded":"2024-03-11T21:27:44.747481+0000","last_fullsized":"2024-03-11T21:27:44.747481+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:26:57.974827+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:37.512018+0000","last_clean_scrub_stamp":"2024-03-11T21:26:57.974827+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:28:10.300007+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,7],"acting":[2,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.8","version":"74'15","reported_seq":116,"reported_epoch":105,"state":"active+clean","last_fresh":"2024-03-11T21:27:44.743087+0000","last_change":"2024-03-11T21:26:50.934584+0000","last_active":"2024-03-11T21:27:44.743087+0000","last_peered":"2024-03-11T21:27:44.743087+0000","last_clean":"2024-03-11T21:27:44.743087+0000","last_became_active":"2024-03-11T21:25:40.761938+0000","last_became_peered":"2024-03-11T21:25:40.761938+0000","last_unstale":"2024-03-11T21:27:44.743087+0000","last_undegraded":"2024-03-11T21:27:44.743087+0000","last_fullsized":"2024-03-11T21:27:44.743087+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'15","last_scrub_stamp":"2024-03-11T21:26:50.934489+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:39.525554+0000","last_clean_scrub_stamp":"2024-03-11T21:26:50.934489+0000","objects_scrubbed":7,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:28:08.719408+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.9","version":"94'9","reported_seq":125,"reported_epoch":105,"state":"active+clean","last_fresh":"2024-03-11T21:27:44.738711+0000","last_change":"2024-03-11T21:26:53.422673+0000","last_active":"2024-03-11T21:27:44.738711+0000","last_peered":"2024-03-11T21:27:44.738711+0000","last_clean":"2024-03-11T21:27:44.738711+0000","last_became_active":"2024-03-11T21:25:42.584461+0000","last_became_peered":"2024-03-11T21:25:42.584461+0000","last_unstale":"2024-03-11T21:27:44.738711+0000","last_undegraded":"2024-03-11T21:27:44.738711+0000","last_fullsized":"2024-03-11T21:27:44.738711+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"94'9","last_scrub_stamp":"2024-03-11T21:26:53.422552+0000","last_deep_scrub":"94'9","last_deep_scrub_stamp":"2024-03-11T21:26:53.422552+0000","last_clean_scrub_stamp":"2024-03-11T21:26:53.422552+0000","objects_scrubbed":1,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:28:07.835997+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":7,"num_pg":16,"stat_sum":{"num_bytes":1329332224,"num_objects":20284,"num_object_clones":0,"num_object_copies":60852,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":20284,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20284,"num_write_kb":1298176,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":3989635072,"data_stored":3989635072,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":20284,"ondisk_log_size":20284,"up":48,"acting":48,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":1531,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":24,"num_read_kb":24,"num_write":10,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":4593,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":6,"ondisk_log_size":6,"up":96,"acting":96,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":212,"num_read_kb":306,"num_write":179,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1771104,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1781760,"data_stored":1771104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":128,"ondisk_log_size":128,"up":3,"acting":3,"num_store_stats":3},{"poolid":3,"num_pg":3,"stat_sum":{"num_bytes":408,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":128,"num_read_kb":123,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1224,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":9,"acting":9,"num_store_stats":7},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":3702,"num_objects":178,"num_object_clones":0,"num_object_copies":534,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":178,"num_whiteouts":0,"num_read":698,"num_read_kb":455,"num_write":417,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":417792,"data_stored":11106,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":420,"ondisk_log_size":420,"up":96,"acting":96,"num_store_stats":8},{"poolid":5,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":67,"ondisk_log_size":67,"up":96,"acting":96,"num_store_stats":8},{"poolid":6,"num_pg":32,"stat_sum":{"num_bytes":383,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1149,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2,"ondisk_log_size":2,"up":96,"acting":96,"num_store_stats":8}],"osd_stats":[{"osd":7,"up_from":60,"seq":257698037790,"num_pgs":61,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":424976,"kb_used_data":389640,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93320176,"statfs":{"total":95995035648,"available":95559860224,"internally_reserved":0,"allocated":398991360,"data_stored":398610637,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":36174282},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,1],"upper_bound":8},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":50,"seq":214748364836,"num_pgs":49,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":399756,"kb_used_data":364420,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93345396,"statfs":{"total":95995035648,"available":95585685504,"internally_reserved":0,"allocated":373166080,"data_stored":372789746,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":36174278},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[3,2,0,1],"upper_bound":16},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574910,"num_pgs":59,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":487400,"kb_used_data":452064,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93257752,"statfs":{"total":95995035648,"available":95495938048,"internally_reserved":0,"allocated":462913536,"data_stored":462508094,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738435,"num_pgs":55,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":651604,"kb_used_data":612108,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":93093548,"statfs":{"total":95995035648,"available":95327793152,"internally_reserved":0,"allocated":626798592,"data_stored":626413990,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":40434117},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[4,0,2],"upper_bound":8},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411384,"num_pgs":45,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":482368,"kb_used_data":447032,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93262784,"statfs":{"total":95995035648,"available":95501090816,"internally_reserved":0,"allocated":457760768,"data_stored":457395510,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,2,0,1],"upper_bound":16},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":26,"seq":111669149748,"num_pgs":64,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":730600,"kb_used_data":703584,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93014552,"statfs":{"total":95995035648,"available":95246901248,"internally_reserved":0,"allocated":720470016,"data_stored":720064908,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":4,"up_from":32,"seq":137438953517,"num_pgs":58,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":437912,"kb_used_data":402576,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93307240,"statfs":{"total":95995035648,"available":95546613760,"internally_reserved":0,"allocated":412237824,"data_stored":411848274,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":41,"seq":176093659176,"num_pgs":53,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":352516,"kb_used_data":317180,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93392636,"statfs":{"total":95995035648,"available":95634059264,"internally_reserved":0,"allocated":324792320,"data_stored":324423017,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1049,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":46,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":957,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":574,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":436,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1003,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":528,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":1320,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":81920,"data_stored":2118,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":1018,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":1898,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":1474,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":990,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":1034,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":1254,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":370,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":383,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":370,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":672399360,"data_stored":672399360,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":502595584,"data_stored":502595584,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":492765184,"data_stored":492765184,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":749010944,"data_stored":749010944,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":416874496,"data_stored":416874496,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":324927488,"data_stored":324927488,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":415301632,"data_stored":415301632,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":415760384,"data_stored":415760384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-11T21:27:57.918 INFO:tasks.cephadm.ceph_manager.ceph:would drop below min_pgs, proj -9, currently 1 2024-03-11T21:27:57.947 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:27:57.994 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:27:57.995 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:27:57.995 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:27:58.077 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:27:57 smithi097 ceph-mon[33317]: from='client.24751 v1:172.21.15.92:0/3347941960' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-11T21:27:58.083 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:27:57 smithi092 ceph-mon[29572]: from='client.24751 v1:172.21.15.92:0/3347941960' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-11T21:27:58.084 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:27:57 smithi092 ceph-mon[37563]: from='client.24751 v1:172.21.15.92:0/3347941960' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-11T21:27:58.084 INFO:journalctl@ceph.mgr.y.smithi092.stdout:Mar 11 21:27:57 smithi092 ceph-00db0016-dfed-11ee-95c8-87774f69a715-mgr-y[29801]: ::ffff:172.21.15.97 - - [11/Mar/2024:21:27:57] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-11T21:27:58.096 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:27:58.138 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:27:58.139 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:27:58.139 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:27:58.207 INFO:teuthology.orchestra.run.smithi092.stdout:{ 2024-03-11T21:27:58.207 INFO:teuthology.orchestra.run.smithi092.stdout: "size": 20, 2024-03-11T21:27:58.207 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 600, 2024-03-11T21:27:58.208 INFO:teuthology.orchestra.run.smithi092.stdout: "ops": [ 2024-03-11T21:27:58.208 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:27:58.208 INFO:teuthology.orchestra.run.smithi092.stdout: "description": "osd_op(client.34715.0:15943 7.6 7:67c3153e:::benchmark_data_smithi092_76040_object15942:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e105)", 2024-03-11T21:27:58.208 INFO:teuthology.orchestra.run.smithi092.stdout: "initiated_at": "2024-03-11T21:27:50.069446+0000", 2024-03-11T21:27:58.208 INFO:teuthology.orchestra.run.smithi092.stdout: "age": 8.1510500209999996, 2024-03-11T21:27:58.208 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.075989397, 2024-03-11T21:27:58.208 INFO:teuthology.orchestra.run.smithi092.stdout: "type_data": { 2024-03-11T21:27:58.208 INFO:teuthology.orchestra.run.smithi092.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-11T21:27:58.208 INFO:teuthology.orchestra.run.smithi092.stdout: "client_info": { 2024-03-11T21:27:58.208 INFO:teuthology.orchestra.run.smithi092.stdout: "client": "client.34715", 2024-03-11T21:27:58.208 INFO:teuthology.orchestra.run.smithi092.stdout: "client_addr": "v1:172.21.15.92:0/2403132573", 2024-03-11T21:27:58.208 INFO:teuthology.orchestra.run.smithi092.stdout: "tid": 15943 2024-03-11T21:27:58.208 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:27:58.208 INFO:teuthology.orchestra.run.smithi092.stdout: "events": [ 2024-03-11T21:27:58.209 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:27:58.209 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "initiated", 2024-03-11T21:27:58.209 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:27:50.069446+0000", 2024-03-11T21:27:58.209 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-03-11T21:27:58.209 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:27:58.209 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:27:58.209 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "header_read", 2024-03-11T21:27:58.209 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:27:50.069446+0000", 2024-03-11T21:27:58.209 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-03-11T21:27:58.209 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:27:58.209 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:27:58.209 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "throttled", 2024-03-11T21:27:58.209 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:27:50.069448+0000", 2024-03-11T21:27:58.210 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.5740000000000001e-06 2024-03-11T21:27:58.210 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:27:58.210 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:27:58.210 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "all_read", 2024-03-11T21:27:58.210 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:27:50.069470+0000", 2024-03-11T21:27:58.210 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 2.2325000000000001e-05 2024-03-11T21:27:58.210 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:27:58.210 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:27:58.210 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "dispatched", 2024-03-11T21:27:58.210 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:27:50.069477+0000", 2024-03-11T21:27:58.210 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 7.3699999999999997e-06 2024-03-11T21:27:58.210 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:27:58.210 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:27:58.210 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "queued_for_pg", 2024-03-11T21:27:58.211 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:27:50.069483+0000", 2024-03-11T21:27:58.211 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 6.0399999999999998e-06 2024-03-11T21:27:58.211 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:27:58.211 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:27:58.211 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-03-11T21:27:58.211 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:27:50.069959+0000", 2024-03-11T21:27:58.211 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00047594 2024-03-11T21:27:58.211 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:27:58.211 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:27:58.211 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "started", 2024-03-11T21:27:58.211 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:27:50.070495+0000", 2024-03-11T21:27:58.211 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00053564800000000001 2024-03-11T21:27:58.211 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:27:58.212 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:27:58.212 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "waiting for subops from 3,7", 2024-03-11T21:27:58.212 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:27:50.070614+0000", 2024-03-11T21:27:58.212 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00011900000000000001 2024-03-11T21:27:58.212 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:27:58.212 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:27:58.212 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "sub_op_commit_rec", 2024-03-11T21:27:58.212 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:27:50.144312+0000", 2024-03-11T21:27:58.212 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.073697713999999998 2024-03-11T21:27:58.212 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:27:58.212 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:27:58.212 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "sub_op_commit_rec", 2024-03-11T21:27:58.212 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:27:50.145278+0000", 2024-03-11T21:27:58.212 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00096628299999999996 2024-03-11T21:27:58.213 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:27:58.213 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:27:58.213 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "op_commit", 2024-03-11T21:27:58.213 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:27:50.145339+0000", 2024-03-11T21:27:58.213 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 6.143e-05 2024-03-11T21:27:58.213 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:27:58.213 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:27:58.213 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "commit_sent", 2024-03-11T21:27:58.213 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:27:50.145387+0000", 2024-03-11T21:27:58.213 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 4.7382000000000001e-05 2024-03-11T21:27:58.213 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:27:58.213 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:27:58.213 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "done", 2024-03-11T21:27:58.214 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:27:50.145435+0000", 2024-03-11T21:27:58.214 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 4.8690999999999998e-05 2024-03-11T21:27:58.214 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-03-11T21:27:58.214 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-03-11T21:27:58.214 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-03-11T21:27:58.214 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:27:58.214 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:27:58.214 INFO:teuthology.orchestra.run.smithi092.stdout: "description": "osd_op(client.34715.0:15945 7.6 7:63e77498:::benchmark_data_smithi092_76040_object15944:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e105)", 2024-03-11T21:27:58.214 INFO:teuthology.orchestra.run.smithi092.stdout: "initiated_at": "2024-03-11T21:27:50.069494+0000", 2024-03-11T21:27:58.214 INFO:teuthology.orchestra.run.smithi092.stdout: "age": 8.1510024320000003, 2024-03-11T21:27:58.214 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.076589351, 2024-03-11T21:27:58.214 INFO:teuthology.orchestra.run.smithi092.stdout: "type_data": { 2024-03-11T21:27:58.214 INFO:teuthology.orchestra.run.smithi092.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-11T21:27:58.214 INFO:teuthology.orchestra.run.smithi092.stdout: "client_info": { 2024-03-11T21:27:58.215 INFO:teuthology.orchestra.run.smithi092.stdout: "client": "client.34715", 2024-03-11T21:27:58.215 INFO:teuthology.orchestra.run.smithi092.stdout: "client_addr": "v1:172.21.15.92:0/2403132573", 2024-03-11T21:27:58.215 INFO:teuthology.orchestra.run.smithi092.stdout: "tid": 15945 2024-03-11T21:27:58.215 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:27:58.215 INFO:teuthology.orchestra.run.smithi092.stdout: "events": [ 2024-03-11T21:27:58.215 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:27:58.215 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "initiated", 2024-03-11T21:27:58.215 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:27:50.069494+0000", 2024-03-11T21:27:58.215 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-03-11T21:27:58.215 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:27:58.215 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:27:58.215 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "header_read", 2024-03-11T21:27:58.215 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:27:50.069494+0000", 2024-03-11T21:27:58.215 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-03-11T21:27:58.216 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:27:58.216 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:27:58.216 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "throttled", 2024-03-11T21:27:58.216 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:27:50.069494+0000", 2024-03-11T21:27:58.216 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 7.3900000000000004e-07 2024-03-11T21:27:58.216 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:27:58.216 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:27:58.216 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "all_read", 2024-03-11T21:27:58.216 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:27:50.069514+0000", 2024-03-11T21:27:58.216 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.9681e-05 2024-03-11T21:27:58.216 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:27:58.216 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:27:58.216 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "dispatched", 2024-03-11T21:27:58.216 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:27:50.069520+0000", 2024-03-11T21:27:58.217 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 5.8499999999999999e-06 2024-03-11T21:27:58.217 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:27:58.217 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:27:58.217 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "queued_for_pg", 2024-03-11T21:27:58.217 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:27:50.069526+0000", 2024-03-11T21:27:58.217 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 5.5770000000000001e-06 2024-03-11T21:27:58.217 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:27:58.217 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:27:58.217 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-03-11T21:27:58.217 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:27:50.143141+0000", 2024-03-11T21:27:58.217 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.073615626000000003 2024-03-11T21:27:58.217 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:27:58.217 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:27:58.217 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "started", 2024-03-11T21:27:58.217 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:27:50.143304+0000", 2024-03-11T21:27:58.218 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00016294399999999999 2024-03-11T21:27:58.218 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:27:58.218 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:27:58.218 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "waiting for subops from 3,7", 2024-03-11T21:27:58.218 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:27:50.143489+0000", 2024-03-11T21:27:58.218 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.000184756 2024-03-11T21:27:58.218 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:27:58.218 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:27:58.218 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "sub_op_commit_rec", 2024-03-11T21:27:58.218 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:27:50.145317+0000", 2024-03-11T21:27:58.218 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.001827803 2024-03-11T21:27:58.218 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:27:58.218 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:27:58.218 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "op_commit", 2024-03-11T21:27:58.219 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:27:50.145444+0000", 2024-03-11T21:27:58.219 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00012734500000000001 2024-03-11T21:27:58.219 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:27:58.219 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:27:58.219 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "sub_op_commit_rec", 2024-03-11T21:27:58.219 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:27:50.145983+0000", 2024-03-11T21:27:58.219 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00053867200000000002 2024-03-11T21:27:58.219 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:27:58.219 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:27:58.219 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "commit_sent", 2024-03-11T21:27:58.219 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:27:50.146048+0000", 2024-03-11T21:27:58.219 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 6.5625999999999998e-05 2024-03-11T21:27:58.219 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:27:58.219 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:27:58.220 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "done", 2024-03-11T21:27:58.220 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:27:50.146083+0000", 2024-03-11T21:27:58.220 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 3.4731999999999999e-05 2024-03-11T21:27:58.220 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-03-11T21:27:58.220 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-03-11T21:27:58.220 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-03-11T21:27:58.220 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:27:58.220 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:27:58.220 INFO:teuthology.orchestra.run.smithi092.stdout: "description": "osd_op(client.34715.0:15946 7.f 7:f3ca32ac:::benchmark_data_smithi092_76040_object15945:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e105)", 2024-03-11T21:27:58.220 INFO:teuthology.orchestra.run.smithi092.stdout: "initiated_at": "2024-03-11T21:27:50.069814+0000", 2024-03-11T21:27:58.220 INFO:teuthology.orchestra.run.smithi092.stdout: "age": 8.1506824770000001, 2024-03-11T21:27:58.220 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.075325880999999997, 2024-03-11T21:27:58.220 INFO:teuthology.orchestra.run.smithi092.stdout: "type_data": { 2024-03-11T21:27:58.220 INFO:teuthology.orchestra.run.smithi092.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-11T21:27:58.220 INFO:teuthology.orchestra.run.smithi092.stdout: "client_info": { 2024-03-11T21:27:58.221 INFO:teuthology.orchestra.run.smithi092.stdout: "client": "client.34715", 2024-03-11T21:27:58.221 INFO:teuthology.orchestra.run.smithi092.stdout: "client_addr": "v1:172.21.15.92:0/2403132573", 2024-03-11T21:27:58.221 INFO:teuthology.orchestra.run.smithi092.stdout: "tid": 15946 2024-03-11T21:27:58.221 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:27:58.221 INFO:teuthology.orchestra.run.smithi092.stdout: "events": [ 2024-03-11T21:27:58.221 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:27:58.221 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "initiated", 2024-03-11T21:27:58.221 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:27:50.069814+0000", 2024-03-11T21:27:58.221 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-03-11T21:27:58.221 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:27:58.221 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:27:58.221 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "header_read", 2024-03-11T21:27:58.221 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:27:50.069814+0000", 2024-03-11T21:27:58.221 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-03-11T21:27:58.222 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:27:58.222 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:27:58.222 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "throttled", 2024-03-11T21:27:58.222 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:27:50.069815+0000", 2024-03-11T21:27:58.222 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.127e-06 2024-03-11T21:27:58.222 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:27:58.222 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:27:58.222 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "all_read", 2024-03-11T21:27:58.222 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:27:50.069843+0000", 2024-03-11T21:27:58.222 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 2.8167e-05 2024-03-11T21:27:58.222 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:27:58.222 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:27:58.222 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "dispatched", 2024-03-11T21:27:58.223 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:27:50.069851+0000", 2024-03-11T21:27:58.223 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 8.2830000000000001e-06 2024-03-11T21:27:58.223 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:27:58.223 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:27:58.223 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "queued_for_pg", 2024-03-11T21:27:58.223 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:27:50.069858+0000", 2024-03-11T21:27:58.223 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 7.0910000000000004e-06 2024-03-11T21:27:58.223 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:27:58.223 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:27:58.223 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-03-11T21:27:58.223 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:27:50.069961+0000", 2024-03-11T21:27:58.223 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00010289999999999999 2024-03-11T21:27:58.223 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:27:58.223 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:27:58.223 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "started", 2024-03-11T21:27:58.224 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:27:50.070092+0000", 2024-03-11T21:27:58.224 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.000131173 2024-03-11T21:27:58.224 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:27:58.224 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:27:58.224 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "waiting for subops from 3,7", 2024-03-11T21:27:58.224 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:27:50.070258+0000", 2024-03-11T21:27:58.224 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.000165565 2024-03-11T21:27:58.224 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:27:58.224 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:27:58.224 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "op_commit", 2024-03-11T21:27:58.224 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:27:50.142765+0000", 2024-03-11T21:27:58.224 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.072507524000000004 2024-03-11T21:27:58.224 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:27:58.224 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:27:58.225 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "sub_op_commit_rec", 2024-03-11T21:27:58.225 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:27:50.144995+0000", 2024-03-11T21:27:58.225 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.0022298029999999998 2024-03-11T21:27:58.225 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:27:58.225 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:27:58.225 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "sub_op_commit_rec", 2024-03-11T21:27:58.225 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:27:50.145044+0000", 2024-03-11T21:27:58.225 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 4.8976e-05 2024-03-11T21:27:58.225 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:27:58.225 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:27:58.225 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "commit_sent", 2024-03-11T21:27:58.225 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:27:50.145097+0000", 2024-03-11T21:27:58.225 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 5.2808e-05 2024-03-11T21:27:58.225 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:27:58.226 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:27:58.226 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "done", 2024-03-11T21:27:58.226 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:27:50.145140+0000", 2024-03-11T21:27:58.226 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 4.2463999999999999e-05 2024-03-11T21:27:58.226 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-03-11T21:27:58.226 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-03-11T21:27:58.226 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-03-11T21:27:58.226 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:27:58.226 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:27:58.226 INFO:teuthology.orchestra.run.smithi092.stdout: "description": "osd_op(client.34715.0:15947 7.9 7:9a3d547c:::benchmark_data_smithi092_76040_object15946:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e105)", 2024-03-11T21:27:58.226 INFO:teuthology.orchestra.run.smithi092.stdout: "initiated_at": "2024-03-11T21:27:50.069877+0000", 2024-03-11T21:27:58.226 INFO:teuthology.orchestra.run.smithi092.stdout: "age": 8.1506193820000004, 2024-03-11T21:27:58.226 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.073152764999999995, 2024-03-11T21:27:58.226 INFO:teuthology.orchestra.run.smithi092.stdout: "type_data": { 2024-03-11T21:27:58.227 INFO:teuthology.orchestra.run.smithi092.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-11T21:27:58.227 INFO:teuthology.orchestra.run.smithi092.stdout: "client_info": { 2024-03-11T21:27:58.227 INFO:teuthology.orchestra.run.smithi092.stdout: "client": "client.34715", 2024-03-11T21:27:58.227 INFO:teuthology.orchestra.run.smithi092.stdout: "client_addr": "v1:172.21.15.92:0/2403132573", 2024-03-11T21:27:58.227 INFO:teuthology.orchestra.run.smithi092.stdout: "tid": 15947 2024-03-11T21:27:58.227 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:27:58.227 INFO:teuthology.orchestra.run.smithi092.stdout: "events": [ 2024-03-11T21:27:58.227 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:27:58.227 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "initiated", 2024-03-11T21:27:58.227 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:27:50.069877+0000", 2024-03-11T21:27:58.227 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-03-11T21:27:58.227 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:27:58.227 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:27:58.227 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "header_read", 2024-03-11T21:27:58.228 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:27:50.069877+0000", 2024-03-11T21:27:58.228 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-03-11T21:27:58.228 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:27:58.228 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:27:58.228 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "throttled", 2024-03-11T21:27:58.228 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:27:50.069878+0000", 2024-03-11T21:27:58.228 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 8.6799999999999999e-07 2024-03-11T21:27:58.228 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:27:58.228 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:27:58.228 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "all_read", 2024-03-11T21:27:58.228 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:27:50.069896+0000", 2024-03-11T21:27:58.228 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.8620000000000001e-05 2024-03-11T21:27:58.228 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:27:58.228 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:27:58.229 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "dispatched", 2024-03-11T21:27:58.229 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:27:50.069902+0000", 2024-03-11T21:27:58.229 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 6.2009999999999996e-06 2024-03-11T21:27:58.229 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:27:58.229 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:27:58.229 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "queued_for_pg", 2024-03-11T21:27:58.229 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:27:50.069909+0000", 2024-03-11T21:27:58.229 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 6.1870000000000002e-06 2024-03-11T21:27:58.229 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:27:58.229 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:27:58.229 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-03-11T21:27:58.229 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:27:50.070040+0000", 2024-03-11T21:27:58.229 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.000131068 2024-03-11T21:27:58.229 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:27:58.229 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:27:58.230 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "started", 2024-03-11T21:27:58.230 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:27:50.070157+0000", 2024-03-11T21:27:58.230 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00011712 2024-03-11T21:27:58.230 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:27:58.230 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:27:58.230 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "waiting for subops from 1,4", 2024-03-11T21:27:58.230 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:27:50.070290+0000", 2024-03-11T21:27:58.230 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.000132781 2024-03-11T21:27:58.230 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:27:58.230 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:27:58.230 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "op_commit", 2024-03-11T21:27:58.230 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:27:50.142261+0000", 2024-03-11T21:27:58.230 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.071971025999999994 2024-03-11T21:27:58.230 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:27:58.231 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:27:58.231 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "sub_op_commit_rec", 2024-03-11T21:27:58.231 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:27:50.142866+0000", 2024-03-11T21:27:58.231 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00060502799999999997 2024-03-11T21:27:58.231 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:27:58.231 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:27:58.231 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "sub_op_commit_rec", 2024-03-11T21:27:58.231 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:27:50.142916+0000", 2024-03-11T21:27:58.231 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 5.0448000000000002e-05 2024-03-11T21:27:58.231 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:27:58.231 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:27:58.231 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "commit_sent", 2024-03-11T21:27:58.231 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:27:50.142973+0000", 2024-03-11T21:27:58.231 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 5.6484999999999999e-05 2024-03-11T21:27:58.231 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:27:58.232 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:27:58.232 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "done", 2024-03-11T21:27:58.232 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:27:50.143029+0000", 2024-03-11T21:27:58.232 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 5.6932999999999999e-05 2024-03-11T21:27:58.232 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-03-11T21:27:58.232 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-03-11T21:27:58.232 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-03-11T21:27:58.232 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:27:58.232 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:27:58.232 INFO:teuthology.orchestra.run.smithi092.stdout: "description": "osd_repop(client.34715.0:15949 7.7 e105/99 7:e1ebd7ef:::benchmark_data_smithi092_76040_object15948:head v 105'1028, mlcod=105'1025)", 2024-03-11T21:27:58.232 INFO:teuthology.orchestra.run.smithi092.stdout: "initiated_at": "2024-03-11T21:27:50.071646+0000", 2024-03-11T21:27:58.232 INFO:teuthology.orchestra.run.smithi092.stdout: "age": 8.1488502860000001, 2024-03-11T21:27:58.232 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.074146726999999996, 2024-03-11T21:27:58.232 INFO:teuthology.orchestra.run.smithi092.stdout: "type_data": { 2024-03-11T21:27:58.232 INFO:teuthology.orchestra.run.smithi092.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-11T21:27:58.233 INFO:teuthology.orchestra.run.smithi092.stdout: "events": [ 2024-03-11T21:27:58.233 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:27:58.233 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "initiated", 2024-03-11T21:27:58.233 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:27:50.071646+0000", 2024-03-11T21:27:58.233 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-03-11T21:27:58.233 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:27:58.233 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:27:58.233 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "header_read", 2024-03-11T21:27:58.233 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:27:50.071646+0000", 2024-03-11T21:27:58.233 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-03-11T21:27:58.233 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:27:58.233 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:27:58.233 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "throttled", 2024-03-11T21:27:58.233 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:27:50.071647+0000", 2024-03-11T21:27:58.233 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.3170000000000001e-06 2024-03-11T21:27:58.234 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:27:58.234 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:27:58.234 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "all_read", 2024-03-11T21:27:58.234 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:27:50.071681+0000", 2024-03-11T21:27:58.234 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 3.4112999999999997e-05 2024-03-11T21:27:58.234 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:27:58.234 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:27:58.235 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "dispatched", 2024-03-11T21:27:58.235 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:27:50.142716+0000", 2024-03-11T21:27:58.235 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.071034326999999994 2024-03-11T21:27:58.235 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:27:58.235 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:27:58.235 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "queued_for_pg", 2024-03-11T21:27:58.235 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:27:50.142727+0000", 2024-03-11T21:27:58.235 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.1705000000000001e-05 2024-03-11T21:27:58.235 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:27:58.235 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:27:58.236 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-03-11T21:27:58.236 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:27:50.142799+0000", 2024-03-11T21:27:58.236 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 7.1914000000000003e-05 2024-03-11T21:27:58.236 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:27:58.236 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:27:58.236 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "started", 2024-03-11T21:27:58.236 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:27:50.142819+0000", 2024-03-11T21:27:58.236 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.9658999999999999e-05 2024-03-11T21:27:58.236 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:27:58.236 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:27:58.236 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "commit_sent", 2024-03-11T21:27:58.236 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:27:50.145738+0000", 2024-03-11T21:27:58.236 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.0029194910000000002 2024-03-11T21:27:58.236 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:27:58.237 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:27:58.237 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "done", 2024-03-11T21:27:58.237 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:27:50.145793+0000", 2024-03-11T21:27:58.237 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 5.4200999999999998e-05 2024-03-11T21:27:58.237 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-03-11T21:27:58.237 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-03-11T21:27:58.237 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-03-11T21:27:58.237 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:27:58.237 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:27:58.237 INFO:teuthology.orchestra.run.smithi092.stdout: "description": "osd_repop(client.34715.0:15952 7.e e105/99 7:7096d29d:::benchmark_data_smithi092_76040_object15951:head v 105'974, mlcod=105'972)", 2024-03-11T21:27:58.237 INFO:teuthology.orchestra.run.smithi092.stdout: "initiated_at": "2024-03-11T21:27:50.071711+0000", 2024-03-11T21:27:58.237 INFO:teuthology.orchestra.run.smithi092.stdout: "age": 8.1487851540000005, 2024-03-11T21:27:58.237 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.073763256999999999, 2024-03-11T21:27:58.237 INFO:teuthology.orchestra.run.smithi092.stdout: "type_data": { 2024-03-11T21:27:58.238 INFO:teuthology.orchestra.run.smithi092.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-11T21:27:58.238 INFO:teuthology.orchestra.run.smithi092.stdout: "events": [ 2024-03-11T21:27:58.238 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:27:58.238 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "initiated", 2024-03-11T21:27:58.238 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:27:50.071711+0000", 2024-03-11T21:27:58.238 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-03-11T21:27:58.238 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:27:58.238 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:27:58.238 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "header_read", 2024-03-11T21:27:58.238 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:27:50.071711+0000", 2024-03-11T21:27:58.238 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-03-11T21:27:58.238 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:27:58.239 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:27:58.239 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "throttled", 2024-03-11T21:27:58.239 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:27:50.071712+0000", 2024-03-11T21:27:58.239 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 6.5899999999999996e-07 2024-03-11T21:27:58.239 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:27:58.239 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:27:58.239 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "all_read", 2024-03-11T21:27:58.239 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:27:50.071732+0000", 2024-03-11T21:27:58.239 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 2.0288e-05 2024-03-11T21:27:58.239 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:27:58.239 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:27:58.239 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "dispatched", 2024-03-11T21:27:58.239 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:27:50.142763+0000", 2024-03-11T21:27:58.239 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.071030842999999996 2024-03-11T21:27:58.240 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:27:58.240 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:27:58.240 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "queued_for_pg", 2024-03-11T21:27:58.240 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:27:50.142769+0000", 2024-03-11T21:27:58.240 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 6.3369999999999996e-06 2024-03-11T21:27:58.240 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:27:58.240 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:27:58.240 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-03-11T21:27:58.240 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:27:50.143047+0000", 2024-03-11T21:27:58.240 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00027801399999999998 2024-03-11T21:27:58.240 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:27:58.240 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:27:58.240 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "started", 2024-03-11T21:27:58.240 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:27:50.143067+0000", 2024-03-11T21:27:58.241 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.9610999999999999e-05 2024-03-11T21:27:58.241 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:27:58.241 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:27:58.241 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "commit_sent", 2024-03-11T21:27:58.241 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:27:50.145445+0000", 2024-03-11T21:27:58.241 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.0023780979999999999 2024-03-11T21:27:58.241 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:27:58.241 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:27:58.241 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "done", 2024-03-11T21:27:58.241 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:27:50.145474+0000", 2024-03-11T21:27:58.241 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 2.9407000000000002e-05 2024-03-11T21:27:58.241 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-03-11T21:27:58.241 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-03-11T21:27:58.241 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-03-11T21:27:58.242 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:27:58.242 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:27:58.242 INFO:teuthology.orchestra.run.smithi092.stdout: "description": "osd_op(client.34715.0:16784 7.6 7:6b68ea5e:::benchmark_data_smithi092_76040_object16783:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e105)", 2024-03-11T21:27:58.242 INFO:teuthology.orchestra.run.smithi092.stdout: "initiated_at": "2024-03-11T21:27:50.453367+0000", 2024-03-11T21:27:58.242 INFO:teuthology.orchestra.run.smithi092.stdout: "age": 7.7671295230000004, 2024-03-11T21:27:58.242 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.069834113000000003, 2024-03-11T21:27:58.242 INFO:teuthology.orchestra.run.smithi092.stdout: "type_data": { 2024-03-11T21:27:58.242 INFO:teuthology.orchestra.run.smithi092.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-11T21:27:58.242 INFO:teuthology.orchestra.run.smithi092.stdout: "client_info": { 2024-03-11T21:27:58.242 INFO:teuthology.orchestra.run.smithi092.stdout: "client": "client.34715", 2024-03-11T21:27:58.242 INFO:teuthology.orchestra.run.smithi092.stdout: "client_addr": "v1:172.21.15.92:0/2403132573", 2024-03-11T21:27:58.242 INFO:teuthology.orchestra.run.smithi092.stdout: "tid": 16784 2024-03-11T21:27:58.242 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:27:58.243 INFO:teuthology.orchestra.run.smithi092.stdout: "events": [ 2024-03-11T21:27:58.243 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:27:58.243 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "initiated", 2024-03-11T21:27:58.243 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:27:50.453367+0000", 2024-03-11T21:27:58.243 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-03-11T21:27:58.243 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:27:58.243 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:27:58.243 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "header_read", 2024-03-11T21:27:58.243 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:27:50.453367+0000", 2024-03-11T21:27:58.243 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-03-11T21:27:58.243 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:27:58.243 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:27:58.243 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "throttled", 2024-03-11T21:27:58.243 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:27:50.453368+0000", 2024-03-11T21:27:58.243 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.4300000000000001e-06 2024-03-11T21:27:58.244 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:27:58.244 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:27:58.244 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "all_read", 2024-03-11T21:27:58.244 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:27:50.453395+0000", 2024-03-11T21:27:58.244 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 2.7121e-05 2024-03-11T21:27:58.244 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:27:58.244 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:27:58.244 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "dispatched", 2024-03-11T21:27:58.244 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:27:50.453405+0000", 2024-03-11T21:27:58.244 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.0142999999999999e-05 2024-03-11T21:27:58.244 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:27:58.244 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:27:58.244 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "queued_for_pg", 2024-03-11T21:27:58.244 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:27:50.453413+0000", 2024-03-11T21:27:58.245 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 7.8730000000000007e-06 2024-03-11T21:27:58.245 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:27:58.245 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:27:58.245 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-03-11T21:27:58.245 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:27:50.453790+0000", 2024-03-11T21:27:58.245 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00037727600000000001 2024-03-11T21:27:58.245 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:27:58.245 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:27:58.245 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "started", 2024-03-11T21:27:58.245 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:27:50.453913+0000", 2024-03-11T21:27:58.245 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00012297999999999999 2024-03-11T21:27:58.245 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:27:58.245 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:27:58.245 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "waiting for subops from 3,7", 2024-03-11T21:27:58.245 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:27:50.454044+0000", 2024-03-11T21:27:58.246 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.000130894 2024-03-11T21:27:58.246 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:27:58.246 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:27:58.246 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "op_commit", 2024-03-11T21:27:58.246 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:27:50.455122+0000", 2024-03-11T21:27:58.246 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.0010772830000000001 2024-03-11T21:27:58.246 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:27:58.246 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:27:58.246 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "sub_op_commit_rec", 2024-03-11T21:27:58.246 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:27:50.456305+0000", 2024-03-11T21:27:58.246 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.0011829640000000001 2024-03-11T21:27:58.246 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:27:58.246 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:27:58.246 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "sub_op_commit_rec", 2024-03-11T21:27:58.247 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:27:50.523106+0000", 2024-03-11T21:27:58.247 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.066801456999999995 2024-03-11T21:27:58.247 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:27:58.247 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:27:58.247 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "commit_sent", 2024-03-11T21:27:58.247 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:27:50.523159+0000", 2024-03-11T21:27:58.247 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 5.2567e-05 2024-03-11T21:27:58.247 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:27:58.247 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:27:58.247 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "done", 2024-03-11T21:27:58.247 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:27:50.523201+0000", 2024-03-11T21:27:58.247 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 4.2125000000000001e-05 2024-03-11T21:27:58.247 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-03-11T21:27:58.247 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-03-11T21:27:58.248 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-03-11T21:27:58.248 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:27:58.248 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:27:58.248 INFO:teuthology.orchestra.run.smithi092.stdout: "description": "osd_op(client.34715.0:21625 7.9 7:98d2f441:::benchmark_data_smithi092_76040_object21624:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e105)", 2024-03-11T21:27:58.248 INFO:teuthology.orchestra.run.smithi092.stdout: "initiated_at": "2024-03-11T21:27:52.543604+0000", 2024-03-11T21:27:58.248 INFO:teuthology.orchestra.run.smithi092.stdout: "age": 5.6768916120000004, 2024-03-11T21:27:58.248 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.100324792, 2024-03-11T21:27:58.248 INFO:teuthology.orchestra.run.smithi092.stdout: "type_data": { 2024-03-11T21:27:58.248 INFO:teuthology.orchestra.run.smithi092.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-11T21:27:58.248 INFO:teuthology.orchestra.run.smithi092.stdout: "client_info": { 2024-03-11T21:27:58.248 INFO:teuthology.orchestra.run.smithi092.stdout: "client": "client.34715", 2024-03-11T21:27:58.248 INFO:teuthology.orchestra.run.smithi092.stdout: "client_addr": "v1:172.21.15.92:0/2403132573", 2024-03-11T21:27:58.248 INFO:teuthology.orchestra.run.smithi092.stdout: "tid": 21625 2024-03-11T21:27:58.248 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:27:58.249 INFO:teuthology.orchestra.run.smithi092.stdout: "events": [ 2024-03-11T21:27:58.249 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:27:58.249 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "initiated", 2024-03-11T21:27:58.249 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:27:52.543604+0000", 2024-03-11T21:27:58.249 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-03-11T21:27:58.249 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:27:58.249 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:27:58.249 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "header_read", 2024-03-11T21:27:58.249 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:27:52.543604+0000", 2024-03-11T21:27:58.249 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-03-11T21:27:58.249 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:27:58.249 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:27:58.249 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "throttled", 2024-03-11T21:27:58.249 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:27:52.543606+0000", 2024-03-11T21:27:58.249 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.8470000000000001e-06 2024-03-11T21:27:58.250 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:27:58.250 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:27:58.250 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "all_read", 2024-03-11T21:27:58.250 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:27:52.543629+0000", 2024-03-11T21:27:58.250 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 2.2359999999999999e-05 2024-03-11T21:27:58.250 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:27:58.250 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:27:58.250 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "dispatched", 2024-03-11T21:27:58.250 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:27:52.543639+0000", 2024-03-11T21:27:58.250 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 9.9089999999999995e-06 2024-03-11T21:27:58.250 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:27:58.250 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:27:58.250 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "queued_for_pg", 2024-03-11T21:27:58.251 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:27:52.543647+0000", 2024-03-11T21:27:58.251 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 7.9019999999999999e-06 2024-03-11T21:27:58.251 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:27:58.251 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:27:58.251 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-03-11T21:27:58.251 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:27:52.602139+0000", 2024-03-11T21:27:58.251 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.058492028000000001 2024-03-11T21:27:58.251 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:27:58.251 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:27:58.251 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "started", 2024-03-11T21:27:58.251 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:27:52.602260+0000", 2024-03-11T21:27:58.251 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.000121727 2024-03-11T21:27:58.251 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:27:58.251 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:27:58.251 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "waiting for subops from 1,4", 2024-03-11T21:27:58.252 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:27:52.602399+0000", 2024-03-11T21:27:58.252 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00013832699999999999 2024-03-11T21:27:58.252 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:27:58.252 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:27:58.252 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "op_commit", 2024-03-11T21:27:58.252 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:27:52.604745+0000", 2024-03-11T21:27:58.252 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.002346042 2024-03-11T21:27:58.252 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:27:58.252 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:27:58.252 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "sub_op_commit_rec", 2024-03-11T21:27:58.252 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:27:52.604780+0000", 2024-03-11T21:27:58.252 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 3.5732000000000003e-05 2024-03-11T21:27:58.252 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:27:58.252 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:27:58.253 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "sub_op_commit_rec", 2024-03-11T21:27:58.253 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:27:52.643838+0000", 2024-03-11T21:27:58.253 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.039057671000000002 2024-03-11T21:27:58.253 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:27:58.253 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:27:58.253 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "commit_sent", 2024-03-11T21:27:58.253 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:27:52.643890+0000", 2024-03-11T21:27:58.253 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 5.1894000000000002e-05 2024-03-11T21:27:58.253 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:27:58.253 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:27:58.253 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "done", 2024-03-11T21:27:58.253 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:27:52.643929+0000", 2024-03-11T21:27:58.253 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 3.9353000000000001e-05 2024-03-11T21:27:58.253 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-03-11T21:27:58.253 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-03-11T21:27:58.254 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-03-11T21:27:58.254 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:27:58.254 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:27:58.254 INFO:teuthology.orchestra.run.smithi092.stdout: "description": "osd_op(client.34715.0:27056 7.9 7:98e161bb:::benchmark_data_smithi092_76040_object27055:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e105)", 2024-03-11T21:27:58.254 INFO:teuthology.orchestra.run.smithi092.stdout: "initiated_at": "2024-03-11T21:27:55.091450+0000", 2024-03-11T21:27:58.254 INFO:teuthology.orchestra.run.smithi092.stdout: "age": 3.129046094, 2024-03-11T21:27:58.254 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.068391780999999999, 2024-03-11T21:27:58.254 INFO:teuthology.orchestra.run.smithi092.stdout: "type_data": { 2024-03-11T21:27:58.254 INFO:teuthology.orchestra.run.smithi092.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-11T21:27:58.254 INFO:teuthology.orchestra.run.smithi092.stdout: "client_info": { 2024-03-11T21:27:58.254 INFO:teuthology.orchestra.run.smithi092.stdout: "client": "client.34715", 2024-03-11T21:27:58.254 INFO:teuthology.orchestra.run.smithi092.stdout: "client_addr": "v1:172.21.15.92:0/2403132573", 2024-03-11T21:27:58.254 INFO:teuthology.orchestra.run.smithi092.stdout: "tid": 27056 2024-03-11T21:27:58.255 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:27:58.255 INFO:teuthology.orchestra.run.smithi092.stdout: "events": [ 2024-03-11T21:27:58.255 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:27:58.255 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "initiated", 2024-03-11T21:27:58.255 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:27:55.091450+0000", 2024-03-11T21:27:58.255 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-03-11T21:27:58.255 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:27:58.255 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:27:58.255 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "header_read", 2024-03-11T21:27:58.255 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:27:55.091450+0000", 2024-03-11T21:27:58.255 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-03-11T21:27:58.255 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:27:58.255 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:27:58.255 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "throttled", 2024-03-11T21:27:58.255 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:27:55.091452+0000", 2024-03-11T21:27:58.256 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.7600000000000001e-06 2024-03-11T21:27:58.256 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:27:58.256 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:27:58.256 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "all_read", 2024-03-11T21:27:58.256 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:27:55.091480+0000", 2024-03-11T21:27:58.256 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 2.8065e-05 2024-03-11T21:27:58.256 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:27:58.256 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:27:58.256 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "dispatched", 2024-03-11T21:27:58.256 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:27:55.091491+0000", 2024-03-11T21:27:58.256 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.0825000000000001e-05 2024-03-11T21:27:58.256 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:27:58.256 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:27:58.256 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "queued_for_pg", 2024-03-11T21:27:58.257 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:27:55.091499+0000", 2024-03-11T21:27:58.257 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 8.0269999999999995e-06 2024-03-11T21:27:58.257 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:27:58.257 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:27:58.257 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-03-11T21:27:58.257 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:27:55.092980+0000", 2024-03-11T21:27:58.257 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.001481766 2024-03-11T21:27:58.257 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:27:58.257 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:27:58.257 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "started", 2024-03-11T21:27:58.257 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:27:55.093119+0000", 2024-03-11T21:27:58.257 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.000138722 2024-03-11T21:27:58.257 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:27:58.257 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:27:58.257 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "waiting for subops from 1,4", 2024-03-11T21:27:58.258 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:27:55.093286+0000", 2024-03-11T21:27:58.258 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00016689100000000001 2024-03-11T21:27:58.258 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:27:58.258 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:27:58.258 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "op_commit", 2024-03-11T21:27:58.258 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:27:55.095859+0000", 2024-03-11T21:27:58.258 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.0025725399999999999 2024-03-11T21:27:58.258 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:27:58.258 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:27:58.258 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "sub_op_commit_rec", 2024-03-11T21:27:58.258 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:27:55.159717+0000", 2024-03-11T21:27:58.258 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.063858528999999997 2024-03-11T21:27:58.258 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:27:58.258 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:27:58.259 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "sub_op_commit_rec", 2024-03-11T21:27:58.259 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:27:55.159750+0000", 2024-03-11T21:27:58.259 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 3.2716999999999999e-05 2024-03-11T21:27:58.259 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:27:58.259 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:27:58.259 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "commit_sent", 2024-03-11T21:27:58.259 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:27:55.159797+0000", 2024-03-11T21:27:58.259 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 4.7639000000000002e-05 2024-03-11T21:27:58.259 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:27:58.259 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:27:58.259 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "done", 2024-03-11T21:27:58.259 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:27:55.159842+0000", 2024-03-11T21:27:58.259 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 4.4299999999999999e-05 2024-03-11T21:27:58.259 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-03-11T21:27:58.260 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-03-11T21:27:58.260 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-03-11T21:27:58.260 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:27:58.260 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:27:58.260 INFO:teuthology.orchestra.run.smithi092.stdout: "description": "osd_op(client.34715.0:27064 7.f 7:fe2040ff:::benchmark_data_smithi092_76040_object27063:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e105)", 2024-03-11T21:27:58.260 INFO:teuthology.orchestra.run.smithi092.stdout: "initiated_at": "2024-03-11T21:27:55.095705+0000", 2024-03-11T21:27:58.260 INFO:teuthology.orchestra.run.smithi092.stdout: "age": 3.1247909250000001, 2024-03-11T21:27:58.260 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.069831330999999996, 2024-03-11T21:27:58.260 INFO:teuthology.orchestra.run.smithi092.stdout: "type_data": { 2024-03-11T21:27:58.260 INFO:teuthology.orchestra.run.smithi092.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-11T21:27:58.260 INFO:teuthology.orchestra.run.smithi092.stdout: "client_info": { 2024-03-11T21:27:58.260 INFO:teuthology.orchestra.run.smithi092.stdout: "client": "client.34715", 2024-03-11T21:27:58.260 INFO:teuthology.orchestra.run.smithi092.stdout: "client_addr": "v1:172.21.15.92:0/2403132573", 2024-03-11T21:27:58.260 INFO:teuthology.orchestra.run.smithi092.stdout: "tid": 27064 2024-03-11T21:27:58.260 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:27:58.261 INFO:teuthology.orchestra.run.smithi092.stdout: "events": [ 2024-03-11T21:27:58.261 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:27:58.261 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "initiated", 2024-03-11T21:27:58.261 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:27:55.095705+0000", 2024-03-11T21:27:58.261 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-03-11T21:27:58.261 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:27:58.261 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:27:58.261 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "header_read", 2024-03-11T21:27:58.261 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:27:55.095705+0000", 2024-03-11T21:27:58.261 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-03-11T21:27:58.261 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:27:58.261 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:27:58.262 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:27:58.263 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "throttled", 2024-03-11T21:27:58.263 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:27:55.095707+0000", 2024-03-11T21:27:58.263 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.6500000000000001e-06 2024-03-11T21:27:58.263 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:27:58.263 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:27:58.263 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "all_read", 2024-03-11T21:27:58.263 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:27:55.095729+0000", 2024-03-11T21:27:58.263 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 2.1980999999999998e-05 2024-03-11T21:27:58.263 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:27:58.263 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:27:58.263 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "dispatched", 2024-03-11T21:27:58.264 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:27:55.095738+0000", 2024-03-11T21:27:58.264 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 8.9080000000000001e-06 2024-03-11T21:27:58.264 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:27:58.264 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:27:58.264 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "queued_for_pg", 2024-03-11T21:27:58.264 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:27:55.095745+0000", 2024-03-11T21:27:58.264 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 7.0369999999999997e-06 2024-03-11T21:27:58.264 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:27:58.264 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:27:58.264 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-03-11T21:27:58.264 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:27:55.095964+0000", 2024-03-11T21:27:58.264 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00021944 2024-03-11T21:27:58.264 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:27:58.264 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:27:58.265 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "started", 2024-03-11T21:27:58.265 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:27:55.157057+0000", 2024-03-11T21:27:58.265 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.061093033999999997 2024-03-11T21:27:58.265 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:27:58.265 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:27:58.265 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "waiting for subops from 3,7", 2024-03-11T21:27:58.265 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:27:55.157209+0000", 2024-03-11T21:27:58.265 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00015175599999999999 2024-03-11T21:27:58.265 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:27:58.265 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:27:58.265 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "op_commit", 2024-03-11T21:27:58.265 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:27:55.159289+0000", 2024-03-11T21:27:58.265 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.0020797350000000001 2024-03-11T21:27:58.265 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:27:58.265 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:27:58.266 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "sub_op_commit_rec", 2024-03-11T21:27:58.266 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:27:55.163908+0000", 2024-03-11T21:27:58.266 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.0046195170000000001 2024-03-11T21:27:58.266 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:27:58.266 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:27:58.266 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "sub_op_commit_rec", 2024-03-11T21:27:58.266 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:27:55.165443+0000", 2024-03-11T21:27:58.266 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.0015344480000000001 2024-03-11T21:27:58.266 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:27:58.266 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:27:58.266 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "commit_sent", 2024-03-11T21:27:58.266 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:27:55.165493+0000", 2024-03-11T21:27:58.266 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 5.0138e-05 2024-03-11T21:27:58.267 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:27:58.267 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:27:58.267 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "done", 2024-03-11T21:27:58.267 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:27:55.165537+0000", 2024-03-11T21:27:58.267 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 4.3686999999999997e-05 2024-03-11T21:27:58.267 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-03-11T21:27:58.267 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-03-11T21:27:58.267 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-03-11T21:27:58.267 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:27:58.267 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:27:58.267 INFO:teuthology.orchestra.run.smithi092.stdout: "description": "osd_op(client.34715.0:28466 7.9 7:9334e072:::benchmark_data_smithi092_76040_object28465:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e105)", 2024-03-11T21:27:58.267 INFO:teuthology.orchestra.run.smithi092.stdout: "initiated_at": "2024-03-11T21:27:55.845117+0000", 2024-03-11T21:27:58.267 INFO:teuthology.orchestra.run.smithi092.stdout: "age": 2.3753793129999998, 2024-03-11T21:27:58.267 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.069099585000000005, 2024-03-11T21:27:58.267 INFO:teuthology.orchestra.run.smithi092.stdout: "type_data": { 2024-03-11T21:27:58.268 INFO:teuthology.orchestra.run.smithi092.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-11T21:27:58.268 INFO:teuthology.orchestra.run.smithi092.stdout: "client_info": { 2024-03-11T21:27:58.268 INFO:teuthology.orchestra.run.smithi092.stdout: "client": "client.34715", 2024-03-11T21:27:58.268 INFO:teuthology.orchestra.run.smithi092.stdout: "client_addr": "v1:172.21.15.92:0/2403132573", 2024-03-11T21:27:58.268 INFO:teuthology.orchestra.run.smithi092.stdout: "tid": 28466 2024-03-11T21:27:58.268 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:27:58.268 INFO:teuthology.orchestra.run.smithi092.stdout: "events": [ 2024-03-11T21:27:58.268 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:27:58.268 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "initiated", 2024-03-11T21:27:58.268 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:27:55.845117+0000", 2024-03-11T21:27:58.268 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-03-11T21:27:58.268 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:27:58.268 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:27:58.268 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "header_read", 2024-03-11T21:27:58.269 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:27:55.845117+0000", 2024-03-11T21:27:58.269 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-03-11T21:27:58.269 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:27:58.269 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:27:58.269 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "throttled", 2024-03-11T21:27:58.269 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:27:55.845118+0000", 2024-03-11T21:27:58.269 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 9.0100000000000003e-07 2024-03-11T21:27:58.269 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:27:58.269 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:27:58.269 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "all_read", 2024-03-11T21:27:58.269 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:27:55.845144+0000", 2024-03-11T21:27:58.269 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 2.6681000000000001e-05 2024-03-11T21:27:58.269 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:27:58.269 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:27:58.269 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "dispatched", 2024-03-11T21:27:58.270 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:27:55.845152+0000", 2024-03-11T21:27:58.270 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 8.0050000000000004e-06 2024-03-11T21:27:58.270 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:27:58.270 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:27:58.270 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "queued_for_pg", 2024-03-11T21:27:58.270 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:27:55.845159+0000", 2024-03-11T21:27:58.270 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 6.9829999999999999e-06 2024-03-11T21:27:58.270 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:27:58.270 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:27:58.270 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-03-11T21:27:58.270 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:27:55.845217+0000", 2024-03-11T21:27:58.270 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 5.7720999999999998e-05 2024-03-11T21:27:58.270 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:27:58.271 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:27:58.271 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "started", 2024-03-11T21:27:58.271 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:27:55.845347+0000", 2024-03-11T21:27:58.271 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00013006899999999999 2024-03-11T21:27:58.271 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:27:58.271 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:27:58.271 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "waiting for subops from 1,4", 2024-03-11T21:27:58.271 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:27:55.845482+0000", 2024-03-11T21:27:58.271 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.000134461 2024-03-11T21:27:58.271 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:27:58.271 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:27:58.271 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "op_commit", 2024-03-11T21:27:58.271 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:27:55.848198+0000", 2024-03-11T21:27:58.271 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.0027165959999999999 2024-03-11T21:27:58.272 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:27:58.272 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:27:58.272 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "sub_op_commit_rec", 2024-03-11T21:27:58.272 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:27:55.848249+0000", 2024-03-11T21:27:58.272 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 5.0705999999999999e-05 2024-03-11T21:27:58.272 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:27:58.272 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:27:58.272 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "sub_op_commit_rec", 2024-03-11T21:27:58.272 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:27:55.914133+0000", 2024-03-11T21:27:58.272 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.065884317999999997 2024-03-11T21:27:58.272 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:27:58.272 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:27:58.272 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "commit_sent", 2024-03-11T21:27:58.272 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:27:55.914178+0000", 2024-03-11T21:27:58.272 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 4.5145000000000002e-05 2024-03-11T21:27:58.273 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:27:58.273 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:27:58.273 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "done", 2024-03-11T21:27:58.273 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:27:55.914216+0000", 2024-03-11T21:27:58.273 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 3.7999e-05 2024-03-11T21:27:58.273 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-03-11T21:27:58.273 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-03-11T21:27:58.273 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-03-11T21:27:58.273 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:27:58.273 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:27:58.273 INFO:teuthology.orchestra.run.smithi092.stdout: "description": "osd_op(client.34715.0:28473 7.6 7:6ecc34ee:::benchmark_data_smithi092_76040_object28472:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e105)", 2024-03-11T21:27:58.273 INFO:teuthology.orchestra.run.smithi092.stdout: "initiated_at": "2024-03-11T21:27:55.845280+0000", 2024-03-11T21:27:58.273 INFO:teuthology.orchestra.run.smithi092.stdout: "age": 2.3752158909999999, 2024-03-11T21:27:58.273 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.067755423999999995, 2024-03-11T21:27:58.273 INFO:teuthology.orchestra.run.smithi092.stdout: "type_data": { 2024-03-11T21:27:58.274 INFO:teuthology.orchestra.run.smithi092.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-11T21:27:58.274 INFO:teuthology.orchestra.run.smithi092.stdout: "client_info": { 2024-03-11T21:27:58.274 INFO:teuthology.orchestra.run.smithi092.stdout: "client": "client.34715", 2024-03-11T21:27:58.274 INFO:teuthology.orchestra.run.smithi092.stdout: "client_addr": "v1:172.21.15.92:0/2403132573", 2024-03-11T21:27:58.274 INFO:teuthology.orchestra.run.smithi092.stdout: "tid": 28473 2024-03-11T21:27:58.274 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:27:58.274 INFO:teuthology.orchestra.run.smithi092.stdout: "events": [ 2024-03-11T21:27:58.274 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:27:58.274 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "initiated", 2024-03-11T21:27:58.274 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:27:55.845280+0000", 2024-03-11T21:27:58.274 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-03-11T21:27:58.274 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:27:58.274 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:27:58.274 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "header_read", 2024-03-11T21:27:58.275 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:27:55.845280+0000", 2024-03-11T21:27:58.275 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-03-11T21:27:58.275 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:27:58.275 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:27:58.275 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "throttled", 2024-03-11T21:27:58.275 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:27:55.845281+0000", 2024-03-11T21:27:58.275 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 7.4399999999999999e-07 2024-03-11T21:27:58.275 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:27:58.275 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:27:58.275 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "all_read", 2024-03-11T21:27:58.275 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:27:55.845305+0000", 2024-03-11T21:27:58.275 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 2.3748e-05 2024-03-11T21:27:58.275 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:27:58.275 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:27:58.275 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "dispatched", 2024-03-11T21:27:58.276 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:27:55.845312+0000", 2024-03-11T21:27:58.276 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 6.9120000000000001e-06 2024-03-11T21:27:58.276 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:27:58.276 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:27:58.276 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "queued_for_pg", 2024-03-11T21:27:58.276 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:27:55.845318+0000", 2024-03-11T21:27:58.276 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 6.2580000000000001e-06 2024-03-11T21:27:58.276 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:27:58.276 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:27:58.276 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-03-11T21:27:58.276 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:27:55.847476+0000", 2024-03-11T21:27:58.276 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.002157932 2024-03-11T21:27:58.276 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:27:58.276 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:27:58.276 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "started", 2024-03-11T21:27:58.277 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:27:55.847596+0000", 2024-03-11T21:27:58.277 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.000119987 2024-03-11T21:27:58.277 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:27:58.277 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:27:58.277 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "waiting for subops from 3,7", 2024-03-11T21:27:58.277 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:27:55.847724+0000", 2024-03-11T21:27:58.277 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00012828399999999999 2024-03-11T21:27:58.277 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:27:58.277 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:27:58.277 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "sub_op_commit_rec", 2024-03-11T21:27:58.277 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:27:55.849979+0000", 2024-03-11T21:27:58.277 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.0022552940000000001 2024-03-11T21:27:58.277 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:27:58.277 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:27:58.277 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "op_commit", 2024-03-11T21:27:58.277 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:27:55.850017+0000", 2024-03-11T21:27:58.278 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 3.7591e-05 2024-03-11T21:27:58.278 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:27:58.278 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:27:58.278 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "sub_op_commit_rec", 2024-03-11T21:27:58.278 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:27:55.912920+0000", 2024-03-11T21:27:58.278 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.062903333000000006 2024-03-11T21:27:58.278 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:27:58.278 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:27:58.278 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "commit_sent", 2024-03-11T21:27:58.278 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:27:55.912975+0000", 2024-03-11T21:27:58.278 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 5.4503000000000003e-05 2024-03-11T21:27:58.278 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:27:58.278 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:27:58.279 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "done", 2024-03-11T21:27:58.279 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:27:55.913036+0000", 2024-03-11T21:27:58.279 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 6.0838000000000003e-05 2024-03-11T21:27:58.279 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-03-11T21:27:58.279 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-03-11T21:27:58.279 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-03-11T21:27:58.279 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:27:58.279 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:27:58.279 INFO:teuthology.orchestra.run.smithi092.stdout: "description": "osd_op(client.34715.0:28482 7.9 7:936b7921:::benchmark_data_smithi092_76040_object28481:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e105)", 2024-03-11T21:27:58.279 INFO:teuthology.orchestra.run.smithi092.stdout: "initiated_at": "2024-03-11T21:27:55.851203+0000", 2024-03-11T21:27:58.279 INFO:teuthology.orchestra.run.smithi092.stdout: "age": 2.3692930730000001, 2024-03-11T21:27:58.279 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.068111777999999998, 2024-03-11T21:27:58.279 INFO:teuthology.orchestra.run.smithi092.stdout: "type_data": { 2024-03-11T21:27:58.279 INFO:teuthology.orchestra.run.smithi092.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-11T21:27:58.280 INFO:teuthology.orchestra.run.smithi092.stdout: "client_info": { 2024-03-11T21:27:58.280 INFO:teuthology.orchestra.run.smithi092.stdout: "client": "client.34715", 2024-03-11T21:27:58.280 INFO:teuthology.orchestra.run.smithi092.stdout: "client_addr": "v1:172.21.15.92:0/2403132573", 2024-03-11T21:27:58.280 INFO:teuthology.orchestra.run.smithi092.stdout: "tid": 28482 2024-03-11T21:27:58.280 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:27:58.280 INFO:teuthology.orchestra.run.smithi092.stdout: "events": [ 2024-03-11T21:27:58.280 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:27:58.280 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "initiated", 2024-03-11T21:27:58.280 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:27:55.851203+0000", 2024-03-11T21:27:58.280 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-03-11T21:27:58.280 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:27:58.280 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:27:58.280 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "header_read", 2024-03-11T21:27:58.280 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:27:55.851203+0000", 2024-03-11T21:27:58.281 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-03-11T21:27:58.281 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:27:58.281 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:27:58.281 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "throttled", 2024-03-11T21:27:58.281 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:27:55.851204+0000", 2024-03-11T21:27:58.281 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 8.6899999999999996e-07 2024-03-11T21:27:58.281 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:27:58.281 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:27:58.281 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "all_read", 2024-03-11T21:27:58.281 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:27:55.851225+0000", 2024-03-11T21:27:58.281 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 2.1432000000000001e-05 2024-03-11T21:27:58.281 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:27:58.281 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:27:58.281 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "dispatched", 2024-03-11T21:27:58.281 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:27:55.851232+0000", 2024-03-11T21:27:58.282 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 6.8639999999999998e-06 2024-03-11T21:27:58.282 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:27:58.282 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:27:58.282 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "queued_for_pg", 2024-03-11T21:27:58.282 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:27:55.851238+0000", 2024-03-11T21:27:58.282 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 5.93e-06 2024-03-11T21:27:58.282 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:27:58.282 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:27:58.282 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-03-11T21:27:58.282 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:27:55.851549+0000", 2024-03-11T21:27:58.282 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00031080600000000001 2024-03-11T21:27:58.282 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:27:58.282 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:27:58.282 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "started", 2024-03-11T21:27:58.283 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:27:55.851714+0000", 2024-03-11T21:27:58.283 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00016482399999999999 2024-03-11T21:27:58.283 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:27:58.283 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:27:58.283 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "waiting for subops from 1,4", 2024-03-11T21:27:58.283 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:27:55.851852+0000", 2024-03-11T21:27:58.283 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00013873299999999999 2024-03-11T21:27:58.283 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:27:58.283 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:27:58.283 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "op_commit", 2024-03-11T21:27:58.283 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:27:55.912592+0000", 2024-03-11T21:27:58.283 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.060739382000000001 2024-03-11T21:27:58.283 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:27:58.283 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:27:58.284 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "sub_op_commit_rec", 2024-03-11T21:27:58.284 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:27:55.915764+0000", 2024-03-11T21:27:58.284 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.0031716679999999999 2024-03-11T21:27:58.284 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:27:58.284 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:27:58.284 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "sub_op_commit_rec", 2024-03-11T21:27:58.284 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:27:55.919180+0000", 2024-03-11T21:27:58.284 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.003416166 2024-03-11T21:27:58.284 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:27:58.284 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:27:58.284 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "commit_sent", 2024-03-11T21:27:58.284 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:27:55.919261+0000", 2024-03-11T21:27:58.284 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 8.1013000000000005e-05 2024-03-11T21:27:58.284 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:27:58.284 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:27:58.285 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "done", 2024-03-11T21:27:58.285 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:27:55.919315+0000", 2024-03-11T21:27:58.285 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 5.4091000000000001e-05 2024-03-11T21:27:58.285 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-03-11T21:27:58.285 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-03-11T21:27:58.285 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-03-11T21:27:58.285 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:27:58.285 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:27:58.285 INFO:teuthology.orchestra.run.smithi092.stdout: "description": "osd_op(client.34715.0:28485 7.6 7:6ac62981:::benchmark_data_smithi092_76040_object28484:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e105)", 2024-03-11T21:27:58.285 INFO:teuthology.orchestra.run.smithi092.stdout: "initiated_at": "2024-03-11T21:27:55.852868+0000", 2024-03-11T21:27:58.285 INFO:teuthology.orchestra.run.smithi092.stdout: "age": 2.3676276409999999, 2024-03-11T21:27:58.285 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.067341769999999995, 2024-03-11T21:27:58.285 INFO:teuthology.orchestra.run.smithi092.stdout: "type_data": { 2024-03-11T21:27:58.285 INFO:teuthology.orchestra.run.smithi092.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-11T21:27:58.286 INFO:teuthology.orchestra.run.smithi092.stdout: "client_info": { 2024-03-11T21:27:58.286 INFO:teuthology.orchestra.run.smithi092.stdout: "client": "client.34715", 2024-03-11T21:27:58.286 INFO:teuthology.orchestra.run.smithi092.stdout: "client_addr": "v1:172.21.15.92:0/2403132573", 2024-03-11T21:27:58.286 INFO:teuthology.orchestra.run.smithi092.stdout: "tid": 28485 2024-03-11T21:27:58.286 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:27:58.286 INFO:teuthology.orchestra.run.smithi092.stdout: "events": [ 2024-03-11T21:27:58.286 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:27:58.286 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "initiated", 2024-03-11T21:27:58.286 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:27:55.852868+0000", 2024-03-11T21:27:58.286 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-03-11T21:27:58.286 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:27:58.286 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:27:58.286 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "header_read", 2024-03-11T21:27:58.286 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:27:55.852868+0000", 2024-03-11T21:27:58.287 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-03-11T21:27:58.287 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:27:58.287 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:27:58.287 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "throttled", 2024-03-11T21:27:58.287 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:27:55.852870+0000", 2024-03-11T21:27:58.287 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.5209999999999999e-06 2024-03-11T21:27:58.287 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:27:58.287 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:27:58.287 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "all_read", 2024-03-11T21:27:58.287 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:27:55.852896+0000", 2024-03-11T21:27:58.287 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 2.6128000000000001e-05 2024-03-11T21:27:58.287 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:27:58.287 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:27:58.287 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "dispatched", 2024-03-11T21:27:58.287 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:27:55.912555+0000", 2024-03-11T21:27:58.288 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.059658911000000002 2024-03-11T21:27:58.288 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:27:58.288 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:27:58.288 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "queued_for_pg", 2024-03-11T21:27:58.288 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:27:55.912571+0000", 2024-03-11T21:27:58.288 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.5823000000000001e-05 2024-03-11T21:27:58.288 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:27:58.288 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:27:58.288 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-03-11T21:27:58.288 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:27:55.913606+0000", 2024-03-11T21:27:58.288 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.0010356339999999999 2024-03-11T21:27:58.288 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:27:58.288 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:27:58.288 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "started", 2024-03-11T21:27:58.288 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:27:55.913730+0000", 2024-03-11T21:27:58.289 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.000123869 2024-03-11T21:27:58.289 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:27:58.289 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:27:58.289 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "waiting for subops from 3,7", 2024-03-11T21:27:58.289 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:27:55.913860+0000", 2024-03-11T21:27:58.289 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.000129958 2024-03-11T21:27:58.289 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:27:58.289 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:27:58.289 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "op_commit", 2024-03-11T21:27:58.289 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:27:55.916093+0000", 2024-03-11T21:27:58.289 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.0022324699999999999 2024-03-11T21:27:58.289 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:27:58.289 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:27:58.289 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "sub_op_commit_rec", 2024-03-11T21:27:58.290 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:27:55.920095+0000", 2024-03-11T21:27:58.290 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.0040022929999999996 2024-03-11T21:27:58.290 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:27:58.290 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:27:58.290 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "sub_op_commit_rec", 2024-03-11T21:27:58.290 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:27:55.920129+0000", 2024-03-11T21:27:58.290 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 3.3813999999999998e-05 2024-03-11T21:27:58.290 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:27:58.290 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:27:58.290 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "commit_sent", 2024-03-11T21:27:58.290 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:27:55.920174+0000", 2024-03-11T21:27:58.290 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 4.4994000000000003e-05 2024-03-11T21:27:58.290 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:27:58.290 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:27:58.291 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "done", 2024-03-11T21:27:58.291 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:27:55.920210+0000", 2024-03-11T21:27:58.291 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 3.6355e-05 2024-03-11T21:27:58.291 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-03-11T21:27:58.291 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-03-11T21:27:58.291 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-03-11T21:27:58.291 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:27:58.291 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:27:58.291 INFO:teuthology.orchestra.run.smithi092.stdout: "description": "osd_op(client.34715.0:29198 7.4 7:2e7cbdf5:::benchmark_data_smithi092_76040_object29197:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e105)", 2024-03-11T21:27:58.291 INFO:teuthology.orchestra.run.smithi092.stdout: "initiated_at": "2024-03-11T21:27:56.337468+0000", 2024-03-11T21:27:58.291 INFO:teuthology.orchestra.run.smithi092.stdout: "age": 1.883027843, 2024-03-11T21:27:58.291 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.066704732000000003, 2024-03-11T21:27:58.291 INFO:teuthology.orchestra.run.smithi092.stdout: "type_data": { 2024-03-11T21:27:58.291 INFO:teuthology.orchestra.run.smithi092.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-11T21:27:58.292 INFO:teuthology.orchestra.run.smithi092.stdout: "client_info": { 2024-03-11T21:27:58.292 INFO:teuthology.orchestra.run.smithi092.stdout: "client": "client.34715", 2024-03-11T21:27:58.292 INFO:teuthology.orchestra.run.smithi092.stdout: "client_addr": "v1:172.21.15.92:0/2403132573", 2024-03-11T21:27:58.292 INFO:teuthology.orchestra.run.smithi092.stdout: "tid": 29198 2024-03-11T21:27:58.292 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:27:58.292 INFO:teuthology.orchestra.run.smithi092.stdout: "events": [ 2024-03-11T21:27:58.292 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:27:58.292 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "initiated", 2024-03-11T21:27:58.292 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:27:56.337468+0000", 2024-03-11T21:27:58.292 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-03-11T21:27:58.292 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:27:58.292 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:27:58.292 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "header_read", 2024-03-11T21:27:58.292 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:27:56.337468+0000", 2024-03-11T21:27:58.292 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-03-11T21:27:58.293 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:27:58.293 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:27:58.293 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "throttled", 2024-03-11T21:27:58.293 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:27:56.337470+0000", 2024-03-11T21:27:58.293 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.7489999999999999e-06 2024-03-11T21:27:58.293 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:27:58.293 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:27:58.293 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "all_read", 2024-03-11T21:27:58.293 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:27:56.337495+0000", 2024-03-11T21:27:58.293 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 2.4831999999999999e-05 2024-03-11T21:27:58.293 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:27:58.293 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:27:58.293 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "dispatched", 2024-03-11T21:27:58.294 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:27:56.337506+0000", 2024-03-11T21:27:58.294 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.1171e-05 2024-03-11T21:27:58.294 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:27:58.294 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:27:58.294 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "queued_for_pg", 2024-03-11T21:27:58.294 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:27:56.337514+0000", 2024-03-11T21:27:58.294 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 7.6990000000000003e-06 2024-03-11T21:27:58.294 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:27:58.294 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:27:58.294 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-03-11T21:27:58.294 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:27:56.337655+0000", 2024-03-11T21:27:58.294 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.0001416 2024-03-11T21:27:58.294 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:27:58.294 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:27:58.295 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "started", 2024-03-11T21:27:58.295 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:27:56.337780+0000", 2024-03-11T21:27:58.295 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00012506600000000001 2024-03-11T21:27:58.295 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:27:58.295 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:27:58.295 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "waiting for subops from 1,4", 2024-03-11T21:27:58.295 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:27:56.337914+0000", 2024-03-11T21:27:58.295 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00013343 2024-03-11T21:27:58.295 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:27:58.295 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:27:58.295 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "op_commit", 2024-03-11T21:27:58.295 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:27:56.339845+0000", 2024-03-11T21:27:58.295 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.0019314270000000001 2024-03-11T21:27:58.295 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:27:58.296 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:27:58.296 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "sub_op_commit_rec", 2024-03-11T21:27:58.296 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:27:56.404007+0000", 2024-03-11T21:27:58.296 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.064161433000000004 2024-03-11T21:27:58.296 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:27:58.296 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:27:58.296 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "sub_op_commit_rec", 2024-03-11T21:27:58.296 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:27:56.404091+0000", 2024-03-11T21:27:58.296 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 8.4553000000000002e-05 2024-03-11T21:27:58.296 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:27:58.296 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:27:58.296 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "commit_sent", 2024-03-11T21:27:58.296 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:27:56.404136+0000", 2024-03-11T21:27:58.296 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 4.464e-05 2024-03-11T21:27:58.297 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:27:58.297 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:27:58.297 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "done", 2024-03-11T21:27:58.297 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:27:56.404173+0000", 2024-03-11T21:27:58.297 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 3.7132000000000003e-05 2024-03-11T21:27:58.297 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-03-11T21:27:58.297 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-03-11T21:27:58.297 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-03-11T21:27:58.297 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:27:58.297 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:27:58.297 INFO:teuthology.orchestra.run.smithi092.stdout: "description": "osd_op(client.34715.0:29211 7.4 7:2134c779:::benchmark_data_smithi092_76040_object29210:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e105)", 2024-03-11T21:27:58.297 INFO:teuthology.orchestra.run.smithi092.stdout: "initiated_at": "2024-03-11T21:27:56.341578+0000", 2024-03-11T21:27:58.297 INFO:teuthology.orchestra.run.smithi092.stdout: "age": 1.878918487, 2024-03-11T21:27:58.297 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.069955965999999994, 2024-03-11T21:27:58.297 INFO:teuthology.orchestra.run.smithi092.stdout: "type_data": { 2024-03-11T21:27:58.298 INFO:teuthology.orchestra.run.smithi092.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-11T21:27:58.298 INFO:teuthology.orchestra.run.smithi092.stdout: "client_info": { 2024-03-11T21:27:58.298 INFO:teuthology.orchestra.run.smithi092.stdout: "client": "client.34715", 2024-03-11T21:27:58.298 INFO:teuthology.orchestra.run.smithi092.stdout: "client_addr": "v1:172.21.15.92:0/2403132573", 2024-03-11T21:27:58.298 INFO:teuthology.orchestra.run.smithi092.stdout: "tid": 29211 2024-03-11T21:27:58.298 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:27:58.298 INFO:teuthology.orchestra.run.smithi092.stdout: "events": [ 2024-03-11T21:27:58.298 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:27:58.298 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "initiated", 2024-03-11T21:27:58.298 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:27:56.341578+0000", 2024-03-11T21:27:58.298 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-03-11T21:27:58.298 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:27:58.298 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:27:58.298 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "header_read", 2024-03-11T21:27:58.299 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:27:56.341578+0000", 2024-03-11T21:27:58.299 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-03-11T21:27:58.299 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:27:58.299 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:27:58.299 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "throttled", 2024-03-11T21:27:58.299 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:27:56.341579+0000", 2024-03-11T21:27:58.299 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.2279999999999999e-06 2024-03-11T21:27:58.299 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:27:58.299 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:27:58.299 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "all_read", 2024-03-11T21:27:58.299 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:27:56.341601+0000", 2024-03-11T21:27:58.299 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 2.247e-05 2024-03-11T21:27:58.299 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:27:58.299 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:27:58.300 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "dispatched", 2024-03-11T21:27:58.300 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:27:56.341612+0000", 2024-03-11T21:27:58.300 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.0326000000000001e-05 2024-03-11T21:27:58.300 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:27:58.300 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:27:58.300 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "queued_for_pg", 2024-03-11T21:27:58.300 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:27:56.341619+0000", 2024-03-11T21:27:58.300 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 7.3499999999999999e-06 2024-03-11T21:27:58.300 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:27:58.300 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:27:58.300 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-03-11T21:27:58.300 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:27:56.404494+0000", 2024-03-11T21:27:58.300 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.062874722999999993 2024-03-11T21:27:58.300 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:27:58.300 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:27:58.301 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "started", 2024-03-11T21:27:58.301 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:27:56.404622+0000", 2024-03-11T21:27:58.301 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00012780300000000001 2024-03-11T21:27:58.301 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:27:58.301 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:27:58.301 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "waiting for subops from 1,4", 2024-03-11T21:27:58.301 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:27:56.404813+0000", 2024-03-11T21:27:58.301 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.000191841 2024-03-11T21:27:58.301 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:27:58.301 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:27:58.301 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "sub_op_commit_rec", 2024-03-11T21:27:58.301 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:27:56.407075+0000", 2024-03-11T21:27:58.301 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.0022612169999999998 2024-03-11T21:27:58.301 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:27:58.301 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:27:58.302 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "sub_op_commit_rec", 2024-03-11T21:27:58.302 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:27:56.407108+0000", 2024-03-11T21:27:58.302 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 3.3219999999999997e-05 2024-03-11T21:27:58.302 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:27:58.302 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:27:58.302 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "op_commit", 2024-03-11T21:27:58.303 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:27:56.411436+0000", 2024-03-11T21:27:58.303 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.0043280669999999997 2024-03-11T21:27:58.303 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:27:58.303 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:27:58.303 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "commit_sent", 2024-03-11T21:27:58.303 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:27:56.411489+0000", 2024-03-11T21:27:58.303 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 5.2933999999999998e-05 2024-03-11T21:27:58.304 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:27:58.304 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:27:58.304 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "done", 2024-03-11T21:27:58.304 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:27:56.411534+0000", 2024-03-11T21:27:58.304 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 4.4787000000000003e-05 2024-03-11T21:27:58.304 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-03-11T21:27:58.304 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-03-11T21:27:58.304 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-03-11T21:27:58.304 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:27:58.304 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:27:58.304 INFO:teuthology.orchestra.run.smithi092.stdout: "description": "osd_op(client.34715.0:32053 7.f 7:f9396c4c:::benchmark_data_smithi092_76040_object32052:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e105)", 2024-03-11T21:27:58.304 INFO:teuthology.orchestra.run.smithi092.stdout: "initiated_at": "2024-03-11T21:27:57.785258+0000", 2024-03-11T21:27:58.304 INFO:teuthology.orchestra.run.smithi092.stdout: "age": 0.43523793300000002, 2024-03-11T21:27:58.305 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.083056137000000002, 2024-03-11T21:27:58.305 INFO:teuthology.orchestra.run.smithi092.stdout: "type_data": { 2024-03-11T21:27:58.305 INFO:teuthology.orchestra.run.smithi092.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-11T21:27:58.305 INFO:teuthology.orchestra.run.smithi092.stdout: "client_info": { 2024-03-11T21:27:58.305 INFO:teuthology.orchestra.run.smithi092.stdout: "client": "client.34715", 2024-03-11T21:27:58.305 INFO:teuthology.orchestra.run.smithi092.stdout: "client_addr": "v1:172.21.15.92:0/2403132573", 2024-03-11T21:27:58.305 INFO:teuthology.orchestra.run.smithi092.stdout: "tid": 32053 2024-03-11T21:27:58.305 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:27:58.305 INFO:teuthology.orchestra.run.smithi092.stdout: "events": [ 2024-03-11T21:27:58.305 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:27:58.305 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "initiated", 2024-03-11T21:27:58.305 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:27:57.785258+0000", 2024-03-11T21:27:58.305 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-03-11T21:27:58.306 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:27:58.306 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:27:58.306 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "header_read", 2024-03-11T21:27:58.306 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:27:57.785258+0000", 2024-03-11T21:27:58.306 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-03-11T21:27:58.306 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:27:58.306 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:27:58.306 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "throttled", 2024-03-11T21:27:58.306 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:27:57.785301+0000", 2024-03-11T21:27:58.306 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 4.2889999999999998e-05 2024-03-11T21:27:58.306 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:27:58.306 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:27:58.306 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "all_read", 2024-03-11T21:27:58.306 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:27:57.785330+0000", 2024-03-11T21:27:58.307 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 2.8660999999999999e-05 2024-03-11T21:27:58.307 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:27:58.307 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:27:58.307 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "dispatched", 2024-03-11T21:27:58.307 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:27:57.785346+0000", 2024-03-11T21:27:58.307 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.6042e-05 2024-03-11T21:27:58.307 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:27:58.307 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:27:58.307 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "queued_for_pg", 2024-03-11T21:27:58.307 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:27:57.785365+0000", 2024-03-11T21:27:58.307 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.9508999999999999e-05 2024-03-11T21:27:58.307 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:27:58.307 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:27:58.307 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-03-11T21:27:58.308 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:27:57.785434+0000", 2024-03-11T21:27:58.308 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 6.8664999999999994e-05 2024-03-11T21:27:58.308 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:27:58.308 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:27:58.308 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "started", 2024-03-11T21:27:58.308 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:27:57.785572+0000", 2024-03-11T21:27:58.308 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00013767900000000001 2024-03-11T21:27:58.308 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:27:58.308 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:27:58.308 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "waiting for subops from 3,7", 2024-03-11T21:27:58.308 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:27:57.785702+0000", 2024-03-11T21:27:58.308 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.000130405 2024-03-11T21:27:58.308 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:27:58.308 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:27:58.309 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "sub_op_commit_rec", 2024-03-11T21:27:58.309 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:27:57.788559+0000", 2024-03-11T21:27:58.309 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.0028570420000000002 2024-03-11T21:27:58.309 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:27:58.309 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:27:58.309 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "op_commit", 2024-03-11T21:27:58.309 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:27:57.790923+0000", 2024-03-11T21:27:58.309 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.002363768 2024-03-11T21:27:58.309 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:27:58.309 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:27:58.309 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "sub_op_commit_rec", 2024-03-11T21:27:58.309 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:27:57.868229+0000", 2024-03-11T21:27:58.309 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.077306484999999994 2024-03-11T21:27:58.309 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:27:58.309 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:27:58.310 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "commit_sent", 2024-03-11T21:27:58.310 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:27:57.868279+0000", 2024-03-11T21:27:58.310 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 4.9231999999999999e-05 2024-03-11T21:27:58.310 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:27:58.310 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:27:58.310 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "done", 2024-03-11T21:27:58.310 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:27:57.868314+0000", 2024-03-11T21:27:58.310 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 3.5759000000000001e-05 2024-03-11T21:27:58.310 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-03-11T21:27:58.310 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-03-11T21:27:58.310 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-03-11T21:27:58.310 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:27:58.310 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:27:58.311 INFO:teuthology.orchestra.run.smithi092.stdout: "description": "osd_op(client.34715.0:32056 7.9 7:94ee601a:::benchmark_data_smithi092_76040_object32055:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e105)", 2024-03-11T21:27:58.311 INFO:teuthology.orchestra.run.smithi092.stdout: "initiated_at": "2024-03-11T21:27:57.785869+0000", 2024-03-11T21:27:58.311 INFO:teuthology.orchestra.run.smithi092.stdout: "age": 0.43462738000000001, 2024-03-11T21:27:58.311 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.077940068000000001, 2024-03-11T21:27:58.311 INFO:teuthology.orchestra.run.smithi092.stdout: "type_data": { 2024-03-11T21:27:58.311 INFO:teuthology.orchestra.run.smithi092.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-11T21:27:58.311 INFO:teuthology.orchestra.run.smithi092.stdout: "client_info": { 2024-03-11T21:27:58.311 INFO:teuthology.orchestra.run.smithi092.stdout: "client": "client.34715", 2024-03-11T21:27:58.311 INFO:teuthology.orchestra.run.smithi092.stdout: "client_addr": "v1:172.21.15.92:0/2403132573", 2024-03-11T21:27:58.311 INFO:teuthology.orchestra.run.smithi092.stdout: "tid": 32056 2024-03-11T21:27:58.311 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:27:58.311 INFO:teuthology.orchestra.run.smithi092.stdout: "events": [ 2024-03-11T21:27:58.311 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:27:58.311 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "initiated", 2024-03-11T21:27:58.312 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:27:57.785869+0000", 2024-03-11T21:27:58.312 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-03-11T21:27:58.312 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:27:58.312 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:27:58.312 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "header_read", 2024-03-11T21:27:58.312 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:27:57.785869+0000", 2024-03-11T21:27:58.312 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-03-11T21:27:58.312 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:27:58.312 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:27:58.312 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "throttled", 2024-03-11T21:27:58.312 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:27:57.785870+0000", 2024-03-11T21:27:58.312 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.122e-06 2024-03-11T21:27:58.312 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:27:58.312 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:27:58.313 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "all_read", 2024-03-11T21:27:58.313 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:27:57.785893+0000", 2024-03-11T21:27:58.313 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 2.2891000000000002e-05 2024-03-11T21:27:58.313 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:27:58.313 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:27:58.313 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "dispatched", 2024-03-11T21:27:58.313 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:27:57.785901+0000", 2024-03-11T21:27:58.313 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 8.2889999999999998e-06 2024-03-11T21:27:58.313 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:27:58.313 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:27:58.313 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "queued_for_pg", 2024-03-11T21:27:58.313 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:27:57.785908+0000", 2024-03-11T21:27:58.313 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 7.2520000000000002e-06 2024-03-11T21:27:58.313 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:27:58.313 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:27:58.314 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-03-11T21:27:58.314 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:27:57.786090+0000", 2024-03-11T21:27:58.314 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.000182041 2024-03-11T21:27:58.314 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:27:58.314 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:27:58.314 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "started", 2024-03-11T21:27:58.314 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:27:57.786327+0000", 2024-03-11T21:27:58.314 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.000236393 2024-03-11T21:27:58.314 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:27:58.314 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:27:58.314 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "waiting for subops from 1,4", 2024-03-11T21:27:58.314 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:27:57.786453+0000", 2024-03-11T21:27:58.314 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.000125854 2024-03-11T21:27:58.314 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:27:58.315 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:27:58.315 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "op_commit", 2024-03-11T21:27:58.315 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:27:57.787887+0000", 2024-03-11T21:27:58.315 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.0014339470000000001 2024-03-11T21:27:58.315 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:27:58.315 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:27:58.315 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "sub_op_commit_rec", 2024-03-11T21:27:58.315 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:27:57.789971+0000", 2024-03-11T21:27:58.315 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.002084299 2024-03-11T21:27:58.315 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:27:58.315 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:27:58.315 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "sub_op_commit_rec", 2024-03-11T21:27:58.315 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:27:57.863638+0000", 2024-03-11T21:27:58.315 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.073667344999999995 2024-03-11T21:27:58.316 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:27:58.316 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:27:58.316 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "commit_sent", 2024-03-11T21:27:58.316 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:27:57.863730+0000", 2024-03-11T21:27:58.316 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 9.2181000000000004e-05 2024-03-11T21:27:58.316 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:27:58.316 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:27:58.316 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "done", 2024-03-11T21:27:58.316 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:27:57.863809+0000", 2024-03-11T21:27:58.316 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 7.8454000000000005e-05 2024-03-11T21:27:58.316 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-03-11T21:27:58.316 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-03-11T21:27:58.316 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-03-11T21:27:58.316 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:27:58.316 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:27:58.317 INFO:teuthology.orchestra.run.smithi092.stdout: "description": "osd_op(client.34715.0:32060 7.6 7:681f1ffd:::benchmark_data_smithi092_76040_object32059:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e105)", 2024-03-11T21:27:58.317 INFO:teuthology.orchestra.run.smithi092.stdout: "initiated_at": "2024-03-11T21:27:57.787807+0000", 2024-03-11T21:27:58.317 INFO:teuthology.orchestra.run.smithi092.stdout: "age": 0.432688713, 2024-03-11T21:27:58.317 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.080338912999999998, 2024-03-11T21:27:58.317 INFO:teuthology.orchestra.run.smithi092.stdout: "type_data": { 2024-03-11T21:27:58.317 INFO:teuthology.orchestra.run.smithi092.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-11T21:27:58.317 INFO:teuthology.orchestra.run.smithi092.stdout: "client_info": { 2024-03-11T21:27:58.317 INFO:teuthology.orchestra.run.smithi092.stdout: "client": "client.34715", 2024-03-11T21:27:58.317 INFO:teuthology.orchestra.run.smithi092.stdout: "client_addr": "v1:172.21.15.92:0/2403132573", 2024-03-11T21:27:58.317 INFO:teuthology.orchestra.run.smithi092.stdout: "tid": 32060 2024-03-11T21:27:58.317 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:27:58.317 INFO:teuthology.orchestra.run.smithi092.stdout: "events": [ 2024-03-11T21:27:58.317 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:27:58.318 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "initiated", 2024-03-11T21:27:58.318 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:27:57.787807+0000", 2024-03-11T21:27:58.318 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-03-11T21:27:58.318 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:27:58.318 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:27:58.318 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "header_read", 2024-03-11T21:27:58.318 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:27:57.787807+0000", 2024-03-11T21:27:58.318 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-03-11T21:27:58.318 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:27:58.318 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:27:58.318 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "throttled", 2024-03-11T21:27:58.318 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:27:57.787814+0000", 2024-03-11T21:27:58.318 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 6.9600000000000003e-06 2024-03-11T21:27:58.318 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:27:58.319 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:27:58.319 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "all_read", 2024-03-11T21:27:58.319 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:27:57.787846+0000", 2024-03-11T21:27:58.319 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 3.1222e-05 2024-03-11T21:27:58.319 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:27:58.319 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:27:58.319 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "dispatched", 2024-03-11T21:27:58.319 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:27:57.787860+0000", 2024-03-11T21:27:58.319 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.4837999999999999e-05 2024-03-11T21:27:58.319 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:27:58.319 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:27:58.319 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "queued_for_pg", 2024-03-11T21:27:58.319 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:27:57.787869+0000", 2024-03-11T21:27:58.319 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 8.9749999999999996e-06 2024-03-11T21:27:58.320 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:27:58.320 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:27:58.320 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-03-11T21:27:58.320 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:27:57.787941+0000", 2024-03-11T21:27:58.320 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 7.1749999999999996e-05 2024-03-11T21:27:58.320 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:27:58.320 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:27:58.320 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "started", 2024-03-11T21:27:58.320 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:27:57.788101+0000", 2024-03-11T21:27:58.320 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00016001599999999999 2024-03-11T21:27:58.320 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:27:58.320 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:27:58.320 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "waiting for subops from 3,7", 2024-03-11T21:27:58.320 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:27:57.788237+0000", 2024-03-11T21:27:58.321 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00013557900000000001 2024-03-11T21:27:58.321 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:27:58.321 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:27:58.321 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "op_commit", 2024-03-11T21:27:58.321 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:27:57.790441+0000", 2024-03-11T21:27:58.321 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.0022043340000000001 2024-03-11T21:27:58.321 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:27:58.321 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:27:58.321 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "sub_op_commit_rec", 2024-03-11T21:27:58.321 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:27:57.791119+0000", 2024-03-11T21:27:58.321 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00067825199999999996 2024-03-11T21:27:58.321 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:27:58.321 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:27:58.321 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "sub_op_commit_rec", 2024-03-11T21:27:58.322 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:27:57.868030+0000", 2024-03-11T21:27:58.322 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.076911101999999995 2024-03-11T21:27:58.322 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:27:58.322 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:27:58.322 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "commit_sent", 2024-03-11T21:27:58.322 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:27:57.868088+0000", 2024-03-11T21:27:58.322 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 5.8047000000000002e-05 2024-03-11T21:27:58.322 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:27:58.322 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:27:58.322 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "done", 2024-03-11T21:27:58.322 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:27:57.868146+0000", 2024-03-11T21:27:58.322 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 5.7837999999999997e-05 2024-03-11T21:27:58.322 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-03-11T21:27:58.323 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-03-11T21:27:58.323 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-03-11T21:27:58.323 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:27:58.323 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:27:58.323 INFO:teuthology.orchestra.run.smithi092.stdout: "description": "osd_op(client.34715.0:32062 7.f 7:fbc35423:::benchmark_data_smithi092_76040_object32061:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e105)", 2024-03-11T21:27:58.323 INFO:teuthology.orchestra.run.smithi092.stdout: "initiated_at": "2024-03-11T21:27:57.788554+0000", 2024-03-11T21:27:58.323 INFO:teuthology.orchestra.run.smithi092.stdout: "age": 0.43194177499999997, 2024-03-11T21:27:58.323 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.079885136999999995, 2024-03-11T21:27:58.323 INFO:teuthology.orchestra.run.smithi092.stdout: "type_data": { 2024-03-11T21:27:58.323 INFO:teuthology.orchestra.run.smithi092.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-11T21:27:58.323 INFO:teuthology.orchestra.run.smithi092.stdout: "client_info": { 2024-03-11T21:27:58.323 INFO:teuthology.orchestra.run.smithi092.stdout: "client": "client.34715", 2024-03-11T21:27:58.323 INFO:teuthology.orchestra.run.smithi092.stdout: "client_addr": "v1:172.21.15.92:0/2403132573", 2024-03-11T21:27:58.323 INFO:teuthology.orchestra.run.smithi092.stdout: "tid": 32062 2024-03-11T21:27:58.324 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:27:58.324 INFO:teuthology.orchestra.run.smithi092.stdout: "events": [ 2024-03-11T21:27:58.324 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:27:58.324 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "initiated", 2024-03-11T21:27:58.324 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:27:57.788554+0000", 2024-03-11T21:27:58.324 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-03-11T21:27:58.324 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:27:58.324 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:27:58.324 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "header_read", 2024-03-11T21:27:58.324 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:27:57.788554+0000", 2024-03-11T21:27:58.324 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-03-11T21:27:58.324 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:27:58.324 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:27:58.324 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "throttled", 2024-03-11T21:27:58.324 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:27:57.788555+0000", 2024-03-11T21:27:58.325 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.1620000000000001e-06 2024-03-11T21:27:58.325 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:27:58.325 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:27:58.325 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "all_read", 2024-03-11T21:27:58.325 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:27:57.788593+0000", 2024-03-11T21:27:58.325 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 3.7296999999999999e-05 2024-03-11T21:27:58.325 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:27:58.325 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:27:58.325 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "dispatched", 2024-03-11T21:27:58.325 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:27:57.788601+0000", 2024-03-11T21:27:58.325 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 8.0120000000000005e-06 2024-03-11T21:27:58.325 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:27:58.325 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:27:58.325 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "queued_for_pg", 2024-03-11T21:27:58.326 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:27:57.788608+0000", 2024-03-11T21:27:58.326 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 7.311e-06 2024-03-11T21:27:58.326 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:27:58.326 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:27:58.326 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-03-11T21:27:58.326 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:27:57.788643+0000", 2024-03-11T21:27:58.326 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 3.4978999999999998e-05 2024-03-11T21:27:58.326 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:27:58.326 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:27:58.326 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "started", 2024-03-11T21:27:58.326 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:27:57.788757+0000", 2024-03-11T21:27:58.326 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.000113545 2024-03-11T21:27:58.326 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:27:58.326 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:27:58.326 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "waiting for subops from 3,7", 2024-03-11T21:27:58.327 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:27:57.788881+0000", 2024-03-11T21:27:58.327 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00012469900000000001 2024-03-11T21:27:58.327 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:27:58.327 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:27:58.327 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "op_commit", 2024-03-11T21:27:58.327 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:27:57.791014+0000", 2024-03-11T21:27:58.327 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.0021331190000000002 2024-03-11T21:27:58.327 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:27:58.327 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:27:58.327 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "sub_op_commit_rec", 2024-03-11T21:27:58.327 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:27:57.791519+0000", 2024-03-11T21:27:58.327 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00050470500000000004 2024-03-11T21:27:58.327 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:27:58.327 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:27:58.327 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "sub_op_commit_rec", 2024-03-11T21:27:58.328 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:27:57.868350+0000", 2024-03-11T21:27:58.328 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.076831224000000004 2024-03-11T21:27:58.328 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:27:58.328 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:27:58.328 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "commit_sent", 2024-03-11T21:27:58.328 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:27:57.868401+0000", 2024-03-11T21:27:58.328 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 5.1005999999999999e-05 2024-03-11T21:27:58.328 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:27:58.328 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:27:58.328 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "done", 2024-03-11T21:27:58.328 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:27:57.868439+0000", 2024-03-11T21:27:58.328 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 3.8077999999999997e-05 2024-03-11T21:27:58.328 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-03-11T21:27:58.328 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-03-11T21:27:58.329 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-03-11T21:27:58.329 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-03-11T21:27:58.329 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-03-11T21:27:58.329 INFO:teuthology.orchestra.run.smithi092.stdout:} 2024-03-11T21:27:58.332 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:27:58.333 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:27:58.333 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:27:58.434 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:27:58.464 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:27:58.465 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:27:58.465 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:27:58.566 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:27:58.600 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:27:58.601 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:27:58.601 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:27:58.620 INFO:tasks.radosbench.radosbench.0.smithi092.stdout: 14 16 33537 33521 149.629 112.562 0.00418601 0.00667771 2024-03-11T21:27:58.702 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:27:58.738 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:27:58.738 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:27:58.739 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:27:58.840 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:27:58.870 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:27:58.871 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:27:58.871 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:27:58.972 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:27:59.013 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:27:59.014 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:27:59.014 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:27:59.076 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:27:58 smithi097 ceph-mon[33317]: pgmap v69: 148 pgs: 148 active+clean; 1.7 GiB data, 5.3 GiB used, 710 GiB / 715 GiB avail; 1.2 KiB/s rd, 149 MiB/s wr, 2.38k op/s 2024-03-11T21:27:59.111 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:27:58 smithi092 ceph-mon[29572]: pgmap v69: 148 pgs: 148 active+clean; 1.7 GiB data, 5.3 GiB used, 710 GiB / 715 GiB avail; 1.2 KiB/s rd, 149 MiB/s wr, 2.38k op/s 2024-03-11T21:27:59.111 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:27:58 smithi092 ceph-mon[37563]: pgmap v69: 148 pgs: 148 active+clean; 1.7 GiB data, 5.3 GiB used, 710 GiB / 715 GiB avail; 1.2 KiB/s rd, 149 MiB/s wr, 2.38k op/s 2024-03-11T21:27:59.115 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:27:59.146 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:27:59.147 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:27:59.147 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:27:59.248 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:27:59.279 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:27:59.279 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:27:59.280 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:27:59.381 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:27:59.420 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:27:59.420 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:27:59.421 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:27:59.521 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:27:59.552 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:27:59.552 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:27:59.553 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:27:59.622 INFO:tasks.radosbench.radosbench.0.smithi092.stdout: 15 16 35517 35501 147.903 123.75 0.00624371 0.00675575 2024-03-11T21:27:59.653 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:27:59.695 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:27:59.695 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:27:59.695 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:27:59.796 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:27:59.835 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:27:59.836 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:27:59.836 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:27:59.937 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:27:59.985 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:27:59.986 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:27:59.986 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:28:00.087 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:28:00.119 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:28:00.120 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:28:00.120 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:28:00.221 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:28:00.254 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:28:00.254 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:28:00.255 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:28:00.355 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:28:00.385 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:28:00.386 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:28:00.386 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:28:00.482 DEBUG:teuthology.orchestra.run.smithi092:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid 00db0016-dfed-11ee-95c8-87774f69a715 -- ceph daemon osd.1 dump_ops_in_flight 2024-03-11T21:28:00.487 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:28:00.538 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:28:00.539 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:28:00.540 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:28:00.621 INFO:tasks.radosbench.radosbench.0.smithi092.stdout: 16 16 37424 37408 146.107 119.188 0.00533825 0.00683974 2024-03-11T21:28:00.641 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:28:00.690 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:28:00.691 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:28:00.691 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:28:00.792 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:28:00.819 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:28:00.820 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:28:00.820 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:28:00.921 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:28:00.952 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:28:00.953 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:28:00.953 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:28:00.992 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:28:00 smithi092 ceph-mon[29572]: pgmap v70: 148 pgs: 148 active+clean; 1.9 GiB data, 5.7 GiB used, 710 GiB / 715 GiB avail; 853 B/s rd, 120 MiB/s wr, 1.92k op/s 2024-03-11T21:28:00.992 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:28:00 smithi092 ceph-mon[37563]: pgmap v70: 148 pgs: 148 active+clean; 1.9 GiB data, 5.7 GiB used, 710 GiB / 715 GiB avail; 853 B/s rd, 120 MiB/s wr, 1.92k op/s 2024-03-11T21:28:01.000 INFO:teuthology.orchestra.run.smithi092.stderr:Inferring config /var/lib/ceph/00db0016-dfed-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T21:28:01.054 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:28:01.076 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:28:00 smithi097 ceph-mon[33317]: pgmap v70: 148 pgs: 148 active+clean; 1.9 GiB data, 5.7 GiB used, 710 GiB / 715 GiB avail; 853 B/s rd, 120 MiB/s wr, 1.92k op/s 2024-03-11T21:28:01.112 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:28:01.112 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:28:01.113 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:28:01.214 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:28:01.241 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:28:01.241 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:28:01.242 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:28:01.343 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:28:01.371 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:28:01.371 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:28:01.371 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:28:01.472 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:28:01.515 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:28:01.516 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:28:01.516 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:28:01.577 INFO:journalctl@ceph.node-exporter.b.smithi097.stdout:Mar 11 21:28:01 smithi097 bash[66565]: Getting image source signatures 2024-03-11T21:28:01.577 INFO:journalctl@ceph.node-exporter.b.smithi097.stdout:Mar 11 21:28:01 smithi097 bash[66565]: Copying blob sha256:c0e87333d3805a7b7e1c25783dd25f27da0fe70a8e262eee7ed190c8dc843fa7 2024-03-11T21:28:01.577 INFO:journalctl@ceph.node-exporter.b.smithi097.stdout:Mar 11 21:28:01 smithi097 bash[66565]: Copying blob sha256:22b70bddd3acadc892fca4c2af4260629bfda5dfd11ebc106a93ce24e752b5ed 2024-03-11T21:28:01.577 INFO:journalctl@ceph.node-exporter.b.smithi097.stdout:Mar 11 21:28:01 smithi097 bash[66565]: Copying blob sha256:5c12815fee558b157dee7f7509dedbaba0a8379098858a65ec869e1f1526ea0c 2024-03-11T21:28:01.617 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:28:01.620 INFO:tasks.radosbench.radosbench.0.smithi092.stdout: 17 16 38886 38870 142.888 91.375 0.00892929 0.0069929 2024-03-11T21:28:01.657 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:28:01.658 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:28:01.658 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:28:01.759 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:28:01.810 INFO:teuthology.orchestra.run.smithi092.stderr:noscrub is unset 2024-03-11T21:28:01.827 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:28:01.830 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:28:01.830 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:28:01.861 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:28:01 smithi092 ceph-mon[29572]: from='client.? v1:172.21.15.92:0/1301513073' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-11T21:28:01.861 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:28:01 smithi092 ceph-mon[37563]: from='client.? v1:172.21.15.92:0/1301513073' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-11T21:28:01.932 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:28:01.962 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:28:01.963 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:28:01.963 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:28:02.064 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:28:02.076 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:28:01 smithi097 ceph-mon[33317]: from='client.? v1:172.21.15.92:0/1301513073' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-11T21:28:02.077 INFO:journalctl@ceph.iscsi.iscsi.a.smithi097.stdout:Mar 11 21:28:02 smithi097 ceph-00db0016-dfed-11ee-95c8-87774f69a715-iscsi-iscsi-a[60943]: debug there is no tcmu-runner data available 2024-03-11T21:28:02.103 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:28:02.104 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:28:02.104 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:28:02.205 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:28:02.235 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:28:02.236 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:28:02.236 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:28:02.337 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:28:02.371 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:28:02.372 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:28:02.372 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:28:02.473 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:28:02.511 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:28:02.511 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:28:02.512 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:28:02.612 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:28:02.622 INFO:tasks.radosbench.radosbench.0.smithi092.stdout: 18 16 40472 40456 140.453 99.125 0.00607487 0.00711237 2024-03-11T21:28:02.658 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:28:02.658 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:28:02.659 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:28:02.759 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:28:02.788 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:28:02.789 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:28:02.789 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:28:02.861 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:28:02 smithi092 ceph-mon[29572]: pgmap v71: 148 pgs: 148 active+clean; 1.9 GiB data, 5.7 GiB used, 710 GiB / 715 GiB avail; 853 B/s rd, 120 MiB/s wr, 1.92k op/s 2024-03-11T21:28:02.861 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:28:02 smithi092 ceph-mon[29572]: purged_snaps scrub starts 2024-03-11T21:28:02.862 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:28:02 smithi092 ceph-mon[29572]: purged_snaps scrub ok 2024-03-11T21:28:02.862 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:28:02 smithi092 ceph-mon[29572]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-11T21:28:02.862 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:28:02 smithi092 ceph-mon[29572]: from='client.? v1:172.21.15.92:0/1301513073' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-03-11T21:28:02.862 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:28:02 smithi092 ceph-mon[29572]: osdmap e106: 8 total, 8 up, 8 in 2024-03-11T21:28:02.862 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:28:02 smithi092 ceph-mon[37563]: pgmap v71: 148 pgs: 148 active+clean; 1.9 GiB data, 5.7 GiB used, 710 GiB / 715 GiB avail; 853 B/s rd, 120 MiB/s wr, 1.92k op/s 2024-03-11T21:28:02.862 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:28:02 smithi092 ceph-mon[37563]: purged_snaps scrub starts 2024-03-11T21:28:02.862 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:28:02 smithi092 ceph-mon[37563]: purged_snaps scrub ok 2024-03-11T21:28:02.862 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:28:02 smithi092 ceph-mon[37563]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-11T21:28:02.862 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:28:02 smithi092 ceph-mon[37563]: from='client.? v1:172.21.15.92:0/1301513073' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-03-11T21:28:02.862 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:28:02 smithi092 ceph-mon[37563]: osdmap e106: 8 total, 8 up, 8 in 2024-03-11T21:28:02.890 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:28:02.918 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5, 6, 7] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7] 2024-03-11T21:28:02.919 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-03-11T21:28:02.919 INFO:tasks.thrashosds.thrasher:thrash_pg_upmap 2024-03-11T21:28:02.919 DEBUG:teuthology.orchestra.run.smithi092:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid 00db0016-dfed-11ee-95c8-87774f69a715 -- ceph osd dump -f json-pretty 2024-03-11T21:28:02.920 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:28:02.920 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:28:02.921 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:28:03.021 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:28:03.047 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:28:03.048 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:28:03.048 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:28:03.076 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:28:02 smithi097 ceph-mon[33317]: pgmap v71: 148 pgs: 148 active+clean; 1.9 GiB data, 5.7 GiB used, 710 GiB / 715 GiB avail; 853 B/s rd, 120 MiB/s wr, 1.92k op/s 2024-03-11T21:28:03.076 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:28:02 smithi097 ceph-mon[33317]: purged_snaps scrub starts 2024-03-11T21:28:03.077 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:28:02 smithi097 ceph-mon[33317]: purged_snaps scrub ok 2024-03-11T21:28:03.077 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:28:02 smithi097 ceph-mon[33317]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-11T21:28:03.077 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:28:02 smithi097 ceph-mon[33317]: from='client.? v1:172.21.15.92:0/1301513073' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-03-11T21:28:03.077 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:28:02 smithi097 ceph-mon[33317]: osdmap e106: 8 total, 8 up, 8 in 2024-03-11T21:28:03.149 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:28:03.179 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:28:03.179 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:28:03.180 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:28:03.281 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:28:03.309 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:28:03.309 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:28:03.310 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:28:03.410 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:28:03.461 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:28:03.461 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:28:03.462 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:28:03.562 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:28:03.591 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:28:03.592 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:28:03.592 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:28:03.611 INFO:teuthology.orchestra.run.smithi092.stderr:Inferring config /var/lib/ceph/00db0016-dfed-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T21:28:03.621 INFO:tasks.radosbench.radosbench.0.smithi092.stdout: 19 16 42002 41986 138.088 95.625 0.0279474 0.00723584 2024-03-11T21:28:03.685 INFO:teuthology.orchestra.run.smithi092.stdout:osd.0: {} 2024-03-11T21:28:03.693 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:28:03.723 INFO:teuthology.orchestra.run.smithi092.stdout:osd.1: {} 2024-03-11T21:28:03.745 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:28:03.746 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:28:03.746 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:28:03.764 INFO:teuthology.orchestra.run.smithi092.stdout:osd.2: {} 2024-03-11T21:28:03.803 INFO:teuthology.orchestra.run.smithi092.stdout:osd.3: {} 2024-03-11T21:28:03.842 INFO:teuthology.orchestra.run.smithi092.stdout:osd.4: {} 2024-03-11T21:28:03.848 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:28:03.875 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:28:03.876 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:28:03.876 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:28:03.881 INFO:teuthology.orchestra.run.smithi092.stdout:osd.5: {} 2024-03-11T21:28:03.921 INFO:teuthology.orchestra.run.smithi092.stdout:osd.6: {} 2024-03-11T21:28:03.940 INFO:teuthology.orchestra.run.smithi092.stderr:Inferring config /var/lib/ceph/00db0016-dfed-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T21:28:03.965 INFO:teuthology.orchestra.run.smithi092.stdout:osd.7: {} 2024-03-11T21:28:03.976 INFO:teuthology.orchestra.run.smithi092.stderr:osd.0: osd_enable_op_tracker = '' 2024-03-11T21:28:03.976 INFO:teuthology.orchestra.run.smithi092.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-11T21:28:03.976 INFO:teuthology.orchestra.run.smithi092.stderr:osd.2: osd_enable_op_tracker = '' 2024-03-11T21:28:03.976 INFO:teuthology.orchestra.run.smithi092.stderr:osd.3: osd_enable_op_tracker = '' 2024-03-11T21:28:03.976 INFO:teuthology.orchestra.run.smithi092.stderr:osd.4: osd_enable_op_tracker = '' 2024-03-11T21:28:03.976 INFO:teuthology.orchestra.run.smithi092.stderr:osd.5: osd_enable_op_tracker = '' 2024-03-11T21:28:03.976 INFO:teuthology.orchestra.run.smithi092.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-11T21:28:03.976 INFO:teuthology.orchestra.run.smithi092.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-11T21:28:03.978 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:28:04.006 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:28:04.007 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:28:04.007 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:28:04.077 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:28:03 smithi097 ceph-mon[33317]: from='client.24469 v1:172.21.15.97:0/404208310' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:28:04.077 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:28:03 smithi097 ceph-mon[33317]: 6.c scrub starts 2024-03-11T21:28:04.077 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:28:03 smithi097 ceph-mon[33317]: 6.c scrub ok 2024-03-11T21:28:04.077 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:28:03 smithi097 ceph-mon[33317]: from='client.? v1:172.21.15.92:0/954226875' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-11T21:28:04.108 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:28:04.112 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:28:03 smithi092 ceph-mon[29572]: from='client.24469 v1:172.21.15.97:0/404208310' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:28:04.112 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:28:03 smithi092 ceph-mon[29572]: 6.c scrub starts 2024-03-11T21:28:04.112 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:28:03 smithi092 ceph-mon[29572]: 6.c scrub ok 2024-03-11T21:28:04.112 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:28:03 smithi092 ceph-mon[29572]: from='client.? v1:172.21.15.92:0/954226875' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-11T21:28:04.113 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:28:03 smithi092 ceph-mon[37563]: from='client.24469 v1:172.21.15.97:0/404208310' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:28:04.113 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:28:03 smithi092 ceph-mon[37563]: 6.c scrub starts 2024-03-11T21:28:04.113 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:28:03 smithi092 ceph-mon[37563]: 6.c scrub ok 2024-03-11T21:28:04.113 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:28:03 smithi092 ceph-mon[37563]: from='client.? v1:172.21.15.92:0/954226875' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-11T21:28:04.150 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:28:04.151 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:28:04.151 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:28:04.252 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:28:04.290 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:28:04.291 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:28:04.291 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:28:04.392 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:28:04.420 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:28:04.421 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:28:04.421 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:28:04.522 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:28:04.561 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:28:04.562 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:28:04.563 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:28:04.622 INFO:tasks.radosbench.radosbench.0.smithi092.stdout:2024-03-11T21:28:04.639925+0000 min lat: 0.00131184 max lat: 0.102714 avg lat: 0.0073261 2024-03-11T21:28:04.622 INFO:tasks.radosbench.radosbench.0.smithi092.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-03-11T21:28:04.622 INFO:tasks.radosbench.radosbench.0.smithi092.stdout: 20 16 43671 43655 136.399 104.312 0.00604029 0.0073261 2024-03-11T21:28:04.664 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:28:04.701 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:28:04.701 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:28:04.702 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:28:04.803 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:28:04.835 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:28:04.835 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:28:04.835 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:28:04.938 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:28:04.970 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:28:04.970 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:28:04.970 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:28:05.056 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:28:04 smithi092 ceph-mon[29572]: pgmap v73: 148 pgs: 148 active+clean; 2.4 GiB data, 7.4 GiB used, 708 GiB / 715 GiB avail; 1023 B/s rd, 119 MiB/s wr, 1.91k op/s 2024-03-11T21:28:05.056 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:28:04 smithi092 ceph-mon[29572]: 2.d scrub starts 2024-03-11T21:28:05.056 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:28:04 smithi092 ceph-mon[29572]: 2.d scrub ok 2024-03-11T21:28:05.056 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:28:04 smithi092 ceph-mon[29572]: 6.b scrub starts 2024-03-11T21:28:05.056 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:28:04 smithi092 ceph-mon[29572]: 6.b scrub ok 2024-03-11T21:28:05.056 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:28:04 smithi092 ceph-mon[29572]: 1.0 scrub starts 2024-03-11T21:28:05.056 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:28:04 smithi092 ceph-mon[29572]: 1.0 scrub ok 2024-03-11T21:28:05.056 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:28:04 smithi092 ceph-mon[29572]: 5.f scrub starts 2024-03-11T21:28:05.056 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:28:04 smithi092 ceph-mon[29572]: 5.f scrub ok 2024-03-11T21:28:05.056 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:28:04 smithi092 ceph-mon[29572]: 2.7 scrub starts 2024-03-11T21:28:05.056 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:28:04 smithi092 ceph-mon[29572]: 2.7 scrub ok 2024-03-11T21:28:05.057 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:28:04 smithi092 ceph-mon[37563]: pgmap v73: 148 pgs: 148 active+clean; 2.4 GiB data, 7.4 GiB used, 708 GiB / 715 GiB avail; 1023 B/s rd, 119 MiB/s wr, 1.91k op/s 2024-03-11T21:28:05.057 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:28:04 smithi092 ceph-mon[37563]: 2.d scrub starts 2024-03-11T21:28:05.057 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:28:04 smithi092 ceph-mon[37563]: 2.d scrub ok 2024-03-11T21:28:05.057 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:28:04 smithi092 ceph-mon[37563]: 6.b scrub starts 2024-03-11T21:28:05.057 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:28:04 smithi092 ceph-mon[37563]: 6.b scrub ok 2024-03-11T21:28:05.057 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:28:04 smithi092 ceph-mon[37563]: 1.0 scrub starts 2024-03-11T21:28:05.057 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:28:04 smithi092 ceph-mon[37563]: 1.0 scrub ok 2024-03-11T21:28:05.057 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:28:04 smithi092 ceph-mon[37563]: 5.f scrub starts 2024-03-11T21:28:05.057 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:28:04 smithi092 ceph-mon[37563]: 5.f scrub ok 2024-03-11T21:28:05.057 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:28:04 smithi092 ceph-mon[37563]: 2.7 scrub starts 2024-03-11T21:28:05.057 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:28:04 smithi092 ceph-mon[37563]: 2.7 scrub ok 2024-03-11T21:28:05.072 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:28:05.076 DEBUG:teuthology.orchestra.run.smithi092:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid 00db0016-dfed-11ee-95c8-87774f69a715 -- ceph osd unset nodeep-scrub 2024-03-11T21:28:05.077 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:28:04 smithi097 ceph-mon[33317]: pgmap v73: 148 pgs: 148 active+clean; 2.4 GiB data, 7.4 GiB used, 708 GiB / 715 GiB avail; 1023 B/s rd, 119 MiB/s wr, 1.91k op/s 2024-03-11T21:28:05.077 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:28:04 smithi097 ceph-mon[33317]: 2.d scrub starts 2024-03-11T21:28:05.077 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:28:04 smithi097 ceph-mon[33317]: 2.d scrub ok 2024-03-11T21:28:05.077 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:28:04 smithi097 ceph-mon[33317]: 6.b scrub starts 2024-03-11T21:28:05.077 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:28:04 smithi097 ceph-mon[33317]: 6.b scrub ok 2024-03-11T21:28:05.077 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:28:04 smithi097 ceph-mon[33317]: 1.0 scrub starts 2024-03-11T21:28:05.077 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:28:04 smithi097 ceph-mon[33317]: 1.0 scrub ok 2024-03-11T21:28:05.077 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:28:04 smithi097 ceph-mon[33317]: 5.f scrub starts 2024-03-11T21:28:05.077 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:28:04 smithi097 ceph-mon[33317]: 5.f scrub ok 2024-03-11T21:28:05.077 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:28:04 smithi097 ceph-mon[33317]: 2.7 scrub starts 2024-03-11T21:28:05.077 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:28:04 smithi097 ceph-mon[33317]: 2.7 scrub ok 2024-03-11T21:28:05.111 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:28:05.112 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:28:05.112 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:28:05.214 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:28:05.263 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:28:05.263 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:28:05.263 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:28:05.364 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:28:05.413 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:28:05.413 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:28:05.414 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:28:05.515 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:28:05.557 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:28:05.558 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:28:05.558 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:28:05.622 INFO:tasks.radosbench.radosbench.0.smithi092.stdout: 21 16 45372 45356 134.966 106.312 0.0148287 0.0074041 2024-03-11T21:28:05.659 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:28:05.690 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:28:05.691 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:28:05.692 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:28:05.793 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:28:05.834 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:28:05.835 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:28:05.835 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:28:05.936 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:28:05.967 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:28:05.968 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:28:05.968 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:28:06.069 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:28:06.099 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:28:06.100 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:28:06.100 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:28:06.201 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:28:06.251 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:28:06.252 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:28:06.252 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:28:06.353 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:28:06.396 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:28:06.397 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:28:06.398 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:28:06.498 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:28:06.528 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:28:06.529 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:28:06.529 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:28:06.622 INFO:tasks.radosbench.radosbench.0.smithi092.stdout: 22 16 47068 47052 133.648 106 0.0191627 0.00747536 2024-03-11T21:28:06.630 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:28:06.659 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:28:06.660 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:28:06.660 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:28:06.761 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:28:06.793 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:28:06.793 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:28:06.794 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:28:06.827 INFO:journalctl@ceph.node-exporter.b.smithi097.stdout:Mar 11 21:28:06 smithi097 bash[66565]: Copying config sha256:0da6a335fe1356545476b749c68f022c897de3a2139e8f0054f6937349ee2b83 2024-03-11T21:28:06.827 INFO:journalctl@ceph.node-exporter.b.smithi097.stdout:Mar 11 21:28:06 smithi097 bash[66565]: Writing manifest to image destination 2024-03-11T21:28:06.863 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:28:06 smithi092 ceph-mon[29572]: pgmap v74: 148 pgs: 148 active+clean; 2.4 GiB data, 7.4 GiB used, 708 GiB / 715 GiB avail; 1023 B/s rd, 119 MiB/s wr, 1.91k op/s 2024-03-11T21:28:06.863 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:28:06 smithi092 ceph-mon[29572]: 2.a scrub starts 2024-03-11T21:28:06.863 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:28:06 smithi092 ceph-mon[29572]: 2.a scrub ok 2024-03-11T21:28:06.863 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:28:06 smithi092 ceph-mon[29572]: 6.7 scrub starts 2024-03-11T21:28:06.863 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:28:06 smithi092 ceph-mon[29572]: 6.7 scrub ok 2024-03-11T21:28:06.863 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:28:06 smithi092 ceph-mon[29572]: 5.1f scrub starts 2024-03-11T21:28:06.863 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:28:06 smithi092 ceph-mon[29572]: 5.1f scrub ok 2024-03-11T21:28:06.864 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:28:06 smithi092 ceph-mon[29572]: from='mgr.14607 v1:172.21.15.92:0/1702717597' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T21:28:06.864 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:28:06 smithi092 ceph-mon[29572]: 3.1 scrub starts 2024-03-11T21:28:06.864 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:28:06 smithi092 ceph-mon[29572]: 3.1 scrub ok 2024-03-11T21:28:06.864 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:28:06 smithi092 ceph-mon[37563]: pgmap v74: 148 pgs: 148 active+clean; 2.4 GiB data, 7.4 GiB used, 708 GiB / 715 GiB avail; 1023 B/s rd, 119 MiB/s wr, 1.91k op/s 2024-03-11T21:28:06.864 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:28:06 smithi092 ceph-mon[37563]: 2.a scrub starts 2024-03-11T21:28:06.864 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:28:06 smithi092 ceph-mon[37563]: 2.a scrub ok 2024-03-11T21:28:06.864 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:28:06 smithi092 ceph-mon[37563]: 6.7 scrub starts 2024-03-11T21:28:06.864 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:28:06 smithi092 ceph-mon[37563]: 6.7 scrub ok 2024-03-11T21:28:06.864 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:28:06 smithi092 ceph-mon[37563]: 5.1f scrub starts 2024-03-11T21:28:06.864 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:28:06 smithi092 ceph-mon[37563]: 5.1f scrub ok 2024-03-11T21:28:06.864 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:28:06 smithi092 ceph-mon[37563]: from='mgr.14607 v1:172.21.15.92:0/1702717597' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T21:28:06.864 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:28:06 smithi092 ceph-mon[37563]: 3.1 scrub starts 2024-03-11T21:28:06.865 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:28:06 smithi092 ceph-mon[37563]: 3.1 scrub ok 2024-03-11T21:28:06.895 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:28:06.946 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:28:06.947 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:28:06.947 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:28:07.048 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:28:07.086 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:28:07.087 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:28:07.087 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:28:07.115 DEBUG:teuthology.orchestra.run.smithi092:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid 00db0016-dfed-11ee-95c8-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-03-11T21:28:07.148 INFO:journalctl@ceph.node-exporter.b.smithi097.stdout:Mar 11 21:28:06 smithi097 podman[66565]: 2024-03-11 21:28:06.915498147 +0000 UTC m=+11.305492524 image pull 0da6a335fe1356545476b749c68f022c897de3a2139e8f0054f6937349ee2b83 quay.io/prometheus/node-exporter:v1.5.0 2024-03-11T21:28:07.148 INFO:journalctl@ceph.node-exporter.b.smithi097.stdout:Mar 11 21:28:07 smithi097 podman[66565]: 2024-03-11 21:28:07.047358973 +0000 UTC m=+11.437353331 container create 02c16113a38a9503d3369f699573e38bf8895d2509a21b25536a2ac3ac5623a3 (image=quay.io/prometheus/node-exporter:v1.5.0, name=ceph-00db0016-dfed-11ee-95c8-87774f69a715-node-exporter-b, maintainer=The Prometheus Authors ) 2024-03-11T21:28:07.148 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:28:06 smithi097 ceph-mon[33317]: pgmap v74: 148 pgs: 148 active+clean; 2.4 GiB data, 7.4 GiB used, 708 GiB / 715 GiB avail; 1023 B/s rd, 119 MiB/s wr, 1.91k op/s 2024-03-11T21:28:07.148 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:28:06 smithi097 ceph-mon[33317]: 2.a scrub starts 2024-03-11T21:28:07.148 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:28:06 smithi097 ceph-mon[33317]: 2.a scrub ok 2024-03-11T21:28:07.148 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:28:06 smithi097 ceph-mon[33317]: 6.7 scrub starts 2024-03-11T21:28:07.149 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:28:06 smithi097 ceph-mon[33317]: 6.7 scrub ok 2024-03-11T21:28:07.149 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:28:06 smithi097 ceph-mon[33317]: 5.1f scrub starts 2024-03-11T21:28:07.149 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:28:06 smithi097 ceph-mon[33317]: 5.1f scrub ok 2024-03-11T21:28:07.149 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:28:06 smithi097 ceph-mon[33317]: from='mgr.14607 v1:172.21.15.92:0/1702717597' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T21:28:07.149 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:28:06 smithi097 ceph-mon[33317]: 3.1 scrub starts 2024-03-11T21:28:07.149 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:28:06 smithi097 ceph-mon[33317]: 3.1 scrub ok 2024-03-11T21:28:07.188 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:28:07.212 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:28:07.213 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:28:07.213 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:28:07.314 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:28:07.346 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:28:07.346 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:28:07.347 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:28:07.448 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:28:07.471 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:28:07.471 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:28:07.472 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:28:07.573 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:28:07.578 INFO:journalctl@ceph.node-exporter.b.smithi097.stdout:Mar 11 21:28:07 smithi097 podman[66565]: 2024-03-11 21:28:07.283544853 +0000 UTC m=+11.673539226 container init 02c16113a38a9503d3369f699573e38bf8895d2509a21b25536a2ac3ac5623a3 (image=quay.io/prometheus/node-exporter:v1.5.0, name=ceph-00db0016-dfed-11ee-95c8-87774f69a715-node-exporter-b, maintainer=The Prometheus Authors ) 2024-03-11T21:28:07.578 INFO:journalctl@ceph.node-exporter.b.smithi097.stdout:Mar 11 21:28:07 smithi097 podman[66565]: 2024-03-11 21:28:07.288423871 +0000 UTC m=+11.678418236 container start 02c16113a38a9503d3369f699573e38bf8895d2509a21b25536a2ac3ac5623a3 (image=quay.io/prometheus/node-exporter:v1.5.0, name=ceph-00db0016-dfed-11ee-95c8-87774f69a715-node-exporter-b, maintainer=The Prometheus Authors ) 2024-03-11T21:28:07.578 INFO:journalctl@ceph.node-exporter.b.smithi097.stdout:Mar 11 21:28:07 smithi097 ceph-00db0016-dfed-11ee-95c8-87774f69a715-node-exporter-b[67444]: ts=2024-03-11T21:28:07.292Z caller=node_exporter.go:180 level=info msg="Starting node_exporter" version="(version=1.5.0, branch=HEAD, revision=1b48970ffcf5630534fb00bb0687d73c66d1c959)" 2024-03-11T21:28:07.578 INFO:journalctl@ceph.node-exporter.b.smithi097.stdout:Mar 11 21:28:07 smithi097 ceph-00db0016-dfed-11ee-95c8-87774f69a715-node-exporter-b[67444]: ts=2024-03-11T21:28:07.292Z caller=node_exporter.go:181 level=info msg="Build context" build_context="(go=go1.19.3, user=root@6e7732a7b81b, date=20221129-18:59:09)" 2024-03-11T21:28:07.578 INFO:journalctl@ceph.node-exporter.b.smithi097.stdout:Mar 11 21:28:07 smithi097 ceph-00db0016-dfed-11ee-95c8-87774f69a715-node-exporter-b[67444]: ts=2024-03-11T21:28:07.293Z caller=filesystem_common.go:111 level=info collector=filesystem msg="Parsed flag --collector.filesystem.mount-points-exclude" flag=^/(dev|proc|run/credentials/.+|sys|var/lib/docker/.+|var/lib/containers/storage/.+)($|/) 2024-03-11T21:28:07.578 INFO:journalctl@ceph.node-exporter.b.smithi097.stdout:Mar 11 21:28:07 smithi097 ceph-00db0016-dfed-11ee-95c8-87774f69a715-node-exporter-b[67444]: ts=2024-03-11T21:28:07.293Z caller=filesystem_common.go:113 level=info collector=filesystem msg="Parsed flag --collector.filesystem.fs-types-exclude" flag=^(autofs|binfmt_misc|bpf|cgroup2?|configfs|debugfs|devpts|devtmpfs|fusectl|hugetlbfs|iso9660|mqueue|nsfs|overlay|proc|procfs|pstore|rpc_pipefs|securityfs|selinuxfs|squashfs|sysfs|tracefs)$ 2024-03-11T21:28:07.578 INFO:journalctl@ceph.node-exporter.b.smithi097.stdout:Mar 11 21:28:07 smithi097 ceph-00db0016-dfed-11ee-95c8-87774f69a715-node-exporter-b[67444]: ts=2024-03-11T21:28:07.293Z caller=diskstats_common.go:111 level=info collector=diskstats msg="Parsed flag --collector.diskstats.device-exclude" flag=^(ram|loop|fd|(h|s|v|xv)d[a-z]|nvme\d+n\d+p)\d+$ 2024-03-11T21:28:07.578 INFO:journalctl@ceph.node-exporter.b.smithi097.stdout:Mar 11 21:28:07 smithi097 ceph-00db0016-dfed-11ee-95c8-87774f69a715-node-exporter-b[67444]: ts=2024-03-11T21:28:07.294Z caller=diskstats_linux.go:264 level=error collector=diskstats msg="Failed to open directory, disabling udev device properties" path=/run/udev/data 2024-03-11T21:28:07.578 INFO:journalctl@ceph.node-exporter.b.smithi097.stdout:Mar 11 21:28:07 smithi097 ceph-00db0016-dfed-11ee-95c8-87774f69a715-node-exporter-b[67444]: ts=2024-03-11T21:28:07.294Z caller=node_exporter.go:110 level=info msg="Enabled collectors" 2024-03-11T21:28:07.578 INFO:journalctl@ceph.node-exporter.b.smithi097.stdout:Mar 11 21:28:07 smithi097 ceph-00db0016-dfed-11ee-95c8-87774f69a715-node-exporter-b[67444]: ts=2024-03-11T21:28:07.294Z caller=node_exporter.go:117 level=info collector=arp 2024-03-11T21:28:07.578 INFO:journalctl@ceph.node-exporter.b.smithi097.stdout:Mar 11 21:28:07 smithi097 ceph-00db0016-dfed-11ee-95c8-87774f69a715-node-exporter-b[67444]: ts=2024-03-11T21:28:07.294Z caller=node_exporter.go:117 level=info collector=bcache 2024-03-11T21:28:07.578 INFO:journalctl@ceph.node-exporter.b.smithi097.stdout:Mar 11 21:28:07 smithi097 ceph-00db0016-dfed-11ee-95c8-87774f69a715-node-exporter-b[67444]: ts=2024-03-11T21:28:07.294Z caller=node_exporter.go:117 level=info collector=bonding 2024-03-11T21:28:07.579 INFO:journalctl@ceph.node-exporter.b.smithi097.stdout:Mar 11 21:28:07 smithi097 ceph-00db0016-dfed-11ee-95c8-87774f69a715-node-exporter-b[67444]: ts=2024-03-11T21:28:07.294Z caller=node_exporter.go:117 level=info collector=btrfs 2024-03-11T21:28:07.579 INFO:journalctl@ceph.node-exporter.b.smithi097.stdout:Mar 11 21:28:07 smithi097 ceph-00db0016-dfed-11ee-95c8-87774f69a715-node-exporter-b[67444]: ts=2024-03-11T21:28:07.294Z caller=node_exporter.go:117 level=info collector=conntrack 2024-03-11T21:28:07.579 INFO:journalctl@ceph.node-exporter.b.smithi097.stdout:Mar 11 21:28:07 smithi097 ceph-00db0016-dfed-11ee-95c8-87774f69a715-node-exporter-b[67444]: ts=2024-03-11T21:28:07.294Z caller=node_exporter.go:117 level=info collector=cpu 2024-03-11T21:28:07.579 INFO:journalctl@ceph.node-exporter.b.smithi097.stdout:Mar 11 21:28:07 smithi097 ceph-00db0016-dfed-11ee-95c8-87774f69a715-node-exporter-b[67444]: ts=2024-03-11T21:28:07.294Z caller=node_exporter.go:117 level=info collector=cpufreq 2024-03-11T21:28:07.579 INFO:journalctl@ceph.node-exporter.b.smithi097.stdout:Mar 11 21:28:07 smithi097 ceph-00db0016-dfed-11ee-95c8-87774f69a715-node-exporter-b[67444]: ts=2024-03-11T21:28:07.294Z caller=node_exporter.go:117 level=info collector=diskstats 2024-03-11T21:28:07.579 INFO:journalctl@ceph.node-exporter.b.smithi097.stdout:Mar 11 21:28:07 smithi097 ceph-00db0016-dfed-11ee-95c8-87774f69a715-node-exporter-b[67444]: ts=2024-03-11T21:28:07.294Z caller=node_exporter.go:117 level=info collector=dmi 2024-03-11T21:28:07.579 INFO:journalctl@ceph.node-exporter.b.smithi097.stdout:Mar 11 21:28:07 smithi097 ceph-00db0016-dfed-11ee-95c8-87774f69a715-node-exporter-b[67444]: ts=2024-03-11T21:28:07.294Z caller=node_exporter.go:117 level=info collector=edac 2024-03-11T21:28:07.580 INFO:journalctl@ceph.node-exporter.b.smithi097.stdout:Mar 11 21:28:07 smithi097 ceph-00db0016-dfed-11ee-95c8-87774f69a715-node-exporter-b[67444]: ts=2024-03-11T21:28:07.294Z caller=node_exporter.go:117 level=info collector=entropy 2024-03-11T21:28:07.580 INFO:journalctl@ceph.node-exporter.b.smithi097.stdout:Mar 11 21:28:07 smithi097 ceph-00db0016-dfed-11ee-95c8-87774f69a715-node-exporter-b[67444]: ts=2024-03-11T21:28:07.294Z caller=node_exporter.go:117 level=info collector=fibrechannel 2024-03-11T21:28:07.580 INFO:journalctl@ceph.node-exporter.b.smithi097.stdout:Mar 11 21:28:07 smithi097 ceph-00db0016-dfed-11ee-95c8-87774f69a715-node-exporter-b[67444]: ts=2024-03-11T21:28:07.294Z caller=node_exporter.go:117 level=info collector=filefd 2024-03-11T21:28:07.580 INFO:journalctl@ceph.node-exporter.b.smithi097.stdout:Mar 11 21:28:07 smithi097 ceph-00db0016-dfed-11ee-95c8-87774f69a715-node-exporter-b[67444]: ts=2024-03-11T21:28:07.294Z caller=node_exporter.go:117 level=info collector=filesystem 2024-03-11T21:28:07.580 INFO:journalctl@ceph.node-exporter.b.smithi097.stdout:Mar 11 21:28:07 smithi097 ceph-00db0016-dfed-11ee-95c8-87774f69a715-node-exporter-b[67444]: ts=2024-03-11T21:28:07.294Z caller=node_exporter.go:117 level=info collector=hwmon 2024-03-11T21:28:07.580 INFO:journalctl@ceph.node-exporter.b.smithi097.stdout:Mar 11 21:28:07 smithi097 ceph-00db0016-dfed-11ee-95c8-87774f69a715-node-exporter-b[67444]: ts=2024-03-11T21:28:07.294Z caller=node_exporter.go:117 level=info collector=infiniband 2024-03-11T21:28:07.580 INFO:journalctl@ceph.node-exporter.b.smithi097.stdout:Mar 11 21:28:07 smithi097 ceph-00db0016-dfed-11ee-95c8-87774f69a715-node-exporter-b[67444]: ts=2024-03-11T21:28:07.294Z caller=node_exporter.go:117 level=info collector=ipvs 2024-03-11T21:28:07.580 INFO:journalctl@ceph.node-exporter.b.smithi097.stdout:Mar 11 21:28:07 smithi097 ceph-00db0016-dfed-11ee-95c8-87774f69a715-node-exporter-b[67444]: ts=2024-03-11T21:28:07.294Z caller=node_exporter.go:117 level=info collector=loadavg 2024-03-11T21:28:07.580 INFO:journalctl@ceph.node-exporter.b.smithi097.stdout:Mar 11 21:28:07 smithi097 ceph-00db0016-dfed-11ee-95c8-87774f69a715-node-exporter-b[67444]: ts=2024-03-11T21:28:07.294Z caller=node_exporter.go:117 level=info collector=mdadm 2024-03-11T21:28:07.580 INFO:journalctl@ceph.node-exporter.b.smithi097.stdout:Mar 11 21:28:07 smithi097 ceph-00db0016-dfed-11ee-95c8-87774f69a715-node-exporter-b[67444]: ts=2024-03-11T21:28:07.294Z caller=node_exporter.go:117 level=info collector=meminfo 2024-03-11T21:28:07.580 INFO:journalctl@ceph.node-exporter.b.smithi097.stdout:Mar 11 21:28:07 smithi097 ceph-00db0016-dfed-11ee-95c8-87774f69a715-node-exporter-b[67444]: ts=2024-03-11T21:28:07.294Z caller=node_exporter.go:117 level=info collector=netclass 2024-03-11T21:28:07.580 INFO:journalctl@ceph.node-exporter.b.smithi097.stdout:Mar 11 21:28:07 smithi097 ceph-00db0016-dfed-11ee-95c8-87774f69a715-node-exporter-b[67444]: ts=2024-03-11T21:28:07.294Z caller=node_exporter.go:117 level=info collector=netdev 2024-03-11T21:28:07.580 INFO:journalctl@ceph.node-exporter.b.smithi097.stdout:Mar 11 21:28:07 smithi097 ceph-00db0016-dfed-11ee-95c8-87774f69a715-node-exporter-b[67444]: ts=2024-03-11T21:28:07.294Z caller=node_exporter.go:117 level=info collector=netstat 2024-03-11T21:28:07.580 INFO:journalctl@ceph.node-exporter.b.smithi097.stdout:Mar 11 21:28:07 smithi097 ceph-00db0016-dfed-11ee-95c8-87774f69a715-node-exporter-b[67444]: ts=2024-03-11T21:28:07.294Z caller=node_exporter.go:117 level=info collector=nfs 2024-03-11T21:28:07.581 INFO:journalctl@ceph.node-exporter.b.smithi097.stdout:Mar 11 21:28:07 smithi097 ceph-00db0016-dfed-11ee-95c8-87774f69a715-node-exporter-b[67444]: ts=2024-03-11T21:28:07.294Z caller=node_exporter.go:117 level=info collector=nfsd 2024-03-11T21:28:07.581 INFO:journalctl@ceph.node-exporter.b.smithi097.stdout:Mar 11 21:28:07 smithi097 ceph-00db0016-dfed-11ee-95c8-87774f69a715-node-exporter-b[67444]: ts=2024-03-11T21:28:07.294Z caller=node_exporter.go:117 level=info collector=nvme 2024-03-11T21:28:07.581 INFO:journalctl@ceph.node-exporter.b.smithi097.stdout:Mar 11 21:28:07 smithi097 ceph-00db0016-dfed-11ee-95c8-87774f69a715-node-exporter-b[67444]: ts=2024-03-11T21:28:07.294Z caller=node_exporter.go:117 level=info collector=os 2024-03-11T21:28:07.581 INFO:journalctl@ceph.node-exporter.b.smithi097.stdout:Mar 11 21:28:07 smithi097 ceph-00db0016-dfed-11ee-95c8-87774f69a715-node-exporter-b[67444]: ts=2024-03-11T21:28:07.294Z caller=node_exporter.go:117 level=info collector=powersupplyclass 2024-03-11T21:28:07.581 INFO:journalctl@ceph.node-exporter.b.smithi097.stdout:Mar 11 21:28:07 smithi097 ceph-00db0016-dfed-11ee-95c8-87774f69a715-node-exporter-b[67444]: ts=2024-03-11T21:28:07.294Z caller=node_exporter.go:117 level=info collector=pressure 2024-03-11T21:28:07.581 INFO:journalctl@ceph.node-exporter.b.smithi097.stdout:Mar 11 21:28:07 smithi097 ceph-00db0016-dfed-11ee-95c8-87774f69a715-node-exporter-b[67444]: ts=2024-03-11T21:28:07.294Z caller=node_exporter.go:117 level=info collector=rapl 2024-03-11T21:28:07.581 INFO:journalctl@ceph.node-exporter.b.smithi097.stdout:Mar 11 21:28:07 smithi097 ceph-00db0016-dfed-11ee-95c8-87774f69a715-node-exporter-b[67444]: ts=2024-03-11T21:28:07.294Z caller=node_exporter.go:117 level=info collector=schedstat 2024-03-11T21:28:07.581 INFO:journalctl@ceph.node-exporter.b.smithi097.stdout:Mar 11 21:28:07 smithi097 ceph-00db0016-dfed-11ee-95c8-87774f69a715-node-exporter-b[67444]: ts=2024-03-11T21:28:07.294Z caller=node_exporter.go:117 level=info collector=selinux 2024-03-11T21:28:07.581 INFO:journalctl@ceph.node-exporter.b.smithi097.stdout:Mar 11 21:28:07 smithi097 ceph-00db0016-dfed-11ee-95c8-87774f69a715-node-exporter-b[67444]: ts=2024-03-11T21:28:07.294Z caller=node_exporter.go:117 level=info collector=sockstat 2024-03-11T21:28:07.581 INFO:journalctl@ceph.node-exporter.b.smithi097.stdout:Mar 11 21:28:07 smithi097 ceph-00db0016-dfed-11ee-95c8-87774f69a715-node-exporter-b[67444]: ts=2024-03-11T21:28:07.294Z caller=node_exporter.go:117 level=info collector=softnet 2024-03-11T21:28:07.581 INFO:journalctl@ceph.node-exporter.b.smithi097.stdout:Mar 11 21:28:07 smithi097 ceph-00db0016-dfed-11ee-95c8-87774f69a715-node-exporter-b[67444]: ts=2024-03-11T21:28:07.294Z caller=node_exporter.go:117 level=info collector=stat 2024-03-11T21:28:07.581 INFO:journalctl@ceph.node-exporter.b.smithi097.stdout:Mar 11 21:28:07 smithi097 ceph-00db0016-dfed-11ee-95c8-87774f69a715-node-exporter-b[67444]: ts=2024-03-11T21:28:07.294Z caller=node_exporter.go:117 level=info collector=tapestats 2024-03-11T21:28:07.581 INFO:journalctl@ceph.node-exporter.b.smithi097.stdout:Mar 11 21:28:07 smithi097 ceph-00db0016-dfed-11ee-95c8-87774f69a715-node-exporter-b[67444]: ts=2024-03-11T21:28:07.294Z caller=node_exporter.go:117 level=info collector=textfile 2024-03-11T21:28:07.581 INFO:journalctl@ceph.node-exporter.b.smithi097.stdout:Mar 11 21:28:07 smithi097 ceph-00db0016-dfed-11ee-95c8-87774f69a715-node-exporter-b[67444]: ts=2024-03-11T21:28:07.294Z caller=node_exporter.go:117 level=info collector=thermal_zone 2024-03-11T21:28:07.581 INFO:journalctl@ceph.node-exporter.b.smithi097.stdout:Mar 11 21:28:07 smithi097 ceph-00db0016-dfed-11ee-95c8-87774f69a715-node-exporter-b[67444]: ts=2024-03-11T21:28:07.294Z caller=node_exporter.go:117 level=info collector=time 2024-03-11T21:28:07.581 INFO:journalctl@ceph.node-exporter.b.smithi097.stdout:Mar 11 21:28:07 smithi097 ceph-00db0016-dfed-11ee-95c8-87774f69a715-node-exporter-b[67444]: ts=2024-03-11T21:28:07.294Z caller=node_exporter.go:117 level=info collector=udp_queues 2024-03-11T21:28:07.582 INFO:journalctl@ceph.node-exporter.b.smithi097.stdout:Mar 11 21:28:07 smithi097 ceph-00db0016-dfed-11ee-95c8-87774f69a715-node-exporter-b[67444]: ts=2024-03-11T21:28:07.294Z caller=node_exporter.go:117 level=info collector=uname 2024-03-11T21:28:07.582 INFO:journalctl@ceph.node-exporter.b.smithi097.stdout:Mar 11 21:28:07 smithi097 ceph-00db0016-dfed-11ee-95c8-87774f69a715-node-exporter-b[67444]: ts=2024-03-11T21:28:07.294Z caller=node_exporter.go:117 level=info collector=vmstat 2024-03-11T21:28:07.582 INFO:journalctl@ceph.node-exporter.b.smithi097.stdout:Mar 11 21:28:07 smithi097 ceph-00db0016-dfed-11ee-95c8-87774f69a715-node-exporter-b[67444]: ts=2024-03-11T21:28:07.294Z caller=node_exporter.go:117 level=info collector=xfs 2024-03-11T21:28:07.582 INFO:journalctl@ceph.node-exporter.b.smithi097.stdout:Mar 11 21:28:07 smithi097 ceph-00db0016-dfed-11ee-95c8-87774f69a715-node-exporter-b[67444]: ts=2024-03-11T21:28:07.294Z caller=node_exporter.go:117 level=info collector=zfs 2024-03-11T21:28:07.582 INFO:journalctl@ceph.node-exporter.b.smithi097.stdout:Mar 11 21:28:07 smithi097 ceph-00db0016-dfed-11ee-95c8-87774f69a715-node-exporter-b[67444]: ts=2024-03-11T21:28:07.294Z caller=tls_config.go:232 level=info msg="Listening on" address=[::]:9100 2024-03-11T21:28:07.582 INFO:journalctl@ceph.node-exporter.b.smithi097.stdout:Mar 11 21:28:07 smithi097 ceph-00db0016-dfed-11ee-95c8-87774f69a715-node-exporter-b[67444]: ts=2024-03-11T21:28:07.294Z caller=tls_config.go:235 level=info msg="TLS is disabled." http2=false address=[::]:9100 2024-03-11T21:28:07.582 INFO:journalctl@ceph.node-exporter.b.smithi097.stdout:Mar 11 21:28:07 smithi097 bash[66565]: 02c16113a38a9503d3369f699573e38bf8895d2509a21b25536a2ac3ac5623a3 2024-03-11T21:28:07.582 INFO:journalctl@ceph.node-exporter.b.smithi097.stdout:Mar 11 21:28:07 smithi097 systemd[1]: Started Ceph node-exporter.b for 00db0016-dfed-11ee-95c8-87774f69a715. 2024-03-11T21:28:07.604 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:28:07.605 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:28:07.605 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:28:07.622 INFO:tasks.radosbench.radosbench.0.smithi092.stdout: 23 16 48458 48442 131.614 86.875 0.00758112 0.00759266 2024-03-11T21:28:07.706 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:28:07.752 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:28:07.752 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:28:07.753 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:28:07.854 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:28:07.889 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:28:07.890 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:28:07.891 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:28:07.925 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:28:07 smithi092 ceph-mon[29572]: purged_snaps scrub starts 2024-03-11T21:28:07.925 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:28:07 smithi092 ceph-mon[29572]: purged_snaps scrub ok 2024-03-11T21:28:07.925 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:28:07 smithi092 ceph-mon[29572]: osdmap e107: 8 total, 8 up, 8 in 2024-03-11T21:28:07.925 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:28:07 smithi092 ceph-mon[29572]: from='mgr.14607 ' entity='mgr.y' 2024-03-11T21:28:07.925 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:28:07 smithi092 ceph-mon[29572]: from='mgr.14607 ' entity='mgr.y' 2024-03-11T21:28:07.925 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:28:07 smithi092 ceph-mon[29572]: from='mgr.14607 ' entity='mgr.y' 2024-03-11T21:28:07.925 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:28:07 smithi092 ceph-mon[29572]: from='mgr.14607 ' entity='mgr.y' 2024-03-11T21:28:07.925 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:28:07 smithi092 ceph-mon[29572]: from='mgr.14607 ' entity='mgr.y' 2024-03-11T21:28:07.925 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:28:07 smithi092 ceph-mon[29572]: from='mgr.14607 v1:172.21.15.92:0/1702717597' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T21:28:07.926 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:28:07 smithi092 ceph-mon[37563]: purged_snaps scrub starts 2024-03-11T21:28:07.926 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:28:07 smithi092 ceph-mon[37563]: purged_snaps scrub ok 2024-03-11T21:28:07.926 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:28:07 smithi092 ceph-mon[37563]: osdmap e107: 8 total, 8 up, 8 in 2024-03-11T21:28:07.926 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:28:07 smithi092 ceph-mon[37563]: from='mgr.14607 ' entity='mgr.y' 2024-03-11T21:28:07.926 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:28:07 smithi092 ceph-mon[37563]: from='mgr.14607 ' entity='mgr.y' 2024-03-11T21:28:07.926 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:28:07 smithi092 ceph-mon[37563]: from='mgr.14607 ' entity='mgr.y' 2024-03-11T21:28:07.926 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:28:07 smithi092 ceph-mon[37563]: from='mgr.14607 ' entity='mgr.y' 2024-03-11T21:28:07.926 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:28:07 smithi092 ceph-mon[37563]: from='mgr.14607 ' entity='mgr.y' 2024-03-11T21:28:07.926 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:28:07 smithi092 ceph-mon[37563]: from='mgr.14607 v1:172.21.15.92:0/1702717597' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T21:28:07.991 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:28:08.051 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:28:08.052 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:28:08.052 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:28:08.076 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:28:07 smithi097 ceph-mon[33317]: purged_snaps scrub starts 2024-03-11T21:28:08.077 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:28:07 smithi097 ceph-mon[33317]: purged_snaps scrub ok 2024-03-11T21:28:08.077 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:28:07 smithi097 ceph-mon[33317]: osdmap e107: 8 total, 8 up, 8 in 2024-03-11T21:28:08.077 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:28:07 smithi097 ceph-mon[33317]: from='mgr.14607 ' entity='mgr.y' 2024-03-11T21:28:08.077 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:28:07 smithi097 ceph-mon[33317]: from='mgr.14607 ' entity='mgr.y' 2024-03-11T21:28:08.077 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:28:07 smithi097 ceph-mon[33317]: from='mgr.14607 ' entity='mgr.y' 2024-03-11T21:28:08.077 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:28:07 smithi097 ceph-mon[33317]: from='mgr.14607 ' entity='mgr.y' 2024-03-11T21:28:08.077 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:28:07 smithi097 ceph-mon[33317]: from='mgr.14607 ' entity='mgr.y' 2024-03-11T21:28:08.077 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:28:07 smithi097 ceph-mon[33317]: from='mgr.14607 v1:172.21.15.92:0/1702717597' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T21:28:08.153 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:28:08.202 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:28:08.202 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:28:08.203 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:28:08.303 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:28:08.343 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:28:08.344 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:28:08.344 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:28:08.361 INFO:journalctl@ceph.mgr.y.smithi092.stdout:Mar 11 21:28:07 smithi092 ceph-00db0016-dfed-11ee-95c8-87774f69a715-mgr-y[29801]: ::ffff:172.21.15.97 - - [11/Mar/2024:21:28:07] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-11T21:28:08.445 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:28:08.486 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:28:08.487 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:28:08.487 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:28:08.588 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:28:08.620 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:28:08.620 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:28:08.620 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:28:08.624 INFO:tasks.radosbench.radosbench.0.smithi092.stdout: 24 15 49756 49741 129.505 81.1875 0.00782719 0.00771634 2024-03-11T21:28:08.722 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:28:08.756 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:28:08.756 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:28:08.757 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:28:08.843 INFO:teuthology.orchestra.run.smithi092.stderr:Inferring config /var/lib/ceph/00db0016-dfed-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T21:28:08.858 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:28:08.861 INFO:teuthology.orchestra.run.smithi092.stderr:Inferring config /var/lib/ceph/00db0016-dfed-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T21:28:08.887 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:28:08.888 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:28:08.888 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:28:08.904 INFO:teuthology.orchestra.run.smithi092.stdout:ERROR: (22) Invalid argument 2024-03-11T21:28:08.904 INFO:teuthology.orchestra.run.smithi092.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-11T21:28:08.921 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:28:08 smithi092 ceph-mon[29572]: pgmap v76: 148 pgs: 148 active+clean; 2.6 GiB data, 8.2 GiB used, 707 GiB / 715 GiB avail; 1.2 KiB/s rd, 98 MiB/s wr, 1.57k op/s 2024-03-11T21:28:08.921 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:28:08 smithi092 ceph-mon[29572]: 2.9 scrub starts 2024-03-11T21:28:08.921 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:28:08 smithi092 ceph-mon[29572]: 2.9 scrub ok 2024-03-11T21:28:08.921 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:28:08 smithi092 ceph-mon[29572]: 2.1e scrub starts 2024-03-11T21:28:08.921 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:28:08 smithi092 ceph-mon[29572]: 2.1e scrub ok 2024-03-11T21:28:08.921 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:28:08 smithi092 ceph-mon[29572]: 4.9 scrub starts 2024-03-11T21:28:08.922 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:28:08 smithi092 ceph-mon[29572]: 4.9 scrub ok 2024-03-11T21:28:08.922 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:28:08 smithi092 ceph-mon[37563]: pgmap v76: 148 pgs: 148 active+clean; 2.6 GiB data, 8.2 GiB used, 707 GiB / 715 GiB avail; 1.2 KiB/s rd, 98 MiB/s wr, 1.57k op/s 2024-03-11T21:28:08.922 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:28:08 smithi092 ceph-mon[37563]: 2.9 scrub starts 2024-03-11T21:28:08.922 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:28:08 smithi092 ceph-mon[37563]: 2.9 scrub ok 2024-03-11T21:28:08.922 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:28:08 smithi092 ceph-mon[37563]: 2.1e scrub starts 2024-03-11T21:28:08.922 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:28:08 smithi092 ceph-mon[37563]: 2.1e scrub ok 2024-03-11T21:28:08.922 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:28:08 smithi092 ceph-mon[37563]: 4.9 scrub starts 2024-03-11T21:28:08.922 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:28:08 smithi092 ceph-mon[37563]: 4.9 scrub ok 2024-03-11T21:28:08.956 INFO:teuthology.orchestra.run.smithi092.stdout: 2024-03-11T21:28:08.956 INFO:teuthology.orchestra.run.smithi092.stdout:{ 2024-03-11T21:28:08.956 INFO:teuthology.orchestra.run.smithi092.stdout: "epoch": 107, 2024-03-11T21:28:08.956 INFO:teuthology.orchestra.run.smithi092.stdout: "fsid": "00db0016-dfed-11ee-95c8-87774f69a715", 2024-03-11T21:28:08.956 INFO:teuthology.orchestra.run.smithi092.stdout: "created": "2024-03-11T21:20:10.488059+0000", 2024-03-11T21:28:08.956 INFO:teuthology.orchestra.run.smithi092.stdout: "modified": "2024-03-11T21:28:06.830290+0000", 2024-03-11T21:28:08.956 INFO:teuthology.orchestra.run.smithi092.stdout: "last_up_change": "2024-03-11T21:25:26.018427+0000", 2024-03-11T21:28:08.956 INFO:teuthology.orchestra.run.smithi092.stdout: "last_in_change": "2024-03-11T21:25:08.189557+0000", 2024-03-11T21:28:08.956 INFO:teuthology.orchestra.run.smithi092.stdout: "flags": "nodeep-scrub,sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit", 2024-03-11T21:28:08.956 INFO:teuthology.orchestra.run.smithi092.stdout: "flags_num": 5804032, 2024-03-11T21:28:08.957 INFO:teuthology.orchestra.run.smithi092.stdout: "flags_set": [ 2024-03-11T21:28:08.957 INFO:teuthology.orchestra.run.smithi092.stdout: "nodeep-scrub", 2024-03-11T21:28:08.957 INFO:teuthology.orchestra.run.smithi092.stdout: "pglog_hardlimit", 2024-03-11T21:28:08.957 INFO:teuthology.orchestra.run.smithi092.stdout: "purged_snapdirs", 2024-03-11T21:28:08.957 INFO:teuthology.orchestra.run.smithi092.stdout: "recovery_deletes", 2024-03-11T21:28:08.957 INFO:teuthology.orchestra.run.smithi092.stdout: "sortbitwise" 2024-03-11T21:28:08.957 INFO:teuthology.orchestra.run.smithi092.stdout: ], 2024-03-11T21:28:08.957 INFO:teuthology.orchestra.run.smithi092.stdout: "crush_version": 18, 2024-03-11T21:28:08.957 INFO:teuthology.orchestra.run.smithi092.stdout: "full_ratio": 0.94999998807907104, 2024-03-11T21:28:08.957 INFO:teuthology.orchestra.run.smithi092.stdout: "backfillfull_ratio": 0.89999997615814209, 2024-03-11T21:28:08.957 INFO:teuthology.orchestra.run.smithi092.stdout: "nearfull_ratio": 0.85000002384185791, 2024-03-11T21:28:08.957 INFO:teuthology.orchestra.run.smithi092.stdout: "cluster_snapshot": "", 2024-03-11T21:28:08.957 INFO:teuthology.orchestra.run.smithi092.stdout: "pool_max": 7, 2024-03-11T21:28:08.957 INFO:teuthology.orchestra.run.smithi092.stdout: "max_osd": 8, 2024-03-11T21:28:08.957 INFO:teuthology.orchestra.run.smithi092.stdout: "require_min_compat_client": "luminous", 2024-03-11T21:28:08.958 INFO:teuthology.orchestra.run.smithi092.stdout: "min_compat_client": "luminous", 2024-03-11T21:28:08.958 INFO:teuthology.orchestra.run.smithi092.stdout: "require_osd_release": "squid", 2024-03-11T21:28:08.958 INFO:teuthology.orchestra.run.smithi092.stdout: "allow_crimson": false, 2024-03-11T21:28:08.958 INFO:teuthology.orchestra.run.smithi092.stdout: "pools": [ 2024-03-11T21:28:08.958 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:28:08.958 INFO:teuthology.orchestra.run.smithi092.stdout: "pool": 1, 2024-03-11T21:28:08.958 INFO:teuthology.orchestra.run.smithi092.stdout: "pool_name": ".mgr", 2024-03-11T21:28:08.958 INFO:teuthology.orchestra.run.smithi092.stdout: "create_time": "2024-03-11T21:23:23.805029+0000", 2024-03-11T21:28:08.958 INFO:teuthology.orchestra.run.smithi092.stdout: "flags": 1, 2024-03-11T21:28:08.958 INFO:teuthology.orchestra.run.smithi092.stdout: "flags_names": "hashpspool", 2024-03-11T21:28:08.958 INFO:teuthology.orchestra.run.smithi092.stdout: "type": 1, 2024-03-11T21:28:08.958 INFO:teuthology.orchestra.run.smithi092.stdout: "size": 3, 2024-03-11T21:28:08.959 INFO:teuthology.orchestra.run.smithi092.stdout: "min_size": 2, 2024-03-11T21:28:08.959 INFO:teuthology.orchestra.run.smithi092.stdout: "crush_rule": 0, 2024-03-11T21:28:08.959 INFO:teuthology.orchestra.run.smithi092.stdout: "peering_crush_bucket_count": 0, 2024-03-11T21:28:08.959 INFO:teuthology.orchestra.run.smithi092.stdout: "peering_crush_bucket_target": 0, 2024-03-11T21:28:08.959 INFO:teuthology.orchestra.run.smithi092.stdout: "peering_crush_bucket_barrier": 0, 2024-03-11T21:28:08.959 INFO:teuthology.orchestra.run.smithi092.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-03-11T21:28:08.959 INFO:teuthology.orchestra.run.smithi092.stdout: "object_hash": 2, 2024-03-11T21:28:08.959 INFO:teuthology.orchestra.run.smithi092.stdout: "pg_autoscale_mode": "off", 2024-03-11T21:28:08.959 INFO:teuthology.orchestra.run.smithi092.stdout: "pg_num": 1, 2024-03-11T21:28:08.959 INFO:teuthology.orchestra.run.smithi092.stdout: "pg_placement_num": 1, 2024-03-11T21:28:08.959 INFO:teuthology.orchestra.run.smithi092.stdout: "pg_placement_num_target": 1, 2024-03-11T21:28:08.959 INFO:teuthology.orchestra.run.smithi092.stdout: "pg_num_target": 1, 2024-03-11T21:28:08.959 INFO:teuthology.orchestra.run.smithi092.stdout: "pg_num_pending": 1, 2024-03-11T21:28:08.960 INFO:teuthology.orchestra.run.smithi092.stdout: "last_pg_merge_meta": { 2024-03-11T21:28:08.960 INFO:teuthology.orchestra.run.smithi092.stdout: "source_pgid": "0.0", 2024-03-11T21:28:08.960 INFO:teuthology.orchestra.run.smithi092.stdout: "ready_epoch": 0, 2024-03-11T21:28:08.960 INFO:teuthology.orchestra.run.smithi092.stdout: "last_epoch_started": 0, 2024-03-11T21:28:08.960 INFO:teuthology.orchestra.run.smithi092.stdout: "last_epoch_clean": 0, 2024-03-11T21:28:08.960 INFO:teuthology.orchestra.run.smithi092.stdout: "source_version": "0'0", 2024-03-11T21:28:08.960 INFO:teuthology.orchestra.run.smithi092.stdout: "target_version": "0'0" 2024-03-11T21:28:08.960 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:28:08.960 INFO:teuthology.orchestra.run.smithi092.stdout: "last_change": "22", 2024-03-11T21:28:08.960 INFO:teuthology.orchestra.run.smithi092.stdout: "last_force_op_resend": "0", 2024-03-11T21:28:08.960 INFO:teuthology.orchestra.run.smithi092.stdout: "last_force_op_resend_prenautilus": "0", 2024-03-11T21:28:08.960 INFO:teuthology.orchestra.run.smithi092.stdout: "last_force_op_resend_preluminous": "0", 2024-03-11T21:28:08.960 INFO:teuthology.orchestra.run.smithi092.stdout: "auid": 0, 2024-03-11T21:28:08.960 INFO:teuthology.orchestra.run.smithi092.stdout: "snap_mode": "selfmanaged", 2024-03-11T21:28:08.961 INFO:teuthology.orchestra.run.smithi092.stdout: "snap_seq": 0, 2024-03-11T21:28:08.961 INFO:teuthology.orchestra.run.smithi092.stdout: "snap_epoch": 0, 2024-03-11T21:28:08.961 INFO:teuthology.orchestra.run.smithi092.stdout: "pool_snaps": [], 2024-03-11T21:28:08.961 INFO:teuthology.orchestra.run.smithi092.stdout: "removed_snaps": "[]", 2024-03-11T21:28:08.961 INFO:teuthology.orchestra.run.smithi092.stdout: "quota_max_bytes": 0, 2024-03-11T21:28:08.961 INFO:teuthology.orchestra.run.smithi092.stdout: "quota_max_objects": 0, 2024-03-11T21:28:08.961 INFO:teuthology.orchestra.run.smithi092.stdout: "tiers": [], 2024-03-11T21:28:08.961 INFO:teuthology.orchestra.run.smithi092.stdout: "tier_of": -1, 2024-03-11T21:28:08.961 INFO:teuthology.orchestra.run.smithi092.stdout: "read_tier": -1, 2024-03-11T21:28:08.961 INFO:teuthology.orchestra.run.smithi092.stdout: "write_tier": -1, 2024-03-11T21:28:08.961 INFO:teuthology.orchestra.run.smithi092.stdout: "cache_mode": "none", 2024-03-11T21:28:08.961 INFO:teuthology.orchestra.run.smithi092.stdout: "target_max_bytes": 0, 2024-03-11T21:28:08.961 INFO:teuthology.orchestra.run.smithi092.stdout: "target_max_objects": 0, 2024-03-11T21:28:08.962 INFO:teuthology.orchestra.run.smithi092.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-03-11T21:28:08.962 INFO:teuthology.orchestra.run.smithi092.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-03-11T21:28:08.962 INFO:teuthology.orchestra.run.smithi092.stdout: "cache_target_full_ratio_micro": 800000, 2024-03-11T21:28:08.962 INFO:teuthology.orchestra.run.smithi092.stdout: "cache_min_flush_age": 0, 2024-03-11T21:28:08.962 INFO:teuthology.orchestra.run.smithi092.stdout: "cache_min_evict_age": 0, 2024-03-11T21:28:08.962 INFO:teuthology.orchestra.run.smithi092.stdout: "erasure_code_profile": "", 2024-03-11T21:28:08.962 INFO:teuthology.orchestra.run.smithi092.stdout: "hit_set_params": { 2024-03-11T21:28:08.962 INFO:teuthology.orchestra.run.smithi092.stdout: "type": "none" 2024-03-11T21:28:08.962 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:28:08.962 INFO:teuthology.orchestra.run.smithi092.stdout: "hit_set_period": 0, 2024-03-11T21:28:08.962 INFO:teuthology.orchestra.run.smithi092.stdout: "hit_set_count": 0, 2024-03-11T21:28:08.962 INFO:teuthology.orchestra.run.smithi092.stdout: "use_gmt_hitset": true, 2024-03-11T21:28:08.962 INFO:teuthology.orchestra.run.smithi092.stdout: "min_read_recency_for_promote": 0, 2024-03-11T21:28:08.963 INFO:teuthology.orchestra.run.smithi092.stdout: "min_write_recency_for_promote": 0, 2024-03-11T21:28:08.963 INFO:teuthology.orchestra.run.smithi092.stdout: "hit_set_grade_decay_rate": 0, 2024-03-11T21:28:08.963 INFO:teuthology.orchestra.run.smithi092.stdout: "hit_set_search_last_n": 0, 2024-03-11T21:28:08.963 INFO:teuthology.orchestra.run.smithi092.stdout: "grade_table": [], 2024-03-11T21:28:08.963 INFO:teuthology.orchestra.run.smithi092.stdout: "stripe_width": 0, 2024-03-11T21:28:08.963 INFO:teuthology.orchestra.run.smithi092.stdout: "expected_num_objects": 0, 2024-03-11T21:28:08.963 INFO:teuthology.orchestra.run.smithi092.stdout: "fast_read": false, 2024-03-11T21:28:08.963 INFO:teuthology.orchestra.run.smithi092.stdout: "options": { 2024-03-11T21:28:08.963 INFO:teuthology.orchestra.run.smithi092.stdout: "pg_num_max": 32, 2024-03-11T21:28:08.963 INFO:teuthology.orchestra.run.smithi092.stdout: "pg_num_min": 1 2024-03-11T21:28:08.963 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:28:08.963 INFO:teuthology.orchestra.run.smithi092.stdout: "application_metadata": { 2024-03-11T21:28:08.963 INFO:teuthology.orchestra.run.smithi092.stdout: "mgr": {} 2024-03-11T21:28:08.963 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:28:08.964 INFO:teuthology.orchestra.run.smithi092.stdout: "read_balance": { 2024-03-11T21:28:08.964 INFO:teuthology.orchestra.run.smithi092.stdout: "score_type": "Fair distribution", 2024-03-11T21:28:08.964 INFO:teuthology.orchestra.run.smithi092.stdout: "score_acting": 7.8899998664855957, 2024-03-11T21:28:08.964 INFO:teuthology.orchestra.run.smithi092.stdout: "score_stable": 7.8899998664855957, 2024-03-11T21:28:08.964 INFO:teuthology.orchestra.run.smithi092.stdout: "optimal_score": 0.37999999523162842, 2024-03-11T21:28:08.964 INFO:teuthology.orchestra.run.smithi092.stdout: "raw_score_acting": 3, 2024-03-11T21:28:08.964 INFO:teuthology.orchestra.run.smithi092.stdout: "raw_score_stable": 3, 2024-03-11T21:28:08.964 INFO:teuthology.orchestra.run.smithi092.stdout: "primary_affinity_weighted": 1, 2024-03-11T21:28:08.964 INFO:teuthology.orchestra.run.smithi092.stdout: "average_primary_affinity": 1, 2024-03-11T21:28:08.964 INFO:teuthology.orchestra.run.smithi092.stdout: "average_primary_affinity_weighted": 1 2024-03-11T21:28:08.964 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-03-11T21:28:08.964 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:28:08.964 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:28:08.964 INFO:teuthology.orchestra.run.smithi092.stdout: "pool": 2, 2024-03-11T21:28:08.965 INFO:teuthology.orchestra.run.smithi092.stdout: "pool_name": ".rgw.root", 2024-03-11T21:28:08.965 INFO:teuthology.orchestra.run.smithi092.stdout: "create_time": "2024-03-11T21:25:37.044504+0000", 2024-03-11T21:28:08.965 INFO:teuthology.orchestra.run.smithi092.stdout: "flags": 1, 2024-03-11T21:28:08.965 INFO:teuthology.orchestra.run.smithi092.stdout: "flags_names": "hashpspool", 2024-03-11T21:28:08.965 INFO:teuthology.orchestra.run.smithi092.stdout: "type": 1, 2024-03-11T21:28:08.965 INFO:teuthology.orchestra.run.smithi092.stdout: "size": 3, 2024-03-11T21:28:08.965 INFO:teuthology.orchestra.run.smithi092.stdout: "min_size": 2, 2024-03-11T21:28:08.965 INFO:teuthology.orchestra.run.smithi092.stdout: "crush_rule": 0, 2024-03-11T21:28:08.965 INFO:teuthology.orchestra.run.smithi092.stdout: "peering_crush_bucket_count": 0, 2024-03-11T21:28:08.965 INFO:teuthology.orchestra.run.smithi092.stdout: "peering_crush_bucket_target": 0, 2024-03-11T21:28:08.965 INFO:teuthology.orchestra.run.smithi092.stdout: "peering_crush_bucket_barrier": 0, 2024-03-11T21:28:08.965 INFO:teuthology.orchestra.run.smithi092.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-03-11T21:28:08.965 INFO:teuthology.orchestra.run.smithi092.stdout: "object_hash": 2, 2024-03-11T21:28:08.966 INFO:teuthology.orchestra.run.smithi092.stdout: "pg_autoscale_mode": "off", 2024-03-11T21:28:08.966 INFO:teuthology.orchestra.run.smithi092.stdout: "pg_num": 32, 2024-03-11T21:28:08.966 INFO:teuthology.orchestra.run.smithi092.stdout: "pg_placement_num": 32, 2024-03-11T21:28:08.966 INFO:teuthology.orchestra.run.smithi092.stdout: "pg_placement_num_target": 32, 2024-03-11T21:28:08.966 INFO:teuthology.orchestra.run.smithi092.stdout: "pg_num_target": 32, 2024-03-11T21:28:08.966 INFO:teuthology.orchestra.run.smithi092.stdout: "pg_num_pending": 32, 2024-03-11T21:28:08.966 INFO:teuthology.orchestra.run.smithi092.stdout: "last_pg_merge_meta": { 2024-03-11T21:28:08.966 INFO:teuthology.orchestra.run.smithi092.stdout: "source_pgid": "0.0", 2024-03-11T21:28:08.966 INFO:teuthology.orchestra.run.smithi092.stdout: "ready_epoch": 0, 2024-03-11T21:28:08.966 INFO:teuthology.orchestra.run.smithi092.stdout: "last_epoch_started": 0, 2024-03-11T21:28:08.966 INFO:teuthology.orchestra.run.smithi092.stdout: "last_epoch_clean": 0, 2024-03-11T21:28:08.966 INFO:teuthology.orchestra.run.smithi092.stdout: "source_version": "0'0", 2024-03-11T21:28:08.966 INFO:teuthology.orchestra.run.smithi092.stdout: "target_version": "0'0" 2024-03-11T21:28:08.966 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:28:08.967 INFO:teuthology.orchestra.run.smithi092.stdout: "last_change": "68", 2024-03-11T21:28:08.967 INFO:teuthology.orchestra.run.smithi092.stdout: "last_force_op_resend": "0", 2024-03-11T21:28:08.967 INFO:teuthology.orchestra.run.smithi092.stdout: "last_force_op_resend_prenautilus": "0", 2024-03-11T21:28:08.967 INFO:teuthology.orchestra.run.smithi092.stdout: "last_force_op_resend_preluminous": "0", 2024-03-11T21:28:08.967 INFO:teuthology.orchestra.run.smithi092.stdout: "auid": 0, 2024-03-11T21:28:08.967 INFO:teuthology.orchestra.run.smithi092.stdout: "snap_mode": "selfmanaged", 2024-03-11T21:28:08.967 INFO:teuthology.orchestra.run.smithi092.stdout: "snap_seq": 0, 2024-03-11T21:28:08.967 INFO:teuthology.orchestra.run.smithi092.stdout: "snap_epoch": 0, 2024-03-11T21:28:08.967 INFO:teuthology.orchestra.run.smithi092.stdout: "pool_snaps": [], 2024-03-11T21:28:08.967 INFO:teuthology.orchestra.run.smithi092.stdout: "removed_snaps": "[]", 2024-03-11T21:28:08.967 INFO:teuthology.orchestra.run.smithi092.stdout: "quota_max_bytes": 0, 2024-03-11T21:28:08.967 INFO:teuthology.orchestra.run.smithi092.stdout: "quota_max_objects": 0, 2024-03-11T21:28:08.967 INFO:teuthology.orchestra.run.smithi092.stdout: "tiers": [], 2024-03-11T21:28:08.967 INFO:teuthology.orchestra.run.smithi092.stdout: "tier_of": -1, 2024-03-11T21:28:08.967 INFO:teuthology.orchestra.run.smithi092.stdout: "read_tier": -1, 2024-03-11T21:28:08.968 INFO:teuthology.orchestra.run.smithi092.stdout: "write_tier": -1, 2024-03-11T21:28:08.968 INFO:teuthology.orchestra.run.smithi092.stdout: "cache_mode": "none", 2024-03-11T21:28:08.968 INFO:teuthology.orchestra.run.smithi092.stdout: "target_max_bytes": 0, 2024-03-11T21:28:08.968 INFO:teuthology.orchestra.run.smithi092.stdout: "target_max_objects": 0, 2024-03-11T21:28:08.968 INFO:teuthology.orchestra.run.smithi092.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-03-11T21:28:08.968 INFO:teuthology.orchestra.run.smithi092.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-03-11T21:28:08.968 INFO:teuthology.orchestra.run.smithi092.stdout: "cache_target_full_ratio_micro": 800000, 2024-03-11T21:28:08.968 INFO:teuthology.orchestra.run.smithi092.stdout: "cache_min_flush_age": 0, 2024-03-11T21:28:08.968 INFO:teuthology.orchestra.run.smithi092.stdout: "cache_min_evict_age": 0, 2024-03-11T21:28:08.968 INFO:teuthology.orchestra.run.smithi092.stdout: "erasure_code_profile": "", 2024-03-11T21:28:08.968 INFO:teuthology.orchestra.run.smithi092.stdout: "hit_set_params": { 2024-03-11T21:28:08.968 INFO:teuthology.orchestra.run.smithi092.stdout: "type": "none" 2024-03-11T21:28:08.968 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:28:08.968 INFO:teuthology.orchestra.run.smithi092.stdout: "hit_set_period": 0, 2024-03-11T21:28:08.969 INFO:teuthology.orchestra.run.smithi092.stdout: "hit_set_count": 0, 2024-03-11T21:28:08.969 INFO:teuthology.orchestra.run.smithi092.stdout: "use_gmt_hitset": true, 2024-03-11T21:28:08.969 INFO:teuthology.orchestra.run.smithi092.stdout: "min_read_recency_for_promote": 0, 2024-03-11T21:28:08.969 INFO:teuthology.orchestra.run.smithi092.stdout: "min_write_recency_for_promote": 0, 2024-03-11T21:28:08.969 INFO:teuthology.orchestra.run.smithi092.stdout: "hit_set_grade_decay_rate": 0, 2024-03-11T21:28:08.969 INFO:teuthology.orchestra.run.smithi092.stdout: "hit_set_search_last_n": 0, 2024-03-11T21:28:08.969 INFO:teuthology.orchestra.run.smithi092.stdout: "grade_table": [], 2024-03-11T21:28:08.969 INFO:teuthology.orchestra.run.smithi092.stdout: "stripe_width": 0, 2024-03-11T21:28:08.969 INFO:teuthology.orchestra.run.smithi092.stdout: "expected_num_objects": 0, 2024-03-11T21:28:08.969 INFO:teuthology.orchestra.run.smithi092.stdout: "fast_read": false, 2024-03-11T21:28:08.969 INFO:teuthology.orchestra.run.smithi092.stdout: "options": {}, 2024-03-11T21:28:08.969 INFO:teuthology.orchestra.run.smithi092.stdout: "application_metadata": { 2024-03-11T21:28:08.969 INFO:teuthology.orchestra.run.smithi092.stdout: "rgw": {} 2024-03-11T21:28:08.969 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:28:08.969 INFO:teuthology.orchestra.run.smithi092.stdout: "read_balance": { 2024-03-11T21:28:08.970 INFO:teuthology.orchestra.run.smithi092.stdout: "score_type": "Fair distribution", 2024-03-11T21:28:08.970 INFO:teuthology.orchestra.run.smithi092.stdout: "score_acting": 1.5, 2024-03-11T21:28:08.970 INFO:teuthology.orchestra.run.smithi092.stdout: "score_stable": 1.5, 2024-03-11T21:28:08.970 INFO:teuthology.orchestra.run.smithi092.stdout: "optimal_score": 1, 2024-03-11T21:28:08.970 INFO:teuthology.orchestra.run.smithi092.stdout: "raw_score_acting": 1.5, 2024-03-11T21:28:08.970 INFO:teuthology.orchestra.run.smithi092.stdout: "raw_score_stable": 1.5, 2024-03-11T21:28:08.970 INFO:teuthology.orchestra.run.smithi092.stdout: "primary_affinity_weighted": 1, 2024-03-11T21:28:08.970 INFO:teuthology.orchestra.run.smithi092.stdout: "average_primary_affinity": 1, 2024-03-11T21:28:08.970 INFO:teuthology.orchestra.run.smithi092.stdout: "average_primary_affinity_weighted": 1 2024-03-11T21:28:08.970 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-03-11T21:28:08.970 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:28:08.970 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:28:08.970 INFO:teuthology.orchestra.run.smithi092.stdout: "pool": 3, 2024-03-11T21:28:08.970 INFO:teuthology.orchestra.run.smithi092.stdout: "pool_name": "datapool", 2024-03-11T21:28:08.971 INFO:teuthology.orchestra.run.smithi092.stdout: "create_time": "2024-03-11T21:25:37.280820+0000", 2024-03-11T21:28:08.971 INFO:teuthology.orchestra.run.smithi092.stdout: "flags": 8193, 2024-03-11T21:28:08.971 INFO:teuthology.orchestra.run.smithi092.stdout: "flags_names": "hashpspool,selfmanaged_snaps", 2024-03-11T21:28:08.971 INFO:teuthology.orchestra.run.smithi092.stdout: "type": 1, 2024-03-11T21:28:08.971 INFO:teuthology.orchestra.run.smithi092.stdout: "size": 3, 2024-03-11T21:28:08.971 INFO:teuthology.orchestra.run.smithi092.stdout: "min_size": 2, 2024-03-11T21:28:08.971 INFO:teuthology.orchestra.run.smithi092.stdout: "crush_rule": 0, 2024-03-11T21:28:08.971 INFO:teuthology.orchestra.run.smithi092.stdout: "peering_crush_bucket_count": 0, 2024-03-11T21:28:08.971 INFO:teuthology.orchestra.run.smithi092.stdout: "peering_crush_bucket_target": 0, 2024-03-11T21:28:08.971 INFO:teuthology.orchestra.run.smithi092.stdout: "peering_crush_bucket_barrier": 0, 2024-03-11T21:28:08.971 INFO:teuthology.orchestra.run.smithi092.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-03-11T21:28:08.971 INFO:teuthology.orchestra.run.smithi092.stdout: "object_hash": 2, 2024-03-11T21:28:08.971 INFO:teuthology.orchestra.run.smithi092.stdout: "pg_autoscale_mode": "off", 2024-03-11T21:28:08.971 INFO:teuthology.orchestra.run.smithi092.stdout: "pg_num": 3, 2024-03-11T21:28:08.972 INFO:teuthology.orchestra.run.smithi092.stdout: "pg_placement_num": 3, 2024-03-11T21:28:08.972 INFO:teuthology.orchestra.run.smithi092.stdout: "pg_placement_num_target": 3, 2024-03-11T21:28:08.972 INFO:teuthology.orchestra.run.smithi092.stdout: "pg_num_target": 3, 2024-03-11T21:28:08.972 INFO:teuthology.orchestra.run.smithi092.stdout: "pg_num_pending": 3, 2024-03-11T21:28:08.972 INFO:teuthology.orchestra.run.smithi092.stdout: "last_pg_merge_meta": { 2024-03-11T21:28:08.972 INFO:teuthology.orchestra.run.smithi092.stdout: "source_pgid": "0.0", 2024-03-11T21:28:08.972 INFO:teuthology.orchestra.run.smithi092.stdout: "ready_epoch": 0, 2024-03-11T21:28:08.972 INFO:teuthology.orchestra.run.smithi092.stdout: "last_epoch_started": 0, 2024-03-11T21:28:08.972 INFO:teuthology.orchestra.run.smithi092.stdout: "last_epoch_clean": 0, 2024-03-11T21:28:08.972 INFO:teuthology.orchestra.run.smithi092.stdout: "source_version": "0'0", 2024-03-11T21:28:08.972 INFO:teuthology.orchestra.run.smithi092.stdout: "target_version": "0'0" 2024-03-11T21:28:08.972 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:28:08.972 INFO:teuthology.orchestra.run.smithi092.stdout: "last_change": "71", 2024-03-11T21:28:08.972 INFO:teuthology.orchestra.run.smithi092.stdout: "last_force_op_resend": "0", 2024-03-11T21:28:08.973 INFO:teuthology.orchestra.run.smithi092.stdout: "last_force_op_resend_prenautilus": "0", 2024-03-11T21:28:08.973 INFO:teuthology.orchestra.run.smithi092.stdout: "last_force_op_resend_preluminous": "0", 2024-03-11T21:28:08.973 INFO:teuthology.orchestra.run.smithi092.stdout: "auid": 0, 2024-03-11T21:28:08.973 INFO:teuthology.orchestra.run.smithi092.stdout: "snap_mode": "selfmanaged", 2024-03-11T21:28:08.973 INFO:teuthology.orchestra.run.smithi092.stdout: "snap_seq": 2, 2024-03-11T21:28:08.973 INFO:teuthology.orchestra.run.smithi092.stdout: "snap_epoch": 71, 2024-03-11T21:28:08.973 INFO:teuthology.orchestra.run.smithi092.stdout: "pool_snaps": [], 2024-03-11T21:28:08.973 INFO:teuthology.orchestra.run.smithi092.stdout: "removed_snaps": "[]", 2024-03-11T21:28:08.973 INFO:teuthology.orchestra.run.smithi092.stdout: "quota_max_bytes": 0, 2024-03-11T21:28:08.973 INFO:teuthology.orchestra.run.smithi092.stdout: "quota_max_objects": 0, 2024-03-11T21:28:08.973 INFO:teuthology.orchestra.run.smithi092.stdout: "tiers": [], 2024-03-11T21:28:08.973 INFO:teuthology.orchestra.run.smithi092.stdout: "tier_of": -1, 2024-03-11T21:28:08.973 INFO:teuthology.orchestra.run.smithi092.stdout: "read_tier": -1, 2024-03-11T21:28:08.973 INFO:teuthology.orchestra.run.smithi092.stdout: "write_tier": -1, 2024-03-11T21:28:08.973 INFO:teuthology.orchestra.run.smithi092.stdout: "cache_mode": "none", 2024-03-11T21:28:08.974 INFO:teuthology.orchestra.run.smithi092.stdout: "target_max_bytes": 0, 2024-03-11T21:28:08.974 INFO:teuthology.orchestra.run.smithi092.stdout: "target_max_objects": 0, 2024-03-11T21:28:08.974 INFO:teuthology.orchestra.run.smithi092.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-03-11T21:28:08.974 INFO:teuthology.orchestra.run.smithi092.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-03-11T21:28:08.974 INFO:teuthology.orchestra.run.smithi092.stdout: "cache_target_full_ratio_micro": 800000, 2024-03-11T21:28:08.974 INFO:teuthology.orchestra.run.smithi092.stdout: "cache_min_flush_age": 0, 2024-03-11T21:28:08.974 INFO:teuthology.orchestra.run.smithi092.stdout: "cache_min_evict_age": 0, 2024-03-11T21:28:08.974 INFO:teuthology.orchestra.run.smithi092.stdout: "erasure_code_profile": "", 2024-03-11T21:28:08.974 INFO:teuthology.orchestra.run.smithi092.stdout: "hit_set_params": { 2024-03-11T21:28:08.974 INFO:teuthology.orchestra.run.smithi092.stdout: "type": "none" 2024-03-11T21:28:08.974 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:28:08.974 INFO:teuthology.orchestra.run.smithi092.stdout: "hit_set_period": 0, 2024-03-11T21:28:08.974 INFO:teuthology.orchestra.run.smithi092.stdout: "hit_set_count": 0, 2024-03-11T21:28:08.975 INFO:teuthology.orchestra.run.smithi092.stdout: "use_gmt_hitset": true, 2024-03-11T21:28:08.975 INFO:teuthology.orchestra.run.smithi092.stdout: "min_read_recency_for_promote": 0, 2024-03-11T21:28:08.975 INFO:teuthology.orchestra.run.smithi092.stdout: "min_write_recency_for_promote": 0, 2024-03-11T21:28:08.975 INFO:teuthology.orchestra.run.smithi092.stdout: "hit_set_grade_decay_rate": 0, 2024-03-11T21:28:08.975 INFO:teuthology.orchestra.run.smithi092.stdout: "hit_set_search_last_n": 0, 2024-03-11T21:28:08.975 INFO:teuthology.orchestra.run.smithi092.stdout: "grade_table": [], 2024-03-11T21:28:08.975 INFO:teuthology.orchestra.run.smithi092.stdout: "stripe_width": 0, 2024-03-11T21:28:08.975 INFO:teuthology.orchestra.run.smithi092.stdout: "expected_num_objects": 0, 2024-03-11T21:28:08.975 INFO:teuthology.orchestra.run.smithi092.stdout: "fast_read": false, 2024-03-11T21:28:08.975 INFO:teuthology.orchestra.run.smithi092.stdout: "options": {}, 2024-03-11T21:28:08.975 INFO:teuthology.orchestra.run.smithi092.stdout: "application_metadata": { 2024-03-11T21:28:08.975 INFO:teuthology.orchestra.run.smithi092.stdout: "rbd": {} 2024-03-11T21:28:08.975 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:28:08.975 INFO:teuthology.orchestra.run.smithi092.stdout: "read_balance": { 2024-03-11T21:28:08.975 INFO:teuthology.orchestra.run.smithi092.stdout: "score_type": "Fair distribution", 2024-03-11T21:28:08.976 INFO:teuthology.orchestra.run.smithi092.stdout: "score_acting": 2.6500000953674316, 2024-03-11T21:28:08.976 INFO:teuthology.orchestra.run.smithi092.stdout: "score_stable": 2.6500000953674316, 2024-03-11T21:28:08.976 INFO:teuthology.orchestra.run.smithi092.stdout: "optimal_score": 0.87999999523162842, 2024-03-11T21:28:08.976 INFO:teuthology.orchestra.run.smithi092.stdout: "raw_score_acting": 2.3299999237060547, 2024-03-11T21:28:08.976 INFO:teuthology.orchestra.run.smithi092.stdout: "raw_score_stable": 2.3299999237060547, 2024-03-11T21:28:08.976 INFO:teuthology.orchestra.run.smithi092.stdout: "primary_affinity_weighted": 1, 2024-03-11T21:28:08.976 INFO:teuthology.orchestra.run.smithi092.stdout: "average_primary_affinity": 1, 2024-03-11T21:28:08.976 INFO:teuthology.orchestra.run.smithi092.stdout: "average_primary_affinity_weighted": 1 2024-03-11T21:28:08.976 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-03-11T21:28:08.976 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:28:08.976 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:28:08.976 INFO:teuthology.orchestra.run.smithi092.stdout: "pool": 4, 2024-03-11T21:28:08.976 INFO:teuthology.orchestra.run.smithi092.stdout: "pool_name": "default.rgw.log", 2024-03-11T21:28:08.977 INFO:teuthology.orchestra.run.smithi092.stdout: "create_time": "2024-03-11T21:25:38.850207+0000", 2024-03-11T21:28:08.977 INFO:teuthology.orchestra.run.smithi092.stdout: "flags": 1, 2024-03-11T21:28:08.977 INFO:teuthology.orchestra.run.smithi092.stdout: "flags_names": "hashpspool", 2024-03-11T21:28:08.977 INFO:teuthology.orchestra.run.smithi092.stdout: "type": 1, 2024-03-11T21:28:08.977 INFO:teuthology.orchestra.run.smithi092.stdout: "size": 3, 2024-03-11T21:28:08.977 INFO:teuthology.orchestra.run.smithi092.stdout: "min_size": 2, 2024-03-11T21:28:08.977 INFO:teuthology.orchestra.run.smithi092.stdout: "crush_rule": 0, 2024-03-11T21:28:08.977 INFO:teuthology.orchestra.run.smithi092.stdout: "peering_crush_bucket_count": 0, 2024-03-11T21:28:08.977 INFO:teuthology.orchestra.run.smithi092.stdout: "peering_crush_bucket_target": 0, 2024-03-11T21:28:08.977 INFO:teuthology.orchestra.run.smithi092.stdout: "peering_crush_bucket_barrier": 0, 2024-03-11T21:28:08.977 INFO:teuthology.orchestra.run.smithi092.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-03-11T21:28:08.977 INFO:teuthology.orchestra.run.smithi092.stdout: "object_hash": 2, 2024-03-11T21:28:08.977 INFO:teuthology.orchestra.run.smithi092.stdout: "pg_autoscale_mode": "off", 2024-03-11T21:28:08.977 INFO:teuthology.orchestra.run.smithi092.stdout: "pg_num": 32, 2024-03-11T21:28:08.977 INFO:teuthology.orchestra.run.smithi092.stdout: "pg_placement_num": 32, 2024-03-11T21:28:08.978 INFO:teuthology.orchestra.run.smithi092.stdout: "pg_placement_num_target": 32, 2024-03-11T21:28:08.978 INFO:teuthology.orchestra.run.smithi092.stdout: "pg_num_target": 32, 2024-03-11T21:28:08.978 INFO:teuthology.orchestra.run.smithi092.stdout: "pg_num_pending": 32, 2024-03-11T21:28:08.978 INFO:teuthology.orchestra.run.smithi092.stdout: "last_pg_merge_meta": { 2024-03-11T21:28:08.978 INFO:teuthology.orchestra.run.smithi092.stdout: "source_pgid": "0.0", 2024-03-11T21:28:08.978 INFO:teuthology.orchestra.run.smithi092.stdout: "ready_epoch": 0, 2024-03-11T21:28:08.978 INFO:teuthology.orchestra.run.smithi092.stdout: "last_epoch_started": 0, 2024-03-11T21:28:08.978 INFO:teuthology.orchestra.run.smithi092.stdout: "last_epoch_clean": 0, 2024-03-11T21:28:08.978 INFO:teuthology.orchestra.run.smithi092.stdout: "source_version": "0'0", 2024-03-11T21:28:08.978 INFO:teuthology.orchestra.run.smithi092.stdout: "target_version": "0'0" 2024-03-11T21:28:08.978 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:28:08.978 INFO:teuthology.orchestra.run.smithi092.stdout: "last_change": "70", 2024-03-11T21:28:08.978 INFO:teuthology.orchestra.run.smithi092.stdout: "last_force_op_resend": "0", 2024-03-11T21:28:08.978 INFO:teuthology.orchestra.run.smithi092.stdout: "last_force_op_resend_prenautilus": "0", 2024-03-11T21:28:08.979 INFO:teuthology.orchestra.run.smithi092.stdout: "last_force_op_resend_preluminous": "0", 2024-03-11T21:28:08.979 INFO:teuthology.orchestra.run.smithi092.stdout: "auid": 0, 2024-03-11T21:28:08.979 INFO:teuthology.orchestra.run.smithi092.stdout: "snap_mode": "selfmanaged", 2024-03-11T21:28:08.979 INFO:teuthology.orchestra.run.smithi092.stdout: "snap_seq": 0, 2024-03-11T21:28:08.979 INFO:teuthology.orchestra.run.smithi092.stdout: "snap_epoch": 0, 2024-03-11T21:28:08.979 INFO:teuthology.orchestra.run.smithi092.stdout: "pool_snaps": [], 2024-03-11T21:28:08.979 INFO:teuthology.orchestra.run.smithi092.stdout: "removed_snaps": "[]", 2024-03-11T21:28:08.979 INFO:teuthology.orchestra.run.smithi092.stdout: "quota_max_bytes": 0, 2024-03-11T21:28:08.979 INFO:teuthology.orchestra.run.smithi092.stdout: "quota_max_objects": 0, 2024-03-11T21:28:08.979 INFO:teuthology.orchestra.run.smithi092.stdout: "tiers": [], 2024-03-11T21:28:08.979 INFO:teuthology.orchestra.run.smithi092.stdout: "tier_of": -1, 2024-03-11T21:28:08.979 INFO:teuthology.orchestra.run.smithi092.stdout: "read_tier": -1, 2024-03-11T21:28:08.979 INFO:teuthology.orchestra.run.smithi092.stdout: "write_tier": -1, 2024-03-11T21:28:08.979 INFO:teuthology.orchestra.run.smithi092.stdout: "cache_mode": "none", 2024-03-11T21:28:08.979 INFO:teuthology.orchestra.run.smithi092.stdout: "target_max_bytes": 0, 2024-03-11T21:28:08.980 INFO:teuthology.orchestra.run.smithi092.stdout: "target_max_objects": 0, 2024-03-11T21:28:08.980 INFO:teuthology.orchestra.run.smithi092.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-03-11T21:28:08.980 INFO:teuthology.orchestra.run.smithi092.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-03-11T21:28:08.980 INFO:teuthology.orchestra.run.smithi092.stdout: "cache_target_full_ratio_micro": 800000, 2024-03-11T21:28:08.980 INFO:teuthology.orchestra.run.smithi092.stdout: "cache_min_flush_age": 0, 2024-03-11T21:28:08.980 INFO:teuthology.orchestra.run.smithi092.stdout: "cache_min_evict_age": 0, 2024-03-11T21:28:08.980 INFO:teuthology.orchestra.run.smithi092.stdout: "erasure_code_profile": "", 2024-03-11T21:28:08.980 INFO:teuthology.orchestra.run.smithi092.stdout: "hit_set_params": { 2024-03-11T21:28:08.980 INFO:teuthology.orchestra.run.smithi092.stdout: "type": "none" 2024-03-11T21:28:08.980 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:28:08.980 INFO:teuthology.orchestra.run.smithi092.stdout: "hit_set_period": 0, 2024-03-11T21:28:08.980 INFO:teuthology.orchestra.run.smithi092.stdout: "hit_set_count": 0, 2024-03-11T21:28:08.980 INFO:teuthology.orchestra.run.smithi092.stdout: "use_gmt_hitset": true, 2024-03-11T21:28:08.980 INFO:teuthology.orchestra.run.smithi092.stdout: "min_read_recency_for_promote": 0, 2024-03-11T21:28:08.981 INFO:teuthology.orchestra.run.smithi092.stdout: "min_write_recency_for_promote": 0, 2024-03-11T21:28:08.981 INFO:teuthology.orchestra.run.smithi092.stdout: "hit_set_grade_decay_rate": 0, 2024-03-11T21:28:08.981 INFO:teuthology.orchestra.run.smithi092.stdout: "hit_set_search_last_n": 0, 2024-03-11T21:28:08.981 INFO:teuthology.orchestra.run.smithi092.stdout: "grade_table": [], 2024-03-11T21:28:08.981 INFO:teuthology.orchestra.run.smithi092.stdout: "stripe_width": 0, 2024-03-11T21:28:08.981 INFO:teuthology.orchestra.run.smithi092.stdout: "expected_num_objects": 0, 2024-03-11T21:28:08.981 INFO:teuthology.orchestra.run.smithi092.stdout: "fast_read": false, 2024-03-11T21:28:08.981 INFO:teuthology.orchestra.run.smithi092.stdout: "options": {}, 2024-03-11T21:28:08.981 INFO:teuthology.orchestra.run.smithi092.stdout: "application_metadata": { 2024-03-11T21:28:08.981 INFO:teuthology.orchestra.run.smithi092.stdout: "rgw": {} 2024-03-11T21:28:08.981 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:28:08.981 INFO:teuthology.orchestra.run.smithi092.stdout: "read_balance": { 2024-03-11T21:28:08.981 INFO:teuthology.orchestra.run.smithi092.stdout: "score_type": "Fair distribution", 2024-03-11T21:28:08.982 INFO:teuthology.orchestra.run.smithi092.stdout: "score_acting": 2.25, 2024-03-11T21:28:08.982 INFO:teuthology.orchestra.run.smithi092.stdout: "score_stable": 2.25, 2024-03-11T21:28:08.982 INFO:teuthology.orchestra.run.smithi092.stdout: "optimal_score": 1, 2024-03-11T21:28:08.982 INFO:teuthology.orchestra.run.smithi092.stdout: "raw_score_acting": 2.25, 2024-03-11T21:28:08.982 INFO:teuthology.orchestra.run.smithi092.stdout: "raw_score_stable": 2.25, 2024-03-11T21:28:08.982 INFO:teuthology.orchestra.run.smithi092.stdout: "primary_affinity_weighted": 1, 2024-03-11T21:28:08.982 INFO:teuthology.orchestra.run.smithi092.stdout: "average_primary_affinity": 1, 2024-03-11T21:28:08.982 INFO:teuthology.orchestra.run.smithi092.stdout: "average_primary_affinity_weighted": 1 2024-03-11T21:28:08.982 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-03-11T21:28:08.982 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:28:08.982 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:28:08.982 INFO:teuthology.orchestra.run.smithi092.stdout: "pool": 5, 2024-03-11T21:28:08.982 INFO:teuthology.orchestra.run.smithi092.stdout: "pool_name": "default.rgw.control", 2024-03-11T21:28:08.982 INFO:teuthology.orchestra.run.smithi092.stdout: "create_time": "2024-03-11T21:25:40.794686+0000", 2024-03-11T21:28:08.983 INFO:teuthology.orchestra.run.smithi092.stdout: "flags": 1, 2024-03-11T21:28:08.983 INFO:teuthology.orchestra.run.smithi092.stdout: "flags_names": "hashpspool", 2024-03-11T21:28:08.983 INFO:teuthology.orchestra.run.smithi092.stdout: "type": 1, 2024-03-11T21:28:08.983 INFO:teuthology.orchestra.run.smithi092.stdout: "size": 3, 2024-03-11T21:28:08.983 INFO:teuthology.orchestra.run.smithi092.stdout: "min_size": 2, 2024-03-11T21:28:08.983 INFO:teuthology.orchestra.run.smithi092.stdout: "crush_rule": 0, 2024-03-11T21:28:08.983 INFO:teuthology.orchestra.run.smithi092.stdout: "peering_crush_bucket_count": 0, 2024-03-11T21:28:08.983 INFO:teuthology.orchestra.run.smithi092.stdout: "peering_crush_bucket_target": 0, 2024-03-11T21:28:08.983 INFO:teuthology.orchestra.run.smithi092.stdout: "peering_crush_bucket_barrier": 0, 2024-03-11T21:28:08.983 INFO:teuthology.orchestra.run.smithi092.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-03-11T21:28:08.983 INFO:teuthology.orchestra.run.smithi092.stdout: "object_hash": 2, 2024-03-11T21:28:08.983 INFO:teuthology.orchestra.run.smithi092.stdout: "pg_autoscale_mode": "off", 2024-03-11T21:28:08.983 INFO:teuthology.orchestra.run.smithi092.stdout: "pg_num": 32, 2024-03-11T21:28:08.983 INFO:teuthology.orchestra.run.smithi092.stdout: "pg_placement_num": 32, 2024-03-11T21:28:08.983 INFO:teuthology.orchestra.run.smithi092.stdout: "pg_placement_num_target": 32, 2024-03-11T21:28:08.984 INFO:teuthology.orchestra.run.smithi092.stdout: "pg_num_target": 32, 2024-03-11T21:28:08.984 INFO:teuthology.orchestra.run.smithi092.stdout: "pg_num_pending": 32, 2024-03-11T21:28:08.984 INFO:teuthology.orchestra.run.smithi092.stdout: "last_pg_merge_meta": { 2024-03-11T21:28:08.984 INFO:teuthology.orchestra.run.smithi092.stdout: "source_pgid": "0.0", 2024-03-11T21:28:08.984 INFO:teuthology.orchestra.run.smithi092.stdout: "ready_epoch": 0, 2024-03-11T21:28:08.984 INFO:teuthology.orchestra.run.smithi092.stdout: "last_epoch_started": 0, 2024-03-11T21:28:08.984 INFO:teuthology.orchestra.run.smithi092.stdout: "last_epoch_clean": 0, 2024-03-11T21:28:08.984 INFO:teuthology.orchestra.run.smithi092.stdout: "source_version": "0'0", 2024-03-11T21:28:08.984 INFO:teuthology.orchestra.run.smithi092.stdout: "target_version": "0'0" 2024-03-11T21:28:08.984 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:28:08.984 INFO:teuthology.orchestra.run.smithi092.stdout: "last_change": "72", 2024-03-11T21:28:08.984 INFO:teuthology.orchestra.run.smithi092.stdout: "last_force_op_resend": "0", 2024-03-11T21:28:08.984 INFO:teuthology.orchestra.run.smithi092.stdout: "last_force_op_resend_prenautilus": "0", 2024-03-11T21:28:08.984 INFO:teuthology.orchestra.run.smithi092.stdout: "last_force_op_resend_preluminous": "0", 2024-03-11T21:28:08.984 INFO:teuthology.orchestra.run.smithi092.stdout: "auid": 0, 2024-03-11T21:28:08.984 INFO:teuthology.orchestra.run.smithi092.stdout: "snap_mode": "selfmanaged", 2024-03-11T21:28:08.985 INFO:teuthology.orchestra.run.smithi092.stdout: "snap_seq": 0, 2024-03-11T21:28:08.985 INFO:teuthology.orchestra.run.smithi092.stdout: "snap_epoch": 0, 2024-03-11T21:28:08.985 INFO:teuthology.orchestra.run.smithi092.stdout: "pool_snaps": [], 2024-03-11T21:28:08.985 INFO:teuthology.orchestra.run.smithi092.stdout: "removed_snaps": "[]", 2024-03-11T21:28:08.985 INFO:teuthology.orchestra.run.smithi092.stdout: "quota_max_bytes": 0, 2024-03-11T21:28:08.985 INFO:teuthology.orchestra.run.smithi092.stdout: "quota_max_objects": 0, 2024-03-11T21:28:08.985 INFO:teuthology.orchestra.run.smithi092.stdout: "tiers": [], 2024-03-11T21:28:08.985 INFO:teuthology.orchestra.run.smithi092.stdout: "tier_of": -1, 2024-03-11T21:28:08.985 INFO:teuthology.orchestra.run.smithi092.stdout: "read_tier": -1, 2024-03-11T21:28:08.985 INFO:teuthology.orchestra.run.smithi092.stdout: "write_tier": -1, 2024-03-11T21:28:08.985 INFO:teuthology.orchestra.run.smithi092.stdout: "cache_mode": "none", 2024-03-11T21:28:08.985 INFO:teuthology.orchestra.run.smithi092.stdout: "target_max_bytes": 0, 2024-03-11T21:28:08.985 INFO:teuthology.orchestra.run.smithi092.stdout: "target_max_objects": 0, 2024-03-11T21:28:08.985 INFO:teuthology.orchestra.run.smithi092.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-03-11T21:28:08.986 INFO:teuthology.orchestra.run.smithi092.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-03-11T21:28:08.986 INFO:teuthology.orchestra.run.smithi092.stdout: "cache_target_full_ratio_micro": 800000, 2024-03-11T21:28:08.986 INFO:teuthology.orchestra.run.smithi092.stdout: "cache_min_flush_age": 0, 2024-03-11T21:28:08.986 INFO:teuthology.orchestra.run.smithi092.stdout: "cache_min_evict_age": 0, 2024-03-11T21:28:08.986 INFO:teuthology.orchestra.run.smithi092.stdout: "erasure_code_profile": "", 2024-03-11T21:28:08.986 INFO:teuthology.orchestra.run.smithi092.stdout: "hit_set_params": { 2024-03-11T21:28:08.986 INFO:teuthology.orchestra.run.smithi092.stdout: "type": "none" 2024-03-11T21:28:08.986 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:28:08.986 INFO:teuthology.orchestra.run.smithi092.stdout: "hit_set_period": 0, 2024-03-11T21:28:08.986 INFO:teuthology.orchestra.run.smithi092.stdout: "hit_set_count": 0, 2024-03-11T21:28:08.986 INFO:teuthology.orchestra.run.smithi092.stdout: "use_gmt_hitset": true, 2024-03-11T21:28:08.986 INFO:teuthology.orchestra.run.smithi092.stdout: "min_read_recency_for_promote": 0, 2024-03-11T21:28:08.986 INFO:teuthology.orchestra.run.smithi092.stdout: "min_write_recency_for_promote": 0, 2024-03-11T21:28:08.986 INFO:teuthology.orchestra.run.smithi092.stdout: "hit_set_grade_decay_rate": 0, 2024-03-11T21:28:08.986 INFO:teuthology.orchestra.run.smithi092.stdout: "hit_set_search_last_n": 0, 2024-03-11T21:28:08.986 INFO:teuthology.orchestra.run.smithi092.stdout: "grade_table": [], 2024-03-11T21:28:08.987 INFO:teuthology.orchestra.run.smithi092.stdout: "stripe_width": 0, 2024-03-11T21:28:08.987 INFO:teuthology.orchestra.run.smithi092.stdout: "expected_num_objects": 0, 2024-03-11T21:28:08.987 INFO:teuthology.orchestra.run.smithi092.stdout: "fast_read": false, 2024-03-11T21:28:08.987 INFO:teuthology.orchestra.run.smithi092.stdout: "options": {}, 2024-03-11T21:28:08.987 INFO:teuthology.orchestra.run.smithi092.stdout: "application_metadata": { 2024-03-11T21:28:08.987 INFO:teuthology.orchestra.run.smithi092.stdout: "rgw": {} 2024-03-11T21:28:08.987 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:28:08.987 INFO:teuthology.orchestra.run.smithi092.stdout: "read_balance": { 2024-03-11T21:28:08.987 INFO:teuthology.orchestra.run.smithi092.stdout: "score_type": "Fair distribution", 2024-03-11T21:28:08.987 INFO:teuthology.orchestra.run.smithi092.stdout: "score_acting": 1.25, 2024-03-11T21:28:08.987 INFO:teuthology.orchestra.run.smithi092.stdout: "score_stable": 1.25, 2024-03-11T21:28:08.987 INFO:teuthology.orchestra.run.smithi092.stdout: "optimal_score": 1, 2024-03-11T21:28:08.987 INFO:teuthology.orchestra.run.smithi092.stdout: "raw_score_acting": 1.25, 2024-03-11T21:28:08.987 INFO:teuthology.orchestra.run.smithi092.stdout: "raw_score_stable": 1.25, 2024-03-11T21:28:08.987 INFO:teuthology.orchestra.run.smithi092.stdout: "primary_affinity_weighted": 1, 2024-03-11T21:28:08.987 INFO:teuthology.orchestra.run.smithi092.stdout: "average_primary_affinity": 1, 2024-03-11T21:28:08.988 INFO:teuthology.orchestra.run.smithi092.stdout: "average_primary_affinity_weighted": 1 2024-03-11T21:28:08.988 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-03-11T21:28:08.988 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:28:08.988 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:28:08.988 INFO:teuthology.orchestra.run.smithi092.stdout: "pool": 6, 2024-03-11T21:28:08.988 INFO:teuthology.orchestra.run.smithi092.stdout: "pool_name": "default.rgw.meta", 2024-03-11T21:28:08.988 INFO:teuthology.orchestra.run.smithi092.stdout: "create_time": "2024-03-11T21:25:42.872077+0000", 2024-03-11T21:28:08.988 INFO:teuthology.orchestra.run.smithi092.stdout: "flags": 1, 2024-03-11T21:28:08.988 INFO:teuthology.orchestra.run.smithi092.stdout: "flags_names": "hashpspool", 2024-03-11T21:28:08.989 INFO:teuthology.orchestra.run.smithi092.stdout: "type": 1, 2024-03-11T21:28:08.989 INFO:teuthology.orchestra.run.smithi092.stdout: "size": 3, 2024-03-11T21:28:08.989 INFO:teuthology.orchestra.run.smithi092.stdout: "min_size": 2, 2024-03-11T21:28:08.989 INFO:teuthology.orchestra.run.smithi092.stdout: "crush_rule": 0, 2024-03-11T21:28:08.989 INFO:teuthology.orchestra.run.smithi092.stdout: "peering_crush_bucket_count": 0, 2024-03-11T21:28:08.989 INFO:teuthology.orchestra.run.smithi092.stdout: "peering_crush_bucket_target": 0, 2024-03-11T21:28:08.989 INFO:teuthology.orchestra.run.smithi092.stdout: "peering_crush_bucket_barrier": 0, 2024-03-11T21:28:08.989 INFO:teuthology.orchestra.run.smithi092.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-03-11T21:28:08.989 INFO:teuthology.orchestra.run.smithi092.stdout: "object_hash": 2, 2024-03-11T21:28:08.989 INFO:teuthology.orchestra.run.smithi092.stdout: "pg_autoscale_mode": "off", 2024-03-11T21:28:08.989 INFO:teuthology.orchestra.run.smithi092.stdout: "pg_num": 32, 2024-03-11T21:28:08.989 INFO:teuthology.orchestra.run.smithi092.stdout: "pg_placement_num": 32, 2024-03-11T21:28:08.989 INFO:teuthology.orchestra.run.smithi092.stdout: "pg_placement_num_target": 32, 2024-03-11T21:28:08.989 INFO:teuthology.orchestra.run.smithi092.stdout: "pg_num_target": 32, 2024-03-11T21:28:08.990 INFO:teuthology.orchestra.run.smithi092.stdout: "pg_num_pending": 32, 2024-03-11T21:28:08.990 INFO:teuthology.orchestra.run.smithi092.stdout: "last_pg_merge_meta": { 2024-03-11T21:28:08.990 INFO:teuthology.orchestra.run.smithi092.stdout: "source_pgid": "0.0", 2024-03-11T21:28:08.990 INFO:teuthology.orchestra.run.smithi092.stdout: "ready_epoch": 0, 2024-03-11T21:28:08.990 INFO:teuthology.orchestra.run.smithi092.stdout: "last_epoch_started": 0, 2024-03-11T21:28:08.990 INFO:teuthology.orchestra.run.smithi092.stdout: "last_epoch_clean": 0, 2024-03-11T21:28:08.990 INFO:teuthology.orchestra.run.smithi092.stdout: "source_version": "0'0", 2024-03-11T21:28:08.990 INFO:teuthology.orchestra.run.smithi092.stdout: "target_version": "0'0" 2024-03-11T21:28:08.990 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:28:08.990 INFO:teuthology.orchestra.run.smithi092.stdout: "last_change": "74", 2024-03-11T21:28:08.990 INFO:teuthology.orchestra.run.smithi092.stdout: "last_force_op_resend": "0", 2024-03-11T21:28:08.990 INFO:teuthology.orchestra.run.smithi092.stdout: "last_force_op_resend_prenautilus": "0", 2024-03-11T21:28:08.990 INFO:teuthology.orchestra.run.smithi092.stdout: "last_force_op_resend_preluminous": "0", 2024-03-11T21:28:08.991 INFO:teuthology.orchestra.run.smithi092.stdout: "auid": 0, 2024-03-11T21:28:08.991 INFO:teuthology.orchestra.run.smithi092.stdout: "snap_mode": "selfmanaged", 2024-03-11T21:28:08.991 INFO:teuthology.orchestra.run.smithi092.stdout: "snap_seq": 0, 2024-03-11T21:28:08.991 INFO:teuthology.orchestra.run.smithi092.stdout: "snap_epoch": 0, 2024-03-11T21:28:08.991 INFO:teuthology.orchestra.run.smithi092.stdout: "pool_snaps": [], 2024-03-11T21:28:08.991 INFO:teuthology.orchestra.run.smithi092.stdout: "removed_snaps": "[]", 2024-03-11T21:28:08.991 INFO:teuthology.orchestra.run.smithi092.stdout: "quota_max_bytes": 0, 2024-03-11T21:28:08.991 INFO:teuthology.orchestra.run.smithi092.stdout: "quota_max_objects": 0, 2024-03-11T21:28:08.991 INFO:teuthology.orchestra.run.smithi092.stdout: "tiers": [], 2024-03-11T21:28:08.991 INFO:teuthology.orchestra.run.smithi092.stdout: "tier_of": -1, 2024-03-11T21:28:08.991 INFO:teuthology.orchestra.run.smithi092.stdout: "read_tier": -1, 2024-03-11T21:28:08.991 INFO:teuthology.orchestra.run.smithi092.stdout: "write_tier": -1, 2024-03-11T21:28:08.991 INFO:teuthology.orchestra.run.smithi092.stdout: "cache_mode": "none", 2024-03-11T21:28:08.991 INFO:teuthology.orchestra.run.smithi092.stdout: "target_max_bytes": 0, 2024-03-11T21:28:08.992 INFO:teuthology.orchestra.run.smithi092.stdout: "target_max_objects": 0, 2024-03-11T21:28:08.992 INFO:teuthology.orchestra.run.smithi092.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-03-11T21:28:08.992 INFO:teuthology.orchestra.run.smithi092.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-03-11T21:28:08.992 INFO:teuthology.orchestra.run.smithi092.stdout: "cache_target_full_ratio_micro": 800000, 2024-03-11T21:28:08.992 INFO:teuthology.orchestra.run.smithi092.stdout: "cache_min_flush_age": 0, 2024-03-11T21:28:08.992 INFO:teuthology.orchestra.run.smithi092.stdout: "cache_min_evict_age": 0, 2024-03-11T21:28:08.992 INFO:teuthology.orchestra.run.smithi092.stdout: "erasure_code_profile": "", 2024-03-11T21:28:08.992 INFO:teuthology.orchestra.run.smithi092.stdout: "hit_set_params": { 2024-03-11T21:28:08.992 INFO:teuthology.orchestra.run.smithi092.stdout: "type": "none" 2024-03-11T21:28:08.992 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:28:08.992 INFO:teuthology.orchestra.run.smithi092.stdout: "hit_set_period": 0, 2024-03-11T21:28:08.992 INFO:teuthology.orchestra.run.smithi092.stdout: "hit_set_count": 0, 2024-03-11T21:28:08.992 INFO:teuthology.orchestra.run.smithi092.stdout: "use_gmt_hitset": true, 2024-03-11T21:28:08.992 INFO:teuthology.orchestra.run.smithi092.stdout: "min_read_recency_for_promote": 0, 2024-03-11T21:28:08.993 INFO:teuthology.orchestra.run.smithi092.stdout: "min_write_recency_for_promote": 0, 2024-03-11T21:28:08.993 INFO:teuthology.orchestra.run.smithi092.stdout: "hit_set_grade_decay_rate": 0, 2024-03-11T21:28:08.993 INFO:teuthology.orchestra.run.smithi092.stdout: "hit_set_search_last_n": 0, 2024-03-11T21:28:08.993 INFO:teuthology.orchestra.run.smithi092.stdout: "grade_table": [], 2024-03-11T21:28:08.993 INFO:teuthology.orchestra.run.smithi092.stdout: "stripe_width": 0, 2024-03-11T21:28:08.993 INFO:teuthology.orchestra.run.smithi092.stdout: "expected_num_objects": 0, 2024-03-11T21:28:08.993 INFO:teuthology.orchestra.run.smithi092.stdout: "fast_read": false, 2024-03-11T21:28:08.993 INFO:teuthology.orchestra.run.smithi092.stdout: "options": { 2024-03-11T21:28:08.993 INFO:teuthology.orchestra.run.smithi092.stdout: "pg_autoscale_bias": 4 2024-03-11T21:28:08.993 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:28:08.993 INFO:teuthology.orchestra.run.smithi092.stdout: "application_metadata": { 2024-03-11T21:28:08.993 INFO:teuthology.orchestra.run.smithi092.stdout: "rgw": {} 2024-03-11T21:28:08.993 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:28:08.993 INFO:teuthology.orchestra.run.smithi092.stdout: "read_balance": { 2024-03-11T21:28:08.993 INFO:teuthology.orchestra.run.smithi092.stdout: "score_type": "Fair distribution", 2024-03-11T21:28:08.994 INFO:teuthology.orchestra.run.smithi092.stdout: "score_acting": 1.75, 2024-03-11T21:28:08.994 INFO:teuthology.orchestra.run.smithi092.stdout: "score_stable": 1.75, 2024-03-11T21:28:08.994 INFO:teuthology.orchestra.run.smithi092.stdout: "optimal_score": 1, 2024-03-11T21:28:08.994 INFO:teuthology.orchestra.run.smithi092.stdout: "raw_score_acting": 1.75, 2024-03-11T21:28:08.994 INFO:teuthology.orchestra.run.smithi092.stdout: "raw_score_stable": 1.75, 2024-03-11T21:28:08.994 INFO:teuthology.orchestra.run.smithi092.stdout: "primary_affinity_weighted": 1, 2024-03-11T21:28:08.994 INFO:teuthology.orchestra.run.smithi092.stdout: "average_primary_affinity": 1, 2024-03-11T21:28:08.994 INFO:teuthology.orchestra.run.smithi092.stdout: "average_primary_affinity_weighted": 1 2024-03-11T21:28:08.994 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-03-11T21:28:08.994 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:28:08.994 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:28:08.994 INFO:teuthology.orchestra.run.smithi092.stdout: "pool": 7, 2024-03-11T21:28:08.994 INFO:teuthology.orchestra.run.smithi092.stdout: "pool_name": "unique_pool_0", 2024-03-11T21:28:08.994 INFO:teuthology.orchestra.run.smithi092.stdout: "create_time": "2024-03-11T21:27:35.489822+0000", 2024-03-11T21:28:08.995 INFO:teuthology.orchestra.run.smithi092.stdout: "flags": 1, 2024-03-11T21:28:08.995 INFO:teuthology.orchestra.run.smithi092.stdout: "flags_names": "hashpspool", 2024-03-11T21:28:08.995 INFO:teuthology.orchestra.run.smithi092.stdout: "type": 1, 2024-03-11T21:28:08.995 INFO:teuthology.orchestra.run.smithi092.stdout: "size": 3, 2024-03-11T21:28:08.995 INFO:teuthology.orchestra.run.smithi092.stdout: "min_size": 2, 2024-03-11T21:28:08.995 INFO:teuthology.orchestra.run.smithi092.stdout: "crush_rule": 0, 2024-03-11T21:28:08.995 INFO:teuthology.orchestra.run.smithi092.stdout: "peering_crush_bucket_count": 0, 2024-03-11T21:28:08.995 INFO:teuthology.orchestra.run.smithi092.stdout: "peering_crush_bucket_target": 0, 2024-03-11T21:28:08.995 INFO:teuthology.orchestra.run.smithi092.stdout: "peering_crush_bucket_barrier": 0, 2024-03-11T21:28:08.995 INFO:teuthology.orchestra.run.smithi092.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-03-11T21:28:08.995 INFO:teuthology.orchestra.run.smithi092.stdout: "object_hash": 2, 2024-03-11T21:28:08.995 INFO:teuthology.orchestra.run.smithi092.stdout: "pg_autoscale_mode": "off", 2024-03-11T21:28:08.995 INFO:teuthology.orchestra.run.smithi092.stdout: "pg_num": 16, 2024-03-11T21:28:08.995 INFO:teuthology.orchestra.run.smithi092.stdout: "pg_placement_num": 16, 2024-03-11T21:28:08.996 INFO:teuthology.orchestra.run.smithi092.stdout: "pg_placement_num_target": 16, 2024-03-11T21:28:08.996 INFO:teuthology.orchestra.run.smithi092.stdout: "pg_num_target": 16, 2024-03-11T21:28:08.996 INFO:teuthology.orchestra.run.smithi092.stdout: "pg_num_pending": 16, 2024-03-11T21:28:08.996 INFO:teuthology.orchestra.run.smithi092.stdout: "last_pg_merge_meta": { 2024-03-11T21:28:08.996 INFO:teuthology.orchestra.run.smithi092.stdout: "source_pgid": "0.0", 2024-03-11T21:28:08.996 INFO:teuthology.orchestra.run.smithi092.stdout: "ready_epoch": 0, 2024-03-11T21:28:08.996 INFO:teuthology.orchestra.run.smithi092.stdout: "last_epoch_started": 0, 2024-03-11T21:28:08.996 INFO:teuthology.orchestra.run.smithi092.stdout: "last_epoch_clean": 0, 2024-03-11T21:28:08.996 INFO:teuthology.orchestra.run.smithi092.stdout: "source_version": "0'0", 2024-03-11T21:28:08.996 INFO:teuthology.orchestra.run.smithi092.stdout: "target_version": "0'0" 2024-03-11T21:28:08.996 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:28:08.996 INFO:teuthology.orchestra.run.smithi092.stdout: "last_change": "104", 2024-03-11T21:28:08.996 INFO:teuthology.orchestra.run.smithi092.stdout: "last_force_op_resend": "0", 2024-03-11T21:28:08.996 INFO:teuthology.orchestra.run.smithi092.stdout: "last_force_op_resend_prenautilus": "0", 2024-03-11T21:28:08.996 INFO:teuthology.orchestra.run.smithi092.stdout: "last_force_op_resend_preluminous": "0", 2024-03-11T21:28:08.997 INFO:teuthology.orchestra.run.smithi092.stdout: "auid": 0, 2024-03-11T21:28:08.997 INFO:teuthology.orchestra.run.smithi092.stdout: "snap_mode": "selfmanaged", 2024-03-11T21:28:08.997 INFO:teuthology.orchestra.run.smithi092.stdout: "snap_seq": 0, 2024-03-11T21:28:08.997 INFO:teuthology.orchestra.run.smithi092.stdout: "snap_epoch": 0, 2024-03-11T21:28:08.997 INFO:teuthology.orchestra.run.smithi092.stdout: "pool_snaps": [], 2024-03-11T21:28:08.997 INFO:teuthology.orchestra.run.smithi092.stdout: "removed_snaps": "[]", 2024-03-11T21:28:08.997 INFO:teuthology.orchestra.run.smithi092.stdout: "quota_max_bytes": 0, 2024-03-11T21:28:08.997 INFO:teuthology.orchestra.run.smithi092.stdout: "quota_max_objects": 0, 2024-03-11T21:28:08.997 INFO:teuthology.orchestra.run.smithi092.stdout: "tiers": [], 2024-03-11T21:28:08.997 INFO:teuthology.orchestra.run.smithi092.stdout: "tier_of": -1, 2024-03-11T21:28:08.997 INFO:teuthology.orchestra.run.smithi092.stdout: "read_tier": -1, 2024-03-11T21:28:08.997 INFO:teuthology.orchestra.run.smithi092.stdout: "write_tier": -1, 2024-03-11T21:28:08.997 INFO:teuthology.orchestra.run.smithi092.stdout: "cache_mode": "none", 2024-03-11T21:28:08.997 INFO:teuthology.orchestra.run.smithi092.stdout: "target_max_bytes": 0, 2024-03-11T21:28:08.998 INFO:teuthology.orchestra.run.smithi092.stdout: "target_max_objects": 0, 2024-03-11T21:28:08.998 INFO:teuthology.orchestra.run.smithi092.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-03-11T21:28:08.998 INFO:teuthology.orchestra.run.smithi092.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-03-11T21:28:08.998 INFO:teuthology.orchestra.run.smithi092.stdout: "cache_target_full_ratio_micro": 800000, 2024-03-11T21:28:08.998 INFO:teuthology.orchestra.run.smithi092.stdout: "cache_min_flush_age": 0, 2024-03-11T21:28:08.998 INFO:teuthology.orchestra.run.smithi092.stdout: "cache_min_evict_age": 0, 2024-03-11T21:28:08.998 INFO:teuthology.orchestra.run.smithi092.stdout: "erasure_code_profile": "", 2024-03-11T21:28:08.998 INFO:teuthology.orchestra.run.smithi092.stdout: "hit_set_params": { 2024-03-11T21:28:08.998 INFO:teuthology.orchestra.run.smithi092.stdout: "type": "none" 2024-03-11T21:28:08.998 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:28:08.998 INFO:teuthology.orchestra.run.smithi092.stdout: "hit_set_period": 0, 2024-03-11T21:28:08.998 INFO:teuthology.orchestra.run.smithi092.stdout: "hit_set_count": 0, 2024-03-11T21:28:08.998 INFO:teuthology.orchestra.run.smithi092.stdout: "use_gmt_hitset": true, 2024-03-11T21:28:08.998 INFO:teuthology.orchestra.run.smithi092.stdout: "min_read_recency_for_promote": 0, 2024-03-11T21:28:08.999 INFO:teuthology.orchestra.run.smithi092.stdout: "min_write_recency_for_promote": 0, 2024-03-11T21:28:08.999 INFO:teuthology.orchestra.run.smithi092.stdout: "hit_set_grade_decay_rate": 0, 2024-03-11T21:28:08.999 INFO:teuthology.orchestra.run.smithi092.stdout: "hit_set_search_last_n": 0, 2024-03-11T21:28:08.999 INFO:teuthology.orchestra.run.smithi092.stdout: "grade_table": [], 2024-03-11T21:28:08.999 INFO:teuthology.orchestra.run.smithi092.stdout: "stripe_width": 0, 2024-03-11T21:28:08.999 INFO:teuthology.orchestra.run.smithi092.stdout: "expected_num_objects": 0, 2024-03-11T21:28:08.999 INFO:teuthology.orchestra.run.smithi092.stdout: "fast_read": false, 2024-03-11T21:28:08.999 INFO:teuthology.orchestra.run.smithi092.stdout: "options": {}, 2024-03-11T21:28:08.999 INFO:teuthology.orchestra.run.smithi092.stdout: "application_metadata": { 2024-03-11T21:28:08.999 INFO:teuthology.orchestra.run.smithi092.stdout: "rados": {} 2024-03-11T21:28:08.999 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:28:08.999 INFO:teuthology.orchestra.run.smithi092.stdout: "read_balance": { 2024-03-11T21:28:08.999 INFO:teuthology.orchestra.run.smithi092.stdout: "score_type": "Fair distribution", 2024-03-11T21:28:08.999 INFO:teuthology.orchestra.run.smithi092.stdout: "score_acting": 2, 2024-03-11T21:28:09.000 INFO:teuthology.orchestra.run.smithi092.stdout: "score_stable": 2, 2024-03-11T21:28:09.000 INFO:teuthology.orchestra.run.smithi092.stdout: "optimal_score": 1, 2024-03-11T21:28:09.000 INFO:teuthology.orchestra.run.smithi092.stdout: "raw_score_acting": 2, 2024-03-11T21:28:09.000 INFO:teuthology.orchestra.run.smithi092.stdout: "raw_score_stable": 2, 2024-03-11T21:28:09.000 INFO:teuthology.orchestra.run.smithi092.stdout: "primary_affinity_weighted": 1, 2024-03-11T21:28:09.000 INFO:teuthology.orchestra.run.smithi092.stdout: "average_primary_affinity": 1, 2024-03-11T21:28:09.000 INFO:teuthology.orchestra.run.smithi092.stdout: "average_primary_affinity_weighted": 1 2024-03-11T21:28:09.000 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-03-11T21:28:09.000 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-03-11T21:28:09.000 INFO:teuthology.orchestra.run.smithi092.stdout: ], 2024-03-11T21:28:09.000 INFO:teuthology.orchestra.run.smithi092.stdout: "osds": [ 2024-03-11T21:28:09.000 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:28:09.000 INFO:teuthology.orchestra.run.smithi092.stdout: "osd": 0, 2024-03-11T21:28:09.000 INFO:teuthology.orchestra.run.smithi092.stdout: "uuid": "79f15843-f0ec-44aa-931e-4e07a74d3768", 2024-03-11T21:28:09.001 INFO:teuthology.orchestra.run.smithi092.stdout: "up": 1, 2024-03-11T21:28:09.001 INFO:teuthology.orchestra.run.smithi092.stdout: "in": 1, 2024-03-11T21:28:09.001 INFO:teuthology.orchestra.run.smithi092.stdout: "weight": 1, 2024-03-11T21:28:09.001 INFO:teuthology.orchestra.run.smithi092.stdout: "primary_affinity": 1, 2024-03-11T21:28:09.001 INFO:teuthology.orchestra.run.smithi092.stdout: "last_clean_begin": 0, 2024-03-11T21:28:09.001 INFO:teuthology.orchestra.run.smithi092.stdout: "last_clean_end": 0, 2024-03-11T21:28:09.001 INFO:teuthology.orchestra.run.smithi092.stdout: "up_from": 8, 2024-03-11T21:28:09.001 INFO:teuthology.orchestra.run.smithi092.stdout: "up_thru": 99, 2024-03-11T21:28:09.001 INFO:teuthology.orchestra.run.smithi092.stdout: "down_at": 0, 2024-03-11T21:28:09.001 INFO:teuthology.orchestra.run.smithi092.stdout: "lost_at": 0, 2024-03-11T21:28:09.001 INFO:teuthology.orchestra.run.smithi092.stdout: "public_addrs": { 2024-03-11T21:28:09.001 INFO:teuthology.orchestra.run.smithi092.stdout: "addrvec": [ 2024-03-11T21:28:09.001 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:28:09.001 INFO:teuthology.orchestra.run.smithi092.stdout: "type": "v1", 2024-03-11T21:28:09.002 INFO:teuthology.orchestra.run.smithi092.stdout: "addr": "172.21.15.92:6801", 2024-03-11T21:28:09.002 INFO:teuthology.orchestra.run.smithi092.stdout: "nonce": 2844100201 2024-03-11T21:28:09.002 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-03-11T21:28:09.002 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-03-11T21:28:09.002 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:28:09.002 INFO:teuthology.orchestra.run.smithi092.stdout: "cluster_addrs": { 2024-03-11T21:28:09.002 INFO:teuthology.orchestra.run.smithi092.stdout: "addrvec": [ 2024-03-11T21:28:09.002 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:28:09.002 INFO:teuthology.orchestra.run.smithi092.stdout: "type": "v1", 2024-03-11T21:28:09.002 INFO:teuthology.orchestra.run.smithi092.stdout: "addr": "172.21.15.92:6802", 2024-03-11T21:28:09.002 INFO:teuthology.orchestra.run.smithi092.stdout: "nonce": 2844100201 2024-03-11T21:28:09.002 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-03-11T21:28:09.002 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-03-11T21:28:09.002 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:28:09.003 INFO:teuthology.orchestra.run.smithi092.stdout: "heartbeat_back_addrs": { 2024-03-11T21:28:09.003 INFO:teuthology.orchestra.run.smithi092.stdout: "addrvec": [ 2024-03-11T21:28:09.003 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:28:09.003 INFO:teuthology.orchestra.run.smithi092.stdout: "type": "v1", 2024-03-11T21:28:09.003 INFO:teuthology.orchestra.run.smithi092.stdout: "addr": "172.21.15.92:6804", 2024-03-11T21:28:09.003 INFO:teuthology.orchestra.run.smithi092.stdout: "nonce": 2844100201 2024-03-11T21:28:09.003 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-03-11T21:28:09.003 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-03-11T21:28:09.003 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:28:09.003 INFO:teuthology.orchestra.run.smithi092.stdout: "heartbeat_front_addrs": { 2024-03-11T21:28:09.003 INFO:teuthology.orchestra.run.smithi092.stdout: "addrvec": [ 2024-03-11T21:28:09.003 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:28:09.003 INFO:teuthology.orchestra.run.smithi092.stdout: "type": "v1", 2024-03-11T21:28:09.003 INFO:teuthology.orchestra.run.smithi092.stdout: "addr": "172.21.15.92:6803", 2024-03-11T21:28:09.004 INFO:teuthology.orchestra.run.smithi092.stdout: "nonce": 2844100201 2024-03-11T21:28:09.004 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-03-11T21:28:09.004 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-03-11T21:28:09.004 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:28:09.004 INFO:teuthology.orchestra.run.smithi092.stdout: "public_addr": "172.21.15.92:6801/2844100201", 2024-03-11T21:28:09.004 INFO:teuthology.orchestra.run.smithi092.stdout: "cluster_addr": "172.21.15.92:6802/2844100201", 2024-03-11T21:28:09.004 INFO:teuthology.orchestra.run.smithi092.stdout: "heartbeat_back_addr": "172.21.15.92:6804/2844100201", 2024-03-11T21:28:09.004 INFO:teuthology.orchestra.run.smithi092.stdout: "heartbeat_front_addr": "172.21.15.92:6803/2844100201", 2024-03-11T21:28:09.004 INFO:teuthology.orchestra.run.smithi092.stdout: "state": [ 2024-03-11T21:28:09.004 INFO:teuthology.orchestra.run.smithi092.stdout: "exists", 2024-03-11T21:28:09.004 INFO:teuthology.orchestra.run.smithi092.stdout: "up" 2024-03-11T21:28:09.004 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-03-11T21:28:09.004 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:28:09.004 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:28:09.005 INFO:teuthology.orchestra.run.smithi092.stdout: "osd": 1, 2024-03-11T21:28:09.005 INFO:teuthology.orchestra.run.smithi092.stdout: "uuid": "3a5a0e78-1293-4ace-a93f-69156098571d", 2024-03-11T21:28:09.005 INFO:teuthology.orchestra.run.smithi092.stdout: "up": 1, 2024-03-11T21:28:09.005 INFO:teuthology.orchestra.run.smithi092.stdout: "in": 1, 2024-03-11T21:28:09.005 INFO:teuthology.orchestra.run.smithi092.stdout: "weight": 1, 2024-03-11T21:28:09.005 INFO:teuthology.orchestra.run.smithi092.stdout: "primary_affinity": 1, 2024-03-11T21:28:09.005 INFO:teuthology.orchestra.run.smithi092.stdout: "last_clean_begin": 0, 2024-03-11T21:28:09.005 INFO:teuthology.orchestra.run.smithi092.stdout: "last_clean_end": 0, 2024-03-11T21:28:09.005 INFO:teuthology.orchestra.run.smithi092.stdout: "up_from": 13, 2024-03-11T21:28:09.005 INFO:teuthology.orchestra.run.smithi092.stdout: "up_thru": 99, 2024-03-11T21:28:09.005 INFO:teuthology.orchestra.run.smithi092.stdout: "down_at": 0, 2024-03-11T21:28:09.005 INFO:teuthology.orchestra.run.smithi092.stdout: "lost_at": 0, 2024-03-11T21:28:09.005 INFO:teuthology.orchestra.run.smithi092.stdout: "public_addrs": { 2024-03-11T21:28:09.005 INFO:teuthology.orchestra.run.smithi092.stdout: "addrvec": [ 2024-03-11T21:28:09.005 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:28:09.006 INFO:teuthology.orchestra.run.smithi092.stdout: "type": "v1", 2024-03-11T21:28:09.006 INFO:teuthology.orchestra.run.smithi092.stdout: "addr": "172.21.15.92:6805", 2024-03-11T21:28:09.006 INFO:teuthology.orchestra.run.smithi092.stdout: "nonce": 3577871157 2024-03-11T21:28:09.006 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-03-11T21:28:09.006 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-03-11T21:28:09.006 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:28:09.006 INFO:teuthology.orchestra.run.smithi092.stdout: "cluster_addrs": { 2024-03-11T21:28:09.006 INFO:teuthology.orchestra.run.smithi092.stdout: "addrvec": [ 2024-03-11T21:28:09.006 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:28:09.006 INFO:teuthology.orchestra.run.smithi092.stdout: "type": "v1", 2024-03-11T21:28:09.006 INFO:teuthology.orchestra.run.smithi092.stdout: "addr": "172.21.15.92:6806", 2024-03-11T21:28:09.006 INFO:teuthology.orchestra.run.smithi092.stdout: "nonce": 3577871157 2024-03-11T21:28:09.007 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-03-11T21:28:09.007 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-03-11T21:28:09.007 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:28:09.007 INFO:teuthology.orchestra.run.smithi092.stdout: "heartbeat_back_addrs": { 2024-03-11T21:28:09.007 INFO:teuthology.orchestra.run.smithi092.stdout: "addrvec": [ 2024-03-11T21:28:09.007 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:28:09.007 INFO:teuthology.orchestra.run.smithi092.stdout: "type": "v1", 2024-03-11T21:28:09.007 INFO:teuthology.orchestra.run.smithi092.stdout: "addr": "172.21.15.92:6808", 2024-03-11T21:28:09.007 INFO:teuthology.orchestra.run.smithi092.stdout: "nonce": 3577871157 2024-03-11T21:28:09.007 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-03-11T21:28:09.007 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-03-11T21:28:09.007 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:28:09.007 INFO:teuthology.orchestra.run.smithi092.stdout: "heartbeat_front_addrs": { 2024-03-11T21:28:09.007 INFO:teuthology.orchestra.run.smithi092.stdout: "addrvec": [ 2024-03-11T21:28:09.008 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:28:09.008 INFO:teuthology.orchestra.run.smithi092.stdout: "type": "v1", 2024-03-11T21:28:09.008 INFO:teuthology.orchestra.run.smithi092.stdout: "addr": "172.21.15.92:6807", 2024-03-11T21:28:09.008 INFO:teuthology.orchestra.run.smithi092.stdout: "nonce": 3577871157 2024-03-11T21:28:09.008 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-03-11T21:28:09.008 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-03-11T21:28:09.008 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:28:09.008 INFO:teuthology.orchestra.run.smithi092.stdout: "public_addr": "172.21.15.92:6805/3577871157", 2024-03-11T21:28:09.008 INFO:teuthology.orchestra.run.smithi092.stdout: "cluster_addr": "172.21.15.92:6806/3577871157", 2024-03-11T21:28:09.008 INFO:teuthology.orchestra.run.smithi092.stdout: "heartbeat_back_addr": "172.21.15.92:6808/3577871157", 2024-03-11T21:28:09.008 INFO:teuthology.orchestra.run.smithi092.stdout: "heartbeat_front_addr": "172.21.15.92:6807/3577871157", 2024-03-11T21:28:09.008 INFO:teuthology.orchestra.run.smithi092.stdout: "state": [ 2024-03-11T21:28:09.008 INFO:teuthology.orchestra.run.smithi092.stdout: "exists", 2024-03-11T21:28:09.008 INFO:teuthology.orchestra.run.smithi092.stdout: "up" 2024-03-11T21:28:09.008 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-03-11T21:28:09.009 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:28:09.009 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:28:09.009 INFO:teuthology.orchestra.run.smithi092.stdout: "osd": 2, 2024-03-11T21:28:09.009 INFO:teuthology.orchestra.run.smithi092.stdout: "uuid": "f207b743-0df4-40ed-a735-950e272fea60", 2024-03-11T21:28:09.009 INFO:teuthology.orchestra.run.smithi092.stdout: "up": 1, 2024-03-11T21:28:09.009 INFO:teuthology.orchestra.run.smithi092.stdout: "in": 1, 2024-03-11T21:28:09.009 INFO:teuthology.orchestra.run.smithi092.stdout: "weight": 1, 2024-03-11T21:28:09.009 INFO:teuthology.orchestra.run.smithi092.stdout: "primary_affinity": 1, 2024-03-11T21:28:09.009 INFO:teuthology.orchestra.run.smithi092.stdout: "last_clean_begin": 0, 2024-03-11T21:28:09.009 INFO:teuthology.orchestra.run.smithi092.stdout: "last_clean_end": 0, 2024-03-11T21:28:09.009 INFO:teuthology.orchestra.run.smithi092.stdout: "up_from": 18, 2024-03-11T21:28:09.009 INFO:teuthology.orchestra.run.smithi092.stdout: "up_thru": 99, 2024-03-11T21:28:09.009 INFO:teuthology.orchestra.run.smithi092.stdout: "down_at": 0, 2024-03-11T21:28:09.009 INFO:teuthology.orchestra.run.smithi092.stdout: "lost_at": 0, 2024-03-11T21:28:09.009 INFO:teuthology.orchestra.run.smithi092.stdout: "public_addrs": { 2024-03-11T21:28:09.009 INFO:teuthology.orchestra.run.smithi092.stdout: "addrvec": [ 2024-03-11T21:28:09.010 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:28:09.010 INFO:teuthology.orchestra.run.smithi092.stdout: "type": "v1", 2024-03-11T21:28:09.010 INFO:teuthology.orchestra.run.smithi092.stdout: "addr": "172.21.15.92:6809", 2024-03-11T21:28:09.010 INFO:teuthology.orchestra.run.smithi092.stdout: "nonce": 760825152 2024-03-11T21:28:09.010 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-03-11T21:28:09.010 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-03-11T21:28:09.010 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:28:09.010 INFO:teuthology.orchestra.run.smithi092.stdout: "cluster_addrs": { 2024-03-11T21:28:09.010 INFO:teuthology.orchestra.run.smithi092.stdout: "addrvec": [ 2024-03-11T21:28:09.010 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:28:09.010 INFO:teuthology.orchestra.run.smithi092.stdout: "type": "v1", 2024-03-11T21:28:09.010 INFO:teuthology.orchestra.run.smithi092.stdout: "addr": "172.21.15.92:6810", 2024-03-11T21:28:09.010 INFO:teuthology.orchestra.run.smithi092.stdout: "nonce": 760825152 2024-03-11T21:28:09.010 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-03-11T21:28:09.010 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-03-11T21:28:09.011 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:28:09.011 INFO:teuthology.orchestra.run.smithi092.stdout: "heartbeat_back_addrs": { 2024-03-11T21:28:09.011 INFO:teuthology.orchestra.run.smithi092.stdout: "addrvec": [ 2024-03-11T21:28:09.011 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:28:09.011 INFO:teuthology.orchestra.run.smithi092.stdout: "type": "v1", 2024-03-11T21:28:09.011 INFO:teuthology.orchestra.run.smithi092.stdout: "addr": "172.21.15.92:6812", 2024-03-11T21:28:09.011 INFO:teuthology.orchestra.run.smithi092.stdout: "nonce": 760825152 2024-03-11T21:28:09.011 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-03-11T21:28:09.011 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-03-11T21:28:09.011 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:28:09.011 INFO:teuthology.orchestra.run.smithi092.stdout: "heartbeat_front_addrs": { 2024-03-11T21:28:09.011 INFO:teuthology.orchestra.run.smithi092.stdout: "addrvec": [ 2024-03-11T21:28:09.011 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:28:09.011 INFO:teuthology.orchestra.run.smithi092.stdout: "type": "v1", 2024-03-11T21:28:09.012 INFO:teuthology.orchestra.run.smithi092.stdout: "addr": "172.21.15.92:6811", 2024-03-11T21:28:09.012 INFO:teuthology.orchestra.run.smithi092.stdout: "nonce": 760825152 2024-03-11T21:28:09.012 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-03-11T21:28:09.012 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-03-11T21:28:09.012 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:28:09.012 INFO:teuthology.orchestra.run.smithi092.stdout: "public_addr": "172.21.15.92:6809/760825152", 2024-03-11T21:28:09.012 INFO:teuthology.orchestra.run.smithi092.stdout: "cluster_addr": "172.21.15.92:6810/760825152", 2024-03-11T21:28:09.012 INFO:teuthology.orchestra.run.smithi092.stdout: "heartbeat_back_addr": "172.21.15.92:6812/760825152", 2024-03-11T21:28:09.012 INFO:teuthology.orchestra.run.smithi092.stdout: "heartbeat_front_addr": "172.21.15.92:6811/760825152", 2024-03-11T21:28:09.012 INFO:teuthology.orchestra.run.smithi092.stdout: "state": [ 2024-03-11T21:28:09.012 INFO:teuthology.orchestra.run.smithi092.stdout: "exists", 2024-03-11T21:28:09.012 INFO:teuthology.orchestra.run.smithi092.stdout: "up" 2024-03-11T21:28:09.012 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-03-11T21:28:09.012 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:28:09.012 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:28:09.012 INFO:teuthology.orchestra.run.smithi092.stdout: "osd": 3, 2024-03-11T21:28:09.013 INFO:teuthology.orchestra.run.smithi092.stdout: "uuid": "bbb6f69a-8f0f-47e6-937a-062e1902ae13", 2024-03-11T21:28:09.013 INFO:teuthology.orchestra.run.smithi092.stdout: "up": 1, 2024-03-11T21:28:09.013 INFO:teuthology.orchestra.run.smithi092.stdout: "in": 1, 2024-03-11T21:28:09.013 INFO:teuthology.orchestra.run.smithi092.stdout: "weight": 1, 2024-03-11T21:28:09.013 INFO:teuthology.orchestra.run.smithi092.stdout: "primary_affinity": 1, 2024-03-11T21:28:09.013 INFO:teuthology.orchestra.run.smithi092.stdout: "last_clean_begin": 0, 2024-03-11T21:28:09.013 INFO:teuthology.orchestra.run.smithi092.stdout: "last_clean_end": 0, 2024-03-11T21:28:09.013 INFO:teuthology.orchestra.run.smithi092.stdout: "up_from": 26, 2024-03-11T21:28:09.013 INFO:teuthology.orchestra.run.smithi092.stdout: "up_thru": 99, 2024-03-11T21:28:09.013 INFO:teuthology.orchestra.run.smithi092.stdout: "down_at": 0, 2024-03-11T21:28:09.013 INFO:teuthology.orchestra.run.smithi092.stdout: "lost_at": 0, 2024-03-11T21:28:09.013 INFO:teuthology.orchestra.run.smithi092.stdout: "public_addrs": { 2024-03-11T21:28:09.013 INFO:teuthology.orchestra.run.smithi092.stdout: "addrvec": [ 2024-03-11T21:28:09.013 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:28:09.013 INFO:teuthology.orchestra.run.smithi092.stdout: "type": "v1", 2024-03-11T21:28:09.013 INFO:teuthology.orchestra.run.smithi092.stdout: "addr": "172.21.15.92:6813", 2024-03-11T21:28:09.014 INFO:teuthology.orchestra.run.smithi092.stdout: "nonce": 670596264 2024-03-11T21:28:09.014 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-03-11T21:28:09.014 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-03-11T21:28:09.014 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:28:09.014 INFO:teuthology.orchestra.run.smithi092.stdout: "cluster_addrs": { 2024-03-11T21:28:09.014 INFO:teuthology.orchestra.run.smithi092.stdout: "addrvec": [ 2024-03-11T21:28:09.014 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:28:09.014 INFO:teuthology.orchestra.run.smithi092.stdout: "type": "v1", 2024-03-11T21:28:09.014 INFO:teuthology.orchestra.run.smithi092.stdout: "addr": "172.21.15.92:6814", 2024-03-11T21:28:09.014 INFO:teuthology.orchestra.run.smithi092.stdout: "nonce": 670596264 2024-03-11T21:28:09.015 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-03-11T21:28:09.015 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-03-11T21:28:09.015 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:28:09.015 INFO:teuthology.orchestra.run.smithi092.stdout: "heartbeat_back_addrs": { 2024-03-11T21:28:09.015 INFO:teuthology.orchestra.run.smithi092.stdout: "addrvec": [ 2024-03-11T21:28:09.015 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:28:09.015 INFO:teuthology.orchestra.run.smithi092.stdout: "type": "v1", 2024-03-11T21:28:09.015 INFO:teuthology.orchestra.run.smithi092.stdout: "addr": "172.21.15.92:6816", 2024-03-11T21:28:09.015 INFO:teuthology.orchestra.run.smithi092.stdout: "nonce": 670596264 2024-03-11T21:28:09.015 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-03-11T21:28:09.015 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-03-11T21:28:09.015 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:28:09.015 INFO:teuthology.orchestra.run.smithi092.stdout: "heartbeat_front_addrs": { 2024-03-11T21:28:09.015 INFO:teuthology.orchestra.run.smithi092.stdout: "addrvec": [ 2024-03-11T21:28:09.015 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:28:09.015 INFO:teuthology.orchestra.run.smithi092.stdout: "type": "v1", 2024-03-11T21:28:09.016 INFO:teuthology.orchestra.run.smithi092.stdout: "addr": "172.21.15.92:6815", 2024-03-11T21:28:09.016 INFO:teuthology.orchestra.run.smithi092.stdout: "nonce": 670596264 2024-03-11T21:28:09.016 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-03-11T21:28:09.016 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-03-11T21:28:09.016 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:28:09.016 INFO:teuthology.orchestra.run.smithi092.stdout: "public_addr": "172.21.15.92:6813/670596264", 2024-03-11T21:28:09.016 INFO:teuthology.orchestra.run.smithi092.stdout: "cluster_addr": "172.21.15.92:6814/670596264", 2024-03-11T21:28:09.016 INFO:teuthology.orchestra.run.smithi092.stdout: "heartbeat_back_addr": "172.21.15.92:6816/670596264", 2024-03-11T21:28:09.016 INFO:teuthology.orchestra.run.smithi092.stdout: "heartbeat_front_addr": "172.21.15.92:6815/670596264", 2024-03-11T21:28:09.016 INFO:teuthology.orchestra.run.smithi092.stdout: "state": [ 2024-03-11T21:28:09.016 INFO:teuthology.orchestra.run.smithi092.stdout: "exists", 2024-03-11T21:28:09.016 INFO:teuthology.orchestra.run.smithi092.stdout: "up" 2024-03-11T21:28:09.016 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-03-11T21:28:09.016 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:28:09.016 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:28:09.016 INFO:teuthology.orchestra.run.smithi092.stdout: "osd": 4, 2024-03-11T21:28:09.017 INFO:teuthology.orchestra.run.smithi092.stdout: "uuid": "00266cbc-d2cf-4dee-8a96-3753ebc95ca8", 2024-03-11T21:28:09.017 INFO:teuthology.orchestra.run.smithi092.stdout: "up": 1, 2024-03-11T21:28:09.017 INFO:teuthology.orchestra.run.smithi092.stdout: "in": 1, 2024-03-11T21:28:09.017 INFO:teuthology.orchestra.run.smithi092.stdout: "weight": 1, 2024-03-11T21:28:09.017 INFO:teuthology.orchestra.run.smithi092.stdout: "primary_affinity": 1, 2024-03-11T21:28:09.017 INFO:teuthology.orchestra.run.smithi092.stdout: "last_clean_begin": 0, 2024-03-11T21:28:09.017 INFO:teuthology.orchestra.run.smithi092.stdout: "last_clean_end": 0, 2024-03-11T21:28:09.017 INFO:teuthology.orchestra.run.smithi092.stdout: "up_from": 32, 2024-03-11T21:28:09.017 INFO:teuthology.orchestra.run.smithi092.stdout: "up_thru": 99, 2024-03-11T21:28:09.017 INFO:teuthology.orchestra.run.smithi092.stdout: "down_at": 0, 2024-03-11T21:28:09.017 INFO:teuthology.orchestra.run.smithi092.stdout: "lost_at": 0, 2024-03-11T21:28:09.017 INFO:teuthology.orchestra.run.smithi092.stdout: "public_addrs": { 2024-03-11T21:28:09.017 INFO:teuthology.orchestra.run.smithi092.stdout: "addrvec": [ 2024-03-11T21:28:09.017 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:28:09.017 INFO:teuthology.orchestra.run.smithi092.stdout: "type": "v1", 2024-03-11T21:28:09.017 INFO:teuthology.orchestra.run.smithi092.stdout: "addr": "172.21.15.97:6800", 2024-03-11T21:28:09.017 INFO:teuthology.orchestra.run.smithi092.stdout: "nonce": 420733620 2024-03-11T21:28:09.018 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-03-11T21:28:09.018 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-03-11T21:28:09.018 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:28:09.018 INFO:teuthology.orchestra.run.smithi092.stdout: "cluster_addrs": { 2024-03-11T21:28:09.018 INFO:teuthology.orchestra.run.smithi092.stdout: "addrvec": [ 2024-03-11T21:28:09.018 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:28:09.018 INFO:teuthology.orchestra.run.smithi092.stdout: "type": "v1", 2024-03-11T21:28:09.018 INFO:teuthology.orchestra.run.smithi092.stdout: "addr": "172.21.15.97:6801", 2024-03-11T21:28:09.018 INFO:teuthology.orchestra.run.smithi092.stdout: "nonce": 420733620 2024-03-11T21:28:09.018 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-03-11T21:28:09.018 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-03-11T21:28:09.018 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:28:09.018 INFO:teuthology.orchestra.run.smithi092.stdout: "heartbeat_back_addrs": { 2024-03-11T21:28:09.018 INFO:teuthology.orchestra.run.smithi092.stdout: "addrvec": [ 2024-03-11T21:28:09.018 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:28:09.019 INFO:teuthology.orchestra.run.smithi092.stdout: "type": "v1", 2024-03-11T21:28:09.019 INFO:teuthology.orchestra.run.smithi092.stdout: "addr": "172.21.15.97:6803", 2024-03-11T21:28:09.019 INFO:teuthology.orchestra.run.smithi092.stdout: "nonce": 420733620 2024-03-11T21:28:09.019 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-03-11T21:28:09.019 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-03-11T21:28:09.019 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:28:09.019 INFO:teuthology.orchestra.run.smithi092.stdout: "heartbeat_front_addrs": { 2024-03-11T21:28:09.019 INFO:teuthology.orchestra.run.smithi092.stdout: "addrvec": [ 2024-03-11T21:28:09.019 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:28:09.019 INFO:teuthology.orchestra.run.smithi092.stdout: "type": "v1", 2024-03-11T21:28:09.019 INFO:teuthology.orchestra.run.smithi092.stdout: "addr": "172.21.15.97:6802", 2024-03-11T21:28:09.019 INFO:teuthology.orchestra.run.smithi092.stdout: "nonce": 420733620 2024-03-11T21:28:09.019 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-03-11T21:28:09.019 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-03-11T21:28:09.019 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:28:09.020 INFO:teuthology.orchestra.run.smithi092.stdout: "public_addr": "172.21.15.97:6800/420733620", 2024-03-11T21:28:09.020 INFO:teuthology.orchestra.run.smithi092.stdout: "cluster_addr": "172.21.15.97:6801/420733620", 2024-03-11T21:28:09.020 INFO:teuthology.orchestra.run.smithi092.stdout: "heartbeat_back_addr": "172.21.15.97:6803/420733620", 2024-03-11T21:28:09.020 INFO:teuthology.orchestra.run.smithi092.stdout: "heartbeat_front_addr": "172.21.15.97:6802/420733620", 2024-03-11T21:28:09.020 INFO:teuthology.orchestra.run.smithi092.stdout: "state": [ 2024-03-11T21:28:09.020 INFO:teuthology.orchestra.run.smithi092.stdout: "exists", 2024-03-11T21:28:09.020 INFO:teuthology.orchestra.run.smithi092.stdout: "up" 2024-03-11T21:28:09.020 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-03-11T21:28:09.020 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:28:09.020 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:28:09.020 INFO:teuthology.orchestra.run.smithi092.stdout: "osd": 5, 2024-03-11T21:28:09.020 INFO:teuthology.orchestra.run.smithi092.stdout: "uuid": "593a6f6b-8874-40ae-8848-d51c5723a5e4", 2024-03-11T21:28:09.020 INFO:teuthology.orchestra.run.smithi092.stdout: "up": 1, 2024-03-11T21:28:09.020 INFO:teuthology.orchestra.run.smithi092.stdout: "in": 1, 2024-03-11T21:28:09.021 INFO:teuthology.orchestra.run.smithi092.stdout: "weight": 1, 2024-03-11T21:28:09.021 INFO:teuthology.orchestra.run.smithi092.stdout: "primary_affinity": 1, 2024-03-11T21:28:09.021 INFO:teuthology.orchestra.run.smithi092.stdout: "last_clean_begin": 0, 2024-03-11T21:28:09.021 INFO:teuthology.orchestra.run.smithi092.stdout: "last_clean_end": 0, 2024-03-11T21:28:09.021 INFO:teuthology.orchestra.run.smithi092.stdout: "up_from": 41, 2024-03-11T21:28:09.021 INFO:teuthology.orchestra.run.smithi092.stdout: "up_thru": 99, 2024-03-11T21:28:09.021 INFO:teuthology.orchestra.run.smithi092.stdout: "down_at": 0, 2024-03-11T21:28:09.021 INFO:teuthology.orchestra.run.smithi092.stdout: "lost_at": 0, 2024-03-11T21:28:09.021 INFO:teuthology.orchestra.run.smithi092.stdout: "public_addrs": { 2024-03-11T21:28:09.021 INFO:teuthology.orchestra.run.smithi092.stdout: "addrvec": [ 2024-03-11T21:28:09.021 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:28:09.021 INFO:teuthology.orchestra.run.smithi092.stdout: "type": "v1", 2024-03-11T21:28:09.021 INFO:teuthology.orchestra.run.smithi092.stdout: "addr": "172.21.15.97:6804", 2024-03-11T21:28:09.021 INFO:teuthology.orchestra.run.smithi092.stdout: "nonce": 312056041 2024-03-11T21:28:09.021 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-03-11T21:28:09.022 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-03-11T21:28:09.022 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:28:09.022 INFO:teuthology.orchestra.run.smithi092.stdout: "cluster_addrs": { 2024-03-11T21:28:09.022 INFO:teuthology.orchestra.run.smithi092.stdout: "addrvec": [ 2024-03-11T21:28:09.022 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:28:09.022 INFO:teuthology.orchestra.run.smithi092.stdout: "type": "v1", 2024-03-11T21:28:09.022 INFO:teuthology.orchestra.run.smithi092.stdout: "addr": "172.21.15.97:6805", 2024-03-11T21:28:09.022 INFO:teuthology.orchestra.run.smithi092.stdout: "nonce": 312056041 2024-03-11T21:28:09.022 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-03-11T21:28:09.022 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-03-11T21:28:09.022 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:28:09.022 INFO:teuthology.orchestra.run.smithi092.stdout: "heartbeat_back_addrs": { 2024-03-11T21:28:09.022 INFO:teuthology.orchestra.run.smithi092.stdout: "addrvec": [ 2024-03-11T21:28:09.022 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:28:09.022 INFO:teuthology.orchestra.run.smithi092.stdout: "type": "v1", 2024-03-11T21:28:09.023 INFO:teuthology.orchestra.run.smithi092.stdout: "addr": "172.21.15.97:6807", 2024-03-11T21:28:09.023 INFO:teuthology.orchestra.run.smithi092.stdout: "nonce": 312056041 2024-03-11T21:28:09.023 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-03-11T21:28:09.023 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-03-11T21:28:09.023 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:28:09.023 INFO:teuthology.orchestra.run.smithi092.stdout: "heartbeat_front_addrs": { 2024-03-11T21:28:09.023 INFO:teuthology.orchestra.run.smithi092.stdout: "addrvec": [ 2024-03-11T21:28:09.023 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:28:09.023 INFO:teuthology.orchestra.run.smithi092.stdout: "type": "v1", 2024-03-11T21:28:09.023 INFO:teuthology.orchestra.run.smithi092.stdout: "addr": "172.21.15.97:6806", 2024-03-11T21:28:09.023 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:28:09.024 INFO:teuthology.orchestra.run.smithi092.stdout: "nonce": 312056041 2024-03-11T21:28:09.024 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-03-11T21:28:09.024 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-03-11T21:28:09.024 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:28:09.024 INFO:teuthology.orchestra.run.smithi092.stdout: "public_addr": "172.21.15.97:6804/312056041", 2024-03-11T21:28:09.025 INFO:teuthology.orchestra.run.smithi092.stdout: "cluster_addr": "172.21.15.97:6805/312056041", 2024-03-11T21:28:09.025 INFO:teuthology.orchestra.run.smithi092.stdout: "heartbeat_back_addr": "172.21.15.97:6807/312056041", 2024-03-11T21:28:09.025 INFO:teuthology.orchestra.run.smithi092.stdout: "heartbeat_front_addr": "172.21.15.97:6806/312056041", 2024-03-11T21:28:09.025 INFO:teuthology.orchestra.run.smithi092.stdout: "state": [ 2024-03-11T21:28:09.025 INFO:teuthology.orchestra.run.smithi092.stdout: "exists", 2024-03-11T21:28:09.025 INFO:teuthology.orchestra.run.smithi092.stdout: "up" 2024-03-11T21:28:09.025 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-03-11T21:28:09.025 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:28:09.025 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:28:09.025 INFO:teuthology.orchestra.run.smithi092.stdout: "osd": 6, 2024-03-11T21:28:09.025 INFO:teuthology.orchestra.run.smithi092.stdout: "uuid": "91035ecc-6f1a-4cec-bda6-090af9d53eff", 2024-03-11T21:28:09.025 INFO:teuthology.orchestra.run.smithi092.stdout: "up": 1, 2024-03-11T21:28:09.025 INFO:teuthology.orchestra.run.smithi092.stdout: "in": 1, 2024-03-11T21:28:09.025 INFO:teuthology.orchestra.run.smithi092.stdout: "weight": 1, 2024-03-11T21:28:09.025 INFO:teuthology.orchestra.run.smithi092.stdout: "primary_affinity": 1, 2024-03-11T21:28:09.026 INFO:teuthology.orchestra.run.smithi092.stdout: "last_clean_begin": 0, 2024-03-11T21:28:09.026 INFO:teuthology.orchestra.run.smithi092.stdout: "last_clean_end": 0, 2024-03-11T21:28:09.026 INFO:teuthology.orchestra.run.smithi092.stdout: "up_from": 50, 2024-03-11T21:28:09.026 INFO:teuthology.orchestra.run.smithi092.stdout: "up_thru": 99, 2024-03-11T21:28:09.026 INFO:teuthology.orchestra.run.smithi092.stdout: "down_at": 0, 2024-03-11T21:28:09.026 INFO:teuthology.orchestra.run.smithi092.stdout: "lost_at": 0, 2024-03-11T21:28:09.026 INFO:teuthology.orchestra.run.smithi092.stdout: "public_addrs": { 2024-03-11T21:28:09.026 INFO:teuthology.orchestra.run.smithi092.stdout: "addrvec": [ 2024-03-11T21:28:09.026 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:28:09.026 INFO:teuthology.orchestra.run.smithi092.stdout: "type": "v1", 2024-03-11T21:28:09.026 INFO:teuthology.orchestra.run.smithi092.stdout: "addr": "172.21.15.97:6808", 2024-03-11T21:28:09.026 INFO:teuthology.orchestra.run.smithi092.stdout: "nonce": 178994999 2024-03-11T21:28:09.026 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-03-11T21:28:09.026 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-03-11T21:28:09.027 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:28:09.027 INFO:teuthology.orchestra.run.smithi092.stdout: "cluster_addrs": { 2024-03-11T21:28:09.027 INFO:teuthology.orchestra.run.smithi092.stdout: "addrvec": [ 2024-03-11T21:28:09.027 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:28:09.027 INFO:teuthology.orchestra.run.smithi092.stdout: "type": "v1", 2024-03-11T21:28:09.027 INFO:teuthology.orchestra.run.smithi092.stdout: "addr": "172.21.15.97:6809", 2024-03-11T21:28:09.027 INFO:teuthology.orchestra.run.smithi092.stdout: "nonce": 178994999 2024-03-11T21:28:09.027 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-03-11T21:28:09.027 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-03-11T21:28:09.027 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:28:09.027 INFO:teuthology.orchestra.run.smithi092.stdout: "heartbeat_back_addrs": { 2024-03-11T21:28:09.027 INFO:teuthology.orchestra.run.smithi092.stdout: "addrvec": [ 2024-03-11T21:28:09.027 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:28:09.027 INFO:teuthology.orchestra.run.smithi092.stdout: "type": "v1", 2024-03-11T21:28:09.027 INFO:teuthology.orchestra.run.smithi092.stdout: "addr": "172.21.15.97:6811", 2024-03-11T21:28:09.027 INFO:teuthology.orchestra.run.smithi092.stdout: "nonce": 178994999 2024-03-11T21:28:09.027 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-03-11T21:28:09.028 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-03-11T21:28:09.028 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:28:09.028 INFO:teuthology.orchestra.run.smithi092.stdout: "heartbeat_front_addrs": { 2024-03-11T21:28:09.028 INFO:teuthology.orchestra.run.smithi092.stdout: "addrvec": [ 2024-03-11T21:28:09.028 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:28:09.028 INFO:teuthology.orchestra.run.smithi092.stdout: "type": "v1", 2024-03-11T21:28:09.028 INFO:teuthology.orchestra.run.smithi092.stdout: "addr": "172.21.15.97:6810", 2024-03-11T21:28:09.028 INFO:teuthology.orchestra.run.smithi092.stdout: "nonce": 178994999 2024-03-11T21:28:09.028 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-03-11T21:28:09.028 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-03-11T21:28:09.028 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:28:09.028 INFO:teuthology.orchestra.run.smithi092.stdout: "public_addr": "172.21.15.97:6808/178994999", 2024-03-11T21:28:09.028 INFO:teuthology.orchestra.run.smithi092.stdout: "cluster_addr": "172.21.15.97:6809/178994999", 2024-03-11T21:28:09.028 INFO:teuthology.orchestra.run.smithi092.stdout: "heartbeat_back_addr": "172.21.15.97:6811/178994999", 2024-03-11T21:28:09.028 INFO:teuthology.orchestra.run.smithi092.stdout: "heartbeat_front_addr": "172.21.15.97:6810/178994999", 2024-03-11T21:28:09.028 INFO:teuthology.orchestra.run.smithi092.stdout: "state": [ 2024-03-11T21:28:09.029 INFO:teuthology.orchestra.run.smithi092.stdout: "exists", 2024-03-11T21:28:09.029 INFO:teuthology.orchestra.run.smithi092.stdout: "up" 2024-03-11T21:28:09.029 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-03-11T21:28:09.029 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:28:09.029 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:28:09.029 INFO:teuthology.orchestra.run.smithi092.stdout: "osd": 7, 2024-03-11T21:28:09.029 INFO:teuthology.orchestra.run.smithi092.stdout: "uuid": "3eb28c24-664a-4bd6-869d-6bf5db98f52f", 2024-03-11T21:28:09.029 INFO:teuthology.orchestra.run.smithi092.stdout: "up": 1, 2024-03-11T21:28:09.029 INFO:teuthology.orchestra.run.smithi092.stdout: "in": 1, 2024-03-11T21:28:09.029 INFO:teuthology.orchestra.run.smithi092.stdout: "weight": 1, 2024-03-11T21:28:09.029 INFO:teuthology.orchestra.run.smithi092.stdout: "primary_affinity": 1, 2024-03-11T21:28:09.029 INFO:teuthology.orchestra.run.smithi092.stdout: "last_clean_begin": 0, 2024-03-11T21:28:09.029 INFO:teuthology.orchestra.run.smithi092.stdout: "last_clean_end": 0, 2024-03-11T21:28:09.029 INFO:teuthology.orchestra.run.smithi092.stdout: "up_from": 60, 2024-03-11T21:28:09.029 INFO:teuthology.orchestra.run.smithi092.stdout: "up_thru": 99, 2024-03-11T21:28:09.029 INFO:teuthology.orchestra.run.smithi092.stdout: "down_at": 0, 2024-03-11T21:28:09.030 INFO:teuthology.orchestra.run.smithi092.stdout: "lost_at": 0, 2024-03-11T21:28:09.030 INFO:teuthology.orchestra.run.smithi092.stdout: "public_addrs": { 2024-03-11T21:28:09.030 INFO:teuthology.orchestra.run.smithi092.stdout: "addrvec": [ 2024-03-11T21:28:09.030 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:28:09.030 INFO:teuthology.orchestra.run.smithi092.stdout: "type": "v1", 2024-03-11T21:28:09.030 INFO:teuthology.orchestra.run.smithi092.stdout: "addr": "172.21.15.97:6812", 2024-03-11T21:28:09.030 INFO:teuthology.orchestra.run.smithi092.stdout: "nonce": 3241560509 2024-03-11T21:28:09.030 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-03-11T21:28:09.030 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-03-11T21:28:09.030 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:28:09.030 INFO:teuthology.orchestra.run.smithi092.stdout: "cluster_addrs": { 2024-03-11T21:28:09.030 INFO:teuthology.orchestra.run.smithi092.stdout: "addrvec": [ 2024-03-11T21:28:09.030 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:28:09.031 INFO:teuthology.orchestra.run.smithi092.stdout: "type": "v1", 2024-03-11T21:28:09.031 INFO:teuthology.orchestra.run.smithi092.stdout: "addr": "172.21.15.97:6813", 2024-03-11T21:28:09.031 INFO:teuthology.orchestra.run.smithi092.stdout: "nonce": 3241560509 2024-03-11T21:28:09.031 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-03-11T21:28:09.031 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-03-11T21:28:09.031 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:28:09.031 INFO:teuthology.orchestra.run.smithi092.stdout: "heartbeat_back_addrs": { 2024-03-11T21:28:09.031 INFO:teuthology.orchestra.run.smithi092.stdout: "addrvec": [ 2024-03-11T21:28:09.031 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:28:09.031 INFO:teuthology.orchestra.run.smithi092.stdout: "type": "v1", 2024-03-11T21:28:09.031 INFO:teuthology.orchestra.run.smithi092.stdout: "addr": "172.21.15.97:6815", 2024-03-11T21:28:09.031 INFO:teuthology.orchestra.run.smithi092.stdout: "nonce": 3241560509 2024-03-11T21:28:09.031 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-03-11T21:28:09.031 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-03-11T21:28:09.031 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:28:09.032 INFO:teuthology.orchestra.run.smithi092.stdout: "heartbeat_front_addrs": { 2024-03-11T21:28:09.032 INFO:teuthology.orchestra.run.smithi092.stdout: "addrvec": [ 2024-03-11T21:28:09.032 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:28:09.032 INFO:teuthology.orchestra.run.smithi092.stdout: "type": "v1", 2024-03-11T21:28:09.032 INFO:teuthology.orchestra.run.smithi092.stdout: "addr": "172.21.15.97:6814", 2024-03-11T21:28:09.032 INFO:teuthology.orchestra.run.smithi092.stdout: "nonce": 3241560509 2024-03-11T21:28:09.032 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-03-11T21:28:09.032 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-03-11T21:28:09.032 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:28:09.032 INFO:teuthology.orchestra.run.smithi092.stdout: "public_addr": "172.21.15.97:6812/3241560509", 2024-03-11T21:28:09.032 INFO:teuthology.orchestra.run.smithi092.stdout: "cluster_addr": "172.21.15.97:6813/3241560509", 2024-03-11T21:28:09.032 INFO:teuthology.orchestra.run.smithi092.stdout: "heartbeat_back_addr": "172.21.15.97:6815/3241560509", 2024-03-11T21:28:09.032 INFO:teuthology.orchestra.run.smithi092.stdout: "heartbeat_front_addr": "172.21.15.97:6814/3241560509", 2024-03-11T21:28:09.032 INFO:teuthology.orchestra.run.smithi092.stdout: "state": [ 2024-03-11T21:28:09.032 INFO:teuthology.orchestra.run.smithi092.stdout: "exists", 2024-03-11T21:28:09.033 INFO:teuthology.orchestra.run.smithi092.stdout: "up" 2024-03-11T21:28:09.033 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-03-11T21:28:09.033 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-03-11T21:28:09.033 INFO:teuthology.orchestra.run.smithi092.stdout: ], 2024-03-11T21:28:09.033 INFO:teuthology.orchestra.run.smithi092.stdout: "osd_xinfo": [ 2024-03-11T21:28:09.033 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:28:09.033 INFO:teuthology.orchestra.run.smithi092.stdout: "osd": 0, 2024-03-11T21:28:09.033 INFO:teuthology.orchestra.run.smithi092.stdout: "down_stamp": "0.000000", 2024-03-11T21:28:09.033 INFO:teuthology.orchestra.run.smithi092.stdout: "laggy_probability": 0, 2024-03-11T21:28:09.033 INFO:teuthology.orchestra.run.smithi092.stdout: "laggy_interval": 0, 2024-03-11T21:28:09.033 INFO:teuthology.orchestra.run.smithi092.stdout: "features": 4540701547738038271, 2024-03-11T21:28:09.033 INFO:teuthology.orchestra.run.smithi092.stdout: "old_weight": 0, 2024-03-11T21:28:09.033 INFO:teuthology.orchestra.run.smithi092.stdout: "last_purged_snaps_scrub": "2024-03-11T21:27:40.685501+0000", 2024-03-11T21:28:09.033 INFO:teuthology.orchestra.run.smithi092.stdout: "dead_epoch": 0 2024-03-11T21:28:09.033 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:28:09.034 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:28:09.034 INFO:teuthology.orchestra.run.smithi092.stdout: "osd": 1, 2024-03-11T21:28:09.034 INFO:teuthology.orchestra.run.smithi092.stdout: "down_stamp": "0.000000", 2024-03-11T21:28:09.034 INFO:teuthology.orchestra.run.smithi092.stdout: "laggy_probability": 0, 2024-03-11T21:28:09.034 INFO:teuthology.orchestra.run.smithi092.stdout: "laggy_interval": 0, 2024-03-11T21:28:09.034 INFO:teuthology.orchestra.run.smithi092.stdout: "features": 4540701547738038271, 2024-03-11T21:28:09.034 INFO:teuthology.orchestra.run.smithi092.stdout: "old_weight": 0, 2024-03-11T21:28:09.034 INFO:teuthology.orchestra.run.smithi092.stdout: "last_purged_snaps_scrub": "2024-03-11T21:27:38.877730+0000", 2024-03-11T21:28:09.034 INFO:teuthology.orchestra.run.smithi092.stdout: "dead_epoch": 0 2024-03-11T21:28:09.034 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:28:09.034 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:28:09.034 INFO:teuthology.orchestra.run.smithi092.stdout: "osd": 2, 2024-03-11T21:28:09.034 INFO:teuthology.orchestra.run.smithi092.stdout: "down_stamp": "0.000000", 2024-03-11T21:28:09.034 INFO:teuthology.orchestra.run.smithi092.stdout: "laggy_probability": 0, 2024-03-11T21:28:09.035 INFO:teuthology.orchestra.run.smithi092.stdout: "laggy_interval": 0, 2024-03-11T21:28:09.035 INFO:teuthology.orchestra.run.smithi092.stdout: "features": 4540701547738038271, 2024-03-11T21:28:09.035 INFO:teuthology.orchestra.run.smithi092.stdout: "old_weight": 0, 2024-03-11T21:28:09.035 INFO:teuthology.orchestra.run.smithi092.stdout: "last_purged_snaps_scrub": "2024-03-11T21:27:05.095427+0000", 2024-03-11T21:28:09.035 INFO:teuthology.orchestra.run.smithi092.stdout: "dead_epoch": 0 2024-03-11T21:28:09.035 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:28:09.035 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:28:09.035 INFO:teuthology.orchestra.run.smithi092.stdout: "osd": 3, 2024-03-11T21:28:09.035 INFO:teuthology.orchestra.run.smithi092.stdout: "down_stamp": "0.000000", 2024-03-11T21:28:09.035 INFO:teuthology.orchestra.run.smithi092.stdout: "laggy_probability": 0, 2024-03-11T21:28:09.035 INFO:teuthology.orchestra.run.smithi092.stdout: "laggy_interval": 0, 2024-03-11T21:28:09.035 INFO:teuthology.orchestra.run.smithi092.stdout: "features": 4540701547738038271, 2024-03-11T21:28:09.035 INFO:teuthology.orchestra.run.smithi092.stdout: "old_weight": 0, 2024-03-11T21:28:09.035 INFO:teuthology.orchestra.run.smithi092.stdout: "last_purged_snaps_scrub": "2024-03-11T21:27:39.377465+0000", 2024-03-11T21:28:09.035 INFO:teuthology.orchestra.run.smithi092.stdout: "dead_epoch": 0 2024-03-11T21:28:09.035 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:28:09.036 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:28:09.036 INFO:teuthology.orchestra.run.smithi092.stdout: "osd": 4, 2024-03-11T21:28:09.036 INFO:teuthology.orchestra.run.smithi092.stdout: "down_stamp": "0.000000", 2024-03-11T21:28:09.036 INFO:teuthology.orchestra.run.smithi092.stdout: "laggy_probability": 0, 2024-03-11T21:28:09.036 INFO:teuthology.orchestra.run.smithi092.stdout: "laggy_interval": 0, 2024-03-11T21:28:09.036 INFO:teuthology.orchestra.run.smithi092.stdout: "features": 4540701547738038271, 2024-03-11T21:28:09.036 INFO:teuthology.orchestra.run.smithi092.stdout: "old_weight": 0, 2024-03-11T21:28:09.036 INFO:teuthology.orchestra.run.smithi092.stdout: "last_purged_snaps_scrub": "2024-03-11T21:28:01.650521+0000", 2024-03-11T21:28:09.036 INFO:teuthology.orchestra.run.smithi092.stdout: "dead_epoch": 0 2024-03-11T21:28:09.036 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:28:09.036 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:28:09.036 INFO:teuthology.orchestra.run.smithi092.stdout: "osd": 5, 2024-03-11T21:28:09.036 INFO:teuthology.orchestra.run.smithi092.stdout: "down_stamp": "0.000000", 2024-03-11T21:28:09.036 INFO:teuthology.orchestra.run.smithi092.stdout: "laggy_probability": 0, 2024-03-11T21:28:09.036 INFO:teuthology.orchestra.run.smithi092.stdout: "laggy_interval": 0, 2024-03-11T21:28:09.036 INFO:teuthology.orchestra.run.smithi092.stdout: "features": 4540701547738038271, 2024-03-11T21:28:09.037 INFO:teuthology.orchestra.run.smithi092.stdout: "old_weight": 0, 2024-03-11T21:28:09.037 INFO:teuthology.orchestra.run.smithi092.stdout: "last_purged_snaps_scrub": "2024-03-11T21:27:13.349177+0000", 2024-03-11T21:28:09.037 INFO:teuthology.orchestra.run.smithi092.stdout: "dead_epoch": 0 2024-03-11T21:28:09.037 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:28:09.037 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:28:09.037 INFO:teuthology.orchestra.run.smithi092.stdout: "osd": 6, 2024-03-11T21:28:09.037 INFO:teuthology.orchestra.run.smithi092.stdout: "down_stamp": "0.000000", 2024-03-11T21:28:09.037 INFO:teuthology.orchestra.run.smithi092.stdout: "laggy_probability": 0, 2024-03-11T21:28:09.037 INFO:teuthology.orchestra.run.smithi092.stdout: "laggy_interval": 0, 2024-03-11T21:28:09.037 INFO:teuthology.orchestra.run.smithi092.stdout: "features": 4540701547738038271, 2024-03-11T21:28:09.037 INFO:teuthology.orchestra.run.smithi092.stdout: "old_weight": 0, 2024-03-11T21:28:09.037 INFO:teuthology.orchestra.run.smithi092.stdout: "last_purged_snaps_scrub": "2024-03-11T21:27:26.483299+0000", 2024-03-11T21:28:09.037 INFO:teuthology.orchestra.run.smithi092.stdout: "dead_epoch": 0 2024-03-11T21:28:09.037 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:28:09.038 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:28:09.038 INFO:teuthology.orchestra.run.smithi092.stdout: "osd": 7, 2024-03-11T21:28:09.038 INFO:teuthology.orchestra.run.smithi092.stdout: "down_stamp": "0.000000", 2024-03-11T21:28:09.038 INFO:teuthology.orchestra.run.smithi092.stdout: "laggy_probability": 0, 2024-03-11T21:28:09.038 INFO:teuthology.orchestra.run.smithi092.stdout: "laggy_interval": 0, 2024-03-11T21:28:09.038 INFO:teuthology.orchestra.run.smithi092.stdout: "features": 4540701547738038271, 2024-03-11T21:28:09.038 INFO:teuthology.orchestra.run.smithi092.stdout: "old_weight": 0, 2024-03-11T21:28:09.038 INFO:teuthology.orchestra.run.smithi092.stdout: "last_purged_snaps_scrub": "2024-03-11T21:28:06.763712+0000", 2024-03-11T21:28:09.038 INFO:teuthology.orchestra.run.smithi092.stdout: "dead_epoch": 0 2024-03-11T21:28:09.038 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-03-11T21:28:09.038 INFO:teuthology.orchestra.run.smithi092.stdout: ], 2024-03-11T21:28:09.039 INFO:teuthology.orchestra.run.smithi092.stdout: "pg_upmap": [], 2024-03-11T21:28:09.039 INFO:teuthology.orchestra.run.smithi092.stdout: "pg_upmap_items": [ 2024-03-11T21:28:09.039 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:28:09.039 INFO:teuthology.orchestra.run.smithi092.stdout: "pgid": "2.3", 2024-03-11T21:28:09.039 INFO:teuthology.orchestra.run.smithi092.stdout: "mappings": [ 2024-03-11T21:28:09.039 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:28:09.039 INFO:teuthology.orchestra.run.smithi092.stdout: "from": 7, 2024-03-11T21:28:09.039 INFO:teuthology.orchestra.run.smithi092.stdout: "to": 4 2024-03-11T21:28:09.039 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-03-11T21:28:09.039 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-03-11T21:28:09.039 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:28:09.039 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:28:09.039 INFO:teuthology.orchestra.run.smithi092.stdout: "pgid": "4.f", 2024-03-11T21:28:09.039 INFO:teuthology.orchestra.run.smithi092.stdout: "mappings": [ 2024-03-11T21:28:09.040 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:28:09.040 INFO:teuthology.orchestra.run.smithi092.stdout: "from": 1, 2024-03-11T21:28:09.040 INFO:teuthology.orchestra.run.smithi092.stdout: "to": 2 2024-03-11T21:28:09.040 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-03-11T21:28:09.040 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-03-11T21:28:09.040 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:28:09.040 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:28:09.040 INFO:teuthology.orchestra.run.smithi092.stdout: "pgid": "4.13", 2024-03-11T21:28:09.040 INFO:teuthology.orchestra.run.smithi092.stdout: "mappings": [ 2024-03-11T21:28:09.040 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:28:09.040 INFO:teuthology.orchestra.run.smithi092.stdout: "from": 1, 2024-03-11T21:28:09.040 INFO:teuthology.orchestra.run.smithi092.stdout: "to": 2 2024-03-11T21:28:09.040 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-03-11T21:28:09.040 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-03-11T21:28:09.041 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:28:09.041 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:28:09.041 INFO:teuthology.orchestra.run.smithi092.stdout: "pgid": "4.17", 2024-03-11T21:28:09.041 INFO:teuthology.orchestra.run.smithi092.stdout: "mappings": [ 2024-03-11T21:28:09.041 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:28:09.041 INFO:teuthology.orchestra.run.smithi092.stdout: "from": 1, 2024-03-11T21:28:09.041 INFO:teuthology.orchestra.run.smithi092.stdout: "to": 5 2024-03-11T21:28:09.041 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-03-11T21:28:09.041 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-03-11T21:28:09.041 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-03-11T21:28:09.041 INFO:teuthology.orchestra.run.smithi092.stdout: ], 2024-03-11T21:28:09.041 INFO:teuthology.orchestra.run.smithi092.stdout: "pg_upmap_primaries": [], 2024-03-11T21:28:09.041 INFO:teuthology.orchestra.run.smithi092.stdout: "pg_temp": [], 2024-03-11T21:28:09.041 INFO:teuthology.orchestra.run.smithi092.stdout: "primary_temp": [], 2024-03-11T21:28:09.041 INFO:teuthology.orchestra.run.smithi092.stdout: "blocklist": { 2024-03-11T21:28:09.042 INFO:teuthology.orchestra.run.smithi092.stdout: "172.21.15.92:0/303695351": "2024-03-12T21:26:20.943013+0000", 2024-03-11T21:28:09.042 INFO:teuthology.orchestra.run.smithi092.stdout: "172.21.15.92:0/437911229": "2024-03-12T21:26:20.943013+0000", 2024-03-11T21:28:09.042 INFO:teuthology.orchestra.run.smithi092.stdout: "172.21.15.92:0/4246054050": "2024-03-12T21:26:20.943013+0000", 2024-03-11T21:28:09.042 INFO:teuthology.orchestra.run.smithi092.stdout: "172.21.15.92:0/2912406656": "2024-03-12T21:26:20.943013+0000", 2024-03-11T21:28:09.042 INFO:teuthology.orchestra.run.smithi092.stdout: "172.21.15.92:0/2706821241": "2024-03-12T21:26:20.943013+0000", 2024-03-11T21:28:09.042 INFO:teuthology.orchestra.run.smithi092.stdout: "172.21.15.92:0/3817549754": "2024-03-12T21:21:09.473916+0000", 2024-03-11T21:28:09.042 INFO:teuthology.orchestra.run.smithi092.stdout: "172.21.15.92:0/2211594569": "2024-03-12T21:21:09.473916+0000", 2024-03-11T21:28:09.042 INFO:teuthology.orchestra.run.smithi092.stdout: "172.21.15.92:0/3569900179": "2024-03-12T21:20:43.677016+0000", 2024-03-11T21:28:09.042 INFO:teuthology.orchestra.run.smithi092.stdout: "172.21.15.92:0/3380359534": "2024-03-12T21:20:43.677016+0000", 2024-03-11T21:28:09.042 INFO:teuthology.orchestra.run.smithi092.stdout: "172.21.15.92:0/2509373130": "2024-03-12T21:20:43.677016+0000", 2024-03-11T21:28:09.042 INFO:teuthology.orchestra.run.smithi092.stdout: "172.21.15.92:6800/2343383852": "2024-03-12T21:26:20.943013+0000", 2024-03-11T21:28:09.042 INFO:teuthology.orchestra.run.smithi092.stdout: "172.21.15.92:6800/1559007462": "2024-03-12T21:20:43.677016+0000", 2024-03-11T21:28:09.042 INFO:teuthology.orchestra.run.smithi092.stdout: "172.21.15.92:6800/1270902819": "2024-03-12T21:21:09.473916+0000", 2024-03-11T21:28:09.042 INFO:teuthology.orchestra.run.smithi092.stdout: "172.21.15.92:0/3562645553": "2024-03-12T21:21:09.473916+0000" 2024-03-11T21:28:09.043 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:28:09.043 INFO:teuthology.orchestra.run.smithi092.stdout: "range_blocklist": {}, 2024-03-11T21:28:09.043 INFO:teuthology.orchestra.run.smithi092.stdout: "erasure_code_profiles": { 2024-03-11T21:28:09.043 INFO:teuthology.orchestra.run.smithi092.stdout: "default": { 2024-03-11T21:28:09.043 INFO:teuthology.orchestra.run.smithi092.stdout: "crush-failure-domain": "osd", 2024-03-11T21:28:09.043 INFO:teuthology.orchestra.run.smithi092.stdout: "k": "2", 2024-03-11T21:28:09.043 INFO:teuthology.orchestra.run.smithi092.stdout: "m": "1", 2024-03-11T21:28:09.043 INFO:teuthology.orchestra.run.smithi092.stdout: "plugin": "jerasure", 2024-03-11T21:28:09.043 INFO:teuthology.orchestra.run.smithi092.stdout: "technique": "reed_sol_van" 2024-03-11T21:28:09.043 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-03-11T21:28:09.043 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:28:09.043 INFO:teuthology.orchestra.run.smithi092.stdout: "removed_snaps_queue": [], 2024-03-11T21:28:09.043 INFO:teuthology.orchestra.run.smithi092.stdout: "new_removed_snaps": [], 2024-03-11T21:28:09.043 INFO:teuthology.orchestra.run.smithi092.stdout: "new_purged_snaps": [], 2024-03-11T21:28:09.043 INFO:teuthology.orchestra.run.smithi092.stdout: "crush_node_flags": {}, 2024-03-11T21:28:09.044 INFO:teuthology.orchestra.run.smithi092.stdout: "device_class_flags": {}, 2024-03-11T21:28:09.044 INFO:teuthology.orchestra.run.smithi092.stdout: "stretch_mode": { 2024-03-11T21:28:09.044 INFO:teuthology.orchestra.run.smithi092.stdout: "stretch_mode_enabled": false, 2024-03-11T21:28:09.044 INFO:teuthology.orchestra.run.smithi092.stdout: "stretch_bucket_count": 0, 2024-03-11T21:28:09.044 INFO:teuthology.orchestra.run.smithi092.stdout: "degraded_stretch_mode": 0, 2024-03-11T21:28:09.044 INFO:teuthology.orchestra.run.smithi092.stdout: "recovering_stretch_mode": 0, 2024-03-11T21:28:09.044 INFO:teuthology.orchestra.run.smithi092.stdout: "stretch_mode_bucket": 0 2024-03-11T21:28:09.044 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-03-11T21:28:09.044 INFO:teuthology.orchestra.run.smithi092.stdout:} 2024-03-11T21:28:09.077 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:28:08 smithi097 ceph-mon[33317]: pgmap v76: 148 pgs: 148 active+clean; 2.6 GiB data, 8.2 GiB used, 707 GiB / 715 GiB avail; 1.2 KiB/s rd, 98 MiB/s wr, 1.57k op/s 2024-03-11T21:28:09.077 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:28:08 smithi097 ceph-mon[33317]: 2.9 scrub starts 2024-03-11T21:28:09.077 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:28:08 smithi097 ceph-mon[33317]: 2.9 scrub ok 2024-03-11T21:28:09.077 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:28:08 smithi097 ceph-mon[33317]: 2.1e scrub starts 2024-03-11T21:28:09.077 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:28:08 smithi097 ceph-mon[33317]: 2.1e scrub ok 2024-03-11T21:28:09.077 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:28:08 smithi097 ceph-mon[33317]: 4.9 scrub starts 2024-03-11T21:28:09.077 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:28:08 smithi097 ceph-mon[33317]: 4.9 scrub ok 2024-03-11T21:28:09.087 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:28:09.087 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:28:09.088 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:28:09.189 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:28:09.230 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:28:09.231 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:28:09.231 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:28:09.332 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:28:09.385 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:28:09.386 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:28:09.386 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:28:09.487 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:28:09.532 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:28:09.533 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:28:09.533 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:28:09.624 INFO:tasks.radosbench.radosbench.0.smithi092.stdout: 25 16 51120 51104 127.732 85.1875 0.0256281 0.00782284 2024-03-11T21:28:09.634 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:28:09.666 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:28:09.667 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:28:09.667 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:28:09.768 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:28:09.799 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:28:09.800 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:28:09.800 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:28:09.901 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:28:09.932 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:28:09.934 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:28:09.934 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:28:10.035 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:28:10.070 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:28:10.071 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:28:10.071 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:28:10.111 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:28:09 smithi092 ceph-mon[29572]: 4.2 scrub starts 2024-03-11T21:28:10.111 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:28:09 smithi092 ceph-mon[29572]: 4.2 scrub ok 2024-03-11T21:28:10.111 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:28:09 smithi092 ceph-mon[29572]: from='client.? v1:172.21.15.92:0/1254996042' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json-pretty"}]: dispatch 2024-03-11T21:28:10.111 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:28:09 smithi092 ceph-mon[37563]: 4.2 scrub starts 2024-03-11T21:28:10.111 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:28:09 smithi092 ceph-mon[37563]: 4.2 scrub ok 2024-03-11T21:28:10.111 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:28:09 smithi092 ceph-mon[37563]: from='client.? v1:172.21.15.92:0/1254996042' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json-pretty"}]: dispatch 2024-03-11T21:28:10.157 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:28:09 smithi097 ceph-mon[33317]: 4.2 scrub starts 2024-03-11T21:28:10.157 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:28:09 smithi097 ceph-mon[33317]: 4.2 scrub ok 2024-03-11T21:28:10.157 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:28:09 smithi097 ceph-mon[33317]: from='client.? v1:172.21.15.92:0/1254996042' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json-pretty"}]: dispatch 2024-03-11T21:28:10.172 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:28:10.206 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:28:10.206 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:28:10.207 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:28:10.308 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:28:10.346 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:28:10.349 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:28:10.350 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:28:10.382 INFO:tasks.thrashosds.thrasher:j is {'epoch': 107, 'fsid': '00db0016-dfed-11ee-95c8-87774f69a715', 'created': '2024-03-11T21:20:10.488059+0000', 'modified': '2024-03-11T21:28:06.830290+0000', 'last_up_change': '2024-03-11T21:25:26.018427+0000', 'last_in_change': '2024-03-11T21:25:08.189557+0000', 'flags': 'nodeep-scrub,sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit', 'flags_num': 5804032, 'flags_set': ['nodeep-scrub', 'pglog_hardlimit', 'purged_snapdirs', 'recovery_deletes', 'sortbitwise'], 'crush_version': 18, 'full_ratio': 0.949999988079071, 'backfillfull_ratio': 0.8999999761581421, 'nearfull_ratio': 0.8500000238418579, 'cluster_snapshot': '', 'pool_max': 7, 'max_osd': 8, 'require_min_compat_client': 'luminous', 'min_compat_client': 'luminous', 'require_osd_release': 'squid', 'allow_crimson': False, 'pools': [{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-11T21:23:23.805029+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '22', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 7.889999866485596, 'score_stable': 7.889999866485596, 'optimal_score': 0.3799999952316284, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': '.rgw.root', 'create_time': '2024-03-11T21:25:37.044504+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '68', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rgw': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'datapool', 'create_time': '2024-03-11T21:25:37.280820+0000', 'flags': 8193, 'flags_names': 'hashpspool,selfmanaged_snaps', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 3, 'pg_placement_num': 3, 'pg_placement_num_target': 3, 'pg_num_target': 3, 'pg_num_pending': 3, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '71', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 2, 'snap_epoch': 71, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 2.6500000953674316, 'score_stable': 2.6500000953674316, 'optimal_score': 0.8799999952316284, 'raw_score_acting': 2.3299999237060547, 'raw_score_stable': 2.3299999237060547, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'default.rgw.log', 'create_time': '2024-03-11T21:25:38.850207+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '70', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rgw': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 2.25, 'score_stable': 2.25, 'optimal_score': 1, 'raw_score_acting': 2.25, 'raw_score_stable': 2.25, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 5, 'pool_name': 'default.rgw.control', 'create_time': '2024-03-11T21:25:40.794686+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '72', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rgw': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.25, 'score_stable': 1.25, 'optimal_score': 1, 'raw_score_acting': 1.25, 'raw_score_stable': 1.25, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 6, 'pool_name': 'default.rgw.meta', 'create_time': '2024-03-11T21:25:42.872077+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '74', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4}, 'application_metadata': {'rgw': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.75, 'score_stable': 1.75, 'optimal_score': 1, 'raw_score_acting': 1.75, 'raw_score_stable': 1.75, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 7, 'pool_name': 'unique_pool_0', 'create_time': '2024-03-11T21:27:35.489822+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 16, 'pg_placement_num': 16, 'pg_placement_num_target': 16, 'pg_num_target': 16, 'pg_num_pending': 16, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '104', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rados': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 2, 'score_stable': 2, 'optimal_score': 1, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}], 'osds': [{'osd': 0, 'uuid': '79f15843-f0ec-44aa-931e-4e07a74d3768', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 8, 'up_thru': 99, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.92:6801', 'nonce': 2844100201}]}, 'cluster_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.92:6802', 'nonce': 2844100201}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.92:6804', 'nonce': 2844100201}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.92:6803', 'nonce': 2844100201}]}, 'public_addr': '172.21.15.92:6801/2844100201', 'cluster_addr': '172.21.15.92:6802/2844100201', 'heartbeat_back_addr': '172.21.15.92:6804/2844100201', 'heartbeat_front_addr': '172.21.15.92:6803/2844100201', 'state': ['exists', 'up']}, {'osd': 1, 'uuid': '3a5a0e78-1293-4ace-a93f-69156098571d', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 13, 'up_thru': 99, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.92:6805', 'nonce': 3577871157}]}, 'cluster_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.92:6806', 'nonce': 3577871157}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.92:6808', 'nonce': 3577871157}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.92:6807', 'nonce': 3577871157}]}, 'public_addr': '172.21.15.92:6805/3577871157', 'cluster_addr': '172.21.15.92:6806/3577871157', 'heartbeat_back_addr': '172.21.15.92:6808/3577871157', 'heartbeat_front_addr': '172.21.15.92:6807/3577871157', 'state': ['exists', 'up']}, {'osd': 2, 'uuid': 'f207b743-0df4-40ed-a735-950e272fea60', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 18, 'up_thru': 99, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.92:6809', 'nonce': 760825152}]}, 'cluster_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.92:6810', 'nonce': 760825152}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.92:6812', 'nonce': 760825152}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.92:6811', 'nonce': 760825152}]}, 'public_addr': '172.21.15.92:6809/760825152', 'cluster_addr': '172.21.15.92:6810/760825152', 'heartbeat_back_addr': '172.21.15.92:6812/760825152', 'heartbeat_front_addr': '172.21.15.92:6811/760825152', 'state': ['exists', 'up']}, {'osd': 3, 'uuid': 'bbb6f69a-8f0f-47e6-937a-062e1902ae13', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 26, 'up_thru': 99, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.92:6813', 'nonce': 670596264}]}, 'cluster_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.92:6814', 'nonce': 670596264}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.92:6816', 'nonce': 670596264}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.92:6815', 'nonce': 670596264}]}, 'public_addr': '172.21.15.92:6813/670596264', 'cluster_addr': '172.21.15.92:6814/670596264', 'heartbeat_back_addr': '172.21.15.92:6816/670596264', 'heartbeat_front_addr': '172.21.15.92:6815/670596264', 'state': ['exists', 'up']}, {'osd': 4, 'uuid': '00266cbc-d2cf-4dee-8a96-3753ebc95ca8', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 32, 'up_thru': 99, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.97:6800', 'nonce': 420733620}]}, 'cluster_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.97:6801', 'nonce': 420733620}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.97:6803', 'nonce': 420733620}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.97:6802', 'nonce': 420733620}]}, 'public_addr': '172.21.15.97:6800/420733620', 'cluster_addr': '172.21.15.97:6801/420733620', 'heartbeat_back_addr': '172.21.15.97:6803/420733620', 'heartbeat_front_addr': '172.21.15.97:6802/420733620', 'state': ['exists', 'up']}, {'osd': 5, 'uuid': '593a6f6b-8874-40ae-8848-d51c5723a5e4', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 41, 'up_thru': 99, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.97:6804', 'nonce': 312056041}]}, 'cluster_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.97:6805', 'nonce': 312056041}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.97:6807', 'nonce': 312056041}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.97:6806', 'nonce': 312056041}]}, 'public_addr': '172.21.15.97:6804/312056041', 'cluster_addr': '172.21.15.97:6805/312056041', 'heartbeat_back_addr': '172.21.15.97:6807/312056041', 'heartbeat_front_addr': '172.21.15.97:6806/312056041', 'state': ['exists', 'up']}, {'osd': 6, 'uuid': '91035ecc-6f1a-4cec-bda6-090af9d53eff', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 50, 'up_thru': 99, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.97:6808', 'nonce': 178994999}]}, 'cluster_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.97:6809', 'nonce': 178994999}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.97:6811', 'nonce': 178994999}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.97:6810', 'nonce': 178994999}]}, 'public_addr': '172.21.15.97:6808/178994999', 'cluster_addr': '172.21.15.97:6809/178994999', 'heartbeat_back_addr': '172.21.15.97:6811/178994999', 'heartbeat_front_addr': '172.21.15.97:6810/178994999', 'state': ['exists', 'up']}, {'osd': 7, 'uuid': '3eb28c24-664a-4bd6-869d-6bf5db98f52f', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 60, 'up_thru': 99, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.97:6812', 'nonce': 3241560509}]}, 'cluster_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.97:6813', 'nonce': 3241560509}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.97:6815', 'nonce': 3241560509}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.97:6814', 'nonce': 3241560509}]}, 'public_addr': '172.21.15.97:6812/3241560509', 'cluster_addr': '172.21.15.97:6813/3241560509', 'heartbeat_back_addr': '172.21.15.97:6815/3241560509', 'heartbeat_front_addr': '172.21.15.97:6814/3241560509', 'state': ['exists', 'up']}], 'osd_xinfo': [{'osd': 0, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-03-11T21:27:40.685501+0000', 'dead_epoch': 0}, {'osd': 1, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-03-11T21:27:38.877730+0000', 'dead_epoch': 0}, {'osd': 2, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-03-11T21:27:05.095427+0000', 'dead_epoch': 0}, {'osd': 3, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-03-11T21:27:39.377465+0000', 'dead_epoch': 0}, {'osd': 4, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-03-11T21:28:01.650521+0000', 'dead_epoch': 0}, {'osd': 5, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-03-11T21:27:13.349177+0000', 'dead_epoch': 0}, {'osd': 6, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-03-11T21:27:26.483299+0000', 'dead_epoch': 0}, {'osd': 7, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-03-11T21:28:06.763712+0000', 'dead_epoch': 0}], 'pg_upmap': [], 'pg_upmap_items': [{'pgid': '2.3', 'mappings': [{'from': 7, 'to': 4}]}, {'pgid': '4.f', 'mappings': [{'from': 1, 'to': 2}]}, {'pgid': '4.13', 'mappings': [{'from': 1, 'to': 2}]}, {'pgid': '4.17', 'mappings': [{'from': 1, 'to': 5}]}], 'pg_upmap_primaries': [], 'pg_temp': [], 'primary_temp': [], 'blocklist': {'172.21.15.92:0/303695351': '2024-03-12T21:26:20.943013+0000', '172.21.15.92:0/437911229': '2024-03-12T21:26:20.943013+0000', '172.21.15.92:0/4246054050': '2024-03-12T21:26:20.943013+0000', '172.21.15.92:0/2912406656': '2024-03-12T21:26:20.943013+0000', '172.21.15.92:0/2706821241': '2024-03-12T21:26:20.943013+0000', '172.21.15.92:0/3817549754': '2024-03-12T21:21:09.473916+0000', '172.21.15.92:0/2211594569': '2024-03-12T21:21:09.473916+0000', '172.21.15.92:0/3569900179': '2024-03-12T21:20:43.677016+0000', '172.21.15.92:0/3380359534': '2024-03-12T21:20:43.677016+0000', '172.21.15.92:0/2509373130': '2024-03-12T21:20:43.677016+0000', '172.21.15.92:6800/2343383852': '2024-03-12T21:26:20.943013+0000', '172.21.15.92:6800/1559007462': '2024-03-12T21:20:43.677016+0000', '172.21.15.92:6800/1270902819': '2024-03-12T21:21:09.473916+0000', '172.21.15.92:0/3562645553': '2024-03-12T21:21:09.473916+0000'}, 'range_blocklist': {}, 'erasure_code_profiles': {'default': {'crush-failure-domain': 'osd', 'k': '2', 'm': '1', 'plugin': 'jerasure', 'technique': 'reed_sol_van'}}, 'removed_snaps_queue': [], 'new_removed_snaps': [], 'new_purged_snaps': [], 'crush_node_flags': {}, 'device_class_flags': {}, 'stretch_mode': {'stretch_mode_enabled': False, 'stretch_bucket_count': 0, 'degraded_stretch_mode': 0, 'recovering_stretch_mode': 0, 'stretch_mode_bucket': 0}} 2024-03-11T21:28:10.383 DEBUG:teuthology.orchestra.run.smithi092:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid 00db0016-dfed-11ee-95c8-87774f69a715 -- ceph pg dump --format=json 2024-03-11T21:28:10.451 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:28:10.520 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:28:10.521 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:28:10.521 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:28:10.622 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:28:10.624 INFO:tasks.radosbench.radosbench.0.smithi092.stdout: 26 16 52673 52657 126.552 97.0625 0.00974448 0.00789686 2024-03-11T21:28:10.651 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:28:10.651 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:28:10.652 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:28:10.753 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:28:10.784 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:28:10.784 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:28:10.785 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:28:10.885 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:28:10.887 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:28:10 smithi092 ceph-mon[37563]: pgmap v77: 148 pgs: 148 active+clean; 2.9 GiB data, 8.9 GiB used, 706 GiB / 715 GiB avail; 1.2 KiB/s rd, 134 MiB/s wr, 2.15k op/s 2024-03-11T21:28:10.887 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:28:10 smithi092 ceph-mon[37563]: 2.14 scrub starts 2024-03-11T21:28:10.887 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:28:10 smithi092 ceph-mon[37563]: 2.14 scrub ok 2024-03-11T21:28:10.887 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:28:10 smithi092 ceph-mon[37563]: 4.16 scrub starts 2024-03-11T21:28:10.887 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:28:10 smithi092 ceph-mon[37563]: 4.16 scrub ok 2024-03-11T21:28:10.887 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:28:10 smithi092 ceph-mon[29572]: pgmap v77: 148 pgs: 148 active+clean; 2.9 GiB data, 8.9 GiB used, 706 GiB / 715 GiB avail; 1.2 KiB/s rd, 134 MiB/s wr, 2.15k op/s 2024-03-11T21:28:10.887 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:28:10 smithi092 ceph-mon[29572]: 2.14 scrub starts 2024-03-11T21:28:10.887 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:28:10 smithi092 ceph-mon[29572]: 2.14 scrub ok 2024-03-11T21:28:10.887 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:28:10 smithi092 ceph-mon[29572]: 4.16 scrub starts 2024-03-11T21:28:10.887 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:28:10 smithi092 ceph-mon[29572]: 4.16 scrub ok 2024-03-11T21:28:10.910 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:28:10.911 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:28:10.911 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:28:11.012 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:28:11.043 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:28:11.044 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:28:11.044 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:28:11.145 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:28:11.174 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:28:11.174 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:28:11.175 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:28:11.275 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:28:11.306 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:28:11.306 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:28:11.307 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:28:11.326 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:28:10 smithi097 ceph-mon[33317]: pgmap v77: 148 pgs: 148 active+clean; 2.9 GiB data, 8.9 GiB used, 706 GiB / 715 GiB avail; 1.2 KiB/s rd, 134 MiB/s wr, 2.15k op/s 2024-03-11T21:28:11.327 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:28:10 smithi097 ceph-mon[33317]: 2.14 scrub starts 2024-03-11T21:28:11.327 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:28:10 smithi097 ceph-mon[33317]: 2.14 scrub ok 2024-03-11T21:28:11.327 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:28:10 smithi097 ceph-mon[33317]: 4.16 scrub starts 2024-03-11T21:28:11.327 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:28:10 smithi097 ceph-mon[33317]: 4.16 scrub ok 2024-03-11T21:28:11.408 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:28:11.439 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:28:11.439 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:28:11.439 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:28:11.540 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:28:11.572 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:28:11.572 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:28:11.573 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:28:11.624 INFO:tasks.radosbench.radosbench.0.smithi092.stdout: 27 16 54064 54048 125.085 86.9375 0.00879036 0.00798906 2024-03-11T21:28:11.674 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:28:11.713 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:28:11.713 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:28:11.714 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:28:11.815 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:28:11.836 DEBUG:teuthology.orchestra.run.smithi092:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid 00db0016-dfed-11ee-95c8-87774f69a715 -- ceph daemon osd.1 dump_blocked_ops 2024-03-11T21:28:11.846 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:28:11.846 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:28:11.846 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:28:11.947 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:28:11.990 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:28:11.991 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:28:11.992 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:28:12.092 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:28:12.112 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:28:11 smithi092 ceph-mon[29572]: 6.1 scrub starts 2024-03-11T21:28:12.112 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:28:11 smithi092 ceph-mon[29572]: 6.1 scrub ok 2024-03-11T21:28:12.112 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:28:11 smithi092 ceph-mon[29572]: 6.1a scrub starts 2024-03-11T21:28:12.112 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:28:11 smithi092 ceph-mon[29572]: 6.1a scrub ok 2024-03-11T21:28:12.112 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:28:11 smithi092 ceph-mon[29572]: 2.2 scrub starts 2024-03-11T21:28:12.112 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:28:11 smithi092 ceph-mon[29572]: 2.2 scrub ok 2024-03-11T21:28:12.112 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:28:11 smithi092 ceph-mon[29572]: 2.e scrub starts 2024-03-11T21:28:12.113 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:28:11 smithi092 ceph-mon[29572]: 2.e scrub ok 2024-03-11T21:28:12.113 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:28:11 smithi092 ceph-mon[29572]: from='mgr.14607 ' entity='mgr.y' 2024-03-11T21:28:12.113 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:28:11 smithi092 ceph-mon[37563]: 6.1 scrub starts 2024-03-11T21:28:12.113 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:28:11 smithi092 ceph-mon[37563]: 6.1 scrub ok 2024-03-11T21:28:12.113 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:28:11 smithi092 ceph-mon[37563]: 6.1a scrub starts 2024-03-11T21:28:12.113 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:28:11 smithi092 ceph-mon[37563]: 6.1a scrub ok 2024-03-11T21:28:12.113 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:28:11 smithi092 ceph-mon[37563]: 2.2 scrub starts 2024-03-11T21:28:12.113 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:28:11 smithi092 ceph-mon[37563]: 2.2 scrub ok 2024-03-11T21:28:12.113 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:28:11 smithi092 ceph-mon[37563]: 2.e scrub starts 2024-03-11T21:28:12.113 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:28:11 smithi092 ceph-mon[37563]: 2.e scrub ok 2024-03-11T21:28:12.114 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:28:11 smithi092 ceph-mon[37563]: from='mgr.14607 ' entity='mgr.y' 2024-03-11T21:28:12.139 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:28:12.139 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:28:12.139 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:28:12.240 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:28:12.284 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:28:12.284 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:28:12.285 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:28:12.327 INFO:journalctl@ceph.iscsi.iscsi.a.smithi097.stdout:Mar 11 21:28:12 smithi097 ceph-00db0016-dfed-11ee-95c8-87774f69a715-iscsi-iscsi-a[60943]: debug there is no tcmu-runner data available 2024-03-11T21:28:12.327 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:28:11 smithi097 ceph-mon[33317]: 6.1 scrub starts 2024-03-11T21:28:12.327 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:28:11 smithi097 ceph-mon[33317]: 6.1 scrub ok 2024-03-11T21:28:12.327 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:28:11 smithi097 ceph-mon[33317]: 6.1a scrub starts 2024-03-11T21:28:12.327 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:28:11 smithi097 ceph-mon[33317]: 6.1a scrub ok 2024-03-11T21:28:12.327 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:28:11 smithi097 ceph-mon[33317]: 2.2 scrub starts 2024-03-11T21:28:12.327 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:28:11 smithi097 ceph-mon[33317]: 2.2 scrub ok 2024-03-11T21:28:12.327 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:28:11 smithi097 ceph-mon[33317]: 2.e scrub starts 2024-03-11T21:28:12.327 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:28:11 smithi097 ceph-mon[33317]: 2.e scrub ok 2024-03-11T21:28:12.327 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:28:11 smithi097 ceph-mon[33317]: from='mgr.14607 ' entity='mgr.y' 2024-03-11T21:28:12.385 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:28:12.421 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:28:12.422 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:28:12.422 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:28:12.523 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:28:12.573 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:28:12.574 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:28:12.574 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:28:12.624 INFO:tasks.radosbench.radosbench.0.smithi092.stdout: 28 16 55440 55424 123.689 86 0.00277491 0.00806778 2024-03-11T21:28:12.676 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:28:12.710 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:28:12.711 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:28:12.711 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:28:12.812 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:28:12.835 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:28:12.835 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:28:12.836 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:28:12.937 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:28:12.976 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:28:12.977 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:28:12.978 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:28:13.077 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:28:12 smithi097 ceph-mon[33317]: pgmap v78: 148 pgs: 148 active+clean; 2.9 GiB data, 8.9 GiB used, 706 GiB / 715 GiB avail; 1.1 KiB/s rd, 117 MiB/s wr, 1.87k op/s 2024-03-11T21:28:13.077 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:28:12 smithi097 ceph-mon[33317]: 2.1d scrub starts 2024-03-11T21:28:13.077 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:28:12 smithi097 ceph-mon[33317]: 2.1d scrub ok 2024-03-11T21:28:13.077 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:28:12 smithi097 ceph-mon[33317]: 6.1e scrub starts 2024-03-11T21:28:13.077 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:28:12 smithi097 ceph-mon[33317]: 6.1e scrub ok 2024-03-11T21:28:13.078 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:28:13.109 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:28:13.110 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:28:13.110 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:28:13.211 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:28:13.250 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:28:13.251 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:28:13.251 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:28:13.284 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:28:12 smithi092 ceph-mon[29572]: pgmap v78: 148 pgs: 148 active+clean; 2.9 GiB data, 8.9 GiB used, 706 GiB / 715 GiB avail; 1.1 KiB/s rd, 117 MiB/s wr, 1.87k op/s 2024-03-11T21:28:13.284 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:28:12 smithi092 ceph-mon[29572]: 2.1d scrub starts 2024-03-11T21:28:13.284 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:28:12 smithi092 ceph-mon[29572]: 2.1d scrub ok 2024-03-11T21:28:13.284 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:28:12 smithi092 ceph-mon[29572]: 6.1e scrub starts 2024-03-11T21:28:13.284 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:28:12 smithi092 ceph-mon[29572]: 6.1e scrub ok 2024-03-11T21:28:13.285 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:28:12 smithi092 ceph-mon[37563]: pgmap v78: 148 pgs: 148 active+clean; 2.9 GiB data, 8.9 GiB used, 706 GiB / 715 GiB avail; 1.1 KiB/s rd, 117 MiB/s wr, 1.87k op/s 2024-03-11T21:28:13.285 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:28:12 smithi092 ceph-mon[37563]: 2.1d scrub starts 2024-03-11T21:28:13.285 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:28:12 smithi092 ceph-mon[37563]: 2.1d scrub ok 2024-03-11T21:28:13.285 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:28:12 smithi092 ceph-mon[37563]: 6.1e scrub starts 2024-03-11T21:28:13.285 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:28:12 smithi092 ceph-mon[37563]: 6.1e scrub ok 2024-03-11T21:28:13.352 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:28:13.392 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:28:13.393 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:28:13.393 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:28:13.494 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:28:13.531 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:28:13.534 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:28:13.535 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:28:13.624 INFO:tasks.radosbench.radosbench.0.smithi092.stdout: 29 16 56731 56715 122.206 80.6875 0.0024219 0.00815828 2024-03-11T21:28:13.636 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:28:13.668 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:28:13.669 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:28:13.669 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:28:13.770 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:28:13.799 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:28:13.799 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:28:13.800 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:28:13.900 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:28:13.940 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:28:13.941 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:28:13.941 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:28:14.042 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:28:14.087 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:28:14.087 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:28:14.087 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:28:14.188 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:28:14.227 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:28:14.228 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:28:14.229 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:28:14.327 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:28:13 smithi097 ceph-mon[33317]: from='client.24469 v1:172.21.15.97:0/404208310' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:28:14.327 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:28:13 smithi097 ceph-mon[33317]: 5.4 scrub starts 2024-03-11T21:28:14.327 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:28:13 smithi097 ceph-mon[33317]: 5.4 scrub ok 2024-03-11T21:28:14.327 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:28:13 smithi097 ceph-mon[33317]: 2.3 scrub starts 2024-03-11T21:28:14.327 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:28:13 smithi097 ceph-mon[33317]: 2.3 scrub ok 2024-03-11T21:28:14.334 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:28:14.366 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:28:13 smithi092 ceph-mon[29572]: from='client.24469 v1:172.21.15.97:0/404208310' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:28:14.366 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:28:13 smithi092 ceph-mon[29572]: 5.4 scrub starts 2024-03-11T21:28:14.367 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:28:13 smithi092 ceph-mon[29572]: 5.4 scrub ok 2024-03-11T21:28:14.367 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:28:13 smithi092 ceph-mon[29572]: 2.3 scrub starts 2024-03-11T21:28:14.367 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:28:13 smithi092 ceph-mon[29572]: 2.3 scrub ok 2024-03-11T21:28:14.367 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:28:13 smithi092 ceph-mon[37563]: from='client.24469 v1:172.21.15.97:0/404208310' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:28:14.367 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:28:13 smithi092 ceph-mon[37563]: 5.4 scrub starts 2024-03-11T21:28:14.367 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:28:13 smithi092 ceph-mon[37563]: 5.4 scrub ok 2024-03-11T21:28:14.367 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:28:13 smithi092 ceph-mon[37563]: 2.3 scrub starts 2024-03-11T21:28:14.367 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:28:13 smithi092 ceph-mon[37563]: 2.3 scrub ok 2024-03-11T21:28:14.375 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:28:14.376 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:28:14.376 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:28:14.477 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:28:14.510 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:28:14.510 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:28:14.511 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:28:14.611 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:28:14.626 INFO:tasks.radosbench.radosbench.0.smithi092.stdout: 30 15 58077 58062 120.933 84.1875 0.00420313 0.00826361 2024-03-11T21:28:14.650 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:28:14.651 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:28:14.651 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:28:14.752 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:28:14.789 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:28:14.790 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:28:14.790 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:28:14.891 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:28:14.918 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:28:14.919 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:28:14.919 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:28:15.020 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:28:15.065 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:28:15.066 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:28:15.067 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:28:15.167 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:28:15.206 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:28:15.207 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:28:15.207 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:28:15.308 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:28:15.326 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:28:14 smithi097 ceph-mon[33317]: pgmap v79: 148 pgs: 148 active+clean; 3.3 GiB data, 10 GiB used, 705 GiB / 715 GiB avail; 1023 B/s rd, 95 MiB/s wr, 1.52k op/s 2024-03-11T21:28:15.326 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:28:14 smithi097 ceph-mon[33317]: from='mgr.14607 ' entity='mgr.y' 2024-03-11T21:28:15.327 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:28:14 smithi097 ceph-mon[33317]: from='mgr.14607 ' entity='mgr.y' 2024-03-11T21:28:15.327 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:28:14 smithi097 ceph-mon[33317]: 2.1f scrub starts 2024-03-11T21:28:15.327 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:28:14 smithi097 ceph-mon[33317]: 2.1f scrub ok 2024-03-11T21:28:15.327 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:28:14 smithi097 ceph-mon[33317]: purged_snaps scrub starts 2024-03-11T21:28:15.327 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:28:14 smithi097 ceph-mon[33317]: purged_snaps scrub ok 2024-03-11T21:28:15.344 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:28:15.344 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:28:15.345 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:28:15.361 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:28:14 smithi092 ceph-mon[29572]: pgmap v79: 148 pgs: 148 active+clean; 3.3 GiB data, 10 GiB used, 705 GiB / 715 GiB avail; 1023 B/s rd, 95 MiB/s wr, 1.52k op/s 2024-03-11T21:28:15.361 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:28:14 smithi092 ceph-mon[29572]: from='mgr.14607 ' entity='mgr.y' 2024-03-11T21:28:15.361 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:28:14 smithi092 ceph-mon[29572]: from='mgr.14607 ' entity='mgr.y' 2024-03-11T21:28:15.361 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:28:14 smithi092 ceph-mon[29572]: 2.1f scrub starts 2024-03-11T21:28:15.361 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:28:14 smithi092 ceph-mon[29572]: 2.1f scrub ok 2024-03-11T21:28:15.361 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:28:14 smithi092 ceph-mon[29572]: purged_snaps scrub starts 2024-03-11T21:28:15.361 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:28:14 smithi092 ceph-mon[29572]: purged_snaps scrub ok 2024-03-11T21:28:15.362 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:28:14 smithi092 ceph-mon[37563]: pgmap v79: 148 pgs: 148 active+clean; 3.3 GiB data, 10 GiB used, 705 GiB / 715 GiB avail; 1023 B/s rd, 95 MiB/s wr, 1.52k op/s 2024-03-11T21:28:15.362 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:28:14 smithi092 ceph-mon[37563]: from='mgr.14607 ' entity='mgr.y' 2024-03-11T21:28:15.362 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:28:14 smithi092 ceph-mon[37563]: from='mgr.14607 ' entity='mgr.y' 2024-03-11T21:28:15.362 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:28:14 smithi092 ceph-mon[37563]: 2.1f scrub starts 2024-03-11T21:28:15.362 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:28:14 smithi092 ceph-mon[37563]: 2.1f scrub ok 2024-03-11T21:28:15.362 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:28:14 smithi092 ceph-mon[37563]: purged_snaps scrub starts 2024-03-11T21:28:15.362 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:28:14 smithi092 ceph-mon[37563]: purged_snaps scrub ok 2024-03-11T21:28:15.446 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:28:15.479 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:28:15.479 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:28:15.480 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:28:15.581 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:28:15.618 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:28:15.618 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:28:15.619 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:28:15.626 INFO:tasks.radosbench.radosbench.0.smithi092.stdout: 31 16 59390 59374 119.677 82 0.0158973 0.00835003 2024-03-11T21:28:15.720 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:28:15.785 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:28:15.785 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:28:15.786 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:28:15.887 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:28:15.911 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:28:15.912 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:28:15.912 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:28:16.013 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:28:16.042 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:28:16.043 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:28:16.043 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:28:16.144 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:28:16.181 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:28:16.181 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:28:16.182 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:28:16.283 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:28:16.322 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:28:16.322 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:28:16.323 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:28:16.326 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:28:16 smithi097 ceph-mon[33317]: 5.9 scrub starts 2024-03-11T21:28:16.326 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:28:16 smithi097 ceph-mon[33317]: 5.9 scrub ok 2024-03-11T21:28:16.327 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:28:16 smithi097 ceph-mon[33317]: 4.1b scrub starts 2024-03-11T21:28:16.327 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:28:16 smithi097 ceph-mon[33317]: 4.1b scrub ok 2024-03-11T21:28:16.327 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:28:16 smithi097 ceph-mon[33317]: pgmap v80: 148 pgs: 148 active+clean; 3.3 GiB data, 10 GiB used, 705 GiB / 715 GiB avail; 1023 B/s rd, 95 MiB/s wr, 1.52k op/s 2024-03-11T21:28:16.327 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:28:16 smithi097 ceph-mon[33317]: osdmap e108: 8 total, 8 up, 8 in 2024-03-11T21:28:16.327 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:28:16 smithi097 ceph-mon[33317]: 3.2 scrub starts 2024-03-11T21:28:16.327 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:28:16 smithi097 ceph-mon[33317]: 3.2 scrub ok 2024-03-11T21:28:16.361 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:28:16 smithi092 ceph-mon[37563]: 5.9 scrub starts 2024-03-11T21:28:16.361 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:28:16 smithi092 ceph-mon[37563]: 5.9 scrub ok 2024-03-11T21:28:16.361 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:28:16 smithi092 ceph-mon[37563]: 4.1b scrub starts 2024-03-11T21:28:16.361 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:28:16 smithi092 ceph-mon[37563]: 4.1b scrub ok 2024-03-11T21:28:16.361 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:28:16 smithi092 ceph-mon[37563]: pgmap v80: 148 pgs: 148 active+clean; 3.3 GiB data, 10 GiB used, 705 GiB / 715 GiB avail; 1023 B/s rd, 95 MiB/s wr, 1.52k op/s 2024-03-11T21:28:16.361 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:28:16 smithi092 ceph-mon[37563]: osdmap e108: 8 total, 8 up, 8 in 2024-03-11T21:28:16.361 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:28:16 smithi092 ceph-mon[37563]: 3.2 scrub starts 2024-03-11T21:28:16.361 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:28:16 smithi092 ceph-mon[37563]: 3.2 scrub ok 2024-03-11T21:28:16.362 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:28:16 smithi092 ceph-mon[29572]: 5.9 scrub starts 2024-03-11T21:28:16.362 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:28:16 smithi092 ceph-mon[29572]: 5.9 scrub ok 2024-03-11T21:28:16.362 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:28:16 smithi092 ceph-mon[29572]: 4.1b scrub starts 2024-03-11T21:28:16.362 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:28:16 smithi092 ceph-mon[29572]: 4.1b scrub ok 2024-03-11T21:28:16.362 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:28:16 smithi092 ceph-mon[29572]: pgmap v80: 148 pgs: 148 active+clean; 3.3 GiB data, 10 GiB used, 705 GiB / 715 GiB avail; 1023 B/s rd, 95 MiB/s wr, 1.52k op/s 2024-03-11T21:28:16.362 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:28:16 smithi092 ceph-mon[29572]: osdmap e108: 8 total, 8 up, 8 in 2024-03-11T21:28:16.362 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:28:16 smithi092 ceph-mon[29572]: 3.2 scrub starts 2024-03-11T21:28:16.362 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:28:16 smithi092 ceph-mon[29572]: 3.2 scrub ok 2024-03-11T21:28:16.424 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:28:16.461 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:28:16.462 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:28:16.462 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:28:16.563 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:28:16.602 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:28:16.603 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:28:16.603 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:28:16.626 INFO:tasks.radosbench.radosbench.0.smithi092.stdout: 32 16 60584 60568 118.269 74.625 0.00300794 0.008446 2024-03-11T21:28:16.704 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:28:16.749 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:28:16.750 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:28:16.750 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:28:16.851 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:28:16.890 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:28:16.891 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:28:16.891 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:28:16.992 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:28:17.024 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:28:17.025 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:28:17.025 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:28:17.126 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:28:17.153 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:28:17.154 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:28:17.154 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:28:17.255 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:28:17.286 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:28:17.287 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:28:17.288 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:28:17.327 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:28:17 smithi097 ceph-mon[33317]: 6.1b scrub starts 2024-03-11T21:28:17.327 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:28:17 smithi097 ceph-mon[33317]: 6.1b scrub ok 2024-03-11T21:28:17.327 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:28:17 smithi097 ceph-mon[33317]: 2.18 scrub starts 2024-03-11T21:28:17.327 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:28:17 smithi097 ceph-mon[33317]: 2.18 scrub ok 2024-03-11T21:28:17.361 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:28:17 smithi092 ceph-mon[29572]: 6.1b scrub starts 2024-03-11T21:28:17.361 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:28:17 smithi092 ceph-mon[29572]: 6.1b scrub ok 2024-03-11T21:28:17.361 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:28:17 smithi092 ceph-mon[29572]: 2.18 scrub starts 2024-03-11T21:28:17.361 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:28:17 smithi092 ceph-mon[29572]: 2.18 scrub ok 2024-03-11T21:28:17.361 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:28:17 smithi092 ceph-mon[37563]: 6.1b scrub starts 2024-03-11T21:28:17.361 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:28:17 smithi092 ceph-mon[37563]: 6.1b scrub ok 2024-03-11T21:28:17.362 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:28:17 smithi092 ceph-mon[37563]: 2.18 scrub starts 2024-03-11T21:28:17.362 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:28:17 smithi092 ceph-mon[37563]: 2.18 scrub ok 2024-03-11T21:28:17.389 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:28:17.413 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:28:17.414 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:28:17.414 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:28:17.515 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:28:17.539 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:28:17.539 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:28:17.539 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:28:17.626 INFO:tasks.radosbench.radosbench.0.smithi092.stdout: 33 16 61685 61669 116.77 68.8125 0.011434 0.00855857 2024-03-11T21:28:17.641 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:28:17.675 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:28:17.676 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:28:17.677 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:28:17.778 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:28:17.819 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:28:17.820 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:28:17.820 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:28:17.921 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:28:17.947 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:28:17.947 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:28:17.948 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:28:18.048 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:28:18.095 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:28:18.096 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:28:18.096 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:28:18.111 INFO:journalctl@ceph.mgr.y.smithi092.stdout:Mar 11 21:28:17 smithi092 ceph-00db0016-dfed-11ee-95c8-87774f69a715-mgr-y[29801]: ::ffff:172.21.15.97 - - [11/Mar/2024:21:28:17] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-11T21:28:18.111 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:28:18 smithi092 ceph-mon[29572]: 5.18 scrub starts 2024-03-11T21:28:18.111 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:28:18 smithi092 ceph-mon[29572]: 5.18 scrub ok 2024-03-11T21:28:18.111 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:28:18 smithi092 ceph-mon[29572]: pgmap v82: 148 pgs: 148 active+clean; 3.5 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 1023 B/s rd, 90 MiB/s wr, 1.44k op/s 2024-03-11T21:28:18.112 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:28:18 smithi092 ceph-mon[29572]: 2.1a scrub starts 2024-03-11T21:28:18.112 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:28:18 smithi092 ceph-mon[29572]: 2.1a scrub ok 2024-03-11T21:28:18.112 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:28:18 smithi092 ceph-mon[29572]: 2.1 scrub starts 2024-03-11T21:28:18.112 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:28:18 smithi092 ceph-mon[29572]: 2.1 scrub ok 2024-03-11T21:28:18.112 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:28:18 smithi092 ceph-mon[29572]: 5.1e scrub starts 2024-03-11T21:28:18.112 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:28:18 smithi092 ceph-mon[29572]: 5.1e scrub ok 2024-03-11T21:28:18.112 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:28:18 smithi092 ceph-mon[37563]: 5.18 scrub starts 2024-03-11T21:28:18.112 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:28:18 smithi092 ceph-mon[37563]: 5.18 scrub ok 2024-03-11T21:28:18.112 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:28:18 smithi092 ceph-mon[37563]: pgmap v82: 148 pgs: 148 active+clean; 3.5 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 1023 B/s rd, 90 MiB/s wr, 1.44k op/s 2024-03-11T21:28:18.112 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:28:18 smithi092 ceph-mon[37563]: 2.1a scrub starts 2024-03-11T21:28:18.112 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:28:18 smithi092 ceph-mon[37563]: 2.1a scrub ok 2024-03-11T21:28:18.113 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:28:18 smithi092 ceph-mon[37563]: 2.1 scrub starts 2024-03-11T21:28:18.113 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:28:18 smithi092 ceph-mon[37563]: 2.1 scrub ok 2024-03-11T21:28:18.113 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:28:18 smithi092 ceph-mon[37563]: 5.1e scrub starts 2024-03-11T21:28:18.113 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:28:18 smithi092 ceph-mon[37563]: 5.1e scrub ok 2024-03-11T21:28:18.197 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:28:18.240 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:28:18.240 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:28:18.241 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:28:18.327 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:28:18 smithi097 ceph-mon[33317]: 5.18 scrub starts 2024-03-11T21:28:18.327 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:28:18 smithi097 ceph-mon[33317]: 5.18 scrub ok 2024-03-11T21:28:18.327 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:28:18 smithi097 ceph-mon[33317]: pgmap v82: 148 pgs: 148 active+clean; 3.5 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 1023 B/s rd, 90 MiB/s wr, 1.44k op/s 2024-03-11T21:28:18.327 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:28:18 smithi097 ceph-mon[33317]: 2.1a scrub starts 2024-03-11T21:28:18.327 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:28:18 smithi097 ceph-mon[33317]: 2.1a scrub ok 2024-03-11T21:28:18.327 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:28:18 smithi097 ceph-mon[33317]: 2.1 scrub starts 2024-03-11T21:28:18.327 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:28:18 smithi097 ceph-mon[33317]: 2.1 scrub ok 2024-03-11T21:28:18.327 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:28:18 smithi097 ceph-mon[33317]: 5.1e scrub starts 2024-03-11T21:28:18.327 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:28:18 smithi097 ceph-mon[33317]: 5.1e scrub ok 2024-03-11T21:28:18.342 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:28:18.379 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:28:18.380 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:28:18.380 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:28:18.481 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:28:18.511 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:28:18.512 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:28:18.512 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:28:18.613 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:28:18.627 INFO:tasks.radosbench.radosbench.0.smithi092.stdout: 34 16 62847 62831 115.467 72.625 0.00832057 0.00865499 2024-03-11T21:28:18.650 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:28:18.651 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:28:18.651 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:28:18.752 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:28:18.789 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:28:18.789 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:28:18.790 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:28:18.890 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:28:18.913 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:28:18.914 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:28:18.914 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:28:19.015 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:28:19.047 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:28:19.047 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:28:19.048 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:28:19.149 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:28:19.180 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:28:19.180 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:28:19.180 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:28:19.281 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:28:19.312 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:28:19.312 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:28:19.312 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:28:19.326 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:28:19 smithi097 ceph-mon[33317]: 2.15 scrub starts 2024-03-11T21:28:19.327 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:28:19 smithi097 ceph-mon[33317]: 2.0 scrub starts 2024-03-11T21:28:19.327 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:28:19 smithi097 ceph-mon[33317]: 2.15 scrub ok 2024-03-11T21:28:19.327 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:28:19 smithi097 ceph-mon[33317]: 2.0 scrub ok 2024-03-11T21:28:19.327 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:28:19 smithi097 ceph-mon[33317]: 4.1f scrub starts 2024-03-11T21:28:19.327 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:28:19 smithi097 ceph-mon[33317]: 4.1f scrub ok 2024-03-11T21:28:19.327 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:28:19 smithi097 ceph-mon[33317]: 4.0 scrub starts 2024-03-11T21:28:19.327 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:28:19 smithi097 ceph-mon[33317]: 4.0 scrub ok 2024-03-11T21:28:19.361 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:28:19 smithi092 ceph-mon[29572]: 2.15 scrub starts 2024-03-11T21:28:19.361 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:28:19 smithi092 ceph-mon[29572]: 2.0 scrub starts 2024-03-11T21:28:19.361 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:28:19 smithi092 ceph-mon[29572]: 2.15 scrub ok 2024-03-11T21:28:19.361 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:28:19 smithi092 ceph-mon[29572]: 2.0 scrub ok 2024-03-11T21:28:19.361 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:28:19 smithi092 ceph-mon[29572]: 4.1f scrub starts 2024-03-11T21:28:19.361 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:28:19 smithi092 ceph-mon[29572]: 4.1f scrub ok 2024-03-11T21:28:19.361 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:28:19 smithi092 ceph-mon[29572]: 4.0 scrub starts 2024-03-11T21:28:19.361 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:28:19 smithi092 ceph-mon[29572]: 4.0 scrub ok 2024-03-11T21:28:19.362 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:28:19 smithi092 ceph-mon[37563]: 2.15 scrub starts 2024-03-11T21:28:19.362 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:28:19 smithi092 ceph-mon[37563]: 2.0 scrub starts 2024-03-11T21:28:19.362 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:28:19 smithi092 ceph-mon[37563]: 2.15 scrub ok 2024-03-11T21:28:19.362 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:28:19 smithi092 ceph-mon[37563]: 2.0 scrub ok 2024-03-11T21:28:19.362 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:28:19 smithi092 ceph-mon[37563]: 4.1f scrub starts 2024-03-11T21:28:19.362 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:28:19 smithi092 ceph-mon[37563]: 4.1f scrub ok 2024-03-11T21:28:19.362 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:28:19 smithi092 ceph-mon[37563]: 4.0 scrub starts 2024-03-11T21:28:19.362 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:28:19 smithi092 ceph-mon[37563]: 4.0 scrub ok 2024-03-11T21:28:19.414 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:28:19.445 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:28:19.445 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:28:19.446 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:28:19.547 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:28:19.579 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:28:19.579 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:28:19.580 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:28:19.628 INFO:tasks.radosbench.radosbench.0.smithi092.stdout: 35 16 64020 64004 114.263 73.3125 0.00958209 0.00874707 2024-03-11T21:28:19.680 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:28:19.711 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:28:19.712 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:28:19.712 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:28:19.813 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:28:19.855 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:28:19.856 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:28:19.856 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:28:19.957 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:28:20.000 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:28:20.000 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:28:20.000 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:28:20.101 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:28:20.111 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:28:20 smithi092 ceph-mon[29572]: 4.1a scrub starts 2024-03-11T21:28:20.111 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:28:20 smithi092 ceph-mon[29572]: 4.1a scrub ok 2024-03-11T21:28:20.111 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:28:20 smithi092 ceph-mon[29572]: pgmap v83: 148 pgs: 148 active+clean; 3.7 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 1023 B/s rd, 84 MiB/s wr, 1.34k op/s 2024-03-11T21:28:20.111 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:28:20 smithi092 ceph-mon[29572]: 6.a scrub starts 2024-03-11T21:28:20.111 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:28:20 smithi092 ceph-mon[29572]: 6.a scrub ok 2024-03-11T21:28:20.112 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:28:20 smithi092 ceph-mon[37563]: 4.1a scrub starts 2024-03-11T21:28:20.112 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:28:20 smithi092 ceph-mon[37563]: 4.1a scrub ok 2024-03-11T21:28:20.112 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:28:20 smithi092 ceph-mon[37563]: pgmap v83: 148 pgs: 148 active+clean; 3.7 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 1023 B/s rd, 84 MiB/s wr, 1.34k op/s 2024-03-11T21:28:20.112 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:28:20 smithi092 ceph-mon[37563]: 6.a scrub starts 2024-03-11T21:28:20.112 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:28:20 smithi092 ceph-mon[37563]: 6.a scrub ok 2024-03-11T21:28:20.141 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:28:20.142 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:28:20.142 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:28:20.243 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:28:20.274 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:28:20.275 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:28:20.276 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:28:20.326 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:28:20 smithi097 ceph-mon[33317]: 4.1a scrub starts 2024-03-11T21:28:20.327 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:28:20 smithi097 ceph-mon[33317]: 4.1a scrub ok 2024-03-11T21:28:20.327 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:28:20 smithi097 ceph-mon[33317]: pgmap v83: 148 pgs: 148 active+clean; 3.7 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 1023 B/s rd, 84 MiB/s wr, 1.34k op/s 2024-03-11T21:28:20.327 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:28:20 smithi097 ceph-mon[33317]: 6.a scrub starts 2024-03-11T21:28:20.327 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:28:20 smithi097 ceph-mon[33317]: 6.a scrub ok 2024-03-11T21:28:20.377 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:28:20.428 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:28:20.428 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:28:20.429 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:28:20.530 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:28:20.567 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:28:20.567 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:28:20.568 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:28:20.628 INFO:tasks.radosbench.radosbench.0.smithi092.stdout: 36 16 65395 65379 113.476 85.9375 0.00506116 0.00880772 2024-03-11T21:28:20.668 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:28:20.700 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:28:20.701 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:28:20.701 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:28:20.802 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:28:20.835 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:28:20.836 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:28:20.837 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:28:20.937 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:28:20.971 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:28:20.971 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:28:20.971 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:28:21.019 INFO:teuthology.orchestra.run.smithi092.stderr:Inferring config /var/lib/ceph/00db0016-dfed-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T21:28:21.072 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:28:21.102 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:28:21.103 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:28:21.103 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:28:21.204 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:28:21.253 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:28:21.254 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:28:21.254 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:28:21.308 INFO:teuthology.orchestra.run.smithi092.stderr:Inferring config /var/lib/ceph/00db0016-dfed-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T21:28:21.317 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:28:21 smithi092 ceph-mon[29572]: 2.6 scrub starts 2024-03-11T21:28:21.318 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:28:21 smithi092 ceph-mon[29572]: 2.6 scrub ok 2024-03-11T21:28:21.318 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:28:21 smithi092 ceph-mon[29572]: 6.12 scrub starts 2024-03-11T21:28:21.318 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:28:21 smithi092 ceph-mon[29572]: 6.12 scrub ok 2024-03-11T21:28:21.318 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:28:21 smithi092 ceph-mon[37563]: 2.6 scrub starts 2024-03-11T21:28:21.318 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:28:21 smithi092 ceph-mon[37563]: 2.6 scrub ok 2024-03-11T21:28:21.318 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:28:21 smithi092 ceph-mon[37563]: 6.12 scrub starts 2024-03-11T21:28:21.318 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:28:21 smithi092 ceph-mon[37563]: 6.12 scrub ok 2024-03-11T21:28:21.355 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:28:21.387 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:28:21.388 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:28:21.388 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:28:21.489 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:28:21.536 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:28:21.537 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:28:21.537 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:28:21.576 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:28:21 smithi097 ceph-mon[33317]: 2.6 scrub starts 2024-03-11T21:28:21.576 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:28:21 smithi097 ceph-mon[33317]: 2.6 scrub ok 2024-03-11T21:28:21.577 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:28:21 smithi097 ceph-mon[33317]: 6.12 scrub starts 2024-03-11T21:28:21.577 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:28:21 smithi097 ceph-mon[33317]: 6.12 scrub ok 2024-03-11T21:28:21.628 INFO:tasks.radosbench.radosbench.0.smithi092.stdout: 37 16 66665 66649 112.554 79.375 0.0153303 0.00887888 2024-03-11T21:28:21.638 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:28:21.666 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:28:21.666 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:28:21.667 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:28:21.768 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:28:21.795 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:28:21.796 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:28:21.797 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:28:21.889 INFO:teuthology.orchestra.run.smithi092.stdout:osd.0: {} 2024-03-11T21:28:21.897 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:28:21.923 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:28:21.924 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:28:21.924 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:28:21.948 INFO:teuthology.orchestra.run.smithi092.stdout:osd.1: {} 2024-03-11T21:28:21.998 INFO:teuthology.orchestra.run.smithi092.stdout:osd.2: {} 2024-03-11T21:28:22.025 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:28:22.053 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:28:22.054 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:28:22.054 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:28:22.062 INFO:teuthology.orchestra.run.smithi092.stdout:osd.3: {} 2024-03-11T21:28:22.101 INFO:teuthology.orchestra.run.smithi092.stdout:osd.4: {} 2024-03-11T21:28:22.146 INFO:teuthology.orchestra.run.smithi092.stdout:osd.5: {} 2024-03-11T21:28:22.147 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:28:22 smithi092 ceph-mon[29572]: pgmap v84: 148 pgs: 148 active+clean; 3.7 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 1022 B/s rd, 84 MiB/s wr, 1.34k op/s 2024-03-11T21:28:22.147 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:28:22 smithi092 ceph-mon[29572]: from='mgr.14607 v1:172.21.15.92:0/1702717597' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T21:28:22.147 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:28:22 smithi092 ceph-mon[29572]: from='client.? v1:172.21.15.92:0/1218977369' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-11T21:28:22.147 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:28:22 smithi092 ceph-mon[37563]: pgmap v84: 148 pgs: 148 active+clean; 3.7 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 1022 B/s rd, 84 MiB/s wr, 1.34k op/s 2024-03-11T21:28:22.147 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:28:22 smithi092 ceph-mon[37563]: from='mgr.14607 v1:172.21.15.92:0/1702717597' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T21:28:22.147 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:28:22 smithi092 ceph-mon[37563]: from='client.? v1:172.21.15.92:0/1218977369' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-11T21:28:22.155 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:28:22.179 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:28:22.180 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:28:22.180 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:28:22.190 INFO:teuthology.orchestra.run.smithi092.stdout:osd.6: {} 2024-03-11T21:28:22.238 INFO:teuthology.orchestra.run.smithi092.stdout:osd.7: {} 2024-03-11T21:28:22.248 INFO:teuthology.orchestra.run.smithi092.stderr:osd.0: osd_enable_op_tracker = '' 2024-03-11T21:28:22.248 INFO:teuthology.orchestra.run.smithi092.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-11T21:28:22.248 INFO:teuthology.orchestra.run.smithi092.stderr:osd.2: osd_enable_op_tracker = '' 2024-03-11T21:28:22.248 INFO:teuthology.orchestra.run.smithi092.stderr:osd.3: osd_enable_op_tracker = '' 2024-03-11T21:28:22.248 INFO:teuthology.orchestra.run.smithi092.stderr:osd.4: osd_enable_op_tracker = '' 2024-03-11T21:28:22.248 INFO:teuthology.orchestra.run.smithi092.stderr:osd.5: osd_enable_op_tracker = '' 2024-03-11T21:28:22.248 INFO:teuthology.orchestra.run.smithi092.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-11T21:28:22.248 INFO:teuthology.orchestra.run.smithi092.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-11T21:28:22.281 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:28:22.308 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:28:22.308 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:28:22.309 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:28:22.327 INFO:journalctl@ceph.iscsi.iscsi.a.smithi097.stdout:Mar 11 21:28:22 smithi097 ceph-00db0016-dfed-11ee-95c8-87774f69a715-iscsi-iscsi-a[60943]: debug there is no tcmu-runner data available 2024-03-11T21:28:22.327 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:28:22 smithi097 ceph-mon[33317]: pgmap v84: 148 pgs: 148 active+clean; 3.7 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 1022 B/s rd, 84 MiB/s wr, 1.34k op/s 2024-03-11T21:28:22.327 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:28:22 smithi097 ceph-mon[33317]: from='mgr.14607 v1:172.21.15.92:0/1702717597' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T21:28:22.327 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:28:22 smithi097 ceph-mon[33317]: from='client.? v1:172.21.15.92:0/1218977369' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-11T21:28:22.410 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:28:22.477 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:28:22.477 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:28:22.477 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:28:22.578 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:28:22.624 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:28:22.625 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:28:22.625 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:28:22.628 INFO:tasks.radosbench.radosbench.0.smithi092.stdout: 38 16 67589 67573 111.11 57.75 0.0171097 0.00899507 2024-03-11T21:28:22.726 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:28:22.755 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:28:22.756 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:28:22.756 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:28:22.857 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:28:22.880 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:28:22.881 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:28:22.881 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:28:22.982 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:28:23.009 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:28:23.010 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:28:23.010 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:28:23.111 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:28:23.113 INFO:teuthology.orchestra.run.smithi092.stderr:nodeep-scrub is unset 2024-03-11T21:28:23.137 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:28:23.138 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:28:23.138 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:28:23.239 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:28:23.267 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:28:23.268 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:28:23.268 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:28:23.361 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:28:23 smithi092 ceph-mon[29572]: 4.4 scrub starts 2024-03-11T21:28:23.361 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:28:23 smithi092 ceph-mon[29572]: 4.4 scrub ok 2024-03-11T21:28:23.361 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:28:23 smithi092 ceph-mon[29572]: from='client.24469 v1:172.21.15.97:0/404208310' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:28:23.361 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:28:23 smithi092 ceph-mon[29572]: from='client.? v1:172.21.15.92:0/159561909' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-11T21:28:23.361 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:28:23 smithi092 ceph-mon[37563]: 4.4 scrub starts 2024-03-11T21:28:23.361 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:28:23 smithi092 ceph-mon[37563]: 4.4 scrub ok 2024-03-11T21:28:23.362 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:28:23 smithi092 ceph-mon[37563]: from='client.24469 v1:172.21.15.97:0/404208310' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:28:23.362 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:28:23 smithi092 ceph-mon[37563]: from='client.? v1:172.21.15.92:0/159561909' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-11T21:28:23.369 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:28:23.411 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:28:23.411 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:28:23.412 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:28:23.512 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:28:23.551 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:28:23.551 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:28:23.552 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:28:23.577 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:28:23 smithi097 ceph-mon[33317]: 4.4 scrub starts 2024-03-11T21:28:23.577 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:28:23 smithi097 ceph-mon[33317]: 4.4 scrub ok 2024-03-11T21:28:23.577 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:28:23 smithi097 ceph-mon[33317]: from='client.24469 v1:172.21.15.97:0/404208310' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:28:23.577 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:28:23 smithi097 ceph-mon[33317]: from='client.? v1:172.21.15.92:0/159561909' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-11T21:28:23.629 INFO:tasks.radosbench.radosbench.0.smithi092.stdout: 39 16 68692 68676 110.028 68.9375 0.00499313 0.0090836 2024-03-11T21:28:23.652 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:28:23.707 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:28:23.707 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:28:23.708 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:28:23.808 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:28:23.837 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:28:23.837 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:28:23.838 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:28:23.938 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:28:23.979 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:28:23.980 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:28:23.980 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:28:24.081 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:28:24.129 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:28:24.129 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:28:24.130 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:28:24.231 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:28:24.262 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:28:24.262 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:28:24.263 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:28:24.298 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:28:24 smithi092 ceph-mon[29572]: 4.18 scrub starts 2024-03-11T21:28:24.298 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:28:24 smithi092 ceph-mon[29572]: 4.18 scrub ok 2024-03-11T21:28:24.299 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:28:24 smithi092 ceph-mon[29572]: pgmap v85: 148 pgs: 148 active+clean; 4.1 GiB data, 13 GiB used, 703 GiB / 715 GiB avail; 1022 B/s rd, 78 MiB/s wr, 1.24k op/s 2024-03-11T21:28:24.299 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:28:24 smithi092 ceph-mon[29572]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-03-11T21:28:24.299 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:28:24 smithi092 ceph-mon[29572]: Cluster is now healthy 2024-03-11T21:28:24.299 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:28:24 smithi092 ceph-mon[29572]: from='client.? v1:172.21.15.92:0/159561909' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-03-11T21:28:24.299 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:28:24 smithi092 ceph-mon[29572]: osdmap e109: 8 total, 8 up, 8 in 2024-03-11T21:28:24.299 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:28:24 smithi092 ceph-mon[29572]: 6.d scrub starts 2024-03-11T21:28:24.299 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:28:24 smithi092 ceph-mon[29572]: 6.d scrub ok 2024-03-11T21:28:24.299 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:28:24 smithi092 ceph-mon[37563]: 4.18 scrub starts 2024-03-11T21:28:24.299 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:28:24 smithi092 ceph-mon[37563]: 4.18 scrub ok 2024-03-11T21:28:24.299 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:28:24 smithi092 ceph-mon[37563]: pgmap v85: 148 pgs: 148 active+clean; 4.1 GiB data, 13 GiB used, 703 GiB / 715 GiB avail; 1022 B/s rd, 78 MiB/s wr, 1.24k op/s 2024-03-11T21:28:24.299 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:28:24 smithi092 ceph-mon[37563]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-03-11T21:28:24.299 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:28:24 smithi092 ceph-mon[37563]: Cluster is now healthy 2024-03-11T21:28:24.300 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:28:24 smithi092 ceph-mon[37563]: from='client.? v1:172.21.15.92:0/159561909' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-03-11T21:28:24.300 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:28:24 smithi092 ceph-mon[37563]: osdmap e109: 8 total, 8 up, 8 in 2024-03-11T21:28:24.300 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:28:24 smithi092 ceph-mon[37563]: 6.d scrub starts 2024-03-11T21:28:24.300 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:28:24 smithi092 ceph-mon[37563]: 6.d scrub ok 2024-03-11T21:28:24.363 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:28:24.391 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:28:24.392 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:28:24.392 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:28:24.493 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:28:24.534 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:28:24.535 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:28:24.536 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:28:24.577 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:28:24 smithi097 ceph-mon[33317]: 4.18 scrub starts 2024-03-11T21:28:24.577 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:28:24 smithi097 ceph-mon[33317]: 4.18 scrub ok 2024-03-11T21:28:24.577 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:28:24 smithi097 ceph-mon[33317]: pgmap v85: 148 pgs: 148 active+clean; 4.1 GiB data, 13 GiB used, 703 GiB / 715 GiB avail; 1022 B/s rd, 78 MiB/s wr, 1.24k op/s 2024-03-11T21:28:24.577 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:28:24 smithi097 ceph-mon[33317]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-03-11T21:28:24.577 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:28:24 smithi097 ceph-mon[33317]: Cluster is now healthy 2024-03-11T21:28:24.578 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:28:24 smithi097 ceph-mon[33317]: from='client.? v1:172.21.15.92:0/159561909' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-03-11T21:28:24.578 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:28:24 smithi097 ceph-mon[33317]: osdmap e109: 8 total, 8 up, 8 in 2024-03-11T21:28:24.578 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:28:24 smithi097 ceph-mon[33317]: 6.d scrub starts 2024-03-11T21:28:24.578 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:28:24 smithi097 ceph-mon[33317]: 6.d scrub ok 2024-03-11T21:28:24.629 INFO:tasks.radosbench.radosbench.0.smithi092.stdout:2024-03-11T21:28:24.647010+0000 min lat: 0.00131184 max lat: 0.114779 avg lat: 0.00915452 2024-03-11T21:28:24.629 INFO:tasks.radosbench.radosbench.0.smithi092.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-03-11T21:28:24.629 INFO:tasks.radosbench.radosbench.0.smithi092.stdout: 40 16 69900 69884 109.165 75.5 0.00652355 0.00915452 2024-03-11T21:28:24.637 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:28:24.666 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:28:24.667 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:28:24.667 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:28:24.768 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:28:24.793 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:28:24.794 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:28:24.794 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:28:24.895 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:28:24.924 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:28:24.925 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:28:24.926 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:28:25.026 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:28:25.067 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:28:25.067 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:28:25.068 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:28:25.169 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:28:25.225 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:28:25.226 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:28:25.230 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:28:25.331 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:28:25.375 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:28:25.375 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:28:25.376 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:28:25.477 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:28:25.512 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:28:25 smithi092 ceph-mon[29572]: 5.1 scrub starts 2024-03-11T21:28:25.512 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:28:25 smithi092 ceph-mon[29572]: 5.1 scrub ok 2024-03-11T21:28:25.512 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:28:25 smithi092 ceph-mon[29572]: 4.a deep-scrub starts 2024-03-11T21:28:25.513 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:28:25 smithi092 ceph-mon[29572]: 4.a deep-scrub ok 2024-03-11T21:28:25.513 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:28:25 smithi092 ceph-mon[29572]: 4.10 deep-scrub starts 2024-03-11T21:28:25.513 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:28:25 smithi092 ceph-mon[29572]: 4.10 deep-scrub ok 2024-03-11T21:28:25.513 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:28:25 smithi092 ceph-mon[37563]: 5.1 scrub starts 2024-03-11T21:28:25.513 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:28:25 smithi092 ceph-mon[37563]: 5.1 scrub ok 2024-03-11T21:28:25.513 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:28:25 smithi092 ceph-mon[37563]: 4.a deep-scrub starts 2024-03-11T21:28:25.513 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:28:25 smithi092 ceph-mon[37563]: 4.a deep-scrub ok 2024-03-11T21:28:25.513 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:28:25 smithi092 ceph-mon[37563]: 4.10 deep-scrub starts 2024-03-11T21:28:25.513 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:28:25 smithi092 ceph-mon[37563]: 4.10 deep-scrub ok 2024-03-11T21:28:25.528 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:28:25.528 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:28:25.528 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:28:25.576 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:28:25 smithi097 ceph-mon[33317]: 5.1 scrub starts 2024-03-11T21:28:25.577 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:28:25 smithi097 ceph-mon[33317]: 5.1 scrub ok 2024-03-11T21:28:25.577 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:28:25 smithi097 ceph-mon[33317]: 4.a deep-scrub starts 2024-03-11T21:28:25.577 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:28:25 smithi097 ceph-mon[33317]: 4.a deep-scrub ok 2024-03-11T21:28:25.577 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:28:25 smithi097 ceph-mon[33317]: 4.10 deep-scrub starts 2024-03-11T21:28:25.577 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:28:25 smithi097 ceph-mon[33317]: 4.10 deep-scrub ok 2024-03-11T21:28:25.629 INFO:tasks.radosbench.radosbench.0.smithi092.stdout: 41 16 71089 71073 108.315 74.3125 0.00574066 0.00922619 2024-03-11T21:28:25.629 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:28:25.659 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:28:25.660 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:28:25.661 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:28:25.761 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:28:25.789 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:28:25.789 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:28:25.790 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:28:25.891 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:28:25.935 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:28:25.936 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:28:25.936 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:28:26.037 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:28:26.080 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:28:26.081 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:28:26.081 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:28:26.181 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:28:26.224 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:28:26.225 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:28:26.225 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:28:26.326 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:28:26.357 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:28:26.358 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:28:26.358 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:28:26.361 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:28:26 smithi092 ceph-mon[29572]: 4.e deep-scrub starts 2024-03-11T21:28:26.361 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:28:26 smithi092 ceph-mon[29572]: 4.e deep-scrub ok 2024-03-11T21:28:26.361 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:28:26 smithi092 ceph-mon[29572]: pgmap v87: 148 pgs: 148 active+clean; 4.1 GiB data, 13 GiB used, 703 GiB / 715 GiB avail; 1.0 KiB/s rd, 78 MiB/s wr, 1.25k op/s 2024-03-11T21:28:26.361 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:28:26 smithi092 ceph-mon[29572]: 4.8 deep-scrub starts 2024-03-11T21:28:26.361 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:28:26 smithi092 ceph-mon[29572]: 4.8 deep-scrub ok 2024-03-11T21:28:26.361 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:28:26 smithi092 ceph-mon[37563]: 4.e deep-scrub starts 2024-03-11T21:28:26.361 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:28:26 smithi092 ceph-mon[37563]: 4.e deep-scrub ok 2024-03-11T21:28:26.361 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:28:26 smithi092 ceph-mon[37563]: pgmap v87: 148 pgs: 148 active+clean; 4.1 GiB data, 13 GiB used, 703 GiB / 715 GiB avail; 1.0 KiB/s rd, 78 MiB/s wr, 1.25k op/s 2024-03-11T21:28:26.361 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:28:26 smithi092 ceph-mon[37563]: 4.8 deep-scrub starts 2024-03-11T21:28:26.361 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:28:26 smithi092 ceph-mon[37563]: 4.8 deep-scrub ok 2024-03-11T21:28:26.459 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:28:26.487 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:28:26.487 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:28:26.488 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:28:26.577 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:28:26 smithi097 ceph-mon[33317]: 4.e deep-scrub starts 2024-03-11T21:28:26.577 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:28:26 smithi097 ceph-mon[33317]: 4.e deep-scrub ok 2024-03-11T21:28:26.577 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:28:26 smithi097 ceph-mon[33317]: pgmap v87: 148 pgs: 148 active+clean; 4.1 GiB data, 13 GiB used, 703 GiB / 715 GiB avail; 1.0 KiB/s rd, 78 MiB/s wr, 1.25k op/s 2024-03-11T21:28:26.577 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:28:26 smithi097 ceph-mon[33317]: 4.8 deep-scrub starts 2024-03-11T21:28:26.577 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:28:26 smithi097 ceph-mon[33317]: 4.8 deep-scrub ok 2024-03-11T21:28:26.589 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:28:26.616 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:28:26.616 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:28:26.616 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:28:26.635 INFO:tasks.radosbench.radosbench.0.smithi092.stdout: 42 16 72366 72350 107.621 79.8125 0.0152237 0.00928649 2024-03-11T21:28:26.717 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:28:26.748 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:28:26.749 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:28:26.750 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:28:26.850 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:28:26.878 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:28:26.879 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:28:26.879 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:28:26.980 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:28:27.022 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:28:27.023 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:28:27.023 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:28:27.124 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:28:27.155 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:28:27.155 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:28:27.156 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:28:27.193 DEBUG:teuthology.orchestra.run.smithi092:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid 00db0016-dfed-11ee-95c8-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-03-11T21:28:27.257 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:28:27.285 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:28:27.286 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:28:27.286 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:28:27.313 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:28:27 smithi092 ceph-mon[29572]: 6.3 deep-scrub starts 2024-03-11T21:28:27.313 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:28:27 smithi092 ceph-mon[29572]: 6.3 deep-scrub ok 2024-03-11T21:28:27.314 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:28:27 smithi092 ceph-mon[29572]: 2.17 deep-scrub starts 2024-03-11T21:28:27.314 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:28:27 smithi092 ceph-mon[29572]: 2.17 deep-scrub ok 2024-03-11T21:28:27.314 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:28:27 smithi092 ceph-mon[29572]: 4.1e scrub starts 2024-03-11T21:28:27.314 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:28:27 smithi092 ceph-mon[29572]: 4.1e scrub ok 2024-03-11T21:28:27.314 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:28:27 smithi092 ceph-mon[29572]: 2.12 scrub starts 2024-03-11T21:28:27.314 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:28:27 smithi092 ceph-mon[29572]: 2.12 scrub ok 2024-03-11T21:28:27.314 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:28:27 smithi092 ceph-mon[37563]: 6.3 deep-scrub starts 2024-03-11T21:28:27.314 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:28:27 smithi092 ceph-mon[37563]: 6.3 deep-scrub ok 2024-03-11T21:28:27.314 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:28:27 smithi092 ceph-mon[37563]: 2.17 deep-scrub starts 2024-03-11T21:28:27.314 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:28:27 smithi092 ceph-mon[37563]: 2.17 deep-scrub ok 2024-03-11T21:28:27.314 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:28:27 smithi092 ceph-mon[37563]: 4.1e scrub starts 2024-03-11T21:28:27.314 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:28:27 smithi092 ceph-mon[37563]: 4.1e scrub ok 2024-03-11T21:28:27.315 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:28:27 smithi092 ceph-mon[37563]: 2.12 scrub starts 2024-03-11T21:28:27.315 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:28:27 smithi092 ceph-mon[37563]: 2.12 scrub ok 2024-03-11T21:28:27.387 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:28:27.436 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:28:27.437 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:28:27.437 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:28:27.538 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:28:27.566 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:28:27.567 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:28:27.567 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:28:27.576 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:28:27 smithi097 ceph-mon[33317]: 6.3 deep-scrub starts 2024-03-11T21:28:27.577 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:28:27 smithi097 ceph-mon[33317]: 6.3 deep-scrub ok 2024-03-11T21:28:27.577 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:28:27 smithi097 ceph-mon[33317]: 2.17 deep-scrub starts 2024-03-11T21:28:27.577 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:28:27 smithi097 ceph-mon[33317]: 2.17 deep-scrub ok 2024-03-11T21:28:27.577 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:28:27 smithi097 ceph-mon[33317]: 4.1e scrub starts 2024-03-11T21:28:27.577 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:28:27 smithi097 ceph-mon[33317]: 4.1e scrub ok 2024-03-11T21:28:27.577 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:28:27 smithi097 ceph-mon[33317]: 2.12 scrub starts 2024-03-11T21:28:27.577 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:28:27 smithi097 ceph-mon[33317]: 2.12 scrub ok 2024-03-11T21:28:27.636 INFO:tasks.radosbench.radosbench.0.smithi092.stdout: 43 16 73538 73522 106.821 73.25 0.00943054 0.00935583 2024-03-11T21:28:27.668 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:28:27.699 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:28:27.700 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:28:27.700 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:28:27.801 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:28:27.833 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:28:27.834 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:28:27.834 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:28:27.861 DEBUG:teuthology.orchestra.run.smithi092:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid 00db0016-dfed-11ee-95c8-87774f69a715 -- ceph osd set noscrub 2024-03-11T21:28:27.921 INFO:teuthology.orchestra.run.smithi092.stdout:{ 2024-03-11T21:28:27.921 INFO:teuthology.orchestra.run.smithi092.stdout: "ops": [], 2024-03-11T21:28:27.921 INFO:teuthology.orchestra.run.smithi092.stdout: "complaint_time": 30, 2024-03-11T21:28:27.922 INFO:teuthology.orchestra.run.smithi092.stdout: "num_blocked_ops": 0 2024-03-11T21:28:27.922 INFO:teuthology.orchestra.run.smithi092.stdout:} 2024-03-11T21:28:27.935 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:28:28.038 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:28:28.039 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:28:28.040 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:28:28.140 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:28:28.149 INFO:journalctl@ceph.mgr.y.smithi092.stdout:Mar 11 21:28:27 smithi092 ceph-00db0016-dfed-11ee-95c8-87774f69a715-mgr-y[29801]: ::ffff:172.21.15.97 - - [11/Mar/2024:21:28:27] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-11T21:28:28.166 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:28:28.167 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:28:28.168 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:28:28.268 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:28:28.320 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:28:28.320 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:28:28.321 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:28:28.421 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:28:28.449 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:28:28.450 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:28:28.450 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:28:28.551 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:28:28.566 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:28:28 smithi092 ceph-mon[29572]: 3.0 scrub starts 2024-03-11T21:28:28.567 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:28:28 smithi092 ceph-mon[29572]: 3.0 scrub ok 2024-03-11T21:28:28.567 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:28:28 smithi092 ceph-mon[29572]: 5.10 scrub starts 2024-03-11T21:28:28.567 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:28:28 smithi092 ceph-mon[29572]: 5.10 scrub ok 2024-03-11T21:28:28.567 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:28:28 smithi092 ceph-mon[29572]: pgmap v88: 148 pgs: 148 active+clean; 4.2 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 1023 B/s rd, 76 MiB/s wr, 1.21k op/s 2024-03-11T21:28:28.567 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:28:28 smithi092 ceph-mon[29572]: 5.d deep-scrub starts 2024-03-11T21:28:28.567 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:28:28 smithi092 ceph-mon[29572]: 5.d deep-scrub ok 2024-03-11T21:28:28.567 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:28:28 smithi092 ceph-mon[29572]: 4.3 scrub starts 2024-03-11T21:28:28.567 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:28:28 smithi092 ceph-mon[29572]: 4.3 scrub ok 2024-03-11T21:28:28.567 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:28:28 smithi092 ceph-mon[29572]: 6.13 scrub starts 2024-03-11T21:28:28.567 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:28:28 smithi092 ceph-mon[29572]: 6.13 scrub ok 2024-03-11T21:28:28.567 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:28:28 smithi092 ceph-mon[29572]: 4.15 scrub starts 2024-03-11T21:28:28.567 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:28:28 smithi092 ceph-mon[29572]: 4.15 scrub ok 2024-03-11T21:28:28.568 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:28:28 smithi092 ceph-mon[37563]: 3.0 scrub starts 2024-03-11T21:28:28.568 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:28:28 smithi092 ceph-mon[37563]: 3.0 scrub ok 2024-03-11T21:28:28.568 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:28:28 smithi092 ceph-mon[37563]: 5.10 scrub starts 2024-03-11T21:28:28.568 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:28:28 smithi092 ceph-mon[37563]: 5.10 scrub ok 2024-03-11T21:28:28.568 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:28:28 smithi092 ceph-mon[37563]: pgmap v88: 148 pgs: 148 active+clean; 4.2 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 1023 B/s rd, 76 MiB/s wr, 1.21k op/s 2024-03-11T21:28:28.568 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:28:28 smithi092 ceph-mon[37563]: 5.d deep-scrub starts 2024-03-11T21:28:28.568 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:28:28 smithi092 ceph-mon[37563]: 5.d deep-scrub ok 2024-03-11T21:28:28.568 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:28:28 smithi092 ceph-mon[37563]: 4.3 scrub starts 2024-03-11T21:28:28.568 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:28:28 smithi092 ceph-mon[37563]: 4.3 scrub ok 2024-03-11T21:28:28.568 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:28:28 smithi092 ceph-mon[37563]: 6.13 scrub starts 2024-03-11T21:28:28.568 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:28:28 smithi092 ceph-mon[37563]: 6.13 scrub ok 2024-03-11T21:28:28.568 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:28:28 smithi092 ceph-mon[37563]: 4.15 scrub starts 2024-03-11T21:28:28.568 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:28:28 smithi092 ceph-mon[37563]: 4.15 scrub ok 2024-03-11T21:28:28.577 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:28:28 smithi097 ceph-mon[33317]: 3.0 scrub starts 2024-03-11T21:28:28.577 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:28:28 smithi097 ceph-mon[33317]: 3.0 scrub ok 2024-03-11T21:28:28.577 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:28:28 smithi097 ceph-mon[33317]: 5.10 scrub starts 2024-03-11T21:28:28.577 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:28:28 smithi097 ceph-mon[33317]: 5.10 scrub ok 2024-03-11T21:28:28.577 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:28:28 smithi097 ceph-mon[33317]: pgmap v88: 148 pgs: 148 active+clean; 4.2 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 1023 B/s rd, 76 MiB/s wr, 1.21k op/s 2024-03-11T21:28:28.577 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:28:28 smithi097 ceph-mon[33317]: 5.d deep-scrub starts 2024-03-11T21:28:28.577 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:28:28 smithi097 ceph-mon[33317]: 5.d deep-scrub ok 2024-03-11T21:28:28.577 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:28:28 smithi097 ceph-mon[33317]: 4.3 scrub starts 2024-03-11T21:28:28.577 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:28:28 smithi097 ceph-mon[33317]: 4.3 scrub ok 2024-03-11T21:28:28.577 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:28:28 smithi097 ceph-mon[33317]: 6.13 scrub starts 2024-03-11T21:28:28.577 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:28:28 smithi097 ceph-mon[33317]: 6.13 scrub ok 2024-03-11T21:28:28.577 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:28:28 smithi097 ceph-mon[33317]: 4.15 scrub starts 2024-03-11T21:28:28.578 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:28:28 smithi097 ceph-mon[33317]: 4.15 scrub ok 2024-03-11T21:28:28.578 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:28:28.579 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:28:28.579 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:28:28.636 INFO:tasks.radosbench.radosbench.0.smithi092.stdout: 44 16 74596 74580 105.896 66.125 0.0109241 0.00943828 2024-03-11T21:28:28.680 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:28:28.699 DEBUG:teuthology.orchestra.run.smithi092:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid 00db0016-dfed-11ee-95c8-87774f69a715 -- ceph daemon osd.1 dump_historic_ops 2024-03-11T21:28:28.733 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:28:28.733 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:28:28.734 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:28:28.782 INFO:teuthology.orchestra.run.smithi092.stdout: 2024-03-11T21:28:28.800 INFO:teuthology.orchestra.run.smithi092.stderr:dumped all 2024-03-11T21:28:28.834 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:28:28.860 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:28:28.861 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:28:28.861 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:28:28.962 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:28:28.991 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:28:28.992 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:28:28.992 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:28:29.093 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:28:29.141 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:28:29.142 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:28:29.143 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:28:29.243 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:28:29.286 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:28:29.287 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:28:29.287 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:28:29.388 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:28:29.434 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:28:29.435 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:28:29.435 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:28:29.451 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:28:29 smithi092 ceph-mon[29572]: 2.4 scrub starts 2024-03-11T21:28:29.451 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:28:29 smithi092 ceph-mon[29572]: 2.4 scrub ok 2024-03-11T21:28:29.451 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:28:29 smithi092 ceph-mon[29572]: 6.2 scrub starts 2024-03-11T21:28:29.451 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:28:29 smithi092 ceph-mon[29572]: 6.2 scrub ok 2024-03-11T21:28:29.451 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:28:29 smithi092 ceph-mon[29572]: 6.1f scrub starts 2024-03-11T21:28:29.452 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:28:29 smithi092 ceph-mon[29572]: 6.1f scrub ok 2024-03-11T21:28:29.452 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:28:29 smithi092 ceph-mon[29572]: from='client.14880 v1:172.21.15.92:0/1032091905' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-11T21:28:29.452 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:28:29 smithi092 ceph-mon[37563]: 2.4 scrub starts 2024-03-11T21:28:29.452 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:28:29 smithi092 ceph-mon[37563]: 2.4 scrub ok 2024-03-11T21:28:29.452 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:28:29 smithi092 ceph-mon[37563]: 6.2 scrub starts 2024-03-11T21:28:29.452 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:28:29 smithi092 ceph-mon[37563]: 6.2 scrub ok 2024-03-11T21:28:29.452 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:28:29 smithi092 ceph-mon[37563]: 6.1f scrub starts 2024-03-11T21:28:29.452 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:28:29 smithi092 ceph-mon[37563]: 6.1f scrub ok 2024-03-11T21:28:29.452 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:28:29 smithi092 ceph-mon[37563]: from='client.14880 v1:172.21.15.92:0/1032091905' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-11T21:28:29.536 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:28:29.576 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:28:29 smithi097 ceph-mon[33317]: 2.4 scrub starts 2024-03-11T21:28:29.576 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:28:29 smithi097 ceph-mon[33317]: 2.4 scrub ok 2024-03-11T21:28:29.577 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:28:29 smithi097 ceph-mon[33317]: 6.2 scrub starts 2024-03-11T21:28:29.577 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:28:29 smithi097 ceph-mon[33317]: 6.2 scrub ok 2024-03-11T21:28:29.577 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:28:29 smithi097 ceph-mon[33317]: 6.1f scrub starts 2024-03-11T21:28:29.577 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:28:29 smithi097 ceph-mon[33317]: 6.1f scrub ok 2024-03-11T21:28:29.577 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:28:29 smithi097 ceph-mon[33317]: from='client.14880 v1:172.21.15.92:0/1032091905' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-11T21:28:29.578 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:28:29.578 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:28:29.579 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:28:29.636 INFO:tasks.radosbench.radosbench.0.smithi092.stdout: 45 16 75685 75669 105.055 68.0625 0.0130314 0.00951377 2024-03-11T21:28:29.679 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:28:29.718 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:28:29.719 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:28:29.719 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:28:29.789 INFO:teuthology.orchestra.run.smithi092.stdout:{"pg_ready":true,"pg_map":{"version":88,"stamp":"2024-03-11T21:28:26.994826+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":4555282856,"num_objects":69697,"num_object_clones":0,"num_object_copies":209091,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":69697,"num_whiteouts":0,"num_read":1097,"num_read_kb":943,"num_write":70114,"num_write_kb":4450397,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":21,"num_bytes_recovered":1771324,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":70150,"ondisk_log_size":70150,"up":444,"acting":444,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":444,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":749961216,"kb_used":13812088,"kb_used_data":13344760,"kb_used_omap":12,"kb_used_meta":467251,"kb_avail":736149128,"statfs":{"total":767960285184,"available":753816707072,"internally_reserved":0,"allocated":13665034240,"data_stored":13661801505,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12752,"internal_metadata":478465584},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,2,6,7,5,5,2],"upper_bound":128},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":794230784,"num_objects":12119,"num_object_clones":0,"num_object_copies":36357,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":12119,"num_whiteouts":0,"num_read":10,"num_read_kb":10,"num_write":12119,"num_write_kb":775616,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"10.008244"},"pg_stats":[{"pgid":"6.1b","version":"0'0","reported_seq":97,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-03-11T21:28:16.650491+0000","last_change":"2024-03-11T21:28:16.650491+0000","last_active":"2024-03-11T21:28:16.650491+0000","last_peered":"2024-03-11T21:28:16.650491+0000","last_clean":"2024-03-11T21:28:16.650491+0000","last_became_active":"2024-03-11T21:25:45.302931+0000","last_became_peered":"2024-03-11T21:25:45.302931+0000","last_unstale":"2024-03-11T21:28:16.650491+0000","last_undegraded":"2024-03-11T21:28:16.650491+0000","last_fullsized":"2024-03-11T21:28:16.650491+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:28:16.650384+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:43.559452+0000","last_clean_scrub_stamp":"2024-03-11T21:28:16.650384+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:29:44.614978+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":96,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-03-11T21:28:23.144848+0000","last_change":"2024-03-11T21:28:16.734432+0000","last_active":"2024-03-11T21:28:23.144848+0000","last_peered":"2024-03-11T21:28:23.144848+0000","last_clean":"2024-03-11T21:28:23.144848+0000","last_became_active":"2024-03-11T21:25:42.578893+0000","last_became_peered":"2024-03-11T21:25:42.578893+0000","last_unstale":"2024-03-11T21:28:23.144848+0000","last_undegraded":"2024-03-11T21:28:23.144848+0000","last_fullsized":"2024-03-11T21:28:23.144848+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:28:16.734315+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:41.544550+0000","last_clean_scrub_stamp":"2024-03-11T21:28:16.734315+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:29:17.932001+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1f","version":"0'0","reported_seq":110,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-03-11T21:28:15.081211+0000","last_change":"2024-03-11T21:28:14.483927+0000","last_active":"2024-03-11T21:28:15.081211+0000","last_peered":"2024-03-11T21:28:15.081211+0000","last_clean":"2024-03-11T21:28:15.081211+0000","last_became_active":"2024-03-11T21:25:38.540510+0000","last_became_peered":"2024-03-11T21:25:38.540510+0000","last_unstale":"2024-03-11T21:28:15.081211+0000","last_undegraded":"2024-03-11T21:28:15.081211+0000","last_fullsized":"2024-03-11T21:28:15.081211+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:28:14.483813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:37.512018+0000","last_clean_scrub_stamp":"2024-03-11T21:28:14.483813+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:29:26.482626+0000","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,4],"acting":[0,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.19","version":"74'15","reported_seq":122,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-03-11T21:28:15.076897+0000","last_change":"2024-03-11T21:27:12.751287+0000","last_active":"2024-03-11T21:28:15.076897+0000","last_peered":"2024-03-11T21:28:15.076897+0000","last_clean":"2024-03-11T21:28:15.076897+0000","last_became_active":"2024-03-11T21:25:40.554207+0000","last_became_peered":"2024-03-11T21:25:40.554207+0000","last_unstale":"2024-03-11T21:28:15.076897+0000","last_undegraded":"2024-03-11T21:28:15.076897+0000","last_fullsized":"2024-03-11T21:28:15.076897+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'15","last_scrub_stamp":"2024-03-11T21:27:12.751184+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:39.525554+0000","last_clean_scrub_stamp":"2024-03-11T21:27:12.751184+0000","objects_scrubbed":7,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:28:24.326457+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.1a","version":"0'0","reported_seq":92,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-03-11T21:28:23.143715+0000","last_change":"2024-03-11T21:28:10.634383+0000","last_active":"2024-03-11T21:28:23.143715+0000","last_peered":"2024-03-11T21:28:23.143715+0000","last_clean":"2024-03-11T21:28:23.143715+0000","last_became_active":"2024-03-11T21:25:44.590923+0000","last_became_peered":"2024-03-11T21:25:44.590923+0000","last_unstale":"2024-03-11T21:28:23.143715+0000","last_undegraded":"2024-03-11T21:28:23.143715+0000","last_fullsized":"2024-03-11T21:28:23.143715+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:28:10.634303+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:43.559452+0000","last_clean_scrub_stamp":"2024-03-11T21:28:10.634303+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:29:25.662002+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,1],"acting":[4,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":96,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-03-11T21:28:23.155289+0000","last_change":"2024-03-11T21:27:17.264407+0000","last_active":"2024-03-11T21:28:23.155289+0000","last_peered":"2024-03-11T21:28:23.155289+0000","last_clean":"2024-03-11T21:28:23.155289+0000","last_became_active":"2024-03-11T21:25:42.578424+0000","last_became_peered":"2024-03-11T21:25:42.578424+0000","last_unstale":"2024-03-11T21:28:23.155289+0000","last_undegraded":"2024-03-11T21:28:23.155289+0000","last_fullsized":"2024-03-11T21:28:23.155289+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:27:17.264317+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:41.544550+0000","last_clean_scrub_stamp":"2024-03-11T21:27:17.264317+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:28:32.538444+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,7],"acting":[1,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1e","version":"0'0","reported_seq":109,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-03-11T21:28:15.077351+0000","last_change":"2024-03-11T21:28:07.604544+0000","last_active":"2024-03-11T21:28:15.077351+0000","last_peered":"2024-03-11T21:28:15.077351+0000","last_clean":"2024-03-11T21:28:15.077351+0000","last_became_active":"2024-03-11T21:25:38.537283+0000","last_became_peered":"2024-03-11T21:25:38.537283+0000","last_unstale":"2024-03-11T21:28:15.077351+0000","last_undegraded":"2024-03-11T21:28:15.077351+0000","last_fullsized":"2024-03-11T21:28:15.077351+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:28:07.604432+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:37.512018+0000","last_clean_scrub_stamp":"2024-03-11T21:28:07.604432+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:29:20.709213+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.18","version":"74'9","reported_seq":124,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-03-11T21:28:23.145040+0000","last_change":"2024-03-11T21:28:22.778031+0000","last_active":"2024-03-11T21:28:23.145040+0000","last_peered":"2024-03-11T21:28:23.145040+0000","last_clean":"2024-03-11T21:28:23.145040+0000","last_became_active":"2024-03-11T21:25:40.552216+0000","last_became_peered":"2024-03-11T21:25:40.552216+0000","last_unstale":"2024-03-11T21:28:23.145040+0000","last_undegraded":"2024-03-11T21:28:23.145040+0000","last_fullsized":"2024-03-11T21:28:23.145040+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'9","last_scrub_stamp":"2024-03-11T21:28:22.777781+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:39.525554+0000","last_clean_scrub_stamp":"2024-03-11T21:28:22.777781+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:29:29.319933+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":91,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-03-11T21:28:23.151542+0000","last_change":"2024-03-11T21:27:29.953758+0000","last_active":"2024-03-11T21:28:23.151542+0000","last_peered":"2024-03-11T21:28:23.151542+0000","last_clean":"2024-03-11T21:28:23.151542+0000","last_became_active":"2024-03-11T21:25:44.590234+0000","last_became_peered":"2024-03-11T21:25:44.590234+0000","last_unstale":"2024-03-11T21:28:23.151542+0000","last_undegraded":"2024-03-11T21:28:23.151542+0000","last_fullsized":"2024-03-11T21:28:23.151542+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:27:29.953654+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:43.559452+0000","last_clean_scrub_stamp":"2024-03-11T21:27:29.953654+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:28:30.681967+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3],"acting":[5,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":95,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-03-11T21:28:23.123781+0000","last_change":"2024-03-11T21:27:18.434331+0000","last_active":"2024-03-11T21:28:23.123781+0000","last_peered":"2024-03-11T21:28:23.123781+0000","last_clean":"2024-03-11T21:28:23.123781+0000","last_became_active":"2024-03-11T21:25:42.580533+0000","last_became_peered":"2024-03-11T21:25:42.580533+0000","last_unstale":"2024-03-11T21:28:23.123781+0000","last_undegraded":"2024-03-11T21:28:23.123781+0000","last_fullsized":"2024-03-11T21:28:23.123781+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:27:18.434247+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:41.544550+0000","last_clean_scrub_stamp":"2024-03-11T21:27:18.434247+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:28:39.839411+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.1d","version":"0'0","reported_seq":111,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-03-11T21:28:23.124152+0000","last_change":"2024-03-11T21:28:11.681226+0000","last_active":"2024-03-11T21:28:23.124152+0000","last_peered":"2024-03-11T21:28:23.124152+0000","last_clean":"2024-03-11T21:28:23.124152+0000","last_became_active":"2024-03-11T21:25:38.542628+0000","last_became_peered":"2024-03-11T21:25:38.542628+0000","last_unstale":"2024-03-11T21:28:23.124152+0000","last_undegraded":"2024-03-11T21:28:23.124152+0000","last_fullsized":"2024-03-11T21:28:23.124152+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:28:11.681096+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:37.512018+0000","last_clean_scrub_stamp":"2024-03-11T21:28:11.681096+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:29:22.527860+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,0],"acting":[7,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.1b","version":"74'5","reported_seq":110,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-03-11T21:28:23.144262+0000","last_change":"2024-03-11T21:28:14.743893+0000","last_active":"2024-03-11T21:28:23.144262+0000","last_peered":"2024-03-11T21:28:23.144262+0000","last_clean":"2024-03-11T21:28:23.144262+0000","last_became_active":"2024-03-11T21:25:40.555826+0000","last_became_peered":"2024-03-11T21:25:40.555826+0000","last_unstale":"2024-03-11T21:28:23.144262+0000","last_undegraded":"2024-03-11T21:28:23.144262+0000","last_fullsized":"2024-03-11T21:28:23.144262+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'5","last_scrub_stamp":"2024-03-11T21:28:14.743756+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:39.525554+0000","last_clean_scrub_stamp":"2024-03-11T21:28:14.743756+0000","objects_scrubbed":2,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:29:25.133975+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":11,"num_read_kb":7,"num_write":6,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,1],"acting":[4,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":90,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-03-11T21:28:15.079308+0000","last_change":"2024-03-11T21:27:29.476225+0000","last_active":"2024-03-11T21:28:15.079308+0000","last_peered":"2024-03-11T21:28:15.079308+0000","last_clean":"2024-03-11T21:28:15.079308+0000","last_became_active":"2024-03-11T21:25:44.589786+0000","last_became_peered":"2024-03-11T21:25:44.589786+0000","last_unstale":"2024-03-11T21:28:15.079308+0000","last_undegraded":"2024-03-11T21:28:15.079308+0000","last_fullsized":"2024-03-11T21:28:15.079308+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:27:29.476144+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:43.559452+0000","last_clean_scrub_stamp":"2024-03-11T21:27:29.476144+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:28:53.520269+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,7],"acting":[0,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":95,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-03-11T21:28:23.151777+0000","last_change":"2024-03-11T21:27:18.957913+0000","last_active":"2024-03-11T21:28:23.151777+0000","last_peered":"2024-03-11T21:28:23.151777+0000","last_clean":"2024-03-11T21:28:23.151777+0000","last_became_active":"2024-03-11T21:25:42.573831+0000","last_became_peered":"2024-03-11T21:25:42.573831+0000","last_unstale":"2024-03-11T21:28:23.151777+0000","last_undegraded":"2024-03-11T21:28:23.151777+0000","last_fullsized":"2024-03-11T21:28:23.151777+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:27:18.957808+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:41.544550+0000","last_clean_scrub_stamp":"2024-03-11T21:27:18.957808+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:28:37.225502+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,7],"acting":[5,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1c","version":"0'0","reported_seq":103,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-03-11T21:28:23.124025+0000","last_change":"2024-03-11T21:27:04.375295+0000","last_active":"2024-03-11T21:28:23.124025+0000","last_peered":"2024-03-11T21:28:23.124025+0000","last_clean":"2024-03-11T21:28:23.124025+0000","last_became_active":"2024-03-11T21:25:38.541364+0000","last_became_peered":"2024-03-11T21:25:38.541364+0000","last_unstale":"2024-03-11T21:28:23.124025+0000","last_undegraded":"2024-03-11T21:28:23.124025+0000","last_fullsized":"2024-03-11T21:28:23.124025+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:27:04.375194+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:37.512018+0000","last_clean_scrub_stamp":"2024-03-11T21:27:04.375194+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.1a","version":"74'9","reported_seq":124,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-03-11T21:28:23.143782+0000","last_change":"2024-03-11T21:28:18.725957+0000","last_active":"2024-03-11T21:28:23.143782+0000","last_peered":"2024-03-11T21:28:23.143782+0000","last_clean":"2024-03-11T21:28:23.143782+0000","last_became_active":"2024-03-11T21:25:40.556629+0000","last_became_peered":"2024-03-11T21:25:40.556629+0000","last_unstale":"2024-03-11T21:28:23.143782+0000","last_undegraded":"2024-03-11T21:28:23.143782+0000","last_fullsized":"2024-03-11T21:28:23.143782+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'9","last_scrub_stamp":"2024-03-11T21:28:18.725837+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:39.525554+0000","last_clean_scrub_stamp":"2024-03-11T21:28:18.725837+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:29:27.821802+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0],"acting":[4,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1f","version":"0'0","reported_seq":89,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-03-11T21:28:15.083699+0000","last_change":"2024-03-11T21:27:04.620314+0000","last_active":"2024-03-11T21:28:15.083699+0000","last_peered":"2024-03-11T21:28:15.083699+0000","last_clean":"2024-03-11T21:28:15.083699+0000","last_became_active":"2024-03-11T21:25:45.302637+0000","last_became_peered":"2024-03-11T21:25:45.302637+0000","last_unstale":"2024-03-11T21:28:15.083699+0000","last_undegraded":"2024-03-11T21:28:15.083699+0000","last_fullsized":"2024-03-11T21:28:15.083699+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:27:04.620217+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:43.559452+0000","last_clean_scrub_stamp":"2024-03-11T21:27:04.620217+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1c","version":"0'0","reported_seq":96,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-03-11T21:28:23.144458+0000","last_change":"2024-03-11T21:27:10.787962+0000","last_active":"2024-03-11T21:28:23.144458+0000","last_peered":"2024-03-11T21:28:23.144458+0000","last_clean":"2024-03-11T21:28:23.144458+0000","last_became_active":"2024-03-11T21:25:42.576064+0000","last_became_peered":"2024-03-11T21:25:42.576064+0000","last_unstale":"2024-03-11T21:28:23.144458+0000","last_undegraded":"2024-03-11T21:28:23.144458+0000","last_fullsized":"2024-03-11T21:28:23.144458+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:27:10.787826+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:41.544550+0000","last_clean_scrub_stamp":"2024-03-11T21:27:10.787826+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:28:32.419330+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1b","version":"0'0","reported_seq":101,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-03-11T21:28:15.080625+0000","last_change":"2024-03-11T21:26:55.579982+0000","last_active":"2024-03-11T21:28:15.080625+0000","last_peered":"2024-03-11T21:28:15.080625+0000","last_clean":"2024-03-11T21:28:15.080625+0000","last_became_active":"2024-03-11T21:25:38.537520+0000","last_became_peered":"2024-03-11T21:25:38.537520+0000","last_unstale":"2024-03-11T21:28:15.080625+0000","last_undegraded":"2024-03-11T21:28:15.080625+0000","last_fullsized":"2024-03-11T21:28:15.080625+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:26:55.579901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:37.512018+0000","last_clean_scrub_stamp":"2024-03-11T21:26:55.579901+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:28:17.884453+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1d","version":"74'12","reported_seq":120,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-03-11T21:28:15.077079+0000","last_change":"2024-03-11T21:27:01.629574+0000","last_active":"2024-03-11T21:28:15.077079+0000","last_peered":"2024-03-11T21:28:15.077079+0000","last_clean":"2024-03-11T21:28:15.077079+0000","last_became_active":"2024-03-11T21:25:40.556330+0000","last_became_peered":"2024-03-11T21:25:40.556330+0000","last_unstale":"2024-03-11T21:28:15.077079+0000","last_undegraded":"2024-03-11T21:28:15.077079+0000","last_fullsized":"2024-03-11T21:28:15.077079+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'12","last_scrub_stamp":"2024-03-11T21:27:01.629471+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:39.525554+0000","last_clean_scrub_stamp":"2024-03-11T21:27:01.629471+0000","objects_scrubbed":5,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:28:26.432819+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":25,"num_read_kb":16,"num_write":14,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.1e","version":"0'0","reported_seq":92,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-03-11T21:28:23.144789+0000","last_change":"2024-03-11T21:28:11.684315+0000","last_active":"2024-03-11T21:28:23.144789+0000","last_peered":"2024-03-11T21:28:23.144789+0000","last_clean":"2024-03-11T21:28:23.144789+0000","last_became_active":"2024-03-11T21:25:45.300355+0000","last_became_peered":"2024-03-11T21:25:45.300355+0000","last_unstale":"2024-03-11T21:28:23.144789+0000","last_undegraded":"2024-03-11T21:28:23.144789+0000","last_fullsized":"2024-03-11T21:28:23.144789+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:28:11.684174+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:43.559452+0000","last_clean_scrub_stamp":"2024-03-11T21:28:11.684174+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:29:32.796638+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5],"acting":[4,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1d","version":"0'0","reported_seq":96,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-03-11T21:28:23.156725+0000","last_change":"2024-03-11T21:27:09.199590+0000","last_active":"2024-03-11T21:28:23.156725+0000","last_peered":"2024-03-11T21:28:23.156725+0000","last_clean":"2024-03-11T21:28:23.156725+0000","last_became_active":"2024-03-11T21:25:42.579407+0000","last_became_peered":"2024-03-11T21:25:42.579407+0000","last_unstale":"2024-03-11T21:28:23.156725+0000","last_undegraded":"2024-03-11T21:28:23.156725+0000","last_fullsized":"2024-03-11T21:28:23.156725+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:27:09.199513+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:27:09.199513+0000","last_clean_scrub_stamp":"2024-03-11T21:27:09.199513+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:28:28.109684+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1a","version":"0'0","reported_seq":112,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-03-11T21:28:23.128190+0000","last_change":"2024-03-11T21:28:16.994328+0000","last_active":"2024-03-11T21:28:23.128190+0000","last_peered":"2024-03-11T21:28:23.128190+0000","last_clean":"2024-03-11T21:28:23.128190+0000","last_became_active":"2024-03-11T21:25:38.539725+0000","last_became_peered":"2024-03-11T21:25:38.539725+0000","last_unstale":"2024-03-11T21:28:23.128190+0000","last_undegraded":"2024-03-11T21:28:23.128190+0000","last_fullsized":"2024-03-11T21:28:23.128190+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:28:16.994251+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:37.512018+0000","last_clean_scrub_stamp":"2024-03-11T21:28:16.994251+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:29:45.088652+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.1c","version":"74'15","reported_seq":122,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-03-11T21:28:15.072104+0000","last_change":"2024-03-11T21:27:04.045043+0000","last_active":"2024-03-11T21:28:15.072104+0000","last_peered":"2024-03-11T21:28:15.072104+0000","last_clean":"2024-03-11T21:28:15.072104+0000","last_became_active":"2024-03-11T21:25:40.551453+0000","last_became_peered":"2024-03-11T21:25:40.551453+0000","last_unstale":"2024-03-11T21:28:15.072104+0000","last_undegraded":"2024-03-11T21:28:15.072104+0000","last_fullsized":"2024-03-11T21:28:15.072104+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'15","last_scrub_stamp":"2024-03-11T21:27:04.044946+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:39.525554+0000","last_clean_scrub_stamp":"2024-03-11T21:27:04.044946+0000","objects_scrubbed":7,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:28:23.430891+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.1d","version":"0'0","reported_seq":92,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-03-11T21:28:23.156352+0000","last_change":"2024-03-11T21:27:20.249317+0000","last_active":"2024-03-11T21:28:23.156352+0000","last_peered":"2024-03-11T21:28:23.156352+0000","last_clean":"2024-03-11T21:28:23.156352+0000","last_became_active":"2024-03-11T21:25:44.588845+0000","last_became_peered":"2024-03-11T21:25:44.588845+0000","last_unstale":"2024-03-11T21:28:23.156352+0000","last_undegraded":"2024-03-11T21:28:23.156352+0000","last_fullsized":"2024-03-11T21:28:23.156352+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:27:20.249231+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:43.559452+0000","last_clean_scrub_stamp":"2024-03-11T21:27:20.249231+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:28:48.831204+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.1e","version":"0'0","reported_seq":102,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-03-11T21:28:17.464830+0000","last_change":"2024-03-11T21:28:17.464869+0000","last_active":"2024-03-11T21:28:17.464830+0000","last_peered":"2024-03-11T21:28:17.464830+0000","last_clean":"2024-03-11T21:28:17.464830+0000","last_became_active":"2024-03-11T21:25:42.582959+0000","last_became_peered":"2024-03-11T21:25:42.582959+0000","last_unstale":"2024-03-11T21:28:17.464830+0000","last_undegraded":"2024-03-11T21:28:17.464830+0000","last_fullsized":"2024-03-11T21:28:17.464830+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:28:17.464401+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:26:58.335431+0000","last_clean_scrub_stamp":"2024-03-11T21:28:17.464401+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:29:40.047363+0000","scrub_duration":20,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.19","version":"67'2","reported_seq":108,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-03-11T21:28:15.079751+0000","last_change":"2024-03-11T21:27:27.823258+0000","last_active":"2024-03-11T21:28:15.079751+0000","last_peered":"2024-03-11T21:28:15.079751+0000","last_clean":"2024-03-11T21:28:15.079751+0000","last_became_active":"2024-03-11T21:25:38.539243+0000","last_became_peered":"2024-03-11T21:25:38.539243+0000","last_unstale":"2024-03-11T21:28:15.079751+0000","last_undegraded":"2024-03-11T21:28:15.079751+0000","last_fullsized":"2024-03-11T21:28:15.079751+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'2","last_scrub_stamp":"2024-03-11T21:27:27.823167+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:37.512018+0000","last_clean_scrub_stamp":"2024-03-11T21:27:27.823167+0000","objects_scrubbed":2,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:28:51.277114+0000","scrub_duration":19,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":957,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":7,"num_read_kb":7,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,0],"acting":[3,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1f","version":"74'11","reported_seq":127,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-03-11T21:28:23.128667+0000","last_change":"2024-03-11T21:28:18.041390+0000","last_active":"2024-03-11T21:28:23.128667+0000","last_peered":"2024-03-11T21:28:23.128667+0000","last_clean":"2024-03-11T21:28:23.128667+0000","last_became_active":"2024-03-11T21:25:40.554686+0000","last_became_peered":"2024-03-11T21:25:40.554686+0000","last_unstale":"2024-03-11T21:28:23.128667+0000","last_undegraded":"2024-03-11T21:28:23.128667+0000","last_fullsized":"2024-03-11T21:28:23.128667+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'11","last_scrub_stamp":"2024-03-11T21:28:18.041260+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:39.525554+0000","last_clean_scrub_stamp":"2024-03-11T21:28:18.041260+0000","objects_scrubbed":5,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:29:23.336510+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,1],"acting":[6,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.1c","version":"74'1","reported_seq":92,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-03-11T21:28:23.124507+0000","last_change":"2024-03-11T21:27:22.466670+0000","last_active":"2024-03-11T21:28:23.124507+0000","last_peered":"2024-03-11T21:28:23.124507+0000","last_clean":"2024-03-11T21:28:23.124507+0000","last_became_active":"2024-03-11T21:25:44.593326+0000","last_became_peered":"2024-03-11T21:25:44.593326+0000","last_unstale":"2024-03-11T21:28:23.124507+0000","last_undegraded":"2024-03-11T21:28:23.124507+0000","last_fullsized":"2024-03-11T21:28:23.124507+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'1","last_scrub_stamp":"2024-03-11T21:27:22.466585+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:43.559452+0000","last_clean_scrub_stamp":"2024-03-11T21:27:22.466585+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:28:38.188343+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":370,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.1f","version":"74'8","reported_seq":149,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-03-11T21:28:23.128686+0000","last_change":"2024-03-11T21:28:06.034479+0000","last_active":"2024-03-11T21:28:23.128686+0000","last_peered":"2024-03-11T21:28:23.128686+0000","last_clean":"2024-03-11T21:28:23.128686+0000","last_became_active":"2024-03-11T21:25:42.577401+0000","last_became_peered":"2024-03-11T21:25:42.577401+0000","last_unstale":"2024-03-11T21:28:23.128686+0000","last_undegraded":"2024-03-11T21:28:23.128686+0000","last_fullsized":"2024-03-11T21:28:23.128686+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'8","last_scrub_stamp":"2024-03-11T21:28:06.034236+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:41.544550+0000","last_clean_scrub_stamp":"2024-03-11T21:28:06.034236+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:29:10.600889+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.18","version":"67'1","reported_seq":117,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-03-11T21:28:23.151549+0000","last_change":"2024-03-11T21:28:16.829077+0000","last_active":"2024-03-11T21:28:23.151549+0000","last_peered":"2024-03-11T21:28:23.151549+0000","last_clean":"2024-03-11T21:28:23.151549+0000","last_became_active":"2024-03-11T21:25:38.538321+0000","last_became_peered":"2024-03-11T21:25:38.538321+0000","last_unstale":"2024-03-11T21:28:23.151549+0000","last_undegraded":"2024-03-11T21:28:23.151549+0000","last_fullsized":"2024-03-11T21:28:23.151549+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'1","last_scrub_stamp":"2024-03-11T21:28:16.828998+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:37.512018+0000","last_clean_scrub_stamp":"2024-03-11T21:28:16.828998+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:29:33.110060+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":436,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":7,"num_read_kb":7,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,7],"acting":[5,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1e","version":"74'10","reported_seq":113,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-03-11T21:28:15.080503+0000","last_change":"2024-03-11T21:26:42.521121+0000","last_active":"2024-03-11T21:28:15.080503+0000","last_peered":"2024-03-11T21:28:15.080503+0000","last_clean":"2024-03-11T21:28:15.080503+0000","last_became_active":"2024-03-11T21:25:40.763438+0000","last_became_peered":"2024-03-11T21:25:40.763438+0000","last_unstale":"2024-03-11T21:28:15.080503+0000","last_undegraded":"2024-03-11T21:28:15.080503+0000","last_fullsized":"2024-03-11T21:28:15.080503+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'10","last_scrub_stamp":"2024-03-11T21:26:42.521016+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:39.525554+0000","last_clean_scrub_stamp":"2024-03-11T21:26:42.521016+0000","objects_scrubbed":5,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.13","version":"0'0","reported_seq":89,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-03-11T21:28:15.080956+0000","last_change":"2024-03-11T21:27:03.623032+0000","last_active":"2024-03-11T21:28:15.080956+0000","last_peered":"2024-03-11T21:28:15.080956+0000","last_clean":"2024-03-11T21:28:15.080956+0000","last_became_active":"2024-03-11T21:25:45.302102+0000","last_became_peered":"2024-03-11T21:25:45.302102+0000","last_unstale":"2024-03-11T21:28:15.080956+0000","last_undegraded":"2024-03-11T21:28:15.080956+0000","last_fullsized":"2024-03-11T21:28:15.080956+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:27:03.622909+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:43.559452+0000","last_clean_scrub_stamp":"2024-03-11T21:27:03.622909+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":103,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-03-11T21:28:26.537945+0000","last_change":"2024-03-11T21:28:26.537966+0000","last_active":"2024-03-11T21:28:26.537945+0000","last_peered":"2024-03-11T21:28:26.537945+0000","last_clean":"2024-03-11T21:28:26.537945+0000","last_became_active":"2024-03-11T21:25:42.579781+0000","last_became_peered":"2024-03-11T21:25:42.579781+0000","last_unstale":"2024-03-11T21:28:26.537945+0000","last_undegraded":"2024-03-11T21:28:26.537945+0000","last_fullsized":"2024-03-11T21:28:26.537945+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:28:26.537878+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:41.544550+0000","last_clean_scrub_stamp":"2024-03-11T21:28:26.537878+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:29:43.076912+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.17","version":"0'0","reported_seq":112,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-03-11T21:28:25.918190+0000","last_change":"2024-03-11T21:28:25.918222+0000","last_active":"2024-03-11T21:28:25.918190+0000","last_peered":"2024-03-11T21:28:25.918190+0000","last_clean":"2024-03-11T21:28:25.918190+0000","last_became_active":"2024-03-11T21:25:38.540349+0000","last_became_peered":"2024-03-11T21:25:38.540349+0000","last_unstale":"2024-03-11T21:28:25.918190+0000","last_undegraded":"2024-03-11T21:28:25.918190+0000","last_fullsized":"2024-03-11T21:28:25.918190+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:28:25.918101+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:28:25.918101+0000","last_clean_scrub_stamp":"2024-03-11T21:28:25.918101+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:29:47.849509+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,2],"acting":[6,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.11","version":"74'11","reported_seq":116,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-03-11T21:28:15.080014+0000","last_change":"2024-03-11T21:27:22.793832+0000","last_active":"2024-03-11T21:28:15.080014+0000","last_peered":"2024-03-11T21:28:15.080014+0000","last_clean":"2024-03-11T21:28:15.080014+0000","last_became_active":"2024-03-11T21:25:40.760664+0000","last_became_peered":"2024-03-11T21:25:40.760664+0000","last_unstale":"2024-03-11T21:28:15.080014+0000","last_undegraded":"2024-03-11T21:28:15.080014+0000","last_fullsized":"2024-03-11T21:28:15.080014+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'11","last_scrub_stamp":"2024-03-11T21:27:22.793725+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:39.525554+0000","last_clean_scrub_stamp":"2024-03-11T21:27:22.793725+0000","objects_scrubbed":5,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:28:28.896623+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":99,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-03-11T21:28:23.123490+0000","last_change":"2024-03-11T21:28:19.525623+0000","last_active":"2024-03-11T21:28:23.123490+0000","last_peered":"2024-03-11T21:28:23.123490+0000","last_clean":"2024-03-11T21:28:23.123490+0000","last_became_active":"2024-03-11T21:25:44.591068+0000","last_became_peered":"2024-03-11T21:25:44.591068+0000","last_unstale":"2024-03-11T21:28:23.123490+0000","last_undegraded":"2024-03-11T21:28:23.123490+0000","last_fullsized":"2024-03-11T21:28:23.123490+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:28:19.525508+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:43.559452+0000","last_clean_scrub_stamp":"2024-03-11T21:28:19.525508+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:29:27.438631+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,4],"acting":[7,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":96,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-03-11T21:28:23.128184+0000","last_change":"2024-03-11T21:27:09.976879+0000","last_active":"2024-03-11T21:28:23.128184+0000","last_peered":"2024-03-11T21:28:23.128184+0000","last_clean":"2024-03-11T21:28:23.128184+0000","last_became_active":"2024-03-11T21:25:42.584148+0000","last_became_peered":"2024-03-11T21:25:42.584148+0000","last_unstale":"2024-03-11T21:28:23.128184+0000","last_undegraded":"2024-03-11T21:28:23.128184+0000","last_fullsized":"2024-03-11T21:28:23.128184+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:27:09.976747+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:27:09.976747+0000","last_clean_scrub_stamp":"2024-03-11T21:27:09.976747+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:28:32.267721+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.16","version":"0'0","reported_seq":103,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-03-11T21:28:23.153217+0000","last_change":"2024-03-11T21:27:11.939012+0000","last_active":"2024-03-11T21:28:23.153217+0000","last_peered":"2024-03-11T21:28:23.153217+0000","last_clean":"2024-03-11T21:28:23.153217+0000","last_became_active":"2024-03-11T21:25:38.539103+0000","last_became_peered":"2024-03-11T21:25:38.539103+0000","last_unstale":"2024-03-11T21:28:23.153217+0000","last_undegraded":"2024-03-11T21:28:23.153217+0000","last_fullsized":"2024-03-11T21:28:23.153217+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:27:11.938909+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:27:11.938909+0000","last_clean_scrub_stamp":"2024-03-11T21:27:11.938909+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:28:32.608220+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,2],"acting":[5,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.10","version":"94'6","reported_seq":105,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-03-11T21:28:15.076783+0000","last_change":"2024-03-11T21:26:58.601920+0000","last_active":"2024-03-11T21:28:15.076783+0000","last_peered":"2024-03-11T21:28:15.076783+0000","last_clean":"2024-03-11T21:28:15.076783+0000","last_became_active":"2024-03-11T21:25:40.552720+0000","last_became_peered":"2024-03-11T21:25:40.552720+0000","last_unstale":"2024-03-11T21:28:15.076783+0000","last_undegraded":"2024-03-11T21:28:15.076783+0000","last_fullsized":"2024-03-11T21:28:15.076783+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"94'6","last_scrub_stamp":"2024-03-11T21:26:58.601831+0000","last_deep_scrub":"94'6","last_deep_scrub_stamp":"2024-03-11T21:26:58.601831+0000","last_clean_scrub_stamp":"2024-03-11T21:26:58.601831+0000","objects_scrubbed":2,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":6,"num_read_kb":4,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6],"acting":[3,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":89,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-03-11T21:28:15.076783+0000","last_change":"2024-03-11T21:27:05.669068+0000","last_active":"2024-03-11T21:28:15.076783+0000","last_peered":"2024-03-11T21:28:15.076783+0000","last_clean":"2024-03-11T21:28:15.076783+0000","last_became_active":"2024-03-11T21:25:44.583181+0000","last_became_peered":"2024-03-11T21:25:44.583181+0000","last_unstale":"2024-03-11T21:28:15.076783+0000","last_undegraded":"2024-03-11T21:28:15.076783+0000","last_fullsized":"2024-03-11T21:28:15.076783+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:27:05.668971+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:43.559452+0000","last_clean_scrub_stamp":"2024-03-11T21:27:05.668971+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:28:34.193069+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":96,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-03-11T21:28:23.157419+0000","last_change":"2024-03-11T21:27:21.207387+0000","last_active":"2024-03-11T21:28:23.157419+0000","last_peered":"2024-03-11T21:28:23.157419+0000","last_clean":"2024-03-11T21:28:23.157419+0000","last_became_active":"2024-03-11T21:25:42.573158+0000","last_became_peered":"2024-03-11T21:25:42.573158+0000","last_unstale":"2024-03-11T21:28:23.157419+0000","last_undegraded":"2024-03-11T21:28:23.157419+0000","last_fullsized":"2024-03-11T21:28:23.157419+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:27:21.207303+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:27:21.207303+0000","last_clean_scrub_stamp":"2024-03-11T21:27:21.207303+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:28:31.619875+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.15","version":"0'0","reported_seq":112,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-03-11T21:28:23.156564+0000","last_change":"2024-03-11T21:28:17.525764+0000","last_active":"2024-03-11T21:28:23.156564+0000","last_peered":"2024-03-11T21:28:23.156564+0000","last_clean":"2024-03-11T21:28:23.156564+0000","last_became_active":"2024-03-11T21:25:38.541705+0000","last_became_peered":"2024-03-11T21:25:38.541705+0000","last_unstale":"2024-03-11T21:28:23.156564+0000","last_undegraded":"2024-03-11T21:28:23.156564+0000","last_fullsized":"2024-03-11T21:28:23.156564+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:28:17.525649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:26:44.860499+0000","last_clean_scrub_stamp":"2024-03-11T21:28:17.525649+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:29:31.562181+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.13","version":"74'11","reported_seq":137,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-03-11T21:28:23.144104+0000","last_change":"2024-03-11T21:27:03.784968+0000","last_active":"2024-03-11T21:28:23.144104+0000","last_peered":"2024-03-11T21:28:23.144104+0000","last_clean":"2024-03-11T21:28:23.144104+0000","last_became_active":"2024-03-11T21:26:11.214879+0000","last_became_peered":"2024-03-11T21:26:11.214879+0000","last_unstale":"2024-03-11T21:28:23.144104+0000","last_undegraded":"2024-03-11T21:28:23.144104+0000","last_fullsized":"2024-03-11T21:28:23.144104+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'11","last_scrub_stamp":"2024-03-11T21:27:03.784851+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:39.525554+0000","last_clean_scrub_stamp":"2024-03-11T21:27:03.784851+0000","objects_scrubbed":5,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,2],"acting":[4,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.10","version":"0'0","reported_seq":90,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-03-11T21:28:15.078866+0000","last_change":"2024-03-11T21:27:25.417528+0000","last_active":"2024-03-11T21:28:15.078866+0000","last_peered":"2024-03-11T21:28:15.078866+0000","last_clean":"2024-03-11T21:28:15.078866+0000","last_became_active":"2024-03-11T21:25:44.591189+0000","last_became_peered":"2024-03-11T21:25:44.591189+0000","last_unstale":"2024-03-11T21:28:15.078866+0000","last_undegraded":"2024-03-11T21:28:15.078866+0000","last_fullsized":"2024-03-11T21:28:15.078866+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:27:25.417450+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:27:25.417450+0000","last_clean_scrub_stamp":"2024-03-11T21:27:25.417450+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:28:52.772668+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,1],"acting":[0,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":93,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-03-11T21:28:15.080701+0000","last_change":"2024-03-11T21:27:06.637829+0000","last_active":"2024-03-11T21:28:15.080701+0000","last_peered":"2024-03-11T21:28:15.080701+0000","last_clean":"2024-03-11T21:28:15.080701+0000","last_became_active":"2024-03-11T21:25:42.576425+0000","last_became_peered":"2024-03-11T21:25:42.576425+0000","last_unstale":"2024-03-11T21:28:15.080701+0000","last_undegraded":"2024-03-11T21:28:15.080701+0000","last_fullsized":"2024-03-11T21:28:15.080701+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:27:06.637752+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:41.544550+0000","last_clean_scrub_stamp":"2024-03-11T21:27:06.637752+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:28:24.953840+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.14","version":"0'0","reported_seq":112,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-03-11T21:28:23.129032+0000","last_change":"2024-03-11T21:28:09.912659+0000","last_active":"2024-03-11T21:28:23.129032+0000","last_peered":"2024-03-11T21:28:23.129032+0000","last_clean":"2024-03-11T21:28:23.129032+0000","last_became_active":"2024-03-11T21:25:38.541852+0000","last_became_peered":"2024-03-11T21:25:38.541852+0000","last_unstale":"2024-03-11T21:28:23.129032+0000","last_undegraded":"2024-03-11T21:28:23.129032+0000","last_fullsized":"2024-03-11T21:28:23.129032+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:28:09.912544+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:37.512018+0000","last_clean_scrub_stamp":"2024-03-11T21:28:09.912544+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:29:18.193923+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,5],"acting":[6,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.12","version":"74'9","reported_seq":116,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-03-11T21:28:23.157238+0000","last_change":"2024-03-11T21:27:15.286166+0000","last_active":"2024-03-11T21:28:23.157238+0000","last_peered":"2024-03-11T21:28:23.157238+0000","last_clean":"2024-03-11T21:28:23.157238+0000","last_became_active":"2024-03-11T21:25:40.547082+0000","last_became_peered":"2024-03-11T21:25:40.547082+0000","last_unstale":"2024-03-11T21:28:23.157238+0000","last_undegraded":"2024-03-11T21:28:23.157238+0000","last_fullsized":"2024-03-11T21:28:23.157238+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'9","last_scrub_stamp":"2024-03-11T21:27:15.286044+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:39.525554+0000","last_clean_scrub_stamp":"2024-03-11T21:27:15.286044+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:28:29.151259+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":92,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-03-11T21:28:23.144874+0000","last_change":"2024-03-11T21:27:09.801239+0000","last_active":"2024-03-11T21:28:23.144874+0000","last_peered":"2024-03-11T21:28:23.144874+0000","last_clean":"2024-03-11T21:28:23.144874+0000","last_became_active":"2024-03-11T21:25:44.589586+0000","last_became_peered":"2024-03-11T21:25:44.589586+0000","last_unstale":"2024-03-11T21:28:23.144874+0000","last_undegraded":"2024-03-11T21:28:23.144874+0000","last_fullsized":"2024-03-11T21:28:23.144874+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:27:09.801138+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:27:09.801138+0000","last_clean_scrub_stamp":"2024-03-11T21:27:09.801138+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:28:37.698118+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,5],"acting":[4,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.14","version":"74'8","reported_seq":134,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-03-11T21:28:15.080295+0000","last_change":"2024-03-11T21:27:24.803037+0000","last_active":"2024-03-11T21:28:15.080295+0000","last_peered":"2024-03-11T21:28:15.080295+0000","last_clean":"2024-03-11T21:28:15.080295+0000","last_became_active":"2024-03-11T21:25:42.583266+0000","last_became_peered":"2024-03-11T21:25:42.583266+0000","last_unstale":"2024-03-11T21:28:15.080295+0000","last_undegraded":"2024-03-11T21:28:15.080295+0000","last_fullsized":"2024-03-11T21:28:15.080295+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'8","last_scrub_stamp":"2024-03-11T21:27:24.802932+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:41.544550+0000","last_clean_scrub_stamp":"2024-03-11T21:27:24.802932+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:28:44.617485+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,2],"acting":[3,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.13","version":"0'0","reported_seq":102,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-03-11T21:28:15.079810+0000","last_change":"2024-03-11T21:27:12.315278+0000","last_active":"2024-03-11T21:28:15.079810+0000","last_peered":"2024-03-11T21:28:15.079810+0000","last_clean":"2024-03-11T21:28:15.079810+0000","last_became_active":"2024-03-11T21:25:38.539467+0000","last_became_peered":"2024-03-11T21:25:38.539467+0000","last_unstale":"2024-03-11T21:28:15.079810+0000","last_undegraded":"2024-03-11T21:28:15.079810+0000","last_fullsized":"2024-03-11T21:28:15.079810+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:27:12.315200+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:37.512018+0000","last_clean_scrub_stamp":"2024-03-11T21:27:12.315200+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:28:29.228183+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.15","version":"94'13","reported_seq":119,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-03-11T21:28:23.152400+0000","last_change":"2024-03-11T21:26:59.013939+0000","last_active":"2024-03-11T21:28:23.152400+0000","last_peered":"2024-03-11T21:28:23.152400+0000","last_clean":"2024-03-11T21:28:23.152400+0000","last_became_active":"2024-03-11T21:25:40.760907+0000","last_became_peered":"2024-03-11T21:25:40.760907+0000","last_unstale":"2024-03-11T21:28:23.152400+0000","last_undegraded":"2024-03-11T21:28:23.152400+0000","last_fullsized":"2024-03-11T21:28:23.152400+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"94'13","last_scrub_stamp":"2024-03-11T21:26:59.013823+0000","last_deep_scrub":"94'13","last_deep_scrub_stamp":"2024-03-11T21:26:59.013823+0000","last_clean_scrub_stamp":"2024-03-11T21:26:59.013823+0000","objects_scrubbed":4,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:28:25.819110+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.16","version":"0'0","reported_seq":90,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-03-11T21:28:15.080051+0000","last_change":"2024-03-11T21:26:57.367233+0000","last_active":"2024-03-11T21:28:15.080051+0000","last_peered":"2024-03-11T21:28:15.080051+0000","last_clean":"2024-03-11T21:28:15.080051+0000","last_became_active":"2024-03-11T21:25:44.583393+0000","last_became_peered":"2024-03-11T21:25:44.583393+0000","last_unstale":"2024-03-11T21:28:15.080051+0000","last_undegraded":"2024-03-11T21:28:15.080051+0000","last_fullsized":"2024-03-11T21:28:15.080051+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:26:57.367155+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:43.559452+0000","last_clean_scrub_stamp":"2024-03-11T21:26:57.367155+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:28:24.056062+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.15","version":"98'9","reported_seq":138,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-03-11T21:28:23.152208+0000","last_change":"2024-03-11T21:27:34.948935+0000","last_active":"2024-03-11T21:28:23.152208+0000","last_peered":"2024-03-11T21:28:23.152208+0000","last_clean":"2024-03-11T21:28:23.152208+0000","last_became_active":"2024-03-11T21:25:42.572555+0000","last_became_peered":"2024-03-11T21:25:42.572555+0000","last_unstale":"2024-03-11T21:28:23.152208+0000","last_undegraded":"2024-03-11T21:28:23.152208+0000","last_fullsized":"2024-03-11T21:28:23.152208+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"98'9","last_scrub_stamp":"2024-03-11T21:27:34.948855+0000","last_deep_scrub":"98'9","last_deep_scrub_stamp":"2024-03-11T21:27:34.948855+0000","last_clean_scrub_stamp":"2024-03-11T21:27:34.948855+0000","objects_scrubbed":1,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:28:45.397760+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.12","version":"0'0","reported_seq":111,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-03-11T21:28:26.701693+0000","last_change":"2024-03-11T21:28:26.701731+0000","last_active":"2024-03-11T21:28:26.701693+0000","last_peered":"2024-03-11T21:28:26.701693+0000","last_clean":"2024-03-11T21:28:26.701693+0000","last_became_active":"2024-03-11T21:25:38.543962+0000","last_became_peered":"2024-03-11T21:25:38.543962+0000","last_unstale":"2024-03-11T21:28:26.701693+0000","last_undegraded":"2024-03-11T21:28:26.701693+0000","last_fullsized":"2024-03-11T21:28:26.701693+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:28:26.701620+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:37.512018+0000","last_clean_scrub_stamp":"2024-03-11T21:28:26.701620+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:29:51.689535+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.14","version":"74'10","reported_seq":112,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-03-11T21:28:15.077282+0000","last_change":"2024-03-11T21:27:19.754590+0000","last_active":"2024-03-11T21:28:15.077282+0000","last_peered":"2024-03-11T21:28:15.077282+0000","last_clean":"2024-03-11T21:28:15.077282+0000","last_became_active":"2024-03-11T21:25:40.760620+0000","last_became_peered":"2024-03-11T21:25:40.760620+0000","last_unstale":"2024-03-11T21:28:15.077282+0000","last_undegraded":"2024-03-11T21:28:15.077282+0000","last_fullsized":"2024-03-11T21:28:15.077282+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'10","last_scrub_stamp":"2024-03-11T21:27:19.754499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:39.525554+0000","last_clean_scrub_stamp":"2024-03-11T21:27:19.754499+0000","objects_scrubbed":5,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:28:40.603422+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.15","version":"0'0","reported_seq":91,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-03-11T21:28:23.124077+0000","last_change":"2024-03-11T21:27:00.378136+0000","last_active":"2024-03-11T21:28:23.124077+0000","last_peered":"2024-03-11T21:28:23.124077+0000","last_clean":"2024-03-11T21:28:23.124077+0000","last_became_active":"2024-03-11T21:25:45.302501+0000","last_became_peered":"2024-03-11T21:25:45.302501+0000","last_unstale":"2024-03-11T21:28:23.124077+0000","last_undegraded":"2024-03-11T21:28:23.124077+0000","last_fullsized":"2024-03-11T21:28:23.124077+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:27:00.377987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:43.559452+0000","last_clean_scrub_stamp":"2024-03-11T21:27:00.377987+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":95,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-03-11T21:28:23.153104+0000","last_change":"2024-03-11T21:27:33.987450+0000","last_active":"2024-03-11T21:28:23.153104+0000","last_peered":"2024-03-11T21:28:23.153104+0000","last_clean":"2024-03-11T21:28:23.153104+0000","last_became_active":"2024-03-11T21:25:42.577351+0000","last_became_peered":"2024-03-11T21:25:42.577351+0000","last_unstale":"2024-03-11T21:28:23.153104+0000","last_undegraded":"2024-03-11T21:28:23.153104+0000","last_fullsized":"2024-03-11T21:28:23.153104+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:27:33.987367+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:41.544550+0000","last_clean_scrub_stamp":"2024-03-11T21:27:33.987367+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:28:41.819337+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,1],"acting":[5,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.11","version":"0'0","reported_seq":104,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-03-11T21:28:23.128289+0000","last_change":"2024-03-11T21:27:16.985709+0000","last_active":"2024-03-11T21:28:23.128289+0000","last_peered":"2024-03-11T21:28:23.128289+0000","last_clean":"2024-03-11T21:28:23.128289+0000","last_became_active":"2024-03-11T21:25:38.537415+0000","last_became_peered":"2024-03-11T21:25:38.537415+0000","last_unstale":"2024-03-11T21:28:23.128289+0000","last_undegraded":"2024-03-11T21:28:23.128289+0000","last_fullsized":"2024-03-11T21:28:23.128289+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:27:16.985601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:37.512018+0000","last_clean_scrub_stamp":"2024-03-11T21:27:16.985601+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:28:27.096179+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.17","version":"97'9","reported_seq":123,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-03-11T21:28:15.083793+0000","last_change":"2024-03-11T21:27:25.803221+0000","last_active":"2024-03-11T21:28:15.083793+0000","last_peered":"2024-03-11T21:28:15.083793+0000","last_clean":"2024-03-11T21:28:15.083793+0000","last_became_active":"2024-03-11T21:26:11.219020+0000","last_became_peered":"2024-03-11T21:26:11.219020+0000","last_unstale":"2024-03-11T21:28:15.083793+0000","last_undegraded":"2024-03-11T21:28:15.083793+0000","last_fullsized":"2024-03-11T21:28:15.083793+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"97'9","last_scrub_stamp":"2024-03-11T21:27:25.803132+0000","last_deep_scrub":"97'9","last_deep_scrub_stamp":"2024-03-11T21:27:25.803132+0000","last_clean_scrub_stamp":"2024-03-11T21:27:25.803132+0000","objects_scrubbed":3,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:28:40.421034+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":9,"num_read_kb":6,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,5],"acting":[3,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.14","version":"0'0","reported_seq":89,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-03-11T21:28:15.071784+0000","last_change":"2024-03-11T21:27:21.063894+0000","last_active":"2024-03-11T21:28:15.071784+0000","last_peered":"2024-03-11T21:28:15.071784+0000","last_clean":"2024-03-11T21:28:15.071784+0000","last_became_active":"2024-03-11T21:25:44.587439+0000","last_became_peered":"2024-03-11T21:25:44.587439+0000","last_unstale":"2024-03-11T21:28:15.071784+0000","last_undegraded":"2024-03-11T21:28:15.071784+0000","last_fullsized":"2024-03-11T21:28:15.071784+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:27:21.063817+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:43.559452+0000","last_clean_scrub_stamp":"2024-03-11T21:27:21.063817+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:28:29.835843+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,7],"acting":[2,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":93,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-03-11T21:28:15.083879+0000","last_change":"2024-03-11T21:27:15.766288+0000","last_active":"2024-03-11T21:28:15.083879+0000","last_peered":"2024-03-11T21:28:15.083879+0000","last_clean":"2024-03-11T21:28:15.083879+0000","last_became_active":"2024-03-11T21:25:42.583514+0000","last_became_peered":"2024-03-11T21:25:42.583514+0000","last_unstale":"2024-03-11T21:28:15.083879+0000","last_undegraded":"2024-03-11T21:28:15.083879+0000","last_fullsized":"2024-03-11T21:28:15.083879+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:27:15.766188+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:41.544550+0000","last_clean_scrub_stamp":"2024-03-11T21:27:15.766188+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:28:22.407684+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.10","version":"0'0","reported_seq":101,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-03-11T21:28:15.071905+0000","last_change":"2024-03-11T21:27:09.937508+0000","last_active":"2024-03-11T21:28:15.071905+0000","last_peered":"2024-03-11T21:28:15.071905+0000","last_clean":"2024-03-11T21:28:15.071905+0000","last_became_active":"2024-03-11T21:25:38.533138+0000","last_became_peered":"2024-03-11T21:25:38.533138+0000","last_unstale":"2024-03-11T21:28:15.071905+0000","last_undegraded":"2024-03-11T21:28:15.071905+0000","last_fullsized":"2024-03-11T21:28:15.071905+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:27:09.937426+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:37.512018+0000","last_clean_scrub_stamp":"2024-03-11T21:27:09.937426+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:28:23.648986+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,0],"acting":[2,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.16","version":"74'9","reported_seq":122,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-03-11T21:28:15.080286+0000","last_change":"2024-03-11T21:28:10.399010+0000","last_active":"2024-03-11T21:28:15.080286+0000","last_peered":"2024-03-11T21:28:15.080286+0000","last_clean":"2024-03-11T21:28:15.080286+0000","last_became_active":"2024-03-11T21:25:40.763434+0000","last_became_peered":"2024-03-11T21:25:40.763434+0000","last_unstale":"2024-03-11T21:28:15.080286+0000","last_undegraded":"2024-03-11T21:28:15.080286+0000","last_fullsized":"2024-03-11T21:28:15.080286+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'9","last_scrub_stamp":"2024-03-11T21:28:10.398261+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:39.525554+0000","last_clean_scrub_stamp":"2024-03-11T21:28:10.398261+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:29:12.053153+0000","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.a","version":"108'4204","reported_seq":4231,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-03-11T21:28:22.034844+0000","last_change":"2024-03-11T21:27:36.693277+0000","last_active":"2024-03-11T21:28:22.034844+0000","last_peered":"2024-03-11T21:28:22.034844+0000","last_clean":"2024-03-11T21:28:22.034844+0000","last_became_active":"2024-03-11T21:27:36.692988+0000","last_became_peered":"2024-03-11T21:27:36.692988+0000","last_unstale":"2024-03-11T21:28:22.034844+0000","last_undegraded":"2024-03-11T21:28:22.034844+0000","last_fullsized":"2024-03-11T21:28:22.034844+0000","mapping_epoch":99,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:27:35.667522+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:27:35.667522+0000","last_clean_scrub_stamp":"2024-03-11T21:27:35.667522+0000","objects_scrubbed":0,"log_size":4204,"log_dups_size":0,"ondisk_log_size":4204,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:28:43.758763+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":275513344,"num_objects":4204,"num_object_clones":0,"num_object_copies":12612,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4204,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4204,"num_write_kb":269056,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5],"acting":[3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.b","version":"0'0","reported_seq":89,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-03-11T21:28:15.080776+0000","last_change":"2024-03-11T21:28:03.724789+0000","last_active":"2024-03-11T21:28:15.080776+0000","last_peered":"2024-03-11T21:28:15.080776+0000","last_clean":"2024-03-11T21:28:15.080776+0000","last_became_active":"2024-03-11T21:25:44.590071+0000","last_became_peered":"2024-03-11T21:25:44.590071+0000","last_unstale":"2024-03-11T21:28:15.080776+0000","last_undegraded":"2024-03-11T21:28:15.080776+0000","last_fullsized":"2024-03-11T21:28:15.080776+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:28:03.724666+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:43.559452+0000","last_clean_scrub_stamp":"2024-03-11T21:28:03.724666+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:29:23.617411+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.f","version":"67'2","reported_seq":116,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-03-11T21:28:23.144727+0000","last_change":"2024-03-11T21:27:21.959075+0000","last_active":"2024-03-11T21:28:23.144727+0000","last_peered":"2024-03-11T21:28:23.144727+0000","last_clean":"2024-03-11T21:28:23.144727+0000","last_became_active":"2024-03-11T21:25:38.537711+0000","last_became_peered":"2024-03-11T21:25:38.537711+0000","last_unstale":"2024-03-11T21:28:23.144727+0000","last_undegraded":"2024-03-11T21:28:23.144727+0000","last_fullsized":"2024-03-11T21:28:23.144727+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'2","last_scrub_stamp":"2024-03-11T21:27:21.958970+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:37.512018+0000","last_clean_scrub_stamp":"2024-03-11T21:27:21.958970+0000","objects_scrubbed":2,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:28:34.591168+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":92,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":10,"num_read_kb":10,"num_write":4,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,7],"acting":[4,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.9","version":"74'12","reported_seq":123,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-03-11T21:28:23.143996+0000","last_change":"2024-03-11T21:28:07.735179+0000","last_active":"2024-03-11T21:28:23.143996+0000","last_peered":"2024-03-11T21:28:23.143996+0000","last_clean":"2024-03-11T21:28:23.143996+0000","last_became_active":"2024-03-11T21:25:40.552267+0000","last_became_peered":"2024-03-11T21:25:40.552267+0000","last_unstale":"2024-03-11T21:28:23.143996+0000","last_undegraded":"2024-03-11T21:28:23.143996+0000","last_fullsized":"2024-03-11T21:28:23.143996+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'12","last_scrub_stamp":"2024-03-11T21:28:07.735010+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:39.525554+0000","last_clean_scrub_stamp":"2024-03-11T21:28:07.735010+0000","objects_scrubbed":5,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:29:16.999453+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":25,"num_read_kb":16,"num_write":14,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,3],"acting":[4,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":93,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-03-11T21:28:15.071816+0000","last_change":"2024-03-11T21:27:08.954258+0000","last_active":"2024-03-11T21:28:15.071816+0000","last_peered":"2024-03-11T21:28:15.071816+0000","last_clean":"2024-03-11T21:28:15.071816+0000","last_became_active":"2024-03-11T21:25:42.573987+0000","last_became_peered":"2024-03-11T21:25:42.573987+0000","last_unstale":"2024-03-11T21:28:15.071816+0000","last_undegraded":"2024-03-11T21:28:15.071816+0000","last_fullsized":"2024-03-11T21:28:15.071816+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:27:08.954143+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:41.544550+0000","last_clean_scrub_stamp":"2024-03-11T21:27:08.954143+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:28:34.087042+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.1","version":"109'4600","reported_seq":4630,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-03-11T21:28:26.951511+0000","last_change":"2024-03-11T21:27:36.697329+0000","last_active":"2024-03-11T21:28:26.951511+0000","last_peered":"2024-03-11T21:28:26.951511+0000","last_clean":"2024-03-11T21:28:26.951511+0000","last_became_active":"2024-03-11T21:27:36.696990+0000","last_became_peered":"2024-03-11T21:27:36.696990+0000","last_unstale":"2024-03-11T21:28:26.951511+0000","last_undegraded":"2024-03-11T21:28:26.951511+0000","last_fullsized":"2024-03-11T21:28:26.951511+0000","mapping_epoch":99,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:27:35.667522+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:27:35.667522+0000","last_clean_scrub_stamp":"2024-03-11T21:27:35.667522+0000","objects_scrubbed":0,"log_size":4600,"log_dups_size":0,"ondisk_log_size":4600,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:28:46.268780+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":301465600,"num_objects":4600,"num_object_clones":0,"num_object_copies":13800,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4600,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4600,"num_write_kb":294400,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.0","version":"0'0","reported_seq":90,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-03-11T21:28:15.078959+0000","last_change":"2024-03-11T21:27:23.389441+0000","last_active":"2024-03-11T21:28:15.078959+0000","last_peered":"2024-03-11T21:28:15.078959+0000","last_clean":"2024-03-11T21:28:15.078959+0000","last_became_active":"2024-03-11T21:25:44.584565+0000","last_became_peered":"2024-03-11T21:25:44.584565+0000","last_unstale":"2024-03-11T21:28:15.078959+0000","last_undegraded":"2024-03-11T21:28:15.078959+0000","last_fullsized":"2024-03-11T21:28:15.078959+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:27:23.389362+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:43.559452+0000","last_clean_scrub_stamp":"2024-03-11T21:27:23.389362+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:28:49.772995+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":104,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-03-11T21:28:23.159505+0000","last_change":"2024-03-11T21:27:06.092300+0000","last_active":"2024-03-11T21:28:23.159505+0000","last_peered":"2024-03-11T21:28:23.159505+0000","last_clean":"2024-03-11T21:28:23.159505+0000","last_became_active":"2024-03-11T21:25:38.541760+0000","last_became_peered":"2024-03-11T21:25:38.541760+0000","last_unstale":"2024-03-11T21:28:23.159505+0000","last_undegraded":"2024-03-11T21:28:23.159505+0000","last_fullsized":"2024-03-11T21:28:23.159505+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:27:06.092220+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:37.512018+0000","last_clean_scrub_stamp":"2024-03-11T21:27:06.092220+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,7],"acting":[1,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.2","version":"94'15","reported_seq":128,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-03-11T21:28:23.157328+0000","last_change":"2024-03-11T21:28:08.398044+0000","last_active":"2024-03-11T21:28:23.157328+0000","last_peered":"2024-03-11T21:28:23.157328+0000","last_clean":"2024-03-11T21:28:23.157328+0000","last_became_active":"2024-03-11T21:25:40.548833+0000","last_became_peered":"2024-03-11T21:25:40.548833+0000","last_unstale":"2024-03-11T21:28:23.157328+0000","last_undegraded":"2024-03-11T21:28:23.157328+0000","last_fullsized":"2024-03-11T21:28:23.157328+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"94'15","last_scrub_stamp":"2024-03-11T21:28:08.397342+0000","last_deep_scrub":"94'15","last_deep_scrub_stamp":"2024-03-11T21:26:49.993060+0000","last_clean_scrub_stamp":"2024-03-11T21:28:08.397342+0000","objects_scrubbed":5,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:29:11.389113+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.3","version":"74'8","reported_seq":135,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-03-11T21:28:15.079996+0000","last_change":"2024-03-11T21:27:06.321926+0000","last_active":"2024-03-11T21:28:15.079996+0000","last_peered":"2024-03-11T21:28:15.079996+0000","last_clean":"2024-03-11T21:28:15.079996+0000","last_became_active":"2024-03-11T21:25:42.578161+0000","last_became_peered":"2024-03-11T21:25:42.578161+0000","last_unstale":"2024-03-11T21:28:15.079996+0000","last_undegraded":"2024-03-11T21:28:15.079996+0000","last_fullsized":"2024-03-11T21:28:15.079996+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'8","last_scrub_stamp":"2024-03-11T21:27:06.321832+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:41.544550+0000","last_clean_scrub_stamp":"2024-03-11T21:27:06.321832+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:28:27.084331+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,5],"acting":[0,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.7","version":"109'4618","reported_seq":4648,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-03-11T21:28:26.942296+0000","last_change":"2024-03-11T21:27:36.695426+0000","last_active":"2024-03-11T21:28:26.942296+0000","last_peered":"2024-03-11T21:28:26.942296+0000","last_clean":"2024-03-11T21:28:26.942296+0000","last_became_active":"2024-03-11T21:27:36.695055+0000","last_became_peered":"2024-03-11T21:27:36.695055+0000","last_unstale":"2024-03-11T21:28:26.942296+0000","last_undegraded":"2024-03-11T21:28:26.942296+0000","last_fullsized":"2024-03-11T21:28:26.942296+0000","mapping_epoch":99,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:27:35.667522+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:27:35.667522+0000","last_clean_scrub_stamp":"2024-03-11T21:27:35.667522+0000","objects_scrubbed":0,"log_size":4618,"log_dups_size":0,"ondisk_log_size":4618,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:29:04.544436+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":302645248,"num_objects":4618,"num_object_clones":0,"num_object_copies":13854,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4618,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4618,"num_write_kb":295552,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,3],"acting":[6,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":89,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-03-11T21:28:15.079643+0000","last_change":"2024-03-11T21:27:07.651113+0000","last_active":"2024-03-11T21:28:15.079643+0000","last_peered":"2024-03-11T21:28:15.079643+0000","last_clean":"2024-03-11T21:28:15.079643+0000","last_became_active":"2024-03-11T21:25:44.585223+0000","last_became_peered":"2024-03-11T21:25:44.585223+0000","last_unstale":"2024-03-11T21:28:15.079643+0000","last_undegraded":"2024-03-11T21:28:15.079643+0000","last_fullsized":"2024-03-11T21:28:15.079643+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:27:07.651016+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:43.559452+0000","last_clean_scrub_stamp":"2024-03-11T21:27:07.651016+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,7],"acting":[3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":111,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-03-11T21:28:23.151917+0000","last_change":"2024-03-11T21:28:10.753819+0000","last_active":"2024-03-11T21:28:23.151917+0000","last_peered":"2024-03-11T21:28:23.151917+0000","last_clean":"2024-03-11T21:28:23.151917+0000","last_became_active":"2024-03-11T21:25:38.538540+0000","last_became_peered":"2024-03-11T21:25:38.538540+0000","last_unstale":"2024-03-11T21:28:23.151917+0000","last_undegraded":"2024-03-11T21:28:23.151917+0000","last_fullsized":"2024-03-11T21:28:23.151917+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:28:10.753706+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:37.512018+0000","last_clean_scrub_stamp":"2024-03-11T21:28:10.753706+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:29:15.610522+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.4","version":"74'28","reported_seq":190,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-03-11T21:28:23.160067+0000","last_change":"2024-03-11T21:28:21.455801+0000","last_active":"2024-03-11T21:28:23.160067+0000","last_peered":"2024-03-11T21:28:23.160067+0000","last_clean":"2024-03-11T21:28:23.160067+0000","last_became_active":"2024-03-11T21:25:40.552902+0000","last_became_peered":"2024-03-11T21:25:40.552902+0000","last_unstale":"2024-03-11T21:28:23.160067+0000","last_undegraded":"2024-03-11T21:28:23.160067+0000","last_fullsized":"2024-03-11T21:28:23.160067+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'28","last_scrub_stamp":"2024-03-11T21:28:21.455629+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:39.525554+0000","last_clean_scrub_stamp":"2024-03-11T21:28:21.455629+0000","objects_scrubbed":10,"log_size":28,"log_dups_size":0,"ondisk_log_size":28,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:29:48.292369+0000","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":358,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":48,"num_read_kb":33,"num_write":26,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3],"acting":[1,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":94,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-03-11T21:28:15.083369+0000","last_change":"2024-03-11T21:27:20.393288+0000","last_active":"2024-03-11T21:28:15.083369+0000","last_peered":"2024-03-11T21:28:15.083369+0000","last_clean":"2024-03-11T21:28:15.083369+0000","last_became_active":"2024-03-11T21:25:42.572673+0000","last_became_peered":"2024-03-11T21:25:42.572673+0000","last_unstale":"2024-03-11T21:28:15.083369+0000","last_undegraded":"2024-03-11T21:28:15.083369+0000","last_fullsized":"2024-03-11T21:28:15.083369+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:27:20.393205+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:41.544550+0000","last_clean_scrub_stamp":"2024-03-11T21:27:20.393205+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:28:49.759997+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.4","version":"108'4292","reported_seq":4320,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-03-11T21:28:21.968912+0000","last_change":"2024-03-11T21:27:36.691053+0000","last_active":"2024-03-11T21:28:21.968912+0000","last_peered":"2024-03-11T21:28:21.968912+0000","last_clean":"2024-03-11T21:28:21.968912+0000","last_became_active":"2024-03-11T21:27:36.690751+0000","last_became_peered":"2024-03-11T21:27:36.690751+0000","last_unstale":"2024-03-11T21:28:21.968912+0000","last_undegraded":"2024-03-11T21:28:21.968912+0000","last_fullsized":"2024-03-11T21:28:21.968912+0000","mapping_epoch":99,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:27:35.667522+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:27:35.667522+0000","last_clean_scrub_stamp":"2024-03-11T21:27:35.667522+0000","objects_scrubbed":0,"log_size":4292,"log_dups_size":0,"ondisk_log_size":4292,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:28:49.540586+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":281280512,"num_objects":4292,"num_object_clones":0,"num_object_copies":12876,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4292,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4292,"num_write_kb":274688,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,1],"acting":[0,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.5","version":"0'0","reported_seq":91,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-03-11T21:28:23.124246+0000","last_change":"2024-03-11T21:27:05.338089+0000","last_active":"2024-03-11T21:28:23.124246+0000","last_peered":"2024-03-11T21:28:23.124246+0000","last_clean":"2024-03-11T21:28:23.124246+0000","last_became_active":"2024-03-11T21:25:45.302120+0000","last_became_peered":"2024-03-11T21:25:45.302120+0000","last_unstale":"2024-03-11T21:28:23.124246+0000","last_undegraded":"2024-03-11T21:28:23.124246+0000","last_fullsized":"2024-03-11T21:28:23.124246+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:27:05.337991+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:43.559452+0000","last_clean_scrub_stamp":"2024-03-11T21:27:05.337991+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:28:33.442171+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,3],"acting":[7,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.0","version":"76'5","reported_seq":285,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-03-11T21:28:26.389894+0000","last_change":"2024-03-11T21:28:26.389894+0000","last_active":"2024-03-11T21:28:26.389894+0000","last_peered":"2024-03-11T21:28:26.389894+0000","last_clean":"2024-03-11T21:28:26.389894+0000","last_became_active":"2024-03-11T21:25:38.536890+0000","last_became_peered":"2024-03-11T21:25:38.536890+0000","last_unstale":"2024-03-11T21:28:26.389894+0000","last_undegraded":"2024-03-11T21:28:26.389894+0000","last_fullsized":"2024-03-11T21:28:26.389894+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"76'5","last_scrub_stamp":"2024-03-11T21:28:26.389561+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:37.512018+0000","last_clean_scrub_stamp":"2024-03-11T21:28:26.389561+0000","objects_scrubbed":1,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:29:43.918091+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0.000753559,"stat_sum":{"num_bytes":389,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":163,"num_read_kb":158,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,6],"acting":[1,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":109,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-03-11T21:28:17.037006+0000","last_change":"2024-03-11T21:28:17.037045+0000","last_active":"2024-03-11T21:28:17.037006+0000","last_peered":"2024-03-11T21:28:17.037006+0000","last_clean":"2024-03-11T21:28:17.037006+0000","last_became_active":"2024-03-11T21:25:38.532984+0000","last_became_peered":"2024-03-11T21:25:38.532984+0000","last_unstale":"2024-03-11T21:28:17.037006+0000","last_undegraded":"2024-03-11T21:28:17.037006+0000","last_fullsized":"2024-03-11T21:28:17.037006+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:28:17.036907+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:26:48.997454+0000","last_clean_scrub_stamp":"2024-03-11T21:28:17.036907+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:29:25.260052+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.7","version":"97'18","reported_seq":132,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-03-11T21:28:23.157419+0000","last_change":"2024-03-11T21:27:19.239524+0000","last_active":"2024-03-11T21:28:23.157419+0000","last_peered":"2024-03-11T21:28:23.157419+0000","last_clean":"2024-03-11T21:28:23.157419+0000","last_became_active":"2024-03-11T21:25:40.541769+0000","last_became_peered":"2024-03-11T21:25:40.541769+0000","last_unstale":"2024-03-11T21:28:23.157419+0000","last_undegraded":"2024-03-11T21:28:23.157419+0000","last_fullsized":"2024-03-11T21:28:23.157419+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"97'18","last_scrub_stamp":"2024-03-11T21:27:19.239404+0000","last_deep_scrub":"97'18","last_deep_scrub_stamp":"2024-03-11T21:27:19.239404+0000","last_clean_scrub_stamp":"2024-03-11T21:27:19.239404+0000","objects_scrubbed":5,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:28:31.185368+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":19,"num_write":16,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":93,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-03-11T21:28:15.075400+0000","last_change":"2024-03-11T21:27:02.039134+0000","last_active":"2024-03-11T21:28:15.075400+0000","last_peered":"2024-03-11T21:28:15.075400+0000","last_clean":"2024-03-11T21:28:15.075400+0000","last_became_active":"2024-03-11T21:25:42.581710+0000","last_became_peered":"2024-03-11T21:25:42.581710+0000","last_unstale":"2024-03-11T21:28:15.075400+0000","last_undegraded":"2024-03-11T21:28:15.075400+0000","last_fullsized":"2024-03-11T21:28:15.075400+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:27:02.039041+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:41.544550+0000","last_clean_scrub_stamp":"2024-03-11T21:27:02.039041+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:28:26.912856+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7],"acting":[2,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.6","version":"108'4158","reported_seq":4184,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-03-11T21:28:21.977464+0000","last_change":"2024-03-11T21:27:36.695308+0000","last_active":"2024-03-11T21:28:21.977464+0000","last_peered":"2024-03-11T21:28:21.977464+0000","last_clean":"2024-03-11T21:28:21.977464+0000","last_became_active":"2024-03-11T21:27:36.695018+0000","last_became_peered":"2024-03-11T21:27:36.695018+0000","last_unstale":"2024-03-11T21:28:21.977464+0000","last_undegraded":"2024-03-11T21:28:21.977464+0000","last_fullsized":"2024-03-11T21:28:21.977464+0000","mapping_epoch":99,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:27:35.667522+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:27:35.667522+0000","last_clean_scrub_stamp":"2024-03-11T21:27:35.667522+0000","objects_scrubbed":0,"log_size":4158,"log_dups_size":0,"ondisk_log_size":4158,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:28:37.233229+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":272498688,"num_objects":4158,"num_object_clones":0,"num_object_copies":12474,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4158,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4158,"num_write_kb":266112,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.7","version":"74'1","reported_seq":92,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-03-11T21:28:23.152396+0000","last_change":"2024-03-11T21:28:05.716314+0000","last_active":"2024-03-11T21:28:23.152396+0000","last_peered":"2024-03-11T21:28:23.152396+0000","last_clean":"2024-03-11T21:28:23.152396+0000","last_became_active":"2024-03-11T21:25:44.591376+0000","last_became_peered":"2024-03-11T21:25:44.591376+0000","last_unstale":"2024-03-11T21:28:23.152396+0000","last_undegraded":"2024-03-11T21:28:23.152396+0000","last_fullsized":"2024-03-11T21:28:23.152396+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'1","last_scrub_stamp":"2024-03-11T21:28:05.716180+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:43.559452+0000","last_clean_scrub_stamp":"2024-03-11T21:28:05.716180+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:29:28.966516+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,4],"acting":[5,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2","version":"71'2","reported_seq":118,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-03-11T21:28:15.701555+0000","last_change":"2024-03-11T21:28:15.701596+0000","last_active":"2024-03-11T21:28:15.701555+0000","last_peered":"2024-03-11T21:28:15.701555+0000","last_clean":"2024-03-11T21:28:15.701555+0000","last_became_active":"2024-03-11T21:25:38.539413+0000","last_became_peered":"2024-03-11T21:25:38.539413+0000","last_unstale":"2024-03-11T21:28:15.701555+0000","last_undegraded":"2024-03-11T21:28:15.701555+0000","last_fullsized":"2024-03-11T21:28:15.701555+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"71'2","last_scrub_stamp":"2024-03-11T21:28:15.701477+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:37.512018+0000","last_clean_scrub_stamp":"2024-03-11T21:28:15.701477+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:29:26.797762+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0.0014206449999999999,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":117,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-03-11T21:28:23.151784+0000","last_change":"2024-03-11T21:28:12.795411+0000","last_active":"2024-03-11T21:28:23.151784+0000","last_peered":"2024-03-11T21:28:23.151784+0000","last_clean":"2024-03-11T21:28:23.151784+0000","last_became_active":"2024-03-11T21:26:11.216188+0000","last_became_peered":"2024-03-11T21:26:11.216188+0000","last_unstale":"2024-03-11T21:28:23.151784+0000","last_undegraded":"2024-03-11T21:28:23.151784+0000","last_fullsized":"2024-03-11T21:28:23.151784+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:28:12.795259+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:37.512018+0000","last_clean_scrub_stamp":"2024-03-11T21:28:12.795259+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:29:14.049738+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,4],"acting":[5,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"87'128","reported_seq":356,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-03-11T21:28:23.123405+0000","last_change":"2024-03-11T21:28:03.750801+0000","last_active":"2024-03-11T21:28:23.123405+0000","last_peered":"2024-03-11T21:28:23.123405+0000","last_clean":"2024-03-11T21:28:23.123405+0000","last_became_active":"2024-03-11T21:25:30.502597+0000","last_became_peered":"2024-03-11T21:25:30.502597+0000","last_unstale":"2024-03-11T21:28:23.123405+0000","last_undegraded":"2024-03-11T21:28:23.123405+0000","last_fullsized":"2024-03-11T21:28:23.123405+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"87'128","last_scrub_stamp":"2024-03-11T21:28:03.750641+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:24.261974+0000","last_clean_scrub_stamp":"2024-03-11T21:28:03.750641+0000","objects_scrubbed":2,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:29:12.310597+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":212,"num_read_kb":306,"num_write":179,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1771104,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.5","version":"74'16","reported_seq":156,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-03-11T21:28:23.128526+0000","last_change":"2024-03-11T21:27:13.982835+0000","last_active":"2024-03-11T21:28:23.128526+0000","last_peered":"2024-03-11T21:28:23.128526+0000","last_clean":"2024-03-11T21:28:23.128526+0000","last_became_active":"2024-03-11T21:25:40.761457+0000","last_became_peered":"2024-03-11T21:25:40.761457+0000","last_unstale":"2024-03-11T21:28:23.128526+0000","last_undegraded":"2024-03-11T21:28:23.128526+0000","last_fullsized":"2024-03-11T21:28:23.128526+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'16","last_scrub_stamp":"2024-03-11T21:27:13.982681+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:39.525554+0000","last_clean_scrub_stamp":"2024-03-11T21:27:13.982681+0000","objects_scrubbed":8,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:28:28.254017+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":154,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":25,"num_read_kb":15,"num_write":13,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":103,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-03-11T21:28:23.124310+0000","last_change":"2024-03-11T21:28:12.639736+0000","last_active":"2024-03-11T21:28:23.124310+0000","last_peered":"2024-03-11T21:28:23.124310+0000","last_clean":"2024-03-11T21:28:23.124310+0000","last_became_active":"2024-03-11T21:25:42.586226+0000","last_became_peered":"2024-03-11T21:25:42.586226+0000","last_unstale":"2024-03-11T21:28:23.124310+0000","last_undegraded":"2024-03-11T21:28:23.124310+0000","last_fullsized":"2024-03-11T21:28:23.124310+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:28:12.639653+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:41.544550+0000","last_clean_scrub_stamp":"2024-03-11T21:28:12.639653+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:29:22.433321+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,5],"acting":[7,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.5","version":"109'4594","reported_seq":4623,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-03-11T21:28:26.970277+0000","last_change":"2024-03-11T21:27:36.692195+0000","last_active":"2024-03-11T21:28:26.970277+0000","last_peered":"2024-03-11T21:28:26.970277+0000","last_clean":"2024-03-11T21:28:26.970277+0000","last_became_active":"2024-03-11T21:27:36.691428+0000","last_became_peered":"2024-03-11T21:27:36.691428+0000","last_unstale":"2024-03-11T21:28:26.970277+0000","last_undegraded":"2024-03-11T21:28:26.970277+0000","last_fullsized":"2024-03-11T21:28:26.970277+0000","mapping_epoch":99,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:27:35.667522+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:27:35.667522+0000","last_clean_scrub_stamp":"2024-03-11T21:27:35.667522+0000","objects_scrubbed":0,"log_size":4594,"log_dups_size":0,"ondisk_log_size":4594,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:29:01.981820+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":301072384,"num_objects":4594,"num_object_clones":0,"num_object_copies":13782,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4594,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4594,"num_write_kb":294016,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,0],"acting":[7,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.4","version":"0'0","reported_seq":92,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-03-11T21:28:23.159912+0000","last_change":"2024-03-11T21:27:00.119584+0000","last_active":"2024-03-11T21:28:23.159912+0000","last_peered":"2024-03-11T21:28:23.159912+0000","last_clean":"2024-03-11T21:28:23.159912+0000","last_became_active":"2024-03-11T21:25:44.594188+0000","last_became_peered":"2024-03-11T21:25:44.594188+0000","last_unstale":"2024-03-11T21:28:23.159912+0000","last_undegraded":"2024-03-11T21:28:23.159912+0000","last_fullsized":"2024-03-11T21:28:23.159912+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:27:00.119508+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:43.559452+0000","last_clean_scrub_stamp":"2024-03-11T21:27:00.119508+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":117,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-03-11T21:28:15.081809+0000","last_change":"2024-03-11T21:28:06.322812+0000","last_active":"2024-03-11T21:28:15.081809+0000","last_peered":"2024-03-11T21:28:15.081809+0000","last_clean":"2024-03-11T21:28:15.081809+0000","last_became_active":"2024-03-11T21:25:38.543834+0000","last_became_peered":"2024-03-11T21:25:38.543834+0000","last_unstale":"2024-03-11T21:28:15.081809+0000","last_undegraded":"2024-03-11T21:28:15.081809+0000","last_fullsized":"2024-03-11T21:28:15.081809+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:28:06.322702+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:37.512018+0000","last_clean_scrub_stamp":"2024-03-11T21:28:06.322702+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:29:32.617923+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0.00089342499999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":111,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-03-11T21:28:23.123333+0000","last_change":"2024-03-11T21:28:17.526308+0000","last_active":"2024-03-11T21:28:23.123333+0000","last_peered":"2024-03-11T21:28:23.123333+0000","last_clean":"2024-03-11T21:28:23.123333+0000","last_became_active":"2024-03-11T21:25:38.541617+0000","last_became_peered":"2024-03-11T21:25:38.541617+0000","last_unstale":"2024-03-11T21:28:23.123333+0000","last_undegraded":"2024-03-11T21:28:23.123333+0000","last_fullsized":"2024-03-11T21:28:23.123333+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:28:17.526232+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:37.512018+0000","last_clean_scrub_stamp":"2024-03-11T21:28:17.526232+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:29:18.725065+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,0],"acting":[7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.6","version":"74'12","reported_seq":116,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-03-11T21:28:15.080128+0000","last_change":"2024-03-11T21:27:01.353450+0000","last_active":"2024-03-11T21:28:15.080128+0000","last_peered":"2024-03-11T21:28:15.080128+0000","last_clean":"2024-03-11T21:28:15.080128+0000","last_became_active":"2024-03-11T21:25:40.550388+0000","last_became_peered":"2024-03-11T21:25:40.550388+0000","last_unstale":"2024-03-11T21:28:15.080128+0000","last_undegraded":"2024-03-11T21:28:15.080128+0000","last_fullsized":"2024-03-11T21:28:15.080128+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'12","last_scrub_stamp":"2024-03-11T21:27:01.353347+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:39.525554+0000","last_clean_scrub_stamp":"2024-03-11T21:27:01.353347+0000","objects_scrubbed":6,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:28:25.846115+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":18,"num_read_kb":12,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":95,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-03-11T21:28:23.152357+0000","last_change":"2024-03-11T21:27:32.991020+0000","last_active":"2024-03-11T21:28:23.152357+0000","last_peered":"2024-03-11T21:28:23.152357+0000","last_clean":"2024-03-11T21:28:23.152357+0000","last_became_active":"2024-03-11T21:25:42.572324+0000","last_became_peered":"2024-03-11T21:25:42.572324+0000","last_unstale":"2024-03-11T21:28:23.152357+0000","last_undegraded":"2024-03-11T21:28:23.152357+0000","last_fullsized":"2024-03-11T21:28:23.152357+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:27:32.990943+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:41.544550+0000","last_clean_scrub_stamp":"2024-03-11T21:27:32.990943+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:28:42.753007+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.0","version":"109'4428","reported_seq":4457,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-03-11T21:28:26.943352+0000","last_change":"2024-03-11T21:27:36.694516+0000","last_active":"2024-03-11T21:28:26.943352+0000","last_peered":"2024-03-11T21:28:26.943352+0000","last_clean":"2024-03-11T21:28:26.943352+0000","last_became_active":"2024-03-11T21:27:36.694232+0000","last_became_peered":"2024-03-11T21:27:36.694232+0000","last_unstale":"2024-03-11T21:28:26.943352+0000","last_undegraded":"2024-03-11T21:28:26.943352+0000","last_fullsized":"2024-03-11T21:28:26.943352+0000","mapping_epoch":99,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:27:35.667522+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:27:35.667522+0000","last_clean_scrub_stamp":"2024-03-11T21:27:35.667522+0000","objects_scrubbed":0,"log_size":4428,"log_dups_size":0,"ondisk_log_size":4428,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:28:44.939811+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":290193408,"num_objects":4428,"num_object_clones":0,"num_object_copies":13284,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4428,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4428,"num_write_kb":283392,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3],"acting":[5,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":100,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-03-11T21:28:23.155371+0000","last_change":"2024-03-11T21:28:10.405572+0000","last_active":"2024-03-11T21:28:23.155371+0000","last_peered":"2024-03-11T21:28:23.155371+0000","last_clean":"2024-03-11T21:28:23.155371+0000","last_became_active":"2024-03-11T21:25:45.301416+0000","last_became_peered":"2024-03-11T21:25:45.301416+0000","last_unstale":"2024-03-11T21:28:23.155371+0000","last_undegraded":"2024-03-11T21:28:23.155371+0000","last_fullsized":"2024-03-11T21:28:23.155371+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:28:10.405058+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:43.559452+0000","last_clean_scrub_stamp":"2024-03-11T21:28:10.405058+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:29:12.495627+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":103,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-03-11T21:28:23.124344+0000","last_change":"2024-03-11T21:26:57.333768+0000","last_active":"2024-03-11T21:28:23.124344+0000","last_peered":"2024-03-11T21:28:23.124344+0000","last_clean":"2024-03-11T21:28:23.124344+0000","last_became_active":"2024-03-11T21:25:38.545148+0000","last_became_peered":"2024-03-11T21:25:38.545148+0000","last_unstale":"2024-03-11T21:28:23.124344+0000","last_undegraded":"2024-03-11T21:28:23.124344+0000","last_fullsized":"2024-03-11T21:28:23.124344+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:26:57.333669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:26:57.333669+0000","last_clean_scrub_stamp":"2024-03-11T21:26:57.333669+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:28:23.923732+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.3","version":"74'19","reported_seq":134,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-03-11T21:28:15.079729+0000","last_change":"2024-03-11T21:27:14.321667+0000","last_active":"2024-03-11T21:28:15.079729+0000","last_peered":"2024-03-11T21:28:15.079729+0000","last_clean":"2024-03-11T21:28:15.079729+0000","last_became_active":"2024-03-11T21:25:40.760594+0000","last_became_peered":"2024-03-11T21:25:40.760594+0000","last_unstale":"2024-03-11T21:28:15.079729+0000","last_undegraded":"2024-03-11T21:28:15.079729+0000","last_fullsized":"2024-03-11T21:28:15.079729+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'19","last_scrub_stamp":"2024-03-11T21:27:14.321574+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:39.525554+0000","last_clean_scrub_stamp":"2024-03-11T21:27:14.321574+0000","objects_scrubbed":8,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:28:21.292097+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":39,"num_read_kb":25,"num_write":22,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":96,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-03-11T21:28:23.128290+0000","last_change":"2024-03-11T21:27:05.035735+0000","last_active":"2024-03-11T21:28:23.128290+0000","last_peered":"2024-03-11T21:28:23.128290+0000","last_clean":"2024-03-11T21:28:23.128290+0000","last_became_active":"2024-03-11T21:25:42.579045+0000","last_became_peered":"2024-03-11T21:25:42.579045+0000","last_unstale":"2024-03-11T21:28:23.128290+0000","last_undegraded":"2024-03-11T21:28:23.128290+0000","last_fullsized":"2024-03-11T21:28:23.128290+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:27:05.035647+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:41.544550+0000","last_clean_scrub_stamp":"2024-03-11T21:27:05.035647+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:28:32.345929+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.3","version":"108'4083","reported_seq":4110,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-03-11T21:28:22.031863+0000","last_change":"2024-03-11T21:27:36.693897+0000","last_active":"2024-03-11T21:28:22.031863+0000","last_peered":"2024-03-11T21:28:22.031863+0000","last_clean":"2024-03-11T21:28:22.031863+0000","last_became_active":"2024-03-11T21:27:36.693506+0000","last_became_peered":"2024-03-11T21:27:36.693506+0000","last_unstale":"2024-03-11T21:28:22.031863+0000","last_undegraded":"2024-03-11T21:28:22.031863+0000","last_fullsized":"2024-03-11T21:28:22.031863+0000","mapping_epoch":99,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:27:35.667522+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:27:35.667522+0000","last_clean_scrub_stamp":"2024-03-11T21:27:35.667522+0000","objects_scrubbed":0,"log_size":4083,"log_dups_size":0,"ondisk_log_size":4083,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:28:54.029141+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":267583488,"num_objects":4083,"num_object_clones":0,"num_object_copies":12249,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4083,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4083,"num_write_kb":261312,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,4],"acting":[3,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":92,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-03-11T21:28:23.143939+0000","last_change":"2024-03-11T21:26:50.836566+0000","last_active":"2024-03-11T21:28:23.143939+0000","last_peered":"2024-03-11T21:28:23.143939+0000","last_clean":"2024-03-11T21:28:23.143939+0000","last_became_active":"2024-03-11T21:25:44.583638+0000","last_became_peered":"2024-03-11T21:25:44.583638+0000","last_unstale":"2024-03-11T21:28:23.143939+0000","last_undegraded":"2024-03-11T21:28:23.143939+0000","last_fullsized":"2024-03-11T21:28:23.143939+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:26:50.836479+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:43.559452+0000","last_clean_scrub_stamp":"2024-03-11T21:26:50.836479+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.6","version":"67'1","reported_seq":113,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-03-11T21:28:23.157628+0000","last_change":"2024-03-11T21:28:19.473355+0000","last_active":"2024-03-11T21:28:23.157628+0000","last_peered":"2024-03-11T21:28:23.157628+0000","last_clean":"2024-03-11T21:28:23.157628+0000","last_became_active":"2024-03-11T21:25:38.536142+0000","last_became_peered":"2024-03-11T21:25:38.536142+0000","last_unstale":"2024-03-11T21:28:23.157628+0000","last_undegraded":"2024-03-11T21:28:23.157628+0000","last_fullsized":"2024-03-11T21:28:23.157628+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'1","last_scrub_stamp":"2024-03-11T21:28:19.473186+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:37.512018+0000","last_clean_scrub_stamp":"2024-03-11T21:28:19.473186+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:29:25.075010+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,4],"acting":[1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.0","version":"93'26","reported_seq":145,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-03-11T21:28:18.664584+0000","last_change":"2024-03-11T21:28:18.664637+0000","last_active":"2024-03-11T21:28:18.664584+0000","last_peered":"2024-03-11T21:28:18.664584+0000","last_clean":"2024-03-11T21:28:18.664584+0000","last_became_active":"2024-03-11T21:25:40.761716+0000","last_became_peered":"2024-03-11T21:25:40.761716+0000","last_unstale":"2024-03-11T21:28:18.664584+0000","last_undegraded":"2024-03-11T21:28:18.664584+0000","last_fullsized":"2024-03-11T21:28:18.664584+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"93'26","last_scrub_stamp":"2024-03-11T21:28:18.664409+0000","last_deep_scrub":"93'26","last_deep_scrub_stamp":"2024-03-11T21:26:40.680324+0000","last_clean_scrub_stamp":"2024-03-11T21:28:18.664409+0000","objects_scrubbed":8,"log_size":26,"log_dups_size":0,"ondisk_log_size":26,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:29:35.615493+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":34,"num_read_kb":22,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":104,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-03-11T21:28:23.769650+0000","last_change":"2024-03-11T21:28:23.769677+0000","last_active":"2024-03-11T21:28:23.769650+0000","last_peered":"2024-03-11T21:28:23.769650+0000","last_clean":"2024-03-11T21:28:23.769650+0000","last_became_active":"2024-03-11T21:25:42.583647+0000","last_became_peered":"2024-03-11T21:25:42.583647+0000","last_unstale":"2024-03-11T21:28:23.769650+0000","last_undegraded":"2024-03-11T21:28:23.769650+0000","last_fullsized":"2024-03-11T21:28:23.769650+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:28:23.769586+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:26:51.863621+0000","last_clean_scrub_stamp":"2024-03-11T21:28:23.769586+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:29:32.731064+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,7],"acting":[4,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.2","version":"108'4172","reported_seq":4198,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-03-11T21:28:22.044488+0000","last_change":"2024-03-11T21:27:36.692676+0000","last_active":"2024-03-11T21:28:22.044488+0000","last_peered":"2024-03-11T21:28:22.044488+0000","last_clean":"2024-03-11T21:28:22.044488+0000","last_became_active":"2024-03-11T21:27:36.692405+0000","last_became_peered":"2024-03-11T21:27:36.692405+0000","last_unstale":"2024-03-11T21:28:22.044488+0000","last_undegraded":"2024-03-11T21:28:22.044488+0000","last_fullsized":"2024-03-11T21:28:22.044488+0000","mapping_epoch":99,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:27:35.667522+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:27:35.667522+0000","last_clean_scrub_stamp":"2024-03-11T21:27:35.667522+0000","objects_scrubbed":0,"log_size":4172,"log_dups_size":0,"ondisk_log_size":4172,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:28:46.439683+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":273416192,"num_objects":4172,"num_object_clones":0,"num_object_copies":12516,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4172,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4172,"num_write_kb":267008,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.3","version":"0'0","reported_seq":99,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-03-11T21:28:25.493560+0000","last_change":"2024-03-11T21:28:25.493599+0000","last_active":"2024-03-11T21:28:25.493560+0000","last_peered":"2024-03-11T21:28:25.493560+0000","last_clean":"2024-03-11T21:28:25.493560+0000","last_became_active":"2024-03-11T21:25:45.302719+0000","last_became_peered":"2024-03-11T21:25:45.302719+0000","last_unstale":"2024-03-11T21:28:25.493560+0000","last_undegraded":"2024-03-11T21:28:25.493560+0000","last_fullsized":"2024-03-11T21:28:25.493560+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:28:25.493325+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:28:25.493325+0000","last_clean_scrub_stamp":"2024-03-11T21:28:25.493325+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:29:27.683604+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,2],"acting":[7,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":112,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-03-11T21:28:23.128564+0000","last_change":"2024-03-11T21:28:04.057863+0000","last_active":"2024-03-11T21:28:23.128564+0000","last_peered":"2024-03-11T21:28:23.128564+0000","last_clean":"2024-03-11T21:28:23.128564+0000","last_became_active":"2024-03-11T21:25:38.540013+0000","last_became_peered":"2024-03-11T21:25:38.540013+0000","last_unstale":"2024-03-11T21:28:23.128564+0000","last_undegraded":"2024-03-11T21:28:23.128564+0000","last_fullsized":"2024-03-11T21:28:23.128564+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:28:04.057757+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:37.512018+0000","last_clean_scrub_stamp":"2024-03-11T21:28:04.057757+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:29:12.016405+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,2],"acting":[6,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.1","version":"74'14","reported_seq":121,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-03-11T21:28:23.143806+0000","last_change":"2024-03-11T21:27:22.932743+0000","last_active":"2024-03-11T21:28:23.143806+0000","last_peered":"2024-03-11T21:28:23.143806+0000","last_clean":"2024-03-11T21:28:23.143806+0000","last_became_active":"2024-03-11T21:25:40.546219+0000","last_became_peered":"2024-03-11T21:25:40.546219+0000","last_unstale":"2024-03-11T21:28:23.143806+0000","last_undegraded":"2024-03-11T21:28:23.143806+0000","last_fullsized":"2024-03-11T21:28:23.143806+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'14","last_scrub_stamp":"2024-03-11T21:27:22.932642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:39.525554+0000","last_clean_scrub_stamp":"2024-03-11T21:27:22.932642+0000","objects_scrubbed":7,"log_size":14,"log_dups_size":0,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:28:36.086651+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":21,"num_read_kb":14,"num_write":14,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6],"acting":[4,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.0","version":"98'9","reported_seq":135,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-03-11T21:28:15.076582+0000","last_change":"2024-03-11T21:27:29.745799+0000","last_active":"2024-03-11T21:28:15.076582+0000","last_peered":"2024-03-11T21:28:15.076582+0000","last_clean":"2024-03-11T21:28:15.076582+0000","last_became_active":"2024-03-11T21:25:42.576126+0000","last_became_peered":"2024-03-11T21:25:42.576126+0000","last_unstale":"2024-03-11T21:28:15.076582+0000","last_undegraded":"2024-03-11T21:28:15.076582+0000","last_fullsized":"2024-03-11T21:28:15.076582+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"98'9","last_scrub_stamp":"2024-03-11T21:27:29.745711+0000","last_deep_scrub":"98'9","last_deep_scrub_stamp":"2024-03-11T21:27:29.745711+0000","last_clean_scrub_stamp":"2024-03-11T21:27:29.745711+0000","objects_scrubbed":1,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:28:34.303318+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.d","version":"109'4661","reported_seq":4691,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-03-11T21:28:26.969690+0000","last_change":"2024-03-11T21:27:36.695471+0000","last_active":"2024-03-11T21:28:26.969690+0000","last_peered":"2024-03-11T21:28:26.969690+0000","last_clean":"2024-03-11T21:28:26.969690+0000","last_became_active":"2024-03-11T21:27:36.695186+0000","last_became_peered":"2024-03-11T21:27:36.695186+0000","last_unstale":"2024-03-11T21:28:26.969690+0000","last_undegraded":"2024-03-11T21:28:26.969690+0000","last_fullsized":"2024-03-11T21:28:26.969690+0000","mapping_epoch":99,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:27:35.667522+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:27:35.667522+0000","last_clean_scrub_stamp":"2024-03-11T21:27:35.667522+0000","objects_scrubbed":0,"log_size":4661,"log_dups_size":0,"ondisk_log_size":4661,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:28:55.480832+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":305463296,"num_objects":4661,"num_object_clones":0,"num_object_copies":13983,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4661,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4661,"num_write_kb":298304,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,0],"acting":[6,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.c","version":"0'0","reported_seq":89,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-03-11T21:28:15.076824+0000","last_change":"2024-03-11T21:28:02.703398+0000","last_active":"2024-03-11T21:28:15.076824+0000","last_peered":"2024-03-11T21:28:15.076824+0000","last_clean":"2024-03-11T21:28:15.076824+0000","last_became_active":"2024-03-11T21:25:45.302207+0000","last_became_peered":"2024-03-11T21:25:45.302207+0000","last_unstale":"2024-03-11T21:28:15.076824+0000","last_undegraded":"2024-03-11T21:28:15.076824+0000","last_fullsized":"2024-03-11T21:28:15.076824+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:28:02.703154+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:43.559452+0000","last_clean_scrub_stamp":"2024-03-11T21:28:02.703154+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:29:31.096224+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.f","version":"0'0","reported_seq":103,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-03-11T21:28:23.152233+0000","last_change":"2024-03-11T21:28:03.765879+0000","last_active":"2024-03-11T21:28:23.152233+0000","last_peered":"2024-03-11T21:28:23.152233+0000","last_clean":"2024-03-11T21:28:23.152233+0000","last_became_active":"2024-03-11T21:25:42.575935+0000","last_became_peered":"2024-03-11T21:25:42.575935+0000","last_unstale":"2024-03-11T21:28:23.152233+0000","last_undegraded":"2024-03-11T21:28:23.152233+0000","last_fullsized":"2024-03-11T21:28:23.152233+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:28:03.765784+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:41.544550+0000","last_clean_scrub_stamp":"2024-03-11T21:28:03.765784+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:29:08.965186+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.8","version":"0'0","reported_seq":103,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-03-11T21:28:23.123426+0000","last_change":"2024-03-11T21:26:48.424013+0000","last_active":"2024-03-11T21:28:23.123426+0000","last_peered":"2024-03-11T21:28:23.123426+0000","last_clean":"2024-03-11T21:28:23.123426+0000","last_became_active":"2024-03-11T21:25:38.540910+0000","last_became_peered":"2024-03-11T21:25:38.540910+0000","last_unstale":"2024-03-11T21:28:23.123426+0000","last_undegraded":"2024-03-11T21:28:23.123426+0000","last_fullsized":"2024-03-11T21:28:23.123426+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:26:48.423910+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:37.512018+0000","last_clean_scrub_stamp":"2024-03-11T21:26:48.423910+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,1],"acting":[7,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.e","version":"109'16","reported_seq":132,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-03-11T21:28:24.739477+0000","last_change":"2024-03-11T21:28:24.739573+0000","last_active":"2024-03-11T21:28:24.739477+0000","last_peered":"2024-03-11T21:28:24.739477+0000","last_clean":"2024-03-11T21:28:24.739477+0000","last_became_active":"2024-03-11T21:25:40.552178+0000","last_became_peered":"2024-03-11T21:25:40.552178+0000","last_unstale":"2024-03-11T21:28:24.739477+0000","last_undegraded":"2024-03-11T21:28:24.739477+0000","last_fullsized":"2024-03-11T21:28:24.739477+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"109'16","last_scrub_stamp":"2024-03-11T21:28:24.739395+0000","last_deep_scrub":"109'16","last_deep_scrub_stamp":"2024-03-11T21:28:24.739395+0000","last_clean_scrub_stamp":"2024-03-11T21:28:24.739395+0000","objects_scrubbed":5,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:29:43.197007+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.c","version":"109'4482","reported_seq":4512,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-03-11T21:28:26.951868+0000","last_change":"2024-03-11T21:27:36.693712+0000","last_active":"2024-03-11T21:28:26.951868+0000","last_peered":"2024-03-11T21:28:26.951868+0000","last_clean":"2024-03-11T21:28:26.951868+0000","last_became_active":"2024-03-11T21:27:36.693325+0000","last_became_peered":"2024-03-11T21:27:36.693325+0000","last_unstale":"2024-03-11T21:28:26.951868+0000","last_undegraded":"2024-03-11T21:28:26.951868+0000","last_fullsized":"2024-03-11T21:28:26.951868+0000","mapping_epoch":99,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:27:35.667522+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:27:35.667522+0000","last_clean_scrub_stamp":"2024-03-11T21:27:35.667522+0000","objects_scrubbed":0,"log_size":4482,"log_dups_size":0,"ondisk_log_size":4482,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:28:41.510375+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":293732352,"num_objects":4482,"num_object_clones":0,"num_object_copies":13446,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4482,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4482,"num_write_kb":286848,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,7],"acting":[6,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.d","version":"0'0","reported_seq":99,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-03-11T21:28:23.796906+0000","last_change":"2024-03-11T21:28:23.796927+0000","last_active":"2024-03-11T21:28:23.796906+0000","last_peered":"2024-03-11T21:28:23.796906+0000","last_clean":"2024-03-11T21:28:23.796906+0000","last_became_active":"2024-03-11T21:25:44.590333+0000","last_became_peered":"2024-03-11T21:25:44.590333+0000","last_unstale":"2024-03-11T21:28:23.796906+0000","last_undegraded":"2024-03-11T21:28:23.796906+0000","last_fullsized":"2024-03-11T21:28:23.796906+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:28:23.796842+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:43.559452+0000","last_clean_scrub_stamp":"2024-03-11T21:28:23.796842+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:29:49.049133+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.e","version":"74'8","reported_seq":138,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-03-11T21:28:23.144655+0000","last_change":"2024-03-11T21:27:19.970228+0000","last_active":"2024-03-11T21:28:23.144655+0000","last_peered":"2024-03-11T21:28:23.144655+0000","last_clean":"2024-03-11T21:28:23.144655+0000","last_became_active":"2024-03-11T21:25:42.576742+0000","last_became_peered":"2024-03-11T21:25:42.576742+0000","last_unstale":"2024-03-11T21:28:23.144655+0000","last_undegraded":"2024-03-11T21:28:23.144655+0000","last_fullsized":"2024-03-11T21:28:23.144655+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'8","last_scrub_stamp":"2024-03-11T21:27:19.970141+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:41.544550+0000","last_clean_scrub_stamp":"2024-03-11T21:27:19.970141+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:28:48.877453+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,0],"acting":[4,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.9","version":"0'0","reported_seq":112,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-03-11T21:28:23.155193+0000","last_change":"2024-03-11T21:28:07.421777+0000","last_active":"2024-03-11T21:28:23.155193+0000","last_peered":"2024-03-11T21:28:23.155193+0000","last_clean":"2024-03-11T21:28:23.155193+0000","last_became_active":"2024-03-11T21:25:38.540976+0000","last_became_peered":"2024-03-11T21:25:38.540976+0000","last_unstale":"2024-03-11T21:28:23.155193+0000","last_undegraded":"2024-03-11T21:28:23.155193+0000","last_fullsized":"2024-03-11T21:28:23.155193+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:28:07.421657+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:26:50.991712+0000","last_clean_scrub_stamp":"2024-03-11T21:28:07.421657+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:29:15.081862+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,3],"acting":[1,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.f","version":"74'15","reported_seq":82,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-03-11T21:28:15.072326+0000","last_change":"2024-03-11T21:27:28.081459+0000","last_active":"2024-03-11T21:28:15.072326+0000","last_peered":"2024-03-11T21:28:15.072326+0000","last_clean":"2024-03-11T21:28:15.072326+0000","last_became_active":"2024-03-11T21:26:11.216224+0000","last_became_peered":"2024-03-11T21:26:11.216224+0000","last_unstale":"2024-03-11T21:28:15.072326+0000","last_undegraded":"2024-03-11T21:28:15.072326+0000","last_fullsized":"2024-03-11T21:28:15.072326+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'15","last_scrub_stamp":"2024-03-11T21:27:28.081356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:39.525554+0000","last_clean_scrub_stamp":"2024-03-11T21:27:28.081356+0000","objects_scrubbed":7,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:28:57.158997+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,4],"acting":[2,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.f","version":"108'4190","reported_seq":4218,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-03-11T21:28:21.975254+0000","last_change":"2024-03-11T21:27:36.694341+0000","last_active":"2024-03-11T21:28:21.975254+0000","last_peered":"2024-03-11T21:28:21.975254+0000","last_clean":"2024-03-11T21:28:21.975254+0000","last_became_active":"2024-03-11T21:27:36.694032+0000","last_became_peered":"2024-03-11T21:27:36.694032+0000","last_unstale":"2024-03-11T21:28:21.975254+0000","last_undegraded":"2024-03-11T21:28:21.975254+0000","last_fullsized":"2024-03-11T21:28:21.975254+0000","mapping_epoch":99,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:27:35.667522+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:27:35.667522+0000","last_clean_scrub_stamp":"2024-03-11T21:27:35.667522+0000","objects_scrubbed":0,"log_size":4190,"log_dups_size":0,"ondisk_log_size":4190,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:28:41.584726+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":274595840,"num_objects":4190,"num_object_clones":0,"num_object_copies":12570,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4190,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4190,"num_write_kb":268160,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.e","version":"0'0","reported_seq":92,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-03-11T21:28:23.144573+0000","last_change":"2024-03-11T21:27:31.792816+0000","last_active":"2024-03-11T21:28:23.144573+0000","last_peered":"2024-03-11T21:28:23.144573+0000","last_clean":"2024-03-11T21:28:23.144573+0000","last_became_active":"2024-03-11T21:25:44.590198+0000","last_became_peered":"2024-03-11T21:25:44.590198+0000","last_unstale":"2024-03-11T21:28:23.144573+0000","last_undegraded":"2024-03-11T21:28:23.144573+0000","last_fullsized":"2024-03-11T21:28:23.144573+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:27:31.792707+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:43.559452+0000","last_clean_scrub_stamp":"2024-03-11T21:27:31.792707+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:28:51.775206+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.d","version":"74'8","reported_seq":134,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-03-11T21:28:15.072244+0000","last_change":"2024-03-11T21:26:59.021603+0000","last_active":"2024-03-11T21:28:15.072244+0000","last_peered":"2024-03-11T21:28:15.072244+0000","last_clean":"2024-03-11T21:28:15.072244+0000","last_became_active":"2024-03-11T21:25:42.575583+0000","last_became_peered":"2024-03-11T21:25:42.575583+0000","last_unstale":"2024-03-11T21:28:15.072244+0000","last_undegraded":"2024-03-11T21:28:15.072244+0000","last_fullsized":"2024-03-11T21:28:15.072244+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'8","last_scrub_stamp":"2024-03-11T21:26:59.021523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:41.544550+0000","last_clean_scrub_stamp":"2024-03-11T21:26:59.021523+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:28:21.121669+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,5],"acting":[2,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.a","version":"0'0","reported_seq":112,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-03-11T21:28:23.157086+0000","last_change":"2024-03-11T21:28:05.349499+0000","last_active":"2024-03-11T21:28:23.157086+0000","last_peered":"2024-03-11T21:28:23.157086+0000","last_clean":"2024-03-11T21:28:23.157086+0000","last_became_active":"2024-03-11T21:25:38.541045+0000","last_became_peered":"2024-03-11T21:25:38.541045+0000","last_unstale":"2024-03-11T21:28:23.157086+0000","last_undegraded":"2024-03-11T21:28:23.157086+0000","last_fullsized":"2024-03-11T21:28:23.157086+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:28:05.349379+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:37.512018+0000","last_clean_scrub_stamp":"2024-03-11T21:28:05.349379+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:29:20.671751+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,7],"acting":[1,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.c","version":"74'10","reported_seq":115,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-03-11T21:28:23.144307+0000","last_change":"2024-03-11T21:27:00.807955+0000","last_active":"2024-03-11T21:28:23.144307+0000","last_peered":"2024-03-11T21:28:23.144307+0000","last_clean":"2024-03-11T21:28:23.144307+0000","last_became_active":"2024-03-11T21:25:40.550885+0000","last_became_peered":"2024-03-11T21:25:40.550885+0000","last_unstale":"2024-03-11T21:28:23.144307+0000","last_undegraded":"2024-03-11T21:28:23.144307+0000","last_fullsized":"2024-03-11T21:28:23.144307+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'10","last_scrub_stamp":"2024-03-11T21:27:00.807849+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:39.525554+0000","last_clean_scrub_stamp":"2024-03-11T21:27:00.807849+0000","objects_scrubbed":5,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:28:27.818066+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.e","version":"108'4167","reported_seq":4194,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-03-11T21:28:21.965321+0000","last_change":"2024-03-11T21:27:36.692095+0000","last_active":"2024-03-11T21:28:21.965321+0000","last_peered":"2024-03-11T21:28:21.965321+0000","last_clean":"2024-03-11T21:28:21.965321+0000","last_became_active":"2024-03-11T21:27:36.691800+0000","last_became_peered":"2024-03-11T21:27:36.691800+0000","last_unstale":"2024-03-11T21:28:21.965321+0000","last_undegraded":"2024-03-11T21:28:21.965321+0000","last_fullsized":"2024-03-11T21:28:21.965321+0000","mapping_epoch":99,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:27:35.667522+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:27:35.667522+0000","last_clean_scrub_stamp":"2024-03-11T21:27:35.667522+0000","objects_scrubbed":0,"log_size":4167,"log_dups_size":0,"ondisk_log_size":4167,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:28:54.001153+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":273088512,"num_objects":4167,"num_object_clones":0,"num_object_copies":12501,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4167,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4167,"num_write_kb":266688,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,0],"acting":[2,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.f","version":"0'0","reported_seq":89,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-03-11T21:28:15.072472+0000","last_change":"2024-03-11T21:27:03.011122+0000","last_active":"2024-03-11T21:28:15.072472+0000","last_peered":"2024-03-11T21:28:15.072472+0000","last_clean":"2024-03-11T21:28:15.072472+0000","last_became_active":"2024-03-11T21:25:44.593414+0000","last_became_peered":"2024-03-11T21:25:44.593414+0000","last_unstale":"2024-03-11T21:28:15.072472+0000","last_undegraded":"2024-03-11T21:28:15.072472+0000","last_fullsized":"2024-03-11T21:28:15.072472+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:27:03.010969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:27:03.010969+0000","last_clean_scrub_stamp":"2024-03-11T21:27:03.010969+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:28:22.636838+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,4],"acting":[2,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":96,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-03-11T21:28:23.159835+0000","last_change":"2024-03-11T21:27:22.232224+0000","last_active":"2024-03-11T21:28:23.159835+0000","last_peered":"2024-03-11T21:28:23.159835+0000","last_clean":"2024-03-11T21:28:23.159835+0000","last_became_active":"2024-03-11T21:25:42.579409+0000","last_became_peered":"2024-03-11T21:25:42.579409+0000","last_unstale":"2024-03-11T21:28:23.159835+0000","last_undegraded":"2024-03-11T21:28:23.159835+0000","last_fullsized":"2024-03-11T21:28:23.159835+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:27:22.232114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:41.544550+0000","last_clean_scrub_stamp":"2024-03-11T21:27:22.232114+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:28:27.887527+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.b","version":"0'0","reported_seq":103,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-03-11T21:28:23.123659+0000","last_change":"2024-03-11T21:27:17.475508+0000","last_active":"2024-03-11T21:28:23.123659+0000","last_peered":"2024-03-11T21:28:23.123659+0000","last_clean":"2024-03-11T21:28:23.123659+0000","last_became_active":"2024-03-11T21:25:38.543377+0000","last_became_peered":"2024-03-11T21:25:38.543377+0000","last_unstale":"2024-03-11T21:28:23.123659+0000","last_undegraded":"2024-03-11T21:28:23.123659+0000","last_fullsized":"2024-03-11T21:28:23.123659+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:27:17.475421+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:37.512018+0000","last_clean_scrub_stamp":"2024-03-11T21:27:17.475421+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:28:32.112149+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,5],"acting":[7,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.d","version":"74'17","reported_seq":128,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-03-11T21:28:23.144421+0000","last_change":"2024-03-11T21:27:02.816136+0000","last_active":"2024-03-11T21:28:23.144421+0000","last_peered":"2024-03-11T21:28:23.144421+0000","last_clean":"2024-03-11T21:28:23.144421+0000","last_became_active":"2024-03-11T21:25:40.553454+0000","last_became_peered":"2024-03-11T21:25:40.553454+0000","last_unstale":"2024-03-11T21:28:23.144421+0000","last_undegraded":"2024-03-11T21:28:23.144421+0000","last_fullsized":"2024-03-11T21:28:23.144421+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'17","last_scrub_stamp":"2024-03-11T21:27:02.815980+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:39.525554+0000","last_clean_scrub_stamp":"2024-03-11T21:27:02.815980+0000","objects_scrubbed":8,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":29,"num_read_kb":19,"num_write":18,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,1],"acting":[4,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.9","version":"108'4179","reported_seq":4207,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-03-11T21:28:21.990159+0000","last_change":"2024-03-11T21:27:36.693276+0000","last_active":"2024-03-11T21:28:21.990159+0000","last_peered":"2024-03-11T21:28:21.990159+0000","last_clean":"2024-03-11T21:28:21.990159+0000","last_became_active":"2024-03-11T21:27:36.692978+0000","last_became_peered":"2024-03-11T21:27:36.692978+0000","last_unstale":"2024-03-11T21:28:21.990159+0000","last_undegraded":"2024-03-11T21:28:21.990159+0000","last_fullsized":"2024-03-11T21:28:21.990159+0000","mapping_epoch":99,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:27:35.667522+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:27:35.667522+0000","last_clean_scrub_stamp":"2024-03-11T21:27:35.667522+0000","objects_scrubbed":0,"log_size":4179,"log_dups_size":0,"ondisk_log_size":4179,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:29:00.262706+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":273874944,"num_objects":4179,"num_object_clones":0,"num_object_copies":12537,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4179,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4179,"num_write_kb":267456,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":91,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-03-11T21:28:23.123296+0000","last_change":"2024-03-11T21:27:07.304361+0000","last_active":"2024-03-11T21:28:23.123296+0000","last_peered":"2024-03-11T21:28:23.123296+0000","last_clean":"2024-03-11T21:28:23.123296+0000","last_became_active":"2024-03-11T21:25:44.589973+0000","last_became_peered":"2024-03-11T21:25:44.589973+0000","last_unstale":"2024-03-11T21:28:23.123296+0000","last_undegraded":"2024-03-11T21:28:23.123296+0000","last_fullsized":"2024-03-11T21:28:23.123296+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:27:07.304264+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:43.559452+0000","last_clean_scrub_stamp":"2024-03-11T21:27:07.304264+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:28:32.511531+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,3],"acting":[7,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":93,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-03-11T21:28:15.074769+0000","last_change":"2024-03-11T21:27:29.075249+0000","last_active":"2024-03-11T21:28:15.074769+0000","last_peered":"2024-03-11T21:28:15.074769+0000","last_clean":"2024-03-11T21:28:15.074769+0000","last_became_active":"2024-03-11T21:25:42.574515+0000","last_became_peered":"2024-03-11T21:25:42.574515+0000","last_unstale":"2024-03-11T21:28:15.074769+0000","last_undegraded":"2024-03-11T21:28:15.074769+0000","last_fullsized":"2024-03-11T21:28:15.074769+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:27:29.075166+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:41.544550+0000","last_clean_scrub_stamp":"2024-03-11T21:27:29.075166+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:28:52.194040+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,5],"acting":[2,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.c","version":"0'0","reported_seq":101,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-03-11T21:28:15.072021+0000","last_change":"2024-03-11T21:27:24.105818+0000","last_active":"2024-03-11T21:28:15.072021+0000","last_peered":"2024-03-11T21:28:15.072021+0000","last_clean":"2024-03-11T21:28:15.072021+0000","last_became_active":"2024-03-11T21:25:38.533240+0000","last_became_peered":"2024-03-11T21:25:38.533240+0000","last_unstale":"2024-03-11T21:28:15.072021+0000","last_undegraded":"2024-03-11T21:28:15.072021+0000","last_fullsized":"2024-03-11T21:28:15.072021+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:27:24.105731+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:37.512018+0000","last_clean_scrub_stamp":"2024-03-11T21:27:24.105731+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:28:44.133006+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,0],"acting":[2,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.a","version":"109'28","reported_seq":148,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-03-11T21:28:24.016520+0000","last_change":"2024-03-11T21:28:24.016603+0000","last_active":"2024-03-11T21:28:24.016520+0000","last_peered":"2024-03-11T21:28:24.016520+0000","last_clean":"2024-03-11T21:28:24.016520+0000","last_became_active":"2024-03-11T21:25:40.760698+0000","last_became_peered":"2024-03-11T21:25:40.760698+0000","last_unstale":"2024-03-11T21:28:24.016520+0000","last_undegraded":"2024-03-11T21:28:24.016520+0000","last_fullsized":"2024-03-11T21:28:24.016520+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"109'28","last_scrub_stamp":"2024-03-11T21:28:24.016389+0000","last_deep_scrub":"109'28","last_deep_scrub_stamp":"2024-03-11T21:28:24.016389+0000","last_clean_scrub_stamp":"2024-03-11T21:28:24.016389+0000","objects_scrubbed":9,"log_size":28,"log_dups_size":0,"ondisk_log_size":28,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:29:37.183547+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":32,"num_read_kb":21,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,7],"acting":[6,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.8","version":"108'4112","reported_seq":4139,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-03-11T21:28:21.973313+0000","last_change":"2024-03-11T21:27:36.693400+0000","last_active":"2024-03-11T21:28:21.973313+0000","last_peered":"2024-03-11T21:28:21.973313+0000","last_clean":"2024-03-11T21:28:21.973313+0000","last_became_active":"2024-03-11T21:27:36.693060+0000","last_became_peered":"2024-03-11T21:27:36.693060+0000","last_unstale":"2024-03-11T21:28:21.973313+0000","last_undegraded":"2024-03-11T21:28:21.973313+0000","last_fullsized":"2024-03-11T21:28:21.973313+0000","mapping_epoch":99,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:27:35.667522+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:27:35.667522+0000","last_clean_scrub_stamp":"2024-03-11T21:27:35.667522+0000","objects_scrubbed":0,"log_size":4112,"log_dups_size":0,"ondisk_log_size":4112,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:28:37.218739+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":269484032,"num_objects":4112,"num_object_clones":0,"num_object_copies":12336,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4112,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4112,"num_write_kb":263168,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,4],"acting":[2,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":90,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-03-11T21:28:15.082091+0000","last_change":"2024-03-11T21:27:11.362017+0000","last_active":"2024-03-11T21:28:15.082091+0000","last_peered":"2024-03-11T21:28:15.082091+0000","last_clean":"2024-03-11T21:28:15.082091+0000","last_became_active":"2024-03-11T21:25:44.584154+0000","last_became_peered":"2024-03-11T21:25:44.584154+0000","last_unstale":"2024-03-11T21:28:15.082091+0000","last_undegraded":"2024-03-11T21:28:15.082091+0000","last_fullsized":"2024-03-11T21:28:15.082091+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:27:11.361913+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:27:11.361913+0000","last_clean_scrub_stamp":"2024-03-11T21:27:11.361913+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:28:25.991645+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":93,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-03-11T21:28:15.071819+0000","last_change":"2024-03-11T21:27:10.909699+0000","last_active":"2024-03-11T21:28:15.071819+0000","last_peered":"2024-03-11T21:28:15.071819+0000","last_clean":"2024-03-11T21:28:15.071819+0000","last_became_active":"2024-03-11T21:25:42.576679+0000","last_became_peered":"2024-03-11T21:25:42.576679+0000","last_unstale":"2024-03-11T21:28:15.071819+0000","last_undegraded":"2024-03-11T21:28:15.071819+0000","last_fullsized":"2024-03-11T21:28:15.071819+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:27:10.909543+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:41.544550+0000","last_clean_scrub_stamp":"2024-03-11T21:27:10.909543+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:28:29.725395+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,3],"acting":[2,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.d","version":"0'0","reported_seq":112,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-03-11T21:28:23.156633+0000","last_change":"2024-03-11T21:28:03.343281+0000","last_active":"2024-03-11T21:28:23.156633+0000","last_peered":"2024-03-11T21:28:23.156633+0000","last_clean":"2024-03-11T21:28:23.156633+0000","last_became_active":"2024-03-11T21:25:38.539681+0000","last_became_peered":"2024-03-11T21:25:38.539681+0000","last_unstale":"2024-03-11T21:28:23.156633+0000","last_undegraded":"2024-03-11T21:28:23.156633+0000","last_fullsized":"2024-03-11T21:28:23.156633+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:28:03.343147+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:37.512018+0000","last_clean_scrub_stamp":"2024-03-11T21:28:03.343147+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:29:15.956568+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,3],"acting":[1,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.b","version":"74'9","reported_seq":114,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-03-11T21:28:15.079882+0000","last_change":"2024-03-11T21:27:24.392581+0000","last_active":"2024-03-11T21:28:15.079882+0000","last_peered":"2024-03-11T21:28:15.079882+0000","last_clean":"2024-03-11T21:28:15.079882+0000","last_became_active":"2024-03-11T21:25:40.549612+0000","last_became_peered":"2024-03-11T21:25:40.549612+0000","last_unstale":"2024-03-11T21:28:15.079882+0000","last_undegraded":"2024-03-11T21:28:15.079882+0000","last_fullsized":"2024-03-11T21:28:15.079882+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'9","last_scrub_stamp":"2024-03-11T21:27:24.392483+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:39.525554+0000","last_clean_scrub_stamp":"2024-03-11T21:27:24.392483+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:28:37.805493+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.b","version":"109'4559","reported_seq":4589,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-03-11T21:28:26.958731+0000","last_change":"2024-03-11T21:27:36.691639+0000","last_active":"2024-03-11T21:28:26.958731+0000","last_peered":"2024-03-11T21:28:26.958731+0000","last_clean":"2024-03-11T21:28:26.958731+0000","last_became_active":"2024-03-11T21:27:36.691322+0000","last_became_peered":"2024-03-11T21:27:36.691322+0000","last_unstale":"2024-03-11T21:28:26.958731+0000","last_undegraded":"2024-03-11T21:28:26.958731+0000","last_fullsized":"2024-03-11T21:28:26.958731+0000","mapping_epoch":99,"log_start":"0'0","ondisk_log_start":"0'0","created":99,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:27:35.667522+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:27:35.667522+0000","last_clean_scrub_stamp":"2024-03-11T21:27:35.667522+0000","objects_scrubbed":0,"log_size":4559,"log_dups_size":0,"ondisk_log_size":4559,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:29:04.800550+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":298778624,"num_objects":4559,"num_object_clones":0,"num_object_copies":13677,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4559,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4559,"num_write_kb":291776,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,6],"acting":[1,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":99,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-03-11T21:28:23.151747+0000","last_change":"2024-03-11T21:28:19.815768+0000","last_active":"2024-03-11T21:28:23.151747+0000","last_peered":"2024-03-11T21:28:23.151747+0000","last_clean":"2024-03-11T21:28:23.151747+0000","last_became_active":"2024-03-11T21:25:45.300333+0000","last_became_peered":"2024-03-11T21:25:45.300333+0000","last_unstale":"2024-03-11T21:28:23.151747+0000","last_undegraded":"2024-03-11T21:28:23.151747+0000","last_fullsized":"2024-03-11T21:28:23.151747+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:28:19.815688+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:43.559452+0000","last_clean_scrub_stamp":"2024-03-11T21:28:19.815688+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:29:48.868489+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,0],"acting":[5,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.e","version":"0'0","reported_seq":109,"reported_epoch":108,"state":"active+clean","last_fresh":"2024-03-11T21:28:15.075230+0000","last_change":"2024-03-11T21:28:11.016096+0000","last_active":"2024-03-11T21:28:15.075230+0000","last_peered":"2024-03-11T21:28:15.075230+0000","last_clean":"2024-03-11T21:28:15.075230+0000","last_became_active":"2024-03-11T21:25:38.539954+0000","last_became_peered":"2024-03-11T21:25:38.539954+0000","last_unstale":"2024-03-11T21:28:15.075230+0000","last_undegraded":"2024-03-11T21:28:15.075230+0000","last_fullsized":"2024-03-11T21:28:15.075230+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:28:11.015887+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:37.512018+0000","last_clean_scrub_stamp":"2024-03-11T21:28:11.015887+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:29:16.519153+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,7],"acting":[2,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.8","version":"109'22","reported_seq":139,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-03-11T21:28:25.743148+0000","last_change":"2024-03-11T21:28:25.743214+0000","last_active":"2024-03-11T21:28:25.743148+0000","last_peered":"2024-03-11T21:28:25.743148+0000","last_clean":"2024-03-11T21:28:25.743148+0000","last_became_active":"2024-03-11T21:25:40.761938+0000","last_became_peered":"2024-03-11T21:25:40.761938+0000","last_unstale":"2024-03-11T21:28:25.743148+0000","last_undegraded":"2024-03-11T21:28:25.743148+0000","last_fullsized":"2024-03-11T21:28:25.743148+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"109'22","last_scrub_stamp":"2024-03-11T21:28:25.743058+0000","last_deep_scrub":"109'22","last_deep_scrub_stamp":"2024-03-11T21:28:25.743058+0000","last_clean_scrub_stamp":"2024-03-11T21:28:25.743058+0000","objects_scrubbed":7,"log_size":22,"log_dups_size":0,"ondisk_log_size":22,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:29:41.395729+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.9","version":"94'9","reported_seq":149,"reported_epoch":109,"state":"active+clean","last_fresh":"2024-03-11T21:28:23.123563+0000","last_change":"2024-03-11T21:28:14.615170+0000","last_active":"2024-03-11T21:28:23.123563+0000","last_peered":"2024-03-11T21:28:23.123563+0000","last_clean":"2024-03-11T21:28:23.123563+0000","last_became_active":"2024-03-11T21:25:42.584461+0000","last_became_peered":"2024-03-11T21:25:42.584461+0000","last_unstale":"2024-03-11T21:28:23.123563+0000","last_undegraded":"2024-03-11T21:28:23.123563+0000","last_fullsized":"2024-03-11T21:28:23.123563+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"94'9","last_scrub_stamp":"2024-03-11T21:28:14.615041+0000","last_deep_scrub":"94'9","last_deep_scrub_stamp":"2024-03-11T21:26:53.422552+0000","last_clean_scrub_stamp":"2024-03-11T21:28:14.615041+0000","objects_scrubbed":1,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:29:18.296913+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":7,"num_pg":16,"stat_sum":{"num_bytes":4554686464,"num_objects":69499,"num_object_clones":0,"num_object_copies":208497,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":69499,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":69499,"num_write_kb":4447936,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":13765312512,"data_stored":13765312512,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":69499,"ondisk_log_size":69499,"up":48,"acting":48,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":1531,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":24,"num_read_kb":24,"num_write":10,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":4593,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":6,"ondisk_log_size":6,"up":96,"acting":96,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":212,"num_read_kb":306,"num_write":179,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1771104,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1781760,"data_stored":1771104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":128,"ondisk_log_size":128,"up":3,"acting":3,"num_store_stats":3},{"poolid":3,"num_pg":3,"stat_sum":{"num_bytes":408,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":163,"num_read_kb":158,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1224,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":9,"acting":9,"num_store_stats":7},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":3702,"num_objects":178,"num_object_clones":0,"num_object_copies":534,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":178,"num_whiteouts":0,"num_read":698,"num_read_kb":455,"num_write":417,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":417792,"data_stored":11106,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":441,"ondisk_log_size":441,"up":96,"acting":96,"num_store_stats":8},{"poolid":5,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":67,"ondisk_log_size":67,"up":96,"acting":96,"num_store_stats":8},{"poolid":6,"num_pg":32,"stat_sum":{"num_bytes":383,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1149,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2,"ondisk_log_size":2,"up":96,"acting":96,"num_store_stats":8}],"osd_stats":[{"osd":7,"up_from":60,"seq":257698037797,"num_pgs":61,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1491776,"kb_used_data":1438392,"kb_used_omap":1,"kb_used_meta":53374,"kb_avail":92253376,"statfs":{"total":95995035648,"available":94467457024,"internally_reserved":0,"allocated":1472913408,"data_stored":1472511095,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":54655434},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":50,"seq":214748364843,"num_pgs":49,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1519740,"kb_used_data":1466356,"kb_used_omap":1,"kb_used_meta":53374,"kb_avail":92225412,"statfs":{"total":95995035648,"available":94438821888,"internally_reserved":0,"allocated":1501548544,"data_stored":1501150620,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":54655430},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,3],"upper_bound":32},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574917,"num_pgs":59,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1792408,"kb_used_data":1734864,"kb_used_omap":1,"kb_used_meta":57534,"kb_avail":91952744,"statfs":{"total":95995035648,"available":94159609856,"internally_reserved":0,"allocated":1776500736,"data_stored":1776073704,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":58915269},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,2],"upper_bound":16},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738441,"num_pgs":55,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2211320,"kb_used_data":2139888,"kb_used_omap":1,"kb_used_meta":71422,"kb_avail":91533832,"statfs":{"total":95995035648,"available":93730643968,"internally_reserved":0,"allocated":2191245312,"data_stored":2190844555,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":73136581},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,5,5,1,5,2],"upper_bound":128},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411390,"num_pgs":45,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1622692,"kb_used_data":1573468,"kb_used_omap":1,"kb_used_meta":49214,"kb_avail":92122460,"statfs":{"total":95995035648,"available":94333399040,"internally_reserved":0,"allocated":1611231232,"data_stored":1610849819,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":50395589},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":3,"up_from":26,"seq":111669149754,"num_pgs":64,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2463564,"kb_used_data":2387972,"kb_used_omap":1,"kb_used_meta":75582,"kb_avail":91281588,"statfs":{"total":95995035648,"available":93472346112,"internally_reserved":0,"allocated":2445283328,"data_stored":2444862065,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":77396423},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":4,"up_from":32,"seq":137438953524,"num_pgs":58,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1504840,"kb_used_data":1451456,"kb_used_omap":1,"kb_used_meta":53374,"kb_avail":92240312,"statfs":{"total":95995035648,"available":94454079488,"internally_reserved":0,"allocated":1486290944,"data_stored":1485879804,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":54655429},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,1,0,1],"upper_bound":32},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":41,"seq":176093659183,"num_pgs":53,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1205748,"kb_used_data":1152364,"kb_used_omap":1,"kb_used_meta":53374,"kb_avail":92539404,"statfs":{"total":95995035648,"available":94760349696,"internally_reserved":0,"allocated":1180020736,"data_stored":1179629843,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":54655429},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1],"upper_bound":4},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1049,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":46,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":957,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":574,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":436,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1003,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":528,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":1320,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":81920,"data_stored":2118,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":1018,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":1898,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":1474,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":990,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":1034,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":1254,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":370,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":383,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":370,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":2212495360,"data_stored":2212495360,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":1796276224,"data_stored":1796276224,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":1634926592,"data_stored":1634926592,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":2461335552,"data_stored":2461335552,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":1490550784,"data_stored":1490550784,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":1185153024,"data_stored":1185153024,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":1502216192,"data_stored":1502216192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":1482358784,"data_stored":1482358784,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-11T21:28:29.796 INFO:tasks.thrashosds.thrasher:Setting 6.a to [4, 0, 3] 2024-03-11T21:28:29.796 INFO:tasks.thrashosds.thrasher:cmd ['osd', 'pg-upmap', '6.a', '4', '0', '3'] 2024-03-11T21:28:29.796 DEBUG:teuthology.orchestra.run.smithi092:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid 00db0016-dfed-11ee-95c8-87774f69a715 -- ceph osd pg-upmap 6.a 4 0 3 2024-03-11T21:28:29.803 INFO:teuthology.orchestra.run.smithi092.stderr:Inferring config /var/lib/ceph/00db0016-dfed-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T21:28:29.821 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:28:29.882 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:28:29.883 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:28:29.883 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:28:29.985 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:28:30.035 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:28:30.036 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:28:30.036 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:28:30.137 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:28:30.165 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:28:30.165 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:28:30.166 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:28:30.267 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:28:30.280 INFO:teuthology.orchestra.run.smithi092.stderr:Inferring config /var/lib/ceph/00db0016-dfed-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T21:28:30.280 INFO:teuthology.orchestra.run.smithi092.stderr:Inferring config /var/lib/ceph/00db0016-dfed-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T21:28:30.329 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:28:30.330 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:28:30.330 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:28:30.361 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:28:30 smithi092 ceph-mon[29572]: 2.11 scrub starts 2024-03-11T21:28:30.361 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:28:30 smithi092 ceph-mon[29572]: 2.11 scrub ok 2024-03-11T21:28:30.361 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:28:30 smithi092 ceph-mon[29572]: pgmap v89: 148 pgs: 148 active+clean; 4.4 GiB data, 14 GiB used, 702 GiB / 715 GiB avail; 1023 B/s rd, 74 MiB/s wr, 1.18k op/s 2024-03-11T21:28:30.361 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:28:30 smithi092 ceph-mon[29572]: 6.16 scrub starts 2024-03-11T21:28:30.362 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:28:30 smithi092 ceph-mon[29572]: 6.16 scrub ok 2024-03-11T21:28:30.362 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:28:30 smithi092 ceph-mon[37563]: 2.11 scrub starts 2024-03-11T21:28:30.362 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:28:30 smithi092 ceph-mon[37563]: 2.11 scrub ok 2024-03-11T21:28:30.362 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:28:30 smithi092 ceph-mon[37563]: pgmap v89: 148 pgs: 148 active+clean; 4.4 GiB data, 14 GiB used, 702 GiB / 715 GiB avail; 1023 B/s rd, 74 MiB/s wr, 1.18k op/s 2024-03-11T21:28:30.362 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:28:30 smithi092 ceph-mon[37563]: 6.16 scrub starts 2024-03-11T21:28:30.362 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:28:30 smithi092 ceph-mon[37563]: 6.16 scrub ok 2024-03-11T21:28:30.431 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:28:30.462 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:28:30.462 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:28:30.462 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:28:30.563 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:28:30.576 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:28:30 smithi097 ceph-mon[33317]: 2.11 scrub starts 2024-03-11T21:28:30.577 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:28:30 smithi097 ceph-mon[33317]: 2.11 scrub ok 2024-03-11T21:28:30.577 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:28:30 smithi097 ceph-mon[33317]: pgmap v89: 148 pgs: 148 active+clean; 4.4 GiB data, 14 GiB used, 702 GiB / 715 GiB avail; 1023 B/s rd, 74 MiB/s wr, 1.18k op/s 2024-03-11T21:28:30.577 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:28:30 smithi097 ceph-mon[33317]: 6.16 scrub starts 2024-03-11T21:28:30.577 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:28:30 smithi097 ceph-mon[33317]: 6.16 scrub ok 2024-03-11T21:28:30.591 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:28:30.592 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:28:30.592 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:28:30.636 INFO:tasks.radosbench.radosbench.0.smithi092.stdout: 46 15 76785 76770 104.267 68.8125 0.0138839 0.0095847 2024-03-11T21:28:30.693 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:28:30.721 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:28:30.722 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:28:30.722 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:28:30.823 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:28:30.863 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:28:30.864 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:28:30.864 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:28:30.965 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:28:30.996 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:28:30.996 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:28:30.997 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:28:31.098 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:28:31.143 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:28:31.144 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:28:31.144 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:28:31.245 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:28:31.276 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:28:31.277 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:28:31.277 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:28:31.361 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:28:31 smithi092 ceph-mon[29572]: 6.4 scrub starts 2024-03-11T21:28:31.361 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:28:31 smithi092 ceph-mon[29572]: 6.4 scrub ok 2024-03-11T21:28:31.361 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:28:31 smithi092 ceph-mon[29572]: 2.8 scrub starts 2024-03-11T21:28:31.362 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:28:31 smithi092 ceph-mon[29572]: 2.8 scrub ok 2024-03-11T21:28:31.362 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:28:31 smithi092 ceph-mon[29572]: 4.d scrub starts 2024-03-11T21:28:31.362 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:28:31 smithi092 ceph-mon[29572]: 4.d scrub ok 2024-03-11T21:28:31.362 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:28:31 smithi092 ceph-mon[29572]: 6.f scrub starts 2024-03-11T21:28:31.362 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:28:31 smithi092 ceph-mon[29572]: 6.f scrub ok 2024-03-11T21:28:31.362 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:28:31 smithi092 ceph-mon[29572]: 4.6 scrub starts 2024-03-11T21:28:31.362 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:28:31 smithi092 ceph-mon[29572]: 5.c scrub starts 2024-03-11T21:28:31.362 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:28:31 smithi092 ceph-mon[29572]: 4.6 scrub ok 2024-03-11T21:28:31.362 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:28:31 smithi092 ceph-mon[29572]: 6.6 deep-scrub starts 2024-03-11T21:28:31.362 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:28:31 smithi092 ceph-mon[29572]: 6.6 deep-scrub ok 2024-03-11T21:28:31.362 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:28:31 smithi092 ceph-mon[37563]: 6.4 scrub starts 2024-03-11T21:28:31.363 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:28:31 smithi092 ceph-mon[37563]: 6.4 scrub ok 2024-03-11T21:28:31.363 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:28:31 smithi092 ceph-mon[37563]: 2.8 scrub starts 2024-03-11T21:28:31.363 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:28:31 smithi092 ceph-mon[37563]: 2.8 scrub ok 2024-03-11T21:28:31.363 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:28:31 smithi092 ceph-mon[37563]: 4.d scrub starts 2024-03-11T21:28:31.363 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:28:31 smithi092 ceph-mon[37563]: 4.d scrub ok 2024-03-11T21:28:31.363 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:28:31 smithi092 ceph-mon[37563]: 6.f scrub starts 2024-03-11T21:28:31.363 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:28:31 smithi092 ceph-mon[37563]: 6.f scrub ok 2024-03-11T21:28:31.363 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:28:31 smithi092 ceph-mon[37563]: 4.6 scrub starts 2024-03-11T21:28:31.363 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:28:31 smithi092 ceph-mon[37563]: 5.c scrub starts 2024-03-11T21:28:31.363 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:28:31 smithi092 ceph-mon[37563]: 4.6 scrub ok 2024-03-11T21:28:31.363 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:28:31 smithi092 ceph-mon[37563]: 6.6 deep-scrub starts 2024-03-11T21:28:31.363 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:28:31 smithi092 ceph-mon[37563]: 6.6 deep-scrub ok 2024-03-11T21:28:31.378 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:28:31.420 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:28:31.420 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:28:31.421 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:28:31.522 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:28:31.551 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:28:31.552 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:28:31.552 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:28:31.577 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:28:31 smithi097 ceph-mon[33317]: 6.4 scrub starts 2024-03-11T21:28:31.578 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:28:31 smithi097 ceph-mon[33317]: 6.4 scrub ok 2024-03-11T21:28:31.578 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:28:31 smithi097 ceph-mon[33317]: 2.8 scrub starts 2024-03-11T21:28:31.578 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:28:31 smithi097 ceph-mon[33317]: 2.8 scrub ok 2024-03-11T21:28:31.578 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:28:31 smithi097 ceph-mon[33317]: 4.d scrub starts 2024-03-11T21:28:31.578 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:28:31 smithi097 ceph-mon[33317]: 4.d scrub ok 2024-03-11T21:28:31.578 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:28:31 smithi097 ceph-mon[33317]: 6.f scrub starts 2024-03-11T21:28:31.578 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:28:31 smithi097 ceph-mon[33317]: 6.f scrub ok 2024-03-11T21:28:31.578 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:28:31 smithi097 ceph-mon[33317]: 4.6 scrub starts 2024-03-11T21:28:31.578 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:28:31 smithi097 ceph-mon[33317]: 5.c scrub starts 2024-03-11T21:28:31.578 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:28:31 smithi097 ceph-mon[33317]: 4.6 scrub ok 2024-03-11T21:28:31.578 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:28:31 smithi097 ceph-mon[33317]: 6.6 deep-scrub starts 2024-03-11T21:28:31.578 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:28:31 smithi097 ceph-mon[33317]: 6.6 deep-scrub ok 2024-03-11T21:28:31.636 INFO:tasks.radosbench.radosbench.0.smithi092.stdout: 47 16 78035 78019 103.709 78.0625 0.00528559 0.00963768 2024-03-11T21:28:31.654 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:28:31.686 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:28:31.687 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:28:31.687 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:28:31.788 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:28:31.814 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:28:31.815 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:28:31.815 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:28:31.916 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:28:31.964 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:28:31.965 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:28:31.965 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:28:32.066 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:28:32.116 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:28:32.116 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:28:32.116 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:28:32.217 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:28:32.257 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:28:32.258 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:28:32.258 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:28:32.327 INFO:journalctl@ceph.iscsi.iscsi.a.smithi097.stdout:Mar 11 21:28:32 smithi097 ceph-00db0016-dfed-11ee-95c8-87774f69a715-iscsi-iscsi-a[60943]: debug there is no tcmu-runner data available 2024-03-11T21:28:32.327 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:28:32 smithi097 ceph-mon[33317]: 5.c scrub ok 2024-03-11T21:28:32.327 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:28:32 smithi097 ceph-mon[33317]: 4.5 scrub starts 2024-03-11T21:28:32.327 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:28:32 smithi097 ceph-mon[33317]: 4.5 scrub ok 2024-03-11T21:28:32.327 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:28:32 smithi097 ceph-mon[33317]: pgmap v90: 148 pgs: 148 active+clean; 4.4 GiB data, 14 GiB used, 702 GiB / 715 GiB avail; 1023 B/s rd, 74 MiB/s wr, 1.18k op/s 2024-03-11T21:28:32.327 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:28:32 smithi097 ceph-mon[33317]: 5.1d scrub starts 2024-03-11T21:28:32.327 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:28:32 smithi097 ceph-mon[33317]: 5.1d scrub ok 2024-03-11T21:28:32.359 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:28:32.361 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:28:32 smithi092 ceph-mon[29572]: 5.c scrub ok 2024-03-11T21:28:32.361 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:28:32 smithi092 ceph-mon[29572]: 4.5 scrub starts 2024-03-11T21:28:32.361 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:28:32 smithi092 ceph-mon[29572]: 4.5 scrub ok 2024-03-11T21:28:32.361 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:28:32 smithi092 ceph-mon[29572]: pgmap v90: 148 pgs: 148 active+clean; 4.4 GiB data, 14 GiB used, 702 GiB / 715 GiB avail; 1023 B/s rd, 74 MiB/s wr, 1.18k op/s 2024-03-11T21:28:32.361 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:28:32 smithi092 ceph-mon[29572]: 5.1d scrub starts 2024-03-11T21:28:32.361 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:28:32 smithi092 ceph-mon[29572]: 5.1d scrub ok 2024-03-11T21:28:32.361 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:28:32 smithi092 ceph-mon[37563]: 5.c scrub ok 2024-03-11T21:28:32.361 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:28:32 smithi092 ceph-mon[37563]: 4.5 scrub starts 2024-03-11T21:28:32.361 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:28:32 smithi092 ceph-mon[37563]: 4.5 scrub ok 2024-03-11T21:28:32.361 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:28:32 smithi092 ceph-mon[37563]: pgmap v90: 148 pgs: 148 active+clean; 4.4 GiB data, 14 GiB used, 702 GiB / 715 GiB avail; 1023 B/s rd, 74 MiB/s wr, 1.18k op/s 2024-03-11T21:28:32.362 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:28:32 smithi092 ceph-mon[37563]: 5.1d scrub starts 2024-03-11T21:28:32.362 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:28:32 smithi092 ceph-mon[37563]: 5.1d scrub ok 2024-03-11T21:28:32.388 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:28:32.389 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:28:32.389 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:28:32.490 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:28:32.552 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:28:32.553 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:28:32.553 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:28:32.638 INFO:tasks.radosbench.radosbench.0.smithi092.stdout: 48 16 79216 79200 103.082 73.8125 0.0137282 0.00969556 2024-03-11T21:28:32.654 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:28:32.696 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:28:32.696 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:28:32.697 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:28:32.797 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:28:32.834 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:28:32.835 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:28:32.835 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:28:32.936 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:28:32.965 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:28:32.966 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:28:32.966 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:28:33.067 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:28:33.106 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:28:33.107 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:28:33.107 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:28:33.208 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:28:33.259 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:28:33.260 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:28:33.260 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:28:33.291 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:28:33 smithi092 ceph-mon[29572]: from='client.24469 v1:172.21.15.97:0/404208310' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:28:33.291 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:28:33 smithi092 ceph-mon[29572]: 4.12 scrub starts 2024-03-11T21:28:33.291 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:28:33 smithi092 ceph-mon[29572]: 4.12 scrub ok 2024-03-11T21:28:33.291 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:28:33 smithi092 ceph-mon[37563]: from='client.24469 v1:172.21.15.97:0/404208310' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:28:33.291 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:28:33 smithi092 ceph-mon[37563]: 4.12 scrub starts 2024-03-11T21:28:33.291 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:28:33 smithi092 ceph-mon[37563]: 4.12 scrub ok 2024-03-11T21:28:33.361 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:28:33.405 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:28:33.406 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:28:33.406 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:28:33.507 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:28:33.561 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:28:33.562 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:28:33.562 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:28:33.576 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:28:33 smithi097 ceph-mon[33317]: from='client.24469 v1:172.21.15.97:0/404208310' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:28:33.577 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:28:33 smithi097 ceph-mon[33317]: 4.12 scrub starts 2024-03-11T21:28:33.577 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:28:33 smithi097 ceph-mon[33317]: 4.12 scrub ok 2024-03-11T21:28:33.638 INFO:tasks.radosbench.radosbench.0.smithi092.stdout: 49 16 80391 80375 102.477 73.4375 0.0114254 0.00975276 2024-03-11T21:28:33.663 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:28:33.693 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:28:33.693 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:28:33.694 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:28:33.795 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:28:33.836 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:28:33.836 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:28:33.836 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:28:33.937 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:28:33.966 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:28:33.967 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:28:33.967 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:28:34.068 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:28:34.124 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:28:34.125 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:28:34.125 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:28:34.226 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:28:34.248 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:28:34 smithi092 ceph-mon[29572]: 2.1c deep-scrub starts 2024-03-11T21:28:34.249 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:28:34 smithi092 ceph-mon[29572]: 2.1c deep-scrub ok 2024-03-11T21:28:34.249 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:28:34 smithi092 ceph-mon[29572]: 5.11 scrub starts 2024-03-11T21:28:34.249 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:28:34 smithi092 ceph-mon[29572]: 5.11 scrub ok 2024-03-11T21:28:34.249 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:28:34 smithi092 ceph-mon[29572]: 4.13 deep-scrub starts 2024-03-11T21:28:34.249 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:28:34 smithi092 ceph-mon[29572]: 4.13 deep-scrub ok 2024-03-11T21:28:34.249 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:28:34 smithi092 ceph-mon[29572]: pgmap v91: 148 pgs: 148 active+clean; 4.8 GiB data, 15 GiB used, 701 GiB / 715 GiB avail; 1023 B/s rd, 71 MiB/s wr, 1.14k op/s 2024-03-11T21:28:34.249 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:28:34 smithi092 ceph-mon[29572]: 4.7 scrub starts 2024-03-11T21:28:34.249 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:28:34 smithi092 ceph-mon[29572]: 4.7 scrub ok 2024-03-11T21:28:34.249 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:28:34 smithi092 ceph-mon[29572]: 2.1b deep-scrub starts 2024-03-11T21:28:34.249 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:28:34 smithi092 ceph-mon[29572]: 2.1b deep-scrub ok 2024-03-11T21:28:34.250 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:28:34 smithi092 ceph-mon[37563]: 2.1c deep-scrub starts 2024-03-11T21:28:34.250 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:28:34 smithi092 ceph-mon[37563]: 2.1c deep-scrub ok 2024-03-11T21:28:34.250 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:28:34 smithi092 ceph-mon[37563]: 5.11 scrub starts 2024-03-11T21:28:34.250 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:28:34 smithi092 ceph-mon[37563]: 5.11 scrub ok 2024-03-11T21:28:34.250 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:28:34 smithi092 ceph-mon[37563]: 4.13 deep-scrub starts 2024-03-11T21:28:34.250 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:28:34 smithi092 ceph-mon[37563]: 4.13 deep-scrub ok 2024-03-11T21:28:34.250 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:28:34 smithi092 ceph-mon[37563]: pgmap v91: 148 pgs: 148 active+clean; 4.8 GiB data, 15 GiB used, 701 GiB / 715 GiB avail; 1023 B/s rd, 71 MiB/s wr, 1.14k op/s 2024-03-11T21:28:34.250 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:28:34 smithi092 ceph-mon[37563]: 4.7 scrub starts 2024-03-11T21:28:34.250 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:28:34 smithi092 ceph-mon[37563]: 4.7 scrub ok 2024-03-11T21:28:34.250 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:28:34 smithi092 ceph-mon[37563]: 2.1b deep-scrub starts 2024-03-11T21:28:34.250 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:28:34 smithi092 ceph-mon[37563]: 2.1b deep-scrub ok 2024-03-11T21:28:34.272 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:28:34.272 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:28:34.273 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:28:34.373 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:28:34.384 INFO:teuthology.orchestra.run.smithi092.stdout:osd.0: {} 2024-03-11T21:28:34.419 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:28:34.419 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:28:34.419 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:28:34.422 INFO:teuthology.orchestra.run.smithi092.stdout:osd.1: {} 2024-03-11T21:28:34.473 INFO:teuthology.orchestra.run.smithi092.stdout:osd.2: {} 2024-03-11T21:28:34.519 INFO:teuthology.orchestra.run.smithi092.stdout:osd.3: {} 2024-03-11T21:28:34.520 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:28:34.559 INFO:teuthology.orchestra.run.smithi092.stdout:osd.4: {} 2024-03-11T21:28:34.570 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:28:34.571 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:28:34.571 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:28:34.576 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:28:34 smithi097 ceph-mon[33317]: 2.1c deep-scrub starts 2024-03-11T21:28:34.577 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:28:34 smithi097 ceph-mon[33317]: 2.1c deep-scrub ok 2024-03-11T21:28:34.577 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:28:34 smithi097 ceph-mon[33317]: 5.11 scrub starts 2024-03-11T21:28:34.577 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:28:34 smithi097 ceph-mon[33317]: 5.11 scrub ok 2024-03-11T21:28:34.577 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:28:34 smithi097 ceph-mon[33317]: 4.13 deep-scrub starts 2024-03-11T21:28:34.577 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:28:34 smithi097 ceph-mon[33317]: 4.13 deep-scrub ok 2024-03-11T21:28:34.577 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:28:34 smithi097 ceph-mon[33317]: pgmap v91: 148 pgs: 148 active+clean; 4.8 GiB data, 15 GiB used, 701 GiB / 715 GiB avail; 1023 B/s rd, 71 MiB/s wr, 1.14k op/s 2024-03-11T21:28:34.577 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:28:34 smithi097 ceph-mon[33317]: 4.7 scrub starts 2024-03-11T21:28:34.577 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:28:34 smithi097 ceph-mon[33317]: 4.7 scrub ok 2024-03-11T21:28:34.577 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:28:34 smithi097 ceph-mon[33317]: 2.1b deep-scrub starts 2024-03-11T21:28:34.577 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:28:34 smithi097 ceph-mon[33317]: 2.1b deep-scrub ok 2024-03-11T21:28:34.597 INFO:teuthology.orchestra.run.smithi092.stdout:osd.5: {} 2024-03-11T21:28:34.639 INFO:teuthology.orchestra.run.smithi092.stdout:osd.6: {} 2024-03-11T21:28:34.639 INFO:tasks.radosbench.radosbench.0.smithi092.stdout: 50 16 81478 81462 101.786 67.9375 0.0153392 0.00981957 2024-03-11T21:28:34.673 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:28:34.676 INFO:teuthology.orchestra.run.smithi092.stdout:osd.7: {} 2024-03-11T21:28:34.686 INFO:teuthology.orchestra.run.smithi092.stderr:osd.0: osd_enable_op_tracker = '' 2024-03-11T21:28:34.686 INFO:teuthology.orchestra.run.smithi092.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-11T21:28:34.686 INFO:teuthology.orchestra.run.smithi092.stderr:osd.2: osd_enable_op_tracker = '' 2024-03-11T21:28:34.686 INFO:teuthology.orchestra.run.smithi092.stderr:osd.3: osd_enable_op_tracker = '' 2024-03-11T21:28:34.686 INFO:teuthology.orchestra.run.smithi092.stderr:osd.4: osd_enable_op_tracker = '' 2024-03-11T21:28:34.686 INFO:teuthology.orchestra.run.smithi092.stderr:osd.5: osd_enable_op_tracker = '' 2024-03-11T21:28:34.686 INFO:teuthology.orchestra.run.smithi092.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-11T21:28:34.686 INFO:teuthology.orchestra.run.smithi092.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-11T21:28:34.714 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:28:34.715 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:28:34.716 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:28:34.816 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:28:34.845 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:28:34.845 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:28:34.846 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:28:34.947 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:28:34.975 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:28:34.976 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:28:34.976 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:28:35.077 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:28:35.108 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:28:35.108 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:28:35.109 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:28:35.210 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:28:35.251 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:28:35.252 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:28:35.252 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:28:35.353 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:28:35.391 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:28:35.391 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:28:35.391 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:28:35.492 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:28:35.531 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:28:35.532 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:28:35.532 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:28:35.576 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:28:35 smithi097 ceph-mon[33317]: 6.15 scrub starts 2024-03-11T21:28:35.577 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:28:35 smithi097 ceph-mon[33317]: 6.15 scrub ok 2024-03-11T21:28:35.577 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:28:35 smithi097 ceph-mon[33317]: 5.2 scrub starts 2024-03-11T21:28:35.577 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:28:35 smithi097 ceph-mon[33317]: 5.2 scrub ok 2024-03-11T21:28:35.577 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:28:35 smithi097 ceph-mon[33317]: 5.12 scrub starts 2024-03-11T21:28:35.577 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:28:35 smithi097 ceph-mon[33317]: 5.12 scrub ok 2024-03-11T21:28:35.577 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:28:35 smithi097 ceph-mon[33317]: from='client.? v1:172.21.15.92:0/3030051232' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-11T21:28:35.577 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:28:35 smithi097 ceph-mon[33317]: 6.19 scrub starts 2024-03-11T21:28:35.577 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:28:35 smithi097 ceph-mon[33317]: 6.19 scrub ok 2024-03-11T21:28:35.598 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:28:35 smithi092 ceph-mon[29572]: 6.15 scrub starts 2024-03-11T21:28:35.598 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:28:35 smithi092 ceph-mon[29572]: 6.15 scrub ok 2024-03-11T21:28:35.598 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:28:35 smithi092 ceph-mon[29572]: 5.2 scrub starts 2024-03-11T21:28:35.599 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:28:35 smithi092 ceph-mon[29572]: 5.2 scrub ok 2024-03-11T21:28:35.599 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:28:35 smithi092 ceph-mon[29572]: 5.12 scrub starts 2024-03-11T21:28:35.599 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:28:35 smithi092 ceph-mon[29572]: 5.12 scrub ok 2024-03-11T21:28:35.599 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:28:35 smithi092 ceph-mon[29572]: from='client.? v1:172.21.15.92:0/3030051232' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-11T21:28:35.599 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:28:35 smithi092 ceph-mon[29572]: 6.19 scrub starts 2024-03-11T21:28:35.599 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:28:35 smithi092 ceph-mon[29572]: 6.19 scrub ok 2024-03-11T21:28:35.599 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:28:35 smithi092 ceph-mon[37563]: 6.15 scrub starts 2024-03-11T21:28:35.599 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:28:35 smithi092 ceph-mon[37563]: 6.15 scrub ok 2024-03-11T21:28:35.600 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:28:35 smithi092 ceph-mon[37563]: 5.2 scrub starts 2024-03-11T21:28:35.600 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:28:35 smithi092 ceph-mon[37563]: 5.2 scrub ok 2024-03-11T21:28:35.600 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:28:35 smithi092 ceph-mon[37563]: 5.12 scrub starts 2024-03-11T21:28:35.600 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:28:35 smithi092 ceph-mon[37563]: 5.12 scrub ok 2024-03-11T21:28:35.600 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:28:35 smithi092 ceph-mon[37563]: from='client.? v1:172.21.15.92:0/3030051232' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-11T21:28:35.600 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:28:35 smithi092 ceph-mon[37563]: 6.19 scrub starts 2024-03-11T21:28:35.600 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:28:35 smithi092 ceph-mon[37563]: 6.19 scrub ok 2024-03-11T21:28:35.633 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:28:35.638 INFO:tasks.radosbench.radosbench.0.smithi092.stdout: 51 16 82630 82614 101.202 72 0.00896823 0.00987587 2024-03-11T21:28:35.673 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:28:35.674 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:28:35.674 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:28:35.775 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:28:35.822 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:28:35.823 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:28:35.823 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:28:35.924 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:28:35.952 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:28:35.953 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:28:35.953 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:28:36.054 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:28:36.106 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:28:36.107 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:28:36.107 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:28:36.208 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:28:36.253 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:28:36.254 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:28:36.254 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:28:36.355 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:28:36.384 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:28:36.385 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:28:36.385 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:28:36.486 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:28:36.528 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:28:36.528 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:28:36.529 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:28:36.577 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:28:36 smithi097 ceph-mon[33317]: pgmap v92: 148 pgs: 148 active+clean; 4.8 GiB data, 15 GiB used, 701 GiB / 715 GiB avail; 862 B/s rd, 60 MiB/s wr, 963 op/s 2024-03-11T21:28:36.577 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:28:36 smithi097 ceph-mon[33317]: 4.1c scrub starts 2024-03-11T21:28:36.577 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:28:36 smithi097 ceph-mon[33317]: 4.1c scrub ok 2024-03-11T21:28:36.577 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:28:36 smithi097 ceph-mon[33317]: 5.17 scrub starts 2024-03-11T21:28:36.577 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:28:36 smithi097 ceph-mon[33317]: 5.17 scrub ok 2024-03-11T21:28:36.577 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:28:36 smithi097 ceph-mon[33317]: from='mgr.14607 ' entity='mgr.y' 2024-03-11T21:28:36.577 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:28:36 smithi097 ceph-mon[33317]: from='mgr.14607 v1:172.21.15.92:0/1702717597' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T21:28:36.584 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:28:36 smithi092 ceph-mon[29572]: pgmap v92: 148 pgs: 148 active+clean; 4.8 GiB data, 15 GiB used, 701 GiB / 715 GiB avail; 862 B/s rd, 60 MiB/s wr, 963 op/s 2024-03-11T21:28:36.584 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:28:36 smithi092 ceph-mon[29572]: 4.1c scrub starts 2024-03-11T21:28:36.584 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:28:36 smithi092 ceph-mon[29572]: 4.1c scrub ok 2024-03-11T21:28:36.584 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:28:36 smithi092 ceph-mon[29572]: 5.17 scrub starts 2024-03-11T21:28:36.584 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:28:36 smithi092 ceph-mon[29572]: 5.17 scrub ok 2024-03-11T21:28:36.584 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:28:36 smithi092 ceph-mon[29572]: from='mgr.14607 ' entity='mgr.y' 2024-03-11T21:28:36.584 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:28:36 smithi092 ceph-mon[29572]: from='mgr.14607 v1:172.21.15.92:0/1702717597' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T21:28:36.584 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:28:36 smithi092 ceph-mon[37563]: pgmap v92: 148 pgs: 148 active+clean; 4.8 GiB data, 15 GiB used, 701 GiB / 715 GiB avail; 862 B/s rd, 60 MiB/s wr, 963 op/s 2024-03-11T21:28:36.584 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:28:36 smithi092 ceph-mon[37563]: 4.1c scrub starts 2024-03-11T21:28:36.584 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:28:36 smithi092 ceph-mon[37563]: 4.1c scrub ok 2024-03-11T21:28:36.584 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:28:36 smithi092 ceph-mon[37563]: 5.17 scrub starts 2024-03-11T21:28:36.585 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:28:36 smithi092 ceph-mon[37563]: 5.17 scrub ok 2024-03-11T21:28:36.585 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:28:36 smithi092 ceph-mon[37563]: from='mgr.14607 ' entity='mgr.y' 2024-03-11T21:28:36.585 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:28:36 smithi092 ceph-mon[37563]: from='mgr.14607 v1:172.21.15.92:0/1702717597' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T21:28:36.630 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:28:36.638 INFO:tasks.radosbench.radosbench.0.smithi092.stdout: 52 16 83767 83751 100.623 71.0625 0.00928358 0.00993344 2024-03-11T21:28:36.661 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:28:36.661 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:28:36.662 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:28:36.763 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:28:36.799 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:28:36.800 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:28:36.801 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:28:36.901 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:28:36.930 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:28:36.930 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:28:36.930 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:28:37.031 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:28:37.072 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:28:37.073 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:28:37.073 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:28:37.174 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:28:37.202 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:28:37.203 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:28:37.203 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:28:37.304 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:28:37.373 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:28:37.374 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:28:37.375 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:28:37.475 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:28:37.501 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:28:37.502 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:28:37.503 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:28:37.563 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:28:37 smithi092 ceph-mon[29572]: 2.5 scrub starts 2024-03-11T21:28:37.563 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:28:37 smithi092 ceph-mon[29572]: 2.5 scrub ok 2024-03-11T21:28:37.563 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:28:37 smithi092 ceph-mon[29572]: from='mgr.14607 ' entity='mgr.y' 2024-03-11T21:28:37.563 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:28:37 smithi092 ceph-mon[29572]: from='mgr.14607 ' entity='mgr.y' 2024-03-11T21:28:37.563 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:28:37 smithi092 ceph-mon[37563]: 2.5 scrub starts 2024-03-11T21:28:37.563 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:28:37 smithi092 ceph-mon[37563]: 2.5 scrub ok 2024-03-11T21:28:37.564 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:28:37 smithi092 ceph-mon[37563]: from='mgr.14607 ' entity='mgr.y' 2024-03-11T21:28:37.564 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:28:37 smithi092 ceph-mon[37563]: from='mgr.14607 ' entity='mgr.y' 2024-03-11T21:28:37.576 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:28:37 smithi097 ceph-mon[33317]: 2.5 scrub starts 2024-03-11T21:28:37.576 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:28:37 smithi097 ceph-mon[33317]: 2.5 scrub ok 2024-03-11T21:28:37.576 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:28:37 smithi097 ceph-mon[33317]: from='mgr.14607 ' entity='mgr.y' 2024-03-11T21:28:37.577 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:28:37 smithi097 ceph-mon[33317]: from='mgr.14607 ' entity='mgr.y' 2024-03-11T21:28:37.580 INFO:teuthology.orchestra.run.smithi092.stderr:Inferring config /var/lib/ceph/00db0016-dfed-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T21:28:37.604 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:28:37.639 INFO:tasks.radosbench.radosbench.0.smithi092.stdout: 53 15 84756 84741 99.8916 61.875 0.0159582 0.0100033 2024-03-11T21:28:37.670 INFO:teuthology.orchestra.run.smithi092.stdout:ERROR: (22) Invalid argument 2024-03-11T21:28:37.670 INFO:teuthology.orchestra.run.smithi092.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-11T21:28:37.680 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:28:37.681 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:28:37.681 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:28:37.782 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:28:37.809 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:28:37.810 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:28:37.810 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:28:37.911 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:28:37.938 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:28:37.939 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:28:37.940 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:28:38.041 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:28:38.087 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:28:38.087 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:28:38.088 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:28:38.188 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:28:38.231 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:28:38.231 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:28:38.232 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:28:38.240 INFO:teuthology.orchestra.run.smithi092.stderr:noscrub is set 2024-03-11T21:28:38.241 INFO:journalctl@ceph.mgr.y.smithi092.stdout:Mar 11 21:28:37 smithi092 ceph-00db0016-dfed-11ee-95c8-87774f69a715-mgr-y[29801]: ::ffff:172.21.15.97 - - [11/Mar/2024:21:28:37] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-11T21:28:38.333 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:28:38.380 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:28:38.383 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:28:38.384 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:28:38.485 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:28:38.514 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:28:38.515 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:28:38.515 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:28:38.577 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:28:38 smithi097 ceph-mon[33317]: 2.b scrub starts 2024-03-11T21:28:38.577 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:28:38 smithi097 ceph-mon[33317]: 2.b scrub ok 2024-03-11T21:28:38.577 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:28:38 smithi097 ceph-mon[33317]: 4.c scrub starts 2024-03-11T21:28:38.577 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:28:38 smithi097 ceph-mon[33317]: 4.c scrub ok 2024-03-11T21:28:38.577 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:28:38 smithi097 ceph-mon[33317]: pgmap v93: 148 pgs: 148 active+clean; 4.9 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 852 B/s rd, 71 MiB/s wr, 1.13k op/s 2024-03-11T21:28:38.577 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:28:38 smithi097 ceph-mon[33317]: 2.10 scrub starts 2024-03-11T21:28:38.577 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:28:38 smithi097 ceph-mon[33317]: 2.10 scrub ok 2024-03-11T21:28:38.577 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:28:38 smithi097 ceph-mon[33317]: 4.19 scrub starts 2024-03-11T21:28:38.577 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:28:38 smithi097 ceph-mon[33317]: 4.19 scrub ok 2024-03-11T21:28:38.578 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:28:38 smithi097 ceph-mon[33317]: 2.16 scrub starts 2024-03-11T21:28:38.578 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:28:38 smithi097 ceph-mon[33317]: 2.16 scrub ok 2024-03-11T21:28:38.578 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:28:38 smithi097 ceph-mon[33317]: from='client.? v1:172.21.15.92:0/2291984314' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-11T21:28:38.578 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:28:38 smithi097 ceph-mon[33317]: from='client.24793 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-11T21:28:38.578 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:28:38 smithi097 ceph-mon[33317]: from='mgr.14607 v1:172.21.15.92:0/1702717597' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T21:28:38.578 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:28:38 smithi097 ceph-mon[33317]: from='mgr.14607 v1:172.21.15.92:0/1702717597' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T21:28:38.578 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:28:38 smithi097 ceph-mon[33317]: from='mgr.14607 ' entity='mgr.y' 2024-03-11T21:28:38.611 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:28:38 smithi092 ceph-mon[29572]: 2.b scrub starts 2024-03-11T21:28:38.611 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:28:38 smithi092 ceph-mon[29572]: 2.b scrub ok 2024-03-11T21:28:38.611 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:28:38 smithi092 ceph-mon[29572]: 4.c scrub starts 2024-03-11T21:28:38.611 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:28:38 smithi092 ceph-mon[29572]: 4.c scrub ok 2024-03-11T21:28:38.611 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:28:38 smithi092 ceph-mon[29572]: pgmap v93: 148 pgs: 148 active+clean; 4.9 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 852 B/s rd, 71 MiB/s wr, 1.13k op/s 2024-03-11T21:28:38.611 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:28:38 smithi092 ceph-mon[29572]: 2.10 scrub starts 2024-03-11T21:28:38.611 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:28:38 smithi092 ceph-mon[29572]: 2.10 scrub ok 2024-03-11T21:28:38.612 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:28:38 smithi092 ceph-mon[29572]: 4.19 scrub starts 2024-03-11T21:28:38.612 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:28:38 smithi092 ceph-mon[29572]: 4.19 scrub ok 2024-03-11T21:28:38.612 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:28:38 smithi092 ceph-mon[29572]: 2.16 scrub starts 2024-03-11T21:28:38.612 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:28:38 smithi092 ceph-mon[29572]: 2.16 scrub ok 2024-03-11T21:28:38.612 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:28:38 smithi092 ceph-mon[29572]: from='client.? v1:172.21.15.92:0/2291984314' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-11T21:28:38.612 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:28:38 smithi092 ceph-mon[29572]: from='client.24793 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-11T21:28:38.612 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:28:38 smithi092 ceph-mon[29572]: from='mgr.14607 v1:172.21.15.92:0/1702717597' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T21:28:38.612 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:28:38 smithi092 ceph-mon[29572]: from='mgr.14607 v1:172.21.15.92:0/1702717597' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T21:28:38.612 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:28:38 smithi092 ceph-mon[29572]: from='mgr.14607 ' entity='mgr.y' 2024-03-11T21:28:38.612 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:28:38 smithi092 ceph-mon[37563]: 2.b scrub starts 2024-03-11T21:28:38.612 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:28:38 smithi092 ceph-mon[37563]: 2.b scrub ok 2024-03-11T21:28:38.613 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:28:38 smithi092 ceph-mon[37563]: 4.c scrub starts 2024-03-11T21:28:38.613 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:28:38 smithi092 ceph-mon[37563]: 4.c scrub ok 2024-03-11T21:28:38.613 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:28:38 smithi092 ceph-mon[37563]: pgmap v93: 148 pgs: 148 active+clean; 4.9 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 852 B/s rd, 71 MiB/s wr, 1.13k op/s 2024-03-11T21:28:38.613 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:28:38 smithi092 ceph-mon[37563]: 2.10 scrub starts 2024-03-11T21:28:38.613 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:28:38 smithi092 ceph-mon[37563]: 2.10 scrub ok 2024-03-11T21:28:38.613 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:28:38 smithi092 ceph-mon[37563]: 4.19 scrub starts 2024-03-11T21:28:38.613 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:28:38 smithi092 ceph-mon[37563]: 4.19 scrub ok 2024-03-11T21:28:38.613 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:28:38 smithi092 ceph-mon[37563]: 2.16 scrub starts 2024-03-11T21:28:38.613 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:28:38 smithi092 ceph-mon[37563]: 2.16 scrub ok 2024-03-11T21:28:38.613 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:28:38 smithi092 ceph-mon[37563]: from='client.? v1:172.21.15.92:0/2291984314' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-11T21:28:38.613 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:28:38 smithi092 ceph-mon[37563]: from='client.24793 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-11T21:28:38.613 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:28:38 smithi092 ceph-mon[37563]: from='mgr.14607 v1:172.21.15.92:0/1702717597' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T21:28:38.613 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:28:38 smithi092 ceph-mon[37563]: from='mgr.14607 v1:172.21.15.92:0/1702717597' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T21:28:38.614 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:28:38 smithi092 ceph-mon[37563]: from='mgr.14607 ' entity='mgr.y' 2024-03-11T21:28:38.616 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:28:38.639 INFO:tasks.radosbench.radosbench.0.smithi092.stdout: 54 16 85728 85712 99.1657 60.6875 0.0104192 0.0100781 2024-03-11T21:28:38.659 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:28:38.659 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:28:38.660 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:28:38.760 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:28:38.804 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:28:38.804 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:28:38.805 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:28:38.906 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:28:38.953 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:28:38.953 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:28:38.953 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:28:39.054 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:28:39.084 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:28:39.085 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:28:39.085 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:28:39.186 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:28:39.232 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:28:39.232 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:28:39.233 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:28:39.333 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:28:39.398 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:28:39.398 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:28:39.399 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:28:39.500 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:28:39.547 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:28:39.548 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:28:39.548 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:28:39.577 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:28:39 smithi097 ceph-mon[33317]: 6.8 deep-scrub starts 2024-03-11T21:28:39.577 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:28:39 smithi097 ceph-mon[33317]: 6.8 deep-scrub ok 2024-03-11T21:28:39.577 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:28:39 smithi097 ceph-mon[33317]: purged_snaps scrub starts 2024-03-11T21:28:39.577 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:28:39 smithi097 ceph-mon[33317]: purged_snaps scrub ok 2024-03-11T21:28:39.577 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:28:39 smithi097 ceph-mon[33317]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-03-11T21:28:39.577 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:28:39 smithi097 ceph-mon[33317]: from='client.24793 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-03-11T21:28:39.577 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:28:39 smithi097 ceph-mon[33317]: osdmap e110: 8 total, 8 up, 8 in 2024-03-11T21:28:39.577 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:28:39 smithi097 ceph-mon[33317]: Reconfiguring alertmanager.a (dependencies changed)... 2024-03-11T21:28:39.577 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:28:39 smithi097 ceph-mon[33317]: Reconfiguring daemon alertmanager.a on smithi092 2024-03-11T21:28:39.582 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:28:39 smithi092 ceph-mon[29572]: 6.8 deep-scrub starts 2024-03-11T21:28:39.582 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:28:39 smithi092 ceph-mon[29572]: 6.8 deep-scrub ok 2024-03-11T21:28:39.583 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:28:39 smithi092 ceph-mon[29572]: purged_snaps scrub starts 2024-03-11T21:28:39.583 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:28:39 smithi092 ceph-mon[29572]: purged_snaps scrub ok 2024-03-11T21:28:39.583 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:28:39 smithi092 ceph-mon[29572]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-03-11T21:28:39.583 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:28:39 smithi092 ceph-mon[29572]: from='client.24793 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-03-11T21:28:39.583 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:28:39 smithi092 ceph-mon[29572]: osdmap e110: 8 total, 8 up, 8 in 2024-03-11T21:28:39.583 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:28:39 smithi092 ceph-mon[29572]: Reconfiguring alertmanager.a (dependencies changed)... 2024-03-11T21:28:39.583 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:28:39 smithi092 ceph-mon[29572]: Reconfiguring daemon alertmanager.a on smithi092 2024-03-11T21:28:39.583 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:28:39 smithi092 ceph-mon[37563]: 6.8 deep-scrub starts 2024-03-11T21:28:39.583 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:28:39 smithi092 ceph-mon[37563]: 6.8 deep-scrub ok 2024-03-11T21:28:39.583 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:28:39 smithi092 ceph-mon[37563]: purged_snaps scrub starts 2024-03-11T21:28:39.583 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:28:39 smithi092 ceph-mon[37563]: purged_snaps scrub ok 2024-03-11T21:28:39.583 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:28:39 smithi092 ceph-mon[37563]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-03-11T21:28:39.584 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:28:39 smithi092 ceph-mon[37563]: from='client.24793 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-03-11T21:28:39.584 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:28:39 smithi092 ceph-mon[37563]: osdmap e110: 8 total, 8 up, 8 in 2024-03-11T21:28:39.584 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:28:39 smithi092 ceph-mon[37563]: Reconfiguring alertmanager.a (dependencies changed)... 2024-03-11T21:28:39.584 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:28:39 smithi092 ceph-mon[37563]: Reconfiguring daemon alertmanager.a on smithi092 2024-03-11T21:28:39.639 INFO:tasks.radosbench.radosbench.0.smithi092.stdout: 55 16 86791 86775 98.5707 66.4375 0.0115049 0.0101399 2024-03-11T21:28:39.650 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:28:39.689 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:28:39.690 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:28:39.690 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:28:39.791 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:28:39.833 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:28:39.834 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:28:39.834 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:28:39.935 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:28:39.966 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:28:39.967 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:28:39.968 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:28:40.068 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:28:40.110 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:28:40.111 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:28:40.111 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:28:40.212 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:28:40.256 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:28:40.257 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:28:40.257 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:28:40.358 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:28:40.388 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:28:40.388 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:28:40.389 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:28:40.415 DEBUG:teuthology.orchestra.run.smithi092:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid 00db0016-dfed-11ee-95c8-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-03-11T21:28:40.456 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:28:40 smithi092 ceph-mon[29572]: pgmap v95: 148 pgs: 148 active+clean; 5.1 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 1023 B/s rd, 71 MiB/s wr, 1.14k op/s 2024-03-11T21:28:40.456 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:28:40 smithi092 ceph-mon[37563]: pgmap v95: 148 pgs: 148 active+clean; 5.1 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 1023 B/s rd, 71 MiB/s wr, 1.14k op/s 2024-03-11T21:28:40.490 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:28:40.547 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:28:40.547 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:28:40.547 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:28:40.576 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:28:40 smithi097 ceph-mon[33317]: pgmap v95: 148 pgs: 148 active+clean; 5.1 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 1023 B/s rd, 71 MiB/s wr, 1.14k op/s 2024-03-11T21:28:40.639 INFO:tasks.radosbench.radosbench.0.smithi092.stdout: 56 16 87891 87875 98.0382 68.75 0.0143682 0.0101952 2024-03-11T21:28:40.649 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:28:40.676 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:28:40.677 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:28:40.677 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:28:40.778 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:28:40.820 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:28:40.820 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:28:40.821 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:28:40.922 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:28:40.967 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:28:40.967 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:28:40.967 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:28:41.068 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:28:41.109 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:28:41.109 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:28:41.109 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:28:41.211 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:28:41.251 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:28:41.251 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:28:41.252 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:28:41.319 DEBUG:teuthology.orchestra.run.smithi092:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid 00db0016-dfed-11ee-95c8-87774f69a715 -- ceph osd set nodeep-scrub 2024-03-11T21:28:41.353 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:28:41.383 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:28:41.383 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:28:41.383 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:28:41.484 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:28:41.512 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:28:41.513 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:28:41.513 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:28:41.614 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:28:41.639 INFO:tasks.radosbench.radosbench.0.smithi092.stdout: 57 16 88990 88974 97.5233 68.6875 0.0128555 0.0102491 2024-03-11T21:28:41.660 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:28:41.661 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:28:41.661 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:28:41.762 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:28:41.804 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:28:41.805 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:28:41.805 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:28:41.885 DEBUG:teuthology.orchestra.run.smithi092:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid 00db0016-dfed-11ee-95c8-87774f69a715 -- ceph daemon osd.2 dump_ops_in_flight 2024-03-11T21:28:41.906 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:28:41.933 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:28:41.934 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:28:41.934 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:28:42.035 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:28:42.128 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:28:42.129 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:28:42.129 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:28:42.230 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:28:42.278 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:28:42.279 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:28:42.279 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:28:42.326 INFO:journalctl@ceph.iscsi.iscsi.a.smithi097.stdout:Mar 11 21:28:42 smithi097 ceph-00db0016-dfed-11ee-95c8-87774f69a715-iscsi-iscsi-a[60943]: debug there is no tcmu-runner data available 2024-03-11T21:28:42.327 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:28:42 smithi097 ceph-mon[33317]: pgmap v96: 148 pgs: 148 active+clean; 5.1 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 1023 B/s rd, 71 MiB/s wr, 1.14k op/s 2024-03-11T21:28:42.380 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:28:42.410 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:28:42.410 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:28:42.411 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:28:42.511 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:28:42.556 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:28:42.556 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:28:42.556 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:28:42.589 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:28:42 smithi092 ceph-mon[29572]: pgmap v96: 148 pgs: 148 active+clean; 5.1 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 1023 B/s rd, 71 MiB/s wr, 1.14k op/s 2024-03-11T21:28:42.590 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:28:42 smithi092 ceph-mon[37563]: pgmap v96: 148 pgs: 148 active+clean; 5.1 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 1023 B/s rd, 71 MiB/s wr, 1.14k op/s 2024-03-11T21:28:42.639 INFO:tasks.radosbench.radosbench.0.smithi092.stdout: 58 16 90039 90023 96.9724 65.5625 0.0154346 0.0103057 2024-03-11T21:28:42.657 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:28:42.722 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:28:42.723 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:28:42.723 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:28:42.824 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:28:42.854 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:28:42.855 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:28:42.855 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:28:42.956 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:28:42.987 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:28:42.987 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:28:42.988 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:28:43.089 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:28:43.120 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:28:43.121 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:28:43.121 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:28:43.222 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:28:43.272 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:28:43.273 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:28:43.274 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:28:43.374 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:28:43.401 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:28:43.402 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:28:43.402 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:28:43.504 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:28:43.532 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:28:43.533 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:28:43.533 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:28:43.576 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:28:43 smithi097 ceph-mon[33317]: from='client.24469 v1:172.21.15.97:0/404208310' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:28:43.611 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:28:43 smithi092 ceph-mon[29572]: from='client.24469 v1:172.21.15.97:0/404208310' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:28:43.611 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:28:43 smithi092 ceph-mon[37563]: from='client.24469 v1:172.21.15.97:0/404208310' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:28:43.634 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:28:43.639 INFO:tasks.radosbench.radosbench.0.smithi092.stdout: 59 16 91137 91121 96.4917 68.625 0.0127465 0.0103583 2024-03-11T21:28:43.662 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:28:43.663 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:28:43.663 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:28:43.725 INFO:teuthology.orchestra.run.smithi092.stderr:Inferring config /var/lib/ceph/00db0016-dfed-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T21:28:43.730 INFO:teuthology.orchestra.run.smithi092.stderr:Inferring config /var/lib/ceph/00db0016-dfed-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T21:28:43.764 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:28:43.835 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:28:43.836 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:28:43.836 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:28:43.937 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:28:43.966 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:28:43.967 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:28:43.967 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:28:44.068 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:28:44.096 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:28:44.097 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:28:44.097 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:28:44.198 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:28:44.227 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:28:44.227 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:28:44.228 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:28:44.316 INFO:teuthology.orchestra.run.smithi092.stderr:set 6.a pg_upmap mapping to [4,0,3] 2024-03-11T21:28:44.328 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:28:44.372 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:28:44.372 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:28:44.373 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:28:44.474 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:28:44.504 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:28:44.505 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:28:44.505 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:28:44.576 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:28:44 smithi097 ceph-mon[33317]: pgmap v97: 148 pgs: 148 active+clean; 5.5 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 1023 B/s rd, 68 MiB/s wr, 1.09k op/s 2024-03-11T21:28:44.577 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:28:44 smithi097 ceph-mon[33317]: from='client.? v1:172.21.15.92:0/596726611' entity='client.admin' cmd=[{"prefix": "osd pg-upmap", "pgid": "6.a", "id": [4, 0, 3]}]: dispatch 2024-03-11T21:28:44.606 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:28:44.610 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:28:44 smithi092 ceph-mon[29572]: pgmap v97: 148 pgs: 148 active+clean; 5.5 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 1023 B/s rd, 68 MiB/s wr, 1.09k op/s 2024-03-11T21:28:44.611 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:28:44 smithi092 ceph-mon[29572]: from='client.? v1:172.21.15.92:0/596726611' entity='client.admin' cmd=[{"prefix": "osd pg-upmap", "pgid": "6.a", "id": [4, 0, 3]}]: dispatch 2024-03-11T21:28:44.611 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:28:44 smithi092 ceph-mon[37563]: pgmap v97: 148 pgs: 148 active+clean; 5.5 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 1023 B/s rd, 68 MiB/s wr, 1.09k op/s 2024-03-11T21:28:44.611 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:28:44 smithi092 ceph-mon[37563]: from='client.? v1:172.21.15.92:0/596726611' entity='client.admin' cmd=[{"prefix": "osd pg-upmap", "pgid": "6.a", "id": [4, 0, 3]}]: dispatch 2024-03-11T21:28:44.640 INFO:tasks.radosbench.radosbench.0.smithi092.stdout:2024-03-11T21:28:44.657943+0000 min lat: 0.00131184 max lat: 0.114779 avg lat: 0.0104244 2024-03-11T21:28:44.640 INFO:tasks.radosbench.radosbench.0.smithi092.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-03-11T21:28:44.640 INFO:tasks.radosbench.radosbench.0.smithi092.stdout: 60 16 92096 92080 95.8825 59.9375 0.0200657 0.0104244 2024-03-11T21:28:44.644 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:28:44.645 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:28:44.645 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:28:44.746 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:28:44.790 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:28:44.791 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:28:44.791 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:28:44.892 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:28:44.925 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:28:44.925 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:28:44.925 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:28:45.026 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:28:45.058 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:28:45.059 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:28:45.059 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:28:45.160 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:28:45.188 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:28:45.189 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:28:45.189 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:28:45.290 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:28:45.320 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:28:45.321 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:28:45.322 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:28:45.423 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:28:45.452 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:28:45.452 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:28:45.453 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:28:45.554 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:28:45.589 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:28:45.590 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:28:45.590 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:28:45.611 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:28:45 smithi092 ceph-mon[29572]: from='client.? v1:172.21.15.92:0/596726611' entity='client.admin' cmd='[{"prefix": "osd pg-upmap", "pgid": "6.a", "id": [4, 0, 3]}]': finished 2024-03-11T21:28:45.611 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:28:45 smithi092 ceph-mon[29572]: osdmap e111: 8 total, 8 up, 8 in 2024-03-11T21:28:45.611 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:28:45 smithi092 ceph-mon[37563]: from='client.? v1:172.21.15.92:0/596726611' entity='client.admin' cmd='[{"prefix": "osd pg-upmap", "pgid": "6.a", "id": [4, 0, 3]}]': finished 2024-03-11T21:28:45.611 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:28:45 smithi092 ceph-mon[37563]: osdmap e111: 8 total, 8 up, 8 in 2024-03-11T21:28:45.640 INFO:tasks.radosbench.radosbench.0.smithi092.stdout: 61 16 93161 93145 95.4019 66.5625 0.00952994 0.0104767 2024-03-11T21:28:45.691 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:28:45.722 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:28:45.722 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:28:45.723 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:28:45.823 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:28:45.826 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:28:45 smithi097 ceph-mon[33317]: from='client.? v1:172.21.15.92:0/596726611' entity='client.admin' cmd='[{"prefix": "osd pg-upmap", "pgid": "6.a", "id": [4, 0, 3]}]': finished 2024-03-11T21:28:45.827 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:28:45 smithi097 ceph-mon[33317]: osdmap e111: 8 total, 8 up, 8 in 2024-03-11T21:28:45.869 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:28:45.869 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:28:45.870 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:28:45.970 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:28:46.009 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:28:46.010 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:28:46.010 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:28:46.111 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:28:46.135 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:28:46.135 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:28:46.135 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:28:46.236 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:28:46.265 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:28:46.266 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:28:46.266 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:28:46.367 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:28:46.412 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:28:46.415 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:28:46.416 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:28:46.516 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:28:46.543 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:28:46.545 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:28:46.545 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:28:46.611 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:28:46 smithi092 ceph-mon[29572]: pgmap v99: 148 pgs: 148 active+clean; 5.5 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 1.2 KiB/s rd, 68 MiB/s wr, 1.09k op/s 2024-03-11T21:28:46.611 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:28:46 smithi092 ceph-mon[29572]: osdmap e112: 8 total, 8 up, 8 in 2024-03-11T21:28:46.611 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:28:46 smithi092 ceph-mon[29572]: purged_snaps scrub starts 2024-03-11T21:28:46.611 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:28:46 smithi092 ceph-mon[29572]: purged_snaps scrub ok 2024-03-11T21:28:46.611 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:28:46 smithi092 ceph-mon[37563]: pgmap v99: 148 pgs: 148 active+clean; 5.5 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 1.2 KiB/s rd, 68 MiB/s wr, 1.09k op/s 2024-03-11T21:28:46.611 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:28:46 smithi092 ceph-mon[37563]: osdmap e112: 8 total, 8 up, 8 in 2024-03-11T21:28:46.611 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:28:46 smithi092 ceph-mon[37563]: purged_snaps scrub starts 2024-03-11T21:28:46.612 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:28:46 smithi092 ceph-mon[37563]: purged_snaps scrub ok 2024-03-11T21:28:46.640 INFO:tasks.radosbench.radosbench.0.smithi092.stdout: 62 16 94012 93996 94.7211 53.1875 0.00530109 0.0105529 2024-03-11T21:28:46.646 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:28:46.694 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:28:46.695 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:28:46.695 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:28:46.796 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:28:46.820 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:28:46.820 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:28:46.820 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:28:46.826 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:28:46 smithi097 ceph-mon[33317]: pgmap v99: 148 pgs: 148 active+clean; 5.5 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 1.2 KiB/s rd, 68 MiB/s wr, 1.09k op/s 2024-03-11T21:28:46.827 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:28:46 smithi097 ceph-mon[33317]: osdmap e112: 8 total, 8 up, 8 in 2024-03-11T21:28:46.827 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:28:46 smithi097 ceph-mon[33317]: purged_snaps scrub starts 2024-03-11T21:28:46.827 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:28:46 smithi097 ceph-mon[33317]: purged_snaps scrub ok 2024-03-11T21:28:46.921 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:28:46.951 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:28:46.952 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:28:46.952 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:28:47.053 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:28:47.087 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:28:47.088 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:28:47.088 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:28:47.189 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:28:47.213 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:28:47.213 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:28:47.214 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:28:47.314 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:28:47.354 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:28:47.358 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:28:47.358 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:28:47.459 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:28:47.488 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:28:47.489 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:28:47.489 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:28:47.590 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:28:47.611 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:28:47 smithi092 ceph-mon[29572]: osdmap e113: 8 total, 8 up, 8 in 2024-03-11T21:28:47.611 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:28:47 smithi092 ceph-mon[37563]: osdmap e113: 8 total, 8 up, 8 in 2024-03-11T21:28:47.631 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:28:47.631 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:28:47.632 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:28:47.640 INFO:tasks.radosbench.radosbench.0.smithi092.stdout: 63 16 94970 94954 94.1681 59.875 0.00979671 0.0106144 2024-03-11T21:28:47.732 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:28:47.769 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:28:47.769 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:28:47.770 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:28:47.827 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:28:47 smithi097 ceph-mon[33317]: osdmap e113: 8 total, 8 up, 8 in 2024-03-11T21:28:47.870 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:28:47.899 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:28:47.899 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:28:47.899 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:28:48.000 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:28:48.027 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:28:48.028 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:28:48.028 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:28:48.129 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:28:48.178 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:28:48.179 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:28:48.179 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:28:48.280 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:28:48.310 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:28:48.311 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:28:48.312 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:28:48.361 INFO:journalctl@ceph.mgr.y.smithi092.stdout:Mar 11 21:28:47 smithi092 ceph-00db0016-dfed-11ee-95c8-87774f69a715-mgr-y[29801]: ::ffff:172.21.15.97 - - [11/Mar/2024:21:28:47] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-11T21:28:48.412 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:28:48.450 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:28:48.451 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:28:48.451 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:28:48.552 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:28:48.582 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:28:48.583 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:28:48.583 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:28:48.640 INFO:tasks.radosbench.radosbench.0.smithi092.stdout: 64 16 95872 95856 93.5777 56.375 0.00575453 0.0106809 2024-03-11T21:28:48.684 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:28:48.721 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:28:48.722 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:28:48.722 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:28:48.823 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:28:48.826 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:28:48 smithi097 ceph-mon[33317]: pgmap v102: 148 pgs: 148 active+clean; 5.6 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 853 B/s rd, 73 MiB/s wr, 1.17k op/s 2024-03-11T21:28:48.826 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:28:48 smithi097 ceph-mon[33317]: osdmap e114: 8 total, 8 up, 8 in 2024-03-11T21:28:48.853 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:28:48.854 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:28:48.855 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:28:48.860 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:28:48 smithi092 ceph-mon[29572]: pgmap v102: 148 pgs: 148 active+clean; 5.6 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 853 B/s rd, 73 MiB/s wr, 1.17k op/s 2024-03-11T21:28:48.861 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:28:48 smithi092 ceph-mon[29572]: osdmap e114: 8 total, 8 up, 8 in 2024-03-11T21:28:48.861 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:28:48 smithi092 ceph-mon[37563]: pgmap v102: 148 pgs: 148 active+clean; 5.6 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 853 B/s rd, 73 MiB/s wr, 1.17k op/s 2024-03-11T21:28:48.861 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:28:48 smithi092 ceph-mon[37563]: osdmap e114: 8 total, 8 up, 8 in 2024-03-11T21:28:48.955 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:28:48.981 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:28:48.981 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:28:48.982 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:28:49.082 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:28:49.123 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:28:49.124 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:28:49.124 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:28:49.225 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:28:49.269 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:28:49.270 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:28:49.270 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:28:49.371 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:28:49.402 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:28:49.403 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:28:49.403 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:28:49.504 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:28:49.547 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:28:49.547 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:28:49.547 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:28:49.640 INFO:tasks.radosbench.radosbench.0.smithi092.stdout: 65 16 97002 96986 93.2245 70.625 0.0106323 0.0107217 2024-03-11T21:28:49.648 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:28:49.680 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:28:49.681 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:28:49.681 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:28:49.753 INFO:teuthology.orchestra.run.smithi092.stderr:Inferring config /var/lib/ceph/00db0016-dfed-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T21:28:49.781 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:28:49.827 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:28:49.827 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:28:49.828 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:28:49.928 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:28:49.960 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:28:49.961 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:28:49.961 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:28:50.062 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:28:50.100 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:28:50.101 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:28:50.101 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:28:50.202 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:28:50.234 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:28:50.235 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:28:50.235 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:28:50.336 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:28:50.362 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:28:50.363 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:28:50.364 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:28:50.464 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:28:50.510 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:28:50.511 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:28:50.511 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:28:50.577 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:28:50 smithi092 ceph-mon[37563]: pgmap v104: 148 pgs: 148 active+clean; 5.8 GiB data, 18 GiB used, 698 GiB / 715 GiB avail; 1.1 KiB/s rd, 66 MiB/s wr, 1.06k op/s 2024-03-11T21:28:50.577 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:28:50 smithi092 ceph-mon[29572]: pgmap v104: 148 pgs: 148 active+clean; 5.8 GiB data, 18 GiB used, 698 GiB / 715 GiB avail; 1.1 KiB/s rd, 66 MiB/s wr, 1.06k op/s 2024-03-11T21:28:50.612 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:28:50.641 INFO:tasks.radosbench.radosbench.0.smithi092.stdout: 66 15 98108 98093 92.8603 69.1875 0.0201066 0.0107639 2024-03-11T21:28:50.643 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:28:50.643 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:28:50.644 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:28:50.745 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:28:50.775 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:28:50.775 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:28:50.776 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:28:50.827 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:28:50 smithi097 ceph-mon[33317]: pgmap v104: 148 pgs: 148 active+clean; 5.8 GiB data, 18 GiB used, 698 GiB / 715 GiB avail; 1.1 KiB/s rd, 66 MiB/s wr, 1.06k op/s 2024-03-11T21:28:50.876 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:28:50.910 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:28:50.910 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:28:50.911 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:28:51.011 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:28:51.046 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:28:51.047 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:28:51.047 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:28:51.148 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:28:51.192 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:28:51.193 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:28:51.193 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:28:51.294 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:28:51.344 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:28:51.344 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:28:51.345 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:28:51.445 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:28:51.481 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:28:51.482 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:28:51.482 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:28:51.583 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:28:51.611 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:28:51 smithi092 ceph-mon[29572]: purged_snaps scrub starts 2024-03-11T21:28:51.611 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:28:51 smithi092 ceph-mon[29572]: purged_snaps scrub ok 2024-03-11T21:28:51.611 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:28:51 smithi092 ceph-mon[29572]: osdmap e115: 8 total, 8 up, 8 in 2024-03-11T21:28:51.611 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:28:51 smithi092 ceph-mon[29572]: from='mgr.14607 ' entity='mgr.y' 2024-03-11T21:28:51.611 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:28:51 smithi092 ceph-mon[29572]: from='mgr.14607 v1:172.21.15.92:0/1702717597' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T21:28:51.612 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:28:51 smithi092 ceph-mon[37563]: purged_snaps scrub starts 2024-03-11T21:28:51.612 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:28:51 smithi092 ceph-mon[37563]: purged_snaps scrub ok 2024-03-11T21:28:51.612 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:28:51 smithi092 ceph-mon[37563]: osdmap e115: 8 total, 8 up, 8 in 2024-03-11T21:28:51.612 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:28:51 smithi092 ceph-mon[37563]: from='mgr.14607 ' entity='mgr.y' 2024-03-11T21:28:51.612 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:28:51 smithi092 ceph-mon[37563]: from='mgr.14607 v1:172.21.15.92:0/1702717597' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T21:28:51.612 INFO:journalctl@ceph.alertmanager.a.smithi092.stdout:Mar 11 21:28:51 smithi092 systemd[1]: Stopping Ceph alertmanager.a for 00db0016-dfed-11ee-95c8-87774f69a715... 2024-03-11T21:28:51.622 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:28:51.622 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:28:51.623 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:28:51.640 INFO:tasks.radosbench.radosbench.0.smithi092.stdout: 67 16 99194 99178 92.4865 67.8125 0.0180156 0.0108072 2024-03-11T21:28:51.724 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:28:51.752 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:28:51.753 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:28:51.754 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:28:51.826 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:28:51 smithi097 ceph-mon[33317]: purged_snaps scrub starts 2024-03-11T21:28:51.827 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:28:51 smithi097 ceph-mon[33317]: purged_snaps scrub ok 2024-03-11T21:28:51.827 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:28:51 smithi097 ceph-mon[33317]: osdmap e115: 8 total, 8 up, 8 in 2024-03-11T21:28:51.827 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:28:51 smithi097 ceph-mon[33317]: from='mgr.14607 ' entity='mgr.y' 2024-03-11T21:28:51.827 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:28:51 smithi097 ceph-mon[33317]: from='mgr.14607 v1:172.21.15.92:0/1702717597' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T21:28:51.854 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:28:51.887 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:28:51.887 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:28:51.888 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:28:51.988 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:28:52.021 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:28:52.021 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:28:52.022 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:28:52.122 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:28:52.163 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:28:52.164 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:28:52.164 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:28:52.265 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:28:52.311 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:28:52.311 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:28:52.312 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:28:52.406 INFO:journalctl@ceph.iscsi.iscsi.a.smithi097.stdout:Mar 11 21:28:52 smithi097 ceph-00db0016-dfed-11ee-95c8-87774f69a715-iscsi-iscsi-a[60943]: debug there is no tcmu-runner data available 2024-03-11T21:28:52.412 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:28:52.442 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:28:52.443 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:28:52.443 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:28:52.544 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:28:52.587 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:28:52.587 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:28:52.588 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:28:52.611 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:28:52 smithi092 ceph-mon[29572]: pgmap v106: 148 pgs: 148 active+clean; 5.8 GiB data, 18 GiB used, 698 GiB / 715 GiB avail; 905 B/s rd, 55 MiB/s wr, 873 op/s 2024-03-11T21:28:52.611 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:28:52 smithi092 ceph-mon[37563]: pgmap v106: 148 pgs: 148 active+clean; 5.8 GiB data, 18 GiB used, 698 GiB / 715 GiB avail; 905 B/s rd, 55 MiB/s wr, 873 op/s 2024-03-11T21:28:52.640 INFO:tasks.radosbench.radosbench.0.smithi092.stdout: 68 16 100261 100245 92.1071 66.6875 0.00660191 0.0108521 2024-03-11T21:28:52.689 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:28:52.722 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:28:52.723 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:28:52.723 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:28:52.824 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:28:52.826 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:28:52 smithi097 ceph-mon[33317]: pgmap v106: 148 pgs: 148 active+clean; 5.8 GiB data, 18 GiB used, 698 GiB / 715 GiB avail; 905 B/s rd, 55 MiB/s wr, 873 op/s 2024-03-11T21:28:52.853 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:28:52.854 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:28:52.855 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:28:52.956 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:28:52.995 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:28:52.996 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:28:52.996 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:28:53.097 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:28:53.124 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:28:53.125 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:28:53.125 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:28:53.226 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:28:53.267 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:28:53.267 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:28:53.267 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:28:53.368 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:28:53.411 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:28:53.412 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:28:53.412 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:28:53.513 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:28:53.553 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:28:53.554 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:28:53.554 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:28:53.641 INFO:tasks.radosbench.radosbench.0.smithi092.stdout: 69 16 101331 101315 91.741 66.875 0.00499482 0.0108952 2024-03-11T21:28:53.655 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:28:53.700 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:28:53.701 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:28:53.701 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:28:53.802 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:28:53.826 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:28:53 smithi097 ceph-mon[33317]: from='client.24469 v1:172.21.15.97:0/404208310' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:28:53.826 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:28:53 smithi097 ceph-mon[33317]: purged_snaps scrub starts 2024-03-11T21:28:53.827 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:28:53 smithi097 ceph-mon[33317]: purged_snaps scrub ok 2024-03-11T21:28:53.827 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:28:53 smithi097 ceph-mon[33317]: osdmap e116: 8 total, 8 up, 8 in 2024-03-11T21:28:53.847 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:28:53.847 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:28:53.848 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:28:53.861 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:28:53 smithi092 ceph-mon[29572]: from='client.24469 v1:172.21.15.97:0/404208310' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:28:53.861 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:28:53 smithi092 ceph-mon[29572]: purged_snaps scrub starts 2024-03-11T21:28:53.861 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:28:53 smithi092 ceph-mon[29572]: purged_snaps scrub ok 2024-03-11T21:28:53.861 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:28:53 smithi092 ceph-mon[29572]: osdmap e116: 8 total, 8 up, 8 in 2024-03-11T21:28:53.861 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:28:53 smithi092 ceph-mon[37563]: from='client.24469 v1:172.21.15.97:0/404208310' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:28:53.861 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:28:53 smithi092 ceph-mon[37563]: purged_snaps scrub starts 2024-03-11T21:28:53.861 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:28:53 smithi092 ceph-mon[37563]: purged_snaps scrub ok 2024-03-11T21:28:53.861 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:28:53 smithi092 ceph-mon[37563]: osdmap e116: 8 total, 8 up, 8 in 2024-03-11T21:28:53.949 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:28:53.980 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:28:53.981 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:28:53.981 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:28:54.082 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:28:54.118 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:28:54.119 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:28:54.119 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:28:54.220 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:28:54.272 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:28:54.273 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:28:54.273 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:28:54.374 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:28:54.422 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:28:54.423 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:28:54.423 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:28:54.471 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:28:54 smithi092 ceph-mon[29572]: pgmap v107: 148 pgs: 148 active+clean; 6.1 GiB data, 19 GiB used, 697 GiB / 715 GiB avail; 1.5 KiB/s rd, 96 MiB/s wr, 1.54k op/s 2024-03-11T21:28:54.472 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:28:54 smithi092 ceph-mon[37563]: pgmap v107: 148 pgs: 148 active+clean; 6.1 GiB data, 19 GiB used, 697 GiB / 715 GiB avail; 1.5 KiB/s rd, 96 MiB/s wr, 1.54k op/s 2024-03-11T21:28:54.524 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:28:54.552 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:28:54.552 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:28:54.553 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:28:54.641 INFO:tasks.radosbench.radosbench.0.smithi092.stdout: 70 16 102393 102377 91.3786 66.375 0.0133269 0.0109385 2024-03-11T21:28:54.653 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:28:54.698 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:28:54.699 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:28:54.699 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:28:54.800 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:28:54.826 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:28:54 smithi097 ceph-mon[33317]: pgmap v107: 148 pgs: 148 active+clean; 6.1 GiB data, 19 GiB used, 697 GiB / 715 GiB avail; 1.5 KiB/s rd, 96 MiB/s wr, 1.54k op/s 2024-03-11T21:28:54.828 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:28:54.829 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:28:54.829 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:28:54.930 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:28:54.973 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:28:54.973 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:28:54.974 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:28:55.074 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:28:55.117 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:28:55.118 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:28:55.119 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:28:55.219 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:28:55.247 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:28:55.247 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:28:55.248 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:28:55.349 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:28:55.361 INFO:journalctl@ceph.alertmanager.a.smithi092.stdout:Mar 11 21:28:55 smithi092 ceph-00db0016-dfed-11ee-95c8-87774f69a715-alertmanager-a[72702]: ts=2024-03-11T21:28:55.044Z caller=main.go:583 level=info msg="Received SIGTERM, exiting gracefully..." 2024-03-11T21:28:55.361 INFO:journalctl@ceph.alertmanager.a.smithi092.stdout:Mar 11 21:28:55 smithi092 podman[83335]: 2024-03-11 21:28:55.27988874 +0000 UTC m=+1.025100546 container died bea3b94d892511f16d09aff1f908794e43c75beb055ed367ef24c96249cb49df (image=quay.io/prometheus/alertmanager:v0.25.0, name=ceph-00db0016-dfed-11ee-95c8-87774f69a715-alertmanager-a, maintainer=The Prometheus Authors ) 2024-03-11T21:28:55.362 INFO:teuthology.orchestra.run.smithi092.stdout:ERROR: (22) Invalid argument 2024-03-11T21:28:55.362 INFO:teuthology.orchestra.run.smithi092.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-11T21:28:55.375 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:28:55.376 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:28:55.376 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:28:55.477 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:28:55.504 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:28:55.504 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:28:55.505 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:28:55.606 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:28:55.641 INFO:tasks.radosbench.radosbench.0.smithi092.stdout: 71 16 103276 103260 90.8689 55.1875 0.0239768 0.011 2024-03-11T21:28:55.647 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:28:55.647 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:28:55.648 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:28:55.749 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:28:55.787 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:28:55.787 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:28:55.787 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:28:55.888 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:28:55.925 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:28:55.926 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:28:55.926 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:28:56.027 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:28:56.071 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:28:56.072 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:28:56.072 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:28:56.111 INFO:journalctl@ceph.alertmanager.a.smithi092.stdout:Mar 11 21:28:55 smithi092 podman[83335]: 2024-03-11 21:28:55.728713669 +0000 UTC m=+1.473925466 container cleanup bea3b94d892511f16d09aff1f908794e43c75beb055ed367ef24c96249cb49df (image=quay.io/prometheus/alertmanager:v0.25.0, name=ceph-00db0016-dfed-11ee-95c8-87774f69a715-alertmanager-a, maintainer=The Prometheus Authors ) 2024-03-11T21:28:56.111 INFO:journalctl@ceph.alertmanager.a.smithi092.stdout:Mar 11 21:28:55 smithi092 bash[83335]: ceph-00db0016-dfed-11ee-95c8-87774f69a715-alertmanager-a 2024-03-11T21:28:56.173 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:28:56.213 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:28:56.214 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:28:56.214 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:28:56.315 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:28:56.343 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:28:56.344 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:28:56.344 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:28:56.444 INFO:journalctl@ceph.alertmanager.a.smithi092.stdout:Mar 11 21:28:56 smithi092 podman[83442]: 2024-03-11 21:28:56.182309819 +0000 UTC m=+1.076584854 container remove bea3b94d892511f16d09aff1f908794e43c75beb055ed367ef24c96249cb49df (image=quay.io/prometheus/alertmanager:v0.25.0, name=ceph-00db0016-dfed-11ee-95c8-87774f69a715-alertmanager-a, maintainer=The Prometheus Authors ) 2024-03-11T21:28:56.445 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:28:56.487 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:28:56.487 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:28:56.488 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:28:56.589 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:28:56.616 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:28:56.617 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:28:56.617 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:28:56.643 INFO:tasks.radosbench.radosbench.0.smithi092.stdout: 72 16 104165 104149 90.3763 55.5625 0.00481671 0.0110603 2024-03-11T21:28:56.707 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5, 6, 7] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7] 2024-03-11T21:28:56.708 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-03-11T21:28:56.708 INFO:tasks.thrashosds.thrasher:fixing pg num pool .mgr 2024-03-11T21:28:56.708 DEBUG:teuthology.orchestra.run.smithi092:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid 00db0016-dfed-11ee-95c8-87774f69a715 -- ceph pg dump --format=json 2024-03-11T21:28:56.709 INFO:teuthology.orchestra.run.smithi092.stdout:osd.0: {} 2024-03-11T21:28:56.709 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:28:56 smithi092 ceph-mon[29572]: pgmap v109: 148 pgs: 148 active+clean; 6.1 GiB data, 19 GiB used, 697 GiB / 715 GiB avail; 747 B/s rd, 48 MiB/s wr, 771 op/s 2024-03-11T21:28:56.709 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:28:56 smithi092 ceph-mon[29572]: purged_snaps scrub starts 2024-03-11T21:28:56.709 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:28:56 smithi092 ceph-mon[29572]: purged_snaps scrub ok 2024-03-11T21:28:56.710 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:28:56 smithi092 ceph-mon[37563]: pgmap v109: 148 pgs: 148 active+clean; 6.1 GiB data, 19 GiB used, 697 GiB / 715 GiB avail; 747 B/s rd, 48 MiB/s wr, 771 op/s 2024-03-11T21:28:56.710 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:28:56 smithi092 ceph-mon[37563]: purged_snaps scrub starts 2024-03-11T21:28:56.710 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:28:56 smithi092 ceph-mon[37563]: purged_snaps scrub ok 2024-03-11T21:28:56.718 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:28:56.744 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:28:56.745 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:28:56.745 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:28:56.763 INFO:teuthology.orchestra.run.smithi092.stdout:osd.1: {} 2024-03-11T21:28:56.789 INFO:teuthology.orchestra.run.smithi092.stdout:osd.2: {} 2024-03-11T21:28:56.827 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:28:56 smithi097 ceph-mon[33317]: pgmap v109: 148 pgs: 148 active+clean; 6.1 GiB data, 19 GiB used, 697 GiB / 715 GiB avail; 747 B/s rd, 48 MiB/s wr, 771 op/s 2024-03-11T21:28:56.827 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:28:56 smithi097 ceph-mon[33317]: purged_snaps scrub starts 2024-03-11T21:28:56.827 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:28:56 smithi097 ceph-mon[33317]: purged_snaps scrub ok 2024-03-11T21:28:56.846 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:28:56.860 INFO:teuthology.orchestra.run.smithi092.stdout:osd.3: {} 2024-03-11T21:28:56.870 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:28:56.871 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:28:56.871 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:28:56.902 INFO:teuthology.orchestra.run.smithi092.stdout:osd.4: {} 2024-03-11T21:28:56.941 INFO:teuthology.orchestra.run.smithi092.stdout:osd.5: {} 2024-03-11T21:28:56.973 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:28:56.982 INFO:teuthology.orchestra.run.smithi092.stdout:osd.6: {} 2024-03-11T21:28:56.997 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:28:56.998 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:28:56.998 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:28:57.016 INFO:teuthology.orchestra.run.smithi092.stdout:osd.7: {} 2024-03-11T21:28:57.024 INFO:teuthology.orchestra.run.smithi092.stderr:osd.0: osd_enable_op_tracker = '' 2024-03-11T21:28:57.024 INFO:teuthology.orchestra.run.smithi092.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-11T21:28:57.024 INFO:teuthology.orchestra.run.smithi092.stderr:osd.2: osd_enable_op_tracker = '' 2024-03-11T21:28:57.024 INFO:teuthology.orchestra.run.smithi092.stderr:osd.3: osd_enable_op_tracker = '' 2024-03-11T21:28:57.024 INFO:teuthology.orchestra.run.smithi092.stderr:osd.4: osd_enable_op_tracker = '' 2024-03-11T21:28:57.024 INFO:teuthology.orchestra.run.smithi092.stderr:osd.5: osd_enable_op_tracker = '' 2024-03-11T21:28:57.024 INFO:teuthology.orchestra.run.smithi092.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-11T21:28:57.024 INFO:teuthology.orchestra.run.smithi092.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-11T21:28:57.099 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:28:57.143 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:28:57.143 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:28:57.144 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:28:57.245 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:28:57.282 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:28:57.282 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:28:57.282 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:28:57.383 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:28:57.412 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:28:57.413 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:28:57.413 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:28:57.487 INFO:teuthology.orchestra.run.smithi092.stderr:nodeep-scrub is set 2024-03-11T21:28:57.514 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:28:57.543 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:28:57.544 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:28:57.544 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:28:57.643 INFO:tasks.radosbench.radosbench.0.smithi092.stdout: 73 16 105002 104986 89.855 52.3125 0.0089364 0.0111242 2024-03-11T21:28:57.645 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:28:57.676 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:28:57.677 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:28:57.677 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:28:57.778 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:28:57.814 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:28:57.815 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:28:57.815 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:28:57.826 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:28:57 smithi097 ceph-mon[33317]: osdmap e117: 8 total, 8 up, 8 in 2024-03-11T21:28:57.826 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:28:57 smithi097 ceph-mon[33317]: from='client.? v1:172.21.15.92:0/3432085845' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-11T21:28:57.826 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:28:57 smithi097 ceph-mon[33317]: from='client.? v1:172.21.15.92:0/1070920643' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-11T21:28:57.826 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:28:57 smithi097 ceph-mon[33317]: from='client.24805 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-11T21:28:57.861 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:28:57 smithi092 ceph-mon[29572]: osdmap e117: 8 total, 8 up, 8 in 2024-03-11T21:28:57.861 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:28:57 smithi092 ceph-mon[29572]: from='client.? v1:172.21.15.92:0/3432085845' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-11T21:28:57.861 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:28:57 smithi092 ceph-mon[29572]: from='client.? v1:172.21.15.92:0/1070920643' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-11T21:28:57.861 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:28:57 smithi092 ceph-mon[29572]: from='client.24805 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-11T21:28:57.861 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:28:57 smithi092 ceph-mon[37563]: osdmap e117: 8 total, 8 up, 8 in 2024-03-11T21:28:57.862 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:28:57 smithi092 ceph-mon[37563]: from='client.? v1:172.21.15.92:0/3432085845' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-11T21:28:57.862 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:28:57 smithi092 ceph-mon[37563]: from='client.? v1:172.21.15.92:0/1070920643' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-11T21:28:57.862 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:28:57 smithi092 ceph-mon[37563]: from='client.24805 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-11T21:28:57.862 INFO:journalctl@ceph.alertmanager.a.smithi092.stdout:Mar 11 21:28:57 smithi092 podman[83442]: 2024-03-11 21:28:57.463038324 +0000 UTC m=+2.357313357 volume remove 44b788cd7107f4e9f36d15ed43b805a5c6ef32d26dd6e7cd4bc7b825d0bb58f9 2024-03-11T21:28:57.916 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:28:57.959 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:28:57.960 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:28:57.960 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:28:58.061 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:28:58.093 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:28:58.094 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:28:58.094 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:28:58.195 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:28:58.226 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:28:58.227 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:28:58.227 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:28:58.286 INFO:journalctl@ceph.mgr.y.smithi092.stdout:Mar 11 21:28:57 smithi092 ceph-00db0016-dfed-11ee-95c8-87774f69a715-mgr-y[29801]: ::ffff:172.21.15.97 - - [11/Mar/2024:21:28:57] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-11T21:28:58.328 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:28:58.335 DEBUG:teuthology.orchestra.run.smithi092:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid 00db0016-dfed-11ee-95c8-87774f69a715 -- ceph daemon osd.2 dump_blocked_ops 2024-03-11T21:28:58.382 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:28:58.382 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:28:58.383 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:28:58.483 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:28:58.532 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:28:58.532 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:28:58.532 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:28:58.611 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:28:58 smithi092 ceph-mon[29572]: pgmap v111: 148 pgs: 148 active+clean; 6.2 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 773 B/s rd, 67 MiB/s wr, 1.07k op/s 2024-03-11T21:28:58.611 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:28:58 smithi092 ceph-mon[29572]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-11T21:28:58.611 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:28:58 smithi092 ceph-mon[29572]: from='client.24805 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-03-11T21:28:58.611 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:28:58 smithi092 ceph-mon[29572]: osdmap e118: 8 total, 8 up, 8 in 2024-03-11T21:28:58.611 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:28:58 smithi092 ceph-mon[29572]: osdmap e119: 8 total, 8 up, 8 in 2024-03-11T21:28:58.611 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:28:58 smithi092 ceph-mon[37563]: pgmap v111: 148 pgs: 148 active+clean; 6.2 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 773 B/s rd, 67 MiB/s wr, 1.07k op/s 2024-03-11T21:28:58.612 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:28:58 smithi092 ceph-mon[37563]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-11T21:28:58.612 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:28:58 smithi092 ceph-mon[37563]: from='client.24805 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-03-11T21:28:58.612 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:28:58 smithi092 ceph-mon[37563]: osdmap e118: 8 total, 8 up, 8 in 2024-03-11T21:28:58.612 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:28:58 smithi092 ceph-mon[37563]: osdmap e119: 8 total, 8 up, 8 in 2024-03-11T21:28:58.633 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:28:58.643 INFO:tasks.radosbench.radosbench.0.smithi092.stdout: 74 16 106066 106050 89.5394 66.5 0.0132335 0.0111631 2024-03-11T21:28:58.662 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:28:58.662 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:28:58.663 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:28:58.764 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:28:58.826 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:28:58 smithi097 ceph-mon[33317]: pgmap v111: 148 pgs: 148 active+clean; 6.2 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 773 B/s rd, 67 MiB/s wr, 1.07k op/s 2024-03-11T21:28:58.826 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:28:58 smithi097 ceph-mon[33317]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-11T21:28:58.827 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:28:58 smithi097 ceph-mon[33317]: from='client.24805 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-03-11T21:28:58.827 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:28:58 smithi097 ceph-mon[33317]: osdmap e118: 8 total, 8 up, 8 in 2024-03-11T21:28:58.827 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:28:58 smithi097 ceph-mon[33317]: osdmap e119: 8 total, 8 up, 8 in 2024-03-11T21:28:58.834 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:28:58.835 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:28:58.836 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:28:58.936 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:28:58.964 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:28:58.965 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:28:58.965 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:28:59.066 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:28:59.095 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:28:59.096 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:28:59.096 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:28:59.197 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:28:59.235 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:28:59.236 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:28:59.237 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:28:59.338 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:28:59.366 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:28:59.366 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:28:59.367 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:28:59.390 INFO:teuthology.orchestra.run.smithi092.stderr:Inferring config /var/lib/ceph/00db0016-dfed-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T21:28:59.394 INFO:teuthology.orchestra.run.smithi092.stderr:Inferring config /var/lib/ceph/00db0016-dfed-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T21:28:59.468 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:28:59.499 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:28:59.499 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:28:59.500 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:28:59.601 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:28:59.630 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:28:59.630 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:28:59.631 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:28:59.643 INFO:tasks.radosbench.radosbench.0.smithi092.stdout: 75 16 107086 107070 89.1955 63.75 0.00844344 0.0112059 2024-03-11T21:28:59.731 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:28:59.774 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:28:59.775 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:28:59.775 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:28:59.810 INFO:journalctl@ceph.alertmanager.a.smithi092.stdout:Mar 11 21:28:59 smithi092 systemd[1]: ceph-00db0016-dfed-11ee-95c8-87774f69a715@alertmanager.a.service: Deactivated successfully. 2024-03-11T21:28:59.810 INFO:journalctl@ceph.alertmanager.a.smithi092.stdout:Mar 11 21:28:59 smithi092 systemd[1]: Stopped Ceph alertmanager.a for 00db0016-dfed-11ee-95c8-87774f69a715. 2024-03-11T21:28:59.810 INFO:journalctl@ceph.alertmanager.a.smithi092.stdout:Mar 11 21:28:59 smithi092 systemd[1]: ceph-00db0016-dfed-11ee-95c8-87774f69a715@alertmanager.a.service: Consumed 1.184s CPU time. 2024-03-11T21:28:59.810 INFO:journalctl@ceph.alertmanager.a.smithi092.stdout:Mar 11 21:28:59 smithi092 systemd[1]: Starting Ceph alertmanager.a for 00db0016-dfed-11ee-95c8-87774f69a715... 2024-03-11T21:28:59.877 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:28:59.907 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:28:59.908 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:28:59.908 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:29:00.009 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:29:00.054 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:29:00.055 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:29:00.055 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:29:00.156 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:29:00.185 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:29:00.186 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:29:00.186 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:29:00.287 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:29:00.327 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:29:00.327 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:29:00.328 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:29:00.428 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:29:00.466 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:29:00.467 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:29:00.468 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:29:00.568 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:29:00.596 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:29:00.597 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:29:00.597 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:29:00.643 INFO:tasks.radosbench.radosbench.0.smithi092.stdout: 76 16 108152 108136 88.8985 66.625 0.0128459 0.0112438 2024-03-11T21:29:00.698 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:29:00.735 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:29:00.735 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:29:00.735 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:29:00.809 DEBUG:teuthology.orchestra.run.smithi092:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid 00db0016-dfed-11ee-95c8-87774f69a715 -- ceph osd unset noscrub 2024-03-11T21:29:00.826 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:29:00 smithi097 ceph-mon[33317]: pgmap v114: 148 pgs: 148 active+clean; 6.4 GiB data, 20 GiB used, 696 GiB / 715 GiB avail; 874 B/s rd, 51 MiB/s wr, 824 op/s 2024-03-11T21:29:00.837 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:29:00.849 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:29:00 smithi092 ceph-mon[29572]: pgmap v114: 148 pgs: 148 active+clean; 6.4 GiB data, 20 GiB used, 696 GiB / 715 GiB avail; 874 B/s rd, 51 MiB/s wr, 824 op/s 2024-03-11T21:29:00.849 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:29:00 smithi092 ceph-mon[37563]: pgmap v114: 148 pgs: 148 active+clean; 6.4 GiB data, 20 GiB used, 696 GiB / 715 GiB avail; 874 B/s rd, 51 MiB/s wr, 824 op/s 2024-03-11T21:29:00.863 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:29:00.864 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:29:00.864 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:29:00.965 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:29:01.017 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:29:01.018 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:29:01.018 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:29:01.119 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:29:01.168 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:29:01.169 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:29:01.169 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:29:01.270 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:29:01.333 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:29:01.334 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:29:01.335 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:29:01.338 DEBUG:teuthology.orchestra.run.smithi092:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid 00db0016-dfed-11ee-95c8-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-03-11T21:29:01.435 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:29:01.463 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:29:01.464 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:29:01.464 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:29:01.565 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:29:01.628 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:29:01.629 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:29:01.630 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:29:01.643 INFO:tasks.radosbench.radosbench.0.smithi092.stdout: 77 16 109140 109124 88.5454 61.75 0.0155118 0.0112891 2024-03-11T21:29:01.730 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:29:01.759 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:29:01.760 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:29:01.760 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:29:01.861 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:29:01.891 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:29:01.892 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:29:01.892 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:29:01.936 INFO:teuthology.orchestra.run.smithi092.stderr:Inferring config /var/lib/ceph/00db0016-dfed-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T21:29:01.993 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:29:02.022 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:29:02.023 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:29:02.023 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:29:02.124 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:29:02.152 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:29:02.153 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:29:02.153 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:29:02.254 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:29:02.289 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:29:02.289 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:29:02.290 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:29:02.384 INFO:journalctl@ceph.iscsi.iscsi.a.smithi097.stdout:Mar 11 21:29:02 smithi097 ceph-00db0016-dfed-11ee-95c8-87774f69a715-iscsi-iscsi-a[60943]: debug there is no tcmu-runner data available 2024-03-11T21:29:02.391 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:29:02.419 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:29:02.419 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:29:02.420 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:29:02.521 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:29:02.564 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:29:02.564 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:29:02.565 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:29:02.644 INFO:tasks.radosbench.radosbench.0.smithi092.stdout: 78 16 110156 110140 88.2243 63.5 0.0135736 0.0113293 2024-03-11T21:29:02.665 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:29:02.718 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:29:02 smithi092 ceph-mon[29572]: pgmap v115: 148 pgs: 148 active+clean; 6.4 GiB data, 20 GiB used, 696 GiB / 715 GiB avail; 853 B/s rd, 50 MiB/s wr, 804 op/s 2024-03-11T21:29:02.718 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:29:02 smithi092 ceph-mon[37563]: pgmap v115: 148 pgs: 148 active+clean; 6.4 GiB data, 20 GiB used, 696 GiB / 715 GiB avail; 853 B/s rd, 50 MiB/s wr, 804 op/s 2024-03-11T21:29:02.723 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:29:02.724 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:29:02.725 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:29:02.825 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:29:02.826 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:29:02 smithi097 ceph-mon[33317]: pgmap v115: 148 pgs: 148 active+clean; 6.4 GiB data, 20 GiB used, 696 GiB / 715 GiB avail; 853 B/s rd, 50 MiB/s wr, 804 op/s 2024-03-11T21:29:02.854 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:29:02.854 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:29:02.855 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:29:02.955 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:29:02.987 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:29:02.988 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:29:02.988 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:29:03.089 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:29:03.132 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:29:03.132 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:29:03.133 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:29:03.233 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:29:03.279 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:29:03.279 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:29:03.280 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:29:03.380 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:29:03.411 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:29:03.411 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:29:03.412 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:29:03.512 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:29:03.552 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:29:03.553 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:29:03.553 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:29:03.561 INFO:teuthology.orchestra.run.smithi092.stdout:{ 2024-03-11T21:29:03.562 INFO:teuthology.orchestra.run.smithi092.stdout: "ops": [], 2024-03-11T21:29:03.562 INFO:teuthology.orchestra.run.smithi092.stdout: "complaint_time": 30, 2024-03-11T21:29:03.562 INFO:teuthology.orchestra.run.smithi092.stdout: "num_blocked_ops": 0 2024-03-11T21:29:03.562 INFO:teuthology.orchestra.run.smithi092.stdout:} 2024-03-11T21:29:03.644 INFO:tasks.radosbench.radosbench.0.smithi092.stdout: 79 16 111181 111165 87.9185 64.0625 0.0084467 0.0113694 2024-03-11T21:29:03.654 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:29:03.696 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:29:03.697 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:29:03.697 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:29:03.798 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:29:03.826 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:29:03 smithi097 ceph-mon[33317]: from='client.24469 v1:172.21.15.97:0/404208310' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:29:03.827 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:29:03.828 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:29:03.828 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:29:03.860 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:29:03 smithi092 ceph-mon[29572]: from='client.24469 v1:172.21.15.97:0/404208310' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:29:03.861 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:29:03 smithi092 ceph-mon[37563]: from='client.24469 v1:172.21.15.97:0/404208310' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:29:03.929 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:29:03.958 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:29:03.959 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:29:03.959 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:29:04.060 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:29:04.098 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:29:04.099 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:29:04.099 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:29:04.200 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:29:04.201 DEBUG:teuthology.orchestra.run.smithi092:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid 00db0016-dfed-11ee-95c8-87774f69a715 -- ceph daemon osd.2 dump_historic_ops 2024-03-11T21:29:04.282 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:29:04.285 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:29:04.286 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:29:04.386 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:29:04.415 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:29:04.416 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:29:04.416 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:29:04.517 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:29:04.545 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:29:04.546 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:29:04.546 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:29:04.644 INFO:tasks.radosbench.radosbench.0.smithi092.stdout:2024-03-11T21:29:04.662279+0000 min lat: 0.00131184 max lat: 0.17729 avg lat: 0.0114048 2024-03-11T21:29:04.644 INFO:tasks.radosbench.radosbench.0.smithi092.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-03-11T21:29:04.644 INFO:tasks.radosbench.radosbench.0.smithi092.stdout: 80 16 112233 112217 87.6414 65.75 0.0227822 0.0114048 2024-03-11T21:29:04.647 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:29:04.674 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:29:04.675 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:29:04.675 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:29:04.776 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:29:04.821 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:29:04.821 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:29:04.822 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:29:04.826 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:29:04 smithi097 ceph-mon[33317]: pgmap v116: 148 pgs: 148 active+clean; 6.7 GiB data, 21 GiB used, 695 GiB / 715 GiB avail; 1.5 KiB/s rd, 95 MiB/s wr, 1.52k op/s 2024-03-11T21:29:04.861 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:29:04 smithi092 ceph-mon[29572]: pgmap v116: 148 pgs: 148 active+clean; 6.7 GiB data, 21 GiB used, 695 GiB / 715 GiB avail; 1.5 KiB/s rd, 95 MiB/s wr, 1.52k op/s 2024-03-11T21:29:04.861 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:29:04 smithi092 ceph-mon[37563]: pgmap v116: 148 pgs: 148 active+clean; 6.7 GiB data, 21 GiB used, 695 GiB / 715 GiB avail; 1.5 KiB/s rd, 95 MiB/s wr, 1.52k op/s 2024-03-11T21:29:04.922 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:29:04.984 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:29:04.985 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:29:04.985 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:29:05.086 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:29:05.116 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:29:05.116 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:29:05.116 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:29:05.217 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:29:05.261 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:29:05.262 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:29:05.262 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:29:05.363 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:29:05.377 INFO:teuthology.orchestra.run.smithi092.stderr:Inferring config /var/lib/ceph/00db0016-dfed-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T21:29:05.389 INFO:teuthology.orchestra.run.smithi092.stderr:Inferring config /var/lib/ceph/00db0016-dfed-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T21:29:05.448 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:29:05.449 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:29:05.450 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:29:05.550 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:29:05.580 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:29:05.580 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:29:05.581 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:29:05.644 INFO:tasks.radosbench.radosbench.0.smithi092.stdout: 81 16 113190 113174 87.2978 59.8125 0.00960256 0.0114495 2024-03-11T21:29:05.682 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:29:05.707 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:29:05.708 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:29:05.709 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:29:05.723 INFO:journalctl@ceph.alertmanager.a.smithi092.stdout:Mar 11 21:29:05 smithi092 podman[84650]: 2024-03-11 21:29:05.382361807 +0000 UTC m=+0.042306298 image pull c8568f914cd25b2062c44e9f79f9c18da6e3b85fe0c47a12a2191c61426c2b19 quay.io/prometheus/alertmanager:v0.25.0 2024-03-11T21:29:05.750 INFO:teuthology.orchestra.run.smithi092.stdout: 2024-03-11T21:29:05.764 INFO:teuthology.orchestra.run.smithi092.stderr:dumped all 2024-03-11T21:29:05.809 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:29:05.837 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:29:05.837 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:29:05.838 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:29:05.939 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:29:05.964 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:29:05.965 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:29:05.965 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:29:06.066 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:29:06.110 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:29:06.111 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:29:06.111 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:29:06.212 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:29:06.240 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:29:06.242 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:29:06.242 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:29:06.343 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:29:06.374 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:29:06.375 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:29:06.375 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:29:06.476 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:29:06.514 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:29:06.515 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:29:06.515 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:29:06.618 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:29:06.644 INFO:tasks.radosbench.radosbench.0.smithi092.stdout: 82 15 114239 114224 87.0335 65.625 0.0121187 0.0114852 2024-03-11T21:29:06.651 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:29:06.651 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:29:06.652 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:29:06.752 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:29:06.791 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:29:06.792 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:29:06.792 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:29:06.807 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:29:06 smithi092 ceph-mon[29572]: pgmap v117: 148 pgs: 148 active+clean; 6.7 GiB data, 21 GiB used, 695 GiB / 715 GiB avail; 1.2 KiB/s rd, 64 MiB/s wr, 1.02k op/s 2024-03-11T21:29:06.807 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:29:06 smithi092 ceph-mon[29572]: from='client.24811 v1:172.21.15.92:0/2915644090' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-11T21:29:06.807 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:29:06 smithi092 ceph-mon[29572]: from='mgr.14607 v1:172.21.15.92:0/1702717597' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T21:29:06.807 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:29:06 smithi092 ceph-mon[37563]: pgmap v117: 148 pgs: 148 active+clean; 6.7 GiB data, 21 GiB used, 695 GiB / 715 GiB avail; 1.2 KiB/s rd, 64 MiB/s wr, 1.02k op/s 2024-03-11T21:29:06.807 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:29:06 smithi092 ceph-mon[37563]: from='client.24811 v1:172.21.15.92:0/2915644090' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-11T21:29:06.807 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:29:06 smithi092 ceph-mon[37563]: from='mgr.14607 v1:172.21.15.92:0/1702717597' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T21:29:06.826 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:29:06 smithi097 ceph-mon[33317]: pgmap v117: 148 pgs: 148 active+clean; 6.7 GiB data, 21 GiB used, 695 GiB / 715 GiB avail; 1.2 KiB/s rd, 64 MiB/s wr, 1.02k op/s 2024-03-11T21:29:06.827 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:29:06 smithi097 ceph-mon[33317]: from='client.24811 v1:172.21.15.92:0/2915644090' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-11T21:29:06.827 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:29:06 smithi097 ceph-mon[33317]: from='mgr.14607 v1:172.21.15.92:0/1702717597' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T21:29:06.854 INFO:teuthology.orchestra.run.smithi092.stdout:{"pg_ready":true,"pg_map":{"version":117,"stamp":"2024-03-11T21:29:05.011384+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":7178754472,"num_objects":109728,"num_object_clones":0,"num_object_copies":329184,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":109728,"num_whiteouts":0,"num_read":1132,"num_read_kb":978,"num_write":110145,"num_write_kb":7012381,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":21,"num_bytes_recovered":1771324,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":89864,"ondisk_log_size":89864,"up":444,"acting":444,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":444,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":749961216,"kb_used":21532192,"kb_used_data":20924636,"kb_used_omap":12,"kb_used_meta":607475,"kb_avail":728429024,"statfs":{"total":767960285184,"available":745911320576,"internally_reserved":0,"allocated":21426827264,"data_stored":21423148128,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12752,"internal_metadata":622054960},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[3,7,4,9,5,2],"upper_bound":64},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":533725184,"num_objects":8144,"num_object_clones":0,"num_object_copies":24432,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8144,"num_whiteouts":0,"num_read":10,"num_read_kb":10,"num_write":8144,"num_write_kb":521216,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.002319"},"pg_stats":[{"pgid":"6.1b","version":"0'0","reported_seq":119,"reported_epoch":119,"state":"active+clean","last_fresh":"2024-03-11T21:28:58.170019+0000","last_change":"2024-03-11T21:28:16.650491+0000","last_active":"2024-03-11T21:28:58.170019+0000","last_peered":"2024-03-11T21:28:58.170019+0000","last_clean":"2024-03-11T21:28:58.170019+0000","last_became_active":"2024-03-11T21:25:45.302931+0000","last_became_peered":"2024-03-11T21:25:45.302931+0000","last_unstale":"2024-03-11T21:28:58.170019+0000","last_undegraded":"2024-03-11T21:28:58.170019+0000","last_fullsized":"2024-03-11T21:28:58.170019+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:28:16.650384+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:43.559452+0000","last_clean_scrub_stamp":"2024-03-11T21:28:16.650384+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:29:44.614978+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":116,"reported_epoch":119,"state":"active+clean","last_fresh":"2024-03-11T21:28:58.154770+0000","last_change":"2024-03-11T21:28:16.734432+0000","last_active":"2024-03-11T21:28:58.154770+0000","last_peered":"2024-03-11T21:28:58.154770+0000","last_clean":"2024-03-11T21:28:58.154770+0000","last_became_active":"2024-03-11T21:25:42.578893+0000","last_became_peered":"2024-03-11T21:25:42.578893+0000","last_unstale":"2024-03-11T21:28:58.154770+0000","last_undegraded":"2024-03-11T21:28:58.154770+0000","last_fullsized":"2024-03-11T21:28:58.154770+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:28:16.734315+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:41.544550+0000","last_clean_scrub_stamp":"2024-03-11T21:28:16.734315+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:29:17.932001+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1f","version":"0'0","reported_seq":132,"reported_epoch":119,"state":"active+clean","last_fresh":"2024-03-11T21:28:58.164876+0000","last_change":"2024-03-11T21:28:14.483927+0000","last_active":"2024-03-11T21:28:58.164876+0000","last_peered":"2024-03-11T21:28:58.164876+0000","last_clean":"2024-03-11T21:28:58.164876+0000","last_became_active":"2024-03-11T21:25:38.540510+0000","last_became_peered":"2024-03-11T21:25:38.540510+0000","last_unstale":"2024-03-11T21:28:58.164876+0000","last_undegraded":"2024-03-11T21:28:58.164876+0000","last_fullsized":"2024-03-11T21:28:58.164876+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:28:14.483813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:37.512018+0000","last_clean_scrub_stamp":"2024-03-11T21:28:14.483813+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:29:26.482626+0000","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,4],"acting":[0,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.19","version":"74'15","reported_seq":152,"reported_epoch":119,"state":"active+clean","last_fresh":"2024-03-11T21:28:58.169710+0000","last_change":"2024-03-11T21:28:37.436744+0000","last_active":"2024-03-11T21:28:58.169710+0000","last_peered":"2024-03-11T21:28:58.169710+0000","last_clean":"2024-03-11T21:28:58.169710+0000","last_became_active":"2024-03-11T21:25:40.554207+0000","last_became_peered":"2024-03-11T21:25:40.554207+0000","last_unstale":"2024-03-11T21:28:58.169710+0000","last_undegraded":"2024-03-11T21:28:58.169710+0000","last_fullsized":"2024-03-11T21:28:58.169710+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'15","last_scrub_stamp":"2024-03-11T21:28:37.436520+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:39.525554+0000","last_clean_scrub_stamp":"2024-03-11T21:28:37.436520+0000","objects_scrubbed":7,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:29:47.196615+0000","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.1a","version":"0'0","reported_seq":112,"reported_epoch":119,"state":"active+clean","last_fresh":"2024-03-11T21:28:58.154994+0000","last_change":"2024-03-11T21:28:10.634383+0000","last_active":"2024-03-11T21:28:58.154994+0000","last_peered":"2024-03-11T21:28:58.154994+0000","last_clean":"2024-03-11T21:28:58.154994+0000","last_became_active":"2024-03-11T21:25:44.590923+0000","last_became_peered":"2024-03-11T21:25:44.590923+0000","last_unstale":"2024-03-11T21:28:58.154994+0000","last_undegraded":"2024-03-11T21:28:58.154994+0000","last_fullsized":"2024-03-11T21:28:58.154994+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:28:10.634303+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:43.559452+0000","last_clean_scrub_stamp":"2024-03-11T21:28:10.634303+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:29:25.662002+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,1],"acting":[4,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":116,"reported_epoch":119,"state":"active+clean","last_fresh":"2024-03-11T21:28:58.159315+0000","last_change":"2024-03-11T21:27:17.264407+0000","last_active":"2024-03-11T21:28:58.159315+0000","last_peered":"2024-03-11T21:28:58.159315+0000","last_clean":"2024-03-11T21:28:58.159315+0000","last_became_active":"2024-03-11T21:25:42.578424+0000","last_became_peered":"2024-03-11T21:25:42.578424+0000","last_unstale":"2024-03-11T21:28:58.159315+0000","last_undegraded":"2024-03-11T21:28:58.159315+0000","last_fullsized":"2024-03-11T21:28:58.159315+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:27:17.264317+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:41.544550+0000","last_clean_scrub_stamp":"2024-03-11T21:27:17.264317+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:28:59.451115+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,7],"acting":[1,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1e","version":"0'0","reported_seq":131,"reported_epoch":119,"state":"active+clean","last_fresh":"2024-03-11T21:28:58.169727+0000","last_change":"2024-03-11T21:28:07.604544+0000","last_active":"2024-03-11T21:28:58.169727+0000","last_peered":"2024-03-11T21:28:58.169727+0000","last_clean":"2024-03-11T21:28:58.169727+0000","last_became_active":"2024-03-11T21:25:38.537283+0000","last_became_peered":"2024-03-11T21:25:38.537283+0000","last_unstale":"2024-03-11T21:28:58.169727+0000","last_undegraded":"2024-03-11T21:28:58.169727+0000","last_fullsized":"2024-03-11T21:28:58.169727+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:28:07.604432+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:37.512018+0000","last_clean_scrub_stamp":"2024-03-11T21:28:07.604432+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:29:20.709213+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.18","version":"74'9","reported_seq":144,"reported_epoch":119,"state":"active+clean","last_fresh":"2024-03-11T21:28:58.155013+0000","last_change":"2024-03-11T21:28:22.778031+0000","last_active":"2024-03-11T21:28:58.155013+0000","last_peered":"2024-03-11T21:28:58.155013+0000","last_clean":"2024-03-11T21:28:58.155013+0000","last_became_active":"2024-03-11T21:25:40.552216+0000","last_became_peered":"2024-03-11T21:25:40.552216+0000","last_unstale":"2024-03-11T21:28:58.155013+0000","last_undegraded":"2024-03-11T21:28:58.155013+0000","last_fullsized":"2024-03-11T21:28:58.155013+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'9","last_scrub_stamp":"2024-03-11T21:28:22.777781+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:39.525554+0000","last_clean_scrub_stamp":"2024-03-11T21:28:22.777781+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:29:29.319933+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":119,"reported_epoch":119,"state":"active+clean","last_fresh":"2024-03-11T21:28:58.163562+0000","last_change":"2024-03-11T21:28:34.755710+0000","last_active":"2024-03-11T21:28:58.163562+0000","last_peered":"2024-03-11T21:28:58.163562+0000","last_clean":"2024-03-11T21:28:58.163562+0000","last_became_active":"2024-03-11T21:25:44.590234+0000","last_became_peered":"2024-03-11T21:25:44.590234+0000","last_unstale":"2024-03-11T21:28:58.163562+0000","last_undegraded":"2024-03-11T21:28:58.163562+0000","last_fullsized":"2024-03-11T21:28:58.163562+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:28:34.755583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:43.559452+0000","last_clean_scrub_stamp":"2024-03-11T21:28:34.755583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:29:41.673503+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3],"acting":[5,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":115,"reported_epoch":119,"state":"active+clean","last_fresh":"2024-03-11T21:28:58.168404+0000","last_change":"2024-03-11T21:27:18.434331+0000","last_active":"2024-03-11T21:28:58.168404+0000","last_peered":"2024-03-11T21:28:58.168404+0000","last_clean":"2024-03-11T21:28:58.168404+0000","last_became_active":"2024-03-11T21:25:42.580533+0000","last_became_peered":"2024-03-11T21:25:42.580533+0000","last_unstale":"2024-03-11T21:28:58.168404+0000","last_undegraded":"2024-03-11T21:28:58.168404+0000","last_fullsized":"2024-03-11T21:28:58.168404+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:27:18.434247+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:41.544550+0000","last_clean_scrub_stamp":"2024-03-11T21:27:18.434247+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.1d","version":"0'0","reported_seq":131,"reported_epoch":119,"state":"active+clean","last_fresh":"2024-03-11T21:28:58.169557+0000","last_change":"2024-03-11T21:28:11.681226+0000","last_active":"2024-03-11T21:28:58.169557+0000","last_peered":"2024-03-11T21:28:58.169557+0000","last_clean":"2024-03-11T21:28:58.169557+0000","last_became_active":"2024-03-11T21:25:38.542628+0000","last_became_peered":"2024-03-11T21:25:38.542628+0000","last_unstale":"2024-03-11T21:28:58.169557+0000","last_undegraded":"2024-03-11T21:28:58.169557+0000","last_fullsized":"2024-03-11T21:28:58.169557+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:28:11.681096+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:37.512018+0000","last_clean_scrub_stamp":"2024-03-11T21:28:11.681096+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:29:22.527860+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,0],"acting":[7,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.1b","version":"74'5","reported_seq":130,"reported_epoch":119,"state":"active+clean","last_fresh":"2024-03-11T21:28:58.157256+0000","last_change":"2024-03-11T21:28:14.743893+0000","last_active":"2024-03-11T21:28:58.157256+0000","last_peered":"2024-03-11T21:28:58.157256+0000","last_clean":"2024-03-11T21:28:58.157256+0000","last_became_active":"2024-03-11T21:25:40.555826+0000","last_became_peered":"2024-03-11T21:25:40.555826+0000","last_unstale":"2024-03-11T21:28:58.157256+0000","last_undegraded":"2024-03-11T21:28:58.157256+0000","last_fullsized":"2024-03-11T21:28:58.157256+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'5","last_scrub_stamp":"2024-03-11T21:28:14.743756+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:39.525554+0000","last_clean_scrub_stamp":"2024-03-11T21:28:14.743756+0000","objects_scrubbed":2,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:29:25.133975+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":11,"num_read_kb":7,"num_write":6,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,1],"acting":[4,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":112,"reported_epoch":119,"state":"active+clean","last_fresh":"2024-03-11T21:28:58.157263+0000","last_change":"2024-03-11T21:27:29.476225+0000","last_active":"2024-03-11T21:28:58.157263+0000","last_peered":"2024-03-11T21:28:58.157263+0000","last_clean":"2024-03-11T21:28:58.157263+0000","last_became_active":"2024-03-11T21:25:44.589786+0000","last_became_peered":"2024-03-11T21:25:44.589786+0000","last_unstale":"2024-03-11T21:28:58.157263+0000","last_undegraded":"2024-03-11T21:28:58.157263+0000","last_fullsized":"2024-03-11T21:28:58.157263+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:27:29.476144+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:43.559452+0000","last_clean_scrub_stamp":"2024-03-11T21:27:29.476144+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:29:02.343726+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,7],"acting":[0,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":115,"reported_epoch":119,"state":"active+clean","last_fresh":"2024-03-11T21:28:58.165179+0000","last_change":"2024-03-11T21:27:18.957913+0000","last_active":"2024-03-11T21:28:58.165179+0000","last_peered":"2024-03-11T21:28:58.165179+0000","last_clean":"2024-03-11T21:28:58.165179+0000","last_became_active":"2024-03-11T21:25:42.573831+0000","last_became_peered":"2024-03-11T21:25:42.573831+0000","last_unstale":"2024-03-11T21:28:58.165179+0000","last_undegraded":"2024-03-11T21:28:58.165179+0000","last_fullsized":"2024-03-11T21:28:58.165179+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:27:18.957808+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:41.544550+0000","last_clean_scrub_stamp":"2024-03-11T21:27:18.957808+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:29:01.738040+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,7],"acting":[5,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1c","version":"0'0","reported_seq":131,"reported_epoch":119,"state":"active+clean","last_fresh":"2024-03-11T21:28:58.168686+0000","last_change":"2024-03-11T21:28:32.566552+0000","last_active":"2024-03-11T21:28:58.168686+0000","last_peered":"2024-03-11T21:28:58.168686+0000","last_clean":"2024-03-11T21:28:58.168686+0000","last_became_active":"2024-03-11T21:25:38.541364+0000","last_became_peered":"2024-03-11T21:25:38.541364+0000","last_unstale":"2024-03-11T21:28:58.168686+0000","last_undegraded":"2024-03-11T21:28:58.168686+0000","last_fullsized":"2024-03-11T21:28:58.168686+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:28:32.566437+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:28:32.566437+0000","last_clean_scrub_stamp":"2024-03-11T21:28:32.566437+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:29:50.714312+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.1a","version":"74'9","reported_seq":144,"reported_epoch":119,"state":"active+clean","last_fresh":"2024-03-11T21:28:58.155094+0000","last_change":"2024-03-11T21:28:18.725957+0000","last_active":"2024-03-11T21:28:58.155094+0000","last_peered":"2024-03-11T21:28:58.155094+0000","last_clean":"2024-03-11T21:28:58.155094+0000","last_became_active":"2024-03-11T21:25:40.556629+0000","last_became_peered":"2024-03-11T21:25:40.556629+0000","last_unstale":"2024-03-11T21:28:58.155094+0000","last_undegraded":"2024-03-11T21:28:58.155094+0000","last_fullsized":"2024-03-11T21:28:58.155094+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'9","last_scrub_stamp":"2024-03-11T21:28:18.725837+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:39.525554+0000","last_clean_scrub_stamp":"2024-03-11T21:28:18.725837+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:29:27.821802+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0],"acting":[4,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1f","version":"0'0","reported_seq":119,"reported_epoch":119,"state":"active+clean","last_fresh":"2024-03-11T21:28:58.169990+0000","last_change":"2024-03-11T21:28:28.559558+0000","last_active":"2024-03-11T21:28:58.169990+0000","last_peered":"2024-03-11T21:28:58.169990+0000","last_clean":"2024-03-11T21:28:58.169990+0000","last_became_active":"2024-03-11T21:25:45.302637+0000","last_became_peered":"2024-03-11T21:25:45.302637+0000","last_unstale":"2024-03-11T21:28:58.169990+0000","last_undegraded":"2024-03-11T21:28:58.169990+0000","last_fullsized":"2024-03-11T21:28:58.169990+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:28:28.559406+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:43.559452+0000","last_clean_scrub_stamp":"2024-03-11T21:28:28.559406+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:29:36.695979+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1c","version":"0'0","reported_seq":116,"reported_epoch":119,"state":"active+clean","last_fresh":"2024-03-11T21:28:58.157073+0000","last_change":"2024-03-11T21:27:10.787962+0000","last_active":"2024-03-11T21:28:58.157073+0000","last_peered":"2024-03-11T21:28:58.157073+0000","last_clean":"2024-03-11T21:28:58.157073+0000","last_became_active":"2024-03-11T21:25:42.576064+0000","last_became_peered":"2024-03-11T21:25:42.576064+0000","last_unstale":"2024-03-11T21:28:58.157073+0000","last_undegraded":"2024-03-11T21:28:58.157073+0000","last_fullsized":"2024-03-11T21:28:58.157073+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:27:10.787826+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:41.544550+0000","last_clean_scrub_stamp":"2024-03-11T21:27:10.787826+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:29:03.083210+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1b","version":"0'0","reported_seq":131,"reported_epoch":119,"state":"active+clean","last_fresh":"2024-03-11T21:28:58.170228+0000","last_change":"2024-03-11T21:28:33.582952+0000","last_active":"2024-03-11T21:28:58.170228+0000","last_peered":"2024-03-11T21:28:58.170228+0000","last_clean":"2024-03-11T21:28:58.170228+0000","last_became_active":"2024-03-11T21:25:38.537520+0000","last_became_peered":"2024-03-11T21:25:38.537520+0000","last_unstale":"2024-03-11T21:28:58.170228+0000","last_undegraded":"2024-03-11T21:28:58.170228+0000","last_fullsized":"2024-03-11T21:28:58.170228+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:28:33.582712+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:28:33.582712+0000","last_clean_scrub_stamp":"2024-03-11T21:28:33.582712+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:29:44.595087+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1d","version":"74'12","reported_seq":142,"reported_epoch":119,"state":"active+clean","last_fresh":"2024-03-11T21:28:58.168943+0000","last_change":"2024-03-11T21:27:01.629574+0000","last_active":"2024-03-11T21:28:58.168943+0000","last_peered":"2024-03-11T21:28:58.168943+0000","last_clean":"2024-03-11T21:28:58.168943+0000","last_became_active":"2024-03-11T21:25:40.556330+0000","last_became_peered":"2024-03-11T21:25:40.556330+0000","last_unstale":"2024-03-11T21:28:58.168943+0000","last_undegraded":"2024-03-11T21:28:58.168943+0000","last_fullsized":"2024-03-11T21:28:58.168943+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'12","last_scrub_stamp":"2024-03-11T21:27:01.629471+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:39.525554+0000","last_clean_scrub_stamp":"2024-03-11T21:27:01.629471+0000","objects_scrubbed":5,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":25,"num_read_kb":16,"num_write":14,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.1e","version":"0'0","reported_seq":112,"reported_epoch":119,"state":"active+clean","last_fresh":"2024-03-11T21:28:58.156549+0000","last_change":"2024-03-11T21:28:11.684315+0000","last_active":"2024-03-11T21:28:58.156549+0000","last_peered":"2024-03-11T21:28:58.156549+0000","last_clean":"2024-03-11T21:28:58.156549+0000","last_became_active":"2024-03-11T21:25:45.300355+0000","last_became_peered":"2024-03-11T21:25:45.300355+0000","last_unstale":"2024-03-11T21:28:58.156549+0000","last_undegraded":"2024-03-11T21:28:58.156549+0000","last_fullsized":"2024-03-11T21:28:58.156549+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:28:11.684174+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:43.559452+0000","last_clean_scrub_stamp":"2024-03-11T21:28:11.684174+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:29:32.796638+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5],"acting":[4,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1d","version":"0'0","reported_seq":124,"reported_epoch":119,"state":"active+clean","last_fresh":"2024-03-11T21:28:58.159672+0000","last_change":"2024-03-11T21:28:31.304566+0000","last_active":"2024-03-11T21:28:58.159672+0000","last_peered":"2024-03-11T21:28:58.159672+0000","last_clean":"2024-03-11T21:28:58.159672+0000","last_became_active":"2024-03-11T21:25:42.579407+0000","last_became_peered":"2024-03-11T21:25:42.579407+0000","last_unstale":"2024-03-11T21:28:58.159672+0000","last_undegraded":"2024-03-11T21:28:58.159672+0000","last_fullsized":"2024-03-11T21:28:58.159672+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:28:31.304120+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:27:09.199513+0000","last_clean_scrub_stamp":"2024-03-11T21:28:31.304120+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:29:45.672976+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1a","version":"0'0","reported_seq":132,"reported_epoch":119,"state":"active+clean","last_fresh":"2024-03-11T21:28:58.164798+0000","last_change":"2024-03-11T21:28:16.994328+0000","last_active":"2024-03-11T21:28:58.164798+0000","last_peered":"2024-03-11T21:28:58.164798+0000","last_clean":"2024-03-11T21:28:58.164798+0000","last_became_active":"2024-03-11T21:25:38.539725+0000","last_became_peered":"2024-03-11T21:25:38.539725+0000","last_unstale":"2024-03-11T21:28:58.164798+0000","last_undegraded":"2024-03-11T21:28:58.164798+0000","last_fullsized":"2024-03-11T21:28:58.164798+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:28:16.994251+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:37.512018+0000","last_clean_scrub_stamp":"2024-03-11T21:28:16.994251+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:29:45.088652+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.1c","version":"74'15","reported_seq":152,"reported_epoch":119,"state":"active+clean","last_fresh":"2024-03-11T21:28:58.162315+0000","last_change":"2024-03-11T21:28:35.104768+0000","last_active":"2024-03-11T21:28:58.162315+0000","last_peered":"2024-03-11T21:28:58.162315+0000","last_clean":"2024-03-11T21:28:58.162315+0000","last_became_active":"2024-03-11T21:25:40.551453+0000","last_became_peered":"2024-03-11T21:25:40.551453+0000","last_unstale":"2024-03-11T21:28:58.162315+0000","last_undegraded":"2024-03-11T21:28:58.162315+0000","last_fullsized":"2024-03-11T21:28:58.162315+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'15","last_scrub_stamp":"2024-03-11T21:28:35.104626+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:39.525554+0000","last_clean_scrub_stamp":"2024-03-11T21:28:35.104626+0000","objects_scrubbed":7,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:29:50.086022+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.1d","version":"0'0","reported_seq":112,"reported_epoch":119,"state":"active+clean","last_fresh":"2024-03-11T21:28:58.159106+0000","last_change":"2024-03-11T21:27:20.249317+0000","last_active":"2024-03-11T21:28:58.159106+0000","last_peered":"2024-03-11T21:28:58.159106+0000","last_clean":"2024-03-11T21:28:58.159106+0000","last_became_active":"2024-03-11T21:25:44.588845+0000","last_became_peered":"2024-03-11T21:25:44.588845+0000","last_unstale":"2024-03-11T21:28:58.159106+0000","last_undegraded":"2024-03-11T21:28:58.159106+0000","last_fullsized":"2024-03-11T21:28:58.159106+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:27:20.249231+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:43.559452+0000","last_clean_scrub_stamp":"2024-03-11T21:27:20.249231+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:28:59.451146+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.1e","version":"0'0","reported_seq":124,"reported_epoch":119,"state":"active+clean","last_fresh":"2024-03-11T21:28:58.165293+0000","last_change":"2024-03-11T21:28:17.464869+0000","last_active":"2024-03-11T21:28:58.165293+0000","last_peered":"2024-03-11T21:28:58.165293+0000","last_clean":"2024-03-11T21:28:58.165293+0000","last_became_active":"2024-03-11T21:25:42.582959+0000","last_became_peered":"2024-03-11T21:25:42.582959+0000","last_unstale":"2024-03-11T21:28:58.165293+0000","last_undegraded":"2024-03-11T21:28:58.165293+0000","last_fullsized":"2024-03-11T21:28:58.165293+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:28:17.464401+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:26:58.335431+0000","last_clean_scrub_stamp":"2024-03-11T21:28:17.464401+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:29:40.047363+0000","scrub_duration":20,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.19","version":"67'2","reported_seq":130,"reported_epoch":119,"state":"active+clean","last_fresh":"2024-03-11T21:28:58.169859+0000","last_change":"2024-03-11T21:27:27.823258+0000","last_active":"2024-03-11T21:28:58.169859+0000","last_peered":"2024-03-11T21:28:58.169859+0000","last_clean":"2024-03-11T21:28:58.169859+0000","last_became_active":"2024-03-11T21:25:38.539243+0000","last_became_peered":"2024-03-11T21:25:38.539243+0000","last_unstale":"2024-03-11T21:28:58.169859+0000","last_undegraded":"2024-03-11T21:28:58.169859+0000","last_fullsized":"2024-03-11T21:28:58.169859+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'2","last_scrub_stamp":"2024-03-11T21:27:27.823167+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:37.512018+0000","last_clean_scrub_stamp":"2024-03-11T21:27:27.823167+0000","objects_scrubbed":2,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":19,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":957,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":7,"num_read_kb":7,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,0],"acting":[3,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1f","version":"74'11","reported_seq":147,"reported_epoch":119,"state":"active+clean","last_fresh":"2024-03-11T21:28:58.165899+0000","last_change":"2024-03-11T21:28:18.041390+0000","last_active":"2024-03-11T21:28:58.165899+0000","last_peered":"2024-03-11T21:28:58.165899+0000","last_clean":"2024-03-11T21:28:58.165899+0000","last_became_active":"2024-03-11T21:25:40.554686+0000","last_became_peered":"2024-03-11T21:25:40.554686+0000","last_unstale":"2024-03-11T21:28:58.165899+0000","last_undegraded":"2024-03-11T21:28:58.165899+0000","last_fullsized":"2024-03-11T21:28:58.165899+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'11","last_scrub_stamp":"2024-03-11T21:28:18.041260+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:39.525554+0000","last_clean_scrub_stamp":"2024-03-11T21:28:18.041260+0000","objects_scrubbed":5,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:29:23.336510+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,1],"acting":[6,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.1c","version":"74'1","reported_seq":112,"reported_epoch":119,"state":"active+clean","last_fresh":"2024-03-11T21:28:58.169002+0000","last_change":"2024-03-11T21:27:22.466670+0000","last_active":"2024-03-11T21:28:58.169002+0000","last_peered":"2024-03-11T21:28:58.169002+0000","last_clean":"2024-03-11T21:28:58.169002+0000","last_became_active":"2024-03-11T21:25:44.593326+0000","last_became_peered":"2024-03-11T21:25:44.593326+0000","last_unstale":"2024-03-11T21:28:58.169002+0000","last_undegraded":"2024-03-11T21:28:58.169002+0000","last_fullsized":"2024-03-11T21:28:58.169002+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'1","last_scrub_stamp":"2024-03-11T21:27:22.466585+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:43.559452+0000","last_clean_scrub_stamp":"2024-03-11T21:27:22.466585+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":370,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.1f","version":"74'8","reported_seq":176,"reported_epoch":119,"state":"active+clean","last_fresh":"2024-03-11T21:28:58.165853+0000","last_change":"2024-03-11T21:28:06.034479+0000","last_active":"2024-03-11T21:28:58.165853+0000","last_peered":"2024-03-11T21:28:58.165853+0000","last_clean":"2024-03-11T21:28:58.165853+0000","last_became_active":"2024-03-11T21:25:42.577401+0000","last_became_peered":"2024-03-11T21:25:42.577401+0000","last_unstale":"2024-03-11T21:28:58.165853+0000","last_undegraded":"2024-03-11T21:28:58.165853+0000","last_fullsized":"2024-03-11T21:28:58.165853+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'8","last_scrub_stamp":"2024-03-11T21:28:06.034236+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:41.544550+0000","last_clean_scrub_stamp":"2024-03-11T21:28:06.034236+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:29:10.600889+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.18","version":"67'1","reported_seq":137,"reported_epoch":119,"state":"active+clean","last_fresh":"2024-03-11T21:28:58.163591+0000","last_change":"2024-03-11T21:28:16.829077+0000","last_active":"2024-03-11T21:28:58.163591+0000","last_peered":"2024-03-11T21:28:58.163591+0000","last_clean":"2024-03-11T21:28:58.163591+0000","last_became_active":"2024-03-11T21:25:38.538321+0000","last_became_peered":"2024-03-11T21:25:38.538321+0000","last_unstale":"2024-03-11T21:28:58.163591+0000","last_undegraded":"2024-03-11T21:28:58.163591+0000","last_fullsized":"2024-03-11T21:28:58.163591+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'1","last_scrub_stamp":"2024-03-11T21:28:16.828998+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:37.512018+0000","last_clean_scrub_stamp":"2024-03-11T21:28:16.828998+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:29:33.110060+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":436,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":7,"num_read_kb":7,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,7],"acting":[5,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1e","version":"74'10","reported_seq":143,"reported_epoch":119,"state":"active+clean","last_fresh":"2024-03-11T21:28:58.165192+0000","last_change":"2024-03-11T21:28:26.392381+0000","last_active":"2024-03-11T21:28:58.165192+0000","last_peered":"2024-03-11T21:28:58.165192+0000","last_clean":"2024-03-11T21:28:58.165192+0000","last_became_active":"2024-03-11T21:25:40.763438+0000","last_became_peered":"2024-03-11T21:25:40.763438+0000","last_unstale":"2024-03-11T21:28:58.165192+0000","last_undegraded":"2024-03-11T21:28:58.165192+0000","last_fullsized":"2024-03-11T21:28:58.165192+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'10","last_scrub_stamp":"2024-03-11T21:28:26.392246+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:39.525554+0000","last_clean_scrub_stamp":"2024-03-11T21:28:26.392246+0000","objects_scrubbed":5,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:29:40.807743+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.13","version":"0'0","reported_seq":119,"reported_epoch":119,"state":"active+clean","last_fresh":"2024-03-11T21:28:58.170543+0000","last_change":"2024-03-11T21:28:27.600720+0000","last_active":"2024-03-11T21:28:58.170543+0000","last_peered":"2024-03-11T21:28:58.170543+0000","last_clean":"2024-03-11T21:28:58.170543+0000","last_became_active":"2024-03-11T21:25:45.302102+0000","last_became_peered":"2024-03-11T21:25:45.302102+0000","last_unstale":"2024-03-11T21:28:58.170543+0000","last_undegraded":"2024-03-11T21:28:58.170543+0000","last_fullsized":"2024-03-11T21:28:58.170543+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:28:27.600574+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:43.559452+0000","last_clean_scrub_stamp":"2024-03-11T21:28:27.600574+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:29:32.095820+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":123,"reported_epoch":119,"state":"active+clean","last_fresh":"2024-03-11T21:28:58.169722+0000","last_change":"2024-03-11T21:28:26.537966+0000","last_active":"2024-03-11T21:28:58.169722+0000","last_peered":"2024-03-11T21:28:58.169722+0000","last_clean":"2024-03-11T21:28:58.169722+0000","last_became_active":"2024-03-11T21:25:42.579781+0000","last_became_peered":"2024-03-11T21:25:42.579781+0000","last_unstale":"2024-03-11T21:28:58.169722+0000","last_undegraded":"2024-03-11T21:28:58.169722+0000","last_fullsized":"2024-03-11T21:28:58.169722+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:28:26.537878+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:41.544550+0000","last_clean_scrub_stamp":"2024-03-11T21:28:26.537878+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:29:43.076912+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.17","version":"0'0","reported_seq":132,"reported_epoch":119,"state":"active+clean","last_fresh":"2024-03-11T21:28:58.165656+0000","last_change":"2024-03-11T21:28:25.918222+0000","last_active":"2024-03-11T21:28:58.165656+0000","last_peered":"2024-03-11T21:28:58.165656+0000","last_clean":"2024-03-11T21:28:58.165656+0000","last_became_active":"2024-03-11T21:25:38.540349+0000","last_became_peered":"2024-03-11T21:25:38.540349+0000","last_unstale":"2024-03-11T21:28:58.165656+0000","last_undegraded":"2024-03-11T21:28:58.165656+0000","last_fullsized":"2024-03-11T21:28:58.165656+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:28:25.918101+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:28:25.918101+0000","last_clean_scrub_stamp":"2024-03-11T21:28:25.918101+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:29:47.849509+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,2],"acting":[6,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.11","version":"74'11","reported_seq":138,"reported_epoch":119,"state":"active+clean","last_fresh":"2024-03-11T21:28:58.170050+0000","last_change":"2024-03-11T21:27:22.793832+0000","last_active":"2024-03-11T21:28:58.170050+0000","last_peered":"2024-03-11T21:28:58.170050+0000","last_clean":"2024-03-11T21:28:58.170050+0000","last_became_active":"2024-03-11T21:25:40.760664+0000","last_became_peered":"2024-03-11T21:25:40.760664+0000","last_unstale":"2024-03-11T21:28:58.170050+0000","last_undegraded":"2024-03-11T21:28:58.170050+0000","last_fullsized":"2024-03-11T21:28:58.170050+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'11","last_scrub_stamp":"2024-03-11T21:27:22.793725+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:39.525554+0000","last_clean_scrub_stamp":"2024-03-11T21:27:22.793725+0000","objects_scrubbed":5,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":119,"reported_epoch":119,"state":"active+clean","last_fresh":"2024-03-11T21:28:58.168075+0000","last_change":"2024-03-11T21:28:19.525623+0000","last_active":"2024-03-11T21:28:58.168075+0000","last_peered":"2024-03-11T21:28:58.168075+0000","last_clean":"2024-03-11T21:28:58.168075+0000","last_became_active":"2024-03-11T21:25:44.591068+0000","last_became_peered":"2024-03-11T21:25:44.591068+0000","last_unstale":"2024-03-11T21:28:58.168075+0000","last_undegraded":"2024-03-11T21:28:58.168075+0000","last_fullsized":"2024-03-11T21:28:58.168075+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:28:19.525508+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:43.559452+0000","last_clean_scrub_stamp":"2024-03-11T21:28:19.525508+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:29:27.438631+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,4],"acting":[7,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":124,"reported_epoch":119,"state":"active+clean","last_fresh":"2024-03-11T21:28:58.165029+0000","last_change":"2024-03-11T21:28:32.783682+0000","last_active":"2024-03-11T21:28:58.165029+0000","last_peered":"2024-03-11T21:28:58.165029+0000","last_clean":"2024-03-11T21:28:58.165029+0000","last_became_active":"2024-03-11T21:25:42.584148+0000","last_became_peered":"2024-03-11T21:25:42.584148+0000","last_unstale":"2024-03-11T21:28:58.165029+0000","last_undegraded":"2024-03-11T21:28:58.165029+0000","last_fullsized":"2024-03-11T21:28:58.165029+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:28:32.783565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:27:09.976747+0000","last_clean_scrub_stamp":"2024-03-11T21:28:32.783565+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:29:36.980081+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.16","version":"0'0","reported_seq":131,"reported_epoch":119,"state":"active+clean","last_fresh":"2024-03-11T21:28:58.165462+0000","last_change":"2024-03-11T21:28:37.734762+0000","last_active":"2024-03-11T21:28:58.165462+0000","last_peered":"2024-03-11T21:28:58.165462+0000","last_clean":"2024-03-11T21:28:58.165462+0000","last_became_active":"2024-03-11T21:25:38.539103+0000","last_became_peered":"2024-03-11T21:25:38.539103+0000","last_unstale":"2024-03-11T21:28:58.165462+0000","last_undegraded":"2024-03-11T21:28:58.165462+0000","last_fullsized":"2024-03-11T21:28:58.165462+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:28:37.734633+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:27:11.938909+0000","last_clean_scrub_stamp":"2024-03-11T21:28:37.734633+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:29:40.854282+0000","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,2],"acting":[5,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.10","version":"94'6","reported_seq":135,"reported_epoch":119,"state":"active+clean","last_fresh":"2024-03-11T21:28:58.169334+0000","last_change":"2024-03-11T21:28:24.604117+0000","last_active":"2024-03-11T21:28:58.169334+0000","last_peered":"2024-03-11T21:28:58.169334+0000","last_clean":"2024-03-11T21:28:58.169334+0000","last_became_active":"2024-03-11T21:25:40.552720+0000","last_became_peered":"2024-03-11T21:25:40.552720+0000","last_unstale":"2024-03-11T21:28:58.169334+0000","last_undegraded":"2024-03-11T21:28:58.169334+0000","last_fullsized":"2024-03-11T21:28:58.169334+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"94'6","last_scrub_stamp":"2024-03-11T21:28:24.603974+0000","last_deep_scrub":"94'6","last_deep_scrub_stamp":"2024-03-11T21:28:24.603974+0000","last_clean_scrub_stamp":"2024-03-11T21:28:24.603974+0000","objects_scrubbed":2,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:29:27.655269+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":6,"num_read_kb":4,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6],"acting":[3,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":111,"reported_epoch":119,"state":"active+clean","last_fresh":"2024-03-11T21:28:58.169556+0000","last_change":"2024-03-11T21:27:05.669068+0000","last_active":"2024-03-11T21:28:58.169556+0000","last_peered":"2024-03-11T21:28:58.169556+0000","last_clean":"2024-03-11T21:28:58.169556+0000","last_became_active":"2024-03-11T21:25:44.583181+0000","last_became_peered":"2024-03-11T21:25:44.583181+0000","last_unstale":"2024-03-11T21:28:58.169556+0000","last_undegraded":"2024-03-11T21:28:58.169556+0000","last_fullsized":"2024-03-11T21:28:58.169556+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:27:05.668971+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:43.559452+0000","last_clean_scrub_stamp":"2024-03-11T21:27:05.668971+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":124,"reported_epoch":119,"state":"active+clean","last_fresh":"2024-03-11T21:28:58.163211+0000","last_change":"2024-03-11T21:28:34.310861+0000","last_active":"2024-03-11T21:28:58.163211+0000","last_peered":"2024-03-11T21:28:58.163211+0000","last_clean":"2024-03-11T21:28:58.163211+0000","last_became_active":"2024-03-11T21:25:42.573158+0000","last_became_peered":"2024-03-11T21:25:42.573158+0000","last_unstale":"2024-03-11T21:28:58.163211+0000","last_undegraded":"2024-03-11T21:28:58.163211+0000","last_fullsized":"2024-03-11T21:28:58.163211+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:28:34.310410+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:27:21.207303+0000","last_clean_scrub_stamp":"2024-03-11T21:28:34.310410+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:29:46.914835+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.15","version":"0'0","reported_seq":132,"reported_epoch":119,"state":"active+clean","last_fresh":"2024-03-11T21:28:58.159377+0000","last_change":"2024-03-11T21:28:17.525764+0000","last_active":"2024-03-11T21:28:58.159377+0000","last_peered":"2024-03-11T21:28:58.159377+0000","last_clean":"2024-03-11T21:28:58.159377+0000","last_became_active":"2024-03-11T21:25:38.541705+0000","last_became_peered":"2024-03-11T21:25:38.541705+0000","last_unstale":"2024-03-11T21:28:58.159377+0000","last_undegraded":"2024-03-11T21:28:58.159377+0000","last_fullsized":"2024-03-11T21:28:58.159377+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:28:17.525649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:26:44.860499+0000","last_clean_scrub_stamp":"2024-03-11T21:28:17.525649+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:29:31.562181+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.13","version":"109'16","reported_seq":170,"reported_epoch":119,"state":"active+clean","last_fresh":"2024-03-11T21:28:58.157029+0000","last_change":"2024-03-11T21:28:32.854572+0000","last_active":"2024-03-11T21:28:58.157029+0000","last_peered":"2024-03-11T21:28:58.157029+0000","last_clean":"2024-03-11T21:28:58.157029+0000","last_became_active":"2024-03-11T21:26:11.214879+0000","last_became_peered":"2024-03-11T21:26:11.214879+0000","last_unstale":"2024-03-11T21:28:58.157029+0000","last_undegraded":"2024-03-11T21:28:58.157029+0000","last_fullsized":"2024-03-11T21:28:58.157029+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"109'16","last_scrub_stamp":"2024-03-11T21:28:32.854393+0000","last_deep_scrub":"109'16","last_deep_scrub_stamp":"2024-03-11T21:28:32.854393+0000","last_clean_scrub_stamp":"2024-03-11T21:28:32.854393+0000","objects_scrubbed":5,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:29:37.486089+0000","scrub_duration":16,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,2],"acting":[4,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.10","version":"0'0","reported_seq":112,"reported_epoch":119,"state":"active+clean","last_fresh":"2024-03-11T21:28:58.156739+0000","last_change":"2024-03-11T21:27:25.417528+0000","last_active":"2024-03-11T21:28:58.156739+0000","last_peered":"2024-03-11T21:28:58.156739+0000","last_clean":"2024-03-11T21:28:58.156739+0000","last_became_active":"2024-03-11T21:25:44.591189+0000","last_became_peered":"2024-03-11T21:25:44.591189+0000","last_unstale":"2024-03-11T21:28:58.156739+0000","last_undegraded":"2024-03-11T21:28:58.156739+0000","last_fullsized":"2024-03-11T21:28:58.156739+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:27:25.417450+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:27:25.417450+0000","last_clean_scrub_stamp":"2024-03-11T21:27:25.417450+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:29:02.343699+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,1],"acting":[0,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":115,"reported_epoch":119,"state":"active+clean","last_fresh":"2024-03-11T21:28:58.170310+0000","last_change":"2024-03-11T21:27:06.637829+0000","last_active":"2024-03-11T21:28:58.170310+0000","last_peered":"2024-03-11T21:28:58.170310+0000","last_clean":"2024-03-11T21:28:58.170310+0000","last_became_active":"2024-03-11T21:25:42.576425+0000","last_became_peered":"2024-03-11T21:25:42.576425+0000","last_unstale":"2024-03-11T21:28:58.170310+0000","last_undegraded":"2024-03-11T21:28:58.170310+0000","last_fullsized":"2024-03-11T21:28:58.170310+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:27:06.637752+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:41.544550+0000","last_clean_scrub_stamp":"2024-03-11T21:27:06.637752+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.14","version":"0'0","reported_seq":132,"reported_epoch":119,"state":"active+clean","last_fresh":"2024-03-11T21:28:58.165403+0000","last_change":"2024-03-11T21:28:09.912659+0000","last_active":"2024-03-11T21:28:58.165403+0000","last_peered":"2024-03-11T21:28:58.165403+0000","last_clean":"2024-03-11T21:28:58.165403+0000","last_became_active":"2024-03-11T21:25:38.541852+0000","last_became_peered":"2024-03-11T21:25:38.541852+0000","last_unstale":"2024-03-11T21:28:58.165403+0000","last_undegraded":"2024-03-11T21:28:58.165403+0000","last_fullsized":"2024-03-11T21:28:58.165403+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:28:09.912544+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:37.512018+0000","last_clean_scrub_stamp":"2024-03-11T21:28:09.912544+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:29:18.193923+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,5],"acting":[6,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.12","version":"74'9","reported_seq":144,"reported_epoch":119,"state":"active+clean","last_fresh":"2024-03-11T21:28:58.163007+0000","last_change":"2024-03-11T21:28:32.283534+0000","last_active":"2024-03-11T21:28:58.163007+0000","last_peered":"2024-03-11T21:28:58.163007+0000","last_clean":"2024-03-11T21:28:58.163007+0000","last_became_active":"2024-03-11T21:25:40.547082+0000","last_became_peered":"2024-03-11T21:25:40.547082+0000","last_unstale":"2024-03-11T21:28:58.163007+0000","last_undegraded":"2024-03-11T21:28:58.163007+0000","last_fullsized":"2024-03-11T21:28:58.163007+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'9","last_scrub_stamp":"2024-03-11T21:28:32.283410+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:39.525554+0000","last_clean_scrub_stamp":"2024-03-11T21:28:32.283410+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:29:39.936188+0000","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":112,"reported_epoch":119,"state":"active+clean","last_fresh":"2024-03-11T21:28:58.157028+0000","last_change":"2024-03-11T21:27:09.801239+0000","last_active":"2024-03-11T21:28:58.157028+0000","last_peered":"2024-03-11T21:28:58.157028+0000","last_clean":"2024-03-11T21:28:58.157028+0000","last_became_active":"2024-03-11T21:25:44.589586+0000","last_became_peered":"2024-03-11T21:25:44.589586+0000","last_unstale":"2024-03-11T21:28:58.157028+0000","last_undegraded":"2024-03-11T21:28:58.157028+0000","last_fullsized":"2024-03-11T21:28:58.157028+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:27:09.801138+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:27:09.801138+0000","last_clean_scrub_stamp":"2024-03-11T21:27:09.801138+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:29:03.083295+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,5],"acting":[4,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.14","version":"74'8","reported_seq":164,"reported_epoch":119,"state":"active+clean","last_fresh":"2024-03-11T21:28:58.170780+0000","last_change":"2024-03-11T21:27:24.803037+0000","last_active":"2024-03-11T21:28:58.170780+0000","last_peered":"2024-03-11T21:28:58.170780+0000","last_clean":"2024-03-11T21:28:58.170780+0000","last_became_active":"2024-03-11T21:25:42.583266+0000","last_became_peered":"2024-03-11T21:25:42.583266+0000","last_unstale":"2024-03-11T21:28:58.170780+0000","last_undegraded":"2024-03-11T21:28:58.170780+0000","last_fullsized":"2024-03-11T21:28:58.170780+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'8","last_scrub_stamp":"2024-03-11T21:27:24.802932+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:41.544550+0000","last_clean_scrub_stamp":"2024-03-11T21:27:24.802932+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:29:03.493355+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,2],"acting":[3,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.13","version":"0'0","reported_seq":124,"reported_epoch":119,"state":"active+clean","last_fresh":"2024-03-11T21:28:58.157593+0000","last_change":"2024-03-11T21:27:12.315278+0000","last_active":"2024-03-11T21:28:58.157593+0000","last_peered":"2024-03-11T21:28:58.157593+0000","last_clean":"2024-03-11T21:28:58.157593+0000","last_became_active":"2024-03-11T21:25:38.539467+0000","last_became_peered":"2024-03-11T21:25:38.539467+0000","last_unstale":"2024-03-11T21:28:58.157593+0000","last_undegraded":"2024-03-11T21:28:58.157593+0000","last_fullsized":"2024-03-11T21:28:58.157593+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:27:12.315200+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:37.512018+0000","last_clean_scrub_stamp":"2024-03-11T21:27:12.315200+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:29:02.343567+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.15","version":"94'13","reported_seq":147,"reported_epoch":119,"state":"active+clean","last_fresh":"2024-03-11T21:28:58.164533+0000","last_change":"2024-03-11T21:28:27.691406+0000","last_active":"2024-03-11T21:28:58.164533+0000","last_peered":"2024-03-11T21:28:58.164533+0000","last_clean":"2024-03-11T21:28:58.164533+0000","last_became_active":"2024-03-11T21:25:40.760907+0000","last_became_peered":"2024-03-11T21:25:40.760907+0000","last_unstale":"2024-03-11T21:28:58.164533+0000","last_undegraded":"2024-03-11T21:28:58.164533+0000","last_fullsized":"2024-03-11T21:28:58.164533+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"94'13","last_scrub_stamp":"2024-03-11T21:28:27.691249+0000","last_deep_scrub":"94'13","last_deep_scrub_stamp":"2024-03-11T21:26:59.013823+0000","last_clean_scrub_stamp":"2024-03-11T21:28:27.691249+0000","objects_scrubbed":4,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:29:40.580610+0000","scrub_duration":28,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.16","version":"0'0","reported_seq":120,"reported_epoch":119,"state":"active+clean","last_fresh":"2024-03-11T21:28:58.164401+0000","last_change":"2024-03-11T21:28:29.368481+0000","last_active":"2024-03-11T21:28:58.164401+0000","last_peered":"2024-03-11T21:28:58.164401+0000","last_clean":"2024-03-11T21:28:58.164401+0000","last_became_active":"2024-03-11T21:25:44.583393+0000","last_became_peered":"2024-03-11T21:25:44.583393+0000","last_unstale":"2024-03-11T21:28:58.164401+0000","last_undegraded":"2024-03-11T21:28:58.164401+0000","last_fullsized":"2024-03-11T21:28:58.164401+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:28:29.368377+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:43.559452+0000","last_clean_scrub_stamp":"2024-03-11T21:28:29.368377+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:29:46.910936+0000","scrub_duration":27,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.15","version":"98'9","reported_seq":165,"reported_epoch":119,"state":"active+clean","last_fresh":"2024-03-11T21:28:58.164434+0000","last_change":"2024-03-11T21:27:34.948935+0000","last_active":"2024-03-11T21:28:58.164434+0000","last_peered":"2024-03-11T21:28:58.164434+0000","last_clean":"2024-03-11T21:28:58.164434+0000","last_became_active":"2024-03-11T21:25:42.572555+0000","last_became_peered":"2024-03-11T21:25:42.572555+0000","last_unstale":"2024-03-11T21:28:58.164434+0000","last_undegraded":"2024-03-11T21:28:58.164434+0000","last_fullsized":"2024-03-11T21:28:58.164434+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"98'9","last_scrub_stamp":"2024-03-11T21:27:34.948855+0000","last_deep_scrub":"98'9","last_deep_scrub_stamp":"2024-03-11T21:27:34.948855+0000","last_clean_scrub_stamp":"2024-03-11T21:27:34.948855+0000","objects_scrubbed":1,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:29:01.738122+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.12","version":"0'0","reported_seq":131,"reported_epoch":119,"state":"active+clean","last_fresh":"2024-03-11T21:28:58.163835+0000","last_change":"2024-03-11T21:28:26.701731+0000","last_active":"2024-03-11T21:28:58.163835+0000","last_peered":"2024-03-11T21:28:58.163835+0000","last_clean":"2024-03-11T21:28:58.163835+0000","last_became_active":"2024-03-11T21:25:38.543962+0000","last_became_peered":"2024-03-11T21:25:38.543962+0000","last_unstale":"2024-03-11T21:28:58.163835+0000","last_undegraded":"2024-03-11T21:28:58.163835+0000","last_fullsized":"2024-03-11T21:28:58.163835+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:28:26.701620+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:37.512018+0000","last_clean_scrub_stamp":"2024-03-11T21:28:26.701620+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:29:51.689535+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.14","version":"74'10","reported_seq":134,"reported_epoch":119,"state":"active+clean","last_fresh":"2024-03-11T21:28:58.170519+0000","last_change":"2024-03-11T21:27:19.754590+0000","last_active":"2024-03-11T21:28:58.170519+0000","last_peered":"2024-03-11T21:28:58.170519+0000","last_clean":"2024-03-11T21:28:58.170519+0000","last_became_active":"2024-03-11T21:25:40.760620+0000","last_became_peered":"2024-03-11T21:25:40.760620+0000","last_unstale":"2024-03-11T21:28:58.170519+0000","last_undegraded":"2024-03-11T21:28:58.170519+0000","last_fullsized":"2024-03-11T21:28:58.170519+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'10","last_scrub_stamp":"2024-03-11T21:27:19.754499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:39.525554+0000","last_clean_scrub_stamp":"2024-03-11T21:27:19.754499+0000","objects_scrubbed":5,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.15","version":"0'0","reported_seq":119,"reported_epoch":119,"state":"active+clean","last_fresh":"2024-03-11T21:28:58.169466+0000","last_change":"2024-03-11T21:28:33.526147+0000","last_active":"2024-03-11T21:28:58.169466+0000","last_peered":"2024-03-11T21:28:58.169466+0000","last_clean":"2024-03-11T21:28:58.169466+0000","last_became_active":"2024-03-11T21:25:45.302501+0000","last_became_peered":"2024-03-11T21:25:45.302501+0000","last_unstale":"2024-03-11T21:28:58.169466+0000","last_undegraded":"2024-03-11T21:28:58.169466+0000","last_fullsized":"2024-03-11T21:28:58.169466+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:28:33.526015+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:43.559452+0000","last_clean_scrub_stamp":"2024-03-11T21:28:33.526015+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:30:01.748461+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":115,"reported_epoch":119,"state":"active+clean","last_fresh":"2024-03-11T21:28:58.165376+0000","last_change":"2024-03-11T21:27:33.987450+0000","last_active":"2024-03-11T21:28:58.165376+0000","last_peered":"2024-03-11T21:28:58.165376+0000","last_clean":"2024-03-11T21:28:58.165376+0000","last_became_active":"2024-03-11T21:25:42.577351+0000","last_became_peered":"2024-03-11T21:25:42.577351+0000","last_unstale":"2024-03-11T21:28:58.165376+0000","last_undegraded":"2024-03-11T21:28:58.165376+0000","last_fullsized":"2024-03-11T21:28:58.165376+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:27:33.987367+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:41.544550+0000","last_clean_scrub_stamp":"2024-03-11T21:27:33.987367+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:29:01.738058+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,1],"acting":[5,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.11","version":"0'0","reported_seq":132,"reported_epoch":119,"state":"active+clean","last_fresh":"2024-03-11T21:28:58.165138+0000","last_change":"2024-03-11T21:28:28.897384+0000","last_active":"2024-03-11T21:28:58.165138+0000","last_peered":"2024-03-11T21:28:58.165138+0000","last_clean":"2024-03-11T21:28:58.165138+0000","last_became_active":"2024-03-11T21:25:38.537415+0000","last_became_peered":"2024-03-11T21:25:38.537415+0000","last_unstale":"2024-03-11T21:28:58.165138+0000","last_undegraded":"2024-03-11T21:28:58.165138+0000","last_fullsized":"2024-03-11T21:28:58.165138+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:28:28.897294+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:37.512018+0000","last_clean_scrub_stamp":"2024-03-11T21:28:28.897294+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:29:53.616989+0000","scrub_duration":16,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.17","version":"97'9","reported_seq":145,"reported_epoch":119,"state":"active+clean","last_fresh":"2024-03-11T21:28:58.170085+0000","last_change":"2024-03-11T21:27:25.803221+0000","last_active":"2024-03-11T21:28:58.170085+0000","last_peered":"2024-03-11T21:28:58.170085+0000","last_clean":"2024-03-11T21:28:58.170085+0000","last_became_active":"2024-03-11T21:26:11.219020+0000","last_became_peered":"2024-03-11T21:26:11.219020+0000","last_unstale":"2024-03-11T21:28:58.170085+0000","last_undegraded":"2024-03-11T21:28:58.170085+0000","last_fullsized":"2024-03-11T21:28:58.170085+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"97'9","last_scrub_stamp":"2024-03-11T21:27:25.803132+0000","last_deep_scrub":"97'9","last_deep_scrub_stamp":"2024-03-11T21:27:25.803132+0000","last_clean_scrub_stamp":"2024-03-11T21:27:25.803132+0000","objects_scrubbed":3,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":9,"num_read_kb":6,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,5],"acting":[3,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.14","version":"0'0","reported_seq":111,"reported_epoch":119,"state":"active+clean","last_fresh":"2024-03-11T21:28:58.161967+0000","last_change":"2024-03-11T21:27:21.063894+0000","last_active":"2024-03-11T21:28:58.161967+0000","last_peered":"2024-03-11T21:28:58.161967+0000","last_clean":"2024-03-11T21:28:58.161967+0000","last_became_active":"2024-03-11T21:25:44.587439+0000","last_became_peered":"2024-03-11T21:25:44.587439+0000","last_unstale":"2024-03-11T21:28:58.161967+0000","last_undegraded":"2024-03-11T21:28:58.161967+0000","last_fullsized":"2024-03-11T21:28:58.161967+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:27:21.063817+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:43.559452+0000","last_clean_scrub_stamp":"2024-03-11T21:27:21.063817+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:29:00.225048+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,7],"acting":[2,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":123,"reported_epoch":119,"state":"active+clean","last_fresh":"2024-03-11T21:28:58.170176+0000","last_change":"2024-03-11T21:28:35.504528+0000","last_active":"2024-03-11T21:28:58.170176+0000","last_peered":"2024-03-11T21:28:58.170176+0000","last_clean":"2024-03-11T21:28:58.170176+0000","last_became_active":"2024-03-11T21:25:42.583514+0000","last_became_peered":"2024-03-11T21:25:42.583514+0000","last_unstale":"2024-03-11T21:28:58.170176+0000","last_undegraded":"2024-03-11T21:28:58.170176+0000","last_fullsized":"2024-03-11T21:28:58.170176+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:28:35.504414+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:41.544550+0000","last_clean_scrub_stamp":"2024-03-11T21:28:35.504414+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:29:59.062600+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.10","version":"0'0","reported_seq":131,"reported_epoch":119,"state":"active+clean","last_fresh":"2024-03-11T21:28:58.168118+0000","last_change":"2024-03-11T21:28:37.151842+0000","last_active":"2024-03-11T21:28:58.168118+0000","last_peered":"2024-03-11T21:28:58.168118+0000","last_clean":"2024-03-11T21:28:58.168118+0000","last_became_active":"2024-03-11T21:25:38.533138+0000","last_became_peered":"2024-03-11T21:25:38.533138+0000","last_unstale":"2024-03-11T21:28:58.168118+0000","last_undegraded":"2024-03-11T21:28:58.168118+0000","last_fullsized":"2024-03-11T21:28:58.168118+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:28:37.151573+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:37.512018+0000","last_clean_scrub_stamp":"2024-03-11T21:28:37.151573+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:29:59.767185+0000","scrub_duration":32,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,0],"acting":[2,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.16","version":"74'9","reported_seq":144,"reported_epoch":119,"state":"active+clean","last_fresh":"2024-03-11T21:28:58.164771+0000","last_change":"2024-03-11T21:28:10.399010+0000","last_active":"2024-03-11T21:28:58.164771+0000","last_peered":"2024-03-11T21:28:58.164771+0000","last_clean":"2024-03-11T21:28:58.164771+0000","last_became_active":"2024-03-11T21:25:40.763434+0000","last_became_peered":"2024-03-11T21:25:40.763434+0000","last_unstale":"2024-03-11T21:28:58.164771+0000","last_undegraded":"2024-03-11T21:28:58.164771+0000","last_fullsized":"2024-03-11T21:28:58.164771+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'9","last_scrub_stamp":"2024-03-11T21:28:10.398261+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:39.525554+0000","last_clean_scrub_stamp":"2024-03-11T21:28:10.398261+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:29:12.053153+0000","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.a","version":"119'6819","reported_seq":6868,"reported_epoch":119,"state":"active+clean","last_fresh":"2024-03-11T21:29:02.127669+0000","last_change":"2024-03-11T21:27:36.693277+0000","last_active":"2024-03-11T21:29:02.127669+0000","last_peered":"2024-03-11T21:29:02.127669+0000","last_clean":"2024-03-11T21:29:02.127669+0000","last_became_active":"2024-03-11T21:27:36.692988+0000","last_became_peered":"2024-03-11T21:27:36.692988+0000","last_unstale":"2024-03-11T21:29:02.127669+0000","last_undegraded":"2024-03-11T21:29:02.127669+0000","last_fullsized":"2024-03-11T21:29:02.127669+0000","mapping_epoch":99,"log_start":"105'2200","ondisk_log_start":"105'2200","created":99,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:27:35.667522+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:27:35.667522+0000","last_clean_scrub_stamp":"2024-03-11T21:27:35.667522+0000","objects_scrubbed":0,"log_size":4619,"log_dups_size":0,"ondisk_log_size":4619,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:29:03.493326+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":446889984,"num_objects":6819,"num_object_clones":0,"num_object_copies":20457,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6819,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6819,"num_write_kb":436416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5],"acting":[3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.b","version":"0'0","reported_seq":111,"reported_epoch":119,"state":"active+clean","last_fresh":"2024-03-11T21:28:58.170375+0000","last_change":"2024-03-11T21:28:03.724789+0000","last_active":"2024-03-11T21:28:58.170375+0000","last_peered":"2024-03-11T21:28:58.170375+0000","last_clean":"2024-03-11T21:28:58.170375+0000","last_became_active":"2024-03-11T21:25:44.590071+0000","last_became_peered":"2024-03-11T21:25:44.590071+0000","last_unstale":"2024-03-11T21:28:58.170375+0000","last_undegraded":"2024-03-11T21:28:58.170375+0000","last_fullsized":"2024-03-11T21:28:58.170375+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:28:03.724666+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:43.559452+0000","last_clean_scrub_stamp":"2024-03-11T21:28:03.724666+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:29:23.617411+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.f","version":"67'2","reported_seq":136,"reported_epoch":119,"state":"active+clean","last_fresh":"2024-03-11T21:28:58.156904+0000","last_change":"2024-03-11T21:27:21.959075+0000","last_active":"2024-03-11T21:28:58.156904+0000","last_peered":"2024-03-11T21:28:58.156904+0000","last_clean":"2024-03-11T21:28:58.156904+0000","last_became_active":"2024-03-11T21:25:38.537711+0000","last_became_peered":"2024-03-11T21:25:38.537711+0000","last_unstale":"2024-03-11T21:28:58.156904+0000","last_undegraded":"2024-03-11T21:28:58.156904+0000","last_fullsized":"2024-03-11T21:28:58.156904+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'2","last_scrub_stamp":"2024-03-11T21:27:21.958970+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:37.512018+0000","last_clean_scrub_stamp":"2024-03-11T21:27:21.958970+0000","objects_scrubbed":2,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:29:03.083242+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":92,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":10,"num_read_kb":10,"num_write":4,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,7],"acting":[4,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.9","version":"74'12","reported_seq":143,"reported_epoch":119,"state":"active+clean","last_fresh":"2024-03-11T21:28:58.155813+0000","last_change":"2024-03-11T21:28:07.735179+0000","last_active":"2024-03-11T21:28:58.155813+0000","last_peered":"2024-03-11T21:28:58.155813+0000","last_clean":"2024-03-11T21:28:58.155813+0000","last_became_active":"2024-03-11T21:25:40.552267+0000","last_became_peered":"2024-03-11T21:25:40.552267+0000","last_unstale":"2024-03-11T21:28:58.155813+0000","last_undegraded":"2024-03-11T21:28:58.155813+0000","last_fullsized":"2024-03-11T21:28:58.155813+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'12","last_scrub_stamp":"2024-03-11T21:28:07.735010+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:39.525554+0000","last_clean_scrub_stamp":"2024-03-11T21:28:07.735010+0000","objects_scrubbed":5,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:29:16.999453+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":25,"num_read_kb":16,"num_write":14,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,3],"acting":[4,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":115,"reported_epoch":119,"state":"active+clean","last_fresh":"2024-03-11T21:28:58.168026+0000","last_change":"2024-03-11T21:27:08.954258+0000","last_active":"2024-03-11T21:28:58.168026+0000","last_peered":"2024-03-11T21:28:58.168026+0000","last_clean":"2024-03-11T21:28:58.168026+0000","last_became_active":"2024-03-11T21:25:42.573987+0000","last_became_peered":"2024-03-11T21:25:42.573987+0000","last_unstale":"2024-03-11T21:28:58.168026+0000","last_undegraded":"2024-03-11T21:28:58.168026+0000","last_fullsized":"2024-03-11T21:28:58.168026+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:27:08.954143+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:41.544550+0000","last_clean_scrub_stamp":"2024-03-11T21:27:08.954143+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:29:00.225082+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.1","version":"119'6915","reported_seq":6965,"reported_epoch":119,"state":"active+clean","last_fresh":"2024-03-11T21:29:01.933115+0000","last_change":"2024-03-11T21:27:36.697329+0000","last_active":"2024-03-11T21:29:01.933115+0000","last_peered":"2024-03-11T21:29:01.933115+0000","last_clean":"2024-03-11T21:29:01.933115+0000","last_became_active":"2024-03-11T21:27:36.696990+0000","last_became_peered":"2024-03-11T21:27:36.696990+0000","last_unstale":"2024-03-11T21:29:01.933115+0000","last_undegraded":"2024-03-11T21:29:01.933115+0000","last_fullsized":"2024-03-11T21:29:01.933115+0000","mapping_epoch":99,"log_start":"105'1800","ondisk_log_start":"105'1800","created":99,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:27:35.667522+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:27:35.667522+0000","last_clean_scrub_stamp":"2024-03-11T21:27:35.667522+0000","objects_scrubbed":0,"log_size":5115,"log_dups_size":0,"ondisk_log_size":5115,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:29:03.083323+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":453181440,"num_objects":6915,"num_object_clones":0,"num_object_copies":20745,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6915,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6915,"num_write_kb":442560,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.0","version":"0'0","reported_seq":112,"reported_epoch":119,"state":"active+clean","last_fresh":"2024-03-11T21:28:58.156851+0000","last_change":"2024-03-11T21:27:23.389441+0000","last_active":"2024-03-11T21:28:58.156851+0000","last_peered":"2024-03-11T21:28:58.156851+0000","last_clean":"2024-03-11T21:28:58.156851+0000","last_became_active":"2024-03-11T21:25:44.584565+0000","last_became_peered":"2024-03-11T21:25:44.584565+0000","last_unstale":"2024-03-11T21:28:58.156851+0000","last_undegraded":"2024-03-11T21:28:58.156851+0000","last_fullsized":"2024-03-11T21:28:58.156851+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:27:23.389362+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:43.559452+0000","last_clean_scrub_stamp":"2024-03-11T21:27:23.389362+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":132,"reported_epoch":119,"state":"active+clean","last_fresh":"2024-03-11T21:28:58.160888+0000","last_change":"2024-03-11T21:28:27.403947+0000","last_active":"2024-03-11T21:28:58.160888+0000","last_peered":"2024-03-11T21:28:58.160888+0000","last_clean":"2024-03-11T21:28:58.160888+0000","last_became_active":"2024-03-11T21:25:38.541760+0000","last_became_peered":"2024-03-11T21:25:38.541760+0000","last_unstale":"2024-03-11T21:28:58.160888+0000","last_undegraded":"2024-03-11T21:28:58.160888+0000","last_fullsized":"2024-03-11T21:28:58.160888+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:28:27.403692+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:37.512018+0000","last_clean_scrub_stamp":"2024-03-11T21:28:27.403692+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:29:31.254351+0000","scrub_duration":34,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,7],"acting":[1,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.2","version":"94'15","reported_seq":148,"reported_epoch":119,"state":"active+clean","last_fresh":"2024-03-11T21:28:58.163116+0000","last_change":"2024-03-11T21:28:08.398044+0000","last_active":"2024-03-11T21:28:58.163116+0000","last_peered":"2024-03-11T21:28:58.163116+0000","last_clean":"2024-03-11T21:28:58.163116+0000","last_became_active":"2024-03-11T21:25:40.548833+0000","last_became_peered":"2024-03-11T21:25:40.548833+0000","last_unstale":"2024-03-11T21:28:58.163116+0000","last_undegraded":"2024-03-11T21:28:58.163116+0000","last_fullsized":"2024-03-11T21:28:58.163116+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"94'15","last_scrub_stamp":"2024-03-11T21:28:08.397342+0000","last_deep_scrub":"94'15","last_deep_scrub_stamp":"2024-03-11T21:26:49.993060+0000","last_clean_scrub_stamp":"2024-03-11T21:28:08.397342+0000","objects_scrubbed":5,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:29:11.389113+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.3","version":"74'8","reported_seq":165,"reported_epoch":119,"state":"active+clean","last_fresh":"2024-03-11T21:28:58.157558+0000","last_change":"2024-03-11T21:27:06.321926+0000","last_active":"2024-03-11T21:28:58.157558+0000","last_peered":"2024-03-11T21:28:58.157558+0000","last_clean":"2024-03-11T21:28:58.157558+0000","last_became_active":"2024-03-11T21:25:42.578161+0000","last_became_peered":"2024-03-11T21:25:42.578161+0000","last_unstale":"2024-03-11T21:28:58.157558+0000","last_undegraded":"2024-03-11T21:28:58.157558+0000","last_fullsized":"2024-03-11T21:28:58.157558+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'8","last_scrub_stamp":"2024-03-11T21:27:06.321832+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:41.544550+0000","last_clean_scrub_stamp":"2024-03-11T21:27:06.321832+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:29:02.343533+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,5],"acting":[0,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.7","version":"119'6871","reported_seq":6921,"reported_epoch":119,"state":"active+clean","last_fresh":"2024-03-11T21:29:01.988545+0000","last_change":"2024-03-11T21:27:36.695426+0000","last_active":"2024-03-11T21:29:01.988545+0000","last_peered":"2024-03-11T21:29:01.988545+0000","last_clean":"2024-03-11T21:29:01.988545+0000","last_became_active":"2024-03-11T21:27:36.695055+0000","last_became_peered":"2024-03-11T21:27:36.695055+0000","last_unstale":"2024-03-11T21:29:01.988545+0000","last_undegraded":"2024-03-11T21:29:01.988545+0000","last_fullsized":"2024-03-11T21:29:01.988545+0000","mapping_epoch":99,"log_start":"105'600","ondisk_log_start":"105'600","created":99,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:27:35.667522+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:27:35.667522+0000","last_clean_scrub_stamp":"2024-03-11T21:27:35.667522+0000","objects_scrubbed":0,"log_size":6271,"log_dups_size":0,"ondisk_log_size":6271,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:29:04.544436+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":450297856,"num_objects":6871,"num_object_clones":0,"num_object_copies":20613,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6871,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6871,"num_write_kb":439744,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,3],"acting":[6,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":119,"reported_epoch":119,"state":"active+clean","last_fresh":"2024-03-11T21:28:58.169752+0000","last_change":"2024-03-11T21:28:30.563856+0000","last_active":"2024-03-11T21:28:58.169752+0000","last_peered":"2024-03-11T21:28:58.169752+0000","last_clean":"2024-03-11T21:28:58.169752+0000","last_became_active":"2024-03-11T21:25:44.585223+0000","last_became_peered":"2024-03-11T21:25:44.585223+0000","last_unstale":"2024-03-11T21:28:58.169752+0000","last_undegraded":"2024-03-11T21:28:58.169752+0000","last_fullsized":"2024-03-11T21:28:58.169752+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:28:30.563712+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:28:30.563712+0000","last_clean_scrub_stamp":"2024-03-11T21:28:30.563712+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:29:35.182681+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,7],"acting":[3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":131,"reported_epoch":119,"state":"active+clean","last_fresh":"2024-03-11T21:28:58.163889+0000","last_change":"2024-03-11T21:28:10.753819+0000","last_active":"2024-03-11T21:28:58.163889+0000","last_peered":"2024-03-11T21:28:58.163889+0000","last_clean":"2024-03-11T21:28:58.163889+0000","last_became_active":"2024-03-11T21:25:38.538540+0000","last_became_peered":"2024-03-11T21:25:38.538540+0000","last_unstale":"2024-03-11T21:28:58.163889+0000","last_undegraded":"2024-03-11T21:28:58.163889+0000","last_fullsized":"2024-03-11T21:28:58.163889+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:28:10.753706+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:37.512018+0000","last_clean_scrub_stamp":"2024-03-11T21:28:10.753706+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:29:15.610522+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.4","version":"74'28","reported_seq":217,"reported_epoch":119,"state":"active+clean","last_fresh":"2024-03-11T21:28:58.160838+0000","last_change":"2024-03-11T21:28:21.455801+0000","last_active":"2024-03-11T21:28:58.160838+0000","last_peered":"2024-03-11T21:28:58.160838+0000","last_clean":"2024-03-11T21:28:58.160838+0000","last_became_active":"2024-03-11T21:25:40.552902+0000","last_became_peered":"2024-03-11T21:25:40.552902+0000","last_unstale":"2024-03-11T21:28:58.160838+0000","last_undegraded":"2024-03-11T21:28:58.160838+0000","last_fullsized":"2024-03-11T21:28:58.160838+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'28","last_scrub_stamp":"2024-03-11T21:28:21.455629+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:39.525554+0000","last_clean_scrub_stamp":"2024-03-11T21:28:21.455629+0000","objects_scrubbed":10,"log_size":28,"log_dups_size":0,"ondisk_log_size":28,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:29:48.292369+0000","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":358,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":48,"num_read_kb":33,"num_write":26,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3],"acting":[1,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":116,"reported_epoch":119,"state":"active+clean","last_fresh":"2024-03-11T21:28:58.170367+0000","last_change":"2024-03-11T21:27:20.393288+0000","last_active":"2024-03-11T21:28:58.170367+0000","last_peered":"2024-03-11T21:28:58.170367+0000","last_clean":"2024-03-11T21:28:58.170367+0000","last_became_active":"2024-03-11T21:25:42.572673+0000","last_became_peered":"2024-03-11T21:25:42.572673+0000","last_unstale":"2024-03-11T21:28:58.170367+0000","last_undegraded":"2024-03-11T21:28:58.170367+0000","last_fullsized":"2024-03-11T21:28:58.170367+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:27:20.393205+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:41.544550+0000","last_clean_scrub_stamp":"2024-03-11T21:27:20.393205+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.4","version":"119'6894","reported_seq":6944,"reported_epoch":119,"state":"active+clean","last_fresh":"2024-03-11T21:29:02.012449+0000","last_change":"2024-03-11T21:27:36.691053+0000","last_active":"2024-03-11T21:29:02.012449+0000","last_peered":"2024-03-11T21:29:02.012449+0000","last_clean":"2024-03-11T21:29:02.012449+0000","last_became_active":"2024-03-11T21:27:36.690751+0000","last_became_peered":"2024-03-11T21:27:36.690751+0000","last_unstale":"2024-03-11T21:29:02.012449+0000","last_undegraded":"2024-03-11T21:29:02.012449+0000","last_fullsized":"2024-03-11T21:29:02.012449+0000","mapping_epoch":99,"log_start":"105'1400","ondisk_log_start":"105'1400","created":99,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:27:35.667522+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:27:35.667522+0000","last_clean_scrub_stamp":"2024-03-11T21:27:35.667522+0000","objects_scrubbed":0,"log_size":5494,"log_dups_size":0,"ondisk_log_size":5494,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:29:06.439024+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":451805184,"num_objects":6894,"num_object_clones":0,"num_object_copies":20682,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6894,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6894,"num_write_kb":441216,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,1],"acting":[0,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.5","version":"0'0","reported_seq":111,"reported_epoch":119,"state":"active+clean","last_fresh":"2024-03-11T21:28:58.169737+0000","last_change":"2024-03-11T21:27:05.338089+0000","last_active":"2024-03-11T21:28:58.169737+0000","last_peered":"2024-03-11T21:28:58.169737+0000","last_clean":"2024-03-11T21:28:58.169737+0000","last_became_active":"2024-03-11T21:25:45.302120+0000","last_became_peered":"2024-03-11T21:25:45.302120+0000","last_unstale":"2024-03-11T21:28:58.169737+0000","last_undegraded":"2024-03-11T21:28:58.169737+0000","last_fullsized":"2024-03-11T21:28:58.169737+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:27:05.337991+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:43.559452+0000","last_clean_scrub_stamp":"2024-03-11T21:27:05.337991+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,3],"acting":[7,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.0","version":"76'5","reported_seq":340,"reported_epoch":119,"state":"active+clean","last_fresh":"2024-03-11T21:29:00.937881+0000","last_change":"2024-03-11T21:28:26.389894+0000","last_active":"2024-03-11T21:29:00.937881+0000","last_peered":"2024-03-11T21:29:00.937881+0000","last_clean":"2024-03-11T21:29:00.937881+0000","last_became_active":"2024-03-11T21:25:38.536890+0000","last_became_peered":"2024-03-11T21:25:38.536890+0000","last_unstale":"2024-03-11T21:29:00.937881+0000","last_undegraded":"2024-03-11T21:29:00.937881+0000","last_fullsized":"2024-03-11T21:29:00.937881+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"76'5","last_scrub_stamp":"2024-03-11T21:28:26.389561+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:37.512018+0000","last_clean_scrub_stamp":"2024-03-11T21:28:26.389561+0000","objects_scrubbed":1,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:29:43.918091+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0.000753559,"stat_sum":{"num_bytes":389,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":198,"num_read_kb":193,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,6],"acting":[1,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":131,"reported_epoch":119,"state":"active+clean","last_fresh":"2024-03-11T21:28:58.161862+0000","last_change":"2024-03-11T21:28:17.037045+0000","last_active":"2024-03-11T21:28:58.161862+0000","last_peered":"2024-03-11T21:28:58.161862+0000","last_clean":"2024-03-11T21:28:58.161862+0000","last_became_active":"2024-03-11T21:25:38.532984+0000","last_became_peered":"2024-03-11T21:25:38.532984+0000","last_unstale":"2024-03-11T21:28:58.161862+0000","last_undegraded":"2024-03-11T21:28:58.161862+0000","last_fullsized":"2024-03-11T21:28:58.161862+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:28:17.036907+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:26:48.997454+0000","last_clean_scrub_stamp":"2024-03-11T21:28:17.036907+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:29:25.260052+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.7","version":"97'18","reported_seq":160,"reported_epoch":119,"state":"active+clean","last_fresh":"2024-03-11T21:28:58.159010+0000","last_change":"2024-03-11T21:28:33.258307+0000","last_active":"2024-03-11T21:28:58.159010+0000","last_peered":"2024-03-11T21:28:58.159010+0000","last_clean":"2024-03-11T21:28:58.159010+0000","last_became_active":"2024-03-11T21:25:40.541769+0000","last_became_peered":"2024-03-11T21:25:40.541769+0000","last_unstale":"2024-03-11T21:28:58.159010+0000","last_undegraded":"2024-03-11T21:28:58.159010+0000","last_fullsized":"2024-03-11T21:28:58.159010+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"97'18","last_scrub_stamp":"2024-03-11T21:28:33.258170+0000","last_deep_scrub":"97'18","last_deep_scrub_stamp":"2024-03-11T21:27:19.239404+0000","last_clean_scrub_stamp":"2024-03-11T21:28:33.258170+0000","objects_scrubbed":5,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:29:43.862175+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":19,"num_write":16,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":115,"reported_epoch":119,"state":"active+clean","last_fresh":"2024-03-11T21:28:58.163344+0000","last_change":"2024-03-11T21:27:02.039134+0000","last_active":"2024-03-11T21:28:58.163344+0000","last_peered":"2024-03-11T21:28:58.163344+0000","last_clean":"2024-03-11T21:28:58.163344+0000","last_became_active":"2024-03-11T21:25:42.581710+0000","last_became_peered":"2024-03-11T21:25:42.581710+0000","last_unstale":"2024-03-11T21:28:58.163344+0000","last_undegraded":"2024-03-11T21:28:58.163344+0000","last_fullsized":"2024-03-11T21:28:58.163344+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:27:02.039041+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:41.544550+0000","last_clean_scrub_stamp":"2024-03-11T21:27:02.039041+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:29:00.224975+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7],"acting":[2,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.6","version":"119'6872","reported_seq":6922,"reported_epoch":119,"state":"active+clean","last_fresh":"2024-03-11T21:29:02.083527+0000","last_change":"2024-03-11T21:27:36.695308+0000","last_active":"2024-03-11T21:29:02.083527+0000","last_peered":"2024-03-11T21:29:02.083527+0000","last_clean":"2024-03-11T21:29:02.083527+0000","last_became_active":"2024-03-11T21:27:36.695018+0000","last_became_peered":"2024-03-11T21:27:36.695018+0000","last_unstale":"2024-03-11T21:29:02.083527+0000","last_undegraded":"2024-03-11T21:29:02.083527+0000","last_fullsized":"2024-03-11T21:29:02.083527+0000","mapping_epoch":99,"log_start":"105'1400","ondisk_log_start":"105'1400","created":99,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:27:35.667522+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:27:35.667522+0000","last_clean_scrub_stamp":"2024-03-11T21:27:35.667522+0000","objects_scrubbed":0,"log_size":5472,"log_dups_size":0,"ondisk_log_size":5472,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:29:02.343603+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":450363392,"num_objects":6872,"num_object_clones":0,"num_object_copies":20616,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6872,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6872,"num_write_kb":439808,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.7","version":"74'1","reported_seq":112,"reported_epoch":119,"state":"active+clean","last_fresh":"2024-03-11T21:28:58.165211+0000","last_change":"2024-03-11T21:28:05.716314+0000","last_active":"2024-03-11T21:28:58.165211+0000","last_peered":"2024-03-11T21:28:58.165211+0000","last_clean":"2024-03-11T21:28:58.165211+0000","last_became_active":"2024-03-11T21:25:44.591376+0000","last_became_peered":"2024-03-11T21:25:44.591376+0000","last_unstale":"2024-03-11T21:28:58.165211+0000","last_undegraded":"2024-03-11T21:28:58.165211+0000","last_fullsized":"2024-03-11T21:28:58.165211+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'1","last_scrub_stamp":"2024-03-11T21:28:05.716180+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:43.559452+0000","last_clean_scrub_stamp":"2024-03-11T21:28:05.716180+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:29:28.966516+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,4],"acting":[5,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2","version":"71'2","reported_seq":140,"reported_epoch":119,"state":"active+clean","last_fresh":"2024-03-11T21:28:58.169091+0000","last_change":"2024-03-11T21:28:15.701596+0000","last_active":"2024-03-11T21:28:58.169091+0000","last_peered":"2024-03-11T21:28:58.169091+0000","last_clean":"2024-03-11T21:28:58.169091+0000","last_became_active":"2024-03-11T21:25:38.539413+0000","last_became_peered":"2024-03-11T21:25:38.539413+0000","last_unstale":"2024-03-11T21:28:58.169091+0000","last_undegraded":"2024-03-11T21:28:58.169091+0000","last_fullsized":"2024-03-11T21:28:58.169091+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"71'2","last_scrub_stamp":"2024-03-11T21:28:15.701477+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:37.512018+0000","last_clean_scrub_stamp":"2024-03-11T21:28:15.701477+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:29:26.797762+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0.0014206449999999999,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":137,"reported_epoch":119,"state":"active+clean","last_fresh":"2024-03-11T21:28:58.165262+0000","last_change":"2024-03-11T21:28:12.795411+0000","last_active":"2024-03-11T21:28:58.165262+0000","last_peered":"2024-03-11T21:28:58.165262+0000","last_clean":"2024-03-11T21:28:58.165262+0000","last_became_active":"2024-03-11T21:26:11.216188+0000","last_became_peered":"2024-03-11T21:26:11.216188+0000","last_unstale":"2024-03-11T21:28:58.165262+0000","last_undegraded":"2024-03-11T21:28:58.165262+0000","last_fullsized":"2024-03-11T21:28:58.165262+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:28:12.795259+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:37.512018+0000","last_clean_scrub_stamp":"2024-03-11T21:28:12.795259+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:29:14.049738+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,4],"acting":[5,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"87'128","reported_seq":376,"reported_epoch":119,"state":"active+clean","last_fresh":"2024-03-11T21:28:58.169393+0000","last_change":"2024-03-11T21:28:03.750801+0000","last_active":"2024-03-11T21:28:58.169393+0000","last_peered":"2024-03-11T21:28:58.169393+0000","last_clean":"2024-03-11T21:28:58.169393+0000","last_became_active":"2024-03-11T21:25:30.502597+0000","last_became_peered":"2024-03-11T21:25:30.502597+0000","last_unstale":"2024-03-11T21:28:58.169393+0000","last_undegraded":"2024-03-11T21:28:58.169393+0000","last_fullsized":"2024-03-11T21:28:58.169393+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"87'128","last_scrub_stamp":"2024-03-11T21:28:03.750641+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:24.261974+0000","last_clean_scrub_stamp":"2024-03-11T21:28:03.750641+0000","objects_scrubbed":2,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:29:12.310597+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":212,"num_read_kb":306,"num_write":179,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1771104,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.5","version":"74'16","reported_seq":191,"reported_epoch":119,"state":"active+clean","last_fresh":"2024-03-11T21:28:58.165723+0000","last_change":"2024-03-11T21:28:30.831871+0000","last_active":"2024-03-11T21:28:58.165723+0000","last_peered":"2024-03-11T21:28:58.165723+0000","last_clean":"2024-03-11T21:28:58.165723+0000","last_became_active":"2024-03-11T21:25:40.761457+0000","last_became_peered":"2024-03-11T21:25:40.761457+0000","last_unstale":"2024-03-11T21:28:58.165723+0000","last_undegraded":"2024-03-11T21:28:58.165723+0000","last_fullsized":"2024-03-11T21:28:58.165723+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'16","last_scrub_stamp":"2024-03-11T21:28:30.831780+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:39.525554+0000","last_clean_scrub_stamp":"2024-03-11T21:28:30.831780+0000","objects_scrubbed":8,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:29:54.428390+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":154,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":25,"num_read_kb":15,"num_write":13,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":123,"reported_epoch":119,"state":"active+clean","last_fresh":"2024-03-11T21:28:58.168860+0000","last_change":"2024-03-11T21:28:12.639736+0000","last_active":"2024-03-11T21:28:58.168860+0000","last_peered":"2024-03-11T21:28:58.168860+0000","last_clean":"2024-03-11T21:28:58.168860+0000","last_became_active":"2024-03-11T21:25:42.586226+0000","last_became_peered":"2024-03-11T21:25:42.586226+0000","last_unstale":"2024-03-11T21:28:58.168860+0000","last_undegraded":"2024-03-11T21:28:58.168860+0000","last_fullsized":"2024-03-11T21:28:58.168860+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:28:12.639653+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:41.544550+0000","last_clean_scrub_stamp":"2024-03-11T21:28:12.639653+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:29:22.433321+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,5],"acting":[7,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.5","version":"119'6821","reported_seq":6870,"reported_epoch":119,"state":"active+clean","last_fresh":"2024-03-11T21:29:01.985132+0000","last_change":"2024-03-11T21:27:36.692195+0000","last_active":"2024-03-11T21:29:01.985132+0000","last_peered":"2024-03-11T21:29:01.985132+0000","last_clean":"2024-03-11T21:29:01.985132+0000","last_became_active":"2024-03-11T21:27:36.691428+0000","last_became_peered":"2024-03-11T21:27:36.691428+0000","last_unstale":"2024-03-11T21:29:01.985132+0000","last_undegraded":"2024-03-11T21:29:01.985132+0000","last_fullsized":"2024-03-11T21:29:01.985132+0000","mapping_epoch":99,"log_start":"105'1900","ondisk_log_start":"105'1900","created":99,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:27:35.667522+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:27:35.667522+0000","last_clean_scrub_stamp":"2024-03-11T21:27:35.667522+0000","objects_scrubbed":0,"log_size":4921,"log_dups_size":0,"ondisk_log_size":4921,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":447021056,"num_objects":6821,"num_object_clones":0,"num_object_copies":20463,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6821,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6821,"num_write_kb":436544,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,0],"acting":[7,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.4","version":"0'0","reported_seq":120,"reported_epoch":119,"state":"active+clean","last_fresh":"2024-03-11T21:28:58.161122+0000","last_change":"2024-03-11T21:28:29.376413+0000","last_active":"2024-03-11T21:28:58.161122+0000","last_peered":"2024-03-11T21:28:58.161122+0000","last_clean":"2024-03-11T21:28:58.161122+0000","last_became_active":"2024-03-11T21:25:44.594188+0000","last_became_peered":"2024-03-11T21:25:44.594188+0000","last_unstale":"2024-03-11T21:28:58.161122+0000","last_undegraded":"2024-03-11T21:28:58.161122+0000","last_fullsized":"2024-03-11T21:28:58.161122+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:28:29.375618+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:43.559452+0000","last_clean_scrub_stamp":"2024-03-11T21:28:29.375618+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:29:49.691757+0000","scrub_duration":32,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":139,"reported_epoch":119,"state":"active+clean","last_fresh":"2024-03-11T21:28:58.157191+0000","last_change":"2024-03-11T21:28:06.322812+0000","last_active":"2024-03-11T21:28:58.157191+0000","last_peered":"2024-03-11T21:28:58.157191+0000","last_clean":"2024-03-11T21:28:58.157191+0000","last_became_active":"2024-03-11T21:25:38.543834+0000","last_became_peered":"2024-03-11T21:25:38.543834+0000","last_unstale":"2024-03-11T21:28:58.157191+0000","last_undegraded":"2024-03-11T21:28:58.157191+0000","last_fullsized":"2024-03-11T21:28:58.157191+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:28:06.322702+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:37.512018+0000","last_clean_scrub_stamp":"2024-03-11T21:28:06.322702+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:29:32.617923+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0.00089342499999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":131,"reported_epoch":119,"state":"active+clean","last_fresh":"2024-03-11T21:28:58.169305+0000","last_change":"2024-03-11T21:28:17.526308+0000","last_active":"2024-03-11T21:28:58.169305+0000","last_peered":"2024-03-11T21:28:58.169305+0000","last_clean":"2024-03-11T21:28:58.169305+0000","last_became_active":"2024-03-11T21:25:38.541617+0000","last_became_peered":"2024-03-11T21:25:38.541617+0000","last_unstale":"2024-03-11T21:28:58.169305+0000","last_undegraded":"2024-03-11T21:28:58.169305+0000","last_fullsized":"2024-03-11T21:28:58.169305+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:28:17.526232+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:37.512018+0000","last_clean_scrub_stamp":"2024-03-11T21:28:17.526232+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:29:18.725065+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,0],"acting":[7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.6","version":"74'12","reported_seq":146,"reported_epoch":119,"state":"active+clean","last_fresh":"2024-03-11T21:28:58.164502+0000","last_change":"2024-03-11T21:28:30.335163+0000","last_active":"2024-03-11T21:28:58.164502+0000","last_peered":"2024-03-11T21:28:58.164502+0000","last_clean":"2024-03-11T21:28:58.164502+0000","last_became_active":"2024-03-11T21:25:40.550388+0000","last_became_peered":"2024-03-11T21:25:40.550388+0000","last_unstale":"2024-03-11T21:28:58.164502+0000","last_undegraded":"2024-03-11T21:28:58.164502+0000","last_fullsized":"2024-03-11T21:28:58.164502+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'12","last_scrub_stamp":"2024-03-11T21:28:30.335019+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:39.525554+0000","last_clean_scrub_stamp":"2024-03-11T21:28:30.335019+0000","objects_scrubbed":6,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:29:42.338429+0000","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":18,"num_read_kb":12,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":115,"reported_epoch":119,"state":"active+clean","last_fresh":"2024-03-11T21:28:58.165121+0000","last_change":"2024-03-11T21:27:32.991020+0000","last_active":"2024-03-11T21:28:58.165121+0000","last_peered":"2024-03-11T21:28:58.165121+0000","last_clean":"2024-03-11T21:28:58.165121+0000","last_became_active":"2024-03-11T21:25:42.572324+0000","last_became_peered":"2024-03-11T21:25:42.572324+0000","last_unstale":"2024-03-11T21:28:58.165121+0000","last_undegraded":"2024-03-11T21:28:58.165121+0000","last_fullsized":"2024-03-11T21:28:58.165121+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:27:32.990943+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:41.544550+0000","last_clean_scrub_stamp":"2024-03-11T21:27:32.990943+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:29:01.738075+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.0","version":"119'6710","reported_seq":6758,"reported_epoch":119,"state":"active+clean","last_fresh":"2024-03-11T21:29:01.969264+0000","last_change":"2024-03-11T21:27:36.694516+0000","last_active":"2024-03-11T21:29:01.969264+0000","last_peered":"2024-03-11T21:29:01.969264+0000","last_clean":"2024-03-11T21:29:01.969264+0000","last_became_active":"2024-03-11T21:27:36.694232+0000","last_became_peered":"2024-03-11T21:27:36.694232+0000","last_unstale":"2024-03-11T21:29:01.969264+0000","last_undegraded":"2024-03-11T21:29:01.969264+0000","last_fullsized":"2024-03-11T21:29:01.969264+0000","mapping_epoch":99,"log_start":"105'900","ondisk_log_start":"105'900","created":99,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:27:35.667522+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:27:35.667522+0000","last_clean_scrub_stamp":"2024-03-11T21:27:35.667522+0000","objects_scrubbed":0,"log_size":5810,"log_dups_size":0,"ondisk_log_size":5810,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":439746560,"num_objects":6710,"num_object_clones":0,"num_object_copies":20130,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6710,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6710,"num_write_kb":429440,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3],"acting":[5,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":120,"reported_epoch":119,"state":"active+clean","last_fresh":"2024-03-11T21:28:58.159390+0000","last_change":"2024-03-11T21:28:10.405572+0000","last_active":"2024-03-11T21:28:58.159390+0000","last_peered":"2024-03-11T21:28:58.159390+0000","last_clean":"2024-03-11T21:28:58.159390+0000","last_became_active":"2024-03-11T21:25:45.301416+0000","last_became_peered":"2024-03-11T21:25:45.301416+0000","last_unstale":"2024-03-11T21:28:58.159390+0000","last_undegraded":"2024-03-11T21:28:58.159390+0000","last_fullsized":"2024-03-11T21:28:58.159390+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:28:10.405058+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:43.559452+0000","last_clean_scrub_stamp":"2024-03-11T21:28:10.405058+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:29:12.495627+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":131,"reported_epoch":119,"state":"active+clean","last_fresh":"2024-03-11T21:28:58.170214+0000","last_change":"2024-03-11T21:28:35.467464+0000","last_active":"2024-03-11T21:28:58.170214+0000","last_peered":"2024-03-11T21:28:58.170214+0000","last_clean":"2024-03-11T21:28:58.170214+0000","last_became_active":"2024-03-11T21:25:38.545148+0000","last_became_peered":"2024-03-11T21:25:38.545148+0000","last_unstale":"2024-03-11T21:28:58.170214+0000","last_undegraded":"2024-03-11T21:28:58.170214+0000","last_fullsized":"2024-03-11T21:28:58.170214+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:28:35.467339+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:26:57.333669+0000","last_clean_scrub_stamp":"2024-03-11T21:28:35.467339+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:29:55.375727+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.3","version":"74'19","reported_seq":164,"reported_epoch":119,"state":"active+clean","last_fresh":"2024-03-11T21:28:58.157191+0000","last_change":"2024-03-11T21:28:27.344174+0000","last_active":"2024-03-11T21:28:58.157191+0000","last_peered":"2024-03-11T21:28:58.157191+0000","last_clean":"2024-03-11T21:28:58.157191+0000","last_became_active":"2024-03-11T21:25:40.760594+0000","last_became_peered":"2024-03-11T21:25:40.760594+0000","last_unstale":"2024-03-11T21:28:58.157191+0000","last_undegraded":"2024-03-11T21:28:58.157191+0000","last_fullsized":"2024-03-11T21:28:58.157191+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'19","last_scrub_stamp":"2024-03-11T21:28:27.344026+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:39.525554+0000","last_clean_scrub_stamp":"2024-03-11T21:28:27.344026+0000","objects_scrubbed":8,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:29:39.040372+0000","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":39,"num_read_kb":25,"num_write":22,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":124,"reported_epoch":119,"state":"active+clean","last_fresh":"2024-03-11T21:28:58.164876+0000","last_change":"2024-03-11T21:28:33.767437+0000","last_active":"2024-03-11T21:28:58.164876+0000","last_peered":"2024-03-11T21:28:58.164876+0000","last_clean":"2024-03-11T21:28:58.164876+0000","last_became_active":"2024-03-11T21:25:42.579045+0000","last_became_peered":"2024-03-11T21:25:42.579045+0000","last_unstale":"2024-03-11T21:28:58.164876+0000","last_undegraded":"2024-03-11T21:28:58.164876+0000","last_fullsized":"2024-03-11T21:28:58.164876+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:28:33.767315+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:41.544550+0000","last_clean_scrub_stamp":"2024-03-11T21:28:33.767315+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:29:36.711423+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.3","version":"119'6769","reported_seq":6818,"reported_epoch":119,"state":"active+clean","last_fresh":"2024-03-11T21:29:02.038081+0000","last_change":"2024-03-11T21:27:36.693897+0000","last_active":"2024-03-11T21:29:02.038081+0000","last_peered":"2024-03-11T21:29:02.038081+0000","last_clean":"2024-03-11T21:29:02.038081+0000","last_became_active":"2024-03-11T21:27:36.693506+0000","last_became_peered":"2024-03-11T21:27:36.693506+0000","last_unstale":"2024-03-11T21:29:02.038081+0000","last_undegraded":"2024-03-11T21:29:02.038081+0000","last_fullsized":"2024-03-11T21:29:02.038081+0000","mapping_epoch":99,"log_start":"105'2100","ondisk_log_start":"105'2100","created":99,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:27:35.667522+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:27:35.667522+0000","last_clean_scrub_stamp":"2024-03-11T21:27:35.667522+0000","objects_scrubbed":0,"log_size":4669,"log_dups_size":0,"ondisk_log_size":4669,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:29:02.440859+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":443613184,"num_objects":6769,"num_object_clones":0,"num_object_copies":20307,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6769,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6769,"num_write_kb":433216,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,4],"acting":[3,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":120,"reported_epoch":119,"state":"active+clean","last_fresh":"2024-03-11T21:28:58.155400+0000","last_change":"2024-03-11T21:28:27.786854+0000","last_active":"2024-03-11T21:28:58.155400+0000","last_peered":"2024-03-11T21:28:58.155400+0000","last_clean":"2024-03-11T21:28:58.155400+0000","last_became_active":"2024-03-11T21:25:44.583638+0000","last_became_peered":"2024-03-11T21:25:44.583638+0000","last_unstale":"2024-03-11T21:28:58.155400+0000","last_undegraded":"2024-03-11T21:28:58.155400+0000","last_fullsized":"2024-03-11T21:28:58.155400+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:28:27.786740+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:43.559452+0000","last_clean_scrub_stamp":"2024-03-11T21:28:27.786740+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:29:30.840139+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.6","version":"67'1","reported_seq":133,"reported_epoch":119,"state":"active+clean","last_fresh":"2024-03-11T21:28:58.160522+0000","last_change":"2024-03-11T21:28:19.473355+0000","last_active":"2024-03-11T21:28:58.160522+0000","last_peered":"2024-03-11T21:28:58.160522+0000","last_clean":"2024-03-11T21:28:58.160522+0000","last_became_active":"2024-03-11T21:25:38.536142+0000","last_became_peered":"2024-03-11T21:25:38.536142+0000","last_unstale":"2024-03-11T21:28:58.160522+0000","last_undegraded":"2024-03-11T21:28:58.160522+0000","last_fullsized":"2024-03-11T21:28:58.160522+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'1","last_scrub_stamp":"2024-03-11T21:28:19.473186+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:37.512018+0000","last_clean_scrub_stamp":"2024-03-11T21:28:19.473186+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:29:25.075010+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,4],"acting":[1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.0","version":"93'26","reported_seq":167,"reported_epoch":119,"state":"active+clean","last_fresh":"2024-03-11T21:28:58.169172+0000","last_change":"2024-03-11T21:28:18.664637+0000","last_active":"2024-03-11T21:28:58.169172+0000","last_peered":"2024-03-11T21:28:58.169172+0000","last_clean":"2024-03-11T21:28:58.169172+0000","last_became_active":"2024-03-11T21:25:40.761716+0000","last_became_peered":"2024-03-11T21:25:40.761716+0000","last_unstale":"2024-03-11T21:28:58.169172+0000","last_undegraded":"2024-03-11T21:28:58.169172+0000","last_fullsized":"2024-03-11T21:28:58.169172+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"93'26","last_scrub_stamp":"2024-03-11T21:28:18.664409+0000","last_deep_scrub":"93'26","last_deep_scrub_stamp":"2024-03-11T21:26:40.680324+0000","last_clean_scrub_stamp":"2024-03-11T21:28:18.664409+0000","objects_scrubbed":8,"log_size":26,"log_dups_size":0,"ondisk_log_size":26,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:29:35.615493+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":34,"num_read_kb":22,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":124,"reported_epoch":119,"state":"active+clean","last_fresh":"2024-03-11T21:28:58.155899+0000","last_change":"2024-03-11T21:28:23.769677+0000","last_active":"2024-03-11T21:28:58.155899+0000","last_peered":"2024-03-11T21:28:58.155899+0000","last_clean":"2024-03-11T21:28:58.155899+0000","last_became_active":"2024-03-11T21:25:42.583647+0000","last_became_peered":"2024-03-11T21:25:42.583647+0000","last_unstale":"2024-03-11T21:28:58.155899+0000","last_undegraded":"2024-03-11T21:28:58.155899+0000","last_fullsized":"2024-03-11T21:28:58.155899+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:28:23.769586+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:26:51.863621+0000","last_clean_scrub_stamp":"2024-03-11T21:28:23.769586+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:29:32.731064+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,7],"acting":[4,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.2","version":"119'6858","reported_seq":6907,"reported_epoch":119,"state":"active+clean","last_fresh":"2024-03-11T21:29:02.110669+0000","last_change":"2024-03-11T21:27:36.692676+0000","last_active":"2024-03-11T21:29:02.110669+0000","last_peered":"2024-03-11T21:29:02.110669+0000","last_clean":"2024-03-11T21:29:02.110669+0000","last_became_active":"2024-03-11T21:27:36.692405+0000","last_became_peered":"2024-03-11T21:27:36.692405+0000","last_unstale":"2024-03-11T21:29:02.110669+0000","last_undegraded":"2024-03-11T21:29:02.110669+0000","last_fullsized":"2024-03-11T21:29:02.110669+0000","mapping_epoch":99,"log_start":"105'2223","ondisk_log_start":"105'2223","created":99,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:27:35.667522+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:27:35.667522+0000","last_clean_scrub_stamp":"2024-03-11T21:27:35.667522+0000","objects_scrubbed":0,"log_size":4635,"log_dups_size":0,"ondisk_log_size":4635,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:29:03.493403+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":449445888,"num_objects":6858,"num_object_clones":0,"num_object_copies":20574,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6858,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6858,"num_write_kb":438912,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.3","version":"0'0","reported_seq":119,"reported_epoch":119,"state":"active+clean","last_fresh":"2024-03-11T21:28:58.168488+0000","last_change":"2024-03-11T21:28:25.493599+0000","last_active":"2024-03-11T21:28:58.168488+0000","last_peered":"2024-03-11T21:28:58.168488+0000","last_clean":"2024-03-11T21:28:58.168488+0000","last_became_active":"2024-03-11T21:25:45.302719+0000","last_became_peered":"2024-03-11T21:25:45.302719+0000","last_unstale":"2024-03-11T21:28:58.168488+0000","last_undegraded":"2024-03-11T21:28:58.168488+0000","last_fullsized":"2024-03-11T21:28:58.168488+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:28:25.493325+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:28:25.493325+0000","last_clean_scrub_stamp":"2024-03-11T21:28:25.493325+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:29:27.683604+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,2],"acting":[7,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":132,"reported_epoch":119,"state":"active+clean","last_fresh":"2024-03-11T21:28:58.165737+0000","last_change":"2024-03-11T21:28:04.057863+0000","last_active":"2024-03-11T21:28:58.165737+0000","last_peered":"2024-03-11T21:28:58.165737+0000","last_clean":"2024-03-11T21:28:58.165737+0000","last_became_active":"2024-03-11T21:25:38.540013+0000","last_became_peered":"2024-03-11T21:25:38.540013+0000","last_unstale":"2024-03-11T21:28:58.165737+0000","last_undegraded":"2024-03-11T21:28:58.165737+0000","last_fullsized":"2024-03-11T21:28:58.165737+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:28:04.057757+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:37.512018+0000","last_clean_scrub_stamp":"2024-03-11T21:28:04.057757+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:29:12.016405+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,2],"acting":[6,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.1","version":"74'14","reported_seq":141,"reported_epoch":119,"state":"active+clean","last_fresh":"2024-03-11T21:28:58.155519+0000","last_change":"2024-03-11T21:27:22.932743+0000","last_active":"2024-03-11T21:28:58.155519+0000","last_peered":"2024-03-11T21:28:58.155519+0000","last_clean":"2024-03-11T21:28:58.155519+0000","last_became_active":"2024-03-11T21:25:40.546219+0000","last_became_peered":"2024-03-11T21:25:40.546219+0000","last_unstale":"2024-03-11T21:28:58.155519+0000","last_undegraded":"2024-03-11T21:28:58.155519+0000","last_fullsized":"2024-03-11T21:28:58.155519+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'14","last_scrub_stamp":"2024-03-11T21:27:22.932642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:39.525554+0000","last_clean_scrub_stamp":"2024-03-11T21:27:22.932642+0000","objects_scrubbed":7,"log_size":14,"log_dups_size":0,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:29:03.083270+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":21,"num_read_kb":14,"num_write":14,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6],"acting":[4,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.0","version":"98'9","reported_seq":165,"reported_epoch":119,"state":"active+clean","last_fresh":"2024-03-11T21:28:58.169116+0000","last_change":"2024-03-11T21:27:29.745799+0000","last_active":"2024-03-11T21:28:58.169116+0000","last_peered":"2024-03-11T21:28:58.169116+0000","last_clean":"2024-03-11T21:28:58.169116+0000","last_became_active":"2024-03-11T21:25:42.576126+0000","last_became_peered":"2024-03-11T21:25:42.576126+0000","last_unstale":"2024-03-11T21:28:58.169116+0000","last_undegraded":"2024-03-11T21:28:58.169116+0000","last_fullsized":"2024-03-11T21:28:58.169116+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"98'9","last_scrub_stamp":"2024-03-11T21:27:29.745711+0000","last_deep_scrub":"98'9","last_deep_scrub_stamp":"2024-03-11T21:27:29.745711+0000","last_clean_scrub_stamp":"2024-03-11T21:27:29.745711+0000","objects_scrubbed":1,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:29:03.493241+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.d","version":"119'6942","reported_seq":6992,"reported_epoch":119,"state":"active+clean","last_fresh":"2024-03-11T21:29:01.955163+0000","last_change":"2024-03-11T21:27:36.695471+0000","last_active":"2024-03-11T21:29:01.955163+0000","last_peered":"2024-03-11T21:29:01.955163+0000","last_clean":"2024-03-11T21:29:01.955163+0000","last_became_active":"2024-03-11T21:27:36.695186+0000","last_became_peered":"2024-03-11T21:27:36.695186+0000","last_unstale":"2024-03-11T21:29:01.955163+0000","last_undegraded":"2024-03-11T21:29:01.955163+0000","last_fullsized":"2024-03-11T21:29:01.955163+0000","mapping_epoch":99,"log_start":"105'600","ondisk_log_start":"105'600","created":99,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:27:35.667522+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:27:35.667522+0000","last_clean_scrub_stamp":"2024-03-11T21:27:35.667522+0000","objects_scrubbed":0,"log_size":6342,"log_dups_size":0,"ondisk_log_size":6342,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":454950912,"num_objects":6942,"num_object_clones":0,"num_object_copies":20826,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6942,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6942,"num_write_kb":444288,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,0],"acting":[6,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.c","version":"0'0","reported_seq":111,"reported_epoch":119,"state":"active+clean","last_fresh":"2024-03-11T21:28:58.170265+0000","last_change":"2024-03-11T21:28:02.703398+0000","last_active":"2024-03-11T21:28:58.170265+0000","last_peered":"2024-03-11T21:28:58.170265+0000","last_clean":"2024-03-11T21:28:58.170265+0000","last_became_active":"2024-03-11T21:25:45.302207+0000","last_became_peered":"2024-03-11T21:25:45.302207+0000","last_unstale":"2024-03-11T21:28:58.170265+0000","last_undegraded":"2024-03-11T21:28:58.170265+0000","last_fullsized":"2024-03-11T21:28:58.170265+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:28:02.703154+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:43.559452+0000","last_clean_scrub_stamp":"2024-03-11T21:28:02.703154+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:29:31.096224+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.f","version":"0'0","reported_seq":123,"reported_epoch":119,"state":"active+clean","last_fresh":"2024-03-11T21:28:58.164911+0000","last_change":"2024-03-11T21:28:03.765879+0000","last_active":"2024-03-11T21:28:58.164911+0000","last_peered":"2024-03-11T21:28:58.164911+0000","last_clean":"2024-03-11T21:28:58.164911+0000","last_became_active":"2024-03-11T21:25:42.575935+0000","last_became_peered":"2024-03-11T21:25:42.575935+0000","last_unstale":"2024-03-11T21:28:58.164911+0000","last_undegraded":"2024-03-11T21:28:58.164911+0000","last_fullsized":"2024-03-11T21:28:58.164911+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:28:03.765784+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:41.544550+0000","last_clean_scrub_stamp":"2024-03-11T21:28:03.765784+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:29:08.965186+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.8","version":"0'0","reported_seq":131,"reported_epoch":119,"state":"active+clean","last_fresh":"2024-03-11T21:28:58.169483+0000","last_change":"2024-03-11T21:28:29.562522+0000","last_active":"2024-03-11T21:28:58.169483+0000","last_peered":"2024-03-11T21:28:58.169483+0000","last_clean":"2024-03-11T21:28:58.169483+0000","last_became_active":"2024-03-11T21:25:38.540910+0000","last_became_peered":"2024-03-11T21:25:38.540910+0000","last_unstale":"2024-03-11T21:28:58.169483+0000","last_undegraded":"2024-03-11T21:28:58.169483+0000","last_fullsized":"2024-03-11T21:28:58.169483+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:28:29.562407+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:37.512018+0000","last_clean_scrub_stamp":"2024-03-11T21:28:29.562407+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:29:56.105469+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,1],"acting":[7,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.e","version":"109'16","reported_seq":152,"reported_epoch":119,"state":"active+clean","last_fresh":"2024-03-11T21:28:58.156644+0000","last_change":"2024-03-11T21:28:24.739573+0000","last_active":"2024-03-11T21:28:58.156644+0000","last_peered":"2024-03-11T21:28:58.156644+0000","last_clean":"2024-03-11T21:28:58.156644+0000","last_became_active":"2024-03-11T21:25:40.552178+0000","last_became_peered":"2024-03-11T21:25:40.552178+0000","last_unstale":"2024-03-11T21:28:58.156644+0000","last_undegraded":"2024-03-11T21:28:58.156644+0000","last_fullsized":"2024-03-11T21:28:58.156644+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"109'16","last_scrub_stamp":"2024-03-11T21:28:24.739395+0000","last_deep_scrub":"109'16","last_deep_scrub_stamp":"2024-03-11T21:28:24.739395+0000","last_clean_scrub_stamp":"2024-03-11T21:28:24.739395+0000","objects_scrubbed":5,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:29:43.197007+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.c","version":"119'6861","reported_seq":6911,"reported_epoch":119,"state":"active+clean","last_fresh":"2024-03-11T21:29:01.975265+0000","last_change":"2024-03-11T21:27:36.693712+0000","last_active":"2024-03-11T21:29:01.975265+0000","last_peered":"2024-03-11T21:29:01.975265+0000","last_clean":"2024-03-11T21:29:01.975265+0000","last_became_active":"2024-03-11T21:27:36.693325+0000","last_became_peered":"2024-03-11T21:27:36.693325+0000","last_unstale":"2024-03-11T21:29:01.975265+0000","last_undegraded":"2024-03-11T21:29:01.975265+0000","last_fullsized":"2024-03-11T21:29:01.975265+0000","mapping_epoch":99,"log_start":"105'600","ondisk_log_start":"105'600","created":99,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:27:35.667522+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:27:35.667522+0000","last_clean_scrub_stamp":"2024-03-11T21:27:35.667522+0000","objects_scrubbed":0,"log_size":6261,"log_dups_size":0,"ondisk_log_size":6261,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:29:03.772150+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":449642496,"num_objects":6861,"num_object_clones":0,"num_object_copies":20583,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6861,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6861,"num_write_kb":439104,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,7],"acting":[6,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.d","version":"0'0","reported_seq":119,"reported_epoch":119,"state":"active+clean","last_fresh":"2024-03-11T21:28:58.164244+0000","last_change":"2024-03-11T21:28:23.796927+0000","last_active":"2024-03-11T21:28:58.164244+0000","last_peered":"2024-03-11T21:28:58.164244+0000","last_clean":"2024-03-11T21:28:58.164244+0000","last_became_active":"2024-03-11T21:25:44.590333+0000","last_became_peered":"2024-03-11T21:25:44.590333+0000","last_unstale":"2024-03-11T21:28:58.164244+0000","last_undegraded":"2024-03-11T21:28:58.164244+0000","last_fullsized":"2024-03-11T21:28:58.164244+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:28:23.796842+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:43.559452+0000","last_clean_scrub_stamp":"2024-03-11T21:28:23.796842+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:29:49.049133+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.e","version":"74'8","reported_seq":165,"reported_epoch":119,"state":"active+clean","last_fresh":"2024-03-11T21:28:58.156443+0000","last_change":"2024-03-11T21:27:19.970228+0000","last_active":"2024-03-11T21:28:58.156443+0000","last_peered":"2024-03-11T21:28:58.156443+0000","last_clean":"2024-03-11T21:28:58.156443+0000","last_became_active":"2024-03-11T21:25:42.576742+0000","last_became_peered":"2024-03-11T21:25:42.576742+0000","last_unstale":"2024-03-11T21:28:58.156443+0000","last_undegraded":"2024-03-11T21:28:58.156443+0000","last_fullsized":"2024-03-11T21:28:58.156443+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'8","last_scrub_stamp":"2024-03-11T21:27:19.970141+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:41.544550+0000","last_clean_scrub_stamp":"2024-03-11T21:27:19.970141+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:29:03.083353+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,0],"acting":[4,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.9","version":"0'0","reported_seq":132,"reported_epoch":119,"state":"active+clean","last_fresh":"2024-03-11T21:28:58.159232+0000","last_change":"2024-03-11T21:28:07.421777+0000","last_active":"2024-03-11T21:28:58.159232+0000","last_peered":"2024-03-11T21:28:58.159232+0000","last_clean":"2024-03-11T21:28:58.159232+0000","last_became_active":"2024-03-11T21:25:38.540976+0000","last_became_peered":"2024-03-11T21:25:38.540976+0000","last_unstale":"2024-03-11T21:28:58.159232+0000","last_undegraded":"2024-03-11T21:28:58.159232+0000","last_fullsized":"2024-03-11T21:28:58.159232+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:28:07.421657+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:26:50.991712+0000","last_clean_scrub_stamp":"2024-03-11T21:28:07.421657+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:29:15.081862+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,3],"acting":[1,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.f","version":"74'15","reported_seq":105,"reported_epoch":119,"state":"active+clean","last_fresh":"2024-03-11T21:28:58.162704+0000","last_change":"2024-03-11T21:27:28.081459+0000","last_active":"2024-03-11T21:28:58.162704+0000","last_peered":"2024-03-11T21:28:58.162704+0000","last_clean":"2024-03-11T21:28:58.162704+0000","last_became_active":"2024-03-11T21:26:11.216224+0000","last_became_peered":"2024-03-11T21:26:11.216224+0000","last_unstale":"2024-03-11T21:28:58.162704+0000","last_undegraded":"2024-03-11T21:28:58.162704+0000","last_fullsized":"2024-03-11T21:28:58.162704+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'15","last_scrub_stamp":"2024-03-11T21:27:28.081356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:39.525554+0000","last_clean_scrub_stamp":"2024-03-11T21:27:28.081356+0000","objects_scrubbed":7,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,4],"acting":[2,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.f","version":"119'6852","reported_seq":6902,"reported_epoch":119,"state":"active+clean","last_fresh":"2024-03-11T21:29:02.068820+0000","last_change":"2024-03-11T21:27:36.694341+0000","last_active":"2024-03-11T21:29:02.068820+0000","last_peered":"2024-03-11T21:29:02.068820+0000","last_clean":"2024-03-11T21:29:02.068820+0000","last_became_active":"2024-03-11T21:27:36.694032+0000","last_became_peered":"2024-03-11T21:27:36.694032+0000","last_unstale":"2024-03-11T21:29:02.068820+0000","last_undegraded":"2024-03-11T21:29:02.068820+0000","last_fullsized":"2024-03-11T21:29:02.068820+0000","mapping_epoch":99,"log_start":"105'1300","ondisk_log_start":"105'1300","created":99,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:27:35.667522+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:27:35.667522+0000","last_clean_scrub_stamp":"2024-03-11T21:27:35.667522+0000","objects_scrubbed":0,"log_size":5552,"log_dups_size":0,"ondisk_log_size":5552,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:29:02.343669+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":449052672,"num_objects":6852,"num_object_clones":0,"num_object_copies":20556,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6852,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6852,"num_write_kb":438528,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.e","version":"0'0","reported_seq":112,"reported_epoch":119,"state":"active+clean","last_fresh":"2024-03-11T21:28:58.156298+0000","last_change":"2024-03-11T21:27:31.792816+0000","last_active":"2024-03-11T21:28:58.156298+0000","last_peered":"2024-03-11T21:28:58.156298+0000","last_clean":"2024-03-11T21:28:58.156298+0000","last_became_active":"2024-03-11T21:25:44.590198+0000","last_became_peered":"2024-03-11T21:25:44.590198+0000","last_unstale":"2024-03-11T21:28:58.156298+0000","last_undegraded":"2024-03-11T21:28:58.156298+0000","last_fullsized":"2024-03-11T21:28:58.156298+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:27:31.792707+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:43.559452+0000","last_clean_scrub_stamp":"2024-03-11T21:27:31.792707+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:29:03.083378+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.d","version":"109'9","reported_seq":173,"reported_epoch":119,"state":"active+clean","last_fresh":"2024-03-11T21:28:58.162528+0000","last_change":"2024-03-11T21:28:27.083117+0000","last_active":"2024-03-11T21:28:58.162528+0000","last_peered":"2024-03-11T21:28:58.162528+0000","last_clean":"2024-03-11T21:28:58.162528+0000","last_became_active":"2024-03-11T21:25:42.575583+0000","last_became_peered":"2024-03-11T21:25:42.575583+0000","last_unstale":"2024-03-11T21:28:58.162528+0000","last_undegraded":"2024-03-11T21:28:58.162528+0000","last_fullsized":"2024-03-11T21:28:58.162528+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"109'9","last_scrub_stamp":"2024-03-11T21:28:27.082975+0000","last_deep_scrub":"109'9","last_deep_scrub_stamp":"2024-03-11T21:28:27.082975+0000","last_clean_scrub_stamp":"2024-03-11T21:28:27.082975+0000","objects_scrubbed":1,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:29:36.610810+0000","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,5],"acting":[2,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.a","version":"0'0","reported_seq":132,"reported_epoch":119,"state":"active+clean","last_fresh":"2024-03-11T21:28:58.162246+0000","last_change":"2024-03-11T21:28:05.349499+0000","last_active":"2024-03-11T21:28:58.162246+0000","last_peered":"2024-03-11T21:28:58.162246+0000","last_clean":"2024-03-11T21:28:58.162246+0000","last_became_active":"2024-03-11T21:25:38.541045+0000","last_became_peered":"2024-03-11T21:25:38.541045+0000","last_unstale":"2024-03-11T21:28:58.162246+0000","last_undegraded":"2024-03-11T21:28:58.162246+0000","last_fullsized":"2024-03-11T21:28:58.162246+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:28:05.349379+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:37.512018+0000","last_clean_scrub_stamp":"2024-03-11T21:28:05.349379+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:29:20.671751+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,7],"acting":[1,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.c","version":"74'10","reported_seq":143,"reported_epoch":119,"state":"active+clean","last_fresh":"2024-03-11T21:28:58.156928+0000","last_change":"2024-03-11T21:28:36.845566+0000","last_active":"2024-03-11T21:28:58.156928+0000","last_peered":"2024-03-11T21:28:58.156928+0000","last_clean":"2024-03-11T21:28:58.156928+0000","last_became_active":"2024-03-11T21:25:40.550885+0000","last_became_peered":"2024-03-11T21:25:40.550885+0000","last_unstale":"2024-03-11T21:28:58.156928+0000","last_undegraded":"2024-03-11T21:28:58.156928+0000","last_fullsized":"2024-03-11T21:28:58.156928+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'10","last_scrub_stamp":"2024-03-11T21:28:36.845331+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:39.525554+0000","last_clean_scrub_stamp":"2024-03-11T21:28:36.845331+0000","objects_scrubbed":5,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:29:57.436270+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.e","version":"119'6916","reported_seq":6965,"reported_epoch":119,"state":"active+clean","last_fresh":"2024-03-11T21:29:01.983248+0000","last_change":"2024-03-11T21:27:36.692095+0000","last_active":"2024-03-11T21:29:01.983248+0000","last_peered":"2024-03-11T21:29:01.983248+0000","last_clean":"2024-03-11T21:29:01.983248+0000","last_became_active":"2024-03-11T21:27:36.691800+0000","last_became_peered":"2024-03-11T21:27:36.691800+0000","last_unstale":"2024-03-11T21:29:01.983248+0000","last_undegraded":"2024-03-11T21:29:01.983248+0000","last_fullsized":"2024-03-11T21:29:01.983248+0000","mapping_epoch":99,"log_start":"105'200","ondisk_log_start":"105'200","created":99,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:27:35.667522+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:27:35.667522+0000","last_clean_scrub_stamp":"2024-03-11T21:27:35.667522+0000","objects_scrubbed":0,"log_size":6716,"log_dups_size":0,"ondisk_log_size":6716,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":453246976,"num_objects":6916,"num_object_clones":0,"num_object_copies":20748,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6916,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6916,"num_write_kb":442624,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,0],"acting":[2,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.f","version":"0'0","reported_seq":119,"reported_epoch":119,"state":"active+clean","last_fresh":"2024-03-11T21:28:58.162842+0000","last_change":"2024-03-11T21:28:30.121487+0000","last_active":"2024-03-11T21:28:58.162842+0000","last_peered":"2024-03-11T21:28:58.162842+0000","last_clean":"2024-03-11T21:28:58.162842+0000","last_became_active":"2024-03-11T21:25:44.593414+0000","last_became_peered":"2024-03-11T21:25:44.593414+0000","last_unstale":"2024-03-11T21:28:58.162842+0000","last_undegraded":"2024-03-11T21:28:58.162842+0000","last_fullsized":"2024-03-11T21:28:58.162842+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:28:30.121365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:27:03.010969+0000","last_clean_scrub_stamp":"2024-03-11T21:28:30.121365+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:29:37.949963+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,4],"acting":[2,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":124,"reported_epoch":119,"state":"active+clean","last_fresh":"2024-03-11T21:28:58.161038+0000","last_change":"2024-03-11T21:28:30.339359+0000","last_active":"2024-03-11T21:28:58.161038+0000","last_peered":"2024-03-11T21:28:58.161038+0000","last_clean":"2024-03-11T21:28:58.161038+0000","last_became_active":"2024-03-11T21:25:42.579409+0000","last_became_peered":"2024-03-11T21:25:42.579409+0000","last_unstale":"2024-03-11T21:28:58.161038+0000","last_undegraded":"2024-03-11T21:28:58.161038+0000","last_fullsized":"2024-03-11T21:28:58.161038+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:28:30.339255+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:41.544550+0000","last_clean_scrub_stamp":"2024-03-11T21:28:30.339255+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:29:47.720331+0000","scrub_duration":22,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.b","version":"0'0","reported_seq":131,"reported_epoch":119,"state":"active+clean","last_fresh":"2024-03-11T21:28:58.168217+0000","last_change":"2024-03-11T21:28:36.500085+0000","last_active":"2024-03-11T21:28:58.168217+0000","last_peered":"2024-03-11T21:28:58.168217+0000","last_clean":"2024-03-11T21:28:58.168217+0000","last_became_active":"2024-03-11T21:25:38.543377+0000","last_became_peered":"2024-03-11T21:25:38.543377+0000","last_unstale":"2024-03-11T21:28:58.168217+0000","last_undegraded":"2024-03-11T21:28:58.168217+0000","last_fullsized":"2024-03-11T21:28:58.168217+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:28:36.499974+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:37.512018+0000","last_clean_scrub_stamp":"2024-03-11T21:28:36.499974+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:29:38.307877+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,5],"acting":[7,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.d","version":"74'17","reported_seq":156,"reported_epoch":119,"state":"active+clean","last_fresh":"2024-03-11T21:28:58.155985+0000","last_change":"2024-03-11T21:28:29.801778+0000","last_active":"2024-03-11T21:28:58.155985+0000","last_peered":"2024-03-11T21:28:58.155985+0000","last_clean":"2024-03-11T21:28:58.155985+0000","last_became_active":"2024-03-11T21:25:40.553454+0000","last_became_peered":"2024-03-11T21:25:40.553454+0000","last_unstale":"2024-03-11T21:28:58.155985+0000","last_undegraded":"2024-03-11T21:28:58.155985+0000","last_fullsized":"2024-03-11T21:28:58.155985+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'17","last_scrub_stamp":"2024-03-11T21:28:29.801670+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:39.525554+0000","last_clean_scrub_stamp":"2024-03-11T21:28:29.801670+0000","objects_scrubbed":8,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:29:34.512249+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":29,"num_read_kb":19,"num_write":18,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,1],"acting":[4,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.9","version":"119'6814","reported_seq":6863,"reported_epoch":119,"state":"active+clean","last_fresh":"2024-03-11T21:29:02.064634+0000","last_change":"2024-03-11T21:27:36.693276+0000","last_active":"2024-03-11T21:29:02.064634+0000","last_peered":"2024-03-11T21:29:02.064634+0000","last_clean":"2024-03-11T21:29:02.064634+0000","last_became_active":"2024-03-11T21:27:36.692978+0000","last_became_peered":"2024-03-11T21:27:36.692978+0000","last_unstale":"2024-03-11T21:29:02.064634+0000","last_undegraded":"2024-03-11T21:29:02.064634+0000","last_fullsized":"2024-03-11T21:29:02.064634+0000","mapping_epoch":99,"log_start":"105'1300","ondisk_log_start":"105'1300","created":99,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:27:35.667522+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:27:35.667522+0000","last_clean_scrub_stamp":"2024-03-11T21:27:35.667522+0000","objects_scrubbed":0,"log_size":5514,"log_dups_size":0,"ondisk_log_size":5514,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:29:06.439117+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":446562304,"num_objects":6814,"num_object_clones":0,"num_object_copies":20442,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6814,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6814,"num_write_kb":436096,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":119,"reported_epoch":119,"state":"active+clean","last_fresh":"2024-03-11T21:28:58.169206+0000","last_change":"2024-03-11T21:28:37.561913+0000","last_active":"2024-03-11T21:28:58.169206+0000","last_peered":"2024-03-11T21:28:58.169206+0000","last_clean":"2024-03-11T21:28:58.169206+0000","last_became_active":"2024-03-11T21:25:44.589973+0000","last_became_peered":"2024-03-11T21:25:44.589973+0000","last_unstale":"2024-03-11T21:28:58.169206+0000","last_undegraded":"2024-03-11T21:28:58.169206+0000","last_fullsized":"2024-03-11T21:28:58.169206+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:28:37.561826+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:28:37.561826+0000","last_clean_scrub_stamp":"2024-03-11T21:28:37.561826+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:29:56.242385+0000","scrub_duration":25,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,3],"acting":[7,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":115,"reported_epoch":119,"state":"active+clean","last_fresh":"2024-03-11T21:28:58.162274+0000","last_change":"2024-03-11T21:27:29.075249+0000","last_active":"2024-03-11T21:28:58.162274+0000","last_peered":"2024-03-11T21:28:58.162274+0000","last_clean":"2024-03-11T21:28:58.162274+0000","last_became_active":"2024-03-11T21:25:42.574515+0000","last_became_peered":"2024-03-11T21:25:42.574515+0000","last_unstale":"2024-03-11T21:28:58.162274+0000","last_undegraded":"2024-03-11T21:28:58.162274+0000","last_fullsized":"2024-03-11T21:28:58.162274+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:27:29.075166+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:41.544550+0000","last_clean_scrub_stamp":"2024-03-11T21:27:29.075166+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:28:58.218185+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,5],"acting":[2,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.c","version":"0'0","reported_seq":123,"reported_epoch":119,"state":"active+clean","last_fresh":"2024-03-11T21:28:58.162225+0000","last_change":"2024-03-11T21:27:24.105818+0000","last_active":"2024-03-11T21:28:58.162225+0000","last_peered":"2024-03-11T21:28:58.162225+0000","last_clean":"2024-03-11T21:28:58.162225+0000","last_became_active":"2024-03-11T21:25:38.533240+0000","last_became_peered":"2024-03-11T21:25:38.533240+0000","last_unstale":"2024-03-11T21:28:58.162225+0000","last_undegraded":"2024-03-11T21:28:58.162225+0000","last_fullsized":"2024-03-11T21:28:58.162225+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:27:24.105731+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:37.512018+0000","last_clean_scrub_stamp":"2024-03-11T21:27:24.105731+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,0],"acting":[2,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.a","version":"109'28","reported_seq":168,"reported_epoch":119,"state":"active+clean","last_fresh":"2024-03-11T21:28:58.164706+0000","last_change":"2024-03-11T21:28:24.016603+0000","last_active":"2024-03-11T21:28:58.164706+0000","last_peered":"2024-03-11T21:28:58.164706+0000","last_clean":"2024-03-11T21:28:58.164706+0000","last_became_active":"2024-03-11T21:25:40.760698+0000","last_became_peered":"2024-03-11T21:25:40.760698+0000","last_unstale":"2024-03-11T21:28:58.164706+0000","last_undegraded":"2024-03-11T21:28:58.164706+0000","last_fullsized":"2024-03-11T21:28:58.164706+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"109'28","last_scrub_stamp":"2024-03-11T21:28:24.016389+0000","last_deep_scrub":"109'28","last_deep_scrub_stamp":"2024-03-11T21:28:24.016389+0000","last_clean_scrub_stamp":"2024-03-11T21:28:24.016389+0000","objects_scrubbed":9,"log_size":28,"log_dups_size":0,"ondisk_log_size":28,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:29:37.183547+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":32,"num_read_kb":21,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,7],"acting":[6,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.8","version":"119'6769","reported_seq":6818,"reported_epoch":119,"state":"active+clean","last_fresh":"2024-03-11T21:29:01.991867+0000","last_change":"2024-03-11T21:27:36.693400+0000","last_active":"2024-03-11T21:29:01.991867+0000","last_peered":"2024-03-11T21:29:01.991867+0000","last_clean":"2024-03-11T21:29:01.991867+0000","last_became_active":"2024-03-11T21:27:36.693060+0000","last_became_peered":"2024-03-11T21:27:36.693060+0000","last_unstale":"2024-03-11T21:29:01.991867+0000","last_undegraded":"2024-03-11T21:29:01.991867+0000","last_fullsized":"2024-03-11T21:29:01.991867+0000","mapping_epoch":99,"log_start":"105'100","ondisk_log_start":"105'100","created":99,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:27:35.667522+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:27:35.667522+0000","last_clean_scrub_stamp":"2024-03-11T21:27:35.667522+0000","objects_scrubbed":0,"log_size":6669,"log_dups_size":0,"ondisk_log_size":6669,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:29:04.212862+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":443613184,"num_objects":6769,"num_object_clones":0,"num_object_copies":20307,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6769,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6769,"num_write_kb":433216,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,4],"acting":[2,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":112,"reported_epoch":119,"state":"active+clean","last_fresh":"2024-03-11T21:28:58.157510+0000","last_change":"2024-03-11T21:27:11.362017+0000","last_active":"2024-03-11T21:28:58.157510+0000","last_peered":"2024-03-11T21:28:58.157510+0000","last_clean":"2024-03-11T21:28:58.157510+0000","last_became_active":"2024-03-11T21:25:44.584154+0000","last_became_peered":"2024-03-11T21:25:44.584154+0000","last_unstale":"2024-03-11T21:28:58.157510+0000","last_undegraded":"2024-03-11T21:28:58.157510+0000","last_fullsized":"2024-03-11T21:28:58.157510+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:27:11.361913+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:27:11.361913+0000","last_clean_scrub_stamp":"2024-03-11T21:27:11.361913+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:29:02.343496+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":115,"reported_epoch":119,"state":"active+clean","last_fresh":"2024-03-11T21:28:58.162876+0000","last_change":"2024-03-11T21:27:10.909699+0000","last_active":"2024-03-11T21:28:58.162876+0000","last_peered":"2024-03-11T21:28:58.162876+0000","last_clean":"2024-03-11T21:28:58.162876+0000","last_became_active":"2024-03-11T21:25:42.576679+0000","last_became_peered":"2024-03-11T21:25:42.576679+0000","last_unstale":"2024-03-11T21:28:58.162876+0000","last_undegraded":"2024-03-11T21:28:58.162876+0000","last_fullsized":"2024-03-11T21:28:58.162876+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:27:10.909543+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:41.544550+0000","last_clean_scrub_stamp":"2024-03-11T21:27:10.909543+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:29:00.225018+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,3],"acting":[2,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.d","version":"0'0","reported_seq":132,"reported_epoch":119,"state":"active+clean","last_fresh":"2024-03-11T21:28:58.159529+0000","last_change":"2024-03-11T21:28:03.343281+0000","last_active":"2024-03-11T21:28:58.159529+0000","last_peered":"2024-03-11T21:28:58.159529+0000","last_clean":"2024-03-11T21:28:58.159529+0000","last_became_active":"2024-03-11T21:25:38.539681+0000","last_became_peered":"2024-03-11T21:25:38.539681+0000","last_unstale":"2024-03-11T21:28:58.159529+0000","last_undegraded":"2024-03-11T21:28:58.159529+0000","last_fullsized":"2024-03-11T21:28:58.159529+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:28:03.343147+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:37.512018+0000","last_clean_scrub_stamp":"2024-03-11T21:28:03.343147+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:29:15.956568+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,3],"acting":[1,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.b","version":"74'9","reported_seq":136,"reported_epoch":119,"state":"active+clean","last_fresh":"2024-03-11T21:28:58.157326+0000","last_change":"2024-03-11T21:27:24.392581+0000","last_active":"2024-03-11T21:28:58.157326+0000","last_peered":"2024-03-11T21:28:58.157326+0000","last_clean":"2024-03-11T21:28:58.157326+0000","last_became_active":"2024-03-11T21:25:40.549612+0000","last_became_peered":"2024-03-11T21:25:40.549612+0000","last_unstale":"2024-03-11T21:28:58.157326+0000","last_undegraded":"2024-03-11T21:28:58.157326+0000","last_fullsized":"2024-03-11T21:28:58.157326+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'9","last_scrub_stamp":"2024-03-11T21:27:24.392483+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:39.525554+0000","last_clean_scrub_stamp":"2024-03-11T21:27:24.392483+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:29:02.343636+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.b","version":"119'6847","reported_seq":6897,"reported_epoch":119,"state":"active+clean","last_fresh":"2024-03-11T21:29:01.972886+0000","last_change":"2024-03-11T21:27:36.691639+0000","last_active":"2024-03-11T21:29:01.972886+0000","last_peered":"2024-03-11T21:29:01.972886+0000","last_clean":"2024-03-11T21:29:01.972886+0000","last_became_active":"2024-03-11T21:27:36.691322+0000","last_became_peered":"2024-03-11T21:27:36.691322+0000","last_unstale":"2024-03-11T21:29:01.972886+0000","last_undegraded":"2024-03-11T21:29:01.972886+0000","last_fullsized":"2024-03-11T21:29:01.972886+0000","mapping_epoch":99,"log_start":"105'1700","ondisk_log_start":"105'1700","created":99,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:27:35.667522+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:27:35.667522+0000","last_clean_scrub_stamp":"2024-03-11T21:27:35.667522+0000","objects_scrubbed":0,"log_size":5147,"log_dups_size":0,"ondisk_log_size":5147,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:29:04.800550+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":448724992,"num_objects":6847,"num_object_clones":0,"num_object_copies":20541,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6847,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6847,"num_write_kb":438208,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,6],"acting":[1,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":25,"reported_epoch":119,"state":"active+clean","last_fresh":"2024-03-11T21:28:58.155476+0000","last_change":"2024-03-11T21:28:45.372681+0000","last_active":"2024-03-11T21:28:58.155476+0000","last_peered":"2024-03-11T21:28:58.155476+0000","last_clean":"2024-03-11T21:28:58.155476+0000","last_became_active":"2024-03-11T21:28:45.372452+0000","last_became_peered":"2024-03-11T21:28:45.372452+0000","last_unstale":"2024-03-11T21:28:58.155476+0000","last_undegraded":"2024-03-11T21:28:58.155476+0000","last_fullsized":"2024-03-11T21:28:58.155476+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:28:19.815688+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:43.559452+0000","last_clean_scrub_stamp":"2024-03-11T21:28:19.815688+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:29:39.127540+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,3],"acting":[4,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.e","version":"0'0","reported_seq":131,"reported_epoch":119,"state":"active+clean","last_fresh":"2024-03-11T21:28:58.163179+0000","last_change":"2024-03-11T21:28:11.016096+0000","last_active":"2024-03-11T21:28:58.163179+0000","last_peered":"2024-03-11T21:28:58.163179+0000","last_clean":"2024-03-11T21:28:58.163179+0000","last_became_active":"2024-03-11T21:25:38.539954+0000","last_became_peered":"2024-03-11T21:25:38.539954+0000","last_unstale":"2024-03-11T21:28:58.163179+0000","last_undegraded":"2024-03-11T21:28:58.163179+0000","last_fullsized":"2024-03-11T21:28:58.163179+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:28:11.015887+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:37.512018+0000","last_clean_scrub_stamp":"2024-03-11T21:28:11.015887+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:29:16.519153+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,7],"acting":[2,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.8","version":"109'22","reported_seq":159,"reported_epoch":119,"state":"active+clean","last_fresh":"2024-03-11T21:28:58.163604+0000","last_change":"2024-03-11T21:28:25.743214+0000","last_active":"2024-03-11T21:28:58.163604+0000","last_peered":"2024-03-11T21:28:58.163604+0000","last_clean":"2024-03-11T21:28:58.163604+0000","last_became_active":"2024-03-11T21:25:40.761938+0000","last_became_peered":"2024-03-11T21:25:40.761938+0000","last_unstale":"2024-03-11T21:28:58.163604+0000","last_undegraded":"2024-03-11T21:28:58.163604+0000","last_fullsized":"2024-03-11T21:28:58.163604+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"109'22","last_scrub_stamp":"2024-03-11T21:28:25.743058+0000","last_deep_scrub":"109'22","last_deep_scrub_stamp":"2024-03-11T21:28:25.743058+0000","last_clean_scrub_stamp":"2024-03-11T21:28:25.743058+0000","objects_scrubbed":7,"log_size":22,"log_dups_size":0,"ondisk_log_size":22,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:29:41.395729+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.9","version":"94'9","reported_seq":176,"reported_epoch":119,"state":"active+clean","last_fresh":"2024-03-11T21:28:58.168174+0000","last_change":"2024-03-11T21:28:14.615170+0000","last_active":"2024-03-11T21:28:58.168174+0000","last_peered":"2024-03-11T21:28:58.168174+0000","last_clean":"2024-03-11T21:28:58.168174+0000","last_became_active":"2024-03-11T21:25:42.584461+0000","last_became_peered":"2024-03-11T21:25:42.584461+0000","last_unstale":"2024-03-11T21:28:58.168174+0000","last_undegraded":"2024-03-11T21:28:58.168174+0000","last_fullsized":"2024-03-11T21:28:58.168174+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"94'9","last_scrub_stamp":"2024-03-11T21:28:14.615041+0000","last_deep_scrub":"94'9","last_deep_scrub_stamp":"2024-03-11T21:26:53.422552+0000","last_clean_scrub_stamp":"2024-03-11T21:28:14.615041+0000","objects_scrubbed":1,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:29:18.296913+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":7,"num_pg":16,"stat_sum":{"num_bytes":7178158080,"num_objects":109530,"num_object_clones":0,"num_object_copies":328590,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":109530,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":109530,"num_write_kb":7009920,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":21535522816,"data_stored":21535522816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":89207,"ondisk_log_size":89207,"up":48,"acting":48,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":1531,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":24,"num_read_kb":24,"num_write":10,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":4593,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":6,"ondisk_log_size":6,"up":96,"acting":96,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":212,"num_read_kb":306,"num_write":179,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1771104,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1781760,"data_stored":1771104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":128,"ondisk_log_size":128,"up":3,"acting":3,"num_store_stats":3},{"poolid":3,"num_pg":3,"stat_sum":{"num_bytes":408,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":198,"num_read_kb":193,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1224,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":9,"acting":9,"num_store_stats":7},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":3702,"num_objects":178,"num_object_clones":0,"num_object_copies":534,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":178,"num_whiteouts":0,"num_read":698,"num_read_kb":455,"num_write":417,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":417792,"data_stored":11106,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":446,"ondisk_log_size":446,"up":96,"acting":96,"num_store_stats":8},{"poolid":5,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":68,"ondisk_log_size":68,"up":96,"acting":96,"num_store_stats":8},{"poolid":6,"num_pg":32,"stat_sum":{"num_bytes":383,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1149,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2,"ondisk_log_size":2,"up":96,"acting":96,"num_store_stats":8}],"osd_stats":[{"osd":7,"up_from":60,"seq":257698037804,"num_pgs":61,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2243384,"kb_used_data":2176112,"kb_used_omap":1,"kb_used_meta":67262,"kb_avail":91501768,"statfs":{"total":95995035648,"available":93697810432,"internally_reserved":0,"allocated":2228338688,"data_stored":2227882613,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":68876746},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":50,"seq":214748364850,"num_pgs":48,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2268148,"kb_used_data":2200876,"kb_used_omap":1,"kb_used_meta":67262,"kb_avail":91477004,"statfs":{"total":95995035648,"available":93672452096,"internally_reserved":0,"allocated":2253697024,"data_stored":2253245338,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":68876742},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,2,1,1,1,1],"upper_bound":64},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574924,"num_pgs":59,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2689936,"kb_used_data":2614344,"kb_used_omap":1,"kb_used_meta":75582,"kb_avail":91055216,"statfs":{"total":95995035648,"available":93240541184,"internally_reserved":0,"allocated":2677088256,"data_stored":2676607462,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":77396421},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,1,1],"upper_bound":16},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738449,"num_pgs":55,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":3594428,"kb_used_data":3500788,"kb_used_omap":1,"kb_used_meta":93630,"kb_avail":90150724,"statfs":{"total":95995035648,"available":92314341376,"internally_reserved":0,"allocated":3584806912,"data_stored":3584346958,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":95877573},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,3],"upper_bound":16},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411398,"num_pgs":45,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2671980,"kb_used_data":2600544,"kb_used_omap":1,"kb_used_meta":71422,"kb_avail":91073172,"statfs":{"total":95995035648,"available":93258928128,"internally_reserved":0,"allocated":2662957056,"data_stored":2662516446,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":73136581},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,0,2,2],"upper_bound":32},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":3,"up_from":26,"seq":111669149762,"num_pgs":65,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":4025488,"kb_used_data":3917960,"kb_used_omap":1,"kb_used_meta":107518,"kb_avail":89719664,"statfs":{"total":95995035648,"available":91872935936,"internally_reserved":0,"allocated":4011991040,"data_stored":4011510580,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":110098887},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,3,1,2,2,1],"upper_bound":64},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":4,"up_from":32,"seq":137438953531,"num_pgs":59,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2237696,"kb_used_data":2170424,"kb_used_omap":1,"kb_used_meta":67262,"kb_avail":91507456,"statfs":{"total":95995035648,"available":93703634944,"internally_reserved":0,"allocated":2222514176,"data_stored":2222049274,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":68876741},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":41,"seq":176093659190,"num_pgs":52,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1801132,"kb_used_data":1743588,"kb_used_omap":1,"kb_used_meta":57534,"kb_avail":91944020,"statfs":{"total":95995035648,"available":94150676480,"internally_reserved":0,"allocated":1785434112,"data_stored":1784989457,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":58915269},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1049,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":46,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":957,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":574,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":436,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1003,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":528,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":1320,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":81920,"data_stored":2118,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":1018,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":1898,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":1474,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":990,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":1034,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":1254,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":370,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":383,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":370,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":3605528576,"data_stored":3605528576,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":2690646016,"data_stored":2690646016,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":2683043840,"data_stored":2683043840,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4035182592,"data_stored":4035182592,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":2237595648,"data_stored":2237595648,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":1787691008,"data_stored":1787691008,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":2257190912,"data_stored":2257190912,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":2238644224,"data_stored":2238644224,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-11T21:29:06.861 DEBUG:teuthology.orchestra.run.smithi092:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid 00db0016-dfed-11ee-95c8-87774f69a715 -- ceph osd pool set .mgr pgp_num 1 2024-03-11T21:29:06.893 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:29:06.940 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:29:06.940 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:29:06.941 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:29:07.041 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:29:07.069 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:29:07.069 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:29:07.070 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:29:07.171 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:29:07.199 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:29:07.199 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:29:07.200 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:29:07.301 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:29:07.349 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:29:07.350 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:29:07.350 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:29:07.451 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:29:07.484 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:29:07.485 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:29:07.485 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:29:07.586 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:29:07.610 INFO:journalctl@ceph.alertmanager.a.smithi092.stdout:Mar 11 21:29:07 smithi092 podman[84650]: 2024-03-11 21:29:07.353789144 +0000 UTC m=+2.013733613 volume create 631a33a17d119ca9f206939521b4e19f0a6f733b1a9f317862f3bdc9a1e37143 2024-03-11T21:29:07.623 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:29:07.624 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:29:07.624 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:29:07.644 INFO:tasks.radosbench.radosbench.0.smithi092.stdout: 83 16 115242 115226 86.7394 62.625 0.0248099 0.0115243 2024-03-11T21:29:07.725 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:29:07.765 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:29:07.766 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:29:07.766 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:29:07.867 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:29:07.893 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:29:07.894 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:29:07.894 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:29:07.995 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:29:08.023 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:29:08.024 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:29:08.025 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:29:08.126 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:29:08.166 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:29:08.167 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:29:08.167 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:29:08.268 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:29:08.297 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:29:08.298 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:29:08.298 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:29:08.361 INFO:journalctl@ceph.mgr.y.smithi092.stdout:Mar 11 21:29:07 smithi092 ceph-00db0016-dfed-11ee-95c8-87774f69a715-mgr-y[29801]: ::ffff:172.21.15.97 - - [11/Mar/2024:21:29:07] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-11T21:29:08.399 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:29:08.426 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:29:08.427 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:29:08.428 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:29:08.528 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:29:08.570 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:29:08.571 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:29:08.571 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:29:08.644 INFO:tasks.radosbench.radosbench.0.smithi092.stdout: 84 16 116164 116148 86.3928 57.625 0.0154782 0.0115702 2024-03-11T21:29:08.672 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:29:08.713 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:29:08.713 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:29:08.714 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:29:08.815 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:29:08.826 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:29:08 smithi097 ceph-mon[33317]: pgmap v118: 148 pgs: 148 active+clean; 6.8 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 537 B/s rd, 46 MiB/s wr, 740 op/s 2024-03-11T21:29:08.838 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:29:08.839 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:29:08.839 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:29:08.861 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:29:08 smithi092 ceph-mon[29572]: pgmap v118: 148 pgs: 148 active+clean; 6.8 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 537 B/s rd, 46 MiB/s wr, 740 op/s 2024-03-11T21:29:08.861 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:29:08 smithi092 ceph-mon[37563]: pgmap v118: 148 pgs: 148 active+clean; 6.8 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 537 B/s rd, 46 MiB/s wr, 740 op/s 2024-03-11T21:29:08.861 INFO:journalctl@ceph.alertmanager.a.smithi092.stdout:Mar 11 21:29:08 smithi092 podman[84650]: 2024-03-11 21:29:08.380154938 +0000 UTC m=+3.040099413 container create bddae98c3565f589db50ce3c14882c9a14da74bb75ef29d00e1b4c80145f9275 (image=quay.io/prometheus/alertmanager:v0.25.0, name=ceph-00db0016-dfed-11ee-95c8-87774f69a715-alertmanager-a, maintainer=The Prometheus Authors ) 2024-03-11T21:29:08.940 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:29:08.963 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:29:08.964 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:29:08.964 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:29:09.065 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:29:09.092 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:29:09.093 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:29:09.093 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:29:09.194 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:29:09.246 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:29:09.247 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:29:09.247 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:29:09.348 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:29:09.376 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:29:09.377 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:29:09.377 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:29:09.478 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:29:09.523 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:29:09.523 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:29:09.523 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:29:09.624 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:29:09.644 INFO:tasks.radosbench.radosbench.0.smithi092.stdout: 85 16 117011 116995 85.9993 52.9375 0.0198069 0.011623 2024-03-11T21:29:09.654 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:29:09.654 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:29:09.654 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:29:09.755 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:29:09.787 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:29:09.788 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:29:09.789 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:29:09.889 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:29:09.917 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:29:09.918 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:29:09.918 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:29:10.019 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:29:10.057 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:29:10.058 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:29:10.058 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:29:10.159 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:29:10.189 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:29:10.189 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:29:10.190 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:29:10.290 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:29:10.314 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:29:10.315 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:29:10.315 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:29:10.416 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:29:10.445 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:29:10.446 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:29:10.447 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:29:10.516 INFO:teuthology.orchestra.run.smithi092.stderr:noscrub is unset 2024-03-11T21:29:10.547 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:29:10.577 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:29:10.578 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:29:10.578 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:29:10.644 INFO:tasks.radosbench.radosbench.0.smithi092.stdout: 86 16 117989 117973 85.71 61.125 0.0195411 0.0116622 2024-03-11T21:29:10.679 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:29:10.720 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:29:10.721 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:29:10.721 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:29:10.822 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:29:10.826 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:29:10 smithi097 ceph-mon[33317]: pgmap v119: 148 pgs: 148 active+clean; 7.0 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 943 B/s rd, 59 MiB/s wr, 942 op/s 2024-03-11T21:29:10.826 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:29:10 smithi097 ceph-mon[33317]: from='client.? v1:172.21.15.92:0/4167050267' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-11T21:29:10.850 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:29:10.851 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:29:10.852 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:29:10.860 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:29:10 smithi092 ceph-mon[29572]: pgmap v119: 148 pgs: 148 active+clean; 7.0 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 943 B/s rd, 59 MiB/s wr, 942 op/s 2024-03-11T21:29:10.860 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:29:10 smithi092 ceph-mon[29572]: from='client.? v1:172.21.15.92:0/4167050267' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-11T21:29:10.861 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:29:10 smithi092 ceph-mon[37563]: pgmap v119: 148 pgs: 148 active+clean; 7.0 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 943 B/s rd, 59 MiB/s wr, 942 op/s 2024-03-11T21:29:10.861 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:29:10 smithi092 ceph-mon[37563]: from='client.? v1:172.21.15.92:0/4167050267' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-11T21:29:10.952 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:29:10.982 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:29:10.982 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:29:10.983 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:29:11.084 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:29:11.122 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:29:11.123 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:29:11.123 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:29:11.224 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:29:11.250 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:29:11.251 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:29:11.251 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:29:11.352 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:29:11.380 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:29:11.381 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:29:11.381 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:29:11.482 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:29:11.512 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:29:11.513 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:29:11.513 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:29:11.614 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:29:11.644 INFO:tasks.radosbench.radosbench.0.smithi092.stdout: 87 16 119039 119023 85.4791 65.625 0.0170779 0.0116938 2024-03-11T21:29:11.645 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:29:11.645 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:29:11.646 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:29:11.747 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:29:11.792 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:29:11.793 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:29:11.793 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:29:11.827 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:29:11 smithi097 ceph-mon[33317]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-11T21:29:11.827 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:29:11 smithi097 ceph-mon[33317]: from='client.? v1:172.21.15.92:0/4167050267' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-03-11T21:29:11.827 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:29:11 smithi097 ceph-mon[33317]: osdmap e120: 8 total, 8 up, 8 in 2024-03-11T21:29:11.827 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:29:11 smithi097 ceph-mon[33317]: 5.1c scrub starts 2024-03-11T21:29:11.827 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:29:11 smithi097 ceph-mon[33317]: 5.1c scrub ok 2024-03-11T21:29:11.861 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:29:11 smithi092 ceph-mon[29572]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-11T21:29:11.861 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:29:11 smithi092 ceph-mon[29572]: from='client.? v1:172.21.15.92:0/4167050267' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-03-11T21:29:11.861 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:29:11 smithi092 ceph-mon[29572]: osdmap e120: 8 total, 8 up, 8 in 2024-03-11T21:29:11.861 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:29:11 smithi092 ceph-mon[29572]: 5.1c scrub starts 2024-03-11T21:29:11.861 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:29:11 smithi092 ceph-mon[29572]: 5.1c scrub ok 2024-03-11T21:29:11.862 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:29:11 smithi092 ceph-mon[37563]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-11T21:29:11.862 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:29:11 smithi092 ceph-mon[37563]: from='client.? v1:172.21.15.92:0/4167050267' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-03-11T21:29:11.862 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:29:11 smithi092 ceph-mon[37563]: osdmap e120: 8 total, 8 up, 8 in 2024-03-11T21:29:11.863 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:29:11 smithi092 ceph-mon[37563]: 5.1c scrub starts 2024-03-11T21:29:11.863 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:29:11 smithi092 ceph-mon[37563]: 5.1c scrub ok 2024-03-11T21:29:11.894 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:29:11.933 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:29:11.934 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:29:11.934 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:29:12.035 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:29:12.065 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:29:12.066 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:29:12.066 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:29:12.167 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:29:12.213 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:29:12.213 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:29:12.213 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:29:12.314 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:29:12.379 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:29:12.380 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:29:12.380 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:29:12.481 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:29:12.514 INFO:journalctl@ceph.iscsi.iscsi.a.smithi097.stdout:Mar 11 21:29:12 smithi097 ceph-00db0016-dfed-11ee-95c8-87774f69a715-iscsi-iscsi-a[60943]: debug there is no tcmu-runner data available 2024-03-11T21:29:12.516 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:29:12.517 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:29:12.518 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:29:12.619 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:29:12.647 INFO:tasks.radosbench.radosbench.0.smithi092.stdout: 88 16 120073 120057 85.2403 64.625 0.0203239 0.0117267 2024-03-11T21:29:12.648 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:29:12.649 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:29:12.649 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:29:12.750 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:29:12.778 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:29:12 smithi092 ceph-mon[29572]: pgmap v121: 148 pgs: 148 active+clean; 7.0 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 1023 B/s rd, 64 MiB/s wr, 1.02k op/s 2024-03-11T21:29:12.779 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:29:12 smithi092 ceph-mon[29572]: 5.6 scrub starts 2024-03-11T21:29:12.779 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:29:12 smithi092 ceph-mon[29572]: 5.6 scrub ok 2024-03-11T21:29:12.779 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:29:12 smithi092 ceph-mon[29572]: 5.19 scrub starts 2024-03-11T21:29:12.779 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:29:12 smithi092 ceph-mon[29572]: 5.19 scrub ok 2024-03-11T21:29:12.779 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:29:12 smithi092 ceph-mon[29572]: 7.7 scrub starts 2024-03-11T21:29:12.783 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:29:12 smithi092 ceph-mon[37563]: pgmap v121: 148 pgs: 148 active+clean; 7.0 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 1023 B/s rd, 64 MiB/s wr, 1.02k op/s 2024-03-11T21:29:12.784 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:29:12 smithi092 ceph-mon[37563]: 5.6 scrub starts 2024-03-11T21:29:12.784 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:29:12 smithi092 ceph-mon[37563]: 5.6 scrub ok 2024-03-11T21:29:12.784 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:29:12 smithi092 ceph-mon[37563]: 5.19 scrub starts 2024-03-11T21:29:12.784 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:29:12 smithi092 ceph-mon[37563]: 5.19 scrub ok 2024-03-11T21:29:12.784 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:29:12 smithi092 ceph-mon[37563]: 7.7 scrub starts 2024-03-11T21:29:12.805 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:29:12.805 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:29:12.806 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:29:12.826 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:29:12 smithi097 ceph-mon[33317]: pgmap v121: 148 pgs: 148 active+clean; 7.0 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 1023 B/s rd, 64 MiB/s wr, 1.02k op/s 2024-03-11T21:29:12.827 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:29:12 smithi097 ceph-mon[33317]: 5.6 scrub starts 2024-03-11T21:29:12.827 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:29:12 smithi097 ceph-mon[33317]: 5.6 scrub ok 2024-03-11T21:29:12.827 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:29:12 smithi097 ceph-mon[33317]: 5.19 scrub starts 2024-03-11T21:29:12.827 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:29:12 smithi097 ceph-mon[33317]: 5.19 scrub ok 2024-03-11T21:29:12.827 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:29:12 smithi097 ceph-mon[33317]: 7.7 scrub starts 2024-03-11T21:29:12.906 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:29:12.945 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:29:12.946 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:29:12.946 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:29:13.047 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:29:13.075 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:29:13.076 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:29:13.076 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:29:13.177 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:29:13.207 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:29:13.207 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:29:13.208 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:29:13.308 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:29:13.327 INFO:teuthology.orchestra.run.smithi092.stdout:{ 2024-03-11T21:29:13.327 INFO:teuthology.orchestra.run.smithi092.stdout: "size": 20, 2024-03-11T21:29:13.327 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 600, 2024-03-11T21:29:13.327 INFO:teuthology.orchestra.run.smithi092.stdout: "ops": [ 2024-03-11T21:29:13.327 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:29:13.327 INFO:teuthology.orchestra.run.smithi092.stdout: "description": "osd_op(client.34715.0:15952 7.e 7:7096d29d:::benchmark_data_smithi092_76040_object15951:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e105)", 2024-03-11T21:29:13.328 INFO:teuthology.orchestra.run.smithi092.stdout: "initiated_at": "2024-03-11T21:27:50.070486+0000", 2024-03-11T21:29:13.328 INFO:teuthology.orchestra.run.smithi092.stdout: "age": 83.270899874999998, 2024-03-11T21:29:13.328 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.076347771999999994, 2024-03-11T21:29:13.328 INFO:teuthology.orchestra.run.smithi092.stdout: "type_data": { 2024-03-11T21:29:13.328 INFO:teuthology.orchestra.run.smithi092.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-11T21:29:13.328 INFO:teuthology.orchestra.run.smithi092.stdout: "client_info": { 2024-03-11T21:29:13.328 INFO:teuthology.orchestra.run.smithi092.stdout: "client": "client.34715", 2024-03-11T21:29:13.328 INFO:teuthology.orchestra.run.smithi092.stdout: "client_addr": "v1:172.21.15.92:0/2403132573", 2024-03-11T21:29:13.328 INFO:teuthology.orchestra.run.smithi092.stdout: "tid": 15952 2024-03-11T21:29:13.328 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:29:13.328 INFO:teuthology.orchestra.run.smithi092.stdout: "events": [ 2024-03-11T21:29:13.328 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:29:13.328 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "initiated", 2024-03-11T21:29:13.328 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:27:50.070486+0000", 2024-03-11T21:29:13.329 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-03-11T21:29:13.329 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:29:13.329 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:29:13.329 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "header_read", 2024-03-11T21:29:13.329 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:27:50.070486+0000", 2024-03-11T21:29:13.329 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-03-11T21:29:13.329 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:29:13.329 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:29:13.329 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "throttled", 2024-03-11T21:29:13.329 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:27:50.070488+0000", 2024-03-11T21:29:13.329 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 2.1100000000000001e-06 2024-03-11T21:29:13.329 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:29:13.329 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:29:13.329 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "all_read", 2024-03-11T21:29:13.330 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:27:50.070512+0000", 2024-03-11T21:29:13.330 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 2.4108999999999999e-05 2024-03-11T21:29:13.330 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:29:13.330 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:29:13.330 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "dispatched", 2024-03-11T21:29:13.330 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:27:50.070523+0000", 2024-03-11T21:29:13.330 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.0942e-05 2024-03-11T21:29:13.330 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:29:13.330 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:29:13.330 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "queued_for_pg", 2024-03-11T21:29:13.330 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:27:50.070530+0000", 2024-03-11T21:29:13.330 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 7.6020000000000002e-06 2024-03-11T21:29:13.330 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:29:13.330 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:29:13.331 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-03-11T21:29:13.331 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:27:50.070582+0000", 2024-03-11T21:29:13.331 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 5.1415000000000002e-05 2024-03-11T21:29:13.331 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:29:13.331 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:29:13.331 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "started", 2024-03-11T21:29:13.331 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:27:50.070700+0000", 2024-03-11T21:29:13.331 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.000117991 2024-03-11T21:29:13.331 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:29:13.331 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:29:13.331 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "waiting for subops from 0,5", 2024-03-11T21:29:13.331 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:27:50.070869+0000", 2024-03-11T21:29:13.331 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.000169483 2024-03-11T21:29:13.331 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:29:13.332 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:29:13.332 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "op_commit", 2024-03-11T21:29:13.332 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:27:50.071772+0000", 2024-03-11T21:29:13.332 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00090245099999999997 2024-03-11T21:29:13.332 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:29:13.332 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:29:13.332 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "sub_op_commit_rec", 2024-03-11T21:29:13.332 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:27:50.072487+0000", 2024-03-11T21:29:13.332 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00071514300000000001 2024-03-11T21:29:13.332 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:29:13.332 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:29:13.332 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "sub_op_commit_rec", 2024-03-11T21:29:13.332 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:27:50.146668+0000", 2024-03-11T21:29:13.332 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.074180605999999996 2024-03-11T21:29:13.333 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:29:13.333 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:29:13.333 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "commit_sent", 2024-03-11T21:29:13.333 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:27:50.146774+0000", 2024-03-11T21:29:13.333 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.000106079 2024-03-11T21:29:13.333 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:29:13.333 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:29:13.333 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "done", 2024-03-11T21:29:13.333 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:27:50.146833+0000", 2024-03-11T21:29:13.333 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 5.9840999999999998e-05 2024-03-11T21:29:13.333 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-03-11T21:29:13.333 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-03-11T21:29:13.333 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-03-11T21:29:13.333 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:29:13.334 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:29:13.334 INFO:teuthology.orchestra.run.smithi092.stdout: "description": "osd_op(client.34715.0:32047 7.8 7:1cc332d5:::benchmark_data_smithi092_76040_object32046:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e105)", 2024-03-11T21:29:13.334 INFO:teuthology.orchestra.run.smithi092.stdout: "initiated_at": "2024-03-11T21:27:57.779953+0000", 2024-03-11T21:29:13.334 INFO:teuthology.orchestra.run.smithi092.stdout: "age": 75.561432131999993, 2024-03-11T21:29:13.334 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.084056040999999998, 2024-03-11T21:29:13.334 INFO:teuthology.orchestra.run.smithi092.stdout: "type_data": { 2024-03-11T21:29:13.334 INFO:teuthology.orchestra.run.smithi092.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-11T21:29:13.334 INFO:teuthology.orchestra.run.smithi092.stdout: "client_info": { 2024-03-11T21:29:13.334 INFO:teuthology.orchestra.run.smithi092.stdout: "client": "client.34715", 2024-03-11T21:29:13.334 INFO:teuthology.orchestra.run.smithi092.stdout: "client_addr": "v1:172.21.15.92:0/2403132573", 2024-03-11T21:29:13.334 INFO:teuthology.orchestra.run.smithi092.stdout: "tid": 32047 2024-03-11T21:29:13.334 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:29:13.334 INFO:teuthology.orchestra.run.smithi092.stdout: "events": [ 2024-03-11T21:29:13.335 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:29:13.335 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "initiated", 2024-03-11T21:29:13.335 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:27:57.779953+0000", 2024-03-11T21:29:13.335 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-03-11T21:29:13.335 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:29:13.335 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:29:13.335 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "header_read", 2024-03-11T21:29:13.335 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:27:57.779953+0000", 2024-03-11T21:29:13.335 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-03-11T21:29:13.335 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:29:13.335 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:29:13.335 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "throttled", 2024-03-11T21:29:13.335 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:27:57.779956+0000", 2024-03-11T21:29:13.335 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 2.069e-06 2024-03-11T21:29:13.335 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:29:13.336 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:29:13.336 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "all_read", 2024-03-11T21:29:13.336 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:27:57.779982+0000", 2024-03-11T21:29:13.336 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 2.6601000000000002e-05 2024-03-11T21:29:13.336 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:29:13.336 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:29:13.336 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "dispatched", 2024-03-11T21:29:13.336 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:27:57.780005+0000", 2024-03-11T21:29:13.336 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 2.2756e-05 2024-03-11T21:29:13.336 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:29:13.336 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:29:13.336 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "queued_for_pg", 2024-03-11T21:29:13.336 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:27:57.780014+0000", 2024-03-11T21:29:13.336 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 8.7630000000000006e-06 2024-03-11T21:29:13.336 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:29:13.337 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:29:13.337 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-03-11T21:29:13.337 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:27:57.780080+0000", 2024-03-11T21:29:13.337 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 6.6018000000000004e-05 2024-03-11T21:29:13.337 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:29:13.337 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:29:13.337 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "started", 2024-03-11T21:29:13.337 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:27:57.781113+0000", 2024-03-11T21:29:13.337 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.001033312 2024-03-11T21:29:13.337 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:29:13.337 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:29:13.337 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "waiting for subops from 4,7", 2024-03-11T21:29:13.337 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:27:57.781655+0000", 2024-03-11T21:29:13.338 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.000542341 2024-03-11T21:29:13.338 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:29:13.338 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:29:13.338 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "sub_op_commit_rec", 2024-03-11T21:29:13.338 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:27:57.783494+0000", 2024-03-11T21:29:13.338 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.0018391499999999999 2024-03-11T21:29:13.338 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:29:13.338 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:29:13.338 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "op_commit", 2024-03-11T21:29:13.338 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:27:57.783972+0000", 2024-03-11T21:29:13.338 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00047797699999999998 2024-03-11T21:29:13.338 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:29:13.338 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:29:13.338 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "sub_op_commit_rec", 2024-03-11T21:29:13.339 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:27:57.863900+0000", 2024-03-11T21:29:13.339 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.079927953999999996 2024-03-11T21:29:13.339 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:29:13.339 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:29:13.339 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "commit_sent", 2024-03-11T21:29:13.339 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:27:57.863960+0000", 2024-03-11T21:29:13.339 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 5.9917999999999998e-05 2024-03-11T21:29:13.339 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:29:13.339 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:29:13.339 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "done", 2024-03-11T21:29:13.339 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:27:57.864009+0000", 2024-03-11T21:29:13.339 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 4.9181999999999998e-05 2024-03-11T21:29:13.339 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-03-11T21:29:13.339 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-03-11T21:29:13.340 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-03-11T21:29:13.340 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:29:13.340 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:29:13.340 INFO:teuthology.orchestra.run.smithi092.stdout: "description": "osd_op(client.34715.0:70030 7.e 7:7517dc90:::benchmark_data_smithi092_76040_object70029:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e109)", 2024-03-11T21:29:13.340 INFO:teuthology.orchestra.run.smithi092.stdout: "initiated_at": "2024-03-11T21:28:24.740192+0000", 2024-03-11T21:29:13.340 INFO:teuthology.orchestra.run.smithi092.stdout: "age": 48.601193553000002, 2024-03-11T21:29:13.340 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.074230372000000003, 2024-03-11T21:29:13.340 INFO:teuthology.orchestra.run.smithi092.stdout: "type_data": { 2024-03-11T21:29:13.340 INFO:teuthology.orchestra.run.smithi092.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-11T21:29:13.340 INFO:teuthology.orchestra.run.smithi092.stdout: "client_info": { 2024-03-11T21:29:13.340 INFO:teuthology.orchestra.run.smithi092.stdout: "client": "client.34715", 2024-03-11T21:29:13.341 INFO:teuthology.orchestra.run.smithi092.stdout: "client_addr": "v1:172.21.15.92:0/2403132573", 2024-03-11T21:29:13.341 INFO:teuthology.orchestra.run.smithi092.stdout: "tid": 70030 2024-03-11T21:29:13.342 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:29:13.342 INFO:teuthology.orchestra.run.smithi092.stdout: "events": [ 2024-03-11T21:29:13.342 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:29:13.342 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "initiated", 2024-03-11T21:29:13.342 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:28:24.740192+0000", 2024-03-11T21:29:13.342 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-03-11T21:29:13.342 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:29:13.342 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:29:13.342 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "header_read", 2024-03-11T21:29:13.342 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:28:24.740192+0000", 2024-03-11T21:29:13.342 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-03-11T21:29:13.342 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:29:13.342 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:29:13.343 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "throttled", 2024-03-11T21:29:13.343 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:28:24.740193+0000", 2024-03-11T21:29:13.343 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 7.1399999999999996e-07 2024-03-11T21:29:13.343 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:29:13.343 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:29:13.343 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "all_read", 2024-03-11T21:29:13.343 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:28:24.740217+0000", 2024-03-11T21:29:13.343 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 2.4737e-05 2024-03-11T21:29:13.343 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:29:13.343 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:29:13.343 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "dispatched", 2024-03-11T21:29:13.343 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:28:24.740224+0000", 2024-03-11T21:29:13.343 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 6.6290000000000004e-06 2024-03-11T21:29:13.343 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:29:13.344 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:29:13.344 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "queued_for_pg", 2024-03-11T21:29:13.344 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:28:24.740230+0000", 2024-03-11T21:29:13.344 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 5.7939999999999999e-06 2024-03-11T21:29:13.344 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:29:13.344 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:29:13.344 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-03-11T21:29:13.344 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:28:24.740492+0000", 2024-03-11T21:29:13.344 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.000261847 2024-03-11T21:29:13.344 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:29:13.344 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:29:13.344 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "started", 2024-03-11T21:29:13.344 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:28:24.740598+0000", 2024-03-11T21:29:13.344 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.000106598 2024-03-11T21:29:13.344 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:29:13.345 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:29:13.345 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "waiting for subops from 0,5", 2024-03-11T21:29:13.345 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:28:24.740722+0000", 2024-03-11T21:29:13.345 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00012385 2024-03-11T21:29:13.345 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:29:13.345 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:29:13.345 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "sub_op_commit_rec", 2024-03-11T21:29:13.345 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:28:24.745569+0000", 2024-03-11T21:29:13.345 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.0048465490000000003 2024-03-11T21:29:13.345 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:29:13.345 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:29:13.345 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "op_commit", 2024-03-11T21:29:13.345 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:28:24.745581+0000", 2024-03-11T21:29:13.346 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.2310000000000001e-05 2024-03-11T21:29:13.346 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:29:13.346 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:29:13.346 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "sub_op_commit_rec", 2024-03-11T21:29:13.346 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:28:24.814295+0000", 2024-03-11T21:29:13.346 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.068713508000000006 2024-03-11T21:29:13.346 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:29:13.346 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:29:13.346 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "commit_sent", 2024-03-11T21:29:13.346 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:28:24.814376+0000", 2024-03-11T21:29:13.346 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 8.1097e-05 2024-03-11T21:29:13.346 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:29:13.346 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:29:13.346 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "done", 2024-03-11T21:29:13.347 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:28:24.814422+0000", 2024-03-11T21:29:13.347 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 4.6739000000000001e-05 2024-03-11T21:29:13.347 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-03-11T21:29:13.347 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-03-11T21:29:13.347 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-03-11T21:29:13.347 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:29:13.347 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:29:13.347 INFO:teuthology.orchestra.run.smithi092.stdout: "description": "osd_op(client.34715.0:70031 7.e 7:79dad857:::benchmark_data_smithi092_76040_object70030:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e109)", 2024-03-11T21:29:13.347 INFO:teuthology.orchestra.run.smithi092.stdout: "initiated_at": "2024-03-11T21:28:24.740242+0000", 2024-03-11T21:29:13.347 INFO:teuthology.orchestra.run.smithi092.stdout: "age": 48.601143575000002, 2024-03-11T21:29:13.347 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.074380246999999997, 2024-03-11T21:29:13.347 INFO:teuthology.orchestra.run.smithi092.stdout: "type_data": { 2024-03-11T21:29:13.347 INFO:teuthology.orchestra.run.smithi092.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-11T21:29:13.347 INFO:teuthology.orchestra.run.smithi092.stdout: "client_info": { 2024-03-11T21:29:13.348 INFO:teuthology.orchestra.run.smithi092.stdout: "client": "client.34715", 2024-03-11T21:29:13.348 INFO:teuthology.orchestra.run.smithi092.stdout: "client_addr": "v1:172.21.15.92:0/2403132573", 2024-03-11T21:29:13.348 INFO:teuthology.orchestra.run.smithi092.stdout: "tid": 70031 2024-03-11T21:29:13.348 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:29:13.348 INFO:teuthology.orchestra.run.smithi092.stdout: "events": [ 2024-03-11T21:29:13.348 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:29:13.348 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "initiated", 2024-03-11T21:29:13.348 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:28:24.740242+0000", 2024-03-11T21:29:13.348 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-03-11T21:29:13.348 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:29:13.348 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:29:13.348 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "header_read", 2024-03-11T21:29:13.348 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:28:24.740242+0000", 2024-03-11T21:29:13.348 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-03-11T21:29:13.349 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:29:13.349 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:29:13.349 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "throttled", 2024-03-11T21:29:13.349 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:28:24.740243+0000", 2024-03-11T21:29:13.349 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 6.0100000000000005e-07 2024-03-11T21:29:13.349 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:29:13.349 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:29:13.349 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "all_read", 2024-03-11T21:29:13.349 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:28:24.740262+0000", 2024-03-11T21:29:13.349 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.9264000000000001e-05 2024-03-11T21:29:13.349 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:29:13.349 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:29:13.349 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "dispatched", 2024-03-11T21:29:13.350 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:28:24.740267+0000", 2024-03-11T21:29:13.350 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 5.609e-06 2024-03-11T21:29:13.350 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:29:13.350 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:29:13.350 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "queued_for_pg", 2024-03-11T21:29:13.350 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:28:24.740273+0000", 2024-03-11T21:29:13.350 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 5.0320000000000003e-06 2024-03-11T21:29:13.350 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:29:13.350 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:29:13.350 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-03-11T21:29:13.350 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:28:24.743224+0000", 2024-03-11T21:29:13.350 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.0029509969999999999 2024-03-11T21:29:13.350 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:29:13.351 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:29:13.351 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "started", 2024-03-11T21:29:13.351 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:28:24.743389+0000", 2024-03-11T21:29:13.351 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00016508399999999999 2024-03-11T21:29:13.351 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:29:13.351 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:29:13.351 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "waiting for subops from 0,5", 2024-03-11T21:29:13.351 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:28:24.743542+0000", 2024-03-11T21:29:13.351 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00015306599999999999 2024-03-11T21:29:13.351 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:29:13.351 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:29:13.351 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "op_commit", 2024-03-11T21:29:13.351 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:28:24.745736+0000", 2024-03-11T21:29:13.352 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.002194292 2024-03-11T21:29:13.352 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:29:13.352 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:29:13.352 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "sub_op_commit_rec", 2024-03-11T21:29:13.352 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:28:24.745962+0000", 2024-03-11T21:29:13.352 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00022648500000000001 2024-03-11T21:29:13.352 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:29:13.352 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:29:13.352 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "sub_op_commit_rec", 2024-03-11T21:29:13.352 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:28:24.814467+0000", 2024-03-11T21:29:13.352 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.068504240999999993 2024-03-11T21:29:13.352 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:29:13.352 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:29:13.352 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "commit_sent", 2024-03-11T21:29:13.353 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:28:24.814519+0000", 2024-03-11T21:29:13.353 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 5.2046999999999998e-05 2024-03-11T21:29:13.353 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:29:13.353 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:29:13.353 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "done", 2024-03-11T21:29:13.353 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:28:24.814622+0000", 2024-03-11T21:29:13.353 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.000103529 2024-03-11T21:29:13.353 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-03-11T21:29:13.353 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-03-11T21:29:13.353 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-03-11T21:29:13.354 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:29:13.354 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:29:13.354 INFO:teuthology.orchestra.run.smithi092.stdout: "description": "osd_op(client.34715.0:104678 7.e 7:77299287:::benchmark_data_smithi092_76040_object104677:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e117)", 2024-03-11T21:29:13.354 INFO:teuthology.orchestra.run.smithi092.stdout: "initiated_at": "2024-03-11T21:28:57.288999+0000", 2024-03-11T21:29:13.354 INFO:teuthology.orchestra.run.smithi092.stdout: "age": 16.052386354999999, 2024-03-11T21:29:13.354 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.092710055999999999, 2024-03-11T21:29:13.354 INFO:teuthology.orchestra.run.smithi092.stdout: "type_data": { 2024-03-11T21:29:13.354 INFO:teuthology.orchestra.run.smithi092.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-11T21:29:13.354 INFO:teuthology.orchestra.run.smithi092.stdout: "client_info": { 2024-03-11T21:29:13.354 INFO:teuthology.orchestra.run.smithi092.stdout: "client": "client.34715", 2024-03-11T21:29:13.354 INFO:teuthology.orchestra.run.smithi092.stdout: "client_addr": "v1:172.21.15.92:0/2403132573", 2024-03-11T21:29:13.354 INFO:teuthology.orchestra.run.smithi092.stdout: "tid": 104678 2024-03-11T21:29:13.354 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:29:13.355 INFO:teuthology.orchestra.run.smithi092.stdout: "events": [ 2024-03-11T21:29:13.355 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:29:13.355 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "initiated", 2024-03-11T21:29:13.355 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:28:57.288999+0000", 2024-03-11T21:29:13.355 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-03-11T21:29:13.355 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:29:13.355 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:29:13.355 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "header_read", 2024-03-11T21:29:13.355 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:28:57.288999+0000", 2024-03-11T21:29:13.355 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-03-11T21:29:13.355 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:29:13.355 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:29:13.356 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "throttled", 2024-03-11T21:29:13.356 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:28:57.289041+0000", 2024-03-11T21:29:13.356 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 4.1856e-05 2024-03-11T21:29:13.356 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:29:13.356 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:29:13.356 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "all_read", 2024-03-11T21:29:13.356 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:28:57.289064+0000", 2024-03-11T21:29:13.356 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 2.3112000000000002e-05 2024-03-11T21:29:13.356 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:29:13.356 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:29:13.356 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "dispatched", 2024-03-11T21:29:13.356 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:28:57.289075+0000", 2024-03-11T21:29:13.357 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.0312000000000001e-05 2024-03-11T21:29:13.357 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:29:13.357 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:29:13.357 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "queued_for_pg", 2024-03-11T21:29:13.357 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:28:57.289082+0000", 2024-03-11T21:29:13.357 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 7.3599999999999998e-06 2024-03-11T21:29:13.357 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:29:13.357 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:29:13.357 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-03-11T21:29:13.357 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:28:57.289159+0000", 2024-03-11T21:29:13.357 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 7.7372000000000004e-05 2024-03-11T21:29:13.357 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:29:13.357 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:29:13.358 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "started", 2024-03-11T21:29:13.358 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:28:57.289288+0000", 2024-03-11T21:29:13.358 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00012846999999999999 2024-03-11T21:29:13.358 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:29:13.358 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:29:13.358 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "waiting for subops from 0,5", 2024-03-11T21:29:13.358 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:28:57.289421+0000", 2024-03-11T21:29:13.358 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00013358000000000001 2024-03-11T21:29:13.358 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:29:13.358 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:29:13.358 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "sub_op_commit_rec", 2024-03-11T21:29:13.358 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:28:57.378143+0000", 2024-03-11T21:29:13.358 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.088721513000000002 2024-03-11T21:29:13.359 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:29:13.359 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:29:13.359 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "sub_op_commit_rec", 2024-03-11T21:29:13.359 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:28:57.378266+0000", 2024-03-11T21:29:13.359 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00012305700000000001 2024-03-11T21:29:13.359 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:29:13.359 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:29:13.359 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "op_commit", 2024-03-11T21:29:13.359 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:28:57.381595+0000", 2024-03-11T21:29:13.359 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.0033288409999999999 2024-03-11T21:29:13.359 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:29:13.359 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:29:13.360 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "commit_sent", 2024-03-11T21:29:13.360 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:28:57.381664+0000", 2024-03-11T21:29:13.361 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 6.9708000000000004e-05 2024-03-11T21:29:13.362 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:29:13.362 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:29:13.362 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "done", 2024-03-11T21:29:13.362 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:28:57.381709+0000", 2024-03-11T21:29:13.362 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 4.4875e-05 2024-03-11T21:29:13.362 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-03-11T21:29:13.362 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-03-11T21:29:13.362 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-03-11T21:29:13.362 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:29:13.362 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:29:13.362 INFO:teuthology.orchestra.run.smithi092.stdout: "description": "osd_op(client.34715.0:104682 7.e 7:7b3e8d68:::benchmark_data_smithi092_76040_object104681:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e117)", 2024-03-11T21:29:13.363 INFO:teuthology.orchestra.run.smithi092.stdout: "initiated_at": "2024-03-11T21:28:57.292170+0000", 2024-03-11T21:29:13.363 INFO:teuthology.orchestra.run.smithi092.stdout: "age": 16.049215414999999, 2024-03-11T21:29:13.363 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.092283122999999995, 2024-03-11T21:29:13.363 INFO:teuthology.orchestra.run.smithi092.stdout: "type_data": { 2024-03-11T21:29:13.363 INFO:teuthology.orchestra.run.smithi092.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-11T21:29:13.363 INFO:teuthology.orchestra.run.smithi092.stdout: "client_info": { 2024-03-11T21:29:13.363 INFO:teuthology.orchestra.run.smithi092.stdout: "client": "client.34715", 2024-03-11T21:29:13.363 INFO:teuthology.orchestra.run.smithi092.stdout: "client_addr": "v1:172.21.15.92:0/2403132573", 2024-03-11T21:29:13.363 INFO:teuthology.orchestra.run.smithi092.stdout: "tid": 104682 2024-03-11T21:29:13.363 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:29:13.363 INFO:teuthology.orchestra.run.smithi092.stdout: "events": [ 2024-03-11T21:29:13.363 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:29:13.363 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "initiated", 2024-03-11T21:29:13.363 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:28:57.292170+0000", 2024-03-11T21:29:13.364 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-03-11T21:29:13.364 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:29:13.364 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:29:13.364 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "header_read", 2024-03-11T21:29:13.364 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:28:57.292170+0000", 2024-03-11T21:29:13.364 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-03-11T21:29:13.364 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:29:13.364 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:29:13.364 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "throttled", 2024-03-11T21:29:13.364 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:28:57.292173+0000", 2024-03-11T21:29:13.364 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 2.542e-06 2024-03-11T21:29:13.364 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:29:13.364 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:29:13.364 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "all_read", 2024-03-11T21:29:13.365 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:28:57.292200+0000", 2024-03-11T21:29:13.365 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 2.7288e-05 2024-03-11T21:29:13.365 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:29:13.365 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:29:13.365 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "dispatched", 2024-03-11T21:29:13.365 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:28:57.375541+0000", 2024-03-11T21:29:13.365 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.083340996000000001 2024-03-11T21:29:13.365 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:29:13.365 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:29:13.365 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "queued_for_pg", 2024-03-11T21:29:13.365 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:28:57.375557+0000", 2024-03-11T21:29:13.365 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.5933000000000001e-05 2024-03-11T21:29:13.365 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:29:13.365 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:29:13.366 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-03-11T21:29:13.366 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:28:57.378303+0000", 2024-03-11T21:29:13.366 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.002745824 2024-03-11T21:29:13.366 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:29:13.366 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:29:13.366 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "started", 2024-03-11T21:29:13.366 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:28:57.378444+0000", 2024-03-11T21:29:13.366 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00014127399999999999 2024-03-11T21:29:13.366 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:29:13.366 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:29:13.366 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "waiting for subops from 0,5", 2024-03-11T21:29:13.366 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:28:57.378537+0000", 2024-03-11T21:29:13.366 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 9.3263000000000005e-05 2024-03-11T21:29:13.366 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:29:13.367 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:29:13.367 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "op_commit", 2024-03-11T21:29:13.367 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:28:57.381723+0000", 2024-03-11T21:29:13.367 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.0031855989999999999 2024-03-11T21:29:13.367 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:29:13.367 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:29:13.367 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "sub_op_commit_rec", 2024-03-11T21:29:13.367 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:28:57.381875+0000", 2024-03-11T21:29:13.367 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00015235199999999999 2024-03-11T21:29:13.367 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:29:13.367 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:29:13.367 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "sub_op_commit_rec", 2024-03-11T21:29:13.367 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:28:57.384116+0000", 2024-03-11T21:29:13.368 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.002240279 2024-03-11T21:29:13.368 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:29:13.368 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:29:13.368 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "commit_sent", 2024-03-11T21:29:13.368 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:28:57.384313+0000", 2024-03-11T21:29:13.368 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.000197648 2024-03-11T21:29:13.368 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:29:13.368 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:29:13.368 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "done", 2024-03-11T21:29:13.368 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:28:57.384453+0000", 2024-03-11T21:29:13.368 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.000140125 2024-03-11T21:29:13.368 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-03-11T21:29:13.368 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-03-11T21:29:13.368 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-03-11T21:29:13.369 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:29:13.369 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:29:13.369 INFO:teuthology.orchestra.run.smithi092.stdout: "description": "osd_op(client.34715.0:110559 7.e 7:7374bcc8:::benchmark_data_smithi092_76040_object110558:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e119)", 2024-03-11T21:29:13.369 INFO:teuthology.orchestra.run.smithi092.stdout: "initiated_at": "2024-03-11T21:29:03.032051+0000", 2024-03-11T21:29:13.369 INFO:teuthology.orchestra.run.smithi092.stdout: "age": 10.309334208999999, 2024-03-11T21:29:13.369 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.076736466000000003, 2024-03-11T21:29:13.369 INFO:teuthology.orchestra.run.smithi092.stdout: "type_data": { 2024-03-11T21:29:13.369 INFO:teuthology.orchestra.run.smithi092.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-11T21:29:13.369 INFO:teuthology.orchestra.run.smithi092.stdout: "client_info": { 2024-03-11T21:29:13.369 INFO:teuthology.orchestra.run.smithi092.stdout: "client": "client.34715", 2024-03-11T21:29:13.369 INFO:teuthology.orchestra.run.smithi092.stdout: "client_addr": "v1:172.21.15.92:0/2403132573", 2024-03-11T21:29:13.369 INFO:teuthology.orchestra.run.smithi092.stdout: "tid": 110559 2024-03-11T21:29:13.369 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:29:13.369 INFO:teuthology.orchestra.run.smithi092.stdout: "events": [ 2024-03-11T21:29:13.369 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:29:13.369 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "initiated", 2024-03-11T21:29:13.370 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:29:03.032051+0000", 2024-03-11T21:29:13.370 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-03-11T21:29:13.370 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:29:13.370 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:29:13.370 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "header_read", 2024-03-11T21:29:13.370 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:29:03.032051+0000", 2024-03-11T21:29:13.370 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-03-11T21:29:13.370 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:29:13.370 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:29:13.370 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "throttled", 2024-03-11T21:29:13.370 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:29:03.032053+0000", 2024-03-11T21:29:13.370 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.291e-06 2024-03-11T21:29:13.370 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:29:13.371 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:29:13.371 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "all_read", 2024-03-11T21:29:13.371 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:29:03.032079+0000", 2024-03-11T21:29:13.371 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 2.6186e-05 2024-03-11T21:29:13.371 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:29:13.371 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:29:13.371 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "dispatched", 2024-03-11T21:29:13.371 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:29:03.032093+0000", 2024-03-11T21:29:13.371 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.4644000000000001e-05 2024-03-11T21:29:13.371 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:29:13.371 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:29:13.371 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "queued_for_pg", 2024-03-11T21:29:13.371 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:29:03.032102+0000", 2024-03-11T21:29:13.371 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 8.6179999999999995e-06 2024-03-11T21:29:13.371 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:29:13.372 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:29:13.372 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:29:13.372 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:29:13.373 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:29:13.373 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-03-11T21:29:13.373 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:29:03.059399+0000", 2024-03-11T21:29:13.373 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.02729678 2024-03-11T21:29:13.373 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:29:13.373 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:29:13.373 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "started", 2024-03-11T21:29:13.374 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:29:03.059542+0000", 2024-03-11T21:29:13.374 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.000143304 2024-03-11T21:29:13.374 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:29:13.374 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:29:13.374 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "waiting for subops from 0,5", 2024-03-11T21:29:13.374 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:29:03.059690+0000", 2024-03-11T21:29:13.374 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00014825400000000001 2024-03-11T21:29:13.374 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:29:13.374 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:29:13.374 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "sub_op_commit_rec", 2024-03-11T21:29:13.374 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:29:03.064538+0000", 2024-03-11T21:29:13.374 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.0048477279999999999 2024-03-11T21:29:13.374 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:29:13.375 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:29:13.375 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "op_commit", 2024-03-11T21:29:13.375 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:29:03.064582+0000", 2024-03-11T21:29:13.375 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 4.3748000000000002e-05 2024-03-11T21:29:13.375 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:29:13.375 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:29:13.375 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "sub_op_commit_rec", 2024-03-11T21:29:13.375 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:29:03.108689+0000", 2024-03-11T21:29:13.375 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.044107199999999999 2024-03-11T21:29:13.375 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:29:13.375 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:29:13.375 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "commit_sent", 2024-03-11T21:29:13.375 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:29:03.108745+0000", 2024-03-11T21:29:13.375 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 5.5553000000000002e-05 2024-03-11T21:29:13.375 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:29:13.376 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:29:13.376 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "done", 2024-03-11T21:29:13.376 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:29:03.108788+0000", 2024-03-11T21:29:13.376 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 4.316e-05 2024-03-11T21:29:13.376 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-03-11T21:29:13.376 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-03-11T21:29:13.376 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-03-11T21:29:13.376 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:29:13.376 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:29:13.376 INFO:teuthology.orchestra.run.smithi092.stdout: "description": "osd_op(client.34715.0:115732 7.e 7:7582fdce:::benchmark_data_smithi092_76040_object115731:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e119)", 2024-03-11T21:29:13.376 INFO:teuthology.orchestra.run.smithi092.stdout: "initiated_at": "2024-03-11T21:29:08.166630+0000", 2024-03-11T21:29:13.376 INFO:teuthology.orchestra.run.smithi092.stdout: "age": 5.1747553909999997, 2024-03-11T21:29:13.376 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.099177280000000007, 2024-03-11T21:29:13.376 INFO:teuthology.orchestra.run.smithi092.stdout: "type_data": { 2024-03-11T21:29:13.377 INFO:teuthology.orchestra.run.smithi092.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-11T21:29:13.377 INFO:teuthology.orchestra.run.smithi092.stdout: "client_info": { 2024-03-11T21:29:13.377 INFO:teuthology.orchestra.run.smithi092.stdout: "client": "client.34715", 2024-03-11T21:29:13.377 INFO:teuthology.orchestra.run.smithi092.stdout: "client_addr": "v1:172.21.15.92:0/2403132573", 2024-03-11T21:29:13.377 INFO:teuthology.orchestra.run.smithi092.stdout: "tid": 115732 2024-03-11T21:29:13.377 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:29:13.377 INFO:teuthology.orchestra.run.smithi092.stdout: "events": [ 2024-03-11T21:29:13.377 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:29:13.377 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "initiated", 2024-03-11T21:29:13.377 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:29:08.166630+0000", 2024-03-11T21:29:13.377 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-03-11T21:29:13.377 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:29:13.377 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:29:13.377 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "header_read", 2024-03-11T21:29:13.377 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:29:08.166630+0000", 2024-03-11T21:29:13.378 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-03-11T21:29:13.378 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:29:13.378 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:29:13.378 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "throttled", 2024-03-11T21:29:13.378 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:29:08.166634+0000", 2024-03-11T21:29:13.378 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 3.4680000000000001e-06 2024-03-11T21:29:13.378 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:29:13.378 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:29:13.378 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "all_read", 2024-03-11T21:29:13.378 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:29:08.166660+0000", 2024-03-11T21:29:13.378 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 2.6512e-05 2024-03-11T21:29:13.378 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:29:13.378 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:29:13.378 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "dispatched", 2024-03-11T21:29:13.378 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:29:08.166674+0000", 2024-03-11T21:29:13.379 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.382e-05 2024-03-11T21:29:13.379 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:29:13.379 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:29:13.379 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "queued_for_pg", 2024-03-11T21:29:13.379 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:29:08.166685+0000", 2024-03-11T21:29:13.379 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.1277999999999999e-05 2024-03-11T21:29:13.379 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:29:13.379 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:29:13.379 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-03-11T21:29:13.379 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:29:08.166750+0000", 2024-03-11T21:29:13.379 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 6.4522000000000003e-05 2024-03-11T21:29:13.379 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:29:13.379 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:29:13.379 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "started", 2024-03-11T21:29:13.380 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:29:08.166894+0000", 2024-03-11T21:29:13.380 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.000144499 2024-03-11T21:29:13.380 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:29:13.380 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:29:13.380 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "waiting for subops from 0,5", 2024-03-11T21:29:13.380 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:29:08.167063+0000", 2024-03-11T21:29:13.380 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00016860700000000001 2024-03-11T21:29:13.380 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:29:13.380 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:29:13.380 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "sub_op_commit_rec", 2024-03-11T21:29:13.380 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:29:08.172918+0000", 2024-03-11T21:29:13.380 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.0058549780000000003 2024-03-11T21:29:13.380 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:29:13.380 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:29:13.380 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "op_commit", 2024-03-11T21:29:13.381 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:29:08.181135+0000", 2024-03-11T21:29:13.381 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.0082175979999999996 2024-03-11T21:29:13.381 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:29:13.381 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:29:13.381 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "sub_op_commit_rec", 2024-03-11T21:29:13.381 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:29:08.265705+0000", 2024-03-11T21:29:13.381 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.084569474000000006 2024-03-11T21:29:13.381 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:29:13.381 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:29:13.381 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "commit_sent", 2024-03-11T21:29:13.381 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:29:08.265763+0000", 2024-03-11T21:29:13.381 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 5.8242999999999998e-05 2024-03-11T21:29:13.381 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:29:13.381 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:29:13.381 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "done", 2024-03-11T21:29:13.382 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:29:08.265807+0000", 2024-03-11T21:29:13.382 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 4.4280999999999998e-05 2024-03-11T21:29:13.382 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-03-11T21:29:13.382 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-03-11T21:29:13.382 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-03-11T21:29:13.382 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:29:13.382 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:29:13.382 INFO:teuthology.orchestra.run.smithi092.stdout: "description": "osd_op(client.34715.0:115741 7.e 7:72f34471:::benchmark_data_smithi092_76040_object115740:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e119)", 2024-03-11T21:29:13.382 INFO:teuthology.orchestra.run.smithi092.stdout: "initiated_at": "2024-03-11T21:29:08.181657+0000", 2024-03-11T21:29:13.382 INFO:teuthology.orchestra.run.smithi092.stdout: "age": 5.1597281910000001, 2024-03-11T21:29:13.382 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.084254920999999997, 2024-03-11T21:29:13.382 INFO:teuthology.orchestra.run.smithi092.stdout: "type_data": { 2024-03-11T21:29:13.382 INFO:teuthology.orchestra.run.smithi092.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-11T21:29:13.382 INFO:teuthology.orchestra.run.smithi092.stdout: "client_info": { 2024-03-11T21:29:13.383 INFO:teuthology.orchestra.run.smithi092.stdout: "client": "client.34715", 2024-03-11T21:29:13.383 INFO:teuthology.orchestra.run.smithi092.stdout: "client_addr": "v1:172.21.15.92:0/2403132573", 2024-03-11T21:29:13.383 INFO:teuthology.orchestra.run.smithi092.stdout: "tid": 115741 2024-03-11T21:29:13.383 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:29:13.383 INFO:teuthology.orchestra.run.smithi092.stdout: "events": [ 2024-03-11T21:29:13.383 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:29:13.383 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "initiated", 2024-03-11T21:29:13.383 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:29:08.181657+0000", 2024-03-11T21:29:13.383 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-03-11T21:29:13.383 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:29:13.383 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:29:13.383 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "header_read", 2024-03-11T21:29:13.383 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:29:08.181657+0000", 2024-03-11T21:29:13.383 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-03-11T21:29:13.383 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:29:13.384 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:29:13.384 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "throttled", 2024-03-11T21:29:13.384 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:29:08.181659+0000", 2024-03-11T21:29:13.384 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.6980000000000001e-06 2024-03-11T21:29:13.384 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:29:13.384 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:29:13.384 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "all_read", 2024-03-11T21:29:13.384 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:29:08.181688+0000", 2024-03-11T21:29:13.384 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 2.8827999999999999e-05 2024-03-11T21:29:13.384 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:29:13.384 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:29:13.384 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "dispatched", 2024-03-11T21:29:13.384 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:29:08.181700+0000", 2024-03-11T21:29:13.384 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.1939999999999999e-05 2024-03-11T21:29:13.384 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:29:13.385 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:29:13.385 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "queued_for_pg", 2024-03-11T21:29:13.385 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:29:08.181708+0000", 2024-03-11T21:29:13.385 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 8.5890000000000003e-06 2024-03-11T21:29:13.385 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:29:13.385 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:29:13.385 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-03-11T21:29:13.385 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:29:08.181755+0000", 2024-03-11T21:29:13.385 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 4.6702000000000001e-05 2024-03-11T21:29:13.385 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:29:13.385 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:29:13.385 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "started", 2024-03-11T21:29:13.385 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:29:08.181870+0000", 2024-03-11T21:29:13.385 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00011508 2024-03-11T21:29:13.386 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:29:13.386 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:29:13.386 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "waiting for subops from 0,5", 2024-03-11T21:29:13.386 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:29:08.182022+0000", 2024-03-11T21:29:13.386 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00015190099999999999 2024-03-11T21:29:13.386 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:29:13.386 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:29:13.386 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "sub_op_commit_rec", 2024-03-11T21:29:13.386 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:29:08.184981+0000", 2024-03-11T21:29:13.386 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.0029593559999999998 2024-03-11T21:29:13.386 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:29:13.386 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:29:13.386 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "op_commit", 2024-03-11T21:29:13.386 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:29:08.186599+0000", 2024-03-11T21:29:13.387 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.0016174830000000001 2024-03-11T21:29:13.387 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:29:13.387 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:29:13.387 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "sub_op_commit_rec", 2024-03-11T21:29:13.387 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:29:08.265841+0000", 2024-03-11T21:29:13.387 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.079241696 2024-03-11T21:29:13.387 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:29:13.387 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:29:13.387 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "commit_sent", 2024-03-11T21:29:13.387 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:29:08.265883+0000", 2024-03-11T21:29:13.387 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 4.2534000000000003e-05 2024-03-11T21:29:13.387 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:29:13.387 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:29:13.388 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "done", 2024-03-11T21:29:13.388 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:29:08.265912+0000", 2024-03-11T21:29:13.388 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 2.9113999999999999e-05 2024-03-11T21:29:13.388 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-03-11T21:29:13.388 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-03-11T21:29:13.388 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-03-11T21:29:13.388 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:29:13.388 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:29:13.388 INFO:teuthology.orchestra.run.smithi092.stdout: "description": "osd_op(client.34715.0:115749 7.8 7:183d5f6b:::benchmark_data_smithi092_76040_object115748:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e119)", 2024-03-11T21:29:13.389 INFO:teuthology.orchestra.run.smithi092.stdout: "initiated_at": "2024-03-11T21:29:08.204342+0000", 2024-03-11T21:29:13.389 INFO:teuthology.orchestra.run.smithi092.stdout: "age": 5.137043781, 2024-03-11T21:29:13.389 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.077972018000000004, 2024-03-11T21:29:13.389 INFO:teuthology.orchestra.run.smithi092.stdout: "type_data": { 2024-03-11T21:29:13.389 INFO:teuthology.orchestra.run.smithi092.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-11T21:29:13.389 INFO:teuthology.orchestra.run.smithi092.stdout: "client_info": { 2024-03-11T21:29:13.389 INFO:teuthology.orchestra.run.smithi092.stdout: "client": "client.34715", 2024-03-11T21:29:13.389 INFO:teuthology.orchestra.run.smithi092.stdout: "client_addr": "v1:172.21.15.92:0/2403132573", 2024-03-11T21:29:13.389 INFO:teuthology.orchestra.run.smithi092.stdout: "tid": 115749 2024-03-11T21:29:13.389 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:29:13.389 INFO:teuthology.orchestra.run.smithi092.stdout: "events": [ 2024-03-11T21:29:13.389 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:29:13.390 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "initiated", 2024-03-11T21:29:13.390 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:29:08.204342+0000", 2024-03-11T21:29:13.390 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-03-11T21:29:13.390 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:29:13.390 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:29:13.390 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "header_read", 2024-03-11T21:29:13.390 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:29:08.204342+0000", 2024-03-11T21:29:13.390 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-03-11T21:29:13.390 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:29:13.390 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:29:13.390 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "throttled", 2024-03-11T21:29:13.390 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:29:08.204344+0000", 2024-03-11T21:29:13.391 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.855e-06 2024-03-11T21:29:13.391 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:29:13.391 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:29:13.391 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "all_read", 2024-03-11T21:29:13.391 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:29:08.204385+0000", 2024-03-11T21:29:13.391 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 4.0883999999999997e-05 2024-03-11T21:29:13.391 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:29:13.391 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:29:13.391 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "dispatched", 2024-03-11T21:29:13.391 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:29:08.204399+0000", 2024-03-11T21:29:13.391 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.4072e-05 2024-03-11T21:29:13.392 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:29:13.392 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:29:13.392 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "queued_for_pg", 2024-03-11T21:29:13.392 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:29:08.204408+0000", 2024-03-11T21:29:13.392 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 9.2799999999999992e-06 2024-03-11T21:29:13.392 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:29:13.392 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:29:13.392 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-03-11T21:29:13.392 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:29:08.204505+0000", 2024-03-11T21:29:13.392 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 9.6830000000000006e-05 2024-03-11T21:29:13.392 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:29:13.392 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:29:13.393 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "started", 2024-03-11T21:29:13.393 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:29:08.265094+0000", 2024-03-11T21:29:13.393 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.060589257000000001 2024-03-11T21:29:13.393 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:29:13.393 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:29:13.393 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "waiting for subops from 4,7", 2024-03-11T21:29:13.393 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:29:08.267947+0000", 2024-03-11T21:29:13.393 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.0028534350000000001 2024-03-11T21:29:13.393 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:29:13.393 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:29:13.393 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "sub_op_commit_rec", 2024-03-11T21:29:13.394 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:29:08.282150+0000", 2024-03-11T21:29:13.394 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.014202177999999999 2024-03-11T21:29:13.394 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:29:13.394 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:29:13.394 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "op_commit", 2024-03-11T21:29:13.394 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:29:08.282161+0000", 2024-03-11T21:29:13.394 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.1663e-05 2024-03-11T21:29:13.394 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:29:13.394 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:29:13.394 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "sub_op_commit_rec", 2024-03-11T21:29:13.394 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:29:08.282192+0000", 2024-03-11T21:29:13.394 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 3.0923000000000002e-05 2024-03-11T21:29:13.395 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:29:13.395 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:29:13.395 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "commit_sent", 2024-03-11T21:29:13.395 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:29:08.282248+0000", 2024-03-11T21:29:13.395 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 5.5933000000000001e-05 2024-03-11T21:29:13.395 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:29:13.395 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:29:13.395 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "done", 2024-03-11T21:29:13.395 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:29:08.282314+0000", 2024-03-11T21:29:13.396 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 6.5708000000000002e-05 2024-03-11T21:29:13.396 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-03-11T21:29:13.396 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-03-11T21:29:13.396 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-03-11T21:29:13.396 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:29:13.396 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:29:13.396 INFO:teuthology.orchestra.run.smithi092.stdout: "description": "osd_op(client.34715.0:116287 7.8 7:161bac11:::benchmark_data_smithi092_76040_object116286:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e119)", 2024-03-11T21:29:13.396 INFO:teuthology.orchestra.run.smithi092.stdout: "initiated_at": "2024-03-11T21:29:08.771810+0000", 2024-03-11T21:29:13.396 INFO:teuthology.orchestra.run.smithi092.stdout: "age": 4.5695760170000002, 2024-03-11T21:29:13.396 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.10587118700000001, 2024-03-11T21:29:13.396 INFO:teuthology.orchestra.run.smithi092.stdout: "type_data": { 2024-03-11T21:29:13.396 INFO:teuthology.orchestra.run.smithi092.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-11T21:29:13.397 INFO:teuthology.orchestra.run.smithi092.stdout: "client_info": { 2024-03-11T21:29:13.397 INFO:teuthology.orchestra.run.smithi092.stdout: "client": "client.34715", 2024-03-11T21:29:13.397 INFO:teuthology.orchestra.run.smithi092.stdout: "client_addr": "v1:172.21.15.92:0/2403132573", 2024-03-11T21:29:13.397 INFO:teuthology.orchestra.run.smithi092.stdout: "tid": 116287 2024-03-11T21:29:13.397 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:29:13.397 INFO:teuthology.orchestra.run.smithi092.stdout: "events": [ 2024-03-11T21:29:13.397 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:29:13.397 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "initiated", 2024-03-11T21:29:13.397 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:29:08.771810+0000", 2024-03-11T21:29:13.397 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-03-11T21:29:13.397 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:29:13.398 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:29:13.398 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "header_read", 2024-03-11T21:29:13.398 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:29:08.771810+0000", 2024-03-11T21:29:13.398 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-03-11T21:29:13.398 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:29:13.398 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:29:13.398 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "throttled", 2024-03-11T21:29:13.398 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:29:08.771811+0000", 2024-03-11T21:29:13.398 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.793e-06 2024-03-11T21:29:13.398 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:29:13.398 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:29:13.398 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "all_read", 2024-03-11T21:29:13.399 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:29:08.771832+0000", 2024-03-11T21:29:13.399 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 2.0641000000000001e-05 2024-03-11T21:29:13.399 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:29:13.399 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:29:13.399 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "dispatched", 2024-03-11T21:29:13.399 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:29:08.771841+0000", 2024-03-11T21:29:13.399 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 9.2939999999999994e-06 2024-03-11T21:29:13.399 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:29:13.399 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:29:13.399 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "queued_for_pg", 2024-03-11T21:29:13.399 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:29:08.771849+0000", 2024-03-11T21:29:13.399 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 7.2239999999999998e-06 2024-03-11T21:29:13.399 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:29:13.400 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:29:13.400 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-03-11T21:29:13.400 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:29:08.771933+0000", 2024-03-11T21:29:13.400 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 8.4158000000000003e-05 2024-03-11T21:29:13.400 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:29:13.400 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:29:13.400 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "started", 2024-03-11T21:29:13.400 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:29:08.772098+0000", 2024-03-11T21:29:13.400 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00016576199999999999 2024-03-11T21:29:13.400 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:29:13.400 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:29:13.400 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "waiting for subops from 4,7", 2024-03-11T21:29:13.401 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:29:08.772361+0000", 2024-03-11T21:29:13.401 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00026245400000000002 2024-03-11T21:29:13.401 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:29:13.401 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:29:13.401 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "op_commit", 2024-03-11T21:29:13.401 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:29:08.877415+0000", 2024-03-11T21:29:13.401 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.10505452 2024-03-11T21:29:13.401 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:29:13.401 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:29:13.401 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "sub_op_commit_rec", 2024-03-11T21:29:13.401 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:29:08.877469+0000", 2024-03-11T21:29:13.401 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 5.3115000000000002e-05 2024-03-11T21:29:13.402 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:29:13.402 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:29:13.402 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "sub_op_commit_rec", 2024-03-11T21:29:13.402 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:29:08.877521+0000", 2024-03-11T21:29:13.402 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 5.2649999999999999e-05 2024-03-11T21:29:13.402 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:29:13.402 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:29:13.402 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "commit_sent", 2024-03-11T21:29:13.402 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:29:08.877632+0000", 2024-03-11T21:29:13.402 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00011079800000000001 2024-03-11T21:29:13.402 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:29:13.403 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:29:13.403 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "done", 2024-03-11T21:29:13.403 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:29:08.877681+0000", 2024-03-11T21:29:13.403 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 4.8778e-05 2024-03-11T21:29:13.403 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-03-11T21:29:13.403 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-03-11T21:29:13.403 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-03-11T21:29:13.403 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:29:13.403 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:29:13.403 INFO:teuthology.orchestra.run.smithi092.stdout: "description": "osd_repop(client.34715.0:116286 7.5 e119/99 7:a74abdea:::benchmark_data_smithi092_76040_object116285:head v 119'7286, mlcod=119'7283)", 2024-03-11T21:29:13.403 INFO:teuthology.orchestra.run.smithi092.stdout: "initiated_at": "2024-03-11T21:29:08.772243+0000", 2024-03-11T21:29:13.403 INFO:teuthology.orchestra.run.smithi092.stdout: "age": 4.5691427359999999, 2024-03-11T21:29:13.404 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.105122486, 2024-03-11T21:29:13.404 INFO:teuthology.orchestra.run.smithi092.stdout: "type_data": { 2024-03-11T21:29:13.404 INFO:teuthology.orchestra.run.smithi092.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-11T21:29:13.404 INFO:teuthology.orchestra.run.smithi092.stdout: "events": [ 2024-03-11T21:29:13.404 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:29:13.404 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "initiated", 2024-03-11T21:29:13.404 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:29:08.772243+0000", 2024-03-11T21:29:13.404 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-03-11T21:29:13.404 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:29:13.404 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:29:13.404 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "header_read", 2024-03-11T21:29:13.404 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:29:08.772243+0000", 2024-03-11T21:29:13.404 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-03-11T21:29:13.405 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:29:13.405 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:29:13.405 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "throttled", 2024-03-11T21:29:13.405 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:29:08.772244+0000", 2024-03-11T21:29:13.405 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 9.7600000000000006e-07 2024-03-11T21:29:13.405 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:29:13.405 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:29:13.405 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "all_read", 2024-03-11T21:29:13.405 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:29:08.772287+0000", 2024-03-11T21:29:13.405 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 4.3052e-05 2024-03-11T21:29:13.405 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:29:13.406 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:29:13.406 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "dispatched", 2024-03-11T21:29:13.406 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:29:08.772295+0000", 2024-03-11T21:29:13.406 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 7.7989999999999994e-06 2024-03-11T21:29:13.406 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:29:13.406 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:29:13.406 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "queued_for_pg", 2024-03-11T21:29:13.406 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:29:08.772301+0000", 2024-03-11T21:29:13.406 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 6.1319999999999999e-06 2024-03-11T21:29:13.406 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:29:13.406 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:29:13.406 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-03-11T21:29:13.407 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:29:08.772348+0000", 2024-03-11T21:29:13.407 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 4.6999000000000002e-05 2024-03-11T21:29:13.407 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:29:13.407 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:29:13.407 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "started", 2024-03-11T21:29:13.407 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:29:08.772377+0000", 2024-03-11T21:29:13.407 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 2.9246000000000001e-05 2024-03-11T21:29:13.407 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:29:13.407 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:29:13.407 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "commit_sent", 2024-03-11T21:29:13.407 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:29:08.775548+0000", 2024-03-11T21:29:13.407 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.0031711740000000001 2024-03-11T21:29:13.408 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:29:13.408 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:29:13.408 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "done", 2024-03-11T21:29:13.408 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:29:08.877365+0000", 2024-03-11T21:29:13.408 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.101817108 2024-03-11T21:29:13.408 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-03-11T21:29:13.408 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-03-11T21:29:13.408 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-03-11T21:29:13.408 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:29:13.408 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:29:13.408 INFO:teuthology.orchestra.run.smithi092.stdout: "description": "osd_repop_reply(client.34715.0:116287 7.8 e119/99 ondisk, result = 0)", 2024-03-11T21:29:13.408 INFO:teuthology.orchestra.run.smithi092.stdout: "initiated_at": "2024-03-11T21:29:08.776070+0000", 2024-03-11T21:29:13.408 INFO:teuthology.orchestra.run.smithi092.stdout: "age": 4.5653156839999998, 2024-03-11T21:29:13.409 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.10140663900000001, 2024-03-11T21:29:13.409 INFO:teuthology.orchestra.run.smithi092.stdout: "type_data": { 2024-03-11T21:29:13.409 INFO:teuthology.orchestra.run.smithi092.stdout: "flag_point": "started", 2024-03-11T21:29:13.409 INFO:teuthology.orchestra.run.smithi092.stdout: "events": [ 2024-03-11T21:29:13.409 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:29:13.409 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "initiated", 2024-03-11T21:29:13.409 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:29:08.776070+0000", 2024-03-11T21:29:13.409 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-03-11T21:29:13.409 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:29:13.409 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:29:13.409 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "header_read", 2024-03-11T21:29:13.409 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:29:08.776070+0000", 2024-03-11T21:29:13.410 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-03-11T21:29:13.410 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:29:13.410 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:29:13.410 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "throttled", 2024-03-11T21:29:13.410 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:29:08.776071+0000", 2024-03-11T21:29:13.410 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.265e-06 2024-03-11T21:29:13.410 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:29:13.410 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:29:13.410 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "all_read", 2024-03-11T21:29:13.410 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:29:08.776075+0000", 2024-03-11T21:29:13.410 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 4.2930000000000002e-06 2024-03-11T21:29:13.410 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:29:13.411 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:29:13.411 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "dispatched", 2024-03-11T21:29:13.411 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:29:08.877397+0000", 2024-03-11T21:29:13.411 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.10132176699999999 2024-03-11T21:29:13.411 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:29:13.411 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:29:13.411 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "queued_for_pg", 2024-03-11T21:29:13.411 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:29:08.877406+0000", 2024-03-11T21:29:13.411 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 9.1779999999999992e-06 2024-03-11T21:29:13.411 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:29:13.411 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:29:13.411 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-03-11T21:29:13.412 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:29:08.877454+0000", 2024-03-11T21:29:13.412 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 4.8087000000000002e-05 2024-03-11T21:29:13.412 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:29:13.412 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:29:13.412 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "started", 2024-03-11T21:29:13.412 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:29:08.877463+0000", 2024-03-11T21:29:13.412 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 8.5350000000000004e-06 2024-03-11T21:29:13.412 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:29:13.412 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:29:13.412 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "done", 2024-03-11T21:29:13.412 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:29:08.877477+0000", 2024-03-11T21:29:13.412 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.3514e-05 2024-03-11T21:29:13.412 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-03-11T21:29:13.413 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-03-11T21:29:13.413 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-03-11T21:29:13.413 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:29:13.413 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:29:13.413 INFO:teuthology.orchestra.run.smithi092.stdout: "description": "osd_op(client.34715.0:116331 7.8 7:19efdca4:::benchmark_data_smithi092_76040_object116330:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e119)", 2024-03-11T21:29:13.413 INFO:teuthology.orchestra.run.smithi092.stdout: "initiated_at": "2024-03-11T21:29:08.912349+0000", 2024-03-11T21:29:13.413 INFO:teuthology.orchestra.run.smithi092.stdout: "age": 4.4290365859999996, 2024-03-11T21:29:13.413 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.077934325999999998, 2024-03-11T21:29:13.413 INFO:teuthology.orchestra.run.smithi092.stdout: "type_data": { 2024-03-11T21:29:13.413 INFO:teuthology.orchestra.run.smithi092.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-11T21:29:13.413 INFO:teuthology.orchestra.run.smithi092.stdout: "client_info": { 2024-03-11T21:29:13.413 INFO:teuthology.orchestra.run.smithi092.stdout: "client": "client.34715", 2024-03-11T21:29:13.414 INFO:teuthology.orchestra.run.smithi092.stdout: "client_addr": "v1:172.21.15.92:0/2403132573", 2024-03-11T21:29:13.414 INFO:teuthology.orchestra.run.smithi092.stdout: "tid": 116331 2024-03-11T21:29:13.414 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:29:13.414 INFO:teuthology.orchestra.run.smithi092.stdout: "events": [ 2024-03-11T21:29:13.414 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:29:13.414 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "initiated", 2024-03-11T21:29:13.414 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:29:08.912349+0000", 2024-03-11T21:29:13.414 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-03-11T21:29:13.414 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:29:13.414 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:29:13.414 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "header_read", 2024-03-11T21:29:13.415 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:29:08.912349+0000", 2024-03-11T21:29:13.415 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-03-11T21:29:13.415 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:29:13.415 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:29:13.415 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "throttled", 2024-03-11T21:29:13.415 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:29:08.912351+0000", 2024-03-11T21:29:13.415 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 2.396e-06 2024-03-11T21:29:13.415 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:29:13.415 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:29:13.415 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "all_read", 2024-03-11T21:29:13.415 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:29:08.912376+0000", 2024-03-11T21:29:13.415 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 2.4978e-05 2024-03-11T21:29:13.415 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:29:13.416 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:29:13.416 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "dispatched", 2024-03-11T21:29:13.416 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:29:08.912389+0000", 2024-03-11T21:29:13.416 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.2842e-05 2024-03-11T21:29:13.416 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:29:13.416 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:29:13.416 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "queued_for_pg", 2024-03-11T21:29:13.416 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:29:08.912398+0000", 2024-03-11T21:29:13.416 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 8.3240000000000003e-06 2024-03-11T21:29:13.416 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:29:13.416 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:29:13.416 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-03-11T21:29:13.416 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:29:08.912791+0000", 2024-03-11T21:29:13.417 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00039321199999999999 2024-03-11T21:29:13.417 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:29:13.417 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:29:13.417 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "started", 2024-03-11T21:29:13.417 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:29:08.912909+0000", 2024-03-11T21:29:13.417 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.000118385 2024-03-11T21:29:13.417 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:29:13.417 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:29:13.417 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "waiting for subops from 4,7", 2024-03-11T21:29:13.417 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:29:08.913056+0000", 2024-03-11T21:29:13.417 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00014714799999999999 2024-03-11T21:29:13.417 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:29:13.417 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:29:13.418 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "sub_op_commit_rec", 2024-03-11T21:29:13.418 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:29:08.990012+0000", 2024-03-11T21:29:13.418 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.076955421999999996 2024-03-11T21:29:13.418 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:29:13.418 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:29:13.419 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "op_commit", 2024-03-11T21:29:13.419 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:29:08.990030+0000", 2024-03-11T21:29:13.419 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.7852000000000001e-05 2024-03-11T21:29:13.419 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:29:13.419 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:29:13.419 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "sub_op_commit_rec", 2024-03-11T21:29:13.419 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:29:08.990087+0000", 2024-03-11T21:29:13.419 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 5.7371999999999999e-05 2024-03-11T21:29:13.419 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:29:13.420 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:29:13.420 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "commit_sent", 2024-03-11T21:29:13.420 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:29:08.990143+0000", 2024-03-11T21:29:13.420 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 5.6450999999999999e-05 2024-03-11T21:29:13.420 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:29:13.420 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:29:13.420 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "done", 2024-03-11T21:29:13.420 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:29:08.990283+0000", 2024-03-11T21:29:13.420 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.000139944 2024-03-11T21:29:13.420 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-03-11T21:29:13.420 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-03-11T21:29:13.420 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-03-11T21:29:13.420 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:29:13.420 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:29:13.421 INFO:teuthology.orchestra.run.smithi092.stdout: "description": "osd_repop(client.34715.0:116312 7.a e119/99 7:519e54d9:::benchmark_data_smithi092_76040_object116311:head v 119'7233, mlcod=119'7231)", 2024-03-11T21:29:13.421 INFO:teuthology.orchestra.run.smithi092.stdout: "initiated_at": "2024-03-11T21:29:08.914334+0000", 2024-03-11T21:29:13.421 INFO:teuthology.orchestra.run.smithi092.stdout: "age": 4.4270512240000004, 2024-03-11T21:29:13.421 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.077253109, 2024-03-11T21:29:13.421 INFO:teuthology.orchestra.run.smithi092.stdout: "type_data": { 2024-03-11T21:29:13.421 INFO:teuthology.orchestra.run.smithi092.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-11T21:29:13.421 INFO:teuthology.orchestra.run.smithi092.stdout: "events": [ 2024-03-11T21:29:13.421 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:29:13.421 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "initiated", 2024-03-11T21:29:13.421 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:29:08.914334+0000", 2024-03-11T21:29:13.421 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-03-11T21:29:13.422 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:29:13.422 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:29:13.422 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "header_read", 2024-03-11T21:29:13.422 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:29:08.914334+0000", 2024-03-11T21:29:13.422 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-03-11T21:29:13.422 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:29:13.422 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:29:13.422 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "throttled", 2024-03-11T21:29:13.422 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:29:08.914336+0000", 2024-03-11T21:29:13.422 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.7990000000000001e-06 2024-03-11T21:29:13.422 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:29:13.423 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:29:13.423 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "all_read", 2024-03-11T21:29:13.423 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:29:08.914367+0000", 2024-03-11T21:29:13.423 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 3.0978000000000001e-05 2024-03-11T21:29:13.423 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:29:13.423 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:29:13.423 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "dispatched", 2024-03-11T21:29:13.423 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:29:08.914376+0000", 2024-03-11T21:29:13.423 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 9.2289999999999992e-06 2024-03-11T21:29:13.423 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:29:13.423 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:29:13.423 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "queued_for_pg", 2024-03-11T21:29:13.423 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:29:08.914383+0000", 2024-03-11T21:29:13.424 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 6.781e-06 2024-03-11T21:29:13.424 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:29:13.424 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:29:13.424 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-03-11T21:29:13.424 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:29:08.914500+0000", 2024-03-11T21:29:13.424 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.000116919 2024-03-11T21:29:13.424 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:29:13.424 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:29:13.424 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "started", 2024-03-11T21:29:13.424 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:29:08.914520+0000", 2024-03-11T21:29:13.424 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.9677000000000002e-05 2024-03-11T21:29:13.424 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:29:13.425 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:29:13.425 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "commit_sent", 2024-03-11T21:29:13.425 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:29:08.991541+0000", 2024-03-11T21:29:13.425 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.077021521999999995 2024-03-11T21:29:13.425 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:29:13.425 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:29:13.425 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "done", 2024-03-11T21:29:13.425 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:29:08.991587+0000", 2024-03-11T21:29:13.425 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 4.6204e-05 2024-03-11T21:29:13.425 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-03-11T21:29:13.425 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-03-11T21:29:13.426 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-03-11T21:29:13.426 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:29:13.426 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:29:13.426 INFO:teuthology.orchestra.run.smithi092.stdout: "description": "osd_repop(client.34715.0:116325 7.a e119/99 7:50f8f4bc:::benchmark_data_smithi092_76040_object116324:head v 119'7234, mlcod=119'7231)", 2024-03-11T21:29:13.426 INFO:teuthology.orchestra.run.smithi092.stdout: "initiated_at": "2024-03-11T21:29:08.914403+0000", 2024-03-11T21:29:13.426 INFO:teuthology.orchestra.run.smithi092.stdout: "age": 4.4269823170000002, 2024-03-11T21:29:13.426 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.079377257000000007, 2024-03-11T21:29:13.426 INFO:teuthology.orchestra.run.smithi092.stdout: "type_data": { 2024-03-11T21:29:13.426 INFO:teuthology.orchestra.run.smithi092.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-11T21:29:13.426 INFO:teuthology.orchestra.run.smithi092.stdout: "events": [ 2024-03-11T21:29:13.426 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:29:13.426 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "initiated", 2024-03-11T21:29:13.426 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:29:08.914403+0000", 2024-03-11T21:29:13.427 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-03-11T21:29:13.427 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:29:13.427 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:29:13.427 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "header_read", 2024-03-11T21:29:13.427 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:29:08.914403+0000", 2024-03-11T21:29:13.427 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-03-11T21:29:13.427 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:29:13.427 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:29:13.427 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "throttled", 2024-03-11T21:29:13.427 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:29:08.914404+0000", 2024-03-11T21:29:13.427 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 5.6599999999999996e-07 2024-03-11T21:29:13.427 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:29:13.427 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:29:13.428 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "all_read", 2024-03-11T21:29:13.428 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:29:08.914431+0000", 2024-03-11T21:29:13.428 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 2.7061000000000001e-05 2024-03-11T21:29:13.428 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:29:13.428 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:29:13.428 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "dispatched", 2024-03-11T21:29:13.428 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:29:08.914437+0000", 2024-03-11T21:29:13.428 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 6.0000000000000002e-06 2024-03-11T21:29:13.428 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:29:13.428 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:29:13.428 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "queued_for_pg", 2024-03-11T21:29:13.428 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:29:08.914442+0000", 2024-03-11T21:29:13.429 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 4.741e-06 2024-03-11T21:29:13.429 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:29:13.429 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:29:13.429 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-03-11T21:29:13.429 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:29:08.986768+0000", 2024-03-11T21:29:13.429 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.072325946000000002 2024-03-11T21:29:13.429 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:29:13.429 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:29:13.429 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "started", 2024-03-11T21:29:13.429 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:29:08.986797+0000", 2024-03-11T21:29:13.429 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 2.9764000000000002e-05 2024-03-11T21:29:13.429 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:29:13.429 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:29:13.429 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "commit_sent", 2024-03-11T21:29:13.430 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:29:08.993737+0000", 2024-03-11T21:29:13.430 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.0069398070000000001 2024-03-11T21:29:13.430 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:29:13.430 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:29:13.430 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "done", 2024-03-11T21:29:13.430 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:29:08.993781+0000", 2024-03-11T21:29:13.430 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 4.3371999999999997e-05 2024-03-11T21:29:13.430 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-03-11T21:29:13.430 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-03-11T21:29:13.430 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-03-11T21:29:13.430 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:29:13.430 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:29:13.431 INFO:teuthology.orchestra.run.smithi092.stdout: "description": "osd_op(client.34715.0:116333 7.8 7:12ed9624:::benchmark_data_smithi092_76040_object116332:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e119)", 2024-03-11T21:29:13.431 INFO:teuthology.orchestra.run.smithi092.stdout: "initiated_at": "2024-03-11T21:29:08.915546+0000", 2024-03-11T21:29:13.431 INFO:teuthology.orchestra.run.smithi092.stdout: "age": 4.4258400089999999, 2024-03-11T21:29:13.431 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.082294330999999998, 2024-03-11T21:29:13.431 INFO:teuthology.orchestra.run.smithi092.stdout: "type_data": { 2024-03-11T21:29:13.431 INFO:teuthology.orchestra.run.smithi092.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-11T21:29:13.431 INFO:teuthology.orchestra.run.smithi092.stdout: "client_info": { 2024-03-11T21:29:13.431 INFO:teuthology.orchestra.run.smithi092.stdout: "client": "client.34715", 2024-03-11T21:29:13.431 INFO:teuthology.orchestra.run.smithi092.stdout: "client_addr": "v1:172.21.15.92:0/2403132573", 2024-03-11T21:29:13.431 INFO:teuthology.orchestra.run.smithi092.stdout: "tid": 116333 2024-03-11T21:29:13.431 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:29:13.431 INFO:teuthology.orchestra.run.smithi092.stdout: "events": [ 2024-03-11T21:29:13.432 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:29:13.432 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "initiated", 2024-03-11T21:29:13.432 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:29:08.915546+0000", 2024-03-11T21:29:13.432 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-03-11T21:29:13.432 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:29:13.432 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:29:13.432 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "header_read", 2024-03-11T21:29:13.432 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:29:08.915546+0000", 2024-03-11T21:29:13.432 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-03-11T21:29:13.432 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:29:13.432 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:29:13.432 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "throttled", 2024-03-11T21:29:13.432 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:29:08.915548+0000", 2024-03-11T21:29:13.433 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 2.3159999999999999e-06 2024-03-11T21:29:13.433 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:29:13.433 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:29:13.433 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "all_read", 2024-03-11T21:29:13.433 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:29:08.915571+0000", 2024-03-11T21:29:13.433 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 2.2838999999999999e-05 2024-03-11T21:29:13.433 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:29:13.433 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:29:13.433 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "dispatched", 2024-03-11T21:29:13.433 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:29:08.915582+0000", 2024-03-11T21:29:13.433 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.1673e-05 2024-03-11T21:29:13.433 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:29:13.434 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:29:13.434 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "queued_for_pg", 2024-03-11T21:29:13.434 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:29:08.915590+0000", 2024-03-11T21:29:13.434 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 7.9289999999999998e-06 2024-03-11T21:29:13.434 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:29:13.434 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:29:13.434 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-03-11T21:29:13.434 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:29:08.916889+0000", 2024-03-11T21:29:13.434 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.001298537 2024-03-11T21:29:13.434 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:29:13.434 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:29:13.434 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "started", 2024-03-11T21:29:13.435 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:29:08.917043+0000", 2024-03-11T21:29:13.435 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00015451500000000001 2024-03-11T21:29:13.435 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:29:13.435 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:29:13.435 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "waiting for subops from 4,7", 2024-03-11T21:29:13.435 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:29:08.917187+0000", 2024-03-11T21:29:13.435 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00014380799999999999 2024-03-11T21:29:13.435 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:29:13.435 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:29:13.435 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "sub_op_commit_rec", 2024-03-11T21:29:13.435 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:29:08.990374+0000", 2024-03-11T21:29:13.435 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.073186651000000005 2024-03-11T21:29:13.435 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:29:13.436 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:29:13.436 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "sub_op_commit_rec", 2024-03-11T21:29:13.436 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:29:08.990416+0000", 2024-03-11T21:29:13.436 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 4.2548e-05 2024-03-11T21:29:13.436 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:29:13.436 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:29:13.436 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "op_commit", 2024-03-11T21:29:13.436 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:29:08.997697+0000", 2024-03-11T21:29:13.436 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.0072802450000000003 2024-03-11T21:29:13.436 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:29:13.436 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:29:13.436 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "commit_sent", 2024-03-11T21:29:13.437 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:29:08.997791+0000", 2024-03-11T21:29:13.437 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 9.4796000000000005e-05 2024-03-11T21:29:13.437 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:29:13.437 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:29:13.437 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "done", 2024-03-11T21:29:13.437 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:29:08.997840+0000", 2024-03-11T21:29:13.437 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 4.8473999999999997e-05 2024-03-11T21:29:13.437 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-03-11T21:29:13.437 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-03-11T21:29:13.437 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-03-11T21:29:13.437 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:29:13.437 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:29:13.438 INFO:teuthology.orchestra.run.smithi092.stdout: "description": "osd_op(client.34715.0:116335 7.8 7:1fa7fab8:::benchmark_data_smithi092_76040_object116334:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e119)", 2024-03-11T21:29:13.438 INFO:teuthology.orchestra.run.smithi092.stdout: "initiated_at": "2024-03-11T21:29:08.916101+0000", 2024-03-11T21:29:13.438 INFO:teuthology.orchestra.run.smithi092.stdout: "age": 4.4252843469999998, 2024-03-11T21:29:13.438 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.081878851000000002, 2024-03-11T21:29:13.438 INFO:teuthology.orchestra.run.smithi092.stdout: "type_data": { 2024-03-11T21:29:13.438 INFO:teuthology.orchestra.run.smithi092.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-11T21:29:13.438 INFO:teuthology.orchestra.run.smithi092.stdout: "client_info": { 2024-03-11T21:29:13.438 INFO:teuthology.orchestra.run.smithi092.stdout: "client": "client.34715", 2024-03-11T21:29:13.438 INFO:teuthology.orchestra.run.smithi092.stdout: "client_addr": "v1:172.21.15.92:0/2403132573", 2024-03-11T21:29:13.438 INFO:teuthology.orchestra.run.smithi092.stdout: "tid": 116335 2024-03-11T21:29:13.438 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:29:13.438 INFO:teuthology.orchestra.run.smithi092.stdout: "events": [ 2024-03-11T21:29:13.439 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:29:13.439 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "initiated", 2024-03-11T21:29:13.439 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:29:08.916101+0000", 2024-03-11T21:29:13.439 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-03-11T21:29:13.439 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:29:13.439 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:29:13.439 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "header_read", 2024-03-11T21:29:13.439 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:29:08.916101+0000", 2024-03-11T21:29:13.439 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-03-11T21:29:13.439 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:29:13.439 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:29:13.439 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "throttled", 2024-03-11T21:29:13.440 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:29:08.916103+0000", 2024-03-11T21:29:13.440 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.5910000000000001e-06 2024-03-11T21:29:13.440 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:29:13.440 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:29:13.440 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "all_read", 2024-03-11T21:29:13.440 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:29:08.916124+0000", 2024-03-11T21:29:13.440 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 2.1481e-05 2024-03-11T21:29:13.440 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:29:13.440 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:29:13.440 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "dispatched", 2024-03-11T21:29:13.440 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:29:08.916135+0000", 2024-03-11T21:29:13.441 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.0721e-05 2024-03-11T21:29:13.441 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:29:13.441 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:29:13.441 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "queued_for_pg", 2024-03-11T21:29:13.441 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:29:08.916142+0000", 2024-03-11T21:29:13.441 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 7.3390000000000004e-06 2024-03-11T21:29:13.441 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:29:13.441 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:29:13.441 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-03-11T21:29:13.441 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:29:08.990446+0000", 2024-03-11T21:29:13.441 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.07430349 2024-03-11T21:29:13.441 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:29:13.441 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:29:13.442 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "started", 2024-03-11T21:29:13.442 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:29:08.990549+0000", 2024-03-11T21:29:13.442 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.000103267 2024-03-11T21:29:13.442 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:29:13.442 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:29:13.442 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "waiting for subops from 4,7", 2024-03-11T21:29:13.442 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:29:08.990686+0000", 2024-03-11T21:29:13.442 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00013649800000000001 2024-03-11T21:29:13.442 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:29:13.442 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:29:13.442 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "sub_op_commit_rec", 2024-03-11T21:29:13.443 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:29:08.997639+0000", 2024-03-11T21:29:13.443 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.0069533199999999998 2024-03-11T21:29:13.443 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:29:13.443 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:29:13.443 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "sub_op_commit_rec", 2024-03-11T21:29:13.443 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:29:08.997683+0000", 2024-03-11T21:29:13.443 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 4.3591000000000003e-05 2024-03-11T21:29:13.443 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:29:13.443 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:29:13.443 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "op_commit", 2024-03-11T21:29:13.443 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:29:08.997850+0000", 2024-03-11T21:29:13.443 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00016767099999999999 2024-03-11T21:29:13.443 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:29:13.444 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:29:13.444 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "commit_sent", 2024-03-11T21:29:13.444 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:29:08.997943+0000", 2024-03-11T21:29:13.444 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 9.2633000000000006e-05 2024-03-11T21:29:13.444 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:29:13.444 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:29:13.444 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "done", 2024-03-11T21:29:13.444 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:29:08.997980+0000", 2024-03-11T21:29:13.444 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 3.7249000000000002e-05 2024-03-11T21:29:13.444 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-03-11T21:29:13.444 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-03-11T21:29:13.444 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-03-11T21:29:13.444 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:29:13.445 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:29:13.445 INFO:teuthology.orchestra.run.smithi092.stdout: "description": "osd_repop(client.34715.0:116329 7.a e119/99 7:5da92f61:::benchmark_data_smithi092_76040_object116328:head v 119'7235, mlcod=119'7231)", 2024-03-11T21:29:13.445 INFO:teuthology.orchestra.run.smithi092.stdout: "initiated_at": "2024-03-11T21:29:08.916526+0000", 2024-03-11T21:29:13.445 INFO:teuthology.orchestra.run.smithi092.stdout: "age": 4.4248600769999999, 2024-03-11T21:29:13.445 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.077278870999999999, 2024-03-11T21:29:13.445 INFO:teuthology.orchestra.run.smithi092.stdout: "type_data": { 2024-03-11T21:29:13.445 INFO:teuthology.orchestra.run.smithi092.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-11T21:29:13.445 INFO:teuthology.orchestra.run.smithi092.stdout: "events": [ 2024-03-11T21:29:13.445 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:29:13.445 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "initiated", 2024-03-11T21:29:13.445 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:29:08.916526+0000", 2024-03-11T21:29:13.445 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-03-11T21:29:13.446 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:29:13.446 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:29:13.446 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "header_read", 2024-03-11T21:29:13.446 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:29:08.916526+0000", 2024-03-11T21:29:13.446 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-03-11T21:29:13.446 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:29:13.446 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:29:13.446 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "throttled", 2024-03-11T21:29:13.446 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:29:08.916527+0000", 2024-03-11T21:29:13.446 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.181e-06 2024-03-11T21:29:13.446 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:29:13.446 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:29:13.446 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "all_read", 2024-03-11T21:29:13.447 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:29:08.916550+0000", 2024-03-11T21:29:13.447 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 2.2979000000000001e-05 2024-03-11T21:29:13.447 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:29:13.447 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:29:13.447 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "dispatched", 2024-03-11T21:29:13.447 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:29:08.916559+0000", 2024-03-11T21:29:13.447 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 9.1079999999999999e-06 2024-03-11T21:29:13.447 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:29:13.447 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:29:13.447 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "queued_for_pg", 2024-03-11T21:29:13.447 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:29:08.916565+0000", 2024-03-11T21:29:13.447 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 6.6470000000000001e-06 2024-03-11T21:29:13.447 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:29:13.448 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:29:13.448 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-03-11T21:29:13.448 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:29:08.989163+0000", 2024-03-11T21:29:13.448 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.072597996999999997 2024-03-11T21:29:13.448 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:29:13.448 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:29:13.448 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "started", 2024-03-11T21:29:13.448 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:29:08.989192+0000", 2024-03-11T21:29:13.448 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 2.8917000000000001e-05 2024-03-11T21:29:13.448 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:29:13.448 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:29:13.448 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "commit_sent", 2024-03-11T21:29:13.448 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:29:08.993783+0000", 2024-03-11T21:29:13.448 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.0045908340000000002 2024-03-11T21:29:13.449 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:29:13.449 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:29:13.449 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "done", 2024-03-11T21:29:13.449 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:29:08.993804+0000", 2024-03-11T21:29:13.449 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 2.1208000000000001e-05 2024-03-11T21:29:13.449 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-03-11T21:29:13.449 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-03-11T21:29:13.449 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-03-11T21:29:13.449 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:29:13.449 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:29:13.449 INFO:teuthology.orchestra.run.smithi092.stdout: "description": "osd_op(client.34715.0:116339 7.e 7:7e6296f0:::benchmark_data_smithi092_76040_object116338:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e119)", 2024-03-11T21:29:13.449 INFO:teuthology.orchestra.run.smithi092.stdout: "initiated_at": "2024-03-11T21:29:08.918203+0000", 2024-03-11T21:29:13.449 INFO:teuthology.orchestra.run.smithi092.stdout: "age": 4.4231822440000004, 2024-03-11T21:29:13.450 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.084915199999999996, 2024-03-11T21:29:13.450 INFO:teuthology.orchestra.run.smithi092.stdout: "type_data": { 2024-03-11T21:29:13.450 INFO:teuthology.orchestra.run.smithi092.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-11T21:29:13.450 INFO:teuthology.orchestra.run.smithi092.stdout: "client_info": { 2024-03-11T21:29:13.450 INFO:teuthology.orchestra.run.smithi092.stdout: "client": "client.34715", 2024-03-11T21:29:13.450 INFO:teuthology.orchestra.run.smithi092.stdout: "client_addr": "v1:172.21.15.92:0/2403132573", 2024-03-11T21:29:13.450 INFO:teuthology.orchestra.run.smithi092.stdout: "tid": 116339 2024-03-11T21:29:13.450 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:29:13.450 INFO:teuthology.orchestra.run.smithi092.stdout: "events": [ 2024-03-11T21:29:13.450 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:29:13.450 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "initiated", 2024-03-11T21:29:13.450 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:29:08.918203+0000", 2024-03-11T21:29:13.451 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-03-11T21:29:13.451 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:29:13.451 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:29:13.451 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "header_read", 2024-03-11T21:29:13.451 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:29:08.918203+0000", 2024-03-11T21:29:13.451 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-03-11T21:29:13.451 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:29:13.451 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:29:13.451 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "throttled", 2024-03-11T21:29:13.451 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:29:08.918205+0000", 2024-03-11T21:29:13.451 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.933e-06 2024-03-11T21:29:13.451 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:29:13.451 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:29:13.452 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "all_read", 2024-03-11T21:29:13.452 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:29:08.918226+0000", 2024-03-11T21:29:13.452 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 2.1098999999999999e-05 2024-03-11T21:29:13.452 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:29:13.452 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:29:13.452 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "dispatched", 2024-03-11T21:29:13.452 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:29:08.986734+0000", 2024-03-11T21:29:13.452 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.068507799999999994 2024-03-11T21:29:13.452 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:29:13.452 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:29:13.452 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "queued_for_pg", 2024-03-11T21:29:13.452 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:29:08.986752+0000", 2024-03-11T21:29:13.452 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.8315999999999999e-05 2024-03-11T21:29:13.452 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:29:13.453 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:29:13.453 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-03-11T21:29:13.453 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:29:08.986828+0000", 2024-03-11T21:29:13.453 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 7.5916999999999998e-05 2024-03-11T21:29:13.453 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:29:13.453 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:29:13.453 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "started", 2024-03-11T21:29:13.453 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:29:08.986957+0000", 2024-03-11T21:29:13.453 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00012831600000000001 2024-03-11T21:29:13.453 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:29:13.453 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:29:13.453 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "waiting for subops from 0,5", 2024-03-11T21:29:13.453 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:29:08.987118+0000", 2024-03-11T21:29:13.454 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00016082999999999999 2024-03-11T21:29:13.454 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:29:13.454 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:29:13.454 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "sub_op_commit_rec", 2024-03-11T21:29:13.454 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:29:08.993823+0000", 2024-03-11T21:29:13.454 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.0067059010000000002 2024-03-11T21:29:13.454 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:29:13.454 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:29:13.454 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "op_commit", 2024-03-11T21:29:13.454 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:29:08.993877+0000", 2024-03-11T21:29:13.454 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 5.3913999999999999e-05 2024-03-11T21:29:13.454 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:29:13.454 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:29:13.455 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "sub_op_commit_rec", 2024-03-11T21:29:13.455 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:29:09.003018+0000", 2024-03-11T21:29:13.455 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.0091407610000000007 2024-03-11T21:29:13.455 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:29:13.455 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:29:13.455 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "commit_sent", 2024-03-11T21:29:13.455 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:29:09.003074+0000", 2024-03-11T21:29:13.455 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 5.6165000000000002e-05 2024-03-11T21:29:13.455 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:29:13.455 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:29:13.455 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "done", 2024-03-11T21:29:13.455 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:29:09.003119+0000", 2024-03-11T21:29:13.456 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 4.4248000000000001e-05 2024-03-11T21:29:13.456 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-03-11T21:29:13.456 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-03-11T21:29:13.456 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-03-11T21:29:13.456 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-03-11T21:29:13.456 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-03-11T21:29:13.456 INFO:teuthology.orchestra.run.smithi092.stdout:} 2024-03-11T21:29:13.473 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:29:13.500 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:29:13.501 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:29:13.502 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:29:13.602 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:29:13.610 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:29:13 smithi092 ceph-mon[29572]: from='client.24469 v1:172.21.15.97:0/404208310' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:29:13.610 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:29:13 smithi092 ceph-mon[29572]: 6.9 scrub starts 2024-03-11T21:29:13.611 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:29:13 smithi092 ceph-mon[29572]: 6.9 scrub ok 2024-03-11T21:29:13.611 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:29:13 smithi092 ceph-mon[29572]: 6.1d scrub starts 2024-03-11T21:29:13.611 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:29:13 smithi092 ceph-mon[29572]: 6.1d scrub ok 2024-03-11T21:29:13.611 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:29:13 smithi092 ceph-mon[29572]: 1.0 scrub starts 2024-03-11T21:29:13.611 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:29:13 smithi092 ceph-mon[29572]: 1.0 scrub ok 2024-03-11T21:29:13.611 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:29:13 smithi092 ceph-mon[37563]: from='client.24469 v1:172.21.15.97:0/404208310' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:29:13.611 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:29:13 smithi092 ceph-mon[37563]: 6.9 scrub starts 2024-03-11T21:29:13.611 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:29:13 smithi092 ceph-mon[37563]: 6.9 scrub ok 2024-03-11T21:29:13.611 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:29:13 smithi092 ceph-mon[37563]: 6.1d scrub starts 2024-03-11T21:29:13.612 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:29:13 smithi092 ceph-mon[37563]: 6.1d scrub ok 2024-03-11T21:29:13.612 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:29:13 smithi092 ceph-mon[37563]: 1.0 scrub starts 2024-03-11T21:29:13.612 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:29:13 smithi092 ceph-mon[37563]: 1.0 scrub ok 2024-03-11T21:29:13.633 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:29:13.633 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:29:13.634 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:29:13.647 INFO:tasks.radosbench.radosbench.0.smithi092.stdout: 89 16 121037 121021 84.9595 60.25 0.0213358 0.0117655 2024-03-11T21:29:13.734 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:29:13.778 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:29:13.778 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:29:13.778 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:29:13.826 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:29:13 smithi097 ceph-mon[33317]: from='client.24469 v1:172.21.15.97:0/404208310' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:29:13.826 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:29:13 smithi097 ceph-mon[33317]: 6.9 scrub starts 2024-03-11T21:29:13.827 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:29:13 smithi097 ceph-mon[33317]: 6.9 scrub ok 2024-03-11T21:29:13.827 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:29:13 smithi097 ceph-mon[33317]: 6.1d scrub starts 2024-03-11T21:29:13.827 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:29:13 smithi097 ceph-mon[33317]: 6.1d scrub ok 2024-03-11T21:29:13.827 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:29:13 smithi097 ceph-mon[33317]: 1.0 scrub starts 2024-03-11T21:29:13.827 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:29:13 smithi097 ceph-mon[33317]: 1.0 scrub ok 2024-03-11T21:29:13.879 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:29:13.912 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:29:13.913 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:29:13.913 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:29:14.014 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:29:14.054 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:29:14.055 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:29:14.055 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:29:14.156 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:29:14.194 INFO:teuthology.orchestra.run.smithi092.stdout:osd.0: {} 2024-03-11T21:29:14.201 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:29:14.202 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:29:14.203 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:29:14.240 INFO:teuthology.orchestra.run.smithi092.stdout:osd.1: {} 2024-03-11T21:29:14.278 INFO:teuthology.orchestra.run.smithi092.stdout:osd.2: {} 2024-03-11T21:29:14.304 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:29:14.323 INFO:teuthology.orchestra.run.smithi092.stdout:osd.3: {} 2024-03-11T21:29:14.345 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:29:14.346 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:29:14.346 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:29:14.367 INFO:teuthology.orchestra.run.smithi092.stdout:osd.4: {} 2024-03-11T21:29:14.405 INFO:teuthology.orchestra.run.smithi092.stdout:osd.5: {} 2024-03-11T21:29:14.440 INFO:teuthology.orchestra.run.smithi092.stdout:osd.6: {} 2024-03-11T21:29:14.448 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:29:14.478 INFO:teuthology.orchestra.run.smithi092.stdout:osd.7: {} 2024-03-11T21:29:14.489 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:29:14.489 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:29:14.490 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:29:14.490 INFO:teuthology.orchestra.run.smithi092.stderr:osd.0: osd_enable_op_tracker = '' 2024-03-11T21:29:14.490 INFO:teuthology.orchestra.run.smithi092.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-11T21:29:14.490 INFO:teuthology.orchestra.run.smithi092.stderr:osd.2: osd_enable_op_tracker = '' 2024-03-11T21:29:14.490 INFO:teuthology.orchestra.run.smithi092.stderr:osd.3: osd_enable_op_tracker = '' 2024-03-11T21:29:14.490 INFO:teuthology.orchestra.run.smithi092.stderr:osd.4: osd_enable_op_tracker = '' 2024-03-11T21:29:14.490 INFO:teuthology.orchestra.run.smithi092.stderr:osd.5: osd_enable_op_tracker = '' 2024-03-11T21:29:14.491 INFO:teuthology.orchestra.run.smithi092.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-11T21:29:14.491 INFO:teuthology.orchestra.run.smithi092.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-11T21:29:14.590 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:29:14.622 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:29:14.623 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:29:14.623 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:29:14.646 INFO:tasks.radosbench.radosbench.0.smithi092.stdout: 90 13 121908 121895 84.6225 54.625 0.0157516 0.0118079 2024-03-11T21:29:14.724 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:29:14.748 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:29:14.749 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:29:14.749 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:29:14.827 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:29:14 smithi097 ceph-mon[33317]: pgmap v122: 148 pgs: 1 active+clean+scrubbing, 147 active+clean; 7.3 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 1023 B/s rd, 62 MiB/s wr, 995 op/s 2024-03-11T21:29:14.827 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:29:14 smithi097 ceph-mon[33317]: 7.b scrub starts 2024-03-11T21:29:14.827 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:29:14 smithi097 ceph-mon[33317]: 6.5 scrub starts 2024-03-11T21:29:14.827 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:29:14 smithi097 ceph-mon[33317]: 6.5 scrub ok 2024-03-11T21:29:14.827 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:29:14 smithi097 ceph-mon[33317]: 5.13 scrub starts 2024-03-11T21:29:14.827 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:29:14 smithi097 ceph-mon[33317]: 5.13 scrub ok 2024-03-11T21:29:14.827 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:29:14 smithi097 ceph-mon[33317]: 5.1f scrub starts 2024-03-11T21:29:14.827 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:29:14 smithi097 ceph-mon[33317]: 5.1f scrub ok 2024-03-11T21:29:14.827 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:29:14 smithi097 ceph-mon[33317]: from='client.? v1:172.21.15.92:0/4043846878' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-11T21:29:14.850 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:29:14.860 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:29:14 smithi092 ceph-mon[37563]: pgmap v122: 148 pgs: 1 active+clean+scrubbing, 147 active+clean; 7.3 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 1023 B/s rd, 62 MiB/s wr, 995 op/s 2024-03-11T21:29:14.861 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:29:14 smithi092 ceph-mon[37563]: 7.b scrub starts 2024-03-11T21:29:14.861 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:29:14 smithi092 ceph-mon[37563]: 6.5 scrub starts 2024-03-11T21:29:14.861 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:29:14 smithi092 ceph-mon[37563]: 6.5 scrub ok 2024-03-11T21:29:14.861 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:29:14 smithi092 ceph-mon[37563]: 5.13 scrub starts 2024-03-11T21:29:14.861 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:29:14 smithi092 ceph-mon[37563]: 5.13 scrub ok 2024-03-11T21:29:14.861 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:29:14 smithi092 ceph-mon[37563]: 5.1f scrub starts 2024-03-11T21:29:14.861 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:29:14 smithi092 ceph-mon[37563]: 5.1f scrub ok 2024-03-11T21:29:14.861 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:29:14 smithi092 ceph-mon[37563]: from='client.? v1:172.21.15.92:0/4043846878' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-11T21:29:14.861 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:29:14 smithi092 ceph-mon[29572]: pgmap v122: 148 pgs: 1 active+clean+scrubbing, 147 active+clean; 7.3 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 1023 B/s rd, 62 MiB/s wr, 995 op/s 2024-03-11T21:29:14.861 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:29:14 smithi092 ceph-mon[29572]: 7.b scrub starts 2024-03-11T21:29:14.862 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:29:14 smithi092 ceph-mon[29572]: 6.5 scrub starts 2024-03-11T21:29:14.862 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:29:14 smithi092 ceph-mon[29572]: 6.5 scrub ok 2024-03-11T21:29:14.862 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:29:14 smithi092 ceph-mon[29572]: 5.13 scrub starts 2024-03-11T21:29:14.862 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:29:14 smithi092 ceph-mon[29572]: 5.13 scrub ok 2024-03-11T21:29:14.862 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:29:14 smithi092 ceph-mon[29572]: 5.1f scrub starts 2024-03-11T21:29:14.862 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:29:14 smithi092 ceph-mon[29572]: 5.1f scrub ok 2024-03-11T21:29:14.862 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:29:14 smithi092 ceph-mon[29572]: from='client.? v1:172.21.15.92:0/4043846878' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-11T21:29:14.874 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:29:14.875 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:29:14.875 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:29:14.929 DEBUG:teuthology.orchestra.run.smithi092:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid 00db0016-dfed-11ee-95c8-87774f69a715 -- ceph daemon osd.3 dump_ops_in_flight 2024-03-11T21:29:14.975 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:29:15.040 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:29:15.040 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:29:15.040 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:29:15.141 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:29:15.166 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:29:15.167 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:29:15.167 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:29:15.268 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:29:15.292 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:29:15.292 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:29:15.292 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:29:15.361 INFO:journalctl@ceph.alertmanager.a.smithi092.stdout:Mar 11 21:29:15 smithi092 podman[84650]: 2024-03-11 21:29:15.046766549 +0000 UTC m=+9.706711041 container init bddae98c3565f589db50ce3c14882c9a14da74bb75ef29d00e1b4c80145f9275 (image=quay.io/prometheus/alertmanager:v0.25.0, name=ceph-00db0016-dfed-11ee-95c8-87774f69a715-alertmanager-a, maintainer=The Prometheus Authors ) 2024-03-11T21:29:15.361 INFO:journalctl@ceph.alertmanager.a.smithi092.stdout:Mar 11 21:29:15 smithi092 podman[84650]: 2024-03-11 21:29:15.053365188 +0000 UTC m=+9.713309662 container start bddae98c3565f589db50ce3c14882c9a14da74bb75ef29d00e1b4c80145f9275 (image=quay.io/prometheus/alertmanager:v0.25.0, name=ceph-00db0016-dfed-11ee-95c8-87774f69a715-alertmanager-a, maintainer=The Prometheus Authors ) 2024-03-11T21:29:15.361 INFO:journalctl@ceph.alertmanager.a.smithi092.stdout:Mar 11 21:29:15 smithi092 ceph-00db0016-dfed-11ee-95c8-87774f69a715-alertmanager-a[85324]: ts=2024-03-11T21:29:15.069Z caller=main.go:240 level=info msg="Starting Alertmanager" version="(version=0.25.0, branch=HEAD, revision=258fab7cdd551f2cf251ed0348f0ad7289aee789)" 2024-03-11T21:29:15.361 INFO:journalctl@ceph.alertmanager.a.smithi092.stdout:Mar 11 21:29:15 smithi092 ceph-00db0016-dfed-11ee-95c8-87774f69a715-alertmanager-a[85324]: ts=2024-03-11T21:29:15.069Z caller=main.go:241 level=info build_context="(go=go1.19.4, user=root@abe866dd5717, date=20221222-14:51:36)" 2024-03-11T21:29:15.361 INFO:journalctl@ceph.alertmanager.a.smithi092.stdout:Mar 11 21:29:15 smithi092 ceph-00db0016-dfed-11ee-95c8-87774f69a715-alertmanager-a[85324]: ts=2024-03-11T21:29:15.070Z caller=cluster.go:185 level=info component=cluster msg="setting advertise address explicitly" addr=172.21.15.92 port=9094 2024-03-11T21:29:15.361 INFO:journalctl@ceph.alertmanager.a.smithi092.stdout:Mar 11 21:29:15 smithi092 ceph-00db0016-dfed-11ee-95c8-87774f69a715-alertmanager-a[85324]: ts=2024-03-11T21:29:15.071Z caller=cluster.go:681 level=info component=cluster msg="Waiting for gossip to settle..." interval=2s 2024-03-11T21:29:15.362 INFO:journalctl@ceph.alertmanager.a.smithi092.stdout:Mar 11 21:29:15 smithi092 ceph-00db0016-dfed-11ee-95c8-87774f69a715-alertmanager-a[85324]: ts=2024-03-11T21:29:15.094Z caller=coordinator.go:113 level=info component=configuration msg="Loading configuration file" file=/etc/alertmanager/alertmanager.yml 2024-03-11T21:29:15.362 INFO:journalctl@ceph.alertmanager.a.smithi092.stdout:Mar 11 21:29:15 smithi092 ceph-00db0016-dfed-11ee-95c8-87774f69a715-alertmanager-a[85324]: ts=2024-03-11T21:29:15.095Z caller=coordinator.go:126 level=info component=configuration msg="Completed loading of configuration file" file=/etc/alertmanager/alertmanager.yml 2024-03-11T21:29:15.362 INFO:journalctl@ceph.alertmanager.a.smithi092.stdout:Mar 11 21:29:15 smithi092 ceph-00db0016-dfed-11ee-95c8-87774f69a715-alertmanager-a[85324]: ts=2024-03-11T21:29:15.099Z caller=tls_config.go:232 level=info msg="Listening on" address=[::]:9093 2024-03-11T21:29:15.362 INFO:journalctl@ceph.alertmanager.a.smithi092.stdout:Mar 11 21:29:15 smithi092 ceph-00db0016-dfed-11ee-95c8-87774f69a715-alertmanager-a[85324]: ts=2024-03-11T21:29:15.099Z caller=tls_config.go:235 level=info msg="TLS is disabled." http2=false address=[::]:9093 2024-03-11T21:29:15.394 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:29:15.417 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:29:15.418 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:29:15.418 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:29:15.519 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:29:15.544 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:29:15.545 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:29:15.545 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:29:15.646 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:29:15.647 INFO:tasks.radosbench.radosbench.0.smithi092.stdout:Total time run: 90.0678 2024-03-11T21:29:15.647 INFO:tasks.radosbench.radosbench.0.smithi092.stdout:Total writes made: 121908 2024-03-11T21:29:15.647 INFO:tasks.radosbench.radosbench.0.smithi092.stdout:Write size: 65536 2024-03-11T21:29:15.647 INFO:tasks.radosbench.radosbench.0.smithi092.stdout:Object size: 65536 2024-03-11T21:29:15.647 INFO:tasks.radosbench.radosbench.0.smithi092.stdout:Bandwidth (MB/sec): 84.5946 2024-03-11T21:29:15.647 INFO:tasks.radosbench.radosbench.0.smithi092.stdout:Stddev Bandwidth: 33.4954 2024-03-11T21:29:15.647 INFO:tasks.radosbench.radosbench.0.smithi092.stdout:Max bandwidth (MB/sec): 205.125 2024-03-11T21:29:15.647 INFO:tasks.radosbench.radosbench.0.smithi092.stdout:Min bandwidth (MB/sec): 52.3125 2024-03-11T21:29:15.648 INFO:tasks.radosbench.radosbench.0.smithi092.stdout:Average IOPS: 1353 2024-03-11T21:29:15.648 INFO:tasks.radosbench.radosbench.0.smithi092.stdout:Stddev IOPS: 535.926 2024-03-11T21:29:15.648 INFO:tasks.radosbench.radosbench.0.smithi092.stdout:Max IOPS: 3282 2024-03-11T21:29:15.648 INFO:tasks.radosbench.radosbench.0.smithi092.stdout:Min IOPS: 837 2024-03-11T21:29:15.648 INFO:tasks.radosbench.radosbench.0.smithi092.stdout:Average Latency(s): 0.0118151 2024-03-11T21:29:15.648 INFO:tasks.radosbench.radosbench.0.smithi092.stdout:Stddev Latency(s): 0.00909382 2024-03-11T21:29:15.648 INFO:tasks.radosbench.radosbench.0.smithi092.stdout:Max latency(s): 0.17729 2024-03-11T21:29:15.648 INFO:tasks.radosbench.radosbench.0.smithi092.stdout:Min latency(s): 0.00131184 2024-03-11T21:29:15.652 INFO:tasks.radosbench.radosbench.0.smithi092.stdout:Cleaning up (deleting benchmark objects) 2024-03-11T21:29:15.681 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:29:15.681 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:29:15.682 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:29:15.783 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:29:15.811 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:29:15.812 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:29:15.812 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:29:15.826 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:29:15 smithi097 ceph-mon[33317]: 5.a scrub starts 2024-03-11T21:29:15.827 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:29:15 smithi097 ceph-mon[33317]: 5.a scrub ok 2024-03-11T21:29:15.827 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:29:15 smithi097 ceph-mon[33317]: 5.3 scrub starts 2024-03-11T21:29:15.827 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:29:15 smithi097 ceph-mon[33317]: 5.3 scrub ok 2024-03-11T21:29:15.827 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:29:15 smithi097 ceph-mon[33317]: 7.7 scrub ok 2024-03-11T21:29:15.827 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:29:15 smithi097 ceph-mon[33317]: 7.b scrub ok 2024-03-11T21:29:15.861 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:29:15 smithi092 ceph-mon[29572]: 5.a scrub starts 2024-03-11T21:29:15.861 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:29:15 smithi092 ceph-mon[29572]: 5.a scrub ok 2024-03-11T21:29:15.861 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:29:15 smithi092 ceph-mon[29572]: 5.3 scrub starts 2024-03-11T21:29:15.861 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:29:15 smithi092 ceph-mon[29572]: 5.3 scrub ok 2024-03-11T21:29:15.861 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:29:15 smithi092 ceph-mon[29572]: 7.7 scrub ok 2024-03-11T21:29:15.861 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:29:15 smithi092 ceph-mon[29572]: 7.b scrub ok 2024-03-11T21:29:15.861 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:29:15 smithi092 ceph-mon[37563]: 5.a scrub starts 2024-03-11T21:29:15.861 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:29:15 smithi092 ceph-mon[37563]: 5.a scrub ok 2024-03-11T21:29:15.861 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:29:15 smithi092 ceph-mon[37563]: 5.3 scrub starts 2024-03-11T21:29:15.861 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:29:15 smithi092 ceph-mon[37563]: 5.3 scrub ok 2024-03-11T21:29:15.862 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:29:15 smithi092 ceph-mon[37563]: 7.7 scrub ok 2024-03-11T21:29:15.862 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:29:15 smithi092 ceph-mon[37563]: 7.b scrub ok 2024-03-11T21:29:15.913 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:29:15.941 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:29:15.942 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:29:15.942 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:29:16.043 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:29:16.073 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:29:16.073 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:29:16.074 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:29:16.175 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:29:16.218 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:29:16.222 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:29:16.222 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:29:16.245 INFO:journalctl@ceph.alertmanager.a.smithi092.stdout:Mar 11 21:29:15 smithi092 bash[84650]: bddae98c3565f589db50ce3c14882c9a14da74bb75ef29d00e1b4c80145f9275 2024-03-11T21:29:16.245 INFO:journalctl@ceph.alertmanager.a.smithi092.stdout:Mar 11 21:29:15 smithi092 systemd[1]: Started Ceph alertmanager.a for 00db0016-dfed-11ee-95c8-87774f69a715. 2024-03-11T21:29:16.323 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:29:16.362 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:29:16.362 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:29:16.362 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:29:16.463 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:29:16.496 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:29:16.497 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:29:16.497 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:29:16.598 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:29:16.636 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:29:16.637 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:29:16.637 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:29:16.738 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:29:16.775 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:29:16.776 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:29:16.777 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:29:16.827 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:29:16 smithi097 ceph-mon[33317]: pgmap v123: 148 pgs: 1 active+clean+scrubbing, 147 active+clean; 7.3 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 1023 B/s rd, 62 MiB/s wr, 995 op/s 2024-03-11T21:29:16.827 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:29:16 smithi097 ceph-mon[33317]: 4.2 scrub starts 2024-03-11T21:29:16.827 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:29:16 smithi097 ceph-mon[33317]: 4.2 scrub ok 2024-03-11T21:29:16.827 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:29:16 smithi097 ceph-mon[33317]: 5.1b scrub starts 2024-03-11T21:29:16.827 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:29:16 smithi097 ceph-mon[33317]: 5.1b scrub ok 2024-03-11T21:29:16.827 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:29:16 smithi097 ceph-mon[33317]: from='mgr.14607 ' entity='mgr.y' 2024-03-11T21:29:16.827 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:29:16 smithi097 ceph-mon[33317]: from='mgr.14607 ' entity='mgr.y' 2024-03-11T21:29:16.861 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:29:16 smithi092 ceph-mon[29572]: pgmap v123: 148 pgs: 1 active+clean+scrubbing, 147 active+clean; 7.3 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 1023 B/s rd, 62 MiB/s wr, 995 op/s 2024-03-11T21:29:16.861 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:29:16 smithi092 ceph-mon[29572]: 4.2 scrub starts 2024-03-11T21:29:16.861 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:29:16 smithi092 ceph-mon[29572]: 4.2 scrub ok 2024-03-11T21:29:16.861 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:29:16 smithi092 ceph-mon[29572]: 5.1b scrub starts 2024-03-11T21:29:16.861 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:29:16 smithi092 ceph-mon[29572]: 5.1b scrub ok 2024-03-11T21:29:16.861 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:29:16 smithi092 ceph-mon[29572]: from='mgr.14607 ' entity='mgr.y' 2024-03-11T21:29:16.861 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:29:16 smithi092 ceph-mon[29572]: from='mgr.14607 ' entity='mgr.y' 2024-03-11T21:29:16.861 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:29:16 smithi092 ceph-mon[37563]: pgmap v123: 148 pgs: 1 active+clean+scrubbing, 147 active+clean; 7.3 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 1023 B/s rd, 62 MiB/s wr, 995 op/s 2024-03-11T21:29:16.861 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:29:16 smithi092 ceph-mon[37563]: 4.2 scrub starts 2024-03-11T21:29:16.862 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:29:16 smithi092 ceph-mon[37563]: 4.2 scrub ok 2024-03-11T21:29:16.862 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:29:16 smithi092 ceph-mon[37563]: 5.1b scrub starts 2024-03-11T21:29:16.862 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:29:16 smithi092 ceph-mon[37563]: 5.1b scrub ok 2024-03-11T21:29:16.862 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:29:16 smithi092 ceph-mon[37563]: from='mgr.14607 ' entity='mgr.y' 2024-03-11T21:29:16.862 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:29:16 smithi092 ceph-mon[37563]: from='mgr.14607 ' entity='mgr.y' 2024-03-11T21:29:16.877 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:29:16.918 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:29:16.919 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:29:16.919 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:29:17.020 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:29:17.080 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:29:17.080 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:29:17.081 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:29:17.088 INFO:teuthology.orchestra.run.smithi092.stderr:Inferring config /var/lib/ceph/00db0016-dfed-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T21:29:17.160 INFO:teuthology.orchestra.run.smithi092.stderr:Inferring config /var/lib/ceph/00db0016-dfed-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T21:29:17.182 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:29:17.210 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:29:17.211 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:29:17.211 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:29:17.259 INFO:journalctl@ceph.alertmanager.a.smithi092.stdout:Mar 11 21:29:17 smithi092 ceph-00db0016-dfed-11ee-95c8-87774f69a715-alertmanager-a[85324]: ts=2024-03-11T21:29:17.071Z caller=cluster.go:706 level=info component=cluster msg="gossip not settled" polls=0 before=0 now=1 elapsed=2.000115985s 2024-03-11T21:29:17.312 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:29:17.361 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:29:17.361 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:29:17.362 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:29:17.463 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:29:17.500 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:29:17.500 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:29:17.501 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:29:17.601 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:29:17.630 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:29:17.631 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:29:17.631 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:29:17.732 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:29:17.759 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:29:17.760 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:29:17.760 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:29:17.827 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:29:17 smithi097 ceph-mon[33317]: Reconfiguring prometheus.a (dependencies changed)... 2024-03-11T21:29:17.827 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:29:17 smithi097 ceph-mon[33317]: 6.14 scrub starts 2024-03-11T21:29:17.827 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:29:17 smithi097 ceph-mon[33317]: 6.14 scrub ok 2024-03-11T21:29:17.827 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:29:17 smithi097 ceph-mon[33317]: Reconfiguring daemon prometheus.a on smithi097 2024-03-11T21:29:17.827 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:29:17 smithi097 ceph-mon[33317]: 6.1 scrub starts 2024-03-11T21:29:17.827 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:29:17 smithi097 ceph-mon[33317]: 2.13 scrub starts 2024-03-11T21:29:17.827 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:29:17 smithi097 ceph-mon[33317]: 2.13 scrub ok 2024-03-11T21:29:17.827 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:29:17 smithi097 ceph-mon[33317]: 6.1 scrub ok 2024-03-11T21:29:17.848 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:29:17 smithi092 ceph-mon[37563]: Reconfiguring prometheus.a (dependencies changed)... 2024-03-11T21:29:17.848 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:29:17 smithi092 ceph-mon[37563]: 6.14 scrub starts 2024-03-11T21:29:17.848 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:29:17 smithi092 ceph-mon[37563]: 6.14 scrub ok 2024-03-11T21:29:17.848 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:29:17 smithi092 ceph-mon[37563]: Reconfiguring daemon prometheus.a on smithi097 2024-03-11T21:29:17.848 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:29:17 smithi092 ceph-mon[37563]: 6.1 scrub starts 2024-03-11T21:29:17.848 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:29:17 smithi092 ceph-mon[37563]: 2.13 scrub starts 2024-03-11T21:29:17.848 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:29:17 smithi092 ceph-mon[37563]: 2.13 scrub ok 2024-03-11T21:29:17.848 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:29:17 smithi092 ceph-mon[37563]: 6.1 scrub ok 2024-03-11T21:29:17.848 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:29:17 smithi092 ceph-mon[29572]: Reconfiguring prometheus.a (dependencies changed)... 2024-03-11T21:29:17.849 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:29:17 smithi092 ceph-mon[29572]: 6.14 scrub starts 2024-03-11T21:29:17.849 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:29:17 smithi092 ceph-mon[29572]: 6.14 scrub ok 2024-03-11T21:29:17.849 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:29:17 smithi092 ceph-mon[29572]: Reconfiguring daemon prometheus.a on smithi097 2024-03-11T21:29:17.849 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:29:17 smithi092 ceph-mon[29572]: 6.1 scrub starts 2024-03-11T21:29:17.849 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:29:17 smithi092 ceph-mon[29572]: 2.13 scrub starts 2024-03-11T21:29:17.849 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:29:17 smithi092 ceph-mon[29572]: 2.13 scrub ok 2024-03-11T21:29:17.849 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:29:17 smithi092 ceph-mon[29572]: 6.1 scrub ok 2024-03-11T21:29:17.861 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:29:17.894 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:29:17.895 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:29:17.895 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:29:17.996 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:29:18.037 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:29:18.037 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:29:18.037 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:29:18.110 INFO:journalctl@ceph.mgr.y.smithi092.stdout:Mar 11 21:29:17 smithi092 ceph-00db0016-dfed-11ee-95c8-87774f69a715-mgr-y[29801]: ::ffff:172.21.15.97 - - [11/Mar/2024:21:29:17] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-11T21:29:18.138 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:29:18.176 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:29:18.177 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:29:18.178 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:29:18.278 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:29:18.311 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:29:18.312 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:29:18.312 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:29:18.413 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:29:18.445 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:29:18.445 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:29:18.446 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:29:18.547 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:29:18.575 DEBUG:teuthology.orchestra.run.smithi092:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid 00db0016-dfed-11ee-95c8-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-03-11T21:29:18.589 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:29:18.589 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:29:18.590 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:29:18.691 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:29:18.736 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:29:18.737 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:29:18.737 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:29:18.783 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:29:18 smithi097 ceph-mon[33317]: pgmap v124: 148 pgs: 148 active+clean; 7.3 GiB data, 23 GiB used, 693 GiB / 715 GiB avail; 1.1 KiB/s rd, 56 MiB/s wr, 949 op/s 2024-03-11T21:29:18.783 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:29:18 smithi097 ceph-mon[33317]: 6.1c scrub starts 2024-03-11T21:29:18.783 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:29:18 smithi097 ceph-mon[33317]: 6.1c scrub ok 2024-03-11T21:29:18.783 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:29:18 smithi097 ceph-mon[33317]: 4.1d scrub starts 2024-03-11T21:29:18.783 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:29:18 smithi097 ceph-mon[33317]: 4.1d scrub ok 2024-03-11T21:29:18.783 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:29:18 smithi097 ceph-mon[33317]: 7.c scrub starts 2024-03-11T21:29:18.783 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:29:18 smithi097 ceph-mon[33317]: 2.f scrub starts 2024-03-11T21:29:18.784 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:29:18 smithi097 ceph-mon[33317]: 2.f scrub ok 2024-03-11T21:29:18.838 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:29:18.861 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:29:18 smithi092 ceph-mon[37563]: pgmap v124: 148 pgs: 148 active+clean; 7.3 GiB data, 23 GiB used, 693 GiB / 715 GiB avail; 1.1 KiB/s rd, 56 MiB/s wr, 949 op/s 2024-03-11T21:29:18.861 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:29:18 smithi092 ceph-mon[37563]: 6.1c scrub starts 2024-03-11T21:29:18.861 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:29:18 smithi092 ceph-mon[37563]: 6.1c scrub ok 2024-03-11T21:29:18.862 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:29:18 smithi092 ceph-mon[37563]: 4.1d scrub starts 2024-03-11T21:29:18.862 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:29:18 smithi092 ceph-mon[37563]: 4.1d scrub ok 2024-03-11T21:29:18.862 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:29:18 smithi092 ceph-mon[37563]: 7.c scrub starts 2024-03-11T21:29:18.862 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:29:18 smithi092 ceph-mon[37563]: 2.f scrub starts 2024-03-11T21:29:18.862 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:29:18 smithi092 ceph-mon[37563]: 2.f scrub ok 2024-03-11T21:29:18.862 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:29:18 smithi092 ceph-mon[29572]: pgmap v124: 148 pgs: 148 active+clean; 7.3 GiB data, 23 GiB used, 693 GiB / 715 GiB avail; 1.1 KiB/s rd, 56 MiB/s wr, 949 op/s 2024-03-11T21:29:18.862 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:29:18 smithi092 ceph-mon[29572]: 6.1c scrub starts 2024-03-11T21:29:18.862 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:29:18 smithi092 ceph-mon[29572]: 6.1c scrub ok 2024-03-11T21:29:18.862 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:29:18 smithi092 ceph-mon[29572]: 4.1d scrub starts 2024-03-11T21:29:18.863 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:29:18 smithi092 ceph-mon[29572]: 4.1d scrub ok 2024-03-11T21:29:18.863 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:29:18 smithi092 ceph-mon[29572]: 7.c scrub starts 2024-03-11T21:29:18.863 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:29:18 smithi092 ceph-mon[29572]: 2.f scrub starts 2024-03-11T21:29:18.863 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:29:18 smithi092 ceph-mon[29572]: 2.f scrub ok 2024-03-11T21:29:18.868 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:29:18.869 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:29:18.869 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:29:18.970 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:29:19.019 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:29:19.020 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:29:19.021 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:29:19.060 DEBUG:teuthology.orchestra.run.smithi092:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid 00db0016-dfed-11ee-95c8-87774f69a715 -- ceph osd unset nodeep-scrub 2024-03-11T21:29:19.122 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:29:19.151 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:29:19.151 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:29:19.152 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:29:19.252 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:29:19.281 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:29:19.281 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:29:19.282 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:29:19.382 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:29:19.423 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:29:19.424 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:29:19.424 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:29:19.525 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:29:19.556 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:29:19.557 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:29:19.557 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:29:19.658 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:29:19.686 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:29:19.687 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:29:19.687 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:29:19.788 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:29:19.826 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:29:19 smithi097 ceph-mon[33317]: 2.9 scrub starts 2024-03-11T21:29:19.827 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:29:19 smithi097 ceph-mon[33317]: 2.9 scrub ok 2024-03-11T21:29:19.846 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:29:19.847 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:29:19.847 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:29:19.861 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:29:19 smithi092 ceph-mon[29572]: 2.9 scrub starts 2024-03-11T21:29:19.861 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:29:19 smithi092 ceph-mon[29572]: 2.9 scrub ok 2024-03-11T21:29:19.861 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:29:19 smithi092 ceph-mon[37563]: 2.9 scrub starts 2024-03-11T21:29:19.861 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:29:19 smithi092 ceph-mon[37563]: 2.9 scrub ok 2024-03-11T21:29:19.948 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:29:19.986 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:29:19.987 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:29:19.987 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:29:20.088 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:29:20.131 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:29:20.131 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:29:20.132 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:29:20.233 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:29:20.274 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:29:20.274 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:29:20.275 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:29:20.327 INFO:journalctl@ceph.prometheus.a.smithi097.stdout:Mar 11 21:29:20 smithi097 systemd[1]: Stopping Ceph prometheus.a for 00db0016-dfed-11ee-95c8-87774f69a715... 2024-03-11T21:29:20.328 INFO:teuthology.orchestra.run.smithi092.stdout:ERROR: (22) Invalid argument 2024-03-11T21:29:20.328 INFO:teuthology.orchestra.run.smithi092.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-11T21:29:20.375 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:29:20.404 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:29:20.405 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:29:20.405 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:29:20.506 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:29:20.537 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:29:20.537 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:29:20.537 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:29:20.638 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:29:20.670 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:29:20.671 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:29:20.671 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:29:20.752 INFO:journalctl@ceph.prometheus.a.smithi097.stdout:Mar 11 21:29:20 smithi097 ceph-00db0016-dfed-11ee-95c8-87774f69a715-prometheus-a[62608]: ts=2024-03-11T21:29:20.451Z caller=main.go:840 level=warn msg="Received SIGTERM, exiting gracefully..." 2024-03-11T21:29:20.752 INFO:journalctl@ceph.prometheus.a.smithi097.stdout:Mar 11 21:29:20 smithi097 ceph-00db0016-dfed-11ee-95c8-87774f69a715-prometheus-a[62608]: ts=2024-03-11T21:29:20.452Z caller=main.go:864 level=info msg="Stopping scrape discovery manager..." 2024-03-11T21:29:20.752 INFO:journalctl@ceph.prometheus.a.smithi097.stdout:Mar 11 21:29:20 smithi097 ceph-00db0016-dfed-11ee-95c8-87774f69a715-prometheus-a[62608]: ts=2024-03-11T21:29:20.452Z caller=main.go:878 level=info msg="Stopping notify discovery manager..." 2024-03-11T21:29:20.752 INFO:journalctl@ceph.prometheus.a.smithi097.stdout:Mar 11 21:29:20 smithi097 ceph-00db0016-dfed-11ee-95c8-87774f69a715-prometheus-a[62608]: ts=2024-03-11T21:29:20.452Z caller=manager.go:988 level=info component="rule manager" msg="Stopping rule manager..." 2024-03-11T21:29:20.752 INFO:journalctl@ceph.prometheus.a.smithi097.stdout:Mar 11 21:29:20 smithi097 ceph-00db0016-dfed-11ee-95c8-87774f69a715-prometheus-a[62608]: ts=2024-03-11T21:29:20.452Z caller=manager.go:998 level=info component="rule manager" msg="Rule manager stopped" 2024-03-11T21:29:20.752 INFO:journalctl@ceph.prometheus.a.smithi097.stdout:Mar 11 21:29:20 smithi097 ceph-00db0016-dfed-11ee-95c8-87774f69a715-prometheus-a[62608]: ts=2024-03-11T21:29:20.452Z caller=main.go:915 level=info msg="Stopping scrape manager..." 2024-03-11T21:29:20.752 INFO:journalctl@ceph.prometheus.a.smithi097.stdout:Mar 11 21:29:20 smithi097 ceph-00db0016-dfed-11ee-95c8-87774f69a715-prometheus-a[62608]: ts=2024-03-11T21:29:20.452Z caller=main.go:860 level=info msg="Scrape discovery manager stopped" 2024-03-11T21:29:20.752 INFO:journalctl@ceph.prometheus.a.smithi097.stdout:Mar 11 21:29:20 smithi097 ceph-00db0016-dfed-11ee-95c8-87774f69a715-prometheus-a[62608]: ts=2024-03-11T21:29:20.452Z caller=main.go:874 level=info msg="Notify discovery manager stopped" 2024-03-11T21:29:20.752 INFO:journalctl@ceph.prometheus.a.smithi097.stdout:Mar 11 21:29:20 smithi097 ceph-00db0016-dfed-11ee-95c8-87774f69a715-prometheus-a[62608]: ts=2024-03-11T21:29:20.452Z caller=main.go:907 level=info msg="Scrape manager stopped" 2024-03-11T21:29:20.752 INFO:journalctl@ceph.prometheus.a.smithi097.stdout:Mar 11 21:29:20 smithi097 ceph-00db0016-dfed-11ee-95c8-87774f69a715-prometheus-a[62608]: ts=2024-03-11T21:29:20.453Z caller=notifier.go:606 level=info component=notifier msg="Stopping notification manager..." 2024-03-11T21:29:20.752 INFO:journalctl@ceph.prometheus.a.smithi097.stdout:Mar 11 21:29:20 smithi097 ceph-00db0016-dfed-11ee-95c8-87774f69a715-prometheus-a[62608]: ts=2024-03-11T21:29:20.453Z caller=main.go:1135 level=info msg="Notifier manager stopped" 2024-03-11T21:29:20.752 INFO:journalctl@ceph.prometheus.a.smithi097.stdout:Mar 11 21:29:20 smithi097 ceph-00db0016-dfed-11ee-95c8-87774f69a715-prometheus-a[62608]: ts=2024-03-11T21:29:20.453Z caller=main.go:1147 level=info msg="See you next time!" 2024-03-11T21:29:20.753 INFO:journalctl@ceph.prometheus.a.smithi097.stdout:Mar 11 21:29:20 smithi097 podman[72684]: 2024-03-11 21:29:20.46191598 +0000 UTC m=+0.129105995 container died 905dc8aea8e89f027807d67743354a08f5c0189061939f389bc0a2c4c5ce4207 (image=quay.io/prometheus/prometheus:v2.43.0, name=ceph-00db0016-dfed-11ee-95c8-87774f69a715-prometheus-a, maintainer=The Prometheus Authors ) 2024-03-11T21:29:20.753 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:29:20 smithi097 ceph-mon[33317]: pgmap v125: 148 pgs: 148 active+clean; 7.3 GiB data, 23 GiB used, 692 GiB / 715 GiB avail; 1.1 KiB/s rd, 45 MiB/s wr, 880 op/s 2024-03-11T21:29:20.753 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:29:20 smithi097 ceph-mon[33317]: 7.c scrub ok 2024-03-11T21:29:20.753 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:29:20 smithi097 ceph-mon[33317]: 4.11 scrub starts 2024-03-11T21:29:20.753 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:29:20 smithi097 ceph-mon[33317]: 4.11 scrub ok 2024-03-11T21:29:20.753 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:29:20 smithi097 ceph-mon[33317]: 5.16 scrub starts 2024-03-11T21:29:20.753 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:29:20 smithi097 ceph-mon[33317]: 5.16 scrub ok 2024-03-11T21:29:20.753 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:29:20 smithi097 ceph-mon[33317]: 4.1 scrub starts 2024-03-11T21:29:20.753 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:29:20 smithi097 ceph-mon[33317]: 4.1 scrub ok 2024-03-11T21:29:20.773 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:29:20.819 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:29:20.819 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:29:20.819 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:29:20.861 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:29:20 smithi092 ceph-mon[29572]: pgmap v125: 148 pgs: 148 active+clean; 7.3 GiB data, 23 GiB used, 692 GiB / 715 GiB avail; 1.1 KiB/s rd, 45 MiB/s wr, 880 op/s 2024-03-11T21:29:20.861 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:29:20 smithi092 ceph-mon[29572]: 7.c scrub ok 2024-03-11T21:29:20.861 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:29:20 smithi092 ceph-mon[29572]: 4.11 scrub starts 2024-03-11T21:29:20.861 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:29:20 smithi092 ceph-mon[29572]: 4.11 scrub ok 2024-03-11T21:29:20.861 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:29:20 smithi092 ceph-mon[29572]: 5.16 scrub starts 2024-03-11T21:29:20.861 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:29:20 smithi092 ceph-mon[29572]: 5.16 scrub ok 2024-03-11T21:29:20.861 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:29:20 smithi092 ceph-mon[29572]: 4.1 scrub starts 2024-03-11T21:29:20.861 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:29:20 smithi092 ceph-mon[29572]: 4.1 scrub ok 2024-03-11T21:29:20.862 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:29:20 smithi092 ceph-mon[37563]: pgmap v125: 148 pgs: 148 active+clean; 7.3 GiB data, 23 GiB used, 692 GiB / 715 GiB avail; 1.1 KiB/s rd, 45 MiB/s wr, 880 op/s 2024-03-11T21:29:20.862 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:29:20 smithi092 ceph-mon[37563]: 7.c scrub ok 2024-03-11T21:29:20.862 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:29:20 smithi092 ceph-mon[37563]: 4.11 scrub starts 2024-03-11T21:29:20.862 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:29:20 smithi092 ceph-mon[37563]: 4.11 scrub ok 2024-03-11T21:29:20.862 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:29:20 smithi092 ceph-mon[37563]: 5.16 scrub starts 2024-03-11T21:29:20.862 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:29:20 smithi092 ceph-mon[37563]: 5.16 scrub ok 2024-03-11T21:29:20.862 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:29:20 smithi092 ceph-mon[37563]: 4.1 scrub starts 2024-03-11T21:29:20.862 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:29:20 smithi092 ceph-mon[37563]: 4.1 scrub ok 2024-03-11T21:29:20.921 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:29:20.950 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:29:20.951 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:29:20.951 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:29:21.036 DEBUG:teuthology.orchestra.run.smithi092:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid 00db0016-dfed-11ee-95c8-87774f69a715 -- ceph daemon osd.3 dump_blocked_ops 2024-03-11T21:29:21.052 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:29:21.082 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:29:21.083 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:29:21.083 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:29:21.184 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:29:21.213 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:29:21.213 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:29:21.214 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:29:21.315 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:29:21.344 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:29:21.344 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:29:21.345 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:29:21.392 INFO:teuthology.orchestra.run.smithi092.stderr:Inferring config /var/lib/ceph/00db0016-dfed-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T21:29:21.433 INFO:teuthology.orchestra.run.smithi092.stderr:Inferring config /var/lib/ceph/00db0016-dfed-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T21:29:21.445 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:29:21.472 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:29:21.473 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:29:21.474 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:29:21.574 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:29:21.604 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:29:21.604 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:29:21.605 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:29:21.705 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:29:21.736 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:29:21.737 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:29:21.737 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:29:21.827 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:29:21 smithi097 ceph-mon[33317]: 2.d scrub starts 2024-03-11T21:29:21.827 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:29:21 smithi097 ceph-mon[33317]: 2.d scrub ok 2024-03-11T21:29:21.827 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:29:21 smithi097 ceph-mon[33317]: from='mgr.14607 v1:172.21.15.92:0/1702717597' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T21:29:21.838 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:29:21.861 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:29:21 smithi092 ceph-mon[29572]: 2.d scrub starts 2024-03-11T21:29:21.861 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:29:21 smithi092 ceph-mon[29572]: 2.d scrub ok 2024-03-11T21:29:21.861 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:29:21 smithi092 ceph-mon[29572]: from='mgr.14607 v1:172.21.15.92:0/1702717597' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T21:29:21.861 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:29:21 smithi092 ceph-mon[37563]: 2.d scrub starts 2024-03-11T21:29:21.861 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:29:21 smithi092 ceph-mon[37563]: 2.d scrub ok 2024-03-11T21:29:21.861 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:29:21 smithi092 ceph-mon[37563]: from='mgr.14607 v1:172.21.15.92:0/1702717597' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T21:29:21.872 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:29:21.872 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:29:21.873 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:29:21.973 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:29:21.998 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:29:21.999 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:29:21.999 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:29:22.100 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:29:22.134 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:29:22.135 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:29:22.135 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:29:22.236 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:29:22.279 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:29:22.280 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:29:22.280 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:29:22.326 INFO:journalctl@ceph.iscsi.iscsi.a.smithi097.stdout:Mar 11 21:29:22 smithi097 ceph-00db0016-dfed-11ee-95c8-87774f69a715-iscsi-iscsi-a[60943]: debug there is no tcmu-runner data available 2024-03-11T21:29:22.327 INFO:journalctl@ceph.prometheus.a.smithi097.stdout:Mar 11 21:29:21 smithi097 podman[72684]: 2024-03-11 21:29:21.88611728 +0000 UTC m=+1.553307279 container cleanup 905dc8aea8e89f027807d67743354a08f5c0189061939f389bc0a2c4c5ce4207 (image=quay.io/prometheus/prometheus:v2.43.0, name=ceph-00db0016-dfed-11ee-95c8-87774f69a715-prometheus-a, maintainer=The Prometheus Authors ) 2024-03-11T21:29:22.327 INFO:journalctl@ceph.prometheus.a.smithi097.stdout:Mar 11 21:29:21 smithi097 bash[72684]: ceph-00db0016-dfed-11ee-95c8-87774f69a715-prometheus-a 2024-03-11T21:29:22.381 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:29:22.424 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:29:22.425 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:29:22.425 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:29:22.530 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:29:22.570 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:29:22.571 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:29:22.571 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:29:22.591 INFO:teuthology.orchestra.run.smithi092.stderr:set pool 1 pgp_num to 1 2024-03-11T21:29:22.672 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:29:22.709 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:29:22.710 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:29:22.710 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:29:22.811 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:29:22.852 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:29:22.852 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:29:22.852 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:29:22.860 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:29:22 smithi092 ceph-mon[29572]: pgmap v126: 148 pgs: 148 active+clean; 7.3 GiB data, 23 GiB used, 692 GiB / 715 GiB avail; 1.0 KiB/s rd, 43 MiB/s wr, 838 op/s 2024-03-11T21:29:22.861 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:29:22 smithi092 ceph-mon[29572]: 2.a scrub starts 2024-03-11T21:29:22.861 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:29:22 smithi092 ceph-mon[29572]: 2.a scrub ok 2024-03-11T21:29:22.861 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:29:22 smithi092 ceph-mon[29572]: 5.1a scrub starts 2024-03-11T21:29:22.861 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:29:22 smithi092 ceph-mon[29572]: 5.1a scrub ok 2024-03-11T21:29:22.861 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:29:22 smithi092 ceph-mon[29572]: from='client.? v1:172.21.15.92:0/661715623' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num", "val": "1"}]: dispatch 2024-03-11T21:29:22.861 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:29:22 smithi092 ceph-mon[29572]: from='client.24826 ' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num", "val": "1"}]: dispatch 2024-03-11T21:29:22.861 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:29:22 smithi092 ceph-mon[29572]: 7.d scrub starts 2024-03-11T21:29:22.861 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:29:22 smithi092 ceph-mon[29572]: 5.7 scrub starts 2024-03-11T21:29:22.861 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:29:22 smithi092 ceph-mon[29572]: 5.7 scrub ok 2024-03-11T21:29:22.862 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:29:22 smithi092 ceph-mon[37563]: pgmap v126: 148 pgs: 148 active+clean; 7.3 GiB data, 23 GiB used, 692 GiB / 715 GiB avail; 1.0 KiB/s rd, 43 MiB/s wr, 838 op/s 2024-03-11T21:29:22.862 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:29:22 smithi092 ceph-mon[37563]: 2.a scrub starts 2024-03-11T21:29:22.862 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:29:22 smithi092 ceph-mon[37563]: 2.a scrub ok 2024-03-11T21:29:22.862 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:29:22 smithi092 ceph-mon[37563]: 5.1a scrub starts 2024-03-11T21:29:22.862 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:29:22 smithi092 ceph-mon[37563]: 5.1a scrub ok 2024-03-11T21:29:22.862 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:29:22 smithi092 ceph-mon[37563]: from='client.? v1:172.21.15.92:0/661715623' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num", "val": "1"}]: dispatch 2024-03-11T21:29:22.862 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:29:22 smithi092 ceph-mon[37563]: from='client.24826 ' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num", "val": "1"}]: dispatch 2024-03-11T21:29:22.863 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:29:22 smithi092 ceph-mon[37563]: 7.d scrub starts 2024-03-11T21:29:22.863 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:29:22 smithi092 ceph-mon[37563]: 5.7 scrub starts 2024-03-11T21:29:22.863 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:29:22 smithi092 ceph-mon[37563]: 5.7 scrub ok 2024-03-11T21:29:22.954 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:29:22.995 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:29:22.995 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:29:22.996 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:29:23.076 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:29:22 smithi097 ceph-mon[33317]: pgmap v126: 148 pgs: 148 active+clean; 7.3 GiB data, 23 GiB used, 692 GiB / 715 GiB avail; 1.0 KiB/s rd, 43 MiB/s wr, 838 op/s 2024-03-11T21:29:23.077 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:29:22 smithi097 ceph-mon[33317]: 2.a scrub starts 2024-03-11T21:29:23.077 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:29:22 smithi097 ceph-mon[33317]: 2.a scrub ok 2024-03-11T21:29:23.077 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:29:22 smithi097 ceph-mon[33317]: 5.1a scrub starts 2024-03-11T21:29:23.077 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:29:22 smithi097 ceph-mon[33317]: 5.1a scrub ok 2024-03-11T21:29:23.077 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:29:22 smithi097 ceph-mon[33317]: from='client.? v1:172.21.15.92:0/661715623' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num", "val": "1"}]: dispatch 2024-03-11T21:29:23.077 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:29:22 smithi097 ceph-mon[33317]: from='client.24826 ' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num", "val": "1"}]: dispatch 2024-03-11T21:29:23.077 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:29:22 smithi097 ceph-mon[33317]: 7.d scrub starts 2024-03-11T21:29:23.077 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:29:22 smithi097 ceph-mon[33317]: 5.7 scrub starts 2024-03-11T21:29:23.077 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:29:22 smithi097 ceph-mon[33317]: 5.7 scrub ok 2024-03-11T21:29:23.097 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:29:23.121 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:29:23.121 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:29:23.121 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:29:23.142 INFO:teuthology.orchestra.run.smithi092.stderr:Inferring config /var/lib/ceph/00db0016-dfed-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T21:29:23.222 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:29:23.270 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:29:23.271 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:29:23.271 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:29:23.372 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:29:23.397 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:29:23.397 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:29:23.397 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:29:23.498 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:29:23.530 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:29:23.531 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:29:23.531 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:29:23.632 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:29:23.671 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:29:23.672 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:29:23.672 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:29:23.773 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:29:23.807 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:29:23.808 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:29:23.808 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:29:23.861 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:29:23 smithi092 ceph-mon[29572]: from='client.24469 v1:172.21.15.97:0/404208310' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:29:23.861 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:29:23 smithi092 ceph-mon[29572]: 7.5 scrub starts 2024-03-11T21:29:23.861 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:29:23 smithi092 ceph-mon[29572]: from='client.24826 ' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num", "val": "1"}]': finished 2024-03-11T21:29:23.861 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:29:23 smithi092 ceph-mon[29572]: osdmap e121: 8 total, 8 up, 8 in 2024-03-11T21:29:23.861 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:29:23 smithi092 ceph-mon[29572]: 6.11 scrub starts 2024-03-11T21:29:23.861 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:29:23 smithi092 ceph-mon[29572]: 6.11 scrub ok 2024-03-11T21:29:23.861 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:29:23 smithi092 ceph-mon[29572]: purged_snaps scrub starts 2024-03-11T21:29:23.861 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:29:23 smithi092 ceph-mon[29572]: purged_snaps scrub ok 2024-03-11T21:29:23.862 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:29:23 smithi092 ceph-mon[29572]: 7.0 scrub starts 2024-03-11T21:29:23.862 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:29:23 smithi092 ceph-mon[29572]: purged_snaps scrub starts 2024-03-11T21:29:23.862 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:29:23 smithi092 ceph-mon[29572]: purged_snaps scrub ok 2024-03-11T21:29:23.862 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:29:23 smithi092 ceph-mon[29572]: 6.17 scrub starts 2024-03-11T21:29:23.862 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:29:23 smithi092 ceph-mon[29572]: 6.17 scrub ok 2024-03-11T21:29:23.862 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:29:23 smithi092 ceph-mon[37563]: from='client.24469 v1:172.21.15.97:0/404208310' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:29:23.862 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:29:23 smithi092 ceph-mon[37563]: 7.5 scrub starts 2024-03-11T21:29:23.862 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:29:23 smithi092 ceph-mon[37563]: from='client.24826 ' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num", "val": "1"}]': finished 2024-03-11T21:29:23.862 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:29:23 smithi092 ceph-mon[37563]: osdmap e121: 8 total, 8 up, 8 in 2024-03-11T21:29:23.862 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:29:23 smithi092 ceph-mon[37563]: 6.11 scrub starts 2024-03-11T21:29:23.862 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:29:23 smithi092 ceph-mon[37563]: 6.11 scrub ok 2024-03-11T21:29:23.863 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:29:23 smithi092 ceph-mon[37563]: purged_snaps scrub starts 2024-03-11T21:29:23.863 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:29:23 smithi092 ceph-mon[37563]: purged_snaps scrub ok 2024-03-11T21:29:23.863 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:29:23 smithi092 ceph-mon[37563]: 7.0 scrub starts 2024-03-11T21:29:23.863 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:29:23 smithi092 ceph-mon[37563]: purged_snaps scrub starts 2024-03-11T21:29:23.863 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:29:23 smithi092 ceph-mon[37563]: purged_snaps scrub ok 2024-03-11T21:29:23.863 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:29:23 smithi092 ceph-mon[37563]: 6.17 scrub starts 2024-03-11T21:29:23.863 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:29:23 smithi092 ceph-mon[37563]: 6.17 scrub ok 2024-03-11T21:29:23.909 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:29:23.951 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:29:23.952 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:29:23.952 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:29:24.053 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:29:24.077 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:29:23 smithi097 ceph-mon[33317]: from='client.24469 v1:172.21.15.97:0/404208310' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:29:24.077 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:29:23 smithi097 ceph-mon[33317]: 7.5 scrub starts 2024-03-11T21:29:24.077 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:29:23 smithi097 ceph-mon[33317]: from='client.24826 ' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num", "val": "1"}]': finished 2024-03-11T21:29:24.077 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:29:23 smithi097 ceph-mon[33317]: osdmap e121: 8 total, 8 up, 8 in 2024-03-11T21:29:24.077 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:29:23 smithi097 ceph-mon[33317]: 6.11 scrub starts 2024-03-11T21:29:24.077 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:29:23 smithi097 ceph-mon[33317]: 6.11 scrub ok 2024-03-11T21:29:24.077 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:29:23 smithi097 ceph-mon[33317]: purged_snaps scrub starts 2024-03-11T21:29:24.077 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:29:23 smithi097 ceph-mon[33317]: purged_snaps scrub ok 2024-03-11T21:29:24.077 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:29:23 smithi097 ceph-mon[33317]: 7.0 scrub starts 2024-03-11T21:29:24.077 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:29:23 smithi097 ceph-mon[33317]: purged_snaps scrub starts 2024-03-11T21:29:24.077 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:29:23 smithi097 ceph-mon[33317]: purged_snaps scrub ok 2024-03-11T21:29:24.078 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:29:23 smithi097 ceph-mon[33317]: 6.17 scrub starts 2024-03-11T21:29:24.078 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:29:23 smithi097 ceph-mon[33317]: 6.17 scrub ok 2024-03-11T21:29:24.087 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:29:24.087 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:29:24.088 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:29:24.188 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:29:24.230 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:29:24.230 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:29:24.230 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:29:24.331 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:29:24.368 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:29:24.368 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:29:24.369 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:29:24.469 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:29:24.515 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:29:24.515 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:29:24.516 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:29:24.604 INFO:journalctl@ceph.prometheus.a.smithi097.stdout:Mar 11 21:29:24 smithi097 podman[72714]: 2024-03-11 21:29:24.365289693 +0000 UTC m=+3.903370150 container remove 905dc8aea8e89f027807d67743354a08f5c0189061939f389bc0a2c4c5ce4207 (image=quay.io/prometheus/prometheus:v2.43.0, name=ceph-00db0016-dfed-11ee-95c8-87774f69a715-prometheus-a, maintainer=The Prometheus Authors ) 2024-03-11T21:29:24.616 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:29:24.655 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:29:24.655 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:29:24.656 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:29:24.756 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:29:24.795 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:29:24.796 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:29:24.796 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:29:24.861 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:29:24 smithi092 ceph-mon[29572]: pgmap v128: 148 pgs: 1 active+clean+scrubbing, 147 active+clean; 7.0 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 1.1 KiB/s rd, 15 MiB/s wr, 921 op/s 2024-03-11T21:29:24.861 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:29:24 smithi092 ceph-mon[29572]: 5.9 scrub starts 2024-03-11T21:29:24.861 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:29:24 smithi092 ceph-mon[29572]: 5.9 scrub ok 2024-03-11T21:29:24.861 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:29:24 smithi092 ceph-mon[29572]: osdmap e122: 8 total, 8 up, 8 in 2024-03-11T21:29:24.861 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:29:24 smithi092 ceph-mon[29572]: 5.8 scrub starts 2024-03-11T21:29:24.861 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:29:24 smithi092 ceph-mon[29572]: 5.8 scrub ok 2024-03-11T21:29:24.861 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:29:24 smithi092 ceph-mon[37563]: pgmap v128: 148 pgs: 1 active+clean+scrubbing, 147 active+clean; 7.0 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 1.1 KiB/s rd, 15 MiB/s wr, 921 op/s 2024-03-11T21:29:24.861 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:29:24 smithi092 ceph-mon[37563]: 5.9 scrub starts 2024-03-11T21:29:24.862 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:29:24 smithi092 ceph-mon[37563]: 5.9 scrub ok 2024-03-11T21:29:24.862 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:29:24 smithi092 ceph-mon[37563]: osdmap e122: 8 total, 8 up, 8 in 2024-03-11T21:29:24.862 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:29:24 smithi092 ceph-mon[37563]: 5.8 scrub starts 2024-03-11T21:29:24.862 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:29:24 smithi092 ceph-mon[37563]: 5.8 scrub ok 2024-03-11T21:29:24.897 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:29:24.929 INFO:journalctl@ceph.prometheus.a.smithi097.stdout:Mar 11 21:29:24 smithi097 systemd[1]: ceph-00db0016-dfed-11ee-95c8-87774f69a715@prometheus.a.service: Deactivated successfully. 2024-03-11T21:29:24.929 INFO:journalctl@ceph.prometheus.a.smithi097.stdout:Mar 11 21:29:24 smithi097 systemd[1]: Stopped Ceph prometheus.a for 00db0016-dfed-11ee-95c8-87774f69a715. 2024-03-11T21:29:24.929 INFO:journalctl@ceph.prometheus.a.smithi097.stdout:Mar 11 21:29:24 smithi097 systemd[1]: ceph-00db0016-dfed-11ee-95c8-87774f69a715@prometheus.a.service: Consumed 1.425s CPU time. 2024-03-11T21:29:24.929 INFO:journalctl@ceph.prometheus.a.smithi097.stdout:Mar 11 21:29:24 smithi097 systemd[1]: Starting Ceph prometheus.a for 00db0016-dfed-11ee-95c8-87774f69a715... 2024-03-11T21:29:24.929 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:29:24 smithi097 ceph-mon[33317]: pgmap v128: 148 pgs: 1 active+clean+scrubbing, 147 active+clean; 7.0 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 1.1 KiB/s rd, 15 MiB/s wr, 921 op/s 2024-03-11T21:29:24.929 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:29:24 smithi097 ceph-mon[33317]: 5.9 scrub starts 2024-03-11T21:29:24.929 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:29:24 smithi097 ceph-mon[33317]: 5.9 scrub ok 2024-03-11T21:29:24.930 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:29:24 smithi097 ceph-mon[33317]: osdmap e122: 8 total, 8 up, 8 in 2024-03-11T21:29:24.930 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:29:24 smithi097 ceph-mon[33317]: 5.8 scrub starts 2024-03-11T21:29:24.930 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:29:24 smithi097 ceph-mon[33317]: 5.8 scrub ok 2024-03-11T21:29:24.931 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:29:24.931 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:29:24.932 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:29:25.032 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:29:25.072 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:29:25.073 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:29:25.073 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:29:25.174 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:29:25.204 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:29:25.205 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:29:25.205 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:29:25.236 INFO:journalctl@ceph.prometheus.a.smithi097.stdout:Mar 11 21:29:24 smithi097 podman[73116]: 2024-03-11 21:29:24.850721983 +0000 UTC m=+0.018627400 image pull a07b618ecd1dce142bce4c52f0e80982eaf1f14265a2415c2d35978ccaa0a464 quay.io/prometheus/prometheus:v2.43.0 2024-03-11T21:29:25.236 INFO:journalctl@ceph.prometheus.a.smithi097.stdout:Mar 11 21:29:25 smithi097 podman[73116]: 2024-03-11 21:29:25.000039379 +0000 UTC m=+0.167944789 container create cc9a759a8c93574c1f7b2325c32c3dabb61c319a00fdfd60c1a7307137141f14 (image=quay.io/prometheus/prometheus:v2.43.0, name=ceph-00db0016-dfed-11ee-95c8-87774f69a715-prometheus-a, maintainer=The Prometheus Authors ) 2024-03-11T21:29:25.307 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:29:25.341 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:29:25.342 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:29:25.343 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:29:25.362 INFO:journalctl@ceph.alertmanager.a.smithi092.stdout:Mar 11 21:29:25 smithi092 ceph-00db0016-dfed-11ee-95c8-87774f69a715-alertmanager-a[85324]: ts=2024-03-11T21:29:25.074Z caller=cluster.go:698 level=info component=cluster msg="gossip settled; proceeding" elapsed=10.003263537s 2024-03-11T21:29:25.443 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:29:25.495 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:29:25.495 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:29:25.495 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:29:25.577 INFO:journalctl@ceph.prometheus.a.smithi097.stdout:Mar 11 21:29:25 smithi097 podman[73116]: 2024-03-11 21:29:25.253606061 +0000 UTC m=+0.421511482 container init cc9a759a8c93574c1f7b2325c32c3dabb61c319a00fdfd60c1a7307137141f14 (image=quay.io/prometheus/prometheus:v2.43.0, name=ceph-00db0016-dfed-11ee-95c8-87774f69a715-prometheus-a, maintainer=The Prometheus Authors ) 2024-03-11T21:29:25.577 INFO:journalctl@ceph.prometheus.a.smithi097.stdout:Mar 11 21:29:25 smithi097 podman[73116]: 2024-03-11 21:29:25.258120995 +0000 UTC m=+0.426026405 container start cc9a759a8c93574c1f7b2325c32c3dabb61c319a00fdfd60c1a7307137141f14 (image=quay.io/prometheus/prometheus:v2.43.0, name=ceph-00db0016-dfed-11ee-95c8-87774f69a715-prometheus-a, maintainer=The Prometheus Authors ) 2024-03-11T21:29:25.577 INFO:journalctl@ceph.prometheus.a.smithi097.stdout:Mar 11 21:29:25 smithi097 ceph-00db0016-dfed-11ee-95c8-87774f69a715-prometheus-a[73147]: ts=2024-03-11T21:29:25.298Z caller=main.go:564 level=info msg="Starting Prometheus Server" mode=server version="(version=2.43.0, branch=HEAD, revision=edfc3bcd025dd6fe296c167a14a216cab1e552ee)" 2024-03-11T21:29:25.577 INFO:journalctl@ceph.prometheus.a.smithi097.stdout:Mar 11 21:29:25 smithi097 ceph-00db0016-dfed-11ee-95c8-87774f69a715-prometheus-a[73147]: ts=2024-03-11T21:29:25.298Z caller=main.go:569 level=info build_context="(go=go1.19.7, platform=linux/amd64, user=root@8a0ee342e522, date=20230321-12:56:07, tags=netgo,builtinassets)" 2024-03-11T21:29:25.577 INFO:journalctl@ceph.prometheus.a.smithi097.stdout:Mar 11 21:29:25 smithi097 ceph-00db0016-dfed-11ee-95c8-87774f69a715-prometheus-a[73147]: ts=2024-03-11T21:29:25.298Z caller=main.go:570 level=info host_details="(Linux 5.14.0-427.el9.x86_64 #1 SMP PREEMPT_DYNAMIC Fri Feb 23 04:45:07 UTC 2024 x86_64 smithi097 (none))" 2024-03-11T21:29:25.577 INFO:journalctl@ceph.prometheus.a.smithi097.stdout:Mar 11 21:29:25 smithi097 ceph-00db0016-dfed-11ee-95c8-87774f69a715-prometheus-a[73147]: ts=2024-03-11T21:29:25.298Z caller=main.go:571 level=info fd_limits="(soft=1048576, hard=1048576)" 2024-03-11T21:29:25.577 INFO:journalctl@ceph.prometheus.a.smithi097.stdout:Mar 11 21:29:25 smithi097 ceph-00db0016-dfed-11ee-95c8-87774f69a715-prometheus-a[73147]: ts=2024-03-11T21:29:25.298Z caller=main.go:572 level=info vm_limits="(soft=unlimited, hard=unlimited)" 2024-03-11T21:29:25.577 INFO:journalctl@ceph.prometheus.a.smithi097.stdout:Mar 11 21:29:25 smithi097 ceph-00db0016-dfed-11ee-95c8-87774f69a715-prometheus-a[73147]: ts=2024-03-11T21:29:25.302Z caller=web.go:561 level=info component=web msg="Start listening for connections" address=:9095 2024-03-11T21:29:25.578 INFO:journalctl@ceph.prometheus.a.smithi097.stdout:Mar 11 21:29:25 smithi097 ceph-00db0016-dfed-11ee-95c8-87774f69a715-prometheus-a[73147]: ts=2024-03-11T21:29:25.303Z caller=main.go:1005 level=info msg="Starting TSDB ..." 2024-03-11T21:29:25.578 INFO:journalctl@ceph.prometheus.a.smithi097.stdout:Mar 11 21:29:25 smithi097 ceph-00db0016-dfed-11ee-95c8-87774f69a715-prometheus-a[73147]: ts=2024-03-11T21:29:25.304Z caller=tls_config.go:232 level=info component=web msg="Listening on" address=[::]:9095 2024-03-11T21:29:25.578 INFO:journalctl@ceph.prometheus.a.smithi097.stdout:Mar 11 21:29:25 smithi097 ceph-00db0016-dfed-11ee-95c8-87774f69a715-prometheus-a[73147]: ts=2024-03-11T21:29:25.304Z caller=tls_config.go:235 level=info component=web msg="TLS is disabled." http2=false address=[::]:9095 2024-03-11T21:29:25.578 INFO:journalctl@ceph.prometheus.a.smithi097.stdout:Mar 11 21:29:25 smithi097 ceph-00db0016-dfed-11ee-95c8-87774f69a715-prometheus-a[73147]: ts=2024-03-11T21:29:25.306Z caller=head.go:587 level=info component=tsdb msg="Replaying on-disk memory mappable chunks if any" 2024-03-11T21:29:25.578 INFO:journalctl@ceph.prometheus.a.smithi097.stdout:Mar 11 21:29:25 smithi097 ceph-00db0016-dfed-11ee-95c8-87774f69a715-prometheus-a[73147]: ts=2024-03-11T21:29:25.306Z caller=head.go:658 level=info component=tsdb msg="On-disk memory mappable chunks replay completed" duration=2.891µs 2024-03-11T21:29:25.578 INFO:journalctl@ceph.prometheus.a.smithi097.stdout:Mar 11 21:29:25 smithi097 ceph-00db0016-dfed-11ee-95c8-87774f69a715-prometheus-a[73147]: ts=2024-03-11T21:29:25.306Z caller=head.go:664 level=info component=tsdb msg="Replaying WAL, this may take a while" 2024-03-11T21:29:25.578 INFO:journalctl@ceph.prometheus.a.smithi097.stdout:Mar 11 21:29:25 smithi097 ceph-00db0016-dfed-11ee-95c8-87774f69a715-prometheus-a[73147]: ts=2024-03-11T21:29:25.307Z caller=head.go:735 level=info component=tsdb msg="WAL segment loaded" segment=0 maxSegment=1 2024-03-11T21:29:25.578 INFO:journalctl@ceph.prometheus.a.smithi097.stdout:Mar 11 21:29:25 smithi097 ceph-00db0016-dfed-11ee-95c8-87774f69a715-prometheus-a[73147]: ts=2024-03-11T21:29:25.307Z caller=head.go:735 level=info component=tsdb msg="WAL segment loaded" segment=1 maxSegment=1 2024-03-11T21:29:25.578 INFO:journalctl@ceph.prometheus.a.smithi097.stdout:Mar 11 21:29:25 smithi097 ceph-00db0016-dfed-11ee-95c8-87774f69a715-prometheus-a[73147]: ts=2024-03-11T21:29:25.307Z caller=head.go:772 level=info component=tsdb msg="WAL replay completed" checkpoint_replay_duration=27.442µs wal_replay_duration=1.040858ms wbl_replay_duration=148ns total_replay_duration=1.248588ms 2024-03-11T21:29:25.578 INFO:journalctl@ceph.prometheus.a.smithi097.stdout:Mar 11 21:29:25 smithi097 ceph-00db0016-dfed-11ee-95c8-87774f69a715-prometheus-a[73147]: ts=2024-03-11T21:29:25.309Z caller=main.go:1026 level=info fs_type=XFS_SUPER_MAGIC 2024-03-11T21:29:25.578 INFO:journalctl@ceph.prometheus.a.smithi097.stdout:Mar 11 21:29:25 smithi097 ceph-00db0016-dfed-11ee-95c8-87774f69a715-prometheus-a[73147]: ts=2024-03-11T21:29:25.309Z caller=main.go:1029 level=info msg="TSDB started" 2024-03-11T21:29:25.578 INFO:journalctl@ceph.prometheus.a.smithi097.stdout:Mar 11 21:29:25 smithi097 ceph-00db0016-dfed-11ee-95c8-87774f69a715-prometheus-a[73147]: ts=2024-03-11T21:29:25.309Z caller=main.go:1209 level=info msg="Loading configuration file" filename=/etc/prometheus/prometheus.yml 2024-03-11T21:29:25.579 INFO:journalctl@ceph.prometheus.a.smithi097.stdout:Mar 11 21:29:25 smithi097 ceph-00db0016-dfed-11ee-95c8-87774f69a715-prometheus-a[73147]: ts=2024-03-11T21:29:25.355Z caller=main.go:1246 level=info msg="Completed loading of configuration file" filename=/etc/prometheus/prometheus.yml totalDuration=45.788094ms db_storage=1.283µs remote_storage=1.62µs web_handler=380ns query_engine=782ns scrape=652.41µs scrape_sd=109.058µs notify=106.684µs notify_sd=26.833µs rules=43.393192ms tracing=7.458µs 2024-03-11T21:29:25.579 INFO:journalctl@ceph.prometheus.a.smithi097.stdout:Mar 11 21:29:25 smithi097 ceph-00db0016-dfed-11ee-95c8-87774f69a715-prometheus-a[73147]: ts=2024-03-11T21:29:25.355Z caller=main.go:990 level=info msg="Server is ready to receive web requests." 2024-03-11T21:29:25.579 INFO:journalctl@ceph.prometheus.a.smithi097.stdout:Mar 11 21:29:25 smithi097 ceph-00db0016-dfed-11ee-95c8-87774f69a715-prometheus-a[73147]: ts=2024-03-11T21:29:25.355Z caller=manager.go:974 level=info component="rule manager" msg="Starting rule manager..." 2024-03-11T21:29:25.579 INFO:journalctl@ceph.prometheus.a.smithi097.stdout:Mar 11 21:29:25 smithi097 bash[73116]: cc9a759a8c93574c1f7b2325c32c3dabb61c319a00fdfd60c1a7307137141f14 2024-03-11T21:29:25.579 INFO:journalctl@ceph.prometheus.a.smithi097.stdout:Mar 11 21:29:25 smithi097 systemd[1]: Started Ceph prometheus.a for 00db0016-dfed-11ee-95c8-87774f69a715. 2024-03-11T21:29:25.596 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:29:25.650 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:29:25.651 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:29:25.651 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:29:25.679 INFO:journalctl@ceph.mgr.y.smithi092.stdout:Mar 11 21:29:25 smithi092 ceph-00db0016-dfed-11ee-95c8-87774f69a715-mgr-y[29801]: [11/Mar/2024:21:29:25] ENGINE Bus STOPPING 2024-03-11T21:29:25.679 INFO:journalctl@ceph.mgr.y.smithi092.stdout:Mar 11 21:29:25 smithi092 ceph-00db0016-dfed-11ee-95c8-87774f69a715-mgr-y[29801]: [11/Mar/2024:21:29:25] ENGINE HTTP Server cherrypy._cpwsgi_server.CPWSGIServer(('::', 9283)) shut down 2024-03-11T21:29:25.680 INFO:journalctl@ceph.mgr.y.smithi092.stdout:Mar 11 21:29:25 smithi092 ceph-00db0016-dfed-11ee-95c8-87774f69a715-mgr-y[29801]: [11/Mar/2024:21:29:25] ENGINE Bus STOPPED 2024-03-11T21:29:25.680 INFO:journalctl@ceph.mgr.y.smithi092.stdout:Mar 11 21:29:25 smithi092 ceph-00db0016-dfed-11ee-95c8-87774f69a715-mgr-y[29801]: [11/Mar/2024:21:29:25] ENGINE Bus STARTING 2024-03-11T21:29:25.680 INFO:journalctl@ceph.mgr.y.smithi092.stdout:Mar 11 21:29:25 smithi092 ceph-00db0016-dfed-11ee-95c8-87774f69a715-mgr-y[29801]: [11/Mar/2024:21:29:25] ENGINE Serving on http://:::9283 2024-03-11T21:29:25.680 INFO:journalctl@ceph.mgr.y.smithi092.stdout:Mar 11 21:29:25 smithi092 ceph-00db0016-dfed-11ee-95c8-87774f69a715-mgr-y[29801]: [11/Mar/2024:21:29:25] ENGINE Bus STARTED 2024-03-11T21:29:25.680 INFO:journalctl@ceph.mgr.y.smithi092.stdout:Mar 11 21:29:25 smithi092 ceph-00db0016-dfed-11ee-95c8-87774f69a715-mgr-y[29801]: [11/Mar/2024:21:29:25] ENGINE Bus STOPPING 2024-03-11T21:29:25.680 INFO:journalctl@ceph.mgr.y.smithi092.stdout:Mar 11 21:29:25 smithi092 ceph-00db0016-dfed-11ee-95c8-87774f69a715-mgr-y[29801]: [11/Mar/2024:21:29:25] ENGINE HTTP Server cherrypy._cpwsgi_server.CPWSGIServer(('::', 9283)) shut down 2024-03-11T21:29:25.680 INFO:journalctl@ceph.mgr.y.smithi092.stdout:Mar 11 21:29:25 smithi092 ceph-00db0016-dfed-11ee-95c8-87774f69a715-mgr-y[29801]: [11/Mar/2024:21:29:25] ENGINE Bus STOPPED 2024-03-11T21:29:25.680 INFO:journalctl@ceph.mgr.y.smithi092.stdout:Mar 11 21:29:25 smithi092 ceph-00db0016-dfed-11ee-95c8-87774f69a715-mgr-y[29801]: [11/Mar/2024:21:29:25] ENGINE Bus STARTING 2024-03-11T21:29:25.680 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:29:25 smithi092 ceph-mon[37563]: 7.d scrub ok 2024-03-11T21:29:25.680 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:29:25 smithi092 ceph-mon[37563]: 2.0 scrub starts 2024-03-11T21:29:25.680 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:29:25 smithi092 ceph-mon[37563]: 2.0 scrub ok 2024-03-11T21:29:25.681 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:29:25 smithi092 ceph-mon[37563]: 5.0 scrub starts 2024-03-11T21:29:25.681 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:29:25 smithi092 ceph-mon[37563]: 5.0 scrub ok 2024-03-11T21:29:25.681 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:29:25 smithi092 ceph-mon[37563]: purged_snaps scrub starts 2024-03-11T21:29:25.681 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:29:25 smithi092 ceph-mon[37563]: purged_snaps scrub ok 2024-03-11T21:29:25.681 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:29:25 smithi092 ceph-mon[37563]: 7.5 scrub ok 2024-03-11T21:29:25.681 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:29:25 smithi092 ceph-mon[37563]: from='mgr.14607 ' entity='mgr.y' 2024-03-11T21:29:25.681 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:29:25 smithi092 ceph-mon[37563]: from='mgr.14607 ' entity='mgr.y' 2024-03-11T21:29:25.681 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:29:25 smithi092 ceph-mon[37563]: from='mgr.14607 v1:172.21.15.92:0/1702717597' entity='mgr.y' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2024-03-11T21:29:25.681 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:29:25 smithi092 ceph-mon[37563]: from='mgr.14607 v1:172.21.15.92:0/1702717597' entity='mgr.y' cmd=[{"prefix": "dashboard set-alertmanager-api-host", "value": "http://smithi092.front.sepia.ceph.com:9093"}]: dispatch 2024-03-11T21:29:25.681 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:29:25 smithi092 ceph-mon[37563]: from='mgr.14607 ' entity='mgr.y' 2024-03-11T21:29:25.681 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:29:25 smithi092 ceph-mon[37563]: from='mgr.14607 v1:172.21.15.92:0/1702717597' entity='mgr.y' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-03-11T21:29:25.681 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:29:25 smithi092 ceph-mon[37563]: from='mgr.14607 v1:172.21.15.92:0/1702717597' entity='mgr.y' cmd=[{"prefix": "dashboard set-prometheus-api-host", "value": "http://smithi097.front.sepia.ceph.com:9095"}]: dispatch 2024-03-11T21:29:25.681 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:29:25 smithi092 ceph-mon[37563]: from='mgr.14607 ' entity='mgr.y' 2024-03-11T21:29:25.681 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:29:25 smithi092 ceph-mon[37563]: from='mgr.14607 v1:172.21.15.92:0/1702717597' entity='mgr.y' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2024-03-11T21:29:25.681 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:29:25 smithi092 ceph-mon[37563]: from='mgr.14607 v1:172.21.15.92:0/1702717597' entity='mgr.y' cmd=[{"prefix": "dashboard set-grafana-api-url", "value": "https://smithi097.front.sepia.ceph.com:3000"}]: dispatch 2024-03-11T21:29:25.682 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:29:25 smithi092 ceph-mon[37563]: from='mgr.14607 ' entity='mgr.y' 2024-03-11T21:29:25.682 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:29:25 smithi092 ceph-mon[37563]: from='mgr.14607 v1:172.21.15.92:0/1702717597' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T21:29:25.682 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:29:25 smithi092 ceph-mon[29572]: 7.d scrub ok 2024-03-11T21:29:25.682 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:29:25 smithi092 ceph-mon[29572]: 2.0 scrub starts 2024-03-11T21:29:25.682 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:29:25 smithi092 ceph-mon[29572]: 2.0 scrub ok 2024-03-11T21:29:25.682 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:29:25 smithi092 ceph-mon[29572]: 5.0 scrub starts 2024-03-11T21:29:25.682 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:29:25 smithi092 ceph-mon[29572]: 5.0 scrub ok 2024-03-11T21:29:25.682 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:29:25 smithi092 ceph-mon[29572]: purged_snaps scrub starts 2024-03-11T21:29:25.682 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:29:25 smithi092 ceph-mon[29572]: purged_snaps scrub ok 2024-03-11T21:29:25.683 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:29:25 smithi092 ceph-mon[29572]: 7.5 scrub ok 2024-03-11T21:29:25.683 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:29:25 smithi092 ceph-mon[29572]: from='mgr.14607 ' entity='mgr.y' 2024-03-11T21:29:25.683 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:29:25 smithi092 ceph-mon[29572]: from='mgr.14607 ' entity='mgr.y' 2024-03-11T21:29:25.683 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:29:25 smithi092 ceph-mon[29572]: from='mgr.14607 v1:172.21.15.92:0/1702717597' entity='mgr.y' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2024-03-11T21:29:25.683 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:29:25 smithi092 ceph-mon[29572]: from='mgr.14607 v1:172.21.15.92:0/1702717597' entity='mgr.y' cmd=[{"prefix": "dashboard set-alertmanager-api-host", "value": "http://smithi092.front.sepia.ceph.com:9093"}]: dispatch 2024-03-11T21:29:25.683 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:29:25 smithi092 ceph-mon[29572]: from='mgr.14607 ' entity='mgr.y' 2024-03-11T21:29:25.683 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:29:25 smithi092 ceph-mon[29572]: from='mgr.14607 v1:172.21.15.92:0/1702717597' entity='mgr.y' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-03-11T21:29:25.683 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:29:25 smithi092 ceph-mon[29572]: from='mgr.14607 v1:172.21.15.92:0/1702717597' entity='mgr.y' cmd=[{"prefix": "dashboard set-prometheus-api-host", "value": "http://smithi097.front.sepia.ceph.com:9095"}]: dispatch 2024-03-11T21:29:25.683 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:29:25 smithi092 ceph-mon[29572]: from='mgr.14607 ' entity='mgr.y' 2024-03-11T21:29:25.683 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:29:25 smithi092 ceph-mon[29572]: from='mgr.14607 v1:172.21.15.92:0/1702717597' entity='mgr.y' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2024-03-11T21:29:25.683 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:29:25 smithi092 ceph-mon[29572]: from='mgr.14607 v1:172.21.15.92:0/1702717597' entity='mgr.y' cmd=[{"prefix": "dashboard set-grafana-api-url", "value": "https://smithi097.front.sepia.ceph.com:3000"}]: dispatch 2024-03-11T21:29:25.683 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:29:25 smithi092 ceph-mon[29572]: from='mgr.14607 ' entity='mgr.y' 2024-03-11T21:29:25.683 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:29:25 smithi092 ceph-mon[29572]: from='mgr.14607 v1:172.21.15.92:0/1702717597' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T21:29:25.752 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:29:25.780 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:29:25.781 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:29:25.781 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:29:25.882 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:29:25.913 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:29:25.914 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:29:25.914 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:29:26.015 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:29:26.066 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:29:26.067 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:29:26.068 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:29:26.077 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:29:25 smithi097 ceph-mon[33317]: 7.d scrub ok 2024-03-11T21:29:26.077 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:29:25 smithi097 ceph-mon[33317]: 2.0 scrub starts 2024-03-11T21:29:26.077 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:29:25 smithi097 ceph-mon[33317]: 2.0 scrub ok 2024-03-11T21:29:26.077 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:29:25 smithi097 ceph-mon[33317]: 5.0 scrub starts 2024-03-11T21:29:26.077 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:29:25 smithi097 ceph-mon[33317]: 5.0 scrub ok 2024-03-11T21:29:26.077 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:29:25 smithi097 ceph-mon[33317]: purged_snaps scrub starts 2024-03-11T21:29:26.077 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:29:25 smithi097 ceph-mon[33317]: purged_snaps scrub ok 2024-03-11T21:29:26.077 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:29:25 smithi097 ceph-mon[33317]: 7.5 scrub ok 2024-03-11T21:29:26.077 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:29:25 smithi097 ceph-mon[33317]: from='mgr.14607 ' entity='mgr.y' 2024-03-11T21:29:26.077 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:29:25 smithi097 ceph-mon[33317]: from='mgr.14607 ' entity='mgr.y' 2024-03-11T21:29:26.077 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:29:25 smithi097 ceph-mon[33317]: from='mgr.14607 v1:172.21.15.92:0/1702717597' entity='mgr.y' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2024-03-11T21:29:26.078 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:29:25 smithi097 ceph-mon[33317]: from='mgr.14607 v1:172.21.15.92:0/1702717597' entity='mgr.y' cmd=[{"prefix": "dashboard set-alertmanager-api-host", "value": "http://smithi092.front.sepia.ceph.com:9093"}]: dispatch 2024-03-11T21:29:26.078 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:29:25 smithi097 ceph-mon[33317]: from='mgr.14607 ' entity='mgr.y' 2024-03-11T21:29:26.078 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:29:25 smithi097 ceph-mon[33317]: from='mgr.14607 v1:172.21.15.92:0/1702717597' entity='mgr.y' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-03-11T21:29:26.078 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:29:25 smithi097 ceph-mon[33317]: from='mgr.14607 v1:172.21.15.92:0/1702717597' entity='mgr.y' cmd=[{"prefix": "dashboard set-prometheus-api-host", "value": "http://smithi097.front.sepia.ceph.com:9095"}]: dispatch 2024-03-11T21:29:26.078 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:29:25 smithi097 ceph-mon[33317]: from='mgr.14607 ' entity='mgr.y' 2024-03-11T21:29:26.078 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:29:25 smithi097 ceph-mon[33317]: from='mgr.14607 v1:172.21.15.92:0/1702717597' entity='mgr.y' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2024-03-11T21:29:26.078 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:29:25 smithi097 ceph-mon[33317]: from='mgr.14607 v1:172.21.15.92:0/1702717597' entity='mgr.y' cmd=[{"prefix": "dashboard set-grafana-api-url", "value": "https://smithi097.front.sepia.ceph.com:3000"}]: dispatch 2024-03-11T21:29:26.078 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:29:25 smithi097 ceph-mon[33317]: from='mgr.14607 ' entity='mgr.y' 2024-03-11T21:29:26.078 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:29:25 smithi097 ceph-mon[33317]: from='mgr.14607 v1:172.21.15.92:0/1702717597' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T21:29:26.111 INFO:journalctl@ceph.mgr.y.smithi092.stdout:Mar 11 21:29:25 smithi092 ceph-00db0016-dfed-11ee-95c8-87774f69a715-mgr-y[29801]: [11/Mar/2024:21:29:25] ENGINE Serving on http://:::9283 2024-03-11T21:29:26.111 INFO:journalctl@ceph.mgr.y.smithi092.stdout:Mar 11 21:29:25 smithi092 ceph-00db0016-dfed-11ee-95c8-87774f69a715-mgr-y[29801]: [11/Mar/2024:21:29:25] ENGINE Bus STARTED 2024-03-11T21:29:26.111 INFO:journalctl@ceph.mgr.y.smithi092.stdout:Mar 11 21:29:25 smithi092 ceph-00db0016-dfed-11ee-95c8-87774f69a715-mgr-y[29801]: [11/Mar/2024:21:29:25] ENGINE Bus STOPPING 2024-03-11T21:29:26.111 INFO:journalctl@ceph.mgr.y.smithi092.stdout:Mar 11 21:29:25 smithi092 ceph-00db0016-dfed-11ee-95c8-87774f69a715-mgr-y[29801]: [11/Mar/2024:21:29:25] ENGINE HTTP Server cherrypy._cpwsgi_server.CPWSGIServer(('::', 9283)) shut down 2024-03-11T21:29:26.111 INFO:journalctl@ceph.mgr.y.smithi092.stdout:Mar 11 21:29:25 smithi092 ceph-00db0016-dfed-11ee-95c8-87774f69a715-mgr-y[29801]: [11/Mar/2024:21:29:25] ENGINE Bus STOPPED 2024-03-11T21:29:26.111 INFO:journalctl@ceph.mgr.y.smithi092.stdout:Mar 11 21:29:25 smithi092 ceph-00db0016-dfed-11ee-95c8-87774f69a715-mgr-y[29801]: [11/Mar/2024:21:29:25] ENGINE Bus STARTING 2024-03-11T21:29:26.112 INFO:journalctl@ceph.mgr.y.smithi092.stdout:Mar 11 21:29:25 smithi092 ceph-00db0016-dfed-11ee-95c8-87774f69a715-mgr-y[29801]: [11/Mar/2024:21:29:25] ENGINE Serving on http://:::9283 2024-03-11T21:29:26.112 INFO:journalctl@ceph.mgr.y.smithi092.stdout:Mar 11 21:29:25 smithi092 ceph-00db0016-dfed-11ee-95c8-87774f69a715-mgr-y[29801]: [11/Mar/2024:21:29:25] ENGINE Bus STARTED 2024-03-11T21:29:26.169 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:29:26.231 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:29:26.231 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:29:26.232 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:29:26.332 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:29:26.380 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:29:26.380 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:29:26.380 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:29:26.481 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:29:26.511 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:29:26.511 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:29:26.512 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:29:26.613 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:29:26.642 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:29:26.642 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:29:26.643 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:29:26.744 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:29:26.771 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:29:26.771 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:29:26.772 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:29:26.826 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:29:26 smithi097 ceph-mon[33317]: pgmap v130: 148 pgs: 1 active+clean+scrubbing, 147 active+clean; 7.0 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 1.2 KiB/s rd, 12 MiB/s wr, 970 op/s 2024-03-11T21:29:26.827 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:29:26 smithi097 ceph-mon[33317]: 7.6 scrub starts 2024-03-11T21:29:26.827 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:29:26 smithi097 ceph-mon[33317]: from='mon.? v1:172.21.15.97:0/414987216' entity='mon.' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2024-03-11T21:29:26.827 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:29:26 smithi097 ceph-mon[33317]: from='mon.? v1:172.21.15.97:0/414987216' entity='mon.' cmd=[{"prefix": "dashboard set-alertmanager-api-host", "value": "http://smithi092.front.sepia.ceph.com:9093"}]: dispatch 2024-03-11T21:29:26.827 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:29:26 smithi097 ceph-mon[33317]: 7.0 scrub ok 2024-03-11T21:29:26.827 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:29:26 smithi097 ceph-mon[33317]: from='mon.? v1:172.21.15.97:0/414987216' entity='mon.' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-03-11T21:29:26.827 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:29:26 smithi097 ceph-mon[33317]: from='mon.? v1:172.21.15.97:0/414987216' entity='mon.' cmd=[{"prefix": "dashboard set-prometheus-api-host", "value": "http://smithi097.front.sepia.ceph.com:9095"}]: dispatch 2024-03-11T21:29:26.827 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:29:26 smithi097 ceph-mon[33317]: from='mon.? v1:172.21.15.97:0/414987216' entity='mon.' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2024-03-11T21:29:26.827 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:29:26 smithi097 ceph-mon[33317]: from='mon.? v1:172.21.15.97:0/414987216' entity='mon.' cmd=[{"prefix": "dashboard set-grafana-api-url", "value": "https://smithi097.front.sepia.ceph.com:3000"}]: dispatch 2024-03-11T21:29:26.827 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:29:26 smithi097 ceph-mon[33317]: osdmap e123: 8 total, 8 up, 8 in 2024-03-11T21:29:26.827 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:29:26 smithi097 ceph-mon[33317]: 2.7 scrub starts 2024-03-11T21:29:26.828 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:29:26 smithi097 ceph-mon[33317]: 2.7 scrub ok 2024-03-11T21:29:26.833 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:29:26 smithi092 ceph-mon[29572]: pgmap v130: 148 pgs: 1 active+clean+scrubbing, 147 active+clean; 7.0 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 1.2 KiB/s rd, 12 MiB/s wr, 970 op/s 2024-03-11T21:29:26.833 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:29:26 smithi092 ceph-mon[29572]: 7.6 scrub starts 2024-03-11T21:29:26.833 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:29:26 smithi092 ceph-mon[29572]: from='mon.? v1:172.21.15.97:0/414987216' entity='mon.' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2024-03-11T21:29:26.833 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:29:26 smithi092 ceph-mon[29572]: from='mon.? v1:172.21.15.97:0/414987216' entity='mon.' cmd=[{"prefix": "dashboard set-alertmanager-api-host", "value": "http://smithi092.front.sepia.ceph.com:9093"}]: dispatch 2024-03-11T21:29:26.833 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:29:26 smithi092 ceph-mon[29572]: 7.0 scrub ok 2024-03-11T21:29:26.833 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:29:26 smithi092 ceph-mon[29572]: from='mon.? v1:172.21.15.97:0/414987216' entity='mon.' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-03-11T21:29:26.834 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:29:26 smithi092 ceph-mon[29572]: from='mon.? v1:172.21.15.97:0/414987216' entity='mon.' cmd=[{"prefix": "dashboard set-prometheus-api-host", "value": "http://smithi097.front.sepia.ceph.com:9095"}]: dispatch 2024-03-11T21:29:26.834 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:29:26 smithi092 ceph-mon[29572]: from='mon.? v1:172.21.15.97:0/414987216' entity='mon.' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2024-03-11T21:29:26.834 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:29:26 smithi092 ceph-mon[29572]: from='mon.? v1:172.21.15.97:0/414987216' entity='mon.' cmd=[{"prefix": "dashboard set-grafana-api-url", "value": "https://smithi097.front.sepia.ceph.com:3000"}]: dispatch 2024-03-11T21:29:26.834 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:29:26 smithi092 ceph-mon[29572]: osdmap e123: 8 total, 8 up, 8 in 2024-03-11T21:29:26.834 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:29:26 smithi092 ceph-mon[29572]: 2.7 scrub starts 2024-03-11T21:29:26.834 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:29:26 smithi092 ceph-mon[29572]: 2.7 scrub ok 2024-03-11T21:29:26.834 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:29:26 smithi092 ceph-mon[37563]: pgmap v130: 148 pgs: 1 active+clean+scrubbing, 147 active+clean; 7.0 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 1.2 KiB/s rd, 12 MiB/s wr, 970 op/s 2024-03-11T21:29:26.835 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:29:26 smithi092 ceph-mon[37563]: 7.6 scrub starts 2024-03-11T21:29:26.835 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:29:26 smithi092 ceph-mon[37563]: from='mon.? v1:172.21.15.97:0/414987216' entity='mon.' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2024-03-11T21:29:26.835 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:29:26 smithi092 ceph-mon[37563]: from='mon.? v1:172.21.15.97:0/414987216' entity='mon.' cmd=[{"prefix": "dashboard set-alertmanager-api-host", "value": "http://smithi092.front.sepia.ceph.com:9093"}]: dispatch 2024-03-11T21:29:26.835 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:29:26 smithi092 ceph-mon[37563]: 7.0 scrub ok 2024-03-11T21:29:26.835 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:29:26 smithi092 ceph-mon[37563]: from='mon.? v1:172.21.15.97:0/414987216' entity='mon.' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-03-11T21:29:26.835 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:29:26 smithi092 ceph-mon[37563]: from='mon.? v1:172.21.15.97:0/414987216' entity='mon.' cmd=[{"prefix": "dashboard set-prometheus-api-host", "value": "http://smithi097.front.sepia.ceph.com:9095"}]: dispatch 2024-03-11T21:29:26.835 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:29:26 smithi092 ceph-mon[37563]: from='mon.? v1:172.21.15.97:0/414987216' entity='mon.' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2024-03-11T21:29:26.835 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:29:26 smithi092 ceph-mon[37563]: from='mon.? v1:172.21.15.97:0/414987216' entity='mon.' cmd=[{"prefix": "dashboard set-grafana-api-url", "value": "https://smithi097.front.sepia.ceph.com:3000"}]: dispatch 2024-03-11T21:29:26.835 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:29:26 smithi092 ceph-mon[37563]: osdmap e123: 8 total, 8 up, 8 in 2024-03-11T21:29:26.836 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:29:26 smithi092 ceph-mon[37563]: 2.7 scrub starts 2024-03-11T21:29:26.836 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:29:26 smithi092 ceph-mon[37563]: 2.7 scrub ok 2024-03-11T21:29:26.873 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:29:26.901 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:29:26.902 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:29:26.902 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:29:27.003 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:29:27.051 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:29:27.051 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:29:27.052 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:29:27.084 INFO:teuthology.orchestra.run.smithi092.stdout:osd.0: {} 2024-03-11T21:29:27.126 INFO:teuthology.orchestra.run.smithi092.stdout:osd.1: {} 2024-03-11T21:29:27.154 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:29:27.171 INFO:teuthology.orchestra.run.smithi092.stdout:osd.2: {} 2024-03-11T21:29:27.197 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:29:27.197 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:29:27.197 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:29:27.248 INFO:teuthology.orchestra.run.smithi092.stdout:osd.3: {} 2024-03-11T21:29:27.288 INFO:teuthology.orchestra.run.smithi092.stdout:osd.4: {} 2024-03-11T21:29:27.299 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:29:27.328 INFO:teuthology.orchestra.run.smithi092.stdout:osd.5: {} 2024-03-11T21:29:27.329 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:29:27.330 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:29:27.331 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:29:27.368 INFO:teuthology.orchestra.run.smithi092.stdout:osd.6: {} 2024-03-11T21:29:27.411 INFO:teuthology.orchestra.run.smithi092.stdout:osd.7: {} 2024-03-11T21:29:27.418 INFO:teuthology.orchestra.run.smithi092.stderr:osd.0: osd_enable_op_tracker = '' 2024-03-11T21:29:27.418 INFO:teuthology.orchestra.run.smithi092.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-11T21:29:27.418 INFO:teuthology.orchestra.run.smithi092.stderr:osd.2: osd_enable_op_tracker = '' 2024-03-11T21:29:27.418 INFO:teuthology.orchestra.run.smithi092.stderr:osd.3: osd_enable_op_tracker = '' 2024-03-11T21:29:27.418 INFO:teuthology.orchestra.run.smithi092.stderr:osd.4: osd_enable_op_tracker = '' 2024-03-11T21:29:27.419 INFO:teuthology.orchestra.run.smithi092.stderr:osd.5: osd_enable_op_tracker = '' 2024-03-11T21:29:27.419 INFO:teuthology.orchestra.run.smithi092.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-11T21:29:27.419 INFO:teuthology.orchestra.run.smithi092.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-11T21:29:27.431 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:29:27.457 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:29:27.458 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:29:27.458 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:29:27.559 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:29:27.607 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:29:27.609 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:29:27.609 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:29:27.711 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:29:27.747 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:29:27.747 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:29:27.748 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:29:27.848 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:29:27.878 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:29:27.878 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:29:27.878 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:29:27.942 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:29:27 smithi092 ceph-mon[29572]: 4.17 scrub starts 2024-03-11T21:29:27.942 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:29:27 smithi092 ceph-mon[29572]: 4.17 scrub ok 2024-03-11T21:29:27.942 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:29:27 smithi092 ceph-mon[29572]: from='client.? v1:172.21.15.92:0/1158829350' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-11T21:29:27.942 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:29:27 smithi092 ceph-mon[29572]: 7.8 scrub starts 2024-03-11T21:29:27.943 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:29:27 smithi092 ceph-mon[29572]: 5.4 scrub starts 2024-03-11T21:29:27.943 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:29:27 smithi092 ceph-mon[29572]: 5.4 scrub ok 2024-03-11T21:29:27.943 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:29:27 smithi092 ceph-mon[37563]: 4.17 scrub starts 2024-03-11T21:29:27.943 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:29:27 smithi092 ceph-mon[37563]: 4.17 scrub ok 2024-03-11T21:29:27.943 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:29:27 smithi092 ceph-mon[37563]: from='client.? v1:172.21.15.92:0/1158829350' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-11T21:29:27.943 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:29:27 smithi092 ceph-mon[37563]: 7.8 scrub starts 2024-03-11T21:29:27.943 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:29:27 smithi092 ceph-mon[37563]: 5.4 scrub starts 2024-03-11T21:29:27.943 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:29:27 smithi092 ceph-mon[37563]: 5.4 scrub ok 2024-03-11T21:29:27.979 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:29:28.010 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:29:28.010 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:29:28.010 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:29:28.077 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:29:27 smithi097 ceph-mon[33317]: 4.17 scrub starts 2024-03-11T21:29:28.077 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:29:27 smithi097 ceph-mon[33317]: 4.17 scrub ok 2024-03-11T21:29:28.077 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:29:27 smithi097 ceph-mon[33317]: from='client.? v1:172.21.15.92:0/1158829350' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-11T21:29:28.077 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:29:27 smithi097 ceph-mon[33317]: 7.8 scrub starts 2024-03-11T21:29:28.077 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:29:27 smithi097 ceph-mon[33317]: 5.4 scrub starts 2024-03-11T21:29:28.077 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:29:27 smithi097 ceph-mon[33317]: 5.4 scrub ok 2024-03-11T21:29:28.112 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:29:28.164 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:29:28.165 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:29:28.165 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:29:28.266 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:29:28.290 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:29:28.291 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:29:28.291 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:29:28.392 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:29:28.419 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:29:28.420 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:29:28.421 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:29:28.521 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:29:28.549 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:29:28.549 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:29:28.549 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:29:28.650 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:29:28.680 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:29:28.681 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:29:28.681 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:29:28.782 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:29:28.810 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:29:28.810 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:29:28.811 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:29:28.827 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:29:28 smithi097 ceph-mon[33317]: pgmap v132: 148 pgs: 148 active+clean; 6.9 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 853 B/s rd, 0 B/s wr, 1.16k op/s 2024-03-11T21:29:28.827 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:29:28 smithi097 ceph-mon[33317]: 4.14 scrub starts 2024-03-11T21:29:28.827 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:29:28 smithi097 ceph-mon[33317]: 4.14 scrub ok 2024-03-11T21:29:28.827 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:29:28 smithi097 ceph-mon[33317]: 5.15 scrub starts 2024-03-11T21:29:28.827 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:29:28 smithi097 ceph-mon[33317]: 5.15 scrub ok 2024-03-11T21:29:28.827 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:29:28 smithi097 ceph-mon[33317]: from='client.? v1:172.21.15.92:0/3716748779' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-11T21:29:28.911 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:29:28.950 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:29:28.951 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:29:28.951 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:29:28.965 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:29:28 smithi092 ceph-mon[29572]: pgmap v132: 148 pgs: 148 active+clean; 6.9 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 853 B/s rd, 0 B/s wr, 1.16k op/s 2024-03-11T21:29:28.966 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:29:28 smithi092 ceph-mon[29572]: 4.14 scrub starts 2024-03-11T21:29:28.966 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:29:28 smithi092 ceph-mon[29572]: 4.14 scrub ok 2024-03-11T21:29:28.966 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:29:28 smithi092 ceph-mon[29572]: 5.15 scrub starts 2024-03-11T21:29:28.966 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:29:28 smithi092 ceph-mon[29572]: 5.15 scrub ok 2024-03-11T21:29:28.966 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:29:28 smithi092 ceph-mon[29572]: from='client.? v1:172.21.15.92:0/3716748779' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-11T21:29:28.966 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:29:28 smithi092 ceph-mon[37563]: pgmap v132: 148 pgs: 148 active+clean; 6.9 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 853 B/s rd, 0 B/s wr, 1.16k op/s 2024-03-11T21:29:28.966 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:29:28 smithi092 ceph-mon[37563]: 4.14 scrub starts 2024-03-11T21:29:28.966 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:29:28 smithi092 ceph-mon[37563]: 4.14 scrub ok 2024-03-11T21:29:28.966 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:29:28 smithi092 ceph-mon[37563]: 5.15 scrub starts 2024-03-11T21:29:28.966 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:29:28 smithi092 ceph-mon[37563]: 5.15 scrub ok 2024-03-11T21:29:28.966 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:29:28 smithi092 ceph-mon[37563]: from='client.? v1:172.21.15.92:0/3716748779' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-11T21:29:29.052 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:29:29.097 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:29:29.098 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:29:29.098 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:29:29.168 INFO:teuthology.orchestra.run.smithi092.stderr:nodeep-scrub is unset 2024-03-11T21:29:29.199 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:29:29.229 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:29:29.229 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:29:29.230 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:29:29.330 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:29:29.376 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:29:29.378 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:29:29.379 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:29:29.480 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:29:29.509 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:29:29.509 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:29:29.510 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:29:29.611 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:29:29.641 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:29:29.642 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:29:29.642 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:29:29.742 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:29:29.776 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:29:29.776 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:29:29.776 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:29:29.806 INFO:teuthology.orchestra.run.smithi092.stdout:{ 2024-03-11T21:29:29.807 INFO:teuthology.orchestra.run.smithi092.stdout: "ops": [], 2024-03-11T21:29:29.807 INFO:teuthology.orchestra.run.smithi092.stdout: "complaint_time": 30, 2024-03-11T21:29:29.807 INFO:teuthology.orchestra.run.smithi092.stdout: "num_blocked_ops": 0 2024-03-11T21:29:29.807 INFO:teuthology.orchestra.run.smithi092.stdout:} 2024-03-11T21:29:29.861 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:29:29 smithi092 ceph-mon[29572]: 7.6 scrub ok 2024-03-11T21:29:29.861 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:29:29 smithi092 ceph-mon[29572]: 5.f scrub starts 2024-03-11T21:29:29.861 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:29:29 smithi092 ceph-mon[29572]: 5.f scrub ok 2024-03-11T21:29:29.861 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:29:29 smithi092 ceph-mon[29572]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-03-11T21:29:29.861 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:29:29 smithi092 ceph-mon[29572]: Cluster is now healthy 2024-03-11T21:29:29.861 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:29:29 smithi092 ceph-mon[29572]: from='client.? v1:172.21.15.92:0/3716748779' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-03-11T21:29:29.861 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:29:29 smithi092 ceph-mon[29572]: osdmap e124: 8 total, 8 up, 8 in 2024-03-11T21:29:29.861 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:29:29 smithi092 ceph-mon[29572]: 7.1 scrub starts 2024-03-11T21:29:29.861 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:29:29 smithi092 ceph-mon[29572]: 7.8 scrub ok 2024-03-11T21:29:29.862 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:29:29 smithi092 ceph-mon[37563]: 7.6 scrub ok 2024-03-11T21:29:29.862 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:29:29 smithi092 ceph-mon[37563]: 5.f scrub starts 2024-03-11T21:29:29.862 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:29:29 smithi092 ceph-mon[37563]: 5.f scrub ok 2024-03-11T21:29:29.862 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:29:29 smithi092 ceph-mon[37563]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-03-11T21:29:29.862 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:29:29 smithi092 ceph-mon[37563]: Cluster is now healthy 2024-03-11T21:29:29.862 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:29:29 smithi092 ceph-mon[37563]: from='client.? v1:172.21.15.92:0/3716748779' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-03-11T21:29:29.862 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:29:29 smithi092 ceph-mon[37563]: osdmap e124: 8 total, 8 up, 8 in 2024-03-11T21:29:29.862 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:29:29 smithi092 ceph-mon[37563]: 7.1 scrub starts 2024-03-11T21:29:29.862 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:29:29 smithi092 ceph-mon[37563]: 7.8 scrub ok 2024-03-11T21:29:29.877 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:29:29.905 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:29:29.906 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:29:29.906 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:29:30.007 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:29:30.035 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:29:30.038 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:29:30.038 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:29:30.076 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:29:29 smithi097 ceph-mon[33317]: 7.6 scrub ok 2024-03-11T21:29:30.076 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:29:29 smithi097 ceph-mon[33317]: 5.f scrub starts 2024-03-11T21:29:30.076 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:29:29 smithi097 ceph-mon[33317]: 5.f scrub ok 2024-03-11T21:29:30.076 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:29:29 smithi097 ceph-mon[33317]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-03-11T21:29:30.077 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:29:29 smithi097 ceph-mon[33317]: Cluster is now healthy 2024-03-11T21:29:30.077 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:29:29 smithi097 ceph-mon[33317]: from='client.? v1:172.21.15.92:0/3716748779' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-03-11T21:29:30.077 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:29:29 smithi097 ceph-mon[33317]: osdmap e124: 8 total, 8 up, 8 in 2024-03-11T21:29:30.077 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:29:29 smithi097 ceph-mon[33317]: 7.1 scrub starts 2024-03-11T21:29:30.077 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:29:29 smithi097 ceph-mon[33317]: 7.8 scrub ok 2024-03-11T21:29:30.139 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:29:30.197 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:29:30.198 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:29:30.199 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:29:30.299 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:29:30.339 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:29:30.340 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:29:30.341 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:29:30.441 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:29:30.485 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:29:30.486 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:29:30.486 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:29:30.587 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:29:30.610 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:29:30.611 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:29:30.611 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:29:30.712 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:29:30.739 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5, 6, 7] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7] 2024-03-11T21:29:30.739 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-03-11T21:29:30.739 INFO:tasks.thrashosds.thrasher:Shrinking pool unique_pool_0 2024-03-11T21:29:30.740 INFO:tasks.cephadm.ceph_manager.ceph:contract_pool unique_pool_0 by 10 min 8 2024-03-11T21:29:30.740 DEBUG:teuthology.orchestra.run.smithi092:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid 00db0016-dfed-11ee-95c8-87774f69a715 -- ceph pg dump --format=json 2024-03-11T21:29:30.755 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:29:30.756 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:29:30.756 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:29:30.826 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:29:30 smithi097 ceph-mon[33317]: pgmap v133: 148 pgs: 1 active+clean+scrubbing, 147 active+clean; 6.8 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 797 B/s rd, 0 B/s wr, 694 op/s 2024-03-11T21:29:30.826 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:29:30 smithi097 ceph-mon[33317]: 2.14 scrub starts 2024-03-11T21:29:30.827 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:29:30 smithi097 ceph-mon[33317]: 2.14 scrub ok 2024-03-11T21:29:30.827 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:29:30 smithi097 ceph-mon[33317]: 2.c deep-scrub starts 2024-03-11T21:29:30.827 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:29:30 smithi097 ceph-mon[33317]: 2.c deep-scrub ok 2024-03-11T21:29:30.857 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:29:30.880 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:29:30.881 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:29:30.881 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:29:30.982 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:29:31.023 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:29:31.023 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:29:31.024 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:29:31.111 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:29:30 smithi092 ceph-mon[29572]: pgmap v133: 148 pgs: 1 active+clean+scrubbing, 147 active+clean; 6.8 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 797 B/s rd, 0 B/s wr, 694 op/s 2024-03-11T21:29:31.111 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:29:30 smithi092 ceph-mon[29572]: 2.14 scrub starts 2024-03-11T21:29:31.111 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:29:30 smithi092 ceph-mon[29572]: 2.14 scrub ok 2024-03-11T21:29:31.111 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:29:30 smithi092 ceph-mon[29572]: 2.c deep-scrub starts 2024-03-11T21:29:31.111 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:29:30 smithi092 ceph-mon[29572]: 2.c deep-scrub ok 2024-03-11T21:29:31.111 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:29:30 smithi092 ceph-mon[37563]: pgmap v133: 148 pgs: 1 active+clean+scrubbing, 147 active+clean; 6.8 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 797 B/s rd, 0 B/s wr, 694 op/s 2024-03-11T21:29:31.111 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:29:30 smithi092 ceph-mon[37563]: 2.14 scrub starts 2024-03-11T21:29:31.111 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:29:30 smithi092 ceph-mon[37563]: 2.14 scrub ok 2024-03-11T21:29:31.111 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:29:30 smithi092 ceph-mon[37563]: 2.c deep-scrub starts 2024-03-11T21:29:31.111 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:29:30 smithi092 ceph-mon[37563]: 2.c deep-scrub ok 2024-03-11T21:29:31.125 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:29:31.127 DEBUG:teuthology.orchestra.run.smithi092:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid 00db0016-dfed-11ee-95c8-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-03-11T21:29:31.171 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:29:31.172 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:29:31.172 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:29:31.273 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:29:31.330 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:29:31.331 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:29:31.331 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:29:31.432 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:29:31.472 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:29:31.473 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:29:31.474 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:29:31.574 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:29:31.607 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:29:31.608 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:29:31.608 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:29:31.709 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:29:31.732 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:29:31.733 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:29:31.733 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:29:31.826 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:29:31 smithi097 ceph-mon[33317]: 7.1 scrub ok 2024-03-11T21:29:31.826 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:29:31 smithi097 ceph-mon[33317]: 2.3 scrub starts 2024-03-11T21:29:31.827 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:29:31 smithi097 ceph-mon[33317]: 2.3 scrub ok 2024-03-11T21:29:31.827 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:29:31 smithi097 ceph-mon[33317]: 4.1f deep-scrub starts 2024-03-11T21:29:31.827 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:29:31 smithi097 ceph-mon[33317]: 4.1f deep-scrub ok 2024-03-11T21:29:31.827 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:29:31 smithi097 ceph-mon[33317]: 5.e scrub starts 2024-03-11T21:29:31.827 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:29:31 smithi097 ceph-mon[33317]: 5.e scrub ok 2024-03-11T21:29:31.834 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:29:31.862 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:29:31.863 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:29:31.863 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:29:31.964 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:29:31.996 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:29:31.996 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:29:31.997 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:29:32.098 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:29:32.110 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:29:31 smithi092 ceph-mon[29572]: 7.1 scrub ok 2024-03-11T21:29:32.111 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:29:31 smithi092 ceph-mon[29572]: 2.3 scrub starts 2024-03-11T21:29:32.111 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:29:31 smithi092 ceph-mon[29572]: 2.3 scrub ok 2024-03-11T21:29:32.111 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:29:31 smithi092 ceph-mon[29572]: 4.1f deep-scrub starts 2024-03-11T21:29:32.111 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:29:31 smithi092 ceph-mon[29572]: 4.1f deep-scrub ok 2024-03-11T21:29:32.111 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:29:31 smithi092 ceph-mon[29572]: 5.e scrub starts 2024-03-11T21:29:32.111 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:29:31 smithi092 ceph-mon[29572]: 5.e scrub ok 2024-03-11T21:29:32.111 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:29:31 smithi092 ceph-mon[37563]: 7.1 scrub ok 2024-03-11T21:29:32.111 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:29:31 smithi092 ceph-mon[37563]: 2.3 scrub starts 2024-03-11T21:29:32.111 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:29:31 smithi092 ceph-mon[37563]: 2.3 scrub ok 2024-03-11T21:29:32.111 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:29:31 smithi092 ceph-mon[37563]: 4.1f deep-scrub starts 2024-03-11T21:29:32.111 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:29:31 smithi092 ceph-mon[37563]: 4.1f deep-scrub ok 2024-03-11T21:29:32.111 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:29:31 smithi092 ceph-mon[37563]: 5.e scrub starts 2024-03-11T21:29:32.112 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:29:31 smithi092 ceph-mon[37563]: 5.e scrub ok 2024-03-11T21:29:32.128 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:29:32.128 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:29:32.129 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:29:32.229 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:29:32.253 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:29:32.254 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:29:32.254 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:29:32.355 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:29:32.394 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:29:32.395 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:29:32.395 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:29:32.496 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:29:32.531 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:29:32.532 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:29:32.532 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:29:32.576 INFO:journalctl@ceph.iscsi.iscsi.a.smithi097.stdout:Mar 11 21:29:32 smithi097 ceph-00db0016-dfed-11ee-95c8-87774f69a715-iscsi-iscsi-a[60943]: debug there is no tcmu-runner data available 2024-03-11T21:29:32.633 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:29:32.669 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:29:32.669 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:29:32.670 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:29:32.708 DEBUG:teuthology.orchestra.run.smithi092:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid 00db0016-dfed-11ee-95c8-87774f69a715 -- ceph daemon osd.3 dump_historic_ops 2024-03-11T21:29:32.771 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:29:32.798 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:29:32.798 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:29:32.798 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:29:32.899 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:29:32.951 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:29:32.951 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:29:32.952 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:29:33.052 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:29:33.106 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:29:33.106 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:29:33.107 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:29:33.111 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:29:32 smithi092 ceph-mon[29572]: pgmap v135: 148 pgs: 1 active+clean+scrubbing, 147 active+clean; 6.8 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 691 B/s rd, 0 B/s wr, 602 op/s 2024-03-11T21:29:33.111 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:29:32 smithi092 ceph-mon[29572]: 4.b scrub starts 2024-03-11T21:29:33.112 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:29:32 smithi092 ceph-mon[29572]: 4.b scrub ok 2024-03-11T21:29:33.112 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:29:32 smithi092 ceph-mon[29572]: from='mgr.14607 ' entity='mgr.y' 2024-03-11T21:29:33.112 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:29:32 smithi092 ceph-mon[29572]: from='mgr.14607 ' entity='mgr.y' 2024-03-11T21:29:33.112 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:29:32 smithi092 ceph-mon[29572]: 2.2 scrub starts 2024-03-11T21:29:33.112 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:29:32 smithi092 ceph-mon[29572]: 2.2 scrub ok 2024-03-11T21:29:33.113 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:29:32 smithi092 ceph-mon[37563]: pgmap v135: 148 pgs: 1 active+clean+scrubbing, 147 active+clean; 6.8 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 691 B/s rd, 0 B/s wr, 602 op/s 2024-03-11T21:29:33.113 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:29:32 smithi092 ceph-mon[37563]: 4.b scrub starts 2024-03-11T21:29:33.113 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:29:32 smithi092 ceph-mon[37563]: 4.b scrub ok 2024-03-11T21:29:33.113 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:29:32 smithi092 ceph-mon[37563]: from='mgr.14607 ' entity='mgr.y' 2024-03-11T21:29:33.113 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:29:32 smithi092 ceph-mon[37563]: from='mgr.14607 ' entity='mgr.y' 2024-03-11T21:29:33.113 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:29:32 smithi092 ceph-mon[37563]: 2.2 scrub starts 2024-03-11T21:29:33.113 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:29:32 smithi092 ceph-mon[37563]: 2.2 scrub ok 2024-03-11T21:29:33.208 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:29:33.255 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:29:33.256 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:29:33.257 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:29:33.327 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:29:32 smithi097 ceph-mon[33317]: pgmap v135: 148 pgs: 1 active+clean+scrubbing, 147 active+clean; 6.8 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 691 B/s rd, 0 B/s wr, 602 op/s 2024-03-11T21:29:33.327 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:29:32 smithi097 ceph-mon[33317]: 4.b scrub starts 2024-03-11T21:29:33.327 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:29:32 smithi097 ceph-mon[33317]: 4.b scrub ok 2024-03-11T21:29:33.327 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:29:32 smithi097 ceph-mon[33317]: from='mgr.14607 ' entity='mgr.y' 2024-03-11T21:29:33.327 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:29:32 smithi097 ceph-mon[33317]: from='mgr.14607 ' entity='mgr.y' 2024-03-11T21:29:33.327 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:29:32 smithi097 ceph-mon[33317]: 2.2 scrub starts 2024-03-11T21:29:33.327 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:29:32 smithi097 ceph-mon[33317]: 2.2 scrub ok 2024-03-11T21:29:33.358 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:29:33.389 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:29:33.389 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:29:33.390 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:29:33.490 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:29:33.528 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:29:33.528 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:29:33.529 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:29:33.629 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:29:33.645 INFO:teuthology.orchestra.run.smithi092.stderr:Inferring config /var/lib/ceph/00db0016-dfed-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T21:29:33.652 INFO:teuthology.orchestra.run.smithi092.stderr:Inferring config /var/lib/ceph/00db0016-dfed-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T21:29:33.704 INFO:teuthology.orchestra.run.smithi092.stderr:Inferring config /var/lib/ceph/00db0016-dfed-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T21:29:33.704 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:29:33.705 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:29:33.705 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:29:33.806 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:29:33.854 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:29:33.854 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:29:33.855 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:29:33.955 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:29:33.986 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:29:33.987 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:29:33.988 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:29:34.088 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:29:34.111 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:29:33 smithi092 ceph-mon[29572]: from='client.24469 v1:172.21.15.97:0/404208310' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:29:34.111 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:29:33 smithi092 ceph-mon[29572]: 2.6 scrub starts 2024-03-11T21:29:34.111 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:29:33 smithi092 ceph-mon[29572]: 2.6 scrub ok 2024-03-11T21:29:34.111 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:29:33 smithi092 ceph-mon[37563]: from='client.24469 v1:172.21.15.97:0/404208310' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:29:34.111 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:29:33 smithi092 ceph-mon[37563]: 2.6 scrub starts 2024-03-11T21:29:34.111 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:29:33 smithi092 ceph-mon[37563]: 2.6 scrub ok 2024-03-11T21:29:34.112 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:29:34.113 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:29:34.113 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:29:34.214 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:29:34.255 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:29:34.256 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:29:34.256 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:29:34.326 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:29:33 smithi097 ceph-mon[33317]: from='client.24469 v1:172.21.15.97:0/404208310' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:29:34.327 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:29:33 smithi097 ceph-mon[33317]: 2.6 scrub starts 2024-03-11T21:29:34.327 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:29:33 smithi097 ceph-mon[33317]: 2.6 scrub ok 2024-03-11T21:29:34.357 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:29:34.385 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:29:34.386 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:29:34.387 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:29:34.488 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:29:34.524 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:29:34.524 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:29:34.524 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:29:34.625 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:29:34.682 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:29:34.682 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:29:34.682 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:29:34.783 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:29:34.831 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:29:34.832 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:29:34.832 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:29:34.933 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:29:34.972 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:29:34.973 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:29:34.973 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:29:34.985 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:29:34 smithi092 ceph-mon[29572]: pgmap v136: 148 pgs: 148 active+clean; 6.5 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.11k op/s 2024-03-11T21:29:34.985 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:29:34 smithi092 ceph-mon[29572]: 7.f deep-scrub starts 2024-03-11T21:29:34.985 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:29:34 smithi092 ceph-mon[29572]: 7.a scrub starts 2024-03-11T21:29:34.985 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:29:34 smithi092 ceph-mon[37563]: pgmap v136: 148 pgs: 148 active+clean; 6.5 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.11k op/s 2024-03-11T21:29:34.985 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:29:34 smithi092 ceph-mon[37563]: 7.f deep-scrub starts 2024-03-11T21:29:34.986 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:29:34 smithi092 ceph-mon[37563]: 7.a scrub starts 2024-03-11T21:29:35.074 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:29:35.103 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:29:35.103 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:29:35.104 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:29:35.205 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:29:35.234 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:29:35.235 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:29:35.235 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:29:35.326 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:29:34 smithi097 ceph-mon[33317]: pgmap v136: 148 pgs: 148 active+clean; 6.5 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.11k op/s 2024-03-11T21:29:35.326 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:29:34 smithi097 ceph-mon[33317]: 7.f deep-scrub starts 2024-03-11T21:29:35.327 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:29:34 smithi097 ceph-mon[33317]: 7.a scrub starts 2024-03-11T21:29:35.336 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:29:35.371 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:29:35.372 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:29:35.373 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:29:35.473 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:29:35.502 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:29:35.502 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:29:35.503 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:29:35.604 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:29:35.613 DEBUG:teuthology.orchestra.run.smithi092:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid 00db0016-dfed-11ee-95c8-87774f69a715 -- ceph osd set noscrub 2024-03-11T21:29:35.644 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:29:35.645 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:29:35.645 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:29:35.746 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:29:35.772 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:29:35.772 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:29:35.773 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:29:35.874 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:29:35.921 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:29:35.924 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:29:35.924 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:29:36.025 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:29:36.066 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:29:36.066 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:29:36.067 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:29:36.167 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:29:36.195 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:29:36.196 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:29:36.196 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:29:36.297 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:29:36.326 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:29:35 smithi097 ceph-mon[33317]: 6.e scrub starts 2024-03-11T21:29:36.326 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:29:35 smithi097 ceph-mon[33317]: 6.e scrub ok 2024-03-11T21:29:36.327 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:29:35 smithi097 ceph-mon[33317]: 5.14 scrub starts 2024-03-11T21:29:36.327 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:29:35 smithi097 ceph-mon[33317]: 5.14 scrub ok 2024-03-11T21:29:36.327 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:29:35 smithi097 ceph-mon[33317]: 6.7 deep-scrub starts 2024-03-11T21:29:36.327 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:29:35 smithi097 ceph-mon[33317]: 6.7 deep-scrub ok 2024-03-11T21:29:36.327 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:29:35 smithi097 ceph-mon[33317]: 2.4 scrub starts 2024-03-11T21:29:36.327 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:29:35 smithi097 ceph-mon[33317]: 2.4 scrub ok 2024-03-11T21:29:36.331 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:29:36.332 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:29:36.332 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:29:36.361 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:29:35 smithi092 ceph-mon[29572]: 6.e scrub starts 2024-03-11T21:29:36.361 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:29:35 smithi092 ceph-mon[29572]: 6.e scrub ok 2024-03-11T21:29:36.361 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:29:35 smithi092 ceph-mon[29572]: 5.14 scrub starts 2024-03-11T21:29:36.361 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:29:35 smithi092 ceph-mon[29572]: 5.14 scrub ok 2024-03-11T21:29:36.361 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:29:35 smithi092 ceph-mon[29572]: 6.7 deep-scrub starts 2024-03-11T21:29:36.361 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:29:35 smithi092 ceph-mon[29572]: 6.7 deep-scrub ok 2024-03-11T21:29:36.361 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:29:35 smithi092 ceph-mon[29572]: 2.4 scrub starts 2024-03-11T21:29:36.361 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:29:35 smithi092 ceph-mon[29572]: 2.4 scrub ok 2024-03-11T21:29:36.361 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:29:35 smithi092 ceph-mon[37563]: 6.e scrub starts 2024-03-11T21:29:36.362 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:29:35 smithi092 ceph-mon[37563]: 6.e scrub ok 2024-03-11T21:29:36.362 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:29:35 smithi092 ceph-mon[37563]: 5.14 scrub starts 2024-03-11T21:29:36.362 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:29:35 smithi092 ceph-mon[37563]: 5.14 scrub ok 2024-03-11T21:29:36.362 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:29:35 smithi092 ceph-mon[37563]: 6.7 deep-scrub starts 2024-03-11T21:29:36.362 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:29:35 smithi092 ceph-mon[37563]: 6.7 deep-scrub ok 2024-03-11T21:29:36.362 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:29:35 smithi092 ceph-mon[37563]: 2.4 scrub starts 2024-03-11T21:29:36.362 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:29:35 smithi092 ceph-mon[37563]: 2.4 scrub ok 2024-03-11T21:29:36.433 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:29:36.468 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:29:36.469 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:29:36.469 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:29:36.570 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:29:36.593 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:29:36.594 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:29:36.594 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:29:36.695 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:29:36.724 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:29:36.724 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:29:36.725 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:29:36.826 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:29:36.864 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:29:36.864 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:29:36.865 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:29:36.966 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:29:36.994 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:29:36.995 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:29:36.995 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:29:37.096 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:29:37.110 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:29:36 smithi092 ceph-mon[29572]: pgmap v137: 148 pgs: 148 active+clean; 6.5 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 1.1 KiB/s rd, 0 B/s wr, 948 op/s 2024-03-11T21:29:37.111 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:29:36 smithi092 ceph-mon[29572]: 7.a scrub ok 2024-03-11T21:29:37.111 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:29:36 smithi092 ceph-mon[29572]: 2.18 scrub starts 2024-03-11T21:29:37.111 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:29:36 smithi092 ceph-mon[29572]: 2.18 scrub ok 2024-03-11T21:29:37.111 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:29:36 smithi092 ceph-mon[29572]: from='mgr.14607 ' entity='mgr.y' 2024-03-11T21:29:37.111 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:29:36 smithi092 ceph-mon[29572]: from='mgr.14607 v1:172.21.15.92:0/1702717597' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T21:29:37.111 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:29:36 smithi092 ceph-mon[37563]: pgmap v137: 148 pgs: 148 active+clean; 6.5 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 1.1 KiB/s rd, 0 B/s wr, 948 op/s 2024-03-11T21:29:37.111 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:29:36 smithi092 ceph-mon[37563]: 7.a scrub ok 2024-03-11T21:29:37.111 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:29:36 smithi092 ceph-mon[37563]: 2.18 scrub starts 2024-03-11T21:29:37.111 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:29:36 smithi092 ceph-mon[37563]: 2.18 scrub ok 2024-03-11T21:29:37.111 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:29:36 smithi092 ceph-mon[37563]: from='mgr.14607 ' entity='mgr.y' 2024-03-11T21:29:37.111 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:29:36 smithi092 ceph-mon[37563]: from='mgr.14607 v1:172.21.15.92:0/1702717597' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T21:29:37.124 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:29:37.125 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:29:37.125 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:29:37.226 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:29:37.256 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:29:37.256 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:29:37.257 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:29:37.326 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:29:36 smithi097 ceph-mon[33317]: pgmap v137: 148 pgs: 148 active+clean; 6.5 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 1.1 KiB/s rd, 0 B/s wr, 948 op/s 2024-03-11T21:29:37.326 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:29:36 smithi097 ceph-mon[33317]: 7.a scrub ok 2024-03-11T21:29:37.327 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:29:36 smithi097 ceph-mon[33317]: 2.18 scrub starts 2024-03-11T21:29:37.327 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:29:36 smithi097 ceph-mon[33317]: 2.18 scrub ok 2024-03-11T21:29:37.327 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:29:36 smithi097 ceph-mon[33317]: from='mgr.14607 ' entity='mgr.y' 2024-03-11T21:29:37.327 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:29:36 smithi097 ceph-mon[33317]: from='mgr.14607 v1:172.21.15.92:0/1702717597' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T21:29:37.357 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:29:37.388 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:29:37.389 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:29:37.389 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:29:37.490 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:29:37.524 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:29:37.524 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:29:37.525 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:29:37.625 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:29:37.665 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:29:37.666 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:29:37.666 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:29:37.767 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:29:37.804 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:29:37.805 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:29:37.805 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:29:37.906 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:29:37.951 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:29:37.952 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:29:37.952 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:29:38.053 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:29:38.081 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:29:38.082 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:29:38.082 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:29:38.183 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:29:38.203 INFO:journalctl@ceph.mgr.y.smithi092.stdout:Mar 11 21:29:37 smithi092 ceph-00db0016-dfed-11ee-95c8-87774f69a715-mgr-y[29801]: ::ffff:172.21.15.97 - - [11/Mar/2024:21:29:37] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-11T21:29:38.204 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:29:37 smithi092 ceph-mon[37563]: 5.2 deep-scrub starts 2024-03-11T21:29:38.204 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:29:37 smithi092 ceph-mon[37563]: 5.2 deep-scrub ok 2024-03-11T21:29:38.204 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:29:37 smithi092 ceph-mon[37563]: 5.b scrub starts 2024-03-11T21:29:38.204 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:29:37 smithi092 ceph-mon[37563]: 5.b scrub ok 2024-03-11T21:29:38.204 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:29:37 smithi092 ceph-mon[29572]: 5.2 deep-scrub starts 2024-03-11T21:29:38.204 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:29:37 smithi092 ceph-mon[29572]: 5.2 deep-scrub ok 2024-03-11T21:29:38.204 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:29:37 smithi092 ceph-mon[29572]: 5.b scrub starts 2024-03-11T21:29:38.204 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:29:37 smithi092 ceph-mon[29572]: 5.b scrub ok 2024-03-11T21:29:38.222 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:29:38.223 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:29:38.223 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:29:38.324 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:29:38.329 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:29:37 smithi097 ceph-mon[33317]: 5.2 deep-scrub starts 2024-03-11T21:29:38.330 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:29:37 smithi097 ceph-mon[33317]: 5.2 deep-scrub ok 2024-03-11T21:29:38.330 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:29:37 smithi097 ceph-mon[33317]: 5.b scrub starts 2024-03-11T21:29:38.330 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:29:37 smithi097 ceph-mon[33317]: 5.b scrub ok 2024-03-11T21:29:38.383 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:29:38.384 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:29:38.384 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:29:38.484 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:29:38.512 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:29:38.513 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:29:38.513 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:29:38.614 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:29:38.649 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:29:38.649 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:29:38.650 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:29:38.750 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:29:38.778 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:29:38.779 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:29:38.780 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:29:38.880 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:29:38.933 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:29:38.934 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:29:38.934 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:29:39.035 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:29:39.072 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:29:38 smithi092 ceph-mon[29572]: pgmap v138: 148 pgs: 148 active+clean; 6.4 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 895 op/s 2024-03-11T21:29:39.073 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:29:38 smithi092 ceph-mon[29572]: 4.9 scrub starts 2024-03-11T21:29:39.073 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:29:38 smithi092 ceph-mon[29572]: 4.9 scrub ok 2024-03-11T21:29:39.073 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:29:38 smithi092 ceph-mon[29572]: 7.4 scrub starts 2024-03-11T21:29:39.073 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:29:38 smithi092 ceph-mon[37563]: pgmap v138: 148 pgs: 148 active+clean; 6.4 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 895 op/s 2024-03-11T21:29:39.073 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:29:38 smithi092 ceph-mon[37563]: 4.9 scrub starts 2024-03-11T21:29:39.073 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:29:38 smithi092 ceph-mon[37563]: 4.9 scrub ok 2024-03-11T21:29:39.073 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:29:38 smithi092 ceph-mon[37563]: 7.4 scrub starts 2024-03-11T21:29:39.076 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:29:39.076 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:29:39.077 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:29:39.177 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:29:39.205 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:29:39.207 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:29:39.208 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:29:39.309 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:29:39.326 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:29:38 smithi097 ceph-mon[33317]: pgmap v138: 148 pgs: 148 active+clean; 6.4 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 895 op/s 2024-03-11T21:29:39.326 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:29:38 smithi097 ceph-mon[33317]: 4.9 scrub starts 2024-03-11T21:29:39.326 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:29:38 smithi097 ceph-mon[33317]: 4.9 scrub ok 2024-03-11T21:29:39.327 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:29:38 smithi097 ceph-mon[33317]: 7.4 scrub starts 2024-03-11T21:29:39.336 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:29:39.337 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:29:39.337 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:29:39.438 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:29:39.503 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:29:39.503 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:29:39.503 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:29:39.604 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:29:39.647 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:29:39.648 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:29:39.648 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:29:39.749 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:29:39.785 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:29:39.786 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:29:39.786 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:29:39.887 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:29:39.960 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:29:39.961 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:29:39.961 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:29:40.062 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:29:40.091 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:29:40.092 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:29:40.092 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:29:40.193 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:29:40.205 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:29:39 smithi092 ceph-mon[29572]: 5.5 scrub starts 2024-03-11T21:29:40.206 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:29:39 smithi092 ceph-mon[29572]: 5.5 scrub ok 2024-03-11T21:29:40.206 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:29:39 smithi092 ceph-mon[29572]: 2.1d scrub starts 2024-03-11T21:29:40.206 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:29:39 smithi092 ceph-mon[29572]: 2.1d scrub ok 2024-03-11T21:29:40.206 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:29:39 smithi092 ceph-mon[37563]: 5.5 scrub starts 2024-03-11T21:29:40.206 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:29:39 smithi092 ceph-mon[37563]: 5.5 scrub ok 2024-03-11T21:29:40.206 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:29:39 smithi092 ceph-mon[37563]: 2.1d scrub starts 2024-03-11T21:29:40.206 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:29:39 smithi092 ceph-mon[37563]: 2.1d scrub ok 2024-03-11T21:29:40.220 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:29:40.220 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:29:40.221 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:29:40.259 INFO:teuthology.orchestra.run.smithi092.stdout:osd.0: {} 2024-03-11T21:29:40.259 INFO:teuthology.orchestra.run.smithi092.stdout:osd.1: {} 2024-03-11T21:29:40.286 INFO:teuthology.orchestra.run.smithi092.stdout:osd.2: {} 2024-03-11T21:29:40.321 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:29:40.326 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:29:39 smithi097 ceph-mon[33317]: 5.5 scrub starts 2024-03-11T21:29:40.326 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:29:39 smithi097 ceph-mon[33317]: 5.5 scrub ok 2024-03-11T21:29:40.326 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:29:39 smithi097 ceph-mon[33317]: 2.1d scrub starts 2024-03-11T21:29:40.326 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:29:39 smithi097 ceph-mon[33317]: 2.1d scrub ok 2024-03-11T21:29:40.332 INFO:teuthology.orchestra.run.smithi092.stdout:osd.3: {} 2024-03-11T21:29:40.378 INFO:teuthology.orchestra.run.smithi092.stdout:osd.4: {} 2024-03-11T21:29:40.381 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:29:40.382 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:29:40.382 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:29:40.417 INFO:teuthology.orchestra.run.smithi092.stdout:osd.5: {} 2024-03-11T21:29:40.456 INFO:teuthology.orchestra.run.smithi092.stdout:osd.6: {} 2024-03-11T21:29:40.483 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:29:40.494 INFO:teuthology.orchestra.run.smithi092.stdout:osd.7: {} 2024-03-11T21:29:40.497 INFO:teuthology.orchestra.run.smithi092.stderr:osd.0: osd_enable_op_tracker = '' 2024-03-11T21:29:40.497 INFO:teuthology.orchestra.run.smithi092.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-11T21:29:40.497 INFO:teuthology.orchestra.run.smithi092.stderr:osd.2: osd_enable_op_tracker = '' 2024-03-11T21:29:40.497 INFO:teuthology.orchestra.run.smithi092.stderr:osd.3: osd_enable_op_tracker = '' 2024-03-11T21:29:40.497 INFO:teuthology.orchestra.run.smithi092.stderr:osd.4: osd_enable_op_tracker = '' 2024-03-11T21:29:40.497 INFO:teuthology.orchestra.run.smithi092.stderr:osd.5: osd_enable_op_tracker = '' 2024-03-11T21:29:40.497 INFO:teuthology.orchestra.run.smithi092.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-11T21:29:40.497 INFO:teuthology.orchestra.run.smithi092.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-11T21:29:40.528 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:29:40.528 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:29:40.528 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:29:40.629 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:29:40.673 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:29:40.673 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:29:40.674 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:29:40.774 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:29:40.801 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:29:40.802 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:29:40.802 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:29:40.903 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:29:40.927 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:29:40.928 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:29:40.928 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:29:41.029 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:29:41.079 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:29:41.080 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:29:41.080 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:29:41.181 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:29:41.196 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:29:40 smithi092 ceph-mon[29572]: pgmap v139: 148 pgs: 1 active+clean+scrubbing+deep, 147 active+clean; 6.2 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 895 op/s 2024-03-11T21:29:41.196 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:29:40 smithi092 ceph-mon[29572]: 7.4 scrub ok 2024-03-11T21:29:41.196 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:29:40 smithi092 ceph-mon[29572]: 5.18 scrub starts 2024-03-11T21:29:41.196 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:29:40 smithi092 ceph-mon[29572]: 5.18 scrub ok 2024-03-11T21:29:41.196 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:29:40 smithi092 ceph-mon[29572]: 6.0 deep-scrub starts 2024-03-11T21:29:41.196 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:29:40 smithi092 ceph-mon[29572]: 6.0 deep-scrub ok 2024-03-11T21:29:41.196 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:29:40 smithi092 ceph-mon[29572]: 7.2 scrub starts 2024-03-11T21:29:41.197 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:29:40 smithi092 ceph-mon[29572]: 5.11 deep-scrub starts 2024-03-11T21:29:41.197 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:29:40 smithi092 ceph-mon[29572]: 5.11 deep-scrub ok 2024-03-11T21:29:41.197 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:29:40 smithi092 ceph-mon[29572]: from='client.? v1:172.21.15.92:0/149656503' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-11T21:29:41.197 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:29:40 smithi092 ceph-mon[37563]: pgmap v139: 148 pgs: 1 active+clean+scrubbing+deep, 147 active+clean; 6.2 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 895 op/s 2024-03-11T21:29:41.197 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:29:40 smithi092 ceph-mon[37563]: 7.4 scrub ok 2024-03-11T21:29:41.197 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:29:40 smithi092 ceph-mon[37563]: 5.18 scrub starts 2024-03-11T21:29:41.197 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:29:40 smithi092 ceph-mon[37563]: 5.18 scrub ok 2024-03-11T21:29:41.197 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:29:40 smithi092 ceph-mon[37563]: 6.0 deep-scrub starts 2024-03-11T21:29:41.197 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:29:40 smithi092 ceph-mon[37563]: 6.0 deep-scrub ok 2024-03-11T21:29:41.197 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:29:40 smithi092 ceph-mon[37563]: 7.2 scrub starts 2024-03-11T21:29:41.197 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:29:40 smithi092 ceph-mon[37563]: 5.11 deep-scrub starts 2024-03-11T21:29:41.197 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:29:40 smithi092 ceph-mon[37563]: 5.11 deep-scrub ok 2024-03-11T21:29:41.198 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:29:40 smithi092 ceph-mon[37563]: from='client.? v1:172.21.15.92:0/149656503' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-11T21:29:41.207 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:29:41.208 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:29:41.208 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:29:41.309 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:29:41.326 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:29:40 smithi097 ceph-mon[33317]: pgmap v139: 148 pgs: 1 active+clean+scrubbing+deep, 147 active+clean; 6.2 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 895 op/s 2024-03-11T21:29:41.326 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:29:40 smithi097 ceph-mon[33317]: 7.4 scrub ok 2024-03-11T21:29:41.326 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:29:40 smithi097 ceph-mon[33317]: 5.18 scrub starts 2024-03-11T21:29:41.327 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:29:40 smithi097 ceph-mon[33317]: 5.18 scrub ok 2024-03-11T21:29:41.327 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:29:40 smithi097 ceph-mon[33317]: 6.0 deep-scrub starts 2024-03-11T21:29:41.327 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:29:40 smithi097 ceph-mon[33317]: 6.0 deep-scrub ok 2024-03-11T21:29:41.327 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:29:40 smithi097 ceph-mon[33317]: 7.2 scrub starts 2024-03-11T21:29:41.327 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:29:40 smithi097 ceph-mon[33317]: 5.11 deep-scrub starts 2024-03-11T21:29:41.327 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:29:40 smithi097 ceph-mon[33317]: 5.11 deep-scrub ok 2024-03-11T21:29:41.327 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:29:40 smithi097 ceph-mon[33317]: from='client.? v1:172.21.15.92:0/149656503' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-11T21:29:41.335 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:29:41.335 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:29:41.336 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:29:41.436 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:29:41.463 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:29:41.464 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:29:41.464 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:29:41.548 INFO:teuthology.orchestra.run.smithi092.stdout:ERROR: (22) Invalid argument 2024-03-11T21:29:41.549 INFO:teuthology.orchestra.run.smithi092.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-11T21:29:41.565 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:29:41.596 INFO:teuthology.orchestra.run.smithi092.stdout: 2024-03-11T21:29:41.603 INFO:teuthology.orchestra.run.smithi092.stderr:dumped all 2024-03-11T21:29:41.622 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:29:41.622 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:29:41.623 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:29:41.723 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:29:41.763 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:29:41.763 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:29:41.764 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:29:41.864 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:29:41.894 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:29:41.895 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:29:41.895 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:29:41.996 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:29:42.024 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:29:42.026 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:29:42.027 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:29:42.095 INFO:teuthology.orchestra.run.smithi092.stderr:Inferring config /var/lib/ceph/00db0016-dfed-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T21:29:42.110 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:29:41 smithi092 ceph-mon[29572]: 4.1b scrub starts 2024-03-11T21:29:42.111 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:29:41 smithi092 ceph-mon[29572]: 4.1b scrub ok 2024-03-11T21:29:42.111 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:29:41 smithi092 ceph-mon[29572]: 6.12 scrub starts 2024-03-11T21:29:42.111 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:29:41 smithi092 ceph-mon[29572]: 6.12 scrub ok 2024-03-11T21:29:42.111 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:29:41 smithi092 ceph-mon[37563]: 4.1b scrub starts 2024-03-11T21:29:42.111 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:29:41 smithi092 ceph-mon[37563]: 4.1b scrub ok 2024-03-11T21:29:42.111 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:29:41 smithi092 ceph-mon[37563]: 6.12 scrub starts 2024-03-11T21:29:42.111 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:29:41 smithi092 ceph-mon[37563]: 6.12 scrub ok 2024-03-11T21:29:42.128 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:29:42.159 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:29:42.160 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:29:42.160 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:29:42.261 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:29:42.288 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:29:42.289 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:29:42.289 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:29:42.326 INFO:journalctl@ceph.iscsi.iscsi.a.smithi097.stdout:Mar 11 21:29:42 smithi097 ceph-00db0016-dfed-11ee-95c8-87774f69a715-iscsi-iscsi-a[60943]: debug there is no tcmu-runner data available 2024-03-11T21:29:42.327 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:29:41 smithi097 ceph-mon[33317]: 4.1b scrub starts 2024-03-11T21:29:42.327 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:29:41 smithi097 ceph-mon[33317]: 4.1b scrub ok 2024-03-11T21:29:42.327 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:29:41 smithi097 ceph-mon[33317]: 6.12 scrub starts 2024-03-11T21:29:42.327 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:29:41 smithi097 ceph-mon[33317]: 6.12 scrub ok 2024-03-11T21:29:42.390 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:29:42.439 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:29:42.439 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:29:42.440 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:29:42.540 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:29:42.574 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:29:42.574 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:29:42.574 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:29:42.675 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:29:42.712 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:29:42.713 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:29:42.714 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:29:42.814 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:29:42.843 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:29:42.844 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:29:42.844 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:29:42.945 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:29:42.976 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:29:42.976 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:29:42.977 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:29:43.077 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:29:43.105 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:29:43.105 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:29:43.106 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:29:43.206 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:29:43.234 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:29:43.234 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:29:43.235 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:29:43.239 DEBUG:teuthology.orchestra.run.smithi092:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid 00db0016-dfed-11ee-95c8-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-03-11T21:29:43.276 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:29:42 smithi092 ceph-mon[29572]: pgmap v140: 148 pgs: 1 active+clean+scrubbing+deep, 147 active+clean; 6.2 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 863 B/s rd, 0 B/s wr, 755 op/s 2024-03-11T21:29:43.276 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:29:42 smithi092 ceph-mon[29572]: from='client.24838 v1:172.21.15.92:0/2269699080' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-11T21:29:43.276 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:29:42 smithi092 ceph-mon[29572]: 6.1a scrub starts 2024-03-11T21:29:43.276 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:29:42 smithi092 ceph-mon[29572]: 6.1a scrub ok 2024-03-11T21:29:43.276 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:29:42 smithi092 ceph-mon[29572]: 6.10 scrub starts 2024-03-11T21:29:43.276 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:29:42 smithi092 ceph-mon[29572]: 6.10 scrub ok 2024-03-11T21:29:43.276 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:29:42 smithi092 ceph-mon[29572]: 7.2 scrub ok 2024-03-11T21:29:43.277 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:29:42 smithi092 ceph-mon[29572]: 4.15 deep-scrub starts 2024-03-11T21:29:43.277 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:29:42 smithi092 ceph-mon[29572]: 4.15 deep-scrub ok 2024-03-11T21:29:43.277 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:29:42 smithi092 ceph-mon[29572]: 6.3 deep-scrub starts 2024-03-11T21:29:43.277 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:29:42 smithi092 ceph-mon[29572]: 6.3 deep-scrub ok 2024-03-11T21:29:43.277 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:29:42 smithi092 ceph-mon[37563]: pgmap v140: 148 pgs: 1 active+clean+scrubbing+deep, 147 active+clean; 6.2 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 863 B/s rd, 0 B/s wr, 755 op/s 2024-03-11T21:29:43.277 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:29:42 smithi092 ceph-mon[37563]: from='client.24838 v1:172.21.15.92:0/2269699080' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-11T21:29:43.277 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:29:42 smithi092 ceph-mon[37563]: 6.1a scrub starts 2024-03-11T21:29:43.277 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:29:42 smithi092 ceph-mon[37563]: 6.1a scrub ok 2024-03-11T21:29:43.277 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:29:42 smithi092 ceph-mon[37563]: 6.10 scrub starts 2024-03-11T21:29:43.277 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:29:42 smithi092 ceph-mon[37563]: 6.10 scrub ok 2024-03-11T21:29:43.278 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:29:42 smithi092 ceph-mon[37563]: 7.2 scrub ok 2024-03-11T21:29:43.278 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:29:42 smithi092 ceph-mon[37563]: 4.15 deep-scrub starts 2024-03-11T21:29:43.278 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:29:42 smithi092 ceph-mon[37563]: 4.15 deep-scrub ok 2024-03-11T21:29:43.278 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:29:42 smithi092 ceph-mon[37563]: 6.3 deep-scrub starts 2024-03-11T21:29:43.278 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:29:42 smithi092 ceph-mon[37563]: 6.3 deep-scrub ok 2024-03-11T21:29:43.326 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:29:42 smithi097 ceph-mon[33317]: pgmap v140: 148 pgs: 1 active+clean+scrubbing+deep, 147 active+clean; 6.2 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 863 B/s rd, 0 B/s wr, 755 op/s 2024-03-11T21:29:43.327 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:29:42 smithi097 ceph-mon[33317]: from='client.24838 v1:172.21.15.92:0/2269699080' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-11T21:29:43.327 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:29:42 smithi097 ceph-mon[33317]: 6.1a scrub starts 2024-03-11T21:29:43.327 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:29:42 smithi097 ceph-mon[33317]: 6.1a scrub ok 2024-03-11T21:29:43.327 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:29:42 smithi097 ceph-mon[33317]: 6.10 scrub starts 2024-03-11T21:29:43.327 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:29:42 smithi097 ceph-mon[33317]: 6.10 scrub ok 2024-03-11T21:29:43.327 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:29:42 smithi097 ceph-mon[33317]: 7.2 scrub ok 2024-03-11T21:29:43.327 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:29:42 smithi097 ceph-mon[33317]: 4.15 deep-scrub starts 2024-03-11T21:29:43.327 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:29:42 smithi097 ceph-mon[33317]: 4.15 deep-scrub ok 2024-03-11T21:29:43.327 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:29:42 smithi097 ceph-mon[33317]: 6.3 deep-scrub starts 2024-03-11T21:29:43.327 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:29:42 smithi097 ceph-mon[33317]: 6.3 deep-scrub ok 2024-03-11T21:29:43.336 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:29:43.384 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:29:43.385 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:29:43.385 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:29:43.486 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:29:43.514 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:29:43.515 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:29:43.515 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:29:43.616 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:29:43.644 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:29:43.645 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:29:43.645 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:29:43.746 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:29:43.781 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:29:43.781 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:29:43.782 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:29:43.882 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:29:43.906 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:29:43.907 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:29:43.907 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:29:44.008 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:29:44.039 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:29:44.040 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:29:44.040 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:29:44.141 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:29:44.170 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:29:44.171 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:29:44.171 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:29:44.272 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:29:44.299 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:29:44.300 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:29:44.300 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:29:44.326 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:29:43 smithi097 ceph-mon[33317]: from='client.24469 v1:172.21.15.97:0/404208310' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:29:44.327 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:29:43 smithi097 ceph-mon[33317]: 2.19 scrub starts 2024-03-11T21:29:44.327 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:29:43 smithi097 ceph-mon[33317]: 2.19 scrub ok 2024-03-11T21:29:44.327 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:29:43 smithi097 ceph-mon[33317]: 2.16 scrub starts 2024-03-11T21:29:44.327 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:29:43 smithi097 ceph-mon[33317]: 2.16 scrub ok 2024-03-11T21:29:44.327 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:29:43 smithi097 ceph-mon[33317]: 2.15 scrub starts 2024-03-11T21:29:44.327 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:29:43 smithi097 ceph-mon[33317]: 2.15 scrub ok 2024-03-11T21:29:44.359 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:29:43 smithi092 ceph-mon[29572]: from='client.24469 v1:172.21.15.97:0/404208310' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:29:44.359 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:29:43 smithi092 ceph-mon[29572]: 2.19 scrub starts 2024-03-11T21:29:44.359 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:29:43 smithi092 ceph-mon[29572]: 2.19 scrub ok 2024-03-11T21:29:44.359 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:29:43 smithi092 ceph-mon[29572]: 2.16 scrub starts 2024-03-11T21:29:44.360 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:29:43 smithi092 ceph-mon[29572]: 2.16 scrub ok 2024-03-11T21:29:44.360 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:29:43 smithi092 ceph-mon[29572]: 2.15 scrub starts 2024-03-11T21:29:44.360 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:29:43 smithi092 ceph-mon[29572]: 2.15 scrub ok 2024-03-11T21:29:44.360 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:29:43 smithi092 ceph-mon[37563]: from='client.24469 v1:172.21.15.97:0/404208310' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:29:44.360 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:29:43 smithi092 ceph-mon[37563]: 2.19 scrub starts 2024-03-11T21:29:44.360 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:29:43 smithi092 ceph-mon[37563]: 2.19 scrub ok 2024-03-11T21:29:44.360 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:29:43 smithi092 ceph-mon[37563]: 2.16 scrub starts 2024-03-11T21:29:44.360 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:29:43 smithi092 ceph-mon[37563]: 2.16 scrub ok 2024-03-11T21:29:44.360 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:29:43 smithi092 ceph-mon[37563]: 2.15 scrub starts 2024-03-11T21:29:44.360 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:29:43 smithi092 ceph-mon[37563]: 2.15 scrub ok 2024-03-11T21:29:44.402 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:29:44.404 DEBUG:teuthology.orchestra.run.smithi097:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid 00db0016-dfed-11ee-95c8-87774f69a715 -- ceph daemon osd.4 dump_ops_in_flight 2024-03-11T21:29:44.435 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:29:44.436 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:29:44.436 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:29:44.537 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:29:44.568 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:29:44.569 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:29:44.569 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:29:44.670 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:29:44.702 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:29:44.703 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:29:44.704 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:29:44.735 INFO:teuthology.orchestra.run.smithi097.stderr:Inferring config /var/lib/ceph/00db0016-dfed-11ee-95c8-87774f69a715/mon.b/config 2024-03-11T21:29:44.805 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:29:44.838 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:29:44.838 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:29:44.839 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:29:44.940 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:29:44.969 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:29:44.969 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:29:44.969 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:29:45.070 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:29:45.107 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:29:45.107 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:29:45.108 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:29:45.175 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:29:44 smithi097 ceph-mon[33317]: pgmap v141: 148 pgs: 1 active+clean+scrubbing+deep, 147 active+clean; 6.0 GiB data, 19 GiB used, 697 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.08k op/s 2024-03-11T21:29:45.175 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:29:44 smithi097 ceph-mon[33317]: 6.18 scrub starts 2024-03-11T21:29:45.175 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:29:44 smithi097 ceph-mon[33317]: 6.18 scrub ok 2024-03-11T21:29:45.175 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:29:44 smithi097 ceph-mon[33317]: 4.1a scrub starts 2024-03-11T21:29:45.175 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:29:44 smithi097 ceph-mon[33317]: 4.1a scrub ok 2024-03-11T21:29:45.209 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:29:45.253 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:29:45.254 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:29:45.255 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:29:45.259 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:29:44 smithi092 ceph-mon[29572]: pgmap v141: 148 pgs: 1 active+clean+scrubbing+deep, 147 active+clean; 6.0 GiB data, 19 GiB used, 697 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.08k op/s 2024-03-11T21:29:45.259 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:29:44 smithi092 ceph-mon[29572]: 6.18 scrub starts 2024-03-11T21:29:45.259 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:29:44 smithi092 ceph-mon[29572]: 6.18 scrub ok 2024-03-11T21:29:45.260 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:29:44 smithi092 ceph-mon[29572]: 4.1a scrub starts 2024-03-11T21:29:45.260 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:29:44 smithi092 ceph-mon[29572]: 4.1a scrub ok 2024-03-11T21:29:45.260 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:29:44 smithi092 ceph-mon[37563]: pgmap v141: 148 pgs: 1 active+clean+scrubbing+deep, 147 active+clean; 6.0 GiB data, 19 GiB used, 697 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.08k op/s 2024-03-11T21:29:45.260 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:29:44 smithi092 ceph-mon[37563]: 6.18 scrub starts 2024-03-11T21:29:45.260 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:29:44 smithi092 ceph-mon[37563]: 6.18 scrub ok 2024-03-11T21:29:45.260 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:29:44 smithi092 ceph-mon[37563]: 4.1a scrub starts 2024-03-11T21:29:45.260 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:29:44 smithi092 ceph-mon[37563]: 4.1a scrub ok 2024-03-11T21:29:45.356 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:29:45.392 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:29:45.392 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:29:45.393 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:29:45.494 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:29:45.527 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:29:45.528 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:29:45.528 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:29:45.629 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:29:45.656 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:29:45.656 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:29:45.657 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:29:45.758 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:29:45.809 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:29:45.810 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:29:45.810 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:29:45.911 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:29:45.941 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:29:45.941 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:29:45.941 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:29:46.042 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:29:46.083 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:29:46.084 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:29:46.084 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:29:46.146 INFO:teuthology.orchestra.run.smithi092.stdout:{"pg_ready":true,"pg_map":{"version":140,"stamp":"2024-03-11T21:29:41.026525+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":6667180480,"num_objects":101923,"num_object_clones":0,"num_object_copies":305769,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":101923,"num_whiteouts":0,"num_read":1168,"num_read_kb":1014,"num_write":142708,"num_write_kb":7804574,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":21,"num_bytes_recovered":1771324,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":90305,"ondisk_log_size":90305,"up":444,"acting":444,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":444,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":749961216,"kb_used":20320360,"kb_used_data":19640296,"kb_used_omap":12,"kb_used_meta":679667,"kb_avail":729640856,"statfs":{"total":767960285184,"available":747152236544,"internally_reserved":0,"allocated":20111663104,"data_stored":20108535000,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12752,"internal_metadata":695979568},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,4,7,6,8,9,14,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-586416128,"num_objects":-8948,"num_object_clones":0,"num_object_copies":-26844,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-8948,"num_whiteouts":0,"num_read":10,"num_read_kb":10,"num_write":8948,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"11.853312"},"pg_stats":[{"pgid":"6.1b","version":"0'0","reported_seq":129,"reported_epoch":124,"state":"active+clean","last_fresh":"2024-03-11T21:29:29.180739+0000","last_change":"2024-03-11T21:28:16.650491+0000","last_active":"2024-03-11T21:29:29.180739+0000","last_peered":"2024-03-11T21:29:29.180739+0000","last_clean":"2024-03-11T21:29:29.180739+0000","last_became_active":"2024-03-11T21:25:45.302931+0000","last_became_peered":"2024-03-11T21:25:45.302931+0000","last_unstale":"2024-03-11T21:29:29.180739+0000","last_undegraded":"2024-03-11T21:29:29.180739+0000","last_fullsized":"2024-03-11T21:29:29.180739+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:28:16.650384+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:43.559452+0000","last_clean_scrub_stamp":"2024-03-11T21:28:16.650384+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:29:44.614978+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":126,"reported_epoch":124,"state":"active+clean","last_fresh":"2024-03-11T21:29:29.171468+0000","last_change":"2024-03-11T21:28:16.734432+0000","last_active":"2024-03-11T21:29:29.171468+0000","last_peered":"2024-03-11T21:29:29.171468+0000","last_clean":"2024-03-11T21:29:29.171468+0000","last_became_active":"2024-03-11T21:25:42.578893+0000","last_became_peered":"2024-03-11T21:25:42.578893+0000","last_unstale":"2024-03-11T21:29:29.171468+0000","last_undegraded":"2024-03-11T21:29:29.171468+0000","last_fullsized":"2024-03-11T21:29:29.171468+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:28:16.734315+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:41.544550+0000","last_clean_scrub_stamp":"2024-03-11T21:28:16.734315+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1f","version":"0'0","reported_seq":142,"reported_epoch":124,"state":"active+clean","last_fresh":"2024-03-11T21:29:29.173457+0000","last_change":"2024-03-11T21:28:14.483927+0000","last_active":"2024-03-11T21:29:29.173457+0000","last_peered":"2024-03-11T21:29:29.173457+0000","last_clean":"2024-03-11T21:29:29.173457+0000","last_became_active":"2024-03-11T21:25:38.540510+0000","last_became_peered":"2024-03-11T21:25:38.540510+0000","last_unstale":"2024-03-11T21:29:29.173457+0000","last_undegraded":"2024-03-11T21:29:29.173457+0000","last_fullsized":"2024-03-11T21:29:29.173457+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:28:14.483813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:37.512018+0000","last_clean_scrub_stamp":"2024-03-11T21:28:14.483813+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,4],"acting":[0,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.19","version":"74'15","reported_seq":162,"reported_epoch":124,"state":"active+clean","last_fresh":"2024-03-11T21:29:29.184572+0000","last_change":"2024-03-11T21:28:37.436744+0000","last_active":"2024-03-11T21:29:29.184572+0000","last_peered":"2024-03-11T21:29:29.184572+0000","last_clean":"2024-03-11T21:29:29.184572+0000","last_became_active":"2024-03-11T21:25:40.554207+0000","last_became_peered":"2024-03-11T21:25:40.554207+0000","last_unstale":"2024-03-11T21:29:29.184572+0000","last_undegraded":"2024-03-11T21:29:29.184572+0000","last_fullsized":"2024-03-11T21:29:29.184572+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'15","last_scrub_stamp":"2024-03-11T21:28:37.436520+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:39.525554+0000","last_clean_scrub_stamp":"2024-03-11T21:28:37.436520+0000","objects_scrubbed":7,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:29:47.196615+0000","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.1a","version":"0'0","reported_seq":122,"reported_epoch":124,"state":"active+clean","last_fresh":"2024-03-11T21:29:29.171754+0000","last_change":"2024-03-11T21:28:10.634383+0000","last_active":"2024-03-11T21:29:29.171754+0000","last_peered":"2024-03-11T21:29:29.171754+0000","last_clean":"2024-03-11T21:29:29.171754+0000","last_became_active":"2024-03-11T21:25:44.590923+0000","last_became_peered":"2024-03-11T21:25:44.590923+0000","last_unstale":"2024-03-11T21:29:29.171754+0000","last_undegraded":"2024-03-11T21:29:29.171754+0000","last_fullsized":"2024-03-11T21:29:29.171754+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:28:10.634303+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:43.559452+0000","last_clean_scrub_stamp":"2024-03-11T21:28:10.634303+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,1],"acting":[4,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":134,"reported_epoch":124,"state":"active+clean","last_fresh":"2024-03-11T21:29:29.177318+0000","last_change":"2024-03-11T21:29:11.409331+0000","last_active":"2024-03-11T21:29:29.177318+0000","last_peered":"2024-03-11T21:29:29.177318+0000","last_clean":"2024-03-11T21:29:29.177318+0000","last_became_active":"2024-03-11T21:25:42.578424+0000","last_became_peered":"2024-03-11T21:25:42.578424+0000","last_unstale":"2024-03-11T21:29:29.177318+0000","last_undegraded":"2024-03-11T21:29:29.177318+0000","last_fullsized":"2024-03-11T21:29:29.177318+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:29:11.408737+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:41.544550+0000","last_clean_scrub_stamp":"2024-03-11T21:29:11.408737+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:30:33.475054+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,7],"acting":[1,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1e","version":"0'0","reported_seq":141,"reported_epoch":124,"state":"active+clean","last_fresh":"2024-03-11T21:29:29.182091+0000","last_change":"2024-03-11T21:28:07.604544+0000","last_active":"2024-03-11T21:29:29.182091+0000","last_peered":"2024-03-11T21:29:29.182091+0000","last_clean":"2024-03-11T21:29:29.182091+0000","last_became_active":"2024-03-11T21:25:38.537283+0000","last_became_peered":"2024-03-11T21:25:38.537283+0000","last_unstale":"2024-03-11T21:29:29.182091+0000","last_undegraded":"2024-03-11T21:29:29.182091+0000","last_fullsized":"2024-03-11T21:29:29.182091+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:28:07.604432+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:37.512018+0000","last_clean_scrub_stamp":"2024-03-11T21:28:07.604432+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.18","version":"74'9","reported_seq":154,"reported_epoch":124,"state":"active+clean","last_fresh":"2024-03-11T21:29:29.172453+0000","last_change":"2024-03-11T21:28:22.778031+0000","last_active":"2024-03-11T21:29:29.172453+0000","last_peered":"2024-03-11T21:29:29.172453+0000","last_clean":"2024-03-11T21:29:29.172453+0000","last_became_active":"2024-03-11T21:25:40.552216+0000","last_became_peered":"2024-03-11T21:25:40.552216+0000","last_unstale":"2024-03-11T21:29:29.172453+0000","last_undegraded":"2024-03-11T21:29:29.172453+0000","last_fullsized":"2024-03-11T21:29:29.172453+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'9","last_scrub_stamp":"2024-03-11T21:28:22.777781+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:39.525554+0000","last_clean_scrub_stamp":"2024-03-11T21:28:22.777781+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:29:29.319933+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":129,"reported_epoch":124,"state":"active+clean","last_fresh":"2024-03-11T21:29:29.182309+0000","last_change":"2024-03-11T21:28:34.755710+0000","last_active":"2024-03-11T21:29:29.182309+0000","last_peered":"2024-03-11T21:29:29.182309+0000","last_clean":"2024-03-11T21:29:29.182309+0000","last_became_active":"2024-03-11T21:25:44.590234+0000","last_became_peered":"2024-03-11T21:25:44.590234+0000","last_unstale":"2024-03-11T21:29:29.182309+0000","last_undegraded":"2024-03-11T21:29:29.182309+0000","last_fullsized":"2024-03-11T21:29:29.182309+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:28:34.755583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:43.559452+0000","last_clean_scrub_stamp":"2024-03-11T21:28:34.755583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:29:41.673503+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3],"acting":[5,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":133,"reported_epoch":124,"state":"active+clean","last_fresh":"2024-03-11T21:29:29.175398+0000","last_change":"2024-03-11T21:29:21.533307+0000","last_active":"2024-03-11T21:29:29.175398+0000","last_peered":"2024-03-11T21:29:29.175398+0000","last_clean":"2024-03-11T21:29:29.175398+0000","last_became_active":"2024-03-11T21:25:42.580533+0000","last_became_peered":"2024-03-11T21:25:42.580533+0000","last_unstale":"2024-03-11T21:29:29.175398+0000","last_undegraded":"2024-03-11T21:29:29.175398+0000","last_fullsized":"2024-03-11T21:29:29.175398+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:29:21.533216+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:41.544550+0000","last_clean_scrub_stamp":"2024-03-11T21:29:21.533216+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:30:49.291696+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.1d","version":"0'0","reported_seq":141,"reported_epoch":124,"state":"active+clean","last_fresh":"2024-03-11T21:29:29.175498+0000","last_change":"2024-03-11T21:28:11.681226+0000","last_active":"2024-03-11T21:29:29.175498+0000","last_peered":"2024-03-11T21:29:29.175498+0000","last_clean":"2024-03-11T21:29:29.175498+0000","last_became_active":"2024-03-11T21:25:38.542628+0000","last_became_peered":"2024-03-11T21:25:38.542628+0000","last_unstale":"2024-03-11T21:29:29.175498+0000","last_undegraded":"2024-03-11T21:29:29.175498+0000","last_fullsized":"2024-03-11T21:29:29.175498+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:28:11.681096+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:37.512018+0000","last_clean_scrub_stamp":"2024-03-11T21:28:11.681096+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,0],"acting":[7,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.1b","version":"74'5","reported_seq":140,"reported_epoch":124,"state":"active+clean","last_fresh":"2024-03-11T21:29:29.172300+0000","last_change":"2024-03-11T21:28:14.743893+0000","last_active":"2024-03-11T21:29:29.172300+0000","last_peered":"2024-03-11T21:29:29.172300+0000","last_clean":"2024-03-11T21:29:29.172300+0000","last_became_active":"2024-03-11T21:25:40.555826+0000","last_became_peered":"2024-03-11T21:25:40.555826+0000","last_unstale":"2024-03-11T21:29:29.172300+0000","last_undegraded":"2024-03-11T21:29:29.172300+0000","last_fullsized":"2024-03-11T21:29:29.172300+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'5","last_scrub_stamp":"2024-03-11T21:28:14.743756+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:39.525554+0000","last_clean_scrub_stamp":"2024-03-11T21:28:14.743756+0000","objects_scrubbed":2,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":11,"num_read_kb":7,"num_write":6,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,1],"acting":[4,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":122,"reported_epoch":124,"state":"active+clean","last_fresh":"2024-03-11T21:29:29.171847+0000","last_change":"2024-03-11T21:27:29.476225+0000","last_active":"2024-03-11T21:29:29.171847+0000","last_peered":"2024-03-11T21:29:29.171847+0000","last_clean":"2024-03-11T21:29:29.171847+0000","last_became_active":"2024-03-11T21:25:44.589786+0000","last_became_peered":"2024-03-11T21:25:44.589786+0000","last_unstale":"2024-03-11T21:29:29.171847+0000","last_undegraded":"2024-03-11T21:29:29.171847+0000","last_fullsized":"2024-03-11T21:29:29.171847+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:27:29.476144+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:43.559452+0000","last_clean_scrub_stamp":"2024-03-11T21:27:29.476144+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,7],"acting":[0,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":133,"reported_epoch":124,"state":"active+clean","last_fresh":"2024-03-11T21:29:29.182523+0000","last_change":"2024-03-11T21:29:15.903693+0000","last_active":"2024-03-11T21:29:29.182523+0000","last_peered":"2024-03-11T21:29:29.182523+0000","last_clean":"2024-03-11T21:29:29.182523+0000","last_became_active":"2024-03-11T21:25:42.573831+0000","last_became_peered":"2024-03-11T21:25:42.573831+0000","last_unstale":"2024-03-11T21:29:29.182523+0000","last_undegraded":"2024-03-11T21:29:29.182523+0000","last_fullsized":"2024-03-11T21:29:29.182523+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:29:15.903611+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:41.544550+0000","last_clean_scrub_stamp":"2024-03-11T21:29:15.903611+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:30:25.018388+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,7],"acting":[5,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1c","version":"0'0","reported_seq":141,"reported_epoch":124,"state":"active+clean","last_fresh":"2024-03-11T21:29:29.175330+0000","last_change":"2024-03-11T21:28:32.566552+0000","last_active":"2024-03-11T21:29:29.175330+0000","last_peered":"2024-03-11T21:29:29.175330+0000","last_clean":"2024-03-11T21:29:29.175330+0000","last_became_active":"2024-03-11T21:25:38.541364+0000","last_became_peered":"2024-03-11T21:25:38.541364+0000","last_unstale":"2024-03-11T21:29:29.175330+0000","last_undegraded":"2024-03-11T21:29:29.175330+0000","last_fullsized":"2024-03-11T21:29:29.175330+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:28:32.566437+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:28:32.566437+0000","last_clean_scrub_stamp":"2024-03-11T21:28:32.566437+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:29:50.714312+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.1a","version":"74'9","reported_seq":154,"reported_epoch":124,"state":"active+clean","last_fresh":"2024-03-11T21:29:29.171839+0000","last_change":"2024-03-11T21:28:18.725957+0000","last_active":"2024-03-11T21:29:29.171839+0000","last_peered":"2024-03-11T21:29:29.171839+0000","last_clean":"2024-03-11T21:29:29.171839+0000","last_became_active":"2024-03-11T21:25:40.556629+0000","last_became_peered":"2024-03-11T21:25:40.556629+0000","last_unstale":"2024-03-11T21:29:29.171839+0000","last_undegraded":"2024-03-11T21:29:29.171839+0000","last_fullsized":"2024-03-11T21:29:29.171839+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'9","last_scrub_stamp":"2024-03-11T21:28:18.725837+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:39.525554+0000","last_clean_scrub_stamp":"2024-03-11T21:28:18.725837+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0],"acting":[4,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1f","version":"0'0","reported_seq":129,"reported_epoch":124,"state":"active+clean","last_fresh":"2024-03-11T21:29:29.181251+0000","last_change":"2024-03-11T21:28:28.559558+0000","last_active":"2024-03-11T21:29:29.181251+0000","last_peered":"2024-03-11T21:29:29.181251+0000","last_clean":"2024-03-11T21:29:29.181251+0000","last_became_active":"2024-03-11T21:25:45.302637+0000","last_became_peered":"2024-03-11T21:25:45.302637+0000","last_unstale":"2024-03-11T21:29:29.181251+0000","last_undegraded":"2024-03-11T21:29:29.181251+0000","last_fullsized":"2024-03-11T21:29:29.181251+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:28:28.559406+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:43.559452+0000","last_clean_scrub_stamp":"2024-03-11T21:28:28.559406+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:29:36.695979+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1c","version":"0'0","reported_seq":134,"reported_epoch":124,"state":"active+clean","last_fresh":"2024-03-11T21:29:29.173887+0000","last_change":"2024-03-11T21:29:11.195663+0000","last_active":"2024-03-11T21:29:29.173887+0000","last_peered":"2024-03-11T21:29:29.173887+0000","last_clean":"2024-03-11T21:29:29.173887+0000","last_became_active":"2024-03-11T21:25:42.576064+0000","last_became_peered":"2024-03-11T21:25:42.576064+0000","last_unstale":"2024-03-11T21:29:29.173887+0000","last_undegraded":"2024-03-11T21:29:29.173887+0000","last_fullsized":"2024-03-11T21:29:29.173887+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:29:11.195539+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:41.544550+0000","last_clean_scrub_stamp":"2024-03-11T21:29:11.195539+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:30:26.525306+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1b","version":"0'0","reported_seq":141,"reported_epoch":124,"state":"active+clean","last_fresh":"2024-03-11T21:29:29.180948+0000","last_change":"2024-03-11T21:28:33.582952+0000","last_active":"2024-03-11T21:29:29.180948+0000","last_peered":"2024-03-11T21:29:29.180948+0000","last_clean":"2024-03-11T21:29:29.180948+0000","last_became_active":"2024-03-11T21:25:38.537520+0000","last_became_peered":"2024-03-11T21:25:38.537520+0000","last_unstale":"2024-03-11T21:29:29.180948+0000","last_undegraded":"2024-03-11T21:29:29.180948+0000","last_fullsized":"2024-03-11T21:29:29.180948+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:28:33.582712+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:28:33.582712+0000","last_clean_scrub_stamp":"2024-03-11T21:28:33.582712+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:29:44.595087+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1d","version":"74'12","reported_seq":160,"reported_epoch":124,"state":"active+clean","last_fresh":"2024-03-11T21:29:29.180954+0000","last_change":"2024-03-11T21:29:17.662843+0000","last_active":"2024-03-11T21:29:29.180954+0000","last_peered":"2024-03-11T21:29:29.180954+0000","last_clean":"2024-03-11T21:29:29.180954+0000","last_became_active":"2024-03-11T21:25:40.556330+0000","last_became_peered":"2024-03-11T21:25:40.556330+0000","last_unstale":"2024-03-11T21:29:29.180954+0000","last_undegraded":"2024-03-11T21:29:29.180954+0000","last_fullsized":"2024-03-11T21:29:29.180954+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'12","last_scrub_stamp":"2024-03-11T21:29:17.662708+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:39.525554+0000","last_clean_scrub_stamp":"2024-03-11T21:29:17.662708+0000","objects_scrubbed":5,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:30:40.617043+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":25,"num_read_kb":16,"num_write":14,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.1e","version":"0'0","reported_seq":122,"reported_epoch":124,"state":"active+clean","last_fresh":"2024-03-11T21:29:29.173063+0000","last_change":"2024-03-11T21:28:11.684315+0000","last_active":"2024-03-11T21:29:29.173063+0000","last_peered":"2024-03-11T21:29:29.173063+0000","last_clean":"2024-03-11T21:29:29.173063+0000","last_became_active":"2024-03-11T21:25:45.300355+0000","last_became_peered":"2024-03-11T21:25:45.300355+0000","last_unstale":"2024-03-11T21:29:29.173063+0000","last_undegraded":"2024-03-11T21:29:29.173063+0000","last_fullsized":"2024-03-11T21:29:29.173063+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:28:11.684174+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:43.559452+0000","last_clean_scrub_stamp":"2024-03-11T21:28:11.684174+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:29:32.796638+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5],"acting":[4,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1d","version":"0'0","reported_seq":134,"reported_epoch":124,"state":"active+clean","last_fresh":"2024-03-11T21:29:29.177734+0000","last_change":"2024-03-11T21:28:31.304566+0000","last_active":"2024-03-11T21:29:29.177734+0000","last_peered":"2024-03-11T21:29:29.177734+0000","last_clean":"2024-03-11T21:29:29.177734+0000","last_became_active":"2024-03-11T21:25:42.579407+0000","last_became_peered":"2024-03-11T21:25:42.579407+0000","last_unstale":"2024-03-11T21:29:29.177734+0000","last_undegraded":"2024-03-11T21:29:29.177734+0000","last_fullsized":"2024-03-11T21:29:29.177734+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:28:31.304120+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:27:09.199513+0000","last_clean_scrub_stamp":"2024-03-11T21:28:31.304120+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:29:45.672976+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1a","version":"0'0","reported_seq":142,"reported_epoch":124,"state":"active+clean","last_fresh":"2024-03-11T21:29:29.177750+0000","last_change":"2024-03-11T21:28:16.994328+0000","last_active":"2024-03-11T21:29:29.177750+0000","last_peered":"2024-03-11T21:29:29.177750+0000","last_clean":"2024-03-11T21:29:29.177750+0000","last_became_active":"2024-03-11T21:25:38.539725+0000","last_became_peered":"2024-03-11T21:25:38.539725+0000","last_unstale":"2024-03-11T21:29:29.177750+0000","last_undegraded":"2024-03-11T21:29:29.177750+0000","last_fullsized":"2024-03-11T21:29:29.177750+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:28:16.994251+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:37.512018+0000","last_clean_scrub_stamp":"2024-03-11T21:28:16.994251+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:29:45.088652+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.1c","version":"74'15","reported_seq":162,"reported_epoch":124,"state":"active+clean","last_fresh":"2024-03-11T21:29:29.180615+0000","last_change":"2024-03-11T21:28:35.104768+0000","last_active":"2024-03-11T21:29:29.180615+0000","last_peered":"2024-03-11T21:29:29.180615+0000","last_clean":"2024-03-11T21:29:29.180615+0000","last_became_active":"2024-03-11T21:25:40.551453+0000","last_became_peered":"2024-03-11T21:25:40.551453+0000","last_unstale":"2024-03-11T21:29:29.180615+0000","last_undegraded":"2024-03-11T21:29:29.180615+0000","last_fullsized":"2024-03-11T21:29:29.180615+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'15","last_scrub_stamp":"2024-03-11T21:28:35.104626+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:39.525554+0000","last_clean_scrub_stamp":"2024-03-11T21:28:35.104626+0000","objects_scrubbed":7,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:29:50.086022+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.1d","version":"0'0","reported_seq":130,"reported_epoch":124,"state":"active+clean","last_fresh":"2024-03-11T21:29:29.177249+0000","last_change":"2024-03-11T21:29:12.415949+0000","last_active":"2024-03-11T21:29:29.177249+0000","last_peered":"2024-03-11T21:29:29.177249+0000","last_clean":"2024-03-11T21:29:29.177249+0000","last_became_active":"2024-03-11T21:25:44.588845+0000","last_became_peered":"2024-03-11T21:25:44.588845+0000","last_unstale":"2024-03-11T21:29:29.177249+0000","last_undegraded":"2024-03-11T21:29:29.177249+0000","last_fullsized":"2024-03-11T21:29:29.177249+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:29:12.415585+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:43.559452+0000","last_clean_scrub_stamp":"2024-03-11T21:29:12.415585+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:30:38.109694+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.1e","version":"0'0","reported_seq":134,"reported_epoch":124,"state":"active+clean","last_fresh":"2024-03-11T21:29:29.172041+0000","last_change":"2024-03-11T21:28:17.464869+0000","last_active":"2024-03-11T21:29:29.172041+0000","last_peered":"2024-03-11T21:29:29.172041+0000","last_clean":"2024-03-11T21:29:29.172041+0000","last_became_active":"2024-03-11T21:25:42.582959+0000","last_became_peered":"2024-03-11T21:25:42.582959+0000","last_unstale":"2024-03-11T21:29:29.172041+0000","last_undegraded":"2024-03-11T21:29:29.172041+0000","last_fullsized":"2024-03-11T21:29:29.172041+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:28:17.464401+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:26:58.335431+0000","last_clean_scrub_stamp":"2024-03-11T21:28:17.464401+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:29:40.047363+0000","scrub_duration":20,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.19","version":"67'2","reported_seq":140,"reported_epoch":124,"state":"active+clean","last_fresh":"2024-03-11T21:29:29.184659+0000","last_change":"2024-03-11T21:27:27.823258+0000","last_active":"2024-03-11T21:29:29.184659+0000","last_peered":"2024-03-11T21:29:29.184659+0000","last_clean":"2024-03-11T21:29:29.184659+0000","last_became_active":"2024-03-11T21:25:38.539243+0000","last_became_peered":"2024-03-11T21:25:38.539243+0000","last_unstale":"2024-03-11T21:29:29.184659+0000","last_undegraded":"2024-03-11T21:29:29.184659+0000","last_fullsized":"2024-03-11T21:29:29.184659+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'2","last_scrub_stamp":"2024-03-11T21:27:27.823167+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:37.512018+0000","last_clean_scrub_stamp":"2024-03-11T21:27:27.823167+0000","objects_scrubbed":2,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":19,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":957,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":7,"num_read_kb":7,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,0],"acting":[3,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1f","version":"124'16","reported_seq":170,"reported_epoch":124,"state":"active+clean","last_fresh":"2024-03-11T21:29:30.873563+0000","last_change":"2024-03-11T21:29:30.873609+0000","last_active":"2024-03-11T21:29:30.873563+0000","last_peered":"2024-03-11T21:29:30.873563+0000","last_clean":"2024-03-11T21:29:30.873563+0000","last_became_active":"2024-03-11T21:25:40.554686+0000","last_became_peered":"2024-03-11T21:25:40.554686+0000","last_unstale":"2024-03-11T21:29:30.873563+0000","last_undegraded":"2024-03-11T21:29:30.873563+0000","last_fullsized":"2024-03-11T21:29:30.873563+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"124'16","last_scrub_stamp":"2024-03-11T21:29:30.873490+0000","last_deep_scrub":"124'16","last_deep_scrub_stamp":"2024-03-11T21:29:30.873490+0000","last_clean_scrub_stamp":"2024-03-11T21:29:30.873490+0000","objects_scrubbed":5,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:30:45.014037+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,1],"acting":[6,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.1c","version":"74'1","reported_seq":130,"reported_epoch":124,"state":"active+clean","last_fresh":"2024-03-11T21:29:29.175521+0000","last_change":"2024-03-11T21:29:17.510892+0000","last_active":"2024-03-11T21:29:29.175521+0000","last_peered":"2024-03-11T21:29:29.175521+0000","last_clean":"2024-03-11T21:29:29.175521+0000","last_became_active":"2024-03-11T21:25:44.593326+0000","last_became_peered":"2024-03-11T21:25:44.593326+0000","last_unstale":"2024-03-11T21:29:29.175521+0000","last_undegraded":"2024-03-11T21:29:29.175521+0000","last_fullsized":"2024-03-11T21:29:29.175521+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'1","last_scrub_stamp":"2024-03-11T21:29:17.510760+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:43.559452+0000","last_clean_scrub_stamp":"2024-03-11T21:29:17.510760+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:30:36.033626+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":370,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.1f","version":"74'8","reported_seq":201,"reported_epoch":124,"state":"active+clean","last_fresh":"2024-03-11T21:29:29.178472+0000","last_change":"2024-03-11T21:29:13.781905+0000","last_active":"2024-03-11T21:29:29.178472+0000","last_peered":"2024-03-11T21:29:29.178472+0000","last_clean":"2024-03-11T21:29:29.178472+0000","last_became_active":"2024-03-11T21:25:42.577401+0000","last_became_peered":"2024-03-11T21:25:42.577401+0000","last_unstale":"2024-03-11T21:29:29.178472+0000","last_undegraded":"2024-03-11T21:29:29.178472+0000","last_fullsized":"2024-03-11T21:29:29.178472+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'8","last_scrub_stamp":"2024-03-11T21:29:13.781764+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:41.544550+0000","last_clean_scrub_stamp":"2024-03-11T21:29:13.781764+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:30:36.265797+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.18","version":"67'1","reported_seq":155,"reported_epoch":124,"state":"active+clean","last_fresh":"2024-03-11T21:29:35.918580+0000","last_change":"2024-03-11T21:29:35.918627+0000","last_active":"2024-03-11T21:29:35.918580+0000","last_peered":"2024-03-11T21:29:35.918580+0000","last_clean":"2024-03-11T21:29:35.918580+0000","last_became_active":"2024-03-11T21:25:38.538321+0000","last_became_peered":"2024-03-11T21:25:38.538321+0000","last_unstale":"2024-03-11T21:29:35.918580+0000","last_undegraded":"2024-03-11T21:29:35.918580+0000","last_fullsized":"2024-03-11T21:29:35.918580+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'1","last_scrub_stamp":"2024-03-11T21:29:35.918494+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:37.512018+0000","last_clean_scrub_stamp":"2024-03-11T21:29:35.918494+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:30:59.355910+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":436,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":7,"num_read_kb":7,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,7],"acting":[5,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1e","version":"74'10","reported_seq":153,"reported_epoch":124,"state":"active+clean","last_fresh":"2024-03-11T21:29:29.171941+0000","last_change":"2024-03-11T21:28:26.392381+0000","last_active":"2024-03-11T21:29:29.171941+0000","last_peered":"2024-03-11T21:29:29.171941+0000","last_clean":"2024-03-11T21:29:29.171941+0000","last_became_active":"2024-03-11T21:25:40.763438+0000","last_became_peered":"2024-03-11T21:25:40.763438+0000","last_unstale":"2024-03-11T21:29:29.171941+0000","last_undegraded":"2024-03-11T21:29:29.171941+0000","last_fullsized":"2024-03-11T21:29:29.171941+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'10","last_scrub_stamp":"2024-03-11T21:28:26.392246+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:39.525554+0000","last_clean_scrub_stamp":"2024-03-11T21:28:26.392246+0000","objects_scrubbed":5,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:29:40.807743+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.13","version":"0'0","reported_seq":129,"reported_epoch":124,"state":"active+clean","last_fresh":"2024-03-11T21:29:29.181293+0000","last_change":"2024-03-11T21:28:27.600720+0000","last_active":"2024-03-11T21:29:29.181293+0000","last_peered":"2024-03-11T21:29:29.181293+0000","last_clean":"2024-03-11T21:29:29.181293+0000","last_became_active":"2024-03-11T21:25:45.302102+0000","last_became_peered":"2024-03-11T21:25:45.302102+0000","last_unstale":"2024-03-11T21:29:29.181293+0000","last_undegraded":"2024-03-11T21:29:29.181293+0000","last_fullsized":"2024-03-11T21:29:29.181293+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:28:27.600574+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:43.559452+0000","last_clean_scrub_stamp":"2024-03-11T21:28:27.600574+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:29:32.095820+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":133,"reported_epoch":124,"state":"active+clean","last_fresh":"2024-03-11T21:29:29.175732+0000","last_change":"2024-03-11T21:28:26.537966+0000","last_active":"2024-03-11T21:29:29.175732+0000","last_peered":"2024-03-11T21:29:29.175732+0000","last_clean":"2024-03-11T21:29:29.175732+0000","last_became_active":"2024-03-11T21:25:42.579781+0000","last_became_peered":"2024-03-11T21:25:42.579781+0000","last_unstale":"2024-03-11T21:29:29.175732+0000","last_undegraded":"2024-03-11T21:29:29.175732+0000","last_fullsized":"2024-03-11T21:29:29.175732+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:28:26.537878+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:41.544550+0000","last_clean_scrub_stamp":"2024-03-11T21:28:26.537878+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:29:43.076912+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.17","version":"0'0","reported_seq":142,"reported_epoch":124,"state":"active+clean","last_fresh":"2024-03-11T21:29:29.178272+0000","last_change":"2024-03-11T21:28:25.918222+0000","last_active":"2024-03-11T21:29:29.178272+0000","last_peered":"2024-03-11T21:29:29.178272+0000","last_clean":"2024-03-11T21:29:29.178272+0000","last_became_active":"2024-03-11T21:25:38.540349+0000","last_became_peered":"2024-03-11T21:25:38.540349+0000","last_unstale":"2024-03-11T21:29:29.178272+0000","last_undegraded":"2024-03-11T21:29:29.178272+0000","last_fullsized":"2024-03-11T21:29:29.178272+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:28:25.918101+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:28:25.918101+0000","last_clean_scrub_stamp":"2024-03-11T21:28:25.918101+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:29:47.849509+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,2],"acting":[6,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.11","version":"74'11","reported_seq":156,"reported_epoch":124,"state":"active+clean","last_fresh":"2024-03-11T21:29:29.184761+0000","last_change":"2024-03-11T21:29:19.659819+0000","last_active":"2024-03-11T21:29:29.184761+0000","last_peered":"2024-03-11T21:29:29.184761+0000","last_clean":"2024-03-11T21:29:29.184761+0000","last_became_active":"2024-03-11T21:25:40.760664+0000","last_became_peered":"2024-03-11T21:25:40.760664+0000","last_unstale":"2024-03-11T21:29:29.184761+0000","last_undegraded":"2024-03-11T21:29:29.184761+0000","last_fullsized":"2024-03-11T21:29:29.184761+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'11","last_scrub_stamp":"2024-03-11T21:29:19.659691+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:39.525554+0000","last_clean_scrub_stamp":"2024-03-11T21:29:19.659691+0000","objects_scrubbed":5,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:30:33.482852+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":129,"reported_epoch":124,"state":"active+clean","last_fresh":"2024-03-11T21:29:29.175057+0000","last_change":"2024-03-11T21:28:19.525623+0000","last_active":"2024-03-11T21:29:29.175057+0000","last_peered":"2024-03-11T21:29:29.175057+0000","last_clean":"2024-03-11T21:29:29.175057+0000","last_became_active":"2024-03-11T21:25:44.591068+0000","last_became_peered":"2024-03-11T21:25:44.591068+0000","last_unstale":"2024-03-11T21:29:29.175057+0000","last_undegraded":"2024-03-11T21:29:29.175057+0000","last_fullsized":"2024-03-11T21:29:29.175057+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:28:19.525508+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:43.559452+0000","last_clean_scrub_stamp":"2024-03-11T21:28:19.525508+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,4],"acting":[7,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":134,"reported_epoch":124,"state":"active+clean","last_fresh":"2024-03-11T21:29:29.177799+0000","last_change":"2024-03-11T21:28:32.783682+0000","last_active":"2024-03-11T21:29:29.177799+0000","last_peered":"2024-03-11T21:29:29.177799+0000","last_clean":"2024-03-11T21:29:29.177799+0000","last_became_active":"2024-03-11T21:25:42.584148+0000","last_became_peered":"2024-03-11T21:25:42.584148+0000","last_unstale":"2024-03-11T21:29:29.177799+0000","last_undegraded":"2024-03-11T21:29:29.177799+0000","last_fullsized":"2024-03-11T21:29:29.177799+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:28:32.783565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:27:09.976747+0000","last_clean_scrub_stamp":"2024-03-11T21:28:32.783565+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:29:36.980081+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.16","version":"0'0","reported_seq":141,"reported_epoch":124,"state":"active+clean","last_fresh":"2024-03-11T21:29:29.183330+0000","last_change":"2024-03-11T21:28:37.734762+0000","last_active":"2024-03-11T21:29:29.183330+0000","last_peered":"2024-03-11T21:29:29.183330+0000","last_clean":"2024-03-11T21:29:29.183330+0000","last_became_active":"2024-03-11T21:25:38.539103+0000","last_became_peered":"2024-03-11T21:25:38.539103+0000","last_unstale":"2024-03-11T21:29:29.183330+0000","last_undegraded":"2024-03-11T21:29:29.183330+0000","last_fullsized":"2024-03-11T21:29:29.183330+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:28:37.734633+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:27:11.938909+0000","last_clean_scrub_stamp":"2024-03-11T21:28:37.734633+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:29:40.854282+0000","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,2],"acting":[5,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.10","version":"94'6","reported_seq":145,"reported_epoch":124,"state":"active+clean","last_fresh":"2024-03-11T21:29:29.181788+0000","last_change":"2024-03-11T21:28:24.604117+0000","last_active":"2024-03-11T21:29:29.181788+0000","last_peered":"2024-03-11T21:29:29.181788+0000","last_clean":"2024-03-11T21:29:29.181788+0000","last_became_active":"2024-03-11T21:25:40.552720+0000","last_became_peered":"2024-03-11T21:25:40.552720+0000","last_unstale":"2024-03-11T21:29:29.181788+0000","last_undegraded":"2024-03-11T21:29:29.181788+0000","last_fullsized":"2024-03-11T21:29:29.181788+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"94'6","last_scrub_stamp":"2024-03-11T21:28:24.603974+0000","last_deep_scrub":"94'6","last_deep_scrub_stamp":"2024-03-11T21:28:24.603974+0000","last_clean_scrub_stamp":"2024-03-11T21:28:24.603974+0000","objects_scrubbed":2,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":6,"num_read_kb":4,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6],"acting":[3,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":129,"reported_epoch":124,"state":"active+clean","last_fresh":"2024-03-11T21:29:29.184482+0000","last_change":"2024-03-11T21:29:22.673347+0000","last_active":"2024-03-11T21:29:29.184482+0000","last_peered":"2024-03-11T21:29:29.184482+0000","last_clean":"2024-03-11T21:29:29.184482+0000","last_became_active":"2024-03-11T21:25:44.583181+0000","last_became_peered":"2024-03-11T21:25:44.583181+0000","last_unstale":"2024-03-11T21:29:29.184482+0000","last_undegraded":"2024-03-11T21:29:29.184482+0000","last_fullsized":"2024-03-11T21:29:29.184482+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:29:22.673239+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:43.559452+0000","last_clean_scrub_stamp":"2024-03-11T21:29:22.673239+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:30:31.246576+0000","scrub_duration":18,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":134,"reported_epoch":124,"state":"active+clean","last_fresh":"2024-03-11T21:29:29.179472+0000","last_change":"2024-03-11T21:28:34.310861+0000","last_active":"2024-03-11T21:29:29.179472+0000","last_peered":"2024-03-11T21:29:29.179472+0000","last_clean":"2024-03-11T21:29:29.179472+0000","last_became_active":"2024-03-11T21:25:42.573158+0000","last_became_peered":"2024-03-11T21:25:42.573158+0000","last_unstale":"2024-03-11T21:29:29.179472+0000","last_undegraded":"2024-03-11T21:29:29.179472+0000","last_fullsized":"2024-03-11T21:29:29.179472+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:28:34.310410+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:27:21.207303+0000","last_clean_scrub_stamp":"2024-03-11T21:28:34.310410+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:29:46.914835+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.15","version":"0'0","reported_seq":142,"reported_epoch":124,"state":"active+clean","last_fresh":"2024-03-11T21:29:29.177450+0000","last_change":"2024-03-11T21:28:17.525764+0000","last_active":"2024-03-11T21:29:29.177450+0000","last_peered":"2024-03-11T21:29:29.177450+0000","last_clean":"2024-03-11T21:29:29.177450+0000","last_became_active":"2024-03-11T21:25:38.541705+0000","last_became_peered":"2024-03-11T21:25:38.541705+0000","last_unstale":"2024-03-11T21:29:29.177450+0000","last_undegraded":"2024-03-11T21:29:29.177450+0000","last_fullsized":"2024-03-11T21:29:29.177450+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:28:17.525649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:26:44.860499+0000","last_clean_scrub_stamp":"2024-03-11T21:28:17.525649+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:29:31.562181+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.13","version":"109'16","reported_seq":180,"reported_epoch":124,"state":"active+clean","last_fresh":"2024-03-11T21:29:29.172205+0000","last_change":"2024-03-11T21:28:32.854572+0000","last_active":"2024-03-11T21:29:29.172205+0000","last_peered":"2024-03-11T21:29:29.172205+0000","last_clean":"2024-03-11T21:29:29.172205+0000","last_became_active":"2024-03-11T21:26:11.214879+0000","last_became_peered":"2024-03-11T21:26:11.214879+0000","last_unstale":"2024-03-11T21:29:29.172205+0000","last_undegraded":"2024-03-11T21:29:29.172205+0000","last_fullsized":"2024-03-11T21:29:29.172205+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"109'16","last_scrub_stamp":"2024-03-11T21:28:32.854393+0000","last_deep_scrub":"109'16","last_deep_scrub_stamp":"2024-03-11T21:28:32.854393+0000","last_clean_scrub_stamp":"2024-03-11T21:28:32.854393+0000","objects_scrubbed":5,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:29:37.486089+0000","scrub_duration":16,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,2],"acting":[4,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.10","version":"0'0","reported_seq":122,"reported_epoch":124,"state":"active+clean","last_fresh":"2024-03-11T21:29:29.171353+0000","last_change":"2024-03-11T21:27:25.417528+0000","last_active":"2024-03-11T21:29:29.171353+0000","last_peered":"2024-03-11T21:29:29.171353+0000","last_clean":"2024-03-11T21:29:29.171353+0000","last_became_active":"2024-03-11T21:25:44.591189+0000","last_became_peered":"2024-03-11T21:25:44.591189+0000","last_unstale":"2024-03-11T21:29:29.171353+0000","last_undegraded":"2024-03-11T21:29:29.171353+0000","last_fullsized":"2024-03-11T21:29:29.171353+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:27:25.417450+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:27:25.417450+0000","last_clean_scrub_stamp":"2024-03-11T21:27:25.417450+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,1],"acting":[0,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":133,"reported_epoch":124,"state":"active+clean","last_fresh":"2024-03-11T21:29:29.181034+0000","last_change":"2024-03-11T21:29:13.597022+0000","last_active":"2024-03-11T21:29:29.181034+0000","last_peered":"2024-03-11T21:29:29.181034+0000","last_clean":"2024-03-11T21:29:29.181034+0000","last_became_active":"2024-03-11T21:25:42.576425+0000","last_became_peered":"2024-03-11T21:25:42.576425+0000","last_unstale":"2024-03-11T21:29:29.181034+0000","last_undegraded":"2024-03-11T21:29:29.181034+0000","last_fullsized":"2024-03-11T21:29:29.181034+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:29:13.596758+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:41.544550+0000","last_clean_scrub_stamp":"2024-03-11T21:29:13.596758+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:30:32.902072+0000","scrub_duration":55,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.14","version":"0'0","reported_seq":150,"reported_epoch":124,"state":"active+clean","last_fresh":"2024-03-11T21:29:29.901611+0000","last_change":"2024-03-11T21:29:29.901645+0000","last_active":"2024-03-11T21:29:29.901611+0000","last_peered":"2024-03-11T21:29:29.901611+0000","last_clean":"2024-03-11T21:29:29.901611+0000","last_became_active":"2024-03-11T21:25:38.541852+0000","last_became_peered":"2024-03-11T21:25:38.541852+0000","last_unstale":"2024-03-11T21:29:29.901611+0000","last_undegraded":"2024-03-11T21:29:29.901611+0000","last_fullsized":"2024-03-11T21:29:29.901611+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:29:29.901544+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:37.512018+0000","last_clean_scrub_stamp":"2024-03-11T21:29:29.901544+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:30:53.801781+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,5],"acting":[6,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.12","version":"74'9","reported_seq":154,"reported_epoch":124,"state":"active+clean","last_fresh":"2024-03-11T21:29:29.179267+0000","last_change":"2024-03-11T21:28:32.283534+0000","last_active":"2024-03-11T21:29:29.179267+0000","last_peered":"2024-03-11T21:29:29.179267+0000","last_clean":"2024-03-11T21:29:29.179267+0000","last_became_active":"2024-03-11T21:25:40.547082+0000","last_became_peered":"2024-03-11T21:25:40.547082+0000","last_unstale":"2024-03-11T21:29:29.179267+0000","last_undegraded":"2024-03-11T21:29:29.179267+0000","last_fullsized":"2024-03-11T21:29:29.179267+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'9","last_scrub_stamp":"2024-03-11T21:28:32.283410+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:39.525554+0000","last_clean_scrub_stamp":"2024-03-11T21:28:32.283410+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:29:39.936188+0000","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":130,"reported_epoch":124,"state":"active+clean","last_fresh":"2024-03-11T21:29:29.173537+0000","last_change":"2024-03-11T21:29:23.214125+0000","last_active":"2024-03-11T21:29:29.173537+0000","last_peered":"2024-03-11T21:29:29.173537+0000","last_clean":"2024-03-11T21:29:29.173537+0000","last_became_active":"2024-03-11T21:25:44.589586+0000","last_became_peered":"2024-03-11T21:25:44.589586+0000","last_unstale":"2024-03-11T21:29:29.173537+0000","last_undegraded":"2024-03-11T21:29:29.173537+0000","last_fullsized":"2024-03-11T21:29:29.173537+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:29:23.214015+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:27:09.801138+0000","last_clean_scrub_stamp":"2024-03-11T21:29:23.214015+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:30:49.151024+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,5],"acting":[4,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.14","version":"74'8","reported_seq":189,"reported_epoch":124,"state":"active+clean","last_fresh":"2024-03-11T21:29:34.599235+0000","last_change":"2024-03-11T21:29:34.599277+0000","last_active":"2024-03-11T21:29:34.599235+0000","last_peered":"2024-03-11T21:29:34.599235+0000","last_clean":"2024-03-11T21:29:34.599235+0000","last_became_active":"2024-03-11T21:25:42.583266+0000","last_became_peered":"2024-03-11T21:25:42.583266+0000","last_unstale":"2024-03-11T21:29:34.599235+0000","last_undegraded":"2024-03-11T21:29:34.599235+0000","last_fullsized":"2024-03-11T21:29:34.599235+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'8","last_scrub_stamp":"2024-03-11T21:29:34.599162+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:41.544550+0000","last_clean_scrub_stamp":"2024-03-11T21:29:34.599162+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:30:45.386715+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,2],"acting":[3,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.13","version":"0'0","reported_seq":142,"reported_epoch":124,"state":"active+clean","last_fresh":"2024-03-11T21:29:29.172730+0000","last_change":"2024-03-11T21:29:16.449075+0000","last_active":"2024-03-11T21:29:29.172730+0000","last_peered":"2024-03-11T21:29:29.172730+0000","last_clean":"2024-03-11T21:29:29.172730+0000","last_became_active":"2024-03-11T21:25:38.539467+0000","last_became_peered":"2024-03-11T21:25:38.539467+0000","last_unstale":"2024-03-11T21:29:29.172730+0000","last_undegraded":"2024-03-11T21:29:29.172730+0000","last_fullsized":"2024-03-11T21:29:29.172730+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:29:16.448818+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:37.512018+0000","last_clean_scrub_stamp":"2024-03-11T21:29:16.448818+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:30:45.928952+0000","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.15","version":"94'13","reported_seq":157,"reported_epoch":124,"state":"active+clean","last_fresh":"2024-03-11T21:29:29.183084+0000","last_change":"2024-03-11T21:28:27.691406+0000","last_active":"2024-03-11T21:29:29.183084+0000","last_peered":"2024-03-11T21:29:29.183084+0000","last_clean":"2024-03-11T21:29:29.183084+0000","last_became_active":"2024-03-11T21:25:40.760907+0000","last_became_peered":"2024-03-11T21:25:40.760907+0000","last_unstale":"2024-03-11T21:29:29.183084+0000","last_undegraded":"2024-03-11T21:29:29.183084+0000","last_fullsized":"2024-03-11T21:29:29.183084+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"94'13","last_scrub_stamp":"2024-03-11T21:28:27.691249+0000","last_deep_scrub":"94'13","last_deep_scrub_stamp":"2024-03-11T21:26:59.013823+0000","last_clean_scrub_stamp":"2024-03-11T21:28:27.691249+0000","objects_scrubbed":4,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:29:40.580610+0000","scrub_duration":28,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.16","version":"0'0","reported_seq":130,"reported_epoch":124,"state":"active+clean","last_fresh":"2024-03-11T21:29:29.171398+0000","last_change":"2024-03-11T21:28:29.368481+0000","last_active":"2024-03-11T21:29:29.171398+0000","last_peered":"2024-03-11T21:29:29.171398+0000","last_clean":"2024-03-11T21:29:29.171398+0000","last_became_active":"2024-03-11T21:25:44.583393+0000","last_became_peered":"2024-03-11T21:25:44.583393+0000","last_unstale":"2024-03-11T21:29:29.171398+0000","last_undegraded":"2024-03-11T21:29:29.171398+0000","last_fullsized":"2024-03-11T21:29:29.171398+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:28:29.368377+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:43.559452+0000","last_clean_scrub_stamp":"2024-03-11T21:28:29.368377+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:29:46.910936+0000","scrub_duration":27,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.15","version":"98'9","reported_seq":190,"reported_epoch":124,"state":"active+clean","last_fresh":"2024-03-11T21:29:29.183031+0000","last_change":"2024-03-11T21:29:27.875893+0000","last_active":"2024-03-11T21:29:29.183031+0000","last_peered":"2024-03-11T21:29:29.183031+0000","last_clean":"2024-03-11T21:29:29.183031+0000","last_became_active":"2024-03-11T21:25:42.572555+0000","last_became_peered":"2024-03-11T21:25:42.572555+0000","last_unstale":"2024-03-11T21:29:29.183031+0000","last_undegraded":"2024-03-11T21:29:29.183031+0000","last_fullsized":"2024-03-11T21:29:29.183031+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"98'9","last_scrub_stamp":"2024-03-11T21:29:27.875760+0000","last_deep_scrub":"98'9","last_deep_scrub_stamp":"2024-03-11T21:27:34.948855+0000","last_clean_scrub_stamp":"2024-03-11T21:29:27.875760+0000","objects_scrubbed":1,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:30:49.671154+0000","scrub_duration":18,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.12","version":"0'0","reported_seq":141,"reported_epoch":124,"state":"active+clean","last_fresh":"2024-03-11T21:29:29.182458+0000","last_change":"2024-03-11T21:28:26.701731+0000","last_active":"2024-03-11T21:29:29.182458+0000","last_peered":"2024-03-11T21:29:29.182458+0000","last_clean":"2024-03-11T21:29:29.182458+0000","last_became_active":"2024-03-11T21:25:38.543962+0000","last_became_peered":"2024-03-11T21:25:38.543962+0000","last_unstale":"2024-03-11T21:29:29.182458+0000","last_undegraded":"2024-03-11T21:29:29.182458+0000","last_fullsized":"2024-03-11T21:29:29.182458+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:28:26.701620+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:37.512018+0000","last_clean_scrub_stamp":"2024-03-11T21:28:26.701620+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:29:51.689535+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.14","version":"74'10","reported_seq":152,"reported_epoch":124,"state":"active+clean","last_fresh":"2024-03-11T21:29:29.187318+0000","last_change":"2024-03-11T21:29:27.547718+0000","last_active":"2024-03-11T21:29:29.187318+0000","last_peered":"2024-03-11T21:29:29.187318+0000","last_clean":"2024-03-11T21:29:29.187318+0000","last_became_active":"2024-03-11T21:25:40.760620+0000","last_became_peered":"2024-03-11T21:25:40.760620+0000","last_unstale":"2024-03-11T21:29:29.187318+0000","last_undegraded":"2024-03-11T21:29:29.187318+0000","last_fullsized":"2024-03-11T21:29:29.187318+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'10","last_scrub_stamp":"2024-03-11T21:29:27.547582+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:39.525554+0000","last_clean_scrub_stamp":"2024-03-11T21:29:27.547582+0000","objects_scrubbed":5,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:30:43.351094+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.15","version":"0'0","reported_seq":129,"reported_epoch":124,"state":"active+clean","last_fresh":"2024-03-11T21:29:29.175407+0000","last_change":"2024-03-11T21:28:33.526147+0000","last_active":"2024-03-11T21:29:29.175407+0000","last_peered":"2024-03-11T21:29:29.175407+0000","last_clean":"2024-03-11T21:29:29.175407+0000","last_became_active":"2024-03-11T21:25:45.302501+0000","last_became_peered":"2024-03-11T21:25:45.302501+0000","last_unstale":"2024-03-11T21:29:29.175407+0000","last_undegraded":"2024-03-11T21:29:29.175407+0000","last_fullsized":"2024-03-11T21:29:29.175407+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:28:33.526015+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:43.559452+0000","last_clean_scrub_stamp":"2024-03-11T21:28:33.526015+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:30:01.748461+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":133,"reported_epoch":124,"state":"active+clean","last_fresh":"2024-03-11T21:29:29.183250+0000","last_change":"2024-03-11T21:29:19.880114+0000","last_active":"2024-03-11T21:29:29.183250+0000","last_peered":"2024-03-11T21:29:29.183250+0000","last_clean":"2024-03-11T21:29:29.183250+0000","last_became_active":"2024-03-11T21:25:42.577351+0000","last_became_peered":"2024-03-11T21:25:42.577351+0000","last_unstale":"2024-03-11T21:29:29.183250+0000","last_undegraded":"2024-03-11T21:29:29.183250+0000","last_fullsized":"2024-03-11T21:29:29.183250+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:29:19.879985+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:41.544550+0000","last_clean_scrub_stamp":"2024-03-11T21:29:19.879985+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:30:44.647434+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,1],"acting":[5,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.11","version":"0'0","reported_seq":142,"reported_epoch":124,"state":"active+clean","last_fresh":"2024-03-11T21:29:29.177896+0000","last_change":"2024-03-11T21:28:28.897384+0000","last_active":"2024-03-11T21:29:29.177896+0000","last_peered":"2024-03-11T21:29:29.177896+0000","last_clean":"2024-03-11T21:29:29.177896+0000","last_became_active":"2024-03-11T21:25:38.537415+0000","last_became_peered":"2024-03-11T21:25:38.537415+0000","last_unstale":"2024-03-11T21:29:29.177896+0000","last_undegraded":"2024-03-11T21:29:29.177896+0000","last_fullsized":"2024-03-11T21:29:29.177896+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:28:28.897294+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:37.512018+0000","last_clean_scrub_stamp":"2024-03-11T21:28:28.897294+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:29:53.616989+0000","scrub_duration":16,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.17","version":"97'9","reported_seq":163,"reported_epoch":124,"state":"active+clean","last_fresh":"2024-03-11T21:29:29.181356+0000","last_change":"2024-03-11T21:29:26.522324+0000","last_active":"2024-03-11T21:29:29.181356+0000","last_peered":"2024-03-11T21:29:29.181356+0000","last_clean":"2024-03-11T21:29:29.181356+0000","last_became_active":"2024-03-11T21:26:11.219020+0000","last_became_peered":"2024-03-11T21:26:11.219020+0000","last_unstale":"2024-03-11T21:29:29.181356+0000","last_undegraded":"2024-03-11T21:29:29.181356+0000","last_fullsized":"2024-03-11T21:29:29.181356+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"97'9","last_scrub_stamp":"2024-03-11T21:29:26.522238+0000","last_deep_scrub":"97'9","last_deep_scrub_stamp":"2024-03-11T21:27:25.803132+0000","last_clean_scrub_stamp":"2024-03-11T21:29:26.522238+0000","objects_scrubbed":3,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:30:43.232148+0000","scrub_duration":17,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":9,"num_read_kb":6,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,5],"acting":[3,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.14","version":"0'0","reported_seq":129,"reported_epoch":124,"state":"active+clean","last_fresh":"2024-03-11T21:29:29.180289+0000","last_change":"2024-03-11T21:29:16.233178+0000","last_active":"2024-03-11T21:29:29.180289+0000","last_peered":"2024-03-11T21:29:29.180289+0000","last_clean":"2024-03-11T21:29:29.180289+0000","last_became_active":"2024-03-11T21:25:44.587439+0000","last_became_peered":"2024-03-11T21:25:44.587439+0000","last_unstale":"2024-03-11T21:29:29.180289+0000","last_undegraded":"2024-03-11T21:29:29.180289+0000","last_fullsized":"2024-03-11T21:29:29.180289+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:29:16.233061+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:43.559452+0000","last_clean_scrub_stamp":"2024-03-11T21:29:16.233061+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:30:41.963480+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,7],"acting":[2,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":133,"reported_epoch":124,"state":"active+clean","last_fresh":"2024-03-11T21:29:29.181363+0000","last_change":"2024-03-11T21:28:35.504528+0000","last_active":"2024-03-11T21:29:29.181363+0000","last_peered":"2024-03-11T21:29:29.181363+0000","last_clean":"2024-03-11T21:29:29.181363+0000","last_became_active":"2024-03-11T21:25:42.583514+0000","last_became_peered":"2024-03-11T21:25:42.583514+0000","last_unstale":"2024-03-11T21:29:29.181363+0000","last_undegraded":"2024-03-11T21:29:29.181363+0000","last_fullsized":"2024-03-11T21:29:29.181363+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:28:35.504414+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:41.544550+0000","last_clean_scrub_stamp":"2024-03-11T21:28:35.504414+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:29:59.062600+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.10","version":"0'0","reported_seq":141,"reported_epoch":124,"state":"active+clean","last_fresh":"2024-03-11T21:29:29.186516+0000","last_change":"2024-03-11T21:28:37.151842+0000","last_active":"2024-03-11T21:29:29.186516+0000","last_peered":"2024-03-11T21:29:29.186516+0000","last_clean":"2024-03-11T21:29:29.186516+0000","last_became_active":"2024-03-11T21:25:38.533138+0000","last_became_peered":"2024-03-11T21:25:38.533138+0000","last_unstale":"2024-03-11T21:29:29.186516+0000","last_undegraded":"2024-03-11T21:29:29.186516+0000","last_fullsized":"2024-03-11T21:29:29.186516+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:28:37.151573+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:37.512018+0000","last_clean_scrub_stamp":"2024-03-11T21:28:37.151573+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:29:59.767185+0000","scrub_duration":32,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,0],"acting":[2,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.16","version":"74'9","reported_seq":154,"reported_epoch":124,"state":"active+clean","last_fresh":"2024-03-11T21:29:29.171693+0000","last_change":"2024-03-11T21:28:10.399010+0000","last_active":"2024-03-11T21:29:29.171693+0000","last_peered":"2024-03-11T21:29:29.171693+0000","last_clean":"2024-03-11T21:29:29.171693+0000","last_became_active":"2024-03-11T21:25:40.763434+0000","last_became_peered":"2024-03-11T21:25:40.763434+0000","last_unstale":"2024-03-11T21:29:29.171693+0000","last_undegraded":"2024-03-11T21:29:29.171693+0000","last_fullsized":"2024-03-11T21:29:29.171693+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'9","last_scrub_stamp":"2024-03-11T21:28:10.398261+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:39.525554+0000","last_clean_scrub_stamp":"2024-03-11T21:28:10.398261+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.a","version":"124'8805","reported_seq":8937,"reported_epoch":124,"state":"active+clean","last_fresh":"2024-03-11T21:29:37.188416+0000","last_change":"2024-03-11T21:29:35.563939+0000","last_active":"2024-03-11T21:29:37.188416+0000","last_peered":"2024-03-11T21:29:37.188416+0000","last_clean":"2024-03-11T21:29:37.188416+0000","last_became_active":"2024-03-11T21:27:36.692988+0000","last_became_peered":"2024-03-11T21:27:36.692988+0000","last_unstale":"2024-03-11T21:29:37.188416+0000","last_undegraded":"2024-03-11T21:29:37.188416+0000","last_fullsized":"2024-03-11T21:29:37.188416+0000","mapping_epoch":99,"log_start":"108'4100","ondisk_log_start":"108'4100","created":99,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"124'8703","last_scrub_stamp":"2024-03-11T21:29:35.563670+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:27:35.667522+0000","last_clean_scrub_stamp":"2024-03-11T21:29:35.563670+0000","objects_scrubbed":6518,"log_size":4705,"log_dups_size":0,"ondisk_log_size":4705,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:30:55.937742+0000","scrub_duration":1963,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":417792000,"num_objects":6375,"num_object_clones":0,"num_object_copies":19125,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6375,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8805,"num_write_kb":485760,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5],"acting":[3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.b","version":"0'0","reported_seq":121,"reported_epoch":124,"state":"active+clean","last_fresh":"2024-03-11T21:29:29.181113+0000","last_change":"2024-03-11T21:28:03.724789+0000","last_active":"2024-03-11T21:29:29.181113+0000","last_peered":"2024-03-11T21:29:29.181113+0000","last_clean":"2024-03-11T21:29:29.181113+0000","last_became_active":"2024-03-11T21:25:44.590071+0000","last_became_peered":"2024-03-11T21:25:44.590071+0000","last_unstale":"2024-03-11T21:29:29.181113+0000","last_undegraded":"2024-03-11T21:29:29.181113+0000","last_fullsized":"2024-03-11T21:29:29.181113+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:28:03.724666+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:43.559452+0000","last_clean_scrub_stamp":"2024-03-11T21:28:03.724666+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.f","version":"67'2","reported_seq":154,"reported_epoch":124,"state":"active+clean","last_fresh":"2024-03-11T21:29:29.173377+0000","last_change":"2024-03-11T21:29:18.211797+0000","last_active":"2024-03-11T21:29:29.173377+0000","last_peered":"2024-03-11T21:29:29.173377+0000","last_clean":"2024-03-11T21:29:29.173377+0000","last_became_active":"2024-03-11T21:25:38.537711+0000","last_became_peered":"2024-03-11T21:25:38.537711+0000","last_unstale":"2024-03-11T21:29:29.173377+0000","last_undegraded":"2024-03-11T21:29:29.173377+0000","last_fullsized":"2024-03-11T21:29:29.173377+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'2","last_scrub_stamp":"2024-03-11T21:29:18.211668+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:37.512018+0000","last_clean_scrub_stamp":"2024-03-11T21:29:18.211668+0000","objects_scrubbed":2,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:30:33.869349+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":92,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":10,"num_read_kb":10,"num_write":4,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,7],"acting":[4,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.9","version":"74'12","reported_seq":153,"reported_epoch":124,"state":"active+clean","last_fresh":"2024-03-11T21:29:29.174393+0000","last_change":"2024-03-11T21:28:07.735179+0000","last_active":"2024-03-11T21:29:29.174393+0000","last_peered":"2024-03-11T21:29:29.174393+0000","last_clean":"2024-03-11T21:29:29.174393+0000","last_became_active":"2024-03-11T21:25:40.552267+0000","last_became_peered":"2024-03-11T21:25:40.552267+0000","last_unstale":"2024-03-11T21:29:29.174393+0000","last_undegraded":"2024-03-11T21:29:29.174393+0000","last_fullsized":"2024-03-11T21:29:29.174393+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'12","last_scrub_stamp":"2024-03-11T21:28:07.735010+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:39.525554+0000","last_clean_scrub_stamp":"2024-03-11T21:28:07.735010+0000","objects_scrubbed":5,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":25,"num_read_kb":16,"num_write":14,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,3],"acting":[4,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":133,"reported_epoch":124,"state":"active+clean","last_fresh":"2024-03-11T21:29:29.186427+0000","last_change":"2024-03-11T21:29:24.240202+0000","last_active":"2024-03-11T21:29:29.186427+0000","last_peered":"2024-03-11T21:29:29.186427+0000","last_clean":"2024-03-11T21:29:29.186427+0000","last_became_active":"2024-03-11T21:25:42.573987+0000","last_became_peered":"2024-03-11T21:25:42.573987+0000","last_unstale":"2024-03-11T21:29:29.186427+0000","last_undegraded":"2024-03-11T21:29:29.186427+0000","last_fullsized":"2024-03-11T21:29:29.186427+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:29:24.240087+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:41.544550+0000","last_clean_scrub_stamp":"2024-03-11T21:29:24.240087+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:30:35.854726+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.1","version":"124'8977","reported_seq":9114,"reported_epoch":124,"state":"active+clean","last_fresh":"2024-03-11T21:29:36.981750+0000","last_change":"2024-03-11T21:29:30.764868+0000","last_active":"2024-03-11T21:29:36.981750+0000","last_peered":"2024-03-11T21:29:36.981750+0000","last_clean":"2024-03-11T21:29:36.981750+0000","last_became_active":"2024-03-11T21:27:36.696990+0000","last_became_peered":"2024-03-11T21:27:36.696990+0000","last_unstale":"2024-03-11T21:29:36.981750+0000","last_undegraded":"2024-03-11T21:29:36.981750+0000","last_fullsized":"2024-03-11T21:29:36.981750+0000","mapping_epoch":99,"log_start":"108'3800","ondisk_log_start":"108'3800","created":99,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"124'8603","last_scrub_stamp":"2024-03-11T21:29:30.764371+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:27:35.667522+0000","last_clean_scrub_stamp":"2024-03-11T21:29:30.764371+0000","objects_scrubbed":6947,"log_size":5177,"log_dups_size":0,"ondisk_log_size":5177,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:30:40.150260+0000","scrub_duration":1564,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":427622400,"num_objects":6525,"num_object_clones":0,"num_object_copies":19575,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6525,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8977,"num_write_kb":496064,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.0","version":"0'0","reported_seq":122,"reported_epoch":124,"state":"active+clean","last_fresh":"2024-03-11T21:29:29.171467+0000","last_change":"2024-03-11T21:27:23.389441+0000","last_active":"2024-03-11T21:29:29.171467+0000","last_peered":"2024-03-11T21:29:29.171467+0000","last_clean":"2024-03-11T21:29:29.171467+0000","last_became_active":"2024-03-11T21:25:44.584565+0000","last_became_peered":"2024-03-11T21:25:44.584565+0000","last_unstale":"2024-03-11T21:29:29.171467+0000","last_undegraded":"2024-03-11T21:29:29.171467+0000","last_fullsized":"2024-03-11T21:29:29.171467+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:27:23.389362+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:43.559452+0000","last_clean_scrub_stamp":"2024-03-11T21:27:23.389362+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":150,"reported_epoch":124,"state":"active+clean","last_fresh":"2024-03-11T21:29:35.214562+0000","last_change":"2024-03-11T21:29:35.214647+0000","last_active":"2024-03-11T21:29:35.214562+0000","last_peered":"2024-03-11T21:29:35.214562+0000","last_clean":"2024-03-11T21:29:35.214562+0000","last_became_active":"2024-03-11T21:25:38.541760+0000","last_became_peered":"2024-03-11T21:25:38.541760+0000","last_unstale":"2024-03-11T21:29:35.214562+0000","last_undegraded":"2024-03-11T21:29:35.214562+0000","last_fullsized":"2024-03-11T21:29:35.214562+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:29:35.214430+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:37.512018+0000","last_clean_scrub_stamp":"2024-03-11T21:29:35.214430+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:30:56.085742+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,7],"acting":[1,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.2","version":"94'15","reported_seq":166,"reported_epoch":124,"state":"active+clean","last_fresh":"2024-03-11T21:29:29.179356+0000","last_change":"2024-03-11T21:29:15.476583+0000","last_active":"2024-03-11T21:29:29.179356+0000","last_peered":"2024-03-11T21:29:29.179356+0000","last_clean":"2024-03-11T21:29:29.179356+0000","last_became_active":"2024-03-11T21:25:40.548833+0000","last_became_peered":"2024-03-11T21:25:40.548833+0000","last_unstale":"2024-03-11T21:29:29.179356+0000","last_undegraded":"2024-03-11T21:29:29.179356+0000","last_fullsized":"2024-03-11T21:29:29.179356+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"94'15","last_scrub_stamp":"2024-03-11T21:29:15.476492+0000","last_deep_scrub":"94'15","last_deep_scrub_stamp":"2024-03-11T21:26:49.993060+0000","last_clean_scrub_stamp":"2024-03-11T21:29:15.476492+0000","objects_scrubbed":5,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:30:32.561565+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.3","version":"74'8","reported_seq":190,"reported_epoch":124,"state":"active+clean","last_fresh":"2024-03-11T21:29:29.172951+0000","last_change":"2024-03-11T21:29:14.348866+0000","last_active":"2024-03-11T21:29:29.172951+0000","last_peered":"2024-03-11T21:29:29.172951+0000","last_clean":"2024-03-11T21:29:29.172951+0000","last_became_active":"2024-03-11T21:25:42.578161+0000","last_became_peered":"2024-03-11T21:25:42.578161+0000","last_unstale":"2024-03-11T21:29:29.172951+0000","last_undegraded":"2024-03-11T21:29:29.172951+0000","last_fullsized":"2024-03-11T21:29:29.172951+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'8","last_scrub_stamp":"2024-03-11T21:29:14.348744+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:41.544550+0000","last_clean_scrub_stamp":"2024-03-11T21:29:14.348744+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:30:42.257563+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,5],"acting":[0,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.7","version":"124'8942","reported_seq":9084,"reported_epoch":124,"state":"active+clean","last_fresh":"2024-03-11T21:29:37.017409+0000","last_change":"2024-03-11T21:29:14.421348+0000","last_active":"2024-03-11T21:29:37.017409+0000","last_peered":"2024-03-11T21:29:37.017409+0000","last_clean":"2024-03-11T21:29:37.017409+0000","last_became_active":"2024-03-11T21:27:36.695055+0000","last_became_peered":"2024-03-11T21:27:36.695055+0000","last_unstale":"2024-03-11T21:29:37.017409+0000","last_undegraded":"2024-03-11T21:29:37.017409+0000","last_fullsized":"2024-03-11T21:29:37.017409+0000","mapping_epoch":99,"log_start":"106'2600","ondisk_log_start":"106'2600","created":99,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"120'7638","last_scrub_stamp":"2024-03-11T21:29:14.421096+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:27:35.667522+0000","last_clean_scrub_stamp":"2024-03-11T21:29:14.421096+0000","objects_scrubbed":7542,"log_size":6342,"log_dups_size":0,"ondisk_log_size":6342,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:30:26.187612+0000","scrub_duration":2593,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":417202176,"num_objects":6366,"num_object_clones":0,"num_object_copies":19098,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6366,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8942,"num_write_kb":489856,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,3],"acting":[6,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":129,"reported_epoch":124,"state":"active+clean","last_fresh":"2024-03-11T21:29:29.182184+0000","last_change":"2024-03-11T21:28:30.563856+0000","last_active":"2024-03-11T21:29:29.182184+0000","last_peered":"2024-03-11T21:29:29.182184+0000","last_clean":"2024-03-11T21:29:29.182184+0000","last_became_active":"2024-03-11T21:25:44.585223+0000","last_became_peered":"2024-03-11T21:25:44.585223+0000","last_unstale":"2024-03-11T21:29:29.182184+0000","last_undegraded":"2024-03-11T21:29:29.182184+0000","last_fullsized":"2024-03-11T21:29:29.182184+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:28:30.563712+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:28:30.563712+0000","last_clean_scrub_stamp":"2024-03-11T21:28:30.563712+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:29:35.182681+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,7],"acting":[3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":149,"reported_epoch":124,"state":"active+clean","last_fresh":"2024-03-11T21:29:31.912841+0000","last_change":"2024-03-11T21:29:31.912865+0000","last_active":"2024-03-11T21:29:31.912841+0000","last_peered":"2024-03-11T21:29:31.912841+0000","last_clean":"2024-03-11T21:29:31.912841+0000","last_became_active":"2024-03-11T21:25:38.538540+0000","last_became_peered":"2024-03-11T21:25:38.538540+0000","last_unstale":"2024-03-11T21:29:31.912841+0000","last_undegraded":"2024-03-11T21:29:31.912841+0000","last_fullsized":"2024-03-11T21:29:31.912841+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:29:31.912762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:37.512018+0000","last_clean_scrub_stamp":"2024-03-11T21:29:31.912762+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:30:47.746683+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.4","version":"74'28","reported_seq":234,"reported_epoch":124,"state":"active+clean","last_fresh":"2024-03-11T21:29:29.177955+0000","last_change":"2024-03-11T21:28:21.455801+0000","last_active":"2024-03-11T21:29:29.177955+0000","last_peered":"2024-03-11T21:29:29.177955+0000","last_clean":"2024-03-11T21:29:29.177955+0000","last_became_active":"2024-03-11T21:25:40.552902+0000","last_became_peered":"2024-03-11T21:25:40.552902+0000","last_unstale":"2024-03-11T21:29:29.177955+0000","last_undegraded":"2024-03-11T21:29:29.177955+0000","last_fullsized":"2024-03-11T21:29:29.177955+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'28","last_scrub_stamp":"2024-03-11T21:28:21.455629+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:39.525554+0000","last_clean_scrub_stamp":"2024-03-11T21:28:21.455629+0000","objects_scrubbed":10,"log_size":28,"log_dups_size":0,"ondisk_log_size":28,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:29:48.292369+0000","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":358,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":48,"num_read_kb":33,"num_write":26,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3],"acting":[1,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":126,"reported_epoch":124,"state":"active+clean","last_fresh":"2024-03-11T21:29:29.176863+0000","last_change":"2024-03-11T21:27:20.393288+0000","last_active":"2024-03-11T21:29:29.176863+0000","last_peered":"2024-03-11T21:29:29.176863+0000","last_clean":"2024-03-11T21:29:29.176863+0000","last_became_active":"2024-03-11T21:25:42.572673+0000","last_became_peered":"2024-03-11T21:25:42.572673+0000","last_unstale":"2024-03-11T21:29:29.176863+0000","last_undegraded":"2024-03-11T21:29:29.176863+0000","last_fullsized":"2024-03-11T21:29:29.176863+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:27:20.393205+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:41.544550+0000","last_clean_scrub_stamp":"2024-03-11T21:27:20.393205+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.4","version":"124'8980","reported_seq":9040,"reported_epoch":124,"state":"active+clean","last_fresh":"2024-03-11T21:29:37.160667+0000","last_change":"2024-03-11T21:27:36.691053+0000","last_active":"2024-03-11T21:29:37.160667+0000","last_peered":"2024-03-11T21:29:37.160667+0000","last_clean":"2024-03-11T21:29:37.160667+0000","last_became_active":"2024-03-11T21:27:36.690751+0000","last_became_peered":"2024-03-11T21:27:36.690751+0000","last_unstale":"2024-03-11T21:29:37.160667+0000","last_undegraded":"2024-03-11T21:29:37.160667+0000","last_fullsized":"2024-03-11T21:29:37.160667+0000","mapping_epoch":99,"log_start":"107'3500","ondisk_log_start":"107'3500","created":99,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:27:35.667522+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:27:35.667522+0000","last_clean_scrub_stamp":"2024-03-11T21:27:35.667522+0000","objects_scrubbed":0,"log_size":5480,"log_dups_size":0,"ondisk_log_size":5480,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":415105024,"num_objects":6334,"num_object_clones":0,"num_object_copies":19002,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6334,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8980,"num_write_kb":490048,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,1],"acting":[0,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.5","version":"0'0","reported_seq":129,"reported_epoch":124,"state":"active+clean","last_fresh":"2024-03-11T21:29:29.175660+0000","last_change":"2024-03-11T21:29:13.477555+0000","last_active":"2024-03-11T21:29:29.175660+0000","last_peered":"2024-03-11T21:29:29.175660+0000","last_clean":"2024-03-11T21:29:29.175660+0000","last_became_active":"2024-03-11T21:25:45.302120+0000","last_became_peered":"2024-03-11T21:25:45.302120+0000","last_unstale":"2024-03-11T21:29:29.175660+0000","last_undegraded":"2024-03-11T21:29:29.175660+0000","last_fullsized":"2024-03-11T21:29:29.175660+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:29:13.477470+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:43.559452+0000","last_clean_scrub_stamp":"2024-03-11T21:29:13.477470+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:30:15.300585+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,3],"acting":[7,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.0","version":"76'5","reported_seq":385,"reported_epoch":124,"state":"active+clean","last_fresh":"2024-03-11T21:29:36.032704+0000","last_change":"2024-03-11T21:28:26.389894+0000","last_active":"2024-03-11T21:29:36.032704+0000","last_peered":"2024-03-11T21:29:36.032704+0000","last_clean":"2024-03-11T21:29:36.032704+0000","last_became_active":"2024-03-11T21:25:38.536890+0000","last_became_peered":"2024-03-11T21:25:38.536890+0000","last_unstale":"2024-03-11T21:29:36.032704+0000","last_undegraded":"2024-03-11T21:29:36.032704+0000","last_fullsized":"2024-03-11T21:29:36.032704+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"76'5","last_scrub_stamp":"2024-03-11T21:28:26.389561+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:37.512018+0000","last_clean_scrub_stamp":"2024-03-11T21:28:26.389561+0000","objects_scrubbed":1,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:29:43.918091+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0.000753559,"stat_sum":{"num_bytes":389,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":233,"num_read_kb":228,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,6],"acting":[1,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":141,"reported_epoch":124,"state":"active+clean","last_fresh":"2024-03-11T21:29:29.180113+0000","last_change":"2024-03-11T21:28:17.037045+0000","last_active":"2024-03-11T21:29:29.180113+0000","last_peered":"2024-03-11T21:29:29.180113+0000","last_clean":"2024-03-11T21:29:29.180113+0000","last_became_active":"2024-03-11T21:25:38.532984+0000","last_became_peered":"2024-03-11T21:25:38.532984+0000","last_unstale":"2024-03-11T21:29:29.180113+0000","last_undegraded":"2024-03-11T21:29:29.180113+0000","last_fullsized":"2024-03-11T21:29:29.180113+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:28:17.036907+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:26:48.997454+0000","last_clean_scrub_stamp":"2024-03-11T21:28:17.036907+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.7","version":"97'18","reported_seq":170,"reported_epoch":124,"state":"active+clean","last_fresh":"2024-03-11T21:29:29.177723+0000","last_change":"2024-03-11T21:28:33.258307+0000","last_active":"2024-03-11T21:29:29.177723+0000","last_peered":"2024-03-11T21:29:29.177723+0000","last_clean":"2024-03-11T21:29:29.177723+0000","last_became_active":"2024-03-11T21:25:40.541769+0000","last_became_peered":"2024-03-11T21:25:40.541769+0000","last_unstale":"2024-03-11T21:29:29.177723+0000","last_undegraded":"2024-03-11T21:29:29.177723+0000","last_fullsized":"2024-03-11T21:29:29.177723+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"97'18","last_scrub_stamp":"2024-03-11T21:28:33.258170+0000","last_deep_scrub":"97'18","last_deep_scrub_stamp":"2024-03-11T21:27:19.239404+0000","last_clean_scrub_stamp":"2024-03-11T21:28:33.258170+0000","objects_scrubbed":5,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:29:43.862175+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":19,"num_write":16,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":133,"reported_epoch":124,"state":"active+clean","last_fresh":"2024-03-11T21:29:29.182340+0000","last_change":"2024-03-11T21:29:11.218638+0000","last_active":"2024-03-11T21:29:29.182340+0000","last_peered":"2024-03-11T21:29:29.182340+0000","last_clean":"2024-03-11T21:29:29.182340+0000","last_became_active":"2024-03-11T21:25:42.581710+0000","last_became_peered":"2024-03-11T21:25:42.581710+0000","last_unstale":"2024-03-11T21:29:29.182340+0000","last_undegraded":"2024-03-11T21:29:29.182340+0000","last_fullsized":"2024-03-11T21:29:29.182340+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:29:11.218424+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:41.544550+0000","last_clean_scrub_stamp":"2024-03-11T21:29:11.218424+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:30:14.242205+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7],"acting":[2,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.6","version":"124'8926","reported_seq":9064,"reported_epoch":124,"state":"active+clean","last_fresh":"2024-03-11T21:29:37.200242+0000","last_change":"2024-03-11T21:29:28.397100+0000","last_active":"2024-03-11T21:29:37.200242+0000","last_peered":"2024-03-11T21:29:37.200242+0000","last_clean":"2024-03-11T21:29:37.200242+0000","last_became_active":"2024-03-11T21:27:36.695018+0000","last_became_peered":"2024-03-11T21:27:36.695018+0000","last_unstale":"2024-03-11T21:29:37.200242+0000","last_undegraded":"2024-03-11T21:29:37.200242+0000","last_fullsized":"2024-03-11T21:29:37.200242+0000","mapping_epoch":99,"log_start":"107'3400","ondisk_log_start":"107'3400","created":99,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"123'8430","last_scrub_stamp":"2024-03-11T21:29:28.396488+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:27:35.667522+0000","last_clean_scrub_stamp":"2024-03-11T21:29:28.396488+0000","objects_scrubbed":6922,"log_size":5526,"log_dups_size":0,"ondisk_log_size":5526,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":4,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:30:41.089781+0000","scrub_duration":3034,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":416546816,"num_objects":6356,"num_object_clones":0,"num_object_copies":19068,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6356,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8926,"num_write_kb":489024,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.7","version":"124'2","reported_seq":131,"reported_epoch":124,"state":"active+clean","last_fresh":"2024-03-11T21:29:34.956971+0000","last_change":"2024-03-11T21:29:34.957004+0000","last_active":"2024-03-11T21:29:34.956971+0000","last_peered":"2024-03-11T21:29:34.956971+0000","last_clean":"2024-03-11T21:29:34.956971+0000","last_became_active":"2024-03-11T21:25:44.591376+0000","last_became_peered":"2024-03-11T21:25:44.591376+0000","last_unstale":"2024-03-11T21:29:34.956971+0000","last_undegraded":"2024-03-11T21:29:34.956971+0000","last_fullsized":"2024-03-11T21:29:34.956971+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"124'2","last_scrub_stamp":"2024-03-11T21:29:34.956908+0000","last_deep_scrub":"124'2","last_deep_scrub_stamp":"2024-03-11T21:29:34.956908+0000","last_clean_scrub_stamp":"2024-03-11T21:29:34.956908+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:30:59.701177+0000","scrub_duration":68,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,4],"acting":[5,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2","version":"71'2","reported_seq":150,"reported_epoch":124,"state":"active+clean","last_fresh":"2024-03-11T21:29:29.181953+0000","last_change":"2024-03-11T21:28:15.701596+0000","last_active":"2024-03-11T21:29:29.181953+0000","last_peered":"2024-03-11T21:29:29.181953+0000","last_clean":"2024-03-11T21:29:29.181953+0000","last_became_active":"2024-03-11T21:25:38.539413+0000","last_became_peered":"2024-03-11T21:25:38.539413+0000","last_unstale":"2024-03-11T21:29:29.181953+0000","last_undegraded":"2024-03-11T21:29:29.181953+0000","last_fullsized":"2024-03-11T21:29:29.181953+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"71'2","last_scrub_stamp":"2024-03-11T21:28:15.701477+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:37.512018+0000","last_clean_scrub_stamp":"2024-03-11T21:28:15.701477+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0.0014206449999999999,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":155,"reported_epoch":124,"state":"active+clean","last_fresh":"2024-03-11T21:29:30.862463+0000","last_change":"2024-03-11T21:29:30.862499+0000","last_active":"2024-03-11T21:29:30.862463+0000","last_peered":"2024-03-11T21:29:30.862463+0000","last_clean":"2024-03-11T21:29:30.862463+0000","last_became_active":"2024-03-11T21:26:11.216188+0000","last_became_peered":"2024-03-11T21:26:11.216188+0000","last_unstale":"2024-03-11T21:29:30.862463+0000","last_undegraded":"2024-03-11T21:29:30.862463+0000","last_fullsized":"2024-03-11T21:29:30.862463+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:29:30.862374+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:37.512018+0000","last_clean_scrub_stamp":"2024-03-11T21:29:30.862374+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:31:00.536669+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,4],"acting":[5,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"87'128","reported_seq":396,"reported_epoch":124,"state":"active+clean","last_fresh":"2024-03-11T21:29:29.175662+0000","last_change":"2024-03-11T21:29:12.458886+0000","last_active":"2024-03-11T21:29:29.175662+0000","last_peered":"2024-03-11T21:29:29.175662+0000","last_clean":"2024-03-11T21:29:29.175662+0000","last_became_active":"2024-03-11T21:25:30.502597+0000","last_became_peered":"2024-03-11T21:25:30.502597+0000","last_unstale":"2024-03-11T21:29:29.175662+0000","last_undegraded":"2024-03-11T21:29:29.175662+0000","last_fullsized":"2024-03-11T21:29:29.175662+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"87'128","last_scrub_stamp":"2024-03-11T21:29:12.458790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:24.261974+0000","last_clean_scrub_stamp":"2024-03-11T21:29:12.458790+0000","objects_scrubbed":2,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:30:13.969704+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":212,"num_read_kb":306,"num_write":179,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1771104,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.5","version":"74'16","reported_seq":208,"reported_epoch":124,"state":"active+clean","last_fresh":"2024-03-11T21:29:29.178336+0000","last_change":"2024-03-11T21:28:30.831871+0000","last_active":"2024-03-11T21:29:29.178336+0000","last_peered":"2024-03-11T21:29:29.178336+0000","last_clean":"2024-03-11T21:29:29.178336+0000","last_became_active":"2024-03-11T21:25:40.761457+0000","last_became_peered":"2024-03-11T21:25:40.761457+0000","last_unstale":"2024-03-11T21:29:29.178336+0000","last_undegraded":"2024-03-11T21:29:29.178336+0000","last_fullsized":"2024-03-11T21:29:29.178336+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'16","last_scrub_stamp":"2024-03-11T21:28:30.831780+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:39.525554+0000","last_clean_scrub_stamp":"2024-03-11T21:28:30.831780+0000","objects_scrubbed":8,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:29:54.428390+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":154,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":25,"num_read_kb":15,"num_write":13,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":141,"reported_epoch":124,"state":"active+clean","last_fresh":"2024-03-11T21:29:29.175426+0000","last_change":"2024-03-11T21:29:27.423931+0000","last_active":"2024-03-11T21:29:29.175426+0000","last_peered":"2024-03-11T21:29:29.175426+0000","last_clean":"2024-03-11T21:29:29.175426+0000","last_became_active":"2024-03-11T21:25:42.586226+0000","last_became_peered":"2024-03-11T21:25:42.586226+0000","last_unstale":"2024-03-11T21:29:29.175426+0000","last_undegraded":"2024-03-11T21:29:29.175426+0000","last_fullsized":"2024-03-11T21:29:29.175426+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:29:27.423847+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:41.544550+0000","last_clean_scrub_stamp":"2024-03-11T21:29:27.423847+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:30:27.900641+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,5],"acting":[7,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.5","version":"124'8871","reported_seq":9010,"reported_epoch":124,"state":"active+clean","last_fresh":"2024-03-11T21:29:36.983341+0000","last_change":"2024-03-11T21:29:25.044218+0000","last_active":"2024-03-11T21:29:36.983341+0000","last_peered":"2024-03-11T21:29:36.983341+0000","last_clean":"2024-03-11T21:29:36.983341+0000","last_became_active":"2024-03-11T21:27:36.691428+0000","last_became_peered":"2024-03-11T21:27:36.691428+0000","last_unstale":"2024-03-11T21:29:36.983341+0000","last_undegraded":"2024-03-11T21:29:36.983341+0000","last_fullsized":"2024-03-11T21:29:36.983341+0000","mapping_epoch":99,"log_start":"108'3900","ondisk_log_start":"108'3900","created":99,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"122'8195","last_scrub_stamp":"2024-03-11T21:29:25.043785+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:27:35.667522+0000","last_clean_scrub_stamp":"2024-03-11T21:29:25.043785+0000","objects_scrubbed":7129,"log_size":4971,"log_dups_size":0,"ondisk_log_size":4971,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:30:27.426053+0000","scrub_duration":2517,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":418447360,"num_objects":6385,"num_object_clones":0,"num_object_copies":19155,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6385,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8871,"num_write_kb":488192,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,0],"acting":[7,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.4","version":"0'0","reported_seq":130,"reported_epoch":124,"state":"active+clean","last_fresh":"2024-03-11T21:29:29.179724+0000","last_change":"2024-03-11T21:28:29.376413+0000","last_active":"2024-03-11T21:29:29.179724+0000","last_peered":"2024-03-11T21:29:29.179724+0000","last_clean":"2024-03-11T21:29:29.179724+0000","last_became_active":"2024-03-11T21:25:44.594188+0000","last_became_peered":"2024-03-11T21:25:44.594188+0000","last_unstale":"2024-03-11T21:29:29.179724+0000","last_undegraded":"2024-03-11T21:29:29.179724+0000","last_fullsized":"2024-03-11T21:29:29.179724+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:28:29.375618+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:43.559452+0000","last_clean_scrub_stamp":"2024-03-11T21:28:29.375618+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:29:49.691757+0000","scrub_duration":32,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":149,"reported_epoch":124,"state":"active+clean","last_fresh":"2024-03-11T21:29:29.171363+0000","last_change":"2024-03-11T21:28:06.322812+0000","last_active":"2024-03-11T21:29:29.171363+0000","last_peered":"2024-03-11T21:29:29.171363+0000","last_clean":"2024-03-11T21:29:29.171363+0000","last_became_active":"2024-03-11T21:25:38.543834+0000","last_became_peered":"2024-03-11T21:25:38.543834+0000","last_unstale":"2024-03-11T21:29:29.171363+0000","last_undegraded":"2024-03-11T21:29:29.171363+0000","last_fullsized":"2024-03-11T21:29:29.171363+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:28:06.322702+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:37.512018+0000","last_clean_scrub_stamp":"2024-03-11T21:28:06.322702+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:29:32.617923+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0.00089342499999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":149,"reported_epoch":124,"state":"active+clean","last_fresh":"2024-03-11T21:29:29.174941+0000","last_change":"2024-03-11T21:29:24.524752+0000","last_active":"2024-03-11T21:29:29.174941+0000","last_peered":"2024-03-11T21:29:29.174941+0000","last_clean":"2024-03-11T21:29:29.174941+0000","last_became_active":"2024-03-11T21:25:38.541617+0000","last_became_peered":"2024-03-11T21:25:38.541617+0000","last_unstale":"2024-03-11T21:29:29.174941+0000","last_undegraded":"2024-03-11T21:29:29.174941+0000","last_fullsized":"2024-03-11T21:29:29.174941+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:29:24.524618+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:37.512018+0000","last_clean_scrub_stamp":"2024-03-11T21:29:24.524618+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:30:31.415455+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,0],"acting":[7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.6","version":"74'12","reported_seq":156,"reported_epoch":124,"state":"active+clean","last_fresh":"2024-03-11T21:29:29.171501+0000","last_change":"2024-03-11T21:28:30.335163+0000","last_active":"2024-03-11T21:29:29.171501+0000","last_peered":"2024-03-11T21:29:29.171501+0000","last_clean":"2024-03-11T21:29:29.171501+0000","last_became_active":"2024-03-11T21:25:40.550388+0000","last_became_peered":"2024-03-11T21:25:40.550388+0000","last_unstale":"2024-03-11T21:29:29.171501+0000","last_undegraded":"2024-03-11T21:29:29.171501+0000","last_fullsized":"2024-03-11T21:29:29.171501+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'12","last_scrub_stamp":"2024-03-11T21:28:30.335019+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:39.525554+0000","last_clean_scrub_stamp":"2024-03-11T21:28:30.335019+0000","objects_scrubbed":6,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:29:42.338429+0000","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":18,"num_read_kb":12,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":133,"reported_epoch":124,"state":"active+clean","last_fresh":"2024-03-11T21:29:29.183467+0000","last_change":"2024-03-11T21:29:21.924703+0000","last_active":"2024-03-11T21:29:29.183467+0000","last_peered":"2024-03-11T21:29:29.183467+0000","last_clean":"2024-03-11T21:29:29.183467+0000","last_became_active":"2024-03-11T21:25:42.572324+0000","last_became_peered":"2024-03-11T21:25:42.572324+0000","last_unstale":"2024-03-11T21:29:29.183467+0000","last_undegraded":"2024-03-11T21:29:29.183467+0000","last_fullsized":"2024-03-11T21:29:29.183467+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:29:21.924587+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:41.544550+0000","last_clean_scrub_stamp":"2024-03-11T21:29:21.924587+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:30:43.446075+0000","scrub_duration":16,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.0","version":"124'8710","reported_seq":8847,"reported_epoch":124,"state":"active+clean","last_fresh":"2024-03-11T21:29:36.951695+0000","last_change":"2024-03-11T21:29:25.488344+0000","last_active":"2024-03-11T21:29:36.951695+0000","last_peered":"2024-03-11T21:29:36.951695+0000","last_clean":"2024-03-11T21:29:36.951695+0000","last_became_active":"2024-03-11T21:27:36.694232+0000","last_became_peered":"2024-03-11T21:27:36.694232+0000","last_unstale":"2024-03-11T21:29:36.951695+0000","last_undegraded":"2024-03-11T21:29:36.951695+0000","last_fullsized":"2024-03-11T21:29:36.951695+0000","mapping_epoch":99,"log_start":"107'2900","ondisk_log_start":"107'2900","created":99,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"122'8091","last_scrub_stamp":"2024-03-11T21:29:25.488240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:27:35.667522+0000","last_clean_scrub_stamp":"2024-03-11T21:29:25.488240+0000","objects_scrubbed":6917,"log_size":5810,"log_dups_size":0,"ondisk_log_size":5810,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:30:29.899089+0000","scrub_duration":2534,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":408158208,"num_objects":6228,"num_object_clones":0,"num_object_copies":18684,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6228,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8710,"num_write_kb":478016,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3],"acting":[5,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":138,"reported_epoch":124,"state":"active+clean","last_fresh":"2024-03-11T21:29:29.177402+0000","last_change":"2024-03-11T21:29:16.451118+0000","last_active":"2024-03-11T21:29:29.177402+0000","last_peered":"2024-03-11T21:29:29.177402+0000","last_clean":"2024-03-11T21:29:29.177402+0000","last_became_active":"2024-03-11T21:25:45.301416+0000","last_became_peered":"2024-03-11T21:25:45.301416+0000","last_unstale":"2024-03-11T21:29:29.177402+0000","last_undegraded":"2024-03-11T21:29:29.177402+0000","last_fullsized":"2024-03-11T21:29:29.177402+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:29:16.451010+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:43.559452+0000","last_clean_scrub_stamp":"2024-03-11T21:29:16.451010+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:30:18.596466+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":141,"reported_epoch":124,"state":"active+clean","last_fresh":"2024-03-11T21:29:29.175758+0000","last_change":"2024-03-11T21:28:35.467464+0000","last_active":"2024-03-11T21:29:29.175758+0000","last_peered":"2024-03-11T21:29:29.175758+0000","last_clean":"2024-03-11T21:29:29.175758+0000","last_became_active":"2024-03-11T21:25:38.545148+0000","last_became_peered":"2024-03-11T21:25:38.545148+0000","last_unstale":"2024-03-11T21:29:29.175758+0000","last_undegraded":"2024-03-11T21:29:29.175758+0000","last_fullsized":"2024-03-11T21:29:29.175758+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:28:35.467339+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:26:57.333669+0000","last_clean_scrub_stamp":"2024-03-11T21:28:35.467339+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:29:55.375727+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.3","version":"74'19","reported_seq":174,"reported_epoch":124,"state":"active+clean","last_fresh":"2024-03-11T21:29:29.172610+0000","last_change":"2024-03-11T21:28:27.344174+0000","last_active":"2024-03-11T21:29:29.172610+0000","last_peered":"2024-03-11T21:29:29.172610+0000","last_clean":"2024-03-11T21:29:29.172610+0000","last_became_active":"2024-03-11T21:25:40.760594+0000","last_became_peered":"2024-03-11T21:25:40.760594+0000","last_unstale":"2024-03-11T21:29:29.172610+0000","last_undegraded":"2024-03-11T21:29:29.172610+0000","last_fullsized":"2024-03-11T21:29:29.172610+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'19","last_scrub_stamp":"2024-03-11T21:28:27.344026+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:39.525554+0000","last_clean_scrub_stamp":"2024-03-11T21:28:27.344026+0000","objects_scrubbed":8,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:29:39.040372+0000","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":39,"num_read_kb":25,"num_write":22,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":142,"reported_epoch":124,"state":"active+clean","last_fresh":"2024-03-11T21:29:36.918990+0000","last_change":"2024-03-11T21:29:36.919010+0000","last_active":"2024-03-11T21:29:36.918990+0000","last_peered":"2024-03-11T21:29:36.918990+0000","last_clean":"2024-03-11T21:29:36.918990+0000","last_became_active":"2024-03-11T21:25:42.579045+0000","last_became_peered":"2024-03-11T21:25:42.579045+0000","last_unstale":"2024-03-11T21:29:36.918990+0000","last_undegraded":"2024-03-11T21:29:36.918990+0000","last_fullsized":"2024-03-11T21:29:36.918990+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:29:36.918936+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:29:36.918936+0000","last_clean_scrub_stamp":"2024-03-11T21:29:36.918936+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:31:05.295611+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.3","version":"124'8850","reported_seq":8909,"reported_epoch":124,"state":"active+clean","last_fresh":"2024-03-11T21:29:37.158426+0000","last_change":"2024-03-11T21:27:36.693897+0000","last_active":"2024-03-11T21:29:37.158426+0000","last_peered":"2024-03-11T21:29:37.158426+0000","last_clean":"2024-03-11T21:29:37.158426+0000","last_became_active":"2024-03-11T21:27:36.693506+0000","last_became_peered":"2024-03-11T21:27:36.693506+0000","last_unstale":"2024-03-11T21:29:37.158426+0000","last_undegraded":"2024-03-11T21:29:37.158426+0000","last_fullsized":"2024-03-11T21:29:37.158426+0000","mapping_epoch":99,"log_start":"109'4200","ondisk_log_start":"109'4200","created":99,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:27:35.667522+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:27:35.667522+0000","last_clean_scrub_stamp":"2024-03-11T21:27:35.667522+0000","objects_scrubbed":0,"log_size":4650,"log_dups_size":0,"ondisk_log_size":4650,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":410910720,"num_objects":6270,"num_object_clones":0,"num_object_copies":18810,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6270,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8850,"num_write_kb":483840,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,4],"acting":[3,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":130,"reported_epoch":124,"state":"active+clean","last_fresh":"2024-03-11T21:29:29.172096+0000","last_change":"2024-03-11T21:28:27.786854+0000","last_active":"2024-03-11T21:29:29.172096+0000","last_peered":"2024-03-11T21:29:29.172096+0000","last_clean":"2024-03-11T21:29:29.172096+0000","last_became_active":"2024-03-11T21:25:44.583638+0000","last_became_peered":"2024-03-11T21:25:44.583638+0000","last_unstale":"2024-03-11T21:29:29.172096+0000","last_undegraded":"2024-03-11T21:29:29.172096+0000","last_fullsized":"2024-03-11T21:29:29.172096+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:28:27.786740+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:43.559452+0000","last_clean_scrub_stamp":"2024-03-11T21:28:27.786740+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:29:30.840139+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.6","version":"67'1","reported_seq":151,"reported_epoch":124,"state":"active+clean","last_fresh":"2024-03-11T21:29:33.236138+0000","last_change":"2024-03-11T21:29:33.236180+0000","last_active":"2024-03-11T21:29:33.236138+0000","last_peered":"2024-03-11T21:29:33.236138+0000","last_clean":"2024-03-11T21:29:33.236138+0000","last_became_active":"2024-03-11T21:25:38.536142+0000","last_became_peered":"2024-03-11T21:25:38.536142+0000","last_unstale":"2024-03-11T21:29:33.236138+0000","last_undegraded":"2024-03-11T21:29:33.236138+0000","last_fullsized":"2024-03-11T21:29:33.236138+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'1","last_scrub_stamp":"2024-03-11T21:29:33.236064+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:37.512018+0000","last_clean_scrub_stamp":"2024-03-11T21:29:33.236064+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:30:53.084578+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,4],"acting":[1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.0","version":"93'26","reported_seq":177,"reported_epoch":124,"state":"active+clean","last_fresh":"2024-03-11T21:29:29.181638+0000","last_change":"2024-03-11T21:28:18.664637+0000","last_active":"2024-03-11T21:29:29.181638+0000","last_peered":"2024-03-11T21:29:29.181638+0000","last_clean":"2024-03-11T21:29:29.181638+0000","last_became_active":"2024-03-11T21:25:40.761716+0000","last_became_peered":"2024-03-11T21:25:40.761716+0000","last_unstale":"2024-03-11T21:29:29.181638+0000","last_undegraded":"2024-03-11T21:29:29.181638+0000","last_fullsized":"2024-03-11T21:29:29.181638+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"93'26","last_scrub_stamp":"2024-03-11T21:28:18.664409+0000","last_deep_scrub":"93'26","last_deep_scrub_stamp":"2024-03-11T21:26:40.680324+0000","last_clean_scrub_stamp":"2024-03-11T21:28:18.664409+0000","objects_scrubbed":8,"log_size":26,"log_dups_size":0,"ondisk_log_size":26,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:29:35.615493+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":34,"num_read_kb":22,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":134,"reported_epoch":124,"state":"active+clean","last_fresh":"2024-03-11T21:29:29.174423+0000","last_change":"2024-03-11T21:28:23.769677+0000","last_active":"2024-03-11T21:29:29.174423+0000","last_peered":"2024-03-11T21:29:29.174423+0000","last_clean":"2024-03-11T21:29:29.174423+0000","last_became_active":"2024-03-11T21:25:42.583647+0000","last_became_peered":"2024-03-11T21:25:42.583647+0000","last_unstale":"2024-03-11T21:29:29.174423+0000","last_undegraded":"2024-03-11T21:29:29.174423+0000","last_fullsized":"2024-03-11T21:29:29.174423+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:28:23.769586+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:26:51.863621+0000","last_clean_scrub_stamp":"2024-03-11T21:28:23.769586+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:29:32.731064+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,7],"acting":[4,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.2","version":"124'8905","reported_seq":8964,"reported_epoch":124,"state":"active+clean","last_fresh":"2024-03-11T21:29:37.189947+0000","last_change":"2024-03-11T21:27:36.692676+0000","last_active":"2024-03-11T21:29:37.189947+0000","last_peered":"2024-03-11T21:29:37.189947+0000","last_clean":"2024-03-11T21:29:37.189947+0000","last_became_active":"2024-03-11T21:27:36.692405+0000","last_became_peered":"2024-03-11T21:27:36.692405+0000","last_unstale":"2024-03-11T21:29:37.189947+0000","last_undegraded":"2024-03-11T21:29:37.189947+0000","last_fullsized":"2024-03-11T21:29:37.189947+0000","mapping_epoch":99,"log_start":"108'4223","ondisk_log_start":"108'4223","created":99,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:27:35.667522+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:27:35.667522+0000","last_clean_scrub_stamp":"2024-03-11T21:27:35.667522+0000","objects_scrubbed":0,"log_size":4682,"log_dups_size":0,"ondisk_log_size":4682,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":416612352,"num_objects":6357,"num_object_clones":0,"num_object_copies":19071,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6357,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8905,"num_write_kb":488384,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.3","version":"0'0","reported_seq":129,"reported_epoch":124,"state":"active+clean","last_fresh":"2024-03-11T21:29:29.175357+0000","last_change":"2024-03-11T21:28:25.493599+0000","last_active":"2024-03-11T21:29:29.175357+0000","last_peered":"2024-03-11T21:29:29.175357+0000","last_clean":"2024-03-11T21:29:29.175357+0000","last_became_active":"2024-03-11T21:25:45.302719+0000","last_became_peered":"2024-03-11T21:25:45.302719+0000","last_unstale":"2024-03-11T21:29:29.175357+0000","last_undegraded":"2024-03-11T21:29:29.175357+0000","last_fullsized":"2024-03-11T21:29:29.175357+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:28:25.493325+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:28:25.493325+0000","last_clean_scrub_stamp":"2024-03-11T21:28:25.493325+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,2],"acting":[7,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":150,"reported_epoch":124,"state":"active+clean","last_fresh":"2024-03-11T21:29:29.178326+0000","last_change":"2024-03-11T21:29:25.893663+0000","last_active":"2024-03-11T21:29:29.178326+0000","last_peered":"2024-03-11T21:29:29.178326+0000","last_clean":"2024-03-11T21:29:29.178326+0000","last_became_active":"2024-03-11T21:25:38.540013+0000","last_became_peered":"2024-03-11T21:25:38.540013+0000","last_unstale":"2024-03-11T21:29:29.178326+0000","last_undegraded":"2024-03-11T21:29:29.178326+0000","last_fullsized":"2024-03-11T21:29:29.178326+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:29:25.893481+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:37.512018+0000","last_clean_scrub_stamp":"2024-03-11T21:29:25.893481+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:30:33.229119+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,2],"acting":[6,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.1","version":"74'14","reported_seq":159,"reported_epoch":124,"state":"active+clean","last_fresh":"2024-03-11T21:29:29.174224+0000","last_change":"2024-03-11T21:29:20.149316+0000","last_active":"2024-03-11T21:29:29.174224+0000","last_peered":"2024-03-11T21:29:29.174224+0000","last_clean":"2024-03-11T21:29:29.174224+0000","last_became_active":"2024-03-11T21:25:40.546219+0000","last_became_peered":"2024-03-11T21:25:40.546219+0000","last_unstale":"2024-03-11T21:29:29.174224+0000","last_undegraded":"2024-03-11T21:29:29.174224+0000","last_fullsized":"2024-03-11T21:29:29.174224+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'14","last_scrub_stamp":"2024-03-11T21:29:20.149195+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:39.525554+0000","last_clean_scrub_stamp":"2024-03-11T21:29:20.149195+0000","objects_scrubbed":7,"log_size":14,"log_dups_size":0,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:30:27.628593+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":21,"num_read_kb":14,"num_write":14,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6],"acting":[4,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.0","version":"98'9","reported_seq":190,"reported_epoch":124,"state":"active+clean","last_fresh":"2024-03-11T21:29:29.181585+0000","last_change":"2024-03-11T21:29:24.578770+0000","last_active":"2024-03-11T21:29:29.181585+0000","last_peered":"2024-03-11T21:29:29.181585+0000","last_clean":"2024-03-11T21:29:29.181585+0000","last_became_active":"2024-03-11T21:25:42.576126+0000","last_became_peered":"2024-03-11T21:25:42.576126+0000","last_unstale":"2024-03-11T21:29:29.181585+0000","last_undegraded":"2024-03-11T21:29:29.181585+0000","last_fullsized":"2024-03-11T21:29:29.181585+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"98'9","last_scrub_stamp":"2024-03-11T21:29:24.578653+0000","last_deep_scrub":"98'9","last_deep_scrub_stamp":"2024-03-11T21:27:29.745711+0000","last_clean_scrub_stamp":"2024-03-11T21:29:24.578653+0000","objects_scrubbed":1,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:30:41.571039+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.d","version":"124'8944","reported_seq":9086,"reported_epoch":124,"state":"active+clean","last_fresh":"2024-03-11T21:29:36.947533+0000","last_change":"2024-03-11T21:29:24.416513+0000","last_active":"2024-03-11T21:29:36.947533+0000","last_peered":"2024-03-11T21:29:36.947533+0000","last_clean":"2024-03-11T21:29:36.947533+0000","last_became_active":"2024-03-11T21:27:36.695186+0000","last_became_peered":"2024-03-11T21:27:36.695186+0000","last_unstale":"2024-03-11T21:29:36.947533+0000","last_undegraded":"2024-03-11T21:29:36.947533+0000","last_fullsized":"2024-03-11T21:29:36.947533+0000","mapping_epoch":99,"log_start":"106'2600","ondisk_log_start":"106'2600","created":99,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"122'8194","last_scrub_stamp":"2024-03-11T21:29:24.416369+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:27:35.667522+0000","last_clean_scrub_stamp":"2024-03-11T21:29:24.416369+0000","objects_scrubbed":7168,"log_size":6344,"log_dups_size":0,"ondisk_log_size":6344,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:30:52.718192+0000","scrub_duration":2563,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":417136664,"num_objects":6366,"num_object_clones":0,"num_object_copies":19098,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6366,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":8944,"num_write_kb":489857,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,0],"acting":[6,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.c","version":"0'0","reported_seq":121,"reported_epoch":124,"state":"active+clean","last_fresh":"2024-03-11T21:29:29.183792+0000","last_change":"2024-03-11T21:28:02.703398+0000","last_active":"2024-03-11T21:29:29.183792+0000","last_peered":"2024-03-11T21:29:29.183792+0000","last_clean":"2024-03-11T21:29:29.183792+0000","last_became_active":"2024-03-11T21:25:45.302207+0000","last_became_peered":"2024-03-11T21:25:45.302207+0000","last_unstale":"2024-03-11T21:29:29.183792+0000","last_undegraded":"2024-03-11T21:29:29.183792+0000","last_fullsized":"2024-03-11T21:29:29.183792+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:28:02.703154+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:43.559452+0000","last_clean_scrub_stamp":"2024-03-11T21:28:02.703154+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:29:31.096224+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.f","version":"0'0","reported_seq":141,"reported_epoch":124,"state":"active+clean","last_fresh":"2024-03-11T21:29:29.183286+0000","last_change":"2024-03-11T21:29:28.844148+0000","last_active":"2024-03-11T21:29:29.183286+0000","last_peered":"2024-03-11T21:29:29.183286+0000","last_clean":"2024-03-11T21:29:29.183286+0000","last_became_active":"2024-03-11T21:25:42.575935+0000","last_became_peered":"2024-03-11T21:25:42.575935+0000","last_unstale":"2024-03-11T21:29:29.183286+0000","last_undegraded":"2024-03-11T21:29:29.183286+0000","last_fullsized":"2024-03-11T21:29:29.183286+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:29:28.844034+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:41.544550+0000","last_clean_scrub_stamp":"2024-03-11T21:29:28.844034+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:30:58.088688+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.8","version":"0'0","reported_seq":141,"reported_epoch":124,"state":"active+clean","last_fresh":"2024-03-11T21:29:29.175609+0000","last_change":"2024-03-11T21:28:29.562522+0000","last_active":"2024-03-11T21:29:29.175609+0000","last_peered":"2024-03-11T21:29:29.175609+0000","last_clean":"2024-03-11T21:29:29.175609+0000","last_became_active":"2024-03-11T21:25:38.540910+0000","last_became_peered":"2024-03-11T21:25:38.540910+0000","last_unstale":"2024-03-11T21:29:29.175609+0000","last_undegraded":"2024-03-11T21:29:29.175609+0000","last_fullsized":"2024-03-11T21:29:29.175609+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:28:29.562407+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:37.512018+0000","last_clean_scrub_stamp":"2024-03-11T21:28:29.562407+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:29:56.105469+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,1],"acting":[7,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.e","version":"109'16","reported_seq":162,"reported_epoch":124,"state":"active+clean","last_fresh":"2024-03-11T21:29:29.173162+0000","last_change":"2024-03-11T21:28:24.739573+0000","last_active":"2024-03-11T21:29:29.173162+0000","last_peered":"2024-03-11T21:29:29.173162+0000","last_clean":"2024-03-11T21:29:29.173162+0000","last_became_active":"2024-03-11T21:25:40.552178+0000","last_became_peered":"2024-03-11T21:25:40.552178+0000","last_unstale":"2024-03-11T21:29:29.173162+0000","last_undegraded":"2024-03-11T21:29:29.173162+0000","last_fullsized":"2024-03-11T21:29:29.173162+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"109'16","last_scrub_stamp":"2024-03-11T21:28:24.739395+0000","last_deep_scrub":"109'16","last_deep_scrub_stamp":"2024-03-11T21:28:24.739395+0000","last_clean_scrub_stamp":"2024-03-11T21:28:24.739395+0000","objects_scrubbed":5,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:29:43.197007+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.c","version":"124'8833","reported_seq":8974,"reported_epoch":124,"state":"active+clean","last_fresh":"2024-03-11T21:29:37.014147+0000","last_change":"2024-03-11T21:29:19.524086+0000","last_active":"2024-03-11T21:29:37.014147+0000","last_peered":"2024-03-11T21:29:37.014147+0000","last_clean":"2024-03-11T21:29:37.014147+0000","last_became_active":"2024-03-11T21:27:36.693325+0000","last_became_peered":"2024-03-11T21:27:36.693325+0000","last_unstale":"2024-03-11T21:29:37.014147+0000","last_undegraded":"2024-03-11T21:29:37.014147+0000","last_fullsized":"2024-03-11T21:29:37.014147+0000","mapping_epoch":99,"log_start":"106'2500","ondisk_log_start":"106'2500","created":99,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"120'7860","last_scrub_stamp":"2024-03-11T21:29:19.523794+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:27:35.667522+0000","last_clean_scrub_stamp":"2024-03-11T21:29:19.523794+0000","objects_scrubbed":7382,"log_size":6333,"log_dups_size":0,"ondisk_log_size":6333,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:30:30.468530+0000","scrub_duration":1664,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":416612352,"num_objects":6357,"num_object_clones":0,"num_object_copies":19071,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6357,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8833,"num_write_kb":486080,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,7],"acting":[6,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.d","version":"0'0","reported_seq":129,"reported_epoch":124,"state":"active+clean","last_fresh":"2024-03-11T21:29:29.182857+0000","last_change":"2024-03-11T21:28:23.796927+0000","last_active":"2024-03-11T21:29:29.182857+0000","last_peered":"2024-03-11T21:29:29.182857+0000","last_clean":"2024-03-11T21:29:29.182857+0000","last_became_active":"2024-03-11T21:25:44.590333+0000","last_became_peered":"2024-03-11T21:25:44.590333+0000","last_unstale":"2024-03-11T21:29:29.182857+0000","last_undegraded":"2024-03-11T21:29:29.182857+0000","last_fullsized":"2024-03-11T21:29:29.182857+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:28:23.796842+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:43.559452+0000","last_clean_scrub_stamp":"2024-03-11T21:28:23.796842+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:29:49.049133+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.e","version":"74'8","reported_seq":190,"reported_epoch":124,"state":"active+clean","last_fresh":"2024-03-11T21:29:31.222014+0000","last_change":"2024-03-11T21:29:31.222014+0000","last_active":"2024-03-11T21:29:31.222014+0000","last_peered":"2024-03-11T21:29:31.222014+0000","last_clean":"2024-03-11T21:29:31.222014+0000","last_became_active":"2024-03-11T21:25:42.576742+0000","last_became_peered":"2024-03-11T21:25:42.576742+0000","last_unstale":"2024-03-11T21:29:31.222014+0000","last_undegraded":"2024-03-11T21:29:31.222014+0000","last_fullsized":"2024-03-11T21:29:31.222014+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'8","last_scrub_stamp":"2024-03-11T21:29:31.221924+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:41.544550+0000","last_clean_scrub_stamp":"2024-03-11T21:29:31.221924+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:30:37.328426+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,0],"acting":[4,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.9","version":"0'0","reported_seq":150,"reported_epoch":124,"state":"active+clean","last_fresh":"2024-03-11T21:29:29.177219+0000","last_change":"2024-03-11T21:29:18.412164+0000","last_active":"2024-03-11T21:29:29.177219+0000","last_peered":"2024-03-11T21:29:29.177219+0000","last_clean":"2024-03-11T21:29:29.177219+0000","last_became_active":"2024-03-11T21:25:38.540976+0000","last_became_peered":"2024-03-11T21:25:38.540976+0000","last_unstale":"2024-03-11T21:29:29.177219+0000","last_undegraded":"2024-03-11T21:29:29.177219+0000","last_fullsized":"2024-03-11T21:29:29.177219+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:29:18.411652+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:26:50.991712+0000","last_clean_scrub_stamp":"2024-03-11T21:29:18.411652+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:30:46.625284+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,3],"acting":[1,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.f","version":"74'15","reported_seq":115,"reported_epoch":124,"state":"active+clean","last_fresh":"2024-03-11T21:29:29.180845+0000","last_change":"2024-03-11T21:27:28.081459+0000","last_active":"2024-03-11T21:29:29.180845+0000","last_peered":"2024-03-11T21:29:29.180845+0000","last_clean":"2024-03-11T21:29:29.180845+0000","last_became_active":"2024-03-11T21:26:11.216224+0000","last_became_peered":"2024-03-11T21:26:11.216224+0000","last_unstale":"2024-03-11T21:29:29.180845+0000","last_undegraded":"2024-03-11T21:29:29.180845+0000","last_fullsized":"2024-03-11T21:29:29.180845+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'15","last_scrub_stamp":"2024-03-11T21:27:28.081356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:39.525554+0000","last_clean_scrub_stamp":"2024-03-11T21:27:28.081356+0000","objects_scrubbed":7,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,4],"acting":[2,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.f","version":"124'9477","reported_seq":9536,"reported_epoch":124,"state":"active+clean+scrubbing+deep","last_fresh":"2024-03-11T21:29:37.221208+0000","last_change":"2024-03-11T21:29:33.365993+0000","last_active":"2024-03-11T21:29:37.221208+0000","last_peered":"2024-03-11T21:29:37.221208+0000","last_clean":"2024-03-11T21:29:37.221208+0000","last_became_active":"2024-03-11T21:27:36.694032+0000","last_became_peered":"2024-03-11T21:27:36.694032+0000","last_unstale":"2024-03-11T21:29:37.221208+0000","last_undegraded":"2024-03-11T21:29:37.221208+0000","last_fullsized":"2024-03-11T21:29:37.221208+0000","mapping_epoch":99,"log_start":"108'4001","ondisk_log_start":"108'4001","created":99,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:27:35.667522+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:27:35.667522+0000","last_clean_scrub_stamp":"2024-03-11T21:27:35.667522+0000","objects_scrubbed":524,"log_size":5476,"log_dups_size":0,"ondisk_log_size":5476,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 3s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":419299328,"num_objects":6398,"num_object_clones":0,"num_object_copies":19194,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6398,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9004,"num_write_kb":492864,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.e","version":"0'0","reported_seq":130,"reported_epoch":124,"state":"active+clean","last_fresh":"2024-03-11T21:29:34.312189+0000","last_change":"2024-03-11T21:29:34.312222+0000","last_active":"2024-03-11T21:29:34.312189+0000","last_peered":"2024-03-11T21:29:34.312189+0000","last_clean":"2024-03-11T21:29:34.312189+0000","last_became_active":"2024-03-11T21:25:44.590198+0000","last_became_peered":"2024-03-11T21:25:44.590198+0000","last_unstale":"2024-03-11T21:29:34.312189+0000","last_undegraded":"2024-03-11T21:29:34.312189+0000","last_fullsized":"2024-03-11T21:29:34.312189+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:29:34.312119+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:43.559452+0000","last_clean_scrub_stamp":"2024-03-11T21:29:34.312119+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:30:44.187207+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.d","version":"109'9","reported_seq":190,"reported_epoch":124,"state":"active+clean","last_fresh":"2024-03-11T21:29:29.188391+0000","last_change":"2024-03-11T21:28:27.083117+0000","last_active":"2024-03-11T21:29:29.188391+0000","last_peered":"2024-03-11T21:29:29.188391+0000","last_clean":"2024-03-11T21:29:29.188391+0000","last_became_active":"2024-03-11T21:25:42.575583+0000","last_became_peered":"2024-03-11T21:25:42.575583+0000","last_unstale":"2024-03-11T21:29:29.188391+0000","last_undegraded":"2024-03-11T21:29:29.188391+0000","last_fullsized":"2024-03-11T21:29:29.188391+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"109'9","last_scrub_stamp":"2024-03-11T21:28:27.082975+0000","last_deep_scrub":"109'9","last_deep_scrub_stamp":"2024-03-11T21:28:27.082975+0000","last_clean_scrub_stamp":"2024-03-11T21:28:27.082975+0000","objects_scrubbed":1,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:29:36.610810+0000","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,5],"acting":[2,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.a","version":"0'0","reported_seq":150,"reported_epoch":124,"state":"active+clean","last_fresh":"2024-03-11T21:29:29.179118+0000","last_change":"2024-03-11T21:29:21.416439+0000","last_active":"2024-03-11T21:29:29.179118+0000","last_peered":"2024-03-11T21:29:29.179118+0000","last_clean":"2024-03-11T21:29:29.179118+0000","last_became_active":"2024-03-11T21:25:38.541045+0000","last_became_peered":"2024-03-11T21:25:38.541045+0000","last_unstale":"2024-03-11T21:29:29.179118+0000","last_undegraded":"2024-03-11T21:29:29.179118+0000","last_fullsized":"2024-03-11T21:29:29.179118+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:29:21.416326+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:37.512018+0000","last_clean_scrub_stamp":"2024-03-11T21:29:21.416326+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:30:29.114951+0000","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,7],"acting":[1,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.c","version":"74'10","reported_seq":153,"reported_epoch":124,"state":"active+clean","last_fresh":"2024-03-11T21:29:29.173794+0000","last_change":"2024-03-11T21:28:36.845566+0000","last_active":"2024-03-11T21:29:29.173794+0000","last_peered":"2024-03-11T21:29:29.173794+0000","last_clean":"2024-03-11T21:29:29.173794+0000","last_became_active":"2024-03-11T21:25:40.550885+0000","last_became_peered":"2024-03-11T21:25:40.550885+0000","last_unstale":"2024-03-11T21:29:29.173794+0000","last_undegraded":"2024-03-11T21:29:29.173794+0000","last_fullsized":"2024-03-11T21:29:29.173794+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'10","last_scrub_stamp":"2024-03-11T21:28:36.845331+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:39.525554+0000","last_clean_scrub_stamp":"2024-03-11T21:28:36.845331+0000","objects_scrubbed":5,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:29:57.436270+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.e","version":"124'8891","reported_seq":8950,"reported_epoch":124,"state":"active+clean","last_fresh":"2024-03-11T21:29:36.959534+0000","last_change":"2024-03-11T21:27:36.692095+0000","last_active":"2024-03-11T21:29:36.959534+0000","last_peered":"2024-03-11T21:29:36.959534+0000","last_clean":"2024-03-11T21:29:36.959534+0000","last_became_active":"2024-03-11T21:27:36.691800+0000","last_became_peered":"2024-03-11T21:27:36.691800+0000","last_unstale":"2024-03-11T21:29:36.959534+0000","last_undegraded":"2024-03-11T21:29:36.959534+0000","last_fullsized":"2024-03-11T21:29:36.959534+0000","mapping_epoch":99,"log_start":"105'2200","ondisk_log_start":"105'2200","created":99,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:27:35.667522+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:27:35.667522+0000","last_clean_scrub_stamp":"2024-03-11T21:27:35.667522+0000","objects_scrubbed":0,"log_size":6691,"log_dups_size":0,"ondisk_log_size":6691,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":422510592,"num_objects":6447,"num_object_clones":0,"num_object_copies":19341,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6447,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8891,"num_write_kb":490816,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,0],"acting":[2,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.f","version":"0'0","reported_seq":129,"reported_epoch":124,"state":"active+clean","last_fresh":"2024-03-11T21:29:29.180998+0000","last_change":"2024-03-11T21:28:30.121487+0000","last_active":"2024-03-11T21:29:29.180998+0000","last_peered":"2024-03-11T21:29:29.180998+0000","last_clean":"2024-03-11T21:29:29.180998+0000","last_became_active":"2024-03-11T21:25:44.593414+0000","last_became_peered":"2024-03-11T21:25:44.593414+0000","last_unstale":"2024-03-11T21:29:29.180998+0000","last_undegraded":"2024-03-11T21:29:29.180998+0000","last_fullsized":"2024-03-11T21:29:29.180998+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:28:30.121365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:27:03.010969+0000","last_clean_scrub_stamp":"2024-03-11T21:28:30.121365+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:29:37.949963+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,4],"acting":[2,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":134,"reported_epoch":124,"state":"active+clean","last_fresh":"2024-03-11T21:29:29.179623+0000","last_change":"2024-03-11T21:28:30.339359+0000","last_active":"2024-03-11T21:29:29.179623+0000","last_peered":"2024-03-11T21:29:29.179623+0000","last_clean":"2024-03-11T21:29:29.179623+0000","last_became_active":"2024-03-11T21:25:42.579409+0000","last_became_peered":"2024-03-11T21:25:42.579409+0000","last_unstale":"2024-03-11T21:29:29.179623+0000","last_undegraded":"2024-03-11T21:29:29.179623+0000","last_fullsized":"2024-03-11T21:29:29.179623+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:28:30.339255+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:41.544550+0000","last_clean_scrub_stamp":"2024-03-11T21:28:30.339255+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:29:47.720331+0000","scrub_duration":22,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.b","version":"0'0","reported_seq":141,"reported_epoch":124,"state":"active+clean","last_fresh":"2024-03-11T21:29:29.175132+0000","last_change":"2024-03-11T21:28:36.500085+0000","last_active":"2024-03-11T21:29:29.175132+0000","last_peered":"2024-03-11T21:29:29.175132+0000","last_clean":"2024-03-11T21:29:29.175132+0000","last_became_active":"2024-03-11T21:25:38.543377+0000","last_became_peered":"2024-03-11T21:25:38.543377+0000","last_unstale":"2024-03-11T21:29:29.175132+0000","last_undegraded":"2024-03-11T21:29:29.175132+0000","last_fullsized":"2024-03-11T21:29:29.175132+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:28:36.499974+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:37.512018+0000","last_clean_scrub_stamp":"2024-03-11T21:28:36.499974+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:29:38.307877+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,5],"acting":[7,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.d","version":"74'17","reported_seq":166,"reported_epoch":124,"state":"active+clean","last_fresh":"2024-03-11T21:29:29.173209+0000","last_change":"2024-03-11T21:28:29.801778+0000","last_active":"2024-03-11T21:29:29.173209+0000","last_peered":"2024-03-11T21:29:29.173209+0000","last_clean":"2024-03-11T21:29:29.173209+0000","last_became_active":"2024-03-11T21:25:40.553454+0000","last_became_peered":"2024-03-11T21:25:40.553454+0000","last_unstale":"2024-03-11T21:29:29.173209+0000","last_undegraded":"2024-03-11T21:29:29.173209+0000","last_fullsized":"2024-03-11T21:29:29.173209+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'17","last_scrub_stamp":"2024-03-11T21:28:29.801670+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:39.525554+0000","last_clean_scrub_stamp":"2024-03-11T21:28:29.801670+0000","objects_scrubbed":8,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:29:34.512249+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":29,"num_read_kb":19,"num_write":18,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,1],"acting":[4,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.9","version":"124'8842","reported_seq":8902,"reported_epoch":124,"state":"active+clean","last_fresh":"2024-03-11T21:29:37.162925+0000","last_change":"2024-03-11T21:27:36.693276+0000","last_active":"2024-03-11T21:29:37.162925+0000","last_peered":"2024-03-11T21:29:37.162925+0000","last_clean":"2024-03-11T21:29:37.162925+0000","last_became_active":"2024-03-11T21:27:36.692978+0000","last_became_peered":"2024-03-11T21:27:36.692978+0000","last_unstale":"2024-03-11T21:29:37.162925+0000","last_undegraded":"2024-03-11T21:29:37.162925+0000","last_fullsized":"2024-03-11T21:29:37.162925+0000","mapping_epoch":99,"log_start":"107'3300","ondisk_log_start":"107'3300","created":99,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:27:35.667522+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:27:35.667522+0000","last_clean_scrub_stamp":"2024-03-11T21:27:35.667522+0000","objects_scrubbed":0,"log_size":5542,"log_dups_size":0,"ondisk_log_size":5542,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":414580736,"num_objects":6326,"num_object_clones":0,"num_object_copies":18978,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6326,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8842,"num_write_kb":485376,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":129,"reported_epoch":124,"state":"active+clean","last_fresh":"2024-03-11T21:29:29.174935+0000","last_change":"2024-03-11T21:28:37.561913+0000","last_active":"2024-03-11T21:29:29.174935+0000","last_peered":"2024-03-11T21:29:29.174935+0000","last_clean":"2024-03-11T21:29:29.174935+0000","last_became_active":"2024-03-11T21:25:44.589973+0000","last_became_peered":"2024-03-11T21:25:44.589973+0000","last_unstale":"2024-03-11T21:29:29.174935+0000","last_undegraded":"2024-03-11T21:29:29.174935+0000","last_fullsized":"2024-03-11T21:29:29.174935+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:28:37.561826+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:28:37.561826+0000","last_clean_scrub_stamp":"2024-03-11T21:28:37.561826+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:29:56.242385+0000","scrub_duration":25,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,3],"acting":[7,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":125,"reported_epoch":124,"state":"active+clean","last_fresh":"2024-03-11T21:29:29.181428+0000","last_change":"2024-03-11T21:27:29.075249+0000","last_active":"2024-03-11T21:29:29.181428+0000","last_peered":"2024-03-11T21:29:29.181428+0000","last_clean":"2024-03-11T21:29:29.181428+0000","last_became_active":"2024-03-11T21:25:42.574515+0000","last_became_peered":"2024-03-11T21:25:42.574515+0000","last_unstale":"2024-03-11T21:29:29.181428+0000","last_undegraded":"2024-03-11T21:29:29.181428+0000","last_fullsized":"2024-03-11T21:29:29.181428+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:27:29.075166+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:41.544550+0000","last_clean_scrub_stamp":"2024-03-11T21:27:29.075166+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,5],"acting":[2,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.c","version":"0'0","reported_seq":141,"reported_epoch":124,"state":"active+clean","last_fresh":"2024-03-11T21:29:30.279993+0000","last_change":"2024-03-11T21:29:30.280126+0000","last_active":"2024-03-11T21:29:30.279993+0000","last_peered":"2024-03-11T21:29:30.279993+0000","last_clean":"2024-03-11T21:29:30.279993+0000","last_became_active":"2024-03-11T21:25:38.533240+0000","last_became_peered":"2024-03-11T21:25:38.533240+0000","last_unstale":"2024-03-11T21:29:30.279993+0000","last_undegraded":"2024-03-11T21:29:30.279993+0000","last_fullsized":"2024-03-11T21:29:30.279993+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:29:30.279910+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:29:30.279910+0000","last_clean_scrub_stamp":"2024-03-11T21:29:30.279910+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:30:30.532672+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,0],"acting":[2,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.a","version":"109'28","reported_seq":178,"reported_epoch":124,"state":"active+clean","last_fresh":"2024-03-11T21:29:29.177690+0000","last_change":"2024-03-11T21:28:24.016603+0000","last_active":"2024-03-11T21:29:29.177690+0000","last_peered":"2024-03-11T21:29:29.177690+0000","last_clean":"2024-03-11T21:29:29.177690+0000","last_became_active":"2024-03-11T21:25:40.760698+0000","last_became_peered":"2024-03-11T21:25:40.760698+0000","last_unstale":"2024-03-11T21:29:29.177690+0000","last_undegraded":"2024-03-11T21:29:29.177690+0000","last_fullsized":"2024-03-11T21:29:29.177690+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"109'28","last_scrub_stamp":"2024-03-11T21:28:24.016389+0000","last_deep_scrub":"109'28","last_deep_scrub_stamp":"2024-03-11T21:28:24.016389+0000","last_clean_scrub_stamp":"2024-03-11T21:28:24.016389+0000","objects_scrubbed":9,"log_size":28,"log_dups_size":0,"ondisk_log_size":28,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:29:37.183547+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":32,"num_read_kb":21,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,7],"acting":[6,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.8","version":"124'8748","reported_seq":8883,"reported_epoch":124,"state":"active+clean","last_fresh":"2024-03-11T21:29:37.006461+0000","last_change":"2024-03-11T21:29:29.451971+0000","last_active":"2024-03-11T21:29:37.006461+0000","last_peered":"2024-03-11T21:29:37.006461+0000","last_clean":"2024-03-11T21:29:37.006461+0000","last_became_active":"2024-03-11T21:27:36.693060+0000","last_became_peered":"2024-03-11T21:27:36.693060+0000","last_unstale":"2024-03-11T21:29:37.006461+0000","last_undegraded":"2024-03-11T21:29:37.006461+0000","last_fullsized":"2024-03-11T21:29:37.006461+0000","mapping_epoch":99,"log_start":"105'2000","ondisk_log_start":"105'2000","created":99,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"124'8286","last_scrub_stamp":"2024-03-11T21:29:29.451663+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:27:35.667522+0000","last_clean_scrub_stamp":"2024-03-11T21:29:29.451663+0000","objects_scrubbed":6788,"log_size":6748,"log_dups_size":0,"ondisk_log_size":6748,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:30:50.146833+0000","scrub_duration":2302,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":410386432,"num_objects":6262,"num_object_clones":0,"num_object_copies":18786,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6262,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8748,"num_write_kb":480320,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,4],"acting":[2,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":130,"reported_epoch":124,"state":"active+clean","last_fresh":"2024-03-11T21:29:29.171660+0000","last_change":"2024-03-11T21:29:12.356729+0000","last_active":"2024-03-11T21:29:29.171660+0000","last_peered":"2024-03-11T21:29:29.171660+0000","last_clean":"2024-03-11T21:29:29.171660+0000","last_became_active":"2024-03-11T21:25:44.584154+0000","last_became_peered":"2024-03-11T21:25:44.584154+0000","last_unstale":"2024-03-11T21:29:29.171660+0000","last_undegraded":"2024-03-11T21:29:29.171660+0000","last_fullsized":"2024-03-11T21:29:29.171660+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:29:12.356609+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:27:11.361913+0000","last_clean_scrub_stamp":"2024-03-11T21:29:12.356609+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:30:23.076325+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":133,"reported_epoch":124,"state":"active+clean","last_fresh":"2024-03-11T21:29:29.181849+0000","last_change":"2024-03-11T21:29:14.281270+0000","last_active":"2024-03-11T21:29:29.181849+0000","last_peered":"2024-03-11T21:29:29.181849+0000","last_clean":"2024-03-11T21:29:29.181849+0000","last_became_active":"2024-03-11T21:25:42.576679+0000","last_became_peered":"2024-03-11T21:25:42.576679+0000","last_unstale":"2024-03-11T21:29:29.181849+0000","last_undegraded":"2024-03-11T21:29:29.181849+0000","last_fullsized":"2024-03-11T21:29:29.181849+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:29:14.281024+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:41.544550+0000","last_clean_scrub_stamp":"2024-03-11T21:29:14.281024+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:30:32.795772+0000","scrub_duration":22,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,3],"acting":[2,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.d","version":"0'0","reported_seq":150,"reported_epoch":124,"state":"active+clean","last_fresh":"2024-03-11T21:29:29.177589+0000","last_change":"2024-03-11T21:29:20.439881+0000","last_active":"2024-03-11T21:29:29.177589+0000","last_peered":"2024-03-11T21:29:29.177589+0000","last_clean":"2024-03-11T21:29:29.177589+0000","last_became_active":"2024-03-11T21:25:38.539681+0000","last_became_peered":"2024-03-11T21:25:38.539681+0000","last_unstale":"2024-03-11T21:29:29.177589+0000","last_undegraded":"2024-03-11T21:29:29.177589+0000","last_fullsized":"2024-03-11T21:29:29.177589+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:29:20.439742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:37.512018+0000","last_clean_scrub_stamp":"2024-03-11T21:29:20.439742+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:30:41.853759+0000","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,3],"acting":[1,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.b","version":"74'9","reported_seq":154,"reported_epoch":124,"state":"active+clean","last_fresh":"2024-03-11T21:29:31.300976+0000","last_change":"2024-03-11T21:29:31.301042+0000","last_active":"2024-03-11T21:29:31.300976+0000","last_peered":"2024-03-11T21:29:31.300976+0000","last_clean":"2024-03-11T21:29:31.300976+0000","last_became_active":"2024-03-11T21:25:40.549612+0000","last_became_peered":"2024-03-11T21:25:40.549612+0000","last_unstale":"2024-03-11T21:29:31.300976+0000","last_undegraded":"2024-03-11T21:29:31.300976+0000","last_fullsized":"2024-03-11T21:29:31.300976+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'9","last_scrub_stamp":"2024-03-11T21:29:31.300892+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:39.525554+0000","last_clean_scrub_stamp":"2024-03-11T21:29:31.300892+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:30:45.970479+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.b","version":"124'8865","reported_seq":9009,"reported_epoch":124,"state":"active+clean","last_fresh":"2024-03-11T21:29:37.012269+0000","last_change":"2024-03-11T21:29:15.125794+0000","last_active":"2024-03-11T21:29:37.012269+0000","last_peered":"2024-03-11T21:29:37.012269+0000","last_clean":"2024-03-11T21:29:37.012269+0000","last_became_active":"2024-03-11T21:27:36.691322+0000","last_became_peered":"2024-03-11T21:27:36.691322+0000","last_unstale":"2024-03-11T21:29:37.012269+0000","last_undegraded":"2024-03-11T21:29:37.012269+0000","last_fullsized":"2024-03-11T21:29:37.012269+0000","mapping_epoch":99,"log_start":"108'3700","ondisk_log_start":"108'3700","created":99,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"120'7619","last_scrub_stamp":"2024-03-11T21:29:15.125670+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:27:35.667522+0000","last_clean_scrub_stamp":"2024-03-11T21:29:15.125670+0000","objects_scrubbed":7603,"log_size":5165,"log_dups_size":0,"ondisk_log_size":5165,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:30:24.493599+0000","scrub_duration":1670,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":417660928,"num_objects":6373,"num_object_clones":0,"num_object_copies":19119,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6373,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8865,"num_write_kb":487616,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,6],"acting":[1,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":35,"reported_epoch":124,"state":"active+clean","last_fresh":"2024-03-11T21:29:29.171546+0000","last_change":"2024-03-11T21:28:45.372681+0000","last_active":"2024-03-11T21:29:29.171546+0000","last_peered":"2024-03-11T21:29:29.171546+0000","last_clean":"2024-03-11T21:29:29.171546+0000","last_became_active":"2024-03-11T21:28:45.372452+0000","last_became_peered":"2024-03-11T21:28:45.372452+0000","last_unstale":"2024-03-11T21:29:29.171546+0000","last_undegraded":"2024-03-11T21:29:29.171546+0000","last_fullsized":"2024-03-11T21:29:29.171546+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:28:19.815688+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:43.559452+0000","last_clean_scrub_stamp":"2024-03-11T21:28:19.815688+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:29:39.127540+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,3],"acting":[4,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.e","version":"0'0","reported_seq":141,"reported_epoch":124,"state":"active+clean","last_fresh":"2024-03-11T21:29:29.182137+0000","last_change":"2024-03-11T21:28:11.016096+0000","last_active":"2024-03-11T21:29:29.182137+0000","last_peered":"2024-03-11T21:29:29.182137+0000","last_clean":"2024-03-11T21:29:29.182137+0000","last_became_active":"2024-03-11T21:25:38.539954+0000","last_became_peered":"2024-03-11T21:25:38.539954+0000","last_unstale":"2024-03-11T21:29:29.182137+0000","last_undegraded":"2024-03-11T21:29:29.182137+0000","last_fullsized":"2024-03-11T21:29:29.182137+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:28:11.015887+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:37.512018+0000","last_clean_scrub_stamp":"2024-03-11T21:28:11.015887+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,7],"acting":[2,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.8","version":"109'22","reported_seq":169,"reported_epoch":124,"state":"active+clean","last_fresh":"2024-03-11T21:29:29.182330+0000","last_change":"2024-03-11T21:28:25.743214+0000","last_active":"2024-03-11T21:29:29.182330+0000","last_peered":"2024-03-11T21:29:29.182330+0000","last_clean":"2024-03-11T21:29:29.182330+0000","last_became_active":"2024-03-11T21:25:40.761938+0000","last_became_peered":"2024-03-11T21:25:40.761938+0000","last_unstale":"2024-03-11T21:29:29.182330+0000","last_undegraded":"2024-03-11T21:29:29.182330+0000","last_fullsized":"2024-03-11T21:29:29.182330+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"109'22","last_scrub_stamp":"2024-03-11T21:28:25.743058+0000","last_deep_scrub":"109'22","last_deep_scrub_stamp":"2024-03-11T21:28:25.743058+0000","last_clean_scrub_stamp":"2024-03-11T21:28:25.743058+0000","objects_scrubbed":7,"log_size":22,"log_dups_size":0,"ondisk_log_size":22,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:29:41.395729+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.9","version":"94'9","reported_seq":201,"reported_epoch":124,"state":"active+clean","last_fresh":"2024-03-11T21:29:29.175168+0000","last_change":"2024-03-11T21:29:23.565598+0000","last_active":"2024-03-11T21:29:29.175168+0000","last_peered":"2024-03-11T21:29:29.175168+0000","last_clean":"2024-03-11T21:29:29.175168+0000","last_became_active":"2024-03-11T21:25:42.584461+0000","last_became_peered":"2024-03-11T21:25:42.584461+0000","last_unstale":"2024-03-11T21:29:29.175168+0000","last_undegraded":"2024-03-11T21:29:29.175168+0000","last_fullsized":"2024-03-11T21:29:29.175168+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"94'9","last_scrub_stamp":"2024-03-11T21:29:23.565483+0000","last_deep_scrub":"94'9","last_deep_scrub_stamp":"2024-03-11T21:26:53.422552+0000","last_clean_scrub_stamp":"2024-03-11T21:29:23.565483+0000","objects_scrubbed":1,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:30:43.080330+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":7,"num_pg":16,"stat_sum":{"num_bytes":6666584088,"num_objects":101725,"num_object_clones":0,"num_object_copies":305175,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":101725,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":142093,"num_write_kb":7802113,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":20001927168,"data_stored":20001914952,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":89642,"ondisk_log_size":89642,"up":48,"acting":48,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":1531,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":24,"num_read_kb":24,"num_write":10,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":4593,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":6,"ondisk_log_size":6,"up":96,"acting":96,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":212,"num_read_kb":306,"num_write":179,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1771104,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1781760,"data_stored":1771104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":128,"ondisk_log_size":128,"up":3,"acting":3,"num_store_stats":3},{"poolid":3,"num_pg":3,"stat_sum":{"num_bytes":408,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":233,"num_read_kb":228,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1224,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":9,"acting":9,"num_store_stats":7},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":3702,"num_objects":178,"num_object_clones":0,"num_object_copies":534,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":178,"num_whiteouts":0,"num_read":698,"num_read_kb":455,"num_write":417,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":417792,"data_stored":11106,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":451,"ondisk_log_size":451,"up":96,"acting":96,"num_store_stats":8},{"poolid":5,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":68,"ondisk_log_size":68,"up":96,"acting":96,"num_store_stats":8},{"poolid":6,"num_pg":32,"stat_sum":{"num_bytes":383,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1149,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":96,"acting":96,"num_store_stats":8}],"osd_stats":[{"osd":7,"up_from":60,"seq":257698037811,"num_pgs":61,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2120112,"kb_used_data":2044520,"kb_used_omap":1,"kb_used_meta":75582,"kb_avail":91625040,"statfs":{"total":95995035648,"available":93824040960,"internally_reserved":0,"allocated":2093588480,"data_stored":2093202811,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":77396426},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,2],"upper_bound":8},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":50,"seq":214748364857,"num_pgs":48,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2125936,"kb_used_data":2050344,"kb_used_omap":1,"kb_used_meta":75582,"kb_avail":91619216,"statfs":{"total":95995035648,"available":93818077184,"internally_reserved":0,"allocated":2099552256,"data_stored":2099166904,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":77396422},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,1,2,1],"upper_bound":64},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574931,"num_pgs":59,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2540364,"kb_used_data":2450884,"kb_used_omap":1,"kb_used_meta":89470,"kb_avail":91204788,"statfs":{"total":95995035648,"available":93393702912,"internally_reserved":0,"allocated":2509705216,"data_stored":2509290756,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":91617733},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,1],"upper_bound":8},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738456,"num_pgs":55,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":3383992,"kb_used_data":3284784,"kb_used_omap":1,"kb_used_meta":99198,"kb_avail":90361160,"statfs":{"total":95995035648,"available":92529827840,"internally_reserved":0,"allocated":3363618816,"data_stored":3363225196,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":101579205},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,3,2,5,8,14,1],"upper_bound":256},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411405,"num_pgs":45,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2542048,"kb_used_data":2456728,"kb_used_omap":1,"kb_used_meta":85310,"kb_avail":91203104,"statfs":{"total":95995035648,"available":93391978496,"internally_reserved":0,"allocated":2515689472,"data_stored":2515319268,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":87357893},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,0,3],"upper_bound":16},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":3,"up_from":26,"seq":111669149769,"num_pgs":65,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":3783432,"kb_used_data":3679808,"kb_used_omap":1,"kb_used_meta":103358,"kb_avail":89961720,"statfs":{"total":95995035648,"available":92120801280,"internally_reserved":0,"allocated":3768123392,"data_stored":3767713338,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":105839047},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,1],"upper_bound":32},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":4,"up_from":32,"seq":137438953538,"num_pgs":59,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2118840,"kb_used_data":2043248,"kb_used_omap":1,"kb_used_meta":75582,"kb_avail":91626312,"statfs":{"total":95995035648,"available":93825343488,"internally_reserved":0,"allocated":2092285952,"data_stored":2091891456,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":77396421},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":41,"seq":176093659197,"num_pgs":52,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1705636,"kb_used_data":1629980,"kb_used_omap":1,"kb_used_meta":75582,"kb_avail":92039516,"statfs":{"total":95995035648,"available":94248464384,"internally_reserved":0,"allocated":1669099520,"data_stored":1668725271,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":77396421},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,1],"upper_bound":8},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1049,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":46,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":957,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":574,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":436,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1003,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":528,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":1320,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":81920,"data_stored":2118,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":1018,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":1898,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":1474,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":990,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":1034,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":1254,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":370,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":383,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":370,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":3338735616,"data_stored":3338731544,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":2490306560,"data_stored":2490302488,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":2497839104,"data_stored":2497839104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":3748593664,"data_stored":3748593664,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":2080440320,"data_stored":2080440320,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":1667039232,"data_stored":1667039232,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":2096107520,"data_stored":2096103448,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":2082865152,"data_stored":2082865152,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-11T21:29:46.152 INFO:tasks.cephadm.ceph_manager.ceph:would drop below min_pgs, proj 6, currently 16 2024-03-11T21:29:46.185 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:29:46.214 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:29:46.215 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:29:46.215 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:29:46.249 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:29:45 smithi092 ceph-mon[29572]: 7.9 deep-scrub starts 2024-03-11T21:29:46.250 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:29:45 smithi092 ceph-mon[29572]: 4.8 scrub starts 2024-03-11T21:29:46.250 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:29:45 smithi092 ceph-mon[29572]: 4.8 scrub ok 2024-03-11T21:29:46.250 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:29:45 smithi092 ceph-mon[29572]: 4.12 scrub starts 2024-03-11T21:29:46.250 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:29:45 smithi092 ceph-mon[29572]: 4.12 scrub ok 2024-03-11T21:29:46.250 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:29:45 smithi092 ceph-mon[29572]: 4.18 scrub starts 2024-03-11T21:29:46.250 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:29:45 smithi092 ceph-mon[29572]: 4.18 scrub ok 2024-03-11T21:29:46.250 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:29:45 smithi092 ceph-mon[37563]: 7.9 deep-scrub starts 2024-03-11T21:29:46.250 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:29:45 smithi092 ceph-mon[37563]: 4.8 scrub starts 2024-03-11T21:29:46.250 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:29:45 smithi092 ceph-mon[37563]: 4.8 scrub ok 2024-03-11T21:29:46.250 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:29:45 smithi092 ceph-mon[37563]: 4.12 scrub starts 2024-03-11T21:29:46.251 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:29:45 smithi092 ceph-mon[37563]: 4.12 scrub ok 2024-03-11T21:29:46.251 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:29:45 smithi092 ceph-mon[37563]: 4.18 scrub starts 2024-03-11T21:29:46.251 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:29:45 smithi092 ceph-mon[37563]: 4.18 scrub ok 2024-03-11T21:29:46.316 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:29:46.326 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:29:45 smithi097 ceph-mon[33317]: 7.9 deep-scrub starts 2024-03-11T21:29:46.326 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:29:45 smithi097 ceph-mon[33317]: 4.8 scrub starts 2024-03-11T21:29:46.326 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:29:45 smithi097 ceph-mon[33317]: 4.8 scrub ok 2024-03-11T21:29:46.326 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:29:45 smithi097 ceph-mon[33317]: 4.12 scrub starts 2024-03-11T21:29:46.326 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:29:45 smithi097 ceph-mon[33317]: 4.12 scrub ok 2024-03-11T21:29:46.327 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:29:45 smithi097 ceph-mon[33317]: 4.18 scrub starts 2024-03-11T21:29:46.327 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:29:45 smithi097 ceph-mon[33317]: 4.18 scrub ok 2024-03-11T21:29:46.358 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:29:46.359 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:29:46.359 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:29:46.460 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:29:46.494 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:29:46.495 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:29:46.495 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:29:46.596 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:29:46.624 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:29:46.625 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:29:46.625 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:29:46.726 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:29:46.752 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:29:46.753 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:29:46.754 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:29:46.854 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:29:46.918 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:29:46.918 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:29:46.918 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:29:46.985 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:29:46 smithi092 ceph-mon[29572]: pgmap v142: 148 pgs: 1 active+clean+scrubbing+deep, 147 active+clean; 6.0 GiB data, 19 GiB used, 697 GiB / 715 GiB avail; 852 B/s rd, 0 B/s wr, 709 op/s 2024-03-11T21:29:46.985 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:29:46 smithi092 ceph-mon[29572]: 7.3 scrub starts 2024-03-11T21:29:46.985 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:29:46 smithi092 ceph-mon[37563]: pgmap v142: 148 pgs: 1 active+clean+scrubbing+deep, 147 active+clean; 6.0 GiB data, 19 GiB used, 697 GiB / 715 GiB avail; 852 B/s rd, 0 B/s wr, 709 op/s 2024-03-11T21:29:46.985 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:29:46 smithi092 ceph-mon[37563]: 7.3 scrub starts 2024-03-11T21:29:46.995 INFO:teuthology.orchestra.run.smithi092.stderr:Inferring config /var/lib/ceph/00db0016-dfed-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T21:29:47.019 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:29:47.024 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:29:46 smithi097 ceph-mon[33317]: pgmap v142: 148 pgs: 1 active+clean+scrubbing+deep, 147 active+clean; 6.0 GiB data, 19 GiB used, 697 GiB / 715 GiB avail; 852 B/s rd, 0 B/s wr, 709 op/s 2024-03-11T21:29:47.024 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:29:46 smithi097 ceph-mon[33317]: 7.3 scrub starts 2024-03-11T21:29:47.046 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:29:47.047 INFO:teuthology.orchestra.run.smithi097.stdout:ERROR: (22) Invalid argument 2024-03-11T21:29:47.047 INFO:teuthology.orchestra.run.smithi097.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-11T21:29:47.047 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:29:47.048 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:29:47.148 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:29:47.174 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:29:47.175 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:29:47.176 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:29:47.276 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:29:47.303 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:29:47.304 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:29:47.304 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:29:47.405 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:29:47.432 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:29:47.433 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:29:47.433 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:29:47.534 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:29:47.569 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:29:47.569 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:29:47.570 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:29:47.671 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:29:47.698 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:29:47.699 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:29:47.699 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:29:47.800 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:29:47.827 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:29:47.827 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:29:47.828 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:29:47.929 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:29:47.984 DEBUG:teuthology.orchestra.run.smithi097:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid 00db0016-dfed-11ee-95c8-87774f69a715 -- ceph daemon osd.4 dump_blocked_ops 2024-03-11T21:29:47.986 INFO:teuthology.orchestra.run.smithi092.stderr:noscrub is set 2024-03-11T21:29:47.992 INFO:journalctl@ceph.mgr.y.smithi092.stdout:Mar 11 21:29:47 smithi092 ceph-00db0016-dfed-11ee-95c8-87774f69a715-mgr-y[29801]: ::ffff:172.21.15.97 - - [11/Mar/2024:21:29:47] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-11T21:29:47.993 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:29:47 smithi092 ceph-mon[29572]: 2.b scrub starts 2024-03-11T21:29:47.993 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:29:47 smithi092 ceph-mon[29572]: 2.b scrub ok 2024-03-11T21:29:47.993 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:29:47 smithi092 ceph-mon[29572]: 6.19 scrub starts 2024-03-11T21:29:47.993 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:29:47 smithi092 ceph-mon[29572]: 6.2 scrub starts 2024-03-11T21:29:47.993 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:29:47 smithi092 ceph-mon[29572]: 6.2 scrub ok 2024-03-11T21:29:47.993 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:29:47 smithi092 ceph-mon[29572]: 7.e scrub starts 2024-03-11T21:29:47.993 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:29:47 smithi092 ceph-mon[29572]: from='client.? v1:172.21.15.92:0/2008660207' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-11T21:29:47.993 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:29:47 smithi092 ceph-mon[29572]: from='client.34817 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-11T21:29:47.993 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:29:47 smithi092 ceph-mon[29572]: from='mgr.14607 ' entity='mgr.y' 2024-03-11T21:29:47.994 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:29:47 smithi092 ceph-mon[29572]: from='mgr.14607 ' entity='mgr.y' 2024-03-11T21:29:47.994 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:29:47 smithi092 ceph-mon[37563]: 2.b scrub starts 2024-03-11T21:29:47.994 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:29:47 smithi092 ceph-mon[37563]: 2.b scrub ok 2024-03-11T21:29:47.994 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:29:47 smithi092 ceph-mon[37563]: 6.19 scrub starts 2024-03-11T21:29:47.994 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:29:47 smithi092 ceph-mon[37563]: 6.2 scrub starts 2024-03-11T21:29:47.994 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:29:47 smithi092 ceph-mon[37563]: 6.2 scrub ok 2024-03-11T21:29:47.994 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:29:47 smithi092 ceph-mon[37563]: 7.e scrub starts 2024-03-11T21:29:47.994 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:29:47 smithi092 ceph-mon[37563]: from='client.? v1:172.21.15.92:0/2008660207' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-11T21:29:47.994 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:29:47 smithi092 ceph-mon[37563]: from='client.34817 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-11T21:29:47.994 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:29:47 smithi092 ceph-mon[37563]: from='mgr.14607 ' entity='mgr.y' 2024-03-11T21:29:47.994 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:29:47 smithi092 ceph-mon[37563]: from='mgr.14607 ' entity='mgr.y' 2024-03-11T21:29:48.014 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:29:48.015 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:29:48.015 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:29:48.116 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:29:48.160 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:29:48.160 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:29:48.161 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:29:48.262 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:29:48.282 INFO:teuthology.orchestra.run.smithi097.stderr:Inferring config /var/lib/ceph/00db0016-dfed-11ee-95c8-87774f69a715/mon.b/config 2024-03-11T21:29:48.312 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:29:48.313 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:29:48.313 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:29:48.326 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:29:47 smithi097 ceph-mon[33317]: 2.b scrub starts 2024-03-11T21:29:48.441 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:29:47 smithi097 ceph-mon[33317]: 2.b scrub ok 2024-03-11T21:29:48.441 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:29:47 smithi097 ceph-mon[33317]: 6.19 scrub starts 2024-03-11T21:29:48.441 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:29:47 smithi097 ceph-mon[33317]: 6.2 scrub starts 2024-03-11T21:29:48.441 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:29:47 smithi097 ceph-mon[33317]: 6.2 scrub ok 2024-03-11T21:29:48.441 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:29:47 smithi097 ceph-mon[33317]: 7.e scrub starts 2024-03-11T21:29:48.441 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:29:47 smithi097 ceph-mon[33317]: from='client.? v1:172.21.15.92:0/2008660207' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-11T21:29:48.441 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:29:47 smithi097 ceph-mon[33317]: from='client.34817 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-11T21:29:48.441 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:29:47 smithi097 ceph-mon[33317]: from='mgr.14607 ' entity='mgr.y' 2024-03-11T21:29:48.441 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:29:47 smithi097 ceph-mon[33317]: from='mgr.14607 ' entity='mgr.y' 2024-03-11T21:29:48.442 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:29:48.484 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:29:48.485 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:29:48.486 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:29:48.586 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:29:48.632 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:29:48.632 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:29:48.633 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:29:48.733 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:29:48.773 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:29:48.774 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:29:48.774 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:29:48.875 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:29:48.905 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:29:48.906 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:29:48.906 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:29:49.007 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:29:49.046 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:29:49.047 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:29:49.047 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:29:49.051 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:29:48 smithi092 ceph-mon[29572]: 6.19 scrub ok 2024-03-11T21:29:49.051 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:29:48 smithi092 ceph-mon[29572]: pgmap v143: 148 pgs: 1 active+clean+scrubbing, 146 active+clean, 1 active+clean+scrubbing+deep; 5.9 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 852 B/s rd, 0 B/s wr, 835 op/s 2024-03-11T21:29:49.051 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:29:48 smithi092 ceph-mon[29572]: 7.3 scrub ok 2024-03-11T21:29:49.051 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:29:48 smithi092 ceph-mon[29572]: 5.10 scrub starts 2024-03-11T21:29:49.051 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:29:48 smithi092 ceph-mon[29572]: 5.10 scrub ok 2024-03-11T21:29:49.051 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:29:48 smithi092 ceph-mon[29572]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-03-11T21:29:49.051 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:29:48 smithi092 ceph-mon[29572]: from='client.34817 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-03-11T21:29:49.052 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:29:48 smithi092 ceph-mon[29572]: osdmap e125: 8 total, 8 up, 8 in 2024-03-11T21:29:49.052 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:29:48 smithi092 ceph-mon[29572]: 4.7 scrub starts 2024-03-11T21:29:49.052 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:29:48 smithi092 ceph-mon[37563]: 6.19 scrub ok 2024-03-11T21:29:49.052 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:29:48 smithi092 ceph-mon[37563]: pgmap v143: 148 pgs: 1 active+clean+scrubbing, 146 active+clean, 1 active+clean+scrubbing+deep; 5.9 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 852 B/s rd, 0 B/s wr, 835 op/s 2024-03-11T21:29:49.052 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:29:48 smithi092 ceph-mon[37563]: 7.3 scrub ok 2024-03-11T21:29:49.052 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:29:48 smithi092 ceph-mon[37563]: 5.10 scrub starts 2024-03-11T21:29:49.052 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:29:48 smithi092 ceph-mon[37563]: 5.10 scrub ok 2024-03-11T21:29:49.052 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:29:48 smithi092 ceph-mon[37563]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-03-11T21:29:49.052 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:29:48 smithi092 ceph-mon[37563]: from='client.34817 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-03-11T21:29:49.053 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:29:48 smithi092 ceph-mon[37563]: osdmap e125: 8 total, 8 up, 8 in 2024-03-11T21:29:49.053 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:29:48 smithi092 ceph-mon[37563]: 4.7 scrub starts 2024-03-11T21:29:49.076 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:29:48 smithi097 ceph-mon[33317]: 6.19 scrub ok 2024-03-11T21:29:49.076 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:29:48 smithi097 ceph-mon[33317]: pgmap v143: 148 pgs: 1 active+clean+scrubbing, 146 active+clean, 1 active+clean+scrubbing+deep; 5.9 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 852 B/s rd, 0 B/s wr, 835 op/s 2024-03-11T21:29:49.077 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:29:48 smithi097 ceph-mon[33317]: 7.3 scrub ok 2024-03-11T21:29:49.077 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:29:48 smithi097 ceph-mon[33317]: 5.10 scrub starts 2024-03-11T21:29:49.077 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:29:48 smithi097 ceph-mon[33317]: 5.10 scrub ok 2024-03-11T21:29:49.077 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:29:48 smithi097 ceph-mon[33317]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-03-11T21:29:49.077 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:29:48 smithi097 ceph-mon[33317]: from='client.34817 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-03-11T21:29:49.077 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:29:48 smithi097 ceph-mon[33317]: osdmap e125: 8 total, 8 up, 8 in 2024-03-11T21:29:49.077 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:29:48 smithi097 ceph-mon[33317]: 4.7 scrub starts 2024-03-11T21:29:49.149 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:29:49.193 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:29:49.194 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:29:49.194 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:29:49.295 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:29:49.334 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:29:49.334 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:29:49.335 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:29:49.435 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:29:49.468 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:29:49.468 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:29:49.468 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:29:49.569 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:29:49.608 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:29:49.609 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:29:49.609 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:29:49.710 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:29:49.740 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:29:49.741 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:29:49.741 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:29:49.842 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:29:49.866 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:29:49.867 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:29:49.868 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:29:49.968 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:29:50.009 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:29:50.010 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:29:50.011 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:29:50.076 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:29:49 smithi097 ceph-mon[33317]: from='mgr.14607 v1:172.21.15.92:0/1702717597' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T21:29:50.077 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:29:49 smithi097 ceph-mon[33317]: from='mgr.14607 v1:172.21.15.92:0/1702717597' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T21:29:50.077 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:29:49 smithi097 ceph-mon[33317]: from='mgr.14607 ' entity='mgr.y' 2024-03-11T21:29:50.111 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:29:50.142 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:29:50.142 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:29:50.143 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:29:50.244 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:29:50.260 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:29:49 smithi092 ceph-mon[29572]: from='mgr.14607 v1:172.21.15.92:0/1702717597' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T21:29:50.260 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:29:49 smithi092 ceph-mon[29572]: from='mgr.14607 v1:172.21.15.92:0/1702717597' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T21:29:50.260 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:29:49 smithi092 ceph-mon[29572]: from='mgr.14607 ' entity='mgr.y' 2024-03-11T21:29:50.260 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:29:49 smithi092 ceph-mon[37563]: from='mgr.14607 v1:172.21.15.92:0/1702717597' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T21:29:50.261 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:29:49 smithi092 ceph-mon[37563]: from='mgr.14607 v1:172.21.15.92:0/1702717597' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T21:29:50.261 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:29:49 smithi092 ceph-mon[37563]: from='mgr.14607 ' entity='mgr.y' 2024-03-11T21:29:50.277 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:29:50.278 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:29:50.278 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:29:50.379 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:29:50.418 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:29:50.419 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:29:50.419 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:29:50.520 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:29:50.555 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:29:50.556 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:29:50.556 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:29:50.658 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:29:50.686 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:29:50.686 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:29:50.687 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:29:50.788 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:29:50.813 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:29:50.813 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:29:50.813 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:29:50.914 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:29:50.940 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:29:50.941 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:29:50.941 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:29:51.042 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:29:51.072 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:29:51.073 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:29:51.073 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:29:51.099 DEBUG:teuthology.orchestra.run.smithi092:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid 00db0016-dfed-11ee-95c8-87774f69a715 -- ceph osd set nodeep-scrub 2024-03-11T21:29:51.153 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5, 6, 7] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7] 2024-03-11T21:29:51.154 DEBUG:teuthology.orchestra.run.smithi092:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid 00db0016-dfed-11ee-95c8-87774f69a715 -- ceph osd reweight 0 1 2024-03-11T21:29:51.174 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:29:51.234 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:29:51.235 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:29:51.235 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:29:51.324 INFO:teuthology.orchestra.run.smithi097.stdout:ERROR: (22) Invalid argument 2024-03-11T21:29:51.324 INFO:teuthology.orchestra.run.smithi097.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-11T21:29:51.324 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:29:50 smithi097 ceph-mon[33317]: pgmap v145: 148 pgs: 2 active+clean+scrubbing, 144 active+clean, 2 active+clean+scrubbing+deep; 5.7 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 791 op/s 2024-03-11T21:29:51.324 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:29:50 smithi097 ceph-mon[33317]: purged_snaps scrub starts 2024-03-11T21:29:51.324 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:29:50 smithi097 ceph-mon[33317]: purged_snaps scrub ok 2024-03-11T21:29:51.336 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:29:51.360 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:29:50 smithi092 ceph-mon[29572]: pgmap v145: 148 pgs: 2 active+clean+scrubbing, 144 active+clean, 2 active+clean+scrubbing+deep; 5.7 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 791 op/s 2024-03-11T21:29:51.360 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:29:51 smithi092 ceph-mon[29572]: purged_snaps scrub starts 2024-03-11T21:29:51.361 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:29:51 smithi092 ceph-mon[29572]: purged_snaps scrub ok 2024-03-11T21:29:51.361 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:29:50 smithi092 ceph-mon[37563]: pgmap v145: 148 pgs: 2 active+clean+scrubbing, 144 active+clean, 2 active+clean+scrubbing+deep; 5.7 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 791 op/s 2024-03-11T21:29:51.361 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:29:51 smithi092 ceph-mon[37563]: purged_snaps scrub starts 2024-03-11T21:29:51.361 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:29:51 smithi092 ceph-mon[37563]: purged_snaps scrub ok 2024-03-11T21:29:51.391 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:29:51.391 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:29:51.392 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:29:51.493 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:29:51.525 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:29:51.526 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:29:51.526 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:29:51.627 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:29:51.656 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:29:51.657 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:29:51.657 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:29:51.758 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:29:51.783 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:29:51.784 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:29:51.784 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:29:51.885 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:29:51.927 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:29:51.927 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:29:51.927 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:29:52.028 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:29:52.056 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:29:52.056 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:29:52.057 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:29:52.158 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:29:52.167 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:29:52 smithi092 ceph-mon[29572]: osdmap e126: 8 total, 8 up, 8 in 2024-03-11T21:29:52.167 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:29:52 smithi092 ceph-mon[29572]: pgmap v147: 148 pgs: 2 active+clean+scrubbing, 144 active+clean, 2 active+clean+scrubbing+deep; 5.7 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 639 B/s rd, 0 B/s wr, 488 op/s 2024-03-11T21:29:52.167 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:29:52 smithi092 ceph-mon[29572]: from='mgr.14607 ' entity='mgr.y' 2024-03-11T21:29:52.167 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:29:52 smithi092 ceph-mon[29572]: from='mgr.14607 v1:172.21.15.92:0/1702717597' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T21:29:52.168 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:29:52 smithi092 ceph-mon[29572]: from='client.? v1:172.21.15.92:0/348786822' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-11T21:29:52.169 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:29:52 smithi092 ceph-mon[37563]: osdmap e126: 8 total, 8 up, 8 in 2024-03-11T21:29:52.170 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:29:52 smithi092 ceph-mon[37563]: pgmap v147: 148 pgs: 2 active+clean+scrubbing, 144 active+clean, 2 active+clean+scrubbing+deep; 5.7 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 639 B/s rd, 0 B/s wr, 488 op/s 2024-03-11T21:29:52.170 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:29:52 smithi092 ceph-mon[37563]: from='mgr.14607 ' entity='mgr.y' 2024-03-11T21:29:52.170 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:29:52 smithi092 ceph-mon[37563]: from='mgr.14607 v1:172.21.15.92:0/1702717597' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T21:29:52.170 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:29:52 smithi092 ceph-mon[37563]: from='client.? v1:172.21.15.92:0/348786822' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-11T21:29:52.194 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:29:52.194 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:29:52.194 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:29:52.295 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:29:52.326 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:29:52 smithi097 ceph-mon[33317]: osdmap e126: 8 total, 8 up, 8 in 2024-03-11T21:29:52.327 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:29:52 smithi097 ceph-mon[33317]: pgmap v147: 148 pgs: 2 active+clean+scrubbing, 144 active+clean, 2 active+clean+scrubbing+deep; 5.7 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 639 B/s rd, 0 B/s wr, 488 op/s 2024-03-11T21:29:52.327 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:29:52 smithi097 ceph-mon[33317]: from='mgr.14607 ' entity='mgr.y' 2024-03-11T21:29:52.327 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:29:52 smithi097 ceph-mon[33317]: from='mgr.14607 v1:172.21.15.92:0/1702717597' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T21:29:52.327 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:29:52 smithi097 ceph-mon[33317]: from='client.? v1:172.21.15.92:0/348786822' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-11T21:29:52.327 INFO:journalctl@ceph.iscsi.iscsi.a.smithi097.stdout:Mar 11 21:29:52 smithi097 ceph-00db0016-dfed-11ee-95c8-87774f69a715-iscsi-iscsi-a[60943]: debug there is no tcmu-runner data available 2024-03-11T21:29:52.331 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:29:52.331 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:29:52.332 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:29:52.362 INFO:teuthology.orchestra.run.smithi092.stdout:osd.0: {} 2024-03-11T21:29:52.362 INFO:teuthology.orchestra.run.smithi092.stdout:osd.1: {} 2024-03-11T21:29:52.362 INFO:teuthology.orchestra.run.smithi092.stdout:osd.2: {} 2024-03-11T21:29:52.363 INFO:teuthology.orchestra.run.smithi092.stdout:osd.3: {} 2024-03-11T21:29:52.363 INFO:teuthology.orchestra.run.smithi092.stdout:osd.4: {} 2024-03-11T21:29:52.363 INFO:teuthology.orchestra.run.smithi092.stdout:osd.5: {} 2024-03-11T21:29:52.363 INFO:teuthology.orchestra.run.smithi092.stdout:osd.6: {} 2024-03-11T21:29:52.363 INFO:teuthology.orchestra.run.smithi092.stdout:osd.7: {} 2024-03-11T21:29:52.364 INFO:teuthology.orchestra.run.smithi092.stderr:osd.0: osd_enable_op_tracker = '' 2024-03-11T21:29:52.364 INFO:teuthology.orchestra.run.smithi092.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-11T21:29:52.364 INFO:teuthology.orchestra.run.smithi092.stderr:osd.2: osd_enable_op_tracker = '' 2024-03-11T21:29:52.364 INFO:teuthology.orchestra.run.smithi092.stderr:osd.3: osd_enable_op_tracker = '' 2024-03-11T21:29:52.364 INFO:teuthology.orchestra.run.smithi092.stderr:osd.4: osd_enable_op_tracker = '' 2024-03-11T21:29:52.364 INFO:teuthology.orchestra.run.smithi092.stderr:osd.5: osd_enable_op_tracker = '' 2024-03-11T21:29:52.364 INFO:teuthology.orchestra.run.smithi092.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-11T21:29:52.365 INFO:teuthology.orchestra.run.smithi092.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-11T21:29:52.432 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:29:52.467 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:29:52.468 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:29:52.468 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:29:52.569 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:29:52.593 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:29:52.594 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:29:52.594 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:29:52.695 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:29:52.726 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:29:52.726 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:29:52.727 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:29:52.828 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:29:52.841 INFO:teuthology.orchestra.run.smithi092.stderr:Inferring config /var/lib/ceph/00db0016-dfed-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T21:29:52.882 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:29:52.883 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:29:52.883 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:29:52.970 DEBUG:teuthology.orchestra.run.smithi097:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid 00db0016-dfed-11ee-95c8-87774f69a715 -- ceph daemon osd.4 dump_historic_ops 2024-03-11T21:29:52.983 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:29:53.045 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:29:53.046 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:29:53.046 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:29:53.147 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:29:53.183 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:29:53.184 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:29:53.184 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:29:53.284 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:29:53 smithi092 ceph-mon[29572]: from='client.24469 v1:172.21.15.97:0/404208310' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:29:53.284 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:29:53 smithi092 ceph-mon[37563]: from='client.24469 v1:172.21.15.97:0/404208310' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:29:53.285 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:29:53.287 INFO:teuthology.orchestra.run.smithi097.stderr:Inferring config /var/lib/ceph/00db0016-dfed-11ee-95c8-87774f69a715/mon.b/config 2024-03-11T21:29:53.326 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:29:53 smithi097 ceph-mon[33317]: from='client.24469 v1:172.21.15.97:0/404208310' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:29:53.350 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:29:53.350 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:29:53.351 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:29:53.405 INFO:teuthology.orchestra.run.smithi092.stderr:Inferring config /var/lib/ceph/00db0016-dfed-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T21:29:53.452 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:29:53.483 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:29:53.483 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:29:53.484 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:29:53.584 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:29:53.614 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:29:53.614 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:29:53.615 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:29:53.715 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:29:53.754 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:29:53.755 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:29:53.755 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:29:53.856 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:29:53.884 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:29:53.885 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:29:53.885 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:29:53.986 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:29:54.033 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:29:54.034 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:29:54.034 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:29:54.135 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:29:54.162 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:29:54.162 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:29:54.163 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:29:54.263 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:29:54.304 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:29:54.305 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:29:54.305 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:29:54.326 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:29:54 smithi097 ceph-mon[33317]: pgmap v148: 148 pgs: 146 active+clean, 2 active+clean+scrubbing+deep; 5.5 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 876 op/s 2024-03-11T21:29:54.360 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:29:54 smithi092 ceph-mon[29572]: pgmap v148: 148 pgs: 146 active+clean, 2 active+clean+scrubbing+deep; 5.5 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 876 op/s 2024-03-11T21:29:54.361 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:29:54 smithi092 ceph-mon[37563]: pgmap v148: 148 pgs: 146 active+clean, 2 active+clean+scrubbing+deep; 5.5 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 876 op/s 2024-03-11T21:29:54.406 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:29:54.442 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:29:54.443 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:29:54.443 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:29:54.546 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:29:54.574 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:29:54.574 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:29:54.575 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:29:54.676 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:29:54.718 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:29:54.718 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:29:54.719 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:29:54.820 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:29:54.847 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:29:54.848 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:29:54.848 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:29:54.949 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:29:54.989 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:29:54.990 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:29:54.990 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:29:55.091 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:29:55.131 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:29:55.131 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:29:55.132 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:29:55.191 INFO:teuthology.orchestra.run.smithi097.stdout:{ 2024-03-11T21:29:55.191 INFO:teuthology.orchestra.run.smithi097.stdout: "size": 20, 2024-03-11T21:29:55.191 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 600, 2024-03-11T21:29:55.191 INFO:teuthology.orchestra.run.smithi097.stdout: "ops": [ 2024-03-11T21:29:55.192 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:29:55.192 INFO:teuthology.orchestra.run.smithi097.stdout: "description": "osd_op(client.34715.0:16783 7.1 7:8a03f820:::benchmark_data_smithi092_76040_object16782:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e105)", 2024-03-11T21:29:55.192 INFO:teuthology.orchestra.run.smithi097.stdout: "initiated_at": "2024-03-11T21:27:50.452605+0000", 2024-03-11T21:29:55.192 INFO:teuthology.orchestra.run.smithi097.stdout: "age": 124.75142874399999, 2024-03-11T21:29:55.192 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.067317150000000006, 2024-03-11T21:29:55.192 INFO:teuthology.orchestra.run.smithi097.stdout: "type_data": { 2024-03-11T21:29:55.192 INFO:teuthology.orchestra.run.smithi097.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-11T21:29:55.192 INFO:teuthology.orchestra.run.smithi097.stdout: "client_info": { 2024-03-11T21:29:55.192 INFO:teuthology.orchestra.run.smithi097.stdout: "client": "client.34715", 2024-03-11T21:29:55.192 INFO:teuthology.orchestra.run.smithi097.stdout: "client_addr": "v1:172.21.15.92:0/2403132573", 2024-03-11T21:29:55.192 INFO:teuthology.orchestra.run.smithi097.stdout: "tid": 16783 2024-03-11T21:29:55.192 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:29:55.192 INFO:teuthology.orchestra.run.smithi097.stdout: "events": [ 2024-03-11T21:29:55.192 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:29:55.193 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "initiated", 2024-03-11T21:29:55.193 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:27:50.452605+0000", 2024-03-11T21:29:55.193 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-03-11T21:29:55.193 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:29:55.193 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:29:55.193 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "header_read", 2024-03-11T21:29:55.193 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:27:50.452605+0000", 2024-03-11T21:29:55.193 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-03-11T21:29:55.193 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:29:55.193 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:29:55.193 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "throttled", 2024-03-11T21:29:55.193 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:27:50.452607+0000", 2024-03-11T21:29:55.193 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.6109999999999999e-06 2024-03-11T21:29:55.193 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:29:55.194 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:29:55.194 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "all_read", 2024-03-11T21:29:55.194 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:27:50.452667+0000", 2024-03-11T21:29:55.194 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 6.0016999999999998e-05 2024-03-11T21:29:55.194 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:29:55.194 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:29:55.194 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "dispatched", 2024-03-11T21:29:55.194 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:27:50.452676+0000", 2024-03-11T21:29:55.194 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 9.5599999999999999e-06 2024-03-11T21:29:55.194 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:29:55.194 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:29:55.194 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "queued_for_pg", 2024-03-11T21:29:55.194 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:27:50.452684+0000", 2024-03-11T21:29:55.195 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 7.8569999999999995e-06 2024-03-11T21:29:55.195 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:29:55.195 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:29:55.195 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "reached_pg", 2024-03-11T21:29:55.195 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:27:50.452754+0000", 2024-03-11T21:29:55.195 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 6.9499000000000007e-05 2024-03-11T21:29:55.195 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:29:55.195 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:29:55.195 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "started", 2024-03-11T21:29:55.195 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:27:50.452866+0000", 2024-03-11T21:29:55.195 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.000112749 2024-03-11T21:29:55.195 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:29:55.195 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:29:55.196 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "waiting for subops from 3,6", 2024-03-11T21:29:55.196 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:27:50.452997+0000", 2024-03-11T21:29:55.196 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.00013031800000000001 2024-03-11T21:29:55.196 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:29:55.196 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:29:55.196 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "op_commit", 2024-03-11T21:29:55.196 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:27:50.453760+0000", 2024-03-11T21:29:55.196 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.00076315599999999997 2024-03-11T21:29:55.196 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:29:55.196 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:29:55.196 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "sub_op_commit_rec", 2024-03-11T21:29:55.196 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:27:50.453961+0000", 2024-03-11T21:29:55.196 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.00020143 2024-03-11T21:29:55.196 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:29:55.197 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:29:55.197 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "sub_op_commit_rec", 2024-03-11T21:29:55.197 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:27:50.519846+0000", 2024-03-11T21:29:55.197 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.065884531999999996 2024-03-11T21:29:55.197 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:29:55.197 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:29:55.197 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "commit_sent", 2024-03-11T21:29:55.197 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:27:50.519883+0000", 2024-03-11T21:29:55.197 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 3.7533000000000001e-05 2024-03-11T21:29:55.197 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:29:55.197 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:29:55.197 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "done", 2024-03-11T21:29:55.197 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:27:50.519922+0000", 2024-03-11T21:29:55.197 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 3.8887999999999998e-05 2024-03-11T21:29:55.197 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-03-11T21:29:55.198 INFO:teuthology.orchestra.run.smithi097.stdout: ] 2024-03-11T21:29:55.198 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-03-11T21:29:55.198 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:29:55.198 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:29:55.198 INFO:teuthology.orchestra.run.smithi097.stdout: "description": "osd_op(client.34715.0:16787 7.1 7:8daed005:::benchmark_data_smithi092_76040_object16786:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e105)", 2024-03-11T21:29:55.198 INFO:teuthology.orchestra.run.smithi097.stdout: "initiated_at": "2024-03-11T21:27:50.455699+0000", 2024-03-11T21:29:55.198 INFO:teuthology.orchestra.run.smithi097.stdout: "age": 124.748334536, 2024-03-11T21:29:55.198 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.067379470999999996, 2024-03-11T21:29:55.198 INFO:teuthology.orchestra.run.smithi097.stdout: "type_data": { 2024-03-11T21:29:55.198 INFO:teuthology.orchestra.run.smithi097.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-11T21:29:55.198 INFO:teuthology.orchestra.run.smithi097.stdout: "client_info": { 2024-03-11T21:29:55.198 INFO:teuthology.orchestra.run.smithi097.stdout: "client": "client.34715", 2024-03-11T21:29:55.198 INFO:teuthology.orchestra.run.smithi097.stdout: "client_addr": "v1:172.21.15.92:0/2403132573", 2024-03-11T21:29:55.198 INFO:teuthology.orchestra.run.smithi097.stdout: "tid": 16787 2024-03-11T21:29:55.199 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:29:55.199 INFO:teuthology.orchestra.run.smithi097.stdout: "events": [ 2024-03-11T21:29:55.199 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:29:55.199 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "initiated", 2024-03-11T21:29:55.199 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:27:50.455699+0000", 2024-03-11T21:29:55.199 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-03-11T21:29:55.199 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:29:55.199 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:29:55.199 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "header_read", 2024-03-11T21:29:55.199 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:27:50.455699+0000", 2024-03-11T21:29:55.199 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-03-11T21:29:55.199 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:29:55.200 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:29:55.200 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "throttled", 2024-03-11T21:29:55.200 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:27:50.455701+0000", 2024-03-11T21:29:55.200 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.42e-06 2024-03-11T21:29:55.200 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:29:55.200 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:29:55.200 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "all_read", 2024-03-11T21:29:55.200 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:27:50.455758+0000", 2024-03-11T21:29:55.200 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 5.7346999999999998e-05 2024-03-11T21:29:55.200 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:29:55.200 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:29:55.200 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "dispatched", 2024-03-11T21:29:55.200 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:27:50.455769+0000", 2024-03-11T21:29:55.200 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.1122e-05 2024-03-11T21:29:55.200 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:29:55.201 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:29:55.201 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "queued_for_pg", 2024-03-11T21:29:55.201 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:27:50.455777+0000", 2024-03-11T21:29:55.201 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 7.6279999999999996e-06 2024-03-11T21:29:55.201 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:29:55.201 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:29:55.201 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "reached_pg", 2024-03-11T21:29:55.201 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:27:50.455823+0000", 2024-03-11T21:29:55.201 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 4.6659000000000002e-05 2024-03-11T21:29:55.201 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:29:55.201 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:29:55.201 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "started", 2024-03-11T21:29:55.201 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:27:50.455906+0000", 2024-03-11T21:29:55.201 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 8.2381999999999998e-05 2024-03-11T21:29:55.202 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:29:55.202 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:29:55.202 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "waiting for subops from 3,6", 2024-03-11T21:29:55.202 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:27:50.456020+0000", 2024-03-11T21:29:55.202 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.000114207 2024-03-11T21:29:55.202 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:29:55.202 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:29:55.202 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "op_commit", 2024-03-11T21:29:55.202 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:27:50.456839+0000", 2024-03-11T21:29:55.202 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.00081941399999999997 2024-03-11T21:29:55.202 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:29:55.202 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:29:55.202 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "sub_op_commit_rec", 2024-03-11T21:29:55.202 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:27:50.457192+0000", 2024-03-11T21:29:55.203 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.00035303400000000001 2024-03-11T21:29:55.203 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:29:55.203 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:29:55.203 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "sub_op_commit_rec", 2024-03-11T21:29:55.203 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:27:50.522980+0000", 2024-03-11T21:29:55.203 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.065787727000000004 2024-03-11T21:29:55.203 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:29:55.203 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:29:55.203 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "commit_sent", 2024-03-11T21:29:55.203 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:27:50.523035+0000", 2024-03-11T21:29:55.203 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 5.5322999999999999e-05 2024-03-11T21:29:55.203 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:29:55.203 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:29:55.203 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "done", 2024-03-11T21:29:55.203 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:27:50.523079+0000", 2024-03-11T21:29:55.204 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 4.3207999999999997e-05 2024-03-11T21:29:55.204 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-03-11T21:29:55.204 INFO:teuthology.orchestra.run.smithi097.stdout: ] 2024-03-11T21:29:55.204 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-03-11T21:29:55.204 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:29:55.204 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:29:55.204 INFO:teuthology.orchestra.run.smithi097.stdout: "description": "osd_op(client.34715.0:21627 7.1 7:89792512:::benchmark_data_smithi092_76040_object21626:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e105)", 2024-03-11T21:29:55.204 INFO:teuthology.orchestra.run.smithi097.stdout: "initiated_at": "2024-03-11T21:27:52.544188+0000", 2024-03-11T21:29:55.204 INFO:teuthology.orchestra.run.smithi097.stdout: "age": 122.659845303, 2024-03-11T21:29:55.204 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.060220893999999997, 2024-03-11T21:29:55.204 INFO:teuthology.orchestra.run.smithi097.stdout: "type_data": { 2024-03-11T21:29:55.204 INFO:teuthology.orchestra.run.smithi097.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-11T21:29:55.204 INFO:teuthology.orchestra.run.smithi097.stdout: "client_info": { 2024-03-11T21:29:55.205 INFO:teuthology.orchestra.run.smithi097.stdout: "client": "client.34715", 2024-03-11T21:29:55.205 INFO:teuthology.orchestra.run.smithi097.stdout: "client_addr": "v1:172.21.15.92:0/2403132573", 2024-03-11T21:29:55.205 INFO:teuthology.orchestra.run.smithi097.stdout: "tid": 21627 2024-03-11T21:29:55.205 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:29:55.205 INFO:teuthology.orchestra.run.smithi097.stdout: "events": [ 2024-03-11T21:29:55.205 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:29:55.205 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "initiated", 2024-03-11T21:29:55.205 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:27:52.544188+0000", 2024-03-11T21:29:55.205 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-03-11T21:29:55.205 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:29:55.205 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:29:55.205 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "header_read", 2024-03-11T21:29:55.205 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:27:52.544188+0000", 2024-03-11T21:29:55.206 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-03-11T21:29:55.206 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:29:55.206 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:29:55.206 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "throttled", 2024-03-11T21:29:55.206 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:27:52.544201+0000", 2024-03-11T21:29:55.206 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.2262999999999999e-05 2024-03-11T21:29:55.206 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:29:55.206 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:29:55.206 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "all_read", 2024-03-11T21:29:55.206 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:27:52.544248+0000", 2024-03-11T21:29:55.206 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 4.7790000000000002e-05 2024-03-11T21:29:55.206 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:29:55.206 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:29:55.206 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "dispatched", 2024-03-11T21:29:55.207 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:27:52.544259+0000", 2024-03-11T21:29:55.207 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.0710000000000001e-05 2024-03-11T21:29:55.207 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:29:55.207 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:29:55.207 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "queued_for_pg", 2024-03-11T21:29:55.207 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:27:52.544266+0000", 2024-03-11T21:29:55.207 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 6.409e-06 2024-03-11T21:29:55.207 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:29:55.207 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:29:55.207 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "reached_pg", 2024-03-11T21:29:55.207 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:27:52.544316+0000", 2024-03-11T21:29:55.207 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 5.0104e-05 2024-03-11T21:29:55.207 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:29:55.207 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:29:55.207 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "started", 2024-03-11T21:29:55.208 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:27:52.544436+0000", 2024-03-11T21:29:55.208 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.00012040399999999999 2024-03-11T21:29:55.208 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:29:55.208 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:29:55.208 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "waiting for subops from 3,6", 2024-03-11T21:29:55.208 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:27:52.544572+0000", 2024-03-11T21:29:55.208 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.000136026 2024-03-11T21:29:55.208 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:29:55.208 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:29:55.208 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "op_commit", 2024-03-11T21:29:55.208 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:27:52.545495+0000", 2024-03-11T21:29:55.208 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.00092301799999999995 2024-03-11T21:29:55.208 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:29:55.208 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:29:55.208 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "sub_op_commit_rec", 2024-03-11T21:29:55.209 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:27:52.546689+0000", 2024-03-11T21:29:55.209 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.0011939489999999999 2024-03-11T21:29:55.209 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:29:55.209 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:29:55.209 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "sub_op_commit_rec", 2024-03-11T21:29:55.209 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:27:52.604322+0000", 2024-03-11T21:29:55.209 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.057633223999999997 2024-03-11T21:29:55.209 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:29:55.209 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:29:55.209 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "commit_sent", 2024-03-11T21:29:55.209 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:27:52.604358+0000", 2024-03-11T21:29:55.209 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 3.5351000000000001e-05 2024-03-11T21:29:55.209 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:29:55.209 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:29:55.210 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "done", 2024-03-11T21:29:55.210 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:27:52.604409+0000", 2024-03-11T21:29:55.210 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 5.1646e-05 2024-03-11T21:29:55.210 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-03-11T21:29:55.210 INFO:teuthology.orchestra.run.smithi097.stdout: ] 2024-03-11T21:29:55.210 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-03-11T21:29:55.210 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:29:55.210 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:29:55.210 INFO:teuthology.orchestra.run.smithi097.stdout: "description": "osd_op(client.34715.0:26786 7.1 7:84ab4d09:::benchmark_data_smithi092_76040_object26785:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e105)", 2024-03-11T21:29:55.210 INFO:teuthology.orchestra.run.smithi097.stdout: "initiated_at": "2024-03-11T21:27:54.914937+0000", 2024-03-11T21:29:55.210 INFO:teuthology.orchestra.run.smithi097.stdout: "age": 120.289096456, 2024-03-11T21:29:55.210 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.061022461, 2024-03-11T21:29:55.210 INFO:teuthology.orchestra.run.smithi097.stdout: "type_data": { 2024-03-11T21:29:55.211 INFO:teuthology.orchestra.run.smithi097.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-11T21:29:55.211 INFO:teuthology.orchestra.run.smithi097.stdout: "client_info": { 2024-03-11T21:29:55.211 INFO:teuthology.orchestra.run.smithi097.stdout: "client": "client.34715", 2024-03-11T21:29:55.211 INFO:teuthology.orchestra.run.smithi097.stdout: "client_addr": "v1:172.21.15.92:0/2403132573", 2024-03-11T21:29:55.211 INFO:teuthology.orchestra.run.smithi097.stdout: "tid": 26786 2024-03-11T21:29:55.211 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:29:55.211 INFO:teuthology.orchestra.run.smithi097.stdout: "events": [ 2024-03-11T21:29:55.211 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:29:55.211 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "initiated", 2024-03-11T21:29:55.211 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:27:54.914937+0000", 2024-03-11T21:29:55.211 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-03-11T21:29:55.211 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:29:55.211 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:29:55.211 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "header_read", 2024-03-11T21:29:55.211 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:27:54.914937+0000", 2024-03-11T21:29:55.212 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-03-11T21:29:55.212 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:29:55.212 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:29:55.212 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "throttled", 2024-03-11T21:29:55.212 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:27:54.914940+0000", 2024-03-11T21:29:55.212 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 2.2419999999999999e-06 2024-03-11T21:29:55.212 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:29:55.212 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:29:55.212 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "all_read", 2024-03-11T21:29:55.212 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:27:54.914991+0000", 2024-03-11T21:29:55.212 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 5.1306e-05 2024-03-11T21:29:55.212 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:29:55.212 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:29:55.212 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "dispatched", 2024-03-11T21:29:55.213 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:27:54.915005+0000", 2024-03-11T21:29:55.213 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.4664999999999999e-05 2024-03-11T21:29:55.213 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:29:55.213 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:29:55.213 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "queued_for_pg", 2024-03-11T21:29:55.213 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:27:54.915013+0000", 2024-03-11T21:29:55.213 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 7.204e-06 2024-03-11T21:29:55.213 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:29:55.213 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:29:55.213 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "reached_pg", 2024-03-11T21:29:55.213 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:27:54.915067+0000", 2024-03-11T21:29:55.213 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 5.4762000000000002e-05 2024-03-11T21:29:55.213 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:29:55.213 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:29:55.213 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "started", 2024-03-11T21:29:55.214 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:27:54.915221+0000", 2024-03-11T21:29:55.214 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.00015334999999999999 2024-03-11T21:29:55.214 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:29:55.214 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:29:55.214 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "waiting for subops from 3,6", 2024-03-11T21:29:55.214 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:27:54.915360+0000", 2024-03-11T21:29:55.214 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.00013939700000000001 2024-03-11T21:29:55.214 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:29:55.214 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:29:55.214 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "sub_op_commit_rec", 2024-03-11T21:29:55.214 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:27:54.917783+0000", 2024-03-11T21:29:55.214 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.0024230530000000001 2024-03-11T21:29:55.214 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:29:55.214 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:29:55.215 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "op_commit", 2024-03-11T21:29:55.215 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:27:54.917796+0000", 2024-03-11T21:29:55.215 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.3106e-05 2024-03-11T21:29:55.215 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:29:55.215 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:29:55.215 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "sub_op_commit_rec", 2024-03-11T21:29:55.215 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:27:54.975871+0000", 2024-03-11T21:29:55.215 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.058074842000000002 2024-03-11T21:29:55.215 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:29:55.215 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:29:55.215 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "commit_sent", 2024-03-11T21:29:55.215 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:27:54.975924+0000", 2024-03-11T21:29:55.215 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 5.2438999999999997e-05 2024-03-11T21:29:55.215 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:29:55.215 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:29:55.216 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "done", 2024-03-11T21:29:55.216 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:27:54.975960+0000", 2024-03-11T21:29:55.216 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 3.6094999999999999e-05 2024-03-11T21:29:55.216 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-03-11T21:29:55.216 INFO:teuthology.orchestra.run.smithi097.stdout: ] 2024-03-11T21:29:55.216 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-03-11T21:29:55.216 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:29:55.216 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:29:55.216 INFO:teuthology.orchestra.run.smithi097.stdout: "description": "osd_op(client.34715.0:26787 7.1 7:8a56bf81:::benchmark_data_smithi092_76040_object26786:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e105)", 2024-03-11T21:29:55.216 INFO:teuthology.orchestra.run.smithi097.stdout: "initiated_at": "2024-03-11T21:27:54.915033+0000", 2024-03-11T21:29:55.216 INFO:teuthology.orchestra.run.smithi097.stdout: "age": 120.289000477, 2024-03-11T21:29:55.216 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.062267007999999999, 2024-03-11T21:29:55.216 INFO:teuthology.orchestra.run.smithi097.stdout: "type_data": { 2024-03-11T21:29:55.216 INFO:teuthology.orchestra.run.smithi097.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-11T21:29:55.216 INFO:teuthology.orchestra.run.smithi097.stdout: "client_info": { 2024-03-11T21:29:55.217 INFO:teuthology.orchestra.run.smithi097.stdout: "client": "client.34715", 2024-03-11T21:29:55.217 INFO:teuthology.orchestra.run.smithi097.stdout: "client_addr": "v1:172.21.15.92:0/2403132573", 2024-03-11T21:29:55.217 INFO:teuthology.orchestra.run.smithi097.stdout: "tid": 26787 2024-03-11T21:29:55.217 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:29:55.217 INFO:teuthology.orchestra.run.smithi097.stdout: "events": [ 2024-03-11T21:29:55.217 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:29:55.217 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "initiated", 2024-03-11T21:29:55.217 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:27:54.915033+0000", 2024-03-11T21:29:55.217 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-03-11T21:29:55.217 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:29:55.217 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:29:55.217 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "header_read", 2024-03-11T21:29:55.218 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:27:54.915033+0000", 2024-03-11T21:29:55.218 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-03-11T21:29:55.218 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:29:55.218 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:29:55.218 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "throttled", 2024-03-11T21:29:55.218 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:27:54.915034+0000", 2024-03-11T21:29:55.218 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 8.78e-07 2024-03-11T21:29:55.218 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:29:55.218 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:29:55.218 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "all_read", 2024-03-11T21:29:55.218 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:27:54.915073+0000", 2024-03-11T21:29:55.218 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 3.8433000000000003e-05 2024-03-11T21:29:55.218 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:29:55.218 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:29:55.219 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "dispatched", 2024-03-11T21:29:55.219 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:27:54.915080+0000", 2024-03-11T21:29:55.219 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 7.2470000000000002e-06 2024-03-11T21:29:55.219 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:29:55.219 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:29:55.219 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "queued_for_pg", 2024-03-11T21:29:55.219 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:27:54.915088+0000", 2024-03-11T21:29:55.219 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 8.0390000000000005e-06 2024-03-11T21:29:55.219 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:29:55.219 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:29:55.219 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "reached_pg", 2024-03-11T21:29:55.219 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:27:54.916571+0000", 2024-03-11T21:29:55.219 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.001483269 2024-03-11T21:29:55.219 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:29:55.220 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:29:55.220 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "started", 2024-03-11T21:29:55.220 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:27:54.916695+0000", 2024-03-11T21:29:55.220 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.00012341300000000001 2024-03-11T21:29:55.220 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:29:55.220 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:29:55.220 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "waiting for subops from 3,6", 2024-03-11T21:29:55.220 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:27:54.916843+0000", 2024-03-11T21:29:55.220 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.000148708 2024-03-11T21:29:55.220 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:29:55.220 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:29:55.220 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "op_commit", 2024-03-11T21:29:55.220 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:27:54.917907+0000", 2024-03-11T21:29:55.220 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.0010639110000000001 2024-03-11T21:29:55.220 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:29:55.221 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:29:55.221 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "sub_op_commit_rec", 2024-03-11T21:29:55.221 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:27:54.918072+0000", 2024-03-11T21:29:55.221 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.000165191 2024-03-11T21:29:55.221 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:29:55.221 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:29:55.221 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "sub_op_commit_rec", 2024-03-11T21:29:55.221 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:27:54.977198+0000", 2024-03-11T21:29:55.221 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.05912568 2024-03-11T21:29:55.221 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:29:55.221 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:29:55.221 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "commit_sent", 2024-03-11T21:29:55.221 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:27:54.977251+0000", 2024-03-11T21:29:55.221 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 5.3406999999999999e-05 2024-03-11T21:29:55.222 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:29:55.222 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:29:55.222 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "done", 2024-03-11T21:29:55.222 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:27:54.977300+0000", 2024-03-11T21:29:55.222 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 4.8832000000000003e-05 2024-03-11T21:29:55.222 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-03-11T21:29:55.222 INFO:teuthology.orchestra.run.smithi097.stdout: ] 2024-03-11T21:29:55.222 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-03-11T21:29:55.222 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:29:55.222 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:29:55.222 INFO:teuthology.orchestra.run.smithi097.stdout: "description": "osd_op(client.34715.0:27067 7.1 7:8785ae58:::benchmark_data_smithi092_76040_object27066:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e105)", 2024-03-11T21:29:55.222 INFO:teuthology.orchestra.run.smithi097.stdout: "initiated_at": "2024-03-11T21:27:55.096430+0000", 2024-03-11T21:29:55.222 INFO:teuthology.orchestra.run.smithi097.stdout: "age": 120.107603927, 2024-03-11T21:29:55.223 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.062796144999999998, 2024-03-11T21:29:55.223 INFO:teuthology.orchestra.run.smithi097.stdout: "type_data": { 2024-03-11T21:29:55.223 INFO:teuthology.orchestra.run.smithi097.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-11T21:29:55.223 INFO:teuthology.orchestra.run.smithi097.stdout: "client_info": { 2024-03-11T21:29:55.223 INFO:teuthology.orchestra.run.smithi097.stdout: "client": "client.34715", 2024-03-11T21:29:55.223 INFO:teuthology.orchestra.run.smithi097.stdout: "client_addr": "v1:172.21.15.92:0/2403132573", 2024-03-11T21:29:55.223 INFO:teuthology.orchestra.run.smithi097.stdout: "tid": 27067 2024-03-11T21:29:55.223 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:29:55.223 INFO:teuthology.orchestra.run.smithi097.stdout: "events": [ 2024-03-11T21:29:55.223 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:29:55.223 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "initiated", 2024-03-11T21:29:55.223 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:27:55.096430+0000", 2024-03-11T21:29:55.223 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-03-11T21:29:55.224 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:29:55.224 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:29:55.224 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "header_read", 2024-03-11T21:29:55.224 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:27:55.096430+0000", 2024-03-11T21:29:55.224 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-03-11T21:29:55.224 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:29:55.224 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:29:55.224 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "throttled", 2024-03-11T21:29:55.224 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:27:55.096431+0000", 2024-03-11T21:29:55.224 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.1799999999999999e-06 2024-03-11T21:29:55.224 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:29:55.224 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:29:55.224 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "all_read", 2024-03-11T21:29:55.224 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:27:55.096493+0000", 2024-03-11T21:29:55.225 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 6.1626999999999998e-05 2024-03-11T21:29:55.225 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:29:55.225 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:29:55.225 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "dispatched", 2024-03-11T21:29:55.225 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:27:55.096503+0000", 2024-03-11T21:29:55.225 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.0062e-05 2024-03-11T21:29:55.225 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:29:55.225 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:29:55.225 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "queued_for_pg", 2024-03-11T21:29:55.225 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:27:55.096520+0000", 2024-03-11T21:29:55.225 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.7408e-05 2024-03-11T21:29:55.225 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:29:55.225 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:29:55.225 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "reached_pg", 2024-03-11T21:29:55.225 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:27:55.097610+0000", 2024-03-11T21:29:55.226 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.001089509 2024-03-11T21:29:55.226 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:29:55.226 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:29:55.226 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "started", 2024-03-11T21:29:55.226 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:27:55.097725+0000", 2024-03-11T21:29:55.226 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.000115828 2024-03-11T21:29:55.226 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:29:55.226 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:29:55.226 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "waiting for subops from 3,6", 2024-03-11T21:29:55.226 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:27:55.097876+0000", 2024-03-11T21:29:55.226 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.00015042899999999999 2024-03-11T21:29:55.226 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:29:55.226 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:29:55.226 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "op_commit", 2024-03-11T21:29:55.227 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:27:55.099820+0000", 2024-03-11T21:29:55.227 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.0019438649999999999 2024-03-11T21:29:55.227 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:29:55.227 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:29:55.227 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "sub_op_commit_rec", 2024-03-11T21:29:55.227 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:27:55.101271+0000", 2024-03-11T21:29:55.227 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.0014509379999999999 2024-03-11T21:29:55.227 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:29:55.227 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:29:55.227 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "sub_op_commit_rec", 2024-03-11T21:29:55.227 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:27:55.159075+0000", 2024-03-11T21:29:55.227 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.057804045999999998 2024-03-11T21:29:55.227 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:29:55.227 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:29:55.227 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "commit_sent", 2024-03-11T21:29:55.228 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:27:55.159169+0000", 2024-03-11T21:29:55.228 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 9.4046e-05 2024-03-11T21:29:55.228 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:29:55.228 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:29:55.228 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "done", 2024-03-11T21:29:55.228 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:27:55.159226+0000", 2024-03-11T21:29:55.228 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 5.7207000000000003e-05 2024-03-11T21:29:55.228 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-03-11T21:29:55.228 INFO:teuthology.orchestra.run.smithi097.stdout: ] 2024-03-11T21:29:55.228 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-03-11T21:29:55.228 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:29:55.228 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:29:55.228 INFO:teuthology.orchestra.run.smithi097.stdout: "description": "osd_op(client.34715.0:27068 7.1 7:849d8468:::benchmark_data_smithi092_76040_object27067:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e105)", 2024-03-11T21:29:55.228 INFO:teuthology.orchestra.run.smithi097.stdout: "initiated_at": "2024-03-11T21:27:55.096529+0000", 2024-03-11T21:29:55.229 INFO:teuthology.orchestra.run.smithi097.stdout: "age": 120.107504932, 2024-03-11T21:29:55.229 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.064478016999999999, 2024-03-11T21:29:55.229 INFO:teuthology.orchestra.run.smithi097.stdout: "type_data": { 2024-03-11T21:29:55.229 INFO:teuthology.orchestra.run.smithi097.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-11T21:29:55.229 INFO:teuthology.orchestra.run.smithi097.stdout: "client_info": { 2024-03-11T21:29:55.229 INFO:teuthology.orchestra.run.smithi097.stdout: "client": "client.34715", 2024-03-11T21:29:55.229 INFO:teuthology.orchestra.run.smithi097.stdout: "client_addr": "v1:172.21.15.92:0/2403132573", 2024-03-11T21:29:55.229 INFO:teuthology.orchestra.run.smithi097.stdout: "tid": 27068 2024-03-11T21:29:55.229 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:29:55.229 INFO:teuthology.orchestra.run.smithi097.stdout: "events": [ 2024-03-11T21:29:55.229 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:29:55.229 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "initiated", 2024-03-11T21:29:55.229 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:27:55.096529+0000", 2024-03-11T21:29:55.229 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-03-11T21:29:55.229 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:29:55.230 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:29:55.230 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "header_read", 2024-03-11T21:29:55.230 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:27:55.096529+0000", 2024-03-11T21:29:55.230 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-03-11T21:29:55.230 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:29:55.230 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:29:55.230 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "throttled", 2024-03-11T21:29:55.230 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:27:55.096530+0000", 2024-03-11T21:29:55.230 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 8.5199999999999995e-07 2024-03-11T21:29:55.230 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:29:55.231 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:29:55.231 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "all_read", 2024-03-11T21:29:55.231 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:27:55.096554+0000", 2024-03-11T21:29:55.231 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 2.4488999999999999e-05 2024-03-11T21:29:55.231 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:29:55.231 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:29:55.231 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "dispatched", 2024-03-11T21:29:55.231 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:27:55.096559+0000", 2024-03-11T21:29:55.231 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 4.8420000000000004e-06 2024-03-11T21:29:55.231 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:29:55.231 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:29:55.231 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "queued_for_pg", 2024-03-11T21:29:55.231 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:27:55.096563+0000", 2024-03-11T21:29:55.231 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 3.551e-06 2024-03-11T21:29:55.232 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:29:55.232 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:29:55.233 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "reached_pg", 2024-03-11T21:29:55.233 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:27:55.098713+0000", 2024-03-11T21:29:55.233 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.0021501139999999998 2024-03-11T21:29:55.233 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:29:55.233 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:29:55.233 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "started", 2024-03-11T21:29:55.233 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:27:55.098831+0000", 2024-03-11T21:29:55.233 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.00011882799999999999 2024-03-11T21:29:55.233 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:29:55.233 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:29:55.233 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "waiting for subops from 3,6", 2024-03-11T21:29:55.233 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:27:55.099003+0000", 2024-03-11T21:29:55.234 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.00017182699999999999 2024-03-11T21:29:55.234 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:29:55.234 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:29:55.234 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "op_commit", 2024-03-11T21:29:55.234 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:27:55.099927+0000", 2024-03-11T21:29:55.234 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.00092343999999999998 2024-03-11T21:29:55.234 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:29:55.234 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:29:55.234 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "sub_op_commit_rec", 2024-03-11T21:29:55.234 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:27:55.101477+0000", 2024-03-11T21:29:55.234 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.0015505060000000001 2024-03-11T21:29:55.234 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:29:55.234 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:29:55.234 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "sub_op_commit_rec", 2024-03-11T21:29:55.235 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:27:55.160896+0000", 2024-03-11T21:29:55.235 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.059419242999999997 2024-03-11T21:29:55.235 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:29:55.235 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:29:55.235 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "commit_sent", 2024-03-11T21:29:55.235 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:27:55.160965+0000", 2024-03-11T21:29:55.235 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 6.8412999999999997e-05 2024-03-11T21:29:55.235 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:29:55.235 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:29:55.235 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "done", 2024-03-11T21:29:55.235 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:27:55.161007+0000", 2024-03-11T21:29:55.235 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 4.1912000000000001e-05 2024-03-11T21:29:55.235 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-03-11T21:29:55.235 INFO:teuthology.orchestra.run.smithi097.stdout: ] 2024-03-11T21:29:55.236 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-03-11T21:29:55.236 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:29:55.236 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:29:55.236 INFO:teuthology.orchestra.run.smithi097.stdout: "description": "osd_op(client.34715.0:28489 7.1 7:81d41a6b:::benchmark_data_smithi092_76040_object28488:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e105)", 2024-03-11T21:29:55.236 INFO:teuthology.orchestra.run.smithi097.stdout: "initiated_at": "2024-03-11T21:27:55.854472+0000", 2024-03-11T21:29:55.236 INFO:teuthology.orchestra.run.smithi097.stdout: "age": 119.349561872, 2024-03-11T21:29:55.236 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.060864736000000003, 2024-03-11T21:29:55.236 INFO:teuthology.orchestra.run.smithi097.stdout: "type_data": { 2024-03-11T21:29:55.236 INFO:teuthology.orchestra.run.smithi097.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-11T21:29:55.236 INFO:teuthology.orchestra.run.smithi097.stdout: "client_info": { 2024-03-11T21:29:55.236 INFO:teuthology.orchestra.run.smithi097.stdout: "client": "client.34715", 2024-03-11T21:29:55.236 INFO:teuthology.orchestra.run.smithi097.stdout: "client_addr": "v1:172.21.15.92:0/2403132573", 2024-03-11T21:29:55.236 INFO:teuthology.orchestra.run.smithi097.stdout: "tid": 28489 2024-03-11T21:29:55.236 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:29:55.237 INFO:teuthology.orchestra.run.smithi097.stdout: "events": [ 2024-03-11T21:29:55.237 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:29:55.237 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "initiated", 2024-03-11T21:29:55.237 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:27:55.854472+0000", 2024-03-11T21:29:55.237 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-03-11T21:29:55.237 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:29:55.237 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:29:55.237 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "header_read", 2024-03-11T21:29:55.237 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:27:55.854472+0000", 2024-03-11T21:29:55.237 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-03-11T21:29:55.237 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:29:55.237 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:29:55.237 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "throttled", 2024-03-11T21:29:55.237 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:27:55.854473+0000", 2024-03-11T21:29:55.238 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.367e-06 2024-03-11T21:29:55.238 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:29:55.238 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:29:55.238 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "all_read", 2024-03-11T21:29:55.238 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:27:55.854544+0000", 2024-03-11T21:29:55.238 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 7.1103e-05 2024-03-11T21:29:55.238 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:29:55.238 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:29:55.238 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "dispatched", 2024-03-11T21:29:55.238 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:27:55.854554+0000", 2024-03-11T21:29:55.238 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 9.2580000000000001e-06 2024-03-11T21:29:55.238 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:29:55.238 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:29:55.239 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "queued_for_pg", 2024-03-11T21:29:55.239 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:27:55.854559+0000", 2024-03-11T21:29:55.239 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 5.5899999999999998e-06 2024-03-11T21:29:55.239 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:29:55.239 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:29:55.239 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "reached_pg", 2024-03-11T21:29:55.239 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:27:55.854604+0000", 2024-03-11T21:29:55.239 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 4.4990999999999997e-05 2024-03-11T21:29:55.239 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:29:55.239 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:29:55.239 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "started", 2024-03-11T21:29:55.239 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:27:55.854691+0000", 2024-03-11T21:29:55.239 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 8.6839e-05 2024-03-11T21:29:55.239 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:29:55.239 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:29:55.240 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "waiting for subops from 3,6", 2024-03-11T21:29:55.240 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:27:55.854787+0000", 2024-03-11T21:29:55.240 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 9.6063000000000005e-05 2024-03-11T21:29:55.240 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:29:55.240 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:29:55.240 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "op_commit", 2024-03-11T21:29:55.240 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:27:55.855641+0000", 2024-03-11T21:29:55.240 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.00085429799999999997 2024-03-11T21:29:55.240 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:29:55.240 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:29:55.240 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "sub_op_commit_rec", 2024-03-11T21:29:55.240 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:27:55.855795+0000", 2024-03-11T21:29:55.240 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.00015403899999999999 2024-03-11T21:29:55.240 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:29:55.240 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:29:55.241 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "sub_op_commit_rec", 2024-03-11T21:29:55.241 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:27:55.915207+0000", 2024-03-11T21:29:55.241 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.05941167 2024-03-11T21:29:55.241 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:29:55.241 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:29:55.241 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "commit_sent", 2024-03-11T21:29:55.241 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:27:55.915275+0000", 2024-03-11T21:29:55.241 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 6.7923999999999995e-05 2024-03-11T21:29:55.241 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:29:55.241 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:29:55.241 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "done", 2024-03-11T21:29:55.241 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:27:55.915337+0000", 2024-03-11T21:29:55.241 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 6.1593999999999993e-05 2024-03-11T21:29:55.241 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-03-11T21:29:55.242 INFO:teuthology.orchestra.run.smithi097.stdout: ] 2024-03-11T21:29:55.242 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-03-11T21:29:55.242 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:29:55.242 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:29:55.242 INFO:teuthology.orchestra.run.smithi097.stdout: "description": "osd_repop(client.34715.0:32056 7.9 e105/99 7:94ee601a:::benchmark_data_smithi092_76040_object32055:head v 105'2003, mlcod=105'2001)", 2024-03-11T21:29:55.242 INFO:teuthology.orchestra.run.smithi097.stdout: "initiated_at": "2024-03-11T21:27:57.787048+0000", 2024-03-11T21:29:55.242 INFO:teuthology.orchestra.run.smithi097.stdout: "age": 117.416985595, 2024-03-11T21:29:55.242 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.075118210000000005, 2024-03-11T21:29:55.242 INFO:teuthology.orchestra.run.smithi097.stdout: "type_data": { 2024-03-11T21:29:55.242 INFO:teuthology.orchestra.run.smithi097.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-11T21:29:55.242 INFO:teuthology.orchestra.run.smithi097.stdout: "events": [ 2024-03-11T21:29:55.242 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:29:55.242 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "initiated", 2024-03-11T21:29:55.242 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:27:57.787048+0000", 2024-03-11T21:29:55.243 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-03-11T21:29:55.243 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:29:55.243 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:29:55.243 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "header_read", 2024-03-11T21:29:55.243 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:27:57.787048+0000", 2024-03-11T21:29:55.243 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-03-11T21:29:55.243 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:29:55.243 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:29:55.243 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "throttled", 2024-03-11T21:29:55.243 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:27:57.787049+0000", 2024-03-11T21:29:55.243 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.336e-06 2024-03-11T21:29:55.243 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:29:55.243 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:29:55.243 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "all_read", 2024-03-11T21:29:55.244 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:27:57.787109+0000", 2024-03-11T21:29:55.244 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 5.9802000000000001e-05 2024-03-11T21:29:55.244 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:29:55.244 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:29:55.244 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "dispatched", 2024-03-11T21:29:55.244 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:27:57.787119+0000", 2024-03-11T21:29:55.244 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.0086e-05 2024-03-11T21:29:55.244 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:29:55.244 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:29:55.244 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "queued_for_pg", 2024-03-11T21:29:55.244 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:27:57.787127+0000", 2024-03-11T21:29:55.244 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 7.1879999999999996e-06 2024-03-11T21:29:55.244 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:29:55.244 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:29:55.244 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "reached_pg", 2024-03-11T21:29:55.245 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:27:57.787181+0000", 2024-03-11T21:29:55.245 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 5.4469999999999999e-05 2024-03-11T21:29:55.245 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:29:55.245 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:29:55.245 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "started", 2024-03-11T21:29:55.245 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:27:57.787200+0000", 2024-03-11T21:29:55.245 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.9247000000000001e-05 2024-03-11T21:29:55.245 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:29:55.245 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:29:55.245 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "commit_sent", 2024-03-11T21:29:55.245 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:27:57.788472+0000", 2024-03-11T21:29:55.245 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.0012718390000000001 2024-03-11T21:29:55.245 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:29:55.245 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:29:55.245 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "done", 2024-03-11T21:29:55.246 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:27:57.862166+0000", 2024-03-11T21:29:55.246 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.073694241999999993 2024-03-11T21:29:55.246 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-03-11T21:29:55.246 INFO:teuthology.orchestra.run.smithi097.stdout: ] 2024-03-11T21:29:55.246 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-03-11T21:29:55.246 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:29:55.246 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:29:55.246 INFO:teuthology.orchestra.run.smithi097.stdout: "description": "osd_op(client.34715.0:32061 7.1 7:89d95748:::benchmark_data_smithi092_76040_object32060:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e105)", 2024-03-11T21:29:55.246 INFO:teuthology.orchestra.run.smithi097.stdout: "initiated_at": "2024-03-11T21:27:57.788025+0000", 2024-03-11T21:29:55.246 INFO:teuthology.orchestra.run.smithi097.stdout: "age": 117.416008715, 2024-03-11T21:29:55.246 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.077315224000000002, 2024-03-11T21:29:55.246 INFO:teuthology.orchestra.run.smithi097.stdout: "type_data": { 2024-03-11T21:29:55.246 INFO:teuthology.orchestra.run.smithi097.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-11T21:29:55.247 INFO:teuthology.orchestra.run.smithi097.stdout: "client_info": { 2024-03-11T21:29:55.247 INFO:teuthology.orchestra.run.smithi097.stdout: "client": "client.34715", 2024-03-11T21:29:55.247 INFO:teuthology.orchestra.run.smithi097.stdout: "client_addr": "v1:172.21.15.92:0/2403132573", 2024-03-11T21:29:55.247 INFO:teuthology.orchestra.run.smithi097.stdout: "tid": 32061 2024-03-11T21:29:55.247 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:29:55.247 INFO:teuthology.orchestra.run.smithi097.stdout: "events": [ 2024-03-11T21:29:55.247 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:29:55.247 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "initiated", 2024-03-11T21:29:55.247 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:27:57.788025+0000", 2024-03-11T21:29:55.247 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-03-11T21:29:55.247 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:29:55.247 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:29:55.247 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "header_read", 2024-03-11T21:29:55.247 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:27:57.788025+0000", 2024-03-11T21:29:55.247 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-03-11T21:29:55.248 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:29:55.248 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:29:55.248 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "throttled", 2024-03-11T21:29:55.248 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:27:57.788026+0000", 2024-03-11T21:29:55.248 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.336e-06 2024-03-11T21:29:55.248 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:29:55.248 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:29:55.248 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "all_read", 2024-03-11T21:29:55.248 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:27:57.788090+0000", 2024-03-11T21:29:55.248 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 6.4092999999999998e-05 2024-03-11T21:29:55.248 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:29:55.248 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:29:55.248 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "dispatched", 2024-03-11T21:29:55.248 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:27:57.788102+0000", 2024-03-11T21:29:55.249 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.1698999999999999e-05 2024-03-11T21:29:55.249 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:29:55.249 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:29:55.249 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "queued_for_pg", 2024-03-11T21:29:55.249 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:27:57.788119+0000", 2024-03-11T21:29:55.249 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.6439e-05 2024-03-11T21:29:55.249 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:29:55.249 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:29:55.249 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "reached_pg", 2024-03-11T21:29:55.249 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:27:57.788161+0000", 2024-03-11T21:29:55.249 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 4.2035e-05 2024-03-11T21:29:55.249 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:29:55.249 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:29:55.249 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "started", 2024-03-11T21:29:55.249 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:27:57.788259+0000", 2024-03-11T21:29:55.250 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 9.8090000000000004e-05 2024-03-11T21:29:55.250 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:29:55.250 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:29:55.250 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "waiting for subops from 3,6", 2024-03-11T21:29:55.250 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:27:57.788380+0000", 2024-03-11T21:29:55.250 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.000121737 2024-03-11T21:29:55.250 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:29:55.250 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:29:55.250 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "sub_op_commit_rec", 2024-03-11T21:29:55.250 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:27:57.863304+0000", 2024-03-11T21:29:55.250 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.074923654000000006 2024-03-11T21:29:55.250 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:29:55.250 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:29:55.250 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "sub_op_commit_rec", 2024-03-11T21:29:55.251 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:27:57.863419+0000", 2024-03-11T21:29:55.251 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.000114957 2024-03-11T21:29:55.251 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:29:55.251 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:29:55.251 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "op_commit", 2024-03-11T21:29:55.251 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:27:57.865245+0000", 2024-03-11T21:29:55.251 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.001826102 2024-03-11T21:29:55.251 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:29:55.251 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:29:55.251 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "commit_sent", 2024-03-11T21:29:55.251 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:27:57.865297+0000", 2024-03-11T21:29:55.251 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 5.1529000000000001e-05 2024-03-11T21:29:55.251 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:29:55.251 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:29:55.251 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "done", 2024-03-11T21:29:55.252 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:27:57.865340+0000", 2024-03-11T21:29:55.252 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 4.3553000000000001e-05 2024-03-11T21:29:55.252 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-03-11T21:29:55.252 INFO:teuthology.orchestra.run.smithi097.stdout: ] 2024-03-11T21:29:55.252 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-03-11T21:29:55.252 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:29:55.252 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:29:55.252 INFO:teuthology.orchestra.run.smithi097.stdout: "description": "osd_op(client.34715.0:32063 7.1 7:8c2d6d23:::benchmark_data_smithi092_76040_object32062:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e105)", 2024-03-11T21:29:55.252 INFO:teuthology.orchestra.run.smithi097.stdout: "initiated_at": "2024-03-11T21:27:57.788779+0000", 2024-03-11T21:29:55.252 INFO:teuthology.orchestra.run.smithi097.stdout: "age": 117.41525434, 2024-03-11T21:29:55.252 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.079669661000000003, 2024-03-11T21:29:55.252 INFO:teuthology.orchestra.run.smithi097.stdout: "type_data": { 2024-03-11T21:29:55.252 INFO:teuthology.orchestra.run.smithi097.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-11T21:29:55.252 INFO:teuthology.orchestra.run.smithi097.stdout: "client_info": { 2024-03-11T21:29:55.253 INFO:teuthology.orchestra.run.smithi097.stdout: "client": "client.34715", 2024-03-11T21:29:55.253 INFO:teuthology.orchestra.run.smithi097.stdout: "client_addr": "v1:172.21.15.92:0/2403132573", 2024-03-11T21:29:55.253 INFO:teuthology.orchestra.run.smithi097.stdout: "tid": 32063 2024-03-11T21:29:55.253 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:29:55.253 INFO:teuthology.orchestra.run.smithi097.stdout: "events": [ 2024-03-11T21:29:55.253 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:29:55.253 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "initiated", 2024-03-11T21:29:55.253 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:27:57.788779+0000", 2024-03-11T21:29:55.253 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-03-11T21:29:55.253 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:29:55.253 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:29:55.253 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "header_read", 2024-03-11T21:29:55.253 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:27:57.788779+0000", 2024-03-11T21:29:55.253 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-03-11T21:29:55.253 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:29:55.254 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:29:55.254 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "throttled", 2024-03-11T21:29:55.254 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:27:57.788780+0000", 2024-03-11T21:29:55.254 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.0270000000000001e-06 2024-03-11T21:29:55.254 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:29:55.254 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:29:55.254 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "all_read", 2024-03-11T21:29:55.254 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:27:57.788872+0000", 2024-03-11T21:29:55.254 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 9.1278000000000003e-05 2024-03-11T21:29:55.254 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:29:55.254 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:29:55.254 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "dispatched", 2024-03-11T21:29:55.254 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:27:57.863322+0000", 2024-03-11T21:29:55.255 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.074450159000000002 2024-03-11T21:29:55.255 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:29:55.255 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:29:55.255 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "queued_for_pg", 2024-03-11T21:29:55.255 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:27:57.863331+0000", 2024-03-11T21:29:55.255 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 9.5389999999999996e-06 2024-03-11T21:29:55.255 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:29:55.255 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:29:55.255 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "reached_pg", 2024-03-11T21:29:55.255 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:27:57.863454+0000", 2024-03-11T21:29:55.255 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.00012228299999999999 2024-03-11T21:29:55.255 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:29:55.255 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:29:55.255 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "started", 2024-03-11T21:29:55.255 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:27:57.863601+0000", 2024-03-11T21:29:55.256 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.000147186 2024-03-11T21:29:55.256 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:29:55.256 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:29:55.256 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "waiting for subops from 3,6", 2024-03-11T21:29:55.256 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:27:57.863761+0000", 2024-03-11T21:29:55.256 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.00015977999999999999 2024-03-11T21:29:55.256 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:29:55.256 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:29:55.256 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "sub_op_commit_rec", 2024-03-11T21:29:55.256 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:27:57.866971+0000", 2024-03-11T21:29:55.256 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.0032101230000000001 2024-03-11T21:29:55.256 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:29:55.256 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:29:55.256 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "op_commit", 2024-03-11T21:29:55.256 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:27:57.866993+0000", 2024-03-11T21:29:55.257 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 2.1897000000000001e-05 2024-03-11T21:29:55.257 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:29:55.257 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:29:55.257 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "sub_op_commit_rec", 2024-03-11T21:29:55.257 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:27:57.868344+0000", 2024-03-11T21:29:55.257 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.001351798 2024-03-11T21:29:55.257 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:29:55.257 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:29:55.257 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "commit_sent", 2024-03-11T21:29:55.257 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:27:57.868402+0000", 2024-03-11T21:29:55.257 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 5.7862000000000003e-05 2024-03-11T21:29:55.257 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:29:55.257 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:29:55.257 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "done", 2024-03-11T21:29:55.257 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:27:57.868449+0000", 2024-03-11T21:29:55.258 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 4.6728999999999999e-05 2024-03-11T21:29:55.258 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-03-11T21:29:55.258 INFO:teuthology.orchestra.run.smithi097.stdout: ] 2024-03-11T21:29:55.258 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-03-11T21:29:55.258 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:29:55.258 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:29:55.258 INFO:teuthology.orchestra.run.smithi097.stdout: "description": "osd_repop_reply(client.34715.0:32061 7.1 e105/99 ondisk, result = 0)", 2024-03-11T21:29:55.258 INFO:teuthology.orchestra.run.smithi097.stdout: "initiated_at": "2024-03-11T21:27:57.789455+0000", 2024-03-11T21:29:55.258 INFO:teuthology.orchestra.run.smithi097.stdout: "age": 117.41457920000001, 2024-03-11T21:29:55.258 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.073857473000000007, 2024-03-11T21:29:55.258 INFO:teuthology.orchestra.run.smithi097.stdout: "type_data": { 2024-03-11T21:29:55.258 INFO:teuthology.orchestra.run.smithi097.stdout: "flag_point": "started", 2024-03-11T21:29:55.258 INFO:teuthology.orchestra.run.smithi097.stdout: "events": [ 2024-03-11T21:29:55.258 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:29:55.259 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "initiated", 2024-03-11T21:29:55.259 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:27:57.789455+0000", 2024-03-11T21:29:55.259 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-03-11T21:29:55.259 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:29:55.259 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:29:55.259 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "header_read", 2024-03-11T21:29:55.259 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:27:57.789455+0000", 2024-03-11T21:29:55.259 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-03-11T21:29:55.259 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:29:55.259 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:29:55.259 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "throttled", 2024-03-11T21:29:55.259 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:27:57.789455+0000", 2024-03-11T21:29:55.259 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 8.6600000000000005e-07 2024-03-11T21:29:55.259 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:29:55.259 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:29:55.260 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "all_read", 2024-03-11T21:29:55.260 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:27:57.789459+0000", 2024-03-11T21:29:55.260 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 3.41e-06 2024-03-11T21:29:55.260 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:29:55.260 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:29:55.260 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "dispatched", 2024-03-11T21:29:55.260 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:27:57.862118+0000", 2024-03-11T21:29:55.260 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.072659661 2024-03-11T21:29:55.260 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:29:55.260 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:29:55.260 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "queued_for_pg", 2024-03-11T21:29:55.260 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:27:57.862128+0000", 2024-03-11T21:29:55.260 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 9.5689999999999994e-06 2024-03-11T21:29:55.260 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:29:55.261 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:29:55.261 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "reached_pg", 2024-03-11T21:29:55.261 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:27:57.863288+0000", 2024-03-11T21:29:55.261 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.00116 2024-03-11T21:29:55.261 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:29:55.261 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:29:55.261 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "started", 2024-03-11T21:29:55.261 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:27:57.863298+0000", 2024-03-11T21:29:55.261 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 9.8439999999999993e-06 2024-03-11T21:29:55.261 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:29:55.261 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:29:55.261 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "done", 2024-03-11T21:29:55.261 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:27:57.863312+0000", 2024-03-11T21:29:55.261 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.4123e-05 2024-03-11T21:29:55.262 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-03-11T21:29:55.262 INFO:teuthology.orchestra.run.smithi097.stdout: ] 2024-03-11T21:29:55.262 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-03-11T21:29:55.262 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:29:55.262 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:29:55.262 INFO:teuthology.orchestra.run.smithi097.stdout: "description": "osd_repop_reply(client.34715.0:32061 7.1 e105/99 ondisk, result = 0)", 2024-03-11T21:29:55.262 INFO:teuthology.orchestra.run.smithi097.stdout: "initiated_at": "2024-03-11T21:27:57.790324+0000", 2024-03-11T21:29:55.262 INFO:teuthology.orchestra.run.smithi097.stdout: "age": 117.413709381, 2024-03-11T21:29:55.262 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.073103739000000001, 2024-03-11T21:29:55.262 INFO:teuthology.orchestra.run.smithi097.stdout: "type_data": { 2024-03-11T21:29:55.262 INFO:teuthology.orchestra.run.smithi097.stdout: "flag_point": "started", 2024-03-11T21:29:55.262 INFO:teuthology.orchestra.run.smithi097.stdout: "events": [ 2024-03-11T21:29:55.262 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:29:55.262 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "initiated", 2024-03-11T21:29:55.263 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:27:57.790324+0000", 2024-03-11T21:29:55.263 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-03-11T21:29:55.263 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:29:55.263 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:29:55.263 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "header_read", 2024-03-11T21:29:55.263 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:27:57.790324+0000", 2024-03-11T21:29:55.263 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-03-11T21:29:55.263 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:29:55.263 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:29:55.263 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "throttled", 2024-03-11T21:29:55.263 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:27:57.790325+0000", 2024-03-11T21:29:55.263 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 8.5300000000000003e-07 2024-03-11T21:29:55.263 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:29:55.263 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:29:55.264 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "all_read", 2024-03-11T21:29:55.264 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:27:57.790328+0000", 2024-03-11T21:29:55.264 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 3.2880000000000001e-06 2024-03-11T21:29:55.264 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:29:55.264 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:29:55.264 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "dispatched", 2024-03-11T21:29:55.264 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:27:57.862092+0000", 2024-03-11T21:29:55.264 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.071763847000000006 2024-03-11T21:29:55.264 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:29:55.264 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:29:55.264 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "queued_for_pg", 2024-03-11T21:29:55.264 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:27:57.862119+0000", 2024-03-11T21:29:55.264 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 2.6288e-05 2024-03-11T21:29:55.264 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:29:55.264 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:29:55.265 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "reached_pg", 2024-03-11T21:29:55.265 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:27:57.863405+0000", 2024-03-11T21:29:55.265 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.001286117 2024-03-11T21:29:55.265 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:29:55.265 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:29:55.265 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "started", 2024-03-11T21:29:55.265 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:27:57.863413+0000", 2024-03-11T21:29:55.265 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 8.1689999999999992e-06 2024-03-11T21:29:55.265 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:29:55.265 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:29:55.265 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "done", 2024-03-11T21:29:55.265 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:27:57.863428+0000", 2024-03-11T21:29:55.265 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.5177000000000001e-05 2024-03-11T21:29:55.265 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-03-11T21:29:55.266 INFO:teuthology.orchestra.run.smithi097.stdout: ] 2024-03-11T21:29:55.266 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-03-11T21:29:55.266 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:29:55.266 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:29:55.266 INFO:teuthology.orchestra.run.smithi097.stdout: "description": "osd_op(client.34715.0:70024 7.1 7:8cacfe50:::benchmark_data_smithi092_76040_object70023:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e109)", 2024-03-11T21:29:55.266 INFO:teuthology.orchestra.run.smithi097.stdout: "initiated_at": "2024-03-11T21:28:24.738322+0000", 2024-03-11T21:29:55.266 INFO:teuthology.orchestra.run.smithi097.stdout: "age": 90.465711704, 2024-03-11T21:29:55.266 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.069497423000000003, 2024-03-11T21:29:55.266 INFO:teuthology.orchestra.run.smithi097.stdout: "type_data": { 2024-03-11T21:29:55.266 INFO:teuthology.orchestra.run.smithi097.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-11T21:29:55.266 INFO:teuthology.orchestra.run.smithi097.stdout: "client_info": { 2024-03-11T21:29:55.266 INFO:teuthology.orchestra.run.smithi097.stdout: "client": "client.34715", 2024-03-11T21:29:55.266 INFO:teuthology.orchestra.run.smithi097.stdout: "client_addr": "v1:172.21.15.92:0/2403132573", 2024-03-11T21:29:55.266 INFO:teuthology.orchestra.run.smithi097.stdout: "tid": 70024 2024-03-11T21:29:55.266 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:29:55.267 INFO:teuthology.orchestra.run.smithi097.stdout: "events": [ 2024-03-11T21:29:55.267 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:29:55.267 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "initiated", 2024-03-11T21:29:55.267 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:28:24.738322+0000", 2024-03-11T21:29:55.267 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-03-11T21:29:55.267 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:29:55.267 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:29:55.267 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "header_read", 2024-03-11T21:29:55.267 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:28:24.738322+0000", 2024-03-11T21:29:55.267 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-03-11T21:29:55.267 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:29:55.267 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:29:55.267 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "throttled", 2024-03-11T21:29:55.267 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:28:24.738324+0000", 2024-03-11T21:29:55.267 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 2.2249999999999999e-06 2024-03-11T21:29:55.268 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:29:55.268 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:29:55.268 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "all_read", 2024-03-11T21:29:55.268 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:28:24.738384+0000", 2024-03-11T21:29:55.268 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 5.9314000000000002e-05 2024-03-11T21:29:55.268 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:29:55.268 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:29:55.268 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "dispatched", 2024-03-11T21:29:55.268 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:28:24.738394+0000", 2024-03-11T21:29:55.268 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.0644e-05 2024-03-11T21:29:55.268 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:29:55.268 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:29:55.268 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "queued_for_pg", 2024-03-11T21:29:55.268 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:28:24.738401+0000", 2024-03-11T21:29:55.269 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 6.9319999999999999e-06 2024-03-11T21:29:55.269 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:29:55.269 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:29:55.269 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "reached_pg", 2024-03-11T21:29:55.269 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:28:24.738460+0000", 2024-03-11T21:29:55.269 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 5.8434999999999999e-05 2024-03-11T21:29:55.269 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:29:55.269 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:29:55.269 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "started", 2024-03-11T21:29:55.269 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:28:24.738584+0000", 2024-03-11T21:29:55.269 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.00012423499999999999 2024-03-11T21:29:55.269 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:29:55.269 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:29:55.269 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "waiting for subops from 3,6", 2024-03-11T21:29:55.270 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:28:24.738712+0000", 2024-03-11T21:29:55.270 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.00012846400000000001 2024-03-11T21:29:55.270 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:29:55.270 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:29:55.270 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "sub_op_commit_rec", 2024-03-11T21:29:55.270 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:28:24.743450+0000", 2024-03-11T21:29:55.270 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.0047372250000000003 2024-03-11T21:29:55.270 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:29:55.270 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:29:55.270 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "op_commit", 2024-03-11T21:29:55.270 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:28:24.743472+0000", 2024-03-11T21:29:55.270 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 2.2925000000000002e-05 2024-03-11T21:29:55.270 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:29:55.271 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:29:55.271 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "sub_op_commit_rec", 2024-03-11T21:29:55.271 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:28:24.807679+0000", 2024-03-11T21:29:55.271 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.064206073000000002 2024-03-11T21:29:55.271 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:29:55.271 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:29:55.271 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "commit_sent", 2024-03-11T21:29:55.271 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:28:24.807784+0000", 2024-03-11T21:29:55.271 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.000105691 2024-03-11T21:29:55.271 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:29:55.271 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:29:55.271 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "done", 2024-03-11T21:29:55.271 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:28:24.807819+0000", 2024-03-11T21:29:55.272 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 3.5259999999999998e-05 2024-03-11T21:29:55.272 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-03-11T21:29:55.272 INFO:teuthology.orchestra.run.smithi097.stdout: ] 2024-03-11T21:29:55.272 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-03-11T21:29:55.272 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:29:55.272 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:29:55.272 INFO:teuthology.orchestra.run.smithi097.stdout: "description": "osd_op(client.34715.0:70026 7.1 7:866c3959:::benchmark_data_smithi092_76040_object70025:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e109)", 2024-03-11T21:29:55.272 INFO:teuthology.orchestra.run.smithi097.stdout: "initiated_at": "2024-03-11T21:28:24.738536+0000", 2024-03-11T21:29:55.272 INFO:teuthology.orchestra.run.smithi097.stdout: "age": 90.465497522000007, 2024-03-11T21:29:55.272 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.069385899000000001, 2024-03-11T21:29:55.272 INFO:teuthology.orchestra.run.smithi097.stdout: "type_data": { 2024-03-11T21:29:55.272 INFO:teuthology.orchestra.run.smithi097.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-11T21:29:55.272 INFO:teuthology.orchestra.run.smithi097.stdout: "client_info": { 2024-03-11T21:29:55.272 INFO:teuthology.orchestra.run.smithi097.stdout: "client": "client.34715", 2024-03-11T21:29:55.273 INFO:teuthology.orchestra.run.smithi097.stdout: "client_addr": "v1:172.21.15.92:0/2403132573", 2024-03-11T21:29:55.273 INFO:teuthology.orchestra.run.smithi097.stdout: "tid": 70026 2024-03-11T21:29:55.273 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:29:55.273 INFO:teuthology.orchestra.run.smithi097.stdout: "events": [ 2024-03-11T21:29:55.273 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:29:55.273 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "initiated", 2024-03-11T21:29:55.273 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:28:24.738536+0000", 2024-03-11T21:29:55.273 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-03-11T21:29:55.273 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:29:55.273 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:29:55.273 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "header_read", 2024-03-11T21:29:55.273 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:28:24.738536+0000", 2024-03-11T21:29:55.273 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-03-11T21:29:55.274 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:29:55.274 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:29:55.274 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "throttled", 2024-03-11T21:29:55.274 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:28:24.738537+0000", 2024-03-11T21:29:55.274 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.026e-06 2024-03-11T21:29:55.274 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:29:55.274 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:29:55.274 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "all_read", 2024-03-11T21:29:55.274 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:28:24.738611+0000", 2024-03-11T21:29:55.274 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 7.3594e-05 2024-03-11T21:29:55.274 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:29:55.274 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:29:55.274 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "dispatched", 2024-03-11T21:29:55.275 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:28:24.738621+0000", 2024-03-11T21:29:55.275 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.0628999999999999e-05 2024-03-11T21:29:55.275 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:29:55.275 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:29:55.275 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "queued_for_pg", 2024-03-11T21:29:55.275 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:28:24.738629+0000", 2024-03-11T21:29:55.275 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 7.2529999999999998e-06 2024-03-11T21:29:55.275 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:29:55.275 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:29:55.275 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "reached_pg", 2024-03-11T21:29:55.275 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:28:24.740821+0000", 2024-03-11T21:29:55.275 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.0021927489999999999 2024-03-11T21:29:55.275 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:29:55.276 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:29:55.276 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "started", 2024-03-11T21:29:55.276 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:28:24.740945+0000", 2024-03-11T21:29:55.276 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.00012362300000000001 2024-03-11T21:29:55.276 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:29:55.276 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:29:55.276 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "waiting for subops from 3,6", 2024-03-11T21:29:55.276 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:28:24.741046+0000", 2024-03-11T21:29:55.276 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.000100975 2024-03-11T21:29:55.276 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:29:55.276 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:29:55.276 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "op_commit", 2024-03-11T21:29:55.276 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:28:24.743548+0000", 2024-03-11T21:29:55.276 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.0025022820000000002 2024-03-11T21:29:55.277 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:29:55.277 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:29:55.277 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "sub_op_commit_rec", 2024-03-11T21:29:55.277 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:28:24.743766+0000", 2024-03-11T21:29:55.277 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.00021741999999999999 2024-03-11T21:29:55.277 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:29:55.277 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:29:55.277 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "sub_op_commit_rec", 2024-03-11T21:29:55.277 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:28:24.807853+0000", 2024-03-11T21:29:55.277 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.064087273 2024-03-11T21:29:55.277 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:29:55.277 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:29:55.277 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "commit_sent", 2024-03-11T21:29:55.277 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:28:24.807887+0000", 2024-03-11T21:29:55.277 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 3.3779000000000003e-05 2024-03-11T21:29:55.278 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:29:55.278 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:29:55.278 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "done", 2024-03-11T21:29:55.278 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:28:24.807922+0000", 2024-03-11T21:29:55.278 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 3.5296000000000002e-05 2024-03-11T21:29:55.278 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-03-11T21:29:55.278 INFO:teuthology.orchestra.run.smithi097.stdout: ] 2024-03-11T21:29:55.278 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-03-11T21:29:55.278 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:29:55.278 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:29:55.278 INFO:teuthology.orchestra.run.smithi097.stdout: "description": "osd_op(client.34715.0:115996 7.1 7:825d57d9:::benchmark_data_smithi092_76040_object115995:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e119)", 2024-03-11T21:29:55.278 INFO:teuthology.orchestra.run.smithi097.stdout: "initiated_at": "2024-03-11T21:29:08.463013+0000", 2024-03-11T21:29:55.278 INFO:teuthology.orchestra.run.smithi097.stdout: "age": 46.741020984999999, 2024-03-11T21:29:55.278 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.065927562999999995, 2024-03-11T21:29:55.279 INFO:teuthology.orchestra.run.smithi097.stdout: "type_data": { 2024-03-11T21:29:55.279 INFO:teuthology.orchestra.run.smithi097.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-11T21:29:55.279 INFO:teuthology.orchestra.run.smithi097.stdout: "client_info": { 2024-03-11T21:29:55.279 INFO:teuthology.orchestra.run.smithi097.stdout: "client": "client.34715", 2024-03-11T21:29:55.279 INFO:teuthology.orchestra.run.smithi097.stdout: "client_addr": "v1:172.21.15.92:0/2403132573", 2024-03-11T21:29:55.279 INFO:teuthology.orchestra.run.smithi097.stdout: "tid": 115996 2024-03-11T21:29:55.279 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:29:55.279 INFO:teuthology.orchestra.run.smithi097.stdout: "events": [ 2024-03-11T21:29:55.279 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:29:55.279 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "initiated", 2024-03-11T21:29:55.279 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:29:08.463013+0000", 2024-03-11T21:29:55.279 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-03-11T21:29:55.279 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:29:55.279 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:29:55.279 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "header_read", 2024-03-11T21:29:55.280 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:29:08.463013+0000", 2024-03-11T21:29:55.280 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-03-11T21:29:55.280 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:29:55.280 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:29:55.280 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "throttled", 2024-03-11T21:29:55.280 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:29:08.463014+0000", 2024-03-11T21:29:55.280 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.4720000000000001e-06 2024-03-11T21:29:55.280 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:29:55.280 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:29:55.280 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "all_read", 2024-03-11T21:29:55.280 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:29:08.463052+0000", 2024-03-11T21:29:55.280 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 3.8003000000000002e-05 2024-03-11T21:29:55.280 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:29:55.280 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:29:55.280 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "dispatched", 2024-03-11T21:29:55.281 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:29:08.463066+0000", 2024-03-11T21:29:55.281 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.3869000000000001e-05 2024-03-11T21:29:55.281 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:29:55.281 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:29:55.281 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "queued_for_pg", 2024-03-11T21:29:55.281 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:29:08.463074+0000", 2024-03-11T21:29:55.281 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 7.7190000000000001e-06 2024-03-11T21:29:55.281 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:29:55.281 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:29:55.281 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "reached_pg", 2024-03-11T21:29:55.281 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:29:08.464705+0000", 2024-03-11T21:29:55.281 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.001631144 2024-03-11T21:29:55.281 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:29:55.281 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:29:55.281 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "started", 2024-03-11T21:29:55.282 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:29:08.464839+0000", 2024-03-11T21:29:55.282 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.000134041 2024-03-11T21:29:55.282 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:29:55.282 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:29:55.282 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "waiting for subops from 3,6", 2024-03-11T21:29:55.282 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:29:08.464988+0000", 2024-03-11T21:29:55.282 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.000148529 2024-03-11T21:29:55.282 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:29:55.282 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:29:55.282 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "op_commit", 2024-03-11T21:29:55.282 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:29:08.467087+0000", 2024-03-11T21:29:55.282 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.002099409 2024-03-11T21:29:55.282 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:29:55.282 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:29:55.282 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "sub_op_commit_rec", 2024-03-11T21:29:55.283 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:29:08.467258+0000", 2024-03-11T21:29:55.283 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.00017126500000000001 2024-03-11T21:29:55.283 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:29:55.283 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:29:55.283 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "sub_op_commit_rec", 2024-03-11T21:29:55.283 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:29:08.528832+0000", 2024-03-11T21:29:55.283 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.061574265000000003 2024-03-11T21:29:55.283 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:29:55.283 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:29:55.283 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "commit_sent", 2024-03-11T21:29:55.283 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:29:08.528895+0000", 2024-03-11T21:29:55.283 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 6.2652999999999998e-05 2024-03-11T21:29:55.283 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:29:55.283 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:29:55.284 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "done", 2024-03-11T21:29:55.284 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:29:08.528940+0000", 2024-03-11T21:29:55.284 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 4.5194000000000001e-05 2024-03-11T21:29:55.284 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-03-11T21:29:55.284 INFO:teuthology.orchestra.run.smithi097.stdout: ] 2024-03-11T21:29:55.284 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-03-11T21:29:55.284 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:29:55.284 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:29:55.284 INFO:teuthology.orchestra.run.smithi097.stdout: "description": "osd_op(client.34715.0:116275 7.1 7:889dcf6e:::benchmark_data_smithi092_76040_object116274:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e119)", 2024-03-11T21:29:55.284 INFO:teuthology.orchestra.run.smithi097.stdout: "initiated_at": "2024-03-11T21:29:08.765237+0000", 2024-03-11T21:29:55.284 INFO:teuthology.orchestra.run.smithi097.stdout: "age": 46.438796932999999, 2024-03-11T21:29:55.284 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.11311642700000001, 2024-03-11T21:29:55.284 INFO:teuthology.orchestra.run.smithi097.stdout: "type_data": { 2024-03-11T21:29:55.284 INFO:teuthology.orchestra.run.smithi097.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-11T21:29:55.284 INFO:teuthology.orchestra.run.smithi097.stdout: "client_info": { 2024-03-11T21:29:55.285 INFO:teuthology.orchestra.run.smithi097.stdout: "client": "client.34715", 2024-03-11T21:29:55.285 INFO:teuthology.orchestra.run.smithi097.stdout: "client_addr": "v1:172.21.15.92:0/2403132573", 2024-03-11T21:29:55.285 INFO:teuthology.orchestra.run.smithi097.stdout: "tid": 116275 2024-03-11T21:29:55.285 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:29:55.285 INFO:teuthology.orchestra.run.smithi097.stdout: "events": [ 2024-03-11T21:29:55.285 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:29:55.285 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "initiated", 2024-03-11T21:29:55.285 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:29:08.765237+0000", 2024-03-11T21:29:55.285 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-03-11T21:29:55.285 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:29:55.285 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:29:55.285 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "header_read", 2024-03-11T21:29:55.285 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:29:08.765237+0000", 2024-03-11T21:29:55.285 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-03-11T21:29:55.286 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:29:55.286 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:29:55.286 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "throttled", 2024-03-11T21:29:55.286 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:29:08.765239+0000", 2024-03-11T21:29:55.286 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.9400000000000001e-06 2024-03-11T21:29:55.286 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:29:55.287 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:29:55.288 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:29:55.288 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "all_read", 2024-03-11T21:29:55.288 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:29:08.765317+0000", 2024-03-11T21:29:55.288 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 7.8537999999999999e-05 2024-03-11T21:29:55.288 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:29:55.288 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:29:55.288 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "dispatched", 2024-03-11T21:29:55.288 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:29:08.765331+0000", 2024-03-11T21:29:55.288 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.3453e-05 2024-03-11T21:29:55.288 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:29:55.288 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:29:55.288 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "queued_for_pg", 2024-03-11T21:29:55.288 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:29:08.765339+0000", 2024-03-11T21:29:55.289 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 7.7980000000000006e-06 2024-03-11T21:29:55.289 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:29:55.289 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:29:55.289 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "reached_pg", 2024-03-11T21:29:55.289 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:29:08.765377+0000", 2024-03-11T21:29:55.289 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 3.8593000000000001e-05 2024-03-11T21:29:55.289 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:29:55.289 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:29:55.289 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "started", 2024-03-11T21:29:55.289 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:29:08.765489+0000", 2024-03-11T21:29:55.289 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.00011158300000000001 2024-03-11T21:29:55.289 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:29:55.289 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:29:55.289 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "waiting for subops from 3,6", 2024-03-11T21:29:55.289 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:29:08.765629+0000", 2024-03-11T21:29:55.290 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.00014014299999999999 2024-03-11T21:29:55.290 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:29:55.290 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:29:55.290 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "op_commit", 2024-03-11T21:29:55.290 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:29:08.769442+0000", 2024-03-11T21:29:55.290 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.0038130519999999999 2024-03-11T21:29:55.290 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:29:55.290 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:29:55.290 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "sub_op_commit_rec", 2024-03-11T21:29:55.290 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:29:08.771245+0000", 2024-03-11T21:29:55.290 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.0018028300000000001 2024-03-11T21:29:55.290 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:29:55.290 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:29:55.291 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "sub_op_commit_rec", 2024-03-11T21:29:55.291 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:29:08.878254+0000", 2024-03-11T21:29:55.291 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.107009544 2024-03-11T21:29:55.291 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:29:55.291 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:29:55.291 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "commit_sent", 2024-03-11T21:29:55.291 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:29:08.878307+0000", 2024-03-11T21:29:55.291 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 5.27e-05 2024-03-11T21:29:55.291 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:29:55.291 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:29:55.291 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "done", 2024-03-11T21:29:55.291 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:29:08.878353+0000", 2024-03-11T21:29:55.291 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 4.6252999999999999e-05 2024-03-11T21:29:55.291 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-03-11T21:29:55.292 INFO:teuthology.orchestra.run.smithi097.stdout: ] 2024-03-11T21:29:55.292 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-03-11T21:29:55.292 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:29:55.292 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:29:55.292 INFO:teuthology.orchestra.run.smithi097.stdout: "description": "osd_op(client.34715.0:116285 7.1 7:8ebc24c4:::benchmark_data_smithi092_76040_object116284:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e119)", 2024-03-11T21:29:55.292 INFO:teuthology.orchestra.run.smithi097.stdout: "initiated_at": "2024-03-11T21:29:08.771631+0000", 2024-03-11T21:29:55.292 INFO:teuthology.orchestra.run.smithi097.stdout: "age": 46.432402691, 2024-03-11T21:29:55.292 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.121431314, 2024-03-11T21:29:55.292 INFO:teuthology.orchestra.run.smithi097.stdout: "type_data": { 2024-03-11T21:29:55.292 INFO:teuthology.orchestra.run.smithi097.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-11T21:29:55.292 INFO:teuthology.orchestra.run.smithi097.stdout: "client_info": { 2024-03-11T21:29:55.292 INFO:teuthology.orchestra.run.smithi097.stdout: "client": "client.34715", 2024-03-11T21:29:55.293 INFO:teuthology.orchestra.run.smithi097.stdout: "client_addr": "v1:172.21.15.92:0/2403132573", 2024-03-11T21:29:55.293 INFO:teuthology.orchestra.run.smithi097.stdout: "tid": 116285 2024-03-11T21:29:55.293 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:29:55.293 INFO:teuthology.orchestra.run.smithi097.stdout: "events": [ 2024-03-11T21:29:55.293 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:29:55.293 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "initiated", 2024-03-11T21:29:55.293 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:29:08.771631+0000", 2024-03-11T21:29:55.293 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-03-11T21:29:55.293 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:29:55.293 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:29:55.293 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "header_read", 2024-03-11T21:29:55.293 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:29:08.771631+0000", 2024-03-11T21:29:55.293 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-03-11T21:29:55.293 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:29:55.293 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:29:55.294 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "throttled", 2024-03-11T21:29:55.294 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:29:08.771633+0000", 2024-03-11T21:29:55.294 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 2.013e-06 2024-03-11T21:29:55.294 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:29:55.294 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:29:55.294 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "all_read", 2024-03-11T21:29:55.294 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:29:08.771682+0000", 2024-03-11T21:29:55.294 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 4.9419999999999998e-05 2024-03-11T21:29:55.294 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:29:55.294 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:29:55.294 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "dispatched", 2024-03-11T21:29:55.294 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:29:08.771695+0000", 2024-03-11T21:29:55.294 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.2486e-05 2024-03-11T21:29:55.294 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:29:55.295 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:29:55.295 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "queued_for_pg", 2024-03-11T21:29:55.295 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:29:08.771703+0000", 2024-03-11T21:29:55.295 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 8.4460000000000001e-06 2024-03-11T21:29:55.295 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:29:55.295 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:29:55.295 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "reached_pg", 2024-03-11T21:29:55.295 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:29:08.771756+0000", 2024-03-11T21:29:55.295 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 5.2992999999999999e-05 2024-03-11T21:29:55.295 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:29:55.295 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:29:55.295 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "started", 2024-03-11T21:29:55.295 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:29:08.771917+0000", 2024-03-11T21:29:55.295 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.00016044900000000001 2024-03-11T21:29:55.295 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:29:55.296 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:29:55.296 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "waiting for subops from 3,6", 2024-03-11T21:29:55.296 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:29:08.772013+0000", 2024-03-11T21:29:55.296 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 9.6419e-05 2024-03-11T21:29:55.296 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:29:55.296 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:29:55.296 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "sub_op_commit_rec", 2024-03-11T21:29:55.296 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:29:08.778457+0000", 2024-03-11T21:29:55.296 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.0064432739999999997 2024-03-11T21:29:55.296 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:29:55.296 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:29:55.296 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "op_commit", 2024-03-11T21:29:55.296 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:29:08.778552+0000", 2024-03-11T21:29:55.296 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 9.5730999999999995e-05 2024-03-11T21:29:55.296 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:29:55.297 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:29:55.297 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "sub_op_commit_rec", 2024-03-11T21:29:55.297 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:29:08.892956+0000", 2024-03-11T21:29:55.297 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.114403941 2024-03-11T21:29:55.297 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:29:55.297 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:29:55.297 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "commit_sent", 2024-03-11T21:29:55.297 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:29:08.893016+0000", 2024-03-11T21:29:55.297 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 5.982e-05 2024-03-11T21:29:55.297 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:29:55.297 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:29:55.297 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "done", 2024-03-11T21:29:55.297 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:29:08.893062+0000", 2024-03-11T21:29:55.297 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 4.6322000000000001e-05 2024-03-11T21:29:55.297 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-03-11T21:29:55.298 INFO:teuthology.orchestra.run.smithi097.stdout: ] 2024-03-11T21:29:55.298 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-03-11T21:29:55.298 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:29:55.298 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:29:55.298 INFO:teuthology.orchestra.run.smithi097.stdout: "description": "osd_op(client.34715.0:116912 7.1 7:8e70114b:::benchmark_data_smithi092_76040_object116911:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e119)", 2024-03-11T21:29:55.298 INFO:teuthology.orchestra.run.smithi097.stdout: "initiated_at": "2024-03-11T21:29:09.520557+0000", 2024-03-11T21:29:55.298 INFO:teuthology.orchestra.run.smithi097.stdout: "age": 45.683476360999997, 2024-03-11T21:29:55.298 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.063563035000000004, 2024-03-11T21:29:55.298 INFO:teuthology.orchestra.run.smithi097.stdout: "type_data": { 2024-03-11T21:29:55.298 INFO:teuthology.orchestra.run.smithi097.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-11T21:29:55.298 INFO:teuthology.orchestra.run.smithi097.stdout: "client_info": { 2024-03-11T21:29:55.298 INFO:teuthology.orchestra.run.smithi097.stdout: "client": "client.34715", 2024-03-11T21:29:55.298 INFO:teuthology.orchestra.run.smithi097.stdout: "client_addr": "v1:172.21.15.92:0/2403132573", 2024-03-11T21:29:55.298 INFO:teuthology.orchestra.run.smithi097.stdout: "tid": 116912 2024-03-11T21:29:55.299 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:29:55.299 INFO:teuthology.orchestra.run.smithi097.stdout: "events": [ 2024-03-11T21:29:55.299 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:29:55.299 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "initiated", 2024-03-11T21:29:55.299 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:29:09.520557+0000", 2024-03-11T21:29:55.299 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-03-11T21:29:55.299 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:29:55.299 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:29:55.299 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "header_read", 2024-03-11T21:29:55.299 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:29:09.520557+0000", 2024-03-11T21:29:55.299 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-03-11T21:29:55.299 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:29:55.299 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:29:55.299 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "throttled", 2024-03-11T21:29:55.299 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:29:09.520558+0000", 2024-03-11T21:29:55.300 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 9.2099999999999995e-07 2024-03-11T21:29:55.300 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:29:55.300 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:29:55.300 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "all_read", 2024-03-11T21:29:55.300 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:29:09.520616+0000", 2024-03-11T21:29:55.300 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 5.7379000000000001e-05 2024-03-11T21:29:55.300 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:29:55.300 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:29:55.300 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "dispatched", 2024-03-11T21:29:55.300 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:29:09.520631+0000", 2024-03-11T21:29:55.300 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.5761999999999999e-05 2024-03-11T21:29:55.300 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:29:55.300 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:29:55.300 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "queued_for_pg", 2024-03-11T21:29:55.300 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:29:09.520640+0000", 2024-03-11T21:29:55.301 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 8.4700000000000002e-06 2024-03-11T21:29:55.301 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:29:55.301 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:29:55.301 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "reached_pg", 2024-03-11T21:29:55.301 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:29:09.522377+0000", 2024-03-11T21:29:55.301 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.0017373460000000001 2024-03-11T21:29:55.301 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:29:55.301 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:29:55.301 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "started", 2024-03-11T21:29:55.301 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:29:09.522503+0000", 2024-03-11T21:29:55.301 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.000125909 2024-03-11T21:29:55.301 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:29:55.301 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:29:55.301 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "waiting for subops from 3,6", 2024-03-11T21:29:55.302 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:29:09.522603+0000", 2024-03-11T21:29:55.302 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 9.9902999999999995e-05 2024-03-11T21:29:55.302 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:29:55.302 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:29:55.302 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "sub_op_commit_rec", 2024-03-11T21:29:55.302 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:29:09.525032+0000", 2024-03-11T21:29:55.302 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.0024290169999999999 2024-03-11T21:29:55.302 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:29:55.302 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:29:55.302 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "op_commit", 2024-03-11T21:29:55.302 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:29:09.527076+0000", 2024-03-11T21:29:55.302 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.0020435539999999999 2024-03-11T21:29:55.302 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:29:55.303 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:29:55.303 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "sub_op_commit_rec", 2024-03-11T21:29:55.303 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:29:09.583955+0000", 2024-03-11T21:29:55.303 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.056878999 2024-03-11T21:29:55.303 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:29:55.303 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:29:55.303 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "commit_sent", 2024-03-11T21:29:55.303 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:29:09.584062+0000", 2024-03-11T21:29:55.303 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.000107687 2024-03-11T21:29:55.303 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:29:55.303 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:29:55.303 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "done", 2024-03-11T21:29:55.303 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:29:09.584120+0000", 2024-03-11T21:29:55.303 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 5.8087999999999997e-05 2024-03-11T21:29:55.303 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-03-11T21:29:55.304 INFO:teuthology.orchestra.run.smithi097.stdout: ] 2024-03-11T21:29:55.304 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-03-11T21:29:55.304 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:29:55.304 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:29:55.304 INFO:teuthology.orchestra.run.smithi097.stdout: "description": "osd_op(client.34715.0:136195 7.1 7:8932f32c:::benchmark_data_smithi092_76040_object14284:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e124)", 2024-03-11T21:29:55.304 INFO:teuthology.orchestra.run.smithi097.stdout: "initiated_at": "2024-03-11T21:29:30.578255+0000", 2024-03-11T21:29:55.304 INFO:teuthology.orchestra.run.smithi097.stdout: "age": 24.625778557, 2024-03-11T21:29:55.304 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.064222313000000003, 2024-03-11T21:29:55.304 INFO:teuthology.orchestra.run.smithi097.stdout: "type_data": { 2024-03-11T21:29:55.304 INFO:teuthology.orchestra.run.smithi097.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-11T21:29:55.304 INFO:teuthology.orchestra.run.smithi097.stdout: "client_info": { 2024-03-11T21:29:55.304 INFO:teuthology.orchestra.run.smithi097.stdout: "client": "client.34715", 2024-03-11T21:29:55.304 INFO:teuthology.orchestra.run.smithi097.stdout: "client_addr": "v1:172.21.15.92:0/2403132573", 2024-03-11T21:29:55.304 INFO:teuthology.orchestra.run.smithi097.stdout: "tid": 136195 2024-03-11T21:29:55.305 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:29:55.305 INFO:teuthology.orchestra.run.smithi097.stdout: "events": [ 2024-03-11T21:29:55.305 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:29:55.305 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "initiated", 2024-03-11T21:29:55.305 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:29:30.578255+0000", 2024-03-11T21:29:55.305 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-03-11T21:29:55.305 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:29:55.305 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:29:55.305 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "header_read", 2024-03-11T21:29:55.305 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:29:30.578255+0000", 2024-03-11T21:29:55.305 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-03-11T21:29:55.305 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:29:55.305 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:29:55.305 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "throttled", 2024-03-11T21:29:55.305 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:29:30.578257+0000", 2024-03-11T21:29:55.306 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.9180000000000001e-06 2024-03-11T21:29:55.306 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:29:55.306 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:29:55.306 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "all_read", 2024-03-11T21:29:55.306 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:29:30.578263+0000", 2024-03-11T21:29:55.306 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 5.8170000000000004e-06 2024-03-11T21:29:55.306 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:29:55.306 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:29:55.306 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "dispatched", 2024-03-11T21:29:55.306 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:29:30.578274+0000", 2024-03-11T21:29:55.306 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.0937999999999999e-05 2024-03-11T21:29:55.306 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:29:55.306 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:29:55.306 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "queued_for_pg", 2024-03-11T21:29:55.306 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:29:30.578282+0000", 2024-03-11T21:29:55.307 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 7.7500000000000003e-06 2024-03-11T21:29:55.307 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:29:55.307 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:29:55.307 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "reached_pg", 2024-03-11T21:29:55.307 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:29:30.580051+0000", 2024-03-11T21:29:55.307 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.001769512 2024-03-11T21:29:55.307 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:29:55.307 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:29:55.307 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "started", 2024-03-11T21:29:55.307 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:29:30.580141+0000", 2024-03-11T21:29:55.307 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 9.0290000000000005e-05 2024-03-11T21:29:55.307 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:29:55.307 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:29:55.307 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "waiting for subops from 3,6", 2024-03-11T21:29:55.308 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:29:30.580216+0000", 2024-03-11T21:29:55.308 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 7.4737e-05 2024-03-11T21:29:55.308 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:29:55.308 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:29:55.308 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "op_commit", 2024-03-11T21:29:55.308 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:29:30.584204+0000", 2024-03-11T21:29:55.308 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.0039877649999999999 2024-03-11T21:29:55.308 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:29:55.308 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:29:55.308 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "sub_op_commit_rec", 2024-03-11T21:29:55.308 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:29:30.584243+0000", 2024-03-11T21:29:55.308 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 3.9032999999999997e-05 2024-03-11T21:29:55.308 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:29:55.308 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:29:55.308 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "sub_op_commit_rec", 2024-03-11T21:29:55.309 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:29:30.642340+0000", 2024-03-11T21:29:55.309 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.058097459999999997 2024-03-11T21:29:55.309 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:29:55.309 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:29:55.309 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "commit_sent", 2024-03-11T21:29:55.309 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:29:30.642406+0000", 2024-03-11T21:29:55.309 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 6.5951000000000006e-05 2024-03-11T21:29:55.309 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:29:55.309 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:29:55.309 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "done", 2024-03-11T21:29:55.309 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:29:30.642477+0000", 2024-03-11T21:29:55.309 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 7.1142000000000004e-05 2024-03-11T21:29:55.309 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-03-11T21:29:55.309 INFO:teuthology.orchestra.run.smithi097.stdout: ] 2024-03-11T21:29:55.309 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-03-11T21:29:55.310 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-03-11T21:29:55.310 INFO:teuthology.orchestra.run.smithi097.stdout: ] 2024-03-11T21:29:55.310 INFO:teuthology.orchestra.run.smithi097.stdout:} 2024-03-11T21:29:55.330 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:29:55.331 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:29:55.331 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:29:55.337 DEBUG:teuthology.orchestra.run.smithi092:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid 00db0016-dfed-11ee-95c8-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-03-11T21:29:55.432 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:29:55.459 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:29:55.460 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:29:55.460 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:29:55.561 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:29:55.588 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:29:55.589 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:29:55.590 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:29:55.691 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:29:55.719 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:29:55.720 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:29:55.720 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:29:55.821 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:29:55.850 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:29:55.851 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:29:55.851 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:29:55.952 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:29:56.004 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:29:56.004 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:29:56.005 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:29:56.105 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:29:56.110 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:29:56 smithi092 ceph-mon[29572]: pgmap v149: 148 pgs: 146 active+clean, 2 active+clean+scrubbing+deep; 5.5 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 687 op/s 2024-03-11T21:29:56.111 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:29:56 smithi092 ceph-mon[37563]: pgmap v149: 148 pgs: 146 active+clean, 2 active+clean+scrubbing+deep; 5.5 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 687 op/s 2024-03-11T21:29:56.140 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:29:56.140 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:29:56.141 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:29:56.241 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:29:56.269 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:29:56.270 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:29:56.270 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:29:56.326 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:29:56 smithi097 ceph-mon[33317]: pgmap v149: 148 pgs: 146 active+clean, 2 active+clean+scrubbing+deep; 5.5 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 687 op/s 2024-03-11T21:29:56.329 DEBUG:teuthology.orchestra.run.smithi097:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid 00db0016-dfed-11ee-95c8-87774f69a715 -- ceph daemon osd.5 dump_ops_in_flight 2024-03-11T21:29:56.371 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:29:56.406 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:29:56.406 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:29:56.407 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:29:56.507 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:29:56.534 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:29:56.535 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:29:56.535 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:29:56.636 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:29:56.637 INFO:teuthology.orchestra.run.smithi097.stderr:Inferring config /var/lib/ceph/00db0016-dfed-11ee-95c8-87774f69a715/mon.b/config 2024-03-11T21:29:56.677 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:29:56.678 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:29:56.678 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:29:56.779 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:29:56.807 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:29:56.808 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:29:56.808 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:29:56.909 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:29:56.936 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:29:56.937 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:29:56.938 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:29:57.039 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:29:57.088 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:29:57.089 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:29:57.089 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:29:57.190 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:29:57.195 INFO:teuthology.orchestra.run.smithi092.stderr:Inferring config /var/lib/ceph/00db0016-dfed-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T21:29:57.258 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:29:57.259 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:29:57.260 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:29:57.360 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:29:57.388 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:29:57.388 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:29:57.389 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:29:57.489 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:29:57.515 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:29:57.516 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:29:57.516 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:29:57.617 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:29:57.658 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:29:57.658 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:29:57.659 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:29:57.759 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:29:57.806 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:29:57.806 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:29:57.806 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:29:57.907 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:29:57.943 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:29:57.944 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:29:57.944 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:29:58.045 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:29:58.077 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:29:58.078 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:29:58.079 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:29:58.096 INFO:teuthology.orchestra.run.smithi092.stderr:nodeep-scrub is set 2024-03-11T21:29:58.096 INFO:teuthology.orchestra.run.smithi092.stderr:reweighted osd.0 to 1 (10000) 2024-03-11T21:29:58.166 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:29:58 smithi097 ceph-mon[33317]: pgmap v150: 148 pgs: 146 active+clean, 2 active+clean+scrubbing+deep; 5.5 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 565 B/s rd, 0 B/s wr, 488 op/s 2024-03-11T21:29:58.166 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:29:58 smithi097 ceph-mon[33317]: from='client.? v1:172.21.15.92:0/1148914874' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]: dispatch 2024-03-11T21:29:58.166 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:29:58 smithi097 ceph-mon[33317]: from='client.24856 ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]: dispatch 2024-03-11T21:29:58.166 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:29:58 smithi097 ceph-mon[33317]: from='client.? v1:172.21.15.92:0/2291500351' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-11T21:29:58.167 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:29:58 smithi097 ceph-mon[33317]: from='client.24862 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-11T21:29:58.180 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:29:58.203 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:29:58.204 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:29:58.204 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:29:58.305 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:29:58.346 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:29:58.347 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:29:58.347 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:29:58.360 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:29:58 smithi092 ceph-mon[29572]: pgmap v150: 148 pgs: 146 active+clean, 2 active+clean+scrubbing+deep; 5.5 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 565 B/s rd, 0 B/s wr, 488 op/s 2024-03-11T21:29:58.360 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:29:58 smithi092 ceph-mon[29572]: from='client.? v1:172.21.15.92:0/1148914874' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]: dispatch 2024-03-11T21:29:58.360 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:29:58 smithi092 ceph-mon[29572]: from='client.24856 ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]: dispatch 2024-03-11T21:29:58.361 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:29:58 smithi092 ceph-mon[29572]: from='client.? v1:172.21.15.92:0/2291500351' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-11T21:29:58.361 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:29:58 smithi092 ceph-mon[29572]: from='client.24862 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-11T21:29:58.361 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:29:58 smithi092 ceph-mon[37563]: pgmap v150: 148 pgs: 146 active+clean, 2 active+clean+scrubbing+deep; 5.5 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 565 B/s rd, 0 B/s wr, 488 op/s 2024-03-11T21:29:58.361 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:29:58 smithi092 ceph-mon[37563]: from='client.? v1:172.21.15.92:0/1148914874' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]: dispatch 2024-03-11T21:29:58.361 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:29:58 smithi092 ceph-mon[37563]: from='client.24856 ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]: dispatch 2024-03-11T21:29:58.361 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:29:58 smithi092 ceph-mon[37563]: from='client.? v1:172.21.15.92:0/2291500351' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-11T21:29:58.361 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:29:58 smithi092 ceph-mon[37563]: from='client.24862 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-11T21:29:58.361 INFO:journalctl@ceph.mgr.y.smithi092.stdout:Mar 11 21:29:57 smithi092 ceph-00db0016-dfed-11ee-95c8-87774f69a715-mgr-y[29801]: ::ffff:172.21.15.97 - - [11/Mar/2024:21:29:57] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-11T21:29:58.448 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:29:58.476 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:29:58.477 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:29:58.477 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:29:58.578 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:29:58.620 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:29:58.621 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:29:58.621 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:29:58.694 INFO:teuthology.orchestra.run.smithi097.stdout:{ 2024-03-11T21:29:58.694 INFO:teuthology.orchestra.run.smithi097.stdout: "ops": [ 2024-03-11T21:29:58.694 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:29:58.694 INFO:teuthology.orchestra.run.smithi097.stdout: "description": "osd_op(client.34715.0:157499 7.0 7:062e5855:::benchmark_data_smithi092_76040_object35588:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e127)", 2024-03-11T21:29:58.694 INFO:teuthology.orchestra.run.smithi097.stdout: "initiated_at": "2024-03-11T21:29:58.696749+0000", 2024-03-11T21:29:58.694 INFO:teuthology.orchestra.run.smithi097.stdout: "age": 0.012395788, 2024-03-11T21:29:58.694 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.012426639, 2024-03-11T21:29:58.694 INFO:teuthology.orchestra.run.smithi097.stdout: "type_data": { 2024-03-11T21:29:58.695 INFO:teuthology.orchestra.run.smithi097.stdout: "flag_point": "waiting for sub ops", 2024-03-11T21:29:58.695 INFO:teuthology.orchestra.run.smithi097.stdout: "client_info": { 2024-03-11T21:29:58.695 INFO:teuthology.orchestra.run.smithi097.stdout: "client": "client.34715", 2024-03-11T21:29:58.695 INFO:teuthology.orchestra.run.smithi097.stdout: "client_addr": "v1:172.21.15.92:0/2403132573", 2024-03-11T21:29:58.695 INFO:teuthology.orchestra.run.smithi097.stdout: "tid": 157499 2024-03-11T21:29:58.695 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:29:58.695 INFO:teuthology.orchestra.run.smithi097.stdout: "events": [ 2024-03-11T21:29:58.695 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:29:58.695 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "initiated", 2024-03-11T21:29:58.695 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:29:58.696749+0000", 2024-03-11T21:29:58.695 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-03-11T21:29:58.695 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:29:58.696 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:29:58.696 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "header_read", 2024-03-11T21:29:58.696 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:29:58.696749+0000", 2024-03-11T21:29:58.696 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-03-11T21:29:58.696 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:29:58.696 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:29:58.696 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "throttled", 2024-03-11T21:29:58.696 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:29:58.696751+0000", 2024-03-11T21:29:58.696 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 2.2309999999999999e-06 2024-03-11T21:29:58.696 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:29:58.696 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:29:58.696 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "all_read", 2024-03-11T21:29:58.696 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:29:58.696759+0000", 2024-03-11T21:29:58.696 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 7.182e-06 2024-03-11T21:29:58.697 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:29:58.697 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:29:58.697 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "dispatched", 2024-03-11T21:29:58.697 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:29:58.696775+0000", 2024-03-11T21:29:58.697 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.6629999999999998e-05 2024-03-11T21:29:58.697 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:29:58.697 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:29:58.697 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "queued_for_pg", 2024-03-11T21:29:58.697 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:29:58.696786+0000", 2024-03-11T21:29:58.697 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.0319000000000001e-05 2024-03-11T21:29:58.697 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:29:58.697 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:29:58.697 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "reached_pg", 2024-03-11T21:29:58.697 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:29:58.696852+0000", 2024-03-11T21:29:58.698 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 6.6749999999999996e-05 2024-03-11T21:29:58.698 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:29:58.698 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:29:58.698 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "started", 2024-03-11T21:29:58.698 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:29:58.696974+0000", 2024-03-11T21:29:58.698 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.000121483 2024-03-11T21:29:58.698 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:29:58.698 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:29:58.698 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "waiting for subops from 1,3", 2024-03-11T21:29:58.698 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:29:58.697087+0000", 2024-03-11T21:29:58.698 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.000112921 2024-03-11T21:29:58.698 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:29:58.698 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:29:58.698 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "op_commit", 2024-03-11T21:29:58.699 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:29:58.699540+0000", 2024-03-11T21:29:58.699 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.002453705 2024-03-11T21:29:58.699 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-03-11T21:29:58.699 INFO:teuthology.orchestra.run.smithi097.stdout: ] 2024-03-11T21:29:58.699 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-03-11T21:29:58.699 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-03-11T21:29:58.699 INFO:teuthology.orchestra.run.smithi097.stdout: ], 2024-03-11T21:29:58.699 INFO:teuthology.orchestra.run.smithi097.stdout: "num_ops": 1 2024-03-11T21:29:58.699 INFO:teuthology.orchestra.run.smithi097.stdout:} 2024-03-11T21:29:58.722 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:29:58.755 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:29:58.756 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:29:58.756 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:29:58.857 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:29:58.901 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:29:58.902 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:29:58.902 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:29:59.003 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:29:59.032 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:29:59.033 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:29:59.033 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:29:59.134 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:29:59.181 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:29:59.182 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:29:59.182 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:29:59.283 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:29:59.322 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:29:59.323 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:29:59.323 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:29:59.326 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:29:59 smithi097 ceph-mon[33317]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-11T21:29:59.326 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:29:59 smithi097 ceph-mon[33317]: from='client.24856 ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]': finished 2024-03-11T21:29:59.326 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:29:59 smithi097 ceph-mon[33317]: from='client.24862 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-03-11T21:29:59.326 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:29:59 smithi097 ceph-mon[33317]: osdmap e127: 8 total, 8 up, 8 in 2024-03-11T21:29:59.360 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:29:59 smithi092 ceph-mon[29572]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-11T21:29:59.361 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:29:59 smithi092 ceph-mon[29572]: from='client.24856 ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]': finished 2024-03-11T21:29:59.361 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:29:59 smithi092 ceph-mon[29572]: from='client.24862 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-03-11T21:29:59.361 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:29:59 smithi092 ceph-mon[29572]: osdmap e127: 8 total, 8 up, 8 in 2024-03-11T21:29:59.361 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:29:59 smithi092 ceph-mon[37563]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-11T21:29:59.361 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:29:59 smithi092 ceph-mon[37563]: from='client.24856 ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]': finished 2024-03-11T21:29:59.361 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:29:59 smithi092 ceph-mon[37563]: from='client.24862 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-03-11T21:29:59.361 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:29:59 smithi092 ceph-mon[37563]: osdmap e127: 8 total, 8 up, 8 in 2024-03-11T21:29:59.425 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:29:59.455 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:29:59.455 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:29:59.456 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:29:59.556 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:29:59.587 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:29:59.587 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:29:59.588 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:29:59.638 DEBUG:teuthology.orchestra.run.smithi097:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid 00db0016-dfed-11ee-95c8-87774f69a715 -- ceph daemon osd.5 dump_blocked_ops 2024-03-11T21:29:59.689 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:29:59.725 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:29:59.726 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:29:59.726 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:29:59.827 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:29:59.867 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:29:59.868 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:29:59.869 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:29:59.969 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:29:59.979 INFO:teuthology.orchestra.run.smithi097.stderr:Inferring config /var/lib/ceph/00db0016-dfed-11ee-95c8-87774f69a715/mon.b/config 2024-03-11T21:30:00.014 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:30:00.015 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:30:00.015 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:30:00.116 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:30:00.159 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:30:00.160 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:30:00.160 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:30:00.261 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:30:00.296 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:30:00.296 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:30:00.296 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:30:00.371 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:30:00 smithi092 ceph-mon[29572]: pgmap v152: 148 pgs: 146 active+clean, 2 active+clean+scrubbing+deep; 5.3 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 818 op/s 2024-03-11T21:30:00.371 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:30:00 smithi092 ceph-mon[29572]: Health detail: HEALTH_WARN noscrub,nodeep-scrub flag(s) set 2024-03-11T21:30:00.372 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:30:00 smithi092 ceph-mon[29572]: [WRN] OSDMAP_FLAGS: noscrub,nodeep-scrub flag(s) set 2024-03-11T21:30:00.372 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:30:00 smithi092 ceph-mon[37563]: pgmap v152: 148 pgs: 146 active+clean, 2 active+clean+scrubbing+deep; 5.3 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 818 op/s 2024-03-11T21:30:00.372 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:30:00 smithi092 ceph-mon[37563]: Health detail: HEALTH_WARN noscrub,nodeep-scrub flag(s) set 2024-03-11T21:30:00.372 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:30:00 smithi092 ceph-mon[37563]: [WRN] OSDMAP_FLAGS: noscrub,nodeep-scrub flag(s) set 2024-03-11T21:30:00.398 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:30:00.402 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:30:00 smithi097 ceph-mon[33317]: pgmap v152: 148 pgs: 146 active+clean, 2 active+clean+scrubbing+deep; 5.3 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 818 op/s 2024-03-11T21:30:00.402 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:30:00 smithi097 ceph-mon[33317]: Health detail: HEALTH_WARN noscrub,nodeep-scrub flag(s) set 2024-03-11T21:30:00.402 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:30:00 smithi097 ceph-mon[33317]: [WRN] OSDMAP_FLAGS: noscrub,nodeep-scrub flag(s) set 2024-03-11T21:30:00.437 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:30:00.438 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:30:00.438 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:30:00.539 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:30:00.568 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:30:00.569 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:30:00.569 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:30:00.670 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:30:00.717 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:30:00.718 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:30:00.718 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:30:00.819 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:30:00.822 INFO:teuthology.orchestra.run.smithi092.stdout:osd.0: {} 2024-03-11T21:30:00.848 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:30:00.849 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:30:00.850 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:30:00.867 INFO:teuthology.orchestra.run.smithi092.stdout:osd.1: {} 2024-03-11T21:30:00.911 INFO:teuthology.orchestra.run.smithi092.stdout:osd.2: {} 2024-03-11T21:30:00.950 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:30:00.952 INFO:teuthology.orchestra.run.smithi092.stdout:osd.3: {} 2024-03-11T21:30:00.981 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:30:00.981 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:30:00.982 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:30:00.991 INFO:teuthology.orchestra.run.smithi092.stdout:osd.4: {} 2024-03-11T21:30:01.033 INFO:teuthology.orchestra.run.smithi092.stdout:osd.5: {} 2024-03-11T21:30:01.069 INFO:teuthology.orchestra.run.smithi092.stdout:osd.6: {} 2024-03-11T21:30:01.083 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:30:01.114 INFO:teuthology.orchestra.run.smithi092.stdout:osd.7: {} 2024-03-11T21:30:01.126 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:30:01.127 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:30:01.127 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:30:01.127 INFO:teuthology.orchestra.run.smithi092.stderr:osd.0: osd_enable_op_tracker = '' 2024-03-11T21:30:01.127 INFO:teuthology.orchestra.run.smithi092.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-11T21:30:01.128 INFO:teuthology.orchestra.run.smithi092.stderr:osd.2: osd_enable_op_tracker = '' 2024-03-11T21:30:01.128 INFO:teuthology.orchestra.run.smithi092.stderr:osd.3: osd_enable_op_tracker = '' 2024-03-11T21:30:01.128 INFO:teuthology.orchestra.run.smithi092.stderr:osd.4: osd_enable_op_tracker = '' 2024-03-11T21:30:01.128 INFO:teuthology.orchestra.run.smithi092.stderr:osd.5: osd_enable_op_tracker = '' 2024-03-11T21:30:01.128 INFO:teuthology.orchestra.run.smithi092.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-11T21:30:01.128 INFO:teuthology.orchestra.run.smithi092.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-11T21:30:01.228 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:30:01.242 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:30:01 smithi092 ceph-mon[29572]: from='client.? v1:172.21.15.92:0/2712822254' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-11T21:30:01.244 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:30:01 smithi092 ceph-mon[37563]: from='client.? v1:172.21.15.92:0/2712822254' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-11T21:30:01.267 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:30:01.268 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:30:01.268 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:30:01.369 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:30:01.396 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:30:01 smithi097 ceph-mon[33317]: from='client.? v1:172.21.15.92:0/2712822254' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-11T21:30:01.398 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:30:01.398 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:30:01.399 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:30:01.499 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:30:01.530 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:30:01.531 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:30:01.531 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:30:01.589 DEBUG:teuthology.orchestra.run.smithi092:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid 00db0016-dfed-11ee-95c8-87774f69a715 -- ceph osd reweight 1 1 2024-03-11T21:30:01.632 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:30:01.684 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:30:01.685 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:30:01.685 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:30:01.786 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:30:01.815 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:30:01.816 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:30:01.817 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:30:01.917 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:30:01.954 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:30:01.955 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:30:01.955 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:30:02.056 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:30:02.086 INFO:teuthology.orchestra.run.smithi097.stdout:ERROR: (22) Invalid argument 2024-03-11T21:30:02.086 INFO:teuthology.orchestra.run.smithi097.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-11T21:30:02.098 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:30:02.098 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:30:02.098 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:30:02.182 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:30:02 smithi092 ceph-mon[29572]: pgmap v153: 148 pgs: 146 active+clean, 2 active+clean+scrubbing+deep; 5.3 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 658 op/s 2024-03-11T21:30:02.182 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:30:02 smithi092 ceph-mon[37563]: pgmap v153: 148 pgs: 146 active+clean, 2 active+clean+scrubbing+deep; 5.3 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 658 op/s 2024-03-11T21:30:02.200 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:30:02.230 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:30:02.230 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:30:02.231 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:30:02.327 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:30:02 smithi097 ceph-mon[33317]: pgmap v153: 148 pgs: 146 active+clean, 2 active+clean+scrubbing+deep; 5.3 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 658 op/s 2024-03-11T21:30:02.327 INFO:journalctl@ceph.iscsi.iscsi.a.smithi097.stdout:Mar 11 21:30:02 smithi097 ceph-00db0016-dfed-11ee-95c8-87774f69a715-iscsi-iscsi-a[60943]: debug there is no tcmu-runner data available 2024-03-11T21:30:02.331 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:30:02.355 INFO:teuthology.orchestra.run.smithi092.stderr:Inferring config /var/lib/ceph/00db0016-dfed-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T21:30:02.360 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:30:02.361 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:30:02.361 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:30:02.462 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:30:02.495 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:30:02.495 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:30:02.496 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:30:02.597 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:30:02.638 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:30:02.638 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:30:02.639 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:30:02.740 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:30:02.772 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:30:02.773 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:30:02.773 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:30:02.874 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:30:02.919 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:30:02.919 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:30:02.920 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:30:02.924 DEBUG:teuthology.orchestra.run.smithi097:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid 00db0016-dfed-11ee-95c8-87774f69a715 -- ceph daemon osd.5 dump_historic_ops 2024-03-11T21:30:03.020 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:30:03.058 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:30:03.059 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:30:03.059 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:30:03.160 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:30:03.195 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:30:03.196 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:30:03.196 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:30:03.245 DEBUG:teuthology.orchestra.run.smithi092:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid 00db0016-dfed-11ee-95c8-87774f69a715 -- ceph osd unset noscrub 2024-03-11T21:30:03.260 INFO:teuthology.orchestra.run.smithi097.stderr:Inferring config /var/lib/ceph/00db0016-dfed-11ee-95c8-87774f69a715/mon.b/config 2024-03-11T21:30:03.297 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:30:03.342 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:30:03.342 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:30:03.342 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:30:03.440 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:30:03 smithi097 ceph-mon[33317]: purged_snaps scrub starts 2024-03-11T21:30:03.440 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:30:03 smithi097 ceph-mon[33317]: purged_snaps scrub ok 2024-03-11T21:30:03.440 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:30:03 smithi097 ceph-mon[33317]: osdmap e128: 8 total, 8 up, 8 in 2024-03-11T21:30:03.440 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:30:03 smithi097 ceph-mon[33317]: from='client.24469 v1:172.21.15.97:0/404208310' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:30:03.443 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:30:03.488 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:30:03.488 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:30:03.489 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:30:03.541 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:30:03 smithi092 ceph-mon[29572]: purged_snaps scrub starts 2024-03-11T21:30:03.541 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:30:03 smithi092 ceph-mon[29572]: purged_snaps scrub ok 2024-03-11T21:30:03.541 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:30:03 smithi092 ceph-mon[29572]: osdmap e128: 8 total, 8 up, 8 in 2024-03-11T21:30:03.541 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:30:03 smithi092 ceph-mon[29572]: from='client.24469 v1:172.21.15.97:0/404208310' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:30:03.542 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:30:03 smithi092 ceph-mon[37563]: purged_snaps scrub starts 2024-03-11T21:30:03.542 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:30:03 smithi092 ceph-mon[37563]: purged_snaps scrub ok 2024-03-11T21:30:03.542 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:30:03 smithi092 ceph-mon[37563]: osdmap e128: 8 total, 8 up, 8 in 2024-03-11T21:30:03.542 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:30:03 smithi092 ceph-mon[37563]: from='client.24469 v1:172.21.15.97:0/404208310' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:30:03.589 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:30:03.622 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:30:03.623 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:30:03.623 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:30:03.724 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:30:03.777 INFO:teuthology.orchestra.run.smithi092.stderr:Inferring config /var/lib/ceph/00db0016-dfed-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T21:30:03.785 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:30:03.786 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:30:03.786 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:30:03.887 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:30:03.937 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:30:03.938 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:30:03.938 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:30:04.039 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:30:04.066 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:30:04.067 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:30:04.067 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:30:04.168 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:30:04.199 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:30:04.200 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:30:04.200 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:30:04.226 DEBUG:teuthology.orchestra.run.smithi092:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid 00db0016-dfed-11ee-95c8-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-03-11T21:30:04.301 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:30:04.359 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:30:04.359 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:30:04.360 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:30:04.416 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:30:04 smithi092 ceph-mon[29572]: pgmap v155: 148 pgs: 148 active+clean; 5.0 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 1.1 KiB/s rd, 0 B/s wr, 1.02k op/s 2024-03-11T21:30:04.417 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:30:04 smithi092 ceph-mon[37563]: pgmap v155: 148 pgs: 148 active+clean; 5.0 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 1.1 KiB/s rd, 0 B/s wr, 1.02k op/s 2024-03-11T21:30:04.444 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:30:04 smithi097 ceph-mon[33317]: pgmap v155: 148 pgs: 148 active+clean; 5.0 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 1.1 KiB/s rd, 0 B/s wr, 1.02k op/s 2024-03-11T21:30:04.460 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:30:04.487 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:30:04.488 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:30:04.488 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:30:04.589 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:30:04.618 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:30:04.618 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:30:04.618 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:30:04.719 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:30:04.760 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:30:04.761 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:30:04.761 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:30:04.862 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:30:04.891 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:30:04.891 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:30:04.892 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:30:04.993 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:30:05.050 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:30:05.050 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:30:05.051 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:30:05.152 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:30:05.170 INFO:teuthology.orchestra.run.smithi092.stderr:reweighted osd.1 to 1 (10000) 2024-03-11T21:30:05.187 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:30:05.187 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:30:05.188 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:30:05.289 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:30:05.327 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:30:05.327 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:30:05.327 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:30:05.364 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:30:05 smithi097 ceph-mon[33317]: from='client.? v1:172.21.15.92:0/2324504069' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]: dispatch 2024-03-11T21:30:05.428 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:30:05.467 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:30:05.467 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:30:05.468 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:30:05.525 INFO:teuthology.orchestra.run.smithi097.stdout:ERROR: (22) Invalid argument 2024-03-11T21:30:05.525 INFO:teuthology.orchestra.run.smithi097.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-11T21:30:05.569 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:30:05.597 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:30:05.597 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:30:05.598 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:30:05.610 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:30:05 smithi092 ceph-mon[29572]: from='client.? v1:172.21.15.92:0/2324504069' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]: dispatch 2024-03-11T21:30:05.611 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:30:05 smithi092 ceph-mon[37563]: from='client.? v1:172.21.15.92:0/2324504069' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]: dispatch 2024-03-11T21:30:05.698 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:30:05.735 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:30:05.736 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:30:05.736 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:30:05.837 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:30:05.880 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:30:05.881 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:30:05.881 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:30:05.982 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:30:06.012 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:30:06.013 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:30:06.013 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:30:06.114 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:30:06.161 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:30:06.161 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:30:06.161 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:30:06.262 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:30:06.309 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:30:06.309 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:30:06.309 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:30:06.326 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:30:06 smithi097 ceph-mon[33317]: pgmap v156: 148 pgs: 148 active+clean; 5.0 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 1.1 KiB/s rd, 0 B/s wr, 855 op/s 2024-03-11T21:30:06.327 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:30:06 smithi097 ceph-mon[33317]: from='client.? v1:172.21.15.92:0/2324504069' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]': finished 2024-03-11T21:30:06.327 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:30:06 smithi097 ceph-mon[33317]: osdmap e129: 8 total, 8 up, 8 in 2024-03-11T21:30:06.374 DEBUG:teuthology.orchestra.run.smithi097:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid 00db0016-dfed-11ee-95c8-87774f69a715 -- ceph daemon osd.6 dump_ops_in_flight 2024-03-11T21:30:06.410 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:30:06.451 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:30:06.451 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:30:06.451 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:30:06.488 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:30:06 smithi092 ceph-mon[29572]: pgmap v156: 148 pgs: 148 active+clean; 5.0 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 1.1 KiB/s rd, 0 B/s wr, 855 op/s 2024-03-11T21:30:06.488 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:30:06 smithi092 ceph-mon[29572]: from='client.? v1:172.21.15.92:0/2324504069' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]': finished 2024-03-11T21:30:06.488 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:30:06 smithi092 ceph-mon[29572]: osdmap e129: 8 total, 8 up, 8 in 2024-03-11T21:30:06.489 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:30:06 smithi092 ceph-mon[37563]: pgmap v156: 148 pgs: 148 active+clean; 5.0 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 1.1 KiB/s rd, 0 B/s wr, 855 op/s 2024-03-11T21:30:06.489 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:30:06 smithi092 ceph-mon[37563]: from='client.? v1:172.21.15.92:0/2324504069' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]': finished 2024-03-11T21:30:06.489 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:30:06 smithi092 ceph-mon[37563]: osdmap e129: 8 total, 8 up, 8 in 2024-03-11T21:30:06.535 DEBUG:teuthology.orchestra.run.smithi092:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid 00db0016-dfed-11ee-95c8-87774f69a715 -- ceph osd reweight 2 1 2024-03-11T21:30:06.553 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:30:06.629 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:30:06.629 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:30:06.630 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:30:06.702 INFO:teuthology.orchestra.run.smithi097.stderr:Inferring config /var/lib/ceph/00db0016-dfed-11ee-95c8-87774f69a715/mon.b/config 2024-03-11T21:30:06.731 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:30:06.760 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:30:06.761 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:30:06.761 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:30:06.777 INFO:teuthology.orchestra.run.smithi092.stderr:Inferring config /var/lib/ceph/00db0016-dfed-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T21:30:06.862 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:30:06.907 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:30:06.908 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:30:06.908 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:30:07.009 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:30:07.047 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:30:07.048 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:30:07.048 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:30:07.149 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:30:07.189 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:30:07.190 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:30:07.190 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:30:07.291 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:30:07.304 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:30:07 smithi092 ceph-mon[29572]: from='mgr.14607 v1:172.21.15.92:0/1702717597' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T21:30:07.304 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:30:07 smithi092 ceph-mon[37563]: from='mgr.14607 v1:172.21.15.92:0/1702717597' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T21:30:07.341 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:30:07.341 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:30:07.342 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:30:07.442 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:30:07.491 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:30:07.492 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:30:07.492 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:30:07.576 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:30:07 smithi097 ceph-mon[33317]: from='mgr.14607 v1:172.21.15.92:0/1702717597' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T21:30:07.593 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:30:07.642 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:30:07.643 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:30:07.643 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:30:07.744 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:30:07.778 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:30:07.778 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:30:07.779 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:30:07.882 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:30:07.936 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:30:07.936 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:30:07.937 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:30:08.038 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:30:08.077 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:30:08.078 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:30:08.078 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:30:08.179 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:30:08.206 INFO:journalctl@ceph.mgr.y.smithi092.stdout:Mar 11 21:30:07 smithi092 ceph-00db0016-dfed-11ee-95c8-87774f69a715-mgr-y[29801]: ::ffff:172.21.15.97 - - [11/Mar/2024:21:30:07] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-11T21:30:08.209 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:30:08.209 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:30:08.210 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:30:08.218 INFO:teuthology.orchestra.run.smithi092.stderr:noscrub is unset 2024-03-11T21:30:08.311 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:30:08.345 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:30:08.345 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:30:08.346 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:30:08.428 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:30:08 smithi097 ceph-mon[33317]: pgmap v158: 148 pgs: 148 active+clean; 4.9 GiB data, 16 GiB used, 700 GiB / 715 GiB avail; 511 B/s rd, 0 B/s wr, 829 op/s 2024-03-11T21:30:08.428 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:30:08 smithi097 ceph-mon[33317]: from='client.? v1:172.21.15.92:0/2337755897' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-11T21:30:08.428 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:30:08 smithi097 ceph-mon[33317]: from='client.34850 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-11T21:30:08.446 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:30:08.479 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:30:08.480 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:30:08.480 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:30:08.581 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:30:08.599 INFO:teuthology.orchestra.run.smithi097.stdout:ERROR: (22) Invalid argument 2024-03-11T21:30:08.599 INFO:teuthology.orchestra.run.smithi097.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-11T21:30:08.611 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:30:08 smithi092 ceph-mon[29572]: pgmap v158: 148 pgs: 148 active+clean; 4.9 GiB data, 16 GiB used, 700 GiB / 715 GiB avail; 511 B/s rd, 0 B/s wr, 829 op/s 2024-03-11T21:30:08.611 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:30:08 smithi092 ceph-mon[29572]: from='client.? v1:172.21.15.92:0/2337755897' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-11T21:30:08.611 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:30:08 smithi092 ceph-mon[29572]: from='client.34850 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-11T21:30:08.612 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:30:08 smithi092 ceph-mon[37563]: pgmap v158: 148 pgs: 148 active+clean; 4.9 GiB data, 16 GiB used, 700 GiB / 715 GiB avail; 511 B/s rd, 0 B/s wr, 829 op/s 2024-03-11T21:30:08.612 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:30:08 smithi092 ceph-mon[37563]: from='client.? v1:172.21.15.92:0/2337755897' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-11T21:30:08.612 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:30:08 smithi092 ceph-mon[37563]: from='client.34850 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-11T21:30:08.621 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:30:08.622 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:30:08.622 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:30:08.723 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:30:08.766 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:30:08.767 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:30:08.767 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:30:08.868 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:30:08.899 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:30:08.899 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:30:08.899 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:30:09.000 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:30:09.037 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:30:09.037 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:30:09.037 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:30:09.138 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:30:09.179 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:30:09.180 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:30:09.180 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:30:09.281 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:30:09.309 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:30:09 smithi097 ceph-mon[33317]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-11T21:30:09.309 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:30:09 smithi097 ceph-mon[33317]: from='client.34850 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-03-11T21:30:09.310 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:30:09 smithi097 ceph-mon[33317]: osdmap e130: 8 total, 8 up, 8 in 2024-03-11T21:30:09.312 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:30:09.313 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:30:09.314 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:30:09.348 DEBUG:teuthology.orchestra.run.smithi097:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid 00db0016-dfed-11ee-95c8-87774f69a715 -- ceph daemon osd.6 dump_blocked_ops 2024-03-11T21:30:09.415 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:30:09.465 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:30:09.466 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:30:09.466 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:30:09.567 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:30:09.604 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:30:09 smithi092 ceph-mon[29572]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-11T21:30:09.604 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:30:09 smithi092 ceph-mon[29572]: from='client.34850 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-03-11T21:30:09.604 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:30:09 smithi092 ceph-mon[29572]: osdmap e130: 8 total, 8 up, 8 in 2024-03-11T21:30:09.605 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:30:09 smithi092 ceph-mon[37563]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-11T21:30:09.605 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:30:09 smithi092 ceph-mon[37563]: from='client.34850 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-03-11T21:30:09.605 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:30:09 smithi092 ceph-mon[37563]: osdmap e130: 8 total, 8 up, 8 in 2024-03-11T21:30:09.610 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:30:09.611 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:30:09.611 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:30:09.712 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:30:09.735 INFO:teuthology.orchestra.run.smithi097.stderr:Inferring config /var/lib/ceph/00db0016-dfed-11ee-95c8-87774f69a715/mon.b/config 2024-03-11T21:30:09.746 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:30:09.747 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:30:09.747 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:30:09.848 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:30:09.871 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:30:09.871 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:30:09.872 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:30:09.972 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:30:10.014 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:30:10.014 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:30:10.015 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:30:10.115 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:30:10.149 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:30:10.149 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:30:10.149 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:30:10.250 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:30:10.286 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:30:10.286 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:30:10.287 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:30:10.372 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:30:10 smithi097 ceph-mon[33317]: 5.1 scrub starts 2024-03-11T21:30:10.372 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:30:10 smithi097 ceph-mon[33317]: 5.1 scrub ok 2024-03-11T21:30:10.372 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:30:10 smithi097 ceph-mon[33317]: pgmap v160: 148 pgs: 148 active+clean; 4.7 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 743 B/s rd, 0 B/s wr, 1.06k op/s 2024-03-11T21:30:10.388 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:30:10.428 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:30:10.428 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:30:10.429 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:30:10.530 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:30:10.562 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:30:10.563 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:30:10.564 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:30:10.610 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:30:10 smithi092 ceph-mon[29572]: 5.1 scrub starts 2024-03-11T21:30:10.611 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:30:10 smithi092 ceph-mon[29572]: 5.1 scrub ok 2024-03-11T21:30:10.611 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:30:10 smithi092 ceph-mon[29572]: pgmap v160: 148 pgs: 148 active+clean; 4.7 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 743 B/s rd, 0 B/s wr, 1.06k op/s 2024-03-11T21:30:10.611 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:30:10 smithi092 ceph-mon[37563]: 5.1 scrub starts 2024-03-11T21:30:10.611 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:30:10 smithi092 ceph-mon[37563]: 5.1 scrub ok 2024-03-11T21:30:10.611 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:30:10 smithi092 ceph-mon[37563]: pgmap v160: 148 pgs: 148 active+clean; 4.7 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 743 B/s rd, 0 B/s wr, 1.06k op/s 2024-03-11T21:30:10.665 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:30:10.696 INFO:teuthology.orchestra.run.smithi092.stderr:Inferring config /var/lib/ceph/00db0016-dfed-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T21:30:10.715 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:30:10.716 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:30:10.716 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:30:10.817 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:30:10.861 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:30:10.861 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:30:10.861 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:30:10.962 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:30:10.996 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:30:10.996 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:30:10.996 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:30:11.098 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:30:11.128 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:30:11.128 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:30:11.129 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:30:11.229 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:30:11.270 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:30:11.270 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:30:11.271 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:30:11.372 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:30:11.405 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:30:11.405 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:30:11.406 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:30:11.506 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:30:11.536 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:30:11.536 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:30:11.537 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:30:11.637 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:30:11.675 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:30:11.676 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:30:11.676 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:30:11.777 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:30:11.817 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:30:11.817 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:30:11.817 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:30:11.906 INFO:teuthology.orchestra.run.smithi097.stdout:ERROR: (22) Invalid argument 2024-03-11T21:30:11.907 INFO:teuthology.orchestra.run.smithi097.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-11T21:30:11.918 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:30:11.958 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:30:11.959 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:30:11.959 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:30:12.060 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:30:12.098 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:30:12.099 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:30:12.099 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:30:12.200 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:30:12.253 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:30:12.254 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:30:12.254 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:30:12.355 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:30:12.376 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:30:12 smithi092 ceph-mon[29572]: 4.a scrub starts 2024-03-11T21:30:12.376 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:30:12 smithi092 ceph-mon[29572]: 4.a scrub ok 2024-03-11T21:30:12.376 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:30:12 smithi092 ceph-mon[29572]: pgmap v161: 148 pgs: 148 active+clean; 4.7 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 639 B/s rd, 0 B/s wr, 651 op/s 2024-03-11T21:30:12.376 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:30:12 smithi092 ceph-mon[29572]: 7.f scrub starts 2024-03-11T21:30:12.376 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:30:12 smithi092 ceph-mon[37563]: 4.a scrub starts 2024-03-11T21:30:12.376 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:30:12 smithi092 ceph-mon[37563]: 4.a scrub ok 2024-03-11T21:30:12.376 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:30:12 smithi092 ceph-mon[37563]: pgmap v161: 148 pgs: 148 active+clean; 4.7 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 639 B/s rd, 0 B/s wr, 651 op/s 2024-03-11T21:30:12.377 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:30:12 smithi092 ceph-mon[37563]: 7.f scrub starts 2024-03-11T21:30:12.399 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:30:12.400 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:30:12.400 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:30:12.498 INFO:teuthology.orchestra.run.smithi092.stdout:osd.0: {} 2024-03-11T21:30:12.501 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:30:12.540 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:30:12.541 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:30:12.541 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:30:12.559 INFO:teuthology.orchestra.run.smithi092.stdout:osd.1: {} 2024-03-11T21:30:12.576 INFO:journalctl@ceph.iscsi.iscsi.a.smithi097.stdout:Mar 11 21:30:12 smithi097 ceph-00db0016-dfed-11ee-95c8-87774f69a715-iscsi-iscsi-a[60943]: debug there is no tcmu-runner data available 2024-03-11T21:30:12.576 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:30:12 smithi097 ceph-mon[33317]: 4.a scrub starts 2024-03-11T21:30:12.577 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:30:12 smithi097 ceph-mon[33317]: 4.a scrub ok 2024-03-11T21:30:12.577 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:30:12 smithi097 ceph-mon[33317]: pgmap v161: 148 pgs: 148 active+clean; 4.7 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 639 B/s rd, 0 B/s wr, 651 op/s 2024-03-11T21:30:12.577 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:30:12 smithi097 ceph-mon[33317]: 7.f scrub starts 2024-03-11T21:30:12.605 INFO:teuthology.orchestra.run.smithi092.stdout:osd.2: {} 2024-03-11T21:30:12.642 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:30:12.649 INFO:teuthology.orchestra.run.smithi092.stdout:osd.3: {} 2024-03-11T21:30:12.672 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:30:12.673 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:30:12.673 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:30:12.686 INFO:teuthology.orchestra.run.smithi092.stdout:osd.4: {} 2024-03-11T21:30:12.710 DEBUG:teuthology.orchestra.run.smithi092:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid 00db0016-dfed-11ee-95c8-87774f69a715 -- ceph osd unset nodeep-scrub 2024-03-11T21:30:12.723 INFO:teuthology.orchestra.run.smithi092.stdout:osd.5: {} 2024-03-11T21:30:12.762 INFO:teuthology.orchestra.run.smithi092.stdout:osd.6: {} 2024-03-11T21:30:12.774 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:30:12.802 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:30:12.803 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:30:12.803 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:30:12.805 INFO:teuthology.orchestra.run.smithi092.stdout:osd.7: {} 2024-03-11T21:30:12.810 INFO:teuthology.orchestra.run.smithi092.stderr:osd.0: osd_enable_op_tracker = '' 2024-03-11T21:30:12.810 INFO:teuthology.orchestra.run.smithi092.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-11T21:30:12.810 INFO:teuthology.orchestra.run.smithi092.stderr:osd.2: osd_enable_op_tracker = '' 2024-03-11T21:30:12.810 INFO:teuthology.orchestra.run.smithi092.stderr:osd.3: osd_enable_op_tracker = '' 2024-03-11T21:30:12.811 INFO:teuthology.orchestra.run.smithi092.stderr:osd.4: osd_enable_op_tracker = '' 2024-03-11T21:30:12.811 INFO:teuthology.orchestra.run.smithi092.stderr:osd.5: osd_enable_op_tracker = '' 2024-03-11T21:30:12.811 INFO:teuthology.orchestra.run.smithi092.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-11T21:30:12.811 INFO:teuthology.orchestra.run.smithi092.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-11T21:30:12.904 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:30:12.930 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:30:12.931 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:30:12.931 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:30:13.032 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:30:13.060 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:30:13.060 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:30:13.061 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:30:13.161 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:30:13.220 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:30:13.221 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:30:13.221 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:30:13.286 DEBUG:teuthology.orchestra.run.smithi097:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid 00db0016-dfed-11ee-95c8-87774f69a715 -- ceph daemon osd.6 dump_historic_ops 2024-03-11T21:30:13.322 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:30:13.362 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:30:13.362 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:30:13.362 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:30:13.463 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:30:13.494 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:30:13.494 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:30:13.495 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:30:13.496 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:30:13 smithi092 ceph-mon[29572]: 6.1e scrub starts 2024-03-11T21:30:13.496 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:30:13 smithi092 ceph-mon[29572]: 6.1e scrub ok 2024-03-11T21:30:13.496 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:30:13 smithi092 ceph-mon[29572]: 2.1c scrub starts 2024-03-11T21:30:13.496 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:30:13 smithi092 ceph-mon[29572]: 2.1c scrub ok 2024-03-11T21:30:13.496 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:30:13 smithi092 ceph-mon[29572]: 4.7 scrub starts 2024-03-11T21:30:13.496 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:30:13 smithi092 ceph-mon[29572]: 4.7 scrub ok 2024-03-11T21:30:13.496 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:30:13 smithi092 ceph-mon[29572]: 2.1a scrub starts 2024-03-11T21:30:13.496 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:30:13 smithi092 ceph-mon[29572]: 2.1a scrub ok 2024-03-11T21:30:13.496 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:30:13 smithi092 ceph-mon[29572]: 6.d scrub starts 2024-03-11T21:30:13.496 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:30:13 smithi092 ceph-mon[29572]: 6.d scrub ok 2024-03-11T21:30:13.496 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:30:13 smithi092 ceph-mon[29572]: from='client.24469 v1:172.21.15.97:0/404208310' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:30:13.496 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:30:13 smithi092 ceph-mon[29572]: from='client.? v1:172.21.15.92:0/1132190549' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-11T21:30:13.496 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:30:13 smithi092 ceph-mon[29572]: purged_snaps scrub starts 2024-03-11T21:30:13.497 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:30:13 smithi092 ceph-mon[29572]: purged_snaps scrub ok 2024-03-11T21:30:13.497 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:30:13 smithi092 ceph-mon[29572]: 7.9 scrub starts 2024-03-11T21:30:13.497 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:30:13 smithi092 ceph-mon[37563]: 6.1e scrub starts 2024-03-11T21:30:13.497 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:30:13 smithi092 ceph-mon[37563]: 6.1e scrub ok 2024-03-11T21:30:13.497 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:30:13 smithi092 ceph-mon[37563]: 2.1c scrub starts 2024-03-11T21:30:13.497 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:30:13 smithi092 ceph-mon[37563]: 2.1c scrub ok 2024-03-11T21:30:13.497 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:30:13 smithi092 ceph-mon[37563]: 4.7 scrub starts 2024-03-11T21:30:13.497 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:30:13 smithi092 ceph-mon[37563]: 4.7 scrub ok 2024-03-11T21:30:13.497 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:30:13 smithi092 ceph-mon[37563]: 2.1a scrub starts 2024-03-11T21:30:13.497 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:30:13 smithi092 ceph-mon[37563]: 2.1a scrub ok 2024-03-11T21:30:13.497 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:30:13 smithi092 ceph-mon[37563]: 6.d scrub starts 2024-03-11T21:30:13.497 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:30:13 smithi092 ceph-mon[37563]: 6.d scrub ok 2024-03-11T21:30:13.498 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:30:13 smithi092 ceph-mon[37563]: from='client.24469 v1:172.21.15.97:0/404208310' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:30:13.498 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:30:13 smithi092 ceph-mon[37563]: from='client.? v1:172.21.15.92:0/1132190549' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-11T21:30:13.498 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:30:13 smithi092 ceph-mon[37563]: purged_snaps scrub starts 2024-03-11T21:30:13.498 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:30:13 smithi092 ceph-mon[37563]: purged_snaps scrub ok 2024-03-11T21:30:13.498 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:30:13 smithi092 ceph-mon[37563]: 7.9 scrub starts 2024-03-11T21:30:13.578 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:30:13 smithi097 ceph-mon[33317]: 6.1e scrub starts 2024-03-11T21:30:13.578 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:30:13 smithi097 ceph-mon[33317]: 6.1e scrub ok 2024-03-11T21:30:13.578 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:30:13 smithi097 ceph-mon[33317]: 2.1c scrub starts 2024-03-11T21:30:13.578 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:30:13 smithi097 ceph-mon[33317]: 2.1c scrub ok 2024-03-11T21:30:13.578 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:30:13 smithi097 ceph-mon[33317]: 4.7 scrub starts 2024-03-11T21:30:13.578 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:30:13 smithi097 ceph-mon[33317]: 4.7 scrub ok 2024-03-11T21:30:13.579 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:30:13 smithi097 ceph-mon[33317]: 2.1a scrub starts 2024-03-11T21:30:13.579 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:30:13 smithi097 ceph-mon[33317]: 2.1a scrub ok 2024-03-11T21:30:13.579 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:30:13 smithi097 ceph-mon[33317]: 6.d scrub starts 2024-03-11T21:30:13.579 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:30:13 smithi097 ceph-mon[33317]: 6.d scrub ok 2024-03-11T21:30:13.579 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:30:13 smithi097 ceph-mon[33317]: from='client.24469 v1:172.21.15.97:0/404208310' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:30:13.579 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:30:13 smithi097 ceph-mon[33317]: from='client.? v1:172.21.15.92:0/1132190549' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-11T21:30:13.579 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:30:13 smithi097 ceph-mon[33317]: purged_snaps scrub starts 2024-03-11T21:30:13.579 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:30:13 smithi097 ceph-mon[33317]: purged_snaps scrub ok 2024-03-11T21:30:13.579 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:30:13 smithi097 ceph-mon[33317]: 7.9 scrub starts 2024-03-11T21:30:13.596 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:30:13.613 INFO:teuthology.orchestra.run.smithi097.stderr:Inferring config /var/lib/ceph/00db0016-dfed-11ee-95c8-87774f69a715/mon.b/config 2024-03-11T21:30:13.629 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:30:13.629 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:30:13.629 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:30:13.730 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:30:13.758 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:30:13.758 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:30:13.759 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:30:13.859 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:30:13.885 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:30:13.886 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:30:13.887 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:30:13.987 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:30:14.021 INFO:teuthology.orchestra.run.smithi092.stderr:Inferring config /var/lib/ceph/00db0016-dfed-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T21:30:14.053 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:30:14.054 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:30:14.054 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:30:14.155 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:30:14.184 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:30:14.185 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:30:14.186 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:30:14.287 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:30:14.326 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:30:14 smithi097 ceph-mon[33317]: 2.5 scrub starts 2024-03-11T21:30:14.327 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:30:14 smithi097 ceph-mon[33317]: 2.5 scrub ok 2024-03-11T21:30:14.327 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:30:14 smithi097 ceph-mon[33317]: 2.17 scrub starts 2024-03-11T21:30:14.327 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:30:14 smithi097 ceph-mon[33317]: 2.17 scrub ok 2024-03-11T21:30:14.327 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:30:14 smithi097 ceph-mon[33317]: 7.e scrub starts 2024-03-11T21:30:14.327 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:30:14 smithi097 ceph-mon[33317]: purged_snaps scrub starts 2024-03-11T21:30:14.327 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:30:14 smithi097 ceph-mon[33317]: purged_snaps scrub ok 2024-03-11T21:30:14.327 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:30:14 smithi097 ceph-mon[33317]: 2.12 scrub starts 2024-03-11T21:30:14.327 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:30:14 smithi097 ceph-mon[33317]: 2.12 scrub ok 2024-03-11T21:30:14.327 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:30:14 smithi097 ceph-mon[33317]: purged_snaps scrub starts 2024-03-11T21:30:14.327 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:30:14 smithi097 ceph-mon[33317]: purged_snaps scrub ok 2024-03-11T21:30:14.327 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:30:14 smithi097 ceph-mon[33317]: pgmap v162: 148 pgs: 2 active+clean+scrubbing, 146 active+clean; 4.4 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.32k op/s 2024-03-11T21:30:14.327 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:30:14 smithi097 ceph-mon[33317]: osdmap e131: 8 total, 8 up, 8 in 2024-03-11T21:30:14.327 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:30:14 smithi097 ceph-mon[33317]: 7.f scrub ok 2024-03-11T21:30:14.327 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:30:14 smithi097 ceph-mon[33317]: 4.16 scrub starts 2024-03-11T21:30:14.328 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:30:14 smithi097 ceph-mon[33317]: 4.16 scrub ok 2024-03-11T21:30:14.329 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:30:14.329 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:30:14.329 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:30:14.426 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:30:14 smithi092 ceph-mon[29572]: 2.5 scrub starts 2024-03-11T21:30:14.427 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:30:14 smithi092 ceph-mon[29572]: 2.5 scrub ok 2024-03-11T21:30:14.427 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:30:14 smithi092 ceph-mon[29572]: 2.17 scrub starts 2024-03-11T21:30:14.427 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:30:14 smithi092 ceph-mon[29572]: 2.17 scrub ok 2024-03-11T21:30:14.427 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:30:14 smithi092 ceph-mon[29572]: 7.e scrub starts 2024-03-11T21:30:14.427 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:30:14 smithi092 ceph-mon[29572]: purged_snaps scrub starts 2024-03-11T21:30:14.427 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:30:14 smithi092 ceph-mon[29572]: purged_snaps scrub ok 2024-03-11T21:30:14.427 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:30:14 smithi092 ceph-mon[29572]: 2.12 scrub starts 2024-03-11T21:30:14.427 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:30:14 smithi092 ceph-mon[29572]: 2.12 scrub ok 2024-03-11T21:30:14.427 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:30:14 smithi092 ceph-mon[29572]: purged_snaps scrub starts 2024-03-11T21:30:14.427 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:30:14 smithi092 ceph-mon[29572]: purged_snaps scrub ok 2024-03-11T21:30:14.427 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:30:14 smithi092 ceph-mon[29572]: pgmap v162: 148 pgs: 2 active+clean+scrubbing, 146 active+clean; 4.4 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.32k op/s 2024-03-11T21:30:14.427 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:30:14 smithi092 ceph-mon[29572]: osdmap e131: 8 total, 8 up, 8 in 2024-03-11T21:30:14.428 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:30:14 smithi092 ceph-mon[29572]: 7.f scrub ok 2024-03-11T21:30:14.428 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:30:14 smithi092 ceph-mon[29572]: 4.16 scrub starts 2024-03-11T21:30:14.428 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:30:14 smithi092 ceph-mon[29572]: 4.16 scrub ok 2024-03-11T21:30:14.429 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:30:14 smithi092 ceph-mon[37563]: 2.5 scrub starts 2024-03-11T21:30:14.429 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:30:14 smithi092 ceph-mon[37563]: 2.5 scrub ok 2024-03-11T21:30:14.429 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:30:14 smithi092 ceph-mon[37563]: 2.17 scrub starts 2024-03-11T21:30:14.429 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:30:14 smithi092 ceph-mon[37563]: 2.17 scrub ok 2024-03-11T21:30:14.429 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:30:14 smithi092 ceph-mon[37563]: 7.e scrub starts 2024-03-11T21:30:14.430 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:30:14 smithi092 ceph-mon[37563]: purged_snaps scrub starts 2024-03-11T21:30:14.430 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:30:14 smithi092 ceph-mon[37563]: purged_snaps scrub ok 2024-03-11T21:30:14.430 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:30:14 smithi092 ceph-mon[37563]: 2.12 scrub starts 2024-03-11T21:30:14.430 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:30:14 smithi092 ceph-mon[37563]: 2.12 scrub ok 2024-03-11T21:30:14.430 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:30:14 smithi092 ceph-mon[37563]: purged_snaps scrub starts 2024-03-11T21:30:14.430 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:30:14 smithi092 ceph-mon[37563]: purged_snaps scrub ok 2024-03-11T21:30:14.430 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:30:14 smithi092 ceph-mon[37563]: pgmap v162: 148 pgs: 2 active+clean+scrubbing, 146 active+clean; 4.4 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.32k op/s 2024-03-11T21:30:14.430 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:30:14 smithi092 ceph-mon[37563]: osdmap e131: 8 total, 8 up, 8 in 2024-03-11T21:30:14.430 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:30:14 smithi092 ceph-mon[37563]: 7.f scrub ok 2024-03-11T21:30:14.430 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:30:14 smithi092 ceph-mon[37563]: 4.16 scrub starts 2024-03-11T21:30:14.430 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:30:14 smithi092 ceph-mon[37563]: 4.16 scrub ok 2024-03-11T21:30:14.431 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:30:14.460 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:30:14.461 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:30:14.461 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:30:14.562 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:30:14.605 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:30:14.606 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:30:14.606 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:30:14.707 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:30:14.734 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:30:14.735 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:30:14.735 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:30:14.836 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:30:14.875 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:30:14.875 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:30:14.875 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:30:14.976 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:30:15.008 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:30:15.008 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:30:15.008 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:30:15.109 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:30:15.136 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:30:15.137 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:30:15.137 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:30:15.238 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:30:15.270 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:30:15.271 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:30:15.271 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:30:15.361 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:30:15 smithi092 ceph-mon[29572]: 2.8 scrub starts 2024-03-11T21:30:15.361 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:30:15 smithi092 ceph-mon[29572]: 2.8 scrub ok 2024-03-11T21:30:15.361 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:30:15 smithi092 ceph-mon[29572]: 4.f scrub starts 2024-03-11T21:30:15.361 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:30:15 smithi092 ceph-mon[29572]: 4.f scrub ok 2024-03-11T21:30:15.361 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:30:15 smithi092 ceph-mon[29572]: 7.9 scrub ok 2024-03-11T21:30:15.361 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:30:15 smithi092 ceph-mon[29572]: 7.e scrub ok 2024-03-11T21:30:15.361 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:30:15 smithi092 ceph-mon[29572]: 2.1f scrub starts 2024-03-11T21:30:15.361 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:30:15 smithi092 ceph-mon[29572]: 2.1f scrub ok 2024-03-11T21:30:15.361 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:30:15 smithi092 ceph-mon[37563]: 2.8 scrub starts 2024-03-11T21:30:15.362 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:30:15 smithi092 ceph-mon[37563]: 2.8 scrub ok 2024-03-11T21:30:15.362 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:30:15 smithi092 ceph-mon[37563]: 4.f scrub starts 2024-03-11T21:30:15.362 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:30:15 smithi092 ceph-mon[37563]: 4.f scrub ok 2024-03-11T21:30:15.362 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:30:15 smithi092 ceph-mon[37563]: 7.9 scrub ok 2024-03-11T21:30:15.362 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:30:15 smithi092 ceph-mon[37563]: 7.e scrub ok 2024-03-11T21:30:15.362 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:30:15 smithi092 ceph-mon[37563]: 2.1f scrub starts 2024-03-11T21:30:15.362 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:30:15 smithi092 ceph-mon[37563]: 2.1f scrub ok 2024-03-11T21:30:15.371 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:30:15.392 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:30:15 smithi097 ceph-mon[33317]: 2.8 scrub starts 2024-03-11T21:30:15.392 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:30:15 smithi097 ceph-mon[33317]: 2.8 scrub ok 2024-03-11T21:30:15.392 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:30:15 smithi097 ceph-mon[33317]: 4.f scrub starts 2024-03-11T21:30:15.392 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:30:15 smithi097 ceph-mon[33317]: 4.f scrub ok 2024-03-11T21:30:15.392 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:30:15 smithi097 ceph-mon[33317]: 7.9 scrub ok 2024-03-11T21:30:15.393 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:30:15 smithi097 ceph-mon[33317]: 7.e scrub ok 2024-03-11T21:30:15.393 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:30:15 smithi097 ceph-mon[33317]: 2.1f scrub starts 2024-03-11T21:30:15.393 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:30:15 smithi097 ceph-mon[33317]: 2.1f scrub ok 2024-03-11T21:30:15.401 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:30:15.402 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:30:15.402 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:30:15.503 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:30:15.536 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:30:15.536 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:30:15.536 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:30:15.547 DEBUG:teuthology.orchestra.run.smithi092:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid 00db0016-dfed-11ee-95c8-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-03-11T21:30:15.572 INFO:teuthology.orchestra.run.smithi097.stdout:{ 2024-03-11T21:30:15.572 INFO:teuthology.orchestra.run.smithi097.stdout: "size": 20, 2024-03-11T21:30:15.572 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 600, 2024-03-11T21:30:15.572 INFO:teuthology.orchestra.run.smithi097.stdout: "ops": [ 2024-03-11T21:30:15.572 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:30:15.573 INFO:teuthology.orchestra.run.smithi097.stdout: "description": "osd_op(client.34715.0:116281 7.7 7:e9a6f278:::benchmark_data_smithi092_76040_object116280:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e119)", 2024-03-11T21:30:15.573 INFO:teuthology.orchestra.run.smithi097.stdout: "initiated_at": "2024-03-11T21:29:08.769942+0000", 2024-03-11T21:30:15.573 INFO:teuthology.orchestra.run.smithi097.stdout: "age": 66.817643365999999, 2024-03-11T21:30:15.573 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.12800686, 2024-03-11T21:30:15.573 INFO:teuthology.orchestra.run.smithi097.stdout: "type_data": { 2024-03-11T21:30:15.573 INFO:teuthology.orchestra.run.smithi097.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-11T21:30:15.573 INFO:teuthology.orchestra.run.smithi097.stdout: "client_info": { 2024-03-11T21:30:15.573 INFO:teuthology.orchestra.run.smithi097.stdout: "client": "client.34715", 2024-03-11T21:30:15.573 INFO:teuthology.orchestra.run.smithi097.stdout: "client_addr": "v1:172.21.15.92:0/2403132573", 2024-03-11T21:30:15.573 INFO:teuthology.orchestra.run.smithi097.stdout: "tid": 116281 2024-03-11T21:30:15.573 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:30:15.573 INFO:teuthology.orchestra.run.smithi097.stdout: "events": [ 2024-03-11T21:30:15.573 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:30:15.574 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "initiated", 2024-03-11T21:30:15.574 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:29:08.769942+0000", 2024-03-11T21:30:15.574 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-03-11T21:30:15.574 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:30:15.574 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:30:15.574 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "header_read", 2024-03-11T21:30:15.574 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:29:08.769942+0000", 2024-03-11T21:30:15.574 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-03-11T21:30:15.574 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:30:15.574 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:30:15.574 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "throttled", 2024-03-11T21:30:15.574 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:29:08.769943+0000", 2024-03-11T21:30:15.575 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.4219999999999999e-06 2024-03-11T21:30:15.575 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:30:15.575 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:30:15.575 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "all_read", 2024-03-11T21:30:15.575 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:29:08.769972+0000", 2024-03-11T21:30:15.575 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 2.8693000000000001e-05 2024-03-11T21:30:15.575 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:30:15.575 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:30:15.575 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "dispatched", 2024-03-11T21:30:15.575 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:29:08.769979+0000", 2024-03-11T21:30:15.575 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 7.3300000000000001e-06 2024-03-11T21:30:15.576 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:30:15.576 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:30:15.576 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "queued_for_pg", 2024-03-11T21:30:15.576 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:29:08.769985+0000", 2024-03-11T21:30:15.576 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 5.8560000000000003e-06 2024-03-11T21:30:15.576 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:30:15.576 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:30:15.576 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "reached_pg", 2024-03-11T21:30:15.576 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:29:08.770242+0000", 2024-03-11T21:30:15.576 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.00025719099999999998 2024-03-11T21:30:15.576 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:30:15.576 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:30:15.576 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "started", 2024-03-11T21:30:15.576 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:29:08.770373+0000", 2024-03-11T21:30:15.577 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.000130918 2024-03-11T21:30:15.577 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:30:15.577 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:30:15.577 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "waiting for subops from 0,3", 2024-03-11T21:30:15.577 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:29:08.770761+0000", 2024-03-11T21:30:15.577 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.00038815099999999999 2024-03-11T21:30:15.577 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:30:15.577 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:30:15.577 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "op_commit", 2024-03-11T21:30:15.577 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:29:08.778257+0000", 2024-03-11T21:30:15.577 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.0074957110000000004 2024-03-11T21:30:15.577 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:30:15.577 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:30:15.577 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "sub_op_commit_rec", 2024-03-11T21:30:15.578 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:29:08.882012+0000", 2024-03-11T21:30:15.578 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.10375451199999999 2024-03-11T21:30:15.578 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:30:15.578 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:30:15.578 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "sub_op_commit_rec", 2024-03-11T21:30:15.578 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:29:08.897835+0000", 2024-03-11T21:30:15.578 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.015823554 2024-03-11T21:30:15.578 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:30:15.578 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:30:15.578 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "commit_sent", 2024-03-11T21:30:15.578 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:29:08.897893+0000", 2024-03-11T21:30:15.579 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 5.7494999999999998e-05 2024-03-11T21:30:15.579 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:30:15.579 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:30:15.579 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "done", 2024-03-11T21:30:15.579 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:29:08.897949+0000", 2024-03-11T21:30:15.579 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 5.6026999999999997e-05 2024-03-11T21:30:15.579 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-03-11T21:30:15.579 INFO:teuthology.orchestra.run.smithi097.stdout: ] 2024-03-11T21:30:15.579 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-03-11T21:30:15.579 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:30:15.579 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:30:15.579 INFO:teuthology.orchestra.run.smithi097.stdout: "description": "osd_op(client.34715.0:139390 7.7 7:e33e1b77:::benchmark_data_smithi092_76040_object17479:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e124)", 2024-03-11T21:30:15.579 INFO:teuthology.orchestra.run.smithi097.stdout: "initiated_at": "2024-03-11T21:29:33.973801+0000", 2024-03-11T21:30:15.579 INFO:teuthology.orchestra.run.smithi097.stdout: "age": 41.613784445999997, 2024-03-11T21:30:15.580 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.17549214799999999, 2024-03-11T21:30:15.580 INFO:teuthology.orchestra.run.smithi097.stdout: "type_data": { 2024-03-11T21:30:15.580 INFO:teuthology.orchestra.run.smithi097.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-11T21:30:15.580 INFO:teuthology.orchestra.run.smithi097.stdout: "client_info": { 2024-03-11T21:30:15.580 INFO:teuthology.orchestra.run.smithi097.stdout: "client": "client.34715", 2024-03-11T21:30:15.580 INFO:teuthology.orchestra.run.smithi097.stdout: "client_addr": "v1:172.21.15.92:0/2403132573", 2024-03-11T21:30:15.580 INFO:teuthology.orchestra.run.smithi097.stdout: "tid": 139390 2024-03-11T21:30:15.580 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:30:15.580 INFO:teuthology.orchestra.run.smithi097.stdout: "events": [ 2024-03-11T21:30:15.580 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:30:15.580 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "initiated", 2024-03-11T21:30:15.580 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:29:33.973801+0000", 2024-03-11T21:30:15.581 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-03-11T21:30:15.581 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:30:15.581 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:30:15.581 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "header_read", 2024-03-11T21:30:15.581 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:29:33.973801+0000", 2024-03-11T21:30:15.581 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-03-11T21:30:15.581 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:30:15.581 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:30:15.581 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "throttled", 2024-03-11T21:30:15.581 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:29:33.973802+0000", 2024-03-11T21:30:15.581 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.35e-06 2024-03-11T21:30:15.582 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:30:15.582 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:30:15.582 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "all_read", 2024-03-11T21:30:15.582 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:29:33.973806+0000", 2024-03-11T21:30:15.582 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 4.1919999999999998e-06 2024-03-11T21:30:15.582 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:30:15.582 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:30:15.582 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "dispatched", 2024-03-11T21:30:15.582 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:29:33.973817+0000", 2024-03-11T21:30:15.582 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.0295000000000001e-05 2024-03-11T21:30:15.582 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:30:15.582 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:30:15.582 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "queued_for_pg", 2024-03-11T21:30:15.582 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:29:33.973822+0000", 2024-03-11T21:30:15.582 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 5.8529999999999997e-06 2024-03-11T21:30:15.583 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:30:15.583 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:30:15.583 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "reached_pg", 2024-03-11T21:30:15.583 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:29:33.973871+0000", 2024-03-11T21:30:15.583 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 4.8217999999999998e-05 2024-03-11T21:30:15.583 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:30:15.583 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:30:15.583 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "started", 2024-03-11T21:30:15.583 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:29:33.973969+0000", 2024-03-11T21:30:15.583 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 9.8329e-05 2024-03-11T21:30:15.583 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:30:15.583 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:30:15.583 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "waiting for subops from 0,3", 2024-03-11T21:30:15.583 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:29:33.974040+0000", 2024-03-11T21:30:15.584 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 7.1097e-05 2024-03-11T21:30:15.584 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:30:15.584 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:30:15.584 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "op_commit", 2024-03-11T21:30:15.584 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:29:33.976796+0000", 2024-03-11T21:30:15.584 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.002755632 2024-03-11T21:30:15.584 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:30:15.584 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:30:15.584 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "sub_op_commit_rec", 2024-03-11T21:30:15.584 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:29:33.991263+0000", 2024-03-11T21:30:15.584 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.014467117 2024-03-11T21:30:15.584 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:30:15.584 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:30:15.584 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "sub_op_commit_rec", 2024-03-11T21:30:15.585 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:29:34.149224+0000", 2024-03-11T21:30:15.585 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.15796159600000001 2024-03-11T21:30:15.585 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:30:15.585 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:30:15.585 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "commit_sent", 2024-03-11T21:30:15.585 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:29:34.149258+0000", 2024-03-11T21:30:15.585 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 3.3912999999999999e-05 2024-03-11T21:30:15.585 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:30:15.585 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:30:15.585 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "done", 2024-03-11T21:30:15.585 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:29:34.149293+0000", 2024-03-11T21:30:15.585 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 3.4555999999999999e-05 2024-03-11T21:30:15.585 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-03-11T21:30:15.585 INFO:teuthology.orchestra.run.smithi097.stdout: ] 2024-03-11T21:30:15.585 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-03-11T21:30:15.586 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:30:15.586 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:30:15.586 INFO:teuthology.orchestra.run.smithi097.stdout: "description": "osd_op(client.34715.0:139397 7.7 7:ec3d637a:::benchmark_data_smithi092_76040_object17486:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e124)", 2024-03-11T21:30:15.586 INFO:teuthology.orchestra.run.smithi097.stdout: "initiated_at": "2024-03-11T21:29:33.982578+0000", 2024-03-11T21:30:15.586 INFO:teuthology.orchestra.run.smithi097.stdout: "age": 41.605007649999997, 2024-03-11T21:30:15.586 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.16841289000000001, 2024-03-11T21:30:15.586 INFO:teuthology.orchestra.run.smithi097.stdout: "type_data": { 2024-03-11T21:30:15.586 INFO:teuthology.orchestra.run.smithi097.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-11T21:30:15.586 INFO:teuthology.orchestra.run.smithi097.stdout: "client_info": { 2024-03-11T21:30:15.586 INFO:teuthology.orchestra.run.smithi097.stdout: "client": "client.34715", 2024-03-11T21:30:15.586 INFO:teuthology.orchestra.run.smithi097.stdout: "client_addr": "v1:172.21.15.92:0/2403132573", 2024-03-11T21:30:15.586 INFO:teuthology.orchestra.run.smithi097.stdout: "tid": 139397 2024-03-11T21:30:15.586 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:30:15.586 INFO:teuthology.orchestra.run.smithi097.stdout: "events": [ 2024-03-11T21:30:15.587 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:30:15.587 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "initiated", 2024-03-11T21:30:15.588 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:29:33.982578+0000", 2024-03-11T21:30:15.588 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-03-11T21:30:15.588 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:30:15.588 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:30:15.588 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "header_read", 2024-03-11T21:30:15.588 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:29:33.982578+0000", 2024-03-11T21:30:15.588 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-03-11T21:30:15.588 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:30:15.588 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:30:15.588 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "throttled", 2024-03-11T21:30:15.588 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:29:33.982580+0000", 2024-03-11T21:30:15.588 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 2.249e-06 2024-03-11T21:30:15.588 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:30:15.588 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:30:15.588 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "all_read", 2024-03-11T21:30:15.589 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:29:33.982586+0000", 2024-03-11T21:30:15.589 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 6.2430000000000002e-06 2024-03-11T21:30:15.589 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:30:15.589 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:30:15.589 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "dispatched", 2024-03-11T21:30:15.589 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:29:33.982600+0000", 2024-03-11T21:30:15.589 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.411e-05 2024-03-11T21:30:15.589 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:30:15.589 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:30:15.589 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "queued_for_pg", 2024-03-11T21:30:15.589 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:29:33.982609+0000", 2024-03-11T21:30:15.589 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 8.935e-06 2024-03-11T21:30:15.589 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:30:15.589 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:30:15.590 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "reached_pg", 2024-03-11T21:30:15.590 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:29:33.982672+0000", 2024-03-11T21:30:15.590 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 6.2643000000000003e-05 2024-03-11T21:30:15.590 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:30:15.590 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:30:15.590 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "started", 2024-03-11T21:30:15.590 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:29:33.982792+0000", 2024-03-11T21:30:15.590 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.00012063899999999999 2024-03-11T21:30:15.590 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:30:15.590 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:30:15.590 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "waiting for subops from 0,3", 2024-03-11T21:30:15.590 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:29:33.982906+0000", 2024-03-11T21:30:15.590 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.000114123 2024-03-11T21:30:15.591 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:30:15.591 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:30:15.591 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "op_commit", 2024-03-11T21:30:15.591 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:29:33.988300+0000", 2024-03-11T21:30:15.591 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.0053937610000000004 2024-03-11T21:30:15.591 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:30:15.591 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:30:15.591 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "sub_op_commit_rec", 2024-03-11T21:30:15.592 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:29:34.000785+0000", 2024-03-11T21:30:15.592 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.012484567 2024-03-11T21:30:15.592 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:30:15.592 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:30:15.592 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "sub_op_commit_rec", 2024-03-11T21:30:15.592 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:29:34.150912+0000", 2024-03-11T21:30:15.592 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.15012709499999999 2024-03-11T21:30:15.592 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:30:15.592 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:30:15.592 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "commit_sent", 2024-03-11T21:30:15.592 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:29:34.150944+0000", 2024-03-11T21:30:15.592 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 3.2005999999999998e-05 2024-03-11T21:30:15.592 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:30:15.592 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:30:15.593 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "done", 2024-03-11T21:30:15.593 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:29:34.150990+0000", 2024-03-11T21:30:15.593 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 4.6519e-05 2024-03-11T21:30:15.593 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-03-11T21:30:15.593 INFO:teuthology.orchestra.run.smithi097.stdout: ] 2024-03-11T21:30:15.593 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-03-11T21:30:15.593 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:30:15.593 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:30:15.593 INFO:teuthology.orchestra.run.smithi097.stdout: "description": "osd_op(client.34715.0:144300 7.7 7:e2e00930:::benchmark_data_smithi092_76040_object22389:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e124)", 2024-03-11T21:30:15.593 INFO:teuthology.orchestra.run.smithi097.stdout: "initiated_at": "2024-03-11T21:29:39.693117+0000", 2024-03-11T21:30:15.593 INFO:teuthology.orchestra.run.smithi097.stdout: "age": 35.894467874, 2024-03-11T21:30:15.593 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.135023847, 2024-03-11T21:30:15.593 INFO:teuthology.orchestra.run.smithi097.stdout: "type_data": { 2024-03-11T21:30:15.593 INFO:teuthology.orchestra.run.smithi097.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-11T21:30:15.593 INFO:teuthology.orchestra.run.smithi097.stdout: "client_info": { 2024-03-11T21:30:15.594 INFO:teuthology.orchestra.run.smithi097.stdout: "client": "client.34715", 2024-03-11T21:30:15.594 INFO:teuthology.orchestra.run.smithi097.stdout: "client_addr": "v1:172.21.15.92:0/2403132573", 2024-03-11T21:30:15.594 INFO:teuthology.orchestra.run.smithi097.stdout: "tid": 144300 2024-03-11T21:30:15.594 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:30:15.594 INFO:teuthology.orchestra.run.smithi097.stdout: "events": [ 2024-03-11T21:30:15.594 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:30:15.594 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "initiated", 2024-03-11T21:30:15.594 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:29:39.693117+0000", 2024-03-11T21:30:15.594 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-03-11T21:30:15.594 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:30:15.594 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:30:15.594 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "header_read", 2024-03-11T21:30:15.594 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:29:39.693117+0000", 2024-03-11T21:30:15.594 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-03-11T21:30:15.595 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:30:15.595 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:30:15.595 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "throttled", 2024-03-11T21:30:15.595 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:29:39.693119+0000", 2024-03-11T21:30:15.595 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.889e-06 2024-03-11T21:30:15.595 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:30:15.595 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:30:15.595 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "all_read", 2024-03-11T21:30:15.595 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:29:39.693125+0000", 2024-03-11T21:30:15.595 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 5.677e-06 2024-03-11T21:30:15.595 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:30:15.595 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:30:15.595 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "dispatched", 2024-03-11T21:30:15.595 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:29:39.693139+0000", 2024-03-11T21:30:15.596 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.3631999999999999e-05 2024-03-11T21:30:15.596 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:30:15.596 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:30:15.596 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "queued_for_pg", 2024-03-11T21:30:15.596 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:29:39.693146+0000", 2024-03-11T21:30:15.596 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 7.571e-06 2024-03-11T21:30:15.596 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:30:15.596 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:30:15.596 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "reached_pg", 2024-03-11T21:30:15.596 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:29:39.693200+0000", 2024-03-11T21:30:15.596 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 5.4194999999999999e-05 2024-03-11T21:30:15.596 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:30:15.596 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:30:15.596 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "started", 2024-03-11T21:30:15.596 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:29:39.693320+0000", 2024-03-11T21:30:15.597 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.00011953 2024-03-11T21:30:15.597 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:30:15.597 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:30:15.597 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "waiting for subops from 0,3", 2024-03-11T21:30:15.597 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:29:39.693452+0000", 2024-03-11T21:30:15.597 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.00013235499999999999 2024-03-11T21:30:15.597 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:30:15.597 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:30:15.597 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "op_commit", 2024-03-11T21:30:15.597 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:29:39.696359+0000", 2024-03-11T21:30:15.597 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.002906788 2024-03-11T21:30:15.597 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:30:15.597 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:30:15.597 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "sub_op_commit_rec", 2024-03-11T21:30:15.597 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:29:39.729028+0000", 2024-03-11T21:30:15.598 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.032669147000000003 2024-03-11T21:30:15.598 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:30:15.598 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:30:15.598 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "sub_op_commit_rec", 2024-03-11T21:30:15.598 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:29:39.828070+0000", 2024-03-11T21:30:15.598 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.099042023000000007 2024-03-11T21:30:15.598 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:30:15.598 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:30:15.598 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "commit_sent", 2024-03-11T21:30:15.598 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:29:39.828103+0000", 2024-03-11T21:30:15.599 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 3.3101e-05 2024-03-11T21:30:15.599 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:30:15.599 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:30:15.599 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "done", 2024-03-11T21:30:15.599 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:29:39.828141+0000", 2024-03-11T21:30:15.599 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 3.7938999999999997e-05 2024-03-11T21:30:15.599 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-03-11T21:30:15.599 INFO:teuthology.orchestra.run.smithi097.stdout: ] 2024-03-11T21:30:15.599 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-03-11T21:30:15.599 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:30:15.599 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:30:15.599 INFO:teuthology.orchestra.run.smithi097.stdout: "description": "osd_op(client.34715.0:144480 7.7 7:e7fe194c:::benchmark_data_smithi092_76040_object22569:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e124)", 2024-03-11T21:30:15.599 INFO:teuthology.orchestra.run.smithi097.stdout: "initiated_at": "2024-03-11T21:29:39.897362+0000", 2024-03-11T21:30:15.599 INFO:teuthology.orchestra.run.smithi097.stdout: "age": 35.690223310999997, 2024-03-11T21:30:15.600 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.12777027899999999, 2024-03-11T21:30:15.600 INFO:teuthology.orchestra.run.smithi097.stdout: "type_data": { 2024-03-11T21:30:15.600 INFO:teuthology.orchestra.run.smithi097.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-11T21:30:15.600 INFO:teuthology.orchestra.run.smithi097.stdout: "client_info": { 2024-03-11T21:30:15.600 INFO:teuthology.orchestra.run.smithi097.stdout: "client": "client.34715", 2024-03-11T21:30:15.600 INFO:teuthology.orchestra.run.smithi097.stdout: "client_addr": "v1:172.21.15.92:0/2403132573", 2024-03-11T21:30:15.600 INFO:teuthology.orchestra.run.smithi097.stdout: "tid": 144480 2024-03-11T21:30:15.600 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:30:15.600 INFO:teuthology.orchestra.run.smithi097.stdout: "events": [ 2024-03-11T21:30:15.600 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:30:15.600 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "initiated", 2024-03-11T21:30:15.600 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:29:39.897362+0000", 2024-03-11T21:30:15.600 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-03-11T21:30:15.600 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:30:15.600 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:30:15.601 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "header_read", 2024-03-11T21:30:15.601 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:29:39.897362+0000", 2024-03-11T21:30:15.601 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-03-11T21:30:15.601 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:30:15.601 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:30:15.601 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "throttled", 2024-03-11T21:30:15.601 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:29:39.897363+0000", 2024-03-11T21:30:15.601 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.4139999999999999e-06 2024-03-11T21:30:15.601 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:30:15.602 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:30:15.602 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "all_read", 2024-03-11T21:30:15.602 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:29:39.897369+0000", 2024-03-11T21:30:15.602 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 5.6500000000000001e-06 2024-03-11T21:30:15.602 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:30:15.602 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:30:15.602 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "dispatched", 2024-03-11T21:30:15.602 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:29:39.897382+0000", 2024-03-11T21:30:15.602 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.2962999999999999e-05 2024-03-11T21:30:15.602 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:30:15.602 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:30:15.602 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "queued_for_pg", 2024-03-11T21:30:15.603 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:29:39.897388+0000", 2024-03-11T21:30:15.603 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 5.9429999999999997e-06 2024-03-11T21:30:15.603 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:30:15.603 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:30:15.603 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "reached_pg", 2024-03-11T21:30:15.603 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:29:39.899927+0000", 2024-03-11T21:30:15.603 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.002539201 2024-03-11T21:30:15.603 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:30:15.603 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:30:15.603 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "started", 2024-03-11T21:30:15.603 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:29:39.900054+0000", 2024-03-11T21:30:15.603 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.00012728800000000001 2024-03-11T21:30:15.603 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:30:15.603 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:30:15.603 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "waiting for subops from 0,3", 2024-03-11T21:30:15.604 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:29:39.900182+0000", 2024-03-11T21:30:15.604 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.00012769300000000001 2024-03-11T21:30:15.604 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:30:15.604 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:30:15.604 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "op_commit", 2024-03-11T21:30:15.604 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:29:39.903208+0000", 2024-03-11T21:30:15.604 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.0030261239999999998 2024-03-11T21:30:15.604 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:30:15.604 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:30:15.604 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "sub_op_commit_rec", 2024-03-11T21:30:15.604 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:29:39.945796+0000", 2024-03-11T21:30:15.604 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.042588295999999998 2024-03-11T21:30:15.604 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:30:15.604 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:30:15.605 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "sub_op_commit_rec", 2024-03-11T21:30:15.605 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:29:40.025063+0000", 2024-03-11T21:30:15.605 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.079266485999999997 2024-03-11T21:30:15.605 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:30:15.605 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:30:15.605 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "commit_sent", 2024-03-11T21:30:15.605 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:29:40.025093+0000", 2024-03-11T21:30:15.605 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 3.0389999999999999e-05 2024-03-11T21:30:15.605 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:30:15.605 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:30:15.605 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "done", 2024-03-11T21:30:15.605 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:29:40.025132+0000", 2024-03-11T21:30:15.605 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 3.8831000000000001e-05 2024-03-11T21:30:15.605 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-03-11T21:30:15.606 INFO:teuthology.orchestra.run.smithi097.stdout: ] 2024-03-11T21:30:15.606 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-03-11T21:30:15.606 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:30:15.606 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:30:15.606 INFO:teuthology.orchestra.run.smithi097.stdout: "description": "osd_op(client.34715.0:153209 7.7 7:ed96a7d9:::benchmark_data_smithi092_76040_object31298:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e126)", 2024-03-11T21:30:15.606 INFO:teuthology.orchestra.run.smithi097.stdout: "initiated_at": "2024-03-11T21:29:52.339978+0000", 2024-03-11T21:30:15.606 INFO:teuthology.orchestra.run.smithi097.stdout: "age": 23.247607622, 2024-03-11T21:30:15.606 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.130714104, 2024-03-11T21:30:15.606 INFO:teuthology.orchestra.run.smithi097.stdout: "type_data": { 2024-03-11T21:30:15.606 INFO:teuthology.orchestra.run.smithi097.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-11T21:30:15.606 INFO:teuthology.orchestra.run.smithi097.stdout: "client_info": { 2024-03-11T21:30:15.606 INFO:teuthology.orchestra.run.smithi097.stdout: "client": "client.34715", 2024-03-11T21:30:15.606 INFO:teuthology.orchestra.run.smithi097.stdout: "client_addr": "v1:172.21.15.92:0/2403132573", 2024-03-11T21:30:15.606 INFO:teuthology.orchestra.run.smithi097.stdout: "tid": 153209 2024-03-11T21:30:15.607 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:30:15.607 INFO:teuthology.orchestra.run.smithi097.stdout: "events": [ 2024-03-11T21:30:15.607 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:30:15.607 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "initiated", 2024-03-11T21:30:15.607 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:29:52.339978+0000", 2024-03-11T21:30:15.607 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-03-11T21:30:15.607 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:30:15.607 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:30:15.607 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "header_read", 2024-03-11T21:30:15.607 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:29:52.339978+0000", 2024-03-11T21:30:15.607 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-03-11T21:30:15.607 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:30:15.607 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:30:15.607 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "throttled", 2024-03-11T21:30:15.608 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:29:52.339980+0000", 2024-03-11T21:30:15.608 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 2.0490000000000002e-06 2024-03-11T21:30:15.608 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:30:15.608 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:30:15.608 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "all_read", 2024-03-11T21:30:15.608 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:29:52.339985+0000", 2024-03-11T21:30:15.608 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 4.9110000000000001e-06 2024-03-11T21:30:15.608 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:30:15.608 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:30:15.608 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "dispatched", 2024-03-11T21:30:15.608 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:29:52.339995+0000", 2024-03-11T21:30:15.608 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.0706e-05 2024-03-11T21:30:15.608 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:30:15.608 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:30:15.608 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "queued_for_pg", 2024-03-11T21:30:15.609 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:29:52.340002+0000", 2024-03-11T21:30:15.609 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 6.9160000000000004e-06 2024-03-11T21:30:15.609 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:30:15.609 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:30:15.609 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "reached_pg", 2024-03-11T21:30:15.609 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:29:52.340074+0000", 2024-03-11T21:30:15.609 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 7.1897000000000006e-05 2024-03-11T21:30:15.609 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:30:15.609 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:30:15.609 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "started", 2024-03-11T21:30:15.609 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:29:52.340212+0000", 2024-03-11T21:30:15.609 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.00013782900000000001 2024-03-11T21:30:15.609 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:30:15.609 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:30:15.609 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "waiting for subops from 0,3", 2024-03-11T21:30:15.610 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:29:52.340329+0000", 2024-03-11T21:30:15.610 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.000117428 2024-03-11T21:30:15.610 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:30:15.610 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:30:15.610 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "op_commit", 2024-03-11T21:30:15.610 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:29:52.344131+0000", 2024-03-11T21:30:15.610 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.003801582 2024-03-11T21:30:15.610 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:30:15.610 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:30:15.610 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "sub_op_commit_rec", 2024-03-11T21:30:15.610 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:29:52.357426+0000", 2024-03-11T21:30:15.610 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.013295039999999999 2024-03-11T21:30:15.610 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:30:15.610 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:30:15.611 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "sub_op_commit_rec", 2024-03-11T21:30:15.611 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:29:52.470631+0000", 2024-03-11T21:30:15.611 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.11320485299999999 2024-03-11T21:30:15.611 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:30:15.611 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:30:15.611 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "commit_sent", 2024-03-11T21:30:15.611 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:29:52.470665+0000", 2024-03-11T21:30:15.611 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 3.3986000000000003e-05 2024-03-11T21:30:15.611 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:30:15.611 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:30:15.611 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "done", 2024-03-11T21:30:15.611 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:29:52.470692+0000", 2024-03-11T21:30:15.611 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 2.6906999999999998e-05 2024-03-11T21:30:15.611 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-03-11T21:30:15.612 INFO:teuthology.orchestra.run.smithi097.stdout: ] 2024-03-11T21:30:15.612 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-03-11T21:30:15.612 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:30:15.612 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:30:15.612 INFO:teuthology.orchestra.run.smithi097.stdout: "description": "osd_op(client.34715.0:153402 7.7 7:e10f6c31:::benchmark_data_smithi092_76040_object31491:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e126)", 2024-03-11T21:30:15.612 INFO:teuthology.orchestra.run.smithi097.stdout: "initiated_at": "2024-03-11T21:29:52.570450+0000", 2024-03-11T21:30:15.612 INFO:teuthology.orchestra.run.smithi097.stdout: "age": 23.017135367000002, 2024-03-11T21:30:15.612 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.131075313, 2024-03-11T21:30:15.612 INFO:teuthology.orchestra.run.smithi097.stdout: "type_data": { 2024-03-11T21:30:15.612 INFO:teuthology.orchestra.run.smithi097.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-11T21:30:15.612 INFO:teuthology.orchestra.run.smithi097.stdout: "client_info": { 2024-03-11T21:30:15.612 INFO:teuthology.orchestra.run.smithi097.stdout: "client": "client.34715", 2024-03-11T21:30:15.612 INFO:teuthology.orchestra.run.smithi097.stdout: "client_addr": "v1:172.21.15.92:0/2403132573", 2024-03-11T21:30:15.612 INFO:teuthology.orchestra.run.smithi097.stdout: "tid": 153402 2024-03-11T21:30:15.613 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:30:15.613 INFO:teuthology.orchestra.run.smithi097.stdout: "events": [ 2024-03-11T21:30:15.613 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:30:15.613 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "initiated", 2024-03-11T21:30:15.613 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:29:52.570450+0000", 2024-03-11T21:30:15.613 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-03-11T21:30:15.613 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:30:15.613 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:30:15.613 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "header_read", 2024-03-11T21:30:15.613 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:29:52.570450+0000", 2024-03-11T21:30:15.613 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-03-11T21:30:15.613 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:30:15.613 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:30:15.613 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "throttled", 2024-03-11T21:30:15.614 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:29:52.570452+0000", 2024-03-11T21:30:15.614 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.7320000000000001e-06 2024-03-11T21:30:15.614 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:30:15.614 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:30:15.614 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "all_read", 2024-03-11T21:30:15.614 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:29:52.570457+0000", 2024-03-11T21:30:15.614 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 5.4620000000000004e-06 2024-03-11T21:30:15.614 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:30:15.614 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:30:15.614 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "dispatched", 2024-03-11T21:30:15.614 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:29:52.570467+0000", 2024-03-11T21:30:15.615 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.0152e-05 2024-03-11T21:30:15.615 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:30:15.615 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:30:15.615 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "queued_for_pg", 2024-03-11T21:30:15.615 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:29:52.570473+0000", 2024-03-11T21:30:15.615 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 5.9780000000000002e-06 2024-03-11T21:30:15.615 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:30:15.615 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:30:15.615 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "reached_pg", 2024-03-11T21:30:15.615 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:29:52.570516+0000", 2024-03-11T21:30:15.615 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 4.3167999999999997e-05 2024-03-11T21:30:15.615 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:30:15.615 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:30:15.615 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "started", 2024-03-11T21:30:15.616 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:29:52.570633+0000", 2024-03-11T21:30:15.616 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.000116631 2024-03-11T21:30:15.616 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:30:15.616 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:30:15.616 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "waiting for subops from 0,3", 2024-03-11T21:30:15.616 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:29:52.570773+0000", 2024-03-11T21:30:15.616 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.00014012399999999999 2024-03-11T21:30:15.616 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:30:15.616 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:30:15.616 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "op_commit", 2024-03-11T21:30:15.616 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:29:52.573627+0000", 2024-03-11T21:30:15.616 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.0028536780000000001 2024-03-11T21:30:15.616 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:30:15.616 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:30:15.616 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "sub_op_commit_rec", 2024-03-11T21:30:15.617 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:29:52.594284+0000", 2024-03-11T21:30:15.617 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.020657149 2024-03-11T21:30:15.617 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:30:15.617 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:30:15.617 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "sub_op_commit_rec", 2024-03-11T21:30:15.617 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:29:52.701421+0000", 2024-03-11T21:30:15.617 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.107137526 2024-03-11T21:30:15.617 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:30:15.617 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:30:15.617 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "commit_sent", 2024-03-11T21:30:15.617 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:29:52.701479+0000", 2024-03-11T21:30:15.617 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 5.7379000000000001e-05 2024-03-11T21:30:15.617 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:30:15.617 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:30:15.618 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "done", 2024-03-11T21:30:15.618 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:29:52.701525+0000", 2024-03-11T21:30:15.618 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 4.6334000000000001e-05 2024-03-11T21:30:15.618 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-03-11T21:30:15.618 INFO:teuthology.orchestra.run.smithi097.stdout: ] 2024-03-11T21:30:15.618 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-03-11T21:30:15.618 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:30:15.618 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:30:15.618 INFO:teuthology.orchestra.run.smithi097.stdout: "description": "osd_op(client.34715.0:153875 7.7 7:e90aeecd:::benchmark_data_smithi092_76040_object31964:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e126)", 2024-03-11T21:30:15.618 INFO:teuthology.orchestra.run.smithi097.stdout: "initiated_at": "2024-03-11T21:29:53.270395+0000", 2024-03-11T21:30:15.618 INFO:teuthology.orchestra.run.smithi097.stdout: "age": 22.317190318000002, 2024-03-11T21:30:15.618 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.13311833100000001, 2024-03-11T21:30:15.618 INFO:teuthology.orchestra.run.smithi097.stdout: "type_data": { 2024-03-11T21:30:15.618 INFO:teuthology.orchestra.run.smithi097.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-11T21:30:15.619 INFO:teuthology.orchestra.run.smithi097.stdout: "client_info": { 2024-03-11T21:30:15.619 INFO:teuthology.orchestra.run.smithi097.stdout: "client": "client.34715", 2024-03-11T21:30:15.619 INFO:teuthology.orchestra.run.smithi097.stdout: "client_addr": "v1:172.21.15.92:0/2403132573", 2024-03-11T21:30:15.619 INFO:teuthology.orchestra.run.smithi097.stdout: "tid": 153875 2024-03-11T21:30:15.619 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:30:15.619 INFO:teuthology.orchestra.run.smithi097.stdout: "events": [ 2024-03-11T21:30:15.619 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:30:15.619 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "initiated", 2024-03-11T21:30:15.619 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:29:53.270395+0000", 2024-03-11T21:30:15.619 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-03-11T21:30:15.619 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:30:15.619 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:30:15.619 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "header_read", 2024-03-11T21:30:15.619 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:29:53.270395+0000", 2024-03-11T21:30:15.619 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-03-11T21:30:15.620 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:30:15.620 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:30:15.620 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "throttled", 2024-03-11T21:30:15.620 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:29:53.270396+0000", 2024-03-11T21:30:15.620 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 6.3099999999999997e-07 2024-03-11T21:30:15.620 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:30:15.620 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:30:15.620 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "all_read", 2024-03-11T21:30:15.620 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:29:53.270398+0000", 2024-03-11T21:30:15.620 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 2.7250000000000002e-06 2024-03-11T21:30:15.620 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:30:15.620 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:30:15.620 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "dispatched", 2024-03-11T21:30:15.621 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:29:53.270404+0000", 2024-03-11T21:30:15.621 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 5.4600000000000002e-06 2024-03-11T21:30:15.621 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:30:15.621 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:30:15.621 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "queued_for_pg", 2024-03-11T21:30:15.621 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:29:53.270412+0000", 2024-03-11T21:30:15.621 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 8.0169999999999996e-06 2024-03-11T21:30:15.621 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:30:15.621 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:30:15.621 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "reached_pg", 2024-03-11T21:30:15.621 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:29:53.273086+0000", 2024-03-11T21:30:15.621 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.0026740679999999999 2024-03-11T21:30:15.621 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:30:15.621 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:30:15.621 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "started", 2024-03-11T21:30:15.622 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:29:53.273206+0000", 2024-03-11T21:30:15.622 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.000119879 2024-03-11T21:30:15.622 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:30:15.622 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:30:15.622 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "waiting for subops from 0,3", 2024-03-11T21:30:15.622 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:29:53.273332+0000", 2024-03-11T21:30:15.622 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.00012594400000000001 2024-03-11T21:30:15.622 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:30:15.622 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:30:15.622 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "op_commit", 2024-03-11T21:30:15.622 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:29:53.287050+0000", 2024-03-11T21:30:15.622 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.013718584000000001 2024-03-11T21:30:15.622 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:30:15.622 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:30:15.622 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "sub_op_commit_rec", 2024-03-11T21:30:15.623 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:29:53.298207+0000", 2024-03-11T21:30:15.623 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.011156998 2024-03-11T21:30:15.623 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:30:15.623 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:30:15.623 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "sub_op_commit_rec", 2024-03-11T21:30:15.623 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:29:53.403446+0000", 2024-03-11T21:30:15.623 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.105238767 2024-03-11T21:30:15.623 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:30:15.623 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:30:15.623 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "commit_sent", 2024-03-11T21:30:15.623 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:29:53.403480+0000", 2024-03-11T21:30:15.623 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 3.3951000000000001e-05 2024-03-11T21:30:15.623 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:30:15.623 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:30:15.624 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "done", 2024-03-11T21:30:15.624 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:29:53.403513+0000", 2024-03-11T21:30:15.624 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 3.3306999999999998e-05 2024-03-11T21:30:15.624 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-03-11T21:30:15.624 INFO:teuthology.orchestra.run.smithi097.stdout: ] 2024-03-11T21:30:15.624 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-03-11T21:30:15.624 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:30:15.624 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:30:15.624 INFO:teuthology.orchestra.run.smithi097.stdout: "description": "osd_op(client.34715.0:153881 7.7 7:e4cc13ba:::benchmark_data_smithi092_76040_object31970:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e126)", 2024-03-11T21:30:15.624 INFO:teuthology.orchestra.run.smithi097.stdout: "initiated_at": "2024-03-11T21:29:53.275406+0000", 2024-03-11T21:30:15.624 INFO:teuthology.orchestra.run.smithi097.stdout: "age": 22.312179347000001, 2024-03-11T21:30:15.624 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.12821152599999999, 2024-03-11T21:30:15.624 INFO:teuthology.orchestra.run.smithi097.stdout: "type_data": { 2024-03-11T21:30:15.624 INFO:teuthology.orchestra.run.smithi097.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-11T21:30:15.625 INFO:teuthology.orchestra.run.smithi097.stdout: "client_info": { 2024-03-11T21:30:15.625 INFO:teuthology.orchestra.run.smithi097.stdout: "client": "client.34715", 2024-03-11T21:30:15.625 INFO:teuthology.orchestra.run.smithi097.stdout: "client_addr": "v1:172.21.15.92:0/2403132573", 2024-03-11T21:30:15.625 INFO:teuthology.orchestra.run.smithi097.stdout: "tid": 153881 2024-03-11T21:30:15.625 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:30:15.625 INFO:teuthology.orchestra.run.smithi097.stdout: "events": [ 2024-03-11T21:30:15.625 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:30:15.625 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "initiated", 2024-03-11T21:30:15.625 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:29:53.275406+0000", 2024-03-11T21:30:15.625 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-03-11T21:30:15.625 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:30:15.625 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:30:15.625 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "header_read", 2024-03-11T21:30:15.625 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:29:53.275406+0000", 2024-03-11T21:30:15.626 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-03-11T21:30:15.626 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:30:15.626 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:30:15.626 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "throttled", 2024-03-11T21:30:15.626 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:29:53.275408+0000", 2024-03-11T21:30:15.626 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 2.2630000000000002e-06 2024-03-11T21:30:15.626 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:30:15.626 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:30:15.626 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "all_read", 2024-03-11T21:30:15.626 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:29:53.275414+0000", 2024-03-11T21:30:15.626 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 6.1240000000000002e-06 2024-03-11T21:30:15.626 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:30:15.626 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:30:15.626 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "dispatched", 2024-03-11T21:30:15.627 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:29:53.275426+0000", 2024-03-11T21:30:15.627 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.1489e-05 2024-03-11T21:30:15.627 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:30:15.627 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:30:15.627 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "queued_for_pg", 2024-03-11T21:30:15.627 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:29:53.275434+0000", 2024-03-11T21:30:15.627 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 7.824e-06 2024-03-11T21:30:15.627 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:30:15.627 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:30:15.627 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "reached_pg", 2024-03-11T21:30:15.627 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:29:53.275967+0000", 2024-03-11T21:30:15.627 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.00053317000000000002 2024-03-11T21:30:15.627 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:30:15.627 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:30:15.627 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "started", 2024-03-11T21:30:15.627 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:29:53.276088+0000", 2024-03-11T21:30:15.628 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.00012078099999999999 2024-03-11T21:30:15.628 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:30:15.628 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:30:15.628 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "waiting for subops from 0,3", 2024-03-11T21:30:15.628 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:29:53.276204+0000", 2024-03-11T21:30:15.628 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.00011660899999999999 2024-03-11T21:30:15.628 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:30:15.628 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:30:15.628 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "op_commit", 2024-03-11T21:30:15.628 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:29:53.287267+0000", 2024-03-11T21:30:15.628 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.011062934 2024-03-11T21:30:15.628 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:30:15.628 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:30:15.628 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "sub_op_commit_rec", 2024-03-11T21:30:15.629 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:29:53.298241+0000", 2024-03-11T21:30:15.629 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.010973546000000001 2024-03-11T21:30:15.629 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:30:15.629 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:30:15.629 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "sub_op_commit_rec", 2024-03-11T21:30:15.630 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:29:53.403549+0000", 2024-03-11T21:30:15.630 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.105308238 2024-03-11T21:30:15.630 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:30:15.630 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:30:15.630 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "commit_sent", 2024-03-11T21:30:15.631 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:29:53.403583+0000", 2024-03-11T21:30:15.631 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 3.4094000000000003e-05 2024-03-11T21:30:15.631 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:30:15.631 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:30:15.631 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "done", 2024-03-11T21:30:15.631 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:29:53.403617+0000", 2024-03-11T21:30:15.631 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 3.4453999999999999e-05 2024-03-11T21:30:15.631 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-03-11T21:30:15.631 INFO:teuthology.orchestra.run.smithi097.stdout: ] 2024-03-11T21:30:15.631 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-03-11T21:30:15.631 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:30:15.631 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:30:15.631 INFO:teuthology.orchestra.run.smithi097.stdout: "description": "osd_op(client.34715.0:153882 7.7 7:eb8589cc:::benchmark_data_smithi092_76040_object31971:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e126)", 2024-03-11T21:30:15.631 INFO:teuthology.orchestra.run.smithi097.stdout: "initiated_at": "2024-03-11T21:29:53.277548+0000", 2024-03-11T21:30:15.632 INFO:teuthology.orchestra.run.smithi097.stdout: "age": 22.310037514000001, 2024-03-11T21:30:15.632 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.14272742099999999, 2024-03-11T21:30:15.632 INFO:teuthology.orchestra.run.smithi097.stdout: "type_data": { 2024-03-11T21:30:15.632 INFO:teuthology.orchestra.run.smithi097.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-11T21:30:15.632 INFO:teuthology.orchestra.run.smithi097.stdout: "client_info": { 2024-03-11T21:30:15.632 INFO:teuthology.orchestra.run.smithi097.stdout: "client": "client.34715", 2024-03-11T21:30:15.632 INFO:teuthology.orchestra.run.smithi097.stdout: "client_addr": "v1:172.21.15.92:0/2403132573", 2024-03-11T21:30:15.632 INFO:teuthology.orchestra.run.smithi097.stdout: "tid": 153882 2024-03-11T21:30:15.632 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:30:15.632 INFO:teuthology.orchestra.run.smithi097.stdout: "events": [ 2024-03-11T21:30:15.632 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:30:15.632 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "initiated", 2024-03-11T21:30:15.632 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:29:53.277548+0000", 2024-03-11T21:30:15.632 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-03-11T21:30:15.632 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:30:15.633 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:30:15.633 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "header_read", 2024-03-11T21:30:15.633 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:29:53.277548+0000", 2024-03-11T21:30:15.633 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-03-11T21:30:15.633 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:30:15.633 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:30:15.633 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "throttled", 2024-03-11T21:30:15.633 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:29:53.277549+0000", 2024-03-11T21:30:15.633 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.395e-06 2024-03-11T21:30:15.633 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:30:15.633 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:30:15.633 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "all_read", 2024-03-11T21:30:15.633 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:29:53.277554+0000", 2024-03-11T21:30:15.633 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 5.0440000000000003e-06 2024-03-11T21:30:15.634 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:30:15.634 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:30:15.634 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "dispatched", 2024-03-11T21:30:15.634 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:29:53.277562+0000", 2024-03-11T21:30:15.634 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 7.4749999999999996e-06 2024-03-11T21:30:15.634 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:30:15.634 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:30:15.634 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "queued_for_pg", 2024-03-11T21:30:15.634 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:29:53.277568+0000", 2024-03-11T21:30:15.634 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 6.2480000000000001e-06 2024-03-11T21:30:15.634 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:30:15.634 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:30:15.634 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "reached_pg", 2024-03-11T21:30:15.634 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:29:53.280778+0000", 2024-03-11T21:30:15.635 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.003210559 2024-03-11T21:30:15.635 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:30:15.635 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:30:15.635 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "started", 2024-03-11T21:30:15.635 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:29:53.280862+0000", 2024-03-11T21:30:15.635 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 8.3116999999999997e-05 2024-03-11T21:30:15.635 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:30:15.635 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:30:15.635 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "waiting for subops from 0,3", 2024-03-11T21:30:15.635 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:29:53.280936+0000", 2024-03-11T21:30:15.635 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 7.4949000000000004e-05 2024-03-11T21:30:15.635 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:30:15.635 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:30:15.635 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "op_commit", 2024-03-11T21:30:15.635 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:29:53.287272+0000", 2024-03-11T21:30:15.636 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.0063358640000000001 2024-03-11T21:30:15.636 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:30:15.636 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:30:15.636 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "sub_op_commit_rec", 2024-03-11T21:30:15.636 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:29:53.298265+0000", 2024-03-11T21:30:15.636 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.010992985 2024-03-11T21:30:15.636 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:30:15.636 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:30:15.636 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "sub_op_commit_rec", 2024-03-11T21:30:15.636 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:29:53.420214+0000", 2024-03-11T21:30:15.636 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.121949115 2024-03-11T21:30:15.636 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:30:15.636 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:30:15.636 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "commit_sent", 2024-03-11T21:30:15.637 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:29:53.420247+0000", 2024-03-11T21:30:15.637 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 3.2975999999999997e-05 2024-03-11T21:30:15.637 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:30:15.637 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:30:15.637 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "done", 2024-03-11T21:30:15.637 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:29:53.420275+0000", 2024-03-11T21:30:15.637 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 2.7693999999999999e-05 2024-03-11T21:30:15.637 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-03-11T21:30:15.637 INFO:teuthology.orchestra.run.smithi097.stdout: ] 2024-03-11T21:30:15.637 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-03-11T21:30:15.637 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:30:15.637 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:30:15.637 INFO:teuthology.orchestra.run.smithi097.stdout: "description": "osd_op(client.34715.0:153883 7.7 7:ee50744c:::benchmark_data_smithi092_76040_object31972:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e126)", 2024-03-11T21:30:15.637 INFO:teuthology.orchestra.run.smithi097.stdout: "initiated_at": "2024-03-11T21:29:53.277575+0000", 2024-03-11T21:30:15.637 INFO:teuthology.orchestra.run.smithi097.stdout: "age": 22.310009795999999, 2024-03-11T21:30:15.638 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.14281391299999999, 2024-03-11T21:30:15.638 INFO:teuthology.orchestra.run.smithi097.stdout: "type_data": { 2024-03-11T21:30:15.638 INFO:teuthology.orchestra.run.smithi097.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-11T21:30:15.638 INFO:teuthology.orchestra.run.smithi097.stdout: "client_info": { 2024-03-11T21:30:15.638 INFO:teuthology.orchestra.run.smithi097.stdout: "client": "client.34715", 2024-03-11T21:30:15.638 INFO:teuthology.orchestra.run.smithi097.stdout: "client_addr": "v1:172.21.15.92:0/2403132573", 2024-03-11T21:30:15.638 INFO:teuthology.orchestra.run.smithi097.stdout: "tid": 153883 2024-03-11T21:30:15.638 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:30:15.638 INFO:teuthology.orchestra.run.smithi097.stdout: "events": [ 2024-03-11T21:30:15.638 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:30:15.638 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "initiated", 2024-03-11T21:30:15.638 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:29:53.277575+0000", 2024-03-11T21:30:15.638 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-03-11T21:30:15.638 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:30:15.638 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:30:15.639 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "header_read", 2024-03-11T21:30:15.639 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:29:53.277575+0000", 2024-03-11T21:30:15.639 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-03-11T21:30:15.639 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:30:15.639 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:30:15.639 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "throttled", 2024-03-11T21:30:15.639 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:29:53.277576+0000", 2024-03-11T21:30:15.639 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 4.46e-07 2024-03-11T21:30:15.639 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:30:15.639 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:30:15.639 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "all_read", 2024-03-11T21:30:15.639 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:29:53.277578+0000", 2024-03-11T21:30:15.639 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 2.3769999999999999e-06 2024-03-11T21:30:15.639 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:30:15.639 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:30:15.640 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "dispatched", 2024-03-11T21:30:15.640 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:29:53.277583+0000", 2024-03-11T21:30:15.640 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 4.5680000000000001e-06 2024-03-11T21:30:15.640 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:30:15.640 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:30:15.640 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "queued_for_pg", 2024-03-11T21:30:15.640 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:29:53.277587+0000", 2024-03-11T21:30:15.640 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 4.4549999999999997e-06 2024-03-11T21:30:15.640 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:30:15.640 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:30:15.640 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "reached_pg", 2024-03-11T21:30:15.640 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:29:53.284150+0000", 2024-03-11T21:30:15.640 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.0065629340000000003 2024-03-11T21:30:15.640 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:30:15.641 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:30:15.641 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "started", 2024-03-11T21:30:15.641 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:29:53.284267+0000", 2024-03-11T21:30:15.641 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.000117298 2024-03-11T21:30:15.641 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:30:15.641 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:30:15.641 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "waiting for subops from 0,3", 2024-03-11T21:30:15.641 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:29:53.284390+0000", 2024-03-11T21:30:15.641 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.000122924 2024-03-11T21:30:15.641 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:30:15.641 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:30:15.641 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "op_commit", 2024-03-11T21:30:15.641 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:29:53.287278+0000", 2024-03-11T21:30:15.641 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.0028878670000000001 2024-03-11T21:30:15.642 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:30:15.642 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:30:15.642 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "sub_op_commit_rec", 2024-03-11T21:30:15.642 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:29:53.298314+0000", 2024-03-11T21:30:15.642 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.011035794 2024-03-11T21:30:15.642 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:30:15.642 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:30:15.642 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "sub_op_commit_rec", 2024-03-11T21:30:15.642 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:29:53.420307+0000", 2024-03-11T21:30:15.642 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.121992795 2024-03-11T21:30:15.642 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:30:15.642 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:30:15.642 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "commit_sent", 2024-03-11T21:30:15.642 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:29:53.420349+0000", 2024-03-11T21:30:15.642 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 4.2508e-05 2024-03-11T21:30:15.643 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:30:15.643 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:30:15.643 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "done", 2024-03-11T21:30:15.643 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:29:53.420389+0000", 2024-03-11T21:30:15.643 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 3.9947000000000002e-05 2024-03-11T21:30:15.643 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-03-11T21:30:15.643 INFO:teuthology.orchestra.run.smithi097.stdout: ] 2024-03-11T21:30:15.643 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-03-11T21:30:15.643 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:30:15.643 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:30:15.643 INFO:teuthology.orchestra.run.smithi097.stdout: "description": "osd_op(client.34715.0:154121 7.7 7:e1e262da:::benchmark_data_smithi092_76040_object32210:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e126)", 2024-03-11T21:30:15.643 INFO:teuthology.orchestra.run.smithi097.stdout: "initiated_at": "2024-03-11T21:29:53.656871+0000", 2024-03-11T21:30:15.643 INFO:teuthology.orchestra.run.smithi097.stdout: "age": 21.930714692999999, 2024-03-11T21:30:15.644 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.12801322900000001, 2024-03-11T21:30:15.644 INFO:teuthology.orchestra.run.smithi097.stdout: "type_data": { 2024-03-11T21:30:15.644 INFO:teuthology.orchestra.run.smithi097.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-11T21:30:15.644 INFO:teuthology.orchestra.run.smithi097.stdout: "client_info": { 2024-03-11T21:30:15.644 INFO:teuthology.orchestra.run.smithi097.stdout: "client": "client.34715", 2024-03-11T21:30:15.644 INFO:teuthology.orchestra.run.smithi097.stdout: "client_addr": "v1:172.21.15.92:0/2403132573", 2024-03-11T21:30:15.644 INFO:teuthology.orchestra.run.smithi097.stdout: "tid": 154121 2024-03-11T21:30:15.644 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:30:15.644 INFO:teuthology.orchestra.run.smithi097.stdout: "events": [ 2024-03-11T21:30:15.644 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:30:15.644 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "initiated", 2024-03-11T21:30:15.644 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:29:53.656871+0000", 2024-03-11T21:30:15.644 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-03-11T21:30:15.644 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:30:15.644 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:30:15.645 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "header_read", 2024-03-11T21:30:15.645 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:29:53.656871+0000", 2024-03-11T21:30:15.645 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-03-11T21:30:15.645 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:30:15.645 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:30:15.645 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "throttled", 2024-03-11T21:30:15.645 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:29:53.656872+0000", 2024-03-11T21:30:15.645 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.302e-06 2024-03-11T21:30:15.645 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:30:15.645 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:30:15.645 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "all_read", 2024-03-11T21:30:15.645 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:29:53.656877+0000", 2024-03-11T21:30:15.645 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 4.9819999999999999e-06 2024-03-11T21:30:15.645 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:30:15.645 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:30:15.646 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "dispatched", 2024-03-11T21:30:15.646 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:29:53.656889+0000", 2024-03-11T21:30:15.646 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.216e-05 2024-03-11T21:30:15.646 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:30:15.646 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:30:15.646 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "queued_for_pg", 2024-03-11T21:30:15.646 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:29:53.656897+0000", 2024-03-11T21:30:15.646 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 7.8189999999999992e-06 2024-03-11T21:30:15.646 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:30:15.646 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:30:15.646 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "reached_pg", 2024-03-11T21:30:15.646 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:29:53.656934+0000", 2024-03-11T21:30:15.646 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 3.7115e-05 2024-03-11T21:30:15.646 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:30:15.647 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:30:15.647 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "started", 2024-03-11T21:30:15.647 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:29:53.657006+0000", 2024-03-11T21:30:15.647 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 7.1894e-05 2024-03-11T21:30:15.647 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:30:15.647 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:30:15.647 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "waiting for subops from 0,3", 2024-03-11T21:30:15.647 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:29:53.657082+0000", 2024-03-11T21:30:15.647 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 7.5964000000000007e-05 2024-03-11T21:30:15.647 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:30:15.647 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:30:15.647 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "op_commit", 2024-03-11T21:30:15.647 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:29:53.659524+0000", 2024-03-11T21:30:15.647 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.0024426019999999999 2024-03-11T21:30:15.648 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:30:15.648 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:30:15.648 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "sub_op_commit_rec", 2024-03-11T21:30:15.648 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:29:53.664310+0000", 2024-03-11T21:30:15.648 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.0047856779999999998 2024-03-11T21:30:15.648 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:30:15.648 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:30:15.648 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "sub_op_commit_rec", 2024-03-11T21:30:15.648 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:29:53.784824+0000", 2024-03-11T21:30:15.648 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.12051442699999999 2024-03-11T21:30:15.648 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:30:15.648 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:30:15.648 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "commit_sent", 2024-03-11T21:30:15.648 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:29:53.784855+0000", 2024-03-11T21:30:15.648 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 3.0391000000000001e-05 2024-03-11T21:30:15.649 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:30:15.649 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:30:15.649 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "done", 2024-03-11T21:30:15.649 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:29:53.784884+0000", 2024-03-11T21:30:15.649 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 2.8895e-05 2024-03-11T21:30:15.649 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-03-11T21:30:15.649 INFO:teuthology.orchestra.run.smithi097.stdout: ] 2024-03-11T21:30:15.649 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-03-11T21:30:15.649 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:30:15.649 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:30:15.649 INFO:teuthology.orchestra.run.smithi097.stdout: "description": "osd_op(client.34715.0:156176 7.7 7:eabb1b87:::benchmark_data_smithi092_76040_object34265:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e126)", 2024-03-11T21:30:15.649 INFO:teuthology.orchestra.run.smithi097.stdout: "initiated_at": "2024-03-11T21:29:56.529603+0000", 2024-03-11T21:30:15.649 INFO:teuthology.orchestra.run.smithi097.stdout: "age": 19.057982402, 2024-03-11T21:30:15.649 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.157669753, 2024-03-11T21:30:15.650 INFO:teuthology.orchestra.run.smithi097.stdout: "type_data": { 2024-03-11T21:30:15.650 INFO:teuthology.orchestra.run.smithi097.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-11T21:30:15.650 INFO:teuthology.orchestra.run.smithi097.stdout: "client_info": { 2024-03-11T21:30:15.650 INFO:teuthology.orchestra.run.smithi097.stdout: "client": "client.34715", 2024-03-11T21:30:15.650 INFO:teuthology.orchestra.run.smithi097.stdout: "client_addr": "v1:172.21.15.92:0/2403132573", 2024-03-11T21:30:15.650 INFO:teuthology.orchestra.run.smithi097.stdout: "tid": 156176 2024-03-11T21:30:15.650 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:30:15.650 INFO:teuthology.orchestra.run.smithi097.stdout: "events": [ 2024-03-11T21:30:15.650 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:30:15.650 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "initiated", 2024-03-11T21:30:15.650 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:29:56.529603+0000", 2024-03-11T21:30:15.650 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-03-11T21:30:15.650 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:30:15.650 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:30:15.650 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "header_read", 2024-03-11T21:30:15.651 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:29:56.529603+0000", 2024-03-11T21:30:15.651 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-03-11T21:30:15.651 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:30:15.651 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:30:15.651 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "throttled", 2024-03-11T21:30:15.651 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:29:56.529604+0000", 2024-03-11T21:30:15.651 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.57e-06 2024-03-11T21:30:15.651 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:30:15.651 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:30:15.651 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "all_read", 2024-03-11T21:30:15.651 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:29:56.529609+0000", 2024-03-11T21:30:15.651 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 4.561e-06 2024-03-11T21:30:15.651 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:30:15.651 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:30:15.651 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "dispatched", 2024-03-11T21:30:15.652 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:29:56.529619+0000", 2024-03-11T21:30:15.652 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 9.5880000000000004e-06 2024-03-11T21:30:15.652 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:30:15.652 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:30:15.652 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "queued_for_pg", 2024-03-11T21:30:15.652 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:29:56.529626+0000", 2024-03-11T21:30:15.652 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 7.2429999999999999e-06 2024-03-11T21:30:15.652 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:30:15.652 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:30:15.652 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "reached_pg", 2024-03-11T21:30:15.652 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:29:56.529687+0000", 2024-03-11T21:30:15.652 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 6.1316999999999996e-05 2024-03-11T21:30:15.652 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:30:15.652 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:30:15.652 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "started", 2024-03-11T21:30:15.653 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:29:56.529851+0000", 2024-03-11T21:30:15.653 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.000163802 2024-03-11T21:30:15.653 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:30:15.653 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:30:15.653 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "waiting for subops from 0,3", 2024-03-11T21:30:15.653 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:29:56.529986+0000", 2024-03-11T21:30:15.653 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.00013493700000000001 2024-03-11T21:30:15.653 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:30:15.653 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:30:15.653 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "op_commit", 2024-03-11T21:30:15.653 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:29:56.533514+0000", 2024-03-11T21:30:15.653 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.0035286390000000001 2024-03-11T21:30:15.653 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:30:15.653 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:30:15.653 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "sub_op_commit_rec", 2024-03-11T21:30:15.654 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:29:56.533820+0000", 2024-03-11T21:30:15.654 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.00030585699999999998 2024-03-11T21:30:15.654 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:30:15.654 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:30:15.654 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "sub_op_commit_rec", 2024-03-11T21:30:15.654 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:29:56.687203+0000", 2024-03-11T21:30:15.654 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.153382565 2024-03-11T21:30:15.654 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:30:15.654 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:30:15.654 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "commit_sent", 2024-03-11T21:30:15.654 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:29:56.687238+0000", 2024-03-11T21:30:15.654 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 3.4650000000000002e-05 2024-03-11T21:30:15.654 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:30:15.654 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:30:15.655 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "done", 2024-03-11T21:30:15.655 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:29:56.687273+0000", 2024-03-11T21:30:15.655 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 3.5024000000000002e-05 2024-03-11T21:30:15.655 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-03-11T21:30:15.655 INFO:teuthology.orchestra.run.smithi097.stdout: ] 2024-03-11T21:30:15.655 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-03-11T21:30:15.655 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:30:15.655 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:30:15.655 INFO:teuthology.orchestra.run.smithi097.stdout: "description": "osd_op(client.34715.0:156470 7.7 7:e13ffbf0:::benchmark_data_smithi092_76040_object34559:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e126)", 2024-03-11T21:30:15.655 INFO:teuthology.orchestra.run.smithi097.stdout: "initiated_at": "2024-03-11T21:29:56.990545+0000", 2024-03-11T21:30:15.655 INFO:teuthology.orchestra.run.smithi097.stdout: "age": 18.597039901999999, 2024-03-11T21:30:15.655 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.12930302799999999, 2024-03-11T21:30:15.655 INFO:teuthology.orchestra.run.smithi097.stdout: "type_data": { 2024-03-11T21:30:15.655 INFO:teuthology.orchestra.run.smithi097.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-11T21:30:15.657 INFO:teuthology.orchestra.run.smithi097.stdout: "client_info": { 2024-03-11T21:30:15.657 INFO:teuthology.orchestra.run.smithi097.stdout: "client": "client.34715", 2024-03-11T21:30:15.657 INFO:teuthology.orchestra.run.smithi097.stdout: "client_addr": "v1:172.21.15.92:0/2403132573", 2024-03-11T21:30:15.657 INFO:teuthology.orchestra.run.smithi097.stdout: "tid": 156470 2024-03-11T21:30:15.657 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:30:15.657 INFO:teuthology.orchestra.run.smithi097.stdout: "events": [ 2024-03-11T21:30:15.657 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:30:15.657 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "initiated", 2024-03-11T21:30:15.657 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:29:56.990545+0000", 2024-03-11T21:30:15.657 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-03-11T21:30:15.658 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:30:15.658 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:30:15.658 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "header_read", 2024-03-11T21:30:15.658 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:29:56.990545+0000", 2024-03-11T21:30:15.658 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-03-11T21:30:15.658 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:30:15.658 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:30:15.658 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "throttled", 2024-03-11T21:30:15.658 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:29:56.990547+0000", 2024-03-11T21:30:15.658 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.9240000000000001e-06 2024-03-11T21:30:15.658 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:30:15.658 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:30:15.658 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "all_read", 2024-03-11T21:30:15.659 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:29:56.990553+0000", 2024-03-11T21:30:15.659 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 5.3199999999999999e-06 2024-03-11T21:30:15.659 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:30:15.659 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:30:15.659 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "dispatched", 2024-03-11T21:30:15.659 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:29:56.990565+0000", 2024-03-11T21:30:15.659 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.2055999999999999e-05 2024-03-11T21:30:15.659 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:30:15.659 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:30:15.659 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "queued_for_pg", 2024-03-11T21:30:15.659 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:29:56.990572+0000", 2024-03-11T21:30:15.659 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 7.8669999999999994e-06 2024-03-11T21:30:15.659 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:30:15.659 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:30:15.659 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "reached_pg", 2024-03-11T21:30:15.660 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:29:56.990618+0000", 2024-03-11T21:30:15.660 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 4.5118999999999999e-05 2024-03-11T21:30:15.660 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:30:15.660 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:30:15.660 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "started", 2024-03-11T21:30:15.660 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:29:56.990769+0000", 2024-03-11T21:30:15.660 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.00015170200000000001 2024-03-11T21:30:15.660 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:30:15.660 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:30:15.660 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "waiting for subops from 0,3", 2024-03-11T21:30:15.660 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:29:56.990885+0000", 2024-03-11T21:30:15.660 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.00011616399999999999 2024-03-11T21:30:15.660 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:30:15.660 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:30:15.661 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "op_commit", 2024-03-11T21:30:15.661 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:29:56.993283+0000", 2024-03-11T21:30:15.661 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.0023971019999999999 2024-03-11T21:30:15.661 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:30:15.661 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:30:15.661 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "sub_op_commit_rec", 2024-03-11T21:30:15.661 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:29:57.043680+0000", 2024-03-11T21:30:15.661 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.050397191000000001 2024-03-11T21:30:15.661 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:30:15.661 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:30:15.661 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "sub_op_commit_rec", 2024-03-11T21:30:15.661 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:29:57.119764+0000", 2024-03-11T21:30:15.661 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.076084497000000001 2024-03-11T21:30:15.661 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:30:15.662 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:30:15.662 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "commit_sent", 2024-03-11T21:30:15.662 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:29:57.119812+0000", 2024-03-11T21:30:15.662 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 4.7991999999999997e-05 2024-03-11T21:30:15.662 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:30:15.662 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:30:15.662 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "done", 2024-03-11T21:30:15.662 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:29:57.119848+0000", 2024-03-11T21:30:15.662 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 3.6093999999999997e-05 2024-03-11T21:30:15.662 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-03-11T21:30:15.662 INFO:teuthology.orchestra.run.smithi097.stdout: ] 2024-03-11T21:30:15.662 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-03-11T21:30:15.662 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:30:15.662 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:30:15.662 INFO:teuthology.orchestra.run.smithi097.stdout: "description": "osd_op(client.34715.0:156595 7.7 7:efb16ea0:::benchmark_data_smithi092_76040_object34684:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e126)", 2024-03-11T21:30:15.663 INFO:teuthology.orchestra.run.smithi097.stdout: "initiated_at": "2024-03-11T21:29:57.229266+0000", 2024-03-11T21:30:15.663 INFO:teuthology.orchestra.run.smithi097.stdout: "age": 18.358319115, 2024-03-11T21:30:15.663 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.138283932, 2024-03-11T21:30:15.663 INFO:teuthology.orchestra.run.smithi097.stdout: "type_data": { 2024-03-11T21:30:15.663 INFO:teuthology.orchestra.run.smithi097.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-11T21:30:15.663 INFO:teuthology.orchestra.run.smithi097.stdout: "client_info": { 2024-03-11T21:30:15.663 INFO:teuthology.orchestra.run.smithi097.stdout: "client": "client.34715", 2024-03-11T21:30:15.663 INFO:teuthology.orchestra.run.smithi097.stdout: "client_addr": "v1:172.21.15.92:0/2403132573", 2024-03-11T21:30:15.663 INFO:teuthology.orchestra.run.smithi097.stdout: "tid": 156595 2024-03-11T21:30:15.663 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:30:15.663 INFO:teuthology.orchestra.run.smithi097.stdout: "events": [ 2024-03-11T21:30:15.663 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:30:15.663 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "initiated", 2024-03-11T21:30:15.663 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:29:57.229266+0000", 2024-03-11T21:30:15.664 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-03-11T21:30:15.664 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:30:15.664 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:30:15.664 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "header_read", 2024-03-11T21:30:15.664 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:29:57.229266+0000", 2024-03-11T21:30:15.664 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-03-11T21:30:15.664 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:30:15.664 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:30:15.664 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "throttled", 2024-03-11T21:30:15.664 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:29:57.229267+0000", 2024-03-11T21:30:15.664 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 9.0599999999999999e-07 2024-03-11T21:30:15.664 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:30:15.664 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:30:15.664 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "all_read", 2024-03-11T21:30:15.664 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:29:57.229271+0000", 2024-03-11T21:30:15.665 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 3.6720000000000002e-06 2024-03-11T21:30:15.665 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:30:15.665 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:30:15.665 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "dispatched", 2024-03-11T21:30:15.665 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:29:57.229280+0000", 2024-03-11T21:30:15.665 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 9.1200000000000008e-06 2024-03-11T21:30:15.665 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:30:15.665 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:30:15.665 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "queued_for_pg", 2024-03-11T21:30:15.665 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:29:57.229287+0000", 2024-03-11T21:30:15.665 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 7.7200000000000006e-06 2024-03-11T21:30:15.665 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:30:15.665 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:30:15.665 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "reached_pg", 2024-03-11T21:30:15.665 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:29:57.229331+0000", 2024-03-11T21:30:15.666 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 4.3940999999999998e-05 2024-03-11T21:30:15.666 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:30:15.666 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:30:15.666 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "started", 2024-03-11T21:30:15.666 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:29:57.229410+0000", 2024-03-11T21:30:15.666 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 7.8553000000000005e-05 2024-03-11T21:30:15.666 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:30:15.666 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:30:15.666 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "waiting for subops from 0,3", 2024-03-11T21:30:15.666 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:29:57.229481+0000", 2024-03-11T21:30:15.666 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 7.0650999999999998e-05 2024-03-11T21:30:15.666 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:30:15.666 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:30:15.666 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "op_commit", 2024-03-11T21:30:15.666 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:29:57.231663+0000", 2024-03-11T21:30:15.667 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.002181887 2024-03-11T21:30:15.667 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:30:15.667 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:30:15.667 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "sub_op_commit_rec", 2024-03-11T21:30:15.667 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:29:57.253437+0000", 2024-03-11T21:30:15.667 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.021774216999999998 2024-03-11T21:30:15.667 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:30:15.667 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:30:15.667 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "sub_op_commit_rec", 2024-03-11T21:30:15.667 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:29:57.367448+0000", 2024-03-11T21:30:15.667 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.114011184 2024-03-11T21:30:15.667 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:30:15.667 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:30:15.667 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "commit_sent", 2024-03-11T21:30:15.667 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:29:57.367505+0000", 2024-03-11T21:30:15.668 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 5.6657999999999999e-05 2024-03-11T21:30:15.668 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:30:15.668 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:30:15.668 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "done", 2024-03-11T21:30:15.668 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:29:57.367550+0000", 2024-03-11T21:30:15.668 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 4.5423000000000002e-05 2024-03-11T21:30:15.668 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-03-11T21:30:15.668 INFO:teuthology.orchestra.run.smithi097.stdout: ] 2024-03-11T21:30:15.668 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-03-11T21:30:15.668 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:30:15.668 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:30:15.668 INFO:teuthology.orchestra.run.smithi097.stdout: "description": "osd_op(client.34715.0:156683 7.7 7:ec65b82b:::benchmark_data_smithi092_76040_object34772:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e126)", 2024-03-11T21:30:15.668 INFO:teuthology.orchestra.run.smithi097.stdout: "initiated_at": "2024-03-11T21:29:57.397512+0000", 2024-03-11T21:30:15.668 INFO:teuthology.orchestra.run.smithi097.stdout: "age": 18.190073379000001, 2024-03-11T21:30:15.669 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.15765780400000001, 2024-03-11T21:30:15.669 INFO:teuthology.orchestra.run.smithi097.stdout: "type_data": { 2024-03-11T21:30:15.669 INFO:teuthology.orchestra.run.smithi097.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-11T21:30:15.669 INFO:teuthology.orchestra.run.smithi097.stdout: "client_info": { 2024-03-11T21:30:15.669 INFO:teuthology.orchestra.run.smithi097.stdout: "client": "client.34715", 2024-03-11T21:30:15.669 INFO:teuthology.orchestra.run.smithi097.stdout: "client_addr": "v1:172.21.15.92:0/2403132573", 2024-03-11T21:30:15.669 INFO:teuthology.orchestra.run.smithi097.stdout: "tid": 156683 2024-03-11T21:30:15.669 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:30:15.669 INFO:teuthology.orchestra.run.smithi097.stdout: "events": [ 2024-03-11T21:30:15.669 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:30:15.669 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "initiated", 2024-03-11T21:30:15.669 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:29:57.397512+0000", 2024-03-11T21:30:15.669 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-03-11T21:30:15.669 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:30:15.670 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:30:15.670 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "header_read", 2024-03-11T21:30:15.670 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:29:57.397512+0000", 2024-03-11T21:30:15.670 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-03-11T21:30:15.670 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:30:15.670 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:30:15.670 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "throttled", 2024-03-11T21:30:15.670 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:29:57.397513+0000", 2024-03-11T21:30:15.670 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 9.7600000000000006e-07 2024-03-11T21:30:15.670 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:30:15.670 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:30:15.670 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "all_read", 2024-03-11T21:30:15.670 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:29:57.397516+0000", 2024-03-11T21:30:15.670 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 3.608e-06 2024-03-11T21:30:15.671 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:30:15.671 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:30:15.671 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "dispatched", 2024-03-11T21:30:15.671 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:29:57.397525+0000", 2024-03-11T21:30:15.671 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 8.3459999999999994e-06 2024-03-11T21:30:15.671 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:30:15.671 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:30:15.671 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "queued_for_pg", 2024-03-11T21:30:15.671 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:29:57.397530+0000", 2024-03-11T21:30:15.671 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 5.164e-06 2024-03-11T21:30:15.671 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:30:15.671 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:30:15.671 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "reached_pg", 2024-03-11T21:30:15.671 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:29:57.401870+0000", 2024-03-11T21:30:15.672 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.004340259 2024-03-11T21:30:15.672 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:30:15.672 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:30:15.672 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "started", 2024-03-11T21:30:15.672 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:29:57.401987+0000", 2024-03-11T21:30:15.672 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.000116589 2024-03-11T21:30:15.672 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:30:15.672 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:30:15.672 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "waiting for subops from 0,3", 2024-03-11T21:30:15.672 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:29:57.402100+0000", 2024-03-11T21:30:15.672 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.000113542 2024-03-11T21:30:15.672 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:30:15.672 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:30:15.672 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "op_commit", 2024-03-11T21:30:15.672 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:29:57.405258+0000", 2024-03-11T21:30:15.673 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.0031580279999999998 2024-03-11T21:30:15.673 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:30:15.673 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:30:15.673 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "sub_op_commit_rec", 2024-03-11T21:30:15.673 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:29:57.429205+0000", 2024-03-11T21:30:15.673 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.023946505 2024-03-11T21:30:15.673 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:30:15.673 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:30:15.673 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "sub_op_commit_rec", 2024-03-11T21:30:15.673 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:29:57.555099+0000", 2024-03-11T21:30:15.673 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.12589394600000001 2024-03-11T21:30:15.673 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:30:15.673 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:30:15.673 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "commit_sent", 2024-03-11T21:30:15.674 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:29:57.555136+0000", 2024-03-11T21:30:15.674 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 3.7712000000000001e-05 2024-03-11T21:30:15.674 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:30:15.674 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:30:15.674 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "done", 2024-03-11T21:30:15.674 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:29:57.555170+0000", 2024-03-11T21:30:15.674 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 3.3129000000000001e-05 2024-03-11T21:30:15.674 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-03-11T21:30:15.674 INFO:teuthology.orchestra.run.smithi097.stdout: ] 2024-03-11T21:30:15.674 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-03-11T21:30:15.674 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:30:15.674 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:30:15.674 INFO:teuthology.orchestra.run.smithi097.stdout: "description": "osd_op(client.34715.0:156720 7.7 7:e8c159b2:::benchmark_data_smithi092_76040_object34809:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e126)", 2024-03-11T21:30:15.674 INFO:teuthology.orchestra.run.smithi097.stdout: "initiated_at": "2024-03-11T21:29:57.455164+0000", 2024-03-11T21:30:15.675 INFO:teuthology.orchestra.run.smithi097.stdout: "age": 18.132421016999999, 2024-03-11T21:30:15.675 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.13071795999999999, 2024-03-11T21:30:15.675 INFO:teuthology.orchestra.run.smithi097.stdout: "type_data": { 2024-03-11T21:30:15.675 INFO:teuthology.orchestra.run.smithi097.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-11T21:30:15.675 INFO:teuthology.orchestra.run.smithi097.stdout: "client_info": { 2024-03-11T21:30:15.675 INFO:teuthology.orchestra.run.smithi097.stdout: "client": "client.34715", 2024-03-11T21:30:15.675 INFO:teuthology.orchestra.run.smithi097.stdout: "client_addr": "v1:172.21.15.92:0/2403132573", 2024-03-11T21:30:15.675 INFO:teuthology.orchestra.run.smithi097.stdout: "tid": 156720 2024-03-11T21:30:15.675 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:30:15.675 INFO:teuthology.orchestra.run.smithi097.stdout: "events": [ 2024-03-11T21:30:15.675 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:30:15.675 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "initiated", 2024-03-11T21:30:15.676 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:29:57.455164+0000", 2024-03-11T21:30:15.676 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-03-11T21:30:15.676 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:30:15.676 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:30:15.676 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "header_read", 2024-03-11T21:30:15.676 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:29:57.455164+0000", 2024-03-11T21:30:15.676 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-03-11T21:30:15.676 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:30:15.676 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:30:15.676 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "throttled", 2024-03-11T21:30:15.676 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:29:57.455165+0000", 2024-03-11T21:30:15.676 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.1200000000000001e-06 2024-03-11T21:30:15.676 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:30:15.676 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:30:15.677 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "all_read", 2024-03-11T21:30:15.677 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:29:57.455170+0000", 2024-03-11T21:30:15.677 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 4.4299999999999999e-06 2024-03-11T21:30:15.677 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:30:15.677 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:30:15.677 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "dispatched", 2024-03-11T21:30:15.677 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:29:57.455178+0000", 2024-03-11T21:30:15.677 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 8.5350000000000004e-06 2024-03-11T21:30:15.677 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:30:15.677 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:30:15.677 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "queued_for_pg", 2024-03-11T21:30:15.677 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:29:57.455185+0000", 2024-03-11T21:30:15.677 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 6.6579999999999996e-06 2024-03-11T21:30:15.677 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:30:15.677 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:30:15.678 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "reached_pg", 2024-03-11T21:30:15.678 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:29:57.455237+0000", 2024-03-11T21:30:15.678 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 5.1678000000000002e-05 2024-03-11T21:30:15.678 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:30:15.678 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:30:15.678 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "started", 2024-03-11T21:30:15.678 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:29:57.455332+0000", 2024-03-11T21:30:15.678 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 9.5357000000000002e-05 2024-03-11T21:30:15.678 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:30:15.678 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:30:15.678 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "waiting for subops from 0,3", 2024-03-11T21:30:15.678 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:29:57.455458+0000", 2024-03-11T21:30:15.678 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.00012579299999999999 2024-03-11T21:30:15.678 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:30:15.679 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:30:15.679 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "op_commit", 2024-03-11T21:30:15.679 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:29:57.458563+0000", 2024-03-11T21:30:15.679 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.0031057039999999999 2024-03-11T21:30:15.679 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:30:15.679 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:30:15.679 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "sub_op_commit_rec", 2024-03-11T21:30:15.679 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:29:57.492724+0000", 2024-03-11T21:30:15.679 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.034160596000000001 2024-03-11T21:30:15.679 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:30:15.679 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:30:15.679 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "sub_op_commit_rec", 2024-03-11T21:30:15.679 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:29:57.585777+0000", 2024-03-11T21:30:15.679 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.093052977999999995 2024-03-11T21:30:15.679 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:30:15.680 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:30:15.680 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "commit_sent", 2024-03-11T21:30:15.680 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:29:57.585832+0000", 2024-03-11T21:30:15.680 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 5.5331999999999998e-05 2024-03-11T21:30:15.680 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:30:15.680 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:30:15.680 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "done", 2024-03-11T21:30:15.680 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:29:57.585882+0000", 2024-03-11T21:30:15.680 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 4.9778999999999999e-05 2024-03-11T21:30:15.680 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-03-11T21:30:15.680 INFO:teuthology.orchestra.run.smithi097.stdout: ] 2024-03-11T21:30:15.680 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-03-11T21:30:15.680 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:30:15.680 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:30:15.680 INFO:teuthology.orchestra.run.smithi097.stdout: "description": "osd_op(client.34715.0:156910 7.7 7:ed8c3ad9:::benchmark_data_smithi092_76040_object34999:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e126)", 2024-03-11T21:30:15.681 INFO:teuthology.orchestra.run.smithi097.stdout: "initiated_at": "2024-03-11T21:29:57.825647+0000", 2024-03-11T21:30:15.681 INFO:teuthology.orchestra.run.smithi097.stdout: "age": 17.761938241999999, 2024-03-11T21:30:15.681 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.13931633800000001, 2024-03-11T21:30:15.681 INFO:teuthology.orchestra.run.smithi097.stdout: "type_data": { 2024-03-11T21:30:15.681 INFO:teuthology.orchestra.run.smithi097.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-11T21:30:15.681 INFO:teuthology.orchestra.run.smithi097.stdout: "client_info": { 2024-03-11T21:30:15.681 INFO:teuthology.orchestra.run.smithi097.stdout: "client": "client.34715", 2024-03-11T21:30:15.681 INFO:teuthology.orchestra.run.smithi097.stdout: "client_addr": "v1:172.21.15.92:0/2403132573", 2024-03-11T21:30:15.681 INFO:teuthology.orchestra.run.smithi097.stdout: "tid": 156910 2024-03-11T21:30:15.681 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:30:15.681 INFO:teuthology.orchestra.run.smithi097.stdout: "events": [ 2024-03-11T21:30:15.681 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:30:15.681 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "initiated", 2024-03-11T21:30:15.681 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:29:57.825647+0000", 2024-03-11T21:30:15.681 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-03-11T21:30:15.682 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:30:15.682 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:30:15.682 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "header_read", 2024-03-11T21:30:15.682 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:29:57.825647+0000", 2024-03-11T21:30:15.682 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-03-11T21:30:15.682 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:30:15.682 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:30:15.682 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "throttled", 2024-03-11T21:30:15.682 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:29:57.825649+0000", 2024-03-11T21:30:15.682 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 2.0719999999999998e-06 2024-03-11T21:30:15.682 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:30:15.682 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:30:15.682 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "all_read", 2024-03-11T21:30:15.682 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:29:57.825655+0000", 2024-03-11T21:30:15.683 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 6.2140000000000001e-06 2024-03-11T21:30:15.683 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:30:15.683 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:30:15.683 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "dispatched", 2024-03-11T21:30:15.683 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:29:57.825669+0000", 2024-03-11T21:30:15.683 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.3509999999999999e-05 2024-03-11T21:30:15.683 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:30:15.683 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:30:15.683 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "queued_for_pg", 2024-03-11T21:30:15.683 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:29:57.825676+0000", 2024-03-11T21:30:15.683 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 6.8460000000000002e-06 2024-03-11T21:30:15.683 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:30:15.683 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:30:15.683 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "reached_pg", 2024-03-11T21:30:15.683 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:29:57.825730+0000", 2024-03-11T21:30:15.684 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 5.4211e-05 2024-03-11T21:30:15.684 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:30:15.684 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:30:15.684 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "started", 2024-03-11T21:30:15.684 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:29:57.825869+0000", 2024-03-11T21:30:15.684 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.000139574 2024-03-11T21:30:15.684 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:30:15.684 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:30:15.684 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "waiting for subops from 0,3", 2024-03-11T21:30:15.684 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:29:57.825987+0000", 2024-03-11T21:30:15.684 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.00011783299999999999 2024-03-11T21:30:15.684 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:30:15.684 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:30:15.684 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "op_commit", 2024-03-11T21:30:15.685 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:29:57.829193+0000", 2024-03-11T21:30:15.685 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.0032057159999999999 2024-03-11T21:30:15.685 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:30:15.685 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:30:15.685 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "sub_op_commit_rec", 2024-03-11T21:30:15.685 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:29:57.862874+0000", 2024-03-11T21:30:15.685 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.033680871000000001 2024-03-11T21:30:15.685 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:30:15.685 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:30:15.685 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "sub_op_commit_rec", 2024-03-11T21:30:15.685 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:29:57.964899+0000", 2024-03-11T21:30:15.685 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.102024935 2024-03-11T21:30:15.685 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:30:15.685 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:30:15.685 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "commit_sent", 2024-03-11T21:30:15.686 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:29:57.964930+0000", 2024-03-11T21:30:15.686 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 3.1390999999999998e-05 2024-03-11T21:30:15.686 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:30:15.686 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:30:15.686 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "done", 2024-03-11T21:30:15.686 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:29:57.964963+0000", 2024-03-11T21:30:15.686 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 3.3164999999999998e-05 2024-03-11T21:30:15.686 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-03-11T21:30:15.686 INFO:teuthology.orchestra.run.smithi097.stdout: ] 2024-03-11T21:30:15.686 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-03-11T21:30:15.686 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:30:15.686 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:30:15.686 INFO:teuthology.orchestra.run.smithi097.stdout: "description": "osd_op(client.34715.0:157118 7.7 7:ef798aee:::benchmark_data_smithi092_76040_object35207:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e127)", 2024-03-11T21:30:15.686 INFO:teuthology.orchestra.run.smithi097.stdout: "initiated_at": "2024-03-11T21:29:58.129269+0000", 2024-03-11T21:30:15.687 INFO:teuthology.orchestra.run.smithi097.stdout: "age": 17.458315717000001, 2024-03-11T21:30:15.687 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.18586082100000001, 2024-03-11T21:30:15.687 INFO:teuthology.orchestra.run.smithi097.stdout: "type_data": { 2024-03-11T21:30:15.687 INFO:teuthology.orchestra.run.smithi097.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-11T21:30:15.687 INFO:teuthology.orchestra.run.smithi097.stdout: "client_info": { 2024-03-11T21:30:15.687 INFO:teuthology.orchestra.run.smithi097.stdout: "client": "client.34715", 2024-03-11T21:30:15.687 INFO:teuthology.orchestra.run.smithi097.stdout: "client_addr": "v1:172.21.15.92:0/2403132573", 2024-03-11T21:30:15.687 INFO:teuthology.orchestra.run.smithi097.stdout: "tid": 157118 2024-03-11T21:30:15.687 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:30:15.687 INFO:teuthology.orchestra.run.smithi097.stdout: "events": [ 2024-03-11T21:30:15.687 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:30:15.687 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "initiated", 2024-03-11T21:30:15.687 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:29:58.129269+0000", 2024-03-11T21:30:15.687 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-03-11T21:30:15.687 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:30:15.688 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:30:15.688 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "header_read", 2024-03-11T21:30:15.688 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:29:58.129269+0000", 2024-03-11T21:30:15.688 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-03-11T21:30:15.688 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:30:15.688 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:30:15.688 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "throttled", 2024-03-11T21:30:15.688 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:29:58.129271+0000", 2024-03-11T21:30:15.688 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.7519999999999999e-06 2024-03-11T21:30:15.688 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:30:15.688 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:30:15.688 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "all_read", 2024-03-11T21:30:15.688 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:29:58.129276+0000", 2024-03-11T21:30:15.688 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 5.1279999999999999e-06 2024-03-11T21:30:15.688 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:30:15.689 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:30:15.689 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "dispatched", 2024-03-11T21:30:15.689 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:29:58.129286+0000", 2024-03-11T21:30:15.689 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.0062e-05 2024-03-11T21:30:15.689 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:30:15.689 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:30:15.689 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "queued_for_pg", 2024-03-11T21:30:15.689 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:29:58.129292+0000", 2024-03-11T21:30:15.689 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 5.8769999999999998e-06 2024-03-11T21:30:15.689 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:30:15.689 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:30:15.689 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "reached_pg", 2024-03-11T21:30:15.689 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:29:58.129334+0000", 2024-03-11T21:30:15.689 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 4.1641000000000003e-05 2024-03-11T21:30:15.689 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:30:15.690 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:30:15.690 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "started", 2024-03-11T21:30:15.690 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:29:58.129426+0000", 2024-03-11T21:30:15.690 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 9.2511999999999998e-05 2024-03-11T21:30:15.690 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:30:15.690 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:30:15.690 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "waiting for subops from 0,3", 2024-03-11T21:30:15.690 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:29:58.129504+0000", 2024-03-11T21:30:15.690 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 7.7631000000000003e-05 2024-03-11T21:30:15.690 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:30:15.690 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:30:15.690 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "op_commit", 2024-03-11T21:30:15.690 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:29:58.132108+0000", 2024-03-11T21:30:15.690 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.0026044060000000001 2024-03-11T21:30:15.690 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:30:15.691 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:30:15.691 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "sub_op_commit_rec", 2024-03-11T21:30:15.691 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:29:58.149431+0000", 2024-03-11T21:30:15.691 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.017322596999999999 2024-03-11T21:30:15.691 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:30:15.691 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:30:15.691 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "sub_op_commit_rec", 2024-03-11T21:30:15.691 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:29:58.315001+0000", 2024-03-11T21:30:15.691 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.16557005799999999 2024-03-11T21:30:15.691 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:30:15.691 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:30:15.691 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "commit_sent", 2024-03-11T21:30:15.691 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:29:58.315060+0000", 2024-03-11T21:30:15.691 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 5.9039999999999997e-05 2024-03-11T21:30:15.691 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:30:15.692 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:30:15.692 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "done", 2024-03-11T21:30:15.692 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:29:58.315130+0000", 2024-03-11T21:30:15.692 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 7.0117000000000006e-05 2024-03-11T21:30:15.692 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-03-11T21:30:15.692 INFO:teuthology.orchestra.run.smithi097.stdout: ] 2024-03-11T21:30:15.692 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-03-11T21:30:15.692 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:30:15.692 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:30:15.692 INFO:teuthology.orchestra.run.smithi097.stdout: "description": "osd_op(client.34715.0:157129 7.7 7:e1f3a7b2:::benchmark_data_smithi092_76040_object35218:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e127)", 2024-03-11T21:30:15.692 INFO:teuthology.orchestra.run.smithi097.stdout: "initiated_at": "2024-03-11T21:29:58.156664+0000", 2024-03-11T21:30:15.692 INFO:teuthology.orchestra.run.smithi097.stdout: "age": 17.430921452, 2024-03-11T21:30:15.692 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.15858186099999999, 2024-03-11T21:30:15.692 INFO:teuthology.orchestra.run.smithi097.stdout: "type_data": { 2024-03-11T21:30:15.692 INFO:teuthology.orchestra.run.smithi097.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-11T21:30:15.693 INFO:teuthology.orchestra.run.smithi097.stdout: "client_info": { 2024-03-11T21:30:15.693 INFO:teuthology.orchestra.run.smithi097.stdout: "client": "client.34715", 2024-03-11T21:30:15.693 INFO:teuthology.orchestra.run.smithi097.stdout: "client_addr": "v1:172.21.15.92:0/2403132573", 2024-03-11T21:30:15.693 INFO:teuthology.orchestra.run.smithi097.stdout: "tid": 157129 2024-03-11T21:30:15.693 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:30:15.693 INFO:teuthology.orchestra.run.smithi097.stdout: "events": [ 2024-03-11T21:30:15.693 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:30:15.693 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "initiated", 2024-03-11T21:30:15.693 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:29:58.156664+0000", 2024-03-11T21:30:15.693 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-03-11T21:30:15.693 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:30:15.693 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:30:15.693 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "header_read", 2024-03-11T21:30:15.693 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:29:58.156664+0000", 2024-03-11T21:30:15.693 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-03-11T21:30:15.694 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:30:15.694 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:30:15.694 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "throttled", 2024-03-11T21:30:15.694 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:29:58.156666+0000", 2024-03-11T21:30:15.694 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 2.0640000000000001e-06 2024-03-11T21:30:15.694 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:30:15.694 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:30:15.694 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "all_read", 2024-03-11T21:30:15.694 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:29:58.156676+0000", 2024-03-11T21:30:15.694 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 9.9960000000000005e-06 2024-03-11T21:30:15.694 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:30:15.694 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:30:15.694 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "dispatched", 2024-03-11T21:30:15.694 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:29:58.156689+0000", 2024-03-11T21:30:15.694 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.3631e-05 2024-03-11T21:30:15.695 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:30:15.695 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:30:15.695 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "queued_for_pg", 2024-03-11T21:30:15.695 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:29:58.156698+0000", 2024-03-11T21:30:15.695 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 8.9290000000000004e-06 2024-03-11T21:30:15.695 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:30:15.695 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:30:15.695 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "reached_pg", 2024-03-11T21:30:15.695 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:29:58.156752+0000", 2024-03-11T21:30:15.695 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 5.3560000000000002e-05 2024-03-11T21:30:15.695 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:30:15.695 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:30:15.695 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "started", 2024-03-11T21:30:15.695 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:29:58.156844+0000", 2024-03-11T21:30:15.696 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 9.2029999999999998e-05 2024-03-11T21:30:15.696 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:30:15.696 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:30:15.696 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "waiting for subops from 0,3", 2024-03-11T21:30:15.696 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:29:58.156922+0000", 2024-03-11T21:30:15.696 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 7.7653999999999999e-05 2024-03-11T21:30:15.696 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:30:15.696 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:30:15.696 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "op_commit", 2024-03-11T21:30:15.696 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:29:58.159472+0000", 2024-03-11T21:30:15.696 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.0025506600000000002 2024-03-11T21:30:15.696 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:30:15.696 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:30:15.696 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "sub_op_commit_rec", 2024-03-11T21:30:15.696 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:29:58.177848+0000", 2024-03-11T21:30:15.697 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.01837538 2024-03-11T21:30:15.697 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:30:15.697 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:30:15.697 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "sub_op_commit_rec", 2024-03-11T21:30:15.697 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:29:58.315184+0000", 2024-03-11T21:30:15.697 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.13733662199999999 2024-03-11T21:30:15.697 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:30:15.697 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:30:15.697 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "commit_sent", 2024-03-11T21:30:15.697 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:29:58.315220+0000", 2024-03-11T21:30:15.697 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 3.5357e-05 2024-03-11T21:30:15.697 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:30:15.697 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:30:15.697 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "done", 2024-03-11T21:30:15.697 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:29:58.315246+0000", 2024-03-11T21:30:15.698 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 2.5978000000000001e-05 2024-03-11T21:30:15.698 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-03-11T21:30:15.698 INFO:teuthology.orchestra.run.smithi097.stdout: ] 2024-03-11T21:30:15.698 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-03-11T21:30:15.698 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-03-11T21:30:15.698 INFO:teuthology.orchestra.run.smithi097.stdout: ] 2024-03-11T21:30:15.698 INFO:teuthology.orchestra.run.smithi097.stdout:} 2024-03-11T21:30:15.698 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:30:15.728 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:30:15.728 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:30:15.729 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:30:15.829 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:30:15.856 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:30:15.857 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:30:15.857 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:30:15.958 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:30:16.003 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:30:16.003 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:30:16.004 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:30:16.104 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:30:16.147 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:30:16.148 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:30:16.148 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:30:16.249 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:30:16.270 INFO:teuthology.orchestra.run.smithi092.stderr:reweighted osd.2 to 1 (10000) 2024-03-11T21:30:16.299 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:30:16.300 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:30:16.300 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:30:16.401 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:30:16.441 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:30:16.442 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:30:16.443 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:30:16.543 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:30:16.576 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:30:16 smithi097 ceph-mon[33317]: 3.0 scrub starts 2024-03-11T21:30:16.576 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:30:16 smithi097 ceph-mon[33317]: 3.0 scrub ok 2024-03-11T21:30:16.577 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:30:16 smithi097 ceph-mon[33317]: 2.11 scrub starts 2024-03-11T21:30:16.577 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:30:16 smithi097 ceph-mon[33317]: 2.11 scrub ok 2024-03-11T21:30:16.577 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:30:16 smithi097 ceph-mon[33317]: 2.e scrub starts 2024-03-11T21:30:16.577 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:30:16 smithi097 ceph-mon[33317]: 2.e scrub ok 2024-03-11T21:30:16.577 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:30:16 smithi097 ceph-mon[33317]: pgmap v164: 148 pgs: 2 active+clean+scrubbing, 146 active+clean; 4.4 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.07k op/s 2024-03-11T21:30:16.577 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:30:16 smithi097 ceph-mon[33317]: from='client.? v1:172.21.15.92:0/1464198237' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]: dispatch 2024-03-11T21:30:16.598 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:30:16.598 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:30:16.598 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:30:16.610 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:30:16 smithi092 ceph-mon[29572]: 3.0 scrub starts 2024-03-11T21:30:16.611 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:30:16 smithi092 ceph-mon[29572]: 3.0 scrub ok 2024-03-11T21:30:16.611 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:30:16 smithi092 ceph-mon[29572]: 2.11 scrub starts 2024-03-11T21:30:16.611 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:30:16 smithi092 ceph-mon[29572]: 2.11 scrub ok 2024-03-11T21:30:16.611 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:30:16 smithi092 ceph-mon[29572]: 2.e scrub starts 2024-03-11T21:30:16.611 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:30:16 smithi092 ceph-mon[29572]: 2.e scrub ok 2024-03-11T21:30:16.611 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:30:16 smithi092 ceph-mon[29572]: pgmap v164: 148 pgs: 2 active+clean+scrubbing, 146 active+clean; 4.4 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.07k op/s 2024-03-11T21:30:16.611 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:30:16 smithi092 ceph-mon[29572]: from='client.? v1:172.21.15.92:0/1464198237' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]: dispatch 2024-03-11T21:30:16.612 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:30:16 smithi092 ceph-mon[37563]: 3.0 scrub starts 2024-03-11T21:30:16.612 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:30:16 smithi092 ceph-mon[37563]: 3.0 scrub ok 2024-03-11T21:30:16.612 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:30:16 smithi092 ceph-mon[37563]: 2.11 scrub starts 2024-03-11T21:30:16.612 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:30:16 smithi092 ceph-mon[37563]: 2.11 scrub ok 2024-03-11T21:30:16.612 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:30:16 smithi092 ceph-mon[37563]: 2.e scrub starts 2024-03-11T21:30:16.612 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:30:16 smithi092 ceph-mon[37563]: 2.e scrub ok 2024-03-11T21:30:16.612 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:30:16 smithi092 ceph-mon[37563]: pgmap v164: 148 pgs: 2 active+clean+scrubbing, 146 active+clean; 4.4 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.07k op/s 2024-03-11T21:30:16.612 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:30:16 smithi092 ceph-mon[37563]: from='client.? v1:172.21.15.92:0/1464198237' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]: dispatch 2024-03-11T21:30:16.699 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:30:16.727 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:30:16.728 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:30:16.728 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:30:16.758 DEBUG:teuthology.orchestra.run.smithi097:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid 00db0016-dfed-11ee-95c8-87774f69a715 -- ceph daemon osd.7 dump_ops_in_flight 2024-03-11T21:30:16.829 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:30:16.868 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:30:16.868 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:30:16.868 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:30:16.969 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:30:17.003 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:30:17.003 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:30:17.004 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:30:17.079 INFO:teuthology.orchestra.run.smithi092.stderr:Inferring config /var/lib/ceph/00db0016-dfed-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T21:30:17.089 DEBUG:teuthology.orchestra.run.smithi092:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid 00db0016-dfed-11ee-95c8-87774f69a715 -- ceph osd reweight 3 1 2024-03-11T21:30:17.105 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:30:17.134 INFO:teuthology.orchestra.run.smithi097.stderr:Inferring config /var/lib/ceph/00db0016-dfed-11ee-95c8-87774f69a715/mon.b/config 2024-03-11T21:30:17.137 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:30:17.138 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:30:17.139 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:30:17.240 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:30:17.270 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:30:17.270 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:30:17.271 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:30:17.293 INFO:teuthology.orchestra.run.smithi092.stderr:nodeep-scrub is unset 2024-03-11T21:30:17.372 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:30:17.399 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:30:17.400 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:30:17.400 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:30:17.501 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:30:17.532 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:30:17.532 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:30:17.533 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:30:17.545 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:30:17 smithi097 ceph-mon[33317]: from='client.? v1:172.21.15.92:0/1464198237' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]': finished 2024-03-11T21:30:17.546 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:30:17 smithi097 ceph-mon[33317]: osdmap e132: 8 total, 8 up, 8 in 2024-03-11T21:30:17.546 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:30:17 smithi097 ceph-mon[33317]: from='client.? v1:172.21.15.92:0/953598547' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-11T21:30:17.546 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:30:17 smithi097 ceph-mon[33317]: from='client.34868 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-11T21:30:17.611 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:30:17 smithi092 ceph-mon[29572]: from='client.? v1:172.21.15.92:0/1464198237' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]': finished 2024-03-11T21:30:17.611 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:30:17 smithi092 ceph-mon[29572]: osdmap e132: 8 total, 8 up, 8 in 2024-03-11T21:30:17.611 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:30:17 smithi092 ceph-mon[29572]: from='client.? v1:172.21.15.92:0/953598547' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-11T21:30:17.611 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:30:17 smithi092 ceph-mon[29572]: from='client.34868 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-11T21:30:17.611 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:30:17 smithi092 ceph-mon[37563]: from='client.? v1:172.21.15.92:0/1464198237' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]': finished 2024-03-11T21:30:17.611 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:30:17 smithi092 ceph-mon[37563]: osdmap e132: 8 total, 8 up, 8 in 2024-03-11T21:30:17.611 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:30:17 smithi092 ceph-mon[37563]: from='client.? v1:172.21.15.92:0/953598547' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-11T21:30:17.611 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:30:17 smithi092 ceph-mon[37563]: from='client.34868 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-11T21:30:17.634 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:30:17.669 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:30:17.669 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:30:17.670 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:30:17.771 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:30:17.801 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:30:17.801 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:30:17.801 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:30:17.902 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:30:17.933 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:30:17.934 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:30:17.934 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:30:18.035 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:30:18.081 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:30:18.081 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:30:18.082 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:30:18.111 INFO:journalctl@ceph.mgr.y.smithi092.stdout:Mar 11 21:30:17 smithi092 ceph-00db0016-dfed-11ee-95c8-87774f69a715-mgr-y[29801]: ::ffff:172.21.15.97 - - [11/Mar/2024:21:30:17] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-11T21:30:18.182 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:30:18.230 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:30:18.231 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:30:18.231 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:30:18.332 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:30:18.369 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:30:18.370 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:30:18.370 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:30:18.471 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:30:18.511 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:30:18.512 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:30:18.512 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:30:18.574 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:30:18 smithi092 ceph-mon[29572]: 6.8 scrub starts 2024-03-11T21:30:18.574 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:30:18 smithi092 ceph-mon[29572]: 6.8 scrub ok 2024-03-11T21:30:18.574 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:30:18 smithi092 ceph-mon[29572]: 2.1 scrub starts 2024-03-11T21:30:18.574 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:30:18 smithi092 ceph-mon[29572]: 2.1 scrub ok 2024-03-11T21:30:18.574 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:30:18 smithi092 ceph-mon[29572]: pgmap v166: 148 pgs: 1 active+clean+scrubbing, 147 active+clean; 4.3 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 639 B/s rd, 0 B/s wr, 885 op/s 2024-03-11T21:30:18.574 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:30:18 smithi092 ceph-mon[29572]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-03-11T21:30:18.574 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:30:18 smithi092 ceph-mon[29572]: Cluster is now healthy 2024-03-11T21:30:18.574 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:30:18 smithi092 ceph-mon[29572]: from='client.34868 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-03-11T21:30:18.574 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:30:18 smithi092 ceph-mon[29572]: osdmap e133: 8 total, 8 up, 8 in 2024-03-11T21:30:18.574 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:30:18 smithi092 ceph-mon[29572]: 2.1e scrub starts 2024-03-11T21:30:18.574 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:30:18 smithi092 ceph-mon[29572]: 2.1e scrub ok 2024-03-11T21:30:18.575 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:30:18 smithi092 ceph-mon[37563]: 6.8 scrub starts 2024-03-11T21:30:18.575 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:30:18 smithi092 ceph-mon[37563]: 6.8 scrub ok 2024-03-11T21:30:18.575 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:30:18 smithi092 ceph-mon[37563]: 2.1 scrub starts 2024-03-11T21:30:18.575 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:30:18 smithi092 ceph-mon[37563]: 2.1 scrub ok 2024-03-11T21:30:18.575 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:30:18 smithi092 ceph-mon[37563]: pgmap v166: 148 pgs: 1 active+clean+scrubbing, 147 active+clean; 4.3 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 639 B/s rd, 0 B/s wr, 885 op/s 2024-03-11T21:30:18.575 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:30:18 smithi092 ceph-mon[37563]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-03-11T21:30:18.575 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:30:18 smithi092 ceph-mon[37563]: Cluster is now healthy 2024-03-11T21:30:18.575 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:30:18 smithi092 ceph-mon[37563]: from='client.34868 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-03-11T21:30:18.575 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:30:18 smithi092 ceph-mon[37563]: osdmap e133: 8 total, 8 up, 8 in 2024-03-11T21:30:18.575 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:30:18 smithi092 ceph-mon[37563]: 2.1e scrub starts 2024-03-11T21:30:18.575 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:30:18 smithi092 ceph-mon[37563]: 2.1e scrub ok 2024-03-11T21:30:18.576 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:30:18 smithi097 ceph-mon[33317]: 6.8 scrub starts 2024-03-11T21:30:18.576 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:30:18 smithi097 ceph-mon[33317]: 6.8 scrub ok 2024-03-11T21:30:18.576 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:30:18 smithi097 ceph-mon[33317]: 2.1 scrub starts 2024-03-11T21:30:18.576 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:30:18 smithi097 ceph-mon[33317]: 2.1 scrub ok 2024-03-11T21:30:18.577 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:30:18 smithi097 ceph-mon[33317]: pgmap v166: 148 pgs: 1 active+clean+scrubbing, 147 active+clean; 4.3 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 639 B/s rd, 0 B/s wr, 885 op/s 2024-03-11T21:30:18.577 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:30:18 smithi097 ceph-mon[33317]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-03-11T21:30:18.577 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:30:18 smithi097 ceph-mon[33317]: Cluster is now healthy 2024-03-11T21:30:18.577 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:30:18 smithi097 ceph-mon[33317]: from='client.34868 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-03-11T21:30:18.577 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:30:18 smithi097 ceph-mon[33317]: osdmap e133: 8 total, 8 up, 8 in 2024-03-11T21:30:18.577 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:30:18 smithi097 ceph-mon[33317]: 2.1e scrub starts 2024-03-11T21:30:18.577 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:30:18 smithi097 ceph-mon[33317]: 2.1e scrub ok 2024-03-11T21:30:18.613 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:30:18.646 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:30:18.647 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:30:18.647 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:30:18.748 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:30:18.778 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:30:18.778 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:30:18.779 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:30:18.879 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:30:18.907 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:30:18.908 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:30:18.908 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:30:19.009 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:30:19.057 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:30:19.058 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:30:19.058 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:30:19.160 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:30:19.189 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:30:19.189 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:30:19.190 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:30:19.291 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:30:19.331 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:30:19.345 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:30:19.345 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:30:19.446 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:30:19.452 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:30:19 smithi092 ceph-mon[29572]: 6.15 scrub starts 2024-03-11T21:30:19.452 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:30:19 smithi092 ceph-mon[29572]: 6.15 scrub ok 2024-03-11T21:30:19.452 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:30:19 smithi092 ceph-mon[29572]: 4.5 scrub starts 2024-03-11T21:30:19.453 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:30:19 smithi092 ceph-mon[29572]: 4.5 scrub ok 2024-03-11T21:30:19.453 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:30:19 smithi092 ceph-mon[29572]: 6.b scrub starts 2024-03-11T21:30:19.453 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:30:19 smithi092 ceph-mon[29572]: 6.b scrub ok 2024-03-11T21:30:19.453 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:30:19 smithi092 ceph-mon[37563]: 6.15 scrub starts 2024-03-11T21:30:19.453 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:30:19 smithi092 ceph-mon[37563]: 6.15 scrub ok 2024-03-11T21:30:19.453 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:30:19 smithi092 ceph-mon[37563]: 4.5 scrub starts 2024-03-11T21:30:19.453 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:30:19 smithi092 ceph-mon[37563]: 4.5 scrub ok 2024-03-11T21:30:19.453 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:30:19 smithi092 ceph-mon[37563]: 6.b scrub starts 2024-03-11T21:30:19.453 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:30:19 smithi092 ceph-mon[37563]: 6.b scrub ok 2024-03-11T21:30:19.472 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:30:19.473 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:30:19.473 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:30:19.531 INFO:teuthology.orchestra.run.smithi092.stderr:Inferring config /var/lib/ceph/00db0016-dfed-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T21:30:19.550 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:30:19 smithi097 ceph-mon[33317]: 6.15 scrub starts 2024-03-11T21:30:19.550 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:30:19 smithi097 ceph-mon[33317]: 6.15 scrub ok 2024-03-11T21:30:19.551 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:30:19 smithi097 ceph-mon[33317]: 4.5 scrub starts 2024-03-11T21:30:19.551 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:30:19 smithi097 ceph-mon[33317]: 4.5 scrub ok 2024-03-11T21:30:19.551 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:30:19 smithi097 ceph-mon[33317]: 6.b scrub starts 2024-03-11T21:30:19.551 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:30:19 smithi097 ceph-mon[33317]: 6.b scrub ok 2024-03-11T21:30:19.574 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:30:19.587 INFO:teuthology.orchestra.run.smithi092.stdout:osd.0: {} 2024-03-11T21:30:19.635 INFO:teuthology.orchestra.run.smithi092.stdout:osd.1: {} 2024-03-11T21:30:19.640 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:30:19.641 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:30:19.641 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:30:19.671 INFO:teuthology.orchestra.run.smithi092.stdout:osd.2: {} 2024-03-11T21:30:19.710 INFO:teuthology.orchestra.run.smithi092.stdout:osd.3: {} 2024-03-11T21:30:19.740 INFO:teuthology.orchestra.run.smithi092.stdout:osd.4: {} 2024-03-11T21:30:19.742 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:30:19.776 INFO:teuthology.orchestra.run.smithi092.stdout:osd.5: {} 2024-03-11T21:30:19.786 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:30:19.787 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:30:19.787 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:30:19.812 INFO:teuthology.orchestra.run.smithi092.stdout:osd.6: {} 2024-03-11T21:30:19.831 INFO:teuthology.orchestra.run.smithi092.stdout:osd.7: {} 2024-03-11T21:30:19.832 INFO:teuthology.orchestra.run.smithi092.stderr:osd.0: osd_enable_op_tracker = '' 2024-03-11T21:30:19.833 INFO:teuthology.orchestra.run.smithi092.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-11T21:30:19.833 INFO:teuthology.orchestra.run.smithi092.stderr:osd.2: osd_enable_op_tracker = '' 2024-03-11T21:30:19.833 INFO:teuthology.orchestra.run.smithi092.stderr:osd.3: osd_enable_op_tracker = '' 2024-03-11T21:30:19.833 INFO:teuthology.orchestra.run.smithi092.stderr:osd.4: osd_enable_op_tracker = '' 2024-03-11T21:30:19.833 INFO:teuthology.orchestra.run.smithi092.stderr:osd.5: osd_enable_op_tracker = '' 2024-03-11T21:30:19.833 INFO:teuthology.orchestra.run.smithi092.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-11T21:30:19.833 INFO:teuthology.orchestra.run.smithi092.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-11T21:30:19.888 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:30:19.929 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:30:19.930 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:30:19.930 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:30:20.031 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:30:20.059 DEBUG:teuthology.orchestra.run.smithi092:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid 00db0016-dfed-11ee-95c8-87774f69a715 -- ceph osd set noscrub 2024-03-11T21:30:20.070 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:30:20.070 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:30:20.071 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:30:20.172 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:30:20.197 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:30:20.198 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:30:20.198 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:30:20.299 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:30:20.341 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:30:20.342 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:30:20.342 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:30:20.443 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:30:20.455 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:30:20 smithi092 ceph-mon[29572]: 5.1d deep-scrub starts 2024-03-11T21:30:20.455 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:30:20 smithi092 ceph-mon[29572]: 5.1d deep-scrub ok 2024-03-11T21:30:20.455 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:30:20 smithi092 ceph-mon[29572]: pgmap v168: 148 pgs: 148 active+clean; 4.1 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 852 B/s rd, 0 B/s wr, 797 op/s 2024-03-11T21:30:20.455 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:30:20 smithi092 ceph-mon[29572]: from='client.? v1:172.21.15.92:0/3924413307' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-11T21:30:20.455 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:30:20 smithi092 ceph-mon[29572]: 3.2 scrub starts 2024-03-11T21:30:20.455 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:30:20 smithi092 ceph-mon[29572]: 3.2 scrub ok 2024-03-11T21:30:20.455 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:30:20 smithi092 ceph-mon[37563]: 5.1d deep-scrub starts 2024-03-11T21:30:20.456 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:30:20 smithi092 ceph-mon[37563]: 5.1d deep-scrub ok 2024-03-11T21:30:20.456 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:30:20 smithi092 ceph-mon[37563]: pgmap v168: 148 pgs: 148 active+clean; 4.1 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 852 B/s rd, 0 B/s wr, 797 op/s 2024-03-11T21:30:20.456 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:30:20 smithi092 ceph-mon[37563]: from='client.? v1:172.21.15.92:0/3924413307' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-11T21:30:20.456 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:30:20 smithi092 ceph-mon[37563]: 3.2 scrub starts 2024-03-11T21:30:20.456 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:30:20 smithi092 ceph-mon[37563]: 3.2 scrub ok 2024-03-11T21:30:20.490 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:30:20.491 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:30:20.491 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:30:20.491 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:30:20 smithi097 ceph-mon[33317]: 5.1d deep-scrub starts 2024-03-11T21:30:20.492 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:30:20 smithi097 ceph-mon[33317]: 5.1d deep-scrub ok 2024-03-11T21:30:20.492 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:30:20 smithi097 ceph-mon[33317]: pgmap v168: 148 pgs: 148 active+clean; 4.1 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 852 B/s rd, 0 B/s wr, 797 op/s 2024-03-11T21:30:20.492 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:30:20 smithi097 ceph-mon[33317]: from='client.? v1:172.21.15.92:0/3924413307' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-11T21:30:20.492 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:30:20 smithi097 ceph-mon[33317]: 3.2 scrub starts 2024-03-11T21:30:20.492 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:30:20 smithi097 ceph-mon[33317]: 3.2 scrub ok 2024-03-11T21:30:20.592 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:30:20.632 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:30:20.633 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:30:20.633 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:30:20.734 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:30:20.767 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:30:20.767 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:30:20.767 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:30:20.868 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:30:20.911 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:30:20.912 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:30:20.912 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:30:21.013 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:30:21.049 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:30:21.050 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:30:21.050 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:30:21.151 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:30:21.187 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:30:21.187 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:30:21.188 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:30:21.201 INFO:teuthology.orchestra.run.smithi097.stdout:ERROR: (22) Invalid argument 2024-03-11T21:30:21.202 INFO:teuthology.orchestra.run.smithi097.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-11T21:30:21.288 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:30:21.321 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:30:21.322 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:30:21.322 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:30:21.423 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:30:21.457 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:30:21.458 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:30:21.459 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:30:21.559 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:30:21.577 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:30:21 smithi097 ceph-mon[33317]: 5.12 scrub starts 2024-03-11T21:30:21.577 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:30:21 smithi097 ceph-mon[33317]: 5.12 scrub ok 2024-03-11T21:30:21.577 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:30:21 smithi097 ceph-mon[33317]: 4.d scrub starts 2024-03-11T21:30:21.577 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:30:21 smithi097 ceph-mon[33317]: 4.d scrub ok 2024-03-11T21:30:21.596 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:30:21.596 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:30:21.597 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:30:21.610 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:30:21 smithi092 ceph-mon[29572]: 5.12 scrub starts 2024-03-11T21:30:21.611 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:30:21 smithi092 ceph-mon[29572]: 5.12 scrub ok 2024-03-11T21:30:21.611 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:30:21 smithi092 ceph-mon[29572]: 4.d scrub starts 2024-03-11T21:30:21.611 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:30:21 smithi092 ceph-mon[29572]: 4.d scrub ok 2024-03-11T21:30:21.611 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:30:21 smithi092 ceph-mon[37563]: 5.12 scrub starts 2024-03-11T21:30:21.611 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:30:21 smithi092 ceph-mon[37563]: 5.12 scrub ok 2024-03-11T21:30:21.611 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:30:21 smithi092 ceph-mon[37563]: 4.d scrub starts 2024-03-11T21:30:21.611 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:30:21 smithi092 ceph-mon[37563]: 4.d scrub ok 2024-03-11T21:30:21.698 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:30:21.754 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:30:21.754 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:30:21.754 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:30:21.855 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:30:21.886 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:30:21.887 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:30:21.887 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:30:21.988 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:30:22.019 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:30:22.020 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:30:22.020 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:30:22.070 DEBUG:teuthology.orchestra.run.smithi097:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid 00db0016-dfed-11ee-95c8-87774f69a715 -- ceph daemon osd.7 dump_blocked_ops 2024-03-11T21:30:22.121 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:30:22.157 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:30:22.158 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:30:22.158 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:30:22.249 INFO:teuthology.orchestra.run.smithi092.stderr:Inferring config /var/lib/ceph/00db0016-dfed-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T21:30:22.258 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:30:22.287 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:30:22.287 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:30:22.288 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:30:22.323 INFO:journalctl@ceph.iscsi.iscsi.a.smithi097.stdout:Mar 11 21:30:22 smithi097 ceph-00db0016-dfed-11ee-95c8-87774f69a715-iscsi-iscsi-a[60943]: debug there is no tcmu-runner data available 2024-03-11T21:30:22.323 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:30:22 smithi097 ceph-mon[33317]: 5.d scrub starts 2024-03-11T21:30:22.323 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:30:22 smithi097 ceph-mon[33317]: 5.d scrub ok 2024-03-11T21:30:22.323 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:30:22 smithi097 ceph-mon[33317]: pgmap v169: 148 pgs: 148 active+clean; 4.1 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 656 B/s rd, 0 B/s wr, 614 op/s 2024-03-11T21:30:22.324 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:30:22 smithi097 ceph-mon[33317]: from='mgr.14607 ' entity='mgr.y' 2024-03-11T21:30:22.324 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:30:22 smithi097 ceph-mon[33317]: from='mgr.14607 v1:172.21.15.92:0/1702717597' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T21:30:22.324 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:30:22 smithi097 ceph-mon[33317]: 4.10 scrub starts 2024-03-11T21:30:22.324 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:30:22 smithi097 ceph-mon[33317]: 4.10 scrub ok 2024-03-11T21:30:22.360 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:30:22 smithi092 ceph-mon[29572]: 5.d scrub starts 2024-03-11T21:30:22.361 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:30:22 smithi092 ceph-mon[29572]: 5.d scrub ok 2024-03-11T21:30:22.361 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:30:22 smithi092 ceph-mon[29572]: pgmap v169: 148 pgs: 148 active+clean; 4.1 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 656 B/s rd, 0 B/s wr, 614 op/s 2024-03-11T21:30:22.361 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:30:22 smithi092 ceph-mon[29572]: from='mgr.14607 ' entity='mgr.y' 2024-03-11T21:30:22.361 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:30:22 smithi092 ceph-mon[29572]: from='mgr.14607 v1:172.21.15.92:0/1702717597' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T21:30:22.361 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:30:22 smithi092 ceph-mon[29572]: 4.10 scrub starts 2024-03-11T21:30:22.361 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:30:22 smithi092 ceph-mon[29572]: 4.10 scrub ok 2024-03-11T21:30:22.361 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:30:22 smithi092 ceph-mon[37563]: 5.d scrub starts 2024-03-11T21:30:22.361 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:30:22 smithi092 ceph-mon[37563]: 5.d scrub ok 2024-03-11T21:30:22.361 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:30:22 smithi092 ceph-mon[37563]: pgmap v169: 148 pgs: 148 active+clean; 4.1 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 656 B/s rd, 0 B/s wr, 614 op/s 2024-03-11T21:30:22.361 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:30:22 smithi092 ceph-mon[37563]: from='mgr.14607 ' entity='mgr.y' 2024-03-11T21:30:22.362 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:30:22 smithi092 ceph-mon[37563]: from='mgr.14607 v1:172.21.15.92:0/1702717597' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T21:30:22.362 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:30:22 smithi092 ceph-mon[37563]: 4.10 scrub starts 2024-03-11T21:30:22.362 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:30:22 smithi092 ceph-mon[37563]: 4.10 scrub ok 2024-03-11T21:30:22.388 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:30:22.428 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:30:22.429 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:30:22.429 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:30:22.455 INFO:teuthology.orchestra.run.smithi097.stderr:Inferring config /var/lib/ceph/00db0016-dfed-11ee-95c8-87774f69a715/mon.b/config 2024-03-11T21:30:22.530 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:30:22.595 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:30:22.596 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:30:22.596 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:30:22.697 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:30:22.747 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:30:22.748 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:30:22.748 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:30:22.849 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:30:22.893 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:30:22.894 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:30:22.894 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:30:22.995 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:30:23.023 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:30:23.024 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:30:23.025 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:30:23.125 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:30:23.177 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:30:23.177 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:30:23.177 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:30:23.278 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:30:23.316 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:30:23.316 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:30:23.317 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:30:23.345 INFO:teuthology.orchestra.run.smithi092.stderr:reweighted osd.3 to 1 (10000) 2024-03-11T21:30:23.418 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:30:23.446 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:30:23.447 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:30:23.447 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:30:23.548 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:30:23.576 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:30:23 smithi097 ceph-mon[33317]: 6.f scrub starts 2024-03-11T21:30:23.577 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:30:23 smithi097 ceph-mon[33317]: 6.f scrub ok 2024-03-11T21:30:23.577 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:30:23 smithi097 ceph-mon[33317]: from='client.24469 v1:172.21.15.97:0/404208310' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:30:23.577 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:30:23 smithi097 ceph-mon[33317]: 3.1 scrub starts 2024-03-11T21:30:23.577 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:30:23 smithi097 ceph-mon[33317]: 3.1 scrub ok 2024-03-11T21:30:23.577 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:30:23 smithi097 ceph-mon[33317]: from='client.? v1:172.21.15.92:0/3511054976' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]: dispatch 2024-03-11T21:30:23.578 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:30:23.579 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:30:23.579 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:30:23.610 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:30:23 smithi092 ceph-mon[29572]: 6.f scrub starts 2024-03-11T21:30:23.611 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:30:23 smithi092 ceph-mon[29572]: 6.f scrub ok 2024-03-11T21:30:23.611 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:30:23 smithi092 ceph-mon[29572]: from='client.24469 v1:172.21.15.97:0/404208310' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:30:23.611 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:30:23 smithi092 ceph-mon[29572]: 3.1 scrub starts 2024-03-11T21:30:23.611 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:30:23 smithi092 ceph-mon[29572]: 3.1 scrub ok 2024-03-11T21:30:23.611 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:30:23 smithi092 ceph-mon[29572]: from='client.? v1:172.21.15.92:0/3511054976' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]: dispatch 2024-03-11T21:30:23.611 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:30:23 smithi092 ceph-mon[37563]: 6.f scrub starts 2024-03-11T21:30:23.611 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:30:23 smithi092 ceph-mon[37563]: 6.f scrub ok 2024-03-11T21:30:23.611 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:30:23 smithi092 ceph-mon[37563]: from='client.24469 v1:172.21.15.97:0/404208310' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:30:23.612 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:30:23 smithi092 ceph-mon[37563]: 3.1 scrub starts 2024-03-11T21:30:23.612 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:30:23 smithi092 ceph-mon[37563]: 3.1 scrub ok 2024-03-11T21:30:23.612 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:30:23 smithi092 ceph-mon[37563]: from='client.? v1:172.21.15.92:0/3511054976' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]: dispatch 2024-03-11T21:30:23.679 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:30:23.712 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:30:23.712 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:30:23.713 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:30:23.713 DEBUG:teuthology.orchestra.run.smithi092:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid 00db0016-dfed-11ee-95c8-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-03-11T21:30:23.814 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:30:23.842 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:30:23.843 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:30:23.843 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:30:23.944 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:30:23.984 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:30:23.984 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:30:23.985 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:30:24.086 DEBUG:teuthology.orchestra.run.smithi092:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid 00db0016-dfed-11ee-95c8-87774f69a715 -- ceph osd reweight 4 1 2024-03-11T21:30:24.086 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:30:24.151 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:30:24.154 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:30:24.154 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:30:24.255 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:30:24.282 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:30:24.283 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:30:24.284 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:30:24.338 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:30:24 smithi092 ceph-mon[37563]: 1.0 scrub starts 2024-03-11T21:30:24.338 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:30:24 smithi092 ceph-mon[37563]: 1.0 scrub ok 2024-03-11T21:30:24.338 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:30:24 smithi092 ceph-mon[37563]: 5.c deep-scrub starts 2024-03-11T21:30:24.338 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:30:24 smithi092 ceph-mon[37563]: 5.c deep-scrub ok 2024-03-11T21:30:24.338 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:30:24 smithi092 ceph-mon[37563]: pgmap v170: 148 pgs: 148 active+clean; 3.8 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.25k op/s 2024-03-11T21:30:24.338 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:30:24 smithi092 ceph-mon[37563]: 4.13 scrub starts 2024-03-11T21:30:24.338 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:30:24 smithi092 ceph-mon[37563]: 4.13 scrub ok 2024-03-11T21:30:24.338 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:30:24 smithi092 ceph-mon[37563]: from='client.? v1:172.21.15.92:0/3511054976' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]': finished 2024-03-11T21:30:24.339 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:30:24 smithi092 ceph-mon[37563]: osdmap e134: 8 total, 8 up, 8 in 2024-03-11T21:30:24.339 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:30:24 smithi092 ceph-mon[37563]: 6.c scrub starts 2024-03-11T21:30:24.339 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:30:24 smithi092 ceph-mon[37563]: 6.c scrub ok 2024-03-11T21:30:24.339 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:30:24 smithi092 ceph-mon[37563]: 4.3 scrub starts 2024-03-11T21:30:24.339 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:30:24 smithi092 ceph-mon[37563]: 4.3 scrub ok 2024-03-11T21:30:24.384 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:30:24.427 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:30:24.428 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:30:24.428 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:30:24.529 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:30:24.575 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:30:24.575 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:30:24.575 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:30:24.576 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:30:24 smithi097 ceph-mon[33317]: 1.0 scrub starts 2024-03-11T21:30:24.576 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:30:24 smithi097 ceph-mon[33317]: 1.0 scrub ok 2024-03-11T21:30:24.576 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:30:24 smithi097 ceph-mon[33317]: 5.c deep-scrub starts 2024-03-11T21:30:24.576 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:30:24 smithi097 ceph-mon[33317]: 5.c deep-scrub ok 2024-03-11T21:30:24.577 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:30:24 smithi097 ceph-mon[33317]: pgmap v170: 148 pgs: 148 active+clean; 3.8 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.25k op/s 2024-03-11T21:30:24.577 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:30:24 smithi097 ceph-mon[33317]: 4.13 scrub starts 2024-03-11T21:30:24.577 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:30:24 smithi097 ceph-mon[33317]: 4.13 scrub ok 2024-03-11T21:30:24.577 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:30:24 smithi097 ceph-mon[33317]: from='client.? v1:172.21.15.92:0/3511054976' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]': finished 2024-03-11T21:30:24.577 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:30:24 smithi097 ceph-mon[33317]: osdmap e134: 8 total, 8 up, 8 in 2024-03-11T21:30:24.577 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:30:24 smithi097 ceph-mon[33317]: 6.c scrub starts 2024-03-11T21:30:24.577 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:30:24 smithi097 ceph-mon[33317]: 6.c scrub ok 2024-03-11T21:30:24.577 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:30:24 smithi097 ceph-mon[33317]: 4.3 scrub starts 2024-03-11T21:30:24.577 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:30:24 smithi097 ceph-mon[33317]: 4.3 scrub ok 2024-03-11T21:30:24.610 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:30:24 smithi092 ceph-mon[29572]: 1.0 scrub starts 2024-03-11T21:30:24.611 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:30:24 smithi092 ceph-mon[29572]: 1.0 scrub ok 2024-03-11T21:30:24.611 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:30:24 smithi092 ceph-mon[29572]: 5.c deep-scrub starts 2024-03-11T21:30:24.611 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:30:24 smithi092 ceph-mon[29572]: 5.c deep-scrub ok 2024-03-11T21:30:24.611 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:30:24 smithi092 ceph-mon[29572]: pgmap v170: 148 pgs: 148 active+clean; 3.8 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.25k op/s 2024-03-11T21:30:24.611 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:30:24 smithi092 ceph-mon[29572]: 4.13 scrub starts 2024-03-11T21:30:24.611 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:30:24 smithi092 ceph-mon[29572]: 4.13 scrub ok 2024-03-11T21:30:24.611 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:30:24 smithi092 ceph-mon[29572]: from='client.? v1:172.21.15.92:0/3511054976' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]': finished 2024-03-11T21:30:24.611 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:30:24 smithi092 ceph-mon[29572]: osdmap e134: 8 total, 8 up, 8 in 2024-03-11T21:30:24.611 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:30:24 smithi092 ceph-mon[29572]: 6.c scrub starts 2024-03-11T21:30:24.611 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:30:24 smithi092 ceph-mon[29572]: 6.c scrub ok 2024-03-11T21:30:24.611 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:30:24 smithi092 ceph-mon[29572]: 4.3 scrub starts 2024-03-11T21:30:24.611 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:30:24 smithi092 ceph-mon[29572]: 4.3 scrub ok 2024-03-11T21:30:24.677 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:30:24.729 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:30:24.729 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:30:24.730 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:30:24.750 INFO:teuthology.orchestra.run.smithi092.stderr:Inferring config /var/lib/ceph/00db0016-dfed-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T21:30:24.830 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:30:24.886 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:30:24.887 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:30:24.888 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:30:24.988 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:30:25.022 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:30:25.022 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:30:25.023 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:30:25.123 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:30:25.153 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:30:25.154 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:30:25.154 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:30:25.253 INFO:teuthology.orchestra.run.smithi097.stdout:ERROR: (22) Invalid argument 2024-03-11T21:30:25.253 INFO:teuthology.orchestra.run.smithi097.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-11T21:30:25.255 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:30:25.285 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:30:25.286 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:30:25.286 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:30:25.387 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:30:25.434 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:30:25.435 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:30:25.435 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:30:25.536 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:30:25.564 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:30:25.565 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:30:25.565 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:30:25.666 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:30:25.709 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:30:25.710 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:30:25.711 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:30:25.812 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:30:25.858 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:30:25.859 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:30:25.859 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:30:25.960 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:30:25.991 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:30:25.992 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:30:25.992 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:30:26.093 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:30:26.121 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:30:26.121 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:30:26.122 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:30:26.223 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:30:26.252 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:30:26.253 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:30:26.253 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:30:26.354 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:30:26.399 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:30:26.399 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:30:26.400 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:30:26.446 INFO:teuthology.orchestra.run.smithi092.stderr:noscrub is set 2024-03-11T21:30:26.501 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:30:26.530 DEBUG:teuthology.orchestra.run.smithi097:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid 00db0016-dfed-11ee-95c8-87774f69a715 -- ceph daemon osd.7 dump_historic_ops 2024-03-11T21:30:26.543 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:30:26.543 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:30:26.544 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:30:26.611 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:30:26 smithi092 ceph-mon[29572]: 4.4 scrub starts 2024-03-11T21:30:26.611 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:30:26 smithi092 ceph-mon[29572]: 4.4 scrub ok 2024-03-11T21:30:26.611 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:30:26 smithi092 ceph-mon[29572]: pgmap v172: 148 pgs: 148 active+clean; 3.8 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.03k op/s 2024-03-11T21:30:26.611 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:30:26 smithi092 ceph-mon[29572]: 6.a scrub starts 2024-03-11T21:30:26.611 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:30:26 smithi092 ceph-mon[29572]: 6.a scrub ok 2024-03-11T21:30:26.611 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:30:26 smithi092 ceph-mon[29572]: from='client.? v1:172.21.15.92:0/1515880413' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-11T21:30:26.611 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:30:26 smithi092 ceph-mon[29572]: from='client.34886 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-11T21:30:26.611 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:30:26 smithi092 ceph-mon[37563]: 4.4 scrub starts 2024-03-11T21:30:26.611 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:30:26 smithi092 ceph-mon[37563]: 4.4 scrub ok 2024-03-11T21:30:26.611 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:30:26 smithi092 ceph-mon[37563]: pgmap v172: 148 pgs: 148 active+clean; 3.8 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.03k op/s 2024-03-11T21:30:26.612 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:30:26 smithi092 ceph-mon[37563]: 6.a scrub starts 2024-03-11T21:30:26.612 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:30:26 smithi092 ceph-mon[37563]: 6.a scrub ok 2024-03-11T21:30:26.612 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:30:26 smithi092 ceph-mon[37563]: from='client.? v1:172.21.15.92:0/1515880413' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-11T21:30:26.612 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:30:26 smithi092 ceph-mon[37563]: from='client.34886 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-11T21:30:26.644 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:30:26.682 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:30:26.682 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:30:26.683 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:30:26.712 INFO:teuthology.orchestra.run.smithi092.stderr:Inferring config /var/lib/ceph/00db0016-dfed-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T21:30:26.784 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:30:26.822 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:30:26.822 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:30:26.823 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:30:26.826 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:30:26 smithi097 ceph-mon[33317]: 4.4 scrub starts 2024-03-11T21:30:26.826 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:30:26 smithi097 ceph-mon[33317]: 4.4 scrub ok 2024-03-11T21:30:26.826 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:30:26 smithi097 ceph-mon[33317]: pgmap v172: 148 pgs: 148 active+clean; 3.8 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.03k op/s 2024-03-11T21:30:26.826 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:30:26 smithi097 ceph-mon[33317]: 6.a scrub starts 2024-03-11T21:30:26.827 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:30:26 smithi097 ceph-mon[33317]: 6.a scrub ok 2024-03-11T21:30:26.827 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:30:26 smithi097 ceph-mon[33317]: from='client.? v1:172.21.15.92:0/1515880413' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-11T21:30:26.827 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:30:26 smithi097 ceph-mon[33317]: from='client.34886 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-11T21:30:26.869 INFO:teuthology.orchestra.run.smithi097.stderr:Inferring config /var/lib/ceph/00db0016-dfed-11ee-95c8-87774f69a715/mon.b/config 2024-03-11T21:30:26.924 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:30:26.970 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:30:26.970 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:30:26.971 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:30:27.071 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:30:27.120 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:30:27.120 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:30:27.120 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:30:27.221 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:30:27.251 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:30:27.252 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:30:27.252 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:30:27.354 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:30:27.409 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:30:27.410 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:30:27.410 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:30:27.511 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:30:27.538 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:30:27.539 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:30:27.539 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:30:27.576 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:30:27 smithi097 ceph-mon[33317]: 5.1e scrub starts 2024-03-11T21:30:27.577 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:30:27 smithi097 ceph-mon[33317]: 5.1e scrub ok 2024-03-11T21:30:27.577 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:30:27 smithi097 ceph-mon[33317]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-03-11T21:30:27.577 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:30:27 smithi097 ceph-mon[33317]: from='client.34886 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-03-11T21:30:27.577 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:30:27 smithi097 ceph-mon[33317]: osdmap e135: 8 total, 8 up, 8 in 2024-03-11T21:30:27.610 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:30:27 smithi092 ceph-mon[29572]: 5.1e scrub starts 2024-03-11T21:30:27.611 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:30:27 smithi092 ceph-mon[29572]: 5.1e scrub ok 2024-03-11T21:30:27.611 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:30:27 smithi092 ceph-mon[29572]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-03-11T21:30:27.611 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:30:27 smithi092 ceph-mon[29572]: from='client.34886 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-03-11T21:30:27.611 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:30:27 smithi092 ceph-mon[29572]: osdmap e135: 8 total, 8 up, 8 in 2024-03-11T21:30:27.611 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:30:27 smithi092 ceph-mon[37563]: 5.1e scrub starts 2024-03-11T21:30:27.611 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:30:27 smithi092 ceph-mon[37563]: 5.1e scrub ok 2024-03-11T21:30:27.611 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:30:27 smithi092 ceph-mon[37563]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-03-11T21:30:27.611 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:30:27 smithi092 ceph-mon[37563]: from='client.34886 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-03-11T21:30:27.611 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:30:27 smithi092 ceph-mon[37563]: osdmap e135: 8 total, 8 up, 8 in 2024-03-11T21:30:27.641 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:30:27.670 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:30:27.671 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:30:27.671 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:30:27.772 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:30:27.808 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:30:27.809 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:30:27.809 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:30:27.910 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:30:27.948 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:30:27.949 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:30:27.949 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:30:28.050 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:30:28.078 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:30:28.079 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:30:28.079 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:30:28.180 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:30:28.184 INFO:journalctl@ceph.mgr.y.smithi092.stdout:Mar 11 21:30:27 smithi092 ceph-00db0016-dfed-11ee-95c8-87774f69a715-mgr-y[29801]: ::ffff:172.21.15.97 - - [11/Mar/2024:21:30:27] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-11T21:30:28.227 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:30:28.227 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:30:28.227 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:30:28.328 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:30:28.395 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:30:28.396 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:30:28.396 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:30:28.463 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:30:28 smithi092 ceph-mon[29572]: pgmap v174: 148 pgs: 148 active+clean; 3.7 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 639 B/s rd, 0 B/s wr, 896 op/s 2024-03-11T21:30:28.464 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:30:28 smithi092 ceph-mon[37563]: pgmap v174: 148 pgs: 148 active+clean; 3.7 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 639 B/s rd, 0 B/s wr, 896 op/s 2024-03-11T21:30:28.497 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:30:28.539 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:30:28.539 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:30:28.540 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:30:28.628 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:30:28 smithi097 ceph-mon[33317]: pgmap v174: 148 pgs: 148 active+clean; 3.7 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 639 B/s rd, 0 B/s wr, 896 op/s 2024-03-11T21:30:28.640 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:30:28.668 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:30:28.669 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:30:28.669 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:30:28.681 INFO:teuthology.orchestra.run.smithi092.stdout:osd.0: {} 2024-03-11T21:30:28.723 INFO:teuthology.orchestra.run.smithi092.stdout:osd.1: {} 2024-03-11T21:30:28.766 INFO:teuthology.orchestra.run.smithi092.stdout:osd.2: {} 2024-03-11T21:30:28.770 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:30:28.807 INFO:teuthology.orchestra.run.smithi092.stdout:osd.3: {} 2024-03-11T21:30:28.810 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:30:28.811 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:30:28.812 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:30:28.847 INFO:teuthology.orchestra.run.smithi092.stdout:osd.4: {} 2024-03-11T21:30:28.892 INFO:teuthology.orchestra.run.smithi092.stdout:osd.5: {} 2024-03-11T21:30:28.913 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:30:28.916 INFO:teuthology.orchestra.run.smithi097.stdout:ERROR: (22) Invalid argument 2024-03-11T21:30:28.916 INFO:teuthology.orchestra.run.smithi097.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-11T21:30:28.932 INFO:teuthology.orchestra.run.smithi092.stdout:osd.6: {} 2024-03-11T21:30:28.944 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:30:28.945 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:30:28.945 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:30:28.970 INFO:teuthology.orchestra.run.smithi092.stdout:osd.7: {} 2024-03-11T21:30:28.982 INFO:teuthology.orchestra.run.smithi092.stderr:osd.0: osd_enable_op_tracker = '' 2024-03-11T21:30:28.982 INFO:teuthology.orchestra.run.smithi092.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-11T21:30:28.982 INFO:teuthology.orchestra.run.smithi092.stderr:osd.2: osd_enable_op_tracker = '' 2024-03-11T21:30:28.982 INFO:teuthology.orchestra.run.smithi092.stderr:osd.3: osd_enable_op_tracker = '' 2024-03-11T21:30:28.982 INFO:teuthology.orchestra.run.smithi092.stderr:osd.4: osd_enable_op_tracker = '' 2024-03-11T21:30:28.982 INFO:teuthology.orchestra.run.smithi092.stderr:osd.5: osd_enable_op_tracker = '' 2024-03-11T21:30:28.982 INFO:teuthology.orchestra.run.smithi092.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-11T21:30:28.983 INFO:teuthology.orchestra.run.smithi092.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-11T21:30:29.046 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:30:29.091 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:30:29.091 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:30:29.092 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:30:29.193 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:30:29.237 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:30:29.237 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:30:29.238 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:30:29.338 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:30:29.377 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:30:29.378 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:30:29.379 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:30:29.479 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:30:29.510 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:30:29.511 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:30:29.511 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:30:29.576 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:30:29 smithi097 ceph-mon[33317]: from='client.? v1:172.21.15.92:0/2527450725' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-11T21:30:29.613 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:30:29.655 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:30:29.656 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:30:29.656 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:30:29.666 DEBUG:teuthology.orchestra.run.smithi092:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid 00db0016-dfed-11ee-95c8-87774f69a715 -- ceph daemon osd.0 dump_ops_in_flight 2024-03-11T21:30:29.745 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:30:29 smithi092 ceph-mon[29572]: from='client.? v1:172.21.15.92:0/2527450725' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-11T21:30:29.745 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:30:29 smithi092 ceph-mon[37563]: from='client.? v1:172.21.15.92:0/2527450725' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-11T21:30:29.758 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:30:29.825 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:30:29.826 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:30:29.826 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:30:29.927 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:30:29.991 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:30:29.992 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:30:29.992 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:30:30.093 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:30:30.102 DEBUG:teuthology.orchestra.run.smithi092:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid 00db0016-dfed-11ee-95c8-87774f69a715 -- ceph osd set nodeep-scrub 2024-03-11T21:30:30.120 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:30:30.121 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:30:30.121 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:30:30.222 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:30:30.273 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:30:30.273 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:30:30.273 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:30:30.374 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:30:30.417 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:30:30 smithi092 ceph-mon[29572]: pgmap v175: 148 pgs: 148 active+clean; 3.5 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.31k op/s 2024-03-11T21:30:30.420 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:30:30 smithi092 ceph-mon[37563]: pgmap v175: 148 pgs: 148 active+clean; 3.5 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.31k op/s 2024-03-11T21:30:30.425 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:30:30.425 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:30:30.426 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:30:30.526 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:30:30.555 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:30:30.556 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:30:30.556 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:30:30.657 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:30:30.687 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:30:30.688 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:30:30.688 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:30:30.789 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:30:30.819 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:30:30.819 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:30:30.820 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:30:30.826 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:30:30 smithi097 ceph-mon[33317]: pgmap v175: 148 pgs: 148 active+clean; 3.5 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.31k op/s 2024-03-11T21:30:30.921 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:30:30.948 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:30:30.948 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:30:30.949 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:30:31.050 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:30:31.080 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:30:31.080 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:30:31.081 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:30:31.181 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:30:31.211 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:30:31.212 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:30:31.212 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:30:31.313 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:30:31.356 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:30:31.357 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:30:31.357 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:30:31.458 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:30:31.482 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:30:31.483 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:30:31.483 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:30:31.584 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:30:31.624 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:30:31.624 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:30:31.624 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:30:31.725 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:30:31.761 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:30:31.762 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:30:31.762 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:30:31.863 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:30:31.891 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:30:31.892 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:30:31.892 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:30:31.993 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:30:32.022 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:30:32.022 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:30:32.023 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:30:32.124 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:30:32.152 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:30:32.153 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:30:32.154 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:30:32.254 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:30:32.292 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:30:32.293 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:30:32.293 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:30:32.394 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:30:32.440 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:30:32.441 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:30:32.441 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:30:32.542 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:30:32.570 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:30:32.571 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:30:32.571 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:30:32.576 INFO:journalctl@ceph.iscsi.iscsi.a.smithi097.stdout:Mar 11 21:30:32 smithi097 ceph-00db0016-dfed-11ee-95c8-87774f69a715-iscsi-iscsi-a[60943]: debug there is no tcmu-runner data available 2024-03-11T21:30:32.576 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:30:32 smithi097 ceph-mon[33317]: pgmap v176: 148 pgs: 148 active+clean; 3.5 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 639 B/s rd, 0 B/s wr, 657 op/s 2024-03-11T21:30:32.663 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:30:32 smithi092 ceph-mon[29572]: pgmap v176: 148 pgs: 148 active+clean; 3.5 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 639 B/s rd, 0 B/s wr, 657 op/s 2024-03-11T21:30:32.663 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:30:32 smithi092 ceph-mon[37563]: pgmap v176: 148 pgs: 148 active+clean; 3.5 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 639 B/s rd, 0 B/s wr, 657 op/s 2024-03-11T21:30:32.672 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:30:32.720 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:30:32.721 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:30:32.721 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:30:32.822 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:30:32.856 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:30:32.856 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:30:32.856 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:30:32.957 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:30:32.988 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:30:32.988 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:30:32.988 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:30:33.089 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:30:33.129 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:30:33.130 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:30:33.130 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:30:33.231 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:30:33.271 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:30:33.272 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:30:33.272 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:30:33.373 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:30:33.402 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:30:33.402 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:30:33.403 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:30:33.504 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:30:33.530 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:30:33.531 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:30:33.532 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:30:33.632 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:30:33.661 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:30:33.662 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:30:33.662 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:30:33.763 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:30:33.790 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:30:33.791 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:30:33.791 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:30:33.826 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:30:33 smithi097 ceph-mon[33317]: from='client.24469 v1:172.21.15.97:0/404208310' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:30:33.860 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:30:33 smithi092 ceph-mon[29572]: from='client.24469 v1:172.21.15.97:0/404208310' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:30:33.861 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:30:33 smithi092 ceph-mon[37563]: from='client.24469 v1:172.21.15.97:0/404208310' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:30:33.892 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:30:33.918 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:30:33.919 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:30:33.919 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:30:34.020 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:30:34.053 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:30:34.053 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:30:34.053 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:30:34.154 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:30:34.192 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:30:34.192 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:30:34.193 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:30:34.293 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:30:34.322 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:30:34.323 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:30:34.323 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:30:34.424 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:30:34.435 INFO:teuthology.orchestra.run.smithi092.stderr:Inferring config /var/lib/ceph/00db0016-dfed-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T21:30:34.499 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:30:34.500 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:30:34.501 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:30:34.587 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:30:34 smithi092 ceph-mon[29572]: pgmap v177: 148 pgs: 148 active+clean; 3.2 GiB data, 10 GiB used, 705 GiB / 715 GiB avail; 1.0 KiB/s rd, 0 B/s wr, 1.04k op/s 2024-03-11T21:30:34.587 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:30:34 smithi092 ceph-mon[37563]: pgmap v177: 148 pgs: 148 active+clean; 3.2 GiB data, 10 GiB used, 705 GiB / 715 GiB avail; 1.0 KiB/s rd, 0 B/s wr, 1.04k op/s 2024-03-11T21:30:34.602 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:30:34.628 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:30:34.629 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:30:34.629 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:30:34.730 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:30:34.746 INFO:teuthology.orchestra.run.smithi092.stderr:Inferring config /var/lib/ceph/00db0016-dfed-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T21:30:34.757 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:30:34.758 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:30:34.758 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:30:34.826 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:30:34 smithi097 ceph-mon[33317]: pgmap v177: 148 pgs: 148 active+clean; 3.2 GiB data, 10 GiB used, 705 GiB / 715 GiB avail; 1.0 KiB/s rd, 0 B/s wr, 1.04k op/s 2024-03-11T21:30:34.860 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:30:34.886 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:30:34.886 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:30:34.887 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:30:34.988 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:30:35.033 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:30:35.034 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:30:35.034 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:30:35.135 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:30:35.164 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:30:35.164 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:30:35.165 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:30:35.266 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:30:35.293 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:30:35.293 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:30:35.294 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:30:35.395 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:30:35.419 INFO:teuthology.orchestra.run.smithi092.stderr:reweighted osd.4 to 1 (10000) 2024-03-11T21:30:35.433 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:30:35.434 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:30:35.435 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:30:35.535 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:30:35.566 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:30:35.567 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:30:35.567 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:30:35.610 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:30:35 smithi092 ceph-mon[29572]: from='client.? v1:172.21.15.92:0/1141660441' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]: dispatch 2024-03-11T21:30:35.611 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:30:35 smithi092 ceph-mon[29572]: from='client.34895 ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]: dispatch 2024-03-11T21:30:35.611 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:30:35 smithi092 ceph-mon[37563]: from='client.? v1:172.21.15.92:0/1141660441' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]: dispatch 2024-03-11T21:30:35.611 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:30:35 smithi092 ceph-mon[37563]: from='client.34895 ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]: dispatch 2024-03-11T21:30:35.613 DEBUG:teuthology.orchestra.run.smithi092:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid 00db0016-dfed-11ee-95c8-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-03-11T21:30:35.669 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:30:35.697 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:30:35.697 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:30:35.697 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:30:35.798 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:30:35.826 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:30:35 smithi097 ceph-mon[33317]: from='client.? v1:172.21.15.92:0/1141660441' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]: dispatch 2024-03-11T21:30:35.826 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:30:35 smithi097 ceph-mon[33317]: from='client.34895 ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]: dispatch 2024-03-11T21:30:35.848 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:30:35.848 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:30:35.849 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:30:35.949 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:30:35.980 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:30:35.981 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:30:35.981 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:30:36.082 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:30:36.106 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:30:36.106 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:30:36.106 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:30:36.207 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:30:36.245 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:30:36.245 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:30:36.246 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:30:36.346 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:30:36.374 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:30:36.374 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:30:36.375 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:30:36.475 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:30:36.505 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:30:36.505 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:30:36.506 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:30:36.607 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:30:36.641 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:30:36.641 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:30:36.641 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:30:36.700 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:30:36 smithi092 ceph-mon[29572]: pgmap v178: 148 pgs: 148 active+clean; 3.2 GiB data, 10 GiB used, 705 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 1.01k op/s 2024-03-11T21:30:36.700 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:30:36 smithi092 ceph-mon[29572]: from='client.34895 ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]': finished 2024-03-11T21:30:36.700 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:30:36 smithi092 ceph-mon[29572]: osdmap e136: 8 total, 8 up, 8 in 2024-03-11T21:30:36.700 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:30:36 smithi092 ceph-mon[29572]: from='mgr.14607 ' entity='mgr.y' 2024-03-11T21:30:36.700 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:30:36 smithi092 ceph-mon[29572]: from='mgr.14607 v1:172.21.15.92:0/1702717597' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T21:30:36.701 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:30:36 smithi092 ceph-mon[37563]: pgmap v178: 148 pgs: 148 active+clean; 3.2 GiB data, 10 GiB used, 705 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 1.01k op/s 2024-03-11T21:30:36.701 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:30:36 smithi092 ceph-mon[37563]: from='client.34895 ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]': finished 2024-03-11T21:30:36.701 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:30:36 smithi092 ceph-mon[37563]: osdmap e136: 8 total, 8 up, 8 in 2024-03-11T21:30:36.701 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:30:36 smithi092 ceph-mon[37563]: from='mgr.14607 ' entity='mgr.y' 2024-03-11T21:30:36.701 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:30:36 smithi092 ceph-mon[37563]: from='mgr.14607 v1:172.21.15.92:0/1702717597' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T21:30:36.742 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:30:36.774 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:30:36.775 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:30:36.775 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:30:36.826 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:30:36 smithi097 ceph-mon[33317]: pgmap v178: 148 pgs: 148 active+clean; 3.2 GiB data, 10 GiB used, 705 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 1.01k op/s 2024-03-11T21:30:36.827 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:30:36 smithi097 ceph-mon[33317]: from='client.34895 ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]': finished 2024-03-11T21:30:36.827 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:30:36 smithi097 ceph-mon[33317]: osdmap e136: 8 total, 8 up, 8 in 2024-03-11T21:30:36.827 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:30:36 smithi097 ceph-mon[33317]: from='mgr.14607 ' entity='mgr.y' 2024-03-11T21:30:36.827 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:30:36 smithi097 ceph-mon[33317]: from='mgr.14607 v1:172.21.15.92:0/1702717597' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T21:30:36.876 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:30:36.906 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:30:36.907 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:30:36.907 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:30:37.008 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:30:37.050 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:30:37.051 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:30:37.051 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:30:37.152 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:30:37.181 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:30:37.182 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:30:37.182 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:30:37.283 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:30:37.330 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:30:37.331 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:30:37.331 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:30:37.432 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:30:37.459 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:30:37.460 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:30:37.460 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:30:37.561 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:30:37.589 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:30:37.589 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:30:37.590 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:30:37.691 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:30:37.735 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:30:37.735 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:30:37.735 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:30:37.836 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:30:37.875 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:30:37.876 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:30:37.876 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:30:37.977 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:30:38.008 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:30:38.008 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:30:38.008 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:30:38.109 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:30:38.113 DEBUG:teuthology.orchestra.run.smithi092:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid 00db0016-dfed-11ee-95c8-87774f69a715 -- ceph osd reweight 5 1 2024-03-11T21:30:38.137 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:30:38.138 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:30:38.138 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:30:38.161 INFO:journalctl@ceph.mgr.y.smithi092.stdout:Mar 11 21:30:37 smithi092 ceph-00db0016-dfed-11ee-95c8-87774f69a715-mgr-y[29801]: ::ffff:172.21.15.97 - - [11/Mar/2024:21:30:37] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-11T21:30:38.240 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:30:38.267 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:30:38.268 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:30:38.269 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:30:38.370 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:30:38.429 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:30:38.430 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:30:38.431 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:30:38.531 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:30:38.559 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:30:38.560 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:30:38.560 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:30:38.661 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:30:38.687 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:30:38 smithi092 ceph-mon[29572]: pgmap v180: 148 pgs: 148 active+clean; 3.1 GiB data, 10 GiB used, 705 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 999 op/s 2024-03-11T21:30:38.687 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:30:38 smithi092 ceph-mon[37563]: pgmap v180: 148 pgs: 148 active+clean; 3.1 GiB data, 10 GiB used, 705 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 999 op/s 2024-03-11T21:30:38.707 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:30:38.708 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:30:38.708 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:30:38.809 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:30:38.826 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:30:38 smithi097 ceph-mon[33317]: pgmap v180: 148 pgs: 148 active+clean; 3.1 GiB data, 10 GiB used, 705 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 999 op/s 2024-03-11T21:30:38.839 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:30:38.839 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:30:38.840 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:30:38.940 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:30:38.980 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:30:38.980 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:30:38.981 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:30:39.081 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:30:39.110 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:30:39.110 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:30:39.111 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:30:39.205 INFO:teuthology.orchestra.run.smithi092.stderr:Inferring config /var/lib/ceph/00db0016-dfed-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T21:30:39.212 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:30:39.279 INFO:teuthology.orchestra.run.smithi092.stdout:{ 2024-03-11T21:30:39.279 INFO:teuthology.orchestra.run.smithi092.stdout: "ops": [ 2024-03-11T21:30:39.279 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:30:39.279 INFO:teuthology.orchestra.run.smithi092.stdout: "description": "osd_repop_reply(client.34715.0:199100 7.4 e136/99)", 2024-03-11T21:30:39.279 INFO:teuthology.orchestra.run.smithi092.stdout: "initiated_at": "2024-03-11T21:30:39.293542+0000", 2024-03-11T21:30:39.279 INFO:teuthology.orchestra.run.smithi092.stdout: "age": 0.000983631, 2024-03-11T21:30:39.279 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.001011629, 2024-03-11T21:30:39.279 INFO:teuthology.orchestra.run.smithi092.stdout: "type_data": { 2024-03-11T21:30:39.279 INFO:teuthology.orchestra.run.smithi092.stdout: "flag_point": "queued for pg", 2024-03-11T21:30:39.279 INFO:teuthology.orchestra.run.smithi092.stdout: "events": [ 2024-03-11T21:30:39.279 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:30:39.280 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "initiated", 2024-03-11T21:30:39.280 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:30:39.293542+0000", 2024-03-11T21:30:39.280 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-03-11T21:30:39.280 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:30:39.280 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:30:39.280 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "header_read", 2024-03-11T21:30:39.280 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:30:39.293542+0000", 2024-03-11T21:30:39.280 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-03-11T21:30:39.280 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:30:39.280 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:30:39.280 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "throttled", 2024-03-11T21:30:39.280 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:30:39.293543+0000", 2024-03-11T21:30:39.280 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 3.58e-07 2024-03-11T21:30:39.281 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:30:39.281 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:30:39.281 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "all_read", 2024-03-11T21:30:39.281 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:30:39.293545+0000", 2024-03-11T21:30:39.281 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 2.5799999999999999e-06 2024-03-11T21:30:39.281 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:30:39.281 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:30:39.281 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "dispatched", 2024-03-11T21:30:39.281 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:30:39.293551+0000", 2024-03-11T21:30:39.281 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 5.6189999999999999e-06 2024-03-11T21:30:39.281 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:30:39.281 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:30:39.281 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "queued_for_pg", 2024-03-11T21:30:39.282 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:30:39.293556+0000", 2024-03-11T21:30:39.282 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 5.0690000000000001e-06 2024-03-11T21:30:39.282 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-03-11T21:30:39.282 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-03-11T21:30:39.282 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-03-11T21:30:39.282 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:30:39.282 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:30:39.282 INFO:teuthology.orchestra.run.smithi092.stdout: "description": "osd_repop_reply(client.34715.0:199100 7.4 e136/99)", 2024-03-11T21:30:39.282 INFO:teuthology.orchestra.run.smithi092.stdout: "initiated_at": "2024-03-11T21:30:39.293646+0000", 2024-03-11T21:30:39.282 INFO:teuthology.orchestra.run.smithi092.stdout: "age": 0.00087966699999999999, 2024-03-11T21:30:39.282 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00096329399999999996, 2024-03-11T21:30:39.282 INFO:teuthology.orchestra.run.smithi092.stdout: "type_data": { 2024-03-11T21:30:39.282 INFO:teuthology.orchestra.run.smithi092.stdout: "flag_point": "queued for pg", 2024-03-11T21:30:39.282 INFO:teuthology.orchestra.run.smithi092.stdout: "events": [ 2024-03-11T21:30:39.283 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:30:39.283 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "initiated", 2024-03-11T21:30:39.283 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:30:39.293646+0000", 2024-03-11T21:30:39.283 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-03-11T21:30:39.283 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:30:39.283 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:30:39.283 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "header_read", 2024-03-11T21:30:39.283 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:30:39.293646+0000", 2024-03-11T21:30:39.283 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-03-11T21:30:39.283 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:30:39.283 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:30:39.283 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "throttled", 2024-03-11T21:30:39.283 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:30:39.293647+0000", 2024-03-11T21:30:39.284 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 7.4300000000000002e-07 2024-03-11T21:30:39.284 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:30:39.284 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:30:39.284 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "all_read", 2024-03-11T21:30:39.284 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:30:39.293651+0000", 2024-03-11T21:30:39.284 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 4.1470000000000003e-06 2024-03-11T21:30:39.284 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:30:39.284 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:30:39.284 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "dispatched", 2024-03-11T21:30:39.284 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:30:39.293658+0000", 2024-03-11T21:30:39.284 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 6.9299999999999997e-06 2024-03-11T21:30:39.284 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:30:39.284 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:30:39.284 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "queued_for_pg", 2024-03-11T21:30:39.284 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:30:39.293665+0000", 2024-03-11T21:30:39.285 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 6.2949999999999999e-06 2024-03-11T21:30:39.285 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-03-11T21:30:39.285 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-03-11T21:30:39.285 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-03-11T21:30:39.285 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:30:39.285 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:30:39.285 INFO:teuthology.orchestra.run.smithi092.stdout: "description": "osd_op(client.34715.0:199111 7.9 7:95cf75ab:::benchmark_data_smithi092_76040_object77200:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e136)", 2024-03-11T21:30:39.285 INFO:teuthology.orchestra.run.smithi092.stdout: "initiated_at": "2024-03-11T21:30:39.293680+0000", 2024-03-11T21:30:39.285 INFO:teuthology.orchestra.run.smithi092.stdout: "age": 0.00084588299999999996, 2024-03-11T21:30:39.285 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00098864299999999999, 2024-03-11T21:30:39.285 INFO:teuthology.orchestra.run.smithi092.stdout: "type_data": { 2024-03-11T21:30:39.285 INFO:teuthology.orchestra.run.smithi092.stdout: "flag_point": "waiting for sub ops", 2024-03-11T21:30:39.285 INFO:teuthology.orchestra.run.smithi092.stdout: "client_info": { 2024-03-11T21:30:39.285 INFO:teuthology.orchestra.run.smithi092.stdout: "client": "client.34715", 2024-03-11T21:30:39.285 INFO:teuthology.orchestra.run.smithi092.stdout: "client_addr": "v1:172.21.15.92:0/2403132573", 2024-03-11T21:30:39.286 INFO:teuthology.orchestra.run.smithi092.stdout: "tid": 199111 2024-03-11T21:30:39.286 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:30:39.286 INFO:teuthology.orchestra.run.smithi092.stdout: "events": [ 2024-03-11T21:30:39.286 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:30:39.286 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "initiated", 2024-03-11T21:30:39.286 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:30:39.293680+0000", 2024-03-11T21:30:39.286 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-03-11T21:30:39.286 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:30:39.286 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:30:39.286 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "header_read", 2024-03-11T21:30:39.286 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:30:39.293680+0000", 2024-03-11T21:30:39.286 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-03-11T21:30:39.286 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:30:39.286 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:30:39.286 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "throttled", 2024-03-11T21:30:39.287 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:30:39.293682+0000", 2024-03-11T21:30:39.287 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.7379999999999999e-06 2024-03-11T21:30:39.287 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:30:39.287 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:30:39.287 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "all_read", 2024-03-11T21:30:39.287 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:30:39.293687+0000", 2024-03-11T21:30:39.287 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 5.259e-06 2024-03-11T21:30:39.287 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:30:39.287 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:30:39.287 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "dispatched", 2024-03-11T21:30:39.287 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:30:39.293696+0000", 2024-03-11T21:30:39.287 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 8.7970000000000006e-06 2024-03-11T21:30:39.287 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:30:39.287 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:30:39.288 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "queued_for_pg", 2024-03-11T21:30:39.288 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:30:39.293704+0000", 2024-03-11T21:30:39.288 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 7.8429999999999993e-06 2024-03-11T21:30:39.288 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:30:39.288 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:30:39.288 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-03-11T21:30:39.288 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:30:39.294113+0000", 2024-03-11T21:30:39.288 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00040874100000000001 2024-03-11T21:30:39.288 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:30:39.288 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:30:39.288 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "started", 2024-03-11T21:30:39.288 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:30:39.294255+0000", 2024-03-11T21:30:39.288 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.000142225 2024-03-11T21:30:39.288 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:30:39.289 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:30:39.289 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "waiting for subops from 1,4", 2024-03-11T21:30:39.289 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:30:39.294388+0000", 2024-03-11T21:30:39.289 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.000133557 2024-03-11T21:30:39.289 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-03-11T21:30:39.289 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-03-11T21:30:39.289 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-03-11T21:30:39.289 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:30:39.289 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:30:39.289 INFO:teuthology.orchestra.run.smithi092.stdout: "description": "osd_repop(client.34715.0:199109 7.e e136/99)", 2024-03-11T21:30:39.289 INFO:teuthology.orchestra.run.smithi092.stdout: "initiated_at": "2024-03-11T21:30:39.293719+0000", 2024-03-11T21:30:39.289 INFO:teuthology.orchestra.run.smithi092.stdout: "age": 0.00080721700000000005, 2024-03-11T21:30:39.289 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.0010242719999999999, 2024-03-11T21:30:39.289 INFO:teuthology.orchestra.run.smithi092.stdout: "type_data": { 2024-03-11T21:30:39.290 INFO:teuthology.orchestra.run.smithi092.stdout: "flag_point": "queued for pg", 2024-03-11T21:30:39.290 INFO:teuthology.orchestra.run.smithi092.stdout: "events": [ 2024-03-11T21:30:39.290 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:30:39.290 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "initiated", 2024-03-11T21:30:39.290 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:30:39.293719+0000", 2024-03-11T21:30:39.290 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-03-11T21:30:39.290 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:30:39.290 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:30:39.290 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "header_read", 2024-03-11T21:30:39.290 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:30:39.293719+0000", 2024-03-11T21:30:39.290 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-03-11T21:30:39.290 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:30:39.290 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:30:39.291 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "throttled", 2024-03-11T21:30:39.291 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:30:39.293719+0000", 2024-03-11T21:30:39.291 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 5.9400000000000005e-07 2024-03-11T21:30:39.291 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:30:39.291 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:30:39.291 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "all_read", 2024-03-11T21:30:39.291 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:30:39.293727+0000", 2024-03-11T21:30:39.291 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 7.8169999999999999e-06 2024-03-11T21:30:39.291 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:30:39.291 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:30:39.291 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "dispatched", 2024-03-11T21:30:39.291 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:30:39.293734+0000", 2024-03-11T21:30:39.291 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 6.6409999999999996e-06 2024-03-11T21:30:39.291 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:30:39.291 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:30:39.292 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "queued_for_pg", 2024-03-11T21:30:39.292 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:30:39.293740+0000", 2024-03-11T21:30:39.292 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 5.7459999999999997e-06 2024-03-11T21:30:39.292 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-03-11T21:30:39.292 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-03-11T21:30:39.292 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-03-11T21:30:39.292 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:30:39.292 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:30:39.292 INFO:teuthology.orchestra.run.smithi092.stdout: "description": "osd_op(client.34715.0:199092 7.f 7:f1d8d2c9:::benchmark_data_smithi092_76040_object77181:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e136)", 2024-03-11T21:30:39.292 INFO:teuthology.orchestra.run.smithi092.stdout: "initiated_at": "2024-03-11T21:30:39.262629+0000", 2024-03-11T21:30:39.292 INFO:teuthology.orchestra.run.smithi092.stdout: "age": 0.031897396000000001, 2024-03-11T21:30:39.292 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.032166093, 2024-03-11T21:30:39.292 INFO:teuthology.orchestra.run.smithi092.stdout: "type_data": { 2024-03-11T21:30:39.292 INFO:teuthology.orchestra.run.smithi092.stdout: "flag_point": "waiting for sub ops", 2024-03-11T21:30:39.293 INFO:teuthology.orchestra.run.smithi092.stdout: "client_info": { 2024-03-11T21:30:39.293 INFO:teuthology.orchestra.run.smithi092.stdout: "client": "client.34715", 2024-03-11T21:30:39.293 INFO:teuthology.orchestra.run.smithi092.stdout: "client_addr": "v1:172.21.15.92:0/2403132573", 2024-03-11T21:30:39.293 INFO:teuthology.orchestra.run.smithi092.stdout: "tid": 199092 2024-03-11T21:30:39.293 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:30:39.293 INFO:teuthology.orchestra.run.smithi092.stdout: "events": [ 2024-03-11T21:30:39.294 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:30:39.294 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "initiated", 2024-03-11T21:30:39.294 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:30:39.262629+0000", 2024-03-11T21:30:39.294 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-03-11T21:30:39.294 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:30:39.294 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:30:39.294 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "header_read", 2024-03-11T21:30:39.294 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:30:39.262629+0000", 2024-03-11T21:30:39.294 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-03-11T21:30:39.294 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:30:39.294 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:30:39.294 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "throttled", 2024-03-11T21:30:39.294 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:30:39.262631+0000", 2024-03-11T21:30:39.295 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 2.0590000000000001e-06 2024-03-11T21:30:39.295 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:30:39.295 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:30:39.295 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "all_read", 2024-03-11T21:30:39.295 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:30:39.262637+0000", 2024-03-11T21:30:39.295 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 5.9920000000000004e-06 2024-03-11T21:30:39.295 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:30:39.295 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:30:39.295 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "dispatched", 2024-03-11T21:30:39.295 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:30:39.262646+0000", 2024-03-11T21:30:39.295 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 9.7289999999999995e-06 2024-03-11T21:30:39.295 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:30:39.295 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:30:39.295 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "queued_for_pg", 2024-03-11T21:30:39.296 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:30:39.262654+0000", 2024-03-11T21:30:39.296 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 7.3490000000000003e-06 2024-03-11T21:30:39.296 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:30:39.296 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:30:39.296 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-03-11T21:30:39.296 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:30:39.265353+0000", 2024-03-11T21:30:39.296 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.0026987460000000001 2024-03-11T21:30:39.296 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:30:39.296 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:30:39.296 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "started", 2024-03-11T21:30:39.296 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:30:39.265481+0000", 2024-03-11T21:30:39.296 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00012861799999999999 2024-03-11T21:30:39.296 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:30:39.296 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:30:39.296 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "waiting for subops from 3,7", 2024-03-11T21:30:39.297 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:30:39.265602+0000", 2024-03-11T21:30:39.297 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00012113299999999999 2024-03-11T21:30:39.297 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:30:39.297 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:30:39.297 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "op_commit", 2024-03-11T21:30:39.297 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:30:39.269992+0000", 2024-03-11T21:30:39.297 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.0043897509999999999 2024-03-11T21:30:39.297 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:30:39.297 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:30:39.297 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "sub_op_commit_rec", 2024-03-11T21:30:39.297 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:30:39.278503+0000", 2024-03-11T21:30:39.297 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.0085112060000000003 2024-03-11T21:30:39.297 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-03-11T21:30:39.297 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-03-11T21:30:39.297 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-03-11T21:30:39.298 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:30:39.298 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:30:39.298 INFO:teuthology.orchestra.run.smithi092.stdout: "description": "osd_op(client.34715.0:199100 7.4 7:2ff391b6:::benchmark_data_smithi092_76040_object77189:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e136)", 2024-03-11T21:30:39.298 INFO:teuthology.orchestra.run.smithi092.stdout: "initiated_at": "2024-03-11T21:30:39.278189+0000", 2024-03-11T21:30:39.298 INFO:teuthology.orchestra.run.smithi092.stdout: "age": 0.016336705, 2024-03-11T21:30:39.298 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.016694819999999999, 2024-03-11T21:30:39.298 INFO:teuthology.orchestra.run.smithi092.stdout: "type_data": { 2024-03-11T21:30:39.298 INFO:teuthology.orchestra.run.smithi092.stdout: "flag_point": "waiting for sub ops", 2024-03-11T21:30:39.298 INFO:teuthology.orchestra.run.smithi092.stdout: "client_info": { 2024-03-11T21:30:39.298 INFO:teuthology.orchestra.run.smithi092.stdout: "client": "client.34715", 2024-03-11T21:30:39.298 INFO:teuthology.orchestra.run.smithi092.stdout: "client_addr": "v1:172.21.15.92:0/2403132573", 2024-03-11T21:30:39.298 INFO:teuthology.orchestra.run.smithi092.stdout: "tid": 199100 2024-03-11T21:30:39.298 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:30:39.299 INFO:teuthology.orchestra.run.smithi092.stdout: "events": [ 2024-03-11T21:30:39.299 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:30:39.299 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "initiated", 2024-03-11T21:30:39.299 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:30:39.278189+0000", 2024-03-11T21:30:39.299 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-03-11T21:30:39.299 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:30:39.299 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:30:39.299 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "header_read", 2024-03-11T21:30:39.299 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:30:39.278189+0000", 2024-03-11T21:30:39.299 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-03-11T21:30:39.299 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:30:39.299 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:30:39.299 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "throttled", 2024-03-11T21:30:39.299 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:30:39.278191+0000", 2024-03-11T21:30:39.299 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.9240000000000001e-06 2024-03-11T21:30:39.300 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:30:39.300 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:30:39.300 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "all_read", 2024-03-11T21:30:39.300 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:30:39.278198+0000", 2024-03-11T21:30:39.300 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 7.1099999999999997e-06 2024-03-11T21:30:39.300 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:30:39.300 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:30:39.300 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "dispatched", 2024-03-11T21:30:39.300 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:30:39.278209+0000", 2024-03-11T21:30:39.300 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.0319000000000001e-05 2024-03-11T21:30:39.300 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:30:39.300 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:30:39.300 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "queued_for_pg", 2024-03-11T21:30:39.300 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:30:39.278217+0000", 2024-03-11T21:30:39.301 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 8.0039999999999999e-06 2024-03-11T21:30:39.301 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:30:39.301 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:30:39.301 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-03-11T21:30:39.301 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:30:39.281831+0000", 2024-03-11T21:30:39.301 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.0036139779999999999 2024-03-11T21:30:39.301 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:30:39.301 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:30:39.301 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "started", 2024-03-11T21:30:39.301 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:30:39.281966+0000", 2024-03-11T21:30:39.301 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00013511400000000001 2024-03-11T21:30:39.301 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:30:39.301 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:30:39.301 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "waiting for subops from 1,4", 2024-03-11T21:30:39.301 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:30:39.282102+0000", 2024-03-11T21:30:39.302 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00013599400000000001 2024-03-11T21:30:39.302 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:30:39.302 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:30:39.302 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "sub_op_commit_rec", 2024-03-11T21:30:39.302 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:30:39.294845+0000", 2024-03-11T21:30:39.302 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.012742903999999999 2024-03-11T21:30:39.302 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:30:39.302 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:30:39.302 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "op_commit", 2024-03-11T21:30:39.302 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:30:39.294857+0000", 2024-03-11T21:30:39.302 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.1891e-05 2024-03-11T21:30:39.302 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:30:39.302 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:30:39.303 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "sub_op_commit_rec", 2024-03-11T21:30:39.303 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:30:39.294888+0000", 2024-03-11T21:30:39.303 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 3.1059000000000002e-05 2024-03-11T21:30:39.303 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-03-11T21:30:39.303 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-03-11T21:30:39.303 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-03-11T21:30:39.303 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:30:39.303 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:30:39.303 INFO:teuthology.orchestra.run.smithi092.stdout: "description": "osd_op(client.34715.0:199104 7.9 7:9ce7f82d:::benchmark_data_smithi092_76040_object77193:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e136)", 2024-03-11T21:30:39.303 INFO:teuthology.orchestra.run.smithi092.stdout: "initiated_at": "2024-03-11T21:30:39.278226+0000", 2024-03-11T21:30:39.303 INFO:teuthology.orchestra.run.smithi092.stdout: "age": 0.016300149999999999, 2024-03-11T21:30:39.303 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.016767792, 2024-03-11T21:30:39.303 INFO:teuthology.orchestra.run.smithi092.stdout: "type_data": { 2024-03-11T21:30:39.303 INFO:teuthology.orchestra.run.smithi092.stdout: "flag_point": "waiting for sub ops", 2024-03-11T21:30:39.303 INFO:teuthology.orchestra.run.smithi092.stdout: "client_info": { 2024-03-11T21:30:39.304 INFO:teuthology.orchestra.run.smithi092.stdout: "client": "client.34715", 2024-03-11T21:30:39.304 INFO:teuthology.orchestra.run.smithi092.stdout: "client_addr": "v1:172.21.15.92:0/2403132573", 2024-03-11T21:30:39.304 INFO:teuthology.orchestra.run.smithi092.stdout: "tid": 199104 2024-03-11T21:30:39.304 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:30:39.304 INFO:teuthology.orchestra.run.smithi092.stdout: "events": [ 2024-03-11T21:30:39.304 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:30:39.304 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "initiated", 2024-03-11T21:30:39.304 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:30:39.278226+0000", 2024-03-11T21:30:39.304 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-03-11T21:30:39.304 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:30:39.304 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:30:39.304 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "header_read", 2024-03-11T21:30:39.304 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:30:39.278226+0000", 2024-03-11T21:30:39.304 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-03-11T21:30:39.304 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:30:39.305 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:30:39.305 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "throttled", 2024-03-11T21:30:39.305 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:30:39.278227+0000", 2024-03-11T21:30:39.305 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 5.8999999999999996e-07 2024-03-11T21:30:39.305 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:30:39.305 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:30:39.305 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "all_read", 2024-03-11T21:30:39.305 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:30:39.278229+0000", 2024-03-11T21:30:39.305 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 2.9670000000000001e-06 2024-03-11T21:30:39.305 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:30:39.305 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:30:39.305 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "dispatched", 2024-03-11T21:30:39.305 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:30:39.278235+0000", 2024-03-11T21:30:39.305 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 5.4790000000000004e-06 2024-03-11T21:30:39.305 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:30:39.306 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:30:39.306 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "queued_for_pg", 2024-03-11T21:30:39.306 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:30:39.278240+0000", 2024-03-11T21:30:39.306 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 5.4090000000000002e-06 2024-03-11T21:30:39.306 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:30:39.306 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:30:39.306 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-03-11T21:30:39.306 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:30:39.278372+0000", 2024-03-11T21:30:39.306 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.000131901 2024-03-11T21:30:39.306 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:30:39.306 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:30:39.306 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "started", 2024-03-11T21:30:39.306 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:30:39.278523+0000", 2024-03-11T21:30:39.306 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00015097099999999999 2024-03-11T21:30:39.307 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:30:39.307 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:30:39.307 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "waiting for subops from 1,4", 2024-03-11T21:30:39.307 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:30:39.278663+0000", 2024-03-11T21:30:39.307 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00013951 2024-03-11T21:30:39.307 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:30:39.307 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:30:39.307 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "sub_op_commit_rec", 2024-03-11T21:30:39.307 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:30:39.286872+0000", 2024-03-11T21:30:39.307 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.0082097309999999996 2024-03-11T21:30:39.307 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:30:39.307 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:30:39.307 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "sub_op_commit_rec", 2024-03-11T21:30:39.307 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:30:39.294076+0000", 2024-03-11T21:30:39.308 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.0072032759999999998 2024-03-11T21:30:39.308 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-03-11T21:30:39.308 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-03-11T21:30:39.308 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-03-11T21:30:39.308 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:30:39.308 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:30:39.308 INFO:teuthology.orchestra.run.smithi092.stdout: "description": "osd_repop(client.34715.0:199097 7.e e136/99 7:78c095a7:::benchmark_data_smithi092_76040_object77186:head v 136'12521, mlcod=136'12519)", 2024-03-11T21:30:39.308 INFO:teuthology.orchestra.run.smithi092.stdout: "initiated_at": "2024-03-11T21:30:39.278268+0000", 2024-03-11T21:30:39.308 INFO:teuthology.orchestra.run.smithi092.stdout: "age": 0.016257799999999999, 2024-03-11T21:30:39.308 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.016896893999999999, 2024-03-11T21:30:39.308 INFO:teuthology.orchestra.run.smithi092.stdout: "type_data": { 2024-03-11T21:30:39.308 INFO:teuthology.orchestra.run.smithi092.stdout: "flag_point": "started", 2024-03-11T21:30:39.308 INFO:teuthology.orchestra.run.smithi092.stdout: "events": [ 2024-03-11T21:30:39.308 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:30:39.308 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "initiated", 2024-03-11T21:30:39.309 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:30:39.278268+0000", 2024-03-11T21:30:39.309 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-03-11T21:30:39.309 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:30:39.309 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:30:39.309 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "header_read", 2024-03-11T21:30:39.309 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:30:39.278268+0000", 2024-03-11T21:30:39.309 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-03-11T21:30:39.309 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:30:39.309 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:30:39.309 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "throttled", 2024-03-11T21:30:39.309 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:30:39.278269+0000", 2024-03-11T21:30:39.309 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 7.85e-07 2024-03-11T21:30:39.309 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:30:39.310 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:30:39.310 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "all_read", 2024-03-11T21:30:39.310 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:30:39.278278+0000", 2024-03-11T21:30:39.310 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 9.081e-06 2024-03-11T21:30:39.310 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:30:39.310 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:30:39.310 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "dispatched", 2024-03-11T21:30:39.310 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:30:39.278286+0000", 2024-03-11T21:30:39.310 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 7.4399999999999999e-06 2024-03-11T21:30:39.310 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:30:39.310 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:30:39.310 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "queued_for_pg", 2024-03-11T21:30:39.310 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:30:39.278291+0000", 2024-03-11T21:30:39.311 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 5.609e-06 2024-03-11T21:30:39.311 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:30:39.311 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:30:39.311 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-03-11T21:30:39.311 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:30:39.278547+0000", 2024-03-11T21:30:39.311 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00025591200000000001 2024-03-11T21:30:39.311 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:30:39.311 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:30:39.311 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "started", 2024-03-11T21:30:39.311 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:30:39.278571+0000", 2024-03-11T21:30:39.311 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 2.3949000000000001e-05 2024-03-11T21:30:39.311 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-03-11T21:30:39.311 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-03-11T21:30:39.312 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-03-11T21:30:39.312 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:30:39.312 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:30:39.312 INFO:teuthology.orchestra.run.smithi092.stdout: "description": "osd_repop(client.34715.0:199101 7.e e136/99 7:7f316912:::benchmark_data_smithi092_76040_object77190:head v 136'12522, mlcod=136'12519)", 2024-03-11T21:30:39.312 INFO:teuthology.orchestra.run.smithi092.stdout: "initiated_at": "2024-03-11T21:30:39.278306+0000", 2024-03-11T21:30:39.312 INFO:teuthology.orchestra.run.smithi092.stdout: "age": 0.016220104999999999, 2024-03-11T21:30:39.312 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.016919836000000001, 2024-03-11T21:30:39.312 INFO:teuthology.orchestra.run.smithi092.stdout: "type_data": { 2024-03-11T21:30:39.312 INFO:teuthology.orchestra.run.smithi092.stdout: "flag_point": "started", 2024-03-11T21:30:39.312 INFO:teuthology.orchestra.run.smithi092.stdout: "events": [ 2024-03-11T21:30:39.312 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:30:39.312 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "initiated", 2024-03-11T21:30:39.312 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:30:39.278306+0000", 2024-03-11T21:30:39.313 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-03-11T21:30:39.313 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:30:39.313 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:30:39.313 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "header_read", 2024-03-11T21:30:39.313 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:30:39.278306+0000", 2024-03-11T21:30:39.313 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-03-11T21:30:39.313 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:30:39.313 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:30:39.313 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "throttled", 2024-03-11T21:30:39.313 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:30:39.278307+0000", 2024-03-11T21:30:39.313 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 5.51e-07 2024-03-11T21:30:39.313 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:30:39.313 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:30:39.313 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "all_read", 2024-03-11T21:30:39.314 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:30:39.278311+0000", 2024-03-11T21:30:39.314 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 4.4309999999999996e-06 2024-03-11T21:30:39.314 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:30:39.314 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:30:39.314 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "dispatched", 2024-03-11T21:30:39.314 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:30:39.278317+0000", 2024-03-11T21:30:39.314 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 5.8799999999999996e-06 2024-03-11T21:30:39.314 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:30:39.314 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:30:39.314 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "queued_for_pg", 2024-03-11T21:30:39.314 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:30:39.278322+0000", 2024-03-11T21:30:39.314 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 4.8570000000000003e-06 2024-03-11T21:30:39.314 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:30:39.315 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:30:39.315 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-03-11T21:30:39.315 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:30:39.286752+0000", 2024-03-11T21:30:39.315 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.0084300499999999997 2024-03-11T21:30:39.315 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:30:39.315 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:30:39.315 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "started", 2024-03-11T21:30:39.315 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:30:39.286775+0000", 2024-03-11T21:30:39.315 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 2.2878e-05 2024-03-11T21:30:39.315 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-03-11T21:30:39.315 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-03-11T21:30:39.315 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-03-11T21:30:39.315 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:30:39.315 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:30:39.316 INFO:teuthology.orchestra.run.smithi092.stdout: "description": "osd_repop(client.34715.0:199107 7.7 e136/99 7:ec882fa9:::benchmark_data_smithi092_76040_object77196:head v 136'12519, mlcod=136'12517)", 2024-03-11T21:30:39.316 INFO:teuthology.orchestra.run.smithi092.stdout: "initiated_at": "2024-03-11T21:30:39.289027+0000", 2024-03-11T21:30:39.316 INFO:teuthology.orchestra.run.smithi092.stdout: "age": 0.005499065, 2024-03-11T21:30:39.316 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.0062772460000000002, 2024-03-11T21:30:39.316 INFO:teuthology.orchestra.run.smithi092.stdout: "type_data": { 2024-03-11T21:30:39.316 INFO:teuthology.orchestra.run.smithi092.stdout: "flag_point": "started", 2024-03-11T21:30:39.316 INFO:teuthology.orchestra.run.smithi092.stdout: "events": [ 2024-03-11T21:30:39.316 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:30:39.316 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "initiated", 2024-03-11T21:30:39.316 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:30:39.289027+0000", 2024-03-11T21:30:39.316 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-03-11T21:30:39.316 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:30:39.317 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:30:39.317 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "header_read", 2024-03-11T21:30:39.317 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:30:39.289027+0000", 2024-03-11T21:30:39.317 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-03-11T21:30:39.317 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:30:39.317 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:30:39.317 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "throttled", 2024-03-11T21:30:39.317 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:30:39.289029+0000", 2024-03-11T21:30:39.317 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.8700000000000001e-06 2024-03-11T21:30:39.318 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:30:39.318 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:30:39.318 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "all_read", 2024-03-11T21:30:39.318 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:30:39.289038+0000", 2024-03-11T21:30:39.318 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 8.6659999999999997e-06 2024-03-11T21:30:39.318 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:30:39.318 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:30:39.318 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "dispatched", 2024-03-11T21:30:39.318 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:30:39.289056+0000", 2024-03-11T21:30:39.318 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.8343999999999999e-05 2024-03-11T21:30:39.319 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:30:39.319 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:30:39.319 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "queued_for_pg", 2024-03-11T21:30:39.319 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:30:39.289065+0000", 2024-03-11T21:30:39.319 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 8.8249999999999994e-06 2024-03-11T21:30:39.319 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:30:39.319 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:30:39.319 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-03-11T21:30:39.319 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:30:39.294463+0000", 2024-03-11T21:30:39.319 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.0053983529999999998 2024-03-11T21:30:39.319 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:30:39.319 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:30:39.319 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "started", 2024-03-11T21:30:39.319 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:30:39.294485+0000", 2024-03-11T21:30:39.319 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 2.2107e-05 2024-03-11T21:30:39.320 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-03-11T21:30:39.320 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-03-11T21:30:39.320 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-03-11T21:30:39.320 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-03-11T21:30:39.320 INFO:teuthology.orchestra.run.smithi092.stdout: ], 2024-03-11T21:30:39.320 INFO:teuthology.orchestra.run.smithi092.stdout: "num_ops": 10 2024-03-11T21:30:39.320 INFO:teuthology.orchestra.run.smithi092.stdout:} 2024-03-11T21:30:39.320 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:30:39.321 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:30:39.321 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:30:39.422 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:30:39.449 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:30:39.450 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:30:39.450 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:30:39.551 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:30:39.587 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:30:39.588 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:30:39.588 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:30:39.689 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:30:39.733 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:30:39.733 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:30:39.734 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:30:39.835 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:30:39.866 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:30:39.867 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:30:39.867 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:30:39.968 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:30:40.014 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:30:40.014 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:30:40.015 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:30:40.115 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:30:40.143 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:30:40.143 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:30:40.144 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:30:40.245 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:30:40.275 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:30:40.276 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:30:40.276 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:30:40.377 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:30:40.414 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:30:40.415 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:30:40.415 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:30:40.453 INFO:teuthology.orchestra.run.smithi092.stderr:nodeep-scrub is set 2024-03-11T21:30:40.516 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:30:40.546 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:30:40.547 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:30:40.547 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:30:40.648 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:30:40.675 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:30:40.676 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:30:40.676 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:30:40.777 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:30:40.812 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:30:40.813 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:30:40.813 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:30:40.826 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:30:40 smithi097 ceph-mon[33317]: pgmap v181: 148 pgs: 148 active+clean; 2.9 GiB data, 9.5 GiB used, 706 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 1.00k op/s 2024-03-11T21:30:40.826 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:30:40 smithi097 ceph-mon[33317]: from='client.? v1:172.21.15.92:0/2666352591' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-11T21:30:40.860 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:30:40 smithi092 ceph-mon[29572]: pgmap v181: 148 pgs: 148 active+clean; 2.9 GiB data, 9.5 GiB used, 706 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 1.00k op/s 2024-03-11T21:30:40.861 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:30:40 smithi092 ceph-mon[29572]: from='client.? v1:172.21.15.92:0/2666352591' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-11T21:30:40.861 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:30:40 smithi092 ceph-mon[37563]: pgmap v181: 148 pgs: 148 active+clean; 2.9 GiB data, 9.5 GiB used, 706 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 1.00k op/s 2024-03-11T21:30:40.861 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:30:40 smithi092 ceph-mon[37563]: from='client.? v1:172.21.15.92:0/2666352591' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-11T21:30:40.913 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:30:40.940 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:30:40.941 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:30:40.941 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:30:41.042 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:30:41.086 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:30:41.087 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:30:41.087 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:30:41.188 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:30:41.235 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:30:41.235 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:30:41.235 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:30:41.336 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:30:41.369 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:30:41.370 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:30:41.370 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:30:41.471 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:30:41.502 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:30:41.503 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:30:41.503 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:30:41.551 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:30:41 smithi092 ceph-mon[29572]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-11T21:30:41.551 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:30:41 smithi092 ceph-mon[29572]: from='client.? v1:172.21.15.92:0/2666352591' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-03-11T21:30:41.551 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:30:41 smithi092 ceph-mon[29572]: osdmap e137: 8 total, 8 up, 8 in 2024-03-11T21:30:41.551 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:30:41 smithi092 ceph-mon[37563]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-11T21:30:41.552 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:30:41 smithi092 ceph-mon[37563]: from='client.? v1:172.21.15.92:0/2666352591' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-03-11T21:30:41.552 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:30:41 smithi092 ceph-mon[37563]: osdmap e137: 8 total, 8 up, 8 in 2024-03-11T21:30:41.568 INFO:teuthology.orchestra.run.smithi092.stderr:Inferring config /var/lib/ceph/00db0016-dfed-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T21:30:41.604 DEBUG:teuthology.orchestra.run.smithi092:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid 00db0016-dfed-11ee-95c8-87774f69a715 -- ceph daemon osd.0 dump_blocked_ops 2024-03-11T21:30:41.605 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:30:41.634 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:30:41.634 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:30:41.634 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:30:41.735 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:30:41.764 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:30:41.765 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:30:41.765 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:30:41.826 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:30:41 smithi097 ceph-mon[33317]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-11T21:30:41.826 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:30:41 smithi097 ceph-mon[33317]: from='client.? v1:172.21.15.92:0/2666352591' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-03-11T21:30:41.826 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:30:41 smithi097 ceph-mon[33317]: osdmap e137: 8 total, 8 up, 8 in 2024-03-11T21:30:41.866 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:30:41.930 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:30:41.931 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:30:41.931 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:30:42.032 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:30:42.075 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:30:42.076 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:30:42.076 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:30:42.177 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:30:42.200 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:30:42.201 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:30:42.201 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:30:42.302 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:30:42.351 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:30:42.351 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:30:42.351 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:30:42.452 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:30:42.480 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:30:42.481 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:30:42.481 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:30:42.576 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:30:42 smithi097 ceph-mon[33317]: pgmap v183: 148 pgs: 148 active+clean; 2.9 GiB data, 9.5 GiB used, 706 GiB / 715 GiB avail; 639 B/s rd, 0 B/s wr, 648 op/s 2024-03-11T21:30:42.577 INFO:journalctl@ceph.iscsi.iscsi.a.smithi097.stdout:Mar 11 21:30:42 smithi097 ceph-00db0016-dfed-11ee-95c8-87774f69a715-iscsi-iscsi-a[60943]: debug there is no tcmu-runner data available 2024-03-11T21:30:42.582 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:30:42.610 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:30:42 smithi092 ceph-mon[29572]: pgmap v183: 148 pgs: 148 active+clean; 2.9 GiB data, 9.5 GiB used, 706 GiB / 715 GiB avail; 639 B/s rd, 0 B/s wr, 648 op/s 2024-03-11T21:30:42.611 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:30:42 smithi092 ceph-mon[37563]: pgmap v183: 148 pgs: 148 active+clean; 2.9 GiB data, 9.5 GiB used, 706 GiB / 715 GiB avail; 639 B/s rd, 0 B/s wr, 648 op/s 2024-03-11T21:30:42.623 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:30:42.624 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:30:42.624 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:30:42.725 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:30:42.766 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:30:42.767 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:30:42.768 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:30:42.868 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:30:42.923 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:30:42.924 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:30:42.924 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:30:43.025 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:30:43.073 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:30:43.073 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:30:43.074 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:30:43.174 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:30:43.222 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:30:43.223 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:30:43.223 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:30:43.324 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:30:43.353 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:30:43.354 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:30:43.355 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:30:43.455 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:30:43.500 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:30:43.500 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:30:43.500 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:30:43.601 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:30:43.610 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:30:43 smithi092 ceph-mon[29572]: from='client.24469 v1:172.21.15.97:0/404208310' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:30:43.610 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:30:43 smithi092 ceph-mon[37563]: from='client.24469 v1:172.21.15.97:0/404208310' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:30:43.627 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:30:43.628 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:30:43.628 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:30:43.729 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:30:43.753 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:30:43.754 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:30:43.754 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:30:43.826 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:30:43 smithi097 ceph-mon[33317]: from='client.24469 v1:172.21.15.97:0/404208310' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:30:43.856 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:30:43.894 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:30:43.895 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:30:43.895 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:30:43.996 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:30:44.024 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:30:44.025 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:30:44.026 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:30:44.127 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:30:44.171 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:30:44.171 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:30:44.171 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:30:44.272 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:30:44.313 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:30:44.313 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:30:44.313 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:30:44.414 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:30:44.448 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:30:44.449 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:30:44.450 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:30:44.551 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:30:44.593 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:30:44.593 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:30:44.594 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:30:44.662 DEBUG:teuthology.orchestra.run.smithi092:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid 00db0016-dfed-11ee-95c8-87774f69a715 -- ceph osd unset noscrub 2024-03-11T21:30:44.695 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:30:44.756 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:30:44.756 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:30:44.757 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:30:44.788 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:30:44 smithi092 ceph-mon[29572]: pgmap v184: 148 pgs: 148 active+clean; 2.5 GiB data, 8.6 GiB used, 707 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.30k op/s 2024-03-11T21:30:44.789 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:30:44 smithi092 ceph-mon[37563]: pgmap v184: 148 pgs: 148 active+clean; 2.5 GiB data, 8.6 GiB used, 707 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.30k op/s 2024-03-11T21:30:44.826 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:30:44 smithi097 ceph-mon[33317]: pgmap v184: 148 pgs: 148 active+clean; 2.5 GiB data, 8.6 GiB used, 707 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.30k op/s 2024-03-11T21:30:44.858 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:30:44.883 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:30:44.884 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:30:44.884 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:30:44.986 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:30:45.016 INFO:teuthology.orchestra.run.smithi092.stderr:Inferring config /var/lib/ceph/00db0016-dfed-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T21:30:45.028 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:30:45.028 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:30:45.029 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:30:45.129 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:30:45.138 INFO:teuthology.orchestra.run.smithi092.stdout:osd.0: {} 2024-03-11T21:30:45.157 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:30:45.158 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:30:45.158 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:30:45.183 INFO:teuthology.orchestra.run.smithi092.stdout:osd.1: {} 2024-03-11T21:30:45.223 INFO:teuthology.orchestra.run.smithi092.stdout:osd.2: {} 2024-03-11T21:30:45.259 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:30:45.269 INFO:teuthology.orchestra.run.smithi092.stdout:osd.3: {} 2024-03-11T21:30:45.287 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:30:45.288 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:30:45.289 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:30:45.315 INFO:teuthology.orchestra.run.smithi092.stdout:osd.4: {} 2024-03-11T21:30:45.356 INFO:teuthology.orchestra.run.smithi092.stdout:osd.5: {} 2024-03-11T21:30:45.389 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:30:45.398 INFO:teuthology.orchestra.run.smithi092.stdout:osd.6: {} 2024-03-11T21:30:45.435 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:30:45.435 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:30:45.436 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:30:45.436 INFO:teuthology.orchestra.run.smithi092.stdout:osd.7: {} 2024-03-11T21:30:45.446 INFO:teuthology.orchestra.run.smithi092.stderr:osd.0: osd_enable_op_tracker = '' 2024-03-11T21:30:45.446 INFO:teuthology.orchestra.run.smithi092.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-11T21:30:45.446 INFO:teuthology.orchestra.run.smithi092.stderr:osd.2: osd_enable_op_tracker = '' 2024-03-11T21:30:45.447 INFO:teuthology.orchestra.run.smithi092.stderr:osd.3: osd_enable_op_tracker = '' 2024-03-11T21:30:45.447 INFO:teuthology.orchestra.run.smithi092.stderr:osd.4: osd_enable_op_tracker = '' 2024-03-11T21:30:45.447 INFO:teuthology.orchestra.run.smithi092.stderr:osd.5: osd_enable_op_tracker = '' 2024-03-11T21:30:45.447 INFO:teuthology.orchestra.run.smithi092.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-11T21:30:45.447 INFO:teuthology.orchestra.run.smithi092.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-11T21:30:45.537 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:30:45.579 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:30:45.580 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:30:45.580 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:30:45.610 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:30:45 smithi092 ceph-mon[37563]: purged_snaps scrub starts 2024-03-11T21:30:45.610 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:30:45 smithi092 ceph-mon[37563]: purged_snaps scrub ok 2024-03-11T21:30:45.611 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:30:45 smithi092 ceph-mon[37563]: osdmap e138: 8 total, 8 up, 8 in 2024-03-11T21:30:45.611 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:30:45 smithi092 ceph-mon[37563]: from='client.? v1:172.21.15.92:0/3872948796' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-11T21:30:45.611 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:30:45 smithi092 ceph-mon[29572]: purged_snaps scrub starts 2024-03-11T21:30:45.611 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:30:45 smithi092 ceph-mon[29572]: purged_snaps scrub ok 2024-03-11T21:30:45.611 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:30:45 smithi092 ceph-mon[29572]: osdmap e138: 8 total, 8 up, 8 in 2024-03-11T21:30:45.611 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:30:45 smithi092 ceph-mon[29572]: from='client.? v1:172.21.15.92:0/3872948796' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-11T21:30:45.681 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:30:45.735 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:30:45.736 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:30:45.736 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:30:45.826 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:30:45 smithi097 ceph-mon[33317]: purged_snaps scrub starts 2024-03-11T21:30:45.827 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:30:45 smithi097 ceph-mon[33317]: purged_snaps scrub ok 2024-03-11T21:30:45.827 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:30:45 smithi097 ceph-mon[33317]: osdmap e138: 8 total, 8 up, 8 in 2024-03-11T21:30:45.827 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:30:45 smithi097 ceph-mon[33317]: from='client.? v1:172.21.15.92:0/3872948796' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-11T21:30:45.837 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:30:45.866 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:30:45.867 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:30:45.867 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:30:45.968 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:30:45.991 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:30:45.992 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:30:45.992 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:30:46.093 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:30:46.121 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:30:46.121 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:30:46.122 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:30:46.222 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:30:46.253 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:30:46.254 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:30:46.254 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:30:46.355 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:30:46.400 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:30:46.401 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:30:46.401 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:30:46.502 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:30:46.537 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:30:46.538 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:30:46.539 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:30:46.610 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:30:46 smithi092 ceph-mon[37563]: pgmap v186: 148 pgs: 148 active+clean; 2.5 GiB data, 8.6 GiB used, 707 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.06k op/s 2024-03-11T21:30:46.611 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:30:46 smithi092 ceph-mon[29572]: pgmap v186: 148 pgs: 148 active+clean; 2.5 GiB data, 8.6 GiB used, 707 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.06k op/s 2024-03-11T21:30:46.639 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:30:46.685 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:30:46.685 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:30:46.686 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:30:46.787 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:30:46.815 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:30:46.815 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:30:46.816 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:30:46.826 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:30:46 smithi097 ceph-mon[33317]: pgmap v186: 148 pgs: 148 active+clean; 2.5 GiB data, 8.6 GiB used, 707 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.06k op/s 2024-03-11T21:30:46.917 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:30:46.980 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:30:46.981 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:30:46.981 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:30:47.082 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:30:47.111 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:30:47.111 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:30:47.112 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:30:47.212 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:30:47.249 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:30:47.249 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:30:47.250 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:30:47.350 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:30:47.380 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:30:47.381 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:30:47.381 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:30:47.482 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:30:47.513 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:30:47.514 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:30:47.514 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:30:47.521 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:30:47 smithi092 ceph-mon[37563]: from='client.? v1:172.21.15.92:0/1331496587' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]: dispatch 2024-03-11T21:30:47.521 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:30:47 smithi092 ceph-mon[37563]: from='client.34910 ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]: dispatch 2024-03-11T21:30:47.521 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:30:47 smithi092 ceph-mon[29572]: from='client.? v1:172.21.15.92:0/1331496587' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]: dispatch 2024-03-11T21:30:47.521 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:30:47 smithi092 ceph-mon[29572]: from='client.34910 ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]: dispatch 2024-03-11T21:30:47.521 INFO:teuthology.orchestra.run.smithi092.stderr:reweighted osd.5 to 1 (10000) 2024-03-11T21:30:47.615 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:30:47.654 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:30:47.654 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:30:47.655 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:30:47.756 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:30:47.800 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:30:47.801 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:30:47.801 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:30:47.826 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:30:47 smithi097 ceph-mon[33317]: from='client.? v1:172.21.15.92:0/1331496587' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]: dispatch 2024-03-11T21:30:47.826 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:30:47 smithi097 ceph-mon[33317]: from='client.34910 ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]: dispatch 2024-03-11T21:30:47.902 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:30:47.946 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:30:47.947 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:30:47.948 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:30:48.048 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:30:48.071 INFO:teuthology.orchestra.run.smithi092.stdout:ERROR: (22) Invalid argument 2024-03-11T21:30:48.071 INFO:teuthology.orchestra.run.smithi092.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-11T21:30:48.092 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:30:48.092 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:30:48.092 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:30:48.193 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:30:48.223 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:30:48.224 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:30:48.224 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:30:48.325 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:30:48.353 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:30:48.353 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:30:48.354 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:30:48.360 INFO:journalctl@ceph.mgr.y.smithi092.stdout:Mar 11 21:30:47 smithi092 ceph-00db0016-dfed-11ee-95c8-87774f69a715-mgr-y[29801]: ::ffff:172.21.15.97 - - [11/Mar/2024:21:30:47] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-11T21:30:48.455 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:30:48.489 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:30:48.489 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:30:48.490 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:30:48.590 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:30:48.635 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:30:48.636 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:30:48.636 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:30:48.737 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:30:48.776 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:30:48.777 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:30:48.777 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:30:48.814 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:30:48 smithi092 ceph-mon[29572]: pgmap v187: 148 pgs: 148 active+clean; 2.4 GiB data, 8.2 GiB used, 707 GiB / 715 GiB avail; 639 B/s rd, 0 B/s wr, 892 op/s 2024-03-11T21:30:48.814 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:30:48 smithi092 ceph-mon[29572]: from='client.34910 ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]': finished 2024-03-11T21:30:48.814 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:30:48 smithi092 ceph-mon[29572]: osdmap e139: 8 total, 8 up, 8 in 2024-03-11T21:30:48.814 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:30:48 smithi092 ceph-mon[37563]: pgmap v187: 148 pgs: 148 active+clean; 2.4 GiB data, 8.2 GiB used, 707 GiB / 715 GiB avail; 639 B/s rd, 0 B/s wr, 892 op/s 2024-03-11T21:30:48.814 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:30:48 smithi092 ceph-mon[37563]: from='client.34910 ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]': finished 2024-03-11T21:30:48.814 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:30:48 smithi092 ceph-mon[37563]: osdmap e139: 8 total, 8 up, 8 in 2024-03-11T21:30:48.826 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:30:48 smithi097 ceph-mon[33317]: pgmap v187: 148 pgs: 148 active+clean; 2.4 GiB data, 8.2 GiB used, 707 GiB / 715 GiB avail; 639 B/s rd, 0 B/s wr, 892 op/s 2024-03-11T21:30:48.826 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:30:48 smithi097 ceph-mon[33317]: from='client.34910 ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]': finished 2024-03-11T21:30:48.826 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:30:48 smithi097 ceph-mon[33317]: osdmap e139: 8 total, 8 up, 8 in 2024-03-11T21:30:48.862 DEBUG:teuthology.orchestra.run.smithi092:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid 00db0016-dfed-11ee-95c8-87774f69a715 -- ceph osd reweight 6 1 2024-03-11T21:30:48.878 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:30:48.879 INFO:teuthology.orchestra.run.smithi092.stderr:Inferring config /var/lib/ceph/00db0016-dfed-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T21:30:48.945 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:30:48.945 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:30:48.945 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:30:49.046 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:30:49.075 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:30:49.076 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:30:49.076 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:30:49.177 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:30:49.223 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:30:49.223 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:30:49.223 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:30:49.298 DEBUG:teuthology.orchestra.run.smithi092:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid 00db0016-dfed-11ee-95c8-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-03-11T21:30:49.325 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:30:49.351 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:30:49.352 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:30:49.352 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:30:49.358 DEBUG:teuthology.orchestra.run.smithi092:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid 00db0016-dfed-11ee-95c8-87774f69a715 -- ceph daemon osd.0 dump_historic_ops 2024-03-11T21:30:49.453 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:30:49.571 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:30:49.571 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:30:49.572 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:30:49.610 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:30:49 smithi092 ceph-mon[29572]: from='mgr.14607 v1:172.21.15.92:0/1702717597' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T21:30:49.613 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:30:49 smithi092 ceph-mon[37563]: from='mgr.14607 v1:172.21.15.92:0/1702717597' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T21:30:49.673 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:30:49.707 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:30:49.708 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:30:49.708 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:30:49.809 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:30:49.826 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:30:49 smithi097 ceph-mon[33317]: from='mgr.14607 v1:172.21.15.92:0/1702717597' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T21:30:49.849 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:30:49.850 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:30:49.850 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:30:49.951 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:30:50.008 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:30:50.009 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:30:50.009 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:30:50.110 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:30:50.160 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:30:50.161 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:30:50.161 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:30:50.262 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:30:50.301 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:30:50.301 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:30:50.301 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:30:50.402 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:30:50.443 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:30:50.444 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:30:50.444 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:30:50.460 INFO:teuthology.orchestra.run.smithi092.stderr:Inferring config /var/lib/ceph/00db0016-dfed-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T21:30:50.546 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:30:50.578 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:30:50.579 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:30:50.579 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:30:50.680 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:30:50.708 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:30:50.708 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:30:50.709 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:30:50.727 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:30:50 smithi097 ceph-mon[33317]: pgmap v189: 148 pgs: 148 active+clean; 2.2 GiB data, 7.7 GiB used, 708 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.29k op/s 2024-03-11T21:30:50.810 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:30:50.828 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:30:50 smithi092 ceph-mon[29572]: pgmap v189: 148 pgs: 148 active+clean; 2.2 GiB data, 7.7 GiB used, 708 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.29k op/s 2024-03-11T21:30:50.829 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:30:50 smithi092 ceph-mon[37563]: pgmap v189: 148 pgs: 148 active+clean; 2.2 GiB data, 7.7 GiB used, 708 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.29k op/s 2024-03-11T21:30:50.862 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:30:50.863 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:30:50.863 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:30:50.964 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:30:51.011 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:30:51.012 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:30:51.012 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:30:51.113 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:30:51.153 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:30:51.153 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:30:51.154 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:30:51.254 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:30:51.291 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:30:51.291 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:30:51.291 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:30:51.392 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:30:51.432 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:30:51.433 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:30:51.433 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:30:51.534 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:30:51.577 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:30:51.577 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:30:51.578 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:30:51.678 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:30:51.706 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:30:51.707 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:30:51.707 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:30:51.729 INFO:teuthology.orchestra.run.smithi092.stderr:Inferring config /var/lib/ceph/00db0016-dfed-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T21:30:51.729 INFO:teuthology.orchestra.run.smithi092.stderr:Inferring config /var/lib/ceph/00db0016-dfed-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T21:30:51.808 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:30:51.826 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:30:51 smithi097 ceph-mon[33317]: purged_snaps scrub starts 2024-03-11T21:30:51.826 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:30:51 smithi097 ceph-mon[33317]: purged_snaps scrub ok 2024-03-11T21:30:51.826 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:30:51 smithi097 ceph-mon[33317]: osdmap e140: 8 total, 8 up, 8 in 2024-03-11T21:30:51.827 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:30:51 smithi097 ceph-mon[33317]: purged_snaps scrub starts 2024-03-11T21:30:51.827 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:30:51 smithi097 ceph-mon[33317]: purged_snaps scrub ok 2024-03-11T21:30:51.827 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:30:51 smithi097 ceph-mon[33317]: from='mgr.14607 v1:172.21.15.92:0/1702717597' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T21:30:51.849 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:30:51 smithi092 ceph-mon[29572]: purged_snaps scrub starts 2024-03-11T21:30:51.849 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:30:51 smithi092 ceph-mon[29572]: purged_snaps scrub ok 2024-03-11T21:30:51.849 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:30:51 smithi092 ceph-mon[29572]: osdmap e140: 8 total, 8 up, 8 in 2024-03-11T21:30:51.849 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:30:51 smithi092 ceph-mon[29572]: purged_snaps scrub starts 2024-03-11T21:30:51.849 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:30:51 smithi092 ceph-mon[29572]: purged_snaps scrub ok 2024-03-11T21:30:51.849 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:30:51 smithi092 ceph-mon[29572]: from='mgr.14607 v1:172.21.15.92:0/1702717597' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T21:30:51.850 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:30:51 smithi092 ceph-mon[37563]: purged_snaps scrub starts 2024-03-11T21:30:51.850 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:30:51 smithi092 ceph-mon[37563]: purged_snaps scrub ok 2024-03-11T21:30:51.850 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:30:51 smithi092 ceph-mon[37563]: osdmap e140: 8 total, 8 up, 8 in 2024-03-11T21:30:51.850 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:30:51 smithi092 ceph-mon[37563]: purged_snaps scrub starts 2024-03-11T21:30:51.851 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:30:51 smithi092 ceph-mon[37563]: purged_snaps scrub ok 2024-03-11T21:30:51.851 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:30:51 smithi092 ceph-mon[37563]: from='mgr.14607 v1:172.21.15.92:0/1702717597' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T21:30:51.864 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:30:51.864 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:30:51.865 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:30:51.965 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:30:51.996 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:30:51.997 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:30:51.997 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:30:52.098 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:30:52.129 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:30:52.129 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:30:52.130 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:30:52.231 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:30:52.255 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:30:52.256 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:30:52.256 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:30:52.357 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:30:52.385 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:30:52.385 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:30:52.386 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:30:52.487 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:30:52.545 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:30:52.546 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:30:52.547 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:30:52.558 INFO:journalctl@ceph.iscsi.iscsi.a.smithi097.stdout:Mar 11 21:30:52 smithi097 ceph-00db0016-dfed-11ee-95c8-87774f69a715-iscsi-iscsi-a[60943]: debug there is no tcmu-runner data available 2024-03-11T21:30:52.647 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:30:52.686 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:30:52.687 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:30:52.687 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:30:52.788 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:30:52.819 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:30:52.820 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:30:52.820 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:30:52.826 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:30:52 smithi097 ceph-mon[33317]: pgmap v191: 148 pgs: 148 active+clean; 2.2 GiB data, 7.7 GiB used, 708 GiB / 715 GiB avail; 778 B/s rd, 0 B/s wr, 775 op/s 2024-03-11T21:30:52.826 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:30:52 smithi097 ceph-mon[33317]: osdmap e141: 8 total, 8 up, 8 in 2024-03-11T21:30:52.860 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:30:52 smithi092 ceph-mon[37563]: pgmap v191: 148 pgs: 148 active+clean; 2.2 GiB data, 7.7 GiB used, 708 GiB / 715 GiB avail; 778 B/s rd, 0 B/s wr, 775 op/s 2024-03-11T21:30:52.860 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:30:52 smithi092 ceph-mon[37563]: osdmap e141: 8 total, 8 up, 8 in 2024-03-11T21:30:52.861 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:30:52 smithi092 ceph-mon[29572]: pgmap v191: 148 pgs: 148 active+clean; 2.2 GiB data, 7.7 GiB used, 708 GiB / 715 GiB avail; 778 B/s rd, 0 B/s wr, 775 op/s 2024-03-11T21:30:52.861 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:30:52 smithi092 ceph-mon[29572]: osdmap e141: 8 total, 8 up, 8 in 2024-03-11T21:30:52.921 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:30:52.951 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:30:52.951 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:30:52.951 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:30:53.052 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:30:53.090 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:30:53.091 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:30:53.091 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:30:53.192 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:30:53.221 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:30:53.222 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:30:53.222 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:30:53.324 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:30:53.367 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:30:53.368 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:30:53.369 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:30:53.470 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:30:53.518 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:30:53.519 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:30:53.519 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:30:53.620 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:30:53.648 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:30:53.649 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:30:53.649 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:30:53.750 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:30:53.778 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:30:53.778 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:30:53.779 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:30:53.785 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:30:53 smithi092 ceph-mon[29572]: from='client.24469 v1:172.21.15.97:0/404208310' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:30:53.785 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:30:53 smithi092 ceph-mon[29572]: purged_snaps scrub starts 2024-03-11T21:30:53.785 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:30:53 smithi092 ceph-mon[29572]: purged_snaps scrub ok 2024-03-11T21:30:53.785 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:30:53 smithi092 ceph-mon[37563]: from='client.24469 v1:172.21.15.97:0/404208310' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:30:53.785 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:30:53 smithi092 ceph-mon[37563]: purged_snaps scrub starts 2024-03-11T21:30:53.786 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:30:53 smithi092 ceph-mon[37563]: purged_snaps scrub ok 2024-03-11T21:30:53.826 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:30:53 smithi097 ceph-mon[33317]: from='client.24469 v1:172.21.15.97:0/404208310' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:30:53.826 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:30:53 smithi097 ceph-mon[33317]: purged_snaps scrub starts 2024-03-11T21:30:53.826 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:30:53 smithi097 ceph-mon[33317]: purged_snaps scrub ok 2024-03-11T21:30:53.880 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:30:53.929 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:30:53.930 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:30:53.930 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:30:54.031 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:30:54.067 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:30:54.068 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:30:54.068 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:30:54.169 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:30:54.207 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:30:54.207 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:30:54.207 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:30:54.308 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:30:54.346 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:30:54.346 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:30:54.346 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:30:54.447 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:30:54.474 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:30:54.475 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:30:54.475 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:30:54.576 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:30:54.605 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:30:54.605 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:30:54.606 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:30:54.707 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:30:54.745 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:30:54.745 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:30:54.746 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:30:54.825 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:30:54 smithi092 ceph-mon[29572]: pgmap v193: 148 pgs: 148 active+clean; 1.9 GiB data, 6.7 GiB used, 708 GiB / 715 GiB avail; 1.7 KiB/s rd, 0 B/s wr, 1.38k op/s 2024-03-11T21:30:54.825 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:30:54 smithi092 ceph-mon[29572]: osdmap e142: 8 total, 8 up, 8 in 2024-03-11T21:30:54.825 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:30:54 smithi092 ceph-mon[37563]: pgmap v193: 148 pgs: 148 active+clean; 1.9 GiB data, 6.7 GiB used, 708 GiB / 715 GiB avail; 1.7 KiB/s rd, 0 B/s wr, 1.38k op/s 2024-03-11T21:30:54.825 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:30:54 smithi092 ceph-mon[37563]: osdmap e142: 8 total, 8 up, 8 in 2024-03-11T21:30:54.826 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:30:54 smithi097 ceph-mon[33317]: pgmap v193: 148 pgs: 148 active+clean; 1.9 GiB data, 6.7 GiB used, 708 GiB / 715 GiB avail; 1.7 KiB/s rd, 0 B/s wr, 1.38k op/s 2024-03-11T21:30:54.826 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:30:54 smithi097 ceph-mon[33317]: osdmap e142: 8 total, 8 up, 8 in 2024-03-11T21:30:54.847 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:30:54.873 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:30:54.874 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:30:54.874 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:30:54.975 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:30:55.003 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:30:55.004 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:30:55.005 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:30:55.105 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:30:55.137 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:30:55.137 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:30:55.137 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:30:55.238 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:30:55.275 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:30:55.275 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:30:55.276 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:30:55.377 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:30:55.418 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:30:55.418 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:30:55.418 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:30:55.519 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:30:55.560 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:30:55.560 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:30:55.561 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:30:55.603 INFO:teuthology.orchestra.run.smithi092.stderr:noscrub is unset 2024-03-11T21:30:55.662 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:30:55.705 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:30:55.706 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:30:55.707 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:30:55.808 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:30:55.826 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:30:55 smithi097 ceph-mon[33317]: from='client.? v1:172.21.15.92:0/2600415585' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-11T21:30:55.826 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:30:55 smithi097 ceph-mon[33317]: from='client.34913 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-11T21:30:55.852 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:30:55.852 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:30:55.852 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:30:55.860 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:30:55 smithi092 ceph-mon[29572]: from='client.? v1:172.21.15.92:0/2600415585' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-11T21:30:55.861 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:30:55 smithi092 ceph-mon[29572]: from='client.34913 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-11T21:30:55.861 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:30:55 smithi092 ceph-mon[37563]: from='client.? v1:172.21.15.92:0/2600415585' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-11T21:30:55.861 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:30:55 smithi092 ceph-mon[37563]: from='client.34913 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-11T21:30:55.954 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:30:55.983 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:30:55.984 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:30:55.984 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:30:56.085 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:30:56.116 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:30:56.117 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:30:56.117 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:30:56.218 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:30:56.246 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:30:56.247 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:30:56.250 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:30:56.351 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:30:56.391 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:30:56.392 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:30:56.393 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:30:56.493 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:30:56.520 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:30:56.520 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:30:56.521 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:30:56.622 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:30:56.659 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:30:56.659 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:30:56.659 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:30:56.760 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:30:56.790 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:30:56.791 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:30:56.791 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:30:56.826 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:30:56 smithi092 ceph-mon[29572]: pgmap v195: 148 pgs: 148 active+clean; 1.9 GiB data, 6.7 GiB used, 708 GiB / 715 GiB avail; 852 B/s rd, 0 B/s wr, 847 op/s 2024-03-11T21:30:56.826 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:30:56 smithi092 ceph-mon[29572]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-11T21:30:56.826 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:30:56 smithi092 ceph-mon[29572]: from='client.34913 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-03-11T21:30:56.826 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:30:56 smithi092 ceph-mon[29572]: osdmap e143: 8 total, 8 up, 8 in 2024-03-11T21:30:56.826 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:30:56 smithi092 ceph-mon[29572]: 6.5 scrub starts 2024-03-11T21:30:56.826 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:30:56 smithi092 ceph-mon[29572]: 6.5 scrub ok 2024-03-11T21:30:56.827 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:30:56 smithi092 ceph-mon[29572]: from='mgr.14607 ' entity='mgr.y' 2024-03-11T21:30:56.827 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:30:56 smithi092 ceph-mon[29572]: from='mgr.14607 ' entity='mgr.y' 2024-03-11T21:30:56.827 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:30:56 smithi092 ceph-mon[37563]: pgmap v195: 148 pgs: 148 active+clean; 1.9 GiB data, 6.7 GiB used, 708 GiB / 715 GiB avail; 852 B/s rd, 0 B/s wr, 847 op/s 2024-03-11T21:30:56.827 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:30:56 smithi092 ceph-mon[37563]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-11T21:30:56.827 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:30:56 smithi092 ceph-mon[37563]: from='client.34913 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-03-11T21:30:56.827 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:30:56 smithi092 ceph-mon[37563]: osdmap e143: 8 total, 8 up, 8 in 2024-03-11T21:30:56.827 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:30:56 smithi092 ceph-mon[37563]: 6.5 scrub starts 2024-03-11T21:30:56.827 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:30:56 smithi092 ceph-mon[37563]: 6.5 scrub ok 2024-03-11T21:30:56.827 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:30:56 smithi092 ceph-mon[37563]: from='mgr.14607 ' entity='mgr.y' 2024-03-11T21:30:56.827 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:30:56 smithi092 ceph-mon[37563]: from='mgr.14607 ' entity='mgr.y' 2024-03-11T21:30:56.828 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:30:56 smithi097 ceph-mon[33317]: pgmap v195: 148 pgs: 148 active+clean; 1.9 GiB data, 6.7 GiB used, 708 GiB / 715 GiB avail; 852 B/s rd, 0 B/s wr, 847 op/s 2024-03-11T21:30:56.828 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:30:56 smithi097 ceph-mon[33317]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-11T21:30:56.828 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:30:56 smithi097 ceph-mon[33317]: from='client.34913 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-03-11T21:30:56.828 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:30:56 smithi097 ceph-mon[33317]: osdmap e143: 8 total, 8 up, 8 in 2024-03-11T21:30:56.828 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:30:56 smithi097 ceph-mon[33317]: 6.5 scrub starts 2024-03-11T21:30:56.828 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:30:56 smithi097 ceph-mon[33317]: 6.5 scrub ok 2024-03-11T21:30:56.828 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:30:56 smithi097 ceph-mon[33317]: from='mgr.14607 ' entity='mgr.y' 2024-03-11T21:30:56.828 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:30:56 smithi097 ceph-mon[33317]: from='mgr.14607 ' entity='mgr.y' 2024-03-11T21:30:56.892 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:30:56.922 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:30:56.922 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:30:56.923 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:30:57.023 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:30:57.058 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:30:57.058 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:30:57.059 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:30:57.159 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:30:57.191 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:30:57.192 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:30:57.192 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:30:57.293 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:30:57.322 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:30:57.322 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:30:57.323 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:30:57.423 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:30:57.451 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:30:57.452 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:30:57.453 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:30:57.553 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:30:57.605 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:30:57.606 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:30:57.606 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:30:57.672 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:30:57 smithi092 ceph-mon[29572]: 4.e scrub starts 2024-03-11T21:30:57.672 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:30:57 smithi092 ceph-mon[29572]: 4.e scrub ok 2024-03-11T21:30:57.673 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:30:57 smithi092 ceph-mon[37563]: 4.e scrub starts 2024-03-11T21:30:57.673 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:30:57 smithi092 ceph-mon[37563]: 4.e scrub ok 2024-03-11T21:30:57.707 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:30:57.755 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:30:57.756 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:30:57.756 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:30:57.857 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:30:57.897 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:30:57.897 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:30:57.897 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:30:57.998 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:30:58.037 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:30:58.037 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:30:58.037 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:30:58.076 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:30:57 smithi097 ceph-mon[33317]: 4.e scrub starts 2024-03-11T21:30:58.076 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:30:57 smithi097 ceph-mon[33317]: 4.e scrub ok 2024-03-11T21:30:58.138 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:30:58.169 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:30:58.169 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:30:58.170 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:30:58.253 INFO:journalctl@ceph.mgr.y.smithi092.stdout:Mar 11 21:30:57 smithi092 ceph-00db0016-dfed-11ee-95c8-87774f69a715-mgr-y[29801]: ::ffff:172.21.15.97 - - [11/Mar/2024:21:30:57] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-11T21:30:58.271 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:30:58.299 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:30:58.300 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:30:58.300 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:30:58.401 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:30:58.437 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:30:58.437 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:30:58.437 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:30:58.538 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:30:58.567 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:30:58.568 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:30:58.568 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:30:58.616 INFO:teuthology.orchestra.run.smithi092.stderr:reweighted osd.6 to 1 (10000) 2024-03-11T21:30:58.669 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:30:58.732 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:30:58.733 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:30:58.733 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:30:58.834 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:30:58.861 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:30:58.861 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:30:58.862 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:30:58.862 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:30:58 smithi092 ceph-mon[37563]: pgmap v197: 148 pgs: 148 active+clean; 1.8 GiB data, 6.4 GiB used, 709 GiB / 715 GiB avail; 852 B/s rd, 0 B/s wr, 1.16k op/s 2024-03-11T21:30:58.862 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:30:58 smithi092 ceph-mon[37563]: 7.a scrub starts 2024-03-11T21:30:58.862 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:30:58 smithi092 ceph-mon[37563]: 5.b scrub starts 2024-03-11T21:30:58.862 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:30:58 smithi092 ceph-mon[37563]: 5.b scrub ok 2024-03-11T21:30:58.862 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:30:58 smithi092 ceph-mon[37563]: 4.1e scrub starts 2024-03-11T21:30:58.863 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:30:58 smithi092 ceph-mon[37563]: 4.1e scrub ok 2024-03-11T21:30:58.863 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:30:58 smithi092 ceph-mon[37563]: 4.c scrub starts 2024-03-11T21:30:58.863 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:30:58 smithi092 ceph-mon[37563]: 4.c scrub ok 2024-03-11T21:30:58.863 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:30:58 smithi092 ceph-mon[37563]: from='client.? v1:172.21.15.92:0/2887827524' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]: dispatch 2024-03-11T21:30:58.863 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:30:58 smithi092 ceph-mon[37563]: 7.a scrub ok 2024-03-11T21:30:58.863 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:30:58 smithi092 ceph-mon[29572]: pgmap v197: 148 pgs: 148 active+clean; 1.8 GiB data, 6.4 GiB used, 709 GiB / 715 GiB avail; 852 B/s rd, 0 B/s wr, 1.16k op/s 2024-03-11T21:30:58.863 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:30:58 smithi092 ceph-mon[29572]: 7.a scrub starts 2024-03-11T21:30:58.863 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:30:58 smithi092 ceph-mon[29572]: 5.b scrub starts 2024-03-11T21:30:58.863 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:30:58 smithi092 ceph-mon[29572]: 5.b scrub ok 2024-03-11T21:30:58.863 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:30:58 smithi092 ceph-mon[29572]: 4.1e scrub starts 2024-03-11T21:30:58.863 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:30:58 smithi092 ceph-mon[29572]: 4.1e scrub ok 2024-03-11T21:30:58.863 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:30:58 smithi092 ceph-mon[29572]: 4.c scrub starts 2024-03-11T21:30:58.863 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:30:58 smithi092 ceph-mon[29572]: 4.c scrub ok 2024-03-11T21:30:58.864 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:30:58 smithi092 ceph-mon[29572]: from='client.? v1:172.21.15.92:0/2887827524' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]: dispatch 2024-03-11T21:30:58.864 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:30:58 smithi092 ceph-mon[29572]: 7.a scrub ok 2024-03-11T21:30:58.962 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:30:58.992 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:30:58.993 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:30:58.994 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:30:59.076 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:30:58 smithi097 ceph-mon[33317]: pgmap v197: 148 pgs: 148 active+clean; 1.8 GiB data, 6.4 GiB used, 709 GiB / 715 GiB avail; 852 B/s rd, 0 B/s wr, 1.16k op/s 2024-03-11T21:30:59.076 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:30:58 smithi097 ceph-mon[33317]: 7.a scrub starts 2024-03-11T21:30:59.077 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:30:58 smithi097 ceph-mon[33317]: 5.b scrub starts 2024-03-11T21:30:59.077 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:30:58 smithi097 ceph-mon[33317]: 5.b scrub ok 2024-03-11T21:30:59.077 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:30:58 smithi097 ceph-mon[33317]: 4.1e scrub starts 2024-03-11T21:30:59.077 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:30:58 smithi097 ceph-mon[33317]: 4.1e scrub ok 2024-03-11T21:30:59.077 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:30:58 smithi097 ceph-mon[33317]: 4.c scrub starts 2024-03-11T21:30:59.077 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:30:58 smithi097 ceph-mon[33317]: 4.c scrub ok 2024-03-11T21:30:59.077 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:30:58 smithi097 ceph-mon[33317]: from='client.? v1:172.21.15.92:0/2887827524' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]: dispatch 2024-03-11T21:30:59.077 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:30:58 smithi097 ceph-mon[33317]: 7.a scrub ok 2024-03-11T21:30:59.095 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:30:59.103 INFO:teuthology.orchestra.run.smithi092.stdout:ERROR: (22) Invalid argument 2024-03-11T21:30:59.103 INFO:teuthology.orchestra.run.smithi092.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-11T21:30:59.124 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:30:59.125 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:30:59.125 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:30:59.226 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:30:59.266 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:30:59.266 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:30:59.267 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:30:59.367 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:30:59.396 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:30:59.396 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:30:59.397 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:30:59.498 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:30:59.525 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:30:59.526 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:30:59.527 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:30:59.627 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:30:59.653 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:30:59.654 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:30:59.654 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:30:59.755 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:30:59.792 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:30:59.792 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:30:59.792 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:30:59.860 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:30:59 smithi092 ceph-mon[29572]: 7.8 scrub starts 2024-03-11T21:30:59.861 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:30:59 smithi092 ceph-mon[29572]: from='client.? v1:172.21.15.92:0/2887827524' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]': finished 2024-03-11T21:30:59.861 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:30:59 smithi092 ceph-mon[29572]: osdmap e144: 8 total, 8 up, 8 in 2024-03-11T21:30:59.861 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:30:59 smithi092 ceph-mon[29572]: 7.5 scrub starts 2024-03-11T21:30:59.861 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:30:59 smithi092 ceph-mon[29572]: 6.4 scrub starts 2024-03-11T21:30:59.861 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:30:59 smithi092 ceph-mon[29572]: 6.4 scrub ok 2024-03-11T21:30:59.861 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:30:59 smithi092 ceph-mon[29572]: 7.5 scrub ok 2024-03-11T21:30:59.861 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:30:59 smithi092 ceph-mon[37563]: 7.8 scrub starts 2024-03-11T21:30:59.861 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:30:59 smithi092 ceph-mon[37563]: from='client.? v1:172.21.15.92:0/2887827524' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]': finished 2024-03-11T21:30:59.861 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:30:59 smithi092 ceph-mon[37563]: osdmap e144: 8 total, 8 up, 8 in 2024-03-11T21:30:59.861 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:30:59 smithi092 ceph-mon[37563]: 7.5 scrub starts 2024-03-11T21:30:59.861 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:30:59 smithi092 ceph-mon[37563]: 6.4 scrub starts 2024-03-11T21:30:59.861 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:30:59 smithi092 ceph-mon[37563]: 6.4 scrub ok 2024-03-11T21:30:59.862 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:30:59 smithi092 ceph-mon[37563]: 7.5 scrub ok 2024-03-11T21:30:59.893 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:30:59.921 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:30:59.921 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:30:59.922 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:31:00.022 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:31:00.053 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:31:00.054 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:31:00.054 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:31:00.076 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:30:59 smithi097 ceph-mon[33317]: 7.8 scrub starts 2024-03-11T21:31:00.076 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:30:59 smithi097 ceph-mon[33317]: from='client.? v1:172.21.15.92:0/2887827524' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]': finished 2024-03-11T21:31:00.076 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:30:59 smithi097 ceph-mon[33317]: osdmap e144: 8 total, 8 up, 8 in 2024-03-11T21:31:00.076 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:30:59 smithi097 ceph-mon[33317]: 7.5 scrub starts 2024-03-11T21:31:00.077 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:30:59 smithi097 ceph-mon[33317]: 6.4 scrub starts 2024-03-11T21:31:00.077 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:30:59 smithi097 ceph-mon[33317]: 6.4 scrub ok 2024-03-11T21:31:00.077 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:30:59 smithi097 ceph-mon[33317]: 7.5 scrub ok 2024-03-11T21:31:00.155 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:31:00.214 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:31:00.214 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:31:00.214 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:31:00.301 DEBUG:teuthology.orchestra.run.smithi092:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid 00db0016-dfed-11ee-95c8-87774f69a715 -- ceph osd unset nodeep-scrub 2024-03-11T21:31:00.316 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:31:00.343 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:31:00.344 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:31:00.345 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:31:00.446 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:31:00.494 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:31:00.494 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:31:00.494 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:31:00.595 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:31:00.658 DEBUG:teuthology.orchestra.run.smithi092:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid 00db0016-dfed-11ee-95c8-87774f69a715 -- ceph osd reweight 7 1 2024-03-11T21:31:00.659 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:31:00.660 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:31:00.660 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:31:00.761 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:31:00.824 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:31:00.825 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:31:00.825 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:31:00.859 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:31:00 smithi092 ceph-mon[29572]: 7.8 scrub ok 2024-03-11T21:31:00.860 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:31:00 smithi092 ceph-mon[29572]: pgmap v199: 148 pgs: 1 active+clean+scrubbing, 147 active+clean; 1.6 GiB data, 5.8 GiB used, 709 GiB / 715 GiB avail; 852 B/s rd, 0 B/s wr, 841 op/s 2024-03-11T21:31:00.860 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:31:00 smithi092 ceph-mon[29572]: 5.1c scrub starts 2024-03-11T21:31:00.860 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:31:00 smithi092 ceph-mon[29572]: 5.1c scrub ok 2024-03-11T21:31:00.860 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:31:00 smithi092 ceph-mon[29572]: 2.16 scrub starts 2024-03-11T21:31:00.860 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:31:00 smithi092 ceph-mon[29572]: 2.16 scrub ok 2024-03-11T21:31:00.860 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:31:00 smithi092 ceph-mon[29572]: 7.7 scrub starts 2024-03-11T21:31:00.860 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:31:00 smithi092 ceph-mon[29572]: 7.7 scrub ok 2024-03-11T21:31:00.861 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:31:00 smithi092 ceph-mon[37563]: 7.8 scrub ok 2024-03-11T21:31:00.862 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:31:00 smithi092 ceph-mon[37563]: pgmap v199: 148 pgs: 1 active+clean+scrubbing, 147 active+clean; 1.6 GiB data, 5.8 GiB used, 709 GiB / 715 GiB avail; 852 B/s rd, 0 B/s wr, 841 op/s 2024-03-11T21:31:00.863 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:31:00 smithi092 ceph-mon[37563]: 5.1c scrub starts 2024-03-11T21:31:00.863 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:31:00 smithi092 ceph-mon[37563]: 5.1c scrub ok 2024-03-11T21:31:00.863 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:31:00 smithi092 ceph-mon[37563]: 2.16 scrub starts 2024-03-11T21:31:00.863 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:31:00 smithi092 ceph-mon[37563]: 2.16 scrub ok 2024-03-11T21:31:00.863 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:31:00 smithi092 ceph-mon[37563]: 7.7 scrub starts 2024-03-11T21:31:00.863 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:31:00 smithi092 ceph-mon[37563]: 7.7 scrub ok 2024-03-11T21:31:00.926 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:31:00.998 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:31:00.999 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:31:00.999 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:31:01.076 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:31:00 smithi097 ceph-mon[33317]: 7.8 scrub ok 2024-03-11T21:31:01.077 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:31:00 smithi097 ceph-mon[33317]: pgmap v199: 148 pgs: 1 active+clean+scrubbing, 147 active+clean; 1.6 GiB data, 5.8 GiB used, 709 GiB / 715 GiB avail; 852 B/s rd, 0 B/s wr, 841 op/s 2024-03-11T21:31:01.077 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:31:00 smithi097 ceph-mon[33317]: 5.1c scrub starts 2024-03-11T21:31:01.077 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:31:00 smithi097 ceph-mon[33317]: 5.1c scrub ok 2024-03-11T21:31:01.077 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:31:00 smithi097 ceph-mon[33317]: 2.16 scrub starts 2024-03-11T21:31:01.077 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:31:00 smithi097 ceph-mon[33317]: 2.16 scrub ok 2024-03-11T21:31:01.077 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:31:00 smithi097 ceph-mon[33317]: 7.7 scrub starts 2024-03-11T21:31:01.077 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:31:00 smithi097 ceph-mon[33317]: 7.7 scrub ok 2024-03-11T21:31:01.101 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:31:01.162 DEBUG:teuthology.orchestra.run.smithi092:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid 00db0016-dfed-11ee-95c8-87774f69a715 -- ceph daemon osd.1 dump_ops_in_flight 2024-03-11T21:31:01.186 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:31:01.186 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:31:01.187 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:31:01.287 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:31:01.363 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:31:01.363 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:31:01.363 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:31:01.464 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:31:01.491 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:31:01.492 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:31:01.492 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:31:01.593 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:31:01.619 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:31:01.620 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:31:01.620 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:31:01.691 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:31:01 smithi092 ceph-mon[29572]: 6.13 scrub starts 2024-03-11T21:31:01.691 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:31:01 smithi092 ceph-mon[29572]: 6.13 scrub ok 2024-03-11T21:31:01.691 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:31:01 smithi092 ceph-mon[29572]: 7.c scrub starts 2024-03-11T21:31:01.691 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:31:01 smithi092 ceph-mon[29572]: 5.1b scrub starts 2024-03-11T21:31:01.691 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:31:01 smithi092 ceph-mon[29572]: 5.1b scrub ok 2024-03-11T21:31:01.691 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:31:01 smithi092 ceph-mon[29572]: 5.4 scrub starts 2024-03-11T21:31:01.691 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:31:01 smithi092 ceph-mon[29572]: 5.4 scrub ok 2024-03-11T21:31:01.691 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:31:01 smithi092 ceph-mon[29572]: 7.c scrub ok 2024-03-11T21:31:01.691 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:31:01 smithi092 ceph-mon[37563]: 6.13 scrub starts 2024-03-11T21:31:01.691 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:31:01 smithi092 ceph-mon[37563]: 6.13 scrub ok 2024-03-11T21:31:01.692 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:31:01 smithi092 ceph-mon[37563]: 7.c scrub starts 2024-03-11T21:31:01.692 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:31:01 smithi092 ceph-mon[37563]: 5.1b scrub starts 2024-03-11T21:31:01.692 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:31:01 smithi092 ceph-mon[37563]: 5.1b scrub ok 2024-03-11T21:31:01.692 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:31:01 smithi092 ceph-mon[37563]: 5.4 scrub starts 2024-03-11T21:31:01.692 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:31:01 smithi092 ceph-mon[37563]: 5.4 scrub ok 2024-03-11T21:31:01.692 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:31:01 smithi092 ceph-mon[37563]: 7.c scrub ok 2024-03-11T21:31:01.705 INFO:teuthology.orchestra.run.smithi092.stderr:Inferring config /var/lib/ceph/00db0016-dfed-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T21:31:01.721 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:31:01.802 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:31:01.803 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:31:01.803 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:31:01.817 INFO:teuthology.orchestra.run.smithi092.stderr:Inferring config /var/lib/ceph/00db0016-dfed-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T21:31:01.827 INFO:teuthology.orchestra.run.smithi092.stderr:Inferring config /var/lib/ceph/00db0016-dfed-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T21:31:01.828 INFO:teuthology.orchestra.run.smithi092.stdout:osd.0: {} 2024-03-11T21:31:01.869 INFO:teuthology.orchestra.run.smithi092.stdout:osd.1: {} 2024-03-11T21:31:01.904 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:31:01.915 INFO:teuthology.orchestra.run.smithi092.stdout:osd.2: {} 2024-03-11T21:31:01.952 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:31:01.953 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:31:01.953 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:31:01.961 INFO:teuthology.orchestra.run.smithi092.stdout:osd.3: {} 2024-03-11T21:31:02.000 INFO:teuthology.orchestra.run.smithi092.stdout:osd.4: {} 2024-03-11T21:31:02.039 INFO:teuthology.orchestra.run.smithi092.stdout:osd.5: {} 2024-03-11T21:31:02.054 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:31:02.076 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:31:01 smithi097 ceph-mon[33317]: 6.13 scrub starts 2024-03-11T21:31:02.076 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:31:01 smithi097 ceph-mon[33317]: 6.13 scrub ok 2024-03-11T21:31:02.076 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:31:01 smithi097 ceph-mon[33317]: 7.c scrub starts 2024-03-11T21:31:02.076 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:31:01 smithi097 ceph-mon[33317]: 5.1b scrub starts 2024-03-11T21:31:02.076 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:31:01 smithi097 ceph-mon[33317]: 5.1b scrub ok 2024-03-11T21:31:02.076 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:31:01 smithi097 ceph-mon[33317]: 5.4 scrub starts 2024-03-11T21:31:02.077 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:31:01 smithi097 ceph-mon[33317]: 5.4 scrub ok 2024-03-11T21:31:02.077 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:31:01 smithi097 ceph-mon[33317]: 7.c scrub ok 2024-03-11T21:31:02.080 INFO:teuthology.orchestra.run.smithi092.stdout:osd.6: {} 2024-03-11T21:31:02.122 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:31:02.123 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:31:02.123 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:31:02.128 INFO:teuthology.orchestra.run.smithi092.stdout:osd.7: {} 2024-03-11T21:31:02.140 INFO:teuthology.orchestra.run.smithi092.stderr:osd.0: osd_enable_op_tracker = '' 2024-03-11T21:31:02.140 INFO:teuthology.orchestra.run.smithi092.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-11T21:31:02.140 INFO:teuthology.orchestra.run.smithi092.stderr:osd.2: osd_enable_op_tracker = '' 2024-03-11T21:31:02.140 INFO:teuthology.orchestra.run.smithi092.stderr:osd.3: osd_enable_op_tracker = '' 2024-03-11T21:31:02.140 INFO:teuthology.orchestra.run.smithi092.stderr:osd.4: osd_enable_op_tracker = '' 2024-03-11T21:31:02.141 INFO:teuthology.orchestra.run.smithi092.stderr:osd.5: osd_enable_op_tracker = '' 2024-03-11T21:31:02.141 INFO:teuthology.orchestra.run.smithi092.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-11T21:31:02.141 INFO:teuthology.orchestra.run.smithi092.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-11T21:31:02.224 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:31:02.254 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:31:02.255 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:31:02.255 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:31:02.356 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:31:02.394 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:31:02.395 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:31:02.395 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:31:02.496 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:31:02.543 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:31:02.544 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:31:02.544 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:31:02.576 INFO:journalctl@ceph.iscsi.iscsi.a.smithi097.stdout:Mar 11 21:31:02 smithi097 ceph-00db0016-dfed-11ee-95c8-87774f69a715-iscsi-iscsi-a[60943]: debug there is no tcmu-runner data available 2024-03-11T21:31:02.645 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:31:02.691 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:31:02.691 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:31:02.692 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:31:02.794 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:31:02.820 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:31:02.821 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:31:02.821 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:31:02.860 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:31:02 smithi092 ceph-mon[29572]: pgmap v200: 148 pgs: 1 active+clean+scrubbing, 147 active+clean; 1.6 GiB data, 5.8 GiB used, 709 GiB / 715 GiB avail; 685 B/s rd, 0 B/s wr, 676 op/s 2024-03-11T21:31:02.861 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:31:02 smithi092 ceph-mon[29572]: 6.6 scrub starts 2024-03-11T21:31:02.861 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:31:02 smithi092 ceph-mon[29572]: 6.6 scrub ok 2024-03-11T21:31:02.861 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:31:02 smithi092 ceph-mon[29572]: 4.1c scrub starts 2024-03-11T21:31:02.861 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:31:02 smithi092 ceph-mon[29572]: 4.1c scrub ok 2024-03-11T21:31:02.861 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:31:02 smithi092 ceph-mon[29572]: 2.7 scrub starts 2024-03-11T21:31:02.861 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:31:02 smithi092 ceph-mon[29572]: 2.7 scrub ok 2024-03-11T21:31:02.861 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:31:02 smithi092 ceph-mon[29572]: from='client.? v1:172.21.15.92:0/2506463253' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-11T21:31:02.861 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:31:02 smithi092 ceph-mon[29572]: 6.1 scrub starts 2024-03-11T21:31:02.861 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:31:02 smithi092 ceph-mon[29572]: 6.1 scrub ok 2024-03-11T21:31:02.861 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:31:02 smithi092 ceph-mon[37563]: pgmap v200: 148 pgs: 1 active+clean+scrubbing, 147 active+clean; 1.6 GiB data, 5.8 GiB used, 709 GiB / 715 GiB avail; 685 B/s rd, 0 B/s wr, 676 op/s 2024-03-11T21:31:02.861 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:31:02 smithi092 ceph-mon[37563]: 6.6 scrub starts 2024-03-11T21:31:02.861 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:31:02 smithi092 ceph-mon[37563]: 6.6 scrub ok 2024-03-11T21:31:02.862 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:31:02 smithi092 ceph-mon[37563]: 4.1c scrub starts 2024-03-11T21:31:02.862 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:31:02 smithi092 ceph-mon[37563]: 4.1c scrub ok 2024-03-11T21:31:02.862 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:31:02 smithi092 ceph-mon[37563]: 2.7 scrub starts 2024-03-11T21:31:02.862 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:31:02 smithi092 ceph-mon[37563]: 2.7 scrub ok 2024-03-11T21:31:02.862 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:31:02 smithi092 ceph-mon[37563]: from='client.? v1:172.21.15.92:0/2506463253' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-11T21:31:02.862 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:31:02 smithi092 ceph-mon[37563]: 6.1 scrub starts 2024-03-11T21:31:02.862 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:31:02 smithi092 ceph-mon[37563]: 6.1 scrub ok 2024-03-11T21:31:02.922 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:31:02.951 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:31:02.951 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:31:02.951 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:31:03.052 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:31:03.076 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:31:02 smithi097 ceph-mon[33317]: pgmap v200: 148 pgs: 1 active+clean+scrubbing, 147 active+clean; 1.6 GiB data, 5.8 GiB used, 709 GiB / 715 GiB avail; 685 B/s rd, 0 B/s wr, 676 op/s 2024-03-11T21:31:03.076 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:31:02 smithi097 ceph-mon[33317]: 6.6 scrub starts 2024-03-11T21:31:03.076 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:31:02 smithi097 ceph-mon[33317]: 6.6 scrub ok 2024-03-11T21:31:03.076 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:31:02 smithi097 ceph-mon[33317]: 4.1c scrub starts 2024-03-11T21:31:03.076 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:31:02 smithi097 ceph-mon[33317]: 4.1c scrub ok 2024-03-11T21:31:03.077 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:31:02 smithi097 ceph-mon[33317]: 2.7 scrub starts 2024-03-11T21:31:03.077 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:31:02 smithi097 ceph-mon[33317]: 2.7 scrub ok 2024-03-11T21:31:03.077 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:31:02 smithi097 ceph-mon[33317]: from='client.? v1:172.21.15.92:0/2506463253' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-11T21:31:03.077 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:31:02 smithi097 ceph-mon[33317]: 6.1 scrub starts 2024-03-11T21:31:03.077 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:31:02 smithi097 ceph-mon[33317]: 6.1 scrub ok 2024-03-11T21:31:03.089 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:31:03.090 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:31:03.090 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:31:03.191 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:31:03.232 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:31:03.232 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:31:03.233 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:31:03.335 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:31:03.375 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:31:03.376 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:31:03.377 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:31:03.478 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:31:03.510 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:31:03.511 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:31:03.511 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:31:03.614 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:31:03.669 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:31:03.670 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:31:03.670 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:31:03.771 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:31:03.810 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:31:03.810 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:31:03.811 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:31:03.912 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:31:03.940 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:31:03.941 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:31:03.941 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:31:04.043 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:31:04.058 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:31:03 smithi092 ceph-mon[29572]: from='client.24469 v1:172.21.15.97:0/404208310' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:31:04.059 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:31:03 smithi092 ceph-mon[29572]: 4.1 scrub starts 2024-03-11T21:31:04.059 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:31:03 smithi092 ceph-mon[29572]: 4.1 scrub ok 2024-03-11T21:31:04.059 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:31:03 smithi092 ceph-mon[29572]: 5.1f scrub starts 2024-03-11T21:31:04.059 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:31:03 smithi092 ceph-mon[29572]: 5.1f scrub ok 2024-03-11T21:31:04.059 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:31:03 smithi092 ceph-mon[29572]: 7.b scrub starts 2024-03-11T21:31:04.059 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:31:03 smithi092 ceph-mon[29572]: 7.b scrub ok 2024-03-11T21:31:04.059 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:31:03 smithi092 ceph-mon[37563]: from='client.24469 v1:172.21.15.97:0/404208310' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:31:04.059 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:31:03 smithi092 ceph-mon[37563]: 4.1 scrub starts 2024-03-11T21:31:04.059 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:31:03 smithi092 ceph-mon[37563]: 4.1 scrub ok 2024-03-11T21:31:04.060 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:31:03 smithi092 ceph-mon[37563]: 5.1f scrub starts 2024-03-11T21:31:04.060 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:31:03 smithi092 ceph-mon[37563]: 5.1f scrub ok 2024-03-11T21:31:04.060 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:31:03 smithi092 ceph-mon[37563]: 7.b scrub starts 2024-03-11T21:31:04.060 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:31:03 smithi092 ceph-mon[37563]: 7.b scrub ok 2024-03-11T21:31:04.076 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:31:03 smithi097 ceph-mon[33317]: from='client.24469 v1:172.21.15.97:0/404208310' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:31:04.076 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:31:03 smithi097 ceph-mon[33317]: 4.1 scrub starts 2024-03-11T21:31:04.076 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:31:03 smithi097 ceph-mon[33317]: 4.1 scrub ok 2024-03-11T21:31:04.076 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:31:03 smithi097 ceph-mon[33317]: 5.1f scrub starts 2024-03-11T21:31:04.077 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:31:03 smithi097 ceph-mon[33317]: 5.1f scrub ok 2024-03-11T21:31:04.077 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:31:03 smithi097 ceph-mon[33317]: 7.b scrub starts 2024-03-11T21:31:04.077 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:31:03 smithi097 ceph-mon[33317]: 7.b scrub ok 2024-03-11T21:31:04.091 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:31:04.092 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:31:04.092 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:31:04.193 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:31:04.219 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:31:04.219 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:31:04.220 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:31:04.320 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:31:04.367 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:31:04.368 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:31:04.368 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:31:04.469 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:31:04.497 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:31:04.498 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:31:04.498 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:31:04.599 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:31:04.632 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:31:04.632 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:31:04.632 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:31:04.733 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:31:04.763 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:31:04.764 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:31:04.764 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:31:04.821 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:31:04 smithi092 ceph-mon[29572]: pgmap v201: 148 pgs: 148 active+clean; 1.3 GiB data, 5.0 GiB used, 710 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.21k op/s 2024-03-11T21:31:04.821 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:31:04 smithi092 ceph-mon[29572]: 4.0 scrub starts 2024-03-11T21:31:04.821 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:31:04 smithi092 ceph-mon[29572]: 2.10 scrub starts 2024-03-11T21:31:04.821 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:31:04 smithi092 ceph-mon[29572]: 4.0 scrub ok 2024-03-11T21:31:04.821 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:31:04 smithi092 ceph-mon[29572]: 2.10 scrub ok 2024-03-11T21:31:04.822 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:31:04 smithi092 ceph-mon[29572]: 2.f scrub starts 2024-03-11T21:31:04.822 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:31:04 smithi092 ceph-mon[29572]: 2.f scrub ok 2024-03-11T21:31:04.822 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:31:04 smithi092 ceph-mon[37563]: pgmap v201: 148 pgs: 148 active+clean; 1.3 GiB data, 5.0 GiB used, 710 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.21k op/s 2024-03-11T21:31:04.822 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:31:04 smithi092 ceph-mon[37563]: 4.0 scrub starts 2024-03-11T21:31:04.822 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:31:04 smithi092 ceph-mon[37563]: 2.10 scrub starts 2024-03-11T21:31:04.822 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:31:04 smithi092 ceph-mon[37563]: 4.0 scrub ok 2024-03-11T21:31:04.822 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:31:04 smithi092 ceph-mon[37563]: 2.10 scrub ok 2024-03-11T21:31:04.822 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:31:04 smithi092 ceph-mon[37563]: 2.f scrub starts 2024-03-11T21:31:04.822 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:31:04 smithi092 ceph-mon[37563]: 2.f scrub ok 2024-03-11T21:31:04.865 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:31:04.892 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:31:04.892 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:31:04.893 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:31:04.994 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:31:05.057 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:31:05.058 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:31:05.059 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:31:05.076 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:31:04 smithi097 ceph-mon[33317]: pgmap v201: 148 pgs: 148 active+clean; 1.3 GiB data, 5.0 GiB used, 710 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.21k op/s 2024-03-11T21:31:05.076 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:31:04 smithi097 ceph-mon[33317]: 4.0 scrub starts 2024-03-11T21:31:05.076 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:31:04 smithi097 ceph-mon[33317]: 2.10 scrub starts 2024-03-11T21:31:05.076 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:31:04 smithi097 ceph-mon[33317]: 4.0 scrub ok 2024-03-11T21:31:05.077 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:31:04 smithi097 ceph-mon[33317]: 2.10 scrub ok 2024-03-11T21:31:05.077 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:31:04 smithi097 ceph-mon[33317]: 2.f scrub starts 2024-03-11T21:31:05.077 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:31:04 smithi097 ceph-mon[33317]: 2.f scrub ok 2024-03-11T21:31:05.159 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:31:05.206 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:31:05.207 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:31:05.207 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:31:05.308 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:31:05.338 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:31:05.339 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:31:05.340 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:31:05.441 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:31:05.481 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:31:05.481 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:31:05.481 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:31:05.582 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:31:05.624 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:31:05.624 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:31:05.625 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:31:05.725 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:31:05.754 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:31:05.754 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:31:05.755 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:31:05.856 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:31:05.889 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:31:05.890 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:31:05.890 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:31:05.991 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:31:06.032 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:31:06.033 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:31:06.033 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:31:06.076 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:31:05 smithi097 ceph-mon[33317]: 6.1f scrub starts 2024-03-11T21:31:06.076 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:31:05 smithi097 ceph-mon[33317]: 6.1f scrub ok 2024-03-11T21:31:06.076 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:31:05 smithi097 ceph-mon[33317]: 4.6 scrub starts 2024-03-11T21:31:06.077 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:31:05 smithi097 ceph-mon[33317]: 4.6 scrub ok 2024-03-11T21:31:06.110 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:31:05 smithi092 ceph-mon[29572]: 6.1f scrub starts 2024-03-11T21:31:06.111 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:31:05 smithi092 ceph-mon[29572]: 6.1f scrub ok 2024-03-11T21:31:06.111 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:31:05 smithi092 ceph-mon[29572]: 4.6 scrub starts 2024-03-11T21:31:06.111 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:31:05 smithi092 ceph-mon[29572]: 4.6 scrub ok 2024-03-11T21:31:06.111 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:31:05 smithi092 ceph-mon[37563]: 6.1f scrub starts 2024-03-11T21:31:06.111 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:31:05 smithi092 ceph-mon[37563]: 6.1f scrub ok 2024-03-11T21:31:06.111 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:31:05 smithi092 ceph-mon[37563]: 4.6 scrub starts 2024-03-11T21:31:06.111 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:31:05 smithi092 ceph-mon[37563]: 4.6 scrub ok 2024-03-11T21:31:06.135 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:31:06.171 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:31:06.172 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:31:06.173 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:31:06.273 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:31:06.309 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:31:06.310 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:31:06.310 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:31:06.411 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:31:06.458 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:31:06.459 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:31:06.459 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:31:06.560 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:31:06.597 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:31:06.598 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:31:06.598 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:31:06.699 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:31:06.731 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:31:06.732 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:31:06.732 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:31:06.833 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:31:06.869 DEBUG:teuthology.orchestra.run.smithi092:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid 00db0016-dfed-11ee-95c8-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-03-11T21:31:06.872 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:31:06.873 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:31:06.873 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:31:06.914 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:31:06 smithi092 ceph-mon[29572]: pgmap v202: 148 pgs: 148 active+clean; 1.3 GiB data, 5.0 GiB used, 710 GiB / 715 GiB avail; 1.1 KiB/s rd, 0 B/s wr, 1.02k op/s 2024-03-11T21:31:06.914 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:31:06 smithi092 ceph-mon[29572]: 2.1b scrub starts 2024-03-11T21:31:06.914 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:31:06 smithi092 ceph-mon[29572]: 2.1b scrub ok 2024-03-11T21:31:06.914 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:31:06 smithi092 ceph-mon[29572]: 5.e scrub starts 2024-03-11T21:31:06.914 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:31:06 smithi092 ceph-mon[29572]: 5.e scrub ok 2024-03-11T21:31:06.914 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:31:06 smithi092 ceph-mon[29572]: from='mgr.14607 v1:172.21.15.92:0/1702717597' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T21:31:06.914 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:31:06 smithi092 ceph-mon[29572]: 2.0 scrub starts 2024-03-11T21:31:06.914 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:31:06 smithi092 ceph-mon[29572]: 2.0 scrub ok 2024-03-11T21:31:06.915 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:31:06 smithi092 ceph-mon[37563]: pgmap v202: 148 pgs: 148 active+clean; 1.3 GiB data, 5.0 GiB used, 710 GiB / 715 GiB avail; 1.1 KiB/s rd, 0 B/s wr, 1.02k op/s 2024-03-11T21:31:06.915 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:31:06 smithi092 ceph-mon[37563]: 2.1b scrub starts 2024-03-11T21:31:06.915 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:31:06 smithi092 ceph-mon[37563]: 2.1b scrub ok 2024-03-11T21:31:06.915 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:31:06 smithi092 ceph-mon[37563]: 5.e scrub starts 2024-03-11T21:31:06.915 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:31:06 smithi092 ceph-mon[37563]: 5.e scrub ok 2024-03-11T21:31:06.915 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:31:06 smithi092 ceph-mon[37563]: from='mgr.14607 v1:172.21.15.92:0/1702717597' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T21:31:06.915 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:31:06 smithi092 ceph-mon[37563]: 2.0 scrub starts 2024-03-11T21:31:06.915 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:31:06 smithi092 ceph-mon[37563]: 2.0 scrub ok 2024-03-11T21:31:06.974 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:31:07.045 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:31:07.045 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:31:07.046 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:31:07.076 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:31:06 smithi097 ceph-mon[33317]: pgmap v202: 148 pgs: 148 active+clean; 1.3 GiB data, 5.0 GiB used, 710 GiB / 715 GiB avail; 1.1 KiB/s rd, 0 B/s wr, 1.02k op/s 2024-03-11T21:31:07.076 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:31:06 smithi097 ceph-mon[33317]: 2.1b scrub starts 2024-03-11T21:31:07.076 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:31:06 smithi097 ceph-mon[33317]: 2.1b scrub ok 2024-03-11T21:31:07.076 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:31:06 smithi097 ceph-mon[33317]: 5.e scrub starts 2024-03-11T21:31:07.077 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:31:06 smithi097 ceph-mon[33317]: 5.e scrub ok 2024-03-11T21:31:07.077 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:31:06 smithi097 ceph-mon[33317]: from='mgr.14607 v1:172.21.15.92:0/1702717597' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T21:31:07.077 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:31:06 smithi097 ceph-mon[33317]: 2.0 scrub starts 2024-03-11T21:31:07.077 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:31:06 smithi097 ceph-mon[33317]: 2.0 scrub ok 2024-03-11T21:31:07.146 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:31:07.174 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:31:07.175 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:31:07.175 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:31:07.276 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:31:07.306 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:31:07.307 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:31:07.307 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:31:07.408 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:31:07.463 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:31:07.464 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:31:07.464 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:31:07.565 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:31:07.593 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:31:07.594 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:31:07.594 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:31:07.695 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:31:07.725 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:31:07 smithi092 ceph-mon[29572]: 5.6 scrub starts 2024-03-11T21:31:07.725 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:31:07 smithi092 ceph-mon[29572]: 5.6 scrub ok 2024-03-11T21:31:07.725 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:31:07 smithi092 ceph-mon[29572]: 6.1b scrub starts 2024-03-11T21:31:07.725 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:31:07 smithi092 ceph-mon[29572]: 6.1b scrub ok 2024-03-11T21:31:07.725 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:31:07 smithi092 ceph-mon[29572]: 4.1f scrub starts 2024-03-11T21:31:07.725 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:31:07 smithi092 ceph-mon[29572]: 4.1f scrub ok 2024-03-11T21:31:07.726 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:31:07 smithi092 ceph-mon[29572]: 7.1 scrub starts 2024-03-11T21:31:07.726 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:31:07 smithi092 ceph-mon[29572]: 7.1 scrub ok 2024-03-11T21:31:07.726 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:31:07 smithi092 ceph-mon[29572]: 2.a scrub starts 2024-03-11T21:31:07.726 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:31:07 smithi092 ceph-mon[29572]: 2.a scrub ok 2024-03-11T21:31:07.726 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:31:07 smithi092 ceph-mon[29572]: 6.1c scrub starts 2024-03-11T21:31:07.726 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:31:07 smithi092 ceph-mon[29572]: 6.1c scrub ok 2024-03-11T21:31:07.726 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:31:07 smithi092 ceph-mon[37563]: 5.6 scrub starts 2024-03-11T21:31:07.726 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:31:07 smithi092 ceph-mon[37563]: 5.6 scrub ok 2024-03-11T21:31:07.726 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:31:07 smithi092 ceph-mon[37563]: 6.1b scrub starts 2024-03-11T21:31:07.726 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:31:07 smithi092 ceph-mon[37563]: 6.1b scrub ok 2024-03-11T21:31:07.726 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:31:07 smithi092 ceph-mon[37563]: 4.1f scrub starts 2024-03-11T21:31:07.726 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:31:07 smithi092 ceph-mon[37563]: 4.1f scrub ok 2024-03-11T21:31:07.727 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:31:07 smithi092 ceph-mon[37563]: 7.1 scrub starts 2024-03-11T21:31:07.727 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:31:07 smithi092 ceph-mon[37563]: 7.1 scrub ok 2024-03-11T21:31:07.727 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:31:07 smithi092 ceph-mon[37563]: 2.a scrub starts 2024-03-11T21:31:07.727 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:31:07 smithi092 ceph-mon[37563]: 2.a scrub ok 2024-03-11T21:31:07.727 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:31:07 smithi092 ceph-mon[37563]: 6.1c scrub starts 2024-03-11T21:31:07.727 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:31:07 smithi092 ceph-mon[37563]: 6.1c scrub ok 2024-03-11T21:31:07.749 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:31:07.750 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:31:07.750 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:31:07.851 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:31:07.888 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:31:07.888 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:31:07.888 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:31:07.976 INFO:journalctl@ceph.mgr.y.smithi092.stdout:Mar 11 21:31:07 smithi092 ceph-00db0016-dfed-11ee-95c8-87774f69a715-mgr-y[29801]: ::ffff:172.21.15.97 - - [11/Mar/2024:21:31:07] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-11T21:31:07.990 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:31:08.030 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:31:08.030 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:31:08.030 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:31:08.076 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:31:07 smithi097 ceph-mon[33317]: 5.6 scrub starts 2024-03-11T21:31:08.076 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:31:07 smithi097 ceph-mon[33317]: 5.6 scrub ok 2024-03-11T21:31:08.077 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:31:07 smithi097 ceph-mon[33317]: 6.1b scrub starts 2024-03-11T21:31:08.077 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:31:07 smithi097 ceph-mon[33317]: 6.1b scrub ok 2024-03-11T21:31:08.077 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:31:07 smithi097 ceph-mon[33317]: 4.1f scrub starts 2024-03-11T21:31:08.077 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:31:07 smithi097 ceph-mon[33317]: 4.1f scrub ok 2024-03-11T21:31:08.077 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:31:07 smithi097 ceph-mon[33317]: 7.1 scrub starts 2024-03-11T21:31:08.077 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:31:07 smithi097 ceph-mon[33317]: 7.1 scrub ok 2024-03-11T21:31:08.077 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:31:07 smithi097 ceph-mon[33317]: 2.a scrub starts 2024-03-11T21:31:08.077 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:31:07 smithi097 ceph-mon[33317]: 2.a scrub ok 2024-03-11T21:31:08.077 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:31:07 smithi097 ceph-mon[33317]: 6.1c scrub starts 2024-03-11T21:31:08.077 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:31:07 smithi097 ceph-mon[33317]: 6.1c scrub ok 2024-03-11T21:31:08.131 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:31:08.175 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:31:08.176 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:31:08.176 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:31:08.277 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:31:08.295 INFO:teuthology.orchestra.run.smithi092.stdout:{ 2024-03-11T21:31:08.295 INFO:teuthology.orchestra.run.smithi092.stdout: "ops": [ 2024-03-11T21:31:08.296 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:31:08.296 INFO:teuthology.orchestra.run.smithi092.stdout: "description": "osd_repop(client.34715.0:228288 7.0 e144/99 7:096aef4d:::benchmark_data_smithi092_76040_object106377:head v 144'13978, mlcod=144'13976)", 2024-03-11T21:31:08.296 INFO:teuthology.orchestra.run.smithi092.stdout: "initiated_at": "2024-03-11T21:31:08.297755+0000", 2024-03-11T21:31:08.296 INFO:teuthology.orchestra.run.smithi092.stdout: "age": 0.0032746289999999998, 2024-03-11T21:31:08.296 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.0033056380000000001, 2024-03-11T21:31:08.296 INFO:teuthology.orchestra.run.smithi092.stdout: "type_data": { 2024-03-11T21:31:08.296 INFO:teuthology.orchestra.run.smithi092.stdout: "flag_point": "started", 2024-03-11T21:31:08.296 INFO:teuthology.orchestra.run.smithi092.stdout: "events": [ 2024-03-11T21:31:08.296 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:31:08.296 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "initiated", 2024-03-11T21:31:08.296 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:31:08.297755+0000", 2024-03-11T21:31:08.296 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-03-11T21:31:08.296 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:31:08.297 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:31:08.297 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "header_read", 2024-03-11T21:31:08.297 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:31:08.297755+0000", 2024-03-11T21:31:08.297 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-03-11T21:31:08.297 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:31:08.297 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:31:08.297 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "throttled", 2024-03-11T21:31:08.297 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:31:08.297757+0000", 2024-03-11T21:31:08.297 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.4640000000000001e-06 2024-03-11T21:31:08.297 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:31:08.297 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:31:08.297 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "all_read", 2024-03-11T21:31:08.297 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:31:08.297765+0000", 2024-03-11T21:31:08.297 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 8.2840000000000006e-06 2024-03-11T21:31:08.298 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:31:08.298 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:31:08.298 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "dispatched", 2024-03-11T21:31:08.298 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:31:08.297780+0000", 2024-03-11T21:31:08.298 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.5081e-05 2024-03-11T21:31:08.298 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:31:08.298 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:31:08.298 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "queued_for_pg", 2024-03-11T21:31:08.298 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:31:08.297788+0000", 2024-03-11T21:31:08.298 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 7.5850000000000002e-06 2024-03-11T21:31:08.298 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:31:08.298 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:31:08.298 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-03-11T21:31:08.298 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:31:08.297856+0000", 2024-03-11T21:31:08.299 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 6.8539000000000002e-05 2024-03-11T21:31:08.299 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:31:08.299 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:31:08.299 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "started", 2024-03-11T21:31:08.299 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:31:08.297876+0000", 2024-03-11T21:31:08.299 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.9677000000000002e-05 2024-03-11T21:31:08.299 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-03-11T21:31:08.299 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-03-11T21:31:08.299 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-03-11T21:31:08.299 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-03-11T21:31:08.299 INFO:teuthology.orchestra.run.smithi092.stdout: ], 2024-03-11T21:31:08.299 INFO:teuthology.orchestra.run.smithi092.stdout: "num_ops": 1 2024-03-11T21:31:08.299 INFO:teuthology.orchestra.run.smithi092.stdout:} 2024-03-11T21:31:08.325 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:31:08.325 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:31:08.325 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:31:08.426 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:31:08.452 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:31:08.453 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:31:08.454 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:31:08.554 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:31:08.581 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:31:08.582 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:31:08.582 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:31:08.683 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:31:08.723 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:31:08.723 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:31:08.723 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:31:08.824 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:31:08.870 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:31:08.871 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:31:08.871 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:31:08.972 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:31:08.999 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:31:08.999 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:31:09.000 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:31:09.076 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:31:08 smithi097 ceph-mon[33317]: pgmap v203: 148 pgs: 148 active+clean; 1.2 GiB data, 4.6 GiB used, 711 GiB / 715 GiB avail; 1022 B/s rd, 0 B/s wr, 970 op/s 2024-03-11T21:31:09.076 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:31:08 smithi097 ceph-mon[33317]: 4.19 scrub starts 2024-03-11T21:31:09.076 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:31:08 smithi097 ceph-mon[33317]: 4.19 scrub ok 2024-03-11T21:31:09.076 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:31:08 smithi097 ceph-mon[33317]: 7.d scrub starts 2024-03-11T21:31:09.076 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:31:08 smithi097 ceph-mon[33317]: 7.d scrub ok 2024-03-11T21:31:09.101 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:31:09.110 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:31:08 smithi092 ceph-mon[29572]: pgmap v203: 148 pgs: 148 active+clean; 1.2 GiB data, 4.6 GiB used, 711 GiB / 715 GiB avail; 1022 B/s rd, 0 B/s wr, 970 op/s 2024-03-11T21:31:09.111 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:31:08 smithi092 ceph-mon[29572]: 4.19 scrub starts 2024-03-11T21:31:09.111 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:31:08 smithi092 ceph-mon[29572]: 4.19 scrub ok 2024-03-11T21:31:09.111 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:31:08 smithi092 ceph-mon[29572]: 7.d scrub starts 2024-03-11T21:31:09.111 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:31:08 smithi092 ceph-mon[29572]: 7.d scrub ok 2024-03-11T21:31:09.111 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:31:08 smithi092 ceph-mon[37563]: pgmap v203: 148 pgs: 148 active+clean; 1.2 GiB data, 4.6 GiB used, 711 GiB / 715 GiB avail; 1022 B/s rd, 0 B/s wr, 970 op/s 2024-03-11T21:31:09.111 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:31:08 smithi092 ceph-mon[37563]: 4.19 scrub starts 2024-03-11T21:31:09.111 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:31:08 smithi092 ceph-mon[37563]: 4.19 scrub ok 2024-03-11T21:31:09.111 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:31:08 smithi092 ceph-mon[37563]: 7.d scrub starts 2024-03-11T21:31:09.111 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:31:08 smithi092 ceph-mon[37563]: 7.d scrub ok 2024-03-11T21:31:09.131 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:31:09.132 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:31:09.132 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:31:09.233 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:31:09.261 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:31:09.262 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:31:09.262 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:31:09.343 DEBUG:teuthology.orchestra.run.smithi092:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid 00db0016-dfed-11ee-95c8-87774f69a715 -- ceph daemon osd.1 dump_blocked_ops 2024-03-11T21:31:09.363 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:31:09.393 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:31:09.393 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:31:09.394 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:31:09.494 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:31:09.521 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:31:09.522 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:31:09.522 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:31:09.623 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:31:09.651 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:31:09.651 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:31:09.652 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:31:09.752 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:31:09.781 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:31:09.782 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:31:09.782 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:31:09.883 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:31:09.910 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:31:09.910 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:31:09.911 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:31:09.972 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:31:09 smithi092 ceph-mon[29572]: 5.17 scrub starts 2024-03-11T21:31:09.973 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:31:09 smithi092 ceph-mon[29572]: 5.17 scrub ok 2024-03-11T21:31:09.973 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:31:09 smithi092 ceph-mon[29572]: 6.16 scrub starts 2024-03-11T21:31:09.973 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:31:09 smithi092 ceph-mon[29572]: 6.16 scrub ok 2024-03-11T21:31:09.973 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:31:09 smithi092 ceph-mon[29572]: 4.2 scrub starts 2024-03-11T21:31:09.973 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:31:09 smithi092 ceph-mon[29572]: 4.2 scrub ok 2024-03-11T21:31:09.973 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:31:09 smithi092 ceph-mon[29572]: from='client.? v1:172.21.15.92:0/1762611047' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]: dispatch 2024-03-11T21:31:09.973 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:31:09 smithi092 ceph-mon[29572]: from='client.24946 ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]: dispatch 2024-03-11T21:31:09.973 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:31:09 smithi092 ceph-mon[29572]: 2.c scrub starts 2024-03-11T21:31:09.973 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:31:09 smithi092 ceph-mon[29572]: 2.c scrub ok 2024-03-11T21:31:09.974 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:31:09 smithi092 ceph-mon[37563]: 5.17 scrub starts 2024-03-11T21:31:09.974 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:31:09 smithi092 ceph-mon[37563]: 5.17 scrub ok 2024-03-11T21:31:09.974 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:31:09 smithi092 ceph-mon[37563]: 6.16 scrub starts 2024-03-11T21:31:09.974 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:31:09 smithi092 ceph-mon[37563]: 6.16 scrub ok 2024-03-11T21:31:09.974 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:31:09 smithi092 ceph-mon[37563]: 4.2 scrub starts 2024-03-11T21:31:09.974 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:31:09 smithi092 ceph-mon[37563]: 4.2 scrub ok 2024-03-11T21:31:09.974 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:31:09 smithi092 ceph-mon[37563]: from='client.? v1:172.21.15.92:0/1762611047' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]: dispatch 2024-03-11T21:31:09.974 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:31:09 smithi092 ceph-mon[37563]: from='client.24946 ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]: dispatch 2024-03-11T21:31:09.974 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:31:09 smithi092 ceph-mon[37563]: 2.c scrub starts 2024-03-11T21:31:09.974 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:31:09 smithi092 ceph-mon[37563]: 2.c scrub ok 2024-03-11T21:31:10.012 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:31:10.052 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:31:10.053 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:31:10.054 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:31:10.076 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:31:09 smithi097 ceph-mon[33317]: 5.17 scrub starts 2024-03-11T21:31:10.076 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:31:09 smithi097 ceph-mon[33317]: 5.17 scrub ok 2024-03-11T21:31:10.076 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:31:09 smithi097 ceph-mon[33317]: 6.16 scrub starts 2024-03-11T21:31:10.077 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:31:09 smithi097 ceph-mon[33317]: 6.16 scrub ok 2024-03-11T21:31:10.077 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:31:09 smithi097 ceph-mon[33317]: 4.2 scrub starts 2024-03-11T21:31:10.077 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:31:09 smithi097 ceph-mon[33317]: 4.2 scrub ok 2024-03-11T21:31:10.077 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:31:09 smithi097 ceph-mon[33317]: from='client.? v1:172.21.15.92:0/1762611047' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]: dispatch 2024-03-11T21:31:10.077 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:31:09 smithi097 ceph-mon[33317]: from='client.24946 ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]: dispatch 2024-03-11T21:31:10.077 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:31:09 smithi097 ceph-mon[33317]: 2.c scrub starts 2024-03-11T21:31:10.077 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:31:09 smithi097 ceph-mon[33317]: 2.c scrub ok 2024-03-11T21:31:10.154 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:31:10.201 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:31:10.201 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:31:10.201 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:31:10.302 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:31:10.334 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:31:10.335 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:31:10.335 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:31:10.436 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:31:10.463 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:31:10.463 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:31:10.464 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:31:10.493 INFO:teuthology.orchestra.run.smithi092.stderr:reweighted osd.7 to 1 (10000) 2024-03-11T21:31:10.565 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:31:10.595 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:31:10.596 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:31:10.596 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:31:10.697 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:31:10.736 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:31:10.737 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:31:10.737 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:31:10.838 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:31:10.884 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:31:10.886 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:31:10.886 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:31:10.987 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:31:11.029 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:31:11.030 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:31:11.030 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:31:11.076 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:31:10 smithi097 ceph-mon[33317]: pgmap v204: 148 pgs: 148 active+clean; 1.0 GiB data, 4.0 GiB used, 711 GiB / 715 GiB avail; 979 B/s rd, 0 B/s wr, 939 op/s 2024-03-11T21:31:11.076 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:31:10 smithi097 ceph-mon[33317]: 2.14 scrub starts 2024-03-11T21:31:11.076 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:31:10 smithi097 ceph-mon[33317]: 2.14 scrub ok 2024-03-11T21:31:11.077 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:31:10 smithi097 ceph-mon[33317]: 6.e scrub starts 2024-03-11T21:31:11.077 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:31:10 smithi097 ceph-mon[33317]: 6.e scrub ok 2024-03-11T21:31:11.077 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:31:10 smithi097 ceph-mon[33317]: from='client.24946 ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]': finished 2024-03-11T21:31:11.077 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:31:10 smithi097 ceph-mon[33317]: osdmap e145: 8 total, 8 up, 8 in 2024-03-11T21:31:11.111 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:31:10 smithi092 ceph-mon[29572]: pgmap v204: 148 pgs: 148 active+clean; 1.0 GiB data, 4.0 GiB used, 711 GiB / 715 GiB avail; 979 B/s rd, 0 B/s wr, 939 op/s 2024-03-11T21:31:11.112 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:31:10 smithi092 ceph-mon[29572]: 2.14 scrub starts 2024-03-11T21:31:11.112 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:31:10 smithi092 ceph-mon[29572]: 2.14 scrub ok 2024-03-11T21:31:11.112 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:31:10 smithi092 ceph-mon[29572]: 6.e scrub starts 2024-03-11T21:31:11.112 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:31:10 smithi092 ceph-mon[29572]: 6.e scrub ok 2024-03-11T21:31:11.112 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:31:10 smithi092 ceph-mon[29572]: from='client.24946 ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]': finished 2024-03-11T21:31:11.112 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:31:10 smithi092 ceph-mon[29572]: osdmap e145: 8 total, 8 up, 8 in 2024-03-11T21:31:11.113 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:31:10 smithi092 ceph-mon[37563]: pgmap v204: 148 pgs: 148 active+clean; 1.0 GiB data, 4.0 GiB used, 711 GiB / 715 GiB avail; 979 B/s rd, 0 B/s wr, 939 op/s 2024-03-11T21:31:11.113 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:31:10 smithi092 ceph-mon[37563]: 2.14 scrub starts 2024-03-11T21:31:11.113 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:31:10 smithi092 ceph-mon[37563]: 2.14 scrub ok 2024-03-11T21:31:11.113 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:31:10 smithi092 ceph-mon[37563]: 6.e scrub starts 2024-03-11T21:31:11.113 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:31:10 smithi092 ceph-mon[37563]: 6.e scrub ok 2024-03-11T21:31:11.113 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:31:10 smithi092 ceph-mon[37563]: from='client.24946 ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]': finished 2024-03-11T21:31:11.113 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:31:10 smithi092 ceph-mon[37563]: osdmap e145: 8 total, 8 up, 8 in 2024-03-11T21:31:11.132 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:31:11.158 INFO:tasks.cephadm.ceph_manager.ceph:waiting for recovery to complete 2024-03-11T21:31:11.158 DEBUG:teuthology.orchestra.run.smithi092:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid 00db0016-dfed-11ee-95c8-87774f69a715 -- ceph pg dump --format=json 2024-03-11T21:31:11.159 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:31:11.159 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:31:11.159 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:31:11.260 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:31:11.288 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:31:11.288 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:31:11.289 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:31:11.390 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:31:11.456 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:31:11.457 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:31:11.457 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:31:11.558 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:31:11.585 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:31:11.585 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:31:11.586 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:31:11.609 INFO:teuthology.orchestra.run.smithi092.stderr:Inferring config /var/lib/ceph/00db0016-dfed-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T21:31:11.687 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:31:11.745 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:31:11.746 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:31:11.747 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:31:11.848 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:31:11.874 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:31:11.875 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:31:11.876 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:31:11.977 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:31:12.027 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:31:12.027 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:31:12.028 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:31:12.068 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:31:11 smithi092 ceph-mon[29572]: 6.9 scrub starts 2024-03-11T21:31:12.068 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:31:11 smithi092 ceph-mon[29572]: 6.9 scrub ok 2024-03-11T21:31:12.068 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:31:11 smithi092 ceph-mon[29572]: 5.18 scrub starts 2024-03-11T21:31:12.068 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:31:11 smithi092 ceph-mon[29572]: 5.18 scrub ok 2024-03-11T21:31:12.068 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:31:11 smithi092 ceph-mon[29572]: 5.a scrub starts 2024-03-11T21:31:12.068 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:31:11 smithi092 ceph-mon[29572]: 5.a scrub ok 2024-03-11T21:31:12.069 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:31:11 smithi092 ceph-mon[37563]: 6.9 scrub starts 2024-03-11T21:31:12.069 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:31:11 smithi092 ceph-mon[37563]: 6.9 scrub ok 2024-03-11T21:31:12.069 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:31:11 smithi092 ceph-mon[37563]: 5.18 scrub starts 2024-03-11T21:31:12.069 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:31:11 smithi092 ceph-mon[37563]: 5.18 scrub ok 2024-03-11T21:31:12.069 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:31:11 smithi092 ceph-mon[37563]: 5.a scrub starts 2024-03-11T21:31:12.069 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:31:11 smithi092 ceph-mon[37563]: 5.a scrub ok 2024-03-11T21:31:12.076 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:31:11 smithi097 ceph-mon[33317]: 6.9 scrub starts 2024-03-11T21:31:12.076 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:31:11 smithi097 ceph-mon[33317]: 6.9 scrub ok 2024-03-11T21:31:12.076 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:31:11 smithi097 ceph-mon[33317]: 5.18 scrub starts 2024-03-11T21:31:12.076 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:31:11 smithi097 ceph-mon[33317]: 5.18 scrub ok 2024-03-11T21:31:12.076 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:31:11 smithi097 ceph-mon[33317]: 5.a scrub starts 2024-03-11T21:31:12.076 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:31:11 smithi097 ceph-mon[33317]: 5.a scrub ok 2024-03-11T21:31:12.128 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:31:12.158 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:31:12.158 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:31:12.159 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:31:12.260 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:31:12.288 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:31:12.289 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:31:12.289 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:31:12.390 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:31:12.420 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:31:12.421 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:31:12.421 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:31:12.522 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:31:12.549 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:31:12.550 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:31:12.550 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:31:12.576 INFO:journalctl@ceph.iscsi.iscsi.a.smithi097.stdout:Mar 11 21:31:12 smithi097 ceph-00db0016-dfed-11ee-95c8-87774f69a715-iscsi-iscsi-a[60943]: debug there is no tcmu-runner data available 2024-03-11T21:31:12.651 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:31:12.691 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:31:12.692 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:31:12.692 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:31:12.697 INFO:teuthology.orchestra.run.smithi092.stderr:Inferring config /var/lib/ceph/00db0016-dfed-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T21:31:12.736 INFO:teuthology.orchestra.run.smithi092.stderr:nodeep-scrub is unset 2024-03-11T21:31:12.793 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:31:12.835 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:31:12.836 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:31:12.836 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:31:12.937 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:31:12.966 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:31:12.966 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:31:12.967 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:31:13.067 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:31:13.076 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:31:12 smithi097 ceph-mon[33317]: pgmap v206: 148 pgs: 148 active+clean; 1.0 GiB data, 4.0 GiB used, 711 GiB / 715 GiB avail; 1022 B/s rd, 0 B/s wr, 981 op/s 2024-03-11T21:31:13.076 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:31:12 smithi097 ceph-mon[33317]: 5.2 scrub starts 2024-03-11T21:31:13.076 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:31:12 smithi097 ceph-mon[33317]: 5.2 scrub ok 2024-03-11T21:31:13.076 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:31:12 smithi097 ceph-mon[33317]: 7.6 scrub starts 2024-03-11T21:31:13.076 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:31:12 smithi097 ceph-mon[33317]: 7.0 scrub starts 2024-03-11T21:31:13.076 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:31:12 smithi097 ceph-mon[33317]: 7.0 scrub ok 2024-03-11T21:31:13.077 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:31:12 smithi097 ceph-mon[33317]: 7.6 scrub ok 2024-03-11T21:31:13.077 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:31:12 smithi097 ceph-mon[33317]: from='client.? v1:172.21.15.92:0/1187901982' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-11T21:31:13.077 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:31:12 smithi097 ceph-mon[33317]: from='client.34928 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-11T21:31:13.077 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:31:12 smithi097 ceph-mon[33317]: 5.8 scrub starts 2024-03-11T21:31:13.077 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:31:12 smithi097 ceph-mon[33317]: 5.8 scrub ok 2024-03-11T21:31:13.095 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:31:13.096 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:31:13.096 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:31:13.110 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:31:12 smithi092 ceph-mon[29572]: pgmap v206: 148 pgs: 148 active+clean; 1.0 GiB data, 4.0 GiB used, 711 GiB / 715 GiB avail; 1022 B/s rd, 0 B/s wr, 981 op/s 2024-03-11T21:31:13.110 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:31:12 smithi092 ceph-mon[29572]: 5.2 scrub starts 2024-03-11T21:31:13.110 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:31:12 smithi092 ceph-mon[29572]: 5.2 scrub ok 2024-03-11T21:31:13.110 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:31:12 smithi092 ceph-mon[29572]: 7.6 scrub starts 2024-03-11T21:31:13.111 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:31:12 smithi092 ceph-mon[29572]: 7.0 scrub starts 2024-03-11T21:31:13.111 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:31:12 smithi092 ceph-mon[29572]: 7.0 scrub ok 2024-03-11T21:31:13.111 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:31:12 smithi092 ceph-mon[29572]: 7.6 scrub ok 2024-03-11T21:31:13.111 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:31:12 smithi092 ceph-mon[29572]: from='client.? v1:172.21.15.92:0/1187901982' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-11T21:31:13.111 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:31:12 smithi092 ceph-mon[29572]: from='client.34928 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-11T21:31:13.111 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:31:12 smithi092 ceph-mon[29572]: 5.8 scrub starts 2024-03-11T21:31:13.111 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:31:12 smithi092 ceph-mon[29572]: 5.8 scrub ok 2024-03-11T21:31:13.111 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:31:12 smithi092 ceph-mon[37563]: pgmap v206: 148 pgs: 148 active+clean; 1.0 GiB data, 4.0 GiB used, 711 GiB / 715 GiB avail; 1022 B/s rd, 0 B/s wr, 981 op/s 2024-03-11T21:31:13.111 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:31:12 smithi092 ceph-mon[37563]: 5.2 scrub starts 2024-03-11T21:31:13.112 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:31:12 smithi092 ceph-mon[37563]: 5.2 scrub ok 2024-03-11T21:31:13.112 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:31:12 smithi092 ceph-mon[37563]: 7.6 scrub starts 2024-03-11T21:31:13.112 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:31:12 smithi092 ceph-mon[37563]: 7.0 scrub starts 2024-03-11T21:31:13.112 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:31:12 smithi092 ceph-mon[37563]: 7.0 scrub ok 2024-03-11T21:31:13.112 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:31:12 smithi092 ceph-mon[37563]: 7.6 scrub ok 2024-03-11T21:31:13.112 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:31:12 smithi092 ceph-mon[37563]: from='client.? v1:172.21.15.92:0/1187901982' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-11T21:31:13.112 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:31:12 smithi092 ceph-mon[37563]: from='client.34928 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-11T21:31:13.112 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:31:12 smithi092 ceph-mon[37563]: 5.8 scrub starts 2024-03-11T21:31:13.112 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:31:12 smithi092 ceph-mon[37563]: 5.8 scrub ok 2024-03-11T21:31:13.197 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:31:13.241 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:31:13.242 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:31:13.242 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:31:13.343 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:31:13.395 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:31:13.396 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:31:13.396 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:31:13.497 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:31:13.523 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:31:13.523 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:31:13.524 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:31:13.624 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:31:13.673 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:31:13.674 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:31:13.674 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:31:13.775 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:31:13.802 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:31:13.803 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:31:13.803 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:31:13.839 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:31:13 smithi092 ceph-mon[29572]: from='client.24469 v1:172.21.15.97:0/404208310' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:31:13.840 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:31:13 smithi092 ceph-mon[29572]: 5.11 scrub starts 2024-03-11T21:31:13.840 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:31:13 smithi092 ceph-mon[29572]: 5.11 scrub ok 2024-03-11T21:31:13.840 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:31:13 smithi092 ceph-mon[29572]: 5.7 scrub starts 2024-03-11T21:31:13.840 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:31:13 smithi092 ceph-mon[29572]: 5.7 scrub ok 2024-03-11T21:31:13.840 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:31:13 smithi092 ceph-mon[29572]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-03-11T21:31:13.840 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:31:13 smithi092 ceph-mon[29572]: Cluster is now healthy 2024-03-11T21:31:13.840 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:31:13 smithi092 ceph-mon[29572]: from='client.34928 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-03-11T21:31:13.840 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:31:13 smithi092 ceph-mon[29572]: osdmap e146: 8 total, 8 up, 8 in 2024-03-11T21:31:13.840 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:31:13 smithi092 ceph-mon[29572]: from='mgr.14607 ' entity='mgr.y' 2024-03-11T21:31:13.840 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:31:13 smithi092 ceph-mon[29572]: from='mgr.14607 ' entity='mgr.y' 2024-03-11T21:31:13.841 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:31:13 smithi092 ceph-mon[37563]: from='client.24469 v1:172.21.15.97:0/404208310' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:31:13.841 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:31:13 smithi092 ceph-mon[37563]: 5.11 scrub starts 2024-03-11T21:31:13.841 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:31:13 smithi092 ceph-mon[37563]: 5.11 scrub ok 2024-03-11T21:31:13.841 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:31:13 smithi092 ceph-mon[37563]: 5.7 scrub starts 2024-03-11T21:31:13.841 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:31:13 smithi092 ceph-mon[37563]: 5.7 scrub ok 2024-03-11T21:31:13.841 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:31:13 smithi092 ceph-mon[37563]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-03-11T21:31:13.841 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:31:13 smithi092 ceph-mon[37563]: Cluster is now healthy 2024-03-11T21:31:13.841 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:31:13 smithi092 ceph-mon[37563]: from='client.34928 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-03-11T21:31:13.841 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:31:13 smithi092 ceph-mon[37563]: osdmap e146: 8 total, 8 up, 8 in 2024-03-11T21:31:13.841 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:31:13 smithi092 ceph-mon[37563]: from='mgr.14607 ' entity='mgr.y' 2024-03-11T21:31:13.841 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:31:13 smithi092 ceph-mon[37563]: from='mgr.14607 ' entity='mgr.y' 2024-03-11T21:31:13.904 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:31:13.933 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:31:13.934 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:31:13.935 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:31:14.035 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:31:14.066 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:31:14.067 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:31:14.067 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:31:14.076 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:31:13 smithi097 ceph-mon[33317]: from='client.24469 v1:172.21.15.97:0/404208310' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:31:14.076 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:31:13 smithi097 ceph-mon[33317]: 5.11 scrub starts 2024-03-11T21:31:14.076 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:31:13 smithi097 ceph-mon[33317]: 5.11 scrub ok 2024-03-11T21:31:14.076 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:31:13 smithi097 ceph-mon[33317]: 5.7 scrub starts 2024-03-11T21:31:14.076 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:31:13 smithi097 ceph-mon[33317]: 5.7 scrub ok 2024-03-11T21:31:14.076 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:31:13 smithi097 ceph-mon[33317]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-03-11T21:31:14.076 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:31:13 smithi097 ceph-mon[33317]: Cluster is now healthy 2024-03-11T21:31:14.076 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:31:13 smithi097 ceph-mon[33317]: from='client.34928 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-03-11T21:31:14.076 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:31:13 smithi097 ceph-mon[33317]: osdmap e146: 8 total, 8 up, 8 in 2024-03-11T21:31:14.076 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:31:13 smithi097 ceph-mon[33317]: from='mgr.14607 ' entity='mgr.y' 2024-03-11T21:31:14.076 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:31:13 smithi097 ceph-mon[33317]: from='mgr.14607 ' entity='mgr.y' 2024-03-11T21:31:14.168 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:31:14.232 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:31:14.233 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:31:14.233 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:31:14.334 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:31:14.388 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:31:14.388 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:31:14.389 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:31:14.490 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:31:14.533 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:31:14.534 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:31:14.534 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:31:14.635 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:31:14.668 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:31:14.669 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:31:14.669 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:31:14.770 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:31:14.815 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:31:14.815 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:31:14.816 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:31:14.916 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:31:14.943 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:31:14.944 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:31:14.944 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:31:15.045 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:31:15.072 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:31:15.072 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:31:15.073 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:31:15.173 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:31:15.202 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:31:15.203 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:31:15.203 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:31:15.304 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:31:15.326 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:31:14 smithi097 ceph-mon[33317]: pgmap v208: 148 pgs: 148 active+clean; 717 MiB data, 3.1 GiB used, 712 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.28k op/s 2024-03-11T21:31:15.326 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:31:14 smithi097 ceph-mon[33317]: from='mgr.14607 v1:172.21.15.92:0/1702717597' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T21:31:15.327 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:31:14 smithi097 ceph-mon[33317]: from='mgr.14607 v1:172.21.15.92:0/1702717597' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T21:31:15.327 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:31:14 smithi097 ceph-mon[33317]: from='mgr.14607 ' entity='mgr.y' 2024-03-11T21:31:15.339 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:31:15.340 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:31:15.340 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:31:15.360 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:31:14 smithi092 ceph-mon[29572]: pgmap v208: 148 pgs: 148 active+clean; 717 MiB data, 3.1 GiB used, 712 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.28k op/s 2024-03-11T21:31:15.361 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:31:15 smithi092 ceph-mon[29572]: from='mgr.14607 v1:172.21.15.92:0/1702717597' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T21:31:15.361 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:31:15 smithi092 ceph-mon[29572]: from='mgr.14607 v1:172.21.15.92:0/1702717597' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T21:31:15.361 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:31:15 smithi092 ceph-mon[29572]: from='mgr.14607 ' entity='mgr.y' 2024-03-11T21:31:15.361 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:31:14 smithi092 ceph-mon[37563]: pgmap v208: 148 pgs: 148 active+clean; 717 MiB data, 3.1 GiB used, 712 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.28k op/s 2024-03-11T21:31:15.361 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:31:15 smithi092 ceph-mon[37563]: from='mgr.14607 v1:172.21.15.92:0/1702717597' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T21:31:15.361 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:31:15 smithi092 ceph-mon[37563]: from='mgr.14607 v1:172.21.15.92:0/1702717597' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T21:31:15.361 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:31:15 smithi092 ceph-mon[37563]: from='mgr.14607 ' entity='mgr.y' 2024-03-11T21:31:15.441 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:31:15.470 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:31:15.471 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:31:15.471 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:31:15.572 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:31:15.600 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:31:15.601 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:31:15.601 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:31:15.702 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:31:15.728 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:31:15.729 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:31:15.729 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:31:15.830 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:31:15.858 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:31:15.859 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:31:15.859 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:31:15.960 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:31:16.001 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:31:16.002 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:31:16.003 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:31:16.103 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:31:16.131 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:31:16.132 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:31:16.132 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:31:16.231 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:31:15 smithi092 ceph-mon[29572]: 6.11 scrub starts 2024-03-11T21:31:16.231 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:31:15 smithi092 ceph-mon[29572]: 6.11 scrub ok 2024-03-11T21:31:16.231 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:31:15 smithi092 ceph-mon[29572]: 5.16 scrub starts 2024-03-11T21:31:16.231 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:31:15 smithi092 ceph-mon[29572]: 5.16 scrub ok 2024-03-11T21:31:16.232 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:31:15 smithi092 ceph-mon[29572]: 6.14 scrub starts 2024-03-11T21:31:16.232 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:31:15 smithi092 ceph-mon[29572]: 6.14 scrub ok 2024-03-11T21:31:16.232 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:31:15 smithi092 ceph-mon[37563]: 6.11 scrub starts 2024-03-11T21:31:16.232 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:31:15 smithi092 ceph-mon[37563]: 6.11 scrub ok 2024-03-11T21:31:16.232 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:31:15 smithi092 ceph-mon[37563]: 5.16 scrub starts 2024-03-11T21:31:16.232 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:31:15 smithi092 ceph-mon[37563]: 5.16 scrub ok 2024-03-11T21:31:16.232 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:31:15 smithi092 ceph-mon[37563]: 6.14 scrub starts 2024-03-11T21:31:16.232 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:31:15 smithi092 ceph-mon[37563]: 6.14 scrub ok 2024-03-11T21:31:16.232 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:31:16.309 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:31:16.310 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:31:16.310 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:31:16.326 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:31:15 smithi097 ceph-mon[33317]: 6.11 scrub starts 2024-03-11T21:31:16.326 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:31:15 smithi097 ceph-mon[33317]: 6.11 scrub ok 2024-03-11T21:31:16.326 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:31:15 smithi097 ceph-mon[33317]: 5.16 scrub starts 2024-03-11T21:31:16.326 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:31:15 smithi097 ceph-mon[33317]: 5.16 scrub ok 2024-03-11T21:31:16.327 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:31:15 smithi097 ceph-mon[33317]: 6.14 scrub starts 2024-03-11T21:31:16.327 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:31:15 smithi097 ceph-mon[33317]: 6.14 scrub ok 2024-03-11T21:31:16.411 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:31:16.450 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:31:16.451 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:31:16.452 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:31:16.552 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:31:16.595 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:31:16.596 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:31:16.596 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:31:16.697 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:31:16.725 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:31:16.726 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:31:16.726 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:31:16.827 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:31:16.872 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:31:16.872 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:31:16.873 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:31:16.889 INFO:teuthology.orchestra.run.smithi092.stderr:Inferring config /var/lib/ceph/00db0016-dfed-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T21:31:16.974 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:31:17.002 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:31:17.003 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:31:17.003 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:31:17.104 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:31:17.152 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:31:17.153 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:31:17.153 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:31:17.242 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:31:16 smithi092 ceph-mon[29572]: pgmap v209: 148 pgs: 148 active+clean; 717 MiB data, 3.1 GiB used, 712 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.04k op/s 2024-03-11T21:31:17.242 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:31:16 smithi092 ceph-mon[29572]: 5.13 deep-scrub starts 2024-03-11T21:31:17.242 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:31:16 smithi092 ceph-mon[29572]: 5.13 deep-scrub ok 2024-03-11T21:31:17.242 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:31:16 smithi092 ceph-mon[29572]: 5.3 scrub starts 2024-03-11T21:31:17.242 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:31:16 smithi092 ceph-mon[29572]: 5.3 scrub ok 2024-03-11T21:31:17.242 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:31:16 smithi092 ceph-mon[29572]: 5.19 scrub starts 2024-03-11T21:31:17.242 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:31:16 smithi092 ceph-mon[29572]: 5.19 scrub ok 2024-03-11T21:31:17.242 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:31:16 smithi092 ceph-mon[29572]: 7.e scrub starts 2024-03-11T21:31:17.242 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:31:16 smithi092 ceph-mon[29572]: 7.e scrub ok 2024-03-11T21:31:17.242 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:31:16 smithi092 ceph-mon[37563]: pgmap v209: 148 pgs: 148 active+clean; 717 MiB data, 3.1 GiB used, 712 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.04k op/s 2024-03-11T21:31:17.243 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:31:16 smithi092 ceph-mon[37563]: 5.13 deep-scrub starts 2024-03-11T21:31:17.243 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:31:16 smithi092 ceph-mon[37563]: 5.13 deep-scrub ok 2024-03-11T21:31:17.243 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:31:16 smithi092 ceph-mon[37563]: 5.3 scrub starts 2024-03-11T21:31:17.243 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:31:16 smithi092 ceph-mon[37563]: 5.3 scrub ok 2024-03-11T21:31:17.243 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:31:16 smithi092 ceph-mon[37563]: 5.19 scrub starts 2024-03-11T21:31:17.243 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:31:16 smithi092 ceph-mon[37563]: 5.19 scrub ok 2024-03-11T21:31:17.243 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:31:16 smithi092 ceph-mon[37563]: 7.e scrub starts 2024-03-11T21:31:17.243 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:31:16 smithi092 ceph-mon[37563]: 7.e scrub ok 2024-03-11T21:31:17.253 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:31:17.294 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:31:17.295 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:31:17.295 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:31:17.326 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:31:16 smithi097 ceph-mon[33317]: pgmap v209: 148 pgs: 148 active+clean; 717 MiB data, 3.1 GiB used, 712 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.04k op/s 2024-03-11T21:31:17.326 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:31:16 smithi097 ceph-mon[33317]: 5.13 deep-scrub starts 2024-03-11T21:31:17.326 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:31:16 smithi097 ceph-mon[33317]: 5.13 deep-scrub ok 2024-03-11T21:31:17.326 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:31:16 smithi097 ceph-mon[33317]: 5.3 scrub starts 2024-03-11T21:31:17.327 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:31:16 smithi097 ceph-mon[33317]: 5.3 scrub ok 2024-03-11T21:31:17.327 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:31:16 smithi097 ceph-mon[33317]: 5.19 scrub starts 2024-03-11T21:31:17.327 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:31:16 smithi097 ceph-mon[33317]: 5.19 scrub ok 2024-03-11T21:31:17.327 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:31:16 smithi097 ceph-mon[33317]: 7.e scrub starts 2024-03-11T21:31:17.327 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:31:16 smithi097 ceph-mon[33317]: 7.e scrub ok 2024-03-11T21:31:17.395 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:31:17.422 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:31:17.423 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:31:17.423 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:31:17.525 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:31:17.552 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:31:17.552 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:31:17.553 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:31:17.654 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:31:17.679 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:31:17.680 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:31:17.680 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:31:17.781 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:31:17.823 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:31:17.823 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:31:17.824 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:31:17.925 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:31:17.956 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:31:17.956 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:31:17.957 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:31:18.058 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:31:18.085 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:31:18.086 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:31:18.086 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:31:18.110 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:31:18 smithi092 ceph-mon[37563]: 4.9 deep-scrub starts 2024-03-11T21:31:18.110 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:31:18 smithi092 ceph-mon[37563]: 2.2 scrub starts 2024-03-11T21:31:18.110 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:31:18 smithi092 ceph-mon[37563]: 2.2 scrub ok 2024-03-11T21:31:18.111 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:31:18 smithi092 ceph-mon[37563]: 4.9 deep-scrub ok 2024-03-11T21:31:18.111 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:31:18 smithi092 ceph-mon[37563]: 2.e scrub starts 2024-03-11T21:31:18.111 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:31:18 smithi092 ceph-mon[37563]: 2.e scrub ok 2024-03-11T21:31:18.111 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:31:18 smithi092 ceph-mon[29572]: 4.9 deep-scrub starts 2024-03-11T21:31:18.111 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:31:18 smithi092 ceph-mon[29572]: 2.2 scrub starts 2024-03-11T21:31:18.111 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:31:18 smithi092 ceph-mon[29572]: 2.2 scrub ok 2024-03-11T21:31:18.111 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:31:18 smithi092 ceph-mon[29572]: 4.9 deep-scrub ok 2024-03-11T21:31:18.111 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:31:18 smithi092 ceph-mon[29572]: 2.e scrub starts 2024-03-11T21:31:18.111 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:31:18 smithi092 ceph-mon[29572]: 2.e scrub ok 2024-03-11T21:31:18.112 INFO:journalctl@ceph.mgr.y.smithi092.stdout:Mar 11 21:31:17 smithi092 ceph-00db0016-dfed-11ee-95c8-87774f69a715-mgr-y[29801]: ::ffff:172.21.15.97 - - [11/Mar/2024:21:31:17] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-11T21:31:18.187 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:31:18.214 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:31:18.215 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:31:18.215 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:31:18.235 DEBUG:teuthology.orchestra.run.smithi092:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid 00db0016-dfed-11ee-95c8-87774f69a715 -- ceph osd set noscrub 2024-03-11T21:31:18.316 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:31:18.326 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:31:18 smithi097 ceph-mon[33317]: 4.9 deep-scrub starts 2024-03-11T21:31:18.326 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:31:18 smithi097 ceph-mon[33317]: 2.2 scrub starts 2024-03-11T21:31:18.326 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:31:18 smithi097 ceph-mon[33317]: 2.2 scrub ok 2024-03-11T21:31:18.326 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:31:18 smithi097 ceph-mon[33317]: 4.9 deep-scrub ok 2024-03-11T21:31:18.326 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:31:18 smithi097 ceph-mon[33317]: 2.e scrub starts 2024-03-11T21:31:18.326 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:31:18 smithi097 ceph-mon[33317]: 2.e scrub ok 2024-03-11T21:31:18.348 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:31:18.349 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:31:18.349 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:31:18.450 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:31:18.501 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:31:18.501 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:31:18.502 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:31:18.602 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:31:18.632 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:31:18.632 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:31:18.633 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:31:18.733 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:31:18.764 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:31:18.764 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:31:18.765 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:31:18.865 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:31:18.895 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:31:18.896 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:31:18.896 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:31:18.997 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:31:19.040 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:31:19.041 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:31:19.041 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:31:19.142 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:31:19.165 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:31:19.166 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:31:19.166 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:31:19.267 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:31:19.297 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:31:19.297 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:31:19.298 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:31:19.326 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:31:19 smithi097 ceph-mon[33317]: pgmap v210: 148 pgs: 148 active+clean; 595 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail; 639 B/s rd, 0 B/s wr, 876 op/s 2024-03-11T21:31:19.326 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:31:19 smithi097 ceph-mon[33317]: 4.11 scrub starts 2024-03-11T21:31:19.326 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:31:19 smithi097 ceph-mon[33317]: 4.11 scrub ok 2024-03-11T21:31:19.360 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:31:19 smithi092 ceph-mon[29572]: pgmap v210: 148 pgs: 148 active+clean; 595 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail; 639 B/s rd, 0 B/s wr, 876 op/s 2024-03-11T21:31:19.360 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:31:19 smithi092 ceph-mon[29572]: 4.11 scrub starts 2024-03-11T21:31:19.360 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:31:19 smithi092 ceph-mon[29572]: 4.11 scrub ok 2024-03-11T21:31:19.361 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:31:19 smithi092 ceph-mon[37563]: pgmap v210: 148 pgs: 148 active+clean; 595 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail; 639 B/s rd, 0 B/s wr, 876 op/s 2024-03-11T21:31:19.361 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:31:19 smithi092 ceph-mon[37563]: 4.11 scrub starts 2024-03-11T21:31:19.361 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:31:19 smithi092 ceph-mon[37563]: 4.11 scrub ok 2024-03-11T21:31:19.398 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:31:19.454 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:31:19.455 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:31:19.455 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:31:19.556 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:31:19.599 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:31:19.600 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:31:19.600 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:31:19.622 INFO:teuthology.orchestra.run.smithi092.stdout:osd.0: {} 2024-03-11T21:31:19.664 INFO:teuthology.orchestra.run.smithi092.stdout:osd.1: {} 2024-03-11T21:31:19.701 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:31:19.708 INFO:teuthology.orchestra.run.smithi092.stdout:osd.2: {} 2024-03-11T21:31:19.727 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:31:19.727 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:31:19.728 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:31:19.753 INFO:teuthology.orchestra.run.smithi092.stdout:osd.3: {} 2024-03-11T21:31:19.809 INFO:teuthology.orchestra.run.smithi092.stdout:osd.4: {} 2024-03-11T21:31:19.828 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:31:19.855 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:31:19.856 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:31:19.856 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:31:19.858 INFO:teuthology.orchestra.run.smithi092.stdout:ERROR: (22) Invalid argument 2024-03-11T21:31:19.858 INFO:teuthology.orchestra.run.smithi092.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-11T21:31:19.859 INFO:teuthology.orchestra.run.smithi092.stdout:osd.5: {} 2024-03-11T21:31:19.902 INFO:teuthology.orchestra.run.smithi092.stdout:osd.6: {} 2024-03-11T21:31:19.943 INFO:teuthology.orchestra.run.smithi092.stdout:osd.7: {} 2024-03-11T21:31:19.947 INFO:teuthology.orchestra.run.smithi092.stderr:osd.0: osd_enable_op_tracker = '' 2024-03-11T21:31:19.948 INFO:teuthology.orchestra.run.smithi092.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-11T21:31:19.948 INFO:teuthology.orchestra.run.smithi092.stderr:osd.2: osd_enable_op_tracker = '' 2024-03-11T21:31:19.948 INFO:teuthology.orchestra.run.smithi092.stderr:osd.3: osd_enable_op_tracker = '' 2024-03-11T21:31:19.948 INFO:teuthology.orchestra.run.smithi092.stderr:osd.4: osd_enable_op_tracker = '' 2024-03-11T21:31:19.948 INFO:teuthology.orchestra.run.smithi092.stderr:osd.5: osd_enable_op_tracker = '' 2024-03-11T21:31:19.948 INFO:teuthology.orchestra.run.smithi092.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-11T21:31:19.948 INFO:teuthology.orchestra.run.smithi092.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-11T21:31:19.956 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:31:20.000 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:31:20.000 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:31:20.000 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:31:20.101 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:31:20.143 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:31:20.144 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:31:20.144 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:31:20.245 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:31:20.278 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:31:20.279 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:31:20.280 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:31:20.314 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:31:20 smithi092 ceph-mon[29572]: 2.13 scrub starts 2024-03-11T21:31:20.314 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:31:20 smithi092 ceph-mon[29572]: 2.13 scrub ok 2024-03-11T21:31:20.314 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:31:20 smithi092 ceph-mon[29572]: 6.1d scrub starts 2024-03-11T21:31:20.314 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:31:20 smithi092 ceph-mon[29572]: 6.1d scrub ok 2024-03-11T21:31:20.314 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:31:20 smithi092 ceph-mon[29572]: from='client.? v1:172.21.15.92:0/3308043167' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-11T21:31:20.314 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:31:20 smithi092 ceph-mon[37563]: 2.13 scrub starts 2024-03-11T21:31:20.314 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:31:20 smithi092 ceph-mon[37563]: 2.13 scrub ok 2024-03-11T21:31:20.314 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:31:20 smithi092 ceph-mon[37563]: 6.1d scrub starts 2024-03-11T21:31:20.314 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:31:20 smithi092 ceph-mon[37563]: 6.1d scrub ok 2024-03-11T21:31:20.315 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:31:20 smithi092 ceph-mon[37563]: from='client.? v1:172.21.15.92:0/3308043167' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-11T21:31:20.326 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:31:20 smithi097 ceph-mon[33317]: 2.13 scrub starts 2024-03-11T21:31:20.326 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:31:20 smithi097 ceph-mon[33317]: 2.13 scrub ok 2024-03-11T21:31:20.326 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:31:20 smithi097 ceph-mon[33317]: 6.1d scrub starts 2024-03-11T21:31:20.326 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:31:20 smithi097 ceph-mon[33317]: 6.1d scrub ok 2024-03-11T21:31:20.326 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:31:20 smithi097 ceph-mon[33317]: from='client.? v1:172.21.15.92:0/3308043167' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-11T21:31:20.381 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:31:20.410 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:31:20.410 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:31:20.411 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:31:20.512 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:31:20.560 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:31:20.561 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:31:20.561 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:31:20.662 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:31:20.700 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:31:20.701 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:31:20.701 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:31:20.802 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:31:20.841 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:31:20.842 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:31:20.842 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:31:20.943 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:31:20.980 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:31:20.980 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:31:20.981 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:31:21.082 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:31:21.133 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:31:21.134 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:31:21.134 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:31:21.235 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:31:21.276 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:31:21.277 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:31:21.277 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:31:21.312 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:31:21 smithi092 ceph-mon[29572]: pgmap v211: 148 pgs: 148 active+clean; 395 MiB data, 2.2 GiB used, 713 GiB / 715 GiB avail; 1.1 KiB/s rd, 0 B/s wr, 1.09k op/s 2024-03-11T21:31:21.312 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:31:21 smithi092 ceph-mon[29572]: 5.9 scrub starts 2024-03-11T21:31:21.312 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:31:21 smithi092 ceph-mon[29572]: 5.9 scrub ok 2024-03-11T21:31:21.312 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:31:21 smithi092 ceph-mon[29572]: 5.15 deep-scrub starts 2024-03-11T21:31:21.312 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:31:21 smithi092 ceph-mon[29572]: 5.15 deep-scrub ok 2024-03-11T21:31:21.313 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:31:21 smithi092 ceph-mon[37563]: pgmap v211: 148 pgs: 148 active+clean; 395 MiB data, 2.2 GiB used, 713 GiB / 715 GiB avail; 1.1 KiB/s rd, 0 B/s wr, 1.09k op/s 2024-03-11T21:31:21.313 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:31:21 smithi092 ceph-mon[37563]: 5.9 scrub starts 2024-03-11T21:31:21.313 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:31:21 smithi092 ceph-mon[37563]: 5.9 scrub ok 2024-03-11T21:31:21.313 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:31:21 smithi092 ceph-mon[37563]: 5.15 deep-scrub starts 2024-03-11T21:31:21.313 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:31:21 smithi092 ceph-mon[37563]: 5.15 deep-scrub ok 2024-03-11T21:31:21.326 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:31:21 smithi097 ceph-mon[33317]: pgmap v211: 148 pgs: 148 active+clean; 395 MiB data, 2.2 GiB used, 713 GiB / 715 GiB avail; 1.1 KiB/s rd, 0 B/s wr, 1.09k op/s 2024-03-11T21:31:21.326 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:31:21 smithi097 ceph-mon[33317]: 5.9 scrub starts 2024-03-11T21:31:21.326 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:31:21 smithi097 ceph-mon[33317]: 5.9 scrub ok 2024-03-11T21:31:21.326 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:31:21 smithi097 ceph-mon[33317]: 5.15 deep-scrub starts 2024-03-11T21:31:21.326 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:31:21 smithi097 ceph-mon[33317]: 5.15 deep-scrub ok 2024-03-11T21:31:21.378 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:31:21.412 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:31:21.412 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:31:21.413 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:31:21.513 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:31:21.548 INFO:teuthology.orchestra.run.smithi092.stderr:Inferring config /var/lib/ceph/00db0016-dfed-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T21:31:21.556 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:31:21.557 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:31:21.557 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:31:21.658 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:31:21.708 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:31:21.709 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:31:21.709 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:31:21.810 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:31:21.842 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:31:21.843 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:31:21.843 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:31:21.944 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:31:21.976 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:31:21.977 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:31:21.977 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:31:22.025 DEBUG:teuthology.orchestra.run.smithi092:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid 00db0016-dfed-11ee-95c8-87774f69a715 -- ceph daemon osd.1 dump_historic_ops 2024-03-11T21:31:22.078 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:31:22.106 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:31:22.106 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:31:22.107 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:31:22.208 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:31:22.235 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:31:22.236 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:31:22.236 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:31:22.237 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:31:22 smithi097 ceph-mon[33317]: 4.b scrub starts 2024-03-11T21:31:22.237 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:31:22 smithi097 ceph-mon[33317]: 4.b scrub ok 2024-03-11T21:31:22.238 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:31:22 smithi097 ceph-mon[33317]: 4.1d scrub starts 2024-03-11T21:31:22.238 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:31:22 smithi097 ceph-mon[33317]: 4.1d scrub ok 2024-03-11T21:31:22.238 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:31:22 smithi097 ceph-mon[33317]: 2.b scrub starts 2024-03-11T21:31:22.238 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:31:22 smithi097 ceph-mon[33317]: 2.b scrub ok 2024-03-11T21:31:22.238 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:31:22 smithi097 ceph-mon[33317]: pgmap v212: 148 pgs: 148 active+clean; 395 MiB data, 2.2 GiB used, 713 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 1.02k op/s 2024-03-11T21:31:22.238 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:31:22 smithi097 ceph-mon[33317]: 2.1 scrub starts 2024-03-11T21:31:22.238 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:31:22 smithi097 ceph-mon[33317]: 2.1 scrub ok 2024-03-11T21:31:22.238 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:31:22 smithi097 ceph-mon[33317]: from='mgr.14607 ' entity='mgr.y' 2024-03-11T21:31:22.238 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:31:22 smithi097 ceph-mon[33317]: from='mgr.14607 v1:172.21.15.92:0/1702717597' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T21:31:22.337 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:31:22.360 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:31:22 smithi092 ceph-mon[29572]: 4.b scrub starts 2024-03-11T21:31:22.361 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:31:22 smithi092 ceph-mon[29572]: 4.b scrub ok 2024-03-11T21:31:22.361 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:31:22 smithi092 ceph-mon[29572]: 4.1d scrub starts 2024-03-11T21:31:22.361 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:31:22 smithi092 ceph-mon[29572]: 4.1d scrub ok 2024-03-11T21:31:22.361 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:31:22 smithi092 ceph-mon[29572]: 2.b scrub starts 2024-03-11T21:31:22.361 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:31:22 smithi092 ceph-mon[29572]: 2.b scrub ok 2024-03-11T21:31:22.361 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:31:22 smithi092 ceph-mon[29572]: pgmap v212: 148 pgs: 148 active+clean; 395 MiB data, 2.2 GiB used, 713 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 1.02k op/s 2024-03-11T21:31:22.361 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:31:22 smithi092 ceph-mon[29572]: 2.1 scrub starts 2024-03-11T21:31:22.361 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:31:22 smithi092 ceph-mon[29572]: 2.1 scrub ok 2024-03-11T21:31:22.361 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:31:22 smithi092 ceph-mon[29572]: from='mgr.14607 ' entity='mgr.y' 2024-03-11T21:31:22.361 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:31:22 smithi092 ceph-mon[29572]: from='mgr.14607 v1:172.21.15.92:0/1702717597' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T21:31:22.364 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:31:22 smithi092 ceph-mon[37563]: 4.b scrub starts 2024-03-11T21:31:22.364 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:31:22 smithi092 ceph-mon[37563]: 4.b scrub ok 2024-03-11T21:31:22.364 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:31:22 smithi092 ceph-mon[37563]: 4.1d scrub starts 2024-03-11T21:31:22.364 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:31:22 smithi092 ceph-mon[37563]: 4.1d scrub ok 2024-03-11T21:31:22.364 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:31:22 smithi092 ceph-mon[37563]: 2.b scrub starts 2024-03-11T21:31:22.365 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:31:22 smithi092 ceph-mon[37563]: 2.b scrub ok 2024-03-11T21:31:22.365 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:31:22 smithi092 ceph-mon[37563]: pgmap v212: 148 pgs: 148 active+clean; 395 MiB data, 2.2 GiB used, 713 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 1.02k op/s 2024-03-11T21:31:22.365 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:31:22 smithi092 ceph-mon[37563]: 2.1 scrub starts 2024-03-11T21:31:22.365 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:31:22 smithi092 ceph-mon[37563]: 2.1 scrub ok 2024-03-11T21:31:22.365 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:31:22 smithi092 ceph-mon[37563]: from='mgr.14607 ' entity='mgr.y' 2024-03-11T21:31:22.365 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:31:22 smithi092 ceph-mon[37563]: from='mgr.14607 v1:172.21.15.92:0/1702717597' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T21:31:22.380 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:31:22.380 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:31:22.381 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:31:22.481 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:31:22.540 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:31:22.540 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:31:22.540 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:31:22.576 INFO:journalctl@ceph.iscsi.iscsi.a.smithi097.stdout:Mar 11 21:31:22 smithi097 ceph-00db0016-dfed-11ee-95c8-87774f69a715-iscsi-iscsi-a[60943]: debug there is no tcmu-runner data available 2024-03-11T21:31:22.641 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:31:22.670 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:31:22.670 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:31:22.671 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:31:22.772 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:31:22.816 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:31:22.817 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:31:22.817 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:31:22.918 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:31:22.956 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:31:22.957 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:31:22.957 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:31:23.021 INFO:teuthology.orchestra.run.smithi092.stdout: 2024-03-11T21:31:23.026 INFO:teuthology.orchestra.run.smithi092.stderr:dumped all 2024-03-11T21:31:23.058 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:31:23.091 DEBUG:teuthology.orchestra.run.smithi092:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid 00db0016-dfed-11ee-95c8-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-03-11T21:31:23.102 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:31:23.103 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:31:23.103 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:31:23.204 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:31:23.247 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:31:23.248 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:31:23.249 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:31:23.326 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:31:23 smithi097 ceph-mon[33317]: 5.0 scrub starts 2024-03-11T21:31:23.326 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:31:23 smithi097 ceph-mon[33317]: 5.0 scrub ok 2024-03-11T21:31:23.326 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:31:23 smithi097 ceph-mon[33317]: from='client.24469 v1:172.21.15.97:0/404208310' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:31:23.327 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:31:23 smithi097 ceph-mon[33317]: from='client.15075 v1:172.21.15.92:0/4172324767' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-11T21:31:23.350 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:31:23.360 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:31:23 smithi092 ceph-mon[29572]: 5.0 scrub starts 2024-03-11T21:31:23.360 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:31:23 smithi092 ceph-mon[29572]: 5.0 scrub ok 2024-03-11T21:31:23.361 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:31:23 smithi092 ceph-mon[29572]: from='client.24469 v1:172.21.15.97:0/404208310' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:31:23.361 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:31:23 smithi092 ceph-mon[29572]: from='client.15075 v1:172.21.15.92:0/4172324767' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-11T21:31:23.361 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:31:23 smithi092 ceph-mon[37563]: 5.0 scrub starts 2024-03-11T21:31:23.361 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:31:23 smithi092 ceph-mon[37563]: 5.0 scrub ok 2024-03-11T21:31:23.361 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:31:23 smithi092 ceph-mon[37563]: from='client.24469 v1:172.21.15.97:0/404208310' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:31:23.361 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:31:23 smithi092 ceph-mon[37563]: from='client.15075 v1:172.21.15.92:0/4172324767' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-11T21:31:23.375 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:31:23.376 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:31:23.376 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:31:23.477 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:31:23.507 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:31:23.508 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:31:23.509 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:31:23.590 INFO:tasks.radosbench.radosbench.0.smithi092.stdout:Removed 121908 objects 2024-03-11T21:31:23.590 INFO:tasks.radosbench.radosbench.0.smithi092.stdout:Clean up completed and total clean up time :127.937 2024-03-11T21:31:23.594 INFO:teuthology.orchestra.run.smithi092.stderr:Inferring config /var/lib/ceph/00db0016-dfed-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T21:31:23.609 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:31:23.634 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:31:23.634 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:31:23.634 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:31:23.736 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:31:23.760 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:31:23.760 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:31:23.760 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:31:23.861 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:31:23.885 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:31:23.886 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:31:23.886 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:31:23.987 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:31:24.012 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:31:24.013 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:31:24.013 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:31:24.078 INFO:teuthology.orchestra.run.smithi092.stdout:{"pg_ready":true,"pg_map":{"version":212,"stamp":"2024-03-11T21:31:21.079151+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":414456256,"num_objects":6514,"num_object_clones":0,"num_object_copies":19542,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6514,"num_whiteouts":0,"num_read":1267,"num_read_kb":1113,"num_write":238117,"num_write_kb":7804574,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":21,"num_bytes_recovered":1771324,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":90233,"ondisk_log_size":90233,"up":444,"acting":444,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":444,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":749961216,"kb_used":2312040,"kb_used_data":1313000,"kb_used_omap":12,"kb_used_meta":998963,"kb_avail":747649176,"statfs":{"total":767960285184,"available":765592756224,"internally_reserved":0,"allocated":1344512000,"data_stored":1341389512,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12752,"internal_metadata":1022938672},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[3,7,4,9,4,3,2],"upper_bound":128},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-668925952,"num_objects":-10207,"num_object_clones":0,"num_object_copies":-30621,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-10207,"num_whiteouts":0,"num_read":10,"num_read_kb":10,"num_write":10207,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"10.002701"},"pg_stats":[{"pgid":"6.1b","version":"0'0","reported_seq":182,"reported_epoch":146,"state":"active+clean","last_fresh":"2024-03-11T21:31:12.751767+0000","last_change":"2024-03-11T21:31:06.310603+0000","last_active":"2024-03-11T21:31:12.751767+0000","last_peered":"2024-03-11T21:31:12.751767+0000","last_clean":"2024-03-11T21:31:12.751767+0000","last_became_active":"2024-03-11T21:25:45.302931+0000","last_became_peered":"2024-03-11T21:25:45.302931+0000","last_unstale":"2024-03-11T21:31:12.751767+0000","last_undegraded":"2024-03-11T21:31:12.751767+0000","last_fullsized":"2024-03-11T21:31:12.751767+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:31:06.310490+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:43.559452+0000","last_clean_scrub_stamp":"2024-03-11T21:31:06.310490+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:32:16.162571+0000","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":187,"reported_epoch":146,"state":"active+clean","last_fresh":"2024-03-11T21:31:12.732783+0000","last_change":"2024-03-11T21:31:10.598620+0000","last_active":"2024-03-11T21:31:12.732783+0000","last_peered":"2024-03-11T21:31:12.732783+0000","last_clean":"2024-03-11T21:31:12.732783+0000","last_became_active":"2024-03-11T21:25:42.578893+0000","last_became_peered":"2024-03-11T21:25:42.578893+0000","last_unstale":"2024-03-11T21:31:12.732783+0000","last_undegraded":"2024-03-11T21:31:12.732783+0000","last_fullsized":"2024-03-11T21:31:12.732783+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:31:10.598515+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:41.544550+0000","last_clean_scrub_stamp":"2024-03-11T21:31:10.598515+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:32:32.512946+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1f","version":"0'0","reported_seq":195,"reported_epoch":146,"state":"active+clean","last_fresh":"2024-03-11T21:31:12.741206+0000","last_change":"2024-03-11T21:30:14.771032+0000","last_active":"2024-03-11T21:31:12.741206+0000","last_peered":"2024-03-11T21:31:12.741206+0000","last_clean":"2024-03-11T21:31:12.741206+0000","last_became_active":"2024-03-11T21:25:38.540510+0000","last_became_peered":"2024-03-11T21:25:38.540510+0000","last_unstale":"2024-03-11T21:31:12.741206+0000","last_undegraded":"2024-03-11T21:31:12.741206+0000","last_fullsized":"2024-03-11T21:31:12.741206+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:30:14.770920+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:37.512018+0000","last_clean_scrub_stamp":"2024-03-11T21:30:14.770920+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:31:30.680425+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,4],"acting":[0,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.19","version":"74'15","reported_seq":215,"reported_epoch":146,"state":"active+clean","last_fresh":"2024-03-11T21:31:12.753904+0000","last_change":"2024-03-11T21:31:07.311590+0000","last_active":"2024-03-11T21:31:12.753904+0000","last_peered":"2024-03-11T21:31:12.753904+0000","last_clean":"2024-03-11T21:31:12.753904+0000","last_became_active":"2024-03-11T21:25:40.554207+0000","last_became_peered":"2024-03-11T21:25:40.554207+0000","last_unstale":"2024-03-11T21:31:12.753904+0000","last_undegraded":"2024-03-11T21:31:12.753904+0000","last_fullsized":"2024-03-11T21:31:12.753904+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'15","last_scrub_stamp":"2024-03-11T21:31:07.311470+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:39.525554+0000","last_clean_scrub_stamp":"2024-03-11T21:31:07.311470+0000","objects_scrubbed":7,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:32:36.263715+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.1a","version":"0'0","reported_seq":175,"reported_epoch":146,"state":"active+clean","last_fresh":"2024-03-11T21:31:12.733928+0000","last_change":"2024-03-11T21:29:41.255464+0000","last_active":"2024-03-11T21:31:12.733928+0000","last_peered":"2024-03-11T21:31:12.733928+0000","last_clean":"2024-03-11T21:31:12.733928+0000","last_became_active":"2024-03-11T21:25:44.590923+0000","last_became_peered":"2024-03-11T21:25:44.590923+0000","last_unstale":"2024-03-11T21:31:12.733928+0000","last_undegraded":"2024-03-11T21:31:12.733928+0000","last_fullsized":"2024-03-11T21:31:12.733928+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:29:41.255337+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:43.559452+0000","last_clean_scrub_stamp":"2024-03-11T21:29:41.255337+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,1],"acting":[4,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":187,"reported_epoch":146,"state":"active+clean","last_fresh":"2024-03-11T21:31:15.812505+0000","last_change":"2024-03-11T21:31:15.812546+0000","last_active":"2024-03-11T21:31:15.812505+0000","last_peered":"2024-03-11T21:31:15.812505+0000","last_clean":"2024-03-11T21:31:15.812505+0000","last_became_active":"2024-03-11T21:25:42.578424+0000","last_became_peered":"2024-03-11T21:25:42.578424+0000","last_unstale":"2024-03-11T21:31:15.812505+0000","last_undegraded":"2024-03-11T21:31:15.812505+0000","last_fullsized":"2024-03-11T21:31:15.812505+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:31:15.812251+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:41.544550+0000","last_clean_scrub_stamp":"2024-03-11T21:31:15.812251+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:32:21.508207+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,7],"acting":[1,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1e","version":"0'0","reported_seq":194,"reported_epoch":146,"state":"active+clean","last_fresh":"2024-03-11T21:31:12.750797+0000","last_change":"2024-03-11T21:30:17.913666+0000","last_active":"2024-03-11T21:31:12.750797+0000","last_peered":"2024-03-11T21:31:12.750797+0000","last_clean":"2024-03-11T21:31:12.750797+0000","last_became_active":"2024-03-11T21:25:38.537283+0000","last_became_peered":"2024-03-11T21:25:38.537283+0000","last_unstale":"2024-03-11T21:31:12.750797+0000","last_undegraded":"2024-03-11T21:31:12.750797+0000","last_fullsized":"2024-03-11T21:31:12.750797+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:30:17.913550+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:37.512018+0000","last_clean_scrub_stamp":"2024-03-11T21:30:17.913550+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:31:38.998596+0000","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.18","version":"74'9","reported_seq":207,"reported_epoch":146,"state":"active+clean","last_fresh":"2024-03-11T21:31:12.732959+0000","last_change":"2024-03-11T21:29:45.187402+0000","last_active":"2024-03-11T21:31:12.732959+0000","last_peered":"2024-03-11T21:31:12.732959+0000","last_clean":"2024-03-11T21:31:12.732959+0000","last_became_active":"2024-03-11T21:25:40.552216+0000","last_became_peered":"2024-03-11T21:25:40.552216+0000","last_unstale":"2024-03-11T21:31:12.732959+0000","last_undegraded":"2024-03-11T21:31:12.732959+0000","last_fullsized":"2024-03-11T21:31:12.732959+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'9","last_scrub_stamp":"2024-03-11T21:29:45.187250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:39.525554+0000","last_clean_scrub_stamp":"2024-03-11T21:29:45.187250+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":182,"reported_epoch":146,"state":"active+clean","last_fresh":"2024-03-11T21:31:12.755383+0000","last_change":"2024-03-11T21:29:47.008124+0000","last_active":"2024-03-11T21:31:12.755383+0000","last_peered":"2024-03-11T21:31:12.755383+0000","last_clean":"2024-03-11T21:31:12.755383+0000","last_became_active":"2024-03-11T21:25:44.590234+0000","last_became_peered":"2024-03-11T21:25:44.590234+0000","last_unstale":"2024-03-11T21:31:12.755383+0000","last_undegraded":"2024-03-11T21:31:12.755383+0000","last_fullsized":"2024-03-11T21:31:12.755383+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:29:47.008018+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:43.559452+0000","last_clean_scrub_stamp":"2024-03-11T21:29:47.008018+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":133,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3],"acting":[5,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":178,"reported_epoch":146,"state":"active+clean","last_fresh":"2024-03-11T21:31:12.741752+0000","last_change":"2024-03-11T21:29:21.533307+0000","last_active":"2024-03-11T21:31:12.741752+0000","last_peered":"2024-03-11T21:31:12.741752+0000","last_clean":"2024-03-11T21:31:12.741752+0000","last_became_active":"2024-03-11T21:25:42.580533+0000","last_became_peered":"2024-03-11T21:25:42.580533+0000","last_unstale":"2024-03-11T21:31:12.741752+0000","last_undegraded":"2024-03-11T21:31:12.741752+0000","last_fullsized":"2024-03-11T21:31:12.741752+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:29:21.533216+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:41.544550+0000","last_clean_scrub_stamp":"2024-03-11T21:29:21.533216+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.1d","version":"0'0","reported_seq":194,"reported_epoch":146,"state":"active+clean","last_fresh":"2024-03-11T21:31:12.742299+0000","last_change":"2024-03-11T21:29:39.481465+0000","last_active":"2024-03-11T21:31:12.742299+0000","last_peered":"2024-03-11T21:31:12.742299+0000","last_clean":"2024-03-11T21:31:12.742299+0000","last_became_active":"2024-03-11T21:25:38.542628+0000","last_became_peered":"2024-03-11T21:25:38.542628+0000","last_unstale":"2024-03-11T21:31:12.742299+0000","last_undegraded":"2024-03-11T21:31:12.742299+0000","last_fullsized":"2024-03-11T21:31:12.742299+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:29:39.481382+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:37.512018+0000","last_clean_scrub_stamp":"2024-03-11T21:29:39.481382+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,0],"acting":[7,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.1b","version":"74'5","reported_seq":193,"reported_epoch":146,"state":"active+clean","last_fresh":"2024-03-11T21:31:12.733382+0000","last_change":"2024-03-11T21:29:40.303599+0000","last_active":"2024-03-11T21:31:12.733382+0000","last_peered":"2024-03-11T21:31:12.733382+0000","last_clean":"2024-03-11T21:31:12.733382+0000","last_became_active":"2024-03-11T21:25:40.555826+0000","last_became_peered":"2024-03-11T21:25:40.555826+0000","last_unstale":"2024-03-11T21:31:12.733382+0000","last_undegraded":"2024-03-11T21:31:12.733382+0000","last_fullsized":"2024-03-11T21:31:12.733382+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'5","last_scrub_stamp":"2024-03-11T21:29:40.303460+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:39.525554+0000","last_clean_scrub_stamp":"2024-03-11T21:29:40.303460+0000","objects_scrubbed":2,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":11,"num_read_kb":7,"num_write":6,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,1],"acting":[4,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":175,"reported_epoch":146,"state":"active+clean","last_fresh":"2024-03-11T21:31:12.737236+0000","last_change":"2024-03-11T21:29:43.509900+0000","last_active":"2024-03-11T21:31:12.737236+0000","last_peered":"2024-03-11T21:31:12.737236+0000","last_clean":"2024-03-11T21:31:12.737236+0000","last_became_active":"2024-03-11T21:25:44.589786+0000","last_became_peered":"2024-03-11T21:25:44.589786+0000","last_unstale":"2024-03-11T21:31:12.737236+0000","last_undegraded":"2024-03-11T21:31:12.737236+0000","last_fullsized":"2024-03-11T21:31:12.737236+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:29:43.509792+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:43.559452+0000","last_clean_scrub_stamp":"2024-03-11T21:29:43.509792+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,7],"acting":[0,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":186,"reported_epoch":146,"state":"active+clean","last_fresh":"2024-03-11T21:31:12.755877+0000","last_change":"2024-03-11T21:31:00.533483+0000","last_active":"2024-03-11T21:31:12.755877+0000","last_peered":"2024-03-11T21:31:12.755877+0000","last_clean":"2024-03-11T21:31:12.755877+0000","last_became_active":"2024-03-11T21:25:42.573831+0000","last_became_peered":"2024-03-11T21:25:42.573831+0000","last_unstale":"2024-03-11T21:31:12.755877+0000","last_undegraded":"2024-03-11T21:31:12.755877+0000","last_fullsized":"2024-03-11T21:31:12.755877+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:31:00.533376+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:41.544550+0000","last_clean_scrub_stamp":"2024-03-11T21:31:00.533376+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:32:04.433484+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,7],"acting":[5,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1c","version":"0'0","reported_seq":194,"reported_epoch":146,"state":"active+clean","last_fresh":"2024-03-11T21:31:12.741971+0000","last_change":"2024-03-11T21:30:11.669644+0000","last_active":"2024-03-11T21:31:12.741971+0000","last_peered":"2024-03-11T21:31:12.741971+0000","last_clean":"2024-03-11T21:31:12.741971+0000","last_became_active":"2024-03-11T21:25:38.541364+0000","last_became_peered":"2024-03-11T21:25:38.541364+0000","last_unstale":"2024-03-11T21:31:12.741971+0000","last_undegraded":"2024-03-11T21:31:12.741971+0000","last_fullsized":"2024-03-11T21:31:12.741971+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:30:11.668879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:28:32.566437+0000","last_clean_scrub_stamp":"2024-03-11T21:30:11.668879+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:31:15.011614+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.1a","version":"74'9","reported_seq":207,"reported_epoch":146,"state":"active+clean","last_fresh":"2024-03-11T21:31:12.734014+0000","last_change":"2024-03-11T21:29:44.227568+0000","last_active":"2024-03-11T21:31:12.734014+0000","last_peered":"2024-03-11T21:31:12.734014+0000","last_clean":"2024-03-11T21:31:12.734014+0000","last_became_active":"2024-03-11T21:25:40.556629+0000","last_became_peered":"2024-03-11T21:25:40.556629+0000","last_unstale":"2024-03-11T21:31:12.734014+0000","last_undegraded":"2024-03-11T21:31:12.734014+0000","last_fullsized":"2024-03-11T21:31:12.734014+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'9","last_scrub_stamp":"2024-03-11T21:29:44.227421+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:39.525554+0000","last_clean_scrub_stamp":"2024-03-11T21:29:44.227421+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0],"acting":[4,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1f","version":"0'0","reported_seq":182,"reported_epoch":146,"state":"active+clean","last_fresh":"2024-03-11T21:31:12.756177+0000","last_change":"2024-03-11T21:31:04.253653+0000","last_active":"2024-03-11T21:31:12.756177+0000","last_peered":"2024-03-11T21:31:12.756177+0000","last_clean":"2024-03-11T21:31:12.756177+0000","last_became_active":"2024-03-11T21:25:45.302637+0000","last_became_peered":"2024-03-11T21:25:45.302637+0000","last_unstale":"2024-03-11T21:31:12.756177+0000","last_undegraded":"2024-03-11T21:31:12.756177+0000","last_fullsized":"2024-03-11T21:31:12.756177+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:31:04.253137+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:43.559452+0000","last_clean_scrub_stamp":"2024-03-11T21:31:04.253137+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:32:05.471991+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1c","version":"0'0","reported_seq":187,"reported_epoch":146,"state":"active+clean","last_fresh":"2024-03-11T21:31:12.734337+0000","last_change":"2024-03-11T21:30:59.496458+0000","last_active":"2024-03-11T21:31:12.734337+0000","last_peered":"2024-03-11T21:31:12.734337+0000","last_clean":"2024-03-11T21:31:12.734337+0000","last_became_active":"2024-03-11T21:25:42.576064+0000","last_became_peered":"2024-03-11T21:25:42.576064+0000","last_unstale":"2024-03-11T21:31:12.734337+0000","last_undegraded":"2024-03-11T21:31:12.734337+0000","last_fullsized":"2024-03-11T21:31:12.734337+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:30:59.496330+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:41.544550+0000","last_clean_scrub_stamp":"2024-03-11T21:30:59.496330+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:32:27.314836+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1b","version":"0'0","reported_seq":194,"reported_epoch":146,"state":"active+clean","last_fresh":"2024-03-11T21:31:12.751946+0000","last_change":"2024-03-11T21:31:05.273080+0000","last_active":"2024-03-11T21:31:12.751946+0000","last_peered":"2024-03-11T21:31:12.751946+0000","last_clean":"2024-03-11T21:31:12.751946+0000","last_became_active":"2024-03-11T21:25:38.537520+0000","last_became_peered":"2024-03-11T21:25:38.537520+0000","last_unstale":"2024-03-11T21:31:12.751946+0000","last_undegraded":"2024-03-11T21:31:12.751946+0000","last_fullsized":"2024-03-11T21:31:12.751946+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:31:05.272969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:28:33.582712+0000","last_clean_scrub_stamp":"2024-03-11T21:31:05.272969+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:32:11.268801+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1d","version":"74'12","reported_seq":205,"reported_epoch":146,"state":"active+clean","last_fresh":"2024-03-11T21:31:12.750031+0000","last_change":"2024-03-11T21:29:17.662843+0000","last_active":"2024-03-11T21:31:12.750031+0000","last_peered":"2024-03-11T21:31:12.750031+0000","last_clean":"2024-03-11T21:31:12.750031+0000","last_became_active":"2024-03-11T21:25:40.556330+0000","last_became_peered":"2024-03-11T21:25:40.556330+0000","last_unstale":"2024-03-11T21:31:12.750031+0000","last_undegraded":"2024-03-11T21:31:12.750031+0000","last_fullsized":"2024-03-11T21:31:12.750031+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'12","last_scrub_stamp":"2024-03-11T21:29:17.662708+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:39.525554+0000","last_clean_scrub_stamp":"2024-03-11T21:29:17.662708+0000","objects_scrubbed":5,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":25,"num_read_kb":16,"num_write":14,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.1e","version":"0'0","reported_seq":175,"reported_epoch":146,"state":"active+clean","last_fresh":"2024-03-11T21:31:12.733935+0000","last_change":"2024-03-11T21:30:11.322364+0000","last_active":"2024-03-11T21:31:12.733935+0000","last_peered":"2024-03-11T21:31:12.733935+0000","last_clean":"2024-03-11T21:31:12.733935+0000","last_became_active":"2024-03-11T21:25:45.300355+0000","last_became_peered":"2024-03-11T21:25:45.300355+0000","last_unstale":"2024-03-11T21:31:12.733935+0000","last_undegraded":"2024-03-11T21:31:12.733935+0000","last_fullsized":"2024-03-11T21:31:12.733935+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:30:11.322178+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:43.559452+0000","last_clean_scrub_stamp":"2024-03-11T21:30:11.322178+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:31:39.815208+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5],"acting":[4,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1d","version":"0'0","reported_seq":187,"reported_epoch":146,"state":"active+clean","last_fresh":"2024-03-11T21:31:12.760436+0000","last_change":"2024-03-11T21:30:18.678642+0000","last_active":"2024-03-11T21:31:12.760436+0000","last_peered":"2024-03-11T21:31:12.760436+0000","last_clean":"2024-03-11T21:31:12.760436+0000","last_became_active":"2024-03-11T21:25:42.579407+0000","last_became_peered":"2024-03-11T21:25:42.579407+0000","last_unstale":"2024-03-11T21:31:12.760436+0000","last_undegraded":"2024-03-11T21:31:12.760436+0000","last_fullsized":"2024-03-11T21:31:12.760436+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:30:18.678531+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:30:18.678531+0000","last_clean_scrub_stamp":"2024-03-11T21:30:18.678531+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:31:21.186234+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1a","version":"0'0","reported_seq":195,"reported_epoch":146,"state":"active+clean","last_fresh":"2024-03-11T21:31:12.744134+0000","last_change":"2024-03-11T21:30:11.816854+0000","last_active":"2024-03-11T21:31:12.744134+0000","last_peered":"2024-03-11T21:31:12.744134+0000","last_clean":"2024-03-11T21:31:12.744134+0000","last_became_active":"2024-03-11T21:25:38.539725+0000","last_became_peered":"2024-03-11T21:25:38.539725+0000","last_unstale":"2024-03-11T21:31:12.744134+0000","last_undegraded":"2024-03-11T21:31:12.744134+0000","last_fullsized":"2024-03-11T21:31:12.744134+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:30:11.816720+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:37.512018+0000","last_clean_scrub_stamp":"2024-03-11T21:30:11.816720+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:31:40.935055+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.1c","version":"74'15","reported_seq":215,"reported_epoch":146,"state":"active+clean","last_fresh":"2024-03-11T21:31:12.743391+0000","last_change":"2024-03-11T21:31:01.272353+0000","last_active":"2024-03-11T21:31:12.743391+0000","last_peered":"2024-03-11T21:31:12.743391+0000","last_clean":"2024-03-11T21:31:12.743391+0000","last_became_active":"2024-03-11T21:25:40.551453+0000","last_became_peered":"2024-03-11T21:25:40.551453+0000","last_unstale":"2024-03-11T21:31:12.743391+0000","last_undegraded":"2024-03-11T21:31:12.743391+0000","last_fullsized":"2024-03-11T21:31:12.743391+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'15","last_scrub_stamp":"2024-03-11T21:31:01.272189+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:39.525554+0000","last_clean_scrub_stamp":"2024-03-11T21:31:01.272189+0000","objects_scrubbed":7,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:32:24.109315+0000","scrub_duration":22,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.1d","version":"0'0","reported_seq":175,"reported_epoch":146,"state":"active+clean","last_fresh":"2024-03-11T21:31:12.759831+0000","last_change":"2024-03-11T21:29:12.415949+0000","last_active":"2024-03-11T21:31:12.759831+0000","last_peered":"2024-03-11T21:31:12.759831+0000","last_clean":"2024-03-11T21:31:12.759831+0000","last_became_active":"2024-03-11T21:25:44.588845+0000","last_became_peered":"2024-03-11T21:25:44.588845+0000","last_unstale":"2024-03-11T21:31:12.759831+0000","last_undegraded":"2024-03-11T21:31:12.759831+0000","last_fullsized":"2024-03-11T21:31:12.759831+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:29:12.415585+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:43.559452+0000","last_clean_scrub_stamp":"2024-03-11T21:29:12.415585+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.1e","version":"0'0","reported_seq":187,"reported_epoch":146,"state":"active+clean","last_fresh":"2024-03-11T21:31:12.739964+0000","last_change":"2024-03-11T21:30:26.111125+0000","last_active":"2024-03-11T21:31:12.739964+0000","last_peered":"2024-03-11T21:31:12.739964+0000","last_clean":"2024-03-11T21:31:12.739964+0000","last_became_active":"2024-03-11T21:25:42.582959+0000","last_became_peered":"2024-03-11T21:25:42.582959+0000","last_unstale":"2024-03-11T21:31:12.739964+0000","last_undegraded":"2024-03-11T21:31:12.739964+0000","last_fullsized":"2024-03-11T21:31:12.739964+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:30:26.110855+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:26:58.335431+0000","last_clean_scrub_stamp":"2024-03-11T21:30:26.110855+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:31:36.208967+0000","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.19","version":"67'2","reported_seq":193,"reported_epoch":146,"state":"active+clean","last_fresh":"2024-03-11T21:31:12.754066+0000","last_change":"2024-03-11T21:29:42.815863+0000","last_active":"2024-03-11T21:31:12.754066+0000","last_peered":"2024-03-11T21:31:12.754066+0000","last_clean":"2024-03-11T21:31:12.754066+0000","last_became_active":"2024-03-11T21:25:38.539243+0000","last_became_peered":"2024-03-11T21:25:38.539243+0000","last_unstale":"2024-03-11T21:31:12.754066+0000","last_undegraded":"2024-03-11T21:31:12.754066+0000","last_fullsized":"2024-03-11T21:31:12.754066+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'2","last_scrub_stamp":"2024-03-11T21:29:42.815740+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:37.512018+0000","last_clean_scrub_stamp":"2024-03-11T21:29:42.815740+0000","objects_scrubbed":2,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":957,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":7,"num_read_kb":7,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,0],"acting":[3,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1f","version":"124'16","reported_seq":223,"reported_epoch":146,"state":"active+clean","last_fresh":"2024-03-11T21:31:12.744749+0000","last_change":"2024-03-11T21:31:06.449601+0000","last_active":"2024-03-11T21:31:12.744749+0000","last_peered":"2024-03-11T21:31:12.744749+0000","last_clean":"2024-03-11T21:31:12.744749+0000","last_became_active":"2024-03-11T21:25:40.554686+0000","last_became_peered":"2024-03-11T21:25:40.554686+0000","last_unstale":"2024-03-11T21:31:12.744749+0000","last_undegraded":"2024-03-11T21:31:12.744749+0000","last_fullsized":"2024-03-11T21:31:12.744749+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"124'16","last_scrub_stamp":"2024-03-11T21:31:06.449509+0000","last_deep_scrub":"124'16","last_deep_scrub_stamp":"2024-03-11T21:29:30.873490+0000","last_clean_scrub_stamp":"2024-03-11T21:31:06.449509+0000","objects_scrubbed":5,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:32:20.104923+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,1],"acting":[6,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.1c","version":"74'1","reported_seq":183,"reported_epoch":146,"state":"active+clean","last_fresh":"2024-03-11T21:31:12.742316+0000","last_change":"2024-03-11T21:31:07.489348+0000","last_active":"2024-03-11T21:31:12.742316+0000","last_peered":"2024-03-11T21:31:12.742316+0000","last_clean":"2024-03-11T21:31:12.742316+0000","last_became_active":"2024-03-11T21:25:44.593326+0000","last_became_peered":"2024-03-11T21:25:44.593326+0000","last_unstale":"2024-03-11T21:31:12.742316+0000","last_undegraded":"2024-03-11T21:31:12.742316+0000","last_fullsized":"2024-03-11T21:31:12.742316+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'1","last_scrub_stamp":"2024-03-11T21:31:07.489215+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:43.559452+0000","last_clean_scrub_stamp":"2024-03-11T21:31:07.489215+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:32:24.472953+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":370,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.1f","version":"74'8","reported_seq":274,"reported_epoch":146,"state":"active+clean","last_fresh":"2024-03-11T21:31:12.744756+0000","last_change":"2024-03-11T21:31:02.499201+0000","last_active":"2024-03-11T21:31:12.744756+0000","last_peered":"2024-03-11T21:31:12.744756+0000","last_clean":"2024-03-11T21:31:12.744756+0000","last_became_active":"2024-03-11T21:25:42.577401+0000","last_became_peered":"2024-03-11T21:25:42.577401+0000","last_unstale":"2024-03-11T21:31:12.744756+0000","last_undegraded":"2024-03-11T21:31:12.744756+0000","last_fullsized":"2024-03-11T21:31:12.744756+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'8","last_scrub_stamp":"2024-03-11T21:31:02.499061+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:41.544550+0000","last_clean_scrub_stamp":"2024-03-11T21:31:02.499061+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:32:20.379279+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.18","version":"67'1","reported_seq":200,"reported_epoch":146,"state":"active+clean","last_fresh":"2024-03-11T21:31:12.755354+0000","last_change":"2024-03-11T21:29:35.918627+0000","last_active":"2024-03-11T21:31:12.755354+0000","last_peered":"2024-03-11T21:31:12.755354+0000","last_clean":"2024-03-11T21:31:12.755354+0000","last_became_active":"2024-03-11T21:25:38.538321+0000","last_became_peered":"2024-03-11T21:25:38.538321+0000","last_unstale":"2024-03-11T21:31:12.755354+0000","last_undegraded":"2024-03-11T21:31:12.755354+0000","last_fullsized":"2024-03-11T21:31:12.755354+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'1","last_scrub_stamp":"2024-03-11T21:29:35.918494+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:37.512018+0000","last_clean_scrub_stamp":"2024-03-11T21:29:35.918494+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":436,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":7,"num_read_kb":7,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,7],"acting":[5,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1e","version":"74'10","reported_seq":206,"reported_epoch":146,"state":"active+clean","last_fresh":"2024-03-11T21:31:12.739878+0000","last_change":"2024-03-11T21:30:57.371847+0000","last_active":"2024-03-11T21:31:12.739878+0000","last_peered":"2024-03-11T21:31:12.739878+0000","last_clean":"2024-03-11T21:31:12.739878+0000","last_became_active":"2024-03-11T21:25:40.763438+0000","last_became_peered":"2024-03-11T21:25:40.763438+0000","last_unstale":"2024-03-11T21:31:12.739878+0000","last_undegraded":"2024-03-11T21:31:12.739878+0000","last_fullsized":"2024-03-11T21:31:12.739878+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'10","last_scrub_stamp":"2024-03-11T21:30:57.371717+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:39.525554+0000","last_clean_scrub_stamp":"2024-03-11T21:30:57.371717+0000","objects_scrubbed":5,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:32:03.436916+0000","scrub_duration":27,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.13","version":"0'0","reported_seq":182,"reported_epoch":146,"state":"active+clean","last_fresh":"2024-03-11T21:31:12.753805+0000","last_change":"2024-03-11T21:31:00.239533+0000","last_active":"2024-03-11T21:31:12.753805+0000","last_peered":"2024-03-11T21:31:12.753805+0000","last_clean":"2024-03-11T21:31:12.753805+0000","last_became_active":"2024-03-11T21:25:45.302102+0000","last_became_peered":"2024-03-11T21:25:45.302102+0000","last_unstale":"2024-03-11T21:31:12.753805+0000","last_undegraded":"2024-03-11T21:31:12.753805+0000","last_fullsized":"2024-03-11T21:31:12.753805+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:31:00.238325+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:43.559452+0000","last_clean_scrub_stamp":"2024-03-11T21:31:00.238325+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:32:15.077313+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":186,"reported_epoch":146,"state":"active+clean","last_fresh":"2024-03-11T21:31:12.741753+0000","last_change":"2024-03-11T21:29:47.630346+0000","last_active":"2024-03-11T21:31:12.741753+0000","last_peered":"2024-03-11T21:31:12.741753+0000","last_clean":"2024-03-11T21:31:12.741753+0000","last_became_active":"2024-03-11T21:25:42.579781+0000","last_became_peered":"2024-03-11T21:25:42.579781+0000","last_unstale":"2024-03-11T21:31:12.741753+0000","last_undegraded":"2024-03-11T21:31:12.741753+0000","last_fullsized":"2024-03-11T21:31:12.741753+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:29:47.630271+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:41.544550+0000","last_clean_scrub_stamp":"2024-03-11T21:29:47.630271+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.17","version":"0'0","reported_seq":195,"reported_epoch":146,"state":"active+clean","last_fresh":"2024-03-11T21:31:12.744609+0000","last_change":"2024-03-11T21:30:12.782541+0000","last_active":"2024-03-11T21:31:12.744609+0000","last_peered":"2024-03-11T21:31:12.744609+0000","last_clean":"2024-03-11T21:31:12.744609+0000","last_became_active":"2024-03-11T21:25:38.540349+0000","last_became_peered":"2024-03-11T21:25:38.540349+0000","last_unstale":"2024-03-11T21:31:12.744609+0000","last_undegraded":"2024-03-11T21:31:12.744609+0000","last_fullsized":"2024-03-11T21:31:12.744609+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:30:12.782413+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:28:25.918101+0000","last_clean_scrub_stamp":"2024-03-11T21:30:12.782413+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:31:35.677574+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,2],"acting":[6,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.11","version":"74'11","reported_seq":209,"reported_epoch":146,"state":"active+clean","last_fresh":"2024-03-11T21:31:17.450204+0000","last_change":"2024-03-11T21:31:17.450254+0000","last_active":"2024-03-11T21:31:17.450204+0000","last_peered":"2024-03-11T21:31:17.450204+0000","last_clean":"2024-03-11T21:31:17.450204+0000","last_became_active":"2024-03-11T21:25:40.760664+0000","last_became_peered":"2024-03-11T21:25:40.760664+0000","last_unstale":"2024-03-11T21:31:17.450204+0000","last_undegraded":"2024-03-11T21:31:17.450204+0000","last_fullsized":"2024-03-11T21:31:17.450204+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'11","last_scrub_stamp":"2024-03-11T21:31:17.450051+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:39.525554+0000","last_clean_scrub_stamp":"2024-03-11T21:31:17.450051+0000","objects_scrubbed":5,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:32:37.763712+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":182,"reported_epoch":146,"state":"active+clean","last_fresh":"2024-03-11T21:31:12.741519+0000","last_change":"2024-03-11T21:29:41.454129+0000","last_active":"2024-03-11T21:31:12.741519+0000","last_peered":"2024-03-11T21:31:12.741519+0000","last_clean":"2024-03-11T21:31:12.741519+0000","last_became_active":"2024-03-11T21:25:44.591068+0000","last_became_peered":"2024-03-11T21:25:44.591068+0000","last_unstale":"2024-03-11T21:31:12.741519+0000","last_undegraded":"2024-03-11T21:31:12.741519+0000","last_fullsized":"2024-03-11T21:31:12.741519+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:29:41.454046+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:43.559452+0000","last_clean_scrub_stamp":"2024-03-11T21:29:41.454046+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,4],"acting":[7,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":195,"reported_epoch":146,"state":"active+clean","last_fresh":"2024-03-11T21:31:12.745748+0000","last_change":"2024-03-11T21:31:12.423773+0000","last_active":"2024-03-11T21:31:12.745748+0000","last_peered":"2024-03-11T21:31:12.745748+0000","last_clean":"2024-03-11T21:31:12.745748+0000","last_became_active":"2024-03-11T21:25:42.584148+0000","last_became_peered":"2024-03-11T21:25:42.584148+0000","last_unstale":"2024-03-11T21:31:12.745748+0000","last_undegraded":"2024-03-11T21:31:12.745748+0000","last_fullsized":"2024-03-11T21:31:12.745748+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:31:12.423671+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:29:39.923895+0000","last_clean_scrub_stamp":"2024-03-11T21:31:12.423671+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:32:18.156840+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.16","version":"0'0","reported_seq":202,"reported_epoch":146,"state":"active+clean","last_fresh":"2024-03-11T21:31:12.755995+0000","last_change":"2024-03-11T21:30:59.528390+0000","last_active":"2024-03-11T21:31:12.755995+0000","last_peered":"2024-03-11T21:31:12.755995+0000","last_clean":"2024-03-11T21:31:12.755995+0000","last_became_active":"2024-03-11T21:25:38.539103+0000","last_became_peered":"2024-03-11T21:25:38.539103+0000","last_unstale":"2024-03-11T21:31:12.755995+0000","last_undegraded":"2024-03-11T21:31:12.755995+0000","last_fullsized":"2024-03-11T21:31:12.755995+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:30:59.528265+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:27:11.938909+0000","last_clean_scrub_stamp":"2024-03-11T21:30:59.528265+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:32:24.866196+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,2],"acting":[5,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.10","version":"94'6","reported_seq":198,"reported_epoch":146,"state":"active+clean","last_fresh":"2024-03-11T21:31:12.752604+0000","last_change":"2024-03-11T21:30:21.932499+0000","last_active":"2024-03-11T21:31:12.752604+0000","last_peered":"2024-03-11T21:31:12.752604+0000","last_clean":"2024-03-11T21:31:12.752604+0000","last_became_active":"2024-03-11T21:25:40.552720+0000","last_became_peered":"2024-03-11T21:25:40.552720+0000","last_unstale":"2024-03-11T21:31:12.752604+0000","last_undegraded":"2024-03-11T21:31:12.752604+0000","last_fullsized":"2024-03-11T21:31:12.752604+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"94'6","last_scrub_stamp":"2024-03-11T21:30:21.932340+0000","last_deep_scrub":"94'6","last_deep_scrub_stamp":"2024-03-11T21:28:24.603974+0000","last_clean_scrub_stamp":"2024-03-11T21:30:21.932340+0000","objects_scrubbed":2,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:31:48.924443+0000","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":6,"num_read_kb":4,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6],"acting":[3,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":182,"reported_epoch":146,"state":"active+clean","last_fresh":"2024-03-11T21:31:14.338844+0000","last_change":"2024-03-11T21:31:14.338908+0000","last_active":"2024-03-11T21:31:14.338844+0000","last_peered":"2024-03-11T21:31:14.338844+0000","last_clean":"2024-03-11T21:31:14.338844+0000","last_became_active":"2024-03-11T21:25:44.583181+0000","last_became_peered":"2024-03-11T21:25:44.583181+0000","last_unstale":"2024-03-11T21:31:14.338844+0000","last_undegraded":"2024-03-11T21:31:14.338844+0000","last_fullsized":"2024-03-11T21:31:14.338844+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:31:14.338640+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:43.559452+0000","last_clean_scrub_stamp":"2024-03-11T21:31:14.338640+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:32:30.708648+0000","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":187,"reported_epoch":146,"state":"active+clean","last_fresh":"2024-03-11T21:31:12.760984+0000","last_change":"2024-03-11T21:30:19.680660+0000","last_active":"2024-03-11T21:31:12.760984+0000","last_peered":"2024-03-11T21:31:12.760984+0000","last_clean":"2024-03-11T21:31:12.760984+0000","last_became_active":"2024-03-11T21:25:42.573158+0000","last_became_peered":"2024-03-11T21:25:42.573158+0000","last_unstale":"2024-03-11T21:31:12.760984+0000","last_undegraded":"2024-03-11T21:31:12.760984+0000","last_fullsized":"2024-03-11T21:31:12.760984+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:30:19.680539+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:27:21.207303+0000","last_clean_scrub_stamp":"2024-03-11T21:30:19.680539+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:31:24.945950+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.15","version":"0'0","reported_seq":195,"reported_epoch":146,"state":"active+clean","last_fresh":"2024-03-11T21:31:12.760118+0000","last_change":"2024-03-11T21:29:43.022451+0000","last_active":"2024-03-11T21:31:12.760118+0000","last_peered":"2024-03-11T21:31:12.760118+0000","last_clean":"2024-03-11T21:31:12.760118+0000","last_became_active":"2024-03-11T21:25:38.541705+0000","last_became_peered":"2024-03-11T21:25:38.541705+0000","last_unstale":"2024-03-11T21:31:12.760118+0000","last_undegraded":"2024-03-11T21:31:12.760118+0000","last_fullsized":"2024-03-11T21:31:12.760118+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:29:43.022321+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:26:44.860499+0000","last_clean_scrub_stamp":"2024-03-11T21:29:43.022321+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.13","version":"109'16","reported_seq":233,"reported_epoch":146,"state":"active+clean","last_fresh":"2024-03-11T21:31:12.733236+0000","last_change":"2024-03-11T21:30:23.314515+0000","last_active":"2024-03-11T21:31:12.733236+0000","last_peered":"2024-03-11T21:31:12.733236+0000","last_clean":"2024-03-11T21:31:12.733236+0000","last_became_active":"2024-03-11T21:26:11.214879+0000","last_became_peered":"2024-03-11T21:26:11.214879+0000","last_unstale":"2024-03-11T21:31:12.733236+0000","last_undegraded":"2024-03-11T21:31:12.733236+0000","last_fullsized":"2024-03-11T21:31:12.733236+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"109'16","last_scrub_stamp":"2024-03-11T21:30:23.314410+0000","last_deep_scrub":"109'16","last_deep_scrub_stamp":"2024-03-11T21:28:32.854393+0000","last_clean_scrub_stamp":"2024-03-11T21:30:23.314410+0000","objects_scrubbed":5,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:31:50.913510+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,2],"acting":[4,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.10","version":"0'0","reported_seq":175,"reported_epoch":146,"state":"active+clean","last_fresh":"2024-03-11T21:31:12.736749+0000","last_change":"2024-03-11T21:29:41.528967+0000","last_active":"2024-03-11T21:31:12.736749+0000","last_peered":"2024-03-11T21:31:12.736749+0000","last_clean":"2024-03-11T21:31:12.736749+0000","last_became_active":"2024-03-11T21:25:44.591189+0000","last_became_peered":"2024-03-11T21:25:44.591189+0000","last_unstale":"2024-03-11T21:31:12.736749+0000","last_undegraded":"2024-03-11T21:31:12.736749+0000","last_fullsized":"2024-03-11T21:31:12.736749+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:29:41.528760+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:27:25.417450+0000","last_clean_scrub_stamp":"2024-03-11T21:29:41.528760+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,1],"acting":[0,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":186,"reported_epoch":146,"state":"active+clean","last_fresh":"2024-03-11T21:31:15.368203+0000","last_change":"2024-03-11T21:31:15.368570+0000","last_active":"2024-03-11T21:31:15.368203+0000","last_peered":"2024-03-11T21:31:15.368203+0000","last_clean":"2024-03-11T21:31:15.368203+0000","last_became_active":"2024-03-11T21:25:42.576425+0000","last_became_peered":"2024-03-11T21:25:42.576425+0000","last_unstale":"2024-03-11T21:31:15.368203+0000","last_undegraded":"2024-03-11T21:31:15.368203+0000","last_fullsized":"2024-03-11T21:31:15.368203+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:31:15.367898+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:31:15.367898+0000","last_clean_scrub_stamp":"2024-03-11T21:31:15.367898+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:32:26.039743+0000","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.14","version":"0'0","reported_seq":203,"reported_epoch":146,"state":"active+clean","last_fresh":"2024-03-11T21:31:12.744444+0000","last_change":"2024-03-11T21:31:09.407017+0000","last_active":"2024-03-11T21:31:12.744444+0000","last_peered":"2024-03-11T21:31:12.744444+0000","last_clean":"2024-03-11T21:31:12.744444+0000","last_became_active":"2024-03-11T21:25:38.541852+0000","last_became_peered":"2024-03-11T21:25:38.541852+0000","last_unstale":"2024-03-11T21:31:12.744444+0000","last_undegraded":"2024-03-11T21:31:12.744444+0000","last_fullsized":"2024-03-11T21:31:12.744444+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:31:09.406884+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:37.512018+0000","last_clean_scrub_stamp":"2024-03-11T21:31:09.406884+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:32:09.476644+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,5],"acting":[6,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.12","version":"74'9","reported_seq":207,"reported_epoch":146,"state":"active+clean","last_fresh":"2024-03-11T21:31:12.760882+0000","last_change":"2024-03-11T21:29:45.013132+0000","last_active":"2024-03-11T21:31:12.760882+0000","last_peered":"2024-03-11T21:31:12.760882+0000","last_clean":"2024-03-11T21:31:12.760882+0000","last_became_active":"2024-03-11T21:25:40.547082+0000","last_became_peered":"2024-03-11T21:25:40.547082+0000","last_unstale":"2024-03-11T21:31:12.760882+0000","last_undegraded":"2024-03-11T21:31:12.760882+0000","last_fullsized":"2024-03-11T21:31:12.760882+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'9","last_scrub_stamp":"2024-03-11T21:29:45.012981+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:39.525554+0000","last_clean_scrub_stamp":"2024-03-11T21:29:45.012981+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":175,"reported_epoch":146,"state":"active+clean","last_fresh":"2024-03-11T21:31:12.734842+0000","last_change":"2024-03-11T21:29:23.214125+0000","last_active":"2024-03-11T21:31:12.734842+0000","last_peered":"2024-03-11T21:31:12.734842+0000","last_clean":"2024-03-11T21:31:12.734842+0000","last_became_active":"2024-03-11T21:25:44.589586+0000","last_became_peered":"2024-03-11T21:25:44.589586+0000","last_unstale":"2024-03-11T21:31:12.734842+0000","last_undegraded":"2024-03-11T21:31:12.734842+0000","last_fullsized":"2024-03-11T21:31:12.734842+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:29:23.214015+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:27:09.801138+0000","last_clean_scrub_stamp":"2024-03-11T21:29:23.214015+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,5],"acting":[4,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.14","version":"74'8","reported_seq":254,"reported_epoch":146,"state":"active+clean","last_fresh":"2024-03-11T21:31:12.751709+0000","last_change":"2024-03-11T21:29:34.599277+0000","last_active":"2024-03-11T21:31:12.751709+0000","last_peered":"2024-03-11T21:31:12.751709+0000","last_clean":"2024-03-11T21:31:12.751709+0000","last_became_active":"2024-03-11T21:25:42.583266+0000","last_became_peered":"2024-03-11T21:25:42.583266+0000","last_unstale":"2024-03-11T21:31:12.751709+0000","last_undegraded":"2024-03-11T21:31:12.751709+0000","last_fullsized":"2024-03-11T21:31:12.751709+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'8","last_scrub_stamp":"2024-03-11T21:29:34.599162+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:41.544550+0000","last_clean_scrub_stamp":"2024-03-11T21:29:34.599162+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,2],"acting":[3,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.13","version":"0'0","reported_seq":187,"reported_epoch":146,"state":"active+clean","last_fresh":"2024-03-11T21:31:12.737161+0000","last_change":"2024-03-11T21:29:16.449075+0000","last_active":"2024-03-11T21:31:12.737161+0000","last_peered":"2024-03-11T21:31:12.737161+0000","last_clean":"2024-03-11T21:31:12.737161+0000","last_became_active":"2024-03-11T21:25:38.539467+0000","last_became_peered":"2024-03-11T21:25:38.539467+0000","last_unstale":"2024-03-11T21:31:12.737161+0000","last_undegraded":"2024-03-11T21:31:12.737161+0000","last_fullsized":"2024-03-11T21:31:12.737161+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:29:16.448818+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:37.512018+0000","last_clean_scrub_stamp":"2024-03-11T21:29:16.448818+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.15","version":"94'13","reported_seq":210,"reported_epoch":146,"state":"active+clean","last_fresh":"2024-03-11T21:31:12.756168+0000","last_change":"2024-03-11T21:29:41.853277+0000","last_active":"2024-03-11T21:31:12.756168+0000","last_peered":"2024-03-11T21:31:12.756168+0000","last_clean":"2024-03-11T21:31:12.756168+0000","last_became_active":"2024-03-11T21:25:40.760907+0000","last_became_peered":"2024-03-11T21:25:40.760907+0000","last_unstale":"2024-03-11T21:31:12.756168+0000","last_undegraded":"2024-03-11T21:31:12.756168+0000","last_fullsized":"2024-03-11T21:31:12.756168+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"94'13","last_scrub_stamp":"2024-03-11T21:29:41.853158+0000","last_deep_scrub":"94'13","last_deep_scrub_stamp":"2024-03-11T21:29:41.853158+0000","last_clean_scrub_stamp":"2024-03-11T21:29:41.853158+0000","objects_scrubbed":4,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.16","version":"0'0","reported_seq":183,"reported_epoch":146,"state":"active+clean","last_fresh":"2024-03-11T21:31:12.740104+0000","last_change":"2024-03-11T21:31:08.360701+0000","last_active":"2024-03-11T21:31:12.740104+0000","last_peered":"2024-03-11T21:31:12.740104+0000","last_clean":"2024-03-11T21:31:12.740104+0000","last_became_active":"2024-03-11T21:25:44.583393+0000","last_became_peered":"2024-03-11T21:25:44.583393+0000","last_unstale":"2024-03-11T21:31:12.740104+0000","last_undegraded":"2024-03-11T21:31:12.740104+0000","last_fullsized":"2024-03-11T21:31:12.740104+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:31:08.360582+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:43.559452+0000","last_clean_scrub_stamp":"2024-03-11T21:31:08.360582+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:32:18.732280+0000","scrub_duration":17,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.15","version":"98'9","reported_seq":255,"reported_epoch":146,"state":"active+clean","last_fresh":"2024-03-11T21:31:12.756177+0000","last_change":"2024-03-11T21:29:27.875893+0000","last_active":"2024-03-11T21:31:12.756177+0000","last_peered":"2024-03-11T21:31:12.756177+0000","last_clean":"2024-03-11T21:31:12.756177+0000","last_became_active":"2024-03-11T21:25:42.572555+0000","last_became_peered":"2024-03-11T21:25:42.572555+0000","last_unstale":"2024-03-11T21:31:12.756177+0000","last_undegraded":"2024-03-11T21:31:12.756177+0000","last_fullsized":"2024-03-11T21:31:12.756177+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"98'9","last_scrub_stamp":"2024-03-11T21:29:27.875760+0000","last_deep_scrub":"98'9","last_deep_scrub_stamp":"2024-03-11T21:27:34.948855+0000","last_clean_scrub_stamp":"2024-03-11T21:29:27.875760+0000","objects_scrubbed":1,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":18,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.12","version":"0'0","reported_seq":194,"reported_epoch":146,"state":"active+clean","last_fresh":"2024-03-11T21:31:12.755642+0000","last_change":"2024-03-11T21:30:13.021786+0000","last_active":"2024-03-11T21:31:12.755642+0000","last_peered":"2024-03-11T21:31:12.755642+0000","last_clean":"2024-03-11T21:31:12.755642+0000","last_became_active":"2024-03-11T21:25:38.543962+0000","last_became_peered":"2024-03-11T21:25:38.543962+0000","last_unstale":"2024-03-11T21:31:12.755642+0000","last_undegraded":"2024-03-11T21:31:12.755642+0000","last_fullsized":"2024-03-11T21:31:12.755642+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:30:13.021666+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:37.512018+0000","last_clean_scrub_stamp":"2024-03-11T21:30:13.021666+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:31:24.623576+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.14","version":"74'10","reported_seq":197,"reported_epoch":146,"state":"active+clean","last_fresh":"2024-03-11T21:31:12.750834+0000","last_change":"2024-03-11T21:29:27.547718+0000","last_active":"2024-03-11T21:31:12.750834+0000","last_peered":"2024-03-11T21:31:12.750834+0000","last_clean":"2024-03-11T21:31:12.750834+0000","last_became_active":"2024-03-11T21:25:40.760620+0000","last_became_peered":"2024-03-11T21:25:40.760620+0000","last_unstale":"2024-03-11T21:31:12.750834+0000","last_undegraded":"2024-03-11T21:31:12.750834+0000","last_fullsized":"2024-03-11T21:31:12.750834+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'10","last_scrub_stamp":"2024-03-11T21:29:27.547582+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:39.525554+0000","last_clean_scrub_stamp":"2024-03-11T21:29:27.547582+0000","objects_scrubbed":5,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.15","version":"0'0","reported_seq":182,"reported_epoch":146,"state":"active+clean","last_fresh":"2024-03-11T21:31:12.741824+0000","last_change":"2024-03-11T21:30:17.607310+0000","last_active":"2024-03-11T21:31:12.741824+0000","last_peered":"2024-03-11T21:31:12.741824+0000","last_clean":"2024-03-11T21:31:12.741824+0000","last_became_active":"2024-03-11T21:25:45.302501+0000","last_became_peered":"2024-03-11T21:25:45.302501+0000","last_unstale":"2024-03-11T21:31:12.741824+0000","last_undegraded":"2024-03-11T21:31:12.741824+0000","last_fullsized":"2024-03-11T21:31:12.741824+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:30:17.607199+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:43.559452+0000","last_clean_scrub_stamp":"2024-03-11T21:30:17.607199+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:31:28.961236+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":186,"reported_epoch":146,"state":"active+clean","last_fresh":"2024-03-11T21:31:14.608033+0000","last_change":"2024-03-11T21:31:14.608055+0000","last_active":"2024-03-11T21:31:14.608033+0000","last_peered":"2024-03-11T21:31:14.608033+0000","last_clean":"2024-03-11T21:31:14.608033+0000","last_became_active":"2024-03-11T21:25:42.577351+0000","last_became_peered":"2024-03-11T21:25:42.577351+0000","last_unstale":"2024-03-11T21:31:14.608033+0000","last_undegraded":"2024-03-11T21:31:14.608033+0000","last_fullsized":"2024-03-11T21:31:14.608033+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:31:14.607971+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:41.544550+0000","last_clean_scrub_stamp":"2024-03-11T21:31:14.607971+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:32:26.368031+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,1],"acting":[5,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.11","version":"0'0","reported_seq":195,"reported_epoch":146,"state":"active+clean","last_fresh":"2024-03-11T21:31:12.745771+0000","last_change":"2024-03-11T21:30:14.818836+0000","last_active":"2024-03-11T21:31:12.745771+0000","last_peered":"2024-03-11T21:31:12.745771+0000","last_clean":"2024-03-11T21:31:12.745771+0000","last_became_active":"2024-03-11T21:25:38.537415+0000","last_became_peered":"2024-03-11T21:25:38.537415+0000","last_unstale":"2024-03-11T21:31:12.745771+0000","last_undegraded":"2024-03-11T21:31:12.745771+0000","last_fullsized":"2024-03-11T21:31:12.745771+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:30:14.818712+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:37.512018+0000","last_clean_scrub_stamp":"2024-03-11T21:30:14.818712+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:31:40.512363+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.17","version":"97'9","reported_seq":208,"reported_epoch":146,"state":"active+clean","last_fresh":"2024-03-11T21:31:12.756380+0000","last_change":"2024-03-11T21:29:26.522324+0000","last_active":"2024-03-11T21:31:12.756380+0000","last_peered":"2024-03-11T21:31:12.756380+0000","last_clean":"2024-03-11T21:31:12.756380+0000","last_became_active":"2024-03-11T21:26:11.219020+0000","last_became_peered":"2024-03-11T21:26:11.219020+0000","last_unstale":"2024-03-11T21:31:12.756380+0000","last_undegraded":"2024-03-11T21:31:12.756380+0000","last_fullsized":"2024-03-11T21:31:12.756380+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"97'9","last_scrub_stamp":"2024-03-11T21:29:26.522238+0000","last_deep_scrub":"97'9","last_deep_scrub_stamp":"2024-03-11T21:27:25.803132+0000","last_clean_scrub_stamp":"2024-03-11T21:29:26.522238+0000","objects_scrubbed":3,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":17,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":9,"num_read_kb":6,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,5],"acting":[3,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.14","version":"0'0","reported_seq":182,"reported_epoch":146,"state":"active+clean","last_fresh":"2024-03-11T21:31:15.409720+0000","last_change":"2024-03-11T21:31:15.409758+0000","last_active":"2024-03-11T21:31:15.409720+0000","last_peered":"2024-03-11T21:31:15.409720+0000","last_clean":"2024-03-11T21:31:15.409720+0000","last_became_active":"2024-03-11T21:25:44.587439+0000","last_became_peered":"2024-03-11T21:25:44.587439+0000","last_unstale":"2024-03-11T21:31:15.409720+0000","last_undegraded":"2024-03-11T21:31:15.409720+0000","last_fullsized":"2024-03-11T21:31:15.409720+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:31:15.409641+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:43.559452+0000","last_clean_scrub_stamp":"2024-03-11T21:31:15.409641+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:32:24.347649+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,7],"acting":[2,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":186,"reported_epoch":146,"state":"active+clean","last_fresh":"2024-03-11T21:31:12.756481+0000","last_change":"2024-03-11T21:31:08.284623+0000","last_active":"2024-03-11T21:31:12.756481+0000","last_peered":"2024-03-11T21:31:12.756481+0000","last_clean":"2024-03-11T21:31:12.756481+0000","last_became_active":"2024-03-11T21:25:42.583514+0000","last_became_peered":"2024-03-11T21:25:42.583514+0000","last_unstale":"2024-03-11T21:31:12.756481+0000","last_undegraded":"2024-03-11T21:31:12.756481+0000","last_fullsized":"2024-03-11T21:31:12.756481+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:31:08.284511+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:41.544550+0000","last_clean_scrub_stamp":"2024-03-11T21:31:08.284511+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:32:27.662416+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.10","version":"0'0","reported_seq":194,"reported_epoch":146,"state":"active+clean","last_fresh":"2024-03-11T21:31:12.741435+0000","last_change":"2024-03-11T21:31:03.260526+0000","last_active":"2024-03-11T21:31:12.741435+0000","last_peered":"2024-03-11T21:31:12.741435+0000","last_clean":"2024-03-11T21:31:12.741435+0000","last_became_active":"2024-03-11T21:25:38.533138+0000","last_became_peered":"2024-03-11T21:25:38.533138+0000","last_unstale":"2024-03-11T21:31:12.741435+0000","last_undegraded":"2024-03-11T21:31:12.741435+0000","last_fullsized":"2024-03-11T21:31:12.741435+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:31:03.260408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:37.512018+0000","last_clean_scrub_stamp":"2024-03-11T21:31:03.260408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:32:29.743555+0000","scrub_duration":29,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,0],"acting":[2,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.16","version":"74'9","reported_seq":207,"reported_epoch":146,"state":"active+clean","last_fresh":"2024-03-11T21:31:12.739630+0000","last_change":"2024-03-11T21:30:13.857912+0000","last_active":"2024-03-11T21:31:12.739630+0000","last_peered":"2024-03-11T21:31:12.739630+0000","last_clean":"2024-03-11T21:31:12.739630+0000","last_became_active":"2024-03-11T21:25:40.763434+0000","last_became_peered":"2024-03-11T21:25:40.763434+0000","last_unstale":"2024-03-11T21:31:12.739630+0000","last_undegraded":"2024-03-11T21:31:12.739630+0000","last_fullsized":"2024-03-11T21:31:12.739630+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'9","last_scrub_stamp":"2024-03-11T21:30:13.857760+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:39.525554+0000","last_clean_scrub_stamp":"2024-03-11T21:30:13.857760+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:31:22.046967+0000","scrub_duration":40,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.a","version":"146'14779","reported_seq":14982,"reported_epoch":146,"state":"active+clean","last_fresh":"2024-03-11T21:31:17.457876+0000","last_change":"2024-03-11T21:30:57.938479+0000","last_active":"2024-03-11T21:31:17.457876+0000","last_peered":"2024-03-11T21:31:17.457876+0000","last_clean":"2024-03-11T21:31:17.457876+0000","last_became_active":"2024-03-11T21:27:36.692988+0000","last_became_peered":"2024-03-11T21:27:36.692988+0000","last_unstale":"2024-03-11T21:31:17.457876+0000","last_undegraded":"2024-03-11T21:31:17.457876+0000","last_fullsized":"2024-03-11T21:31:17.457876+0000","mapping_epoch":99,"log_start":"128'10100","ondisk_log_start":"128'10100","created":99,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"143'13593","last_scrub_stamp":"2024-03-11T21:30:57.938271+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:27:35.667522+0000","last_clean_scrub_stamp":"2024-03-11T21:30:57.938271+0000","objects_scrubbed":1614,"log_size":4679,"log_dups_size":0,"ondisk_log_size":4679,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:32:07.370996+0000","scrub_duration":767,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26279936,"num_objects":401,"num_object_clones":0,"num_object_copies":1203,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":401,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14779,"num_write_kb":485760,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5],"acting":[3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.b","version":"0'0","reported_seq":174,"reported_epoch":146,"state":"active+clean","last_fresh":"2024-03-11T21:31:12.752101+0000","last_change":"2024-03-11T21:30:19.008385+0000","last_active":"2024-03-11T21:31:12.752101+0000","last_peered":"2024-03-11T21:31:12.752101+0000","last_clean":"2024-03-11T21:31:12.752101+0000","last_became_active":"2024-03-11T21:25:44.590071+0000","last_became_peered":"2024-03-11T21:25:44.590071+0000","last_unstale":"2024-03-11T21:31:12.752101+0000","last_undegraded":"2024-03-11T21:31:12.752101+0000","last_fullsized":"2024-03-11T21:31:12.752101+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:30:19.006959+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:43.559452+0000","last_clean_scrub_stamp":"2024-03-11T21:30:19.006959+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:31:25.655303+0000","scrub_duration":70,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.f","version":"67'2","reported_seq":207,"reported_epoch":146,"state":"active+clean","last_fresh":"2024-03-11T21:31:12.734253+0000","last_change":"2024-03-11T21:31:03.533706+0000","last_active":"2024-03-11T21:31:12.734253+0000","last_peered":"2024-03-11T21:31:12.734253+0000","last_clean":"2024-03-11T21:31:12.734253+0000","last_became_active":"2024-03-11T21:25:38.537711+0000","last_became_peered":"2024-03-11T21:25:38.537711+0000","last_unstale":"2024-03-11T21:31:12.734253+0000","last_undegraded":"2024-03-11T21:31:12.734253+0000","last_fullsized":"2024-03-11T21:31:12.734253+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'2","last_scrub_stamp":"2024-03-11T21:31:03.533547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:37.512018+0000","last_clean_scrub_stamp":"2024-03-11T21:31:03.533547+0000","objects_scrubbed":2,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:32:27.195878+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":92,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":10,"num_read_kb":10,"num_write":4,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,7],"acting":[4,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.9","version":"146'17","reported_seq":219,"reported_epoch":146,"state":"active+clean","last_fresh":"2024-03-11T21:31:16.615095+0000","last_change":"2024-03-11T21:31:16.615182+0000","last_active":"2024-03-11T21:31:16.615095+0000","last_peered":"2024-03-11T21:31:16.615095+0000","last_clean":"2024-03-11T21:31:16.615095+0000","last_became_active":"2024-03-11T21:25:40.552267+0000","last_became_peered":"2024-03-11T21:25:40.552267+0000","last_unstale":"2024-03-11T21:31:16.615095+0000","last_undegraded":"2024-03-11T21:31:16.615095+0000","last_fullsized":"2024-03-11T21:31:16.615095+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"146'17","last_scrub_stamp":"2024-03-11T21:31:16.615010+0000","last_deep_scrub":"146'17","last_deep_scrub_stamp":"2024-03-11T21:31:16.615010+0000","last_clean_scrub_stamp":"2024-03-11T21:31:16.615010+0000","objects_scrubbed":5,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:32:36.347947+0000","scrub_duration":27,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":25,"num_read_kb":16,"num_write":14,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,3],"acting":[4,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":186,"reported_epoch":146,"state":"active+clean","last_fresh":"2024-03-11T21:31:12.741392+0000","last_change":"2024-03-11T21:31:12.434593+0000","last_active":"2024-03-11T21:31:12.741392+0000","last_peered":"2024-03-11T21:31:12.741392+0000","last_clean":"2024-03-11T21:31:12.741392+0000","last_became_active":"2024-03-11T21:25:42.573987+0000","last_became_peered":"2024-03-11T21:25:42.573987+0000","last_unstale":"2024-03-11T21:31:12.741392+0000","last_undegraded":"2024-03-11T21:31:12.741392+0000","last_fullsized":"2024-03-11T21:31:12.741392+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:31:12.434479+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:41.544550+0000","last_clean_scrub_stamp":"2024-03-11T21:31:12.434479+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:32:19.616693+0000","scrub_duration":17,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.1","version":"146'15059","reported_seq":15260,"reported_epoch":146,"state":"active+clean","last_fresh":"2024-03-11T21:31:17.010140+0000","last_change":"2024-03-11T21:31:06.819421+0000","last_active":"2024-03-11T21:31:17.010140+0000","last_peered":"2024-03-11T21:31:17.010140+0000","last_clean":"2024-03-11T21:31:17.010140+0000","last_became_active":"2024-03-11T21:27:36.696990+0000","last_became_peered":"2024-03-11T21:27:36.696990+0000","last_unstale":"2024-03-11T21:31:17.010140+0000","last_undegraded":"2024-03-11T21:31:17.010140+0000","last_fullsized":"2024-03-11T21:31:17.010140+0000","mapping_epoch":99,"log_start":"126'9900","ondisk_log_start":"126'9900","created":99,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"144'14362","last_scrub_stamp":"2024-03-11T21:31:06.819152+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:27:35.667522+0000","last_clean_scrub_stamp":"2024-03-11T21:31:06.819152+0000","objects_scrubbed":1146,"log_size":5159,"log_dups_size":0,"ondisk_log_size":5159,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:32:18.841039+0000","scrub_duration":263,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":29032448,"num_objects":443,"num_object_clones":0,"num_object_copies":1329,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":443,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15059,"num_write_kb":496064,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.0","version":"0'0","reported_seq":175,"reported_epoch":146,"state":"active+clean","last_fresh":"2024-03-11T21:31:12.736864+0000","last_change":"2024-03-11T21:29:39.498904+0000","last_active":"2024-03-11T21:31:12.736864+0000","last_peered":"2024-03-11T21:31:12.736864+0000","last_clean":"2024-03-11T21:31:12.736864+0000","last_became_active":"2024-03-11T21:25:44.584565+0000","last_became_peered":"2024-03-11T21:25:44.584565+0000","last_unstale":"2024-03-11T21:31:12.736864+0000","last_undegraded":"2024-03-11T21:31:12.736864+0000","last_fullsized":"2024-03-11T21:31:12.736864+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:29:39.498784+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:29:39.498784+0000","last_clean_scrub_stamp":"2024-03-11T21:29:39.498784+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":195,"reported_epoch":146,"state":"active+clean","last_fresh":"2024-03-11T21:31:12.757917+0000","last_change":"2024-03-11T21:29:35.214647+0000","last_active":"2024-03-11T21:31:12.757917+0000","last_peered":"2024-03-11T21:31:12.757917+0000","last_clean":"2024-03-11T21:31:12.757917+0000","last_became_active":"2024-03-11T21:25:38.541760+0000","last_became_peered":"2024-03-11T21:25:38.541760+0000","last_unstale":"2024-03-11T21:31:12.757917+0000","last_undegraded":"2024-03-11T21:31:12.757917+0000","last_fullsized":"2024-03-11T21:31:12.757917+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:29:35.214430+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:37.512018+0000","last_clean_scrub_stamp":"2024-03-11T21:29:35.214430+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,7],"acting":[1,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.2","version":"94'15","reported_seq":219,"reported_epoch":146,"state":"active+clean","last_fresh":"2024-03-11T21:31:12.760887+0000","last_change":"2024-03-11T21:31:08.956264+0000","last_active":"2024-03-11T21:31:12.760887+0000","last_peered":"2024-03-11T21:31:12.760887+0000","last_clean":"2024-03-11T21:31:12.760887+0000","last_became_active":"2024-03-11T21:25:40.548833+0000","last_became_peered":"2024-03-11T21:25:40.548833+0000","last_unstale":"2024-03-11T21:31:12.760887+0000","last_undegraded":"2024-03-11T21:31:12.760887+0000","last_fullsized":"2024-03-11T21:31:12.760887+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"94'15","last_scrub_stamp":"2024-03-11T21:31:08.955971+0000","last_deep_scrub":"94'15","last_deep_scrub_stamp":"2024-03-11T21:26:49.993060+0000","last_clean_scrub_stamp":"2024-03-11T21:31:08.955971+0000","objects_scrubbed":5,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:32:37.727507+0000","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.3","version":"74'8","reported_seq":263,"reported_epoch":146,"state":"active+clean","last_fresh":"2024-03-11T21:31:15.465563+0000","last_change":"2024-03-11T21:31:15.465617+0000","last_active":"2024-03-11T21:31:15.465563+0000","last_peered":"2024-03-11T21:31:15.465563+0000","last_clean":"2024-03-11T21:31:15.465563+0000","last_became_active":"2024-03-11T21:25:42.578161+0000","last_became_peered":"2024-03-11T21:25:42.578161+0000","last_unstale":"2024-03-11T21:31:15.465563+0000","last_undegraded":"2024-03-11T21:31:15.465563+0000","last_fullsized":"2024-03-11T21:31:15.465563+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'8","last_scrub_stamp":"2024-03-11T21:31:15.465460+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:41.544550+0000","last_clean_scrub_stamp":"2024-03-11T21:31:15.465460+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:32:43.379445+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,5],"acting":[0,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.7","version":"146'14871","reported_seq":15083,"reported_epoch":146,"state":"active+clean","last_fresh":"2024-03-11T21:31:17.046895+0000","last_change":"2024-03-11T21:31:00.103228+0000","last_active":"2024-03-11T21:31:17.046895+0000","last_peered":"2024-03-11T21:31:17.046895+0000","last_clean":"2024-03-11T21:31:17.046895+0000","last_became_active":"2024-03-11T21:27:36.695055+0000","last_became_peered":"2024-03-11T21:27:36.695055+0000","last_unstale":"2024-03-11T21:31:17.046895+0000","last_undegraded":"2024-03-11T21:31:17.046895+0000","last_fullsized":"2024-03-11T21:31:17.046895+0000","mapping_epoch":99,"log_start":"124'8600","ondisk_log_start":"124'8600","created":99,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"144'13822","last_scrub_stamp":"2024-03-11T21:31:00.103084+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:27:35.667522+0000","last_clean_scrub_stamp":"2024-03-11T21:31:00.103084+0000","objects_scrubbed":1506,"log_size":6271,"log_dups_size":0,"ondisk_log_size":6271,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:32:17.242400+0000","scrub_duration":561,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":28639232,"num_objects":437,"num_object_clones":0,"num_object_copies":1311,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":437,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14871,"num_write_kb":489856,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,3],"acting":[6,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":182,"reported_epoch":146,"state":"active+clean","last_fresh":"2024-03-11T21:31:12.750886+0000","last_change":"2024-03-11T21:31:01.241438+0000","last_active":"2024-03-11T21:31:12.750886+0000","last_peered":"2024-03-11T21:31:12.750886+0000","last_clean":"2024-03-11T21:31:12.750886+0000","last_became_active":"2024-03-11T21:25:44.585223+0000","last_became_peered":"2024-03-11T21:25:44.585223+0000","last_unstale":"2024-03-11T21:31:12.750886+0000","last_undegraded":"2024-03-11T21:31:12.750886+0000","last_fullsized":"2024-03-11T21:31:12.750886+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:31:01.241180+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:28:30.563712+0000","last_clean_scrub_stamp":"2024-03-11T21:31:01.241180+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:32:22.284772+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,7],"acting":[3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":202,"reported_epoch":146,"state":"active+clean","last_fresh":"2024-03-11T21:31:16.608321+0000","last_change":"2024-03-11T21:31:16.608343+0000","last_active":"2024-03-11T21:31:16.608321+0000","last_peered":"2024-03-11T21:31:16.608321+0000","last_clean":"2024-03-11T21:31:16.608321+0000","last_became_active":"2024-03-11T21:25:38.538540+0000","last_became_peered":"2024-03-11T21:25:38.538540+0000","last_unstale":"2024-03-11T21:31:16.608321+0000","last_undegraded":"2024-03-11T21:31:16.608321+0000","last_fullsized":"2024-03-11T21:31:16.608321+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:31:16.608266+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:37.512018+0000","last_clean_scrub_stamp":"2024-03-11T21:31:16.608266+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:32:30.358883+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.4","version":"74'28","reported_seq":307,"reported_epoch":146,"state":"active+clean","last_fresh":"2024-03-11T21:31:12.757866+0000","last_change":"2024-03-11T21:30:24.750195+0000","last_active":"2024-03-11T21:31:12.757866+0000","last_peered":"2024-03-11T21:31:12.757866+0000","last_clean":"2024-03-11T21:31:12.757866+0000","last_became_active":"2024-03-11T21:25:40.552902+0000","last_became_peered":"2024-03-11T21:25:40.552902+0000","last_unstale":"2024-03-11T21:31:12.757866+0000","last_undegraded":"2024-03-11T21:31:12.757866+0000","last_fullsized":"2024-03-11T21:31:12.757866+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'28","last_scrub_stamp":"2024-03-11T21:30:24.750058+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:39.525554+0000","last_clean_scrub_stamp":"2024-03-11T21:30:24.750058+0000","objects_scrubbed":10,"log_size":28,"log_dups_size":0,"ondisk_log_size":28,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:31:36.994296+0000","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":358,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":48,"num_read_kb":33,"num_write":26,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3],"acting":[1,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":179,"reported_epoch":146,"state":"active+clean","last_fresh":"2024-03-11T21:31:12.738104+0000","last_change":"2024-03-11T21:29:38.388702+0000","last_active":"2024-03-11T21:31:12.738104+0000","last_peered":"2024-03-11T21:31:12.738104+0000","last_clean":"2024-03-11T21:31:12.738104+0000","last_became_active":"2024-03-11T21:25:42.572673+0000","last_became_peered":"2024-03-11T21:25:42.572673+0000","last_unstale":"2024-03-11T21:31:12.738104+0000","last_undegraded":"2024-03-11T21:31:12.738104+0000","last_fullsized":"2024-03-11T21:31:12.738104+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:29:38.388578+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:41.544550+0000","last_clean_scrub_stamp":"2024-03-11T21:29:38.388578+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.4","version":"146'14941","reported_seq":15117,"reported_epoch":146,"state":"active+clean","last_fresh":"2024-03-11T21:31:17.773497+0000","last_change":"2024-03-11T21:29:39.193418+0000","last_active":"2024-03-11T21:31:17.773497+0000","last_peered":"2024-03-11T21:31:17.773497+0000","last_clean":"2024-03-11T21:31:17.773497+0000","last_became_active":"2024-03-11T21:27:36.690751+0000","last_became_peered":"2024-03-11T21:27:36.690751+0000","last_unstale":"2024-03-11T21:31:17.773497+0000","last_undegraded":"2024-03-11T21:31:17.773497+0000","last_fullsized":"2024-03-11T21:31:17.773497+0000","mapping_epoch":99,"log_start":"124'9400","ondisk_log_start":"124'9400","created":99,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"124'9100","last_scrub_stamp":"2024-03-11T21:29:39.193151+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:27:35.667522+0000","last_clean_scrub_stamp":"2024-03-11T21:29:39.193151+0000","objects_scrubbed":6273,"log_size":5541,"log_dups_size":0,"ondisk_log_size":5541,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"queued for scrub","scrub_duration":1792,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":24444928,"num_objects":373,"num_object_clones":0,"num_object_copies":1119,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":373,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14941,"num_write_kb":490048,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,1],"acting":[0,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.5","version":"0'0","reported_seq":182,"reported_epoch":146,"state":"active+clean","last_fresh":"2024-03-11T21:31:12.742393+0000","last_change":"2024-03-11T21:30:55.708176+0000","last_active":"2024-03-11T21:31:12.742393+0000","last_peered":"2024-03-11T21:31:12.742393+0000","last_clean":"2024-03-11T21:31:12.742393+0000","last_became_active":"2024-03-11T21:25:45.302120+0000","last_became_peered":"2024-03-11T21:25:45.302120+0000","last_unstale":"2024-03-11T21:31:12.742393+0000","last_undegraded":"2024-03-11T21:31:12.742393+0000","last_fullsized":"2024-03-11T21:31:12.742393+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:30:55.708059+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:43.559452+0000","last_clean_scrub_stamp":"2024-03-11T21:30:55.708059+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:32:11.142437+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,3],"acting":[7,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.0","version":"76'5","reported_seq":537,"reported_epoch":146,"state":"active+clean","last_fresh":"2024-03-11T21:31:15.315462+0000","last_change":"2024-03-11T21:30:14.726351+0000","last_active":"2024-03-11T21:31:15.315462+0000","last_peered":"2024-03-11T21:31:15.315462+0000","last_clean":"2024-03-11T21:31:15.315462+0000","last_became_active":"2024-03-11T21:25:38.536890+0000","last_became_peered":"2024-03-11T21:25:38.536890+0000","last_unstale":"2024-03-11T21:31:15.315462+0000","last_undegraded":"2024-03-11T21:31:15.315462+0000","last_fullsized":"2024-03-11T21:31:15.315462+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"76'5","last_scrub_stamp":"2024-03-11T21:30:14.726214+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:37.512018+0000","last_clean_scrub_stamp":"2024-03-11T21:30:14.726214+0000","objects_scrubbed":1,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:31:18.797414+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0.000753559,"stat_sum":{"num_bytes":389,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":332,"num_read_kb":327,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,6],"acting":[1,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":194,"reported_epoch":146,"state":"active+clean","last_fresh":"2024-03-11T21:31:12.741139+0000","last_change":"2024-03-11T21:30:16.960087+0000","last_active":"2024-03-11T21:31:12.741139+0000","last_peered":"2024-03-11T21:31:12.741139+0000","last_clean":"2024-03-11T21:31:12.741139+0000","last_became_active":"2024-03-11T21:25:38.532984+0000","last_became_peered":"2024-03-11T21:25:38.532984+0000","last_unstale":"2024-03-11T21:31:12.741139+0000","last_undegraded":"2024-03-11T21:31:12.741139+0000","last_fullsized":"2024-03-11T21:31:12.741139+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:30:16.959969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:26:48.997454+0000","last_clean_scrub_stamp":"2024-03-11T21:30:16.959969+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:31:19.491250+0000","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.7","version":"97'18","reported_seq":226,"reported_epoch":146,"state":"active+clean","last_fresh":"2024-03-11T21:31:12.757451+0000","last_change":"2024-03-11T21:30:11.805557+0000","last_active":"2024-03-11T21:31:12.757451+0000","last_peered":"2024-03-11T21:31:12.757451+0000","last_clean":"2024-03-11T21:31:12.757451+0000","last_became_active":"2024-03-11T21:25:40.541769+0000","last_became_peered":"2024-03-11T21:25:40.541769+0000","last_unstale":"2024-03-11T21:31:12.757451+0000","last_undegraded":"2024-03-11T21:31:12.757451+0000","last_fullsized":"2024-03-11T21:31:12.757451+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"97'18","last_scrub_stamp":"2024-03-11T21:30:11.805394+0000","last_deep_scrub":"97'18","last_deep_scrub_stamp":"2024-03-11T21:27:19.239404+0000","last_clean_scrub_stamp":"2024-03-11T21:30:11.805394+0000","objects_scrubbed":5,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:31:16.981036+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":19,"num_write":16,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":186,"reported_epoch":146,"state":"active+clean","last_fresh":"2024-03-11T21:31:12.747336+0000","last_change":"2024-03-11T21:31:06.279911+0000","last_active":"2024-03-11T21:31:12.747336+0000","last_peered":"2024-03-11T21:31:12.747336+0000","last_clean":"2024-03-11T21:31:12.747336+0000","last_became_active":"2024-03-11T21:25:42.581710+0000","last_became_peered":"2024-03-11T21:25:42.581710+0000","last_unstale":"2024-03-11T21:31:12.747336+0000","last_undegraded":"2024-03-11T21:31:12.747336+0000","last_fullsized":"2024-03-11T21:31:12.747336+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:31:06.279751+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:41.544550+0000","last_clean_scrub_stamp":"2024-03-11T21:31:06.279751+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:32:31.343510+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7],"acting":[2,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.6","version":"146'14915","reported_seq":15112,"reported_epoch":146,"state":"active+clean","last_fresh":"2024-03-11T21:31:17.802639+0000","last_change":"2024-03-11T21:31:11.813944+0000","last_active":"2024-03-11T21:31:17.802639+0000","last_peered":"2024-03-11T21:31:17.802639+0000","last_clean":"2024-03-11T21:31:17.802639+0000","last_became_active":"2024-03-11T21:27:36.695018+0000","last_became_peered":"2024-03-11T21:27:36.695018+0000","last_unstale":"2024-03-11T21:31:17.802639+0000","last_undegraded":"2024-03-11T21:31:17.802639+0000","last_fullsized":"2024-03-11T21:31:17.802639+0000","mapping_epoch":99,"log_start":"124'9400","ondisk_log_start":"124'9400","created":99,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"145'14537","last_scrub_stamp":"2024-03-11T21:31:11.813806+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:27:35.667522+0000","last_clean_scrub_stamp":"2024-03-11T21:31:11.813806+0000","objects_scrubbed":758,"log_size":5515,"log_dups_size":0,"ondisk_log_size":5515,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:32:30.283716+0000","scrub_duration":408,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":24051712,"num_objects":367,"num_object_clones":0,"num_object_copies":1101,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":367,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14915,"num_write_kb":489024,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.7","version":"124'2","reported_seq":176,"reported_epoch":146,"state":"active+clean","last_fresh":"2024-03-11T21:31:12.756546+0000","last_change":"2024-03-11T21:29:34.957004+0000","last_active":"2024-03-11T21:31:12.756546+0000","last_peered":"2024-03-11T21:31:12.756546+0000","last_clean":"2024-03-11T21:31:12.756546+0000","last_became_active":"2024-03-11T21:25:44.591376+0000","last_became_peered":"2024-03-11T21:25:44.591376+0000","last_unstale":"2024-03-11T21:31:12.756546+0000","last_undegraded":"2024-03-11T21:31:12.756546+0000","last_fullsized":"2024-03-11T21:31:12.756546+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"124'2","last_scrub_stamp":"2024-03-11T21:29:34.956908+0000","last_deep_scrub":"124'2","last_deep_scrub_stamp":"2024-03-11T21:29:34.956908+0000","last_clean_scrub_stamp":"2024-03-11T21:29:34.956908+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":68,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,4],"acting":[5,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2","version":"71'2","reported_seq":203,"reported_epoch":146,"state":"active+clean","last_fresh":"2024-03-11T21:31:12.761383+0000","last_change":"2024-03-11T21:30:19.973353+0000","last_active":"2024-03-11T21:31:12.761383+0000","last_peered":"2024-03-11T21:31:12.761383+0000","last_clean":"2024-03-11T21:31:12.761383+0000","last_became_active":"2024-03-11T21:25:38.539413+0000","last_became_peered":"2024-03-11T21:25:38.539413+0000","last_unstale":"2024-03-11T21:31:12.761383+0000","last_undegraded":"2024-03-11T21:31:12.761383+0000","last_fullsized":"2024-03-11T21:31:12.761383+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"71'2","last_scrub_stamp":"2024-03-11T21:30:19.973188+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:37.512018+0000","last_clean_scrub_stamp":"2024-03-11T21:30:19.973188+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:31:40.539347+0000","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0.0014206449999999999,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":200,"reported_epoch":146,"state":"active+clean","last_fresh":"2024-03-11T21:31:12.755945+0000","last_change":"2024-03-11T21:29:30.862499+0000","last_active":"2024-03-11T21:31:12.755945+0000","last_peered":"2024-03-11T21:31:12.755945+0000","last_clean":"2024-03-11T21:31:12.755945+0000","last_became_active":"2024-03-11T21:26:11.216188+0000","last_became_peered":"2024-03-11T21:26:11.216188+0000","last_unstale":"2024-03-11T21:31:12.755945+0000","last_undegraded":"2024-03-11T21:31:12.755945+0000","last_fullsized":"2024-03-11T21:31:12.755945+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:29:30.862374+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:37.512018+0000","last_clean_scrub_stamp":"2024-03-11T21:29:30.862374+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,4],"acting":[5,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"87'128","reported_seq":448,"reported_epoch":146,"state":"active+clean","last_fresh":"2024-03-11T21:31:12.741509+0000","last_change":"2024-03-11T21:30:22.564543+0000","last_active":"2024-03-11T21:31:12.741509+0000","last_peered":"2024-03-11T21:31:12.741509+0000","last_clean":"2024-03-11T21:31:12.741509+0000","last_became_active":"2024-03-11T21:25:30.502597+0000","last_became_peered":"2024-03-11T21:25:30.502597+0000","last_unstale":"2024-03-11T21:31:12.741509+0000","last_undegraded":"2024-03-11T21:31:12.741509+0000","last_fullsized":"2024-03-11T21:31:12.741509+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"87'128","last_scrub_stamp":"2024-03-11T21:30:22.564368+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:24.261974+0000","last_clean_scrub_stamp":"2024-03-11T21:30:22.564368+0000","objects_scrubbed":2,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:31:48.461461+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":212,"num_read_kb":306,"num_write":179,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1771104,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.5","version":"74'16","reported_seq":281,"reported_epoch":146,"state":"active+clean","last_fresh":"2024-03-11T21:31:12.744787+0000","last_change":"2024-03-11T21:30:17.783268+0000","last_active":"2024-03-11T21:31:12.744787+0000","last_peered":"2024-03-11T21:31:12.744787+0000","last_clean":"2024-03-11T21:31:12.744787+0000","last_became_active":"2024-03-11T21:25:40.761457+0000","last_became_peered":"2024-03-11T21:25:40.761457+0000","last_unstale":"2024-03-11T21:31:12.744787+0000","last_undegraded":"2024-03-11T21:31:12.744787+0000","last_fullsized":"2024-03-11T21:31:12.744787+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'16","last_scrub_stamp":"2024-03-11T21:30:17.783007+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:39.525554+0000","last_clean_scrub_stamp":"2024-03-11T21:30:17.783007+0000","objects_scrubbed":8,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:31:40.049704+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":154,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":25,"num_read_kb":15,"num_write":13,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":194,"reported_epoch":146,"state":"active+clean","last_fresh":"2024-03-11T21:31:12.742204+0000","last_change":"2024-03-11T21:31:00.607444+0000","last_active":"2024-03-11T21:31:12.742204+0000","last_peered":"2024-03-11T21:31:12.742204+0000","last_clean":"2024-03-11T21:31:12.742204+0000","last_became_active":"2024-03-11T21:25:42.586226+0000","last_became_peered":"2024-03-11T21:25:42.586226+0000","last_unstale":"2024-03-11T21:31:12.742204+0000","last_undegraded":"2024-03-11T21:31:12.742204+0000","last_fullsized":"2024-03-11T21:31:12.742204+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:31:00.607366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:41.544550+0000","last_clean_scrub_stamp":"2024-03-11T21:31:00.607366+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:32:10.161500+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,5],"acting":[7,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.5","version":"146'14842","reported_seq":15049,"reported_epoch":146,"state":"active+clean","last_fresh":"2024-03-11T21:31:17.040876+0000","last_change":"2024-03-11T21:30:59.104725+0000","last_active":"2024-03-11T21:31:17.040876+0000","last_peered":"2024-03-11T21:31:17.040876+0000","last_clean":"2024-03-11T21:31:17.040876+0000","last_became_active":"2024-03-11T21:27:36.691428+0000","last_became_peered":"2024-03-11T21:27:36.691428+0000","last_unstale":"2024-03-11T21:31:17.040876+0000","last_undegraded":"2024-03-11T21:31:17.040876+0000","last_fullsized":"2024-03-11T21:31:17.040876+0000","mapping_epoch":99,"log_start":"127'9900","ondisk_log_start":"127'9900","created":99,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"144'13718","last_scrub_stamp":"2024-03-11T21:30:59.104513+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:27:35.667522+0000","last_clean_scrub_stamp":"2024-03-11T21:30:59.104513+0000","objects_scrubbed":1548,"log_size":4942,"log_dups_size":0,"ondisk_log_size":4942,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:32:24.841804+0000","scrub_duration":440,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":27131904,"num_objects":414,"num_object_clones":0,"num_object_copies":1242,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":414,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14842,"num_write_kb":488192,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,0],"acting":[7,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.4","version":"0'0","reported_seq":183,"reported_epoch":146,"state":"active+clean","last_fresh":"2024-03-11T21:31:12.758157+0000","last_change":"2024-03-11T21:30:58.758256+0000","last_active":"2024-03-11T21:31:12.758157+0000","last_peered":"2024-03-11T21:31:12.758157+0000","last_clean":"2024-03-11T21:31:12.758157+0000","last_became_active":"2024-03-11T21:25:44.594188+0000","last_became_peered":"2024-03-11T21:25:44.594188+0000","last_unstale":"2024-03-11T21:31:12.758157+0000","last_undegraded":"2024-03-11T21:31:12.758157+0000","last_fullsized":"2024-03-11T21:31:12.758157+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:30:58.757819+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:43.559452+0000","last_clean_scrub_stamp":"2024-03-11T21:30:58.757819+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:32:17.709798+0000","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":202,"reported_epoch":146,"state":"active+clean","last_fresh":"2024-03-11T21:31:12.737518+0000","last_change":"2024-03-11T21:30:22.969948+0000","last_active":"2024-03-11T21:31:12.737518+0000","last_peered":"2024-03-11T21:31:12.737518+0000","last_clean":"2024-03-11T21:31:12.737518+0000","last_became_active":"2024-03-11T21:25:38.543834+0000","last_became_peered":"2024-03-11T21:25:38.543834+0000","last_unstale":"2024-03-11T21:31:12.737518+0000","last_undegraded":"2024-03-11T21:31:12.737518+0000","last_fullsized":"2024-03-11T21:31:12.737518+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:30:22.969840+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:37.512018+0000","last_clean_scrub_stamp":"2024-03-11T21:30:22.969840+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:31:46.044467+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0.00089342499999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":202,"reported_epoch":146,"state":"active+clean","last_fresh":"2024-03-11T21:31:12.741438+0000","last_change":"2024-03-11T21:31:06.457491+0000","last_active":"2024-03-11T21:31:12.741438+0000","last_peered":"2024-03-11T21:31:12.741438+0000","last_clean":"2024-03-11T21:31:12.741438+0000","last_became_active":"2024-03-11T21:25:38.541617+0000","last_became_peered":"2024-03-11T21:25:38.541617+0000","last_unstale":"2024-03-11T21:31:12.741438+0000","last_undegraded":"2024-03-11T21:31:12.741438+0000","last_fullsized":"2024-03-11T21:31:12.741438+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:31:06.457393+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:37.512018+0000","last_clean_scrub_stamp":"2024-03-11T21:31:06.457393+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:32:33.915715+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,0],"acting":[7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.6","version":"74'12","reported_seq":209,"reported_epoch":146,"state":"active+clean","last_fresh":"2024-03-11T21:31:12.739439+0000","last_change":"2024-03-11T21:31:04.383837+0000","last_active":"2024-03-11T21:31:12.739439+0000","last_peered":"2024-03-11T21:31:12.739439+0000","last_clean":"2024-03-11T21:31:12.739439+0000","last_became_active":"2024-03-11T21:25:40.550388+0000","last_became_peered":"2024-03-11T21:25:40.550388+0000","last_unstale":"2024-03-11T21:31:12.739439+0000","last_undegraded":"2024-03-11T21:31:12.739439+0000","last_fullsized":"2024-03-11T21:31:12.739439+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'12","last_scrub_stamp":"2024-03-11T21:31:04.383580+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:39.525554+0000","last_clean_scrub_stamp":"2024-03-11T21:31:04.383580+0000","objects_scrubbed":6,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:32:16.235650+0000","scrub_duration":25,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":18,"num_read_kb":12,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":186,"reported_epoch":146,"state":"active+clean","last_fresh":"2024-03-11T21:31:12.756496+0000","last_change":"2024-03-11T21:31:12.634176+0000","last_active":"2024-03-11T21:31:12.756496+0000","last_peered":"2024-03-11T21:31:12.756496+0000","last_clean":"2024-03-11T21:31:12.756496+0000","last_became_active":"2024-03-11T21:25:42.572324+0000","last_became_peered":"2024-03-11T21:25:42.572324+0000","last_unstale":"2024-03-11T21:31:12.756496+0000","last_undegraded":"2024-03-11T21:31:12.756496+0000","last_fullsized":"2024-03-11T21:31:12.756496+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:31:12.634040+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:41.544550+0000","last_clean_scrub_stamp":"2024-03-11T21:31:12.634040+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:32:31.480900+0000","scrub_duration":16,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.0","version":"146'14542","reported_seq":14738,"reported_epoch":146,"state":"active+clean","last_fresh":"2024-03-11T21:31:16.961829+0000","last_change":"2024-03-11T21:31:11.737460+0000","last_active":"2024-03-11T21:31:16.961829+0000","last_peered":"2024-03-11T21:31:16.961829+0000","last_clean":"2024-03-11T21:31:16.961829+0000","last_became_active":"2024-03-11T21:27:36.694232+0000","last_became_peered":"2024-03-11T21:27:36.694232+0000","last_unstale":"2024-03-11T21:31:16.961829+0000","last_undegraded":"2024-03-11T21:31:16.961829+0000","last_fullsized":"2024-03-11T21:31:16.961829+0000","mapping_epoch":99,"log_start":"124'8700","ondisk_log_start":"124'8700","created":99,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"145'14196","last_scrub_stamp":"2024-03-11T21:31:11.737323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:27:35.667522+0000","last_clean_scrub_stamp":"2024-03-11T21:31:11.737323+0000","objects_scrubbed":743,"log_size":5842,"log_dups_size":0,"ondisk_log_size":5842,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:32:28.001263+0000","scrub_duration":161,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25952256,"num_objects":396,"num_object_clones":0,"num_object_copies":1188,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":396,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14542,"num_write_kb":478016,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3],"acting":[5,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":191,"reported_epoch":146,"state":"active+clean","last_fresh":"2024-03-11T21:31:12.759718+0000","last_change":"2024-03-11T21:31:01.858727+0000","last_active":"2024-03-11T21:31:12.759718+0000","last_peered":"2024-03-11T21:31:12.759718+0000","last_clean":"2024-03-11T21:31:12.759718+0000","last_became_active":"2024-03-11T21:25:45.301416+0000","last_became_peered":"2024-03-11T21:25:45.301416+0000","last_unstale":"2024-03-11T21:31:12.759718+0000","last_undegraded":"2024-03-11T21:31:12.759718+0000","last_fullsized":"2024-03-11T21:31:12.759718+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:31:01.856502+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:43.559452+0000","last_clean_scrub_stamp":"2024-03-11T21:31:01.856502+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:32:30.487243+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":194,"reported_epoch":146,"state":"active+clean","last_fresh":"2024-03-11T21:31:12.742440+0000","last_change":"2024-03-11T21:30:12.625380+0000","last_active":"2024-03-11T21:31:12.742440+0000","last_peered":"2024-03-11T21:31:12.742440+0000","last_clean":"2024-03-11T21:31:12.742440+0000","last_became_active":"2024-03-11T21:25:38.545148+0000","last_became_peered":"2024-03-11T21:25:38.545148+0000","last_unstale":"2024-03-11T21:31:12.742440+0000","last_undegraded":"2024-03-11T21:31:12.742440+0000","last_fullsized":"2024-03-11T21:31:12.742440+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:30:12.625276+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:26:57.333669+0000","last_clean_scrub_stamp":"2024-03-11T21:30:12.625276+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:31:28.500050+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.3","version":"74'19","reported_seq":227,"reported_epoch":146,"state":"active+clean","last_fresh":"2024-03-11T21:31:12.737069+0000","last_change":"2024-03-11T21:30:23.995660+0000","last_active":"2024-03-11T21:31:12.737069+0000","last_peered":"2024-03-11T21:31:12.737069+0000","last_clean":"2024-03-11T21:31:12.737069+0000","last_became_active":"2024-03-11T21:25:40.760594+0000","last_became_peered":"2024-03-11T21:25:40.760594+0000","last_unstale":"2024-03-11T21:31:12.737069+0000","last_undegraded":"2024-03-11T21:31:12.737069+0000","last_fullsized":"2024-03-11T21:31:12.737069+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'19","last_scrub_stamp":"2024-03-11T21:30:23.995503+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:39.525554+0000","last_clean_scrub_stamp":"2024-03-11T21:30:23.995503+0000","objects_scrubbed":8,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:31:28.880234+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":39,"num_read_kb":25,"num_write":22,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":195,"reported_epoch":146,"state":"active+clean","last_fresh":"2024-03-11T21:31:12.744168+0000","last_change":"2024-03-11T21:31:11.390780+0000","last_active":"2024-03-11T21:31:12.744168+0000","last_peered":"2024-03-11T21:31:12.744168+0000","last_clean":"2024-03-11T21:31:12.744168+0000","last_became_active":"2024-03-11T21:25:42.579045+0000","last_became_peered":"2024-03-11T21:25:42.579045+0000","last_unstale":"2024-03-11T21:31:12.744168+0000","last_undegraded":"2024-03-11T21:31:12.744168+0000","last_fullsized":"2024-03-11T21:31:12.744168+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:31:11.390707+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:29:36.918936+0000","last_clean_scrub_stamp":"2024-03-11T21:31:11.390707+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:32:13.656422+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.3","version":"146'14709","reported_seq":14877,"reported_epoch":146,"state":"active+clean","last_fresh":"2024-03-11T21:31:17.356309+0000","last_change":"2024-03-11T21:29:47.452562+0000","last_active":"2024-03-11T21:31:17.356309+0000","last_peered":"2024-03-11T21:31:17.356309+0000","last_clean":"2024-03-11T21:31:17.356309+0000","last_became_active":"2024-03-11T21:27:36.693506+0000","last_became_peered":"2024-03-11T21:27:36.693506+0000","last_unstale":"2024-03-11T21:31:17.356309+0000","last_undegraded":"2024-03-11T21:31:17.356309+0000","last_fullsized":"2024-03-11T21:31:17.356309+0000","mapping_epoch":99,"log_start":"127'10000","ondisk_log_start":"127'10000","created":99,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"124'9352","last_scrub_stamp":"2024-03-11T21:29:47.452230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:27:35.667522+0000","last_clean_scrub_stamp":"2024-03-11T21:29:47.452230+0000","objects_scrubbed":5797,"log_size":4709,"log_dups_size":0,"ondisk_log_size":4709,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"queued for scrub","scrub_duration":1664,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26935296,"num_objects":411,"num_object_clones":0,"num_object_copies":1233,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":411,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14709,"num_write_kb":483840,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,4],"acting":[3,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":183,"reported_epoch":146,"state":"active+clean","last_fresh":"2024-03-11T21:31:12.734330+0000","last_change":"2024-03-11T21:29:47.154729+0000","last_active":"2024-03-11T21:31:12.734330+0000","last_peered":"2024-03-11T21:31:12.734330+0000","last_clean":"2024-03-11T21:31:12.734330+0000","last_became_active":"2024-03-11T21:25:44.583638+0000","last_became_peered":"2024-03-11T21:25:44.583638+0000","last_unstale":"2024-03-11T21:31:12.734330+0000","last_undegraded":"2024-03-11T21:31:12.734330+0000","last_fullsized":"2024-03-11T21:31:12.734330+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:29:47.154619+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:43.559452+0000","last_clean_scrub_stamp":"2024-03-11T21:29:47.154619+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.6","version":"67'1","reported_seq":196,"reported_epoch":146,"state":"active+clean","last_fresh":"2024-03-11T21:31:12.757175+0000","last_change":"2024-03-11T21:29:33.236180+0000","last_active":"2024-03-11T21:31:12.757175+0000","last_peered":"2024-03-11T21:31:12.757175+0000","last_clean":"2024-03-11T21:31:12.757175+0000","last_became_active":"2024-03-11T21:25:38.536142+0000","last_became_peered":"2024-03-11T21:25:38.536142+0000","last_unstale":"2024-03-11T21:31:12.757175+0000","last_undegraded":"2024-03-11T21:31:12.757175+0000","last_fullsized":"2024-03-11T21:31:12.757175+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'1","last_scrub_stamp":"2024-03-11T21:29:33.236064+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:37.512018+0000","last_clean_scrub_stamp":"2024-03-11T21:29:33.236064+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,4],"acting":[1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.0","version":"93'26","reported_seq":230,"reported_epoch":146,"state":"active+clean","last_fresh":"2024-03-11T21:31:12.752474+0000","last_change":"2024-03-11T21:31:03.246972+0000","last_active":"2024-03-11T21:31:12.752474+0000","last_peered":"2024-03-11T21:31:12.752474+0000","last_clean":"2024-03-11T21:31:12.752474+0000","last_became_active":"2024-03-11T21:25:40.761716+0000","last_became_peered":"2024-03-11T21:25:40.761716+0000","last_unstale":"2024-03-11T21:31:12.752474+0000","last_undegraded":"2024-03-11T21:31:12.752474+0000","last_fullsized":"2024-03-11T21:31:12.752474+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"93'26","last_scrub_stamp":"2024-03-11T21:31:03.244599+0000","last_deep_scrub":"93'26","last_deep_scrub_stamp":"2024-03-11T21:26:40.680324+0000","last_clean_scrub_stamp":"2024-03-11T21:31:03.244599+0000","objects_scrubbed":8,"log_size":26,"log_dups_size":0,"ondisk_log_size":26,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:32:13.957518+0000","scrub_duration":19,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":34,"num_read_kb":22,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":187,"reported_epoch":146,"state":"active+clean","last_fresh":"2024-03-11T21:31:12.734786+0000","last_change":"2024-03-11T21:30:08.277864+0000","last_active":"2024-03-11T21:31:12.734786+0000","last_peered":"2024-03-11T21:31:12.734786+0000","last_clean":"2024-03-11T21:31:12.734786+0000","last_became_active":"2024-03-11T21:25:42.583647+0000","last_became_peered":"2024-03-11T21:25:42.583647+0000","last_unstale":"2024-03-11T21:31:12.734786+0000","last_undegraded":"2024-03-11T21:31:12.734786+0000","last_fullsized":"2024-03-11T21:31:12.734786+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:30:08.277733+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:26:51.863621+0000","last_clean_scrub_stamp":"2024-03-11T21:30:08.277733+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:31:13.113000+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,7],"acting":[4,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.2","version":"146'14885","reported_seq":15059,"reported_epoch":146,"state":"active+clean","last_fresh":"2024-03-11T21:31:17.443169+0000","last_change":"2024-03-11T21:29:41.726756+0000","last_active":"2024-03-11T21:31:17.443169+0000","last_peered":"2024-03-11T21:31:17.443169+0000","last_clean":"2024-03-11T21:31:17.443169+0000","last_became_active":"2024-03-11T21:27:36.692405+0000","last_became_peered":"2024-03-11T21:27:36.692405+0000","last_unstale":"2024-03-11T21:31:17.443169+0000","last_undegraded":"2024-03-11T21:31:17.443169+0000","last_fullsized":"2024-03-11T21:31:17.443169+0000","mapping_epoch":99,"log_start":"128'10223","ondisk_log_start":"128'10223","created":99,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"124'9134","last_scrub_stamp":"2024-03-11T21:29:41.726402+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:27:35.667522+0000","last_clean_scrub_stamp":"2024-03-11T21:29:41.726402+0000","objects_scrubbed":6183,"log_size":4662,"log_dups_size":0,"ondisk_log_size":4662,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"queued for scrub","scrub_duration":2000,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":24707072,"num_objects":377,"num_object_clones":0,"num_object_copies":1131,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":377,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14885,"num_write_kb":488384,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.3","version":"0'0","reported_seq":182,"reported_epoch":146,"state":"active+clean","last_fresh":"2024-03-11T21:31:12.741957+0000","last_change":"2024-03-11T21:29:42.485840+0000","last_active":"2024-03-11T21:31:12.741957+0000","last_peered":"2024-03-11T21:31:12.741957+0000","last_clean":"2024-03-11T21:31:12.741957+0000","last_became_active":"2024-03-11T21:25:45.302719+0000","last_became_peered":"2024-03-11T21:25:45.302719+0000","last_unstale":"2024-03-11T21:31:12.741957+0000","last_undegraded":"2024-03-11T21:31:12.741957+0000","last_fullsized":"2024-03-11T21:31:12.741957+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:29:42.485740+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:29:42.485740+0000","last_clean_scrub_stamp":"2024-03-11T21:29:42.485740+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,2],"acting":[7,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":203,"reported_epoch":146,"state":"active+clean","last_fresh":"2024-03-11T21:31:12.744654+0000","last_change":"2024-03-11T21:31:01.530536+0000","last_active":"2024-03-11T21:31:12.744654+0000","last_peered":"2024-03-11T21:31:12.744654+0000","last_clean":"2024-03-11T21:31:12.744654+0000","last_became_active":"2024-03-11T21:25:38.540013+0000","last_became_peered":"2024-03-11T21:25:38.540013+0000","last_unstale":"2024-03-11T21:31:12.744654+0000","last_undegraded":"2024-03-11T21:31:12.744654+0000","last_fullsized":"2024-03-11T21:31:12.744654+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:31:01.530407+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:37.512018+0000","last_clean_scrub_stamp":"2024-03-11T21:31:01.530407+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:32:06.743781+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,2],"acting":[6,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.1","version":"74'14","reported_seq":212,"reported_epoch":146,"state":"active+clean","last_fresh":"2024-03-11T21:31:12.734045+0000","last_change":"2024-03-11T21:31:02.486605+0000","last_active":"2024-03-11T21:31:12.734045+0000","last_peered":"2024-03-11T21:31:12.734045+0000","last_clean":"2024-03-11T21:31:12.734045+0000","last_became_active":"2024-03-11T21:25:40.546219+0000","last_became_peered":"2024-03-11T21:25:40.546219+0000","last_unstale":"2024-03-11T21:31:12.734045+0000","last_undegraded":"2024-03-11T21:31:12.734045+0000","last_fullsized":"2024-03-11T21:31:12.734045+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'14","last_scrub_stamp":"2024-03-11T21:31:02.486457+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:39.525554+0000","last_clean_scrub_stamp":"2024-03-11T21:31:02.486457+0000","objects_scrubbed":7,"log_size":14,"log_dups_size":0,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:32:07.493431+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":21,"num_read_kb":14,"num_write":14,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6],"acting":[4,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.0","version":"98'9","reported_seq":255,"reported_epoch":146,"state":"active+clean","last_fresh":"2024-03-11T21:31:12.752429+0000","last_change":"2024-03-11T21:29:24.578770+0000","last_active":"2024-03-11T21:31:12.752429+0000","last_peered":"2024-03-11T21:31:12.752429+0000","last_clean":"2024-03-11T21:31:12.752429+0000","last_became_active":"2024-03-11T21:25:42.576126+0000","last_became_peered":"2024-03-11T21:25:42.576126+0000","last_unstale":"2024-03-11T21:31:12.752429+0000","last_undegraded":"2024-03-11T21:31:12.752429+0000","last_fullsized":"2024-03-11T21:31:12.752429+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"98'9","last_scrub_stamp":"2024-03-11T21:29:24.578653+0000","last_deep_scrub":"98'9","last_deep_scrub_stamp":"2024-03-11T21:27:29.745711+0000","last_clean_scrub_stamp":"2024-03-11T21:29:24.578653+0000","objects_scrubbed":1,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.d","version":"146'14949","reported_seq":15156,"reported_epoch":146,"state":"active+clean","last_fresh":"2024-03-11T21:31:17.049386+0000","last_change":"2024-03-11T21:31:07.816875+0000","last_active":"2024-03-11T21:31:17.049386+0000","last_peered":"2024-03-11T21:31:17.049386+0000","last_clean":"2024-03-11T21:31:17.049386+0000","last_became_active":"2024-03-11T21:27:36.695186+0000","last_became_peered":"2024-03-11T21:27:36.695186+0000","last_unstale":"2024-03-11T21:31:17.049386+0000","last_undegraded":"2024-03-11T21:31:17.049386+0000","last_fullsized":"2024-03-11T21:31:17.049386+0000","mapping_epoch":99,"log_start":"124'8600","ondisk_log_start":"124'8600","created":99,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"144'14363","last_scrub_stamp":"2024-03-11T21:31:07.816767+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:27:35.667522+0000","last_clean_scrub_stamp":"2024-03-11T21:31:07.816767+0000","objects_scrubbed":959,"log_size":6349,"log_dups_size":0,"ondisk_log_size":6349,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:32:36.874959+0000","scrub_duration":402,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":23592984,"num_objects":361,"num_object_clones":0,"num_object_copies":1083,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":361,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":14949,"num_write_kb":489857,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,0],"acting":[6,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.c","version":"0'0","reported_seq":174,"reported_epoch":146,"state":"active+clean","last_fresh":"2024-03-11T21:31:12.750565+0000","last_change":"2024-03-11T21:30:23.986312+0000","last_active":"2024-03-11T21:31:12.750565+0000","last_peered":"2024-03-11T21:31:12.750565+0000","last_clean":"2024-03-11T21:31:12.750565+0000","last_became_active":"2024-03-11T21:25:45.302207+0000","last_became_peered":"2024-03-11T21:25:45.302207+0000","last_unstale":"2024-03-11T21:31:12.750565+0000","last_undegraded":"2024-03-11T21:31:12.750565+0000","last_fullsized":"2024-03-11T21:31:12.750565+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:30:23.986191+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:43.559452+0000","last_clean_scrub_stamp":"2024-03-11T21:30:23.986191+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:31:25.299859+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.f","version":"0'0","reported_seq":186,"reported_epoch":146,"state":"active+clean","last_fresh":"2024-03-11T21:31:12.756407+0000","last_change":"2024-03-11T21:29:28.844148+0000","last_active":"2024-03-11T21:31:12.756407+0000","last_peered":"2024-03-11T21:31:12.756407+0000","last_clean":"2024-03-11T21:31:12.756407+0000","last_became_active":"2024-03-11T21:25:42.575935+0000","last_became_peered":"2024-03-11T21:25:42.575935+0000","last_unstale":"2024-03-11T21:31:12.756407+0000","last_undegraded":"2024-03-11T21:31:12.756407+0000","last_fullsized":"2024-03-11T21:31:12.756407+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:29:28.844034+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:41.544550+0000","last_clean_scrub_stamp":"2024-03-11T21:29:28.844034+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.8","version":"0'0","reported_seq":194,"reported_epoch":146,"state":"active+clean","last_fresh":"2024-03-11T21:31:12.741912+0000","last_change":"2024-03-11T21:30:13.584857+0000","last_active":"2024-03-11T21:31:12.741912+0000","last_peered":"2024-03-11T21:31:12.741912+0000","last_clean":"2024-03-11T21:31:12.741912+0000","last_became_active":"2024-03-11T21:25:38.540910+0000","last_became_peered":"2024-03-11T21:25:38.540910+0000","last_unstale":"2024-03-11T21:31:12.741912+0000","last_undegraded":"2024-03-11T21:31:12.741912+0000","last_fullsized":"2024-03-11T21:31:12.741912+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:30:13.584506+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:37.512018+0000","last_clean_scrub_stamp":"2024-03-11T21:30:13.584506+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:31:39.126385+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,1],"acting":[7,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.e","version":"109'16","reported_seq":215,"reported_epoch":146,"state":"active+clean","last_fresh":"2024-03-11T21:31:12.734028+0000","last_change":"2024-03-11T21:30:56.453809+0000","last_active":"2024-03-11T21:31:12.734028+0000","last_peered":"2024-03-11T21:31:12.734028+0000","last_clean":"2024-03-11T21:31:12.734028+0000","last_became_active":"2024-03-11T21:25:40.552178+0000","last_became_peered":"2024-03-11T21:25:40.552178+0000","last_unstale":"2024-03-11T21:31:12.734028+0000","last_undegraded":"2024-03-11T21:31:12.734028+0000","last_fullsized":"2024-03-11T21:31:12.734028+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"109'16","last_scrub_stamp":"2024-03-11T21:30:56.453670+0000","last_deep_scrub":"109'16","last_deep_scrub_stamp":"2024-03-11T21:28:24.739395+0000","last_clean_scrub_stamp":"2024-03-11T21:30:56.453670+0000","objects_scrubbed":5,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:32:01.053867+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.c","version":"146'14810","reported_seq":15018,"reported_epoch":146,"state":"active+clean","last_fresh":"2024-03-11T21:31:17.025849+0000","last_change":"2024-03-11T21:31:00.837025+0000","last_active":"2024-03-11T21:31:17.025849+0000","last_peered":"2024-03-11T21:31:17.025849+0000","last_clean":"2024-03-11T21:31:17.025849+0000","last_became_active":"2024-03-11T21:27:36.693325+0000","last_became_peered":"2024-03-11T21:27:36.693325+0000","last_unstale":"2024-03-11T21:31:17.025849+0000","last_undegraded":"2024-03-11T21:31:17.025849+0000","last_fullsized":"2024-03-11T21:31:17.025849+0000","mapping_epoch":99,"log_start":"124'8500","ondisk_log_start":"124'8500","created":99,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"144'13774","last_scrub_stamp":"2024-03-11T21:31:00.836924+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:27:35.667522+0000","last_clean_scrub_stamp":"2024-03-11T21:31:00.836924+0000","objects_scrubbed":1423,"log_size":6310,"log_dups_size":0,"ondisk_log_size":6310,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:32:07.800734+0000","scrub_duration":338,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":24903680,"num_objects":380,"num_object_clones":0,"num_object_copies":1140,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":380,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14810,"num_write_kb":486080,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,7],"acting":[6,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.d","version":"0'0","reported_seq":182,"reported_epoch":146,"state":"active+clean","last_fresh":"2024-03-11T21:31:12.755977+0000","last_change":"2024-03-11T21:30:11.993990+0000","last_active":"2024-03-11T21:31:12.755977+0000","last_peered":"2024-03-11T21:31:12.755977+0000","last_clean":"2024-03-11T21:31:12.755977+0000","last_became_active":"2024-03-11T21:25:44.590333+0000","last_became_peered":"2024-03-11T21:25:44.590333+0000","last_unstale":"2024-03-11T21:31:12.755977+0000","last_undegraded":"2024-03-11T21:31:12.755977+0000","last_fullsized":"2024-03-11T21:31:12.755977+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:30:11.993862+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:43.559452+0000","last_clean_scrub_stamp":"2024-03-11T21:30:11.993862+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:31:18.236956+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.e","version":"74'8","reported_seq":263,"reported_epoch":146,"state":"active+clean","last_fresh":"2024-03-11T21:31:12.733833+0000","last_change":"2024-03-11T21:31:05.564402+0000","last_active":"2024-03-11T21:31:12.733833+0000","last_peered":"2024-03-11T21:31:12.733833+0000","last_clean":"2024-03-11T21:31:12.733833+0000","last_became_active":"2024-03-11T21:25:42.576742+0000","last_became_peered":"2024-03-11T21:25:42.576742+0000","last_unstale":"2024-03-11T21:31:12.733833+0000","last_undegraded":"2024-03-11T21:31:12.733833+0000","last_fullsized":"2024-03-11T21:31:12.733833+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'8","last_scrub_stamp":"2024-03-11T21:31:05.564320+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:41.544550+0000","last_clean_scrub_stamp":"2024-03-11T21:31:05.564320+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:32:33.999249+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,0],"acting":[4,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.9","version":"0'0","reported_seq":195,"reported_epoch":146,"state":"active+clean","last_fresh":"2024-03-11T21:31:12.759542+0000","last_change":"2024-03-11T21:29:18.412164+0000","last_active":"2024-03-11T21:31:12.759542+0000","last_peered":"2024-03-11T21:31:12.759542+0000","last_clean":"2024-03-11T21:31:12.759542+0000","last_became_active":"2024-03-11T21:25:38.540976+0000","last_became_peered":"2024-03-11T21:25:38.540976+0000","last_unstale":"2024-03-11T21:31:12.759542+0000","last_undegraded":"2024-03-11T21:31:12.759542+0000","last_fullsized":"2024-03-11T21:31:12.759542+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:29:18.411652+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:26:50.991712+0000","last_clean_scrub_stamp":"2024-03-11T21:29:18.411652+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,3],"acting":[1,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.f","version":"74'15","reported_seq":168,"reported_epoch":146,"state":"active+clean","last_fresh":"2024-03-11T21:31:12.743748+0000","last_change":"2024-03-11T21:30:13.987515+0000","last_active":"2024-03-11T21:31:12.743748+0000","last_peered":"2024-03-11T21:31:12.743748+0000","last_clean":"2024-03-11T21:31:12.743748+0000","last_became_active":"2024-03-11T21:26:11.216224+0000","last_became_peered":"2024-03-11T21:26:11.216224+0000","last_unstale":"2024-03-11T21:31:12.743748+0000","last_undegraded":"2024-03-11T21:31:12.743748+0000","last_fullsized":"2024-03-11T21:31:12.743748+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'15","last_scrub_stamp":"2024-03-11T21:30:13.987375+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:39.525554+0000","last_clean_scrub_stamp":"2024-03-11T21:30:13.987375+0000","objects_scrubbed":7,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:31:36.359932+0000","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,4],"acting":[2,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.f","version":"146'18212","reported_seq":18510,"reported_epoch":146,"state":"active+clean","last_fresh":"2024-03-11T21:31:17.759648+0000","last_change":"2024-03-11T21:30:13.830365+0000","last_active":"2024-03-11T21:31:17.759648+0000","last_peered":"2024-03-11T21:31:17.759648+0000","last_clean":"2024-03-11T21:31:17.759648+0000","last_became_active":"2024-03-11T21:27:36.694032+0000","last_became_peered":"2024-03-11T21:27:36.694032+0000","last_unstale":"2024-03-11T21:31:17.759648+0000","last_undegraded":"2024-03-11T21:31:17.759648+0000","last_fullsized":"2024-03-11T21:31:17.759648+0000","mapping_epoch":99,"log_start":"126'12702","ondisk_log_start":"126'12702","created":99,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"131'14092","last_scrub_stamp":"2024-03-11T21:30:13.830084+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:27:35.667522+0000","last_clean_scrub_stamp":"2024-03-11T21:30:13.830084+0000","objects_scrubbed":4557,"log_size":5510,"log_dups_size":0,"ondisk_log_size":5510,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:31:42.886014+0000","scrub_duration":2014,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25755648,"num_objects":393,"num_object_clones":0,"num_object_copies":1179,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":393,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15009,"num_write_kb":492864,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.e","version":"0'0","reported_seq":183,"reported_epoch":146,"state":"active+clean","last_fresh":"2024-03-11T21:31:12.733704+0000","last_change":"2024-03-11T21:31:09.597590+0000","last_active":"2024-03-11T21:31:12.733704+0000","last_peered":"2024-03-11T21:31:12.733704+0000","last_clean":"2024-03-11T21:31:12.733704+0000","last_became_active":"2024-03-11T21:25:44.590198+0000","last_became_peered":"2024-03-11T21:25:44.590198+0000","last_unstale":"2024-03-11T21:31:12.733704+0000","last_undegraded":"2024-03-11T21:31:12.733704+0000","last_fullsized":"2024-03-11T21:31:12.733704+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:31:09.597448+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:43.559452+0000","last_clean_scrub_stamp":"2024-03-11T21:31:09.597448+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:32:30.206261+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.d","version":"109'9","reported_seq":263,"reported_epoch":146,"state":"active+clean","last_fresh":"2024-03-11T21:31:12.746593+0000","last_change":"2024-03-11T21:30:20.940788+0000","last_active":"2024-03-11T21:31:12.746593+0000","last_peered":"2024-03-11T21:31:12.746593+0000","last_clean":"2024-03-11T21:31:12.746593+0000","last_became_active":"2024-03-11T21:25:42.575583+0000","last_became_peered":"2024-03-11T21:25:42.575583+0000","last_unstale":"2024-03-11T21:31:12.746593+0000","last_undegraded":"2024-03-11T21:31:12.746593+0000","last_fullsized":"2024-03-11T21:31:12.746593+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"109'9","last_scrub_stamp":"2024-03-11T21:30:20.940485+0000","last_deep_scrub":"109'9","last_deep_scrub_stamp":"2024-03-11T21:28:27.082975+0000","last_clean_scrub_stamp":"2024-03-11T21:30:20.940485+0000","objects_scrubbed":1,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:31:43.649095+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,5],"acting":[2,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.a","version":"0'0","reported_seq":203,"reported_epoch":146,"state":"active+clean","last_fresh":"2024-03-11T21:31:12.760777+0000","last_change":"2024-03-11T21:31:06.898504+0000","last_active":"2024-03-11T21:31:12.760777+0000","last_peered":"2024-03-11T21:31:12.760777+0000","last_clean":"2024-03-11T21:31:12.760777+0000","last_became_active":"2024-03-11T21:25:38.541045+0000","last_became_peered":"2024-03-11T21:25:38.541045+0000","last_unstale":"2024-03-11T21:31:12.760777+0000","last_undegraded":"2024-03-11T21:31:12.760777+0000","last_fullsized":"2024-03-11T21:31:12.760777+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:31:06.898394+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:37.512018+0000","last_clean_scrub_stamp":"2024-03-11T21:31:06.898394+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:32:14.529726+0000","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,7],"acting":[1,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.c","version":"74'10","reported_seq":206,"reported_epoch":146,"state":"active+clean","last_fresh":"2024-03-11T21:31:12.733697+0000","last_change":"2024-03-11T21:30:57.495457+0000","last_active":"2024-03-11T21:31:12.733697+0000","last_peered":"2024-03-11T21:31:12.733697+0000","last_clean":"2024-03-11T21:31:12.733697+0000","last_became_active":"2024-03-11T21:25:40.550885+0000","last_became_peered":"2024-03-11T21:25:40.550885+0000","last_unstale":"2024-03-11T21:31:12.733697+0000","last_undegraded":"2024-03-11T21:31:12.733697+0000","last_fullsized":"2024-03-11T21:31:12.733697+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'10","last_scrub_stamp":"2024-03-11T21:30:57.495302+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:39.525554+0000","last_clean_scrub_stamp":"2024-03-11T21:30:57.495302+0000","objects_scrubbed":5,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:32:04.367621+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.e","version":"146'14945","reported_seq":15142,"reported_epoch":146,"state":"active+clean","last_fresh":"2024-03-11T21:31:17.157383+0000","last_change":"2024-03-11T21:31:16.547286+0000","last_active":"2024-03-11T21:31:17.157383+0000","last_peered":"2024-03-11T21:31:17.157383+0000","last_clean":"2024-03-11T21:31:17.157383+0000","last_became_active":"2024-03-11T21:27:36.691800+0000","last_became_peered":"2024-03-11T21:27:36.691800+0000","last_unstale":"2024-03-11T21:31:17.157383+0000","last_undegraded":"2024-03-11T21:31:17.157383+0000","last_fullsized":"2024-03-11T21:31:17.157383+0000","mapping_epoch":99,"log_start":"122'8200","ondisk_log_start":"122'8200","created":99,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"146'14900","last_scrub_stamp":"2024-03-11T21:31:16.547085+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:27:35.667522+0000","last_clean_scrub_stamp":"2024-03-11T21:31:16.547085+0000","objects_scrubbed":441,"log_size":6745,"log_dups_size":0,"ondisk_log_size":6745,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:32:26.495904+0000","scrub_duration":185,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":25755648,"num_objects":393,"num_object_clones":0,"num_object_copies":1179,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":393,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14945,"num_write_kb":490816,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,0],"acting":[2,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.f","version":"0'0","reported_seq":182,"reported_epoch":146,"state":"active+clean","last_fresh":"2024-03-11T21:31:12.743895+0000","last_change":"2024-03-11T21:30:21.979887+0000","last_active":"2024-03-11T21:31:12.743895+0000","last_peered":"2024-03-11T21:31:12.743895+0000","last_clean":"2024-03-11T21:31:12.743895+0000","last_became_active":"2024-03-11T21:25:44.593414+0000","last_became_peered":"2024-03-11T21:25:44.593414+0000","last_unstale":"2024-03-11T21:31:12.743895+0000","last_undegraded":"2024-03-11T21:31:12.743895+0000","last_fullsized":"2024-03-11T21:31:12.743895+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:30:21.979536+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:27:03.010969+0000","last_clean_scrub_stamp":"2024-03-11T21:30:21.979536+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:31:47.911652+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,4],"acting":[2,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":187,"reported_epoch":146,"state":"active+clean","last_fresh":"2024-03-11T21:31:12.758075+0000","last_change":"2024-03-11T21:30:22.750174+0000","last_active":"2024-03-11T21:31:12.758075+0000","last_peered":"2024-03-11T21:31:12.758075+0000","last_clean":"2024-03-11T21:31:12.758075+0000","last_became_active":"2024-03-11T21:25:42.579409+0000","last_became_peered":"2024-03-11T21:25:42.579409+0000","last_unstale":"2024-03-11T21:31:12.758075+0000","last_undegraded":"2024-03-11T21:31:12.758075+0000","last_fullsized":"2024-03-11T21:31:12.758075+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:30:22.750053+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:30:22.750053+0000","last_clean_scrub_stamp":"2024-03-11T21:30:22.750053+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:31:45.145778+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.b","version":"0'0","reported_seq":194,"reported_epoch":146,"state":"active+clean","last_fresh":"2024-03-11T21:31:12.741629+0000","last_change":"2024-03-11T21:29:46.582228+0000","last_active":"2024-03-11T21:31:12.741629+0000","last_peered":"2024-03-11T21:31:12.741629+0000","last_clean":"2024-03-11T21:31:12.741629+0000","last_became_active":"2024-03-11T21:25:38.543377+0000","last_became_peered":"2024-03-11T21:25:38.543377+0000","last_unstale":"2024-03-11T21:31:12.741629+0000","last_undegraded":"2024-03-11T21:31:12.741629+0000","last_fullsized":"2024-03-11T21:31:12.741629+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:29:46.582119+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:37.512018+0000","last_clean_scrub_stamp":"2024-03-11T21:29:46.582119+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,5],"acting":[7,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.d","version":"74'17","reported_seq":219,"reported_epoch":146,"state":"active+clean","last_fresh":"2024-03-11T21:31:12.733430+0000","last_change":"2024-03-11T21:30:20.287636+0000","last_active":"2024-03-11T21:31:12.733430+0000","last_peered":"2024-03-11T21:31:12.733430+0000","last_clean":"2024-03-11T21:31:12.733430+0000","last_became_active":"2024-03-11T21:25:40.553454+0000","last_became_peered":"2024-03-11T21:25:40.553454+0000","last_unstale":"2024-03-11T21:31:12.733430+0000","last_undegraded":"2024-03-11T21:31:12.733430+0000","last_fullsized":"2024-03-11T21:31:12.733430+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'17","last_scrub_stamp":"2024-03-11T21:30:20.287535+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:39.525554+0000","last_clean_scrub_stamp":"2024-03-11T21:30:20.287535+0000","objects_scrubbed":8,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:31:40.112321+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":29,"num_read_kb":19,"num_write":18,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,1],"acting":[4,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.9","version":"146'16998","reported_seq":17245,"reported_epoch":146,"state":"active+clean","last_fresh":"2024-03-11T21:31:17.748601+0000","last_change":"2024-03-11T21:30:14.522862+0000","last_active":"2024-03-11T21:31:17.748601+0000","last_peered":"2024-03-11T21:31:17.748601+0000","last_clean":"2024-03-11T21:31:17.748601+0000","last_became_active":"2024-03-11T21:27:36.692978+0000","last_became_peered":"2024-03-11T21:27:36.692978+0000","last_unstale":"2024-03-11T21:31:17.748601+0000","last_undegraded":"2024-03-11T21:31:17.748601+0000","last_fullsized":"2024-03-11T21:31:17.748601+0000","mapping_epoch":99,"log_start":"126'11501","ondisk_log_start":"126'11501","created":99,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"131'13017","last_scrub_stamp":"2024-03-11T21:30:14.519102+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:27:35.667522+0000","last_clean_scrub_stamp":"2024-03-11T21:30:14.519102+0000","objects_scrubbed":4381,"log_size":5497,"log_dups_size":0,"ondisk_log_size":5497,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:31:44.028644+0000","scrub_duration":1664,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":23330816,"num_objects":356,"num_object_clones":0,"num_object_copies":1068,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":356,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14812,"num_write_kb":485376,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":182,"reported_epoch":146,"state":"active+clean","last_fresh":"2024-03-11T21:31:12.741426+0000","last_change":"2024-03-11T21:30:16.650959+0000","last_active":"2024-03-11T21:31:12.741426+0000","last_peered":"2024-03-11T21:31:12.741426+0000","last_clean":"2024-03-11T21:31:12.741426+0000","last_became_active":"2024-03-11T21:25:44.589973+0000","last_became_peered":"2024-03-11T21:25:44.589973+0000","last_unstale":"2024-03-11T21:31:12.741426+0000","last_undegraded":"2024-03-11T21:31:12.741426+0000","last_fullsized":"2024-03-11T21:31:12.741426+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:30:16.650854+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:28:37.561826+0000","last_clean_scrub_stamp":"2024-03-11T21:30:16.650854+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:31:43.938865+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,3],"acting":[7,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":186,"reported_epoch":146,"state":"active+clean","last_fresh":"2024-03-11T21:31:12.742105+0000","last_change":"2024-03-11T21:30:57.336406+0000","last_active":"2024-03-11T21:31:12.742105+0000","last_peered":"2024-03-11T21:31:12.742105+0000","last_clean":"2024-03-11T21:31:12.742105+0000","last_became_active":"2024-03-11T21:25:42.574515+0000","last_became_peered":"2024-03-11T21:25:42.574515+0000","last_unstale":"2024-03-11T21:31:12.742105+0000","last_undegraded":"2024-03-11T21:31:12.742105+0000","last_fullsized":"2024-03-11T21:31:12.742105+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:30:57.336188+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:41.544550+0000","last_clean_scrub_stamp":"2024-03-11T21:30:57.336188+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:32:12.876558+0000","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,5],"acting":[2,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.c","version":"0'0","reported_seq":194,"reported_epoch":146,"state":"active+clean","last_fresh":"2024-03-11T21:31:12.743297+0000","last_change":"2024-03-11T21:31:09.381415+0000","last_active":"2024-03-11T21:31:12.743297+0000","last_peered":"2024-03-11T21:31:12.743297+0000","last_clean":"2024-03-11T21:31:12.743297+0000","last_became_active":"2024-03-11T21:25:38.533240+0000","last_became_peered":"2024-03-11T21:25:38.533240+0000","last_unstale":"2024-03-11T21:31:12.743297+0000","last_undegraded":"2024-03-11T21:31:12.743297+0000","last_fullsized":"2024-03-11T21:31:12.743297+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:31:09.381301+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:29:30.279910+0000","last_clean_scrub_stamp":"2024-03-11T21:31:09.381301+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:32:38.869288+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,0],"acting":[2,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.a","version":"109'28","reported_seq":231,"reported_epoch":146,"state":"active+clean","last_fresh":"2024-03-11T21:31:12.744062+0000","last_change":"2024-03-11T21:30:10.797455+0000","last_active":"2024-03-11T21:31:12.744062+0000","last_peered":"2024-03-11T21:31:12.744062+0000","last_clean":"2024-03-11T21:31:12.744062+0000","last_became_active":"2024-03-11T21:25:40.760698+0000","last_became_peered":"2024-03-11T21:25:40.760698+0000","last_unstale":"2024-03-11T21:31:12.744062+0000","last_undegraded":"2024-03-11T21:31:12.744062+0000","last_fullsized":"2024-03-11T21:31:12.744062+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"109'28","last_scrub_stamp":"2024-03-11T21:30:10.797357+0000","last_deep_scrub":"109'28","last_deep_scrub_stamp":"2024-03-11T21:28:24.016389+0000","last_clean_scrub_stamp":"2024-03-11T21:30:10.797357+0000","objects_scrubbed":9,"log_size":28,"log_dups_size":0,"ondisk_log_size":28,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:31:21.723235+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":32,"num_read_kb":21,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,7],"acting":[6,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.8","version":"146'14599","reported_seq":14802,"reported_epoch":146,"state":"active+clean","last_fresh":"2024-03-11T21:31:17.101708+0000","last_change":"2024-03-11T21:30:58.801387+0000","last_active":"2024-03-11T21:31:17.101708+0000","last_peered":"2024-03-11T21:31:17.101708+0000","last_clean":"2024-03-11T21:31:17.101708+0000","last_became_active":"2024-03-11T21:27:36.693060+0000","last_became_peered":"2024-03-11T21:27:36.693060+0000","last_unstale":"2024-03-11T21:31:17.101708+0000","last_undegraded":"2024-03-11T21:31:17.101708+0000","last_fullsized":"2024-03-11T21:31:17.101708+0000","mapping_epoch":99,"log_start":"120'7900","ondisk_log_start":"120'7900","created":99,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"144'13479","last_scrub_stamp":"2024-03-11T21:30:58.800987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:27:35.667522+0000","last_clean_scrub_stamp":"2024-03-11T21:30:58.800987+0000","objects_scrubbed":1545,"log_size":6699,"log_dups_size":0,"ondisk_log_size":6699,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:32:27.160455+0000","scrub_duration":524,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26935296,"num_objects":411,"num_object_clones":0,"num_object_copies":1233,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":411,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14599,"num_write_kb":480320,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,4],"acting":[2,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":183,"reported_epoch":146,"state":"active+clean","last_fresh":"2024-03-11T21:31:12.737807+0000","last_change":"2024-03-11T21:31:10.397757+0000","last_active":"2024-03-11T21:31:12.737807+0000","last_peered":"2024-03-11T21:31:12.737807+0000","last_clean":"2024-03-11T21:31:12.737807+0000","last_became_active":"2024-03-11T21:25:44.584154+0000","last_became_peered":"2024-03-11T21:25:44.584154+0000","last_unstale":"2024-03-11T21:31:12.737807+0000","last_undegraded":"2024-03-11T21:31:12.737807+0000","last_fullsized":"2024-03-11T21:31:12.737807+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:31:10.397515+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:27:11.361913+0000","last_clean_scrub_stamp":"2024-03-11T21:31:10.397515+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:32:15.530582+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":186,"reported_epoch":146,"state":"active+clean","last_fresh":"2024-03-11T21:31:12.741969+0000","last_change":"2024-03-11T21:31:11.390022+0000","last_active":"2024-03-11T21:31:12.741969+0000","last_peered":"2024-03-11T21:31:12.741969+0000","last_clean":"2024-03-11T21:31:12.741969+0000","last_became_active":"2024-03-11T21:25:42.576679+0000","last_became_peered":"2024-03-11T21:25:42.576679+0000","last_unstale":"2024-03-11T21:31:12.741969+0000","last_undegraded":"2024-03-11T21:31:12.741969+0000","last_fullsized":"2024-03-11T21:31:12.741969+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:31:11.389869+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:41.544550+0000","last_clean_scrub_stamp":"2024-03-11T21:31:11.389869+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:32:39.283658+0000","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,3],"acting":[2,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.d","version":"0'0","reported_seq":195,"reported_epoch":146,"state":"active+clean","last_fresh":"2024-03-11T21:31:12.760282+0000","last_change":"2024-03-11T21:29:20.439881+0000","last_active":"2024-03-11T21:31:12.760282+0000","last_peered":"2024-03-11T21:31:12.760282+0000","last_clean":"2024-03-11T21:31:12.760282+0000","last_became_active":"2024-03-11T21:25:38.539681+0000","last_became_peered":"2024-03-11T21:25:38.539681+0000","last_unstale":"2024-03-11T21:31:12.760282+0000","last_undegraded":"2024-03-11T21:31:12.760282+0000","last_fullsized":"2024-03-11T21:31:12.760282+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:29:20.439742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:37.512018+0000","last_clean_scrub_stamp":"2024-03-11T21:29:20.439742+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,3],"acting":[1,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.b","version":"74'9","reported_seq":199,"reported_epoch":146,"state":"active+clean","last_fresh":"2024-03-11T21:31:12.737239+0000","last_change":"2024-03-11T21:29:31.301042+0000","last_active":"2024-03-11T21:31:12.737239+0000","last_peered":"2024-03-11T21:31:12.737239+0000","last_clean":"2024-03-11T21:31:12.737239+0000","last_became_active":"2024-03-11T21:25:40.549612+0000","last_became_peered":"2024-03-11T21:25:40.549612+0000","last_unstale":"2024-03-11T21:31:12.737239+0000","last_undegraded":"2024-03-11T21:31:12.737239+0000","last_fullsized":"2024-03-11T21:31:12.737239+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'9","last_scrub_stamp":"2024-03-11T21:29:31.300892+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:39.525554+0000","last_clean_scrub_stamp":"2024-03-11T21:29:31.300892+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.b","version":"146'14835","reported_seq":15046,"reported_epoch":146,"state":"active+clean","last_fresh":"2024-03-11T21:31:17.082595+0000","last_change":"2024-03-11T21:31:03.330969+0000","last_active":"2024-03-11T21:31:17.082595+0000","last_peered":"2024-03-11T21:31:17.082595+0000","last_clean":"2024-03-11T21:31:17.082595+0000","last_became_active":"2024-03-11T21:27:36.691322+0000","last_became_peered":"2024-03-11T21:27:36.691322+0000","last_unstale":"2024-03-11T21:31:17.082595+0000","last_undegraded":"2024-03-11T21:31:17.082595+0000","last_fullsized":"2024-03-11T21:31:17.082595+0000","mapping_epoch":99,"log_start":"126'9700","ondisk_log_start":"126'9700","created":99,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"144'13944","last_scrub_stamp":"2024-03-11T21:31:03.330626+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:27:35.667522+0000","last_clean_scrub_stamp":"2024-03-11T21:31:03.330626+0000","objects_scrubbed":1311,"log_size":5135,"log_dups_size":0,"ondisk_log_size":5135,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:32:09.099528+0000","scrub_duration":469,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":26411008,"num_objects":403,"num_object_clones":0,"num_object_copies":1209,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":403,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14835,"num_write_kb":487616,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,6],"acting":[1,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":88,"reported_epoch":146,"state":"active+clean","last_fresh":"2024-03-11T21:31:12.733390+0000","last_change":"2024-03-11T21:30:25.380496+0000","last_active":"2024-03-11T21:31:12.733390+0000","last_peered":"2024-03-11T21:31:12.733390+0000","last_clean":"2024-03-11T21:31:12.733390+0000","last_became_active":"2024-03-11T21:28:45.372452+0000","last_became_peered":"2024-03-11T21:28:45.372452+0000","last_unstale":"2024-03-11T21:31:12.733390+0000","last_undegraded":"2024-03-11T21:31:12.733390+0000","last_fullsized":"2024-03-11T21:31:12.733390+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:30:25.380313+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:43.559452+0000","last_clean_scrub_stamp":"2024-03-11T21:30:25.380313+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:31:34.770501+0000","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,3],"acting":[4,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.e","version":"0'0","reported_seq":194,"reported_epoch":146,"state":"active+clean","last_fresh":"2024-03-11T21:31:12.747148+0000","last_change":"2024-03-11T21:30:14.983740+0000","last_active":"2024-03-11T21:31:12.747148+0000","last_peered":"2024-03-11T21:31:12.747148+0000","last_clean":"2024-03-11T21:31:12.747148+0000","last_became_active":"2024-03-11T21:25:38.539954+0000","last_became_peered":"2024-03-11T21:25:38.539954+0000","last_unstale":"2024-03-11T21:31:12.747148+0000","last_undegraded":"2024-03-11T21:31:12.747148+0000","last_fullsized":"2024-03-11T21:31:12.747148+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:30:14.983460+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:37.512018+0000","last_clean_scrub_stamp":"2024-03-11T21:30:14.983460+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:31:16.594855+0000","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,7],"acting":[2,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.8","version":"109'22","reported_seq":222,"reported_epoch":146,"state":"active+clean","last_fresh":"2024-03-11T21:31:12.755395+0000","last_change":"2024-03-11T21:29:44.835802+0000","last_active":"2024-03-11T21:31:12.755395+0000","last_peered":"2024-03-11T21:31:12.755395+0000","last_clean":"2024-03-11T21:31:12.755395+0000","last_became_active":"2024-03-11T21:25:40.761938+0000","last_became_peered":"2024-03-11T21:25:40.761938+0000","last_unstale":"2024-03-11T21:31:12.755395+0000","last_undegraded":"2024-03-11T21:31:12.755395+0000","last_fullsized":"2024-03-11T21:31:12.755395+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"109'22","last_scrub_stamp":"2024-03-11T21:29:44.835663+0000","last_deep_scrub":"109'22","last_deep_scrub_stamp":"2024-03-11T21:28:25.743058+0000","last_clean_scrub_stamp":"2024-03-11T21:29:44.835663+0000","objects_scrubbed":7,"log_size":22,"log_dups_size":0,"ondisk_log_size":22,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.9","version":"94'9","reported_seq":266,"reported_epoch":146,"state":"active+clean","last_fresh":"2024-03-11T21:31:12.741662+0000","last_change":"2024-03-11T21:29:23.565598+0000","last_active":"2024-03-11T21:31:12.741662+0000","last_peered":"2024-03-11T21:31:12.741662+0000","last_clean":"2024-03-11T21:31:12.741662+0000","last_became_active":"2024-03-11T21:25:42.584461+0000","last_became_peered":"2024-03-11T21:25:42.584461+0000","last_unstale":"2024-03-11T21:31:12.741662+0000","last_undegraded":"2024-03-11T21:31:12.741662+0000","last_fullsized":"2024-03-11T21:31:12.741662+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"94'9","last_scrub_stamp":"2024-03-11T21:29:23.565483+0000","last_deep_scrub":"94'9","last_deep_scrub_stamp":"2024-03-11T21:26:53.422552+0000","last_clean_scrub_stamp":"2024-03-11T21:29:23.565483+0000","objects_scrubbed":1,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":7,"num_pg":16,"stat_sum":{"num_bytes":413859864,"num_objects":6316,"num_object_clones":0,"num_object_copies":18948,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6316,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":237502,"num_write_kb":7802113,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1253126144,"data_stored":1253113928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":89565,"ondisk_log_size":89565,"up":48,"acting":48,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":1531,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":24,"num_read_kb":24,"num_write":10,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":4593,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":6,"ondisk_log_size":6,"up":96,"acting":96,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":212,"num_read_kb":306,"num_write":179,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1771104,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1781760,"data_stored":1771104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":128,"ondisk_log_size":128,"up":3,"acting":3,"num_store_stats":3},{"poolid":3,"num_pg":3,"stat_sum":{"num_bytes":408,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":332,"num_read_kb":327,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1224,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":9,"acting":9,"num_store_stats":7},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":3702,"num_objects":178,"num_object_clones":0,"num_object_copies":534,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":178,"num_whiteouts":0,"num_read":698,"num_read_kb":455,"num_write":417,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":417792,"data_stored":11106,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":456,"ondisk_log_size":456,"up":96,"acting":96,"num_store_stats":8},{"poolid":5,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":68,"ondisk_log_size":68,"up":96,"acting":96,"num_store_stats":8},{"poolid":6,"num_pg":32,"stat_sum":{"num_bytes":383,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1149,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":96,"acting":96,"num_store_stats":8}],"osd_stats":[{"osd":7,"up_from":60,"seq":257698037831,"num_pgs":61,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":258504,"kb_used_data":146816,"kb_used_omap":1,"kb_used_meta":111678,"kb_avail":93486648,"statfs":{"total":95995035648,"available":95730327552,"internally_reserved":0,"allocated":150339584,"data_stored":149954617,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":114358730},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,0,0,2],"upper_bound":32},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":50,"seq":214748364877,"num_pgs":48,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":259656,"kb_used_data":143808,"kb_used_omap":1,"kb_used_meta":115838,"kb_avail":93485496,"statfs":{"total":95995035648,"available":95729147904,"internally_reserved":0,"allocated":147259392,"data_stored":146874742,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":118618566},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,1,4],"upper_bound":16},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574951,"num_pgs":59,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":294500,"kb_used_data":160604,"kb_used_omap":1,"kb_used_meta":133886,"kb_avail":93450652,"statfs":{"total":95995035648,"available":95693467648,"internally_reserved":0,"allocated":164458496,"data_stored":164044738,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":137099717},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1],"upper_bound":4},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738476,"num_pgs":55,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":350672,"kb_used_data":197320,"kb_used_omap":1,"kb_used_meta":153342,"kb_avail":93394480,"statfs":{"total":95995035648,"available":95635947520,"internally_reserved":0,"allocated":202055680,"data_stored":201662762,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":157022661},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,1,3,1,3,2],"upper_bound":128},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411425,"num_pgs":45,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":298232,"kb_used_data":176816,"kb_used_omap":1,"kb_used_meta":121406,"kb_avail":93446920,"statfs":{"total":95995035648,"available":95689646080,"internally_reserved":0,"allocated":181059584,"data_stored":180690082,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":124320197},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,0,1,1],"upper_bound":32},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":3,"up_from":26,"seq":111669149789,"num_pgs":65,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":394976,"kb_used_data":229144,"kb_used_omap":1,"kb_used_meta":165822,"kb_avail":93350176,"statfs":{"total":95995035648,"available":95590580224,"internally_reserved":0,"allocated":234643456,"data_stored":234234104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":169802183},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,1,1,1],"upper_bound":16},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":4,"up_from":32,"seq":137438953558,"num_pgs":59,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":252816,"kb_used_data":145288,"kb_used_omap":1,"kb_used_meta":107518,"kb_avail":93492336,"statfs":{"total":95995035648,"available":95736152064,"internally_reserved":0,"allocated":148774912,"data_stored":148381118,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":110098885},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1],"upper_bound":8},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":41,"seq":176093659217,"num_pgs":52,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":202684,"kb_used_data":113204,"kb_used_omap":1,"kb_used_meta":89470,"kb_avail":93542468,"statfs":{"total":95995035648,"available":95787487232,"internally_reserved":0,"allocated":115920896,"data_stored":115547349,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":91617733},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,2],"upper_bound":4},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1049,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":46,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":957,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":574,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":436,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1003,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":528,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":1320,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":81920,"data_stored":2118,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":1018,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":1898,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":1474,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":990,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":1034,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":1254,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":370,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":383,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":370,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":190582784,"data_stored":190578712,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":154210304,"data_stored":154206232,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":160890880,"data_stored":160890880,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":231997440,"data_stored":231997440,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":139198464,"data_stored":139198464,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":107544576,"data_stored":107544576,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":132255744,"data_stored":132251672,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":136445952,"data_stored":136445952,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-11T21:31:24.086 DEBUG:teuthology.orchestra.run.smithi092:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid 00db0016-dfed-11ee-95c8-87774f69a715 -- ceph pg dump --format=json 2024-03-11T21:31:24.114 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:31:24.153 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:31:24.153 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:31:24.154 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:31:24.186 INFO:tasks.cephadm.ceph_manager.ceph:removing pool_name unique_pool_0 2024-03-11T21:31:24.186 DEBUG:teuthology.orchestra.run.smithi092:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid 00db0016-dfed-11ee-95c8-87774f69a715 -- ceph osd pool rm unique_pool_0 unique_pool_0 --yes-i-really-really-mean-it 2024-03-11T21:31:24.254 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:31:24.279 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:31:24.280 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:31:24.281 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:31:24.326 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:31:24 smithi097 ceph-mon[33317]: purged_snaps scrub starts 2024-03-11T21:31:24.326 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:31:24 smithi097 ceph-mon[33317]: purged_snaps scrub ok 2024-03-11T21:31:24.326 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:31:24 smithi097 ceph-mon[33317]: 4.a scrub starts 2024-03-11T21:31:24.326 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:31:24 smithi097 ceph-mon[33317]: 4.a scrub ok 2024-03-11T21:31:24.327 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:31:24 smithi097 ceph-mon[33317]: 6.10 scrub starts 2024-03-11T21:31:24.327 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:31:24 smithi097 ceph-mon[33317]: 6.10 scrub ok 2024-03-11T21:31:24.327 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:31:24 smithi097 ceph-mon[33317]: osdmap e147: 8 total, 8 up, 8 in 2024-03-11T21:31:24.327 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:31:24 smithi097 ceph-mon[33317]: pgmap v214: 148 pgs: 148 active+clean; 75 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 1.03k op/s 2024-03-11T21:31:24.360 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:31:24 smithi092 ceph-mon[37563]: purged_snaps scrub starts 2024-03-11T21:31:24.360 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:31:24 smithi092 ceph-mon[37563]: purged_snaps scrub ok 2024-03-11T21:31:24.360 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:31:24 smithi092 ceph-mon[37563]: 4.a scrub starts 2024-03-11T21:31:24.361 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:31:24 smithi092 ceph-mon[37563]: 4.a scrub ok 2024-03-11T21:31:24.361 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:31:24 smithi092 ceph-mon[37563]: 6.10 scrub starts 2024-03-11T21:31:24.361 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:31:24 smithi092 ceph-mon[37563]: 6.10 scrub ok 2024-03-11T21:31:24.361 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:31:24 smithi092 ceph-mon[37563]: osdmap e147: 8 total, 8 up, 8 in 2024-03-11T21:31:24.361 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:31:24 smithi092 ceph-mon[37563]: pgmap v214: 148 pgs: 148 active+clean; 75 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 1.03k op/s 2024-03-11T21:31:24.361 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:31:24 smithi092 ceph-mon[29572]: purged_snaps scrub starts 2024-03-11T21:31:24.361 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:31:24 smithi092 ceph-mon[29572]: purged_snaps scrub ok 2024-03-11T21:31:24.361 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:31:24 smithi092 ceph-mon[29572]: 4.a scrub starts 2024-03-11T21:31:24.361 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:31:24 smithi092 ceph-mon[29572]: 4.a scrub ok 2024-03-11T21:31:24.361 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:31:24 smithi092 ceph-mon[29572]: 6.10 scrub starts 2024-03-11T21:31:24.361 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:31:24 smithi092 ceph-mon[29572]: 6.10 scrub ok 2024-03-11T21:31:24.362 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:31:24 smithi092 ceph-mon[29572]: osdmap e147: 8 total, 8 up, 8 in 2024-03-11T21:31:24.362 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:31:24 smithi092 ceph-mon[29572]: pgmap v214: 148 pgs: 148 active+clean; 75 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 1.03k op/s 2024-03-11T21:31:24.382 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:31:24.406 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:31:24.407 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:31:24.407 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:31:24.508 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:31:24.531 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:31:24.532 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:31:24.532 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:31:24.633 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:31:24.660 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:31:24.660 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:31:24.661 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:31:24.762 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:31:24.789 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:31:24.790 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:31:24.790 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:31:24.891 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:31:24.915 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:31:24.915 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:31:24.916 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:31:25.016 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:31:25.045 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:31:25.045 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:31:25.046 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:31:25.147 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:31:25.174 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:31:25.174 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:31:25.175 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:31:25.276 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:31:25.303 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:31:25.303 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:31:25.304 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:31:25.326 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:31:25 smithi097 ceph-mon[33317]: 5.f scrub starts 2024-03-11T21:31:25.326 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:31:25 smithi097 ceph-mon[33317]: 5.f scrub ok 2024-03-11T21:31:25.360 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:31:25 smithi092 ceph-mon[37563]: 5.f scrub starts 2024-03-11T21:31:25.360 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:31:25 smithi092 ceph-mon[37563]: 5.f scrub ok 2024-03-11T21:31:25.360 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:31:25 smithi092 ceph-mon[29572]: 5.f scrub starts 2024-03-11T21:31:25.361 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:31:25 smithi092 ceph-mon[29572]: 5.f scrub ok 2024-03-11T21:31:25.405 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:31:25.430 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:31:25.431 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:31:25.431 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:31:25.532 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:31:25.560 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:31:25.561 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:31:25.561 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:31:25.667 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:31:25.696 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:31:25.696 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:31:25.697 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:31:25.798 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:31:25.823 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:31:25.823 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:31:25.823 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:31:25.924 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:31:25.953 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:31:25.954 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:31:25.954 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:31:26.055 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:31:26.066 INFO:teuthology.orchestra.run.smithi092.stderr:noscrub is set 2024-03-11T21:31:26.082 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:31:26.082 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:31:26.082 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:31:26.183 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:31:26.209 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:31:26.210 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:31:26.210 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:31:26.311 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:31:26.331 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:31:26 smithi097 ceph-mon[33317]: pgmap v215: 148 pgs: 148 active+clean; 75 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 1.03k op/s 2024-03-11T21:31:26.331 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:31:26 smithi097 ceph-mon[33317]: from='client.? v1:172.21.15.92:0/373970759' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-11T21:31:26.338 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:31:26 smithi092 ceph-mon[29572]: pgmap v215: 148 pgs: 148 active+clean; 75 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 1.03k op/s 2024-03-11T21:31:26.338 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:31:26 smithi092 ceph-mon[29572]: from='client.? v1:172.21.15.92:0/373970759' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-11T21:31:26.339 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:31:26 smithi092 ceph-mon[37563]: pgmap v215: 148 pgs: 148 active+clean; 75 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 1.03k op/s 2024-03-11T21:31:26.339 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:31:26 smithi092 ceph-mon[37563]: from='client.? v1:172.21.15.92:0/373970759' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-11T21:31:26.355 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:31:26.356 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:31:26.356 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:31:26.457 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:31:26.491 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:31:26.492 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:31:26.492 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:31:26.550 INFO:teuthology.orchestra.run.smithi092.stderr:Inferring config /var/lib/ceph/00db0016-dfed-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T21:31:26.584 INFO:teuthology.orchestra.run.smithi092.stderr:Inferring config /var/lib/ceph/00db0016-dfed-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T21:31:26.593 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:31:26.625 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:31:26.626 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:31:26.626 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:31:26.727 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:31:26.759 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:31:26.760 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:31:26.760 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:31:26.861 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:31:26.894 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:31:26.895 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:31:26.895 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:31:26.996 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:31:27.024 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:31:27.025 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:31:27.025 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:31:27.127 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:31:27.152 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:31:27.152 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:31:27.152 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:31:27.253 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:31:27.282 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:31:27.282 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:31:27.283 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:31:27.326 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:31:27 smithi097 ceph-mon[33317]: 7.2 scrub starts 2024-03-11T21:31:27.326 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:31:27 smithi097 ceph-mon[33317]: 7.2 scrub ok 2024-03-11T21:31:27.326 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:31:27 smithi097 ceph-mon[33317]: 6.17 scrub starts 2024-03-11T21:31:27.326 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:31:27 smithi097 ceph-mon[33317]: 6.17 scrub ok 2024-03-11T21:31:27.326 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:31:27 smithi097 ceph-mon[33317]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-03-11T21:31:27.327 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:31:27 smithi097 ceph-mon[33317]: from='client.? v1:172.21.15.92:0/373970759' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-03-11T21:31:27.327 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:31:27 smithi097 ceph-mon[33317]: osdmap e148: 8 total, 8 up, 8 in 2024-03-11T21:31:27.360 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:31:27 smithi092 ceph-mon[37563]: 7.2 scrub starts 2024-03-11T21:31:27.360 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:31:27 smithi092 ceph-mon[37563]: 7.2 scrub ok 2024-03-11T21:31:27.360 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:31:27 smithi092 ceph-mon[37563]: 6.17 scrub starts 2024-03-11T21:31:27.361 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:31:27 smithi092 ceph-mon[37563]: 6.17 scrub ok 2024-03-11T21:31:27.361 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:31:27 smithi092 ceph-mon[37563]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-03-11T21:31:27.361 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:31:27 smithi092 ceph-mon[37563]: from='client.? v1:172.21.15.92:0/373970759' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-03-11T21:31:27.361 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:31:27 smithi092 ceph-mon[37563]: osdmap e148: 8 total, 8 up, 8 in 2024-03-11T21:31:27.361 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:31:27 smithi092 ceph-mon[29572]: 7.2 scrub starts 2024-03-11T21:31:27.361 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:31:27 smithi092 ceph-mon[29572]: 7.2 scrub ok 2024-03-11T21:31:27.361 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:31:27 smithi092 ceph-mon[29572]: 6.17 scrub starts 2024-03-11T21:31:27.361 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:31:27 smithi092 ceph-mon[29572]: 6.17 scrub ok 2024-03-11T21:31:27.361 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:31:27 smithi092 ceph-mon[29572]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-03-11T21:31:27.362 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:31:27 smithi092 ceph-mon[29572]: from='client.? v1:172.21.15.92:0/373970759' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-03-11T21:31:27.362 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:31:27 smithi092 ceph-mon[29572]: osdmap e148: 8 total, 8 up, 8 in 2024-03-11T21:31:27.384 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:31:27.413 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:31:27.413 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:31:27.414 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:31:27.446 INFO:teuthology.orchestra.run.smithi092.stderr:Inferring config /var/lib/ceph/00db0016-dfed-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T21:31:27.515 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:31:27.539 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:31:27.540 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:31:27.540 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:31:27.641 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:31:27.665 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:31:27.665 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:31:27.665 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:31:27.766 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:31:27.793 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:31:27.794 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:31:27.795 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:31:27.895 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:31:27.920 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:31:27.921 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:31:27.921 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:31:28.022 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:31:28.048 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:31:28.049 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:31:28.049 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:31:28.150 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:31:28.175 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:31:28.176 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:31:28.176 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:31:28.276 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:31:28.304 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:31:28.304 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:31:28.305 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:31:28.326 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:31:28 smithi097 ceph-mon[33317]: pgmap v217: 148 pgs: 148 active+clean; 42 MiB data, 1.1 GiB used, 714 GiB / 715 GiB avail; 639 B/s rd, 0 B/s wr, 708 op/s 2024-03-11T21:31:28.360 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:31:28 smithi092 ceph-mon[29572]: pgmap v217: 148 pgs: 148 active+clean; 42 MiB data, 1.1 GiB used, 714 GiB / 715 GiB avail; 639 B/s rd, 0 B/s wr, 708 op/s 2024-03-11T21:31:28.361 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:31:28 smithi092 ceph-mon[37563]: pgmap v217: 148 pgs: 148 active+clean; 42 MiB data, 1.1 GiB used, 714 GiB / 715 GiB avail; 639 B/s rd, 0 B/s wr, 708 op/s 2024-03-11T21:31:28.361 INFO:journalctl@ceph.mgr.y.smithi092.stdout:Mar 11 21:31:27 smithi092 ceph-00db0016-dfed-11ee-95c8-87774f69a715-mgr-y[29801]: ::ffff:172.21.15.97 - - [11/Mar/2024:21:31:27] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-11T21:31:28.406 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:31:28.432 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:31:28.433 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:31:28.433 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:31:28.534 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:31:28.559 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:31:28.559 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:31:28.560 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:31:28.661 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:31:28.689 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:31:28.689 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:31:28.690 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:31:28.791 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:31:28.817 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:31:28.818 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:31:28.818 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:31:28.919 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:31:28.946 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:31:28.947 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:31:28.947 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:31:29.048 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:31:29.073 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:31:29.073 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:31:29.074 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:31:29.174 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:31:29.202 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:31:29.203 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:31:29.203 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:31:29.304 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:31:29.325 INFO:teuthology.orchestra.run.smithi092.stdout:ERROR: (22) Invalid argument 2024-03-11T21:31:29.325 INFO:teuthology.orchestra.run.smithi092.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-11T21:31:29.331 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:31:29.331 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:31:29.332 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:31:29.433 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:31:29.458 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:31:29.458 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:31:29.458 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:31:29.559 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:31:29.583 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:31:29.583 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:31:29.583 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:31:29.684 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:31:29.712 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:31:29.712 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:31:29.713 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:31:29.814 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:31:29.838 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:31:29.838 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:31:29.839 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:31:29.939 INFO:teuthology.orchestra.run.smithi092.stdout:osd.0: {} 2024-03-11T21:31:29.940 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:31:29.961 INFO:teuthology.orchestra.run.smithi092.stdout:osd.1: {} 2024-03-11T21:31:29.967 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:31:29.968 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:31:29.968 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:31:29.979 INFO:teuthology.orchestra.run.smithi092.stdout:osd.2: {} 2024-03-11T21:31:30.000 INFO:teuthology.orchestra.run.smithi092.stdout:osd.3: {} 2024-03-11T21:31:30.019 INFO:teuthology.orchestra.run.smithi092.stdout:osd.4: {} 2024-03-11T21:31:30.040 INFO:teuthology.orchestra.run.smithi092.stdout:osd.5: {} 2024-03-11T21:31:30.061 INFO:teuthology.orchestra.run.smithi092.stdout:osd.6: {} 2024-03-11T21:31:30.069 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:31:30.081 INFO:teuthology.orchestra.run.smithi092.stdout:osd.7: {} 2024-03-11T21:31:30.082 INFO:teuthology.orchestra.run.smithi092.stderr:osd.0: osd_enable_op_tracker = '' 2024-03-11T21:31:30.082 INFO:teuthology.orchestra.run.smithi092.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-11T21:31:30.082 INFO:teuthology.orchestra.run.smithi092.stderr:osd.2: osd_enable_op_tracker = '' 2024-03-11T21:31:30.083 INFO:teuthology.orchestra.run.smithi092.stderr:osd.3: osd_enable_op_tracker = '' 2024-03-11T21:31:30.083 INFO:teuthology.orchestra.run.smithi092.stderr:osd.4: osd_enable_op_tracker = '' 2024-03-11T21:31:30.083 INFO:teuthology.orchestra.run.smithi092.stderr:osd.5: osd_enable_op_tracker = '' 2024-03-11T21:31:30.083 INFO:teuthology.orchestra.run.smithi092.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-11T21:31:30.083 INFO:teuthology.orchestra.run.smithi092.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-11T21:31:30.096 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:31:30.097 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:31:30.097 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:31:30.198 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:31:30.222 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:31:30.223 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:31:30.223 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:31:30.324 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:31:30.352 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:31:30.353 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:31:30.353 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:31:30.360 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:31:30 smithi092 ceph-mon[29572]: pgmap v218: 148 pgs: 148 active+clean; 582 KiB data, 955 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 790 op/s 2024-03-11T21:31:30.360 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:31:30 smithi092 ceph-mon[29572]: from='client.? v1:172.21.15.92:0/536617606' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-11T21:31:30.360 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:31:30 smithi092 ceph-mon[37563]: pgmap v218: 148 pgs: 148 active+clean; 582 KiB data, 955 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 790 op/s 2024-03-11T21:31:30.361 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:31:30 smithi092 ceph-mon[37563]: from='client.? v1:172.21.15.92:0/536617606' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-11T21:31:30.454 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:31:30.481 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:31:30.482 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:31:30.482 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:31:30.576 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:31:30 smithi097 ceph-mon[33317]: pgmap v218: 148 pgs: 148 active+clean; 582 KiB data, 955 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 790 op/s 2024-03-11T21:31:30.576 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:31:30 smithi097 ceph-mon[33317]: from='client.? v1:172.21.15.92:0/536617606' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-11T21:31:30.583 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:31:30.612 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:31:30.613 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:31:30.613 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:31:30.714 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:31:30.739 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:31:30.740 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:31:30.740 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:31:30.841 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:31:30.866 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:31:30.867 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:31:30.867 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:31:30.968 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:31:30.996 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:31:30.997 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:31:30.997 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:31:31.098 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:31:31.125 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:31:31.125 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:31:31.126 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:31:31.227 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:31:31.252 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:31:31.253 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:31:31.253 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:31:31.354 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:31:31.373 DEBUG:teuthology.orchestra.run.smithi092:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid 00db0016-dfed-11ee-95c8-87774f69a715 -- ceph daemon osd.2 dump_ops_in_flight 2024-03-11T21:31:31.379 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:31:31.380 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:31:31.380 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:31:31.481 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:31:31.506 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:31:31.507 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:31:31.507 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:31:31.607 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:31:31.631 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:31:31.632 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:31:31.632 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:31:31.733 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:31:31.761 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:31:31.762 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:31:31.762 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:31:31.863 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:31:31.888 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:31:31.888 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:31:31.889 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:31:31.989 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:31:32.015 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:31:32.015 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:31:32.016 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:31:32.116 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:31:32.141 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:31:32.141 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:31:32.142 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:31:32.243 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:31:32.248 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:31:32 smithi097 ceph-mon[33317]: pgmap v219: 148 pgs: 148 active+clean; 582 KiB data, 955 MiB used, 714 GiB / 715 GiB avail; 637 B/s rd, 0 B/s wr, 148 op/s 2024-03-11T21:31:32.267 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:31:32.267 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:31:32.268 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:31:32.360 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:31:32 smithi092 ceph-mon[29572]: pgmap v219: 148 pgs: 148 active+clean; 582 KiB data, 955 MiB used, 714 GiB / 715 GiB avail; 637 B/s rd, 0 B/s wr, 148 op/s 2024-03-11T21:31:32.360 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:31:32 smithi092 ceph-mon[37563]: pgmap v219: 148 pgs: 148 active+clean; 582 KiB data, 955 MiB used, 714 GiB / 715 GiB avail; 637 B/s rd, 0 B/s wr, 148 op/s 2024-03-11T21:31:32.369 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:31:32.394 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:31:32.395 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:31:32.396 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:31:32.496 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:31:32.520 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:31:32.520 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:31:32.520 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:31:32.576 INFO:journalctl@ceph.iscsi.iscsi.a.smithi097.stdout:Mar 11 21:31:32 smithi097 ceph-00db0016-dfed-11ee-95c8-87774f69a715-iscsi-iscsi-a[60943]: debug there is no tcmu-runner data available 2024-03-11T21:31:32.622 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:31:32.651 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:31:32.652 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:31:32.652 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:31:32.753 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:31:32.777 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:31:32.777 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:31:32.777 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:31:32.878 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:31:32.903 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:31:32.904 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:31:32.904 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:31:32.962 DEBUG:teuthology.orchestra.run.smithi092:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid 00db0016-dfed-11ee-95c8-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-03-11T21:31:33.005 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:31:33.030 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:31:33.034 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:31:33.035 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:31:33.136 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:31:33.159 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:31:33.160 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:31:33.161 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:31:33.261 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:31:33.290 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:31:33.290 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:31:33.291 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:31:33.392 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:31:33.419 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:31:33.420 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:31:33.420 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:31:33.521 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:31:33.547 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:31:33.547 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:31:33.547 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:31:33.576 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:31:33 smithi097 ceph-mon[33317]: from='client.24469 v1:172.21.15.97:0/404208310' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:31:33.576 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:31:33 smithi097 ceph-mon[33317]: purged_snaps scrub starts 2024-03-11T21:31:33.576 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:31:33 smithi097 ceph-mon[33317]: purged_snaps scrub ok 2024-03-11T21:31:33.610 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:31:33 smithi092 ceph-mon[37563]: from='client.24469 v1:172.21.15.97:0/404208310' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:31:33.610 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:31:33 smithi092 ceph-mon[37563]: purged_snaps scrub starts 2024-03-11T21:31:33.610 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:31:33 smithi092 ceph-mon[37563]: purged_snaps scrub ok 2024-03-11T21:31:33.611 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:31:33 smithi092 ceph-mon[29572]: from='client.24469 v1:172.21.15.97:0/404208310' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:31:33.611 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:31:33 smithi092 ceph-mon[29572]: purged_snaps scrub starts 2024-03-11T21:31:33.611 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:31:33 smithi092 ceph-mon[29572]: purged_snaps scrub ok 2024-03-11T21:31:33.648 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:31:33.678 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:31:33.678 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:31:33.678 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:31:33.779 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:31:33.806 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:31:33.807 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:31:33.807 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:31:33.827 INFO:teuthology.orchestra.run.smithi092.stderr:Inferring config /var/lib/ceph/00db0016-dfed-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T21:31:33.907 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:31:33.931 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:31:33.932 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:31:33.932 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:31:34.033 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:31:34.060 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:31:34.061 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:31:34.061 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:31:34.070 INFO:teuthology.orchestra.run.smithi092.stdout: 2024-03-11T21:31:34.072 INFO:teuthology.orchestra.run.smithi092.stderr:dumped all 2024-03-11T21:31:34.136 DEBUG:teuthology.orchestra.run.smithi092:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid 00db0016-dfed-11ee-95c8-87774f69a715 -- ceph osd set nodeep-scrub 2024-03-11T21:31:34.162 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:31:34.204 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:31:34.204 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:31:34.205 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:31:34.305 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:31:34.333 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:31:34.334 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:31:34.334 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:31:34.360 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:31:34 smithi092 ceph-mon[29572]: purged_snaps scrub starts 2024-03-11T21:31:34.360 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:31:34 smithi092 ceph-mon[29572]: purged_snaps scrub ok 2024-03-11T21:31:34.360 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:31:34 smithi092 ceph-mon[29572]: pgmap v220: 148 pgs: 148 active+clean; 582 KiB data, 955 MiB used, 714 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 119 op/s 2024-03-11T21:31:34.361 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:31:34 smithi092 ceph-mon[29572]: osdmap e149: 8 total, 8 up, 8 in 2024-03-11T21:31:34.361 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:31:34 smithi092 ceph-mon[37563]: purged_snaps scrub starts 2024-03-11T21:31:34.361 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:31:34 smithi092 ceph-mon[37563]: purged_snaps scrub ok 2024-03-11T21:31:34.361 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:31:34 smithi092 ceph-mon[37563]: pgmap v220: 148 pgs: 148 active+clean; 582 KiB data, 955 MiB used, 714 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 119 op/s 2024-03-11T21:31:34.361 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:31:34 smithi092 ceph-mon[37563]: osdmap e149: 8 total, 8 up, 8 in 2024-03-11T21:31:34.435 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:31:34.467 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:31:34.467 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:31:34.467 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:31:34.569 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:31:34.576 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:31:34 smithi097 ceph-mon[33317]: purged_snaps scrub starts 2024-03-11T21:31:34.576 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:31:34 smithi097 ceph-mon[33317]: purged_snaps scrub ok 2024-03-11T21:31:34.576 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:31:34 smithi097 ceph-mon[33317]: pgmap v220: 148 pgs: 148 active+clean; 582 KiB data, 955 MiB used, 714 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 119 op/s 2024-03-11T21:31:34.576 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:31:34 smithi097 ceph-mon[33317]: osdmap e149: 8 total, 8 up, 8 in 2024-03-11T21:31:34.597 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:31:34.598 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:31:34.598 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:31:34.699 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:31:34.723 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:31:34.724 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:31:34.725 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:31:34.825 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:31:34.853 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:31:34.854 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:31:34.854 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:31:34.955 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:31:34.981 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:31:34.982 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:31:34.982 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:31:35.084 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:31:35.110 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:31:35.111 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:31:35.111 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:31:35.212 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:31:35.236 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:31:35.237 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:31:35.237 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:31:35.338 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:31:35.360 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:31:35 smithi092 ceph-mon[29572]: from='client.15093 v1:172.21.15.92:0/2034970522' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-11T21:31:35.360 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:31:35 smithi092 ceph-mon[37563]: from='client.15093 v1:172.21.15.92:0/2034970522' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-11T21:31:35.361 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:31:35.362 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:31:35.362 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:31:35.463 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:31:35.487 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:31:35.488 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:31:35.488 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:31:35.576 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:31:35 smithi097 ceph-mon[33317]: from='client.15093 v1:172.21.15.92:0/2034970522' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-11T21:31:35.588 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:31:35.613 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:31:35.613 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:31:35.613 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:31:35.661 INFO:teuthology.orchestra.run.smithi092.stdout:{"pg_ready":true,"pg_map":{"version":221,"stamp":"2024-03-11T21:31:33.217193+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":596392,"num_objects":198,"num_object_clones":0,"num_object_copies":594,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":198,"num_whiteouts":0,"num_read":1282,"num_read_kb":1128,"num_write":244433,"num_write_kb":7804574,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":21,"num_bytes_recovered":1771324,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":90049,"ondisk_log_size":90049,"up":444,"acting":444,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":444,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":749961216,"kb_used":978044,"kb_used_data":8092,"kb_used_omap":12,"kb_used_meta":969843,"kb_avail":748983172,"statfs":{"total":767960285184,"available":766958768128,"internally_reserved":0,"allocated":8286208,"data_stored":5089904,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12752,"internal_metadata":993119792},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-77594648,"num_objects":-1185,"num_object_clones":0,"num_object_copies":-3555,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-1185,"num_whiteouts":0,"num_read":10,"num_read_kb":10,"num_write":1185,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.137229"},"pg_stats":[{"pgid":"6.1b","version":"0'0","reported_seq":186,"reported_epoch":148,"state":"active+clean","last_fresh":"2024-03-11T21:31:26.088949+0000","last_change":"2024-03-11T21:31:06.310603+0000","last_active":"2024-03-11T21:31:26.088949+0000","last_peered":"2024-03-11T21:31:26.088949+0000","last_clean":"2024-03-11T21:31:26.088949+0000","last_became_active":"2024-03-11T21:25:45.302931+0000","last_became_peered":"2024-03-11T21:25:45.302931+0000","last_unstale":"2024-03-11T21:31:26.088949+0000","last_undegraded":"2024-03-11T21:31:26.088949+0000","last_fullsized":"2024-03-11T21:31:26.088949+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:31:06.310490+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:43.559452+0000","last_clean_scrub_stamp":"2024-03-11T21:31:06.310490+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:32:16.162571+0000","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":191,"reported_epoch":148,"state":"active+clean","last_fresh":"2024-03-11T21:31:26.083096+0000","last_change":"2024-03-11T21:31:10.598620+0000","last_active":"2024-03-11T21:31:26.083096+0000","last_peered":"2024-03-11T21:31:26.083096+0000","last_clean":"2024-03-11T21:31:26.083096+0000","last_became_active":"2024-03-11T21:25:42.578893+0000","last_became_peered":"2024-03-11T21:25:42.578893+0000","last_unstale":"2024-03-11T21:31:26.083096+0000","last_undegraded":"2024-03-11T21:31:26.083096+0000","last_fullsized":"2024-03-11T21:31:26.083096+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:31:10.598515+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:41.544550+0000","last_clean_scrub_stamp":"2024-03-11T21:31:10.598515+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:32:32.512946+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1f","version":"0'0","reported_seq":199,"reported_epoch":148,"state":"active+clean","last_fresh":"2024-03-11T21:31:26.086685+0000","last_change":"2024-03-11T21:30:14.771032+0000","last_active":"2024-03-11T21:31:26.086685+0000","last_peered":"2024-03-11T21:31:26.086685+0000","last_clean":"2024-03-11T21:31:26.086685+0000","last_became_active":"2024-03-11T21:25:38.540510+0000","last_became_peered":"2024-03-11T21:25:38.540510+0000","last_unstale":"2024-03-11T21:31:26.086685+0000","last_undegraded":"2024-03-11T21:31:26.086685+0000","last_fullsized":"2024-03-11T21:31:26.086685+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:30:14.770920+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:37.512018+0000","last_clean_scrub_stamp":"2024-03-11T21:30:14.770920+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:31:30.680425+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,4],"acting":[0,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.19","version":"74'15","reported_seq":219,"reported_epoch":148,"state":"active+clean","last_fresh":"2024-03-11T21:31:26.087278+0000","last_change":"2024-03-11T21:31:07.311590+0000","last_active":"2024-03-11T21:31:26.087278+0000","last_peered":"2024-03-11T21:31:26.087278+0000","last_clean":"2024-03-11T21:31:26.087278+0000","last_became_active":"2024-03-11T21:25:40.554207+0000","last_became_peered":"2024-03-11T21:25:40.554207+0000","last_unstale":"2024-03-11T21:31:26.087278+0000","last_undegraded":"2024-03-11T21:31:26.087278+0000","last_fullsized":"2024-03-11T21:31:26.087278+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'15","last_scrub_stamp":"2024-03-11T21:31:07.311470+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:39.525554+0000","last_clean_scrub_stamp":"2024-03-11T21:31:07.311470+0000","objects_scrubbed":7,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:32:36.263715+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.1a","version":"0'0","reported_seq":179,"reported_epoch":148,"state":"active+clean","last_fresh":"2024-03-11T21:31:26.083446+0000","last_change":"2024-03-11T21:29:41.255464+0000","last_active":"2024-03-11T21:31:26.083446+0000","last_peered":"2024-03-11T21:31:26.083446+0000","last_clean":"2024-03-11T21:31:26.083446+0000","last_became_active":"2024-03-11T21:25:44.590923+0000","last_became_peered":"2024-03-11T21:25:44.590923+0000","last_unstale":"2024-03-11T21:31:26.083446+0000","last_undegraded":"2024-03-11T21:31:26.083446+0000","last_fullsized":"2024-03-11T21:31:26.083446+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:29:41.255337+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:43.559452+0000","last_clean_scrub_stamp":"2024-03-11T21:29:41.255337+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,1],"acting":[4,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":191,"reported_epoch":148,"state":"active+clean","last_fresh":"2024-03-11T21:31:26.531127+0000","last_change":"2024-03-11T21:31:15.812546+0000","last_active":"2024-03-11T21:31:26.531127+0000","last_peered":"2024-03-11T21:31:26.531127+0000","last_clean":"2024-03-11T21:31:26.531127+0000","last_became_active":"2024-03-11T21:25:42.578424+0000","last_became_peered":"2024-03-11T21:25:42.578424+0000","last_unstale":"2024-03-11T21:31:26.531127+0000","last_undegraded":"2024-03-11T21:31:26.531127+0000","last_fullsized":"2024-03-11T21:31:26.531127+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:31:15.812251+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:41.544550+0000","last_clean_scrub_stamp":"2024-03-11T21:31:15.812251+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:32:21.508207+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,7],"acting":[1,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1e","version":"0'0","reported_seq":198,"reported_epoch":148,"state":"active+clean","last_fresh":"2024-03-11T21:31:26.087766+0000","last_change":"2024-03-11T21:30:17.913666+0000","last_active":"2024-03-11T21:31:26.087766+0000","last_peered":"2024-03-11T21:31:26.087766+0000","last_clean":"2024-03-11T21:31:26.087766+0000","last_became_active":"2024-03-11T21:25:38.537283+0000","last_became_peered":"2024-03-11T21:25:38.537283+0000","last_unstale":"2024-03-11T21:31:26.087766+0000","last_undegraded":"2024-03-11T21:31:26.087766+0000","last_fullsized":"2024-03-11T21:31:26.087766+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:30:17.913550+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:37.512018+0000","last_clean_scrub_stamp":"2024-03-11T21:30:17.913550+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:31:38.998596+0000","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.18","version":"74'9","reported_seq":211,"reported_epoch":148,"state":"active+clean","last_fresh":"2024-03-11T21:31:26.083281+0000","last_change":"2024-03-11T21:29:45.187402+0000","last_active":"2024-03-11T21:31:26.083281+0000","last_peered":"2024-03-11T21:31:26.083281+0000","last_clean":"2024-03-11T21:31:26.083281+0000","last_became_active":"2024-03-11T21:25:40.552216+0000","last_became_peered":"2024-03-11T21:25:40.552216+0000","last_unstale":"2024-03-11T21:31:26.083281+0000","last_undegraded":"2024-03-11T21:31:26.083281+0000","last_fullsized":"2024-03-11T21:31:26.083281+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'9","last_scrub_stamp":"2024-03-11T21:29:45.187250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:39.525554+0000","last_clean_scrub_stamp":"2024-03-11T21:29:45.187250+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":186,"reported_epoch":148,"state":"active+clean","last_fresh":"2024-03-11T21:31:26.531356+0000","last_change":"2024-03-11T21:29:47.008124+0000","last_active":"2024-03-11T21:31:26.531356+0000","last_peered":"2024-03-11T21:31:26.531356+0000","last_clean":"2024-03-11T21:31:26.531356+0000","last_became_active":"2024-03-11T21:25:44.590234+0000","last_became_peered":"2024-03-11T21:25:44.590234+0000","last_unstale":"2024-03-11T21:31:26.531356+0000","last_undegraded":"2024-03-11T21:31:26.531356+0000","last_fullsized":"2024-03-11T21:31:26.531356+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:29:47.008018+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:43.559452+0000","last_clean_scrub_stamp":"2024-03-11T21:29:47.008018+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":133,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3],"acting":[5,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":182,"reported_epoch":148,"state":"active+clean","last_fresh":"2024-03-11T21:31:26.530822+0000","last_change":"2024-03-11T21:29:21.533307+0000","last_active":"2024-03-11T21:31:26.530822+0000","last_peered":"2024-03-11T21:31:26.530822+0000","last_clean":"2024-03-11T21:31:26.530822+0000","last_became_active":"2024-03-11T21:25:42.580533+0000","last_became_peered":"2024-03-11T21:25:42.580533+0000","last_unstale":"2024-03-11T21:31:26.530822+0000","last_undegraded":"2024-03-11T21:31:26.530822+0000","last_fullsized":"2024-03-11T21:31:26.530822+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:29:21.533216+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:41.544550+0000","last_clean_scrub_stamp":"2024-03-11T21:29:21.533216+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.1d","version":"0'0","reported_seq":198,"reported_epoch":148,"state":"active+clean","last_fresh":"2024-03-11T21:31:26.531649+0000","last_change":"2024-03-11T21:29:39.481465+0000","last_active":"2024-03-11T21:31:26.531649+0000","last_peered":"2024-03-11T21:31:26.531649+0000","last_clean":"2024-03-11T21:31:26.531649+0000","last_became_active":"2024-03-11T21:25:38.542628+0000","last_became_peered":"2024-03-11T21:25:38.542628+0000","last_unstale":"2024-03-11T21:31:26.531649+0000","last_undegraded":"2024-03-11T21:31:26.531649+0000","last_fullsized":"2024-03-11T21:31:26.531649+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:29:39.481382+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:37.512018+0000","last_clean_scrub_stamp":"2024-03-11T21:29:39.481382+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,0],"acting":[7,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.1b","version":"74'5","reported_seq":197,"reported_epoch":148,"state":"active+clean","last_fresh":"2024-03-11T21:31:26.083646+0000","last_change":"2024-03-11T21:29:40.303599+0000","last_active":"2024-03-11T21:31:26.083646+0000","last_peered":"2024-03-11T21:31:26.083646+0000","last_clean":"2024-03-11T21:31:26.083646+0000","last_became_active":"2024-03-11T21:25:40.555826+0000","last_became_peered":"2024-03-11T21:25:40.555826+0000","last_unstale":"2024-03-11T21:31:26.083646+0000","last_undegraded":"2024-03-11T21:31:26.083646+0000","last_fullsized":"2024-03-11T21:31:26.083646+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'5","last_scrub_stamp":"2024-03-11T21:29:40.303460+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:39.525554+0000","last_clean_scrub_stamp":"2024-03-11T21:29:40.303460+0000","objects_scrubbed":2,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":11,"num_read_kb":7,"num_write":6,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,1],"acting":[4,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":179,"reported_epoch":148,"state":"active+clean","last_fresh":"2024-03-11T21:31:26.086824+0000","last_change":"2024-03-11T21:29:43.509900+0000","last_active":"2024-03-11T21:31:26.086824+0000","last_peered":"2024-03-11T21:31:26.086824+0000","last_clean":"2024-03-11T21:31:26.086824+0000","last_became_active":"2024-03-11T21:25:44.589786+0000","last_became_peered":"2024-03-11T21:25:44.589786+0000","last_unstale":"2024-03-11T21:31:26.086824+0000","last_undegraded":"2024-03-11T21:31:26.086824+0000","last_fullsized":"2024-03-11T21:31:26.086824+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:29:43.509792+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:43.559452+0000","last_clean_scrub_stamp":"2024-03-11T21:29:43.509792+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,7],"acting":[0,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":190,"reported_epoch":148,"state":"active+clean","last_fresh":"2024-03-11T21:31:26.532831+0000","last_change":"2024-03-11T21:31:00.533483+0000","last_active":"2024-03-11T21:31:26.532831+0000","last_peered":"2024-03-11T21:31:26.532831+0000","last_clean":"2024-03-11T21:31:26.532831+0000","last_became_active":"2024-03-11T21:25:42.573831+0000","last_became_peered":"2024-03-11T21:25:42.573831+0000","last_unstale":"2024-03-11T21:31:26.532831+0000","last_undegraded":"2024-03-11T21:31:26.532831+0000","last_fullsized":"2024-03-11T21:31:26.532831+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:31:00.533376+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:41.544550+0000","last_clean_scrub_stamp":"2024-03-11T21:31:00.533376+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:32:04.433484+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,7],"acting":[5,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1c","version":"0'0","reported_seq":198,"reported_epoch":148,"state":"active+clean","last_fresh":"2024-03-11T21:31:26.532116+0000","last_change":"2024-03-11T21:30:11.669644+0000","last_active":"2024-03-11T21:31:26.532116+0000","last_peered":"2024-03-11T21:31:26.532116+0000","last_clean":"2024-03-11T21:31:26.532116+0000","last_became_active":"2024-03-11T21:25:38.541364+0000","last_became_peered":"2024-03-11T21:25:38.541364+0000","last_unstale":"2024-03-11T21:31:26.532116+0000","last_undegraded":"2024-03-11T21:31:26.532116+0000","last_fullsized":"2024-03-11T21:31:26.532116+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:30:11.668879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:28:32.566437+0000","last_clean_scrub_stamp":"2024-03-11T21:30:11.668879+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.1a","version":"74'9","reported_seq":211,"reported_epoch":148,"state":"active+clean","last_fresh":"2024-03-11T21:31:26.083582+0000","last_change":"2024-03-11T21:29:44.227568+0000","last_active":"2024-03-11T21:31:26.083582+0000","last_peered":"2024-03-11T21:31:26.083582+0000","last_clean":"2024-03-11T21:31:26.083582+0000","last_became_active":"2024-03-11T21:25:40.556629+0000","last_became_peered":"2024-03-11T21:25:40.556629+0000","last_unstale":"2024-03-11T21:31:26.083582+0000","last_undegraded":"2024-03-11T21:31:26.083582+0000","last_fullsized":"2024-03-11T21:31:26.083582+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'9","last_scrub_stamp":"2024-03-11T21:29:44.227421+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:39.525554+0000","last_clean_scrub_stamp":"2024-03-11T21:29:44.227421+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0],"acting":[4,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1f","version":"0'0","reported_seq":186,"reported_epoch":148,"state":"active+clean","last_fresh":"2024-03-11T21:31:26.087673+0000","last_change":"2024-03-11T21:31:04.253653+0000","last_active":"2024-03-11T21:31:26.087673+0000","last_peered":"2024-03-11T21:31:26.087673+0000","last_clean":"2024-03-11T21:31:26.087673+0000","last_became_active":"2024-03-11T21:25:45.302637+0000","last_became_peered":"2024-03-11T21:25:45.302637+0000","last_unstale":"2024-03-11T21:31:26.087673+0000","last_undegraded":"2024-03-11T21:31:26.087673+0000","last_fullsized":"2024-03-11T21:31:26.087673+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:31:04.253137+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:43.559452+0000","last_clean_scrub_stamp":"2024-03-11T21:31:04.253137+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:32:05.471991+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1c","version":"0'0","reported_seq":191,"reported_epoch":148,"state":"active+clean","last_fresh":"2024-03-11T21:31:26.083902+0000","last_change":"2024-03-11T21:30:59.496458+0000","last_active":"2024-03-11T21:31:26.083902+0000","last_peered":"2024-03-11T21:31:26.083902+0000","last_clean":"2024-03-11T21:31:26.083902+0000","last_became_active":"2024-03-11T21:25:42.576064+0000","last_became_peered":"2024-03-11T21:25:42.576064+0000","last_unstale":"2024-03-11T21:31:26.083902+0000","last_undegraded":"2024-03-11T21:31:26.083902+0000","last_fullsized":"2024-03-11T21:31:26.083902+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:30:59.496330+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:41.544550+0000","last_clean_scrub_stamp":"2024-03-11T21:30:59.496330+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:32:27.314836+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1b","version":"0'0","reported_seq":198,"reported_epoch":148,"state":"active+clean","last_fresh":"2024-03-11T21:31:26.088384+0000","last_change":"2024-03-11T21:31:05.273080+0000","last_active":"2024-03-11T21:31:26.088384+0000","last_peered":"2024-03-11T21:31:26.088384+0000","last_clean":"2024-03-11T21:31:26.088384+0000","last_became_active":"2024-03-11T21:25:38.537520+0000","last_became_peered":"2024-03-11T21:25:38.537520+0000","last_unstale":"2024-03-11T21:31:26.088384+0000","last_undegraded":"2024-03-11T21:31:26.088384+0000","last_fullsized":"2024-03-11T21:31:26.088384+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:31:05.272969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:28:33.582712+0000","last_clean_scrub_stamp":"2024-03-11T21:31:05.272969+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:32:11.268801+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1d","version":"74'12","reported_seq":217,"reported_epoch":148,"state":"active+clean","last_fresh":"2024-03-11T21:31:26.087430+0000","last_change":"2024-03-11T21:31:20.514563+0000","last_active":"2024-03-11T21:31:26.087430+0000","last_peered":"2024-03-11T21:31:26.087430+0000","last_clean":"2024-03-11T21:31:26.087430+0000","last_became_active":"2024-03-11T21:25:40.556330+0000","last_became_peered":"2024-03-11T21:25:40.556330+0000","last_unstale":"2024-03-11T21:31:26.087430+0000","last_undegraded":"2024-03-11T21:31:26.087430+0000","last_fullsized":"2024-03-11T21:31:26.087430+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'12","last_scrub_stamp":"2024-03-11T21:31:20.514423+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:39.525554+0000","last_clean_scrub_stamp":"2024-03-11T21:31:20.514423+0000","objects_scrubbed":5,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:32:20.604067+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":25,"num_read_kb":16,"num_write":14,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.1e","version":"0'0","reported_seq":179,"reported_epoch":148,"state":"active+clean","last_fresh":"2024-03-11T21:31:26.084335+0000","last_change":"2024-03-11T21:30:11.322364+0000","last_active":"2024-03-11T21:31:26.084335+0000","last_peered":"2024-03-11T21:31:26.084335+0000","last_clean":"2024-03-11T21:31:26.084335+0000","last_became_active":"2024-03-11T21:25:45.300355+0000","last_became_peered":"2024-03-11T21:25:45.300355+0000","last_unstale":"2024-03-11T21:31:26.084335+0000","last_undegraded":"2024-03-11T21:31:26.084335+0000","last_fullsized":"2024-03-11T21:31:26.084335+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:30:11.322178+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:43.559452+0000","last_clean_scrub_stamp":"2024-03-11T21:30:11.322178+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:31:39.815208+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5],"acting":[4,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1d","version":"0'0","reported_seq":191,"reported_epoch":148,"state":"active+clean","last_fresh":"2024-03-11T21:31:26.531421+0000","last_change":"2024-03-11T21:30:18.678642+0000","last_active":"2024-03-11T21:31:26.531421+0000","last_peered":"2024-03-11T21:31:26.531421+0000","last_clean":"2024-03-11T21:31:26.531421+0000","last_became_active":"2024-03-11T21:25:42.579407+0000","last_became_peered":"2024-03-11T21:25:42.579407+0000","last_unstale":"2024-03-11T21:31:26.531421+0000","last_undegraded":"2024-03-11T21:31:26.531421+0000","last_fullsized":"2024-03-11T21:31:26.531421+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:30:18.678531+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:30:18.678531+0000","last_clean_scrub_stamp":"2024-03-11T21:30:18.678531+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1a","version":"0'0","reported_seq":199,"reported_epoch":148,"state":"active+clean","last_fresh":"2024-03-11T21:31:26.531179+0000","last_change":"2024-03-11T21:30:11.816854+0000","last_active":"2024-03-11T21:31:26.531179+0000","last_peered":"2024-03-11T21:31:26.531179+0000","last_clean":"2024-03-11T21:31:26.531179+0000","last_became_active":"2024-03-11T21:25:38.539725+0000","last_became_peered":"2024-03-11T21:25:38.539725+0000","last_unstale":"2024-03-11T21:31:26.531179+0000","last_undegraded":"2024-03-11T21:31:26.531179+0000","last_fullsized":"2024-03-11T21:31:26.531179+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:30:11.816720+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:37.512018+0000","last_clean_scrub_stamp":"2024-03-11T21:30:11.816720+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:31:40.935055+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.1c","version":"74'15","reported_seq":219,"reported_epoch":148,"state":"active+clean","last_fresh":"2024-03-11T21:31:26.530384+0000","last_change":"2024-03-11T21:31:01.272353+0000","last_active":"2024-03-11T21:31:26.530384+0000","last_peered":"2024-03-11T21:31:26.530384+0000","last_clean":"2024-03-11T21:31:26.530384+0000","last_became_active":"2024-03-11T21:25:40.551453+0000","last_became_peered":"2024-03-11T21:25:40.551453+0000","last_unstale":"2024-03-11T21:31:26.530384+0000","last_undegraded":"2024-03-11T21:31:26.530384+0000","last_fullsized":"2024-03-11T21:31:26.530384+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'15","last_scrub_stamp":"2024-03-11T21:31:01.272189+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:39.525554+0000","last_clean_scrub_stamp":"2024-03-11T21:31:01.272189+0000","objects_scrubbed":7,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:32:24.109315+0000","scrub_duration":22,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.1d","version":"0'0","reported_seq":187,"reported_epoch":148,"state":"active+clean","last_fresh":"2024-03-11T21:31:26.531142+0000","last_change":"2024-03-11T21:31:18.798813+0000","last_active":"2024-03-11T21:31:26.531142+0000","last_peered":"2024-03-11T21:31:26.531142+0000","last_clean":"2024-03-11T21:31:26.531142+0000","last_became_active":"2024-03-11T21:25:44.588845+0000","last_became_peered":"2024-03-11T21:25:44.588845+0000","last_unstale":"2024-03-11T21:31:26.531142+0000","last_undegraded":"2024-03-11T21:31:26.531142+0000","last_fullsized":"2024-03-11T21:31:26.531142+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:31:18.798701+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:43.559452+0000","last_clean_scrub_stamp":"2024-03-11T21:31:18.798701+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:32:47.454620+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.1e","version":"0'0","reported_seq":191,"reported_epoch":148,"state":"active+clean","last_fresh":"2024-03-11T21:31:26.087712+0000","last_change":"2024-03-11T21:30:26.111125+0000","last_active":"2024-03-11T21:31:26.087712+0000","last_peered":"2024-03-11T21:31:26.087712+0000","last_clean":"2024-03-11T21:31:26.087712+0000","last_became_active":"2024-03-11T21:25:42.582959+0000","last_became_peered":"2024-03-11T21:25:42.582959+0000","last_unstale":"2024-03-11T21:31:26.087712+0000","last_undegraded":"2024-03-11T21:31:26.087712+0000","last_fullsized":"2024-03-11T21:31:26.087712+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:30:26.110855+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:26:58.335431+0000","last_clean_scrub_stamp":"2024-03-11T21:30:26.110855+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:31:36.208967+0000","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.19","version":"67'2","reported_seq":197,"reported_epoch":148,"state":"active+clean","last_fresh":"2024-03-11T21:31:26.087430+0000","last_change":"2024-03-11T21:29:42.815863+0000","last_active":"2024-03-11T21:31:26.087430+0000","last_peered":"2024-03-11T21:31:26.087430+0000","last_clean":"2024-03-11T21:31:26.087430+0000","last_became_active":"2024-03-11T21:25:38.539243+0000","last_became_peered":"2024-03-11T21:25:38.539243+0000","last_unstale":"2024-03-11T21:31:26.087430+0000","last_undegraded":"2024-03-11T21:31:26.087430+0000","last_fullsized":"2024-03-11T21:31:26.087430+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'2","last_scrub_stamp":"2024-03-11T21:29:42.815740+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:37.512018+0000","last_clean_scrub_stamp":"2024-03-11T21:29:42.815740+0000","objects_scrubbed":2,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":957,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":7,"num_read_kb":7,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,0],"acting":[3,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1f","version":"124'16","reported_seq":227,"reported_epoch":148,"state":"active+clean","last_fresh":"2024-03-11T21:31:26.532373+0000","last_change":"2024-03-11T21:31:06.449601+0000","last_active":"2024-03-11T21:31:26.532373+0000","last_peered":"2024-03-11T21:31:26.532373+0000","last_clean":"2024-03-11T21:31:26.532373+0000","last_became_active":"2024-03-11T21:25:40.554686+0000","last_became_peered":"2024-03-11T21:25:40.554686+0000","last_unstale":"2024-03-11T21:31:26.532373+0000","last_undegraded":"2024-03-11T21:31:26.532373+0000","last_fullsized":"2024-03-11T21:31:26.532373+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"124'16","last_scrub_stamp":"2024-03-11T21:31:06.449509+0000","last_deep_scrub":"124'16","last_deep_scrub_stamp":"2024-03-11T21:29:30.873490+0000","last_clean_scrub_stamp":"2024-03-11T21:31:06.449509+0000","objects_scrubbed":5,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:32:20.104923+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,1],"acting":[6,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.1c","version":"74'1","reported_seq":187,"reported_epoch":148,"state":"active+clean","last_fresh":"2024-03-11T21:31:26.532522+0000","last_change":"2024-03-11T21:31:07.489348+0000","last_active":"2024-03-11T21:31:26.532522+0000","last_peered":"2024-03-11T21:31:26.532522+0000","last_clean":"2024-03-11T21:31:26.532522+0000","last_became_active":"2024-03-11T21:25:44.593326+0000","last_became_peered":"2024-03-11T21:25:44.593326+0000","last_unstale":"2024-03-11T21:31:26.532522+0000","last_undegraded":"2024-03-11T21:31:26.532522+0000","last_fullsized":"2024-03-11T21:31:26.532522+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'1","last_scrub_stamp":"2024-03-11T21:31:07.489215+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:43.559452+0000","last_clean_scrub_stamp":"2024-03-11T21:31:07.489215+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:32:24.472953+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":370,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.1f","version":"74'8","reported_seq":281,"reported_epoch":148,"state":"active+clean","last_fresh":"2024-03-11T21:31:26.532317+0000","last_change":"2024-03-11T21:31:02.499201+0000","last_active":"2024-03-11T21:31:26.532317+0000","last_peered":"2024-03-11T21:31:26.532317+0000","last_clean":"2024-03-11T21:31:26.532317+0000","last_became_active":"2024-03-11T21:25:42.577401+0000","last_became_peered":"2024-03-11T21:25:42.577401+0000","last_unstale":"2024-03-11T21:31:26.532317+0000","last_undegraded":"2024-03-11T21:31:26.532317+0000","last_fullsized":"2024-03-11T21:31:26.532317+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'8","last_scrub_stamp":"2024-03-11T21:31:02.499061+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:41.544550+0000","last_clean_scrub_stamp":"2024-03-11T21:31:02.499061+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:32:20.379279+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.18","version":"67'1","reported_seq":204,"reported_epoch":148,"state":"active+clean","last_fresh":"2024-03-11T21:31:26.532170+0000","last_change":"2024-03-11T21:29:35.918627+0000","last_active":"2024-03-11T21:31:26.532170+0000","last_peered":"2024-03-11T21:31:26.532170+0000","last_clean":"2024-03-11T21:31:26.532170+0000","last_became_active":"2024-03-11T21:25:38.538321+0000","last_became_peered":"2024-03-11T21:25:38.538321+0000","last_unstale":"2024-03-11T21:31:26.532170+0000","last_undegraded":"2024-03-11T21:31:26.532170+0000","last_fullsized":"2024-03-11T21:31:26.532170+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'1","last_scrub_stamp":"2024-03-11T21:29:35.918494+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:37.512018+0000","last_clean_scrub_stamp":"2024-03-11T21:29:35.918494+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":436,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":7,"num_read_kb":7,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,7],"acting":[5,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1e","version":"74'10","reported_seq":210,"reported_epoch":148,"state":"active+clean","last_fresh":"2024-03-11T21:31:26.087617+0000","last_change":"2024-03-11T21:30:57.371847+0000","last_active":"2024-03-11T21:31:26.087617+0000","last_peered":"2024-03-11T21:31:26.087617+0000","last_clean":"2024-03-11T21:31:26.087617+0000","last_became_active":"2024-03-11T21:25:40.763438+0000","last_became_peered":"2024-03-11T21:25:40.763438+0000","last_unstale":"2024-03-11T21:31:26.087617+0000","last_undegraded":"2024-03-11T21:31:26.087617+0000","last_fullsized":"2024-03-11T21:31:26.087617+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'10","last_scrub_stamp":"2024-03-11T21:30:57.371717+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:39.525554+0000","last_clean_scrub_stamp":"2024-03-11T21:30:57.371717+0000","objects_scrubbed":5,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:32:03.436916+0000","scrub_duration":27,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.13","version":"0'0","reported_seq":186,"reported_epoch":148,"state":"active+clean","last_fresh":"2024-03-11T21:31:26.088750+0000","last_change":"2024-03-11T21:31:00.239533+0000","last_active":"2024-03-11T21:31:26.088750+0000","last_peered":"2024-03-11T21:31:26.088750+0000","last_clean":"2024-03-11T21:31:26.088750+0000","last_became_active":"2024-03-11T21:25:45.302102+0000","last_became_peered":"2024-03-11T21:25:45.302102+0000","last_unstale":"2024-03-11T21:31:26.088750+0000","last_undegraded":"2024-03-11T21:31:26.088750+0000","last_fullsized":"2024-03-11T21:31:26.088750+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:31:00.238325+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:43.559452+0000","last_clean_scrub_stamp":"2024-03-11T21:31:00.238325+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:32:15.077313+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":190,"reported_epoch":148,"state":"active+clean","last_fresh":"2024-03-11T21:31:26.532332+0000","last_change":"2024-03-11T21:29:47.630346+0000","last_active":"2024-03-11T21:31:26.532332+0000","last_peered":"2024-03-11T21:31:26.532332+0000","last_clean":"2024-03-11T21:31:26.532332+0000","last_became_active":"2024-03-11T21:25:42.579781+0000","last_became_peered":"2024-03-11T21:25:42.579781+0000","last_unstale":"2024-03-11T21:31:26.532332+0000","last_undegraded":"2024-03-11T21:31:26.532332+0000","last_fullsized":"2024-03-11T21:31:26.532332+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:29:47.630271+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:41.544550+0000","last_clean_scrub_stamp":"2024-03-11T21:29:47.630271+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.17","version":"0'0","reported_seq":199,"reported_epoch":148,"state":"active+clean","last_fresh":"2024-03-11T21:31:26.532371+0000","last_change":"2024-03-11T21:30:12.782541+0000","last_active":"2024-03-11T21:31:26.532371+0000","last_peered":"2024-03-11T21:31:26.532371+0000","last_clean":"2024-03-11T21:31:26.532371+0000","last_became_active":"2024-03-11T21:25:38.540349+0000","last_became_peered":"2024-03-11T21:25:38.540349+0000","last_unstale":"2024-03-11T21:31:26.532371+0000","last_undegraded":"2024-03-11T21:31:26.532371+0000","last_fullsized":"2024-03-11T21:31:26.532371+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:30:12.782413+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:28:25.918101+0000","last_clean_scrub_stamp":"2024-03-11T21:30:12.782413+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:31:35.677574+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,2],"acting":[6,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.11","version":"74'11","reported_seq":213,"reported_epoch":148,"state":"active+clean","last_fresh":"2024-03-11T21:31:26.088582+0000","last_change":"2024-03-11T21:31:17.450254+0000","last_active":"2024-03-11T21:31:26.088582+0000","last_peered":"2024-03-11T21:31:26.088582+0000","last_clean":"2024-03-11T21:31:26.088582+0000","last_became_active":"2024-03-11T21:25:40.760664+0000","last_became_peered":"2024-03-11T21:25:40.760664+0000","last_unstale":"2024-03-11T21:31:26.088582+0000","last_undegraded":"2024-03-11T21:31:26.088582+0000","last_fullsized":"2024-03-11T21:31:26.088582+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'11","last_scrub_stamp":"2024-03-11T21:31:17.450051+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:39.525554+0000","last_clean_scrub_stamp":"2024-03-11T21:31:17.450051+0000","objects_scrubbed":5,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:32:37.763712+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":186,"reported_epoch":148,"state":"active+clean","last_fresh":"2024-03-11T21:31:26.530632+0000","last_change":"2024-03-11T21:29:41.454129+0000","last_active":"2024-03-11T21:31:26.530632+0000","last_peered":"2024-03-11T21:31:26.530632+0000","last_clean":"2024-03-11T21:31:26.530632+0000","last_became_active":"2024-03-11T21:25:44.591068+0000","last_became_peered":"2024-03-11T21:25:44.591068+0000","last_unstale":"2024-03-11T21:31:26.530632+0000","last_undegraded":"2024-03-11T21:31:26.530632+0000","last_fullsized":"2024-03-11T21:31:26.530632+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:29:41.454046+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:43.559452+0000","last_clean_scrub_stamp":"2024-03-11T21:29:41.454046+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,4],"acting":[7,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":199,"reported_epoch":148,"state":"active+clean","last_fresh":"2024-03-11T21:31:26.530726+0000","last_change":"2024-03-11T21:31:12.423773+0000","last_active":"2024-03-11T21:31:26.530726+0000","last_peered":"2024-03-11T21:31:26.530726+0000","last_clean":"2024-03-11T21:31:26.530726+0000","last_became_active":"2024-03-11T21:25:42.584148+0000","last_became_peered":"2024-03-11T21:25:42.584148+0000","last_unstale":"2024-03-11T21:31:26.530726+0000","last_undegraded":"2024-03-11T21:31:26.530726+0000","last_fullsized":"2024-03-11T21:31:26.530726+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:31:12.423671+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:29:39.923895+0000","last_clean_scrub_stamp":"2024-03-11T21:31:12.423671+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:32:18.156840+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.16","version":"0'0","reported_seq":206,"reported_epoch":148,"state":"active+clean","last_fresh":"2024-03-11T21:31:26.532585+0000","last_change":"2024-03-11T21:30:59.528390+0000","last_active":"2024-03-11T21:31:26.532585+0000","last_peered":"2024-03-11T21:31:26.532585+0000","last_clean":"2024-03-11T21:31:26.532585+0000","last_became_active":"2024-03-11T21:25:38.539103+0000","last_became_peered":"2024-03-11T21:25:38.539103+0000","last_unstale":"2024-03-11T21:31:26.532585+0000","last_undegraded":"2024-03-11T21:31:26.532585+0000","last_fullsized":"2024-03-11T21:31:26.532585+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:30:59.528265+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:27:11.938909+0000","last_clean_scrub_stamp":"2024-03-11T21:30:59.528265+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:32:24.866196+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,2],"acting":[5,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.10","version":"94'6","reported_seq":202,"reported_epoch":148,"state":"active+clean","last_fresh":"2024-03-11T21:31:26.086299+0000","last_change":"2024-03-11T21:30:21.932499+0000","last_active":"2024-03-11T21:31:26.086299+0000","last_peered":"2024-03-11T21:31:26.086299+0000","last_clean":"2024-03-11T21:31:26.086299+0000","last_became_active":"2024-03-11T21:25:40.552720+0000","last_became_peered":"2024-03-11T21:25:40.552720+0000","last_unstale":"2024-03-11T21:31:26.086299+0000","last_undegraded":"2024-03-11T21:31:26.086299+0000","last_fullsized":"2024-03-11T21:31:26.086299+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"94'6","last_scrub_stamp":"2024-03-11T21:30:21.932340+0000","last_deep_scrub":"94'6","last_deep_scrub_stamp":"2024-03-11T21:28:24.603974+0000","last_clean_scrub_stamp":"2024-03-11T21:30:21.932340+0000","objects_scrubbed":2,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:31:48.924443+0000","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":6,"num_read_kb":4,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6],"acting":[3,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":186,"reported_epoch":148,"state":"active+clean","last_fresh":"2024-03-11T21:31:26.085802+0000","last_change":"2024-03-11T21:31:14.338908+0000","last_active":"2024-03-11T21:31:26.085802+0000","last_peered":"2024-03-11T21:31:26.085802+0000","last_clean":"2024-03-11T21:31:26.085802+0000","last_became_active":"2024-03-11T21:25:44.583181+0000","last_became_peered":"2024-03-11T21:25:44.583181+0000","last_unstale":"2024-03-11T21:31:26.085802+0000","last_undegraded":"2024-03-11T21:31:26.085802+0000","last_fullsized":"2024-03-11T21:31:26.085802+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:31:14.338640+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:43.559452+0000","last_clean_scrub_stamp":"2024-03-11T21:31:14.338640+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:32:30.708648+0000","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":191,"reported_epoch":148,"state":"active+clean","last_fresh":"2024-03-11T21:31:26.530703+0000","last_change":"2024-03-11T21:30:19.680660+0000","last_active":"2024-03-11T21:31:26.530703+0000","last_peered":"2024-03-11T21:31:26.530703+0000","last_clean":"2024-03-11T21:31:26.530703+0000","last_became_active":"2024-03-11T21:25:42.573158+0000","last_became_peered":"2024-03-11T21:25:42.573158+0000","last_unstale":"2024-03-11T21:31:26.530703+0000","last_undegraded":"2024-03-11T21:31:26.530703+0000","last_fullsized":"2024-03-11T21:31:26.530703+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:30:19.680539+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:27:21.207303+0000","last_clean_scrub_stamp":"2024-03-11T21:30:19.680539+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.15","version":"0'0","reported_seq":199,"reported_epoch":148,"state":"active+clean","last_fresh":"2024-03-11T21:31:26.531285+0000","last_change":"2024-03-11T21:29:43.022451+0000","last_active":"2024-03-11T21:31:26.531285+0000","last_peered":"2024-03-11T21:31:26.531285+0000","last_clean":"2024-03-11T21:31:26.531285+0000","last_became_active":"2024-03-11T21:25:38.541705+0000","last_became_peered":"2024-03-11T21:25:38.541705+0000","last_unstale":"2024-03-11T21:31:26.531285+0000","last_undegraded":"2024-03-11T21:31:26.531285+0000","last_fullsized":"2024-03-11T21:31:26.531285+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:29:43.022321+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:26:44.860499+0000","last_clean_scrub_stamp":"2024-03-11T21:29:43.022321+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.13","version":"109'16","reported_seq":237,"reported_epoch":148,"state":"active+clean","last_fresh":"2024-03-11T21:31:26.083548+0000","last_change":"2024-03-11T21:30:23.314515+0000","last_active":"2024-03-11T21:31:26.083548+0000","last_peered":"2024-03-11T21:31:26.083548+0000","last_clean":"2024-03-11T21:31:26.083548+0000","last_became_active":"2024-03-11T21:26:11.214879+0000","last_became_peered":"2024-03-11T21:26:11.214879+0000","last_unstale":"2024-03-11T21:31:26.083548+0000","last_undegraded":"2024-03-11T21:31:26.083548+0000","last_fullsized":"2024-03-11T21:31:26.083548+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"109'16","last_scrub_stamp":"2024-03-11T21:30:23.314410+0000","last_deep_scrub":"109'16","last_deep_scrub_stamp":"2024-03-11T21:28:32.854393+0000","last_clean_scrub_stamp":"2024-03-11T21:30:23.314410+0000","objects_scrubbed":5,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:31:50.913510+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,2],"acting":[4,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.10","version":"0'0","reported_seq":187,"reported_epoch":148,"state":"active+clean","last_fresh":"2024-03-11T21:31:26.085921+0000","last_change":"2024-03-11T21:31:22.541299+0000","last_active":"2024-03-11T21:31:26.085921+0000","last_peered":"2024-03-11T21:31:26.085921+0000","last_clean":"2024-03-11T21:31:26.085921+0000","last_became_active":"2024-03-11T21:25:44.591189+0000","last_became_peered":"2024-03-11T21:25:44.591189+0000","last_unstale":"2024-03-11T21:31:26.085921+0000","last_undegraded":"2024-03-11T21:31:26.085921+0000","last_fullsized":"2024-03-11T21:31:26.085921+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:31:22.541178+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:27:25.417450+0000","last_clean_scrub_stamp":"2024-03-11T21:31:22.541178+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:32:32.828094+0000","scrub_duration":18,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,1],"acting":[0,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":190,"reported_epoch":148,"state":"active+clean","last_fresh":"2024-03-11T21:31:26.088473+0000","last_change":"2024-03-11T21:31:15.368570+0000","last_active":"2024-03-11T21:31:26.088473+0000","last_peered":"2024-03-11T21:31:26.088473+0000","last_clean":"2024-03-11T21:31:26.088473+0000","last_became_active":"2024-03-11T21:25:42.576425+0000","last_became_peered":"2024-03-11T21:25:42.576425+0000","last_unstale":"2024-03-11T21:31:26.088473+0000","last_undegraded":"2024-03-11T21:31:26.088473+0000","last_fullsized":"2024-03-11T21:31:26.088473+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:31:15.367898+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:31:15.367898+0000","last_clean_scrub_stamp":"2024-03-11T21:31:15.367898+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:32:26.039743+0000","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.14","version":"0'0","reported_seq":207,"reported_epoch":148,"state":"active+clean","last_fresh":"2024-03-11T21:31:26.531370+0000","last_change":"2024-03-11T21:31:09.407017+0000","last_active":"2024-03-11T21:31:26.531370+0000","last_peered":"2024-03-11T21:31:26.531370+0000","last_clean":"2024-03-11T21:31:26.531370+0000","last_became_active":"2024-03-11T21:25:38.541852+0000","last_became_peered":"2024-03-11T21:25:38.541852+0000","last_unstale":"2024-03-11T21:31:26.531370+0000","last_undegraded":"2024-03-11T21:31:26.531370+0000","last_fullsized":"2024-03-11T21:31:26.531370+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:31:09.406884+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:37.512018+0000","last_clean_scrub_stamp":"2024-03-11T21:31:09.406884+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:32:09.476644+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,5],"acting":[6,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.12","version":"74'9","reported_seq":211,"reported_epoch":148,"state":"active+clean","last_fresh":"2024-03-11T21:31:26.530514+0000","last_change":"2024-03-11T21:29:45.013132+0000","last_active":"2024-03-11T21:31:26.530514+0000","last_peered":"2024-03-11T21:31:26.530514+0000","last_clean":"2024-03-11T21:31:26.530514+0000","last_became_active":"2024-03-11T21:25:40.547082+0000","last_became_peered":"2024-03-11T21:25:40.547082+0000","last_unstale":"2024-03-11T21:31:26.530514+0000","last_undegraded":"2024-03-11T21:31:26.530514+0000","last_fullsized":"2024-03-11T21:31:26.530514+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'9","last_scrub_stamp":"2024-03-11T21:29:45.012981+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:39.525554+0000","last_clean_scrub_stamp":"2024-03-11T21:29:45.012981+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":187,"reported_epoch":148,"state":"active+clean","last_fresh":"2024-03-11T21:31:26.084388+0000","last_change":"2024-03-11T21:31:25.682563+0000","last_active":"2024-03-11T21:31:26.084388+0000","last_peered":"2024-03-11T21:31:26.084388+0000","last_clean":"2024-03-11T21:31:26.084388+0000","last_became_active":"2024-03-11T21:25:44.589586+0000","last_became_peered":"2024-03-11T21:25:44.589586+0000","last_unstale":"2024-03-11T21:31:26.084388+0000","last_undegraded":"2024-03-11T21:31:26.084388+0000","last_fullsized":"2024-03-11T21:31:26.084388+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:31:25.682446+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:27:09.801138+0000","last_clean_scrub_stamp":"2024-03-11T21:31:25.682446+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:32:34.081976+0000","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,5],"acting":[4,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.14","version":"74'8","reported_seq":261,"reported_epoch":148,"state":"active+clean","last_fresh":"2024-03-11T21:31:26.088938+0000","last_change":"2024-03-11T21:29:34.599277+0000","last_active":"2024-03-11T21:31:26.088938+0000","last_peered":"2024-03-11T21:31:26.088938+0000","last_clean":"2024-03-11T21:31:26.088938+0000","last_became_active":"2024-03-11T21:25:42.583266+0000","last_became_peered":"2024-03-11T21:25:42.583266+0000","last_unstale":"2024-03-11T21:31:26.088938+0000","last_undegraded":"2024-03-11T21:31:26.088938+0000","last_fullsized":"2024-03-11T21:31:26.088938+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'8","last_scrub_stamp":"2024-03-11T21:29:34.599162+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:41.544550+0000","last_clean_scrub_stamp":"2024-03-11T21:29:34.599162+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:31:31.562033+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,2],"acting":[3,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.13","version":"0'0","reported_seq":199,"reported_epoch":148,"state":"active+clean","last_fresh":"2024-03-11T21:31:26.086418+0000","last_change":"2024-03-11T21:31:18.506150+0000","last_active":"2024-03-11T21:31:26.086418+0000","last_peered":"2024-03-11T21:31:26.086418+0000","last_clean":"2024-03-11T21:31:26.086418+0000","last_became_active":"2024-03-11T21:25:38.539467+0000","last_became_peered":"2024-03-11T21:25:38.539467+0000","last_unstale":"2024-03-11T21:31:26.086418+0000","last_undegraded":"2024-03-11T21:31:26.086418+0000","last_fullsized":"2024-03-11T21:31:26.086418+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:31:18.499630+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:37.512018+0000","last_clean_scrub_stamp":"2024-03-11T21:31:18.499630+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:32:38.370145+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.15","version":"94'13","reported_seq":214,"reported_epoch":148,"state":"active+clean","last_fresh":"2024-03-11T21:31:26.531824+0000","last_change":"2024-03-11T21:29:41.853277+0000","last_active":"2024-03-11T21:31:26.531824+0000","last_peered":"2024-03-11T21:31:26.531824+0000","last_clean":"2024-03-11T21:31:26.531824+0000","last_became_active":"2024-03-11T21:25:40.760907+0000","last_became_peered":"2024-03-11T21:25:40.760907+0000","last_unstale":"2024-03-11T21:31:26.531824+0000","last_undegraded":"2024-03-11T21:31:26.531824+0000","last_fullsized":"2024-03-11T21:31:26.531824+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"94'13","last_scrub_stamp":"2024-03-11T21:29:41.853158+0000","last_deep_scrub":"94'13","last_deep_scrub_stamp":"2024-03-11T21:29:41.853158+0000","last_clean_scrub_stamp":"2024-03-11T21:29:41.853158+0000","objects_scrubbed":4,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.16","version":"0'0","reported_seq":187,"reported_epoch":148,"state":"active+clean","last_fresh":"2024-03-11T21:31:26.086542+0000","last_change":"2024-03-11T21:31:08.360701+0000","last_active":"2024-03-11T21:31:26.086542+0000","last_peered":"2024-03-11T21:31:26.086542+0000","last_clean":"2024-03-11T21:31:26.086542+0000","last_became_active":"2024-03-11T21:25:44.583393+0000","last_became_peered":"2024-03-11T21:25:44.583393+0000","last_unstale":"2024-03-11T21:31:26.086542+0000","last_undegraded":"2024-03-11T21:31:26.086542+0000","last_fullsized":"2024-03-11T21:31:26.086542+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:31:08.360582+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:43.559452+0000","last_clean_scrub_stamp":"2024-03-11T21:31:08.360582+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:32:18.732280+0000","scrub_duration":17,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.15","version":"98'9","reported_seq":270,"reported_epoch":148,"state":"active+clean","last_fresh":"2024-03-11T21:31:26.531699+0000","last_change":"2024-03-11T21:31:20.625060+0000","last_active":"2024-03-11T21:31:26.531699+0000","last_peered":"2024-03-11T21:31:26.531699+0000","last_clean":"2024-03-11T21:31:26.531699+0000","last_became_active":"2024-03-11T21:25:42.572555+0000","last_became_peered":"2024-03-11T21:25:42.572555+0000","last_unstale":"2024-03-11T21:31:26.531699+0000","last_undegraded":"2024-03-11T21:31:26.531699+0000","last_fullsized":"2024-03-11T21:31:26.531699+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"98'9","last_scrub_stamp":"2024-03-11T21:31:20.624923+0000","last_deep_scrub":"98'9","last_deep_scrub_stamp":"2024-03-11T21:31:20.624923+0000","last_clean_scrub_stamp":"2024-03-11T21:31:20.624923+0000","objects_scrubbed":1,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:32:45.365157+0000","scrub_duration":25,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.12","version":"0'0","reported_seq":198,"reported_epoch":148,"state":"active+clean","last_fresh":"2024-03-11T21:31:26.531701+0000","last_change":"2024-03-11T21:30:13.021786+0000","last_active":"2024-03-11T21:31:26.531701+0000","last_peered":"2024-03-11T21:31:26.531701+0000","last_clean":"2024-03-11T21:31:26.531701+0000","last_became_active":"2024-03-11T21:25:38.543962+0000","last_became_peered":"2024-03-11T21:25:38.543962+0000","last_unstale":"2024-03-11T21:31:26.531701+0000","last_undegraded":"2024-03-11T21:31:26.531701+0000","last_fullsized":"2024-03-11T21:31:26.531701+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:30:13.021666+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:37.512018+0000","last_clean_scrub_stamp":"2024-03-11T21:30:13.021666+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.14","version":"74'10","reported_seq":201,"reported_epoch":148,"state":"active+clean","last_fresh":"2024-03-11T21:31:26.087978+0000","last_change":"2024-03-11T21:29:27.547718+0000","last_active":"2024-03-11T21:31:26.087978+0000","last_peered":"2024-03-11T21:31:26.087978+0000","last_clean":"2024-03-11T21:31:26.087978+0000","last_became_active":"2024-03-11T21:25:40.760620+0000","last_became_peered":"2024-03-11T21:25:40.760620+0000","last_unstale":"2024-03-11T21:31:26.087978+0000","last_undegraded":"2024-03-11T21:31:26.087978+0000","last_fullsized":"2024-03-11T21:31:26.087978+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'10","last_scrub_stamp":"2024-03-11T21:29:27.547582+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:39.525554+0000","last_clean_scrub_stamp":"2024-03-11T21:29:27.547582+0000","objects_scrubbed":5,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.15","version":"0'0","reported_seq":186,"reported_epoch":148,"state":"active+clean","last_fresh":"2024-03-11T21:31:26.531538+0000","last_change":"2024-03-11T21:30:17.607310+0000","last_active":"2024-03-11T21:31:26.531538+0000","last_peered":"2024-03-11T21:31:26.531538+0000","last_clean":"2024-03-11T21:31:26.531538+0000","last_became_active":"2024-03-11T21:25:45.302501+0000","last_became_peered":"2024-03-11T21:25:45.302501+0000","last_unstale":"2024-03-11T21:31:26.531538+0000","last_undegraded":"2024-03-11T21:31:26.531538+0000","last_fullsized":"2024-03-11T21:31:26.531538+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:30:17.607199+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:43.559452+0000","last_clean_scrub_stamp":"2024-03-11T21:30:17.607199+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:31:28.961236+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":190,"reported_epoch":148,"state":"active+clean","last_fresh":"2024-03-11T21:31:26.532541+0000","last_change":"2024-03-11T21:31:14.608055+0000","last_active":"2024-03-11T21:31:26.532541+0000","last_peered":"2024-03-11T21:31:26.532541+0000","last_clean":"2024-03-11T21:31:26.532541+0000","last_became_active":"2024-03-11T21:25:42.577351+0000","last_became_peered":"2024-03-11T21:25:42.577351+0000","last_unstale":"2024-03-11T21:31:26.532541+0000","last_undegraded":"2024-03-11T21:31:26.532541+0000","last_fullsized":"2024-03-11T21:31:26.532541+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:31:14.607971+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:41.544550+0000","last_clean_scrub_stamp":"2024-03-11T21:31:14.607971+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:32:26.368031+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,1],"acting":[5,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.11","version":"0'0","reported_seq":199,"reported_epoch":148,"state":"active+clean","last_fresh":"2024-03-11T21:31:26.530838+0000","last_change":"2024-03-11T21:30:14.818836+0000","last_active":"2024-03-11T21:31:26.530838+0000","last_peered":"2024-03-11T21:31:26.530838+0000","last_clean":"2024-03-11T21:31:26.530838+0000","last_became_active":"2024-03-11T21:25:38.537415+0000","last_became_peered":"2024-03-11T21:25:38.537415+0000","last_unstale":"2024-03-11T21:31:26.530838+0000","last_undegraded":"2024-03-11T21:31:26.530838+0000","last_fullsized":"2024-03-11T21:31:26.530838+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:30:14.818712+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:37.512018+0000","last_clean_scrub_stamp":"2024-03-11T21:30:14.818712+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:31:40.512363+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.17","version":"97'9","reported_seq":212,"reported_epoch":148,"state":"active+clean","last_fresh":"2024-03-11T21:31:26.087801+0000","last_change":"2024-03-11T21:29:26.522324+0000","last_active":"2024-03-11T21:31:26.087801+0000","last_peered":"2024-03-11T21:31:26.087801+0000","last_clean":"2024-03-11T21:31:26.087801+0000","last_became_active":"2024-03-11T21:26:11.219020+0000","last_became_peered":"2024-03-11T21:26:11.219020+0000","last_unstale":"2024-03-11T21:31:26.087801+0000","last_undegraded":"2024-03-11T21:31:26.087801+0000","last_fullsized":"2024-03-11T21:31:26.087801+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"97'9","last_scrub_stamp":"2024-03-11T21:29:26.522238+0000","last_deep_scrub":"97'9","last_deep_scrub_stamp":"2024-03-11T21:27:25.803132+0000","last_clean_scrub_stamp":"2024-03-11T21:29:26.522238+0000","objects_scrubbed":3,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":17,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":9,"num_read_kb":6,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,5],"acting":[3,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.14","version":"0'0","reported_seq":186,"reported_epoch":148,"state":"active+clean","last_fresh":"2024-03-11T21:31:26.530213+0000","last_change":"2024-03-11T21:31:15.409758+0000","last_active":"2024-03-11T21:31:26.530213+0000","last_peered":"2024-03-11T21:31:26.530213+0000","last_clean":"2024-03-11T21:31:26.530213+0000","last_became_active":"2024-03-11T21:25:44.587439+0000","last_became_peered":"2024-03-11T21:25:44.587439+0000","last_unstale":"2024-03-11T21:31:26.530213+0000","last_undegraded":"2024-03-11T21:31:26.530213+0000","last_fullsized":"2024-03-11T21:31:26.530213+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:31:15.409641+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:43.559452+0000","last_clean_scrub_stamp":"2024-03-11T21:31:15.409641+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:32:24.347649+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,7],"acting":[2,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":190,"reported_epoch":148,"state":"active+clean","last_fresh":"2024-03-11T21:31:26.087906+0000","last_change":"2024-03-11T21:31:08.284623+0000","last_active":"2024-03-11T21:31:26.087906+0000","last_peered":"2024-03-11T21:31:26.087906+0000","last_clean":"2024-03-11T21:31:26.087906+0000","last_became_active":"2024-03-11T21:25:42.583514+0000","last_became_peered":"2024-03-11T21:25:42.583514+0000","last_unstale":"2024-03-11T21:31:26.087906+0000","last_undegraded":"2024-03-11T21:31:26.087906+0000","last_fullsized":"2024-03-11T21:31:26.087906+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:31:08.284511+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:41.544550+0000","last_clean_scrub_stamp":"2024-03-11T21:31:08.284511+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:32:27.662416+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.10","version":"0'0","reported_seq":198,"reported_epoch":148,"state":"active+clean","last_fresh":"2024-03-11T21:31:26.529992+0000","last_change":"2024-03-11T21:31:03.260526+0000","last_active":"2024-03-11T21:31:26.529992+0000","last_peered":"2024-03-11T21:31:26.529992+0000","last_clean":"2024-03-11T21:31:26.529992+0000","last_became_active":"2024-03-11T21:25:38.533138+0000","last_became_peered":"2024-03-11T21:25:38.533138+0000","last_unstale":"2024-03-11T21:31:26.529992+0000","last_undegraded":"2024-03-11T21:31:26.529992+0000","last_fullsized":"2024-03-11T21:31:26.529992+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:31:03.260408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:37.512018+0000","last_clean_scrub_stamp":"2024-03-11T21:31:03.260408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:32:29.743555+0000","scrub_duration":29,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,0],"acting":[2,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.16","version":"74'9","reported_seq":211,"reported_epoch":148,"state":"active+clean","last_fresh":"2024-03-11T21:31:26.086867+0000","last_change":"2024-03-11T21:30:13.857912+0000","last_active":"2024-03-11T21:31:26.086867+0000","last_peered":"2024-03-11T21:31:26.086867+0000","last_clean":"2024-03-11T21:31:26.086867+0000","last_became_active":"2024-03-11T21:25:40.763434+0000","last_became_peered":"2024-03-11T21:25:40.763434+0000","last_unstale":"2024-03-11T21:31:26.086867+0000","last_undegraded":"2024-03-11T21:31:26.086867+0000","last_fullsized":"2024-03-11T21:31:26.086867+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'9","last_scrub_stamp":"2024-03-11T21:30:13.857760+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:39.525554+0000","last_clean_scrub_stamp":"2024-03-11T21:30:13.857760+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":40,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.a","version":"147'15180","reported_seq":15387,"reported_epoch":148,"state":"active+clean","last_fresh":"2024-03-11T21:31:26.087568+0000","last_change":"2024-03-11T21:30:57.938479+0000","last_active":"2024-03-11T21:31:26.087568+0000","last_peered":"2024-03-11T21:31:26.087568+0000","last_clean":"2024-03-11T21:31:26.087568+0000","last_became_active":"2024-03-11T21:27:36.692988+0000","last_became_peered":"2024-03-11T21:27:36.692988+0000","last_unstale":"2024-03-11T21:31:26.087568+0000","last_undegraded":"2024-03-11T21:31:26.087568+0000","last_fullsized":"2024-03-11T21:31:26.087568+0000","mapping_epoch":99,"log_start":"130'10500","ondisk_log_start":"130'10500","created":99,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"143'13593","last_scrub_stamp":"2024-03-11T21:30:57.938271+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:27:35.667522+0000","last_clean_scrub_stamp":"2024-03-11T21:30:57.938271+0000","objects_scrubbed":1614,"log_size":4680,"log_dups_size":0,"ondisk_log_size":4680,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:32:07.370996+0000","scrub_duration":767,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15180,"num_write_kb":485760,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5],"acting":[3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.b","version":"0'0","reported_seq":178,"reported_epoch":148,"state":"active+clean","last_fresh":"2024-03-11T21:31:26.088555+0000","last_change":"2024-03-11T21:30:19.008385+0000","last_active":"2024-03-11T21:31:26.088555+0000","last_peered":"2024-03-11T21:31:26.088555+0000","last_clean":"2024-03-11T21:31:26.088555+0000","last_became_active":"2024-03-11T21:25:44.590071+0000","last_became_peered":"2024-03-11T21:25:44.590071+0000","last_unstale":"2024-03-11T21:31:26.088555+0000","last_undegraded":"2024-03-11T21:31:26.088555+0000","last_fullsized":"2024-03-11T21:31:26.088555+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:30:19.006959+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:43.559452+0000","last_clean_scrub_stamp":"2024-03-11T21:30:19.006959+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":70,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.f","version":"67'2","reported_seq":211,"reported_epoch":148,"state":"active+clean","last_fresh":"2024-03-11T21:31:26.084263+0000","last_change":"2024-03-11T21:31:03.533706+0000","last_active":"2024-03-11T21:31:26.084263+0000","last_peered":"2024-03-11T21:31:26.084263+0000","last_clean":"2024-03-11T21:31:26.084263+0000","last_became_active":"2024-03-11T21:25:38.537711+0000","last_became_peered":"2024-03-11T21:25:38.537711+0000","last_unstale":"2024-03-11T21:31:26.084263+0000","last_undegraded":"2024-03-11T21:31:26.084263+0000","last_fullsized":"2024-03-11T21:31:26.084263+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'2","last_scrub_stamp":"2024-03-11T21:31:03.533547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:37.512018+0000","last_clean_scrub_stamp":"2024-03-11T21:31:03.533547+0000","objects_scrubbed":2,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:32:27.195878+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":92,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":10,"num_read_kb":10,"num_write":4,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,7],"acting":[4,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.9","version":"146'17","reported_seq":223,"reported_epoch":148,"state":"active+clean","last_fresh":"2024-03-11T21:31:26.083887+0000","last_change":"2024-03-11T21:31:16.615182+0000","last_active":"2024-03-11T21:31:26.083887+0000","last_peered":"2024-03-11T21:31:26.083887+0000","last_clean":"2024-03-11T21:31:26.083887+0000","last_became_active":"2024-03-11T21:25:40.552267+0000","last_became_peered":"2024-03-11T21:25:40.552267+0000","last_unstale":"2024-03-11T21:31:26.083887+0000","last_undegraded":"2024-03-11T21:31:26.083887+0000","last_fullsized":"2024-03-11T21:31:26.083887+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"146'17","last_scrub_stamp":"2024-03-11T21:31:16.615010+0000","last_deep_scrub":"146'17","last_deep_scrub_stamp":"2024-03-11T21:31:16.615010+0000","last_clean_scrub_stamp":"2024-03-11T21:31:16.615010+0000","objects_scrubbed":5,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:32:36.347947+0000","scrub_duration":27,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":25,"num_read_kb":16,"num_write":14,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,3],"acting":[4,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":190,"reported_epoch":148,"state":"active+clean","last_fresh":"2024-03-11T21:31:26.529969+0000","last_change":"2024-03-11T21:31:12.434593+0000","last_active":"2024-03-11T21:31:26.529969+0000","last_peered":"2024-03-11T21:31:26.529969+0000","last_clean":"2024-03-11T21:31:26.529969+0000","last_became_active":"2024-03-11T21:25:42.573987+0000","last_became_peered":"2024-03-11T21:25:42.573987+0000","last_unstale":"2024-03-11T21:31:26.529969+0000","last_undegraded":"2024-03-11T21:31:26.529969+0000","last_fullsized":"2024-03-11T21:31:26.529969+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:31:12.434479+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:41.544550+0000","last_clean_scrub_stamp":"2024-03-11T21:31:12.434479+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:32:19.616693+0000","scrub_duration":17,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.1","version":"147'15502","reported_seq":15707,"reported_epoch":148,"state":"active+clean","last_fresh":"2024-03-11T21:31:26.083212+0000","last_change":"2024-03-11T21:31:06.819421+0000","last_active":"2024-03-11T21:31:26.083212+0000","last_peered":"2024-03-11T21:31:26.083212+0000","last_clean":"2024-03-11T21:31:26.083212+0000","last_became_active":"2024-03-11T21:27:36.696990+0000","last_became_peered":"2024-03-11T21:27:36.696990+0000","last_unstale":"2024-03-11T21:31:26.083212+0000","last_undegraded":"2024-03-11T21:31:26.083212+0000","last_fullsized":"2024-03-11T21:31:26.083212+0000","mapping_epoch":99,"log_start":"128'10400","ondisk_log_start":"128'10400","created":99,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"144'14362","last_scrub_stamp":"2024-03-11T21:31:06.819152+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:27:35.667522+0000","last_clean_scrub_stamp":"2024-03-11T21:31:06.819152+0000","objects_scrubbed":1146,"log_size":5102,"log_dups_size":0,"ondisk_log_size":5102,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:32:18.841039+0000","scrub_duration":263,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15502,"num_write_kb":496064,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.0","version":"0'0","reported_seq":179,"reported_epoch":148,"state":"active+clean","last_fresh":"2024-03-11T21:31:26.085929+0000","last_change":"2024-03-11T21:29:39.498904+0000","last_active":"2024-03-11T21:31:26.085929+0000","last_peered":"2024-03-11T21:31:26.085929+0000","last_clean":"2024-03-11T21:31:26.085929+0000","last_became_active":"2024-03-11T21:25:44.584565+0000","last_became_peered":"2024-03-11T21:25:44.584565+0000","last_unstale":"2024-03-11T21:31:26.085929+0000","last_undegraded":"2024-03-11T21:31:26.085929+0000","last_fullsized":"2024-03-11T21:31:26.085929+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:29:39.498784+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:29:39.498784+0000","last_clean_scrub_stamp":"2024-03-11T21:29:39.498784+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":199,"reported_epoch":148,"state":"active+clean","last_fresh":"2024-03-11T21:31:26.530916+0000","last_change":"2024-03-11T21:29:35.214647+0000","last_active":"2024-03-11T21:31:26.530916+0000","last_peered":"2024-03-11T21:31:26.530916+0000","last_clean":"2024-03-11T21:31:26.530916+0000","last_became_active":"2024-03-11T21:25:38.541760+0000","last_became_peered":"2024-03-11T21:25:38.541760+0000","last_unstale":"2024-03-11T21:31:26.530916+0000","last_undegraded":"2024-03-11T21:31:26.530916+0000","last_fullsized":"2024-03-11T21:31:26.530916+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:29:35.214430+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:37.512018+0000","last_clean_scrub_stamp":"2024-03-11T21:29:35.214430+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,7],"acting":[1,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.2","version":"94'15","reported_seq":223,"reported_epoch":148,"state":"active+clean","last_fresh":"2024-03-11T21:31:26.530605+0000","last_change":"2024-03-11T21:31:08.956264+0000","last_active":"2024-03-11T21:31:26.530605+0000","last_peered":"2024-03-11T21:31:26.530605+0000","last_clean":"2024-03-11T21:31:26.530605+0000","last_became_active":"2024-03-11T21:25:40.548833+0000","last_became_peered":"2024-03-11T21:25:40.548833+0000","last_unstale":"2024-03-11T21:31:26.530605+0000","last_undegraded":"2024-03-11T21:31:26.530605+0000","last_fullsized":"2024-03-11T21:31:26.530605+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"94'15","last_scrub_stamp":"2024-03-11T21:31:08.955971+0000","last_deep_scrub":"94'15","last_deep_scrub_stamp":"2024-03-11T21:26:49.993060+0000","last_clean_scrub_stamp":"2024-03-11T21:31:08.955971+0000","objects_scrubbed":5,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:32:37.727507+0000","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.3","version":"74'8","reported_seq":270,"reported_epoch":148,"state":"active+clean","last_fresh":"2024-03-11T21:31:26.087094+0000","last_change":"2024-03-11T21:31:15.465617+0000","last_active":"2024-03-11T21:31:26.087094+0000","last_peered":"2024-03-11T21:31:26.087094+0000","last_clean":"2024-03-11T21:31:26.087094+0000","last_became_active":"2024-03-11T21:25:42.578161+0000","last_became_peered":"2024-03-11T21:25:42.578161+0000","last_unstale":"2024-03-11T21:31:26.087094+0000","last_undegraded":"2024-03-11T21:31:26.087094+0000","last_fullsized":"2024-03-11T21:31:26.087094+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'8","last_scrub_stamp":"2024-03-11T21:31:15.465460+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:41.544550+0000","last_clean_scrub_stamp":"2024-03-11T21:31:15.465460+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:32:43.379445+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,5],"acting":[0,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.7","version":"147'15308","reported_seq":15525,"reported_epoch":148,"state":"active+clean","last_fresh":"2024-03-11T21:31:26.532016+0000","last_change":"2024-03-11T21:31:00.103228+0000","last_active":"2024-03-11T21:31:26.532016+0000","last_peered":"2024-03-11T21:31:26.532016+0000","last_clean":"2024-03-11T21:31:26.532016+0000","last_became_active":"2024-03-11T21:27:36.695055+0000","last_became_peered":"2024-03-11T21:27:36.695055+0000","last_unstale":"2024-03-11T21:31:26.532016+0000","last_undegraded":"2024-03-11T21:31:26.532016+0000","last_fullsized":"2024-03-11T21:31:26.532016+0000","mapping_epoch":99,"log_start":"124'9000","ondisk_log_start":"124'9000","created":99,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"144'13822","last_scrub_stamp":"2024-03-11T21:31:00.103084+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:27:35.667522+0000","last_clean_scrub_stamp":"2024-03-11T21:31:00.103084+0000","objects_scrubbed":1506,"log_size":6308,"log_dups_size":0,"ondisk_log_size":6308,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:32:17.242400+0000","scrub_duration":561,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15308,"num_write_kb":489856,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,3],"acting":[6,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":186,"reported_epoch":148,"state":"active+clean","last_fresh":"2024-03-11T21:31:26.088217+0000","last_change":"2024-03-11T21:31:01.241438+0000","last_active":"2024-03-11T21:31:26.088217+0000","last_peered":"2024-03-11T21:31:26.088217+0000","last_clean":"2024-03-11T21:31:26.088217+0000","last_became_active":"2024-03-11T21:25:44.585223+0000","last_became_peered":"2024-03-11T21:25:44.585223+0000","last_unstale":"2024-03-11T21:31:26.088217+0000","last_undegraded":"2024-03-11T21:31:26.088217+0000","last_fullsized":"2024-03-11T21:31:26.088217+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:31:01.241180+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:28:30.563712+0000","last_clean_scrub_stamp":"2024-03-11T21:31:01.241180+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:32:22.284772+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,7],"acting":[3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":206,"reported_epoch":148,"state":"active+clean","last_fresh":"2024-03-11T21:31:26.532656+0000","last_change":"2024-03-11T21:31:16.608343+0000","last_active":"2024-03-11T21:31:26.532656+0000","last_peered":"2024-03-11T21:31:26.532656+0000","last_clean":"2024-03-11T21:31:26.532656+0000","last_became_active":"2024-03-11T21:25:38.538540+0000","last_became_peered":"2024-03-11T21:25:38.538540+0000","last_unstale":"2024-03-11T21:31:26.532656+0000","last_undegraded":"2024-03-11T21:31:26.532656+0000","last_fullsized":"2024-03-11T21:31:26.532656+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:31:16.608266+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:37.512018+0000","last_clean_scrub_stamp":"2024-03-11T21:31:16.608266+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:32:30.358883+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.4","version":"74'28","reported_seq":314,"reported_epoch":148,"state":"active+clean","last_fresh":"2024-03-11T21:31:26.530929+0000","last_change":"2024-03-11T21:30:24.750195+0000","last_active":"2024-03-11T21:31:26.530929+0000","last_peered":"2024-03-11T21:31:26.530929+0000","last_clean":"2024-03-11T21:31:26.530929+0000","last_became_active":"2024-03-11T21:25:40.552902+0000","last_became_peered":"2024-03-11T21:25:40.552902+0000","last_unstale":"2024-03-11T21:31:26.530929+0000","last_undegraded":"2024-03-11T21:31:26.530929+0000","last_fullsized":"2024-03-11T21:31:26.530929+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'28","last_scrub_stamp":"2024-03-11T21:30:24.750058+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:39.525554+0000","last_clean_scrub_stamp":"2024-03-11T21:30:24.750058+0000","objects_scrubbed":10,"log_size":28,"log_dups_size":0,"ondisk_log_size":28,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:31:36.994296+0000","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":358,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":48,"num_read_kb":33,"num_write":26,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3],"acting":[1,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":183,"reported_epoch":148,"state":"active+clean","last_fresh":"2024-03-11T21:31:26.086818+0000","last_change":"2024-03-11T21:29:38.388702+0000","last_active":"2024-03-11T21:31:26.086818+0000","last_peered":"2024-03-11T21:31:26.086818+0000","last_clean":"2024-03-11T21:31:26.086818+0000","last_became_active":"2024-03-11T21:25:42.572673+0000","last_became_peered":"2024-03-11T21:25:42.572673+0000","last_unstale":"2024-03-11T21:31:26.086818+0000","last_undegraded":"2024-03-11T21:31:26.086818+0000","last_fullsized":"2024-03-11T21:31:26.086818+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:29:38.388578+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:41.544550+0000","last_clean_scrub_stamp":"2024-03-11T21:29:38.388578+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.4","version":"147'15314","reported_seq":15494,"reported_epoch":148,"state":"active+clean","last_fresh":"2024-03-11T21:31:26.086311+0000","last_change":"2024-03-11T21:29:39.193418+0000","last_active":"2024-03-11T21:31:26.086311+0000","last_peered":"2024-03-11T21:31:26.086311+0000","last_clean":"2024-03-11T21:31:26.086311+0000","last_became_active":"2024-03-11T21:27:36.690751+0000","last_became_peered":"2024-03-11T21:27:36.690751+0000","last_unstale":"2024-03-11T21:31:26.086311+0000","last_undegraded":"2024-03-11T21:31:26.086311+0000","last_fullsized":"2024-03-11T21:31:26.086311+0000","mapping_epoch":99,"log_start":"126'9800","ondisk_log_start":"126'9800","created":99,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"124'9100","last_scrub_stamp":"2024-03-11T21:29:39.193151+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:27:35.667522+0000","last_clean_scrub_stamp":"2024-03-11T21:29:39.193151+0000","objects_scrubbed":6273,"log_size":5514,"log_dups_size":0,"ondisk_log_size":5514,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"queued for scrub","scrub_duration":1792,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15314,"num_write_kb":490048,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,1],"acting":[0,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.5","version":"0'0","reported_seq":186,"reported_epoch":148,"state":"active+clean","last_fresh":"2024-03-11T21:31:26.531820+0000","last_change":"2024-03-11T21:30:55.708176+0000","last_active":"2024-03-11T21:31:26.531820+0000","last_peered":"2024-03-11T21:31:26.531820+0000","last_clean":"2024-03-11T21:31:26.531820+0000","last_became_active":"2024-03-11T21:25:45.302120+0000","last_became_peered":"2024-03-11T21:25:45.302120+0000","last_unstale":"2024-03-11T21:31:26.531820+0000","last_undegraded":"2024-03-11T21:31:26.531820+0000","last_fullsized":"2024-03-11T21:31:26.531820+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:30:55.708059+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:43.559452+0000","last_clean_scrub_stamp":"2024-03-11T21:30:55.708059+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:32:11.142437+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,3],"acting":[7,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.0","version":"76'5","reported_seq":556,"reported_epoch":148,"state":"active+clean","last_fresh":"2024-03-11T21:31:30.347564+0000","last_change":"2024-03-11T21:30:14.726351+0000","last_active":"2024-03-11T21:31:30.347564+0000","last_peered":"2024-03-11T21:31:30.347564+0000","last_clean":"2024-03-11T21:31:30.347564+0000","last_became_active":"2024-03-11T21:25:38.536890+0000","last_became_peered":"2024-03-11T21:25:38.536890+0000","last_unstale":"2024-03-11T21:31:30.347564+0000","last_undegraded":"2024-03-11T21:31:30.347564+0000","last_fullsized":"2024-03-11T21:31:30.347564+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"76'5","last_scrub_stamp":"2024-03-11T21:30:14.726214+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:37.512018+0000","last_clean_scrub_stamp":"2024-03-11T21:30:14.726214+0000","objects_scrubbed":1,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:31:31.690340+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0.000753559,"stat_sum":{"num_bytes":389,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":347,"num_read_kb":342,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,6],"acting":[1,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":206,"reported_epoch":148,"state":"active+clean","last_fresh":"2024-03-11T21:31:26.529927+0000","last_change":"2024-03-11T21:31:21.359528+0000","last_active":"2024-03-11T21:31:26.529927+0000","last_peered":"2024-03-11T21:31:26.529927+0000","last_clean":"2024-03-11T21:31:26.529927+0000","last_became_active":"2024-03-11T21:25:38.532984+0000","last_became_peered":"2024-03-11T21:25:38.532984+0000","last_unstale":"2024-03-11T21:31:26.529927+0000","last_undegraded":"2024-03-11T21:31:26.529927+0000","last_fullsized":"2024-03-11T21:31:26.529927+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:31:21.359298+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:26:48.997454+0000","last_clean_scrub_stamp":"2024-03-11T21:31:21.359298+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:32:43.745058+0000","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.7","version":"97'18","reported_seq":230,"reported_epoch":148,"state":"active+clean","last_fresh":"2024-03-11T21:31:26.530815+0000","last_change":"2024-03-11T21:30:11.805557+0000","last_active":"2024-03-11T21:31:26.530815+0000","last_peered":"2024-03-11T21:31:26.530815+0000","last_clean":"2024-03-11T21:31:26.530815+0000","last_became_active":"2024-03-11T21:25:40.541769+0000","last_became_peered":"2024-03-11T21:25:40.541769+0000","last_unstale":"2024-03-11T21:31:26.530815+0000","last_undegraded":"2024-03-11T21:31:26.530815+0000","last_fullsized":"2024-03-11T21:31:26.530815+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"97'18","last_scrub_stamp":"2024-03-11T21:30:11.805394+0000","last_deep_scrub":"97'18","last_deep_scrub_stamp":"2024-03-11T21:27:19.239404+0000","last_clean_scrub_stamp":"2024-03-11T21:30:11.805394+0000","objects_scrubbed":5,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":19,"num_write":16,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":190,"reported_epoch":148,"state":"active+clean","last_fresh":"2024-03-11T21:31:26.530761+0000","last_change":"2024-03-11T21:31:06.279911+0000","last_active":"2024-03-11T21:31:26.530761+0000","last_peered":"2024-03-11T21:31:26.530761+0000","last_clean":"2024-03-11T21:31:26.530761+0000","last_became_active":"2024-03-11T21:25:42.581710+0000","last_became_peered":"2024-03-11T21:25:42.581710+0000","last_unstale":"2024-03-11T21:31:26.530761+0000","last_undegraded":"2024-03-11T21:31:26.530761+0000","last_fullsized":"2024-03-11T21:31:26.530761+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:31:06.279751+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:41.544550+0000","last_clean_scrub_stamp":"2024-03-11T21:31:06.279751+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:32:31.343510+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7],"acting":[2,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.6","version":"147'15282","reported_seq":15484,"reported_epoch":148,"state":"active+clean","last_fresh":"2024-03-11T21:31:26.086739+0000","last_change":"2024-03-11T21:31:11.813944+0000","last_active":"2024-03-11T21:31:26.086739+0000","last_peered":"2024-03-11T21:31:26.086739+0000","last_clean":"2024-03-11T21:31:26.086739+0000","last_became_active":"2024-03-11T21:27:36.695018+0000","last_became_peered":"2024-03-11T21:27:36.695018+0000","last_unstale":"2024-03-11T21:31:26.086739+0000","last_undegraded":"2024-03-11T21:31:26.086739+0000","last_fullsized":"2024-03-11T21:31:26.086739+0000","mapping_epoch":99,"log_start":"126'9800","ondisk_log_start":"126'9800","created":99,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"145'14537","last_scrub_stamp":"2024-03-11T21:31:11.813806+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:27:35.667522+0000","last_clean_scrub_stamp":"2024-03-11T21:31:11.813806+0000","objects_scrubbed":758,"log_size":5482,"log_dups_size":0,"ondisk_log_size":5482,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:32:30.283716+0000","scrub_duration":408,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15282,"num_write_kb":489024,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.7","version":"124'2","reported_seq":180,"reported_epoch":148,"state":"active+clean","last_fresh":"2024-03-11T21:31:26.532648+0000","last_change":"2024-03-11T21:29:34.957004+0000","last_active":"2024-03-11T21:31:26.532648+0000","last_peered":"2024-03-11T21:31:26.532648+0000","last_clean":"2024-03-11T21:31:26.532648+0000","last_became_active":"2024-03-11T21:25:44.591376+0000","last_became_peered":"2024-03-11T21:25:44.591376+0000","last_unstale":"2024-03-11T21:31:26.532648+0000","last_undegraded":"2024-03-11T21:31:26.532648+0000","last_fullsized":"2024-03-11T21:31:26.532648+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"124'2","last_scrub_stamp":"2024-03-11T21:29:34.956908+0000","last_deep_scrub":"124'2","last_deep_scrub_stamp":"2024-03-11T21:29:34.956908+0000","last_clean_scrub_stamp":"2024-03-11T21:29:34.956908+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":68,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,4],"acting":[5,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2","version":"71'2","reported_seq":207,"reported_epoch":148,"state":"active+clean","last_fresh":"2024-03-11T21:31:26.087300+0000","last_change":"2024-03-11T21:30:19.973353+0000","last_active":"2024-03-11T21:31:26.087300+0000","last_peered":"2024-03-11T21:31:26.087300+0000","last_clean":"2024-03-11T21:31:26.087300+0000","last_became_active":"2024-03-11T21:25:38.539413+0000","last_became_peered":"2024-03-11T21:25:38.539413+0000","last_unstale":"2024-03-11T21:31:26.087300+0000","last_undegraded":"2024-03-11T21:31:26.087300+0000","last_fullsized":"2024-03-11T21:31:26.087300+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"71'2","last_scrub_stamp":"2024-03-11T21:30:19.973188+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:37.512018+0000","last_clean_scrub_stamp":"2024-03-11T21:30:19.973188+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:31:40.539347+0000","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0.0014206449999999999,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":204,"reported_epoch":148,"state":"active+clean","last_fresh":"2024-03-11T21:31:26.532891+0000","last_change":"2024-03-11T21:29:30.862499+0000","last_active":"2024-03-11T21:31:26.532891+0000","last_peered":"2024-03-11T21:31:26.532891+0000","last_clean":"2024-03-11T21:31:26.532891+0000","last_became_active":"2024-03-11T21:26:11.216188+0000","last_became_peered":"2024-03-11T21:26:11.216188+0000","last_unstale":"2024-03-11T21:31:26.532891+0000","last_undegraded":"2024-03-11T21:31:26.532891+0000","last_fullsized":"2024-03-11T21:31:26.532891+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:29:30.862374+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:37.512018+0000","last_clean_scrub_stamp":"2024-03-11T21:29:30.862374+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,4],"acting":[5,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"87'128","reported_seq":453,"reported_epoch":148,"state":"active+clean","last_fresh":"2024-03-11T21:31:26.532162+0000","last_change":"2024-03-11T21:30:22.564543+0000","last_active":"2024-03-11T21:31:26.532162+0000","last_peered":"2024-03-11T21:31:26.532162+0000","last_clean":"2024-03-11T21:31:26.532162+0000","last_became_active":"2024-03-11T21:25:30.502597+0000","last_became_peered":"2024-03-11T21:25:30.502597+0000","last_unstale":"2024-03-11T21:31:26.532162+0000","last_undegraded":"2024-03-11T21:31:26.532162+0000","last_fullsized":"2024-03-11T21:31:26.532162+0000","mapping_epoch":63,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":64,"parent":"0.0","parent_split_bits":0,"last_scrub":"87'128","last_scrub_stamp":"2024-03-11T21:30:22.564368+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:24.261974+0000","last_clean_scrub_stamp":"2024-03-11T21:30:22.564368+0000","objects_scrubbed":2,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:31:48.461461+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":212,"num_read_kb":306,"num_write":179,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1771104,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.5","version":"74'16","reported_seq":288,"reported_epoch":148,"state":"active+clean","last_fresh":"2024-03-11T21:31:26.531180+0000","last_change":"2024-03-11T21:30:17.783268+0000","last_active":"2024-03-11T21:31:26.531180+0000","last_peered":"2024-03-11T21:31:26.531180+0000","last_clean":"2024-03-11T21:31:26.531180+0000","last_became_active":"2024-03-11T21:25:40.761457+0000","last_became_peered":"2024-03-11T21:25:40.761457+0000","last_unstale":"2024-03-11T21:31:26.531180+0000","last_undegraded":"2024-03-11T21:31:26.531180+0000","last_fullsized":"2024-03-11T21:31:26.531180+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'16","last_scrub_stamp":"2024-03-11T21:30:17.783007+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:39.525554+0000","last_clean_scrub_stamp":"2024-03-11T21:30:17.783007+0000","objects_scrubbed":8,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:31:40.049704+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":154,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":25,"num_read_kb":15,"num_write":13,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":198,"reported_epoch":148,"state":"active+clean","last_fresh":"2024-03-11T21:31:26.532365+0000","last_change":"2024-03-11T21:31:00.607444+0000","last_active":"2024-03-11T21:31:26.532365+0000","last_peered":"2024-03-11T21:31:26.532365+0000","last_clean":"2024-03-11T21:31:26.532365+0000","last_became_active":"2024-03-11T21:25:42.586226+0000","last_became_peered":"2024-03-11T21:25:42.586226+0000","last_unstale":"2024-03-11T21:31:26.532365+0000","last_undegraded":"2024-03-11T21:31:26.532365+0000","last_fullsized":"2024-03-11T21:31:26.532365+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:31:00.607366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:41.544550+0000","last_clean_scrub_stamp":"2024-03-11T21:31:00.607366+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:32:10.161500+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,5],"acting":[7,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.5","version":"147'15256","reported_seq":15467,"reported_epoch":148,"state":"active+clean","last_fresh":"2024-03-11T21:31:26.531418+0000","last_change":"2024-03-11T21:30:59.104725+0000","last_active":"2024-03-11T21:31:26.531418+0000","last_peered":"2024-03-11T21:31:26.531418+0000","last_clean":"2024-03-11T21:31:26.531418+0000","last_became_active":"2024-03-11T21:27:36.691428+0000","last_became_peered":"2024-03-11T21:27:36.691428+0000","last_unstale":"2024-03-11T21:31:26.531418+0000","last_undegraded":"2024-03-11T21:31:26.531418+0000","last_fullsized":"2024-03-11T21:31:26.531418+0000","mapping_epoch":99,"log_start":"129'10300","ondisk_log_start":"129'10300","created":99,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"144'13718","last_scrub_stamp":"2024-03-11T21:30:59.104513+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:27:35.667522+0000","last_clean_scrub_stamp":"2024-03-11T21:30:59.104513+0000","objects_scrubbed":1548,"log_size":4956,"log_dups_size":0,"ondisk_log_size":4956,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:32:24.841804+0000","scrub_duration":440,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15256,"num_write_kb":488192,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,0],"acting":[7,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.4","version":"0'0","reported_seq":187,"reported_epoch":148,"state":"active+clean","last_fresh":"2024-03-11T21:31:26.531056+0000","last_change":"2024-03-11T21:30:58.758256+0000","last_active":"2024-03-11T21:31:26.531056+0000","last_peered":"2024-03-11T21:31:26.531056+0000","last_clean":"2024-03-11T21:31:26.531056+0000","last_became_active":"2024-03-11T21:25:44.594188+0000","last_became_peered":"2024-03-11T21:25:44.594188+0000","last_unstale":"2024-03-11T21:31:26.531056+0000","last_undegraded":"2024-03-11T21:31:26.531056+0000","last_fullsized":"2024-03-11T21:31:26.531056+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:30:58.757819+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:43.559452+0000","last_clean_scrub_stamp":"2024-03-11T21:30:58.757819+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:32:17.709798+0000","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":206,"reported_epoch":148,"state":"active+clean","last_fresh":"2024-03-11T21:31:26.086188+0000","last_change":"2024-03-11T21:30:22.969948+0000","last_active":"2024-03-11T21:31:26.086188+0000","last_peered":"2024-03-11T21:31:26.086188+0000","last_clean":"2024-03-11T21:31:26.086188+0000","last_became_active":"2024-03-11T21:25:38.543834+0000","last_became_peered":"2024-03-11T21:25:38.543834+0000","last_unstale":"2024-03-11T21:31:26.086188+0000","last_undegraded":"2024-03-11T21:31:26.086188+0000","last_fullsized":"2024-03-11T21:31:26.086188+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:30:22.969840+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:37.512018+0000","last_clean_scrub_stamp":"2024-03-11T21:30:22.969840+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:31:46.044467+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0.00089342499999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":206,"reported_epoch":148,"state":"active+clean","last_fresh":"2024-03-11T21:31:26.531952+0000","last_change":"2024-03-11T21:31:06.457491+0000","last_active":"2024-03-11T21:31:26.531952+0000","last_peered":"2024-03-11T21:31:26.531952+0000","last_clean":"2024-03-11T21:31:26.531952+0000","last_became_active":"2024-03-11T21:25:38.541617+0000","last_became_peered":"2024-03-11T21:25:38.541617+0000","last_unstale":"2024-03-11T21:31:26.531952+0000","last_undegraded":"2024-03-11T21:31:26.531952+0000","last_fullsized":"2024-03-11T21:31:26.531952+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:31:06.457393+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:37.512018+0000","last_clean_scrub_stamp":"2024-03-11T21:31:06.457393+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:32:33.915715+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,0],"acting":[7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.6","version":"74'12","reported_seq":213,"reported_epoch":148,"state":"active+clean","last_fresh":"2024-03-11T21:31:26.086640+0000","last_change":"2024-03-11T21:31:04.383837+0000","last_active":"2024-03-11T21:31:26.086640+0000","last_peered":"2024-03-11T21:31:26.086640+0000","last_clean":"2024-03-11T21:31:26.086640+0000","last_became_active":"2024-03-11T21:25:40.550388+0000","last_became_peered":"2024-03-11T21:25:40.550388+0000","last_unstale":"2024-03-11T21:31:26.086640+0000","last_undegraded":"2024-03-11T21:31:26.086640+0000","last_fullsized":"2024-03-11T21:31:26.086640+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'12","last_scrub_stamp":"2024-03-11T21:31:04.383580+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:39.525554+0000","last_clean_scrub_stamp":"2024-03-11T21:31:04.383580+0000","objects_scrubbed":6,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:32:16.235650+0000","scrub_duration":25,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":18,"num_read_kb":12,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":190,"reported_epoch":148,"state":"active+clean","last_fresh":"2024-03-11T21:31:26.532550+0000","last_change":"2024-03-11T21:31:12.634176+0000","last_active":"2024-03-11T21:31:26.532550+0000","last_peered":"2024-03-11T21:31:26.532550+0000","last_clean":"2024-03-11T21:31:26.532550+0000","last_became_active":"2024-03-11T21:25:42.572324+0000","last_became_peered":"2024-03-11T21:25:42.572324+0000","last_unstale":"2024-03-11T21:31:26.532550+0000","last_undegraded":"2024-03-11T21:31:26.532550+0000","last_fullsized":"2024-03-11T21:31:26.532550+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:31:12.634040+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:41.544550+0000","last_clean_scrub_stamp":"2024-03-11T21:31:12.634040+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:32:31.480900+0000","scrub_duration":16,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.0","version":"147'14938","reported_seq":15138,"reported_epoch":148,"state":"active+clean","last_fresh":"2024-03-11T21:31:26.530489+0000","last_change":"2024-03-11T21:31:11.737460+0000","last_active":"2024-03-11T21:31:26.530489+0000","last_peered":"2024-03-11T21:31:26.530489+0000","last_clean":"2024-03-11T21:31:26.530489+0000","last_became_active":"2024-03-11T21:27:36.694232+0000","last_became_peered":"2024-03-11T21:27:36.694232+0000","last_unstale":"2024-03-11T21:31:26.530489+0000","last_undegraded":"2024-03-11T21:31:26.530489+0000","last_fullsized":"2024-03-11T21:31:26.530489+0000","mapping_epoch":99,"log_start":"124'9100","ondisk_log_start":"124'9100","created":99,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"145'14196","last_scrub_stamp":"2024-03-11T21:31:11.737323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:27:35.667522+0000","last_clean_scrub_stamp":"2024-03-11T21:31:11.737323+0000","objects_scrubbed":743,"log_size":5838,"log_dups_size":0,"ondisk_log_size":5838,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:32:28.001263+0000","scrub_duration":161,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14938,"num_write_kb":478016,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3],"acting":[5,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":195,"reported_epoch":148,"state":"active+clean","last_fresh":"2024-03-11T21:31:26.531216+0000","last_change":"2024-03-11T21:31:01.858727+0000","last_active":"2024-03-11T21:31:26.531216+0000","last_peered":"2024-03-11T21:31:26.531216+0000","last_clean":"2024-03-11T21:31:26.531216+0000","last_became_active":"2024-03-11T21:25:45.301416+0000","last_became_peered":"2024-03-11T21:25:45.301416+0000","last_unstale":"2024-03-11T21:31:26.531216+0000","last_undegraded":"2024-03-11T21:31:26.531216+0000","last_fullsized":"2024-03-11T21:31:26.531216+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:31:01.856502+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:43.559452+0000","last_clean_scrub_stamp":"2024-03-11T21:31:01.856502+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:32:30.487243+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":198,"reported_epoch":148,"state":"active+clean","last_fresh":"2024-03-11T21:31:26.531952+0000","last_change":"2024-03-11T21:30:12.625380+0000","last_active":"2024-03-11T21:31:26.531952+0000","last_peered":"2024-03-11T21:31:26.531952+0000","last_clean":"2024-03-11T21:31:26.531952+0000","last_became_active":"2024-03-11T21:25:38.545148+0000","last_became_peered":"2024-03-11T21:25:38.545148+0000","last_unstale":"2024-03-11T21:31:26.531952+0000","last_undegraded":"2024-03-11T21:31:26.531952+0000","last_fullsized":"2024-03-11T21:31:26.531952+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:30:12.625276+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:26:57.333669+0000","last_clean_scrub_stamp":"2024-03-11T21:30:12.625276+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:31:28.500050+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.3","version":"74'19","reported_seq":231,"reported_epoch":148,"state":"active+clean","last_fresh":"2024-03-11T21:31:26.086320+0000","last_change":"2024-03-11T21:30:23.995660+0000","last_active":"2024-03-11T21:31:26.086320+0000","last_peered":"2024-03-11T21:31:26.086320+0000","last_clean":"2024-03-11T21:31:26.086320+0000","last_became_active":"2024-03-11T21:25:40.760594+0000","last_became_peered":"2024-03-11T21:25:40.760594+0000","last_unstale":"2024-03-11T21:31:26.086320+0000","last_undegraded":"2024-03-11T21:31:26.086320+0000","last_fullsized":"2024-03-11T21:31:26.086320+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'19","last_scrub_stamp":"2024-03-11T21:30:23.995503+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:39.525554+0000","last_clean_scrub_stamp":"2024-03-11T21:30:23.995503+0000","objects_scrubbed":8,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:31:28.880234+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":39,"num_read_kb":25,"num_write":22,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":199,"reported_epoch":148,"state":"active+clean","last_fresh":"2024-03-11T21:31:26.531297+0000","last_change":"2024-03-11T21:31:11.390780+0000","last_active":"2024-03-11T21:31:26.531297+0000","last_peered":"2024-03-11T21:31:26.531297+0000","last_clean":"2024-03-11T21:31:26.531297+0000","last_became_active":"2024-03-11T21:25:42.579045+0000","last_became_peered":"2024-03-11T21:25:42.579045+0000","last_unstale":"2024-03-11T21:31:26.531297+0000","last_undegraded":"2024-03-11T21:31:26.531297+0000","last_fullsized":"2024-03-11T21:31:26.531297+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:31:11.390707+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:29:36.918936+0000","last_clean_scrub_stamp":"2024-03-11T21:31:11.390707+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:32:13.656422+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.3","version":"147'15120","reported_seq":15293,"reported_epoch":148,"state":"active+clean","last_fresh":"2024-03-11T21:31:26.088648+0000","last_change":"2024-03-11T21:29:47.452562+0000","last_active":"2024-03-11T21:31:26.088648+0000","last_peered":"2024-03-11T21:31:26.088648+0000","last_clean":"2024-03-11T21:31:26.088648+0000","last_became_active":"2024-03-11T21:27:36.693506+0000","last_became_peered":"2024-03-11T21:27:36.693506+0000","last_unstale":"2024-03-11T21:31:26.088648+0000","last_undegraded":"2024-03-11T21:31:26.088648+0000","last_fullsized":"2024-03-11T21:31:26.088648+0000","mapping_epoch":99,"log_start":"130'10500","ondisk_log_start":"130'10500","created":99,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"124'9352","last_scrub_stamp":"2024-03-11T21:29:47.452230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:27:35.667522+0000","last_clean_scrub_stamp":"2024-03-11T21:29:47.452230+0000","objects_scrubbed":5797,"log_size":4620,"log_dups_size":0,"ondisk_log_size":4620,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"queued for scrub","scrub_duration":1664,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15120,"num_write_kb":483840,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,4],"acting":[3,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":187,"reported_epoch":148,"state":"active+clean","last_fresh":"2024-03-11T21:31:26.083810+0000","last_change":"2024-03-11T21:29:47.154729+0000","last_active":"2024-03-11T21:31:26.083810+0000","last_peered":"2024-03-11T21:31:26.083810+0000","last_clean":"2024-03-11T21:31:26.083810+0000","last_became_active":"2024-03-11T21:25:44.583638+0000","last_became_peered":"2024-03-11T21:25:44.583638+0000","last_unstale":"2024-03-11T21:31:26.083810+0000","last_undegraded":"2024-03-11T21:31:26.083810+0000","last_fullsized":"2024-03-11T21:31:26.083810+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:29:47.154619+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:43.559452+0000","last_clean_scrub_stamp":"2024-03-11T21:29:47.154619+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.6","version":"67'1","reported_seq":200,"reported_epoch":148,"state":"active+clean","last_fresh":"2024-03-11T21:31:26.530605+0000","last_change":"2024-03-11T21:29:33.236180+0000","last_active":"2024-03-11T21:31:26.530605+0000","last_peered":"2024-03-11T21:31:26.530605+0000","last_clean":"2024-03-11T21:31:26.530605+0000","last_became_active":"2024-03-11T21:25:38.536142+0000","last_became_peered":"2024-03-11T21:25:38.536142+0000","last_unstale":"2024-03-11T21:31:26.530605+0000","last_undegraded":"2024-03-11T21:31:26.530605+0000","last_fullsized":"2024-03-11T21:31:26.530605+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'1","last_scrub_stamp":"2024-03-11T21:29:33.236064+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:37.512018+0000","last_clean_scrub_stamp":"2024-03-11T21:29:33.236064+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,4],"acting":[1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.0","version":"93'26","reported_seq":234,"reported_epoch":148,"state":"active+clean","last_fresh":"2024-03-11T21:31:26.086157+0000","last_change":"2024-03-11T21:31:03.246972+0000","last_active":"2024-03-11T21:31:26.086157+0000","last_peered":"2024-03-11T21:31:26.086157+0000","last_clean":"2024-03-11T21:31:26.086157+0000","last_became_active":"2024-03-11T21:25:40.761716+0000","last_became_peered":"2024-03-11T21:25:40.761716+0000","last_unstale":"2024-03-11T21:31:26.086157+0000","last_undegraded":"2024-03-11T21:31:26.086157+0000","last_fullsized":"2024-03-11T21:31:26.086157+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"93'26","last_scrub_stamp":"2024-03-11T21:31:03.244599+0000","last_deep_scrub":"93'26","last_deep_scrub_stamp":"2024-03-11T21:26:40.680324+0000","last_clean_scrub_stamp":"2024-03-11T21:31:03.244599+0000","objects_scrubbed":8,"log_size":26,"log_dups_size":0,"ondisk_log_size":26,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:32:13.957518+0000","scrub_duration":19,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":34,"num_read_kb":22,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":191,"reported_epoch":148,"state":"active+clean","last_fresh":"2024-03-11T21:31:26.083594+0000","last_change":"2024-03-11T21:30:08.277864+0000","last_active":"2024-03-11T21:31:26.083594+0000","last_peered":"2024-03-11T21:31:26.083594+0000","last_clean":"2024-03-11T21:31:26.083594+0000","last_became_active":"2024-03-11T21:25:42.583647+0000","last_became_peered":"2024-03-11T21:25:42.583647+0000","last_unstale":"2024-03-11T21:31:26.083594+0000","last_undegraded":"2024-03-11T21:31:26.083594+0000","last_fullsized":"2024-03-11T21:31:26.083594+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:30:08.277733+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:26:51.863621+0000","last_clean_scrub_stamp":"2024-03-11T21:30:08.277733+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,7],"acting":[4,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.2","version":"147'15262","reported_seq":15448,"reported_epoch":148,"state":"active+clean","last_fresh":"2024-03-11T21:31:26.087131+0000","last_change":"2024-03-11T21:31:25.615274+0000","last_active":"2024-03-11T21:31:26.087131+0000","last_peered":"2024-03-11T21:31:26.087131+0000","last_clean":"2024-03-11T21:31:26.087131+0000","last_became_active":"2024-03-11T21:27:36.692405+0000","last_became_peered":"2024-03-11T21:27:36.692405+0000","last_unstale":"2024-03-11T21:31:26.087131+0000","last_undegraded":"2024-03-11T21:31:26.087131+0000","last_fullsized":"2024-03-11T21:31:26.087131+0000","mapping_epoch":99,"log_start":"130'10623","ondisk_log_start":"130'10623","created":99,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"147'15262","last_scrub_stamp":"2024-03-11T21:31:25.615166+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:27:35.667522+0000","last_clean_scrub_stamp":"2024-03-11T21:31:25.615166+0000","objects_scrubbed":0,"log_size":4639,"log_dups_size":0,"ondisk_log_size":4639,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:32:31.376715+0000","scrub_duration":20,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15262,"num_write_kb":488384,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.3","version":"0'0","reported_seq":186,"reported_epoch":148,"state":"active+clean","last_fresh":"2024-03-11T21:31:26.531711+0000","last_change":"2024-03-11T21:29:42.485840+0000","last_active":"2024-03-11T21:31:26.531711+0000","last_peered":"2024-03-11T21:31:26.531711+0000","last_clean":"2024-03-11T21:31:26.531711+0000","last_became_active":"2024-03-11T21:25:45.302719+0000","last_became_peered":"2024-03-11T21:25:45.302719+0000","last_unstale":"2024-03-11T21:31:26.531711+0000","last_undegraded":"2024-03-11T21:31:26.531711+0000","last_fullsized":"2024-03-11T21:31:26.531711+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:29:42.485740+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:29:42.485740+0000","last_clean_scrub_stamp":"2024-03-11T21:29:42.485740+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,2],"acting":[7,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":207,"reported_epoch":148,"state":"active+clean","last_fresh":"2024-03-11T21:31:26.532173+0000","last_change":"2024-03-11T21:31:01.530536+0000","last_active":"2024-03-11T21:31:26.532173+0000","last_peered":"2024-03-11T21:31:26.532173+0000","last_clean":"2024-03-11T21:31:26.532173+0000","last_became_active":"2024-03-11T21:25:38.540013+0000","last_became_peered":"2024-03-11T21:25:38.540013+0000","last_unstale":"2024-03-11T21:31:26.532173+0000","last_undegraded":"2024-03-11T21:31:26.532173+0000","last_fullsized":"2024-03-11T21:31:26.532173+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:31:01.530407+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:37.512018+0000","last_clean_scrub_stamp":"2024-03-11T21:31:01.530407+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:32:06.743781+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,2],"acting":[6,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.1","version":"74'14","reported_seq":216,"reported_epoch":148,"state":"active+clean","last_fresh":"2024-03-11T21:31:26.083427+0000","last_change":"2024-03-11T21:31:02.486605+0000","last_active":"2024-03-11T21:31:26.083427+0000","last_peered":"2024-03-11T21:31:26.083427+0000","last_clean":"2024-03-11T21:31:26.083427+0000","last_became_active":"2024-03-11T21:25:40.546219+0000","last_became_peered":"2024-03-11T21:25:40.546219+0000","last_unstale":"2024-03-11T21:31:26.083427+0000","last_undegraded":"2024-03-11T21:31:26.083427+0000","last_fullsized":"2024-03-11T21:31:26.083427+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'14","last_scrub_stamp":"2024-03-11T21:31:02.486457+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:39.525554+0000","last_clean_scrub_stamp":"2024-03-11T21:31:02.486457+0000","objects_scrubbed":7,"log_size":14,"log_dups_size":0,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:32:07.493431+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":21,"num_read_kb":14,"num_write":14,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6],"acting":[4,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.0","version":"98'9","reported_seq":270,"reported_epoch":148,"state":"active+clean","last_fresh":"2024-03-11T21:31:26.086102+0000","last_change":"2024-03-11T21:31:21.562516+0000","last_active":"2024-03-11T21:31:26.086102+0000","last_peered":"2024-03-11T21:31:26.086102+0000","last_clean":"2024-03-11T21:31:26.086102+0000","last_became_active":"2024-03-11T21:25:42.576126+0000","last_became_peered":"2024-03-11T21:25:42.576126+0000","last_unstale":"2024-03-11T21:31:26.086102+0000","last_undegraded":"2024-03-11T21:31:26.086102+0000","last_fullsized":"2024-03-11T21:31:26.086102+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"98'9","last_scrub_stamp":"2024-03-11T21:31:21.562395+0000","last_deep_scrub":"98'9","last_deep_scrub_stamp":"2024-03-11T21:27:29.745711+0000","last_clean_scrub_stamp":"2024-03-11T21:31:21.562395+0000","objects_scrubbed":1,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:32:31.326990+0000","scrub_duration":17,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.d","version":"147'15310","reported_seq":15521,"reported_epoch":148,"state":"active+clean","last_fresh":"2024-03-11T21:31:26.530857+0000","last_change":"2024-03-11T21:31:07.816875+0000","last_active":"2024-03-11T21:31:26.530857+0000","last_peered":"2024-03-11T21:31:26.530857+0000","last_clean":"2024-03-11T21:31:26.530857+0000","last_became_active":"2024-03-11T21:27:36.695186+0000","last_became_peered":"2024-03-11T21:27:36.695186+0000","last_unstale":"2024-03-11T21:31:26.530857+0000","last_undegraded":"2024-03-11T21:31:26.530857+0000","last_fullsized":"2024-03-11T21:31:26.530857+0000","mapping_epoch":99,"log_start":"124'9000","ondisk_log_start":"124'9000","created":99,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"144'14363","last_scrub_stamp":"2024-03-11T21:31:07.816767+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:27:35.667522+0000","last_clean_scrub_stamp":"2024-03-11T21:31:07.816767+0000","objects_scrubbed":959,"log_size":6310,"log_dups_size":0,"ondisk_log_size":6310,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:32:36.874959+0000","scrub_duration":402,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":15310,"num_write_kb":489857,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,0],"acting":[6,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.c","version":"0'0","reported_seq":178,"reported_epoch":148,"state":"active+clean","last_fresh":"2024-03-11T21:31:26.087822+0000","last_change":"2024-03-11T21:30:23.986312+0000","last_active":"2024-03-11T21:31:26.087822+0000","last_peered":"2024-03-11T21:31:26.087822+0000","last_clean":"2024-03-11T21:31:26.087822+0000","last_became_active":"2024-03-11T21:25:45.302207+0000","last_became_peered":"2024-03-11T21:25:45.302207+0000","last_unstale":"2024-03-11T21:31:26.087822+0000","last_undegraded":"2024-03-11T21:31:26.087822+0000","last_fullsized":"2024-03-11T21:31:26.087822+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:30:23.986191+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:43.559452+0000","last_clean_scrub_stamp":"2024-03-11T21:30:23.986191+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.f","version":"0'0","reported_seq":198,"reported_epoch":148,"state":"active+clean","last_fresh":"2024-03-11T21:31:26.532329+0000","last_change":"2024-03-11T21:31:24.574334+0000","last_active":"2024-03-11T21:31:26.532329+0000","last_peered":"2024-03-11T21:31:26.532329+0000","last_clean":"2024-03-11T21:31:26.532329+0000","last_became_active":"2024-03-11T21:25:42.575935+0000","last_became_peered":"2024-03-11T21:25:42.575935+0000","last_unstale":"2024-03-11T21:31:26.532329+0000","last_undegraded":"2024-03-11T21:31:26.532329+0000","last_fullsized":"2024-03-11T21:31:26.532329+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:31:24.574239+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:41.544550+0000","last_clean_scrub_stamp":"2024-03-11T21:31:24.574239+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:32:33.744328+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.8","version":"0'0","reported_seq":198,"reported_epoch":148,"state":"active+clean","last_fresh":"2024-03-11T21:31:26.532214+0000","last_change":"2024-03-11T21:30:13.584857+0000","last_active":"2024-03-11T21:31:26.532214+0000","last_peered":"2024-03-11T21:31:26.532214+0000","last_clean":"2024-03-11T21:31:26.532214+0000","last_became_active":"2024-03-11T21:25:38.540910+0000","last_became_peered":"2024-03-11T21:25:38.540910+0000","last_unstale":"2024-03-11T21:31:26.532214+0000","last_undegraded":"2024-03-11T21:31:26.532214+0000","last_fullsized":"2024-03-11T21:31:26.532214+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:30:13.584506+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:37.512018+0000","last_clean_scrub_stamp":"2024-03-11T21:30:13.584506+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:31:39.126385+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,1],"acting":[7,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.e","version":"109'16","reported_seq":219,"reported_epoch":148,"state":"active+clean","last_fresh":"2024-03-11T21:31:26.084370+0000","last_change":"2024-03-11T21:30:56.453809+0000","last_active":"2024-03-11T21:31:26.084370+0000","last_peered":"2024-03-11T21:31:26.084370+0000","last_clean":"2024-03-11T21:31:26.084370+0000","last_became_active":"2024-03-11T21:25:40.552178+0000","last_became_peered":"2024-03-11T21:25:40.552178+0000","last_unstale":"2024-03-11T21:31:26.084370+0000","last_undegraded":"2024-03-11T21:31:26.084370+0000","last_fullsized":"2024-03-11T21:31:26.084370+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"109'16","last_scrub_stamp":"2024-03-11T21:30:56.453670+0000","last_deep_scrub":"109'16","last_deep_scrub_stamp":"2024-03-11T21:28:24.739395+0000","last_clean_scrub_stamp":"2024-03-11T21:30:56.453670+0000","objects_scrubbed":5,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:32:01.053867+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.c","version":"147'15190","reported_seq":15402,"reported_epoch":148,"state":"active+clean","last_fresh":"2024-03-11T21:31:26.531176+0000","last_change":"2024-03-11T21:31:00.837025+0000","last_active":"2024-03-11T21:31:26.531176+0000","last_peered":"2024-03-11T21:31:26.531176+0000","last_clean":"2024-03-11T21:31:26.531176+0000","last_became_active":"2024-03-11T21:27:36.693325+0000","last_became_peered":"2024-03-11T21:27:36.693325+0000","last_unstale":"2024-03-11T21:31:26.531176+0000","last_undegraded":"2024-03-11T21:31:26.531176+0000","last_fullsized":"2024-03-11T21:31:26.531176+0000","mapping_epoch":99,"log_start":"124'8900","ondisk_log_start":"124'8900","created":99,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"144'13774","last_scrub_stamp":"2024-03-11T21:31:00.836924+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:27:35.667522+0000","last_clean_scrub_stamp":"2024-03-11T21:31:00.836924+0000","objects_scrubbed":1423,"log_size":6290,"log_dups_size":0,"ondisk_log_size":6290,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:32:07.800734+0000","scrub_duration":338,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15190,"num_write_kb":486080,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,7],"acting":[6,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.d","version":"0'0","reported_seq":186,"reported_epoch":148,"state":"active+clean","last_fresh":"2024-03-11T21:31:26.531464+0000","last_change":"2024-03-11T21:30:11.993990+0000","last_active":"2024-03-11T21:31:26.531464+0000","last_peered":"2024-03-11T21:31:26.531464+0000","last_clean":"2024-03-11T21:31:26.531464+0000","last_became_active":"2024-03-11T21:25:44.590333+0000","last_became_peered":"2024-03-11T21:25:44.590333+0000","last_unstale":"2024-03-11T21:31:26.531464+0000","last_undegraded":"2024-03-11T21:31:26.531464+0000","last_fullsized":"2024-03-11T21:31:26.531464+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:30:11.993862+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:43.559452+0000","last_clean_scrub_stamp":"2024-03-11T21:30:11.993862+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.e","version":"74'8","reported_seq":270,"reported_epoch":148,"state":"active+clean","last_fresh":"2024-03-11T21:31:26.084231+0000","last_change":"2024-03-11T21:31:05.564402+0000","last_active":"2024-03-11T21:31:26.084231+0000","last_peered":"2024-03-11T21:31:26.084231+0000","last_clean":"2024-03-11T21:31:26.084231+0000","last_became_active":"2024-03-11T21:25:42.576742+0000","last_became_peered":"2024-03-11T21:25:42.576742+0000","last_unstale":"2024-03-11T21:31:26.084231+0000","last_undegraded":"2024-03-11T21:31:26.084231+0000","last_fullsized":"2024-03-11T21:31:26.084231+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'8","last_scrub_stamp":"2024-03-11T21:31:05.564320+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:41.544550+0000","last_clean_scrub_stamp":"2024-03-11T21:31:05.564320+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:32:33.999249+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,0],"acting":[4,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.9","version":"0'0","reported_seq":199,"reported_epoch":148,"state":"active+clean","last_fresh":"2024-03-11T21:31:26.531031+0000","last_change":"2024-03-11T21:29:18.412164+0000","last_active":"2024-03-11T21:31:26.531031+0000","last_peered":"2024-03-11T21:31:26.531031+0000","last_clean":"2024-03-11T21:31:26.531031+0000","last_became_active":"2024-03-11T21:25:38.540976+0000","last_became_peered":"2024-03-11T21:25:38.540976+0000","last_unstale":"2024-03-11T21:31:26.531031+0000","last_undegraded":"2024-03-11T21:31:26.531031+0000","last_fullsized":"2024-03-11T21:31:26.531031+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:29:18.411652+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:26:50.991712+0000","last_clean_scrub_stamp":"2024-03-11T21:29:18.411652+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,3],"acting":[1,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.f","version":"74'15","reported_seq":172,"reported_epoch":148,"state":"active+clean","last_fresh":"2024-03-11T21:31:26.530537+0000","last_change":"2024-03-11T21:30:13.987515+0000","last_active":"2024-03-11T21:31:26.530537+0000","last_peered":"2024-03-11T21:31:26.530537+0000","last_clean":"2024-03-11T21:31:26.530537+0000","last_became_active":"2024-03-11T21:26:11.216224+0000","last_became_peered":"2024-03-11T21:26:11.216224+0000","last_unstale":"2024-03-11T21:31:26.530537+0000","last_undegraded":"2024-03-11T21:31:26.530537+0000","last_fullsized":"2024-03-11T21:31:26.530537+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'15","last_scrub_stamp":"2024-03-11T21:30:13.987375+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:39.525554+0000","last_clean_scrub_stamp":"2024-03-11T21:30:13.987375+0000","objects_scrubbed":7,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:31:36.359932+0000","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,4],"acting":[2,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.f","version":"147'18605","reported_seq":18907,"reported_epoch":148,"state":"active+clean","last_fresh":"2024-03-11T21:31:26.086646+0000","last_change":"2024-03-11T21:30:13.830365+0000","last_active":"2024-03-11T21:31:26.086646+0000","last_peered":"2024-03-11T21:31:26.086646+0000","last_clean":"2024-03-11T21:31:26.086646+0000","last_became_active":"2024-03-11T21:27:36.694032+0000","last_became_peered":"2024-03-11T21:27:36.694032+0000","last_unstale":"2024-03-11T21:31:26.086646+0000","last_undegraded":"2024-03-11T21:31:26.086646+0000","last_fullsized":"2024-03-11T21:31:26.086646+0000","mapping_epoch":99,"log_start":"126'13102","ondisk_log_start":"126'13102","created":99,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"131'14092","last_scrub_stamp":"2024-03-11T21:30:13.830084+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:27:35.667522+0000","last_clean_scrub_stamp":"2024-03-11T21:30:13.830084+0000","objects_scrubbed":4557,"log_size":5503,"log_dups_size":0,"ondisk_log_size":5503,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:31:42.886014+0000","scrub_duration":2014,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15402,"num_write_kb":492864,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.e","version":"0'0","reported_seq":187,"reported_epoch":148,"state":"active+clean","last_fresh":"2024-03-11T21:31:26.084091+0000","last_change":"2024-03-11T21:31:09.597590+0000","last_active":"2024-03-11T21:31:26.084091+0000","last_peered":"2024-03-11T21:31:26.084091+0000","last_clean":"2024-03-11T21:31:26.084091+0000","last_became_active":"2024-03-11T21:25:44.590198+0000","last_became_peered":"2024-03-11T21:25:44.590198+0000","last_unstale":"2024-03-11T21:31:26.084091+0000","last_undegraded":"2024-03-11T21:31:26.084091+0000","last_fullsized":"2024-03-11T21:31:26.084091+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:31:09.597448+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:43.559452+0000","last_clean_scrub_stamp":"2024-03-11T21:31:09.597448+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:32:30.206261+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.d","version":"109'9","reported_seq":270,"reported_epoch":148,"state":"active+clean","last_fresh":"2024-03-11T21:31:26.530674+0000","last_change":"2024-03-11T21:30:20.940788+0000","last_active":"2024-03-11T21:31:26.530674+0000","last_peered":"2024-03-11T21:31:26.530674+0000","last_clean":"2024-03-11T21:31:26.530674+0000","last_became_active":"2024-03-11T21:25:42.575583+0000","last_became_peered":"2024-03-11T21:25:42.575583+0000","last_unstale":"2024-03-11T21:31:26.530674+0000","last_undegraded":"2024-03-11T21:31:26.530674+0000","last_fullsized":"2024-03-11T21:31:26.530674+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"109'9","last_scrub_stamp":"2024-03-11T21:30:20.940485+0000","last_deep_scrub":"109'9","last_deep_scrub_stamp":"2024-03-11T21:28:27.082975+0000","last_clean_scrub_stamp":"2024-03-11T21:30:20.940485+0000","objects_scrubbed":1,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:31:43.649095+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,5],"acting":[2,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.a","version":"0'0","reported_seq":207,"reported_epoch":148,"state":"active+clean","last_fresh":"2024-03-11T21:31:26.530363+0000","last_change":"2024-03-11T21:31:06.898504+0000","last_active":"2024-03-11T21:31:26.530363+0000","last_peered":"2024-03-11T21:31:26.530363+0000","last_clean":"2024-03-11T21:31:26.530363+0000","last_became_active":"2024-03-11T21:25:38.541045+0000","last_became_peered":"2024-03-11T21:25:38.541045+0000","last_unstale":"2024-03-11T21:31:26.530363+0000","last_undegraded":"2024-03-11T21:31:26.530363+0000","last_fullsized":"2024-03-11T21:31:26.530363+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:31:06.898394+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:37.512018+0000","last_clean_scrub_stamp":"2024-03-11T21:31:06.898394+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:32:14.529726+0000","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,7],"acting":[1,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.c","version":"74'10","reported_seq":210,"reported_epoch":148,"state":"active+clean","last_fresh":"2024-03-11T21:31:26.083775+0000","last_change":"2024-03-11T21:30:57.495457+0000","last_active":"2024-03-11T21:31:26.083775+0000","last_peered":"2024-03-11T21:31:26.083775+0000","last_clean":"2024-03-11T21:31:26.083775+0000","last_became_active":"2024-03-11T21:25:40.550885+0000","last_became_peered":"2024-03-11T21:25:40.550885+0000","last_unstale":"2024-03-11T21:31:26.083775+0000","last_undegraded":"2024-03-11T21:31:26.083775+0000","last_fullsized":"2024-03-11T21:31:26.083775+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'10","last_scrub_stamp":"2024-03-11T21:30:57.495302+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:39.525554+0000","last_clean_scrub_stamp":"2024-03-11T21:30:57.495302+0000","objects_scrubbed":5,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:32:04.367621+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.e","version":"147'15338","reported_seq":15539,"reported_epoch":148,"state":"active+clean","last_fresh":"2024-03-11T21:31:26.530413+0000","last_change":"2024-03-11T21:31:16.547286+0000","last_active":"2024-03-11T21:31:26.530413+0000","last_peered":"2024-03-11T21:31:26.530413+0000","last_clean":"2024-03-11T21:31:26.530413+0000","last_became_active":"2024-03-11T21:27:36.691800+0000","last_became_peered":"2024-03-11T21:27:36.691800+0000","last_unstale":"2024-03-11T21:31:26.530413+0000","last_undegraded":"2024-03-11T21:31:26.530413+0000","last_fullsized":"2024-03-11T21:31:26.530413+0000","mapping_epoch":99,"log_start":"124'8600","ondisk_log_start":"124'8600","created":99,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"146'14900","last_scrub_stamp":"2024-03-11T21:31:16.547085+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:27:35.667522+0000","last_clean_scrub_stamp":"2024-03-11T21:31:16.547085+0000","objects_scrubbed":441,"log_size":6738,"log_dups_size":0,"ondisk_log_size":6738,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:32:26.495904+0000","scrub_duration":185,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15338,"num_write_kb":490816,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,0],"acting":[2,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.f","version":"0'0","reported_seq":186,"reported_epoch":148,"state":"active+clean","last_fresh":"2024-03-11T21:31:26.530691+0000","last_change":"2024-03-11T21:30:21.979887+0000","last_active":"2024-03-11T21:31:26.530691+0000","last_peered":"2024-03-11T21:31:26.530691+0000","last_clean":"2024-03-11T21:31:26.530691+0000","last_became_active":"2024-03-11T21:25:44.593414+0000","last_became_peered":"2024-03-11T21:25:44.593414+0000","last_unstale":"2024-03-11T21:31:26.530691+0000","last_undegraded":"2024-03-11T21:31:26.530691+0000","last_fullsized":"2024-03-11T21:31:26.530691+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:30:21.979536+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:27:03.010969+0000","last_clean_scrub_stamp":"2024-03-11T21:30:21.979536+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:31:47.911652+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,4],"acting":[2,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":191,"reported_epoch":148,"state":"active+clean","last_fresh":"2024-03-11T21:31:26.531020+0000","last_change":"2024-03-11T21:30:22.750174+0000","last_active":"2024-03-11T21:31:26.531020+0000","last_peered":"2024-03-11T21:31:26.531020+0000","last_clean":"2024-03-11T21:31:26.531020+0000","last_became_active":"2024-03-11T21:25:42.579409+0000","last_became_peered":"2024-03-11T21:25:42.579409+0000","last_unstale":"2024-03-11T21:31:26.531020+0000","last_undegraded":"2024-03-11T21:31:26.531020+0000","last_fullsized":"2024-03-11T21:31:26.531020+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:30:22.750053+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:30:22.750053+0000","last_clean_scrub_stamp":"2024-03-11T21:30:22.750053+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:31:45.145778+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.b","version":"0'0","reported_seq":206,"reported_epoch":148,"state":"active+clean","last_fresh":"2024-03-11T21:31:26.531339+0000","last_change":"2024-03-11T21:31:20.539366+0000","last_active":"2024-03-11T21:31:26.531339+0000","last_peered":"2024-03-11T21:31:26.531339+0000","last_clean":"2024-03-11T21:31:26.531339+0000","last_became_active":"2024-03-11T21:25:38.543377+0000","last_became_peered":"2024-03-11T21:25:38.543377+0000","last_unstale":"2024-03-11T21:31:26.531339+0000","last_undegraded":"2024-03-11T21:31:26.531339+0000","last_fullsized":"2024-03-11T21:31:26.531339+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:31:20.539243+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:37.512018+0000","last_clean_scrub_stamp":"2024-03-11T21:31:20.539243+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:32:38.890953+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,5],"acting":[7,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.d","version":"74'17","reported_seq":223,"reported_epoch":148,"state":"active+clean","last_fresh":"2024-03-11T21:31:26.084074+0000","last_change":"2024-03-11T21:30:20.287636+0000","last_active":"2024-03-11T21:31:26.084074+0000","last_peered":"2024-03-11T21:31:26.084074+0000","last_clean":"2024-03-11T21:31:26.084074+0000","last_became_active":"2024-03-11T21:25:40.553454+0000","last_became_peered":"2024-03-11T21:25:40.553454+0000","last_unstale":"2024-03-11T21:31:26.084074+0000","last_undegraded":"2024-03-11T21:31:26.084074+0000","last_fullsized":"2024-03-11T21:31:26.084074+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'17","last_scrub_stamp":"2024-03-11T21:30:20.287535+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:39.525554+0000","last_clean_scrub_stamp":"2024-03-11T21:30:20.287535+0000","objects_scrubbed":8,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:31:40.112321+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":29,"num_read_kb":19,"num_write":18,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,1],"acting":[4,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.9","version":"147'17354","reported_seq":17605,"reported_epoch":148,"state":"active+clean","last_fresh":"2024-03-11T21:31:26.086081+0000","last_change":"2024-03-11T21:30:14.522862+0000","last_active":"2024-03-11T21:31:26.086081+0000","last_peered":"2024-03-11T21:31:26.086081+0000","last_clean":"2024-03-11T21:31:26.086081+0000","last_became_active":"2024-03-11T21:27:36.692978+0000","last_became_peered":"2024-03-11T21:27:36.692978+0000","last_unstale":"2024-03-11T21:31:26.086081+0000","last_undegraded":"2024-03-11T21:31:26.086081+0000","last_fullsized":"2024-03-11T21:31:26.086081+0000","mapping_epoch":99,"log_start":"126'11801","ondisk_log_start":"126'11801","created":99,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"131'13017","last_scrub_stamp":"2024-03-11T21:30:14.519102+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:27:35.667522+0000","last_clean_scrub_stamp":"2024-03-11T21:30:14.519102+0000","objects_scrubbed":4381,"log_size":5553,"log_dups_size":0,"ondisk_log_size":5553,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:31:44.028644+0000","scrub_duration":1664,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15168,"num_write_kb":485376,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":186,"reported_epoch":148,"state":"active+clean","last_fresh":"2024-03-11T21:31:26.530774+0000","last_change":"2024-03-11T21:30:16.650959+0000","last_active":"2024-03-11T21:31:26.530774+0000","last_peered":"2024-03-11T21:31:26.530774+0000","last_clean":"2024-03-11T21:31:26.530774+0000","last_became_active":"2024-03-11T21:25:44.589973+0000","last_became_peered":"2024-03-11T21:25:44.589973+0000","last_unstale":"2024-03-11T21:31:26.530774+0000","last_undegraded":"2024-03-11T21:31:26.530774+0000","last_fullsized":"2024-03-11T21:31:26.530774+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:30:16.650854+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:28:37.561826+0000","last_clean_scrub_stamp":"2024-03-11T21:30:16.650854+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:31:43.938865+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,3],"acting":[7,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":190,"reported_epoch":148,"state":"active+clean","last_fresh":"2024-03-11T21:31:26.530307+0000","last_change":"2024-03-11T21:30:57.336406+0000","last_active":"2024-03-11T21:31:26.530307+0000","last_peered":"2024-03-11T21:31:26.530307+0000","last_clean":"2024-03-11T21:31:26.530307+0000","last_became_active":"2024-03-11T21:25:42.574515+0000","last_became_peered":"2024-03-11T21:25:42.574515+0000","last_unstale":"2024-03-11T21:31:26.530307+0000","last_undegraded":"2024-03-11T21:31:26.530307+0000","last_fullsized":"2024-03-11T21:31:26.530307+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:30:57.336188+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:41.544550+0000","last_clean_scrub_stamp":"2024-03-11T21:30:57.336188+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:32:12.876558+0000","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,5],"acting":[2,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.c","version":"0'0","reported_seq":198,"reported_epoch":148,"state":"active+clean","last_fresh":"2024-03-11T21:31:26.530352+0000","last_change":"2024-03-11T21:31:09.381415+0000","last_active":"2024-03-11T21:31:26.530352+0000","last_peered":"2024-03-11T21:31:26.530352+0000","last_clean":"2024-03-11T21:31:26.530352+0000","last_became_active":"2024-03-11T21:25:38.533240+0000","last_became_peered":"2024-03-11T21:25:38.533240+0000","last_unstale":"2024-03-11T21:31:26.530352+0000","last_undegraded":"2024-03-11T21:31:26.530352+0000","last_fullsized":"2024-03-11T21:31:26.530352+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:31:09.381301+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:29:30.279910+0000","last_clean_scrub_stamp":"2024-03-11T21:31:09.381301+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:32:38.869288+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,0],"acting":[2,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.a","version":"109'28","reported_seq":243,"reported_epoch":148,"state":"active+clean","last_fresh":"2024-03-11T21:31:26.531071+0000","last_change":"2024-03-11T21:31:22.522787+0000","last_active":"2024-03-11T21:31:26.531071+0000","last_peered":"2024-03-11T21:31:26.531071+0000","last_clean":"2024-03-11T21:31:26.531071+0000","last_became_active":"2024-03-11T21:25:40.760698+0000","last_became_peered":"2024-03-11T21:25:40.760698+0000","last_unstale":"2024-03-11T21:31:26.531071+0000","last_undegraded":"2024-03-11T21:31:26.531071+0000","last_fullsized":"2024-03-11T21:31:26.531071+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"109'28","last_scrub_stamp":"2024-03-11T21:31:22.522676+0000","last_deep_scrub":"109'28","last_deep_scrub_stamp":"2024-03-11T21:28:24.016389+0000","last_clean_scrub_stamp":"2024-03-11T21:31:22.522676+0000","objects_scrubbed":9,"log_size":28,"log_dups_size":0,"ondisk_log_size":28,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:32:31.422599+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":32,"num_read_kb":21,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,7],"acting":[6,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.8","version":"147'15010","reported_seq":15217,"reported_epoch":148,"state":"active+clean","last_fresh":"2024-03-11T21:31:26.529862+0000","last_change":"2024-03-11T21:30:58.801387+0000","last_active":"2024-03-11T21:31:26.529862+0000","last_peered":"2024-03-11T21:31:26.529862+0000","last_clean":"2024-03-11T21:31:26.529862+0000","last_became_active":"2024-03-11T21:27:36.693060+0000","last_became_peered":"2024-03-11T21:27:36.693060+0000","last_unstale":"2024-03-11T21:31:26.529862+0000","last_undegraded":"2024-03-11T21:31:26.529862+0000","last_fullsized":"2024-03-11T21:31:26.529862+0000","mapping_epoch":99,"log_start":"124'8300","ondisk_log_start":"124'8300","created":99,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"144'13479","last_scrub_stamp":"2024-03-11T21:30:58.800987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:27:35.667522+0000","last_clean_scrub_stamp":"2024-03-11T21:30:58.800987+0000","objects_scrubbed":1545,"log_size":6710,"log_dups_size":0,"ondisk_log_size":6710,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:32:27.160455+0000","scrub_duration":524,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15010,"num_write_kb":480320,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,4],"acting":[2,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":187,"reported_epoch":148,"state":"active+clean","last_fresh":"2024-03-11T21:31:26.086518+0000","last_change":"2024-03-11T21:31:10.397757+0000","last_active":"2024-03-11T21:31:26.086518+0000","last_peered":"2024-03-11T21:31:26.086518+0000","last_clean":"2024-03-11T21:31:26.086518+0000","last_became_active":"2024-03-11T21:25:44.584154+0000","last_became_peered":"2024-03-11T21:25:44.584154+0000","last_unstale":"2024-03-11T21:31:26.086518+0000","last_undegraded":"2024-03-11T21:31:26.086518+0000","last_fullsized":"2024-03-11T21:31:26.086518+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:31:10.397515+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:27:11.361913+0000","last_clean_scrub_stamp":"2024-03-11T21:31:10.397515+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:32:15.530582+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":190,"reported_epoch":148,"state":"active+clean","last_fresh":"2024-03-11T21:31:26.531214+0000","last_change":"2024-03-11T21:31:11.390022+0000","last_active":"2024-03-11T21:31:26.531214+0000","last_peered":"2024-03-11T21:31:26.531214+0000","last_clean":"2024-03-11T21:31:26.531214+0000","last_became_active":"2024-03-11T21:25:42.576679+0000","last_became_peered":"2024-03-11T21:25:42.576679+0000","last_unstale":"2024-03-11T21:31:26.531214+0000","last_undegraded":"2024-03-11T21:31:26.531214+0000","last_fullsized":"2024-03-11T21:31:26.531214+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:31:11.389869+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:41.544550+0000","last_clean_scrub_stamp":"2024-03-11T21:31:11.389869+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:32:39.283658+0000","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,3],"acting":[2,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.d","version":"0'0","reported_seq":199,"reported_epoch":148,"state":"active+clean","last_fresh":"2024-03-11T21:31:26.531379+0000","last_change":"2024-03-11T21:29:20.439881+0000","last_active":"2024-03-11T21:31:26.531379+0000","last_peered":"2024-03-11T21:31:26.531379+0000","last_clean":"2024-03-11T21:31:26.531379+0000","last_became_active":"2024-03-11T21:25:38.539681+0000","last_became_peered":"2024-03-11T21:25:38.539681+0000","last_unstale":"2024-03-11T21:31:26.531379+0000","last_undegraded":"2024-03-11T21:31:26.531379+0000","last_fullsized":"2024-03-11T21:31:26.531379+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:29:20.439742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:37.512018+0000","last_clean_scrub_stamp":"2024-03-11T21:29:20.439742+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,3],"acting":[1,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.b","version":"74'9","reported_seq":211,"reported_epoch":148,"state":"active+clean","last_fresh":"2024-03-11T21:31:26.086509+0000","last_change":"2024-03-11T21:31:20.485241+0000","last_active":"2024-03-11T21:31:26.086509+0000","last_peered":"2024-03-11T21:31:26.086509+0000","last_clean":"2024-03-11T21:31:26.086509+0000","last_became_active":"2024-03-11T21:25:40.549612+0000","last_became_peered":"2024-03-11T21:25:40.549612+0000","last_unstale":"2024-03-11T21:31:26.086509+0000","last_undegraded":"2024-03-11T21:31:26.086509+0000","last_fullsized":"2024-03-11T21:31:26.086509+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'9","last_scrub_stamp":"2024-03-11T21:31:20.485090+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:39.525554+0000","last_clean_scrub_stamp":"2024-03-11T21:31:20.485090+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:32:28.057879+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.b","version":"147'15238","reported_seq":15453,"reported_epoch":148,"state":"active+clean","last_fresh":"2024-03-11T21:31:26.530756+0000","last_change":"2024-03-11T21:31:03.330969+0000","last_active":"2024-03-11T21:31:26.530756+0000","last_peered":"2024-03-11T21:31:26.530756+0000","last_clean":"2024-03-11T21:31:26.530756+0000","last_became_active":"2024-03-11T21:27:36.691322+0000","last_became_peered":"2024-03-11T21:27:36.691322+0000","last_unstale":"2024-03-11T21:31:26.530756+0000","last_undegraded":"2024-03-11T21:31:26.530756+0000","last_fullsized":"2024-03-11T21:31:26.530756+0000","mapping_epoch":99,"log_start":"128'10100","ondisk_log_start":"128'10100","created":99,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"144'13944","last_scrub_stamp":"2024-03-11T21:31:03.330626+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:27:35.667522+0000","last_clean_scrub_stamp":"2024-03-11T21:31:03.330626+0000","objects_scrubbed":1311,"log_size":5138,"log_dups_size":0,"ondisk_log_size":5138,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:32:09.099528+0000","scrub_duration":469,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15238,"num_write_kb":487616,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,6],"acting":[1,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":92,"reported_epoch":148,"state":"active+clean","last_fresh":"2024-03-11T21:31:26.083305+0000","last_change":"2024-03-11T21:30:25.380496+0000","last_active":"2024-03-11T21:31:26.083305+0000","last_peered":"2024-03-11T21:31:26.083305+0000","last_clean":"2024-03-11T21:31:26.083305+0000","last_became_active":"2024-03-11T21:28:45.372452+0000","last_became_peered":"2024-03-11T21:28:45.372452+0000","last_unstale":"2024-03-11T21:31:26.083305+0000","last_undegraded":"2024-03-11T21:31:26.083305+0000","last_fullsized":"2024-03-11T21:31:26.083305+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:30:25.380313+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:43.559452+0000","last_clean_scrub_stamp":"2024-03-11T21:30:25.380313+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:31:34.770501+0000","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,3],"acting":[4,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.e","version":"0'0","reported_seq":206,"reported_epoch":148,"state":"active+clean","last_fresh":"2024-03-11T21:31:26.530566+0000","last_change":"2024-03-11T21:31:17.409333+0000","last_active":"2024-03-11T21:31:26.530566+0000","last_peered":"2024-03-11T21:31:26.530566+0000","last_clean":"2024-03-11T21:31:26.530566+0000","last_became_active":"2024-03-11T21:25:38.539954+0000","last_became_peered":"2024-03-11T21:25:38.539954+0000","last_unstale":"2024-03-11T21:31:26.530566+0000","last_undegraded":"2024-03-11T21:31:26.530566+0000","last_fullsized":"2024-03-11T21:31:26.530566+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:31:17.408572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:37.512018+0000","last_clean_scrub_stamp":"2024-03-11T21:31:17.408572+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:32:41.434864+0000","scrub_duration":21,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,7],"acting":[2,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.8","version":"109'22","reported_seq":226,"reported_epoch":148,"state":"active+clean","last_fresh":"2024-03-11T21:31:26.532278+0000","last_change":"2024-03-11T21:29:44.835802+0000","last_active":"2024-03-11T21:31:26.532278+0000","last_peered":"2024-03-11T21:31:26.532278+0000","last_clean":"2024-03-11T21:31:26.532278+0000","last_became_active":"2024-03-11T21:25:40.761938+0000","last_became_peered":"2024-03-11T21:25:40.761938+0000","last_unstale":"2024-03-11T21:31:26.532278+0000","last_undegraded":"2024-03-11T21:31:26.532278+0000","last_fullsized":"2024-03-11T21:31:26.532278+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"109'22","last_scrub_stamp":"2024-03-11T21:29:44.835663+0000","last_deep_scrub":"109'22","last_deep_scrub_stamp":"2024-03-11T21:28:25.743058+0000","last_clean_scrub_stamp":"2024-03-11T21:29:44.835663+0000","objects_scrubbed":7,"log_size":22,"log_dups_size":0,"ondisk_log_size":22,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.9","version":"94'9","reported_seq":281,"reported_epoch":148,"state":"active+clean","last_fresh":"2024-03-11T21:31:26.531273+0000","last_change":"2024-03-11T21:31:19.528959+0000","last_active":"2024-03-11T21:31:26.531273+0000","last_peered":"2024-03-11T21:31:26.531273+0000","last_clean":"2024-03-11T21:31:26.531273+0000","last_became_active":"2024-03-11T21:25:42.584461+0000","last_became_peered":"2024-03-11T21:25:42.584461+0000","last_unstale":"2024-03-11T21:31:26.531273+0000","last_undegraded":"2024-03-11T21:31:26.531273+0000","last_fullsized":"2024-03-11T21:31:26.531273+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"94'9","last_scrub_stamp":"2024-03-11T21:31:19.528840+0000","last_deep_scrub":"94'9","last_deep_scrub_stamp":"2024-03-11T21:26:53.422552+0000","last_clean_scrub_stamp":"2024-03-11T21:31:19.528840+0000","objects_scrubbed":1,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:32:48.650755+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":7,"num_pg":16,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":243818,"num_write_kb":7802113,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":89381,"ondisk_log_size":89381,"up":48,"acting":48,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":1531,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":24,"num_read_kb":24,"num_write":10,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":4593,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":6,"ondisk_log_size":6,"up":96,"acting":96,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":212,"num_read_kb":306,"num_write":179,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1771104,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1781760,"data_stored":1771104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":128,"ondisk_log_size":128,"up":3,"acting":3,"num_store_stats":3},{"poolid":3,"num_pg":3,"stat_sum":{"num_bytes":408,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":347,"num_read_kb":342,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1224,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":9,"acting":9,"num_store_stats":7},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":3702,"num_objects":178,"num_object_clones":0,"num_object_copies":534,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":178,"num_whiteouts":0,"num_read":698,"num_read_kb":455,"num_write":417,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":417792,"data_stored":11106,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":456,"ondisk_log_size":456,"up":96,"acting":96,"num_store_stats":8},{"poolid":5,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":68,"ondisk_log_size":68,"up":96,"acting":96,"num_store_stats":8},{"poolid":6,"num_pg":32,"stat_sum":{"num_bytes":383,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1149,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":96,"acting":96,"num_store_stats":8}],"osd_stats":[{"osd":7,"up_from":60,"seq":257698037834,"num_pgs":61,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":104740,"kb_used_data":1368,"kb_used_omap":1,"kb_used_meta":103358,"kb_avail":93640412,"statfs":{"total":95995035648,"available":95887781888,"internally_reserved":0,"allocated":1400832,"data_stored":1005111,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":105839050},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":50,"seq":214748364880,"num_pgs":48,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":108896,"kb_used_data":1364,"kb_used_omap":1,"kb_used_meta":107518,"kb_avail":93636256,"statfs":{"total":95995035648,"available":95883526144,"internally_reserved":0,"allocated":1396736,"data_stored":1005404,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":110098886},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574954,"num_pgs":59,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":126396,"kb_used_data":816,"kb_used_omap":1,"kb_used_meta":125566,"kb_avail":93618756,"statfs":{"total":95995035648,"available":95865606144,"internally_reserved":0,"allocated":835584,"data_stored":415144,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":128580037},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738479,"num_pgs":55,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":163048,"kb_used_data":1372,"kb_used_omap":1,"kb_used_meta":161662,"kb_avail":93582104,"statfs":{"total":95995035648,"available":95828074496,"internally_reserved":0,"allocated":1404928,"data_stored":1005328,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":165542341},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411428,"num_pgs":45,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":130516,"kb_used_data":776,"kb_used_omap":1,"kb_used_meta":129726,"kb_avail":93614636,"statfs":{"total":95995035648,"available":95861387264,"internally_reserved":0,"allocated":794624,"data_stored":414368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":132839877},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":26,"seq":111669149792,"num_pgs":65,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":162492,"kb_used_data":816,"kb_used_omap":1,"kb_used_meta":161662,"kb_avail":93582660,"statfs":{"total":95995035648,"available":95828643840,"internally_reserved":0,"allocated":835584,"data_stored":415478,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":165542343},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":32,"seq":137438953561,"num_pgs":59,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":100012,"kb_used_data":800,"kb_used_omap":1,"kb_used_meta":99198,"kb_avail":93645140,"statfs":{"total":95995035648,"available":95892623360,"internally_reserved":0,"allocated":819200,"data_stored":414652,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":101579205},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":41,"seq":176093659220,"num_pgs":52,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":81944,"kb_used_data":780,"kb_used_omap":1,"kb_used_meta":81150,"kb_avail":93663208,"statfs":{"total":95995035648,"available":95911124992,"internally_reserved":0,"allocated":798720,"data_stored":414419,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":83098053},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1049,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":46,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":957,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":574,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":436,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1003,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":528,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":1320,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":81920,"data_stored":2118,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":1018,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":1898,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":1474,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":990,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":1034,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":1254,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":370,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":383,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":370,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-11T21:31:35.668 INFO:tasks.cephadm.ceph_manager.ceph:recovered! 2024-03-11T21:31:35.669 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-03-11T21:31:35.669 INFO:tasks.thrashosds.thrasher:Reweighting osd 7 to 0.12537148830939518 2024-03-11T21:31:35.669 DEBUG:teuthology.orchestra.run.smithi092:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid 00db0016-dfed-11ee-95c8-87774f69a715 -- ceph osd reweight 7 0.12537148830939518 2024-03-11T21:31:35.714 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:31:35.738 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:31:35.740 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:31:35.740 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:31:35.841 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:31:35.865 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:31:35.865 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:31:35.866 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:31:35.967 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:31:35.990 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:31:35.991 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:31:35.991 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:31:36.092 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:31:36.117 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:31:36.117 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:31:36.118 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:31:36.141 INFO:teuthology.orchestra.run.smithi092.stderr:Inferring config /var/lib/ceph/00db0016-dfed-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T21:31:36.141 INFO:teuthology.orchestra.run.smithi092.stderr:Inferring config /var/lib/ceph/00db0016-dfed-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T21:31:36.218 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:31:36.224 INFO:teuthology.orchestra.run.smithi092.stderr:pool 'unique_pool_0' removed 2024-03-11T21:31:36.262 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:31:36.262 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:31:36.262 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:31:36.363 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:31:36.388 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:31:36.389 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:31:36.389 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:31:36.485 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:31:36 smithi092 ceph-mon[29572]: pgmap v222: 148 pgs: 148 active+clean; 582 KiB data, 955 MiB used, 714 GiB / 715 GiB avail; 1.1 KiB/s rd, 0 B/s wr, 132 op/s 2024-03-11T21:31:36.485 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:31:36 smithi092 ceph-mon[29572]: from='client.? v1:172.21.15.92:0/3838210494' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "unique_pool_0", "pool2": "unique_pool_0", "yes_i_really_really_mean_it": true}]: dispatch 2024-03-11T21:31:36.486 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:31:36 smithi092 ceph-mon[37563]: pgmap v222: 148 pgs: 148 active+clean; 582 KiB data, 955 MiB used, 714 GiB / 715 GiB avail; 1.1 KiB/s rd, 0 B/s wr, 132 op/s 2024-03-11T21:31:36.486 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:31:36 smithi092 ceph-mon[37563]: from='client.? v1:172.21.15.92:0/3838210494' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "unique_pool_0", "pool2": "unique_pool_0", "yes_i_really_really_mean_it": true}]: dispatch 2024-03-11T21:31:36.490 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:31:36.515 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:31:36.516 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:31:36.516 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:31:36.576 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:31:36 smithi097 ceph-mon[33317]: pgmap v222: 148 pgs: 148 active+clean; 582 KiB data, 955 MiB used, 714 GiB / 715 GiB avail; 1.1 KiB/s rd, 0 B/s wr, 132 op/s 2024-03-11T21:31:36.577 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:31:36 smithi097 ceph-mon[33317]: from='client.? v1:172.21.15.92:0/3838210494' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "unique_pool_0", "pool2": "unique_pool_0", "yes_i_really_really_mean_it": true}]: dispatch 2024-03-11T21:31:36.616 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:31:36.643 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:31:36.643 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:31:36.644 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:31:36.744 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:31:36.769 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:31:36.769 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:31:36.769 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:31:36.870 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:31:36.894 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:31:36.895 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:31:36.895 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:31:36.996 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:31:37.026 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:31:37.026 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:31:37.027 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:31:37.127 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:31:37.152 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:31:37.152 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:31:37.152 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:31:37.253 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:31:37.277 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:31:37.277 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:31:37.278 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:31:37.378 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:31:37.402 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:31:37.403 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:31:37.403 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:31:37.504 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:31:37.528 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:31:37.528 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:31:37.528 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:31:37.529 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:31:37 smithi092 ceph-mon[29572]: from='client.? v1:172.21.15.92:0/3838210494' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "unique_pool_0", "pool2": "unique_pool_0", "yes_i_really_really_mean_it": true}]': finished 2024-03-11T21:31:37.529 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:31:37 smithi092 ceph-mon[29572]: osdmap e150: 8 total, 8 up, 8 in 2024-03-11T21:31:37.529 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:31:37 smithi092 ceph-mon[29572]: from='mgr.14607 ' entity='mgr.y' 2024-03-11T21:31:37.529 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:31:37 smithi092 ceph-mon[29572]: from='mgr.14607 v1:172.21.15.92:0/1702717597' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T21:31:37.529 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:31:37 smithi092 ceph-mon[37563]: from='client.? v1:172.21.15.92:0/3838210494' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "unique_pool_0", "pool2": "unique_pool_0", "yes_i_really_really_mean_it": true}]': finished 2024-03-11T21:31:37.529 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:31:37 smithi092 ceph-mon[37563]: osdmap e150: 8 total, 8 up, 8 in 2024-03-11T21:31:37.529 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:31:37 smithi092 ceph-mon[37563]: from='mgr.14607 ' entity='mgr.y' 2024-03-11T21:31:37.530 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:31:37 smithi092 ceph-mon[37563]: from='mgr.14607 v1:172.21.15.92:0/1702717597' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T21:31:37.576 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:31:37 smithi097 ceph-mon[33317]: from='client.? v1:172.21.15.92:0/3838210494' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "unique_pool_0", "pool2": "unique_pool_0", "yes_i_really_really_mean_it": true}]': finished 2024-03-11T21:31:37.576 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:31:37 smithi097 ceph-mon[33317]: osdmap e150: 8 total, 8 up, 8 in 2024-03-11T21:31:37.576 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:31:37 smithi097 ceph-mon[33317]: from='mgr.14607 ' entity='mgr.y' 2024-03-11T21:31:37.576 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:31:37 smithi097 ceph-mon[33317]: from='mgr.14607 v1:172.21.15.92:0/1702717597' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T21:31:37.630 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:31:37.655 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:31:37.656 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:31:37.657 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:31:37.757 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:31:37.781 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:31:37.782 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:31:37.782 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:31:37.883 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:31:37.908 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:31:37.909 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:31:37.909 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:31:38.010 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:31:38.034 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:31:38.035 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:31:38.036 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:31:38.136 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:31:38.161 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:31:38.162 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:31:38.162 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:31:38.263 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:31:38.287 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:31:38.288 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:31:38.288 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:31:38.294 INFO:journalctl@ceph.mgr.y.smithi092.stdout:Mar 11 21:31:37 smithi092 ceph-00db0016-dfed-11ee-95c8-87774f69a715-mgr-y[29801]: ::ffff:172.21.15.97 - - [11/Mar/2024:21:31:37] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-11T21:31:38.388 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:31:38.412 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:31:38.413 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:31:38.413 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:31:38.514 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:31:38.542 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:31:38.543 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:31:38.543 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:31:38.610 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:31:38 smithi092 ceph-mon[29572]: pgmap v224: 132 pgs: 132 active+clean; 582 KiB data, 955 MiB used, 714 GiB / 715 GiB avail; 511 B/s rd, 0 op/s 2024-03-11T21:31:38.611 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:31:38 smithi092 ceph-mon[37563]: pgmap v224: 132 pgs: 132 active+clean; 582 KiB data, 955 MiB used, 714 GiB / 715 GiB avail; 511 B/s rd, 0 op/s 2024-03-11T21:31:38.644 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:31:38.671 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:31:38.671 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:31:38.672 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:31:38.773 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:31:38.799 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:31:38.800 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:31:38.800 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:31:38.826 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:31:38 smithi097 ceph-mon[33317]: pgmap v224: 132 pgs: 132 active+clean; 582 KiB data, 955 MiB used, 714 GiB / 715 GiB avail; 511 B/s rd, 0 op/s 2024-03-11T21:31:38.901 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:31:38.925 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:31:38.925 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:31:38.926 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:31:39.027 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:31:39.055 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:31:39.056 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:31:39.056 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:31:39.157 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:31:39.181 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:31:39.181 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:31:39.181 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:31:39.282 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:31:39.307 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:31:39.308 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:31:39.308 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:31:39.409 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:31:39.437 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:31:39.437 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:31:39.438 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:31:39.538 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:31:39.563 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:31:39.564 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:31:39.564 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:31:39.665 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:31:39.689 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:31:39.690 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:31:39.690 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:31:39.744 INFO:teuthology.orchestra.run.smithi092.stdout:{ 2024-03-11T21:31:39.744 INFO:teuthology.orchestra.run.smithi092.stdout: "ops": [], 2024-03-11T21:31:39.745 INFO:teuthology.orchestra.run.smithi092.stdout: "num_ops": 0 2024-03-11T21:31:39.745 INFO:teuthology.orchestra.run.smithi092.stdout:} 2024-03-11T21:31:39.794 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:31:39.818 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:31:39.818 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:31:39.819 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:31:39.919 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:31:39.947 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:31:39.948 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:31:39.948 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:31:40.049 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:31:40.073 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:31:40.074 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:31:40.074 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:31:40.175 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:31:40.203 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:31:40.204 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:31:40.204 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:31:40.305 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:31:40.330 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:31:40.331 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:31:40.331 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:31:40.331 INFO:teuthology.orchestra.run.smithi092.stdout:osd.0: {} 2024-03-11T21:31:40.350 INFO:teuthology.orchestra.run.smithi092.stdout:osd.1: {} 2024-03-11T21:31:40.369 INFO:teuthology.orchestra.run.smithi092.stdout:osd.2: {} 2024-03-11T21:31:40.389 INFO:teuthology.orchestra.run.smithi092.stdout:osd.3: {} 2024-03-11T21:31:40.411 INFO:teuthology.orchestra.run.smithi092.stdout:osd.4: {} 2024-03-11T21:31:40.430 INFO:teuthology.orchestra.run.smithi092.stdout:osd.5: {} 2024-03-11T21:31:40.432 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:31:40.450 INFO:teuthology.orchestra.run.smithi092.stdout:osd.6: {} 2024-03-11T21:31:40.456 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:31:40.456 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:31:40.457 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:31:40.459 INFO:teuthology.orchestra.run.smithi092.stderr:nodeep-scrub is set 2024-03-11T21:31:40.459 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:31:40 smithi092 ceph-mon[29572]: pgmap v225: 132 pgs: 132 active+clean; 582 KiB data, 973 MiB used, 714 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-03-11T21:31:40.459 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:31:40 smithi092 ceph-mon[29572]: from='client.? v1:172.21.15.92:0/2212877055' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-11T21:31:40.459 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:31:40 smithi092 ceph-mon[29572]: from='client.? v1:172.21.15.92:0/850270785' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-11T21:31:40.459 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:31:40 smithi092 ceph-mon[37563]: pgmap v225: 132 pgs: 132 active+clean; 582 KiB data, 973 MiB used, 714 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-03-11T21:31:40.459 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:31:40 smithi092 ceph-mon[37563]: from='client.? v1:172.21.15.92:0/2212877055' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-11T21:31:40.460 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:31:40 smithi092 ceph-mon[37563]: from='client.? v1:172.21.15.92:0/850270785' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-11T21:31:40.473 INFO:teuthology.orchestra.run.smithi092.stdout:osd.7: {} 2024-03-11T21:31:40.475 INFO:teuthology.orchestra.run.smithi092.stderr:osd.0: osd_enable_op_tracker = '' 2024-03-11T21:31:40.475 INFO:teuthology.orchestra.run.smithi092.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-11T21:31:40.475 INFO:teuthology.orchestra.run.smithi092.stderr:osd.2: osd_enable_op_tracker = '' 2024-03-11T21:31:40.475 INFO:teuthology.orchestra.run.smithi092.stderr:osd.3: osd_enable_op_tracker = '' 2024-03-11T21:31:40.475 INFO:teuthology.orchestra.run.smithi092.stderr:osd.4: osd_enable_op_tracker = '' 2024-03-11T21:31:40.476 INFO:teuthology.orchestra.run.smithi092.stderr:osd.5: osd_enable_op_tracker = '' 2024-03-11T21:31:40.476 INFO:teuthology.orchestra.run.smithi092.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-11T21:31:40.476 INFO:teuthology.orchestra.run.smithi092.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-11T21:31:40.557 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:31:40.585 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:31:40.586 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:31:40.586 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:31:40.688 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:31:40.712 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:31:40.713 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:31:40.713 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:31:40.814 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:31:40.819 DEBUG:teuthology.orchestra.run.smithi092:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid 00db0016-dfed-11ee-95c8-87774f69a715 -- ceph daemon osd.2 dump_blocked_ops 2024-03-11T21:31:40.826 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:31:40 smithi097 ceph-mon[33317]: pgmap v225: 132 pgs: 132 active+clean; 582 KiB data, 973 MiB used, 714 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-03-11T21:31:40.826 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:31:40 smithi097 ceph-mon[33317]: from='client.? v1:172.21.15.92:0/2212877055' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-11T21:31:40.826 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:31:40 smithi097 ceph-mon[33317]: from='client.? v1:172.21.15.92:0/850270785' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-11T21:31:40.838 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:31:40.838 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:31:40.839 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:31:40.940 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:31:40.964 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:31:40.965 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:31:40.965 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:31:41.066 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:31:41.091 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:31:41.091 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:31:41.091 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:31:41.192 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:31:41.220 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:31:41.221 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:31:41.221 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:31:41.322 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:31:41.347 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:31:41.348 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:31:41.348 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:31:41.449 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:31:41.473 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:31:41.474 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:31:41.474 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:31:41.574 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:31:41.603 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:31:41.603 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:31:41.603 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:31:41.704 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:31:41.731 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:31:41.731 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:31:41.732 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:31:41.827 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:31:41 smithi097 ceph-mon[33317]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-11T21:31:41.827 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:31:41 smithi097 ceph-mon[33317]: from='client.? v1:172.21.15.92:0/2212877055' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-03-11T21:31:41.827 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:31:41 smithi097 ceph-mon[33317]: osdmap e151: 8 total, 8 up, 8 in 2024-03-11T21:31:41.832 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:31:41.872 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:31:41 smithi092 ceph-mon[29572]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-11T21:31:41.873 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:31:41 smithi092 ceph-mon[29572]: from='client.? v1:172.21.15.92:0/2212877055' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-03-11T21:31:41.873 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:31:41 smithi092 ceph-mon[29572]: osdmap e151: 8 total, 8 up, 8 in 2024-03-11T21:31:41.873 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:31:41 smithi092 ceph-mon[37563]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-11T21:31:41.874 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:31:41 smithi092 ceph-mon[37563]: from='client.? v1:172.21.15.92:0/2212877055' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-03-11T21:31:41.874 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:31:41 smithi092 ceph-mon[37563]: osdmap e151: 8 total, 8 up, 8 in 2024-03-11T21:31:41.877 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:31:41.877 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:31:41.878 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:31:41.978 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:31:42.007 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:31:42.007 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:31:42.008 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:31:42.109 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:31:42.136 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:31:42.137 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:31:42.137 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:31:42.238 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:31:42.266 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:31:42.266 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:31:42.267 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:31:42.367 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:31:42.393 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:31:42.393 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:31:42.394 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:31:42.494 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:31:42.523 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:31:42.523 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:31:42.523 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:31:42.576 INFO:journalctl@ceph.iscsi.iscsi.a.smithi097.stdout:Mar 11 21:31:42 smithi097 ceph-00db0016-dfed-11ee-95c8-87774f69a715-iscsi-iscsi-a[60943]: debug there is no tcmu-runner data available 2024-03-11T21:31:42.576 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:31:42 smithi097 ceph-mon[33317]: pgmap v227: 132 pgs: 132 active+clean; 582 KiB data, 973 MiB used, 714 GiB / 715 GiB avail; 520 B/s rd, 0 op/s 2024-03-11T21:31:42.610 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:31:42 smithi092 ceph-mon[29572]: pgmap v227: 132 pgs: 132 active+clean; 582 KiB data, 973 MiB used, 714 GiB / 715 GiB avail; 520 B/s rd, 0 op/s 2024-03-11T21:31:42.610 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:31:42 smithi092 ceph-mon[37563]: pgmap v227: 132 pgs: 132 active+clean; 582 KiB data, 973 MiB used, 714 GiB / 715 GiB avail; 520 B/s rd, 0 op/s 2024-03-11T21:31:42.625 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:31:42.649 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:31:42.650 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:31:42.650 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:31:42.751 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:31:42.775 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:31:42.775 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:31:42.776 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:31:42.877 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:31:42.906 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:31:42.906 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:31:42.906 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:31:42.965 INFO:teuthology.task.full_sequential:In full_sequential, running task radosbench... 2024-03-11T21:31:42.966 INFO:tasks.radosbench:Beginning radosbench... 2024-03-11T21:31:42.966 INFO:tasks.cephadm.ceph_manager.ceph:creating pool_name unique_pool_1 2024-03-11T21:31:42.966 DEBUG:teuthology.orchestra.run.smithi092:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid 00db0016-dfed-11ee-95c8-87774f69a715 -- ceph osd pool create unique_pool_1 16 2024-03-11T21:31:43.007 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:31:43.036 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:31:43.037 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:31:43.037 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:31:43.061 INFO:teuthology.orchestra.run.smithi092.stderr:Inferring config /var/lib/ceph/00db0016-dfed-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T21:31:43.139 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:31:43.167 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:31:43.167 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:31:43.168 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:31:43.270 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:31:43.298 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:31:43.299 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:31:43.299 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:31:43.400 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:31:43.424 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:31:43.425 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:31:43.425 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:31:43.526 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:31:43.553 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:31:43.554 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:31:43.554 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:31:43.655 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:31:43.683 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:31:43.684 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:31:43.684 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:31:43.725 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:31:43 smithi092 ceph-mon[29572]: from='client.24469 v1:172.21.15.97:0/404208310' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:31:43.725 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:31:43 smithi092 ceph-mon[29572]: purged_snaps scrub starts 2024-03-11T21:31:43.725 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:31:43 smithi092 ceph-mon[29572]: purged_snaps scrub ok 2024-03-11T21:31:43.725 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:31:43 smithi092 ceph-mon[37563]: from='client.24469 v1:172.21.15.97:0/404208310' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:31:43.725 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:31:43 smithi092 ceph-mon[37563]: purged_snaps scrub starts 2024-03-11T21:31:43.725 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:31:43 smithi092 ceph-mon[37563]: purged_snaps scrub ok 2024-03-11T21:31:43.785 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:31:43.812 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:31:43.813 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:31:43.813 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:31:43.826 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:31:43 smithi097 ceph-mon[33317]: from='client.24469 v1:172.21.15.97:0/404208310' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:31:43.826 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:31:43 smithi097 ceph-mon[33317]: purged_snaps scrub starts 2024-03-11T21:31:43.826 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:31:43 smithi097 ceph-mon[33317]: purged_snaps scrub ok 2024-03-11T21:31:43.915 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:31:43.942 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:31:43.943 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:31:43.943 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:31:44.044 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:31:44.070 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:31:44.071 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:31:44.071 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:31:44.172 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:31:44.197 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:31:44.198 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:31:44.198 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:31:44.299 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:31:44.328 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:31:44.328 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:31:44.329 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:31:44.381 DEBUG:teuthology.orchestra.run.smithi092:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid 00db0016-dfed-11ee-95c8-87774f69a715 -- ceph osd unset noscrub 2024-03-11T21:31:44.429 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:31:44.454 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:31:44.455 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:31:44.455 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:31:44.556 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:31:44.559 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:31:44 smithi092 ceph-mon[29572]: pgmap v228: 132 pgs: 132 active+clean; 582 KiB data, 973 MiB used, 714 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-03-11T21:31:44.559 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:31:44 smithi092 ceph-mon[29572]: osdmap e152: 8 total, 8 up, 8 in 2024-03-11T21:31:44.559 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:31:44 smithi092 ceph-mon[37563]: pgmap v228: 132 pgs: 132 active+clean; 582 KiB data, 973 MiB used, 714 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-03-11T21:31:44.559 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:31:44 smithi092 ceph-mon[37563]: osdmap e152: 8 total, 8 up, 8 in 2024-03-11T21:31:44.581 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:31:44.582 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:31:44.582 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:31:44.611 INFO:teuthology.orchestra.run.smithi092.stderr:Inferring config /var/lib/ceph/00db0016-dfed-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T21:31:44.655 INFO:teuthology.orchestra.run.smithi092.stderr:Inferring config /var/lib/ceph/00db0016-dfed-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T21:31:44.683 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:31:44.707 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:31:44.707 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:31:44.708 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:31:44.808 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:31:44.826 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:31:44 smithi097 ceph-mon[33317]: pgmap v228: 132 pgs: 132 active+clean; 582 KiB data, 973 MiB used, 714 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-03-11T21:31:44.826 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:31:44 smithi097 ceph-mon[33317]: osdmap e152: 8 total, 8 up, 8 in 2024-03-11T21:31:44.832 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:31:44.833 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:31:44.833 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:31:44.934 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:31:44.958 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:31:44.958 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:31:44.958 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:31:45.059 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:31:45.083 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:31:45.084 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:31:45.084 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:31:45.185 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:31:45.213 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:31:45.214 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:31:45.215 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:31:45.315 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:31:45.343 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:31:45.344 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:31:45.344 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:31:45.445 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:31:45.472 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:31:45.473 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:31:45.473 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:31:45.529 DEBUG:teuthology.orchestra.run.smithi092:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid 00db0016-dfed-11ee-95c8-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-03-11T21:31:45.574 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:31:45.602 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:31:45.603 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:31:45.603 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:31:45.704 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:31:45.727 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:31:45.727 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:31:45.727 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:31:45.828 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:31:45.852 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:31:45.853 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:31:45.853 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:31:45.954 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:31:45.977 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:31:45.978 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:31:45.978 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:31:46.079 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:31:46.103 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:31:46.104 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:31:46.104 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:31:46.205 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:31:46.229 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:31:46.229 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:31:46.230 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:31:46.331 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:31:46.359 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:31:46.360 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:31:46.360 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:31:46.461 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:31:46.489 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:31:46.490 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:31:46.490 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:31:46.591 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:31:46.610 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:31:46 smithi092 ceph-mon[29572]: pgmap v230: 132 pgs: 132 active+clean; 582 KiB data, 973 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T21:31:46.610 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:31:46 smithi092 ceph-mon[37563]: pgmap v230: 132 pgs: 132 active+clean; 582 KiB data, 973 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T21:31:46.617 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:31:46.618 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:31:46.618 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:31:46.719 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:31:46.746 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:31:46.746 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:31:46.747 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:31:46.822 INFO:teuthology.orchestra.run.smithi092.stderr:Inferring config /var/lib/ceph/00db0016-dfed-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T21:31:46.826 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:31:46 smithi097 ceph-mon[33317]: pgmap v230: 132 pgs: 132 active+clean; 582 KiB data, 973 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T21:31:46.847 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:31:46.874 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:31:46.875 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:31:46.875 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:31:46.976 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:31:47.004 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:31:47.005 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:31:47.005 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:31:47.106 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:31:47.134 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:31:47.135 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:31:47.135 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:31:47.236 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:31:47.263 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:31:47.264 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:31:47.264 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:31:47.366 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:31:47.394 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:31:47.394 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:31:47.395 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:31:47.493 INFO:teuthology.orchestra.run.smithi092.stderr:reweighted osd.7 to 0.125371 (2018) 2024-03-11T21:31:47.495 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:31:47.504 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:31:47 smithi092 ceph-mon[29572]: from='client.? v1:172.21.15.92:0/2568812573' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 7, "weight": 0.12537148830939518}]: dispatch 2024-03-11T21:31:47.505 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:31:47 smithi092 ceph-mon[29572]: from='client.34976 ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 7, "weight": 0.12537148830939518}]: dispatch 2024-03-11T21:31:47.505 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:31:47 smithi092 ceph-mon[37563]: from='client.? v1:172.21.15.92:0/2568812573' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 7, "weight": 0.12537148830939518}]: dispatch 2024-03-11T21:31:47.505 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:31:47 smithi092 ceph-mon[37563]: from='client.34976 ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 7, "weight": 0.12537148830939518}]: dispatch 2024-03-11T21:31:47.521 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:31:47.521 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:31:47.522 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:31:47.623 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:31:47.650 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:31:47.651 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:31:47.651 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:31:47.734 INFO:teuthology.orchestra.run.smithi092.stderr:Inferring config /var/lib/ceph/00db0016-dfed-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T21:31:47.752 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:31:47.780 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:31:47.781 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:31:47.781 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:31:47.826 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:31:47 smithi097 ceph-mon[33317]: from='client.? v1:172.21.15.92:0/2568812573' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 7, "weight": 0.12537148830939518}]: dispatch 2024-03-11T21:31:47.826 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:31:47 smithi097 ceph-mon[33317]: from='client.34976 ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 7, "weight": 0.12537148830939518}]: dispatch 2024-03-11T21:31:47.882 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:31:47.906 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:31:47.907 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:31:47.907 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:31:48.008 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:31:48.036 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:31:48.036 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:31:48.037 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:31:48.110 INFO:journalctl@ceph.mgr.y.smithi092.stdout:Mar 11 21:31:47 smithi092 ceph-00db0016-dfed-11ee-95c8-87774f69a715-mgr-y[29801]: ::ffff:172.21.15.97 - - [11/Mar/2024:21:31:47] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-11T21:31:48.138 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:31:48.162 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:31:48.162 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:31:48.163 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:31:48.263 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:31:48.287 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:31:48.287 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:31:48.288 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:31:48.390 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:31:48.415 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:31:48.415 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:31:48.416 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:31:48.517 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:31:48.541 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:31:48.541 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:31:48.542 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:31:48.643 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:31:48.667 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:31:48.667 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:31:48.668 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:31:48.769 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:31:48.792 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:31:48.793 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:31:48.794 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:31:48.826 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:31:48 smithi097 ceph-mon[33317]: pgmap v231: 132 pgs: 132 active+clean; 582 KiB data, 973 MiB used, 714 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-03-11T21:31:48.826 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:31:48 smithi097 ceph-mon[33317]: from='client.34976 ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 7, "weight": 0.12537148830939518}]': finished 2024-03-11T21:31:48.826 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:31:48 smithi097 ceph-mon[33317]: osdmap e153: 8 total, 8 up, 8 in 2024-03-11T21:31:48.826 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:31:48 smithi097 ceph-mon[33317]: osdmap e154: 8 total, 8 up, 8 in 2024-03-11T21:31:48.860 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:31:48 smithi092 ceph-mon[29572]: pgmap v231: 132 pgs: 132 active+clean; 582 KiB data, 973 MiB used, 714 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-03-11T21:31:48.860 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:31:48 smithi092 ceph-mon[29572]: from='client.34976 ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 7, "weight": 0.12537148830939518}]': finished 2024-03-11T21:31:48.860 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:31:48 smithi092 ceph-mon[29572]: osdmap e153: 8 total, 8 up, 8 in 2024-03-11T21:31:48.860 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:31:48 smithi092 ceph-mon[29572]: osdmap e154: 8 total, 8 up, 8 in 2024-03-11T21:31:48.861 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:31:48 smithi092 ceph-mon[37563]: pgmap v231: 132 pgs: 132 active+clean; 582 KiB data, 973 MiB used, 714 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-03-11T21:31:48.861 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:31:48 smithi092 ceph-mon[37563]: from='client.34976 ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 7, "weight": 0.12537148830939518}]': finished 2024-03-11T21:31:48.861 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:31:48 smithi092 ceph-mon[37563]: osdmap e153: 8 total, 8 up, 8 in 2024-03-11T21:31:48.861 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:31:48 smithi092 ceph-mon[37563]: osdmap e154: 8 total, 8 up, 8 in 2024-03-11T21:31:48.894 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:31:48.918 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:31:48.919 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:31:48.919 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:31:49.020 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:31:49.044 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:31:49.045 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:31:49.045 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:31:49.145 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:31:49.169 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:31:49.170 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:31:49.170 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:31:49.271 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:31:49.295 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:31:49.296 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:31:49.296 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:31:49.397 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:31:49.422 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:31:49.422 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:31:49.423 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:31:49.523 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:31:49.547 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:31:49.547 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:31:49.548 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:31:49.649 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:31:49.672 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:31:49.673 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:31:49.673 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:31:49.774 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:31:49.797 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:31:49.798 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:31:49.798 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:31:49.899 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:31:49.924 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:31:49.925 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:31:49.926 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:31:50.027 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:31:50.050 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:31:50.051 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:31:50.051 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:31:50.152 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:31:50.175 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:31:50.176 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:31:50.176 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:31:50.226 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:31:50 smithi092 ceph-mon[29572]: pgmap v234: 132 pgs: 8 peering, 1 unknown, 123 active+clean; 582 KiB data, 972 MiB used, 714 GiB / 715 GiB avail 2024-03-11T21:31:50.226 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:31:50 smithi092 ceph-mon[29572]: osdmap e155: 8 total, 8 up, 8 in 2024-03-11T21:31:50.226 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:31:50 smithi092 ceph-mon[29572]: Health check failed: Reduced data availability: 2 pgs inactive, 2 pgs peering (PG_AVAILABILITY) 2024-03-11T21:31:50.227 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:31:50 smithi092 ceph-mon[37563]: pgmap v234: 132 pgs: 8 peering, 1 unknown, 123 active+clean; 582 KiB data, 972 MiB used, 714 GiB / 715 GiB avail 2024-03-11T21:31:50.227 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:31:50 smithi092 ceph-mon[37563]: osdmap e155: 8 total, 8 up, 8 in 2024-03-11T21:31:50.227 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:31:50 smithi092 ceph-mon[37563]: Health check failed: Reduced data availability: 2 pgs inactive, 2 pgs peering (PG_AVAILABILITY) 2024-03-11T21:31:50.277 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:31:50.300 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:31:50.301 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:31:50.301 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:31:50.402 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:31:50.430 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:31:50.431 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:31:50.431 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:31:50.531 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:31:50.556 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:31:50.556 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:31:50.557 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:31:50.576 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:31:50 smithi097 ceph-mon[33317]: pgmap v234: 132 pgs: 8 peering, 1 unknown, 123 active+clean; 582 KiB data, 972 MiB used, 714 GiB / 715 GiB avail 2024-03-11T21:31:50.576 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:31:50 smithi097 ceph-mon[33317]: osdmap e155: 8 total, 8 up, 8 in 2024-03-11T21:31:50.576 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:31:50 smithi097 ceph-mon[33317]: Health check failed: Reduced data availability: 2 pgs inactive, 2 pgs peering (PG_AVAILABILITY) 2024-03-11T21:31:50.658 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:31:50.686 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:31:50.686 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:31:50.687 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:31:50.787 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:31:50.814 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:31:50.815 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:31:50.815 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:31:50.916 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:31:50.943 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:31:50.944 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:31:50.944 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:31:50.970 INFO:teuthology.orchestra.run.smithi092.stdout:ERROR: (22) Invalid argument 2024-03-11T21:31:50.970 INFO:teuthology.orchestra.run.smithi092.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-11T21:31:51.046 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:31:51.070 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:31:51.070 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:31:51.071 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:31:51.171 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:31:51.197 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:31:51.197 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:31:51.198 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:31:51.217 INFO:teuthology.orchestra.run.smithi092.stderr:pool 'unique_pool_1' created 2024-03-11T21:31:51.298 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:31:51.326 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:31:51.326 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:31:51.327 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:31:51.428 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:31:51.452 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:31:51.452 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:31:51.453 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:31:51.554 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:31:51.576 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:31:51 smithi097 ceph-mon[33317]: osdmap e156: 8 total, 8 up, 8 in 2024-03-11T21:31:51.576 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:31:51 smithi097 ceph-mon[33317]: from='client.? v1:172.21.15.92:0/2964348057' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "unique_pool_1", "pg_num": 16}]: dispatch 2024-03-11T21:31:51.578 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:31:51.579 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:31:51.579 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:31:51.610 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:31:51 smithi092 ceph-mon[29572]: osdmap e156: 8 total, 8 up, 8 in 2024-03-11T21:31:51.610 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:31:51 smithi092 ceph-mon[29572]: from='client.? v1:172.21.15.92:0/2964348057' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "unique_pool_1", "pg_num": 16}]: dispatch 2024-03-11T21:31:51.611 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:31:51 smithi092 ceph-mon[37563]: osdmap e156: 8 total, 8 up, 8 in 2024-03-11T21:31:51.611 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:31:51 smithi092 ceph-mon[37563]: from='client.? v1:172.21.15.92:0/2964348057' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "unique_pool_1", "pg_num": 16}]: dispatch 2024-03-11T21:31:51.680 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:31:51.696 DEBUG:teuthology.orchestra.run.smithi092:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid 00db0016-dfed-11ee-95c8-87774f69a715 -- ceph daemon osd.2 dump_historic_ops 2024-03-11T21:31:51.706 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:31:51.706 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:31:51.707 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:31:51.807 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:31:51.832 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:31:51.833 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:31:51.833 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:31:51.934 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:31:51.964 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:31:51.964 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:31:51.964 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:31:52.065 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:31:52.098 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:31:52.098 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:31:52.099 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:31:52.199 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:31:52.231 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:31:52.232 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:31:52.232 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:31:52.333 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:31:52.357 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:31:52.358 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:31:52.358 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:31:52.360 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:31:52 smithi092 ceph-mon[29572]: pgmap v237: 132 pgs: 8 peering, 1 unknown, 123 active+clean; 582 KiB data, 972 MiB used, 714 GiB / 715 GiB avail 2024-03-11T21:31:52.360 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:31:52 smithi092 ceph-mon[29572]: from='client.? v1:172.21.15.92:0/2964348057' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "unique_pool_1", "pg_num": 16}]': finished 2024-03-11T21:31:52.360 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:31:52 smithi092 ceph-mon[29572]: osdmap e157: 8 total, 8 up, 8 in 2024-03-11T21:31:52.360 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:31:52 smithi092 ceph-mon[29572]: from='mgr.14607 ' entity='mgr.y' 2024-03-11T21:31:52.360 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:31:52 smithi092 ceph-mon[29572]: from='mgr.14607 v1:172.21.15.92:0/1702717597' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T21:31:52.360 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:31:52 smithi092 ceph-mon[37563]: pgmap v237: 132 pgs: 8 peering, 1 unknown, 123 active+clean; 582 KiB data, 972 MiB used, 714 GiB / 715 GiB avail 2024-03-11T21:31:52.360 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:31:52 smithi092 ceph-mon[37563]: from='client.? v1:172.21.15.92:0/2964348057' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "unique_pool_1", "pg_num": 16}]': finished 2024-03-11T21:31:52.360 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:31:52 smithi092 ceph-mon[37563]: osdmap e157: 8 total, 8 up, 8 in 2024-03-11T21:31:52.361 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:31:52 smithi092 ceph-mon[37563]: from='mgr.14607 ' entity='mgr.y' 2024-03-11T21:31:52.361 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:31:52 smithi092 ceph-mon[37563]: from='mgr.14607 v1:172.21.15.92:0/1702717597' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T21:31:52.459 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:31:52.485 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:31:52.486 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:31:52.486 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:31:52.576 INFO:journalctl@ceph.iscsi.iscsi.a.smithi097.stdout:Mar 11 21:31:52 smithi097 ceph-00db0016-dfed-11ee-95c8-87774f69a715-iscsi-iscsi-a[60943]: debug there is no tcmu-runner data available 2024-03-11T21:31:52.576 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:31:52 smithi097 ceph-mon[33317]: pgmap v237: 132 pgs: 8 peering, 1 unknown, 123 active+clean; 582 KiB data, 972 MiB used, 714 GiB / 715 GiB avail 2024-03-11T21:31:52.576 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:31:52 smithi097 ceph-mon[33317]: from='client.? v1:172.21.15.92:0/2964348057' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "unique_pool_1", "pg_num": 16}]': finished 2024-03-11T21:31:52.577 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:31:52 smithi097 ceph-mon[33317]: osdmap e157: 8 total, 8 up, 8 in 2024-03-11T21:31:52.577 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:31:52 smithi097 ceph-mon[33317]: from='mgr.14607 ' entity='mgr.y' 2024-03-11T21:31:52.577 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:31:52 smithi097 ceph-mon[33317]: from='mgr.14607 v1:172.21.15.92:0/1702717597' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T21:31:52.587 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:31:52.611 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:31:52.611 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:31:52.611 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:31:52.712 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:31:52.737 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:31:52.737 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:31:52.738 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:31:52.838 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:31:52.866 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:31:52.867 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:31:52.867 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:31:52.878 DEBUG:teuthology.orchestra.run.smithi092:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid 00db0016-dfed-11ee-95c8-87774f69a715 -- ceph osd pool application enable unique_pool_1 rados --yes-i-really-mean-it || true 2024-03-11T21:31:52.968 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:31:52.992 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:31:52.992 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:31:52.993 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:31:53.094 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:31:53.118 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:31:53.119 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:31:53.119 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:31:53.219 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:31:53.248 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:31:53.249 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:31:53.249 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:31:53.350 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:31:53.374 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:31:53.374 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:31:53.375 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:31:53.476 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:31:53.489 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:31:53 smithi092 ceph-mon[29572]: osdmap e158: 8 total, 8 up, 8 in 2024-03-11T21:31:53.490 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:31:53 smithi092 ceph-mon[29572]: from='client.24469 v1:172.21.15.97:0/404208310' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:31:53.490 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:31:53 smithi092 ceph-mon[37563]: osdmap e158: 8 total, 8 up, 8 in 2024-03-11T21:31:53.490 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:31:53 smithi092 ceph-mon[37563]: from='client.24469 v1:172.21.15.97:0/404208310' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:31:53.500 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:31:53.501 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:31:53.501 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:31:53.521 INFO:teuthology.orchestra.run.smithi092.stdout:osd.0: {} 2024-03-11T21:31:53.541 INFO:teuthology.orchestra.run.smithi092.stdout:osd.1: {} 2024-03-11T21:31:53.561 INFO:teuthology.orchestra.run.smithi092.stdout:osd.2: {} 2024-03-11T21:31:53.576 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:31:53 smithi097 ceph-mon[33317]: osdmap e158: 8 total, 8 up, 8 in 2024-03-11T21:31:53.576 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:31:53 smithi097 ceph-mon[33317]: from='client.24469 v1:172.21.15.97:0/404208310' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:31:53.580 INFO:teuthology.orchestra.run.smithi092.stdout:osd.3: {} 2024-03-11T21:31:53.601 INFO:teuthology.orchestra.run.smithi092.stdout:osd.4: {} 2024-03-11T21:31:53.602 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:31:53.621 INFO:teuthology.orchestra.run.smithi092.stdout:osd.5: {} 2024-03-11T21:31:53.630 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:31:53.630 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:31:53.631 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:31:53.641 INFO:teuthology.orchestra.run.smithi092.stdout:osd.6: {} 2024-03-11T21:31:53.662 INFO:teuthology.orchestra.run.smithi092.stdout:osd.7: {} 2024-03-11T21:31:53.663 INFO:teuthology.orchestra.run.smithi092.stderr:osd.0: osd_enable_op_tracker = '' 2024-03-11T21:31:53.663 INFO:teuthology.orchestra.run.smithi092.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-11T21:31:53.663 INFO:teuthology.orchestra.run.smithi092.stderr:osd.2: osd_enable_op_tracker = '' 2024-03-11T21:31:53.663 INFO:teuthology.orchestra.run.smithi092.stderr:osd.3: osd_enable_op_tracker = '' 2024-03-11T21:31:53.663 INFO:teuthology.orchestra.run.smithi092.stderr:osd.4: osd_enable_op_tracker = '' 2024-03-11T21:31:53.663 INFO:teuthology.orchestra.run.smithi092.stderr:osd.5: osd_enable_op_tracker = '' 2024-03-11T21:31:53.663 INFO:teuthology.orchestra.run.smithi092.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-11T21:31:53.663 INFO:teuthology.orchestra.run.smithi092.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-11T21:31:53.732 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:31:53.756 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:31:53.756 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:31:53.757 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:31:53.857 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:31:53.881 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:31:53.882 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:31:53.882 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:31:53.983 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:31:54.007 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:31:54.008 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:31:54.008 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:31:54.109 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:31:54.137 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:31:54.138 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:31:54.138 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:31:54.235 INFO:teuthology.orchestra.run.smithi092.stderr:noscrub is unset 2024-03-11T21:31:54.239 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:31:54.267 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:31:54.268 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:31:54.268 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:31:54.369 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:31:54.398 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:31:54.398 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:31:54.399 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:31:54.499 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:31:54.527 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:31:54.527 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:31:54.527 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:31:54.576 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:31:54 smithi097 ceph-mon[33317]: pgmap v240: 148 pgs: 10 creating+peering, 138 active+clean; 582 KiB data, 981 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s; 145 KiB/s, 16 objects/s recovering 2024-03-11T21:31:54.576 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:31:54 smithi097 ceph-mon[33317]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-11T21:31:54.576 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:31:54 smithi097 ceph-mon[33317]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 2 pgs inactive, 2 pgs peering) 2024-03-11T21:31:54.576 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:31:54 smithi097 ceph-mon[33317]: osdmap e159: 8 total, 8 up, 8 in 2024-03-11T21:31:54.576 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:31:54 smithi097 ceph-mon[33317]: from='client.? v1:172.21.15.92:0/2395229587' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-11T21:31:54.577 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:31:54 smithi097 ceph-mon[33317]: from='client.? v1:172.21.15.92:0/861569196' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-11T21:31:54.577 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:31:54 smithi092 ceph-mon[29572]: pgmap v240: 148 pgs: 10 creating+peering, 138 active+clean; 582 KiB data, 981 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s; 145 KiB/s, 16 objects/s recovering 2024-03-11T21:31:54.578 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:31:54 smithi092 ceph-mon[29572]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-11T21:31:54.578 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:31:54 smithi092 ceph-mon[29572]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 2 pgs inactive, 2 pgs peering) 2024-03-11T21:31:54.578 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:31:54 smithi092 ceph-mon[29572]: osdmap e159: 8 total, 8 up, 8 in 2024-03-11T21:31:54.578 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:31:54 smithi092 ceph-mon[29572]: from='client.? v1:172.21.15.92:0/2395229587' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-11T21:31:54.578 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:31:54 smithi092 ceph-mon[29572]: from='client.? v1:172.21.15.92:0/861569196' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-11T21:31:54.578 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:31:54 smithi092 ceph-mon[37563]: pgmap v240: 148 pgs: 10 creating+peering, 138 active+clean; 582 KiB data, 981 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s; 145 KiB/s, 16 objects/s recovering 2024-03-11T21:31:54.578 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:31:54 smithi092 ceph-mon[37563]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-11T21:31:54.578 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:31:54 smithi092 ceph-mon[37563]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 2 pgs inactive, 2 pgs peering) 2024-03-11T21:31:54.578 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:31:54 smithi092 ceph-mon[37563]: osdmap e159: 8 total, 8 up, 8 in 2024-03-11T21:31:54.578 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:31:54 smithi092 ceph-mon[37563]: from='client.? v1:172.21.15.92:0/2395229587' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-11T21:31:54.579 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:31:54 smithi092 ceph-mon[37563]: from='client.? v1:172.21.15.92:0/861569196' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-11T21:31:54.628 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:31:54.654 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:31:54.654 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:31:54.654 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:31:54.755 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:31:54.781 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:31:54.782 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:31:54.783 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:31:54.854 INFO:teuthology.orchestra.run.smithi092.stderr:Inferring config /var/lib/ceph/00db0016-dfed-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T21:31:54.855 INFO:teuthology.orchestra.run.smithi092.stderr:Inferring config /var/lib/ceph/00db0016-dfed-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T21:31:54.883 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:31:54.909 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:31:54.909 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:31:54.910 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:31:54.983 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5, 6, 7] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7] 2024-03-11T21:31:54.984 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-03-11T21:31:54.984 INFO:tasks.thrashosds.thrasher:inject_pause on osd.5 2024-03-11T21:31:54.984 INFO:tasks.thrashosds.thrasher:Testing filestore_inject_stall pause injection for duration 3 2024-03-11T21:31:54.984 INFO:tasks.thrashosds.thrasher:Checking after 0, should_be_down=False 2024-03-11T21:31:54.984 DEBUG:teuthology.orchestra.run.smithi097:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid 00db0016-dfed-11ee-95c8-87774f69a715 -- ceph daemon osd.5 config set filestore_inject_stall 3 2024-03-11T21:31:55.011 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:31:55.052 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:31:55.052 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:31:55.053 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:31:55.154 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:31:55.178 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:31:55.179 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:31:55.179 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:31:55.235 INFO:teuthology.orchestra.run.smithi097.stderr:Inferring config /var/lib/ceph/00db0016-dfed-11ee-95c8-87774f69a715/mon.b/config 2024-03-11T21:31:55.280 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:31:55.304 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:31:55.304 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:31:55.305 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:31:55.326 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:31:55 smithi097 ceph-mon[33317]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-11T21:31:55.326 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:31:55 smithi097 ceph-mon[33317]: from='client.? v1:172.21.15.92:0/861569196' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-03-11T21:31:55.326 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:31:55 smithi097 ceph-mon[33317]: osdmap e160: 8 total, 8 up, 8 in 2024-03-11T21:31:55.326 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:31:55 smithi097 ceph-mon[33317]: 2.1f scrub starts 2024-03-11T21:31:55.326 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:31:55 smithi097 ceph-mon[33317]: 2.1f scrub ok 2024-03-11T21:31:55.406 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:31:55.430 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:31:55.430 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:31:55.431 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:31:55.532 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:31:55.548 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:31:55 smithi092 ceph-mon[29572]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-11T21:31:55.548 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:31:55 smithi092 ceph-mon[29572]: from='client.? v1:172.21.15.92:0/861569196' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-03-11T21:31:55.548 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:31:55 smithi092 ceph-mon[29572]: osdmap e160: 8 total, 8 up, 8 in 2024-03-11T21:31:55.548 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:31:55 smithi092 ceph-mon[29572]: 2.1f scrub starts 2024-03-11T21:31:55.548 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:31:55 smithi092 ceph-mon[29572]: 2.1f scrub ok 2024-03-11T21:31:55.548 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:31:55 smithi092 ceph-mon[37563]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-11T21:31:55.548 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:31:55 smithi092 ceph-mon[37563]: from='client.? v1:172.21.15.92:0/861569196' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-03-11T21:31:55.549 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:31:55 smithi092 ceph-mon[37563]: osdmap e160: 8 total, 8 up, 8 in 2024-03-11T21:31:55.549 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:31:55 smithi092 ceph-mon[37563]: 2.1f scrub starts 2024-03-11T21:31:55.549 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:31:55 smithi092 ceph-mon[37563]: 2.1f scrub ok 2024-03-11T21:31:55.556 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:31:55.557 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:31:55.558 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:31:55.659 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:31:55.684 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:31:55.685 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:31:55.685 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:31:55.786 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:31:55.810 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:31:55.810 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:31:55.811 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:31:55.912 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:31:55.936 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:31:55.937 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:31:55.937 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:31:56.038 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:31:56.061 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:31:56.062 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:31:56.063 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:31:56.167 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:31:56.191 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:31:56.191 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:31:56.191 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:31:56.292 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:31:56.317 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:31:56.317 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:31:56.317 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:31:56.418 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:31:56.442 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:31:56.443 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:31:56.443 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:31:56.486 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:31:56 smithi092 ceph-mon[29572]: 6.3 scrub starts 2024-03-11T21:31:56.486 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:31:56 smithi092 ceph-mon[29572]: 6.3 scrub ok 2024-03-11T21:31:56.486 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:31:56 smithi092 ceph-mon[29572]: pgmap v243: 148 pgs: 10 creating+peering, 138 active+clean; 582 KiB data, 981 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s; 145 KiB/s, 16 objects/s recovering 2024-03-11T21:31:56.486 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:31:56 smithi092 ceph-mon[29572]: 6.12 scrub starts 2024-03-11T21:31:56.486 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:31:56 smithi092 ceph-mon[29572]: 6.12 scrub ok 2024-03-11T21:31:56.487 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:31:56 smithi092 ceph-mon[29572]: 2.d scrub starts 2024-03-11T21:31:56.487 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:31:56 smithi092 ceph-mon[29572]: 2.d scrub ok 2024-03-11T21:31:56.487 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:31:56 smithi092 ceph-mon[37563]: 6.3 scrub starts 2024-03-11T21:31:56.487 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:31:56 smithi092 ceph-mon[37563]: 6.3 scrub ok 2024-03-11T21:31:56.487 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:31:56 smithi092 ceph-mon[37563]: pgmap v243: 148 pgs: 10 creating+peering, 138 active+clean; 582 KiB data, 981 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s; 145 KiB/s, 16 objects/s recovering 2024-03-11T21:31:56.487 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:31:56 smithi092 ceph-mon[37563]: 6.12 scrub starts 2024-03-11T21:31:56.487 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:31:56 smithi092 ceph-mon[37563]: 6.12 scrub ok 2024-03-11T21:31:56.487 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:31:56 smithi092 ceph-mon[37563]: 2.d scrub starts 2024-03-11T21:31:56.487 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:31:56 smithi092 ceph-mon[37563]: 2.d scrub ok 2024-03-11T21:31:56.490 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:31:56 smithi097 ceph-mon[33317]: 6.3 scrub starts 2024-03-11T21:31:56.490 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:31:56 smithi097 ceph-mon[33317]: 6.3 scrub ok 2024-03-11T21:31:56.490 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:31:56 smithi097 ceph-mon[33317]: pgmap v243: 148 pgs: 10 creating+peering, 138 active+clean; 582 KiB data, 981 MiB used, 714 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s; 145 KiB/s, 16 objects/s recovering 2024-03-11T21:31:56.490 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:31:56 smithi097 ceph-mon[33317]: 6.12 scrub starts 2024-03-11T21:31:56.491 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:31:56 smithi097 ceph-mon[33317]: 6.12 scrub ok 2024-03-11T21:31:56.491 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:31:56 smithi097 ceph-mon[33317]: 2.d scrub starts 2024-03-11T21:31:56.491 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:31:56 smithi097 ceph-mon[33317]: 2.d scrub ok 2024-03-11T21:31:56.544 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:31:56.570 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:31:56.574 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:31:56.574 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:31:56.610 DEBUG:teuthology.orchestra.run.smithi092:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid 00db0016-dfed-11ee-95c8-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-03-11T21:31:56.675 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:31:56.699 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:31:56.700 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:31:56.700 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:31:56.801 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:31:56.825 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:31:56.826 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:31:56.826 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:31:56.927 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:31:56.951 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:31:56.951 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:31:56.952 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:31:57.053 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:31:57.054 INFO:teuthology.orchestra.run.smithi097.stdout:{ 2024-03-11T21:31:57.054 INFO:teuthology.orchestra.run.smithi097.stdout: "success": "filestore_inject_stall = '' (not observed, change may require restart) " 2024-03-11T21:31:57.054 INFO:teuthology.orchestra.run.smithi097.stdout:} 2024-03-11T21:31:57.077 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:31:57.078 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:31:57.079 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:31:57.179 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:31:57.210 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:31:57.210 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:31:57.210 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:31:57.235 DEBUG:teuthology.orchestra.run.smithi092:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid 00db0016-dfed-11ee-95c8-87774f69a715 -- ceph osd unset nodeep-scrub 2024-03-11T21:31:57.312 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:31:57.326 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:31:57 smithi097 ceph-mon[33317]: 2.5 scrub starts 2024-03-11T21:31:57.326 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:31:57 smithi097 ceph-mon[33317]: 2.5 scrub ok 2024-03-11T21:31:57.336 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:31:57.337 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:31:57.337 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:31:57.438 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:31:57.463 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:31:57.463 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:31:57.463 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:31:57.547 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:31:57 smithi092 ceph-mon[29572]: 2.5 scrub starts 2024-03-11T21:31:57.547 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:31:57 smithi092 ceph-mon[29572]: 2.5 scrub ok 2024-03-11T21:31:57.547 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:31:57 smithi092 ceph-mon[37563]: 2.5 scrub starts 2024-03-11T21:31:57.547 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:31:57 smithi092 ceph-mon[37563]: 2.5 scrub ok 2024-03-11T21:31:57.564 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:31:57.588 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:31:57.588 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:31:57.589 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:31:57.690 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:31:57.714 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:31:57.715 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:31:57.715 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:31:57.816 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:31:57.840 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:31:57.841 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:31:57.841 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:31:57.942 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:31:57.965 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:31:57.966 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:31:57.966 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:31:58.067 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:31:58.090 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:31:58.091 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:31:58.091 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:31:58.159 INFO:teuthology.orchestra.run.smithi092.stderr:Inferring config /var/lib/ceph/00db0016-dfed-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T21:31:58.174 INFO:journalctl@ceph.mgr.y.smithi092.stdout:Mar 11 21:31:57 smithi092 ceph-00db0016-dfed-11ee-95c8-87774f69a715-mgr-y[29801]: ::ffff:172.21.15.97 - - [11/Mar/2024:21:31:57] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-11T21:31:58.174 INFO:teuthology.orchestra.run.smithi092.stdout:{ 2024-03-11T21:31:58.174 INFO:teuthology.orchestra.run.smithi092.stdout: "size": 20, 2024-03-11T21:31:58.174 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 600, 2024-03-11T21:31:58.174 INFO:teuthology.orchestra.run.smithi092.stdout: "ops": [ 2024-03-11T21:31:58.174 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:31:58.174 INFO:teuthology.orchestra.run.smithi092.stdout: "description": "osd_op(client.34715.0:32047 7.8 7:1cc332d5:::benchmark_data_smithi092_76040_object32046:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e105)", 2024-03-11T21:31:58.174 INFO:teuthology.orchestra.run.smithi092.stdout: "initiated_at": "2024-03-11T21:27:57.779953+0000", 2024-03-11T21:31:58.175 INFO:teuthology.orchestra.run.smithi092.stdout: "age": 240.41059519199999, 2024-03-11T21:31:58.175 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.084056040999999998, 2024-03-11T21:31:58.175 INFO:teuthology.orchestra.run.smithi092.stdout: "type_data": { 2024-03-11T21:31:58.175 INFO:teuthology.orchestra.run.smithi092.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-11T21:31:58.175 INFO:teuthology.orchestra.run.smithi092.stdout: "client_info": { 2024-03-11T21:31:58.175 INFO:teuthology.orchestra.run.smithi092.stdout: "client": "client.34715", 2024-03-11T21:31:58.175 INFO:teuthology.orchestra.run.smithi092.stdout: "client_addr": "v1:172.21.15.92:0/2403132573", 2024-03-11T21:31:58.175 INFO:teuthology.orchestra.run.smithi092.stdout: "tid": 32047 2024-03-11T21:31:58.175 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:31:58.175 INFO:teuthology.orchestra.run.smithi092.stdout: "events": [ 2024-03-11T21:31:58.175 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:31:58.175 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "initiated", 2024-03-11T21:31:58.175 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:27:57.779953+0000", 2024-03-11T21:31:58.175 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-03-11T21:31:58.176 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:31:58.176 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:31:58.176 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "header_read", 2024-03-11T21:31:58.176 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:27:57.779953+0000", 2024-03-11T21:31:58.176 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-03-11T21:31:58.176 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:31:58.176 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:31:58.176 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "throttled", 2024-03-11T21:31:58.176 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:27:57.779956+0000", 2024-03-11T21:31:58.176 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 2.069e-06 2024-03-11T21:31:58.176 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:31:58.176 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:31:58.176 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "all_read", 2024-03-11T21:31:58.176 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:27:57.779982+0000", 2024-03-11T21:31:58.176 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 2.6601000000000002e-05 2024-03-11T21:31:58.177 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:31:58.177 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:31:58.177 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "dispatched", 2024-03-11T21:31:58.177 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:27:57.780005+0000", 2024-03-11T21:31:58.177 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 2.2756e-05 2024-03-11T21:31:58.177 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:31:58.177 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:31:58.177 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "queued_for_pg", 2024-03-11T21:31:58.177 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:27:57.780014+0000", 2024-03-11T21:31:58.177 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 8.7630000000000006e-06 2024-03-11T21:31:58.177 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:31:58.177 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:31:58.177 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-03-11T21:31:58.177 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:27:57.780080+0000", 2024-03-11T21:31:58.178 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 6.6018000000000004e-05 2024-03-11T21:31:58.178 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:31:58.178 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:31:58.178 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "started", 2024-03-11T21:31:58.178 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:27:57.781113+0000", 2024-03-11T21:31:58.178 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.001033312 2024-03-11T21:31:58.178 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:31:58.178 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:31:58.178 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "waiting for subops from 4,7", 2024-03-11T21:31:58.178 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:27:57.781655+0000", 2024-03-11T21:31:58.178 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.000542341 2024-03-11T21:31:58.178 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:31:58.178 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:31:58.179 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "sub_op_commit_rec", 2024-03-11T21:31:58.179 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:27:57.783494+0000", 2024-03-11T21:31:58.179 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.0018391499999999999 2024-03-11T21:31:58.179 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:31:58.179 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:31:58.179 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "op_commit", 2024-03-11T21:31:58.179 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:27:57.783972+0000", 2024-03-11T21:31:58.179 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00047797699999999998 2024-03-11T21:31:58.179 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:31:58.179 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:31:58.179 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "sub_op_commit_rec", 2024-03-11T21:31:58.179 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:27:57.863900+0000", 2024-03-11T21:31:58.180 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.079927953999999996 2024-03-11T21:31:58.180 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:31:58.180 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:31:58.180 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "commit_sent", 2024-03-11T21:31:58.180 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:27:57.863960+0000", 2024-03-11T21:31:58.180 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 5.9917999999999998e-05 2024-03-11T21:31:58.180 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:31:58.180 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:31:58.180 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "done", 2024-03-11T21:31:58.180 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:27:57.864009+0000", 2024-03-11T21:31:58.180 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 4.9181999999999998e-05 2024-03-11T21:31:58.180 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-03-11T21:31:58.180 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-03-11T21:31:58.180 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-03-11T21:31:58.181 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:31:58.181 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:31:58.181 INFO:teuthology.orchestra.run.smithi092.stdout: "description": "osd_op(client.34715.0:104678 7.e 7:77299287:::benchmark_data_smithi092_76040_object104677:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e117)", 2024-03-11T21:31:58.181 INFO:teuthology.orchestra.run.smithi092.stdout: "initiated_at": "2024-03-11T21:28:57.288999+0000", 2024-03-11T21:31:58.181 INFO:teuthology.orchestra.run.smithi092.stdout: "age": 180.90154941500001, 2024-03-11T21:31:58.181 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.092710055999999999, 2024-03-11T21:31:58.181 INFO:teuthology.orchestra.run.smithi092.stdout: "type_data": { 2024-03-11T21:31:58.181 INFO:teuthology.orchestra.run.smithi092.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-11T21:31:58.181 INFO:teuthology.orchestra.run.smithi092.stdout: "client_info": { 2024-03-11T21:31:58.181 INFO:teuthology.orchestra.run.smithi092.stdout: "client": "client.34715", 2024-03-11T21:31:58.181 INFO:teuthology.orchestra.run.smithi092.stdout: "client_addr": "v1:172.21.15.92:0/2403132573", 2024-03-11T21:31:58.181 INFO:teuthology.orchestra.run.smithi092.stdout: "tid": 104678 2024-03-11T21:31:58.181 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:31:58.181 INFO:teuthology.orchestra.run.smithi092.stdout: "events": [ 2024-03-11T21:31:58.181 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:31:58.182 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "initiated", 2024-03-11T21:31:58.182 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:28:57.288999+0000", 2024-03-11T21:31:58.182 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-03-11T21:31:58.182 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:31:58.182 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:31:58.182 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "header_read", 2024-03-11T21:31:58.182 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:28:57.288999+0000", 2024-03-11T21:31:58.182 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-03-11T21:31:58.182 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:31:58.182 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:31:58.182 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "throttled", 2024-03-11T21:31:58.182 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:28:57.289041+0000", 2024-03-11T21:31:58.182 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 4.1856e-05 2024-03-11T21:31:58.183 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:31:58.183 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:31:58.183 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "all_read", 2024-03-11T21:31:58.183 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:28:57.289064+0000", 2024-03-11T21:31:58.183 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 2.3112000000000002e-05 2024-03-11T21:31:58.183 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:31:58.183 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:31:58.183 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "dispatched", 2024-03-11T21:31:58.183 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:28:57.289075+0000", 2024-03-11T21:31:58.183 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.0312000000000001e-05 2024-03-11T21:31:58.183 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:31:58.183 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:31:58.183 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "queued_for_pg", 2024-03-11T21:31:58.184 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:28:57.289082+0000", 2024-03-11T21:31:58.184 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 7.3599999999999998e-06 2024-03-11T21:31:58.184 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:31:58.184 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:31:58.184 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-03-11T21:31:58.184 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:28:57.289159+0000", 2024-03-11T21:31:58.184 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 7.7372000000000004e-05 2024-03-11T21:31:58.184 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:31:58.184 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:31:58.184 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "started", 2024-03-11T21:31:58.184 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:28:57.289288+0000", 2024-03-11T21:31:58.184 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00012846999999999999 2024-03-11T21:31:58.184 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:31:58.184 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:31:58.184 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "waiting for subops from 0,5", 2024-03-11T21:31:58.185 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:28:57.289421+0000", 2024-03-11T21:31:58.185 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00013358000000000001 2024-03-11T21:31:58.185 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:31:58.185 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:31:58.185 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "sub_op_commit_rec", 2024-03-11T21:31:58.185 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:28:57.378143+0000", 2024-03-11T21:31:58.185 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.088721513000000002 2024-03-11T21:31:58.185 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:31:58.185 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:31:58.185 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "sub_op_commit_rec", 2024-03-11T21:31:58.185 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:28:57.378266+0000", 2024-03-11T21:31:58.185 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00012305700000000001 2024-03-11T21:31:58.185 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:31:58.185 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:31:58.185 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "op_commit", 2024-03-11T21:31:58.186 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:28:57.381595+0000", 2024-03-11T21:31:58.186 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.0033288409999999999 2024-03-11T21:31:58.186 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:31:58.186 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:31:58.186 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "commit_sent", 2024-03-11T21:31:58.186 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:28:57.381664+0000", 2024-03-11T21:31:58.186 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 6.9708000000000004e-05 2024-03-11T21:31:58.186 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:31:58.186 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:31:58.186 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "done", 2024-03-11T21:31:58.186 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:28:57.381709+0000", 2024-03-11T21:31:58.186 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 4.4875e-05 2024-03-11T21:31:58.186 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-03-11T21:31:58.186 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-03-11T21:31:58.187 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-03-11T21:31:58.187 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:31:58.187 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:31:58.187 INFO:teuthology.orchestra.run.smithi092.stdout: "description": "osd_op(client.34715.0:104682 7.e 7:7b3e8d68:::benchmark_data_smithi092_76040_object104681:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e117)", 2024-03-11T21:31:58.187 INFO:teuthology.orchestra.run.smithi092.stdout: "initiated_at": "2024-03-11T21:28:57.292170+0000", 2024-03-11T21:31:58.187 INFO:teuthology.orchestra.run.smithi092.stdout: "age": 180.89837847499999, 2024-03-11T21:31:58.187 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.092283122999999995, 2024-03-11T21:31:58.187 INFO:teuthology.orchestra.run.smithi092.stdout: "type_data": { 2024-03-11T21:31:58.187 INFO:teuthology.orchestra.run.smithi092.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-11T21:31:58.187 INFO:teuthology.orchestra.run.smithi092.stdout: "client_info": { 2024-03-11T21:31:58.187 INFO:teuthology.orchestra.run.smithi092.stdout: "client": "client.34715", 2024-03-11T21:31:58.188 INFO:teuthology.orchestra.run.smithi092.stdout: "client_addr": "v1:172.21.15.92:0/2403132573", 2024-03-11T21:31:58.188 INFO:teuthology.orchestra.run.smithi092.stdout: "tid": 104682 2024-03-11T21:31:58.188 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:31:58.188 INFO:teuthology.orchestra.run.smithi092.stdout: "events": [ 2024-03-11T21:31:58.188 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:31:58.188 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "initiated", 2024-03-11T21:31:58.188 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:28:57.292170+0000", 2024-03-11T21:31:58.188 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-03-11T21:31:58.188 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:31:58.189 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:31:58.189 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "header_read", 2024-03-11T21:31:58.189 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:28:57.292170+0000", 2024-03-11T21:31:58.189 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-03-11T21:31:58.189 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:31:58.189 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:31:58.189 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "throttled", 2024-03-11T21:31:58.189 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:28:57.292173+0000", 2024-03-11T21:31:58.189 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 2.542e-06 2024-03-11T21:31:58.189 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:31:58.189 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:31:58.189 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "all_read", 2024-03-11T21:31:58.189 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:28:57.292200+0000", 2024-03-11T21:31:58.189 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 2.7288e-05 2024-03-11T21:31:58.190 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:31:58.190 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:31:58.190 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "dispatched", 2024-03-11T21:31:58.190 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:28:57.375541+0000", 2024-03-11T21:31:58.190 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.083340996000000001 2024-03-11T21:31:58.190 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:31:58.190 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:31:58.190 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "queued_for_pg", 2024-03-11T21:31:58.190 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:28:57.375557+0000", 2024-03-11T21:31:58.190 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.5933000000000001e-05 2024-03-11T21:31:58.190 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:31:58.190 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:31:58.190 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-03-11T21:31:58.190 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:28:57.378303+0000", 2024-03-11T21:31:58.190 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.002745824 2024-03-11T21:31:58.191 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:31:58.191 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:31:58.191 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "started", 2024-03-11T21:31:58.191 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:28:57.378444+0000", 2024-03-11T21:31:58.191 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00014127399999999999 2024-03-11T21:31:58.191 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:31:58.191 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:31:58.191 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "waiting for subops from 0,5", 2024-03-11T21:31:58.191 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:28:57.378537+0000", 2024-03-11T21:31:58.191 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 9.3263000000000005e-05 2024-03-11T21:31:58.191 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:31:58.191 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:31:58.191 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "op_commit", 2024-03-11T21:31:58.191 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:28:57.381723+0000", 2024-03-11T21:31:58.191 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.0031855989999999999 2024-03-11T21:31:58.192 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:31:58.192 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:31:58.192 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "sub_op_commit_rec", 2024-03-11T21:31:58.192 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:28:57.381875+0000", 2024-03-11T21:31:58.192 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00015235199999999999 2024-03-11T21:31:58.192 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:31:58.192 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:31:58.192 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "sub_op_commit_rec", 2024-03-11T21:31:58.192 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:28:57.384116+0000", 2024-03-11T21:31:58.192 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.002240279 2024-03-11T21:31:58.192 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:31:58.192 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:31:58.192 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "commit_sent", 2024-03-11T21:31:58.192 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:28:57.384313+0000", 2024-03-11T21:31:58.192 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.000197648 2024-03-11T21:31:58.193 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:31:58.193 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:31:58.193 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "done", 2024-03-11T21:31:58.193 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:28:57.384453+0000", 2024-03-11T21:31:58.193 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.000140125 2024-03-11T21:31:58.193 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-03-11T21:31:58.193 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-03-11T21:31:58.193 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-03-11T21:31:58.193 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:31:58.193 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:31:58.193 INFO:teuthology.orchestra.run.smithi092.stdout: "description": "osd_op(client.34715.0:115732 7.e 7:7582fdce:::benchmark_data_smithi092_76040_object115731:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e119)", 2024-03-11T21:31:58.193 INFO:teuthology.orchestra.run.smithi092.stdout: "initiated_at": "2024-03-11T21:29:08.166630+0000", 2024-03-11T21:31:58.193 INFO:teuthology.orchestra.run.smithi092.stdout: "age": 170.02391845099999, 2024-03-11T21:31:58.193 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.099177280000000007, 2024-03-11T21:31:58.194 INFO:teuthology.orchestra.run.smithi092.stdout: "type_data": { 2024-03-11T21:31:58.194 INFO:teuthology.orchestra.run.smithi092.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-11T21:31:58.194 INFO:teuthology.orchestra.run.smithi092.stdout: "client_info": { 2024-03-11T21:31:58.194 INFO:teuthology.orchestra.run.smithi092.stdout: "client": "client.34715", 2024-03-11T21:31:58.194 INFO:teuthology.orchestra.run.smithi092.stdout: "client_addr": "v1:172.21.15.92:0/2403132573", 2024-03-11T21:31:58.194 INFO:teuthology.orchestra.run.smithi092.stdout: "tid": 115732 2024-03-11T21:31:58.194 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:31:58.194 INFO:teuthology.orchestra.run.smithi092.stdout: "events": [ 2024-03-11T21:31:58.194 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:31:58.194 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "initiated", 2024-03-11T21:31:58.194 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:29:08.166630+0000", 2024-03-11T21:31:58.194 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-03-11T21:31:58.195 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:31:58.195 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:31:58.195 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "header_read", 2024-03-11T21:31:58.195 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:29:08.166630+0000", 2024-03-11T21:31:58.195 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-03-11T21:31:58.195 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:31:58.195 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:31:58.195 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "throttled", 2024-03-11T21:31:58.195 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:29:08.166634+0000", 2024-03-11T21:31:58.195 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 3.4680000000000001e-06 2024-03-11T21:31:58.195 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:31:58.195 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:31:58.195 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "all_read", 2024-03-11T21:31:58.195 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:29:08.166660+0000", 2024-03-11T21:31:58.196 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 2.6512e-05 2024-03-11T21:31:58.196 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:31:58.196 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:31:58.196 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "dispatched", 2024-03-11T21:31:58.196 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:29:08.166674+0000", 2024-03-11T21:31:58.196 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.382e-05 2024-03-11T21:31:58.196 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:31:58.196 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:31:58.196 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "queued_for_pg", 2024-03-11T21:31:58.196 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:29:08.166685+0000", 2024-03-11T21:31:58.196 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.1277999999999999e-05 2024-03-11T21:31:58.196 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:31:58.197 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:31:58.197 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-03-11T21:31:58.197 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:29:08.166750+0000", 2024-03-11T21:31:58.197 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 6.4522000000000003e-05 2024-03-11T21:31:58.197 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:31:58.197 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:31:58.197 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "started", 2024-03-11T21:31:58.197 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:29:08.166894+0000", 2024-03-11T21:31:58.197 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.000144499 2024-03-11T21:31:58.197 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:31:58.197 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:31:58.197 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "waiting for subops from 0,5", 2024-03-11T21:31:58.197 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:29:08.167063+0000", 2024-03-11T21:31:58.197 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00016860700000000001 2024-03-11T21:31:58.198 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:31:58.198 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:31:58.198 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "sub_op_commit_rec", 2024-03-11T21:31:58.198 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:29:08.172918+0000", 2024-03-11T21:31:58.198 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.0058549780000000003 2024-03-11T21:31:58.198 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:31:58.198 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:31:58.198 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "op_commit", 2024-03-11T21:31:58.198 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:29:08.181135+0000", 2024-03-11T21:31:58.198 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.0082175979999999996 2024-03-11T21:31:58.198 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:31:58.198 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:31:58.198 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "sub_op_commit_rec", 2024-03-11T21:31:58.198 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:29:08.265705+0000", 2024-03-11T21:31:58.198 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.084569474000000006 2024-03-11T21:31:58.199 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:31:58.199 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:31:58.199 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "commit_sent", 2024-03-11T21:31:58.199 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:29:08.265763+0000", 2024-03-11T21:31:58.199 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 5.8242999999999998e-05 2024-03-11T21:31:58.199 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:31:58.199 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:31:58.199 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "done", 2024-03-11T21:31:58.199 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:29:08.265807+0000", 2024-03-11T21:31:58.199 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 4.4280999999999998e-05 2024-03-11T21:31:58.199 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-03-11T21:31:58.199 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-03-11T21:31:58.199 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-03-11T21:31:58.199 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:31:58.199 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:31:58.200 INFO:teuthology.orchestra.run.smithi092.stdout: "description": "osd_op(client.34715.0:115741 7.e 7:72f34471:::benchmark_data_smithi092_76040_object115740:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e119)", 2024-03-11T21:31:58.200 INFO:teuthology.orchestra.run.smithi092.stdout: "initiated_at": "2024-03-11T21:29:08.181657+0000", 2024-03-11T21:31:58.200 INFO:teuthology.orchestra.run.smithi092.stdout: "age": 170.00889125099999, 2024-03-11T21:31:58.200 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.084254920999999997, 2024-03-11T21:31:58.200 INFO:teuthology.orchestra.run.smithi092.stdout: "type_data": { 2024-03-11T21:31:58.200 INFO:teuthology.orchestra.run.smithi092.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-11T21:31:58.200 INFO:teuthology.orchestra.run.smithi092.stdout: "client_info": { 2024-03-11T21:31:58.200 INFO:teuthology.orchestra.run.smithi092.stdout: "client": "client.34715", 2024-03-11T21:31:58.200 INFO:teuthology.orchestra.run.smithi092.stdout: "client_addr": "v1:172.21.15.92:0/2403132573", 2024-03-11T21:31:58.200 INFO:teuthology.orchestra.run.smithi092.stdout: "tid": 115741 2024-03-11T21:31:58.200 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:31:58.200 INFO:teuthology.orchestra.run.smithi092.stdout: "events": [ 2024-03-11T21:31:58.200 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:31:58.200 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "initiated", 2024-03-11T21:31:58.200 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:29:08.181657+0000", 2024-03-11T21:31:58.201 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-03-11T21:31:58.201 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:31:58.201 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:31:58.201 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "header_read", 2024-03-11T21:31:58.201 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:29:08.181657+0000", 2024-03-11T21:31:58.201 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-03-11T21:31:58.201 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:31:58.201 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:31:58.201 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "throttled", 2024-03-11T21:31:58.201 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:29:08.181659+0000", 2024-03-11T21:31:58.201 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.6980000000000001e-06 2024-03-11T21:31:58.201 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:31:58.201 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:31:58.201 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "all_read", 2024-03-11T21:31:58.201 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:29:08.181688+0000", 2024-03-11T21:31:58.202 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 2.8827999999999999e-05 2024-03-11T21:31:58.202 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:31:58.202 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:31:58.202 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "dispatched", 2024-03-11T21:31:58.202 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:29:08.181700+0000", 2024-03-11T21:31:58.202 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.1939999999999999e-05 2024-03-11T21:31:58.202 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:31:58.202 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:31:58.202 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "queued_for_pg", 2024-03-11T21:31:58.202 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:29:08.181708+0000", 2024-03-11T21:31:58.202 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 8.5890000000000003e-06 2024-03-11T21:31:58.202 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:31:58.202 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:31:58.202 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-03-11T21:31:58.202 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:29:08.181755+0000", 2024-03-11T21:31:58.203 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 4.6702000000000001e-05 2024-03-11T21:31:58.203 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:31:58.203 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:31:58.203 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "started", 2024-03-11T21:31:58.203 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:29:08.181870+0000", 2024-03-11T21:31:58.203 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00011508 2024-03-11T21:31:58.203 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:31:58.203 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:31:58.203 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "waiting for subops from 0,5", 2024-03-11T21:31:58.203 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:29:08.182022+0000", 2024-03-11T21:31:58.203 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00015190099999999999 2024-03-11T21:31:58.203 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:31:58.203 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:31:58.203 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "sub_op_commit_rec", 2024-03-11T21:31:58.204 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:29:08.184981+0000", 2024-03-11T21:31:58.204 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.0029593559999999998 2024-03-11T21:31:58.204 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:31:58.204 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:31:58.204 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "op_commit", 2024-03-11T21:31:58.204 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:29:08.186599+0000", 2024-03-11T21:31:58.204 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.0016174830000000001 2024-03-11T21:31:58.204 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:31:58.204 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:31:58.204 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "sub_op_commit_rec", 2024-03-11T21:31:58.204 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:29:08.265841+0000", 2024-03-11T21:31:58.204 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.079241696 2024-03-11T21:31:58.204 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:31:58.204 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:31:58.204 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "commit_sent", 2024-03-11T21:31:58.205 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:29:08.265883+0000", 2024-03-11T21:31:58.205 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 4.2534000000000003e-05 2024-03-11T21:31:58.205 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:31:58.205 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:31:58.205 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "done", 2024-03-11T21:31:58.205 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:29:08.265912+0000", 2024-03-11T21:31:58.205 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 2.9113999999999999e-05 2024-03-11T21:31:58.205 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-03-11T21:31:58.205 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-03-11T21:31:58.205 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-03-11T21:31:58.205 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:31:58.205 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:31:58.205 INFO:teuthology.orchestra.run.smithi092.stdout: "description": "osd_op(client.34715.0:115749 7.8 7:183d5f6b:::benchmark_data_smithi092_76040_object115748:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e119)", 2024-03-11T21:31:58.205 INFO:teuthology.orchestra.run.smithi092.stdout: "initiated_at": "2024-03-11T21:29:08.204342+0000", 2024-03-11T21:31:58.206 INFO:teuthology.orchestra.run.smithi092.stdout: "age": 169.98620684100001, 2024-03-11T21:31:58.206 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.077972018000000004, 2024-03-11T21:31:58.206 INFO:teuthology.orchestra.run.smithi092.stdout: "type_data": { 2024-03-11T21:31:58.206 INFO:teuthology.orchestra.run.smithi092.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-11T21:31:58.206 INFO:teuthology.orchestra.run.smithi092.stdout: "client_info": { 2024-03-11T21:31:58.206 INFO:teuthology.orchestra.run.smithi092.stdout: "client": "client.34715", 2024-03-11T21:31:58.206 INFO:teuthology.orchestra.run.smithi092.stdout: "client_addr": "v1:172.21.15.92:0/2403132573", 2024-03-11T21:31:58.206 INFO:teuthology.orchestra.run.smithi092.stdout: "tid": 115749 2024-03-11T21:31:58.206 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:31:58.206 INFO:teuthology.orchestra.run.smithi092.stdout: "events": [ 2024-03-11T21:31:58.206 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:31:58.206 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "initiated", 2024-03-11T21:31:58.207 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:29:08.204342+0000", 2024-03-11T21:31:58.207 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-03-11T21:31:58.207 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:31:58.207 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:31:58.207 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "header_read", 2024-03-11T21:31:58.207 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:29:08.204342+0000", 2024-03-11T21:31:58.207 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-03-11T21:31:58.207 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:31:58.207 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:31:58.207 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "throttled", 2024-03-11T21:31:58.207 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:29:08.204344+0000", 2024-03-11T21:31:58.207 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.855e-06 2024-03-11T21:31:58.207 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:31:58.207 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:31:58.207 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "all_read", 2024-03-11T21:31:58.208 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:29:08.204385+0000", 2024-03-11T21:31:58.208 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 4.0883999999999997e-05 2024-03-11T21:31:58.208 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:31:58.208 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:31:58.208 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "dispatched", 2024-03-11T21:31:58.208 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:29:08.204399+0000", 2024-03-11T21:31:58.208 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.4072e-05 2024-03-11T21:31:58.208 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:31:58.208 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:31:58.208 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "queued_for_pg", 2024-03-11T21:31:58.208 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:29:08.204408+0000", 2024-03-11T21:31:58.208 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 9.2799999999999992e-06 2024-03-11T21:31:58.208 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:31:58.208 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:31:58.209 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-03-11T21:31:58.209 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:29:08.204505+0000", 2024-03-11T21:31:58.209 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 9.6830000000000006e-05 2024-03-11T21:31:58.209 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:31:58.209 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:31:58.209 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "started", 2024-03-11T21:31:58.209 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:29:08.265094+0000", 2024-03-11T21:31:58.209 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.060589257000000001 2024-03-11T21:31:58.209 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:31:58.209 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:31:58.209 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "waiting for subops from 4,7", 2024-03-11T21:31:58.209 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:29:08.267947+0000", 2024-03-11T21:31:58.209 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.0028534350000000001 2024-03-11T21:31:58.209 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:31:58.209 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:31:58.210 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "sub_op_commit_rec", 2024-03-11T21:31:58.210 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:29:08.282150+0000", 2024-03-11T21:31:58.210 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.014202177999999999 2024-03-11T21:31:58.210 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:31:58.210 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:31:58.210 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "op_commit", 2024-03-11T21:31:58.210 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:29:08.282161+0000", 2024-03-11T21:31:58.210 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.1663e-05 2024-03-11T21:31:58.210 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:31:58.210 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:31:58.210 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "sub_op_commit_rec", 2024-03-11T21:31:58.210 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:29:08.282192+0000", 2024-03-11T21:31:58.210 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 3.0923000000000002e-05 2024-03-11T21:31:58.210 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:31:58.210 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:31:58.211 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "commit_sent", 2024-03-11T21:31:58.211 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:29:08.282248+0000", 2024-03-11T21:31:58.211 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 5.5933000000000001e-05 2024-03-11T21:31:58.211 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:31:58.211 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:31:58.211 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "done", 2024-03-11T21:31:58.211 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:29:08.282314+0000", 2024-03-11T21:31:58.211 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 6.5708000000000002e-05 2024-03-11T21:31:58.211 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-03-11T21:31:58.212 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-03-11T21:31:58.212 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-03-11T21:31:58.212 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:31:58.212 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:31:58.212 INFO:teuthology.orchestra.run.smithi092.stdout: "description": "osd_op(client.34715.0:116287 7.8 7:161bac11:::benchmark_data_smithi092_76040_object116286:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e119)", 2024-03-11T21:31:58.212 INFO:teuthology.orchestra.run.smithi092.stdout: "initiated_at": "2024-03-11T21:29:08.771810+0000", 2024-03-11T21:31:58.212 INFO:teuthology.orchestra.run.smithi092.stdout: "age": 169.418739077, 2024-03-11T21:31:58.212 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.10587118700000001, 2024-03-11T21:31:58.212 INFO:teuthology.orchestra.run.smithi092.stdout: "type_data": { 2024-03-11T21:31:58.212 INFO:teuthology.orchestra.run.smithi092.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-11T21:31:58.212 INFO:teuthology.orchestra.run.smithi092.stdout: "client_info": { 2024-03-11T21:31:58.212 INFO:teuthology.orchestra.run.smithi092.stdout: "client": "client.34715", 2024-03-11T21:31:58.212 INFO:teuthology.orchestra.run.smithi092.stdout: "client_addr": "v1:172.21.15.92:0/2403132573", 2024-03-11T21:31:58.213 INFO:teuthology.orchestra.run.smithi092.stdout: "tid": 116287 2024-03-11T21:31:58.213 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:31:58.213 INFO:teuthology.orchestra.run.smithi092.stdout: "events": [ 2024-03-11T21:31:58.213 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:31:58.213 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "initiated", 2024-03-11T21:31:58.213 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:29:08.771810+0000", 2024-03-11T21:31:58.213 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-03-11T21:31:58.213 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:31:58.213 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:31:58.213 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "header_read", 2024-03-11T21:31:58.213 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:29:08.771810+0000", 2024-03-11T21:31:58.213 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-03-11T21:31:58.213 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:31:58.213 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:31:58.213 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "throttled", 2024-03-11T21:31:58.214 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:29:08.771811+0000", 2024-03-11T21:31:58.214 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.793e-06 2024-03-11T21:31:58.214 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:31:58.214 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:31:58.214 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "all_read", 2024-03-11T21:31:58.214 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:29:08.771832+0000", 2024-03-11T21:31:58.214 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 2.0641000000000001e-05 2024-03-11T21:31:58.214 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:31:58.214 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:31:58.214 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "dispatched", 2024-03-11T21:31:58.214 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:29:08.771841+0000", 2024-03-11T21:31:58.214 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 9.2939999999999994e-06 2024-03-11T21:31:58.214 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:31:58.214 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:31:58.215 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "queued_for_pg", 2024-03-11T21:31:58.215 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:29:08.771849+0000", 2024-03-11T21:31:58.215 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 7.2239999999999998e-06 2024-03-11T21:31:58.215 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:31:58.215 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:31:58.215 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-03-11T21:31:58.215 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:29:08.771933+0000", 2024-03-11T21:31:58.215 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 8.4158000000000003e-05 2024-03-11T21:31:58.215 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:31:58.215 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:31:58.215 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "started", 2024-03-11T21:31:58.215 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:29:08.772098+0000", 2024-03-11T21:31:58.215 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00016576199999999999 2024-03-11T21:31:58.215 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:31:58.215 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:31:58.216 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "waiting for subops from 4,7", 2024-03-11T21:31:58.216 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:29:08.772361+0000", 2024-03-11T21:31:58.216 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00026245400000000002 2024-03-11T21:31:58.216 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:31:58.216 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:31:58.216 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "op_commit", 2024-03-11T21:31:58.216 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:29:08.877415+0000", 2024-03-11T21:31:58.216 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.10505452 2024-03-11T21:31:58.216 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:31:58.216 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:31:58.216 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "sub_op_commit_rec", 2024-03-11T21:31:58.216 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:29:08.877469+0000", 2024-03-11T21:31:58.216 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 5.3115000000000002e-05 2024-03-11T21:31:58.216 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:31:58.216 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:31:58.217 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "sub_op_commit_rec", 2024-03-11T21:31:58.217 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:29:08.877521+0000", 2024-03-11T21:31:58.217 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 5.2649999999999999e-05 2024-03-11T21:31:58.217 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:31:58.217 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:31:58.217 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "commit_sent", 2024-03-11T21:31:58.217 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:29:08.877632+0000", 2024-03-11T21:31:58.217 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00011079800000000001 2024-03-11T21:31:58.217 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:31:58.217 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:31:58.217 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "done", 2024-03-11T21:31:58.217 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:29:08.877681+0000", 2024-03-11T21:31:58.217 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 4.8778e-05 2024-03-11T21:31:58.217 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-03-11T21:31:58.218 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-03-11T21:31:58.218 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-03-11T21:31:58.218 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:31:58.218 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:31:58.218 INFO:teuthology.orchestra.run.smithi092.stdout: "description": "osd_repop(client.34715.0:116286 7.5 e119/99 7:a74abdea:::benchmark_data_smithi092_76040_object116285:head v 119'7286, mlcod=119'7283)", 2024-03-11T21:31:58.218 INFO:teuthology.orchestra.run.smithi092.stdout: "initiated_at": "2024-03-11T21:29:08.772243+0000", 2024-03-11T21:31:58.218 INFO:teuthology.orchestra.run.smithi092.stdout: "age": 169.418305796, 2024-03-11T21:31:58.218 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.105122486, 2024-03-11T21:31:58.218 INFO:teuthology.orchestra.run.smithi092.stdout: "type_data": { 2024-03-11T21:31:58.218 INFO:teuthology.orchestra.run.smithi092.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-11T21:31:58.218 INFO:teuthology.orchestra.run.smithi092.stdout: "events": [ 2024-03-11T21:31:58.218 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:31:58.218 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "initiated", 2024-03-11T21:31:58.218 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:29:08.772243+0000", 2024-03-11T21:31:58.219 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-03-11T21:31:58.219 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:31:58.219 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:31:58.219 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "header_read", 2024-03-11T21:31:58.219 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:29:08.772243+0000", 2024-03-11T21:31:58.219 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-03-11T21:31:58.219 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:31:58.219 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:31:58.219 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "throttled", 2024-03-11T21:31:58.219 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:29:08.772244+0000", 2024-03-11T21:31:58.219 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 9.7600000000000006e-07 2024-03-11T21:31:58.219 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:31:58.219 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:31:58.219 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "all_read", 2024-03-11T21:31:58.220 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:29:08.772287+0000", 2024-03-11T21:31:58.220 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 4.3052e-05 2024-03-11T21:31:58.220 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:31:58.220 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:31:58.220 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "dispatched", 2024-03-11T21:31:58.220 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:29:08.772295+0000", 2024-03-11T21:31:58.220 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 7.7989999999999994e-06 2024-03-11T21:31:58.220 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:31:58.220 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:31:58.220 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "queued_for_pg", 2024-03-11T21:31:58.220 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:29:08.772301+0000", 2024-03-11T21:31:58.220 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 6.1319999999999999e-06 2024-03-11T21:31:58.220 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:31:58.220 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:31:58.220 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-03-11T21:31:58.221 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:29:08.772348+0000", 2024-03-11T21:31:58.221 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 4.6999000000000002e-05 2024-03-11T21:31:58.221 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:31:58.221 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:31:58.221 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "started", 2024-03-11T21:31:58.221 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:29:08.772377+0000", 2024-03-11T21:31:58.221 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 2.9246000000000001e-05 2024-03-11T21:31:58.221 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:31:58.221 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:31:58.221 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "commit_sent", 2024-03-11T21:31:58.221 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:29:08.775548+0000", 2024-03-11T21:31:58.221 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.0031711740000000001 2024-03-11T21:31:58.221 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:31:58.221 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:31:58.221 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "done", 2024-03-11T21:31:58.222 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:29:08.877365+0000", 2024-03-11T21:31:58.222 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.101817108 2024-03-11T21:31:58.222 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-03-11T21:31:58.222 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-03-11T21:31:58.222 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-03-11T21:31:58.222 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:31:58.222 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:31:58.222 INFO:teuthology.orchestra.run.smithi092.stdout: "description": "osd_repop_reply(client.34715.0:116287 7.8 e119/99 ondisk, result = 0)", 2024-03-11T21:31:58.222 INFO:teuthology.orchestra.run.smithi092.stdout: "initiated_at": "2024-03-11T21:29:08.776070+0000", 2024-03-11T21:31:58.222 INFO:teuthology.orchestra.run.smithi092.stdout: "age": 169.41447874400001, 2024-03-11T21:31:58.222 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.10140663900000001, 2024-03-11T21:31:58.222 INFO:teuthology.orchestra.run.smithi092.stdout: "type_data": { 2024-03-11T21:31:58.222 INFO:teuthology.orchestra.run.smithi092.stdout: "flag_point": "started", 2024-03-11T21:31:58.222 INFO:teuthology.orchestra.run.smithi092.stdout: "events": [ 2024-03-11T21:31:58.222 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:31:58.223 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "initiated", 2024-03-11T21:31:58.223 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:29:08.776070+0000", 2024-03-11T21:31:58.223 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-03-11T21:31:58.223 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:31:58.223 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:31:58.223 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "header_read", 2024-03-11T21:31:58.223 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:29:08.776070+0000", 2024-03-11T21:31:58.223 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-03-11T21:31:58.223 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:31:58.223 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:31:58.223 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "throttled", 2024-03-11T21:31:58.223 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:29:08.776071+0000", 2024-03-11T21:31:58.223 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.265e-06 2024-03-11T21:31:58.223 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:31:58.223 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:31:58.224 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "all_read", 2024-03-11T21:31:58.224 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:29:08.776075+0000", 2024-03-11T21:31:58.224 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 4.2930000000000002e-06 2024-03-11T21:31:58.224 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:31:58.224 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:31:58.224 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "dispatched", 2024-03-11T21:31:58.224 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:29:08.877397+0000", 2024-03-11T21:31:58.224 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.10132176699999999 2024-03-11T21:31:58.224 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:31:58.224 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:31:58.224 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "queued_for_pg", 2024-03-11T21:31:58.224 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:29:08.877406+0000", 2024-03-11T21:31:58.224 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 9.1779999999999992e-06 2024-03-11T21:31:58.224 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:31:58.224 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:31:58.225 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-03-11T21:31:58.225 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:29:08.877454+0000", 2024-03-11T21:31:58.225 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 4.8087000000000002e-05 2024-03-11T21:31:58.225 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:31:58.225 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:31:58.225 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "started", 2024-03-11T21:31:58.225 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:29:08.877463+0000", 2024-03-11T21:31:58.225 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 8.5350000000000004e-06 2024-03-11T21:31:58.225 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:31:58.225 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:31:58.225 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "done", 2024-03-11T21:31:58.225 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:29:08.877477+0000", 2024-03-11T21:31:58.225 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.3514e-05 2024-03-11T21:31:58.225 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-03-11T21:31:58.225 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-03-11T21:31:58.225 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-03-11T21:31:58.226 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:31:58.226 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:31:58.226 INFO:teuthology.orchestra.run.smithi092.stdout: "description": "osd_repop(client.34715.0:116325 7.a e119/99 7:50f8f4bc:::benchmark_data_smithi092_76040_object116324:head v 119'7234, mlcod=119'7231)", 2024-03-11T21:31:58.226 INFO:teuthology.orchestra.run.smithi092.stdout: "initiated_at": "2024-03-11T21:29:08.914403+0000", 2024-03-11T21:31:58.226 INFO:teuthology.orchestra.run.smithi092.stdout: "age": 169.27614537700001, 2024-03-11T21:31:58.226 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.079377257000000007, 2024-03-11T21:31:58.226 INFO:teuthology.orchestra.run.smithi092.stdout: "type_data": { 2024-03-11T21:31:58.226 INFO:teuthology.orchestra.run.smithi092.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-11T21:31:58.226 INFO:teuthology.orchestra.run.smithi092.stdout: "events": [ 2024-03-11T21:31:58.226 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:31:58.226 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "initiated", 2024-03-11T21:31:58.226 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:29:08.914403+0000", 2024-03-11T21:31:58.226 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-03-11T21:31:58.226 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:31:58.227 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:31:58.227 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "header_read", 2024-03-11T21:31:58.227 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:29:08.914403+0000", 2024-03-11T21:31:58.227 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-03-11T21:31:58.227 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:31:58.227 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:31:58.227 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "throttled", 2024-03-11T21:31:58.227 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:29:08.914404+0000", 2024-03-11T21:31:58.227 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 5.6599999999999996e-07 2024-03-11T21:31:58.227 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:31:58.227 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:31:58.227 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "all_read", 2024-03-11T21:31:58.227 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:29:08.914431+0000", 2024-03-11T21:31:58.227 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 2.7061000000000001e-05 2024-03-11T21:31:58.227 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:31:58.227 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:31:58.228 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "dispatched", 2024-03-11T21:31:58.228 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:29:08.914437+0000", 2024-03-11T21:31:58.228 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 6.0000000000000002e-06 2024-03-11T21:31:58.228 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:31:58.228 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:31:58.228 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:31:58.230 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "queued_for_pg", 2024-03-11T21:31:58.230 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:29:08.914442+0000", 2024-03-11T21:31:58.230 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 4.741e-06 2024-03-11T21:31:58.230 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:31:58.230 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:31:58.230 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-03-11T21:31:58.230 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:29:08.986768+0000", 2024-03-11T21:31:58.230 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.072325946000000002 2024-03-11T21:31:58.230 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:31:58.230 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:31:58.230 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "started", 2024-03-11T21:31:58.230 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:29:08.986797+0000", 2024-03-11T21:31:58.231 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 2.9764000000000002e-05 2024-03-11T21:31:58.231 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:31:58.231 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:31:58.231 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "commit_sent", 2024-03-11T21:31:58.231 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:29:08.993737+0000", 2024-03-11T21:31:58.231 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.0069398070000000001 2024-03-11T21:31:58.231 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:31:58.231 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:31:58.231 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "done", 2024-03-11T21:31:58.231 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:29:08.993781+0000", 2024-03-11T21:31:58.231 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 4.3371999999999997e-05 2024-03-11T21:31:58.231 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-03-11T21:31:58.231 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-03-11T21:31:58.231 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-03-11T21:31:58.232 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:31:58.232 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:31:58.232 INFO:teuthology.orchestra.run.smithi092.stdout: "description": "osd_op(client.34715.0:116333 7.8 7:12ed9624:::benchmark_data_smithi092_76040_object116332:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e119)", 2024-03-11T21:31:58.232 INFO:teuthology.orchestra.run.smithi092.stdout: "initiated_at": "2024-03-11T21:29:08.915546+0000", 2024-03-11T21:31:58.232 INFO:teuthology.orchestra.run.smithi092.stdout: "age": 169.27500306900001, 2024-03-11T21:31:58.232 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.082294330999999998, 2024-03-11T21:31:58.232 INFO:teuthology.orchestra.run.smithi092.stdout: "type_data": { 2024-03-11T21:31:58.232 INFO:teuthology.orchestra.run.smithi092.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-11T21:31:58.232 INFO:teuthology.orchestra.run.smithi092.stdout: "client_info": { 2024-03-11T21:31:58.232 INFO:teuthology.orchestra.run.smithi092.stdout: "client": "client.34715", 2024-03-11T21:31:58.232 INFO:teuthology.orchestra.run.smithi092.stdout: "client_addr": "v1:172.21.15.92:0/2403132573", 2024-03-11T21:31:58.232 INFO:teuthology.orchestra.run.smithi092.stdout: "tid": 116333 2024-03-11T21:31:58.232 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:31:58.232 INFO:teuthology.orchestra.run.smithi092.stdout: "events": [ 2024-03-11T21:31:58.232 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:31:58.233 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "initiated", 2024-03-11T21:31:58.233 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:29:08.915546+0000", 2024-03-11T21:31:58.233 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-03-11T21:31:58.233 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:31:58.233 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:31:58.233 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "header_read", 2024-03-11T21:31:58.233 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:29:08.915546+0000", 2024-03-11T21:31:58.233 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-03-11T21:31:58.233 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:31:58.233 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:31:58.233 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "throttled", 2024-03-11T21:31:58.233 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:29:08.915548+0000", 2024-03-11T21:31:58.233 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 2.3159999999999999e-06 2024-03-11T21:31:58.233 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:31:58.233 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:31:58.234 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "all_read", 2024-03-11T21:31:58.234 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:29:08.915571+0000", 2024-03-11T21:31:58.234 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 2.2838999999999999e-05 2024-03-11T21:31:58.234 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:31:58.234 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:31:58.234 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "dispatched", 2024-03-11T21:31:58.234 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:29:08.915582+0000", 2024-03-11T21:31:58.234 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.1673e-05 2024-03-11T21:31:58.234 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:31:58.234 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:31:58.234 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "queued_for_pg", 2024-03-11T21:31:58.235 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:29:08.915590+0000", 2024-03-11T21:31:58.235 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 7.9289999999999998e-06 2024-03-11T21:31:58.235 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:31:58.235 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:31:58.235 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-03-11T21:31:58.235 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:29:08.916889+0000", 2024-03-11T21:31:58.235 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.001298537 2024-03-11T21:31:58.235 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:31:58.235 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:31:58.235 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "started", 2024-03-11T21:31:58.235 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:29:08.917043+0000", 2024-03-11T21:31:58.235 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00015451500000000001 2024-03-11T21:31:58.235 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:31:58.235 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:31:58.235 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "waiting for subops from 4,7", 2024-03-11T21:31:58.236 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:29:08.917187+0000", 2024-03-11T21:31:58.236 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00014380799999999999 2024-03-11T21:31:58.236 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:31:58.236 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:31:58.236 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "sub_op_commit_rec", 2024-03-11T21:31:58.236 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:29:08.990374+0000", 2024-03-11T21:31:58.236 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.073186651000000005 2024-03-11T21:31:58.236 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:31:58.236 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:31:58.236 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "sub_op_commit_rec", 2024-03-11T21:31:58.236 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:29:08.990416+0000", 2024-03-11T21:31:58.236 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 4.2548e-05 2024-03-11T21:31:58.236 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:31:58.236 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:31:58.237 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "op_commit", 2024-03-11T21:31:58.237 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:29:08.997697+0000", 2024-03-11T21:31:58.237 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.0072802450000000003 2024-03-11T21:31:58.237 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:31:58.237 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:31:58.237 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "commit_sent", 2024-03-11T21:31:58.237 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:29:08.997791+0000", 2024-03-11T21:31:58.237 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 9.4796000000000005e-05 2024-03-11T21:31:58.237 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:31:58.237 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:31:58.237 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "done", 2024-03-11T21:31:58.237 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:29:08.997840+0000", 2024-03-11T21:31:58.237 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 4.8473999999999997e-05 2024-03-11T21:31:58.237 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-03-11T21:31:58.237 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-03-11T21:31:58.238 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-03-11T21:31:58.238 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:31:58.238 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:31:58.238 INFO:teuthology.orchestra.run.smithi092.stdout: "description": "osd_op(client.34715.0:116335 7.8 7:1fa7fab8:::benchmark_data_smithi092_76040_object116334:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e119)", 2024-03-11T21:31:58.238 INFO:teuthology.orchestra.run.smithi092.stdout: "initiated_at": "2024-03-11T21:29:08.916101+0000", 2024-03-11T21:31:58.238 INFO:teuthology.orchestra.run.smithi092.stdout: "age": 169.274447407, 2024-03-11T21:31:58.238 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.081878851000000002, 2024-03-11T21:31:58.238 INFO:teuthology.orchestra.run.smithi092.stdout: "type_data": { 2024-03-11T21:31:58.238 INFO:teuthology.orchestra.run.smithi092.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-11T21:31:58.238 INFO:teuthology.orchestra.run.smithi092.stdout: "client_info": { 2024-03-11T21:31:58.238 INFO:teuthology.orchestra.run.smithi092.stdout: "client": "client.34715", 2024-03-11T21:31:58.238 INFO:teuthology.orchestra.run.smithi092.stdout: "client_addr": "v1:172.21.15.92:0/2403132573", 2024-03-11T21:31:58.238 INFO:teuthology.orchestra.run.smithi092.stdout: "tid": 116335 2024-03-11T21:31:58.238 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:31:58.239 INFO:teuthology.orchestra.run.smithi092.stdout: "events": [ 2024-03-11T21:31:58.239 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:31:58.239 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "initiated", 2024-03-11T21:31:58.239 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:29:08.916101+0000", 2024-03-11T21:31:58.239 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-03-11T21:31:58.239 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:31:58.239 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:31:58.239 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "header_read", 2024-03-11T21:31:58.239 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:29:08.916101+0000", 2024-03-11T21:31:58.239 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-03-11T21:31:58.239 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:31:58.239 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:31:58.239 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "throttled", 2024-03-11T21:31:58.239 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:29:08.916103+0000", 2024-03-11T21:31:58.239 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.5910000000000001e-06 2024-03-11T21:31:58.240 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:31:58.240 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:31:58.240 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "all_read", 2024-03-11T21:31:58.240 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:29:08.916124+0000", 2024-03-11T21:31:58.240 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 2.1481e-05 2024-03-11T21:31:58.240 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:31:58.240 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:31:58.240 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "dispatched", 2024-03-11T21:31:58.240 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:29:08.916135+0000", 2024-03-11T21:31:58.240 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.0721e-05 2024-03-11T21:31:58.240 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:31:58.240 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:31:58.240 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "queued_for_pg", 2024-03-11T21:31:58.240 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:29:08.916142+0000", 2024-03-11T21:31:58.240 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 7.3390000000000004e-06 2024-03-11T21:31:58.241 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:31:58.241 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:31:58.241 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-03-11T21:31:58.241 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:29:08.990446+0000", 2024-03-11T21:31:58.241 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.07430349 2024-03-11T21:31:58.241 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:31:58.241 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:31:58.241 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "started", 2024-03-11T21:31:58.241 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:29:08.990549+0000", 2024-03-11T21:31:58.241 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.000103267 2024-03-11T21:31:58.241 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:31:58.241 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:31:58.241 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "waiting for subops from 4,7", 2024-03-11T21:31:58.241 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:29:08.990686+0000", 2024-03-11T21:31:58.241 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00013649800000000001 2024-03-11T21:31:58.242 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:31:58.242 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:31:58.242 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "sub_op_commit_rec", 2024-03-11T21:31:58.242 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:29:08.997639+0000", 2024-03-11T21:31:58.242 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.0069533199999999998 2024-03-11T21:31:58.242 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:31:58.242 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:31:58.242 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "sub_op_commit_rec", 2024-03-11T21:31:58.242 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:29:08.997683+0000", 2024-03-11T21:31:58.242 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 4.3591000000000003e-05 2024-03-11T21:31:58.242 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:31:58.242 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:31:58.242 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "op_commit", 2024-03-11T21:31:58.242 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:29:08.997850+0000", 2024-03-11T21:31:58.243 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00016767099999999999 2024-03-11T21:31:58.243 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:31:58.243 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:31:58.243 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "commit_sent", 2024-03-11T21:31:58.243 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:29:08.997943+0000", 2024-03-11T21:31:58.243 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 9.2633000000000006e-05 2024-03-11T21:31:58.243 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:31:58.243 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:31:58.243 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "done", 2024-03-11T21:31:58.243 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:29:08.997980+0000", 2024-03-11T21:31:58.243 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 3.7249000000000002e-05 2024-03-11T21:31:58.243 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-03-11T21:31:58.243 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-03-11T21:31:58.243 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-03-11T21:31:58.244 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:31:58.244 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:31:58.244 INFO:teuthology.orchestra.run.smithi092.stdout: "description": "osd_op(client.34715.0:116339 7.e 7:7e6296f0:::benchmark_data_smithi092_76040_object116338:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e119)", 2024-03-11T21:31:58.244 INFO:teuthology.orchestra.run.smithi092.stdout: "initiated_at": "2024-03-11T21:29:08.918203+0000", 2024-03-11T21:31:58.244 INFO:teuthology.orchestra.run.smithi092.stdout: "age": 169.272345304, 2024-03-11T21:31:58.244 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.084915199999999996, 2024-03-11T21:31:58.244 INFO:teuthology.orchestra.run.smithi092.stdout: "type_data": { 2024-03-11T21:31:58.244 INFO:teuthology.orchestra.run.smithi092.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-11T21:31:58.244 INFO:teuthology.orchestra.run.smithi092.stdout: "client_info": { 2024-03-11T21:31:58.244 INFO:teuthology.orchestra.run.smithi092.stdout: "client": "client.34715", 2024-03-11T21:31:58.244 INFO:teuthology.orchestra.run.smithi092.stdout: "client_addr": "v1:172.21.15.92:0/2403132573", 2024-03-11T21:31:58.244 INFO:teuthology.orchestra.run.smithi092.stdout: "tid": 116339 2024-03-11T21:31:58.244 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:31:58.244 INFO:teuthology.orchestra.run.smithi092.stdout: "events": [ 2024-03-11T21:31:58.244 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:31:58.245 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "initiated", 2024-03-11T21:31:58.245 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:29:08.918203+0000", 2024-03-11T21:31:58.245 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-03-11T21:31:58.245 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:31:58.245 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:31:58.245 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "header_read", 2024-03-11T21:31:58.245 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:29:08.918203+0000", 2024-03-11T21:31:58.245 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-03-11T21:31:58.245 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:31:58.245 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:31:58.245 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "throttled", 2024-03-11T21:31:58.245 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:29:08.918205+0000", 2024-03-11T21:31:58.245 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.933e-06 2024-03-11T21:31:58.245 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:31:58.246 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:31:58.246 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "all_read", 2024-03-11T21:31:58.246 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:29:08.918226+0000", 2024-03-11T21:31:58.246 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 2.1098999999999999e-05 2024-03-11T21:31:58.246 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:31:58.246 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:31:58.246 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "dispatched", 2024-03-11T21:31:58.246 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:29:08.986734+0000", 2024-03-11T21:31:58.246 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.068507799999999994 2024-03-11T21:31:58.246 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:31:58.246 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:31:58.246 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "queued_for_pg", 2024-03-11T21:31:58.246 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:29:08.986752+0000", 2024-03-11T21:31:58.246 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.8315999999999999e-05 2024-03-11T21:31:58.246 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:31:58.247 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:31:58.247 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-03-11T21:31:58.247 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:29:08.986828+0000", 2024-03-11T21:31:58.247 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 7.5916999999999998e-05 2024-03-11T21:31:58.247 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:31:58.247 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:31:58.247 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "started", 2024-03-11T21:31:58.247 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:29:08.986957+0000", 2024-03-11T21:31:58.247 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00012831600000000001 2024-03-11T21:31:58.247 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:31:58.247 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:31:58.247 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "waiting for subops from 0,5", 2024-03-11T21:31:58.247 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:29:08.987118+0000", 2024-03-11T21:31:58.247 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00016082999999999999 2024-03-11T21:31:58.247 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:31:58.248 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:31:58.248 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "sub_op_commit_rec", 2024-03-11T21:31:58.248 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:29:08.993823+0000", 2024-03-11T21:31:58.248 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.0067059010000000002 2024-03-11T21:31:58.248 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:31:58.248 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:31:58.248 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "op_commit", 2024-03-11T21:31:58.248 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:29:08.993877+0000", 2024-03-11T21:31:58.248 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 5.3913999999999999e-05 2024-03-11T21:31:58.248 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:31:58.248 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:31:58.248 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "sub_op_commit_rec", 2024-03-11T21:31:58.248 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:29:09.003018+0000", 2024-03-11T21:31:58.248 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.0091407610000000007 2024-03-11T21:31:58.248 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:31:58.249 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:31:58.249 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "commit_sent", 2024-03-11T21:31:58.249 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:29:09.003074+0000", 2024-03-11T21:31:58.249 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 5.6165000000000002e-05 2024-03-11T21:31:58.249 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:31:58.249 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:31:58.249 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "done", 2024-03-11T21:31:58.249 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:29:09.003119+0000", 2024-03-11T21:31:58.249 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 4.4248000000000001e-05 2024-03-11T21:31:58.249 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-03-11T21:31:58.249 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-03-11T21:31:58.249 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-03-11T21:31:58.249 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:31:58.249 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:31:58.249 INFO:teuthology.orchestra.run.smithi092.stdout: "description": "osd_op(client.34715.0:133972 7.e 7:7ab07efd:::benchmark_data_smithi092_76040_object12061:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e123)", 2024-03-11T21:31:58.250 INFO:teuthology.orchestra.run.smithi092.stdout: "initiated_at": "2024-03-11T21:29:28.264725+0000", 2024-03-11T21:31:58.250 INFO:teuthology.orchestra.run.smithi092.stdout: "age": 149.92582363700001, 2024-03-11T21:31:58.250 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.104608654, 2024-03-11T21:31:58.250 INFO:teuthology.orchestra.run.smithi092.stdout: "type_data": { 2024-03-11T21:31:58.250 INFO:teuthology.orchestra.run.smithi092.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-11T21:31:58.250 INFO:teuthology.orchestra.run.smithi092.stdout: "client_info": { 2024-03-11T21:31:58.250 INFO:teuthology.orchestra.run.smithi092.stdout: "client": "client.34715", 2024-03-11T21:31:58.250 INFO:teuthology.orchestra.run.smithi092.stdout: "client_addr": "v1:172.21.15.92:0/2403132573", 2024-03-11T21:31:58.250 INFO:teuthology.orchestra.run.smithi092.stdout: "tid": 133972 2024-03-11T21:31:58.250 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:31:58.250 INFO:teuthology.orchestra.run.smithi092.stdout: "events": [ 2024-03-11T21:31:58.250 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:31:58.250 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "initiated", 2024-03-11T21:31:58.250 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:29:28.264725+0000", 2024-03-11T21:31:58.251 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-03-11T21:31:58.251 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:31:58.251 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:31:58.251 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "header_read", 2024-03-11T21:31:58.251 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:29:28.264725+0000", 2024-03-11T21:31:58.251 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-03-11T21:31:58.251 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:31:58.251 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:31:58.251 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "throttled", 2024-03-11T21:31:58.251 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:29:28.264728+0000", 2024-03-11T21:31:58.251 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 2.7300000000000001e-06 2024-03-11T21:31:58.251 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:31:58.251 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:31:58.251 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "all_read", 2024-03-11T21:31:58.252 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:29:28.264735+0000", 2024-03-11T21:31:58.252 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 7.4930000000000001e-06 2024-03-11T21:31:58.252 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:31:58.252 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:31:58.252 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "dispatched", 2024-03-11T21:31:58.252 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:29:28.310340+0000", 2024-03-11T21:31:58.252 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.045604599000000003 2024-03-11T21:31:58.252 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:31:58.252 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:31:58.252 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "queued_for_pg", 2024-03-11T21:31:58.252 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:29:28.310368+0000", 2024-03-11T21:31:58.252 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 2.811e-05 2024-03-11T21:31:58.252 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:31:58.252 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:31:58.252 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-03-11T21:31:58.253 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:29:28.310454+0000", 2024-03-11T21:31:58.253 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 8.5987000000000002e-05 2024-03-11T21:31:58.253 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:31:58.253 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:31:58.253 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "started", 2024-03-11T21:31:58.253 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:29:28.310555+0000", 2024-03-11T21:31:58.253 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.000100782 2024-03-11T21:31:58.253 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:31:58.253 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:31:58.253 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "waiting for subops from 0,5", 2024-03-11T21:31:58.253 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:29:28.310638+0000", 2024-03-11T21:31:58.253 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 8.2805000000000004e-05 2024-03-11T21:31:58.253 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:31:58.253 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:31:58.253 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "op_commit", 2024-03-11T21:31:58.254 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:29:28.313111+0000", 2024-03-11T21:31:58.254 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.0024739150000000001 2024-03-11T21:31:58.254 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:31:58.254 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:31:58.254 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "sub_op_commit_rec", 2024-03-11T21:31:58.254 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:29:28.313382+0000", 2024-03-11T21:31:58.254 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.000270492 2024-03-11T21:31:58.254 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:31:58.254 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:31:58.254 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "sub_op_commit_rec", 2024-03-11T21:31:58.254 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:29:28.369094+0000", 2024-03-11T21:31:58.254 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.055711954000000001 2024-03-11T21:31:58.254 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:31:58.254 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:31:58.254 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "commit_sent", 2024-03-11T21:31:58.255 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:29:28.369144+0000", 2024-03-11T21:31:58.255 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 4.9709000000000001e-05 2024-03-11T21:31:58.255 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:31:58.255 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:31:58.255 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "done", 2024-03-11T21:31:58.255 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:29:28.369334+0000", 2024-03-11T21:31:58.255 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00019007799999999999 2024-03-11T21:31:58.255 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-03-11T21:31:58.255 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-03-11T21:31:58.255 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-03-11T21:31:58.255 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:31:58.255 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:31:58.255 INFO:teuthology.orchestra.run.smithi092.stdout: "description": "osd_op(client.34715.0:136172 7.e 7:7d0bc28f:::benchmark_data_smithi092_76040_object14261:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e124)", 2024-03-11T21:31:58.255 INFO:teuthology.orchestra.run.smithi092.stdout: "initiated_at": "2024-03-11T21:29:30.561600+0000", 2024-03-11T21:31:58.255 INFO:teuthology.orchestra.run.smithi092.stdout: "age": 147.62894857500001, 2024-03-11T21:31:58.256 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.079707081999999999, 2024-03-11T21:31:58.256 INFO:teuthology.orchestra.run.smithi092.stdout: "type_data": { 2024-03-11T21:31:58.256 INFO:teuthology.orchestra.run.smithi092.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-11T21:31:58.256 INFO:teuthology.orchestra.run.smithi092.stdout: "client_info": { 2024-03-11T21:31:58.256 INFO:teuthology.orchestra.run.smithi092.stdout: "client": "client.34715", 2024-03-11T21:31:58.256 INFO:teuthology.orchestra.run.smithi092.stdout: "client_addr": "v1:172.21.15.92:0/2403132573", 2024-03-11T21:31:58.256 INFO:teuthology.orchestra.run.smithi092.stdout: "tid": 136172 2024-03-11T21:31:58.256 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:31:58.256 INFO:teuthology.orchestra.run.smithi092.stdout: "events": [ 2024-03-11T21:31:58.256 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:31:58.256 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "initiated", 2024-03-11T21:31:58.256 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:29:30.561600+0000", 2024-03-11T21:31:58.256 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-03-11T21:31:58.256 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:31:58.257 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:31:58.257 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "header_read", 2024-03-11T21:31:58.257 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:29:30.561600+0000", 2024-03-11T21:31:58.257 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-03-11T21:31:58.257 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:31:58.257 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:31:58.257 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "throttled", 2024-03-11T21:31:58.257 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:29:30.561601+0000", 2024-03-11T21:31:58.257 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 8.2099999999999995e-07 2024-03-11T21:31:58.257 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:31:58.257 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:31:58.257 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "all_read", 2024-03-11T21:31:58.257 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:29:30.561605+0000", 2024-03-11T21:31:58.257 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 3.8460000000000001e-06 2024-03-11T21:31:58.257 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:31:58.258 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:31:58.258 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "dispatched", 2024-03-11T21:31:58.258 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:29:30.561612+0000", 2024-03-11T21:31:58.258 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 6.866e-06 2024-03-11T21:31:58.258 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:31:58.258 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:31:58.258 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "queued_for_pg", 2024-03-11T21:31:58.258 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:29:30.561618+0000", 2024-03-11T21:31:58.258 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 6.2389999999999999e-06 2024-03-11T21:31:58.258 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:31:58.258 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:31:58.258 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-03-11T21:31:58.258 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:29:30.567506+0000", 2024-03-11T21:31:58.258 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.005887799 2024-03-11T21:31:58.258 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:31:58.259 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:31:58.259 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "started", 2024-03-11T21:31:58.259 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:29:30.567608+0000", 2024-03-11T21:31:58.259 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.000102569 2024-03-11T21:31:58.259 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:31:58.259 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:31:58.259 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "waiting for subops from 0,5", 2024-03-11T21:31:58.259 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:29:30.567711+0000", 2024-03-11T21:31:58.259 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.000102463 2024-03-11T21:31:58.259 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:31:58.259 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:31:58.259 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "sub_op_commit_rec", 2024-03-11T21:31:58.259 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:29:30.572528+0000", 2024-03-11T21:31:58.259 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.0048171250000000002 2024-03-11T21:31:58.260 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:31:58.260 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:31:58.260 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "op_commit", 2024-03-11T21:31:58.260 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:29:30.572538+0000", 2024-03-11T21:31:58.260 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 9.9550000000000004e-06 2024-03-11T21:31:58.260 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:31:58.260 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:31:58.260 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "sub_op_commit_rec", 2024-03-11T21:31:58.260 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:29:30.641214+0000", 2024-03-11T21:31:58.260 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.068675833000000006 2024-03-11T21:31:58.260 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:31:58.260 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:31:58.260 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "commit_sent", 2024-03-11T21:31:58.260 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:29:30.641265+0000", 2024-03-11T21:31:58.260 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 5.0964999999999997e-05 2024-03-11T21:31:58.261 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:31:58.261 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:31:58.261 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "done", 2024-03-11T21:31:58.261 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:29:30.641307+0000", 2024-03-11T21:31:58.261 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 4.2601000000000001e-05 2024-03-11T21:31:58.261 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-03-11T21:31:58.261 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-03-11T21:31:58.261 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-03-11T21:31:58.261 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:31:58.261 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:31:58.261 INFO:teuthology.orchestra.run.smithi092.stdout: "description": "osd_op(client.34715.0:142044 7.e 7:798f10b5:::benchmark_data_smithi092_76040_object20133:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e124)", 2024-03-11T21:31:58.261 INFO:teuthology.orchestra.run.smithi092.stdout: "initiated_at": "2024-03-11T21:29:37.013602+0000", 2024-03-11T21:31:58.261 INFO:teuthology.orchestra.run.smithi092.stdout: "age": 141.17694652599999, 2024-03-11T21:31:58.261 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.084341611999999996, 2024-03-11T21:31:58.262 INFO:teuthology.orchestra.run.smithi092.stdout: "type_data": { 2024-03-11T21:31:58.262 INFO:teuthology.orchestra.run.smithi092.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-11T21:31:58.262 INFO:teuthology.orchestra.run.smithi092.stdout: "client_info": { 2024-03-11T21:31:58.262 INFO:teuthology.orchestra.run.smithi092.stdout: "client": "client.34715", 2024-03-11T21:31:58.262 INFO:teuthology.orchestra.run.smithi092.stdout: "client_addr": "v1:172.21.15.92:0/2403132573", 2024-03-11T21:31:58.262 INFO:teuthology.orchestra.run.smithi092.stdout: "tid": 142044 2024-03-11T21:31:58.262 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:31:58.262 INFO:teuthology.orchestra.run.smithi092.stdout: "events": [ 2024-03-11T21:31:58.262 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:31:58.262 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "initiated", 2024-03-11T21:31:58.262 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:29:37.013602+0000", 2024-03-11T21:31:58.262 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-03-11T21:31:58.262 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:31:58.262 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:31:58.262 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "header_read", 2024-03-11T21:31:58.263 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:29:37.013602+0000", 2024-03-11T21:31:58.263 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-03-11T21:31:58.263 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:31:58.263 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:31:58.263 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "throttled", 2024-03-11T21:31:58.263 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:29:37.013604+0000", 2024-03-11T21:31:58.263 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 2.3099999999999999e-06 2024-03-11T21:31:58.263 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:31:58.263 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:31:58.263 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "all_read", 2024-03-11T21:31:58.263 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:29:37.013611+0000", 2024-03-11T21:31:58.263 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 7.0500000000000003e-06 2024-03-11T21:31:58.263 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:31:58.263 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:31:58.263 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "dispatched", 2024-03-11T21:31:58.264 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:29:37.013624+0000", 2024-03-11T21:31:58.264 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.2858e-05 2024-03-11T21:31:58.264 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:31:58.264 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:31:58.264 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "queued_for_pg", 2024-03-11T21:31:58.264 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:29:37.013632+0000", 2024-03-11T21:31:58.264 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 7.9689999999999994e-06 2024-03-11T21:31:58.264 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:31:58.264 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:31:58.264 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-03-11T21:31:58.264 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:29:37.013690+0000", 2024-03-11T21:31:58.264 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 5.7472000000000001e-05 2024-03-11T21:31:58.264 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:31:58.264 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:31:58.264 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "started", 2024-03-11T21:31:58.265 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:29:37.013804+0000", 2024-03-11T21:31:58.265 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.000114225 2024-03-11T21:31:58.265 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:31:58.265 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:31:58.265 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "waiting for subops from 0,5", 2024-03-11T21:31:58.265 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:29:37.013905+0000", 2024-03-11T21:31:58.265 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.000101056 2024-03-11T21:31:58.265 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:31:58.265 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:31:58.265 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "op_commit", 2024-03-11T21:31:58.265 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:29:37.017257+0000", 2024-03-11T21:31:58.265 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.0033514679999999998 2024-03-11T21:31:58.265 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:31:58.265 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:31:58.266 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "sub_op_commit_rec", 2024-03-11T21:31:58.266 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:29:37.019214+0000", 2024-03-11T21:31:58.266 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.001957828 2024-03-11T21:31:58.266 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:31:58.266 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:31:58.266 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "sub_op_commit_rec", 2024-03-11T21:31:58.266 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:29:37.097839+0000", 2024-03-11T21:31:58.266 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.078624227000000005 2024-03-11T21:31:58.266 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:31:58.266 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:31:58.266 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "commit_sent", 2024-03-11T21:31:58.266 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:29:37.097895+0000", 2024-03-11T21:31:58.266 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 5.6149000000000001e-05 2024-03-11T21:31:58.266 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:31:58.266 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:31:58.267 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "done", 2024-03-11T21:31:58.267 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:29:37.097944+0000", 2024-03-11T21:31:58.267 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 4.8999999999999998e-05 2024-03-11T21:31:58.267 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-03-11T21:31:58.267 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-03-11T21:31:58.267 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-03-11T21:31:58.267 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:31:58.267 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:31:58.267 INFO:teuthology.orchestra.run.smithi092.stdout: "description": "osd_op(client.34715.0:142052 7.e 7:76608073:::benchmark_data_smithi092_76040_object20141:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e124)", 2024-03-11T21:31:58.267 INFO:teuthology.orchestra.run.smithi092.stdout: "initiated_at": "2024-03-11T21:29:37.019638+0000", 2024-03-11T21:31:58.267 INFO:teuthology.orchestra.run.smithi092.stdout: "age": 141.17091076299999, 2024-03-11T21:31:58.267 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.081651325999999996, 2024-03-11T21:31:58.267 INFO:teuthology.orchestra.run.smithi092.stdout: "type_data": { 2024-03-11T21:31:58.267 INFO:teuthology.orchestra.run.smithi092.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-11T21:31:58.267 INFO:teuthology.orchestra.run.smithi092.stdout: "client_info": { 2024-03-11T21:31:58.268 INFO:teuthology.orchestra.run.smithi092.stdout: "client": "client.34715", 2024-03-11T21:31:58.268 INFO:teuthology.orchestra.run.smithi092.stdout: "client_addr": "v1:172.21.15.92:0/2403132573", 2024-03-11T21:31:58.268 INFO:teuthology.orchestra.run.smithi092.stdout: "tid": 142052 2024-03-11T21:31:58.268 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:31:58.268 INFO:teuthology.orchestra.run.smithi092.stdout: "events": [ 2024-03-11T21:31:58.268 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:31:58.268 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "initiated", 2024-03-11T21:31:58.268 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:29:37.019638+0000", 2024-03-11T21:31:58.268 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-03-11T21:31:58.268 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:31:58.268 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:31:58.268 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "header_read", 2024-03-11T21:31:58.268 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:29:37.019638+0000", 2024-03-11T21:31:58.268 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-03-11T21:31:58.269 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:31:58.269 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:31:58.269 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "throttled", 2024-03-11T21:31:58.269 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:29:37.019640+0000", 2024-03-11T21:31:58.269 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 2.0310000000000001e-06 2024-03-11T21:31:58.269 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:31:58.269 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:31:58.269 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "all_read", 2024-03-11T21:31:58.269 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:29:37.019646+0000", 2024-03-11T21:31:58.269 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 5.728e-06 2024-03-11T21:31:58.269 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:31:58.269 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:31:58.269 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "dispatched", 2024-03-11T21:31:58.269 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:29:37.019656+0000", 2024-03-11T21:31:58.270 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.0849000000000001e-05 2024-03-11T21:31:58.270 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:31:58.270 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:31:58.270 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "queued_for_pg", 2024-03-11T21:31:58.270 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:29:37.019665+0000", 2024-03-11T21:31:58.270 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 8.3520000000000007e-06 2024-03-11T21:31:58.270 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:31:58.270 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:31:58.270 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-03-11T21:31:58.270 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:29:37.019713+0000", 2024-03-11T21:31:58.270 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 4.8458000000000003e-05 2024-03-11T21:31:58.270 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:31:58.270 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:31:58.270 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "started", 2024-03-11T21:31:58.271 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:29:37.019822+0000", 2024-03-11T21:31:58.271 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00010881499999999999 2024-03-11T21:31:58.271 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:31:58.271 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:31:58.271 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "waiting for subops from 0,5", 2024-03-11T21:31:58.271 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:29:37.019928+0000", 2024-03-11T21:31:58.271 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.000105827 2024-03-11T21:31:58.271 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:31:58.271 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:31:58.271 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "op_commit", 2024-03-11T21:31:58.271 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:29:37.023307+0000", 2024-03-11T21:31:58.271 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.003379427 2024-03-11T21:31:58.271 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:31:58.271 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:31:58.271 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "sub_op_commit_rec", 2024-03-11T21:31:58.272 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:29:37.023535+0000", 2024-03-11T21:31:58.272 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00022795399999999999 2024-03-11T21:31:58.272 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:31:58.272 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:31:58.272 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "sub_op_commit_rec", 2024-03-11T21:31:58.272 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:29:37.101180+0000", 2024-03-11T21:31:58.272 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.077644562 2024-03-11T21:31:58.272 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:31:58.272 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:31:58.272 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "commit_sent", 2024-03-11T21:31:58.272 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:29:37.101242+0000", 2024-03-11T21:31:58.272 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 6.1902999999999993e-05 2024-03-11T21:31:58.272 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:31:58.272 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:31:58.272 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "done", 2024-03-11T21:31:58.273 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:29:37.101289+0000", 2024-03-11T21:31:58.273 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 4.7420000000000003e-05 2024-03-11T21:31:58.273 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-03-11T21:31:58.273 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-03-11T21:31:58.273 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-03-11T21:31:58.274 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:31:58.274 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:31:58.274 INFO:teuthology.orchestra.run.smithi092.stdout: "description": "osd_op(client.34715.0:157116 7.e 7:789e4b33:::benchmark_data_smithi092_76040_object35205:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e127)", 2024-03-11T21:31:58.274 INFO:teuthology.orchestra.run.smithi092.stdout: "initiated_at": "2024-03-11T21:29:58.112234+0000", 2024-03-11T21:31:58.274 INFO:teuthology.orchestra.run.smithi092.stdout: "age": 120.078314155, 2024-03-11T21:31:58.274 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.078904863000000006, 2024-03-11T21:31:58.274 INFO:teuthology.orchestra.run.smithi092.stdout: "type_data": { 2024-03-11T21:31:58.274 INFO:teuthology.orchestra.run.smithi092.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-11T21:31:58.274 INFO:teuthology.orchestra.run.smithi092.stdout: "client_info": { 2024-03-11T21:31:58.275 INFO:teuthology.orchestra.run.smithi092.stdout: "client": "client.34715", 2024-03-11T21:31:58.275 INFO:teuthology.orchestra.run.smithi092.stdout: "client_addr": "v1:172.21.15.92:0/2403132573", 2024-03-11T21:31:58.275 INFO:teuthology.orchestra.run.smithi092.stdout: "tid": 157116 2024-03-11T21:31:58.275 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:31:58.275 INFO:teuthology.orchestra.run.smithi092.stdout: "events": [ 2024-03-11T21:31:58.275 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:31:58.275 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "initiated", 2024-03-11T21:31:58.275 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:29:58.112234+0000", 2024-03-11T21:31:58.275 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-03-11T21:31:58.275 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:31:58.275 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:31:58.275 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "header_read", 2024-03-11T21:31:58.275 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:29:58.112234+0000", 2024-03-11T21:31:58.276 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-03-11T21:31:58.276 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:31:58.276 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:31:58.276 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "throttled", 2024-03-11T21:31:58.276 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:29:58.112237+0000", 2024-03-11T21:31:58.276 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 2.2170000000000001e-06 2024-03-11T21:31:58.276 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:31:58.276 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:31:58.276 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "all_read", 2024-03-11T21:31:58.276 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:29:58.112243+0000", 2024-03-11T21:31:58.276 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 6.579e-06 2024-03-11T21:31:58.276 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:31:58.277 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:31:58.277 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "dispatched", 2024-03-11T21:31:58.277 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:29:58.112254+0000", 2024-03-11T21:31:58.277 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.0652000000000001e-05 2024-03-11T21:31:58.277 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:31:58.277 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:31:58.277 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "queued_for_pg", 2024-03-11T21:31:58.277 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:29:58.112262+0000", 2024-03-11T21:31:58.277 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 8.2579999999999995e-06 2024-03-11T21:31:58.277 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:31:58.277 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:31:58.277 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-03-11T21:31:58.277 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:29:58.117782+0000", 2024-03-11T21:31:58.278 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.0055199530000000002 2024-03-11T21:31:58.278 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:31:58.278 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:31:58.278 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "started", 2024-03-11T21:31:58.278 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:29:58.117924+0000", 2024-03-11T21:31:58.278 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00014177100000000001 2024-03-11T21:31:58.278 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:31:58.278 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:31:58.278 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "waiting for subops from 0,5", 2024-03-11T21:31:58.278 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:29:58.118051+0000", 2024-03-11T21:31:58.278 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00012745899999999999 2024-03-11T21:31:58.279 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:31:58.279 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:31:58.279 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "op_commit", 2024-03-11T21:31:58.279 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:29:58.136345+0000", 2024-03-11T21:31:58.279 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.018293489999999999 2024-03-11T21:31:58.279 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:31:58.279 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:31:58.279 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "sub_op_commit_rec", 2024-03-11T21:31:58.279 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:29:58.141436+0000", 2024-03-11T21:31:58.279 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.0050912589999999999 2024-03-11T21:31:58.279 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:31:58.279 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:31:58.279 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "sub_op_commit_rec", 2024-03-11T21:31:58.280 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:29:58.190865+0000", 2024-03-11T21:31:58.280 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.049428699 2024-03-11T21:31:58.280 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:31:58.280 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:31:58.280 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "commit_sent", 2024-03-11T21:31:58.280 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:29:58.191088+0000", 2024-03-11T21:31:58.280 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00022311799999999999 2024-03-11T21:31:58.280 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:31:58.280 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:31:58.280 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "done", 2024-03-11T21:31:58.280 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:29:58.191139+0000", 2024-03-11T21:31:58.280 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 5.1408e-05 2024-03-11T21:31:58.280 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-03-11T21:31:58.281 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-03-11T21:31:58.281 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-03-11T21:31:58.281 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:31:58.281 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:31:58.281 INFO:teuthology.orchestra.run.smithi092.stdout: "description": "osd_op(client.34715.0:157136 7.e 7:745b6815:::benchmark_data_smithi092_76040_object35225:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e127)", 2024-03-11T21:31:58.281 INFO:teuthology.orchestra.run.smithi092.stdout: "initiated_at": "2024-03-11T21:29:58.183594+0000", 2024-03-11T21:31:58.281 INFO:teuthology.orchestra.run.smithi092.stdout: "age": 120.00695514500001, 2024-03-11T21:31:58.281 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.095394920999999994, 2024-03-11T21:31:58.281 INFO:teuthology.orchestra.run.smithi092.stdout: "type_data": { 2024-03-11T21:31:58.281 INFO:teuthology.orchestra.run.smithi092.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-11T21:31:58.281 INFO:teuthology.orchestra.run.smithi092.stdout: "client_info": { 2024-03-11T21:31:58.281 INFO:teuthology.orchestra.run.smithi092.stdout: "client": "client.34715", 2024-03-11T21:31:58.281 INFO:teuthology.orchestra.run.smithi092.stdout: "client_addr": "v1:172.21.15.92:0/2403132573", 2024-03-11T21:31:58.282 INFO:teuthology.orchestra.run.smithi092.stdout: "tid": 157136 2024-03-11T21:31:58.282 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:31:58.282 INFO:teuthology.orchestra.run.smithi092.stdout: "events": [ 2024-03-11T21:31:58.282 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:31:58.282 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "initiated", 2024-03-11T21:31:58.282 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:29:58.183594+0000", 2024-03-11T21:31:58.282 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-03-11T21:31:58.282 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:31:58.282 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:31:58.282 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "header_read", 2024-03-11T21:31:58.282 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:29:58.183594+0000", 2024-03-11T21:31:58.283 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-03-11T21:31:58.283 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:31:58.283 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:31:58.283 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "throttled", 2024-03-11T21:31:58.283 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:29:58.183596+0000", 2024-03-11T21:31:58.283 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 2.4150000000000002e-06 2024-03-11T21:31:58.283 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:31:58.283 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:31:58.283 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "all_read", 2024-03-11T21:31:58.283 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:29:58.183604+0000", 2024-03-11T21:31:58.283 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 8.1950000000000003e-06 2024-03-11T21:31:58.283 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:31:58.283 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:31:58.284 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "dispatched", 2024-03-11T21:31:58.284 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:29:58.183622+0000", 2024-03-11T21:31:58.284 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.7969e-05 2024-03-11T21:31:58.284 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:31:58.284 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:31:58.284 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "queued_for_pg", 2024-03-11T21:31:58.284 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:29:58.183632+0000", 2024-03-11T21:31:58.284 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 9.7499999999999998e-06 2024-03-11T21:31:58.284 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:31:58.284 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:31:58.284 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-03-11T21:31:58.285 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:29:58.183813+0000", 2024-03-11T21:31:58.285 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00018102099999999999 2024-03-11T21:31:58.285 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:31:58.285 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:31:58.285 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "started", 2024-03-11T21:31:58.285 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:29:58.184140+0000", 2024-03-11T21:31:58.285 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00032760200000000003 2024-03-11T21:31:58.285 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:31:58.285 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:31:58.285 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "waiting for subops from 0,5", 2024-03-11T21:31:58.285 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:29:58.184404+0000", 2024-03-11T21:31:58.285 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00026328300000000001 2024-03-11T21:31:58.285 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:31:58.286 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:31:58.286 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "op_commit", 2024-03-11T21:31:58.286 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:29:58.191221+0000", 2024-03-11T21:31:58.286 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.0068170799999999997 2024-03-11T21:31:58.286 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:31:58.286 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:31:58.286 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "sub_op_commit_rec", 2024-03-11T21:31:58.286 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:29:58.191474+0000", 2024-03-11T21:31:58.286 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00025292199999999999 2024-03-11T21:31:58.286 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:31:58.286 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:31:58.286 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "sub_op_commit_rec", 2024-03-11T21:31:58.287 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:29:58.278663+0000", 2024-03-11T21:31:58.287 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.087188838000000005 2024-03-11T21:31:58.287 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:31:58.287 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:31:58.287 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "commit_sent", 2024-03-11T21:31:58.287 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:29:58.278856+0000", 2024-03-11T21:31:58.287 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00019299399999999999 2024-03-11T21:31:58.287 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:31:58.287 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:31:58.287 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "done", 2024-03-11T21:31:58.287 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:29:58.278988+0000", 2024-03-11T21:31:58.287 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00013285200000000001 2024-03-11T21:31:58.287 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-03-11T21:31:58.288 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-03-11T21:31:58.288 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-03-11T21:31:58.288 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:31:58.288 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:31:58.288 INFO:teuthology.orchestra.run.smithi092.stdout: "description": "osd_op(client.34715.0:176635 7.8 7:115faf47:::benchmark_data_smithi092_76040_object54724:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e133)", 2024-03-11T21:31:58.288 INFO:teuthology.orchestra.run.smithi092.stdout: "initiated_at": "2024-03-11T21:30:17.308381+0000", 2024-03-11T21:31:58.288 INFO:teuthology.orchestra.run.smithi092.stdout: "age": 100.882167834, 2024-03-11T21:31:58.288 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.080650902999999996, 2024-03-11T21:31:58.288 INFO:teuthology.orchestra.run.smithi092.stdout: "type_data": { 2024-03-11T21:31:58.288 INFO:teuthology.orchestra.run.smithi092.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-11T21:31:58.288 INFO:teuthology.orchestra.run.smithi092.stdout: "client_info": { 2024-03-11T21:31:58.288 INFO:teuthology.orchestra.run.smithi092.stdout: "client": "client.34715", 2024-03-11T21:31:58.289 INFO:teuthology.orchestra.run.smithi092.stdout: "client_addr": "v1:172.21.15.92:0/2403132573", 2024-03-11T21:31:58.289 INFO:teuthology.orchestra.run.smithi092.stdout: "tid": 176635 2024-03-11T21:31:58.289 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:31:58.289 INFO:teuthology.orchestra.run.smithi092.stdout: "events": [ 2024-03-11T21:31:58.289 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:31:58.289 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "initiated", 2024-03-11T21:31:58.289 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:30:17.308381+0000", 2024-03-11T21:31:58.289 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-03-11T21:31:58.289 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:31:58.289 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:31:58.289 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "header_read", 2024-03-11T21:31:58.289 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:30:17.308381+0000", 2024-03-11T21:31:58.290 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-03-11T21:31:58.290 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:31:58.290 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:31:58.290 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "throttled", 2024-03-11T21:31:58.290 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:30:17.308383+0000", 2024-03-11T21:31:58.290 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 2.052e-06 2024-03-11T21:31:58.290 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:31:58.290 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:31:58.290 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "all_read", 2024-03-11T21:31:58.290 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:30:17.308388+0000", 2024-03-11T21:31:58.290 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 5.524e-06 2024-03-11T21:31:58.290 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:31:58.291 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:31:58.291 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "dispatched", 2024-03-11T21:31:58.291 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:30:17.308400+0000", 2024-03-11T21:31:58.291 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.1212999999999999e-05 2024-03-11T21:31:58.291 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:31:58.291 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:31:58.291 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "queued_for_pg", 2024-03-11T21:31:58.291 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:30:17.308408+0000", 2024-03-11T21:31:58.291 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 8.4689999999999997e-06 2024-03-11T21:31:58.291 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:31:58.291 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:31:58.291 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-03-11T21:31:58.292 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:30:17.308471+0000", 2024-03-11T21:31:58.292 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 6.2539999999999994e-05 2024-03-11T21:31:58.292 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:31:58.292 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:31:58.292 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "op must wait for map", 2024-03-11T21:31:58.292 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:30:17.308476+0000", 2024-03-11T21:31:58.292 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 5.6450000000000002e-06 2024-03-11T21:31:58.292 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:31:58.292 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:31:58.292 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-03-11T21:31:58.292 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:30:17.310072+0000", 2024-03-11T21:31:58.292 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.001595909 2024-03-11T21:31:58.293 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:31:58.293 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:31:58.293 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "started", 2024-03-11T21:31:58.293 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:30:17.310249+0000", 2024-03-11T21:31:58.293 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.000176897 2024-03-11T21:31:58.293 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:31:58.293 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:31:58.293 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "waiting for subops from 4,7", 2024-03-11T21:31:58.293 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:30:17.310357+0000", 2024-03-11T21:31:58.293 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00010811600000000001 2024-03-11T21:31:58.293 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:31:58.293 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:31:58.294 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "sub_op_commit_rec", 2024-03-11T21:31:58.294 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:30:17.314612+0000", 2024-03-11T21:31:58.294 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.0042548430000000003 2024-03-11T21:31:58.294 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:31:58.294 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:31:58.294 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "op_commit", 2024-03-11T21:31:58.294 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:30:17.315171+0000", 2024-03-11T21:31:58.294 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00055859799999999999 2024-03-11T21:31:58.294 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:31:58.294 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:31:58.294 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "sub_op_commit_rec", 2024-03-11T21:31:58.294 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:30:17.388870+0000", 2024-03-11T21:31:58.294 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.073699205000000004 2024-03-11T21:31:58.295 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:31:58.295 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:31:58.295 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "commit_sent", 2024-03-11T21:31:58.295 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:30:17.388974+0000", 2024-03-11T21:31:58.295 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.000104151 2024-03-11T21:31:58.295 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:31:58.295 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:31:58.295 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "done", 2024-03-11T21:31:58.295 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:30:17.389032+0000", 2024-03-11T21:31:58.295 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 5.7741000000000002e-05 2024-03-11T21:31:58.295 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-03-11T21:31:58.295 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-03-11T21:31:58.296 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-03-11T21:31:58.296 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-03-11T21:31:58.296 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-03-11T21:31:58.296 INFO:teuthology.orchestra.run.smithi092.stdout:} 2024-03-11T21:31:58.297 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:31:58.297 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:31:58.298 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:31:58.398 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:31:58.422 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:31:58.423 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:31:58.423 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:31:58.497 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:31:58 smithi092 ceph-mon[29572]: 5.1a scrub starts 2024-03-11T21:31:58.497 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:31:58 smithi092 ceph-mon[29572]: 5.1a scrub ok 2024-03-11T21:31:58.497 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:31:58 smithi092 ceph-mon[29572]: pgmap v244: 148 pgs: 6 creating+peering, 142 active+clean; 582 KiB data, 981 MiB used, 714 GiB / 715 GiB avail; 873 B/s rd, 0 op/s; 99 KiB/s, 11 objects/s recovering 2024-03-11T21:31:58.497 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:31:58 smithi092 ceph-mon[29572]: 2.1d scrub starts 2024-03-11T21:31:58.497 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:31:58 smithi092 ceph-mon[29572]: 2.1d scrub ok 2024-03-11T21:31:58.498 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:31:58 smithi092 ceph-mon[29572]: 5.5 scrub starts 2024-03-11T21:31:58.498 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:31:58 smithi092 ceph-mon[29572]: 5.5 scrub ok 2024-03-11T21:31:58.498 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:31:58 smithi092 ceph-mon[29572]: 4.1b scrub starts 2024-03-11T21:31:58.498 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:31:58 smithi092 ceph-mon[29572]: 4.1b scrub ok 2024-03-11T21:31:58.498 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:31:58 smithi092 ceph-mon[37563]: 5.1a scrub starts 2024-03-11T21:31:58.498 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:31:58 smithi092 ceph-mon[37563]: 5.1a scrub ok 2024-03-11T21:31:58.498 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:31:58 smithi092 ceph-mon[37563]: pgmap v244: 148 pgs: 6 creating+peering, 142 active+clean; 582 KiB data, 981 MiB used, 714 GiB / 715 GiB avail; 873 B/s rd, 0 op/s; 99 KiB/s, 11 objects/s recovering 2024-03-11T21:31:58.498 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:31:58 smithi092 ceph-mon[37563]: 2.1d scrub starts 2024-03-11T21:31:58.498 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:31:58 smithi092 ceph-mon[37563]: 2.1d scrub ok 2024-03-11T21:31:58.498 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:31:58 smithi092 ceph-mon[37563]: 5.5 scrub starts 2024-03-11T21:31:58.499 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:31:58 smithi092 ceph-mon[37563]: 5.5 scrub ok 2024-03-11T21:31:58.499 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:31:58 smithi092 ceph-mon[37563]: 4.1b scrub starts 2024-03-11T21:31:58.499 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:31:58 smithi092 ceph-mon[37563]: 4.1b scrub ok 2024-03-11T21:31:58.523 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:31:58.546 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:31:58.547 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:31:58.548 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:31:58.576 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:31:58 smithi097 ceph-mon[33317]: 5.1a scrub starts 2024-03-11T21:31:58.576 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:31:58 smithi097 ceph-mon[33317]: 5.1a scrub ok 2024-03-11T21:31:58.576 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:31:58 smithi097 ceph-mon[33317]: pgmap v244: 148 pgs: 6 creating+peering, 142 active+clean; 582 KiB data, 981 MiB used, 714 GiB / 715 GiB avail; 873 B/s rd, 0 op/s; 99 KiB/s, 11 objects/s recovering 2024-03-11T21:31:58.576 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:31:58 smithi097 ceph-mon[33317]: 2.1d scrub starts 2024-03-11T21:31:58.576 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:31:58 smithi097 ceph-mon[33317]: 2.1d scrub ok 2024-03-11T21:31:58.576 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:31:58 smithi097 ceph-mon[33317]: 5.5 scrub starts 2024-03-11T21:31:58.577 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:31:58 smithi097 ceph-mon[33317]: 5.5 scrub ok 2024-03-11T21:31:58.577 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:31:58 smithi097 ceph-mon[33317]: 4.1b scrub starts 2024-03-11T21:31:58.577 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:31:58 smithi097 ceph-mon[33317]: 4.1b scrub ok 2024-03-11T21:31:58.648 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:31:58.671 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:31:58.672 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:31:58.672 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:31:58.773 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:31:58.799 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:31:58.800 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:31:58.800 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:31:58.901 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:31:58.925 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:31:58.926 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:31:58.927 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:31:59.028 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:31:59.051 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:31:59.052 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:31:59.052 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:31:59.153 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:31:59.179 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:31:59.180 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:31:59.180 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:31:59.253 INFO:teuthology.orchestra.run.smithi092.stderr:enabled application 'rados' on pool 'unique_pool_1' 2024-03-11T21:31:59.281 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:31:59.304 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:31:59.305 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:31:59.305 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:31:59.406 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:31:59.431 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:31:59.431 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:31:59.431 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:31:59.463 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:31:59 smithi092 ceph-mon[29572]: 6.18 scrub starts 2024-03-11T21:31:59.463 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:31:59 smithi092 ceph-mon[29572]: 6.18 scrub ok 2024-03-11T21:31:59.463 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:31:59 smithi092 ceph-mon[29572]: 2.1c scrub starts 2024-03-11T21:31:59.463 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:31:59 smithi092 ceph-mon[29572]: 2.1c scrub ok 2024-03-11T21:31:59.464 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:31:59 smithi092 ceph-mon[29572]: 6.8 scrub starts 2024-03-11T21:31:59.464 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:31:59 smithi092 ceph-mon[29572]: 6.8 scrub ok 2024-03-11T21:31:59.464 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:31:59 smithi092 ceph-mon[29572]: from='client.? v1:172.21.15.92:0/990600754' entity='client.admin' cmd=[{"prefix": "osd pool application enable", "pool": "unique_pool_1", "app": "rados", "yes_i_really_mean_it": true}]: dispatch 2024-03-11T21:31:59.464 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:31:59 smithi092 ceph-mon[29572]: from='client.35000 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable", "pool": "unique_pool_1", "app": "rados", "yes_i_really_mean_it": true}]: dispatch 2024-03-11T21:31:59.464 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:31:59 smithi092 ceph-mon[37563]: 6.18 scrub starts 2024-03-11T21:31:59.464 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:31:59 smithi092 ceph-mon[37563]: 6.18 scrub ok 2024-03-11T21:31:59.464 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:31:59 smithi092 ceph-mon[37563]: 2.1c scrub starts 2024-03-11T21:31:59.464 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:31:59 smithi092 ceph-mon[37563]: 2.1c scrub ok 2024-03-11T21:31:59.464 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:31:59 smithi092 ceph-mon[37563]: 6.8 scrub starts 2024-03-11T21:31:59.464 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:31:59 smithi092 ceph-mon[37563]: 6.8 scrub ok 2024-03-11T21:31:59.464 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:31:59 smithi092 ceph-mon[37563]: from='client.? v1:172.21.15.92:0/990600754' entity='client.admin' cmd=[{"prefix": "osd pool application enable", "pool": "unique_pool_1", "app": "rados", "yes_i_really_mean_it": true}]: dispatch 2024-03-11T21:31:59.464 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:31:59 smithi092 ceph-mon[37563]: from='client.35000 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable", "pool": "unique_pool_1", "app": "rados", "yes_i_really_mean_it": true}]: dispatch 2024-03-11T21:31:59.532 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:31:59.559 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:31:59.559 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:31:59.560 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:31:59.576 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:31:59 smithi097 ceph-mon[33317]: 6.18 scrub starts 2024-03-11T21:31:59.576 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:31:59 smithi097 ceph-mon[33317]: 6.18 scrub ok 2024-03-11T21:31:59.576 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:31:59 smithi097 ceph-mon[33317]: 2.1c scrub starts 2024-03-11T21:31:59.576 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:31:59 smithi097 ceph-mon[33317]: 2.1c scrub ok 2024-03-11T21:31:59.576 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:31:59 smithi097 ceph-mon[33317]: 6.8 scrub starts 2024-03-11T21:31:59.576 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:31:59 smithi097 ceph-mon[33317]: 6.8 scrub ok 2024-03-11T21:31:59.576 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:31:59 smithi097 ceph-mon[33317]: from='client.? v1:172.21.15.92:0/990600754' entity='client.admin' cmd=[{"prefix": "osd pool application enable", "pool": "unique_pool_1", "app": "rados", "yes_i_really_mean_it": true}]: dispatch 2024-03-11T21:31:59.577 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:31:59 smithi097 ceph-mon[33317]: from='client.35000 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable", "pool": "unique_pool_1", "app": "rados", "yes_i_really_mean_it": true}]: dispatch 2024-03-11T21:31:59.661 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:31:59.686 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:31:59.687 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:31:59.687 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:31:59.788 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:31:59.812 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:31:59.813 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:31:59.813 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:31:59.914 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:31:59.937 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:31:59.938 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:31:59.938 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:32:00.039 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:32:00.065 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:32:00.066 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:32:00.066 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:32:00.167 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:32:00.190 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:32:00.191 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:32:00.191 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:32:00.292 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:32:00.318 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:32:00.319 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:32:00.319 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:32:00.349 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:32:00 smithi092 ceph-mon[29572]: pgmap v245: 148 pgs: 148 active+clean; 582 KiB data, 981 MiB used, 714 GiB / 715 GiB avail; 746 B/s rd, 0 op/s; 84 KiB/s, 5 objects/s recovering 2024-03-11T21:32:00.349 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:32:00 smithi092 ceph-mon[29572]: from='client.35000 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable", "pool": "unique_pool_1", "app": "rados", "yes_i_really_mean_it": true}]': finished 2024-03-11T21:32:00.349 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:32:00 smithi092 ceph-mon[29572]: osdmap e161: 8 total, 8 up, 8 in 2024-03-11T21:32:00.349 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:32:00 smithi092 ceph-mon[29572]: 2.18 scrub starts 2024-03-11T21:32:00.349 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:32:00 smithi092 ceph-mon[29572]: 2.18 scrub ok 2024-03-11T21:32:00.349 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:32:00 smithi092 ceph-mon[37563]: pgmap v245: 148 pgs: 148 active+clean; 582 KiB data, 981 MiB used, 714 GiB / 715 GiB avail; 746 B/s rd, 0 op/s; 84 KiB/s, 5 objects/s recovering 2024-03-11T21:32:00.349 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:32:00 smithi092 ceph-mon[37563]: from='client.35000 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable", "pool": "unique_pool_1", "app": "rados", "yes_i_really_mean_it": true}]': finished 2024-03-11T21:32:00.349 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:32:00 smithi092 ceph-mon[37563]: osdmap e161: 8 total, 8 up, 8 in 2024-03-11T21:32:00.349 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:32:00 smithi092 ceph-mon[37563]: 2.18 scrub starts 2024-03-11T21:32:00.350 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:32:00 smithi092 ceph-mon[37563]: 2.18 scrub ok 2024-03-11T21:32:00.420 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:32:00.444 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:32:00.444 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:32:00.444 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:32:00.545 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:32:00.569 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:32:00.570 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:32:00.570 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:32:00.576 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:32:00 smithi097 ceph-mon[33317]: pgmap v245: 148 pgs: 148 active+clean; 582 KiB data, 981 MiB used, 714 GiB / 715 GiB avail; 746 B/s rd, 0 op/s; 84 KiB/s, 5 objects/s recovering 2024-03-11T21:32:00.576 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:32:00 smithi097 ceph-mon[33317]: from='client.35000 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable", "pool": "unique_pool_1", "app": "rados", "yes_i_really_mean_it": true}]': finished 2024-03-11T21:32:00.576 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:32:00 smithi097 ceph-mon[33317]: osdmap e161: 8 total, 8 up, 8 in 2024-03-11T21:32:00.576 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:32:00 smithi097 ceph-mon[33317]: 2.18 scrub starts 2024-03-11T21:32:00.576 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:32:00 smithi097 ceph-mon[33317]: 2.18 scrub ok 2024-03-11T21:32:00.671 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:32:00.695 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:32:00.696 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:32:00.696 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:32:00.797 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:32:00.825 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:32:00.826 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:32:00.826 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:32:00.927 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:32:00.955 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:32:00.955 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:32:00.956 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:32:01.057 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:32:01.081 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:32:01.082 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:32:01.082 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:32:01.183 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:32:01.211 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:32:01.212 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:32:01.212 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:32:01.313 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:32:01.337 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:32:01.337 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:32:01.338 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:32:01.360 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:32:01 smithi092 ceph-mon[37563]: 4.12 scrub starts 2024-03-11T21:32:01.360 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:32:01 smithi092 ceph-mon[37563]: 4.12 scrub ok 2024-03-11T21:32:01.361 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:32:01 smithi092 ceph-mon[37563]: 2.1a scrub starts 2024-03-11T21:32:01.361 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:32:01 smithi092 ceph-mon[37563]: 2.1a scrub ok 2024-03-11T21:32:01.361 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:32:01 smithi092 ceph-mon[37563]: 4.18 scrub starts 2024-03-11T21:32:01.361 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:32:01 smithi092 ceph-mon[37563]: 4.18 scrub ok 2024-03-11T21:32:01.361 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:32:01 smithi092 ceph-mon[29572]: 4.12 scrub starts 2024-03-11T21:32:01.361 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:32:01 smithi092 ceph-mon[29572]: 4.12 scrub ok 2024-03-11T21:32:01.362 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:32:01 smithi092 ceph-mon[29572]: 2.1a scrub starts 2024-03-11T21:32:01.362 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:32:01 smithi092 ceph-mon[29572]: 2.1a scrub ok 2024-03-11T21:32:01.362 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:32:01 smithi092 ceph-mon[29572]: 4.18 scrub starts 2024-03-11T21:32:01.362 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:32:01 smithi092 ceph-mon[29572]: 4.18 scrub ok 2024-03-11T21:32:01.375 DEBUG:teuthology.orchestra.run.smithi092:> /bin/sh -c 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados --no-log-to-stderr --name client.0 -b 65536 --object-size 65536 -p unique_pool_1 bench 90 write' 2024-03-11T21:32:01.378 INFO:tasks.radosbench:joining radosbench (timing out after 3000s) 2024-03-11T21:32:01.378 INFO:teuthology.orchestra.run:waiting for 3000 2024-03-11T21:32:01.425 INFO:tasks.radosbench.radosbench.0.smithi092.stdout:hints = 1 2024-03-11T21:32:01.426 INFO:tasks.radosbench.radosbench.0.smithi092.stdout:Maintaining 16 concurrent writes of 65536 bytes to objects of size 65536 for up to 90 seconds or 0 objects 2024-03-11T21:32:01.426 INFO:tasks.radosbench.radosbench.0.smithi092.stdout:Object prefix: benchmark_data_smithi092_104865 2024-03-11T21:32:01.426 INFO:tasks.radosbench.radosbench.0.smithi092.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-03-11T21:32:01.426 INFO:tasks.radosbench.radosbench.0.smithi092.stdout: 0 0 0 0 0 0 - 0 2024-03-11T21:32:01.438 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:32:01.464 DEBUG:teuthology.orchestra.run.smithi092:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid 00db0016-dfed-11ee-95c8-87774f69a715 -- ceph daemon osd.3 dump_ops_in_flight 2024-03-11T21:32:01.475 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:32:01.476 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:32:01.477 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:32:01.576 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:32:01 smithi097 ceph-mon[33317]: 4.12 scrub starts 2024-03-11T21:32:01.576 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:32:01 smithi097 ceph-mon[33317]: 4.12 scrub ok 2024-03-11T21:32:01.576 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:32:01 smithi097 ceph-mon[33317]: 2.1a scrub starts 2024-03-11T21:32:01.576 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:32:01 smithi097 ceph-mon[33317]: 2.1a scrub ok 2024-03-11T21:32:01.577 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:32:01 smithi097 ceph-mon[33317]: 4.18 scrub starts 2024-03-11T21:32:01.577 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:32:01 smithi097 ceph-mon[33317]: 4.18 scrub ok 2024-03-11T21:32:01.578 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:32:01.623 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:32:01.624 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:32:01.624 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:32:01.725 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:32:01.781 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:32:01.782 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:32:01.783 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:32:01.883 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:32:01.916 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:32:01.917 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:32:01.918 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:32:02.018 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:32:02.057 INFO:teuthology.orchestra.run.smithi092.stderr:Inferring config /var/lib/ceph/00db0016-dfed-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T21:32:02.072 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:32:02.073 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:32:02.073 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:32:02.121 INFO:teuthology.orchestra.run.smithi092.stderr:Inferring config /var/lib/ceph/00db0016-dfed-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T21:32:02.174 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:32:02.231 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:32:02.231 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:32:02.232 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:32:02.332 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:32:02.362 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:32:02.363 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:32:02.363 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:32:02.378 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:32:02 smithi092 ceph-mon[29572]: 2.17 scrub starts 2024-03-11T21:32:02.378 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:32:02 smithi092 ceph-mon[29572]: 2.17 scrub ok 2024-03-11T21:32:02.378 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:32:02 smithi092 ceph-mon[29572]: pgmap v247: 148 pgs: 148 active+clean; 582 KiB data, 981 MiB used, 714 GiB / 715 GiB avail; 652 B/s rd, 0 op/s 2024-03-11T21:32:02.378 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:32:02 smithi092 ceph-mon[29572]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-03-11T21:32:02.378 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:32:02 smithi092 ceph-mon[29572]: 2.3 scrub starts 2024-03-11T21:32:02.378 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:32:02 smithi092 ceph-mon[29572]: 2.3 scrub ok 2024-03-11T21:32:02.378 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:32:02 smithi092 ceph-mon[29572]: 6.1a scrub starts 2024-03-11T21:32:02.378 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:32:02 smithi092 ceph-mon[29572]: 6.1a scrub ok 2024-03-11T21:32:02.379 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:32:02 smithi092 ceph-mon[37563]: 2.17 scrub starts 2024-03-11T21:32:02.379 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:32:02 smithi092 ceph-mon[37563]: 2.17 scrub ok 2024-03-11T21:32:02.379 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:32:02 smithi092 ceph-mon[37563]: pgmap v247: 148 pgs: 148 active+clean; 582 KiB data, 981 MiB used, 714 GiB / 715 GiB avail; 652 B/s rd, 0 op/s 2024-03-11T21:32:02.379 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:32:02 smithi092 ceph-mon[37563]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-03-11T21:32:02.379 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:32:02 smithi092 ceph-mon[37563]: 2.3 scrub starts 2024-03-11T21:32:02.379 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:32:02 smithi092 ceph-mon[37563]: 2.3 scrub ok 2024-03-11T21:32:02.379 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:32:02 smithi092 ceph-mon[37563]: 6.1a scrub starts 2024-03-11T21:32:02.379 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:32:02 smithi092 ceph-mon[37563]: 6.1a scrub ok 2024-03-11T21:32:02.426 INFO:tasks.radosbench.radosbench.0.smithi092.stdout: 1 16 2885 2869 179.25 179.312 0.00447717 0.00556058 2024-03-11T21:32:02.464 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:32:02.507 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:32:02.508 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:32:02.508 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:32:02.576 INFO:journalctl@ceph.iscsi.iscsi.a.smithi097.stdout:Mar 11 21:32:02 smithi097 ceph-00db0016-dfed-11ee-95c8-87774f69a715-iscsi-iscsi-a[60943]: debug there is no tcmu-runner data available 2024-03-11T21:32:02.576 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:32:02 smithi097 ceph-mon[33317]: 2.17 scrub starts 2024-03-11T21:32:02.576 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:32:02 smithi097 ceph-mon[33317]: 2.17 scrub ok 2024-03-11T21:32:02.576 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:32:02 smithi097 ceph-mon[33317]: pgmap v247: 148 pgs: 148 active+clean; 582 KiB data, 981 MiB used, 714 GiB / 715 GiB avail; 652 B/s rd, 0 op/s 2024-03-11T21:32:02.577 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:32:02 smithi097 ceph-mon[33317]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-03-11T21:32:02.577 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:32:02 smithi097 ceph-mon[33317]: 2.3 scrub starts 2024-03-11T21:32:02.577 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:32:02 smithi097 ceph-mon[33317]: 2.3 scrub ok 2024-03-11T21:32:02.577 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:32:02 smithi097 ceph-mon[33317]: 6.1a scrub starts 2024-03-11T21:32:02.577 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:32:02 smithi097 ceph-mon[33317]: 6.1a scrub ok 2024-03-11T21:32:02.608 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:32:02.640 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:32:02.641 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:32:02.641 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:32:02.742 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:32:02.782 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:32:02.783 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:32:02.783 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:32:02.826 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5, 6, 7] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7] 2024-03-11T21:32:02.827 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-03-11T21:32:02.827 INFO:tasks.thrashosds.thrasher:fixing pg num pool unique_pool_1 2024-03-11T21:32:02.827 DEBUG:teuthology.orchestra.run.smithi092:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid 00db0016-dfed-11ee-95c8-87774f69a715 -- ceph pg dump --format=json 2024-03-11T21:32:02.884 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:32:02.917 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:32:02.918 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:32:02.919 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:32:03.019 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:32:03.064 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:32:03.065 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:32:03.065 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:32:03.166 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:32:03.196 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:32:03.197 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:32:03.197 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:32:03.298 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:32:03.303 INFO:teuthology.orchestra.run.smithi092.stderr:nodeep-scrub is unset 2024-03-11T21:32:03.341 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:32:03.341 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:32:03.342 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:32:03.426 INFO:tasks.radosbench.radosbench.0.smithi092.stdout: 2 15 5598 5583 174.431 169.625 0.00397532 0.00571827 2024-03-11T21:32:03.443 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:32:03.482 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:32:03.483 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:32:03.483 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:32:03.502 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:32:03 smithi092 ceph-mon[37563]: from='client.24469 v1:172.21.15.97:0/404208310' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:32:03.502 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:32:03 smithi092 ceph-mon[37563]: 4.f scrub starts 2024-03-11T21:32:03.502 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:32:03 smithi092 ceph-mon[37563]: 4.f scrub ok 2024-03-11T21:32:03.502 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:32:03 smithi092 ceph-mon[37563]: 2.9 scrub starts 2024-03-11T21:32:03.503 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:32:03 smithi092 ceph-mon[37563]: purged_snaps scrub starts 2024-03-11T21:32:03.503 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:32:03 smithi092 ceph-mon[37563]: purged_snaps scrub ok 2024-03-11T21:32:03.503 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:32:03 smithi092 ceph-mon[37563]: from='client.? v1:172.21.15.92:0/3862354835' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-11T21:32:03.503 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:32:03 smithi092 ceph-mon[37563]: from='client.35015 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-11T21:32:03.503 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:32:03 smithi092 ceph-mon[29572]: from='client.24469 v1:172.21.15.97:0/404208310' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:32:03.504 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:32:03 smithi092 ceph-mon[29572]: 4.f scrub starts 2024-03-11T21:32:03.504 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:32:03 smithi092 ceph-mon[29572]: 4.f scrub ok 2024-03-11T21:32:03.504 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:32:03 smithi092 ceph-mon[29572]: 2.9 scrub starts 2024-03-11T21:32:03.504 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:32:03 smithi092 ceph-mon[29572]: purged_snaps scrub starts 2024-03-11T21:32:03.504 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:32:03 smithi092 ceph-mon[29572]: purged_snaps scrub ok 2024-03-11T21:32:03.504 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:32:03 smithi092 ceph-mon[29572]: from='client.? v1:172.21.15.92:0/3862354835' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-11T21:32:03.505 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:32:03 smithi092 ceph-mon[29572]: from='client.35015 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-11T21:32:03.576 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:32:03 smithi097 ceph-mon[33317]: from='client.24469 v1:172.21.15.97:0/404208310' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:32:03.576 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:32:03 smithi097 ceph-mon[33317]: 4.f scrub starts 2024-03-11T21:32:03.576 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:32:03 smithi097 ceph-mon[33317]: 4.f scrub ok 2024-03-11T21:32:03.576 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:32:03 smithi097 ceph-mon[33317]: 2.9 scrub starts 2024-03-11T21:32:03.576 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:32:03 smithi097 ceph-mon[33317]: purged_snaps scrub starts 2024-03-11T21:32:03.576 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:32:03 smithi097 ceph-mon[33317]: purged_snaps scrub ok 2024-03-11T21:32:03.577 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:32:03 smithi097 ceph-mon[33317]: from='client.? v1:172.21.15.92:0/3862354835' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-11T21:32:03.577 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:32:03 smithi097 ceph-mon[33317]: from='client.35015 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-11T21:32:03.584 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:32:03.624 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:32:03.624 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:32:03.625 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:32:03.726 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:32:03.755 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:32:03.756 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:32:03.757 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:32:03.858 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:32:03.888 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:32:03.889 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:32:03.889 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:32:03.990 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:32:04.042 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:32:04.042 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:32:04.043 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:32:04.144 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:32:04.185 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:32:04.186 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:32:04.186 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:32:04.287 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:32:04.315 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:32:04.315 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:32:04.316 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:32:04.416 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:32:04.426 INFO:tasks.radosbench.radosbench.0.smithi092.stdout: 3 16 8343 8327 173.43 171.5 0.00430859 0.00575381 2024-03-11T21:32:04.450 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:32:04.450 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:32:04.451 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:32:04.552 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:32:04.576 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:32:04 smithi097 ceph-mon[33317]: 2.9 scrub ok 2024-03-11T21:32:04.576 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:32:04 smithi097 ceph-mon[33317]: pgmap v248: 148 pgs: 148 active+clean; 158 MiB data, 1.2 GiB used, 714 GiB / 715 GiB avail; 1.1 KiB/s rd, 18 MiB/s wr, 286 op/s 2024-03-11T21:32:04.576 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:32:04 smithi097 ceph-mon[33317]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-03-11T21:32:04.576 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:32:04 smithi097 ceph-mon[33317]: Cluster is now healthy 2024-03-11T21:32:04.577 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:32:04 smithi097 ceph-mon[33317]: from='client.35015 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-03-11T21:32:04.577 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:32:04 smithi097 ceph-mon[33317]: osdmap e162: 8 total, 8 up, 8 in 2024-03-11T21:32:04.577 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:32:04 smithi097 ceph-mon[33317]: 6.0 scrub starts 2024-03-11T21:32:04.577 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:32:04 smithi097 ceph-mon[33317]: 6.0 scrub ok 2024-03-11T21:32:04.577 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:32:04 smithi097 ceph-mon[33317]: 5.14 scrub starts 2024-03-11T21:32:04.577 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:32:04 smithi097 ceph-mon[33317]: 5.14 scrub ok 2024-03-11T21:32:04.593 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:32:04.594 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:32:04.594 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:32:04.610 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:32:04 smithi092 ceph-mon[29572]: 2.9 scrub ok 2024-03-11T21:32:04.610 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:32:04 smithi092 ceph-mon[29572]: pgmap v248: 148 pgs: 148 active+clean; 158 MiB data, 1.2 GiB used, 714 GiB / 715 GiB avail; 1.1 KiB/s rd, 18 MiB/s wr, 286 op/s 2024-03-11T21:32:04.610 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:32:04 smithi092 ceph-mon[29572]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-03-11T21:32:04.611 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:32:04 smithi092 ceph-mon[29572]: Cluster is now healthy 2024-03-11T21:32:04.611 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:32:04 smithi092 ceph-mon[29572]: from='client.35015 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-03-11T21:32:04.611 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:32:04 smithi092 ceph-mon[29572]: osdmap e162: 8 total, 8 up, 8 in 2024-03-11T21:32:04.611 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:32:04 smithi092 ceph-mon[29572]: 6.0 scrub starts 2024-03-11T21:32:04.611 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:32:04 smithi092 ceph-mon[29572]: 6.0 scrub ok 2024-03-11T21:32:04.611 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:32:04 smithi092 ceph-mon[29572]: 5.14 scrub starts 2024-03-11T21:32:04.611 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:32:04 smithi092 ceph-mon[29572]: 5.14 scrub ok 2024-03-11T21:32:04.611 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:32:04 smithi092 ceph-mon[37563]: 2.9 scrub ok 2024-03-11T21:32:04.611 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:32:04 smithi092 ceph-mon[37563]: pgmap v248: 148 pgs: 148 active+clean; 158 MiB data, 1.2 GiB used, 714 GiB / 715 GiB avail; 1.1 KiB/s rd, 18 MiB/s wr, 286 op/s 2024-03-11T21:32:04.611 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:32:04 smithi092 ceph-mon[37563]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-03-11T21:32:04.612 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:32:04 smithi092 ceph-mon[37563]: Cluster is now healthy 2024-03-11T21:32:04.612 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:32:04 smithi092 ceph-mon[37563]: from='client.35015 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-03-11T21:32:04.612 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:32:04 smithi092 ceph-mon[37563]: osdmap e162: 8 total, 8 up, 8 in 2024-03-11T21:32:04.612 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:32:04 smithi092 ceph-mon[37563]: 6.0 scrub starts 2024-03-11T21:32:04.612 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:32:04 smithi092 ceph-mon[37563]: 6.0 scrub ok 2024-03-11T21:32:04.612 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:32:04 smithi092 ceph-mon[37563]: 5.14 scrub starts 2024-03-11T21:32:04.612 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:32:04 smithi092 ceph-mon[37563]: 5.14 scrub ok 2024-03-11T21:32:04.696 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:32:04.729 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:32:04.730 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:32:04.730 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:32:04.831 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:32:04.878 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:32:04.878 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:32:04.879 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:32:04.981 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:32:05.018 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:32:05.019 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:32:05.019 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:32:05.120 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:32:05.159 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:32:05.160 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:32:05.160 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:32:05.261 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:32:05.295 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:32:05.296 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:32:05.296 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:32:05.347 INFO:teuthology.orchestra.run.smithi092.stderr:Inferring config /var/lib/ceph/00db0016-dfed-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T21:32:05.381 INFO:teuthology.orchestra.run.smithi092.stdout:osd.0: {} 2024-03-11T21:32:05.396 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:32:05.425 INFO:teuthology.orchestra.run.smithi092.stdout:osd.1: {} 2024-03-11T21:32:05.426 INFO:tasks.radosbench.radosbench.0.smithi092.stdout: 4 16 11166 11150 174.177 176.438 0.00423822 0.00573176 2024-03-11T21:32:05.428 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:32:05.429 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:32:05.430 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:32:05.466 INFO:teuthology.orchestra.run.smithi092.stdout:osd.2: {} 2024-03-11T21:32:05.508 INFO:teuthology.orchestra.run.smithi092.stdout:osd.3: {} 2024-03-11T21:32:05.509 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:32:05 smithi092 ceph-mon[29572]: purged_snaps scrub starts 2024-03-11T21:32:05.509 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:32:05 smithi092 ceph-mon[29572]: purged_snaps scrub ok 2024-03-11T21:32:05.509 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:32:05 smithi092 ceph-mon[29572]: osdmap e163: 8 total, 8 up, 8 in 2024-03-11T21:32:05.509 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:32:05 smithi092 ceph-mon[29572]: 2.4 scrub starts 2024-03-11T21:32:05.509 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:32:05 smithi092 ceph-mon[29572]: 4.1a scrub starts 2024-03-11T21:32:05.509 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:32:05 smithi092 ceph-mon[29572]: 4.1a scrub ok 2024-03-11T21:32:05.509 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:32:05 smithi092 ceph-mon[37563]: purged_snaps scrub starts 2024-03-11T21:32:05.510 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:32:05 smithi092 ceph-mon[37563]: purged_snaps scrub ok 2024-03-11T21:32:05.510 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:32:05 smithi092 ceph-mon[37563]: osdmap e163: 8 total, 8 up, 8 in 2024-03-11T21:32:05.510 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:32:05 smithi092 ceph-mon[37563]: 2.4 scrub starts 2024-03-11T21:32:05.510 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:32:05 smithi092 ceph-mon[37563]: 4.1a scrub starts 2024-03-11T21:32:05.510 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:32:05 smithi092 ceph-mon[37563]: 4.1a scrub ok 2024-03-11T21:32:05.530 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:32:05.549 INFO:teuthology.orchestra.run.smithi092.stdout:osd.4: {} 2024-03-11T21:32:05.562 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:32:05.563 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:32:05.563 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:32:05.576 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:32:05 smithi097 ceph-mon[33317]: purged_snaps scrub starts 2024-03-11T21:32:05.576 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:32:05 smithi097 ceph-mon[33317]: purged_snaps scrub ok 2024-03-11T21:32:05.576 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:32:05 smithi097 ceph-mon[33317]: osdmap e163: 8 total, 8 up, 8 in 2024-03-11T21:32:05.576 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:32:05 smithi097 ceph-mon[33317]: 2.4 scrub starts 2024-03-11T21:32:05.576 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:32:05 smithi097 ceph-mon[33317]: 4.1a scrub starts 2024-03-11T21:32:05.577 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:32:05 smithi097 ceph-mon[33317]: 4.1a scrub ok 2024-03-11T21:32:05.593 INFO:teuthology.orchestra.run.smithi092.stdout:osd.5: {} 2024-03-11T21:32:05.634 INFO:teuthology.orchestra.run.smithi092.stdout:osd.6: {} 2024-03-11T21:32:05.664 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:32:05.676 INFO:teuthology.orchestra.run.smithi092.stdout:osd.7: {} 2024-03-11T21:32:05.683 INFO:teuthology.orchestra.run.smithi092.stderr:osd.0: osd_enable_op_tracker = '' 2024-03-11T21:32:05.683 INFO:teuthology.orchestra.run.smithi092.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-11T21:32:05.683 INFO:teuthology.orchestra.run.smithi092.stderr:osd.2: osd_enable_op_tracker = '' 2024-03-11T21:32:05.683 INFO:teuthology.orchestra.run.smithi092.stderr:osd.3: osd_enable_op_tracker = '' 2024-03-11T21:32:05.683 INFO:teuthology.orchestra.run.smithi092.stderr:osd.4: osd_enable_op_tracker = '' 2024-03-11T21:32:05.683 INFO:teuthology.orchestra.run.smithi092.stderr:osd.5: osd_enable_op_tracker = '' 2024-03-11T21:32:05.683 INFO:teuthology.orchestra.run.smithi092.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-11T21:32:05.683 INFO:teuthology.orchestra.run.smithi092.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-11T21:32:05.697 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:32:05.698 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:32:05.698 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:32:05.799 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:32:05.832 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:32:05.833 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:32:05.833 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:32:05.934 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:32:05.974 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:32:05.974 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:32:05.975 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:32:06.075 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:32:06.120 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:32:06.121 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:32:06.121 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:32:06.223 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:32:06.255 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:32:06.255 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:32:06.256 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:32:06.357 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:32:06.396 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:32:06.397 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:32:06.397 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:32:06.427 INFO:tasks.radosbench.radosbench.0.smithi092.stdout: 5 16 13974 13958 174.438 175.5 0.00322698 0.00572508 2024-03-11T21:32:06.499 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:32:06.533 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:32:06.533 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:32:06.534 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:32:06.576 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:32:06 smithi097 ceph-mon[33317]: 2.4 scrub ok 2024-03-11T21:32:06.576 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:32:06 smithi097 ceph-mon[33317]: pgmap v251: 148 pgs: 148 active+clean; 158 MiB data, 1.2 GiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 26 MiB/s wr, 420 op/s 2024-03-11T21:32:06.576 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:32:06 smithi097 ceph-mon[33317]: from='client.? v1:172.21.15.92:0/16449359' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-11T21:32:06.576 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:32:06 smithi097 ceph-mon[33317]: 6.2 scrub starts 2024-03-11T21:32:06.576 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:32:06 smithi097 ceph-mon[33317]: 6.2 scrub ok 2024-03-11T21:32:06.610 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:32:06 smithi092 ceph-mon[37563]: 2.4 scrub ok 2024-03-11T21:32:06.611 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:32:06 smithi092 ceph-mon[37563]: pgmap v251: 148 pgs: 148 active+clean; 158 MiB data, 1.2 GiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 26 MiB/s wr, 420 op/s 2024-03-11T21:32:06.611 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:32:06 smithi092 ceph-mon[37563]: from='client.? v1:172.21.15.92:0/16449359' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-11T21:32:06.611 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:32:06 smithi092 ceph-mon[37563]: 6.2 scrub starts 2024-03-11T21:32:06.611 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:32:06 smithi092 ceph-mon[37563]: 6.2 scrub ok 2024-03-11T21:32:06.611 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:32:06 smithi092 ceph-mon[29572]: 2.4 scrub ok 2024-03-11T21:32:06.611 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:32:06 smithi092 ceph-mon[29572]: pgmap v251: 148 pgs: 148 active+clean; 158 MiB data, 1.2 GiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 26 MiB/s wr, 420 op/s 2024-03-11T21:32:06.611 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:32:06 smithi092 ceph-mon[29572]: from='client.? v1:172.21.15.92:0/16449359' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-11T21:32:06.611 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:32:06 smithi092 ceph-mon[29572]: 6.2 scrub starts 2024-03-11T21:32:06.611 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:32:06 smithi092 ceph-mon[29572]: 6.2 scrub ok 2024-03-11T21:32:06.635 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:32:06.675 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:32:06.676 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:32:06.676 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:32:06.777 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:32:06.790 DEBUG:teuthology.orchestra.run.smithi092:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid 00db0016-dfed-11ee-95c8-87774f69a715 -- ceph osd set noscrub 2024-03-11T21:32:06.809 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:32:06.809 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:32:06.810 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:32:06.911 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:32:06.945 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:32:06.945 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:32:06.946 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:32:07.046 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:32:07.087 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:32:07.088 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:32:07.088 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:32:07.189 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:32:07.232 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:32:07.233 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:32:07.233 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:32:07.334 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:32:07.374 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:32:07.375 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:32:07.375 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:32:07.427 INFO:tasks.radosbench.radosbench.0.smithi092.stdout: 6 16 16549 16533 172.186 160.938 0.00562191 0.00580112 2024-03-11T21:32:07.476 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:32:07.507 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:32:07.508 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:32:07.508 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:32:07.609 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:32:07.645 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:32:07.645 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:32:07.646 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:32:07.746 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:32:07.788 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:32:07.788 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:32:07.789 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:32:07.800 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:32:07 smithi097 ceph-mon[33317]: 2.6 scrub starts 2024-03-11T21:32:07.800 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:32:07 smithi097 ceph-mon[33317]: 2.6 scrub ok 2024-03-11T21:32:07.800 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:32:07 smithi097 ceph-mon[33317]: 2.8 scrub starts 2024-03-11T21:32:07.800 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:32:07 smithi097 ceph-mon[33317]: 2.8 scrub ok 2024-03-11T21:32:07.800 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:32:07 smithi097 ceph-mon[33317]: from='mgr.14607 ' entity='mgr.y' 2024-03-11T21:32:07.800 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:32:07 smithi097 ceph-mon[33317]: from='mgr.14607 v1:172.21.15.92:0/1702717597' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T21:32:07.800 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:32:07 smithi097 ceph-mon[33317]: 6.15 scrub starts 2024-03-11T21:32:07.800 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:32:07 smithi097 ceph-mon[33317]: 6.15 scrub ok 2024-03-11T21:32:07.860 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:32:07 smithi092 ceph-mon[37563]: 2.6 scrub starts 2024-03-11T21:32:07.861 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:32:07 smithi092 ceph-mon[37563]: 2.6 scrub ok 2024-03-11T21:32:07.861 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:32:07 smithi092 ceph-mon[37563]: 2.8 scrub starts 2024-03-11T21:32:07.861 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:32:07 smithi092 ceph-mon[37563]: 2.8 scrub ok 2024-03-11T21:32:07.861 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:32:07 smithi092 ceph-mon[37563]: from='mgr.14607 ' entity='mgr.y' 2024-03-11T21:32:07.861 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:32:07 smithi092 ceph-mon[37563]: from='mgr.14607 v1:172.21.15.92:0/1702717597' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T21:32:07.861 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:32:07 smithi092 ceph-mon[37563]: 6.15 scrub starts 2024-03-11T21:32:07.861 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:32:07 smithi092 ceph-mon[37563]: 6.15 scrub ok 2024-03-11T21:32:07.861 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:32:07 smithi092 ceph-mon[29572]: 2.6 scrub starts 2024-03-11T21:32:07.861 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:32:07 smithi092 ceph-mon[29572]: 2.6 scrub ok 2024-03-11T21:32:07.861 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:32:07 smithi092 ceph-mon[29572]: 2.8 scrub starts 2024-03-11T21:32:07.861 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:32:07 smithi092 ceph-mon[29572]: 2.8 scrub ok 2024-03-11T21:32:07.861 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:32:07 smithi092 ceph-mon[29572]: from='mgr.14607 ' entity='mgr.y' 2024-03-11T21:32:07.862 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:32:07 smithi092 ceph-mon[29572]: from='mgr.14607 v1:172.21.15.92:0/1702717597' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T21:32:07.862 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:32:07 smithi092 ceph-mon[29572]: 6.15 scrub starts 2024-03-11T21:32:07.862 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:32:07 smithi092 ceph-mon[29572]: 6.15 scrub ok 2024-03-11T21:32:07.890 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:32:07.919 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:32:07.919 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:32:07.919 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:32:08.020 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:32:08.054 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:32:08.055 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:32:08.055 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:32:08.156 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:32:08.181 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:32:08.182 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:32:08.182 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:32:08.283 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:32:08.316 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:32:08.317 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:32:08.317 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:32:08.345 INFO:journalctl@ceph.mgr.y.smithi092.stdout:Mar 11 21:32:07 smithi092 ceph-00db0016-dfed-11ee-95c8-87774f69a715-mgr-y[29801]: ::ffff:172.21.15.97 - - [11/Mar/2024:21:32:07] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-11T21:32:08.418 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:32:08.427 INFO:tasks.radosbench.radosbench.0.smithi092.stdout: 7 15 18821 18806 167.881 142.062 0.00608569 0.00594985 2024-03-11T21:32:08.452 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:32:08.453 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:32:08.453 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:32:08.554 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:32:08.586 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:32:08.586 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:32:08.587 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:32:08.610 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:32:08 smithi092 ceph-mon[37563]: pgmap v252: 148 pgs: 148 active+clean; 380 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 654 B/s rd, 48 MiB/s wr, 776 op/s 2024-03-11T21:32:08.610 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:32:08 smithi092 ceph-mon[37563]: 5.1e scrub starts 2024-03-11T21:32:08.610 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:32:08 smithi092 ceph-mon[37563]: 5.1e scrub ok 2024-03-11T21:32:08.610 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:32:08 smithi092 ceph-mon[37563]: 4.8 deep-scrub starts 2024-03-11T21:32:08.610 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:32:08 smithi092 ceph-mon[37563]: 4.8 deep-scrub ok 2024-03-11T21:32:08.610 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:32:08 smithi092 ceph-mon[29572]: pgmap v252: 148 pgs: 148 active+clean; 380 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 654 B/s rd, 48 MiB/s wr, 776 op/s 2024-03-11T21:32:08.611 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:32:08 smithi092 ceph-mon[29572]: 5.1e scrub starts 2024-03-11T21:32:08.611 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:32:08 smithi092 ceph-mon[29572]: 5.1e scrub ok 2024-03-11T21:32:08.611 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:32:08 smithi092 ceph-mon[29572]: 4.8 deep-scrub starts 2024-03-11T21:32:08.611 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:32:08 smithi092 ceph-mon[29572]: 4.8 deep-scrub ok 2024-03-11T21:32:08.687 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:32:08.729 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:32:08.730 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:32:08.730 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:32:08.826 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:32:08 smithi097 ceph-mon[33317]: pgmap v252: 148 pgs: 148 active+clean; 380 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 654 B/s rd, 48 MiB/s wr, 776 op/s 2024-03-11T21:32:08.826 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:32:08 smithi097 ceph-mon[33317]: 5.1e scrub starts 2024-03-11T21:32:08.826 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:32:08 smithi097 ceph-mon[33317]: 5.1e scrub ok 2024-03-11T21:32:08.826 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:32:08 smithi097 ceph-mon[33317]: 4.8 deep-scrub starts 2024-03-11T21:32:08.826 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:32:08 smithi097 ceph-mon[33317]: 4.8 deep-scrub ok 2024-03-11T21:32:08.831 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:32:08.862 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:32:08.863 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:32:08.863 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:32:08.964 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:32:09.008 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:32:09.009 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:32:09.009 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:32:09.110 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:32:09.151 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:32:09.151 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:32:09.151 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:32:09.217 INFO:teuthology.orchestra.run.smithi092.stdout:ERROR: (22) Invalid argument 2024-03-11T21:32:09.217 INFO:teuthology.orchestra.run.smithi092.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-11T21:32:09.253 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:32:09.290 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:32:09.291 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:32:09.292 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:32:09.392 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:32:09.426 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:32:09.427 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:32:09.427 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:32:09.428 INFO:tasks.radosbench.radosbench.0.smithi092.stdout: 8 16 21256 21240 165.91 152.125 0.00298802 0.00602018 2024-03-11T21:32:09.528 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:32:09.559 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:32:09.560 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:32:09.560 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:32:09.604 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:32:09 smithi092 ceph-mon[29572]: 4.17 scrub starts 2024-03-11T21:32:09.604 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:32:09 smithi092 ceph-mon[29572]: 4.17 scrub ok 2024-03-11T21:32:09.604 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:32:09 smithi092 ceph-mon[29572]: purged_snaps scrub starts 2024-03-11T21:32:09.604 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:32:09 smithi092 ceph-mon[29572]: purged_snaps scrub ok 2024-03-11T21:32:09.604 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:32:09 smithi092 ceph-mon[29572]: 5.10 scrub starts 2024-03-11T21:32:09.604 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:32:09 smithi092 ceph-mon[29572]: 5.10 scrub ok 2024-03-11T21:32:09.604 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:32:09 smithi092 ceph-mon[29572]: 4.5 scrub starts 2024-03-11T21:32:09.604 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:32:09 smithi092 ceph-mon[29572]: 4.5 scrub ok 2024-03-11T21:32:09.604 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:32:09 smithi092 ceph-mon[37563]: 4.17 scrub starts 2024-03-11T21:32:09.604 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:32:09 smithi092 ceph-mon[37563]: 4.17 scrub ok 2024-03-11T21:32:09.605 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:32:09 smithi092 ceph-mon[37563]: purged_snaps scrub starts 2024-03-11T21:32:09.605 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:32:09 smithi092 ceph-mon[37563]: purged_snaps scrub ok 2024-03-11T21:32:09.605 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:32:09 smithi092 ceph-mon[37563]: 5.10 scrub starts 2024-03-11T21:32:09.605 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:32:09 smithi092 ceph-mon[37563]: 5.10 scrub ok 2024-03-11T21:32:09.605 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:32:09 smithi092 ceph-mon[37563]: 4.5 scrub starts 2024-03-11T21:32:09.605 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:32:09 smithi092 ceph-mon[37563]: 4.5 scrub ok 2024-03-11T21:32:09.660 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:32:09.699 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:32:09.700 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:32:09.700 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:32:09.801 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:32:09.826 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:32:09 smithi097 ceph-mon[33317]: 4.17 scrub starts 2024-03-11T21:32:09.826 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:32:09 smithi097 ceph-mon[33317]: 4.17 scrub ok 2024-03-11T21:32:09.826 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:32:09 smithi097 ceph-mon[33317]: purged_snaps scrub starts 2024-03-11T21:32:09.826 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:32:09 smithi097 ceph-mon[33317]: purged_snaps scrub ok 2024-03-11T21:32:09.827 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:32:09 smithi097 ceph-mon[33317]: 5.10 scrub starts 2024-03-11T21:32:09.827 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:32:09 smithi097 ceph-mon[33317]: 5.10 scrub ok 2024-03-11T21:32:09.827 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:32:09 smithi097 ceph-mon[33317]: 4.5 scrub starts 2024-03-11T21:32:09.827 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:32:09 smithi097 ceph-mon[33317]: 4.5 scrub ok 2024-03-11T21:32:09.833 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:32:09.833 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:32:09.834 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:32:09.935 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:32:09.961 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:32:09.962 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:32:09.962 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:32:10.063 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:32:10.094 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:32:10.095 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:32:10.095 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:32:10.196 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:32:10.228 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:32:10.228 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:32:10.229 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:32:10.330 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:32:10.357 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:32:10.357 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:32:10.358 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:32:10.427 INFO:tasks.radosbench.radosbench.0.smithi092.stdout: 9 16 23340 23324 161.947 130.25 0.042351 0.00615802 2024-03-11T21:32:10.459 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:32:10.499 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:32:10.500 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:32:10.500 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:32:10.601 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:32:10.632 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:32:10.633 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:32:10.634 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:32:10.734 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:32:10.777 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:32:10.777 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:32:10.778 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:32:10.826 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:32:10 smithi097 ceph-mon[33317]: pgmap v253: 148 pgs: 148 active+clean; 1017 MiB data, 3.9 GiB used, 711 GiB / 715 GiB avail; 1.2 KiB/s rd, 127 MiB/s wr, 2.03k op/s 2024-03-11T21:32:10.826 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:32:10 smithi097 ceph-mon[33317]: osdmap e164: 8 total, 8 up, 8 in 2024-03-11T21:32:10.827 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:32:10 smithi097 ceph-mon[33317]: 6.7 scrub starts 2024-03-11T21:32:10.827 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:32:10 smithi097 ceph-mon[33317]: 6.7 scrub ok 2024-03-11T21:32:10.827 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:32:10 smithi097 ceph-mon[33317]: 2.11 scrub starts 2024-03-11T21:32:10.827 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:32:10 smithi097 ceph-mon[33317]: 2.11 scrub ok 2024-03-11T21:32:10.861 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:32:10 smithi092 ceph-mon[29572]: pgmap v253: 148 pgs: 148 active+clean; 1017 MiB data, 3.9 GiB used, 711 GiB / 715 GiB avail; 1.2 KiB/s rd, 127 MiB/s wr, 2.03k op/s 2024-03-11T21:32:10.861 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:32:10 smithi092 ceph-mon[29572]: osdmap e164: 8 total, 8 up, 8 in 2024-03-11T21:32:10.861 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:32:10 smithi092 ceph-mon[29572]: 6.7 scrub starts 2024-03-11T21:32:10.861 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:32:10 smithi092 ceph-mon[29572]: 6.7 scrub ok 2024-03-11T21:32:10.861 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:32:10 smithi092 ceph-mon[29572]: 2.11 scrub starts 2024-03-11T21:32:10.861 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:32:10 smithi092 ceph-mon[29572]: 2.11 scrub ok 2024-03-11T21:32:10.862 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:32:10 smithi092 ceph-mon[37563]: pgmap v253: 148 pgs: 148 active+clean; 1017 MiB data, 3.9 GiB used, 711 GiB / 715 GiB avail; 1.2 KiB/s rd, 127 MiB/s wr, 2.03k op/s 2024-03-11T21:32:10.862 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:32:10 smithi092 ceph-mon[37563]: osdmap e164: 8 total, 8 up, 8 in 2024-03-11T21:32:10.862 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:32:10 smithi092 ceph-mon[37563]: 6.7 scrub starts 2024-03-11T21:32:10.862 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:32:10 smithi092 ceph-mon[37563]: 6.7 scrub ok 2024-03-11T21:32:10.862 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:32:10 smithi092 ceph-mon[37563]: 2.11 scrub starts 2024-03-11T21:32:10.862 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:32:10 smithi092 ceph-mon[37563]: 2.11 scrub ok 2024-03-11T21:32:10.879 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:32:10.919 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:32:10.919 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:32:10.920 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:32:11.021 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:32:11.060 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:32:11.061 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:32:11.061 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:32:11.115 DEBUG:teuthology.orchestra.run.smithi092:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid 00db0016-dfed-11ee-95c8-87774f69a715 -- ceph daemon osd.3 dump_blocked_ops 2024-03-11T21:32:11.162 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:32:11.191 INFO:teuthology.orchestra.run.smithi092.stdout: 2024-03-11T21:32:11.205 INFO:teuthology.orchestra.run.smithi092.stderr:dumped all 2024-03-11T21:32:11.207 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:32:11.207 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:32:11.208 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:32:11.309 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:32:11.350 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:32:11.352 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:32:11.353 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:32:11.427 INFO:tasks.radosbench.radosbench.0.smithi092.stdout: 10 16 25523 25507 159.395 136.438 0.00404011 0.00626716 2024-03-11T21:32:11.454 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:32:11.500 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:32:11.500 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:32:11.501 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:32:11.601 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:32:11.602 DEBUG:teuthology.orchestra.run.smithi092:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid 00db0016-dfed-11ee-95c8-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-03-11T21:32:11.634 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:32:11.634 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:32:11.635 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:32:11.737 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:32:11.768 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:32:11.769 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:32:11.769 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:32:11.826 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:32:11 smithi097 ceph-mon[33317]: from='client.25042 v1:172.21.15.92:0/2056788888' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-11T21:32:11.826 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:32:11 smithi097 ceph-mon[33317]: 4.15 scrub starts 2024-03-11T21:32:11.826 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:32:11 smithi097 ceph-mon[33317]: 4.15 scrub ok 2024-03-11T21:32:11.826 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:32:11 smithi097 ceph-mon[33317]: 6.5 scrub starts 2024-03-11T21:32:11.826 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:32:11 smithi097 ceph-mon[33317]: 6.5 scrub ok 2024-03-11T21:32:11.860 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:32:11 smithi092 ceph-mon[29572]: from='client.25042 v1:172.21.15.92:0/2056788888' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-11T21:32:11.860 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:32:11 smithi092 ceph-mon[29572]: 4.15 scrub starts 2024-03-11T21:32:11.860 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:32:11 smithi092 ceph-mon[29572]: 4.15 scrub ok 2024-03-11T21:32:11.861 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:32:11 smithi092 ceph-mon[29572]: 6.5 scrub starts 2024-03-11T21:32:11.861 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:32:11 smithi092 ceph-mon[29572]: 6.5 scrub ok 2024-03-11T21:32:11.861 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:32:11 smithi092 ceph-mon[37563]: from='client.25042 v1:172.21.15.92:0/2056788888' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-11T21:32:11.861 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:32:11 smithi092 ceph-mon[37563]: 4.15 scrub starts 2024-03-11T21:32:11.861 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:32:11 smithi092 ceph-mon[37563]: 4.15 scrub ok 2024-03-11T21:32:11.861 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:32:11 smithi092 ceph-mon[37563]: 6.5 scrub starts 2024-03-11T21:32:11.861 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:32:11 smithi092 ceph-mon[37563]: 6.5 scrub ok 2024-03-11T21:32:11.870 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:32:11.900 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:32:11.901 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:32:11.901 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:32:12.002 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:32:12.017 INFO:teuthology.orchestra.run.smithi092.stdout:{"pg_ready":true,"pg_map":{"version":254,"stamp":"2024-03-11T21:32:09.517096+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":1065884072,"num_objects":16453,"num_object_clones":0,"num_object_copies":49359,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":16453,"num_whiteouts":0,"num_read":1302,"num_read_kb":1150,"num_write":16870,"num_write_kb":1042781,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":86,"num_bytes_recovered":2363598,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":16926,"ondisk_log_size":16926,"up":444,"acting":444,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":444,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":749961216,"kb_used":4056768,"kb_used_data":2877244,"kb_used_omap":12,"kb_used_meta":1179443,"kb_avail":745904448,"statfs":{"total":767960285184,"available":763806154752,"internally_reserved":0,"allocated":2946297856,"data_stored":2942922888,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12752,"internal_metadata":1207750192},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,1,1],"upper_bound":8},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":900136960,"num_objects":13735,"num_object_clones":0,"num_object_copies":41205,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13735,"num_whiteouts":0,"num_read":5,"num_read_kb":5,"num_write":13735,"num_write_kb":879040,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.427418"},"pg_stats":[{"pgid":"6.1b","version":"0'0","reported_seq":222,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-03-11T21:32:04.326279+0000","last_change":"2024-03-11T21:31:48.238290+0000","last_active":"2024-03-11T21:32:04.326279+0000","last_peered":"2024-03-11T21:32:04.326279+0000","last_clean":"2024-03-11T21:32:04.326279+0000","last_became_active":"2024-03-11T21:31:48.237220+0000","last_became_peered":"2024-03-11T21:31:48.237220+0000","last_unstale":"2024-03-11T21:32:04.326279+0000","last_undegraded":"2024-03-11T21:32:04.326279+0000","last_fullsized":"2024-03-11T21:32:04.326279+0000","mapping_epoch":153,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":154,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:31:06.310490+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:43.559452+0000","last_clean_scrub_stamp":"2024-03-11T21:31:06.310490+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:32:30.638456+0000","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,1],"acting":[3,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":221,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-03-11T21:32:04.315060+0000","last_change":"2024-03-11T21:31:10.598620+0000","last_active":"2024-03-11T21:32:04.315060+0000","last_peered":"2024-03-11T21:32:04.315060+0000","last_clean":"2024-03-11T21:32:04.315060+0000","last_became_active":"2024-03-11T21:25:42.578893+0000","last_became_peered":"2024-03-11T21:25:42.578893+0000","last_unstale":"2024-03-11T21:32:04.315060+0000","last_undegraded":"2024-03-11T21:32:04.315060+0000","last_fullsized":"2024-03-11T21:32:04.315060+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:31:10.598515+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:41.544550+0000","last_clean_scrub_stamp":"2024-03-11T21:31:10.598515+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:32:32.512946+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1f","version":"0'0","reported_seq":244,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-03-11T21:32:04.330211+0000","last_change":"2024-03-11T21:31:54.380555+0000","last_active":"2024-03-11T21:32:04.330211+0000","last_peered":"2024-03-11T21:32:04.330211+0000","last_clean":"2024-03-11T21:32:04.330211+0000","last_became_active":"2024-03-11T21:31:48.225548+0000","last_became_peered":"2024-03-11T21:31:48.225548+0000","last_unstale":"2024-03-11T21:32:04.330211+0000","last_undegraded":"2024-03-11T21:32:04.330211+0000","last_fullsized":"2024-03-11T21:32:04.330211+0000","mapping_epoch":153,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":154,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:31:54.380472+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:37.512018+0000","last_clean_scrub_stamp":"2024-03-11T21:31:54.380472+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:33:09.244401+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,6],"acting":[0,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.19","version":"74'15","reported_seq":249,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-03-11T21:32:04.323236+0000","last_change":"2024-03-11T21:31:07.311590+0000","last_active":"2024-03-11T21:32:04.323236+0000","last_peered":"2024-03-11T21:32:04.323236+0000","last_clean":"2024-03-11T21:32:04.323236+0000","last_became_active":"2024-03-11T21:25:40.554207+0000","last_became_peered":"2024-03-11T21:25:40.554207+0000","last_unstale":"2024-03-11T21:32:04.323236+0000","last_undegraded":"2024-03-11T21:32:04.323236+0000","last_fullsized":"2024-03-11T21:32:04.323236+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'15","last_scrub_stamp":"2024-03-11T21:31:07.311470+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:39.525554+0000","last_clean_scrub_stamp":"2024-03-11T21:31:07.311470+0000","objects_scrubbed":7,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:32:36.263715+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.1a","version":"0'0","reported_seq":217,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-03-11T21:32:04.315233+0000","last_change":"2024-03-11T21:32:01.718632+0000","last_active":"2024-03-11T21:32:04.315233+0000","last_peered":"2024-03-11T21:32:04.315233+0000","last_clean":"2024-03-11T21:32:04.315233+0000","last_became_active":"2024-03-11T21:25:44.590923+0000","last_became_peered":"2024-03-11T21:25:44.590923+0000","last_unstale":"2024-03-11T21:32:04.315233+0000","last_undegraded":"2024-03-11T21:32:04.315233+0000","last_fullsized":"2024-03-11T21:32:04.315233+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:32:01.718486+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:43.559452+0000","last_clean_scrub_stamp":"2024-03-11T21:32:01.718486+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:33:26.052102+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,1],"acting":[4,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":227,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-03-11T21:32:04.330277+0000","last_change":"2024-03-11T21:31:48.226625+0000","last_active":"2024-03-11T21:32:04.330277+0000","last_peered":"2024-03-11T21:32:04.330277+0000","last_clean":"2024-03-11T21:32:04.330277+0000","last_became_active":"2024-03-11T21:31:48.226180+0000","last_became_peered":"2024-03-11T21:31:48.226180+0000","last_unstale":"2024-03-11T21:32:04.330277+0000","last_undegraded":"2024-03-11T21:32:04.330277+0000","last_fullsized":"2024-03-11T21:32:04.330277+0000","mapping_epoch":153,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":154,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:31:15.812251+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:41.544550+0000","last_clean_scrub_stamp":"2024-03-11T21:31:15.812251+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:32:22.828377+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,2],"acting":[1,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1e","version":"0'0","reported_seq":229,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-03-11T21:32:04.324657+0000","last_change":"2024-03-11T21:30:17.913666+0000","last_active":"2024-03-11T21:32:04.324657+0000","last_peered":"2024-03-11T21:32:04.324657+0000","last_clean":"2024-03-11T21:32:04.324657+0000","last_became_active":"2024-03-11T21:25:38.537283+0000","last_became_peered":"2024-03-11T21:25:38.537283+0000","last_unstale":"2024-03-11T21:32:04.324657+0000","last_undegraded":"2024-03-11T21:32:04.324657+0000","last_fullsized":"2024-03-11T21:32:04.324657+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:30:17.913550+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:37.512018+0000","last_clean_scrub_stamp":"2024-03-11T21:30:17.913550+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.18","version":"74'9","reported_seq":249,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-03-11T21:32:04.314801+0000","last_change":"2024-03-11T21:32:00.674307+0000","last_active":"2024-03-11T21:32:04.314801+0000","last_peered":"2024-03-11T21:32:04.314801+0000","last_clean":"2024-03-11T21:32:04.314801+0000","last_became_active":"2024-03-11T21:25:40.552216+0000","last_became_peered":"2024-03-11T21:25:40.552216+0000","last_unstale":"2024-03-11T21:32:04.314801+0000","last_undegraded":"2024-03-11T21:32:04.314801+0000","last_fullsized":"2024-03-11T21:32:04.314801+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'9","last_scrub_stamp":"2024-03-11T21:32:00.674188+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:39.525554+0000","last_clean_scrub_stamp":"2024-03-11T21:32:00.674188+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:33:07.357445+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":216,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-03-11T21:32:04.317881+0000","last_change":"2024-03-11T21:29:47.008124+0000","last_active":"2024-03-11T21:32:04.317881+0000","last_peered":"2024-03-11T21:32:04.317881+0000","last_clean":"2024-03-11T21:32:04.317881+0000","last_became_active":"2024-03-11T21:25:44.590234+0000","last_became_peered":"2024-03-11T21:25:44.590234+0000","last_unstale":"2024-03-11T21:32:04.317881+0000","last_undegraded":"2024-03-11T21:32:04.317881+0000","last_fullsized":"2024-03-11T21:32:04.317881+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:29:47.008018+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:43.559452+0000","last_clean_scrub_stamp":"2024-03-11T21:29:47.008018+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":133,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3],"acting":[5,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":38,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-03-11T21:32:04.315509+0000","last_change":"2024-03-11T21:31:56.567886+0000","last_active":"2024-03-11T21:32:04.315509+0000","last_peered":"2024-03-11T21:32:04.315509+0000","last_clean":"2024-03-11T21:32:04.315509+0000","last_became_active":"2024-03-11T21:31:48.218366+0000","last_became_peered":"2024-03-11T21:31:48.218366+0000","last_unstale":"2024-03-11T21:32:04.315509+0000","last_undegraded":"2024-03-11T21:32:04.315509+0000","last_fullsized":"2024-03-11T21:32:04.315509+0000","mapping_epoch":153,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":154,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:31:56.567802+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:41.544550+0000","last_clean_scrub_stamp":"2024-03-11T21:31:56.567802+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:33:19.092666+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,0],"acting":[4,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1d","version":"0'0","reported_seq":236,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-03-11T21:32:04.322960+0000","last_change":"2024-03-11T21:31:57.371941+0000","last_active":"2024-03-11T21:32:04.322960+0000","last_peered":"2024-03-11T21:32:04.322960+0000","last_clean":"2024-03-11T21:32:04.322960+0000","last_became_active":"2024-03-11T21:25:38.542628+0000","last_became_peered":"2024-03-11T21:25:38.542628+0000","last_unstale":"2024-03-11T21:32:04.322960+0000","last_undegraded":"2024-03-11T21:32:04.322960+0000","last_fullsized":"2024-03-11T21:32:04.322960+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:31:57.371862+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:37.512018+0000","last_clean_scrub_stamp":"2024-03-11T21:31:57.371862+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:33:26.732272+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,0],"acting":[7,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.1b","version":"74'5","reported_seq":235,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-03-11T21:32:04.316286+0000","last_change":"2024-03-11T21:31:57.621692+0000","last_active":"2024-03-11T21:32:04.316286+0000","last_peered":"2024-03-11T21:32:04.316286+0000","last_clean":"2024-03-11T21:32:04.316286+0000","last_became_active":"2024-03-11T21:25:40.555826+0000","last_became_peered":"2024-03-11T21:25:40.555826+0000","last_unstale":"2024-03-11T21:32:04.316286+0000","last_undegraded":"2024-03-11T21:32:04.316286+0000","last_fullsized":"2024-03-11T21:32:04.316286+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'5","last_scrub_stamp":"2024-03-11T21:31:57.621580+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:39.525554+0000","last_clean_scrub_stamp":"2024-03-11T21:31:57.621580+0000","objects_scrubbed":2,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:33:18.541742+0000","scrub_duration":34,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":11,"num_read_kb":7,"num_write":6,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,1],"acting":[4,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":217,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-03-11T21:32:04.317541+0000","last_change":"2024-03-11T21:31:58.402086+0000","last_active":"2024-03-11T21:32:04.317541+0000","last_peered":"2024-03-11T21:32:04.317541+0000","last_clean":"2024-03-11T21:32:04.317541+0000","last_became_active":"2024-03-11T21:25:44.589786+0000","last_became_peered":"2024-03-11T21:25:44.589786+0000","last_unstale":"2024-03-11T21:32:04.317541+0000","last_undegraded":"2024-03-11T21:32:04.317541+0000","last_fullsized":"2024-03-11T21:32:04.317541+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:31:58.401956+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:43.559452+0000","last_clean_scrub_stamp":"2024-03-11T21:31:58.401956+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:33:09.708327+0000","scrub_duration":20,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,7],"acting":[0,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":226,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-03-11T21:32:04.318038+0000","last_change":"2024-03-11T21:31:48.235333+0000","last_active":"2024-03-11T21:32:04.318038+0000","last_peered":"2024-03-11T21:32:04.318038+0000","last_clean":"2024-03-11T21:32:04.318038+0000","last_became_active":"2024-03-11T21:31:48.234980+0000","last_became_peered":"2024-03-11T21:31:48.234980+0000","last_unstale":"2024-03-11T21:32:04.318038+0000","last_undegraded":"2024-03-11T21:32:04.318038+0000","last_fullsized":"2024-03-11T21:32:04.318038+0000","mapping_epoch":153,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":154,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:31:00.533376+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:41.544550+0000","last_clean_scrub_stamp":"2024-03-11T21:31:00.533376+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:32:21.078260+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,4],"acting":[5,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1c","version":"0'0","reported_seq":236,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-03-11T21:32:04.322950+0000","last_change":"2024-03-11T21:31:58.422366+0000","last_active":"2024-03-11T21:32:04.322950+0000","last_peered":"2024-03-11T21:32:04.322950+0000","last_clean":"2024-03-11T21:32:04.322950+0000","last_became_active":"2024-03-11T21:25:38.541364+0000","last_became_peered":"2024-03-11T21:25:38.541364+0000","last_unstale":"2024-03-11T21:32:04.322950+0000","last_undegraded":"2024-03-11T21:32:04.322950+0000","last_fullsized":"2024-03-11T21:32:04.322950+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:31:58.422287+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:28:32.566437+0000","last_clean_scrub_stamp":"2024-03-11T21:31:58.422287+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:33:23.582317+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.1a","version":"74'9","reported_seq":249,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-03-11T21:32:04.628747+0000","last_change":"2024-03-11T21:32:04.628796+0000","last_active":"2024-03-11T21:32:04.628747+0000","last_peered":"2024-03-11T21:32:04.628747+0000","last_clean":"2024-03-11T21:32:04.628747+0000","last_became_active":"2024-03-11T21:25:40.556629+0000","last_became_peered":"2024-03-11T21:25:40.556629+0000","last_unstale":"2024-03-11T21:32:04.628747+0000","last_undegraded":"2024-03-11T21:32:04.628747+0000","last_fullsized":"2024-03-11T21:32:04.628747+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'9","last_scrub_stamp":"2024-03-11T21:32:04.628652+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:39.525554+0000","last_clean_scrub_stamp":"2024-03-11T21:32:04.628652+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:33:09.872947+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0],"acting":[4,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1f","version":"0'0","reported_seq":217,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-03-11T21:32:04.327018+0000","last_change":"2024-03-11T21:31:04.253653+0000","last_active":"2024-03-11T21:32:04.327018+0000","last_peered":"2024-03-11T21:32:04.327018+0000","last_clean":"2024-03-11T21:32:04.327018+0000","last_became_active":"2024-03-11T21:25:45.302637+0000","last_became_peered":"2024-03-11T21:25:45.302637+0000","last_unstale":"2024-03-11T21:32:04.327018+0000","last_undegraded":"2024-03-11T21:32:04.327018+0000","last_fullsized":"2024-03-11T21:32:04.327018+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:31:04.253137+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:43.559452+0000","last_clean_scrub_stamp":"2024-03-11T21:31:04.253137+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:32:05.471991+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1c","version":"0'0","reported_seq":221,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-03-11T21:32:04.315919+0000","last_change":"2024-03-11T21:30:59.496458+0000","last_active":"2024-03-11T21:32:04.315919+0000","last_peered":"2024-03-11T21:32:04.315919+0000","last_clean":"2024-03-11T21:32:04.315919+0000","last_became_active":"2024-03-11T21:25:42.576064+0000","last_became_peered":"2024-03-11T21:25:42.576064+0000","last_unstale":"2024-03-11T21:32:04.315919+0000","last_undegraded":"2024-03-11T21:32:04.315919+0000","last_fullsized":"2024-03-11T21:32:04.315919+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:30:59.496330+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:41.544550+0000","last_clean_scrub_stamp":"2024-03-11T21:30:59.496330+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:32:27.314836+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1b","version":"0'0","reported_seq":234,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-03-11T21:32:04.326486+0000","last_change":"2024-03-11T21:31:48.233696+0000","last_active":"2024-03-11T21:32:04.326486+0000","last_peered":"2024-03-11T21:32:04.326486+0000","last_clean":"2024-03-11T21:32:04.326486+0000","last_became_active":"2024-03-11T21:31:48.233345+0000","last_became_peered":"2024-03-11T21:31:48.233345+0000","last_unstale":"2024-03-11T21:32:04.326486+0000","last_undegraded":"2024-03-11T21:32:04.326486+0000","last_fullsized":"2024-03-11T21:32:04.326486+0000","mapping_epoch":153,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":154,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:31:05.272969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:28:33.582712+0000","last_clean_scrub_stamp":"2024-03-11T21:31:05.272969+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:32:17.251367+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,4],"acting":[3,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1d","version":"74'12","reported_seq":247,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-03-11T21:32:04.324169+0000","last_change":"2024-03-11T21:31:20.514563+0000","last_active":"2024-03-11T21:32:04.324169+0000","last_peered":"2024-03-11T21:32:04.324169+0000","last_clean":"2024-03-11T21:32:04.324169+0000","last_became_active":"2024-03-11T21:25:40.556330+0000","last_became_peered":"2024-03-11T21:25:40.556330+0000","last_unstale":"2024-03-11T21:32:04.324169+0000","last_undegraded":"2024-03-11T21:32:04.324169+0000","last_fullsized":"2024-03-11T21:32:04.324169+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'12","last_scrub_stamp":"2024-03-11T21:31:20.514423+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:39.525554+0000","last_clean_scrub_stamp":"2024-03-11T21:31:20.514423+0000","objects_scrubbed":5,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:32:20.604067+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":25,"num_read_kb":16,"num_write":14,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.1e","version":"0'0","reported_seq":210,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-03-11T21:32:04.315768+0000","last_change":"2024-03-11T21:30:11.322364+0000","last_active":"2024-03-11T21:32:04.315768+0000","last_peered":"2024-03-11T21:32:04.315768+0000","last_clean":"2024-03-11T21:32:04.315768+0000","last_became_active":"2024-03-11T21:25:45.300355+0000","last_became_peered":"2024-03-11T21:25:45.300355+0000","last_unstale":"2024-03-11T21:32:04.315768+0000","last_undegraded":"2024-03-11T21:32:04.315768+0000","last_fullsized":"2024-03-11T21:32:04.315768+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:30:11.322178+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:43.559452+0000","last_clean_scrub_stamp":"2024-03-11T21:30:11.322178+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5],"acting":[4,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1d","version":"0'0","reported_seq":221,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-03-11T21:32:04.332909+0000","last_change":"2024-03-11T21:30:18.678642+0000","last_active":"2024-03-11T21:32:04.332909+0000","last_peered":"2024-03-11T21:32:04.332909+0000","last_clean":"2024-03-11T21:32:04.332909+0000","last_became_active":"2024-03-11T21:25:42.579407+0000","last_became_peered":"2024-03-11T21:25:42.579407+0000","last_unstale":"2024-03-11T21:32:04.332909+0000","last_undegraded":"2024-03-11T21:32:04.332909+0000","last_fullsized":"2024-03-11T21:32:04.332909+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:30:18.678531+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:30:18.678531+0000","last_clean_scrub_stamp":"2024-03-11T21:30:18.678531+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1a","version":"0'0","reported_seq":243,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-03-11T21:32:04.320857+0000","last_change":"2024-03-11T21:31:59.716486+0000","last_active":"2024-03-11T21:32:04.320857+0000","last_peered":"2024-03-11T21:32:04.320857+0000","last_clean":"2024-03-11T21:32:04.320857+0000","last_became_active":"2024-03-11T21:31:48.222582+0000","last_became_peered":"2024-03-11T21:31:48.222582+0000","last_unstale":"2024-03-11T21:32:04.320857+0000","last_undegraded":"2024-03-11T21:32:04.320857+0000","last_fullsized":"2024-03-11T21:32:04.320857+0000","mapping_epoch":153,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":154,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:31:59.716376+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:37.512018+0000","last_clean_scrub_stamp":"2024-03-11T21:31:59.716376+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:33:05.765786+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,2],"acting":[6,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.1c","version":"74'15","reported_seq":249,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-03-11T21:32:04.330508+0000","last_change":"2024-03-11T21:31:01.272353+0000","last_active":"2024-03-11T21:32:04.330508+0000","last_peered":"2024-03-11T21:32:04.330508+0000","last_clean":"2024-03-11T21:32:04.330508+0000","last_became_active":"2024-03-11T21:25:40.551453+0000","last_became_peered":"2024-03-11T21:25:40.551453+0000","last_unstale":"2024-03-11T21:32:04.330508+0000","last_undegraded":"2024-03-11T21:32:04.330508+0000","last_fullsized":"2024-03-11T21:32:04.330508+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'15","last_scrub_stamp":"2024-03-11T21:31:01.272189+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:39.525554+0000","last_clean_scrub_stamp":"2024-03-11T21:31:01.272189+0000","objects_scrubbed":7,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:32:24.109315+0000","scrub_duration":22,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.1d","version":"0'0","reported_seq":217,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-03-11T21:32:04.325812+0000","last_change":"2024-03-11T21:31:18.798813+0000","last_active":"2024-03-11T21:32:04.325812+0000","last_peered":"2024-03-11T21:32:04.325812+0000","last_clean":"2024-03-11T21:32:04.325812+0000","last_became_active":"2024-03-11T21:25:44.588845+0000","last_became_peered":"2024-03-11T21:25:44.588845+0000","last_unstale":"2024-03-11T21:32:04.325812+0000","last_undegraded":"2024-03-11T21:32:04.325812+0000","last_fullsized":"2024-03-11T21:32:04.325812+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:31:18.798701+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:43.559452+0000","last_clean_scrub_stamp":"2024-03-11T21:31:18.798701+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:32:47.454620+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.1e","version":"0'0","reported_seq":230,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-03-11T21:32:07.397619+0000","last_change":"2024-03-11T21:32:07.397655+0000","last_active":"2024-03-11T21:32:07.397619+0000","last_peered":"2024-03-11T21:32:07.397619+0000","last_clean":"2024-03-11T21:32:07.397619+0000","last_became_active":"2024-03-11T21:25:42.582959+0000","last_became_peered":"2024-03-11T21:25:42.582959+0000","last_unstale":"2024-03-11T21:32:07.397619+0000","last_undegraded":"2024-03-11T21:32:07.397619+0000","last_fullsized":"2024-03-11T21:32:07.397619+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:32:07.397538+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:26:58.335431+0000","last_clean_scrub_stamp":"2024-03-11T21:32:07.397538+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:33:32.766194+0000","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.19","version":"67'2","reported_seq":227,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-03-11T21:32:04.323380+0000","last_change":"2024-03-11T21:29:42.815863+0000","last_active":"2024-03-11T21:32:04.323380+0000","last_peered":"2024-03-11T21:32:04.323380+0000","last_clean":"2024-03-11T21:32:04.323380+0000","last_became_active":"2024-03-11T21:25:38.539243+0000","last_became_peered":"2024-03-11T21:25:38.539243+0000","last_unstale":"2024-03-11T21:32:04.323380+0000","last_undegraded":"2024-03-11T21:32:04.323380+0000","last_fullsized":"2024-03-11T21:32:04.323380+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'2","last_scrub_stamp":"2024-03-11T21:29:42.815740+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:37.512018+0000","last_clean_scrub_stamp":"2024-03-11T21:29:42.815740+0000","objects_scrubbed":2,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":957,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":7,"num_read_kb":7,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,0],"acting":[3,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1f","version":"124'16","reported_seq":258,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-03-11T21:32:04.321657+0000","last_change":"2024-03-11T21:31:06.449601+0000","last_active":"2024-03-11T21:32:04.321657+0000","last_peered":"2024-03-11T21:32:04.321657+0000","last_clean":"2024-03-11T21:32:04.321657+0000","last_became_active":"2024-03-11T21:25:40.554686+0000","last_became_peered":"2024-03-11T21:25:40.554686+0000","last_unstale":"2024-03-11T21:32:04.321657+0000","last_undegraded":"2024-03-11T21:32:04.321657+0000","last_fullsized":"2024-03-11T21:32:04.321657+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"124'16","last_scrub_stamp":"2024-03-11T21:31:06.449509+0000","last_deep_scrub":"124'16","last_deep_scrub_stamp":"2024-03-11T21:29:30.873490+0000","last_clean_scrub_stamp":"2024-03-11T21:31:06.449509+0000","objects_scrubbed":5,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:32:20.104923+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,1],"acting":[6,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.1c","version":"74'1","reported_seq":45,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-03-11T21:32:04.318258+0000","last_change":"2024-03-11T21:31:48.249824+0000","last_active":"2024-03-11T21:32:04.318258+0000","last_peered":"2024-03-11T21:32:04.318258+0000","last_clean":"2024-03-11T21:32:04.318258+0000","last_became_active":"2024-03-11T21:31:48.236162+0000","last_became_peered":"2024-03-11T21:31:48.236162+0000","last_unstale":"2024-03-11T21:32:04.318258+0000","last_undegraded":"2024-03-11T21:32:04.318258+0000","last_fullsized":"2024-03-11T21:32:04.318258+0000","mapping_epoch":153,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":154,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'1","last_scrub_stamp":"2024-03-11T21:31:07.489215+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:43.559452+0000","last_clean_scrub_stamp":"2024-03-11T21:31:07.489215+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:32:26.903791+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":370,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":370,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,3],"acting":[5,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.1f","version":"154'9","reported_seq":329,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-03-11T21:32:04.321652+0000","last_change":"2024-03-11T21:31:48.243462+0000","last_active":"2024-03-11T21:32:04.321652+0000","last_peered":"2024-03-11T21:32:04.321652+0000","last_clean":"2024-03-11T21:32:04.321652+0000","last_became_active":"2024-03-11T21:31:48.227717+0000","last_became_peered":"2024-03-11T21:31:48.227717+0000","last_unstale":"2024-03-11T21:32:04.321652+0000","last_undegraded":"2024-03-11T21:32:04.321652+0000","last_fullsized":"2024-03-11T21:32:04.321652+0000","mapping_epoch":153,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":154,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'8","last_scrub_stamp":"2024-03-11T21:31:02.499061+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:41.544550+0000","last_clean_scrub_stamp":"2024-03-11T21:31:02.499061+0000","objects_scrubbed":1,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:32:05.780313+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,2],"acting":[6,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.18","version":"67'1","reported_seq":252,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-03-11T21:32:04.317864+0000","last_change":"2024-03-11T21:31:59.576103+0000","last_active":"2024-03-11T21:32:04.317864+0000","last_peered":"2024-03-11T21:32:04.317864+0000","last_clean":"2024-03-11T21:32:04.317864+0000","last_became_active":"2024-03-11T21:31:48.225066+0000","last_became_peered":"2024-03-11T21:31:48.225066+0000","last_unstale":"2024-03-11T21:32:04.317864+0000","last_undegraded":"2024-03-11T21:32:04.317864+0000","last_fullsized":"2024-03-11T21:32:04.317864+0000","mapping_epoch":153,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":154,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'1","last_scrub_stamp":"2024-03-11T21:31:59.575976+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:37.512018+0000","last_clean_scrub_stamp":"2024-03-11T21:31:59.575976+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:32:59.997189+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":436,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":7,"num_read_kb":7,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":436,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1e","version":"74'10","reported_seq":259,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-03-11T21:32:04.330678+0000","last_change":"2024-03-11T21:31:48.252894+0000","last_active":"2024-03-11T21:32:04.330678+0000","last_peered":"2024-03-11T21:32:04.330678+0000","last_clean":"2024-03-11T21:32:04.330678+0000","last_became_active":"2024-03-11T21:31:48.231510+0000","last_became_peered":"2024-03-11T21:31:48.231510+0000","last_unstale":"2024-03-11T21:32:04.330678+0000","last_undegraded":"2024-03-11T21:32:04.330678+0000","last_fullsized":"2024-03-11T21:32:04.330678+0000","mapping_epoch":153,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":154,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'10","last_scrub_stamp":"2024-03-11T21:30:57.371717+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:39.525554+0000","last_clean_scrub_stamp":"2024-03-11T21:30:57.371717+0000","objects_scrubbed":5,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:32:15.016682+0000","scrub_duration":27,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1],"acting":[0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.13","version":"0'0","reported_seq":216,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-03-11T21:32:04.328036+0000","last_change":"2024-03-11T21:31:00.239533+0000","last_active":"2024-03-11T21:32:04.328036+0000","last_peered":"2024-03-11T21:32:04.328036+0000","last_clean":"2024-03-11T21:32:04.328036+0000","last_became_active":"2024-03-11T21:25:45.302102+0000","last_became_peered":"2024-03-11T21:25:45.302102+0000","last_unstale":"2024-03-11T21:32:04.328036+0000","last_undegraded":"2024-03-11T21:32:04.328036+0000","last_fullsized":"2024-03-11T21:32:04.328036+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:31:00.238325+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:43.559452+0000","last_clean_scrub_stamp":"2024-03-11T21:31:00.238325+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:32:15.077313+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":30,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-03-11T21:32:04.314839+0000","last_change":"2024-03-11T21:31:48.225206+0000","last_active":"2024-03-11T21:32:04.314839+0000","last_peered":"2024-03-11T21:32:04.314839+0000","last_clean":"2024-03-11T21:32:04.314839+0000","last_became_active":"2024-03-11T21:31:48.224831+0000","last_became_peered":"2024-03-11T21:31:48.224831+0000","last_unstale":"2024-03-11T21:32:04.314839+0000","last_undegraded":"2024-03-11T21:32:04.314839+0000","last_fullsized":"2024-03-11T21:32:04.314839+0000","mapping_epoch":153,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":154,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:29:47.630271+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:41.544550+0000","last_clean_scrub_stamp":"2024-03-11T21:29:47.630271+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,0],"acting":[4,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.17","version":"0'0","reported_seq":237,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-03-11T21:32:04.321543+0000","last_change":"2024-03-11T21:32:00.758583+0000","last_active":"2024-03-11T21:32:04.321543+0000","last_peered":"2024-03-11T21:32:04.321543+0000","last_clean":"2024-03-11T21:32:04.321543+0000","last_became_active":"2024-03-11T21:25:38.540349+0000","last_became_peered":"2024-03-11T21:25:38.540349+0000","last_unstale":"2024-03-11T21:32:04.321543+0000","last_undegraded":"2024-03-11T21:32:04.321543+0000","last_fullsized":"2024-03-11T21:32:04.321543+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:32:00.758492+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:28:25.918101+0000","last_clean_scrub_stamp":"2024-03-11T21:32:00.758492+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:33:27.761241+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,2],"acting":[6,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.11","version":"74'11","reported_seq":261,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-03-11T21:32:04.323637+0000","last_change":"2024-03-11T21:31:48.250922+0000","last_active":"2024-03-11T21:32:04.323637+0000","last_peered":"2024-03-11T21:32:04.323637+0000","last_clean":"2024-03-11T21:32:04.323637+0000","last_became_active":"2024-03-11T21:31:48.229218+0000","last_became_peered":"2024-03-11T21:31:48.229218+0000","last_unstale":"2024-03-11T21:32:04.323637+0000","last_undegraded":"2024-03-11T21:32:04.323637+0000","last_fullsized":"2024-03-11T21:32:04.323637+0000","mapping_epoch":153,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":154,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'11","last_scrub_stamp":"2024-03-11T21:31:17.450051+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:39.525554+0000","last_clean_scrub_stamp":"2024-03-11T21:31:17.450051+0000","objects_scrubbed":5,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:32:31.155922+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":38,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-03-11T21:32:04.331641+0000","last_change":"2024-03-11T21:31:55.360729+0000","last_active":"2024-03-11T21:32:04.331641+0000","last_peered":"2024-03-11T21:32:04.331641+0000","last_clean":"2024-03-11T21:32:04.331641+0000","last_became_active":"2024-03-11T21:31:48.226446+0000","last_became_peered":"2024-03-11T21:31:48.226446+0000","last_unstale":"2024-03-11T21:32:04.331641+0000","last_undegraded":"2024-03-11T21:32:04.331641+0000","last_fullsized":"2024-03-11T21:32:04.331641+0000","mapping_epoch":153,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":154,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:31:55.360650+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:43.559452+0000","last_clean_scrub_stamp":"2024-03-11T21:31:55.360650+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:32:56.657476+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,3],"acting":[2,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":235,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-03-11T21:32:04.320726+0000","last_change":"2024-03-11T21:31:48.223455+0000","last_active":"2024-03-11T21:32:04.320726+0000","last_peered":"2024-03-11T21:32:04.320726+0000","last_clean":"2024-03-11T21:32:04.320726+0000","last_became_active":"2024-03-11T21:31:48.223088+0000","last_became_peered":"2024-03-11T21:31:48.223088+0000","last_unstale":"2024-03-11T21:32:04.320726+0000","last_undegraded":"2024-03-11T21:32:04.320726+0000","last_fullsized":"2024-03-11T21:32:04.320726+0000","mapping_epoch":153,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":154,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:31:12.423671+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:29:39.923895+0000","last_clean_scrub_stamp":"2024-03-11T21:31:12.423671+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:32:19.031986+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,0],"acting":[6,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.16","version":"0'0","reported_seq":236,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-03-11T21:32:04.318273+0000","last_change":"2024-03-11T21:30:59.528390+0000","last_active":"2024-03-11T21:32:04.318273+0000","last_peered":"2024-03-11T21:32:04.318273+0000","last_clean":"2024-03-11T21:32:04.318273+0000","last_became_active":"2024-03-11T21:25:38.539103+0000","last_became_peered":"2024-03-11T21:25:38.539103+0000","last_unstale":"2024-03-11T21:32:04.318273+0000","last_undegraded":"2024-03-11T21:32:04.318273+0000","last_fullsized":"2024-03-11T21:32:04.318273+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:30:59.528265+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:27:11.938909+0000","last_clean_scrub_stamp":"2024-03-11T21:30:59.528265+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:32:24.866196+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,2],"acting":[5,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.10","version":"94'6","reported_seq":232,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-03-11T21:32:04.322778+0000","last_change":"2024-03-11T21:30:21.932499+0000","last_active":"2024-03-11T21:32:04.322778+0000","last_peered":"2024-03-11T21:32:04.322778+0000","last_clean":"2024-03-11T21:32:04.322778+0000","last_became_active":"2024-03-11T21:25:40.552720+0000","last_became_peered":"2024-03-11T21:25:40.552720+0000","last_unstale":"2024-03-11T21:32:04.322778+0000","last_undegraded":"2024-03-11T21:32:04.322778+0000","last_fullsized":"2024-03-11T21:32:04.322778+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"94'6","last_scrub_stamp":"2024-03-11T21:30:21.932340+0000","last_deep_scrub":"94'6","last_deep_scrub_stamp":"2024-03-11T21:28:24.603974+0000","last_clean_scrub_stamp":"2024-03-11T21:30:21.932340+0000","objects_scrubbed":2,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":6,"num_read_kb":4,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6],"acting":[3,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":216,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-03-11T21:32:04.323090+0000","last_change":"2024-03-11T21:31:14.338908+0000","last_active":"2024-03-11T21:32:04.323090+0000","last_peered":"2024-03-11T21:32:04.323090+0000","last_clean":"2024-03-11T21:32:04.323090+0000","last_became_active":"2024-03-11T21:25:44.583181+0000","last_became_peered":"2024-03-11T21:25:44.583181+0000","last_unstale":"2024-03-11T21:32:04.323090+0000","last_undegraded":"2024-03-11T21:32:04.323090+0000","last_fullsized":"2024-03-11T21:32:04.323090+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:31:14.338640+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:43.559452+0000","last_clean_scrub_stamp":"2024-03-11T21:31:14.338640+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:32:30.708648+0000","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":221,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-03-11T21:32:04.330802+0000","last_change":"2024-03-11T21:30:19.680660+0000","last_active":"2024-03-11T21:32:04.330802+0000","last_peered":"2024-03-11T21:32:04.330802+0000","last_clean":"2024-03-11T21:32:04.330802+0000","last_became_active":"2024-03-11T21:25:42.573158+0000","last_became_peered":"2024-03-11T21:25:42.573158+0000","last_unstale":"2024-03-11T21:32:04.330802+0000","last_undegraded":"2024-03-11T21:32:04.330802+0000","last_fullsized":"2024-03-11T21:32:04.330802+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:30:19.680539+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:27:21.207303+0000","last_clean_scrub_stamp":"2024-03-11T21:30:19.680539+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.15","version":"0'0","reported_seq":229,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-03-11T21:32:04.327085+0000","last_change":"2024-03-11T21:29:43.022451+0000","last_active":"2024-03-11T21:32:04.327085+0000","last_peered":"2024-03-11T21:32:04.327085+0000","last_clean":"2024-03-11T21:32:04.327085+0000","last_became_active":"2024-03-11T21:25:38.541705+0000","last_became_peered":"2024-03-11T21:25:38.541705+0000","last_unstale":"2024-03-11T21:32:04.327085+0000","last_undegraded":"2024-03-11T21:32:04.327085+0000","last_fullsized":"2024-03-11T21:32:04.327085+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:29:43.022321+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:26:44.860499+0000","last_clean_scrub_stamp":"2024-03-11T21:29:43.022321+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.13","version":"109'16","reported_seq":267,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-03-11T21:32:04.316206+0000","last_change":"2024-03-11T21:30:23.314515+0000","last_active":"2024-03-11T21:32:04.316206+0000","last_peered":"2024-03-11T21:32:04.316206+0000","last_clean":"2024-03-11T21:32:04.316206+0000","last_became_active":"2024-03-11T21:26:11.214879+0000","last_became_peered":"2024-03-11T21:26:11.214879+0000","last_unstale":"2024-03-11T21:32:04.316206+0000","last_undegraded":"2024-03-11T21:32:04.316206+0000","last_fullsized":"2024-03-11T21:32:04.316206+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"109'16","last_scrub_stamp":"2024-03-11T21:30:23.314410+0000","last_deep_scrub":"109'16","last_deep_scrub_stamp":"2024-03-11T21:28:32.854393+0000","last_clean_scrub_stamp":"2024-03-11T21:30:23.314410+0000","objects_scrubbed":5,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,2],"acting":[4,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.10","version":"0'0","reported_seq":217,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-03-11T21:32:04.316949+0000","last_change":"2024-03-11T21:31:22.541299+0000","last_active":"2024-03-11T21:32:04.316949+0000","last_peered":"2024-03-11T21:32:04.316949+0000","last_clean":"2024-03-11T21:32:04.316949+0000","last_became_active":"2024-03-11T21:25:44.591189+0000","last_became_peered":"2024-03-11T21:25:44.591189+0000","last_unstale":"2024-03-11T21:32:04.316949+0000","last_undegraded":"2024-03-11T21:32:04.316949+0000","last_fullsized":"2024-03-11T21:32:04.316949+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:31:22.541178+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:27:25.417450+0000","last_clean_scrub_stamp":"2024-03-11T21:31:22.541178+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:32:32.828094+0000","scrub_duration":18,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,1],"acting":[0,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":220,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-03-11T21:32:04.326571+0000","last_change":"2024-03-11T21:31:15.368570+0000","last_active":"2024-03-11T21:32:04.326571+0000","last_peered":"2024-03-11T21:32:04.326571+0000","last_clean":"2024-03-11T21:32:04.326571+0000","last_became_active":"2024-03-11T21:25:42.576425+0000","last_became_peered":"2024-03-11T21:25:42.576425+0000","last_unstale":"2024-03-11T21:32:04.326571+0000","last_undegraded":"2024-03-11T21:32:04.326571+0000","last_fullsized":"2024-03-11T21:32:04.326571+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:31:15.367898+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:31:15.367898+0000","last_clean_scrub_stamp":"2024-03-11T21:31:15.367898+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:32:26.039743+0000","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.14","version":"0'0","reported_seq":237,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-03-11T21:32:04.321372+0000","last_change":"2024-03-11T21:31:09.407017+0000","last_active":"2024-03-11T21:32:04.321372+0000","last_peered":"2024-03-11T21:32:04.321372+0000","last_clean":"2024-03-11T21:32:04.321372+0000","last_became_active":"2024-03-11T21:25:38.541852+0000","last_became_peered":"2024-03-11T21:25:38.541852+0000","last_unstale":"2024-03-11T21:32:04.321372+0000","last_undegraded":"2024-03-11T21:32:04.321372+0000","last_fullsized":"2024-03-11T21:32:04.321372+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:31:09.406884+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:37.512018+0000","last_clean_scrub_stamp":"2024-03-11T21:31:09.406884+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:32:09.476644+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,5],"acting":[6,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.12","version":"74'9","reported_seq":249,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-03-11T21:32:04.330603+0000","last_change":"2024-03-11T21:31:59.579237+0000","last_active":"2024-03-11T21:32:04.330603+0000","last_peered":"2024-03-11T21:32:04.330603+0000","last_clean":"2024-03-11T21:32:04.330603+0000","last_became_active":"2024-03-11T21:25:40.547082+0000","last_became_peered":"2024-03-11T21:25:40.547082+0000","last_unstale":"2024-03-11T21:32:04.330603+0000","last_undegraded":"2024-03-11T21:32:04.330603+0000","last_fullsized":"2024-03-11T21:32:04.330603+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'9","last_scrub_stamp":"2024-03-11T21:31:59.579142+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:39.525554+0000","last_clean_scrub_stamp":"2024-03-11T21:31:59.579142+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:33:15.464959+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":217,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-03-11T21:32:04.315472+0000","last_change":"2024-03-11T21:31:25.682563+0000","last_active":"2024-03-11T21:32:04.315472+0000","last_peered":"2024-03-11T21:32:04.315472+0000","last_clean":"2024-03-11T21:32:04.315472+0000","last_became_active":"2024-03-11T21:25:44.589586+0000","last_became_peered":"2024-03-11T21:25:44.589586+0000","last_unstale":"2024-03-11T21:32:04.315472+0000","last_undegraded":"2024-03-11T21:32:04.315472+0000","last_fullsized":"2024-03-11T21:32:04.315472+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:31:25.682446+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:27:09.801138+0000","last_clean_scrub_stamp":"2024-03-11T21:31:25.682446+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:32:34.081976+0000","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,5],"acting":[4,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.14","version":"154'9","reported_seq":316,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-03-11T21:32:04.325951+0000","last_change":"2024-03-11T21:32:03.467267+0000","last_active":"2024-03-11T21:32:04.325951+0000","last_peered":"2024-03-11T21:32:04.325951+0000","last_clean":"2024-03-11T21:32:04.325951+0000","last_became_active":"2024-03-11T21:31:48.230299+0000","last_became_peered":"2024-03-11T21:31:48.230299+0000","last_unstale":"2024-03-11T21:32:04.325951+0000","last_undegraded":"2024-03-11T21:32:04.325951+0000","last_fullsized":"2024-03-11T21:32:04.325951+0000","mapping_epoch":153,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":154,"parent":"0.0","parent_split_bits":0,"last_scrub":"154'9","last_scrub_stamp":"2024-03-11T21:32:03.467138+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:41.544550+0000","last_clean_scrub_stamp":"2024-03-11T21:32:03.467138+0000","objects_scrubbed":1,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:33:14.044164+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,1],"acting":[3,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.13","version":"0'0","reported_seq":229,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-03-11T21:32:04.326978+0000","last_change":"2024-03-11T21:31:18.506150+0000","last_active":"2024-03-11T21:32:04.326978+0000","last_peered":"2024-03-11T21:32:04.326978+0000","last_clean":"2024-03-11T21:32:04.326978+0000","last_became_active":"2024-03-11T21:25:38.539467+0000","last_became_peered":"2024-03-11T21:25:38.539467+0000","last_unstale":"2024-03-11T21:32:04.326978+0000","last_undegraded":"2024-03-11T21:32:04.326978+0000","last_fullsized":"2024-03-11T21:32:04.326978+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:31:18.499630+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:37.512018+0000","last_clean_scrub_stamp":"2024-03-11T21:31:18.499630+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:32:38.370145+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.15","version":"94'13","reported_seq":260,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-03-11T21:32:04.318229+0000","last_change":"2024-03-11T21:31:48.250051+0000","last_active":"2024-03-11T21:32:04.318229+0000","last_peered":"2024-03-11T21:32:04.318229+0000","last_clean":"2024-03-11T21:32:04.318229+0000","last_became_active":"2024-03-11T21:31:48.230445+0000","last_became_peered":"2024-03-11T21:31:48.230445+0000","last_unstale":"2024-03-11T21:32:04.318229+0000","last_undegraded":"2024-03-11T21:32:04.318229+0000","last_fullsized":"2024-03-11T21:32:04.318229+0000","mapping_epoch":153,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":154,"parent":"0.0","parent_split_bits":0,"last_scrub":"94'13","last_scrub_stamp":"2024-03-11T21:29:41.853158+0000","last_deep_scrub":"94'13","last_deep_scrub_stamp":"2024-03-11T21:29:41.853158+0000","last_clean_scrub_stamp":"2024-03-11T21:29:41.853158+0000","objects_scrubbed":4,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,6],"acting":[5,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.16","version":"0'0","reported_seq":223,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-03-11T21:32:04.330448+0000","last_change":"2024-03-11T21:31:48.232275+0000","last_active":"2024-03-11T21:32:04.330448+0000","last_peered":"2024-03-11T21:32:04.330448+0000","last_clean":"2024-03-11T21:32:04.330448+0000","last_became_active":"2024-03-11T21:31:48.231031+0000","last_became_peered":"2024-03-11T21:31:48.231031+0000","last_unstale":"2024-03-11T21:32:04.330448+0000","last_undegraded":"2024-03-11T21:32:04.330448+0000","last_fullsized":"2024-03-11T21:32:04.330448+0000","mapping_epoch":153,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":154,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:31:08.360582+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:43.559452+0000","last_clean_scrub_stamp":"2024-03-11T21:31:08.360582+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:32:09.194183+0000","scrub_duration":17,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,4],"acting":[0,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.15","version":"98'9","reported_seq":307,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-03-11T21:32:04.318409+0000","last_change":"2024-03-11T21:31:20.625060+0000","last_active":"2024-03-11T21:32:04.318409+0000","last_peered":"2024-03-11T21:32:04.318409+0000","last_clean":"2024-03-11T21:32:04.318409+0000","last_became_active":"2024-03-11T21:25:42.572555+0000","last_became_peered":"2024-03-11T21:25:42.572555+0000","last_unstale":"2024-03-11T21:32:04.318409+0000","last_undegraded":"2024-03-11T21:32:04.318409+0000","last_fullsized":"2024-03-11T21:32:04.318409+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"98'9","last_scrub_stamp":"2024-03-11T21:31:20.624923+0000","last_deep_scrub":"98'9","last_deep_scrub_stamp":"2024-03-11T21:31:20.624923+0000","last_clean_scrub_stamp":"2024-03-11T21:31:20.624923+0000","objects_scrubbed":1,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:32:45.365157+0000","scrub_duration":25,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.12","version":"0'0","reported_seq":234,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-03-11T21:32:04.318401+0000","last_change":"2024-03-11T21:31:48.223844+0000","last_active":"2024-03-11T21:32:04.318401+0000","last_peered":"2024-03-11T21:32:04.318401+0000","last_clean":"2024-03-11T21:32:04.318401+0000","last_became_active":"2024-03-11T21:31:48.223495+0000","last_became_peered":"2024-03-11T21:31:48.223495+0000","last_unstale":"2024-03-11T21:32:04.318401+0000","last_undegraded":"2024-03-11T21:32:04.318401+0000","last_fullsized":"2024-03-11T21:32:04.318401+0000","mapping_epoch":153,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":154,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:30:13.021666+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:37.512018+0000","last_clean_scrub_stamp":"2024-03-11T21:30:13.021666+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,0],"acting":[5,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.14","version":"74'10","reported_seq":249,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-03-11T21:32:04.325725+0000","last_change":"2024-03-11T21:31:48.258390+0000","last_active":"2024-03-11T21:32:04.325725+0000","last_peered":"2024-03-11T21:32:04.325725+0000","last_clean":"2024-03-11T21:32:04.325725+0000","last_became_active":"2024-03-11T21:31:48.229849+0000","last_became_peered":"2024-03-11T21:31:48.229849+0000","last_unstale":"2024-03-11T21:32:04.325725+0000","last_undegraded":"2024-03-11T21:32:04.325725+0000","last_fullsized":"2024-03-11T21:32:04.325725+0000","mapping_epoch":153,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":154,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'10","last_scrub_stamp":"2024-03-11T21:29:27.547582+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:39.525554+0000","last_clean_scrub_stamp":"2024-03-11T21:29:27.547582+0000","objects_scrubbed":5,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.15","version":"0'0","reported_seq":38,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-03-11T21:32:06.780151+0000","last_change":"2024-03-11T21:32:06.780189+0000","last_active":"2024-03-11T21:32:06.780151+0000","last_peered":"2024-03-11T21:32:06.780151+0000","last_clean":"2024-03-11T21:32:06.780151+0000","last_became_active":"2024-03-11T21:31:48.223661+0000","last_became_peered":"2024-03-11T21:31:48.223661+0000","last_unstale":"2024-03-11T21:32:06.780151+0000","last_undegraded":"2024-03-11T21:32:06.780151+0000","last_fullsized":"2024-03-11T21:32:06.780151+0000","mapping_epoch":153,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":154,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:32:06.779964+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:43.559452+0000","last_clean_scrub_stamp":"2024-03-11T21:32:06.779964+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:33:12.866979+0000","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,5],"acting":[6,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":220,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-03-11T21:32:04.318213+0000","last_change":"2024-03-11T21:31:14.608055+0000","last_active":"2024-03-11T21:32:04.318213+0000","last_peered":"2024-03-11T21:32:04.318213+0000","last_clean":"2024-03-11T21:32:04.318213+0000","last_became_active":"2024-03-11T21:25:42.577351+0000","last_became_peered":"2024-03-11T21:25:42.577351+0000","last_unstale":"2024-03-11T21:32:04.318213+0000","last_undegraded":"2024-03-11T21:32:04.318213+0000","last_fullsized":"2024-03-11T21:32:04.318213+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:31:14.607971+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:41.544550+0000","last_clean_scrub_stamp":"2024-03-11T21:31:14.607971+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:32:26.368031+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,1],"acting":[5,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.11","version":"0'0","reported_seq":229,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-03-11T21:32:04.320779+0000","last_change":"2024-03-11T21:30:14.818836+0000","last_active":"2024-03-11T21:32:04.320779+0000","last_peered":"2024-03-11T21:32:04.320779+0000","last_clean":"2024-03-11T21:32:04.320779+0000","last_became_active":"2024-03-11T21:25:38.537415+0000","last_became_peered":"2024-03-11T21:25:38.537415+0000","last_unstale":"2024-03-11T21:32:04.320779+0000","last_undegraded":"2024-03-11T21:32:04.320779+0000","last_fullsized":"2024-03-11T21:32:04.320779+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:30:14.818712+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:37.512018+0000","last_clean_scrub_stamp":"2024-03-11T21:30:14.818712+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.17","version":"97'9","reported_seq":256,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-03-11T21:32:04.327112+0000","last_change":"2024-03-11T21:31:48.243354+0000","last_active":"2024-03-11T21:32:04.327112+0000","last_peered":"2024-03-11T21:32:04.327112+0000","last_clean":"2024-03-11T21:32:04.327112+0000","last_became_active":"2024-03-11T21:31:48.224112+0000","last_became_peered":"2024-03-11T21:31:48.224112+0000","last_unstale":"2024-03-11T21:32:04.327112+0000","last_undegraded":"2024-03-11T21:32:04.327112+0000","last_fullsized":"2024-03-11T21:32:04.327112+0000","mapping_epoch":153,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":154,"parent":"0.0","parent_split_bits":0,"last_scrub":"97'9","last_scrub_stamp":"2024-03-11T21:29:26.522238+0000","last_deep_scrub":"97'9","last_deep_scrub_stamp":"2024-03-11T21:27:25.803132+0000","last_clean_scrub_stamp":"2024-03-11T21:29:26.522238+0000","objects_scrubbed":3,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":17,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":9,"num_read_kb":6,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,2],"acting":[3,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.14","version":"0'0","reported_seq":222,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-03-11T21:32:04.330139+0000","last_change":"2024-03-11T21:31:48.218674+0000","last_active":"2024-03-11T21:32:04.330139+0000","last_peered":"2024-03-11T21:32:04.330139+0000","last_clean":"2024-03-11T21:32:04.330139+0000","last_became_active":"2024-03-11T21:31:48.218375+0000","last_became_peered":"2024-03-11T21:31:48.218375+0000","last_unstale":"2024-03-11T21:32:04.330139+0000","last_undegraded":"2024-03-11T21:32:04.330139+0000","last_fullsized":"2024-03-11T21:32:04.330139+0000","mapping_epoch":153,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":154,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:31:15.409641+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:43.559452+0000","last_clean_scrub_stamp":"2024-03-11T21:31:15.409641+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:32:33.638512+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,0],"acting":[2,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":226,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-03-11T21:32:04.327205+0000","last_change":"2024-03-11T21:31:48.230708+0000","last_active":"2024-03-11T21:32:04.327205+0000","last_peered":"2024-03-11T21:32:04.327205+0000","last_clean":"2024-03-11T21:32:04.327205+0000","last_became_active":"2024-03-11T21:31:48.230253+0000","last_became_peered":"2024-03-11T21:31:48.230253+0000","last_unstale":"2024-03-11T21:32:04.327205+0000","last_undegraded":"2024-03-11T21:32:04.327205+0000","last_fullsized":"2024-03-11T21:32:04.327205+0000","mapping_epoch":153,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":154,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:31:08.284511+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:41.544550+0000","last_clean_scrub_stamp":"2024-03-11T21:31:08.284511+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:32:18.615321+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0],"acting":[3,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.10","version":"0'0","reported_seq":228,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-03-11T21:32:04.329487+0000","last_change":"2024-03-11T21:31:03.260526+0000","last_active":"2024-03-11T21:32:04.329487+0000","last_peered":"2024-03-11T21:32:04.329487+0000","last_clean":"2024-03-11T21:32:04.329487+0000","last_became_active":"2024-03-11T21:25:38.533138+0000","last_became_peered":"2024-03-11T21:25:38.533138+0000","last_unstale":"2024-03-11T21:32:04.329487+0000","last_undegraded":"2024-03-11T21:32:04.329487+0000","last_fullsized":"2024-03-11T21:32:04.329487+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:31:03.260408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:37.512018+0000","last_clean_scrub_stamp":"2024-03-11T21:31:03.260408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:32:29.743555+0000","scrub_duration":29,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,0],"acting":[2,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.16","version":"74'9","reported_seq":257,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-03-11T21:32:04.330551+0000","last_change":"2024-03-11T21:31:48.247407+0000","last_active":"2024-03-11T21:32:04.330551+0000","last_peered":"2024-03-11T21:32:04.330551+0000","last_clean":"2024-03-11T21:32:04.330551+0000","last_became_active":"2024-03-11T21:31:48.231280+0000","last_became_peered":"2024-03-11T21:31:48.231280+0000","last_unstale":"2024-03-11T21:32:04.330551+0000","last_undegraded":"2024-03-11T21:32:04.330551+0000","last_fullsized":"2024-03-11T21:32:04.330551+0000","mapping_epoch":153,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":154,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'9","last_scrub_stamp":"2024-03-11T21:30:13.857760+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:39.525554+0000","last_clean_scrub_stamp":"2024-03-11T21:30:13.857760+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":40,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.5","version":"163'989","reported_seq":1011,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-03-11T21:32:07.114048+0000","last_change":"2024-03-11T21:31:52.239825+0000","last_active":"2024-03-11T21:32:07.114048+0000","last_peered":"2024-03-11T21:32:07.114048+0000","last_clean":"2024-03-11T21:32:07.114048+0000","last_became_active":"2024-03-11T21:31:52.239480+0000","last_became_peered":"2024-03-11T21:31:52.239480+0000","last_unstale":"2024-03-11T21:32:07.114048+0000","last_undegraded":"2024-03-11T21:32:07.114048+0000","last_fullsized":"2024-03-11T21:32:07.114048+0000","mapping_epoch":157,"log_start":"0'0","ondisk_log_start":"0'0","created":157,"last_epoch_clean":158,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:31:51.219688+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:31:51.219688+0000","last_clean_scrub_stamp":"2024-03-11T21:31:51.219688+0000","objects_scrubbed":0,"log_size":989,"log_dups_size":0,"ondisk_log_size":989,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:33:18.248439+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":64815104,"num_objects":989,"num_object_clones":0,"num_object_copies":2967,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":989,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":989,"num_write_kb":63296,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,0],"acting":[6,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.b","version":"0'0","reported_seq":214,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-03-11T21:32:04.326653+0000","last_change":"2024-03-11T21:31:48.239140+0000","last_active":"2024-03-11T21:32:04.326653+0000","last_peered":"2024-03-11T21:32:04.326653+0000","last_clean":"2024-03-11T21:32:04.326653+0000","last_became_active":"2024-03-11T21:31:48.238287+0000","last_became_peered":"2024-03-11T21:31:48.238287+0000","last_unstale":"2024-03-11T21:32:04.326653+0000","last_undegraded":"2024-03-11T21:32:04.326653+0000","last_fullsized":"2024-03-11T21:32:04.326653+0000","mapping_epoch":153,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":154,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:30:19.006959+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:43.559452+0000","last_clean_scrub_stamp":"2024-03-11T21:30:19.006959+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":70,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6],"acting":[3,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.f","version":"67'2","reported_seq":241,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-03-11T21:32:04.315364+0000","last_change":"2024-03-11T21:31:03.533706+0000","last_active":"2024-03-11T21:32:04.315364+0000","last_peered":"2024-03-11T21:32:04.315364+0000","last_clean":"2024-03-11T21:32:04.315364+0000","last_became_active":"2024-03-11T21:25:38.537711+0000","last_became_peered":"2024-03-11T21:25:38.537711+0000","last_unstale":"2024-03-11T21:32:04.315364+0000","last_undegraded":"2024-03-11T21:32:04.315364+0000","last_fullsized":"2024-03-11T21:32:04.315364+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'2","last_scrub_stamp":"2024-03-11T21:31:03.533547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:37.512018+0000","last_clean_scrub_stamp":"2024-03-11T21:31:03.533547+0000","objects_scrubbed":2,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:32:27.195878+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":92,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":10,"num_read_kb":10,"num_write":4,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,7],"acting":[4,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.9","version":"146'17","reported_seq":253,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-03-11T21:32:04.315102+0000","last_change":"2024-03-11T21:31:16.615182+0000","last_active":"2024-03-11T21:32:04.315102+0000","last_peered":"2024-03-11T21:32:04.315102+0000","last_clean":"2024-03-11T21:32:04.315102+0000","last_became_active":"2024-03-11T21:25:40.552267+0000","last_became_peered":"2024-03-11T21:25:40.552267+0000","last_unstale":"2024-03-11T21:32:04.315102+0000","last_undegraded":"2024-03-11T21:32:04.315102+0000","last_fullsized":"2024-03-11T21:32:04.315102+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"146'17","last_scrub_stamp":"2024-03-11T21:31:16.615010+0000","last_deep_scrub":"146'17","last_deep_scrub_stamp":"2024-03-11T21:31:16.615010+0000","last_clean_scrub_stamp":"2024-03-11T21:31:16.615010+0000","objects_scrubbed":5,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:32:36.347947+0000","scrub_duration":27,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":25,"num_read_kb":16,"num_write":14,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,3],"acting":[4,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":220,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-03-11T21:32:04.329385+0000","last_change":"2024-03-11T21:31:12.434593+0000","last_active":"2024-03-11T21:32:04.329385+0000","last_peered":"2024-03-11T21:32:04.329385+0000","last_clean":"2024-03-11T21:32:04.329385+0000","last_became_active":"2024-03-11T21:25:42.573987+0000","last_became_peered":"2024-03-11T21:25:42.573987+0000","last_unstale":"2024-03-11T21:32:04.329385+0000","last_undegraded":"2024-03-11T21:32:04.329385+0000","last_fullsized":"2024-03-11T21:32:04.329385+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:31:12.434479+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:41.544550+0000","last_clean_scrub_stamp":"2024-03-11T21:31:12.434479+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:32:19.616693+0000","scrub_duration":17,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"8.e","version":"163'1099","reported_seq":1121,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-03-11T21:32:07.820503+0000","last_change":"2024-03-11T21:31:52.273246+0000","last_active":"2024-03-11T21:32:07.820503+0000","last_peered":"2024-03-11T21:32:07.820503+0000","last_clean":"2024-03-11T21:32:07.820503+0000","last_became_active":"2024-03-11T21:31:52.272382+0000","last_became_peered":"2024-03-11T21:31:52.272382+0000","last_unstale":"2024-03-11T21:32:07.820503+0000","last_undegraded":"2024-03-11T21:32:07.820503+0000","last_fullsized":"2024-03-11T21:32:07.820503+0000","mapping_epoch":157,"log_start":"0'0","ondisk_log_start":"0'0","created":157,"last_epoch_clean":158,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:31:51.219688+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:31:51.219688+0000","last_clean_scrub_stamp":"2024-03-11T21:31:51.219688+0000","objects_scrubbed":0,"log_size":1099,"log_dups_size":0,"ondisk_log_size":1099,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:33:00.550944+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":72024064,"num_objects":1099,"num_object_clones":0,"num_object_copies":3297,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1099,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1099,"num_write_kb":70336,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,4],"acting":[0,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.0","version":"0'0","reported_seq":218,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-03-11T21:32:04.317046+0000","last_change":"2024-03-11T21:32:03.422921+0000","last_active":"2024-03-11T21:32:04.317046+0000","last_peered":"2024-03-11T21:32:04.317046+0000","last_clean":"2024-03-11T21:32:04.317046+0000","last_became_active":"2024-03-11T21:25:44.584565+0000","last_became_peered":"2024-03-11T21:25:44.584565+0000","last_unstale":"2024-03-11T21:32:04.317046+0000","last_undegraded":"2024-03-11T21:32:04.317046+0000","last_fullsized":"2024-03-11T21:32:04.317046+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:32:03.422808+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:29:39.498784+0000","last_clean_scrub_stamp":"2024-03-11T21:32:03.422808+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:33:15.105649+0000","scrub_duration":31,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":244,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-03-11T21:32:04.509596+0000","last_change":"2024-03-11T21:32:04.509633+0000","last_active":"2024-03-11T21:32:04.509596+0000","last_peered":"2024-03-11T21:32:04.509596+0000","last_clean":"2024-03-11T21:32:04.509596+0000","last_became_active":"2024-03-11T21:31:48.224184+0000","last_became_peered":"2024-03-11T21:31:48.224184+0000","last_unstale":"2024-03-11T21:32:04.509596+0000","last_undegraded":"2024-03-11T21:32:04.509596+0000","last_fullsized":"2024-03-11T21:32:04.509596+0000","mapping_epoch":153,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":154,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:32:04.509514+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:37.512018+0000","last_clean_scrub_stamp":"2024-03-11T21:32:04.509514+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:33:08.577665+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.2","version":"94'15","reported_seq":254,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-03-11T21:32:04.330703+0000","last_change":"2024-03-11T21:31:08.956264+0000","last_active":"2024-03-11T21:32:04.330703+0000","last_peered":"2024-03-11T21:32:04.330703+0000","last_clean":"2024-03-11T21:32:04.330703+0000","last_became_active":"2024-03-11T21:25:40.548833+0000","last_became_peered":"2024-03-11T21:25:40.548833+0000","last_unstale":"2024-03-11T21:32:04.330703+0000","last_undegraded":"2024-03-11T21:32:04.330703+0000","last_fullsized":"2024-03-11T21:32:04.330703+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"94'15","last_scrub_stamp":"2024-03-11T21:31:08.955971+0000","last_deep_scrub":"94'15","last_deep_scrub_stamp":"2024-03-11T21:26:49.993060+0000","last_clean_scrub_stamp":"2024-03-11T21:31:08.955971+0000","objects_scrubbed":5,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:32:37.727507+0000","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.3","version":"74'8","reported_seq":308,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-03-11T21:32:04.327165+0000","last_change":"2024-03-11T21:31:15.465617+0000","last_active":"2024-03-11T21:32:04.327165+0000","last_peered":"2024-03-11T21:32:04.327165+0000","last_clean":"2024-03-11T21:32:04.327165+0000","last_became_active":"2024-03-11T21:25:42.578161+0000","last_became_peered":"2024-03-11T21:25:42.578161+0000","last_unstale":"2024-03-11T21:32:04.327165+0000","last_undegraded":"2024-03-11T21:32:04.327165+0000","last_fullsized":"2024-03-11T21:32:04.327165+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'8","last_scrub_stamp":"2024-03-11T21:31:15.465460+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:41.544550+0000","last_clean_scrub_stamp":"2024-03-11T21:31:15.465460+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:32:43.379445+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,5],"acting":[0,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.8","version":"163'1021","reported_seq":1043,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-03-11T21:32:07.010576+0000","last_change":"2024-03-11T21:31:52.240065+0000","last_active":"2024-03-11T21:32:07.010576+0000","last_peered":"2024-03-11T21:32:07.010576+0000","last_clean":"2024-03-11T21:32:07.010576+0000","last_became_active":"2024-03-11T21:31:52.239695+0000","last_became_peered":"2024-03-11T21:31:52.239695+0000","last_unstale":"2024-03-11T21:32:07.010576+0000","last_undegraded":"2024-03-11T21:32:07.010576+0000","last_fullsized":"2024-03-11T21:32:07.010576+0000","mapping_epoch":157,"log_start":"0'0","ondisk_log_start":"0'0","created":157,"last_epoch_clean":158,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:31:51.219688+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:31:51.219688+0000","last_clean_scrub_stamp":"2024-03-11T21:31:51.219688+0000","objects_scrubbed":0,"log_size":1021,"log_dups_size":0,"ondisk_log_size":1021,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:33:09.802875+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":66912256,"num_objects":1021,"num_object_clones":0,"num_object_copies":3063,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1021,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1021,"num_write_kb":65344,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,0],"acting":[5,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":222,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-03-11T21:32:04.324742+0000","last_change":"2024-03-11T21:31:48.236305+0000","last_active":"2024-03-11T21:32:04.324742+0000","last_peered":"2024-03-11T21:32:04.324742+0000","last_clean":"2024-03-11T21:32:04.324742+0000","last_became_active":"2024-03-11T21:31:48.234360+0000","last_became_peered":"2024-03-11T21:31:48.234360+0000","last_unstale":"2024-03-11T21:32:04.324742+0000","last_undegraded":"2024-03-11T21:32:04.324742+0000","last_fullsized":"2024-03-11T21:32:04.324742+0000","mapping_epoch":153,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":154,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:31:01.241180+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:28:30.563712+0000","last_clean_scrub_stamp":"2024-03-11T21:31:01.241180+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:32:24.330555+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,6],"acting":[3,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":237,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-03-11T21:32:04.318427+0000","last_change":"2024-03-11T21:31:16.608343+0000","last_active":"2024-03-11T21:32:04.318427+0000","last_peered":"2024-03-11T21:32:04.318427+0000","last_clean":"2024-03-11T21:32:04.318427+0000","last_became_active":"2024-03-11T21:25:38.538540+0000","last_became_peered":"2024-03-11T21:25:38.538540+0000","last_unstale":"2024-03-11T21:32:04.318427+0000","last_undegraded":"2024-03-11T21:32:04.318427+0000","last_fullsized":"2024-03-11T21:32:04.318427+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:31:16.608266+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:37.512018+0000","last_clean_scrub_stamp":"2024-03-11T21:31:16.608266+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:32:30.358883+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.4","version":"74'28","reported_seq":352,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-03-11T21:32:04.325227+0000","last_change":"2024-03-11T21:30:24.750195+0000","last_active":"2024-03-11T21:32:04.325227+0000","last_peered":"2024-03-11T21:32:04.325227+0000","last_clean":"2024-03-11T21:32:04.325227+0000","last_became_active":"2024-03-11T21:25:40.552902+0000","last_became_peered":"2024-03-11T21:25:40.552902+0000","last_unstale":"2024-03-11T21:32:04.325227+0000","last_undegraded":"2024-03-11T21:32:04.325227+0000","last_fullsized":"2024-03-11T21:32:04.325227+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'28","last_scrub_stamp":"2024-03-11T21:30:24.750058+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:39.525554+0000","last_clean_scrub_stamp":"2024-03-11T21:30:24.750058+0000","objects_scrubbed":10,"log_size":28,"log_dups_size":0,"ondisk_log_size":28,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":358,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":48,"num_read_kb":33,"num_write":26,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3],"acting":[1,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":222,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-03-11T21:32:04.325050+0000","last_change":"2024-03-11T21:31:57.398694+0000","last_active":"2024-03-11T21:32:04.325050+0000","last_peered":"2024-03-11T21:32:04.325050+0000","last_clean":"2024-03-11T21:32:04.325050+0000","last_became_active":"2024-03-11T21:25:42.572673+0000","last_became_peered":"2024-03-11T21:25:42.572673+0000","last_unstale":"2024-03-11T21:32:04.325050+0000","last_undegraded":"2024-03-11T21:32:04.325050+0000","last_fullsized":"2024-03-11T21:32:04.325050+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:31:57.398611+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:41.544550+0000","last_clean_scrub_stamp":"2024-03-11T21:31:57.398611+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:33:06.306030+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.b","version":"163'1084","reported_seq":1106,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-03-11T21:32:07.497391+0000","last_change":"2024-03-11T21:31:52.241413+0000","last_active":"2024-03-11T21:32:07.497391+0000","last_peered":"2024-03-11T21:32:07.497391+0000","last_clean":"2024-03-11T21:32:07.497391+0000","last_became_active":"2024-03-11T21:31:52.241089+0000","last_became_peered":"2024-03-11T21:31:52.241089+0000","last_unstale":"2024-03-11T21:32:07.497391+0000","last_undegraded":"2024-03-11T21:32:07.497391+0000","last_fullsized":"2024-03-11T21:32:07.497391+0000","mapping_epoch":157,"log_start":"0'0","ondisk_log_start":"0'0","created":157,"last_epoch_clean":158,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:31:51.219688+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:31:51.219688+0000","last_clean_scrub_stamp":"2024-03-11T21:31:51.219688+0000","objects_scrubbed":0,"log_size":1084,"log_dups_size":0,"ondisk_log_size":1084,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:33:08.118290+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":71041024,"num_objects":1084,"num_object_clones":0,"num_object_copies":3252,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1084,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1084,"num_write_kb":69376,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,1],"acting":[3,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.5","version":"0'0","reported_seq":30,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-03-11T21:32:04.321735+0000","last_change":"2024-03-11T21:31:48.234938+0000","last_active":"2024-03-11T21:32:04.321735+0000","last_peered":"2024-03-11T21:32:04.321735+0000","last_clean":"2024-03-11T21:32:04.321735+0000","last_became_active":"2024-03-11T21:31:48.234440+0000","last_became_peered":"2024-03-11T21:31:48.234440+0000","last_unstale":"2024-03-11T21:32:04.321735+0000","last_undegraded":"2024-03-11T21:32:04.321735+0000","last_fullsized":"2024-03-11T21:32:04.321735+0000","mapping_epoch":153,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":154,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:30:55.708059+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:43.559452+0000","last_clean_scrub_stamp":"2024-03-11T21:30:55.708059+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,2],"acting":[6,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.0","version":"76'5","reported_seq":622,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-03-11T21:32:05.407942+0000","last_change":"2024-03-11T21:30:14.726351+0000","last_active":"2024-03-11T21:32:05.407942+0000","last_peered":"2024-03-11T21:32:05.407942+0000","last_clean":"2024-03-11T21:32:05.407942+0000","last_became_active":"2024-03-11T21:25:38.536890+0000","last_became_peered":"2024-03-11T21:25:38.536890+0000","last_unstale":"2024-03-11T21:32:05.407942+0000","last_undegraded":"2024-03-11T21:32:05.407942+0000","last_fullsized":"2024-03-11T21:32:05.407942+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"76'5","last_scrub_stamp":"2024-03-11T21:30:14.726214+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:37.512018+0000","last_clean_scrub_stamp":"2024-03-11T21:30:14.726214+0000","objects_scrubbed":1,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0.000753559,"stat_sum":{"num_bytes":389,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":382,"num_read_kb":377,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,6],"acting":[1,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":237,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-03-11T21:32:04.329701+0000","last_change":"2024-03-11T21:31:21.359528+0000","last_active":"2024-03-11T21:32:04.329701+0000","last_peered":"2024-03-11T21:32:04.329701+0000","last_clean":"2024-03-11T21:32:04.329701+0000","last_became_active":"2024-03-11T21:25:38.532984+0000","last_became_peered":"2024-03-11T21:25:38.532984+0000","last_unstale":"2024-03-11T21:32:04.329701+0000","last_undegraded":"2024-03-11T21:32:04.329701+0000","last_fullsized":"2024-03-11T21:32:04.329701+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:31:21.359298+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:26:48.997454+0000","last_clean_scrub_stamp":"2024-03-11T21:31:21.359298+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:32:43.745058+0000","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.7","version":"97'18","reported_seq":260,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-03-11T21:32:04.327960+0000","last_change":"2024-03-11T21:30:11.805557+0000","last_active":"2024-03-11T21:32:04.327960+0000","last_peered":"2024-03-11T21:32:04.327960+0000","last_clean":"2024-03-11T21:32:04.327960+0000","last_became_active":"2024-03-11T21:25:40.541769+0000","last_became_peered":"2024-03-11T21:25:40.541769+0000","last_unstale":"2024-03-11T21:32:04.327960+0000","last_undegraded":"2024-03-11T21:32:04.327960+0000","last_fullsized":"2024-03-11T21:32:04.327960+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"97'18","last_scrub_stamp":"2024-03-11T21:30:11.805394+0000","last_deep_scrub":"97'18","last_deep_scrub_stamp":"2024-03-11T21:27:19.239404+0000","last_clean_scrub_stamp":"2024-03-11T21:30:11.805394+0000","objects_scrubbed":5,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":19,"num_write":16,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":226,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-03-11T21:32:04.330865+0000","last_change":"2024-03-11T21:31:48.236854+0000","last_active":"2024-03-11T21:32:04.330865+0000","last_peered":"2024-03-11T21:32:04.330865+0000","last_clean":"2024-03-11T21:32:04.330865+0000","last_became_active":"2024-03-11T21:31:48.236520+0000","last_became_peered":"2024-03-11T21:31:48.236520+0000","last_unstale":"2024-03-11T21:32:04.330865+0000","last_undegraded":"2024-03-11T21:32:04.330865+0000","last_fullsized":"2024-03-11T21:32:04.330865+0000","mapping_epoch":153,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":154,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:31:06.279751+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:41.544550+0000","last_clean_scrub_stamp":"2024-03-11T21:31:06.279751+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:32:21.307607+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,4],"acting":[2,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"8.9","version":"163'980","reported_seq":1002,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-03-11T21:32:07.050076+0000","last_change":"2024-03-11T21:31:52.241561+0000","last_active":"2024-03-11T21:32:07.050076+0000","last_peered":"2024-03-11T21:32:07.050076+0000","last_clean":"2024-03-11T21:32:07.050076+0000","last_became_active":"2024-03-11T21:31:52.241194+0000","last_became_peered":"2024-03-11T21:31:52.241194+0000","last_unstale":"2024-03-11T21:32:07.050076+0000","last_undegraded":"2024-03-11T21:32:07.050076+0000","last_fullsized":"2024-03-11T21:32:07.050076+0000","mapping_epoch":157,"log_start":"0'0","ondisk_log_start":"0'0","created":157,"last_epoch_clean":158,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:31:51.219688+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:31:51.219688+0000","last_clean_scrub_stamp":"2024-03-11T21:31:51.219688+0000","objects_scrubbed":0,"log_size":980,"log_dups_size":0,"ondisk_log_size":980,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:33:00.072287+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":64225280,"num_objects":980,"num_object_clones":0,"num_object_copies":2940,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":980,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":980,"num_write_kb":62720,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,0],"acting":[6,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.7","version":"124'2","reported_seq":210,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-03-11T21:32:04.318796+0000","last_change":"2024-03-11T21:29:34.957004+0000","last_active":"2024-03-11T21:32:04.318796+0000","last_peered":"2024-03-11T21:32:04.318796+0000","last_clean":"2024-03-11T21:32:04.318796+0000","last_became_active":"2024-03-11T21:25:44.591376+0000","last_became_peered":"2024-03-11T21:25:44.591376+0000","last_unstale":"2024-03-11T21:32:04.318796+0000","last_undegraded":"2024-03-11T21:32:04.318796+0000","last_fullsized":"2024-03-11T21:32:04.318796+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"124'2","last_scrub_stamp":"2024-03-11T21:29:34.956908+0000","last_deep_scrub":"124'2","last_deep_scrub_stamp":"2024-03-11T21:29:34.956908+0000","last_clean_scrub_stamp":"2024-03-11T21:29:34.956908+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":68,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,4],"acting":[5,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2","version":"71'2","reported_seq":238,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-03-11T21:32:04.322982+0000","last_change":"2024-03-11T21:30:19.973353+0000","last_active":"2024-03-11T21:32:04.322982+0000","last_peered":"2024-03-11T21:32:04.322982+0000","last_clean":"2024-03-11T21:32:04.322982+0000","last_became_active":"2024-03-11T21:25:38.539413+0000","last_became_peered":"2024-03-11T21:25:38.539413+0000","last_unstale":"2024-03-11T21:32:04.322982+0000","last_undegraded":"2024-03-11T21:32:04.322982+0000","last_fullsized":"2024-03-11T21:32:04.322982+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"71'2","last_scrub_stamp":"2024-03-11T21:30:19.973188+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:37.512018+0000","last_clean_scrub_stamp":"2024-03-11T21:30:19.973188+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0.0014206449999999999,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":249,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-03-11T21:32:04.318097+0000","last_change":"2024-03-11T21:32:01.617566+0000","last_active":"2024-03-11T21:32:04.318097+0000","last_peered":"2024-03-11T21:32:04.318097+0000","last_clean":"2024-03-11T21:32:04.318097+0000","last_became_active":"2024-03-11T21:31:48.236738+0000","last_became_peered":"2024-03-11T21:31:48.236738+0000","last_unstale":"2024-03-11T21:32:04.318097+0000","last_undegraded":"2024-03-11T21:32:04.318097+0000","last_fullsized":"2024-03-11T21:32:04.318097+0000","mapping_epoch":153,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":154,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:32:01.617459+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:37.512018+0000","last_clean_scrub_stamp":"2024-03-11T21:32:01.617459+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:33:29.445302+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,3],"acting":[5,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"87'128","reported_seq":341,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-03-11T21:32:04.317137+0000","last_change":"2024-03-11T21:31:51.235920+0000","last_active":"2024-03-11T21:32:04.317137+0000","last_peered":"2024-03-11T21:32:04.317137+0000","last_clean":"2024-03-11T21:32:04.317137+0000","last_became_active":"2024-03-11T21:31:51.235503+0000","last_became_peered":"2024-03-11T21:31:51.235503+0000","last_unstale":"2024-03-11T21:32:04.317137+0000","last_undegraded":"2024-03-11T21:32:04.317137+0000","last_fullsized":"2024-03-11T21:32:04.317137+0000","mapping_epoch":156,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":157,"parent":"0.0","parent_split_bits":0,"last_scrub":"87'128","last_scrub_stamp":"2024-03-11T21:30:22.564368+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:24.261974+0000","last_clean_scrub_stamp":"2024-03-11T21:30:22.564368+0000","objects_scrubbed":2,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":198,"num_read_kb":294,"num_write":179,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":2361472,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,1],"acting":[0,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.5","version":"74'16","reported_seq":326,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-03-11T21:32:04.321651+0000","last_change":"2024-03-11T21:30:17.783268+0000","last_active":"2024-03-11T21:32:04.321651+0000","last_peered":"2024-03-11T21:32:04.321651+0000","last_clean":"2024-03-11T21:32:04.321651+0000","last_became_active":"2024-03-11T21:25:40.761457+0000","last_became_peered":"2024-03-11T21:25:40.761457+0000","last_unstale":"2024-03-11T21:32:04.321651+0000","last_undegraded":"2024-03-11T21:32:04.321651+0000","last_fullsized":"2024-03-11T21:32:04.321651+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'16","last_scrub_stamp":"2024-03-11T21:30:17.783007+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:39.525554+0000","last_clean_scrub_stamp":"2024-03-11T21:30:17.783007+0000","objects_scrubbed":8,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":154,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":25,"num_read_kb":15,"num_write":13,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":30,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-03-11T21:32:04.330238+0000","last_change":"2024-03-11T21:31:48.236341+0000","last_active":"2024-03-11T21:32:04.330238+0000","last_peered":"2024-03-11T21:32:04.330238+0000","last_clean":"2024-03-11T21:32:04.330238+0000","last_became_active":"2024-03-11T21:31:48.235988+0000","last_became_peered":"2024-03-11T21:31:48.235988+0000","last_unstale":"2024-03-11T21:32:04.330238+0000","last_undegraded":"2024-03-11T21:32:04.330238+0000","last_fullsized":"2024-03-11T21:32:04.330238+0000","mapping_epoch":153,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":154,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:31:00.607366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:41.544550+0000","last_clean_scrub_stamp":"2024-03-11T21:31:00.607366+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:32:05.069626+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,0],"acting":[2,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"8.a","version":"163'989","reported_seq":1011,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-03-11T21:32:07.166072+0000","last_change":"2024-03-11T21:31:52.272724+0000","last_active":"2024-03-11T21:32:07.166072+0000","last_peered":"2024-03-11T21:32:07.166072+0000","last_clean":"2024-03-11T21:32:07.166072+0000","last_became_active":"2024-03-11T21:31:52.271977+0000","last_became_peered":"2024-03-11T21:31:52.271977+0000","last_unstale":"2024-03-11T21:32:07.166072+0000","last_undegraded":"2024-03-11T21:32:07.166072+0000","last_fullsized":"2024-03-11T21:32:07.166072+0000","mapping_epoch":157,"log_start":"0'0","ondisk_log_start":"0'0","created":157,"last_epoch_clean":158,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:31:51.219688+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:31:51.219688+0000","last_clean_scrub_stamp":"2024-03-11T21:31:51.219688+0000","objects_scrubbed":0,"log_size":989,"log_dups_size":0,"ondisk_log_size":989,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:33:08.449586+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":64815104,"num_objects":989,"num_object_clones":0,"num_object_copies":2967,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":989,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":989,"num_write_kb":63296,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,5],"acting":[1,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.4","version":"0'0","reported_seq":218,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-03-11T21:32:04.325592+0000","last_change":"2024-03-11T21:30:58.758256+0000","last_active":"2024-03-11T21:32:04.325592+0000","last_peered":"2024-03-11T21:32:04.325592+0000","last_clean":"2024-03-11T21:32:04.325592+0000","last_became_active":"2024-03-11T21:25:44.594188+0000","last_became_peered":"2024-03-11T21:25:44.594188+0000","last_unstale":"2024-03-11T21:32:04.325592+0000","last_undegraded":"2024-03-11T21:32:04.325592+0000","last_fullsized":"2024-03-11T21:32:04.325592+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:30:58.757819+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:43.559452+0000","last_clean_scrub_stamp":"2024-03-11T21:30:58.757819+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:32:17.709798+0000","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":237,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-03-11T21:32:04.319724+0000","last_change":"2024-03-11T21:30:22.969948+0000","last_active":"2024-03-11T21:32:04.319724+0000","last_peered":"2024-03-11T21:32:04.319724+0000","last_clean":"2024-03-11T21:32:04.319724+0000","last_became_active":"2024-03-11T21:25:38.543834+0000","last_became_peered":"2024-03-11T21:25:38.543834+0000","last_unstale":"2024-03-11T21:32:04.319724+0000","last_undegraded":"2024-03-11T21:32:04.319724+0000","last_fullsized":"2024-03-11T21:32:04.319724+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:30:22.969840+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:37.512018+0000","last_clean_scrub_stamp":"2024-03-11T21:30:22.969840+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0.00089342499999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":30,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-03-11T21:32:04.329568+0000","last_change":"2024-03-11T21:31:48.235540+0000","last_active":"2024-03-11T21:32:04.329568+0000","last_peered":"2024-03-11T21:32:04.329568+0000","last_clean":"2024-03-11T21:32:04.329568+0000","last_became_active":"2024-03-11T21:31:48.235234+0000","last_became_peered":"2024-03-11T21:31:48.235234+0000","last_unstale":"2024-03-11T21:32:04.329568+0000","last_undegraded":"2024-03-11T21:32:04.329568+0000","last_fullsized":"2024-03-11T21:32:04.329568+0000","mapping_epoch":153,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":154,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:31:06.457393+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:37.512018+0000","last_clean_scrub_stamp":"2024-03-11T21:31:06.457393+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:32:29.582338+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3],"acting":[1,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.6","version":"74'12","reported_seq":243,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-03-11T21:32:04.330525+0000","last_change":"2024-03-11T21:31:04.383837+0000","last_active":"2024-03-11T21:32:04.330525+0000","last_peered":"2024-03-11T21:32:04.330525+0000","last_clean":"2024-03-11T21:32:04.330525+0000","last_became_active":"2024-03-11T21:25:40.550388+0000","last_became_peered":"2024-03-11T21:25:40.550388+0000","last_unstale":"2024-03-11T21:32:04.330525+0000","last_undegraded":"2024-03-11T21:32:04.330525+0000","last_fullsized":"2024-03-11T21:32:04.330525+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'12","last_scrub_stamp":"2024-03-11T21:31:04.383580+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:39.525554+0000","last_clean_scrub_stamp":"2024-03-11T21:31:04.383580+0000","objects_scrubbed":6,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:32:16.235650+0000","scrub_duration":25,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":18,"num_read_kb":12,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":220,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-03-11T21:32:04.318742+0000","last_change":"2024-03-11T21:31:12.634176+0000","last_active":"2024-03-11T21:32:04.318742+0000","last_peered":"2024-03-11T21:32:04.318742+0000","last_clean":"2024-03-11T21:32:04.318742+0000","last_became_active":"2024-03-11T21:25:42.572324+0000","last_became_peered":"2024-03-11T21:25:42.572324+0000","last_unstale":"2024-03-11T21:32:04.318742+0000","last_undegraded":"2024-03-11T21:32:04.318742+0000","last_fullsized":"2024-03-11T21:32:04.318742+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:31:12.634040+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:41.544550+0000","last_clean_scrub_stamp":"2024-03-11T21:31:12.634040+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:32:31.480900+0000","scrub_duration":16,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.f","version":"163'1023","reported_seq":1043,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-03-11T21:32:07.499534+0000","last_change":"2024-03-11T21:31:52.272528+0000","last_active":"2024-03-11T21:32:07.499534+0000","last_peered":"2024-03-11T21:32:07.499534+0000","last_clean":"2024-03-11T21:32:07.499534+0000","last_became_active":"2024-03-11T21:31:52.272191+0000","last_became_peered":"2024-03-11T21:31:52.272191+0000","last_unstale":"2024-03-11T21:32:07.499534+0000","last_undegraded":"2024-03-11T21:32:07.499534+0000","last_fullsized":"2024-03-11T21:32:07.499534+0000","mapping_epoch":157,"log_start":"0'0","ondisk_log_start":"0'0","created":157,"last_epoch_clean":158,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:31:51.219688+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:31:51.219688+0000","last_clean_scrub_stamp":"2024-03-11T21:31:51.219688+0000","objects_scrubbed":0,"log_size":1023,"log_dups_size":0,"ondisk_log_size":1023,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:32:59.655064+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":67043328,"num_objects":1023,"num_object_clones":0,"num_object_copies":3069,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1023,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1023,"num_write_kb":65472,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,1],"acting":[3,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":226,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-03-11T21:32:04.330364+0000","last_change":"2024-03-11T21:31:01.858727+0000","last_active":"2024-03-11T21:32:04.330364+0000","last_peered":"2024-03-11T21:32:04.330364+0000","last_clean":"2024-03-11T21:32:04.330364+0000","last_became_active":"2024-03-11T21:25:45.301416+0000","last_became_peered":"2024-03-11T21:25:45.301416+0000","last_unstale":"2024-03-11T21:32:04.330364+0000","last_undegraded":"2024-03-11T21:32:04.330364+0000","last_fullsized":"2024-03-11T21:32:04.330364+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:31:01.856502+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:43.559452+0000","last_clean_scrub_stamp":"2024-03-11T21:31:01.856502+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:32:30.487243+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":38,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-03-11T21:32:04.324955+0000","last_change":"2024-03-11T21:31:56.381335+0000","last_active":"2024-03-11T21:32:04.324955+0000","last_peered":"2024-03-11T21:32:04.324955+0000","last_clean":"2024-03-11T21:32:04.324955+0000","last_became_active":"2024-03-11T21:31:48.222884+0000","last_became_peered":"2024-03-11T21:31:48.222884+0000","last_unstale":"2024-03-11T21:32:04.324955+0000","last_undegraded":"2024-03-11T21:32:04.324955+0000","last_fullsized":"2024-03-11T21:32:04.324955+0000","mapping_epoch":153,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":154,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:31:56.381231+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:26:57.333669+0000","last_clean_scrub_stamp":"2024-03-11T21:31:56.381231+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:33:03.065697+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,1],"acting":[0,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.3","version":"74'19","reported_seq":286,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-03-11T21:32:04.326879+0000","last_change":"2024-03-11T21:31:48.250641+0000","last_active":"2024-03-11T21:32:04.326879+0000","last_peered":"2024-03-11T21:32:04.326879+0000","last_clean":"2024-03-11T21:32:04.326879+0000","last_became_active":"2024-03-11T21:31:48.223109+0000","last_became_peered":"2024-03-11T21:31:48.223109+0000","last_unstale":"2024-03-11T21:32:04.326879+0000","last_undegraded":"2024-03-11T21:32:04.326879+0000","last_fullsized":"2024-03-11T21:32:04.326879+0000","mapping_epoch":153,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":154,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'19","last_scrub_stamp":"2024-03-11T21:30:23.995503+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:39.525554+0000","last_clean_scrub_stamp":"2024-03-11T21:30:23.995503+0000","objects_scrubbed":8,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":39,"num_read_kb":25,"num_write":22,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,1],"acting":[0,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":230,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-03-11T21:32:04.320943+0000","last_change":"2024-03-11T21:31:11.390780+0000","last_active":"2024-03-11T21:32:04.320943+0000","last_peered":"2024-03-11T21:32:04.320943+0000","last_clean":"2024-03-11T21:32:04.320943+0000","last_became_active":"2024-03-11T21:25:42.579045+0000","last_became_peered":"2024-03-11T21:25:42.579045+0000","last_unstale":"2024-03-11T21:32:04.320943+0000","last_undegraded":"2024-03-11T21:32:04.320943+0000","last_fullsized":"2024-03-11T21:32:04.320943+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:31:11.390707+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:29:36.918936+0000","last_clean_scrub_stamp":"2024-03-11T21:31:11.390707+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:32:13.656422+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"8.c","version":"163'1040","reported_seq":1062,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-03-11T21:32:07.849525+0000","last_change":"2024-03-11T21:31:52.240141+0000","last_active":"2024-03-11T21:32:07.849525+0000","last_peered":"2024-03-11T21:32:07.849525+0000","last_clean":"2024-03-11T21:32:07.849525+0000","last_became_active":"2024-03-11T21:31:52.239816+0000","last_became_peered":"2024-03-11T21:31:52.239816+0000","last_unstale":"2024-03-11T21:32:07.849525+0000","last_undegraded":"2024-03-11T21:32:07.849525+0000","last_fullsized":"2024-03-11T21:32:07.849525+0000","mapping_epoch":157,"log_start":"0'0","ondisk_log_start":"0'0","created":157,"last_epoch_clean":158,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:31:51.219688+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:31:51.219688+0000","last_clean_scrub_stamp":"2024-03-11T21:31:51.219688+0000","objects_scrubbed":0,"log_size":1040,"log_dups_size":0,"ondisk_log_size":1040,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:33:10.290532+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":68157440,"num_objects":1040,"num_object_clones":0,"num_object_copies":3120,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1040,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1040,"num_write_kb":66560,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,5],"acting":[0,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":226,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-03-11T21:32:05.636310+0000","last_change":"2024-03-11T21:32:05.636351+0000","last_active":"2024-03-11T21:32:05.636310+0000","last_peered":"2024-03-11T21:32:05.636310+0000","last_clean":"2024-03-11T21:32:05.636310+0000","last_became_active":"2024-03-11T21:25:44.583638+0000","last_became_peered":"2024-03-11T21:25:44.583638+0000","last_unstale":"2024-03-11T21:32:05.636310+0000","last_undegraded":"2024-03-11T21:32:05.636310+0000","last_fullsized":"2024-03-11T21:32:05.636310+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:32:05.636174+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:43.559452+0000","last_clean_scrub_stamp":"2024-03-11T21:32:05.636174+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:33:33.012723+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.6","version":"67'1","reported_seq":239,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-03-11T21:32:05.556583+0000","last_change":"2024-03-11T21:32:05.556583+0000","last_active":"2024-03-11T21:32:05.556583+0000","last_peered":"2024-03-11T21:32:05.556583+0000","last_clean":"2024-03-11T21:32:05.556583+0000","last_became_active":"2024-03-11T21:25:38.536142+0000","last_became_peered":"2024-03-11T21:25:38.536142+0000","last_unstale":"2024-03-11T21:32:05.556583+0000","last_undegraded":"2024-03-11T21:32:05.556583+0000","last_fullsized":"2024-03-11T21:32:05.556583+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'1","last_scrub_stamp":"2024-03-11T21:32:05.556427+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:37.512018+0000","last_clean_scrub_stamp":"2024-03-11T21:32:05.556427+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:33:11.248441+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,4],"acting":[1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.0","version":"93'26","reported_seq":289,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-03-11T21:32:04.322911+0000","last_change":"2024-03-11T21:31:48.258922+0000","last_active":"2024-03-11T21:32:04.322911+0000","last_peered":"2024-03-11T21:32:04.322911+0000","last_clean":"2024-03-11T21:32:04.322911+0000","last_became_active":"2024-03-11T21:31:48.229925+0000","last_became_peered":"2024-03-11T21:31:48.229925+0000","last_unstale":"2024-03-11T21:32:04.322911+0000","last_undegraded":"2024-03-11T21:32:04.322911+0000","last_fullsized":"2024-03-11T21:32:04.322911+0000","mapping_epoch":153,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":154,"parent":"0.0","parent_split_bits":0,"last_scrub":"93'26","last_scrub_stamp":"2024-03-11T21:31:03.244599+0000","last_deep_scrub":"93'26","last_deep_scrub_stamp":"2024-03-11T21:26:40.680324+0000","last_clean_scrub_stamp":"2024-03-11T21:31:03.244599+0000","objects_scrubbed":8,"log_size":26,"log_dups_size":0,"ondisk_log_size":26,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:32:16.690320+0000","scrub_duration":19,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":34,"num_read_kb":22,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":228,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-03-11T21:32:04.315213+0000","last_change":"2024-03-11T21:31:48.227115+0000","last_active":"2024-03-11T21:32:04.315213+0000","last_peered":"2024-03-11T21:32:04.315213+0000","last_clean":"2024-03-11T21:32:04.315213+0000","last_became_active":"2024-03-11T21:31:48.226442+0000","last_became_peered":"2024-03-11T21:31:48.226442+0000","last_unstale":"2024-03-11T21:32:04.315213+0000","last_undegraded":"2024-03-11T21:32:04.315213+0000","last_fullsized":"2024-03-11T21:32:04.315213+0000","mapping_epoch":153,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":154,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:30:08.277733+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:26:51.863621+0000","last_clean_scrub_stamp":"2024-03-11T21:30:08.277733+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0],"acting":[4,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"8.d","version":"163'1013","reported_seq":1035,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-03-11T21:32:07.006730+0000","last_change":"2024-03-11T21:31:52.272819+0000","last_active":"2024-03-11T21:32:07.006730+0000","last_peered":"2024-03-11T21:32:07.006730+0000","last_clean":"2024-03-11T21:32:07.006730+0000","last_became_active":"2024-03-11T21:31:52.272281+0000","last_became_peered":"2024-03-11T21:31:52.272281+0000","last_unstale":"2024-03-11T21:32:07.006730+0000","last_undegraded":"2024-03-11T21:32:07.006730+0000","last_fullsized":"2024-03-11T21:32:07.006730+0000","mapping_epoch":157,"log_start":"0'0","ondisk_log_start":"0'0","created":157,"last_epoch_clean":158,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:31:51.219688+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:31:51.219688+0000","last_clean_scrub_stamp":"2024-03-11T21:31:51.219688+0000","objects_scrubbed":0,"log_size":1013,"log_dups_size":0,"ondisk_log_size":1013,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:32:52.405338+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":66387968,"num_objects":1013,"num_object_clones":0,"num_object_copies":3039,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1013,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1013,"num_write_kb":64832,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,6],"acting":[5,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.3","version":"0'0","reported_seq":38,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-03-11T21:32:04.321227+0000","last_change":"2024-03-11T21:31:54.671396+0000","last_active":"2024-03-11T21:32:04.321227+0000","last_peered":"2024-03-11T21:32:04.321227+0000","last_clean":"2024-03-11T21:32:04.321227+0000","last_became_active":"2024-03-11T21:31:48.220482+0000","last_became_peered":"2024-03-11T21:31:48.220482+0000","last_unstale":"2024-03-11T21:32:04.321227+0000","last_undegraded":"2024-03-11T21:32:04.321227+0000","last_fullsized":"2024-03-11T21:32:04.321227+0000","mapping_epoch":153,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":154,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:31:54.671300+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:29:42.485740+0000","last_clean_scrub_stamp":"2024-03-11T21:31:54.671300+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:33:20.587123+0000","scrub_duration":16,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,1],"acting":[6,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":243,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-03-11T21:32:04.321549+0000","last_change":"2024-03-11T21:31:48.236203+0000","last_active":"2024-03-11T21:32:04.321549+0000","last_peered":"2024-03-11T21:32:04.321549+0000","last_clean":"2024-03-11T21:32:04.321549+0000","last_became_active":"2024-03-11T21:31:48.235815+0000","last_became_peered":"2024-03-11T21:31:48.235815+0000","last_unstale":"2024-03-11T21:32:04.321549+0000","last_undegraded":"2024-03-11T21:32:04.321549+0000","last_fullsized":"2024-03-11T21:32:04.321549+0000","mapping_epoch":153,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":154,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:31:01.530407+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:37.512018+0000","last_clean_scrub_stamp":"2024-03-11T21:31:01.530407+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:32:22.979366+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,3],"acting":[6,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.1","version":"74'14","reported_seq":247,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-03-11T21:32:04.315554+0000","last_change":"2024-03-11T21:31:02.486605+0000","last_active":"2024-03-11T21:32:04.315554+0000","last_peered":"2024-03-11T21:32:04.315554+0000","last_clean":"2024-03-11T21:32:04.315554+0000","last_became_active":"2024-03-11T21:25:40.546219+0000","last_became_peered":"2024-03-11T21:25:40.546219+0000","last_unstale":"2024-03-11T21:32:04.315554+0000","last_undegraded":"2024-03-11T21:32:04.315554+0000","last_fullsized":"2024-03-11T21:32:04.315554+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'14","last_scrub_stamp":"2024-03-11T21:31:02.486457+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:39.525554+0000","last_clean_scrub_stamp":"2024-03-11T21:31:02.486457+0000","objects_scrubbed":7,"log_size":14,"log_dups_size":0,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:32:07.493431+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":21,"num_read_kb":14,"num_write":14,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6],"acting":[4,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.0","version":"98'9","reported_seq":308,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-03-11T21:32:04.323012+0000","last_change":"2024-03-11T21:31:21.562516+0000","last_active":"2024-03-11T21:32:04.323012+0000","last_peered":"2024-03-11T21:32:04.323012+0000","last_clean":"2024-03-11T21:32:04.323012+0000","last_became_active":"2024-03-11T21:25:42.576126+0000","last_became_peered":"2024-03-11T21:25:42.576126+0000","last_unstale":"2024-03-11T21:32:04.323012+0000","last_undegraded":"2024-03-11T21:32:04.323012+0000","last_fullsized":"2024-03-11T21:32:04.323012+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"98'9","last_scrub_stamp":"2024-03-11T21:31:21.562395+0000","last_deep_scrub":"98'9","last_deep_scrub_stamp":"2024-03-11T21:27:29.745711+0000","last_clean_scrub_stamp":"2024-03-11T21:31:21.562395+0000","objects_scrubbed":1,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:32:31.326990+0000","scrub_duration":17,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.2","version":"163'996","reported_seq":1017,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-03-11T21:32:07.025774+0000","last_change":"2024-03-11T21:31:52.237340+0000","last_active":"2024-03-11T21:32:07.025774+0000","last_peered":"2024-03-11T21:32:07.025774+0000","last_clean":"2024-03-11T21:32:07.025774+0000","last_became_active":"2024-03-11T21:31:52.236955+0000","last_became_peered":"2024-03-11T21:31:52.236955+0000","last_unstale":"2024-03-11T21:32:07.025774+0000","last_undegraded":"2024-03-11T21:32:07.025774+0000","last_fullsized":"2024-03-11T21:32:07.025774+0000","mapping_epoch":157,"log_start":"0'0","ondisk_log_start":"0'0","created":157,"last_epoch_clean":158,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:31:51.219688+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:31:51.219688+0000","last_clean_scrub_stamp":"2024-03-11T21:31:51.219688+0000","objects_scrubbed":0,"log_size":996,"log_dups_size":0,"ondisk_log_size":996,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:33:03.531418+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":65273856,"num_objects":996,"num_object_clones":0,"num_object_copies":2988,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":996,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":996,"num_write_kb":63744,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,5],"acting":[4,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.c","version":"0'0","reported_seq":208,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-03-11T21:32:04.325460+0000","last_change":"2024-03-11T21:30:23.986312+0000","last_active":"2024-03-11T21:32:04.325460+0000","last_peered":"2024-03-11T21:32:04.325460+0000","last_clean":"2024-03-11T21:32:04.325460+0000","last_became_active":"2024-03-11T21:25:45.302207+0000","last_became_peered":"2024-03-11T21:25:45.302207+0000","last_unstale":"2024-03-11T21:32:04.325460+0000","last_undegraded":"2024-03-11T21:32:04.325460+0000","last_fullsized":"2024-03-11T21:32:04.325460+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:30:23.986191+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:43.559452+0000","last_clean_scrub_stamp":"2024-03-11T21:30:23.986191+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.f","version":"0'0","reported_seq":228,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-03-11T21:32:04.318608+0000","last_change":"2024-03-11T21:31:24.574334+0000","last_active":"2024-03-11T21:32:04.318608+0000","last_peered":"2024-03-11T21:32:04.318608+0000","last_clean":"2024-03-11T21:32:04.318608+0000","last_became_active":"2024-03-11T21:25:42.575935+0000","last_became_peered":"2024-03-11T21:25:42.575935+0000","last_unstale":"2024-03-11T21:32:04.318608+0000","last_undegraded":"2024-03-11T21:32:04.318608+0000","last_fullsized":"2024-03-11T21:32:04.318608+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:31:24.574239+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:41.544550+0000","last_clean_scrub_stamp":"2024-03-11T21:31:24.574239+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:32:33.744328+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.8","version":"0'0","reported_seq":38,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-03-11T21:32:06.392103+0000","last_change":"2024-03-11T21:32:06.392141+0000","last_active":"2024-03-11T21:32:06.392103+0000","last_peered":"2024-03-11T21:32:06.392103+0000","last_clean":"2024-03-11T21:32:06.392103+0000","last_became_active":"2024-03-11T21:31:48.222826+0000","last_became_peered":"2024-03-11T21:31:48.222826+0000","last_unstale":"2024-03-11T21:32:06.392103+0000","last_undegraded":"2024-03-11T21:32:06.392103+0000","last_fullsized":"2024-03-11T21:32:06.392103+0000","mapping_epoch":153,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":154,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:32:06.392027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:37.512018+0000","last_clean_scrub_stamp":"2024-03-11T21:32:06.392027+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:33:08.153624+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.e","version":"109'16","reported_seq":249,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-03-11T21:32:04.315435+0000","last_change":"2024-03-11T21:30:56.453809+0000","last_active":"2024-03-11T21:32:04.315435+0000","last_peered":"2024-03-11T21:32:04.315435+0000","last_clean":"2024-03-11T21:32:04.315435+0000","last_became_active":"2024-03-11T21:25:40.552178+0000","last_became_peered":"2024-03-11T21:25:40.552178+0000","last_unstale":"2024-03-11T21:32:04.315435+0000","last_undegraded":"2024-03-11T21:32:04.315435+0000","last_fullsized":"2024-03-11T21:32:04.315435+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"109'16","last_scrub_stamp":"2024-03-11T21:30:56.453670+0000","last_deep_scrub":"109'16","last_deep_scrub_stamp":"2024-03-11T21:28:24.739395+0000","last_clean_scrub_stamp":"2024-03-11T21:30:56.453670+0000","objects_scrubbed":5,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"8.3","version":"163'961","reported_seq":983,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-03-11T21:32:07.169050+0000","last_change":"2024-03-11T21:31:52.237339+0000","last_active":"2024-03-11T21:32:07.169050+0000","last_peered":"2024-03-11T21:32:07.169050+0000","last_clean":"2024-03-11T21:32:07.169050+0000","last_became_active":"2024-03-11T21:31:52.237034+0000","last_became_peered":"2024-03-11T21:31:52.237034+0000","last_unstale":"2024-03-11T21:32:07.169050+0000","last_undegraded":"2024-03-11T21:32:07.169050+0000","last_fullsized":"2024-03-11T21:32:07.169050+0000","mapping_epoch":157,"log_start":"0'0","ondisk_log_start":"0'0","created":157,"last_epoch_clean":158,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:31:51.219688+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:31:51.219688+0000","last_clean_scrub_stamp":"2024-03-11T21:31:51.219688+0000","objects_scrubbed":0,"log_size":961,"log_dups_size":0,"ondisk_log_size":961,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:32:59.838253+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":62980096,"num_objects":961,"num_object_clones":0,"num_object_copies":2883,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":961,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":961,"num_write_kb":61504,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,6],"acting":[1,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.d","version":"0'0","reported_seq":216,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-03-11T21:32:04.318574+0000","last_change":"2024-03-11T21:30:11.993990+0000","last_active":"2024-03-11T21:32:04.318574+0000","last_peered":"2024-03-11T21:32:04.318574+0000","last_clean":"2024-03-11T21:32:04.318574+0000","last_became_active":"2024-03-11T21:25:44.590333+0000","last_became_peered":"2024-03-11T21:25:44.590333+0000","last_unstale":"2024-03-11T21:32:04.318574+0000","last_undegraded":"2024-03-11T21:32:04.318574+0000","last_fullsized":"2024-03-11T21:32:04.318574+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:30:11.993862+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:43.559452+0000","last_clean_scrub_stamp":"2024-03-11T21:30:11.993862+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.e","version":"74'8","reported_seq":307,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-03-11T21:32:04.315874+0000","last_change":"2024-03-11T21:31:05.564402+0000","last_active":"2024-03-11T21:32:04.315874+0000","last_peered":"2024-03-11T21:32:04.315874+0000","last_clean":"2024-03-11T21:32:04.315874+0000","last_became_active":"2024-03-11T21:25:42.576742+0000","last_became_peered":"2024-03-11T21:25:42.576742+0000","last_unstale":"2024-03-11T21:32:04.315874+0000","last_undegraded":"2024-03-11T21:32:04.315874+0000","last_fullsized":"2024-03-11T21:32:04.315874+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'8","last_scrub_stamp":"2024-03-11T21:31:05.564320+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:41.544550+0000","last_clean_scrub_stamp":"2024-03-11T21:31:05.564320+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:32:33.999249+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,0],"acting":[4,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.9","version":"0'0","reported_seq":244,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-03-11T21:32:04.330224+0000","last_change":"2024-03-11T21:32:02.507023+0000","last_active":"2024-03-11T21:32:04.330224+0000","last_peered":"2024-03-11T21:32:04.330224+0000","last_clean":"2024-03-11T21:32:04.330224+0000","last_became_active":"2024-03-11T21:31:48.234424+0000","last_became_peered":"2024-03-11T21:31:48.234424+0000","last_unstale":"2024-03-11T21:32:04.330224+0000","last_undegraded":"2024-03-11T21:32:04.330224+0000","last_fullsized":"2024-03-11T21:32:04.330224+0000","mapping_epoch":153,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":154,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:32:02.506579+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:26:50.991712+0000","last_clean_scrub_stamp":"2024-03-11T21:32:02.506579+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:33:11.511126+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.f","version":"74'15","reported_seq":210,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-03-11T21:32:04.332153+0000","last_change":"2024-03-11T21:32:02.432661+0000","last_active":"2024-03-11T21:32:04.332153+0000","last_peered":"2024-03-11T21:32:04.332153+0000","last_clean":"2024-03-11T21:32:04.332153+0000","last_became_active":"2024-03-11T21:26:11.216224+0000","last_became_peered":"2024-03-11T21:26:11.216224+0000","last_unstale":"2024-03-11T21:32:04.332153+0000","last_undegraded":"2024-03-11T21:32:04.332153+0000","last_fullsized":"2024-03-11T21:32:04.332153+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'15","last_scrub_stamp":"2024-03-11T21:32:02.432529+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:39.525554+0000","last_clean_scrub_stamp":"2024-03-11T21:32:02.432529+0000","objects_scrubbed":7,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:33:02.536334+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,4],"acting":[2,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"8.0","version":"163'1052","reported_seq":1074,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-03-11T21:32:07.485327+0000","last_change":"2024-03-11T21:31:52.239973+0000","last_active":"2024-03-11T21:32:07.485327+0000","last_peered":"2024-03-11T21:32:07.485327+0000","last_clean":"2024-03-11T21:32:07.485327+0000","last_became_active":"2024-03-11T21:31:52.239611+0000","last_became_peered":"2024-03-11T21:31:52.239611+0000","last_unstale":"2024-03-11T21:32:07.485327+0000","last_undegraded":"2024-03-11T21:32:07.485327+0000","last_fullsized":"2024-03-11T21:32:07.485327+0000","mapping_epoch":157,"log_start":"0'0","ondisk_log_start":"0'0","created":157,"last_epoch_clean":158,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:31:51.219688+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:31:51.219688+0000","last_clean_scrub_stamp":"2024-03-11T21:31:51.219688+0000","objects_scrubbed":0,"log_size":1052,"log_dups_size":0,"ondisk_log_size":1052,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:32:53.694197+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":68943872,"num_objects":1052,"num_object_clones":0,"num_object_copies":3156,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1052,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1052,"num_write_kb":67328,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,1],"acting":[3,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.e","version":"0'0","reported_seq":217,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-03-11T21:32:04.315907+0000","last_change":"2024-03-11T21:31:09.597590+0000","last_active":"2024-03-11T21:32:04.315907+0000","last_peered":"2024-03-11T21:32:04.315907+0000","last_clean":"2024-03-11T21:32:04.315907+0000","last_became_active":"2024-03-11T21:25:44.590198+0000","last_became_peered":"2024-03-11T21:25:44.590198+0000","last_unstale":"2024-03-11T21:32:04.315907+0000","last_undegraded":"2024-03-11T21:32:04.315907+0000","last_fullsized":"2024-03-11T21:32:04.315907+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:31:09.597448+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:43.559452+0000","last_clean_scrub_stamp":"2024-03-11T21:31:09.597448+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:32:30.206261+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.d","version":"109'9","reported_seq":307,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-03-11T21:32:04.331329+0000","last_change":"2024-03-11T21:30:20.940788+0000","last_active":"2024-03-11T21:32:04.331329+0000","last_peered":"2024-03-11T21:32:04.331329+0000","last_clean":"2024-03-11T21:32:04.331329+0000","last_became_active":"2024-03-11T21:25:42.575583+0000","last_became_peered":"2024-03-11T21:25:42.575583+0000","last_unstale":"2024-03-11T21:32:04.331329+0000","last_undegraded":"2024-03-11T21:32:04.331329+0000","last_fullsized":"2024-03-11T21:32:04.331329+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"109'9","last_scrub_stamp":"2024-03-11T21:30:20.940485+0000","last_deep_scrub":"109'9","last_deep_scrub_stamp":"2024-03-11T21:28:27.082975+0000","last_clean_scrub_stamp":"2024-03-11T21:30:20.940485+0000","objects_scrubbed":1,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,5],"acting":[2,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.a","version":"0'0","reported_seq":243,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-03-11T21:32:04.330515+0000","last_change":"2024-03-11T21:31:48.235649+0000","last_active":"2024-03-11T21:32:04.330515+0000","last_peered":"2024-03-11T21:32:04.330515+0000","last_clean":"2024-03-11T21:32:04.330515+0000","last_became_active":"2024-03-11T21:31:48.235272+0000","last_became_peered":"2024-03-11T21:31:48.235272+0000","last_unstale":"2024-03-11T21:32:04.330515+0000","last_undegraded":"2024-03-11T21:32:04.330515+0000","last_fullsized":"2024-03-11T21:32:04.330515+0000","mapping_epoch":153,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":154,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:31:06.898394+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:37.512018+0000","last_clean_scrub_stamp":"2024-03-11T21:31:06.898394+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:32:16.716524+0000","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,5],"acting":[1,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.c","version":"74'10","reported_seq":240,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-03-11T21:32:04.315724+0000","last_change":"2024-03-11T21:30:57.495457+0000","last_active":"2024-03-11T21:32:04.315724+0000","last_peered":"2024-03-11T21:32:04.315724+0000","last_clean":"2024-03-11T21:32:04.315724+0000","last_became_active":"2024-03-11T21:25:40.550885+0000","last_became_peered":"2024-03-11T21:25:40.550885+0000","last_unstale":"2024-03-11T21:32:04.315724+0000","last_undegraded":"2024-03-11T21:32:04.315724+0000","last_fullsized":"2024-03-11T21:32:04.315724+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'10","last_scrub_stamp":"2024-03-11T21:30:57.495302+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:39.525554+0000","last_clean_scrub_stamp":"2024-03-11T21:30:57.495302+0000","objects_scrubbed":5,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:32:04.367621+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"8.1","version":"163'974","reported_seq":996,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-03-11T21:32:07.170563+0000","last_change":"2024-03-11T21:31:52.272438+0000","last_active":"2024-03-11T21:32:07.170563+0000","last_peered":"2024-03-11T21:32:07.170563+0000","last_clean":"2024-03-11T21:32:07.170563+0000","last_became_active":"2024-03-11T21:31:52.272076+0000","last_became_peered":"2024-03-11T21:31:52.272076+0000","last_unstale":"2024-03-11T21:32:07.170563+0000","last_undegraded":"2024-03-11T21:32:07.170563+0000","last_fullsized":"2024-03-11T21:32:07.170563+0000","mapping_epoch":157,"log_start":"0'0","ondisk_log_start":"0'0","created":157,"last_epoch_clean":158,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:31:51.219688+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:31:51.219688+0000","last_clean_scrub_stamp":"2024-03-11T21:31:51.219688+0000","objects_scrubbed":0,"log_size":974,"log_dups_size":0,"ondisk_log_size":974,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:33:03.738056+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":63832064,"num_objects":974,"num_object_clones":0,"num_object_copies":2922,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":974,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":974,"num_write_kb":62336,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.f","version":"0'0","reported_seq":216,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-03-11T21:32:04.331997+0000","last_change":"2024-03-11T21:30:21.979887+0000","last_active":"2024-03-11T21:32:04.331997+0000","last_peered":"2024-03-11T21:32:04.331997+0000","last_clean":"2024-03-11T21:32:04.331997+0000","last_became_active":"2024-03-11T21:25:44.593414+0000","last_became_peered":"2024-03-11T21:25:44.593414+0000","last_unstale":"2024-03-11T21:32:04.331997+0000","last_undegraded":"2024-03-11T21:32:04.331997+0000","last_fullsized":"2024-03-11T21:32:04.331997+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:30:21.979536+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:27:03.010969+0000","last_clean_scrub_stamp":"2024-03-11T21:30:21.979536+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,4],"acting":[2,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":221,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-03-11T21:32:04.325479+0000","last_change":"2024-03-11T21:30:22.750174+0000","last_active":"2024-03-11T21:32:04.325479+0000","last_peered":"2024-03-11T21:32:04.325479+0000","last_clean":"2024-03-11T21:32:04.325479+0000","last_became_active":"2024-03-11T21:25:42.579409+0000","last_became_peered":"2024-03-11T21:25:42.579409+0000","last_unstale":"2024-03-11T21:32:04.325479+0000","last_undegraded":"2024-03-11T21:32:04.325479+0000","last_fullsized":"2024-03-11T21:32:04.325479+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:30:22.750053+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:30:22.750053+0000","last_clean_scrub_stamp":"2024-03-11T21:30:22.750053+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.b","version":"0'0","reported_seq":30,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-03-11T21:32:04.316157+0000","last_change":"2024-03-11T21:31:48.229615+0000","last_active":"2024-03-11T21:32:04.316157+0000","last_peered":"2024-03-11T21:32:04.316157+0000","last_clean":"2024-03-11T21:32:04.316157+0000","last_became_active":"2024-03-11T21:31:48.228437+0000","last_became_peered":"2024-03-11T21:31:48.228437+0000","last_unstale":"2024-03-11T21:32:04.316157+0000","last_undegraded":"2024-03-11T21:32:04.316157+0000","last_fullsized":"2024-03-11T21:32:04.316157+0000","mapping_epoch":153,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":154,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:31:20.539243+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:37.512018+0000","last_clean_scrub_stamp":"2024-03-11T21:31:20.539243+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:32:39.183195+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,3],"acting":[4,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.d","version":"74'17","reported_seq":253,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-03-11T21:32:04.315489+0000","last_change":"2024-03-11T21:30:20.287636+0000","last_active":"2024-03-11T21:32:04.315489+0000","last_peered":"2024-03-11T21:32:04.315489+0000","last_clean":"2024-03-11T21:32:04.315489+0000","last_became_active":"2024-03-11T21:25:40.553454+0000","last_became_peered":"2024-03-11T21:25:40.553454+0000","last_unstale":"2024-03-11T21:32:04.315489+0000","last_undegraded":"2024-03-11T21:32:04.315489+0000","last_fullsized":"2024-03-11T21:32:04.315489+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'17","last_scrub_stamp":"2024-03-11T21:30:20.287535+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:39.525554+0000","last_clean_scrub_stamp":"2024-03-11T21:30:20.287535+0000","objects_scrubbed":8,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":29,"num_read_kb":19,"num_write":18,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,1],"acting":[4,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"8.6","version":"163'1115","reported_seq":1137,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-03-11T21:32:07.855825+0000","last_change":"2024-03-11T21:31:52.238026+0000","last_active":"2024-03-11T21:32:07.855825+0000","last_peered":"2024-03-11T21:32:07.855825+0000","last_clean":"2024-03-11T21:32:07.855825+0000","last_became_active":"2024-03-11T21:31:52.237725+0000","last_became_peered":"2024-03-11T21:31:52.237725+0000","last_unstale":"2024-03-11T21:32:07.855825+0000","last_undegraded":"2024-03-11T21:32:07.855825+0000","last_fullsized":"2024-03-11T21:32:07.855825+0000","mapping_epoch":157,"log_start":"0'0","ondisk_log_start":"0'0","created":157,"last_epoch_clean":158,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:31:51.219688+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:31:51.219688+0000","last_clean_scrub_stamp":"2024-03-11T21:31:51.219688+0000","objects_scrubbed":0,"log_size":1115,"log_dups_size":0,"ondisk_log_size":1115,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:33:04.247215+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":73072640,"num_objects":1115,"num_object_clones":0,"num_object_copies":3345,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1115,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1115,"num_write_kb":71360,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,7],"acting":[0,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":38,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-03-11T21:32:04.329952+0000","last_change":"2024-03-11T21:31:58.450281+0000","last_active":"2024-03-11T21:32:04.329952+0000","last_peered":"2024-03-11T21:32:04.329952+0000","last_clean":"2024-03-11T21:32:04.329952+0000","last_became_active":"2024-03-11T21:31:48.226356+0000","last_became_peered":"2024-03-11T21:31:48.226356+0000","last_unstale":"2024-03-11T21:32:04.329952+0000","last_undegraded":"2024-03-11T21:32:04.329952+0000","last_fullsized":"2024-03-11T21:32:04.329952+0000","mapping_epoch":153,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":154,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:31:58.450208+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:28:37.561826+0000","last_clean_scrub_stamp":"2024-03-11T21:31:58.450208+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:33:12.115297+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,6],"acting":[2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":220,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-03-11T21:32:04.329834+0000","last_change":"2024-03-11T21:30:57.336406+0000","last_active":"2024-03-11T21:32:04.329834+0000","last_peered":"2024-03-11T21:32:04.329834+0000","last_clean":"2024-03-11T21:32:04.329834+0000","last_became_active":"2024-03-11T21:25:42.574515+0000","last_became_peered":"2024-03-11T21:25:42.574515+0000","last_unstale":"2024-03-11T21:32:04.329834+0000","last_undegraded":"2024-03-11T21:32:04.329834+0000","last_fullsized":"2024-03-11T21:32:04.329834+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:30:57.336188+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:41.544550+0000","last_clean_scrub_stamp":"2024-03-11T21:30:57.336188+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:32:12.876558+0000","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,5],"acting":[2,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.c","version":"0'0","reported_seq":228,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-03-11T21:32:04.330423+0000","last_change":"2024-03-11T21:31:09.381415+0000","last_active":"2024-03-11T21:32:04.330423+0000","last_peered":"2024-03-11T21:32:04.330423+0000","last_clean":"2024-03-11T21:32:04.330423+0000","last_became_active":"2024-03-11T21:25:38.533240+0000","last_became_peered":"2024-03-11T21:25:38.533240+0000","last_unstale":"2024-03-11T21:32:04.330423+0000","last_undegraded":"2024-03-11T21:32:04.330423+0000","last_fullsized":"2024-03-11T21:32:04.330423+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:31:09.381301+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:29:30.279910+0000","last_clean_scrub_stamp":"2024-03-11T21:31:09.381301+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:32:38.869288+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,0],"acting":[2,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.a","version":"109'28","reported_seq":299,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-03-11T21:32:04.320842+0000","last_change":"2024-03-11T21:31:48.257526+0000","last_active":"2024-03-11T21:32:04.320842+0000","last_peered":"2024-03-11T21:32:04.320842+0000","last_clean":"2024-03-11T21:32:04.320842+0000","last_became_active":"2024-03-11T21:31:48.226789+0000","last_became_peered":"2024-03-11T21:31:48.226789+0000","last_unstale":"2024-03-11T21:32:04.320842+0000","last_undegraded":"2024-03-11T21:32:04.320842+0000","last_fullsized":"2024-03-11T21:32:04.320842+0000","mapping_epoch":153,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":154,"parent":"0.0","parent_split_bits":0,"last_scrub":"109'28","last_scrub_stamp":"2024-03-11T21:31:22.522676+0000","last_deep_scrub":"109'28","last_deep_scrub_stamp":"2024-03-11T21:28:24.016389+0000","last_clean_scrub_stamp":"2024-03-11T21:31:22.522676+0000","objects_scrubbed":9,"log_size":28,"log_dups_size":0,"ondisk_log_size":28,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:32:25.322562+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":32,"num_read_kb":21,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,5],"acting":[6,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"8.7","version":"163'936","reported_seq":957,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-03-11T21:32:07.175367+0000","last_change":"2024-03-11T21:31:52.240303+0000","last_active":"2024-03-11T21:32:07.175367+0000","last_peered":"2024-03-11T21:32:07.175367+0000","last_clean":"2024-03-11T21:32:07.175367+0000","last_became_active":"2024-03-11T21:31:52.239980+0000","last_became_peered":"2024-03-11T21:31:52.239980+0000","last_unstale":"2024-03-11T21:32:07.175367+0000","last_undegraded":"2024-03-11T21:32:07.175367+0000","last_fullsized":"2024-03-11T21:32:07.175367+0000","mapping_epoch":157,"log_start":"0'0","ondisk_log_start":"0'0","created":157,"last_epoch_clean":158,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:31:51.219688+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:31:51.219688+0000","last_clean_scrub_stamp":"2024-03-11T21:31:51.219688+0000","objects_scrubbed":0,"log_size":936,"log_dups_size":0,"ondisk_log_size":936,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:32:57.846310+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":61341696,"num_objects":936,"num_object_clones":0,"num_object_copies":2808,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":936,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":936,"num_write_kb":59904,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,6],"acting":[1,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":223,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-03-11T21:32:04.330615+0000","last_change":"2024-03-11T21:31:48.217329+0000","last_active":"2024-03-11T21:32:04.330615+0000","last_peered":"2024-03-11T21:32:04.330615+0000","last_clean":"2024-03-11T21:32:04.330615+0000","last_became_active":"2024-03-11T21:31:48.216970+0000","last_became_peered":"2024-03-11T21:31:48.216970+0000","last_unstale":"2024-03-11T21:32:04.330615+0000","last_undegraded":"2024-03-11T21:32:04.330615+0000","last_fullsized":"2024-03-11T21:32:04.330615+0000","mapping_epoch":153,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":154,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:31:10.397515+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:27:11.361913+0000","last_clean_scrub_stamp":"2024-03-11T21:31:10.397515+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:32:39.196135+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,3],"acting":[0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":220,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-03-11T21:32:04.331847+0000","last_change":"2024-03-11T21:31:11.390022+0000","last_active":"2024-03-11T21:32:04.331847+0000","last_peered":"2024-03-11T21:32:04.331847+0000","last_clean":"2024-03-11T21:32:04.331847+0000","last_became_active":"2024-03-11T21:25:42.576679+0000","last_became_peered":"2024-03-11T21:25:42.576679+0000","last_unstale":"2024-03-11T21:32:04.331847+0000","last_undegraded":"2024-03-11T21:32:04.331847+0000","last_fullsized":"2024-03-11T21:32:04.331847+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:31:11.389869+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:41.544550+0000","last_clean_scrub_stamp":"2024-03-11T21:31:11.389869+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:32:39.283658+0000","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,3],"acting":[2,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.d","version":"0'0","reported_seq":237,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-03-11T21:32:04.327227+0000","last_change":"2024-03-11T21:31:55.464241+0000","last_active":"2024-03-11T21:32:04.327227+0000","last_peered":"2024-03-11T21:32:04.327227+0000","last_clean":"2024-03-11T21:32:04.327227+0000","last_became_active":"2024-03-11T21:25:38.539681+0000","last_became_peered":"2024-03-11T21:25:38.539681+0000","last_unstale":"2024-03-11T21:32:04.327227+0000","last_undegraded":"2024-03-11T21:32:04.327227+0000","last_fullsized":"2024-03-11T21:32:04.327227+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:31:55.464145+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:37.512018+0000","last_clean_scrub_stamp":"2024-03-11T21:31:55.464145+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:33:06.998087+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,3],"acting":[1,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.b","version":"74'9","reported_seq":241,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-03-11T21:32:04.327070+0000","last_change":"2024-03-11T21:31:20.485241+0000","last_active":"2024-03-11T21:32:04.327070+0000","last_peered":"2024-03-11T21:32:04.327070+0000","last_clean":"2024-03-11T21:32:04.327070+0000","last_became_active":"2024-03-11T21:25:40.549612+0000","last_became_peered":"2024-03-11T21:25:40.549612+0000","last_unstale":"2024-03-11T21:32:04.327070+0000","last_undegraded":"2024-03-11T21:32:04.327070+0000","last_fullsized":"2024-03-11T21:32:04.327070+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'9","last_scrub_stamp":"2024-03-11T21:31:20.485090+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:39.525554+0000","last_clean_scrub_stamp":"2024-03-11T21:31:20.485090+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:32:28.057879+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.4","version":"163'983","reported_seq":1005,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-03-11T21:32:07.004093+0000","last_change":"2024-03-11T21:31:52.272663+0000","last_active":"2024-03-11T21:32:07.004093+0000","last_peered":"2024-03-11T21:32:07.004093+0000","last_clean":"2024-03-11T21:32:07.004093+0000","last_became_active":"2024-03-11T21:31:52.272229+0000","last_became_peered":"2024-03-11T21:31:52.272229+0000","last_unstale":"2024-03-11T21:32:07.004093+0000","last_undegraded":"2024-03-11T21:32:07.004093+0000","last_fullsized":"2024-03-11T21:32:07.004093+0000","mapping_epoch":157,"log_start":"0'0","ondisk_log_start":"0'0","created":157,"last_epoch_clean":158,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:31:51.219688+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:31:51.219688+0000","last_clean_scrub_stamp":"2024-03-11T21:31:51.219688+0000","objects_scrubbed":0,"log_size":983,"log_dups_size":0,"ondisk_log_size":983,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:33:02.217752+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":64421888,"num_objects":983,"num_object_clones":0,"num_object_copies":2949,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":983,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":983,"num_write_kb":62912,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,2],"acting":[5,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":122,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-03-11T21:32:04.315129+0000","last_change":"2024-03-11T21:30:25.380496+0000","last_active":"2024-03-11T21:32:04.315129+0000","last_peered":"2024-03-11T21:32:04.315129+0000","last_clean":"2024-03-11T21:32:04.315129+0000","last_became_active":"2024-03-11T21:28:45.372452+0000","last_became_peered":"2024-03-11T21:28:45.372452+0000","last_unstale":"2024-03-11T21:32:04.315129+0000","last_undegraded":"2024-03-11T21:32:04.315129+0000","last_fullsized":"2024-03-11T21:32:04.315129+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:30:25.380313+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:43.559452+0000","last_clean_scrub_stamp":"2024-03-11T21:30:25.380313+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,3],"acting":[4,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.e","version":"0'0","reported_seq":242,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-03-11T21:32:04.330253+0000","last_change":"2024-03-11T21:31:48.232984+0000","last_active":"2024-03-11T21:32:04.330253+0000","last_peered":"2024-03-11T21:32:04.330253+0000","last_clean":"2024-03-11T21:32:04.330253+0000","last_became_active":"2024-03-11T21:31:48.232420+0000","last_became_peered":"2024-03-11T21:31:48.232420+0000","last_unstale":"2024-03-11T21:32:04.330253+0000","last_undegraded":"2024-03-11T21:32:04.330253+0000","last_fullsized":"2024-03-11T21:32:04.330253+0000","mapping_epoch":153,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":154,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:31:17.408572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:37.512018+0000","last_clean_scrub_stamp":"2024-03-11T21:31:17.408572+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:32:45.134639+0000","scrub_duration":21,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,6],"acting":[2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.8","version":"109'22","reported_seq":278,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-03-11T21:32:04.317974+0000","last_change":"2024-03-11T21:31:48.254935+0000","last_active":"2024-03-11T21:32:04.317974+0000","last_peered":"2024-03-11T21:32:04.317974+0000","last_clean":"2024-03-11T21:32:04.317974+0000","last_became_active":"2024-03-11T21:31:48.231316+0000","last_became_peered":"2024-03-11T21:31:48.231316+0000","last_unstale":"2024-03-11T21:32:04.317974+0000","last_undegraded":"2024-03-11T21:32:04.317974+0000","last_fullsized":"2024-03-11T21:32:04.317974+0000","mapping_epoch":153,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":154,"parent":"0.0","parent_split_bits":0,"last_scrub":"109'22","last_scrub_stamp":"2024-03-11T21:29:44.835663+0000","last_deep_scrub":"109'22","last_deep_scrub_stamp":"2024-03-11T21:28:25.743058+0000","last_clean_scrub_stamp":"2024-03-11T21:29:44.835663+0000","objects_scrubbed":7,"log_size":22,"log_dups_size":0,"ondisk_log_size":22,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,4],"acting":[5,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.9","version":"154'10","reported_seq":124,"reported_epoch":163,"state":"active+clean","last_fresh":"2024-03-11T21:32:04.320884+0000","last_change":"2024-03-11T21:31:48.246094+0000","last_active":"2024-03-11T21:32:04.320884+0000","last_peered":"2024-03-11T21:32:04.320884+0000","last_clean":"2024-03-11T21:32:04.320884+0000","last_became_active":"2024-03-11T21:31:48.223454+0000","last_became_peered":"2024-03-11T21:31:48.223454+0000","last_unstale":"2024-03-11T21:32:04.320884+0000","last_undegraded":"2024-03-11T21:32:04.320884+0000","last_fullsized":"2024-03-11T21:32:04.320884+0000","mapping_epoch":153,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":154,"parent":"0.0","parent_split_bits":0,"last_scrub":"94'9","last_scrub_stamp":"2024-03-11T21:31:19.528840+0000","last_deep_scrub":"94'9","last_deep_scrub_stamp":"2024-03-11T21:26:53.422552+0000","last_clean_scrub_stamp":"2024-03-11T21:31:19.528840+0000","objects_scrubbed":1,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:32:49.277556+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]}],"pool_stats":[{"poolid":8,"num_pg":16,"stat_sum":{"num_bytes":1065287680,"num_objects":16255,"num_object_clones":0,"num_object_copies":48765,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":16255,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16255,"num_write_kb":1040320,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":3170959360,"data_stored":3170959360,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":16255,"ondisk_log_size":16255,"up":48,"acting":48,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":1531,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":24,"num_read_kb":24,"num_write":10,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":436,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":4593,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":6,"ondisk_log_size":6,"up":96,"acting":96,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":198,"num_read_kb":294,"num_write":179,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":2361472,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1781760,"data_stored":1771104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":128,"ondisk_log_size":128,"up":3,"acting":3,"num_store_stats":4},{"poolid":3,"num_pg":3,"stat_sum":{"num_bytes":408,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":382,"num_read_kb":377,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1224,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":9,"acting":9,"num_store_stats":7},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":3702,"num_objects":178,"num_object_clones":0,"num_object_copies":534,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":178,"num_whiteouts":0,"num_read":698,"num_read_kb":455,"num_write":417,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":73,"num_bytes_recovered":1320,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":417792,"data_stored":11106,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":456,"ondisk_log_size":456,"up":96,"acting":96,"num_store_stats":8},{"poolid":5,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":71,"ondisk_log_size":71,"up":96,"acting":96,"num_store_stats":8},{"poolid":6,"num_pg":32,"stat_sum":{"num_bytes":383,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":370,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1149,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":96,"acting":96,"num_store_stats":8}],"osd_stats":[{"osd":7,"up_from":60,"seq":257698037841,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":183584,"kb_used_data":58008,"kb_used_omap":1,"kb_used_meta":125566,"kb_avail":93561568,"statfs":{"total":95995035648,"available":95807045632,"internally_reserved":0,"allocated":59400192,"data_stored":59032936,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":128580042},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1],"upper_bound":2},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":50,"seq":214748364887,"num_pgs":56,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":503832,"kb_used_data":356048,"kb_used_omap":1,"kb_used_meta":147774,"kb_avail":93241320,"statfs":{"total":95995035648,"available":95479111680,"internally_reserved":0,"allocated":364593152,"data_stored":364171841,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":151321030},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574961,"num_pgs":69,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":521924,"kb_used_data":392188,"kb_used_omap":1,"kb_used_meta":129726,"kb_avail":93223228,"statfs":{"total":95995035648,"available":95460585472,"internally_reserved":0,"allocated":401600512,"data_stored":401134257,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":132839877},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738486,"num_pgs":63,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":714392,"kb_used_data":524944,"kb_used_omap":1,"kb_used_meta":189438,"kb_avail":93030760,"statfs":{"total":95995035648,"available":95263498240,"internally_reserved":0,"allocated":537542656,"data_stored":537120723,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":193984965},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1],"upper_bound":4},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411435,"num_pgs":54,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":473228,"kb_used_data":339328,"kb_used_omap":1,"kb_used_meta":133886,"kb_avail":93271924,"statfs":{"total":95995035648,"available":95510450176,"internally_reserved":0,"allocated":347471872,"data_stored":347065297,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":137099717},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":26,"seq":111669149799,"num_pgs":69,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":633520,"kb_used_data":395496,"kb_used_omap":1,"kb_used_meta":238014,"kb_avail":93111632,"statfs":{"total":95995035648,"available":95346311168,"internally_reserved":0,"allocated":404987904,"data_stored":404541739,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":243726791},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":32,"seq":137438953568,"num_pgs":64,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":416096,"kb_used_data":294680,"kb_used_omap":1,"kb_used_meta":121406,"kb_avail":93329056,"statfs":{"total":95995035648,"available":95568953344,"internally_reserved":0,"allocated":301752320,"data_stored":301321453,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":124320197},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1],"upper_bound":8},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":41,"seq":176093659227,"num_pgs":61,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":610192,"kb_used_data":516552,"kb_used_omap":1,"kb_used_meta":93630,"kb_avail":93134960,"statfs":{"total":95995035648,"available":95370199040,"internally_reserved":0,"allocated":528949248,"data_stored":528534642,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":95877573},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1049,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":46,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":957,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":574,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":436,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1439,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":92,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":1320,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":102400,"data_stored":2668,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":1128,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":1898,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":1584,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":1210,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":1144,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":154,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":370,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":383,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":383,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":578617344,"data_stored":578617344,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":448987136,"data_stored":448987136,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":388104192,"data_stored":388104192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":407633920,"data_stored":407633920,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":322895872,"data_stored":322895872,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":576061440,"data_stored":576061440,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":383975424,"data_stored":383975424,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":64684032,"data_stored":64684032,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-11T21:32:12.024 DEBUG:teuthology.orchestra.run.smithi092:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid 00db0016-dfed-11ee-95c8-87774f69a715 -- ceph osd pool set unique_pool_1 pgp_num 16 2024-03-11T21:32:12.048 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:32:12.049 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:32:12.049 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:32:12.150 INFO:teuthology.orchestra.run.smithi092.stderr:Inferring config /var/lib/ceph/00db0016-dfed-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T21:32:12.150 INFO:teuthology.orchestra.run.smithi092.stderr:Inferring config /var/lib/ceph/00db0016-dfed-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T21:32:12.150 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:32:12.156 INFO:teuthology.orchestra.run.smithi092.stderr:Inferring config /var/lib/ceph/00db0016-dfed-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T21:32:12.207 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:32:12.207 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:32:12.208 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:32:12.308 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:32:12.338 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:32:12.338 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:32:12.339 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:32:12.428 INFO:tasks.radosbench.radosbench.0.smithi092.stdout: 11 16 27466 27450 155.936 121.438 0.00537113 0.00640676 2024-03-11T21:32:12.439 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:32:12.488 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:32:12.489 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:32:12.489 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:32:12.576 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:32:12 smithi097 ceph-mon[33317]: pgmap v255: 148 pgs: 148 active+clean; 1017 MiB data, 3.9 GiB used, 711 GiB / 715 GiB avail; 656 B/s rd, 110 MiB/s wr, 1.76k op/s 2024-03-11T21:32:12.577 INFO:journalctl@ceph.iscsi.iscsi.a.smithi097.stdout:Mar 11 21:32:12 smithi097 ceph-00db0016-dfed-11ee-95c8-87774f69a715-iscsi-iscsi-a[60943]: debug there is no tcmu-runner data available 2024-03-11T21:32:12.589 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:32:12.610 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:32:12 smithi092 ceph-mon[29572]: pgmap v255: 148 pgs: 148 active+clean; 1017 MiB data, 3.9 GiB used, 711 GiB / 715 GiB avail; 656 B/s rd, 110 MiB/s wr, 1.76k op/s 2024-03-11T21:32:12.610 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:32:12 smithi092 ceph-mon[37563]: pgmap v255: 148 pgs: 148 active+clean; 1017 MiB data, 3.9 GiB used, 711 GiB / 715 GiB avail; 656 B/s rd, 110 MiB/s wr, 1.76k op/s 2024-03-11T21:32:12.616 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:32:12.616 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:32:12.617 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:32:12.718 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:32:12.758 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:32:12.758 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:32:12.758 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:32:12.859 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:32:12.885 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:32:12.885 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:32:12.886 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:32:12.987 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:32:13.023 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:32:13.023 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:32:13.024 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:32:13.124 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:32:13.156 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:32:13.157 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:32:13.157 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:32:13.258 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:32:13.282 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:32:13.282 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:32:13.282 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:32:13.383 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:32:13.417 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:32:13.418 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:32:13.419 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:32:13.428 INFO:tasks.radosbench.radosbench.0.smithi092.stdout: 12 16 29250 29234 152.233 111.5 0.00416066 0.00656281 2024-03-11T21:32:13.519 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:32:13.552 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:32:13.553 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:32:13.553 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:32:13.654 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:32:13.691 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:32:13.691 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:32:13.691 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:32:13.792 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:32:13.816 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:32:13.817 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:32:13.817 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:32:13.826 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:32:13 smithi097 ceph-mon[33317]: from='client.24469 v1:172.21.15.97:0/404208310' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:32:13.826 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:32:13 smithi097 ceph-mon[33317]: 5.d scrub starts 2024-03-11T21:32:13.826 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:32:13 smithi097 ceph-mon[33317]: 5.d scrub ok 2024-03-11T21:32:13.826 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:32:13 smithi097 ceph-mon[33317]: 2.15 scrub starts 2024-03-11T21:32:13.826 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:32:13 smithi097 ceph-mon[33317]: 2.15 scrub ok 2024-03-11T21:32:13.826 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:32:13 smithi097 ceph-mon[33317]: 6.19 deep-scrub starts 2024-03-11T21:32:13.827 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:32:13 smithi097 ceph-mon[33317]: 6.19 deep-scrub ok 2024-03-11T21:32:13.827 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:32:13 smithi097 ceph-mon[33317]: 5.1f scrub starts 2024-03-11T21:32:13.827 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:32:13 smithi097 ceph-mon[33317]: 5.1f scrub ok 2024-03-11T21:32:13.860 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:32:13 smithi092 ceph-mon[29572]: from='client.24469 v1:172.21.15.97:0/404208310' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:32:13.860 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:32:13 smithi092 ceph-mon[29572]: 5.d scrub starts 2024-03-11T21:32:13.861 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:32:13 smithi092 ceph-mon[29572]: 5.d scrub ok 2024-03-11T21:32:13.861 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:32:13 smithi092 ceph-mon[29572]: 2.15 scrub starts 2024-03-11T21:32:13.861 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:32:13 smithi092 ceph-mon[29572]: 2.15 scrub ok 2024-03-11T21:32:13.861 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:32:13 smithi092 ceph-mon[29572]: 6.19 deep-scrub starts 2024-03-11T21:32:13.861 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:32:13 smithi092 ceph-mon[29572]: 6.19 deep-scrub ok 2024-03-11T21:32:13.861 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:32:13 smithi092 ceph-mon[29572]: 5.1f scrub starts 2024-03-11T21:32:13.861 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:32:13 smithi092 ceph-mon[29572]: 5.1f scrub ok 2024-03-11T21:32:13.861 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:32:13 smithi092 ceph-mon[37563]: from='client.24469 v1:172.21.15.97:0/404208310' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:32:13.861 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:32:13 smithi092 ceph-mon[37563]: 5.d scrub starts 2024-03-11T21:32:13.861 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:32:13 smithi092 ceph-mon[37563]: 5.d scrub ok 2024-03-11T21:32:13.862 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:32:13 smithi092 ceph-mon[37563]: 2.15 scrub starts 2024-03-11T21:32:13.862 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:32:13 smithi092 ceph-mon[37563]: 2.15 scrub ok 2024-03-11T21:32:13.862 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:32:13 smithi092 ceph-mon[37563]: 6.19 deep-scrub starts 2024-03-11T21:32:13.862 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:32:13 smithi092 ceph-mon[37563]: 6.19 deep-scrub ok 2024-03-11T21:32:13.862 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:32:13 smithi092 ceph-mon[37563]: 5.1f scrub starts 2024-03-11T21:32:13.862 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:32:13 smithi092 ceph-mon[37563]: 5.1f scrub ok 2024-03-11T21:32:13.918 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:32:13.949 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:32:13.950 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:32:13.950 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:32:14.051 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:32:14.085 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:32:14.086 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:32:14.086 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:32:14.187 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:32:14.212 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:32:14.212 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:32:14.213 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:32:14.313 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:32:14.344 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:32:14.345 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:32:14.345 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:32:14.428 INFO:tasks.radosbench.radosbench.0.smithi092.stdout: 13 16 30507 30491 146.566 78.5625 0.00231937 0.00681677 2024-03-11T21:32:14.446 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:32:14.476 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:32:14.476 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:32:14.477 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:32:14.577 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:32:14.610 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:32:14.610 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:32:14.611 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:32:14.711 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:32:14.745 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:32:14.746 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:32:14.747 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:32:14.826 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:32:14 smithi097 ceph-mon[33317]: pgmap v256: 148 pgs: 148 active+clean; 1.7 GiB data, 5.9 GiB used, 709 GiB / 715 GiB avail; 1.1 KiB/s rd, 176 MiB/s wr, 2.82k op/s 2024-03-11T21:32:14.826 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:32:14 smithi097 ceph-mon[33317]: 4.14 scrub starts 2024-03-11T21:32:14.826 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:32:14 smithi097 ceph-mon[33317]: 4.14 scrub ok 2024-03-11T21:32:14.826 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:32:14 smithi097 ceph-mon[33317]: 6.d scrub starts 2024-03-11T21:32:14.826 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:32:14 smithi097 ceph-mon[33317]: 6.d scrub ok 2024-03-11T21:32:14.848 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:32:14.860 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:32:14 smithi092 ceph-mon[29572]: pgmap v256: 148 pgs: 148 active+clean; 1.7 GiB data, 5.9 GiB used, 709 GiB / 715 GiB avail; 1.1 KiB/s rd, 176 MiB/s wr, 2.82k op/s 2024-03-11T21:32:14.860 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:32:14 smithi092 ceph-mon[29572]: 4.14 scrub starts 2024-03-11T21:32:14.860 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:32:14 smithi092 ceph-mon[29572]: 4.14 scrub ok 2024-03-11T21:32:14.860 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:32:14 smithi092 ceph-mon[29572]: 6.d scrub starts 2024-03-11T21:32:14.860 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:32:14 smithi092 ceph-mon[29572]: 6.d scrub ok 2024-03-11T21:32:14.860 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:32:14 smithi092 ceph-mon[37563]: pgmap v256: 148 pgs: 148 active+clean; 1.7 GiB data, 5.9 GiB used, 709 GiB / 715 GiB avail; 1.1 KiB/s rd, 176 MiB/s wr, 2.82k op/s 2024-03-11T21:32:14.860 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:32:14 smithi092 ceph-mon[37563]: 4.14 scrub starts 2024-03-11T21:32:14.860 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:32:14 smithi092 ceph-mon[37563]: 4.14 scrub ok 2024-03-11T21:32:14.861 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:32:14 smithi092 ceph-mon[37563]: 6.d scrub starts 2024-03-11T21:32:14.861 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:32:14 smithi092 ceph-mon[37563]: 6.d scrub ok 2024-03-11T21:32:14.874 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:32:14.874 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:32:14.875 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:32:14.975 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:32:15.017 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:32:15.018 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:32:15.018 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:32:15.119 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:32:15.150 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:32:15.151 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:32:15.151 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:32:15.252 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:32:15.292 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:32:15.292 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:32:15.293 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:32:15.393 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:32:15.426 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:32:15.427 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:32:15.427 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:32:15.428 INFO:tasks.radosbench.radosbench.0.smithi092.stdout: 14 16 32219 32203 143.739 107 0.00451533 0.00695087 2024-03-11T21:32:15.527 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:32:15.566 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:32:15.566 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:32:15.567 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:32:15.668 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:32:15.707 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:32:15.707 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:32:15.707 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:32:15.808 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:32:15.826 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:32:15 smithi097 ceph-mon[33317]: 4.16 scrub starts 2024-03-11T21:32:15.826 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:32:15 smithi097 ceph-mon[33317]: 4.16 scrub ok 2024-03-11T21:32:15.826 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:32:15 smithi097 ceph-mon[33317]: 2.12 scrub starts 2024-03-11T21:32:15.826 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:32:15 smithi097 ceph-mon[33317]: 2.12 scrub ok 2024-03-11T21:32:15.826 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:32:15 smithi097 ceph-mon[33317]: 4.7 scrub starts 2024-03-11T21:32:15.826 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:32:15 smithi097 ceph-mon[33317]: 4.7 scrub ok 2024-03-11T21:32:15.827 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:32:15 smithi097 ceph-mon[33317]: from='mgr.14607 v1:172.21.15.92:0/1702717597' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T21:32:15.827 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:32:15 smithi097 ceph-mon[33317]: from='mgr.14607 ' entity='mgr.y' 2024-03-11T21:32:15.827 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:32:15 smithi097 ceph-mon[33317]: from='mgr.14607 ' entity='mgr.y' 2024-03-11T21:32:15.846 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:32:15.847 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:32:15.847 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:32:15.860 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:32:15 smithi092 ceph-mon[29572]: 4.16 scrub starts 2024-03-11T21:32:15.860 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:32:15 smithi092 ceph-mon[29572]: 4.16 scrub ok 2024-03-11T21:32:15.860 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:32:15 smithi092 ceph-mon[29572]: 2.12 scrub starts 2024-03-11T21:32:15.860 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:32:15 smithi092 ceph-mon[29572]: 2.12 scrub ok 2024-03-11T21:32:15.860 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:32:15 smithi092 ceph-mon[29572]: 4.7 scrub starts 2024-03-11T21:32:15.860 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:32:15 smithi092 ceph-mon[29572]: 4.7 scrub ok 2024-03-11T21:32:15.860 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:32:15 smithi092 ceph-mon[29572]: from='mgr.14607 v1:172.21.15.92:0/1702717597' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T21:32:15.861 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:32:15 smithi092 ceph-mon[29572]: from='mgr.14607 ' entity='mgr.y' 2024-03-11T21:32:15.861 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:32:15 smithi092 ceph-mon[29572]: from='mgr.14607 ' entity='mgr.y' 2024-03-11T21:32:15.861 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:32:15 smithi092 ceph-mon[37563]: 4.16 scrub starts 2024-03-11T21:32:15.861 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:32:15 smithi092 ceph-mon[37563]: 4.16 scrub ok 2024-03-11T21:32:15.861 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:32:15 smithi092 ceph-mon[37563]: 2.12 scrub starts 2024-03-11T21:32:15.861 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:32:15 smithi092 ceph-mon[37563]: 2.12 scrub ok 2024-03-11T21:32:15.861 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:32:15 smithi092 ceph-mon[37563]: 4.7 scrub starts 2024-03-11T21:32:15.861 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:32:15 smithi092 ceph-mon[37563]: 4.7 scrub ok 2024-03-11T21:32:15.861 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:32:15 smithi092 ceph-mon[37563]: from='mgr.14607 v1:172.21.15.92:0/1702717597' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T21:32:15.861 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:32:15 smithi092 ceph-mon[37563]: from='mgr.14607 ' entity='mgr.y' 2024-03-11T21:32:15.861 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:32:15 smithi092 ceph-mon[37563]: from='mgr.14607 ' entity='mgr.y' 2024-03-11T21:32:15.948 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:32:15.982 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:32:15.983 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:32:15.984 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:32:16.084 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:32:16.119 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:32:16.119 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:32:16.120 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:32:16.221 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:32:16.245 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:32:16.245 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:32:16.246 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:32:16.346 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:32:16.383 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:32:16.384 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:32:16.384 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:32:16.428 INFO:tasks.radosbench.radosbench.0.smithi092.stdout: 15 16 33884 33868 141.094 104.062 0.00418694 0.00708223 2024-03-11T21:32:16.486 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:32:16.518 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:32:16.519 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:32:16.519 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:32:16.620 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:32:16.662 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:32:16.663 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:32:16.663 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:32:16.764 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:32:16.795 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:32:16.796 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:32:16.796 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:32:16.826 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:32:16 smithi097 ceph-mon[33317]: pgmap v257: 148 pgs: 148 active+clean; 1.7 GiB data, 5.9 GiB used, 709 GiB / 715 GiB avail; 1023 B/s rd, 155 MiB/s wr, 2.48k op/s 2024-03-11T21:32:16.826 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:32:16 smithi097 ceph-mon[33317]: 2.19 scrub starts 2024-03-11T21:32:16.826 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:32:16 smithi097 ceph-mon[33317]: 2.19 scrub ok 2024-03-11T21:32:16.860 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:32:16 smithi092 ceph-mon[29572]: pgmap v257: 148 pgs: 148 active+clean; 1.7 GiB data, 5.9 GiB used, 709 GiB / 715 GiB avail; 1023 B/s rd, 155 MiB/s wr, 2.48k op/s 2024-03-11T21:32:16.860 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:32:16 smithi092 ceph-mon[29572]: 2.19 scrub starts 2024-03-11T21:32:16.860 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:32:16 smithi092 ceph-mon[29572]: 2.19 scrub ok 2024-03-11T21:32:16.861 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:32:16 smithi092 ceph-mon[37563]: pgmap v257: 148 pgs: 148 active+clean; 1.7 GiB data, 5.9 GiB used, 709 GiB / 715 GiB avail; 1023 B/s rd, 155 MiB/s wr, 2.48k op/s 2024-03-11T21:32:16.861 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:32:16 smithi092 ceph-mon[37563]: 2.19 scrub starts 2024-03-11T21:32:16.861 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:32:16 smithi092 ceph-mon[37563]: 2.19 scrub ok 2024-03-11T21:32:16.897 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:32:16.925 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:32:16.925 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:32:16.926 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:32:17.026 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:32:17.066 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:32:17.067 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:32:17.068 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:32:17.169 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:32:17.201 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:32:17.202 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:32:17.202 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:32:17.303 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:32:17.342 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:32:17.343 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:32:17.343 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:32:17.428 INFO:tasks.radosbench.radosbench.0.smithi092.stdout: 16 16 35834 35818 139.892 121.875 0.00561118 0.00714329 2024-03-11T21:32:17.445 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:32:17.481 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:32:17.482 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:32:17.482 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:32:17.583 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:32:17.613 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:32:17.613 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:32:17.614 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:32:17.715 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:32:17.748 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:32:17.748 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:32:17.749 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:32:17.826 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:32:17 smithi097 ceph-mon[33317]: 5.1 scrub starts 2024-03-11T21:32:17.826 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:32:17 smithi097 ceph-mon[33317]: 5.1 scrub ok 2024-03-11T21:32:17.850 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:32:17.860 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:32:17 smithi092 ceph-mon[29572]: 5.1 scrub starts 2024-03-11T21:32:17.860 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:32:17 smithi092 ceph-mon[29572]: 5.1 scrub ok 2024-03-11T21:32:17.860 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:32:17 smithi092 ceph-mon[37563]: 5.1 scrub starts 2024-03-11T21:32:17.860 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:32:17 smithi092 ceph-mon[37563]: 5.1 scrub ok 2024-03-11T21:32:17.881 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:32:17.882 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:32:17.882 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:32:17.983 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:32:18.022 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:32:18.023 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:32:18.023 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:32:18.124 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:32:18.162 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:32:18.162 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:32:18.163 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:32:18.263 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:32:18.303 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:32:18.304 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:32:18.304 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:32:18.305 INFO:journalctl@ceph.mgr.y.smithi092.stdout:Mar 11 21:32:17 smithi092 ceph-00db0016-dfed-11ee-95c8-87774f69a715-mgr-y[29801]: ::ffff:172.21.15.97 - - [11/Mar/2024:21:32:17] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-11T21:32:18.405 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:32:18.428 INFO:tasks.radosbench.radosbench.0.smithi092.stdout: 17 15 37708 37693 138.556 117.188 0.00496895 0.00721225 2024-03-11T21:32:18.437 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:32:18.438 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:32:18.438 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:32:18.539 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:32:18.586 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:32:18.587 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:32:18.587 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:32:18.689 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:32:18.734 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:32:18.734 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:32:18.735 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:32:18.826 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:32:18 smithi097 ceph-mon[33317]: pgmap v258: 148 pgs: 148 active+clean; 1.8 GiB data, 6.4 GiB used, 709 GiB / 715 GiB avail; 1023 B/s rd, 145 MiB/s wr, 2.33k op/s 2024-03-11T21:32:18.826 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:32:18 smithi097 ceph-mon[33317]: 6.f scrub starts 2024-03-11T21:32:18.826 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:32:18 smithi097 ceph-mon[33317]: 6.f scrub ok 2024-03-11T21:32:18.826 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:32:18 smithi097 ceph-mon[33317]: 3.0 deep-scrub starts 2024-03-11T21:32:18.827 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:32:18 smithi097 ceph-mon[33317]: 3.0 deep-scrub ok 2024-03-11T21:32:18.827 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:32:18 smithi097 ceph-mon[33317]: 6.c scrub starts 2024-03-11T21:32:18.827 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:32:18 smithi097 ceph-mon[33317]: 6.c scrub ok 2024-03-11T21:32:18.835 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:32:18.860 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:32:18 smithi092 ceph-mon[29572]: pgmap v258: 148 pgs: 148 active+clean; 1.8 GiB data, 6.4 GiB used, 709 GiB / 715 GiB avail; 1023 B/s rd, 145 MiB/s wr, 2.33k op/s 2024-03-11T21:32:18.861 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:32:18 smithi092 ceph-mon[29572]: 6.f scrub starts 2024-03-11T21:32:18.861 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:32:18 smithi092 ceph-mon[29572]: 6.f scrub ok 2024-03-11T21:32:18.861 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:32:18 smithi092 ceph-mon[29572]: 3.0 deep-scrub starts 2024-03-11T21:32:18.861 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:32:18 smithi092 ceph-mon[29572]: 3.0 deep-scrub ok 2024-03-11T21:32:18.861 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:32:18 smithi092 ceph-mon[29572]: 6.c scrub starts 2024-03-11T21:32:18.861 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:32:18 smithi092 ceph-mon[29572]: 6.c scrub ok 2024-03-11T21:32:18.861 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:32:18 smithi092 ceph-mon[37563]: pgmap v258: 148 pgs: 148 active+clean; 1.8 GiB data, 6.4 GiB used, 709 GiB / 715 GiB avail; 1023 B/s rd, 145 MiB/s wr, 2.33k op/s 2024-03-11T21:32:18.861 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:32:18 smithi092 ceph-mon[37563]: 6.f scrub starts 2024-03-11T21:32:18.861 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:32:18 smithi092 ceph-mon[37563]: 6.f scrub ok 2024-03-11T21:32:18.861 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:32:18 smithi092 ceph-mon[37563]: 3.0 deep-scrub starts 2024-03-11T21:32:18.862 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:32:18 smithi092 ceph-mon[37563]: 3.0 deep-scrub ok 2024-03-11T21:32:18.862 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:32:18 smithi092 ceph-mon[37563]: 6.c scrub starts 2024-03-11T21:32:18.862 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:32:18 smithi092 ceph-mon[37563]: 6.c scrub ok 2024-03-11T21:32:18.867 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:32:18.867 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:32:18.868 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:32:18.969 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:32:18.999 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:32:19.000 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:32:19.000 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:32:19.101 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:32:19.141 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:32:19.142 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:32:19.143 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:32:19.243 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:32:19.271 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:32:19.272 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:32:19.273 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:32:19.373 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:32:19.405 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:32:19.406 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:32:19.406 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:32:19.429 INFO:tasks.radosbench.radosbench.0.smithi092.stdout: 18 16 39416 39400 136.785 106.688 0.0106268 0.00730494 2024-03-11T21:32:19.507 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:32:19.554 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:32:19.554 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:32:19.555 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:32:19.656 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:32:19.703 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:32:19.703 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:32:19.704 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:32:19.804 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:32:19.833 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:32:19.834 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:32:19.835 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:32:19.928 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:32:19 smithi092 ceph-mon[29572]: 2.1e scrub starts 2024-03-11T21:32:19.928 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:32:19 smithi092 ceph-mon[29572]: 2.1e scrub ok 2024-03-11T21:32:19.928 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:32:19 smithi092 ceph-mon[29572]: 6.a scrub starts 2024-03-11T21:32:19.928 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:32:19 smithi092 ceph-mon[29572]: 6.a scrub ok 2024-03-11T21:32:19.928 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:32:19 smithi092 ceph-mon[29572]: from='mgr.14607 ' entity='mgr.y' 2024-03-11T21:32:19.928 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:32:19 smithi092 ceph-mon[29572]: from='mgr.14607 ' entity='mgr.y' 2024-03-11T21:32:19.928 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:32:19 smithi092 ceph-mon[37563]: 2.1e scrub starts 2024-03-11T21:32:19.929 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:32:19 smithi092 ceph-mon[37563]: 2.1e scrub ok 2024-03-11T21:32:19.929 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:32:19 smithi092 ceph-mon[37563]: 6.a scrub starts 2024-03-11T21:32:19.929 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:32:19 smithi092 ceph-mon[37563]: 6.a scrub ok 2024-03-11T21:32:19.929 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:32:19 smithi092 ceph-mon[37563]: from='mgr.14607 ' entity='mgr.y' 2024-03-11T21:32:19.929 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:32:19 smithi092 ceph-mon[37563]: from='mgr.14607 ' entity='mgr.y' 2024-03-11T21:32:19.935 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:32:19.965 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:32:19.966 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:32:19.966 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:32:20.067 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:32:20.076 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:32:19 smithi097 ceph-mon[33317]: 2.1e scrub starts 2024-03-11T21:32:20.076 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:32:19 smithi097 ceph-mon[33317]: 2.1e scrub ok 2024-03-11T21:32:20.076 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:32:19 smithi097 ceph-mon[33317]: 6.a scrub starts 2024-03-11T21:32:20.076 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:32:19 smithi097 ceph-mon[33317]: 6.a scrub ok 2024-03-11T21:32:20.076 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:32:19 smithi097 ceph-mon[33317]: from='mgr.14607 ' entity='mgr.y' 2024-03-11T21:32:20.076 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:32:19 smithi097 ceph-mon[33317]: from='mgr.14607 ' entity='mgr.y' 2024-03-11T21:32:20.098 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:32:20.099 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:32:20.099 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:32:20.200 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:32:20.232 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:32:20.232 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:32:20.233 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:32:20.333 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:32:20.373 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:32:20.373 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:32:20.374 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:32:20.429 INFO:tasks.radosbench.radosbench.0.smithi092.stdout: 19 16 41306 41290 135.802 118.125 0.00764781 0.00735883 2024-03-11T21:32:20.474 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:32:20.507 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:32:20.507 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:32:20.508 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:32:20.608 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:32:20.639 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:32:20.640 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:32:20.640 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:32:20.730 INFO:teuthology.orchestra.run.smithi092.stderr:Inferring config /var/lib/ceph/00db0016-dfed-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T21:32:20.741 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:32:20.771 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:32:20.771 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:32:20.771 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:32:20.872 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:32:20.906 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:32:20.906 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:32:20.907 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:32:21.008 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:32:21.041 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:32:21.042 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:32:21.043 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:32:21.076 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:32:20 smithi097 ceph-mon[33317]: pgmap v259: 148 pgs: 148 active+clean; 2.2 GiB data, 7.5 GiB used, 708 GiB / 715 GiB avail; 1023 B/s rd, 121 MiB/s wr, 1.93k op/s 2024-03-11T21:32:21.076 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:32:20 smithi097 ceph-mon[33317]: 6.1e scrub starts 2024-03-11T21:32:21.076 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:32:20 smithi097 ceph-mon[33317]: 6.1e scrub ok 2024-03-11T21:32:21.076 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:32:20 smithi097 ceph-mon[33317]: purged_snaps scrub starts 2024-03-11T21:32:21.076 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:32:20 smithi097 ceph-mon[33317]: purged_snaps scrub ok 2024-03-11T21:32:21.110 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:32:20 smithi092 ceph-mon[29572]: pgmap v259: 148 pgs: 148 active+clean; 2.2 GiB data, 7.5 GiB used, 708 GiB / 715 GiB avail; 1023 B/s rd, 121 MiB/s wr, 1.93k op/s 2024-03-11T21:32:21.111 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:32:20 smithi092 ceph-mon[29572]: 6.1e scrub starts 2024-03-11T21:32:21.111 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:32:20 smithi092 ceph-mon[29572]: 6.1e scrub ok 2024-03-11T21:32:21.111 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:32:20 smithi092 ceph-mon[29572]: purged_snaps scrub starts 2024-03-11T21:32:21.111 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:32:20 smithi092 ceph-mon[29572]: purged_snaps scrub ok 2024-03-11T21:32:21.111 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:32:20 smithi092 ceph-mon[37563]: pgmap v259: 148 pgs: 148 active+clean; 2.2 GiB data, 7.5 GiB used, 708 GiB / 715 GiB avail; 1023 B/s rd, 121 MiB/s wr, 1.93k op/s 2024-03-11T21:32:21.111 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:32:20 smithi092 ceph-mon[37563]: 6.1e scrub starts 2024-03-11T21:32:21.111 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:32:20 smithi092 ceph-mon[37563]: 6.1e scrub ok 2024-03-11T21:32:21.111 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:32:20 smithi092 ceph-mon[37563]: purged_snaps scrub starts 2024-03-11T21:32:21.111 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:32:20 smithi092 ceph-mon[37563]: purged_snaps scrub ok 2024-03-11T21:32:21.144 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:32:21.216 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:32:21.219 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:32:21.219 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:32:21.320 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:32:21.353 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:32:21.354 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:32:21.354 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:32:21.430 INFO:tasks.radosbench.radosbench.0.smithi092.stdout:2024-03-11T21:32:21.448516+0000 min lat: 0.0013089 max lat: 0.168224 avg lat: 0.00743409 2024-03-11T21:32:21.430 INFO:tasks.radosbench.radosbench.0.smithi092.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-03-11T21:32:21.430 INFO:tasks.radosbench.radosbench.0.smithi092.stdout: 20 16 43026 43010 134.38 107.5 0.00552234 0.00743409 2024-03-11T21:32:21.454 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:32:21.504 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:32:21.504 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:32:21.508 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:32:21.608 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:32:21.652 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:32:21.652 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:32:21.653 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:32:21.754 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:32:21.793 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:32:21.793 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:32:21.794 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:32:21.895 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:32:21.928 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:32:21.928 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:32:21.929 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:32:21.966 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:32:21 smithi092 ceph-mon[29572]: 1.0 scrub starts 2024-03-11T21:32:21.966 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:32:21 smithi092 ceph-mon[29572]: 1.0 scrub ok 2024-03-11T21:32:21.966 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:32:21 smithi092 ceph-mon[29572]: osdmap e165: 8 total, 8 up, 8 in 2024-03-11T21:32:21.966 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:32:21 smithi092 ceph-mon[29572]: from='mgr.14607 v1:172.21.15.92:0/1702717597' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.6", "id": [1, 5]}]: dispatch 2024-03-11T21:32:21.966 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:32:21 smithi092 ceph-mon[29572]: from='mgr.14607 v1:172.21.15.92:0/1702717597' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.1f", "id": [1, 0]}]: dispatch 2024-03-11T21:32:21.966 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:32:21 smithi092 ceph-mon[29572]: from='mgr.14607 v1:172.21.15.92:0/1702717597' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "5.4", "id": [0, 6]}]: dispatch 2024-03-11T21:32:21.966 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:32:21 smithi092 ceph-mon[29572]: from='mgr.14607 v1:172.21.15.92:0/1702717597' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "6.2", "id": [3, 0]}]: dispatch 2024-03-11T21:32:21.966 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:32:21 smithi092 ceph-mon[29572]: from='mgr.14607 ' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.6", "id": [1, 5]}]: dispatch 2024-03-11T21:32:21.967 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:32:21 smithi092 ceph-mon[29572]: from='mgr.14607 v1:172.21.15.92:0/1702717597' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "6.8", "id": [3, 0]}]: dispatch 2024-03-11T21:32:21.967 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:32:21 smithi092 ceph-mon[29572]: from='mgr.14607 ' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.1f", "id": [1, 0]}]: dispatch 2024-03-11T21:32:21.967 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:32:21 smithi092 ceph-mon[29572]: from='mgr.14607 ' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "5.4", "id": [0, 6]}]: dispatch 2024-03-11T21:32:21.967 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:32:21 smithi092 ceph-mon[29572]: from='mgr.14607 ' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "6.2", "id": [3, 0]}]: dispatch 2024-03-11T21:32:21.967 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:32:21 smithi092 ceph-mon[29572]: from='mgr.14607 ' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "6.8", "id": [3, 0]}]: dispatch 2024-03-11T21:32:21.967 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:32:21 smithi092 ceph-mon[29572]: 5.4 deep-scrub starts 2024-03-11T21:32:21.967 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:32:21 smithi092 ceph-mon[29572]: 5.4 deep-scrub ok 2024-03-11T21:32:21.967 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:32:21 smithi092 ceph-mon[29572]: from='mgr.14607 v1:172.21.15.92:0/1702717597' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T21:32:21.967 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:32:21 smithi092 ceph-mon[29572]: from='mgr.14607 v1:172.21.15.92:0/1702717597' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T21:32:21.967 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:32:21 smithi092 ceph-mon[29572]: from='mgr.14607 ' entity='mgr.y' 2024-03-11T21:32:21.967 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:32:21 smithi092 ceph-mon[29572]: from='mgr.14607 v1:172.21.15.92:0/1702717597' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T21:32:21.967 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:32:21 smithi092 ceph-mon[37563]: 1.0 scrub starts 2024-03-11T21:32:21.968 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:32:21 smithi092 ceph-mon[37563]: 1.0 scrub ok 2024-03-11T21:32:21.968 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:32:21 smithi092 ceph-mon[37563]: osdmap e165: 8 total, 8 up, 8 in 2024-03-11T21:32:21.968 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:32:21 smithi092 ceph-mon[37563]: from='mgr.14607 v1:172.21.15.92:0/1702717597' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.6", "id": [1, 5]}]: dispatch 2024-03-11T21:32:21.968 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:32:21 smithi092 ceph-mon[37563]: from='mgr.14607 v1:172.21.15.92:0/1702717597' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.1f", "id": [1, 0]}]: dispatch 2024-03-11T21:32:21.968 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:32:21 smithi092 ceph-mon[37563]: from='mgr.14607 v1:172.21.15.92:0/1702717597' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "5.4", "id": [0, 6]}]: dispatch 2024-03-11T21:32:21.968 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:32:21 smithi092 ceph-mon[37563]: from='mgr.14607 v1:172.21.15.92:0/1702717597' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "6.2", "id": [3, 0]}]: dispatch 2024-03-11T21:32:21.968 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:32:21 smithi092 ceph-mon[37563]: from='mgr.14607 ' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.6", "id": [1, 5]}]: dispatch 2024-03-11T21:32:21.968 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:32:21 smithi092 ceph-mon[37563]: from='mgr.14607 v1:172.21.15.92:0/1702717597' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "6.8", "id": [3, 0]}]: dispatch 2024-03-11T21:32:21.968 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:32:21 smithi092 ceph-mon[37563]: from='mgr.14607 ' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.1f", "id": [1, 0]}]: dispatch 2024-03-11T21:32:21.968 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:32:21 smithi092 ceph-mon[37563]: from='mgr.14607 ' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "5.4", "id": [0, 6]}]: dispatch 2024-03-11T21:32:21.968 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:32:21 smithi092 ceph-mon[37563]: from='mgr.14607 ' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "6.2", "id": [3, 0]}]: dispatch 2024-03-11T21:32:21.968 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:32:21 smithi092 ceph-mon[37563]: from='mgr.14607 ' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "6.8", "id": [3, 0]}]: dispatch 2024-03-11T21:32:21.968 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:32:21 smithi092 ceph-mon[37563]: 5.4 deep-scrub starts 2024-03-11T21:32:21.968 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:32:21 smithi092 ceph-mon[37563]: 5.4 deep-scrub ok 2024-03-11T21:32:21.969 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:32:21 smithi092 ceph-mon[37563]: from='mgr.14607 v1:172.21.15.92:0/1702717597' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T21:32:21.969 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:32:21 smithi092 ceph-mon[37563]: from='mgr.14607 v1:172.21.15.92:0/1702717597' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T21:32:21.969 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:32:21 smithi092 ceph-mon[37563]: from='mgr.14607 ' entity='mgr.y' 2024-03-11T21:32:21.969 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:32:21 smithi092 ceph-mon[37563]: from='mgr.14607 v1:172.21.15.92:0/1702717597' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T21:32:22.029 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:32:22.073 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:32:22.074 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:32:22.074 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:32:22.076 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:32:21 smithi097 ceph-mon[33317]: 1.0 scrub starts 2024-03-11T21:32:22.076 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:32:21 smithi097 ceph-mon[33317]: 1.0 scrub ok 2024-03-11T21:32:22.076 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:32:21 smithi097 ceph-mon[33317]: osdmap e165: 8 total, 8 up, 8 in 2024-03-11T21:32:22.076 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:32:21 smithi097 ceph-mon[33317]: from='mgr.14607 v1:172.21.15.92:0/1702717597' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.6", "id": [1, 5]}]: dispatch 2024-03-11T21:32:22.076 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:32:21 smithi097 ceph-mon[33317]: from='mgr.14607 v1:172.21.15.92:0/1702717597' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.1f", "id": [1, 0]}]: dispatch 2024-03-11T21:32:22.076 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:32:21 smithi097 ceph-mon[33317]: from='mgr.14607 v1:172.21.15.92:0/1702717597' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "5.4", "id": [0, 6]}]: dispatch 2024-03-11T21:32:22.077 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:32:21 smithi097 ceph-mon[33317]: from='mgr.14607 v1:172.21.15.92:0/1702717597' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "6.2", "id": [3, 0]}]: dispatch 2024-03-11T21:32:22.077 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:32:21 smithi097 ceph-mon[33317]: from='mgr.14607 ' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.6", "id": [1, 5]}]: dispatch 2024-03-11T21:32:22.077 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:32:21 smithi097 ceph-mon[33317]: from='mgr.14607 v1:172.21.15.92:0/1702717597' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "6.8", "id": [3, 0]}]: dispatch 2024-03-11T21:32:22.077 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:32:21 smithi097 ceph-mon[33317]: from='mgr.14607 ' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.1f", "id": [1, 0]}]: dispatch 2024-03-11T21:32:22.077 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:32:21 smithi097 ceph-mon[33317]: from='mgr.14607 ' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "5.4", "id": [0, 6]}]: dispatch 2024-03-11T21:32:22.077 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:32:21 smithi097 ceph-mon[33317]: from='mgr.14607 ' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "6.2", "id": [3, 0]}]: dispatch 2024-03-11T21:32:22.077 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:32:21 smithi097 ceph-mon[33317]: from='mgr.14607 ' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "6.8", "id": [3, 0]}]: dispatch 2024-03-11T21:32:22.077 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:32:21 smithi097 ceph-mon[33317]: 5.4 deep-scrub starts 2024-03-11T21:32:22.077 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:32:21 smithi097 ceph-mon[33317]: 5.4 deep-scrub ok 2024-03-11T21:32:22.077 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:32:21 smithi097 ceph-mon[33317]: from='mgr.14607 v1:172.21.15.92:0/1702717597' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T21:32:22.077 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:32:21 smithi097 ceph-mon[33317]: from='mgr.14607 v1:172.21.15.92:0/1702717597' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T21:32:22.077 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:32:21 smithi097 ceph-mon[33317]: from='mgr.14607 ' entity='mgr.y' 2024-03-11T21:32:22.077 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:32:21 smithi097 ceph-mon[33317]: from='mgr.14607 v1:172.21.15.92:0/1702717597' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T21:32:22.175 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:32:22.211 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:32:22.212 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:32:22.212 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:32:22.313 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:32:22.357 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:32:22.357 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:32:22.358 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:32:22.430 INFO:tasks.radosbench.radosbench.0.smithi092.stdout: 21 16 44638 44622 132.779 100.75 0.00322847 0.00752635 2024-03-11T21:32:22.458 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:32:22.496 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:32:22.496 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:32:22.497 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:32:22.576 INFO:journalctl@ceph.iscsi.iscsi.a.smithi097.stdout:Mar 11 21:32:22 smithi097 ceph-00db0016-dfed-11ee-95c8-87774f69a715-iscsi-iscsi-a[60943]: debug there is no tcmu-runner data available 2024-03-11T21:32:22.598 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:32:22.630 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:32:22.631 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:32:22.632 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:32:22.652 INFO:teuthology.orchestra.run.smithi092.stdout:ERROR: (22) Invalid argument 2024-03-11T21:32:22.652 INFO:teuthology.orchestra.run.smithi092.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-11T21:32:22.733 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:32:22.770 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:32:22.770 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:32:22.771 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:32:22.872 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:32:22.908 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:32:22.909 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:32:22.910 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:32:22.934 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:32:22 smithi092 ceph-mon[29572]: pgmap v261: 148 pgs: 148 active+clean; 2.2 GiB data, 7.5 GiB used, 708 GiB / 715 GiB avail; 1023 B/s rd, 121 MiB/s wr, 1.93k op/s 2024-03-11T21:32:22.934 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:32:22 smithi092 ceph-mon[29572]: 5.1b deep-scrub starts 2024-03-11T21:32:22.934 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:32:22 smithi092 ceph-mon[29572]: 5.1b deep-scrub ok 2024-03-11T21:32:22.934 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:32:22 smithi092 ceph-mon[29572]: 3.2 scrub starts 2024-03-11T21:32:22.934 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:32:22 smithi092 ceph-mon[29572]: 3.2 scrub ok 2024-03-11T21:32:22.935 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:32:22 smithi092 ceph-mon[29572]: 4.3 deep-scrub starts 2024-03-11T21:32:22.935 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:32:22 smithi092 ceph-mon[29572]: 4.3 deep-scrub ok 2024-03-11T21:32:22.935 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:32:22 smithi092 ceph-mon[29572]: 4.d deep-scrub starts 2024-03-11T21:32:22.935 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:32:22 smithi092 ceph-mon[29572]: 4.d deep-scrub ok 2024-03-11T21:32:22.935 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:32:22 smithi092 ceph-mon[29572]: from='mgr.14607 ' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.6", "id": [1, 5]}]': finished 2024-03-11T21:32:22.935 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:32:22 smithi092 ceph-mon[29572]: from='mgr.14607 ' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.1f", "id": [1, 0]}]': finished 2024-03-11T21:32:22.936 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:32:22 smithi092 ceph-mon[29572]: from='mgr.14607 ' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "5.4", "id": [0, 6]}]': finished 2024-03-11T21:32:22.936 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:32:22 smithi092 ceph-mon[29572]: from='mgr.14607 ' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "6.2", "id": [3, 0]}]': finished 2024-03-11T21:32:22.936 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:32:22 smithi092 ceph-mon[29572]: from='mgr.14607 ' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "6.8", "id": [3, 0]}]': finished 2024-03-11T21:32:22.936 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:32:22 smithi092 ceph-mon[29572]: osdmap e166: 8 total, 8 up, 8 in 2024-03-11T21:32:22.936 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:32:22 smithi092 ceph-mon[37563]: pgmap v261: 148 pgs: 148 active+clean; 2.2 GiB data, 7.5 GiB used, 708 GiB / 715 GiB avail; 1023 B/s rd, 121 MiB/s wr, 1.93k op/s 2024-03-11T21:32:22.937 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:32:22 smithi092 ceph-mon[37563]: 5.1b deep-scrub starts 2024-03-11T21:32:22.937 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:32:22 smithi092 ceph-mon[37563]: 5.1b deep-scrub ok 2024-03-11T21:32:22.937 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:32:22 smithi092 ceph-mon[37563]: 3.2 scrub starts 2024-03-11T21:32:22.937 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:32:22 smithi092 ceph-mon[37563]: 3.2 scrub ok 2024-03-11T21:32:22.937 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:32:22 smithi092 ceph-mon[37563]: 4.3 deep-scrub starts 2024-03-11T21:32:22.937 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:32:22 smithi092 ceph-mon[37563]: 4.3 deep-scrub ok 2024-03-11T21:32:22.937 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:32:22 smithi092 ceph-mon[37563]: 4.d deep-scrub starts 2024-03-11T21:32:22.938 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:32:22 smithi092 ceph-mon[37563]: 4.d deep-scrub ok 2024-03-11T21:32:22.938 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:32:22 smithi092 ceph-mon[37563]: from='mgr.14607 ' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.6", "id": [1, 5]}]': finished 2024-03-11T21:32:22.938 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:32:22 smithi092 ceph-mon[37563]: from='mgr.14607 ' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.1f", "id": [1, 0]}]': finished 2024-03-11T21:32:22.938 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:32:22 smithi092 ceph-mon[37563]: from='mgr.14607 ' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "5.4", "id": [0, 6]}]': finished 2024-03-11T21:32:22.938 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:32:22 smithi092 ceph-mon[37563]: from='mgr.14607 ' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "6.2", "id": [3, 0]}]': finished 2024-03-11T21:32:22.939 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:32:22 smithi092 ceph-mon[37563]: from='mgr.14607 ' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "6.8", "id": [3, 0]}]': finished 2024-03-11T21:32:22.939 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:32:22 smithi092 ceph-mon[37563]: osdmap e166: 8 total, 8 up, 8 in 2024-03-11T21:32:23.010 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:32:23.059 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:32:23.060 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:32:23.060 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:32:23.076 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:32:22 smithi097 ceph-mon[33317]: pgmap v261: 148 pgs: 148 active+clean; 2.2 GiB data, 7.5 GiB used, 708 GiB / 715 GiB avail; 1023 B/s rd, 121 MiB/s wr, 1.93k op/s 2024-03-11T21:32:23.076 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:32:22 smithi097 ceph-mon[33317]: 5.1b deep-scrub starts 2024-03-11T21:32:23.076 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:32:22 smithi097 ceph-mon[33317]: 5.1b deep-scrub ok 2024-03-11T21:32:23.077 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:32:22 smithi097 ceph-mon[33317]: 3.2 scrub starts 2024-03-11T21:32:23.077 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:32:22 smithi097 ceph-mon[33317]: 3.2 scrub ok 2024-03-11T21:32:23.077 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:32:22 smithi097 ceph-mon[33317]: 4.3 deep-scrub starts 2024-03-11T21:32:23.077 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:32:22 smithi097 ceph-mon[33317]: 4.3 deep-scrub ok 2024-03-11T21:32:23.077 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:32:22 smithi097 ceph-mon[33317]: 4.d deep-scrub starts 2024-03-11T21:32:23.077 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:32:22 smithi097 ceph-mon[33317]: 4.d deep-scrub ok 2024-03-11T21:32:23.077 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:32:22 smithi097 ceph-mon[33317]: from='mgr.14607 ' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.6", "id": [1, 5]}]': finished 2024-03-11T21:32:23.077 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:32:22 smithi097 ceph-mon[33317]: from='mgr.14607 ' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.1f", "id": [1, 0]}]': finished 2024-03-11T21:32:23.077 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:32:22 smithi097 ceph-mon[33317]: from='mgr.14607 ' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "5.4", "id": [0, 6]}]': finished 2024-03-11T21:32:23.077 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:32:22 smithi097 ceph-mon[33317]: from='mgr.14607 ' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "6.2", "id": [3, 0]}]': finished 2024-03-11T21:32:23.078 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:32:22 smithi097 ceph-mon[33317]: from='mgr.14607 ' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "6.8", "id": [3, 0]}]': finished 2024-03-11T21:32:23.078 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:32:22 smithi097 ceph-mon[33317]: osdmap e166: 8 total, 8 up, 8 in 2024-03-11T21:32:23.161 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:32:23.196 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:32:23.196 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:32:23.196 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:32:23.297 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:32:23.339 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:32:23.339 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:32:23.339 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:32:23.393 DEBUG:teuthology.orchestra.run.smithi092:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid 00db0016-dfed-11ee-95c8-87774f69a715 -- ceph daemon osd.3 dump_historic_ops 2024-03-11T21:32:23.430 INFO:tasks.radosbench.radosbench.0.smithi092.stdout: 22 16 46424 46408 131.817 111.625 0.0117424 0.00758126 2024-03-11T21:32:23.441 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:32:23.487 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:32:23.488 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:32:23.488 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:32:23.590 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:32:23.617 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:32:23.618 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:32:23.618 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:32:23.719 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:32:23.753 INFO:teuthology.orchestra.run.smithi092.stdout:osd.0: {} 2024-03-11T21:32:23.769 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:32:23.769 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:32:23.770 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:32:23.794 INFO:teuthology.orchestra.run.smithi092.stdout:osd.1: {} 2024-03-11T21:32:23.836 INFO:teuthology.orchestra.run.smithi092.stdout:osd.2: {} 2024-03-11T21:32:23.871 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:32:23.878 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:32:23 smithi092 ceph-mon[29572]: from='client.24469 v1:172.21.15.97:0/404208310' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:32:23.878 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:32:23 smithi092 ceph-mon[29572]: 6.b scrub starts 2024-03-11T21:32:23.878 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:32:23 smithi092 ceph-mon[29572]: 3.1 scrub starts 2024-03-11T21:32:23.878 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:32:23 smithi092 ceph-mon[29572]: 3.1 scrub ok 2024-03-11T21:32:23.879 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:32:23 smithi092 ceph-mon[29572]: 6.b scrub ok 2024-03-11T21:32:23.879 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:32:23 smithi092 ceph-mon[29572]: osdmap e167: 8 total, 8 up, 8 in 2024-03-11T21:32:23.879 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:32:23 smithi092 ceph-mon[29572]: 2.14 scrub starts 2024-03-11T21:32:23.879 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:32:23 smithi092 ceph-mon[29572]: 2.14 scrub ok 2024-03-11T21:32:23.879 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:32:23 smithi092 ceph-mon[29572]: osdmap e168: 8 total, 8 up, 8 in 2024-03-11T21:32:23.879 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:32:23 smithi092 ceph-mon[29572]: 5.b scrub starts 2024-03-11T21:32:23.879 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:32:23 smithi092 ceph-mon[29572]: 5.b scrub ok 2024-03-11T21:32:23.879 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:32:23 smithi092 ceph-mon[29572]: from='client.? v1:172.21.15.92:0/3730533100' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-11T21:32:23.879 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:32:23 smithi092 ceph-mon[37563]: from='client.24469 v1:172.21.15.97:0/404208310' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:32:23.879 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:32:23 smithi092 ceph-mon[37563]: 6.b scrub starts 2024-03-11T21:32:23.879 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:32:23 smithi092 ceph-mon[37563]: 3.1 scrub starts 2024-03-11T21:32:23.880 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:32:23 smithi092 ceph-mon[37563]: 3.1 scrub ok 2024-03-11T21:32:23.880 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:32:23 smithi092 ceph-mon[37563]: 6.b scrub ok 2024-03-11T21:32:23.880 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:32:23 smithi092 ceph-mon[37563]: osdmap e167: 8 total, 8 up, 8 in 2024-03-11T21:32:23.880 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:32:23 smithi092 ceph-mon[37563]: 2.14 scrub starts 2024-03-11T21:32:23.880 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:32:23 smithi092 ceph-mon[37563]: 2.14 scrub ok 2024-03-11T21:32:23.880 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:32:23 smithi092 ceph-mon[37563]: osdmap e168: 8 total, 8 up, 8 in 2024-03-11T21:32:23.880 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:32:23 smithi092 ceph-mon[37563]: 5.b scrub starts 2024-03-11T21:32:23.880 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:32:23 smithi092 ceph-mon[37563]: 5.b scrub ok 2024-03-11T21:32:23.880 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:32:23 smithi092 ceph-mon[37563]: from='client.? v1:172.21.15.92:0/3730533100' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-11T21:32:23.880 INFO:teuthology.orchestra.run.smithi092.stdout:osd.3: {} 2024-03-11T21:32:23.900 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:32:23.901 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:32:23.902 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:32:23.919 INFO:teuthology.orchestra.run.smithi092.stdout:osd.4: {} 2024-03-11T21:32:23.965 INFO:teuthology.orchestra.run.smithi092.stdout:osd.5: {} 2024-03-11T21:32:24.003 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:32:24.007 INFO:teuthology.orchestra.run.smithi092.stdout:osd.6: {} 2024-03-11T21:32:24.032 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:32:24.032 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:32:24.033 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:32:24.046 INFO:teuthology.orchestra.run.smithi092.stdout:osd.7: {} 2024-03-11T21:32:24.057 INFO:teuthology.orchestra.run.smithi092.stderr:osd.0: osd_enable_op_tracker = '' 2024-03-11T21:32:24.057 INFO:teuthology.orchestra.run.smithi092.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-11T21:32:24.057 INFO:teuthology.orchestra.run.smithi092.stderr:osd.2: osd_enable_op_tracker = '' 2024-03-11T21:32:24.057 INFO:teuthology.orchestra.run.smithi092.stderr:osd.3: osd_enable_op_tracker = '' 2024-03-11T21:32:24.057 INFO:teuthology.orchestra.run.smithi092.stderr:osd.4: osd_enable_op_tracker = '' 2024-03-11T21:32:24.057 INFO:teuthology.orchestra.run.smithi092.stderr:osd.5: osd_enable_op_tracker = '' 2024-03-11T21:32:24.057 INFO:teuthology.orchestra.run.smithi092.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-11T21:32:24.057 INFO:teuthology.orchestra.run.smithi092.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-11T21:32:24.076 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:32:23 smithi097 ceph-mon[33317]: from='client.24469 v1:172.21.15.97:0/404208310' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:32:24.076 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:32:23 smithi097 ceph-mon[33317]: 6.b scrub starts 2024-03-11T21:32:24.076 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:32:23 smithi097 ceph-mon[33317]: 3.1 scrub starts 2024-03-11T21:32:24.076 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:32:23 smithi097 ceph-mon[33317]: 3.1 scrub ok 2024-03-11T21:32:24.076 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:32:23 smithi097 ceph-mon[33317]: 6.b scrub ok 2024-03-11T21:32:24.076 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:32:23 smithi097 ceph-mon[33317]: osdmap e167: 8 total, 8 up, 8 in 2024-03-11T21:32:24.076 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:32:23 smithi097 ceph-mon[33317]: 2.14 scrub starts 2024-03-11T21:32:24.077 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:32:23 smithi097 ceph-mon[33317]: 2.14 scrub ok 2024-03-11T21:32:24.077 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:32:23 smithi097 ceph-mon[33317]: osdmap e168: 8 total, 8 up, 8 in 2024-03-11T21:32:24.077 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:32:23 smithi097 ceph-mon[33317]: 5.b scrub starts 2024-03-11T21:32:24.077 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:32:23 smithi097 ceph-mon[33317]: 5.b scrub ok 2024-03-11T21:32:24.077 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:32:23 smithi097 ceph-mon[33317]: from='client.? v1:172.21.15.92:0/3730533100' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-11T21:32:24.134 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:32:24.180 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:32:24.181 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:32:24.181 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:32:24.282 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:32:24.312 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:32:24.312 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:32:24.313 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:32:24.413 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:32:24.430 INFO:tasks.radosbench.radosbench.0.smithi092.stdout: 23 16 47839 47823 129.93 88.4375 0.0122029 0.00769193 2024-03-11T21:32:24.445 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:32:24.446 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:32:24.446 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:32:24.547 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:32:24.588 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:32:24.588 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:32:24.589 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:32:24.690 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:32:24.731 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:32:24.732 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:32:24.732 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:32:24.810 INFO:teuthology.orchestra.run.smithi092.stderr:noscrub is set 2024-03-11T21:32:24.833 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:32:24.874 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:32:24.874 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:32:24.875 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:32:24.976 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:32:25.007 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:32:25.008 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:32:25.008 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:32:25.054 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:32:24 smithi092 ceph-mon[29572]: pgmap v264: 148 pgs: 4 peering, 144 active+clean; 2.7 GiB data, 9.2 GiB used, 706 GiB / 715 GiB avail; 1.7 KiB/s rd, 157 MiB/s wr, 2.51k op/s; 0 B/s, 0 objects/s recovering 2024-03-11T21:32:25.054 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:32:24 smithi092 ceph-mon[29572]: 6.16 scrub starts 2024-03-11T21:32:25.054 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:32:24 smithi092 ceph-mon[29572]: 6.16 scrub ok 2024-03-11T21:32:25.054 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:32:24 smithi092 ceph-mon[29572]: 5.2 scrub starts 2024-03-11T21:32:25.055 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:32:24 smithi092 ceph-mon[29572]: 5.2 scrub ok 2024-03-11T21:32:25.055 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:32:24 smithi092 ceph-mon[29572]: from='client.? v1:172.21.15.92:0/208428906' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-11T21:32:25.055 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:32:24 smithi092 ceph-mon[29572]: from='client.25054 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-11T21:32:25.055 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:32:24 smithi092 ceph-mon[37563]: pgmap v264: 148 pgs: 4 peering, 144 active+clean; 2.7 GiB data, 9.2 GiB used, 706 GiB / 715 GiB avail; 1.7 KiB/s rd, 157 MiB/s wr, 2.51k op/s; 0 B/s, 0 objects/s recovering 2024-03-11T21:32:25.055 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:32:24 smithi092 ceph-mon[37563]: 6.16 scrub starts 2024-03-11T21:32:25.055 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:32:24 smithi092 ceph-mon[37563]: 6.16 scrub ok 2024-03-11T21:32:25.055 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:32:24 smithi092 ceph-mon[37563]: 5.2 scrub starts 2024-03-11T21:32:25.055 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:32:24 smithi092 ceph-mon[37563]: 5.2 scrub ok 2024-03-11T21:32:25.055 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:32:24 smithi092 ceph-mon[37563]: from='client.? v1:172.21.15.92:0/208428906' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-11T21:32:25.055 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:32:24 smithi092 ceph-mon[37563]: from='client.25054 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-11T21:32:25.076 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:32:24 smithi097 ceph-mon[33317]: pgmap v264: 148 pgs: 4 peering, 144 active+clean; 2.7 GiB data, 9.2 GiB used, 706 GiB / 715 GiB avail; 1.7 KiB/s rd, 157 MiB/s wr, 2.51k op/s; 0 B/s, 0 objects/s recovering 2024-03-11T21:32:25.076 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:32:24 smithi097 ceph-mon[33317]: 6.16 scrub starts 2024-03-11T21:32:25.076 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:32:24 smithi097 ceph-mon[33317]: 6.16 scrub ok 2024-03-11T21:32:25.076 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:32:24 smithi097 ceph-mon[33317]: 5.2 scrub starts 2024-03-11T21:32:25.076 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:32:24 smithi097 ceph-mon[33317]: 5.2 scrub ok 2024-03-11T21:32:25.076 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:32:24 smithi097 ceph-mon[33317]: from='client.? v1:172.21.15.92:0/208428906' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-11T21:32:25.076 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:32:24 smithi097 ceph-mon[33317]: from='client.25054 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-11T21:32:25.109 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:32:25.141 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:32:25.141 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:32:25.142 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:32:25.242 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:32:25.272 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:32:25.273 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:32:25.274 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:32:25.374 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:32:25.398 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:32:25.399 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:32:25.399 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:32:25.430 INFO:tasks.radosbench.radosbench.0.smithi092.stdout: 24 16 49217 49201 128.105 86.125 0.0023676 0.00776581 2024-03-11T21:32:25.500 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:32:25.538 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:32:25.539 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:32:25.539 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:32:25.643 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:32:25.681 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:32:25.681 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:32:25.682 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:32:25.783 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:32:25.811 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:32:25.812 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:32:25.812 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:32:25.913 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:32:25.942 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:32:25.942 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:32:25.943 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:32:26.043 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:32:26.066 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:32:26.067 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:32:26.067 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:32:26.076 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:32:25 smithi097 ceph-mon[33317]: 4.13 deep-scrub starts 2024-03-11T21:32:26.076 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:32:25 smithi097 ceph-mon[33317]: 4.13 deep-scrub ok 2024-03-11T21:32:26.076 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:32:25 smithi097 ceph-mon[33317]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-03-11T21:32:26.076 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:32:25 smithi097 ceph-mon[33317]: from='client.25054 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-03-11T21:32:26.076 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:32:25 smithi097 ceph-mon[33317]: osdmap e169: 8 total, 8 up, 8 in 2024-03-11T21:32:26.110 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:32:25 smithi092 ceph-mon[29572]: 4.13 deep-scrub starts 2024-03-11T21:32:26.110 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:32:25 smithi092 ceph-mon[29572]: 4.13 deep-scrub ok 2024-03-11T21:32:26.110 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:32:25 smithi092 ceph-mon[29572]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-03-11T21:32:26.111 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:32:25 smithi092 ceph-mon[29572]: from='client.25054 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-03-11T21:32:26.111 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:32:25 smithi092 ceph-mon[29572]: osdmap e169: 8 total, 8 up, 8 in 2024-03-11T21:32:26.111 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:32:25 smithi092 ceph-mon[37563]: 4.13 deep-scrub starts 2024-03-11T21:32:26.111 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:32:25 smithi092 ceph-mon[37563]: 4.13 deep-scrub ok 2024-03-11T21:32:26.111 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:32:25 smithi092 ceph-mon[37563]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-03-11T21:32:26.111 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:32:25 smithi092 ceph-mon[37563]: from='client.25054 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-03-11T21:32:26.111 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:32:25 smithi092 ceph-mon[37563]: osdmap e169: 8 total, 8 up, 8 in 2024-03-11T21:32:26.169 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:32:26.200 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:32:26.200 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:32:26.201 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:32:26.302 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:32:26.330 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:32:26.330 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:32:26.331 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:32:26.430 INFO:tasks.radosbench.radosbench.0.smithi092.stdout: 25 15 50527 50512 126.258 81.9375 0.00686575 0.00791378 2024-03-11T21:32:26.432 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:32:26.469 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:32:26.470 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:32:26.470 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:32:26.571 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:32:26.600 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:32:26.601 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:32:26.601 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:32:26.702 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:32:26.737 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:32:26.737 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:32:26.737 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:32:26.838 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:32:26.874 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:32:26.874 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:32:26.875 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:32:26.975 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:32:27.017 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:32:27.017 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:32:27.018 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:32:27.076 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:32:26 smithi097 ceph-mon[33317]: pgmap v267: 148 pgs: 4 peering, 144 active+clean; 2.7 GiB data, 9.2 GiB used, 706 GiB / 715 GiB avail; 1.2 KiB/s rd, 138 MiB/s wr, 2.20k op/s; 0 B/s, 1 objects/s recovering 2024-03-11T21:32:27.110 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:32:26 smithi092 ceph-mon[29572]: pgmap v267: 148 pgs: 4 peering, 144 active+clean; 2.7 GiB data, 9.2 GiB used, 706 GiB / 715 GiB avail; 1.2 KiB/s rd, 138 MiB/s wr, 2.20k op/s; 0 B/s, 1 objects/s recovering 2024-03-11T21:32:27.110 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:32:26 smithi092 ceph-mon[37563]: pgmap v267: 148 pgs: 4 peering, 144 active+clean; 2.7 GiB data, 9.2 GiB used, 706 GiB / 715 GiB avail; 1.2 KiB/s rd, 138 MiB/s wr, 2.20k op/s; 0 B/s, 1 objects/s recovering 2024-03-11T21:32:27.119 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:32:27.149 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:32:27.150 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:32:27.150 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:32:27.251 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:32:27.294 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:32:27.294 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:32:27.294 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:32:27.395 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:32:27.430 INFO:tasks.radosbench.radosbench.0.smithi092.stdout: 26 16 52160 52144 125.325 102 0.0122245 0.0079742 2024-03-11T21:32:27.442 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:32:27.442 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:32:27.443 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:32:27.544 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:32:27.581 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:32:27.581 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:32:27.582 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:32:27.682 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:32:27.714 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:32:27.715 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:32:27.716 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:32:27.816 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:32:27.870 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:32:27.871 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:32:27.871 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:32:27.972 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:32:28.003 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:32:28.003 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:32:28.004 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:32:28.031 INFO:journalctl@ceph.mgr.y.smithi092.stdout:Mar 11 21:32:27 smithi092 ceph-00db0016-dfed-11ee-95c8-87774f69a715-mgr-y[29801]: ::ffff:172.21.15.97 - - [11/Mar/2024:21:32:27] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-11T21:32:28.078 INFO:teuthology.orchestra.run.smithi092.stderr:Inferring config /var/lib/ceph/00db0016-dfed-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T21:32:28.104 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:32:28.135 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:32:28.136 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:32:28.136 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:32:28.217 INFO:teuthology.orchestra.run.smithi092.stderr:set pool 8 pgp_num to 16 2024-03-11T21:32:28.237 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:32:28.278 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:32:28.279 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:32:28.279 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:32:28.380 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:32:28.423 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:32:28.423 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:32:28.424 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:32:28.430 INFO:tasks.radosbench.radosbench.0.smithi092.stdout: 27 16 53564 53548 123.933 87.75 0.00835034 0.008064 2024-03-11T21:32:28.525 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:32:28.556 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:32:28.557 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:32:28.558 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:32:28.658 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:32:28.698 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:32:28.698 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:32:28.699 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:32:28.800 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:32:28.835 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:32:28.836 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:32:28.837 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:32:28.937 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:32:28.976 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:32:28.976 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:32:28.977 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:32:29.076 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:32:28 smithi097 ceph-mon[33317]: pgmap v268: 148 pgs: 3 peering, 145 active+clean; 2.8 GiB data, 9.6 GiB used, 706 GiB / 715 GiB avail; 961 B/s rd, 126 MiB/s wr, 2.02k op/s; 0 B/s, 1 objects/s recovering 2024-03-11T21:32:29.076 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:32:28 smithi097 ceph-mon[33317]: from='client.? v1:172.21.15.92:0/822103310' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "unique_pool_1", "var": "pgp_num", "val": "16"}]: dispatch 2024-03-11T21:32:29.076 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:32:28 smithi097 ceph-mon[33317]: from='client.25060 ' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "unique_pool_1", "var": "pgp_num", "val": "16"}]: dispatch 2024-03-11T21:32:29.076 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:32:28 smithi097 ceph-mon[33317]: from='client.25060 ' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_1", "var": "pgp_num", "val": "16"}]': finished 2024-03-11T21:32:29.076 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:32:28 smithi097 ceph-mon[33317]: osdmap e170: 8 total, 8 up, 8 in 2024-03-11T21:32:29.078 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:32:29.111 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:32:28 smithi092 ceph-mon[29572]: pgmap v268: 148 pgs: 3 peering, 145 active+clean; 2.8 GiB data, 9.6 GiB used, 706 GiB / 715 GiB avail; 961 B/s rd, 126 MiB/s wr, 2.02k op/s; 0 B/s, 1 objects/s recovering 2024-03-11T21:32:29.111 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:32:28 smithi092 ceph-mon[29572]: from='client.? v1:172.21.15.92:0/822103310' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "unique_pool_1", "var": "pgp_num", "val": "16"}]: dispatch 2024-03-11T21:32:29.111 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:32:28 smithi092 ceph-mon[29572]: from='client.25060 ' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "unique_pool_1", "var": "pgp_num", "val": "16"}]: dispatch 2024-03-11T21:32:29.111 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:32:28 smithi092 ceph-mon[29572]: from='client.25060 ' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_1", "var": "pgp_num", "val": "16"}]': finished 2024-03-11T21:32:29.112 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:32:28 smithi092 ceph-mon[29572]: osdmap e170: 8 total, 8 up, 8 in 2024-03-11T21:32:29.112 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:32:28 smithi092 ceph-mon[37563]: pgmap v268: 148 pgs: 3 peering, 145 active+clean; 2.8 GiB data, 9.6 GiB used, 706 GiB / 715 GiB avail; 961 B/s rd, 126 MiB/s wr, 2.02k op/s; 0 B/s, 1 objects/s recovering 2024-03-11T21:32:29.112 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:32:28 smithi092 ceph-mon[37563]: from='client.? v1:172.21.15.92:0/822103310' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "unique_pool_1", "var": "pgp_num", "val": "16"}]: dispatch 2024-03-11T21:32:29.112 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:32:28 smithi092 ceph-mon[37563]: from='client.25060 ' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "unique_pool_1", "var": "pgp_num", "val": "16"}]: dispatch 2024-03-11T21:32:29.112 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:32:28 smithi092 ceph-mon[37563]: from='client.25060 ' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_1", "var": "pgp_num", "val": "16"}]': finished 2024-03-11T21:32:29.112 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:32:28 smithi092 ceph-mon[37563]: osdmap e170: 8 total, 8 up, 8 in 2024-03-11T21:32:29.118 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:32:29.119 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:32:29.119 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:32:29.219 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:32:29.257 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:32:29.258 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:32:29.258 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:32:29.359 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:32:29.398 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:32:29.398 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:32:29.398 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:32:29.420 DEBUG:teuthology.orchestra.run.smithi092:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid 00db0016-dfed-11ee-95c8-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-03-11T21:32:29.430 INFO:tasks.radosbench.radosbench.0.smithi092.stdout: 28 16 55162 55146 123.074 99.875 0.00754388 0.00812013 2024-03-11T21:32:29.499 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:32:29.528 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:32:29.528 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:32:29.529 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:32:29.630 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:32:29.660 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:32:29.661 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:32:29.661 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:32:29.762 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:32:29.793 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:32:29.794 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:32:29.794 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:32:29.895 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:32:29.937 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:32:29.938 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:32:29.938 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:32:30.039 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:32:30.069 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:32:30.070 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:32:30.070 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:32:30.079 DEBUG:teuthology.orchestra.run.smithi092:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid 00db0016-dfed-11ee-95c8-87774f69a715 -- ceph osd set nodeep-scrub 2024-03-11T21:32:30.170 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:32:30.216 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:32:30.217 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:32:30.218 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:32:30.318 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:32:30.330 INFO:teuthology.orchestra.run.smithi092.stderr:Inferring config /var/lib/ceph/00db0016-dfed-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T21:32:30.344 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:32:30.345 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:32:30.345 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:32:30.431 INFO:tasks.radosbench.radosbench.0.smithi092.stdout: 29 16 56585 56569 121.892 88.9375 0.00431985 0.00819838 2024-03-11T21:32:30.446 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:32:30.494 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:32:30.494 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:32:30.495 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:32:30.595 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:32:30.648 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:32:30.649 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:32:30.649 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:32:30.750 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:32:30.781 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:32:30.781 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:32:30.782 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:32:30.883 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:32:30.915 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:32:30.915 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:32:30.916 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:32:31.017 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:32:31.045 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:32:31.046 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:32:31.046 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:32:31.063 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:32:30 smithi092 ceph-mon[37563]: pgmap v270: 148 pgs: 148 active+clean; 3.2 GiB data, 11 GiB used, 705 GiB / 715 GiB avail; 853 B/s rd, 79 MiB/s wr, 1.26k op/s; 18 B/s, 0 objects/s recovering 2024-03-11T21:32:31.064 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:32:30 smithi092 ceph-mon[29572]: pgmap v270: 148 pgs: 148 active+clean; 3.2 GiB data, 11 GiB used, 705 GiB / 715 GiB avail; 853 B/s rd, 79 MiB/s wr, 1.26k op/s; 18 B/s, 0 objects/s recovering 2024-03-11T21:32:31.076 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:32:30 smithi097 ceph-mon[33317]: pgmap v270: 148 pgs: 148 active+clean; 3.2 GiB data, 11 GiB used, 705 GiB / 715 GiB avail; 853 B/s rd, 79 MiB/s wr, 1.26k op/s; 18 B/s, 0 objects/s recovering 2024-03-11T21:32:31.147 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:32:31.192 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:32:31.192 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:32:31.193 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:32:31.294 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:32:31.323 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:32:31.324 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:32:31.324 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:32:31.425 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:32:31.431 INFO:tasks.radosbench.radosbench.0.smithi092.stdout: 30 16 58009 57993 120.795 89 0.00674425 0.00827299 2024-03-11T21:32:31.463 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:32:31.464 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:32:31.464 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:32:31.565 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:32:31.593 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:32:31.594 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:32:31.594 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:32:31.652 INFO:teuthology.orchestra.run.smithi092.stdout:{ 2024-03-11T21:32:31.652 INFO:teuthology.orchestra.run.smithi092.stdout: "size": 20, 2024-03-11T21:32:31.652 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 600, 2024-03-11T21:32:31.652 INFO:teuthology.orchestra.run.smithi092.stdout: "ops": [ 2024-03-11T21:32:31.652 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:32:31.652 INFO:teuthology.orchestra.run.smithi092.stdout: "description": "osd_op(client.34715.0:69655 7.2 7:4c25af8c:::benchmark_data_smithi092_76040_object69654:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e109)", 2024-03-11T21:32:31.652 INFO:teuthology.orchestra.run.smithi092.stdout: "initiated_at": "2024-03-11T21:28:24.370889+0000", 2024-03-11T21:32:31.652 INFO:teuthology.orchestra.run.smithi092.stdout: "age": 246.92243688600001, 2024-03-11T21:32:31.652 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.11310818, 2024-03-11T21:32:31.652 INFO:teuthology.orchestra.run.smithi092.stdout: "type_data": { 2024-03-11T21:32:31.653 INFO:teuthology.orchestra.run.smithi092.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-11T21:32:31.653 INFO:teuthology.orchestra.run.smithi092.stdout: "client_info": { 2024-03-11T21:32:31.653 INFO:teuthology.orchestra.run.smithi092.stdout: "client": "client.34715", 2024-03-11T21:32:31.653 INFO:teuthology.orchestra.run.smithi092.stdout: "client_addr": "v1:172.21.15.92:0/2403132573", 2024-03-11T21:32:31.653 INFO:teuthology.orchestra.run.smithi092.stdout: "tid": 69655 2024-03-11T21:32:31.653 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:32:31.653 INFO:teuthology.orchestra.run.smithi092.stdout: "events": [ 2024-03-11T21:32:31.653 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:32:31.653 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "initiated", 2024-03-11T21:32:31.653 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:28:24.370889+0000", 2024-03-11T21:32:31.653 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-03-11T21:32:31.653 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:32:31.653 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:32:31.653 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "header_read", 2024-03-11T21:32:31.654 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:28:24.370889+0000", 2024-03-11T21:32:31.654 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-03-11T21:32:31.654 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:32:31.654 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:32:31.654 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "throttled", 2024-03-11T21:32:31.654 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:28:24.370890+0000", 2024-03-11T21:32:31.654 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 7.9999999999999996e-07 2024-03-11T21:32:31.654 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:32:31.654 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:32:31.654 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "all_read", 2024-03-11T21:32:31.654 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:28:24.370911+0000", 2024-03-11T21:32:31.654 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 2.1903999999999999e-05 2024-03-11T21:32:31.654 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:32:31.655 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:32:31.655 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "dispatched", 2024-03-11T21:32:31.655 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:28:24.370918+0000", 2024-03-11T21:32:31.655 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 7.0299999999999996e-06 2024-03-11T21:32:31.655 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:32:31.655 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:32:31.655 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "queued_for_pg", 2024-03-11T21:32:31.655 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:28:24.370925+0000", 2024-03-11T21:32:31.655 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 6.6409999999999996e-06 2024-03-11T21:32:31.655 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:32:31.655 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:32:31.655 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-03-11T21:32:31.655 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:28:24.379101+0000", 2024-03-11T21:32:31.656 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.0081760830000000007 2024-03-11T21:32:31.656 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:32:31.656 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:32:31.656 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "started", 2024-03-11T21:32:31.656 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:28:24.379231+0000", 2024-03-11T21:32:31.656 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00013012500000000001 2024-03-11T21:32:31.656 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:32:31.656 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:32:31.656 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "waiting for subops from 1,2", 2024-03-11T21:32:31.656 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:28:24.379367+0000", 2024-03-11T21:32:31.656 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.000136174 2024-03-11T21:32:31.656 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:32:31.656 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:32:31.656 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "sub_op_commit_rec", 2024-03-11T21:32:31.656 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:28:24.385163+0000", 2024-03-11T21:32:31.657 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.005795471 2024-03-11T21:32:31.657 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:32:31.657 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:32:31.657 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "op_commit", 2024-03-11T21:32:31.657 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:28:24.387193+0000", 2024-03-11T21:32:31.657 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.0020303029999999998 2024-03-11T21:32:31.657 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:32:31.657 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:32:31.657 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "sub_op_commit_rec", 2024-03-11T21:32:31.657 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:28:24.483891+0000", 2024-03-11T21:32:31.657 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.096697913999999996 2024-03-11T21:32:31.657 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:32:31.657 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:32:31.657 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "commit_sent", 2024-03-11T21:32:31.657 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:28:24.483948+0000", 2024-03-11T21:32:31.658 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 5.6449000000000001e-05 2024-03-11T21:32:31.658 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:32:31.658 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:32:31.658 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "done", 2024-03-11T21:32:31.658 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:28:24.483997+0000", 2024-03-11T21:32:31.658 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 4.9286000000000002e-05 2024-03-11T21:32:31.658 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-03-11T21:32:31.658 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-03-11T21:32:31.658 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-03-11T21:32:31.658 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:32:31.658 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:32:31.658 INFO:teuthology.orchestra.run.smithi092.stdout: "description": "osd_op(client.34715.0:69659 7.2 7:4b3565ee:::benchmark_data_smithi092_76040_object69658:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e109)", 2024-03-11T21:32:31.658 INFO:teuthology.orchestra.run.smithi092.stdout: "initiated_at": "2024-03-11T21:28:24.374833+0000", 2024-03-11T21:32:31.659 INFO:teuthology.orchestra.run.smithi092.stdout: "age": 246.91849284099999, 2024-03-11T21:32:31.659 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.10933362100000001, 2024-03-11T21:32:31.659 INFO:teuthology.orchestra.run.smithi092.stdout: "type_data": { 2024-03-11T21:32:31.659 INFO:teuthology.orchestra.run.smithi092.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-11T21:32:31.659 INFO:teuthology.orchestra.run.smithi092.stdout: "client_info": { 2024-03-11T21:32:31.659 INFO:teuthology.orchestra.run.smithi092.stdout: "client": "client.34715", 2024-03-11T21:32:31.659 INFO:teuthology.orchestra.run.smithi092.stdout: "client_addr": "v1:172.21.15.92:0/2403132573", 2024-03-11T21:32:31.659 INFO:teuthology.orchestra.run.smithi092.stdout: "tid": 69659 2024-03-11T21:32:31.659 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:32:31.659 INFO:teuthology.orchestra.run.smithi092.stdout: "events": [ 2024-03-11T21:32:31.659 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:32:31.659 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "initiated", 2024-03-11T21:32:31.659 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:28:24.374833+0000", 2024-03-11T21:32:31.659 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-03-11T21:32:31.660 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:32:31.660 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:32:31.660 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "header_read", 2024-03-11T21:32:31.660 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:28:24.374833+0000", 2024-03-11T21:32:31.660 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-03-11T21:32:31.660 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:32:31.660 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:32:31.660 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "throttled", 2024-03-11T21:32:31.660 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:28:24.374834+0000", 2024-03-11T21:32:31.660 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.5379999999999999e-06 2024-03-11T21:32:31.660 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:32:31.660 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:32:31.661 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "all_read", 2024-03-11T21:32:31.661 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:28:24.374855+0000", 2024-03-11T21:32:31.661 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 2.0448000000000001e-05 2024-03-11T21:32:31.661 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:32:31.661 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:32:31.661 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "dispatched", 2024-03-11T21:32:31.661 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:28:24.374865+0000", 2024-03-11T21:32:31.661 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 9.8640000000000008e-06 2024-03-11T21:32:31.661 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:32:31.661 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:32:31.661 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "queued_for_pg", 2024-03-11T21:32:31.661 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:28:24.374872+0000", 2024-03-11T21:32:31.661 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 7.3339999999999996e-06 2024-03-11T21:32:31.662 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:32:31.662 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:32:31.662 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-03-11T21:32:31.662 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:28:24.382375+0000", 2024-03-11T21:32:31.662 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.0075029800000000002 2024-03-11T21:32:31.662 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:32:31.662 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:32:31.662 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "started", 2024-03-11T21:32:31.663 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:28:24.382484+0000", 2024-03-11T21:32:31.663 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00010863500000000001 2024-03-11T21:32:31.663 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:32:31.663 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:32:31.663 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "waiting for subops from 1,2", 2024-03-11T21:32:31.663 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:28:24.382608+0000", 2024-03-11T21:32:31.663 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00012472300000000001 2024-03-11T21:32:31.663 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:32:31.663 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:32:31.663 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "op_commit", 2024-03-11T21:32:31.663 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:28:24.387200+0000", 2024-03-11T21:32:31.664 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.0045912080000000003 2024-03-11T21:32:31.664 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:32:31.664 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:32:31.664 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "sub_op_commit_rec", 2024-03-11T21:32:31.664 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:28:24.484040+0000", 2024-03-11T21:32:31.664 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.096840741999999994 2024-03-11T21:32:31.664 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:32:31.664 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:32:31.664 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "sub_op_commit_rec", 2024-03-11T21:32:31.664 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:28:24.484081+0000", 2024-03-11T21:32:31.665 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 4.0441000000000001e-05 2024-03-11T21:32:31.665 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:32:31.665 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:32:31.665 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "commit_sent", 2024-03-11T21:32:31.665 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:28:24.484131+0000", 2024-03-11T21:32:31.665 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 5.0065999999999998e-05 2024-03-11T21:32:31.665 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:32:31.665 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:32:31.665 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "done", 2024-03-11T21:32:31.665 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:28:24.484166+0000", 2024-03-11T21:32:31.665 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 3.5642000000000002e-05 2024-03-11T21:32:31.665 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-03-11T21:32:31.665 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-03-11T21:32:31.665 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-03-11T21:32:31.666 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:32:31.666 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:32:31.666 INFO:teuthology.orchestra.run.smithi092.stdout: "description": "osd_op(client.34715.0:104673 7.a 7:51b8768e:::benchmark_data_smithi092_76040_object104672:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e117)", 2024-03-11T21:32:31.666 INFO:teuthology.orchestra.run.smithi092.stdout: "initiated_at": "2024-03-11T21:28:57.285489+0000", 2024-03-11T21:32:31.666 INFO:teuthology.orchestra.run.smithi092.stdout: "age": 214.00783703499999, 2024-03-11T21:32:31.666 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.103913641, 2024-03-11T21:32:31.666 INFO:teuthology.orchestra.run.smithi092.stdout: "type_data": { 2024-03-11T21:32:31.666 INFO:teuthology.orchestra.run.smithi092.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-11T21:32:31.666 INFO:teuthology.orchestra.run.smithi092.stdout: "client_info": { 2024-03-11T21:32:31.666 INFO:teuthology.orchestra.run.smithi092.stdout: "client": "client.34715", 2024-03-11T21:32:31.666 INFO:teuthology.orchestra.run.smithi092.stdout: "client_addr": "v1:172.21.15.92:0/2403132573", 2024-03-11T21:32:31.666 INFO:teuthology.orchestra.run.smithi092.stdout: "tid": 104673 2024-03-11T21:32:31.666 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:32:31.667 INFO:teuthology.orchestra.run.smithi092.stdout: "events": [ 2024-03-11T21:32:31.667 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:32:31.667 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "initiated", 2024-03-11T21:32:31.667 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:28:57.285489+0000", 2024-03-11T21:32:31.667 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-03-11T21:32:31.667 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:32:31.667 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:32:31.667 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "header_read", 2024-03-11T21:32:31.667 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:28:57.285489+0000", 2024-03-11T21:32:31.667 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-03-11T21:32:31.667 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:32:31.667 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:32:31.667 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "throttled", 2024-03-11T21:32:31.667 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:28:57.285491+0000", 2024-03-11T21:32:31.667 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 2.5809999999999999e-06 2024-03-11T21:32:31.668 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:32:31.668 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:32:31.668 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "all_read", 2024-03-11T21:32:31.668 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:28:57.285520+0000", 2024-03-11T21:32:31.668 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 2.8486000000000001e-05 2024-03-11T21:32:31.668 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:32:31.668 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:32:31.668 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "dispatched", 2024-03-11T21:32:31.668 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:28:57.285533+0000", 2024-03-11T21:32:31.668 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.3509999999999999e-05 2024-03-11T21:32:31.668 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:32:31.668 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:32:31.668 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "queued_for_pg", 2024-03-11T21:32:31.668 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:28:57.285542+0000", 2024-03-11T21:32:31.669 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 8.7630000000000006e-06 2024-03-11T21:32:31.669 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:32:31.669 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:32:31.669 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-03-11T21:32:31.669 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:28:57.343895+0000", 2024-03-11T21:32:31.669 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.058353500000000003 2024-03-11T21:32:31.669 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:32:31.669 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:32:31.669 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "started", 2024-03-11T21:32:31.669 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:28:57.344035+0000", 2024-03-11T21:32:31.669 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00013953899999999999 2024-03-11T21:32:31.669 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:32:31.669 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:32:31.669 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "waiting for subops from 2,5", 2024-03-11T21:32:31.669 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:28:57.344625+0000", 2024-03-11T21:32:31.670 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00059037599999999999 2024-03-11T21:32:31.670 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:32:31.670 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:32:31.670 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "sub_op_commit_rec", 2024-03-11T21:32:31.670 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:28:57.353304+0000", 2024-03-11T21:32:31.670 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.0086788860000000002 2024-03-11T21:32:31.670 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:32:31.670 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:32:31.670 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "op_commit", 2024-03-11T21:32:31.670 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:28:57.356204+0000", 2024-03-11T21:32:31.670 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.0028994720000000002 2024-03-11T21:32:31.670 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:32:31.670 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:32:31.670 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "sub_op_commit_rec", 2024-03-11T21:32:31.671 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:28:57.389267+0000", 2024-03-11T21:32:31.671 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.033063642999999997 2024-03-11T21:32:31.671 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:32:31.671 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:32:31.671 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "commit_sent", 2024-03-11T21:32:31.671 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:28:57.389358+0000", 2024-03-11T21:32:31.671 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 9.0493000000000002e-05 2024-03-11T21:32:31.671 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:32:31.671 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:32:31.671 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "done", 2024-03-11T21:32:31.671 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:28:57.389402+0000", 2024-03-11T21:32:31.671 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 4.4391999999999998e-05 2024-03-11T21:32:31.671 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-03-11T21:32:31.671 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-03-11T21:32:31.672 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-03-11T21:32:31.672 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:32:31.672 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:32:31.672 INFO:teuthology.orchestra.run.smithi092.stdout: "description": "osd_repop(client.34715.0:116274 7.7 e119/99 7:ec7fdfb9:::benchmark_data_smithi092_76040_object116273:head v 119'7279, mlcod=119'7277)", 2024-03-11T21:32:31.672 INFO:teuthology.orchestra.run.smithi092.stdout: "initiated_at": "2024-03-11T21:29:08.767082+0000", 2024-03-11T21:32:31.672 INFO:teuthology.orchestra.run.smithi092.stdout: "age": 202.52624350400001, 2024-03-11T21:32:31.672 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.11052577099999999, 2024-03-11T21:32:31.672 INFO:teuthology.orchestra.run.smithi092.stdout: "type_data": { 2024-03-11T21:32:31.672 INFO:teuthology.orchestra.run.smithi092.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-11T21:32:31.672 INFO:teuthology.orchestra.run.smithi092.stdout: "events": [ 2024-03-11T21:32:31.672 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:32:31.672 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "initiated", 2024-03-11T21:32:31.672 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:29:08.767082+0000", 2024-03-11T21:32:31.672 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-03-11T21:32:31.673 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:32:31.673 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:32:31.673 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "header_read", 2024-03-11T21:32:31.673 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:29:08.767082+0000", 2024-03-11T21:32:31.673 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-03-11T21:32:31.673 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:32:31.673 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:32:31.673 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "throttled", 2024-03-11T21:32:31.673 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:29:08.767084+0000", 2024-03-11T21:32:31.673 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.643e-06 2024-03-11T21:32:31.673 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:32:31.673 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:32:31.673 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "all_read", 2024-03-11T21:32:31.673 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:29:08.767116+0000", 2024-03-11T21:32:31.674 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 3.2017000000000002e-05 2024-03-11T21:32:31.674 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:32:31.674 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:32:31.674 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "dispatched", 2024-03-11T21:32:31.674 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:29:08.767124+0000", 2024-03-11T21:32:31.674 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 8.5739999999999996e-06 2024-03-11T21:32:31.674 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:32:31.674 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:32:31.674 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "queued_for_pg", 2024-03-11T21:32:31.674 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:29:08.767131+0000", 2024-03-11T21:32:31.674 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 6.545e-06 2024-03-11T21:32:31.674 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:32:31.674 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:32:31.675 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-03-11T21:32:31.675 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:29:08.767220+0000", 2024-03-11T21:32:31.675 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 8.9470999999999998e-05 2024-03-11T21:32:31.675 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:32:31.675 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:32:31.675 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "started", 2024-03-11T21:32:31.675 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:29:08.767239+0000", 2024-03-11T21:32:31.675 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.8865e-05 2024-03-11T21:32:31.675 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:32:31.675 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:32:31.675 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "commit_sent", 2024-03-11T21:32:31.675 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:29:08.770981+0000", 2024-03-11T21:32:31.675 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.0037422570000000001 2024-03-11T21:32:31.675 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:32:31.675 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:32:31.676 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "done", 2024-03-11T21:32:31.676 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:29:08.877608+0000", 2024-03-11T21:32:31.676 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.106626399 2024-03-11T21:32:31.676 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-03-11T21:32:31.676 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-03-11T21:32:31.676 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-03-11T21:32:31.676 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:32:31.676 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:32:31.676 INFO:teuthology.orchestra.run.smithi092.stdout: "description": "osd_repop(client.34715.0:116275 7.1 e119/99 7:889dcf6e:::benchmark_data_smithi092_76040_object116274:head v 119'7376, mlcod=119'7371)", 2024-03-11T21:32:31.676 INFO:teuthology.orchestra.run.smithi092.stdout: "initiated_at": "2024-03-11T21:29:08.767196+0000", 2024-03-11T21:32:31.676 INFO:teuthology.orchestra.run.smithi092.stdout: "age": 202.52612973199999, 2024-03-11T21:32:31.676 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.110578172, 2024-03-11T21:32:31.676 INFO:teuthology.orchestra.run.smithi092.stdout: "type_data": { 2024-03-11T21:32:31.676 INFO:teuthology.orchestra.run.smithi092.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-11T21:32:31.676 INFO:teuthology.orchestra.run.smithi092.stdout: "events": [ 2024-03-11T21:32:31.677 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:32:31.677 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "initiated", 2024-03-11T21:32:31.677 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:29:08.767196+0000", 2024-03-11T21:32:31.677 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-03-11T21:32:31.677 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:32:31.677 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:32:31.677 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "header_read", 2024-03-11T21:32:31.677 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:29:08.767196+0000", 2024-03-11T21:32:31.677 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-03-11T21:32:31.677 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:32:31.677 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:32:31.677 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "throttled", 2024-03-11T21:32:31.677 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:29:08.767197+0000", 2024-03-11T21:32:31.677 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.3960000000000001e-06 2024-03-11T21:32:31.677 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:32:31.678 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:32:31.678 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "all_read", 2024-03-11T21:32:31.678 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:29:08.767237+0000", 2024-03-11T21:32:31.678 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 3.9617999999999999e-05 2024-03-11T21:32:31.678 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:32:31.678 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:32:31.678 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "dispatched", 2024-03-11T21:32:31.678 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:29:08.767247+0000", 2024-03-11T21:32:31.678 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.0074999999999999e-05 2024-03-11T21:32:31.678 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:32:31.678 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:32:31.678 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "queued_for_pg", 2024-03-11T21:32:31.678 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:29:08.767254+0000", 2024-03-11T21:32:31.678 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 6.7680000000000003e-06 2024-03-11T21:32:31.679 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:32:31.679 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:32:31.679 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-03-11T21:32:31.679 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:29:08.767332+0000", 2024-03-11T21:32:31.679 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 7.8096999999999994e-05 2024-03-11T21:32:31.679 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:32:31.679 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:32:31.679 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "started", 2024-03-11T21:32:31.679 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:29:08.767359+0000", 2024-03-11T21:32:31.679 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 2.6879000000000001e-05 2024-03-11T21:32:31.679 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:32:31.679 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:32:31.679 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "commit_sent", 2024-03-11T21:32:31.680 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:29:08.770761+0000", 2024-03-11T21:32:31.680 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.0034018030000000001 2024-03-11T21:32:31.680 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:32:31.680 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:32:31.680 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "done", 2024-03-11T21:32:31.680 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:29:08.877774+0000", 2024-03-11T21:32:31.680 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.10701353600000001 2024-03-11T21:32:31.680 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-03-11T21:32:31.680 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-03-11T21:32:31.680 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-03-11T21:32:31.680 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:32:31.680 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:32:31.680 INFO:teuthology.orchestra.run.smithi092.stdout: "description": "osd_op(client.34715.0:116278 7.a 7:58f001dc:::benchmark_data_smithi092_76040_object116277:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e119)", 2024-03-11T21:32:31.680 INFO:teuthology.orchestra.run.smithi092.stdout: "initiated_at": "2024-03-11T21:29:08.767691+0000", 2024-03-11T21:32:31.681 INFO:teuthology.orchestra.run.smithi092.stdout: "age": 202.525634938, 2024-03-11T21:32:31.681 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.117725041, 2024-03-11T21:32:31.681 INFO:teuthology.orchestra.run.smithi092.stdout: "type_data": { 2024-03-11T21:32:31.681 INFO:teuthology.orchestra.run.smithi092.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-11T21:32:31.681 INFO:teuthology.orchestra.run.smithi092.stdout: "client_info": { 2024-03-11T21:32:31.681 INFO:teuthology.orchestra.run.smithi092.stdout: "client": "client.34715", 2024-03-11T21:32:31.681 INFO:teuthology.orchestra.run.smithi092.stdout: "client_addr": "v1:172.21.15.92:0/2403132573", 2024-03-11T21:32:31.681 INFO:teuthology.orchestra.run.smithi092.stdout: "tid": 116278 2024-03-11T21:32:31.681 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:32:31.681 INFO:teuthology.orchestra.run.smithi092.stdout: "events": [ 2024-03-11T21:32:31.681 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:32:31.681 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "initiated", 2024-03-11T21:32:31.681 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:29:08.767691+0000", 2024-03-11T21:32:31.681 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-03-11T21:32:31.682 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:32:31.682 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:32:31.682 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "header_read", 2024-03-11T21:32:31.682 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:29:08.767691+0000", 2024-03-11T21:32:31.682 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-03-11T21:32:31.682 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:32:31.682 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:32:31.682 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "throttled", 2024-03-11T21:32:31.682 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:29:08.767693+0000", 2024-03-11T21:32:31.682 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 2.5289999999999998e-06 2024-03-11T21:32:31.682 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:32:31.682 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:32:31.682 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "all_read", 2024-03-11T21:32:31.683 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:29:08.767717+0000", 2024-03-11T21:32:31.683 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 2.4082000000000001e-05 2024-03-11T21:32:31.683 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:32:31.683 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:32:31.683 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "dispatched", 2024-03-11T21:32:31.683 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:29:08.767730+0000", 2024-03-11T21:32:31.683 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.2289e-05 2024-03-11T21:32:31.683 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:32:31.683 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:32:31.683 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "queued_for_pg", 2024-03-11T21:32:31.683 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:29:08.767738+0000", 2024-03-11T21:32:31.683 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 7.9249999999999995e-06 2024-03-11T21:32:31.683 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:32:31.683 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:32:31.684 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-03-11T21:32:31.684 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:29:08.767803+0000", 2024-03-11T21:32:31.684 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 6.5607999999999999e-05 2024-03-11T21:32:31.684 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:32:31.684 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:32:31.684 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "started", 2024-03-11T21:32:31.684 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:29:08.767938+0000", 2024-03-11T21:32:31.684 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00013469299999999999 2024-03-11T21:32:31.684 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:32:31.684 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:32:31.684 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "waiting for subops from 2,5", 2024-03-11T21:32:31.684 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:29:08.768092+0000", 2024-03-11T21:32:31.684 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00015369900000000001 2024-03-11T21:32:31.684 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:32:31.685 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:32:31.685 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "op_commit", 2024-03-11T21:32:31.685 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:29:08.877559+0000", 2024-03-11T21:32:31.685 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.109467592 2024-03-11T21:32:31.685 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:32:31.685 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:32:31.685 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "sub_op_commit_rec", 2024-03-11T21:32:31.685 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:29:08.877873+0000", 2024-03-11T21:32:31.685 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00031346899999999998 2024-03-11T21:32:31.685 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:32:31.685 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:32:31.686 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "sub_op_commit_rec", 2024-03-11T21:32:31.686 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:29:08.885310+0000", 2024-03-11T21:32:31.686 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.0074369680000000004 2024-03-11T21:32:31.686 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:32:31.686 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:32:31.686 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "commit_sent", 2024-03-11T21:32:31.686 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:29:08.885368+0000", 2024-03-11T21:32:31.686 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 5.8349e-05 2024-03-11T21:32:31.686 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:32:31.686 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:32:31.686 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "done", 2024-03-11T21:32:31.686 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:29:08.885416+0000", 2024-03-11T21:32:31.686 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 4.7837999999999998e-05 2024-03-11T21:32:31.686 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-03-11T21:32:31.687 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-03-11T21:32:31.687 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-03-11T21:32:31.687 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:32:31.687 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:32:31.687 INFO:teuthology.orchestra.run.smithi092.stdout: "description": "osd_repop(client.34715.0:116273 7.c e119/99 7:37145dec:::benchmark_data_smithi092_76040_object116272:head v 119'7271, mlcod=119'7267)", 2024-03-11T21:32:31.687 INFO:teuthology.orchestra.run.smithi092.stdout: "initiated_at": "2024-03-11T21:29:08.769737+0000", 2024-03-11T21:32:31.687 INFO:teuthology.orchestra.run.smithi092.stdout: "age": 202.52358815100001, 2024-03-11T21:32:31.687 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.10805804400000001, 2024-03-11T21:32:31.687 INFO:teuthology.orchestra.run.smithi092.stdout: "type_data": { 2024-03-11T21:32:31.687 INFO:teuthology.orchestra.run.smithi092.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-11T21:32:31.687 INFO:teuthology.orchestra.run.smithi092.stdout: "events": [ 2024-03-11T21:32:31.687 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:32:31.687 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "initiated", 2024-03-11T21:32:31.687 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:29:08.769737+0000", 2024-03-11T21:32:31.688 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-03-11T21:32:31.688 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:32:31.688 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:32:31.688 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "header_read", 2024-03-11T21:32:31.688 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:29:08.769737+0000", 2024-03-11T21:32:31.688 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-03-11T21:32:31.688 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:32:31.688 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:32:31.688 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "throttled", 2024-03-11T21:32:31.688 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:29:08.769739+0000", 2024-03-11T21:32:31.688 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.3489999999999999e-06 2024-03-11T21:32:31.688 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:32:31.688 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:32:31.688 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "all_read", 2024-03-11T21:32:31.688 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:29:08.769799+0000", 2024-03-11T21:32:31.689 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 5.9707000000000003e-05 2024-03-11T21:32:31.689 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:32:31.689 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:32:31.689 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "dispatched", 2024-03-11T21:32:31.689 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:29:08.769807+0000", 2024-03-11T21:32:31.689 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 8.5299999999999996e-06 2024-03-11T21:32:31.689 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:32:31.689 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:32:31.689 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "queued_for_pg", 2024-03-11T21:32:31.689 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:29:08.769814+0000", 2024-03-11T21:32:31.689 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 7.2910000000000002e-06 2024-03-11T21:32:31.689 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:32:31.689 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:32:31.689 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-03-11T21:32:31.689 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:29:08.769889+0000", 2024-03-11T21:32:31.690 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 7.4742999999999999e-05 2024-03-11T21:32:31.690 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:32:31.690 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:32:31.690 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "started", 2024-03-11T21:32:31.690 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:29:08.769908+0000", 2024-03-11T21:32:31.690 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.8749e-05 2024-03-11T21:32:31.690 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:32:31.690 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:32:31.690 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "commit_sent", 2024-03-11T21:32:31.690 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:29:08.877594+0000", 2024-03-11T21:32:31.690 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.107686051 2024-03-11T21:32:31.690 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:32:31.690 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:32:31.691 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "done", 2024-03-11T21:32:31.691 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:29:08.877796+0000", 2024-03-11T21:32:31.691 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00020162399999999999 2024-03-11T21:32:31.691 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-03-11T21:32:31.691 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-03-11T21:32:31.691 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-03-11T21:32:31.691 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:32:31.691 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:32:31.691 INFO:teuthology.orchestra.run.smithi092.stdout: "description": "osd_repop(client.34715.0:116281 7.7 e119/99 7:e9a6f278:::benchmark_data_smithi092_76040_object116280:head v 119'7280, mlcod=119'7277)", 2024-03-11T21:32:31.691 INFO:teuthology.orchestra.run.smithi092.stdout: "initiated_at": "2024-03-11T21:29:08.770837+0000", 2024-03-11T21:32:31.691 INFO:teuthology.orchestra.run.smithi092.stdout: "age": 202.52248827700001, 2024-03-11T21:32:31.691 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.121888107, 2024-03-11T21:32:31.692 INFO:teuthology.orchestra.run.smithi092.stdout: "type_data": { 2024-03-11T21:32:31.692 INFO:teuthology.orchestra.run.smithi092.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-11T21:32:31.692 INFO:teuthology.orchestra.run.smithi092.stdout: "events": [ 2024-03-11T21:32:31.692 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:32:31.692 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "initiated", 2024-03-11T21:32:31.692 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:29:08.770837+0000", 2024-03-11T21:32:31.692 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-03-11T21:32:31.692 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:32:31.692 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:32:31.692 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "header_read", 2024-03-11T21:32:31.692 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:29:08.770837+0000", 2024-03-11T21:32:31.692 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-03-11T21:32:31.692 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:32:31.692 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:32:31.692 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "throttled", 2024-03-11T21:32:31.693 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:29:08.770842+0000", 2024-03-11T21:32:31.693 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 4.1849999999999997e-06 2024-03-11T21:32:31.693 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:32:31.693 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:32:31.693 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "all_read", 2024-03-11T21:32:31.693 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:29:08.770879+0000", 2024-03-11T21:32:31.693 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 3.7787999999999998e-05 2024-03-11T21:32:31.693 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:32:31.693 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:32:31.693 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "dispatched", 2024-03-11T21:32:31.693 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:29:08.877588+0000", 2024-03-11T21:32:31.693 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.1067085 2024-03-11T21:32:31.693 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:32:31.693 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:32:31.693 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "queued_for_pg", 2024-03-11T21:32:31.693 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:29:08.877600+0000", 2024-03-11T21:32:31.694 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.2429000000000001e-05 2024-03-11T21:32:31.694 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:32:31.694 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:32:31.694 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-03-11T21:32:31.694 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:29:08.877918+0000", 2024-03-11T21:32:31.694 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00031814000000000001 2024-03-11T21:32:31.694 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:32:31.694 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:32:31.694 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "started", 2024-03-11T21:32:31.694 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:29:08.877944+0000", 2024-03-11T21:32:31.694 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 2.5524000000000002e-05 2024-03-11T21:32:31.694 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:32:31.695 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:32:31.695 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "commit_sent", 2024-03-11T21:32:31.695 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:29:08.892645+0000", 2024-03-11T21:32:31.695 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.014701472 2024-03-11T21:32:31.695 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:32:31.695 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:32:31.695 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "done", 2024-03-11T21:32:31.695 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:29:08.892725+0000", 2024-03-11T21:32:31.695 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 8.0068999999999995e-05 2024-03-11T21:32:31.695 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-03-11T21:32:31.695 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-03-11T21:32:31.695 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-03-11T21:32:31.695 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:32:31.695 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:32:31.695 INFO:teuthology.orchestra.run.smithi092.stdout: "description": "osd_repop_reply(client.34715.0:116278 7.a e119/99 ondisk, result = 0)", 2024-03-11T21:32:31.696 INFO:teuthology.orchestra.run.smithi092.stdout: "initiated_at": "2024-03-11T21:29:08.770973+0000", 2024-03-11T21:32:31.696 INFO:teuthology.orchestra.run.smithi092.stdout: "age": 202.52235216899999, 2024-03-11T21:32:31.696 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.106903952, 2024-03-11T21:32:31.696 INFO:teuthology.orchestra.run.smithi092.stdout: "type_data": { 2024-03-11T21:32:31.696 INFO:teuthology.orchestra.run.smithi092.stdout: "flag_point": "started", 2024-03-11T21:32:31.696 INFO:teuthology.orchestra.run.smithi092.stdout: "events": [ 2024-03-11T21:32:31.696 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:32:31.696 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "initiated", 2024-03-11T21:32:31.696 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:29:08.770973+0000", 2024-03-11T21:32:31.696 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-03-11T21:32:31.696 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:32:31.696 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:32:31.696 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "header_read", 2024-03-11T21:32:31.696 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:29:08.770973+0000", 2024-03-11T21:32:31.696 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-03-11T21:32:31.697 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:32:31.697 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:32:31.697 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "throttled", 2024-03-11T21:32:31.697 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:29:08.770975+0000", 2024-03-11T21:32:31.697 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.432e-06 2024-03-11T21:32:31.697 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:32:31.697 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:32:31.697 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "all_read", 2024-03-11T21:32:31.697 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:29:08.770980+0000", 2024-03-11T21:32:31.697 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 5.2610000000000001e-06 2024-03-11T21:32:31.697 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:32:31.697 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:32:31.697 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "dispatched", 2024-03-11T21:32:31.697 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:29:08.877775+0000", 2024-03-11T21:32:31.697 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.106795236 2024-03-11T21:32:31.698 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:32:31.698 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:32:31.698 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "queued_for_pg", 2024-03-11T21:32:31.698 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:29:08.877784+0000", 2024-03-11T21:32:31.698 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 8.3350000000000007e-06 2024-03-11T21:32:31.698 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:32:31.698 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:32:31.698 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-03-11T21:32:31.698 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:29:08.877862+0000", 2024-03-11T21:32:31.698 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 7.7978000000000005e-05 2024-03-11T21:32:31.698 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:32:31.698 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:32:31.698 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "started", 2024-03-11T21:32:31.699 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:29:08.877869+0000", 2024-03-11T21:32:31.699 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 7.1060000000000002e-06 2024-03-11T21:32:31.699 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:32:31.699 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:32:31.699 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "done", 2024-03-11T21:32:31.699 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:29:08.877877+0000", 2024-03-11T21:32:31.699 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 8.6039999999999993e-06 2024-03-11T21:32:31.699 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-03-11T21:32:31.699 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-03-11T21:32:31.699 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-03-11T21:32:31.699 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:32:31.699 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:32:31.699 INFO:teuthology.orchestra.run.smithi092.stdout: "description": "osd_repop(client.34715.0:116285 7.1 e119/99 7:8ebc24c4:::benchmark_data_smithi092_76040_object116284:head v 119'7377, mlcod=119'7374)", 2024-03-11T21:32:31.699 INFO:teuthology.orchestra.run.smithi092.stdout: "initiated_at": "2024-03-11T21:29:08.772084+0000", 2024-03-11T21:32:31.700 INFO:teuthology.orchestra.run.smithi092.stdout: "age": 202.52124167900001, 2024-03-11T21:32:31.700 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.120626209, 2024-03-11T21:32:31.700 INFO:teuthology.orchestra.run.smithi092.stdout: "type_data": { 2024-03-11T21:32:31.700 INFO:teuthology.orchestra.run.smithi092.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-11T21:32:31.700 INFO:teuthology.orchestra.run.smithi092.stdout: "events": [ 2024-03-11T21:32:31.700 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:32:31.700 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "initiated", 2024-03-11T21:32:31.700 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:29:08.772084+0000", 2024-03-11T21:32:31.700 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-03-11T21:32:31.700 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:32:31.700 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:32:31.700 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "header_read", 2024-03-11T21:32:31.700 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:29:08.772084+0000", 2024-03-11T21:32:31.700 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-03-11T21:32:31.701 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:32:31.701 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:32:31.701 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "throttled", 2024-03-11T21:32:31.701 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:29:08.772085+0000", 2024-03-11T21:32:31.701 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.362e-06 2024-03-11T21:32:31.701 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:32:31.701 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:32:31.701 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "all_read", 2024-03-11T21:32:31.701 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:29:08.772115+0000", 2024-03-11T21:32:31.701 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 2.9496999999999999e-05 2024-03-11T21:32:31.701 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:32:31.701 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:32:31.701 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "dispatched", 2024-03-11T21:32:31.701 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:29:08.877618+0000", 2024-03-11T21:32:31.701 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.10550290499999999 2024-03-11T21:32:31.701 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:32:31.702 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:32:31.702 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "queued_for_pg", 2024-03-11T21:32:31.702 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:29:08.877628+0000", 2024-03-11T21:32:31.702 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.0784e-05 2024-03-11T21:32:31.702 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:32:31.702 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:32:31.702 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-03-11T21:32:31.703 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:29:08.878065+0000", 2024-03-11T21:32:31.703 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00043677499999999998 2024-03-11T21:32:31.703 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:32:31.703 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:32:31.703 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "started", 2024-03-11T21:32:31.703 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:29:08.878086+0000", 2024-03-11T21:32:31.703 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 2.1030999999999999e-05 2024-03-11T21:32:31.703 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:32:31.703 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:32:31.703 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "commit_sent", 2024-03-11T21:32:31.703 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:29:08.892658+0000", 2024-03-11T21:32:31.703 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.014571918 2024-03-11T21:32:31.703 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:32:31.703 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:32:31.704 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "done", 2024-03-11T21:32:31.704 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:29:08.892710+0000", 2024-03-11T21:32:31.704 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 5.1937000000000001e-05 2024-03-11T21:32:31.704 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-03-11T21:32:31.704 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-03-11T21:32:31.704 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-03-11T21:32:31.705 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:32:31.705 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:32:31.705 INFO:teuthology.orchestra.run.smithi092.stdout: "description": "osd_op(client.34715.0:191136 7.2 7:46aadf7b:::benchmark_data_smithi092_76040_object69225:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e135)", 2024-03-11T21:32:31.705 INFO:teuthology.orchestra.run.smithi092.stdout: "initiated_at": "2024-03-11T21:30:31.413601+0000", 2024-03-11T21:32:31.705 INFO:teuthology.orchestra.run.smithi092.stdout: "age": 119.879724866, 2024-03-11T21:32:31.705 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.110186913, 2024-03-11T21:32:31.705 INFO:teuthology.orchestra.run.smithi092.stdout: "type_data": { 2024-03-11T21:32:31.705 INFO:teuthology.orchestra.run.smithi092.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-11T21:32:31.705 INFO:teuthology.orchestra.run.smithi092.stdout: "client_info": { 2024-03-11T21:32:31.705 INFO:teuthology.orchestra.run.smithi092.stdout: "client": "client.34715", 2024-03-11T21:32:31.705 INFO:teuthology.orchestra.run.smithi092.stdout: "client_addr": "v1:172.21.15.92:0/2403132573", 2024-03-11T21:32:31.705 INFO:teuthology.orchestra.run.smithi092.stdout: "tid": 191136 2024-03-11T21:32:31.705 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:32:31.705 INFO:teuthology.orchestra.run.smithi092.stdout: "events": [ 2024-03-11T21:32:31.706 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:32:31.706 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "initiated", 2024-03-11T21:32:31.706 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:30:31.413601+0000", 2024-03-11T21:32:31.706 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-03-11T21:32:31.706 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:32:31.706 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:32:31.706 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "header_read", 2024-03-11T21:32:31.706 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:30:31.413601+0000", 2024-03-11T21:32:31.706 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-03-11T21:32:31.706 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:32:31.706 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:32:31.706 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "throttled", 2024-03-11T21:32:31.706 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:30:31.413602+0000", 2024-03-11T21:32:31.706 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.53e-06 2024-03-11T21:32:31.707 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:32:31.707 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:32:31.707 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "all_read", 2024-03-11T21:32:31.707 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:30:31.413606+0000", 2024-03-11T21:32:31.707 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 3.9099999999999998e-06 2024-03-11T21:32:31.707 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:32:31.707 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:32:31.707 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "dispatched", 2024-03-11T21:32:31.707 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:30:31.413614+0000", 2024-03-11T21:32:31.707 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 8.028e-06 2024-03-11T21:32:31.707 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:32:31.707 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:32:31.707 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "queued_for_pg", 2024-03-11T21:32:31.707 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:30:31.413621+0000", 2024-03-11T21:32:31.707 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 6.2979999999999997e-06 2024-03-11T21:32:31.708 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:32:31.708 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:32:31.708 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-03-11T21:32:31.708 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:30:31.413709+0000", 2024-03-11T21:32:31.708 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 8.8525999999999999e-05 2024-03-11T21:32:31.708 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:32:31.708 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:32:31.708 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "started", 2024-03-11T21:32:31.708 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:30:31.413840+0000", 2024-03-11T21:32:31.708 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00013101899999999999 2024-03-11T21:32:31.708 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:32:31.708 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:32:31.708 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "waiting for subops from 1,2", 2024-03-11T21:32:31.708 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:30:31.413947+0000", 2024-03-11T21:32:31.709 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00010681 2024-03-11T21:32:31.709 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:32:31.709 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:32:31.709 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "sub_op_commit_rec", 2024-03-11T21:32:31.709 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:30:31.420373+0000", 2024-03-11T21:32:31.709 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.0064264120000000003 2024-03-11T21:32:31.709 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:32:31.709 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:32:31.709 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "sub_op_commit_rec", 2024-03-11T21:32:31.709 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:30:31.420393+0000", 2024-03-11T21:32:31.709 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.9219999999999999e-05 2024-03-11T21:32:31.709 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:32:31.709 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:32:31.709 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "op_commit", 2024-03-11T21:32:31.709 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:30:31.425394+0000", 2024-03-11T21:32:31.710 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.005001208 2024-03-11T21:32:31.710 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:32:31.710 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:32:31.710 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "commit_sent", 2024-03-11T21:32:31.710 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:30:31.425485+0000", 2024-03-11T21:32:31.710 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 9.1028999999999999e-05 2024-03-11T21:32:31.710 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:32:31.710 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:32:31.710 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "done", 2024-03-11T21:32:31.710 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:30:31.523788+0000", 2024-03-11T21:32:31.710 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.098302923 2024-03-11T21:32:31.710 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-03-11T21:32:31.710 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-03-11T21:32:31.710 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-03-11T21:32:31.710 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:32:31.711 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:32:31.711 INFO:teuthology.orchestra.run.smithi092.stdout: "description": "osd_op(client.34715.0:191141 7.a 7:5952c096:::benchmark_data_smithi092_76040_object69230:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e135)", 2024-03-11T21:32:31.711 INFO:teuthology.orchestra.run.smithi092.stdout: "initiated_at": "2024-03-11T21:30:31.415283+0000", 2024-03-11T21:32:31.711 INFO:teuthology.orchestra.run.smithi092.stdout: "age": 119.878042554, 2024-03-11T21:32:31.711 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.111247662, 2024-03-11T21:32:31.711 INFO:teuthology.orchestra.run.smithi092.stdout: "type_data": { 2024-03-11T21:32:31.711 INFO:teuthology.orchestra.run.smithi092.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-11T21:32:31.711 INFO:teuthology.orchestra.run.smithi092.stdout: "client_info": { 2024-03-11T21:32:31.711 INFO:teuthology.orchestra.run.smithi092.stdout: "client": "client.34715", 2024-03-11T21:32:31.711 INFO:teuthology.orchestra.run.smithi092.stdout: "client_addr": "v1:172.21.15.92:0/2403132573", 2024-03-11T21:32:31.711 INFO:teuthology.orchestra.run.smithi092.stdout: "tid": 191141 2024-03-11T21:32:31.711 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:32:31.711 INFO:teuthology.orchestra.run.smithi092.stdout: "events": [ 2024-03-11T21:32:31.711 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:32:31.712 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "initiated", 2024-03-11T21:32:31.712 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:30:31.415283+0000", 2024-03-11T21:32:31.712 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-03-11T21:32:31.712 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:32:31.712 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:32:31.712 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "header_read", 2024-03-11T21:32:31.712 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:30:31.415283+0000", 2024-03-11T21:32:31.712 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-03-11T21:32:31.712 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:32:31.712 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:32:31.712 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "throttled", 2024-03-11T21:32:31.712 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:30:31.415285+0000", 2024-03-11T21:32:31.712 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.637e-06 2024-03-11T21:32:31.712 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:32:31.712 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:32:31.713 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "all_read", 2024-03-11T21:32:31.713 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:30:31.415290+0000", 2024-03-11T21:32:31.713 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 5.4389999999999999e-06 2024-03-11T21:32:31.713 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:32:31.713 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:32:31.713 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "dispatched", 2024-03-11T21:32:31.713 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:30:31.415301+0000", 2024-03-11T21:32:31.713 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.0713000000000001e-05 2024-03-11T21:32:31.713 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:32:31.713 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:32:31.713 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "queued_for_pg", 2024-03-11T21:32:31.713 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:30:31.415309+0000", 2024-03-11T21:32:31.714 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 7.7479999999999993e-06 2024-03-11T21:32:31.714 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:32:31.714 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:32:31.714 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-03-11T21:32:31.714 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:30:31.420490+0000", 2024-03-11T21:32:31.714 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.0051817490000000003 2024-03-11T21:32:31.714 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:32:31.714 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:32:31.714 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "started", 2024-03-11T21:32:31.714 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:30:31.420560+0000", 2024-03-11T21:32:31.714 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 6.9732000000000003e-05 2024-03-11T21:32:31.714 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:32:31.714 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:32:31.715 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "waiting for subops from 2,5", 2024-03-11T21:32:31.715 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:30:31.420634+0000", 2024-03-11T21:32:31.715 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 7.3934000000000001e-05 2024-03-11T21:32:31.715 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:32:31.715 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:32:31.715 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "sub_op_commit_rec", 2024-03-11T21:32:31.715 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:30:31.526111+0000", 2024-03-11T21:32:31.715 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.105477163 2024-03-11T21:32:31.715 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:32:31.715 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:32:31.715 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "sub_op_commit_rec", 2024-03-11T21:32:31.715 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:30:31.526153+0000", 2024-03-11T21:32:31.715 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 4.1579999999999998e-05 2024-03-11T21:32:31.715 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:32:31.715 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:32:31.716 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "op_commit", 2024-03-11T21:32:31.716 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:30:31.526332+0000", 2024-03-11T21:32:31.716 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00017919600000000001 2024-03-11T21:32:31.716 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:32:31.716 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:32:31.716 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "commit_sent", 2024-03-11T21:32:31.716 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:30:31.526479+0000", 2024-03-11T21:32:31.716 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.000147145 2024-03-11T21:32:31.716 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:32:31.716 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:32:31.716 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "done", 2024-03-11T21:32:31.716 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:30:31.526531+0000", 2024-03-11T21:32:31.716 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 5.1625999999999997e-05 2024-03-11T21:32:31.716 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-03-11T21:32:31.716 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-03-11T21:32:31.716 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-03-11T21:32:31.717 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:32:31.717 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:32:31.717 INFO:teuthology.orchestra.run.smithi092.stdout: "description": "osd_op(client.34715.0:191143 7.a 7:5efa688b:::benchmark_data_smithi092_76040_object69232:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e135)", 2024-03-11T21:32:31.717 INFO:teuthology.orchestra.run.smithi092.stdout: "initiated_at": "2024-03-11T21:30:31.415573+0000", 2024-03-11T21:32:31.717 INFO:teuthology.orchestra.run.smithi092.stdout: "age": 119.877752405, 2024-03-11T21:32:31.717 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.111074409, 2024-03-11T21:32:31.717 INFO:teuthology.orchestra.run.smithi092.stdout: "type_data": { 2024-03-11T21:32:31.717 INFO:teuthology.orchestra.run.smithi092.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-11T21:32:31.717 INFO:teuthology.orchestra.run.smithi092.stdout: "client_info": { 2024-03-11T21:32:31.717 INFO:teuthology.orchestra.run.smithi092.stdout: "client": "client.34715", 2024-03-11T21:32:31.717 INFO:teuthology.orchestra.run.smithi092.stdout: "client_addr": "v1:172.21.15.92:0/2403132573", 2024-03-11T21:32:31.717 INFO:teuthology.orchestra.run.smithi092.stdout: "tid": 191143 2024-03-11T21:32:31.717 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:32:31.717 INFO:teuthology.orchestra.run.smithi092.stdout: "events": [ 2024-03-11T21:32:31.717 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:32:31.718 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "initiated", 2024-03-11T21:32:31.718 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:30:31.415573+0000", 2024-03-11T21:32:31.718 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-03-11T21:32:31.718 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:32:31.718 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:32:31.718 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "header_read", 2024-03-11T21:32:31.718 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:30:31.415573+0000", 2024-03-11T21:32:31.718 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-03-11T21:32:31.718 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:32:31.718 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:32:31.718 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "throttled", 2024-03-11T21:32:31.718 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:30:31.415574+0000", 2024-03-11T21:32:31.718 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 9.4099999999999997e-07 2024-03-11T21:32:31.718 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:32:31.719 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:32:31.719 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "all_read", 2024-03-11T21:32:31.719 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:30:31.415579+0000", 2024-03-11T21:32:31.719 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 4.4580000000000003e-06 2024-03-11T21:32:31.719 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:32:31.719 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:32:31.719 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "dispatched", 2024-03-11T21:32:31.719 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:30:31.415588+0000", 2024-03-11T21:32:31.719 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 9.1949999999999992e-06 2024-03-11T21:32:31.719 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:32:31.719 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:32:31.719 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "queued_for_pg", 2024-03-11T21:32:31.719 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:30:31.415595+0000", 2024-03-11T21:32:31.719 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 6.8199999999999999e-06 2024-03-11T21:32:31.720 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:32:31.720 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:32:31.720 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-03-11T21:32:31.720 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:30:31.423235+0000", 2024-03-11T21:32:31.720 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.0076401960000000001 2024-03-11T21:32:31.720 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:32:31.720 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:32:31.720 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "started", 2024-03-11T21:32:31.720 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:30:31.423365+0000", 2024-03-11T21:32:31.720 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.000130467 2024-03-11T21:32:31.720 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:32:31.720 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:32:31.720 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "waiting for subops from 2,5", 2024-03-11T21:32:31.720 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:30:31.423489+0000", 2024-03-11T21:32:31.720 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00012326899999999999 2024-03-11T21:32:31.721 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:32:31.721 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:32:31.721 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "sub_op_commit_rec", 2024-03-11T21:32:31.721 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:30:31.526199+0000", 2024-03-11T21:32:31.721 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.10271060999999999 2024-03-11T21:32:31.721 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:32:31.721 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:32:31.721 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "sub_op_commit_rec", 2024-03-11T21:32:31.721 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:30:31.526253+0000", 2024-03-11T21:32:31.721 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 5.4222999999999999e-05 2024-03-11T21:32:31.721 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:32:31.721 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:32:31.721 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "op_commit", 2024-03-11T21:32:31.721 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:30:31.526538+0000", 2024-03-11T21:32:31.721 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00028438299999999998 2024-03-11T21:32:31.721 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:32:31.722 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:32:31.722 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "commit_sent", 2024-03-11T21:32:31.722 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:30:31.526611+0000", 2024-03-11T21:32:31.722 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 7.2847000000000002e-05 2024-03-11T21:32:31.722 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:32:31.722 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:32:31.722 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "done", 2024-03-11T21:32:31.723 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:30:31.526648+0000", 2024-03-11T21:32:31.723 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 3.6999999999999998e-05 2024-03-11T21:32:31.723 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-03-11T21:32:31.723 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-03-11T21:32:31.723 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-03-11T21:32:31.723 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:32:31.723 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:32:31.724 INFO:teuthology.orchestra.run.smithi092.stdout: "description": "osd_repop(client.34715.0:191145 7.7 e135/99 7:e7135044:::benchmark_data_smithi092_76040_object69234:head v 135'12032, mlcod=135'12030)", 2024-03-11T21:32:31.724 INFO:teuthology.orchestra.run.smithi092.stdout: "initiated_at": "2024-03-11T21:30:31.420519+0000", 2024-03-11T21:32:31.724 INFO:teuthology.orchestra.run.smithi092.stdout: "age": 119.872806735, 2024-03-11T21:32:31.724 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.10944303, 2024-03-11T21:32:31.724 INFO:teuthology.orchestra.run.smithi092.stdout: "type_data": { 2024-03-11T21:32:31.724 INFO:teuthology.orchestra.run.smithi092.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-11T21:32:31.724 INFO:teuthology.orchestra.run.smithi092.stdout: "events": [ 2024-03-11T21:32:31.724 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:32:31.724 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "initiated", 2024-03-11T21:32:31.724 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:30:31.420519+0000", 2024-03-11T21:32:31.724 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-03-11T21:32:31.725 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:32:31.725 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:32:31.725 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "header_read", 2024-03-11T21:32:31.725 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:30:31.420519+0000", 2024-03-11T21:32:31.725 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-03-11T21:32:31.725 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:32:31.725 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:32:31.725 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "throttled", 2024-03-11T21:32:31.725 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:30:31.420520+0000", 2024-03-11T21:32:31.725 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.2249999999999999e-06 2024-03-11T21:32:31.725 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:32:31.725 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:32:31.726 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "all_read", 2024-03-11T21:32:31.726 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:30:31.420527+0000", 2024-03-11T21:32:31.726 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 6.9410000000000001e-06 2024-03-11T21:32:31.726 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:32:31.726 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:32:31.726 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "dispatched", 2024-03-11T21:32:31.726 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:30:31.420536+0000", 2024-03-11T21:32:31.726 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 9.0289999999999994e-06 2024-03-11T21:32:31.726 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:32:31.726 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:32:31.726 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "queued_for_pg", 2024-03-11T21:32:31.726 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:30:31.420543+0000", 2024-03-11T21:32:31.726 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 7.0330000000000003e-06 2024-03-11T21:32:31.727 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:32:31.727 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:32:31.727 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-03-11T21:32:31.727 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:30:31.420597+0000", 2024-03-11T21:32:31.727 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 5.4385000000000002e-05 2024-03-11T21:32:31.727 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:32:31.727 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:32:31.727 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "started", 2024-03-11T21:32:31.727 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:30:31.420617+0000", 2024-03-11T21:32:31.727 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.9063e-05 2024-03-11T21:32:31.727 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:32:31.727 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:32:31.727 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "commit_sent", 2024-03-11T21:32:31.727 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:30:31.529867+0000", 2024-03-11T21:32:31.727 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.109250029 2024-03-11T21:32:31.728 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:32:31.728 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:32:31.728 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "done", 2024-03-11T21:32:31.728 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:30:31.529962+0000", 2024-03-11T21:32:31.728 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 9.5325e-05 2024-03-11T21:32:31.728 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-03-11T21:32:31.728 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-03-11T21:32:31.728 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-03-11T21:32:31.728 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:32:31.728 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:32:31.728 INFO:teuthology.orchestra.run.smithi092.stdout: "description": "osd_op(client.34715.0:191153 7.a 7:5e51656e:::benchmark_data_smithi092_76040_object69242:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e135)", 2024-03-11T21:32:31.728 INFO:teuthology.orchestra.run.smithi092.stdout: "initiated_at": "2024-03-11T21:30:31.422122+0000", 2024-03-11T21:32:31.728 INFO:teuthology.orchestra.run.smithi092.stdout: "age": 119.871203939, 2024-03-11T21:32:31.728 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.11289640200000001, 2024-03-11T21:32:31.729 INFO:teuthology.orchestra.run.smithi092.stdout: "type_data": { 2024-03-11T21:32:31.729 INFO:teuthology.orchestra.run.smithi092.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-11T21:32:31.729 INFO:teuthology.orchestra.run.smithi092.stdout: "client_info": { 2024-03-11T21:32:31.729 INFO:teuthology.orchestra.run.smithi092.stdout: "client": "client.34715", 2024-03-11T21:32:31.729 INFO:teuthology.orchestra.run.smithi092.stdout: "client_addr": "v1:172.21.15.92:0/2403132573", 2024-03-11T21:32:31.729 INFO:teuthology.orchestra.run.smithi092.stdout: "tid": 191153 2024-03-11T21:32:31.729 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:32:31.729 INFO:teuthology.orchestra.run.smithi092.stdout: "events": [ 2024-03-11T21:32:31.729 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:32:31.729 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "initiated", 2024-03-11T21:32:31.729 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:30:31.422122+0000", 2024-03-11T21:32:31.729 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-03-11T21:32:31.729 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:32:31.729 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:32:31.729 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "header_read", 2024-03-11T21:32:31.730 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:30:31.422122+0000", 2024-03-11T21:32:31.730 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-03-11T21:32:31.730 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:32:31.730 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:32:31.730 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "throttled", 2024-03-11T21:32:31.730 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:30:31.422124+0000", 2024-03-11T21:32:31.730 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.8479999999999999e-06 2024-03-11T21:32:31.730 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:32:31.730 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:32:31.730 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "all_read", 2024-03-11T21:32:31.730 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:30:31.422129+0000", 2024-03-11T21:32:31.730 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 5.6400000000000002e-06 2024-03-11T21:32:31.730 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:32:31.730 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:32:31.731 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "dispatched", 2024-03-11T21:32:31.731 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:30:31.422141+0000", 2024-03-11T21:32:31.731 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.1345999999999999e-05 2024-03-11T21:32:31.731 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:32:31.731 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:32:31.731 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "queued_for_pg", 2024-03-11T21:32:31.731 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:30:31.422148+0000", 2024-03-11T21:32:31.731 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 7.7950000000000008e-06 2024-03-11T21:32:31.731 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:32:31.731 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:32:31.731 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-03-11T21:32:31.731 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:30:31.425446+0000", 2024-03-11T21:32:31.731 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.003298044 2024-03-11T21:32:31.731 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:32:31.731 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:32:31.732 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "started", 2024-03-11T21:32:31.732 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:30:31.523399+0000", 2024-03-11T21:32:31.732 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.097952518000000002 2024-03-11T21:32:31.732 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:32:31.732 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:32:31.732 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "waiting for subops from 2,5", 2024-03-11T21:32:31.732 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:30:31.523536+0000", 2024-03-11T21:32:31.732 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00013672599999999999 2024-03-11T21:32:31.732 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:32:31.732 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:32:31.732 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "sub_op_commit_rec", 2024-03-11T21:32:31.732 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:30:31.526320+0000", 2024-03-11T21:32:31.732 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.0027840880000000001 2024-03-11T21:32:31.732 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:32:31.732 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:32:31.733 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "op_commit", 2024-03-11T21:32:31.733 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:30:31.526656+0000", 2024-03-11T21:32:31.733 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00033663699999999998 2024-03-11T21:32:31.733 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:32:31.733 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:32:31.733 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "sub_op_commit_rec", 2024-03-11T21:32:31.733 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:30:31.534907+0000", 2024-03-11T21:32:31.733 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.0082502040000000006 2024-03-11T21:32:31.733 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:32:31.733 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:32:31.733 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "commit_sent", 2024-03-11T21:32:31.734 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:30:31.534959+0000", 2024-03-11T21:32:31.734 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 5.2272999999999999e-05 2024-03-11T21:32:31.734 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:32:31.734 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:32:31.734 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "done", 2024-03-11T21:32:31.734 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:30:31.535018+0000", 2024-03-11T21:32:31.734 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 5.9283000000000001e-05 2024-03-11T21:32:31.734 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-03-11T21:32:31.734 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-03-11T21:32:31.734 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-03-11T21:32:31.735 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:32:31.735 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:32:31.735 INFO:teuthology.orchestra.run.smithi092.stdout: "description": "osd_repop_reply(client.34715.0:191141 7.a e135/99 ondisk, result = 0)", 2024-03-11T21:32:31.735 INFO:teuthology.orchestra.run.smithi092.stdout: "initiated_at": "2024-03-11T21:30:31.423431+0000", 2024-03-11T21:32:31.735 INFO:teuthology.orchestra.run.smithi092.stdout: "age": 119.869894925, 2024-03-11T21:32:31.735 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.102689715, 2024-03-11T21:32:31.735 INFO:teuthology.orchestra.run.smithi092.stdout: "type_data": { 2024-03-11T21:32:31.735 INFO:teuthology.orchestra.run.smithi092.stdout: "flag_point": "started", 2024-03-11T21:32:31.735 INFO:teuthology.orchestra.run.smithi092.stdout: "events": [ 2024-03-11T21:32:31.735 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:32:31.736 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "initiated", 2024-03-11T21:32:31.736 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:30:31.423431+0000", 2024-03-11T21:32:31.736 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-03-11T21:32:31.736 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:32:31.736 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:32:31.736 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "header_read", 2024-03-11T21:32:31.736 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:30:31.423431+0000", 2024-03-11T21:32:31.736 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-03-11T21:32:31.736 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:32:31.736 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:32:31.736 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "throttled", 2024-03-11T21:32:31.736 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:30:31.423432+0000", 2024-03-11T21:32:31.736 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.285e-06 2024-03-11T21:32:31.737 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:32:31.737 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:32:31.737 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "all_read", 2024-03-11T21:32:31.737 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:30:31.423437+0000", 2024-03-11T21:32:31.737 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 5.3229999999999997e-06 2024-03-11T21:32:31.737 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:32:31.737 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:32:31.738 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "dispatched", 2024-03-11T21:32:31.738 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:30:31.423447+0000", 2024-03-11T21:32:31.738 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 9.4269999999999997e-06 2024-03-11T21:32:31.738 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:32:31.738 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:32:31.738 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "queued_for_pg", 2024-03-11T21:32:31.738 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:30:31.423454+0000", 2024-03-11T21:32:31.738 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 7.1400000000000002e-06 2024-03-11T21:32:31.738 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:32:31.738 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:32:31.738 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-03-11T21:32:31.739 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:30:31.526097+0000", 2024-03-11T21:32:31.739 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.102642709 2024-03-11T21:32:31.739 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:32:31.739 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:32:31.739 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "started", 2024-03-11T21:32:31.739 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:30:31.526104+0000", 2024-03-11T21:32:31.739 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 7.5850000000000002e-06 2024-03-11T21:32:31.739 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:32:31.739 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:32:31.739 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "done", 2024-03-11T21:32:31.739 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:30:31.526120+0000", 2024-03-11T21:32:31.740 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.6246e-05 2024-03-11T21:32:31.740 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-03-11T21:32:31.740 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-03-11T21:32:31.740 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-03-11T21:32:31.740 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:32:31.740 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:32:31.740 INFO:teuthology.orchestra.run.smithi092.stdout: "description": "osd_repop(client.34715.0:191154 7.7 e135/99 7:ef41ab29:::benchmark_data_smithi092_76040_object69243:head v 135'12033, mlcod=135'12030)", 2024-03-11T21:32:31.740 INFO:teuthology.orchestra.run.smithi092.stdout: "initiated_at": "2024-03-11T21:30:31.424025+0000", 2024-03-11T21:32:31.740 INFO:teuthology.orchestra.run.smithi092.stdout: "age": 119.869300243, 2024-03-11T21:32:31.740 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.105995272, 2024-03-11T21:32:31.741 INFO:teuthology.orchestra.run.smithi092.stdout: "type_data": { 2024-03-11T21:32:31.741 INFO:teuthology.orchestra.run.smithi092.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-11T21:32:31.741 INFO:teuthology.orchestra.run.smithi092.stdout: "events": [ 2024-03-11T21:32:31.741 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:32:31.741 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "initiated", 2024-03-11T21:32:31.741 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:30:31.424025+0000", 2024-03-11T21:32:31.741 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-03-11T21:32:31.741 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:32:31.742 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:32:31.742 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "header_read", 2024-03-11T21:32:31.742 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:30:31.424025+0000", 2024-03-11T21:32:31.742 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-03-11T21:32:31.742 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:32:31.742 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:32:31.742 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "throttled", 2024-03-11T21:32:31.742 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:30:31.424027+0000", 2024-03-11T21:32:31.742 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.248e-06 2024-03-11T21:32:31.742 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:32:31.742 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:32:31.742 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "all_read", 2024-03-11T21:32:31.742 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:30:31.424034+0000", 2024-03-11T21:32:31.743 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 7.0990000000000001e-06 2024-03-11T21:32:31.743 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:32:31.743 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:32:31.743 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "dispatched", 2024-03-11T21:32:31.743 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:30:31.424043+0000", 2024-03-11T21:32:31.743 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 9.3889999999999994e-06 2024-03-11T21:32:31.743 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:32:31.743 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:32:31.743 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "queued_for_pg", 2024-03-11T21:32:31.743 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:30:31.424049+0000", 2024-03-11T21:32:31.743 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 6.3319999999999997e-06 2024-03-11T21:32:31.743 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:32:31.743 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:32:31.743 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-03-11T21:32:31.743 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:30:31.424096+0000", 2024-03-11T21:32:31.744 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 4.6088999999999999e-05 2024-03-11T21:32:31.744 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:32:31.744 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:32:31.744 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "started", 2024-03-11T21:32:31.744 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:30:31.424115+0000", 2024-03-11T21:32:31.744 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.9386e-05 2024-03-11T21:32:31.744 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:32:31.744 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:32:31.744 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "commit_sent", 2024-03-11T21:32:31.744 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:30:31.529965+0000", 2024-03-11T21:32:31.744 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.105850494 2024-03-11T21:32:31.744 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:32:31.745 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:32:31.745 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "done", 2024-03-11T21:32:31.745 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:30:31.530021+0000", 2024-03-11T21:32:31.745 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 5.5235000000000002e-05 2024-03-11T21:32:31.745 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-03-11T21:32:31.745 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-03-11T21:32:31.745 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-03-11T21:32:31.745 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:32:31.745 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:32:31.746 INFO:teuthology.orchestra.run.smithi092.stdout: "description": "osd_repop(client.34715.0:239321 7.7 e146/99 7:e125f248:::benchmark_data_smithi092_76040_object117410:head v 146'15001, mlcod=146'14998)", 2024-03-11T21:32:31.746 INFO:teuthology.orchestra.run.smithi092.stdout: "initiated_at": "2024-03-11T21:31:19.084464+0000", 2024-03-11T21:32:31.746 INFO:teuthology.orchestra.run.smithi092.stdout: "age": 72.208861588000005, 2024-03-11T21:32:31.746 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.102475784, 2024-03-11T21:32:31.746 INFO:teuthology.orchestra.run.smithi092.stdout: "type_data": { 2024-03-11T21:32:31.746 INFO:teuthology.orchestra.run.smithi092.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-11T21:32:31.746 INFO:teuthology.orchestra.run.smithi092.stdout: "events": [ 2024-03-11T21:32:31.746 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:32:31.746 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "initiated", 2024-03-11T21:32:31.746 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:31:19.084464+0000", 2024-03-11T21:32:31.747 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-03-11T21:32:31.747 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:32:31.747 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:32:31.747 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "header_read", 2024-03-11T21:32:31.747 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:31:19.084464+0000", 2024-03-11T21:32:31.747 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-03-11T21:32:31.747 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:32:31.747 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:32:31.747 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "throttled", 2024-03-11T21:32:31.748 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:31:19.084465+0000", 2024-03-11T21:32:31.748 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 6.9699999999999995e-07 2024-03-11T21:32:31.748 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:32:31.748 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:32:31.748 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "all_read", 2024-03-11T21:32:31.748 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:31:19.084470+0000", 2024-03-11T21:32:31.748 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 5.7710000000000003e-06 2024-03-11T21:32:31.748 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:32:31.749 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:32:31.749 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "dispatched", 2024-03-11T21:32:31.749 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:31:19.084477+0000", 2024-03-11T21:32:31.749 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 6.6379999999999998e-06 2024-03-11T21:32:31.749 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:32:31.749 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:32:31.749 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "queued_for_pg", 2024-03-11T21:32:31.749 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:31:19.084483+0000", 2024-03-11T21:32:31.749 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 5.7049999999999996e-06 2024-03-11T21:32:31.750 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:32:31.750 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:32:31.750 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-03-11T21:32:31.750 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:31:19.084762+0000", 2024-03-11T21:32:31.750 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00027901599999999999 2024-03-11T21:32:31.750 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:32:31.750 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:32:31.750 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "started", 2024-03-11T21:32:31.750 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:31:19.084781+0000", 2024-03-11T21:32:31.751 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.8974000000000002e-05 2024-03-11T21:32:31.751 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:32:31.751 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:32:31.751 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "commit_sent", 2024-03-11T21:32:31.751 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:31:19.186894+0000", 2024-03-11T21:32:31.751 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.102112895 2024-03-11T21:32:31.751 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:32:31.751 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:32:31.751 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "done", 2024-03-11T21:32:31.751 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:31:19.186940+0000", 2024-03-11T21:32:31.751 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 4.6088000000000003e-05 2024-03-11T21:32:31.752 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-03-11T21:32:31.752 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-03-11T21:32:31.752 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-03-11T21:32:31.752 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:32:31.752 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:32:31.752 INFO:teuthology.orchestra.run.smithi092.stdout: "description": "osd_repop(client.34715.0:239319 7.f e146/99 7:f23e104b:::benchmark_data_smithi092_76040_object117408:head v 146'18293, mlcod=146'18290)", 2024-03-11T21:32:31.752 INFO:teuthology.orchestra.run.smithi092.stdout: "initiated_at": "2024-03-11T21:31:19.088368+0000", 2024-03-11T21:32:31.752 INFO:teuthology.orchestra.run.smithi092.stdout: "age": 72.204958000000005, 2024-03-11T21:32:31.752 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.102002329, 2024-03-11T21:32:31.752 INFO:teuthology.orchestra.run.smithi092.stdout: "type_data": { 2024-03-11T21:32:31.752 INFO:teuthology.orchestra.run.smithi092.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-11T21:32:31.752 INFO:teuthology.orchestra.run.smithi092.stdout: "events": [ 2024-03-11T21:32:31.752 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:32:31.752 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "initiated", 2024-03-11T21:32:31.753 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:31:19.088368+0000", 2024-03-11T21:32:31.753 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-03-11T21:32:31.753 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:32:31.753 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:32:31.753 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "header_read", 2024-03-11T21:32:31.753 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:31:19.088368+0000", 2024-03-11T21:32:31.753 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-03-11T21:32:31.753 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:32:31.753 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:32:31.753 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "throttled", 2024-03-11T21:32:31.753 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:31:19.088369+0000", 2024-03-11T21:32:31.753 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.2619999999999999e-06 2024-03-11T21:32:31.753 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:32:31.753 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:32:31.753 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "all_read", 2024-03-11T21:32:31.753 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:31:19.088376+0000", 2024-03-11T21:32:31.754 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 6.742e-06 2024-03-11T21:32:31.754 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:32:31.754 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:32:31.754 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "dispatched", 2024-03-11T21:32:31.754 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:31:19.088385+0000", 2024-03-11T21:32:31.754 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 9.3530000000000001e-06 2024-03-11T21:32:31.754 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:32:31.754 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:32:31.754 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "queued_for_pg", 2024-03-11T21:32:31.754 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:31:19.088392+0000", 2024-03-11T21:32:31.754 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 6.5390000000000004e-06 2024-03-11T21:32:31.754 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:32:31.754 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:32:31.754 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-03-11T21:32:31.755 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:31:19.088472+0000", 2024-03-11T21:32:31.755 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 8.0280999999999999e-05 2024-03-11T21:32:31.755 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:32:31.755 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:32:31.755 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "started", 2024-03-11T21:32:31.755 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:31:19.088490+0000", 2024-03-11T21:32:31.755 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.8114999999999999e-05 2024-03-11T21:32:31.755 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:32:31.755 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:32:31.755 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "commit_sent", 2024-03-11T21:32:31.755 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:31:19.190325+0000", 2024-03-11T21:32:31.755 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.101834959 2024-03-11T21:32:31.755 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:32:31.756 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:32:31.756 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "done", 2024-03-11T21:32:31.756 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:31:19.190370+0000", 2024-03-11T21:32:31.756 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 4.5077999999999998e-05 2024-03-11T21:32:31.756 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-03-11T21:32:31.756 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-03-11T21:32:31.756 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-03-11T21:32:31.756 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:32:31.756 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:32:31.756 INFO:teuthology.orchestra.run.smithi092.stdout: "description": "osd_repop(client.34715.0:239324 7.f e146/99 7:f70e220a:::benchmark_data_smithi092_76040_object117413:head v 146'18294, mlcod=146'18290)", 2024-03-11T21:32:31.756 INFO:teuthology.orchestra.run.smithi092.stdout: "initiated_at": "2024-03-11T21:31:19.088880+0000", 2024-03-11T21:32:31.756 INFO:teuthology.orchestra.run.smithi092.stdout: "age": 72.204445574000005, 2024-03-11T21:32:31.756 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.104314253, 2024-03-11T21:32:31.756 INFO:teuthology.orchestra.run.smithi092.stdout: "type_data": { 2024-03-11T21:32:31.756 INFO:teuthology.orchestra.run.smithi092.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-11T21:32:31.757 INFO:teuthology.orchestra.run.smithi092.stdout: "events": [ 2024-03-11T21:32:31.757 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:32:31.757 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "initiated", 2024-03-11T21:32:31.757 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:31:19.088880+0000", 2024-03-11T21:32:31.757 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-03-11T21:32:31.757 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:32:31.757 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:32:31.757 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "header_read", 2024-03-11T21:32:31.757 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:31:19.088880+0000", 2024-03-11T21:32:31.757 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-03-11T21:32:31.757 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:32:31.757 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:32:31.757 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "throttled", 2024-03-11T21:32:31.757 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:31:19.088881+0000", 2024-03-11T21:32:31.757 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 8.9400000000000004e-07 2024-03-11T21:32:31.758 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:32:31.758 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:32:31.758 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "all_read", 2024-03-11T21:32:31.758 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:31:19.088887+0000", 2024-03-11T21:32:31.758 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 6.3910000000000003e-06 2024-03-11T21:32:31.758 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:32:31.758 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:32:31.758 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "dispatched", 2024-03-11T21:32:31.758 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:31:19.088897+0000", 2024-03-11T21:32:31.758 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 9.285e-06 2024-03-11T21:32:31.758 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:32:31.759 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:32:31.759 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "queued_for_pg", 2024-03-11T21:32:31.759 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:31:19.088903+0000", 2024-03-11T21:32:31.759 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 6.5019999999999998e-06 2024-03-11T21:32:31.759 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:32:31.759 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:32:31.759 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-03-11T21:32:31.759 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:31:19.091402+0000", 2024-03-11T21:32:31.759 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.0024992719999999999 2024-03-11T21:32:31.759 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:32:31.759 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:32:31.759 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "started", 2024-03-11T21:32:31.759 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:31:19.091422+0000", 2024-03-11T21:32:31.759 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.9632000000000001e-05 2024-03-11T21:32:31.759 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:32:31.760 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:32:31.760 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "commit_sent", 2024-03-11T21:32:31.760 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:31:19.193156+0000", 2024-03-11T21:32:31.760 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.101733882 2024-03-11T21:32:31.760 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:32:31.760 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:32:31.760 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "done", 2024-03-11T21:32:31.760 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:31:19.193194+0000", 2024-03-11T21:32:31.760 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 3.8395000000000001e-05 2024-03-11T21:32:31.760 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-03-11T21:32:31.760 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-03-11T21:32:31.760 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-03-11T21:32:31.760 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-03-11T21:32:31.760 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-03-11T21:32:31.760 INFO:teuthology.orchestra.run.smithi092.stdout:} 2024-03-11T21:32:31.761 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:32:31.793 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:32:31.794 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:32:31.794 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:32:31.895 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:32:31.937 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:32:31.938 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:32:31.938 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:32:32.039 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:32:32.072 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:32:32.072 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:32:32.073 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:32:32.174 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:32:32.208 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:32:32.208 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:32:32.209 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:32:32.309 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:32:32.346 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:32:32.346 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:32:32.347 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:32:32.432 INFO:tasks.radosbench.radosbench.0.smithi092.stdout: 31 15 59496 59481 119.899 93 0.0137739 0.00833498 2024-03-11T21:32:32.447 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:32:32.479 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:32:32.479 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:32:32.480 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:32:32.576 INFO:journalctl@ceph.iscsi.iscsi.a.smithi097.stdout:Mar 11 21:32:32 smithi097 ceph-00db0016-dfed-11ee-95c8-87774f69a715-iscsi-iscsi-a[60943]: debug there is no tcmu-runner data available 2024-03-11T21:32:32.580 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:32:32.612 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:32:32.612 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:32:32.612 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:32:32.713 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:32:32.743 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:32:32.743 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:32:32.744 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:32:32.845 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:32:32.878 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:32:32.879 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:32:32.880 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:32:32.981 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:32:32.997 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:32:32 smithi092 ceph-mon[29572]: pgmap v271: 148 pgs: 148 active+clean; 3.2 GiB data, 11 GiB used, 705 GiB / 715 GiB avail; 649 B/s rd, 60 MiB/s wr, 959 op/s; 13 B/s, 0 objects/s recovering 2024-03-11T21:32:32.997 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:32:32 smithi092 ceph-mon[37563]: pgmap v271: 148 pgs: 148 active+clean; 3.2 GiB data, 11 GiB used, 705 GiB / 715 GiB avail; 649 B/s rd, 60 MiB/s wr, 959 op/s; 13 B/s, 0 objects/s recovering 2024-03-11T21:32:33.022 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:32:33.023 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:32:33.023 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:32:33.076 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:32:32 smithi097 ceph-mon[33317]: pgmap v271: 148 pgs: 148 active+clean; 3.2 GiB data, 11 GiB used, 705 GiB / 715 GiB avail; 649 B/s rd, 60 MiB/s wr, 959 op/s; 13 B/s, 0 objects/s recovering 2024-03-11T21:32:33.124 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:32:33.153 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:32:33.154 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:32:33.154 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:32:33.255 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:32:33.297 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:32:33.298 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:32:33.298 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:32:33.399 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:32:33.432 INFO:tasks.radosbench.radosbench.0.smithi092.stdout: 32 16 60869 60853 118.83 85.75 0.0121934 0.00840973 2024-03-11T21:32:33.442 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:32:33.443 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:32:33.443 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:32:33.544 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:32:33.585 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:32:33.586 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:32:33.586 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:32:33.684 DEBUG:teuthology.orchestra.run.smithi097:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid 00db0016-dfed-11ee-95c8-87774f69a715 -- ceph daemon osd.4 dump_ops_in_flight 2024-03-11T21:32:33.687 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:32:33.758 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:32:33.760 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:32:33.761 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:32:33.810 INFO:teuthology.orchestra.run.smithi092.stderr:Inferring config /var/lib/ceph/00db0016-dfed-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T21:32:33.862 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:32:33.909 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:32:33.909 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:32:33.910 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:32:34.009 INFO:teuthology.orchestra.run.smithi097.stderr:Inferring config /var/lib/ceph/00db0016-dfed-11ee-95c8-87774f69a715/mon.b/config 2024-03-11T21:32:34.010 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:32:34.048 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:32:34.049 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:32:34.049 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:32:34.076 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:32:33 smithi097 ceph-mon[33317]: from='client.24469 v1:172.21.15.97:0/404208310' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:32:34.076 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:32:33 smithi097 ceph-mon[33317]: osdmap e171: 8 total, 8 up, 8 in 2024-03-11T21:32:34.110 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:32:33 smithi092 ceph-mon[29572]: from='client.24469 v1:172.21.15.97:0/404208310' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:32:34.110 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:32:33 smithi092 ceph-mon[29572]: osdmap e171: 8 total, 8 up, 8 in 2024-03-11T21:32:34.111 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:32:33 smithi092 ceph-mon[37563]: from='client.24469 v1:172.21.15.97:0/404208310' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:32:34.111 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:32:33 smithi092 ceph-mon[37563]: osdmap e171: 8 total, 8 up, 8 in 2024-03-11T21:32:34.150 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:32:34.192 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:32:34.195 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:32:34.195 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:32:34.218 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5, 6, 7] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7] 2024-03-11T21:32:34.218 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-03-11T21:32:34.218 INFO:tasks.thrashosds.thrasher:Growing pool unique_pool_1 2024-03-11T21:32:34.218 DEBUG:teuthology.orchestra.run.smithi092:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid 00db0016-dfed-11ee-95c8-87774f69a715 -- ceph pg dump --format=json 2024-03-11T21:32:34.296 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:32:34.326 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:32:34.327 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:32:34.327 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:32:34.428 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:32:34.432 INFO:tasks.radosbench.radosbench.0.smithi092.stdout: 33 16 61959 61943 117.294 68.125 0.0155266 0.00851969 2024-03-11T21:32:34.458 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:32:34.459 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:32:34.459 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:32:34.560 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:32:34.592 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:32:34.593 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:32:34.593 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:32:34.694 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:32:34.742 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:32:34.742 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:32:34.743 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:32:34.844 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:32:34.876 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:32:34.876 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:32:34.877 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:32:34.978 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:32:35.017 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:32:35.018 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:32:35.019 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:32:35.076 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:32:34 smithi097 ceph-mon[33317]: pgmap v272: 148 pgs: 148 active+clean; 3.6 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 1.2 KiB/s rd, 112 MiB/s wr, 1.80k op/s; 13 B/s, 0 objects/s recovering 2024-03-11T21:32:35.111 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:32:34 smithi092 ceph-mon[29572]: pgmap v272: 148 pgs: 148 active+clean; 3.6 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 1.2 KiB/s rd, 112 MiB/s wr, 1.80k op/s; 13 B/s, 0 objects/s recovering 2024-03-11T21:32:35.111 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:32:34 smithi092 ceph-mon[37563]: pgmap v272: 148 pgs: 148 active+clean; 3.6 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 1.2 KiB/s rd, 112 MiB/s wr, 1.80k op/s; 13 B/s, 0 objects/s recovering 2024-03-11T21:32:35.119 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:32:35.150 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:32:35.150 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:32:35.151 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:32:35.252 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:32:35.284 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:32:35.284 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:32:35.285 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:32:35.386 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:32:35.415 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:32:35.416 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:32:35.417 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:32:35.432 INFO:tasks.radosbench.radosbench.0.smithi092.stdout: 34 16 63210 63194 116.143 78.1875 0.0138626 0.00860469 2024-03-11T21:32:35.518 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:32:35.559 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:32:35.560 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:32:35.560 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:32:35.661 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:32:35.700 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:32:35.701 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:32:35.701 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:32:35.802 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:32:35.844 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:32:35.845 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:32:35.845 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:32:35.946 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:32:35.978 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:32:35.979 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:32:35.979 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:32:35.984 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:32:35 smithi092 ceph-mon[29572]: purged_snaps scrub starts 2024-03-11T21:32:35.984 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:32:35 smithi092 ceph-mon[29572]: purged_snaps scrub ok 2024-03-11T21:32:35.984 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:32:35 smithi092 ceph-mon[29572]: purged_snaps scrub starts 2024-03-11T21:32:35.984 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:32:35 smithi092 ceph-mon[29572]: purged_snaps scrub ok 2024-03-11T21:32:35.984 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:32:35 smithi092 ceph-mon[29572]: osdmap e172: 8 total, 8 up, 8 in 2024-03-11T21:32:35.985 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:32:35 smithi092 ceph-mon[37563]: purged_snaps scrub starts 2024-03-11T21:32:35.985 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:32:35 smithi092 ceph-mon[37563]: purged_snaps scrub ok 2024-03-11T21:32:35.985 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:32:35 smithi092 ceph-mon[37563]: purged_snaps scrub starts 2024-03-11T21:32:35.985 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:32:35 smithi092 ceph-mon[37563]: purged_snaps scrub ok 2024-03-11T21:32:35.985 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:32:35 smithi092 ceph-mon[37563]: osdmap e172: 8 total, 8 up, 8 in 2024-03-11T21:32:36.080 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:32:36.121 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:32:36.122 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:32:36.122 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:32:36.161 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:32:35 smithi097 ceph-mon[33317]: purged_snaps scrub starts 2024-03-11T21:32:36.161 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:32:35 smithi097 ceph-mon[33317]: purged_snaps scrub ok 2024-03-11T21:32:36.161 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:32:35 smithi097 ceph-mon[33317]: purged_snaps scrub starts 2024-03-11T21:32:36.161 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:32:35 smithi097 ceph-mon[33317]: purged_snaps scrub ok 2024-03-11T21:32:36.161 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:32:35 smithi097 ceph-mon[33317]: osdmap e172: 8 total, 8 up, 8 in 2024-03-11T21:32:36.223 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:32:36.255 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:32:36.255 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:32:36.256 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:32:36.357 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:32:36.394 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:32:36.395 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:32:36.396 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:32:36.405 INFO:teuthology.orchestra.run.smithi092.stdout:osd.0: {} 2024-03-11T21:32:36.405 INFO:teuthology.orchestra.run.smithi092.stdout:osd.1: {} 2024-03-11T21:32:36.405 INFO:teuthology.orchestra.run.smithi092.stdout:osd.2: {} 2024-03-11T21:32:36.405 INFO:teuthology.orchestra.run.smithi092.stdout:osd.3: {} 2024-03-11T21:32:36.405 INFO:teuthology.orchestra.run.smithi092.stdout:osd.4: {} 2024-03-11T21:32:36.406 INFO:teuthology.orchestra.run.smithi092.stdout:osd.5: {} 2024-03-11T21:32:36.406 INFO:teuthology.orchestra.run.smithi092.stdout:osd.6: {} 2024-03-11T21:32:36.406 INFO:teuthology.orchestra.run.smithi092.stdout:osd.7: {} 2024-03-11T21:32:36.406 INFO:teuthology.orchestra.run.smithi092.stderr:osd.0: osd_enable_op_tracker = '' 2024-03-11T21:32:36.406 INFO:teuthology.orchestra.run.smithi092.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-11T21:32:36.406 INFO:teuthology.orchestra.run.smithi092.stderr:osd.2: osd_enable_op_tracker = '' 2024-03-11T21:32:36.406 INFO:teuthology.orchestra.run.smithi092.stderr:osd.3: osd_enable_op_tracker = '' 2024-03-11T21:32:36.406 INFO:teuthology.orchestra.run.smithi092.stderr:osd.4: osd_enable_op_tracker = '' 2024-03-11T21:32:36.406 INFO:teuthology.orchestra.run.smithi092.stderr:osd.5: osd_enable_op_tracker = '' 2024-03-11T21:32:36.406 INFO:teuthology.orchestra.run.smithi092.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-11T21:32:36.406 INFO:teuthology.orchestra.run.smithi092.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-11T21:32:36.432 INFO:tasks.radosbench.radosbench.0.smithi092.stdout: 35 16 64479 64463 115.091 79.3125 0.0049259 0.00868201 2024-03-11T21:32:36.497 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:32:36.528 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:32:36.529 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:32:36.529 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:32:36.630 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:32:36.671 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:32:36.672 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:32:36.672 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:32:36.773 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:32:36.805 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:32:36.806 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:32:36.806 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:32:36.907 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:32:36.947 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:32:36.948 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:32:36.948 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:32:36.982 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:32:36 smithi092 ceph-mon[37563]: pgmap v275: 148 pgs: 148 active+clean; 3.6 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 744 B/s rd, 67 MiB/s wr, 1.07k op/s 2024-03-11T21:32:36.982 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:32:36 smithi092 ceph-mon[37563]: from='client.? v1:172.21.15.92:0/437541954' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-11T21:32:36.982 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:32:36 smithi092 ceph-mon[37563]: from='mgr.14607 ' entity='mgr.y' 2024-03-11T21:32:36.983 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:32:36 smithi092 ceph-mon[37563]: from='mgr.14607 v1:172.21.15.92:0/1702717597' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T21:32:36.983 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:32:36 smithi092 ceph-mon[29572]: pgmap v275: 148 pgs: 148 active+clean; 3.6 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 744 B/s rd, 67 MiB/s wr, 1.07k op/s 2024-03-11T21:32:36.983 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:32:36 smithi092 ceph-mon[29572]: from='client.? v1:172.21.15.92:0/437541954' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-11T21:32:36.983 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:32:36 smithi092 ceph-mon[29572]: from='mgr.14607 ' entity='mgr.y' 2024-03-11T21:32:36.983 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:32:36 smithi092 ceph-mon[29572]: from='mgr.14607 v1:172.21.15.92:0/1702717597' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T21:32:37.050 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:32:37.090 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:32:37.090 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:32:37.090 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:32:37.191 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:32:37.230 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:32:37.230 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:32:37.231 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:32:37.327 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:32:36 smithi097 ceph-mon[33317]: pgmap v275: 148 pgs: 148 active+clean; 3.6 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 744 B/s rd, 67 MiB/s wr, 1.07k op/s 2024-03-11T21:32:37.327 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:32:36 smithi097 ceph-mon[33317]: from='client.? v1:172.21.15.92:0/437541954' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-11T21:32:37.327 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:32:36 smithi097 ceph-mon[33317]: from='mgr.14607 ' entity='mgr.y' 2024-03-11T21:32:37.327 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:32:36 smithi097 ceph-mon[33317]: from='mgr.14607 v1:172.21.15.92:0/1702717597' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T21:32:37.331 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:32:37.375 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:32:37.375 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:32:37.376 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:32:37.432 INFO:tasks.radosbench.radosbench.0.smithi092.stdout: 36 16 65783 65767 114.158 81.5 0.00341494 0.00874887 2024-03-11T21:32:37.477 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:32:37.506 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:32:37.506 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:32:37.506 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:32:37.607 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:32:37.639 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:32:37.639 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:32:37.639 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:32:37.670 INFO:teuthology.orchestra.run.smithi097.stdout:ERROR: (22) Invalid argument 2024-03-11T21:32:37.670 INFO:teuthology.orchestra.run.smithi097.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-11T21:32:37.741 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:32:37.779 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:32:37.780 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:32:37.780 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:32:37.881 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:32:37.920 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:32:37.920 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:32:37.920 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:32:38.021 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:32:38.057 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:32:38.058 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:32:38.058 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:32:38.159 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:32:38.192 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:32:38.193 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:32:38.193 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:32:38.294 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:32:38.313 INFO:journalctl@ceph.mgr.y.smithi092.stdout:Mar 11 21:32:37 smithi092 ceph-00db0016-dfed-11ee-95c8-87774f69a715-mgr-y[29801]: ::ffff:172.21.15.97 - - [11/Mar/2024:21:32:37] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-11T21:32:38.331 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:32:38.332 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:32:38.332 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:32:38.380 INFO:teuthology.orchestra.run.smithi092.stderr:Inferring config /var/lib/ceph/00db0016-dfed-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T21:32:38.433 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:32:38.435 INFO:tasks.radosbench.radosbench.0.smithi092.stdout: 37 16 66981 66965 113.095 74.875 0.00830886 0.00883703 2024-03-11T21:32:38.470 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:32:38.471 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:32:38.471 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:32:38.572 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:32:38.602 DEBUG:teuthology.orchestra.run.smithi097:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid 00db0016-dfed-11ee-95c8-87774f69a715 -- ceph daemon osd.4 dump_blocked_ops 2024-03-11T21:32:38.613 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:32:38.615 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:32:38.615 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:32:38.718 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:32:38.744 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:32:38.745 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:32:38.745 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:32:38.846 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:32:38.882 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:32:38.882 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:32:38.883 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:32:38.954 INFO:teuthology.orchestra.run.smithi097.stderr:Inferring config /var/lib/ceph/00db0016-dfed-11ee-95c8-87774f69a715/mon.b/config 2024-03-11T21:32:38.984 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:32:39.026 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:32:39.026 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:32:39.027 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:32:39.127 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:32:39.128 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:32:38 smithi097 ceph-mon[33317]: pgmap v276: 148 pgs: 148 active+clean; 3.7 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 639 B/s rd, 70 MiB/s wr, 1.12k op/s 2024-03-11T21:32:39.128 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:32:38 smithi097 ceph-mon[33317]: osdmap e173: 8 total, 8 up, 8 in 2024-03-11T21:32:39.166 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:32:39.167 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:32:39.167 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:32:39.268 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:32:39.301 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:32:39.301 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:32:39.301 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:32:39.305 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:32:38 smithi092 ceph-mon[29572]: pgmap v276: 148 pgs: 148 active+clean; 3.7 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 639 B/s rd, 70 MiB/s wr, 1.12k op/s 2024-03-11T21:32:39.305 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:32:38 smithi092 ceph-mon[29572]: osdmap e173: 8 total, 8 up, 8 in 2024-03-11T21:32:39.305 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:32:38 smithi092 ceph-mon[37563]: pgmap v276: 148 pgs: 148 active+clean; 3.7 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 639 B/s rd, 70 MiB/s wr, 1.12k op/s 2024-03-11T21:32:39.306 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:32:38 smithi092 ceph-mon[37563]: osdmap e173: 8 total, 8 up, 8 in 2024-03-11T21:32:39.402 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:32:39.433 INFO:tasks.radosbench.radosbench.0.smithi092.stdout: 38 16 68197 68181 112.119 76 0.0200038 0.00891438 2024-03-11T21:32:39.441 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:32:39.442 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:32:39.442 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:32:39.543 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:32:39.575 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:32:39.576 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:32:39.576 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:32:39.677 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:32:39.713 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:32:39.713 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:32:39.714 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:32:39.815 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:32:39.847 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:32:39.847 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:32:39.847 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:32:39.948 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:32:39.981 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:32:39.982 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:32:39.982 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:32:40.083 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:32:40.116 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:32:40.116 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:32:40.117 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:32:40.217 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:32:40.255 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:32:40.255 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:32:40.256 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:32:40.357 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:32:40.359 DEBUG:teuthology.orchestra.run.smithi092:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid 00db0016-dfed-11ee-95c8-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-03-11T21:32:40.388 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:32:40.389 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:32:40.389 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:32:40.433 INFO:tasks.radosbench.radosbench.0.smithi092.stdout: 39 15 69494 69479 111.324 81.125 0.00767953 0.00897681 2024-03-11T21:32:40.491 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:32:40.518 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:32:40.519 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:32:40.519 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:32:40.620 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:32:40.670 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:32:40.671 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:32:40.671 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:32:40.772 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:32:40.810 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:32:40.810 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:32:40.811 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:32:40.912 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:32:40.954 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:32:40.954 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:32:40.955 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:32:41.055 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:32:41.077 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:32:40 smithi092 ceph-mon[29572]: pgmap v278: 148 pgs: 148 active+clean; 4.0 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 853 B/s rd, 66 MiB/s wr, 1.05k op/s 2024-03-11T21:32:41.077 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:32:40 smithi092 ceph-mon[37563]: pgmap v278: 148 pgs: 148 active+clean; 4.0 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 853 B/s rd, 66 MiB/s wr, 1.05k op/s 2024-03-11T21:32:41.102 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:32:41.103 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:32:41.103 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:32:41.198 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:32:40 smithi097 ceph-mon[33317]: pgmap v278: 148 pgs: 148 active+clean; 4.0 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 853 B/s rd, 66 MiB/s wr, 1.05k op/s 2024-03-11T21:32:41.205 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:32:41.246 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:32:41.247 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:32:41.247 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:32:41.348 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:32:41.381 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:32:41.382 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:32:41.382 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:32:41.433 INFO:tasks.radosbench.radosbench.0.smithi092.stdout:2024-03-11T21:32:41.451880+0000 min lat: 0.0013089 max lat: 0.168224 avg lat: 0.00905658 2024-03-11T21:32:41.433 INFO:tasks.radosbench.radosbench.0.smithi092.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-03-11T21:32:41.433 INFO:tasks.radosbench.radosbench.0.smithi092.stdout: 40 16 70656 70640 110.355 72.5625 0.00573637 0.00905658 2024-03-11T21:32:41.484 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:32:41.513 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:32:41.514 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:32:41.514 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:32:41.615 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:32:41.646 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:32:41.646 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:32:41.647 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:32:41.666 INFO:teuthology.orchestra.run.smithi097.stdout:ERROR: (22) Invalid argument 2024-03-11T21:32:41.666 INFO:teuthology.orchestra.run.smithi097.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-11T21:32:41.748 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:32:41.780 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:32:41.780 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:32:41.781 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:32:41.826 INFO:teuthology.orchestra.run.smithi092.stderr:Inferring config /var/lib/ceph/00db0016-dfed-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T21:32:41.882 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:32:41.909 INFO:teuthology.orchestra.run.smithi092.stderr:nodeep-scrub is set 2024-03-11T21:32:41.910 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:32:41.910 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:32:41.910 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:32:42.011 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:32:42.041 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:32:42.042 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:32:42.043 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:32:42.076 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:32:41 smithi097 ceph-mon[33317]: from='client.? v1:172.21.15.92:0/2370746709' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-11T21:32:42.076 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:32:41 smithi097 ceph-mon[33317]: from='client.35054 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-11T21:32:42.110 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:32:41 smithi092 ceph-mon[29572]: from='client.? v1:172.21.15.92:0/2370746709' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-11T21:32:42.110 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:32:41 smithi092 ceph-mon[29572]: from='client.35054 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-11T21:32:42.110 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:32:41 smithi092 ceph-mon[37563]: from='client.? v1:172.21.15.92:0/2370746709' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-11T21:32:42.110 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:32:41 smithi092 ceph-mon[37563]: from='client.35054 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-11T21:32:42.144 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:32:42.182 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:32:42.182 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:32:42.183 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:32:42.284 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:32:42.324 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:32:42.324 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:32:42.325 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:32:42.425 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:32:42.434 INFO:tasks.radosbench.radosbench.0.smithi092.stdout: 41 15 71908 71893 109.573 78.3125 0.0257254 0.009122 2024-03-11T21:32:42.472 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:32:42.473 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:32:42.473 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:32:42.574 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:32:42.577 INFO:journalctl@ceph.iscsi.iscsi.a.smithi097.stdout:Mar 11 21:32:42 smithi097 ceph-00db0016-dfed-11ee-95c8-87774f69a715-iscsi-iscsi-a[60943]: debug there is no tcmu-runner data available 2024-03-11T21:32:42.604 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:32:42.605 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:32:42.605 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:32:42.706 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:32:42.734 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:32:42.735 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:32:42.735 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:32:42.836 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:32:42.868 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:32:42.869 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:32:42.869 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:32:42.970 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:32:43.017 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:32:43.018 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:32:43.018 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:32:43.119 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:32:43.156 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:32:43.157 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:32:43.158 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:32:43.165 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:32:42 smithi092 ceph-mon[29572]: pgmap v279: 148 pgs: 148 active+clean; 4.0 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 650 B/s rd, 50 MiB/s wr, 804 op/s 2024-03-11T21:32:43.165 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:32:42 smithi092 ceph-mon[29572]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-11T21:32:43.165 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:32:42 smithi092 ceph-mon[29572]: from='client.35054 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-03-11T21:32:43.165 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:32:42 smithi092 ceph-mon[29572]: osdmap e174: 8 total, 8 up, 8 in 2024-03-11T21:32:43.165 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:32:42 smithi092 ceph-mon[37563]: pgmap v279: 148 pgs: 148 active+clean; 4.0 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 650 B/s rd, 50 MiB/s wr, 804 op/s 2024-03-11T21:32:43.165 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:32:42 smithi092 ceph-mon[37563]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-11T21:32:43.165 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:32:42 smithi092 ceph-mon[37563]: from='client.35054 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-03-11T21:32:43.165 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:32:42 smithi092 ceph-mon[37563]: osdmap e174: 8 total, 8 up, 8 in 2024-03-11T21:32:43.259 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:32:43.297 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:32:43.298 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:32:43.298 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:32:43.326 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:32:42 smithi097 ceph-mon[33317]: pgmap v279: 148 pgs: 148 active+clean; 4.0 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 650 B/s rd, 50 MiB/s wr, 804 op/s 2024-03-11T21:32:43.326 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:32:42 smithi097 ceph-mon[33317]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-11T21:32:43.326 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:32:42 smithi097 ceph-mon[33317]: from='client.35054 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-03-11T21:32:43.326 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:32:42 smithi097 ceph-mon[33317]: osdmap e174: 8 total, 8 up, 8 in 2024-03-11T21:32:43.399 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:32:43.431 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:32:43.432 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:32:43.433 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:32:43.433 INFO:tasks.radosbench.radosbench.0.smithi092.stdout: 42 16 73165 73149 108.834 78.5 0.00877345 0.00918337 2024-03-11T21:32:43.534 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:32:43.563 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:32:43.564 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:32:43.565 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:32:43.665 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:32:43.695 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:32:43.695 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:32:43.695 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:32:43.796 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:32:43.824 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:32:43.825 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:32:43.825 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:32:43.926 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:32:43.975 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:32:43.975 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:32:43.976 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:32:44.018 DEBUG:teuthology.orchestra.run.smithi097:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid 00db0016-dfed-11ee-95c8-87774f69a715 -- ceph daemon osd.4 dump_historic_ops 2024-03-11T21:32:44.076 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:32:44.109 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:32:44.109 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:32:44.110 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:32:44.110 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:32:43 smithi092 ceph-mon[29572]: from='client.24469 v1:172.21.15.97:0/404208310' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:32:44.111 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:32:43 smithi092 ceph-mon[29572]: osdmap e175: 8 total, 8 up, 8 in 2024-03-11T21:32:44.111 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:32:43 smithi092 ceph-mon[37563]: from='client.24469 v1:172.21.15.97:0/404208310' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:32:44.111 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:32:43 smithi092 ceph-mon[37563]: osdmap e175: 8 total, 8 up, 8 in 2024-03-11T21:32:44.211 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:32:44.242 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:32:44.243 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:32:44.247 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:32:44.326 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:32:43 smithi097 ceph-mon[33317]: from='client.24469 v1:172.21.15.97:0/404208310' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:32:44.327 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:32:43 smithi097 ceph-mon[33317]: osdmap e175: 8 total, 8 up, 8 in 2024-03-11T21:32:44.348 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:32:44.363 INFO:teuthology.orchestra.run.smithi097.stderr:Inferring config /var/lib/ceph/00db0016-dfed-11ee-95c8-87774f69a715/mon.b/config 2024-03-11T21:32:44.386 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:32:44.386 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:32:44.386 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:32:44.419 INFO:teuthology.orchestra.run.smithi092.stdout: 2024-03-11T21:32:44.422 INFO:teuthology.orchestra.run.smithi092.stderr:dumped all 2024-03-11T21:32:44.433 INFO:tasks.radosbench.radosbench.0.smithi092.stdout: 43 16 74377 74361 108.063 75.75 0.0115999 0.00924815 2024-03-11T21:32:44.487 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:32:44.538 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:32:44.538 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:32:44.538 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:32:44.639 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:32:44.695 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:32:44.696 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:32:44.696 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:32:44.797 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:32:44.839 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:32:44.840 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:32:44.840 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:32:44.941 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:32:44.978 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:32:44.979 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:32:44.979 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:32:45.076 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:32:44 smithi097 ceph-mon[33317]: pgmap v281: 148 pgs: 148 active+clean; 4.4 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 1.2 KiB/s rd, 97 MiB/s wr, 1.55k op/s 2024-03-11T21:32:45.081 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:32:45.115 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:32:45.115 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:32:45.115 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:32:45.156 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:32:44 smithi092 ceph-mon[29572]: pgmap v281: 148 pgs: 148 active+clean; 4.4 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 1.2 KiB/s rd, 97 MiB/s wr, 1.55k op/s 2024-03-11T21:32:45.157 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:32:44 smithi092 ceph-mon[37563]: pgmap v281: 148 pgs: 148 active+clean; 4.4 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 1.2 KiB/s rd, 97 MiB/s wr, 1.55k op/s 2024-03-11T21:32:45.201 INFO:teuthology.orchestra.run.smithi092.stdout:{"pg_ready":true,"pg_map":{"version":282,"stamp":"2024-03-11T21:32:43.225823+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":4702476712,"num_objects":71943,"num_object_clones":0,"num_object_copies":215829,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":71943,"num_whiteouts":0,"num_read":1337,"num_read_kb":1185,"num_write":72360,"num_write_kb":4594141,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":97,"num_bytes_recovered":2363708,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":72433,"ondisk_log_size":72433,"up":444,"acting":444,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":444,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":749961216,"kb_used":15059676,"kb_used_data":13635732,"kb_used_omap":12,"kb_used_meta":1423795,"kb_avail":734901540,"statfs":{"total":767960285184,"available":752539176960,"internally_reserved":0,"allocated":13962989568,"data_stored":13959184564,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12752,"internal_metadata":1457966640},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":706805760,"num_objects":10785,"num_object_clones":0,"num_object_copies":32355,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10785,"num_whiteouts":0,"num_read":10,"num_read_kb":10,"num_write":10785,"num_write_kb":690240,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.124383"},"pg_stats":[{"pgid":"6.1b","version":"0'0","reported_seq":244,"reported_epoch":174,"state":"active+clean","last_fresh":"2024-03-11T21:32:41.929487+0000","last_change":"2024-03-11T21:31:48.238290+0000","last_active":"2024-03-11T21:32:41.929487+0000","last_peered":"2024-03-11T21:32:41.929487+0000","last_clean":"2024-03-11T21:32:41.929487+0000","last_became_active":"2024-03-11T21:31:48.237220+0000","last_became_peered":"2024-03-11T21:31:48.237220+0000","last_unstale":"2024-03-11T21:32:41.929487+0000","last_undegraded":"2024-03-11T21:32:41.929487+0000","last_fullsized":"2024-03-11T21:32:41.929487+0000","mapping_epoch":153,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":154,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:31:06.310490+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:43.559452+0000","last_clean_scrub_stamp":"2024-03-11T21:31:06.310490+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,1],"acting":[3,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":243,"reported_epoch":174,"state":"active+clean","last_fresh":"2024-03-11T21:32:41.913400+0000","last_change":"2024-03-11T21:31:10.598620+0000","last_active":"2024-03-11T21:32:41.913400+0000","last_peered":"2024-03-11T21:32:41.913400+0000","last_clean":"2024-03-11T21:32:41.913400+0000","last_became_active":"2024-03-11T21:25:42.578893+0000","last_became_peered":"2024-03-11T21:25:42.578893+0000","last_unstale":"2024-03-11T21:32:41.913400+0000","last_undegraded":"2024-03-11T21:32:41.913400+0000","last_fullsized":"2024-03-11T21:32:41.913400+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:31:10.598515+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:41.544550+0000","last_clean_scrub_stamp":"2024-03-11T21:31:10.598515+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1f","version":"0'0","reported_seq":266,"reported_epoch":174,"state":"active+clean","last_fresh":"2024-03-11T21:32:41.918661+0000","last_change":"2024-03-11T21:31:54.380555+0000","last_active":"2024-03-11T21:32:41.918661+0000","last_peered":"2024-03-11T21:32:41.918661+0000","last_clean":"2024-03-11T21:32:41.918661+0000","last_became_active":"2024-03-11T21:31:48.225548+0000","last_became_peered":"2024-03-11T21:31:48.225548+0000","last_unstale":"2024-03-11T21:32:41.918661+0000","last_undegraded":"2024-03-11T21:32:41.918661+0000","last_fullsized":"2024-03-11T21:32:41.918661+0000","mapping_epoch":153,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":154,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:31:54.380472+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:37.512018+0000","last_clean_scrub_stamp":"2024-03-11T21:31:54.380472+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:33:09.244401+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,6],"acting":[0,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.19","version":"74'15","reported_seq":271,"reported_epoch":174,"state":"active+clean","last_fresh":"2024-03-11T21:32:41.925487+0000","last_change":"2024-03-11T21:31:07.311590+0000","last_active":"2024-03-11T21:32:41.925487+0000","last_peered":"2024-03-11T21:32:41.925487+0000","last_clean":"2024-03-11T21:32:41.925487+0000","last_became_active":"2024-03-11T21:25:40.554207+0000","last_became_peered":"2024-03-11T21:25:40.554207+0000","last_unstale":"2024-03-11T21:32:41.925487+0000","last_undegraded":"2024-03-11T21:32:41.925487+0000","last_fullsized":"2024-03-11T21:32:41.925487+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'15","last_scrub_stamp":"2024-03-11T21:31:07.311470+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:39.525554+0000","last_clean_scrub_stamp":"2024-03-11T21:31:07.311470+0000","objects_scrubbed":7,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.1a","version":"0'0","reported_seq":239,"reported_epoch":174,"state":"active+clean","last_fresh":"2024-03-11T21:32:41.913905+0000","last_change":"2024-03-11T21:32:01.718632+0000","last_active":"2024-03-11T21:32:41.913905+0000","last_peered":"2024-03-11T21:32:41.913905+0000","last_clean":"2024-03-11T21:32:41.913905+0000","last_became_active":"2024-03-11T21:25:44.590923+0000","last_became_peered":"2024-03-11T21:25:44.590923+0000","last_unstale":"2024-03-11T21:32:41.913905+0000","last_undegraded":"2024-03-11T21:32:41.913905+0000","last_fullsized":"2024-03-11T21:32:41.913905+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:32:01.718486+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:43.559452+0000","last_clean_scrub_stamp":"2024-03-11T21:32:01.718486+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:33:26.052102+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,1],"acting":[4,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":249,"reported_epoch":174,"state":"active+clean","last_fresh":"2024-03-11T21:32:41.927311+0000","last_change":"2024-03-11T21:31:48.226625+0000","last_active":"2024-03-11T21:32:41.927311+0000","last_peered":"2024-03-11T21:32:41.927311+0000","last_clean":"2024-03-11T21:32:41.927311+0000","last_became_active":"2024-03-11T21:31:48.226180+0000","last_became_peered":"2024-03-11T21:31:48.226180+0000","last_unstale":"2024-03-11T21:32:41.927311+0000","last_undegraded":"2024-03-11T21:32:41.927311+0000","last_fullsized":"2024-03-11T21:32:41.927311+0000","mapping_epoch":153,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":154,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:31:15.812251+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:41.544550+0000","last_clean_scrub_stamp":"2024-03-11T21:31:15.812251+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:32:43.569436+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,2],"acting":[1,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1e","version":"0'0","reported_seq":259,"reported_epoch":174,"state":"active+clean","last_fresh":"2024-03-11T21:32:41.926716+0000","last_change":"2024-03-11T21:32:18.578157+0000","last_active":"2024-03-11T21:32:41.926716+0000","last_peered":"2024-03-11T21:32:41.926716+0000","last_clean":"2024-03-11T21:32:41.926716+0000","last_became_active":"2024-03-11T21:25:38.537283+0000","last_became_peered":"2024-03-11T21:25:38.537283+0000","last_unstale":"2024-03-11T21:32:41.926716+0000","last_undegraded":"2024-03-11T21:32:41.926716+0000","last_fullsized":"2024-03-11T21:32:41.926716+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:32:18.578038+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:37.512018+0000","last_clean_scrub_stamp":"2024-03-11T21:32:18.578038+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:33:34.708987+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.18","version":"74'9","reported_seq":271,"reported_epoch":174,"state":"active+clean","last_fresh":"2024-03-11T21:32:41.913170+0000","last_change":"2024-03-11T21:32:00.674307+0000","last_active":"2024-03-11T21:32:41.913170+0000","last_peered":"2024-03-11T21:32:41.913170+0000","last_clean":"2024-03-11T21:32:41.913170+0000","last_became_active":"2024-03-11T21:25:40.552216+0000","last_became_peered":"2024-03-11T21:25:40.552216+0000","last_unstale":"2024-03-11T21:32:41.913170+0000","last_undegraded":"2024-03-11T21:32:41.913170+0000","last_fullsized":"2024-03-11T21:32:41.913170+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'9","last_scrub_stamp":"2024-03-11T21:32:00.674188+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:39.525554+0000","last_clean_scrub_stamp":"2024-03-11T21:32:00.674188+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:33:07.357445+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":247,"reported_epoch":174,"state":"active+clean","last_fresh":"2024-03-11T21:32:41.928190+0000","last_change":"2024-03-11T21:32:12.654714+0000","last_active":"2024-03-11T21:32:41.928190+0000","last_peered":"2024-03-11T21:32:41.928190+0000","last_clean":"2024-03-11T21:32:41.928190+0000","last_became_active":"2024-03-11T21:25:44.590234+0000","last_became_peered":"2024-03-11T21:25:44.590234+0000","last_unstale":"2024-03-11T21:32:41.928190+0000","last_undegraded":"2024-03-11T21:32:41.928190+0000","last_fullsized":"2024-03-11T21:32:41.928190+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:32:12.654573+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:32:12.654573+0000","last_clean_scrub_stamp":"2024-03-11T21:32:12.654573+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:33:29.244319+0000","scrub_duration":43,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3],"acting":[5,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":60,"reported_epoch":174,"state":"active+clean","last_fresh":"2024-03-11T21:32:41.914171+0000","last_change":"2024-03-11T21:31:56.567886+0000","last_active":"2024-03-11T21:32:41.914171+0000","last_peered":"2024-03-11T21:32:41.914171+0000","last_clean":"2024-03-11T21:32:41.914171+0000","last_became_active":"2024-03-11T21:31:48.218366+0000","last_became_peered":"2024-03-11T21:31:48.218366+0000","last_unstale":"2024-03-11T21:32:41.914171+0000","last_undegraded":"2024-03-11T21:32:41.914171+0000","last_fullsized":"2024-03-11T21:32:41.914171+0000","mapping_epoch":153,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":154,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:31:56.567802+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:41.544550+0000","last_clean_scrub_stamp":"2024-03-11T21:31:56.567802+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:33:19.092666+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,0],"acting":[4,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1d","version":"0'0","reported_seq":258,"reported_epoch":174,"state":"active+clean","last_fresh":"2024-03-11T21:32:41.952987+0000","last_change":"2024-03-11T21:31:57.371941+0000","last_active":"2024-03-11T21:32:41.952987+0000","last_peered":"2024-03-11T21:32:41.952987+0000","last_clean":"2024-03-11T21:32:41.952987+0000","last_became_active":"2024-03-11T21:25:38.542628+0000","last_became_peered":"2024-03-11T21:25:38.542628+0000","last_unstale":"2024-03-11T21:32:41.952987+0000","last_undegraded":"2024-03-11T21:32:41.952987+0000","last_fullsized":"2024-03-11T21:32:41.952987+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:31:57.371862+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:37.512018+0000","last_clean_scrub_stamp":"2024-03-11T21:31:57.371862+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:33:26.732272+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,0],"acting":[7,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.1b","version":"74'5","reported_seq":257,"reported_epoch":174,"state":"active+clean","last_fresh":"2024-03-11T21:32:41.914110+0000","last_change":"2024-03-11T21:31:57.621692+0000","last_active":"2024-03-11T21:32:41.914110+0000","last_peered":"2024-03-11T21:32:41.914110+0000","last_clean":"2024-03-11T21:32:41.914110+0000","last_became_active":"2024-03-11T21:25:40.555826+0000","last_became_peered":"2024-03-11T21:25:40.555826+0000","last_unstale":"2024-03-11T21:32:41.914110+0000","last_undegraded":"2024-03-11T21:32:41.914110+0000","last_fullsized":"2024-03-11T21:32:41.914110+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'5","last_scrub_stamp":"2024-03-11T21:31:57.621580+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:39.525554+0000","last_clean_scrub_stamp":"2024-03-11T21:31:57.621580+0000","objects_scrubbed":2,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:33:18.541742+0000","scrub_duration":34,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":11,"num_read_kb":7,"num_write":6,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,1],"acting":[4,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":239,"reported_epoch":174,"state":"active+clean","last_fresh":"2024-03-11T21:32:41.923510+0000","last_change":"2024-03-11T21:31:58.402086+0000","last_active":"2024-03-11T21:32:41.923510+0000","last_peered":"2024-03-11T21:32:41.923510+0000","last_clean":"2024-03-11T21:32:41.923510+0000","last_became_active":"2024-03-11T21:25:44.589786+0000","last_became_peered":"2024-03-11T21:25:44.589786+0000","last_unstale":"2024-03-11T21:32:41.923510+0000","last_undegraded":"2024-03-11T21:32:41.923510+0000","last_fullsized":"2024-03-11T21:32:41.923510+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:31:58.401956+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:43.559452+0000","last_clean_scrub_stamp":"2024-03-11T21:31:58.401956+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:33:09.708327+0000","scrub_duration":20,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,7],"acting":[0,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":256,"reported_epoch":174,"state":"active+clean","last_fresh":"2024-03-11T21:32:41.928438+0000","last_change":"2024-03-11T21:32:21.535197+0000","last_active":"2024-03-11T21:32:41.928438+0000","last_peered":"2024-03-11T21:32:41.928438+0000","last_clean":"2024-03-11T21:32:41.928438+0000","last_became_active":"2024-03-11T21:31:48.234980+0000","last_became_peered":"2024-03-11T21:31:48.234980+0000","last_unstale":"2024-03-11T21:32:41.928438+0000","last_undegraded":"2024-03-11T21:32:41.928438+0000","last_fullsized":"2024-03-11T21:32:41.928438+0000","mapping_epoch":153,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":154,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:32:21.535034+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:32:21.535034+0000","last_clean_scrub_stamp":"2024-03-11T21:32:21.535034+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:33:32.724425+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,4],"acting":[5,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1c","version":"0'0","reported_seq":258,"reported_epoch":174,"state":"active+clean","last_fresh":"2024-03-11T21:32:41.952959+0000","last_change":"2024-03-11T21:31:58.422366+0000","last_active":"2024-03-11T21:32:41.952959+0000","last_peered":"2024-03-11T21:32:41.952959+0000","last_clean":"2024-03-11T21:32:41.952959+0000","last_became_active":"2024-03-11T21:25:38.541364+0000","last_became_peered":"2024-03-11T21:25:38.541364+0000","last_unstale":"2024-03-11T21:32:41.952959+0000","last_undegraded":"2024-03-11T21:32:41.952959+0000","last_fullsized":"2024-03-11T21:32:41.952959+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:31:58.422287+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:28:32.566437+0000","last_clean_scrub_stamp":"2024-03-11T21:31:58.422287+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:33:23.582317+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.1a","version":"74'9","reported_seq":271,"reported_epoch":174,"state":"active+clean","last_fresh":"2024-03-11T21:32:41.914003+0000","last_change":"2024-03-11T21:32:04.628796+0000","last_active":"2024-03-11T21:32:41.914003+0000","last_peered":"2024-03-11T21:32:41.914003+0000","last_clean":"2024-03-11T21:32:41.914003+0000","last_became_active":"2024-03-11T21:25:40.556629+0000","last_became_peered":"2024-03-11T21:25:40.556629+0000","last_unstale":"2024-03-11T21:32:41.914003+0000","last_undegraded":"2024-03-11T21:32:41.914003+0000","last_fullsized":"2024-03-11T21:32:41.914003+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'9","last_scrub_stamp":"2024-03-11T21:32:04.628652+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:39.525554+0000","last_clean_scrub_stamp":"2024-03-11T21:32:04.628652+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:33:09.872947+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0],"acting":[4,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1f","version":"0'0","reported_seq":239,"reported_epoch":174,"state":"active+clean","last_fresh":"2024-03-11T21:32:41.926200+0000","last_change":"2024-03-11T21:31:04.253653+0000","last_active":"2024-03-11T21:32:41.926200+0000","last_peered":"2024-03-11T21:32:41.926200+0000","last_clean":"2024-03-11T21:32:41.926200+0000","last_became_active":"2024-03-11T21:25:45.302637+0000","last_became_peered":"2024-03-11T21:25:45.302637+0000","last_unstale":"2024-03-11T21:32:41.926200+0000","last_undegraded":"2024-03-11T21:32:41.926200+0000","last_fullsized":"2024-03-11T21:32:41.926200+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:31:04.253137+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:43.559452+0000","last_clean_scrub_stamp":"2024-03-11T21:31:04.253137+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1c","version":"0'0","reported_seq":243,"reported_epoch":174,"state":"active+clean","last_fresh":"2024-03-11T21:32:41.913820+0000","last_change":"2024-03-11T21:30:59.496458+0000","last_active":"2024-03-11T21:32:41.913820+0000","last_peered":"2024-03-11T21:32:41.913820+0000","last_clean":"2024-03-11T21:32:41.913820+0000","last_became_active":"2024-03-11T21:25:42.576064+0000","last_became_peered":"2024-03-11T21:25:42.576064+0000","last_unstale":"2024-03-11T21:32:41.913820+0000","last_undegraded":"2024-03-11T21:32:41.913820+0000","last_fullsized":"2024-03-11T21:32:41.913820+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:30:59.496330+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:41.544550+0000","last_clean_scrub_stamp":"2024-03-11T21:30:59.496330+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1b","version":"0'0","reported_seq":256,"reported_epoch":174,"state":"active+clean","last_fresh":"2024-03-11T21:32:41.929698+0000","last_change":"2024-03-11T21:31:48.233696+0000","last_active":"2024-03-11T21:32:41.929698+0000","last_peered":"2024-03-11T21:32:41.929698+0000","last_clean":"2024-03-11T21:32:41.929698+0000","last_became_active":"2024-03-11T21:31:48.233345+0000","last_became_peered":"2024-03-11T21:31:48.233345+0000","last_unstale":"2024-03-11T21:32:41.929698+0000","last_undegraded":"2024-03-11T21:32:41.929698+0000","last_fullsized":"2024-03-11T21:32:41.929698+0000","mapping_epoch":153,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":154,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:31:05.272969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:28:33.582712+0000","last_clean_scrub_stamp":"2024-03-11T21:31:05.272969+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,4],"acting":[3,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1d","version":"74'12","reported_seq":269,"reported_epoch":174,"state":"active+clean","last_fresh":"2024-03-11T21:32:41.926319+0000","last_change":"2024-03-11T21:31:20.514563+0000","last_active":"2024-03-11T21:32:41.926319+0000","last_peered":"2024-03-11T21:32:41.926319+0000","last_clean":"2024-03-11T21:32:41.926319+0000","last_became_active":"2024-03-11T21:25:40.556330+0000","last_became_peered":"2024-03-11T21:25:40.556330+0000","last_unstale":"2024-03-11T21:32:41.926319+0000","last_undegraded":"2024-03-11T21:32:41.926319+0000","last_fullsized":"2024-03-11T21:32:41.926319+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'12","last_scrub_stamp":"2024-03-11T21:31:20.514423+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:39.525554+0000","last_clean_scrub_stamp":"2024-03-11T21:31:20.514423+0000","objects_scrubbed":5,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":25,"num_read_kb":16,"num_write":14,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.1e","version":"0'0","reported_seq":240,"reported_epoch":174,"state":"active+clean","last_fresh":"2024-03-11T21:32:41.914600+0000","last_change":"2024-03-11T21:32:19.599182+0000","last_active":"2024-03-11T21:32:41.914600+0000","last_peered":"2024-03-11T21:32:41.914600+0000","last_clean":"2024-03-11T21:32:41.914600+0000","last_became_active":"2024-03-11T21:25:45.300355+0000","last_became_peered":"2024-03-11T21:25:45.300355+0000","last_unstale":"2024-03-11T21:32:41.914600+0000","last_undegraded":"2024-03-11T21:32:41.914600+0000","last_fullsized":"2024-03-11T21:32:41.914600+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:32:19.599050+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:43.559452+0000","last_clean_scrub_stamp":"2024-03-11T21:32:19.599050+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:33:45.661826+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5],"acting":[4,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1d","version":"0'0","reported_seq":243,"reported_epoch":174,"state":"active+clean","last_fresh":"2024-03-11T21:32:41.929681+0000","last_change":"2024-03-11T21:30:18.678642+0000","last_active":"2024-03-11T21:32:41.929681+0000","last_peered":"2024-03-11T21:32:41.929681+0000","last_clean":"2024-03-11T21:32:41.929681+0000","last_became_active":"2024-03-11T21:25:42.579407+0000","last_became_peered":"2024-03-11T21:25:42.579407+0000","last_unstale":"2024-03-11T21:32:41.929681+0000","last_undegraded":"2024-03-11T21:32:41.929681+0000","last_fullsized":"2024-03-11T21:32:41.929681+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:30:18.678531+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:30:18.678531+0000","last_clean_scrub_stamp":"2024-03-11T21:30:18.678531+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:32:43.569275+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1a","version":"0'0","reported_seq":265,"reported_epoch":174,"state":"active+clean","last_fresh":"2024-03-11T21:32:41.924218+0000","last_change":"2024-03-11T21:31:59.716486+0000","last_active":"2024-03-11T21:32:41.924218+0000","last_peered":"2024-03-11T21:32:41.924218+0000","last_clean":"2024-03-11T21:32:41.924218+0000","last_became_active":"2024-03-11T21:31:48.222582+0000","last_became_peered":"2024-03-11T21:31:48.222582+0000","last_unstale":"2024-03-11T21:32:41.924218+0000","last_undegraded":"2024-03-11T21:32:41.924218+0000","last_fullsized":"2024-03-11T21:32:41.924218+0000","mapping_epoch":153,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":154,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:31:59.716376+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:37.512018+0000","last_clean_scrub_stamp":"2024-03-11T21:31:59.716376+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:33:05.765786+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,2],"acting":[6,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.1c","version":"74'15","reported_seq":271,"reported_epoch":174,"state":"active+clean","last_fresh":"2024-03-11T21:32:41.939308+0000","last_change":"2024-03-11T21:31:01.272353+0000","last_active":"2024-03-11T21:32:41.939308+0000","last_peered":"2024-03-11T21:32:41.939308+0000","last_clean":"2024-03-11T21:32:41.939308+0000","last_became_active":"2024-03-11T21:25:40.551453+0000","last_became_peered":"2024-03-11T21:25:40.551453+0000","last_unstale":"2024-03-11T21:32:41.939308+0000","last_undegraded":"2024-03-11T21:32:41.939308+0000","last_fullsized":"2024-03-11T21:32:41.939308+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'15","last_scrub_stamp":"2024-03-11T21:31:01.272189+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:39.525554+0000","last_clean_scrub_stamp":"2024-03-11T21:31:01.272189+0000","objects_scrubbed":7,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":22,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.1d","version":"0'0","reported_seq":239,"reported_epoch":174,"state":"active+clean","last_fresh":"2024-03-11T21:32:41.929136+0000","last_change":"2024-03-11T21:31:18.798813+0000","last_active":"2024-03-11T21:32:41.929136+0000","last_peered":"2024-03-11T21:32:41.929136+0000","last_clean":"2024-03-11T21:32:41.929136+0000","last_became_active":"2024-03-11T21:25:44.588845+0000","last_became_peered":"2024-03-11T21:25:44.588845+0000","last_unstale":"2024-03-11T21:32:41.929136+0000","last_undegraded":"2024-03-11T21:32:41.929136+0000","last_fullsized":"2024-03-11T21:32:41.929136+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:31:18.798701+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:43.559452+0000","last_clean_scrub_stamp":"2024-03-11T21:31:18.798701+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:32:47.454620+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.1e","version":"0'0","reported_seq":252,"reported_epoch":174,"state":"active+clean","last_fresh":"2024-03-11T21:32:41.922493+0000","last_change":"2024-03-11T21:32:07.397655+0000","last_active":"2024-03-11T21:32:41.922493+0000","last_peered":"2024-03-11T21:32:41.922493+0000","last_clean":"2024-03-11T21:32:41.922493+0000","last_became_active":"2024-03-11T21:25:42.582959+0000","last_became_peered":"2024-03-11T21:25:42.582959+0000","last_unstale":"2024-03-11T21:32:41.922493+0000","last_undegraded":"2024-03-11T21:32:41.922493+0000","last_fullsized":"2024-03-11T21:32:41.922493+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:32:07.397538+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:26:58.335431+0000","last_clean_scrub_stamp":"2024-03-11T21:32:07.397538+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:33:32.766194+0000","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.19","version":"67'2","reported_seq":257,"reported_epoch":174,"state":"active+clean","last_fresh":"2024-03-11T21:32:41.925607+0000","last_change":"2024-03-11T21:32:15.509684+0000","last_active":"2024-03-11T21:32:41.925607+0000","last_peered":"2024-03-11T21:32:41.925607+0000","last_clean":"2024-03-11T21:32:41.925607+0000","last_became_active":"2024-03-11T21:25:38.539243+0000","last_became_peered":"2024-03-11T21:25:38.539243+0000","last_unstale":"2024-03-11T21:32:41.925607+0000","last_undegraded":"2024-03-11T21:32:41.925607+0000","last_fullsized":"2024-03-11T21:32:41.925607+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'2","last_scrub_stamp":"2024-03-11T21:32:15.509542+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:37.512018+0000","last_clean_scrub_stamp":"2024-03-11T21:32:15.509542+0000","objects_scrubbed":2,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:33:26.153151+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":957,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":7,"num_read_kb":7,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,0],"acting":[3,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1f","version":"124'16","reported_seq":298,"reported_epoch":174,"state":"active+clean","last_fresh":"2024-03-11T21:32:41.925757+0000","last_change":"2024-03-11T21:32:22.821766+0000","last_active":"2024-03-11T21:32:41.925757+0000","last_peered":"2024-03-11T21:32:41.925757+0000","last_clean":"2024-03-11T21:32:41.925757+0000","last_became_active":"2024-03-11T21:32:22.798242+0000","last_became_peered":"2024-03-11T21:32:22.798242+0000","last_unstale":"2024-03-11T21:32:41.925757+0000","last_undegraded":"2024-03-11T21:32:41.925757+0000","last_fullsized":"2024-03-11T21:32:41.925757+0000","mapping_epoch":166,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":167,"parent":"0.0","parent_split_bits":0,"last_scrub":"124'16","last_scrub_stamp":"2024-03-11T21:31:06.449509+0000","last_deep_scrub":"124'16","last_deep_scrub_stamp":"2024-03-11T21:29:30.873490+0000","last_clean_scrub_stamp":"2024-03-11T21:31:06.449509+0000","objects_scrubbed":5,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:32:43.941436+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,0],"acting":[6,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.1c","version":"74'1","reported_seq":67,"reported_epoch":174,"state":"active+clean","last_fresh":"2024-03-11T21:32:41.927532+0000","last_change":"2024-03-11T21:31:48.249824+0000","last_active":"2024-03-11T21:32:41.927532+0000","last_peered":"2024-03-11T21:32:41.927532+0000","last_clean":"2024-03-11T21:32:41.927532+0000","last_became_active":"2024-03-11T21:31:48.236162+0000","last_became_peered":"2024-03-11T21:31:48.236162+0000","last_unstale":"2024-03-11T21:32:41.927532+0000","last_undegraded":"2024-03-11T21:32:41.927532+0000","last_fullsized":"2024-03-11T21:32:41.927532+0000","mapping_epoch":153,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":154,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'1","last_scrub_stamp":"2024-03-11T21:31:07.489215+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:43.559452+0000","last_clean_scrub_stamp":"2024-03-11T21:31:07.489215+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:32:45.331147+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":370,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":370,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,3],"acting":[5,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.1f","version":"154'9","reported_seq":366,"reported_epoch":174,"state":"active+clean","last_fresh":"2024-03-11T21:32:41.925658+0000","last_change":"2024-03-11T21:32:12.885181+0000","last_active":"2024-03-11T21:32:41.925658+0000","last_peered":"2024-03-11T21:32:41.925658+0000","last_clean":"2024-03-11T21:32:41.925658+0000","last_became_active":"2024-03-11T21:31:48.227717+0000","last_became_peered":"2024-03-11T21:31:48.227717+0000","last_unstale":"2024-03-11T21:32:41.925658+0000","last_undegraded":"2024-03-11T21:32:41.925658+0000","last_fullsized":"2024-03-11T21:32:41.925658+0000","mapping_epoch":153,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":154,"parent":"0.0","parent_split_bits":0,"last_scrub":"154'9","last_scrub_stamp":"2024-03-11T21:32:12.885043+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:41.544550+0000","last_clean_scrub_stamp":"2024-03-11T21:32:12.885043+0000","objects_scrubbed":1,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:33:15.212011+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,2],"acting":[6,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.18","version":"67'1","reported_seq":274,"reported_epoch":174,"state":"active+clean","last_fresh":"2024-03-11T21:32:41.927038+0000","last_change":"2024-03-11T21:31:59.576103+0000","last_active":"2024-03-11T21:32:41.927038+0000","last_peered":"2024-03-11T21:32:41.927038+0000","last_clean":"2024-03-11T21:32:41.927038+0000","last_became_active":"2024-03-11T21:31:48.225066+0000","last_became_peered":"2024-03-11T21:31:48.225066+0000","last_unstale":"2024-03-11T21:32:41.927038+0000","last_undegraded":"2024-03-11T21:32:41.927038+0000","last_fullsized":"2024-03-11T21:32:41.927038+0000","mapping_epoch":153,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":154,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'1","last_scrub_stamp":"2024-03-11T21:31:59.575976+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:37.512018+0000","last_clean_scrub_stamp":"2024-03-11T21:31:59.575976+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:32:59.997189+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":436,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":7,"num_read_kb":7,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":436,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1e","version":"74'10","reported_seq":281,"reported_epoch":174,"state":"active+clean","last_fresh":"2024-03-11T21:32:41.922412+0000","last_change":"2024-03-11T21:31:48.252894+0000","last_active":"2024-03-11T21:32:41.922412+0000","last_peered":"2024-03-11T21:32:41.922412+0000","last_clean":"2024-03-11T21:32:41.922412+0000","last_became_active":"2024-03-11T21:31:48.231510+0000","last_became_peered":"2024-03-11T21:31:48.231510+0000","last_unstale":"2024-03-11T21:32:41.922412+0000","last_undegraded":"2024-03-11T21:32:41.922412+0000","last_fullsized":"2024-03-11T21:32:41.922412+0000","mapping_epoch":153,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":154,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'10","last_scrub_stamp":"2024-03-11T21:30:57.371717+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:39.525554+0000","last_clean_scrub_stamp":"2024-03-11T21:30:57.371717+0000","objects_scrubbed":5,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:32:44.469253+0000","scrub_duration":27,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1],"acting":[0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.13","version":"0'0","reported_seq":238,"reported_epoch":174,"state":"active+clean","last_fresh":"2024-03-11T21:32:41.929923+0000","last_change":"2024-03-11T21:31:00.239533+0000","last_active":"2024-03-11T21:32:41.929923+0000","last_peered":"2024-03-11T21:32:41.929923+0000","last_clean":"2024-03-11T21:32:41.929923+0000","last_became_active":"2024-03-11T21:25:45.302102+0000","last_became_peered":"2024-03-11T21:25:45.302102+0000","last_unstale":"2024-03-11T21:32:41.929923+0000","last_undegraded":"2024-03-11T21:32:41.929923+0000","last_fullsized":"2024-03-11T21:32:41.929923+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:31:00.238325+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:43.559452+0000","last_clean_scrub_stamp":"2024-03-11T21:31:00.238325+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":60,"reported_epoch":174,"state":"active+clean","last_fresh":"2024-03-11T21:32:41.913230+0000","last_change":"2024-03-11T21:32:08.612616+0000","last_active":"2024-03-11T21:32:41.913230+0000","last_peered":"2024-03-11T21:32:41.913230+0000","last_clean":"2024-03-11T21:32:41.913230+0000","last_became_active":"2024-03-11T21:31:48.224831+0000","last_became_peered":"2024-03-11T21:31:48.224831+0000","last_unstale":"2024-03-11T21:32:41.913230+0000","last_undegraded":"2024-03-11T21:32:41.913230+0000","last_fullsized":"2024-03-11T21:32:41.913230+0000","mapping_epoch":153,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":154,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:32:08.612508+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:41.544550+0000","last_clean_scrub_stamp":"2024-03-11T21:32:08.612508+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:33:10.063162+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,0],"acting":[4,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.17","version":"0'0","reported_seq":259,"reported_epoch":174,"state":"active+clean","last_fresh":"2024-03-11T21:32:41.925489+0000","last_change":"2024-03-11T21:32:00.758583+0000","last_active":"2024-03-11T21:32:41.925489+0000","last_peered":"2024-03-11T21:32:41.925489+0000","last_clean":"2024-03-11T21:32:41.925489+0000","last_became_active":"2024-03-11T21:25:38.540349+0000","last_became_peered":"2024-03-11T21:25:38.540349+0000","last_unstale":"2024-03-11T21:32:41.925489+0000","last_undegraded":"2024-03-11T21:32:41.925489+0000","last_fullsized":"2024-03-11T21:32:41.925489+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:32:00.758492+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:28:25.918101+0000","last_clean_scrub_stamp":"2024-03-11T21:32:00.758492+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:33:27.761241+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,2],"acting":[6,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.11","version":"74'11","reported_seq":284,"reported_epoch":174,"state":"active+clean","last_fresh":"2024-03-11T21:32:41.925797+0000","last_change":"2024-03-11T21:31:48.250922+0000","last_active":"2024-03-11T21:32:41.925797+0000","last_peered":"2024-03-11T21:32:41.925797+0000","last_clean":"2024-03-11T21:32:41.925797+0000","last_became_active":"2024-03-11T21:31:48.229218+0000","last_became_peered":"2024-03-11T21:31:48.229218+0000","last_unstale":"2024-03-11T21:32:41.925797+0000","last_undegraded":"2024-03-11T21:32:41.925797+0000","last_fullsized":"2024-03-11T21:32:41.925797+0000","mapping_epoch":153,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":154,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'11","last_scrub_stamp":"2024-03-11T21:31:17.450051+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:39.525554+0000","last_clean_scrub_stamp":"2024-03-11T21:31:17.450051+0000","objects_scrubbed":5,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":60,"reported_epoch":174,"state":"active+clean","last_fresh":"2024-03-11T21:32:41.939600+0000","last_change":"2024-03-11T21:31:55.360729+0000","last_active":"2024-03-11T21:32:41.939600+0000","last_peered":"2024-03-11T21:32:41.939600+0000","last_clean":"2024-03-11T21:32:41.939600+0000","last_became_active":"2024-03-11T21:31:48.226446+0000","last_became_peered":"2024-03-11T21:31:48.226446+0000","last_unstale":"2024-03-11T21:32:41.939600+0000","last_undegraded":"2024-03-11T21:32:41.939600+0000","last_fullsized":"2024-03-11T21:32:41.939600+0000","mapping_epoch":153,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":154,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:31:55.360650+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:43.559452+0000","last_clean_scrub_stamp":"2024-03-11T21:31:55.360650+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:32:56.657476+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,3],"acting":[2,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":258,"reported_epoch":174,"state":"active+clean","last_fresh":"2024-03-11T21:32:41.924521+0000","last_change":"2024-03-11T21:31:48.223455+0000","last_active":"2024-03-11T21:32:41.924521+0000","last_peered":"2024-03-11T21:32:41.924521+0000","last_clean":"2024-03-11T21:32:41.924521+0000","last_became_active":"2024-03-11T21:31:48.223088+0000","last_became_peered":"2024-03-11T21:31:48.223088+0000","last_unstale":"2024-03-11T21:32:41.924521+0000","last_undegraded":"2024-03-11T21:32:41.924521+0000","last_fullsized":"2024-03-11T21:32:41.924521+0000","mapping_epoch":153,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":154,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:31:12.423671+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:29:39.923895+0000","last_clean_scrub_stamp":"2024-03-11T21:31:12.423671+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:32:43.941376+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,0],"acting":[6,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.16","version":"0'0","reported_seq":258,"reported_epoch":174,"state":"active+clean","last_fresh":"2024-03-11T21:32:41.927331+0000","last_change":"2024-03-11T21:30:59.528390+0000","last_active":"2024-03-11T21:32:41.927331+0000","last_peered":"2024-03-11T21:32:41.927331+0000","last_clean":"2024-03-11T21:32:41.927331+0000","last_became_active":"2024-03-11T21:25:38.539103+0000","last_became_peered":"2024-03-11T21:25:38.539103+0000","last_unstale":"2024-03-11T21:32:41.927331+0000","last_undegraded":"2024-03-11T21:32:41.927331+0000","last_fullsized":"2024-03-11T21:32:41.927331+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:30:59.528265+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:27:11.938909+0000","last_clean_scrub_stamp":"2024-03-11T21:30:59.528265+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:32:45.331070+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,2],"acting":[5,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.10","version":"94'6","reported_seq":255,"reported_epoch":174,"state":"active+clean","last_fresh":"2024-03-11T21:32:41.931253+0000","last_change":"2024-03-11T21:30:21.932499+0000","last_active":"2024-03-11T21:32:41.931253+0000","last_peered":"2024-03-11T21:32:41.931253+0000","last_clean":"2024-03-11T21:32:41.931253+0000","last_became_active":"2024-03-11T21:25:40.552720+0000","last_became_peered":"2024-03-11T21:25:40.552720+0000","last_unstale":"2024-03-11T21:32:41.931253+0000","last_undegraded":"2024-03-11T21:32:41.931253+0000","last_fullsized":"2024-03-11T21:32:41.931253+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"94'6","last_scrub_stamp":"2024-03-11T21:30:21.932340+0000","last_deep_scrub":"94'6","last_deep_scrub_stamp":"2024-03-11T21:28:24.603974+0000","last_clean_scrub_stamp":"2024-03-11T21:30:21.932340+0000","objects_scrubbed":2,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":6,"num_read_kb":4,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6],"acting":[3,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":238,"reported_epoch":174,"state":"active+clean","last_fresh":"2024-03-11T21:32:41.925310+0000","last_change":"2024-03-11T21:31:14.338908+0000","last_active":"2024-03-11T21:32:41.925310+0000","last_peered":"2024-03-11T21:32:41.925310+0000","last_clean":"2024-03-11T21:32:41.925310+0000","last_became_active":"2024-03-11T21:25:44.583181+0000","last_became_peered":"2024-03-11T21:25:44.583181+0000","last_unstale":"2024-03-11T21:32:41.925310+0000","last_undegraded":"2024-03-11T21:32:41.925310+0000","last_fullsized":"2024-03-11T21:32:41.925310+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:31:14.338640+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:43.559452+0000","last_clean_scrub_stamp":"2024-03-11T21:31:14.338640+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":244,"reported_epoch":174,"state":"active+clean","last_fresh":"2024-03-11T21:32:41.930629+0000","last_change":"2024-03-11T21:30:19.680660+0000","last_active":"2024-03-11T21:32:41.930629+0000","last_peered":"2024-03-11T21:32:41.930629+0000","last_clean":"2024-03-11T21:32:41.930629+0000","last_became_active":"2024-03-11T21:25:42.573158+0000","last_became_peered":"2024-03-11T21:25:42.573158+0000","last_unstale":"2024-03-11T21:32:41.930629+0000","last_undegraded":"2024-03-11T21:32:41.930629+0000","last_fullsized":"2024-03-11T21:32:41.930629+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:30:19.680539+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:27:21.207303+0000","last_clean_scrub_stamp":"2024-03-11T21:30:19.680539+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:32:43.569312+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.15","version":"0'0","reported_seq":259,"reported_epoch":174,"state":"active+clean","last_fresh":"2024-03-11T21:32:41.929338+0000","last_change":"2024-03-11T21:32:12.600491+0000","last_active":"2024-03-11T21:32:41.929338+0000","last_peered":"2024-03-11T21:32:41.929338+0000","last_clean":"2024-03-11T21:32:41.929338+0000","last_became_active":"2024-03-11T21:25:38.541705+0000","last_became_peered":"2024-03-11T21:25:38.541705+0000","last_unstale":"2024-03-11T21:32:41.929338+0000","last_undegraded":"2024-03-11T21:32:41.929338+0000","last_fullsized":"2024-03-11T21:32:41.929338+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:32:12.600388+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:26:44.860499+0000","last_clean_scrub_stamp":"2024-03-11T21:32:12.600388+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:33:39.538663+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.13","version":"109'16","reported_seq":298,"reported_epoch":174,"state":"active+clean","last_fresh":"2024-03-11T21:32:41.913993+0000","last_change":"2024-03-11T21:32:24.657615+0000","last_active":"2024-03-11T21:32:41.913993+0000","last_peered":"2024-03-11T21:32:41.913993+0000","last_clean":"2024-03-11T21:32:41.913993+0000","last_became_active":"2024-03-11T21:26:11.214879+0000","last_became_peered":"2024-03-11T21:26:11.214879+0000","last_unstale":"2024-03-11T21:32:41.913993+0000","last_undegraded":"2024-03-11T21:32:41.913993+0000","last_fullsized":"2024-03-11T21:32:41.913993+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"109'16","last_scrub_stamp":"2024-03-11T21:32:24.657490+0000","last_deep_scrub":"109'16","last_deep_scrub_stamp":"2024-03-11T21:32:24.657490+0000","last_clean_scrub_stamp":"2024-03-11T21:32:24.657490+0000","objects_scrubbed":5,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:33:25.260243+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,2],"acting":[4,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.10","version":"0'0","reported_seq":239,"reported_epoch":174,"state":"active+clean","last_fresh":"2024-03-11T21:32:41.922895+0000","last_change":"2024-03-11T21:31:22.541299+0000","last_active":"2024-03-11T21:32:41.922895+0000","last_peered":"2024-03-11T21:32:41.922895+0000","last_clean":"2024-03-11T21:32:41.922895+0000","last_became_active":"2024-03-11T21:25:44.591189+0000","last_became_peered":"2024-03-11T21:25:44.591189+0000","last_unstale":"2024-03-11T21:32:41.922895+0000","last_undegraded":"2024-03-11T21:32:41.922895+0000","last_fullsized":"2024-03-11T21:32:41.922895+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:31:22.541178+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:27:25.417450+0000","last_clean_scrub_stamp":"2024-03-11T21:31:22.541178+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":18,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,1],"acting":[0,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":242,"reported_epoch":174,"state":"active+clean","last_fresh":"2024-03-11T21:32:41.929773+0000","last_change":"2024-03-11T21:31:15.368570+0000","last_active":"2024-03-11T21:32:41.929773+0000","last_peered":"2024-03-11T21:32:41.929773+0000","last_clean":"2024-03-11T21:32:41.929773+0000","last_became_active":"2024-03-11T21:25:42.576425+0000","last_became_peered":"2024-03-11T21:25:42.576425+0000","last_unstale":"2024-03-11T21:32:41.929773+0000","last_undegraded":"2024-03-11T21:32:41.929773+0000","last_fullsized":"2024-03-11T21:32:41.929773+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:31:15.367898+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:31:15.367898+0000","last_clean_scrub_stamp":"2024-03-11T21:31:15.367898+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.14","version":"0'0","reported_seq":268,"reported_epoch":174,"state":"active+clean","last_fresh":"2024-03-11T21:32:41.924631+0000","last_change":"2024-03-11T21:32:22.917604+0000","last_active":"2024-03-11T21:32:41.924631+0000","last_peered":"2024-03-11T21:32:41.924631+0000","last_clean":"2024-03-11T21:32:41.924631+0000","last_became_active":"2024-03-11T21:25:38.541852+0000","last_became_peered":"2024-03-11T21:25:38.541852+0000","last_unstale":"2024-03-11T21:32:41.924631+0000","last_undegraded":"2024-03-11T21:32:41.924631+0000","last_fullsized":"2024-03-11T21:32:41.924631+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:32:22.917423+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:37.512018+0000","last_clean_scrub_stamp":"2024-03-11T21:32:22.917423+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:33:44.196295+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,5],"acting":[6,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.12","version":"74'9","reported_seq":272,"reported_epoch":174,"state":"active+clean","last_fresh":"2024-03-11T21:32:41.930466+0000","last_change":"2024-03-11T21:31:59.579237+0000","last_active":"2024-03-11T21:32:41.930466+0000","last_peered":"2024-03-11T21:32:41.930466+0000","last_clean":"2024-03-11T21:32:41.930466+0000","last_became_active":"2024-03-11T21:25:40.547082+0000","last_became_peered":"2024-03-11T21:25:40.547082+0000","last_unstale":"2024-03-11T21:32:41.930466+0000","last_undegraded":"2024-03-11T21:32:41.930466+0000","last_fullsized":"2024-03-11T21:32:41.930466+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'9","last_scrub_stamp":"2024-03-11T21:31:59.579142+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:39.525554+0000","last_clean_scrub_stamp":"2024-03-11T21:31:59.579142+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:33:15.464959+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":239,"reported_epoch":174,"state":"active+clean","last_fresh":"2024-03-11T21:32:41.914462+0000","last_change":"2024-03-11T21:31:25.682563+0000","last_active":"2024-03-11T21:32:41.914462+0000","last_peered":"2024-03-11T21:32:41.914462+0000","last_clean":"2024-03-11T21:32:41.914462+0000","last_became_active":"2024-03-11T21:25:44.589586+0000","last_became_peered":"2024-03-11T21:25:44.589586+0000","last_unstale":"2024-03-11T21:32:41.914462+0000","last_undegraded":"2024-03-11T21:32:41.914462+0000","last_fullsized":"2024-03-11T21:32:41.914462+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:31:25.682446+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:27:09.801138+0000","last_clean_scrub_stamp":"2024-03-11T21:31:25.682446+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,5],"acting":[4,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.14","version":"154'9","reported_seq":345,"reported_epoch":174,"state":"active+clean","last_fresh":"2024-03-11T21:32:41.928378+0000","last_change":"2024-03-11T21:32:03.467267+0000","last_active":"2024-03-11T21:32:41.928378+0000","last_peered":"2024-03-11T21:32:41.928378+0000","last_clean":"2024-03-11T21:32:41.928378+0000","last_became_active":"2024-03-11T21:31:48.230299+0000","last_became_peered":"2024-03-11T21:31:48.230299+0000","last_unstale":"2024-03-11T21:32:41.928378+0000","last_undegraded":"2024-03-11T21:32:41.928378+0000","last_fullsized":"2024-03-11T21:32:41.928378+0000","mapping_epoch":153,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":154,"parent":"0.0","parent_split_bits":0,"last_scrub":"154'9","last_scrub_stamp":"2024-03-11T21:32:03.467138+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:41.544550+0000","last_clean_scrub_stamp":"2024-03-11T21:32:03.467138+0000","objects_scrubbed":1,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:33:14.044164+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,1],"acting":[3,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.13","version":"0'0","reported_seq":252,"reported_epoch":174,"state":"active+clean","last_fresh":"2024-03-11T21:32:41.918803+0000","last_change":"2024-03-11T21:31:18.506150+0000","last_active":"2024-03-11T21:32:41.918803+0000","last_peered":"2024-03-11T21:32:41.918803+0000","last_clean":"2024-03-11T21:32:41.918803+0000","last_became_active":"2024-03-11T21:25:38.539467+0000","last_became_peered":"2024-03-11T21:25:38.539467+0000","last_unstale":"2024-03-11T21:32:41.918803+0000","last_undegraded":"2024-03-11T21:32:41.918803+0000","last_fullsized":"2024-03-11T21:32:41.918803+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:31:18.499630+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:37.512018+0000","last_clean_scrub_stamp":"2024-03-11T21:31:18.499630+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:32:44.469282+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.15","version":"94'13","reported_seq":290,"reported_epoch":174,"state":"active+clean","last_fresh":"2024-03-11T21:32:41.927599+0000","last_change":"2024-03-11T21:32:10.610662+0000","last_active":"2024-03-11T21:32:41.927599+0000","last_peered":"2024-03-11T21:32:41.927599+0000","last_clean":"2024-03-11T21:32:41.927599+0000","last_became_active":"2024-03-11T21:31:48.230445+0000","last_became_peered":"2024-03-11T21:31:48.230445+0000","last_unstale":"2024-03-11T21:32:41.927599+0000","last_undegraded":"2024-03-11T21:32:41.927599+0000","last_fullsized":"2024-03-11T21:32:41.927599+0000","mapping_epoch":153,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":154,"parent":"0.0","parent_split_bits":0,"last_scrub":"94'13","last_scrub_stamp":"2024-03-11T21:32:10.610489+0000","last_deep_scrub":"94'13","last_deep_scrub_stamp":"2024-03-11T21:29:41.853158+0000","last_clean_scrub_stamp":"2024-03-11T21:32:10.610489+0000","objects_scrubbed":4,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:33:29.872424+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,6],"acting":[5,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.16","version":"0'0","reported_seq":253,"reported_epoch":174,"state":"active+clean","last_fresh":"2024-03-11T21:32:41.919631+0000","last_change":"2024-03-11T21:32:23.607583+0000","last_active":"2024-03-11T21:32:41.919631+0000","last_peered":"2024-03-11T21:32:41.919631+0000","last_clean":"2024-03-11T21:32:41.919631+0000","last_became_active":"2024-03-11T21:31:48.231031+0000","last_became_peered":"2024-03-11T21:31:48.231031+0000","last_unstale":"2024-03-11T21:32:41.919631+0000","last_undegraded":"2024-03-11T21:32:41.919631+0000","last_fullsized":"2024-03-11T21:32:41.919631+0000","mapping_epoch":153,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":154,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:32:23.607460+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:43.559452+0000","last_clean_scrub_stamp":"2024-03-11T21:32:23.607460+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:33:24.611914+0000","scrub_duration":35,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,4],"acting":[0,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.15","version":"98'9","reported_seq":336,"reported_epoch":174,"state":"active+clean","last_fresh":"2024-03-11T21:32:41.927777+0000","last_change":"2024-03-11T21:31:20.625060+0000","last_active":"2024-03-11T21:32:41.927777+0000","last_peered":"2024-03-11T21:32:41.927777+0000","last_clean":"2024-03-11T21:32:41.927777+0000","last_became_active":"2024-03-11T21:25:42.572555+0000","last_became_peered":"2024-03-11T21:25:42.572555+0000","last_unstale":"2024-03-11T21:32:41.927777+0000","last_undegraded":"2024-03-11T21:32:41.927777+0000","last_fullsized":"2024-03-11T21:32:41.927777+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"98'9","last_scrub_stamp":"2024-03-11T21:31:20.624923+0000","last_deep_scrub":"98'9","last_deep_scrub_stamp":"2024-03-11T21:31:20.624923+0000","last_clean_scrub_stamp":"2024-03-11T21:31:20.624923+0000","objects_scrubbed":1,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:32:45.365157+0000","scrub_duration":25,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.12","version":"0'0","reported_seq":265,"reported_epoch":174,"state":"active+clean","last_fresh":"2024-03-11T21:32:41.927262+0000","last_change":"2024-03-11T21:32:14.555259+0000","last_active":"2024-03-11T21:32:41.927262+0000","last_peered":"2024-03-11T21:32:41.927262+0000","last_clean":"2024-03-11T21:32:41.927262+0000","last_became_active":"2024-03-11T21:31:48.223495+0000","last_became_peered":"2024-03-11T21:31:48.223495+0000","last_unstale":"2024-03-11T21:32:41.927262+0000","last_undegraded":"2024-03-11T21:32:41.927262+0000","last_fullsized":"2024-03-11T21:32:41.927262+0000","mapping_epoch":153,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":154,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:32:14.555187+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:37.512018+0000","last_clean_scrub_stamp":"2024-03-11T21:32:14.555187+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:33:30.496365+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,0],"acting":[5,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.14","version":"74'10","reported_seq":279,"reported_epoch":174,"state":"active+clean","last_fresh":"2024-03-11T21:32:41.928191+0000","last_change":"2024-03-11T21:32:13.490370+0000","last_active":"2024-03-11T21:32:41.928191+0000","last_peered":"2024-03-11T21:32:41.928191+0000","last_clean":"2024-03-11T21:32:41.928191+0000","last_became_active":"2024-03-11T21:31:48.229849+0000","last_became_peered":"2024-03-11T21:31:48.229849+0000","last_unstale":"2024-03-11T21:32:41.928191+0000","last_undegraded":"2024-03-11T21:32:41.928191+0000","last_fullsized":"2024-03-11T21:32:41.928191+0000","mapping_epoch":153,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":154,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'10","last_scrub_stamp":"2024-03-11T21:32:13.490198+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:39.525554+0000","last_clean_scrub_stamp":"2024-03-11T21:32:13.490198+0000","objects_scrubbed":5,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:33:19.120049+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.15","version":"0'0","reported_seq":60,"reported_epoch":174,"state":"active+clean","last_fresh":"2024-03-11T21:32:41.925932+0000","last_change":"2024-03-11T21:32:06.780189+0000","last_active":"2024-03-11T21:32:41.925932+0000","last_peered":"2024-03-11T21:32:41.925932+0000","last_clean":"2024-03-11T21:32:41.925932+0000","last_became_active":"2024-03-11T21:31:48.223661+0000","last_became_peered":"2024-03-11T21:31:48.223661+0000","last_unstale":"2024-03-11T21:32:41.925932+0000","last_undegraded":"2024-03-11T21:32:41.925932+0000","last_fullsized":"2024-03-11T21:32:41.925932+0000","mapping_epoch":153,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":154,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:32:06.779964+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:43.559452+0000","last_clean_scrub_stamp":"2024-03-11T21:32:06.779964+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:33:12.866979+0000","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,5],"acting":[6,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":242,"reported_epoch":174,"state":"active+clean","last_fresh":"2024-03-11T21:32:41.927246+0000","last_change":"2024-03-11T21:31:14.608055+0000","last_active":"2024-03-11T21:32:41.927246+0000","last_peered":"2024-03-11T21:32:41.927246+0000","last_clean":"2024-03-11T21:32:41.927246+0000","last_became_active":"2024-03-11T21:25:42.577351+0000","last_became_peered":"2024-03-11T21:25:42.577351+0000","last_unstale":"2024-03-11T21:32:41.927246+0000","last_undegraded":"2024-03-11T21:32:41.927246+0000","last_fullsized":"2024-03-11T21:32:41.927246+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:31:14.607971+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:41.544550+0000","last_clean_scrub_stamp":"2024-03-11T21:31:14.607971+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:32:45.331100+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,1],"acting":[5,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.11","version":"0'0","reported_seq":260,"reported_epoch":174,"state":"active+clean","last_fresh":"2024-03-11T21:32:41.924607+0000","last_change":"2024-03-11T21:32:09.785387+0000","last_active":"2024-03-11T21:32:41.924607+0000","last_peered":"2024-03-11T21:32:41.924607+0000","last_clean":"2024-03-11T21:32:41.924607+0000","last_became_active":"2024-03-11T21:25:38.537415+0000","last_became_peered":"2024-03-11T21:25:38.537415+0000","last_unstale":"2024-03-11T21:32:41.924607+0000","last_undegraded":"2024-03-11T21:32:41.924607+0000","last_fullsized":"2024-03-11T21:32:41.924607+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:32:09.785314+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:37.512018+0000","last_clean_scrub_stamp":"2024-03-11T21:32:09.785314+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:33:28.459195+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.17","version":"97'9","reported_seq":286,"reported_epoch":174,"state":"active+clean","last_fresh":"2024-03-11T21:32:41.926377+0000","last_change":"2024-03-11T21:32:08.495485+0000","last_active":"2024-03-11T21:32:41.926377+0000","last_peered":"2024-03-11T21:32:41.926377+0000","last_clean":"2024-03-11T21:32:41.926377+0000","last_became_active":"2024-03-11T21:31:48.224112+0000","last_became_peered":"2024-03-11T21:31:48.224112+0000","last_unstale":"2024-03-11T21:32:41.926377+0000","last_undegraded":"2024-03-11T21:32:41.926377+0000","last_fullsized":"2024-03-11T21:32:41.926377+0000","mapping_epoch":153,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":154,"parent":"0.0","parent_split_bits":0,"last_scrub":"97'9","last_scrub_stamp":"2024-03-11T21:32:08.495352+0000","last_deep_scrub":"97'9","last_deep_scrub_stamp":"2024-03-11T21:27:25.803132+0000","last_clean_scrub_stamp":"2024-03-11T21:32:08.495352+0000","objects_scrubbed":3,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:33:33.794394+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":9,"num_read_kb":6,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,2],"acting":[3,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.14","version":"0'0","reported_seq":244,"reported_epoch":174,"state":"active+clean","last_fresh":"2024-03-11T21:32:41.938535+0000","last_change":"2024-03-11T21:31:48.218674+0000","last_active":"2024-03-11T21:32:41.938535+0000","last_peered":"2024-03-11T21:32:41.938535+0000","last_clean":"2024-03-11T21:32:41.938535+0000","last_became_active":"2024-03-11T21:31:48.218375+0000","last_became_peered":"2024-03-11T21:31:48.218375+0000","last_unstale":"2024-03-11T21:32:41.938535+0000","last_undegraded":"2024-03-11T21:32:41.938535+0000","last_fullsized":"2024-03-11T21:32:41.938535+0000","mapping_epoch":153,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":154,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:31:15.409641+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:43.559452+0000","last_clean_scrub_stamp":"2024-03-11T21:31:15.409641+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,0],"acting":[2,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":248,"reported_epoch":174,"state":"active+clean","last_fresh":"2024-03-11T21:32:41.926480+0000","last_change":"2024-03-11T21:31:48.230708+0000","last_active":"2024-03-11T21:32:41.926480+0000","last_peered":"2024-03-11T21:32:41.926480+0000","last_clean":"2024-03-11T21:32:41.926480+0000","last_became_active":"2024-03-11T21:31:48.230253+0000","last_became_peered":"2024-03-11T21:31:48.230253+0000","last_unstale":"2024-03-11T21:32:41.926480+0000","last_undegraded":"2024-03-11T21:32:41.926480+0000","last_fullsized":"2024-03-11T21:32:41.926480+0000","mapping_epoch":153,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":154,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:31:08.284511+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:41.544550+0000","last_clean_scrub_stamp":"2024-03-11T21:31:08.284511+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0],"acting":[3,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.10","version":"0'0","reported_seq":251,"reported_epoch":174,"state":"active+clean","last_fresh":"2024-03-11T21:32:41.937497+0000","last_change":"2024-03-11T21:31:03.260526+0000","last_active":"2024-03-11T21:32:41.937497+0000","last_peered":"2024-03-11T21:32:41.937497+0000","last_clean":"2024-03-11T21:32:41.937497+0000","last_became_active":"2024-03-11T21:25:38.533138+0000","last_became_peered":"2024-03-11T21:25:38.533138+0000","last_unstale":"2024-03-11T21:32:41.937497+0000","last_undegraded":"2024-03-11T21:32:41.937497+0000","last_fullsized":"2024-03-11T21:32:41.937497+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:31:03.260408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:37.512018+0000","last_clean_scrub_stamp":"2024-03-11T21:31:03.260408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":29,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,0],"acting":[2,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.16","version":"74'9","reported_seq":287,"reported_epoch":174,"state":"active+clean","last_fresh":"2024-03-11T21:32:41.922232+0000","last_change":"2024-03-11T21:32:14.414768+0000","last_active":"2024-03-11T21:32:41.922232+0000","last_peered":"2024-03-11T21:32:41.922232+0000","last_clean":"2024-03-11T21:32:41.922232+0000","last_became_active":"2024-03-11T21:31:48.231280+0000","last_became_peered":"2024-03-11T21:31:48.231280+0000","last_unstale":"2024-03-11T21:32:41.922232+0000","last_undegraded":"2024-03-11T21:32:41.922232+0000","last_fullsized":"2024-03-11T21:32:41.922232+0000","mapping_epoch":153,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":154,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'9","last_scrub_stamp":"2024-03-11T21:32:14.414508+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:39.525554+0000","last_clean_scrub_stamp":"2024-03-11T21:32:14.414508+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:33:43.663024+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.5","version":"174'4466","reported_seq":4511,"reported_epoch":174,"state":"active+clean","last_fresh":"2024-03-11T21:32:42.142946+0000","last_change":"2024-03-11T21:31:52.239825+0000","last_active":"2024-03-11T21:32:42.142946+0000","last_peered":"2024-03-11T21:32:42.142946+0000","last_clean":"2024-03-11T21:32:42.142946+0000","last_became_active":"2024-03-11T21:31:52.239480+0000","last_became_peered":"2024-03-11T21:31:52.239480+0000","last_unstale":"2024-03-11T21:32:42.142946+0000","last_undegraded":"2024-03-11T21:32:42.142946+0000","last_fullsized":"2024-03-11T21:32:42.142946+0000","mapping_epoch":157,"log_start":"0'0","ondisk_log_start":"0'0","created":157,"last_epoch_clean":158,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:31:51.219688+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:31:51.219688+0000","last_clean_scrub_stamp":"2024-03-11T21:31:51.219688+0000","objects_scrubbed":0,"log_size":4466,"log_dups_size":0,"ondisk_log_size":4466,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:33:17.826081+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":292683776,"num_objects":4466,"num_object_clones":0,"num_object_copies":13398,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4466,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4466,"num_write_kb":285824,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,0],"acting":[6,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.b","version":"0'0","reported_seq":245,"reported_epoch":174,"state":"active+clean","last_fresh":"2024-03-11T21:32:41.929846+0000","last_change":"2024-03-11T21:32:22.600308+0000","last_active":"2024-03-11T21:32:41.929846+0000","last_peered":"2024-03-11T21:32:41.929846+0000","last_clean":"2024-03-11T21:32:41.929846+0000","last_became_active":"2024-03-11T21:31:48.238287+0000","last_became_peered":"2024-03-11T21:31:48.238287+0000","last_unstale":"2024-03-11T21:32:41.929846+0000","last_undegraded":"2024-03-11T21:32:41.929846+0000","last_fullsized":"2024-03-11T21:32:41.929846+0000","mapping_epoch":153,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":154,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:32:22.598150+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:43.559452+0000","last_clean_scrub_stamp":"2024-03-11T21:32:22.598150+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:33:47.277891+0000","scrub_duration":57,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6],"acting":[3,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.f","version":"67'2","reported_seq":264,"reported_epoch":174,"state":"active+clean","last_fresh":"2024-03-11T21:32:41.914235+0000","last_change":"2024-03-11T21:31:03.533706+0000","last_active":"2024-03-11T21:32:41.914235+0000","last_peered":"2024-03-11T21:32:41.914235+0000","last_clean":"2024-03-11T21:32:41.914235+0000","last_became_active":"2024-03-11T21:25:38.537711+0000","last_became_peered":"2024-03-11T21:25:38.537711+0000","last_unstale":"2024-03-11T21:32:41.914235+0000","last_undegraded":"2024-03-11T21:32:41.914235+0000","last_fullsized":"2024-03-11T21:32:41.914235+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'2","last_scrub_stamp":"2024-03-11T21:31:03.533547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:37.512018+0000","last_clean_scrub_stamp":"2024-03-11T21:31:03.533547+0000","objects_scrubbed":2,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":92,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":10,"num_read_kb":10,"num_write":4,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,7],"acting":[4,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.9","version":"146'17","reported_seq":276,"reported_epoch":174,"state":"active+clean","last_fresh":"2024-03-11T21:32:41.913702+0000","last_change":"2024-03-11T21:31:16.615182+0000","last_active":"2024-03-11T21:32:41.913702+0000","last_peered":"2024-03-11T21:32:41.913702+0000","last_clean":"2024-03-11T21:32:41.913702+0000","last_became_active":"2024-03-11T21:25:40.552267+0000","last_became_peered":"2024-03-11T21:25:40.552267+0000","last_unstale":"2024-03-11T21:32:41.913702+0000","last_undegraded":"2024-03-11T21:32:41.913702+0000","last_fullsized":"2024-03-11T21:32:41.913702+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"146'17","last_scrub_stamp":"2024-03-11T21:31:16.615010+0000","last_deep_scrub":"146'17","last_deep_scrub_stamp":"2024-03-11T21:31:16.615010+0000","last_clean_scrub_stamp":"2024-03-11T21:31:16.615010+0000","objects_scrubbed":5,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:32:42.700216+0000","scrub_duration":27,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":25,"num_read_kb":16,"num_write":14,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,3],"acting":[4,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":243,"reported_epoch":174,"state":"active+clean","last_fresh":"2024-03-11T21:32:41.937422+0000","last_change":"2024-03-11T21:31:12.434593+0000","last_active":"2024-03-11T21:32:41.937422+0000","last_peered":"2024-03-11T21:32:41.937422+0000","last_clean":"2024-03-11T21:32:41.937422+0000","last_became_active":"2024-03-11T21:25:42.573987+0000","last_became_peered":"2024-03-11T21:25:42.573987+0000","last_unstale":"2024-03-11T21:32:41.937422+0000","last_undegraded":"2024-03-11T21:32:41.937422+0000","last_fullsized":"2024-03-11T21:32:41.937422+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:31:12.434479+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:41.544550+0000","last_clean_scrub_stamp":"2024-03-11T21:31:12.434479+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":17,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"8.e","version":"174'4649","reported_seq":4694,"reported_epoch":174,"state":"active+clean","last_fresh":"2024-03-11T21:32:42.833165+0000","last_change":"2024-03-11T21:31:52.273246+0000","last_active":"2024-03-11T21:32:42.833165+0000","last_peered":"2024-03-11T21:32:42.833165+0000","last_clean":"2024-03-11T21:32:42.833165+0000","last_became_active":"2024-03-11T21:31:52.272382+0000","last_became_peered":"2024-03-11T21:31:52.272382+0000","last_unstale":"2024-03-11T21:32:42.833165+0000","last_undegraded":"2024-03-11T21:32:42.833165+0000","last_fullsized":"2024-03-11T21:32:42.833165+0000","mapping_epoch":157,"log_start":"0'0","ondisk_log_start":"0'0","created":157,"last_epoch_clean":158,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:31:51.219688+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:31:51.219688+0000","last_clean_scrub_stamp":"2024-03-11T21:31:51.219688+0000","objects_scrubbed":0,"log_size":4649,"log_dups_size":0,"ondisk_log_size":4649,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:33:18.833285+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":304676864,"num_objects":4649,"num_object_clones":0,"num_object_copies":13947,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4649,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4649,"num_write_kb":297536,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,4],"acting":[0,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.0","version":"0'0","reported_seq":240,"reported_epoch":174,"state":"active+clean","last_fresh":"2024-03-11T21:32:41.922995+0000","last_change":"2024-03-11T21:32:03.422921+0000","last_active":"2024-03-11T21:32:41.922995+0000","last_peered":"2024-03-11T21:32:41.922995+0000","last_clean":"2024-03-11T21:32:41.922995+0000","last_became_active":"2024-03-11T21:25:44.584565+0000","last_became_peered":"2024-03-11T21:25:44.584565+0000","last_unstale":"2024-03-11T21:32:41.922995+0000","last_undegraded":"2024-03-11T21:32:41.922995+0000","last_fullsized":"2024-03-11T21:32:41.922995+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:32:03.422808+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:29:39.498784+0000","last_clean_scrub_stamp":"2024-03-11T21:32:03.422808+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:33:15.105649+0000","scrub_duration":31,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":266,"reported_epoch":174,"state":"active+clean","last_fresh":"2024-03-11T21:32:41.926798+0000","last_change":"2024-03-11T21:32:04.509633+0000","last_active":"2024-03-11T21:32:41.926798+0000","last_peered":"2024-03-11T21:32:41.926798+0000","last_clean":"2024-03-11T21:32:41.926798+0000","last_became_active":"2024-03-11T21:31:48.224184+0000","last_became_peered":"2024-03-11T21:31:48.224184+0000","last_unstale":"2024-03-11T21:32:41.926798+0000","last_undegraded":"2024-03-11T21:32:41.926798+0000","last_fullsized":"2024-03-11T21:32:41.926798+0000","mapping_epoch":153,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":154,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:32:04.509514+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:37.512018+0000","last_clean_scrub_stamp":"2024-03-11T21:32:04.509514+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:33:08.577665+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.2","version":"94'15","reported_seq":276,"reported_epoch":174,"state":"active+clean","last_fresh":"2024-03-11T21:32:41.930514+0000","last_change":"2024-03-11T21:31:08.956264+0000","last_active":"2024-03-11T21:32:41.930514+0000","last_peered":"2024-03-11T21:32:41.930514+0000","last_clean":"2024-03-11T21:32:41.930514+0000","last_became_active":"2024-03-11T21:25:40.548833+0000","last_became_peered":"2024-03-11T21:25:40.548833+0000","last_unstale":"2024-03-11T21:32:41.930514+0000","last_undegraded":"2024-03-11T21:32:41.930514+0000","last_fullsized":"2024-03-11T21:32:41.930514+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"94'15","last_scrub_stamp":"2024-03-11T21:31:08.955971+0000","last_deep_scrub":"94'15","last_deep_scrub_stamp":"2024-03-11T21:26:49.993060+0000","last_clean_scrub_stamp":"2024-03-11T21:31:08.955971+0000","objects_scrubbed":5,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:32:43.569485+0000","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.3","version":"74'8","reported_seq":337,"reported_epoch":174,"state":"active+clean","last_fresh":"2024-03-11T21:32:41.918936+0000","last_change":"2024-03-11T21:31:15.465617+0000","last_active":"2024-03-11T21:32:41.918936+0000","last_peered":"2024-03-11T21:32:41.918936+0000","last_clean":"2024-03-11T21:32:41.918936+0000","last_became_active":"2024-03-11T21:25:42.578161+0000","last_became_peered":"2024-03-11T21:25:42.578161+0000","last_unstale":"2024-03-11T21:32:41.918936+0000","last_undegraded":"2024-03-11T21:32:41.918936+0000","last_fullsized":"2024-03-11T21:32:41.918936+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'8","last_scrub_stamp":"2024-03-11T21:31:15.465460+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:41.544550+0000","last_clean_scrub_stamp":"2024-03-11T21:31:15.465460+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:32:43.379445+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,5],"acting":[0,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.8","version":"174'4562","reported_seq":4606,"reported_epoch":174,"state":"active+clean","last_fresh":"2024-03-11T21:32:42.040342+0000","last_change":"2024-03-11T21:31:52.240065+0000","last_active":"2024-03-11T21:32:42.040342+0000","last_peered":"2024-03-11T21:32:42.040342+0000","last_clean":"2024-03-11T21:32:42.040342+0000","last_became_active":"2024-03-11T21:31:52.239695+0000","last_became_peered":"2024-03-11T21:31:52.239695+0000","last_unstale":"2024-03-11T21:32:42.040342+0000","last_undegraded":"2024-03-11T21:32:42.040342+0000","last_fullsized":"2024-03-11T21:32:42.040342+0000","mapping_epoch":157,"log_start":"0'0","ondisk_log_start":"0'0","created":157,"last_epoch_clean":158,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:31:51.219688+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:31:51.219688+0000","last_clean_scrub_stamp":"2024-03-11T21:31:51.219688+0000","objects_scrubbed":0,"log_size":4562,"log_dups_size":0,"ondisk_log_size":4562,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:32:52.359698+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":298975232,"num_objects":4562,"num_object_clones":0,"num_object_copies":13686,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4562,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4562,"num_write_kb":291968,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,0],"acting":[5,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":245,"reported_epoch":174,"state":"active+clean","last_fresh":"2024-03-11T21:32:41.926792+0000","last_change":"2024-03-11T21:31:48.236305+0000","last_active":"2024-03-11T21:32:41.926792+0000","last_peered":"2024-03-11T21:32:41.926792+0000","last_clean":"2024-03-11T21:32:41.926792+0000","last_became_active":"2024-03-11T21:31:48.234360+0000","last_became_peered":"2024-03-11T21:31:48.234360+0000","last_unstale":"2024-03-11T21:32:41.926792+0000","last_undegraded":"2024-03-11T21:32:41.926792+0000","last_fullsized":"2024-03-11T21:32:41.926792+0000","mapping_epoch":153,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":154,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:31:01.241180+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:28:30.563712+0000","last_clean_scrub_stamp":"2024-03-11T21:31:01.241180+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,6],"acting":[3,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":259,"reported_epoch":174,"state":"active+clean","last_fresh":"2024-03-11T21:32:41.927306+0000","last_change":"2024-03-11T21:31:16.608343+0000","last_active":"2024-03-11T21:32:41.927306+0000","last_peered":"2024-03-11T21:32:41.927306+0000","last_clean":"2024-03-11T21:32:41.927306+0000","last_became_active":"2024-03-11T21:25:38.538540+0000","last_became_peered":"2024-03-11T21:25:38.538540+0000","last_unstale":"2024-03-11T21:32:41.927306+0000","last_undegraded":"2024-03-11T21:32:41.927306+0000","last_fullsized":"2024-03-11T21:32:41.927306+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:31:16.608266+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:37.512018+0000","last_clean_scrub_stamp":"2024-03-11T21:31:16.608266+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:32:45.331174+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.4","version":"74'28","reported_seq":381,"reported_epoch":174,"state":"active+clean","last_fresh":"2024-03-11T21:32:41.926706+0000","last_change":"2024-03-11T21:30:24.750195+0000","last_active":"2024-03-11T21:32:41.926706+0000","last_peered":"2024-03-11T21:32:41.926706+0000","last_clean":"2024-03-11T21:32:41.926706+0000","last_became_active":"2024-03-11T21:25:40.552902+0000","last_became_peered":"2024-03-11T21:25:40.552902+0000","last_unstale":"2024-03-11T21:32:41.926706+0000","last_undegraded":"2024-03-11T21:32:41.926706+0000","last_fullsized":"2024-03-11T21:32:41.926706+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'28","last_scrub_stamp":"2024-03-11T21:30:24.750058+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:39.525554+0000","last_clean_scrub_stamp":"2024-03-11T21:30:24.750058+0000","objects_scrubbed":10,"log_size":28,"log_dups_size":0,"ondisk_log_size":28,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:32:43.569352+0000","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":358,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":48,"num_read_kb":33,"num_write":26,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3],"acting":[1,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":244,"reported_epoch":174,"state":"active+clean","last_fresh":"2024-03-11T21:32:41.921731+0000","last_change":"2024-03-11T21:31:57.398694+0000","last_active":"2024-03-11T21:32:41.921731+0000","last_peered":"2024-03-11T21:32:41.921731+0000","last_clean":"2024-03-11T21:32:41.921731+0000","last_became_active":"2024-03-11T21:25:42.572673+0000","last_became_peered":"2024-03-11T21:25:42.572673+0000","last_unstale":"2024-03-11T21:32:41.921731+0000","last_undegraded":"2024-03-11T21:32:41.921731+0000","last_fullsized":"2024-03-11T21:32:41.921731+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:31:57.398611+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:41.544550+0000","last_clean_scrub_stamp":"2024-03-11T21:31:57.398611+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:33:06.306030+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.b","version":"174'4567","reported_seq":4611,"reported_epoch":174,"state":"active+clean","last_fresh":"2024-03-11T21:32:42.530476+0000","last_change":"2024-03-11T21:31:52.241413+0000","last_active":"2024-03-11T21:32:42.530476+0000","last_peered":"2024-03-11T21:32:42.530476+0000","last_clean":"2024-03-11T21:32:42.530476+0000","last_became_active":"2024-03-11T21:31:52.241089+0000","last_became_peered":"2024-03-11T21:31:52.241089+0000","last_unstale":"2024-03-11T21:32:42.530476+0000","last_undegraded":"2024-03-11T21:32:42.530476+0000","last_fullsized":"2024-03-11T21:32:42.530476+0000","mapping_epoch":157,"log_start":"0'0","ondisk_log_start":"0'0","created":157,"last_epoch_clean":158,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:31:51.219688+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:31:51.219688+0000","last_clean_scrub_stamp":"2024-03-11T21:31:51.219688+0000","objects_scrubbed":0,"log_size":4567,"log_dups_size":0,"ondisk_log_size":4567,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:33:00.420299+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":299302912,"num_objects":4567,"num_object_clones":0,"num_object_copies":13701,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4567,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4567,"num_write_kb":292288,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,1],"acting":[3,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.5","version":"0'0","reported_seq":60,"reported_epoch":174,"state":"active+clean","last_fresh":"2024-03-11T21:32:41.925807+0000","last_change":"2024-03-11T21:32:10.837792+0000","last_active":"2024-03-11T21:32:41.925807+0000","last_peered":"2024-03-11T21:32:41.925807+0000","last_clean":"2024-03-11T21:32:41.925807+0000","last_became_active":"2024-03-11T21:31:48.234440+0000","last_became_peered":"2024-03-11T21:31:48.234440+0000","last_unstale":"2024-03-11T21:32:41.925807+0000","last_undegraded":"2024-03-11T21:32:41.925807+0000","last_fullsized":"2024-03-11T21:32:41.925807+0000","mapping_epoch":153,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":154,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:32:10.837701+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:43.559452+0000","last_clean_scrub_stamp":"2024-03-11T21:32:10.837701+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:33:28.713079+0000","scrub_duration":16,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,2],"acting":[6,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.0","version":"164'6","reported_seq":688,"reported_epoch":174,"state":"active+clean","last_fresh":"2024-03-11T21:32:41.928280+0000","last_change":"2024-03-11T21:32:17.597798+0000","last_active":"2024-03-11T21:32:41.928280+0000","last_peered":"2024-03-11T21:32:41.928280+0000","last_clean":"2024-03-11T21:32:41.928280+0000","last_became_active":"2024-03-11T21:25:38.536890+0000","last_became_peered":"2024-03-11T21:25:38.536890+0000","last_unstale":"2024-03-11T21:32:41.928280+0000","last_undegraded":"2024-03-11T21:32:41.928280+0000","last_fullsized":"2024-03-11T21:32:41.928280+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"164'6","last_scrub_stamp":"2024-03-11T21:32:17.597666+0000","last_deep_scrub":"164'6","last_deep_scrub_stamp":"2024-03-11T21:32:17.597666+0000","last_clean_scrub_stamp":"2024-03-11T21:32:17.597666+0000","objects_scrubbed":1,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:33:38.033052+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0.000753559,"stat_sum":{"num_bytes":389,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":417,"num_read_kb":412,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,6],"acting":[1,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":259,"reported_epoch":174,"state":"active+clean","last_fresh":"2024-03-11T21:32:41.938849+0000","last_change":"2024-03-11T21:31:21.359528+0000","last_active":"2024-03-11T21:32:41.938849+0000","last_peered":"2024-03-11T21:32:41.938849+0000","last_clean":"2024-03-11T21:32:41.938849+0000","last_became_active":"2024-03-11T21:25:38.532984+0000","last_became_peered":"2024-03-11T21:25:38.532984+0000","last_unstale":"2024-03-11T21:32:41.938849+0000","last_undegraded":"2024-03-11T21:32:41.938849+0000","last_fullsized":"2024-03-11T21:32:41.938849+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:31:21.359298+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:26:48.997454+0000","last_clean_scrub_stamp":"2024-03-11T21:31:21.359298+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:32:43.745058+0000","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.7","version":"97'18","reported_seq":291,"reported_epoch":174,"state":"active+clean","last_fresh":"2024-03-11T21:32:41.940428+0000","last_change":"2024-03-11T21:32:14.594198+0000","last_active":"2024-03-11T21:32:41.940428+0000","last_peered":"2024-03-11T21:32:41.940428+0000","last_clean":"2024-03-11T21:32:41.940428+0000","last_became_active":"2024-03-11T21:25:40.541769+0000","last_became_peered":"2024-03-11T21:25:40.541769+0000","last_unstale":"2024-03-11T21:32:41.940428+0000","last_undegraded":"2024-03-11T21:32:41.940428+0000","last_fullsized":"2024-03-11T21:32:41.940428+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"97'18","last_scrub_stamp":"2024-03-11T21:32:14.593767+0000","last_deep_scrub":"97'18","last_deep_scrub_stamp":"2024-03-11T21:27:19.239404+0000","last_clean_scrub_stamp":"2024-03-11T21:32:14.593767+0000","objects_scrubbed":5,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:33:21.350171+0000","scrub_duration":18,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":19,"num_write":16,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":249,"reported_epoch":174,"state":"active+clean","last_fresh":"2024-03-11T21:32:41.941053+0000","last_change":"2024-03-11T21:31:48.236854+0000","last_active":"2024-03-11T21:32:41.941053+0000","last_peered":"2024-03-11T21:32:41.941053+0000","last_clean":"2024-03-11T21:32:41.941053+0000","last_became_active":"2024-03-11T21:31:48.236520+0000","last_became_peered":"2024-03-11T21:31:48.236520+0000","last_unstale":"2024-03-11T21:32:41.941053+0000","last_undegraded":"2024-03-11T21:32:41.941053+0000","last_fullsized":"2024-03-11T21:32:41.941053+0000","mapping_epoch":153,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":154,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:31:06.279751+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:41.544550+0000","last_clean_scrub_stamp":"2024-03-11T21:31:06.279751+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,4],"acting":[2,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"8.9","version":"174'4527","reported_seq":4572,"reported_epoch":174,"state":"active+clean","last_fresh":"2024-03-11T21:32:42.089355+0000","last_change":"2024-03-11T21:31:52.241561+0000","last_active":"2024-03-11T21:32:42.089355+0000","last_peered":"2024-03-11T21:32:42.089355+0000","last_clean":"2024-03-11T21:32:42.089355+0000","last_became_active":"2024-03-11T21:31:52.241194+0000","last_became_peered":"2024-03-11T21:31:52.241194+0000","last_unstale":"2024-03-11T21:32:42.089355+0000","last_undegraded":"2024-03-11T21:32:42.089355+0000","last_fullsized":"2024-03-11T21:32:42.089355+0000","mapping_epoch":157,"log_start":"0'0","ondisk_log_start":"0'0","created":157,"last_epoch_clean":158,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:31:51.219688+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:31:51.219688+0000","last_clean_scrub_stamp":"2024-03-11T21:31:51.219688+0000","objects_scrubbed":0,"log_size":4527,"log_dups_size":0,"ondisk_log_size":4527,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:32:54.353662+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":296681472,"num_objects":4527,"num_object_clones":0,"num_object_copies":13581,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4527,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4527,"num_write_kb":289728,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,0],"acting":[6,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.7","version":"124'2","reported_seq":241,"reported_epoch":174,"state":"active+clean","last_fresh":"2024-03-11T21:32:41.928352+0000","last_change":"2024-03-11T21:32:09.689578+0000","last_active":"2024-03-11T21:32:41.928352+0000","last_peered":"2024-03-11T21:32:41.928352+0000","last_clean":"2024-03-11T21:32:41.928352+0000","last_became_active":"2024-03-11T21:25:44.591376+0000","last_became_peered":"2024-03-11T21:25:44.591376+0000","last_unstale":"2024-03-11T21:32:41.928352+0000","last_undegraded":"2024-03-11T21:32:41.928352+0000","last_fullsized":"2024-03-11T21:32:41.928352+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"124'2","last_scrub_stamp":"2024-03-11T21:32:09.689480+0000","last_deep_scrub":"124'2","last_deep_scrub_stamp":"2024-03-11T21:29:34.956908+0000","last_clean_scrub_stamp":"2024-03-11T21:32:09.689480+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:33:20.464244+0000","scrub_duration":48,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,4],"acting":[5,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2","version":"71'2","reported_seq":268,"reported_epoch":174,"state":"active+clean","last_fresh":"2024-03-11T21:32:41.925486+0000","last_change":"2024-03-11T21:32:21.543178+0000","last_active":"2024-03-11T21:32:41.925486+0000","last_peered":"2024-03-11T21:32:41.925486+0000","last_clean":"2024-03-11T21:32:41.925486+0000","last_became_active":"2024-03-11T21:25:38.539413+0000","last_became_peered":"2024-03-11T21:25:38.539413+0000","last_unstale":"2024-03-11T21:32:41.925486+0000","last_undegraded":"2024-03-11T21:32:41.925486+0000","last_fullsized":"2024-03-11T21:32:41.925486+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"71'2","last_scrub_stamp":"2024-03-11T21:32:21.543053+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:37.512018+0000","last_clean_scrub_stamp":"2024-03-11T21:32:21.543053+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:33:23.042174+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0.0014206449999999999,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":271,"reported_epoch":174,"state":"active+clean","last_fresh":"2024-03-11T21:32:41.928472+0000","last_change":"2024-03-11T21:32:01.617566+0000","last_active":"2024-03-11T21:32:41.928472+0000","last_peered":"2024-03-11T21:32:41.928472+0000","last_clean":"2024-03-11T21:32:41.928472+0000","last_became_active":"2024-03-11T21:31:48.236738+0000","last_became_peered":"2024-03-11T21:31:48.236738+0000","last_unstale":"2024-03-11T21:32:41.928472+0000","last_undegraded":"2024-03-11T21:32:41.928472+0000","last_fullsized":"2024-03-11T21:32:41.928472+0000","mapping_epoch":153,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":154,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:32:01.617459+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:37.512018+0000","last_clean_scrub_stamp":"2024-03-11T21:32:01.617459+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:33:29.445302+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,3],"acting":[5,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"87'128","reported_seq":371,"reported_epoch":174,"state":"active+clean","last_fresh":"2024-03-11T21:32:41.923106+0000","last_change":"2024-03-11T21:32:20.541774+0000","last_active":"2024-03-11T21:32:41.923106+0000","last_peered":"2024-03-11T21:32:41.923106+0000","last_clean":"2024-03-11T21:32:41.923106+0000","last_became_active":"2024-03-11T21:31:51.235503+0000","last_became_peered":"2024-03-11T21:31:51.235503+0000","last_unstale":"2024-03-11T21:32:41.923106+0000","last_undegraded":"2024-03-11T21:32:41.923106+0000","last_fullsized":"2024-03-11T21:32:41.923106+0000","mapping_epoch":156,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":157,"parent":"0.0","parent_split_bits":0,"last_scrub":"87'128","last_scrub_stamp":"2024-03-11T21:32:20.540647+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:24.261974+0000","last_clean_scrub_stamp":"2024-03-11T21:32:20.540647+0000","objects_scrubbed":2,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:33:48.607298+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":198,"num_read_kb":294,"num_write":179,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":2361472,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,1],"acting":[0,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.5","version":"74'16","reported_seq":363,"reported_epoch":174,"state":"active+clean","last_fresh":"2024-03-11T21:32:41.925810+0000","last_change":"2024-03-11T21:32:08.790629+0000","last_active":"2024-03-11T21:32:41.925810+0000","last_peered":"2024-03-11T21:32:41.925810+0000","last_clean":"2024-03-11T21:32:41.925810+0000","last_became_active":"2024-03-11T21:25:40.761457+0000","last_became_peered":"2024-03-11T21:25:40.761457+0000","last_unstale":"2024-03-11T21:32:41.925810+0000","last_undegraded":"2024-03-11T21:32:41.925810+0000","last_fullsized":"2024-03-11T21:32:41.925810+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'16","last_scrub_stamp":"2024-03-11T21:32:08.790534+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:39.525554+0000","last_clean_scrub_stamp":"2024-03-11T21:32:08.790534+0000","objects_scrubbed":8,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:33:16.868656+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":154,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":25,"num_read_kb":15,"num_write":13,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":66,"reported_epoch":174,"state":"active+clean","last_fresh":"2024-03-11T21:32:41.938625+0000","last_change":"2024-03-11T21:32:22.795392+0000","last_active":"2024-03-11T21:32:41.938625+0000","last_peered":"2024-03-11T21:32:41.938625+0000","last_clean":"2024-03-11T21:32:41.938625+0000","last_became_active":"2024-03-11T21:32:22.795065+0000","last_became_peered":"2024-03-11T21:32:22.795065+0000","last_unstale":"2024-03-11T21:32:41.938625+0000","last_undegraded":"2024-03-11T21:32:41.938625+0000","last_fullsized":"2024-03-11T21:32:41.938625+0000","mapping_epoch":166,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":167,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:32:21.266672+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:32:21.266672+0000","last_clean_scrub_stamp":"2024-03-11T21:32:21.266672+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:33:23.214787+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,6],"acting":[2,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"8.a","version":"174'4438","reported_seq":4483,"reported_epoch":174,"state":"active+clean","last_fresh":"2024-03-11T21:32:42.197794+0000","last_change":"2024-03-11T21:31:52.272724+0000","last_active":"2024-03-11T21:32:42.197794+0000","last_peered":"2024-03-11T21:32:42.197794+0000","last_clean":"2024-03-11T21:32:42.197794+0000","last_became_active":"2024-03-11T21:31:52.271977+0000","last_became_peered":"2024-03-11T21:31:52.271977+0000","last_unstale":"2024-03-11T21:32:42.197794+0000","last_undegraded":"2024-03-11T21:32:42.197794+0000","last_fullsized":"2024-03-11T21:32:42.197794+0000","mapping_epoch":157,"log_start":"0'0","ondisk_log_start":"0'0","created":157,"last_epoch_clean":158,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:31:51.219688+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:31:51.219688+0000","last_clean_scrub_stamp":"2024-03-11T21:31:51.219688+0000","objects_scrubbed":0,"log_size":4438,"log_dups_size":0,"ondisk_log_size":4438,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:32:55.030994+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":290848768,"num_objects":4438,"num_object_clones":0,"num_object_copies":13314,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4438,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4438,"num_write_kb":284032,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,5],"acting":[1,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.4","version":"0'0","reported_seq":240,"reported_epoch":174,"state":"active+clean","last_fresh":"2024-03-11T21:32:41.927117+0000","last_change":"2024-03-11T21:30:58.758256+0000","last_active":"2024-03-11T21:32:41.927117+0000","last_peered":"2024-03-11T21:32:41.927117+0000","last_clean":"2024-03-11T21:32:41.927117+0000","last_became_active":"2024-03-11T21:25:44.594188+0000","last_became_peered":"2024-03-11T21:25:44.594188+0000","last_unstale":"2024-03-11T21:32:41.927117+0000","last_undegraded":"2024-03-11T21:32:41.927117+0000","last_fullsized":"2024-03-11T21:32:41.927117+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:30:58.757819+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:43.559452+0000","last_clean_scrub_stamp":"2024-03-11T21:30:58.757819+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:32:43.569419+0000","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":267,"reported_epoch":174,"state":"active+clean","last_fresh":"2024-03-11T21:32:41.919231+0000","last_change":"2024-03-11T21:32:22.561581+0000","last_active":"2024-03-11T21:32:41.919231+0000","last_peered":"2024-03-11T21:32:41.919231+0000","last_clean":"2024-03-11T21:32:41.919231+0000","last_became_active":"2024-03-11T21:25:38.543834+0000","last_became_peered":"2024-03-11T21:25:38.543834+0000","last_unstale":"2024-03-11T21:32:41.919231+0000","last_undegraded":"2024-03-11T21:32:41.919231+0000","last_fullsized":"2024-03-11T21:32:41.919231+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:32:22.560895+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:37.512018+0000","last_clean_scrub_stamp":"2024-03-11T21:32:22.560895+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:33:51.319357+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0.00089342499999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":52,"reported_epoch":174,"state":"active+clean","last_fresh":"2024-03-11T21:32:41.928196+0000","last_change":"2024-03-11T21:31:48.235540+0000","last_active":"2024-03-11T21:32:41.928196+0000","last_peered":"2024-03-11T21:32:41.928196+0000","last_clean":"2024-03-11T21:32:41.928196+0000","last_became_active":"2024-03-11T21:31:48.235234+0000","last_became_peered":"2024-03-11T21:31:48.235234+0000","last_unstale":"2024-03-11T21:32:41.928196+0000","last_undegraded":"2024-03-11T21:32:41.928196+0000","last_fullsized":"2024-03-11T21:32:41.928196+0000","mapping_epoch":153,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":154,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:31:06.457393+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:37.512018+0000","last_clean_scrub_stamp":"2024-03-11T21:31:06.457393+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:32:43.569452+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3],"acting":[1,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.6","version":"74'12","reported_seq":286,"reported_epoch":174,"state":"active+clean","last_fresh":"2024-03-11T21:32:41.921731+0000","last_change":"2024-03-11T21:32:22.816786+0000","last_active":"2024-03-11T21:32:41.921731+0000","last_peered":"2024-03-11T21:32:41.921731+0000","last_clean":"2024-03-11T21:32:41.921731+0000","last_became_active":"2024-03-11T21:32:22.796486+0000","last_became_peered":"2024-03-11T21:32:22.796486+0000","last_unstale":"2024-03-11T21:32:41.921731+0000","last_undegraded":"2024-03-11T21:32:41.921731+0000","last_fullsized":"2024-03-11T21:32:41.921731+0000","mapping_epoch":166,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":167,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'12","last_scrub_stamp":"2024-03-11T21:31:04.383580+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:39.525554+0000","last_clean_scrub_stamp":"2024-03-11T21:31:04.383580+0000","objects_scrubbed":6,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":25,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":18,"num_read_kb":12,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":243,"reported_epoch":174,"state":"active+clean","last_fresh":"2024-03-11T21:32:41.928033+0000","last_change":"2024-03-11T21:31:12.634176+0000","last_active":"2024-03-11T21:32:41.928033+0000","last_peered":"2024-03-11T21:32:41.928033+0000","last_clean":"2024-03-11T21:32:41.928033+0000","last_became_active":"2024-03-11T21:25:42.572324+0000","last_became_peered":"2024-03-11T21:25:42.572324+0000","last_unstale":"2024-03-11T21:32:41.928033+0000","last_undegraded":"2024-03-11T21:32:41.928033+0000","last_fullsized":"2024-03-11T21:32:41.928033+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:31:12.634040+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:41.544550+0000","last_clean_scrub_stamp":"2024-03-11T21:31:12.634040+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:32:45.331201+0000","scrub_duration":16,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.f","version":"174'4442","reported_seq":4487,"reported_epoch":174,"state":"active+clean","last_fresh":"2024-03-11T21:32:42.521840+0000","last_change":"2024-03-11T21:31:52.272528+0000","last_active":"2024-03-11T21:32:42.521840+0000","last_peered":"2024-03-11T21:32:42.521840+0000","last_clean":"2024-03-11T21:32:42.521840+0000","last_became_active":"2024-03-11T21:31:52.272191+0000","last_became_peered":"2024-03-11T21:31:52.272191+0000","last_unstale":"2024-03-11T21:32:42.521840+0000","last_undegraded":"2024-03-11T21:32:42.521840+0000","last_fullsized":"2024-03-11T21:32:42.521840+0000","mapping_epoch":157,"log_start":"0'0","ondisk_log_start":"0'0","created":157,"last_epoch_clean":158,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:31:51.219688+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:31:51.219688+0000","last_clean_scrub_stamp":"2024-03-11T21:31:51.219688+0000","objects_scrubbed":0,"log_size":4442,"log_dups_size":0,"ondisk_log_size":4442,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:32:56.929209+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":291110912,"num_objects":4442,"num_object_clones":0,"num_object_copies":13326,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4442,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4442,"num_write_kb":284288,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,1],"acting":[3,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":248,"reported_epoch":174,"state":"active+clean","last_fresh":"2024-03-11T21:32:41.927391+0000","last_change":"2024-03-11T21:31:01.858727+0000","last_active":"2024-03-11T21:32:41.927391+0000","last_peered":"2024-03-11T21:32:41.927391+0000","last_clean":"2024-03-11T21:32:41.927391+0000","last_became_active":"2024-03-11T21:25:45.301416+0000","last_became_peered":"2024-03-11T21:25:45.301416+0000","last_unstale":"2024-03-11T21:32:41.927391+0000","last_undegraded":"2024-03-11T21:32:41.927391+0000","last_fullsized":"2024-03-11T21:32:41.927391+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:31:01.856502+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:43.559452+0000","last_clean_scrub_stamp":"2024-03-11T21:31:01.856502+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:32:43.569467+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":60,"reported_epoch":174,"state":"active+clean","last_fresh":"2024-03-11T21:32:41.921638+0000","last_change":"2024-03-11T21:31:56.381335+0000","last_active":"2024-03-11T21:32:41.921638+0000","last_peered":"2024-03-11T21:32:41.921638+0000","last_clean":"2024-03-11T21:32:41.921638+0000","last_became_active":"2024-03-11T21:31:48.222884+0000","last_became_peered":"2024-03-11T21:31:48.222884+0000","last_unstale":"2024-03-11T21:32:41.921638+0000","last_undegraded":"2024-03-11T21:32:41.921638+0000","last_fullsized":"2024-03-11T21:32:41.921638+0000","mapping_epoch":153,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":154,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:31:56.381231+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:26:57.333669+0000","last_clean_scrub_stamp":"2024-03-11T21:31:56.381231+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:33:03.065697+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,1],"acting":[0,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.3","version":"165'27","reported_seq":324,"reported_epoch":174,"state":"active+clean","last_fresh":"2024-03-11T21:32:41.918681+0000","last_change":"2024-03-11T21:32:21.576912+0000","last_active":"2024-03-11T21:32:41.918681+0000","last_peered":"2024-03-11T21:32:41.918681+0000","last_clean":"2024-03-11T21:32:41.918681+0000","last_became_active":"2024-03-11T21:31:48.223109+0000","last_became_peered":"2024-03-11T21:31:48.223109+0000","last_unstale":"2024-03-11T21:32:41.918681+0000","last_undegraded":"2024-03-11T21:32:41.918681+0000","last_fullsized":"2024-03-11T21:32:41.918681+0000","mapping_epoch":153,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":154,"parent":"0.0","parent_split_bits":0,"last_scrub":"165'27","last_scrub_stamp":"2024-03-11T21:32:21.576732+0000","last_deep_scrub":"165'27","last_deep_scrub_stamp":"2024-03-11T21:32:21.576732+0000","last_clean_scrub_stamp":"2024-03-11T21:32:21.576732+0000","objects_scrubbed":8,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:33:49.684107+0000","scrub_duration":23,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":39,"num_read_kb":25,"num_write":22,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,1],"acting":[0,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":260,"reported_epoch":174,"state":"active+clean","last_fresh":"2024-03-11T21:32:41.924296+0000","last_change":"2024-03-11T21:32:23.882539+0000","last_active":"2024-03-11T21:32:41.924296+0000","last_peered":"2024-03-11T21:32:41.924296+0000","last_clean":"2024-03-11T21:32:41.924296+0000","last_became_active":"2024-03-11T21:25:42.579045+0000","last_became_peered":"2024-03-11T21:25:42.579045+0000","last_unstale":"2024-03-11T21:32:41.924296+0000","last_undegraded":"2024-03-11T21:32:41.924296+0000","last_fullsized":"2024-03-11T21:32:41.924296+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:32:23.882391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:29:36.918936+0000","last_clean_scrub_stamp":"2024-03-11T21:32:23.882391+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:33:44.401769+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"8.c","version":"174'4459","reported_seq":4504,"reported_epoch":174,"state":"active+clean","last_fresh":"2024-03-11T21:32:42.862954+0000","last_change":"2024-03-11T21:31:52.240141+0000","last_active":"2024-03-11T21:32:42.862954+0000","last_peered":"2024-03-11T21:32:42.862954+0000","last_clean":"2024-03-11T21:32:42.862954+0000","last_became_active":"2024-03-11T21:31:52.239816+0000","last_became_peered":"2024-03-11T21:31:52.239816+0000","last_unstale":"2024-03-11T21:32:42.862954+0000","last_undegraded":"2024-03-11T21:32:42.862954+0000","last_fullsized":"2024-03-11T21:32:42.862954+0000","mapping_epoch":157,"log_start":"0'0","ondisk_log_start":"0'0","created":157,"last_epoch_clean":158,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:31:51.219688+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:31:51.219688+0000","last_clean_scrub_stamp":"2024-03-11T21:31:51.219688+0000","objects_scrubbed":0,"log_size":4459,"log_dups_size":0,"ondisk_log_size":4459,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:33:10.166287+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":292225024,"num_objects":4459,"num_object_clones":0,"num_object_copies":13377,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4459,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4459,"num_write_kb":285376,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,5],"acting":[0,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":254,"reported_epoch":174,"state":"active+clean","last_fresh":"2024-03-11T21:32:41.914387+0000","last_change":"2024-03-11T21:32:22.791904+0000","last_active":"2024-03-11T21:32:41.914387+0000","last_peered":"2024-03-11T21:32:41.914387+0000","last_clean":"2024-03-11T21:32:41.914387+0000","last_became_active":"2024-03-11T21:32:22.791439+0000","last_became_peered":"2024-03-11T21:32:22.791439+0000","last_unstale":"2024-03-11T21:32:41.914387+0000","last_undegraded":"2024-03-11T21:32:41.914387+0000","last_fullsized":"2024-03-11T21:32:41.914387+0000","mapping_epoch":166,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":167,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:32:05.636174+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:43.559452+0000","last_clean_scrub_stamp":"2024-03-11T21:32:05.636174+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:33:17.728230+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,2],"acting":[4,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.6","version":"67'1","reported_seq":261,"reported_epoch":174,"state":"active+clean","last_fresh":"2024-03-11T21:32:41.927513+0000","last_change":"2024-03-11T21:32:05.556583+0000","last_active":"2024-03-11T21:32:41.927513+0000","last_peered":"2024-03-11T21:32:41.927513+0000","last_clean":"2024-03-11T21:32:41.927513+0000","last_became_active":"2024-03-11T21:25:38.536142+0000","last_became_peered":"2024-03-11T21:25:38.536142+0000","last_unstale":"2024-03-11T21:32:41.927513+0000","last_undegraded":"2024-03-11T21:32:41.927513+0000","last_fullsized":"2024-03-11T21:32:41.927513+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'1","last_scrub_stamp":"2024-03-11T21:32:05.556427+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:37.512018+0000","last_clean_scrub_stamp":"2024-03-11T21:32:05.556427+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:33:11.248441+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,4],"acting":[1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.0","version":"93'26","reported_seq":311,"reported_epoch":174,"state":"active+clean","last_fresh":"2024-03-11T21:32:41.931433+0000","last_change":"2024-03-11T21:31:48.258922+0000","last_active":"2024-03-11T21:32:41.931433+0000","last_peered":"2024-03-11T21:32:41.931433+0000","last_clean":"2024-03-11T21:32:41.931433+0000","last_became_active":"2024-03-11T21:31:48.229925+0000","last_became_peered":"2024-03-11T21:31:48.229925+0000","last_unstale":"2024-03-11T21:32:41.931433+0000","last_undegraded":"2024-03-11T21:32:41.931433+0000","last_fullsized":"2024-03-11T21:32:41.931433+0000","mapping_epoch":153,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":154,"parent":"0.0","parent_split_bits":0,"last_scrub":"93'26","last_scrub_stamp":"2024-03-11T21:31:03.244599+0000","last_deep_scrub":"93'26","last_deep_scrub_stamp":"2024-03-11T21:26:40.680324+0000","last_clean_scrub_stamp":"2024-03-11T21:31:03.244599+0000","objects_scrubbed":8,"log_size":26,"log_dups_size":0,"ondisk_log_size":26,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":19,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":34,"num_read_kb":22,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":258,"reported_epoch":174,"state":"active+clean","last_fresh":"2024-03-11T21:32:41.913786+0000","last_change":"2024-03-11T21:32:16.599224+0000","last_active":"2024-03-11T21:32:41.913786+0000","last_peered":"2024-03-11T21:32:41.913786+0000","last_clean":"2024-03-11T21:32:41.913786+0000","last_became_active":"2024-03-11T21:31:48.226442+0000","last_became_peered":"2024-03-11T21:31:48.226442+0000","last_unstale":"2024-03-11T21:32:41.913786+0000","last_undegraded":"2024-03-11T21:32:41.913786+0000","last_fullsized":"2024-03-11T21:32:41.913786+0000","mapping_epoch":153,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":154,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:32:16.599049+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:26:51.863621+0000","last_clean_scrub_stamp":"2024-03-11T21:32:16.599049+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:33:21.882351+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0],"acting":[4,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"8.d","version":"174'4547","reported_seq":4591,"reported_epoch":174,"state":"active+clean","last_fresh":"2024-03-11T21:32:42.037229+0000","last_change":"2024-03-11T21:31:52.272819+0000","last_active":"2024-03-11T21:32:42.037229+0000","last_peered":"2024-03-11T21:32:42.037229+0000","last_clean":"2024-03-11T21:32:42.037229+0000","last_became_active":"2024-03-11T21:31:52.272281+0000","last_became_peered":"2024-03-11T21:31:52.272281+0000","last_unstale":"2024-03-11T21:32:42.037229+0000","last_undegraded":"2024-03-11T21:32:42.037229+0000","last_fullsized":"2024-03-11T21:32:42.037229+0000","mapping_epoch":157,"log_start":"0'0","ondisk_log_start":"0'0","created":157,"last_epoch_clean":158,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:31:51.219688+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:31:51.219688+0000","last_clean_scrub_stamp":"2024-03-11T21:31:51.219688+0000","objects_scrubbed":0,"log_size":4547,"log_dups_size":0,"ondisk_log_size":4547,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:33:01.290866+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":297992192,"num_objects":4547,"num_object_clones":0,"num_object_copies":13641,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4547,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4547,"num_write_kb":291008,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,6],"acting":[5,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.3","version":"0'0","reported_seq":60,"reported_epoch":174,"state":"active+clean","last_fresh":"2024-03-11T21:32:41.925042+0000","last_change":"2024-03-11T21:31:54.671396+0000","last_active":"2024-03-11T21:32:41.925042+0000","last_peered":"2024-03-11T21:32:41.925042+0000","last_clean":"2024-03-11T21:32:41.925042+0000","last_became_active":"2024-03-11T21:31:48.220482+0000","last_became_peered":"2024-03-11T21:31:48.220482+0000","last_unstale":"2024-03-11T21:32:41.925042+0000","last_undegraded":"2024-03-11T21:32:41.925042+0000","last_fullsized":"2024-03-11T21:32:41.925042+0000","mapping_epoch":153,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":154,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:31:54.671300+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:29:42.485740+0000","last_clean_scrub_stamp":"2024-03-11T21:31:54.671300+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:33:20.587123+0000","scrub_duration":16,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,1],"acting":[6,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":266,"reported_epoch":174,"state":"active+clean","last_fresh":"2024-03-11T21:32:41.925572+0000","last_change":"2024-03-11T21:31:48.236203+0000","last_active":"2024-03-11T21:32:41.925572+0000","last_peered":"2024-03-11T21:32:41.925572+0000","last_clean":"2024-03-11T21:32:41.925572+0000","last_became_active":"2024-03-11T21:31:48.235815+0000","last_became_peered":"2024-03-11T21:31:48.235815+0000","last_unstale":"2024-03-11T21:32:41.925572+0000","last_undegraded":"2024-03-11T21:32:41.925572+0000","last_fullsized":"2024-03-11T21:32:41.925572+0000","mapping_epoch":153,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":154,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:31:01.530407+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:37.512018+0000","last_clean_scrub_stamp":"2024-03-11T21:31:01.530407+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:32:43.941405+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,3],"acting":[6,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.1","version":"74'14","reported_seq":269,"reported_epoch":174,"state":"active+clean","last_fresh":"2024-03-11T21:32:41.913511+0000","last_change":"2024-03-11T21:31:02.486605+0000","last_active":"2024-03-11T21:32:41.913511+0000","last_peered":"2024-03-11T21:32:41.913511+0000","last_clean":"2024-03-11T21:32:41.913511+0000","last_became_active":"2024-03-11T21:25:40.546219+0000","last_became_peered":"2024-03-11T21:25:40.546219+0000","last_unstale":"2024-03-11T21:32:41.913511+0000","last_undegraded":"2024-03-11T21:32:41.913511+0000","last_fullsized":"2024-03-11T21:32:41.913511+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'14","last_scrub_stamp":"2024-03-11T21:31:02.486457+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:39.525554+0000","last_clean_scrub_stamp":"2024-03-11T21:31:02.486457+0000","objects_scrubbed":7,"log_size":14,"log_dups_size":0,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":21,"num_read_kb":14,"num_write":14,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6],"acting":[4,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.0","version":"98'9","reported_seq":337,"reported_epoch":174,"state":"active+clean","last_fresh":"2024-03-11T21:32:41.931537+0000","last_change":"2024-03-11T21:31:21.562516+0000","last_active":"2024-03-11T21:32:41.931537+0000","last_peered":"2024-03-11T21:32:41.931537+0000","last_clean":"2024-03-11T21:32:41.931537+0000","last_became_active":"2024-03-11T21:25:42.576126+0000","last_became_peered":"2024-03-11T21:25:42.576126+0000","last_unstale":"2024-03-11T21:32:41.931537+0000","last_undegraded":"2024-03-11T21:32:41.931537+0000","last_fullsized":"2024-03-11T21:32:41.931537+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"98'9","last_scrub_stamp":"2024-03-11T21:31:21.562395+0000","last_deep_scrub":"98'9","last_deep_scrub_stamp":"2024-03-11T21:27:29.745711+0000","last_clean_scrub_stamp":"2024-03-11T21:31:21.562395+0000","objects_scrubbed":1,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":17,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.2","version":"174'4504","reported_seq":4549,"reported_epoch":174,"state":"active+clean","last_fresh":"2024-03-11T21:32:41.983802+0000","last_change":"2024-03-11T21:31:52.237340+0000","last_active":"2024-03-11T21:32:41.983802+0000","last_peered":"2024-03-11T21:32:41.983802+0000","last_clean":"2024-03-11T21:32:41.983802+0000","last_became_active":"2024-03-11T21:31:52.236955+0000","last_became_peered":"2024-03-11T21:31:52.236955+0000","last_unstale":"2024-03-11T21:32:41.983802+0000","last_undegraded":"2024-03-11T21:32:41.983802+0000","last_fullsized":"2024-03-11T21:32:41.983802+0000","mapping_epoch":157,"log_start":"0'0","ondisk_log_start":"0'0","created":157,"last_epoch_clean":158,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:31:51.219688+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:31:51.219688+0000","last_clean_scrub_stamp":"2024-03-11T21:31:51.219688+0000","objects_scrubbed":0,"log_size":4504,"log_dups_size":0,"ondisk_log_size":4504,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:33:05.560741+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":295174144,"num_objects":4504,"num_object_clones":0,"num_object_copies":13512,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4504,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4504,"num_write_kb":288256,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,5],"acting":[4,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.c","version":"0'0","reported_seq":239,"reported_epoch":174,"state":"active+clean","last_fresh":"2024-03-11T21:32:41.927961+0000","last_change":"2024-03-11T21:32:17.600335+0000","last_active":"2024-03-11T21:32:41.927961+0000","last_peered":"2024-03-11T21:32:41.927961+0000","last_clean":"2024-03-11T21:32:41.927961+0000","last_became_active":"2024-03-11T21:25:45.302207+0000","last_became_peered":"2024-03-11T21:25:45.302207+0000","last_unstale":"2024-03-11T21:32:41.927961+0000","last_undegraded":"2024-03-11T21:32:41.927961+0000","last_fullsized":"2024-03-11T21:32:41.927961+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:32:17.600213+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:43.559452+0000","last_clean_scrub_stamp":"2024-03-11T21:32:17.600213+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:33:20.285441+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.f","version":"0'0","reported_seq":251,"reported_epoch":174,"state":"active+clean","last_fresh":"2024-03-11T21:32:41.927846+0000","last_change":"2024-03-11T21:31:24.574334+0000","last_active":"2024-03-11T21:32:41.927846+0000","last_peered":"2024-03-11T21:32:41.927846+0000","last_clean":"2024-03-11T21:32:41.927846+0000","last_became_active":"2024-03-11T21:25:42.575935+0000","last_became_peered":"2024-03-11T21:25:42.575935+0000","last_unstale":"2024-03-11T21:32:41.927846+0000","last_undegraded":"2024-03-11T21:32:41.927846+0000","last_fullsized":"2024-03-11T21:32:41.927846+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:31:24.574239+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:41.544550+0000","last_clean_scrub_stamp":"2024-03-11T21:31:24.574239+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.8","version":"0'0","reported_seq":60,"reported_epoch":174,"state":"active+clean","last_fresh":"2024-03-11T21:32:41.923383+0000","last_change":"2024-03-11T21:32:06.392141+0000","last_active":"2024-03-11T21:32:41.923383+0000","last_peered":"2024-03-11T21:32:41.923383+0000","last_clean":"2024-03-11T21:32:41.923383+0000","last_became_active":"2024-03-11T21:31:48.222826+0000","last_became_peered":"2024-03-11T21:31:48.222826+0000","last_unstale":"2024-03-11T21:32:41.923383+0000","last_undegraded":"2024-03-11T21:32:41.923383+0000","last_fullsized":"2024-03-11T21:32:41.923383+0000","mapping_epoch":153,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":154,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:32:06.392027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:37.512018+0000","last_clean_scrub_stamp":"2024-03-11T21:32:06.392027+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:33:08.153624+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.e","version":"109'16","reported_seq":272,"reported_epoch":174,"state":"active+clean","last_fresh":"2024-03-11T21:32:41.914514+0000","last_change":"2024-03-11T21:30:56.453809+0000","last_active":"2024-03-11T21:32:41.914514+0000","last_peered":"2024-03-11T21:32:41.914514+0000","last_clean":"2024-03-11T21:32:41.914514+0000","last_became_active":"2024-03-11T21:25:40.552178+0000","last_became_peered":"2024-03-11T21:25:40.552178+0000","last_unstale":"2024-03-11T21:32:41.914514+0000","last_undegraded":"2024-03-11T21:32:41.914514+0000","last_fullsized":"2024-03-11T21:32:41.914514+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"109'16","last_scrub_stamp":"2024-03-11T21:30:56.453670+0000","last_deep_scrub":"109'16","last_deep_scrub_stamp":"2024-03-11T21:28:24.739395+0000","last_clean_scrub_stamp":"2024-03-11T21:30:56.453670+0000","objects_scrubbed":5,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"8.3","version":"174'4396","reported_seq":4440,"reported_epoch":174,"state":"active+clean","last_fresh":"2024-03-11T21:32:42.204523+0000","last_change":"2024-03-11T21:31:52.237339+0000","last_active":"2024-03-11T21:32:42.204523+0000","last_peered":"2024-03-11T21:32:42.204523+0000","last_clean":"2024-03-11T21:32:42.204523+0000","last_became_active":"2024-03-11T21:31:52.237034+0000","last_became_peered":"2024-03-11T21:31:52.237034+0000","last_unstale":"2024-03-11T21:32:42.204523+0000","last_undegraded":"2024-03-11T21:32:42.204523+0000","last_fullsized":"2024-03-11T21:32:42.204523+0000","mapping_epoch":157,"log_start":"0'0","ondisk_log_start":"0'0","created":157,"last_epoch_clean":158,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:31:51.219688+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:31:51.219688+0000","last_clean_scrub_stamp":"2024-03-11T21:31:51.219688+0000","objects_scrubbed":0,"log_size":4396,"log_dups_size":0,"ondisk_log_size":4396,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:33:05.442105+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":288096256,"num_objects":4396,"num_object_clones":0,"num_object_copies":13188,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4396,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4396,"num_write_kb":281344,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,6],"acting":[1,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.d","version":"0'0","reported_seq":247,"reported_epoch":174,"state":"active+clean","last_fresh":"2024-03-11T21:32:41.928022+0000","last_change":"2024-03-11T21:32:13.641959+0000","last_active":"2024-03-11T21:32:41.928022+0000","last_peered":"2024-03-11T21:32:41.928022+0000","last_clean":"2024-03-11T21:32:41.928022+0000","last_became_active":"2024-03-11T21:25:44.590333+0000","last_became_peered":"2024-03-11T21:25:44.590333+0000","last_unstale":"2024-03-11T21:32:41.928022+0000","last_undegraded":"2024-03-11T21:32:41.928022+0000","last_fullsized":"2024-03-11T21:32:41.928022+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:32:13.641870+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:43.559452+0000","last_clean_scrub_stamp":"2024-03-11T21:32:13.641870+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:33:40.168472+0000","scrub_duration":45,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.e","version":"74'8","reported_seq":337,"reported_epoch":174,"state":"active+clean","last_fresh":"2024-03-11T21:32:41.914745+0000","last_change":"2024-03-11T21:31:05.564402+0000","last_active":"2024-03-11T21:32:41.914745+0000","last_peered":"2024-03-11T21:32:41.914745+0000","last_clean":"2024-03-11T21:32:41.914745+0000","last_became_active":"2024-03-11T21:25:42.576742+0000","last_became_peered":"2024-03-11T21:25:42.576742+0000","last_unstale":"2024-03-11T21:32:41.914745+0000","last_undegraded":"2024-03-11T21:32:41.914745+0000","last_fullsized":"2024-03-11T21:32:41.914745+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'8","last_scrub_stamp":"2024-03-11T21:31:05.564320+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:41.544550+0000","last_clean_scrub_stamp":"2024-03-11T21:31:05.564320+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,0],"acting":[4,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.9","version":"0'0","reported_seq":266,"reported_epoch":174,"state":"active+clean","last_fresh":"2024-03-11T21:32:41.927223+0000","last_change":"2024-03-11T21:32:02.507023+0000","last_active":"2024-03-11T21:32:41.927223+0000","last_peered":"2024-03-11T21:32:41.927223+0000","last_clean":"2024-03-11T21:32:41.927223+0000","last_became_active":"2024-03-11T21:31:48.234424+0000","last_became_peered":"2024-03-11T21:31:48.234424+0000","last_unstale":"2024-03-11T21:32:41.927223+0000","last_undegraded":"2024-03-11T21:32:41.927223+0000","last_fullsized":"2024-03-11T21:32:41.927223+0000","mapping_epoch":153,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":154,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:32:02.506579+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:26:50.991712+0000","last_clean_scrub_stamp":"2024-03-11T21:32:02.506579+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:33:11.511126+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,2],"acting":[1,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.f","version":"74'15","reported_seq":232,"reported_epoch":174,"state":"active+clean","last_fresh":"2024-03-11T21:32:41.939397+0000","last_change":"2024-03-11T21:32:02.432661+0000","last_active":"2024-03-11T21:32:41.939397+0000","last_peered":"2024-03-11T21:32:41.939397+0000","last_clean":"2024-03-11T21:32:41.939397+0000","last_became_active":"2024-03-11T21:26:11.216224+0000","last_became_peered":"2024-03-11T21:26:11.216224+0000","last_unstale":"2024-03-11T21:32:41.939397+0000","last_undegraded":"2024-03-11T21:32:41.939397+0000","last_fullsized":"2024-03-11T21:32:41.939397+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'15","last_scrub_stamp":"2024-03-11T21:32:02.432529+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:39.525554+0000","last_clean_scrub_stamp":"2024-03-11T21:32:02.432529+0000","objects_scrubbed":7,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:33:02.536334+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,4],"acting":[2,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"8.0","version":"174'4480","reported_seq":4525,"reported_epoch":174,"state":"active+clean","last_fresh":"2024-03-11T21:32:42.539342+0000","last_change":"2024-03-11T21:31:52.239973+0000","last_active":"2024-03-11T21:32:42.539342+0000","last_peered":"2024-03-11T21:32:42.539342+0000","last_clean":"2024-03-11T21:32:42.539342+0000","last_became_active":"2024-03-11T21:31:52.239611+0000","last_became_peered":"2024-03-11T21:31:52.239611+0000","last_unstale":"2024-03-11T21:32:42.539342+0000","last_undegraded":"2024-03-11T21:32:42.539342+0000","last_fullsized":"2024-03-11T21:32:42.539342+0000","mapping_epoch":157,"log_start":"0'0","ondisk_log_start":"0'0","created":157,"last_epoch_clean":158,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:31:51.219688+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:31:51.219688+0000","last_clean_scrub_stamp":"2024-03-11T21:31:51.219688+0000","objects_scrubbed":0,"log_size":4480,"log_dups_size":0,"ondisk_log_size":4480,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:32:58.342661+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":293601280,"num_objects":4480,"num_object_clones":0,"num_object_copies":13440,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4480,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4480,"num_write_kb":286720,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,1],"acting":[3,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.e","version":"0'0","reported_seq":240,"reported_epoch":174,"state":"active+clean","last_fresh":"2024-03-11T21:32:41.914748+0000","last_change":"2024-03-11T21:31:09.597590+0000","last_active":"2024-03-11T21:32:41.914748+0000","last_peered":"2024-03-11T21:32:41.914748+0000","last_clean":"2024-03-11T21:32:41.914748+0000","last_became_active":"2024-03-11T21:25:44.590198+0000","last_became_peered":"2024-03-11T21:25:44.590198+0000","last_unstale":"2024-03-11T21:32:41.914748+0000","last_undegraded":"2024-03-11T21:32:41.914748+0000","last_fullsized":"2024-03-11T21:32:41.914748+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:31:09.597448+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:43.559452+0000","last_clean_scrub_stamp":"2024-03-11T21:31:09.597448+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:32:42.700183+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.d","version":"109'9","reported_seq":345,"reported_epoch":174,"state":"active+clean","last_fresh":"2024-03-11T21:32:41.938858+0000","last_change":"2024-03-11T21:32:12.413588+0000","last_active":"2024-03-11T21:32:41.938858+0000","last_peered":"2024-03-11T21:32:41.938858+0000","last_clean":"2024-03-11T21:32:41.938858+0000","last_became_active":"2024-03-11T21:25:42.575583+0000","last_became_peered":"2024-03-11T21:25:42.575583+0000","last_unstale":"2024-03-11T21:32:41.938858+0000","last_undegraded":"2024-03-11T21:32:41.938858+0000","last_fullsized":"2024-03-11T21:32:41.938858+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"109'9","last_scrub_stamp":"2024-03-11T21:32:12.413216+0000","last_deep_scrub":"109'9","last_deep_scrub_stamp":"2024-03-11T21:28:27.082975+0000","last_clean_scrub_stamp":"2024-03-11T21:32:12.413216+0000","objects_scrubbed":1,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:33:25.590071+0000","scrub_duration":25,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,5],"acting":[2,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.a","version":"0'0","reported_seq":266,"reported_epoch":174,"state":"active+clean","last_fresh":"2024-03-11T21:32:41.930347+0000","last_change":"2024-03-11T21:31:48.235649+0000","last_active":"2024-03-11T21:32:41.930347+0000","last_peered":"2024-03-11T21:32:41.930347+0000","last_clean":"2024-03-11T21:32:41.930347+0000","last_became_active":"2024-03-11T21:31:48.235272+0000","last_became_peered":"2024-03-11T21:31:48.235272+0000","last_unstale":"2024-03-11T21:32:41.930347+0000","last_undegraded":"2024-03-11T21:32:41.930347+0000","last_fullsized":"2024-03-11T21:32:41.930347+0000","mapping_epoch":153,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":154,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:31:06.898394+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:37.512018+0000","last_clean_scrub_stamp":"2024-03-11T21:31:06.898394+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:32:43.569403+0000","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,5],"acting":[1,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.c","version":"74'10","reported_seq":263,"reported_epoch":174,"state":"active+clean","last_fresh":"2024-03-11T21:32:41.913650+0000","last_change":"2024-03-11T21:30:57.495457+0000","last_active":"2024-03-11T21:32:41.913650+0000","last_peered":"2024-03-11T21:32:41.913650+0000","last_clean":"2024-03-11T21:32:41.913650+0000","last_became_active":"2024-03-11T21:25:40.550885+0000","last_became_peered":"2024-03-11T21:25:40.550885+0000","last_unstale":"2024-03-11T21:32:41.913650+0000","last_undegraded":"2024-03-11T21:32:41.913650+0000","last_fullsized":"2024-03-11T21:32:41.913650+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'10","last_scrub_stamp":"2024-03-11T21:30:57.495302+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:39.525554+0000","last_clean_scrub_stamp":"2024-03-11T21:30:57.495302+0000","objects_scrubbed":5,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"8.1","version":"174'4448","reported_seq":4493,"reported_epoch":174,"state":"active+clean","last_fresh":"2024-03-11T21:32:42.188317+0000","last_change":"2024-03-11T21:31:52.272438+0000","last_active":"2024-03-11T21:32:42.188317+0000","last_peered":"2024-03-11T21:32:42.188317+0000","last_clean":"2024-03-11T21:32:42.188317+0000","last_became_active":"2024-03-11T21:31:52.272076+0000","last_became_peered":"2024-03-11T21:31:52.272076+0000","last_unstale":"2024-03-11T21:32:42.188317+0000","last_undegraded":"2024-03-11T21:32:42.188317+0000","last_fullsized":"2024-03-11T21:32:42.188317+0000","mapping_epoch":157,"log_start":"0'0","ondisk_log_start":"0'0","created":157,"last_epoch_clean":158,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:31:51.219688+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:31:51.219688+0000","last_clean_scrub_stamp":"2024-03-11T21:31:51.219688+0000","objects_scrubbed":0,"log_size":4448,"log_dups_size":0,"ondisk_log_size":4448,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:33:01.373954+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":291504128,"num_objects":4448,"num_object_clones":0,"num_object_copies":13344,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4448,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4448,"num_write_kb":284672,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.f","version":"0'0","reported_seq":247,"reported_epoch":174,"state":"active+clean","last_fresh":"2024-03-11T21:32:41.939250+0000","last_change":"2024-03-11T21:32:17.431835+0000","last_active":"2024-03-11T21:32:41.939250+0000","last_peered":"2024-03-11T21:32:41.939250+0000","last_clean":"2024-03-11T21:32:41.939250+0000","last_became_active":"2024-03-11T21:25:44.593414+0000","last_became_peered":"2024-03-11T21:25:44.593414+0000","last_unstale":"2024-03-11T21:32:41.939250+0000","last_undegraded":"2024-03-11T21:32:41.939250+0000","last_fullsized":"2024-03-11T21:32:41.939250+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:32:17.431576+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:27:03.010969+0000","last_clean_scrub_stamp":"2024-03-11T21:32:17.431576+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:33:39.610791+0000","scrub_duration":75,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,4],"acting":[2,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":244,"reported_epoch":174,"state":"active+clean","last_fresh":"2024-03-11T21:32:41.926964+0000","last_change":"2024-03-11T21:30:22.750174+0000","last_active":"2024-03-11T21:32:41.926964+0000","last_peered":"2024-03-11T21:32:41.926964+0000","last_clean":"2024-03-11T21:32:41.926964+0000","last_became_active":"2024-03-11T21:25:42.579409+0000","last_became_peered":"2024-03-11T21:25:42.579409+0000","last_unstale":"2024-03-11T21:32:41.926964+0000","last_undegraded":"2024-03-11T21:32:41.926964+0000","last_fullsized":"2024-03-11T21:32:41.926964+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:30:22.750053+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:30:22.750053+0000","last_clean_scrub_stamp":"2024-03-11T21:30:22.750053+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:32:43.569383+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.b","version":"0'0","reported_seq":52,"reported_epoch":174,"state":"active+clean","last_fresh":"2024-03-11T21:32:41.913772+0000","last_change":"2024-03-11T21:31:48.229615+0000","last_active":"2024-03-11T21:32:41.913772+0000","last_peered":"2024-03-11T21:32:41.913772+0000","last_clean":"2024-03-11T21:32:41.913772+0000","last_became_active":"2024-03-11T21:31:48.228437+0000","last_became_peered":"2024-03-11T21:31:48.228437+0000","last_unstale":"2024-03-11T21:32:41.913772+0000","last_undegraded":"2024-03-11T21:32:41.913772+0000","last_fullsized":"2024-03-11T21:32:41.913772+0000","mapping_epoch":153,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":154,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:31:20.539243+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:37.512018+0000","last_clean_scrub_stamp":"2024-03-11T21:31:20.539243+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,3],"acting":[4,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.d","version":"165'25","reported_seq":292,"reported_epoch":174,"state":"active+clean","last_fresh":"2024-03-11T21:32:41.913615+0000","last_change":"2024-03-11T21:32:21.678485+0000","last_active":"2024-03-11T21:32:41.913615+0000","last_peered":"2024-03-11T21:32:41.913615+0000","last_clean":"2024-03-11T21:32:41.913615+0000","last_became_active":"2024-03-11T21:25:40.553454+0000","last_became_peered":"2024-03-11T21:25:40.553454+0000","last_unstale":"2024-03-11T21:32:41.913615+0000","last_undegraded":"2024-03-11T21:32:41.913615+0000","last_fullsized":"2024-03-11T21:32:41.913615+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"165'25","last_scrub_stamp":"2024-03-11T21:32:21.678328+0000","last_deep_scrub":"165'25","last_deep_scrub_stamp":"2024-03-11T21:32:21.678328+0000","last_clean_scrub_stamp":"2024-03-11T21:32:21.678328+0000","objects_scrubbed":8,"log_size":25,"log_dups_size":0,"ondisk_log_size":25,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:33:42.632132+0000","scrub_duration":21,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":29,"num_read_kb":19,"num_write":18,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,1],"acting":[4,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"8.6","version":"174'4444","reported_seq":4489,"reported_epoch":174,"state":"active+clean","last_fresh":"2024-03-11T21:32:42.852029+0000","last_change":"2024-03-11T21:31:52.238026+0000","last_active":"2024-03-11T21:32:42.852029+0000","last_peered":"2024-03-11T21:32:42.852029+0000","last_clean":"2024-03-11T21:32:42.852029+0000","last_became_active":"2024-03-11T21:31:52.237725+0000","last_became_peered":"2024-03-11T21:31:52.237725+0000","last_unstale":"2024-03-11T21:32:42.852029+0000","last_undegraded":"2024-03-11T21:32:42.852029+0000","last_fullsized":"2024-03-11T21:32:42.852029+0000","mapping_epoch":157,"log_start":"0'0","ondisk_log_start":"0'0","created":157,"last_epoch_clean":158,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:31:51.219688+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:31:51.219688+0000","last_clean_scrub_stamp":"2024-03-11T21:31:51.219688+0000","objects_scrubbed":0,"log_size":4444,"log_dups_size":0,"ondisk_log_size":4444,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:33:03.186494+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":291241984,"num_objects":4444,"num_object_clones":0,"num_object_copies":13332,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4444,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4444,"num_write_kb":284416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,7],"acting":[0,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":66,"reported_epoch":174,"state":"active+clean","last_fresh":"2024-03-11T21:32:41.937249+0000","last_change":"2024-03-11T21:32:22.798321+0000","last_active":"2024-03-11T21:32:41.937249+0000","last_peered":"2024-03-11T21:32:41.937249+0000","last_clean":"2024-03-11T21:32:41.937249+0000","last_became_active":"2024-03-11T21:32:22.797849+0000","last_became_peered":"2024-03-11T21:32:22.797849+0000","last_unstale":"2024-03-11T21:32:41.937249+0000","last_undegraded":"2024-03-11T21:32:41.937249+0000","last_fullsized":"2024-03-11T21:32:41.937249+0000","mapping_epoch":166,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":167,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:31:58.450208+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:28:37.561826+0000","last_clean_scrub_stamp":"2024-03-11T21:31:58.450208+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:33:06.824763+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,6],"acting":[2,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":251,"reported_epoch":174,"state":"active+clean","last_fresh":"2024-03-11T21:32:41.937792+0000","last_change":"2024-03-11T21:32:23.251510+0000","last_active":"2024-03-11T21:32:41.937792+0000","last_peered":"2024-03-11T21:32:41.937792+0000","last_clean":"2024-03-11T21:32:41.937792+0000","last_became_active":"2024-03-11T21:25:42.574515+0000","last_became_peered":"2024-03-11T21:25:42.574515+0000","last_unstale":"2024-03-11T21:32:41.937792+0000","last_undegraded":"2024-03-11T21:32:41.937792+0000","last_fullsized":"2024-03-11T21:32:41.937792+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:32:23.251386+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:41.544550+0000","last_clean_scrub_stamp":"2024-03-11T21:32:23.251386+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:33:40.770572+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,5],"acting":[2,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.c","version":"0'0","reported_seq":251,"reported_epoch":174,"state":"active+clean","last_fresh":"2024-03-11T21:32:41.939229+0000","last_change":"2024-03-11T21:31:09.381415+0000","last_active":"2024-03-11T21:32:41.939229+0000","last_peered":"2024-03-11T21:32:41.939229+0000","last_clean":"2024-03-11T21:32:41.939229+0000","last_became_active":"2024-03-11T21:25:38.533240+0000","last_became_peered":"2024-03-11T21:25:38.533240+0000","last_unstale":"2024-03-11T21:32:41.939229+0000","last_undegraded":"2024-03-11T21:32:41.939229+0000","last_fullsized":"2024-03-11T21:32:41.939229+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:31:09.381301+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:29:30.279910+0000","last_clean_scrub_stamp":"2024-03-11T21:31:09.381301+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,0],"acting":[2,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.a","version":"109'28","reported_seq":322,"reported_epoch":174,"state":"active+clean","last_fresh":"2024-03-11T21:32:41.924122+0000","last_change":"2024-03-11T21:31:48.257526+0000","last_active":"2024-03-11T21:32:41.924122+0000","last_peered":"2024-03-11T21:32:41.924122+0000","last_clean":"2024-03-11T21:32:41.924122+0000","last_became_active":"2024-03-11T21:31:48.226789+0000","last_became_peered":"2024-03-11T21:31:48.226789+0000","last_unstale":"2024-03-11T21:32:41.924122+0000","last_undegraded":"2024-03-11T21:32:41.924122+0000","last_fullsized":"2024-03-11T21:32:41.924122+0000","mapping_epoch":153,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":154,"parent":"0.0","parent_split_bits":0,"last_scrub":"109'28","last_scrub_stamp":"2024-03-11T21:31:22.522676+0000","last_deep_scrub":"109'28","last_deep_scrub_stamp":"2024-03-11T21:28:24.016389+0000","last_clean_scrub_stamp":"2024-03-11T21:31:22.522676+0000","objects_scrubbed":9,"log_size":28,"log_dups_size":0,"ondisk_log_size":28,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:32:43.941466+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":32,"num_read_kb":21,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,5],"acting":[6,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"8.7","version":"174'4423","reported_seq":4468,"reported_epoch":174,"state":"active+clean","last_fresh":"2024-03-11T21:32:42.133663+0000","last_change":"2024-03-11T21:31:52.240303+0000","last_active":"2024-03-11T21:32:42.133663+0000","last_peered":"2024-03-11T21:32:42.133663+0000","last_clean":"2024-03-11T21:32:42.133663+0000","last_became_active":"2024-03-11T21:31:52.239980+0000","last_became_peered":"2024-03-11T21:31:52.239980+0000","last_unstale":"2024-03-11T21:32:42.133663+0000","last_undegraded":"2024-03-11T21:32:42.133663+0000","last_fullsized":"2024-03-11T21:32:42.133663+0000","mapping_epoch":157,"log_start":"0'0","ondisk_log_start":"0'0","created":157,"last_epoch_clean":158,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:31:51.219688+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:31:51.219688+0000","last_clean_scrub_stamp":"2024-03-11T21:31:51.219688+0000","objects_scrubbed":0,"log_size":4423,"log_dups_size":0,"ondisk_log_size":4423,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:32:56.737271+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":289865728,"num_objects":4423,"num_object_clones":0,"num_object_copies":13269,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4423,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4423,"num_write_kb":283072,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,6],"acting":[1,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":246,"reported_epoch":174,"state":"active+clean","last_fresh":"2024-03-11T21:32:41.919553+0000","last_change":"2024-03-11T21:31:48.217329+0000","last_active":"2024-03-11T21:32:41.919553+0000","last_peered":"2024-03-11T21:32:41.919553+0000","last_clean":"2024-03-11T21:32:41.919553+0000","last_became_active":"2024-03-11T21:31:48.216970+0000","last_became_peered":"2024-03-11T21:31:48.216970+0000","last_unstale":"2024-03-11T21:32:41.919553+0000","last_undegraded":"2024-03-11T21:32:41.919553+0000","last_fullsized":"2024-03-11T21:32:41.919553+0000","mapping_epoch":153,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":154,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:31:10.397515+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:27:11.361913+0000","last_clean_scrub_stamp":"2024-03-11T21:31:10.397515+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:32:44.469312+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,3],"acting":[0,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":243,"reported_epoch":174,"state":"active+clean","last_fresh":"2024-03-11T21:32:41.939774+0000","last_change":"2024-03-11T21:31:11.390022+0000","last_active":"2024-03-11T21:32:41.939774+0000","last_peered":"2024-03-11T21:32:41.939774+0000","last_clean":"2024-03-11T21:32:41.939774+0000","last_became_active":"2024-03-11T21:25:42.576679+0000","last_became_peered":"2024-03-11T21:25:42.576679+0000","last_unstale":"2024-03-11T21:32:41.939774+0000","last_undegraded":"2024-03-11T21:32:41.939774+0000","last_fullsized":"2024-03-11T21:32:41.939774+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:31:11.389869+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:41.544550+0000","last_clean_scrub_stamp":"2024-03-11T21:31:11.389869+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,3],"acting":[2,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.d","version":"0'0","reported_seq":260,"reported_epoch":174,"state":"active+clean","last_fresh":"2024-03-11T21:32:41.929483+0000","last_change":"2024-03-11T21:31:55.464241+0000","last_active":"2024-03-11T21:32:41.929483+0000","last_peered":"2024-03-11T21:32:41.929483+0000","last_clean":"2024-03-11T21:32:41.929483+0000","last_became_active":"2024-03-11T21:25:38.539681+0000","last_became_peered":"2024-03-11T21:25:38.539681+0000","last_unstale":"2024-03-11T21:32:41.929483+0000","last_undegraded":"2024-03-11T21:32:41.929483+0000","last_fullsized":"2024-03-11T21:32:41.929483+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:31:55.464145+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:37.512018+0000","last_clean_scrub_stamp":"2024-03-11T21:31:55.464145+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:33:06.998087+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,3],"acting":[1,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.b","version":"74'9","reported_seq":264,"reported_epoch":174,"state":"active+clean","last_fresh":"2024-03-11T21:32:41.918849+0000","last_change":"2024-03-11T21:31:20.485241+0000","last_active":"2024-03-11T21:32:41.918849+0000","last_peered":"2024-03-11T21:32:41.918849+0000","last_clean":"2024-03-11T21:32:41.918849+0000","last_became_active":"2024-03-11T21:25:40.549612+0000","last_became_peered":"2024-03-11T21:25:40.549612+0000","last_unstale":"2024-03-11T21:32:41.918849+0000","last_undegraded":"2024-03-11T21:32:41.918849+0000","last_fullsized":"2024-03-11T21:32:41.918849+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'9","last_scrub_stamp":"2024-03-11T21:31:20.485090+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:39.525554+0000","last_clean_scrub_stamp":"2024-03-11T21:31:20.485090+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.4","version":"174'4393","reported_seq":4438,"reported_epoch":174,"state":"active+clean","last_fresh":"2024-03-11T21:32:42.030268+0000","last_change":"2024-03-11T21:31:52.272663+0000","last_active":"2024-03-11T21:32:42.030268+0000","last_peered":"2024-03-11T21:32:42.030268+0000","last_clean":"2024-03-11T21:32:42.030268+0000","last_became_active":"2024-03-11T21:31:52.272229+0000","last_became_peered":"2024-03-11T21:31:52.272229+0000","last_unstale":"2024-03-11T21:32:42.030268+0000","last_undegraded":"2024-03-11T21:32:42.030268+0000","last_fullsized":"2024-03-11T21:32:42.030268+0000","mapping_epoch":157,"log_start":"0'0","ondisk_log_start":"0'0","created":157,"last_epoch_clean":158,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:31:51.219688+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:31:51.219688+0000","last_clean_scrub_stamp":"2024-03-11T21:31:51.219688+0000","objects_scrubbed":0,"log_size":4393,"log_dups_size":0,"ondisk_log_size":4393,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:33:03.474638+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":287899648,"num_objects":4393,"num_object_clones":0,"num_object_copies":13179,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4393,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4393,"num_write_kb":281152,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,2],"acting":[5,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":152,"reported_epoch":174,"state":"active+clean","last_fresh":"2024-03-11T21:32:41.913382+0000","last_change":"2024-03-11T21:32:18.627466+0000","last_active":"2024-03-11T21:32:41.913382+0000","last_peered":"2024-03-11T21:32:41.913382+0000","last_clean":"2024-03-11T21:32:41.913382+0000","last_became_active":"2024-03-11T21:28:45.372452+0000","last_became_peered":"2024-03-11T21:28:45.372452+0000","last_unstale":"2024-03-11T21:32:41.913382+0000","last_undegraded":"2024-03-11T21:32:41.913382+0000","last_fullsized":"2024-03-11T21:32:41.913382+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:32:18.627290+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:43.559452+0000","last_clean_scrub_stamp":"2024-03-11T21:32:18.627290+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:33:37.229242+0000","scrub_duration":29,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,3],"acting":[4,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.e","version":"0'0","reported_seq":265,"reported_epoch":174,"state":"active+clean","last_fresh":"2024-03-11T21:32:41.940867+0000","last_change":"2024-03-11T21:31:48.232984+0000","last_active":"2024-03-11T21:32:41.940867+0000","last_peered":"2024-03-11T21:32:41.940867+0000","last_clean":"2024-03-11T21:32:41.940867+0000","last_became_active":"2024-03-11T21:31:48.232420+0000","last_became_peered":"2024-03-11T21:31:48.232420+0000","last_unstale":"2024-03-11T21:32:41.940867+0000","last_undegraded":"2024-03-11T21:32:41.940867+0000","last_fullsized":"2024-03-11T21:32:41.940867+0000","mapping_epoch":153,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":154,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:31:17.408572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:37.512018+0000","last_clean_scrub_stamp":"2024-03-11T21:31:17.408572+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:32:45.134639+0000","scrub_duration":21,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,6],"acting":[2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.8","version":"109'22","reported_seq":309,"reported_epoch":174,"state":"active+clean","last_fresh":"2024-03-11T21:32:41.927156+0000","last_change":"2024-03-11T21:32:07.573285+0000","last_active":"2024-03-11T21:32:41.927156+0000","last_peered":"2024-03-11T21:32:41.927156+0000","last_clean":"2024-03-11T21:32:41.927156+0000","last_became_active":"2024-03-11T21:31:48.231316+0000","last_became_peered":"2024-03-11T21:31:48.231316+0000","last_unstale":"2024-03-11T21:32:41.927156+0000","last_undegraded":"2024-03-11T21:32:41.927156+0000","last_fullsized":"2024-03-11T21:32:41.927156+0000","mapping_epoch":153,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":154,"parent":"0.0","parent_split_bits":0,"last_scrub":"109'22","last_scrub_stamp":"2024-03-11T21:32:07.573197+0000","last_deep_scrub":"109'22","last_deep_scrub_stamp":"2024-03-11T21:32:07.573197+0000","last_clean_scrub_stamp":"2024-03-11T21:32:07.573197+0000","objects_scrubbed":7,"log_size":22,"log_dups_size":0,"ondisk_log_size":22,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:33:13.319438+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,4],"acting":[5,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.9","version":"154'10","reported_seq":153,"reported_epoch":174,"state":"active+clean","last_fresh":"2024-03-11T21:32:41.924721+0000","last_change":"2024-03-11T21:31:48.246094+0000","last_active":"2024-03-11T21:32:41.924721+0000","last_peered":"2024-03-11T21:32:41.924721+0000","last_clean":"2024-03-11T21:32:41.924721+0000","last_became_active":"2024-03-11T21:31:48.223454+0000","last_became_peered":"2024-03-11T21:31:48.223454+0000","last_unstale":"2024-03-11T21:32:41.924721+0000","last_undegraded":"2024-03-11T21:32:41.924721+0000","last_fullsized":"2024-03-11T21:32:41.924721+0000","mapping_epoch":153,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":154,"parent":"0.0","parent_split_bits":0,"last_scrub":"94'9","last_scrub_stamp":"2024-03-11T21:31:19.528840+0000","last_deep_scrub":"94'9","last_deep_scrub_stamp":"2024-03-11T21:26:53.422552+0000","last_clean_scrub_stamp":"2024-03-11T21:31:19.528840+0000","objects_scrubbed":1,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:32:49.277556+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]}],"pool_stats":[{"poolid":8,"num_pg":16,"stat_sum":{"num_bytes":4701880320,"num_objects":71745,"num_object_clones":0,"num_object_copies":215235,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":71745,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":71745,"num_write_kb":4591680,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":14097973248,"data_stored":14097973248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":71745,"ondisk_log_size":71745,"up":48,"acting":48,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":1531,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":24,"num_read_kb":24,"num_write":10,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":436,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":4593,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":6,"ondisk_log_size":6,"up":96,"acting":96,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":198,"num_read_kb":294,"num_write":179,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":2361472,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1781760,"data_stored":1771104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":128,"ondisk_log_size":128,"up":3,"acting":3,"num_store_stats":4},{"poolid":3,"num_pg":3,"stat_sum":{"num_bytes":408,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":417,"num_read_kb":412,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1224,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":8,"ondisk_log_size":8,"up":9,"acting":9,"num_store_stats":7},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":3702,"num_objects":178,"num_object_clones":0,"num_object_copies":534,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":178,"num_whiteouts":0,"num_read":698,"num_read_kb":455,"num_write":417,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":84,"num_bytes_recovered":1430,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":417792,"data_stored":11106,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":472,"ondisk_log_size":472,"up":96,"acting":96,"num_store_stats":8},{"poolid":5,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":71,"ondisk_log_size":71,"up":96,"acting":96,"num_store_stats":8},{"poolid":6,"num_pg":32,"stat_sum":{"num_bytes":383,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":370,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1149,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":96,"acting":96,"num_store_stats":8}],"osd_stats":[{"osd":7,"up_from":60,"seq":257698037848,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":404568,"kb_used_data":278992,"kb_used_omap":1,"kb_used_meta":125566,"kb_avail":93340584,"statfs":{"total":95995035648,"available":95580758016,"internally_reserved":0,"allocated":285687808,"data_stored":285268119,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":128580042},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":50,"seq":214748364894,"num_pgs":57,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1875296,"kb_used_data":1709460,"kb_used_omap":1,"kb_used_meta":165822,"kb_avail":91869856,"statfs":{"total":95995035648,"available":94074732544,"internally_reserved":0,"allocated":1750487040,"data_stored":1750008010,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":169802182},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574968,"num_pgs":67,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2131512,"kb_used_data":1965680,"kb_used_omap":1,"kb_used_meta":165822,"kb_avail":91613640,"statfs":{"total":95995035648,"available":93812367360,"internally_reserved":0,"allocated":2012856320,"data_stored":2012341618,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":169802181},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738493,"num_pgs":65,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2586148,"kb_used_data":2316120,"kb_used_omap":1,"kb_used_meta":269950,"kb_avail":91159004,"statfs":{"total":95995035648,"available":93346820096,"internally_reserved":0,"allocated":2371706880,"data_stored":2371223242,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":276429253},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411442,"num_pgs":54,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1857856,"kb_used_data":1696184,"kb_used_omap":1,"kb_used_meta":161662,"kb_avail":91887296,"statfs":{"total":95995035648,"available":94092591104,"internally_reserved":0,"allocated":1736892416,"data_stored":1736433408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":165542341},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":3,"up_from":26,"seq":111669149806,"num_pgs":67,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1971752,"kb_used_data":1697632,"kb_used_omap":1,"kb_used_meta":274110,"kb_avail":91773400,"statfs":{"total":95995035648,"available":93975961600,"internally_reserved":0,"allocated":1738375168,"data_stored":1737876570,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":280689095},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":4,"up_from":32,"seq":137438953575,"num_pgs":64,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1574232,"kb_used_data":1430608,"kb_used_omap":1,"kb_used_meta":143614,"kb_avail":92170920,"statfs":{"total":95995035648,"available":94383022080,"internally_reserved":0,"allocated":1464942592,"data_stored":1464459292,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":147061189},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":41,"seq":176093659234,"num_pgs":62,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2658312,"kb_used_data":2541056,"kb_used_omap":1,"kb_used_meta":117246,"kb_avail":91086840,"statfs":{"total":95995035648,"available":93272924160,"internally_reserved":0,"allocated":2602041344,"data_stored":2601574305,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":120060357},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1049,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":46,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":957,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":574,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":436,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1439,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":92,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":1430,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":98304,"data_stored":2558,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":1128,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":1898,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":1584,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":1210,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":1144,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":154,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":370,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":383,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":383,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":2384723968,"data_stored":2384723968,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":2039283712,"data_stored":2039283712,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":1760690176,"data_stored":1760690176,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":1768161280,"data_stored":1768161280,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":1471545344,"data_stored":1471545344,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":2632581120,"data_stored":2632581120,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":1753350144,"data_stored":1753350144,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":287637504,"data_stored":287637504,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-11T21:32:45.207 INFO:tasks.cephadm.ceph_manager.ceph:increase pool size by 10 2024-03-11T21:32:45.208 DEBUG:teuthology.orchestra.run.smithi092:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid 00db0016-dfed-11ee-95c8-87774f69a715 -- ceph osd pool set unique_pool_1 pg_num 26 2024-03-11T21:32:45.212 DEBUG:teuthology.orchestra.run.smithi092:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid 00db0016-dfed-11ee-95c8-87774f69a715 -- ceph osd unset noscrub 2024-03-11T21:32:45.216 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:32:45.301 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:32:45.302 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:32:45.302 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:32:45.403 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:32:45.433 INFO:tasks.radosbench.radosbench.0.smithi092.stdout: 44 16 75580 75564 107.316 75.1875 0.0189598 0.00931162 2024-03-11T21:32:45.480 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:32:45.481 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:32:45.482 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:32:45.585 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:32:45.623 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:32:45.624 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:32:45.624 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:32:45.725 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:32:45.754 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:32:45.755 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:32:45.756 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:32:45.856 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:32:45.890 INFO:teuthology.orchestra.run.smithi092.stderr:Inferring config /var/lib/ceph/00db0016-dfed-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T21:32:45.903 INFO:teuthology.orchestra.run.smithi092.stdout:osd.0: {} 2024-03-11T21:32:45.920 INFO:teuthology.orchestra.run.smithi092.stderr:Inferring config /var/lib/ceph/00db0016-dfed-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T21:32:45.932 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:32:45.933 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:32:45.933 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:32:45.957 INFO:teuthology.orchestra.run.smithi092.stdout:osd.1: {} 2024-03-11T21:32:46.000 INFO:teuthology.orchestra.run.smithi092.stdout:osd.2: {} 2024-03-11T21:32:46.000 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:32:45 smithi092 ceph-mon[29572]: from='client.35060 v1:172.21.15.92:0/3405451087' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-11T21:32:46.000 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:32:45 smithi092 ceph-mon[29572]: from='client.? v1:172.21.15.92:0/1177665681' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-11T21:32:46.001 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:32:45 smithi092 ceph-mon[37563]: from='client.35060 v1:172.21.15.92:0/3405451087' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-11T21:32:46.001 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:32:45 smithi092 ceph-mon[37563]: from='client.? v1:172.21.15.92:0/1177665681' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-11T21:32:46.034 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:32:46.039 INFO:teuthology.orchestra.run.smithi092.stdout:osd.3: {} 2024-03-11T21:32:46.076 INFO:teuthology.orchestra.run.smithi092.stdout:osd.4: {} 2024-03-11T21:32:46.082 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:32:46.082 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:32:46.083 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:32:46.112 INFO:teuthology.orchestra.run.smithi092.stdout:osd.5: {} 2024-03-11T21:32:46.149 INFO:teuthology.orchestra.run.smithi092.stdout:osd.6: {} 2024-03-11T21:32:46.186 INFO:teuthology.orchestra.run.smithi092.stdout:osd.7: {} 2024-03-11T21:32:46.187 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:32:46.195 INFO:teuthology.orchestra.run.smithi092.stderr:osd.0: osd_enable_op_tracker = '' 2024-03-11T21:32:46.195 INFO:teuthology.orchestra.run.smithi092.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-11T21:32:46.195 INFO:teuthology.orchestra.run.smithi092.stderr:osd.2: osd_enable_op_tracker = '' 2024-03-11T21:32:46.195 INFO:teuthology.orchestra.run.smithi092.stderr:osd.3: osd_enable_op_tracker = '' 2024-03-11T21:32:46.196 INFO:teuthology.orchestra.run.smithi092.stderr:osd.4: osd_enable_op_tracker = '' 2024-03-11T21:32:46.196 INFO:teuthology.orchestra.run.smithi092.stderr:osd.5: osd_enable_op_tracker = '' 2024-03-11T21:32:46.196 INFO:teuthology.orchestra.run.smithi092.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-11T21:32:46.196 INFO:teuthology.orchestra.run.smithi092.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-11T21:32:46.208 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:32:45 smithi097 ceph-mon[33317]: from='client.35060 v1:172.21.15.92:0/3405451087' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-11T21:32:46.209 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:32:45 smithi097 ceph-mon[33317]: from='client.? v1:172.21.15.92:0/1177665681' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-11T21:32:46.230 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:32:46.232 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:32:46.233 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:32:46.333 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:32:46.376 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:32:46.377 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:32:46.377 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:32:46.434 INFO:tasks.radosbench.radosbench.0.smithi092.stdout: 45 16 76637 76621 106.399 66.0625 0.0135965 0.00939288 2024-03-11T21:32:46.478 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:32:46.513 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:32:46.513 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:32:46.514 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:32:46.615 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:32:46.649 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:32:46.650 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:32:46.651 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:32:46.751 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:32:46.790 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:32:46.791 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:32:46.791 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:32:46.892 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:32:46.929 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:32:46.929 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:32:46.930 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:32:46.937 INFO:teuthology.orchestra.run.smithi097.stdout:{ 2024-03-11T21:32:46.937 INFO:teuthology.orchestra.run.smithi097.stdout: "size": 20, 2024-03-11T21:32:46.937 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 600, 2024-03-11T21:32:46.937 INFO:teuthology.orchestra.run.smithi097.stdout: "ops": [ 2024-03-11T21:32:46.937 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:32:46.937 INFO:teuthology.orchestra.run.smithi097.stdout: "description": "osd_op(client.34715.0:16783 7.1 7:8a03f820:::benchmark_data_smithi092_76040_object16782:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e105)", 2024-03-11T21:32:46.937 INFO:teuthology.orchestra.run.smithi097.stdout: "initiated_at": "2024-03-11T21:27:50.452605+0000", 2024-03-11T21:32:46.937 INFO:teuthology.orchestra.run.smithi097.stdout: "age": 296.49652378500002, 2024-03-11T21:32:46.937 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.067317150000000006, 2024-03-11T21:32:46.937 INFO:teuthology.orchestra.run.smithi097.stdout: "type_data": { 2024-03-11T21:32:46.937 INFO:teuthology.orchestra.run.smithi097.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-11T21:32:46.937 INFO:teuthology.orchestra.run.smithi097.stdout: "client_info": { 2024-03-11T21:32:46.937 INFO:teuthology.orchestra.run.smithi097.stdout: "client": "client.34715", 2024-03-11T21:32:46.938 INFO:teuthology.orchestra.run.smithi097.stdout: "client_addr": "v1:172.21.15.92:0/2403132573", 2024-03-11T21:32:46.938 INFO:teuthology.orchestra.run.smithi097.stdout: "tid": 16783 2024-03-11T21:32:46.938 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:32:46.938 INFO:teuthology.orchestra.run.smithi097.stdout: "events": [ 2024-03-11T21:32:46.938 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:32:46.938 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "initiated", 2024-03-11T21:32:46.938 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:27:50.452605+0000", 2024-03-11T21:32:46.938 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-03-11T21:32:46.938 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:32:46.938 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:32:46.938 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "header_read", 2024-03-11T21:32:46.938 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:27:50.452605+0000", 2024-03-11T21:32:46.938 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-03-11T21:32:46.939 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:32:46.939 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:32:46.939 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "throttled", 2024-03-11T21:32:46.939 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:27:50.452607+0000", 2024-03-11T21:32:46.939 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.6109999999999999e-06 2024-03-11T21:32:46.939 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:32:46.939 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:32:46.939 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "all_read", 2024-03-11T21:32:46.939 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:27:50.452667+0000", 2024-03-11T21:32:46.939 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 6.0016999999999998e-05 2024-03-11T21:32:46.939 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:32:46.939 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:32:46.939 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "dispatched", 2024-03-11T21:32:46.939 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:27:50.452676+0000", 2024-03-11T21:32:46.940 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 9.5599999999999999e-06 2024-03-11T21:32:46.940 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:32:46.940 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:32:46.940 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "queued_for_pg", 2024-03-11T21:32:46.940 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:27:50.452684+0000", 2024-03-11T21:32:46.940 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 7.8569999999999995e-06 2024-03-11T21:32:46.940 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:32:46.940 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:32:46.940 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "reached_pg", 2024-03-11T21:32:46.940 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:27:50.452754+0000", 2024-03-11T21:32:46.940 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 6.9499000000000007e-05 2024-03-11T21:32:46.940 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:32:46.940 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:32:46.940 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "started", 2024-03-11T21:32:46.941 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:27:50.452866+0000", 2024-03-11T21:32:46.941 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.000112749 2024-03-11T21:32:46.941 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:32:46.941 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:32:46.941 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "waiting for subops from 3,6", 2024-03-11T21:32:46.941 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:27:50.452997+0000", 2024-03-11T21:32:46.941 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.00013031800000000001 2024-03-11T21:32:46.941 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:32:46.941 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:32:46.941 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "op_commit", 2024-03-11T21:32:46.941 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:27:50.453760+0000", 2024-03-11T21:32:46.941 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.00076315599999999997 2024-03-11T21:32:46.941 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:32:46.941 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:32:46.942 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "sub_op_commit_rec", 2024-03-11T21:32:46.942 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:27:50.453961+0000", 2024-03-11T21:32:46.942 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.00020143 2024-03-11T21:32:46.942 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:32:46.942 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:32:46.942 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "sub_op_commit_rec", 2024-03-11T21:32:46.943 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:27:50.519846+0000", 2024-03-11T21:32:46.943 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.065884531999999996 2024-03-11T21:32:46.943 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:32:46.943 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:32:46.943 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "commit_sent", 2024-03-11T21:32:46.943 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:27:50.519883+0000", 2024-03-11T21:32:46.943 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 3.7533000000000001e-05 2024-03-11T21:32:46.943 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:32:46.943 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:32:46.944 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "done", 2024-03-11T21:32:46.944 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:27:50.519922+0000", 2024-03-11T21:32:46.944 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 3.8887999999999998e-05 2024-03-11T21:32:46.944 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-03-11T21:32:46.944 INFO:teuthology.orchestra.run.smithi097.stdout: ] 2024-03-11T21:32:46.944 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-03-11T21:32:46.944 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:32:46.944 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:32:46.944 INFO:teuthology.orchestra.run.smithi097.stdout: "description": "osd_op(client.34715.0:16787 7.1 7:8daed005:::benchmark_data_smithi092_76040_object16786:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e105)", 2024-03-11T21:32:46.944 INFO:teuthology.orchestra.run.smithi097.stdout: "initiated_at": "2024-03-11T21:27:50.455699+0000", 2024-03-11T21:32:46.944 INFO:teuthology.orchestra.run.smithi097.stdout: "age": 296.49342957699997, 2024-03-11T21:32:46.944 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.067379470999999996, 2024-03-11T21:32:46.944 INFO:teuthology.orchestra.run.smithi097.stdout: "type_data": { 2024-03-11T21:32:46.945 INFO:teuthology.orchestra.run.smithi097.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-11T21:32:46.945 INFO:teuthology.orchestra.run.smithi097.stdout: "client_info": { 2024-03-11T21:32:46.945 INFO:teuthology.orchestra.run.smithi097.stdout: "client": "client.34715", 2024-03-11T21:32:46.945 INFO:teuthology.orchestra.run.smithi097.stdout: "client_addr": "v1:172.21.15.92:0/2403132573", 2024-03-11T21:32:46.945 INFO:teuthology.orchestra.run.smithi097.stdout: "tid": 16787 2024-03-11T21:32:46.945 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:32:46.945 INFO:teuthology.orchestra.run.smithi097.stdout: "events": [ 2024-03-11T21:32:46.945 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:32:46.945 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "initiated", 2024-03-11T21:32:46.945 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:27:50.455699+0000", 2024-03-11T21:32:46.946 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-03-11T21:32:46.946 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:32:46.946 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:32:46.946 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "header_read", 2024-03-11T21:32:46.946 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:27:50.455699+0000", 2024-03-11T21:32:46.946 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-03-11T21:32:46.946 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:32:46.946 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:32:46.946 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "throttled", 2024-03-11T21:32:46.946 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:27:50.455701+0000", 2024-03-11T21:32:46.946 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.42e-06 2024-03-11T21:32:46.946 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:32:46.946 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:32:46.946 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "all_read", 2024-03-11T21:32:46.946 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:27:50.455758+0000", 2024-03-11T21:32:46.947 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 5.7346999999999998e-05 2024-03-11T21:32:46.947 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:32:46.947 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:32:46.947 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "dispatched", 2024-03-11T21:32:46.947 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:27:50.455769+0000", 2024-03-11T21:32:46.947 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.1122e-05 2024-03-11T21:32:46.947 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:32:46.947 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:32:46.947 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "queued_for_pg", 2024-03-11T21:32:46.947 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:27:50.455777+0000", 2024-03-11T21:32:46.947 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 7.6279999999999996e-06 2024-03-11T21:32:46.947 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:32:46.947 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:32:46.947 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "reached_pg", 2024-03-11T21:32:46.948 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:27:50.455823+0000", 2024-03-11T21:32:46.948 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 4.6659000000000002e-05 2024-03-11T21:32:46.948 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:32:46.948 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:32:46.948 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "started", 2024-03-11T21:32:46.948 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:27:50.455906+0000", 2024-03-11T21:32:46.948 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 8.2381999999999998e-05 2024-03-11T21:32:46.948 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:32:46.948 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:32:46.948 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "waiting for subops from 3,6", 2024-03-11T21:32:46.948 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:27:50.456020+0000", 2024-03-11T21:32:46.948 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.000114207 2024-03-11T21:32:46.949 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:32:46.949 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:32:46.949 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "op_commit", 2024-03-11T21:32:46.949 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:27:50.456839+0000", 2024-03-11T21:32:46.949 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.00081941399999999997 2024-03-11T21:32:46.949 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:32:46.949 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:32:46.949 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "sub_op_commit_rec", 2024-03-11T21:32:46.949 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:27:50.457192+0000", 2024-03-11T21:32:46.949 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.00035303400000000001 2024-03-11T21:32:46.949 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:32:46.949 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:32:46.949 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "sub_op_commit_rec", 2024-03-11T21:32:46.949 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:27:50.522980+0000", 2024-03-11T21:32:46.950 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.065787727000000004 2024-03-11T21:32:46.950 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:32:46.950 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:32:46.950 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "commit_sent", 2024-03-11T21:32:46.950 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:27:50.523035+0000", 2024-03-11T21:32:46.950 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 5.5322999999999999e-05 2024-03-11T21:32:46.950 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:32:46.950 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:32:46.950 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "done", 2024-03-11T21:32:46.950 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:27:50.523079+0000", 2024-03-11T21:32:46.950 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 4.3207999999999997e-05 2024-03-11T21:32:46.950 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-03-11T21:32:46.950 INFO:teuthology.orchestra.run.smithi097.stdout: ] 2024-03-11T21:32:46.950 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-03-11T21:32:46.951 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:32:46.951 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:32:46.951 INFO:teuthology.orchestra.run.smithi097.stdout: "description": "osd_repop(client.34715.0:32056 7.9 e105/99 7:94ee601a:::benchmark_data_smithi092_76040_object32055:head v 105'2003, mlcod=105'2001)", 2024-03-11T21:32:46.951 INFO:teuthology.orchestra.run.smithi097.stdout: "initiated_at": "2024-03-11T21:27:57.787048+0000", 2024-03-11T21:32:46.951 INFO:teuthology.orchestra.run.smithi097.stdout: "age": 289.16208063599998, 2024-03-11T21:32:46.951 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.075118210000000005, 2024-03-11T21:32:46.951 INFO:teuthology.orchestra.run.smithi097.stdout: "type_data": { 2024-03-11T21:32:46.951 INFO:teuthology.orchestra.run.smithi097.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-11T21:32:46.951 INFO:teuthology.orchestra.run.smithi097.stdout: "events": [ 2024-03-11T21:32:46.951 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:32:46.951 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "initiated", 2024-03-11T21:32:46.951 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:27:57.787048+0000", 2024-03-11T21:32:46.951 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-03-11T21:32:46.951 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:32:46.952 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:32:46.952 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "header_read", 2024-03-11T21:32:46.952 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:27:57.787048+0000", 2024-03-11T21:32:46.952 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-03-11T21:32:46.952 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:32:46.952 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:32:46.952 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "throttled", 2024-03-11T21:32:46.952 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:27:57.787049+0000", 2024-03-11T21:32:46.952 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.336e-06 2024-03-11T21:32:46.952 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:32:46.952 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:32:46.952 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "all_read", 2024-03-11T21:32:46.952 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:27:57.787109+0000", 2024-03-11T21:32:46.952 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 5.9802000000000001e-05 2024-03-11T21:32:46.953 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:32:46.953 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:32:46.953 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "dispatched", 2024-03-11T21:32:46.953 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:27:57.787119+0000", 2024-03-11T21:32:46.953 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.0086e-05 2024-03-11T21:32:46.953 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:32:46.953 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:32:46.953 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "queued_for_pg", 2024-03-11T21:32:46.953 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:27:57.787127+0000", 2024-03-11T21:32:46.953 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 7.1879999999999996e-06 2024-03-11T21:32:46.953 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:32:46.953 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:32:46.953 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "reached_pg", 2024-03-11T21:32:46.953 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:27:57.787181+0000", 2024-03-11T21:32:46.953 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 5.4469999999999999e-05 2024-03-11T21:32:46.954 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:32:46.954 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:32:46.954 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "started", 2024-03-11T21:32:46.954 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:27:57.787200+0000", 2024-03-11T21:32:46.954 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.9247000000000001e-05 2024-03-11T21:32:46.954 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:32:46.954 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:32:46.954 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "commit_sent", 2024-03-11T21:32:46.955 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:27:57.788472+0000", 2024-03-11T21:32:46.955 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.0012718390000000001 2024-03-11T21:32:46.955 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:32:46.955 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:32:46.955 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "done", 2024-03-11T21:32:46.955 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:27:57.862166+0000", 2024-03-11T21:32:46.955 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.073694241999999993 2024-03-11T21:32:46.955 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-03-11T21:32:46.955 INFO:teuthology.orchestra.run.smithi097.stdout: ] 2024-03-11T21:32:46.955 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-03-11T21:32:46.955 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:32:46.955 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:32:46.955 INFO:teuthology.orchestra.run.smithi097.stdout: "description": "osd_op(client.34715.0:32061 7.1 7:89d95748:::benchmark_data_smithi092_76040_object32060:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e105)", 2024-03-11T21:32:46.955 INFO:teuthology.orchestra.run.smithi097.stdout: "initiated_at": "2024-03-11T21:27:57.788025+0000", 2024-03-11T21:32:46.956 INFO:teuthology.orchestra.run.smithi097.stdout: "age": 289.16110375599999, 2024-03-11T21:32:46.956 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.077315224000000002, 2024-03-11T21:32:46.956 INFO:teuthology.orchestra.run.smithi097.stdout: "type_data": { 2024-03-11T21:32:46.956 INFO:teuthology.orchestra.run.smithi097.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-11T21:32:46.956 INFO:teuthology.orchestra.run.smithi097.stdout: "client_info": { 2024-03-11T21:32:46.956 INFO:teuthology.orchestra.run.smithi097.stdout: "client": "client.34715", 2024-03-11T21:32:46.956 INFO:teuthology.orchestra.run.smithi097.stdout: "client_addr": "v1:172.21.15.92:0/2403132573", 2024-03-11T21:32:46.956 INFO:teuthology.orchestra.run.smithi097.stdout: "tid": 32061 2024-03-11T21:32:46.956 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:32:46.956 INFO:teuthology.orchestra.run.smithi097.stdout: "events": [ 2024-03-11T21:32:46.957 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:32:46.957 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "initiated", 2024-03-11T21:32:46.957 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:27:57.788025+0000", 2024-03-11T21:32:46.957 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-03-11T21:32:46.957 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:32:46.957 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:32:46.957 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "header_read", 2024-03-11T21:32:46.957 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:27:57.788025+0000", 2024-03-11T21:32:46.957 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-03-11T21:32:46.957 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:32:46.957 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:32:46.958 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "throttled", 2024-03-11T21:32:46.958 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:27:57.788026+0000", 2024-03-11T21:32:46.958 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.336e-06 2024-03-11T21:32:46.958 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:32:46.958 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:32:46.958 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "all_read", 2024-03-11T21:32:46.958 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:27:57.788090+0000", 2024-03-11T21:32:46.958 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 6.4092999999999998e-05 2024-03-11T21:32:46.958 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:32:46.958 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:32:46.958 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "dispatched", 2024-03-11T21:32:46.958 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:27:57.788102+0000", 2024-03-11T21:32:46.958 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.1698999999999999e-05 2024-03-11T21:32:46.958 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:32:46.959 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:32:46.959 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "queued_for_pg", 2024-03-11T21:32:46.959 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:27:57.788119+0000", 2024-03-11T21:32:46.959 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.6439e-05 2024-03-11T21:32:46.959 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:32:46.959 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:32:46.959 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "reached_pg", 2024-03-11T21:32:46.959 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:27:57.788161+0000", 2024-03-11T21:32:46.959 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 4.2035e-05 2024-03-11T21:32:46.959 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:32:46.959 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:32:46.959 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "started", 2024-03-11T21:32:46.959 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:27:57.788259+0000", 2024-03-11T21:32:46.959 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 9.8090000000000004e-05 2024-03-11T21:32:46.959 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:32:46.960 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:32:46.960 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "waiting for subops from 3,6", 2024-03-11T21:32:46.960 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:27:57.788380+0000", 2024-03-11T21:32:46.960 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.000121737 2024-03-11T21:32:46.960 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:32:46.960 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:32:46.960 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "sub_op_commit_rec", 2024-03-11T21:32:46.960 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:27:57.863304+0000", 2024-03-11T21:32:46.960 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.074923654000000006 2024-03-11T21:32:46.960 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:32:46.960 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:32:46.960 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "sub_op_commit_rec", 2024-03-11T21:32:46.960 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:27:57.863419+0000", 2024-03-11T21:32:46.960 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.000114957 2024-03-11T21:32:46.960 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:32:46.961 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:32:46.961 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "op_commit", 2024-03-11T21:32:46.961 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:27:57.865245+0000", 2024-03-11T21:32:46.961 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.001826102 2024-03-11T21:32:46.961 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:32:46.961 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:32:46.961 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "commit_sent", 2024-03-11T21:32:46.961 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:27:57.865297+0000", 2024-03-11T21:32:46.961 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 5.1529000000000001e-05 2024-03-11T21:32:46.961 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:32:46.961 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:32:46.961 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "done", 2024-03-11T21:32:46.961 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:27:57.865340+0000", 2024-03-11T21:32:46.961 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 4.3553000000000001e-05 2024-03-11T21:32:46.962 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-03-11T21:32:46.962 INFO:teuthology.orchestra.run.smithi097.stdout: ] 2024-03-11T21:32:46.962 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-03-11T21:32:46.962 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:32:46.962 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:32:46.962 INFO:teuthology.orchestra.run.smithi097.stdout: "description": "osd_op(client.34715.0:32063 7.1 7:8c2d6d23:::benchmark_data_smithi092_76040_object32062:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e105)", 2024-03-11T21:32:46.962 INFO:teuthology.orchestra.run.smithi097.stdout: "initiated_at": "2024-03-11T21:27:57.788779+0000", 2024-03-11T21:32:46.962 INFO:teuthology.orchestra.run.smithi097.stdout: "age": 289.160349381, 2024-03-11T21:32:46.962 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.079669661000000003, 2024-03-11T21:32:46.962 INFO:teuthology.orchestra.run.smithi097.stdout: "type_data": { 2024-03-11T21:32:46.962 INFO:teuthology.orchestra.run.smithi097.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-11T21:32:46.962 INFO:teuthology.orchestra.run.smithi097.stdout: "client_info": { 2024-03-11T21:32:46.962 INFO:teuthology.orchestra.run.smithi097.stdout: "client": "client.34715", 2024-03-11T21:32:46.962 INFO:teuthology.orchestra.run.smithi097.stdout: "client_addr": "v1:172.21.15.92:0/2403132573", 2024-03-11T21:32:46.963 INFO:teuthology.orchestra.run.smithi097.stdout: "tid": 32063 2024-03-11T21:32:46.963 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:32:46.963 INFO:teuthology.orchestra.run.smithi097.stdout: "events": [ 2024-03-11T21:32:46.963 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:32:46.963 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "initiated", 2024-03-11T21:32:46.963 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:27:57.788779+0000", 2024-03-11T21:32:46.963 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-03-11T21:32:46.963 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:32:46.963 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:32:46.963 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "header_read", 2024-03-11T21:32:46.963 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:27:57.788779+0000", 2024-03-11T21:32:46.963 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-03-11T21:32:46.963 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:32:46.963 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:32:46.963 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "throttled", 2024-03-11T21:32:46.964 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:27:57.788780+0000", 2024-03-11T21:32:46.964 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.0270000000000001e-06 2024-03-11T21:32:46.964 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:32:46.964 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:32:46.964 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "all_read", 2024-03-11T21:32:46.964 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:27:57.788872+0000", 2024-03-11T21:32:46.964 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 9.1278000000000003e-05 2024-03-11T21:32:46.964 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:32:46.964 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:32:46.965 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "dispatched", 2024-03-11T21:32:46.965 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:27:57.863322+0000", 2024-03-11T21:32:46.965 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.074450159000000002 2024-03-11T21:32:46.965 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:32:46.965 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:32:46.965 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "queued_for_pg", 2024-03-11T21:32:46.965 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:27:57.863331+0000", 2024-03-11T21:32:46.965 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 9.5389999999999996e-06 2024-03-11T21:32:46.965 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:32:46.965 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:32:46.965 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "reached_pg", 2024-03-11T21:32:46.965 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:27:57.863454+0000", 2024-03-11T21:32:46.965 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.00012228299999999999 2024-03-11T21:32:46.965 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:32:46.966 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:32:46.966 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "started", 2024-03-11T21:32:46.966 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:27:57.863601+0000", 2024-03-11T21:32:46.966 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.000147186 2024-03-11T21:32:46.966 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:32:46.966 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:32:46.967 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "waiting for subops from 3,6", 2024-03-11T21:32:46.967 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:27:57.863761+0000", 2024-03-11T21:32:46.967 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.00015977999999999999 2024-03-11T21:32:46.967 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:32:46.968 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:32:46.968 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "sub_op_commit_rec", 2024-03-11T21:32:46.968 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:27:57.866971+0000", 2024-03-11T21:32:46.968 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.0032101230000000001 2024-03-11T21:32:46.968 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:32:46.968 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:32:46.968 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "op_commit", 2024-03-11T21:32:46.968 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:27:57.866993+0000", 2024-03-11T21:32:46.968 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 2.1897000000000001e-05 2024-03-11T21:32:46.968 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:32:46.968 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:32:46.968 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "sub_op_commit_rec", 2024-03-11T21:32:46.968 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:27:57.868344+0000", 2024-03-11T21:32:46.968 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.001351798 2024-03-11T21:32:46.969 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:32:46.969 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:32:46.969 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "commit_sent", 2024-03-11T21:32:46.969 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:27:57.868402+0000", 2024-03-11T21:32:46.969 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 5.7862000000000003e-05 2024-03-11T21:32:46.969 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:32:46.969 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:32:46.969 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "done", 2024-03-11T21:32:46.969 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:27:57.868449+0000", 2024-03-11T21:32:46.969 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 4.6728999999999999e-05 2024-03-11T21:32:46.969 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-03-11T21:32:46.969 INFO:teuthology.orchestra.run.smithi097.stdout: ] 2024-03-11T21:32:46.969 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-03-11T21:32:46.969 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:32:46.969 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:32:46.970 INFO:teuthology.orchestra.run.smithi097.stdout: "description": "osd_repop_reply(client.34715.0:32061 7.1 e105/99 ondisk, result = 0)", 2024-03-11T21:32:46.970 INFO:teuthology.orchestra.run.smithi097.stdout: "initiated_at": "2024-03-11T21:27:57.789455+0000", 2024-03-11T21:32:46.970 INFO:teuthology.orchestra.run.smithi097.stdout: "age": 289.159674241, 2024-03-11T21:32:46.970 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.073857473000000007, 2024-03-11T21:32:46.970 INFO:teuthology.orchestra.run.smithi097.stdout: "type_data": { 2024-03-11T21:32:46.970 INFO:teuthology.orchestra.run.smithi097.stdout: "flag_point": "started", 2024-03-11T21:32:46.970 INFO:teuthology.orchestra.run.smithi097.stdout: "events": [ 2024-03-11T21:32:46.970 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:32:46.970 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "initiated", 2024-03-11T21:32:46.970 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:27:57.789455+0000", 2024-03-11T21:32:46.970 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-03-11T21:32:46.970 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:32:46.970 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:32:46.970 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "header_read", 2024-03-11T21:32:46.971 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:27:57.789455+0000", 2024-03-11T21:32:46.971 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-03-11T21:32:46.971 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:32:46.971 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:32:46.971 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "throttled", 2024-03-11T21:32:46.971 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:27:57.789455+0000", 2024-03-11T21:32:46.971 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 8.6600000000000005e-07 2024-03-11T21:32:46.971 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:32:46.971 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:32:46.971 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "all_read", 2024-03-11T21:32:46.971 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:27:57.789459+0000", 2024-03-11T21:32:46.971 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 3.41e-06 2024-03-11T21:32:46.971 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:32:46.971 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:32:46.971 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "dispatched", 2024-03-11T21:32:46.972 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:27:57.862118+0000", 2024-03-11T21:32:46.972 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.072659661 2024-03-11T21:32:46.972 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:32:46.972 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:32:46.972 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "queued_for_pg", 2024-03-11T21:32:46.972 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:27:57.862128+0000", 2024-03-11T21:32:46.972 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 9.5689999999999994e-06 2024-03-11T21:32:46.972 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:32:46.972 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:32:46.972 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "reached_pg", 2024-03-11T21:32:46.972 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:27:57.863288+0000", 2024-03-11T21:32:46.972 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.00116 2024-03-11T21:32:46.972 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:32:46.973 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:32:46.973 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "started", 2024-03-11T21:32:46.973 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:27:57.863298+0000", 2024-03-11T21:32:46.973 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 9.8439999999999993e-06 2024-03-11T21:32:46.973 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:32:46.973 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:32:46.973 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "done", 2024-03-11T21:32:46.973 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:27:57.863312+0000", 2024-03-11T21:32:46.973 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.4123e-05 2024-03-11T21:32:46.973 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-03-11T21:32:46.973 INFO:teuthology.orchestra.run.smithi097.stdout: ] 2024-03-11T21:32:46.973 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-03-11T21:32:46.973 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:32:46.973 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:32:46.973 INFO:teuthology.orchestra.run.smithi097.stdout: "description": "osd_repop_reply(client.34715.0:32061 7.1 e105/99 ondisk, result = 0)", 2024-03-11T21:32:46.974 INFO:teuthology.orchestra.run.smithi097.stdout: "initiated_at": "2024-03-11T21:27:57.790324+0000", 2024-03-11T21:32:46.974 INFO:teuthology.orchestra.run.smithi097.stdout: "age": 289.158804422, 2024-03-11T21:32:46.974 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.073103739000000001, 2024-03-11T21:32:46.974 INFO:teuthology.orchestra.run.smithi097.stdout: "type_data": { 2024-03-11T21:32:46.974 INFO:teuthology.orchestra.run.smithi097.stdout: "flag_point": "started", 2024-03-11T21:32:46.974 INFO:teuthology.orchestra.run.smithi097.stdout: "events": [ 2024-03-11T21:32:46.974 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:32:46.974 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "initiated", 2024-03-11T21:32:46.974 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:27:57.790324+0000", 2024-03-11T21:32:46.974 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-03-11T21:32:46.974 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:32:46.974 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:32:46.974 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "header_read", 2024-03-11T21:32:46.974 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:27:57.790324+0000", 2024-03-11T21:32:46.975 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-03-11T21:32:46.975 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:32:46.975 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:32:46.975 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "throttled", 2024-03-11T21:32:46.975 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:27:57.790325+0000", 2024-03-11T21:32:46.975 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 8.5300000000000003e-07 2024-03-11T21:32:46.975 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:32:46.975 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:32:46.975 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "all_read", 2024-03-11T21:32:46.975 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:27:57.790328+0000", 2024-03-11T21:32:46.975 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 3.2880000000000001e-06 2024-03-11T21:32:46.975 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:32:46.975 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:32:46.975 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "dispatched", 2024-03-11T21:32:46.975 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:27:57.862092+0000", 2024-03-11T21:32:46.976 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.071763847000000006 2024-03-11T21:32:46.976 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:32:46.976 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:32:46.976 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "queued_for_pg", 2024-03-11T21:32:46.976 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:27:57.862119+0000", 2024-03-11T21:32:46.976 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 2.6288e-05 2024-03-11T21:32:46.976 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:32:46.976 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:32:46.976 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "reached_pg", 2024-03-11T21:32:46.976 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:27:57.863405+0000", 2024-03-11T21:32:46.976 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.001286117 2024-03-11T21:32:46.976 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:32:46.976 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:32:46.976 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "started", 2024-03-11T21:32:46.976 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:27:57.863413+0000", 2024-03-11T21:32:46.977 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 8.1689999999999992e-06 2024-03-11T21:32:46.977 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:32:46.977 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:32:46.977 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "done", 2024-03-11T21:32:46.977 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:27:57.863428+0000", 2024-03-11T21:32:46.977 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.5177000000000001e-05 2024-03-11T21:32:46.977 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-03-11T21:32:46.977 INFO:teuthology.orchestra.run.smithi097.stdout: ] 2024-03-11T21:32:46.977 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-03-11T21:32:46.977 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:32:46.977 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:32:46.977 INFO:teuthology.orchestra.run.smithi097.stdout: "description": "osd_op(client.34715.0:70024 7.1 7:8cacfe50:::benchmark_data_smithi092_76040_object70023:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e109)", 2024-03-11T21:32:46.977 INFO:teuthology.orchestra.run.smithi097.stdout: "initiated_at": "2024-03-11T21:28:24.738322+0000", 2024-03-11T21:32:46.977 INFO:teuthology.orchestra.run.smithi097.stdout: "age": 262.21080674500001, 2024-03-11T21:32:46.977 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.069497423000000003, 2024-03-11T21:32:46.978 INFO:teuthology.orchestra.run.smithi097.stdout: "type_data": { 2024-03-11T21:32:46.978 INFO:teuthology.orchestra.run.smithi097.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-11T21:32:46.978 INFO:teuthology.orchestra.run.smithi097.stdout: "client_info": { 2024-03-11T21:32:46.978 INFO:teuthology.orchestra.run.smithi097.stdout: "client": "client.34715", 2024-03-11T21:32:46.978 INFO:teuthology.orchestra.run.smithi097.stdout: "client_addr": "v1:172.21.15.92:0/2403132573", 2024-03-11T21:32:46.978 INFO:teuthology.orchestra.run.smithi097.stdout: "tid": 70024 2024-03-11T21:32:46.978 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:32:46.978 INFO:teuthology.orchestra.run.smithi097.stdout: "events": [ 2024-03-11T21:32:46.978 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:32:46.978 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "initiated", 2024-03-11T21:32:46.978 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:28:24.738322+0000", 2024-03-11T21:32:46.978 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-03-11T21:32:46.978 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:32:46.978 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:32:46.979 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "header_read", 2024-03-11T21:32:46.979 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:28:24.738322+0000", 2024-03-11T21:32:46.979 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-03-11T21:32:46.979 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:32:46.979 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:32:46.979 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "throttled", 2024-03-11T21:32:46.979 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:28:24.738324+0000", 2024-03-11T21:32:46.979 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 2.2249999999999999e-06 2024-03-11T21:32:46.979 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:32:46.979 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:32:46.979 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "all_read", 2024-03-11T21:32:46.979 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:28:24.738384+0000", 2024-03-11T21:32:46.979 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 5.9314000000000002e-05 2024-03-11T21:32:46.979 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:32:46.980 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:32:46.980 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "dispatched", 2024-03-11T21:32:46.980 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:28:24.738394+0000", 2024-03-11T21:32:46.980 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.0644e-05 2024-03-11T21:32:46.980 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:32:46.980 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:32:46.980 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "queued_for_pg", 2024-03-11T21:32:46.980 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:28:24.738401+0000", 2024-03-11T21:32:46.980 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 6.9319999999999999e-06 2024-03-11T21:32:46.980 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:32:46.980 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:32:46.980 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "reached_pg", 2024-03-11T21:32:46.980 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:28:24.738460+0000", 2024-03-11T21:32:46.980 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 5.8434999999999999e-05 2024-03-11T21:32:46.981 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:32:46.981 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:32:46.981 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "started", 2024-03-11T21:32:46.981 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:28:24.738584+0000", 2024-03-11T21:32:46.981 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.00012423499999999999 2024-03-11T21:32:46.981 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:32:46.981 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:32:46.981 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "waiting for subops from 3,6", 2024-03-11T21:32:46.981 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:28:24.738712+0000", 2024-03-11T21:32:46.981 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.00012846400000000001 2024-03-11T21:32:46.981 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:32:46.981 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:32:46.981 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "sub_op_commit_rec", 2024-03-11T21:32:46.981 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:28:24.743450+0000", 2024-03-11T21:32:46.982 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.0047372250000000003 2024-03-11T21:32:46.982 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:32:46.982 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:32:46.982 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "op_commit", 2024-03-11T21:32:46.982 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:28:24.743472+0000", 2024-03-11T21:32:46.982 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 2.2925000000000002e-05 2024-03-11T21:32:46.982 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:32:46.982 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:32:46.982 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "sub_op_commit_rec", 2024-03-11T21:32:46.982 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:28:24.807679+0000", 2024-03-11T21:32:46.982 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.064206073000000002 2024-03-11T21:32:46.982 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:32:46.982 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:32:46.982 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "commit_sent", 2024-03-11T21:32:46.983 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:28:24.807784+0000", 2024-03-11T21:32:46.983 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.000105691 2024-03-11T21:32:46.983 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:32:46.983 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:32:46.983 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "done", 2024-03-11T21:32:46.983 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:28:24.807819+0000", 2024-03-11T21:32:46.983 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 3.5259999999999998e-05 2024-03-11T21:32:46.983 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-03-11T21:32:46.983 INFO:teuthology.orchestra.run.smithi097.stdout: ] 2024-03-11T21:32:46.983 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-03-11T21:32:46.983 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:32:46.983 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:32:46.983 INFO:teuthology.orchestra.run.smithi097.stdout: "description": "osd_op(client.34715.0:70026 7.1 7:866c3959:::benchmark_data_smithi092_76040_object70025:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e109)", 2024-03-11T21:32:46.983 INFO:teuthology.orchestra.run.smithi097.stdout: "initiated_at": "2024-03-11T21:28:24.738536+0000", 2024-03-11T21:32:46.984 INFO:teuthology.orchestra.run.smithi097.stdout: "age": 262.21059256299998, 2024-03-11T21:32:46.984 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.069385899000000001, 2024-03-11T21:32:46.984 INFO:teuthology.orchestra.run.smithi097.stdout: "type_data": { 2024-03-11T21:32:46.984 INFO:teuthology.orchestra.run.smithi097.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-11T21:32:46.984 INFO:teuthology.orchestra.run.smithi097.stdout: "client_info": { 2024-03-11T21:32:46.984 INFO:teuthology.orchestra.run.smithi097.stdout: "client": "client.34715", 2024-03-11T21:32:46.984 INFO:teuthology.orchestra.run.smithi097.stdout: "client_addr": "v1:172.21.15.92:0/2403132573", 2024-03-11T21:32:46.984 INFO:teuthology.orchestra.run.smithi097.stdout: "tid": 70026 2024-03-11T21:32:46.984 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:32:46.984 INFO:teuthology.orchestra.run.smithi097.stdout: "events": [ 2024-03-11T21:32:46.984 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:32:46.984 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "initiated", 2024-03-11T21:32:46.984 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:28:24.738536+0000", 2024-03-11T21:32:46.984 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-03-11T21:32:46.984 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:32:46.985 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:32:46.985 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "header_read", 2024-03-11T21:32:46.985 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:28:24.738536+0000", 2024-03-11T21:32:46.985 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-03-11T21:32:46.985 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:32:46.985 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:32:46.985 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "throttled", 2024-03-11T21:32:46.985 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:28:24.738537+0000", 2024-03-11T21:32:46.985 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.026e-06 2024-03-11T21:32:46.985 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:32:46.985 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:32:46.985 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "all_read", 2024-03-11T21:32:46.985 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:28:24.738611+0000", 2024-03-11T21:32:46.985 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 7.3594e-05 2024-03-11T21:32:46.986 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:32:46.986 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:32:46.986 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "dispatched", 2024-03-11T21:32:46.986 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:28:24.738621+0000", 2024-03-11T21:32:46.986 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.0628999999999999e-05 2024-03-11T21:32:46.986 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:32:46.986 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:32:46.986 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "queued_for_pg", 2024-03-11T21:32:46.986 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:28:24.738629+0000", 2024-03-11T21:32:46.986 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 7.2529999999999998e-06 2024-03-11T21:32:46.986 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:32:46.986 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:32:46.986 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "reached_pg", 2024-03-11T21:32:46.986 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:28:24.740821+0000", 2024-03-11T21:32:46.987 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.0021927489999999999 2024-03-11T21:32:46.987 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:32:46.987 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:32:46.987 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "started", 2024-03-11T21:32:46.987 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:28:24.740945+0000", 2024-03-11T21:32:46.987 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.00012362300000000001 2024-03-11T21:32:46.987 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:32:46.987 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:32:46.987 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "waiting for subops from 3,6", 2024-03-11T21:32:46.987 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:28:24.741046+0000", 2024-03-11T21:32:46.987 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.000100975 2024-03-11T21:32:46.987 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:32:46.987 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:32:46.987 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "op_commit", 2024-03-11T21:32:46.987 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:28:24.743548+0000", 2024-03-11T21:32:46.988 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.0025022820000000002 2024-03-11T21:32:46.988 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:32:46.988 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:32:46.988 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "sub_op_commit_rec", 2024-03-11T21:32:46.988 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:28:24.743766+0000", 2024-03-11T21:32:46.988 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.00021741999999999999 2024-03-11T21:32:46.988 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:32:46.988 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:32:46.988 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "sub_op_commit_rec", 2024-03-11T21:32:46.988 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:28:24.807853+0000", 2024-03-11T21:32:46.988 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.064087273 2024-03-11T21:32:46.988 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:32:46.988 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:32:46.988 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "commit_sent", 2024-03-11T21:32:46.989 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:28:24.807887+0000", 2024-03-11T21:32:46.989 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 3.3779000000000003e-05 2024-03-11T21:32:46.989 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:32:46.989 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:32:46.989 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "done", 2024-03-11T21:32:46.989 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:28:24.807922+0000", 2024-03-11T21:32:46.989 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 3.5296000000000002e-05 2024-03-11T21:32:46.989 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-03-11T21:32:46.989 INFO:teuthology.orchestra.run.smithi097.stdout: ] 2024-03-11T21:32:46.989 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-03-11T21:32:46.989 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:32:46.989 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:32:46.989 INFO:teuthology.orchestra.run.smithi097.stdout: "description": "osd_op(client.34715.0:115996 7.1 7:825d57d9:::benchmark_data_smithi092_76040_object115995:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e119)", 2024-03-11T21:32:46.989 INFO:teuthology.orchestra.run.smithi097.stdout: "initiated_at": "2024-03-11T21:29:08.463013+0000", 2024-03-11T21:32:46.990 INFO:teuthology.orchestra.run.smithi097.stdout: "age": 218.48611602599999, 2024-03-11T21:32:46.990 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.065927562999999995, 2024-03-11T21:32:46.990 INFO:teuthology.orchestra.run.smithi097.stdout: "type_data": { 2024-03-11T21:32:46.990 INFO:teuthology.orchestra.run.smithi097.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-11T21:32:46.990 INFO:teuthology.orchestra.run.smithi097.stdout: "client_info": { 2024-03-11T21:32:46.990 INFO:teuthology.orchestra.run.smithi097.stdout: "client": "client.34715", 2024-03-11T21:32:46.990 INFO:teuthology.orchestra.run.smithi097.stdout: "client_addr": "v1:172.21.15.92:0/2403132573", 2024-03-11T21:32:46.990 INFO:teuthology.orchestra.run.smithi097.stdout: "tid": 115996 2024-03-11T21:32:46.990 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:32:46.990 INFO:teuthology.orchestra.run.smithi097.stdout: "events": [ 2024-03-11T21:32:46.990 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:32:46.990 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "initiated", 2024-03-11T21:32:46.990 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:29:08.463013+0000", 2024-03-11T21:32:46.991 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-03-11T21:32:46.991 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:32:46.991 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:32:46.991 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "header_read", 2024-03-11T21:32:46.991 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:29:08.463013+0000", 2024-03-11T21:32:46.991 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-03-11T21:32:46.991 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:32:46.991 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:32:46.991 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "throttled", 2024-03-11T21:32:46.991 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:29:08.463014+0000", 2024-03-11T21:32:46.991 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.4720000000000001e-06 2024-03-11T21:32:46.991 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:32:46.991 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:32:46.991 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "all_read", 2024-03-11T21:32:46.991 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:29:08.463052+0000", 2024-03-11T21:32:46.992 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 3.8003000000000002e-05 2024-03-11T21:32:46.992 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:32:46.992 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:32:46.992 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "dispatched", 2024-03-11T21:32:46.992 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:29:08.463066+0000", 2024-03-11T21:32:46.992 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.3869000000000001e-05 2024-03-11T21:32:46.992 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:32:46.992 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:32:46.992 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "queued_for_pg", 2024-03-11T21:32:46.992 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:29:08.463074+0000", 2024-03-11T21:32:46.992 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 7.7190000000000001e-06 2024-03-11T21:32:46.992 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:32:46.992 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:32:46.992 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "reached_pg", 2024-03-11T21:32:46.993 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:29:08.464705+0000", 2024-03-11T21:32:46.993 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.001631144 2024-03-11T21:32:46.993 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:32:46.993 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:32:46.993 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "started", 2024-03-11T21:32:46.993 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:29:08.464839+0000", 2024-03-11T21:32:46.993 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.000134041 2024-03-11T21:32:46.993 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:32:46.993 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:32:46.993 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "waiting for subops from 3,6", 2024-03-11T21:32:46.993 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:29:08.464988+0000", 2024-03-11T21:32:46.993 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.000148529 2024-03-11T21:32:46.993 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:32:46.993 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:32:46.993 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "op_commit", 2024-03-11T21:32:46.994 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:29:08.467087+0000", 2024-03-11T21:32:46.994 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.002099409 2024-03-11T21:32:46.994 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:32:46.994 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:32:46.994 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "sub_op_commit_rec", 2024-03-11T21:32:46.994 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:29:08.467258+0000", 2024-03-11T21:32:46.994 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.00017126500000000001 2024-03-11T21:32:46.994 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:32:46.994 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:32:46.994 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "sub_op_commit_rec", 2024-03-11T21:32:46.994 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:29:08.528832+0000", 2024-03-11T21:32:46.994 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.061574265000000003 2024-03-11T21:32:46.994 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:32:46.995 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:32:46.995 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "commit_sent", 2024-03-11T21:32:46.995 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:29:08.528895+0000", 2024-03-11T21:32:46.995 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 6.2652999999999998e-05 2024-03-11T21:32:46.995 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:32:46.995 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:32:46.995 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "done", 2024-03-11T21:32:46.995 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:29:08.528940+0000", 2024-03-11T21:32:46.995 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 4.5194000000000001e-05 2024-03-11T21:32:46.995 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-03-11T21:32:46.995 INFO:teuthology.orchestra.run.smithi097.stdout: ] 2024-03-11T21:32:46.995 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-03-11T21:32:46.995 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:32:46.995 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:32:46.996 INFO:teuthology.orchestra.run.smithi097.stdout: "description": "osd_op(client.34715.0:116275 7.1 7:889dcf6e:::benchmark_data_smithi092_76040_object116274:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e119)", 2024-03-11T21:32:46.996 INFO:teuthology.orchestra.run.smithi097.stdout: "initiated_at": "2024-03-11T21:29:08.765237+0000", 2024-03-11T21:32:46.996 INFO:teuthology.orchestra.run.smithi097.stdout: "age": 218.18389197400001, 2024-03-11T21:32:46.996 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.11311642700000001, 2024-03-11T21:32:46.996 INFO:teuthology.orchestra.run.smithi097.stdout: "type_data": { 2024-03-11T21:32:46.996 INFO:teuthology.orchestra.run.smithi097.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-11T21:32:46.996 INFO:teuthology.orchestra.run.smithi097.stdout: "client_info": { 2024-03-11T21:32:46.996 INFO:teuthology.orchestra.run.smithi097.stdout: "client": "client.34715", 2024-03-11T21:32:46.996 INFO:teuthology.orchestra.run.smithi097.stdout: "client_addr": "v1:172.21.15.92:0/2403132573", 2024-03-11T21:32:46.996 INFO:teuthology.orchestra.run.smithi097.stdout: "tid": 116275 2024-03-11T21:32:46.996 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:32:46.996 INFO:teuthology.orchestra.run.smithi097.stdout: "events": [ 2024-03-11T21:32:46.996 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:32:46.996 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "initiated", 2024-03-11T21:32:46.997 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:29:08.765237+0000", 2024-03-11T21:32:46.997 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-03-11T21:32:46.997 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:32:46.997 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:32:46.997 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "header_read", 2024-03-11T21:32:46.997 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:29:08.765237+0000", 2024-03-11T21:32:46.997 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-03-11T21:32:46.997 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:32:46.997 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:32:46.997 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "throttled", 2024-03-11T21:32:46.997 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:29:08.765239+0000", 2024-03-11T21:32:46.997 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.9400000000000001e-06 2024-03-11T21:32:46.997 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:32:46.997 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:32:46.998 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "all_read", 2024-03-11T21:32:46.998 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:29:08.765317+0000", 2024-03-11T21:32:46.998 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 7.8537999999999999e-05 2024-03-11T21:32:46.998 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:32:46.998 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:32:46.998 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "dispatched", 2024-03-11T21:32:46.998 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:29:08.765331+0000", 2024-03-11T21:32:46.998 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.3453e-05 2024-03-11T21:32:46.998 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:32:46.998 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:32:46.998 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "queued_for_pg", 2024-03-11T21:32:46.998 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:29:08.765339+0000", 2024-03-11T21:32:46.998 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 7.7980000000000006e-06 2024-03-11T21:32:46.999 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:32:46.999 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:32:46.999 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "reached_pg", 2024-03-11T21:32:46.999 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:29:08.765377+0000", 2024-03-11T21:32:46.999 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 3.8593000000000001e-05 2024-03-11T21:32:46.999 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:32:46.999 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:32:46.999 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "started", 2024-03-11T21:32:46.999 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:29:08.765489+0000", 2024-03-11T21:32:46.999 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.00011158300000000001 2024-03-11T21:32:46.999 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:32:46.999 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:32:46.999 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "waiting for subops from 3,6", 2024-03-11T21:32:46.999 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:29:08.765629+0000", 2024-03-11T21:32:46.999 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.00014014299999999999 2024-03-11T21:32:47.000 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:32:47.000 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:32:47.000 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "op_commit", 2024-03-11T21:32:47.000 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:29:08.769442+0000", 2024-03-11T21:32:47.000 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.0038130519999999999 2024-03-11T21:32:47.000 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:32:47.000 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:32:47.000 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "sub_op_commit_rec", 2024-03-11T21:32:47.000 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:29:08.771245+0000", 2024-03-11T21:32:47.000 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.0018028300000000001 2024-03-11T21:32:47.000 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:32:47.000 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:32:47.000 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "sub_op_commit_rec", 2024-03-11T21:32:47.000 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:29:08.878254+0000", 2024-03-11T21:32:47.000 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.107009544 2024-03-11T21:32:47.001 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:32:47.001 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:32:47.001 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "commit_sent", 2024-03-11T21:32:47.001 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:29:08.878307+0000", 2024-03-11T21:32:47.001 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 5.27e-05 2024-03-11T21:32:47.001 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:32:47.001 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:32:47.001 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "done", 2024-03-11T21:32:47.001 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:29:08.878353+0000", 2024-03-11T21:32:47.001 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 4.6252999999999999e-05 2024-03-11T21:32:47.001 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-03-11T21:32:47.001 INFO:teuthology.orchestra.run.smithi097.stdout: ] 2024-03-11T21:32:47.001 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-03-11T21:32:47.001 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:32:47.001 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:32:47.002 INFO:teuthology.orchestra.run.smithi097.stdout: "description": "osd_op(client.34715.0:116285 7.1 7:8ebc24c4:::benchmark_data_smithi092_76040_object116284:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e119)", 2024-03-11T21:32:47.002 INFO:teuthology.orchestra.run.smithi097.stdout: "initiated_at": "2024-03-11T21:29:08.771631+0000", 2024-03-11T21:32:47.002 INFO:teuthology.orchestra.run.smithi097.stdout: "age": 218.17749773200001, 2024-03-11T21:32:47.002 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.121431314, 2024-03-11T21:32:47.002 INFO:teuthology.orchestra.run.smithi097.stdout: "type_data": { 2024-03-11T21:32:47.002 INFO:teuthology.orchestra.run.smithi097.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-11T21:32:47.002 INFO:teuthology.orchestra.run.smithi097.stdout: "client_info": { 2024-03-11T21:32:47.002 INFO:teuthology.orchestra.run.smithi097.stdout: "client": "client.34715", 2024-03-11T21:32:47.002 INFO:teuthology.orchestra.run.smithi097.stdout: "client_addr": "v1:172.21.15.92:0/2403132573", 2024-03-11T21:32:47.002 INFO:teuthology.orchestra.run.smithi097.stdout: "tid": 116285 2024-03-11T21:32:47.002 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:32:47.002 INFO:teuthology.orchestra.run.smithi097.stdout: "events": [ 2024-03-11T21:32:47.002 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:32:47.003 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "initiated", 2024-03-11T21:32:47.003 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:29:08.771631+0000", 2024-03-11T21:32:47.003 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-03-11T21:32:47.003 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:32:47.003 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:32:47.003 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "header_read", 2024-03-11T21:32:47.003 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:29:08.771631+0000", 2024-03-11T21:32:47.003 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-03-11T21:32:47.003 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:32:47.003 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:32:47.003 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "throttled", 2024-03-11T21:32:47.003 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:29:08.771633+0000", 2024-03-11T21:32:47.003 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 2.013e-06 2024-03-11T21:32:47.003 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:32:47.003 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:32:47.003 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "all_read", 2024-03-11T21:32:47.004 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:29:08.771682+0000", 2024-03-11T21:32:47.004 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 4.9419999999999998e-05 2024-03-11T21:32:47.004 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:32:47.004 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:32:47.004 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "dispatched", 2024-03-11T21:32:47.004 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:29:08.771695+0000", 2024-03-11T21:32:47.004 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.2486e-05 2024-03-11T21:32:47.004 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:32:47.004 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:32:47.004 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "queued_for_pg", 2024-03-11T21:32:47.004 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:29:08.771703+0000", 2024-03-11T21:32:47.004 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 8.4460000000000001e-06 2024-03-11T21:32:47.004 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:32:47.004 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:32:47.004 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "reached_pg", 2024-03-11T21:32:47.005 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:29:08.771756+0000", 2024-03-11T21:32:47.005 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 5.2992999999999999e-05 2024-03-11T21:32:47.005 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:32:47.005 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:32:47.005 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "started", 2024-03-11T21:32:47.005 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:29:08.771917+0000", 2024-03-11T21:32:47.005 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.00016044900000000001 2024-03-11T21:32:47.005 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:32:47.005 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:32:47.005 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "waiting for subops from 3,6", 2024-03-11T21:32:47.005 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:29:08.772013+0000", 2024-03-11T21:32:47.005 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 9.6419e-05 2024-03-11T21:32:47.005 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:32:47.005 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:32:47.006 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "sub_op_commit_rec", 2024-03-11T21:32:47.006 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:29:08.778457+0000", 2024-03-11T21:32:47.006 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.0064432739999999997 2024-03-11T21:32:47.006 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:32:47.006 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:32:47.006 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "op_commit", 2024-03-11T21:32:47.006 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:29:08.778552+0000", 2024-03-11T21:32:47.006 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 9.5730999999999995e-05 2024-03-11T21:32:47.006 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:32:47.006 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:32:47.006 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "sub_op_commit_rec", 2024-03-11T21:32:47.006 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:29:08.892956+0000", 2024-03-11T21:32:47.006 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.114403941 2024-03-11T21:32:47.006 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:32:47.007 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:32:47.007 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "commit_sent", 2024-03-11T21:32:47.007 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:29:08.893016+0000", 2024-03-11T21:32:47.007 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 5.982e-05 2024-03-11T21:32:47.007 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:32:47.007 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:32:47.007 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "done", 2024-03-11T21:32:47.007 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:29:08.893062+0000", 2024-03-11T21:32:47.007 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 4.6322000000000001e-05 2024-03-11T21:32:47.007 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-03-11T21:32:47.007 INFO:teuthology.orchestra.run.smithi097.stdout: ] 2024-03-11T21:32:47.007 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-03-11T21:32:47.007 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:32:47.007 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:32:47.007 INFO:teuthology.orchestra.run.smithi097.stdout: "description": "osd_op(client.34715.0:178419 7.1 7:8a449f8c:::benchmark_data_smithi092_76040_object56508:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e133)", 2024-03-11T21:32:47.008 INFO:teuthology.orchestra.run.smithi097.stdout: "initiated_at": "2024-03-11T21:30:18.923975+0000", 2024-03-11T21:32:47.008 INFO:teuthology.orchestra.run.smithi097.stdout: "age": 148.02515350100001, 2024-03-11T21:32:47.008 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.082047249000000003, 2024-03-11T21:32:47.008 INFO:teuthology.orchestra.run.smithi097.stdout: "type_data": { 2024-03-11T21:32:47.008 INFO:teuthology.orchestra.run.smithi097.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-11T21:32:47.008 INFO:teuthology.orchestra.run.smithi097.stdout: "client_info": { 2024-03-11T21:32:47.008 INFO:teuthology.orchestra.run.smithi097.stdout: "client": "client.34715", 2024-03-11T21:32:47.008 INFO:teuthology.orchestra.run.smithi097.stdout: "client_addr": "v1:172.21.15.92:0/2403132573", 2024-03-11T21:32:47.008 INFO:teuthology.orchestra.run.smithi097.stdout: "tid": 178419 2024-03-11T21:32:47.008 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:32:47.008 INFO:teuthology.orchestra.run.smithi097.stdout: "events": [ 2024-03-11T21:32:47.008 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:32:47.008 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "initiated", 2024-03-11T21:32:47.008 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:30:18.923975+0000", 2024-03-11T21:32:47.009 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-03-11T21:32:47.009 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:32:47.009 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:32:47.009 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "header_read", 2024-03-11T21:32:47.009 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:30:18.923975+0000", 2024-03-11T21:32:47.009 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-03-11T21:32:47.009 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:32:47.009 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:32:47.009 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "throttled", 2024-03-11T21:32:47.009 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:30:18.923977+0000", 2024-03-11T21:32:47.009 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.827e-06 2024-03-11T21:32:47.009 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:32:47.009 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:32:47.009 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "all_read", 2024-03-11T21:32:47.010 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:30:18.923982+0000", 2024-03-11T21:32:47.010 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 4.9989999999999999e-06 2024-03-11T21:32:47.010 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:32:47.010 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:32:47.010 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "dispatched", 2024-03-11T21:32:47.010 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:30:18.923993+0000", 2024-03-11T21:32:47.010 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.1342000000000001e-05 2024-03-11T21:32:47.010 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:32:47.010 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:32:47.010 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "queued_for_pg", 2024-03-11T21:32:47.010 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:30:18.924001+0000", 2024-03-11T21:32:47.010 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 7.7330000000000003e-06 2024-03-11T21:32:47.010 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:32:47.010 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:32:47.011 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "reached_pg", 2024-03-11T21:32:47.011 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:30:18.924054+0000", 2024-03-11T21:32:47.011 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 5.2494000000000003e-05 2024-03-11T21:32:47.011 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:32:47.011 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:32:47.011 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "started", 2024-03-11T21:32:47.011 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:30:18.924175+0000", 2024-03-11T21:32:47.011 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.00012139200000000001 2024-03-11T21:32:47.011 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:32:47.011 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:32:47.011 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "waiting for subops from 3,6", 2024-03-11T21:32:47.011 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:30:18.924279+0000", 2024-03-11T21:32:47.011 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.000104046 2024-03-11T21:32:47.011 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:32:47.011 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:32:47.012 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "op_commit", 2024-03-11T21:32:47.012 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:30:18.926505+0000", 2024-03-11T21:32:47.012 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.002225557 2024-03-11T21:32:47.012 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:32:47.012 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:32:47.012 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "sub_op_commit_rec", 2024-03-11T21:32:47.012 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:30:18.928277+0000", 2024-03-11T21:32:47.012 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.0017725600000000001 2024-03-11T21:32:47.012 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:32:47.012 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:32:47.012 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "sub_op_commit_rec", 2024-03-11T21:32:47.012 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:30:19.005868+0000", 2024-03-11T21:32:47.012 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.077590846000000005 2024-03-11T21:32:47.012 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:32:47.013 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:32:47.013 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "commit_sent", 2024-03-11T21:32:47.013 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:30:19.005968+0000", 2024-03-11T21:32:47.013 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 9.9586999999999994e-05 2024-03-11T21:32:47.013 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:32:47.013 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:32:47.013 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "done", 2024-03-11T21:32:47.013 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:30:19.006023+0000", 2024-03-11T21:32:47.013 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 5.4866e-05 2024-03-11T21:32:47.013 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-03-11T21:32:47.013 INFO:teuthology.orchestra.run.smithi097.stdout: ] 2024-03-11T21:32:47.013 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-03-11T21:32:47.013 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:32:47.013 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:32:47.013 INFO:teuthology.orchestra.run.smithi097.stdout: "description": "osd_op(client.34715.0:190373 7.1 7:84520120:::benchmark_data_smithi092_76040_object68462:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e135)", 2024-03-11T21:32:47.014 INFO:teuthology.orchestra.run.smithi097.stdout: "initiated_at": "2024-03-11T21:30:30.685204+0000", 2024-03-11T21:32:47.014 INFO:teuthology.orchestra.run.smithi097.stdout: "age": 136.26392501800001, 2024-03-11T21:32:47.014 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.064988606000000004, 2024-03-11T21:32:47.014 INFO:teuthology.orchestra.run.smithi097.stdout: "type_data": { 2024-03-11T21:32:47.014 INFO:teuthology.orchestra.run.smithi097.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-11T21:32:47.014 INFO:teuthology.orchestra.run.smithi097.stdout: "client_info": { 2024-03-11T21:32:47.014 INFO:teuthology.orchestra.run.smithi097.stdout: "client": "client.34715", 2024-03-11T21:32:47.014 INFO:teuthology.orchestra.run.smithi097.stdout: "client_addr": "v1:172.21.15.92:0/2403132573", 2024-03-11T21:32:47.014 INFO:teuthology.orchestra.run.smithi097.stdout: "tid": 190373 2024-03-11T21:32:47.014 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:32:47.014 INFO:teuthology.orchestra.run.smithi097.stdout: "events": [ 2024-03-11T21:32:47.014 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:32:47.014 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "initiated", 2024-03-11T21:32:47.014 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:30:30.685204+0000", 2024-03-11T21:32:47.015 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-03-11T21:32:47.015 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:32:47.015 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:32:47.015 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "header_read", 2024-03-11T21:32:47.015 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:30:30.685204+0000", 2024-03-11T21:32:47.015 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-03-11T21:32:47.015 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:32:47.015 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:32:47.015 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "throttled", 2024-03-11T21:32:47.015 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:30:30.685206+0000", 2024-03-11T21:32:47.015 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 2.2309999999999999e-06 2024-03-11T21:32:47.015 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:32:47.015 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:32:47.015 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "all_read", 2024-03-11T21:32:47.015 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:30:30.685212+0000", 2024-03-11T21:32:47.016 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 5.7189999999999998e-06 2024-03-11T21:32:47.016 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:32:47.016 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:32:47.016 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "dispatched", 2024-03-11T21:32:47.016 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:30:30.685225+0000", 2024-03-11T21:32:47.016 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.2994e-05 2024-03-11T21:32:47.016 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:32:47.016 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:32:47.016 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "queued_for_pg", 2024-03-11T21:32:47.016 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:30:30.685232+0000", 2024-03-11T21:32:47.016 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 7.6469999999999998e-06 2024-03-11T21:32:47.016 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:32:47.016 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:32:47.016 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "reached_pg", 2024-03-11T21:32:47.016 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:30:30.685278+0000", 2024-03-11T21:32:47.017 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 4.6088999999999999e-05 2024-03-11T21:32:47.017 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:32:47.017 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:32:47.017 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "started", 2024-03-11T21:32:47.017 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:30:30.685361+0000", 2024-03-11T21:32:47.017 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 8.2492999999999998e-05 2024-03-11T21:32:47.017 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:32:47.017 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:32:47.017 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "waiting for subops from 3,6", 2024-03-11T21:32:47.017 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:30:30.685431+0000", 2024-03-11T21:32:47.017 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 7.0062999999999996e-05 2024-03-11T21:32:47.017 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:32:47.017 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:32:47.017 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "op_commit", 2024-03-11T21:32:47.017 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:30:30.687728+0000", 2024-03-11T21:32:47.018 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.002297136 2024-03-11T21:32:47.018 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:32:47.018 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:32:47.018 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "sub_op_commit_rec", 2024-03-11T21:32:47.018 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:30:30.687840+0000", 2024-03-11T21:32:47.018 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.000111844 2024-03-11T21:32:47.018 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:32:47.018 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:32:47.018 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "sub_op_commit_rec", 2024-03-11T21:32:47.018 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:30:30.750112+0000", 2024-03-11T21:32:47.018 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.062271871999999999 2024-03-11T21:32:47.018 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:32:47.018 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:32:47.018 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "commit_sent", 2024-03-11T21:32:47.018 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:30:30.750150+0000", 2024-03-11T21:32:47.019 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 3.8587000000000002e-05 2024-03-11T21:32:47.019 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:32:47.019 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:32:47.019 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "done", 2024-03-11T21:32:47.019 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:30:30.750192+0000", 2024-03-11T21:32:47.019 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 4.1931000000000002e-05 2024-03-11T21:32:47.019 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-03-11T21:32:47.019 INFO:teuthology.orchestra.run.smithi097.stdout: ] 2024-03-11T21:32:47.019 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-03-11T21:32:47.019 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:32:47.019 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:32:47.019 INFO:teuthology.orchestra.run.smithi097.stdout: "description": "osd_op(client.34715.0:191317 7.1 7:8827d806:::benchmark_data_smithi092_76040_object69406:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e135)", 2024-03-11T21:32:47.019 INFO:teuthology.orchestra.run.smithi097.stdout: "initiated_at": "2024-03-11T21:30:31.663053+0000", 2024-03-11T21:32:47.019 INFO:teuthology.orchestra.run.smithi097.stdout: "age": 135.28607545400001, 2024-03-11T21:32:47.019 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.092794199999999993, 2024-03-11T21:32:47.020 INFO:teuthology.orchestra.run.smithi097.stdout: "type_data": { 2024-03-11T21:32:47.020 INFO:teuthology.orchestra.run.smithi097.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-11T21:32:47.020 INFO:teuthology.orchestra.run.smithi097.stdout: "client_info": { 2024-03-11T21:32:47.020 INFO:teuthology.orchestra.run.smithi097.stdout: "client": "client.34715", 2024-03-11T21:32:47.020 INFO:teuthology.orchestra.run.smithi097.stdout: "client_addr": "v1:172.21.15.92:0/2403132573", 2024-03-11T21:32:47.020 INFO:teuthology.orchestra.run.smithi097.stdout: "tid": 191317 2024-03-11T21:32:47.020 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:32:47.020 INFO:teuthology.orchestra.run.smithi097.stdout: "events": [ 2024-03-11T21:32:47.020 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:32:47.021 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "initiated", 2024-03-11T21:32:47.021 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:30:31.663053+0000", 2024-03-11T21:32:47.021 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-03-11T21:32:47.021 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:32:47.021 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:32:47.021 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "header_read", 2024-03-11T21:32:47.021 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:30:31.663053+0000", 2024-03-11T21:32:47.022 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-03-11T21:32:47.022 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:32:47.022 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:32:47.022 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "throttled", 2024-03-11T21:32:47.022 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:30:31.663055+0000", 2024-03-11T21:32:47.022 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.906e-06 2024-03-11T21:32:47.022 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:32:47.022 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:32:47.022 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "all_read", 2024-03-11T21:32:47.022 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:30:31.663061+0000", 2024-03-11T21:32:47.022 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 5.4269999999999999e-06 2024-03-11T21:32:47.022 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:32:47.022 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:32:47.022 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "dispatched", 2024-03-11T21:32:47.023 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:30:31.663075+0000", 2024-03-11T21:32:47.023 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.4538999999999999e-05 2024-03-11T21:32:47.023 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:32:47.023 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:32:47.023 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "queued_for_pg", 2024-03-11T21:32:47.023 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:30:31.663083+0000", 2024-03-11T21:32:47.023 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 7.4440000000000002e-06 2024-03-11T21:32:47.023 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:32:47.023 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:32:47.023 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "reached_pg", 2024-03-11T21:32:47.023 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:30:31.663141+0000", 2024-03-11T21:32:47.023 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 5.8196000000000003e-05 2024-03-11T21:32:47.023 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:32:47.023 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:32:47.024 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "started", 2024-03-11T21:32:47.024 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:30:31.663260+0000", 2024-03-11T21:32:47.024 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.00011961800000000001 2024-03-11T21:32:47.024 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:32:47.024 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:32:47.024 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "waiting for subops from 3,6", 2024-03-11T21:32:47.024 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:30:31.663385+0000", 2024-03-11T21:32:47.024 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.00012454499999999999 2024-03-11T21:32:47.024 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:32:47.024 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:32:47.024 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "op_commit", 2024-03-11T21:32:47.024 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:30:31.666203+0000", 2024-03-11T21:32:47.024 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.002818086 2024-03-11T21:32:47.024 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:32:47.024 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:32:47.025 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "sub_op_commit_rec", 2024-03-11T21:32:47.025 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:30:31.668748+0000", 2024-03-11T21:32:47.025 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.0025445350000000001 2024-03-11T21:32:47.025 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:32:47.025 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:32:47.025 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "sub_op_commit_rec", 2024-03-11T21:32:47.025 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:30:31.755747+0000", 2024-03-11T21:32:47.025 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.086999014 2024-03-11T21:32:47.025 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:32:47.025 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:32:47.025 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "commit_sent", 2024-03-11T21:32:47.025 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:30:31.755803+0000", 2024-03-11T21:32:47.025 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 5.6746000000000002e-05 2024-03-11T21:32:47.025 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:32:47.025 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:32:47.026 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "done", 2024-03-11T21:32:47.026 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:30:31.755848+0000", 2024-03-11T21:32:47.026 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 4.4144000000000003e-05 2024-03-11T21:32:47.026 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-03-11T21:32:47.026 INFO:teuthology.orchestra.run.smithi097.stdout: ] 2024-03-11T21:32:47.026 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-03-11T21:32:47.026 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:32:47.026 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:32:47.026 INFO:teuthology.orchestra.run.smithi097.stdout: "description": "osd_op(client.34715.0:192265 7.1 7:8d5f4ce6:::benchmark_data_smithi092_76040_object70354:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e135)", 2024-03-11T21:32:47.026 INFO:teuthology.orchestra.run.smithi097.stdout: "initiated_at": "2024-03-11T21:30:32.633187+0000", 2024-03-11T21:32:47.026 INFO:teuthology.orchestra.run.smithi097.stdout: "age": 134.315941399, 2024-03-11T21:32:47.026 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.064618474999999995, 2024-03-11T21:32:47.026 INFO:teuthology.orchestra.run.smithi097.stdout: "type_data": { 2024-03-11T21:32:47.026 INFO:teuthology.orchestra.run.smithi097.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-11T21:32:47.027 INFO:teuthology.orchestra.run.smithi097.stdout: "client_info": { 2024-03-11T21:32:47.027 INFO:teuthology.orchestra.run.smithi097.stdout: "client": "client.34715", 2024-03-11T21:32:47.027 INFO:teuthology.orchestra.run.smithi097.stdout: "client_addr": "v1:172.21.15.92:0/2403132573", 2024-03-11T21:32:47.027 INFO:teuthology.orchestra.run.smithi097.stdout: "tid": 192265 2024-03-11T21:32:47.027 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:32:47.027 INFO:teuthology.orchestra.run.smithi097.stdout: "events": [ 2024-03-11T21:32:47.027 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:32:47.027 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "initiated", 2024-03-11T21:32:47.027 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:30:32.633187+0000", 2024-03-11T21:32:47.027 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-03-11T21:32:47.027 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:32:47.027 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:32:47.027 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "header_read", 2024-03-11T21:32:47.027 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:30:32.633187+0000", 2024-03-11T21:32:47.028 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-03-11T21:32:47.028 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:32:47.028 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:32:47.028 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "throttled", 2024-03-11T21:32:47.028 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:30:32.633190+0000", 2024-03-11T21:32:47.028 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 2.4430000000000002e-06 2024-03-11T21:32:47.028 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:32:47.028 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:32:47.028 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "all_read", 2024-03-11T21:32:47.028 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:30:32.633195+0000", 2024-03-11T21:32:47.028 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 5.6119999999999998e-06 2024-03-11T21:32:47.028 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:32:47.028 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:32:47.028 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "dispatched", 2024-03-11T21:32:47.028 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:30:32.633208+0000", 2024-03-11T21:32:47.029 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.2536e-05 2024-03-11T21:32:47.029 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:32:47.029 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:32:47.029 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "queued_for_pg", 2024-03-11T21:32:47.029 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:30:32.633216+0000", 2024-03-11T21:32:47.029 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 8.2390000000000002e-06 2024-03-11T21:32:47.029 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:32:47.029 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:32:47.029 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "reached_pg", 2024-03-11T21:32:47.029 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:30:32.633261+0000", 2024-03-11T21:32:47.029 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 4.4357999999999998e-05 2024-03-11T21:32:47.029 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:32:47.029 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:32:47.029 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "started", 2024-03-11T21:32:47.030 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:30:32.633360+0000", 2024-03-11T21:32:47.030 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 9.9821000000000005e-05 2024-03-11T21:32:47.030 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:32:47.030 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:32:47.030 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "waiting for subops from 3,6", 2024-03-11T21:32:47.030 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:30:32.633468+0000", 2024-03-11T21:32:47.030 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.000107632 2024-03-11T21:32:47.030 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:32:47.030 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:32:47.030 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "op_commit", 2024-03-11T21:32:47.030 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:30:32.635514+0000", 2024-03-11T21:32:47.030 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.0020458529999999998 2024-03-11T21:32:47.030 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:32:47.030 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:32:47.031 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "sub_op_commit_rec", 2024-03-11T21:32:47.031 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:30:32.636000+0000", 2024-03-11T21:32:47.031 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.000485804 2024-03-11T21:32:47.031 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:32:47.031 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:32:47.031 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "sub_op_commit_rec", 2024-03-11T21:32:47.031 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:30:32.697721+0000", 2024-03-11T21:32:47.031 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.061721534000000002 2024-03-11T21:32:47.031 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:32:47.031 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:32:47.031 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "commit_sent", 2024-03-11T21:32:47.031 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:30:32.697759+0000", 2024-03-11T21:32:47.031 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 3.7780000000000001e-05 2024-03-11T21:32:47.031 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:32:47.031 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:32:47.032 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "done", 2024-03-11T21:32:47.032 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:30:32.697806+0000", 2024-03-11T21:32:47.032 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 4.6863000000000002e-05 2024-03-11T21:32:47.032 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-03-11T21:32:47.032 INFO:teuthology.orchestra.run.smithi097.stdout: ] 2024-03-11T21:32:47.032 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-03-11T21:32:47.032 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:32:47.032 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:32:47.032 INFO:teuthology.orchestra.run.smithi097.stdout: "description": "osd_op(client.34715.0:225278 7.1 7:806b8c80:::benchmark_data_smithi092_76040_object103367:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e144)", 2024-03-11T21:32:47.032 INFO:teuthology.orchestra.run.smithi097.stdout: "initiated_at": "2024-03-11T21:31:05.281169+0000", 2024-03-11T21:32:47.032 INFO:teuthology.orchestra.run.smithi097.stdout: "age": 101.667959776, 2024-03-11T21:32:47.032 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.067854118000000005, 2024-03-11T21:32:47.032 INFO:teuthology.orchestra.run.smithi097.stdout: "type_data": { 2024-03-11T21:32:47.032 INFO:teuthology.orchestra.run.smithi097.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-11T21:32:47.032 INFO:teuthology.orchestra.run.smithi097.stdout: "client_info": { 2024-03-11T21:32:47.033 INFO:teuthology.orchestra.run.smithi097.stdout: "client": "client.34715", 2024-03-11T21:32:47.033 INFO:teuthology.orchestra.run.smithi097.stdout: "client_addr": "v1:172.21.15.92:0/2403132573", 2024-03-11T21:32:47.033 INFO:teuthology.orchestra.run.smithi097.stdout: "tid": 225278 2024-03-11T21:32:47.033 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:32:47.033 INFO:teuthology.orchestra.run.smithi097.stdout: "events": [ 2024-03-11T21:32:47.033 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:32:47.033 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "initiated", 2024-03-11T21:32:47.033 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:31:05.281169+0000", 2024-03-11T21:32:47.033 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-03-11T21:32:47.033 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:32:47.033 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:32:47.033 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "header_read", 2024-03-11T21:32:47.033 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:31:05.281169+0000", 2024-03-11T21:32:47.033 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-03-11T21:32:47.033 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:32:47.034 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:32:47.034 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "throttled", 2024-03-11T21:32:47.034 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:31:05.281171+0000", 2024-03-11T21:32:47.034 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.5659999999999999e-06 2024-03-11T21:32:47.034 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:32:47.034 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:32:47.034 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "all_read", 2024-03-11T21:32:47.034 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:31:05.281176+0000", 2024-03-11T21:32:47.034 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 4.9309999999999999e-06 2024-03-11T21:32:47.034 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:32:47.034 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:32:47.034 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "dispatched", 2024-03-11T21:32:47.034 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:31:05.281189+0000", 2024-03-11T21:32:47.034 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.3431000000000001e-05 2024-03-11T21:32:47.034 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:32:47.035 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:32:47.035 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "queued_for_pg", 2024-03-11T21:32:47.035 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:31:05.281197+0000", 2024-03-11T21:32:47.035 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 8.0180000000000001e-06 2024-03-11T21:32:47.035 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:32:47.035 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:32:47.035 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "reached_pg", 2024-03-11T21:32:47.035 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:31:05.281259+0000", 2024-03-11T21:32:47.035 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 6.1997999999999998e-05 2024-03-11T21:32:47.035 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:32:47.035 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:32:47.035 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "started", 2024-03-11T21:32:47.035 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:31:05.281394+0000", 2024-03-11T21:32:47.035 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.00013496 2024-03-11T21:32:47.036 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:32:47.036 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:32:47.036 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "waiting for subops from 3,6", 2024-03-11T21:32:47.036 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:31:05.281516+0000", 2024-03-11T21:32:47.036 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.00012200400000000001 2024-03-11T21:32:47.036 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:32:47.036 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:32:47.036 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "sub_op_commit_rec", 2024-03-11T21:32:47.036 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:31:05.284248+0000", 2024-03-11T21:32:47.036 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.0027323500000000001 2024-03-11T21:32:47.036 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:32:47.036 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:32:47.036 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "op_commit", 2024-03-11T21:32:47.036 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:31:05.284343+0000", 2024-03-11T21:32:47.036 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 9.4853999999999997e-05 2024-03-11T21:32:47.037 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:32:47.037 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:32:47.037 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "sub_op_commit_rec", 2024-03-11T21:32:47.037 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:31:05.348919+0000", 2024-03-11T21:32:47.037 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.064576332 2024-03-11T21:32:47.037 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:32:47.037 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:32:47.037 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "commit_sent", 2024-03-11T21:32:47.037 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:31:05.348977+0000", 2024-03-11T21:32:47.037 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 5.7213000000000003e-05 2024-03-11T21:32:47.037 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:32:47.037 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:32:47.037 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "done", 2024-03-11T21:32:47.037 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:31:05.349023+0000", 2024-03-11T21:32:47.037 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 4.6461000000000001e-05 2024-03-11T21:32:47.038 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-03-11T21:32:47.038 INFO:teuthology.orchestra.run.smithi097.stdout: ] 2024-03-11T21:32:47.038 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-03-11T21:32:47.038 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:32:47.038 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:32:47.038 INFO:teuthology.orchestra.run.smithi097.stdout: "description": "osd_op(client.15150.0:49199 8.2 8:4ed75a56:::benchmark_data_smithi092_104865_object49198:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e169)", 2024-03-11T21:32:47.038 INFO:teuthology.orchestra.run.smithi097.stdout: "initiated_at": "2024-03-11T21:32:25.332777+0000", 2024-03-11T21:32:47.038 INFO:teuthology.orchestra.run.smithi097.stdout: "age": 21.616351473999998, 2024-03-11T21:32:47.038 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.141463702, 2024-03-11T21:32:47.038 INFO:teuthology.orchestra.run.smithi097.stdout: "type_data": { 2024-03-11T21:32:47.038 INFO:teuthology.orchestra.run.smithi097.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-11T21:32:47.038 INFO:teuthology.orchestra.run.smithi097.stdout: "client_info": { 2024-03-11T21:32:47.038 INFO:teuthology.orchestra.run.smithi097.stdout: "client": "client.15150", 2024-03-11T21:32:47.038 INFO:teuthology.orchestra.run.smithi097.stdout: "client_addr": "v1:172.21.15.92:0/436606717", 2024-03-11T21:32:47.039 INFO:teuthology.orchestra.run.smithi097.stdout: "tid": 49199 2024-03-11T21:32:47.039 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:32:47.039 INFO:teuthology.orchestra.run.smithi097.stdout: "events": [ 2024-03-11T21:32:47.039 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:32:47.039 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "initiated", 2024-03-11T21:32:47.039 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:32:25.332777+0000", 2024-03-11T21:32:47.039 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-03-11T21:32:47.039 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:32:47.039 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:32:47.039 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "header_read", 2024-03-11T21:32:47.039 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:32:25.332777+0000", 2024-03-11T21:32:47.039 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-03-11T21:32:47.039 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:32:47.039 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:32:47.040 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "throttled", 2024-03-11T21:32:47.040 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:32:25.332779+0000", 2024-03-11T21:32:47.040 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.68e-06 2024-03-11T21:32:47.040 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:32:47.040 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:32:47.040 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "all_read", 2024-03-11T21:32:47.040 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:32:25.332874+0000", 2024-03-11T21:32:47.040 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 9.4918000000000002e-05 2024-03-11T21:32:47.040 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:32:47.040 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:32:47.040 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "dispatched", 2024-03-11T21:32:47.040 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:32:25.332886+0000", 2024-03-11T21:32:47.040 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.1674e-05 2024-03-11T21:32:47.040 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:32:47.040 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:32:47.041 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "queued_for_pg", 2024-03-11T21:32:47.041 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:32:25.332894+0000", 2024-03-11T21:32:47.041 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 8.2470000000000008e-06 2024-03-11T21:32:47.041 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:32:47.041 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:32:47.041 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "reached_pg", 2024-03-11T21:32:47.041 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:32:25.332952+0000", 2024-03-11T21:32:47.041 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 5.7769999999999997e-05 2024-03-11T21:32:47.041 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:32:47.041 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:32:47.041 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "started", 2024-03-11T21:32:47.041 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:32:25.333081+0000", 2024-03-11T21:32:47.041 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.00012933700000000001 2024-03-11T21:32:47.041 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:32:47.041 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:32:47.042 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "waiting for subops from 0,5", 2024-03-11T21:32:47.042 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:32:25.333215+0000", 2024-03-11T21:32:47.042 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.00013378500000000001 2024-03-11T21:32:47.042 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:32:47.042 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:32:47.042 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "op_commit", 2024-03-11T21:32:47.042 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:32:25.337101+0000", 2024-03-11T21:32:47.042 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.0038865089999999998 2024-03-11T21:32:47.042 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:32:47.042 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:32:47.042 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "sub_op_commit_rec", 2024-03-11T21:32:47.042 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:32:25.338320+0000", 2024-03-11T21:32:47.042 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.0012189869999999999 2024-03-11T21:32:47.042 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:32:47.043 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:32:47.043 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "sub_op_commit_rec", 2024-03-11T21:32:47.043 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:32:25.474062+0000", 2024-03-11T21:32:47.043 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.13574159499999999 2024-03-11T21:32:47.043 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:32:47.043 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:32:47.043 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "commit_sent", 2024-03-11T21:32:47.043 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:32:25.474193+0000", 2024-03-11T21:32:47.043 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.00013147300000000001 2024-03-11T21:32:47.043 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:32:47.043 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:32:47.043 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "done", 2024-03-11T21:32:47.043 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:32:25.474241+0000", 2024-03-11T21:32:47.043 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 4.7726999999999999e-05 2024-03-11T21:32:47.044 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-03-11T21:32:47.044 INFO:teuthology.orchestra.run.smithi097.stdout: ] 2024-03-11T21:32:47.044 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-03-11T21:32:47.044 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:32:47.044 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:32:47.044 INFO:teuthology.orchestra.run.smithi097.stdout: "description": "osd_op(client.15150.0:49201 8.2 8:4b6cf877:::benchmark_data_smithi092_104865_object49200:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e169)", 2024-03-11T21:32:47.044 INFO:teuthology.orchestra.run.smithi097.stdout: "initiated_at": "2024-03-11T21:32:25.335165+0000", 2024-03-11T21:32:47.044 INFO:teuthology.orchestra.run.smithi097.stdout: "age": 21.613964219, 2024-03-11T21:32:47.044 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.13921125100000001, 2024-03-11T21:32:47.044 INFO:teuthology.orchestra.run.smithi097.stdout: "type_data": { 2024-03-11T21:32:47.044 INFO:teuthology.orchestra.run.smithi097.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-11T21:32:47.044 INFO:teuthology.orchestra.run.smithi097.stdout: "client_info": { 2024-03-11T21:32:47.044 INFO:teuthology.orchestra.run.smithi097.stdout: "client": "client.15150", 2024-03-11T21:32:47.044 INFO:teuthology.orchestra.run.smithi097.stdout: "client_addr": "v1:172.21.15.92:0/436606717", 2024-03-11T21:32:47.044 INFO:teuthology.orchestra.run.smithi097.stdout: "tid": 49201 2024-03-11T21:32:47.044 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:32:47.045 INFO:teuthology.orchestra.run.smithi097.stdout: "events": [ 2024-03-11T21:32:47.045 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:32:47.045 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "initiated", 2024-03-11T21:32:47.045 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:32:25.335165+0000", 2024-03-11T21:32:47.045 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-03-11T21:32:47.045 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:32:47.045 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:32:47.045 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "header_read", 2024-03-11T21:32:47.045 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:32:25.335165+0000", 2024-03-11T21:32:47.045 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-03-11T21:32:47.045 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:32:47.045 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:32:47.045 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "throttled", 2024-03-11T21:32:47.045 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:32:25.335166+0000", 2024-03-11T21:32:47.046 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.8670000000000001e-06 2024-03-11T21:32:47.046 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:32:47.046 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:32:47.046 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "all_read", 2024-03-11T21:32:47.046 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:32:25.335195+0000", 2024-03-11T21:32:47.046 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 2.8251000000000001e-05 2024-03-11T21:32:47.046 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:32:47.046 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:32:47.046 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "dispatched", 2024-03-11T21:32:47.046 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:32:25.335207+0000", 2024-03-11T21:32:47.046 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.2598e-05 2024-03-11T21:32:47.046 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:32:47.046 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:32:47.046 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "queued_for_pg", 2024-03-11T21:32:47.046 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:32:25.335215+0000", 2024-03-11T21:32:47.047 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 7.7330000000000003e-06 2024-03-11T21:32:47.047 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:32:47.047 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:32:47.047 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "reached_pg", 2024-03-11T21:32:47.047 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:32:25.335330+0000", 2024-03-11T21:32:47.047 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.000115059 2024-03-11T21:32:47.047 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:32:47.047 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:32:47.047 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "started", 2024-03-11T21:32:47.047 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:32:25.335441+0000", 2024-03-11T21:32:47.047 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.00011127400000000001 2024-03-11T21:32:47.047 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:32:47.047 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:32:47.047 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "waiting for subops from 0,5", 2024-03-11T21:32:47.048 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:32:25.335571+0000", 2024-03-11T21:32:47.048 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.00012946600000000001 2024-03-11T21:32:47.048 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:32:47.048 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:32:47.048 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "op_commit", 2024-03-11T21:32:47.048 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:32:25.337253+0000", 2024-03-11T21:32:47.048 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.001682689 2024-03-11T21:32:47.048 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:32:47.048 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:32:47.048 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "sub_op_commit_rec", 2024-03-11T21:32:47.048 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:32:25.338470+0000", 2024-03-11T21:32:47.048 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.001216312 2024-03-11T21:32:47.049 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:32:47.049 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:32:47.049 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "sub_op_commit_rec", 2024-03-11T21:32:47.049 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:32:25.474286+0000", 2024-03-11T21:32:47.049 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.135815931 2024-03-11T21:32:47.049 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:32:47.049 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:32:47.049 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "commit_sent", 2024-03-11T21:32:47.049 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:32:25.474340+0000", 2024-03-11T21:32:47.049 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 5.4073000000000002e-05 2024-03-11T21:32:47.049 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:32:47.049 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:32:47.049 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "done", 2024-03-11T21:32:47.049 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:32:25.474376+0000", 2024-03-11T21:32:47.049 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 3.5998000000000003e-05 2024-03-11T21:32:47.050 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-03-11T21:32:47.050 INFO:teuthology.orchestra.run.smithi097.stdout: ] 2024-03-11T21:32:47.050 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-03-11T21:32:47.050 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:32:47.050 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:32:47.050 INFO:teuthology.orchestra.run.smithi097.stdout: "description": "osd_op(client.15150.0:49930 8.2 8:49571121:::benchmark_data_smithi092_104865_object49929:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e169)", 2024-03-11T21:32:47.050 INFO:teuthology.orchestra.run.smithi097.stdout: "initiated_at": "2024-03-11T21:32:25.949935+0000", 2024-03-11T21:32:47.050 INFO:teuthology.orchestra.run.smithi097.stdout: "age": 20.999194013, 2024-03-11T21:32:47.050 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.078003341000000004, 2024-03-11T21:32:47.050 INFO:teuthology.orchestra.run.smithi097.stdout: "type_data": { 2024-03-11T21:32:47.050 INFO:teuthology.orchestra.run.smithi097.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-11T21:32:47.050 INFO:teuthology.orchestra.run.smithi097.stdout: "client_info": { 2024-03-11T21:32:47.050 INFO:teuthology.orchestra.run.smithi097.stdout: "client": "client.15150", 2024-03-11T21:32:47.051 INFO:teuthology.orchestra.run.smithi097.stdout: "client_addr": "v1:172.21.15.92:0/436606717", 2024-03-11T21:32:47.051 INFO:teuthology.orchestra.run.smithi097.stdout: "tid": 49930 2024-03-11T21:32:47.051 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:32:47.051 INFO:teuthology.orchestra.run.smithi097.stdout: "events": [ 2024-03-11T21:32:47.051 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:32:47.051 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "initiated", 2024-03-11T21:32:47.051 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:32:25.949935+0000", 2024-03-11T21:32:47.051 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-03-11T21:32:47.051 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:32:47.051 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:32:47.051 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "header_read", 2024-03-11T21:32:47.051 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:32:25.949935+0000", 2024-03-11T21:32:47.051 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-03-11T21:32:47.051 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:32:47.051 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:32:47.052 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "throttled", 2024-03-11T21:32:47.052 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:32:25.949938+0000", 2024-03-11T21:32:47.052 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 2.9529999999999999e-06 2024-03-11T21:32:47.052 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:32:47.052 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:32:47.052 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "all_read", 2024-03-11T21:32:47.052 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:32:25.950096+0000", 2024-03-11T21:32:47.052 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.00015875500000000001 2024-03-11T21:32:47.052 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:32:47.052 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:32:47.052 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "dispatched", 2024-03-11T21:32:47.052 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:32:25.950114+0000", 2024-03-11T21:32:47.052 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.7571000000000002e-05 2024-03-11T21:32:47.052 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:32:47.052 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:32:47.053 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "queued_for_pg", 2024-03-11T21:32:47.053 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:32:25.950122+0000", 2024-03-11T21:32:47.053 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 8.0600000000000008e-06 2024-03-11T21:32:47.053 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:32:47.053 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:32:47.053 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "reached_pg", 2024-03-11T21:32:47.053 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:32:25.950377+0000", 2024-03-11T21:32:47.053 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.00025509300000000002 2024-03-11T21:32:47.053 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:32:47.053 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:32:47.053 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "started", 2024-03-11T21:32:47.053 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:32:25.950560+0000", 2024-03-11T21:32:47.053 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.00018232599999999999 2024-03-11T21:32:47.053 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:32:47.053 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:32:47.054 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "waiting for subops from 0,5", 2024-03-11T21:32:47.054 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:32:25.950731+0000", 2024-03-11T21:32:47.054 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.000171721 2024-03-11T21:32:47.054 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:32:47.054 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:32:47.054 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "op_commit", 2024-03-11T21:32:47.054 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:32:25.952771+0000", 2024-03-11T21:32:47.054 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.0020393239999999999 2024-03-11T21:32:47.054 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:32:47.054 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:32:47.054 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "sub_op_commit_rec", 2024-03-11T21:32:47.054 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:32:25.954913+0000", 2024-03-11T21:32:47.054 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.0021423409999999999 2024-03-11T21:32:47.054 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:32:47.054 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:32:47.055 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "sub_op_commit_rec", 2024-03-11T21:32:47.055 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:32:26.027868+0000", 2024-03-11T21:32:47.055 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.072954668 2024-03-11T21:32:47.055 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:32:47.055 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:32:47.055 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "commit_sent", 2024-03-11T21:32:47.055 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:32:26.027903+0000", 2024-03-11T21:32:47.055 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 3.5160000000000002e-05 2024-03-11T21:32:47.055 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:32:47.055 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:32:47.055 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "done", 2024-03-11T21:32:47.055 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:32:26.027938+0000", 2024-03-11T21:32:47.055 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 3.5369e-05 2024-03-11T21:32:47.055 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-03-11T21:32:47.056 INFO:teuthology.orchestra.run.smithi097.stdout: ] 2024-03-11T21:32:47.056 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-03-11T21:32:47.056 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-03-11T21:32:47.056 INFO:teuthology.orchestra.run.smithi097.stdout: ] 2024-03-11T21:32:47.056 INFO:teuthology.orchestra.run.smithi097.stdout:} 2024-03-11T21:32:47.056 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:32:47.098 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:32:47.098 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:32:47.099 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:32:47.199 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:32:47.224 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:32:47.225 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:32:47.225 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:32:47.314 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:32:46 smithi097 ceph-mon[33317]: pgmap v283: 148 pgs: 148 active+clean; 4.4 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 744 B/s rd, 56 MiB/s wr, 888 op/s 2024-03-11T21:32:47.325 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:32:47.360 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:32:46 smithi092 ceph-mon[29572]: pgmap v283: 148 pgs: 148 active+clean; 4.4 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 744 B/s rd, 56 MiB/s wr, 888 op/s 2024-03-11T21:32:47.360 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:32:46 smithi092 ceph-mon[37563]: pgmap v283: 148 pgs: 148 active+clean; 4.4 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 744 B/s rd, 56 MiB/s wr, 888 op/s 2024-03-11T21:32:47.365 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:32:47.366 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:32:47.366 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:32:47.434 INFO:tasks.radosbench.radosbench.0.smithi092.stdout: 46 16 77791 77775 105.654 72.125 0.011116 0.00945909 2024-03-11T21:32:47.467 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:32:47.507 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:32:47.507 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:32:47.507 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:32:47.608 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:32:47.651 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:32:47.652 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:32:47.652 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:32:47.753 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:32:47.793 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:32:47.794 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:32:47.794 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:32:47.795 DEBUG:teuthology.orchestra.run.smithi097:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid 00db0016-dfed-11ee-95c8-87774f69a715 -- ceph daemon osd.5 dump_ops_in_flight 2024-03-11T21:32:47.896 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:32:47.933 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:32:47.934 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:32:47.934 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:32:48.029 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:32:47 smithi097 ceph-mon[33317]: purged_snaps scrub starts 2024-03-11T21:32:48.029 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:32:47 smithi097 ceph-mon[33317]: purged_snaps scrub ok 2024-03-11T21:32:48.029 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:32:47 smithi097 ceph-mon[33317]: osdmap e176: 8 total, 8 up, 8 in 2024-03-11T21:32:48.035 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:32:48.079 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:32:48.079 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:32:48.080 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:32:48.181 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:32:48.193 INFO:teuthology.orchestra.run.smithi097.stderr:Inferring config /var/lib/ceph/00db0016-dfed-11ee-95c8-87774f69a715/mon.b/config 2024-03-11T21:32:48.210 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:32:48.210 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:32:48.210 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:32:48.289 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:32:47 smithi092 ceph-mon[29572]: purged_snaps scrub starts 2024-03-11T21:32:48.289 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:32:47 smithi092 ceph-mon[29572]: purged_snaps scrub ok 2024-03-11T21:32:48.289 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:32:47 smithi092 ceph-mon[29572]: osdmap e176: 8 total, 8 up, 8 in 2024-03-11T21:32:48.289 INFO:journalctl@ceph.mgr.y.smithi092.stdout:Mar 11 21:32:47 smithi092 ceph-00db0016-dfed-11ee-95c8-87774f69a715-mgr-y[29801]: ::ffff:172.21.15.97 - - [11/Mar/2024:21:32:47] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-11T21:32:48.289 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:32:47 smithi092 ceph-mon[37563]: purged_snaps scrub starts 2024-03-11T21:32:48.289 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:32:47 smithi092 ceph-mon[37563]: purged_snaps scrub ok 2024-03-11T21:32:48.290 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:32:47 smithi092 ceph-mon[37563]: osdmap e176: 8 total, 8 up, 8 in 2024-03-11T21:32:48.311 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:32:48.350 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:32:48.351 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:32:48.352 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:32:48.434 INFO:tasks.radosbench.radosbench.0.smithi092.stdout: 47 16 78965 78949 104.967 73.375 0.00267977 0.00951408 2024-03-11T21:32:48.452 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:32:48.493 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:32:48.493 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:32:48.494 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:32:48.594 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:32:48.636 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:32:48.636 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:32:48.636 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:32:48.737 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:32:48.773 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:32:48.773 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:32:48.774 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:32:48.874 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:32:48.910 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:32:48.910 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:32:48.911 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:32:49.012 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:32:49.025 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:32:48 smithi092 ceph-mon[29572]: pgmap v285: 148 pgs: 148 active+clean; 4.5 GiB data, 15 GiB used, 701 GiB / 715 GiB avail; 853 B/s rd, 79 MiB/s wr, 1.26k op/s 2024-03-11T21:32:49.025 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:32:48 smithi092 ceph-mon[29572]: osdmap e177: 8 total, 8 up, 8 in 2024-03-11T21:32:49.025 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:32:48 smithi092 ceph-mon[37563]: pgmap v285: 148 pgs: 148 active+clean; 4.5 GiB data, 15 GiB used, 701 GiB / 715 GiB avail; 853 B/s rd, 79 MiB/s wr, 1.26k op/s 2024-03-11T21:32:49.025 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:32:48 smithi092 ceph-mon[37563]: osdmap e177: 8 total, 8 up, 8 in 2024-03-11T21:32:49.045 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:32:49.046 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:32:49.046 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:32:49.143 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:32:48 smithi097 ceph-mon[33317]: pgmap v285: 148 pgs: 148 active+clean; 4.5 GiB data, 15 GiB used, 701 GiB / 715 GiB avail; 853 B/s rd, 79 MiB/s wr, 1.26k op/s 2024-03-11T21:32:49.144 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:32:48 smithi097 ceph-mon[33317]: osdmap e177: 8 total, 8 up, 8 in 2024-03-11T21:32:49.147 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:32:49.188 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:32:49.189 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:32:49.189 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:32:49.290 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:32:49.324 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:32:49.325 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:32:49.325 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:32:49.426 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:32:49.434 INFO:tasks.radosbench.radosbench.0.smithi092.stdout: 48 16 80245 80229 104.447 80 0.0149463 0.00956935 2024-03-11T21:32:49.461 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:32:49.461 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:32:49.461 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:32:49.562 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:32:49.597 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:32:49.598 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:32:49.598 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:32:49.622 DEBUG:teuthology.orchestra.run.smithi092:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid 00db0016-dfed-11ee-95c8-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-03-11T21:32:49.699 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:32:49.731 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:32:49.731 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:32:49.731 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:32:49.832 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:32:49.885 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:32:49.885 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:32:49.886 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:32:49.987 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:32:50.027 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:32:50.028 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:32:50.028 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:32:50.129 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:32:50.162 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:32:50.163 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:32:50.163 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:32:50.264 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:32:50.302 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:32:50.303 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:32:50.303 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:32:50.404 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:32:50.434 INFO:tasks.radosbench.radosbench.0.smithi092.stdout: 49 16 81429 81413 103.825 74 0.0113584 0.00962637 2024-03-11T21:32:50.442 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:32:50.443 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:32:50.443 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:32:50.544 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:32:50.581 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:32:50.582 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:32:50.582 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:32:50.683 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:32:50.722 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:32:50.723 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:32:50.724 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:32:50.824 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:32:50.857 INFO:teuthology.orchestra.run.smithi097.stdout:{ 2024-03-11T21:32:50.857 INFO:teuthology.orchestra.run.smithi097.stdout: "ops": [ 2024-03-11T21:32:50.857 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:32:50.857 INFO:teuthology.orchestra.run.smithi097.stdout: "description": "osd_op(client.15150.0:81901 8.d 8:bd15cd8f:::benchmark_data_smithi092_104865_object81900:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536 in=65536b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e177)", 2024-03-11T21:32:50.857 INFO:teuthology.orchestra.run.smithi097.stdout: "initiated_at": "2024-03-11T21:32:50.864275+0000", 2024-03-11T21:32:50.858 INFO:teuthology.orchestra.run.smithi097.stdout: "age": 0.010184121000000001, 2024-03-11T21:32:50.858 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.010217992, 2024-03-11T21:32:50.858 INFO:teuthology.orchestra.run.smithi097.stdout: "type_data": { 2024-03-11T21:32:50.858 INFO:teuthology.orchestra.run.smithi097.stdout: "flag_point": "waiting for sub ops", 2024-03-11T21:32:50.858 INFO:teuthology.orchestra.run.smithi097.stdout: "client_info": { 2024-03-11T21:32:50.859 INFO:teuthology.orchestra.run.smithi097.stdout: "client": "client.15150", 2024-03-11T21:32:50.859 INFO:teuthology.orchestra.run.smithi097.stdout: "client_addr": "v1:172.21.15.92:0/436606717", 2024-03-11T21:32:50.859 INFO:teuthology.orchestra.run.smithi097.stdout: "tid": 81901 2024-03-11T21:32:50.859 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:32:50.859 INFO:teuthology.orchestra.run.smithi097.stdout: "events": [ 2024-03-11T21:32:50.859 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:32:50.859 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "initiated", 2024-03-11T21:32:50.859 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:32:50.864275+0000", 2024-03-11T21:32:50.859 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-03-11T21:32:50.859 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:32:50.860 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:32:50.860 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "header_read", 2024-03-11T21:32:50.860 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:32:50.864275+0000", 2024-03-11T21:32:50.860 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-03-11T21:32:50.860 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:32:50.860 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:32:50.860 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "throttled", 2024-03-11T21:32:50.860 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:32:50.864277+0000", 2024-03-11T21:32:50.860 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.8789999999999999e-06 2024-03-11T21:32:50.860 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:32:50.860 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:32:50.860 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "all_read", 2024-03-11T21:32:50.860 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:32:50.864313+0000", 2024-03-11T21:32:50.860 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 3.5760000000000003e-05 2024-03-11T21:32:50.860 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:32:50.861 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:32:50.861 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "dispatched", 2024-03-11T21:32:50.861 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:32:50.864322+0000", 2024-03-11T21:32:50.861 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 9.7110000000000007e-06 2024-03-11T21:32:50.861 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:32:50.861 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:32:50.861 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "queued_for_pg", 2024-03-11T21:32:50.861 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:32:50.864330+0000", 2024-03-11T21:32:50.861 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 7.7929999999999998e-06 2024-03-11T21:32:50.861 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:32:50.861 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:32:50.861 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "reached_pg", 2024-03-11T21:32:50.861 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:32:50.864524+0000", 2024-03-11T21:32:50.861 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.000194043 2024-03-11T21:32:50.862 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:32:50.862 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:32:50.862 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "started", 2024-03-11T21:32:50.862 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:32:50.864641+0000", 2024-03-11T21:32:50.862 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.00011701000000000001 2024-03-11T21:32:50.862 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:32:50.862 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:32:50.862 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "waiting for subops from 2,6", 2024-03-11T21:32:50.862 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:32:50.864782+0000", 2024-03-11T21:32:50.862 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.00014087 2024-03-11T21:32:50.862 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:32:50.862 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:32:50.862 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "sub_op_commit_rec", 2024-03-11T21:32:50.862 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:32:50.873004+0000", 2024-03-11T21:32:50.863 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.0082218099999999995 2024-03-11T21:32:50.863 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:32:50.863 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:32:50.863 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "op_commit", 2024-03-11T21:32:50.863 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:32:50.874115+0000", 2024-03-11T21:32:50.863 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.0011110270000000001 2024-03-11T21:32:50.863 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-03-11T21:32:50.863 INFO:teuthology.orchestra.run.smithi097.stdout: ] 2024-03-11T21:32:50.863 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-03-11T21:32:50.863 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:32:50.863 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:32:50.863 INFO:teuthology.orchestra.run.smithi097.stdout: "description": "osd_op(client.15150.0:81904 8.d 8:b063cd62:::benchmark_data_smithi092_104865_object81903:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536 in=65536b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e177)", 2024-03-11T21:32:50.863 INFO:teuthology.orchestra.run.smithi097.stdout: "initiated_at": "2024-03-11T21:32:50.864344+0000", 2024-03-11T21:32:50.864 INFO:teuthology.orchestra.run.smithi097.stdout: "age": 0.010115262999999999, 2024-03-11T21:32:50.864 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.010243449999999999, 2024-03-11T21:32:50.864 INFO:teuthology.orchestra.run.smithi097.stdout: "type_data": { 2024-03-11T21:32:50.865 INFO:teuthology.orchestra.run.smithi097.stdout: "flag_point": "waiting for sub ops", 2024-03-11T21:32:50.865 INFO:teuthology.orchestra.run.smithi097.stdout: "client_info": { 2024-03-11T21:32:50.865 INFO:teuthology.orchestra.run.smithi097.stdout: "client": "client.15150", 2024-03-11T21:32:50.865 INFO:teuthology.orchestra.run.smithi097.stdout: "client_addr": "v1:172.21.15.92:0/436606717", 2024-03-11T21:32:50.865 INFO:teuthology.orchestra.run.smithi097.stdout: "tid": 81904 2024-03-11T21:32:50.865 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:32:50.866 INFO:teuthology.orchestra.run.smithi097.stdout: "events": [ 2024-03-11T21:32:50.866 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:32:50.866 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "initiated", 2024-03-11T21:32:50.866 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:32:50.864344+0000", 2024-03-11T21:32:50.866 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-03-11T21:32:50.866 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:32:50.866 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:32:50.866 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "header_read", 2024-03-11T21:32:50.866 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:32:50.864344+0000", 2024-03-11T21:32:50.866 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-03-11T21:32:50.866 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:32:50.867 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:32:50.867 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "throttled", 2024-03-11T21:32:50.867 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:32:50.864345+0000", 2024-03-11T21:32:50.867 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 8.7000000000000003e-07 2024-03-11T21:32:50.867 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:32:50.867 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:32:50.867 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "all_read", 2024-03-11T21:32:50.867 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:32:50.864377+0000", 2024-03-11T21:32:50.867 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 3.1832000000000003e-05 2024-03-11T21:32:50.867 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:32:50.867 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:32:50.867 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "dispatched", 2024-03-11T21:32:50.867 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:32:50.864383+0000", 2024-03-11T21:32:50.867 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 6.3899999999999998e-06 2024-03-11T21:32:50.867 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:32:50.868 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:32:50.868 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "queued_for_pg", 2024-03-11T21:32:50.868 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:32:50.864390+0000", 2024-03-11T21:32:50.868 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 6.7089999999999996e-06 2024-03-11T21:32:50.868 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:32:50.868 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:32:50.868 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "reached_pg", 2024-03-11T21:32:50.868 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:32:50.870173+0000", 2024-03-11T21:32:50.868 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.0057837929999999997 2024-03-11T21:32:50.868 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:32:50.868 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:32:50.868 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "started", 2024-03-11T21:32:50.868 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:32:50.870302+0000", 2024-03-11T21:32:50.868 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.000128682 2024-03-11T21:32:50.868 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:32:50.869 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:32:50.869 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "waiting for subops from 2,6", 2024-03-11T21:32:50.869 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:32:50.870453+0000", 2024-03-11T21:32:50.869 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.00015107900000000001 2024-03-11T21:32:50.869 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:32:50.869 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:32:50.869 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "op_commit", 2024-03-11T21:32:50.869 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:32:50.874119+0000", 2024-03-11T21:32:50.869 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.0036663009999999998 2024-03-11T21:32:50.869 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-03-11T21:32:50.869 INFO:teuthology.orchestra.run.smithi097.stdout: ] 2024-03-11T21:32:50.870 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-03-11T21:32:50.870 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:32:50.870 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:32:50.870 INFO:teuthology.orchestra.run.smithi097.stdout: "description": "osd_repop(client.15150.0:81903 8.9 e177/157 8:9a924ef9:::benchmark_data_smithi092_104865_object81902:head v 177'5203, mlcod=177'5201)", 2024-03-11T21:32:50.870 INFO:teuthology.orchestra.run.smithi097.stdout: "initiated_at": "2024-03-11T21:32:50.871017+0000", 2024-03-11T21:32:50.870 INFO:teuthology.orchestra.run.smithi097.stdout: "age": 0.0034422649999999999, 2024-03-11T21:32:50.870 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.0036553079999999999, 2024-03-11T21:32:50.870 INFO:teuthology.orchestra.run.smithi097.stdout: "type_data": { 2024-03-11T21:32:50.870 INFO:teuthology.orchestra.run.smithi097.stdout: "flag_point": "started", 2024-03-11T21:32:50.870 INFO:teuthology.orchestra.run.smithi097.stdout: "events": [ 2024-03-11T21:32:50.870 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:32:50.870 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "initiated", 2024-03-11T21:32:50.871 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:32:50.871017+0000", 2024-03-11T21:32:50.871 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-03-11T21:32:50.871 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:32:50.871 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:32:50.871 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "header_read", 2024-03-11T21:32:50.871 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:32:50.871017+0000", 2024-03-11T21:32:50.871 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-03-11T21:32:50.871 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:32:50.871 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:32:50.871 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "throttled", 2024-03-11T21:32:50.871 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:32:50.871018+0000", 2024-03-11T21:32:50.871 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.1650000000000001e-06 2024-03-11T21:32:50.871 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:32:50.871 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:32:50.872 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "all_read", 2024-03-11T21:32:50.872 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:32:50.871043+0000", 2024-03-11T21:32:50.872 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 2.5392999999999999e-05 2024-03-11T21:32:50.872 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:32:50.872 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:32:50.872 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "dispatched", 2024-03-11T21:32:50.872 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:32:50.871057+0000", 2024-03-11T21:32:50.872 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.3789e-05 2024-03-11T21:32:50.872 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:32:50.872 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:32:50.872 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "queued_for_pg", 2024-03-11T21:32:50.872 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:32:50.871064+0000", 2024-03-11T21:32:50.872 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 6.5130000000000002e-06 2024-03-11T21:32:50.872 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:32:50.872 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:32:50.873 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "reached_pg", 2024-03-11T21:32:50.873 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:32:50.871168+0000", 2024-03-11T21:32:50.873 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.00010466799999999999 2024-03-11T21:32:50.873 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:32:50.873 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:32:50.873 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "started", 2024-03-11T21:32:50.873 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:32:50.871188+0000", 2024-03-11T21:32:50.873 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.982e-05 2024-03-11T21:32:50.873 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-03-11T21:32:50.873 INFO:teuthology.orchestra.run.smithi097.stdout: ] 2024-03-11T21:32:50.873 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-03-11T21:32:50.874 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:32:50.874 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:32:50.875 INFO:teuthology.orchestra.run.smithi097.stdout: "description": "osd_repop(client.15150.0:81908 8.9 e177/157 8:9ec43dc2:::benchmark_data_smithi092_104865_object81907:head v 177'5204, mlcod=177'5201)", 2024-03-11T21:32:50.875 INFO:teuthology.orchestra.run.smithi097.stdout: "initiated_at": "2024-03-11T21:32:50.871742+0000", 2024-03-11T21:32:50.875 INFO:teuthology.orchestra.run.smithi097.stdout: "age": 0.002717358, 2024-03-11T21:32:50.875 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.002988671, 2024-03-11T21:32:50.875 INFO:teuthology.orchestra.run.smithi097.stdout: "type_data": { 2024-03-11T21:32:50.875 INFO:teuthology.orchestra.run.smithi097.stdout: "flag_point": "started", 2024-03-11T21:32:50.875 INFO:teuthology.orchestra.run.smithi097.stdout: "events": [ 2024-03-11T21:32:50.875 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:32:50.875 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "initiated", 2024-03-11T21:32:50.875 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:32:50.871742+0000", 2024-03-11T21:32:50.875 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-03-11T21:32:50.875 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:32:50.875 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:32:50.875 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "header_read", 2024-03-11T21:32:50.876 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:32:50.871742+0000", 2024-03-11T21:32:50.876 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-03-11T21:32:50.876 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:32:50.876 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:32:50.876 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "throttled", 2024-03-11T21:32:50.876 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:32:50.871743+0000", 2024-03-11T21:32:50.876 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 9.09e-07 2024-03-11T21:32:50.876 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:32:50.876 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:32:50.876 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "all_read", 2024-03-11T21:32:50.876 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:32:50.871766+0000", 2024-03-11T21:32:50.876 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 2.2989999999999998e-05 2024-03-11T21:32:50.876 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:32:50.876 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:32:50.877 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "dispatched", 2024-03-11T21:32:50.877 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:32:50.871775+0000", 2024-03-11T21:32:50.877 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 9.2599999999999994e-06 2024-03-11T21:32:50.877 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:32:50.877 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:32:50.877 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "queued_for_pg", 2024-03-11T21:32:50.877 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:32:50.871782+0000", 2024-03-11T21:32:50.877 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 6.8909999999999998e-06 2024-03-11T21:32:50.877 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:32:50.877 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:32:50.877 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "reached_pg", 2024-03-11T21:32:50.877 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:32:50.874360+0000", 2024-03-11T21:32:50.877 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.0025779459999999998 2024-03-11T21:32:50.877 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:32:50.877 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:32:50.878 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "started", 2024-03-11T21:32:50.878 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:32:50.874380+0000", 2024-03-11T21:32:50.878 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 2.0250000000000001e-05 2024-03-11T21:32:50.878 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-03-11T21:32:50.878 INFO:teuthology.orchestra.run.smithi097.stdout: ] 2024-03-11T21:32:50.878 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-03-11T21:32:50.878 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:32:50.878 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:32:50.878 INFO:teuthology.orchestra.run.smithi097.stdout: "description": "osd_op(client.15150.0:81914 8.4 8:2ac356bc:::benchmark_data_smithi092_104865_object81913:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536 in=65536b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e177)", 2024-03-11T21:32:50.878 INFO:teuthology.orchestra.run.smithi097.stdout: "initiated_at": "2024-03-11T21:32:50.873159+0000", 2024-03-11T21:32:50.878 INFO:teuthology.orchestra.run.smithi097.stdout: "age": 0.0013005180000000001, 2024-03-11T21:32:50.878 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.0016328899999999999, 2024-03-11T21:32:50.878 INFO:teuthology.orchestra.run.smithi097.stdout: "type_data": { 2024-03-11T21:32:50.878 INFO:teuthology.orchestra.run.smithi097.stdout: "flag_point": "waiting for sub ops", 2024-03-11T21:32:50.879 INFO:teuthology.orchestra.run.smithi097.stdout: "client_info": { 2024-03-11T21:32:50.879 INFO:teuthology.orchestra.run.smithi097.stdout: "client": "client.15150", 2024-03-11T21:32:50.879 INFO:teuthology.orchestra.run.smithi097.stdout: "client_addr": "v1:172.21.15.92:0/436606717", 2024-03-11T21:32:50.879 INFO:teuthology.orchestra.run.smithi097.stdout: "tid": 81914 2024-03-11T21:32:50.879 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:32:50.879 INFO:teuthology.orchestra.run.smithi097.stdout: "events": [ 2024-03-11T21:32:50.879 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:32:50.879 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "initiated", 2024-03-11T21:32:50.879 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:32:50.873159+0000", 2024-03-11T21:32:50.879 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-03-11T21:32:50.879 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:32:50.879 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:32:50.879 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "header_read", 2024-03-11T21:32:50.879 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:32:50.873159+0000", 2024-03-11T21:32:50.880 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-03-11T21:32:50.880 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:32:50.880 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:32:50.880 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "throttled", 2024-03-11T21:32:50.880 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:32:50.873161+0000", 2024-03-11T21:32:50.880 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 2.2689999999999998e-06 2024-03-11T21:32:50.880 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:32:50.880 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:32:50.880 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "all_read", 2024-03-11T21:32:50.880 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:32:50.873202+0000", 2024-03-11T21:32:50.880 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 4.1347999999999998e-05 2024-03-11T21:32:50.881 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:32:50.881 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:32:50.881 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "dispatched", 2024-03-11T21:32:50.881 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:32:50.873214+0000", 2024-03-11T21:32:50.881 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.1561e-05 2024-03-11T21:32:50.881 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:32:50.881 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:32:50.881 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "queued_for_pg", 2024-03-11T21:32:50.881 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:32:50.873222+0000", 2024-03-11T21:32:50.881 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 8.191e-06 2024-03-11T21:32:50.881 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:32:50.881 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:32:50.881 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "reached_pg", 2024-03-11T21:32:50.882 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:32:50.873357+0000", 2024-03-11T21:32:50.882 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.00013542400000000001 2024-03-11T21:32:50.882 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:32:50.882 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:32:50.882 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "started", 2024-03-11T21:32:50.882 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:32:50.873487+0000", 2024-03-11T21:32:50.882 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.00013008900000000001 2024-03-11T21:32:50.882 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:32:50.882 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:32:50.882 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "waiting for subops from 2,4", 2024-03-11T21:32:50.882 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:32:50.873648+0000", 2024-03-11T21:32:50.882 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.000160867 2024-03-11T21:32:50.882 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-03-11T21:32:50.882 INFO:teuthology.orchestra.run.smithi097.stdout: ] 2024-03-11T21:32:50.883 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-03-11T21:32:50.883 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:32:50.883 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:32:50.883 INFO:teuthology.orchestra.run.smithi097.stdout: "description": "osd_repop(client.15150.0:81913 8.0 e177/157 8:0029cfbf:::benchmark_data_smithi092_104865_object81912:head v 177'5063, mlcod=177'5061)", 2024-03-11T21:32:50.883 INFO:teuthology.orchestra.run.smithi097.stdout: "initiated_at": "2024-03-11T21:32:50.873743+0000", 2024-03-11T21:32:50.883 INFO:teuthology.orchestra.run.smithi097.stdout: "age": 0.00071581699999999999, 2024-03-11T21:32:50.883 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.0011177369999999999, 2024-03-11T21:32:50.883 INFO:teuthology.orchestra.run.smithi097.stdout: "type_data": { 2024-03-11T21:32:50.883 INFO:teuthology.orchestra.run.smithi097.stdout: "flag_point": "started", 2024-03-11T21:32:50.883 INFO:teuthology.orchestra.run.smithi097.stdout: "events": [ 2024-03-11T21:32:50.883 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:32:50.883 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "initiated", 2024-03-11T21:32:50.883 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:32:50.873743+0000", 2024-03-11T21:32:50.883 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-03-11T21:32:50.883 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:32:50.883 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:32:50.884 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "header_read", 2024-03-11T21:32:50.884 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:32:50.873743+0000", 2024-03-11T21:32:50.884 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-03-11T21:32:50.884 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:32:50.884 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:32:50.884 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "throttled", 2024-03-11T21:32:50.884 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:32:50.873745+0000", 2024-03-11T21:32:50.884 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.3480000000000001e-06 2024-03-11T21:32:50.884 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:32:50.884 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:32:50.884 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "all_read", 2024-03-11T21:32:50.884 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:32:50.873776+0000", 2024-03-11T21:32:50.885 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 3.0997999999999997e-05 2024-03-11T21:32:50.885 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:32:50.885 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:32:50.885 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "dispatched", 2024-03-11T21:32:50.885 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:32:50.873785+0000", 2024-03-11T21:32:50.885 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 8.9230000000000008e-06 2024-03-11T21:32:50.885 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:32:50.885 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:32:50.885 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "queued_for_pg", 2024-03-11T21:32:50.885 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:32:50.873792+0000", 2024-03-11T21:32:50.885 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 7.1500000000000002e-06 2024-03-11T21:32:50.885 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:32:50.885 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:32:50.885 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "reached_pg", 2024-03-11T21:32:50.886 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:32:50.873838+0000", 2024-03-11T21:32:50.886 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 4.6505000000000003e-05 2024-03-11T21:32:50.886 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:32:50.886 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:32:50.886 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "started", 2024-03-11T21:32:50.886 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:32:50.873873+0000", 2024-03-11T21:32:50.886 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 3.5175000000000001e-05 2024-03-11T21:32:50.886 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-03-11T21:32:50.886 INFO:teuthology.orchestra.run.smithi097.stdout: ] 2024-03-11T21:32:50.886 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-03-11T21:32:50.886 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-03-11T21:32:50.886 INFO:teuthology.orchestra.run.smithi097.stdout: ], 2024-03-11T21:32:50.886 INFO:teuthology.orchestra.run.smithi097.stdout: "num_ops": 6 2024-03-11T21:32:50.886 INFO:teuthology.orchestra.run.smithi097.stdout:} 2024-03-11T21:32:50.887 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:32:50.887 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:32:50.888 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:32:50.988 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:32:51.022 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:32:51.022 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:32:51.023 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:32:51.123 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:32:51.186 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:32:51.187 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:32:51.187 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:32:51.205 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:32:50 smithi092 ceph-mon[29572]: pgmap v287: 148 pgs: 148 active+clean; 4.7 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 853 B/s rd, 62 MiB/s wr, 990 op/s 2024-03-11T21:32:51.205 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:32:50 smithi092 ceph-mon[37563]: pgmap v287: 148 pgs: 148 active+clean; 4.7 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 853 B/s rd, 62 MiB/s wr, 990 op/s 2024-03-11T21:32:51.289 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:32:51.326 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:32:50 smithi097 ceph-mon[33317]: pgmap v287: 148 pgs: 148 active+clean; 4.7 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 853 B/s rd, 62 MiB/s wr, 990 op/s 2024-03-11T21:32:51.335 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:32:51.336 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:32:51.336 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:32:51.434 INFO:tasks.radosbench.radosbench.0.smithi092.stdout: 50 16 82629 82613 103.249 75 0.0157934 0.00968039 2024-03-11T21:32:51.437 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:32:51.475 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:32:51.476 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:32:51.476 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:32:51.577 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:32:51.585 INFO:teuthology.orchestra.run.smithi092.stderr:Inferring config /var/lib/ceph/00db0016-dfed-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T21:32:51.649 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:32:51.650 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:32:51.651 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:32:51.704 DEBUG:teuthology.orchestra.run.smithi097:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid 00db0016-dfed-11ee-95c8-87774f69a715 -- ceph daemon osd.5 dump_blocked_ops 2024-03-11T21:32:51.751 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:32:51.803 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:32:51.803 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:32:51.804 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:32:51.904 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:32:51.940 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:32:51.940 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:32:51.941 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:32:52.042 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:32:52.091 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:32:52.092 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:32:52.092 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:32:52.098 INFO:teuthology.orchestra.run.smithi097.stderr:Inferring config /var/lib/ceph/00db0016-dfed-11ee-95c8-87774f69a715/mon.b/config 2024-03-11T21:32:52.194 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:32:52.222 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:32:51 smithi092 ceph-mon[29572]: purged_snaps scrub starts 2024-03-11T21:32:52.222 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:32:51 smithi092 ceph-mon[29572]: purged_snaps scrub ok 2024-03-11T21:32:52.225 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:32:51 smithi092 ceph-mon[29572]: osdmap e178: 8 total, 8 up, 8 in 2024-03-11T21:32:52.225 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:32:51 smithi092 ceph-mon[29572]: from='mgr.14607 v1:172.21.15.92:0/1702717597' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T21:32:52.225 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:32:51 smithi092 ceph-mon[37563]: purged_snaps scrub starts 2024-03-11T21:32:52.226 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:32:51 smithi092 ceph-mon[37563]: purged_snaps scrub ok 2024-03-11T21:32:52.226 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:32:51 smithi092 ceph-mon[37563]: osdmap e178: 8 total, 8 up, 8 in 2024-03-11T21:32:52.226 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:32:51 smithi092 ceph-mon[37563]: from='mgr.14607 v1:172.21.15.92:0/1702717597' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T21:32:52.233 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:32:51 smithi097 ceph-mon[33317]: purged_snaps scrub starts 2024-03-11T21:32:52.233 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:32:51 smithi097 ceph-mon[33317]: purged_snaps scrub ok 2024-03-11T21:32:52.233 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:32:51 smithi097 ceph-mon[33317]: osdmap e178: 8 total, 8 up, 8 in 2024-03-11T21:32:52.233 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:32:51 smithi097 ceph-mon[33317]: from='mgr.14607 v1:172.21.15.92:0/1702717597' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T21:32:52.240 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:32:52.240 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:32:52.241 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:32:52.342 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:32:52.381 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:32:52.381 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:32:52.382 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:32:52.434 INFO:tasks.radosbench.radosbench.0.smithi092.stdout: 51 16 83821 83805 102.685 74.5 0.00761381 0.00973409 2024-03-11T21:32:52.483 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:32:52.518 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:32:52.518 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:32:52.518 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:32:52.553 INFO:journalctl@ceph.iscsi.iscsi.a.smithi097.stdout:Mar 11 21:32:52 smithi097 ceph-00db0016-dfed-11ee-95c8-87774f69a715-iscsi-iscsi-a[60943]: debug there is no tcmu-runner data available 2024-03-11T21:32:52.619 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:32:52.656 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:32:52.656 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:32:52.656 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:32:52.757 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:32:52.796 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:32:52.796 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:32:52.797 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:32:52.897 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:32:52.940 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:32:52.940 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:32:52.940 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:32:53.041 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:32:53.076 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:32:52 smithi097 ceph-mon[33317]: pgmap v289: 148 pgs: 148 active+clean; 4.7 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 853 B/s rd, 62 MiB/s wr, 990 op/s 2024-03-11T21:32:53.077 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:32:53.078 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:32:53.078 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:32:53.168 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:32:52 smithi092 ceph-mon[29572]: pgmap v289: 148 pgs: 148 active+clean; 4.7 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 853 B/s rd, 62 MiB/s wr, 990 op/s 2024-03-11T21:32:53.169 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:32:52 smithi092 ceph-mon[37563]: pgmap v289: 148 pgs: 148 active+clean; 4.7 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 853 B/s rd, 62 MiB/s wr, 990 op/s 2024-03-11T21:32:53.179 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:32:53.223 INFO:teuthology.orchestra.run.smithi092.stderr:noscrub is unset 2024-03-11T21:32:53.224 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:32:53.225 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:32:53.226 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:32:53.326 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:32:53.364 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:32:53.365 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:32:53.365 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:32:53.434 INFO:tasks.radosbench.radosbench.0.smithi092.stdout: 52 16 84968 84952 102.089 71.6875 0.007066 0.00979082 2024-03-11T21:32:53.467 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:32:53.505 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:32:53.506 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:32:53.506 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:32:53.607 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:32:53.641 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:32:53.641 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:32:53.641 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:32:53.742 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:32:53.773 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:32:53.774 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:32:53.774 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:32:53.875 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:32:53.917 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:32:53.917 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:32:53.918 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:32:54.018 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:32:54.054 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:32:54.054 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:32:54.055 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:32:54.110 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:32:53 smithi092 ceph-mon[29572]: from='client.24469 v1:172.21.15.97:0/404208310' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:32:54.110 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:32:53 smithi092 ceph-mon[29572]: from='client.? v1:172.21.15.92:0/3696225932' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-11T21:32:54.110 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:32:53 smithi092 ceph-mon[29572]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-11T21:32:54.110 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:32:53 smithi092 ceph-mon[29572]: from='client.? v1:172.21.15.92:0/3696225932' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-03-11T21:32:54.110 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:32:53 smithi092 ceph-mon[29572]: osdmap e179: 8 total, 8 up, 8 in 2024-03-11T21:32:54.111 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:32:53 smithi092 ceph-mon[37563]: from='client.24469 v1:172.21.15.97:0/404208310' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:32:54.111 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:32:53 smithi092 ceph-mon[37563]: from='client.? v1:172.21.15.92:0/3696225932' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-11T21:32:54.111 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:32:53 smithi092 ceph-mon[37563]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-11T21:32:54.111 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:32:53 smithi092 ceph-mon[37563]: from='client.? v1:172.21.15.92:0/3696225932' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-03-11T21:32:54.111 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:32:53 smithi092 ceph-mon[37563]: osdmap e179: 8 total, 8 up, 8 in 2024-03-11T21:32:54.155 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:32:54.186 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:32:54.225 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:32:54.226 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:32:54.250 INFO:teuthology.orchestra.run.smithi097.stdout:{ 2024-03-11T21:32:54.250 INFO:teuthology.orchestra.run.smithi097.stdout: "ops": [], 2024-03-11T21:32:54.250 INFO:teuthology.orchestra.run.smithi097.stdout: "complaint_time": 30, 2024-03-11T21:32:54.250 INFO:teuthology.orchestra.run.smithi097.stdout: "num_blocked_ops": 0 2024-03-11T21:32:54.250 INFO:teuthology.orchestra.run.smithi097.stdout:} 2024-03-11T21:32:54.251 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:32:53 smithi097 ceph-mon[33317]: from='client.24469 v1:172.21.15.97:0/404208310' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:32:54.251 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:32:54 smithi097 ceph-mon[33317]: from='client.? v1:172.21.15.92:0/3696225932' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-11T21:32:54.251 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:32:54 smithi097 ceph-mon[33317]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-11T21:32:54.251 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:32:54 smithi097 ceph-mon[33317]: from='client.? v1:172.21.15.92:0/3696225932' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-03-11T21:32:54.251 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:32:54 smithi097 ceph-mon[33317]: osdmap e179: 8 total, 8 up, 8 in 2024-03-11T21:32:54.327 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:32:54.368 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:32:54.368 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:32:54.369 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:32:54.434 INFO:tasks.radosbench.radosbench.0.smithi092.stdout: 53 16 86110 86094 101.509 71.375 0.00829169 0.00984623 2024-03-11T21:32:54.470 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:32:54.512 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:32:54.512 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:32:54.513 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:32:54.614 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:32:54.649 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:32:54.649 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:32:54.650 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:32:54.751 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:32:54.793 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:32:54.793 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:32:54.793 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:32:54.894 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:32:54.928 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:32:54.929 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:32:54.929 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:32:54.975 DEBUG:teuthology.orchestra.run.smithi097:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid 00db0016-dfed-11ee-95c8-87774f69a715 -- ceph daemon osd.5 dump_historic_ops 2024-03-11T21:32:55.012 INFO:teuthology.orchestra.run.smithi092.stderr:set pool 8 pg_num to 26 2024-03-11T21:32:55.030 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:32:55.069 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:32:55.070 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:32:55.070 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:32:55.110 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:32:55 smithi092 ceph-mon[29572]: pgmap v290: 148 pgs: 148 active+clean; 5.1 GiB data, 17 GiB used, 699 GiB / 715 GiB avail; 1.6 KiB/s rd, 121 MiB/s wr, 1.94k op/s 2024-03-11T21:32:55.110 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:32:55 smithi092 ceph-mon[29572]: 4.1e scrub starts 2024-03-11T21:32:55.111 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:32:55 smithi092 ceph-mon[29572]: 4.1e scrub ok 2024-03-11T21:32:55.111 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:32:55 smithi092 ceph-mon[29572]: 4.e scrub starts 2024-03-11T21:32:55.111 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:32:55 smithi092 ceph-mon[29572]: 4.e scrub ok 2024-03-11T21:32:55.111 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:32:55 smithi092 ceph-mon[29572]: from='client.? v1:172.21.15.92:0/341582292' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "unique_pool_1", "var": "pg_num", "val": "26"}]: dispatch 2024-03-11T21:32:55.111 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:32:55 smithi092 ceph-mon[37563]: pgmap v290: 148 pgs: 148 active+clean; 5.1 GiB data, 17 GiB used, 699 GiB / 715 GiB avail; 1.6 KiB/s rd, 121 MiB/s wr, 1.94k op/s 2024-03-11T21:32:55.111 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:32:55 smithi092 ceph-mon[37563]: 4.1e scrub starts 2024-03-11T21:32:55.111 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:32:55 smithi092 ceph-mon[37563]: 4.1e scrub ok 2024-03-11T21:32:55.111 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:32:55 smithi092 ceph-mon[37563]: 4.e scrub starts 2024-03-11T21:32:55.111 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:32:55 smithi092 ceph-mon[37563]: 4.e scrub ok 2024-03-11T21:32:55.111 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:32:55 smithi092 ceph-mon[37563]: from='client.? v1:172.21.15.92:0/341582292' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "unique_pool_1", "var": "pg_num", "val": "26"}]: dispatch 2024-03-11T21:32:55.171 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:32:55.209 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:32:55.209 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:32:55.210 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:32:55.311 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:32:55.329 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:32:55 smithi097 ceph-mon[33317]: pgmap v290: 148 pgs: 148 active+clean; 5.1 GiB data, 17 GiB used, 699 GiB / 715 GiB avail; 1.6 KiB/s rd, 121 MiB/s wr, 1.94k op/s 2024-03-11T21:32:55.329 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:32:55 smithi097 ceph-mon[33317]: 4.1e scrub starts 2024-03-11T21:32:55.329 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:32:55 smithi097 ceph-mon[33317]: 4.1e scrub ok 2024-03-11T21:32:55.329 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:32:55 smithi097 ceph-mon[33317]: 4.e scrub starts 2024-03-11T21:32:55.329 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:32:55 smithi097 ceph-mon[33317]: 4.e scrub ok 2024-03-11T21:32:55.329 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:32:55 smithi097 ceph-mon[33317]: from='client.? v1:172.21.15.92:0/341582292' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "unique_pool_1", "var": "pg_num", "val": "26"}]: dispatch 2024-03-11T21:32:55.349 INFO:teuthology.orchestra.run.smithi097.stderr:Inferring config /var/lib/ceph/00db0016-dfed-11ee-95c8-87774f69a715/mon.b/config 2024-03-11T21:32:55.354 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:32:55.355 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:32:55.355 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:32:55.434 INFO:tasks.radosbench.radosbench.0.smithi092.stdout: 54 16 87386 87370 101.106 79.75 0.00842812 0.00988569 2024-03-11T21:32:55.456 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:32:55.494 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:32:55.495 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:32:55.496 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:32:55.596 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:32:55.625 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:32:55.626 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:32:55.626 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:32:55.727 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:32:55.758 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:32:55.759 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:32:55.759 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:32:55.860 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:32:55.899 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:32:55.900 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:32:55.900 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:32:56.001 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:32:56.043 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:32:56.044 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:32:56.044 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:32:56.111 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:32:56 smithi092 ceph-mon[29572]: 4.b scrub starts 2024-03-11T21:32:56.111 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:32:56 smithi092 ceph-mon[29572]: 4.b scrub ok 2024-03-11T21:32:56.111 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:32:56 smithi092 ceph-mon[29572]: 4.c scrub starts 2024-03-11T21:32:56.111 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:32:56 smithi092 ceph-mon[29572]: 4.c scrub ok 2024-03-11T21:32:56.111 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:32:56 smithi092 ceph-mon[29572]: from='client.? v1:172.21.15.92:0/341582292' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_1", "var": "pg_num", "val": "26"}]': finished 2024-03-11T21:32:56.111 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:32:56 smithi092 ceph-mon[29572]: osdmap e180: 8 total, 8 up, 8 in 2024-03-11T21:32:56.111 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:32:56 smithi092 ceph-mon[29572]: from='mgr.14607 v1:172.21.15.92:0/1702717597' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "unique_pool_1", "var": "pg_num_actual", "val": "26"}]: dispatch 2024-03-11T21:32:56.111 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:32:56 smithi092 ceph-mon[29572]: from='mgr.14607 ' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "unique_pool_1", "var": "pg_num_actual", "val": "26"}]: dispatch 2024-03-11T21:32:56.111 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:32:56 smithi092 ceph-mon[29572]: 5.8 scrub starts 2024-03-11T21:32:56.111 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:32:56 smithi092 ceph-mon[29572]: 5.8 scrub ok 2024-03-11T21:32:56.112 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:32:56 smithi092 ceph-mon[37563]: 4.b scrub starts 2024-03-11T21:32:56.112 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:32:56 smithi092 ceph-mon[37563]: 4.b scrub ok 2024-03-11T21:32:56.112 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:32:56 smithi092 ceph-mon[37563]: 4.c scrub starts 2024-03-11T21:32:56.112 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:32:56 smithi092 ceph-mon[37563]: 4.c scrub ok 2024-03-11T21:32:56.112 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:32:56 smithi092 ceph-mon[37563]: from='client.? v1:172.21.15.92:0/341582292' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_1", "var": "pg_num", "val": "26"}]': finished 2024-03-11T21:32:56.112 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:32:56 smithi092 ceph-mon[37563]: osdmap e180: 8 total, 8 up, 8 in 2024-03-11T21:32:56.112 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:32:56 smithi092 ceph-mon[37563]: from='mgr.14607 v1:172.21.15.92:0/1702717597' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "unique_pool_1", "var": "pg_num_actual", "val": "26"}]: dispatch 2024-03-11T21:32:56.112 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:32:56 smithi092 ceph-mon[37563]: from='mgr.14607 ' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "unique_pool_1", "var": "pg_num_actual", "val": "26"}]: dispatch 2024-03-11T21:32:56.112 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:32:56 smithi092 ceph-mon[37563]: 5.8 scrub starts 2024-03-11T21:32:56.113 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:32:56 smithi092 ceph-mon[37563]: 5.8 scrub ok 2024-03-11T21:32:56.145 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:32:56.187 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:32:56.187 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:32:56.187 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:32:56.288 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:32:56.313 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:32:56.314 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:32:56.314 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:32:56.326 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:32:56 smithi097 ceph-mon[33317]: 4.b scrub starts 2024-03-11T21:32:56.326 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:32:56 smithi097 ceph-mon[33317]: 4.b scrub ok 2024-03-11T21:32:56.326 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:32:56 smithi097 ceph-mon[33317]: 4.c scrub starts 2024-03-11T21:32:56.326 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:32:56 smithi097 ceph-mon[33317]: 4.c scrub ok 2024-03-11T21:32:56.326 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:32:56 smithi097 ceph-mon[33317]: from='client.? v1:172.21.15.92:0/341582292' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_1", "var": "pg_num", "val": "26"}]': finished 2024-03-11T21:32:56.327 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:32:56 smithi097 ceph-mon[33317]: osdmap e180: 8 total, 8 up, 8 in 2024-03-11T21:32:56.327 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:32:56 smithi097 ceph-mon[33317]: from='mgr.14607 v1:172.21.15.92:0/1702717597' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "unique_pool_1", "var": "pg_num_actual", "val": "26"}]: dispatch 2024-03-11T21:32:56.327 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:32:56 smithi097 ceph-mon[33317]: from='mgr.14607 ' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "unique_pool_1", "var": "pg_num_actual", "val": "26"}]: dispatch 2024-03-11T21:32:56.327 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:32:56 smithi097 ceph-mon[33317]: 5.8 scrub starts 2024-03-11T21:32:56.327 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:32:56 smithi097 ceph-mon[33317]: 5.8 scrub ok 2024-03-11T21:32:56.351 DEBUG:teuthology.orchestra.run.smithi092:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid 00db0016-dfed-11ee-95c8-87774f69a715 -- ceph osd unset nodeep-scrub 2024-03-11T21:32:56.416 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:32:56.434 INFO:tasks.radosbench.radosbench.0.smithi092.stdout: 55 16 88153 88137 100.14 47.9375 0.0960691 0.0099179 2024-03-11T21:32:56.440 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:32:56.441 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:32:56.441 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:32:56.542 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:32:56.543 INFO:teuthology.orchestra.run.smithi092.stdout:osd.0: {} 2024-03-11T21:32:56.543 INFO:teuthology.orchestra.run.smithi092.stdout:osd.1: {} 2024-03-11T21:32:56.543 INFO:teuthology.orchestra.run.smithi092.stdout:osd.2: {} 2024-03-11T21:32:56.543 INFO:teuthology.orchestra.run.smithi092.stdout:osd.3: {} 2024-03-11T21:32:56.543 INFO:teuthology.orchestra.run.smithi092.stdout:osd.4: {} 2024-03-11T21:32:56.543 INFO:teuthology.orchestra.run.smithi092.stdout:osd.5: {} 2024-03-11T21:32:56.544 INFO:teuthology.orchestra.run.smithi092.stdout:osd.6: {} 2024-03-11T21:32:56.544 INFO:teuthology.orchestra.run.smithi092.stdout:osd.7: {} 2024-03-11T21:32:56.544 INFO:teuthology.orchestra.run.smithi092.stderr:osd.0: osd_enable_op_tracker = '' 2024-03-11T21:32:56.544 INFO:teuthology.orchestra.run.smithi092.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-11T21:32:56.544 INFO:teuthology.orchestra.run.smithi092.stderr:osd.2: osd_enable_op_tracker = '' 2024-03-11T21:32:56.544 INFO:teuthology.orchestra.run.smithi092.stderr:osd.3: osd_enable_op_tracker = '' 2024-03-11T21:32:56.544 INFO:teuthology.orchestra.run.smithi092.stderr:osd.4: osd_enable_op_tracker = '' 2024-03-11T21:32:56.544 INFO:teuthology.orchestra.run.smithi092.stderr:osd.5: osd_enable_op_tracker = '' 2024-03-11T21:32:56.544 INFO:teuthology.orchestra.run.smithi092.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-11T21:32:56.544 INFO:teuthology.orchestra.run.smithi092.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-11T21:32:56.568 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:32:56.568 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:32:56.569 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:32:56.669 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:32:56.692 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:32:56.693 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:32:56.693 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:32:56.794 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:32:56.818 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:32:56.819 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:32:56.819 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:32:56.920 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:32:56.948 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:32:56.949 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:32:56.949 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:32:57.050 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:32:57.099 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:32:57.100 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:32:57.100 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:32:57.201 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:32:57.232 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:32:57.232 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:32:57.233 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:32:57.300 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:32:57 smithi097 ceph-mon[33317]: pgmap v293: 148 pgs: 148 active+clean; 5.1 GiB data, 17 GiB used, 699 GiB / 715 GiB avail; 853 B/s rd, 62 MiB/s wr, 1k op/s 2024-03-11T21:32:57.300 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:32:57 smithi097 ceph-mon[33317]: from='mgr.14607 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_1", "var": "pg_num_actual", "val": "26"}]': finished 2024-03-11T21:32:57.301 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:32:57 smithi097 ceph-mon[33317]: osdmap e181: 8 total, 8 up, 8 in 2024-03-11T21:32:57.301 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:32:57 smithi097 ceph-mon[33317]: from='client.? v1:172.21.15.92:0/2863252577' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-11T21:32:57.333 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:32:57.360 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:32:57 smithi092 ceph-mon[29572]: pgmap v293: 148 pgs: 148 active+clean; 5.1 GiB data, 17 GiB used, 699 GiB / 715 GiB avail; 853 B/s rd, 62 MiB/s wr, 1k op/s 2024-03-11T21:32:57.360 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:32:57 smithi092 ceph-mon[29572]: from='mgr.14607 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_1", "var": "pg_num_actual", "val": "26"}]': finished 2024-03-11T21:32:57.360 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:32:57 smithi092 ceph-mon[29572]: osdmap e181: 8 total, 8 up, 8 in 2024-03-11T21:32:57.360 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:32:57 smithi092 ceph-mon[29572]: from='client.? v1:172.21.15.92:0/2863252577' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-11T21:32:57.361 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:32:57 smithi092 ceph-mon[37563]: pgmap v293: 148 pgs: 148 active+clean; 5.1 GiB data, 17 GiB used, 699 GiB / 715 GiB avail; 853 B/s rd, 62 MiB/s wr, 1k op/s 2024-03-11T21:32:57.361 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:32:57 smithi092 ceph-mon[37563]: from='mgr.14607 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_1", "var": "pg_num_actual", "val": "26"}]': finished 2024-03-11T21:32:57.361 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:32:57 smithi092 ceph-mon[37563]: osdmap e181: 8 total, 8 up, 8 in 2024-03-11T21:32:57.361 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:32:57 smithi092 ceph-mon[37563]: from='client.? v1:172.21.15.92:0/2863252577' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-11T21:32:57.374 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:32:57.374 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:32:57.375 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:32:57.434 INFO:tasks.radosbench.radosbench.0.smithi092.stdout: 56 15 88719 88704 98.9842 35.4375 0.027413 0.0100972 2024-03-11T21:32:57.476 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:32:57.499 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:32:57.500 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:32:57.501 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:32:57.602 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:32:57.640 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:32:57.641 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:32:57.641 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:32:57.742 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:32:57.765 INFO:teuthology.orchestra.run.smithi097.stdout:ERROR: (22) Invalid argument 2024-03-11T21:32:57.765 INFO:teuthology.orchestra.run.smithi097.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-11T21:32:57.789 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:32:57.789 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:32:57.790 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:32:57.890 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:32:57.922 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:32:57.922 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:32:57.923 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:32:58.024 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:32:58.067 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:32:58.068 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:32:58.069 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:32:58.169 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:32:58.197 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:32:58.198 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:32:58.198 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:32:58.299 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:32:58.305 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:32:58 smithi097 ceph-mon[33317]: 2.16 scrub starts 2024-03-11T21:32:58.305 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:32:58 smithi097 ceph-mon[33317]: 2.16 scrub ok 2024-03-11T21:32:58.305 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:32:58 smithi097 ceph-mon[33317]: 6.10 scrub starts 2024-03-11T21:32:58.305 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:32:58 smithi097 ceph-mon[33317]: 6.10 scrub ok 2024-03-11T21:32:58.305 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:32:58 smithi097 ceph-mon[33317]: osdmap e182: 8 total, 8 up, 8 in 2024-03-11T21:32:58.305 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:32:58 smithi097 ceph-mon[33317]: from='mgr.14607 v1:172.21.15.92:0/1702717597' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "unique_pool_1", "var": "pgp_num_actual", "val": "26"}]: dispatch 2024-03-11T21:32:58.305 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:32:58 smithi097 ceph-mon[33317]: from='mgr.14607 ' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "unique_pool_1", "var": "pgp_num_actual", "val": "26"}]: dispatch 2024-03-11T21:32:58.305 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:32:58 smithi097 ceph-mon[33317]: 8.7 scrub starts 2024-03-11T21:32:58.331 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:32:58.332 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:32:58.332 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:32:58.360 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:32:58 smithi092 ceph-mon[29572]: 2.16 scrub starts 2024-03-11T21:32:58.360 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:32:58 smithi092 ceph-mon[29572]: 2.16 scrub ok 2024-03-11T21:32:58.360 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:32:58 smithi092 ceph-mon[29572]: 6.10 scrub starts 2024-03-11T21:32:58.360 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:32:58 smithi092 ceph-mon[29572]: 6.10 scrub ok 2024-03-11T21:32:58.360 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:32:58 smithi092 ceph-mon[29572]: osdmap e182: 8 total, 8 up, 8 in 2024-03-11T21:32:58.361 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:32:58 smithi092 ceph-mon[29572]: from='mgr.14607 v1:172.21.15.92:0/1702717597' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "unique_pool_1", "var": "pgp_num_actual", "val": "26"}]: dispatch 2024-03-11T21:32:58.361 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:32:58 smithi092 ceph-mon[29572]: from='mgr.14607 ' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "unique_pool_1", "var": "pgp_num_actual", "val": "26"}]: dispatch 2024-03-11T21:32:58.361 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:32:58 smithi092 ceph-mon[29572]: 8.7 scrub starts 2024-03-11T21:32:58.361 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:32:58 smithi092 ceph-mon[37563]: 2.16 scrub starts 2024-03-11T21:32:58.361 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:32:58 smithi092 ceph-mon[37563]: 2.16 scrub ok 2024-03-11T21:32:58.361 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:32:58 smithi092 ceph-mon[37563]: 6.10 scrub starts 2024-03-11T21:32:58.361 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:32:58 smithi092 ceph-mon[37563]: 6.10 scrub ok 2024-03-11T21:32:58.361 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:32:58 smithi092 ceph-mon[37563]: osdmap e182: 8 total, 8 up, 8 in 2024-03-11T21:32:58.361 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:32:58 smithi092 ceph-mon[37563]: from='mgr.14607 v1:172.21.15.92:0/1702717597' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "unique_pool_1", "var": "pgp_num_actual", "val": "26"}]: dispatch 2024-03-11T21:32:58.361 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:32:58 smithi092 ceph-mon[37563]: from='mgr.14607 ' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "unique_pool_1", "var": "pgp_num_actual", "val": "26"}]: dispatch 2024-03-11T21:32:58.362 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:32:58 smithi092 ceph-mon[37563]: 8.7 scrub starts 2024-03-11T21:32:58.362 INFO:journalctl@ceph.mgr.y.smithi092.stdout:Mar 11 21:32:57 smithi092 ceph-00db0016-dfed-11ee-95c8-87774f69a715-mgr-y[29801]: ::ffff:172.21.15.97 - - [11/Mar/2024:21:32:57] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-11T21:32:58.433 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:32:58.434 INFO:tasks.radosbench.radosbench.0.smithi092.stdout: 57 16 89298 89282 97.8814 36.125 0.00338271 0.0101472 2024-03-11T21:32:58.459 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:32:58.460 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:32:58.460 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:32:58.497 DEBUG:teuthology.orchestra.run.smithi097:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid 00db0016-dfed-11ee-95c8-87774f69a715 -- ceph daemon osd.6 dump_ops_in_flight 2024-03-11T21:32:58.561 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:32:58.588 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:32:58.588 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:32:58.589 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:32:58.637 INFO:teuthology.orchestra.run.smithi092.stderr:Inferring config /var/lib/ceph/00db0016-dfed-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T21:32:58.690 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:32:58.714 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:32:58.715 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:32:58.715 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:32:58.743 INFO:teuthology.orchestra.run.smithi097.stderr:Inferring config /var/lib/ceph/00db0016-dfed-11ee-95c8-87774f69a715/mon.b/config 2024-03-11T21:32:58.817 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:32:58.841 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:32:58.841 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:32:58.841 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:32:58.942 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:32:58.966 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:32:58.967 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:32:58.967 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:32:59.068 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:32:59.094 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:32:59.094 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:32:59.095 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:32:59.120 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:32:59 smithi097 ceph-mon[33317]: pgmap v296: 158 pgs: 3 peering, 10 unknown, 145 active+clean; 5.2 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 19 MiB/s wr, 306 op/s 2024-03-11T21:32:59.120 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:32:59 smithi097 ceph-mon[33317]: 8.18 scrub starts 2024-03-11T21:32:59.120 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:32:59 smithi097 ceph-mon[33317]: 8.6 scrub starts 2024-03-11T21:32:59.120 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:32:59 smithi097 ceph-mon[33317]: 8.0 scrub starts 2024-03-11T21:32:59.120 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:32:59 smithi097 ceph-mon[33317]: 8.2 scrub starts 2024-03-11T21:32:59.120 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:32:59 smithi097 ceph-mon[33317]: 8.9 scrub starts 2024-03-11T21:32:59.120 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:32:59 smithi097 ceph-mon[33317]: from='mgr.14607 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_1", "var": "pgp_num_actual", "val": "26"}]': finished 2024-03-11T21:32:59.120 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:32:59 smithi097 ceph-mon[33317]: osdmap e183: 8 total, 8 up, 8 in 2024-03-11T21:32:59.120 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:32:59 smithi097 ceph-mon[33317]: 8.7 scrub ok 2024-03-11T21:32:59.121 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:32:59 smithi097 ceph-mon[33317]: 8.6 scrub ok 2024-03-11T21:32:59.121 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:32:59 smithi097 ceph-mon[33317]: osdmap e184: 8 total, 8 up, 8 in 2024-03-11T21:32:59.121 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:32:59 smithi097 ceph-mon[33317]: 8.1 scrub starts 2024-03-11T21:32:59.195 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:32:59.222 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:32:59.223 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:32:59.223 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:32:59.324 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:32:59.361 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:32:59 smithi092 ceph-mon[29572]: pgmap v296: 158 pgs: 3 peering, 10 unknown, 145 active+clean; 5.2 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 19 MiB/s wr, 306 op/s 2024-03-11T21:32:59.362 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:32:59 smithi092 ceph-mon[29572]: 8.18 scrub starts 2024-03-11T21:32:59.362 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:32:59 smithi092 ceph-mon[29572]: 8.6 scrub starts 2024-03-11T21:32:59.362 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:32:59 smithi092 ceph-mon[29572]: 8.0 scrub starts 2024-03-11T21:32:59.362 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:32:59 smithi092 ceph-mon[29572]: 8.2 scrub starts 2024-03-11T21:32:59.362 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:32:59 smithi092 ceph-mon[29572]: 8.9 scrub starts 2024-03-11T21:32:59.362 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:32:59 smithi092 ceph-mon[29572]: from='mgr.14607 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_1", "var": "pgp_num_actual", "val": "26"}]': finished 2024-03-11T21:32:59.362 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:32:59 smithi092 ceph-mon[29572]: osdmap e183: 8 total, 8 up, 8 in 2024-03-11T21:32:59.362 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:32:59 smithi092 ceph-mon[29572]: 8.7 scrub ok 2024-03-11T21:32:59.362 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:32:59 smithi092 ceph-mon[29572]: 8.6 scrub ok 2024-03-11T21:32:59.362 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:32:59 smithi092 ceph-mon[29572]: osdmap e184: 8 total, 8 up, 8 in 2024-03-11T21:32:59.362 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:32:59 smithi092 ceph-mon[29572]: 8.1 scrub starts 2024-03-11T21:32:59.363 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:32:59 smithi092 ceph-mon[37563]: pgmap v296: 158 pgs: 3 peering, 10 unknown, 145 active+clean; 5.2 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 19 MiB/s wr, 306 op/s 2024-03-11T21:32:59.363 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:32:59 smithi092 ceph-mon[37563]: 8.18 scrub starts 2024-03-11T21:32:59.363 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:32:59 smithi092 ceph-mon[37563]: 8.6 scrub starts 2024-03-11T21:32:59.363 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:32:59 smithi092 ceph-mon[37563]: 8.0 scrub starts 2024-03-11T21:32:59.363 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:32:59 smithi092 ceph-mon[37563]: 8.2 scrub starts 2024-03-11T21:32:59.363 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:32:59 smithi092 ceph-mon[37563]: 8.9 scrub starts 2024-03-11T21:32:59.363 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:32:59 smithi092 ceph-mon[37563]: from='mgr.14607 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_1", "var": "pgp_num_actual", "val": "26"}]': finished 2024-03-11T21:32:59.363 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:32:59 smithi092 ceph-mon[37563]: osdmap e183: 8 total, 8 up, 8 in 2024-03-11T21:32:59.363 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:32:59 smithi092 ceph-mon[37563]: 8.7 scrub ok 2024-03-11T21:32:59.363 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:32:59 smithi092 ceph-mon[37563]: 8.6 scrub ok 2024-03-11T21:32:59.363 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:32:59 smithi092 ceph-mon[37563]: osdmap e184: 8 total, 8 up, 8 in 2024-03-11T21:32:59.363 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:32:59 smithi092 ceph-mon[37563]: 8.1 scrub starts 2024-03-11T21:32:59.364 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:32:59.365 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:32:59.365 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:32:59.435 INFO:tasks.radosbench.radosbench.0.smithi092.stdout: 58 16 89434 89418 96.3405 8.5 0.00441552 0.0103745 2024-03-11T21:32:59.466 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:32:59.504 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:32:59.505 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:32:59.505 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:32:59.606 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:32:59.643 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:32:59.644 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:32:59.644 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:32:59.745 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:32:59.785 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:32:59.785 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:32:59.785 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:32:59.886 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:32:59.932 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:32:59.933 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:32:59.933 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:33:00.034 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:33:00.067 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:33:00.068 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:33:00.068 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:33:00.169 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:33:00.202 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:33:00.202 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:33:00.202 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:33:00.303 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:33:00.313 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:33:00 smithi092 ceph-mon[37563]: 8.8 scrub starts 2024-03-11T21:33:00.313 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:33:00 smithi092 ceph-mon[37563]: 8.0 scrub ok 2024-03-11T21:33:00.313 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:33:00 smithi092 ceph-mon[37563]: 8.2 scrub ok 2024-03-11T21:33:00.314 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:33:00 smithi092 ceph-mon[37563]: 8.9 scrub ok 2024-03-11T21:33:00.314 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:33:00 smithi092 ceph-mon[37563]: 4.6 scrub starts 2024-03-11T21:33:00.314 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:33:00 smithi092 ceph-mon[37563]: 4.6 scrub ok 2024-03-11T21:33:00.314 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:33:00 smithi092 ceph-mon[37563]: 8.8 scrub ok 2024-03-11T21:33:00.314 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:33:00 smithi092 ceph-mon[37563]: 8.1 scrub ok 2024-03-11T21:33:00.314 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:33:00 smithi092 ceph-mon[37563]: 4.1 scrub starts 2024-03-11T21:33:00.314 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:33:00 smithi092 ceph-mon[37563]: 4.1 scrub ok 2024-03-11T21:33:00.314 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:33:00 smithi092 ceph-mon[37563]: 8.5 scrub starts 2024-03-11T21:33:00.314 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:33:00 smithi092 ceph-mon[37563]: osdmap e185: 8 total, 8 up, 8 in 2024-03-11T21:33:00.314 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:33:00 smithi092 ceph-mon[37563]: 8.19 starting backfill to osd.4 from (0'0,0'0] MAX to 182'5635 2024-03-11T21:33:00.314 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:33:00 smithi092 ceph-mon[37563]: 8.12 starting backfill to osd.2 from (0'0,0'0] MAX to 182'5608 2024-03-11T21:33:00.314 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:33:00 smithi092 ceph-mon[37563]: 8.17 starting backfill to osd.0 from (0'0,0'0] MAX to 182'5492 2024-03-11T21:33:00.314 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:33:00 smithi092 ceph-mon[37563]: 8.10 starting backfill to osd.4 from (0'0,0'0] MAX to 182'5465 2024-03-11T21:33:00.314 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:33:00 smithi092 ceph-mon[37563]: 8.16 starting backfill to osd.2 from (0'0,0'0] MAX to 182'5452 2024-03-11T21:33:00.315 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:33:00 smithi092 ceph-mon[37563]: 8.18 starting backfill to osd.4 from (0'0,0'0] MAX to 182'5657 2024-03-11T21:33:00.315 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:33:00 smithi092 ceph-mon[37563]: 8.11 starting backfill to osd.5 from (0'0,0'0] MAX to 182'5510 2024-03-11T21:33:00.315 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:33:00 smithi092 ceph-mon[37563]: 8.13 starting backfill to osd.2 from (0'0,0'0] MAX to 182'5517 2024-03-11T21:33:00.315 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:33:00 smithi092 ceph-mon[37563]: 8.14 starting backfill to osd.0 from (0'0,0'0] MAX to 182'5473 2024-03-11T21:33:00.315 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:33:00 smithi092 ceph-mon[37563]: 8.15 starting backfill to osd.1 from (0'0,0'0] MAX to 182'5571 2024-03-11T21:33:00.315 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:33:00 smithi092 ceph-mon[37563]: 8.17 starting backfill to osd.2 from (0'0,0'0] MAX to 182'5492 2024-03-11T21:33:00.315 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:33:00 smithi092 ceph-mon[37563]: 8.13 starting backfill to osd.3 from (0'0,0'0] MAX to 182'5517 2024-03-11T21:33:00.315 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:33:00 smithi092 ceph-mon[37563]: 8.10 starting backfill to osd.6 from (0'0,0'0] MAX to 182'5465 2024-03-11T21:33:00.315 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:33:00 smithi092 ceph-mon[37563]: 8.16 starting backfill to osd.3 from (0'0,0'0] MAX to 182'5452 2024-03-11T21:33:00.315 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:33:00 smithi092 ceph-mon[37563]: 8.4 scrub starts 2024-03-11T21:33:00.315 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:33:00 smithi092 ceph-mon[37563]: 8.4 scrub ok 2024-03-11T21:33:00.315 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:33:00 smithi092 ceph-mon[29572]: 8.8 scrub starts 2024-03-11T21:33:00.315 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:33:00 smithi092 ceph-mon[29572]: 8.0 scrub ok 2024-03-11T21:33:00.316 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:33:00 smithi092 ceph-mon[29572]: 8.2 scrub ok 2024-03-11T21:33:00.316 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:33:00 smithi092 ceph-mon[29572]: 8.9 scrub ok 2024-03-11T21:33:00.316 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:33:00 smithi092 ceph-mon[29572]: 4.6 scrub starts 2024-03-11T21:33:00.316 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:33:00 smithi092 ceph-mon[29572]: 4.6 scrub ok 2024-03-11T21:33:00.316 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:33:00 smithi092 ceph-mon[29572]: 8.8 scrub ok 2024-03-11T21:33:00.316 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:33:00 smithi092 ceph-mon[29572]: 8.1 scrub ok 2024-03-11T21:33:00.316 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:33:00 smithi092 ceph-mon[29572]: 4.1 scrub starts 2024-03-11T21:33:00.316 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:33:00 smithi092 ceph-mon[29572]: 4.1 scrub ok 2024-03-11T21:33:00.316 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:33:00 smithi092 ceph-mon[29572]: 8.5 scrub starts 2024-03-11T21:33:00.316 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:33:00 smithi092 ceph-mon[29572]: osdmap e185: 8 total, 8 up, 8 in 2024-03-11T21:33:00.316 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:33:00 smithi092 ceph-mon[29572]: 8.19 starting backfill to osd.4 from (0'0,0'0] MAX to 182'5635 2024-03-11T21:33:00.316 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:33:00 smithi092 ceph-mon[29572]: 8.12 starting backfill to osd.2 from (0'0,0'0] MAX to 182'5608 2024-03-11T21:33:00.316 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:33:00 smithi092 ceph-mon[29572]: 8.17 starting backfill to osd.0 from (0'0,0'0] MAX to 182'5492 2024-03-11T21:33:00.316 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:33:00 smithi092 ceph-mon[29572]: 8.10 starting backfill to osd.4 from (0'0,0'0] MAX to 182'5465 2024-03-11T21:33:00.316 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:33:00 smithi092 ceph-mon[29572]: 8.16 starting backfill to osd.2 from (0'0,0'0] MAX to 182'5452 2024-03-11T21:33:00.317 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:33:00 smithi092 ceph-mon[29572]: 8.18 starting backfill to osd.4 from (0'0,0'0] MAX to 182'5657 2024-03-11T21:33:00.317 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:33:00 smithi092 ceph-mon[29572]: 8.11 starting backfill to osd.5 from (0'0,0'0] MAX to 182'5510 2024-03-11T21:33:00.317 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:33:00 smithi092 ceph-mon[29572]: 8.13 starting backfill to osd.2 from (0'0,0'0] MAX to 182'5517 2024-03-11T21:33:00.317 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:33:00 smithi092 ceph-mon[29572]: 8.14 starting backfill to osd.0 from (0'0,0'0] MAX to 182'5473 2024-03-11T21:33:00.317 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:33:00 smithi092 ceph-mon[29572]: 8.15 starting backfill to osd.1 from (0'0,0'0] MAX to 182'5571 2024-03-11T21:33:00.317 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:33:00 smithi092 ceph-mon[29572]: 8.17 starting backfill to osd.2 from (0'0,0'0] MAX to 182'5492 2024-03-11T21:33:00.317 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:33:00 smithi092 ceph-mon[29572]: 8.13 starting backfill to osd.3 from (0'0,0'0] MAX to 182'5517 2024-03-11T21:33:00.317 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:33:00 smithi092 ceph-mon[29572]: 8.10 starting backfill to osd.6 from (0'0,0'0] MAX to 182'5465 2024-03-11T21:33:00.317 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:33:00 smithi092 ceph-mon[29572]: 8.16 starting backfill to osd.3 from (0'0,0'0] MAX to 182'5452 2024-03-11T21:33:00.317 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:33:00 smithi092 ceph-mon[29572]: 8.4 scrub starts 2024-03-11T21:33:00.317 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:33:00 smithi092 ceph-mon[29572]: 8.4 scrub ok 2024-03-11T21:33:00.326 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:33:00 smithi097 ceph-mon[33317]: 8.8 scrub starts 2024-03-11T21:33:00.326 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:33:00 smithi097 ceph-mon[33317]: 8.0 scrub ok 2024-03-11T21:33:00.326 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:33:00 smithi097 ceph-mon[33317]: 8.2 scrub ok 2024-03-11T21:33:00.327 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:33:00 smithi097 ceph-mon[33317]: 8.9 scrub ok 2024-03-11T21:33:00.327 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:33:00 smithi097 ceph-mon[33317]: 4.6 scrub starts 2024-03-11T21:33:00.327 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:33:00 smithi097 ceph-mon[33317]: 4.6 scrub ok 2024-03-11T21:33:00.327 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:33:00 smithi097 ceph-mon[33317]: 8.8 scrub ok 2024-03-11T21:33:00.327 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:33:00 smithi097 ceph-mon[33317]: 8.1 scrub ok 2024-03-11T21:33:00.327 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:33:00 smithi097 ceph-mon[33317]: 4.1 scrub starts 2024-03-11T21:33:00.327 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:33:00 smithi097 ceph-mon[33317]: 4.1 scrub ok 2024-03-11T21:33:00.327 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:33:00 smithi097 ceph-mon[33317]: 8.5 scrub starts 2024-03-11T21:33:00.327 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:33:00 smithi097 ceph-mon[33317]: osdmap e185: 8 total, 8 up, 8 in 2024-03-11T21:33:00.327 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:33:00 smithi097 ceph-mon[33317]: 8.19 starting backfill to osd.4 from (0'0,0'0] MAX to 182'5635 2024-03-11T21:33:00.327 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:33:00 smithi097 ceph-mon[33317]: 8.12 starting backfill to osd.2 from (0'0,0'0] MAX to 182'5608 2024-03-11T21:33:00.327 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:33:00 smithi097 ceph-mon[33317]: 8.17 starting backfill to osd.0 from (0'0,0'0] MAX to 182'5492 2024-03-11T21:33:00.327 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:33:00 smithi097 ceph-mon[33317]: 8.10 starting backfill to osd.4 from (0'0,0'0] MAX to 182'5465 2024-03-11T21:33:00.328 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:33:00 smithi097 ceph-mon[33317]: 8.16 starting backfill to osd.2 from (0'0,0'0] MAX to 182'5452 2024-03-11T21:33:00.328 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:33:00 smithi097 ceph-mon[33317]: 8.18 starting backfill to osd.4 from (0'0,0'0] MAX to 182'5657 2024-03-11T21:33:00.328 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:33:00 smithi097 ceph-mon[33317]: 8.11 starting backfill to osd.5 from (0'0,0'0] MAX to 182'5510 2024-03-11T21:33:00.328 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:33:00 smithi097 ceph-mon[33317]: 8.13 starting backfill to osd.2 from (0'0,0'0] MAX to 182'5517 2024-03-11T21:33:00.328 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:33:00 smithi097 ceph-mon[33317]: 8.14 starting backfill to osd.0 from (0'0,0'0] MAX to 182'5473 2024-03-11T21:33:00.328 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:33:00 smithi097 ceph-mon[33317]: 8.15 starting backfill to osd.1 from (0'0,0'0] MAX to 182'5571 2024-03-11T21:33:00.328 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:33:00 smithi097 ceph-mon[33317]: 8.17 starting backfill to osd.2 from (0'0,0'0] MAX to 182'5492 2024-03-11T21:33:00.328 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:33:00 smithi097 ceph-mon[33317]: 8.13 starting backfill to osd.3 from (0'0,0'0] MAX to 182'5517 2024-03-11T21:33:00.328 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:33:00 smithi097 ceph-mon[33317]: 8.10 starting backfill to osd.6 from (0'0,0'0] MAX to 182'5465 2024-03-11T21:33:00.328 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:33:00 smithi097 ceph-mon[33317]: 8.16 starting backfill to osd.3 from (0'0,0'0] MAX to 182'5452 2024-03-11T21:33:00.328 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:33:00 smithi097 ceph-mon[33317]: 8.4 scrub starts 2024-03-11T21:33:00.328 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:33:00 smithi097 ceph-mon[33317]: 8.4 scrub ok 2024-03-11T21:33:00.349 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:33:00.350 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:33:00.350 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:33:00.435 INFO:tasks.radosbench.radosbench.0.smithi092.stdout: 59 16 90602 90586 95.9448 73 0.00884348 0.0104175 2024-03-11T21:33:00.450 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:33:00.491 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:33:00.492 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:33:00.492 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:33:00.560 DEBUG:teuthology.orchestra.run.smithi092:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid 00db0016-dfed-11ee-95c8-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-03-11T21:33:00.593 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:33:00.651 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:33:00.652 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:33:00.652 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:33:00.753 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:33:00.803 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:33:00.803 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:33:00.803 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:33:00.867 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5, 6, 7] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7] 2024-03-11T21:33:00.867 DEBUG:teuthology.orchestra.run.smithi092:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid 00db0016-dfed-11ee-95c8-87774f69a715 -- ceph osd reweight 0 1 2024-03-11T21:33:00.904 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:33:00.970 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:33:00.970 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:33:00.970 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:33:01.071 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:33:01.108 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:33:01.108 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:33:01.109 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:33:01.110 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:33:01 smithi092 ceph-mon[29572]: pgmap v299: 158 pgs: 1 active+clean+scrubbing, 3 peering, 3 unknown, 151 active+clean; 5.4 GiB data, 18 GiB used, 698 GiB / 715 GiB avail; 1.2 KiB/s rd, 75 MiB/s wr, 1.20k op/s; 48 KiB/s, 0 objects/s recovering 2024-03-11T21:33:01.110 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:33:01 smithi092 ceph-mon[29572]: 8.12 starting backfill to osd.6 from (0'0,0'0] MAX to 182'5608 2024-03-11T21:33:01.110 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:33:01 smithi092 ceph-mon[29572]: 8.11 starting backfill to osd.6 from (0'0,0'0] MAX to 182'5510 2024-03-11T21:33:01.110 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:33:01 smithi092 ceph-mon[29572]: 8.16 starting backfill to osd.5 from (0'0,0'0] MAX to 182'5452 2024-03-11T21:33:01.110 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:33:01 smithi092 ceph-mon[29572]: 8.15 starting backfill to osd.7 from (0'0,0'0] MAX to 182'5571 2024-03-11T21:33:01.110 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:33:01 smithi092 ceph-mon[29572]: 8.11 starting backfill to osd.7 from (0'0,0'0] MAX to 182'5510 2024-03-11T21:33:01.110 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:33:01 smithi092 ceph-mon[29572]: 8.5 scrub ok 2024-03-11T21:33:01.110 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:33:01 smithi092 ceph-mon[37563]: pgmap v299: 158 pgs: 1 active+clean+scrubbing, 3 peering, 3 unknown, 151 active+clean; 5.4 GiB data, 18 GiB used, 698 GiB / 715 GiB avail; 1.2 KiB/s rd, 75 MiB/s wr, 1.20k op/s; 48 KiB/s, 0 objects/s recovering 2024-03-11T21:33:01.111 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:33:01 smithi092 ceph-mon[37563]: 8.12 starting backfill to osd.6 from (0'0,0'0] MAX to 182'5608 2024-03-11T21:33:01.111 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:33:01 smithi092 ceph-mon[37563]: 8.11 starting backfill to osd.6 from (0'0,0'0] MAX to 182'5510 2024-03-11T21:33:01.111 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:33:01 smithi092 ceph-mon[37563]: 8.16 starting backfill to osd.5 from (0'0,0'0] MAX to 182'5452 2024-03-11T21:33:01.111 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:33:01 smithi092 ceph-mon[37563]: 8.15 starting backfill to osd.7 from (0'0,0'0] MAX to 182'5571 2024-03-11T21:33:01.111 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:33:01 smithi092 ceph-mon[37563]: 8.11 starting backfill to osd.7 from (0'0,0'0] MAX to 182'5510 2024-03-11T21:33:01.111 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:33:01 smithi092 ceph-mon[37563]: 8.5 scrub ok 2024-03-11T21:33:01.147 INFO:teuthology.orchestra.run.smithi097.stdout:ERROR: (22) Invalid argument 2024-03-11T21:33:01.147 INFO:teuthology.orchestra.run.smithi097.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-11T21:33:01.210 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:33:01.243 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:33:01.243 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:33:01.244 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:33:01.326 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:33:01 smithi097 ceph-mon[33317]: pgmap v299: 158 pgs: 1 active+clean+scrubbing, 3 peering, 3 unknown, 151 active+clean; 5.4 GiB data, 18 GiB used, 698 GiB / 715 GiB avail; 1.2 KiB/s rd, 75 MiB/s wr, 1.20k op/s; 48 KiB/s, 0 objects/s recovering 2024-03-11T21:33:01.326 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:33:01 smithi097 ceph-mon[33317]: 8.12 starting backfill to osd.6 from (0'0,0'0] MAX to 182'5608 2024-03-11T21:33:01.326 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:33:01 smithi097 ceph-mon[33317]: 8.11 starting backfill to osd.6 from (0'0,0'0] MAX to 182'5510 2024-03-11T21:33:01.326 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:33:01 smithi097 ceph-mon[33317]: 8.16 starting backfill to osd.5 from (0'0,0'0] MAX to 182'5452 2024-03-11T21:33:01.326 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:33:01 smithi097 ceph-mon[33317]: 8.15 starting backfill to osd.7 from (0'0,0'0] MAX to 182'5571 2024-03-11T21:33:01.326 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:33:01 smithi097 ceph-mon[33317]: 8.11 starting backfill to osd.7 from (0'0,0'0] MAX to 182'5510 2024-03-11T21:33:01.327 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:33:01 smithi097 ceph-mon[33317]: 8.5 scrub ok 2024-03-11T21:33:01.345 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:33:01.375 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:33:01.376 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:33:01.376 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:33:01.435 INFO:tasks.radosbench.radosbench.0.smithi092.stdout:2024-03-11T21:33:01.453957+0000 min lat: 0.0013089 max lat: 1.3116 avg lat: 0.010469 2024-03-11T21:33:01.435 INFO:tasks.radosbench.radosbench.0.smithi092.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-03-11T21:33:01.435 INFO:tasks.radosbench.radosbench.0.smithi092.stdout: 60 16 91683 91667 95.4717 67.5625 0.00951925 0.010469 2024-03-11T21:33:01.477 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:33:01.530 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:33:01.531 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:33:01.531 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:33:01.632 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:33:01.661 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:33:01.662 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:33:01.662 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:33:01.714 INFO:teuthology.orchestra.run.smithi092.stderr:Inferring config /var/lib/ceph/00db0016-dfed-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T21:33:01.726 INFO:teuthology.orchestra.run.smithi092.stderr:Inferring config /var/lib/ceph/00db0016-dfed-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T21:33:01.763 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:33:01.794 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:33:01.795 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:33:01.796 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:33:01.896 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:33:01.953 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:33:01.953 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:33:01.954 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:33:02.006 DEBUG:teuthology.orchestra.run.smithi097:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid 00db0016-dfed-11ee-95c8-87774f69a715 -- ceph daemon osd.6 dump_blocked_ops 2024-03-11T21:33:02.054 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:33:02.089 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:33:02.089 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:33:02.090 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:33:02.191 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:33:02.224 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:33:02.225 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:33:02.225 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:33:02.326 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:33:02.361 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:33:02.361 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:33:02.362 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:33:02.373 INFO:teuthology.orchestra.run.smithi097.stderr:Inferring config /var/lib/ceph/00db0016-dfed-11ee-95c8-87774f69a715/mon.b/config 2024-03-11T21:33:02.435 INFO:tasks.radosbench.radosbench.0.smithi092.stdout: 61 16 92701 92685 94.9496 63.625 0.0247637 0.0105269 2024-03-11T21:33:02.463 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:33:02.503 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:33:02.503 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:33:02.504 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:33:02.604 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:33:02.642 INFO:journalctl@ceph.iscsi.iscsi.a.smithi097.stdout:Mar 11 21:33:02 smithi097 ceph-00db0016-dfed-11ee-95c8-87774f69a715-iscsi-iscsi-a[60943]: debug there is no tcmu-runner data available 2024-03-11T21:33:02.645 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:33:02.645 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:33:02.646 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:33:02.747 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:33:02.787 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:33:02.788 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:33:02.788 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:33:02.889 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:33:02.929 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:33:02.929 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:33:02.930 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:33:03.030 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:33:03.065 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:33:03.066 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:33:03.066 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:33:03.071 INFO:teuthology.orchestra.run.smithi092.stderr:nodeep-scrub is unset 2024-03-11T21:33:03.086 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:33:03 smithi092 ceph-mon[29572]: pgmap v301: 158 pgs: 1 active+clean+scrubbing, 3 peering, 3 unknown, 151 active+clean; 5.4 GiB data, 18 GiB used, 698 GiB / 715 GiB avail; 1.2 KiB/s rd, 55 MiB/s wr, 883 op/s; 47 KiB/s, 0 objects/s recovering 2024-03-11T21:33:03.086 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:33:03 smithi092 ceph-mon[29572]: from='client.? v1:172.21.15.92:0/363793756' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-11T21:33:03.087 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:33:03 smithi092 ceph-mon[29572]: from='client.35084 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-11T21:33:03.087 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:33:03 smithi092 ceph-mon[37563]: pgmap v301: 158 pgs: 1 active+clean+scrubbing, 3 peering, 3 unknown, 151 active+clean; 5.4 GiB data, 18 GiB used, 698 GiB / 715 GiB avail; 1.2 KiB/s rd, 55 MiB/s wr, 883 op/s; 47 KiB/s, 0 objects/s recovering 2024-03-11T21:33:03.087 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:33:03 smithi092 ceph-mon[37563]: from='client.? v1:172.21.15.92:0/363793756' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-11T21:33:03.087 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:33:03 smithi092 ceph-mon[37563]: from='client.35084 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-11T21:33:03.167 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:33:03.201 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:33:03.201 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:33:03.202 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:33:03.258 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:33:03 smithi097 ceph-mon[33317]: pgmap v301: 158 pgs: 1 active+clean+scrubbing, 3 peering, 3 unknown, 151 active+clean; 5.4 GiB data, 18 GiB used, 698 GiB / 715 GiB avail; 1.2 KiB/s rd, 55 MiB/s wr, 883 op/s; 47 KiB/s, 0 objects/s recovering 2024-03-11T21:33:03.259 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:33:03 smithi097 ceph-mon[33317]: from='client.? v1:172.21.15.92:0/363793756' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-11T21:33:03.259 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:33:03 smithi097 ceph-mon[33317]: from='client.35084 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-11T21:33:03.303 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:33:03.349 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:33:03.349 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:33:03.350 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:33:03.435 INFO:tasks.radosbench.radosbench.0.smithi092.stdout: 62 16 93742 93726 94.4675 65.0625 0.0198102 0.0105806 2024-03-11T21:33:03.451 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:33:03.493 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:33:03.494 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:33:03.495 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:33:03.595 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:33:03.637 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:33:03.638 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:33:03.638 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:33:03.739 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:33:03.772 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:33:03.773 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:33:03.773 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:33:03.874 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:33:03.917 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:33:03.917 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:33:03.918 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:33:04.019 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:33:04.056 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:33:04.056 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:33:04.057 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:33:04.158 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:33:04.203 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:33:04.203 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:33:04.204 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:33:04.204 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:33:04 smithi092 ceph-mon[29572]: from='client.24469 v1:172.21.15.97:0/404208310' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:33:04.204 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:33:04 smithi092 ceph-mon[29572]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-03-11T21:33:04.204 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:33:04 smithi092 ceph-mon[29572]: Cluster is now healthy 2024-03-11T21:33:04.204 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:33:04 smithi092 ceph-mon[29572]: from='client.35084 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-03-11T21:33:04.204 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:33:04 smithi092 ceph-mon[29572]: osdmap e186: 8 total, 8 up, 8 in 2024-03-11T21:33:04.204 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:33:04 smithi092 ceph-mon[29572]: pgmap v303: 158 pgs: 2 active+remapped+backfill_wait, 2 active+remapped+backfill_toofull, 6 active+remapped+backfilling, 148 active+clean; 5.7 GiB data, 19 GiB used, 697 GiB / 715 GiB avail; 1013 B/s rd, 0 op/s; 51650/278763 objects misplaced (18.528%); 29 MiB/s, 471 objects/s recovering 2024-03-11T21:33:04.205 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:33:04 smithi092 ceph-mon[29572]: osdmap e187: 8 total, 8 up, 8 in 2024-03-11T21:33:04.205 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:33:04 smithi092 ceph-mon[37563]: from='client.24469 v1:172.21.15.97:0/404208310' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:33:04.205 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:33:04 smithi092 ceph-mon[37563]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-03-11T21:33:04.205 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:33:04 smithi092 ceph-mon[37563]: Cluster is now healthy 2024-03-11T21:33:04.205 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:33:04 smithi092 ceph-mon[37563]: from='client.35084 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-03-11T21:33:04.205 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:33:04 smithi092 ceph-mon[37563]: osdmap e186: 8 total, 8 up, 8 in 2024-03-11T21:33:04.205 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:33:04 smithi092 ceph-mon[37563]: pgmap v303: 158 pgs: 2 active+remapped+backfill_wait, 2 active+remapped+backfill_toofull, 6 active+remapped+backfilling, 148 active+clean; 5.7 GiB data, 19 GiB used, 697 GiB / 715 GiB avail; 1013 B/s rd, 0 op/s; 51650/278763 objects misplaced (18.528%); 29 MiB/s, 471 objects/s recovering 2024-03-11T21:33:04.205 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:33:04 smithi092 ceph-mon[37563]: osdmap e187: 8 total, 8 up, 8 in 2024-03-11T21:33:04.212 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:33:04 smithi097 ceph-mon[33317]: from='client.24469 v1:172.21.15.97:0/404208310' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:33:04.212 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:33:04 smithi097 ceph-mon[33317]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-03-11T21:33:04.212 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:33:04 smithi097 ceph-mon[33317]: Cluster is now healthy 2024-03-11T21:33:04.212 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:33:04 smithi097 ceph-mon[33317]: from='client.35084 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-03-11T21:33:04.212 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:33:04 smithi097 ceph-mon[33317]: osdmap e186: 8 total, 8 up, 8 in 2024-03-11T21:33:04.212 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:33:04 smithi097 ceph-mon[33317]: pgmap v303: 158 pgs: 2 active+remapped+backfill_wait, 2 active+remapped+backfill_toofull, 6 active+remapped+backfilling, 148 active+clean; 5.7 GiB data, 19 GiB used, 697 GiB / 715 GiB avail; 1013 B/s rd, 0 op/s; 51650/278763 objects misplaced (18.528%); 29 MiB/s, 471 objects/s recovering 2024-03-11T21:33:04.212 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:33:04 smithi097 ceph-mon[33317]: osdmap e187: 8 total, 8 up, 8 in 2024-03-11T21:33:04.305 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:33:04.348 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:33:04.348 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:33:04.349 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:33:04.437 INFO:tasks.radosbench.radosbench.0.smithi092.stdout: 63 16 94846 94830 94.0631 69 0.0104123 0.0106265 2024-03-11T21:33:04.450 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:33:04.482 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:33:04.483 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:33:04.483 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:33:04.546 INFO:teuthology.orchestra.run.smithi097.stdout:ERROR: (22) Invalid argument 2024-03-11T21:33:04.546 INFO:teuthology.orchestra.run.smithi097.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-11T21:33:04.584 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:33:04.615 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:33:04.616 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:33:04.616 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:33:04.717 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:33:04.757 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:33:04.757 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:33:04.758 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:33:04.859 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:33:04.899 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:33:04.900 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:33:04.900 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:33:05.001 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:33:05.050 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:33:05.051 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:33:05.051 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:33:05.152 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:33:05.199 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:33:05.200 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:33:05.201 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:33:05.201 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:33:05 smithi097 ceph-mon[33317]: Health check failed: Low space hindering backfill (add storage if this doesn't resolve itself): 2 pgs backfill_toofull (PG_BACKFILL_FULL) 2024-03-11T21:33:05.247 DEBUG:teuthology.orchestra.run.smithi097:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid 00db0016-dfed-11ee-95c8-87774f69a715 -- ceph daemon osd.6 dump_historic_ops 2024-03-11T21:33:05.301 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:33:05.335 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:33:05.336 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:33:05.336 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:33:05.336 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:33:05 smithi092 ceph-mon[29572]: Health check failed: Low space hindering backfill (add storage if this doesn't resolve itself): 2 pgs backfill_toofull (PG_BACKFILL_FULL) 2024-03-11T21:33:05.337 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:33:05 smithi092 ceph-mon[37563]: Health check failed: Low space hindering backfill (add storage if this doesn't resolve itself): 2 pgs backfill_toofull (PG_BACKFILL_FULL) 2024-03-11T21:33:05.436 INFO:tasks.radosbench.radosbench.0.smithi092.stdout: 64 16 96011 95995 93.731 72.8125 0.00716777 0.0106642 2024-03-11T21:33:05.437 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:33:05.478 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:33:05.478 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:33:05.479 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:33:05.580 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:33:05.620 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:33:05.620 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:33:05.621 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:33:05.628 INFO:teuthology.orchestra.run.smithi097.stderr:Inferring config /var/lib/ceph/00db0016-dfed-11ee-95c8-87774f69a715/mon.b/config 2024-03-11T21:33:05.721 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:33:05.753 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:33:05.754 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:33:05.754 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:33:05.855 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:33:05.887 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:33:05.887 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:33:05.888 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:33:05.945 DEBUG:teuthology.orchestra.run.smithi092:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid 00db0016-dfed-11ee-95c8-87774f69a715 -- ceph osd set noscrub 2024-03-11T21:33:05.972 INFO:teuthology.orchestra.run.smithi092.stdout:osd.0: {} 2024-03-11T21:33:05.988 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:33:06.017 INFO:teuthology.orchestra.run.smithi092.stdout:osd.1: {} 2024-03-11T21:33:06.019 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:33:06.019 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:33:06.020 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:33:06.073 INFO:teuthology.orchestra.run.smithi092.stdout:osd.2: {} 2024-03-11T21:33:06.121 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:33:06.128 INFO:teuthology.orchestra.run.smithi092.stdout:osd.3: {} 2024-03-11T21:33:06.175 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:33:06 smithi092 ceph-mon[29572]: pgmap v305: 158 pgs: 2 active+remapped+backfill_wait, 2 active+remapped+backfill_toofull, 6 active+remapped+backfilling, 148 active+clean; 5.7 GiB data, 19 GiB used, 697 GiB / 715 GiB avail; 853 B/s rd, 0 op/s; 51650/278763 objects misplaced (18.528%); 25 MiB/s, 396 objects/s recovering 2024-03-11T21:33:06.175 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:33:06 smithi092 ceph-mon[29572]: from='client.? v1:172.21.15.92:0/1770251064' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-11T21:33:06.175 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:33:06 smithi092 ceph-mon[37563]: pgmap v305: 158 pgs: 2 active+remapped+backfill_wait, 2 active+remapped+backfill_toofull, 6 active+remapped+backfilling, 148 active+clean; 5.7 GiB data, 19 GiB used, 697 GiB / 715 GiB avail; 853 B/s rd, 0 op/s; 51650/278763 objects misplaced (18.528%); 25 MiB/s, 396 objects/s recovering 2024-03-11T21:33:06.175 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:33:06 smithi092 ceph-mon[37563]: from='client.? v1:172.21.15.92:0/1770251064' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-11T21:33:06.176 INFO:teuthology.orchestra.run.smithi092.stdout:osd.4: {} 2024-03-11T21:33:06.210 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:33:06.211 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:33:06.211 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:33:06.214 INFO:teuthology.orchestra.run.smithi092.stdout:osd.5: {} 2024-03-11T21:33:06.265 INFO:teuthology.orchestra.run.smithi092.stdout:osd.6: {} 2024-03-11T21:33:06.308 INFO:teuthology.orchestra.run.smithi092.stdout:osd.7: {} 2024-03-11T21:33:06.312 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:33:06.317 INFO:teuthology.orchestra.run.smithi092.stderr:osd.0: osd_enable_op_tracker = '' 2024-03-11T21:33:06.317 INFO:teuthology.orchestra.run.smithi092.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-11T21:33:06.318 INFO:teuthology.orchestra.run.smithi092.stderr:osd.2: osd_enable_op_tracker = '' 2024-03-11T21:33:06.318 INFO:teuthology.orchestra.run.smithi092.stderr:osd.3: osd_enable_op_tracker = '' 2024-03-11T21:33:06.318 INFO:teuthology.orchestra.run.smithi092.stderr:osd.4: osd_enable_op_tracker = '' 2024-03-11T21:33:06.318 INFO:teuthology.orchestra.run.smithi092.stderr:osd.5: osd_enable_op_tracker = '' 2024-03-11T21:33:06.318 INFO:teuthology.orchestra.run.smithi092.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-11T21:33:06.318 INFO:teuthology.orchestra.run.smithi092.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-11T21:33:06.326 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:33:06 smithi097 ceph-mon[33317]: pgmap v305: 158 pgs: 2 active+remapped+backfill_wait, 2 active+remapped+backfill_toofull, 6 active+remapped+backfilling, 148 active+clean; 5.7 GiB data, 19 GiB used, 697 GiB / 715 GiB avail; 853 B/s rd, 0 op/s; 51650/278763 objects misplaced (18.528%); 25 MiB/s, 396 objects/s recovering 2024-03-11T21:33:06.326 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:33:06 smithi097 ceph-mon[33317]: from='client.? v1:172.21.15.92:0/1770251064' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-11T21:33:06.344 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:33:06.344 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:33:06.345 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:33:06.435 INFO:tasks.radosbench.radosbench.0.smithi092.stdout: 65 16 97001 96985 93.2408 61.875 0.0205744 0.0107198 2024-03-11T21:33:06.445 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:33:06.479 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:33:06.479 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:33:06.480 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:33:06.580 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:33:06.612 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:33:06.613 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:33:06.613 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:33:06.714 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:33:06.760 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:33:06.760 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:33:06.761 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:33:06.862 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:33:06.893 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:33:06.896 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:33:06.897 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:33:06.997 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:33:07.033 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:33:07.034 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:33:07.034 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:33:07.106 INFO:teuthology.orchestra.run.smithi092.stderr:reweighted osd.0 to 1 (10000) 2024-03-11T21:33:07.135 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:33:07.181 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:33:07.182 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:33:07.182 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:33:07.205 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:33:07 smithi097 ceph-mon[33317]: purged_snaps scrub starts 2024-03-11T21:33:07.205 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:33:07 smithi097 ceph-mon[33317]: purged_snaps scrub ok 2024-03-11T21:33:07.205 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:33:07 smithi097 ceph-mon[33317]: osdmap e188: 8 total, 8 up, 8 in 2024-03-11T21:33:07.205 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:33:07 smithi097 ceph-mon[33317]: from='mgr.14607 ' entity='mgr.y' 2024-03-11T21:33:07.205 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:33:07 smithi097 ceph-mon[33317]: from='mgr.14607 v1:172.21.15.92:0/1702717597' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T21:33:07.205 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:33:07 smithi097 ceph-mon[33317]: from='client.? v1:172.21.15.92:0/739942126' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]: dispatch 2024-03-11T21:33:07.283 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:33:07.313 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:33:07.314 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:33:07.314 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:33:07.360 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:33:07 smithi092 ceph-mon[29572]: purged_snaps scrub starts 2024-03-11T21:33:07.360 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:33:07 smithi092 ceph-mon[29572]: purged_snaps scrub ok 2024-03-11T21:33:07.360 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:33:07 smithi092 ceph-mon[29572]: osdmap e188: 8 total, 8 up, 8 in 2024-03-11T21:33:07.360 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:33:07 smithi092 ceph-mon[29572]: from='mgr.14607 ' entity='mgr.y' 2024-03-11T21:33:07.360 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:33:07 smithi092 ceph-mon[29572]: from='mgr.14607 v1:172.21.15.92:0/1702717597' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T21:33:07.360 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:33:07 smithi092 ceph-mon[29572]: from='client.? v1:172.21.15.92:0/739942126' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]: dispatch 2024-03-11T21:33:07.361 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:33:07 smithi092 ceph-mon[37563]: purged_snaps scrub starts 2024-03-11T21:33:07.361 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:33:07 smithi092 ceph-mon[37563]: purged_snaps scrub ok 2024-03-11T21:33:07.361 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:33:07 smithi092 ceph-mon[37563]: osdmap e188: 8 total, 8 up, 8 in 2024-03-11T21:33:07.361 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:33:07 smithi092 ceph-mon[37563]: from='mgr.14607 ' entity='mgr.y' 2024-03-11T21:33:07.361 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:33:07 smithi092 ceph-mon[37563]: from='mgr.14607 v1:172.21.15.92:0/1702717597' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T21:33:07.361 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:33:07 smithi092 ceph-mon[37563]: from='client.? v1:172.21.15.92:0/739942126' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]: dispatch 2024-03-11T21:33:07.415 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:33:07.436 INFO:tasks.radosbench.radosbench.0.smithi092.stdout: 66 16 98034 98018 92.8062 64.5625 0.0155844 0.0107699 2024-03-11T21:33:07.461 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:33:07.462 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:33:07.462 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:33:07.563 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:33:07.587 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:33:07.588 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:33:07.588 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:33:07.689 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:33:07.727 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:33:07.727 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:33:07.728 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:33:07.829 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:33:07.870 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:33:07.870 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:33:07.871 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:33:07.973 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:33:07.999 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:33:07.999 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:33:08.000 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:33:08.004 INFO:teuthology.orchestra.run.smithi097.stdout:{ 2024-03-11T21:33:08.004 INFO:teuthology.orchestra.run.smithi097.stdout: "size": 20, 2024-03-11T21:33:08.004 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 600, 2024-03-11T21:33:08.004 INFO:teuthology.orchestra.run.smithi097.stdout: "ops": [ 2024-03-11T21:33:08.004 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:33:08.005 INFO:teuthology.orchestra.run.smithi097.stdout: "description": "osd_op(client.34715.0:139390 7.7 7:e33e1b77:::benchmark_data_smithi092_76040_object17479:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e124)", 2024-03-11T21:33:08.005 INFO:teuthology.orchestra.run.smithi097.stdout: "initiated_at": "2024-03-11T21:29:33.973801+0000", 2024-03-11T21:33:08.005 INFO:teuthology.orchestra.run.smithi097.stdout: "age": 214.04723497099999, 2024-03-11T21:33:08.005 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.17549214799999999, 2024-03-11T21:33:08.005 INFO:teuthology.orchestra.run.smithi097.stdout: "type_data": { 2024-03-11T21:33:08.005 INFO:teuthology.orchestra.run.smithi097.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-11T21:33:08.005 INFO:teuthology.orchestra.run.smithi097.stdout: "client_info": { 2024-03-11T21:33:08.005 INFO:teuthology.orchestra.run.smithi097.stdout: "client": "client.34715", 2024-03-11T21:33:08.005 INFO:teuthology.orchestra.run.smithi097.stdout: "client_addr": "v1:172.21.15.92:0/2403132573", 2024-03-11T21:33:08.005 INFO:teuthology.orchestra.run.smithi097.stdout: "tid": 139390 2024-03-11T21:33:08.005 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:33:08.005 INFO:teuthology.orchestra.run.smithi097.stdout: "events": [ 2024-03-11T21:33:08.005 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:33:08.005 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "initiated", 2024-03-11T21:33:08.006 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:29:33.973801+0000", 2024-03-11T21:33:08.006 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-03-11T21:33:08.006 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:33:08.006 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:33:08.006 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "header_read", 2024-03-11T21:33:08.006 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:29:33.973801+0000", 2024-03-11T21:33:08.006 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-03-11T21:33:08.006 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:33:08.006 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:33:08.006 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "throttled", 2024-03-11T21:33:08.006 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:29:33.973802+0000", 2024-03-11T21:33:08.006 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.35e-06 2024-03-11T21:33:08.006 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:33:08.006 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:33:08.007 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "all_read", 2024-03-11T21:33:08.007 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:29:33.973806+0000", 2024-03-11T21:33:08.007 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 4.1919999999999998e-06 2024-03-11T21:33:08.007 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:33:08.007 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:33:08.007 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "dispatched", 2024-03-11T21:33:08.007 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:29:33.973817+0000", 2024-03-11T21:33:08.007 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.0295000000000001e-05 2024-03-11T21:33:08.007 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:33:08.007 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:33:08.007 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "queued_for_pg", 2024-03-11T21:33:08.008 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:29:33.973822+0000", 2024-03-11T21:33:08.008 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 5.8529999999999997e-06 2024-03-11T21:33:08.008 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:33:08.008 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:33:08.008 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "reached_pg", 2024-03-11T21:33:08.009 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:29:33.973871+0000", 2024-03-11T21:33:08.009 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 4.8217999999999998e-05 2024-03-11T21:33:08.009 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:33:08.009 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:33:08.009 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "started", 2024-03-11T21:33:08.009 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:29:33.973969+0000", 2024-03-11T21:33:08.009 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 9.8329e-05 2024-03-11T21:33:08.009 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:33:08.009 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:33:08.009 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "waiting for subops from 0,3", 2024-03-11T21:33:08.009 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:29:33.974040+0000", 2024-03-11T21:33:08.010 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 7.1097e-05 2024-03-11T21:33:08.010 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:33:08.010 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:33:08.010 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "op_commit", 2024-03-11T21:33:08.010 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:29:33.976796+0000", 2024-03-11T21:33:08.010 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.002755632 2024-03-11T21:33:08.010 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:33:08.010 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:33:08.010 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "sub_op_commit_rec", 2024-03-11T21:33:08.010 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:29:33.991263+0000", 2024-03-11T21:33:08.010 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.014467117 2024-03-11T21:33:08.010 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:33:08.010 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:33:08.010 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "sub_op_commit_rec", 2024-03-11T21:33:08.011 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:29:34.149224+0000", 2024-03-11T21:33:08.011 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.15796159600000001 2024-03-11T21:33:08.011 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:33:08.011 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:33:08.011 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "commit_sent", 2024-03-11T21:33:08.011 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:29:34.149258+0000", 2024-03-11T21:33:08.011 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 3.3912999999999999e-05 2024-03-11T21:33:08.011 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:33:08.011 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:33:08.011 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "done", 2024-03-11T21:33:08.011 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:29:34.149293+0000", 2024-03-11T21:33:08.011 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 3.4555999999999999e-05 2024-03-11T21:33:08.011 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-03-11T21:33:08.012 INFO:teuthology.orchestra.run.smithi097.stdout: ] 2024-03-11T21:33:08.012 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-03-11T21:33:08.012 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:33:08.012 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:33:08.012 INFO:teuthology.orchestra.run.smithi097.stdout: "description": "osd_op(client.34715.0:139397 7.7 7:ec3d637a:::benchmark_data_smithi092_76040_object17486:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e124)", 2024-03-11T21:33:08.012 INFO:teuthology.orchestra.run.smithi097.stdout: "initiated_at": "2024-03-11T21:29:33.982578+0000", 2024-03-11T21:33:08.012 INFO:teuthology.orchestra.run.smithi097.stdout: "age": 214.03845817499999, 2024-03-11T21:33:08.012 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.16841289000000001, 2024-03-11T21:33:08.012 INFO:teuthology.orchestra.run.smithi097.stdout: "type_data": { 2024-03-11T21:33:08.012 INFO:teuthology.orchestra.run.smithi097.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-11T21:33:08.012 INFO:teuthology.orchestra.run.smithi097.stdout: "client_info": { 2024-03-11T21:33:08.013 INFO:teuthology.orchestra.run.smithi097.stdout: "client": "client.34715", 2024-03-11T21:33:08.013 INFO:teuthology.orchestra.run.smithi097.stdout: "client_addr": "v1:172.21.15.92:0/2403132573", 2024-03-11T21:33:08.013 INFO:teuthology.orchestra.run.smithi097.stdout: "tid": 139397 2024-03-11T21:33:08.013 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:33:08.013 INFO:teuthology.orchestra.run.smithi097.stdout: "events": [ 2024-03-11T21:33:08.013 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:33:08.013 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "initiated", 2024-03-11T21:33:08.013 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:29:33.982578+0000", 2024-03-11T21:33:08.013 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-03-11T21:33:08.013 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:33:08.013 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:33:08.013 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "header_read", 2024-03-11T21:33:08.013 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:29:33.982578+0000", 2024-03-11T21:33:08.013 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-03-11T21:33:08.013 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:33:08.014 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:33:08.014 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "throttled", 2024-03-11T21:33:08.014 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:29:33.982580+0000", 2024-03-11T21:33:08.014 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 2.249e-06 2024-03-11T21:33:08.014 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:33:08.014 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:33:08.014 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "all_read", 2024-03-11T21:33:08.014 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:29:33.982586+0000", 2024-03-11T21:33:08.014 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 6.2430000000000002e-06 2024-03-11T21:33:08.014 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:33:08.014 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:33:08.014 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "dispatched", 2024-03-11T21:33:08.014 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:29:33.982600+0000", 2024-03-11T21:33:08.014 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.411e-05 2024-03-11T21:33:08.015 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:33:08.015 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:33:08.015 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "queued_for_pg", 2024-03-11T21:33:08.015 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:29:33.982609+0000", 2024-03-11T21:33:08.015 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 8.935e-06 2024-03-11T21:33:08.015 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:33:08.015 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:33:08.015 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "reached_pg", 2024-03-11T21:33:08.015 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:29:33.982672+0000", 2024-03-11T21:33:08.015 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 6.2643000000000003e-05 2024-03-11T21:33:08.015 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:33:08.015 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:33:08.015 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "started", 2024-03-11T21:33:08.015 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:29:33.982792+0000", 2024-03-11T21:33:08.015 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.00012063899999999999 2024-03-11T21:33:08.016 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:33:08.016 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:33:08.016 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "waiting for subops from 0,3", 2024-03-11T21:33:08.016 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:29:33.982906+0000", 2024-03-11T21:33:08.016 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.000114123 2024-03-11T21:33:08.016 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:33:08.016 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:33:08.016 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "op_commit", 2024-03-11T21:33:08.016 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:29:33.988300+0000", 2024-03-11T21:33:08.016 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.0053937610000000004 2024-03-11T21:33:08.016 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:33:08.016 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:33:08.016 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "sub_op_commit_rec", 2024-03-11T21:33:08.016 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:29:34.000785+0000", 2024-03-11T21:33:08.016 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.012484567 2024-03-11T21:33:08.017 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:33:08.017 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:33:08.017 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "sub_op_commit_rec", 2024-03-11T21:33:08.017 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:29:34.150912+0000", 2024-03-11T21:33:08.017 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.15012709499999999 2024-03-11T21:33:08.017 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:33:08.017 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:33:08.017 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "commit_sent", 2024-03-11T21:33:08.017 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:29:34.150944+0000", 2024-03-11T21:33:08.017 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 3.2005999999999998e-05 2024-03-11T21:33:08.017 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:33:08.017 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:33:08.017 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "done", 2024-03-11T21:33:08.017 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:29:34.150990+0000", 2024-03-11T21:33:08.017 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 4.6519e-05 2024-03-11T21:33:08.018 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-03-11T21:33:08.018 INFO:teuthology.orchestra.run.smithi097.stdout: ] 2024-03-11T21:33:08.018 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-03-11T21:33:08.018 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:33:08.018 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:33:08.018 INFO:teuthology.orchestra.run.smithi097.stdout: "description": "osd_op(client.34715.0:144300 7.7 7:e2e00930:::benchmark_data_smithi092_76040_object22389:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e124)", 2024-03-11T21:33:08.018 INFO:teuthology.orchestra.run.smithi097.stdout: "initiated_at": "2024-03-11T21:29:39.693117+0000", 2024-03-11T21:33:08.018 INFO:teuthology.orchestra.run.smithi097.stdout: "age": 208.327918399, 2024-03-11T21:33:08.018 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.135023847, 2024-03-11T21:33:08.018 INFO:teuthology.orchestra.run.smithi097.stdout: "type_data": { 2024-03-11T21:33:08.018 INFO:teuthology.orchestra.run.smithi097.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-11T21:33:08.018 INFO:teuthology.orchestra.run.smithi097.stdout: "client_info": { 2024-03-11T21:33:08.018 INFO:teuthology.orchestra.run.smithi097.stdout: "client": "client.34715", 2024-03-11T21:33:08.018 INFO:teuthology.orchestra.run.smithi097.stdout: "client_addr": "v1:172.21.15.92:0/2403132573", 2024-03-11T21:33:08.019 INFO:teuthology.orchestra.run.smithi097.stdout: "tid": 144300 2024-03-11T21:33:08.019 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:33:08.019 INFO:teuthology.orchestra.run.smithi097.stdout: "events": [ 2024-03-11T21:33:08.019 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:33:08.019 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "initiated", 2024-03-11T21:33:08.020 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:29:39.693117+0000", 2024-03-11T21:33:08.020 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-03-11T21:33:08.020 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:33:08.020 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:33:08.020 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "header_read", 2024-03-11T21:33:08.020 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:29:39.693117+0000", 2024-03-11T21:33:08.020 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-03-11T21:33:08.020 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:33:08.020 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:33:08.020 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "throttled", 2024-03-11T21:33:08.020 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:29:39.693119+0000", 2024-03-11T21:33:08.020 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.889e-06 2024-03-11T21:33:08.021 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:33:08.021 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:33:08.021 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "all_read", 2024-03-11T21:33:08.021 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:29:39.693125+0000", 2024-03-11T21:33:08.021 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 5.677e-06 2024-03-11T21:33:08.021 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:33:08.021 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:33:08.021 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "dispatched", 2024-03-11T21:33:08.021 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:29:39.693139+0000", 2024-03-11T21:33:08.021 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.3631999999999999e-05 2024-03-11T21:33:08.021 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:33:08.021 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:33:08.021 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "queued_for_pg", 2024-03-11T21:33:08.021 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:29:39.693146+0000", 2024-03-11T21:33:08.021 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 7.571e-06 2024-03-11T21:33:08.022 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:33:08.022 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:33:08.022 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "reached_pg", 2024-03-11T21:33:08.022 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:29:39.693200+0000", 2024-03-11T21:33:08.022 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 5.4194999999999999e-05 2024-03-11T21:33:08.022 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:33:08.022 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:33:08.022 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "started", 2024-03-11T21:33:08.022 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:29:39.693320+0000", 2024-03-11T21:33:08.022 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.00011953 2024-03-11T21:33:08.022 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:33:08.022 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:33:08.022 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "waiting for subops from 0,3", 2024-03-11T21:33:08.022 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:29:39.693452+0000", 2024-03-11T21:33:08.023 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.00013235499999999999 2024-03-11T21:33:08.023 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:33:08.023 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:33:08.023 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "op_commit", 2024-03-11T21:33:08.023 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:29:39.696359+0000", 2024-03-11T21:33:08.023 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.002906788 2024-03-11T21:33:08.023 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:33:08.023 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:33:08.023 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "sub_op_commit_rec", 2024-03-11T21:33:08.023 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:29:39.729028+0000", 2024-03-11T21:33:08.023 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.032669147000000003 2024-03-11T21:33:08.023 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:33:08.023 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:33:08.023 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "sub_op_commit_rec", 2024-03-11T21:33:08.024 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:29:39.828070+0000", 2024-03-11T21:33:08.024 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.099042023000000007 2024-03-11T21:33:08.024 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:33:08.024 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:33:08.024 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "commit_sent", 2024-03-11T21:33:08.024 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:29:39.828103+0000", 2024-03-11T21:33:08.024 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 3.3101e-05 2024-03-11T21:33:08.024 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:33:08.024 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:33:08.024 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "done", 2024-03-11T21:33:08.024 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:29:39.828141+0000", 2024-03-11T21:33:08.024 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 3.7938999999999997e-05 2024-03-11T21:33:08.024 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-03-11T21:33:08.024 INFO:teuthology.orchestra.run.smithi097.stdout: ] 2024-03-11T21:33:08.024 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-03-11T21:33:08.025 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:33:08.025 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:33:08.025 INFO:teuthology.orchestra.run.smithi097.stdout: "description": "osd_op(client.34715.0:153209 7.7 7:ed96a7d9:::benchmark_data_smithi092_76040_object31298:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e126)", 2024-03-11T21:33:08.025 INFO:teuthology.orchestra.run.smithi097.stdout: "initiated_at": "2024-03-11T21:29:52.339978+0000", 2024-03-11T21:33:08.025 INFO:teuthology.orchestra.run.smithi097.stdout: "age": 195.68105814699999, 2024-03-11T21:33:08.025 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.130714104, 2024-03-11T21:33:08.025 INFO:teuthology.orchestra.run.smithi097.stdout: "type_data": { 2024-03-11T21:33:08.025 INFO:teuthology.orchestra.run.smithi097.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-11T21:33:08.025 INFO:teuthology.orchestra.run.smithi097.stdout: "client_info": { 2024-03-11T21:33:08.025 INFO:teuthology.orchestra.run.smithi097.stdout: "client": "client.34715", 2024-03-11T21:33:08.025 INFO:teuthology.orchestra.run.smithi097.stdout: "client_addr": "v1:172.21.15.92:0/2403132573", 2024-03-11T21:33:08.025 INFO:teuthology.orchestra.run.smithi097.stdout: "tid": 153209 2024-03-11T21:33:08.025 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:33:08.025 INFO:teuthology.orchestra.run.smithi097.stdout: "events": [ 2024-03-11T21:33:08.025 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:33:08.026 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "initiated", 2024-03-11T21:33:08.026 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:29:52.339978+0000", 2024-03-11T21:33:08.026 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-03-11T21:33:08.026 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:33:08.026 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:33:08.026 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "header_read", 2024-03-11T21:33:08.026 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:29:52.339978+0000", 2024-03-11T21:33:08.026 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-03-11T21:33:08.026 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:33:08.026 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:33:08.026 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "throttled", 2024-03-11T21:33:08.026 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:29:52.339980+0000", 2024-03-11T21:33:08.026 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 2.0490000000000002e-06 2024-03-11T21:33:08.026 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:33:08.027 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:33:08.027 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "all_read", 2024-03-11T21:33:08.027 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:29:52.339985+0000", 2024-03-11T21:33:08.027 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 4.9110000000000001e-06 2024-03-11T21:33:08.027 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:33:08.027 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:33:08.027 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "dispatched", 2024-03-11T21:33:08.027 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:29:52.339995+0000", 2024-03-11T21:33:08.027 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.0706e-05 2024-03-11T21:33:08.027 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:33:08.027 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:33:08.027 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "queued_for_pg", 2024-03-11T21:33:08.027 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:29:52.340002+0000", 2024-03-11T21:33:08.027 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 6.9160000000000004e-06 2024-03-11T21:33:08.027 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:33:08.028 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:33:08.028 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "reached_pg", 2024-03-11T21:33:08.028 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:29:52.340074+0000", 2024-03-11T21:33:08.028 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 7.1897000000000006e-05 2024-03-11T21:33:08.028 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:33:08.028 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:33:08.028 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "started", 2024-03-11T21:33:08.028 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:29:52.340212+0000", 2024-03-11T21:33:08.028 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.00013782900000000001 2024-03-11T21:33:08.028 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:33:08.028 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:33:08.028 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "waiting for subops from 0,3", 2024-03-11T21:33:08.028 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:29:52.340329+0000", 2024-03-11T21:33:08.028 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.000117428 2024-03-11T21:33:08.029 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:33:08.029 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:33:08.029 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "op_commit", 2024-03-11T21:33:08.029 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:29:52.344131+0000", 2024-03-11T21:33:08.029 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.003801582 2024-03-11T21:33:08.029 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:33:08.029 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:33:08.029 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "sub_op_commit_rec", 2024-03-11T21:33:08.029 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:29:52.357426+0000", 2024-03-11T21:33:08.029 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.013295039999999999 2024-03-11T21:33:08.029 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:33:08.029 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:33:08.029 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "sub_op_commit_rec", 2024-03-11T21:33:08.029 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:29:52.470631+0000", 2024-03-11T21:33:08.029 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.11320485299999999 2024-03-11T21:33:08.030 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:33:08.030 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:33:08.030 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "commit_sent", 2024-03-11T21:33:08.030 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:29:52.470665+0000", 2024-03-11T21:33:08.030 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 3.3986000000000003e-05 2024-03-11T21:33:08.030 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:33:08.030 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:33:08.030 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "done", 2024-03-11T21:33:08.030 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:29:52.470692+0000", 2024-03-11T21:33:08.030 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 2.6906999999999998e-05 2024-03-11T21:33:08.030 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-03-11T21:33:08.030 INFO:teuthology.orchestra.run.smithi097.stdout: ] 2024-03-11T21:33:08.030 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-03-11T21:33:08.030 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:33:08.030 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:33:08.031 INFO:teuthology.orchestra.run.smithi097.stdout: "description": "osd_op(client.34715.0:153402 7.7 7:e10f6c31:::benchmark_data_smithi092_76040_object31491:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e126)", 2024-03-11T21:33:08.031 INFO:teuthology.orchestra.run.smithi097.stdout: "initiated_at": "2024-03-11T21:29:52.570450+0000", 2024-03-11T21:33:08.031 INFO:teuthology.orchestra.run.smithi097.stdout: "age": 195.45058589199999, 2024-03-11T21:33:08.031 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.131075313, 2024-03-11T21:33:08.031 INFO:teuthology.orchestra.run.smithi097.stdout: "type_data": { 2024-03-11T21:33:08.031 INFO:teuthology.orchestra.run.smithi097.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-11T21:33:08.031 INFO:teuthology.orchestra.run.smithi097.stdout: "client_info": { 2024-03-11T21:33:08.031 INFO:teuthology.orchestra.run.smithi097.stdout: "client": "client.34715", 2024-03-11T21:33:08.031 INFO:teuthology.orchestra.run.smithi097.stdout: "client_addr": "v1:172.21.15.92:0/2403132573", 2024-03-11T21:33:08.031 INFO:teuthology.orchestra.run.smithi097.stdout: "tid": 153402 2024-03-11T21:33:08.032 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:33:08.032 INFO:teuthology.orchestra.run.smithi097.stdout: "events": [ 2024-03-11T21:33:08.032 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:33:08.032 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "initiated", 2024-03-11T21:33:08.032 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:29:52.570450+0000", 2024-03-11T21:33:08.032 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-03-11T21:33:08.032 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:33:08.032 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:33:08.032 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "header_read", 2024-03-11T21:33:08.032 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:29:52.570450+0000", 2024-03-11T21:33:08.032 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-03-11T21:33:08.032 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:33:08.032 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:33:08.032 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "throttled", 2024-03-11T21:33:08.032 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:29:52.570452+0000", 2024-03-11T21:33:08.033 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.7320000000000001e-06 2024-03-11T21:33:08.033 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:33:08.033 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:33:08.033 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "all_read", 2024-03-11T21:33:08.033 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:29:52.570457+0000", 2024-03-11T21:33:08.033 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 5.4620000000000004e-06 2024-03-11T21:33:08.033 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:33:08.033 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:33:08.033 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "dispatched", 2024-03-11T21:33:08.033 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:29:52.570467+0000", 2024-03-11T21:33:08.033 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.0152e-05 2024-03-11T21:33:08.034 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:33:08.034 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:33:08.034 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "queued_for_pg", 2024-03-11T21:33:08.034 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:29:52.570473+0000", 2024-03-11T21:33:08.034 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 5.9780000000000002e-06 2024-03-11T21:33:08.034 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:33:08.034 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:33:08.034 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "reached_pg", 2024-03-11T21:33:08.034 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:29:52.570516+0000", 2024-03-11T21:33:08.034 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 4.3167999999999997e-05 2024-03-11T21:33:08.034 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:33:08.034 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:33:08.034 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "started", 2024-03-11T21:33:08.034 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:29:52.570633+0000", 2024-03-11T21:33:08.035 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.000116631 2024-03-11T21:33:08.035 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:33:08.035 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:33:08.035 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "waiting for subops from 0,3", 2024-03-11T21:33:08.035 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:29:52.570773+0000", 2024-03-11T21:33:08.035 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.00014012399999999999 2024-03-11T21:33:08.035 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:33:08.035 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:33:08.035 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "op_commit", 2024-03-11T21:33:08.035 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:29:52.573627+0000", 2024-03-11T21:33:08.035 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.0028536780000000001 2024-03-11T21:33:08.035 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:33:08.035 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:33:08.035 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "sub_op_commit_rec", 2024-03-11T21:33:08.036 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:29:52.594284+0000", 2024-03-11T21:33:08.036 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.020657149 2024-03-11T21:33:08.036 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:33:08.036 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:33:08.036 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "sub_op_commit_rec", 2024-03-11T21:33:08.036 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:29:52.701421+0000", 2024-03-11T21:33:08.036 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.107137526 2024-03-11T21:33:08.036 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:33:08.036 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:33:08.036 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "commit_sent", 2024-03-11T21:33:08.036 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:29:52.701479+0000", 2024-03-11T21:33:08.036 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 5.7379000000000001e-05 2024-03-11T21:33:08.036 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:33:08.036 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:33:08.037 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "done", 2024-03-11T21:33:08.037 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:29:52.701525+0000", 2024-03-11T21:33:08.037 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 4.6334000000000001e-05 2024-03-11T21:33:08.037 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-03-11T21:33:08.037 INFO:teuthology.orchestra.run.smithi097.stdout: ] 2024-03-11T21:33:08.037 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-03-11T21:33:08.037 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:33:08.037 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:33:08.037 INFO:teuthology.orchestra.run.smithi097.stdout: "description": "osd_op(client.34715.0:153875 7.7 7:e90aeecd:::benchmark_data_smithi092_76040_object31964:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e126)", 2024-03-11T21:33:08.037 INFO:teuthology.orchestra.run.smithi097.stdout: "initiated_at": "2024-03-11T21:29:53.270395+0000", 2024-03-11T21:33:08.037 INFO:teuthology.orchestra.run.smithi097.stdout: "age": 194.75064084300001, 2024-03-11T21:33:08.037 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.13311833100000001, 2024-03-11T21:33:08.037 INFO:teuthology.orchestra.run.smithi097.stdout: "type_data": { 2024-03-11T21:33:08.037 INFO:teuthology.orchestra.run.smithi097.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-11T21:33:08.038 INFO:teuthology.orchestra.run.smithi097.stdout: "client_info": { 2024-03-11T21:33:08.038 INFO:teuthology.orchestra.run.smithi097.stdout: "client": "client.34715", 2024-03-11T21:33:08.038 INFO:teuthology.orchestra.run.smithi097.stdout: "client_addr": "v1:172.21.15.92:0/2403132573", 2024-03-11T21:33:08.038 INFO:teuthology.orchestra.run.smithi097.stdout: "tid": 153875 2024-03-11T21:33:08.038 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:33:08.038 INFO:teuthology.orchestra.run.smithi097.stdout: "events": [ 2024-03-11T21:33:08.038 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:33:08.038 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "initiated", 2024-03-11T21:33:08.038 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:29:53.270395+0000", 2024-03-11T21:33:08.038 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-03-11T21:33:08.038 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:33:08.038 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:33:08.038 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "header_read", 2024-03-11T21:33:08.038 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:29:53.270395+0000", 2024-03-11T21:33:08.039 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-03-11T21:33:08.039 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:33:08.039 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:33:08.039 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "throttled", 2024-03-11T21:33:08.039 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:29:53.270396+0000", 2024-03-11T21:33:08.039 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 6.3099999999999997e-07 2024-03-11T21:33:08.039 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:33:08.039 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:33:08.039 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "all_read", 2024-03-11T21:33:08.039 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:29:53.270398+0000", 2024-03-11T21:33:08.039 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 2.7250000000000002e-06 2024-03-11T21:33:08.039 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:33:08.039 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:33:08.039 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "dispatched", 2024-03-11T21:33:08.039 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:29:53.270404+0000", 2024-03-11T21:33:08.040 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 5.4600000000000002e-06 2024-03-11T21:33:08.040 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:33:08.040 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:33:08.040 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "queued_for_pg", 2024-03-11T21:33:08.040 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:29:53.270412+0000", 2024-03-11T21:33:08.040 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 8.0169999999999996e-06 2024-03-11T21:33:08.040 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:33:08.040 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:33:08.040 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "reached_pg", 2024-03-11T21:33:08.040 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:29:53.273086+0000", 2024-03-11T21:33:08.040 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.0026740679999999999 2024-03-11T21:33:08.040 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:33:08.040 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:33:08.040 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "started", 2024-03-11T21:33:08.040 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:29:53.273206+0000", 2024-03-11T21:33:08.041 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.000119879 2024-03-11T21:33:08.041 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:33:08.041 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:33:08.041 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "waiting for subops from 0,3", 2024-03-11T21:33:08.041 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:29:53.273332+0000", 2024-03-11T21:33:08.041 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.00012594400000000001 2024-03-11T21:33:08.041 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:33:08.041 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:33:08.041 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "op_commit", 2024-03-11T21:33:08.041 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:29:53.287050+0000", 2024-03-11T21:33:08.041 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.013718584000000001 2024-03-11T21:33:08.041 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:33:08.041 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:33:08.041 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "sub_op_commit_rec", 2024-03-11T21:33:08.041 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:29:53.298207+0000", 2024-03-11T21:33:08.042 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.011156998 2024-03-11T21:33:08.042 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:33:08.042 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:33:08.042 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "sub_op_commit_rec", 2024-03-11T21:33:08.042 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:29:53.403446+0000", 2024-03-11T21:33:08.042 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.105238767 2024-03-11T21:33:08.042 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:33:08.042 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:33:08.042 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "commit_sent", 2024-03-11T21:33:08.042 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:29:53.403480+0000", 2024-03-11T21:33:08.043 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 3.3951000000000001e-05 2024-03-11T21:33:08.043 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:33:08.043 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:33:08.043 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "done", 2024-03-11T21:33:08.043 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:29:53.403513+0000", 2024-03-11T21:33:08.043 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 3.3306999999999998e-05 2024-03-11T21:33:08.043 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-03-11T21:33:08.043 INFO:teuthology.orchestra.run.smithi097.stdout: ] 2024-03-11T21:33:08.043 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-03-11T21:33:08.043 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:33:08.043 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:33:08.043 INFO:teuthology.orchestra.run.smithi097.stdout: "description": "osd_op(client.34715.0:153882 7.7 7:eb8589cc:::benchmark_data_smithi092_76040_object31971:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e126)", 2024-03-11T21:33:08.044 INFO:teuthology.orchestra.run.smithi097.stdout: "initiated_at": "2024-03-11T21:29:53.277548+0000", 2024-03-11T21:33:08.044 INFO:teuthology.orchestra.run.smithi097.stdout: "age": 194.743488039, 2024-03-11T21:33:08.044 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.14272742099999999, 2024-03-11T21:33:08.044 INFO:teuthology.orchestra.run.smithi097.stdout: "type_data": { 2024-03-11T21:33:08.044 INFO:teuthology.orchestra.run.smithi097.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-11T21:33:08.044 INFO:teuthology.orchestra.run.smithi097.stdout: "client_info": { 2024-03-11T21:33:08.044 INFO:teuthology.orchestra.run.smithi097.stdout: "client": "client.34715", 2024-03-11T21:33:08.044 INFO:teuthology.orchestra.run.smithi097.stdout: "client_addr": "v1:172.21.15.92:0/2403132573", 2024-03-11T21:33:08.044 INFO:teuthology.orchestra.run.smithi097.stdout: "tid": 153882 2024-03-11T21:33:08.044 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:33:08.044 INFO:teuthology.orchestra.run.smithi097.stdout: "events": [ 2024-03-11T21:33:08.044 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:33:08.044 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "initiated", 2024-03-11T21:33:08.044 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:29:53.277548+0000", 2024-03-11T21:33:08.045 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-03-11T21:33:08.045 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:33:08.045 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:33:08.045 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "header_read", 2024-03-11T21:33:08.045 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:29:53.277548+0000", 2024-03-11T21:33:08.045 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-03-11T21:33:08.045 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:33:08.045 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:33:08.045 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "throttled", 2024-03-11T21:33:08.045 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:29:53.277549+0000", 2024-03-11T21:33:08.045 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.395e-06 2024-03-11T21:33:08.045 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:33:08.045 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:33:08.045 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "all_read", 2024-03-11T21:33:08.045 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:29:53.277554+0000", 2024-03-11T21:33:08.046 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 5.0440000000000003e-06 2024-03-11T21:33:08.046 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:33:08.046 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:33:08.046 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "dispatched", 2024-03-11T21:33:08.046 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:29:53.277562+0000", 2024-03-11T21:33:08.046 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 7.4749999999999996e-06 2024-03-11T21:33:08.046 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:33:08.046 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:33:08.046 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "queued_for_pg", 2024-03-11T21:33:08.046 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:29:53.277568+0000", 2024-03-11T21:33:08.046 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 6.2480000000000001e-06 2024-03-11T21:33:08.046 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:33:08.046 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:33:08.046 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "reached_pg", 2024-03-11T21:33:08.047 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:29:53.280778+0000", 2024-03-11T21:33:08.047 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.003210559 2024-03-11T21:33:08.047 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:33:08.047 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:33:08.047 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "started", 2024-03-11T21:33:08.047 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:29:53.280862+0000", 2024-03-11T21:33:08.047 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 8.3116999999999997e-05 2024-03-11T21:33:08.047 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:33:08.047 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:33:08.047 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "waiting for subops from 0,3", 2024-03-11T21:33:08.047 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:29:53.280936+0000", 2024-03-11T21:33:08.047 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 7.4949000000000004e-05 2024-03-11T21:33:08.047 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:33:08.047 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:33:08.048 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "op_commit", 2024-03-11T21:33:08.048 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:29:53.287272+0000", 2024-03-11T21:33:08.048 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.0063358640000000001 2024-03-11T21:33:08.048 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:33:08.048 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:33:08.048 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "sub_op_commit_rec", 2024-03-11T21:33:08.048 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:29:53.298265+0000", 2024-03-11T21:33:08.048 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.010992985 2024-03-11T21:33:08.048 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:33:08.048 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:33:08.048 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "sub_op_commit_rec", 2024-03-11T21:33:08.048 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:29:53.420214+0000", 2024-03-11T21:33:08.048 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.121949115 2024-03-11T21:33:08.048 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:33:08.048 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:33:08.049 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "commit_sent", 2024-03-11T21:33:08.049 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:29:53.420247+0000", 2024-03-11T21:33:08.049 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 3.2975999999999997e-05 2024-03-11T21:33:08.049 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:33:08.049 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:33:08.049 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "done", 2024-03-11T21:33:08.049 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:29:53.420275+0000", 2024-03-11T21:33:08.049 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 2.7693999999999999e-05 2024-03-11T21:33:08.049 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-03-11T21:33:08.049 INFO:teuthology.orchestra.run.smithi097.stdout: ] 2024-03-11T21:33:08.049 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-03-11T21:33:08.049 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:33:08.049 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:33:08.049 INFO:teuthology.orchestra.run.smithi097.stdout: "description": "osd_op(client.34715.0:153883 7.7 7:ee50744c:::benchmark_data_smithi092_76040_object31972:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e126)", 2024-03-11T21:33:08.050 INFO:teuthology.orchestra.run.smithi097.stdout: "initiated_at": "2024-03-11T21:29:53.277575+0000", 2024-03-11T21:33:08.050 INFO:teuthology.orchestra.run.smithi097.stdout: "age": 194.74346032099999, 2024-03-11T21:33:08.050 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.14281391299999999, 2024-03-11T21:33:08.050 INFO:teuthology.orchestra.run.smithi097.stdout: "type_data": { 2024-03-11T21:33:08.050 INFO:teuthology.orchestra.run.smithi097.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-11T21:33:08.050 INFO:teuthology.orchestra.run.smithi097.stdout: "client_info": { 2024-03-11T21:33:08.050 INFO:teuthology.orchestra.run.smithi097.stdout: "client": "client.34715", 2024-03-11T21:33:08.050 INFO:teuthology.orchestra.run.smithi097.stdout: "client_addr": "v1:172.21.15.92:0/2403132573", 2024-03-11T21:33:08.050 INFO:teuthology.orchestra.run.smithi097.stdout: "tid": 153883 2024-03-11T21:33:08.050 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:33:08.050 INFO:teuthology.orchestra.run.smithi097.stdout: "events": [ 2024-03-11T21:33:08.050 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:33:08.050 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "initiated", 2024-03-11T21:33:08.050 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:29:53.277575+0000", 2024-03-11T21:33:08.051 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-03-11T21:33:08.051 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:33:08.051 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:33:08.051 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "header_read", 2024-03-11T21:33:08.051 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:29:53.277575+0000", 2024-03-11T21:33:08.051 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-03-11T21:33:08.051 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:33:08.051 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:33:08.051 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "throttled", 2024-03-11T21:33:08.051 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:29:53.277576+0000", 2024-03-11T21:33:08.052 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 4.46e-07 2024-03-11T21:33:08.052 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:33:08.052 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:33:08.052 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "all_read", 2024-03-11T21:33:08.052 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:29:53.277578+0000", 2024-03-11T21:33:08.052 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 2.3769999999999999e-06 2024-03-11T21:33:08.052 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:33:08.052 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:33:08.052 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "dispatched", 2024-03-11T21:33:08.052 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:29:53.277583+0000", 2024-03-11T21:33:08.052 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 4.5680000000000001e-06 2024-03-11T21:33:08.052 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:33:08.053 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:33:08.053 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "queued_for_pg", 2024-03-11T21:33:08.053 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:29:53.277587+0000", 2024-03-11T21:33:08.053 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 4.4549999999999997e-06 2024-03-11T21:33:08.053 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:33:08.053 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:33:08.053 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "reached_pg", 2024-03-11T21:33:08.053 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:29:53.284150+0000", 2024-03-11T21:33:08.053 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.0065629340000000003 2024-03-11T21:33:08.053 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:33:08.053 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:33:08.053 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "started", 2024-03-11T21:33:08.053 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:29:53.284267+0000", 2024-03-11T21:33:08.053 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.000117298 2024-03-11T21:33:08.053 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:33:08.054 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:33:08.054 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "waiting for subops from 0,3", 2024-03-11T21:33:08.054 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:29:53.284390+0000", 2024-03-11T21:33:08.054 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.000122924 2024-03-11T21:33:08.054 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:33:08.054 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:33:08.054 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "op_commit", 2024-03-11T21:33:08.054 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:29:53.287278+0000", 2024-03-11T21:33:08.054 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.0028878670000000001 2024-03-11T21:33:08.054 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:33:08.054 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:33:08.054 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "sub_op_commit_rec", 2024-03-11T21:33:08.054 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:29:53.298314+0000", 2024-03-11T21:33:08.055 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.011035794 2024-03-11T21:33:08.055 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:33:08.055 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:33:08.055 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "sub_op_commit_rec", 2024-03-11T21:33:08.055 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:29:53.420307+0000", 2024-03-11T21:33:08.055 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.121992795 2024-03-11T21:33:08.055 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:33:08.055 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:33:08.055 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "commit_sent", 2024-03-11T21:33:08.055 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:29:53.420349+0000", 2024-03-11T21:33:08.055 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 4.2508e-05 2024-03-11T21:33:08.055 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:33:08.055 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:33:08.055 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "done", 2024-03-11T21:33:08.055 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:29:53.420389+0000", 2024-03-11T21:33:08.056 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 3.9947000000000002e-05 2024-03-11T21:33:08.056 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-03-11T21:33:08.056 INFO:teuthology.orchestra.run.smithi097.stdout: ] 2024-03-11T21:33:08.056 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-03-11T21:33:08.056 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:33:08.056 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:33:08.056 INFO:teuthology.orchestra.run.smithi097.stdout: "description": "osd_op(client.34715.0:156176 7.7 7:eabb1b87:::benchmark_data_smithi092_76040_object34265:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e126)", 2024-03-11T21:33:08.056 INFO:teuthology.orchestra.run.smithi097.stdout: "initiated_at": "2024-03-11T21:29:56.529603+0000", 2024-03-11T21:33:08.056 INFO:teuthology.orchestra.run.smithi097.stdout: "age": 191.49143292700001, 2024-03-11T21:33:08.056 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.157669753, 2024-03-11T21:33:08.056 INFO:teuthology.orchestra.run.smithi097.stdout: "type_data": { 2024-03-11T21:33:08.056 INFO:teuthology.orchestra.run.smithi097.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-11T21:33:08.056 INFO:teuthology.orchestra.run.smithi097.stdout: "client_info": { 2024-03-11T21:33:08.056 INFO:teuthology.orchestra.run.smithi097.stdout: "client": "client.34715", 2024-03-11T21:33:08.056 INFO:teuthology.orchestra.run.smithi097.stdout: "client_addr": "v1:172.21.15.92:0/2403132573", 2024-03-11T21:33:08.057 INFO:teuthology.orchestra.run.smithi097.stdout: "tid": 156176 2024-03-11T21:33:08.057 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:33:08.057 INFO:teuthology.orchestra.run.smithi097.stdout: "events": [ 2024-03-11T21:33:08.057 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:33:08.057 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "initiated", 2024-03-11T21:33:08.057 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:29:56.529603+0000", 2024-03-11T21:33:08.057 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-03-11T21:33:08.057 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:33:08.057 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:33:08.057 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "header_read", 2024-03-11T21:33:08.057 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:29:56.529603+0000", 2024-03-11T21:33:08.057 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-03-11T21:33:08.057 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:33:08.057 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:33:08.058 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "throttled", 2024-03-11T21:33:08.058 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:29:56.529604+0000", 2024-03-11T21:33:08.058 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.57e-06 2024-03-11T21:33:08.058 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:33:08.058 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:33:08.058 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "all_read", 2024-03-11T21:33:08.058 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:29:56.529609+0000", 2024-03-11T21:33:08.058 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 4.561e-06 2024-03-11T21:33:08.058 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:33:08.058 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:33:08.058 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "dispatched", 2024-03-11T21:33:08.058 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:29:56.529619+0000", 2024-03-11T21:33:08.058 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 9.5880000000000004e-06 2024-03-11T21:33:08.058 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:33:08.058 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:33:08.059 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "queued_for_pg", 2024-03-11T21:33:08.059 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:29:56.529626+0000", 2024-03-11T21:33:08.059 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 7.2429999999999999e-06 2024-03-11T21:33:08.059 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:33:08.059 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:33:08.059 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "reached_pg", 2024-03-11T21:33:08.059 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:29:56.529687+0000", 2024-03-11T21:33:08.059 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 6.1316999999999996e-05 2024-03-11T21:33:08.059 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:33:08.059 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:33:08.059 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "started", 2024-03-11T21:33:08.059 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:29:56.529851+0000", 2024-03-11T21:33:08.059 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.000163802 2024-03-11T21:33:08.059 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:33:08.060 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:33:08.060 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "waiting for subops from 0,3", 2024-03-11T21:33:08.060 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:29:56.529986+0000", 2024-03-11T21:33:08.060 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.00013493700000000001 2024-03-11T21:33:08.060 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:33:08.060 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:33:08.060 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "op_commit", 2024-03-11T21:33:08.060 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:29:56.533514+0000", 2024-03-11T21:33:08.060 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.0035286390000000001 2024-03-11T21:33:08.060 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:33:08.060 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:33:08.060 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "sub_op_commit_rec", 2024-03-11T21:33:08.060 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:29:56.533820+0000", 2024-03-11T21:33:08.060 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.00030585699999999998 2024-03-11T21:33:08.060 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:33:08.061 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:33:08.061 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "sub_op_commit_rec", 2024-03-11T21:33:08.061 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:29:56.687203+0000", 2024-03-11T21:33:08.061 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.153382565 2024-03-11T21:33:08.062 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:33:08.062 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:33:08.062 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "commit_sent", 2024-03-11T21:33:08.062 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:29:56.687238+0000", 2024-03-11T21:33:08.062 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 3.4650000000000002e-05 2024-03-11T21:33:08.062 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:33:08.062 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:33:08.062 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "done", 2024-03-11T21:33:08.062 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:29:56.687273+0000", 2024-03-11T21:33:08.062 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 3.5024000000000002e-05 2024-03-11T21:33:08.063 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-03-11T21:33:08.063 INFO:teuthology.orchestra.run.smithi097.stdout: ] 2024-03-11T21:33:08.063 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-03-11T21:33:08.063 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:33:08.063 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:33:08.063 INFO:teuthology.orchestra.run.smithi097.stdout: "description": "osd_op(client.34715.0:156470 7.7 7:e13ffbf0:::benchmark_data_smithi092_76040_object34559:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e126)", 2024-03-11T21:33:08.063 INFO:teuthology.orchestra.run.smithi097.stdout: "initiated_at": "2024-03-11T21:29:56.990545+0000", 2024-03-11T21:33:08.063 INFO:teuthology.orchestra.run.smithi097.stdout: "age": 191.03049042699999, 2024-03-11T21:33:08.063 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.12930302799999999, 2024-03-11T21:33:08.063 INFO:teuthology.orchestra.run.smithi097.stdout: "type_data": { 2024-03-11T21:33:08.063 INFO:teuthology.orchestra.run.smithi097.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-11T21:33:08.063 INFO:teuthology.orchestra.run.smithi097.stdout: "client_info": { 2024-03-11T21:33:08.063 INFO:teuthology.orchestra.run.smithi097.stdout: "client": "client.34715", 2024-03-11T21:33:08.063 INFO:teuthology.orchestra.run.smithi097.stdout: "client_addr": "v1:172.21.15.92:0/2403132573", 2024-03-11T21:33:08.064 INFO:teuthology.orchestra.run.smithi097.stdout: "tid": 156470 2024-03-11T21:33:08.064 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:33:08.064 INFO:teuthology.orchestra.run.smithi097.stdout: "events": [ 2024-03-11T21:33:08.064 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:33:08.064 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "initiated", 2024-03-11T21:33:08.064 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:29:56.990545+0000", 2024-03-11T21:33:08.064 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-03-11T21:33:08.064 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:33:08.064 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:33:08.064 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "header_read", 2024-03-11T21:33:08.064 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:29:56.990545+0000", 2024-03-11T21:33:08.064 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-03-11T21:33:08.064 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:33:08.064 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:33:08.065 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "throttled", 2024-03-11T21:33:08.065 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:29:56.990547+0000", 2024-03-11T21:33:08.065 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.9240000000000001e-06 2024-03-11T21:33:08.065 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:33:08.065 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:33:08.065 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "all_read", 2024-03-11T21:33:08.065 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:29:56.990553+0000", 2024-03-11T21:33:08.065 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 5.3199999999999999e-06 2024-03-11T21:33:08.065 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:33:08.065 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:33:08.065 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "dispatched", 2024-03-11T21:33:08.066 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:29:56.990565+0000", 2024-03-11T21:33:08.066 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.2055999999999999e-05 2024-03-11T21:33:08.066 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:33:08.066 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:33:08.066 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "queued_for_pg", 2024-03-11T21:33:08.066 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:29:56.990572+0000", 2024-03-11T21:33:08.066 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 7.8669999999999994e-06 2024-03-11T21:33:08.066 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:33:08.066 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:33:08.066 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "reached_pg", 2024-03-11T21:33:08.066 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:29:56.990618+0000", 2024-03-11T21:33:08.066 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 4.5118999999999999e-05 2024-03-11T21:33:08.066 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:33:08.067 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:33:08.067 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "started", 2024-03-11T21:33:08.067 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:29:56.990769+0000", 2024-03-11T21:33:08.067 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.00015170200000000001 2024-03-11T21:33:08.067 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:33:08.067 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:33:08.067 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "waiting for subops from 0,3", 2024-03-11T21:33:08.067 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:29:56.990885+0000", 2024-03-11T21:33:08.067 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.00011616399999999999 2024-03-11T21:33:08.067 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:33:08.067 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:33:08.067 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "op_commit", 2024-03-11T21:33:08.067 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:29:56.993283+0000", 2024-03-11T21:33:08.067 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.0023971019999999999 2024-03-11T21:33:08.067 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:33:08.068 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:33:08.068 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "sub_op_commit_rec", 2024-03-11T21:33:08.068 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:29:57.043680+0000", 2024-03-11T21:33:08.068 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.050397191000000001 2024-03-11T21:33:08.068 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:33:08.068 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:33:08.068 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "sub_op_commit_rec", 2024-03-11T21:33:08.068 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:29:57.119764+0000", 2024-03-11T21:33:08.068 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.076084497000000001 2024-03-11T21:33:08.068 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:33:08.068 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:33:08.068 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "commit_sent", 2024-03-11T21:33:08.068 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:29:57.119812+0000", 2024-03-11T21:33:08.068 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 4.7991999999999997e-05 2024-03-11T21:33:08.069 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:33:08.069 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:33:08.069 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "done", 2024-03-11T21:33:08.069 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:29:57.119848+0000", 2024-03-11T21:33:08.069 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 3.6093999999999997e-05 2024-03-11T21:33:08.069 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-03-11T21:33:08.069 INFO:teuthology.orchestra.run.smithi097.stdout: ] 2024-03-11T21:33:08.069 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-03-11T21:33:08.069 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:33:08.069 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:33:08.069 INFO:teuthology.orchestra.run.smithi097.stdout: "description": "osd_op(client.34715.0:156595 7.7 7:efb16ea0:::benchmark_data_smithi092_76040_object34684:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e126)", 2024-03-11T21:33:08.069 INFO:teuthology.orchestra.run.smithi097.stdout: "initiated_at": "2024-03-11T21:29:57.229266+0000", 2024-03-11T21:33:08.069 INFO:teuthology.orchestra.run.smithi097.stdout: "age": 190.79176964000001, 2024-03-11T21:33:08.069 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.138283932, 2024-03-11T21:33:08.070 INFO:teuthology.orchestra.run.smithi097.stdout: "type_data": { 2024-03-11T21:33:08.070 INFO:teuthology.orchestra.run.smithi097.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-11T21:33:08.070 INFO:teuthology.orchestra.run.smithi097.stdout: "client_info": { 2024-03-11T21:33:08.070 INFO:teuthology.orchestra.run.smithi097.stdout: "client": "client.34715", 2024-03-11T21:33:08.070 INFO:teuthology.orchestra.run.smithi097.stdout: "client_addr": "v1:172.21.15.92:0/2403132573", 2024-03-11T21:33:08.070 INFO:teuthology.orchestra.run.smithi097.stdout: "tid": 156595 2024-03-11T21:33:08.070 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:33:08.070 INFO:teuthology.orchestra.run.smithi097.stdout: "events": [ 2024-03-11T21:33:08.070 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:33:08.070 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "initiated", 2024-03-11T21:33:08.070 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:29:57.229266+0000", 2024-03-11T21:33:08.070 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-03-11T21:33:08.070 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:33:08.070 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:33:08.070 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "header_read", 2024-03-11T21:33:08.071 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:29:57.229266+0000", 2024-03-11T21:33:08.071 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-03-11T21:33:08.071 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:33:08.071 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:33:08.071 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "throttled", 2024-03-11T21:33:08.071 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:29:57.229267+0000", 2024-03-11T21:33:08.071 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 9.0599999999999999e-07 2024-03-11T21:33:08.071 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:33:08.071 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:33:08.071 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "all_read", 2024-03-11T21:33:08.071 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:29:57.229271+0000", 2024-03-11T21:33:08.071 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 3.6720000000000002e-06 2024-03-11T21:33:08.071 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:33:08.071 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:33:08.072 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "dispatched", 2024-03-11T21:33:08.072 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:29:57.229280+0000", 2024-03-11T21:33:08.072 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 9.1200000000000008e-06 2024-03-11T21:33:08.072 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:33:08.072 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:33:08.072 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "queued_for_pg", 2024-03-11T21:33:08.072 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:29:57.229287+0000", 2024-03-11T21:33:08.072 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 7.7200000000000006e-06 2024-03-11T21:33:08.072 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:33:08.072 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:33:08.072 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "reached_pg", 2024-03-11T21:33:08.072 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:29:57.229331+0000", 2024-03-11T21:33:08.072 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 4.3940999999999998e-05 2024-03-11T21:33:08.072 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:33:08.072 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:33:08.073 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "started", 2024-03-11T21:33:08.073 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:29:57.229410+0000", 2024-03-11T21:33:08.073 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 7.8553000000000005e-05 2024-03-11T21:33:08.073 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:33:08.073 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:33:08.073 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "waiting for subops from 0,3", 2024-03-11T21:33:08.073 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:29:57.229481+0000", 2024-03-11T21:33:08.073 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 7.0650999999999998e-05 2024-03-11T21:33:08.073 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:33:08.073 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:33:08.073 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "op_commit", 2024-03-11T21:33:08.073 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:29:57.231663+0000", 2024-03-11T21:33:08.073 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.002181887 2024-03-11T21:33:08.073 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:33:08.073 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:33:08.074 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "sub_op_commit_rec", 2024-03-11T21:33:08.074 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:29:57.253437+0000", 2024-03-11T21:33:08.074 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.021774216999999998 2024-03-11T21:33:08.074 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:33:08.074 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:33:08.074 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "sub_op_commit_rec", 2024-03-11T21:33:08.074 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:29:57.367448+0000", 2024-03-11T21:33:08.074 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.114011184 2024-03-11T21:33:08.074 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:33:08.074 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:33:08.074 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "commit_sent", 2024-03-11T21:33:08.074 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:29:57.367505+0000", 2024-03-11T21:33:08.074 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 5.6657999999999999e-05 2024-03-11T21:33:08.074 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:33:08.075 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:33:08.075 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "done", 2024-03-11T21:33:08.075 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:29:57.367550+0000", 2024-03-11T21:33:08.075 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 4.5423000000000002e-05 2024-03-11T21:33:08.075 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-03-11T21:33:08.075 INFO:teuthology.orchestra.run.smithi097.stdout: ] 2024-03-11T21:33:08.075 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-03-11T21:33:08.075 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:33:08.075 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:33:08.075 INFO:teuthology.orchestra.run.smithi097.stdout: "description": "osd_op(client.34715.0:156683 7.7 7:ec65b82b:::benchmark_data_smithi092_76040_object34772:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e126)", 2024-03-11T21:33:08.075 INFO:teuthology.orchestra.run.smithi097.stdout: "initiated_at": "2024-03-11T21:29:57.397512+0000", 2024-03-11T21:33:08.076 INFO:teuthology.orchestra.run.smithi097.stdout: "age": 190.623523904, 2024-03-11T21:33:08.076 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.15765780400000001, 2024-03-11T21:33:08.076 INFO:teuthology.orchestra.run.smithi097.stdout: "type_data": { 2024-03-11T21:33:08.076 INFO:teuthology.orchestra.run.smithi097.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-11T21:33:08.076 INFO:teuthology.orchestra.run.smithi097.stdout: "client_info": { 2024-03-11T21:33:08.076 INFO:teuthology.orchestra.run.smithi097.stdout: "client": "client.34715", 2024-03-11T21:33:08.076 INFO:teuthology.orchestra.run.smithi097.stdout: "client_addr": "v1:172.21.15.92:0/2403132573", 2024-03-11T21:33:08.076 INFO:teuthology.orchestra.run.smithi097.stdout: "tid": 156683 2024-03-11T21:33:08.076 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:33:08.076 INFO:teuthology.orchestra.run.smithi097.stdout: "events": [ 2024-03-11T21:33:08.076 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:33:08.076 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "initiated", 2024-03-11T21:33:08.076 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:29:57.397512+0000", 2024-03-11T21:33:08.076 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-03-11T21:33:08.077 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:33:08.077 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:33:08.077 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "header_read", 2024-03-11T21:33:08.077 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:29:57.397512+0000", 2024-03-11T21:33:08.077 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-03-11T21:33:08.077 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:33:08.077 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:33:08.077 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "throttled", 2024-03-11T21:33:08.077 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:29:57.397513+0000", 2024-03-11T21:33:08.077 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 9.7600000000000006e-07 2024-03-11T21:33:08.077 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:33:08.077 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:33:08.077 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "all_read", 2024-03-11T21:33:08.077 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:29:57.397516+0000", 2024-03-11T21:33:08.078 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 3.608e-06 2024-03-11T21:33:08.078 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:33:08.078 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:33:08.078 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "dispatched", 2024-03-11T21:33:08.078 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:29:57.397525+0000", 2024-03-11T21:33:08.078 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 8.3459999999999994e-06 2024-03-11T21:33:08.078 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:33:08.078 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:33:08.078 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "queued_for_pg", 2024-03-11T21:33:08.078 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:29:57.397530+0000", 2024-03-11T21:33:08.078 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 5.164e-06 2024-03-11T21:33:08.078 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:33:08.078 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:33:08.078 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "reached_pg", 2024-03-11T21:33:08.079 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:29:57.401870+0000", 2024-03-11T21:33:08.079 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.004340259 2024-03-11T21:33:08.079 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:33:08.079 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:33:08.079 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "started", 2024-03-11T21:33:08.079 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:29:57.401987+0000", 2024-03-11T21:33:08.079 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.000116589 2024-03-11T21:33:08.079 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:33:08.079 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:33:08.079 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "waiting for subops from 0,3", 2024-03-11T21:33:08.079 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:29:57.402100+0000", 2024-03-11T21:33:08.079 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.000113542 2024-03-11T21:33:08.079 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:33:08.079 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:33:08.080 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "op_commit", 2024-03-11T21:33:08.080 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:29:57.405258+0000", 2024-03-11T21:33:08.080 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.0031580279999999998 2024-03-11T21:33:08.080 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:33:08.080 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:33:08.080 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "sub_op_commit_rec", 2024-03-11T21:33:08.080 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:29:57.429205+0000", 2024-03-11T21:33:08.080 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.023946505 2024-03-11T21:33:08.081 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:33:08.081 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:33:08.081 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "sub_op_commit_rec", 2024-03-11T21:33:08.081 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:29:57.555099+0000", 2024-03-11T21:33:08.081 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.12589394600000001 2024-03-11T21:33:08.081 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:33:08.081 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:33:08.081 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "commit_sent", 2024-03-11T21:33:08.081 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:29:57.555136+0000", 2024-03-11T21:33:08.081 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 3.7712000000000001e-05 2024-03-11T21:33:08.081 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:33:08.081 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:33:08.081 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "done", 2024-03-11T21:33:08.082 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:29:57.555170+0000", 2024-03-11T21:33:08.082 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 3.3129000000000001e-05 2024-03-11T21:33:08.082 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-03-11T21:33:08.082 INFO:teuthology.orchestra.run.smithi097.stdout: ] 2024-03-11T21:33:08.082 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-03-11T21:33:08.082 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:33:08.082 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:33:08.082 INFO:teuthology.orchestra.run.smithi097.stdout: "description": "osd_op(client.34715.0:156720 7.7 7:e8c159b2:::benchmark_data_smithi092_76040_object34809:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e126)", 2024-03-11T21:33:08.082 INFO:teuthology.orchestra.run.smithi097.stdout: "initiated_at": "2024-03-11T21:29:57.455164+0000", 2024-03-11T21:33:08.082 INFO:teuthology.orchestra.run.smithi097.stdout: "age": 190.565871542, 2024-03-11T21:33:08.083 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.13071795999999999, 2024-03-11T21:33:08.083 INFO:teuthology.orchestra.run.smithi097.stdout: "type_data": { 2024-03-11T21:33:08.083 INFO:teuthology.orchestra.run.smithi097.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-11T21:33:08.083 INFO:teuthology.orchestra.run.smithi097.stdout: "client_info": { 2024-03-11T21:33:08.083 INFO:teuthology.orchestra.run.smithi097.stdout: "client": "client.34715", 2024-03-11T21:33:08.083 INFO:teuthology.orchestra.run.smithi097.stdout: "client_addr": "v1:172.21.15.92:0/2403132573", 2024-03-11T21:33:08.083 INFO:teuthology.orchestra.run.smithi097.stdout: "tid": 156720 2024-03-11T21:33:08.083 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:33:08.083 INFO:teuthology.orchestra.run.smithi097.stdout: "events": [ 2024-03-11T21:33:08.083 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:33:08.083 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "initiated", 2024-03-11T21:33:08.083 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:29:57.455164+0000", 2024-03-11T21:33:08.083 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-03-11T21:33:08.083 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:33:08.083 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:33:08.084 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "header_read", 2024-03-11T21:33:08.084 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:29:57.455164+0000", 2024-03-11T21:33:08.084 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-03-11T21:33:08.084 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:33:08.084 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:33:08.084 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "throttled", 2024-03-11T21:33:08.084 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:29:57.455165+0000", 2024-03-11T21:33:08.084 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.1200000000000001e-06 2024-03-11T21:33:08.084 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:33:08.084 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:33:08.084 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "all_read", 2024-03-11T21:33:08.084 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:29:57.455170+0000", 2024-03-11T21:33:08.084 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 4.4299999999999999e-06 2024-03-11T21:33:08.084 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:33:08.085 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:33:08.085 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "dispatched", 2024-03-11T21:33:08.085 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:29:57.455178+0000", 2024-03-11T21:33:08.085 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 8.5350000000000004e-06 2024-03-11T21:33:08.085 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:33:08.085 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:33:08.085 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "queued_for_pg", 2024-03-11T21:33:08.085 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:29:57.455185+0000", 2024-03-11T21:33:08.085 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 6.6579999999999996e-06 2024-03-11T21:33:08.085 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:33:08.085 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:33:08.085 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "reached_pg", 2024-03-11T21:33:08.085 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:29:57.455237+0000", 2024-03-11T21:33:08.085 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 5.1678000000000002e-05 2024-03-11T21:33:08.085 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:33:08.086 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:33:08.086 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "started", 2024-03-11T21:33:08.086 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:29:57.455332+0000", 2024-03-11T21:33:08.086 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 9.5357000000000002e-05 2024-03-11T21:33:08.086 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:33:08.086 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:33:08.086 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "waiting for subops from 0,3", 2024-03-11T21:33:08.086 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:29:57.455458+0000", 2024-03-11T21:33:08.086 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.00012579299999999999 2024-03-11T21:33:08.086 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:33:08.086 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:33:08.086 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "op_commit", 2024-03-11T21:33:08.086 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:29:57.458563+0000", 2024-03-11T21:33:08.086 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.0031057039999999999 2024-03-11T21:33:08.087 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:33:08.087 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:33:08.087 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "sub_op_commit_rec", 2024-03-11T21:33:08.087 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:29:57.492724+0000", 2024-03-11T21:33:08.087 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.034160596000000001 2024-03-11T21:33:08.087 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:33:08.087 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:33:08.087 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "sub_op_commit_rec", 2024-03-11T21:33:08.087 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:29:57.585777+0000", 2024-03-11T21:33:08.087 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.093052977999999995 2024-03-11T21:33:08.087 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:33:08.088 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:33:08.088 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "commit_sent", 2024-03-11T21:33:08.088 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:29:57.585832+0000", 2024-03-11T21:33:08.088 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 5.5331999999999998e-05 2024-03-11T21:33:08.088 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:33:08.088 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:33:08.088 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "done", 2024-03-11T21:33:08.088 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:29:57.585882+0000", 2024-03-11T21:33:08.088 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 4.9778999999999999e-05 2024-03-11T21:33:08.088 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-03-11T21:33:08.088 INFO:teuthology.orchestra.run.smithi097.stdout: ] 2024-03-11T21:33:08.088 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-03-11T21:33:08.088 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:33:08.088 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:33:08.089 INFO:teuthology.orchestra.run.smithi097.stdout: "description": "osd_op(client.34715.0:156910 7.7 7:ed8c3ad9:::benchmark_data_smithi092_76040_object34999:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e126)", 2024-03-11T21:33:08.089 INFO:teuthology.orchestra.run.smithi097.stdout: "initiated_at": "2024-03-11T21:29:57.825647+0000", 2024-03-11T21:33:08.089 INFO:teuthology.orchestra.run.smithi097.stdout: "age": 190.195388767, 2024-03-11T21:33:08.089 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.13931633800000001, 2024-03-11T21:33:08.089 INFO:teuthology.orchestra.run.smithi097.stdout: "type_data": { 2024-03-11T21:33:08.089 INFO:teuthology.orchestra.run.smithi097.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-11T21:33:08.089 INFO:teuthology.orchestra.run.smithi097.stdout: "client_info": { 2024-03-11T21:33:08.089 INFO:teuthology.orchestra.run.smithi097.stdout: "client": "client.34715", 2024-03-11T21:33:08.089 INFO:teuthology.orchestra.run.smithi097.stdout: "client_addr": "v1:172.21.15.92:0/2403132573", 2024-03-11T21:33:08.089 INFO:teuthology.orchestra.run.smithi097.stdout: "tid": 156910 2024-03-11T21:33:08.089 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:33:08.089 INFO:teuthology.orchestra.run.smithi097.stdout: "events": [ 2024-03-11T21:33:08.089 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:33:08.089 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "initiated", 2024-03-11T21:33:08.090 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:29:57.825647+0000", 2024-03-11T21:33:08.090 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-03-11T21:33:08.090 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:33:08.090 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:33:08.090 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "header_read", 2024-03-11T21:33:08.090 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:29:57.825647+0000", 2024-03-11T21:33:08.090 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-03-11T21:33:08.090 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:33:08.090 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:33:08.090 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "throttled", 2024-03-11T21:33:08.090 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:29:57.825649+0000", 2024-03-11T21:33:08.090 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 2.0719999999999998e-06 2024-03-11T21:33:08.090 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:33:08.090 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:33:08.091 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "all_read", 2024-03-11T21:33:08.091 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:29:57.825655+0000", 2024-03-11T21:33:08.091 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 6.2140000000000001e-06 2024-03-11T21:33:08.091 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:33:08.091 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:33:08.091 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "dispatched", 2024-03-11T21:33:08.091 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:29:57.825669+0000", 2024-03-11T21:33:08.091 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.3509999999999999e-05 2024-03-11T21:33:08.091 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:33:08.091 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:33:08.091 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "queued_for_pg", 2024-03-11T21:33:08.091 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:29:57.825676+0000", 2024-03-11T21:33:08.091 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 6.8460000000000002e-06 2024-03-11T21:33:08.091 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:33:08.092 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:33:08.092 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "reached_pg", 2024-03-11T21:33:08.092 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:29:57.825730+0000", 2024-03-11T21:33:08.092 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 5.4211e-05 2024-03-11T21:33:08.092 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:33:08.092 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:33:08.092 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "started", 2024-03-11T21:33:08.092 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:29:57.825869+0000", 2024-03-11T21:33:08.092 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.000139574 2024-03-11T21:33:08.092 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:33:08.092 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:33:08.092 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "waiting for subops from 0,3", 2024-03-11T21:33:08.092 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:29:57.825987+0000", 2024-03-11T21:33:08.092 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.00011783299999999999 2024-03-11T21:33:08.092 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:33:08.093 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:33:08.093 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "op_commit", 2024-03-11T21:33:08.093 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:29:57.829193+0000", 2024-03-11T21:33:08.093 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.0032057159999999999 2024-03-11T21:33:08.093 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:33:08.093 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:33:08.093 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "sub_op_commit_rec", 2024-03-11T21:33:08.093 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:29:57.862874+0000", 2024-03-11T21:33:08.093 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.033680871000000001 2024-03-11T21:33:08.093 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:33:08.093 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:33:08.093 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "sub_op_commit_rec", 2024-03-11T21:33:08.093 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:29:57.964899+0000", 2024-03-11T21:33:08.093 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.102024935 2024-03-11T21:33:08.094 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:33:08.094 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:33:08.094 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "commit_sent", 2024-03-11T21:33:08.094 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:29:57.964930+0000", 2024-03-11T21:33:08.094 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 3.1390999999999998e-05 2024-03-11T21:33:08.094 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:33:08.094 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:33:08.094 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "done", 2024-03-11T21:33:08.094 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:29:57.964963+0000", 2024-03-11T21:33:08.094 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 3.3164999999999998e-05 2024-03-11T21:33:08.094 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-03-11T21:33:08.094 INFO:teuthology.orchestra.run.smithi097.stdout: ] 2024-03-11T21:33:08.094 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-03-11T21:33:08.094 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:33:08.095 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:33:08.095 INFO:teuthology.orchestra.run.smithi097.stdout: "description": "osd_op(client.34715.0:157118 7.7 7:ef798aee:::benchmark_data_smithi092_76040_object35207:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e127)", 2024-03-11T21:33:08.095 INFO:teuthology.orchestra.run.smithi097.stdout: "initiated_at": "2024-03-11T21:29:58.129269+0000", 2024-03-11T21:33:08.095 INFO:teuthology.orchestra.run.smithi097.stdout: "age": 189.89176624199999, 2024-03-11T21:33:08.095 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.18586082100000001, 2024-03-11T21:33:08.095 INFO:teuthology.orchestra.run.smithi097.stdout: "type_data": { 2024-03-11T21:33:08.095 INFO:teuthology.orchestra.run.smithi097.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-11T21:33:08.095 INFO:teuthology.orchestra.run.smithi097.stdout: "client_info": { 2024-03-11T21:33:08.095 INFO:teuthology.orchestra.run.smithi097.stdout: "client": "client.34715", 2024-03-11T21:33:08.095 INFO:teuthology.orchestra.run.smithi097.stdout: "client_addr": "v1:172.21.15.92:0/2403132573", 2024-03-11T21:33:08.095 INFO:teuthology.orchestra.run.smithi097.stdout: "tid": 157118 2024-03-11T21:33:08.095 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:33:08.095 INFO:teuthology.orchestra.run.smithi097.stdout: "events": [ 2024-03-11T21:33:08.095 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:33:08.096 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "initiated", 2024-03-11T21:33:08.096 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:29:58.129269+0000", 2024-03-11T21:33:08.096 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-03-11T21:33:08.096 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:33:08.096 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:33:08.096 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "header_read", 2024-03-11T21:33:08.096 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:29:58.129269+0000", 2024-03-11T21:33:08.096 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-03-11T21:33:08.096 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:33:08.096 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:33:08.096 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "throttled", 2024-03-11T21:33:08.096 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:29:58.129271+0000", 2024-03-11T21:33:08.096 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.7519999999999999e-06 2024-03-11T21:33:08.096 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:33:08.096 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:33:08.097 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "all_read", 2024-03-11T21:33:08.097 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:29:58.129276+0000", 2024-03-11T21:33:08.097 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 5.1279999999999999e-06 2024-03-11T21:33:08.097 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:33:08.097 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:33:08.097 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "dispatched", 2024-03-11T21:33:08.097 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:29:58.129286+0000", 2024-03-11T21:33:08.097 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.0062e-05 2024-03-11T21:33:08.097 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:33:08.097 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:33:08.097 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "queued_for_pg", 2024-03-11T21:33:08.097 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:29:58.129292+0000", 2024-03-11T21:33:08.097 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 5.8769999999999998e-06 2024-03-11T21:33:08.097 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:33:08.098 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:33:08.098 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "reached_pg", 2024-03-11T21:33:08.098 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:29:58.129334+0000", 2024-03-11T21:33:08.098 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 4.1641000000000003e-05 2024-03-11T21:33:08.098 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:33:08.098 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:33:08.098 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "started", 2024-03-11T21:33:08.098 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:29:58.129426+0000", 2024-03-11T21:33:08.098 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 9.2511999999999998e-05 2024-03-11T21:33:08.098 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:33:08.098 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:33:08.098 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "waiting for subops from 0,3", 2024-03-11T21:33:08.098 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:29:58.129504+0000", 2024-03-11T21:33:08.098 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 7.7631000000000003e-05 2024-03-11T21:33:08.098 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:33:08.099 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:33:08.099 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "op_commit", 2024-03-11T21:33:08.099 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:29:58.132108+0000", 2024-03-11T21:33:08.099 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.0026044060000000001 2024-03-11T21:33:08.099 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:33:08.099 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:33:08.099 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "sub_op_commit_rec", 2024-03-11T21:33:08.099 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:29:58.149431+0000", 2024-03-11T21:33:08.099 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.017322596999999999 2024-03-11T21:33:08.099 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:33:08.099 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:33:08.099 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "sub_op_commit_rec", 2024-03-11T21:33:08.099 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:29:58.315001+0000", 2024-03-11T21:33:08.099 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.16557005799999999 2024-03-11T21:33:08.100 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:33:08.100 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:33:08.100 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "commit_sent", 2024-03-11T21:33:08.100 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:29:58.315060+0000", 2024-03-11T21:33:08.100 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 5.9039999999999997e-05 2024-03-11T21:33:08.100 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:33:08.100 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:33:08.100 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "done", 2024-03-11T21:33:08.100 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:29:58.315130+0000", 2024-03-11T21:33:08.100 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 7.0117000000000006e-05 2024-03-11T21:33:08.100 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-03-11T21:33:08.100 INFO:teuthology.orchestra.run.smithi097.stdout: ] 2024-03-11T21:33:08.100 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-03-11T21:33:08.100 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:33:08.100 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:33:08.101 INFO:teuthology.orchestra.run.smithi097.stdout: "description": "osd_op(client.34715.0:157129 7.7 7:e1f3a7b2:::benchmark_data_smithi092_76040_object35218:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e127)", 2024-03-11T21:33:08.101 INFO:teuthology.orchestra.run.smithi097.stdout: "initiated_at": "2024-03-11T21:29:58.156664+0000", 2024-03-11T21:33:08.101 INFO:teuthology.orchestra.run.smithi097.stdout: "age": 189.86437197699999, 2024-03-11T21:33:08.101 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.15858186099999999, 2024-03-11T21:33:08.101 INFO:teuthology.orchestra.run.smithi097.stdout: "type_data": { 2024-03-11T21:33:08.101 INFO:teuthology.orchestra.run.smithi097.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-11T21:33:08.101 INFO:teuthology.orchestra.run.smithi097.stdout: "client_info": { 2024-03-11T21:33:08.101 INFO:teuthology.orchestra.run.smithi097.stdout: "client": "client.34715", 2024-03-11T21:33:08.101 INFO:teuthology.orchestra.run.smithi097.stdout: "client_addr": "v1:172.21.15.92:0/2403132573", 2024-03-11T21:33:08.101 INFO:teuthology.orchestra.run.smithi097.stdout: "tid": 157129 2024-03-11T21:33:08.101 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:33:08.101 INFO:teuthology.orchestra.run.smithi097.stdout: "events": [ 2024-03-11T21:33:08.101 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:33:08.101 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "initiated", 2024-03-11T21:33:08.101 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:29:58.156664+0000", 2024-03-11T21:33:08.102 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-03-11T21:33:08.102 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:33:08.102 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:33:08.102 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "header_read", 2024-03-11T21:33:08.102 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:29:58.156664+0000", 2024-03-11T21:33:08.102 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-03-11T21:33:08.102 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:33:08.102 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:33:08.102 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "throttled", 2024-03-11T21:33:08.102 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:29:58.156666+0000", 2024-03-11T21:33:08.102 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 2.0640000000000001e-06 2024-03-11T21:33:08.102 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:33:08.102 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:33:08.102 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "all_read", 2024-03-11T21:33:08.103 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:29:58.156676+0000", 2024-03-11T21:33:08.103 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 9.9960000000000005e-06 2024-03-11T21:33:08.103 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:33:08.103 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:33:08.103 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "dispatched", 2024-03-11T21:33:08.103 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:29:58.156689+0000", 2024-03-11T21:33:08.103 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.3631e-05 2024-03-11T21:33:08.103 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:33:08.103 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:33:08.103 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "queued_for_pg", 2024-03-11T21:33:08.103 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:29:58.156698+0000", 2024-03-11T21:33:08.103 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 8.9290000000000004e-06 2024-03-11T21:33:08.103 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:33:08.103 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:33:08.104 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "reached_pg", 2024-03-11T21:33:08.104 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:29:58.156752+0000", 2024-03-11T21:33:08.104 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 5.3560000000000002e-05 2024-03-11T21:33:08.104 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:33:08.104 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:33:08.104 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "started", 2024-03-11T21:33:08.104 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:29:58.156844+0000", 2024-03-11T21:33:08.104 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 9.2029999999999998e-05 2024-03-11T21:33:08.104 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:33:08.104 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:33:08.104 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "waiting for subops from 0,3", 2024-03-11T21:33:08.104 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:29:58.156922+0000", 2024-03-11T21:33:08.104 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 7.7653999999999999e-05 2024-03-11T21:33:08.104 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:33:08.105 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:33:08.105 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "op_commit", 2024-03-11T21:33:08.105 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:29:58.159472+0000", 2024-03-11T21:33:08.105 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.0025506600000000002 2024-03-11T21:33:08.105 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:33:08.105 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:33:08.105 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "sub_op_commit_rec", 2024-03-11T21:33:08.105 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:29:58.177848+0000", 2024-03-11T21:33:08.105 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.01837538 2024-03-11T21:33:08.105 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:33:08.105 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:33:08.105 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "sub_op_commit_rec", 2024-03-11T21:33:08.105 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:29:58.315184+0000", 2024-03-11T21:33:08.105 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.13733662199999999 2024-03-11T21:33:08.105 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:33:08.106 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:33:08.106 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "commit_sent", 2024-03-11T21:33:08.106 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:29:58.315220+0000", 2024-03-11T21:33:08.106 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 3.5357e-05 2024-03-11T21:33:08.106 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:33:08.106 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:33:08.106 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "done", 2024-03-11T21:33:08.106 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:29:58.315246+0000", 2024-03-11T21:33:08.106 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 2.5978000000000001e-05 2024-03-11T21:33:08.106 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-03-11T21:33:08.106 INFO:teuthology.orchestra.run.smithi097.stdout: ] 2024-03-11T21:33:08.106 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-03-11T21:33:08.106 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:33:08.106 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:33:08.107 INFO:teuthology.orchestra.run.smithi097.stdout: "description": "osd_op(client.15150.0:49197 8.5 8:a049e1ac:::benchmark_data_smithi092_104865_object49196:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e169)", 2024-03-11T21:33:08.107 INFO:teuthology.orchestra.run.smithi097.stdout: "initiated_at": "2024-03-11T21:32:25.332392+0000", 2024-03-11T21:33:08.107 INFO:teuthology.orchestra.run.smithi097.stdout: "age": 42.688643513000002, 2024-03-11T21:33:08.107 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.139426087, 2024-03-11T21:33:08.107 INFO:teuthology.orchestra.run.smithi097.stdout: "type_data": { 2024-03-11T21:33:08.107 INFO:teuthology.orchestra.run.smithi097.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-11T21:33:08.107 INFO:teuthology.orchestra.run.smithi097.stdout: "client_info": { 2024-03-11T21:33:08.107 INFO:teuthology.orchestra.run.smithi097.stdout: "client": "client.15150", 2024-03-11T21:33:08.107 INFO:teuthology.orchestra.run.smithi097.stdout: "client_addr": "v1:172.21.15.92:0/436606717", 2024-03-11T21:33:08.107 INFO:teuthology.orchestra.run.smithi097.stdout: "tid": 49197 2024-03-11T21:33:08.107 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:33:08.107 INFO:teuthology.orchestra.run.smithi097.stdout: "events": [ 2024-03-11T21:33:08.107 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:33:08.107 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "initiated", 2024-03-11T21:33:08.108 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:32:25.332392+0000", 2024-03-11T21:33:08.108 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-03-11T21:33:08.108 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:33:08.108 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:33:08.108 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "header_read", 2024-03-11T21:33:08.108 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:32:25.332392+0000", 2024-03-11T21:33:08.109 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-03-11T21:33:08.109 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:33:08.109 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:33:08.109 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "throttled", 2024-03-11T21:33:08.109 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:32:25.332393+0000", 2024-03-11T21:33:08.109 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.22e-06 2024-03-11T21:33:08.109 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:33:08.109 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:33:08.109 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "all_read", 2024-03-11T21:33:08.109 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:32:25.332452+0000", 2024-03-11T21:33:08.109 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 5.8075000000000002e-05 2024-03-11T21:33:08.109 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:33:08.109 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:33:08.110 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "dispatched", 2024-03-11T21:33:08.110 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:32:25.332468+0000", 2024-03-11T21:33:08.110 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.6705999999999999e-05 2024-03-11T21:33:08.110 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:33:08.110 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:33:08.110 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "queued_for_pg", 2024-03-11T21:33:08.110 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:32:25.332477+0000", 2024-03-11T21:33:08.110 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 9.1390000000000001e-06 2024-03-11T21:33:08.110 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:33:08.110 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:33:08.110 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "reached_pg", 2024-03-11T21:33:08.110 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:32:25.333583+0000", 2024-03-11T21:33:08.110 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.001105948 2024-03-11T21:33:08.110 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:33:08.111 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:33:08.111 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "started", 2024-03-11T21:33:08.111 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:32:25.333721+0000", 2024-03-11T21:33:08.111 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.000137226 2024-03-11T21:33:08.111 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:33:08.111 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:33:08.111 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "waiting for subops from 0,4", 2024-03-11T21:33:08.111 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:32:25.333874+0000", 2024-03-11T21:33:08.111 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.00015352700000000001 2024-03-11T21:33:08.111 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:33:08.111 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:33:08.112 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "op_commit", 2024-03-11T21:33:08.112 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:32:25.337784+0000", 2024-03-11T21:33:08.112 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.0039102470000000004 2024-03-11T21:33:08.112 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:33:08.112 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:33:08.112 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "sub_op_commit_rec", 2024-03-11T21:33:08.112 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:32:25.337996+0000", 2024-03-11T21:33:08.112 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.000211494 2024-03-11T21:33:08.112 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:33:08.112 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:33:08.112 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "sub_op_commit_rec", 2024-03-11T21:33:08.112 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:32:25.471708+0000", 2024-03-11T21:33:08.112 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.13371229500000001 2024-03-11T21:33:08.112 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:33:08.112 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:33:08.113 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "commit_sent", 2024-03-11T21:33:08.113 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:32:25.471770+0000", 2024-03-11T21:33:08.113 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 6.1469000000000004e-05 2024-03-11T21:33:08.113 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:33:08.113 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:33:08.113 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "done", 2024-03-11T21:33:08.113 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:32:25.471818+0000", 2024-03-11T21:33:08.113 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 4.8741e-05 2024-03-11T21:33:08.113 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-03-11T21:33:08.113 INFO:teuthology.orchestra.run.smithi097.stdout: ] 2024-03-11T21:33:08.113 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-03-11T21:33:08.114 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:33:08.114 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:33:08.114 INFO:teuthology.orchestra.run.smithi097.stdout: "description": "osd_op(client.15150.0:88140 8.9 8:90d4c759:::benchmark_data_smithi092_104865_object88139:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e180)", 2024-03-11T21:33:08.114 INFO:teuthology.orchestra.run.smithi097.stdout: "initiated_at": "2024-03-11T21:32:56.043419+0000", 2024-03-11T21:33:08.114 INFO:teuthology.orchestra.run.smithi097.stdout: "age": 11.977616271, 2024-03-11T21:33:08.114 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.16849561299999999, 2024-03-11T21:33:08.114 INFO:teuthology.orchestra.run.smithi097.stdout: "type_data": { 2024-03-11T21:33:08.114 INFO:teuthology.orchestra.run.smithi097.stdout: "flag_point": "reached pg", 2024-03-11T21:33:08.114 INFO:teuthology.orchestra.run.smithi097.stdout: "client_info": { 2024-03-11T21:33:08.114 INFO:teuthology.orchestra.run.smithi097.stdout: "client": "client.15150", 2024-03-11T21:33:08.114 INFO:teuthology.orchestra.run.smithi097.stdout: "client_addr": "v1:172.21.15.92:0/436606717", 2024-03-11T21:33:08.114 INFO:teuthology.orchestra.run.smithi097.stdout: "tid": 88140 2024-03-11T21:33:08.114 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:33:08.114 INFO:teuthology.orchestra.run.smithi097.stdout: "events": [ 2024-03-11T21:33:08.115 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:33:08.115 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "initiated", 2024-03-11T21:33:08.115 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:32:56.043419+0000", 2024-03-11T21:33:08.115 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-03-11T21:33:08.115 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:33:08.115 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:33:08.115 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "header_read", 2024-03-11T21:33:08.115 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:32:56.043419+0000", 2024-03-11T21:33:08.115 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-03-11T21:33:08.115 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:33:08.115 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:33:08.115 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "throttled", 2024-03-11T21:33:08.115 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:32:56.043423+0000", 2024-03-11T21:33:08.115 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 3.1480000000000002e-06 2024-03-11T21:33:08.116 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:33:08.116 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:33:08.116 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "all_read", 2024-03-11T21:33:08.116 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:32:56.043462+0000", 2024-03-11T21:33:08.116 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 3.9475999999999999e-05 2024-03-11T21:33:08.116 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:33:08.116 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:33:08.116 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "dispatched", 2024-03-11T21:33:08.116 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:32:56.043484+0000", 2024-03-11T21:33:08.116 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 2.1735000000000001e-05 2024-03-11T21:33:08.116 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:33:08.116 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:33:08.116 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "queued_for_pg", 2024-03-11T21:33:08.116 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:32:56.043494+0000", 2024-03-11T21:33:08.117 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 9.9089999999999995e-06 2024-03-11T21:33:08.117 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:33:08.117 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:33:08.117 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "reached_pg", 2024-03-11T21:33:08.117 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:32:56.043553+0000", 2024-03-11T21:33:08.117 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 5.9759999999999997e-05 2024-03-11T21:33:08.117 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:33:08.117 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:33:08.117 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "started", 2024-03-11T21:33:08.117 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:32:56.043676+0000", 2024-03-11T21:33:08.117 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.00012296599999999999 2024-03-11T21:33:08.117 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:33:08.117 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:33:08.117 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "waiting for subops from 0,5", 2024-03-11T21:33:08.118 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:33:08.119 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:32:56.043819+0000", 2024-03-11T21:33:08.119 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.00014223799999999999 2024-03-11T21:33:08.119 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:33:08.119 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:33:08.119 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "op_commit", 2024-03-11T21:33:08.119 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:32:56.046675+0000", 2024-03-11T21:33:08.119 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.0028560899999999999 2024-03-11T21:33:08.119 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:33:08.119 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:33:08.119 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "sub_op_commit_rec", 2024-03-11T21:33:08.120 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:32:56.047365+0000", 2024-03-11T21:33:08.120 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.00069037099999999995 2024-03-11T21:33:08.120 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:33:08.120 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:33:08.120 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "reached_pg", 2024-03-11T21:33:08.120 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:32:56.211897+0000", 2024-03-11T21:33:08.120 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.16453135699999999 2024-03-11T21:33:08.120 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:33:08.120 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:33:08.120 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "done", 2024-03-11T21:33:08.120 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:32:56.211915+0000", 2024-03-11T21:33:08.120 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.8563000000000002e-05 2024-03-11T21:33:08.120 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-03-11T21:33:08.120 INFO:teuthology.orchestra.run.smithi097.stdout: ] 2024-03-11T21:33:08.121 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-03-11T21:33:08.121 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:33:08.121 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:33:08.121 INFO:teuthology.orchestra.run.smithi097.stdout: "description": "osd_op(client.15150.0:88146 8.5 8:ab2e3366:::benchmark_data_smithi092_104865_object88145:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e180)", 2024-03-11T21:33:08.121 INFO:teuthology.orchestra.run.smithi097.stdout: "initiated_at": "2024-03-11T21:32:56.058355+0000", 2024-03-11T21:33:08.121 INFO:teuthology.orchestra.run.smithi097.stdout: "age": 11.962680423, 2024-03-11T21:33:08.121 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.15908792899999999, 2024-03-11T21:33:08.121 INFO:teuthology.orchestra.run.smithi097.stdout: "type_data": { 2024-03-11T21:33:08.121 INFO:teuthology.orchestra.run.smithi097.stdout: "flag_point": "reached pg", 2024-03-11T21:33:08.121 INFO:teuthology.orchestra.run.smithi097.stdout: "client_info": { 2024-03-11T21:33:08.121 INFO:teuthology.orchestra.run.smithi097.stdout: "client": "client.15150", 2024-03-11T21:33:08.121 INFO:teuthology.orchestra.run.smithi097.stdout: "client_addr": "v1:172.21.15.92:0/436606717", 2024-03-11T21:33:08.121 INFO:teuthology.orchestra.run.smithi097.stdout: "tid": 88146 2024-03-11T21:33:08.121 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:33:08.121 INFO:teuthology.orchestra.run.smithi097.stdout: "events": [ 2024-03-11T21:33:08.122 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:33:08.122 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "initiated", 2024-03-11T21:33:08.122 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:32:56.058355+0000", 2024-03-11T21:33:08.122 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-03-11T21:33:08.122 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:33:08.122 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:33:08.123 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "header_read", 2024-03-11T21:33:08.123 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:32:56.058355+0000", 2024-03-11T21:33:08.123 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-03-11T21:33:08.124 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:33:08.124 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:33:08.124 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "throttled", 2024-03-11T21:33:08.124 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:32:56.058358+0000", 2024-03-11T21:33:08.124 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 2.9610000000000001e-06 2024-03-11T21:33:08.124 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:33:08.124 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:33:08.124 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "all_read", 2024-03-11T21:33:08.124 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:32:56.058398+0000", 2024-03-11T21:33:08.124 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 4.0126000000000002e-05 2024-03-11T21:33:08.124 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:33:08.124 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:33:08.125 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "dispatched", 2024-03-11T21:33:08.125 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:32:56.058415+0000", 2024-03-11T21:33:08.125 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.6976000000000001e-05 2024-03-11T21:33:08.125 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:33:08.125 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:33:08.125 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "queued_for_pg", 2024-03-11T21:33:08.125 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:32:56.058426+0000", 2024-03-11T21:33:08.125 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.0190999999999999e-05 2024-03-11T21:33:08.125 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:33:08.125 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:33:08.125 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "reached_pg", 2024-03-11T21:33:08.125 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:32:56.058487+0000", 2024-03-11T21:33:08.125 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 6.1852000000000003e-05 2024-03-11T21:33:08.125 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:33:08.126 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:33:08.126 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "started", 2024-03-11T21:33:08.126 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:32:56.058645+0000", 2024-03-11T21:33:08.126 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.000157324 2024-03-11T21:33:08.126 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:33:08.126 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:33:08.126 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "waiting for subops from 0,4", 2024-03-11T21:33:08.127 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:32:56.058813+0000", 2024-03-11T21:33:08.127 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.00016822699999999999 2024-03-11T21:33:08.127 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:33:08.127 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:33:08.127 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "op_commit", 2024-03-11T21:33:08.127 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:32:56.065565+0000", 2024-03-11T21:33:08.127 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.0067525129999999999 2024-03-11T21:33:08.127 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:33:08.127 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:33:08.128 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "reached_pg", 2024-03-11T21:33:08.129 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:32:56.217422+0000", 2024-03-11T21:33:08.129 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.15185670800000001 2024-03-11T21:33:08.129 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:33:08.129 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:33:08.129 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "done", 2024-03-11T21:33:08.129 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:32:56.217443+0000", 2024-03-11T21:33:08.129 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 2.1050999999999999e-05 2024-03-11T21:33:08.129 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-03-11T21:33:08.129 INFO:teuthology.orchestra.run.smithi097.stdout: ] 2024-03-11T21:33:08.129 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-03-11T21:33:08.129 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:33:08.129 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:33:08.129 INFO:teuthology.orchestra.run.smithi097.stdout: "description": "osd_repop(client.15150.0:88148 8.d e181/157 8:b7832715:::benchmark_data_smithi092_104865_object88147:head v 181'5567, mlcod=180'5565)", 2024-03-11T21:33:08.129 INFO:teuthology.orchestra.run.smithi097.stdout: "initiated_at": "2024-03-11T21:32:56.084147+0000", 2024-03-11T21:33:08.130 INFO:teuthology.orchestra.run.smithi097.stdout: "age": 11.936888550999999, 2024-03-11T21:33:08.130 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.151794016, 2024-03-11T21:33:08.130 INFO:teuthology.orchestra.run.smithi097.stdout: "type_data": { 2024-03-11T21:33:08.130 INFO:teuthology.orchestra.run.smithi097.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-11T21:33:08.130 INFO:teuthology.orchestra.run.smithi097.stdout: "events": [ 2024-03-11T21:33:08.130 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:33:08.130 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "initiated", 2024-03-11T21:33:08.130 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:32:56.084147+0000", 2024-03-11T21:33:08.130 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-03-11T21:33:08.130 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:33:08.130 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:33:08.130 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "header_read", 2024-03-11T21:33:08.131 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:32:56.084147+0000", 2024-03-11T21:33:08.131 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-03-11T21:33:08.131 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:33:08.131 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:33:08.131 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "throttled", 2024-03-11T21:33:08.131 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:32:56.084148+0000", 2024-03-11T21:33:08.131 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 3.5100000000000001e-07 2024-03-11T21:33:08.131 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:33:08.131 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:33:08.131 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "all_read", 2024-03-11T21:33:08.131 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:32:56.084179+0000", 2024-03-11T21:33:08.131 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 3.1275000000000001e-05 2024-03-11T21:33:08.131 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:33:08.132 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:33:08.132 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "dispatched", 2024-03-11T21:33:08.132 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:32:56.084192+0000", 2024-03-11T21:33:08.132 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.3609e-05 2024-03-11T21:33:08.132 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:33:08.132 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:33:08.132 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "queued_for_pg", 2024-03-11T21:33:08.132 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:32:56.084202+0000", 2024-03-11T21:33:08.132 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 9.6460000000000005e-06 2024-03-11T21:33:08.132 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:33:08.132 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:33:08.132 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "reached_pg", 2024-03-11T21:33:08.132 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:32:56.217558+0000", 2024-03-11T21:33:08.133 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.13335607899999999 2024-03-11T21:33:08.133 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:33:08.133 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:33:08.133 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "started", 2024-03-11T21:33:08.133 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:32:56.217581+0000", 2024-03-11T21:33:08.133 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 2.2654e-05 2024-03-11T21:33:08.133 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:33:08.133 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:33:08.133 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "commit_sent", 2024-03-11T21:33:08.133 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:32:56.235851+0000", 2024-03-11T21:33:08.133 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.018270249999999998 2024-03-11T21:33:08.133 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:33:08.133 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:33:08.133 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "done", 2024-03-11T21:33:08.133 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:32:56.235941+0000", 2024-03-11T21:33:08.134 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 9.0152e-05 2024-03-11T21:33:08.134 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-03-11T21:33:08.134 INFO:teuthology.orchestra.run.smithi097.stdout: ] 2024-03-11T21:33:08.134 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-03-11T21:33:08.134 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-03-11T21:33:08.134 INFO:teuthology.orchestra.run.smithi097.stdout: ] 2024-03-11T21:33:08.134 INFO:teuthology.orchestra.run.smithi097.stdout:} 2024-03-11T21:33:08.159 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:33:08.160 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:33:08.161 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:33:08.261 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:33:08.286 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:33:08.287 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:33:08.287 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:33:08.360 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:33:08 smithi092 ceph-mon[29572]: purged_snaps scrub starts 2024-03-11T21:33:08.360 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:33:08 smithi092 ceph-mon[29572]: purged_snaps scrub ok 2024-03-11T21:33:08.360 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:33:08 smithi092 ceph-mon[29572]: pgmap v307: 158 pgs: 2 active+remapped+backfill_wait, 2 active+remapped+backfill_toofull, 6 active+remapped+backfilling, 148 active+clean; 5.7 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 853 B/s rd, 0 op/s; 49898/282765 objects misplaced (17.646%); 49 MiB/s, 781 objects/s recovering 2024-03-11T21:33:08.360 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:33:08 smithi092 ceph-mon[29572]: from='client.? v1:172.21.15.92:0/739942126' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]': finished 2024-03-11T21:33:08.361 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:33:08 smithi092 ceph-mon[29572]: osdmap e189: 8 total, 8 up, 8 in 2024-03-11T21:33:08.361 INFO:journalctl@ceph.mgr.y.smithi092.stdout:Mar 11 21:33:07 smithi092 ceph-00db0016-dfed-11ee-95c8-87774f69a715-mgr-y[29801]: ::ffff:172.21.15.97 - - [11/Mar/2024:21:33:07] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-11T21:33:08.361 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:33:08 smithi092 ceph-mon[37563]: purged_snaps scrub starts 2024-03-11T21:33:08.361 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:33:08 smithi092 ceph-mon[37563]: purged_snaps scrub ok 2024-03-11T21:33:08.361 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:33:08 smithi092 ceph-mon[37563]: pgmap v307: 158 pgs: 2 active+remapped+backfill_wait, 2 active+remapped+backfill_toofull, 6 active+remapped+backfilling, 148 active+clean; 5.7 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 853 B/s rd, 0 op/s; 49898/282765 objects misplaced (17.646%); 49 MiB/s, 781 objects/s recovering 2024-03-11T21:33:08.361 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:33:08 smithi092 ceph-mon[37563]: from='client.? v1:172.21.15.92:0/739942126' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]': finished 2024-03-11T21:33:08.361 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:33:08 smithi092 ceph-mon[37563]: osdmap e189: 8 total, 8 up, 8 in 2024-03-11T21:33:08.363 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:33:08 smithi097 ceph-mon[33317]: purged_snaps scrub starts 2024-03-11T21:33:08.363 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:33:08 smithi097 ceph-mon[33317]: purged_snaps scrub ok 2024-03-11T21:33:08.363 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:33:08 smithi097 ceph-mon[33317]: pgmap v307: 158 pgs: 2 active+remapped+backfill_wait, 2 active+remapped+backfill_toofull, 6 active+remapped+backfilling, 148 active+clean; 5.7 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 853 B/s rd, 0 op/s; 49898/282765 objects misplaced (17.646%); 49 MiB/s, 781 objects/s recovering 2024-03-11T21:33:08.363 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:33:08 smithi097 ceph-mon[33317]: from='client.? v1:172.21.15.92:0/739942126' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]': finished 2024-03-11T21:33:08.363 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:33:08 smithi097 ceph-mon[33317]: osdmap e189: 8 total, 8 up, 8 in 2024-03-11T21:33:08.388 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:33:08.423 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:33:08.424 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:33:08.424 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:33:08.435 INFO:tasks.radosbench.radosbench.0.smithi092.stdout: 67 16 98964 98948 92.2885 58.125 0.0167695 0.010831 2024-03-11T21:33:08.525 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:33:08.548 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:33:08.549 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:33:08.549 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:33:08.650 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:33:08.692 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:33:08.693 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:33:08.693 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:33:08.794 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:33:08.818 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:33:08.818 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:33:08.818 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:33:08.919 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:33:08.954 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:33:08.954 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:33:08.954 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:33:09.055 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:33:09.079 DEBUG:teuthology.orchestra.run.smithi097:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid 00db0016-dfed-11ee-95c8-87774f69a715 -- ceph daemon osd.7 dump_ops_in_flight 2024-03-11T21:33:09.081 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:33:09.082 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:33:09.082 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:33:09.183 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:33:09.223 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:33:09.224 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:33:09.224 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:33:09.325 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:33:09.327 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:33:09 smithi097 ceph-mon[33317]: osdmap e190: 8 total, 8 up, 8 in 2024-03-11T21:33:09.327 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:33:09 smithi097 ceph-mon[33317]: osdmap e191: 8 total, 8 up, 8 in 2024-03-11T21:33:09.369 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:33:09.370 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:33:09.370 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:33:09.436 INFO:tasks.radosbench.radosbench.0.smithi092.stdout: 68 16 99908 99892 91.7989 59 0.0162952 0.0108884 2024-03-11T21:33:09.446 INFO:teuthology.orchestra.run.smithi097.stderr:Inferring config /var/lib/ceph/00db0016-dfed-11ee-95c8-87774f69a715/mon.b/config 2024-03-11T21:33:09.471 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:33:09.514 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:33:09.515 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:33:09.515 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:33:09.515 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:33:09 smithi092 ceph-mon[37563]: osdmap e190: 8 total, 8 up, 8 in 2024-03-11T21:33:09.516 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:33:09 smithi092 ceph-mon[37563]: osdmap e191: 8 total, 8 up, 8 in 2024-03-11T21:33:09.516 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:33:09 smithi092 ceph-mon[29572]: osdmap e190: 8 total, 8 up, 8 in 2024-03-11T21:33:09.516 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:33:09 smithi092 ceph-mon[29572]: osdmap e191: 8 total, 8 up, 8 in 2024-03-11T21:33:09.615 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:33:09.662 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:33:09.663 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:33:09.663 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:33:09.764 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:33:09.807 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:33:09.808 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:33:09.808 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:33:09.909 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:33:09.955 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:33:09.956 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:33:09.957 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:33:09.996 DEBUG:teuthology.orchestra.run.smithi092:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid 00db0016-dfed-11ee-95c8-87774f69a715 -- ceph osd reweight 1 1 2024-03-11T21:33:10.058 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:33:10.137 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:33:10.137 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:33:10.138 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:33:10.238 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:33:10.273 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:33:10.275 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:33:10.276 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:33:10.376 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:33:10.420 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:33:10.421 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:33:10.421 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:33:10.437 INFO:tasks.radosbench.radosbench.0.smithi092.stdout: 69 16 100980 100964 91.439 67 0.0118996 0.0109311 2024-03-11T21:33:10.522 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:33:10.554 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:33:10.555 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:33:10.555 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:33:10.576 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:33:10 smithi097 ceph-mon[33317]: pgmap v311: 158 pgs: 2 active+remapped+backfill_wait, 2 active+remapped+backfill_toofull, 6 active+remapped+backfilling, 148 active+clean; 6.0 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 1.2 KiB/s rd, 83 MiB/s wr, 1.32k op/s; 49242/294615 objects misplaced (16.714%); 61 MiB/s, 981 objects/s recovering 2024-03-11T21:33:10.610 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:33:10 smithi092 ceph-mon[29572]: pgmap v311: 158 pgs: 2 active+remapped+backfill_wait, 2 active+remapped+backfill_toofull, 6 active+remapped+backfilling, 148 active+clean; 6.0 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 1.2 KiB/s rd, 83 MiB/s wr, 1.32k op/s; 49242/294615 objects misplaced (16.714%); 61 MiB/s, 981 objects/s recovering 2024-03-11T21:33:10.611 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:33:10 smithi092 ceph-mon[37563]: pgmap v311: 158 pgs: 2 active+remapped+backfill_wait, 2 active+remapped+backfill_toofull, 6 active+remapped+backfilling, 148 active+clean; 6.0 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 1.2 KiB/s rd, 83 MiB/s wr, 1.32k op/s; 49242/294615 objects misplaced (16.714%); 61 MiB/s, 981 objects/s recovering 2024-03-11T21:33:10.656 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:33:10.686 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:33:10.688 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:33:10.688 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:33:10.789 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:33:10.821 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:33:10.822 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:33:10.822 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:33:10.838 INFO:teuthology.orchestra.run.smithi092.stderr:Inferring config /var/lib/ceph/00db0016-dfed-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T21:33:10.912 INFO:teuthology.orchestra.run.smithi092.stderr:Inferring config /var/lib/ceph/00db0016-dfed-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T21:33:10.923 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:33:10.975 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:33:10.976 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:33:10.976 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:33:11.077 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:33:11.100 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:33:11.101 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:33:11.101 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:33:11.202 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:33:11.248 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:33:11.249 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:33:11.250 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:33:11.350 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:33:11.380 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:33:11.381 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:33:11.381 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:33:11.436 INFO:tasks.radosbench.radosbench.0.smithi092.stdout: 70 16 102016 102000 91.0577 64.75 0.0152715 0.0109765 2024-03-11T21:33:11.483 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:33:11.513 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:33:11.513 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:33:11.513 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:33:11.614 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:33:11.661 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:33:11.662 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:33:11.662 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:33:11.763 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:33:11.799 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:33:11.800 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:33:11.801 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:33:11.901 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:33:11.934 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:33:11.935 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:33:11.936 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:33:12.036 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:33:12.076 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:33:12.077 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:33:12.077 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:33:12.178 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:33:12.211 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:33:12.211 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:33:12.211 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:33:12.312 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:33:12.343 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:33:12.344 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:33:12.344 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:33:12.436 INFO:tasks.radosbench.radosbench.0.smithi092.stdout: 71 16 103189 103173 90.8077 73.3125 0.00694128 0.0110077 2024-03-11T21:33:12.445 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:33:12.479 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:33:12.479 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:33:12.479 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:33:12.576 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:33:12 smithi097 ceph-mon[33317]: pgmap v312: 158 pgs: 2 active+remapped+backfill_wait, 2 active+remapped+backfill_toofull, 6 active+remapped+backfilling, 148 active+clean; 6.0 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 1023 B/s rd, 66 MiB/s wr, 1.06k op/s; 49242/294615 objects misplaced (16.714%); 49 MiB/s, 784 objects/s recovering 2024-03-11T21:33:12.576 INFO:journalctl@ceph.iscsi.iscsi.a.smithi097.stdout:Mar 11 21:33:12 smithi097 ceph-00db0016-dfed-11ee-95c8-87774f69a715-iscsi-iscsi-a[60943]: debug there is no tcmu-runner data available 2024-03-11T21:33:12.580 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:33:12.610 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:33:12 smithi092 ceph-mon[29572]: pgmap v312: 158 pgs: 2 active+remapped+backfill_wait, 2 active+remapped+backfill_toofull, 6 active+remapped+backfilling, 148 active+clean; 6.0 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 1023 B/s rd, 66 MiB/s wr, 1.06k op/s; 49242/294615 objects misplaced (16.714%); 49 MiB/s, 784 objects/s recovering 2024-03-11T21:33:12.610 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:33:12 smithi092 ceph-mon[37563]: pgmap v312: 158 pgs: 2 active+remapped+backfill_wait, 2 active+remapped+backfill_toofull, 6 active+remapped+backfilling, 148 active+clean; 6.0 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 1023 B/s rd, 66 MiB/s wr, 1.06k op/s; 49242/294615 objects misplaced (16.714%); 49 MiB/s, 784 objects/s recovering 2024-03-11T21:33:12.625 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:33:12.626 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:33:12.626 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:33:12.727 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:33:12.770 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:33:12.771 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:33:12.771 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:33:12.821 DEBUG:teuthology.orchestra.run.smithi092:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid 00db0016-dfed-11ee-95c8-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-03-11T21:33:12.872 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:33:12.902 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:33:12.903 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:33:12.903 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:33:13.004 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:33:13.028 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:33:13.028 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:33:13.029 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:33:13.130 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:33:13.162 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:33:13.163 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:33:13.163 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:33:13.264 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:33:13.291 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:33:13.292 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:33:13.292 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:33:13.393 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:33:13.437 INFO:tasks.radosbench.radosbench.0.smithi092.stdout: 72 16 104060 104044 90.3025 54.4375 0.0175711 0.0110688 2024-03-11T21:33:13.459 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:33:13.460 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:33:13.460 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:33:13.494 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:33:13 smithi092 ceph-mon[29572]: from='client.24469 v1:172.21.15.97:0/404208310' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:33:13.494 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:33:13 smithi092 ceph-mon[37563]: from='client.24469 v1:172.21.15.97:0/404208310' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:33:13.562 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:33:13.576 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:33:13 smithi097 ceph-mon[33317]: from='client.24469 v1:172.21.15.97:0/404208310' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:33:13.604 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:33:13.605 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:33:13.605 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:33:13.706 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:33:13.738 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:33:13.739 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:33:13.739 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:33:13.840 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:33:13.884 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:33:13.884 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:33:13.885 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:33:13.985 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:33:14.011 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:33:14.012 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:33:14.012 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:33:14.113 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:33:14.147 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:33:14.148 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:33:14.148 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:33:14.249 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:33:14.289 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:33:14.290 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:33:14.290 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:33:14.391 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:33:14.416 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:33:14.416 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:33:14.416 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:33:14.436 INFO:tasks.radosbench.radosbench.0.smithi092.stdout: 73 16 104848 104832 89.7401 49.25 0.00717329 0.0111333 2024-03-11T21:33:14.517 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:33:14.540 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:33:14.541 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:33:14.541 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:33:14.576 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:33:14 smithi097 ceph-mon[33317]: pgmap v313: 158 pgs: 2 active+remapped+backfill_wait, 2 active+remapped+backfill_toofull, 5 active+remapped+backfilling, 149 active+clean; 6.3 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 1.7 KiB/s rd, 95 MiB/s wr, 1.52k op/s; 48134/310074 objects misplaced (15.523%); 41 MiB/s, 658 objects/s recovering 2024-03-11T21:33:14.610 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:33:14 smithi092 ceph-mon[29572]: pgmap v313: 158 pgs: 2 active+remapped+backfill_wait, 2 active+remapped+backfill_toofull, 5 active+remapped+backfilling, 149 active+clean; 6.3 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 1.7 KiB/s rd, 95 MiB/s wr, 1.52k op/s; 48134/310074 objects misplaced (15.523%); 41 MiB/s, 658 objects/s recovering 2024-03-11T21:33:14.610 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:33:14 smithi092 ceph-mon[37563]: pgmap v313: 158 pgs: 2 active+remapped+backfill_wait, 2 active+remapped+backfill_toofull, 5 active+remapped+backfilling, 149 active+clean; 6.3 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 1.7 KiB/s rd, 95 MiB/s wr, 1.52k op/s; 48134/310074 objects misplaced (15.523%); 41 MiB/s, 658 objects/s recovering 2024-03-11T21:33:14.643 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:33:14.684 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:33:14.687 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:33:14.688 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:33:14.789 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:33:14.814 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:33:14.814 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:33:14.814 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:33:14.915 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:33:14.944 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:33:14.945 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:33:14.945 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:33:15.046 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:33:15.070 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:33:15.071 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:33:15.071 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:33:15.172 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:33:15.202 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:33:15.203 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:33:15.203 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:33:15.304 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:33:15.350 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:33:15.351 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:33:15.351 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:33:15.437 INFO:tasks.radosbench.radosbench.0.smithi092.stdout: 74 15 105548 105533 89.1194 43.8125 0.0100688 0.011216 2024-03-11T21:33:15.454 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:33:15.493 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:33:15.493 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:33:15.494 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:33:15.576 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:33:15 smithi097 ceph-mon[33317]: from='client.? v1:172.21.15.92:0/2823699298' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-11T21:33:15.595 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:33:15.610 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:33:15 smithi092 ceph-mon[29572]: from='client.? v1:172.21.15.92:0/2823699298' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-11T21:33:15.610 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:33:15 smithi092 ceph-mon[37563]: from='client.? v1:172.21.15.92:0/2823699298' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-11T21:33:15.618 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:33:15.619 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:33:15.619 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:33:15.720 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:33:15.750 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:33:15.750 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:33:15.751 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:33:15.765 INFO:teuthology.orchestra.run.smithi092.stderr:noscrub is set 2024-03-11T21:33:15.851 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:33:15.874 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:33:15.875 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:33:15.875 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:33:15.976 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:33:16.002 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:33:16.002 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:33:16.002 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:33:16.103 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:33:16.130 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:33:16.131 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:33:16.131 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:33:16.232 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:33:16.255 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:33:16.256 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:33:16.256 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:33:16.357 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:33:16.401 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:33:16.401 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:33:16.401 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:33:16.437 INFO:tasks.radosbench.radosbench.0.smithi092.stdout: 75 16 106312 106296 88.5669 47.6875 0.00989909 0.0112864 2024-03-11T21:33:16.502 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:33:16.529 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:33:16.529 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:33:16.529 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:33:16.576 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:33:16 smithi097 ceph-mon[33317]: pgmap v314: 158 pgs: 2 active+remapped+backfill_wait, 2 active+remapped+backfill_toofull, 5 active+remapped+backfilling, 149 active+clean; 6.3 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 1.3 KiB/s rd, 71 MiB/s wr, 1.14k op/s; 48134/310074 objects misplaced (15.523%); 31 MiB/s, 494 objects/s recovering 2024-03-11T21:33:16.576 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:33:16 smithi097 ceph-mon[33317]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-03-11T21:33:16.576 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:33:16 smithi097 ceph-mon[33317]: from='client.? v1:172.21.15.92:0/2823699298' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-03-11T21:33:16.576 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:33:16 smithi097 ceph-mon[33317]: osdmap e192: 8 total, 8 up, 8 in 2024-03-11T21:33:16.610 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:33:16 smithi092 ceph-mon[29572]: pgmap v314: 158 pgs: 2 active+remapped+backfill_wait, 2 active+remapped+backfill_toofull, 5 active+remapped+backfilling, 149 active+clean; 6.3 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 1.3 KiB/s rd, 71 MiB/s wr, 1.14k op/s; 48134/310074 objects misplaced (15.523%); 31 MiB/s, 494 objects/s recovering 2024-03-11T21:33:16.610 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:33:16 smithi092 ceph-mon[29572]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-03-11T21:33:16.610 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:33:16 smithi092 ceph-mon[29572]: from='client.? v1:172.21.15.92:0/2823699298' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-03-11T21:33:16.611 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:33:16 smithi092 ceph-mon[29572]: osdmap e192: 8 total, 8 up, 8 in 2024-03-11T21:33:16.611 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:33:16 smithi092 ceph-mon[37563]: pgmap v314: 158 pgs: 2 active+remapped+backfill_wait, 2 active+remapped+backfill_toofull, 5 active+remapped+backfilling, 149 active+clean; 6.3 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 1.3 KiB/s rd, 71 MiB/s wr, 1.14k op/s; 48134/310074 objects misplaced (15.523%); 31 MiB/s, 494 objects/s recovering 2024-03-11T21:33:16.611 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:33:16 smithi092 ceph-mon[37563]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-03-11T21:33:16.611 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:33:16 smithi092 ceph-mon[37563]: from='client.? v1:172.21.15.92:0/2823699298' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-03-11T21:33:16.611 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:33:16 smithi092 ceph-mon[37563]: osdmap e192: 8 total, 8 up, 8 in 2024-03-11T21:33:16.630 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:33:16.664 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:33:16.664 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:33:16.665 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:33:16.766 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:33:16.790 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:33:16.790 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:33:16.790 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:33:16.891 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:33:16.934 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:33:16.934 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:33:16.934 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:33:17.035 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:33:17.076 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:33:17.076 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:33:17.077 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:33:17.178 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:33:17.216 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:33:17.217 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:33:17.217 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:33:17.302 INFO:teuthology.orchestra.run.smithi097.stdout:{ 2024-03-11T21:33:17.302 INFO:teuthology.orchestra.run.smithi097.stdout: "ops": [], 2024-03-11T21:33:17.303 INFO:teuthology.orchestra.run.smithi097.stdout: "num_ops": 0 2024-03-11T21:33:17.303 INFO:teuthology.orchestra.run.smithi097.stdout:} 2024-03-11T21:33:17.318 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:33:17.343 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:33:17.344 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:33:17.344 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:33:17.437 INFO:tasks.radosbench.radosbench.0.smithi092.stdout: 76 16 107235 107219 88.1606 57.6875 0.0173882 0.0113382 2024-03-11T21:33:17.444 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:33:17.476 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:33:17.476 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:33:17.477 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:33:17.578 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:33:17.605 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:33:17.605 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:33:17.605 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:33:17.706 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:33:17.732 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:33:17.733 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:33:17.734 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:33:17.835 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:33:17.864 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:33:17.865 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:33:17.866 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:33:17.966 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:33:17.990 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:33:17.991 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:33:17.991 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:33:18.092 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:33:18.138 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:33:18.139 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:33:18.139 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:33:18.174 DEBUG:teuthology.orchestra.run.smithi097:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid 00db0016-dfed-11ee-95c8-87774f69a715 -- ceph daemon osd.7 dump_blocked_ops 2024-03-11T21:33:18.240 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:33:18.273 INFO:journalctl@ceph.mgr.y.smithi092.stdout:Mar 11 21:33:17 smithi092 ceph-00db0016-dfed-11ee-95c8-87774f69a715-mgr-y[29801]: ::ffff:172.21.15.97 - - [11/Mar/2024:21:33:17] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-11T21:33:18.284 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:33:18.284 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:33:18.285 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:33:18.385 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:33:18.422 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:33:18.422 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:33:18.423 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:33:18.437 INFO:tasks.radosbench.radosbench.0.smithi092.stdout: 77 16 108052 108036 87.6787 51.0625 0.0285151 0.0113973 2024-03-11T21:33:18.498 INFO:teuthology.orchestra.run.smithi097.stderr:Inferring config /var/lib/ceph/00db0016-dfed-11ee-95c8-87774f69a715/mon.b/config 2024-03-11T21:33:18.524 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:33:18.566 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:33:18.566 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:33:18.567 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:33:18.576 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:33:18 smithi097 ceph-mon[33317]: pgmap v316: 158 pgs: 2 active+remapped+backfill_wait, 2 active+remapped+backfill_toofull, 5 active+remapped+backfilling, 149 active+clean; 6.4 GiB data, 22 GiB used, 694 GiB / 715 GiB avail; 576 B/s rd, 43 MiB/s wr, 695 op/s; 48067/313122 objects misplaced (15.351%); 19 MiB/s, 308 objects/s recovering 2024-03-11T21:33:18.576 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:33:18 smithi097 ceph-mon[33317]: osdmap e193: 8 total, 8 up, 8 in 2024-03-11T21:33:18.611 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:33:18 smithi092 ceph-mon[29572]: pgmap v316: 158 pgs: 2 active+remapped+backfill_wait, 2 active+remapped+backfill_toofull, 5 active+remapped+backfilling, 149 active+clean; 6.4 GiB data, 22 GiB used, 694 GiB / 715 GiB avail; 576 B/s rd, 43 MiB/s wr, 695 op/s; 48067/313122 objects misplaced (15.351%); 19 MiB/s, 308 objects/s recovering 2024-03-11T21:33:18.611 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:33:18 smithi092 ceph-mon[29572]: osdmap e193: 8 total, 8 up, 8 in 2024-03-11T21:33:18.611 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:33:18 smithi092 ceph-mon[37563]: pgmap v316: 158 pgs: 2 active+remapped+backfill_wait, 2 active+remapped+backfill_toofull, 5 active+remapped+backfilling, 149 active+clean; 6.4 GiB data, 22 GiB used, 694 GiB / 715 GiB avail; 576 B/s rd, 43 MiB/s wr, 695 op/s; 48067/313122 objects misplaced (15.351%); 19 MiB/s, 308 objects/s recovering 2024-03-11T21:33:18.611 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:33:18 smithi092 ceph-mon[37563]: osdmap e193: 8 total, 8 up, 8 in 2024-03-11T21:33:18.668 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:33:18.709 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:33:18.710 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:33:18.710 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:33:18.811 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:33:18.847 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:33:18.848 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:33:18.848 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:33:18.949 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:33:18.976 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:33:18.977 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:33:18.977 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:33:19.079 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:33:19.108 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:33:19.109 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:33:19.109 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:33:19.210 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:33:19.234 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:33:19.234 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:33:19.235 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:33:19.336 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:33:19.375 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:33:19.376 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:33:19.376 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:33:19.437 INFO:tasks.radosbench.radosbench.0.smithi092.stdout: 78 16 109008 108992 87.3206 59.75 0.00425932 0.0114471 2024-03-11T21:33:19.477 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:33:19.509 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:33:19.509 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:33:19.509 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:33:19.610 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:33:19.647 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:33:19.648 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:33:19.648 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:33:19.749 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:33:19.788 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:33:19.789 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:33:19.790 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:33:19.890 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:33:19.932 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:33:19.933 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:33:19.933 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:33:20.034 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:33:20.065 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:33:20.066 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:33:20.066 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:33:20.167 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:33:20.210 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:33:20.211 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:33:20.211 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:33:20.312 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:33:20.325 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:33:20 smithi097 ceph-mon[33317]: pgmap v318: 158 pgs: 2 active+remapped+backfill_wait, 2 active+remapped+backfill_toofull, 5 active+remapped+backfilling, 149 active+clean; 6.5 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 1.2 KiB/s rd, 72 MiB/s wr, 1.16k op/s; 46416/322377 objects misplaced (14.398%); 35 MiB/s, 554 objects/s recovering 2024-03-11T21:33:20.337 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:33:20.338 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:33:20.338 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:33:20.438 INFO:tasks.radosbench.radosbench.0.smithi092.stdout: 79 16 110065 110049 87.0515 66.0625 0.0102987 0.0114822 2024-03-11T21:33:20.439 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:33:20.474 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:33:20.475 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:33:20.475 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:33:20.576 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:33:20.610 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:33:20 smithi092 ceph-mon[29572]: pgmap v318: 158 pgs: 2 active+remapped+backfill_wait, 2 active+remapped+backfill_toofull, 5 active+remapped+backfilling, 149 active+clean; 6.5 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 1.2 KiB/s rd, 72 MiB/s wr, 1.16k op/s; 46416/322377 objects misplaced (14.398%); 35 MiB/s, 554 objects/s recovering 2024-03-11T21:33:20.610 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:33:20 smithi092 ceph-mon[37563]: pgmap v318: 158 pgs: 2 active+remapped+backfill_wait, 2 active+remapped+backfill_toofull, 5 active+remapped+backfilling, 149 active+clean; 6.5 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 1.2 KiB/s rd, 72 MiB/s wr, 1.16k op/s; 46416/322377 objects misplaced (14.398%); 35 MiB/s, 554 objects/s recovering 2024-03-11T21:33:20.619 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:33:20.619 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:33:20.620 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:33:20.721 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:33:20.745 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:33:20.746 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:33:20.746 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:33:20.847 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:33:20.848 INFO:teuthology.orchestra.run.smithi097.stdout:{ 2024-03-11T21:33:20.848 INFO:teuthology.orchestra.run.smithi097.stdout: "ops": [], 2024-03-11T21:33:20.848 INFO:teuthology.orchestra.run.smithi097.stdout: "complaint_time": 30, 2024-03-11T21:33:20.848 INFO:teuthology.orchestra.run.smithi097.stdout: "num_blocked_ops": 0 2024-03-11T21:33:20.848 INFO:teuthology.orchestra.run.smithi097.stdout:} 2024-03-11T21:33:20.892 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:33:20.893 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:33:20.893 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:33:20.994 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:33:21.035 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:33:21.035 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:33:21.036 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:33:21.137 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:33:21.170 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:33:21.171 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:33:21.171 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:33:21.187 INFO:teuthology.orchestra.run.smithi092.stderr:Inferring config /var/lib/ceph/00db0016-dfed-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T21:33:21.272 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:33:21.322 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:33:21.322 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:33:21.322 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:33:21.423 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:33:21.438 INFO:tasks.radosbench.radosbench.0.smithi092.stdout:2024-03-11T21:33:21.457331+0000 min lat: 0.0013089 max lat: 1.3116 avg lat: 0.0115183 2024-03-11T21:33:21.438 INFO:tasks.radosbench.radosbench.0.smithi092.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-03-11T21:33:21.438 INFO:tasks.radosbench.radosbench.0.smithi092.stdout: 80 16 111111 111095 86.7792 65.375 0.00833209 0.0115183 2024-03-11T21:33:21.477 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:33:21.478 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:33:21.478 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:33:21.579 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:33:21.623 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:33:21.623 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:33:21.624 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:33:21.641 DEBUG:teuthology.orchestra.run.smithi097:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid 00db0016-dfed-11ee-95c8-87774f69a715 -- ceph daemon osd.7 dump_historic_ops 2024-03-11T21:33:21.724 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:33:21.775 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:33:21.776 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:33:21.776 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:33:21.877 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:33:21.932 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:33:21.933 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:33:21.933 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:33:22.011 INFO:teuthology.orchestra.run.smithi097.stderr:Inferring config /var/lib/ceph/00db0016-dfed-11ee-95c8-87774f69a715/mon.b/config 2024-03-11T21:33:22.034 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:33:22.071 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:33:22.072 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:33:22.072 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:33:22.173 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:33:22.229 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:33:22.230 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:33:22.230 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:33:22.331 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:33:22.360 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:33:22.361 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:33:22.361 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:33:22.395 INFO:journalctl@ceph.iscsi.iscsi.a.smithi097.stdout:Mar 11 21:33:22 smithi097 ceph-00db0016-dfed-11ee-95c8-87774f69a715-iscsi-iscsi-a[60943]: debug there is no tcmu-runner data available 2024-03-11T21:33:22.395 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:33:22 smithi097 ceph-mon[33317]: pgmap v319: 158 pgs: 2 active+remapped+backfill_wait, 2 active+remapped+backfill_toofull, 5 active+remapped+backfilling, 149 active+clean; 6.5 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 639 B/s rd, 32 MiB/s wr, 513 op/s; 46416/322377 objects misplaced (14.398%); 16 MiB/s, 263 objects/s recovering 2024-03-11T21:33:22.395 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:33:22 smithi097 ceph-mon[33317]: from='mgr.14607 v1:172.21.15.92:0/1702717597' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T21:33:22.395 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:33:22 smithi097 ceph-mon[33317]: from='mgr.14607 ' entity='mgr.y' 2024-03-11T21:33:22.395 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:33:22 smithi097 ceph-mon[33317]: from='mgr.14607 v1:172.21.15.92:0/1702717597' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T21:33:22.439 INFO:tasks.radosbench.radosbench.0.smithi092.stdout: 81 15 111997 111982 86.3916 55.4375 0.0207626 0.0115703 2024-03-11T21:33:22.462 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:33:22.510 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:33:22.511 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:33:22.511 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:33:22.610 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:33:22 smithi092 ceph-mon[29572]: pgmap v319: 158 pgs: 2 active+remapped+backfill_wait, 2 active+remapped+backfill_toofull, 5 active+remapped+backfilling, 149 active+clean; 6.5 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 639 B/s rd, 32 MiB/s wr, 513 op/s; 46416/322377 objects misplaced (14.398%); 16 MiB/s, 263 objects/s recovering 2024-03-11T21:33:22.610 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:33:22 smithi092 ceph-mon[29572]: from='mgr.14607 v1:172.21.15.92:0/1702717597' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T21:33:22.610 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:33:22 smithi092 ceph-mon[29572]: from='mgr.14607 ' entity='mgr.y' 2024-03-11T21:33:22.610 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:33:22 smithi092 ceph-mon[29572]: from='mgr.14607 v1:172.21.15.92:0/1702717597' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T21:33:22.610 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:33:22 smithi092 ceph-mon[37563]: pgmap v319: 158 pgs: 2 active+remapped+backfill_wait, 2 active+remapped+backfill_toofull, 5 active+remapped+backfilling, 149 active+clean; 6.5 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 639 B/s rd, 32 MiB/s wr, 513 op/s; 46416/322377 objects misplaced (14.398%); 16 MiB/s, 263 objects/s recovering 2024-03-11T21:33:22.611 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:33:22 smithi092 ceph-mon[37563]: from='mgr.14607 v1:172.21.15.92:0/1702717597' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T21:33:22.611 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:33:22 smithi092 ceph-mon[37563]: from='mgr.14607 ' entity='mgr.y' 2024-03-11T21:33:22.611 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:33:22 smithi092 ceph-mon[37563]: from='mgr.14607 v1:172.21.15.92:0/1702717597' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T21:33:22.612 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:33:22.651 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:33:22.651 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:33:22.651 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:33:22.752 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:33:22.782 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:33:22.783 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:33:22.783 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:33:22.886 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:33:22.946 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:33:22.946 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:33:22.947 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:33:23.048 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:33:23.080 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:33:23.081 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:33:23.081 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:33:23.153 DEBUG:teuthology.orchestra.run.smithi092:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid 00db0016-dfed-11ee-95c8-87774f69a715 -- ceph osd set nodeep-scrub 2024-03-11T21:33:23.182 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:33:23.209 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:33:23.210 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:33:23.210 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:33:23.311 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:33:23.326 INFO:teuthology.orchestra.run.smithi092.stderr:reweighted osd.1 to 1 (10000) 2024-03-11T21:33:23.358 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:33:23 smithi092 ceph-mon[29572]: from='client.24469 v1:172.21.15.97:0/404208310' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:33:23.358 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:33:23 smithi092 ceph-mon[29572]: from='client.? v1:172.21.15.92:0/1509318900' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]: dispatch 2024-03-11T21:33:23.359 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:33:23 smithi092 ceph-mon[29572]: from='mgr.14607 v1:172.21.15.92:0/1702717597' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T21:33:23.359 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:33:23 smithi092 ceph-mon[29572]: from='mgr.14607 v1:172.21.15.92:0/1702717597' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T21:33:23.359 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:33:23 smithi092 ceph-mon[29572]: from='mgr.14607 ' entity='mgr.y' 2024-03-11T21:33:23.359 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:33:23 smithi092 ceph-mon[37563]: from='client.24469 v1:172.21.15.97:0/404208310' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:33:23.359 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:33:23 smithi092 ceph-mon[37563]: from='client.? v1:172.21.15.92:0/1509318900' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]: dispatch 2024-03-11T21:33:23.359 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:33:23 smithi092 ceph-mon[37563]: from='mgr.14607 v1:172.21.15.92:0/1702717597' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T21:33:23.359 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:33:23 smithi092 ceph-mon[37563]: from='mgr.14607 v1:172.21.15.92:0/1702717597' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T21:33:23.359 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:33:23 smithi092 ceph-mon[37563]: from='mgr.14607 ' entity='mgr.y' 2024-03-11T21:33:23.393 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:33:23.393 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:33:23.393 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:33:23.439 INFO:tasks.radosbench.radosbench.0.smithi092.stdout: 82 16 112961 112945 86.072 60.1875 0.0229322 0.0116131 2024-03-11T21:33:23.494 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:33:23.525 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:33:23.526 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:33:23.526 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:33:23.576 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:33:23 smithi097 ceph-mon[33317]: from='client.24469 v1:172.21.15.97:0/404208310' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:33:23.576 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:33:23 smithi097 ceph-mon[33317]: from='client.? v1:172.21.15.92:0/1509318900' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]: dispatch 2024-03-11T21:33:23.576 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:33:23 smithi097 ceph-mon[33317]: from='mgr.14607 v1:172.21.15.92:0/1702717597' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T21:33:23.576 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:33:23 smithi097 ceph-mon[33317]: from='mgr.14607 v1:172.21.15.92:0/1702717597' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T21:33:23.576 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:33:23 smithi097 ceph-mon[33317]: from='mgr.14607 ' entity='mgr.y' 2024-03-11T21:33:23.628 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:33:23.677 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:33:23.677 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:33:23.678 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:33:23.726 INFO:teuthology.orchestra.run.smithi092.stdout:osd.0: {} 2024-03-11T21:33:23.746 INFO:teuthology.orchestra.run.smithi092.stdout:osd.1: {} 2024-03-11T21:33:23.779 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:33:23.796 INFO:teuthology.orchestra.run.smithi092.stdout:osd.2: {} 2024-03-11T21:33:23.833 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:33:23.834 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:33:23.834 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:33:23.838 INFO:teuthology.orchestra.run.smithi092.stdout:osd.3: {} 2024-03-11T21:33:23.884 INFO:teuthology.orchestra.run.smithi092.stdout:osd.4: {} 2024-03-11T21:33:23.922 INFO:teuthology.orchestra.run.smithi092.stdout:osd.5: {} 2024-03-11T21:33:23.936 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:33:23.969 INFO:teuthology.orchestra.run.smithi092.stdout:osd.6: {} 2024-03-11T21:33:23.982 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:33:23.982 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:33:23.983 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:33:24.011 INFO:teuthology.orchestra.run.smithi092.stdout:osd.7: {} 2024-03-11T21:33:24.014 INFO:teuthology.orchestra.run.smithi092.stderr:osd.0: osd_enable_op_tracker = '' 2024-03-11T21:33:24.014 INFO:teuthology.orchestra.run.smithi092.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-11T21:33:24.014 INFO:teuthology.orchestra.run.smithi092.stderr:osd.2: osd_enable_op_tracker = '' 2024-03-11T21:33:24.014 INFO:teuthology.orchestra.run.smithi092.stderr:osd.3: osd_enable_op_tracker = '' 2024-03-11T21:33:24.014 INFO:teuthology.orchestra.run.smithi092.stderr:osd.4: osd_enable_op_tracker = '' 2024-03-11T21:33:24.014 INFO:teuthology.orchestra.run.smithi092.stderr:osd.5: osd_enable_op_tracker = '' 2024-03-11T21:33:24.014 INFO:teuthology.orchestra.run.smithi092.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-11T21:33:24.014 INFO:teuthology.orchestra.run.smithi092.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-11T21:33:24.083 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:33:24.123 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:33:24.124 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:33:24.124 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:33:24.155 INFO:teuthology.orchestra.run.smithi097.stdout:ERROR: (22) Invalid argument 2024-03-11T21:33:24.155 INFO:teuthology.orchestra.run.smithi097.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-11T21:33:24.225 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:33:24.265 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:33:24.265 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:33:24.266 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:33:24.367 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:33:24.411 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:33:24.411 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:33:24.412 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:33:24.439 INFO:tasks.radosbench.radosbench.0.smithi092.stdout: 83 16 113972 113956 85.7963 63.1875 0.0256446 0.0116504 2024-03-11T21:33:24.455 DEBUG:teuthology.orchestra.run.smithi092:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid 00db0016-dfed-11ee-95c8-87774f69a715 -- ceph osd reweight 2 1 2024-03-11T21:33:24.512 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:33:24.544 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:33:24.544 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:33:24.545 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:33:24.576 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:33:24 smithi097 ceph-mon[33317]: pgmap v320: 158 pgs: 2 active+remapped+backfill_wait, 2 active+remapped+backfill_toofull, 5 active+remapped+backfilling, 149 active+clean; 6.8 GiB data, 23 GiB used, 692 GiB / 715 GiB avail; 1.2 KiB/s rd, 69 MiB/s wr, 1.11k op/s; 45180/336771 objects misplaced (13.416%); 33 MiB/s, 520 objects/s recovering 2024-03-11T21:33:24.576 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:33:24 smithi097 ceph-mon[33317]: from='client.? v1:172.21.15.92:0/1509318900' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]': finished 2024-03-11T21:33:24.576 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:33:24 smithi097 ceph-mon[33317]: osdmap e194: 8 total, 8 up, 8 in 2024-03-11T21:33:24.576 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:33:24 smithi097 ceph-mon[33317]: from='client.? v1:172.21.15.92:0/91180561' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-11T21:33:24.611 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:33:24 smithi092 ceph-mon[29572]: pgmap v320: 158 pgs: 2 active+remapped+backfill_wait, 2 active+remapped+backfill_toofull, 5 active+remapped+backfilling, 149 active+clean; 6.8 GiB data, 23 GiB used, 692 GiB / 715 GiB avail; 1.2 KiB/s rd, 69 MiB/s wr, 1.11k op/s; 45180/336771 objects misplaced (13.416%); 33 MiB/s, 520 objects/s recovering 2024-03-11T21:33:24.611 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:33:24 smithi092 ceph-mon[29572]: from='client.? v1:172.21.15.92:0/1509318900' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]': finished 2024-03-11T21:33:24.611 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:33:24 smithi092 ceph-mon[29572]: osdmap e194: 8 total, 8 up, 8 in 2024-03-11T21:33:24.611 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:33:24 smithi092 ceph-mon[29572]: from='client.? v1:172.21.15.92:0/91180561' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-11T21:33:24.612 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:33:24 smithi092 ceph-mon[37563]: pgmap v320: 158 pgs: 2 active+remapped+backfill_wait, 2 active+remapped+backfill_toofull, 5 active+remapped+backfilling, 149 active+clean; 6.8 GiB data, 23 GiB used, 692 GiB / 715 GiB avail; 1.2 KiB/s rd, 69 MiB/s wr, 1.11k op/s; 45180/336771 objects misplaced (13.416%); 33 MiB/s, 520 objects/s recovering 2024-03-11T21:33:24.612 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:33:24 smithi092 ceph-mon[37563]: from='client.? v1:172.21.15.92:0/1509318900' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]': finished 2024-03-11T21:33:24.612 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:33:24 smithi092 ceph-mon[37563]: osdmap e194: 8 total, 8 up, 8 in 2024-03-11T21:33:24.612 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:33:24 smithi092 ceph-mon[37563]: from='client.? v1:172.21.15.92:0/91180561' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-11T21:33:24.645 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:33:24.675 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:33:24.676 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:33:24.676 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:33:24.777 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:33:24.808 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:33:24.808 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:33:24.809 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:33:24.909 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:33:24.918 DEBUG:teuthology.orchestra.run.smithi092:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid 00db0016-dfed-11ee-95c8-87774f69a715 -- ceph daemon osd.0 dump_ops_in_flight 2024-03-11T21:33:24.942 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:33:24.943 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:33:24.943 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:33:25.044 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:33:25.099 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:33:25.099 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:33:25.100 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:33:25.200 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:33:25.268 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:33:25.268 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:33:25.269 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:33:25.370 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:33:25.417 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:33:25.417 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:33:25.418 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:33:25.439 INFO:tasks.radosbench.radosbench.0.smithi092.stdout: 84 16 114991 114975 85.533 63.6875 0.00488335 0.0116871 2024-03-11T21:33:25.518 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:33:25.618 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:33:25 smithi092 ceph-mon[29572]: purged_snaps scrub starts 2024-03-11T21:33:25.618 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:33:25 smithi092 ceph-mon[29572]: purged_snaps scrub ok 2024-03-11T21:33:25.619 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:33:25 smithi097 ceph-mon[33317]: purged_snaps scrub starts 2024-03-11T21:33:25.619 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:33:25 smithi097 ceph-mon[33317]: purged_snaps scrub ok 2024-03-11T21:33:25.619 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:33:25 smithi092 ceph-mon[37563]: purged_snaps scrub starts 2024-03-11T21:33:25.619 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:33:25 smithi092 ceph-mon[37563]: purged_snaps scrub ok 2024-03-11T21:33:25.647 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:33:25.648 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:33:25.649 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:33:25.750 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:33:25.801 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:33:25.803 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:33:25.803 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:33:25.906 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:33:25.930 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:33:25.930 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:33:25.931 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:33:26.031 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:33:26.063 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:33:26.063 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:33:26.064 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:33:26.164 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:33:26.204 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:33:26.204 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:33:26.205 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:33:26.306 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:33:26.351 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:33:26.352 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:33:26.352 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:33:26.439 INFO:tasks.radosbench.radosbench.0.smithi092.stdout: 85 16 115954 115938 85.2348 60.1875 0.0130499 0.0117274 2024-03-11T21:33:26.453 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:33:26.486 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:33:26.486 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:33:26.487 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:33:26.588 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:33:26.610 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:33:26 smithi092 ceph-mon[29572]: pgmap v322: 158 pgs: 2 active+remapped+backfill_wait, 2 active+remapped+backfill_toofull, 5 active+remapped+backfilling, 149 active+clean; 6.8 GiB data, 23 GiB used, 692 GiB / 715 GiB avail; 1.2 KiB/s rd, 62 MiB/s wr, 986 op/s; 45180/336771 objects misplaced (13.416%); 29 MiB/s, 469 objects/s recovering 2024-03-11T21:33:26.610 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:33:26 smithi092 ceph-mon[29572]: osdmap e195: 8 total, 8 up, 8 in 2024-03-11T21:33:26.611 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:33:26 smithi092 ceph-mon[37563]: pgmap v322: 158 pgs: 2 active+remapped+backfill_wait, 2 active+remapped+backfill_toofull, 5 active+remapped+backfilling, 149 active+clean; 6.8 GiB data, 23 GiB used, 692 GiB / 715 GiB avail; 1.2 KiB/s rd, 62 MiB/s wr, 986 op/s; 45180/336771 objects misplaced (13.416%); 29 MiB/s, 469 objects/s recovering 2024-03-11T21:33:26.611 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:33:26 smithi092 ceph-mon[37563]: osdmap e195: 8 total, 8 up, 8 in 2024-03-11T21:33:26.620 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:33:26.621 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:33:26.621 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:33:26.722 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:33:26.757 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:33:26.758 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:33:26.758 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:33:26.826 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:33:26 smithi097 ceph-mon[33317]: pgmap v322: 158 pgs: 2 active+remapped+backfill_wait, 2 active+remapped+backfill_toofull, 5 active+remapped+backfilling, 149 active+clean; 6.8 GiB data, 23 GiB used, 692 GiB / 715 GiB avail; 1.2 KiB/s rd, 62 MiB/s wr, 986 op/s; 45180/336771 objects misplaced (13.416%); 29 MiB/s, 469 objects/s recovering 2024-03-11T21:33:26.826 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:33:26 smithi097 ceph-mon[33317]: osdmap e195: 8 total, 8 up, 8 in 2024-03-11T21:33:26.859 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:33:26.879 INFO:teuthology.orchestra.run.smithi092.stderr:Inferring config /var/lib/ceph/00db0016-dfed-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T21:33:26.925 INFO:teuthology.orchestra.run.smithi092.stderr:Inferring config /var/lib/ceph/00db0016-dfed-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T21:33:26.935 INFO:teuthology.orchestra.run.smithi092.stderr:Inferring config /var/lib/ceph/00db0016-dfed-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T21:33:26.945 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:33:26.946 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:33:26.946 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:33:27.047 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:33:27.107 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:33:27.108 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:33:27.108 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:33:27.209 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:33:27.245 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:33:27.246 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:33:27.246 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:33:27.347 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:33:27.383 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:33:27.383 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:33:27.384 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:33:27.439 INFO:tasks.radosbench.radosbench.0.smithi092.stdout: 86 16 116980 116964 84.9893 64.125 0.0189997 0.011758 2024-03-11T21:33:27.484 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:33:27.523 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:33:27.524 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:33:27.524 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:33:27.625 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:33:27.670 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:33:27.670 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:33:27.671 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:33:27.772 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:33:27.805 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:33:27.806 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:33:27.807 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:33:27.907 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:33:27.947 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:33:27.947 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:33:27.948 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:33:28.049 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:33:28.078 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:33:28.079 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:33:28.079 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:33:28.180 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:33:28.220 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:33:28.221 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:33:28.221 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:33:28.262 INFO:journalctl@ceph.mgr.y.smithi092.stdout:Mar 11 21:33:27 smithi092 ceph-00db0016-dfed-11ee-95c8-87774f69a715-mgr-y[29801]: ::ffff:172.21.15.97 - - [11/Mar/2024:21:33:27] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-11T21:33:28.322 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:33:28.355 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:33:28.355 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:33:28.356 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:33:28.440 INFO:tasks.radosbench.radosbench.0.smithi092.stdout: 87 16 118013 117997 84.754 64.5625 0.0102273 0.0117936 2024-03-11T21:33:28.457 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:33:28.499 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:33:28.500 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:33:28.501 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:33:28.601 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:33:28.610 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:33:28 smithi092 ceph-mon[29572]: pgmap v324: 158 pgs: 2 active+remapped+backfill_wait, 2 active+remapped+backfill_toofull, 5 active+remapped+backfilling, 149 active+clean; 6.9 GiB data, 24 GiB used, 691 GiB / 715 GiB avail; 639 B/s rd, 47 MiB/s wr, 752 op/s; 45069/340431 objects misplaced (13.239%); 20 MiB/s, 317 objects/s recovering 2024-03-11T21:33:28.610 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:33:28 smithi092 ceph-mon[37563]: pgmap v324: 158 pgs: 2 active+remapped+backfill_wait, 2 active+remapped+backfill_toofull, 5 active+remapped+backfilling, 149 active+clean; 6.9 GiB data, 24 GiB used, 691 GiB / 715 GiB avail; 639 B/s rd, 47 MiB/s wr, 752 op/s; 45069/340431 objects misplaced (13.239%); 20 MiB/s, 317 objects/s recovering 2024-03-11T21:33:28.646 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:33:28.646 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:33:28.646 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:33:28.747 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:33:28.776 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:33:28.777 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:33:28.777 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:33:28.820 DEBUG:teuthology.orchestra.run.smithi092:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid 00db0016-dfed-11ee-95c8-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-03-11T21:33:28.826 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:33:28 smithi097 ceph-mon[33317]: pgmap v324: 158 pgs: 2 active+remapped+backfill_wait, 2 active+remapped+backfill_toofull, 5 active+remapped+backfilling, 149 active+clean; 6.9 GiB data, 24 GiB used, 691 GiB / 715 GiB avail; 639 B/s rd, 47 MiB/s wr, 752 op/s; 45069/340431 objects misplaced (13.239%); 20 MiB/s, 317 objects/s recovering 2024-03-11T21:33:28.878 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:33:28.943 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:33:28.943 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:33:28.943 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:33:29.044 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:33:29.089 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:33:29.090 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:33:29.090 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:33:29.191 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:33:29.245 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:33:29.246 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:33:29.246 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:33:29.347 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:33:29.393 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:33:29.394 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:33:29.394 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:33:29.441 INFO:tasks.radosbench.radosbench.0.smithi092.stdout: 88 16 119022 119006 84.5066 63.0625 0.00381812 0.0118272 2024-03-11T21:33:29.495 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:33:29.533 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:33:29.534 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:33:29.535 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:33:29.638 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:33:29.676 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:33:29.677 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:33:29.677 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:33:29.778 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:33:29.805 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:33:29.806 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:33:29.806 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:33:29.907 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:33:29.933 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:33:29.933 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:33:29.934 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:33:30.034 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:33:30.060 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:33:30.061 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:33:30.061 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:33:30.162 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:33:30.188 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:33:30.188 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:33:30.189 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:33:30.290 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:33:30.324 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:33:30.324 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:33:30.325 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:33:30.425 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:33:30.441 INFO:tasks.radosbench.radosbench.0.smithi092.stdout: 89 16 119400 119384 83.8225 23.625 0.015064 0.0119251 2024-03-11T21:33:30.456 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:33:30.456 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:33:30.457 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:33:30.557 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:33:30.600 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:33:30.601 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:33:30.601 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:33:30.610 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:33:30 smithi092 ceph-mon[29572]: pgmap v325: 158 pgs: 2 active+remapped+backfill_wait, 2 active+remapped+backfill_toofull, 5 active+remapped+backfilling, 149 active+clean; 7.1 GiB data, 25 GiB used, 691 GiB / 715 GiB avail; 1.2 KiB/s rd, 76 MiB/s wr, 1.22k op/s; 43601/351660 objects misplaced (12.399%); 32 MiB/s, 518 objects/s recovering 2024-03-11T21:33:30.610 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:33:30 smithi092 ceph-mon[29572]: osdmap e196: 8 total, 8 up, 8 in 2024-03-11T21:33:30.610 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:33:30 smithi092 ceph-mon[37563]: pgmap v325: 158 pgs: 2 active+remapped+backfill_wait, 2 active+remapped+backfill_toofull, 5 active+remapped+backfilling, 149 active+clean; 7.1 GiB data, 25 GiB used, 691 GiB / 715 GiB avail; 1.2 KiB/s rd, 76 MiB/s wr, 1.22k op/s; 43601/351660 objects misplaced (12.399%); 32 MiB/s, 518 objects/s recovering 2024-03-11T21:33:30.610 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:33:30 smithi092 ceph-mon[37563]: osdmap e196: 8 total, 8 up, 8 in 2024-03-11T21:33:30.702 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:33:30.747 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:33:30.748 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:33:30.748 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:33:30.826 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:33:30 smithi097 ceph-mon[33317]: pgmap v325: 158 pgs: 2 active+remapped+backfill_wait, 2 active+remapped+backfill_toofull, 5 active+remapped+backfilling, 149 active+clean; 7.1 GiB data, 25 GiB used, 691 GiB / 715 GiB avail; 1.2 KiB/s rd, 76 MiB/s wr, 1.22k op/s; 43601/351660 objects misplaced (12.399%); 32 MiB/s, 518 objects/s recovering 2024-03-11T21:33:30.826 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:33:30 smithi097 ceph-mon[33317]: osdmap e196: 8 total, 8 up, 8 in 2024-03-11T21:33:30.849 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:33:30.895 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:33:30.895 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:33:30.896 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:33:30.996 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:33:31.037 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:33:31.038 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:33:31.038 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:33:31.139 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:33:31.171 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:33:31.172 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:33:31.172 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:33:31.273 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:33:31.319 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:33:31.319 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:33:31.320 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:33:31.421 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:33:31.441 INFO:tasks.radosbench.radosbench.0.smithi092.stdout: 90 1 120498 120497 83.664 69.5625 0.0100008 0.0119471 2024-03-11T21:33:31.446 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:33:31.446 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:33:31.447 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:33:31.547 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:33:31.571 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:33:31.572 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:33:31.572 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:33:31.625 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:33:31 smithi092 ceph-mon[29572]: osdmap e197: 8 total, 8 up, 8 in 2024-03-11T21:33:31.626 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:33:31 smithi092 ceph-mon[37563]: osdmap e197: 8 total, 8 up, 8 in 2024-03-11T21:33:31.673 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:33:31.697 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:33:31.698 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:33:31.698 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:33:31.799 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:33:31.826 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:33:31 smithi097 ceph-mon[33317]: osdmap e197: 8 total, 8 up, 8 in 2024-03-11T21:33:31.831 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:33:31.832 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:33:31.832 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:33:31.933 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:33:31.970 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:33:31.971 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:33:31.971 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:33:32.073 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:33:32.097 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:33:32.098 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:33:32.098 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:33:32.199 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:33:32.236 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:33:32.236 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:33:32.237 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:33:32.337 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:33:32.371 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:33:32.371 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:33:32.372 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:33:32.442 INFO:tasks.radosbench.radosbench.0.smithi092.stdout:Total time run: 90.9572 2024-03-11T21:33:32.442 INFO:tasks.radosbench.radosbench.0.smithi092.stdout:Total writes made: 120498 2024-03-11T21:33:32.442 INFO:tasks.radosbench.radosbench.0.smithi092.stdout:Write size: 65536 2024-03-11T21:33:32.442 INFO:tasks.radosbench.radosbench.0.smithi092.stdout:Object size: 65536 2024-03-11T21:33:32.442 INFO:tasks.radosbench.radosbench.0.smithi092.stdout:Bandwidth (MB/sec): 82.7986 2024-03-11T21:33:32.442 INFO:tasks.radosbench.radosbench.0.smithi092.stdout:Stddev Bandwidth: 34.1007 2024-03-11T21:33:32.442 INFO:tasks.radosbench.radosbench.0.smithi092.stdout:Max bandwidth (MB/sec): 179.312 2024-03-11T21:33:32.442 INFO:tasks.radosbench.radosbench.0.smithi092.stdout:Min bandwidth (MB/sec): 8.5 2024-03-11T21:33:32.443 INFO:tasks.radosbench.radosbench.0.smithi092.stdout:Average IOPS: 1324 2024-03-11T21:33:32.443 INFO:tasks.radosbench.radosbench.0.smithi092.stdout:Stddev IOPS: 545.611 2024-03-11T21:33:32.443 INFO:tasks.radosbench.radosbench.0.smithi092.stdout:Max IOPS: 2869 2024-03-11T21:33:32.443 INFO:tasks.radosbench.radosbench.0.smithi092.stdout:Min IOPS: 136 2024-03-11T21:33:32.443 INFO:tasks.radosbench.radosbench.0.smithi092.stdout:Average Latency(s): 0.0119553 2024-03-11T21:33:32.443 INFO:tasks.radosbench.radosbench.0.smithi092.stdout:Stddev Latency(s): 0.0231807 2024-03-11T21:33:32.443 INFO:tasks.radosbench.radosbench.0.smithi092.stdout:Max latency(s): 1.3116 2024-03-11T21:33:32.443 INFO:tasks.radosbench.radosbench.0.smithi092.stdout:Min latency(s): 0.0013089 2024-03-11T21:33:32.447 INFO:tasks.radosbench.radosbench.0.smithi092.stdout:Cleaning up (deleting benchmark objects) 2024-03-11T21:33:32.473 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:33:32.515 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:33:32.515 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:33:32.515 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:33:32.517 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:33:32 smithi092 ceph-mon[29572]: pgmap v328: 158 pgs: 2 active+remapped+backfill_wait, 2 active+remapped+backfill_toofull, 5 active+remapped+backfilling, 149 active+clean; 7.1 GiB data, 25 GiB used, 691 GiB / 715 GiB avail; 852 B/s rd, 52 MiB/s wr, 827 op/s; 43601/351660 objects misplaced (12.399%); 22 MiB/s, 348 objects/s recovering 2024-03-11T21:33:32.517 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:33:32 smithi092 ceph-mon[29572]: osdmap e198: 8 total, 8 up, 8 in 2024-03-11T21:33:32.517 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:33:32 smithi092 ceph-mon[37563]: pgmap v328: 158 pgs: 2 active+remapped+backfill_wait, 2 active+remapped+backfill_toofull, 5 active+remapped+backfilling, 149 active+clean; 7.1 GiB data, 25 GiB used, 691 GiB / 715 GiB avail; 852 B/s rd, 52 MiB/s wr, 827 op/s; 43601/351660 objects misplaced (12.399%); 22 MiB/s, 348 objects/s recovering 2024-03-11T21:33:32.517 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:33:32 smithi092 ceph-mon[37563]: osdmap e198: 8 total, 8 up, 8 in 2024-03-11T21:33:32.616 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:33:32.646 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:33:32.647 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:33:32.647 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:33:32.748 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:33:32.776 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:33:32.776 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:33:32.777 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:33:32.826 INFO:journalctl@ceph.iscsi.iscsi.a.smithi097.stdout:Mar 11 21:33:32 smithi097 ceph-00db0016-dfed-11ee-95c8-87774f69a715-iscsi-iscsi-a[60943]: debug there is no tcmu-runner data available 2024-03-11T21:33:32.826 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:33:32 smithi097 ceph-mon[33317]: pgmap v328: 158 pgs: 2 active+remapped+backfill_wait, 2 active+remapped+backfill_toofull, 5 active+remapped+backfilling, 149 active+clean; 7.1 GiB data, 25 GiB used, 691 GiB / 715 GiB avail; 852 B/s rd, 52 MiB/s wr, 827 op/s; 43601/351660 objects misplaced (12.399%); 22 MiB/s, 348 objects/s recovering 2024-03-11T21:33:32.826 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:33:32 smithi097 ceph-mon[33317]: osdmap e198: 8 total, 8 up, 8 in 2024-03-11T21:33:32.878 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:33:32.904 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:33:32.905 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:33:32.906 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:33:33.007 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:33:33.049 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:33:33.050 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:33:33.050 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:33:33.151 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:33:33.200 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:33:33.201 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:33:33.201 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:33:33.302 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:33:33.327 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:33:33.328 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:33:33.328 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:33:33.410 INFO:teuthology.orchestra.run.smithi092.stderr:reweighted osd.2 to 1 (10000) 2024-03-11T21:33:33.429 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:33:33.459 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:33:33.460 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:33:33.460 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:33:33.561 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:33:33.587 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:33:33.587 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:33:33.588 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:33:33.610 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:33:33 smithi092 ceph-mon[29572]: osdmap e199: 8 total, 8 up, 8 in 2024-03-11T21:33:33.610 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:33:33 smithi092 ceph-mon[29572]: from='client.24469 v1:172.21.15.97:0/404208310' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:33:33.610 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:33:33 smithi092 ceph-mon[29572]: from='client.? v1:172.21.15.92:0/936052954' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]: dispatch 2024-03-11T21:33:33.610 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:33:33 smithi092 ceph-mon[29572]: from='client.35105 ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]: dispatch 2024-03-11T21:33:33.610 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:33:33 smithi092 ceph-mon[37563]: osdmap e199: 8 total, 8 up, 8 in 2024-03-11T21:33:33.611 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:33:33 smithi092 ceph-mon[37563]: from='client.24469 v1:172.21.15.97:0/404208310' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:33:33.611 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:33:33 smithi092 ceph-mon[37563]: from='client.? v1:172.21.15.92:0/936052954' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]: dispatch 2024-03-11T21:33:33.611 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:33:33 smithi092 ceph-mon[37563]: from='client.35105 ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]: dispatch 2024-03-11T21:33:33.688 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:33:33.730 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:33:33.731 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:33:33.731 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:33:33.826 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:33:33 smithi097 ceph-mon[33317]: osdmap e199: 8 total, 8 up, 8 in 2024-03-11T21:33:33.826 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:33:33 smithi097 ceph-mon[33317]: from='client.24469 v1:172.21.15.97:0/404208310' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:33:33.826 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:33:33 smithi097 ceph-mon[33317]: from='client.? v1:172.21.15.92:0/936052954' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]: dispatch 2024-03-11T21:33:33.826 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:33:33 smithi097 ceph-mon[33317]: from='client.35105 ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]: dispatch 2024-03-11T21:33:33.833 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:33:33.861 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:33:33.861 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:33:33.862 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:33:33.962 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:33:34.004 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:33:34.005 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:33:34.005 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:33:34.107 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:33:34.139 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:33:34.140 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:33:34.140 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:33:34.242 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:33:34.268 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:33:34.269 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:33:34.269 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:33:34.370 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:33:34.399 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:33:34.400 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:33:34.401 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:33:34.501 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:33:34.530 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:33:34.531 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:33:34.531 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:33:34.610 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:33:34 smithi092 ceph-mon[29572]: pgmap v331: 158 pgs: 2 peering, 2 active+remapped+backfill_toofull, 4 active+remapped+backfilling, 150 active+clean; 7.3 GiB data, 26 GiB used, 689 GiB / 715 GiB avail; 1.2 KiB/s rd, 54 MiB/s wr, 904 op/s; 37256/361530 objects misplaced (10.305%); 58 MiB/s, 924 objects/s recovering 2024-03-11T21:33:34.610 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:33:34 smithi092 ceph-mon[29572]: from='client.35105 ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]': finished 2024-03-11T21:33:34.610 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:33:34 smithi092 ceph-mon[29572]: osdmap e200: 8 total, 8 up, 8 in 2024-03-11T21:33:34.610 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:33:34 smithi092 ceph-mon[37563]: pgmap v331: 158 pgs: 2 peering, 2 active+remapped+backfill_toofull, 4 active+remapped+backfilling, 150 active+clean; 7.3 GiB data, 26 GiB used, 689 GiB / 715 GiB avail; 1.2 KiB/s rd, 54 MiB/s wr, 904 op/s; 37256/361530 objects misplaced (10.305%); 58 MiB/s, 924 objects/s recovering 2024-03-11T21:33:34.611 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:33:34 smithi092 ceph-mon[37563]: from='client.35105 ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]': finished 2024-03-11T21:33:34.611 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:33:34 smithi092 ceph-mon[37563]: osdmap e200: 8 total, 8 up, 8 in 2024-03-11T21:33:34.632 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:33:34.665 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:33:34.666 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:33:34.666 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:33:34.767 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:33:34.796 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:33:34.796 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:33:34.796 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:33:34.826 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:33:34 smithi097 ceph-mon[33317]: pgmap v331: 158 pgs: 2 peering, 2 active+remapped+backfill_toofull, 4 active+remapped+backfilling, 150 active+clean; 7.3 GiB data, 26 GiB used, 689 GiB / 715 GiB avail; 1.2 KiB/s rd, 54 MiB/s wr, 904 op/s; 37256/361530 objects misplaced (10.305%); 58 MiB/s, 924 objects/s recovering 2024-03-11T21:33:34.826 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:33:34 smithi097 ceph-mon[33317]: from='client.35105 ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]': finished 2024-03-11T21:33:34.826 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:33:34 smithi097 ceph-mon[33317]: osdmap e200: 8 total, 8 up, 8 in 2024-03-11T21:33:34.898 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:33:34.928 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:33:34.929 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:33:34.929 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:33:35.030 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:33:35.063 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:33:35.064 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:33:35.064 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:33:35.164 INFO:teuthology.orchestra.run.smithi092.stdout:ERROR: (22) Invalid argument 2024-03-11T21:33:35.164 INFO:teuthology.orchestra.run.smithi092.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-11T21:33:35.164 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:33:35.195 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:33:35.196 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:33:35.196 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:33:35.297 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:33:35.340 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:33:35.341 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:33:35.341 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:33:35.442 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:33:35.471 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:33:35.471 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:33:35.472 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:33:35.573 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:33:35.612 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:33:35.613 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:33:35.613 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:33:35.714 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:33:35.746 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:33:35.746 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:33:35.747 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:33:35.848 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:33:35.889 DEBUG:teuthology.orchestra.run.smithi092:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid 00db0016-dfed-11ee-95c8-87774f69a715 -- ceph daemon osd.0 dump_blocked_ops 2024-03-11T21:33:35.890 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:33:35.891 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:33:35.891 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:33:35.993 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:33:36.020 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:33:36.021 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:33:36.021 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:33:36.122 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:33:36.148 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:33:36.149 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:33:36.150 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:33:36.250 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:33:36.303 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:33:36.304 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:33:36.304 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:33:36.405 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:33:36.445 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:33:36.446 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:33:36.447 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:33:36.547 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:33:36.589 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:33:36.590 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:33:36.590 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:33:36.610 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:33:36 smithi092 ceph-mon[29572]: pgmap v333: 158 pgs: 2 peering, 2 active+remapped+backfill_toofull, 4 active+remapped+backfilling, 150 active+clean; 7.3 GiB data, 26 GiB used, 689 GiB / 715 GiB avail; 1.0 KiB/s rd, 45 MiB/s wr, 759 op/s; 37256/361530 objects misplaced (10.305%); 48 MiB/s, 775 objects/s recovering 2024-03-11T21:33:36.610 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:33:36 smithi092 ceph-mon[37563]: pgmap v333: 158 pgs: 2 peering, 2 active+remapped+backfill_toofull, 4 active+remapped+backfilling, 150 active+clean; 7.3 GiB data, 26 GiB used, 689 GiB / 715 GiB avail; 1.0 KiB/s rd, 45 MiB/s wr, 759 op/s; 37256/361530 objects misplaced (10.305%); 48 MiB/s, 775 objects/s recovering 2024-03-11T21:33:36.643 INFO:teuthology.orchestra.run.smithi092.stderr:Inferring config /var/lib/ceph/00db0016-dfed-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T21:33:36.691 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:33:36.692 DEBUG:teuthology.orchestra.run.smithi092:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid 00db0016-dfed-11ee-95c8-87774f69a715 -- ceph osd reweight 3 1 2024-03-11T21:33:36.777 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:33:36.778 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:33:36.778 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:33:36.826 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:33:36 smithi097 ceph-mon[33317]: pgmap v333: 158 pgs: 2 peering, 2 active+remapped+backfill_toofull, 4 active+remapped+backfilling, 150 active+clean; 7.3 GiB data, 26 GiB used, 689 GiB / 715 GiB avail; 1.0 KiB/s rd, 45 MiB/s wr, 759 op/s; 37256/361530 objects misplaced (10.305%); 48 MiB/s, 775 objects/s recovering 2024-03-11T21:33:36.879 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:33:36.908 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:33:36.909 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:33:36.909 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:33:37.010 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:33:37.037 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:33:37.037 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:33:37.038 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:33:37.138 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:33:37.169 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:33:37.170 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:33:37.170 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:33:37.271 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:33:37.315 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:33:37.315 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:33:37.316 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:33:37.416 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:33:37.461 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:33:37.461 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:33:37.462 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:33:37.563 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:33:37.589 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:33:37.589 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:33:37.590 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:33:37.690 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:33:37.721 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:33:37.721 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:33:37.722 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:33:37.788 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:33:37 smithi092 ceph-mon[29572]: from='mgr.14607 v1:172.21.15.92:0/1702717597' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T21:33:37.789 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:33:37 smithi092 ceph-mon[37563]: from='mgr.14607 v1:172.21.15.92:0/1702717597' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T21:33:37.822 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:33:37.826 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:33:37 smithi097 ceph-mon[33317]: from='mgr.14607 v1:172.21.15.92:0/1702717597' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T21:33:37.866 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:33:37.866 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:33:37.867 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:33:37.967 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:33:37.994 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:33:37.995 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:33:37.995 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:33:38.096 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:33:38.110 INFO:journalctl@ceph.mgr.y.smithi092.stdout:Mar 11 21:33:37 smithi092 ceph-00db0016-dfed-11ee-95c8-87774f69a715-mgr-y[29801]: ::ffff:172.21.15.97 - - [11/Mar/2024:21:33:37] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-11T21:33:38.121 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:33:38.122 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:33:38.122 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:33:38.223 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:33:38.263 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:33:38.263 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:33:38.264 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:33:38.364 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:33:38.396 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:33:38.397 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:33:38.397 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:33:38.498 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:33:38.522 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:33:38.522 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:33:38.523 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:33:38.623 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:33:38.654 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:33:38.654 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:33:38.655 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:33:38.755 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:33:38.778 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:33:38.779 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:33:38.779 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:33:38.826 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:33:38 smithi097 ceph-mon[33317]: pgmap v334: 158 pgs: 1 peering, 2 active+remapped+backfill_toofull, 4 active+remapped+backfilling, 151 active+clean; 7.3 GiB data, 25 GiB used, 690 GiB / 715 GiB avail; 1023 B/s rd, 36 MiB/s wr, 834 op/s; 36942/357384 objects misplaced (10.337%); 39 MiB/s, 616 objects/s recovering 2024-03-11T21:33:38.826 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:33:38 smithi097 ceph-mon[33317]: osdmap e201: 8 total, 8 up, 8 in 2024-03-11T21:33:38.826 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:33:38 smithi097 ceph-mon[33317]: from='client.? v1:172.21.15.92:0/514700168' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-11T21:33:38.826 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:33:38 smithi097 ceph-mon[33317]: from='client.35111 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-11T21:33:38.860 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:33:38 smithi092 ceph-mon[29572]: pgmap v334: 158 pgs: 1 peering, 2 active+remapped+backfill_toofull, 4 active+remapped+backfilling, 151 active+clean; 7.3 GiB data, 25 GiB used, 690 GiB / 715 GiB avail; 1023 B/s rd, 36 MiB/s wr, 834 op/s; 36942/357384 objects misplaced (10.337%); 39 MiB/s, 616 objects/s recovering 2024-03-11T21:33:38.860 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:33:38 smithi092 ceph-mon[29572]: osdmap e201: 8 total, 8 up, 8 in 2024-03-11T21:33:38.860 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:33:38 smithi092 ceph-mon[29572]: from='client.? v1:172.21.15.92:0/514700168' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-11T21:33:38.860 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:33:38 smithi092 ceph-mon[29572]: from='client.35111 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-11T21:33:38.860 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:33:38 smithi092 ceph-mon[37563]: pgmap v334: 158 pgs: 1 peering, 2 active+remapped+backfill_toofull, 4 active+remapped+backfilling, 151 active+clean; 7.3 GiB data, 25 GiB used, 690 GiB / 715 GiB avail; 1023 B/s rd, 36 MiB/s wr, 834 op/s; 36942/357384 objects misplaced (10.337%); 39 MiB/s, 616 objects/s recovering 2024-03-11T21:33:38.861 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:33:38 smithi092 ceph-mon[37563]: osdmap e201: 8 total, 8 up, 8 in 2024-03-11T21:33:38.861 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:33:38 smithi092 ceph-mon[37563]: from='client.? v1:172.21.15.92:0/514700168' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-11T21:33:38.861 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:33:38 smithi092 ceph-mon[37563]: from='client.35111 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-11T21:33:38.880 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:33:38.911 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:33:38.912 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:33:38.912 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:33:39.013 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:33:39.045 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:33:39.045 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:33:39.046 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:33:39.147 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:33:39.182 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:33:39.182 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:33:39.183 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:33:39.232 INFO:teuthology.orchestra.run.smithi092.stderr:nodeep-scrub is set 2024-03-11T21:33:39.284 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:33:39.314 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:33:39.314 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:33:39.315 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:33:39.415 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:33:39.449 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:33:39.450 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:33:39.450 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:33:39.551 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:33:39.591 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:33:39.591 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:33:39.592 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:33:39.610 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:33:39 smithi092 ceph-mon[29572]: purged_snaps scrub starts 2024-03-11T21:33:39.610 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:33:39 smithi092 ceph-mon[29572]: purged_snaps scrub ok 2024-03-11T21:33:39.610 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:33:39 smithi092 ceph-mon[29572]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-11T21:33:39.610 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:33:39 smithi092 ceph-mon[29572]: from='client.35111 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-03-11T21:33:39.610 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:33:39 smithi092 ceph-mon[29572]: osdmap e202: 8 total, 8 up, 8 in 2024-03-11T21:33:39.610 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:33:39 smithi092 ceph-mon[37563]: purged_snaps scrub starts 2024-03-11T21:33:39.611 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:33:39 smithi092 ceph-mon[37563]: purged_snaps scrub ok 2024-03-11T21:33:39.611 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:33:39 smithi092 ceph-mon[37563]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-11T21:33:39.611 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:33:39 smithi092 ceph-mon[37563]: from='client.35111 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-03-11T21:33:39.611 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:33:39 smithi092 ceph-mon[37563]: osdmap e202: 8 total, 8 up, 8 in 2024-03-11T21:33:39.692 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:33:39.730 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:33:39.730 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:33:39.731 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:33:39.826 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:33:39 smithi097 ceph-mon[33317]: purged_snaps scrub starts 2024-03-11T21:33:39.826 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:33:39 smithi097 ceph-mon[33317]: purged_snaps scrub ok 2024-03-11T21:33:39.826 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:33:39 smithi097 ceph-mon[33317]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-11T21:33:39.826 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:33:39 smithi097 ceph-mon[33317]: from='client.35111 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-03-11T21:33:39.826 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:33:39 smithi097 ceph-mon[33317]: osdmap e202: 8 total, 8 up, 8 in 2024-03-11T21:33:39.832 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:33:39.859 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:33:39.860 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:33:39.860 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:33:39.961 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:33:39.991 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:33:39.992 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:33:39.992 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:33:40.093 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:33:40.124 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:33:40.125 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:33:40.125 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:33:40.226 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:33:40.266 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:33:40.267 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:33:40.268 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:33:40.369 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:33:40.392 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:33:40.393 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:33:40.393 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:33:40.494 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:33:40.531 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:33:40.531 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:33:40.532 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:33:40.610 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:33:40 smithi092 ceph-mon[29572]: pgmap v336: 158 pgs: 2 active+remapped+backfill_toofull, 4 active+remapped+backfilling, 152 active+clean; 7.0 GiB data, 25 GiB used, 690 GiB / 715 GiB avail; 911 B/s rd, 12 MiB/s wr, 959 op/s; 32565/346596 objects misplaced (9.396%); 31 MiB/s, 501 objects/s recovering 2024-03-11T21:33:40.610 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:33:40 smithi092 ceph-mon[37563]: pgmap v336: 158 pgs: 2 active+remapped+backfill_toofull, 4 active+remapped+backfilling, 152 active+clean; 7.0 GiB data, 25 GiB used, 690 GiB / 715 GiB avail; 911 B/s rd, 12 MiB/s wr, 959 op/s; 32565/346596 objects misplaced (9.396%); 31 MiB/s, 501 objects/s recovering 2024-03-11T21:33:40.633 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:33:40.664 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:33:40.665 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:33:40.665 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:33:40.765 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:33:40.795 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:33:40.796 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:33:40.796 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:33:40.826 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:33:40 smithi097 ceph-mon[33317]: pgmap v336: 158 pgs: 2 active+remapped+backfill_toofull, 4 active+remapped+backfilling, 152 active+clean; 7.0 GiB data, 25 GiB used, 690 GiB / 715 GiB avail; 911 B/s rd, 12 MiB/s wr, 959 op/s; 32565/346596 objects misplaced (9.396%); 31 MiB/s, 501 objects/s recovering 2024-03-11T21:33:40.898 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:33:40.937 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:33:40.938 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:33:40.938 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:33:41.039 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:33:41.067 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:33:41.067 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:33:41.068 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:33:41.168 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:33:41.199 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:33:41.200 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:33:41.200 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:33:41.301 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:33:41.335 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:33:41.336 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:33:41.336 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:33:41.438 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:33:41.471 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:33:41.472 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:33:41.472 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:33:41.573 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:33:41.609 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:33:41.610 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:33:41.610 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:33:41.711 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:33:41.755 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:33:41.756 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:33:41.757 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:33:41.858 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:33:41.890 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:33:41.891 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:33:41.891 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:33:41.992 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:33:42.017 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:33:42.018 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:33:42.019 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:33:42.120 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:33:42.158 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:33:42.158 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:33:42.159 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:33:42.259 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:33:42.301 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:33:42.301 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:33:42.302 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:33:42.403 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:33:42.442 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:33:42.442 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:33:42.443 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:33:42.544 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:33:42.575 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:33:42.575 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:33:42.576 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:33:42.676 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:33:42.707 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:33:42.707 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:33:42.708 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:33:42.808 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:33:42.826 INFO:journalctl@ceph.iscsi.iscsi.a.smithi097.stdout:Mar 11 21:33:42 smithi097 ceph-00db0016-dfed-11ee-95c8-87774f69a715-iscsi-iscsi-a[60943]: debug there is no tcmu-runner data available 2024-03-11T21:33:42.826 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:33:42 smithi097 ceph-mon[33317]: pgmap v338: 158 pgs: 2 active+remapped+backfill_toofull, 4 active+remapped+backfilling, 152 active+clean; 7.0 GiB data, 25 GiB used, 690 GiB / 715 GiB avail; 796 B/s rd, 8.4 KiB/s wr, 646 op/s; 32565/346596 objects misplaced (9.396%); 12 MiB/s, 190 objects/s recovering 2024-03-11T21:33:42.832 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:33:42.833 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:33:42.833 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:33:42.860 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:33:42 smithi092 ceph-mon[29572]: pgmap v338: 158 pgs: 2 active+remapped+backfill_toofull, 4 active+remapped+backfilling, 152 active+clean; 7.0 GiB data, 25 GiB used, 690 GiB / 715 GiB avail; 796 B/s rd, 8.4 KiB/s wr, 646 op/s; 32565/346596 objects misplaced (9.396%); 12 MiB/s, 190 objects/s recovering 2024-03-11T21:33:42.860 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:33:42 smithi092 ceph-mon[37563]: pgmap v338: 158 pgs: 2 active+remapped+backfill_toofull, 4 active+remapped+backfilling, 152 active+clean; 7.0 GiB data, 25 GiB used, 690 GiB / 715 GiB avail; 796 B/s rd, 8.4 KiB/s wr, 646 op/s; 32565/346596 objects misplaced (9.396%); 12 MiB/s, 190 objects/s recovering 2024-03-11T21:33:42.934 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:33:42.967 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:33:42.968 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:33:42.968 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:33:43.069 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:33:43.093 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:33:43.094 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:33:43.094 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:33:43.195 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:33:43.239 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:33:43.239 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:33:43.240 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:33:43.341 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:33:43.371 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:33:43.371 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:33:43.372 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:33:43.472 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:33:43.507 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:33:43.508 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:33:43.508 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:33:43.609 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:33:43.641 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:33:43.641 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:33:43.642 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:33:43.743 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:33:43.774 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:33:43.774 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:33:43.775 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:33:43.826 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:33:43 smithi097 ceph-mon[33317]: from='client.24469 v1:172.21.15.97:0/404208310' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:33:43.826 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:33:43 smithi097 ceph-mon[33317]: osdmap e203: 8 total, 8 up, 8 in 2024-03-11T21:33:43.860 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:33:43 smithi092 ceph-mon[29572]: from='client.24469 v1:172.21.15.97:0/404208310' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:33:43.860 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:33:43 smithi092 ceph-mon[29572]: osdmap e203: 8 total, 8 up, 8 in 2024-03-11T21:33:43.860 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:33:43 smithi092 ceph-mon[37563]: from='client.24469 v1:172.21.15.97:0/404208310' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:33:43.860 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:33:43 smithi092 ceph-mon[37563]: osdmap e203: 8 total, 8 up, 8 in 2024-03-11T21:33:43.876 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:33:43.903 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:33:43.904 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:33:43.904 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:33:44.005 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:33:44.048 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:33:44.048 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:33:44.049 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:33:44.150 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:33:44.185 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:33:44.186 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:33:44.186 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:33:44.287 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:33:44.316 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:33:44.317 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:33:44.317 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:33:44.418 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:33:44.442 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:33:44.442 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:33:44.443 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:33:44.543 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:33:44.591 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:33:44.592 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:33:44.592 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:33:44.610 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:33:44 smithi092 ceph-mon[29572]: pgmap v339: 158 pgs: 2 active+remapped+backfill_toofull, 4 active+remapped+backfilling, 152 active+clean; 6.8 GiB data, 24 GiB used, 691 GiB / 715 GiB avail; 1.4 KiB/s rd, 8.1 KiB/s wr, 1.07k op/s; 31403/335895 objects misplaced (9.349%); 14 MiB/s, 221 objects/s recovering 2024-03-11T21:33:44.610 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:33:44 smithi092 ceph-mon[37563]: pgmap v339: 158 pgs: 2 active+remapped+backfill_toofull, 4 active+remapped+backfilling, 152 active+clean; 6.8 GiB data, 24 GiB used, 691 GiB / 715 GiB avail; 1.4 KiB/s rd, 8.1 KiB/s wr, 1.07k op/s; 31403/335895 objects misplaced (9.349%); 14 MiB/s, 221 objects/s recovering 2024-03-11T21:33:44.693 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:33:44.736 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:33:44.737 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:33:44.737 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:33:44.826 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:33:44 smithi097 ceph-mon[33317]: pgmap v339: 158 pgs: 2 active+remapped+backfill_toofull, 4 active+remapped+backfilling, 152 active+clean; 6.8 GiB data, 24 GiB used, 691 GiB / 715 GiB avail; 1.4 KiB/s rd, 8.1 KiB/s wr, 1.07k op/s; 31403/335895 objects misplaced (9.349%); 14 MiB/s, 221 objects/s recovering 2024-03-11T21:33:44.838 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:33:44.868 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:33:44.869 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:33:44.869 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:33:44.970 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:33:44.999 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:33:45.000 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:33:45.000 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:33:45.101 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:33:45.137 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:33:45.138 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:33:45.138 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:33:45.239 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:33:45.269 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:33:45.269 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:33:45.270 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:33:45.371 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:33:45.403 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:33:45.404 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:33:45.404 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:33:45.505 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:33:45.531 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:33:45.532 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:33:45.532 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:33:45.633 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:33:45.674 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:33:45.674 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:33:45.675 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:33:45.775 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:33:45.807 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:33:45.808 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:33:45.808 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:33:45.908 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:33:45.937 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:33:45.937 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:33:45.938 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:33:45.972 INFO:teuthology.orchestra.run.smithi092.stderr:Inferring config /var/lib/ceph/00db0016-dfed-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T21:33:46.038 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:33:46.064 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:33:46.065 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:33:46.065 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:33:46.166 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:33:46.193 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:33:46.193 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:33:46.194 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:33:46.294 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:33:46.325 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:33:46.326 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:33:46.326 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:33:46.427 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:33:46.451 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:33:46.452 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:33:46.452 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:33:46.524 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:33:46 smithi092 ceph-mon[29572]: pgmap v341: 158 pgs: 2 active+remapped+backfill_toofull, 4 active+remapped+backfilling, 152 active+clean; 6.7 GiB data, 24 GiB used, 691 GiB / 715 GiB avail; 743 B/s rd, 0 B/s wr, 715 op/s; 28586/331845 objects misplaced (8.614%); 11 MiB/s, 176 objects/s recovering 2024-03-11T21:33:46.524 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:33:46 smithi092 ceph-mon[29572]: osdmap e204: 8 total, 8 up, 8 in 2024-03-11T21:33:46.524 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:33:46 smithi092 ceph-mon[29572]: purged_snaps scrub starts 2024-03-11T21:33:46.524 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:33:46 smithi092 ceph-mon[29572]: purged_snaps scrub ok 2024-03-11T21:33:46.524 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:33:46 smithi092 ceph-mon[37563]: pgmap v341: 158 pgs: 2 active+remapped+backfill_toofull, 4 active+remapped+backfilling, 152 active+clean; 6.7 GiB data, 24 GiB used, 691 GiB / 715 GiB avail; 743 B/s rd, 0 B/s wr, 715 op/s; 28586/331845 objects misplaced (8.614%); 11 MiB/s, 176 objects/s recovering 2024-03-11T21:33:46.525 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:33:46 smithi092 ceph-mon[37563]: osdmap e204: 8 total, 8 up, 8 in 2024-03-11T21:33:46.525 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:33:46 smithi092 ceph-mon[37563]: purged_snaps scrub starts 2024-03-11T21:33:46.525 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:33:46 smithi092 ceph-mon[37563]: purged_snaps scrub ok 2024-03-11T21:33:46.553 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:33:46.585 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:33:46.585 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:33:46.586 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:33:46.687 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:33:46.716 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:33:46.717 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:33:46.717 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:33:46.818 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:33:46.826 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:33:46 smithi097 ceph-mon[33317]: pgmap v341: 158 pgs: 2 active+remapped+backfill_toofull, 4 active+remapped+backfilling, 152 active+clean; 6.7 GiB data, 24 GiB used, 691 GiB / 715 GiB avail; 743 B/s rd, 0 B/s wr, 715 op/s; 28586/331845 objects misplaced (8.614%); 11 MiB/s, 176 objects/s recovering 2024-03-11T21:33:46.826 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:33:46 smithi097 ceph-mon[33317]: osdmap e204: 8 total, 8 up, 8 in 2024-03-11T21:33:46.826 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:33:46 smithi097 ceph-mon[33317]: purged_snaps scrub starts 2024-03-11T21:33:46.826 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:33:46 smithi097 ceph-mon[33317]: purged_snaps scrub ok 2024-03-11T21:33:46.847 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:33:46.847 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:33:46.848 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:33:46.949 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:33:46.992 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:33:46.993 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:33:46.993 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:33:47.094 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:33:47.138 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:33:47.139 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:33:47.139 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:33:47.223 DEBUG:teuthology.orchestra.run.smithi092:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid 00db0016-dfed-11ee-95c8-87774f69a715 -- ceph osd unset noscrub 2024-03-11T21:33:47.240 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:33:47.271 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:33:47.271 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:33:47.271 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:33:47.372 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:33:47.400 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:33:47.400 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:33:47.401 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:33:47.502 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:33:47.550 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:33:47.551 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:33:47.551 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:33:47.610 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:33:47 smithi092 ceph-mon[29572]: osdmap e205: 8 total, 8 up, 8 in 2024-03-11T21:33:47.610 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:33:47 smithi092 ceph-mon[37563]: osdmap e205: 8 total, 8 up, 8 in 2024-03-11T21:33:47.652 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:33:47.682 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:33:47.682 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:33:47.682 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:33:47.783 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:33:47.813 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:33:47.813 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:33:47.814 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:33:47.826 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:33:47 smithi097 ceph-mon[33317]: osdmap e205: 8 total, 8 up, 8 in 2024-03-11T21:33:47.915 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:33:47.966 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:33:47.967 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:33:47.967 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:33:47.990 INFO:teuthology.orchestra.run.smithi092.stdout:osd.0: {} 2024-03-11T21:33:48.042 INFO:teuthology.orchestra.run.smithi092.stdout:osd.1: {} 2024-03-11T21:33:48.042 INFO:journalctl@ceph.mgr.y.smithi092.stdout:Mar 11 21:33:47 smithi092 ceph-00db0016-dfed-11ee-95c8-87774f69a715-mgr-y[29801]: ::ffff:172.21.15.97 - - [11/Mar/2024:21:33:47] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-11T21:33:48.069 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:33:48.084 INFO:teuthology.orchestra.run.smithi092.stdout:osd.2: {} 2024-03-11T21:33:48.102 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:33:48.102 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:33:48.103 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:33:48.128 INFO:teuthology.orchestra.run.smithi092.stdout:osd.3: {} 2024-03-11T21:33:48.169 INFO:teuthology.orchestra.run.smithi092.stdout:osd.4: {} 2024-03-11T21:33:48.204 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:33:48.208 INFO:teuthology.orchestra.run.smithi092.stdout:osd.5: {} 2024-03-11T21:33:48.250 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:33:48.251 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:33:48.251 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:33:48.251 INFO:teuthology.orchestra.run.smithi092.stdout:osd.6: {} 2024-03-11T21:33:48.289 INFO:teuthology.orchestra.run.smithi092.stdout:osd.7: {} 2024-03-11T21:33:48.296 INFO:teuthology.orchestra.run.smithi092.stderr:osd.0: osd_enable_op_tracker = '' 2024-03-11T21:33:48.296 INFO:teuthology.orchestra.run.smithi092.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-11T21:33:48.296 INFO:teuthology.orchestra.run.smithi092.stderr:osd.2: osd_enable_op_tracker = '' 2024-03-11T21:33:48.296 INFO:teuthology.orchestra.run.smithi092.stderr:osd.3: osd_enable_op_tracker = '' 2024-03-11T21:33:48.296 INFO:teuthology.orchestra.run.smithi092.stderr:osd.4: osd_enable_op_tracker = '' 2024-03-11T21:33:48.296 INFO:teuthology.orchestra.run.smithi092.stderr:osd.5: osd_enable_op_tracker = '' 2024-03-11T21:33:48.296 INFO:teuthology.orchestra.run.smithi092.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-11T21:33:48.297 INFO:teuthology.orchestra.run.smithi092.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-11T21:33:48.352 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:33:48.395 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:33:48.396 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:33:48.396 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:33:48.497 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:33:48.543 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:33:48.544 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:33:48.545 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:33:48.610 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:33:48 smithi092 ceph-mon[37563]: pgmap v344: 158 pgs: 2 active+remapped+backfill_toofull, 4 active+remapped+backfilling, 152 active+clean; 6.6 GiB data, 24 GiB used, 691 GiB / 715 GiB avail; 853 B/s rd, 0 B/s wr, 1.06k op/s; 28264/327435 objects misplaced (8.632%); 13 MiB/s, 202 objects/s recovering 2024-03-11T21:33:48.610 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:33:48 smithi092 ceph-mon[37563]: from='client.? v1:172.21.15.92:0/1360993792' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-11T21:33:48.610 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:33:48 smithi092 ceph-mon[29572]: pgmap v344: 158 pgs: 2 active+remapped+backfill_toofull, 4 active+remapped+backfilling, 152 active+clean; 6.6 GiB data, 24 GiB used, 691 GiB / 715 GiB avail; 853 B/s rd, 0 B/s wr, 1.06k op/s; 28264/327435 objects misplaced (8.632%); 13 MiB/s, 202 objects/s recovering 2024-03-11T21:33:48.610 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:33:48 smithi092 ceph-mon[29572]: from='client.? v1:172.21.15.92:0/1360993792' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-11T21:33:48.646 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:33:48.675 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:33:48.676 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:33:48.676 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:33:48.771 INFO:teuthology.orchestra.run.smithi092.stderr:Inferring config /var/lib/ceph/00db0016-dfed-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T21:33:48.777 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:33:48.826 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:33:48 smithi097 ceph-mon[33317]: pgmap v344: 158 pgs: 2 active+remapped+backfill_toofull, 4 active+remapped+backfilling, 152 active+clean; 6.6 GiB data, 24 GiB used, 691 GiB / 715 GiB avail; 853 B/s rd, 0 B/s wr, 1.06k op/s; 28264/327435 objects misplaced (8.632%); 13 MiB/s, 202 objects/s recovering 2024-03-11T21:33:48.826 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:33:48 smithi097 ceph-mon[33317]: from='client.? v1:172.21.15.92:0/1360993792' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-11T21:33:48.885 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:33:48.885 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:33:48.886 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:33:48.987 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:33:49.019 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:33:49.019 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:33:49.019 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:33:49.120 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:33:49.161 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:33:49.161 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:33:49.162 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:33:49.263 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:33:49.290 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:33:49.291 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:33:49.291 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:33:49.392 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:33:49.443 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:33:49.444 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:33:49.445 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:33:49.545 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:33:49.577 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:33:49.577 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:33:49.578 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:33:49.678 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:33:49.695 INFO:teuthology.orchestra.run.smithi092.stderr:Inferring config /var/lib/ceph/00db0016-dfed-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T21:33:49.730 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:33:49.731 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:33:49.731 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:33:49.832 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:33:49.878 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:33:49.879 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:33:49.879 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:33:49.980 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:33:50.010 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:33:50.011 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:33:50.011 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:33:50.112 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:33:50.141 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:33:50.141 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:33:50.142 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:33:50.242 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:33:50.269 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:33:50.270 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:33:50.270 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:33:50.371 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:33:50.400 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:33:50.400 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:33:50.400 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:33:50.501 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:33:50.527 INFO:teuthology.orchestra.run.smithi092.stderr:reweighted osd.3 to 1 (10000) 2024-03-11T21:33:50.545 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:33:50.546 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:33:50.546 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:33:50.647 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:33:50.677 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:33:50.678 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:33:50.678 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:33:50.779 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:33:50.810 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:33:50.811 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:33:50.811 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:33:50.826 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:33:50 smithi097 ceph-mon[33317]: purged_snaps scrub starts 2024-03-11T21:33:50.826 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:33:50 smithi097 ceph-mon[33317]: purged_snaps scrub ok 2024-03-11T21:33:50.826 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:33:50 smithi097 ceph-mon[33317]: pgmap v345: 158 pgs: 2 active+remapped+backfill_toofull, 3 active+remapped+backfilling, 153 active+clean; 6.4 GiB data, 23 GiB used, 692 GiB / 715 GiB avail; 852 B/s rd, 0 B/s wr, 1.04k op/s; 23584/317076 objects misplaced (7.438%); 27 MiB/s, 428 objects/s recovering 2024-03-11T21:33:50.826 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:33:50 smithi097 ceph-mon[33317]: osdmap e206: 8 total, 8 up, 8 in 2024-03-11T21:33:50.826 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:33:50 smithi097 ceph-mon[33317]: purged_snaps scrub starts 2024-03-11T21:33:50.827 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:33:50 smithi097 ceph-mon[33317]: purged_snaps scrub ok 2024-03-11T21:33:50.827 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:33:50 smithi097 ceph-mon[33317]: from='client.? v1:172.21.15.92:0/3983396143' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]: dispatch 2024-03-11T21:33:50.827 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:33:50 smithi097 ceph-mon[33317]: from='client.25147 ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]: dispatch 2024-03-11T21:33:50.860 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:33:50 smithi092 ceph-mon[29572]: purged_snaps scrub starts 2024-03-11T21:33:50.860 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:33:50 smithi092 ceph-mon[29572]: purged_snaps scrub ok 2024-03-11T21:33:50.860 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:33:50 smithi092 ceph-mon[29572]: pgmap v345: 158 pgs: 2 active+remapped+backfill_toofull, 3 active+remapped+backfilling, 153 active+clean; 6.4 GiB data, 23 GiB used, 692 GiB / 715 GiB avail; 852 B/s rd, 0 B/s wr, 1.04k op/s; 23584/317076 objects misplaced (7.438%); 27 MiB/s, 428 objects/s recovering 2024-03-11T21:33:50.860 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:33:50 smithi092 ceph-mon[29572]: osdmap e206: 8 total, 8 up, 8 in 2024-03-11T21:33:50.861 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:33:50 smithi092 ceph-mon[29572]: purged_snaps scrub starts 2024-03-11T21:33:50.861 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:33:50 smithi092 ceph-mon[29572]: purged_snaps scrub ok 2024-03-11T21:33:50.861 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:33:50 smithi092 ceph-mon[29572]: from='client.? v1:172.21.15.92:0/3983396143' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]: dispatch 2024-03-11T21:33:50.861 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:33:50 smithi092 ceph-mon[29572]: from='client.25147 ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]: dispatch 2024-03-11T21:33:50.861 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:33:50 smithi092 ceph-mon[37563]: purged_snaps scrub starts 2024-03-11T21:33:50.861 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:33:50 smithi092 ceph-mon[37563]: purged_snaps scrub ok 2024-03-11T21:33:50.861 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:33:50 smithi092 ceph-mon[37563]: pgmap v345: 158 pgs: 2 active+remapped+backfill_toofull, 3 active+remapped+backfilling, 153 active+clean; 6.4 GiB data, 23 GiB used, 692 GiB / 715 GiB avail; 852 B/s rd, 0 B/s wr, 1.04k op/s; 23584/317076 objects misplaced (7.438%); 27 MiB/s, 428 objects/s recovering 2024-03-11T21:33:50.861 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:33:50 smithi092 ceph-mon[37563]: osdmap e206: 8 total, 8 up, 8 in 2024-03-11T21:33:50.861 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:33:50 smithi092 ceph-mon[37563]: purged_snaps scrub starts 2024-03-11T21:33:50.861 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:33:50 smithi092 ceph-mon[37563]: purged_snaps scrub ok 2024-03-11T21:33:50.861 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:33:50 smithi092 ceph-mon[37563]: from='client.? v1:172.21.15.92:0/3983396143' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]: dispatch 2024-03-11T21:33:50.862 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:33:50 smithi092 ceph-mon[37563]: from='client.25147 ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]: dispatch 2024-03-11T21:33:50.912 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:33:50.958 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:33:50.959 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:33:50.959 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:33:51.060 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:33:51.096 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:33:51.097 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:33:51.097 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:33:51.198 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:33:51.226 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:33:51.226 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:33:51.226 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:33:51.327 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:33:51.350 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:33:51.351 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:33:51.351 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:33:51.445 DEBUG:teuthology.orchestra.run.smithi092:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid 00db0016-dfed-11ee-95c8-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-03-11T21:33:51.452 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:33:51.482 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:33:51.483 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:33:51.484 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:33:51.584 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:33:51.618 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:33:51.618 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:33:51.618 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:33:51.719 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:33:51.749 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:33:51.750 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:33:51.750 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:33:51.826 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:33:51 smithi097 ceph-mon[33317]: from='client.25147 ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]': finished 2024-03-11T21:33:51.826 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:33:51 smithi097 ceph-mon[33317]: osdmap e207: 8 total, 8 up, 8 in 2024-03-11T21:33:51.852 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:33:51.860 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:33:51 smithi092 ceph-mon[37563]: from='client.25147 ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]': finished 2024-03-11T21:33:51.860 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:33:51 smithi092 ceph-mon[37563]: osdmap e207: 8 total, 8 up, 8 in 2024-03-11T21:33:51.863 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:33:51 smithi092 ceph-mon[29572]: from='client.25147 ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]': finished 2024-03-11T21:33:51.863 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:33:51 smithi092 ceph-mon[29572]: osdmap e207: 8 total, 8 up, 8 in 2024-03-11T21:33:51.898 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:33:51.899 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:33:51.899 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:33:52.000 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:33:52.031 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:33:52.031 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:33:52.031 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:33:52.132 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:33:52.172 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:33:52.172 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:33:52.172 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:33:52.273 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:33:52.315 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:33:52.315 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:33:52.316 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:33:52.416 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:33:52.459 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:33:52.460 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:33:52.460 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:33:52.543 INFO:journalctl@ceph.iscsi.iscsi.a.smithi097.stdout:Mar 11 21:33:52 smithi097 ceph-00db0016-dfed-11ee-95c8-87774f69a715-iscsi-iscsi-a[60943]: debug there is no tcmu-runner data available 2024-03-11T21:33:52.560 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:33:52.591 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:33:52.592 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:33:52.593 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:33:52.693 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:33:52.735 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:33:52.737 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:33:52.737 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:33:52.812 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:33:52 smithi092 ceph-mon[29572]: pgmap v348: 158 pgs: 2 active+remapped+backfill_toofull, 3 active+remapped+backfilling, 153 active+clean; 6.4 GiB data, 23 GiB used, 692 GiB / 715 GiB avail; 908 B/s rd, 0 B/s wr, 874 op/s; 23584/317076 objects misplaced (7.438%); 18 MiB/s, 294 objects/s recovering 2024-03-11T21:33:52.812 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:33:52 smithi092 ceph-mon[29572]: from='mgr.14607 v1:172.21.15.92:0/1702717597' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T21:33:52.812 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:33:52 smithi092 ceph-mon[37563]: pgmap v348: 158 pgs: 2 active+remapped+backfill_toofull, 3 active+remapped+backfilling, 153 active+clean; 6.4 GiB data, 23 GiB used, 692 GiB / 715 GiB avail; 908 B/s rd, 0 B/s wr, 874 op/s; 23584/317076 objects misplaced (7.438%); 18 MiB/s, 294 objects/s recovering 2024-03-11T21:33:52.812 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:33:52 smithi092 ceph-mon[37563]: from='mgr.14607 v1:172.21.15.92:0/1702717597' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T21:33:52.826 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:33:52 smithi097 ceph-mon[33317]: pgmap v348: 158 pgs: 2 active+remapped+backfill_toofull, 3 active+remapped+backfilling, 153 active+clean; 6.4 GiB data, 23 GiB used, 692 GiB / 715 GiB avail; 908 B/s rd, 0 B/s wr, 874 op/s; 23584/317076 objects misplaced (7.438%); 18 MiB/s, 294 objects/s recovering 2024-03-11T21:33:52.826 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:33:52 smithi097 ceph-mon[33317]: from='mgr.14607 v1:172.21.15.92:0/1702717597' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T21:33:52.838 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:33:52.861 DEBUG:teuthology.orchestra.run.smithi092:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid 00db0016-dfed-11ee-95c8-87774f69a715 -- ceph osd reweight 4 1 2024-03-11T21:33:52.865 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:33:52.867 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:33:52.867 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:33:52.968 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:33:52.993 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:33:52.994 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:33:52.994 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:33:53.095 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:33:53.120 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:33:53.121 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:33:53.121 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:33:53.222 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:33:53.249 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:33:53.250 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:33:53.250 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:33:53.351 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:33:53.402 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:33:53.403 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:33:53.403 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:33:53.504 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:33:53.547 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:33:53.548 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:33:53.548 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:33:53.649 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:33:53.687 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:33:53.688 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:33:53.688 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:33:53.789 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:33:53.812 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:33:53.813 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:33:53.813 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:33:53.826 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:33:53 smithi097 ceph-mon[33317]: from='client.24469 v1:172.21.15.97:0/404208310' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:33:53.826 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:33:53 smithi097 ceph-mon[33317]: osdmap e208: 8 total, 8 up, 8 in 2024-03-11T21:33:53.826 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:33:53 smithi097 ceph-mon[33317]: osdmap e209: 8 total, 8 up, 8 in 2024-03-11T21:33:53.860 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:33:53 smithi092 ceph-mon[29572]: from='client.24469 v1:172.21.15.97:0/404208310' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:33:53.860 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:33:53 smithi092 ceph-mon[29572]: osdmap e208: 8 total, 8 up, 8 in 2024-03-11T21:33:53.860 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:33:53 smithi092 ceph-mon[29572]: osdmap e209: 8 total, 8 up, 8 in 2024-03-11T21:33:53.860 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:33:53 smithi092 ceph-mon[37563]: from='client.24469 v1:172.21.15.97:0/404208310' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:33:53.860 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:33:53 smithi092 ceph-mon[37563]: osdmap e208: 8 total, 8 up, 8 in 2024-03-11T21:33:53.860 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:33:53 smithi092 ceph-mon[37563]: osdmap e209: 8 total, 8 up, 8 in 2024-03-11T21:33:53.915 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:33:53.947 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:33:53.948 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:33:53.948 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:33:54.049 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:33:54.078 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:33:54.078 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:33:54.078 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:33:54.179 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:33:54.210 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:33:54.211 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:33:54.211 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:33:54.312 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:33:54.344 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:33:54.345 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:33:54.345 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:33:54.446 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:33:54.475 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:33:54.476 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:33:54.476 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:33:54.577 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:33:54.608 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:33:54.609 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:33:54.609 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:33:54.710 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:33:54.740 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:33:54.740 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:33:54.741 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:33:54.826 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:33:54 smithi097 ceph-mon[33317]: pgmap v350: 158 pgs: 1 peering, 2 active+remapped+backfill_toofull, 2 active+remapped+backfilling, 153 active+clean; 6.2 GiB data, 23 GiB used, 693 GiB / 715 GiB avail; 1.7 KiB/s rd, 0 B/s wr, 1.23k op/s; 21853/305313 objects misplaced (7.158%); 23 MiB/s, 364 objects/s recovering 2024-03-11T21:33:54.841 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:33:54.860 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:33:54 smithi092 ceph-mon[29572]: pgmap v350: 158 pgs: 1 peering, 2 active+remapped+backfill_toofull, 2 active+remapped+backfilling, 153 active+clean; 6.2 GiB data, 23 GiB used, 693 GiB / 715 GiB avail; 1.7 KiB/s rd, 0 B/s wr, 1.23k op/s; 21853/305313 objects misplaced (7.158%); 23 MiB/s, 364 objects/s recovering 2024-03-11T21:33:54.860 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:33:54 smithi092 ceph-mon[37563]: pgmap v350: 158 pgs: 1 peering, 2 active+remapped+backfill_toofull, 2 active+remapped+backfilling, 153 active+clean; 6.2 GiB data, 23 GiB used, 693 GiB / 715 GiB avail; 1.7 KiB/s rd, 0 B/s wr, 1.23k op/s; 21853/305313 objects misplaced (7.158%); 23 MiB/s, 364 objects/s recovering 2024-03-11T21:33:54.871 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:33:54.872 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:33:54.872 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:33:54.973 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:33:55.002 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:33:55.003 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:33:55.004 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:33:55.104 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:33:55.146 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:33:55.147 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:33:55.147 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:33:55.248 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:33:55.281 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:33:55.282 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:33:55.282 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:33:55.383 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:33:55.408 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:33:55.409 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:33:55.409 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:33:55.510 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:33:55.551 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:33:55.551 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:33:55.551 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:33:55.652 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:33:55.694 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:33:55.694 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:33:55.694 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:33:55.795 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:33:55.819 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:33:55.820 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:33:55.820 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:33:55.921 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:33:55.953 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:33:55.954 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:33:55.954 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:33:56.055 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:33:56.099 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:33:56.099 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:33:56.100 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:33:56.202 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:33:56.232 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:33:56.233 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:33:56.233 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:33:56.335 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:33:56.397 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:33:56.398 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:33:56.399 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:33:56.500 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:33:56.546 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:33:56.547 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:33:56.547 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:33:56.649 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:33:56.681 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:33:56.683 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:33:56.683 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:33:56.759 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:33:56 smithi092 ceph-mon[29572]: pgmap v352: 158 pgs: 1 peering, 2 active+remapped+backfill_toofull, 2 active+remapped+backfilling, 153 active+clean; 6.1 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 914 B/s rd, 0 B/s wr, 901 op/s; 20299/301938 objects misplaced (6.723%); 11 MiB/s, 181 objects/s recovering 2024-03-11T21:33:56.760 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:33:56 smithi092 ceph-mon[37563]: pgmap v352: 158 pgs: 1 peering, 2 active+remapped+backfill_toofull, 2 active+remapped+backfilling, 153 active+clean; 6.1 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 914 B/s rd, 0 B/s wr, 901 op/s; 20299/301938 objects misplaced (6.723%); 11 MiB/s, 181 objects/s recovering 2024-03-11T21:33:56.784 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:33:56.826 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:33:56 smithi097 ceph-mon[33317]: pgmap v352: 158 pgs: 1 peering, 2 active+remapped+backfill_toofull, 2 active+remapped+backfilling, 153 active+clean; 6.1 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 914 B/s rd, 0 B/s wr, 901 op/s; 20299/301938 objects misplaced (6.723%); 11 MiB/s, 181 objects/s recovering 2024-03-11T21:33:56.832 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:33:56.833 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:33:56.833 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:33:56.934 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:33:56.967 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:33:56.968 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:33:56.968 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:33:57.069 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:33:57.103 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:33:57.104 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:33:57.104 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:33:57.205 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:33:57.243 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:33:57.244 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:33:57.244 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:33:57.345 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:33:57.385 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:33:57.385 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:33:57.386 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:33:57.389 INFO:teuthology.orchestra.run.smithi092.stderr:Inferring config /var/lib/ceph/00db0016-dfed-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T21:33:57.408 INFO:teuthology.orchestra.run.smithi092.stderr:Inferring config /var/lib/ceph/00db0016-dfed-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T21:33:57.485 INFO:teuthology.orchestra.run.smithi092.stdout:{ 2024-03-11T21:33:57.486 INFO:teuthology.orchestra.run.smithi092.stdout: "ops": [], 2024-03-11T21:33:57.486 INFO:teuthology.orchestra.run.smithi092.stdout: "complaint_time": 30, 2024-03-11T21:33:57.486 INFO:teuthology.orchestra.run.smithi092.stdout: "num_blocked_ops": 0 2024-03-11T21:33:57.486 INFO:teuthology.orchestra.run.smithi092.stdout:} 2024-03-11T21:33:57.487 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:33:57.567 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:33:57.568 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:33:57.568 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:33:57.590 INFO:teuthology.orchestra.run.smithi092.stderr:noscrub is unset 2024-03-11T21:33:57.668 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:33:57.692 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:33:57 smithi092 ceph-mon[29572]: from='client.? v1:172.21.15.92:0/3859079787' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-11T21:33:57.692 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:33:57 smithi092 ceph-mon[29572]: from='client.25153 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-11T21:33:57.692 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:33:57 smithi092 ceph-mon[37563]: from='client.? v1:172.21.15.92:0/3859079787' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-11T21:33:57.692 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:33:57 smithi092 ceph-mon[37563]: from='client.25153 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-11T21:33:57.718 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:33:57.719 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:33:57.719 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:33:57.821 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:33:57.859 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:33:57.861 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:33:57.861 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:33:57.962 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:33:57.991 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:33:57.992 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:33:57.992 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:33:58.055 INFO:journalctl@ceph.mgr.y.smithi092.stdout:Mar 11 21:33:57 smithi092 ceph-00db0016-dfed-11ee-95c8-87774f69a715-mgr-y[29801]: ::ffff:172.21.15.97 - - [11/Mar/2024:21:33:57] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-11T21:33:58.076 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:33:57 smithi097 ceph-mon[33317]: from='client.? v1:172.21.15.92:0/3859079787' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-11T21:33:58.076 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:33:57 smithi097 ceph-mon[33317]: from='client.25153 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-11T21:33:58.093 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:33:58.125 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:33:58.126 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:33:58.126 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:33:58.227 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:33:58.273 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:33:58.274 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:33:58.274 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:33:58.374 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:33:58.398 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:33:58.399 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:33:58.399 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:33:58.500 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:33:58.536 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:33:58.536 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:33:58.537 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:33:58.637 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:33:58.674 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:33:58.675 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:33:58.675 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:33:58.776 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:33:58.827 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:33:58.827 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:33:58.828 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:33:58.860 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:33:58 smithi092 ceph-mon[29572]: pgmap v353: 158 pgs: 1 peering, 2 active+remapped+backfill_toofull, 2 active+remapped+backfilling, 153 active+clean; 6.0 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 775 B/s rd, 0 B/s wr, 1.00k op/s; 19923/297231 objects misplaced (6.703%); 9.6 MiB/s, 154 objects/s recovering 2024-03-11T21:33:58.860 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:33:58 smithi092 ceph-mon[29572]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-11T21:33:58.860 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:33:58 smithi092 ceph-mon[29572]: from='client.25153 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-03-11T21:33:58.861 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:33:58 smithi092 ceph-mon[29572]: osdmap e210: 8 total, 8 up, 8 in 2024-03-11T21:33:58.861 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:33:58 smithi092 ceph-mon[29572]: 8.12 scrub starts 2024-03-11T21:33:58.861 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:33:58 smithi092 ceph-mon[29572]: osdmap e211: 8 total, 8 up, 8 in 2024-03-11T21:33:58.861 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:33:58 smithi092 ceph-mon[37563]: pgmap v353: 158 pgs: 1 peering, 2 active+remapped+backfill_toofull, 2 active+remapped+backfilling, 153 active+clean; 6.0 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 775 B/s rd, 0 B/s wr, 1.00k op/s; 19923/297231 objects misplaced (6.703%); 9.6 MiB/s, 154 objects/s recovering 2024-03-11T21:33:58.861 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:33:58 smithi092 ceph-mon[37563]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-11T21:33:58.861 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:33:58 smithi092 ceph-mon[37563]: from='client.25153 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-03-11T21:33:58.861 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:33:58 smithi092 ceph-mon[37563]: osdmap e210: 8 total, 8 up, 8 in 2024-03-11T21:33:58.861 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:33:58 smithi092 ceph-mon[37563]: 8.12 scrub starts 2024-03-11T21:33:58.861 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:33:58 smithi092 ceph-mon[37563]: osdmap e211: 8 total, 8 up, 8 in 2024-03-11T21:33:58.929 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:33:58.973 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:33:58.973 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:33:58.974 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:33:59.075 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:33:59.076 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:33:58 smithi097 ceph-mon[33317]: pgmap v353: 158 pgs: 1 peering, 2 active+remapped+backfill_toofull, 2 active+remapped+backfilling, 153 active+clean; 6.0 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 775 B/s rd, 0 B/s wr, 1.00k op/s; 19923/297231 objects misplaced (6.703%); 9.6 MiB/s, 154 objects/s recovering 2024-03-11T21:33:59.076 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:33:58 smithi097 ceph-mon[33317]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-11T21:33:59.076 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:33:58 smithi097 ceph-mon[33317]: from='client.25153 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-03-11T21:33:59.076 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:33:58 smithi097 ceph-mon[33317]: osdmap e210: 8 total, 8 up, 8 in 2024-03-11T21:33:59.076 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:33:58 smithi097 ceph-mon[33317]: 8.12 scrub starts 2024-03-11T21:33:59.076 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:33:58 smithi097 ceph-mon[33317]: osdmap e211: 8 total, 8 up, 8 in 2024-03-11T21:33:59.112 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:33:59.113 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:33:59.113 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:33:59.214 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:33:59.245 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:33:59.246 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:33:59.246 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:33:59.347 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:33:59.405 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:33:59.406 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:33:59.406 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:33:59.432 DEBUG:teuthology.orchestra.run.smithi092:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid 00db0016-dfed-11ee-95c8-87774f69a715 -- ceph daemon osd.0 dump_historic_ops 2024-03-11T21:33:59.507 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:33:59.563 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:33:59.564 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:33:59.564 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:33:59.665 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:33:59.713 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:33:59.714 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:33:59.714 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:33:59.721 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:33:59 smithi092 ceph-mon[29572]: 8.19 scrub starts 2024-03-11T21:33:59.721 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:33:59 smithi092 ceph-mon[29572]: 8.18 scrub starts 2024-03-11T21:33:59.721 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:33:59 smithi092 ceph-mon[29572]: 8.16 scrub starts 2024-03-11T21:33:59.721 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:33:59 smithi092 ceph-mon[29572]: purged_snaps scrub starts 2024-03-11T21:33:59.721 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:33:59 smithi092 ceph-mon[29572]: purged_snaps scrub ok 2024-03-11T21:33:59.721 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:33:59 smithi092 ceph-mon[29572]: 8.12 scrub ok 2024-03-11T21:33:59.721 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:33:59 smithi092 ceph-mon[29572]: 8.19 scrub ok 2024-03-11T21:33:59.722 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:33:59 smithi092 ceph-mon[37563]: 8.19 scrub starts 2024-03-11T21:33:59.722 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:33:59 smithi092 ceph-mon[37563]: 8.18 scrub starts 2024-03-11T21:33:59.722 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:33:59 smithi092 ceph-mon[37563]: 8.16 scrub starts 2024-03-11T21:33:59.722 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:33:59 smithi092 ceph-mon[37563]: purged_snaps scrub starts 2024-03-11T21:33:59.722 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:33:59 smithi092 ceph-mon[37563]: purged_snaps scrub ok 2024-03-11T21:33:59.722 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:33:59 smithi092 ceph-mon[37563]: 8.12 scrub ok 2024-03-11T21:33:59.722 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:33:59 smithi092 ceph-mon[37563]: 8.19 scrub ok 2024-03-11T21:33:59.815 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:33:59.848 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:33:59.849 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:33:59.849 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:33:59.951 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:33:59.984 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:33:59.985 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:33:59.985 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:34:00.076 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:33:59 smithi097 ceph-mon[33317]: 8.19 scrub starts 2024-03-11T21:34:00.076 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:33:59 smithi097 ceph-mon[33317]: 8.18 scrub starts 2024-03-11T21:34:00.077 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:33:59 smithi097 ceph-mon[33317]: 8.16 scrub starts 2024-03-11T21:34:00.077 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:33:59 smithi097 ceph-mon[33317]: purged_snaps scrub starts 2024-03-11T21:34:00.077 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:33:59 smithi097 ceph-mon[33317]: purged_snaps scrub ok 2024-03-11T21:34:00.077 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:33:59 smithi097 ceph-mon[33317]: 8.12 scrub ok 2024-03-11T21:34:00.077 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:33:59 smithi097 ceph-mon[33317]: 8.19 scrub ok 2024-03-11T21:34:00.086 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:34:00.119 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:34:00.119 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:34:00.119 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:34:00.220 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:34:00.261 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:34:00.262 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:34:00.262 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:34:00.363 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:34:00.406 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:34:00.407 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:34:00.407 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:34:00.508 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:34:00.551 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:34:00.552 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:34:00.552 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:34:00.598 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:34:00 smithi092 ceph-mon[29572]: pgmap v356: 158 pgs: 2 active+remapped+backfill_toofull, 2 active+remapped+backfilling, 154 active+clean; 5.8 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 852 B/s rd, 0 B/s wr, 1.09k op/s; 17430/285669 objects misplaced (6.101%); 15 MiB/s, 234 objects/s recovering 2024-03-11T21:34:00.598 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:34:00 smithi092 ceph-mon[29572]: 8.18 scrub ok 2024-03-11T21:34:00.598 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:34:00 smithi092 ceph-mon[29572]: osdmap e212: 8 total, 8 up, 8 in 2024-03-11T21:34:00.598 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:34:00 smithi092 ceph-mon[29572]: 8.16 scrub ok 2024-03-11T21:34:00.653 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:34:00.688 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:34:00.689 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:34:00.689 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:34:00.790 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:34:00.817 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:34:00.817 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:34:00.818 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:34:00.860 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:34:00 smithi092 ceph-mon[37563]: pgmap v356: 158 pgs: 2 active+remapped+backfill_toofull, 2 active+remapped+backfilling, 154 active+clean; 5.8 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 852 B/s rd, 0 B/s wr, 1.09k op/s; 17430/285669 objects misplaced (6.101%); 15 MiB/s, 234 objects/s recovering 2024-03-11T21:34:00.860 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:34:00 smithi092 ceph-mon[37563]: 8.18 scrub ok 2024-03-11T21:34:00.860 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:34:00 smithi092 ceph-mon[37563]: osdmap e212: 8 total, 8 up, 8 in 2024-03-11T21:34:00.860 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:34:00 smithi092 ceph-mon[37563]: 8.16 scrub ok 2024-03-11T21:34:00.919 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:34:00.961 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:34:00.962 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:34:00.962 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:34:01.063 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:34:01.076 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:34:00 smithi097 ceph-mon[33317]: pgmap v356: 158 pgs: 2 active+remapped+backfill_toofull, 2 active+remapped+backfilling, 154 active+clean; 5.8 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 852 B/s rd, 0 B/s wr, 1.09k op/s; 17430/285669 objects misplaced (6.101%); 15 MiB/s, 234 objects/s recovering 2024-03-11T21:34:01.076 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:34:00 smithi097 ceph-mon[33317]: 8.18 scrub ok 2024-03-11T21:34:01.076 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:34:00 smithi097 ceph-mon[33317]: osdmap e212: 8 total, 8 up, 8 in 2024-03-11T21:34:01.076 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:34:00 smithi097 ceph-mon[33317]: 8.16 scrub ok 2024-03-11T21:34:01.106 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:34:01.106 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:34:01.107 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:34:01.208 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:34:01.255 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:34:01.256 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:34:01.256 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:34:01.357 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:34:01.389 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:34:01.389 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:34:01.390 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:34:01.491 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:34:01.528 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:34:01.528 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:34:01.528 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:34:01.629 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:34:01.660 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:34:01.660 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:34:01.660 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:34:01.761 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:34:01.793 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:34:01.794 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:34:01.795 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:34:01.860 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:34:01 smithi092 ceph-mon[29572]: 4.10 scrub starts 2024-03-11T21:34:01.860 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:34:01 smithi092 ceph-mon[29572]: 4.10 scrub ok 2024-03-11T21:34:01.860 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:34:01 smithi092 ceph-mon[37563]: 4.10 scrub starts 2024-03-11T21:34:01.860 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:34:01 smithi092 ceph-mon[37563]: 4.10 scrub ok 2024-03-11T21:34:01.895 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:34:01.924 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:34:01.925 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:34:01.925 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:34:02.026 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:34:02.061 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:34:02.062 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:34:02.062 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:34:02.075 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:34:01 smithi097 ceph-mon[33317]: 4.10 scrub starts 2024-03-11T21:34:02.076 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:34:01 smithi097 ceph-mon[33317]: 4.10 scrub ok 2024-03-11T21:34:02.085 DEBUG:teuthology.orchestra.run.smithi092:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid 00db0016-dfed-11ee-95c8-87774f69a715 -- ceph osd unset nodeep-scrub 2024-03-11T21:34:02.164 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:34:02.190 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:34:02.191 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:34:02.191 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:34:02.292 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:34:02.324 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:34:02.325 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:34:02.325 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:34:02.426 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:34:02.452 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:34:02.452 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:34:02.453 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:34:02.553 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:34:02.580 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:34:02.581 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:34:02.582 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:34:02.622 INFO:journalctl@ceph.iscsi.iscsi.a.smithi097.stdout:Mar 11 21:34:02 smithi097 ceph-00db0016-dfed-11ee-95c8-87774f69a715-iscsi-iscsi-a[60943]: debug there is no tcmu-runner data available 2024-03-11T21:34:02.683 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:34:02.716 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:34:02.716 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:34:02.717 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:34:02.817 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:34:02.851 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:34:02.852 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:34:02.853 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:34:02.953 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:34:02.991 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:34:02.992 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:34:02.992 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:34:03.002 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:34:02 smithi092 ceph-mon[29572]: pgmap v358: 158 pgs: 2 active+remapped+backfill_toofull, 2 active+remapped+backfilling, 154 active+clean; 5.8 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 852 B/s rd, 0 B/s wr, 904 op/s; 17430/285669 objects misplaced (6.101%); 9.5 MiB/s, 152 objects/s recovering 2024-03-11T21:34:03.002 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:34:02 smithi092 ceph-mon[29572]: osdmap e213: 8 total, 8 up, 8 in 2024-03-11T21:34:03.002 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:34:02 smithi092 ceph-mon[29572]: 8.13 scrub starts 2024-03-11T21:34:03.002 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:34:02 smithi092 ceph-mon[29572]: 8.13 scrub ok 2024-03-11T21:34:03.002 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:34:02 smithi092 ceph-mon[37563]: pgmap v358: 158 pgs: 2 active+remapped+backfill_toofull, 2 active+remapped+backfilling, 154 active+clean; 5.8 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 852 B/s rd, 0 B/s wr, 904 op/s; 17430/285669 objects misplaced (6.101%); 9.5 MiB/s, 152 objects/s recovering 2024-03-11T21:34:03.002 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:34:02 smithi092 ceph-mon[37563]: osdmap e213: 8 total, 8 up, 8 in 2024-03-11T21:34:03.002 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:34:02 smithi092 ceph-mon[37563]: 8.13 scrub starts 2024-03-11T21:34:03.002 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:34:02 smithi092 ceph-mon[37563]: 8.13 scrub ok 2024-03-11T21:34:03.076 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:34:02 smithi097 ceph-mon[33317]: pgmap v358: 158 pgs: 2 active+remapped+backfill_toofull, 2 active+remapped+backfilling, 154 active+clean; 5.8 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 852 B/s rd, 0 B/s wr, 904 op/s; 17430/285669 objects misplaced (6.101%); 9.5 MiB/s, 152 objects/s recovering 2024-03-11T21:34:03.076 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:34:02 smithi097 ceph-mon[33317]: osdmap e213: 8 total, 8 up, 8 in 2024-03-11T21:34:03.076 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:34:02 smithi097 ceph-mon[33317]: 8.13 scrub starts 2024-03-11T21:34:03.076 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:34:02 smithi097 ceph-mon[33317]: 8.13 scrub ok 2024-03-11T21:34:03.094 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:34:03.130 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:34:03.131 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:34:03.131 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:34:03.232 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:34:03.288 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:34:03.289 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:34:03.289 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:34:03.390 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:34:03.421 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:34:03.422 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:34:03.422 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:34:03.523 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:34:03.566 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:34:03.566 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:34:03.567 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:34:03.668 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:34:03.698 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:34:03.698 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:34:03.699 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:34:03.800 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:34:03.832 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:34:03.833 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:34:03.833 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:34:03.934 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:34:03.966 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:34:03.966 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:34:03.967 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:34:04.067 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:34:04.076 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:34:03 smithi097 ceph-mon[33317]: from='client.24469 v1:172.21.15.97:0/404208310' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:34:04.076 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:34:03 smithi097 ceph-mon[33317]: osdmap e214: 8 total, 8 up, 8 in 2024-03-11T21:34:04.076 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:34:03 smithi097 ceph-mon[33317]: 8.15 scrub starts 2024-03-11T21:34:04.076 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:34:03 smithi097 ceph-mon[33317]: osdmap e215: 8 total, 8 up, 8 in 2024-03-11T21:34:04.076 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:34:03 smithi097 ceph-mon[33317]: 8.15 scrub ok 2024-03-11T21:34:04.109 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:34:04.110 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:34:03 smithi092 ceph-mon[29572]: from='client.24469 v1:172.21.15.97:0/404208310' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:34:04.110 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:34:03 smithi092 ceph-mon[29572]: osdmap e214: 8 total, 8 up, 8 in 2024-03-11T21:34:04.110 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:34:03 smithi092 ceph-mon[29572]: 8.15 scrub starts 2024-03-11T21:34:04.110 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:34:03 smithi092 ceph-mon[29572]: osdmap e215: 8 total, 8 up, 8 in 2024-03-11T21:34:04.110 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:34:03 smithi092 ceph-mon[29572]: 8.15 scrub ok 2024-03-11T21:34:04.110 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:34:03 smithi092 ceph-mon[37563]: from='client.24469 v1:172.21.15.97:0/404208310' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:34:04.111 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:34:03 smithi092 ceph-mon[37563]: osdmap e214: 8 total, 8 up, 8 in 2024-03-11T21:34:04.111 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:34:03 smithi092 ceph-mon[37563]: 8.15 scrub starts 2024-03-11T21:34:04.111 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:34:03 smithi092 ceph-mon[37563]: osdmap e215: 8 total, 8 up, 8 in 2024-03-11T21:34:04.111 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:34:03 smithi092 ceph-mon[37563]: 8.15 scrub ok 2024-03-11T21:34:04.111 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:34:04.112 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:34:04.212 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:34:04.246 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:34:04.247 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:34:04.247 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:34:04.348 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:34:04.419 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:34:04.420 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:34:04.420 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:34:04.521 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:34:04.553 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:34:04.554 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:34:04.554 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:34:04.655 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:34:04.693 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:34:04.694 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:34:04.694 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:34:04.795 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:34:04.828 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:34:04.829 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:34:04.829 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:34:04.930 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:34:04.964 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:34:04.965 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:34:04.965 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:34:05.066 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:34:05.075 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:34:04 smithi097 ceph-mon[33317]: pgmap v361: 158 pgs: 1 peering, 1 active+remapped+backfill_toofull, 2 active+remapped+backfilling, 154 active+clean; 5.5 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 1.0 KiB/s rd, 1 op/s; 13760/273138 objects misplaced (5.038%) 2024-03-11T21:34:05.076 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:34:04 smithi097 ceph-mon[33317]: Health check update: Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull (PG_BACKFILL_FULL) 2024-03-11T21:34:05.108 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:34:05.109 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:34:05.109 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:34:05.110 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:34:04 smithi092 ceph-mon[29572]: pgmap v361: 158 pgs: 1 peering, 1 active+remapped+backfill_toofull, 2 active+remapped+backfilling, 154 active+clean; 5.5 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 1.0 KiB/s rd, 1 op/s; 13760/273138 objects misplaced (5.038%) 2024-03-11T21:34:05.110 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:34:04 smithi092 ceph-mon[29572]: Health check update: Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull (PG_BACKFILL_FULL) 2024-03-11T21:34:05.110 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:34:04 smithi092 ceph-mon[37563]: pgmap v361: 158 pgs: 1 peering, 1 active+remapped+backfill_toofull, 2 active+remapped+backfilling, 154 active+clean; 5.5 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 1.0 KiB/s rd, 1 op/s; 13760/273138 objects misplaced (5.038%) 2024-03-11T21:34:05.110 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:34:04 smithi092 ceph-mon[37563]: Health check update: Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull (PG_BACKFILL_FULL) 2024-03-11T21:34:05.210 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:34:05.250 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:34:05.250 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:34:05.251 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:34:05.351 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:34:05.387 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:34:05.387 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:34:05.388 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:34:05.489 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:34:05.520 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:34:05.521 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:34:05.521 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:34:05.622 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:34:05.647 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:34:05.648 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:34:05.648 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:34:05.749 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:34:05.786 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:34:05.787 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:34:05.787 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:34:05.888 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:34:05.912 INFO:teuthology.orchestra.run.smithi092.stderr:Inferring config /var/lib/ceph/00db0016-dfed-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T21:34:05.920 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:34:05.921 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:34:05.921 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:34:06.022 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:34:06.039 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:34:05 smithi092 ceph-mon[29572]: 2.1c scrub starts 2024-03-11T21:34:06.039 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:34:05 smithi092 ceph-mon[29572]: 2.1c scrub ok 2024-03-11T21:34:06.039 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:34:05 smithi092 ceph-mon[37563]: 2.1c scrub starts 2024-03-11T21:34:06.039 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:34:05 smithi092 ceph-mon[37563]: 2.1c scrub ok 2024-03-11T21:34:06.056 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:34:06.056 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:34:06.056 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:34:06.076 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:34:05 smithi097 ceph-mon[33317]: 2.1c scrub starts 2024-03-11T21:34:06.076 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:34:05 smithi097 ceph-mon[33317]: 2.1c scrub ok 2024-03-11T21:34:06.158 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:34:06.163 INFO:teuthology.orchestra.run.smithi092.stderr:Inferring config /var/lib/ceph/00db0016-dfed-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T21:34:06.207 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:34:06.209 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:34:06.210 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:34:06.281 INFO:teuthology.orchestra.run.smithi092.stdout:osd.0: {} 2024-03-11T21:34:06.311 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:34:06.335 INFO:teuthology.orchestra.run.smithi092.stdout:osd.1: {} 2024-03-11T21:34:06.354 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:34:06.355 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:34:06.355 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:34:06.374 INFO:teuthology.orchestra.run.smithi092.stdout:osd.2: {} 2024-03-11T21:34:06.415 INFO:teuthology.orchestra.run.smithi092.stdout:osd.3: {} 2024-03-11T21:34:06.454 INFO:teuthology.orchestra.run.smithi092.stdout:osd.4: {} 2024-03-11T21:34:06.457 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:34:06.498 INFO:teuthology.orchestra.run.smithi092.stdout:osd.5: {} 2024-03-11T21:34:06.499 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:34:06.500 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:34:06.500 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:34:06.536 INFO:teuthology.orchestra.run.smithi092.stdout:osd.6: {} 2024-03-11T21:34:06.578 INFO:teuthology.orchestra.run.smithi092.stdout:osd.7: {} 2024-03-11T21:34:06.582 INFO:teuthology.orchestra.run.smithi092.stderr:osd.0: osd_enable_op_tracker = '' 2024-03-11T21:34:06.582 INFO:teuthology.orchestra.run.smithi092.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-11T21:34:06.582 INFO:teuthology.orchestra.run.smithi092.stderr:osd.2: osd_enable_op_tracker = '' 2024-03-11T21:34:06.582 INFO:teuthology.orchestra.run.smithi092.stderr:osd.3: osd_enable_op_tracker = '' 2024-03-11T21:34:06.582 INFO:teuthology.orchestra.run.smithi092.stderr:osd.4: osd_enable_op_tracker = '' 2024-03-11T21:34:06.582 INFO:teuthology.orchestra.run.smithi092.stderr:osd.5: osd_enable_op_tracker = '' 2024-03-11T21:34:06.582 INFO:teuthology.orchestra.run.smithi092.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-11T21:34:06.583 INFO:teuthology.orchestra.run.smithi092.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-11T21:34:06.602 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:34:06.638 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:34:06.639 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:34:06.639 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:34:06.686 INFO:teuthology.orchestra.run.smithi092.stderr:reweighted osd.4 to 1 (10000) 2024-03-11T21:34:06.740 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:34:06.785 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:34:06.786 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:34:06.786 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:34:06.860 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:34:06 smithi092 ceph-mon[29572]: pgmap v363: 158 pgs: 1 active+remapped+backfill_toofull, 1 peering, 2 active+remapped+backfilling, 154 active+clean; 5.5 GiB data, 21 GiB used, 695 GiB / 715 GiB avail; 927 B/s rd, 0 op/s; 13319/270471 objects misplaced (4.924%) 2024-03-11T21:34:06.860 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:34:06 smithi092 ceph-mon[29572]: 6.1f scrub starts 2024-03-11T21:34:06.860 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:34:06 smithi092 ceph-mon[29572]: from='client.? v1:172.21.15.92:0/103799955' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]: dispatch 2024-03-11T21:34:06.860 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:34:06 smithi092 ceph-mon[29572]: from='client.35132 ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]: dispatch 2024-03-11T21:34:06.861 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:34:06 smithi092 ceph-mon[29572]: 6.1f scrub ok 2024-03-11T21:34:06.861 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:34:06 smithi092 ceph-mon[29572]: 2.1d scrub starts 2024-03-11T21:34:06.861 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:34:06 smithi092 ceph-mon[29572]: 2.1d scrub ok 2024-03-11T21:34:06.861 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:34:06 smithi092 ceph-mon[29572]: from='client.? v1:172.21.15.92:0/436590046' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-11T21:34:06.861 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:34:06 smithi092 ceph-mon[37563]: pgmap v363: 158 pgs: 1 active+remapped+backfill_toofull, 1 peering, 2 active+remapped+backfilling, 154 active+clean; 5.5 GiB data, 21 GiB used, 695 GiB / 715 GiB avail; 927 B/s rd, 0 op/s; 13319/270471 objects misplaced (4.924%) 2024-03-11T21:34:06.861 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:34:06 smithi092 ceph-mon[37563]: 6.1f scrub starts 2024-03-11T21:34:06.861 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:34:06 smithi092 ceph-mon[37563]: from='client.? v1:172.21.15.92:0/103799955' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]: dispatch 2024-03-11T21:34:06.861 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:34:06 smithi092 ceph-mon[37563]: from='client.35132 ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]: dispatch 2024-03-11T21:34:06.861 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:34:06 smithi092 ceph-mon[37563]: 6.1f scrub ok 2024-03-11T21:34:06.861 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:34:06 smithi092 ceph-mon[37563]: 2.1d scrub starts 2024-03-11T21:34:06.861 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:34:06 smithi092 ceph-mon[37563]: 2.1d scrub ok 2024-03-11T21:34:06.861 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:34:06 smithi092 ceph-mon[37563]: from='client.? v1:172.21.15.92:0/436590046' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-11T21:34:06.887 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:34:06.920 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:34:06.920 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:34:06.921 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:34:07.022 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:34:07.055 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:34:07.056 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:34:07.056 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:34:07.076 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:34:06 smithi097 ceph-mon[33317]: pgmap v363: 158 pgs: 1 active+remapped+backfill_toofull, 1 peering, 2 active+remapped+backfilling, 154 active+clean; 5.5 GiB data, 21 GiB used, 695 GiB / 715 GiB avail; 927 B/s rd, 0 op/s; 13319/270471 objects misplaced (4.924%) 2024-03-11T21:34:07.076 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:34:06 smithi097 ceph-mon[33317]: 6.1f scrub starts 2024-03-11T21:34:07.076 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:34:06 smithi097 ceph-mon[33317]: from='client.? v1:172.21.15.92:0/103799955' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]: dispatch 2024-03-11T21:34:07.076 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:34:06 smithi097 ceph-mon[33317]: from='client.35132 ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]: dispatch 2024-03-11T21:34:07.076 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:34:06 smithi097 ceph-mon[33317]: 6.1f scrub ok 2024-03-11T21:34:07.077 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:34:06 smithi097 ceph-mon[33317]: 2.1d scrub starts 2024-03-11T21:34:07.077 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:34:06 smithi097 ceph-mon[33317]: 2.1d scrub ok 2024-03-11T21:34:07.077 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:34:06 smithi097 ceph-mon[33317]: from='client.? v1:172.21.15.92:0/436590046' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-11T21:34:07.157 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:34:07.196 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:34:07.197 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:34:07.197 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:34:07.298 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:34:07.328 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:34:07.329 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:34:07.329 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:34:07.430 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:34:07.466 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:34:07.466 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:34:07.467 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:34:07.568 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:34:07.600 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:34:07.601 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:34:07.602 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:34:07.702 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:34:07.738 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:34:07.738 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:34:07.739 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:34:07.839 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:34:07.873 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:34:07.873 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:34:07.874 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:34:07.923 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:34:07 smithi092 ceph-mon[29572]: from='client.35132 ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]': finished 2024-03-11T21:34:07.923 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:34:07 smithi092 ceph-mon[29572]: osdmap e216: 8 total, 8 up, 8 in 2024-03-11T21:34:07.923 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:34:07 smithi092 ceph-mon[29572]: from='mgr.14607 v1:172.21.15.92:0/1702717597' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T21:34:07.924 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:34:07 smithi092 ceph-mon[37563]: from='client.35132 ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]': finished 2024-03-11T21:34:07.924 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:34:07 smithi092 ceph-mon[37563]: osdmap e216: 8 total, 8 up, 8 in 2024-03-11T21:34:07.924 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:34:07 smithi092 ceph-mon[37563]: from='mgr.14607 v1:172.21.15.92:0/1702717597' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T21:34:07.974 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:34:08.001 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:34:08.001 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:34:08.001 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:34:08.076 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:34:07 smithi097 ceph-mon[33317]: from='client.35132 ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]': finished 2024-03-11T21:34:08.076 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:34:07 smithi097 ceph-mon[33317]: osdmap e216: 8 total, 8 up, 8 in 2024-03-11T21:34:08.076 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:34:07 smithi097 ceph-mon[33317]: from='mgr.14607 v1:172.21.15.92:0/1702717597' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T21:34:08.102 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:34:08.129 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:34:08.130 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:34:08.131 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:34:08.181 INFO:journalctl@ceph.mgr.y.smithi092.stdout:Mar 11 21:34:07 smithi092 ceph-00db0016-dfed-11ee-95c8-87774f69a715-mgr-y[29801]: ::ffff:172.21.15.97 - - [11/Mar/2024:21:34:07] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-11T21:34:08.232 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:34:08.258 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:34:08.259 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:34:08.259 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:34:08.360 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:34:08.384 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:34:08.384 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:34:08.385 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:34:08.485 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:34:08.510 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:34:08.511 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:34:08.511 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:34:08.612 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:34:08.637 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:34:08.637 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:34:08.638 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:34:08.738 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:34:08.763 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:34:08.763 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:34:08.764 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:34:08.865 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:34:08.888 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:34:08.889 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:34:08.889 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:34:08.990 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:34:09.026 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:34:09.027 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:34:09.027 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:34:09.076 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:34:08 smithi097 ceph-mon[33317]: pgmap v365: 158 pgs: 1 active+remapped+backfill_toofull, 1 peering, 2 active+remapped+backfilling, 154 active+clean; 5.4 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 930 B/s rd, 0 op/s; 10071/266316 objects misplaced (3.782%) 2024-03-11T21:34:09.076 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:34:08 smithi097 ceph-mon[33317]: 5.6 scrub starts 2024-03-11T21:34:09.076 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:34:08 smithi097 ceph-mon[33317]: 5.6 scrub ok 2024-03-11T21:34:09.076 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:34:08 smithi097 ceph-mon[33317]: 8.17 scrub starts 2024-03-11T21:34:09.076 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:34:08 smithi097 ceph-mon[33317]: osdmap e217: 8 total, 8 up, 8 in 2024-03-11T21:34:09.110 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:34:08 smithi092 ceph-mon[29572]: pgmap v365: 158 pgs: 1 active+remapped+backfill_toofull, 1 peering, 2 active+remapped+backfilling, 154 active+clean; 5.4 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 930 B/s rd, 0 op/s; 10071/266316 objects misplaced (3.782%) 2024-03-11T21:34:09.110 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:34:08 smithi092 ceph-mon[29572]: 5.6 scrub starts 2024-03-11T21:34:09.110 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:34:08 smithi092 ceph-mon[29572]: 5.6 scrub ok 2024-03-11T21:34:09.110 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:34:08 smithi092 ceph-mon[29572]: 8.17 scrub starts 2024-03-11T21:34:09.110 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:34:08 smithi092 ceph-mon[29572]: osdmap e217: 8 total, 8 up, 8 in 2024-03-11T21:34:09.110 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:34:08 smithi092 ceph-mon[37563]: pgmap v365: 158 pgs: 1 active+remapped+backfill_toofull, 1 peering, 2 active+remapped+backfilling, 154 active+clean; 5.4 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 930 B/s rd, 0 op/s; 10071/266316 objects misplaced (3.782%) 2024-03-11T21:34:09.110 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:34:08 smithi092 ceph-mon[37563]: 5.6 scrub starts 2024-03-11T21:34:09.110 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:34:08 smithi092 ceph-mon[37563]: 5.6 scrub ok 2024-03-11T21:34:09.110 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:34:08 smithi092 ceph-mon[37563]: 8.17 scrub starts 2024-03-11T21:34:09.111 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:34:08 smithi092 ceph-mon[37563]: osdmap e217: 8 total, 8 up, 8 in 2024-03-11T21:34:09.128 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:34:09.153 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:34:09.154 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:34:09.154 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:34:09.255 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:34:09.279 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:34:09.280 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:34:09.280 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:34:09.381 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:34:09.406 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:34:09.407 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:34:09.407 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:34:09.508 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:34:09.532 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:34:09.533 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:34:09.534 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:34:09.634 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:34:09.658 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:34:09.659 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:34:09.659 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:34:09.760 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:34:09.789 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:34:09.790 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:34:09.790 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:34:09.841 DEBUG:teuthology.orchestra.run.smithi092:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid 00db0016-dfed-11ee-95c8-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-03-11T21:34:09.888 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:34:09 smithi092 ceph-mon[29572]: 8.3 scrub starts 2024-03-11T21:34:09.889 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:34:09 smithi092 ceph-mon[29572]: 2.13 scrub starts 2024-03-11T21:34:09.889 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:34:09 smithi092 ceph-mon[29572]: 2.13 scrub ok 2024-03-11T21:34:09.889 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:34:09 smithi092 ceph-mon[29572]: osdmap e218: 8 total, 8 up, 8 in 2024-03-11T21:34:09.889 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:34:09 smithi092 ceph-mon[29572]: 2.f scrub starts 2024-03-11T21:34:09.889 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:34:09 smithi092 ceph-mon[29572]: 2.f scrub ok 2024-03-11T21:34:09.889 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:34:09 smithi092 ceph-mon[29572]: purged_snaps scrub starts 2024-03-11T21:34:09.889 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:34:09 smithi092 ceph-mon[29572]: purged_snaps scrub ok 2024-03-11T21:34:09.889 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:34:09 smithi092 ceph-mon[37563]: 8.3 scrub starts 2024-03-11T21:34:09.889 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:34:09 smithi092 ceph-mon[37563]: 2.13 scrub starts 2024-03-11T21:34:09.889 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:34:09 smithi092 ceph-mon[37563]: 2.13 scrub ok 2024-03-11T21:34:09.889 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:34:09 smithi092 ceph-mon[37563]: osdmap e218: 8 total, 8 up, 8 in 2024-03-11T21:34:09.889 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:34:09 smithi092 ceph-mon[37563]: 2.f scrub starts 2024-03-11T21:34:09.889 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:34:09 smithi092 ceph-mon[37563]: 2.f scrub ok 2024-03-11T21:34:09.890 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:34:09 smithi092 ceph-mon[37563]: purged_snaps scrub starts 2024-03-11T21:34:09.890 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:34:09 smithi092 ceph-mon[37563]: purged_snaps scrub ok 2024-03-11T21:34:09.891 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:34:09.921 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:34:09.922 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:34:09.922 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:34:09.990 DEBUG:teuthology.orchestra.run.smithi092:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid 00db0016-dfed-11ee-95c8-87774f69a715 -- ceph osd reweight 5 1 2024-03-11T21:34:10.023 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:34:10.053 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:34:10.053 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:34:10.054 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:34:10.076 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:34:09 smithi097 ceph-mon[33317]: 8.3 scrub starts 2024-03-11T21:34:10.076 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:34:09 smithi097 ceph-mon[33317]: 2.13 scrub starts 2024-03-11T21:34:10.076 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:34:09 smithi097 ceph-mon[33317]: 2.13 scrub ok 2024-03-11T21:34:10.076 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:34:09 smithi097 ceph-mon[33317]: osdmap e218: 8 total, 8 up, 8 in 2024-03-11T21:34:10.076 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:34:09 smithi097 ceph-mon[33317]: 2.f scrub starts 2024-03-11T21:34:10.076 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:34:09 smithi097 ceph-mon[33317]: 2.f scrub ok 2024-03-11T21:34:10.076 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:34:09 smithi097 ceph-mon[33317]: purged_snaps scrub starts 2024-03-11T21:34:10.076 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:34:09 smithi097 ceph-mon[33317]: purged_snaps scrub ok 2024-03-11T21:34:10.155 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:34:10.214 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:34:10.215 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:34:10.216 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:34:10.316 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:34:10.355 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:34:10.356 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:34:10.356 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:34:10.457 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:34:10.487 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:34:10.488 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:34:10.488 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:34:10.589 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:34:10.627 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:34:10.627 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:34:10.628 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:34:10.728 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:34:10.762 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:34:10.763 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:34:10.763 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:34:10.860 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:34:10 smithi092 ceph-mon[37563]: 8.3 scrub ok 2024-03-11T21:34:10.861 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:34:10 smithi092 ceph-mon[37563]: pgmap v368: 158 pgs: 1 peering, 1 active+remapped+backfill_toofull, 1 active+remapped+backfilling, 155 active+clean; 5.2 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 870 B/s rd, 0 B/s wr, 91 op/s; 8708/254643 objects misplaced (3.420%) 2024-03-11T21:34:10.861 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:34:10 smithi092 ceph-mon[37563]: 5.16 scrub starts 2024-03-11T21:34:10.861 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:34:10 smithi092 ceph-mon[37563]: 5.16 scrub ok 2024-03-11T21:34:10.861 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:34:10 smithi092 ceph-mon[37563]: 5.11 scrub starts 2024-03-11T21:34:10.861 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:34:10 smithi092 ceph-mon[37563]: 5.11 scrub ok 2024-03-11T21:34:10.861 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:34:10 smithi092 ceph-mon[37563]: osdmap e219: 8 total, 8 up, 8 in 2024-03-11T21:34:10.861 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:34:10 smithi092 ceph-mon[37563]: 8.17 scrub starts 2024-03-11T21:34:10.861 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:34:10 smithi092 ceph-mon[37563]: 8.17 scrub ok 2024-03-11T21:34:10.862 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:34:10 smithi092 ceph-mon[29572]: 8.3 scrub ok 2024-03-11T21:34:10.862 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:34:10 smithi092 ceph-mon[29572]: pgmap v368: 158 pgs: 1 peering, 1 active+remapped+backfill_toofull, 1 active+remapped+backfilling, 155 active+clean; 5.2 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 870 B/s rd, 0 B/s wr, 91 op/s; 8708/254643 objects misplaced (3.420%) 2024-03-11T21:34:10.862 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:34:10 smithi092 ceph-mon[29572]: 5.16 scrub starts 2024-03-11T21:34:10.862 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:34:10 smithi092 ceph-mon[29572]: 5.16 scrub ok 2024-03-11T21:34:10.862 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:34:10 smithi092 ceph-mon[29572]: 5.11 scrub starts 2024-03-11T21:34:10.862 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:34:10 smithi092 ceph-mon[29572]: 5.11 scrub ok 2024-03-11T21:34:10.862 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:34:10 smithi092 ceph-mon[29572]: osdmap e219: 8 total, 8 up, 8 in 2024-03-11T21:34:10.862 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:34:10 smithi092 ceph-mon[29572]: 8.17 scrub starts 2024-03-11T21:34:10.862 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:34:10 smithi092 ceph-mon[29572]: 8.17 scrub ok 2024-03-11T21:34:10.863 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:34:10.915 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:34:10.915 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:34:10.916 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:34:11.016 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:34:11.058 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:34:11.059 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:34:11.059 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:34:11.076 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:34:10 smithi097 ceph-mon[33317]: 8.3 scrub ok 2024-03-11T21:34:11.076 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:34:10 smithi097 ceph-mon[33317]: pgmap v368: 158 pgs: 1 peering, 1 active+remapped+backfill_toofull, 1 active+remapped+backfilling, 155 active+clean; 5.2 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 870 B/s rd, 0 B/s wr, 91 op/s; 8708/254643 objects misplaced (3.420%) 2024-03-11T21:34:11.076 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:34:10 smithi097 ceph-mon[33317]: 5.16 scrub starts 2024-03-11T21:34:11.076 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:34:10 smithi097 ceph-mon[33317]: 5.16 scrub ok 2024-03-11T21:34:11.076 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:34:10 smithi097 ceph-mon[33317]: 5.11 scrub starts 2024-03-11T21:34:11.076 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:34:10 smithi097 ceph-mon[33317]: 5.11 scrub ok 2024-03-11T21:34:11.076 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:34:10 smithi097 ceph-mon[33317]: osdmap e219: 8 total, 8 up, 8 in 2024-03-11T21:34:11.076 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:34:10 smithi097 ceph-mon[33317]: 8.17 scrub starts 2024-03-11T21:34:11.076 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:34:10 smithi097 ceph-mon[33317]: 8.17 scrub ok 2024-03-11T21:34:11.160 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:34:11.197 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:34:11.198 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:34:11.199 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:34:11.299 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:34:11.332 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:34:11.332 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:34:11.333 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:34:11.433 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:34:11.461 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:34:11.462 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:34:11.462 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:34:11.495 INFO:teuthology.orchestra.run.smithi092.stderr:Inferring config /var/lib/ceph/00db0016-dfed-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T21:34:11.563 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:34:11.594 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:34:11.595 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:34:11.596 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:34:11.696 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:34:11.726 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:34:11.727 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:34:11.727 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:34:11.828 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:34:11.851 INFO:teuthology.orchestra.run.smithi092.stderr:Inferring config /var/lib/ceph/00db0016-dfed-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T21:34:11.861 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:34:11.862 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:34:11.862 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:34:11.963 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:34:11.967 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:34:11 smithi092 ceph-mon[37563]: 8.10 scrub starts 2024-03-11T21:34:11.967 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:34:11 smithi092 ceph-mon[37563]: 4.1c scrub starts 2024-03-11T21:34:11.967 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:34:11 smithi092 ceph-mon[37563]: 4.1c scrub ok 2024-03-11T21:34:11.967 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:34:11 smithi092 ceph-mon[37563]: 2.7 scrub starts 2024-03-11T21:34:11.967 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:34:11 smithi092 ceph-mon[37563]: 2.7 scrub ok 2024-03-11T21:34:11.967 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:34:11 smithi092 ceph-mon[37563]: 6.9 scrub starts 2024-03-11T21:34:11.967 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:34:11 smithi092 ceph-mon[37563]: 8.14 scrub starts 2024-03-11T21:34:11.967 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:34:11 smithi092 ceph-mon[37563]: 6.9 scrub ok 2024-03-11T21:34:11.968 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:34:11 smithi092 ceph-mon[37563]: 8.10 scrub ok 2024-03-11T21:34:11.968 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:34:11 smithi092 ceph-mon[29572]: 8.10 scrub starts 2024-03-11T21:34:11.968 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:34:11 smithi092 ceph-mon[29572]: 4.1c scrub starts 2024-03-11T21:34:11.968 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:34:11 smithi092 ceph-mon[29572]: 4.1c scrub ok 2024-03-11T21:34:11.968 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:34:11 smithi092 ceph-mon[29572]: 2.7 scrub starts 2024-03-11T21:34:11.968 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:34:11 smithi092 ceph-mon[29572]: 2.7 scrub ok 2024-03-11T21:34:11.968 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:34:11 smithi092 ceph-mon[29572]: 6.9 scrub starts 2024-03-11T21:34:11.968 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:34:11 smithi092 ceph-mon[29572]: 8.14 scrub starts 2024-03-11T21:34:11.968 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:34:11 smithi092 ceph-mon[29572]: 6.9 scrub ok 2024-03-11T21:34:11.968 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:34:11 smithi092 ceph-mon[29572]: 8.10 scrub ok 2024-03-11T21:34:12.012 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:34:12.012 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:34:12.013 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:34:12.076 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:34:11 smithi097 ceph-mon[33317]: 8.10 scrub starts 2024-03-11T21:34:12.076 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:34:11 smithi097 ceph-mon[33317]: 4.1c scrub starts 2024-03-11T21:34:12.076 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:34:11 smithi097 ceph-mon[33317]: 4.1c scrub ok 2024-03-11T21:34:12.076 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:34:11 smithi097 ceph-mon[33317]: 2.7 scrub starts 2024-03-11T21:34:12.076 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:34:11 smithi097 ceph-mon[33317]: 2.7 scrub ok 2024-03-11T21:34:12.076 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:34:11 smithi097 ceph-mon[33317]: 6.9 scrub starts 2024-03-11T21:34:12.076 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:34:11 smithi097 ceph-mon[33317]: 8.14 scrub starts 2024-03-11T21:34:12.076 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:34:11 smithi097 ceph-mon[33317]: 6.9 scrub ok 2024-03-11T21:34:12.076 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:34:11 smithi097 ceph-mon[33317]: 8.10 scrub ok 2024-03-11T21:34:12.113 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:34:12.165 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:34:12.165 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:34:12.166 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:34:12.266 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:34:12.299 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:34:12.299 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:34:12.300 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:34:12.401 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:34:12.441 INFO:teuthology.orchestra.run.smithi092.stdout:ERROR: (22) Invalid argument 2024-03-11T21:34:12.441 INFO:teuthology.orchestra.run.smithi092.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-11T21:34:12.441 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:34:12.442 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:34:12.442 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:34:12.542 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:34:12.578 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:34:12.580 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:34:12.580 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:34:12.681 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:34:12.694 INFO:journalctl@ceph.iscsi.iscsi.a.smithi097.stdout:Mar 11 21:34:12 smithi097 ceph-00db0016-dfed-11ee-95c8-87774f69a715-iscsi-iscsi-a[60943]: debug there is no tcmu-runner data available 2024-03-11T21:34:12.719 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:34:12.719 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:34:12.719 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:34:12.822 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:34:12.854 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:34:12.854 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:34:12.855 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:34:12.860 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:34:12 smithi092 ceph-mon[29572]: pgmap v370: 158 pgs: 1 peering, 1 active+remapped+backfill_toofull, 1 active+remapped+backfilling, 155 active+clean; 5.2 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 1.1 KiB/s rd, 0 B/s wr, 120 op/s; 8708/254643 objects misplaced (3.420%) 2024-03-11T21:34:12.860 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:34:12 smithi092 ceph-mon[29572]: 8.14 scrub ok 2024-03-11T21:34:12.860 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:34:12 smithi092 ceph-mon[29572]: 5.3 scrub starts 2024-03-11T21:34:12.860 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:34:12 smithi092 ceph-mon[29572]: 5.3 scrub ok 2024-03-11T21:34:12.860 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:34:12 smithi092 ceph-mon[29572]: 5.1c scrub starts 2024-03-11T21:34:12.860 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:34:12 smithi092 ceph-mon[29572]: 5.1c scrub ok 2024-03-11T21:34:12.860 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:34:12 smithi092 ceph-mon[37563]: pgmap v370: 158 pgs: 1 peering, 1 active+remapped+backfill_toofull, 1 active+remapped+backfilling, 155 active+clean; 5.2 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 1.1 KiB/s rd, 0 B/s wr, 120 op/s; 8708/254643 objects misplaced (3.420%) 2024-03-11T21:34:12.860 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:34:12 smithi092 ceph-mon[37563]: 8.14 scrub ok 2024-03-11T21:34:12.861 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:34:12 smithi092 ceph-mon[37563]: 5.3 scrub starts 2024-03-11T21:34:12.861 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:34:12 smithi092 ceph-mon[37563]: 5.3 scrub ok 2024-03-11T21:34:12.861 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:34:12 smithi092 ceph-mon[37563]: 5.1c scrub starts 2024-03-11T21:34:12.861 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:34:12 smithi092 ceph-mon[37563]: 5.1c scrub ok 2024-03-11T21:34:12.955 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:34:13.003 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:34:13.004 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:34:13.004 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:34:13.076 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:34:12 smithi097 ceph-mon[33317]: pgmap v370: 158 pgs: 1 peering, 1 active+remapped+backfill_toofull, 1 active+remapped+backfilling, 155 active+clean; 5.2 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 1.1 KiB/s rd, 0 B/s wr, 120 op/s; 8708/254643 objects misplaced (3.420%) 2024-03-11T21:34:13.076 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:34:12 smithi097 ceph-mon[33317]: 8.14 scrub ok 2024-03-11T21:34:13.076 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:34:12 smithi097 ceph-mon[33317]: 5.3 scrub starts 2024-03-11T21:34:13.076 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:34:12 smithi097 ceph-mon[33317]: 5.3 scrub ok 2024-03-11T21:34:13.076 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:34:12 smithi097 ceph-mon[33317]: 5.1c scrub starts 2024-03-11T21:34:13.076 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:34:12 smithi097 ceph-mon[33317]: 5.1c scrub ok 2024-03-11T21:34:13.105 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:34:13.136 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:34:13.137 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:34:13.137 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:34:13.238 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:34:13.267 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:34:13.268 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:34:13.268 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:34:13.369 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:34:13.414 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:34:13.415 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:34:13.415 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:34:13.516 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:34:13.546 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:34:13.546 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:34:13.547 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:34:13.648 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:34:13.687 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:34:13.688 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:34:13.688 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:34:13.789 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:34:13.832 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:34:13.832 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:34:13.833 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:34:13.933 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:34:13.974 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:34:13.975 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:34:13.975 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:34:14.069 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:34:13 smithi092 ceph-mon[29572]: 2.10 scrub starts 2024-03-11T21:34:14.069 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:34:13 smithi092 ceph-mon[29572]: 2.10 scrub ok 2024-03-11T21:34:14.069 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:34:13 smithi092 ceph-mon[29572]: 4.1f scrub starts 2024-03-11T21:34:14.069 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:34:13 smithi092 ceph-mon[29572]: 4.1f scrub ok 2024-03-11T21:34:14.069 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:34:13 smithi092 ceph-mon[29572]: from='client.24469 v1:172.21.15.97:0/404208310' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:34:14.069 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:34:13 smithi092 ceph-mon[29572]: 5.1d scrub starts 2024-03-11T21:34:14.069 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:34:13 smithi092 ceph-mon[29572]: 5.1d scrub ok 2024-03-11T21:34:14.070 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:34:13 smithi092 ceph-mon[29572]: 6.e scrub starts 2024-03-11T21:34:14.070 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:34:13 smithi092 ceph-mon[29572]: 6.e scrub ok 2024-03-11T21:34:14.070 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:34:13 smithi092 ceph-mon[37563]: 2.10 scrub starts 2024-03-11T21:34:14.070 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:34:13 smithi092 ceph-mon[37563]: 2.10 scrub ok 2024-03-11T21:34:14.070 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:34:13 smithi092 ceph-mon[37563]: 4.1f scrub starts 2024-03-11T21:34:14.070 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:34:13 smithi092 ceph-mon[37563]: 4.1f scrub ok 2024-03-11T21:34:14.070 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:34:13 smithi092 ceph-mon[37563]: from='client.24469 v1:172.21.15.97:0/404208310' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:34:14.070 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:34:13 smithi092 ceph-mon[37563]: 5.1d scrub starts 2024-03-11T21:34:14.070 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:34:13 smithi092 ceph-mon[37563]: 5.1d scrub ok 2024-03-11T21:34:14.070 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:34:13 smithi092 ceph-mon[37563]: 6.e scrub starts 2024-03-11T21:34:14.070 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:34:13 smithi092 ceph-mon[37563]: 6.e scrub ok 2024-03-11T21:34:14.075 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:34:14.076 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:34:13 smithi097 ceph-mon[33317]: 2.10 scrub starts 2024-03-11T21:34:14.076 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:34:13 smithi097 ceph-mon[33317]: 2.10 scrub ok 2024-03-11T21:34:14.076 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:34:13 smithi097 ceph-mon[33317]: 4.1f scrub starts 2024-03-11T21:34:14.076 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:34:13 smithi097 ceph-mon[33317]: 4.1f scrub ok 2024-03-11T21:34:14.076 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:34:13 smithi097 ceph-mon[33317]: from='client.24469 v1:172.21.15.97:0/404208310' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:34:14.076 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:34:13 smithi097 ceph-mon[33317]: 5.1d scrub starts 2024-03-11T21:34:14.076 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:34:13 smithi097 ceph-mon[33317]: 5.1d scrub ok 2024-03-11T21:34:14.077 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:34:13 smithi097 ceph-mon[33317]: 6.e scrub starts 2024-03-11T21:34:14.077 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:34:13 smithi097 ceph-mon[33317]: 6.e scrub ok 2024-03-11T21:34:14.120 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:34:14.121 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:34:14.121 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:34:14.222 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:34:14.256 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:34:14.257 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:34:14.257 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:34:14.358 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:34:14.396 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:34:14.397 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:34:14.397 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:34:14.498 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:34:14.525 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:34:14.526 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:34:14.526 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:34:14.627 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:34:14.659 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:34:14.664 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:34:14.665 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:34:14.764 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:34:14 smithi092 ceph-mon[29572]: pgmap v371: 158 pgs: 1 peering, 1 active+remapped+backfill_toofull, 156 active+clean; 5.0 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 1.7 KiB/s rd, 1 op/s; 7890/244281 objects misplaced (3.230%) 2024-03-11T21:34:14.764 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:34:14 smithi092 ceph-mon[29572]: 8.e scrub starts 2024-03-11T21:34:14.764 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:34:14 smithi092 ceph-mon[29572]: 6.13 scrub starts 2024-03-11T21:34:14.764 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:34:14 smithi092 ceph-mon[29572]: 6.13 scrub ok 2024-03-11T21:34:14.765 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:34:14 smithi092 ceph-mon[37563]: pgmap v371: 158 pgs: 1 peering, 1 active+remapped+backfill_toofull, 156 active+clean; 5.0 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 1.7 KiB/s rd, 1 op/s; 7890/244281 objects misplaced (3.230%) 2024-03-11T21:34:14.765 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:34:14 smithi092 ceph-mon[37563]: 8.e scrub starts 2024-03-11T21:34:14.765 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:34:14 smithi092 ceph-mon[37563]: 6.13 scrub starts 2024-03-11T21:34:14.765 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:34:14 smithi092 ceph-mon[37563]: 6.13 scrub ok 2024-03-11T21:34:14.765 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:34:14.793 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:34:14.794 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:34:14.795 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:34:14.895 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:34:14.929 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:34:14.930 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:34:14.930 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:34:15.031 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:34:15.062 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:34:15.062 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:34:15.063 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:34:15.076 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:34:14 smithi097 ceph-mon[33317]: pgmap v371: 158 pgs: 1 peering, 1 active+remapped+backfill_toofull, 156 active+clean; 5.0 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 1.7 KiB/s rd, 1 op/s; 7890/244281 objects misplaced (3.230%) 2024-03-11T21:34:15.076 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:34:14 smithi097 ceph-mon[33317]: 8.e scrub starts 2024-03-11T21:34:15.076 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:34:14 smithi097 ceph-mon[33317]: 6.13 scrub starts 2024-03-11T21:34:15.076 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:34:14 smithi097 ceph-mon[33317]: 6.13 scrub ok 2024-03-11T21:34:15.163 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:34:15.194 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:34:15.195 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:34:15.195 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:34:15.296 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:34:15.325 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:34:15.326 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:34:15.326 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:34:15.427 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:34:15.466 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:34:15.467 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:34:15.467 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:34:15.568 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:34:15.599 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:34:15.599 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:34:15.600 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:34:15.701 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:34:15.742 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:34:15.742 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:34:15.743 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:34:15.844 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:34:15.852 DEBUG:teuthology.orchestra.run.smithi092:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid 00db0016-dfed-11ee-95c8-87774f69a715 -- ceph daemon osd.1 dump_ops_in_flight 2024-03-11T21:34:15.889 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:34:15.890 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:34:15.890 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:34:15.991 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:34:16.042 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:34:16.042 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:34:16.043 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:34:16.144 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:34:16.191 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:34:16.192 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:34:16.192 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:34:16.294 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:34:16.355 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:34:16.356 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:34:16.357 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:34:16.381 INFO:teuthology.orchestra.run.smithi092.stdout:osd.0: {} 2024-03-11T21:34:16.427 INFO:teuthology.orchestra.run.smithi092.stdout:osd.1: {} 2024-03-11T21:34:16.457 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:34:16.468 INFO:teuthology.orchestra.run.smithi092.stdout:osd.2: {} 2024-03-11T21:34:16.503 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:34:16.503 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:34:16.503 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:34:16.508 INFO:teuthology.orchestra.run.smithi092.stdout:osd.3: {} 2024-03-11T21:34:16.545 INFO:teuthology.orchestra.run.smithi092.stdout:osd.4: {} 2024-03-11T21:34:16.585 INFO:teuthology.orchestra.run.smithi092.stdout:osd.5: {} 2024-03-11T21:34:16.604 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:34:16.630 INFO:teuthology.orchestra.run.smithi092.stdout:osd.6: {} 2024-03-11T21:34:16.651 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:34:16.652 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:34:16.652 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:34:16.672 INFO:teuthology.orchestra.run.smithi092.stdout:osd.7: {} 2024-03-11T21:34:16.677 INFO:teuthology.orchestra.run.smithi092.stderr:osd.0: osd_enable_op_tracker = '' 2024-03-11T21:34:16.678 INFO:teuthology.orchestra.run.smithi092.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-11T21:34:16.678 INFO:teuthology.orchestra.run.smithi092.stderr:osd.2: osd_enable_op_tracker = '' 2024-03-11T21:34:16.678 INFO:teuthology.orchestra.run.smithi092.stderr:osd.3: osd_enable_op_tracker = '' 2024-03-11T21:34:16.678 INFO:teuthology.orchestra.run.smithi092.stderr:osd.4: osd_enable_op_tracker = '' 2024-03-11T21:34:16.678 INFO:teuthology.orchestra.run.smithi092.stderr:osd.5: osd_enable_op_tracker = '' 2024-03-11T21:34:16.678 INFO:teuthology.orchestra.run.smithi092.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-11T21:34:16.678 INFO:teuthology.orchestra.run.smithi092.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-11T21:34:16.716 INFO:teuthology.orchestra.run.smithi092.stderr:reweighted osd.5 to 1 (10000) 2024-03-11T21:34:16.752 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:34:16.779 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:34:16.780 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:34:16.780 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:34:16.881 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:34:16.917 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:34:16.918 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:34:16.919 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:34:16.960 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:34:16 smithi092 ceph-mon[29572]: pgmap v372: 158 pgs: 1 active+remapped+backfill_toofull, 157 active+clean; 4.9 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 687 B/s rd, 0 op/s; 7518/240969 objects misplaced (3.120%) 2024-03-11T21:34:16.960 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:34:16 smithi092 ceph-mon[29572]: 6.14 scrub starts 2024-03-11T21:34:16.961 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:34:16 smithi092 ceph-mon[29572]: 6.14 scrub ok 2024-03-11T21:34:16.961 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:34:16 smithi092 ceph-mon[29572]: 4.a scrub starts 2024-03-11T21:34:16.961 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:34:16 smithi092 ceph-mon[29572]: 4.a scrub ok 2024-03-11T21:34:16.961 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:34:16 smithi092 ceph-mon[29572]: 5.12 scrub starts 2024-03-11T21:34:16.961 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:34:16 smithi092 ceph-mon[29572]: 5.12 scrub ok 2024-03-11T21:34:16.961 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:34:16 smithi092 ceph-mon[29572]: 8.e scrub ok 2024-03-11T21:34:16.962 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:34:16 smithi092 ceph-mon[29572]: 5.18 scrub starts 2024-03-11T21:34:16.962 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:34:16 smithi092 ceph-mon[29572]: 5.18 scrub ok 2024-03-11T21:34:16.962 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:34:16 smithi092 ceph-mon[29572]: from='client.? v1:172.21.15.92:0/197331733' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]: dispatch 2024-03-11T21:34:16.962 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:34:16 smithi092 ceph-mon[29572]: from='client.? v1:172.21.15.92:0/1235688699' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-11T21:34:16.963 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:34:16 smithi092 ceph-mon[37563]: pgmap v372: 158 pgs: 1 active+remapped+backfill_toofull, 157 active+clean; 4.9 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 687 B/s rd, 0 op/s; 7518/240969 objects misplaced (3.120%) 2024-03-11T21:34:16.963 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:34:16 smithi092 ceph-mon[37563]: 6.14 scrub starts 2024-03-11T21:34:16.963 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:34:16 smithi092 ceph-mon[37563]: 6.14 scrub ok 2024-03-11T21:34:16.963 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:34:16 smithi092 ceph-mon[37563]: 4.a scrub starts 2024-03-11T21:34:16.963 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:34:16 smithi092 ceph-mon[37563]: 4.a scrub ok 2024-03-11T21:34:16.963 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:34:16 smithi092 ceph-mon[37563]: 5.12 scrub starts 2024-03-11T21:34:16.964 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:34:16 smithi092 ceph-mon[37563]: 5.12 scrub ok 2024-03-11T21:34:16.964 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:34:16 smithi092 ceph-mon[37563]: 8.e scrub ok 2024-03-11T21:34:16.964 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:34:16 smithi092 ceph-mon[37563]: 5.18 scrub starts 2024-03-11T21:34:16.964 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:34:16 smithi092 ceph-mon[37563]: 5.18 scrub ok 2024-03-11T21:34:16.964 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:34:16 smithi092 ceph-mon[37563]: from='client.? v1:172.21.15.92:0/197331733' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]: dispatch 2024-03-11T21:34:16.964 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:34:16 smithi092 ceph-mon[37563]: from='client.? v1:172.21.15.92:0/1235688699' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-11T21:34:17.019 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:34:17.051 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:34:17.052 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:34:17.053 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:34:17.076 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:34:16 smithi097 ceph-mon[33317]: pgmap v372: 158 pgs: 1 active+remapped+backfill_toofull, 157 active+clean; 4.9 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 687 B/s rd, 0 op/s; 7518/240969 objects misplaced (3.120%) 2024-03-11T21:34:17.077 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:34:16 smithi097 ceph-mon[33317]: 6.14 scrub starts 2024-03-11T21:34:17.077 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:34:16 smithi097 ceph-mon[33317]: 6.14 scrub ok 2024-03-11T21:34:17.077 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:34:16 smithi097 ceph-mon[33317]: 4.a scrub starts 2024-03-11T21:34:17.077 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:34:16 smithi097 ceph-mon[33317]: 4.a scrub ok 2024-03-11T21:34:17.077 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:34:16 smithi097 ceph-mon[33317]: 5.12 scrub starts 2024-03-11T21:34:17.077 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:34:16 smithi097 ceph-mon[33317]: 5.12 scrub ok 2024-03-11T21:34:17.078 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:34:16 smithi097 ceph-mon[33317]: 8.e scrub ok 2024-03-11T21:34:17.078 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:34:16 smithi097 ceph-mon[33317]: 5.18 scrub starts 2024-03-11T21:34:17.079 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:34:16 smithi097 ceph-mon[33317]: 5.18 scrub ok 2024-03-11T21:34:17.079 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:34:16 smithi097 ceph-mon[33317]: from='client.? v1:172.21.15.92:0/197331733' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]: dispatch 2024-03-11T21:34:17.079 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:34:16 smithi097 ceph-mon[33317]: from='client.? v1:172.21.15.92:0/1235688699' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-11T21:34:17.153 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:34:17.185 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:34:17.186 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:34:17.186 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:34:17.287 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:34:17.317 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:34:17.318 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:34:17.318 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:34:17.419 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:34:17.465 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:34:17.466 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:34:17.466 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:34:17.567 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:34:17.606 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:34:17.607 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:34:17.607 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:34:17.708 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:34:17.738 INFO:teuthology.orchestra.run.smithi092.stderr:nodeep-scrub is unset 2024-03-11T21:34:17.749 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:34:17.750 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:34:17.750 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:34:17.851 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:34:17.873 DEBUG:teuthology.orchestra.run.smithi092:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid 00db0016-dfed-11ee-95c8-87774f69a715 -- ceph osd reweight 6 1 2024-03-11T21:34:17.898 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:34:17.898 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:34:17.898 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:34:17.999 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:34:18.017 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:34:17 smithi092 ceph-mon[29572]: 6.1c scrub starts 2024-03-11T21:34:18.017 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:34:17 smithi092 ceph-mon[29572]: 6.1c scrub ok 2024-03-11T21:34:18.017 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:34:17 smithi092 ceph-mon[29572]: 2.c scrub starts 2024-03-11T21:34:18.017 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:34:17 smithi092 ceph-mon[29572]: 2.c scrub ok 2024-03-11T21:34:18.017 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:34:17 smithi092 ceph-mon[29572]: 2.5 scrub starts 2024-03-11T21:34:18.017 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:34:17 smithi092 ceph-mon[29572]: 2.5 scrub ok 2024-03-11T21:34:18.017 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:34:17 smithi092 ceph-mon[29572]: from='client.? v1:172.21.15.92:0/197331733' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]': finished 2024-03-11T21:34:18.017 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:34:17 smithi092 ceph-mon[29572]: osdmap e220: 8 total, 8 up, 8 in 2024-03-11T21:34:18.017 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:34:17 smithi092 ceph-mon[29572]: 4.0 scrub starts 2024-03-11T21:34:18.017 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:34:17 smithi092 ceph-mon[29572]: 4.0 scrub ok 2024-03-11T21:34:18.017 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:34:17 smithi092 ceph-mon[29572]: from='client.? v1:172.21.15.92:0/3849450407' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-11T21:34:18.018 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:34:17 smithi092 ceph-mon[29572]: from='client.25180 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-11T21:34:18.018 INFO:journalctl@ceph.mgr.y.smithi092.stdout:Mar 11 21:34:17 smithi092 ceph-00db0016-dfed-11ee-95c8-87774f69a715-mgr-y[29801]: ::ffff:172.21.15.97 - - [11/Mar/2024:21:34:17] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-11T21:34:18.018 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:34:17 smithi092 ceph-mon[37563]: 6.1c scrub starts 2024-03-11T21:34:18.018 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:34:17 smithi092 ceph-mon[37563]: 6.1c scrub ok 2024-03-11T21:34:18.018 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:34:17 smithi092 ceph-mon[37563]: 2.c scrub starts 2024-03-11T21:34:18.018 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:34:17 smithi092 ceph-mon[37563]: 2.c scrub ok 2024-03-11T21:34:18.018 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:34:17 smithi092 ceph-mon[37563]: 2.5 scrub starts 2024-03-11T21:34:18.019 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:34:17 smithi092 ceph-mon[37563]: 2.5 scrub ok 2024-03-11T21:34:18.019 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:34:17 smithi092 ceph-mon[37563]: from='client.? v1:172.21.15.92:0/197331733' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]': finished 2024-03-11T21:34:18.019 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:34:17 smithi092 ceph-mon[37563]: osdmap e220: 8 total, 8 up, 8 in 2024-03-11T21:34:18.019 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:34:17 smithi092 ceph-mon[37563]: 4.0 scrub starts 2024-03-11T21:34:18.019 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:34:17 smithi092 ceph-mon[37563]: 4.0 scrub ok 2024-03-11T21:34:18.019 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:34:17 smithi092 ceph-mon[37563]: from='client.? v1:172.21.15.92:0/3849450407' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-11T21:34:18.019 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:34:17 smithi092 ceph-mon[37563]: from='client.25180 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-11T21:34:18.046 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:34:18.047 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:34:18.047 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:34:18.076 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:34:17 smithi097 ceph-mon[33317]: 6.1c scrub starts 2024-03-11T21:34:18.076 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:34:17 smithi097 ceph-mon[33317]: 6.1c scrub ok 2024-03-11T21:34:18.076 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:34:17 smithi097 ceph-mon[33317]: 2.c scrub starts 2024-03-11T21:34:18.076 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:34:17 smithi097 ceph-mon[33317]: 2.c scrub ok 2024-03-11T21:34:18.076 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:34:17 smithi097 ceph-mon[33317]: 2.5 scrub starts 2024-03-11T21:34:18.076 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:34:17 smithi097 ceph-mon[33317]: 2.5 scrub ok 2024-03-11T21:34:18.076 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:34:17 smithi097 ceph-mon[33317]: from='client.? v1:172.21.15.92:0/197331733' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]': finished 2024-03-11T21:34:18.076 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:34:17 smithi097 ceph-mon[33317]: osdmap e220: 8 total, 8 up, 8 in 2024-03-11T21:34:18.077 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:34:17 smithi097 ceph-mon[33317]: 4.0 scrub starts 2024-03-11T21:34:18.077 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:34:17 smithi097 ceph-mon[33317]: 4.0 scrub ok 2024-03-11T21:34:18.077 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:34:17 smithi097 ceph-mon[33317]: from='client.? v1:172.21.15.92:0/3849450407' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-11T21:34:18.077 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:34:17 smithi097 ceph-mon[33317]: from='client.25180 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-11T21:34:18.148 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:34:18.179 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:34:18.180 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:34:18.180 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:34:18.281 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:34:18.314 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:34:18.314 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:34:18.314 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:34:18.415 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:34:18.447 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:34:18.448 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:34:18.448 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:34:18.549 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:34:18.580 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:34:18.581 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:34:18.581 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:34:18.581 INFO:teuthology.orchestra.run.smithi092.stderr:Inferring config /var/lib/ceph/00db0016-dfed-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T21:34:18.682 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:34:18.714 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:34:18.715 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:34:18.716 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:34:18.816 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:34:18.862 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:34:18.862 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:34:18.863 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:34:18.963 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:34:19.006 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:34:19.007 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:34:19.007 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:34:19.076 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:34:18 smithi097 ceph-mon[33317]: pgmap v374: 158 pgs: 1 active+remapped+backfill_toofull, 157 active+clean; 4.8 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 639 B/s rd, 0 op/s; 7518/235506 objects misplaced (3.192%) 2024-03-11T21:34:19.076 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:34:18 smithi097 ceph-mon[33317]: 5.a scrub starts 2024-03-11T21:34:19.076 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:34:18 smithi097 ceph-mon[33317]: 5.a scrub ok 2024-03-11T21:34:19.076 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:34:18 smithi097 ceph-mon[33317]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-03-11T21:34:19.076 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:34:18 smithi097 ceph-mon[33317]: from='client.25180 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-03-11T21:34:19.076 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:34:18 smithi097 ceph-mon[33317]: osdmap e221: 8 total, 8 up, 8 in 2024-03-11T21:34:19.076 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:34:18 smithi097 ceph-mon[33317]: purged_snaps scrub starts 2024-03-11T21:34:19.076 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:34:18 smithi097 ceph-mon[33317]: purged_snaps scrub ok 2024-03-11T21:34:19.108 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:34:19.110 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:34:18 smithi092 ceph-mon[29572]: pgmap v374: 158 pgs: 1 active+remapped+backfill_toofull, 157 active+clean; 4.8 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 639 B/s rd, 0 op/s; 7518/235506 objects misplaced (3.192%) 2024-03-11T21:34:19.110 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:34:18 smithi092 ceph-mon[29572]: 5.a scrub starts 2024-03-11T21:34:19.110 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:34:18 smithi092 ceph-mon[29572]: 5.a scrub ok 2024-03-11T21:34:19.110 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:34:18 smithi092 ceph-mon[29572]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-03-11T21:34:19.110 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:34:18 smithi092 ceph-mon[29572]: from='client.25180 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-03-11T21:34:19.110 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:34:18 smithi092 ceph-mon[29572]: osdmap e221: 8 total, 8 up, 8 in 2024-03-11T21:34:19.110 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:34:18 smithi092 ceph-mon[29572]: purged_snaps scrub starts 2024-03-11T21:34:19.110 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:34:18 smithi092 ceph-mon[29572]: purged_snaps scrub ok 2024-03-11T21:34:19.111 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:34:18 smithi092 ceph-mon[37563]: pgmap v374: 158 pgs: 1 active+remapped+backfill_toofull, 157 active+clean; 4.8 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 639 B/s rd, 0 op/s; 7518/235506 objects misplaced (3.192%) 2024-03-11T21:34:19.111 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:34:18 smithi092 ceph-mon[37563]: 5.a scrub starts 2024-03-11T21:34:19.111 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:34:18 smithi092 ceph-mon[37563]: 5.a scrub ok 2024-03-11T21:34:19.111 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:34:18 smithi092 ceph-mon[37563]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-03-11T21:34:19.111 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:34:18 smithi092 ceph-mon[37563]: from='client.25180 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-03-11T21:34:19.111 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:34:18 smithi092 ceph-mon[37563]: osdmap e221: 8 total, 8 up, 8 in 2024-03-11T21:34:19.111 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:34:18 smithi092 ceph-mon[37563]: purged_snaps scrub starts 2024-03-11T21:34:19.111 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:34:18 smithi092 ceph-mon[37563]: purged_snaps scrub ok 2024-03-11T21:34:19.138 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:34:19.139 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:34:19.139 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:34:19.240 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:34:19.271 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:34:19.272 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:34:19.272 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:34:19.373 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:34:19.415 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:34:19.415 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:34:19.416 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:34:19.517 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:34:19.556 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:34:19.556 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:34:19.557 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:34:19.657 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:34:19.694 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:34:19.695 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:34:19.695 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:34:19.796 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:34:19.840 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:34:19.840 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:34:19.841 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:34:19.941 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:34:19.971 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:34:19.972 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:34:19.972 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:34:19.974 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:34:19 smithi092 ceph-mon[29572]: 2.2 scrub starts 2024-03-11T21:34:19.974 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:34:19 smithi092 ceph-mon[29572]: 2.2 scrub ok 2024-03-11T21:34:19.974 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:34:19 smithi092 ceph-mon[29572]: 2.1 scrub starts 2024-03-11T21:34:19.974 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:34:19 smithi092 ceph-mon[29572]: 2.1 scrub ok 2024-03-11T21:34:19.975 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:34:19 smithi092 ceph-mon[29572]: 5.5 scrub starts 2024-03-11T21:34:19.975 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:34:19 smithi092 ceph-mon[29572]: 5.5 scrub ok 2024-03-11T21:34:19.975 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:34:19 smithi092 ceph-mon[29572]: 5.e scrub starts 2024-03-11T21:34:19.975 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:34:19 smithi092 ceph-mon[29572]: 5.e scrub ok 2024-03-11T21:34:19.975 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:34:19 smithi092 ceph-mon[29572]: osdmap e222: 8 total, 8 up, 8 in 2024-03-11T21:34:19.975 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:34:19 smithi092 ceph-mon[37563]: 2.2 scrub starts 2024-03-11T21:34:19.975 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:34:19 smithi092 ceph-mon[37563]: 2.2 scrub ok 2024-03-11T21:34:19.975 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:34:19 smithi092 ceph-mon[37563]: 2.1 scrub starts 2024-03-11T21:34:19.975 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:34:19 smithi092 ceph-mon[37563]: 2.1 scrub ok 2024-03-11T21:34:19.975 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:34:19 smithi092 ceph-mon[37563]: 5.5 scrub starts 2024-03-11T21:34:19.975 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:34:19 smithi092 ceph-mon[37563]: 5.5 scrub ok 2024-03-11T21:34:19.975 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:34:19 smithi092 ceph-mon[37563]: 5.e scrub starts 2024-03-11T21:34:19.975 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:34:19 smithi092 ceph-mon[37563]: 5.e scrub ok 2024-03-11T21:34:19.976 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:34:19 smithi092 ceph-mon[37563]: osdmap e222: 8 total, 8 up, 8 in 2024-03-11T21:34:20.073 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:34:20.076 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:34:19 smithi097 ceph-mon[33317]: 2.2 scrub starts 2024-03-11T21:34:20.076 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:34:19 smithi097 ceph-mon[33317]: 2.2 scrub ok 2024-03-11T21:34:20.076 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:34:19 smithi097 ceph-mon[33317]: 2.1 scrub starts 2024-03-11T21:34:20.076 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:34:19 smithi097 ceph-mon[33317]: 2.1 scrub ok 2024-03-11T21:34:20.076 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:34:19 smithi097 ceph-mon[33317]: 5.5 scrub starts 2024-03-11T21:34:20.076 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:34:19 smithi097 ceph-mon[33317]: 5.5 scrub ok 2024-03-11T21:34:20.076 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:34:19 smithi097 ceph-mon[33317]: 5.e scrub starts 2024-03-11T21:34:20.077 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:34:19 smithi097 ceph-mon[33317]: 5.e scrub ok 2024-03-11T21:34:20.077 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:34:19 smithi097 ceph-mon[33317]: osdmap e222: 8 total, 8 up, 8 in 2024-03-11T21:34:20.114 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:34:20.115 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:34:20.115 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:34:20.216 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:34:20.253 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:34:20.254 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:34:20.255 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:34:20.355 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:34:20.386 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:34:20.387 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:34:20.387 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:34:20.488 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:34:20.533 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:34:20.534 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:34:20.534 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:34:20.635 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:34:20.687 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:34:20.688 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:34:20.688 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:34:20.788 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:34:20.819 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:34:20.820 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:34:20.820 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:34:20.921 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:34:20.951 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:34:20.952 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:34:20.952 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:34:21.046 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:34:20 smithi092 ceph-mon[29572]: pgmap v377: 158 pgs: 1 active+remapped+backfill_toofull, 157 active+clean; 4.5 GiB data, 18 GiB used, 698 GiB / 715 GiB avail; 852 B/s rd, 0 B/s wr, 428 op/s; 6960/222135 objects misplaced (3.133%) 2024-03-11T21:34:21.046 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:34:20 smithi092 ceph-mon[29572]: 5.9 scrub starts 2024-03-11T21:34:21.046 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:34:20 smithi092 ceph-mon[29572]: 5.9 scrub ok 2024-03-11T21:34:21.047 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:34:20 smithi092 ceph-mon[29572]: 2.e scrub starts 2024-03-11T21:34:21.047 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:34:20 smithi092 ceph-mon[29572]: 2.e scrub ok 2024-03-11T21:34:21.047 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:34:20 smithi092 ceph-mon[29572]: 4.4 scrub starts 2024-03-11T21:34:21.047 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:34:20 smithi092 ceph-mon[29572]: 4.4 scrub ok 2024-03-11T21:34:21.047 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:34:20 smithi092 ceph-mon[37563]: pgmap v377: 158 pgs: 1 active+remapped+backfill_toofull, 157 active+clean; 4.5 GiB data, 18 GiB used, 698 GiB / 715 GiB avail; 852 B/s rd, 0 B/s wr, 428 op/s; 6960/222135 objects misplaced (3.133%) 2024-03-11T21:34:21.047 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:34:20 smithi092 ceph-mon[37563]: 5.9 scrub starts 2024-03-11T21:34:21.047 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:34:20 smithi092 ceph-mon[37563]: 5.9 scrub ok 2024-03-11T21:34:21.047 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:34:20 smithi092 ceph-mon[37563]: 2.e scrub starts 2024-03-11T21:34:21.048 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:34:20 smithi092 ceph-mon[37563]: 2.e scrub ok 2024-03-11T21:34:21.048 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:34:20 smithi092 ceph-mon[37563]: 4.4 scrub starts 2024-03-11T21:34:21.048 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:34:20 smithi092 ceph-mon[37563]: 4.4 scrub ok 2024-03-11T21:34:21.053 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:34:21.076 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:34:20 smithi097 ceph-mon[33317]: pgmap v377: 158 pgs: 1 active+remapped+backfill_toofull, 157 active+clean; 4.5 GiB data, 18 GiB used, 698 GiB / 715 GiB avail; 852 B/s rd, 0 B/s wr, 428 op/s; 6960/222135 objects misplaced (3.133%) 2024-03-11T21:34:21.076 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:34:20 smithi097 ceph-mon[33317]: 5.9 scrub starts 2024-03-11T21:34:21.076 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:34:20 smithi097 ceph-mon[33317]: 5.9 scrub ok 2024-03-11T21:34:21.076 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:34:20 smithi097 ceph-mon[33317]: 2.e scrub starts 2024-03-11T21:34:21.076 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:34:20 smithi097 ceph-mon[33317]: 2.e scrub ok 2024-03-11T21:34:21.076 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:34:20 smithi097 ceph-mon[33317]: 4.4 scrub starts 2024-03-11T21:34:21.076 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:34:20 smithi097 ceph-mon[33317]: 4.4 scrub ok 2024-03-11T21:34:21.096 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:34:21.096 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:34:21.097 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:34:21.197 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:34:21.218 DEBUG:teuthology.orchestra.run.smithi092:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid 00db0016-dfed-11ee-95c8-87774f69a715 -- ceph osd set noscrub 2024-03-11T21:34:21.233 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:34:21.234 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:34:21.234 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:34:21.335 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:34:21.377 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:34:21.378 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:34:21.379 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:34:21.479 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:34:21.512 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:34:21.513 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:34:21.513 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:34:21.614 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:34:21.642 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:34:21.643 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:34:21.644 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:34:21.745 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:34:21.779 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:34:21.780 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:34:21.780 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:34:21.826 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:34:21 smithi097 ceph-mon[33317]: 2.1a scrub starts 2024-03-11T21:34:21.827 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:34:21 smithi097 ceph-mon[33317]: 2.1a scrub ok 2024-03-11T21:34:21.827 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:34:21 smithi097 ceph-mon[33317]: 6.12 scrub starts 2024-03-11T21:34:21.827 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:34:21 smithi097 ceph-mon[33317]: 6.12 scrub ok 2024-03-11T21:34:21.881 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:34:21.905 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:34:21.906 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:34:21.906 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:34:22.007 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:34:22.044 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:34:22.044 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:34:22.045 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:34:22.110 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:34:21 smithi092 ceph-mon[29572]: 2.1a scrub starts 2024-03-11T21:34:22.110 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:34:21 smithi092 ceph-mon[29572]: 2.1a scrub ok 2024-03-11T21:34:22.110 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:34:21 smithi092 ceph-mon[29572]: 6.12 scrub starts 2024-03-11T21:34:22.110 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:34:21 smithi092 ceph-mon[29572]: 6.12 scrub ok 2024-03-11T21:34:22.110 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:34:21 smithi092 ceph-mon[37563]: 2.1a scrub starts 2024-03-11T21:34:22.110 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:34:21 smithi092 ceph-mon[37563]: 2.1a scrub ok 2024-03-11T21:34:22.110 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:34:21 smithi092 ceph-mon[37563]: 6.12 scrub starts 2024-03-11T21:34:22.111 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:34:21 smithi092 ceph-mon[37563]: 6.12 scrub ok 2024-03-11T21:34:22.145 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:34:22.185 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:34:22.186 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:34:22.186 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:34:22.287 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:34:22.324 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:34:22.325 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:34:22.325 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:34:22.426 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:34:22.449 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:34:22.450 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:34:22.450 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:34:22.551 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:34:22.582 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:34:22.582 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:34:22.583 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:34:22.683 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:34:22.721 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:34:22.721 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:34:22.722 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:34:22.767 INFO:journalctl@ceph.iscsi.iscsi.a.smithi097.stdout:Mar 11 21:34:22 smithi097 ceph-00db0016-dfed-11ee-95c8-87774f69a715-iscsi-iscsi-a[60943]: debug there is no tcmu-runner data available 2024-03-11T21:34:22.823 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:34:22.865 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:34:22.866 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:34:22.866 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:34:22.968 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:34:22.995 DEBUG:teuthology.orchestra.run.smithi092:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid 00db0016-dfed-11ee-95c8-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-03-11T21:34:23.012 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:34:23.013 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:34:23.013 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:34:23.076 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:34:22 smithi097 ceph-mon[33317]: pgmap v378: 158 pgs: 1 active+remapped+backfill_toofull, 157 active+clean; 4.5 GiB data, 18 GiB used, 698 GiB / 715 GiB avail; 853 B/s rd, 0 B/s wr, 1.05k op/s; 6960/222135 objects misplaced (3.133%) 2024-03-11T21:34:23.076 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:34:22 smithi097 ceph-mon[33317]: 5.c scrub starts 2024-03-11T21:34:23.076 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:34:22 smithi097 ceph-mon[33317]: 5.c scrub ok 2024-03-11T21:34:23.076 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:34:22 smithi097 ceph-mon[33317]: 4.f scrub starts 2024-03-11T21:34:23.076 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:34:22 smithi097 ceph-mon[33317]: 4.f scrub ok 2024-03-11T21:34:23.076 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:34:22 smithi097 ceph-mon[33317]: from='mgr.14607 ' entity='mgr.y' 2024-03-11T21:34:23.076 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:34:22 smithi097 ceph-mon[33317]: from='mgr.14607 v1:172.21.15.92:0/1702717597' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T21:34:23.076 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:34:22 smithi097 ceph-mon[33317]: 6.15 scrub starts 2024-03-11T21:34:23.096 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:34:22 smithi092 ceph-mon[29572]: pgmap v378: 158 pgs: 1 active+remapped+backfill_toofull, 157 active+clean; 4.5 GiB data, 18 GiB used, 698 GiB / 715 GiB avail; 853 B/s rd, 0 B/s wr, 1.05k op/s; 6960/222135 objects misplaced (3.133%) 2024-03-11T21:34:23.097 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:34:22 smithi092 ceph-mon[29572]: 5.c scrub starts 2024-03-11T21:34:23.097 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:34:22 smithi092 ceph-mon[29572]: 5.c scrub ok 2024-03-11T21:34:23.097 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:34:22 smithi092 ceph-mon[29572]: 4.f scrub starts 2024-03-11T21:34:23.097 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:34:22 smithi092 ceph-mon[29572]: 4.f scrub ok 2024-03-11T21:34:23.097 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:34:22 smithi092 ceph-mon[29572]: from='mgr.14607 ' entity='mgr.y' 2024-03-11T21:34:23.097 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:34:22 smithi092 ceph-mon[29572]: from='mgr.14607 v1:172.21.15.92:0/1702717597' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T21:34:23.097 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:34:22 smithi092 ceph-mon[29572]: 6.15 scrub starts 2024-03-11T21:34:23.098 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:34:22 smithi092 ceph-mon[37563]: pgmap v378: 158 pgs: 1 active+remapped+backfill_toofull, 157 active+clean; 4.5 GiB data, 18 GiB used, 698 GiB / 715 GiB avail; 853 B/s rd, 0 B/s wr, 1.05k op/s; 6960/222135 objects misplaced (3.133%) 2024-03-11T21:34:23.098 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:34:22 smithi092 ceph-mon[37563]: 5.c scrub starts 2024-03-11T21:34:23.098 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:34:22 smithi092 ceph-mon[37563]: 5.c scrub ok 2024-03-11T21:34:23.098 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:34:22 smithi092 ceph-mon[37563]: 4.f scrub starts 2024-03-11T21:34:23.098 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:34:22 smithi092 ceph-mon[37563]: 4.f scrub ok 2024-03-11T21:34:23.098 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:34:22 smithi092 ceph-mon[37563]: from='mgr.14607 ' entity='mgr.y' 2024-03-11T21:34:23.098 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:34:22 smithi092 ceph-mon[37563]: from='mgr.14607 v1:172.21.15.92:0/1702717597' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T21:34:23.098 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:34:22 smithi092 ceph-mon[37563]: 6.15 scrub starts 2024-03-11T21:34:23.114 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:34:23.147 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:34:23.147 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:34:23.147 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:34:23.224 INFO:teuthology.orchestra.run.smithi092.stdout:{ 2024-03-11T21:34:23.225 INFO:teuthology.orchestra.run.smithi092.stdout: "ops": [ 2024-03-11T21:34:23.225 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:34:23.225 INFO:teuthology.orchestra.run.smithi092.stdout: "description": "osd_op(client.15150.0:173320 8.3 8:c5538a32:::benchmark_data_smithi092_104865_object52819:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e222)", 2024-03-11T21:34:23.225 INFO:teuthology.orchestra.run.smithi092.stdout: "initiated_at": "2024-03-11T21:34:22.555281+0000", 2024-03-11T21:34:23.225 INFO:teuthology.orchestra.run.smithi092.stdout: "age": 0.0048546149999999996, 2024-03-11T21:34:23.225 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.0048868499999999999, 2024-03-11T21:34:23.225 INFO:teuthology.orchestra.run.smithi092.stdout: "type_data": { 2024-03-11T21:34:23.225 INFO:teuthology.orchestra.run.smithi092.stdout: "flag_point": "waiting for sub ops", 2024-03-11T21:34:23.225 INFO:teuthology.orchestra.run.smithi092.stdout: "client_info": { 2024-03-11T21:34:23.225 INFO:teuthology.orchestra.run.smithi092.stdout: "client": "client.15150", 2024-03-11T21:34:23.225 INFO:teuthology.orchestra.run.smithi092.stdout: "client_addr": "v1:172.21.15.92:0/436606717", 2024-03-11T21:34:23.225 INFO:teuthology.orchestra.run.smithi092.stdout: "tid": 173320 2024-03-11T21:34:23.225 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:34:23.225 INFO:teuthology.orchestra.run.smithi092.stdout: "events": [ 2024-03-11T21:34:23.226 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:34:23.226 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "initiated", 2024-03-11T21:34:23.226 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:34:22.555281+0000", 2024-03-11T21:34:23.226 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-03-11T21:34:23.226 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:34:23.226 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:34:23.226 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "header_read", 2024-03-11T21:34:23.226 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:34:22.555281+0000", 2024-03-11T21:34:23.226 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-03-11T21:34:23.226 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:34:23.226 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:34:23.226 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "throttled", 2024-03-11T21:34:23.227 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:34:22.555283+0000", 2024-03-11T21:34:23.227 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.3480000000000001e-06 2024-03-11T21:34:23.227 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:34:23.227 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:34:23.227 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "all_read", 2024-03-11T21:34:23.227 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:34:22.555288+0000", 2024-03-11T21:34:23.227 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 4.9760000000000003e-06 2024-03-11T21:34:23.227 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:34:23.227 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:34:23.227 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "dispatched", 2024-03-11T21:34:23.227 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:34:22.555296+0000", 2024-03-11T21:34:23.227 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 8.7369999999999995e-06 2024-03-11T21:34:23.227 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:34:23.227 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:34:23.227 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "queued_for_pg", 2024-03-11T21:34:23.228 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:34:22.555302+0000", 2024-03-11T21:34:23.228 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 5.5969999999999999e-06 2024-03-11T21:34:23.228 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:34:23.228 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:34:23.228 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-03-11T21:34:23.228 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:34:22.555340+0000", 2024-03-11T21:34:23.228 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 3.8145000000000002e-05 2024-03-11T21:34:23.228 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:34:23.228 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:34:23.228 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "started", 2024-03-11T21:34:23.228 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:34:22.555427+0000", 2024-03-11T21:34:23.228 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 8.7198000000000001e-05 2024-03-11T21:34:23.228 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:34:23.229 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:34:23.229 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "waiting for subops from 5,6", 2024-03-11T21:34:23.229 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:34:22.555938+0000", 2024-03-11T21:34:23.229 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.000510445 2024-03-11T21:34:23.229 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-03-11T21:34:23.229 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-03-11T21:34:23.229 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-03-11T21:34:23.229 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:34:23.229 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:34:23.229 INFO:teuthology.orchestra.run.smithi092.stdout: "description": "osd_repop(client.15150.0:173316 8.11 e222/184 8:89549ca0:::benchmark_data_smithi092_104865_object52815:head v 222'8208, mlcod=222'8206)", 2024-03-11T21:34:23.229 INFO:teuthology.orchestra.run.smithi092.stdout: "initiated_at": "2024-03-11T21:34:22.555904+0000", 2024-03-11T21:34:23.229 INFO:teuthology.orchestra.run.smithi092.stdout: "age": 0.0042322979999999998, 2024-03-11T21:34:23.229 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.0043482950000000003, 2024-03-11T21:34:23.230 INFO:teuthology.orchestra.run.smithi092.stdout: "type_data": { 2024-03-11T21:34:23.230 INFO:teuthology.orchestra.run.smithi092.stdout: "flag_point": "started", 2024-03-11T21:34:23.230 INFO:teuthology.orchestra.run.smithi092.stdout: "events": [ 2024-03-11T21:34:23.230 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:34:23.230 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "initiated", 2024-03-11T21:34:23.230 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:34:22.555904+0000", 2024-03-11T21:34:23.230 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-03-11T21:34:23.230 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:34:23.230 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:34:23.230 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "header_read", 2024-03-11T21:34:23.230 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:34:22.555904+0000", 2024-03-11T21:34:23.230 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-03-11T21:34:23.231 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:34:23.231 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:34:23.231 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "throttled", 2024-03-11T21:34:23.231 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:34:22.555905+0000", 2024-03-11T21:34:23.231 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.4160000000000001e-06 2024-03-11T21:34:23.231 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:34:23.231 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:34:23.231 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "all_read", 2024-03-11T21:34:23.231 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:34:22.555913+0000", 2024-03-11T21:34:23.231 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 7.5989999999999996e-06 2024-03-11T21:34:23.231 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:34:23.231 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:34:23.231 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "dispatched", 2024-03-11T21:34:23.232 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:34:22.555923+0000", 2024-03-11T21:34:23.232 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.0205e-05 2024-03-11T21:34:23.232 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:34:23.232 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:34:23.232 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "queued_for_pg", 2024-03-11T21:34:23.232 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:34:22.555930+0000", 2024-03-11T21:34:23.232 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 7.4050000000000003e-06 2024-03-11T21:34:23.232 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:34:23.232 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:34:23.232 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-03-11T21:34:23.232 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:34:22.556112+0000", 2024-03-11T21:34:23.232 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00018144400000000001 2024-03-11T21:34:23.233 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:34:23.233 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:34:23.233 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "started", 2024-03-11T21:34:23.233 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:34:22.556152+0000", 2024-03-11T21:34:23.233 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 4.0003000000000003e-05 2024-03-11T21:34:23.233 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-03-11T21:34:23.233 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-03-11T21:34:23.233 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-03-11T21:34:23.233 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:34:23.233 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:34:23.233 INFO:teuthology.orchestra.run.smithi092.stdout: "description": "osd_op(client.15150.0:173322 8.a 8:587f52b1:::benchmark_data_smithi092_104865_object52821:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e222)", 2024-03-11T21:34:23.234 INFO:teuthology.orchestra.run.smithi092.stdout: "initiated_at": "2024-03-11T21:34:22.555953+0000", 2024-03-11T21:34:23.234 INFO:teuthology.orchestra.run.smithi092.stdout: "age": 0.0041826199999999997, 2024-03-11T21:34:23.234 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.0043609670000000003, 2024-03-11T21:34:23.234 INFO:teuthology.orchestra.run.smithi092.stdout: "type_data": { 2024-03-11T21:34:23.234 INFO:teuthology.orchestra.run.smithi092.stdout: "flag_point": "waiting for sub ops", 2024-03-11T21:34:23.234 INFO:teuthology.orchestra.run.smithi092.stdout: "client_info": { 2024-03-11T21:34:23.234 INFO:teuthology.orchestra.run.smithi092.stdout: "client": "client.15150", 2024-03-11T21:34:23.235 INFO:teuthology.orchestra.run.smithi092.stdout: "client_addr": "v1:172.21.15.92:0/436606717", 2024-03-11T21:34:23.235 INFO:teuthology.orchestra.run.smithi092.stdout: "tid": 173322 2024-03-11T21:34:23.235 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:34:23.235 INFO:teuthology.orchestra.run.smithi092.stdout: "events": [ 2024-03-11T21:34:23.235 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:34:23.235 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "initiated", 2024-03-11T21:34:23.235 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:34:22.555953+0000", 2024-03-11T21:34:23.235 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-03-11T21:34:23.235 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:34:23.235 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:34:23.235 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "header_read", 2024-03-11T21:34:23.235 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:34:22.555953+0000", 2024-03-11T21:34:23.235 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-03-11T21:34:23.235 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:34:23.236 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:34:23.236 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "throttled", 2024-03-11T21:34:23.236 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:34:22.555955+0000", 2024-03-11T21:34:23.236 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.2410000000000001e-06 2024-03-11T21:34:23.236 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:34:23.236 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:34:23.236 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "all_read", 2024-03-11T21:34:23.236 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:34:22.555959+0000", 2024-03-11T21:34:23.236 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 4.189e-06 2024-03-11T21:34:23.237 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:34:23.237 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:34:23.237 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "dispatched", 2024-03-11T21:34:23.237 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:34:22.555967+0000", 2024-03-11T21:34:23.237 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 8.6009999999999995e-06 2024-03-11T21:34:23.237 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:34:23.237 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:34:23.237 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "queued_for_pg", 2024-03-11T21:34:23.237 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:34:22.555974+0000", 2024-03-11T21:34:23.237 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 6.8990000000000004e-06 2024-03-11T21:34:23.238 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:34:23.238 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:34:23.238 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-03-11T21:34:23.238 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:34:22.556133+0000", 2024-03-11T21:34:23.238 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00015833700000000001 2024-03-11T21:34:23.238 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:34:23.238 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:34:23.238 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "started", 2024-03-11T21:34:23.238 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:34:22.556266+0000", 2024-03-11T21:34:23.239 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00013285200000000001 2024-03-11T21:34:23.239 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:34:23.239 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:34:23.239 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "waiting for subops from 2,5", 2024-03-11T21:34:23.239 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:34:22.556389+0000", 2024-03-11T21:34:23.239 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.000123306 2024-03-11T21:34:23.239 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:34:23.239 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:34:23.240 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "op_commit", 2024-03-11T21:34:23.240 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:34:22.560292+0000", 2024-03-11T21:34:23.240 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.0039035329999999998 2024-03-11T21:34:23.240 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-03-11T21:34:23.240 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-03-11T21:34:23.240 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-03-11T21:34:23.240 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-03-11T21:34:23.240 INFO:teuthology.orchestra.run.smithi092.stdout: ], 2024-03-11T21:34:23.240 INFO:teuthology.orchestra.run.smithi092.stdout: "num_ops": 3 2024-03-11T21:34:23.241 INFO:teuthology.orchestra.run.smithi092.stdout:} 2024-03-11T21:34:23.248 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:34:23.295 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:34:23.295 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:34:23.296 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:34:23.397 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:34:23.424 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:34:23.425 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:34:23.425 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:34:23.526 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:34:23.562 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:34:23.562 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:34:23.563 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:34:23.666 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:34:23.701 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:34:23.702 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:34:23.702 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:34:23.803 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:34:23.828 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:34:23.828 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:34:23.829 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:34:23.930 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:34:23.964 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:34:23.965 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:34:23.965 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:34:24.066 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:34:24.080 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:34:23 smithi097 ceph-mon[33317]: 6.15 scrub ok 2024-03-11T21:34:24.080 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:34:23 smithi097 ceph-mon[33317]: from='client.24469 v1:172.21.15.97:0/404208310' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:34:24.080 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:34:23 smithi097 ceph-mon[33317]: 2.a scrub starts 2024-03-11T21:34:24.080 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:34:23 smithi097 ceph-mon[33317]: 2.a scrub ok 2024-03-11T21:34:24.080 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:34:23 smithi097 ceph-mon[33317]: 6.8 scrub starts 2024-03-11T21:34:24.080 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:34:23 smithi097 ceph-mon[33317]: 6.8 scrub ok 2024-03-11T21:34:24.080 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:34:23 smithi097 ceph-mon[33317]: 5.1f scrub starts 2024-03-11T21:34:24.081 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:34:23 smithi097 ceph-mon[33317]: 5.1f scrub ok 2024-03-11T21:34:24.082 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:34:23 smithi097 ceph-mon[33317]: from='mgr.14607 v1:172.21.15.92:0/1702717597' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T21:34:24.082 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:34:23 smithi097 ceph-mon[33317]: 6.4 scrub starts 2024-03-11T21:34:24.082 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:34:23 smithi097 ceph-mon[33317]: 6.4 scrub ok 2024-03-11T21:34:24.101 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:34:24.102 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:34:24.102 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:34:24.110 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:34:23 smithi092 ceph-mon[29572]: 6.15 scrub ok 2024-03-11T21:34:24.110 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:34:23 smithi092 ceph-mon[29572]: from='client.24469 v1:172.21.15.97:0/404208310' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:34:24.110 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:34:23 smithi092 ceph-mon[29572]: 2.a scrub starts 2024-03-11T21:34:24.110 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:34:23 smithi092 ceph-mon[29572]: 2.a scrub ok 2024-03-11T21:34:24.110 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:34:23 smithi092 ceph-mon[29572]: 6.8 scrub starts 2024-03-11T21:34:24.110 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:34:23 smithi092 ceph-mon[29572]: 6.8 scrub ok 2024-03-11T21:34:24.110 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:34:23 smithi092 ceph-mon[29572]: 5.1f scrub starts 2024-03-11T21:34:24.110 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:34:23 smithi092 ceph-mon[29572]: 5.1f scrub ok 2024-03-11T21:34:24.110 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:34:23 smithi092 ceph-mon[29572]: from='mgr.14607 v1:172.21.15.92:0/1702717597' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T21:34:24.110 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:34:23 smithi092 ceph-mon[29572]: 6.4 scrub starts 2024-03-11T21:34:24.110 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:34:23 smithi092 ceph-mon[29572]: 6.4 scrub ok 2024-03-11T21:34:24.111 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:34:23 smithi092 ceph-mon[37563]: 6.15 scrub ok 2024-03-11T21:34:24.111 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:34:23 smithi092 ceph-mon[37563]: from='client.24469 v1:172.21.15.97:0/404208310' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:34:24.111 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:34:23 smithi092 ceph-mon[37563]: 2.a scrub starts 2024-03-11T21:34:24.111 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:34:23 smithi092 ceph-mon[37563]: 2.a scrub ok 2024-03-11T21:34:24.111 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:34:23 smithi092 ceph-mon[37563]: 6.8 scrub starts 2024-03-11T21:34:24.111 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:34:23 smithi092 ceph-mon[37563]: 6.8 scrub ok 2024-03-11T21:34:24.111 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:34:23 smithi092 ceph-mon[37563]: 5.1f scrub starts 2024-03-11T21:34:24.111 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:34:23 smithi092 ceph-mon[37563]: 5.1f scrub ok 2024-03-11T21:34:24.111 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:34:23 smithi092 ceph-mon[37563]: from='mgr.14607 v1:172.21.15.92:0/1702717597' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T21:34:24.112 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:34:23 smithi092 ceph-mon[37563]: 6.4 scrub starts 2024-03-11T21:34:24.112 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:34:23 smithi092 ceph-mon[37563]: 6.4 scrub ok 2024-03-11T21:34:24.203 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:34:24.251 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:34:24.252 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:34:24.252 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:34:24.353 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:34:24.384 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:34:24.384 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:34:24.384 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:34:24.485 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:34:24.521 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:34:24.521 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:34:24.522 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:34:24.622 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:34:24.649 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:34:24.650 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:34:24.650 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:34:24.751 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:34:24.790 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:34:24.791 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:34:24.791 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:34:24.892 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:34:24.930 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:34:24.931 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:34:24.931 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:34:25.032 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:34:25.062 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:34:25.063 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:34:25.063 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:34:25.076 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:34:24 smithi097 ceph-mon[33317]: pgmap v379: 158 pgs: 1 active+remapped+backfill_toofull, 157 active+clean; 4.2 GiB data, 17 GiB used, 699 GiB / 715 GiB avail; 1.6 KiB/s rd, 0 B/s wr, 1.73k op/s; 6960/207801 objects misplaced (3.349%) 2024-03-11T21:34:25.076 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:34:24 smithi097 ceph-mon[33317]: 2.8 scrub starts 2024-03-11T21:34:25.076 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:34:24 smithi097 ceph-mon[33317]: 2.8 scrub ok 2024-03-11T21:34:25.076 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:34:24 smithi097 ceph-mon[33317]: 4.5 scrub starts 2024-03-11T21:34:25.076 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:34:24 smithi097 ceph-mon[33317]: 4.5 scrub ok 2024-03-11T21:34:25.110 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:34:24 smithi092 ceph-mon[29572]: pgmap v379: 158 pgs: 1 active+remapped+backfill_toofull, 157 active+clean; 4.2 GiB data, 17 GiB used, 699 GiB / 715 GiB avail; 1.6 KiB/s rd, 0 B/s wr, 1.73k op/s; 6960/207801 objects misplaced (3.349%) 2024-03-11T21:34:25.110 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:34:24 smithi092 ceph-mon[29572]: 2.8 scrub starts 2024-03-11T21:34:25.110 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:34:24 smithi092 ceph-mon[29572]: 2.8 scrub ok 2024-03-11T21:34:25.110 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:34:24 smithi092 ceph-mon[29572]: 4.5 scrub starts 2024-03-11T21:34:25.110 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:34:24 smithi092 ceph-mon[29572]: 4.5 scrub ok 2024-03-11T21:34:25.111 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:34:24 smithi092 ceph-mon[37563]: pgmap v379: 158 pgs: 1 active+remapped+backfill_toofull, 157 active+clean; 4.2 GiB data, 17 GiB used, 699 GiB / 715 GiB avail; 1.6 KiB/s rd, 0 B/s wr, 1.73k op/s; 6960/207801 objects misplaced (3.349%) 2024-03-11T21:34:25.111 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:34:24 smithi092 ceph-mon[37563]: 2.8 scrub starts 2024-03-11T21:34:25.111 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:34:24 smithi092 ceph-mon[37563]: 2.8 scrub ok 2024-03-11T21:34:25.111 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:34:24 smithi092 ceph-mon[37563]: 4.5 scrub starts 2024-03-11T21:34:25.111 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:34:24 smithi092 ceph-mon[37563]: 4.5 scrub ok 2024-03-11T21:34:25.165 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:34:25.191 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:34:25.192 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:34:25.192 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:34:25.293 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:34:25.328 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:34:25.329 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:34:25.329 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:34:25.430 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:34:25.469 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:34:25.470 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:34:25.470 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:34:25.571 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:34:25.610 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:34:25.610 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:34:25.611 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:34:25.714 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:34:25.748 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:34:25.749 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:34:25.749 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:34:25.850 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:34:25.897 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:34:25.898 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:34:25.898 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:34:25.999 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:34:26.040 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:34:26.040 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:34:26.040 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:34:26.141 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:34:26.165 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:34:26.165 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:34:26.165 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:34:26.266 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:34:26.306 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:34:26.307 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:34:26.307 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:34:26.372 INFO:teuthology.orchestra.run.smithi092.stderr:Inferring config /var/lib/ceph/00db0016-dfed-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T21:34:26.408 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:34:26.411 DEBUG:teuthology.orchestra.run.smithi092:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid 00db0016-dfed-11ee-95c8-87774f69a715 -- ceph daemon osd.1 dump_blocked_ops 2024-03-11T21:34:26.436 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:34:26.437 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:34:26.437 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:34:26.439 INFO:teuthology.orchestra.run.smithi092.stderr:Inferring config /var/lib/ceph/00db0016-dfed-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T21:34:26.476 INFO:teuthology.orchestra.run.smithi092.stderr:Inferring config /var/lib/ceph/00db0016-dfed-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T21:34:26.538 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:34:26.569 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:34:26.570 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:34:26.570 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:34:26.671 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:34:26.700 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:34:26.701 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:34:26.701 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:34:26.802 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:34:26.835 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:34:26.835 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:34:26.835 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:34:26.936 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:34:26.983 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:34:26.983 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:34:26.984 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:34:27.076 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:34:26 smithi097 ceph-mon[33317]: pgmap v380: 158 pgs: 1 active+remapped+backfill_toofull, 157 active+clean; 4.1 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.32k op/s; 6366/203706 objects misplaced (3.125%) 2024-03-11T21:34:27.076 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:34:26 smithi097 ceph-mon[33317]: 5.7 scrub starts 2024-03-11T21:34:27.076 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:34:26 smithi097 ceph-mon[33317]: 5.7 scrub ok 2024-03-11T21:34:27.077 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:34:26 smithi097 ceph-mon[33317]: 6.17 scrub starts 2024-03-11T21:34:27.077 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:34:26 smithi097 ceph-mon[33317]: 6.17 scrub ok 2024-03-11T21:34:27.077 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:34:26 smithi097 ceph-mon[33317]: from='mgr.14607 v1:172.21.15.92:0/1702717597' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T21:34:27.077 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:34:26 smithi097 ceph-mon[33317]: from='mgr.14607 v1:172.21.15.92:0/1702717597' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T21:34:27.077 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:34:26 smithi097 ceph-mon[33317]: from='mgr.14607 ' entity='mgr.y' 2024-03-11T21:34:27.084 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:34:27.110 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:34:26 smithi092 ceph-mon[37563]: pgmap v380: 158 pgs: 1 active+remapped+backfill_toofull, 157 active+clean; 4.1 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.32k op/s; 6366/203706 objects misplaced (3.125%) 2024-03-11T21:34:27.110 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:34:26 smithi092 ceph-mon[37563]: 5.7 scrub starts 2024-03-11T21:34:27.110 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:34:26 smithi092 ceph-mon[37563]: 5.7 scrub ok 2024-03-11T21:34:27.110 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:34:26 smithi092 ceph-mon[37563]: 6.17 scrub starts 2024-03-11T21:34:27.110 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:34:26 smithi092 ceph-mon[37563]: 6.17 scrub ok 2024-03-11T21:34:27.111 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:34:26 smithi092 ceph-mon[37563]: from='mgr.14607 v1:172.21.15.92:0/1702717597' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T21:34:27.111 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:34:26 smithi092 ceph-mon[37563]: from='mgr.14607 v1:172.21.15.92:0/1702717597' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T21:34:27.111 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:34:26 smithi092 ceph-mon[37563]: from='mgr.14607 ' entity='mgr.y' 2024-03-11T21:34:27.111 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:34:26 smithi092 ceph-mon[29572]: pgmap v380: 158 pgs: 1 active+remapped+backfill_toofull, 157 active+clean; 4.1 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.32k op/s; 6366/203706 objects misplaced (3.125%) 2024-03-11T21:34:27.111 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:34:26 smithi092 ceph-mon[29572]: 5.7 scrub starts 2024-03-11T21:34:27.111 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:34:26 smithi092 ceph-mon[29572]: 5.7 scrub ok 2024-03-11T21:34:27.111 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:34:26 smithi092 ceph-mon[29572]: 6.17 scrub starts 2024-03-11T21:34:27.111 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:34:26 smithi092 ceph-mon[29572]: 6.17 scrub ok 2024-03-11T21:34:27.111 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:34:26 smithi092 ceph-mon[29572]: from='mgr.14607 v1:172.21.15.92:0/1702717597' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T21:34:27.112 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:34:26 smithi092 ceph-mon[29572]: from='mgr.14607 v1:172.21.15.92:0/1702717597' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T21:34:27.112 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:34:26 smithi092 ceph-mon[29572]: from='mgr.14607 ' entity='mgr.y' 2024-03-11T21:34:27.124 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:34:27.125 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:34:27.126 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:34:27.227 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:34:27.265 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:34:27.265 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:34:27.265 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:34:27.366 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:34:27.398 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:34:27.399 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:34:27.399 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:34:27.500 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:34:27.534 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:34:27.535 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:34:27.535 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:34:27.636 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:34:27.673 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:34:27.674 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:34:27.674 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:34:27.775 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:34:27.817 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:34:27.818 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:34:27.818 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:34:27.919 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:34:28.289 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:34:27 smithi092 ceph-mon[29572]: 5.f scrub starts 2024-03-11T21:34:28.289 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:34:27 smithi092 ceph-mon[29572]: 5.f scrub ok 2024-03-11T21:34:28.289 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:34:27 smithi092 ceph-mon[29572]: 4.9 deep-scrub starts 2024-03-11T21:34:28.289 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:34:27 smithi092 ceph-mon[29572]: 4.9 deep-scrub ok 2024-03-11T21:34:28.289 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:34:27 smithi092 ceph-mon[29572]: 6.3 scrub starts 2024-03-11T21:34:28.289 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:34:27 smithi092 ceph-mon[29572]: 6.3 scrub ok 2024-03-11T21:34:28.290 INFO:journalctl@ceph.mgr.y.smithi092.stdout:Mar 11 21:34:27 smithi092 ceph-00db0016-dfed-11ee-95c8-87774f69a715-mgr-y[29801]: ::ffff:172.21.15.97 - - [11/Mar/2024:21:34:27] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-11T21:34:28.290 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:34:27 smithi092 ceph-mon[37563]: 5.f scrub starts 2024-03-11T21:34:28.290 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:34:27 smithi092 ceph-mon[37563]: 5.f scrub ok 2024-03-11T21:34:28.290 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:34:27 smithi092 ceph-mon[37563]: 4.9 deep-scrub starts 2024-03-11T21:34:28.290 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:34:27 smithi092 ceph-mon[37563]: 4.9 deep-scrub ok 2024-03-11T21:34:28.290 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:34:27 smithi092 ceph-mon[37563]: 6.3 scrub starts 2024-03-11T21:34:28.291 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:34:27 smithi092 ceph-mon[37563]: 6.3 scrub ok 2024-03-11T21:34:28.291 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:34:27 smithi097 ceph-mon[33317]: 5.f scrub starts 2024-03-11T21:34:28.291 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:34:27 smithi097 ceph-mon[33317]: 5.f scrub ok 2024-03-11T21:34:28.291 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:34:27 smithi097 ceph-mon[33317]: 4.9 deep-scrub starts 2024-03-11T21:34:28.291 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:34:27 smithi097 ceph-mon[33317]: 4.9 deep-scrub ok 2024-03-11T21:34:28.291 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:34:27 smithi097 ceph-mon[33317]: 6.3 scrub starts 2024-03-11T21:34:28.291 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:34:27 smithi097 ceph-mon[33317]: 6.3 scrub ok 2024-03-11T21:34:28.330 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:34:28.331 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:34:28.331 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:34:28.432 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:34:28.475 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:34:28.476 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:34:28.476 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:34:28.577 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:34:28.611 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:34:28.612 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:34:28.612 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:34:28.713 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:34:28.743 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:34:28.744 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:34:28.744 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:34:28.845 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:34:28.860 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:34:28 smithi092 ceph-mon[29572]: pgmap v381: 158 pgs: 1 active+remapped+backfill_toofull, 157 active+clean; 4.0 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 544 B/s rd, 0 B/s wr, 977 op/s; 6366/198564 objects misplaced (3.206%) 2024-03-11T21:34:28.860 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:34:28 smithi092 ceph-mon[29572]: 2.1b deep-scrub starts 2024-03-11T21:34:28.860 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:34:28 smithi092 ceph-mon[29572]: 2.1b deep-scrub ok 2024-03-11T21:34:28.860 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:34:28 smithi092 ceph-mon[37563]: pgmap v381: 158 pgs: 1 active+remapped+backfill_toofull, 157 active+clean; 4.0 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 544 B/s rd, 0 B/s wr, 977 op/s; 6366/198564 objects misplaced (3.206%) 2024-03-11T21:34:28.860 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:34:28 smithi092 ceph-mon[37563]: 2.1b deep-scrub starts 2024-03-11T21:34:28.860 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:34:28 smithi092 ceph-mon[37563]: 2.1b deep-scrub ok 2024-03-11T21:34:28.877 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:34:28.877 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:34:28.878 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:34:28.979 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:34:29.024 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:34:29.024 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:34:29.025 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:34:29.075 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:34:28 smithi097 ceph-mon[33317]: pgmap v381: 158 pgs: 1 active+remapped+backfill_toofull, 157 active+clean; 4.0 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 544 B/s rd, 0 B/s wr, 977 op/s; 6366/198564 objects misplaced (3.206%) 2024-03-11T21:34:29.076 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:34:28 smithi097 ceph-mon[33317]: 2.1b deep-scrub starts 2024-03-11T21:34:29.076 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:34:28 smithi097 ceph-mon[33317]: 2.1b deep-scrub ok 2024-03-11T21:34:29.125 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:34:29.153 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:34:29.154 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:34:29.154 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:34:29.255 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:34:29.295 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:34:29.295 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:34:29.296 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:34:29.397 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:34:29.435 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:34:29.436 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:34:29.436 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:34:29.537 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:34:29.577 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:34:29.577 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:34:29.578 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:34:29.678 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:34:29.710 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:34:29.710 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:34:29.711 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:34:29.811 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:34:29.845 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:34:29.845 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:34:29.845 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:34:29.946 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:34:29.984 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:34:29.985 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:34:29.985 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:34:30.076 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:34:29 smithi097 ceph-mon[33317]: 2.1f scrub starts 2024-03-11T21:34:30.076 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:34:29 smithi097 ceph-mon[33317]: 2.1f scrub ok 2024-03-11T21:34:30.080 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:34:29 smithi092 ceph-mon[29572]: 2.1f scrub starts 2024-03-11T21:34:30.080 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:34:29 smithi092 ceph-mon[29572]: 2.1f scrub ok 2024-03-11T21:34:30.080 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:34:29 smithi092 ceph-mon[37563]: 2.1f scrub starts 2024-03-11T21:34:30.080 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:34:29 smithi092 ceph-mon[37563]: 2.1f scrub ok 2024-03-11T21:34:30.086 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:34:30.117 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:34:30.118 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:34:30.119 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:34:30.219 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:34:30.250 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:34:30.250 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:34:30.250 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:34:30.351 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:34:30.399 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:34:30.400 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:34:30.400 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:34:30.501 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:34:30.533 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:34:30.534 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:34:30.535 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:34:30.635 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:34:30.672 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:34:30.672 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:34:30.673 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:34:30.773 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:34:30.815 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:34:30.815 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:34:30.816 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:34:30.917 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:34:30.949 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:34:30.950 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:34:30.950 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:34:31.051 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:34:31.075 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:34:30 smithi097 ceph-mon[33317]: pgmap v382: 158 pgs: 1 active+remapped+backfill_toofull, 157 active+clean; 3.8 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 985 B/s rd, 0 B/s wr, 1.19k op/s; 5772/185067 objects misplaced (3.119%) 2024-03-11T21:34:31.076 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:34:30 smithi097 ceph-mon[33317]: 2.b scrub starts 2024-03-11T21:34:31.076 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:34:30 smithi097 ceph-mon[33317]: 2.b scrub ok 2024-03-11T21:34:31.076 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:34:30 smithi097 ceph-mon[33317]: 5.19 scrub starts 2024-03-11T21:34:31.076 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:34:30 smithi097 ceph-mon[33317]: 5.19 scrub ok 2024-03-11T21:34:31.086 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:34:31.087 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:34:31.087 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:34:31.110 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:34:30 smithi092 ceph-mon[29572]: pgmap v382: 158 pgs: 1 active+remapped+backfill_toofull, 157 active+clean; 3.8 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 985 B/s rd, 0 B/s wr, 1.19k op/s; 5772/185067 objects misplaced (3.119%) 2024-03-11T21:34:31.110 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:34:30 smithi092 ceph-mon[29572]: 2.b scrub starts 2024-03-11T21:34:31.110 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:34:30 smithi092 ceph-mon[29572]: 2.b scrub ok 2024-03-11T21:34:31.110 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:34:30 smithi092 ceph-mon[29572]: 5.19 scrub starts 2024-03-11T21:34:31.110 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:34:30 smithi092 ceph-mon[29572]: 5.19 scrub ok 2024-03-11T21:34:31.110 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:34:30 smithi092 ceph-mon[37563]: pgmap v382: 158 pgs: 1 active+remapped+backfill_toofull, 157 active+clean; 3.8 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 985 B/s rd, 0 B/s wr, 1.19k op/s; 5772/185067 objects misplaced (3.119%) 2024-03-11T21:34:31.111 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:34:30 smithi092 ceph-mon[37563]: 2.b scrub starts 2024-03-11T21:34:31.111 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:34:30 smithi092 ceph-mon[37563]: 2.b scrub ok 2024-03-11T21:34:31.111 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:34:30 smithi092 ceph-mon[37563]: 5.19 scrub starts 2024-03-11T21:34:31.111 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:34:30 smithi092 ceph-mon[37563]: 5.19 scrub ok 2024-03-11T21:34:31.188 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:34:31.229 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:34:31.230 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:34:31.230 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:34:31.331 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:34:31.366 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:34:31.367 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:34:31.367 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:34:31.468 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:34:31.509 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:34:31.509 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:34:31.510 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:34:31.611 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:34:31.647 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:34:31.647 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:34:31.647 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:34:31.748 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:34:31.777 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:34:31.778 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:34:31.778 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:34:31.879 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:34:31.910 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:34:31.910 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:34:31.911 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:34:32.011 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:34:32.039 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:34:31 smithi092 ceph-mon[29572]: 5.15 scrub starts 2024-03-11T21:34:32.039 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:34:31 smithi092 ceph-mon[29572]: 5.15 scrub ok 2024-03-11T21:34:32.039 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:34:31 smithi092 ceph-mon[29572]: 2.0 scrub starts 2024-03-11T21:34:32.039 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:34:31 smithi092 ceph-mon[29572]: 2.0 scrub ok 2024-03-11T21:34:32.039 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:34:31 smithi092 ceph-mon[37563]: 5.15 scrub starts 2024-03-11T21:34:32.039 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:34:31 smithi092 ceph-mon[37563]: 5.15 scrub ok 2024-03-11T21:34:32.039 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:34:31 smithi092 ceph-mon[37563]: 2.0 scrub starts 2024-03-11T21:34:32.039 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:34:31 smithi092 ceph-mon[37563]: 2.0 scrub ok 2024-03-11T21:34:32.045 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:34:32.045 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:34:32.046 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:34:32.136 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:34:31 smithi097 ceph-mon[33317]: 5.15 scrub starts 2024-03-11T21:34:32.136 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:34:31 smithi097 ceph-mon[33317]: 5.15 scrub ok 2024-03-11T21:34:32.136 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:34:31 smithi097 ceph-mon[33317]: 2.0 scrub starts 2024-03-11T21:34:32.136 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:34:31 smithi097 ceph-mon[33317]: 2.0 scrub ok 2024-03-11T21:34:32.147 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:34:32.192 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:34:32.192 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:34:32.193 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:34:32.293 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:34:32.338 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:34:32.339 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:34:32.339 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:34:32.440 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:34:32.488 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:34:32.489 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:34:32.489 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:34:32.590 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:34:32.639 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:34:32.640 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:34:32.640 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:34:32.741 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:34:32.782 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:34:32.783 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:34:32.783 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:34:32.826 INFO:journalctl@ceph.iscsi.iscsi.a.smithi097.stdout:Mar 11 21:34:32 smithi097 ceph-00db0016-dfed-11ee-95c8-87774f69a715-iscsi-iscsi-a[60943]: debug there is no tcmu-runner data available 2024-03-11T21:34:32.873 INFO:teuthology.orchestra.run.smithi092.stderr:reweighted osd.6 to 1 (10000) 2024-03-11T21:34:32.885 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:34:32.915 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:34:32.916 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:34:32.916 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:34:33.017 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:34:33.044 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:34:33.045 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:34:33.045 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:34:33.110 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:34:32 smithi092 ceph-mon[29572]: pgmap v383: 158 pgs: 1 active+remapped+backfill_toofull, 157 active+clean; 3.8 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 852 B/s rd, 0 B/s wr, 1.03k op/s; 5772/185067 objects misplaced (3.119%) 2024-03-11T21:34:33.110 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:34:32 smithi092 ceph-mon[29572]: 6.18 scrub starts 2024-03-11T21:34:33.111 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:34:32 smithi092 ceph-mon[29572]: 6.18 scrub ok 2024-03-11T21:34:33.111 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:34:32 smithi092 ceph-mon[29572]: from='client.? v1:172.21.15.92:0/1582403224' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]: dispatch 2024-03-11T21:34:33.111 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:34:32 smithi092 ceph-mon[29572]: from='client.35147 ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]: dispatch 2024-03-11T21:34:33.111 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:34:32 smithi092 ceph-mon[37563]: pgmap v383: 158 pgs: 1 active+remapped+backfill_toofull, 157 active+clean; 3.8 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 852 B/s rd, 0 B/s wr, 1.03k op/s; 5772/185067 objects misplaced (3.119%) 2024-03-11T21:34:33.111 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:34:32 smithi092 ceph-mon[37563]: 6.18 scrub starts 2024-03-11T21:34:33.111 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:34:32 smithi092 ceph-mon[37563]: 6.18 scrub ok 2024-03-11T21:34:33.111 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:34:32 smithi092 ceph-mon[37563]: from='client.? v1:172.21.15.92:0/1582403224' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]: dispatch 2024-03-11T21:34:33.111 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:34:32 smithi092 ceph-mon[37563]: from='client.35147 ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]: dispatch 2024-03-11T21:34:33.146 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:34:33.215 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:34:33.215 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:34:33.216 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:34:33.316 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:34:33.326 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:34:32 smithi097 ceph-mon[33317]: pgmap v383: 158 pgs: 1 active+remapped+backfill_toofull, 157 active+clean; 3.8 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 852 B/s rd, 0 B/s wr, 1.03k op/s; 5772/185067 objects misplaced (3.119%) 2024-03-11T21:34:33.326 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:34:32 smithi097 ceph-mon[33317]: 6.18 scrub starts 2024-03-11T21:34:33.326 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:34:32 smithi097 ceph-mon[33317]: 6.18 scrub ok 2024-03-11T21:34:33.326 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:34:32 smithi097 ceph-mon[33317]: from='client.? v1:172.21.15.92:0/1582403224' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]: dispatch 2024-03-11T21:34:33.326 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:34:32 smithi097 ceph-mon[33317]: from='client.35147 ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]: dispatch 2024-03-11T21:34:33.348 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:34:33.348 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:34:33.348 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:34:33.449 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:34:33.485 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:34:33.486 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:34:33.486 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:34:33.587 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:34:33.620 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:34:33.621 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:34:33.621 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:34:33.651 DEBUG:teuthology.orchestra.run.smithi092:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid 00db0016-dfed-11ee-95c8-87774f69a715 -- ceph osd reweight 7 1 2024-03-11T21:34:33.722 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:34:33.766 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:34:33.766 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:34:33.767 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:34:33.868 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:34:33.880 INFO:teuthology.orchestra.run.smithi092.stderr:noscrub is set 2024-03-11T21:34:33.928 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:34:33.929 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:34:33.929 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:34:34.030 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:34:34.060 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:34:34.061 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:34:34.061 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:34:34.151 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:34:33 smithi092 ceph-mon[29572]: 2.18 scrub starts 2024-03-11T21:34:34.151 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:34:33 smithi092 ceph-mon[29572]: 2.18 scrub ok 2024-03-11T21:34:34.151 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:34:33 smithi092 ceph-mon[29572]: from='client.24469 v1:172.21.15.97:0/404208310' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:34:34.151 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:34:33 smithi092 ceph-mon[29572]: 4.18 scrub starts 2024-03-11T21:34:34.151 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:34:33 smithi092 ceph-mon[29572]: 4.18 scrub ok 2024-03-11T21:34:34.151 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:34:33 smithi092 ceph-mon[29572]: from='client.35147 ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]': finished 2024-03-11T21:34:34.151 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:34:33 smithi092 ceph-mon[29572]: osdmap e223: 8 total, 8 up, 8 in 2024-03-11T21:34:34.151 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:34:33 smithi092 ceph-mon[29572]: from='client.? v1:172.21.15.92:0/3891950670' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-11T21:34:34.151 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:34:33 smithi092 ceph-mon[29572]: 5.17 scrub starts 2024-03-11T21:34:34.152 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:34:33 smithi092 ceph-mon[29572]: 5.17 scrub ok 2024-03-11T21:34:34.152 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:34:33 smithi092 ceph-mon[29572]: 6.1 scrub starts 2024-03-11T21:34:34.152 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:34:33 smithi092 ceph-mon[29572]: 6.1 scrub ok 2024-03-11T21:34:34.152 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:34:33 smithi092 ceph-mon[37563]: 2.18 scrub starts 2024-03-11T21:34:34.152 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:34:33 smithi092 ceph-mon[37563]: 2.18 scrub ok 2024-03-11T21:34:34.152 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:34:33 smithi092 ceph-mon[37563]: from='client.24469 v1:172.21.15.97:0/404208310' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:34:34.152 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:34:33 smithi092 ceph-mon[37563]: 4.18 scrub starts 2024-03-11T21:34:34.152 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:34:33 smithi092 ceph-mon[37563]: 4.18 scrub ok 2024-03-11T21:34:34.152 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:34:33 smithi092 ceph-mon[37563]: from='client.35147 ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]': finished 2024-03-11T21:34:34.152 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:34:33 smithi092 ceph-mon[37563]: osdmap e223: 8 total, 8 up, 8 in 2024-03-11T21:34:34.152 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:34:33 smithi092 ceph-mon[37563]: from='client.? v1:172.21.15.92:0/3891950670' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-11T21:34:34.152 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:34:33 smithi092 ceph-mon[37563]: 5.17 scrub starts 2024-03-11T21:34:34.153 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:34:33 smithi092 ceph-mon[37563]: 5.17 scrub ok 2024-03-11T21:34:34.153 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:34:33 smithi092 ceph-mon[37563]: 6.1 scrub starts 2024-03-11T21:34:34.153 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:34:33 smithi092 ceph-mon[37563]: 6.1 scrub ok 2024-03-11T21:34:34.162 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:34:34.208 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:34:34.209 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:34:34.209 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:34:34.310 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:34:34.326 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:34:33 smithi097 ceph-mon[33317]: 2.18 scrub starts 2024-03-11T21:34:34.326 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:34:33 smithi097 ceph-mon[33317]: 2.18 scrub ok 2024-03-11T21:34:34.326 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:34:33 smithi097 ceph-mon[33317]: from='client.24469 v1:172.21.15.97:0/404208310' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:34:34.326 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:34:33 smithi097 ceph-mon[33317]: 4.18 scrub starts 2024-03-11T21:34:34.326 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:34:33 smithi097 ceph-mon[33317]: 4.18 scrub ok 2024-03-11T21:34:34.326 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:34:33 smithi097 ceph-mon[33317]: from='client.35147 ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]': finished 2024-03-11T21:34:34.326 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:34:33 smithi097 ceph-mon[33317]: osdmap e223: 8 total, 8 up, 8 in 2024-03-11T21:34:34.326 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:34:33 smithi097 ceph-mon[33317]: from='client.? v1:172.21.15.92:0/3891950670' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-11T21:34:34.326 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:34:33 smithi097 ceph-mon[33317]: 5.17 scrub starts 2024-03-11T21:34:34.326 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:34:33 smithi097 ceph-mon[33317]: 5.17 scrub ok 2024-03-11T21:34:34.327 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:34:33 smithi097 ceph-mon[33317]: 6.1 scrub starts 2024-03-11T21:34:34.327 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:34:33 smithi097 ceph-mon[33317]: 6.1 scrub ok 2024-03-11T21:34:34.357 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:34:34.357 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:34:34.358 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:34:34.458 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:34:34.501 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:34:34.501 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:34:34.501 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:34:34.602 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:34:34.632 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:34:34.633 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:34:34.633 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:34:34.734 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:34:34.764 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:34:34.764 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:34:34.765 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:34:34.803 INFO:teuthology.orchestra.run.smithi092.stdout:osd.0: {} 2024-03-11T21:34:34.804 INFO:teuthology.orchestra.run.smithi092.stdout:osd.1: {} 2024-03-11T21:34:34.804 INFO:teuthology.orchestra.run.smithi092.stdout:osd.2: {} 2024-03-11T21:34:34.804 INFO:teuthology.orchestra.run.smithi092.stdout:osd.3: {} 2024-03-11T21:34:34.804 INFO:teuthology.orchestra.run.smithi092.stdout:osd.4: {} 2024-03-11T21:34:34.804 INFO:teuthology.orchestra.run.smithi092.stdout:osd.5: {} 2024-03-11T21:34:34.804 INFO:teuthology.orchestra.run.smithi092.stdout:osd.6: {} 2024-03-11T21:34:34.804 INFO:teuthology.orchestra.run.smithi092.stdout:osd.7: {} 2024-03-11T21:34:34.804 INFO:teuthology.orchestra.run.smithi092.stderr:osd.0: osd_enable_op_tracker = '' 2024-03-11T21:34:34.804 INFO:teuthology.orchestra.run.smithi092.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-11T21:34:34.804 INFO:teuthology.orchestra.run.smithi092.stderr:osd.2: osd_enable_op_tracker = '' 2024-03-11T21:34:34.804 INFO:teuthology.orchestra.run.smithi092.stderr:osd.3: osd_enable_op_tracker = '' 2024-03-11T21:34:34.805 INFO:teuthology.orchestra.run.smithi092.stderr:osd.4: osd_enable_op_tracker = '' 2024-03-11T21:34:34.805 INFO:teuthology.orchestra.run.smithi092.stderr:osd.5: osd_enable_op_tracker = '' 2024-03-11T21:34:34.805 INFO:teuthology.orchestra.run.smithi092.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-11T21:34:34.805 INFO:teuthology.orchestra.run.smithi092.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-11T21:34:34.865 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:34:34.897 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:34:34.897 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:34:34.898 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:34:34.999 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:34:35.026 INFO:teuthology.orchestra.run.smithi092.stderr:Inferring config /var/lib/ceph/00db0016-dfed-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T21:34:35.032 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:34:35.033 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:34:35.033 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:34:35.110 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:34:34 smithi092 ceph-mon[29572]: pgmap v385: 158 pgs: 1 active+remapped+backfill_toofull, 157 active+clean; 3.4 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 1.28k op/s; 5772/169470 objects misplaced (3.406%) 2024-03-11T21:34:35.110 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:34:34 smithi092 ceph-mon[29572]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-03-11T21:34:35.110 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:34:34 smithi092 ceph-mon[29572]: from='client.? v1:172.21.15.92:0/3891950670' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-03-11T21:34:35.111 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:34:34 smithi092 ceph-mon[29572]: osdmap e224: 8 total, 8 up, 8 in 2024-03-11T21:34:35.111 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:34:34 smithi092 ceph-mon[29572]: from='client.? v1:172.21.15.92:0/1365410090' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-11T21:34:35.111 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:34:34 smithi092 ceph-mon[37563]: pgmap v385: 158 pgs: 1 active+remapped+backfill_toofull, 157 active+clean; 3.4 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 1.28k op/s; 5772/169470 objects misplaced (3.406%) 2024-03-11T21:34:35.111 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:34:34 smithi092 ceph-mon[37563]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-03-11T21:34:35.111 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:34:34 smithi092 ceph-mon[37563]: from='client.? v1:172.21.15.92:0/3891950670' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-03-11T21:34:35.111 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:34:34 smithi092 ceph-mon[37563]: osdmap e224: 8 total, 8 up, 8 in 2024-03-11T21:34:35.111 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:34:34 smithi092 ceph-mon[37563]: from='client.? v1:172.21.15.92:0/1365410090' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-11T21:34:35.134 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:34:35.168 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:34:35.169 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:34:35.169 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:34:35.270 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:34:35.304 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:34:35.305 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:34:35.305 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:34:35.326 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:34:34 smithi097 ceph-mon[33317]: pgmap v385: 158 pgs: 1 active+remapped+backfill_toofull, 157 active+clean; 3.4 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 1.28k op/s; 5772/169470 objects misplaced (3.406%) 2024-03-11T21:34:35.326 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:34:34 smithi097 ceph-mon[33317]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-03-11T21:34:35.326 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:34:34 smithi097 ceph-mon[33317]: from='client.? v1:172.21.15.92:0/3891950670' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-03-11T21:34:35.326 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:34:34 smithi097 ceph-mon[33317]: osdmap e224: 8 total, 8 up, 8 in 2024-03-11T21:34:35.326 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:34:34 smithi097 ceph-mon[33317]: from='client.? v1:172.21.15.92:0/1365410090' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-11T21:34:35.406 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:34:35.435 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:34:35.436 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:34:35.436 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:34:35.537 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:34:35.575 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:34:35.576 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:34:35.576 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:34:35.677 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:34:35.709 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:34:35.710 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:34:35.710 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:34:35.811 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:34:35.849 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:34:35.850 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:34:35.851 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:34:35.952 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:34:35.982 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:34:35.983 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:34:35.983 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:34:36.084 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:34:36.120 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:34:36.120 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:34:36.121 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:34:36.222 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:34:36.252 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:34:36.252 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:34:36.253 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:34:36.354 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:34:36.381 INFO:teuthology.orchestra.run.smithi092.stderr:Inferring config /var/lib/ceph/00db0016-dfed-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T21:34:36.386 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:34:36.386 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:34:36.387 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:34:36.488 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:34:36.536 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:34:36.536 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:34:36.536 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:34:36.637 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:34:36.674 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:34:36.675 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:34:36.675 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:34:36.716 DEBUG:teuthology.orchestra.run.smithi092:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid 00db0016-dfed-11ee-95c8-87774f69a715 -- ceph osd set nodeep-scrub 2024-03-11T21:34:36.776 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:34:36.806 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:34:36.807 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:34:36.807 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:34:36.908 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:34:36.960 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:34:36.961 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:34:36.961 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:34:37.062 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:34:37.072 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:34:36 smithi092 ceph-mon[29572]: pgmap v387: 158 pgs: 1 active+remapped+backfill_toofull, 157 active+clean; 3.3 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.39k op/s; 5220/165219 objects misplaced (3.159%) 2024-03-11T21:34:37.073 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:34:36 smithi092 ceph-mon[37563]: pgmap v387: 158 pgs: 1 active+remapped+backfill_toofull, 157 active+clean; 3.3 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.39k op/s; 5220/165219 objects misplaced (3.159%) 2024-03-11T21:34:37.117 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:34:37.118 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:34:37.118 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:34:37.219 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:34:37.269 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:34:37.270 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:34:37.270 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:34:37.326 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:34:36 smithi097 ceph-mon[33317]: pgmap v387: 158 pgs: 1 active+remapped+backfill_toofull, 157 active+clean; 3.3 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.39k op/s; 5220/165219 objects misplaced (3.159%) 2024-03-11T21:34:37.371 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:34:37.406 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:34:37.407 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:34:37.407 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:34:37.508 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:34:37.549 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:34:37.550 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:34:37.550 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:34:37.618 DEBUG:teuthology.orchestra.run.smithi092:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid 00db0016-dfed-11ee-95c8-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-03-11T21:34:37.652 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:34:37.705 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:34:37.705 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:34:37.706 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:34:37.807 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:34:37.847 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:34:37.847 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:34:37.848 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:34:37.948 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:34:37.997 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:34:37.998 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:34:37.998 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:34:38.099 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:34:38.133 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:34:38.133 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:34:38.134 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:34:38.235 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:34:38.268 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:34:38.269 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:34:38.270 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:34:38.326 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:34:38 smithi097 ceph-mon[33317]: from='mgr.14607 ' entity='mgr.y' 2024-03-11T21:34:38.326 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:34:38 smithi097 ceph-mon[33317]: from='mgr.14607 v1:172.21.15.92:0/1702717597' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T21:34:38.334 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:34:38 smithi092 ceph-mon[29572]: from='mgr.14607 ' entity='mgr.y' 2024-03-11T21:34:38.335 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:34:38 smithi092 ceph-mon[29572]: from='mgr.14607 v1:172.21.15.92:0/1702717597' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T21:34:38.335 INFO:journalctl@ceph.mgr.y.smithi092.stdout:Mar 11 21:34:37 smithi092 ceph-00db0016-dfed-11ee-95c8-87774f69a715-mgr-y[29801]: ::ffff:172.21.15.97 - - [11/Mar/2024:21:34:37] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-11T21:34:38.335 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:34:38 smithi092 ceph-mon[37563]: from='mgr.14607 ' entity='mgr.y' 2024-03-11T21:34:38.335 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:34:38 smithi092 ceph-mon[37563]: from='mgr.14607 v1:172.21.15.92:0/1702717597' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T21:34:38.371 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:34:38.417 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:34:38.418 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:34:38.419 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:34:38.520 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:34:38.586 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:34:38.587 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:34:38.587 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:34:38.688 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:34:38.721 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:34:38.722 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:34:38.723 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:34:38.824 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:34:38.857 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:34:38.858 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:34:38.858 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:34:38.959 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:34:38.993 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:34:38.993 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:34:38.994 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:34:39.094 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:34:39.139 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:34:39.139 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:34:39.140 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:34:39.192 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:34:39 smithi092 ceph-mon[29572]: pgmap v388: 158 pgs: 1 active+remapped+backfill_toofull, 157 active+clean; 3.2 GiB data, 14 GiB used, 702 GiB / 715 GiB avail; 639 B/s rd, 0 B/s wr, 1.03k op/s; 5220/160290 objects misplaced (3.257%) 2024-03-11T21:34:39.192 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:34:39 smithi092 ceph-mon[37563]: pgmap v388: 158 pgs: 1 active+remapped+backfill_toofull, 157 active+clean; 3.2 GiB data, 14 GiB used, 702 GiB / 715 GiB avail; 639 B/s rd, 0 B/s wr, 1.03k op/s; 5220/160290 objects misplaced (3.257%) 2024-03-11T21:34:39.241 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:34:39.282 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:34:39.283 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:34:39.284 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:34:39.326 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:34:39 smithi097 ceph-mon[33317]: pgmap v388: 158 pgs: 1 active+remapped+backfill_toofull, 157 active+clean; 3.2 GiB data, 14 GiB used, 702 GiB / 715 GiB avail; 639 B/s rd, 0 B/s wr, 1.03k op/s; 5220/160290 objects misplaced (3.257%) 2024-03-11T21:34:39.385 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:34:39.410 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:34:39.411 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:34:39.411 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:34:39.512 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:34:39.514 INFO:teuthology.orchestra.run.smithi092.stdout:ERROR: (22) Invalid argument 2024-03-11T21:34:39.514 INFO:teuthology.orchestra.run.smithi092.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-11T21:34:39.572 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:34:39.572 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:34:39.573 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:34:39.674 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:34:39.715 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:34:39.715 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:34:39.715 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:34:39.816 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:34:39.854 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:34:39.854 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:34:39.855 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:34:39.955 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:34:39.989 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:34:39.989 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:34:39.990 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:34:40.091 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:34:40.099 INFO:teuthology.orchestra.run.smithi092.stderr:reweighted osd.7 to 1 (10000) 2024-03-11T21:34:40.135 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:34:40.136 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:34:40.136 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:34:40.237 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:34:40.261 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:34:40.261 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:34:40.261 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:34:40.266 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:34:40 smithi092 ceph-mon[29572]: purged_snaps scrub starts 2024-03-11T21:34:40.266 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:34:40 smithi092 ceph-mon[29572]: purged_snaps scrub ok 2024-03-11T21:34:40.266 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:34:40 smithi092 ceph-mon[29572]: osdmap e225: 8 total, 8 up, 8 in 2024-03-11T21:34:40.266 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:34:40 smithi092 ceph-mon[29572]: from='client.? v1:172.21.15.92:0/198460862' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]: dispatch 2024-03-11T21:34:40.266 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:34:40 smithi092 ceph-mon[29572]: from='client.35162 ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]: dispatch 2024-03-11T21:34:40.266 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:34:40 smithi092 ceph-mon[37563]: purged_snaps scrub starts 2024-03-11T21:34:40.266 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:34:40 smithi092 ceph-mon[37563]: purged_snaps scrub ok 2024-03-11T21:34:40.266 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:34:40 smithi092 ceph-mon[37563]: osdmap e225: 8 total, 8 up, 8 in 2024-03-11T21:34:40.266 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:34:40 smithi092 ceph-mon[37563]: from='client.? v1:172.21.15.92:0/198460862' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]: dispatch 2024-03-11T21:34:40.266 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:34:40 smithi092 ceph-mon[37563]: from='client.35162 ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]: dispatch 2024-03-11T21:34:40.291 DEBUG:teuthology.orchestra.run.smithi092:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid 00db0016-dfed-11ee-95c8-87774f69a715 -- ceph daemon osd.1 dump_historic_ops 2024-03-11T21:34:40.326 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:34:40 smithi097 ceph-mon[33317]: purged_snaps scrub starts 2024-03-11T21:34:40.326 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:34:40 smithi097 ceph-mon[33317]: purged_snaps scrub ok 2024-03-11T21:34:40.326 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:34:40 smithi097 ceph-mon[33317]: osdmap e225: 8 total, 8 up, 8 in 2024-03-11T21:34:40.326 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:34:40 smithi097 ceph-mon[33317]: from='client.? v1:172.21.15.92:0/198460862' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]: dispatch 2024-03-11T21:34:40.326 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:34:40 smithi097 ceph-mon[33317]: from='client.35162 ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]: dispatch 2024-03-11T21:34:40.362 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:34:40.386 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:34:40.387 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:34:40.387 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:34:40.488 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:34:40.511 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:34:40.511 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:34:40.512 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:34:40.612 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:34:40.636 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:34:40.637 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:34:40.637 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:34:40.738 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:34:40.761 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:34:40.762 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:34:40.762 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:34:40.863 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:34:40.886 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:34:40.887 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:34:40.887 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:34:40.988 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:34:40.996 INFO:tasks.cephadm.ceph_manager.ceph:waiting for recovery to complete 2024-03-11T21:34:40.996 DEBUG:teuthology.orchestra.run.smithi092:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid 00db0016-dfed-11ee-95c8-87774f69a715 -- ceph pg dump --format=json 2024-03-11T21:34:41.013 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:34:41.014 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:34:41.014 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:34:41.038 INFO:teuthology.orchestra.run.smithi092.stderr:Inferring config /var/lib/ceph/00db0016-dfed-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T21:34:41.038 INFO:teuthology.orchestra.run.smithi092.stderr:Inferring config /var/lib/ceph/00db0016-dfed-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T21:34:41.079 INFO:teuthology.orchestra.run.smithi092.stderr:Inferring config /var/lib/ceph/00db0016-dfed-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T21:34:41.115 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:34:41.151 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:34:41.151 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:34:41.152 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:34:41.253 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:34:41.276 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:34:41.277 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:34:41.277 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:34:41.303 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:34:41 smithi092 ceph-mon[29572]: pgmap v390: 158 pgs: 1 active+remapped+backfill_toofull, 157 active+clean; 3.0 GiB data, 13 GiB used, 703 GiB / 715 GiB avail; 813 B/s rd, 0 B/s wr, 1.20k op/s; 4599/146832 objects misplaced (3.132%) 2024-03-11T21:34:41.303 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:34:41 smithi092 ceph-mon[29572]: from='client.35162 ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]': finished 2024-03-11T21:34:41.303 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:34:41 smithi092 ceph-mon[29572]: osdmap e226: 8 total, 8 up, 8 in 2024-03-11T21:34:41.304 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:34:41 smithi092 ceph-mon[37563]: pgmap v390: 158 pgs: 1 active+remapped+backfill_toofull, 157 active+clean; 3.0 GiB data, 13 GiB used, 703 GiB / 715 GiB avail; 813 B/s rd, 0 B/s wr, 1.20k op/s; 4599/146832 objects misplaced (3.132%) 2024-03-11T21:34:41.304 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:34:41 smithi092 ceph-mon[37563]: from='client.35162 ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]': finished 2024-03-11T21:34:41.304 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:34:41 smithi092 ceph-mon[37563]: osdmap e226: 8 total, 8 up, 8 in 2024-03-11T21:34:41.325 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:34:41 smithi097 ceph-mon[33317]: pgmap v390: 158 pgs: 1 active+remapped+backfill_toofull, 157 active+clean; 3.0 GiB data, 13 GiB used, 703 GiB / 715 GiB avail; 813 B/s rd, 0 B/s wr, 1.20k op/s; 4599/146832 objects misplaced (3.132%) 2024-03-11T21:34:41.326 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:34:41 smithi097 ceph-mon[33317]: from='client.35162 ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]': finished 2024-03-11T21:34:41.326 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:34:41 smithi097 ceph-mon[33317]: osdmap e226: 8 total, 8 up, 8 in 2024-03-11T21:34:41.378 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:34:41.403 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:34:41.403 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:34:41.403 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:34:41.504 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:34:41.529 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:34:41.530 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:34:41.530 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:34:41.631 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:34:41.655 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:34:41.656 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:34:41.656 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:34:41.757 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:34:41.784 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:34:41.784 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:34:41.785 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:34:41.885 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:34:41.914 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:34:41.914 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:34:41.914 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:34:42.015 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:34:42.043 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:34:42.043 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:34:42.043 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:34:42.144 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:34:42.170 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:34:42.170 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:34:42.171 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:34:42.271 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:34:42.296 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:34:42.297 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:34:42.297 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:34:42.360 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:34:42 smithi092 ceph-mon[29572]: osdmap e227: 8 total, 8 up, 8 in 2024-03-11T21:34:42.360 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:34:42 smithi092 ceph-mon[29572]: pgmap v393: 158 pgs: 1 active+remapped+backfill_toofull, 157 active+clean; 3.0 GiB data, 13 GiB used, 703 GiB / 715 GiB avail; 853 B/s rd, 0 B/s wr, 1.02k op/s; 4599/146832 objects misplaced (3.132%) 2024-03-11T21:34:42.360 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:34:42 smithi092 ceph-mon[37563]: osdmap e227: 8 total, 8 up, 8 in 2024-03-11T21:34:42.360 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:34:42 smithi092 ceph-mon[37563]: pgmap v393: 158 pgs: 1 active+remapped+backfill_toofull, 157 active+clean; 3.0 GiB data, 13 GiB used, 703 GiB / 715 GiB avail; 853 B/s rd, 0 B/s wr, 1.02k op/s; 4599/146832 objects misplaced (3.132%) 2024-03-11T21:34:42.398 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:34:42.422 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:34:42.422 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:34:42.423 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:34:42.447 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:34:42 smithi097 ceph-mon[33317]: osdmap e227: 8 total, 8 up, 8 in 2024-03-11T21:34:42.448 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:34:42 smithi097 ceph-mon[33317]: pgmap v393: 158 pgs: 1 active+remapped+backfill_toofull, 157 active+clean; 3.0 GiB data, 13 GiB used, 703 GiB / 715 GiB avail; 853 B/s rd, 0 B/s wr, 1.02k op/s; 4599/146832 objects misplaced (3.132%) 2024-03-11T21:34:42.523 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:34:42.549 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:34:42.550 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:34:42.550 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:34:42.651 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:34:42.685 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:34:42.686 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:34:42.686 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:34:42.787 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:34:42.820 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:34:42.821 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:34:42.821 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:34:42.825 INFO:journalctl@ceph.iscsi.iscsi.a.smithi097.stdout:Mar 11 21:34:42 smithi097 ceph-00db0016-dfed-11ee-95c8-87774f69a715-iscsi-iscsi-a[60943]: debug there is no tcmu-runner data available 2024-03-11T21:34:42.922 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:34:42.971 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:34:42.972 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:34:42.972 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:34:43.073 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:34:43.109 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:34:43.110 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:34:43.110 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:34:43.206 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:34:43 smithi092 ceph-mon[29572]: 8.10 starting backfill to osd.7 from (0'0,0'0] MAX to 225'8905 2024-03-11T21:34:43.206 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:34:43 smithi092 ceph-mon[29572]: 8.c starting backfill to osd.7 from (0'0,0'0] MAX to 225'12056 2024-03-11T21:34:43.206 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:34:43 smithi092 ceph-mon[29572]: 8.5 starting backfill to osd.7 from (0'0,0'0] MAX to 225'8902 2024-03-11T21:34:43.206 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:34:43 smithi092 ceph-mon[29572]: 8.12 starting backfill to osd.7 from (0'0,0'0] MAX to 225'8975 2024-03-11T21:34:43.206 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:34:43 smithi092 ceph-mon[29572]: 8.17 starting backfill to osd.7 from (0'0,0'0] MAX to 225'8777 2024-03-11T21:34:43.206 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:34:43 smithi092 ceph-mon[29572]: osdmap e228: 8 total, 8 up, 8 in 2024-03-11T21:34:43.206 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:34:43 smithi092 ceph-mon[29572]: 8.d starting backfill to osd.7 from (0'0,0'0] MAX to 225'12355 2024-03-11T21:34:43.206 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:34:43 smithi092 ceph-mon[29572]: 8.8 starting backfill to osd.7 from (0'0,0'0] MAX to 225'9044 2024-03-11T21:34:43.206 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:34:43 smithi092 ceph-mon[29572]: from='client.24469 v1:172.21.15.97:0/404208310' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:34:43.207 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:34:43 smithi092 ceph-mon[37563]: 8.10 starting backfill to osd.7 from (0'0,0'0] MAX to 225'8905 2024-03-11T21:34:43.207 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:34:43 smithi092 ceph-mon[37563]: 8.c starting backfill to osd.7 from (0'0,0'0] MAX to 225'12056 2024-03-11T21:34:43.207 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:34:43 smithi092 ceph-mon[37563]: 8.5 starting backfill to osd.7 from (0'0,0'0] MAX to 225'8902 2024-03-11T21:34:43.207 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:34:43 smithi092 ceph-mon[37563]: 8.12 starting backfill to osd.7 from (0'0,0'0] MAX to 225'8975 2024-03-11T21:34:43.207 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:34:43 smithi092 ceph-mon[37563]: 8.17 starting backfill to osd.7 from (0'0,0'0] MAX to 225'8777 2024-03-11T21:34:43.207 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:34:43 smithi092 ceph-mon[37563]: osdmap e228: 8 total, 8 up, 8 in 2024-03-11T21:34:43.207 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:34:43 smithi092 ceph-mon[37563]: 8.d starting backfill to osd.7 from (0'0,0'0] MAX to 225'12355 2024-03-11T21:34:43.207 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:34:43 smithi092 ceph-mon[37563]: 8.8 starting backfill to osd.7 from (0'0,0'0] MAX to 225'9044 2024-03-11T21:34:43.207 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:34:43 smithi092 ceph-mon[37563]: from='client.24469 v1:172.21.15.97:0/404208310' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:34:43.212 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:34:43.252 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:34:43.253 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:34:43.254 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:34:43.355 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:34:43.390 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:34:43.391 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:34:43.392 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:34:43.492 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:34:43.539 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:34:43.539 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:34:43.540 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:34:43.576 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:34:43 smithi097 ceph-mon[33317]: 8.10 starting backfill to osd.7 from (0'0,0'0] MAX to 225'8905 2024-03-11T21:34:43.576 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:34:43 smithi097 ceph-mon[33317]: 8.c starting backfill to osd.7 from (0'0,0'0] MAX to 225'12056 2024-03-11T21:34:43.576 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:34:43 smithi097 ceph-mon[33317]: 8.5 starting backfill to osd.7 from (0'0,0'0] MAX to 225'8902 2024-03-11T21:34:43.576 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:34:43 smithi097 ceph-mon[33317]: 8.12 starting backfill to osd.7 from (0'0,0'0] MAX to 225'8975 2024-03-11T21:34:43.576 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:34:43 smithi097 ceph-mon[33317]: 8.17 starting backfill to osd.7 from (0'0,0'0] MAX to 225'8777 2024-03-11T21:34:43.576 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:34:43 smithi097 ceph-mon[33317]: osdmap e228: 8 total, 8 up, 8 in 2024-03-11T21:34:43.576 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:34:43 smithi097 ceph-mon[33317]: 8.d starting backfill to osd.7 from (0'0,0'0] MAX to 225'12355 2024-03-11T21:34:43.576 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:34:43 smithi097 ceph-mon[33317]: 8.8 starting backfill to osd.7 from (0'0,0'0] MAX to 225'9044 2024-03-11T21:34:43.576 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:34:43 smithi097 ceph-mon[33317]: from='client.24469 v1:172.21.15.97:0/404208310' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:34:43.641 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:34:43.678 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:34:43.678 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:34:43.679 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:34:43.779 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:34:43.820 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:34:43.820 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:34:43.820 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:34:43.921 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:34:43.959 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:34:43.959 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:34:43.960 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:34:44.060 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:34:44.107 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:34:44.107 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:34:44.108 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:34:44.209 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:34:44.249 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:34:44.250 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:34:44.250 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:34:44.353 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:34:44.388 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:34:44.389 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:34:44.389 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:34:44.453 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:34:44 smithi092 ceph-mon[29572]: osdmap e229: 8 total, 8 up, 8 in 2024-03-11T21:34:44.453 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:34:44 smithi092 ceph-mon[29572]: pgmap v396: 158 pgs: 2 activating+remapped, 3 remapped+peering, 1 active+remapped+backfill_toofull, 152 active+clean; 2.8 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 704 op/s; 7530/138381 objects misplaced (5.441%); 273 B/s, 10 objects/s recovering 2024-03-11T21:34:44.454 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:34:44 smithi092 ceph-mon[37563]: osdmap e229: 8 total, 8 up, 8 in 2024-03-11T21:34:44.454 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:34:44 smithi092 ceph-mon[37563]: pgmap v396: 158 pgs: 2 activating+remapped, 3 remapped+peering, 1 active+remapped+backfill_toofull, 152 active+clean; 2.8 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 704 op/s; 7530/138381 objects misplaced (5.441%); 273 B/s, 10 objects/s recovering 2024-03-11T21:34:44.490 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:34:44.527 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:34:44.528 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:34:44.528 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:34:44.575 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:34:44 smithi097 ceph-mon[33317]: osdmap e229: 8 total, 8 up, 8 in 2024-03-11T21:34:44.576 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:34:44 smithi097 ceph-mon[33317]: pgmap v396: 158 pgs: 2 activating+remapped, 3 remapped+peering, 1 active+remapped+backfill_toofull, 152 active+clean; 2.8 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 704 op/s; 7530/138381 objects misplaced (5.441%); 273 B/s, 10 objects/s recovering 2024-03-11T21:34:44.629 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:34:44.667 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:34:44.667 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:34:44.668 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:34:44.769 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:34:44.802 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:34:44.802 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:34:44.803 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:34:44.904 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:34:44.944 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:34:44.945 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:34:44.945 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:34:45.046 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:34:45.091 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:34:45.091 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:34:45.092 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:34:45.193 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:34:45.229 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:34:45.230 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:34:45.230 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:34:45.331 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:34:45.367 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:34:45.367 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:34:45.368 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:34:45.433 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:34:45 smithi092 ceph-mon[29572]: osdmap e230: 8 total, 8 up, 8 in 2024-03-11T21:34:45.434 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:34:45 smithi092 ceph-mon[37563]: osdmap e230: 8 total, 8 up, 8 in 2024-03-11T21:34:45.468 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:34:45.506 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:34:45.507 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:34:45.507 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:34:45.576 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:34:45 smithi097 ceph-mon[33317]: osdmap e230: 8 total, 8 up, 8 in 2024-03-11T21:34:45.608 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:34:45.647 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:34:45.648 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:34:45.648 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:34:45.749 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:34:45.782 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:34:45.783 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:34:45.783 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:34:45.884 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:34:45.918 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:34:45.919 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:34:45.919 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:34:46.020 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:34:46.056 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:34:46.056 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:34:46.057 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:34:46.158 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:34:46.191 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:34:46.192 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:34:46.192 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:34:46.293 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:34:46.339 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:34:46.339 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:34:46.339 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:34:46.342 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:34:46 smithi092 ceph-mon[29572]: pgmap v398: 158 pgs: 1 active+remapped+backfill_toofull, 2 active+remapped+backfilling, 2 activating+remapped, 3 remapped+peering, 150 active+clean; 2.8 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 874 op/s; 11251/136245 objects misplaced (8.258%); 1.6 MiB/s, 40 objects/s recovering 2024-03-11T21:34:46.342 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:34:46 smithi092 ceph-mon[37563]: pgmap v398: 158 pgs: 1 active+remapped+backfill_toofull, 2 active+remapped+backfilling, 2 activating+remapped, 3 remapped+peering, 150 active+clean; 2.8 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 874 op/s; 11251/136245 objects misplaced (8.258%); 1.6 MiB/s, 40 objects/s recovering 2024-03-11T21:34:46.440 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:34:46.474 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:34:46.475 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:34:46.475 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:34:46.575 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:34:46 smithi097 ceph-mon[33317]: pgmap v398: 158 pgs: 1 active+remapped+backfill_toofull, 2 active+remapped+backfilling, 2 activating+remapped, 3 remapped+peering, 150 active+clean; 2.8 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 874 op/s; 11251/136245 objects misplaced (8.258%); 1.6 MiB/s, 40 objects/s recovering 2024-03-11T21:34:46.576 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:34:46.622 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:34:46.623 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:34:46.624 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:34:46.725 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:34:46.765 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:34:46.766 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:34:46.766 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:34:46.867 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:34:46.906 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:34:46.907 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:34:46.907 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:34:47.008 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:34:47.046 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:34:47.046 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:34:47.047 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:34:47.138 INFO:teuthology.orchestra.run.smithi092.stdout:ERROR: (22) Invalid argument 2024-03-11T21:34:47.138 INFO:teuthology.orchestra.run.smithi092.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-11T21:34:47.147 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:34:47.180 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:34:47.181 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:34:47.181 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:34:47.282 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:34:47.317 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:34:47.317 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:34:47.317 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:34:47.418 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:34:47.453 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:34:47.454 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:34:47.455 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:34:47.556 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:34:47.605 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:34:47.606 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:34:47.606 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:34:47.633 INFO:teuthology.orchestra.run.smithi092.stderr:Inferring config /var/lib/ceph/00db0016-dfed-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T21:34:47.707 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:34:47.756 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:34:47.757 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:34:47.757 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:34:47.761 INFO:teuthology.orchestra.run.smithi092.stdout:osd.0: {} 2024-03-11T21:34:47.804 INFO:teuthology.orchestra.run.smithi092.stdout:osd.1: {} 2024-03-11T21:34:47.842 INFO:teuthology.orchestra.run.smithi092.stdout:osd.2: {} 2024-03-11T21:34:47.858 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:34:47.885 INFO:teuthology.orchestra.run.smithi092.stdout:osd.3: {} 2024-03-11T21:34:47.898 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:34:47.898 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:34:47.899 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:34:47.930 INFO:teuthology.orchestra.run.smithi092.stdout:osd.4: {} 2024-03-11T21:34:47.967 INFO:teuthology.orchestra.run.smithi092.stdout:osd.5: {} 2024-03-11T21:34:48.000 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:34:48.008 INFO:teuthology.orchestra.run.smithi092.stdout:osd.6: {} 2024-03-11T21:34:48.044 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:34:48.044 INFO:teuthology.orchestra.run.smithi092.stdout:osd.7: {} 2024-03-11T21:34:48.045 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:34:48.045 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:34:48.054 INFO:teuthology.orchestra.run.smithi092.stderr:osd.0: osd_enable_op_tracker = '' 2024-03-11T21:34:48.054 INFO:teuthology.orchestra.run.smithi092.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-11T21:34:48.054 INFO:teuthology.orchestra.run.smithi092.stderr:osd.2: osd_enable_op_tracker = '' 2024-03-11T21:34:48.054 INFO:teuthology.orchestra.run.smithi092.stderr:osd.3: osd_enable_op_tracker = '' 2024-03-11T21:34:48.054 INFO:teuthology.orchestra.run.smithi092.stderr:osd.4: osd_enable_op_tracker = '' 2024-03-11T21:34:48.054 INFO:teuthology.orchestra.run.smithi092.stderr:osd.5: osd_enable_op_tracker = '' 2024-03-11T21:34:48.054 INFO:teuthology.orchestra.run.smithi092.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-11T21:34:48.054 INFO:teuthology.orchestra.run.smithi092.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-11T21:34:48.146 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:34:48.181 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:34:48.181 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:34:48.182 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:34:48.206 INFO:journalctl@ceph.mgr.y.smithi092.stdout:Mar 11 21:34:47 smithi092 ceph-00db0016-dfed-11ee-95c8-87774f69a715-mgr-y[29801]: ::ffff:172.21.15.97 - - [11/Mar/2024:21:34:47] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-11T21:34:48.283 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:34:48.325 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:34:48.326 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:34:48.326 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:34:48.427 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:34:48.465 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:34:48.466 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:34:48.466 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:34:48.567 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:34:48.569 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:34:48 smithi092 ceph-mon[29572]: pgmap v399: 158 pgs: 3 active+remapped+backfill_wait, 1 active+remapped+backfill_toofull, 2 active+remapped+backfilling, 2 activating+remapped, 150 active+clean; 2.7 GiB data, 12 GiB used, 704 GiB / 715 GiB avail; 852 B/s rd, 0 B/s wr, 828 op/s; 16337/131916 objects misplaced (12.384%); 1.1 MiB/s, 27 objects/s recovering 2024-03-11T21:34:48.569 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:34:48 smithi092 ceph-mon[29572]: from='client.? v1:172.21.15.92:0/1022603780' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-11T21:34:48.569 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:34:48 smithi092 ceph-mon[37563]: pgmap v399: 158 pgs: 3 active+remapped+backfill_wait, 1 active+remapped+backfill_toofull, 2 active+remapped+backfilling, 2 activating+remapped, 150 active+clean; 2.7 GiB data, 12 GiB used, 704 GiB / 715 GiB avail; 852 B/s rd, 0 B/s wr, 828 op/s; 16337/131916 objects misplaced (12.384%); 1.1 MiB/s, 27 objects/s recovering 2024-03-11T21:34:48.569 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:34:48 smithi092 ceph-mon[37563]: from='client.? v1:172.21.15.92:0/1022603780' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-11T21:34:48.575 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:34:48 smithi097 ceph-mon[33317]: pgmap v399: 158 pgs: 3 active+remapped+backfill_wait, 1 active+remapped+backfill_toofull, 2 active+remapped+backfilling, 2 activating+remapped, 150 active+clean; 2.7 GiB data, 12 GiB used, 704 GiB / 715 GiB avail; 852 B/s rd, 0 B/s wr, 828 op/s; 16337/131916 objects misplaced (12.384%); 1.1 MiB/s, 27 objects/s recovering 2024-03-11T21:34:48.575 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:34:48 smithi097 ceph-mon[33317]: from='client.? v1:172.21.15.92:0/1022603780' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-11T21:34:48.606 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:34:48.606 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:34:48.606 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:34:48.707 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:34:48.745 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:34:48.746 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:34:48.746 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:34:48.847 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:34:48.851 DEBUG:teuthology.orchestra.run.smithi092:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid 00db0016-dfed-11ee-95c8-87774f69a715 -- ceph daemon osd.2 dump_ops_in_flight 2024-03-11T21:34:48.878 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:34:48.881 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:34:48.882 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:34:48.983 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:34:49.010 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:34:49.011 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:34:49.011 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:34:49.112 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:34:49.146 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:34:49.147 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:34:49.147 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:34:49.218 INFO:teuthology.orchestra.run.smithi092.stderr:nodeep-scrub is set 2024-03-11T21:34:49.249 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:34:49.281 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:34:49.282 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:34:49.282 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:34:49.383 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:34:49.408 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:34:49.409 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:34:49.409 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:34:49.510 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:34:49.552 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:34:49.552 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:34:49.552 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:34:49.575 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:34:49 smithi097 ceph-mon[33317]: osdmap e231: 8 total, 8 up, 8 in 2024-03-11T21:34:49.576 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:34:49 smithi097 ceph-mon[33317]: from='client.? v1:172.21.15.92:0/3741862839' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-11T21:34:49.576 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:34:49 smithi097 ceph-mon[33317]: from='client.25204 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-11T21:34:49.610 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:34:49 smithi092 ceph-mon[29572]: osdmap e231: 8 total, 8 up, 8 in 2024-03-11T21:34:49.610 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:34:49 smithi092 ceph-mon[29572]: from='client.? v1:172.21.15.92:0/3741862839' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-11T21:34:49.610 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:34:49 smithi092 ceph-mon[29572]: from='client.25204 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-11T21:34:49.610 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:34:49 smithi092 ceph-mon[37563]: osdmap e231: 8 total, 8 up, 8 in 2024-03-11T21:34:49.610 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:34:49 smithi092 ceph-mon[37563]: from='client.? v1:172.21.15.92:0/3741862839' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-11T21:34:49.610 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:34:49 smithi092 ceph-mon[37563]: from='client.25204 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-11T21:34:49.653 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:34:49.685 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:34:49.686 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:34:49.686 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:34:49.787 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:34:49.822 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:34:49.823 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:34:49.823 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:34:49.924 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:34:49.961 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:34:49.962 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:34:49.962 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:34:50.063 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:34:50.098 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:34:50.099 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:34:50.099 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:34:50.200 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:34:50.240 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:34:50.241 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:34:50.242 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:34:50.343 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:34:50.378 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:34:50.379 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:34:50.379 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:34:50.476 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:34:50 smithi097 ceph-mon[33317]: pgmap v401: 158 pgs: 3 active+remapped+backfill_wait, 2 active+remapped+backfill_toofull, 3 active+remapped+backfilling, 150 active+clean; 2.4 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 850 B/s rd, 0 B/s wr, 1.03k op/s; 12578/119778 objects misplaced (10.501%); 26 MiB/s, 415 objects/s recovering 2024-03-11T21:34:50.476 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:34:50 smithi097 ceph-mon[33317]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-11T21:34:50.476 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:34:50 smithi097 ceph-mon[33317]: from='client.25204 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-03-11T21:34:50.476 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:34:50 smithi097 ceph-mon[33317]: osdmap e232: 8 total, 8 up, 8 in 2024-03-11T21:34:50.477 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:34:50 smithi097 ceph-mon[33317]: Health check update: Low space hindering backfill (add storage if this doesn't resolve itself): 2 pgs backfill_toofull (PG_BACKFILL_FULL) 2024-03-11T21:34:50.480 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:34:50.509 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:34:50.509 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:34:50.509 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:34:50.602 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:34:50 smithi092 ceph-mon[29572]: pgmap v401: 158 pgs: 3 active+remapped+backfill_wait, 2 active+remapped+backfill_toofull, 3 active+remapped+backfilling, 150 active+clean; 2.4 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 850 B/s rd, 0 B/s wr, 1.03k op/s; 12578/119778 objects misplaced (10.501%); 26 MiB/s, 415 objects/s recovering 2024-03-11T21:34:50.603 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:34:50 smithi092 ceph-mon[29572]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-11T21:34:50.603 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:34:50 smithi092 ceph-mon[29572]: from='client.25204 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-03-11T21:34:50.603 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:34:50 smithi092 ceph-mon[29572]: osdmap e232: 8 total, 8 up, 8 in 2024-03-11T21:34:50.603 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:34:50 smithi092 ceph-mon[29572]: Health check update: Low space hindering backfill (add storage if this doesn't resolve itself): 2 pgs backfill_toofull (PG_BACKFILL_FULL) 2024-03-11T21:34:50.603 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:34:50 smithi092 ceph-mon[37563]: pgmap v401: 158 pgs: 3 active+remapped+backfill_wait, 2 active+remapped+backfill_toofull, 3 active+remapped+backfilling, 150 active+clean; 2.4 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 850 B/s rd, 0 B/s wr, 1.03k op/s; 12578/119778 objects misplaced (10.501%); 26 MiB/s, 415 objects/s recovering 2024-03-11T21:34:50.603 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:34:50 smithi092 ceph-mon[37563]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-11T21:34:50.603 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:34:50 smithi092 ceph-mon[37563]: from='client.25204 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-03-11T21:34:50.603 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:34:50 smithi092 ceph-mon[37563]: osdmap e232: 8 total, 8 up, 8 in 2024-03-11T21:34:50.603 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:34:50 smithi092 ceph-mon[37563]: Health check update: Low space hindering backfill (add storage if this doesn't resolve itself): 2 pgs backfill_toofull (PG_BACKFILL_FULL) 2024-03-11T21:34:50.610 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:34:50.638 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:34:50.639 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:34:50.639 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:34:50.740 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:34:50.768 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:34:50.769 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:34:50.770 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:34:50.870 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:34:50.898 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:34:50.898 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:34:50.899 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:34:51.000 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:34:51.024 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:34:51.025 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:34:51.026 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:34:51.126 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:34:51.150 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:34:51.151 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:34:51.151 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:34:51.252 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:34:51.277 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:34:51.278 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:34:51.279 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:34:51.379 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:34:51.411 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:34:51.412 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:34:51.412 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:34:51.513 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:34:51.547 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:34:51.547 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:34:51.548 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:34:51.575 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:34:51 smithi097 ceph-mon[33317]: osdmap e233: 8 total, 8 up, 8 in 2024-03-11T21:34:51.608 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:34:51 smithi092 ceph-mon[29572]: osdmap e233: 8 total, 8 up, 8 in 2024-03-11T21:34:51.608 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:34:51 smithi092 ceph-mon[37563]: osdmap e233: 8 total, 8 up, 8 in 2024-03-11T21:34:51.649 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:34:51.680 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:34:51.680 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:34:51.681 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:34:51.781 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:34:51.812 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:34:51.813 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:34:51.813 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:34:51.914 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:34:51.939 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:34:51.939 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:34:51.939 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:34:52.040 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:34:52.064 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:34:52.065 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:34:52.065 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:34:52.166 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:34:52.191 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:34:52.192 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:34:52.192 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:34:52.293 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:34:52.319 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:34:52.320 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:34:52.321 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:34:52.422 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:34:52.428 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:34:52 smithi092 ceph-mon[29572]: pgmap v404: 158 pgs: 3 active+remapped+backfill_wait, 2 active+remapped+backfill_toofull, 3 active+remapped+backfilling, 150 active+clean; 2.4 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 853 B/s rd, 0 B/s wr, 915 op/s; 12578/119778 objects misplaced (10.501%); 25 MiB/s, 396 objects/s recovering 2024-03-11T21:34:52.428 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:34:52 smithi092 ceph-mon[29572]: osdmap e234: 8 total, 8 up, 8 in 2024-03-11T21:34:52.428 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:34:52 smithi092 ceph-mon[29572]: purged_snaps scrub starts 2024-03-11T21:34:52.429 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:34:52 smithi092 ceph-mon[29572]: purged_snaps scrub ok 2024-03-11T21:34:52.429 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:34:52 smithi092 ceph-mon[29572]: from='mgr.14607 v1:172.21.15.92:0/1702717597' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T21:34:52.429 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:34:52 smithi092 ceph-mon[37563]: pgmap v404: 158 pgs: 3 active+remapped+backfill_wait, 2 active+remapped+backfill_toofull, 3 active+remapped+backfilling, 150 active+clean; 2.4 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 853 B/s rd, 0 B/s wr, 915 op/s; 12578/119778 objects misplaced (10.501%); 25 MiB/s, 396 objects/s recovering 2024-03-11T21:34:52.429 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:34:52 smithi092 ceph-mon[37563]: osdmap e234: 8 total, 8 up, 8 in 2024-03-11T21:34:52.429 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:34:52 smithi092 ceph-mon[37563]: purged_snaps scrub starts 2024-03-11T21:34:52.429 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:34:52 smithi092 ceph-mon[37563]: purged_snaps scrub ok 2024-03-11T21:34:52.429 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:34:52 smithi092 ceph-mon[37563]: from='mgr.14607 v1:172.21.15.92:0/1702717597' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T21:34:52.462 DEBUG:teuthology.orchestra.run.smithi092:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid 00db0016-dfed-11ee-95c8-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-03-11T21:34:52.468 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:34:52.469 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:34:52.469 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:34:52.570 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:34:52.576 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:34:52 smithi097 ceph-mon[33317]: pgmap v404: 158 pgs: 3 active+remapped+backfill_wait, 2 active+remapped+backfill_toofull, 3 active+remapped+backfilling, 150 active+clean; 2.4 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 853 B/s rd, 0 B/s wr, 915 op/s; 12578/119778 objects misplaced (10.501%); 25 MiB/s, 396 objects/s recovering 2024-03-11T21:34:52.576 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:34:52 smithi097 ceph-mon[33317]: osdmap e234: 8 total, 8 up, 8 in 2024-03-11T21:34:52.576 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:34:52 smithi097 ceph-mon[33317]: purged_snaps scrub starts 2024-03-11T21:34:52.576 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:34:52 smithi097 ceph-mon[33317]: purged_snaps scrub ok 2024-03-11T21:34:52.576 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:34:52 smithi097 ceph-mon[33317]: from='mgr.14607 v1:172.21.15.92:0/1702717597' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T21:34:52.576 INFO:journalctl@ceph.iscsi.iscsi.a.smithi097.stdout:Mar 11 21:34:52 smithi097 ceph-00db0016-dfed-11ee-95c8-87774f69a715-iscsi-iscsi-a[60943]: debug there is no tcmu-runner data available 2024-03-11T21:34:52.609 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:34:52.610 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:34:52.610 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:34:52.711 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:34:52.738 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:34:52.739 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:34:52.739 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:34:52.840 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:34:52.878 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:34:52.879 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:34:52.879 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:34:52.980 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:34:53.015 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:34:53.015 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:34:53.016 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:34:53.116 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:34:53.150 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:34:53.150 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:34:53.151 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:34:53.252 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:34:53.290 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:34:53.290 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:34:53.291 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:34:53.392 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:34:53.422 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:34:53.422 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:34:53.423 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:34:53.524 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:34:53.556 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:34:53.556 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:34:53.557 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:34:53.575 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:34:53 smithi097 ceph-mon[33317]: osdmap e235: 8 total, 8 up, 8 in 2024-03-11T21:34:53.575 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:34:53 smithi097 ceph-mon[33317]: from='client.24469 v1:172.21.15.97:0/404208310' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:34:53.610 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:34:53 smithi092 ceph-mon[29572]: osdmap e235: 8 total, 8 up, 8 in 2024-03-11T21:34:53.610 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:34:53 smithi092 ceph-mon[29572]: from='client.24469 v1:172.21.15.97:0/404208310' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:34:53.610 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:34:53 smithi092 ceph-mon[37563]: osdmap e235: 8 total, 8 up, 8 in 2024-03-11T21:34:53.611 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:34:53 smithi092 ceph-mon[37563]: from='client.24469 v1:172.21.15.97:0/404208310' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:34:53.658 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:34:53.691 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:34:53.692 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:34:53.693 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:34:53.794 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:34:53.834 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:34:53.834 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:34:53.835 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:34:53.935 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:34:53.969 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:34:53.969 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:34:53.970 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:34:54.071 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:34:54.102 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:34:54.103 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:34:54.103 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:34:54.187 DEBUG:teuthology.orchestra.run.smithi092:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid 00db0016-dfed-11ee-95c8-87774f69a715 -- ceph osd unset noscrub 2024-03-11T21:34:54.204 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:34:54.237 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:34:54.237 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:34:54.237 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:34:54.338 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:34:54.383 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:34:54.384 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:34:54.385 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:34:54.415 INFO:teuthology.orchestra.run.smithi092.stdout: 2024-03-11T21:34:54.426 INFO:teuthology.orchestra.run.smithi092.stderr:dumped all 2024-03-11T21:34:54.485 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:34:54.519 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:34:54.520 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:34:54.520 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:34:54.575 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:34:54 smithi097 ceph-mon[33317]: pgmap v407: 158 pgs: 1 peering, 2 active+remapped+backfill_toofull, 3 active+remapped+backfilling, 152 active+clean; 2.2 GiB data, 11 GiB used, 705 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 732 op/s; 8986/111000 objects misplaced (8.095%); 48 MiB/s, 766 objects/s recovering 2024-03-11T21:34:54.576 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:34:54 smithi097 ceph-mon[33317]: from='client.35180 v1:172.21.15.92:0/4242230667' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-11T21:34:54.610 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:34:54 smithi092 ceph-mon[29572]: pgmap v407: 158 pgs: 1 peering, 2 active+remapped+backfill_toofull, 3 active+remapped+backfilling, 152 active+clean; 2.2 GiB data, 11 GiB used, 705 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 732 op/s; 8986/111000 objects misplaced (8.095%); 48 MiB/s, 766 objects/s recovering 2024-03-11T21:34:54.610 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:34:54 smithi092 ceph-mon[29572]: from='client.35180 v1:172.21.15.92:0/4242230667' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-11T21:34:54.610 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:34:54 smithi092 ceph-mon[37563]: pgmap v407: 158 pgs: 1 peering, 2 active+remapped+backfill_toofull, 3 active+remapped+backfilling, 152 active+clean; 2.2 GiB data, 11 GiB used, 705 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 732 op/s; 8986/111000 objects misplaced (8.095%); 48 MiB/s, 766 objects/s recovering 2024-03-11T21:34:54.610 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:34:54 smithi092 ceph-mon[37563]: from='client.35180 v1:172.21.15.92:0/4242230667' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-11T21:34:54.621 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:34:54.650 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:34:54.651 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:34:54.651 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:34:54.752 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:34:54.775 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:34:54.776 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:34:54.776 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:34:54.877 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:34:54.900 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:34:54.901 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:34:54.902 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:34:55.002 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:34:55.027 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:34:55.028 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:34:55.028 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:34:55.129 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:34:55.155 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:34:55.155 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:34:55.155 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:34:55.256 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:34:55.283 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:34:55.284 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:34:55.284 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:34:55.385 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:34:55.422 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:34:55.423 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:34:55.424 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:34:55.524 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:34:55.555 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:34:55.556 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:34:55.556 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:34:55.575 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:34:55 smithi097 ceph-mon[33317]: osdmap e236: 8 total, 8 up, 8 in 2024-03-11T21:34:55.610 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:34:55 smithi092 ceph-mon[29572]: osdmap e236: 8 total, 8 up, 8 in 2024-03-11T21:34:55.610 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:34:55 smithi092 ceph-mon[37563]: osdmap e236: 8 total, 8 up, 8 in 2024-03-11T21:34:55.657 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:34:55.681 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:34:55.682 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:34:55.682 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:34:55.783 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:34:55.807 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:34:55.807 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:34:55.808 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:34:55.908 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:34:55.932 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:34:55.933 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:34:55.933 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:34:56.034 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:34:56.058 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:34:56.062 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:34:56.063 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:34:56.163 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:34:56.187 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:34:56.188 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:34:56.188 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:34:56.289 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:34:56.317 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:34:56.317 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:34:56.318 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:34:56.419 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:34:56.456 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:34:56.456 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:34:56.456 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:34:56.502 INFO:teuthology.orchestra.run.smithi092.stdout:{"pg_ready":true,"pg_map":{"version":407,"stamp":"2024-03-11T21:34:53.160887+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":2412353984,"num_objects":37000,"num_object_clones":0,"num_object_copies":111000,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":8986,"num_objects_unfound":0,"num_objects_dirty":37000,"num_whiteouts":0,"num_read":1468,"num_read_kb":1316,"num_write":132229,"num_write_kb":3701310,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5714,"num_bytes_recovered":366942556,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":140272,"ondisk_log_size":140272,"up":474,"acting":474,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":494,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":749961216,"kb_used":11101032,"kb_used_data":8482576,"kb_used_omap":12,"kb_used_meta":2617779,"kb_avail":738860184,"statfs":{"total":767960285184,"available":756592828416,"internally_reserved":0,"allocated":8686157824,"data_stored":8683312012,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12752,"internal_metadata":2680606256},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[5,1,1,2,0,0,0,0,0,0,16],"upper_bound":2048},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-191758336,"num_objects":-2926,"num_object_clones":0,"num_object_copies":-8778,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-3592,"num_objects_unfound":0,"num_objects_dirty":-2926,"num_whiteouts":0,"num_read":5,"num_read_kb":5,"num_write":2926,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3068,"num_bytes_recovered":201064448,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.001328"},"pg_stats":[{"pgid":"8.15","version":"234'9164","reported_seq":19548,"reported_epoch":234,"state":"active+clean","last_fresh":"2024-03-11T21:34:51.735929+0000","last_change":"2024-03-11T21:34:03.402085+0000","last_active":"2024-03-11T21:34:51.735929+0000","last_peered":"2024-03-11T21:34:51.735929+0000","last_clean":"2024-03-11T21:34:51.735929+0000","last_became_active":"2024-03-11T21:34:02.653146+0000","last_became_peered":"2024-03-11T21:34:02.653146+0000","last_unstale":"2024-03-11T21:34:51.735929+0000","last_undegraded":"2024-03-11T21:34:51.735929+0000","last_fullsized":"2024-03-11T21:34:51.735929+0000","mapping_epoch":213,"log_start":"172'3955","ondisk_log_start":"172'3955","created":181,"last_epoch_clean":214,"parent":"0.0","parent_split_bits":5,"last_scrub":"215'7489","last_scrub_stamp":"2024-03-11T21:34:03.401974+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:31:51.219688+0000","last_clean_scrub_stamp":"2024-03-11T21:34:03.401974+0000","objects_scrubbed":2801,"log_size":5209,"log_dups_size":0,"ondisk_log_size":5209,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:35:33.221837+0000","scrub_duration":598,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":73400320,"num_objects":1120,"num_object_clones":0,"num_object_copies":3360,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1120,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1681,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,6],"acting":[7,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.1b","version":"0'0","reported_seq":373,"reported_epoch":235,"state":"active+clean","last_fresh":"2024-03-11T21:34:52.335756+0000","last_change":"2024-03-11T21:34:41.149143+0000","last_active":"2024-03-11T21:34:52.335756+0000","last_peered":"2024-03-11T21:34:52.335756+0000","last_clean":"2024-03-11T21:34:52.335756+0000","last_became_active":"2024-03-11T21:34:41.145634+0000","last_became_peered":"2024-03-11T21:34:41.145634+0000","last_unstale":"2024-03-11T21:34:52.335756+0000","last_undegraded":"2024-03-11T21:34:52.335756+0000","last_fullsized":"2024-03-11T21:34:52.335756+0000","mapping_epoch":226,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":227,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:31:06.310490+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:43.559452+0000","last_clean_scrub_stamp":"2024-03-11T21:31:06.310490+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:34:55.849375+0000","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":372,"reported_epoch":234,"state":"active+clean","last_fresh":"2024-03-11T21:34:51.294729+0000","last_change":"2024-03-11T21:34:15.738366+0000","last_active":"2024-03-11T21:34:51.294729+0000","last_peered":"2024-03-11T21:34:51.294729+0000","last_clean":"2024-03-11T21:34:51.294729+0000","last_became_active":"2024-03-11T21:25:42.578893+0000","last_became_peered":"2024-03-11T21:25:42.578893+0000","last_unstale":"2024-03-11T21:34:51.294729+0000","last_undegraded":"2024-03-11T21:34:51.294729+0000","last_fullsized":"2024-03-11T21:34:51.294729+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:34:15.738139+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:41.544550+0000","last_clean_scrub_stamp":"2024-03-11T21:34:15.738139+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:35:19.004355+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1f","version":"0'0","reported_seq":395,"reported_epoch":231,"state":"active+clean","last_fresh":"2024-03-11T21:34:48.242061+0000","last_change":"2024-03-11T21:34:41.145243+0000","last_active":"2024-03-11T21:34:48.242061+0000","last_peered":"2024-03-11T21:34:48.242061+0000","last_clean":"2024-03-11T21:34:48.242061+0000","last_became_active":"2024-03-11T21:34:41.144945+0000","last_became_peered":"2024-03-11T21:34:41.144945+0000","last_unstale":"2024-03-11T21:34:48.242061+0000","last_undegraded":"2024-03-11T21:34:48.242061+0000","last_fullsized":"2024-03-11T21:34:48.242061+0000","mapping_epoch":226,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":227,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:34:28.637947+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:37.512018+0000","last_clean_scrub_stamp":"2024-03-11T21:34:28.637947+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:35:53.709415+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,4],"acting":[0,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.19","version":"74'15","reported_seq":394,"reported_epoch":235,"state":"active+clean","last_fresh":"2024-03-11T21:34:52.333846+0000","last_change":"2024-03-11T21:31:07.311590+0000","last_active":"2024-03-11T21:34:52.333846+0000","last_peered":"2024-03-11T21:34:52.333846+0000","last_clean":"2024-03-11T21:34:52.333846+0000","last_became_active":"2024-03-11T21:25:40.554207+0000","last_became_peered":"2024-03-11T21:25:40.554207+0000","last_unstale":"2024-03-11T21:34:52.333846+0000","last_undegraded":"2024-03-11T21:34:52.333846+0000","last_fullsized":"2024-03-11T21:34:52.333846+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'15","last_scrub_stamp":"2024-03-11T21:31:07.311470+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:39.525554+0000","last_clean_scrub_stamp":"2024-03-11T21:31:07.311470+0000","objects_scrubbed":7,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:34:55.849557+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.14","version":"234'8988","reported_seq":15644,"reported_epoch":234,"state":"active+clean","last_fresh":"2024-03-11T21:34:51.876548+0000","last_change":"2024-03-11T21:34:11.342445+0000","last_active":"2024-03-11T21:34:51.876548+0000","last_peered":"2024-03-11T21:34:51.876548+0000","last_clean":"2024-03-11T21:34:51.876548+0000","last_became_active":"2024-03-11T21:33:32.391932+0000","last_became_peered":"2024-03-11T21:33:32.391932+0000","last_unstale":"2024-03-11T21:34:51.876548+0000","last_undegraded":"2024-03-11T21:34:51.876548+0000","last_fullsized":"2024-03-11T21:34:51.876548+0000","mapping_epoch":198,"log_start":"170'3426","ondisk_log_start":"170'3426","created":181,"last_epoch_clean":199,"parent":"0.0","parent_split_bits":5,"last_scrub":"219'7635","last_scrub_stamp":"2024-03-11T21:34:11.342073+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:31:51.219688+0000","last_clean_scrub_stamp":"2024-03-11T21:34:11.342073+0000","objects_scrubbed":2502,"log_size":5562,"log_dups_size":0,"ondisk_log_size":5562,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:35:36.963800+0000","scrub_duration":604,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":74448896,"num_objects":1136,"num_object_clones":0,"num_object_copies":3408,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1136,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1366,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,2],"acting":[4,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1a","version":"0'0","reported_seq":360,"reported_epoch":234,"state":"active+clean","last_fresh":"2024-03-11T21:34:51.294577+0000","last_change":"2024-03-11T21:32:01.718632+0000","last_active":"2024-03-11T21:34:51.294577+0000","last_peered":"2024-03-11T21:34:51.294577+0000","last_clean":"2024-03-11T21:34:51.294577+0000","last_became_active":"2024-03-11T21:25:44.590923+0000","last_became_peered":"2024-03-11T21:25:44.590923+0000","last_unstale":"2024-03-11T21:34:51.294577+0000","last_undegraded":"2024-03-11T21:34:51.294577+0000","last_fullsized":"2024-03-11T21:34:51.294577+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:32:01.718486+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:43.559452+0000","last_clean_scrub_stamp":"2024-03-11T21:32:01.718486+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,1],"acting":[4,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":386,"reported_epoch":235,"state":"active+clean","last_fresh":"2024-03-11T21:34:52.334700+0000","last_change":"2024-03-11T21:34:41.154531+0000","last_active":"2024-03-11T21:34:52.334700+0000","last_peered":"2024-03-11T21:34:52.334700+0000","last_clean":"2024-03-11T21:34:52.334700+0000","last_became_active":"2024-03-11T21:34:41.154216+0000","last_became_peered":"2024-03-11T21:34:41.154216+0000","last_unstale":"2024-03-11T21:34:52.334700+0000","last_undegraded":"2024-03-11T21:34:52.334700+0000","last_fullsized":"2024-03-11T21:34:52.334700+0000","mapping_epoch":226,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":227,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:34:30.609917+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:41.544550+0000","last_clean_scrub_stamp":"2024-03-11T21:34:30.609917+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:35:42.029275+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,7],"acting":[1,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1e","version":"0'0","reported_seq":382,"reported_epoch":235,"state":"active+clean","last_fresh":"2024-03-11T21:34:52.334502+0000","last_change":"2024-03-11T21:32:18.578157+0000","last_active":"2024-03-11T21:34:52.334502+0000","last_peered":"2024-03-11T21:34:52.334502+0000","last_clean":"2024-03-11T21:34:52.334502+0000","last_became_active":"2024-03-11T21:25:38.537283+0000","last_became_peered":"2024-03-11T21:25:38.537283+0000","last_unstale":"2024-03-11T21:34:52.334502+0000","last_undegraded":"2024-03-11T21:34:52.334502+0000","last_fullsized":"2024-03-11T21:34:52.334502+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:32:18.578038+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:37.512018+0000","last_clean_scrub_stamp":"2024-03-11T21:32:18.578038+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:34:55.849726+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.18","version":"74'9","reported_seq":400,"reported_epoch":234,"state":"active+clean","last_fresh":"2024-03-11T21:34:51.293772+0000","last_change":"2024-03-11T21:34:32.763177+0000","last_active":"2024-03-11T21:34:51.293772+0000","last_peered":"2024-03-11T21:34:51.293772+0000","last_clean":"2024-03-11T21:34:51.293772+0000","last_became_active":"2024-03-11T21:25:40.552216+0000","last_became_peered":"2024-03-11T21:25:40.552216+0000","last_unstale":"2024-03-11T21:34:51.293772+0000","last_undegraded":"2024-03-11T21:34:51.293772+0000","last_fullsized":"2024-03-11T21:34:51.293772+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'9","last_scrub_stamp":"2024-03-11T21:34:32.763054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:39.525554+0000","last_clean_scrub_stamp":"2024-03-11T21:34:32.763054+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:35:36.156930+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"8.17","version":"234'9047","reported_seq":21384,"reported_epoch":234,"state":"active+clean","last_fresh":"2024-03-11T21:34:51.915554+0000","last_change":"2024-03-11T21:34:51.320663+0000","last_active":"2024-03-11T21:34:51.915554+0000","last_peered":"2024-03-11T21:34:51.915554+0000","last_clean":"2024-03-11T21:34:51.915554+0000","last_became_active":"2024-03-11T21:34:51.300935+0000","last_became_peered":"2024-03-11T21:34:51.300935+0000","last_unstale":"2024-03-11T21:34:51.915554+0000","last_undegraded":"2024-03-11T21:34:51.915554+0000","last_fullsized":"2024-03-11T21:34:51.915554+0000","mapping_epoch":233,"log_start":"172'3885","ondisk_log_start":"172'3885","created":181,"last_epoch_clean":234,"parent":"0.0","parent_split_bits":5,"last_scrub":"219'7594","last_scrub_stamp":"2024-03-11T21:34:10.514580+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:31:51.219688+0000","last_clean_scrub_stamp":"2024-03-11T21:34:10.514580+0000","objects_scrubbed":2575,"log_size":5162,"log_dups_size":0,"ondisk_log_size":5162,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:35:11.641075+0000","scrub_duration":815,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":72155136,"num_objects":1101,"num_object_clones":0,"num_object_copies":3303,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1101,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1474,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1222,"num_bytes_recovered":80084992,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":368,"reported_epoch":234,"state":"active+clean","last_fresh":"2024-03-11T21:34:51.300400+0000","last_change":"2024-03-11T21:32:12.654714+0000","last_active":"2024-03-11T21:34:51.300400+0000","last_peered":"2024-03-11T21:34:51.300400+0000","last_clean":"2024-03-11T21:34:51.300400+0000","last_became_active":"2024-03-11T21:25:44.590234+0000","last_became_peered":"2024-03-11T21:25:44.590234+0000","last_unstale":"2024-03-11T21:34:51.300400+0000","last_undegraded":"2024-03-11T21:34:51.300400+0000","last_fullsized":"2024-03-11T21:34:51.300400+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:32:12.654573+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:32:12.654573+0000","last_clean_scrub_stamp":"2024-03-11T21:32:12.654573+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":43,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3],"acting":[5,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":25,"reported_epoch":234,"state":"active+clean","last_fresh":"2024-03-11T21:34:51.296784+0000","last_change":"2024-03-11T21:34:41.148957+0000","last_active":"2024-03-11T21:34:51.296784+0000","last_peered":"2024-03-11T21:34:51.296784+0000","last_clean":"2024-03-11T21:34:51.296784+0000","last_became_active":"2024-03-11T21:34:41.145299+0000","last_became_peered":"2024-03-11T21:34:41.145299+0000","last_unstale":"2024-03-11T21:34:51.296784+0000","last_undegraded":"2024-03-11T21:34:51.296784+0000","last_fullsized":"2024-03-11T21:34:51.296784+0000","mapping_epoch":226,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":227,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:31:56.567802+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:41.544550+0000","last_clean_scrub_stamp":"2024-03-11T21:31:56.567802+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.1d","version":"0'0","reported_seq":387,"reported_epoch":234,"state":"active+clean","last_fresh":"2024-03-11T21:34:51.296977+0000","last_change":"2024-03-11T21:34:05.871427+0000","last_active":"2024-03-11T21:34:51.296977+0000","last_peered":"2024-03-11T21:34:51.296977+0000","last_clean":"2024-03-11T21:34:51.296977+0000","last_became_active":"2024-03-11T21:25:38.542628+0000","last_became_peered":"2024-03-11T21:25:38.542628+0000","last_unstale":"2024-03-11T21:34:51.296977+0000","last_undegraded":"2024-03-11T21:34:51.296977+0000","last_fullsized":"2024-03-11T21:34:51.296977+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:34:05.871316+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:37.512018+0000","last_clean_scrub_stamp":"2024-03-11T21:34:05.871316+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:35:08.808268+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,0],"acting":[7,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.1b","version":"74'5","reported_seq":378,"reported_epoch":234,"state":"active+clean","last_fresh":"2024-03-11T21:34:51.294095+0000","last_change":"2024-03-11T21:31:57.621692+0000","last_active":"2024-03-11T21:34:51.294095+0000","last_peered":"2024-03-11T21:34:51.294095+0000","last_clean":"2024-03-11T21:34:51.294095+0000","last_became_active":"2024-03-11T21:25:40.555826+0000","last_became_peered":"2024-03-11T21:25:40.555826+0000","last_unstale":"2024-03-11T21:34:51.294095+0000","last_undegraded":"2024-03-11T21:34:51.294095+0000","last_fullsized":"2024-03-11T21:34:51.294095+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'5","last_scrub_stamp":"2024-03-11T21:31:57.621580+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:39.525554+0000","last_clean_scrub_stamp":"2024-03-11T21:31:57.621580+0000","objects_scrubbed":2,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":34,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":11,"num_read_kb":7,"num_write":6,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,1],"acting":[4,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"8.16","version":"235'9032","reported_seq":22553,"reported_epoch":235,"state":"active+clean","last_fresh":"2024-03-11T21:34:52.720488+0000","last_change":"2024-03-11T21:33:59.690527+0000","last_active":"2024-03-11T21:34:52.720488+0000","last_peered":"2024-03-11T21:34:52.720488+0000","last_clean":"2024-03-11T21:34:52.720488+0000","last_became_active":"2024-03-11T21:33:53.254658+0000","last_became_peered":"2024-03-11T21:33:53.254658+0000","last_unstale":"2024-03-11T21:34:52.720488+0000","last_undegraded":"2024-03-11T21:34:52.720488+0000","last_fullsized":"2024-03-11T21:34:52.720488+0000","mapping_epoch":208,"log_start":"170'3471","ondisk_log_start":"170'3471","created":181,"last_epoch_clean":209,"parent":"0.0","parent_split_bits":5,"last_scrub":"212'7305","last_scrub_stamp":"2024-03-11T21:33:59.690214+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:31:51.219688+0000","last_clean_scrub_stamp":"2024-03-11T21:33:59.690214+0000","objects_scrubbed":2873,"log_size":5561,"log_dups_size":0,"ondisk_log_size":5561,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:35:08.100514+0000","scrub_duration":952,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":74055680,"num_objects":1130,"num_object_clones":0,"num_object_copies":3390,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1130,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1743,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5],"acting":[3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":362,"reported_epoch":231,"state":"active+clean","last_fresh":"2024-03-11T21:34:48.223791+0000","last_change":"2024-03-11T21:34:31.720942+0000","last_active":"2024-03-11T21:34:48.223791+0000","last_peered":"2024-03-11T21:34:48.223791+0000","last_clean":"2024-03-11T21:34:48.223791+0000","last_became_active":"2024-03-11T21:25:44.589786+0000","last_became_peered":"2024-03-11T21:25:44.589786+0000","last_unstale":"2024-03-11T21:34:48.223791+0000","last_undegraded":"2024-03-11T21:34:48.223791+0000","last_fullsized":"2024-03-11T21:34:48.223791+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:34:31.720821+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:43.559452+0000","last_clean_scrub_stamp":"2024-03-11T21:34:31.720821+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:35:39.422862+0000","scrub_duration":33,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,7],"acting":[0,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":383,"reported_epoch":234,"state":"active+clean","last_fresh":"2024-03-11T21:34:51.300580+0000","last_change":"2024-03-11T21:34:41.147546+0000","last_active":"2024-03-11T21:34:51.300580+0000","last_peered":"2024-03-11T21:34:51.300580+0000","last_clean":"2024-03-11T21:34:51.300580+0000","last_became_active":"2024-03-11T21:34:41.147185+0000","last_became_peered":"2024-03-11T21:34:41.147185+0000","last_unstale":"2024-03-11T21:34:51.300580+0000","last_undegraded":"2024-03-11T21:34:51.300580+0000","last_fullsized":"2024-03-11T21:34:51.300580+0000","mapping_epoch":226,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":227,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:32:21.535034+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:32:21.535034+0000","last_clean_scrub_stamp":"2024-03-11T21:32:21.535034+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,7],"acting":[5,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1c","version":"0'0","reported_seq":387,"reported_epoch":234,"state":"active+clean","last_fresh":"2024-03-11T21:34:51.297837+0000","last_change":"2024-03-11T21:34:04.858406+0000","last_active":"2024-03-11T21:34:51.297837+0000","last_peered":"2024-03-11T21:34:51.297837+0000","last_clean":"2024-03-11T21:34:51.297837+0000","last_became_active":"2024-03-11T21:25:38.541364+0000","last_became_peered":"2024-03-11T21:25:38.541364+0000","last_unstale":"2024-03-11T21:34:51.297837+0000","last_undegraded":"2024-03-11T21:34:51.297837+0000","last_fullsized":"2024-03-11T21:34:51.297837+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:34:04.858280+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:28:32.566437+0000","last_clean_scrub_stamp":"2024-03-11T21:34:04.858280+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:35:13.190197+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.1a","version":"74'9","reported_seq":392,"reported_epoch":234,"state":"active+clean","last_fresh":"2024-03-11T21:34:51.294666+0000","last_change":"2024-03-11T21:32:04.628796+0000","last_active":"2024-03-11T21:34:51.294666+0000","last_peered":"2024-03-11T21:34:51.294666+0000","last_clean":"2024-03-11T21:34:51.294666+0000","last_became_active":"2024-03-11T21:25:40.556629+0000","last_became_peered":"2024-03-11T21:25:40.556629+0000","last_unstale":"2024-03-11T21:34:51.294666+0000","last_undegraded":"2024-03-11T21:34:51.294666+0000","last_fullsized":"2024-03-11T21:34:51.294666+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'9","last_scrub_stamp":"2024-03-11T21:32:04.628652+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:39.525554+0000","last_clean_scrub_stamp":"2024-03-11T21:32:04.628652+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0],"acting":[4,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"8.11","version":"230'9125","reported_seq":9313,"reported_epoch":231,"state":"active+remapped+backfill_toofull","last_fresh":"2024-03-11T21:34:48.223227+0000","last_change":"2024-03-11T21:34:30.382622+0000","last_active":"2024-03-11T21:34:48.223227+0000","last_peered":"2024-03-11T21:34:48.223227+0000","last_clean":"2024-03-11T21:32:58.048747+0000","last_became_active":"2024-03-11T21:32:59.353819+0000","last_became_peered":"2024-03-11T21:32:59.353819+0000","last_unstale":"2024-03-11T21:34:48.223227+0000","last_undegraded":"2024-03-11T21:34:48.223227+0000","last_fullsized":"2024-03-11T21:34:48.223227+0000","mapping_epoch":184,"log_start":"173'4138","ondisk_log_start":"173'4138","created":181,"last_epoch_clean":182,"parent":"0.0","parent_split_bits":5,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:31:51.219688+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:31:51.219688+0000","last_clean_scrub_stamp":"2024-03-11T21:31:51.219688+0000","objects_scrubbed":0,"log_size":4987,"log_dups_size":0,"ondisk_log_size":4987,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":80117760,"num_objects":1223,"num_object_clones":0,"num_object_copies":3669,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":3669,"num_objects_unfound":0,"num_objects_dirty":1223,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6383,"num_write_kb":243360,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":131072,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[0,1,2],"avail_no_missing":["0","1","2"],"object_location_counts":[{"shards":"0,1,2","objects":1223}],"blocked_by":[],"up_primary":5,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.1f","version":"0'0","reported_seq":370,"reported_epoch":235,"state":"active+clean","last_fresh":"2024-03-11T21:34:52.334219+0000","last_change":"2024-03-11T21:34:05.863007+0000","last_active":"2024-03-11T21:34:52.334219+0000","last_peered":"2024-03-11T21:34:52.334219+0000","last_clean":"2024-03-11T21:34:52.334219+0000","last_became_active":"2024-03-11T21:25:45.302637+0000","last_became_peered":"2024-03-11T21:25:45.302637+0000","last_unstale":"2024-03-11T21:34:52.334219+0000","last_undegraded":"2024-03-11T21:34:52.334219+0000","last_fullsized":"2024-03-11T21:34:52.334219+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:34:05.862891+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:43.559452+0000","last_clean_scrub_stamp":"2024-03-11T21:34:05.862891+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:35:13.148496+0000","scrub_duration":122,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1c","version":"0'0","reported_seq":372,"reported_epoch":234,"state":"active+clean","last_fresh":"2024-03-11T21:34:51.294259+0000","last_change":"2024-03-11T21:34:11.766184+0000","last_active":"2024-03-11T21:34:51.294259+0000","last_peered":"2024-03-11T21:34:51.294259+0000","last_clean":"2024-03-11T21:34:51.294259+0000","last_became_active":"2024-03-11T21:25:42.576064+0000","last_became_peered":"2024-03-11T21:25:42.576064+0000","last_unstale":"2024-03-11T21:34:51.294259+0000","last_undegraded":"2024-03-11T21:34:51.294259+0000","last_fullsized":"2024-03-11T21:34:51.294259+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:34:11.766066+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:41.544550+0000","last_clean_scrub_stamp":"2024-03-11T21:34:11.766066+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:35:30.920690+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1b","version":"0'0","reported_seq":393,"reported_epoch":235,"state":"active+clean","last_fresh":"2024-03-11T21:34:52.335968+0000","last_change":"2024-03-11T21:34:41.147151+0000","last_active":"2024-03-11T21:34:52.335968+0000","last_peered":"2024-03-11T21:34:52.335968+0000","last_clean":"2024-03-11T21:34:52.335968+0000","last_became_active":"2024-03-11T21:34:41.145184+0000","last_became_peered":"2024-03-11T21:34:41.145184+0000","last_unstale":"2024-03-11T21:34:52.335968+0000","last_undegraded":"2024-03-11T21:34:52.335968+0000","last_fullsized":"2024-03-11T21:34:52.335968+0000","mapping_epoch":226,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":227,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:34:27.998724+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:34:27.998724+0000","last_clean_scrub_stamp":"2024-03-11T21:34:27.998724+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:35:45.823051+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1d","version":"74'12","reported_seq":392,"reported_epoch":235,"state":"active+clean","last_fresh":"2024-03-11T21:34:52.332736+0000","last_change":"2024-03-11T21:31:20.514563+0000","last_active":"2024-03-11T21:34:52.332736+0000","last_peered":"2024-03-11T21:34:52.332736+0000","last_clean":"2024-03-11T21:34:52.332736+0000","last_became_active":"2024-03-11T21:25:40.556330+0000","last_became_peered":"2024-03-11T21:25:40.556330+0000","last_unstale":"2024-03-11T21:34:52.332736+0000","last_undegraded":"2024-03-11T21:34:52.332736+0000","last_fullsized":"2024-03-11T21:34:52.332736+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'12","last_scrub_stamp":"2024-03-11T21:31:20.514423+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:39.525554+0000","last_clean_scrub_stamp":"2024-03-11T21:31:20.514423+0000","objects_scrubbed":5,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:34:55.849418+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":25,"num_read_kb":16,"num_write":14,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.10","version":"233'9171","reported_seq":20504,"reported_epoch":234,"state":"peering","last_fresh":"2024-03-11T21:34:51.297083+0000","last_change":"2024-03-11T21:34:51.297083+0000","last_active":"2024-03-11T21:34:51.297083+0000","last_peered":"2024-03-11T21:34:51.189717+0000","last_clean":"2024-03-11T21:34:40.002419+0000","last_became_active":"2024-03-11T21:34:42.677989+0000","last_became_peered":"2024-03-11T21:34:42.677989+0000","last_unstale":"2024-03-11T21:34:51.297083+0000","last_undegraded":"2024-03-11T21:34:51.297083+0000","last_fullsized":"2024-03-11T21:34:51.297083+0000","mapping_epoch":234,"log_start":"170'3594","ondisk_log_start":"170'3594","created":181,"last_epoch_clean":233,"parent":"0.0","parent_split_bits":5,"last_scrub":"219'7729","last_scrub_stamp":"2024-03-11T21:34:10.807906+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:31:51.219688+0000","last_clean_scrub_stamp":"2024-03-11T21:34:10.807906+0000","objects_scrubbed":2679,"log_size":5577,"log_dups_size":0,"ondisk_log_size":5577,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"no scrub is scheduled","scrub_duration":540,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":80478208,"num_objects":1228,"num_object_clones":0,"num_object_copies":3684,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1228,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1451,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1258,"num_bytes_recovered":82444288,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,4],"acting":[7,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.1e","version":"0'0","reported_seq":361,"reported_epoch":234,"state":"active+clean","last_fresh":"2024-03-11T21:34:51.294323+0000","last_change":"2024-03-11T21:32:19.599182+0000","last_active":"2024-03-11T21:34:51.294323+0000","last_peered":"2024-03-11T21:34:51.294323+0000","last_clean":"2024-03-11T21:34:51.294323+0000","last_became_active":"2024-03-11T21:25:45.300355+0000","last_became_peered":"2024-03-11T21:25:45.300355+0000","last_unstale":"2024-03-11T21:34:51.294323+0000","last_undegraded":"2024-03-11T21:34:51.294323+0000","last_fullsized":"2024-03-11T21:34:51.294323+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:32:19.599050+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:43.559452+0000","last_clean_scrub_stamp":"2024-03-11T21:32:19.599050+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5],"acting":[4,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1d","version":"0'0","reported_seq":374,"reported_epoch":235,"state":"active+clean","last_fresh":"2024-03-11T21:34:52.333731+0000","last_change":"2024-03-11T21:34:12.507829+0000","last_active":"2024-03-11T21:34:52.333731+0000","last_peered":"2024-03-11T21:34:52.333731+0000","last_clean":"2024-03-11T21:34:52.333731+0000","last_became_active":"2024-03-11T21:25:42.579407+0000","last_became_peered":"2024-03-11T21:25:42.579407+0000","last_unstale":"2024-03-11T21:34:52.333731+0000","last_undegraded":"2024-03-11T21:34:52.333731+0000","last_fullsized":"2024-03-11T21:34:52.333731+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:34:12.507702+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:30:18.678531+0000","last_clean_scrub_stamp":"2024-03-11T21:34:12.507702+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:35:16.018362+0000","scrub_duration":16,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1a","version":"0'0","reported_seq":400,"reported_epoch":234,"state":"active+clean","last_fresh":"2024-03-11T21:34:51.298954+0000","last_change":"2024-03-11T21:34:41.145362+0000","last_active":"2024-03-11T21:34:51.298954+0000","last_peered":"2024-03-11T21:34:51.298954+0000","last_clean":"2024-03-11T21:34:51.298954+0000","last_became_active":"2024-03-11T21:34:41.145014+0000","last_became_peered":"2024-03-11T21:34:41.145014+0000","last_unstale":"2024-03-11T21:34:51.298954+0000","last_undegraded":"2024-03-11T21:34:51.298954+0000","last_fullsized":"2024-03-11T21:34:51.298954+0000","mapping_epoch":226,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":227,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:34:20.415006+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:37.512018+0000","last_clean_scrub_stamp":"2024-03-11T21:34:20.415006+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:35:46.870412+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.1c","version":"74'15","reported_seq":402,"reported_epoch":235,"state":"active+clean","last_fresh":"2024-03-11T21:34:52.295918+0000","last_change":"2024-03-11T21:34:10.311308+0000","last_active":"2024-03-11T21:34:52.295918+0000","last_peered":"2024-03-11T21:34:52.295918+0000","last_clean":"2024-03-11T21:34:52.295918+0000","last_became_active":"2024-03-11T21:25:40.551453+0000","last_became_peered":"2024-03-11T21:25:40.551453+0000","last_unstale":"2024-03-11T21:34:52.295918+0000","last_undegraded":"2024-03-11T21:34:52.295918+0000","last_fullsized":"2024-03-11T21:34:52.295918+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'15","last_scrub_stamp":"2024-03-11T21:34:10.311181+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:39.525554+0000","last_clean_scrub_stamp":"2024-03-11T21:34:10.311181+0000","objects_scrubbed":7,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:35:36.061874+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"8.13","version":"235'9135","reported_seq":19393,"reported_epoch":235,"state":"active+clean","last_fresh":"2024-03-11T21:34:52.783397+0000","last_change":"2024-03-11T21:34:02.249988+0000","last_active":"2024-03-11T21:34:52.783397+0000","last_peered":"2024-03-11T21:34:52.783397+0000","last_clean":"2024-03-11T21:34:52.783397+0000","last_became_active":"2024-03-11T21:33:33.425325+0000","last_became_peered":"2024-03-11T21:33:33.425325+0000","last_unstale":"2024-03-11T21:34:52.783397+0000","last_undegraded":"2024-03-11T21:34:52.783397+0000","last_fullsized":"2024-03-11T21:34:52.783397+0000","mapping_epoch":199,"log_start":"170'3596","ondisk_log_start":"170'3596","created":181,"last_epoch_clean":200,"parent":"0.0","parent_split_bits":5,"last_scrub":"213'7420","last_scrub_stamp":"2024-03-11T21:34:02.249703+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:31:51.219688+0000","last_clean_scrub_stamp":"2024-03-11T21:34:02.249703+0000","objects_scrubbed":2850,"log_size":5539,"log_dups_size":0,"ondisk_log_size":5539,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:35:24.228384+0000","scrub_duration":580,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":74186752,"num_objects":1132,"num_object_clones":0,"num_object_copies":3396,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1132,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1718,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,1],"acting":[3,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.1d","version":"0'0","reported_seq":362,"reported_epoch":235,"state":"active+clean","last_fresh":"2024-03-11T21:34:52.333184+0000","last_change":"2024-03-11T21:31:18.798813+0000","last_active":"2024-03-11T21:34:52.333184+0000","last_peered":"2024-03-11T21:34:52.333184+0000","last_clean":"2024-03-11T21:34:52.333184+0000","last_became_active":"2024-03-11T21:25:44.588845+0000","last_became_peered":"2024-03-11T21:25:44.588845+0000","last_unstale":"2024-03-11T21:34:52.333184+0000","last_undegraded":"2024-03-11T21:34:52.333184+0000","last_fullsized":"2024-03-11T21:34:52.333184+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:31:18.798701+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:43.559452+0000","last_clean_scrub_stamp":"2024-03-11T21:31:18.798701+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.1e","version":"0'0","reported_seq":367,"reported_epoch":231,"state":"active+clean","last_fresh":"2024-03-11T21:34:48.243501+0000","last_change":"2024-03-11T21:32:07.397655+0000","last_active":"2024-03-11T21:34:48.243501+0000","last_peered":"2024-03-11T21:34:48.243501+0000","last_clean":"2024-03-11T21:34:48.243501+0000","last_became_active":"2024-03-11T21:25:42.582959+0000","last_became_peered":"2024-03-11T21:25:42.582959+0000","last_unstale":"2024-03-11T21:34:48.243501+0000","last_undegraded":"2024-03-11T21:34:48.243501+0000","last_fullsized":"2024-03-11T21:34:48.243501+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:32:07.397538+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:26:58.335431+0000","last_clean_scrub_stamp":"2024-03-11T21:32:07.397538+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.19","version":"67'2","reported_seq":380,"reported_epoch":235,"state":"active+clean","last_fresh":"2024-03-11T21:34:52.333983+0000","last_change":"2024-03-11T21:32:15.509684+0000","last_active":"2024-03-11T21:34:52.333983+0000","last_peered":"2024-03-11T21:34:52.333983+0000","last_clean":"2024-03-11T21:34:52.333983+0000","last_became_active":"2024-03-11T21:25:38.539243+0000","last_became_peered":"2024-03-11T21:25:38.539243+0000","last_unstale":"2024-03-11T21:34:52.333983+0000","last_undegraded":"2024-03-11T21:34:52.333983+0000","last_fullsized":"2024-03-11T21:34:52.333983+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'2","last_scrub_stamp":"2024-03-11T21:32:15.509542+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:37.512018+0000","last_clean_scrub_stamp":"2024-03-11T21:32:15.509542+0000","objects_scrubbed":2,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:34:55.849709+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":957,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":7,"num_read_kb":7,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,0],"acting":[3,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1f","version":"124'16","reported_seq":427,"reported_epoch":234,"state":"active+clean","last_fresh":"2024-03-11T21:34:51.299498+0000","last_change":"2024-03-11T21:34:12.415133+0000","last_active":"2024-03-11T21:34:51.299498+0000","last_peered":"2024-03-11T21:34:51.299498+0000","last_clean":"2024-03-11T21:34:51.299498+0000","last_became_active":"2024-03-11T21:32:22.798242+0000","last_became_peered":"2024-03-11T21:32:22.798242+0000","last_unstale":"2024-03-11T21:34:51.299498+0000","last_undegraded":"2024-03-11T21:34:51.299498+0000","last_fullsized":"2024-03-11T21:34:51.299498+0000","mapping_epoch":166,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":167,"parent":"0.0","parent_split_bits":0,"last_scrub":"124'16","last_scrub_stamp":"2024-03-11T21:34:12.414804+0000","last_deep_scrub":"124'16","last_deep_scrub_stamp":"2024-03-11T21:29:30.873490+0000","last_clean_scrub_stamp":"2024-03-11T21:34:12.414804+0000","objects_scrubbed":5,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:35:36.222319+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,0],"acting":[6,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"8.12","version":"234'9262","reported_seq":22559,"reported_epoch":234,"state":"active+clean","last_fresh":"2024-03-11T21:34:51.854129+0000","last_change":"2024-03-11T21:34:49.254204+0000","last_active":"2024-03-11T21:34:51.854129+0000","last_peered":"2024-03-11T21:34:51.854129+0000","last_clean":"2024-03-11T21:34:51.854129+0000","last_became_active":"2024-03-11T21:34:49.234452+0000","last_became_peered":"2024-03-11T21:34:49.234452+0000","last_unstale":"2024-03-11T21:34:51.854129+0000","last_undegraded":"2024-03-11T21:34:51.854129+0000","last_fullsized":"2024-03-11T21:34:51.854129+0000","mapping_epoch":231,"log_start":"169'3159","ondisk_log_start":"169'3159","created":181,"last_epoch_clean":232,"parent":"0.0","parent_split_bits":5,"last_scrub":"211'7452","last_scrub_stamp":"2024-03-11T21:33:58.939672+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:31:51.219688+0000","last_clean_scrub_stamp":"2024-03-11T21:33:58.939672+0000","objects_scrubbed":3006,"log_size":6103,"log_dups_size":0,"ondisk_log_size":6103,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:35:28.047990+0000","scrub_duration":697,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":77725696,"num_objects":1186,"num_object_clones":0,"num_object_copies":3558,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1186,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1820,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1368,"num_bytes_recovered":89653248,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,2],"acting":[6,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.1c","version":"74'1","reported_seq":29,"reported_epoch":234,"state":"active+clean","last_fresh":"2024-03-11T21:34:51.297715+0000","last_change":"2024-03-11T21:34:41.161206+0000","last_active":"2024-03-11T21:34:51.297715+0000","last_peered":"2024-03-11T21:34:51.297715+0000","last_clean":"2024-03-11T21:34:51.297715+0000","last_became_active":"2024-03-11T21:34:41.144839+0000","last_became_peered":"2024-03-11T21:34:41.144839+0000","last_unstale":"2024-03-11T21:34:51.297715+0000","last_undegraded":"2024-03-11T21:34:51.297715+0000","last_fullsized":"2024-03-11T21:34:51.297715+0000","mapping_epoch":226,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":227,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'1","last_scrub_stamp":"2024-03-11T21:34:16.378072+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:43.559452+0000","last_clean_scrub_stamp":"2024-03-11T21:34:16.378072+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:35:39.328627+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":370,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":370,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.1f","version":"229'10","reported_seq":531,"reported_epoch":234,"state":"active+clean","last_fresh":"2024-03-11T21:34:51.299452+0000","last_change":"2024-03-11T21:34:41.168287+0000","last_active":"2024-03-11T21:34:51.299452+0000","last_peered":"2024-03-11T21:34:51.299452+0000","last_clean":"2024-03-11T21:34:51.299452+0000","last_became_active":"2024-03-11T21:34:41.145862+0000","last_became_peered":"2024-03-11T21:34:41.145862+0000","last_unstale":"2024-03-11T21:34:51.299452+0000","last_undegraded":"2024-03-11T21:34:51.299452+0000","last_fullsized":"2024-03-11T21:34:51.299452+0000","mapping_epoch":226,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":227,"parent":"0.0","parent_split_bits":0,"last_scrub":"154'9","last_scrub_stamp":"2024-03-11T21:34:23.327729+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:41.544550+0000","last_clean_scrub_stamp":"2024-03-11T21:34:23.327729+0000","objects_scrubbed":1,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:35:46.133503+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.18","version":"67'1","reported_seq":413,"reported_epoch":234,"state":"active+clean","last_fresh":"2024-03-11T21:34:51.301213+0000","last_change":"2024-03-11T21:34:41.156671+0000","last_active":"2024-03-11T21:34:51.301213+0000","last_peered":"2024-03-11T21:34:51.301213+0000","last_clean":"2024-03-11T21:34:51.301213+0000","last_became_active":"2024-03-11T21:34:41.138664+0000","last_became_peered":"2024-03-11T21:34:41.138664+0000","last_unstale":"2024-03-11T21:34:51.301213+0000","last_undegraded":"2024-03-11T21:34:51.301213+0000","last_fullsized":"2024-03-11T21:34:51.301213+0000","mapping_epoch":226,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":227,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'1","last_scrub_stamp":"2024-03-11T21:34:32.363650+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:37.512018+0000","last_clean_scrub_stamp":"2024-03-11T21:34:32.363650+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:35:37.328713+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":436,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":7,"num_read_kb":7,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":872,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,7],"acting":[5,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1e","version":"74'10","reported_seq":422,"reported_epoch":231,"state":"active+clean","last_fresh":"2024-03-11T21:34:48.243417+0000","last_change":"2024-03-11T21:34:41.185277+0000","last_active":"2024-03-11T21:34:48.243417+0000","last_peered":"2024-03-11T21:34:48.243417+0000","last_clean":"2024-03-11T21:34:48.243417+0000","last_became_active":"2024-03-11T21:34:41.147199+0000","last_became_peered":"2024-03-11T21:34:41.147199+0000","last_unstale":"2024-03-11T21:34:48.243417+0000","last_undegraded":"2024-03-11T21:34:48.243417+0000","last_fullsized":"2024-03-11T21:34:48.243417+0000","mapping_epoch":226,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":227,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'10","last_scrub_stamp":"2024-03-11T21:32:53.668918+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:39.525554+0000","last_clean_scrub_stamp":"2024-03-11T21:32:53.668918+0000","objects_scrubbed":5,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.13","version":"0'0","reported_seq":369,"reported_epoch":235,"state":"active+clean","last_fresh":"2024-03-11T21:34:52.336246+0000","last_change":"2024-03-11T21:34:13.822920+0000","last_active":"2024-03-11T21:34:52.336246+0000","last_peered":"2024-03-11T21:34:52.336246+0000","last_clean":"2024-03-11T21:34:52.336246+0000","last_became_active":"2024-03-11T21:25:45.302102+0000","last_became_peered":"2024-03-11T21:25:45.302102+0000","last_unstale":"2024-03-11T21:34:52.336246+0000","last_undegraded":"2024-03-11T21:34:52.336246+0000","last_fullsized":"2024-03-11T21:34:52.336246+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:34:13.822803+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:43.559452+0000","last_clean_scrub_stamp":"2024-03-11T21:34:13.822803+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:35:27.875351+0000","scrub_duration":53,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":25,"reported_epoch":234,"state":"active+clean","last_fresh":"2024-03-11T21:34:51.297734+0000","last_change":"2024-03-11T21:34:41.143920+0000","last_active":"2024-03-11T21:34:51.297734+0000","last_peered":"2024-03-11T21:34:51.297734+0000","last_clean":"2024-03-11T21:34:51.297734+0000","last_became_active":"2024-03-11T21:34:41.142569+0000","last_became_peered":"2024-03-11T21:34:41.142569+0000","last_unstale":"2024-03-11T21:34:51.297734+0000","last_undegraded":"2024-03-11T21:34:51.297734+0000","last_fullsized":"2024-03-11T21:34:51.297734+0000","mapping_epoch":226,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":227,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:32:08.612508+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:41.544550+0000","last_clean_scrub_stamp":"2024-03-11T21:32:08.612508+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.17","version":"0'0","reported_seq":380,"reported_epoch":234,"state":"active+clean","last_fresh":"2024-03-11T21:34:51.299365+0000","last_change":"2024-03-11T21:32:00.758583+0000","last_active":"2024-03-11T21:34:51.299365+0000","last_peered":"2024-03-11T21:34:51.299365+0000","last_clean":"2024-03-11T21:34:51.299365+0000","last_became_active":"2024-03-11T21:25:38.540349+0000","last_became_peered":"2024-03-11T21:25:38.540349+0000","last_unstale":"2024-03-11T21:34:51.299365+0000","last_undegraded":"2024-03-11T21:34:51.299365+0000","last_fullsized":"2024-03-11T21:34:51.299365+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:32:00.758492+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:28:25.918101+0000","last_clean_scrub_stamp":"2024-03-11T21:32:00.758492+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:34:53.059840+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,2],"acting":[6,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.11","version":"74'11","reported_seq":424,"reported_epoch":235,"state":"active+clean","last_fresh":"2024-03-11T21:34:52.334216+0000","last_change":"2024-03-11T21:34:41.188418+0000","last_active":"2024-03-11T21:34:52.334216+0000","last_peered":"2024-03-11T21:34:52.334216+0000","last_clean":"2024-03-11T21:34:52.334216+0000","last_became_active":"2024-03-11T21:34:41.154298+0000","last_became_peered":"2024-03-11T21:34:41.154298+0000","last_unstale":"2024-03-11T21:34:52.334216+0000","last_undegraded":"2024-03-11T21:34:52.334216+0000","last_fullsized":"2024-03-11T21:34:52.334216+0000","mapping_epoch":226,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":227,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'11","last_scrub_stamp":"2024-03-11T21:31:17.450051+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:39.525554+0000","last_clean_scrub_stamp":"2024-03-11T21:31:17.450051+0000","objects_scrubbed":5,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:34:55.849506+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":25,"reported_epoch":234,"state":"active+clean","last_fresh":"2024-03-11T21:34:51.296625+0000","last_change":"2024-03-11T21:34:41.140372+0000","last_active":"2024-03-11T21:34:51.296625+0000","last_peered":"2024-03-11T21:34:51.296625+0000","last_clean":"2024-03-11T21:34:51.296625+0000","last_became_active":"2024-03-11T21:34:41.137235+0000","last_became_peered":"2024-03-11T21:34:41.137235+0000","last_unstale":"2024-03-11T21:34:51.296625+0000","last_undegraded":"2024-03-11T21:34:51.296625+0000","last_fullsized":"2024-03-11T21:34:51.296625+0000","mapping_epoch":226,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":227,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:34:20.478420+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:43.559452+0000","last_clean_scrub_stamp":"2024-03-11T21:34:20.478420+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:35:22.077123+0000","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,4],"acting":[7,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":392,"reported_epoch":234,"state":"active+clean","last_fresh":"2024-03-11T21:34:51.298947+0000","last_change":"2024-03-11T21:34:41.156388+0000","last_active":"2024-03-11T21:34:51.298947+0000","last_peered":"2024-03-11T21:34:51.298947+0000","last_clean":"2024-03-11T21:34:51.298947+0000","last_became_active":"2024-03-11T21:34:41.156036+0000","last_became_peered":"2024-03-11T21:34:41.156036+0000","last_unstale":"2024-03-11T21:34:51.298947+0000","last_undegraded":"2024-03-11T21:34:51.298947+0000","last_fullsized":"2024-03-11T21:34:51.298947+0000","mapping_epoch":226,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":227,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:34:09.424875+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:29:39.923895+0000","last_clean_scrub_stamp":"2024-03-11T21:34:09.424875+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:35:14.239741+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.16","version":"0'0","reported_seq":387,"reported_epoch":234,"state":"active+clean","last_fresh":"2024-03-11T21:34:51.301296+0000","last_change":"2024-03-11T21:32:56.247901+0000","last_active":"2024-03-11T21:34:51.301296+0000","last_peered":"2024-03-11T21:34:51.301296+0000","last_clean":"2024-03-11T21:34:51.301296+0000","last_became_active":"2024-03-11T21:25:38.539103+0000","last_became_peered":"2024-03-11T21:25:38.539103+0000","last_unstale":"2024-03-11T21:34:51.301296+0000","last_undegraded":"2024-03-11T21:34:51.301296+0000","last_fullsized":"2024-03-11T21:34:51.301296+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:32:56.247825+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:27:11.938909+0000","last_clean_scrub_stamp":"2024-03-11T21:32:56.247825+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,2],"acting":[5,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.10","version":"94'6","reported_seq":385,"reported_epoch":235,"state":"active+clean","last_fresh":"2024-03-11T21:34:52.332567+0000","last_change":"2024-03-11T21:34:00.727542+0000","last_active":"2024-03-11T21:34:52.332567+0000","last_peered":"2024-03-11T21:34:52.332567+0000","last_clean":"2024-03-11T21:34:52.332567+0000","last_became_active":"2024-03-11T21:25:40.552720+0000","last_became_peered":"2024-03-11T21:25:40.552720+0000","last_unstale":"2024-03-11T21:34:52.332567+0000","last_undegraded":"2024-03-11T21:34:52.332567+0000","last_fullsized":"2024-03-11T21:34:52.332567+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"94'6","last_scrub_stamp":"2024-03-11T21:34:00.727416+0000","last_deep_scrub":"94'6","last_deep_scrub_stamp":"2024-03-11T21:28:24.603974+0000","last_clean_scrub_stamp":"2024-03-11T21:34:00.727416+0000","objects_scrubbed":2,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:35:06.875929+0000","scrub_duration":20,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":6,"num_read_kb":4,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6],"acting":[3,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":361,"reported_epoch":235,"state":"active+clean","last_fresh":"2024-03-11T21:34:52.333716+0000","last_change":"2024-03-11T21:31:14.338908+0000","last_active":"2024-03-11T21:34:52.333716+0000","last_peered":"2024-03-11T21:34:52.333716+0000","last_clean":"2024-03-11T21:34:52.333716+0000","last_became_active":"2024-03-11T21:25:44.583181+0000","last_became_peered":"2024-03-11T21:25:44.583181+0000","last_unstale":"2024-03-11T21:34:52.333716+0000","last_undegraded":"2024-03-11T21:34:52.333716+0000","last_fullsized":"2024-03-11T21:34:52.333716+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:31:14.338640+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:43.559452+0000","last_clean_scrub_stamp":"2024-03-11T21:31:14.338640+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:34:55.849521+0000","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":374,"reported_epoch":235,"state":"active+clean","last_fresh":"2024-03-11T21:34:52.333315+0000","last_change":"2024-03-11T21:34:15.516387+0000","last_active":"2024-03-11T21:34:52.333315+0000","last_peered":"2024-03-11T21:34:52.333315+0000","last_clean":"2024-03-11T21:34:52.333315+0000","last_became_active":"2024-03-11T21:25:42.573158+0000","last_became_peered":"2024-03-11T21:25:42.573158+0000","last_unstale":"2024-03-11T21:34:52.333315+0000","last_undegraded":"2024-03-11T21:34:52.333315+0000","last_fullsized":"2024-03-11T21:34:52.333315+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:34:15.516278+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:27:21.207303+0000","last_clean_scrub_stamp":"2024-03-11T21:34:15.516278+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:35:36.619704+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.15","version":"0'0","reported_seq":382,"reported_epoch":235,"state":"active+clean","last_fresh":"2024-03-11T21:34:52.333464+0000","last_change":"2024-03-11T21:32:12.600491+0000","last_active":"2024-03-11T21:34:52.333464+0000","last_peered":"2024-03-11T21:34:52.333464+0000","last_clean":"2024-03-11T21:34:52.333464+0000","last_became_active":"2024-03-11T21:25:38.541705+0000","last_became_peered":"2024-03-11T21:25:38.541705+0000","last_unstale":"2024-03-11T21:34:52.333464+0000","last_undegraded":"2024-03-11T21:34:52.333464+0000","last_fullsized":"2024-03-11T21:34:52.333464+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:32:12.600388+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:26:44.860499+0000","last_clean_scrub_stamp":"2024-03-11T21:32:12.600388+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.13","version":"109'16","reported_seq":418,"reported_epoch":234,"state":"active+clean","last_fresh":"2024-03-11T21:34:51.293933+0000","last_change":"2024-03-11T21:32:24.657615+0000","last_active":"2024-03-11T21:34:51.293933+0000","last_peered":"2024-03-11T21:34:51.293933+0000","last_clean":"2024-03-11T21:34:51.293933+0000","last_became_active":"2024-03-11T21:26:11.214879+0000","last_became_peered":"2024-03-11T21:26:11.214879+0000","last_unstale":"2024-03-11T21:34:51.293933+0000","last_undegraded":"2024-03-11T21:34:51.293933+0000","last_fullsized":"2024-03-11T21:34:51.293933+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"109'16","last_scrub_stamp":"2024-03-11T21:32:24.657490+0000","last_deep_scrub":"109'16","last_deep_scrub_stamp":"2024-03-11T21:32:24.657490+0000","last_clean_scrub_stamp":"2024-03-11T21:32:24.657490+0000","objects_scrubbed":5,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,2],"acting":[4,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.10","version":"0'0","reported_seq":362,"reported_epoch":231,"state":"active+clean","last_fresh":"2024-03-11T21:34:48.223123+0000","last_change":"2024-03-11T21:32:56.582937+0000","last_active":"2024-03-11T21:34:48.223123+0000","last_peered":"2024-03-11T21:34:48.223123+0000","last_clean":"2024-03-11T21:34:48.223123+0000","last_became_active":"2024-03-11T21:25:44.591189+0000","last_became_peered":"2024-03-11T21:25:44.591189+0000","last_unstale":"2024-03-11T21:34:48.223123+0000","last_undegraded":"2024-03-11T21:34:48.223123+0000","last_fullsized":"2024-03-11T21:34:48.223123+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:32:56.582834+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:27:25.417450+0000","last_clean_scrub_stamp":"2024-03-11T21:32:56.582834+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":16,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,1],"acting":[0,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":365,"reported_epoch":235,"state":"active+clean","last_fresh":"2024-03-11T21:34:52.336067+0000","last_change":"2024-03-11T21:31:15.368570+0000","last_active":"2024-03-11T21:34:52.336067+0000","last_peered":"2024-03-11T21:34:52.336067+0000","last_clean":"2024-03-11T21:34:52.336067+0000","last_became_active":"2024-03-11T21:25:42.576425+0000","last_became_peered":"2024-03-11T21:25:42.576425+0000","last_unstale":"2024-03-11T21:34:52.336067+0000","last_undegraded":"2024-03-11T21:34:52.336067+0000","last_fullsized":"2024-03-11T21:34:52.336067+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:31:15.367898+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:31:15.367898+0000","last_clean_scrub_stamp":"2024-03-11T21:31:15.367898+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:34:55.849475+0000","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.14","version":"0'0","reported_seq":388,"reported_epoch":234,"state":"active+clean","last_fresh":"2024-03-11T21:34:51.298937+0000","last_change":"2024-03-11T21:32:22.917604+0000","last_active":"2024-03-11T21:34:51.298937+0000","last_peered":"2024-03-11T21:34:51.298937+0000","last_clean":"2024-03-11T21:34:51.298937+0000","last_became_active":"2024-03-11T21:25:38.541852+0000","last_became_peered":"2024-03-11T21:25:38.541852+0000","last_unstale":"2024-03-11T21:34:51.298937+0000","last_undegraded":"2024-03-11T21:34:51.298937+0000","last_fullsized":"2024-03-11T21:34:51.298937+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:32:22.917423+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:37.512018+0000","last_clean_scrub_stamp":"2024-03-11T21:32:22.917423+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:34:53.059875+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,5],"acting":[6,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.12","version":"74'9","reported_seq":394,"reported_epoch":235,"state":"active+clean","last_fresh":"2024-03-11T21:34:52.333128+0000","last_change":"2024-03-11T21:31:59.579237+0000","last_active":"2024-03-11T21:34:52.333128+0000","last_peered":"2024-03-11T21:34:52.333128+0000","last_clean":"2024-03-11T21:34:52.333128+0000","last_became_active":"2024-03-11T21:25:40.547082+0000","last_became_peered":"2024-03-11T21:25:40.547082+0000","last_unstale":"2024-03-11T21:34:52.333128+0000","last_undegraded":"2024-03-11T21:34:52.333128+0000","last_fullsized":"2024-03-11T21:34:52.333128+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'9","last_scrub_stamp":"2024-03-11T21:31:59.579142+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:39.525554+0000","last_clean_scrub_stamp":"2024-03-11T21:31:59.579142+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.19","version":"234'9144","reported_seq":15186,"reported_epoch":234,"state":"active+clean","last_fresh":"2024-03-11T21:34:51.712802+0000","last_change":"2024-03-11T21:33:58.958302+0000","last_active":"2024-03-11T21:34:51.712802+0000","last_peered":"2024-03-11T21:34:51.712802+0000","last_clean":"2024-03-11T21:34:51.712802+0000","last_became_active":"2024-03-11T21:33:08.320798+0000","last_became_peered":"2024-03-11T21:33:08.320798+0000","last_unstale":"2024-03-11T21:34:51.712802+0000","last_undegraded":"2024-03-11T21:34:51.712802+0000","last_fullsized":"2024-03-11T21:34:51.712802+0000","mapping_epoch":190,"log_start":"169'3118","ondisk_log_start":"169'3118","created":181,"last_epoch_clean":191,"parent":"0.0","parent_split_bits":5,"last_scrub":"211'7356","last_scrub_stamp":"2024-03-11T21:33:58.958068+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:31:51.219688+0000","last_clean_scrub_stamp":"2024-03-11T21:33:58.958068+0000","objects_scrubbed":2898,"log_size":6026,"log_dups_size":0,"ondisk_log_size":6026,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:35:19.300821+0000","scrub_duration":775,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":71958528,"num_objects":1098,"num_object_clones":0,"num_object_copies":3294,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1098,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1800,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,0],"acting":[5,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":368,"reported_epoch":234,"state":"active+clean","last_fresh":"2024-03-11T21:34:51.295002+0000","last_change":"2024-03-11T21:34:25.833239+0000","last_active":"2024-03-11T21:34:51.295002+0000","last_peered":"2024-03-11T21:34:51.295002+0000","last_clean":"2024-03-11T21:34:51.295002+0000","last_became_active":"2024-03-11T21:25:44.589586+0000","last_became_peered":"2024-03-11T21:25:44.589586+0000","last_unstale":"2024-03-11T21:34:51.295002+0000","last_undegraded":"2024-03-11T21:34:51.295002+0000","last_fullsized":"2024-03-11T21:34:51.295002+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:34:25.833149+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:27:09.801138+0000","last_clean_scrub_stamp":"2024-03-11T21:34:25.833149+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:35:26.798227+0000","scrub_duration":66,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,5],"acting":[4,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.14","version":"229'10","reported_seq":504,"reported_epoch":235,"state":"active+clean","last_fresh":"2024-03-11T21:34:52.333615+0000","last_change":"2024-03-11T21:34:41.181157+0000","last_active":"2024-03-11T21:34:52.333615+0000","last_peered":"2024-03-11T21:34:52.333615+0000","last_clean":"2024-03-11T21:34:52.333615+0000","last_became_active":"2024-03-11T21:34:41.151329+0000","last_became_peered":"2024-03-11T21:34:41.151329+0000","last_unstale":"2024-03-11T21:34:52.333615+0000","last_undegraded":"2024-03-11T21:34:52.333615+0000","last_fullsized":"2024-03-11T21:34:52.333615+0000","mapping_epoch":226,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":227,"parent":"0.0","parent_split_bits":0,"last_scrub":"154'9","last_scrub_stamp":"2024-03-11T21:32:03.467138+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:41.544550+0000","last_clean_scrub_stamp":"2024-03-11T21:32:03.467138+0000","objects_scrubbed":1,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:34:55.849615+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,2],"acting":[3,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.13","version":"0'0","reported_seq":374,"reported_epoch":231,"state":"active+clean","last_fresh":"2024-03-11T21:34:48.238613+0000","last_change":"2024-03-11T21:34:08.650456+0000","last_active":"2024-03-11T21:34:48.238613+0000","last_peered":"2024-03-11T21:34:48.238613+0000","last_clean":"2024-03-11T21:34:48.238613+0000","last_became_active":"2024-03-11T21:25:38.539467+0000","last_became_peered":"2024-03-11T21:25:38.539467+0000","last_unstale":"2024-03-11T21:34:48.238613+0000","last_undegraded":"2024-03-11T21:34:48.238613+0000","last_fullsized":"2024-03-11T21:34:48.238613+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:34:08.650371+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:37.512018+0000","last_clean_scrub_stamp":"2024-03-11T21:34:08.650371+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:35:32.975966+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.15","version":"94'13","reported_seq":427,"reported_epoch":234,"state":"active+clean","last_fresh":"2024-03-11T21:34:51.300663+0000","last_change":"2024-03-11T21:34:41.173208+0000","last_active":"2024-03-11T21:34:51.300663+0000","last_peered":"2024-03-11T21:34:51.300663+0000","last_clean":"2024-03-11T21:34:51.300663+0000","last_became_active":"2024-03-11T21:34:41.147412+0000","last_became_peered":"2024-03-11T21:34:41.147412+0000","last_unstale":"2024-03-11T21:34:51.300663+0000","last_undegraded":"2024-03-11T21:34:51.300663+0000","last_fullsized":"2024-03-11T21:34:51.300663+0000","mapping_epoch":226,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":227,"parent":"0.0","parent_split_bits":0,"last_scrub":"94'13","last_scrub_stamp":"2024-03-11T21:32:10.610489+0000","last_deep_scrub":"94'13","last_deep_scrub_stamp":"2024-03-11T21:29:41.853158+0000","last_clean_scrub_stamp":"2024-03-11T21:32:10.610489+0000","objects_scrubbed":4,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.18","version":"234'9294","reported_seq":16073,"reported_epoch":234,"state":"active+clean","last_fresh":"2024-03-11T21:34:51.754887+0000","last_change":"2024-03-11T21:33:59.340712+0000","last_active":"2024-03-11T21:34:51.754887+0000","last_peered":"2024-03-11T21:34:51.754887+0000","last_clean":"2024-03-11T21:34:51.754887+0000","last_became_active":"2024-03-11T21:33:30.363159+0000","last_became_peered":"2024-03-11T21:33:30.363159+0000","last_unstale":"2024-03-11T21:34:51.754887+0000","last_undegraded":"2024-03-11T21:34:51.754887+0000","last_fullsized":"2024-03-11T21:34:51.754887+0000","mapping_epoch":196,"log_start":"170'3829","ondisk_log_start":"170'3829","created":181,"last_epoch_clean":197,"parent":"0.0","parent_split_bits":5,"last_scrub":"211'7505","last_scrub_stamp":"2024-03-11T21:33:59.340450+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:31:51.219688+0000","last_clean_scrub_stamp":"2024-03-11T21:33:59.340450+0000","objects_scrubbed":2900,"log_size":5465,"log_dups_size":0,"ondisk_log_size":5465,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:35:29.223040+0000","scrub_duration":648,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":72286208,"num_objects":1103,"num_object_clones":0,"num_object_copies":3309,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1103,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1797,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,5],"acting":[4,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.16","version":"0'0","reported_seq":374,"reported_epoch":231,"state":"active+clean","last_fresh":"2024-03-11T21:34:48.243065+0000","last_change":"2024-03-11T21:34:41.149107+0000","last_active":"2024-03-11T21:34:48.243065+0000","last_peered":"2024-03-11T21:34:48.243065+0000","last_clean":"2024-03-11T21:34:48.243065+0000","last_became_active":"2024-03-11T21:34:41.148012+0000","last_became_peered":"2024-03-11T21:34:41.148012+0000","last_unstale":"2024-03-11T21:34:48.243065+0000","last_undegraded":"2024-03-11T21:34:48.243065+0000","last_fullsized":"2024-03-11T21:34:48.243065+0000","mapping_epoch":226,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":227,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:32:23.607460+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:43.559452+0000","last_clean_scrub_stamp":"2024-03-11T21:32:23.607460+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":35,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.15","version":"98'9","reported_seq":491,"reported_epoch":234,"state":"active+clean","last_fresh":"2024-03-11T21:34:51.300820+0000","last_change":"2024-03-11T21:34:30.363954+0000","last_active":"2024-03-11T21:34:51.300820+0000","last_peered":"2024-03-11T21:34:51.300820+0000","last_clean":"2024-03-11T21:34:51.300820+0000","last_became_active":"2024-03-11T21:25:42.572555+0000","last_became_peered":"2024-03-11T21:25:42.572555+0000","last_unstale":"2024-03-11T21:34:51.300820+0000","last_undegraded":"2024-03-11T21:34:51.300820+0000","last_fullsized":"2024-03-11T21:34:51.300820+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"98'9","last_scrub_stamp":"2024-03-11T21:34:30.363823+0000","last_deep_scrub":"98'9","last_deep_scrub_stamp":"2024-03-11T21:31:20.624923+0000","last_clean_scrub_stamp":"2024-03-11T21:34:30.363823+0000","objects_scrubbed":1,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:35:44.164104+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.12","version":"0'0","reported_seq":391,"reported_epoch":234,"state":"active+clean","last_fresh":"2024-03-11T21:34:51.301217+0000","last_change":"2024-03-11T21:34:41.136310+0000","last_active":"2024-03-11T21:34:51.301217+0000","last_peered":"2024-03-11T21:34:51.301217+0000","last_clean":"2024-03-11T21:34:51.301217+0000","last_became_active":"2024-03-11T21:34:41.134090+0000","last_became_peered":"2024-03-11T21:34:41.134090+0000","last_unstale":"2024-03-11T21:34:51.301217+0000","last_undegraded":"2024-03-11T21:34:51.301217+0000","last_fullsized":"2024-03-11T21:34:51.301217+0000","mapping_epoch":226,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":227,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:32:14.555187+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:37.512018+0000","last_clean_scrub_stamp":"2024-03-11T21:32:14.555187+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.14","version":"74'10","reported_seq":420,"reported_epoch":235,"state":"active+clean","last_fresh":"2024-03-11T21:34:52.333436+0000","last_change":"2024-03-11T21:34:41.178979+0000","last_active":"2024-03-11T21:34:52.333436+0000","last_peered":"2024-03-11T21:34:52.333436+0000","last_clean":"2024-03-11T21:34:52.333436+0000","last_became_active":"2024-03-11T21:34:41.145114+0000","last_became_peered":"2024-03-11T21:34:41.145114+0000","last_unstale":"2024-03-11T21:34:52.333436+0000","last_undegraded":"2024-03-11T21:34:52.333436+0000","last_fullsized":"2024-03-11T21:34:52.333436+0000","mapping_epoch":226,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":227,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'10","last_scrub_stamp":"2024-03-11T21:32:13.490198+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:39.525554+0000","last_clean_scrub_stamp":"2024-03-11T21:32:13.490198+0000","objects_scrubbed":5,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:34:55.849633+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.15","version":"0'0","reported_seq":25,"reported_epoch":234,"state":"active+clean","last_fresh":"2024-03-11T21:34:51.296759+0000","last_change":"2024-03-11T21:34:41.151416+0000","last_active":"2024-03-11T21:34:51.296759+0000","last_peered":"2024-03-11T21:34:51.296759+0000","last_clean":"2024-03-11T21:34:51.296759+0000","last_became_active":"2024-03-11T21:34:41.142760+0000","last_became_peered":"2024-03-11T21:34:41.142760+0000","last_unstale":"2024-03-11T21:34:51.296759+0000","last_undegraded":"2024-03-11T21:34:51.296759+0000","last_fullsized":"2024-03-11T21:34:51.296759+0000","mapping_epoch":226,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":227,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:34:22.372241+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:43.559452+0000","last_clean_scrub_stamp":"2024-03-11T21:34:22.372241+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:35:41.662719+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":371,"reported_epoch":234,"state":"active+clean","last_fresh":"2024-03-11T21:34:51.301199+0000","last_change":"2024-03-11T21:34:09.244924+0000","last_active":"2024-03-11T21:34:51.301199+0000","last_peered":"2024-03-11T21:34:51.301199+0000","last_clean":"2024-03-11T21:34:51.301199+0000","last_became_active":"2024-03-11T21:25:42.577351+0000","last_became_peered":"2024-03-11T21:25:42.577351+0000","last_unstale":"2024-03-11T21:34:51.301199+0000","last_undegraded":"2024-03-11T21:34:51.301199+0000","last_fullsized":"2024-03-11T21:34:51.301199+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:34:09.244852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:41.544550+0000","last_clean_scrub_stamp":"2024-03-11T21:34:09.244852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:35:11.770192+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,1],"acting":[5,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.11","version":"0'0","reported_seq":380,"reported_epoch":234,"state":"active+clean","last_fresh":"2024-03-11T21:34:51.299052+0000","last_change":"2024-03-11T21:32:09.785387+0000","last_active":"2024-03-11T21:34:51.299052+0000","last_peered":"2024-03-11T21:34:51.299052+0000","last_clean":"2024-03-11T21:34:51.299052+0000","last_became_active":"2024-03-11T21:25:38.537415+0000","last_became_peered":"2024-03-11T21:25:38.537415+0000","last_unstale":"2024-03-11T21:34:51.299052+0000","last_undegraded":"2024-03-11T21:34:51.299052+0000","last_fullsized":"2024-03-11T21:34:51.299052+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:32:09.785314+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:37.512018+0000","last_clean_scrub_stamp":"2024-03-11T21:32:09.785314+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:34:53.059858+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.17","version":"97'9","reported_seq":423,"reported_epoch":235,"state":"active+clean","last_fresh":"2024-03-11T21:34:52.334743+0000","last_change":"2024-03-11T21:34:41.173645+0000","last_active":"2024-03-11T21:34:52.334743+0000","last_peered":"2024-03-11T21:34:52.334743+0000","last_clean":"2024-03-11T21:34:52.334743+0000","last_became_active":"2024-03-11T21:34:41.146177+0000","last_became_peered":"2024-03-11T21:34:41.146177+0000","last_unstale":"2024-03-11T21:34:52.334743+0000","last_undegraded":"2024-03-11T21:34:52.334743+0000","last_fullsized":"2024-03-11T21:34:52.334743+0000","mapping_epoch":226,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":227,"parent":"0.0","parent_split_bits":0,"last_scrub":"97'9","last_scrub_stamp":"2024-03-11T21:32:08.495352+0000","last_deep_scrub":"97'9","last_deep_scrub_stamp":"2024-03-11T21:27:25.803132+0000","last_clean_scrub_stamp":"2024-03-11T21:32:08.495352+0000","objects_scrubbed":3,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:34:55.849743+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":9,"num_read_kb":6,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,5],"acting":[3,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.14","version":"0'0","reported_seq":381,"reported_epoch":235,"state":"active+clean","last_fresh":"2024-03-11T21:34:52.295627+0000","last_change":"2024-03-11T21:34:41.137559+0000","last_active":"2024-03-11T21:34:52.295627+0000","last_peered":"2024-03-11T21:34:52.295627+0000","last_clean":"2024-03-11T21:34:52.295627+0000","last_became_active":"2024-03-11T21:34:41.136946+0000","last_became_peered":"2024-03-11T21:34:41.136946+0000","last_unstale":"2024-03-11T21:34:52.295627+0000","last_undegraded":"2024-03-11T21:34:52.295627+0000","last_fullsized":"2024-03-11T21:34:52.295627+0000","mapping_epoch":226,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":227,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:34:15.361586+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:43.559452+0000","last_clean_scrub_stamp":"2024-03-11T21:34:15.361586+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:35:23.055559+0000","scrub_duration":27,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,7],"acting":[2,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":385,"reported_epoch":235,"state":"active+clean","last_fresh":"2024-03-11T21:34:52.334828+0000","last_change":"2024-03-11T21:34:41.146564+0000","last_active":"2024-03-11T21:34:52.334828+0000","last_peered":"2024-03-11T21:34:52.334828+0000","last_clean":"2024-03-11T21:34:52.334828+0000","last_became_active":"2024-03-11T21:34:41.145400+0000","last_became_peered":"2024-03-11T21:34:41.145400+0000","last_unstale":"2024-03-11T21:34:52.334828+0000","last_undegraded":"2024-03-11T21:34:52.334828+0000","last_fullsized":"2024-03-11T21:34:52.334828+0000","mapping_epoch":226,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":227,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:34:33.082577+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:41.544550+0000","last_clean_scrub_stamp":"2024-03-11T21:34:33.082577+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:36:02.359492+0000","scrub_duration":35,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.10","version":"0'0","reported_seq":381,"reported_epoch":235,"state":"active+clean","last_fresh":"2024-03-11T21:34:52.295136+0000","last_change":"2024-03-11T21:34:12.365368+0000","last_active":"2024-03-11T21:34:52.295136+0000","last_peered":"2024-03-11T21:34:52.295136+0000","last_clean":"2024-03-11T21:34:52.295136+0000","last_became_active":"2024-03-11T21:25:38.533138+0000","last_became_peered":"2024-03-11T21:25:38.533138+0000","last_unstale":"2024-03-11T21:34:52.295136+0000","last_undegraded":"2024-03-11T21:34:52.295136+0000","last_fullsized":"2024-03-11T21:34:52.295136+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:34:12.365244+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:37.512018+0000","last_clean_scrub_stamp":"2024-03-11T21:34:12.365244+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:35:36.676182+0000","scrub_duration":17,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,0],"acting":[2,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.16","version":"74'9","reported_seq":418,"reported_epoch":231,"state":"active+clean","last_fresh":"2024-03-11T21:34:48.243237+0000","last_change":"2024-03-11T21:34:41.178232+0000","last_active":"2024-03-11T21:34:48.243237+0000","last_peered":"2024-03-11T21:34:48.243237+0000","last_clean":"2024-03-11T21:34:48.243237+0000","last_became_active":"2024-03-11T21:34:41.148571+0000","last_became_peered":"2024-03-11T21:34:41.148571+0000","last_unstale":"2024-03-11T21:34:48.243237+0000","last_undegraded":"2024-03-11T21:34:48.243237+0000","last_fullsized":"2024-03-11T21:34:48.243237+0000","mapping_epoch":226,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":227,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'9","last_scrub_stamp":"2024-03-11T21:32:14.414508+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:39.525554+0000","last_clean_scrub_stamp":"2024-03-11T21:32:14.414508+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.5","version":"234'9207","reported_seq":9424,"reported_epoch":234,"state":"active+remapped+backfill_toofull","last_fresh":"2024-03-11T21:34:51.648745+0000","last_change":"2024-03-11T21:34:42.731203+0000","last_active":"2024-03-11T21:34:51.648745+0000","last_peered":"2024-03-11T21:34:51.648745+0000","last_clean":"2024-03-11T21:34:40.105655+0000","last_became_active":"2024-03-11T21:34:42.690454+0000","last_became_peered":"2024-03-11T21:34:42.690454+0000","last_unstale":"2024-03-11T21:34:51.648745+0000","last_undegraded":"2024-03-11T21:34:51.648745+0000","last_fullsized":"2024-03-11T21:34:51.648745+0000","mapping_epoch":227,"log_start":"169'3088","ondisk_log_start":"169'3088","created":157,"last_epoch_clean":182,"parent":"0.0","parent_split_bits":0,"last_scrub":"185'5575","last_scrub_stamp":"2024-03-11T21:32:59.559295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:31:51.219688+0000","last_clean_scrub_stamp":"2024-03-11T21:32:59.559295+0000","objects_scrubbed":2833,"log_size":6119,"log_dups_size":0,"ondisk_log_size":6119,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"no scrub is scheduled","scrub_duration":572,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":75563008,"num_objects":1153,"num_object_clones":0,"num_object_copies":3459,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1153,"num_objects_unfound":0,"num_objects_dirty":1153,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3636,"num_write_kb":62592,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,0],"avail_no_missing":["6","0","4"],"object_location_counts":[{"shards":"0,4,6","objects":1153}],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.b","version":"0'0","reported_seq":374,"reported_epoch":235,"state":"active+clean","last_fresh":"2024-03-11T21:34:52.336151+0000","last_change":"2024-03-11T21:34:41.149950+0000","last_active":"2024-03-11T21:34:52.336151+0000","last_peered":"2024-03-11T21:34:52.336151+0000","last_clean":"2024-03-11T21:34:52.336151+0000","last_became_active":"2024-03-11T21:34:41.147208+0000","last_became_peered":"2024-03-11T21:34:41.147208+0000","last_unstale":"2024-03-11T21:34:52.336151+0000","last_undegraded":"2024-03-11T21:34:52.336151+0000","last_fullsized":"2024-03-11T21:34:52.336151+0000","mapping_epoch":226,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":227,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:32:22.598150+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:43.559452+0000","last_clean_scrub_stamp":"2024-03-11T21:32:22.598150+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:34:55.849758+0000","scrub_duration":57,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.f","version":"67'2","reported_seq":393,"reported_epoch":234,"state":"active+clean","last_fresh":"2024-03-11T21:34:51.294872+0000","last_change":"2024-03-11T21:34:08.776614+0000","last_active":"2024-03-11T21:34:51.294872+0000","last_peered":"2024-03-11T21:34:51.294872+0000","last_clean":"2024-03-11T21:34:51.294872+0000","last_became_active":"2024-03-11T21:25:38.537711+0000","last_became_peered":"2024-03-11T21:25:38.537711+0000","last_unstale":"2024-03-11T21:34:51.294872+0000","last_undegraded":"2024-03-11T21:34:51.294872+0000","last_fullsized":"2024-03-11T21:34:51.294872+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'2","last_scrub_stamp":"2024-03-11T21:34:08.776510+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:37.512018+0000","last_clean_scrub_stamp":"2024-03-11T21:34:08.776510+0000","objects_scrubbed":2,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:35:18.719734+0000","scrub_duration":46,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":92,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":10,"num_read_kb":10,"num_write":4,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,7],"acting":[4,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.9","version":"146'17","reported_seq":405,"reported_epoch":234,"state":"active+clean","last_fresh":"2024-03-11T21:34:51.293911+0000","last_change":"2024-03-11T21:34:26.765180+0000","last_active":"2024-03-11T21:34:51.293911+0000","last_peered":"2024-03-11T21:34:51.293911+0000","last_clean":"2024-03-11T21:34:51.293911+0000","last_became_active":"2024-03-11T21:25:40.552267+0000","last_became_peered":"2024-03-11T21:25:40.552267+0000","last_unstale":"2024-03-11T21:34:51.293911+0000","last_undegraded":"2024-03-11T21:34:51.293911+0000","last_fullsized":"2024-03-11T21:34:51.293911+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"146'17","last_scrub_stamp":"2024-03-11T21:34:26.765075+0000","last_deep_scrub":"146'17","last_deep_scrub_stamp":"2024-03-11T21:34:26.765075+0000","last_clean_scrub_stamp":"2024-03-11T21:34:26.765075+0000","objects_scrubbed":5,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:35:27.141569+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":25,"num_read_kb":16,"num_write":14,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,3],"acting":[4,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":374,"reported_epoch":235,"state":"active+clean","last_fresh":"2024-03-11T21:34:52.295042+0000","last_change":"2024-03-11T21:32:55.362571+0000","last_active":"2024-03-11T21:34:52.295042+0000","last_peered":"2024-03-11T21:34:52.295042+0000","last_clean":"2024-03-11T21:34:52.295042+0000","last_became_active":"2024-03-11T21:25:42.573987+0000","last_became_peered":"2024-03-11T21:25:42.573987+0000","last_unstale":"2024-03-11T21:34:52.295042+0000","last_undegraded":"2024-03-11T21:34:52.295042+0000","last_fullsized":"2024-03-11T21:34:52.295042+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:32:55.362458+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:41.544550+0000","last_clean_scrub_stamp":"2024-03-11T21:32:55.362458+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:34:52.399573+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"8.e","version":"231'12777","reported_seq":12997,"reported_epoch":231,"state":"active+clean","last_fresh":"2024-03-11T21:34:48.270191+0000","last_change":"2024-03-11T21:34:15.604651+0000","last_active":"2024-03-11T21:34:48.270191+0000","last_peered":"2024-03-11T21:34:48.270191+0000","last_clean":"2024-03-11T21:34:48.270191+0000","last_became_active":"2024-03-11T21:31:52.272382+0000","last_became_peered":"2024-03-11T21:31:52.272382+0000","last_unstale":"2024-03-11T21:34:48.270191+0000","last_undegraded":"2024-03-11T21:34:48.270191+0000","last_fullsized":"2024-03-11T21:34:48.270191+0000","mapping_epoch":157,"log_start":"202'8259","ondisk_log_start":"202'8259","created":157,"last_epoch_clean":158,"parent":"0.0","parent_split_bits":0,"last_scrub":"219'10491","last_scrub_stamp":"2024-03-11T21:34:15.604036+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:31:51.219688+0000","last_clean_scrub_stamp":"2024-03-11T21:34:15.604036+0000","objects_scrubbed":4819,"log_size":4518,"log_dups_size":0,"ondisk_log_size":4518,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:35:40.194213+0000","scrub_duration":1929,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":161415168,"num_objects":2463,"num_object_clones":0,"num_object_copies":7389,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2463,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12777,"num_write_kb":487680,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,4],"acting":[0,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.0","version":"0'0","reported_seq":355,"reported_epoch":231,"state":"active+clean","last_fresh":"2024-03-11T21:34:48.223213+0000","last_change":"2024-03-11T21:32:03.422921+0000","last_active":"2024-03-11T21:34:48.223213+0000","last_peered":"2024-03-11T21:34:48.223213+0000","last_clean":"2024-03-11T21:34:48.223213+0000","last_became_active":"2024-03-11T21:25:44.584565+0000","last_became_peered":"2024-03-11T21:25:44.584565+0000","last_unstale":"2024-03-11T21:34:48.223213+0000","last_undegraded":"2024-03-11T21:34:48.223213+0000","last_fullsized":"2024-03-11T21:34:48.223213+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:32:03.422808+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:29:39.498784+0000","last_clean_scrub_stamp":"2024-03-11T21:32:03.422808+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":31,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":395,"reported_epoch":235,"state":"active+clean","last_fresh":"2024-03-11T21:34:52.333058+0000","last_change":"2024-03-11T21:34:41.151254+0000","last_active":"2024-03-11T21:34:52.333058+0000","last_peered":"2024-03-11T21:34:52.333058+0000","last_clean":"2024-03-11T21:34:52.333058+0000","last_became_active":"2024-03-11T21:34:41.150913+0000","last_became_peered":"2024-03-11T21:34:41.150913+0000","last_unstale":"2024-03-11T21:34:52.333058+0000","last_undegraded":"2024-03-11T21:34:52.333058+0000","last_fullsized":"2024-03-11T21:34:52.333058+0000","mapping_epoch":226,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":227,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:32:04.509514+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:37.512018+0000","last_clean_scrub_stamp":"2024-03-11T21:32:04.509514+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,7],"acting":[1,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.2","version":"94'15","reported_seq":399,"reported_epoch":235,"state":"active+clean","last_fresh":"2024-03-11T21:34:52.333223+0000","last_change":"2024-03-11T21:31:08.956264+0000","last_active":"2024-03-11T21:34:52.333223+0000","last_peered":"2024-03-11T21:34:52.333223+0000","last_clean":"2024-03-11T21:34:52.333223+0000","last_became_active":"2024-03-11T21:25:40.548833+0000","last_became_peered":"2024-03-11T21:25:40.548833+0000","last_unstale":"2024-03-11T21:34:52.333223+0000","last_undegraded":"2024-03-11T21:34:52.333223+0000","last_fullsized":"2024-03-11T21:34:52.333223+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"94'15","last_scrub_stamp":"2024-03-11T21:31:08.955971+0000","last_deep_scrub":"94'15","last_deep_scrub_stamp":"2024-03-11T21:26:49.993060+0000","last_clean_scrub_stamp":"2024-03-11T21:31:08.955971+0000","objects_scrubbed":5,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.3","version":"74'8","reported_seq":485,"reported_epoch":231,"state":"active+clean","last_fresh":"2024-03-11T21:34:48.239204+0000","last_change":"2024-03-11T21:34:11.753926+0000","last_active":"2024-03-11T21:34:48.239204+0000","last_peered":"2024-03-11T21:34:48.239204+0000","last_clean":"2024-03-11T21:34:48.239204+0000","last_became_active":"2024-03-11T21:25:42.578161+0000","last_became_peered":"2024-03-11T21:25:42.578161+0000","last_unstale":"2024-03-11T21:34:48.239204+0000","last_undegraded":"2024-03-11T21:34:48.239204+0000","last_fullsized":"2024-03-11T21:34:48.239204+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'8","last_scrub_stamp":"2024-03-11T21:34:11.753809+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:41.544550+0000","last_clean_scrub_stamp":"2024-03-11T21:34:11.753809+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:35:12.615444+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,5],"acting":[0,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.8","version":"234'9335","reported_seq":11162,"reported_epoch":234,"state":"active+remapped+backfilling","last_fresh":"2024-03-11T21:34:52.115058+0000","last_change":"2024-03-11T21:34:49.974121+0000","last_active":"2024-03-11T21:34:52.115058+0000","last_peered":"2024-03-11T21:34:52.115058+0000","last_clean":"2024-03-11T21:34:40.062677+0000","last_became_active":"2024-03-11T21:34:42.688468+0000","last_became_peered":"2024-03-11T21:34:42.688468+0000","last_unstale":"2024-03-11T21:34:52.115058+0000","last_undegraded":"2024-03-11T21:34:52.115058+0000","last_fullsized":"2024-03-11T21:34:52.115058+0000","mapping_epoch":227,"log_start":"170'3530","ondisk_log_start":"170'3530","created":157,"last_epoch_clean":182,"parent":"0.0","parent_split_bits":0,"last_scrub":"183'5663","last_scrub_stamp":"2024-03-11T21:32:58.596358+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:31:51.219688+0000","last_clean_scrub_stamp":"2024-03-11T21:32:58.596358+0000","objects_scrubbed":2875,"log_size":5805,"log_dups_size":0,"ondisk_log_size":5805,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"no scrub is scheduled","scrub_duration":327,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":74514432,"num_objects":1137,"num_object_clones":0,"num_object_copies":3411,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":342,"num_objects_unfound":0,"num_objects_dirty":1137,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3672,"num_write_kb":61888,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":805,"num_bytes_recovered":52756480,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,0],"avail_no_missing":["5","0","3"],"object_location_counts":[{"shards":"0,3,5","objects":1137}],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":374,"reported_epoch":235,"state":"active+clean","last_fresh":"2024-03-11T21:34:52.334596+0000","last_change":"2024-03-11T21:34:41.146703+0000","last_active":"2024-03-11T21:34:52.334596+0000","last_peered":"2024-03-11T21:34:52.334596+0000","last_clean":"2024-03-11T21:34:52.334596+0000","last_became_active":"2024-03-11T21:34:41.146073+0000","last_became_peered":"2024-03-11T21:34:41.146073+0000","last_unstale":"2024-03-11T21:34:52.334596+0000","last_undegraded":"2024-03-11T21:34:52.334596+0000","last_fullsized":"2024-03-11T21:34:52.334596+0000","mapping_epoch":226,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":227,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:31:01.241180+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:28:30.563712+0000","last_clean_scrub_stamp":"2024-03-11T21:31:01.241180+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:34:55.849448+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,7],"acting":[3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":388,"reported_epoch":234,"state":"active+clean","last_fresh":"2024-03-11T21:34:51.301289+0000","last_change":"2024-03-11T21:34:18.388334+0000","last_active":"2024-03-11T21:34:51.301289+0000","last_peered":"2024-03-11T21:34:51.301289+0000","last_clean":"2024-03-11T21:34:51.301289+0000","last_became_active":"2024-03-11T21:25:38.538540+0000","last_became_peered":"2024-03-11T21:25:38.538540+0000","last_unstale":"2024-03-11T21:34:51.301289+0000","last_undegraded":"2024-03-11T21:34:51.301289+0000","last_fullsized":"2024-03-11T21:34:51.301289+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:34:18.388258+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:37.512018+0000","last_clean_scrub_stamp":"2024-03-11T21:34:18.388258+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:35:35.043384+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.4","version":"74'28","reported_seq":538,"reported_epoch":235,"state":"active+clean","last_fresh":"2024-03-11T21:34:52.332862+0000","last_change":"2024-03-11T21:34:19.513419+0000","last_active":"2024-03-11T21:34:52.332862+0000","last_peered":"2024-03-11T21:34:52.332862+0000","last_clean":"2024-03-11T21:34:52.332862+0000","last_became_active":"2024-03-11T21:25:40.552902+0000","last_became_peered":"2024-03-11T21:25:40.552902+0000","last_unstale":"2024-03-11T21:34:52.332862+0000","last_undegraded":"2024-03-11T21:34:52.332862+0000","last_fullsized":"2024-03-11T21:34:52.332862+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'28","last_scrub_stamp":"2024-03-11T21:34:19.513252+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:39.525554+0000","last_clean_scrub_stamp":"2024-03-11T21:34:19.513252+0000","objects_scrubbed":10,"log_size":28,"log_dups_size":0,"ondisk_log_size":28,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:35:34.875894+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":358,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":48,"num_read_kb":33,"num_write":26,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3],"acting":[1,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":367,"reported_epoch":231,"state":"active+clean","last_fresh":"2024-03-11T21:34:48.241740+0000","last_change":"2024-03-11T21:34:18.638688+0000","last_active":"2024-03-11T21:34:48.241740+0000","last_peered":"2024-03-11T21:34:48.241740+0000","last_clean":"2024-03-11T21:34:48.241740+0000","last_became_active":"2024-03-11T21:25:42.572673+0000","last_became_peered":"2024-03-11T21:25:42.572673+0000","last_unstale":"2024-03-11T21:34:48.241740+0000","last_undegraded":"2024-03-11T21:34:48.241740+0000","last_fullsized":"2024-03-11T21:34:48.241740+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:34:18.638566+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:41.544550+0000","last_clean_scrub_stamp":"2024-03-11T21:34:18.638566+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:35:21.043366+0000","scrub_duration":18,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.b","version":"235'13025","reported_seq":13196,"reported_epoch":235,"state":"active+clean","last_fresh":"2024-03-11T21:34:52.697061+0000","last_change":"2024-03-11T21:31:52.241413+0000","last_active":"2024-03-11T21:34:52.697061+0000","last_peered":"2024-03-11T21:34:52.697061+0000","last_clean":"2024-03-11T21:34:52.697061+0000","last_became_active":"2024-03-11T21:31:52.241089+0000","last_became_peered":"2024-03-11T21:31:52.241089+0000","last_unstale":"2024-03-11T21:34:52.697061+0000","last_undegraded":"2024-03-11T21:34:52.697061+0000","last_fullsized":"2024-03-11T21:34:52.697061+0000","mapping_epoch":157,"log_start":"202'8192","ondisk_log_start":"202'8192","created":157,"last_epoch_clean":158,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:31:51.219688+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:31:51.219688+0000","last_clean_scrub_stamp":"2024-03-11T21:31:51.219688+0000","objects_scrubbed":0,"log_size":4833,"log_dups_size":0,"ondisk_log_size":4833,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:34:55.849597+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":152895488,"num_objects":2333,"num_object_clones":0,"num_object_copies":6999,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2333,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13025,"num_write_kb":491456,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,1],"acting":[3,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.5","version":"0'0","reported_seq":25,"reported_epoch":234,"state":"active+clean","last_fresh":"2024-03-11T21:34:51.296668+0000","last_change":"2024-03-11T21:34:41.153269+0000","last_active":"2024-03-11T21:34:51.296668+0000","last_peered":"2024-03-11T21:34:51.296668+0000","last_clean":"2024-03-11T21:34:51.296668+0000","last_became_active":"2024-03-11T21:34:41.151087+0000","last_became_peered":"2024-03-11T21:34:41.151087+0000","last_unstale":"2024-03-11T21:34:51.296668+0000","last_undegraded":"2024-03-11T21:34:51.296668+0000","last_fullsized":"2024-03-11T21:34:51.296668+0000","mapping_epoch":226,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":227,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:32:10.837701+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:43.559452+0000","last_clean_scrub_stamp":"2024-03-11T21:32:10.837701+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,3],"acting":[7,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.0","version":"164'6","reported_seq":941,"reported_epoch":235,"state":"active+clean","last_fresh":"2024-03-11T21:34:52.335035+0000","last_change":"2024-03-11T21:32:17.597798+0000","last_active":"2024-03-11T21:34:52.335035+0000","last_peered":"2024-03-11T21:34:52.335035+0000","last_clean":"2024-03-11T21:34:52.335035+0000","last_became_active":"2024-03-11T21:25:38.536890+0000","last_became_peered":"2024-03-11T21:25:38.536890+0000","last_unstale":"2024-03-11T21:34:52.335035+0000","last_undegraded":"2024-03-11T21:34:52.335035+0000","last_fullsized":"2024-03-11T21:34:52.335035+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"164'6","last_scrub_stamp":"2024-03-11T21:32:17.597666+0000","last_deep_scrub":"164'6","last_deep_scrub_stamp":"2024-03-11T21:32:17.597666+0000","last_clean_scrub_stamp":"2024-03-11T21:32:17.597666+0000","objects_scrubbed":1,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0.000753559,"stat_sum":{"num_bytes":389,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":547,"num_read_kb":542,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,6],"acting":[1,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":390,"reported_epoch":235,"state":"active+clean","last_fresh":"2024-03-11T21:34:52.295300+0000","last_change":"2024-03-11T21:34:18.424114+0000","last_active":"2024-03-11T21:34:52.295300+0000","last_peered":"2024-03-11T21:34:52.295300+0000","last_clean":"2024-03-11T21:34:52.295300+0000","last_became_active":"2024-03-11T21:25:38.532984+0000","last_became_peered":"2024-03-11T21:25:38.532984+0000","last_unstale":"2024-03-11T21:34:52.295300+0000","last_undegraded":"2024-03-11T21:34:52.295300+0000","last_fullsized":"2024-03-11T21:34:52.295300+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:34:18.424010+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:26:48.997454+0000","last_clean_scrub_stamp":"2024-03-11T21:34:18.424010+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:35:44.820528+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.7","version":"97'18","reported_seq":414,"reported_epoch":235,"state":"active+clean","last_fresh":"2024-03-11T21:34:52.334233+0000","last_change":"2024-03-11T21:32:14.594198+0000","last_active":"2024-03-11T21:34:52.334233+0000","last_peered":"2024-03-11T21:34:52.334233+0000","last_clean":"2024-03-11T21:34:52.334233+0000","last_became_active":"2024-03-11T21:25:40.541769+0000","last_became_peered":"2024-03-11T21:25:40.541769+0000","last_unstale":"2024-03-11T21:34:52.334233+0000","last_undegraded":"2024-03-11T21:34:52.334233+0000","last_fullsized":"2024-03-11T21:34:52.334233+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"97'18","last_scrub_stamp":"2024-03-11T21:32:14.593767+0000","last_deep_scrub":"97'18","last_deep_scrub_stamp":"2024-03-11T21:27:19.239404+0000","last_clean_scrub_stamp":"2024-03-11T21:32:14.593767+0000","objects_scrubbed":5,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":18,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":19,"num_write":16,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":386,"reported_epoch":235,"state":"active+clean","last_fresh":"2024-03-11T21:34:52.295884+0000","last_change":"2024-03-11T21:34:41.144253+0000","last_active":"2024-03-11T21:34:52.295884+0000","last_peered":"2024-03-11T21:34:52.295884+0000","last_clean":"2024-03-11T21:34:52.295884+0000","last_became_active":"2024-03-11T21:34:41.143660+0000","last_became_peered":"2024-03-11T21:34:41.143660+0000","last_unstale":"2024-03-11T21:34:52.295884+0000","last_undegraded":"2024-03-11T21:34:52.295884+0000","last_fullsized":"2024-03-11T21:34:52.295884+0000","mapping_epoch":226,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":227,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:34:07.290464+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:41.544550+0000","last_clean_scrub_stamp":"2024-03-11T21:34:07.290464+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:35:15.129200+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7],"acting":[2,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"8.9","version":"234'9343","reported_seq":9553,"reported_epoch":234,"state":"active+clean","last_fresh":"2024-03-11T21:34:51.837552+0000","last_change":"2024-03-11T21:32:58.396514+0000","last_active":"2024-03-11T21:34:51.837552+0000","last_peered":"2024-03-11T21:34:51.837552+0000","last_clean":"2024-03-11T21:34:51.837552+0000","last_became_active":"2024-03-11T21:32:57.085769+0000","last_became_peered":"2024-03-11T21:32:57.085769+0000","last_unstale":"2024-03-11T21:34:51.837552+0000","last_undegraded":"2024-03-11T21:34:51.837552+0000","last_fullsized":"2024-03-11T21:34:51.837552+0000","mapping_epoch":157,"log_start":"169'3290","ondisk_log_start":"169'3290","created":157,"last_epoch_clean":182,"parent":"0.0","parent_split_bits":0,"last_scrub":"182'5644","last_scrub_stamp":"2024-03-11T21:32:58.396310+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:31:51.219688+0000","last_clean_scrub_stamp":"2024-03-11T21:32:58.396310+0000","objects_scrubbed":2893,"log_size":6053,"log_dups_size":0,"ondisk_log_size":6053,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:34:53.059927+0000","scrub_duration":386,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":74055680,"num_objects":1130,"num_object_clones":0,"num_object_copies":3390,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1130,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3699,"num_write_kb":61952,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,0],"acting":[6,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.7","version":"124'2","reported_seq":362,"reported_epoch":234,"state":"active+clean","last_fresh":"2024-03-11T21:34:51.301545+0000","last_change":"2024-03-11T21:32:09.689578+0000","last_active":"2024-03-11T21:34:51.301545+0000","last_peered":"2024-03-11T21:34:51.301545+0000","last_clean":"2024-03-11T21:34:51.301545+0000","last_became_active":"2024-03-11T21:25:44.591376+0000","last_became_peered":"2024-03-11T21:25:44.591376+0000","last_unstale":"2024-03-11T21:34:51.301545+0000","last_undegraded":"2024-03-11T21:34:51.301545+0000","last_fullsized":"2024-03-11T21:34:51.301545+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"124'2","last_scrub_stamp":"2024-03-11T21:32:09.689480+0000","last_deep_scrub":"124'2","last_deep_scrub_stamp":"2024-03-11T21:29:34.956908+0000","last_clean_scrub_stamp":"2024-03-11T21:32:09.689480+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":48,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,4],"acting":[5,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2","version":"71'2","reported_seq":391,"reported_epoch":235,"state":"active+clean","last_fresh":"2024-03-11T21:34:52.332784+0000","last_change":"2024-03-11T21:32:21.543178+0000","last_active":"2024-03-11T21:34:52.332784+0000","last_peered":"2024-03-11T21:34:52.332784+0000","last_clean":"2024-03-11T21:34:52.332784+0000","last_became_active":"2024-03-11T21:25:38.539413+0000","last_became_peered":"2024-03-11T21:25:38.539413+0000","last_unstale":"2024-03-11T21:34:52.332784+0000","last_undegraded":"2024-03-11T21:34:52.332784+0000","last_fullsized":"2024-03-11T21:34:52.332784+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"71'2","last_scrub_stamp":"2024-03-11T21:32:21.543053+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:37.512018+0000","last_clean_scrub_stamp":"2024-03-11T21:32:21.543053+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:34:55.849681+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0.0014206449999999999,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":398,"reported_epoch":234,"state":"active+clean","last_fresh":"2024-03-11T21:34:51.300814+0000","last_change":"2024-03-11T21:34:41.147000+0000","last_active":"2024-03-11T21:34:51.300814+0000","last_peered":"2024-03-11T21:34:51.300814+0000","last_clean":"2024-03-11T21:34:51.300814+0000","last_became_active":"2024-03-11T21:34:41.146672+0000","last_became_peered":"2024-03-11T21:34:41.146672+0000","last_unstale":"2024-03-11T21:34:51.300814+0000","last_undegraded":"2024-03-11T21:34:51.300814+0000","last_fullsized":"2024-03-11T21:34:51.300814+0000","mapping_epoch":226,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":227,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:32:01.617459+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:37.512018+0000","last_clean_scrub_stamp":"2024-03-11T21:32:01.617459+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"87'128","reported_seq":496,"reported_epoch":234,"state":"active+clean","last_fresh":"2024-03-11T21:34:51.296406+0000","last_change":"2024-03-11T21:34:44.165013+0000","last_active":"2024-03-11T21:34:51.296406+0000","last_peered":"2024-03-11T21:34:51.296406+0000","last_clean":"2024-03-11T21:34:51.296406+0000","last_became_active":"2024-03-11T21:34:44.164569+0000","last_became_peered":"2024-03-11T21:34:44.164569+0000","last_unstale":"2024-03-11T21:34:51.296406+0000","last_undegraded":"2024-03-11T21:34:51.296406+0000","last_fullsized":"2024-03-11T21:34:51.296406+0000","mapping_epoch":229,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":230,"parent":"0.0","parent_split_bits":0,"last_scrub":"87'128","last_scrub_stamp":"2024-03-11T21:32:20.540647+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:24.261974+0000","last_clean_scrub_stamp":"2024-03-11T21:32:20.540647+0000","objects_scrubbed":2,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":198,"num_read_kb":294,"num_write":179,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":2951840,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.5","version":"74'16","reported_seq":518,"reported_epoch":234,"state":"active+clean","last_fresh":"2024-03-11T21:34:51.299610+0000","last_change":"2024-03-11T21:34:24.320823+0000","last_active":"2024-03-11T21:34:51.299610+0000","last_peered":"2024-03-11T21:34:51.299610+0000","last_clean":"2024-03-11T21:34:51.299610+0000","last_became_active":"2024-03-11T21:25:40.761457+0000","last_became_peered":"2024-03-11T21:25:40.761457+0000","last_unstale":"2024-03-11T21:34:51.299610+0000","last_undegraded":"2024-03-11T21:34:51.299610+0000","last_fullsized":"2024-03-11T21:34:51.299610+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'16","last_scrub_stamp":"2024-03-11T21:34:24.320713+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:39.525554+0000","last_clean_scrub_stamp":"2024-03-11T21:34:24.320713+0000","objects_scrubbed":8,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:35:49.716597+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":154,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":25,"num_read_kb":15,"num_write":13,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":25,"reported_epoch":234,"state":"active+clean","last_fresh":"2024-03-11T21:34:51.297626+0000","last_change":"2024-03-11T21:34:41.141916+0000","last_active":"2024-03-11T21:34:51.297626+0000","last_peered":"2024-03-11T21:34:51.297626+0000","last_clean":"2024-03-11T21:34:51.297626+0000","last_became_active":"2024-03-11T21:34:41.139782+0000","last_became_peered":"2024-03-11T21:34:41.139782+0000","last_unstale":"2024-03-11T21:34:51.297626+0000","last_undegraded":"2024-03-11T21:34:51.297626+0000","last_fullsized":"2024-03-11T21:34:51.297626+0000","mapping_epoch":226,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":227,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:32:21.266672+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:32:21.266672+0000","last_clean_scrub_stamp":"2024-03-11T21:32:21.266672+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,5],"acting":[7,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.a","version":"235'12802","reported_seq":12973,"reported_epoch":235,"state":"active+clean","last_fresh":"2024-03-11T21:34:52.396801+0000","last_change":"2024-03-11T21:31:52.272724+0000","last_active":"2024-03-11T21:34:52.396801+0000","last_peered":"2024-03-11T21:34:52.396801+0000","last_clean":"2024-03-11T21:34:52.396801+0000","last_became_active":"2024-03-11T21:31:52.271977+0000","last_became_peered":"2024-03-11T21:31:52.271977+0000","last_unstale":"2024-03-11T21:34:52.396801+0000","last_undegraded":"2024-03-11T21:34:52.396801+0000","last_fullsized":"2024-03-11T21:34:52.396801+0000","mapping_epoch":157,"log_start":"201'7953","ondisk_log_start":"201'7953","created":157,"last_epoch_clean":158,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:31:51.219688+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:31:51.219688+0000","last_clean_scrub_stamp":"2024-03-11T21:31:51.219688+0000","objects_scrubbed":0,"log_size":4849,"log_dups_size":0,"ondisk_log_size":4849,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":148897792,"num_objects":2272,"num_object_clones":0,"num_object_copies":6816,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2272,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12802,"num_write_kb":482368,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,5],"acting":[1,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.4","version":"0'0","reported_seq":371,"reported_epoch":235,"state":"active+clean","last_fresh":"2024-03-11T21:34:52.333290+0000","last_change":"2024-03-11T21:34:23.531211+0000","last_active":"2024-03-11T21:34:52.333290+0000","last_peered":"2024-03-11T21:34:52.333290+0000","last_clean":"2024-03-11T21:34:52.333290+0000","last_became_active":"2024-03-11T21:25:44.594188+0000","last_became_peered":"2024-03-11T21:25:44.594188+0000","last_unstale":"2024-03-11T21:34:52.333290+0000","last_undegraded":"2024-03-11T21:34:52.333290+0000","last_fullsized":"2024-03-11T21:34:52.333290+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:34:23.531064+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:43.559452+0000","last_clean_scrub_stamp":"2024-03-11T21:34:23.531064+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:35:45.276874+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":382,"reported_epoch":231,"state":"active+clean","last_fresh":"2024-03-11T21:34:48.225822+0000","last_change":"2024-03-11T21:32:22.561581+0000","last_active":"2024-03-11T21:34:48.225822+0000","last_peered":"2024-03-11T21:34:48.225822+0000","last_clean":"2024-03-11T21:34:48.225822+0000","last_became_active":"2024-03-11T21:25:38.543834+0000","last_became_peered":"2024-03-11T21:25:38.543834+0000","last_unstale":"2024-03-11T21:34:48.225822+0000","last_undegraded":"2024-03-11T21:34:48.225822+0000","last_fullsized":"2024-03-11T21:34:48.225822+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:32:22.560895+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:37.512018+0000","last_clean_scrub_stamp":"2024-03-11T21:32:22.560895+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0.00089342499999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":25,"reported_epoch":234,"state":"active+clean","last_fresh":"2024-03-11T21:34:51.296309+0000","last_change":"2024-03-11T21:34:41.150606+0000","last_active":"2024-03-11T21:34:51.296309+0000","last_peered":"2024-03-11T21:34:51.296309+0000","last_clean":"2024-03-11T21:34:51.296309+0000","last_became_active":"2024-03-11T21:34:41.149219+0000","last_became_peered":"2024-03-11T21:34:41.149219+0000","last_unstale":"2024-03-11T21:34:51.296309+0000","last_undegraded":"2024-03-11T21:34:51.296309+0000","last_fullsized":"2024-03-11T21:34:51.296309+0000","mapping_epoch":226,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":227,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:34:31.628014+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:37.512018+0000","last_clean_scrub_stamp":"2024-03-11T21:34:31.628014+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:35:57.776832+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,0],"acting":[7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.6","version":"74'12","reported_seq":409,"reported_epoch":231,"state":"active+clean","last_fresh":"2024-03-11T21:34:48.243152+0000","last_change":"2024-03-11T21:32:58.559118+0000","last_active":"2024-03-11T21:34:48.243152+0000","last_peered":"2024-03-11T21:34:48.243152+0000","last_clean":"2024-03-11T21:34:48.243152+0000","last_became_active":"2024-03-11T21:32:22.796486+0000","last_became_peered":"2024-03-11T21:32:22.796486+0000","last_unstale":"2024-03-11T21:34:48.243152+0000","last_undegraded":"2024-03-11T21:34:48.243152+0000","last_fullsized":"2024-03-11T21:34:48.243152+0000","mapping_epoch":166,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":167,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'12","last_scrub_stamp":"2024-03-11T21:32:58.559016+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:39.525554+0000","last_clean_scrub_stamp":"2024-03-11T21:32:58.559016+0000","objects_scrubbed":6,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":18,"num_read_kb":12,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":372,"reported_epoch":234,"state":"active+clean","last_fresh":"2024-03-11T21:34:51.301474+0000","last_change":"2024-03-11T21:34:25.411699+0000","last_active":"2024-03-11T21:34:51.301474+0000","last_peered":"2024-03-11T21:34:51.301474+0000","last_clean":"2024-03-11T21:34:51.301474+0000","last_became_active":"2024-03-11T21:25:42.572324+0000","last_became_peered":"2024-03-11T21:25:42.572324+0000","last_unstale":"2024-03-11T21:34:51.301474+0000","last_undegraded":"2024-03-11T21:34:51.301474+0000","last_fullsized":"2024-03-11T21:34:51.301474+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:34:25.411598+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:41.544550+0000","last_clean_scrub_stamp":"2024-03-11T21:34:25.411598+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:35:46.761784+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.f","version":"235'12606","reported_seq":12777,"reported_epoch":235,"state":"active+clean","last_fresh":"2024-03-11T21:34:52.734705+0000","last_change":"2024-03-11T21:31:52.272528+0000","last_active":"2024-03-11T21:34:52.734705+0000","last_peered":"2024-03-11T21:34:52.734705+0000","last_clean":"2024-03-11T21:34:52.734705+0000","last_became_active":"2024-03-11T21:31:52.272191+0000","last_became_peered":"2024-03-11T21:31:52.272191+0000","last_unstale":"2024-03-11T21:34:52.734705+0000","last_undegraded":"2024-03-11T21:34:52.734705+0000","last_fullsized":"2024-03-11T21:34:52.734705+0000","mapping_epoch":157,"log_start":"201'7820","ondisk_log_start":"201'7820","created":157,"last_epoch_clean":158,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:31:51.219688+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:31:51.219688+0000","last_clean_scrub_stamp":"2024-03-11T21:31:51.219688+0000","objects_scrubbed":0,"log_size":4786,"log_dups_size":0,"ondisk_log_size":4786,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:34:55.849575+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":144834560,"num_objects":2210,"num_object_clones":0,"num_object_copies":6630,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2210,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12606,"num_write_kb":474112,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,1],"acting":[3,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":379,"reported_epoch":235,"state":"active+clean","last_fresh":"2024-03-11T21:34:52.334746+0000","last_change":"2024-03-11T21:34:33.565178+0000","last_active":"2024-03-11T21:34:52.334746+0000","last_peered":"2024-03-11T21:34:52.334746+0000","last_clean":"2024-03-11T21:34:52.334746+0000","last_became_active":"2024-03-11T21:25:45.301416+0000","last_became_peered":"2024-03-11T21:25:45.301416+0000","last_unstale":"2024-03-11T21:34:52.334746+0000","last_undegraded":"2024-03-11T21:34:52.334746+0000","last_fullsized":"2024-03-11T21:34:52.334746+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:34:33.564652+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:43.559452+0000","last_clean_scrub_stamp":"2024-03-11T21:34:33.564652+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:35:48.030891+0000","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":25,"reported_epoch":234,"state":"active+clean","last_fresh":"2024-03-11T21:34:51.296723+0000","last_change":"2024-03-11T21:34:41.155790+0000","last_active":"2024-03-11T21:34:51.296723+0000","last_peered":"2024-03-11T21:34:51.296723+0000","last_clean":"2024-03-11T21:34:51.296723+0000","last_became_active":"2024-03-11T21:34:41.152721+0000","last_became_peered":"2024-03-11T21:34:41.152721+0000","last_unstale":"2024-03-11T21:34:51.296723+0000","last_undegraded":"2024-03-11T21:34:51.296723+0000","last_fullsized":"2024-03-11T21:34:51.296723+0000","mapping_epoch":226,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":227,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:34:16.591255+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:26:57.333669+0000","last_clean_scrub_stamp":"2024-03-11T21:34:16.591255+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:35:24.096418+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.3","version":"165'27","reported_seq":463,"reported_epoch":231,"state":"active+clean","last_fresh":"2024-03-11T21:34:48.238515+0000","last_change":"2024-03-11T21:34:41.168911+0000","last_active":"2024-03-11T21:34:48.238515+0000","last_peered":"2024-03-11T21:34:48.238515+0000","last_clean":"2024-03-11T21:34:48.238515+0000","last_became_active":"2024-03-11T21:34:41.145607+0000","last_became_peered":"2024-03-11T21:34:41.145607+0000","last_unstale":"2024-03-11T21:34:48.238515+0000","last_undegraded":"2024-03-11T21:34:48.238515+0000","last_fullsized":"2024-03-11T21:34:48.238515+0000","mapping_epoch":226,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":227,"parent":"0.0","parent_split_bits":0,"last_scrub":"165'27","last_scrub_stamp":"2024-03-11T21:32:21.576732+0000","last_deep_scrub":"165'27","last_deep_scrub_stamp":"2024-03-11T21:32:21.576732+0000","last_clean_scrub_stamp":"2024-03-11T21:32:21.576732+0000","objects_scrubbed":8,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":23,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":39,"num_read_kb":25,"num_write":22,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":660,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":381,"reported_epoch":234,"state":"active+clean","last_fresh":"2024-03-11T21:34:51.299053+0000","last_change":"2024-03-11T21:32:23.882539+0000","last_active":"2024-03-11T21:34:51.299053+0000","last_peered":"2024-03-11T21:34:51.299053+0000","last_clean":"2024-03-11T21:34:51.299053+0000","last_became_active":"2024-03-11T21:25:42.579045+0000","last_became_peered":"2024-03-11T21:25:42.579045+0000","last_unstale":"2024-03-11T21:34:51.299053+0000","last_undegraded":"2024-03-11T21:34:51.299053+0000","last_fullsized":"2024-03-11T21:34:51.299053+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:32:23.882391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:29:36.918936+0000","last_clean_scrub_stamp":"2024-03-11T21:32:23.882391+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:34:53.059896+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"8.c","version":"231'12477","reported_seq":13763,"reported_epoch":231,"state":"active+remapped+backfilling","last_fresh":"2024-03-11T21:34:48.271375+0000","last_change":"2024-03-11T21:34:42.711176+0000","last_active":"2024-03-11T21:34:48.271375+0000","last_peered":"2024-03-11T21:34:48.271375+0000","last_clean":"2024-03-11T21:34:40.079870+0000","last_became_active":"2024-03-11T21:34:42.671023+0000","last_became_peered":"2024-03-11T21:34:42.671023+0000","last_unstale":"2024-03-11T21:34:48.271375+0000","last_undegraded":"2024-03-11T21:34:48.271375+0000","last_fullsized":"2024-03-11T21:34:48.271375+0000","mapping_epoch":227,"log_start":"202'7933","ondisk_log_start":"202'7933","created":157,"last_epoch_clean":158,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:31:51.219688+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:31:51.219688+0000","last_clean_scrub_stamp":"2024-03-11T21:31:51.219688+0000","objects_scrubbed":0,"log_size":4544,"log_dups_size":0,"ondisk_log_size":4544,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":157614080,"num_objects":2405,"num_object_clones":0,"num_object_copies":7215,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1876,"num_objects_unfound":0,"num_objects_dirty":2405,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12477,"num_write_kb":476224,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":560,"num_bytes_recovered":36700160,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,5],"avail_no_missing":["0","3","5"],"object_location_counts":[{"shards":"0,3,5","objects":2405}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":375,"reported_epoch":234,"state":"active+clean","last_fresh":"2024-03-11T21:34:51.295047+0000","last_change":"2024-03-11T21:32:22.791904+0000","last_active":"2024-03-11T21:34:51.295047+0000","last_peered":"2024-03-11T21:34:51.295047+0000","last_clean":"2024-03-11T21:34:51.295047+0000","last_became_active":"2024-03-11T21:32:22.791439+0000","last_became_peered":"2024-03-11T21:32:22.791439+0000","last_unstale":"2024-03-11T21:34:51.295047+0000","last_undegraded":"2024-03-11T21:34:51.295047+0000","last_fullsized":"2024-03-11T21:34:51.295047+0000","mapping_epoch":166,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":167,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:32:05.636174+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:43.559452+0000","last_clean_scrub_stamp":"2024-03-11T21:32:05.636174+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,2],"acting":[4,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.6","version":"67'1","reported_seq":384,"reported_epoch":235,"state":"active+clean","last_fresh":"2024-03-11T21:34:52.333717+0000","last_change":"2024-03-11T21:32:05.556583+0000","last_active":"2024-03-11T21:34:52.333717+0000","last_peered":"2024-03-11T21:34:52.333717+0000","last_clean":"2024-03-11T21:34:52.333717+0000","last_became_active":"2024-03-11T21:25:38.536142+0000","last_became_peered":"2024-03-11T21:25:38.536142+0000","last_unstale":"2024-03-11T21:34:52.333717+0000","last_undegraded":"2024-03-11T21:34:52.333717+0000","last_fullsized":"2024-03-11T21:34:52.333717+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'1","last_scrub_stamp":"2024-03-11T21:32:05.556427+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:37.512018+0000","last_clean_scrub_stamp":"2024-03-11T21:32:05.556427+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,4],"acting":[1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.0","version":"93'26","reported_seq":466,"reported_epoch":235,"state":"active+clean","last_fresh":"2024-03-11T21:34:52.332645+0000","last_change":"2024-03-11T21:34:41.181772+0000","last_active":"2024-03-11T21:34:52.332645+0000","last_peered":"2024-03-11T21:34:52.332645+0000","last_clean":"2024-03-11T21:34:52.332645+0000","last_became_active":"2024-03-11T21:34:41.145354+0000","last_became_peered":"2024-03-11T21:34:41.145354+0000","last_unstale":"2024-03-11T21:34:52.332645+0000","last_undegraded":"2024-03-11T21:34:52.332645+0000","last_fullsized":"2024-03-11T21:34:52.332645+0000","mapping_epoch":226,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":227,"parent":"0.0","parent_split_bits":0,"last_scrub":"93'26","last_scrub_stamp":"2024-03-11T21:34:16.863984+0000","last_deep_scrub":"93'26","last_deep_scrub_stamp":"2024-03-11T21:26:40.680324+0000","last_clean_scrub_stamp":"2024-03-11T21:34:16.863984+0000","objects_scrubbed":8,"log_size":26,"log_dups_size":0,"ondisk_log_size":26,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:35:24.709288+0000","scrub_duration":22,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":34,"num_read_kb":22,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":440,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":385,"reported_epoch":234,"state":"active+clean","last_fresh":"2024-03-11T21:34:51.294494+0000","last_change":"2024-03-11T21:34:41.135951+0000","last_active":"2024-03-11T21:34:51.294494+0000","last_peered":"2024-03-11T21:34:51.294494+0000","last_clean":"2024-03-11T21:34:51.294494+0000","last_became_active":"2024-03-11T21:34:41.135551+0000","last_became_peered":"2024-03-11T21:34:41.135551+0000","last_unstale":"2024-03-11T21:34:51.294494+0000","last_undegraded":"2024-03-11T21:34:51.294494+0000","last_fullsized":"2024-03-11T21:34:51.294494+0000","mapping_epoch":226,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":227,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:32:16.599049+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:26:51.863621+0000","last_clean_scrub_stamp":"2024-03-11T21:32:16.599049+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,7],"acting":[4,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"8.d","version":"234'12893","reported_seq":13747,"reported_epoch":234,"state":"active+remapped+backfilling","last_fresh":"2024-03-11T21:34:52.113198+0000","last_change":"2024-03-11T21:34:51.190691+0000","last_active":"2024-03-11T21:34:52.113198+0000","last_peered":"2024-03-11T21:34:52.113198+0000","last_clean":"2024-03-11T21:34:40.110110+0000","last_became_active":"2024-03-11T21:34:42.688412+0000","last_became_peered":"2024-03-11T21:34:42.688412+0000","last_unstale":"2024-03-11T21:34:52.113198+0000","last_undegraded":"2024-03-11T21:34:52.113198+0000","last_fullsized":"2024-03-11T21:34:52.113198+0000","mapping_epoch":227,"log_start":"202'8090","ondisk_log_start":"202'8090","created":157,"last_epoch_clean":158,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:31:51.219688+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:31:51.219688+0000","last_clean_scrub_stamp":"2024-03-11T21:31:51.219688+0000","objects_scrubbed":0,"log_size":4803,"log_dups_size":0,"ondisk_log_size":4803,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":149553176,"num_objects":2283,"num_object_clones":0,"num_object_copies":6849,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1946,"num_objects_unfound":0,"num_objects_dirty":2283,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":12893,"num_write_kb":485569,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":339,"num_bytes_recovered":22216704,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,2],"acting":[5,2,6],"avail_no_missing":["5","2","6"],"object_location_counts":[{"shards":"2,5,6","objects":2283}],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.3","version":"0'0","reported_seq":25,"reported_epoch":234,"state":"active+clean","last_fresh":"2024-03-11T21:34:51.296859+0000","last_change":"2024-03-11T21:34:41.143973+0000","last_active":"2024-03-11T21:34:51.296859+0000","last_peered":"2024-03-11T21:34:51.296859+0000","last_clean":"2024-03-11T21:34:51.296859+0000","last_became_active":"2024-03-11T21:34:41.142683+0000","last_became_peered":"2024-03-11T21:34:41.142683+0000","last_unstale":"2024-03-11T21:34:51.296859+0000","last_undegraded":"2024-03-11T21:34:51.296859+0000","last_fullsized":"2024-03-11T21:34:51.296859+0000","mapping_epoch":226,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":227,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:34:27.353700+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:29:42.485740+0000","last_clean_scrub_stamp":"2024-03-11T21:34:27.353700+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:35:56.768836+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,2],"acting":[7,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":401,"reported_epoch":234,"state":"active+clean","last_fresh":"2024-03-11T21:34:51.299400+0000","last_change":"2024-03-11T21:34:41.146260+0000","last_active":"2024-03-11T21:34:51.299400+0000","last_peered":"2024-03-11T21:34:51.299400+0000","last_clean":"2024-03-11T21:34:51.299400+0000","last_became_active":"2024-03-11T21:34:41.145628+0000","last_became_peered":"2024-03-11T21:34:41.145628+0000","last_unstale":"2024-03-11T21:34:51.299400+0000","last_undegraded":"2024-03-11T21:34:51.299400+0000","last_fullsized":"2024-03-11T21:34:51.299400+0000","mapping_epoch":226,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":227,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:34:10.437356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:37.512018+0000","last_clean_scrub_stamp":"2024-03-11T21:34:10.437356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:35:20.602470+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,2],"acting":[6,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.1","version":"74'14","reported_seq":398,"reported_epoch":234,"state":"active+clean","last_fresh":"2024-03-11T21:34:51.293788+0000","last_change":"2024-03-11T21:32:58.900322+0000","last_active":"2024-03-11T21:34:51.293788+0000","last_peered":"2024-03-11T21:34:51.293788+0000","last_clean":"2024-03-11T21:34:51.293788+0000","last_became_active":"2024-03-11T21:25:40.546219+0000","last_became_peered":"2024-03-11T21:25:40.546219+0000","last_unstale":"2024-03-11T21:34:51.293788+0000","last_undegraded":"2024-03-11T21:34:51.293788+0000","last_fullsized":"2024-03-11T21:34:51.293788+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'14","last_scrub_stamp":"2024-03-11T21:32:58.900223+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:39.525554+0000","last_clean_scrub_stamp":"2024-03-11T21:32:58.900223+0000","objects_scrubbed":7,"log_size":14,"log_dups_size":0,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":21,"num_read_kb":14,"num_write":14,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6],"acting":[4,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.0","version":"98'9","reported_seq":486,"reported_epoch":235,"state":"active+clean","last_fresh":"2024-03-11T21:34:52.332680+0000","last_change":"2024-03-11T21:31:21.562516+0000","last_active":"2024-03-11T21:34:52.332680+0000","last_peered":"2024-03-11T21:34:52.332680+0000","last_clean":"2024-03-11T21:34:52.332680+0000","last_became_active":"2024-03-11T21:25:42.576126+0000","last_became_peered":"2024-03-11T21:25:42.576126+0000","last_unstale":"2024-03-11T21:34:52.332680+0000","last_undegraded":"2024-03-11T21:34:52.332680+0000","last_fullsized":"2024-03-11T21:34:52.332680+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"98'9","last_scrub_stamp":"2024-03-11T21:31:21.562395+0000","last_deep_scrub":"98'9","last_deep_scrub_stamp":"2024-03-11T21:27:29.745711+0000","last_clean_scrub_stamp":"2024-03-11T21:31:21.562395+0000","objects_scrubbed":1,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:34:55.849539+0000","scrub_duration":17,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.2","version":"234'9216","reported_seq":9425,"reported_epoch":234,"state":"active+clean","last_fresh":"2024-03-11T21:34:51.831093+0000","last_change":"2024-03-11T21:32:58.359357+0000","last_active":"2024-03-11T21:34:51.831093+0000","last_peered":"2024-03-11T21:34:51.831093+0000","last_clean":"2024-03-11T21:34:51.831093+0000","last_became_active":"2024-03-11T21:32:57.078777+0000","last_became_peered":"2024-03-11T21:32:57.078777+0000","last_unstale":"2024-03-11T21:34:51.831093+0000","last_undegraded":"2024-03-11T21:34:51.831093+0000","last_fullsized":"2024-03-11T21:34:51.831093+0000","mapping_epoch":157,"log_start":"170'3729","ondisk_log_start":"170'3729","created":157,"last_epoch_clean":182,"parent":"0.0","parent_split_bits":0,"last_scrub":"182'5608","last_scrub_stamp":"2024-03-11T21:32:58.359183+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:31:51.219688+0000","last_clean_scrub_stamp":"2024-03-11T21:32:58.359183+0000","objects_scrubbed":2818,"log_size":5487,"log_dups_size":0,"ondisk_log_size":5487,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":474,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":73138176,"num_objects":1116,"num_object_clones":0,"num_object_copies":3348,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1116,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3610,"num_write_kb":61056,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,5],"acting":[4,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.c","version":"0'0","reported_seq":362,"reported_epoch":235,"state":"active+clean","last_fresh":"2024-03-11T21:34:52.333228+0000","last_change":"2024-03-11T21:32:17.600335+0000","last_active":"2024-03-11T21:34:52.333228+0000","last_peered":"2024-03-11T21:34:52.333228+0000","last_clean":"2024-03-11T21:34:52.333228+0000","last_became_active":"2024-03-11T21:25:45.302207+0000","last_became_peered":"2024-03-11T21:25:45.302207+0000","last_unstale":"2024-03-11T21:34:52.333228+0000","last_undegraded":"2024-03-11T21:34:52.333228+0000","last_fullsized":"2024-03-11T21:34:52.333228+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:32:17.600213+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:43.559452+0000","last_clean_scrub_stamp":"2024-03-11T21:32:17.600213+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:34:55.849649+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.f","version":"0'0","reported_seq":380,"reported_epoch":234,"state":"active+clean","last_fresh":"2024-03-11T21:34:51.301359+0000","last_change":"2024-03-11T21:34:26.357068+0000","last_active":"2024-03-11T21:34:51.301359+0000","last_peered":"2024-03-11T21:34:51.301359+0000","last_clean":"2024-03-11T21:34:51.301359+0000","last_became_active":"2024-03-11T21:25:42.575935+0000","last_became_peered":"2024-03-11T21:25:42.575935+0000","last_unstale":"2024-03-11T21:34:51.301359+0000","last_undegraded":"2024-03-11T21:34:51.301359+0000","last_fullsized":"2024-03-11T21:34:51.301359+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:34:26.356954+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:41.544550+0000","last_clean_scrub_stamp":"2024-03-11T21:34:26.356954+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:35:54.552150+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.8","version":"0'0","reported_seq":25,"reported_epoch":234,"state":"active+clean","last_fresh":"2024-03-11T21:34:51.297830+0000","last_change":"2024-03-11T21:34:41.145417+0000","last_active":"2024-03-11T21:34:51.297830+0000","last_peered":"2024-03-11T21:34:51.297830+0000","last_clean":"2024-03-11T21:34:51.297830+0000","last_became_active":"2024-03-11T21:34:41.144681+0000","last_became_peered":"2024-03-11T21:34:41.144681+0000","last_unstale":"2024-03-11T21:34:51.297830+0000","last_undegraded":"2024-03-11T21:34:51.297830+0000","last_fullsized":"2024-03-11T21:34:51.297830+0000","mapping_epoch":226,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":227,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:34:23.711140+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:37.512018+0000","last_clean_scrub_stamp":"2024-03-11T21:34:23.711140+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:35:29.880606+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,1],"acting":[7,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.e","version":"109'16","reported_seq":401,"reported_epoch":234,"state":"active+clean","last_fresh":"2024-03-11T21:34:51.294220+0000","last_change":"2024-03-11T21:32:53.834261+0000","last_active":"2024-03-11T21:34:51.294220+0000","last_peered":"2024-03-11T21:34:51.294220+0000","last_clean":"2024-03-11T21:34:51.294220+0000","last_became_active":"2024-03-11T21:25:40.552178+0000","last_became_peered":"2024-03-11T21:25:40.552178+0000","last_unstale":"2024-03-11T21:34:51.294220+0000","last_undegraded":"2024-03-11T21:34:51.294220+0000","last_fullsized":"2024-03-11T21:34:51.294220+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"109'16","last_scrub_stamp":"2024-03-11T21:32:53.834165+0000","last_deep_scrub":"109'16","last_deep_scrub_stamp":"2024-03-11T21:28:24.739395+0000","last_clean_scrub_stamp":"2024-03-11T21:32:53.834165+0000","objects_scrubbed":5,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"8.3","version":"235'9092","reported_seq":9302,"reported_epoch":235,"state":"active+clean","last_fresh":"2024-03-11T21:34:52.382607+0000","last_change":"2024-03-11T21:34:09.073683+0000","last_active":"2024-03-11T21:34:52.382607+0000","last_peered":"2024-03-11T21:34:52.382607+0000","last_clean":"2024-03-11T21:34:52.382607+0000","last_became_active":"2024-03-11T21:32:57.081711+0000","last_became_peered":"2024-03-11T21:32:57.081711+0000","last_unstale":"2024-03-11T21:34:52.382607+0000","last_undegraded":"2024-03-11T21:34:52.382607+0000","last_fullsized":"2024-03-11T21:34:52.382607+0000","mapping_epoch":157,"log_start":"171'3739","ondisk_log_start":"171'3739","created":157,"last_epoch_clean":182,"parent":"0.0","parent_split_bits":0,"last_scrub":"218'7578","last_scrub_stamp":"2024-03-11T21:34:09.073424+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:31:51.219688+0000","last_clean_scrub_stamp":"2024-03-11T21:34:09.073424+0000","objects_scrubbed":2660,"log_size":5353,"log_dups_size":0,"ondisk_log_size":5353,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:35:12.983280+0000","scrub_duration":534,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":74907648,"num_objects":1143,"num_object_clones":0,"num_object_copies":3429,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1143,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1517,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,6],"acting":[1,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.d","version":"0'0","reported_seq":368,"reported_epoch":234,"state":"active+clean","last_fresh":"2024-03-11T21:34:51.300984+0000","last_change":"2024-03-11T21:32:13.641959+0000","last_active":"2024-03-11T21:34:51.300984+0000","last_peered":"2024-03-11T21:34:51.300984+0000","last_clean":"2024-03-11T21:34:51.300984+0000","last_became_active":"2024-03-11T21:25:44.590333+0000","last_became_peered":"2024-03-11T21:25:44.590333+0000","last_unstale":"2024-03-11T21:34:51.300984+0000","last_undegraded":"2024-03-11T21:34:51.300984+0000","last_fullsized":"2024-03-11T21:34:51.300984+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:32:13.641870+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:43.559452+0000","last_clean_scrub_stamp":"2024-03-11T21:32:13.641870+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":45,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.e","version":"74'8","reported_seq":492,"reported_epoch":234,"state":"active+clean","last_fresh":"2024-03-11T21:34:51.294488+0000","last_change":"2024-03-11T21:34:18.722515+0000","last_active":"2024-03-11T21:34:51.294488+0000","last_peered":"2024-03-11T21:34:51.294488+0000","last_clean":"2024-03-11T21:34:51.294488+0000","last_became_active":"2024-03-11T21:25:42.576742+0000","last_became_peered":"2024-03-11T21:25:42.576742+0000","last_unstale":"2024-03-11T21:34:51.294488+0000","last_undegraded":"2024-03-11T21:34:51.294488+0000","last_fullsized":"2024-03-11T21:34:51.294488+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'8","last_scrub_stamp":"2024-03-11T21:34:18.722196+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:41.544550+0000","last_clean_scrub_stamp":"2024-03-11T21:34:18.722196+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:35:36.062991+0000","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,0],"acting":[4,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.9","version":"0'0","reported_seq":395,"reported_epoch":235,"state":"active+clean","last_fresh":"2024-03-11T21:34:52.334653+0000","last_change":"2024-03-11T21:34:41.156338+0000","last_active":"2024-03-11T21:34:52.334653+0000","last_peered":"2024-03-11T21:34:52.334653+0000","last_clean":"2024-03-11T21:34:52.334653+0000","last_became_active":"2024-03-11T21:34:41.155893+0000","last_became_peered":"2024-03-11T21:34:41.155893+0000","last_unstale":"2024-03-11T21:34:52.334653+0000","last_undegraded":"2024-03-11T21:34:52.334653+0000","last_fullsized":"2024-03-11T21:34:52.334653+0000","mapping_epoch":226,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":227,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:32:02.506579+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:26:50.991712+0000","last_clean_scrub_stamp":"2024-03-11T21:32:02.506579+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,3],"acting":[1,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.f","version":"74'15","reported_seq":363,"reported_epoch":235,"state":"active+clean","last_fresh":"2024-03-11T21:34:52.296387+0000","last_change":"2024-03-11T21:34:21.519858+0000","last_active":"2024-03-11T21:34:52.296387+0000","last_peered":"2024-03-11T21:34:52.296387+0000","last_clean":"2024-03-11T21:34:52.296387+0000","last_became_active":"2024-03-11T21:26:11.216224+0000","last_became_peered":"2024-03-11T21:26:11.216224+0000","last_unstale":"2024-03-11T21:34:52.296387+0000","last_undegraded":"2024-03-11T21:34:52.296387+0000","last_fullsized":"2024-03-11T21:34:52.296387+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'15","last_scrub_stamp":"2024-03-11T21:34:21.517894+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:39.525554+0000","last_clean_scrub_stamp":"2024-03-11T21:34:21.517894+0000","objects_scrubbed":7,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:35:35.603333+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,4],"acting":[2,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"8.0","version":"235'8953","reported_seq":9165,"reported_epoch":235,"state":"active+clean","last_fresh":"2024-03-11T21:34:52.609449+0000","last_change":"2024-03-11T21:32:58.324488+0000","last_active":"2024-03-11T21:34:52.609449+0000","last_peered":"2024-03-11T21:34:52.609449+0000","last_clean":"2024-03-11T21:34:52.609449+0000","last_became_active":"2024-03-11T21:32:57.071801+0000","last_became_peered":"2024-03-11T21:32:57.071801+0000","last_unstale":"2024-03-11T21:34:52.609449+0000","last_undegraded":"2024-03-11T21:34:52.609449+0000","last_fullsized":"2024-03-11T21:34:52.609449+0000","mapping_epoch":157,"log_start":"170'3348","ondisk_log_start":"170'3348","created":157,"last_epoch_clean":182,"parent":"0.0","parent_split_bits":0,"last_scrub":"183'5466","last_scrub_stamp":"2024-03-11T21:32:58.324387+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:31:51.219688+0000","last_clean_scrub_stamp":"2024-03-11T21:32:58.324387+0000","objects_scrubbed":2659,"log_size":5605,"log_dups_size":0,"ondisk_log_size":5605,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:34:55.849776+0000","scrub_duration":612,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":72679424,"num_objects":1109,"num_object_clones":0,"num_object_copies":3327,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1109,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3490,"num_write_kb":62080,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,1],"acting":[3,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.e","version":"0'0","reported_seq":369,"reported_epoch":234,"state":"active+clean","last_fresh":"2024-03-11T21:34:51.294660+0000","last_change":"2024-03-11T21:34:12.721980+0000","last_active":"2024-03-11T21:34:51.294660+0000","last_peered":"2024-03-11T21:34:51.294660+0000","last_clean":"2024-03-11T21:34:51.294660+0000","last_became_active":"2024-03-11T21:25:44.590198+0000","last_became_peered":"2024-03-11T21:25:44.590198+0000","last_unstale":"2024-03-11T21:34:51.294660+0000","last_undegraded":"2024-03-11T21:34:51.294660+0000","last_fullsized":"2024-03-11T21:34:51.294660+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:34:12.721766+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:43.559452+0000","last_clean_scrub_stamp":"2024-03-11T21:34:12.721766+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:35:37.637969+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.d","version":"109'9","reported_seq":494,"reported_epoch":235,"state":"active+clean","last_fresh":"2024-03-11T21:34:52.295392+0000","last_change":"2024-03-11T21:32:12.413588+0000","last_active":"2024-03-11T21:34:52.295392+0000","last_peered":"2024-03-11T21:34:52.295392+0000","last_clean":"2024-03-11T21:34:52.295392+0000","last_became_active":"2024-03-11T21:25:42.575583+0000","last_became_peered":"2024-03-11T21:25:42.575583+0000","last_unstale":"2024-03-11T21:34:52.295392+0000","last_undegraded":"2024-03-11T21:34:52.295392+0000","last_fullsized":"2024-03-11T21:34:52.295392+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"109'9","last_scrub_stamp":"2024-03-11T21:32:12.413216+0000","last_deep_scrub":"109'9","last_deep_scrub_stamp":"2024-03-11T21:28:27.082975+0000","last_clean_scrub_stamp":"2024-03-11T21:32:12.413216+0000","objects_scrubbed":1,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:34:52.399493+0000","scrub_duration":25,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,5],"acting":[2,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.a","version":"0'0","reported_seq":403,"reported_epoch":235,"state":"active+clean","last_fresh":"2024-03-11T21:34:52.333033+0000","last_change":"2024-03-11T21:34:41.145567+0000","last_active":"2024-03-11T21:34:52.333033+0000","last_peered":"2024-03-11T21:34:52.333033+0000","last_clean":"2024-03-11T21:34:52.333033+0000","last_became_active":"2024-03-11T21:34:41.144773+0000","last_became_peered":"2024-03-11T21:34:41.144773+0000","last_unstale":"2024-03-11T21:34:52.333033+0000","last_undegraded":"2024-03-11T21:34:52.333033+0000","last_fullsized":"2024-03-11T21:34:52.333033+0000","mapping_epoch":226,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":227,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:34:22.526916+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:37.512018+0000","last_clean_scrub_stamp":"2024-03-11T21:34:22.526916+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:35:34.423315+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,7],"acting":[1,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.c","version":"74'10","reported_seq":392,"reported_epoch":234,"state":"active+clean","last_fresh":"2024-03-11T21:34:51.294085+0000","last_change":"2024-03-11T21:32:54.846999+0000","last_active":"2024-03-11T21:34:51.294085+0000","last_peered":"2024-03-11T21:34:51.294085+0000","last_clean":"2024-03-11T21:34:51.294085+0000","last_became_active":"2024-03-11T21:25:40.550885+0000","last_became_peered":"2024-03-11T21:25:40.550885+0000","last_unstale":"2024-03-11T21:34:51.294085+0000","last_undegraded":"2024-03-11T21:34:51.294085+0000","last_fullsized":"2024-03-11T21:34:51.294085+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'10","last_scrub_stamp":"2024-03-11T21:32:54.846584+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:39.525554+0000","last_clean_scrub_stamp":"2024-03-11T21:32:54.846584+0000","objects_scrubbed":5,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"8.1","version":"234'9094","reported_seq":9304,"reported_epoch":235,"state":"active+clean","last_fresh":"2024-03-11T21:34:52.334471+0000","last_change":"2024-03-11T21:32:58.666237+0000","last_active":"2024-03-11T21:34:52.334471+0000","last_peered":"2024-03-11T21:34:52.334471+0000","last_clean":"2024-03-11T21:34:52.334471+0000","last_became_active":"2024-03-11T21:32:57.078036+0000","last_became_peered":"2024-03-11T21:32:57.078036+0000","last_unstale":"2024-03-11T21:34:52.334471+0000","last_undegraded":"2024-03-11T21:34:52.334471+0000","last_fullsized":"2024-03-11T21:34:52.334471+0000","mapping_epoch":157,"log_start":"170'3758","ondisk_log_start":"170'3758","created":157,"last_epoch_clean":182,"parent":"0.0","parent_split_bits":0,"last_scrub":"182'5517","last_scrub_stamp":"2024-03-11T21:32:58.666106+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:31:51.219688+0000","last_clean_scrub_stamp":"2024-03-11T21:32:58.666106+0000","objects_scrubbed":2736,"log_size":5336,"log_dups_size":0,"ondisk_log_size":5336,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":348,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":74121216,"num_objects":1131,"num_object_clones":0,"num_object_copies":3393,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1131,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3577,"num_write_kb":63104,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.f","version":"0'0","reported_seq":370,"reported_epoch":235,"state":"active+clean","last_fresh":"2024-03-11T21:34:52.296500+0000","last_change":"2024-03-11T21:32:17.431835+0000","last_active":"2024-03-11T21:34:52.296500+0000","last_peered":"2024-03-11T21:34:52.296500+0000","last_clean":"2024-03-11T21:34:52.296500+0000","last_became_active":"2024-03-11T21:25:44.593414+0000","last_became_peered":"2024-03-11T21:25:44.593414+0000","last_unstale":"2024-03-11T21:34:52.296500+0000","last_undegraded":"2024-03-11T21:34:52.296500+0000","last_fullsized":"2024-03-11T21:34:52.296500+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:32:17.431576+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:27:03.010969+0000","last_clean_scrub_stamp":"2024-03-11T21:32:17.431576+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:34:52.399522+0000","scrub_duration":75,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,4],"acting":[2,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":375,"reported_epoch":235,"state":"active+clean","last_fresh":"2024-03-11T21:34:52.333204+0000","last_change":"2024-03-11T21:34:21.502786+0000","last_active":"2024-03-11T21:34:52.333204+0000","last_peered":"2024-03-11T21:34:52.333204+0000","last_clean":"2024-03-11T21:34:52.333204+0000","last_became_active":"2024-03-11T21:25:42.579409+0000","last_became_peered":"2024-03-11T21:25:42.579409+0000","last_unstale":"2024-03-11T21:34:52.333204+0000","last_undegraded":"2024-03-11T21:34:52.333204+0000","last_fullsized":"2024-03-11T21:34:52.333204+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:34:21.502521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:30:22.750053+0000","last_clean_scrub_stamp":"2024-03-11T21:34:21.502521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:35:30.013386+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.b","version":"0'0","reported_seq":25,"reported_epoch":234,"state":"active+clean","last_fresh":"2024-03-11T21:34:51.296970+0000","last_change":"2024-03-11T21:34:41.136339+0000","last_active":"2024-03-11T21:34:51.296970+0000","last_peered":"2024-03-11T21:34:51.296970+0000","last_clean":"2024-03-11T21:34:51.296970+0000","last_became_active":"2024-03-11T21:34:41.134987+0000","last_became_peered":"2024-03-11T21:34:41.134987+0000","last_unstale":"2024-03-11T21:34:51.296970+0000","last_undegraded":"2024-03-11T21:34:51.296970+0000","last_fullsized":"2024-03-11T21:34:51.296970+0000","mapping_epoch":226,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":227,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:34:29.735148+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:37.512018+0000","last_clean_scrub_stamp":"2024-03-11T21:34:29.735148+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:35:43.850507+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,5],"acting":[7,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.d","version":"165'25","reported_seq":413,"reported_epoch":234,"state":"active+clean","last_fresh":"2024-03-11T21:34:51.294349+0000","last_change":"2024-03-11T21:32:21.678485+0000","last_active":"2024-03-11T21:34:51.294349+0000","last_peered":"2024-03-11T21:34:51.294349+0000","last_clean":"2024-03-11T21:34:51.294349+0000","last_became_active":"2024-03-11T21:25:40.553454+0000","last_became_peered":"2024-03-11T21:25:40.553454+0000","last_unstale":"2024-03-11T21:34:51.294349+0000","last_undegraded":"2024-03-11T21:34:51.294349+0000","last_fullsized":"2024-03-11T21:34:51.294349+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"165'25","last_scrub_stamp":"2024-03-11T21:32:21.678328+0000","last_deep_scrub":"165'25","last_deep_scrub_stamp":"2024-03-11T21:32:21.678328+0000","last_clean_scrub_stamp":"2024-03-11T21:32:21.678328+0000","objects_scrubbed":8,"log_size":25,"log_dups_size":0,"ondisk_log_size":25,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":21,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":29,"num_read_kb":19,"num_write":18,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,1],"acting":[4,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"8.6","version":"231'8970","reported_seq":9173,"reported_epoch":231,"state":"active+clean","last_fresh":"2024-03-11T21:34:48.243016+0000","last_change":"2024-03-11T21:32:58.205093+0000","last_active":"2024-03-11T21:34:48.243016+0000","last_peered":"2024-03-11T21:34:48.243016+0000","last_clean":"2024-03-11T21:34:48.243016+0000","last_became_active":"2024-03-11T21:32:57.074062+0000","last_became_peered":"2024-03-11T21:32:57.074062+0000","last_unstale":"2024-03-11T21:34:48.243016+0000","last_undegraded":"2024-03-11T21:34:48.243016+0000","last_fullsized":"2024-03-11T21:34:48.243016+0000","mapping_epoch":157,"log_start":"172'3920","ondisk_log_start":"172'3920","created":157,"last_epoch_clean":182,"parent":"0.0","parent_split_bits":0,"last_scrub":"183'5458","last_scrub_stamp":"2024-03-11T21:32:58.204881+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:31:51.219688+0000","last_clean_scrub_stamp":"2024-03-11T21:32:58.204881+0000","objects_scrubbed":2753,"log_size":5050,"log_dups_size":0,"ondisk_log_size":5050,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":640,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":80740352,"num_objects":1232,"num_object_clones":0,"num_object_copies":3696,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1232,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3513,"num_write_kb":63744,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,7],"acting":[0,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":25,"reported_epoch":234,"state":"active+clean","last_fresh":"2024-03-11T21:34:51.297586+0000","last_change":"2024-03-11T21:34:41.149338+0000","last_active":"2024-03-11T21:34:51.297586+0000","last_peered":"2024-03-11T21:34:51.297586+0000","last_clean":"2024-03-11T21:34:51.297586+0000","last_became_active":"2024-03-11T21:34:41.145012+0000","last_became_peered":"2024-03-11T21:34:41.145012+0000","last_unstale":"2024-03-11T21:34:51.297586+0000","last_undegraded":"2024-03-11T21:34:51.297586+0000","last_fullsized":"2024-03-11T21:34:51.297586+0000","mapping_epoch":226,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":227,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:34:22.554606+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:28:37.561826+0000","last_clean_scrub_stamp":"2024-03-11T21:34:22.554606+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:35:37.799908+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,0],"acting":[7,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":374,"reported_epoch":235,"state":"active+clean","last_fresh":"2024-03-11T21:34:52.296285+0000","last_change":"2024-03-11T21:32:23.251510+0000","last_active":"2024-03-11T21:34:52.296285+0000","last_peered":"2024-03-11T21:34:52.296285+0000","last_clean":"2024-03-11T21:34:52.296285+0000","last_became_active":"2024-03-11T21:25:42.574515+0000","last_became_peered":"2024-03-11T21:25:42.574515+0000","last_unstale":"2024-03-11T21:34:52.296285+0000","last_undegraded":"2024-03-11T21:34:52.296285+0000","last_fullsized":"2024-03-11T21:34:52.296285+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:32:23.251386+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:41.544550+0000","last_clean_scrub_stamp":"2024-03-11T21:32:23.251386+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:34:52.399548+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,5],"acting":[2,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.c","version":"0'0","reported_seq":382,"reported_epoch":235,"state":"active+clean","last_fresh":"2024-03-11T21:34:52.295841+0000","last_change":"2024-03-11T21:34:16.385926+0000","last_active":"2024-03-11T21:34:52.295841+0000","last_peered":"2024-03-11T21:34:52.295841+0000","last_clean":"2024-03-11T21:34:52.295841+0000","last_became_active":"2024-03-11T21:25:38.533240+0000","last_became_peered":"2024-03-11T21:25:38.533240+0000","last_unstale":"2024-03-11T21:34:52.295841+0000","last_undegraded":"2024-03-11T21:34:52.295841+0000","last_fullsized":"2024-03-11T21:34:52.295841+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:34:16.385811+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:29:30.279910+0000","last_clean_scrub_stamp":"2024-03-11T21:34:16.385811+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:35:45.125684+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,0],"acting":[2,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.a","version":"109'28","reported_seq":477,"reported_epoch":234,"state":"active+clean","last_fresh":"2024-03-11T21:34:51.298839+0000","last_change":"2024-03-11T21:34:41.169399+0000","last_active":"2024-03-11T21:34:51.298839+0000","last_peered":"2024-03-11T21:34:51.298839+0000","last_clean":"2024-03-11T21:34:51.298839+0000","last_became_active":"2024-03-11T21:34:41.148863+0000","last_became_peered":"2024-03-11T21:34:41.148863+0000","last_unstale":"2024-03-11T21:34:51.298839+0000","last_undegraded":"2024-03-11T21:34:51.298839+0000","last_fullsized":"2024-03-11T21:34:51.298839+0000","mapping_epoch":226,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":227,"parent":"0.0","parent_split_bits":0,"last_scrub":"109'28","last_scrub_stamp":"2024-03-11T21:34:15.438620+0000","last_deep_scrub":"109'28","last_deep_scrub_stamp":"2024-03-11T21:28:24.016389+0000","last_clean_scrub_stamp":"2024-03-11T21:34:15.438620+0000","objects_scrubbed":9,"log_size":28,"log_dups_size":0,"ondisk_log_size":28,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:35:23.047712+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":32,"num_read_kb":21,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,7],"acting":[6,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"8.7","version":"235'8985","reported_seq":9197,"reported_epoch":235,"state":"active+clean","last_fresh":"2024-03-11T21:34:52.378998+0000","last_change":"2024-03-11T21:32:58.092722+0000","last_active":"2024-03-11T21:34:52.378998+0000","last_peered":"2024-03-11T21:34:52.378998+0000","last_clean":"2024-03-11T21:34:52.378998+0000","last_became_active":"2024-03-11T21:32:57.070297+0000","last_became_peered":"2024-03-11T21:32:57.070297+0000","last_unstale":"2024-03-11T21:34:52.378998+0000","last_undegraded":"2024-03-11T21:34:52.378998+0000","last_fullsized":"2024-03-11T21:34:52.378998+0000","mapping_epoch":157,"log_start":"170'3631","ondisk_log_start":"170'3631","created":157,"last_epoch_clean":182,"parent":"0.0","parent_split_bits":0,"last_scrub":"183'5479","last_scrub_stamp":"2024-03-11T21:32:58.091894+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:31:51.219688+0000","last_clean_scrub_stamp":"2024-03-11T21:32:58.091894+0000","objects_scrubbed":2710,"log_size":5354,"log_dups_size":0,"ondisk_log_size":5354,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":724,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":71368704,"num_objects":1089,"num_object_clones":0,"num_object_copies":3267,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1089,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3517,"num_write_kb":60672,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,6],"acting":[1,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":375,"reported_epoch":231,"state":"active+clean","last_fresh":"2024-03-11T21:34:48.226080+0000","last_change":"2024-03-11T21:34:41.161838+0000","last_active":"2024-03-11T21:34:48.226080+0000","last_peered":"2024-03-11T21:34:48.226080+0000","last_clean":"2024-03-11T21:34:48.226080+0000","last_became_active":"2024-03-11T21:34:41.154507+0000","last_became_peered":"2024-03-11T21:34:41.154507+0000","last_unstale":"2024-03-11T21:34:48.226080+0000","last_undegraded":"2024-03-11T21:34:48.226080+0000","last_fullsized":"2024-03-11T21:34:48.226080+0000","mapping_epoch":226,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":227,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:34:10.771846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:27:11.361913+0000","last_clean_scrub_stamp":"2024-03-11T21:34:10.771846+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:35:29.534703+0000","scrub_duration":65,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":374,"reported_epoch":235,"state":"active+clean","last_fresh":"2024-03-11T21:34:52.296447+0000","last_change":"2024-03-11T21:34:17.412793+0000","last_active":"2024-03-11T21:34:52.296447+0000","last_peered":"2024-03-11T21:34:52.296447+0000","last_clean":"2024-03-11T21:34:52.296447+0000","last_became_active":"2024-03-11T21:25:42.576679+0000","last_became_peered":"2024-03-11T21:25:42.576679+0000","last_unstale":"2024-03-11T21:34:52.296447+0000","last_undegraded":"2024-03-11T21:34:52.296447+0000","last_fullsized":"2024-03-11T21:34:52.296447+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:34:17.412687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:41.544550+0000","last_clean_scrub_stamp":"2024-03-11T21:34:17.412687+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:35:37.322196+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,3],"acting":[2,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.d","version":"0'0","reported_seq":383,"reported_epoch":235,"state":"active+clean","last_fresh":"2024-03-11T21:34:52.333602+0000","last_change":"2024-03-11T21:31:55.464241+0000","last_active":"2024-03-11T21:34:52.333602+0000","last_peered":"2024-03-11T21:34:52.333602+0000","last_clean":"2024-03-11T21:34:52.333602+0000","last_became_active":"2024-03-11T21:25:38.539681+0000","last_became_peered":"2024-03-11T21:25:38.539681+0000","last_unstale":"2024-03-11T21:34:52.333602+0000","last_undegraded":"2024-03-11T21:34:52.333602+0000","last_fullsized":"2024-03-11T21:34:52.333602+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:31:55.464145+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:37.512018+0000","last_clean_scrub_stamp":"2024-03-11T21:31:55.464145+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,3],"acting":[1,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.b","version":"74'9","reported_seq":387,"reported_epoch":231,"state":"active+clean","last_fresh":"2024-03-11T21:34:48.239090+0000","last_change":"2024-03-11T21:32:54.624276+0000","last_active":"2024-03-11T21:34:48.239090+0000","last_peered":"2024-03-11T21:34:48.239090+0000","last_clean":"2024-03-11T21:34:48.239090+0000","last_became_active":"2024-03-11T21:25:40.549612+0000","last_became_peered":"2024-03-11T21:25:40.549612+0000","last_unstale":"2024-03-11T21:34:48.239090+0000","last_undegraded":"2024-03-11T21:34:48.239090+0000","last_fullsized":"2024-03-11T21:34:48.239090+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'9","last_scrub_stamp":"2024-03-11T21:32:54.623943+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:39.525554+0000","last_clean_scrub_stamp":"2024-03-11T21:32:54.623943+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.4","version":"234'9067","reported_seq":9276,"reported_epoch":234,"state":"active+clean","last_fresh":"2024-03-11T21:34:51.873457+0000","last_change":"2024-03-11T21:32:59.991354+0000","last_active":"2024-03-11T21:34:51.873457+0000","last_peered":"2024-03-11T21:34:51.873457+0000","last_clean":"2024-03-11T21:34:51.873457+0000","last_became_active":"2024-03-11T21:32:57.087247+0000","last_became_peered":"2024-03-11T21:32:57.087247+0000","last_unstale":"2024-03-11T21:34:51.873457+0000","last_undegraded":"2024-03-11T21:34:51.873457+0000","last_fullsized":"2024-03-11T21:34:51.873457+0000","mapping_epoch":157,"log_start":"169'3172","ondisk_log_start":"169'3172","created":157,"last_epoch_clean":182,"parent":"0.0","parent_split_bits":0,"last_scrub":"185'5515","last_scrub_stamp":"2024-03-11T21:32:59.991099+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:31:51.219688+0000","last_clean_scrub_stamp":"2024-03-11T21:32:59.991099+0000","objects_scrubbed":2803,"log_size":5895,"log_dups_size":0,"ondisk_log_size":5895,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":733,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":74645504,"num_objects":1139,"num_object_clones":0,"num_object_copies":3417,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1139,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3570,"num_write_kb":60992,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,2],"acting":[5,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":273,"reported_epoch":234,"state":"active+clean","last_fresh":"2024-03-11T21:34:51.294369+0000","last_change":"2024-03-11T21:32:18.627466+0000","last_active":"2024-03-11T21:34:51.294369+0000","last_peered":"2024-03-11T21:34:51.294369+0000","last_clean":"2024-03-11T21:34:51.294369+0000","last_became_active":"2024-03-11T21:28:45.372452+0000","last_became_peered":"2024-03-11T21:28:45.372452+0000","last_unstale":"2024-03-11T21:34:51.294369+0000","last_undegraded":"2024-03-11T21:34:51.294369+0000","last_fullsized":"2024-03-11T21:34:51.294369+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:32:18.627290+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:43.559452+0000","last_clean_scrub_stamp":"2024-03-11T21:32:18.627290+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":29,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,3],"acting":[4,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.e","version":"0'0","reported_seq":402,"reported_epoch":235,"state":"active+clean","last_fresh":"2024-03-11T21:34:52.295679+0000","last_change":"2024-03-11T21:34:41.143980+0000","last_active":"2024-03-11T21:34:52.295679+0000","last_peered":"2024-03-11T21:34:52.295679+0000","last_clean":"2024-03-11T21:34:52.295679+0000","last_became_active":"2024-03-11T21:34:41.143532+0000","last_became_peered":"2024-03-11T21:34:41.143532+0000","last_unstale":"2024-03-11T21:34:52.295679+0000","last_undegraded":"2024-03-11T21:34:52.295679+0000","last_fullsized":"2024-03-11T21:34:52.295679+0000","mapping_epoch":226,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":227,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:34:19.476172+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:37.512018+0000","last_clean_scrub_stamp":"2024-03-11T21:34:19.476172+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:35:48.981067+0000","scrub_duration":17,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,7],"acting":[2,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.8","version":"109'22","reported_seq":452,"reported_epoch":234,"state":"active+clean","last_fresh":"2024-03-11T21:34:51.301092+0000","last_change":"2024-03-11T21:34:41.166337+0000","last_active":"2024-03-11T21:34:51.301092+0000","last_peered":"2024-03-11T21:34:51.301092+0000","last_clean":"2024-03-11T21:34:51.301092+0000","last_became_active":"2024-03-11T21:34:41.146631+0000","last_became_peered":"2024-03-11T21:34:41.146631+0000","last_unstale":"2024-03-11T21:34:51.301092+0000","last_undegraded":"2024-03-11T21:34:51.301092+0000","last_fullsized":"2024-03-11T21:34:51.301092+0000","mapping_epoch":226,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":227,"parent":"0.0","parent_split_bits":0,"last_scrub":"109'22","last_scrub_stamp":"2024-03-11T21:32:07.573197+0000","last_deep_scrub":"109'22","last_deep_scrub_stamp":"2024-03-11T21:32:07.573197+0000","last_clean_scrub_stamp":"2024-03-11T21:32:07.573197+0000","objects_scrubbed":7,"log_size":22,"log_dups_size":0,"ondisk_log_size":22,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.9","version":"229'11","reported_seq":31,"reported_epoch":234,"state":"active+clean","last_fresh":"2024-03-11T21:34:51.296536+0000","last_change":"2024-03-11T21:34:41.163190+0000","last_active":"2024-03-11T21:34:51.296536+0000","last_peered":"2024-03-11T21:34:51.296536+0000","last_clean":"2024-03-11T21:34:51.296536+0000","last_became_active":"2024-03-11T21:34:41.156892+0000","last_became_peered":"2024-03-11T21:34:41.156892+0000","last_unstale":"2024-03-11T21:34:51.296536+0000","last_undegraded":"2024-03-11T21:34:51.296536+0000","last_fullsized":"2024-03-11T21:34:51.296536+0000","mapping_epoch":226,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":227,"parent":"0.0","parent_split_bits":0,"last_scrub":"154'10","last_scrub_stamp":"2024-03-11T21:34:19.426261+0000","last_deep_scrub":"94'9","last_deep_scrub_stamp":"2024-03-11T21:26:53.422552+0000","last_clean_scrub_stamp":"2024-03-11T21:34:19.426261+0000","objects_scrubbed":1,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:35:48.765690+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":8,"num_pg":26,"stat_sum":{"num_bytes":2411757592,"num_objects":36802,"num_object_clones":0,"num_object_copies":110406,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":8986,"num_objects_unfound":0,"num_objects_dirty":36802,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":131614,"num_write_kb":3698849,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5554,"num_bytes_recovered":363986944,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8618520576,"data_stored":8618508360,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":139581,"ondisk_log_size":139581,"up":78,"acting":78,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":1531,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":24,"num_read_kb":24,"num_write":10,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":872,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":4593,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":6,"ondisk_log_size":6,"up":96,"acting":96,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":198,"num_read_kb":294,"num_write":179,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":2951840,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":2375680,"data_stored":2361472,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":128,"ondisk_log_size":128,"up":3,"acting":3,"num_store_stats":4},{"poolid":3,"num_pg":3,"stat_sum":{"num_bytes":408,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":547,"num_read_kb":542,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1224,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":8,"ondisk_log_size":8,"up":9,"acting":9,"num_store_stats":7},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":3702,"num_objects":178,"num_object_clones":0,"num_object_copies":534,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":178,"num_whiteouts":0,"num_read":698,"num_read_kb":455,"num_write":417,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":142,"num_bytes_recovered":2530,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":417792,"data_stored":11106,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":472,"ondisk_log_size":472,"up":96,"acting":96,"num_store_stats":8},{"poolid":5,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":74,"ondisk_log_size":74,"up":96,"acting":96,"num_store_stats":8},{"poolid":6,"num_pg":32,"stat_sum":{"num_bytes":383,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":370,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1149,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":96,"acting":96,"num_store_stats":8}],"osd_stats":[{"osd":7,"up_from":60,"seq":257698037874,"num_pgs":68,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":830240,"kb_used_data":618452,"kb_used_omap":1,"kb_used_meta":211646,"kb_avail":92914912,"statfs":{"total":95995035648,"available":95144869888,"internally_reserved":0,"allocated":633294848,"data_stored":632948464,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":216725962},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,0,0,0,0,0,0,16],"upper_bound":2048},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":50,"seq":214748364920,"num_pgs":56,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1296608,"kb_used_data":980820,"kb_used_omap":1,"kb_used_meta":315774,"kb_avail":92448544,"statfs":{"total":95995035648,"available":94667309056,"internally_reserved":0,"allocated":1004359680,"data_stored":1004013613,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":323353030},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1],"upper_bound":2},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574994,"num_pgs":63,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1506548,"kb_used_data":1172712,"kb_used_omap":1,"kb_used_meta":333822,"kb_avail":92238604,"statfs":{"total":95995035648,"available":94452330496,"internally_reserved":0,"allocated":1200857088,"data_stored":1200489971,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":341834181},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,0,1,2],"upper_bound":16},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738518,"num_pgs":66,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1892652,"kb_used_data":1496356,"kb_used_omap":1,"kb_used_meta":396286,"kb_avail":91852500,"statfs":{"total":95995035648,"available":94056960000,"internally_reserved":0,"allocated":1532268544,"data_stored":1531882307,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":405797317},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411468,"num_pgs":51,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1544788,"kb_used_data":1184456,"kb_used_omap":1,"kb_used_meta":360190,"kb_avail":92200364,"statfs":{"total":95995035648,"available":94413172736,"internally_reserved":0,"allocated":1212882944,"data_stored":1212547953,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":368835013},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1],"upper_bound":2},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":3,"up_from":26,"seq":111669149832,"num_pgs":63,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1271924,"kb_used_data":871468,"kb_used_omap":1,"kb_used_meta":400446,"kb_avail":92473228,"statfs":{"total":95995035648,"available":94692585472,"internally_reserved":0,"allocated":892383232,"data_stored":892012463,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":410057159},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":4,"up_from":32,"seq":137438953601,"num_pgs":64,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1280552,"kb_used_data":978524,"kb_used_omap":1,"kb_used_meta":301886,"kb_avail":92464600,"statfs":{"total":95995035648,"available":94683750400,"internally_reserved":0,"allocated":1002008576,"data_stored":1001653365,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":309131717},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,1],"upper_bound":4},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":41,"seq":176093659260,"num_pgs":63,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1477720,"kb_used_data":1179788,"kb_used_omap":1,"kb_used_meta":297726,"kb_avail":92267432,"statfs":{"total":95995035648,"available":94481850368,"internally_reserved":0,"allocated":1208102912,"data_stored":1207763876,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":304871877},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1049,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":46,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":957,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":574,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":436,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1003,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":528,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":1430,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":77824,"data_stored":2008,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":1018,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":1898,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":1474,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":990,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":1034,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":1254,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":370,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":383,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":370,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":1509883904,"data_stored":1509883904,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":1185808384,"data_stored":1185808384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":1196298240,"data_stored":1196294168,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":880214016,"data_stored":880214016,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":993329152,"data_stored":993329152,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":1191317504,"data_stored":1191313432,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":1003032576,"data_stored":1003028504,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":658636800,"data_stored":658636800,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-11T21:34:56.511 DEBUG:teuthology.orchestra.run.smithi092:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid 00db0016-dfed-11ee-95c8-87774f69a715 -- ceph pg dump --format=json 2024-03-11T21:34:56.541 INFO:teuthology.orchestra.run.smithi092.stderr:Inferring config /var/lib/ceph/00db0016-dfed-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T21:34:56.557 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:34:56.569 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:34:56 smithi092 ceph-mon[29572]: pgmap v409: 158 pgs: 1 peering, 2 active+remapped+backfill_toofull, 3 active+remapped+backfilling, 152 active+clean; 2.2 GiB data, 10 GiB used, 705 GiB / 715 GiB avail; 1.0 KiB/s rd, 0 B/s wr, 732 op/s; 7067/109080 objects misplaced (6.479%); 59 MiB/s, 938 objects/s recovering 2024-03-11T21:34:56.569 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:34:56 smithi092 ceph-mon[29572]: osdmap e237: 8 total, 8 up, 8 in 2024-03-11T21:34:56.570 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:34:56 smithi092 ceph-mon[37563]: pgmap v409: 158 pgs: 1 peering, 2 active+remapped+backfill_toofull, 3 active+remapped+backfilling, 152 active+clean; 2.2 GiB data, 10 GiB used, 705 GiB / 715 GiB avail; 1.0 KiB/s rd, 0 B/s wr, 732 op/s; 7067/109080 objects misplaced (6.479%); 59 MiB/s, 938 objects/s recovering 2024-03-11T21:34:56.570 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:34:56 smithi092 ceph-mon[37563]: osdmap e237: 8 total, 8 up, 8 in 2024-03-11T21:34:56.575 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:34:56 smithi097 ceph-mon[33317]: pgmap v409: 158 pgs: 1 peering, 2 active+remapped+backfill_toofull, 3 active+remapped+backfilling, 152 active+clean; 2.2 GiB data, 10 GiB used, 705 GiB / 715 GiB avail; 1.0 KiB/s rd, 0 B/s wr, 732 op/s; 7067/109080 objects misplaced (6.479%); 59 MiB/s, 938 objects/s recovering 2024-03-11T21:34:56.575 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:34:56 smithi097 ceph-mon[33317]: osdmap e237: 8 total, 8 up, 8 in 2024-03-11T21:34:56.619 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:34:56.619 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:34:56.620 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:34:56.667 INFO:teuthology.orchestra.run.smithi092.stderr:Inferring config /var/lib/ceph/00db0016-dfed-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T21:34:56.721 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:34:56.757 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:34:56.758 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:34:56.758 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:34:56.859 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:34:56.895 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:34:56.895 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:34:56.896 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:34:56.997 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:34:57.041 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:34:57.041 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:34:57.042 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:34:57.142 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:34:57.185 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:34:57.186 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:34:57.187 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:34:57.287 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:34:57.331 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:34:57.332 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:34:57.332 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:34:57.433 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:34:57.461 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:34:57.462 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:34:57.462 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:34:57.535 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:34:57 smithi092 ceph-mon[29572]: osdmap e238: 8 total, 8 up, 8 in 2024-03-11T21:34:57.536 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:34:57 smithi092 ceph-mon[37563]: osdmap e238: 8 total, 8 up, 8 in 2024-03-11T21:34:57.563 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:34:57.575 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:34:57 smithi097 ceph-mon[33317]: osdmap e238: 8 total, 8 up, 8 in 2024-03-11T21:34:57.586 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:34:57.586 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:34:57.587 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:34:57.688 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:34:57.711 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:34:57.712 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:34:57.712 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:34:57.812 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:34:57.836 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:34:57.836 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:34:57.837 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:34:57.937 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:34:57.962 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:34:57.963 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:34:57.963 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:34:58.064 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:34:58.088 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:34:58.089 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:34:58.089 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:34:58.110 INFO:journalctl@ceph.mgr.y.smithi092.stdout:Mar 11 21:34:57 smithi092 ceph-00db0016-dfed-11ee-95c8-87774f69a715-mgr-y[29801]: ::ffff:172.21.15.97 - - [11/Mar/2024:21:34:57] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-11T21:34:58.190 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:34:58.214 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:34:58.214 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:34:58.215 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:34:58.316 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:34:58.342 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:34:58.343 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:34:58.343 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:34:58.444 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:34:58.486 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:34:58.487 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:34:58.487 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:34:58.588 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:34:58.610 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:34:58 smithi092 ceph-mon[29572]: pgmap v412: 158 pgs: 1 peering, 2 active+remapped+backfill_toofull, 1 active+clean+remapped, 1 active+remapped+backfilling, 153 active+clean; 2.1 GiB data, 10 GiB used, 705 GiB / 715 GiB avail; 1.0 KiB/s rd, 0 B/s wr, 601 op/s; 4779/106167 objects misplaced (4.501%); 47 MiB/s, 755 objects/s recovering 2024-03-11T21:34:58.610 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:34:58 smithi092 ceph-mon[29572]: osdmap e239: 8 total, 8 up, 8 in 2024-03-11T21:34:58.612 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:34:58 smithi092 ceph-mon[37563]: pgmap v412: 158 pgs: 1 peering, 2 active+remapped+backfill_toofull, 1 active+clean+remapped, 1 active+remapped+backfilling, 153 active+clean; 2.1 GiB data, 10 GiB used, 705 GiB / 715 GiB avail; 1.0 KiB/s rd, 0 B/s wr, 601 op/s; 4779/106167 objects misplaced (4.501%); 47 MiB/s, 755 objects/s recovering 2024-03-11T21:34:58.612 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:34:58 smithi092 ceph-mon[37563]: osdmap e239: 8 total, 8 up, 8 in 2024-03-11T21:34:58.630 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:34:58.630 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:34:58.631 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:34:58.732 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:34:58.770 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:34:58.770 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:34:58.770 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:34:58.826 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:34:58 smithi097 ceph-mon[33317]: pgmap v412: 158 pgs: 1 peering, 2 active+remapped+backfill_toofull, 1 active+clean+remapped, 1 active+remapped+backfilling, 153 active+clean; 2.1 GiB data, 10 GiB used, 705 GiB / 715 GiB avail; 1.0 KiB/s rd, 0 B/s wr, 601 op/s; 4779/106167 objects misplaced (4.501%); 47 MiB/s, 755 objects/s recovering 2024-03-11T21:34:58.826 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:34:58 smithi097 ceph-mon[33317]: osdmap e239: 8 total, 8 up, 8 in 2024-03-11T21:34:58.871 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:34:58.906 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:34:58.907 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:34:58.907 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:34:59.008 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:34:59.046 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:34:59.046 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:34:59.047 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:34:59.148 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:34:59.176 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:34:59.273 INFO:teuthology.orchestra.run.smithi092.stderr:Inferring config /var/lib/ceph/00db0016-dfed-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T21:34:59.274 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:34:59.274 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:34:59.375 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:34:59.395 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:34:59 smithi092 ceph-mon[29572]: osdmap e240: 8 total, 8 up, 8 in 2024-03-11T21:34:59.395 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:34:59 smithi092 ceph-mon[37563]: osdmap e240: 8 total, 8 up, 8 in 2024-03-11T21:34:59.410 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:34:59.411 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:34:59.411 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:34:59.512 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:34:59.551 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:34:59.552 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:34:59.552 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:34:59.653 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:34:59.689 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:34:59.689 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:34:59.690 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:34:59.791 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:34:59.818 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:34:59.819 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:34:59.819 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:34:59.825 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:34:59 smithi097 ceph-mon[33317]: osdmap e240: 8 total, 8 up, 8 in 2024-03-11T21:34:59.920 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:34:59.957 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:34:59.957 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:34:59.958 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:35:00.058 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:35:00.101 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:35:00.101 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:35:00.101 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:35:00.202 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:35:00.242 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:35:00.242 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:35:00.242 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:35:00.343 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:35:00.372 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:35:00.373 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:35:00.373 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:35:00.474 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:35:00.505 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:35:00.506 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:35:00.506 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:35:00.607 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:35:00.610 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:35:00 smithi092 ceph-mon[29572]: pgmap v415: 158 pgs: 2 active+remapped+backfill_toofull, 1 active+clean+remapped, 155 active+clean; 2.0 GiB data, 9.9 GiB used, 705 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 941 op/s; 4089/97788 objects misplaced (4.181%); 38 MiB/s, 605 objects/s recovering 2024-03-11T21:35:00.610 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:35:00 smithi092 ceph-mon[37563]: pgmap v415: 158 pgs: 2 active+remapped+backfill_toofull, 1 active+clean+remapped, 155 active+clean; 2.0 GiB data, 9.9 GiB used, 705 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 941 op/s; 4089/97788 objects misplaced (4.181%); 38 MiB/s, 605 objects/s recovering 2024-03-11T21:35:00.645 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:35:00.645 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:35:00.646 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:35:00.747 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:35:00.796 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:35:00.797 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:35:00.797 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:35:00.826 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:35:00 smithi097 ceph-mon[33317]: pgmap v415: 158 pgs: 2 active+remapped+backfill_toofull, 1 active+clean+remapped, 155 active+clean; 2.0 GiB data, 9.9 GiB used, 705 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 941 op/s; 4089/97788 objects misplaced (4.181%); 38 MiB/s, 605 objects/s recovering 2024-03-11T21:35:00.898 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:35:00.929 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:35:00.930 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:35:00.930 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:35:01.005 INFO:teuthology.orchestra.run.smithi092.stderr:Inferring config /var/lib/ceph/00db0016-dfed-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T21:35:01.031 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:35:01.063 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:35:01.064 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:35:01.064 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:35:01.109 INFO:teuthology.orchestra.run.smithi092.stdout:{ 2024-03-11T21:35:01.109 INFO:teuthology.orchestra.run.smithi092.stdout: "ops": [ 2024-03-11T21:35:01.109 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:35:01.109 INFO:teuthology.orchestra.run.smithi092.stdout: "description": "osd_repop(client.15150.0:212280 8.e e240/157 8:77ad9562:::benchmark_data_smithi092_104865_object91779:head v 240'13403, mlcod=240'13400)", 2024-03-11T21:35:01.109 INFO:teuthology.orchestra.run.smithi092.stdout: "initiated_at": "2024-03-11T21:35:01.121730+0000", 2024-03-11T21:35:01.109 INFO:teuthology.orchestra.run.smithi092.stdout: "age": 0.0042834290000000001, 2024-03-11T21:35:01.109 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.0043134150000000001, 2024-03-11T21:35:01.109 INFO:teuthology.orchestra.run.smithi092.stdout: "type_data": { 2024-03-11T21:35:01.109 INFO:teuthology.orchestra.run.smithi092.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-11T21:35:01.109 INFO:teuthology.orchestra.run.smithi092.stdout: "events": [ 2024-03-11T21:35:01.109 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:35:01.109 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "initiated", 2024-03-11T21:35:01.110 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:35:01.121730+0000", 2024-03-11T21:35:01.110 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-03-11T21:35:01.110 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:35:01.110 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:35:01.110 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "header_read", 2024-03-11T21:35:01.110 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:35:01.121730+0000", 2024-03-11T21:35:01.110 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-03-11T21:35:01.110 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:35:01.110 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:35:01.110 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "throttled", 2024-03-11T21:35:01.111 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:35:01.121732+0000", 2024-03-11T21:35:01.111 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 2.215e-06 2024-03-11T21:35:01.111 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:35:01.111 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:35:01.111 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "all_read", 2024-03-11T21:35:01.111 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:35:01.121742+0000", 2024-03-11T21:35:01.111 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.0380000000000001e-05 2024-03-11T21:35:01.111 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:35:01.111 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:35:01.111 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "dispatched", 2024-03-11T21:35:01.111 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:35:01.121761+0000", 2024-03-11T21:35:01.111 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.8705000000000001e-05 2024-03-11T21:35:01.111 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:35:01.111 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:35:01.111 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "queued_for_pg", 2024-03-11T21:35:01.112 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:35:01.121772+0000", 2024-03-11T21:35:01.112 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.0601999999999999e-05 2024-03-11T21:35:01.112 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:35:01.112 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:35:01.112 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-03-11T21:35:01.112 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:35:01.121957+0000", 2024-03-11T21:35:01.112 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00018589299999999999 2024-03-11T21:35:01.112 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:35:01.112 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:35:01.112 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "started", 2024-03-11T21:35:01.112 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:35:01.122036+0000", 2024-03-11T21:35:01.112 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 7.8959000000000001e-05 2024-03-11T21:35:01.112 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:35:01.112 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:35:01.113 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "commit_sent", 2024-03-11T21:35:01.113 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:35:01.125936+0000", 2024-03-11T21:35:01.113 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.003899876 2024-03-11T21:35:01.113 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-03-11T21:35:01.113 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-03-11T21:35:01.113 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-03-11T21:35:01.113 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:35:01.113 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:35:01.113 INFO:teuthology.orchestra.run.smithi092.stdout: "description": "osd_repop(client.15150.0:212288 8.f e240/157 8:f3ea090c:::benchmark_data_smithi092_104865_object91787:head v 240'13045, mlcod=240'13043)", 2024-03-11T21:35:01.113 INFO:teuthology.orchestra.run.smithi092.stdout: "initiated_at": "2024-03-11T21:35:01.123403+0000", 2024-03-11T21:35:01.113 INFO:teuthology.orchestra.run.smithi092.stdout: "age": 0.002609802, 2024-03-11T21:35:01.113 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.002720456, 2024-03-11T21:35:01.113 INFO:teuthology.orchestra.run.smithi092.stdout: "type_data": { 2024-03-11T21:35:01.113 INFO:teuthology.orchestra.run.smithi092.stdout: "flag_point": "started", 2024-03-11T21:35:01.113 INFO:teuthology.orchestra.run.smithi092.stdout: "events": [ 2024-03-11T21:35:01.114 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:35:01.114 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "initiated", 2024-03-11T21:35:01.114 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:35:01.123403+0000", 2024-03-11T21:35:01.114 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-03-11T21:35:01.114 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:35:01.114 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:35:01.114 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "header_read", 2024-03-11T21:35:01.114 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:35:01.123403+0000", 2024-03-11T21:35:01.114 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-03-11T21:35:01.114 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:35:01.114 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:35:01.114 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "throttled", 2024-03-11T21:35:01.114 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:35:01.123405+0000", 2024-03-11T21:35:01.115 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.66e-06 2024-03-11T21:35:01.115 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:35:01.115 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:35:01.115 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "all_read", 2024-03-11T21:35:01.115 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:35:01.123414+0000", 2024-03-11T21:35:01.115 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 8.9989999999999997e-06 2024-03-11T21:35:01.115 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:35:01.115 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:35:01.115 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "dispatched", 2024-03-11T21:35:01.115 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:35:01.123431+0000", 2024-03-11T21:35:01.115 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.7337999999999999e-05 2024-03-11T21:35:01.115 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:35:01.115 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:35:01.115 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "queued_for_pg", 2024-03-11T21:35:01.116 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:35:01.123440+0000", 2024-03-11T21:35:01.116 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 8.5809999999999997e-06 2024-03-11T21:35:01.116 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:35:01.116 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:35:01.116 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-03-11T21:35:01.116 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:35:01.123559+0000", 2024-03-11T21:35:01.116 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.000118739 2024-03-11T21:35:01.116 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:35:01.116 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:35:01.116 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "started", 2024-03-11T21:35:01.116 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:35:01.123582+0000", 2024-03-11T21:35:01.116 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 2.3281e-05 2024-03-11T21:35:01.116 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-03-11T21:35:01.116 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-03-11T21:35:01.116 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-03-11T21:35:01.117 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:35:01.117 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:35:01.117 INFO:teuthology.orchestra.run.smithi092.stdout: "description": "osd_repop(client.15150.0:212282 8.1 e240/181 8:81ecbdf2:::benchmark_data_smithi092_104865_object91781:head v 240'9330, mlcod=240'9327)", 2024-03-11T21:35:01.117 INFO:teuthology.orchestra.run.smithi092.stdout: "initiated_at": "2024-03-11T21:35:01.125302+0000", 2024-03-11T21:35:01.117 INFO:teuthology.orchestra.run.smithi092.stdout: "age": 0.00071138400000000002, 2024-03-11T21:35:01.117 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00091179300000000004, 2024-03-11T21:35:01.117 INFO:teuthology.orchestra.run.smithi092.stdout: "type_data": { 2024-03-11T21:35:01.117 INFO:teuthology.orchestra.run.smithi092.stdout: "flag_point": "started", 2024-03-11T21:35:01.117 INFO:teuthology.orchestra.run.smithi092.stdout: "events": [ 2024-03-11T21:35:01.117 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:35:01.117 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "initiated", 2024-03-11T21:35:01.117 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:35:01.125302+0000", 2024-03-11T21:35:01.117 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-03-11T21:35:01.117 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:35:01.118 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:35:01.118 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "header_read", 2024-03-11T21:35:01.118 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:35:01.125302+0000", 2024-03-11T21:35:01.118 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-03-11T21:35:01.118 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:35:01.118 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:35:01.118 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "throttled", 2024-03-11T21:35:01.118 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:35:01.125303+0000", 2024-03-11T21:35:01.118 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.4050000000000001e-06 2024-03-11T21:35:01.118 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:35:01.118 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:35:01.118 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "all_read", 2024-03-11T21:35:01.118 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:35:01.125311+0000", 2024-03-11T21:35:01.118 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 7.4989999999999997e-06 2024-03-11T21:35:01.119 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:35:01.119 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:35:01.119 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "dispatched", 2024-03-11T21:35:01.119 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:35:01.125325+0000", 2024-03-11T21:35:01.119 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.4147e-05 2024-03-11T21:35:01.119 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:35:01.119 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:35:01.119 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "queued_for_pg", 2024-03-11T21:35:01.119 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:35:01.125332+0000", 2024-03-11T21:35:01.119 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 7.3209999999999999e-06 2024-03-11T21:35:01.119 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:35:01.119 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:35:01.119 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-03-11T21:35:01.120 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:35:01.125403+0000", 2024-03-11T21:35:01.120 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 7.1357000000000001e-05 2024-03-11T21:35:01.120 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:35:01.120 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:35:01.120 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "started", 2024-03-11T21:35:01.120 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:35:01.125423+0000", 2024-03-11T21:35:01.120 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 2.0081e-05 2024-03-11T21:35:01.120 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-03-11T21:35:01.120 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-03-11T21:35:01.120 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-03-11T21:35:01.120 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-03-11T21:35:01.120 INFO:teuthology.orchestra.run.smithi092.stdout: ], 2024-03-11T21:35:01.120 INFO:teuthology.orchestra.run.smithi092.stdout: "num_ops": 3 2024-03-11T21:35:01.120 INFO:teuthology.orchestra.run.smithi092.stdout:} 2024-03-11T21:35:01.165 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:35:01.209 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:35:01.210 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:35:01.210 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:35:01.310 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:35:01.347 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:35:01.348 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:35:01.348 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:35:01.449 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:35:01.481 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:35:01.481 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:35:01.482 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:35:01.582 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:35:01.621 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:35:01.621 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:35:01.621 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:35:01.722 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:35:01.759 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:35:01.760 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:35:01.760 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:35:01.861 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:35:01.894 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:35:01.894 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:35:01.895 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:35:01.995 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:35:02.032 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:35:02.032 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:35:02.032 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:35:02.133 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:35:02.166 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:35:02.167 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:35:02.167 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:35:02.268 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:35:02.322 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:35:02.322 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:35:02.322 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:35:02.423 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:35:02.463 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:35:02.464 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:35:02.464 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:35:02.565 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:35:02.603 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:35:02.603 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:35:02.604 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:35:02.609 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:35:02 smithi092 ceph-mon[29572]: pgmap v416: 158 pgs: 2 active+remapped+backfill_toofull, 1 active+clean+remapped, 155 active+clean; 2.0 GiB data, 9.9 GiB used, 705 GiB / 715 GiB avail; 876 B/s rd, 0 B/s wr, 645 op/s; 4089/97788 objects misplaced (4.181%); 26 MiB/s, 414 objects/s recovering 2024-03-11T21:35:02.610 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:35:02 smithi092 ceph-mon[37563]: pgmap v416: 158 pgs: 2 active+remapped+backfill_toofull, 1 active+clean+remapped, 155 active+clean; 2.0 GiB data, 9.9 GiB used, 705 GiB / 715 GiB avail; 876 B/s rd, 0 B/s wr, 645 op/s; 4089/97788 objects misplaced (4.181%); 26 MiB/s, 414 objects/s recovering 2024-03-11T21:35:02.704 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:35:02.744 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:35:02.745 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:35:02.745 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:35:02.826 INFO:journalctl@ceph.iscsi.iscsi.a.smithi097.stdout:Mar 11 21:35:02 smithi097 ceph-00db0016-dfed-11ee-95c8-87774f69a715-iscsi-iscsi-a[60943]: debug there is no tcmu-runner data available 2024-03-11T21:35:02.827 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:35:02 smithi097 ceph-mon[33317]: pgmap v416: 158 pgs: 2 active+remapped+backfill_toofull, 1 active+clean+remapped, 155 active+clean; 2.0 GiB data, 9.9 GiB used, 705 GiB / 715 GiB avail; 876 B/s rd, 0 B/s wr, 645 op/s; 4089/97788 objects misplaced (4.181%); 26 MiB/s, 414 objects/s recovering 2024-03-11T21:35:02.846 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:35:02.902 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:35:02.902 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:35:02.903 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:35:02.906 DEBUG:teuthology.orchestra.run.smithi092:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid 00db0016-dfed-11ee-95c8-87774f69a715 -- ceph daemon osd.2 dump_blocked_ops 2024-03-11T21:35:03.004 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:35:03.031 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:35:03.032 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:35:03.032 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:35:03.133 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:35:03.191 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:35:03.194 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:35:03.195 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:35:03.296 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:35:03.325 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:35:03.326 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:35:03.326 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:35:03.398 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:35:03 smithi092 ceph-mon[29572]: from='client.24469 v1:172.21.15.97:0/404208310' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:35:03.398 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:35:03 smithi092 ceph-mon[37563]: from='client.24469 v1:172.21.15.97:0/404208310' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:35:03.427 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:35:03.454 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:35:03.454 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:35:03.455 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:35:03.556 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:35:03.597 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:35:03.598 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:35:03.599 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:35:03.699 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:35:03.746 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:35:03.747 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:35:03.747 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:35:03.797 INFO:teuthology.orchestra.run.smithi092.stdout:osd.0: {} 2024-03-11T21:35:03.810 INFO:teuthology.orchestra.run.smithi092.stdout:osd.1: {} 2024-03-11T21:35:03.825 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:35:03 smithi097 ceph-mon[33317]: from='client.24469 v1:172.21.15.97:0/404208310' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:35:03.848 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:35:03.857 INFO:teuthology.orchestra.run.smithi092.stdout:osd.2: {} 2024-03-11T21:35:03.875 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:35:03.875 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:35:03.876 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:35:03.916 INFO:teuthology.orchestra.run.smithi092.stdout:osd.3: {} 2024-03-11T21:35:03.947 INFO:teuthology.orchestra.run.smithi092.stdout:osd.4: {} 2024-03-11T21:35:03.966 INFO:teuthology.orchestra.run.smithi092.stdout:osd.5: {} 2024-03-11T21:35:03.976 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:35:03.986 INFO:teuthology.orchestra.run.smithi092.stdout:osd.6: {} 2024-03-11T21:35:04.001 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:35:04.002 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:35:04.002 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:35:04.019 INFO:teuthology.orchestra.run.smithi092.stdout:osd.7: {} 2024-03-11T21:35:04.026 INFO:teuthology.orchestra.run.smithi092.stderr:osd.0: osd_enable_op_tracker = '' 2024-03-11T21:35:04.027 INFO:teuthology.orchestra.run.smithi092.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-11T21:35:04.027 INFO:teuthology.orchestra.run.smithi092.stderr:osd.2: osd_enable_op_tracker = '' 2024-03-11T21:35:04.027 INFO:teuthology.orchestra.run.smithi092.stderr:osd.3: osd_enable_op_tracker = '' 2024-03-11T21:35:04.027 INFO:teuthology.orchestra.run.smithi092.stderr:osd.4: osd_enable_op_tracker = '' 2024-03-11T21:35:04.027 INFO:teuthology.orchestra.run.smithi092.stderr:osd.5: osd_enable_op_tracker = '' 2024-03-11T21:35:04.027 INFO:teuthology.orchestra.run.smithi092.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-11T21:35:04.027 INFO:teuthology.orchestra.run.smithi092.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-11T21:35:04.104 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:35:04.136 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:35:04.136 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:35:04.137 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:35:04.237 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:35:04.273 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:35:04.273 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:35:04.273 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:35:04.374 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:35:04.408 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:35:04.409 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:35:04.409 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:35:04.510 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:35:04.545 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:35:04.546 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:35:04.547 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:35:04.647 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:35:04.688 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:35:04.689 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:35:04.690 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:35:04.715 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:35:04 smithi092 ceph-mon[29572]: pgmap v417: 158 pgs: 2 active+remapped+backfill_toofull, 156 active+clean; 1.7 GiB data, 8.8 GiB used, 706 GiB / 715 GiB avail; 1.5 KiB/s rd, 0 B/s wr, 1.20k op/s; 3934/84507 objects misplaced (4.655%); 22 MiB/s, 354 objects/s recovering 2024-03-11T21:35:04.716 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:35:04 smithi092 ceph-mon[29572]: from='client.? v1:172.21.15.92:0/2862964706' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-11T21:35:04.716 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:35:04 smithi092 ceph-mon[29572]: from='client.? v1:172.21.15.92:0/3591126689' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-11T21:35:04.716 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:35:04 smithi092 ceph-mon[37563]: pgmap v417: 158 pgs: 2 active+remapped+backfill_toofull, 156 active+clean; 1.7 GiB data, 8.8 GiB used, 706 GiB / 715 GiB avail; 1.5 KiB/s rd, 0 B/s wr, 1.20k op/s; 3934/84507 objects misplaced (4.655%); 22 MiB/s, 354 objects/s recovering 2024-03-11T21:35:04.716 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:35:04 smithi092 ceph-mon[37563]: from='client.? v1:172.21.15.92:0/2862964706' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-11T21:35:04.716 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:35:04 smithi092 ceph-mon[37563]: from='client.? v1:172.21.15.92:0/3591126689' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-11T21:35:04.791 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:35:04.825 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:35:04 smithi097 ceph-mon[33317]: pgmap v417: 158 pgs: 2 active+remapped+backfill_toofull, 156 active+clean; 1.7 GiB data, 8.8 GiB used, 706 GiB / 715 GiB avail; 1.5 KiB/s rd, 0 B/s wr, 1.20k op/s; 3934/84507 objects misplaced (4.655%); 22 MiB/s, 354 objects/s recovering 2024-03-11T21:35:04.825 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:35:04 smithi097 ceph-mon[33317]: from='client.? v1:172.21.15.92:0/2862964706' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-11T21:35:04.826 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:35:04 smithi097 ceph-mon[33317]: from='client.? v1:172.21.15.92:0/3591126689' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-11T21:35:04.833 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:35:04.833 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:35:04.834 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:35:04.934 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:35:04.969 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:35:04.970 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:35:04.970 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:35:05.071 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:35:05.132 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:35:05.132 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:35:05.132 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:35:05.233 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:35:05.272 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:35:05.273 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:35:05.273 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:35:05.374 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:35:05.417 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:35:05.417 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:35:05.417 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:35:05.518 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:35:05.560 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:35:05.561 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:35:05.561 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:35:05.610 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:35:05 smithi092 ceph-mon[29572]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-11T21:35:05.610 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:35:05 smithi092 ceph-mon[29572]: from='client.? v1:172.21.15.92:0/3591126689' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-03-11T21:35:05.610 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:35:05 smithi092 ceph-mon[29572]: osdmap e241: 8 total, 8 up, 8 in 2024-03-11T21:35:05.610 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:35:05 smithi092 ceph-mon[29572]: 8.d scrub starts 2024-03-11T21:35:05.610 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:35:05 smithi092 ceph-mon[29572]: 6.0 scrub starts 2024-03-11T21:35:05.610 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:35:05 smithi092 ceph-mon[29572]: 8.d scrub ok 2024-03-11T21:35:05.610 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:35:05 smithi092 ceph-mon[37563]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-11T21:35:05.610 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:35:05 smithi092 ceph-mon[37563]: from='client.? v1:172.21.15.92:0/3591126689' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-03-11T21:35:05.611 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:35:05 smithi092 ceph-mon[37563]: osdmap e241: 8 total, 8 up, 8 in 2024-03-11T21:35:05.611 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:35:05 smithi092 ceph-mon[37563]: 8.d scrub starts 2024-03-11T21:35:05.611 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:35:05 smithi092 ceph-mon[37563]: 6.0 scrub starts 2024-03-11T21:35:05.611 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:35:05 smithi092 ceph-mon[37563]: 8.d scrub ok 2024-03-11T21:35:05.662 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:35:05.694 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:35:05.694 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:35:05.694 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:35:05.795 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:35:05.825 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:35:05 smithi097 ceph-mon[33317]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-11T21:35:05.826 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:35:05 smithi097 ceph-mon[33317]: from='client.? v1:172.21.15.92:0/3591126689' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-03-11T21:35:05.826 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:35:05 smithi097 ceph-mon[33317]: osdmap e241: 8 total, 8 up, 8 in 2024-03-11T21:35:05.826 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:35:05 smithi097 ceph-mon[33317]: 8.d scrub starts 2024-03-11T21:35:05.826 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:35:05 smithi097 ceph-mon[33317]: 6.0 scrub starts 2024-03-11T21:35:05.826 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:35:05 smithi097 ceph-mon[33317]: 8.d scrub ok 2024-03-11T21:35:05.837 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:35:05.837 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:35:05.838 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:35:05.938 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:35:05.962 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:35:05.963 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:35:05.963 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:35:06.064 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:35:06.098 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:35:06.099 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:35:06.099 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:35:06.200 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:35:06.224 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:35:06.224 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:35:06.225 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:35:06.325 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:35:06.357 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:35:06.358 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:35:06.358 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:35:06.459 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:35:06.497 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:35:06.497 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:35:06.498 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:35:06.599 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:35:06.636 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:35:06.637 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:35:06.637 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:35:06.738 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:35:06.767 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:35:06.768 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:35:06.768 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:35:06.807 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:35:06 smithi092 ceph-mon[29572]: 6.0 scrub ok 2024-03-11T21:35:06.807 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:35:06 smithi092 ceph-mon[29572]: pgmap v419: 158 pgs: 2 active+remapped+backfill_toofull, 156 active+clean; 1.6 GiB data, 8.7 GiB used, 707 GiB / 715 GiB avail; 1.3 KiB/s rd, 0 B/s wr, 969 op/s; 3358/81252 objects misplaced (4.133%); 2.0 MiB/s, 31 objects/s recovering 2024-03-11T21:35:06.807 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:35:06 smithi092 ceph-mon[29572]: 4.1a scrub starts 2024-03-11T21:35:06.807 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:35:06 smithi092 ceph-mon[29572]: 4.1a scrub ok 2024-03-11T21:35:06.807 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:35:06 smithi092 ceph-mon[29572]: 2.17 scrub starts 2024-03-11T21:35:06.807 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:35:06 smithi092 ceph-mon[29572]: 2.17 scrub ok 2024-03-11T21:35:06.807 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:35:06 smithi092 ceph-mon[37563]: 6.0 scrub ok 2024-03-11T21:35:06.807 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:35:06 smithi092 ceph-mon[37563]: pgmap v419: 158 pgs: 2 active+remapped+backfill_toofull, 156 active+clean; 1.6 GiB data, 8.7 GiB used, 707 GiB / 715 GiB avail; 1.3 KiB/s rd, 0 B/s wr, 969 op/s; 3358/81252 objects misplaced (4.133%); 2.0 MiB/s, 31 objects/s recovering 2024-03-11T21:35:06.807 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:35:06 smithi092 ceph-mon[37563]: 4.1a scrub starts 2024-03-11T21:35:06.807 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:35:06 smithi092 ceph-mon[37563]: 4.1a scrub ok 2024-03-11T21:35:06.807 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:35:06 smithi092 ceph-mon[37563]: 2.17 scrub starts 2024-03-11T21:35:06.808 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:35:06 smithi092 ceph-mon[37563]: 2.17 scrub ok 2024-03-11T21:35:06.825 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:35:06 smithi097 ceph-mon[33317]: 6.0 scrub ok 2024-03-11T21:35:06.825 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:35:06 smithi097 ceph-mon[33317]: pgmap v419: 158 pgs: 2 active+remapped+backfill_toofull, 156 active+clean; 1.6 GiB data, 8.7 GiB used, 707 GiB / 715 GiB avail; 1.3 KiB/s rd, 0 B/s wr, 969 op/s; 3358/81252 objects misplaced (4.133%); 2.0 MiB/s, 31 objects/s recovering 2024-03-11T21:35:06.825 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:35:06 smithi097 ceph-mon[33317]: 4.1a scrub starts 2024-03-11T21:35:06.826 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:35:06 smithi097 ceph-mon[33317]: 4.1a scrub ok 2024-03-11T21:35:06.826 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:35:06 smithi097 ceph-mon[33317]: 2.17 scrub starts 2024-03-11T21:35:06.826 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:35:06 smithi097 ceph-mon[33317]: 2.17 scrub ok 2024-03-11T21:35:06.869 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:35:06.902 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:35:06.902 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:35:06.903 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:35:07.003 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:35:07.053 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:35:07.056 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:35:07.056 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:35:07.140 INFO:teuthology.orchestra.run.smithi092.stderr:noscrub is unset 2024-03-11T21:35:07.157 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:35:07.201 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:35:07.202 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:35:07.202 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:35:07.303 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:35:07.340 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:35:07.341 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:35:07.341 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:35:07.442 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:35:07.480 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:35:07.481 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:35:07.481 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:35:07.582 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:35:07.620 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:35:07.620 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:35:07.621 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:35:07.722 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:35:07.757 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:35:07.758 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:35:07.758 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:35:07.825 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:35:07 smithi097 ceph-mon[33317]: 8.c scrub starts 2024-03-11T21:35:07.826 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:35:07 smithi097 ceph-mon[33317]: 6.1b scrub starts 2024-03-11T21:35:07.826 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:35:07 smithi097 ceph-mon[33317]: 6.1b scrub ok 2024-03-11T21:35:07.826 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:35:07 smithi097 ceph-mon[33317]: from='mgr.14607 v1:172.21.15.92:0/1702717597' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T21:35:07.859 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:35:07.860 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:35:07 smithi092 ceph-mon[29572]: 8.c scrub starts 2024-03-11T21:35:07.860 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:35:07 smithi092 ceph-mon[29572]: 6.1b scrub starts 2024-03-11T21:35:07.860 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:35:07 smithi092 ceph-mon[29572]: 6.1b scrub ok 2024-03-11T21:35:07.860 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:35:07 smithi092 ceph-mon[29572]: from='mgr.14607 v1:172.21.15.92:0/1702717597' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T21:35:07.861 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:35:07 smithi092 ceph-mon[37563]: 8.c scrub starts 2024-03-11T21:35:07.861 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:35:07 smithi092 ceph-mon[37563]: 6.1b scrub starts 2024-03-11T21:35:07.861 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:35:07 smithi092 ceph-mon[37563]: 6.1b scrub ok 2024-03-11T21:35:07.861 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:35:07 smithi092 ceph-mon[37563]: from='mgr.14607 v1:172.21.15.92:0/1702717597' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T21:35:07.893 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:35:07.894 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:35:07.894 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:35:07.995 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:35:08.051 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:35:08.051 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:35:08.051 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:35:08.152 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:35:08.200 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:35:08.200 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:35:08.201 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:35:08.302 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:35:08.343 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:35:08.344 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:35:08.344 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:35:08.360 INFO:journalctl@ceph.mgr.y.smithi092.stdout:Mar 11 21:35:07 smithi092 ceph-00db0016-dfed-11ee-95c8-87774f69a715-mgr-y[29801]: ::ffff:172.21.15.97 - - [11/Mar/2024:21:35:07] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-11T21:35:08.445 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:35:08.486 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:35:08.487 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:35:08.487 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:35:08.516 INFO:teuthology.orchestra.run.smithi092.stdout: 2024-03-11T21:35:08.518 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:35:08 smithi092 ceph-mon[29572]: 8.c scrub ok 2024-03-11T21:35:08.518 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:35:08 smithi092 ceph-mon[29572]: pgmap v420: 158 pgs: 2 active+remapped+backfill_toofull, 156 active+clean; 1.6 GiB data, 8.4 GiB used, 707 GiB / 715 GiB avail; 580 B/s rd, 0 B/s wr, 789 op/s; 3358/76923 objects misplaced (4.365%) 2024-03-11T21:35:08.518 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:35:08 smithi092 ceph-mon[29572]: purged_snaps scrub starts 2024-03-11T21:35:08.518 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:35:08 smithi092 ceph-mon[29572]: purged_snaps scrub ok 2024-03-11T21:35:08.518 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:35:08 smithi092 ceph-mon[29572]: osdmap e242: 8 total, 8 up, 8 in 2024-03-11T21:35:08.518 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:35:08 smithi092 ceph-mon[29572]: 5.d scrub starts 2024-03-11T21:35:08.518 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:35:08 smithi092 ceph-mon[29572]: 5.d scrub ok 2024-03-11T21:35:08.518 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:35:08 smithi092 ceph-mon[29572]: 4.1d scrub starts 2024-03-11T21:35:08.518 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:35:08 smithi092 ceph-mon[29572]: 4.1d scrub ok 2024-03-11T21:35:08.518 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:35:08 smithi092 ceph-mon[29572]: 2.11 scrub starts 2024-03-11T21:35:08.518 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:35:08 smithi092 ceph-mon[29572]: 2.11 scrub ok 2024-03-11T21:35:08.519 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:35:08 smithi092 ceph-mon[37563]: 8.c scrub ok 2024-03-11T21:35:08.519 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:35:08 smithi092 ceph-mon[37563]: pgmap v420: 158 pgs: 2 active+remapped+backfill_toofull, 156 active+clean; 1.6 GiB data, 8.4 GiB used, 707 GiB / 715 GiB avail; 580 B/s rd, 0 B/s wr, 789 op/s; 3358/76923 objects misplaced (4.365%) 2024-03-11T21:35:08.519 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:35:08 smithi092 ceph-mon[37563]: purged_snaps scrub starts 2024-03-11T21:35:08.519 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:35:08 smithi092 ceph-mon[37563]: purged_snaps scrub ok 2024-03-11T21:35:08.519 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:35:08 smithi092 ceph-mon[37563]: osdmap e242: 8 total, 8 up, 8 in 2024-03-11T21:35:08.519 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:35:08 smithi092 ceph-mon[37563]: 5.d scrub starts 2024-03-11T21:35:08.520 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:35:08 smithi092 ceph-mon[37563]: 5.d scrub ok 2024-03-11T21:35:08.520 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:35:08 smithi092 ceph-mon[37563]: 4.1d scrub starts 2024-03-11T21:35:08.520 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:35:08 smithi092 ceph-mon[37563]: 4.1d scrub ok 2024-03-11T21:35:08.520 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:35:08 smithi092 ceph-mon[37563]: 2.11 scrub starts 2024-03-11T21:35:08.520 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:35:08 smithi092 ceph-mon[37563]: 2.11 scrub ok 2024-03-11T21:35:08.529 INFO:teuthology.orchestra.run.smithi092.stderr:dumped all 2024-03-11T21:35:08.588 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:35:08.622 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:35:08.622 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:35:08.623 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:35:08.723 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:35:08.761 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:35:08.762 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:35:08.762 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:35:08.826 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:35:08 smithi097 ceph-mon[33317]: 8.c scrub ok 2024-03-11T21:35:08.826 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:35:08 smithi097 ceph-mon[33317]: pgmap v420: 158 pgs: 2 active+remapped+backfill_toofull, 156 active+clean; 1.6 GiB data, 8.4 GiB used, 707 GiB / 715 GiB avail; 580 B/s rd, 0 B/s wr, 789 op/s; 3358/76923 objects misplaced (4.365%) 2024-03-11T21:35:08.826 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:35:08 smithi097 ceph-mon[33317]: purged_snaps scrub starts 2024-03-11T21:35:08.826 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:35:08 smithi097 ceph-mon[33317]: purged_snaps scrub ok 2024-03-11T21:35:08.826 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:35:08 smithi097 ceph-mon[33317]: osdmap e242: 8 total, 8 up, 8 in 2024-03-11T21:35:08.826 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:35:08 smithi097 ceph-mon[33317]: 5.d scrub starts 2024-03-11T21:35:08.826 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:35:08 smithi097 ceph-mon[33317]: 5.d scrub ok 2024-03-11T21:35:08.826 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:35:08 smithi097 ceph-mon[33317]: 4.1d scrub starts 2024-03-11T21:35:08.826 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:35:08 smithi097 ceph-mon[33317]: 4.1d scrub ok 2024-03-11T21:35:08.826 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:35:08 smithi097 ceph-mon[33317]: 2.11 scrub starts 2024-03-11T21:35:08.826 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:35:08 smithi097 ceph-mon[33317]: 2.11 scrub ok 2024-03-11T21:35:08.863 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:35:08.901 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:35:08.901 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:35:08.902 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:35:09.002 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:35:09.036 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:35:09.036 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:35:09.037 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:35:09.137 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:35:09.172 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:35:09.172 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:35:09.173 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:35:09.256 INFO:teuthology.orchestra.run.smithi092.stdout:{"pg_ready":true,"pg_map":{"version":421,"stamp":"2024-03-11T21:35:07.414057+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":1597938112,"num_objects":24573,"num_object_clones":0,"num_object_copies":73719,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":3172,"num_objects_unfound":0,"num_objects_dirty":24573,"num_whiteouts":0,"num_read":1478,"num_read_kb":1326,"num_write":144656,"num_write_kb":3701310,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9644,"num_bytes_recovered":624433524,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":139569,"ondisk_log_size":139569,"up":474,"acting":474,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":493,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":749961216,"kb_used":8398484,"kb_used_data":5680584,"kb_used_omap":12,"kb_used_meta":2717683,"kb_avail":741562732,"statfs":{"total":767960285184,"available":759360237568,"internally_reserved":0,"allocated":5816918016,"data_stored":5813825123,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12752,"internal_metadata":2782907952},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[3,4,3,3,11,1],"upper_bound":64},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-525795328,"num_objects":-8023,"num_object_clones":0,"num_object_copies":-24069,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-917,"num_objects_unfound":0,"num_objects_dirty":-8023,"num_whiteouts":0,"num_read":5,"num_read_kb":5,"num_write":8023,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.250997"},"pg_stats":[{"pgid":"8.15","version":"241'9630","reported_seq":20028,"reported_epoch":241,"state":"active+clean","last_fresh":"2024-03-11T21:35:07.127365+0000","last_change":"2024-03-11T21:34:03.402085+0000","last_active":"2024-03-11T21:35:07.127365+0000","last_peered":"2024-03-11T21:35:07.127365+0000","last_clean":"2024-03-11T21:35:07.127365+0000","last_became_active":"2024-03-11T21:34:02.653146+0000","last_became_peered":"2024-03-11T21:34:02.653146+0000","last_unstale":"2024-03-11T21:35:07.127365+0000","last_undegraded":"2024-03-11T21:35:07.127365+0000","last_fullsized":"2024-03-11T21:35:07.127365+0000","mapping_epoch":213,"log_start":"176'4877","ondisk_log_start":"176'4877","created":181,"last_epoch_clean":214,"parent":"0.0","parent_split_bits":5,"last_scrub":"215'7489","last_scrub_stamp":"2024-03-11T21:34:03.401974+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:31:51.219688+0000","last_clean_scrub_stamp":"2024-03-11T21:34:03.401974+0000","objects_scrubbed":2801,"log_size":4753,"log_dups_size":0,"ondisk_log_size":4753,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:35:33.221837+0000","scrub_duration":598,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":42860544,"num_objects":654,"num_object_clones":0,"num_object_copies":1962,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":654,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2147,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,6],"acting":[7,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.1b","version":"0'0","reported_seq":383,"reported_epoch":240,"state":"active+clean","last_fresh":"2024-03-11T21:34:58.385841+0000","last_change":"2024-03-11T21:34:41.149143+0000","last_active":"2024-03-11T21:34:58.385841+0000","last_peered":"2024-03-11T21:34:58.385841+0000","last_clean":"2024-03-11T21:34:58.385841+0000","last_became_active":"2024-03-11T21:34:41.145634+0000","last_became_peered":"2024-03-11T21:34:41.145634+0000","last_unstale":"2024-03-11T21:34:58.385841+0000","last_undegraded":"2024-03-11T21:34:58.385841+0000","last_fullsized":"2024-03-11T21:34:58.385841+0000","mapping_epoch":226,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":227,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:31:06.310490+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:43.559452+0000","last_clean_scrub_stamp":"2024-03-11T21:31:06.310490+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:35:01.869912+0000","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":386,"reported_epoch":241,"state":"active+clean","last_fresh":"2024-03-11T21:35:04.388467+0000","last_change":"2024-03-11T21:34:15.738366+0000","last_active":"2024-03-11T21:35:04.388467+0000","last_peered":"2024-03-11T21:35:04.388467+0000","last_clean":"2024-03-11T21:35:04.388467+0000","last_became_active":"2024-03-11T21:25:42.578893+0000","last_became_peered":"2024-03-11T21:25:42.578893+0000","last_unstale":"2024-03-11T21:35:04.388467+0000","last_undegraded":"2024-03-11T21:35:04.388467+0000","last_fullsized":"2024-03-11T21:35:04.388467+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:34:15.738139+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:41.544550+0000","last_clean_scrub_stamp":"2024-03-11T21:34:15.738139+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:35:19.004355+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1f","version":"0'0","reported_seq":413,"reported_epoch":240,"state":"active+clean","last_fresh":"2024-03-11T21:34:58.350359+0000","last_change":"2024-03-11T21:34:41.145243+0000","last_active":"2024-03-11T21:34:58.350359+0000","last_peered":"2024-03-11T21:34:58.350359+0000","last_clean":"2024-03-11T21:34:58.350359+0000","last_became_active":"2024-03-11T21:34:41.144945+0000","last_became_peered":"2024-03-11T21:34:41.144945+0000","last_unstale":"2024-03-11T21:34:58.350359+0000","last_undegraded":"2024-03-11T21:34:58.350359+0000","last_fullsized":"2024-03-11T21:34:58.350359+0000","mapping_epoch":226,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":227,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:34:28.637947+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:37.512018+0000","last_clean_scrub_stamp":"2024-03-11T21:34:28.637947+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:35:53.709415+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,4],"acting":[0,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.19","version":"74'15","reported_seq":404,"reported_epoch":240,"state":"active+clean","last_fresh":"2024-03-11T21:34:58.385628+0000","last_change":"2024-03-11T21:31:07.311590+0000","last_active":"2024-03-11T21:34:58.385628+0000","last_peered":"2024-03-11T21:34:58.385628+0000","last_clean":"2024-03-11T21:34:58.385628+0000","last_became_active":"2024-03-11T21:25:40.554207+0000","last_became_peered":"2024-03-11T21:25:40.554207+0000","last_unstale":"2024-03-11T21:34:58.385628+0000","last_undegraded":"2024-03-11T21:34:58.385628+0000","last_fullsized":"2024-03-11T21:34:58.385628+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'15","last_scrub_stamp":"2024-03-11T21:31:07.311470+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:39.525554+0000","last_clean_scrub_stamp":"2024-03-11T21:31:07.311470+0000","objects_scrubbed":7,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:35:01.870112+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.14","version":"241'9459","reported_seq":16129,"reported_epoch":241,"state":"active+clean","last_fresh":"2024-03-11T21:35:07.062870+0000","last_change":"2024-03-11T21:34:11.342445+0000","last_active":"2024-03-11T21:35:07.062870+0000","last_peered":"2024-03-11T21:35:07.062870+0000","last_clean":"2024-03-11T21:35:07.062870+0000","last_became_active":"2024-03-11T21:33:32.391932+0000","last_became_peered":"2024-03-11T21:33:32.391932+0000","last_unstale":"2024-03-11T21:35:07.062870+0000","last_undegraded":"2024-03-11T21:35:07.062870+0000","last_fullsized":"2024-03-11T21:35:07.062870+0000","mapping_epoch":198,"log_start":"172'4093","ondisk_log_start":"172'4093","created":181,"last_epoch_clean":199,"parent":"0.0","parent_split_bits":5,"last_scrub":"219'7635","last_scrub_stamp":"2024-03-11T21:34:11.342073+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:31:51.219688+0000","last_clean_scrub_stamp":"2024-03-11T21:34:11.342073+0000","objects_scrubbed":2502,"log_size":5366,"log_dups_size":0,"ondisk_log_size":5366,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:35:36.963800+0000","scrub_duration":604,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":43581440,"num_objects":665,"num_object_clones":0,"num_object_copies":1995,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":665,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1837,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,2],"acting":[4,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1a","version":"0'0","reported_seq":374,"reported_epoch":241,"state":"active+clean","last_fresh":"2024-03-11T21:35:04.388513+0000","last_change":"2024-03-11T21:32:01.718632+0000","last_active":"2024-03-11T21:35:04.388513+0000","last_peered":"2024-03-11T21:35:04.388513+0000","last_clean":"2024-03-11T21:35:04.388513+0000","last_became_active":"2024-03-11T21:25:44.590923+0000","last_became_peered":"2024-03-11T21:25:44.590923+0000","last_unstale":"2024-03-11T21:35:04.388513+0000","last_undegraded":"2024-03-11T21:35:04.388513+0000","last_fullsized":"2024-03-11T21:35:04.388513+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:32:01.718486+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:43.559452+0000","last_clean_scrub_stamp":"2024-03-11T21:32:01.718486+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:35:04.446595+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,1],"acting":[4,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":396,"reported_epoch":240,"state":"active+clean","last_fresh":"2024-03-11T21:34:58.347528+0000","last_change":"2024-03-11T21:34:41.154531+0000","last_active":"2024-03-11T21:34:58.347528+0000","last_peered":"2024-03-11T21:34:58.347528+0000","last_clean":"2024-03-11T21:34:58.347528+0000","last_became_active":"2024-03-11T21:34:41.154216+0000","last_became_peered":"2024-03-11T21:34:41.154216+0000","last_unstale":"2024-03-11T21:34:58.347528+0000","last_undegraded":"2024-03-11T21:34:58.347528+0000","last_fullsized":"2024-03-11T21:34:58.347528+0000","mapping_epoch":226,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":227,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:34:30.609917+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:41.544550+0000","last_clean_scrub_stamp":"2024-03-11T21:34:30.609917+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:35:42.029275+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,7],"acting":[1,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1e","version":"0'0","reported_seq":392,"reported_epoch":240,"state":"active+clean","last_fresh":"2024-03-11T21:34:58.386484+0000","last_change":"2024-03-11T21:32:18.578157+0000","last_active":"2024-03-11T21:34:58.386484+0000","last_peered":"2024-03-11T21:34:58.386484+0000","last_clean":"2024-03-11T21:34:58.386484+0000","last_became_active":"2024-03-11T21:25:38.537283+0000","last_became_peered":"2024-03-11T21:25:38.537283+0000","last_unstale":"2024-03-11T21:34:58.386484+0000","last_undegraded":"2024-03-11T21:34:58.386484+0000","last_fullsized":"2024-03-11T21:34:58.386484+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:32:18.578038+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:37.512018+0000","last_clean_scrub_stamp":"2024-03-11T21:32:18.578038+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:35:01.870367+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.18","version":"74'9","reported_seq":414,"reported_epoch":241,"state":"active+clean","last_fresh":"2024-03-11T21:35:04.388271+0000","last_change":"2024-03-11T21:34:32.763177+0000","last_active":"2024-03-11T21:35:04.388271+0000","last_peered":"2024-03-11T21:35:04.388271+0000","last_clean":"2024-03-11T21:35:04.388271+0000","last_became_active":"2024-03-11T21:25:40.552216+0000","last_became_peered":"2024-03-11T21:25:40.552216+0000","last_unstale":"2024-03-11T21:35:04.388271+0000","last_undegraded":"2024-03-11T21:35:04.388271+0000","last_fullsized":"2024-03-11T21:35:04.388271+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'9","last_scrub_stamp":"2024-03-11T21:34:32.763054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:39.525554+0000","last_clean_scrub_stamp":"2024-03-11T21:34:32.763054+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:35:36.156930+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"8.17","version":"241'9491","reported_seq":21842,"reported_epoch":241,"state":"active+clean","last_fresh":"2024-03-11T21:35:07.136910+0000","last_change":"2024-03-11T21:34:51.320663+0000","last_active":"2024-03-11T21:35:07.136910+0000","last_peered":"2024-03-11T21:35:07.136910+0000","last_clean":"2024-03-11T21:35:07.136910+0000","last_became_active":"2024-03-11T21:34:51.300935+0000","last_became_peered":"2024-03-11T21:34:51.300935+0000","last_unstale":"2024-03-11T21:35:07.136910+0000","last_undegraded":"2024-03-11T21:35:07.136910+0000","last_fullsized":"2024-03-11T21:35:07.136910+0000","mapping_epoch":233,"log_start":"175'4693","ondisk_log_start":"175'4693","created":181,"last_epoch_clean":234,"parent":"0.0","parent_split_bits":5,"last_scrub":"219'7594","last_scrub_stamp":"2024-03-11T21:34:10.514580+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:31:51.219688+0000","last_clean_scrub_stamp":"2024-03-11T21:34:10.514580+0000","objects_scrubbed":2575,"log_size":4798,"log_dups_size":0,"ondisk_log_size":4798,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:35:11.641075+0000","scrub_duration":815,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":43057152,"num_objects":657,"num_object_clones":0,"num_object_copies":1971,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":657,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1918,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1222,"num_bytes_recovered":80084992,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":382,"reported_epoch":241,"state":"active+clean","last_fresh":"2024-03-11T21:35:04.397500+0000","last_change":"2024-03-11T21:32:12.654714+0000","last_active":"2024-03-11T21:35:04.397500+0000","last_peered":"2024-03-11T21:35:04.397500+0000","last_clean":"2024-03-11T21:35:04.397500+0000","last_became_active":"2024-03-11T21:25:44.590234+0000","last_became_peered":"2024-03-11T21:25:44.590234+0000","last_unstale":"2024-03-11T21:35:04.397500+0000","last_undegraded":"2024-03-11T21:35:04.397500+0000","last_fullsized":"2024-03-11T21:35:04.397500+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:32:12.654573+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:32:12.654573+0000","last_clean_scrub_stamp":"2024-03-11T21:32:12.654573+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":43,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3],"acting":[5,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":39,"reported_epoch":241,"state":"active+clean","last_fresh":"2024-03-11T21:35:04.396970+0000","last_change":"2024-03-11T21:34:41.148957+0000","last_active":"2024-03-11T21:35:04.396970+0000","last_peered":"2024-03-11T21:35:04.396970+0000","last_clean":"2024-03-11T21:35:04.396970+0000","last_became_active":"2024-03-11T21:34:41.145299+0000","last_became_peered":"2024-03-11T21:34:41.145299+0000","last_unstale":"2024-03-11T21:35:04.396970+0000","last_undegraded":"2024-03-11T21:35:04.396970+0000","last_fullsized":"2024-03-11T21:35:04.396970+0000","mapping_epoch":226,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":227,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:31:56.567802+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:41.544550+0000","last_clean_scrub_stamp":"2024-03-11T21:31:56.567802+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:35:08.003010+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.1d","version":"0'0","reported_seq":401,"reported_epoch":241,"state":"active+clean","last_fresh":"2024-03-11T21:35:04.398006+0000","last_change":"2024-03-11T21:34:05.871427+0000","last_active":"2024-03-11T21:35:04.398006+0000","last_peered":"2024-03-11T21:35:04.398006+0000","last_clean":"2024-03-11T21:35:04.398006+0000","last_became_active":"2024-03-11T21:25:38.542628+0000","last_became_peered":"2024-03-11T21:25:38.542628+0000","last_unstale":"2024-03-11T21:35:04.398006+0000","last_undegraded":"2024-03-11T21:35:04.398006+0000","last_fullsized":"2024-03-11T21:35:04.398006+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:34:05.871316+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:37.512018+0000","last_clean_scrub_stamp":"2024-03-11T21:34:05.871316+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:35:08.808268+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,0],"acting":[7,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.1b","version":"74'5","reported_seq":392,"reported_epoch":241,"state":"active+clean","last_fresh":"2024-03-11T21:35:04.388737+0000","last_change":"2024-03-11T21:31:57.621692+0000","last_active":"2024-03-11T21:35:04.388737+0000","last_peered":"2024-03-11T21:35:04.388737+0000","last_clean":"2024-03-11T21:35:04.388737+0000","last_became_active":"2024-03-11T21:25:40.555826+0000","last_became_peered":"2024-03-11T21:25:40.555826+0000","last_unstale":"2024-03-11T21:35:04.388737+0000","last_undegraded":"2024-03-11T21:35:04.388737+0000","last_fullsized":"2024-03-11T21:35:04.388737+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'5","last_scrub_stamp":"2024-03-11T21:31:57.621580+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:39.525554+0000","last_clean_scrub_stamp":"2024-03-11T21:31:57.621580+0000","objects_scrubbed":2,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:35:04.446516+0000","scrub_duration":34,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":11,"num_read_kb":7,"num_write":6,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,1],"acting":[4,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"8.16","version":"240'9328","reported_seq":22859,"reported_epoch":240,"state":"active+clean","last_fresh":"2024-03-11T21:35:02.742243+0000","last_change":"2024-03-11T21:33:59.690527+0000","last_active":"2024-03-11T21:35:02.742243+0000","last_peered":"2024-03-11T21:35:02.742243+0000","last_clean":"2024-03-11T21:35:02.742243+0000","last_became_active":"2024-03-11T21:33:53.254658+0000","last_became_peered":"2024-03-11T21:33:53.254658+0000","last_unstale":"2024-03-11T21:35:02.742243+0000","last_undegraded":"2024-03-11T21:35:02.742243+0000","last_fullsized":"2024-03-11T21:35:02.742243+0000","mapping_epoch":208,"log_start":"170'3665","ondisk_log_start":"170'3665","created":181,"last_epoch_clean":209,"parent":"0.0","parent_split_bits":5,"last_scrub":"212'7305","last_scrub_stamp":"2024-03-11T21:33:59.690214+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:31:51.219688+0000","last_clean_scrub_stamp":"2024-03-11T21:33:59.690214+0000","objects_scrubbed":2873,"log_size":5663,"log_dups_size":0,"ondisk_log_size":5663,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:35:08.100514+0000","scrub_duration":952,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":54657024,"num_objects":834,"num_object_clones":0,"num_object_copies":2502,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":834,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2039,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5],"acting":[3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":380,"reported_epoch":240,"state":"active+clean","last_fresh":"2024-03-11T21:34:58.347708+0000","last_change":"2024-03-11T21:34:31.720942+0000","last_active":"2024-03-11T21:34:58.347708+0000","last_peered":"2024-03-11T21:34:58.347708+0000","last_clean":"2024-03-11T21:34:58.347708+0000","last_became_active":"2024-03-11T21:25:44.589786+0000","last_became_peered":"2024-03-11T21:25:44.589786+0000","last_unstale":"2024-03-11T21:34:58.347708+0000","last_undegraded":"2024-03-11T21:34:58.347708+0000","last_fullsized":"2024-03-11T21:34:58.347708+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:34:31.720821+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:43.559452+0000","last_clean_scrub_stamp":"2024-03-11T21:34:31.720821+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:35:39.422862+0000","scrub_duration":33,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,7],"acting":[0,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":397,"reported_epoch":241,"state":"active+clean","last_fresh":"2024-03-11T21:35:04.398853+0000","last_change":"2024-03-11T21:34:41.147546+0000","last_active":"2024-03-11T21:35:04.398853+0000","last_peered":"2024-03-11T21:35:04.398853+0000","last_clean":"2024-03-11T21:35:04.398853+0000","last_became_active":"2024-03-11T21:34:41.147185+0000","last_became_peered":"2024-03-11T21:34:41.147185+0000","last_unstale":"2024-03-11T21:35:04.398853+0000","last_undegraded":"2024-03-11T21:35:04.398853+0000","last_fullsized":"2024-03-11T21:35:04.398853+0000","mapping_epoch":226,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":227,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:32:21.535034+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:32:21.535034+0000","last_clean_scrub_stamp":"2024-03-11T21:32:21.535034+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,7],"acting":[5,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1c","version":"0'0","reported_seq":401,"reported_epoch":241,"state":"active+clean","last_fresh":"2024-03-11T21:35:04.396920+0000","last_change":"2024-03-11T21:34:04.858406+0000","last_active":"2024-03-11T21:35:04.396920+0000","last_peered":"2024-03-11T21:35:04.396920+0000","last_clean":"2024-03-11T21:35:04.396920+0000","last_became_active":"2024-03-11T21:25:38.541364+0000","last_became_peered":"2024-03-11T21:25:38.541364+0000","last_unstale":"2024-03-11T21:35:04.396920+0000","last_undegraded":"2024-03-11T21:35:04.396920+0000","last_fullsized":"2024-03-11T21:35:04.396920+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:34:04.858280+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:28:32.566437+0000","last_clean_scrub_stamp":"2024-03-11T21:34:04.858280+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:35:13.190197+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.1a","version":"74'9","reported_seq":414,"reported_epoch":241,"state":"active+clean","last_fresh":"2024-03-11T21:35:05.451864+0000","last_change":"2024-03-11T21:35:05.451932+0000","last_active":"2024-03-11T21:35:05.451864+0000","last_peered":"2024-03-11T21:35:05.451864+0000","last_clean":"2024-03-11T21:35:05.451864+0000","last_became_active":"2024-03-11T21:25:40.556629+0000","last_became_peered":"2024-03-11T21:25:40.556629+0000","last_unstale":"2024-03-11T21:35:05.451864+0000","last_undegraded":"2024-03-11T21:35:05.451864+0000","last_fullsized":"2024-03-11T21:35:05.451864+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'9","last_scrub_stamp":"2024-03-11T21:35:05.451780+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:39.525554+0000","last_clean_scrub_stamp":"2024-03-11T21:35:05.451780+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:36:25.443084+0000","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0],"acting":[4,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"8.11","version":"240'9521","reported_seq":9731,"reported_epoch":240,"state":"active+remapped+backfill_toofull","last_fresh":"2024-03-11T21:35:03.283599+0000","last_change":"2024-03-11T21:35:00.388348+0000","last_active":"2024-03-11T21:35:03.283599+0000","last_peered":"2024-03-11T21:35:03.283599+0000","last_clean":"2024-03-11T21:32:58.048747+0000","last_became_active":"2024-03-11T21:32:59.353819+0000","last_became_peered":"2024-03-11T21:32:59.353819+0000","last_unstale":"2024-03-11T21:35:03.283599+0000","last_undegraded":"2024-03-11T21:35:03.283599+0000","last_fullsized":"2024-03-11T21:35:03.283599+0000","mapping_epoch":184,"log_start":"174'4512","ondisk_log_start":"174'4512","created":181,"last_epoch_clean":182,"parent":"0.0","parent_split_bits":5,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:31:51.219688+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:31:51.219688+0000","last_clean_scrub_stamp":"2024-03-11T21:31:51.219688+0000","objects_scrubbed":0,"log_size":5009,"log_dups_size":0,"ondisk_log_size":5009,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":54165504,"num_objects":827,"num_object_clones":0,"num_object_copies":2481,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":2481,"num_objects_unfound":0,"num_objects_dirty":827,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6779,"num_write_kb":243360,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":131072,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[0,1,2],"avail_no_missing":["0","1","2"],"object_location_counts":[{"shards":"0,1,2","objects":827}],"blocked_by":[],"up_primary":5,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.1f","version":"0'0","reported_seq":380,"reported_epoch":240,"state":"active+clean","last_fresh":"2024-03-11T21:34:58.386426+0000","last_change":"2024-03-11T21:34:05.863007+0000","last_active":"2024-03-11T21:34:58.386426+0000","last_peered":"2024-03-11T21:34:58.386426+0000","last_clean":"2024-03-11T21:34:58.386426+0000","last_became_active":"2024-03-11T21:25:45.302637+0000","last_became_peered":"2024-03-11T21:25:45.302637+0000","last_unstale":"2024-03-11T21:34:58.386426+0000","last_undegraded":"2024-03-11T21:34:58.386426+0000","last_fullsized":"2024-03-11T21:34:58.386426+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:34:05.862891+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:43.559452+0000","last_clean_scrub_stamp":"2024-03-11T21:34:05.862891+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:35:13.148496+0000","scrub_duration":122,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1c","version":"0'0","reported_seq":386,"reported_epoch":241,"state":"active+clean","last_fresh":"2024-03-11T21:35:04.390632+0000","last_change":"2024-03-11T21:34:11.766184+0000","last_active":"2024-03-11T21:35:04.390632+0000","last_peered":"2024-03-11T21:35:04.390632+0000","last_clean":"2024-03-11T21:35:04.390632+0000","last_became_active":"2024-03-11T21:25:42.576064+0000","last_became_peered":"2024-03-11T21:25:42.576064+0000","last_unstale":"2024-03-11T21:35:04.390632+0000","last_undegraded":"2024-03-11T21:35:04.390632+0000","last_fullsized":"2024-03-11T21:35:04.390632+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:34:11.766066+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:41.544550+0000","last_clean_scrub_stamp":"2024-03-11T21:34:11.766066+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:35:30.920690+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1b","version":"0'0","reported_seq":403,"reported_epoch":240,"state":"active+clean","last_fresh":"2024-03-11T21:34:58.386031+0000","last_change":"2024-03-11T21:34:41.147151+0000","last_active":"2024-03-11T21:34:58.386031+0000","last_peered":"2024-03-11T21:34:58.386031+0000","last_clean":"2024-03-11T21:34:58.386031+0000","last_became_active":"2024-03-11T21:34:41.145184+0000","last_became_peered":"2024-03-11T21:34:41.145184+0000","last_unstale":"2024-03-11T21:34:58.386031+0000","last_undegraded":"2024-03-11T21:34:58.386031+0000","last_fullsized":"2024-03-11T21:34:58.386031+0000","mapping_epoch":226,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":227,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:34:27.998724+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:34:27.998724+0000","last_clean_scrub_stamp":"2024-03-11T21:34:27.998724+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:35:45.823051+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1d","version":"74'12","reported_seq":402,"reported_epoch":240,"state":"active+clean","last_fresh":"2024-03-11T21:34:58.386095+0000","last_change":"2024-03-11T21:31:20.514563+0000","last_active":"2024-03-11T21:34:58.386095+0000","last_peered":"2024-03-11T21:34:58.386095+0000","last_clean":"2024-03-11T21:34:58.386095+0000","last_became_active":"2024-03-11T21:25:40.556330+0000","last_became_peered":"2024-03-11T21:25:40.556330+0000","last_unstale":"2024-03-11T21:34:58.386095+0000","last_undegraded":"2024-03-11T21:34:58.386095+0000","last_fullsized":"2024-03-11T21:34:58.386095+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'12","last_scrub_stamp":"2024-03-11T21:31:20.514423+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:39.525554+0000","last_clean_scrub_stamp":"2024-03-11T21:31:20.514423+0000","objects_scrubbed":5,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:35:01.869944+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":25,"num_read_kb":16,"num_write":14,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.10","version":"241'9659","reported_seq":21013,"reported_epoch":241,"state":"active+clean","last_fresh":"2024-03-11T21:35:07.111595+0000","last_change":"2024-03-11T21:34:52.323099+0000","last_active":"2024-03-11T21:35:07.111595+0000","last_peered":"2024-03-11T21:35:07.111595+0000","last_clean":"2024-03-11T21:35:07.111595+0000","last_became_active":"2024-03-11T21:34:52.300002+0000","last_became_peered":"2024-03-11T21:34:52.300002+0000","last_unstale":"2024-03-11T21:35:07.111595+0000","last_undegraded":"2024-03-11T21:35:07.111595+0000","last_fullsized":"2024-03-11T21:35:07.111595+0000","mapping_epoch":234,"log_start":"178'5079","ondisk_log_start":"178'5079","created":181,"last_epoch_clean":235,"parent":"0.0","parent_split_bits":5,"last_scrub":"219'7729","last_scrub_stamp":"2024-03-11T21:34:10.807906+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:31:51.219688+0000","last_clean_scrub_stamp":"2024-03-11T21:34:10.807906+0000","objects_scrubbed":2679,"log_size":4580,"log_dups_size":0,"ondisk_log_size":4580,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:35:30.641361+0000","scrub_duration":540,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":48496640,"num_objects":740,"num_object_clones":0,"num_object_copies":2220,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":740,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1939,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1258,"num_bytes_recovered":82444288,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,4],"acting":[7,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.1e","version":"0'0","reported_seq":375,"reported_epoch":241,"state":"active+clean","last_fresh":"2024-03-11T21:35:04.389068+0000","last_change":"2024-03-11T21:32:19.599182+0000","last_active":"2024-03-11T21:35:04.389068+0000","last_peered":"2024-03-11T21:35:04.389068+0000","last_clean":"2024-03-11T21:35:04.389068+0000","last_became_active":"2024-03-11T21:25:45.300355+0000","last_became_peered":"2024-03-11T21:25:45.300355+0000","last_unstale":"2024-03-11T21:35:04.389068+0000","last_undegraded":"2024-03-11T21:35:04.389068+0000","last_fullsized":"2024-03-11T21:35:04.389068+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:32:19.599050+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:43.559452+0000","last_clean_scrub_stamp":"2024-03-11T21:32:19.599050+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:35:04.446681+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5],"acting":[4,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1d","version":"0'0","reported_seq":384,"reported_epoch":240,"state":"active+clean","last_fresh":"2024-03-11T21:34:58.348617+0000","last_change":"2024-03-11T21:34:12.507829+0000","last_active":"2024-03-11T21:34:58.348617+0000","last_peered":"2024-03-11T21:34:58.348617+0000","last_clean":"2024-03-11T21:34:58.348617+0000","last_became_active":"2024-03-11T21:25:42.579407+0000","last_became_peered":"2024-03-11T21:25:42.579407+0000","last_unstale":"2024-03-11T21:34:58.348617+0000","last_undegraded":"2024-03-11T21:34:58.348617+0000","last_fullsized":"2024-03-11T21:34:58.348617+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:34:12.507702+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:30:18.678531+0000","last_clean_scrub_stamp":"2024-03-11T21:34:12.507702+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:35:16.018362+0000","scrub_duration":16,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1a","version":"0'0","reported_seq":414,"reported_epoch":241,"state":"active+clean","last_fresh":"2024-03-11T21:35:04.399959+0000","last_change":"2024-03-11T21:34:41.145362+0000","last_active":"2024-03-11T21:35:04.399959+0000","last_peered":"2024-03-11T21:35:04.399959+0000","last_clean":"2024-03-11T21:35:04.399959+0000","last_became_active":"2024-03-11T21:34:41.145014+0000","last_became_peered":"2024-03-11T21:34:41.145014+0000","last_unstale":"2024-03-11T21:35:04.399959+0000","last_undegraded":"2024-03-11T21:35:04.399959+0000","last_fullsized":"2024-03-11T21:35:04.399959+0000","mapping_epoch":226,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":227,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:34:20.415006+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:37.512018+0000","last_clean_scrub_stamp":"2024-03-11T21:34:20.415006+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:35:46.870412+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.1c","version":"74'15","reported_seq":414,"reported_epoch":241,"state":"active+clean","last_fresh":"2024-03-11T21:35:04.394054+0000","last_change":"2024-03-11T21:34:10.311308+0000","last_active":"2024-03-11T21:35:04.394054+0000","last_peered":"2024-03-11T21:35:04.394054+0000","last_clean":"2024-03-11T21:35:04.394054+0000","last_became_active":"2024-03-11T21:25:40.551453+0000","last_became_peered":"2024-03-11T21:25:40.551453+0000","last_unstale":"2024-03-11T21:35:04.394054+0000","last_undegraded":"2024-03-11T21:35:04.394054+0000","last_fullsized":"2024-03-11T21:35:04.394054+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'15","last_scrub_stamp":"2024-03-11T21:34:10.311181+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:39.525554+0000","last_clean_scrub_stamp":"2024-03-11T21:34:10.311181+0000","objects_scrubbed":7,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:35:36.061874+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"8.13","version":"240'9442","reported_seq":19710,"reported_epoch":240,"state":"active+clean","last_fresh":"2024-03-11T21:35:02.712205+0000","last_change":"2024-03-11T21:34:02.249988+0000","last_active":"2024-03-11T21:35:02.712205+0000","last_peered":"2024-03-11T21:35:02.712205+0000","last_clean":"2024-03-11T21:35:02.712205+0000","last_became_active":"2024-03-11T21:33:33.425325+0000","last_became_peered":"2024-03-11T21:33:33.425325+0000","last_unstale":"2024-03-11T21:35:02.712205+0000","last_undegraded":"2024-03-11T21:35:02.712205+0000","last_fullsized":"2024-03-11T21:35:02.712205+0000","mapping_epoch":199,"log_start":"172'3923","ondisk_log_start":"172'3923","created":181,"last_epoch_clean":200,"parent":"0.0","parent_split_bits":5,"last_scrub":"213'7420","last_scrub_stamp":"2024-03-11T21:34:02.249703+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:31:51.219688+0000","last_clean_scrub_stamp":"2024-03-11T21:34:02.249703+0000","objects_scrubbed":2850,"log_size":5519,"log_dups_size":0,"ondisk_log_size":5519,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:35:24.228384+0000","scrub_duration":580,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":54067200,"num_objects":825,"num_object_clones":0,"num_object_copies":2475,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":825,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2025,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,1],"acting":[3,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.1d","version":"0'0","reported_seq":372,"reported_epoch":240,"state":"active+clean","last_fresh":"2024-03-11T21:34:58.348104+0000","last_change":"2024-03-11T21:31:18.798813+0000","last_active":"2024-03-11T21:34:58.348104+0000","last_peered":"2024-03-11T21:34:58.348104+0000","last_clean":"2024-03-11T21:34:58.348104+0000","last_became_active":"2024-03-11T21:25:44.588845+0000","last_became_peered":"2024-03-11T21:25:44.588845+0000","last_unstale":"2024-03-11T21:34:58.348104+0000","last_undegraded":"2024-03-11T21:34:58.348104+0000","last_fullsized":"2024-03-11T21:34:58.348104+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:31:18.798701+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:43.559452+0000","last_clean_scrub_stamp":"2024-03-11T21:31:18.798701+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:35:02.646435+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.1e","version":"0'0","reported_seq":385,"reported_epoch":240,"state":"active+clean","last_fresh":"2024-03-11T21:34:58.350595+0000","last_change":"2024-03-11T21:32:07.397655+0000","last_active":"2024-03-11T21:34:58.350595+0000","last_peered":"2024-03-11T21:34:58.350595+0000","last_clean":"2024-03-11T21:34:58.350595+0000","last_became_active":"2024-03-11T21:25:42.582959+0000","last_became_peered":"2024-03-11T21:25:42.582959+0000","last_unstale":"2024-03-11T21:34:58.350595+0000","last_undegraded":"2024-03-11T21:34:58.350595+0000","last_fullsized":"2024-03-11T21:34:58.350595+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:32:07.397538+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:26:58.335431+0000","last_clean_scrub_stamp":"2024-03-11T21:32:07.397538+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:35:01.803760+0000","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.19","version":"67'2","reported_seq":390,"reported_epoch":240,"state":"active+clean","last_fresh":"2024-03-11T21:34:58.385732+0000","last_change":"2024-03-11T21:32:15.509684+0000","last_active":"2024-03-11T21:34:58.385732+0000","last_peered":"2024-03-11T21:34:58.385732+0000","last_clean":"2024-03-11T21:34:58.385732+0000","last_became_active":"2024-03-11T21:25:38.539243+0000","last_became_peered":"2024-03-11T21:25:38.539243+0000","last_unstale":"2024-03-11T21:34:58.385732+0000","last_undegraded":"2024-03-11T21:34:58.385732+0000","last_fullsized":"2024-03-11T21:34:58.385732+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'2","last_scrub_stamp":"2024-03-11T21:32:15.509542+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:37.512018+0000","last_clean_scrub_stamp":"2024-03-11T21:32:15.509542+0000","objects_scrubbed":2,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:35:01.870342+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":957,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":7,"num_read_kb":7,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,0],"acting":[3,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1f","version":"124'16","reported_seq":441,"reported_epoch":241,"state":"active+clean","last_fresh":"2024-03-11T21:35:04.400355+0000","last_change":"2024-03-11T21:34:12.415133+0000","last_active":"2024-03-11T21:35:04.400355+0000","last_peered":"2024-03-11T21:35:04.400355+0000","last_clean":"2024-03-11T21:35:04.400355+0000","last_became_active":"2024-03-11T21:32:22.798242+0000","last_became_peered":"2024-03-11T21:32:22.798242+0000","last_unstale":"2024-03-11T21:35:04.400355+0000","last_undegraded":"2024-03-11T21:35:04.400355+0000","last_fullsized":"2024-03-11T21:35:04.400355+0000","mapping_epoch":166,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":167,"parent":"0.0","parent_split_bits":0,"last_scrub":"124'16","last_scrub_stamp":"2024-03-11T21:34:12.414804+0000","last_deep_scrub":"124'16","last_deep_scrub_stamp":"2024-03-11T21:29:30.873490+0000","last_clean_scrub_stamp":"2024-03-11T21:34:12.414804+0000","objects_scrubbed":5,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:35:36.222319+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,0],"acting":[6,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"8.12","version":"241'9760","reported_seq":23071,"reported_epoch":241,"state":"active+clean","last_fresh":"2024-03-11T21:35:07.167306+0000","last_change":"2024-03-11T21:34:49.254204+0000","last_active":"2024-03-11T21:35:07.167306+0000","last_peered":"2024-03-11T21:35:07.167306+0000","last_clean":"2024-03-11T21:35:07.167306+0000","last_became_active":"2024-03-11T21:34:49.234452+0000","last_became_peered":"2024-03-11T21:34:49.234452+0000","last_unstale":"2024-03-11T21:35:07.167306+0000","last_undegraded":"2024-03-11T21:35:07.167306+0000","last_fullsized":"2024-03-11T21:35:07.167306+0000","mapping_epoch":231,"log_start":"169'3225","ondisk_log_start":"169'3225","created":181,"last_epoch_clean":232,"parent":"0.0","parent_split_bits":5,"last_scrub":"211'7452","last_scrub_stamp":"2024-03-11T21:33:58.939672+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:31:51.219688+0000","last_clean_scrub_stamp":"2024-03-11T21:33:58.939672+0000","objects_scrubbed":3006,"log_size":6535,"log_dups_size":0,"ondisk_log_size":6535,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:35:28.047990+0000","scrub_duration":697,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":45088768,"num_objects":688,"num_object_clones":0,"num_object_copies":2064,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":688,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2318,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1368,"num_bytes_recovered":89653248,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,2],"acting":[6,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.1c","version":"74'1","reported_seq":43,"reported_epoch":241,"state":"active+clean","last_fresh":"2024-03-11T21:35:04.396842+0000","last_change":"2024-03-11T21:34:41.161206+0000","last_active":"2024-03-11T21:35:04.396842+0000","last_peered":"2024-03-11T21:35:04.396842+0000","last_clean":"2024-03-11T21:35:04.396842+0000","last_became_active":"2024-03-11T21:34:41.144839+0000","last_became_peered":"2024-03-11T21:34:41.144839+0000","last_unstale":"2024-03-11T21:35:04.396842+0000","last_undegraded":"2024-03-11T21:35:04.396842+0000","last_fullsized":"2024-03-11T21:35:04.396842+0000","mapping_epoch":226,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":227,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'1","last_scrub_stamp":"2024-03-11T21:34:16.378072+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:43.559452+0000","last_clean_scrub_stamp":"2024-03-11T21:34:16.378072+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:35:39.328627+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":370,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":370,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.1f","version":"229'10","reported_seq":548,"reported_epoch":241,"state":"active+clean","last_fresh":"2024-03-11T21:35:04.400265+0000","last_change":"2024-03-11T21:34:41.168287+0000","last_active":"2024-03-11T21:35:04.400265+0000","last_peered":"2024-03-11T21:35:04.400265+0000","last_clean":"2024-03-11T21:35:04.400265+0000","last_became_active":"2024-03-11T21:34:41.145862+0000","last_became_peered":"2024-03-11T21:34:41.145862+0000","last_unstale":"2024-03-11T21:35:04.400265+0000","last_undegraded":"2024-03-11T21:35:04.400265+0000","last_fullsized":"2024-03-11T21:35:04.400265+0000","mapping_epoch":226,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":227,"parent":"0.0","parent_split_bits":0,"last_scrub":"154'9","last_scrub_stamp":"2024-03-11T21:34:23.327729+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:41.544550+0000","last_clean_scrub_stamp":"2024-03-11T21:34:23.327729+0000","objects_scrubbed":1,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:35:46.133503+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.18","version":"67'1","reported_seq":427,"reported_epoch":241,"state":"active+clean","last_fresh":"2024-03-11T21:35:04.396776+0000","last_change":"2024-03-11T21:34:41.156671+0000","last_active":"2024-03-11T21:35:04.396776+0000","last_peered":"2024-03-11T21:35:04.396776+0000","last_clean":"2024-03-11T21:35:04.396776+0000","last_became_active":"2024-03-11T21:34:41.138664+0000","last_became_peered":"2024-03-11T21:34:41.138664+0000","last_unstale":"2024-03-11T21:35:04.396776+0000","last_undegraded":"2024-03-11T21:35:04.396776+0000","last_fullsized":"2024-03-11T21:35:04.396776+0000","mapping_epoch":226,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":227,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'1","last_scrub_stamp":"2024-03-11T21:34:32.363650+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:37.512018+0000","last_clean_scrub_stamp":"2024-03-11T21:34:32.363650+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:35:37.328713+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":436,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":7,"num_read_kb":7,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":872,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,7],"acting":[5,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1e","version":"74'10","reported_seq":440,"reported_epoch":240,"state":"active+clean","last_fresh":"2024-03-11T21:34:58.350523+0000","last_change":"2024-03-11T21:34:41.185277+0000","last_active":"2024-03-11T21:34:58.350523+0000","last_peered":"2024-03-11T21:34:58.350523+0000","last_clean":"2024-03-11T21:34:58.350523+0000","last_became_active":"2024-03-11T21:34:41.147199+0000","last_became_peered":"2024-03-11T21:34:41.147199+0000","last_unstale":"2024-03-11T21:34:58.350523+0000","last_undegraded":"2024-03-11T21:34:58.350523+0000","last_fullsized":"2024-03-11T21:34:58.350523+0000","mapping_epoch":226,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":227,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'10","last_scrub_stamp":"2024-03-11T21:32:53.668918+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:39.525554+0000","last_clean_scrub_stamp":"2024-03-11T21:32:53.668918+0000","objects_scrubbed":5,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:35:01.803892+0000","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.13","version":"0'0","reported_seq":379,"reported_epoch":240,"state":"active+clean","last_fresh":"2024-03-11T21:34:58.386336+0000","last_change":"2024-03-11T21:34:13.822920+0000","last_active":"2024-03-11T21:34:58.386336+0000","last_peered":"2024-03-11T21:34:58.386336+0000","last_clean":"2024-03-11T21:34:58.386336+0000","last_became_active":"2024-03-11T21:25:45.302102+0000","last_became_peered":"2024-03-11T21:25:45.302102+0000","last_unstale":"2024-03-11T21:34:58.386336+0000","last_undegraded":"2024-03-11T21:34:58.386336+0000","last_fullsized":"2024-03-11T21:34:58.386336+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:34:13.822803+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:43.559452+0000","last_clean_scrub_stamp":"2024-03-11T21:34:13.822803+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:35:27.875351+0000","scrub_duration":53,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":39,"reported_epoch":241,"state":"active+clean","last_fresh":"2024-03-11T21:35:04.397221+0000","last_change":"2024-03-11T21:34:41.143920+0000","last_active":"2024-03-11T21:35:04.397221+0000","last_peered":"2024-03-11T21:35:04.397221+0000","last_clean":"2024-03-11T21:35:04.397221+0000","last_became_active":"2024-03-11T21:34:41.142569+0000","last_became_peered":"2024-03-11T21:34:41.142569+0000","last_unstale":"2024-03-11T21:35:04.397221+0000","last_undegraded":"2024-03-11T21:35:04.397221+0000","last_fullsized":"2024-03-11T21:35:04.397221+0000","mapping_epoch":226,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":227,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:32:08.612508+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:41.544550+0000","last_clean_scrub_stamp":"2024-03-11T21:32:08.612508+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:35:08.003043+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.17","version":"0'0","reported_seq":402,"reported_epoch":241,"state":"active+clean","last_fresh":"2024-03-11T21:35:06.186929+0000","last_change":"2024-03-11T21:35:06.186953+0000","last_active":"2024-03-11T21:35:06.186929+0000","last_peered":"2024-03-11T21:35:06.186929+0000","last_clean":"2024-03-11T21:35:06.186929+0000","last_became_active":"2024-03-11T21:25:38.540349+0000","last_became_peered":"2024-03-11T21:25:38.540349+0000","last_unstale":"2024-03-11T21:35:06.186929+0000","last_undegraded":"2024-03-11T21:35:06.186929+0000","last_fullsized":"2024-03-11T21:35:06.186929+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:35:06.186840+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:28:25.918101+0000","last_clean_scrub_stamp":"2024-03-11T21:35:06.186840+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:36:20.780691+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,2],"acting":[6,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.11","version":"74'11","reported_seq":435,"reported_epoch":240,"state":"active+clean","last_fresh":"2024-03-11T21:34:58.386052+0000","last_change":"2024-03-11T21:34:41.188418+0000","last_active":"2024-03-11T21:34:58.386052+0000","last_peered":"2024-03-11T21:34:58.386052+0000","last_clean":"2024-03-11T21:34:58.386052+0000","last_became_active":"2024-03-11T21:34:41.154298+0000","last_became_peered":"2024-03-11T21:34:41.154298+0000","last_unstale":"2024-03-11T21:34:58.386052+0000","last_undegraded":"2024-03-11T21:34:58.386052+0000","last_fullsized":"2024-03-11T21:34:58.386052+0000","mapping_epoch":226,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":227,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'11","last_scrub_stamp":"2024-03-11T21:31:17.450051+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:39.525554+0000","last_clean_scrub_stamp":"2024-03-11T21:31:17.450051+0000","objects_scrubbed":5,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:35:06.906389+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":39,"reported_epoch":241,"state":"active+clean","last_fresh":"2024-03-11T21:35:04.396814+0000","last_change":"2024-03-11T21:34:41.140372+0000","last_active":"2024-03-11T21:35:04.396814+0000","last_peered":"2024-03-11T21:35:04.396814+0000","last_clean":"2024-03-11T21:35:04.396814+0000","last_became_active":"2024-03-11T21:34:41.137235+0000","last_became_peered":"2024-03-11T21:34:41.137235+0000","last_unstale":"2024-03-11T21:35:04.396814+0000","last_undegraded":"2024-03-11T21:35:04.396814+0000","last_fullsized":"2024-03-11T21:35:04.396814+0000","mapping_epoch":226,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":227,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:34:20.478420+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:43.559452+0000","last_clean_scrub_stamp":"2024-03-11T21:34:20.478420+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:35:22.077123+0000","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,4],"acting":[7,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":407,"reported_epoch":241,"state":"active+clean","last_fresh":"2024-03-11T21:35:04.402290+0000","last_change":"2024-03-11T21:34:41.156388+0000","last_active":"2024-03-11T21:35:04.402290+0000","last_peered":"2024-03-11T21:35:04.402290+0000","last_clean":"2024-03-11T21:35:04.402290+0000","last_became_active":"2024-03-11T21:34:41.156036+0000","last_became_peered":"2024-03-11T21:34:41.156036+0000","last_unstale":"2024-03-11T21:35:04.402290+0000","last_undegraded":"2024-03-11T21:35:04.402290+0000","last_fullsized":"2024-03-11T21:35:04.402290+0000","mapping_epoch":226,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":227,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:34:09.424875+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:29:39.923895+0000","last_clean_scrub_stamp":"2024-03-11T21:34:09.424875+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:35:14.239741+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.16","version":"0'0","reported_seq":401,"reported_epoch":241,"state":"active+clean","last_fresh":"2024-03-11T21:35:04.401590+0000","last_change":"2024-03-11T21:32:56.247901+0000","last_active":"2024-03-11T21:35:04.401590+0000","last_peered":"2024-03-11T21:35:04.401590+0000","last_clean":"2024-03-11T21:35:04.401590+0000","last_became_active":"2024-03-11T21:25:38.539103+0000","last_became_peered":"2024-03-11T21:25:38.539103+0000","last_unstale":"2024-03-11T21:35:04.401590+0000","last_undegraded":"2024-03-11T21:35:04.401590+0000","last_fullsized":"2024-03-11T21:35:04.401590+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:32:56.247825+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:27:11.938909+0000","last_clean_scrub_stamp":"2024-03-11T21:32:56.247825+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,2],"acting":[5,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.10","version":"94'6","reported_seq":396,"reported_epoch":240,"state":"active+clean","last_fresh":"2024-03-11T21:34:58.385623+0000","last_change":"2024-03-11T21:34:00.727542+0000","last_active":"2024-03-11T21:34:58.385623+0000","last_peered":"2024-03-11T21:34:58.385623+0000","last_clean":"2024-03-11T21:34:58.385623+0000","last_became_active":"2024-03-11T21:25:40.552720+0000","last_became_peered":"2024-03-11T21:25:40.552720+0000","last_unstale":"2024-03-11T21:34:58.385623+0000","last_undegraded":"2024-03-11T21:34:58.385623+0000","last_fullsized":"2024-03-11T21:34:58.385623+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"94'6","last_scrub_stamp":"2024-03-11T21:34:00.727416+0000","last_deep_scrub":"94'6","last_deep_scrub_stamp":"2024-03-11T21:28:24.603974+0000","last_clean_scrub_stamp":"2024-03-11T21:34:00.727416+0000","objects_scrubbed":2,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:35:06.875929+0000","scrub_duration":20,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":6,"num_read_kb":4,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6],"acting":[3,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":371,"reported_epoch":240,"state":"active+clean","last_fresh":"2024-03-11T21:34:58.385506+0000","last_change":"2024-03-11T21:31:14.338908+0000","last_active":"2024-03-11T21:34:58.385506+0000","last_peered":"2024-03-11T21:34:58.385506+0000","last_clean":"2024-03-11T21:34:58.385506+0000","last_became_active":"2024-03-11T21:25:44.583181+0000","last_became_peered":"2024-03-11T21:25:44.583181+0000","last_unstale":"2024-03-11T21:34:58.385506+0000","last_undegraded":"2024-03-11T21:34:58.385506+0000","last_fullsized":"2024-03-11T21:34:58.385506+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:31:14.338640+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:43.559452+0000","last_clean_scrub_stamp":"2024-03-11T21:31:14.338640+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:35:01.870054+0000","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":385,"reported_epoch":240,"state":"active+clean","last_fresh":"2024-03-11T21:34:58.348420+0000","last_change":"2024-03-11T21:34:15.516387+0000","last_active":"2024-03-11T21:34:58.348420+0000","last_peered":"2024-03-11T21:34:58.348420+0000","last_clean":"2024-03-11T21:34:58.348420+0000","last_became_active":"2024-03-11T21:25:42.573158+0000","last_became_peered":"2024-03-11T21:25:42.573158+0000","last_unstale":"2024-03-11T21:34:58.348420+0000","last_undegraded":"2024-03-11T21:34:58.348420+0000","last_fullsized":"2024-03-11T21:34:58.348420+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:34:15.516278+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:27:21.207303+0000","last_clean_scrub_stamp":"2024-03-11T21:34:15.516278+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:35:36.619704+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.15","version":"0'0","reported_seq":392,"reported_epoch":240,"state":"active+clean","last_fresh":"2024-03-11T21:34:58.348328+0000","last_change":"2024-03-11T21:32:12.600491+0000","last_active":"2024-03-11T21:34:58.348328+0000","last_peered":"2024-03-11T21:34:58.348328+0000","last_clean":"2024-03-11T21:34:58.348328+0000","last_became_active":"2024-03-11T21:25:38.541705+0000","last_became_peered":"2024-03-11T21:25:38.541705+0000","last_unstale":"2024-03-11T21:34:58.348328+0000","last_undegraded":"2024-03-11T21:34:58.348328+0000","last_fullsized":"2024-03-11T21:34:58.348328+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:32:12.600388+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:26:44.860499+0000","last_clean_scrub_stamp":"2024-03-11T21:32:12.600388+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:35:02.646710+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.13","version":"109'16","reported_seq":433,"reported_epoch":241,"state":"active+clean","last_fresh":"2024-03-11T21:35:04.388643+0000","last_change":"2024-03-11T21:32:24.657615+0000","last_active":"2024-03-11T21:35:04.388643+0000","last_peered":"2024-03-11T21:35:04.388643+0000","last_clean":"2024-03-11T21:35:04.388643+0000","last_became_active":"2024-03-11T21:26:11.214879+0000","last_became_peered":"2024-03-11T21:26:11.214879+0000","last_unstale":"2024-03-11T21:35:04.388643+0000","last_undegraded":"2024-03-11T21:35:04.388643+0000","last_fullsized":"2024-03-11T21:35:04.388643+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"109'16","last_scrub_stamp":"2024-03-11T21:32:24.657490+0000","last_deep_scrub":"109'16","last_deep_scrub_stamp":"2024-03-11T21:32:24.657490+0000","last_clean_scrub_stamp":"2024-03-11T21:32:24.657490+0000","objects_scrubbed":5,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:35:04.446570+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,2],"acting":[4,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.10","version":"0'0","reported_seq":380,"reported_epoch":240,"state":"active+clean","last_fresh":"2024-03-11T21:34:58.347120+0000","last_change":"2024-03-11T21:32:56.582937+0000","last_active":"2024-03-11T21:34:58.347120+0000","last_peered":"2024-03-11T21:34:58.347120+0000","last_clean":"2024-03-11T21:34:58.347120+0000","last_became_active":"2024-03-11T21:25:44.591189+0000","last_became_peered":"2024-03-11T21:25:44.591189+0000","last_unstale":"2024-03-11T21:34:58.347120+0000","last_undegraded":"2024-03-11T21:34:58.347120+0000","last_fullsized":"2024-03-11T21:34:58.347120+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:32:56.582834+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:27:25.417450+0000","last_clean_scrub_stamp":"2024-03-11T21:32:56.582834+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:35:01.804205+0000","scrub_duration":16,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,1],"acting":[0,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":375,"reported_epoch":240,"state":"active+clean","last_fresh":"2024-03-11T21:34:58.386126+0000","last_change":"2024-03-11T21:31:15.368570+0000","last_active":"2024-03-11T21:34:58.386126+0000","last_peered":"2024-03-11T21:34:58.386126+0000","last_clean":"2024-03-11T21:34:58.386126+0000","last_became_active":"2024-03-11T21:25:42.576425+0000","last_became_peered":"2024-03-11T21:25:42.576425+0000","last_unstale":"2024-03-11T21:34:58.386126+0000","last_undegraded":"2024-03-11T21:34:58.386126+0000","last_fullsized":"2024-03-11T21:34:58.386126+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:31:15.367898+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:31:15.367898+0000","last_clean_scrub_stamp":"2024-03-11T21:31:15.367898+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:35:01.869994+0000","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.14","version":"0'0","reported_seq":403,"reported_epoch":241,"state":"active+clean","last_fresh":"2024-03-11T21:35:04.399559+0000","last_change":"2024-03-11T21:32:22.917604+0000","last_active":"2024-03-11T21:35:04.399559+0000","last_peered":"2024-03-11T21:35:04.399559+0000","last_clean":"2024-03-11T21:35:04.399559+0000","last_became_active":"2024-03-11T21:25:38.541852+0000","last_became_peered":"2024-03-11T21:25:38.541852+0000","last_unstale":"2024-03-11T21:35:04.399559+0000","last_undegraded":"2024-03-11T21:35:04.399559+0000","last_fullsized":"2024-03-11T21:35:04.399559+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:32:22.917423+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:37.512018+0000","last_clean_scrub_stamp":"2024-03-11T21:32:22.917423+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,5],"acting":[6,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.12","version":"74'9","reported_seq":405,"reported_epoch":240,"state":"active+clean","last_fresh":"2024-03-11T21:34:58.348241+0000","last_change":"2024-03-11T21:31:59.579237+0000","last_active":"2024-03-11T21:34:58.348241+0000","last_peered":"2024-03-11T21:34:58.348241+0000","last_clean":"2024-03-11T21:34:58.348241+0000","last_became_active":"2024-03-11T21:25:40.547082+0000","last_became_peered":"2024-03-11T21:25:40.547082+0000","last_unstale":"2024-03-11T21:34:58.348241+0000","last_undegraded":"2024-03-11T21:34:58.348241+0000","last_fullsized":"2024-03-11T21:34:58.348241+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'9","last_scrub_stamp":"2024-03-11T21:31:59.579142+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:39.525554+0000","last_clean_scrub_stamp":"2024-03-11T21:31:59.579142+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:35:02.646559+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.19","version":"241'9614","reported_seq":15670,"reported_epoch":241,"state":"active+clean","last_fresh":"2024-03-11T21:35:06.968068+0000","last_change":"2024-03-11T21:33:58.958302+0000","last_active":"2024-03-11T21:35:06.968068+0000","last_peered":"2024-03-11T21:35:06.968068+0000","last_clean":"2024-03-11T21:35:06.968068+0000","last_became_active":"2024-03-11T21:33:08.320798+0000","last_became_peered":"2024-03-11T21:33:08.320798+0000","last_unstale":"2024-03-11T21:35:06.968068+0000","last_undegraded":"2024-03-11T21:35:06.968068+0000","last_fullsized":"2024-03-11T21:35:06.968068+0000","mapping_epoch":190,"log_start":"171'3959","ondisk_log_start":"171'3959","created":181,"last_epoch_clean":191,"parent":"0.0","parent_split_bits":5,"last_scrub":"211'7356","last_scrub_stamp":"2024-03-11T21:33:58.958068+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:31:51.219688+0000","last_clean_scrub_stamp":"2024-03-11T21:33:58.958068+0000","objects_scrubbed":2898,"log_size":5655,"log_dups_size":0,"ondisk_log_size":5655,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:35:19.300821+0000","scrub_duration":775,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":41156608,"num_objects":628,"num_object_clones":0,"num_object_copies":1884,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":628,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2270,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,0],"acting":[5,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":382,"reported_epoch":241,"state":"active+clean","last_fresh":"2024-03-11T21:35:04.389333+0000","last_change":"2024-03-11T21:34:25.833239+0000","last_active":"2024-03-11T21:35:04.389333+0000","last_peered":"2024-03-11T21:35:04.389333+0000","last_clean":"2024-03-11T21:35:04.389333+0000","last_became_active":"2024-03-11T21:25:44.589586+0000","last_became_peered":"2024-03-11T21:25:44.589586+0000","last_unstale":"2024-03-11T21:35:04.389333+0000","last_undegraded":"2024-03-11T21:35:04.389333+0000","last_fullsized":"2024-03-11T21:35:04.389333+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:34:25.833149+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:27:09.801138+0000","last_clean_scrub_stamp":"2024-03-11T21:34:25.833149+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:35:26.798227+0000","scrub_duration":66,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,5],"acting":[4,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.14","version":"229'10","reported_seq":516,"reported_epoch":240,"state":"active+clean","last_fresh":"2024-03-11T21:34:58.386392+0000","last_change":"2024-03-11T21:34:41.181157+0000","last_active":"2024-03-11T21:34:58.386392+0000","last_peered":"2024-03-11T21:34:58.386392+0000","last_clean":"2024-03-11T21:34:58.386392+0000","last_became_active":"2024-03-11T21:34:41.151329+0000","last_became_peered":"2024-03-11T21:34:41.151329+0000","last_unstale":"2024-03-11T21:34:58.386392+0000","last_undegraded":"2024-03-11T21:34:58.386392+0000","last_fullsized":"2024-03-11T21:34:58.386392+0000","mapping_epoch":226,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":227,"parent":"0.0","parent_split_bits":0,"last_scrub":"154'9","last_scrub_stamp":"2024-03-11T21:32:03.467138+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:41.544550+0000","last_clean_scrub_stamp":"2024-03-11T21:32:03.467138+0000","objects_scrubbed":1,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:35:01.870231+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,2],"acting":[3,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.13","version":"0'0","reported_seq":393,"reported_epoch":240,"state":"active+clean","last_fresh":"2024-03-11T21:34:58.350384+0000","last_change":"2024-03-11T21:34:08.650456+0000","last_active":"2024-03-11T21:34:58.350384+0000","last_peered":"2024-03-11T21:34:58.350384+0000","last_clean":"2024-03-11T21:34:58.350384+0000","last_became_active":"2024-03-11T21:25:38.539467+0000","last_became_peered":"2024-03-11T21:25:38.539467+0000","last_unstale":"2024-03-11T21:34:58.350384+0000","last_undegraded":"2024-03-11T21:34:58.350384+0000","last_fullsized":"2024-03-11T21:34:58.350384+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:34:08.650371+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:37.512018+0000","last_clean_scrub_stamp":"2024-03-11T21:34:08.650371+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:35:32.975966+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.15","version":"94'13","reported_seq":441,"reported_epoch":241,"state":"active+clean","last_fresh":"2024-03-11T21:35:04.398329+0000","last_change":"2024-03-11T21:34:41.173208+0000","last_active":"2024-03-11T21:35:04.398329+0000","last_peered":"2024-03-11T21:35:04.398329+0000","last_clean":"2024-03-11T21:35:04.398329+0000","last_became_active":"2024-03-11T21:34:41.147412+0000","last_became_peered":"2024-03-11T21:34:41.147412+0000","last_unstale":"2024-03-11T21:35:04.398329+0000","last_undegraded":"2024-03-11T21:35:04.398329+0000","last_fullsized":"2024-03-11T21:35:04.398329+0000","mapping_epoch":226,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":227,"parent":"0.0","parent_split_bits":0,"last_scrub":"94'13","last_scrub_stamp":"2024-03-11T21:32:10.610489+0000","last_deep_scrub":"94'13","last_deep_scrub_stamp":"2024-03-11T21:29:41.853158+0000","last_clean_scrub_stamp":"2024-03-11T21:32:10.610489+0000","objects_scrubbed":4,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.18","version":"241'9715","reported_seq":16508,"reported_epoch":241,"state":"active+clean","last_fresh":"2024-03-11T21:35:07.023689+0000","last_change":"2024-03-11T21:33:59.340712+0000","last_active":"2024-03-11T21:35:07.023689+0000","last_peered":"2024-03-11T21:35:07.023689+0000","last_clean":"2024-03-11T21:35:07.023689+0000","last_became_active":"2024-03-11T21:33:30.363159+0000","last_became_peered":"2024-03-11T21:33:30.363159+0000","last_unstale":"2024-03-11T21:35:07.023689+0000","last_undegraded":"2024-03-11T21:35:07.023689+0000","last_fullsized":"2024-03-11T21:35:07.023689+0000","mapping_epoch":196,"log_start":"173'4407","ondisk_log_start":"173'4407","created":181,"last_epoch_clean":197,"parent":"0.0","parent_split_bits":5,"last_scrub":"211'7505","last_scrub_stamp":"2024-03-11T21:33:59.340450+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:31:51.219688+0000","last_clean_scrub_stamp":"2024-03-11T21:33:59.340450+0000","objects_scrubbed":2900,"log_size":5308,"log_dups_size":0,"ondisk_log_size":5308,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:35:29.223040+0000","scrub_duration":648,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":44695552,"num_objects":682,"num_object_clones":0,"num_object_copies":2046,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":682,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2218,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,5],"acting":[4,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.16","version":"0'0","reported_seq":392,"reported_epoch":240,"state":"active+clean","last_fresh":"2024-03-11T21:34:58.350271+0000","last_change":"2024-03-11T21:34:41.149107+0000","last_active":"2024-03-11T21:34:58.350271+0000","last_peered":"2024-03-11T21:34:58.350271+0000","last_clean":"2024-03-11T21:34:58.350271+0000","last_became_active":"2024-03-11T21:34:41.148012+0000","last_became_peered":"2024-03-11T21:34:41.148012+0000","last_unstale":"2024-03-11T21:34:58.350271+0000","last_undegraded":"2024-03-11T21:34:58.350271+0000","last_fullsized":"2024-03-11T21:34:58.350271+0000","mapping_epoch":226,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":227,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:32:23.607460+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:43.559452+0000","last_clean_scrub_stamp":"2024-03-11T21:32:23.607460+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:35:01.803724+0000","scrub_duration":35,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.15","version":"98'9","reported_seq":508,"reported_epoch":241,"state":"active+clean","last_fresh":"2024-03-11T21:35:04.398468+0000","last_change":"2024-03-11T21:34:30.363954+0000","last_active":"2024-03-11T21:35:04.398468+0000","last_peered":"2024-03-11T21:35:04.398468+0000","last_clean":"2024-03-11T21:35:04.398468+0000","last_became_active":"2024-03-11T21:25:42.572555+0000","last_became_peered":"2024-03-11T21:25:42.572555+0000","last_unstale":"2024-03-11T21:35:04.398468+0000","last_undegraded":"2024-03-11T21:35:04.398468+0000","last_fullsized":"2024-03-11T21:35:04.398468+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"98'9","last_scrub_stamp":"2024-03-11T21:34:30.363823+0000","last_deep_scrub":"98'9","last_deep_scrub_stamp":"2024-03-11T21:31:20.624923+0000","last_clean_scrub_stamp":"2024-03-11T21:34:30.363823+0000","objects_scrubbed":1,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:35:44.164104+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.12","version":"0'0","reported_seq":406,"reported_epoch":241,"state":"active+clean","last_fresh":"2024-03-11T21:35:04.397910+0000","last_change":"2024-03-11T21:34:41.136310+0000","last_active":"2024-03-11T21:35:04.397910+0000","last_peered":"2024-03-11T21:35:04.397910+0000","last_clean":"2024-03-11T21:35:04.397910+0000","last_became_active":"2024-03-11T21:34:41.134090+0000","last_became_peered":"2024-03-11T21:34:41.134090+0000","last_unstale":"2024-03-11T21:35:04.397910+0000","last_undegraded":"2024-03-11T21:35:04.397910+0000","last_fullsized":"2024-03-11T21:35:04.397910+0000","mapping_epoch":226,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":227,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:32:14.555187+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:37.512018+0000","last_clean_scrub_stamp":"2024-03-11T21:32:14.555187+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.14","version":"74'10","reported_seq":430,"reported_epoch":240,"state":"active+clean","last_fresh":"2024-03-11T21:34:58.386160+0000","last_change":"2024-03-11T21:34:41.178979+0000","last_active":"2024-03-11T21:34:58.386160+0000","last_peered":"2024-03-11T21:34:58.386160+0000","last_clean":"2024-03-11T21:34:58.386160+0000","last_became_active":"2024-03-11T21:34:41.145114+0000","last_became_peered":"2024-03-11T21:34:41.145114+0000","last_unstale":"2024-03-11T21:34:58.386160+0000","last_undegraded":"2024-03-11T21:34:58.386160+0000","last_fullsized":"2024-03-11T21:34:58.386160+0000","mapping_epoch":226,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":227,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'10","last_scrub_stamp":"2024-03-11T21:32:13.490198+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:39.525554+0000","last_clean_scrub_stamp":"2024-03-11T21:32:13.490198+0000","objects_scrubbed":5,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:35:01.870261+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.15","version":"0'0","reported_seq":39,"reported_epoch":241,"state":"active+clean","last_fresh":"2024-03-11T21:35:04.397829+0000","last_change":"2024-03-11T21:34:41.151416+0000","last_active":"2024-03-11T21:35:04.397829+0000","last_peered":"2024-03-11T21:35:04.397829+0000","last_clean":"2024-03-11T21:35:04.397829+0000","last_became_active":"2024-03-11T21:34:41.142760+0000","last_became_peered":"2024-03-11T21:34:41.142760+0000","last_unstale":"2024-03-11T21:35:04.397829+0000","last_undegraded":"2024-03-11T21:35:04.397829+0000","last_fullsized":"2024-03-11T21:35:04.397829+0000","mapping_epoch":226,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":227,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:34:22.372241+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:43.559452+0000","last_clean_scrub_stamp":"2024-03-11T21:34:22.372241+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:35:41.662719+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":385,"reported_epoch":241,"state":"active+clean","last_fresh":"2024-03-11T21:35:04.401859+0000","last_change":"2024-03-11T21:34:09.244924+0000","last_active":"2024-03-11T21:35:04.401859+0000","last_peered":"2024-03-11T21:35:04.401859+0000","last_clean":"2024-03-11T21:35:04.401859+0000","last_became_active":"2024-03-11T21:25:42.577351+0000","last_became_peered":"2024-03-11T21:25:42.577351+0000","last_unstale":"2024-03-11T21:35:04.401859+0000","last_undegraded":"2024-03-11T21:35:04.401859+0000","last_fullsized":"2024-03-11T21:35:04.401859+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:34:09.244852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:41.544550+0000","last_clean_scrub_stamp":"2024-03-11T21:34:09.244852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:35:11.770192+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,1],"acting":[5,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.11","version":"0'0","reported_seq":395,"reported_epoch":241,"state":"active+clean","last_fresh":"2024-03-11T21:35:04.402368+0000","last_change":"2024-03-11T21:32:09.785387+0000","last_active":"2024-03-11T21:35:04.402368+0000","last_peered":"2024-03-11T21:35:04.402368+0000","last_clean":"2024-03-11T21:35:04.402368+0000","last_became_active":"2024-03-11T21:25:38.537415+0000","last_became_peered":"2024-03-11T21:25:38.537415+0000","last_unstale":"2024-03-11T21:35:04.402368+0000","last_undegraded":"2024-03-11T21:35:04.402368+0000","last_fullsized":"2024-03-11T21:35:04.402368+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:32:09.785314+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:37.512018+0000","last_clean_scrub_stamp":"2024-03-11T21:32:09.785314+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.17","version":"97'9","reported_seq":433,"reported_epoch":240,"state":"active+clean","last_fresh":"2024-03-11T21:34:58.386483+0000","last_change":"2024-03-11T21:34:41.173645+0000","last_active":"2024-03-11T21:34:58.386483+0000","last_peered":"2024-03-11T21:34:58.386483+0000","last_clean":"2024-03-11T21:34:58.386483+0000","last_became_active":"2024-03-11T21:34:41.146177+0000","last_became_peered":"2024-03-11T21:34:41.146177+0000","last_unstale":"2024-03-11T21:34:58.386483+0000","last_undegraded":"2024-03-11T21:34:58.386483+0000","last_fullsized":"2024-03-11T21:34:58.386483+0000","mapping_epoch":226,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":227,"parent":"0.0","parent_split_bits":0,"last_scrub":"97'9","last_scrub_stamp":"2024-03-11T21:32:08.495352+0000","last_deep_scrub":"97'9","last_deep_scrub_stamp":"2024-03-11T21:27:25.803132+0000","last_clean_scrub_stamp":"2024-03-11T21:32:08.495352+0000","objects_scrubbed":3,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:35:01.870395+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":9,"num_read_kb":6,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,5],"acting":[3,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.14","version":"0'0","reported_seq":393,"reported_epoch":241,"state":"active+clean","last_fresh":"2024-03-11T21:35:04.393059+0000","last_change":"2024-03-11T21:34:41.137559+0000","last_active":"2024-03-11T21:35:04.393059+0000","last_peered":"2024-03-11T21:35:04.393059+0000","last_clean":"2024-03-11T21:35:04.393059+0000","last_became_active":"2024-03-11T21:34:41.136946+0000","last_became_peered":"2024-03-11T21:34:41.136946+0000","last_unstale":"2024-03-11T21:35:04.393059+0000","last_undegraded":"2024-03-11T21:35:04.393059+0000","last_fullsized":"2024-03-11T21:35:04.393059+0000","mapping_epoch":226,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":227,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:34:15.361586+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:43.559452+0000","last_clean_scrub_stamp":"2024-03-11T21:34:15.361586+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:35:23.055559+0000","scrub_duration":27,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,7],"acting":[2,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":395,"reported_epoch":240,"state":"active+clean","last_fresh":"2024-03-11T21:34:58.386522+0000","last_change":"2024-03-11T21:34:41.146564+0000","last_active":"2024-03-11T21:34:58.386522+0000","last_peered":"2024-03-11T21:34:58.386522+0000","last_clean":"2024-03-11T21:34:58.386522+0000","last_became_active":"2024-03-11T21:34:41.145400+0000","last_became_peered":"2024-03-11T21:34:41.145400+0000","last_unstale":"2024-03-11T21:34:58.386522+0000","last_undegraded":"2024-03-11T21:34:58.386522+0000","last_fullsized":"2024-03-11T21:34:58.386522+0000","mapping_epoch":226,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":227,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:34:33.082577+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:41.544550+0000","last_clean_scrub_stamp":"2024-03-11T21:34:33.082577+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:36:02.359492+0000","scrub_duration":35,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.10","version":"0'0","reported_seq":394,"reported_epoch":241,"state":"active+clean","last_fresh":"2024-03-11T21:35:04.392868+0000","last_change":"2024-03-11T21:34:12.365368+0000","last_active":"2024-03-11T21:35:04.392868+0000","last_peered":"2024-03-11T21:35:04.392868+0000","last_clean":"2024-03-11T21:35:04.392868+0000","last_became_active":"2024-03-11T21:25:38.533138+0000","last_became_peered":"2024-03-11T21:25:38.533138+0000","last_unstale":"2024-03-11T21:35:04.392868+0000","last_undegraded":"2024-03-11T21:35:04.392868+0000","last_fullsized":"2024-03-11T21:35:04.392868+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:34:12.365244+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:37.512018+0000","last_clean_scrub_stamp":"2024-03-11T21:34:12.365244+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:35:36.676182+0000","scrub_duration":17,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,0],"acting":[2,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.16","version":"74'9","reported_seq":436,"reported_epoch":240,"state":"active+clean","last_fresh":"2024-03-11T21:34:58.350405+0000","last_change":"2024-03-11T21:34:41.178232+0000","last_active":"2024-03-11T21:34:58.350405+0000","last_peered":"2024-03-11T21:34:58.350405+0000","last_clean":"2024-03-11T21:34:58.350405+0000","last_became_active":"2024-03-11T21:34:41.148571+0000","last_became_peered":"2024-03-11T21:34:41.148571+0000","last_unstale":"2024-03-11T21:34:58.350405+0000","last_undegraded":"2024-03-11T21:34:58.350405+0000","last_fullsized":"2024-03-11T21:34:58.350405+0000","mapping_epoch":226,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":227,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'9","last_scrub_stamp":"2024-03-11T21:32:14.414508+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:39.525554+0000","last_clean_scrub_stamp":"2024-03-11T21:32:14.414508+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:35:01.803826+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.5","version":"241'9669","reported_seq":9900,"reported_epoch":241,"state":"active+remapped+backfill_toofull","last_fresh":"2024-03-11T21:35:07.180324+0000","last_change":"2024-03-11T21:34:42.731203+0000","last_active":"2024-03-11T21:35:07.180324+0000","last_peered":"2024-03-11T21:35:07.180324+0000","last_clean":"2024-03-11T21:34:40.105655+0000","last_became_active":"2024-03-11T21:34:42.690454+0000","last_became_peered":"2024-03-11T21:34:42.690454+0000","last_unstale":"2024-03-11T21:35:07.180324+0000","last_undegraded":"2024-03-11T21:35:07.180324+0000","last_fullsized":"2024-03-11T21:35:07.180324+0000","mapping_epoch":227,"log_start":"169'3143","ondisk_log_start":"169'3143","created":157,"last_epoch_clean":182,"parent":"0.0","parent_split_bits":0,"last_scrub":"185'5575","last_scrub_stamp":"2024-03-11T21:32:59.559295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:31:51.219688+0000","last_clean_scrub_stamp":"2024-03-11T21:32:59.559295+0000","objects_scrubbed":2833,"log_size":6526,"log_dups_size":0,"ondisk_log_size":6526,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"no scrub is scheduled","scrub_duration":572,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":45285376,"num_objects":691,"num_object_clones":0,"num_object_copies":2073,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":691,"num_objects_unfound":0,"num_objects_dirty":691,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4098,"num_write_kb":62592,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,0],"avail_no_missing":["6","0","4"],"object_location_counts":[{"shards":"0,4,6","objects":691}],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.b","version":"0'0","reported_seq":384,"reported_epoch":240,"state":"active+clean","last_fresh":"2024-03-11T21:34:58.386218+0000","last_change":"2024-03-11T21:34:41.149950+0000","last_active":"2024-03-11T21:34:58.386218+0000","last_peered":"2024-03-11T21:34:58.386218+0000","last_clean":"2024-03-11T21:34:58.386218+0000","last_became_active":"2024-03-11T21:34:41.147208+0000","last_became_peered":"2024-03-11T21:34:41.147208+0000","last_unstale":"2024-03-11T21:34:58.386218+0000","last_undegraded":"2024-03-11T21:34:58.386218+0000","last_fullsized":"2024-03-11T21:34:58.386218+0000","mapping_epoch":226,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":227,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:32:22.598150+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:43.559452+0000","last_clean_scrub_stamp":"2024-03-11T21:32:22.598150+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:35:01.870420+0000","scrub_duration":57,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.f","version":"67'2","reported_seq":407,"reported_epoch":241,"state":"active+clean","last_fresh":"2024-03-11T21:35:04.389160+0000","last_change":"2024-03-11T21:34:08.776614+0000","last_active":"2024-03-11T21:35:04.389160+0000","last_peered":"2024-03-11T21:35:04.389160+0000","last_clean":"2024-03-11T21:35:04.389160+0000","last_became_active":"2024-03-11T21:25:38.537711+0000","last_became_peered":"2024-03-11T21:25:38.537711+0000","last_unstale":"2024-03-11T21:35:04.389160+0000","last_undegraded":"2024-03-11T21:35:04.389160+0000","last_fullsized":"2024-03-11T21:35:04.389160+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'2","last_scrub_stamp":"2024-03-11T21:34:08.776510+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:37.512018+0000","last_clean_scrub_stamp":"2024-03-11T21:34:08.776510+0000","objects_scrubbed":2,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:35:18.719734+0000","scrub_duration":46,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":92,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":10,"num_read_kb":10,"num_write":4,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,7],"acting":[4,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.9","version":"146'17","reported_seq":419,"reported_epoch":241,"state":"active+clean","last_fresh":"2024-03-11T21:35:04.388559+0000","last_change":"2024-03-11T21:34:26.765180+0000","last_active":"2024-03-11T21:35:04.388559+0000","last_peered":"2024-03-11T21:35:04.388559+0000","last_clean":"2024-03-11T21:35:04.388559+0000","last_became_active":"2024-03-11T21:25:40.552267+0000","last_became_peered":"2024-03-11T21:25:40.552267+0000","last_unstale":"2024-03-11T21:35:04.388559+0000","last_undegraded":"2024-03-11T21:35:04.388559+0000","last_fullsized":"2024-03-11T21:35:04.388559+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"146'17","last_scrub_stamp":"2024-03-11T21:34:26.765075+0000","last_deep_scrub":"146'17","last_deep_scrub_stamp":"2024-03-11T21:34:26.765075+0000","last_clean_scrub_stamp":"2024-03-11T21:34:26.765075+0000","objects_scrubbed":5,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:35:27.141569+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":25,"num_read_kb":16,"num_write":14,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,3],"acting":[4,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":386,"reported_epoch":241,"state":"active+clean","last_fresh":"2024-03-11T21:35:04.392584+0000","last_change":"2024-03-11T21:32:55.362571+0000","last_active":"2024-03-11T21:35:04.392584+0000","last_peered":"2024-03-11T21:35:04.392584+0000","last_clean":"2024-03-11T21:35:04.392584+0000","last_became_active":"2024-03-11T21:25:42.573987+0000","last_became_peered":"2024-03-11T21:25:42.573987+0000","last_unstale":"2024-03-11T21:35:04.392584+0000","last_undegraded":"2024-03-11T21:35:04.392584+0000","last_fullsized":"2024-03-11T21:35:04.392584+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:32:55.362458+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:41.544550+0000","last_clean_scrub_stamp":"2024-03-11T21:32:55.362458+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"8.e","version":"240'13557","reported_seq":13795,"reported_epoch":240,"state":"active+clean","last_fresh":"2024-03-11T21:35:03.300025+0000","last_change":"2024-03-11T21:34:15.604651+0000","last_active":"2024-03-11T21:35:03.300025+0000","last_peered":"2024-03-11T21:35:03.300025+0000","last_clean":"2024-03-11T21:35:03.300025+0000","last_became_active":"2024-03-11T21:31:52.272382+0000","last_became_peered":"2024-03-11T21:31:52.272382+0000","last_unstale":"2024-03-11T21:35:03.300025+0000","last_undegraded":"2024-03-11T21:35:03.300025+0000","last_fullsized":"2024-03-11T21:35:03.300025+0000","mapping_epoch":157,"log_start":"209'8959","ondisk_log_start":"209'8959","created":157,"last_epoch_clean":158,"parent":"0.0","parent_split_bits":0,"last_scrub":"219'10491","last_scrub_stamp":"2024-03-11T21:34:15.604036+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:31:51.219688+0000","last_clean_scrub_stamp":"2024-03-11T21:34:15.604036+0000","objects_scrubbed":4819,"log_size":4598,"log_dups_size":0,"ondisk_log_size":4598,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:35:40.194213+0000","scrub_duration":1929,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110297088,"num_objects":1683,"num_object_clones":0,"num_object_copies":5049,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1683,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13557,"num_write_kb":487680,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,4],"acting":[0,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.0","version":"0'0","reported_seq":373,"reported_epoch":240,"state":"active+clean","last_fresh":"2024-03-11T21:34:58.347346+0000","last_change":"2024-03-11T21:32:03.422921+0000","last_active":"2024-03-11T21:34:58.347346+0000","last_peered":"2024-03-11T21:34:58.347346+0000","last_clean":"2024-03-11T21:34:58.347346+0000","last_became_active":"2024-03-11T21:25:44.584565+0000","last_became_peered":"2024-03-11T21:25:44.584565+0000","last_unstale":"2024-03-11T21:34:58.347346+0000","last_undegraded":"2024-03-11T21:34:58.347346+0000","last_fullsized":"2024-03-11T21:34:58.347346+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:32:03.422808+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:29:39.498784+0000","last_clean_scrub_stamp":"2024-03-11T21:32:03.422808+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:35:01.803638+0000","scrub_duration":31,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":405,"reported_epoch":240,"state":"active+clean","last_fresh":"2024-03-11T21:34:58.347337+0000","last_change":"2024-03-11T21:34:41.151254+0000","last_active":"2024-03-11T21:34:58.347337+0000","last_peered":"2024-03-11T21:34:58.347337+0000","last_clean":"2024-03-11T21:34:58.347337+0000","last_became_active":"2024-03-11T21:34:41.150913+0000","last_became_peered":"2024-03-11T21:34:41.150913+0000","last_unstale":"2024-03-11T21:34:58.347337+0000","last_undegraded":"2024-03-11T21:34:58.347337+0000","last_fullsized":"2024-03-11T21:34:58.347337+0000","mapping_epoch":226,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":227,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:32:04.509514+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:37.512018+0000","last_clean_scrub_stamp":"2024-03-11T21:32:04.509514+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:35:02.646586+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,7],"acting":[1,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.2","version":"94'15","reported_seq":409,"reported_epoch":240,"state":"active+clean","last_fresh":"2024-03-11T21:34:58.348293+0000","last_change":"2024-03-11T21:31:08.956264+0000","last_active":"2024-03-11T21:34:58.348293+0000","last_peered":"2024-03-11T21:34:58.348293+0000","last_clean":"2024-03-11T21:34:58.348293+0000","last_became_active":"2024-03-11T21:25:40.548833+0000","last_became_peered":"2024-03-11T21:25:40.548833+0000","last_unstale":"2024-03-11T21:34:58.348293+0000","last_undegraded":"2024-03-11T21:34:58.348293+0000","last_fullsized":"2024-03-11T21:34:58.348293+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"94'15","last_scrub_stamp":"2024-03-11T21:31:08.955971+0000","last_deep_scrub":"94'15","last_deep_scrub_stamp":"2024-03-11T21:26:49.993060+0000","last_clean_scrub_stamp":"2024-03-11T21:31:08.955971+0000","objects_scrubbed":5,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:35:02.646403+0000","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.3","version":"74'8","reported_seq":506,"reported_epoch":240,"state":"active+clean","last_fresh":"2024-03-11T21:34:58.350571+0000","last_change":"2024-03-11T21:34:11.753926+0000","last_active":"2024-03-11T21:34:58.350571+0000","last_peered":"2024-03-11T21:34:58.350571+0000","last_clean":"2024-03-11T21:34:58.350571+0000","last_became_active":"2024-03-11T21:25:42.578161+0000","last_became_peered":"2024-03-11T21:25:42.578161+0000","last_unstale":"2024-03-11T21:34:58.350571+0000","last_undegraded":"2024-03-11T21:34:58.350571+0000","last_fullsized":"2024-03-11T21:34:58.350571+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'8","last_scrub_stamp":"2024-03-11T21:34:11.753809+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:41.544550+0000","last_clean_scrub_stamp":"2024-03-11T21:34:11.753809+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:35:12.615444+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,5],"acting":[0,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.8","version":"241'9812","reported_seq":12337,"reported_epoch":241,"state":"active+clean","last_fresh":"2024-03-11T21:35:07.100938+0000","last_change":"2024-03-11T21:34:55.353961+0000","last_active":"2024-03-11T21:35:07.100938+0000","last_peered":"2024-03-11T21:35:07.100938+0000","last_clean":"2024-03-11T21:35:07.100938+0000","last_became_active":"2024-03-11T21:34:55.331174+0000","last_became_peered":"2024-03-11T21:34:55.331174+0000","last_unstale":"2024-03-11T21:35:07.100938+0000","last_undegraded":"2024-03-11T21:35:07.100938+0000","last_fullsized":"2024-03-11T21:35:07.100938+0000","mapping_epoch":236,"log_start":"173'4327","ondisk_log_start":"173'4327","created":157,"last_epoch_clean":237,"parent":"0.0","parent_split_bits":0,"last_scrub":"183'5663","last_scrub_stamp":"2024-03-11T21:32:58.596358+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:31:51.219688+0000","last_clean_scrub_stamp":"2024-03-11T21:32:58.596358+0000","objects_scrubbed":2875,"log_size":5485,"log_dups_size":0,"ondisk_log_size":5485,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":327,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":43253760,"num_objects":660,"num_object_clones":0,"num_object_copies":1980,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":660,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4149,"num_write_kb":61888,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1143,"num_bytes_recovered":74907648,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":384,"reported_epoch":240,"state":"active+clean","last_fresh":"2024-03-11T21:34:58.386505+0000","last_change":"2024-03-11T21:34:41.146703+0000","last_active":"2024-03-11T21:34:58.386505+0000","last_peered":"2024-03-11T21:34:58.386505+0000","last_clean":"2024-03-11T21:34:58.386505+0000","last_became_active":"2024-03-11T21:34:41.146073+0000","last_became_peered":"2024-03-11T21:34:41.146073+0000","last_unstale":"2024-03-11T21:34:58.386505+0000","last_undegraded":"2024-03-11T21:34:58.386505+0000","last_fullsized":"2024-03-11T21:34:58.386505+0000","mapping_epoch":226,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":227,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:31:01.241180+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:28:30.563712+0000","last_clean_scrub_stamp":"2024-03-11T21:31:01.241180+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:35:01.869969+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,7],"acting":[3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":402,"reported_epoch":241,"state":"active+clean","last_fresh":"2024-03-11T21:35:04.397915+0000","last_change":"2024-03-11T21:34:18.388334+0000","last_active":"2024-03-11T21:35:04.397915+0000","last_peered":"2024-03-11T21:35:04.397915+0000","last_clean":"2024-03-11T21:35:04.397915+0000","last_became_active":"2024-03-11T21:25:38.538540+0000","last_became_peered":"2024-03-11T21:25:38.538540+0000","last_unstale":"2024-03-11T21:35:04.397915+0000","last_undegraded":"2024-03-11T21:35:04.397915+0000","last_fullsized":"2024-03-11T21:35:04.397915+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:34:18.388258+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:37.512018+0000","last_clean_scrub_stamp":"2024-03-11T21:34:18.388258+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:35:35.043384+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.4","version":"74'28","reported_seq":550,"reported_epoch":240,"state":"active+clean","last_fresh":"2024-03-11T21:34:58.347287+0000","last_change":"2024-03-11T21:34:19.513419+0000","last_active":"2024-03-11T21:34:58.347287+0000","last_peered":"2024-03-11T21:34:58.347287+0000","last_clean":"2024-03-11T21:34:58.347287+0000","last_became_active":"2024-03-11T21:25:40.552902+0000","last_became_peered":"2024-03-11T21:25:40.552902+0000","last_unstale":"2024-03-11T21:34:58.347287+0000","last_undegraded":"2024-03-11T21:34:58.347287+0000","last_fullsized":"2024-03-11T21:34:58.347287+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'28","last_scrub_stamp":"2024-03-11T21:34:19.513252+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:39.525554+0000","last_clean_scrub_stamp":"2024-03-11T21:34:19.513252+0000","objects_scrubbed":10,"log_size":28,"log_dups_size":0,"ondisk_log_size":28,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:35:34.875894+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":358,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":48,"num_read_kb":33,"num_write":26,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3],"acting":[1,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":385,"reported_epoch":240,"state":"active+clean","last_fresh":"2024-03-11T21:34:58.350543+0000","last_change":"2024-03-11T21:34:18.638688+0000","last_active":"2024-03-11T21:34:58.350543+0000","last_peered":"2024-03-11T21:34:58.350543+0000","last_clean":"2024-03-11T21:34:58.350543+0000","last_became_active":"2024-03-11T21:25:42.572673+0000","last_became_peered":"2024-03-11T21:25:42.572673+0000","last_unstale":"2024-03-11T21:34:58.350543+0000","last_undegraded":"2024-03-11T21:34:58.350543+0000","last_fullsized":"2024-03-11T21:34:58.350543+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:34:18.638566+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:41.544550+0000","last_clean_scrub_stamp":"2024-03-11T21:34:18.638566+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:35:21.043366+0000","scrub_duration":18,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.b","version":"240'13629","reported_seq":13810,"reported_epoch":240,"state":"active+clean","last_fresh":"2024-03-11T21:35:02.772538+0000","last_change":"2024-03-11T21:31:52.241413+0000","last_active":"2024-03-11T21:35:02.772538+0000","last_peered":"2024-03-11T21:35:02.772538+0000","last_clean":"2024-03-11T21:35:02.772538+0000","last_became_active":"2024-03-11T21:31:52.241089+0000","last_became_peered":"2024-03-11T21:31:52.241089+0000","last_unstale":"2024-03-11T21:35:02.772538+0000","last_undegraded":"2024-03-11T21:35:02.772538+0000","last_fullsized":"2024-03-11T21:35:02.772538+0000","mapping_epoch":157,"log_start":"206'8792","ondisk_log_start":"206'8792","created":157,"last_epoch_clean":158,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:31:51.219688+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:31:51.219688+0000","last_clean_scrub_stamp":"2024-03-11T21:31:51.219688+0000","objects_scrubbed":0,"log_size":4837,"log_dups_size":0,"ondisk_log_size":4837,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:35:06.906576+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":113311744,"num_objects":1729,"num_object_clones":0,"num_object_copies":5187,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1729,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13629,"num_write_kb":491456,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,1],"acting":[3,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.5","version":"0'0","reported_seq":39,"reported_epoch":241,"state":"active+clean","last_fresh":"2024-03-11T21:35:04.397749+0000","last_change":"2024-03-11T21:34:41.153269+0000","last_active":"2024-03-11T21:35:04.397749+0000","last_peered":"2024-03-11T21:35:04.397749+0000","last_clean":"2024-03-11T21:35:04.397749+0000","last_became_active":"2024-03-11T21:34:41.151087+0000","last_became_peered":"2024-03-11T21:34:41.151087+0000","last_unstale":"2024-03-11T21:35:04.397749+0000","last_undegraded":"2024-03-11T21:35:04.397749+0000","last_fullsized":"2024-03-11T21:35:04.397749+0000","mapping_epoch":226,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":227,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:32:10.837701+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:43.559452+0000","last_clean_scrub_stamp":"2024-03-11T21:32:10.837701+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:35:08.003072+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,3],"acting":[7,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.0","version":"164'6","reported_seq":961,"reported_epoch":240,"state":"active+clean","last_fresh":"2024-03-11T21:35:01.023394+0000","last_change":"2024-03-11T21:32:17.597798+0000","last_active":"2024-03-11T21:35:01.023394+0000","last_peered":"2024-03-11T21:35:01.023394+0000","last_clean":"2024-03-11T21:35:01.023394+0000","last_became_active":"2024-03-11T21:25:38.536890+0000","last_became_peered":"2024-03-11T21:25:38.536890+0000","last_unstale":"2024-03-11T21:35:01.023394+0000","last_undegraded":"2024-03-11T21:35:01.023394+0000","last_fullsized":"2024-03-11T21:35:01.023394+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"164'6","last_scrub_stamp":"2024-03-11T21:32:17.597666+0000","last_deep_scrub":"164'6","last_deep_scrub_stamp":"2024-03-11T21:32:17.597666+0000","last_clean_scrub_stamp":"2024-03-11T21:32:17.597666+0000","objects_scrubbed":1,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:35:02.646682+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0.000753559,"stat_sum":{"num_bytes":389,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":557,"num_read_kb":552,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,6],"acting":[1,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":402,"reported_epoch":241,"state":"active+clean","last_fresh":"2024-03-11T21:35:04.393987+0000","last_change":"2024-03-11T21:34:18.424114+0000","last_active":"2024-03-11T21:35:04.393987+0000","last_peered":"2024-03-11T21:35:04.393987+0000","last_clean":"2024-03-11T21:35:04.393987+0000","last_became_active":"2024-03-11T21:25:38.532984+0000","last_became_peered":"2024-03-11T21:25:38.532984+0000","last_unstale":"2024-03-11T21:35:04.393987+0000","last_undegraded":"2024-03-11T21:35:04.393987+0000","last_fullsized":"2024-03-11T21:35:04.393987+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:34:18.424010+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:26:48.997454+0000","last_clean_scrub_stamp":"2024-03-11T21:34:18.424010+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:35:44.820528+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.7","version":"97'18","reported_seq":424,"reported_epoch":240,"state":"active+clean","last_fresh":"2024-03-11T21:34:58.349282+0000","last_change":"2024-03-11T21:32:14.594198+0000","last_active":"2024-03-11T21:34:58.349282+0000","last_peered":"2024-03-11T21:34:58.349282+0000","last_clean":"2024-03-11T21:34:58.349282+0000","last_became_active":"2024-03-11T21:25:40.541769+0000","last_became_peered":"2024-03-11T21:25:40.541769+0000","last_unstale":"2024-03-11T21:34:58.349282+0000","last_undegraded":"2024-03-11T21:34:58.349282+0000","last_fullsized":"2024-03-11T21:34:58.349282+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"97'18","last_scrub_stamp":"2024-03-11T21:32:14.593767+0000","last_deep_scrub":"97'18","last_deep_scrub_stamp":"2024-03-11T21:27:19.239404+0000","last_clean_scrub_stamp":"2024-03-11T21:32:14.593767+0000","objects_scrubbed":5,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:35:02.646621+0000","scrub_duration":18,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":19,"num_write":16,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":398,"reported_epoch":241,"state":"active+clean","last_fresh":"2024-03-11T21:35:04.412166+0000","last_change":"2024-03-11T21:34:41.144253+0000","last_active":"2024-03-11T21:35:04.412166+0000","last_peered":"2024-03-11T21:35:04.412166+0000","last_clean":"2024-03-11T21:35:04.412166+0000","last_became_active":"2024-03-11T21:34:41.143660+0000","last_became_peered":"2024-03-11T21:34:41.143660+0000","last_unstale":"2024-03-11T21:35:04.412166+0000","last_undegraded":"2024-03-11T21:35:04.412166+0000","last_fullsized":"2024-03-11T21:35:04.412166+0000","mapping_epoch":226,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":227,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:34:07.290464+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:41.544550+0000","last_clean_scrub_stamp":"2024-03-11T21:34:07.290464+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:35:15.129200+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7],"acting":[2,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"8.9","version":"241'9815","reported_seq":10039,"reported_epoch":241,"state":"active+clean","last_fresh":"2024-03-11T21:35:07.109764+0000","last_change":"2024-03-11T21:32:58.396514+0000","last_active":"2024-03-11T21:35:07.109764+0000","last_peered":"2024-03-11T21:35:07.109764+0000","last_clean":"2024-03-11T21:35:07.109764+0000","last_became_active":"2024-03-11T21:32:57.085769+0000","last_became_peered":"2024-03-11T21:32:57.085769+0000","last_unstale":"2024-03-11T21:35:07.109764+0000","last_undegraded":"2024-03-11T21:35:07.109764+0000","last_fullsized":"2024-03-11T21:35:07.109764+0000","mapping_epoch":157,"log_start":"169'3327","ondisk_log_start":"169'3327","created":157,"last_epoch_clean":182,"parent":"0.0","parent_split_bits":0,"last_scrub":"182'5644","last_scrub_stamp":"2024-03-11T21:32:58.396310+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:31:51.219688+0000","last_clean_scrub_stamp":"2024-03-11T21:32:58.396310+0000","objects_scrubbed":2893,"log_size":6488,"log_dups_size":0,"ondisk_log_size":6488,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":386,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":43122688,"num_objects":658,"num_object_clones":0,"num_object_copies":1974,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":658,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4171,"num_write_kb":61952,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,0],"acting":[6,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.7","version":"124'2","reported_seq":376,"reported_epoch":241,"state":"active+clean","last_fresh":"2024-03-11T21:35:04.398733+0000","last_change":"2024-03-11T21:32:09.689578+0000","last_active":"2024-03-11T21:35:04.398733+0000","last_peered":"2024-03-11T21:35:04.398733+0000","last_clean":"2024-03-11T21:35:04.398733+0000","last_became_active":"2024-03-11T21:25:44.591376+0000","last_became_peered":"2024-03-11T21:25:44.591376+0000","last_unstale":"2024-03-11T21:35:04.398733+0000","last_undegraded":"2024-03-11T21:35:04.398733+0000","last_fullsized":"2024-03-11T21:35:04.398733+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"124'2","last_scrub_stamp":"2024-03-11T21:32:09.689480+0000","last_deep_scrub":"124'2","last_deep_scrub_stamp":"2024-03-11T21:29:34.956908+0000","last_clean_scrub_stamp":"2024-03-11T21:32:09.689480+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":48,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,4],"acting":[5,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2","version":"71'2","reported_seq":401,"reported_epoch":240,"state":"active+clean","last_fresh":"2024-03-11T21:34:58.385738+0000","last_change":"2024-03-11T21:32:21.543178+0000","last_active":"2024-03-11T21:34:58.385738+0000","last_peered":"2024-03-11T21:34:58.385738+0000","last_clean":"2024-03-11T21:34:58.385738+0000","last_became_active":"2024-03-11T21:25:38.539413+0000","last_became_peered":"2024-03-11T21:25:38.539413+0000","last_unstale":"2024-03-11T21:34:58.385738+0000","last_undegraded":"2024-03-11T21:34:58.385738+0000","last_fullsized":"2024-03-11T21:34:58.385738+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"71'2","last_scrub_stamp":"2024-03-11T21:32:21.543053+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:37.512018+0000","last_clean_scrub_stamp":"2024-03-11T21:32:21.543053+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:35:01.870315+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0.0014206449999999999,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":412,"reported_epoch":241,"state":"active+clean","last_fresh":"2024-03-11T21:35:04.401114+0000","last_change":"2024-03-11T21:34:41.147000+0000","last_active":"2024-03-11T21:35:04.401114+0000","last_peered":"2024-03-11T21:35:04.401114+0000","last_clean":"2024-03-11T21:35:04.401114+0000","last_became_active":"2024-03-11T21:34:41.146672+0000","last_became_peered":"2024-03-11T21:34:41.146672+0000","last_unstale":"2024-03-11T21:35:04.401114+0000","last_undegraded":"2024-03-11T21:35:04.401114+0000","last_fullsized":"2024-03-11T21:35:04.401114+0000","mapping_epoch":226,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":227,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:32:01.617459+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:37.512018+0000","last_clean_scrub_stamp":"2024-03-11T21:32:01.617459+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"87'128","reported_seq":510,"reported_epoch":241,"state":"active+clean","last_fresh":"2024-03-11T21:35:04.396441+0000","last_change":"2024-03-11T21:34:44.165013+0000","last_active":"2024-03-11T21:35:04.396441+0000","last_peered":"2024-03-11T21:35:04.396441+0000","last_clean":"2024-03-11T21:35:04.396441+0000","last_became_active":"2024-03-11T21:34:44.164569+0000","last_became_peered":"2024-03-11T21:34:44.164569+0000","last_unstale":"2024-03-11T21:35:04.396441+0000","last_undegraded":"2024-03-11T21:35:04.396441+0000","last_fullsized":"2024-03-11T21:35:04.396441+0000","mapping_epoch":229,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":230,"parent":"0.0","parent_split_bits":0,"last_scrub":"87'128","last_scrub_stamp":"2024-03-11T21:32:20.540647+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:24.261974+0000","last_clean_scrub_stamp":"2024-03-11T21:32:20.540647+0000","objects_scrubbed":2,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:35:08.003101+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":198,"num_read_kb":294,"num_write":179,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":2951840,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.5","version":"74'16","reported_seq":535,"reported_epoch":241,"state":"active+clean","last_fresh":"2024-03-11T21:35:04.400653+0000","last_change":"2024-03-11T21:34:24.320823+0000","last_active":"2024-03-11T21:35:04.400653+0000","last_peered":"2024-03-11T21:35:04.400653+0000","last_clean":"2024-03-11T21:35:04.400653+0000","last_became_active":"2024-03-11T21:25:40.761457+0000","last_became_peered":"2024-03-11T21:25:40.761457+0000","last_unstale":"2024-03-11T21:35:04.400653+0000","last_undegraded":"2024-03-11T21:35:04.400653+0000","last_fullsized":"2024-03-11T21:35:04.400653+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'16","last_scrub_stamp":"2024-03-11T21:34:24.320713+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:39.525554+0000","last_clean_scrub_stamp":"2024-03-11T21:34:24.320713+0000","objects_scrubbed":8,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:35:49.716597+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":154,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":25,"num_read_kb":15,"num_write":13,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":39,"reported_epoch":241,"state":"active+clean","last_fresh":"2024-03-11T21:35:04.396810+0000","last_change":"2024-03-11T21:34:41.141916+0000","last_active":"2024-03-11T21:35:04.396810+0000","last_peered":"2024-03-11T21:35:04.396810+0000","last_clean":"2024-03-11T21:35:04.396810+0000","last_became_active":"2024-03-11T21:34:41.139782+0000","last_became_peered":"2024-03-11T21:34:41.139782+0000","last_unstale":"2024-03-11T21:35:04.396810+0000","last_undegraded":"2024-03-11T21:35:04.396810+0000","last_fullsized":"2024-03-11T21:35:04.396810+0000","mapping_epoch":226,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":227,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:32:21.266672+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:32:21.266672+0000","last_clean_scrub_stamp":"2024-03-11T21:32:21.266672+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:35:08.003137+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,5],"acting":[7,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.a","version":"240'13419","reported_seq":13598,"reported_epoch":240,"state":"active+clean","last_fresh":"2024-03-11T21:35:02.406063+0000","last_change":"2024-03-11T21:31:52.272724+0000","last_active":"2024-03-11T21:35:02.406063+0000","last_peered":"2024-03-11T21:35:02.406063+0000","last_clean":"2024-03-11T21:35:02.406063+0000","last_became_active":"2024-03-11T21:31:52.271977+0000","last_became_peered":"2024-03-11T21:31:52.271977+0000","last_unstale":"2024-03-11T21:35:02.406063+0000","last_undegraded":"2024-03-11T21:35:02.406063+0000","last_fullsized":"2024-03-11T21:35:02.406063+0000","mapping_epoch":157,"log_start":"207'8653","ondisk_log_start":"207'8653","created":157,"last_epoch_clean":158,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:31:51.219688+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:31:51.219688+0000","last_clean_scrub_stamp":"2024-03-11T21:31:51.219688+0000","objects_scrubbed":0,"log_size":4766,"log_dups_size":0,"ondisk_log_size":4766,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:35:02.646470+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":108462080,"num_objects":1655,"num_object_clones":0,"num_object_copies":4965,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1655,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13419,"num_write_kb":482368,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,5],"acting":[1,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.4","version":"0'0","reported_seq":381,"reported_epoch":240,"state":"active+clean","last_fresh":"2024-03-11T21:34:58.348732+0000","last_change":"2024-03-11T21:34:23.531211+0000","last_active":"2024-03-11T21:34:58.348732+0000","last_peered":"2024-03-11T21:34:58.348732+0000","last_clean":"2024-03-11T21:34:58.348732+0000","last_became_active":"2024-03-11T21:25:44.594188+0000","last_became_peered":"2024-03-11T21:25:44.594188+0000","last_unstale":"2024-03-11T21:34:58.348732+0000","last_undegraded":"2024-03-11T21:34:58.348732+0000","last_fullsized":"2024-03-11T21:34:58.348732+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:34:23.531064+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:43.559452+0000","last_clean_scrub_stamp":"2024-03-11T21:34:23.531064+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:35:45.276874+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":400,"reported_epoch":240,"state":"active+clean","last_fresh":"2024-03-11T21:34:58.348630+0000","last_change":"2024-03-11T21:32:22.561581+0000","last_active":"2024-03-11T21:34:58.348630+0000","last_peered":"2024-03-11T21:34:58.348630+0000","last_clean":"2024-03-11T21:34:58.348630+0000","last_became_active":"2024-03-11T21:25:38.543834+0000","last_became_peered":"2024-03-11T21:25:38.543834+0000","last_unstale":"2024-03-11T21:34:58.348630+0000","last_undegraded":"2024-03-11T21:34:58.348630+0000","last_fullsized":"2024-03-11T21:34:58.348630+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:32:22.560895+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:37.512018+0000","last_clean_scrub_stamp":"2024-03-11T21:32:22.560895+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:35:01.803856+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0.00089342499999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":39,"reported_epoch":241,"state":"active+clean","last_fresh":"2024-03-11T21:35:04.396410+0000","last_change":"2024-03-11T21:34:41.150606+0000","last_active":"2024-03-11T21:35:04.396410+0000","last_peered":"2024-03-11T21:35:04.396410+0000","last_clean":"2024-03-11T21:35:04.396410+0000","last_became_active":"2024-03-11T21:34:41.149219+0000","last_became_peered":"2024-03-11T21:34:41.149219+0000","last_unstale":"2024-03-11T21:35:04.396410+0000","last_undegraded":"2024-03-11T21:35:04.396410+0000","last_fullsized":"2024-03-11T21:35:04.396410+0000","mapping_epoch":226,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":227,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:34:31.628014+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:37.512018+0000","last_clean_scrub_stamp":"2024-03-11T21:34:31.628014+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:35:57.776832+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,0],"acting":[7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.6","version":"74'12","reported_seq":427,"reported_epoch":240,"state":"active+clean","last_fresh":"2024-03-11T21:34:58.350329+0000","last_change":"2024-03-11T21:32:58.559118+0000","last_active":"2024-03-11T21:34:58.350329+0000","last_peered":"2024-03-11T21:34:58.350329+0000","last_clean":"2024-03-11T21:34:58.350329+0000","last_became_active":"2024-03-11T21:32:22.796486+0000","last_became_peered":"2024-03-11T21:32:22.796486+0000","last_unstale":"2024-03-11T21:34:58.350329+0000","last_undegraded":"2024-03-11T21:34:58.350329+0000","last_fullsized":"2024-03-11T21:34:58.350329+0000","mapping_epoch":166,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":167,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'12","last_scrub_stamp":"2024-03-11T21:32:58.559016+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:39.525554+0000","last_clean_scrub_stamp":"2024-03-11T21:32:58.559016+0000","objects_scrubbed":6,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:35:01.803927+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":18,"num_read_kb":12,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":386,"reported_epoch":241,"state":"active+clean","last_fresh":"2024-03-11T21:35:04.398641+0000","last_change":"2024-03-11T21:34:25.411699+0000","last_active":"2024-03-11T21:35:04.398641+0000","last_peered":"2024-03-11T21:35:04.398641+0000","last_clean":"2024-03-11T21:35:04.398641+0000","last_became_active":"2024-03-11T21:25:42.572324+0000","last_became_peered":"2024-03-11T21:25:42.572324+0000","last_unstale":"2024-03-11T21:35:04.398641+0000","last_undegraded":"2024-03-11T21:35:04.398641+0000","last_fullsized":"2024-03-11T21:35:04.398641+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:34:25.411598+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:41.544550+0000","last_clean_scrub_stamp":"2024-03-11T21:34:25.411598+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:35:46.761784+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.f","version":"240'13188","reported_seq":13368,"reported_epoch":240,"state":"active+clean","last_fresh":"2024-03-11T21:35:02.765617+0000","last_change":"2024-03-11T21:31:52.272528+0000","last_active":"2024-03-11T21:35:02.765617+0000","last_peered":"2024-03-11T21:35:02.765617+0000","last_clean":"2024-03-11T21:35:02.765617+0000","last_became_active":"2024-03-11T21:31:52.272191+0000","last_became_peered":"2024-03-11T21:31:52.272191+0000","last_unstale":"2024-03-11T21:35:02.765617+0000","last_undegraded":"2024-03-11T21:35:02.765617+0000","last_fullsized":"2024-03-11T21:35:02.765617+0000","mapping_epoch":157,"log_start":"205'8420","ondisk_log_start":"205'8420","created":157,"last_epoch_clean":158,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:31:51.219688+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:31:51.219688+0000","last_clean_scrub_stamp":"2024-03-11T21:31:51.219688+0000","objects_scrubbed":0,"log_size":4768,"log_dups_size":0,"ondisk_log_size":4768,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:35:06.906541+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":106692608,"num_objects":1628,"num_object_clones":0,"num_object_copies":4884,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1628,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13188,"num_write_kb":474112,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,1],"acting":[3,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":389,"reported_epoch":240,"state":"active+clean","last_fresh":"2024-03-11T21:34:58.347609+0000","last_change":"2024-03-11T21:34:33.565178+0000","last_active":"2024-03-11T21:34:58.347609+0000","last_peered":"2024-03-11T21:34:58.347609+0000","last_clean":"2024-03-11T21:34:58.347609+0000","last_became_active":"2024-03-11T21:25:45.301416+0000","last_became_peered":"2024-03-11T21:25:45.301416+0000","last_unstale":"2024-03-11T21:34:58.347609+0000","last_undegraded":"2024-03-11T21:34:58.347609+0000","last_fullsized":"2024-03-11T21:34:58.347609+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:34:33.564652+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:43.559452+0000","last_clean_scrub_stamp":"2024-03-11T21:34:33.564652+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:35:48.030891+0000","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":39,"reported_epoch":241,"state":"active+clean","last_fresh":"2024-03-11T21:35:04.398006+0000","last_change":"2024-03-11T21:34:41.155790+0000","last_active":"2024-03-11T21:35:04.398006+0000","last_peered":"2024-03-11T21:35:04.398006+0000","last_clean":"2024-03-11T21:35:04.398006+0000","last_became_active":"2024-03-11T21:34:41.152721+0000","last_became_peered":"2024-03-11T21:34:41.152721+0000","last_unstale":"2024-03-11T21:35:04.398006+0000","last_undegraded":"2024-03-11T21:35:04.398006+0000","last_fullsized":"2024-03-11T21:35:04.398006+0000","mapping_epoch":226,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":227,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:34:16.591255+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:26:57.333669+0000","last_clean_scrub_stamp":"2024-03-11T21:34:16.591255+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:35:24.096418+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.3","version":"165'27","reported_seq":481,"reported_epoch":240,"state":"active+clean","last_fresh":"2024-03-11T21:34:58.350236+0000","last_change":"2024-03-11T21:34:41.168911+0000","last_active":"2024-03-11T21:34:58.350236+0000","last_peered":"2024-03-11T21:34:58.350236+0000","last_clean":"2024-03-11T21:34:58.350236+0000","last_became_active":"2024-03-11T21:34:41.145607+0000","last_became_peered":"2024-03-11T21:34:41.145607+0000","last_unstale":"2024-03-11T21:34:58.350236+0000","last_undegraded":"2024-03-11T21:34:58.350236+0000","last_fullsized":"2024-03-11T21:34:58.350236+0000","mapping_epoch":226,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":227,"parent":"0.0","parent_split_bits":0,"last_scrub":"165'27","last_scrub_stamp":"2024-03-11T21:32:21.576732+0000","last_deep_scrub":"165'27","last_deep_scrub_stamp":"2024-03-11T21:32:21.576732+0000","last_clean_scrub_stamp":"2024-03-11T21:32:21.576732+0000","objects_scrubbed":8,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:35:01.803794+0000","scrub_duration":23,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":39,"num_read_kb":25,"num_write":22,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":660,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":395,"reported_epoch":241,"state":"active+clean","last_fresh":"2024-03-11T21:35:04.399981+0000","last_change":"2024-03-11T21:32:23.882539+0000","last_active":"2024-03-11T21:35:04.399981+0000","last_peered":"2024-03-11T21:35:04.399981+0000","last_clean":"2024-03-11T21:35:04.399981+0000","last_became_active":"2024-03-11T21:25:42.579045+0000","last_became_peered":"2024-03-11T21:25:42.579045+0000","last_unstale":"2024-03-11T21:35:04.399981+0000","last_undegraded":"2024-03-11T21:35:04.399981+0000","last_fullsized":"2024-03-11T21:35:04.399981+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:32:23.882391+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:29:36.918936+0000","last_clean_scrub_stamp":"2024-03-11T21:32:23.882391+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"8.c","version":"240'13239","reported_seq":18069,"reported_epoch":240,"state":"active+clean","last_fresh":"2024-03-11T21:35:03.257132+0000","last_change":"2024-03-11T21:34:56.357171+0000","last_active":"2024-03-11T21:35:03.257132+0000","last_peered":"2024-03-11T21:35:03.257132+0000","last_clean":"2024-03-11T21:35:03.257132+0000","last_became_active":"2024-03-11T21:34:56.333422+0000","last_became_peered":"2024-03-11T21:34:56.333422+0000","last_unstale":"2024-03-11T21:35:03.257132+0000","last_undegraded":"2024-03-11T21:35:03.257132+0000","last_fullsized":"2024-03-11T21:35:03.257132+0000","mapping_epoch":237,"log_start":"207'8633","ondisk_log_start":"207'8633","created":157,"last_epoch_clean":238,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:31:51.219688+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:31:51.219688+0000","last_clean_scrub_stamp":"2024-03-11T21:31:51.219688+0000","objects_scrubbed":0,"log_size":4606,"log_dups_size":0,"ondisk_log_size":4606,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":107675648,"num_objects":1643,"num_object_clones":0,"num_object_copies":4929,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1643,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13239,"num_write_kb":476224,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2318,"num_bytes_recovered":151912448,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":389,"reported_epoch":241,"state":"active+clean","last_fresh":"2024-03-11T21:35:04.388839+0000","last_change":"2024-03-11T21:32:22.791904+0000","last_active":"2024-03-11T21:35:04.388839+0000","last_peered":"2024-03-11T21:35:04.388839+0000","last_clean":"2024-03-11T21:35:04.388839+0000","last_became_active":"2024-03-11T21:32:22.791439+0000","last_became_peered":"2024-03-11T21:32:22.791439+0000","last_unstale":"2024-03-11T21:35:04.388839+0000","last_undegraded":"2024-03-11T21:35:04.388839+0000","last_fullsized":"2024-03-11T21:35:04.388839+0000","mapping_epoch":166,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":167,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:32:05.636174+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:43.559452+0000","last_clean_scrub_stamp":"2024-03-11T21:32:05.636174+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:35:04.446487+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,2],"acting":[4,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.6","version":"67'1","reported_seq":394,"reported_epoch":240,"state":"active+clean","last_fresh":"2024-03-11T21:34:58.347776+0000","last_change":"2024-03-11T21:32:05.556583+0000","last_active":"2024-03-11T21:34:58.347776+0000","last_peered":"2024-03-11T21:34:58.347776+0000","last_clean":"2024-03-11T21:34:58.347776+0000","last_became_active":"2024-03-11T21:25:38.536142+0000","last_became_peered":"2024-03-11T21:25:38.536142+0000","last_unstale":"2024-03-11T21:34:58.347776+0000","last_undegraded":"2024-03-11T21:34:58.347776+0000","last_fullsized":"2024-03-11T21:34:58.347776+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'1","last_scrub_stamp":"2024-03-11T21:32:05.556427+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:37.512018+0000","last_clean_scrub_stamp":"2024-03-11T21:32:05.556427+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:35:02.646527+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,4],"acting":[1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.0","version":"93'26","reported_seq":476,"reported_epoch":240,"state":"active+clean","last_fresh":"2024-03-11T21:34:58.385687+0000","last_change":"2024-03-11T21:34:41.181772+0000","last_active":"2024-03-11T21:34:58.385687+0000","last_peered":"2024-03-11T21:34:58.385687+0000","last_clean":"2024-03-11T21:34:58.385687+0000","last_became_active":"2024-03-11T21:34:41.145354+0000","last_became_peered":"2024-03-11T21:34:41.145354+0000","last_unstale":"2024-03-11T21:34:58.385687+0000","last_undegraded":"2024-03-11T21:34:58.385687+0000","last_fullsized":"2024-03-11T21:34:58.385687+0000","mapping_epoch":226,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":227,"parent":"0.0","parent_split_bits":0,"last_scrub":"93'26","last_scrub_stamp":"2024-03-11T21:34:16.863984+0000","last_deep_scrub":"93'26","last_deep_scrub_stamp":"2024-03-11T21:26:40.680324+0000","last_clean_scrub_stamp":"2024-03-11T21:34:16.863984+0000","objects_scrubbed":8,"log_size":26,"log_dups_size":0,"ondisk_log_size":26,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:35:24.709288+0000","scrub_duration":22,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":34,"num_read_kb":22,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":440,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":399,"reported_epoch":241,"state":"active+clean","last_fresh":"2024-03-11T21:35:04.388604+0000","last_change":"2024-03-11T21:34:41.135951+0000","last_active":"2024-03-11T21:35:04.388604+0000","last_peered":"2024-03-11T21:35:04.388604+0000","last_clean":"2024-03-11T21:35:04.388604+0000","last_became_active":"2024-03-11T21:34:41.135551+0000","last_became_peered":"2024-03-11T21:34:41.135551+0000","last_unstale":"2024-03-11T21:35:04.388604+0000","last_undegraded":"2024-03-11T21:35:04.388604+0000","last_fullsized":"2024-03-11T21:35:04.388604+0000","mapping_epoch":226,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":227,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:32:16.599049+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:26:51.863621+0000","last_clean_scrub_stamp":"2024-03-11T21:32:16.599049+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:35:04.446541+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,7],"acting":[4,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"8.d","version":"241'13823","reported_seq":18392,"reported_epoch":241,"state":"active+clean","last_fresh":"2024-03-11T21:35:07.103918+0000","last_change":"2024-03-11T21:35:04.857549+0000","last_active":"2024-03-11T21:35:07.103918+0000","last_peered":"2024-03-11T21:35:07.103918+0000","last_clean":"2024-03-11T21:35:07.103918+0000","last_became_active":"2024-03-11T21:34:58.356843+0000","last_became_peered":"2024-03-11T21:34:58.356843+0000","last_unstale":"2024-03-11T21:35:07.103918+0000","last_undegraded":"2024-03-11T21:35:07.103918+0000","last_fullsized":"2024-03-11T21:35:07.103918+0000","mapping_epoch":239,"log_start":"207'8890","ondisk_log_start":"207'8890","created":157,"last_epoch_clean":240,"parent":"0.0","parent_split_bits":0,"last_scrub":"241'13659","last_scrub_stamp":"2024-03-11T21:35:04.857337+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:31:51.219688+0000","last_clean_scrub_stamp":"2024-03-11T21:35:04.857337+0000","objects_scrubbed":1534,"log_size":4933,"log_dups_size":0,"ondisk_log_size":4933,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:36:19.520848+0000","scrub_duration":449,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":88604696,"num_objects":1353,"num_object_clones":0,"num_object_copies":4059,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1353,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":13823,"num_write_kb":485569,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2173,"num_bytes_recovered":142344216,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,2],"acting":[5,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.3","version":"0'0","reported_seq":39,"reported_epoch":241,"state":"active+clean","last_fresh":"2024-03-11T21:35:04.397536+0000","last_change":"2024-03-11T21:34:41.143973+0000","last_active":"2024-03-11T21:35:04.397536+0000","last_peered":"2024-03-11T21:35:04.397536+0000","last_clean":"2024-03-11T21:35:04.397536+0000","last_became_active":"2024-03-11T21:34:41.142683+0000","last_became_peered":"2024-03-11T21:34:41.142683+0000","last_unstale":"2024-03-11T21:35:04.397536+0000","last_undegraded":"2024-03-11T21:35:04.397536+0000","last_fullsized":"2024-03-11T21:35:04.397536+0000","mapping_epoch":226,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":227,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:34:27.353700+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:29:42.485740+0000","last_clean_scrub_stamp":"2024-03-11T21:34:27.353700+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:35:56.768836+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,2],"acting":[7,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":415,"reported_epoch":241,"state":"active+clean","last_fresh":"2024-03-11T21:35:04.400186+0000","last_change":"2024-03-11T21:34:41.146260+0000","last_active":"2024-03-11T21:35:04.400186+0000","last_peered":"2024-03-11T21:35:04.400186+0000","last_clean":"2024-03-11T21:35:04.400186+0000","last_became_active":"2024-03-11T21:34:41.145628+0000","last_became_peered":"2024-03-11T21:34:41.145628+0000","last_unstale":"2024-03-11T21:35:04.400186+0000","last_undegraded":"2024-03-11T21:35:04.400186+0000","last_fullsized":"2024-03-11T21:35:04.400186+0000","mapping_epoch":226,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":227,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:34:10.437356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:37.512018+0000","last_clean_scrub_stamp":"2024-03-11T21:34:10.437356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:35:20.602470+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,2],"acting":[6,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.1","version":"74'14","reported_seq":412,"reported_epoch":241,"state":"active+clean","last_fresh":"2024-03-11T21:35:04.388500+0000","last_change":"2024-03-11T21:32:58.900322+0000","last_active":"2024-03-11T21:35:04.388500+0000","last_peered":"2024-03-11T21:35:04.388500+0000","last_clean":"2024-03-11T21:35:04.388500+0000","last_became_active":"2024-03-11T21:25:40.546219+0000","last_became_peered":"2024-03-11T21:25:40.546219+0000","last_unstale":"2024-03-11T21:35:04.388500+0000","last_undegraded":"2024-03-11T21:35:04.388500+0000","last_fullsized":"2024-03-11T21:35:04.388500+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'14","last_scrub_stamp":"2024-03-11T21:32:58.900223+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:39.525554+0000","last_clean_scrub_stamp":"2024-03-11T21:32:58.900223+0000","objects_scrubbed":7,"log_size":14,"log_dups_size":0,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:35:04.446713+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":21,"num_read_kb":14,"num_write":14,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6],"acting":[4,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.0","version":"98'9","reported_seq":498,"reported_epoch":240,"state":"active+clean","last_fresh":"2024-03-11T21:34:58.385733+0000","last_change":"2024-03-11T21:31:21.562516+0000","last_active":"2024-03-11T21:34:58.385733+0000","last_peered":"2024-03-11T21:34:58.385733+0000","last_clean":"2024-03-11T21:34:58.385733+0000","last_became_active":"2024-03-11T21:25:42.576126+0000","last_became_peered":"2024-03-11T21:25:42.576126+0000","last_unstale":"2024-03-11T21:34:58.385733+0000","last_undegraded":"2024-03-11T21:34:58.385733+0000","last_fullsized":"2024-03-11T21:34:58.385733+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"98'9","last_scrub_stamp":"2024-03-11T21:31:21.562395+0000","last_deep_scrub":"98'9","last_deep_scrub_stamp":"2024-03-11T21:27:29.745711+0000","last_clean_scrub_stamp":"2024-03-11T21:31:21.562395+0000","objects_scrubbed":1,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:35:01.870083+0000","scrub_duration":17,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.2","version":"241'9678","reported_seq":9901,"reported_epoch":241,"state":"active+clean","last_fresh":"2024-03-11T21:35:07.024518+0000","last_change":"2024-03-11T21:32:58.359357+0000","last_active":"2024-03-11T21:35:07.024518+0000","last_peered":"2024-03-11T21:35:07.024518+0000","last_clean":"2024-03-11T21:35:07.024518+0000","last_became_active":"2024-03-11T21:32:57.078777+0000","last_became_peered":"2024-03-11T21:32:57.078777+0000","last_unstale":"2024-03-11T21:35:07.024518+0000","last_undegraded":"2024-03-11T21:35:07.024518+0000","last_fullsized":"2024-03-11T21:35:07.024518+0000","mapping_epoch":157,"log_start":"173'4418","ondisk_log_start":"173'4418","created":157,"last_epoch_clean":182,"parent":"0.0","parent_split_bits":0,"last_scrub":"182'5608","last_scrub_stamp":"2024-03-11T21:32:58.359183+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:31:51.219688+0000","last_clean_scrub_stamp":"2024-03-11T21:32:58.359183+0000","objects_scrubbed":2818,"log_size":5260,"log_dups_size":0,"ondisk_log_size":5260,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":474,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":42860544,"num_objects":654,"num_object_clones":0,"num_object_copies":1962,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":654,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4072,"num_write_kb":61056,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,5],"acting":[4,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.c","version":"0'0","reported_seq":372,"reported_epoch":240,"state":"active+clean","last_fresh":"2024-03-11T21:34:58.385908+0000","last_change":"2024-03-11T21:32:17.600335+0000","last_active":"2024-03-11T21:34:58.385908+0000","last_peered":"2024-03-11T21:34:58.385908+0000","last_clean":"2024-03-11T21:34:58.385908+0000","last_became_active":"2024-03-11T21:25:45.302207+0000","last_became_peered":"2024-03-11T21:25:45.302207+0000","last_unstale":"2024-03-11T21:34:58.385908+0000","last_undegraded":"2024-03-11T21:34:58.385908+0000","last_fullsized":"2024-03-11T21:34:58.385908+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:32:17.600213+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:43.559452+0000","last_clean_scrub_stamp":"2024-03-11T21:32:17.600213+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:35:01.870286+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.f","version":"0'0","reported_seq":394,"reported_epoch":241,"state":"active+clean","last_fresh":"2024-03-11T21:35:04.398427+0000","last_change":"2024-03-11T21:34:26.357068+0000","last_active":"2024-03-11T21:35:04.398427+0000","last_peered":"2024-03-11T21:35:04.398427+0000","last_clean":"2024-03-11T21:35:04.398427+0000","last_became_active":"2024-03-11T21:25:42.575935+0000","last_became_peered":"2024-03-11T21:25:42.575935+0000","last_unstale":"2024-03-11T21:35:04.398427+0000","last_undegraded":"2024-03-11T21:35:04.398427+0000","last_fullsized":"2024-03-11T21:35:04.398427+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:34:26.356954+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:41.544550+0000","last_clean_scrub_stamp":"2024-03-11T21:34:26.356954+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:35:54.552150+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.8","version":"0'0","reported_seq":39,"reported_epoch":241,"state":"active+clean","last_fresh":"2024-03-11T21:35:04.397085+0000","last_change":"2024-03-11T21:34:41.145417+0000","last_active":"2024-03-11T21:35:04.397085+0000","last_peered":"2024-03-11T21:35:04.397085+0000","last_clean":"2024-03-11T21:35:04.397085+0000","last_became_active":"2024-03-11T21:34:41.144681+0000","last_became_peered":"2024-03-11T21:34:41.144681+0000","last_unstale":"2024-03-11T21:35:04.397085+0000","last_undegraded":"2024-03-11T21:35:04.397085+0000","last_fullsized":"2024-03-11T21:35:04.397085+0000","mapping_epoch":226,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":227,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:34:23.711140+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:37.512018+0000","last_clean_scrub_stamp":"2024-03-11T21:34:23.711140+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:35:29.880606+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,1],"acting":[7,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.e","version":"109'16","reported_seq":415,"reported_epoch":241,"state":"active+clean","last_fresh":"2024-03-11T21:35:04.389000+0000","last_change":"2024-03-11T21:32:53.834261+0000","last_active":"2024-03-11T21:35:04.389000+0000","last_peered":"2024-03-11T21:35:04.389000+0000","last_clean":"2024-03-11T21:35:04.389000+0000","last_became_active":"2024-03-11T21:25:40.552178+0000","last_became_peered":"2024-03-11T21:25:40.552178+0000","last_unstale":"2024-03-11T21:35:04.389000+0000","last_undegraded":"2024-03-11T21:35:04.389000+0000","last_fullsized":"2024-03-11T21:35:04.389000+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"109'16","last_scrub_stamp":"2024-03-11T21:32:53.834165+0000","last_deep_scrub":"109'16","last_deep_scrub_stamp":"2024-03-11T21:28:24.739395+0000","last_clean_scrub_stamp":"2024-03-11T21:32:53.834165+0000","objects_scrubbed":5,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:35:04.446747+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"8.3","version":"240'9398","reported_seq":9618,"reported_epoch":240,"state":"active+clean","last_fresh":"2024-03-11T21:35:02.279023+0000","last_change":"2024-03-11T21:34:09.073683+0000","last_active":"2024-03-11T21:35:02.279023+0000","last_peered":"2024-03-11T21:35:02.279023+0000","last_clean":"2024-03-11T21:35:02.279023+0000","last_became_active":"2024-03-11T21:32:57.081711+0000","last_became_peered":"2024-03-11T21:32:57.081711+0000","last_unstale":"2024-03-11T21:35:02.279023+0000","last_undegraded":"2024-03-11T21:35:02.279023+0000","last_fullsized":"2024-03-11T21:35:02.279023+0000","mapping_epoch":157,"log_start":"171'3752","ondisk_log_start":"171'3752","created":157,"last_epoch_clean":182,"parent":"0.0","parent_split_bits":0,"last_scrub":"218'7578","last_scrub_stamp":"2024-03-11T21:34:09.073424+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:31:51.219688+0000","last_clean_scrub_stamp":"2024-03-11T21:34:09.073424+0000","objects_scrubbed":2660,"log_size":5646,"log_dups_size":0,"ondisk_log_size":5646,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:35:12.983280+0000","scrub_duration":534,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":54853632,"num_objects":837,"num_object_clones":0,"num_object_copies":2511,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":837,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1823,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,6],"acting":[1,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.d","version":"0'0","reported_seq":382,"reported_epoch":241,"state":"active+clean","last_fresh":"2024-03-11T21:35:04.398607+0000","last_change":"2024-03-11T21:32:13.641959+0000","last_active":"2024-03-11T21:35:04.398607+0000","last_peered":"2024-03-11T21:35:04.398607+0000","last_clean":"2024-03-11T21:35:04.398607+0000","last_became_active":"2024-03-11T21:25:44.590333+0000","last_became_peered":"2024-03-11T21:25:44.590333+0000","last_unstale":"2024-03-11T21:35:04.398607+0000","last_undegraded":"2024-03-11T21:35:04.398607+0000","last_fullsized":"2024-03-11T21:35:04.398607+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:32:13.641870+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:43.559452+0000","last_clean_scrub_stamp":"2024-03-11T21:32:13.641870+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":45,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.e","version":"74'8","reported_seq":509,"reported_epoch":241,"state":"active+clean","last_fresh":"2024-03-11T21:35:04.389147+0000","last_change":"2024-03-11T21:34:18.722515+0000","last_active":"2024-03-11T21:35:04.389147+0000","last_peered":"2024-03-11T21:35:04.389147+0000","last_clean":"2024-03-11T21:35:04.389147+0000","last_became_active":"2024-03-11T21:25:42.576742+0000","last_became_peered":"2024-03-11T21:25:42.576742+0000","last_unstale":"2024-03-11T21:35:04.389147+0000","last_undegraded":"2024-03-11T21:35:04.389147+0000","last_fullsized":"2024-03-11T21:35:04.389147+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'8","last_scrub_stamp":"2024-03-11T21:34:18.722196+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:41.544550+0000","last_clean_scrub_stamp":"2024-03-11T21:34:18.722196+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:35:36.062991+0000","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,0],"acting":[4,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.9","version":"0'0","reported_seq":405,"reported_epoch":240,"state":"active+clean","last_fresh":"2024-03-11T21:34:58.347439+0000","last_change":"2024-03-11T21:34:41.156338+0000","last_active":"2024-03-11T21:34:58.347439+0000","last_peered":"2024-03-11T21:34:58.347439+0000","last_clean":"2024-03-11T21:34:58.347439+0000","last_became_active":"2024-03-11T21:34:41.155893+0000","last_became_peered":"2024-03-11T21:34:41.155893+0000","last_unstale":"2024-03-11T21:34:58.347439+0000","last_undegraded":"2024-03-11T21:34:58.347439+0000","last_fullsized":"2024-03-11T21:34:58.347439+0000","mapping_epoch":226,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":227,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:32:02.506579+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:26:50.991712+0000","last_clean_scrub_stamp":"2024-03-11T21:32:02.506579+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:35:02.646649+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,3],"acting":[1,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.f","version":"74'15","reported_seq":375,"reported_epoch":241,"state":"active+clean","last_fresh":"2024-03-11T21:35:04.407423+0000","last_change":"2024-03-11T21:34:21.519858+0000","last_active":"2024-03-11T21:35:04.407423+0000","last_peered":"2024-03-11T21:35:04.407423+0000","last_clean":"2024-03-11T21:35:04.407423+0000","last_became_active":"2024-03-11T21:26:11.216224+0000","last_became_peered":"2024-03-11T21:26:11.216224+0000","last_unstale":"2024-03-11T21:35:04.407423+0000","last_undegraded":"2024-03-11T21:35:04.407423+0000","last_fullsized":"2024-03-11T21:35:04.407423+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'15","last_scrub_stamp":"2024-03-11T21:34:21.517894+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:39.525554+0000","last_clean_scrub_stamp":"2024-03-11T21:34:21.517894+0000","objects_scrubbed":7,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:35:35.603333+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,4],"acting":[2,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"8.0","version":"240'9230","reported_seq":9452,"reported_epoch":240,"state":"active+clean","last_fresh":"2024-03-11T21:35:02.792135+0000","last_change":"2024-03-11T21:32:58.324488+0000","last_active":"2024-03-11T21:35:02.792135+0000","last_peered":"2024-03-11T21:35:02.792135+0000","last_clean":"2024-03-11T21:35:02.792135+0000","last_became_active":"2024-03-11T21:32:57.071801+0000","last_became_peered":"2024-03-11T21:32:57.071801+0000","last_unstale":"2024-03-11T21:35:02.792135+0000","last_undegraded":"2024-03-11T21:35:02.792135+0000","last_fullsized":"2024-03-11T21:35:02.792135+0000","mapping_epoch":157,"log_start":"170'3459","ondisk_log_start":"170'3459","created":157,"last_epoch_clean":182,"parent":"0.0","parent_split_bits":0,"last_scrub":"183'5466","last_scrub_stamp":"2024-03-11T21:32:58.324387+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:31:51.219688+0000","last_clean_scrub_stamp":"2024-03-11T21:32:58.324387+0000","objects_scrubbed":2659,"log_size":5771,"log_dups_size":0,"ondisk_log_size":5771,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:35:06.913481+0000","scrub_duration":612,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":54525952,"num_objects":832,"num_object_clones":0,"num_object_copies":2496,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":832,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3767,"num_write_kb":62080,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,1],"acting":[3,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.e","version":"0'0","reported_seq":383,"reported_epoch":241,"state":"active+clean","last_fresh":"2024-03-11T21:35:04.389179+0000","last_change":"2024-03-11T21:34:12.721980+0000","last_active":"2024-03-11T21:35:04.389179+0000","last_peered":"2024-03-11T21:35:04.389179+0000","last_clean":"2024-03-11T21:35:04.389179+0000","last_became_active":"2024-03-11T21:25:44.590198+0000","last_became_peered":"2024-03-11T21:25:44.590198+0000","last_unstale":"2024-03-11T21:35:04.389179+0000","last_undegraded":"2024-03-11T21:35:04.389179+0000","last_fullsized":"2024-03-11T21:35:04.389179+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:34:12.721766+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:43.559452+0000","last_clean_scrub_stamp":"2024-03-11T21:34:12.721766+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:35:37.637969+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.d","version":"109'9","reported_seq":509,"reported_epoch":241,"state":"active+clean","last_fresh":"2024-03-11T21:35:04.403819+0000","last_change":"2024-03-11T21:32:12.413588+0000","last_active":"2024-03-11T21:35:04.403819+0000","last_peered":"2024-03-11T21:35:04.403819+0000","last_clean":"2024-03-11T21:35:04.403819+0000","last_became_active":"2024-03-11T21:25:42.575583+0000","last_became_peered":"2024-03-11T21:25:42.575583+0000","last_unstale":"2024-03-11T21:35:04.403819+0000","last_undegraded":"2024-03-11T21:35:04.403819+0000","last_fullsized":"2024-03-11T21:35:04.403819+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"109'9","last_scrub_stamp":"2024-03-11T21:32:12.413216+0000","last_deep_scrub":"109'9","last_deep_scrub_stamp":"2024-03-11T21:28:27.082975+0000","last_clean_scrub_stamp":"2024-03-11T21:32:12.413216+0000","objects_scrubbed":1,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":25,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,5],"acting":[2,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.a","version":"0'0","reported_seq":413,"reported_epoch":240,"state":"active+clean","last_fresh":"2024-03-11T21:34:58.348193+0000","last_change":"2024-03-11T21:34:41.145567+0000","last_active":"2024-03-11T21:34:58.348193+0000","last_peered":"2024-03-11T21:34:58.348193+0000","last_clean":"2024-03-11T21:34:58.348193+0000","last_became_active":"2024-03-11T21:34:41.144773+0000","last_became_peered":"2024-03-11T21:34:41.144773+0000","last_unstale":"2024-03-11T21:34:58.348193+0000","last_undegraded":"2024-03-11T21:34:58.348193+0000","last_fullsized":"2024-03-11T21:34:58.348193+0000","mapping_epoch":226,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":227,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:34:22.526916+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:37.512018+0000","last_clean_scrub_stamp":"2024-03-11T21:34:22.526916+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:35:34.423315+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,7],"acting":[1,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.c","version":"74'10","reported_seq":406,"reported_epoch":241,"state":"active+clean","last_fresh":"2024-03-11T21:35:04.390592+0000","last_change":"2024-03-11T21:32:54.846999+0000","last_active":"2024-03-11T21:35:04.390592+0000","last_peered":"2024-03-11T21:35:04.390592+0000","last_clean":"2024-03-11T21:35:04.390592+0000","last_became_active":"2024-03-11T21:25:40.550885+0000","last_became_peered":"2024-03-11T21:25:40.550885+0000","last_unstale":"2024-03-11T21:35:04.390592+0000","last_undegraded":"2024-03-11T21:35:04.390592+0000","last_fullsized":"2024-03-11T21:35:04.390592+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'10","last_scrub_stamp":"2024-03-11T21:32:54.846584+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:39.525554+0000","last_clean_scrub_stamp":"2024-03-11T21:32:54.846584+0000","objects_scrubbed":5,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:35:04.446781+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"8.1","version":"240'9388","reported_seq":9608,"reported_epoch":240,"state":"active+clean","last_fresh":"2024-03-11T21:35:02.400385+0000","last_change":"2024-03-11T21:32:58.666237+0000","last_active":"2024-03-11T21:35:02.400385+0000","last_peered":"2024-03-11T21:35:02.400385+0000","last_clean":"2024-03-11T21:35:02.400385+0000","last_became_active":"2024-03-11T21:32:57.078036+0000","last_became_peered":"2024-03-11T21:32:57.078036+0000","last_unstale":"2024-03-11T21:35:02.400385+0000","last_undegraded":"2024-03-11T21:35:02.400385+0000","last_fullsized":"2024-03-11T21:35:02.400385+0000","mapping_epoch":157,"log_start":"170'3758","ondisk_log_start":"170'3758","created":157,"last_epoch_clean":182,"parent":"0.0","parent_split_bits":0,"last_scrub":"182'5517","last_scrub_stamp":"2024-03-11T21:32:58.666106+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:31:51.219688+0000","last_clean_scrub_stamp":"2024-03-11T21:32:58.666106+0000","objects_scrubbed":2736,"log_size":5630,"log_dups_size":0,"ondisk_log_size":5630,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:35:02.646776+0000","scrub_duration":348,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":54853632,"num_objects":837,"num_object_clones":0,"num_object_copies":2511,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":837,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3871,"num_write_kb":63104,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.f","version":"0'0","reported_seq":382,"reported_epoch":241,"state":"active+clean","last_fresh":"2024-03-11T21:35:04.407267+0000","last_change":"2024-03-11T21:32:17.431835+0000","last_active":"2024-03-11T21:35:04.407267+0000","last_peered":"2024-03-11T21:35:04.407267+0000","last_clean":"2024-03-11T21:35:04.407267+0000","last_became_active":"2024-03-11T21:25:44.593414+0000","last_became_peered":"2024-03-11T21:25:44.593414+0000","last_unstale":"2024-03-11T21:35:04.407267+0000","last_undegraded":"2024-03-11T21:35:04.407267+0000","last_fullsized":"2024-03-11T21:35:04.407267+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:32:17.431576+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:27:03.010969+0000","last_clean_scrub_stamp":"2024-03-11T21:32:17.431576+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":75,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,4],"acting":[2,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":385,"reported_epoch":240,"state":"active+clean","last_fresh":"2024-03-11T21:34:58.347882+0000","last_change":"2024-03-11T21:34:21.502786+0000","last_active":"2024-03-11T21:34:58.347882+0000","last_peered":"2024-03-11T21:34:58.347882+0000","last_clean":"2024-03-11T21:34:58.347882+0000","last_became_active":"2024-03-11T21:25:42.579409+0000","last_became_peered":"2024-03-11T21:25:42.579409+0000","last_unstale":"2024-03-11T21:34:58.347882+0000","last_undegraded":"2024-03-11T21:34:58.347882+0000","last_fullsized":"2024-03-11T21:34:58.347882+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:34:21.502521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:30:22.750053+0000","last_clean_scrub_stamp":"2024-03-11T21:34:21.502521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:35:30.013386+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.b","version":"0'0","reported_seq":39,"reported_epoch":241,"state":"active+clean","last_fresh":"2024-03-11T21:35:04.397636+0000","last_change":"2024-03-11T21:34:41.136339+0000","last_active":"2024-03-11T21:35:04.397636+0000","last_peered":"2024-03-11T21:35:04.397636+0000","last_clean":"2024-03-11T21:35:04.397636+0000","last_became_active":"2024-03-11T21:34:41.134987+0000","last_became_peered":"2024-03-11T21:34:41.134987+0000","last_unstale":"2024-03-11T21:35:04.397636+0000","last_undegraded":"2024-03-11T21:35:04.397636+0000","last_fullsized":"2024-03-11T21:35:04.397636+0000","mapping_epoch":226,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":227,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:34:29.735148+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:37.512018+0000","last_clean_scrub_stamp":"2024-03-11T21:34:29.735148+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:35:43.850507+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,5],"acting":[7,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.d","version":"165'25","reported_seq":427,"reported_epoch":241,"state":"active+clean","last_fresh":"2024-03-11T21:35:04.389061+0000","last_change":"2024-03-11T21:32:21.678485+0000","last_active":"2024-03-11T21:35:04.389061+0000","last_peered":"2024-03-11T21:35:04.389061+0000","last_clean":"2024-03-11T21:35:04.389061+0000","last_became_active":"2024-03-11T21:25:40.553454+0000","last_became_peered":"2024-03-11T21:25:40.553454+0000","last_unstale":"2024-03-11T21:35:04.389061+0000","last_undegraded":"2024-03-11T21:35:04.389061+0000","last_fullsized":"2024-03-11T21:35:04.389061+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"165'25","last_scrub_stamp":"2024-03-11T21:32:21.678328+0000","last_deep_scrub":"165'25","last_deep_scrub_stamp":"2024-03-11T21:32:21.678328+0000","last_clean_scrub_stamp":"2024-03-11T21:32:21.678328+0000","objects_scrubbed":8,"log_size":25,"log_dups_size":0,"ondisk_log_size":25,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:35:04.446654+0000","scrub_duration":21,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":29,"num_read_kb":19,"num_write":18,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,1],"acting":[4,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"8.6","version":"240'9358","reported_seq":9579,"reported_epoch":240,"state":"active+clean","last_fresh":"2024-03-11T21:35:03.261516+0000","last_change":"2024-03-11T21:32:58.205093+0000","last_active":"2024-03-11T21:35:03.261516+0000","last_peered":"2024-03-11T21:35:03.261516+0000","last_clean":"2024-03-11T21:35:03.261516+0000","last_became_active":"2024-03-11T21:32:57.074062+0000","last_became_peered":"2024-03-11T21:32:57.074062+0000","last_unstale":"2024-03-11T21:35:03.261516+0000","last_undegraded":"2024-03-11T21:35:03.261516+0000","last_fullsized":"2024-03-11T21:35:03.261516+0000","mapping_epoch":157,"log_start":"173'4219","ondisk_log_start":"173'4219","created":157,"last_epoch_clean":182,"parent":"0.0","parent_split_bits":0,"last_scrub":"183'5458","last_scrub_stamp":"2024-03-11T21:32:58.204881+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:31:51.219688+0000","last_clean_scrub_stamp":"2024-03-11T21:32:58.204881+0000","objects_scrubbed":2753,"log_size":5139,"log_dups_size":0,"ondisk_log_size":5139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":640,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":55312384,"num_objects":844,"num_object_clones":0,"num_object_copies":2532,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":844,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3901,"num_write_kb":63744,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,7],"acting":[0,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":39,"reported_epoch":241,"state":"active+clean","last_fresh":"2024-03-11T21:35:04.396555+0000","last_change":"2024-03-11T21:34:41.149338+0000","last_active":"2024-03-11T21:35:04.396555+0000","last_peered":"2024-03-11T21:35:04.396555+0000","last_clean":"2024-03-11T21:35:04.396555+0000","last_became_active":"2024-03-11T21:34:41.145012+0000","last_became_peered":"2024-03-11T21:34:41.145012+0000","last_unstale":"2024-03-11T21:35:04.396555+0000","last_undegraded":"2024-03-11T21:35:04.396555+0000","last_fullsized":"2024-03-11T21:35:04.396555+0000","mapping_epoch":226,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":227,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:34:22.554606+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:28:37.561826+0000","last_clean_scrub_stamp":"2024-03-11T21:34:22.554606+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:35:37.799908+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,0],"acting":[7,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":386,"reported_epoch":241,"state":"active+clean","last_fresh":"2024-03-11T21:35:04.396655+0000","last_change":"2024-03-11T21:32:23.251510+0000","last_active":"2024-03-11T21:35:04.396655+0000","last_peered":"2024-03-11T21:35:04.396655+0000","last_clean":"2024-03-11T21:35:04.396655+0000","last_became_active":"2024-03-11T21:25:42.574515+0000","last_became_peered":"2024-03-11T21:25:42.574515+0000","last_unstale":"2024-03-11T21:35:04.396655+0000","last_undegraded":"2024-03-11T21:35:04.396655+0000","last_fullsized":"2024-03-11T21:35:04.396655+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:32:23.251386+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:41.544550+0000","last_clean_scrub_stamp":"2024-03-11T21:32:23.251386+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,5],"acting":[2,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.c","version":"0'0","reported_seq":394,"reported_epoch":241,"state":"active+clean","last_fresh":"2024-03-11T21:35:04.393026+0000","last_change":"2024-03-11T21:34:16.385926+0000","last_active":"2024-03-11T21:35:04.393026+0000","last_peered":"2024-03-11T21:35:04.393026+0000","last_clean":"2024-03-11T21:35:04.393026+0000","last_became_active":"2024-03-11T21:25:38.533240+0000","last_became_peered":"2024-03-11T21:25:38.533240+0000","last_unstale":"2024-03-11T21:35:04.393026+0000","last_undegraded":"2024-03-11T21:35:04.393026+0000","last_fullsized":"2024-03-11T21:35:04.393026+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:34:16.385811+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:29:30.279910+0000","last_clean_scrub_stamp":"2024-03-11T21:34:16.385811+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:35:45.125684+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,0],"acting":[2,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.a","version":"109'28","reported_seq":491,"reported_epoch":241,"state":"active+clean","last_fresh":"2024-03-11T21:35:04.399867+0000","last_change":"2024-03-11T21:34:41.169399+0000","last_active":"2024-03-11T21:35:04.399867+0000","last_peered":"2024-03-11T21:35:04.399867+0000","last_clean":"2024-03-11T21:35:04.399867+0000","last_became_active":"2024-03-11T21:34:41.148863+0000","last_became_peered":"2024-03-11T21:34:41.148863+0000","last_unstale":"2024-03-11T21:35:04.399867+0000","last_undegraded":"2024-03-11T21:35:04.399867+0000","last_fullsized":"2024-03-11T21:35:04.399867+0000","mapping_epoch":226,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":227,"parent":"0.0","parent_split_bits":0,"last_scrub":"109'28","last_scrub_stamp":"2024-03-11T21:34:15.438620+0000","last_deep_scrub":"109'28","last_deep_scrub_stamp":"2024-03-11T21:28:24.016389+0000","last_clean_scrub_stamp":"2024-03-11T21:34:15.438620+0000","objects_scrubbed":9,"log_size":28,"log_dups_size":0,"ondisk_log_size":28,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:35:23.047712+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":32,"num_read_kb":21,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,7],"acting":[6,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"8.7","version":"240'9275","reported_seq":9497,"reported_epoch":240,"state":"active+clean","last_fresh":"2024-03-11T21:35:02.335767+0000","last_change":"2024-03-11T21:32:58.092722+0000","last_active":"2024-03-11T21:35:02.335767+0000","last_peered":"2024-03-11T21:35:02.335767+0000","last_clean":"2024-03-11T21:35:02.335767+0000","last_became_active":"2024-03-11T21:32:57.070297+0000","last_became_peered":"2024-03-11T21:32:57.070297+0000","last_unstale":"2024-03-11T21:35:02.335767+0000","last_undegraded":"2024-03-11T21:35:02.335767+0000","last_fullsized":"2024-03-11T21:35:02.335767+0000","mapping_epoch":157,"log_start":"170'3631","ondisk_log_start":"170'3631","created":157,"last_epoch_clean":182,"parent":"0.0","parent_split_bits":0,"last_scrub":"183'5479","last_scrub_stamp":"2024-03-11T21:32:58.091894+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:31:51.219688+0000","last_clean_scrub_stamp":"2024-03-11T21:32:58.091894+0000","objects_scrubbed":2710,"log_size":5644,"log_dups_size":0,"ondisk_log_size":5644,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:35:02.646743+0000","scrub_duration":724,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":52363264,"num_objects":799,"num_object_clones":0,"num_object_copies":2397,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":799,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3807,"num_write_kb":60672,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,6],"acting":[1,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":393,"reported_epoch":240,"state":"active+clean","last_fresh":"2024-03-11T21:34:58.348892+0000","last_change":"2024-03-11T21:34:41.161838+0000","last_active":"2024-03-11T21:34:58.348892+0000","last_peered":"2024-03-11T21:34:58.348892+0000","last_clean":"2024-03-11T21:34:58.348892+0000","last_became_active":"2024-03-11T21:34:41.154507+0000","last_became_peered":"2024-03-11T21:34:41.154507+0000","last_unstale":"2024-03-11T21:34:58.348892+0000","last_undegraded":"2024-03-11T21:34:58.348892+0000","last_fullsized":"2024-03-11T21:34:58.348892+0000","mapping_epoch":226,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":227,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:34:10.771846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:27:11.361913+0000","last_clean_scrub_stamp":"2024-03-11T21:34:10.771846+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:35:29.534703+0000","scrub_duration":65,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":386,"reported_epoch":241,"state":"active+clean","last_fresh":"2024-03-11T21:35:04.394228+0000","last_change":"2024-03-11T21:34:17.412793+0000","last_active":"2024-03-11T21:35:04.394228+0000","last_peered":"2024-03-11T21:35:04.394228+0000","last_clean":"2024-03-11T21:35:04.394228+0000","last_became_active":"2024-03-11T21:25:42.576679+0000","last_became_peered":"2024-03-11T21:25:42.576679+0000","last_unstale":"2024-03-11T21:35:04.394228+0000","last_undegraded":"2024-03-11T21:35:04.394228+0000","last_fullsized":"2024-03-11T21:35:04.394228+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:34:17.412687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:41.544550+0000","last_clean_scrub_stamp":"2024-03-11T21:34:17.412687+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:35:37.322196+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,3],"acting":[2,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.d","version":"0'0","reported_seq":393,"reported_epoch":240,"state":"active+clean","last_fresh":"2024-03-11T21:34:58.348472+0000","last_change":"2024-03-11T21:31:55.464241+0000","last_active":"2024-03-11T21:34:58.348472+0000","last_peered":"2024-03-11T21:34:58.348472+0000","last_clean":"2024-03-11T21:34:58.348472+0000","last_became_active":"2024-03-11T21:25:38.539681+0000","last_became_peered":"2024-03-11T21:25:38.539681+0000","last_unstale":"2024-03-11T21:34:58.348472+0000","last_undegraded":"2024-03-11T21:34:58.348472+0000","last_fullsized":"2024-03-11T21:34:58.348472+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:31:55.464145+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:37.512018+0000","last_clean_scrub_stamp":"2024-03-11T21:31:55.464145+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:35:02.646497+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,3],"acting":[1,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.b","version":"74'9","reported_seq":405,"reported_epoch":240,"state":"active+clean","last_fresh":"2024-03-11T21:34:58.350436+0000","last_change":"2024-03-11T21:32:54.624276+0000","last_active":"2024-03-11T21:34:58.350436+0000","last_peered":"2024-03-11T21:34:58.350436+0000","last_clean":"2024-03-11T21:34:58.350436+0000","last_became_active":"2024-03-11T21:25:40.549612+0000","last_became_peered":"2024-03-11T21:25:40.549612+0000","last_unstale":"2024-03-11T21:34:58.350436+0000","last_undegraded":"2024-03-11T21:34:58.350436+0000","last_fullsized":"2024-03-11T21:34:58.350436+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'9","last_scrub_stamp":"2024-03-11T21:32:54.623943+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:39.525554+0000","last_clean_scrub_stamp":"2024-03-11T21:32:54.623943+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:35:01.804174+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.4","version":"241'9534","reported_seq":9757,"reported_epoch":241,"state":"active+clean","last_fresh":"2024-03-11T21:35:06.973304+0000","last_change":"2024-03-11T21:32:59.991354+0000","last_active":"2024-03-11T21:35:06.973304+0000","last_peered":"2024-03-11T21:35:06.973304+0000","last_clean":"2024-03-11T21:35:06.973304+0000","last_became_active":"2024-03-11T21:32:57.087247+0000","last_became_peered":"2024-03-11T21:32:57.087247+0000","last_unstale":"2024-03-11T21:35:06.973304+0000","last_undegraded":"2024-03-11T21:35:06.973304+0000","last_fullsized":"2024-03-11T21:35:06.973304+0000","mapping_epoch":157,"log_start":"172'3939","ondisk_log_start":"172'3939","created":157,"last_epoch_clean":182,"parent":"0.0","parent_split_bits":0,"last_scrub":"185'5515","last_scrub_stamp":"2024-03-11T21:32:59.991099+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:31:51.219688+0000","last_clean_scrub_stamp":"2024-03-11T21:32:59.991099+0000","objects_scrubbed":2803,"log_size":5595,"log_dups_size":0,"ondisk_log_size":5595,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":733,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":44040192,"num_objects":672,"num_object_clones":0,"num_object_copies":2016,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":672,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4037,"num_write_kb":60992,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,2],"acting":[5,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":287,"reported_epoch":241,"state":"active+clean","last_fresh":"2024-03-11T21:35:04.388387+0000","last_change":"2024-03-11T21:32:18.627466+0000","last_active":"2024-03-11T21:35:04.388387+0000","last_peered":"2024-03-11T21:35:04.388387+0000","last_clean":"2024-03-11T21:35:04.388387+0000","last_became_active":"2024-03-11T21:28:45.372452+0000","last_became_peered":"2024-03-11T21:28:45.372452+0000","last_unstale":"2024-03-11T21:35:04.388387+0000","last_undegraded":"2024-03-11T21:35:04.388387+0000","last_fullsized":"2024-03-11T21:35:04.388387+0000","mapping_epoch":111,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":112,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:32:18.627290+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:43.559452+0000","last_clean_scrub_stamp":"2024-03-11T21:32:18.627290+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:35:04.446623+0000","scrub_duration":29,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,3],"acting":[4,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.e","version":"0'0","reported_seq":414,"reported_epoch":241,"state":"active+clean","last_fresh":"2024-03-11T21:35:04.411955+0000","last_change":"2024-03-11T21:34:41.143980+0000","last_active":"2024-03-11T21:35:04.411955+0000","last_peered":"2024-03-11T21:35:04.411955+0000","last_clean":"2024-03-11T21:35:04.411955+0000","last_became_active":"2024-03-11T21:34:41.143532+0000","last_became_peered":"2024-03-11T21:34:41.143532+0000","last_unstale":"2024-03-11T21:35:04.411955+0000","last_undegraded":"2024-03-11T21:35:04.411955+0000","last_fullsized":"2024-03-11T21:35:04.411955+0000","mapping_epoch":226,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":227,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:34:19.476172+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:37.512018+0000","last_clean_scrub_stamp":"2024-03-11T21:34:19.476172+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:35:48.981067+0000","scrub_duration":17,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,7],"acting":[2,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.8","version":"109'22","reported_seq":466,"reported_epoch":241,"state":"active+clean","last_fresh":"2024-03-11T21:35:04.396325+0000","last_change":"2024-03-11T21:34:41.166337+0000","last_active":"2024-03-11T21:35:04.396325+0000","last_peered":"2024-03-11T21:35:04.396325+0000","last_clean":"2024-03-11T21:35:04.396325+0000","last_became_active":"2024-03-11T21:34:41.146631+0000","last_became_peered":"2024-03-11T21:34:41.146631+0000","last_unstale":"2024-03-11T21:35:04.396325+0000","last_undegraded":"2024-03-11T21:35:04.396325+0000","last_fullsized":"2024-03-11T21:35:04.396325+0000","mapping_epoch":226,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":227,"parent":"0.0","parent_split_bits":0,"last_scrub":"109'22","last_scrub_stamp":"2024-03-11T21:32:07.573197+0000","last_deep_scrub":"109'22","last_deep_scrub_stamp":"2024-03-11T21:32:07.573197+0000","last_clean_scrub_stamp":"2024-03-11T21:32:07.573197+0000","objects_scrubbed":7,"log_size":22,"log_dups_size":0,"ondisk_log_size":22,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.9","version":"229'11","reported_seq":48,"reported_epoch":241,"state":"active+clean","last_fresh":"2024-03-11T21:35:04.396711+0000","last_change":"2024-03-11T21:34:41.163190+0000","last_active":"2024-03-11T21:35:04.396711+0000","last_peered":"2024-03-11T21:35:04.396711+0000","last_clean":"2024-03-11T21:35:04.396711+0000","last_became_active":"2024-03-11T21:34:41.156892+0000","last_became_peered":"2024-03-11T21:34:41.156892+0000","last_unstale":"2024-03-11T21:35:04.396711+0000","last_undegraded":"2024-03-11T21:35:04.396711+0000","last_fullsized":"2024-03-11T21:35:04.396711+0000","mapping_epoch":226,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":227,"parent":"0.0","parent_split_bits":0,"last_scrub":"154'10","last_scrub_stamp":"2024-03-11T21:34:19.426261+0000","last_deep_scrub":"94'9","last_deep_scrub_stamp":"2024-03-11T21:26:53.422552+0000","last_clean_scrub_stamp":"2024-03-11T21:34:19.426261+0000","objects_scrubbed":1,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:35:48.765690+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":8,"num_pg":26,"stat_sum":{"num_bytes":1597341720,"num_objects":24375,"num_object_clones":0,"num_object_copies":73125,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":3172,"num_objects_unfound":0,"num_objects_dirty":24375,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":144041,"num_write_kb":3698849,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9484,"num_bytes_recovered":621477912,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":5638930432,"data_stored":5638914144,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":138878,"ondisk_log_size":138878,"up":78,"acting":78,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":1531,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":24,"num_read_kb":24,"num_write":10,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":872,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":4593,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":6,"ondisk_log_size":6,"up":96,"acting":96,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":198,"num_read_kb":294,"num_write":179,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":2951840,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":2375680,"data_stored":2361472,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":128,"ondisk_log_size":128,"up":3,"acting":3,"num_store_stats":4},{"poolid":3,"num_pg":3,"stat_sum":{"num_bytes":408,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":557,"num_read_kb":552,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1224,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":8,"ondisk_log_size":8,"up":9,"acting":9,"num_store_stats":7},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":3702,"num_objects":178,"num_object_clones":0,"num_object_copies":534,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":178,"num_whiteouts":0,"num_read":698,"num_read_kb":455,"num_write":417,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":142,"num_bytes_recovered":2530,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":417792,"data_stored":11106,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":472,"ondisk_log_size":472,"up":96,"acting":96,"num_store_stats":8},{"poolid":5,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":74,"ondisk_log_size":74,"up":96,"acting":96,"num_store_stats":8},{"poolid":6,"num_pg":32,"stat_sum":{"num_bytes":383,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":370,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1149,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":96,"acting":96,"num_store_stats":8}],"osd_stats":[{"osd":7,"up_from":60,"seq":257698037877,"num_pgs":68,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":719096,"kb_used_data":499116,"kb_used_omap":1,"kb_used_meta":219966,"kb_avail":93026056,"statfs":{"total":95995035648,"available":95258681344,"internally_reserved":0,"allocated":511094784,"data_stored":510707985,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":225245642},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":50,"seq":214748364923,"num_pgs":56,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":944048,"kb_used_data":624104,"kb_used_omap":1,"kb_used_meta":319934,"kb_avail":92801104,"statfs":{"total":95995035648,"available":95028330496,"internally_reserved":0,"allocated":639082496,"data_stored":638700086,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":327612870},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574996,"num_pgs":63,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1251128,"kb_used_data":904688,"kb_used_omap":1,"kb_used_meta":346302,"kb_avail":92494024,"statfs":{"total":95995035648,"available":94713880576,"internally_reserved":0,"allocated":926400512,"data_stored":926002301,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":354613701},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,1,3,4,1],"upper_bound":64},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738521,"num_pgs":66,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1443564,"kb_used_data":1029220,"kb_used_omap":1,"kb_used_meta":414334,"kb_avail":92301588,"statfs":{"total":95995035648,"available":94516826112,"internally_reserved":0,"allocated":1053921280,"data_stored":1053535937,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":424278469},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,0,0,5],"upper_bound":32},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411471,"num_pgs":51,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1088164,"kb_used_data":719644,"kb_used_omap":1,"kb_used_meta":368510,"kb_avail":92656988,"statfs":{"total":95995035648,"available":94880755712,"internally_reserved":0,"allocated":736915456,"data_stored":736544122,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":377354693},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":3,"up_from":26,"seq":111669149834,"num_pgs":63,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1061244,"kb_used_data":664948,"kb_used_omap":1,"kb_used_meta":396286,"kb_avail":92683908,"statfs":{"total":95995035648,"available":94908321792,"internally_reserved":0,"allocated":680906752,"data_stored":680504889,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":405797319},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,3,1,0,2],"upper_bound":32},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":4,"up_from":32,"seq":137438953604,"num_pgs":64,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":933756,"kb_used_data":609648,"kb_used_omap":1,"kb_used_meta":324094,"kb_avail":92811396,"statfs":{"total":95995035648,"available":95038869504,"internally_reserved":0,"allocated":624279552,"data_stored":623887998,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":331872709},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,1],"upper_bound":8},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":41,"seq":176093659263,"num_pgs":62,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":957484,"kb_used_data":629216,"kb_used_omap":1,"kb_used_meta":328254,"kb_avail":92787668,"statfs":{"total":95995035648,"available":95014572032,"internally_reserved":0,"allocated":644317184,"data_stored":643941805,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":336132549},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1049,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":46,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":957,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":574,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":436,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1003,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":528,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":1430,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":77824,"data_stored":2008,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":1018,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":1898,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":1474,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":990,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":1034,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":1254,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":370,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":383,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":370,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":1027342336,"data_stored":1027342336,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":894894080,"data_stored":894894080,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":705171456,"data_stored":705167384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":651100160,"data_stored":651100160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":603455488,"data_stored":603455488,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":616501248,"data_stored":616497176,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":636424192,"data_stored":636420120,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":504041472,"data_stored":504037400,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-11T21:35:09.263 INFO:tasks.cephadm.ceph_manager.ceph:recovered! 2024-03-11T21:35:09.263 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-03-11T21:35:09.263 INFO:tasks.thrashosds.thrasher:inject_pause on osd.1 2024-03-11T21:35:09.264 INFO:tasks.thrashosds.thrasher:Testing filestore_inject_stall pause injection for duration 3 2024-03-11T21:35:09.264 INFO:tasks.thrashosds.thrasher:Checking after 0, should_be_down=False 2024-03-11T21:35:09.264 DEBUG:teuthology.orchestra.run.smithi092:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid 00db0016-dfed-11ee-95c8-87774f69a715 -- ceph daemon osd.1 config set filestore_inject_stall 3 2024-03-11T21:35:09.273 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:35:09.301 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:35:09.302 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:35:09.303 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:35:09.403 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:35:09.434 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:35:09.434 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:35:09.434 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:35:09.451 INFO:teuthology.orchestra.run.smithi092.stderr:Inferring config /var/lib/ceph/00db0016-dfed-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T21:35:09.495 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:35:09 smithi092 ceph-mon[29572]: 6.16 scrub starts 2024-03-11T21:35:09.495 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:35:09 smithi092 ceph-mon[29572]: 6.16 scrub ok 2024-03-11T21:35:09.495 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:35:09 smithi092 ceph-mon[29572]: 2.3 scrub starts 2024-03-11T21:35:09.495 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:35:09 smithi092 ceph-mon[29572]: 2.3 scrub ok 2024-03-11T21:35:09.495 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:35:09 smithi092 ceph-mon[29572]: 6.2 scrub starts 2024-03-11T21:35:09.495 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:35:09 smithi092 ceph-mon[29572]: 6.2 scrub ok 2024-03-11T21:35:09.495 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:35:09 smithi092 ceph-mon[29572]: from='client.25219 v1:172.21.15.92:0/1968617316' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-11T21:35:09.495 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:35:09 smithi092 ceph-mon[29572]: 2.14 scrub starts 2024-03-11T21:35:09.495 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:35:09 smithi092 ceph-mon[29572]: 2.14 scrub ok 2024-03-11T21:35:09.496 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:35:09 smithi092 ceph-mon[37563]: 6.16 scrub starts 2024-03-11T21:35:09.496 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:35:09 smithi092 ceph-mon[37563]: 6.16 scrub ok 2024-03-11T21:35:09.496 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:35:09 smithi092 ceph-mon[37563]: 2.3 scrub starts 2024-03-11T21:35:09.496 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:35:09 smithi092 ceph-mon[37563]: 2.3 scrub ok 2024-03-11T21:35:09.496 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:35:09 smithi092 ceph-mon[37563]: 6.2 scrub starts 2024-03-11T21:35:09.496 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:35:09 smithi092 ceph-mon[37563]: 6.2 scrub ok 2024-03-11T21:35:09.496 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:35:09 smithi092 ceph-mon[37563]: from='client.25219 v1:172.21.15.92:0/1968617316' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-11T21:35:09.496 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:35:09 smithi092 ceph-mon[37563]: 2.14 scrub starts 2024-03-11T21:35:09.496 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:35:09 smithi092 ceph-mon[37563]: 2.14 scrub ok 2024-03-11T21:35:09.535 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:35:09.567 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:35:09.568 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:35:09.569 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:35:09.669 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:35:09.705 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:35:09.706 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:35:09.706 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:35:09.807 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:35:09.826 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:35:09 smithi097 ceph-mon[33317]: 6.16 scrub starts 2024-03-11T21:35:09.826 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:35:09 smithi097 ceph-mon[33317]: 6.16 scrub ok 2024-03-11T21:35:09.826 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:35:09 smithi097 ceph-mon[33317]: 2.3 scrub starts 2024-03-11T21:35:09.826 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:35:09 smithi097 ceph-mon[33317]: 2.3 scrub ok 2024-03-11T21:35:09.826 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:35:09 smithi097 ceph-mon[33317]: 6.2 scrub starts 2024-03-11T21:35:09.826 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:35:09 smithi097 ceph-mon[33317]: 6.2 scrub ok 2024-03-11T21:35:09.826 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:35:09 smithi097 ceph-mon[33317]: from='client.25219 v1:172.21.15.92:0/1968617316' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-11T21:35:09.826 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:35:09 smithi097 ceph-mon[33317]: 2.14 scrub starts 2024-03-11T21:35:09.826 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:35:09 smithi097 ceph-mon[33317]: 2.14 scrub ok 2024-03-11T21:35:09.848 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:35:09.849 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:35:09.849 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:35:09.950 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:35:09.995 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:35:09.996 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:35:09.996 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:35:10.097 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:35:10.134 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:35:10.135 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:35:10.135 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:35:10.236 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:35:10.277 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:35:10.277 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:35:10.278 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:35:10.379 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:35:10.423 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:35:10.424 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:35:10.424 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:35:10.525 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:35:10.567 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:35:10.567 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:35:10.568 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:35:10.570 DEBUG:teuthology.orchestra.run.smithi092:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid 00db0016-dfed-11ee-95c8-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-03-11T21:35:10.668 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:35:10.711 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:35:10.711 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:35:10.712 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:35:10.812 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:35:10.825 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:35:10 smithi097 ceph-mon[33317]: pgmap v422: 158 pgs: 2 active+remapped+backfill_toofull, 156 active+clean; 1.3 GiB data, 7.5 GiB used, 708 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.40k op/s; 2539/64302 objects misplaced (3.949%) 2024-03-11T21:35:10.825 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:35:10 smithi097 ceph-mon[33317]: 4.1b scrub starts 2024-03-11T21:35:10.826 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:35:10 smithi097 ceph-mon[33317]: 4.1b scrub ok 2024-03-11T21:35:10.826 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:35:10 smithi097 ceph-mon[33317]: 6.6 scrub starts 2024-03-11T21:35:10.826 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:35:10 smithi097 ceph-mon[33317]: 6.6 scrub ok 2024-03-11T21:35:10.841 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:35:10.842 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:35:10.843 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:35:10.860 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:35:10 smithi092 ceph-mon[29572]: pgmap v422: 158 pgs: 2 active+remapped+backfill_toofull, 156 active+clean; 1.3 GiB data, 7.5 GiB used, 708 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.40k op/s; 2539/64302 objects misplaced (3.949%) 2024-03-11T21:35:10.860 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:35:10 smithi092 ceph-mon[29572]: 4.1b scrub starts 2024-03-11T21:35:10.860 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:35:10 smithi092 ceph-mon[29572]: 4.1b scrub ok 2024-03-11T21:35:10.860 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:35:10 smithi092 ceph-mon[29572]: 6.6 scrub starts 2024-03-11T21:35:10.860 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:35:10 smithi092 ceph-mon[29572]: 6.6 scrub ok 2024-03-11T21:35:10.860 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:35:10 smithi092 ceph-mon[37563]: pgmap v422: 158 pgs: 2 active+remapped+backfill_toofull, 156 active+clean; 1.3 GiB data, 7.5 GiB used, 708 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.40k op/s; 2539/64302 objects misplaced (3.949%) 2024-03-11T21:35:10.860 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:35:10 smithi092 ceph-mon[37563]: 4.1b scrub starts 2024-03-11T21:35:10.860 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:35:10 smithi092 ceph-mon[37563]: 4.1b scrub ok 2024-03-11T21:35:10.860 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:35:10 smithi092 ceph-mon[37563]: 6.6 scrub starts 2024-03-11T21:35:10.860 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:35:10 smithi092 ceph-mon[37563]: 6.6 scrub ok 2024-03-11T21:35:10.943 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:35:10.973 INFO:teuthology.orchestra.run.smithi092.stderr:Inferring config /var/lib/ceph/00db0016-dfed-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T21:35:11.002 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:35:11.002 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:35:11.003 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:35:11.062 INFO:teuthology.orchestra.run.smithi092.stderr:Inferring config /var/lib/ceph/00db0016-dfed-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T21:35:11.104 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:35:11.139 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:35:11.139 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:35:11.140 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:35:11.241 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:35:11.276 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:35:11.276 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:35:11.277 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:35:11.377 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:35:11.417 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:35:11.417 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:35:11.418 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:35:11.497 DEBUG:teuthology.orchestra.run.smithi092:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid 00db0016-dfed-11ee-95c8-87774f69a715 -- ceph osd unset nodeep-scrub 2024-03-11T21:35:11.518 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:35:11.547 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:35:11.548 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:35:11.548 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:35:11.649 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:35:11.714 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:35:11.715 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:35:11.715 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:35:11.750 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:35:11 smithi092 ceph-mon[29572]: 6.f scrub starts 2024-03-11T21:35:11.750 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:35:11 smithi092 ceph-mon[29572]: 6.f scrub ok 2024-03-11T21:35:11.750 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:35:11 smithi092 ceph-mon[37563]: 6.f scrub starts 2024-03-11T21:35:11.751 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:35:11 smithi092 ceph-mon[37563]: 6.f scrub ok 2024-03-11T21:35:11.816 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:35:11.825 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:35:11 smithi097 ceph-mon[33317]: 6.f scrub starts 2024-03-11T21:35:11.825 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:35:11 smithi097 ceph-mon[33317]: 6.f scrub ok 2024-03-11T21:35:11.865 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:35:11.865 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:35:11.865 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:35:11.966 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:35:12.002 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:35:12.003 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:35:12.004 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:35:12.104 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:35:12.140 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:35:12.141 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:35:12.141 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:35:12.242 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:35:12.277 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:35:12.278 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:35:12.278 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:35:12.379 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:35:12.409 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:35:12.409 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:35:12.410 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:35:12.510 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:35:12.539 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:35:12.540 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:35:12.540 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:35:12.641 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:35:12.674 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:35:12.675 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:35:12.675 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:35:12.776 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:35:12.814 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:35:12.815 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:35:12.816 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:35:12.826 INFO:journalctl@ceph.iscsi.iscsi.a.smithi097.stdout:Mar 11 21:35:12 smithi097 ceph-00db0016-dfed-11ee-95c8-87774f69a715-iscsi-iscsi-a[60943]: debug there is no tcmu-runner data available 2024-03-11T21:35:12.826 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:35:12 smithi097 ceph-mon[33317]: 5.1e scrub starts 2024-03-11T21:35:12.826 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:35:12 smithi097 ceph-mon[33317]: 5.1e scrub ok 2024-03-11T21:35:12.826 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:35:12 smithi097 ceph-mon[33317]: pgmap v423: 158 pgs: 2 active+remapped+backfill_toofull, 156 active+clean; 1.3 GiB data, 7.5 GiB used, 708 GiB / 715 GiB avail; 639 B/s rd, 0 B/s wr, 842 op/s; 2539/64302 objects misplaced (3.949%) 2024-03-11T21:35:12.826 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:35:12 smithi097 ceph-mon[33317]: 5.2 scrub starts 2024-03-11T21:35:12.826 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:35:12 smithi097 ceph-mon[33317]: 5.2 scrub ok 2024-03-11T21:35:12.859 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:35:12 smithi092 ceph-mon[29572]: 5.1e scrub starts 2024-03-11T21:35:12.860 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:35:12 smithi092 ceph-mon[29572]: 5.1e scrub ok 2024-03-11T21:35:12.860 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:35:12 smithi092 ceph-mon[29572]: pgmap v423: 158 pgs: 2 active+remapped+backfill_toofull, 156 active+clean; 1.3 GiB data, 7.5 GiB used, 708 GiB / 715 GiB avail; 639 B/s rd, 0 B/s wr, 842 op/s; 2539/64302 objects misplaced (3.949%) 2024-03-11T21:35:12.860 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:35:12 smithi092 ceph-mon[29572]: 5.2 scrub starts 2024-03-11T21:35:12.860 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:35:12 smithi092 ceph-mon[29572]: 5.2 scrub ok 2024-03-11T21:35:12.860 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:35:12 smithi092 ceph-mon[37563]: 5.1e scrub starts 2024-03-11T21:35:12.860 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:35:12 smithi092 ceph-mon[37563]: 5.1e scrub ok 2024-03-11T21:35:12.860 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:35:12 smithi092 ceph-mon[37563]: pgmap v423: 158 pgs: 2 active+remapped+backfill_toofull, 156 active+clean; 1.3 GiB data, 7.5 GiB used, 708 GiB / 715 GiB avail; 639 B/s rd, 0 B/s wr, 842 op/s; 2539/64302 objects misplaced (3.949%) 2024-03-11T21:35:12.860 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:35:12 smithi092 ceph-mon[37563]: 5.2 scrub starts 2024-03-11T21:35:12.860 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:35:12 smithi092 ceph-mon[37563]: 5.2 scrub ok 2024-03-11T21:35:12.917 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:35:12.957 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:35:12.957 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:35:12.958 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:35:13.058 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:35:13.093 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:35:13.094 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:35:13.095 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:35:13.195 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:35:13.226 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:35:13.226 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:35:13.227 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:35:13.327 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:35:13.368 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:35:13.368 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:35:13.369 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:35:13.469 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:35:13.504 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:35:13.504 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:35:13.505 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:35:13.605 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:35:13.641 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:35:13.642 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:35:13.642 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:35:13.743 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:35:13.775 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:35:13 smithi092 ceph-mon[29572]: 4.3 scrub starts 2024-03-11T21:35:13.775 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:35:13 smithi092 ceph-mon[29572]: 4.3 scrub ok 2024-03-11T21:35:13.775 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:35:13 smithi092 ceph-mon[29572]: 5.1a scrub starts 2024-03-11T21:35:13.775 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:35:13 smithi092 ceph-mon[29572]: 5.1a scrub ok 2024-03-11T21:35:13.775 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:35:13 smithi092 ceph-mon[29572]: 5.1 scrub starts 2024-03-11T21:35:13.775 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:35:13 smithi092 ceph-mon[29572]: 5.1 scrub ok 2024-03-11T21:35:13.775 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:35:13 smithi092 ceph-mon[29572]: from='client.24469 v1:172.21.15.97:0/404208310' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:35:13.775 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:35:13 smithi092 ceph-mon[29572]: purged_snaps scrub starts 2024-03-11T21:35:13.776 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:35:13 smithi092 ceph-mon[29572]: purged_snaps scrub ok 2024-03-11T21:35:13.776 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:35:13 smithi092 ceph-mon[29572]: 5.13 scrub starts 2024-03-11T21:35:13.776 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:35:13 smithi092 ceph-mon[29572]: 5.13 scrub ok 2024-03-11T21:35:13.776 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:35:13 smithi092 ceph-mon[37563]: 4.3 scrub starts 2024-03-11T21:35:13.776 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:35:13 smithi092 ceph-mon[37563]: 4.3 scrub ok 2024-03-11T21:35:13.776 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:35:13 smithi092 ceph-mon[37563]: 5.1a scrub starts 2024-03-11T21:35:13.776 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:35:13 smithi092 ceph-mon[37563]: 5.1a scrub ok 2024-03-11T21:35:13.776 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:35:13 smithi092 ceph-mon[37563]: 5.1 scrub starts 2024-03-11T21:35:13.776 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:35:13 smithi092 ceph-mon[37563]: 5.1 scrub ok 2024-03-11T21:35:13.776 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:35:13 smithi092 ceph-mon[37563]: from='client.24469 v1:172.21.15.97:0/404208310' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:35:13.776 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:35:13 smithi092 ceph-mon[37563]: purged_snaps scrub starts 2024-03-11T21:35:13.776 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:35:13 smithi092 ceph-mon[37563]: purged_snaps scrub ok 2024-03-11T21:35:13.777 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:35:13 smithi092 ceph-mon[37563]: 5.13 scrub starts 2024-03-11T21:35:13.777 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:35:13 smithi092 ceph-mon[37563]: 5.13 scrub ok 2024-03-11T21:35:13.785 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:35:13.785 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:35:13.785 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:35:13.826 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:35:13 smithi097 ceph-mon[33317]: 4.3 scrub starts 2024-03-11T21:35:13.826 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:35:13 smithi097 ceph-mon[33317]: 4.3 scrub ok 2024-03-11T21:35:13.826 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:35:13 smithi097 ceph-mon[33317]: 5.1a scrub starts 2024-03-11T21:35:13.826 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:35:13 smithi097 ceph-mon[33317]: 5.1a scrub ok 2024-03-11T21:35:13.826 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:35:13 smithi097 ceph-mon[33317]: 5.1 scrub starts 2024-03-11T21:35:13.826 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:35:13 smithi097 ceph-mon[33317]: 5.1 scrub ok 2024-03-11T21:35:13.826 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:35:13 smithi097 ceph-mon[33317]: from='client.24469 v1:172.21.15.97:0/404208310' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:35:13.826 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:35:13 smithi097 ceph-mon[33317]: purged_snaps scrub starts 2024-03-11T21:35:13.826 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:35:13 smithi097 ceph-mon[33317]: purged_snaps scrub ok 2024-03-11T21:35:13.826 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:35:13 smithi097 ceph-mon[33317]: 5.13 scrub starts 2024-03-11T21:35:13.826 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:35:13 smithi097 ceph-mon[33317]: 5.13 scrub ok 2024-03-11T21:35:13.886 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:35:13.927 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:35:13.927 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:35:13.928 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:35:13.984 INFO:teuthology.orchestra.run.smithi092.stdout:ERROR: (22) Invalid argument 2024-03-11T21:35:13.984 INFO:teuthology.orchestra.run.smithi092.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-11T21:35:14.028 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:35:14.062 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:35:14.063 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:35:14.063 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:35:14.164 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:35:14.202 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:35:14.203 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:35:14.203 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:35:14.304 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:35:14.337 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:35:14.338 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:35:14.339 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:35:14.439 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:35:14.471 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:35:14.471 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:35:14.472 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:35:14.573 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:35:14.616 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:35:14.617 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:35:14.617 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:35:14.718 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:35:14.750 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:35:14.751 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:35:14.751 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:35:14.825 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:35:14 smithi097 ceph-mon[33317]: 4.16 scrub starts 2024-03-11T21:35:14.826 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:35:14 smithi097 ceph-mon[33317]: 4.16 scrub ok 2024-03-11T21:35:14.826 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:35:14 smithi097 ceph-mon[33317]: pgmap v424: 158 pgs: 2 active+remapped+backfill_toofull, 156 active+clean; 1014 MiB data, 6.5 GiB used, 709 GiB / 715 GiB avail; 1.1 KiB/s rd, 0 B/s wr, 1.22k op/s; 2335/49230 objects misplaced (4.743%) 2024-03-11T21:35:14.826 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:35:14 smithi097 ceph-mon[33317]: osdmap e243: 8 total, 8 up, 8 in 2024-03-11T21:35:14.839 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:35:14 smithi092 ceph-mon[29572]: 4.16 scrub starts 2024-03-11T21:35:14.839 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:35:14 smithi092 ceph-mon[29572]: 4.16 scrub ok 2024-03-11T21:35:14.839 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:35:14 smithi092 ceph-mon[29572]: pgmap v424: 158 pgs: 2 active+remapped+backfill_toofull, 156 active+clean; 1014 MiB data, 6.5 GiB used, 709 GiB / 715 GiB avail; 1.1 KiB/s rd, 0 B/s wr, 1.22k op/s; 2335/49230 objects misplaced (4.743%) 2024-03-11T21:35:14.839 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:35:14 smithi092 ceph-mon[29572]: osdmap e243: 8 total, 8 up, 8 in 2024-03-11T21:35:14.840 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:35:14 smithi092 ceph-mon[37563]: 4.16 scrub starts 2024-03-11T21:35:14.840 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:35:14 smithi092 ceph-mon[37563]: 4.16 scrub ok 2024-03-11T21:35:14.840 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:35:14 smithi092 ceph-mon[37563]: pgmap v424: 158 pgs: 2 active+remapped+backfill_toofull, 156 active+clean; 1014 MiB data, 6.5 GiB used, 709 GiB / 715 GiB avail; 1.1 KiB/s rd, 0 B/s wr, 1.22k op/s; 2335/49230 objects misplaced (4.743%) 2024-03-11T21:35:14.840 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:35:14 smithi092 ceph-mon[37563]: osdmap e243: 8 total, 8 up, 8 in 2024-03-11T21:35:14.851 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:35:14.881 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:35:14.882 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:35:14.883 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:35:14.983 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:35:15.012 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:35:15.012 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:35:15.013 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:35:15.114 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:35:15.137 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:35:15.138 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:35:15.138 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:35:15.239 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:35:15.263 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:35:15.264 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:35:15.264 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:35:15.365 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:35:15.389 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:35:15.390 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:35:15.391 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:35:15.491 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:35:15.515 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:35:15.516 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:35:15.516 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:35:15.617 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:35:15.658 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:35:15.658 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:35:15.658 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:35:15.759 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:35:15.795 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:35:15.796 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:35:15.797 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:35:15.825 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:35:15 smithi097 ceph-mon[33317]: 3.1 scrub starts 2024-03-11T21:35:15.826 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:35:15 smithi097 ceph-mon[33317]: 3.1 scrub ok 2024-03-11T21:35:15.826 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:35:15 smithi097 ceph-mon[33317]: 4.13 scrub starts 2024-03-11T21:35:15.826 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:35:15 smithi097 ceph-mon[33317]: 4.13 scrub ok 2024-03-11T21:35:15.826 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:35:15 smithi097 ceph-mon[33317]: osdmap e244: 8 total, 8 up, 8 in 2024-03-11T21:35:15.826 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:35:15 smithi097 ceph-mon[33317]: 5.b scrub starts 2024-03-11T21:35:15.826 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:35:15 smithi097 ceph-mon[33317]: 5.b scrub ok 2024-03-11T21:35:15.860 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:35:15 smithi092 ceph-mon[29572]: 3.1 scrub starts 2024-03-11T21:35:15.860 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:35:15 smithi092 ceph-mon[29572]: 3.1 scrub ok 2024-03-11T21:35:15.860 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:35:15 smithi092 ceph-mon[29572]: 4.13 scrub starts 2024-03-11T21:35:15.860 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:35:15 smithi092 ceph-mon[29572]: 4.13 scrub ok 2024-03-11T21:35:15.860 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:35:15 smithi092 ceph-mon[29572]: osdmap e244: 8 total, 8 up, 8 in 2024-03-11T21:35:15.860 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:35:15 smithi092 ceph-mon[29572]: 5.b scrub starts 2024-03-11T21:35:15.860 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:35:15 smithi092 ceph-mon[29572]: 5.b scrub ok 2024-03-11T21:35:15.861 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:35:15 smithi092 ceph-mon[37563]: 3.1 scrub starts 2024-03-11T21:35:15.861 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:35:15 smithi092 ceph-mon[37563]: 3.1 scrub ok 2024-03-11T21:35:15.861 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:35:15 smithi092 ceph-mon[37563]: 4.13 scrub starts 2024-03-11T21:35:15.861 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:35:15 smithi092 ceph-mon[37563]: 4.13 scrub ok 2024-03-11T21:35:15.861 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:35:15 smithi092 ceph-mon[37563]: osdmap e244: 8 total, 8 up, 8 in 2024-03-11T21:35:15.861 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:35:15 smithi092 ceph-mon[37563]: 5.b scrub starts 2024-03-11T21:35:15.861 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:35:15 smithi092 ceph-mon[37563]: 5.b scrub ok 2024-03-11T21:35:15.898 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:35:15.931 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:35:15.931 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:35:15.932 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:35:16.032 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:35:16.061 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:35:16.061 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:35:16.062 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:35:16.106 DEBUG:teuthology.orchestra.run.smithi092:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid 00db0016-dfed-11ee-95c8-87774f69a715 -- ceph daemon osd.2 dump_historic_ops 2024-03-11T21:35:16.162 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:35:16.192 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:35:16.192 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:35:16.192 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:35:16.293 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:35:16.333 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:35:16.333 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:35:16.334 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:35:16.406 INFO:teuthology.orchestra.run.smithi092.stdout:{ 2024-03-11T21:35:16.406 INFO:teuthology.orchestra.run.smithi092.stdout: "success": "filestore_inject_stall = '' (not observed, change may require restart) " 2024-03-11T21:35:16.406 INFO:teuthology.orchestra.run.smithi092.stdout:} 2024-03-11T21:35:16.435 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:35:16.484 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:35:16.484 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:35:16.485 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:35:16.586 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:35:16.610 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:35:16 smithi092 ceph-mon[29572]: pgmap v427: 158 pgs: 2 active+remapped+backfill_toofull, 156 active+clean; 940 MiB data, 6.3 GiB used, 709 GiB / 715 GiB avail; 1.3 KiB/s rd, 0 B/s wr, 1.21k op/s; 1753/45684 objects misplaced (3.837%) 2024-03-11T21:35:16.610 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:35:16 smithi092 ceph-mon[29572]: osdmap e245: 8 total, 8 up, 8 in 2024-03-11T21:35:16.610 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:35:16 smithi092 ceph-mon[29572]: 4.11 scrub starts 2024-03-11T21:35:16.610 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:35:16 smithi092 ceph-mon[29572]: 4.11 scrub ok 2024-03-11T21:35:16.610 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:35:16 smithi092 ceph-mon[29572]: 8.9 scrub starts 2024-03-11T21:35:16.610 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:35:16 smithi092 ceph-mon[29572]: 8.9 scrub ok 2024-03-11T21:35:16.611 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:35:16 smithi092 ceph-mon[37563]: pgmap v427: 158 pgs: 2 active+remapped+backfill_toofull, 156 active+clean; 940 MiB data, 6.3 GiB used, 709 GiB / 715 GiB avail; 1.3 KiB/s rd, 0 B/s wr, 1.21k op/s; 1753/45684 objects misplaced (3.837%) 2024-03-11T21:35:16.611 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:35:16 smithi092 ceph-mon[37563]: osdmap e245: 8 total, 8 up, 8 in 2024-03-11T21:35:16.611 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:35:16 smithi092 ceph-mon[37563]: 4.11 scrub starts 2024-03-11T21:35:16.611 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:35:16 smithi092 ceph-mon[37563]: 4.11 scrub ok 2024-03-11T21:35:16.611 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:35:16 smithi092 ceph-mon[37563]: 8.9 scrub starts 2024-03-11T21:35:16.611 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:35:16 smithi092 ceph-mon[37563]: 8.9 scrub ok 2024-03-11T21:35:16.612 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:35:16.613 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:35:16.613 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:35:16.714 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:35:16.746 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:35:16.747 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:35:16.747 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:35:16.825 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:35:16 smithi097 ceph-mon[33317]: pgmap v427: 158 pgs: 2 active+remapped+backfill_toofull, 156 active+clean; 940 MiB data, 6.3 GiB used, 709 GiB / 715 GiB avail; 1.3 KiB/s rd, 0 B/s wr, 1.21k op/s; 1753/45684 objects misplaced (3.837%) 2024-03-11T21:35:16.826 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:35:16 smithi097 ceph-mon[33317]: osdmap e245: 8 total, 8 up, 8 in 2024-03-11T21:35:16.826 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:35:16 smithi097 ceph-mon[33317]: 4.11 scrub starts 2024-03-11T21:35:16.826 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:35:16 smithi097 ceph-mon[33317]: 4.11 scrub ok 2024-03-11T21:35:16.826 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:35:16 smithi097 ceph-mon[33317]: 8.9 scrub starts 2024-03-11T21:35:16.826 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:35:16 smithi097 ceph-mon[33317]: 8.9 scrub ok 2024-03-11T21:35:16.848 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:35:16.864 INFO:teuthology.orchestra.run.smithi092.stdout:osd.0: {} 2024-03-11T21:35:16.864 INFO:teuthology.orchestra.run.smithi092.stdout:osd.1: {} 2024-03-11T21:35:16.864 INFO:teuthology.orchestra.run.smithi092.stdout:osd.2: {} 2024-03-11T21:35:16.864 INFO:teuthology.orchestra.run.smithi092.stdout:osd.3: {} 2024-03-11T21:35:16.864 INFO:teuthology.orchestra.run.smithi092.stdout:osd.4: {} 2024-03-11T21:35:16.872 INFO:teuthology.orchestra.run.smithi092.stdout:osd.5: {} 2024-03-11T21:35:16.888 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:35:16.888 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:35:16.888 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:35:16.911 INFO:teuthology.orchestra.run.smithi092.stdout:osd.6: {} 2024-03-11T21:35:16.949 INFO:teuthology.orchestra.run.smithi092.stdout:osd.7: {} 2024-03-11T21:35:16.957 INFO:teuthology.orchestra.run.smithi092.stderr:osd.0: osd_enable_op_tracker = '' 2024-03-11T21:35:16.957 INFO:teuthology.orchestra.run.smithi092.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-11T21:35:16.957 INFO:teuthology.orchestra.run.smithi092.stderr:osd.2: osd_enable_op_tracker = '' 2024-03-11T21:35:16.957 INFO:teuthology.orchestra.run.smithi092.stderr:osd.3: osd_enable_op_tracker = '' 2024-03-11T21:35:16.957 INFO:teuthology.orchestra.run.smithi092.stderr:osd.4: osd_enable_op_tracker = '' 2024-03-11T21:35:16.957 INFO:teuthology.orchestra.run.smithi092.stderr:osd.5: osd_enable_op_tracker = '' 2024-03-11T21:35:16.957 INFO:teuthology.orchestra.run.smithi092.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-11T21:35:16.957 INFO:teuthology.orchestra.run.smithi092.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-11T21:35:16.990 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:35:17.041 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:35:17.042 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:35:17.042 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:35:17.143 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:35:17.170 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:35:17.171 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:35:17.171 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:35:17.272 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:35:17.299 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:35:17.300 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:35:17.300 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:35:17.401 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:35:17.431 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:35:17.432 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:35:17.432 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:35:17.533 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:35:17.572 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:35:17.572 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:35:17.572 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:35:17.673 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:35:17.718 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:35:17.719 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:35:17.719 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:35:17.820 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:35:17.825 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:35:17 smithi097 ceph-mon[33317]: 6.7 scrub starts 2024-03-11T21:35:17.825 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:35:17 smithi097 ceph-mon[33317]: 6.1a scrub starts 2024-03-11T21:35:17.825 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:35:17 smithi097 ceph-mon[33317]: 6.1a scrub ok 2024-03-11T21:35:17.826 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:35:17 smithi097 ceph-mon[33317]: 6.7 scrub ok 2024-03-11T21:35:17.826 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:35:17 smithi097 ceph-mon[33317]: from='client.? v1:172.21.15.92:0/2910530520' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-11T21:35:17.826 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:35:17 smithi097 ceph-mon[33317]: 5.8 scrub starts 2024-03-11T21:35:17.826 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:35:17 smithi097 ceph-mon[33317]: 5.8 scrub ok 2024-03-11T21:35:17.858 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:35:17.858 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:35:17.859 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:35:17.859 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:35:17 smithi092 ceph-mon[29572]: 6.7 scrub starts 2024-03-11T21:35:17.860 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:35:17 smithi092 ceph-mon[29572]: 6.1a scrub starts 2024-03-11T21:35:17.860 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:35:17 smithi092 ceph-mon[29572]: 6.1a scrub ok 2024-03-11T21:35:17.860 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:35:17 smithi092 ceph-mon[29572]: 6.7 scrub ok 2024-03-11T21:35:17.860 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:35:17 smithi092 ceph-mon[29572]: from='client.? v1:172.21.15.92:0/2910530520' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-11T21:35:17.860 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:35:17 smithi092 ceph-mon[29572]: 5.8 scrub starts 2024-03-11T21:35:17.860 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:35:17 smithi092 ceph-mon[29572]: 5.8 scrub ok 2024-03-11T21:35:17.860 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:35:17 smithi092 ceph-mon[37563]: 6.7 scrub starts 2024-03-11T21:35:17.860 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:35:17 smithi092 ceph-mon[37563]: 6.1a scrub starts 2024-03-11T21:35:17.860 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:35:17 smithi092 ceph-mon[37563]: 6.1a scrub ok 2024-03-11T21:35:17.860 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:35:17 smithi092 ceph-mon[37563]: 6.7 scrub ok 2024-03-11T21:35:17.861 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:35:17 smithi092 ceph-mon[37563]: from='client.? v1:172.21.15.92:0/2910530520' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-11T21:35:17.861 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:35:17 smithi092 ceph-mon[37563]: 5.8 scrub starts 2024-03-11T21:35:17.861 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:35:17 smithi092 ceph-mon[37563]: 5.8 scrub ok 2024-03-11T21:35:17.959 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:35:17.996 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:35:17.997 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:35:17.998 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:35:18.099 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:35:18.150 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:35:18.151 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:35:18.151 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:35:18.253 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:35:18.284 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:35:18.284 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:35:18.285 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:35:18.362 INFO:journalctl@ceph.mgr.y.smithi092.stdout:Mar 11 21:35:17 smithi092 ceph-00db0016-dfed-11ee-95c8-87774f69a715-mgr-y[29801]: ::ffff:172.21.15.97 - - [11/Mar/2024:21:35:17] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-11T21:35:18.386 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:35:18.430 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:35:18.433 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:35:18.434 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:35:18.534 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:35:18.577 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:35:18.577 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:35:18.577 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:35:18.678 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:35:18.723 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:35:18.723 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:35:18.724 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:35:18.824 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:35:18.826 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:35:18 smithi097 ceph-mon[33317]: 4.1e scrub starts 2024-03-11T21:35:18.826 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:35:18 smithi097 ceph-mon[33317]: 4.1e scrub ok 2024-03-11T21:35:18.826 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:35:18 smithi097 ceph-mon[33317]: 5.10 scrub starts 2024-03-11T21:35:18.826 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:35:18 smithi097 ceph-mon[33317]: 5.10 scrub ok 2024-03-11T21:35:18.826 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:35:18 smithi097 ceph-mon[33317]: pgmap v429: 158 pgs: 2 active+remapped+backfill_toofull, 156 active+clean; 862 MiB data, 6.0 GiB used, 709 GiB / 715 GiB avail; 853 B/s rd, 0 B/s wr, 1.24k op/s; 1753/41964 objects misplaced (4.177%) 2024-03-11T21:35:18.826 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:35:18 smithi097 ceph-mon[33317]: 4.2 scrub starts 2024-03-11T21:35:18.826 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:35:18 smithi097 ceph-mon[33317]: 4.2 scrub ok 2024-03-11T21:35:18.826 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:35:18 smithi097 ceph-mon[33317]: 6.11 scrub starts 2024-03-11T21:35:18.826 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:35:18 smithi097 ceph-mon[33317]: 6.11 scrub ok 2024-03-11T21:35:18.860 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:35:18 smithi092 ceph-mon[29572]: 4.1e scrub starts 2024-03-11T21:35:18.860 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:35:18 smithi092 ceph-mon[29572]: 4.1e scrub ok 2024-03-11T21:35:18.860 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:35:18 smithi092 ceph-mon[29572]: 5.10 scrub starts 2024-03-11T21:35:18.860 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:35:18 smithi092 ceph-mon[29572]: 5.10 scrub ok 2024-03-11T21:35:18.860 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:35:18 smithi092 ceph-mon[29572]: pgmap v429: 158 pgs: 2 active+remapped+backfill_toofull, 156 active+clean; 862 MiB data, 6.0 GiB used, 709 GiB / 715 GiB avail; 853 B/s rd, 0 B/s wr, 1.24k op/s; 1753/41964 objects misplaced (4.177%) 2024-03-11T21:35:18.860 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:35:18 smithi092 ceph-mon[29572]: 4.2 scrub starts 2024-03-11T21:35:18.860 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:35:18 smithi092 ceph-mon[29572]: 4.2 scrub ok 2024-03-11T21:35:18.860 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:35:18 smithi092 ceph-mon[29572]: 6.11 scrub starts 2024-03-11T21:35:18.860 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:35:18 smithi092 ceph-mon[29572]: 6.11 scrub ok 2024-03-11T21:35:18.861 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:35:18 smithi092 ceph-mon[37563]: 4.1e scrub starts 2024-03-11T21:35:18.861 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:35:18 smithi092 ceph-mon[37563]: 4.1e scrub ok 2024-03-11T21:35:18.861 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:35:18 smithi092 ceph-mon[37563]: 5.10 scrub starts 2024-03-11T21:35:18.861 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:35:18 smithi092 ceph-mon[37563]: 5.10 scrub ok 2024-03-11T21:35:18.861 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:35:18 smithi092 ceph-mon[37563]: pgmap v429: 158 pgs: 2 active+remapped+backfill_toofull, 156 active+clean; 862 MiB data, 6.0 GiB used, 709 GiB / 715 GiB avail; 853 B/s rd, 0 B/s wr, 1.24k op/s; 1753/41964 objects misplaced (4.177%) 2024-03-11T21:35:18.861 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:35:18 smithi092 ceph-mon[37563]: 4.2 scrub starts 2024-03-11T21:35:18.861 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:35:18 smithi092 ceph-mon[37563]: 4.2 scrub ok 2024-03-11T21:35:18.861 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:35:18 smithi092 ceph-mon[37563]: 6.11 scrub starts 2024-03-11T21:35:18.861 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:35:18 smithi092 ceph-mon[37563]: 6.11 scrub ok 2024-03-11T21:35:18.867 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:35:18.867 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:35:18.868 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:35:18.931 INFO:teuthology.orchestra.run.smithi092.stderr:Inferring config /var/lib/ceph/00db0016-dfed-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T21:35:18.969 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:35:19.003 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:35:19.003 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:35:19.004 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:35:19.011 INFO:teuthology.orchestra.run.smithi092.stderr:Inferring config /var/lib/ceph/00db0016-dfed-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T21:35:19.105 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:35:19.147 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:35:19.148 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:35:19.149 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:35:19.249 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:35:19.293 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:35:19.294 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:35:19.294 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:35:19.395 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:35:19.435 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:35:19.435 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:35:19.436 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:35:19.537 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:35:19.576 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:35:19.577 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:35:19.577 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:35:19.678 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:35:19.713 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:35:19.713 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:35:19.714 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:35:19.815 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:35:19.825 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:35:19 smithi097 ceph-mon[33317]: purged_snaps scrub starts 2024-03-11T21:35:19.825 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:35:19 smithi097 ceph-mon[33317]: purged_snaps scrub ok 2024-03-11T21:35:19.826 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:35:19 smithi097 ceph-mon[33317]: osdmap e246: 8 total, 8 up, 8 in 2024-03-11T21:35:19.826 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:35:19 smithi097 ceph-mon[33317]: purged_snaps scrub starts 2024-03-11T21:35:19.826 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:35:19 smithi097 ceph-mon[33317]: purged_snaps scrub ok 2024-03-11T21:35:19.852 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:35:19.853 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:35:19.853 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:35:19.859 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:35:19 smithi092 ceph-mon[29572]: purged_snaps scrub starts 2024-03-11T21:35:19.860 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:35:19 smithi092 ceph-mon[29572]: purged_snaps scrub ok 2024-03-11T21:35:19.860 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:35:19 smithi092 ceph-mon[29572]: osdmap e246: 8 total, 8 up, 8 in 2024-03-11T21:35:19.860 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:35:19 smithi092 ceph-mon[29572]: purged_snaps scrub starts 2024-03-11T21:35:19.860 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:35:19 smithi092 ceph-mon[29572]: purged_snaps scrub ok 2024-03-11T21:35:19.860 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:35:19 smithi092 ceph-mon[37563]: purged_snaps scrub starts 2024-03-11T21:35:19.860 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:35:19 smithi092 ceph-mon[37563]: purged_snaps scrub ok 2024-03-11T21:35:19.860 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:35:19 smithi092 ceph-mon[37563]: osdmap e246: 8 total, 8 up, 8 in 2024-03-11T21:35:19.860 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:35:19 smithi092 ceph-mon[37563]: purged_snaps scrub starts 2024-03-11T21:35:19.860 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:35:19 smithi092 ceph-mon[37563]: purged_snaps scrub ok 2024-03-11T21:35:19.954 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:35:19.993 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:35:19.994 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:35:19.995 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:35:20.095 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:35:20.137 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:35:20.137 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:35:20.138 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:35:20.239 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:35:20.273 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:35:20.274 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:35:20.274 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:35:20.375 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:35:20.418 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:35:20.418 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:35:20.419 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:35:20.519 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:35:20.550 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:35:20.550 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:35:20.551 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:35:20.652 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:35:20.692 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:35:20.692 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:35:20.693 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:35:20.793 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:35:20.825 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:35:20.826 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:35:20 smithi097 ceph-mon[33317]: pgmap v431: 158 pgs: 1 active+remapped+backfill_toofull, 157 active+clean; 609 MiB data, 5.2 GiB used, 710 GiB / 715 GiB avail; 898 B/s rd, 0 B/s wr, 930 op/s; 669/29799 objects misplaced (2.245%); 4.8 MiB/s, 76 objects/s recovering 2024-03-11T21:35:20.826 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:35:20 smithi097 ceph-mon[33317]: Health check update: Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull (PG_BACKFILL_FULL) 2024-03-11T21:35:20.826 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:35:20 smithi097 ceph-mon[33317]: osdmap e247: 8 total, 8 up, 8 in 2024-03-11T21:35:20.826 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:35:20 smithi097 ceph-mon[33317]: 6.5 scrub starts 2024-03-11T21:35:20.826 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:35:20 smithi097 ceph-mon[33317]: 6.5 scrub ok 2024-03-11T21:35:20.826 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:35:20.827 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:35:20.853 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:35:20 smithi092 ceph-mon[29572]: pgmap v431: 158 pgs: 1 active+remapped+backfill_toofull, 157 active+clean; 609 MiB data, 5.2 GiB used, 710 GiB / 715 GiB avail; 898 B/s rd, 0 B/s wr, 930 op/s; 669/29799 objects misplaced (2.245%); 4.8 MiB/s, 76 objects/s recovering 2024-03-11T21:35:20.853 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:35:20 smithi092 ceph-mon[29572]: Health check update: Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull (PG_BACKFILL_FULL) 2024-03-11T21:35:20.853 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:35:20 smithi092 ceph-mon[29572]: osdmap e247: 8 total, 8 up, 8 in 2024-03-11T21:35:20.853 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:35:20 smithi092 ceph-mon[29572]: 6.5 scrub starts 2024-03-11T21:35:20.853 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:35:20 smithi092 ceph-mon[29572]: 6.5 scrub ok 2024-03-11T21:35:20.853 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:35:20 smithi092 ceph-mon[37563]: pgmap v431: 158 pgs: 1 active+remapped+backfill_toofull, 157 active+clean; 609 MiB data, 5.2 GiB used, 710 GiB / 715 GiB avail; 898 B/s rd, 0 B/s wr, 930 op/s; 669/29799 objects misplaced (2.245%); 4.8 MiB/s, 76 objects/s recovering 2024-03-11T21:35:20.853 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:35:20 smithi092 ceph-mon[37563]: Health check update: Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull (PG_BACKFILL_FULL) 2024-03-11T21:35:20.853 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:35:20 smithi092 ceph-mon[37563]: osdmap e247: 8 total, 8 up, 8 in 2024-03-11T21:35:20.853 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:35:20 smithi092 ceph-mon[37563]: 6.5 scrub starts 2024-03-11T21:35:20.853 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:35:20 smithi092 ceph-mon[37563]: 6.5 scrub ok 2024-03-11T21:35:20.928 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:35:20.947 DEBUG:teuthology.orchestra.run.smithi092:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid 00db0016-dfed-11ee-95c8-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-03-11T21:35:20.960 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:35:20.961 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:35:20.961 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:35:21.062 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:35:21.115 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:35:21.116 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:35:21.116 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:35:21.218 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:35:21.260 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:35:21.261 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:35:21.261 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:35:21.362 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:35:21.437 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:35:21.437 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:35:21.438 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:35:21.538 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:35:21.575 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:35:21.575 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:35:21.576 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:35:21.610 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:35:21 smithi092 ceph-mon[29572]: 4.6 scrub starts 2024-03-11T21:35:21.610 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:35:21 smithi092 ceph-mon[29572]: 4.6 scrub ok 2024-03-11T21:35:21.610 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:35:21 smithi092 ceph-mon[29572]: 4.8 scrub starts 2024-03-11T21:35:21.610 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:35:21 smithi092 ceph-mon[29572]: 4.8 scrub ok 2024-03-11T21:35:21.610 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:35:21 smithi092 ceph-mon[29572]: 5.0 scrub starts 2024-03-11T21:35:21.610 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:35:21 smithi092 ceph-mon[29572]: 5.0 scrub ok 2024-03-11T21:35:21.610 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:35:21 smithi092 ceph-mon[37563]: 4.6 scrub starts 2024-03-11T21:35:21.610 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:35:21 smithi092 ceph-mon[37563]: 4.6 scrub ok 2024-03-11T21:35:21.611 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:35:21 smithi092 ceph-mon[37563]: 4.8 scrub starts 2024-03-11T21:35:21.611 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:35:21 smithi092 ceph-mon[37563]: 4.8 scrub ok 2024-03-11T21:35:21.611 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:35:21 smithi092 ceph-mon[37563]: 5.0 scrub starts 2024-03-11T21:35:21.611 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:35:21 smithi092 ceph-mon[37563]: 5.0 scrub ok 2024-03-11T21:35:21.676 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:35:21.706 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:35:21.706 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:35:21.707 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:35:21.808 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:35:21.825 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:35:21 smithi097 ceph-mon[33317]: 4.6 scrub starts 2024-03-11T21:35:21.825 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:35:21 smithi097 ceph-mon[33317]: 4.6 scrub ok 2024-03-11T21:35:21.825 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:35:21 smithi097 ceph-mon[33317]: 4.8 scrub starts 2024-03-11T21:35:21.826 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:35:21 smithi097 ceph-mon[33317]: 4.8 scrub ok 2024-03-11T21:35:21.826 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:35:21 smithi097 ceph-mon[33317]: 5.0 scrub starts 2024-03-11T21:35:21.826 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:35:21 smithi097 ceph-mon[33317]: 5.0 scrub ok 2024-03-11T21:35:21.848 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:35:21.849 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:35:21.849 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:35:21.950 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:35:21.987 INFO:teuthology.orchestra.run.smithi092.stderr:Inferring config /var/lib/ceph/00db0016-dfed-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T21:35:22.020 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:35:22.020 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:35:22.020 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:35:22.121 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:35:22.168 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:35:22.168 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:35:22.169 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:35:22.269 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:35:22.294 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:35:22.295 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:35:22.295 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:35:22.396 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:35:22.431 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:35:22.431 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:35:22.432 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:35:22.533 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:35:22.570 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:35:22.571 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:35:22.572 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:35:22.673 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:35:22.708 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:35:22.709 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:35:22.709 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:35:22.810 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:35:22.825 INFO:journalctl@ceph.iscsi.iscsi.a.smithi097.stdout:Mar 11 21:35:22 smithi097 ceph-00db0016-dfed-11ee-95c8-87774f69a715-iscsi-iscsi-a[60943]: debug there is no tcmu-runner data available 2024-03-11T21:35:22.826 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:35:22 smithi097 ceph-mon[33317]: pgmap v433: 158 pgs: 1 active+remapped+backfill_toofull, 157 active+clean; 609 MiB data, 5.2 GiB used, 710 GiB / 715 GiB avail; 853 B/s rd, 0 B/s wr, 883 op/s; 669/29799 objects misplaced (2.245%); 4.5 MiB/s, 72 objects/s recovering 2024-03-11T21:35:22.826 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:35:22 smithi097 ceph-mon[33317]: 6.1d scrub starts 2024-03-11T21:35:22.826 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:35:22 smithi097 ceph-mon[33317]: 6.1d scrub ok 2024-03-11T21:35:22.826 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:35:22 smithi097 ceph-mon[33317]: 1.0 scrub starts 2024-03-11T21:35:22.826 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:35:22 smithi097 ceph-mon[33317]: 1.0 scrub ok 2024-03-11T21:35:22.826 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:35:22 smithi097 ceph-mon[33317]: from='mgr.14607 v1:172.21.15.92:0/1702717597' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.18", "id": [7, 2]}]: dispatch 2024-03-11T21:35:22.826 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:35:22 smithi097 ceph-mon[33317]: from='mgr.14607 ' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.18", "id": [7, 2]}]: dispatch 2024-03-11T21:35:22.826 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:35:22 smithi097 ceph-mon[33317]: 4.19 scrub starts 2024-03-11T21:35:22.827 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:35:22 smithi097 ceph-mon[33317]: 4.19 scrub ok 2024-03-11T21:35:22.827 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:35:22 smithi097 ceph-mon[33317]: from='mgr.14607 v1:172.21.15.92:0/1702717597' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T21:35:22.848 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:35:22.849 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:35:22.849 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:35:22.860 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:35:22 smithi092 ceph-mon[29572]: pgmap v433: 158 pgs: 1 active+remapped+backfill_toofull, 157 active+clean; 609 MiB data, 5.2 GiB used, 710 GiB / 715 GiB avail; 853 B/s rd, 0 B/s wr, 883 op/s; 669/29799 objects misplaced (2.245%); 4.5 MiB/s, 72 objects/s recovering 2024-03-11T21:35:22.860 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:35:22 smithi092 ceph-mon[29572]: 6.1d scrub starts 2024-03-11T21:35:22.860 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:35:22 smithi092 ceph-mon[29572]: 6.1d scrub ok 2024-03-11T21:35:22.860 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:35:22 smithi092 ceph-mon[29572]: 1.0 scrub starts 2024-03-11T21:35:22.860 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:35:22 smithi092 ceph-mon[29572]: 1.0 scrub ok 2024-03-11T21:35:22.860 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:35:22 smithi092 ceph-mon[29572]: from='mgr.14607 v1:172.21.15.92:0/1702717597' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.18", "id": [7, 2]}]: dispatch 2024-03-11T21:35:22.860 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:35:22 smithi092 ceph-mon[29572]: from='mgr.14607 ' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.18", "id": [7, 2]}]: dispatch 2024-03-11T21:35:22.860 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:35:22 smithi092 ceph-mon[29572]: 4.19 scrub starts 2024-03-11T21:35:22.861 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:35:22 smithi092 ceph-mon[29572]: 4.19 scrub ok 2024-03-11T21:35:22.861 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:35:22 smithi092 ceph-mon[29572]: from='mgr.14607 v1:172.21.15.92:0/1702717597' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T21:35:22.861 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:35:22 smithi092 ceph-mon[37563]: pgmap v433: 158 pgs: 1 active+remapped+backfill_toofull, 157 active+clean; 609 MiB data, 5.2 GiB used, 710 GiB / 715 GiB avail; 853 B/s rd, 0 B/s wr, 883 op/s; 669/29799 objects misplaced (2.245%); 4.5 MiB/s, 72 objects/s recovering 2024-03-11T21:35:22.861 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:35:22 smithi092 ceph-mon[37563]: 6.1d scrub starts 2024-03-11T21:35:22.861 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:35:22 smithi092 ceph-mon[37563]: 6.1d scrub ok 2024-03-11T21:35:22.861 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:35:22 smithi092 ceph-mon[37563]: 1.0 scrub starts 2024-03-11T21:35:22.861 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:35:22 smithi092 ceph-mon[37563]: 1.0 scrub ok 2024-03-11T21:35:22.861 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:35:22 smithi092 ceph-mon[37563]: from='mgr.14607 v1:172.21.15.92:0/1702717597' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.18", "id": [7, 2]}]: dispatch 2024-03-11T21:35:22.861 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:35:22 smithi092 ceph-mon[37563]: from='mgr.14607 ' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.18", "id": [7, 2]}]: dispatch 2024-03-11T21:35:22.861 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:35:22 smithi092 ceph-mon[37563]: 4.19 scrub starts 2024-03-11T21:35:22.862 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:35:22 smithi092 ceph-mon[37563]: 4.19 scrub ok 2024-03-11T21:35:22.862 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:35:22 smithi092 ceph-mon[37563]: from='mgr.14607 v1:172.21.15.92:0/1702717597' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T21:35:22.950 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:35:22.984 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:35:22.985 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:35:22.985 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:35:23.086 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:35:23.127 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:35:23.127 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:35:23.127 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:35:23.161 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5, 6, 7] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7] 2024-03-11T21:35:23.161 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-03-11T21:35:23.161 INFO:tasks.thrashosds.thrasher:Killing osd 3, live_osds are [0, 1, 2, 3, 4, 5, 6, 7] 2024-03-11T21:35:23.161 INFO:tasks.cephadm.osd.3:Stopping osd.3... 2024-03-11T21:35:23.161 DEBUG:teuthology.orchestra.run.smithi092:> sudo systemctl stop ceph-00db0016-dfed-11ee-95c8-87774f69a715@osd.3 2024-03-11T21:35:23.228 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:35:23.275 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:35:23.276 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:35:23.276 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:35:23.377 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:35:23.411 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:35:23.411 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:35:23.412 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:35:23.513 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:35:23.545 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:35:23.546 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:35:23.546 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:35:23.561 INFO:journalctl@ceph.osd.3.smithi092.stdout:Mar 11 21:35:23 smithi092 systemd[1]: Stopping Ceph osd.3 for 00db0016-dfed-11ee-95c8-87774f69a715... 2024-03-11T21:35:23.647 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:35:23.683 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:35:23.684 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:35:23.684 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:35:23.785 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:35:23.814 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:35:23.815 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:35:23.816 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:35:23.825 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:35:23 smithi097 ceph-mon[33317]: 8.6 scrub starts 2024-03-11T21:35:23.825 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:35:23 smithi097 ceph-mon[33317]: 8.6 scrub ok 2024-03-11T21:35:23.825 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:35:23 smithi097 ceph-mon[33317]: from='client.24469 v1:172.21.15.97:0/404208310' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:35:23.826 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:35:23 smithi097 ceph-mon[33317]: from='mgr.14607 ' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.18", "id": [7, 2]}]': finished 2024-03-11T21:35:23.826 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:35:23 smithi097 ceph-mon[33317]: osdmap e248: 8 total, 8 up, 8 in 2024-03-11T21:35:23.826 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:35:23 smithi097 ceph-mon[33317]: 5.6 scrub starts 2024-03-11T21:35:23.826 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:35:23 smithi097 ceph-mon[33317]: 5.6 scrub ok 2024-03-11T21:35:23.826 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:35:23 smithi097 ceph-mon[33317]: 8.f scrub starts 2024-03-11T21:35:23.826 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:35:23 smithi097 ceph-mon[33317]: osdmap e249: 8 total, 8 up, 8 in 2024-03-11T21:35:23.826 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:35:23 smithi097 ceph-mon[33317]: 8.f scrub ok 2024-03-11T21:35:23.861 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:35:23 smithi092 ceph-mon[29572]: 8.6 scrub starts 2024-03-11T21:35:23.861 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:35:23 smithi092 ceph-mon[29572]: 8.6 scrub ok 2024-03-11T21:35:23.861 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:35:23 smithi092 ceph-mon[29572]: from='client.24469 v1:172.21.15.97:0/404208310' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:35:23.861 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:35:23 smithi092 ceph-mon[29572]: from='mgr.14607 ' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.18", "id": [7, 2]}]': finished 2024-03-11T21:35:23.861 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:35:23 smithi092 ceph-mon[29572]: osdmap e248: 8 total, 8 up, 8 in 2024-03-11T21:35:23.861 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:35:23 smithi092 ceph-mon[29572]: 5.6 scrub starts 2024-03-11T21:35:23.861 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:35:23 smithi092 ceph-mon[29572]: 5.6 scrub ok 2024-03-11T21:35:23.861 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:35:23 smithi092 ceph-mon[29572]: 8.f scrub starts 2024-03-11T21:35:23.861 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:35:23 smithi092 ceph-mon[29572]: osdmap e249: 8 total, 8 up, 8 in 2024-03-11T21:35:23.862 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:35:23 smithi092 ceph-mon[29572]: 8.f scrub ok 2024-03-11T21:35:23.862 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:35:23 smithi092 ceph-mon[37563]: 8.6 scrub starts 2024-03-11T21:35:23.862 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:35:23 smithi092 ceph-mon[37563]: 8.6 scrub ok 2024-03-11T21:35:23.862 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:35:23 smithi092 ceph-mon[37563]: from='client.24469 v1:172.21.15.97:0/404208310' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:35:23.862 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:35:23 smithi092 ceph-mon[37563]: from='mgr.14607 ' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.18", "id": [7, 2]}]': finished 2024-03-11T21:35:23.862 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:35:23 smithi092 ceph-mon[37563]: osdmap e248: 8 total, 8 up, 8 in 2024-03-11T21:35:23.862 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:35:23 smithi092 ceph-mon[37563]: 5.6 scrub starts 2024-03-11T21:35:23.862 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:35:23 smithi092 ceph-mon[37563]: 5.6 scrub ok 2024-03-11T21:35:23.862 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:35:23 smithi092 ceph-mon[37563]: 8.f scrub starts 2024-03-11T21:35:23.862 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:35:23 smithi092 ceph-mon[37563]: osdmap e249: 8 total, 8 up, 8 in 2024-03-11T21:35:23.863 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:35:23 smithi092 ceph-mon[37563]: 8.f scrub ok 2024-03-11T21:35:23.917 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:35:23.948 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:35:23.948 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:35:23.948 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:35:24.049 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:35:24.072 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:35:24.073 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:35:24.073 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:35:24.174 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:35:24.203 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:35:24.203 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:35:24.204 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:35:24.305 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:35:24.353 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:35:24.354 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:35:24.354 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:35:24.455 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:35:24.478 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:35:24.479 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:35:24.479 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:35:24.561 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:35:24 smithi092 ceph-mon[29572]: pgmap v435: 158 pgs: 1 active+remapped+backfill_toofull, 157 active+clean; 292 MiB data, 4.1 GiB used, 711 GiB / 715 GiB avail; 1.7 KiB/s rd, 0 B/s wr, 1.52k op/s; 669/14592 objects misplaced (4.585%); 4.5 MiB/s, 72 objects/s recovering 2024-03-11T21:35:24.580 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:35:24.613 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:35:24.613 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:35:24.614 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:35:24.714 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:35:24.752 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:35:24.753 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:35:24.753 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:35:24.825 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:35:24 smithi097 ceph-mon[33317]: pgmap v435: 158 pgs: 1 active+remapped+backfill_toofull, 157 active+clean; 292 MiB data, 4.1 GiB used, 711 GiB / 715 GiB avail; 1.7 KiB/s rd, 0 B/s wr, 1.52k op/s; 669/14592 objects misplaced (4.585%); 4.5 MiB/s, 72 objects/s recovering 2024-03-11T21:35:24.854 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:35:24.859 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:35:24 smithi092 ceph-mon[37563]: pgmap v435: 158 pgs: 1 active+remapped+backfill_toofull, 157 active+clean; 292 MiB data, 4.1 GiB used, 711 GiB / 715 GiB avail; 1.7 KiB/s rd, 0 B/s wr, 1.52k op/s; 669/14592 objects misplaced (4.585%); 4.5 MiB/s, 72 objects/s recovering 2024-03-11T21:35:24.890 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:35:24.891 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:35:24.891 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:35:24.992 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:35:25.034 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:35:25.035 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:35:25.035 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:35:25.057 INFO:teuthology.orchestra.run.smithi092.stdout:{ 2024-03-11T21:35:25.057 INFO:teuthology.orchestra.run.smithi092.stdout: "size": 20, 2024-03-11T21:35:25.057 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 600, 2024-03-11T21:35:25.057 INFO:teuthology.orchestra.run.smithi092.stdout: "ops": [ 2024-03-11T21:35:25.057 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:35:25.057 INFO:teuthology.orchestra.run.smithi092.stdout: "description": "osd_op(client.34715.0:32047 7.8 7:1cc332d5:::benchmark_data_smithi092_76040_object32046:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e105)", 2024-03-11T21:35:25.057 INFO:teuthology.orchestra.run.smithi092.stdout: "initiated_at": "2024-03-11T21:27:57.779953+0000", 2024-03-11T21:35:25.057 INFO:teuthology.orchestra.run.smithi092.stdout: "age": 446.66698368700003, 2024-03-11T21:35:25.058 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.084056040999999998, 2024-03-11T21:35:25.058 INFO:teuthology.orchestra.run.smithi092.stdout: "type_data": { 2024-03-11T21:35:25.058 INFO:teuthology.orchestra.run.smithi092.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-11T21:35:25.058 INFO:teuthology.orchestra.run.smithi092.stdout: "client_info": { 2024-03-11T21:35:25.058 INFO:teuthology.orchestra.run.smithi092.stdout: "client": "client.34715", 2024-03-11T21:35:25.058 INFO:teuthology.orchestra.run.smithi092.stdout: "client_addr": "v1:172.21.15.92:0/2403132573", 2024-03-11T21:35:25.058 INFO:teuthology.orchestra.run.smithi092.stdout: "tid": 32047 2024-03-11T21:35:25.058 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:35:25.058 INFO:teuthology.orchestra.run.smithi092.stdout: "events": [ 2024-03-11T21:35:25.058 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:35:25.058 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "initiated", 2024-03-11T21:35:25.058 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:27:57.779953+0000", 2024-03-11T21:35:25.058 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-03-11T21:35:25.059 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:35:25.059 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:35:25.059 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "header_read", 2024-03-11T21:35:25.059 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:27:57.779953+0000", 2024-03-11T21:35:25.059 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-03-11T21:35:25.059 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:35:25.059 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:35:25.059 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "throttled", 2024-03-11T21:35:25.059 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:27:57.779956+0000", 2024-03-11T21:35:25.059 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 2.069e-06 2024-03-11T21:35:25.059 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:35:25.059 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:35:25.059 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "all_read", 2024-03-11T21:35:25.059 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:27:57.779982+0000", 2024-03-11T21:35:25.060 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 2.6601000000000002e-05 2024-03-11T21:35:25.060 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:35:25.060 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:35:25.060 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "dispatched", 2024-03-11T21:35:25.060 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:27:57.780005+0000", 2024-03-11T21:35:25.060 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 2.2756e-05 2024-03-11T21:35:25.060 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:35:25.060 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:35:25.060 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "queued_for_pg", 2024-03-11T21:35:25.060 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:27:57.780014+0000", 2024-03-11T21:35:25.060 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 8.7630000000000006e-06 2024-03-11T21:35:25.060 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:35:25.060 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:35:25.060 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-03-11T21:35:25.061 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:27:57.780080+0000", 2024-03-11T21:35:25.061 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 6.6018000000000004e-05 2024-03-11T21:35:25.061 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:35:25.061 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:35:25.061 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "started", 2024-03-11T21:35:25.061 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:27:57.781113+0000", 2024-03-11T21:35:25.061 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.001033312 2024-03-11T21:35:25.061 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:35:25.061 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:35:25.061 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "waiting for subops from 4,7", 2024-03-11T21:35:25.061 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:27:57.781655+0000", 2024-03-11T21:35:25.061 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.000542341 2024-03-11T21:35:25.061 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:35:25.061 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:35:25.062 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "sub_op_commit_rec", 2024-03-11T21:35:25.062 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:27:57.783494+0000", 2024-03-11T21:35:25.062 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.0018391499999999999 2024-03-11T21:35:25.062 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:35:25.062 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:35:25.062 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "op_commit", 2024-03-11T21:35:25.062 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:27:57.783972+0000", 2024-03-11T21:35:25.062 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00047797699999999998 2024-03-11T21:35:25.062 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:35:25.062 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:35:25.062 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "sub_op_commit_rec", 2024-03-11T21:35:25.062 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:27:57.863900+0000", 2024-03-11T21:35:25.062 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.079927953999999996 2024-03-11T21:35:25.062 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:35:25.063 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:35:25.063 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "commit_sent", 2024-03-11T21:35:25.063 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:27:57.863960+0000", 2024-03-11T21:35:25.063 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 5.9917999999999998e-05 2024-03-11T21:35:25.063 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:35:25.063 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:35:25.063 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "done", 2024-03-11T21:35:25.063 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:27:57.864009+0000", 2024-03-11T21:35:25.063 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 4.9181999999999998e-05 2024-03-11T21:35:25.063 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-03-11T21:35:25.063 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-03-11T21:35:25.063 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-03-11T21:35:25.063 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:35:25.063 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:35:25.064 INFO:teuthology.orchestra.run.smithi092.stdout: "description": "osd_op(client.34715.0:104678 7.e 7:77299287:::benchmark_data_smithi092_76040_object104677:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e117)", 2024-03-11T21:35:25.064 INFO:teuthology.orchestra.run.smithi092.stdout: "initiated_at": "2024-03-11T21:28:57.288999+0000", 2024-03-11T21:35:25.064 INFO:teuthology.orchestra.run.smithi092.stdout: "age": 387.15793790999999, 2024-03-11T21:35:25.064 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.092710055999999999, 2024-03-11T21:35:25.064 INFO:teuthology.orchestra.run.smithi092.stdout: "type_data": { 2024-03-11T21:35:25.064 INFO:teuthology.orchestra.run.smithi092.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-11T21:35:25.064 INFO:teuthology.orchestra.run.smithi092.stdout: "client_info": { 2024-03-11T21:35:25.064 INFO:teuthology.orchestra.run.smithi092.stdout: "client": "client.34715", 2024-03-11T21:35:25.064 INFO:teuthology.orchestra.run.smithi092.stdout: "client_addr": "v1:172.21.15.92:0/2403132573", 2024-03-11T21:35:25.064 INFO:teuthology.orchestra.run.smithi092.stdout: "tid": 104678 2024-03-11T21:35:25.064 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:35:25.064 INFO:teuthology.orchestra.run.smithi092.stdout: "events": [ 2024-03-11T21:35:25.064 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:35:25.064 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "initiated", 2024-03-11T21:35:25.065 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:28:57.288999+0000", 2024-03-11T21:35:25.065 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-03-11T21:35:25.065 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:35:25.065 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:35:25.065 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "header_read", 2024-03-11T21:35:25.065 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:28:57.288999+0000", 2024-03-11T21:35:25.065 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-03-11T21:35:25.065 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:35:25.065 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:35:25.065 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "throttled", 2024-03-11T21:35:25.065 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:28:57.289041+0000", 2024-03-11T21:35:25.065 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 4.1856e-05 2024-03-11T21:35:25.065 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:35:25.065 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:35:25.066 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "all_read", 2024-03-11T21:35:25.066 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:28:57.289064+0000", 2024-03-11T21:35:25.066 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 2.3112000000000002e-05 2024-03-11T21:35:25.066 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:35:25.066 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:35:25.066 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "dispatched", 2024-03-11T21:35:25.066 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:28:57.289075+0000", 2024-03-11T21:35:25.066 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.0312000000000001e-05 2024-03-11T21:35:25.066 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:35:25.066 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:35:25.066 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "queued_for_pg", 2024-03-11T21:35:25.066 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:28:57.289082+0000", 2024-03-11T21:35:25.066 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 7.3599999999999998e-06 2024-03-11T21:35:25.066 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:35:25.067 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:35:25.067 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-03-11T21:35:25.067 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:28:57.289159+0000", 2024-03-11T21:35:25.067 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 7.7372000000000004e-05 2024-03-11T21:35:25.067 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:35:25.067 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:35:25.067 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "started", 2024-03-11T21:35:25.067 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:28:57.289288+0000", 2024-03-11T21:35:25.067 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00012846999999999999 2024-03-11T21:35:25.067 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:35:25.067 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:35:25.067 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "waiting for subops from 0,5", 2024-03-11T21:35:25.067 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:28:57.289421+0000", 2024-03-11T21:35:25.067 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00013358000000000001 2024-03-11T21:35:25.067 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:35:25.068 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:35:25.068 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "sub_op_commit_rec", 2024-03-11T21:35:25.068 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:28:57.378143+0000", 2024-03-11T21:35:25.068 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.088721513000000002 2024-03-11T21:35:25.068 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:35:25.068 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:35:25.068 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "sub_op_commit_rec", 2024-03-11T21:35:25.068 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:28:57.378266+0000", 2024-03-11T21:35:25.068 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00012305700000000001 2024-03-11T21:35:25.068 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:35:25.068 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:35:25.068 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "op_commit", 2024-03-11T21:35:25.068 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:28:57.381595+0000", 2024-03-11T21:35:25.068 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.0033288409999999999 2024-03-11T21:35:25.069 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:35:25.069 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:35:25.069 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "commit_sent", 2024-03-11T21:35:25.069 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:28:57.381664+0000", 2024-03-11T21:35:25.069 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 6.9708000000000004e-05 2024-03-11T21:35:25.069 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:35:25.069 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:35:25.069 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "done", 2024-03-11T21:35:25.069 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:28:57.381709+0000", 2024-03-11T21:35:25.069 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 4.4875e-05 2024-03-11T21:35:25.069 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-03-11T21:35:25.069 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-03-11T21:35:25.069 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-03-11T21:35:25.069 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:35:25.069 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:35:25.070 INFO:teuthology.orchestra.run.smithi092.stdout: "description": "osd_op(client.34715.0:104682 7.e 7:7b3e8d68:::benchmark_data_smithi092_76040_object104681:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e117)", 2024-03-11T21:35:25.070 INFO:teuthology.orchestra.run.smithi092.stdout: "initiated_at": "2024-03-11T21:28:57.292170+0000", 2024-03-11T21:35:25.070 INFO:teuthology.orchestra.run.smithi092.stdout: "age": 387.15476697000003, 2024-03-11T21:35:25.070 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.092283122999999995, 2024-03-11T21:35:25.070 INFO:teuthology.orchestra.run.smithi092.stdout: "type_data": { 2024-03-11T21:35:25.070 INFO:teuthology.orchestra.run.smithi092.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-11T21:35:25.070 INFO:teuthology.orchestra.run.smithi092.stdout: "client_info": { 2024-03-11T21:35:25.070 INFO:teuthology.orchestra.run.smithi092.stdout: "client": "client.34715", 2024-03-11T21:35:25.070 INFO:teuthology.orchestra.run.smithi092.stdout: "client_addr": "v1:172.21.15.92:0/2403132573", 2024-03-11T21:35:25.070 INFO:teuthology.orchestra.run.smithi092.stdout: "tid": 104682 2024-03-11T21:35:25.070 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:35:25.070 INFO:teuthology.orchestra.run.smithi092.stdout: "events": [ 2024-03-11T21:35:25.070 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:35:25.071 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "initiated", 2024-03-11T21:35:25.071 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:28:57.292170+0000", 2024-03-11T21:35:25.071 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-03-11T21:35:25.071 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:35:25.071 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:35:25.071 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "header_read", 2024-03-11T21:35:25.071 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:28:57.292170+0000", 2024-03-11T21:35:25.071 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-03-11T21:35:25.071 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:35:25.071 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:35:25.071 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "throttled", 2024-03-11T21:35:25.071 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:28:57.292173+0000", 2024-03-11T21:35:25.071 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 2.542e-06 2024-03-11T21:35:25.071 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:35:25.071 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:35:25.072 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "all_read", 2024-03-11T21:35:25.072 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:28:57.292200+0000", 2024-03-11T21:35:25.072 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 2.7288e-05 2024-03-11T21:35:25.072 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:35:25.072 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:35:25.072 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "dispatched", 2024-03-11T21:35:25.072 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:28:57.375541+0000", 2024-03-11T21:35:25.072 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.083340996000000001 2024-03-11T21:35:25.072 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:35:25.072 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:35:25.072 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "queued_for_pg", 2024-03-11T21:35:25.072 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:28:57.375557+0000", 2024-03-11T21:35:25.072 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.5933000000000001e-05 2024-03-11T21:35:25.072 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:35:25.073 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:35:25.073 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-03-11T21:35:25.073 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:28:57.378303+0000", 2024-03-11T21:35:25.073 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.002745824 2024-03-11T21:35:25.073 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:35:25.073 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:35:25.073 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "started", 2024-03-11T21:35:25.073 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:28:57.378444+0000", 2024-03-11T21:35:25.073 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00014127399999999999 2024-03-11T21:35:25.073 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:35:25.073 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:35:25.073 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "waiting for subops from 0,5", 2024-03-11T21:35:25.073 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:28:57.378537+0000", 2024-03-11T21:35:25.073 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 9.3263000000000005e-05 2024-03-11T21:35:25.073 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:35:25.074 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:35:25.074 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "op_commit", 2024-03-11T21:35:25.074 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:28:57.381723+0000", 2024-03-11T21:35:25.074 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.0031855989999999999 2024-03-11T21:35:25.074 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:35:25.074 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:35:25.074 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "sub_op_commit_rec", 2024-03-11T21:35:25.074 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:28:57.381875+0000", 2024-03-11T21:35:25.074 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00015235199999999999 2024-03-11T21:35:25.074 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:35:25.074 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:35:25.074 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "sub_op_commit_rec", 2024-03-11T21:35:25.074 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:28:57.384116+0000", 2024-03-11T21:35:25.074 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.002240279 2024-03-11T21:35:25.075 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:35:25.075 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:35:25.075 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "commit_sent", 2024-03-11T21:35:25.075 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:28:57.384313+0000", 2024-03-11T21:35:25.075 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.000197648 2024-03-11T21:35:25.075 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:35:25.075 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:35:25.075 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "done", 2024-03-11T21:35:25.075 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:28:57.384453+0000", 2024-03-11T21:35:25.075 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.000140125 2024-03-11T21:35:25.075 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-03-11T21:35:25.075 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-03-11T21:35:25.075 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-03-11T21:35:25.075 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:35:25.076 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:35:25.076 INFO:teuthology.orchestra.run.smithi092.stdout: "description": "osd_op(client.34715.0:115732 7.e 7:7582fdce:::benchmark_data_smithi092_76040_object115731:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e119)", 2024-03-11T21:35:25.076 INFO:teuthology.orchestra.run.smithi092.stdout: "initiated_at": "2024-03-11T21:29:08.166630+0000", 2024-03-11T21:35:25.076 INFO:teuthology.orchestra.run.smithi092.stdout: "age": 376.280306946, 2024-03-11T21:35:25.076 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.099177280000000007, 2024-03-11T21:35:25.076 INFO:teuthology.orchestra.run.smithi092.stdout: "type_data": { 2024-03-11T21:35:25.076 INFO:teuthology.orchestra.run.smithi092.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-11T21:35:25.076 INFO:teuthology.orchestra.run.smithi092.stdout: "client_info": { 2024-03-11T21:35:25.076 INFO:teuthology.orchestra.run.smithi092.stdout: "client": "client.34715", 2024-03-11T21:35:25.076 INFO:teuthology.orchestra.run.smithi092.stdout: "client_addr": "v1:172.21.15.92:0/2403132573", 2024-03-11T21:35:25.076 INFO:teuthology.orchestra.run.smithi092.stdout: "tid": 115732 2024-03-11T21:35:25.076 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:35:25.076 INFO:teuthology.orchestra.run.smithi092.stdout: "events": [ 2024-03-11T21:35:25.076 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:35:25.076 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "initiated", 2024-03-11T21:35:25.077 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:29:08.166630+0000", 2024-03-11T21:35:25.077 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-03-11T21:35:25.077 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:35:25.077 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:35:25.077 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "header_read", 2024-03-11T21:35:25.077 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:29:08.166630+0000", 2024-03-11T21:35:25.077 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-03-11T21:35:25.077 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:35:25.077 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:35:25.077 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "throttled", 2024-03-11T21:35:25.077 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:29:08.166634+0000", 2024-03-11T21:35:25.077 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 3.4680000000000001e-06 2024-03-11T21:35:25.077 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:35:25.077 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:35:25.077 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "all_read", 2024-03-11T21:35:25.078 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:29:08.166660+0000", 2024-03-11T21:35:25.078 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 2.6512e-05 2024-03-11T21:35:25.078 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:35:25.078 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:35:25.078 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "dispatched", 2024-03-11T21:35:25.078 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:29:08.166674+0000", 2024-03-11T21:35:25.078 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.382e-05 2024-03-11T21:35:25.078 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:35:25.078 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:35:25.078 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "queued_for_pg", 2024-03-11T21:35:25.078 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:29:08.166685+0000", 2024-03-11T21:35:25.078 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.1277999999999999e-05 2024-03-11T21:35:25.078 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:35:25.078 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:35:25.079 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-03-11T21:35:25.079 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:29:08.166750+0000", 2024-03-11T21:35:25.079 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 6.4522000000000003e-05 2024-03-11T21:35:25.079 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:35:25.079 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:35:25.079 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "started", 2024-03-11T21:35:25.079 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:29:08.166894+0000", 2024-03-11T21:35:25.079 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.000144499 2024-03-11T21:35:25.079 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:35:25.079 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:35:25.079 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "waiting for subops from 0,5", 2024-03-11T21:35:25.079 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:29:08.167063+0000", 2024-03-11T21:35:25.079 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00016860700000000001 2024-03-11T21:35:25.079 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:35:25.080 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:35:25.080 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "sub_op_commit_rec", 2024-03-11T21:35:25.080 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:29:08.172918+0000", 2024-03-11T21:35:25.080 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.0058549780000000003 2024-03-11T21:35:25.080 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:35:25.080 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:35:25.080 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "op_commit", 2024-03-11T21:35:25.080 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:29:08.181135+0000", 2024-03-11T21:35:25.080 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.0082175979999999996 2024-03-11T21:35:25.080 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:35:25.080 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:35:25.080 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "sub_op_commit_rec", 2024-03-11T21:35:25.080 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:29:08.265705+0000", 2024-03-11T21:35:25.080 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.084569474000000006 2024-03-11T21:35:25.080 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:35:25.080 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:35:25.081 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "commit_sent", 2024-03-11T21:35:25.081 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:29:08.265763+0000", 2024-03-11T21:35:25.081 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 5.8242999999999998e-05 2024-03-11T21:35:25.081 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:35:25.081 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:35:25.081 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "done", 2024-03-11T21:35:25.081 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:29:08.265807+0000", 2024-03-11T21:35:25.081 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 4.4280999999999998e-05 2024-03-11T21:35:25.081 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-03-11T21:35:25.081 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-03-11T21:35:25.081 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-03-11T21:35:25.081 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:35:25.081 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:35:25.081 INFO:teuthology.orchestra.run.smithi092.stdout: "description": "osd_op(client.34715.0:115741 7.e 7:72f34471:::benchmark_data_smithi092_76040_object115740:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e119)", 2024-03-11T21:35:25.081 INFO:teuthology.orchestra.run.smithi092.stdout: "initiated_at": "2024-03-11T21:29:08.181657+0000", 2024-03-11T21:35:25.082 INFO:teuthology.orchestra.run.smithi092.stdout: "age": 376.26527974599998, 2024-03-11T21:35:25.082 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.084254920999999997, 2024-03-11T21:35:25.082 INFO:teuthology.orchestra.run.smithi092.stdout: "type_data": { 2024-03-11T21:35:25.082 INFO:teuthology.orchestra.run.smithi092.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-11T21:35:25.082 INFO:teuthology.orchestra.run.smithi092.stdout: "client_info": { 2024-03-11T21:35:25.082 INFO:teuthology.orchestra.run.smithi092.stdout: "client": "client.34715", 2024-03-11T21:35:25.082 INFO:teuthology.orchestra.run.smithi092.stdout: "client_addr": "v1:172.21.15.92:0/2403132573", 2024-03-11T21:35:25.082 INFO:teuthology.orchestra.run.smithi092.stdout: "tid": 115741 2024-03-11T21:35:25.082 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:35:25.082 INFO:teuthology.orchestra.run.smithi092.stdout: "events": [ 2024-03-11T21:35:25.082 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:35:25.082 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "initiated", 2024-03-11T21:35:25.082 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:29:08.181657+0000", 2024-03-11T21:35:25.082 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-03-11T21:35:25.083 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:35:25.083 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:35:25.083 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "header_read", 2024-03-11T21:35:25.083 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:29:08.181657+0000", 2024-03-11T21:35:25.083 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-03-11T21:35:25.083 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:35:25.083 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:35:25.083 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "throttled", 2024-03-11T21:35:25.083 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:29:08.181659+0000", 2024-03-11T21:35:25.083 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.6980000000000001e-06 2024-03-11T21:35:25.083 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:35:25.083 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:35:25.083 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "all_read", 2024-03-11T21:35:25.083 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:29:08.181688+0000", 2024-03-11T21:35:25.083 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 2.8827999999999999e-05 2024-03-11T21:35:25.083 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:35:25.084 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:35:25.084 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "dispatched", 2024-03-11T21:35:25.084 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:29:08.181700+0000", 2024-03-11T21:35:25.084 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.1939999999999999e-05 2024-03-11T21:35:25.084 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:35:25.084 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:35:25.084 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "queued_for_pg", 2024-03-11T21:35:25.084 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:29:08.181708+0000", 2024-03-11T21:35:25.084 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 8.5890000000000003e-06 2024-03-11T21:35:25.084 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:35:25.084 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:35:25.084 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-03-11T21:35:25.084 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:29:08.181755+0000", 2024-03-11T21:35:25.084 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 4.6702000000000001e-05 2024-03-11T21:35:25.084 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:35:25.085 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:35:25.085 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "started", 2024-03-11T21:35:25.085 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:29:08.181870+0000", 2024-03-11T21:35:25.085 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00011508 2024-03-11T21:35:25.085 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:35:25.085 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:35:25.085 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "waiting for subops from 0,5", 2024-03-11T21:35:25.085 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:29:08.182022+0000", 2024-03-11T21:35:25.085 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00015190099999999999 2024-03-11T21:35:25.085 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:35:25.085 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:35:25.085 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "sub_op_commit_rec", 2024-03-11T21:35:25.085 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:29:08.184981+0000", 2024-03-11T21:35:25.085 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.0029593559999999998 2024-03-11T21:35:25.086 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:35:25.086 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:35:25.086 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "op_commit", 2024-03-11T21:35:25.086 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:29:08.186599+0000", 2024-03-11T21:35:25.086 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.0016174830000000001 2024-03-11T21:35:25.086 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:35:25.086 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:35:25.086 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "sub_op_commit_rec", 2024-03-11T21:35:25.086 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:29:08.265841+0000", 2024-03-11T21:35:25.086 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.079241696 2024-03-11T21:35:25.086 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:35:25.086 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:35:25.086 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "commit_sent", 2024-03-11T21:35:25.086 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:29:08.265883+0000", 2024-03-11T21:35:25.087 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 4.2534000000000003e-05 2024-03-11T21:35:25.087 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:35:25.087 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:35:25.087 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "done", 2024-03-11T21:35:25.087 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:29:08.265912+0000", 2024-03-11T21:35:25.087 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 2.9113999999999999e-05 2024-03-11T21:35:25.087 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-03-11T21:35:25.087 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-03-11T21:35:25.087 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-03-11T21:35:25.087 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:35:25.087 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:35:25.087 INFO:teuthology.orchestra.run.smithi092.stdout: "description": "osd_op(client.34715.0:116287 7.8 7:161bac11:::benchmark_data_smithi092_76040_object116286:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e119)", 2024-03-11T21:35:25.087 INFO:teuthology.orchestra.run.smithi092.stdout: "initiated_at": "2024-03-11T21:29:08.771810+0000", 2024-03-11T21:35:25.087 INFO:teuthology.orchestra.run.smithi092.stdout: "age": 375.67512757200001, 2024-03-11T21:35:25.088 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.10587118700000001, 2024-03-11T21:35:25.088 INFO:teuthology.orchestra.run.smithi092.stdout: "type_data": { 2024-03-11T21:35:25.088 INFO:teuthology.orchestra.run.smithi092.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-11T21:35:25.088 INFO:teuthology.orchestra.run.smithi092.stdout: "client_info": { 2024-03-11T21:35:25.088 INFO:teuthology.orchestra.run.smithi092.stdout: "client": "client.34715", 2024-03-11T21:35:25.088 INFO:teuthology.orchestra.run.smithi092.stdout: "client_addr": "v1:172.21.15.92:0/2403132573", 2024-03-11T21:35:25.088 INFO:teuthology.orchestra.run.smithi092.stdout: "tid": 116287 2024-03-11T21:35:25.088 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:35:25.088 INFO:teuthology.orchestra.run.smithi092.stdout: "events": [ 2024-03-11T21:35:25.088 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:35:25.088 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "initiated", 2024-03-11T21:35:25.088 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:29:08.771810+0000", 2024-03-11T21:35:25.088 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-03-11T21:35:25.089 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:35:25.089 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:35:25.089 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "header_read", 2024-03-11T21:35:25.089 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:29:08.771810+0000", 2024-03-11T21:35:25.089 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-03-11T21:35:25.089 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:35:25.089 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:35:25.089 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "throttled", 2024-03-11T21:35:25.089 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:29:08.771811+0000", 2024-03-11T21:35:25.089 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.793e-06 2024-03-11T21:35:25.089 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:35:25.089 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:35:25.089 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "all_read", 2024-03-11T21:35:25.089 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:29:08.771832+0000", 2024-03-11T21:35:25.089 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 2.0641000000000001e-05 2024-03-11T21:35:25.090 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:35:25.090 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:35:25.090 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "dispatched", 2024-03-11T21:35:25.090 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:29:08.771841+0000", 2024-03-11T21:35:25.090 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 9.2939999999999994e-06 2024-03-11T21:35:25.090 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:35:25.090 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:35:25.090 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "queued_for_pg", 2024-03-11T21:35:25.090 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:29:08.771849+0000", 2024-03-11T21:35:25.090 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 7.2239999999999998e-06 2024-03-11T21:35:25.090 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:35:25.090 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:35:25.090 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-03-11T21:35:25.090 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:29:08.771933+0000", 2024-03-11T21:35:25.091 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 8.4158000000000003e-05 2024-03-11T21:35:25.091 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:35:25.091 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:35:25.091 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "started", 2024-03-11T21:35:25.091 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:29:08.772098+0000", 2024-03-11T21:35:25.091 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00016576199999999999 2024-03-11T21:35:25.091 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:35:25.091 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:35:25.091 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "waiting for subops from 4,7", 2024-03-11T21:35:25.091 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:29:08.772361+0000", 2024-03-11T21:35:25.091 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00026245400000000002 2024-03-11T21:35:25.091 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:35:25.091 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:35:25.091 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "op_commit", 2024-03-11T21:35:25.091 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:29:08.877415+0000", 2024-03-11T21:35:25.092 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.10505452 2024-03-11T21:35:25.092 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:35:25.092 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:35:25.092 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "sub_op_commit_rec", 2024-03-11T21:35:25.092 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:29:08.877469+0000", 2024-03-11T21:35:25.092 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 5.3115000000000002e-05 2024-03-11T21:35:25.092 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:35:25.092 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:35:25.092 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "sub_op_commit_rec", 2024-03-11T21:35:25.092 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:29:08.877521+0000", 2024-03-11T21:35:25.092 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 5.2649999999999999e-05 2024-03-11T21:35:25.092 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:35:25.092 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:35:25.092 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "commit_sent", 2024-03-11T21:35:25.092 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:29:08.877632+0000", 2024-03-11T21:35:25.093 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00011079800000000001 2024-03-11T21:35:25.093 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:35:25.093 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:35:25.093 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "done", 2024-03-11T21:35:25.093 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:29:08.877681+0000", 2024-03-11T21:35:25.093 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 4.8778e-05 2024-03-11T21:35:25.093 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-03-11T21:35:25.093 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-03-11T21:35:25.093 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-03-11T21:35:25.093 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:35:25.093 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:35:25.093 INFO:teuthology.orchestra.run.smithi092.stdout: "description": "osd_repop(client.34715.0:116286 7.5 e119/99 7:a74abdea:::benchmark_data_smithi092_76040_object116285:head v 119'7286, mlcod=119'7283)", 2024-03-11T21:35:25.093 INFO:teuthology.orchestra.run.smithi092.stdout: "initiated_at": "2024-03-11T21:29:08.772243+0000", 2024-03-11T21:35:25.093 INFO:teuthology.orchestra.run.smithi092.stdout: "age": 375.67469429099998, 2024-03-11T21:35:25.094 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.105122486, 2024-03-11T21:35:25.094 INFO:teuthology.orchestra.run.smithi092.stdout: "type_data": { 2024-03-11T21:35:25.094 INFO:teuthology.orchestra.run.smithi092.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-11T21:35:25.094 INFO:teuthology.orchestra.run.smithi092.stdout: "events": [ 2024-03-11T21:35:25.094 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:35:25.094 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "initiated", 2024-03-11T21:35:25.094 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:29:08.772243+0000", 2024-03-11T21:35:25.094 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-03-11T21:35:25.094 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:35:25.094 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:35:25.094 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "header_read", 2024-03-11T21:35:25.094 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:29:08.772243+0000", 2024-03-11T21:35:25.094 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-03-11T21:35:25.094 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:35:25.094 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:35:25.095 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "throttled", 2024-03-11T21:35:25.095 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:29:08.772244+0000", 2024-03-11T21:35:25.095 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 9.7600000000000006e-07 2024-03-11T21:35:25.095 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:35:25.095 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:35:25.095 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "all_read", 2024-03-11T21:35:25.095 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:29:08.772287+0000", 2024-03-11T21:35:25.095 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 4.3052e-05 2024-03-11T21:35:25.095 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:35:25.095 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:35:25.095 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "dispatched", 2024-03-11T21:35:25.095 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:29:08.772295+0000", 2024-03-11T21:35:25.095 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 7.7989999999999994e-06 2024-03-11T21:35:25.095 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:35:25.095 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:35:25.096 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "queued_for_pg", 2024-03-11T21:35:25.096 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:29:08.772301+0000", 2024-03-11T21:35:25.096 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 6.1319999999999999e-06 2024-03-11T21:35:25.096 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:35:25.096 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:35:25.096 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-03-11T21:35:25.096 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:29:08.772348+0000", 2024-03-11T21:35:25.096 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 4.6999000000000002e-05 2024-03-11T21:35:25.096 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:35:25.096 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:35:25.096 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "started", 2024-03-11T21:35:25.096 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:29:08.772377+0000", 2024-03-11T21:35:25.096 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 2.9246000000000001e-05 2024-03-11T21:35:25.096 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:35:25.097 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:35:25.097 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "commit_sent", 2024-03-11T21:35:25.097 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:29:08.775548+0000", 2024-03-11T21:35:25.097 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.0031711740000000001 2024-03-11T21:35:25.097 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:35:25.097 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:35:25.097 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "done", 2024-03-11T21:35:25.097 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:29:08.877365+0000", 2024-03-11T21:35:25.097 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.101817108 2024-03-11T21:35:25.097 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-03-11T21:35:25.097 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-03-11T21:35:25.097 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-03-11T21:35:25.097 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:35:25.097 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:35:25.097 INFO:teuthology.orchestra.run.smithi092.stdout: "description": "osd_repop_reply(client.34715.0:116287 7.8 e119/99 ondisk, result = 0)", 2024-03-11T21:35:25.097 INFO:teuthology.orchestra.run.smithi092.stdout: "initiated_at": "2024-03-11T21:29:08.776070+0000", 2024-03-11T21:35:25.098 INFO:teuthology.orchestra.run.smithi092.stdout: "age": 375.67086723900002, 2024-03-11T21:35:25.098 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.10140663900000001, 2024-03-11T21:35:25.098 INFO:teuthology.orchestra.run.smithi092.stdout: "type_data": { 2024-03-11T21:35:25.098 INFO:teuthology.orchestra.run.smithi092.stdout: "flag_point": "started", 2024-03-11T21:35:25.098 INFO:teuthology.orchestra.run.smithi092.stdout: "events": [ 2024-03-11T21:35:25.098 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:35:25.098 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "initiated", 2024-03-11T21:35:25.098 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:29:08.776070+0000", 2024-03-11T21:35:25.098 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-03-11T21:35:25.098 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:35:25.098 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:35:25.098 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "header_read", 2024-03-11T21:35:25.098 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:29:08.776070+0000", 2024-03-11T21:35:25.098 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-03-11T21:35:25.099 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:35:25.099 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:35:25.099 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "throttled", 2024-03-11T21:35:25.099 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:29:08.776071+0000", 2024-03-11T21:35:25.099 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.265e-06 2024-03-11T21:35:25.099 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:35:25.099 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:35:25.099 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "all_read", 2024-03-11T21:35:25.099 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:29:08.776075+0000", 2024-03-11T21:35:25.099 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 4.2930000000000002e-06 2024-03-11T21:35:25.099 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:35:25.099 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:35:25.099 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "dispatched", 2024-03-11T21:35:25.100 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:29:08.877397+0000", 2024-03-11T21:35:25.100 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.10132176699999999 2024-03-11T21:35:25.100 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:35:25.100 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:35:25.100 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "queued_for_pg", 2024-03-11T21:35:25.100 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:29:08.877406+0000", 2024-03-11T21:35:25.100 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 9.1779999999999992e-06 2024-03-11T21:35:25.100 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:35:25.100 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:35:25.100 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-03-11T21:35:25.100 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:29:08.877454+0000", 2024-03-11T21:35:25.100 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 4.8087000000000002e-05 2024-03-11T21:35:25.100 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:35:25.100 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:35:25.101 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "started", 2024-03-11T21:35:25.101 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:29:08.877463+0000", 2024-03-11T21:35:25.101 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 8.5350000000000004e-06 2024-03-11T21:35:25.101 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:35:25.101 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:35:25.101 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "done", 2024-03-11T21:35:25.101 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:29:08.877477+0000", 2024-03-11T21:35:25.101 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.3514e-05 2024-03-11T21:35:25.101 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-03-11T21:35:25.101 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-03-11T21:35:25.101 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-03-11T21:35:25.101 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:35:25.101 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:35:25.101 INFO:teuthology.orchestra.run.smithi092.stdout: "description": "osd_op(client.34715.0:116339 7.e 7:7e6296f0:::benchmark_data_smithi092_76040_object116338:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e119)", 2024-03-11T21:35:25.101 INFO:teuthology.orchestra.run.smithi092.stdout: "initiated_at": "2024-03-11T21:29:08.918203+0000", 2024-03-11T21:35:25.102 INFO:teuthology.orchestra.run.smithi092.stdout: "age": 375.52873379900001, 2024-03-11T21:35:25.102 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.084915199999999996, 2024-03-11T21:35:25.102 INFO:teuthology.orchestra.run.smithi092.stdout: "type_data": { 2024-03-11T21:35:25.102 INFO:teuthology.orchestra.run.smithi092.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-11T21:35:25.102 INFO:teuthology.orchestra.run.smithi092.stdout: "client_info": { 2024-03-11T21:35:25.102 INFO:teuthology.orchestra.run.smithi092.stdout: "client": "client.34715", 2024-03-11T21:35:25.102 INFO:teuthology.orchestra.run.smithi092.stdout: "client_addr": "v1:172.21.15.92:0/2403132573", 2024-03-11T21:35:25.102 INFO:teuthology.orchestra.run.smithi092.stdout: "tid": 116339 2024-03-11T21:35:25.102 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:35:25.102 INFO:teuthology.orchestra.run.smithi092.stdout: "events": [ 2024-03-11T21:35:25.102 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:35:25.102 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "initiated", 2024-03-11T21:35:25.102 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:29:08.918203+0000", 2024-03-11T21:35:25.103 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-03-11T21:35:25.103 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:35:25.103 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:35:25.103 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "header_read", 2024-03-11T21:35:25.103 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:29:08.918203+0000", 2024-03-11T21:35:25.103 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-03-11T21:35:25.103 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:35:25.103 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:35:25.103 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "throttled", 2024-03-11T21:35:25.103 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:29:08.918205+0000", 2024-03-11T21:35:25.103 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.933e-06 2024-03-11T21:35:25.103 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:35:25.103 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:35:25.103 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "all_read", 2024-03-11T21:35:25.104 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:29:08.918226+0000", 2024-03-11T21:35:25.104 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 2.1098999999999999e-05 2024-03-11T21:35:25.104 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:35:25.104 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:35:25.104 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "dispatched", 2024-03-11T21:35:25.104 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:29:08.986734+0000", 2024-03-11T21:35:25.104 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.068507799999999994 2024-03-11T21:35:25.104 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:35:25.104 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:35:25.104 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "queued_for_pg", 2024-03-11T21:35:25.104 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:29:08.986752+0000", 2024-03-11T21:35:25.104 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.8315999999999999e-05 2024-03-11T21:35:25.104 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:35:25.104 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:35:25.104 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-03-11T21:35:25.105 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:29:08.986828+0000", 2024-03-11T21:35:25.105 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 7.5916999999999998e-05 2024-03-11T21:35:25.105 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:35:25.105 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:35:25.105 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "started", 2024-03-11T21:35:25.105 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:29:08.986957+0000", 2024-03-11T21:35:25.105 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00012831600000000001 2024-03-11T21:35:25.105 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:35:25.105 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:35:25.105 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "waiting for subops from 0,5", 2024-03-11T21:35:25.105 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:29:08.987118+0000", 2024-03-11T21:35:25.105 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00016082999999999999 2024-03-11T21:35:25.105 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:35:25.105 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:35:25.105 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "sub_op_commit_rec", 2024-03-11T21:35:25.106 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:29:08.993823+0000", 2024-03-11T21:35:25.106 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.0067059010000000002 2024-03-11T21:35:25.106 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:35:25.106 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:35:25.106 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "op_commit", 2024-03-11T21:35:25.106 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:29:08.993877+0000", 2024-03-11T21:35:25.106 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 5.3913999999999999e-05 2024-03-11T21:35:25.106 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:35:25.106 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:35:25.106 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "sub_op_commit_rec", 2024-03-11T21:35:25.106 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:29:09.003018+0000", 2024-03-11T21:35:25.106 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.0091407610000000007 2024-03-11T21:35:25.106 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:35:25.106 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:35:25.106 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "commit_sent", 2024-03-11T21:35:25.107 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:29:09.003074+0000", 2024-03-11T21:35:25.107 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 5.6165000000000002e-05 2024-03-11T21:35:25.107 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:35:25.107 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:35:25.107 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "done", 2024-03-11T21:35:25.107 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:29:09.003119+0000", 2024-03-11T21:35:25.107 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 4.4248000000000001e-05 2024-03-11T21:35:25.107 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-03-11T21:35:25.107 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-03-11T21:35:25.107 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-03-11T21:35:25.107 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:35:25.107 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:35:25.107 INFO:teuthology.orchestra.run.smithi092.stdout: "description": "osd_op(client.34715.0:133972 7.e 7:7ab07efd:::benchmark_data_smithi092_76040_object12061:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e123)", 2024-03-11T21:35:25.107 INFO:teuthology.orchestra.run.smithi092.stdout: "initiated_at": "2024-03-11T21:29:28.264725+0000", 2024-03-11T21:35:25.107 INFO:teuthology.orchestra.run.smithi092.stdout: "age": 356.18221213200002, 2024-03-11T21:35:25.108 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.104608654, 2024-03-11T21:35:25.108 INFO:teuthology.orchestra.run.smithi092.stdout: "type_data": { 2024-03-11T21:35:25.108 INFO:teuthology.orchestra.run.smithi092.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-11T21:35:25.108 INFO:teuthology.orchestra.run.smithi092.stdout: "client_info": { 2024-03-11T21:35:25.108 INFO:teuthology.orchestra.run.smithi092.stdout: "client": "client.34715", 2024-03-11T21:35:25.108 INFO:teuthology.orchestra.run.smithi092.stdout: "client_addr": "v1:172.21.15.92:0/2403132573", 2024-03-11T21:35:25.108 INFO:teuthology.orchestra.run.smithi092.stdout: "tid": 133972 2024-03-11T21:35:25.108 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:35:25.108 INFO:teuthology.orchestra.run.smithi092.stdout: "events": [ 2024-03-11T21:35:25.108 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:35:25.108 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "initiated", 2024-03-11T21:35:25.108 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:29:28.264725+0000", 2024-03-11T21:35:25.108 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-03-11T21:35:25.108 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:35:25.108 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:35:25.109 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "header_read", 2024-03-11T21:35:25.109 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:29:28.264725+0000", 2024-03-11T21:35:25.109 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-03-11T21:35:25.109 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:35:25.109 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:35:25.109 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "throttled", 2024-03-11T21:35:25.109 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:29:28.264728+0000", 2024-03-11T21:35:25.109 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 2.7300000000000001e-06 2024-03-11T21:35:25.109 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:35:25.109 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:35:25.109 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "all_read", 2024-03-11T21:35:25.109 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:29:28.264735+0000", 2024-03-11T21:35:25.109 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 7.4930000000000001e-06 2024-03-11T21:35:25.109 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:35:25.109 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:35:25.110 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "dispatched", 2024-03-11T21:35:25.110 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:29:28.310340+0000", 2024-03-11T21:35:25.110 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.045604599000000003 2024-03-11T21:35:25.110 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:35:25.110 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:35:25.110 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "queued_for_pg", 2024-03-11T21:35:25.110 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:29:28.310368+0000", 2024-03-11T21:35:25.110 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 2.811e-05 2024-03-11T21:35:25.110 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:35:25.110 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:35:25.110 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-03-11T21:35:25.110 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:29:28.310454+0000", 2024-03-11T21:35:25.110 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 8.5987000000000002e-05 2024-03-11T21:35:25.110 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:35:25.111 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:35:25.111 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "started", 2024-03-11T21:35:25.111 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:29:28.310555+0000", 2024-03-11T21:35:25.111 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.000100782 2024-03-11T21:35:25.111 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:35:25.111 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:35:25.111 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "waiting for subops from 0,5", 2024-03-11T21:35:25.111 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:29:28.310638+0000", 2024-03-11T21:35:25.111 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 8.2805000000000004e-05 2024-03-11T21:35:25.111 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:35:25.111 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:35:25.111 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "op_commit", 2024-03-11T21:35:25.111 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:29:28.313111+0000", 2024-03-11T21:35:25.111 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.0024739150000000001 2024-03-11T21:35:25.111 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:35:25.111 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:35:25.112 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "sub_op_commit_rec", 2024-03-11T21:35:25.112 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:29:28.313382+0000", 2024-03-11T21:35:25.112 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.000270492 2024-03-11T21:35:25.112 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:35:25.112 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:35:25.112 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "sub_op_commit_rec", 2024-03-11T21:35:25.112 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:29:28.369094+0000", 2024-03-11T21:35:25.112 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.055711954000000001 2024-03-11T21:35:25.112 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:35:25.112 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:35:25.112 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "commit_sent", 2024-03-11T21:35:25.112 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:29:28.369144+0000", 2024-03-11T21:35:25.112 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 4.9709000000000001e-05 2024-03-11T21:35:25.112 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:35:25.112 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:35:25.113 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "done", 2024-03-11T21:35:25.113 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:29:28.369334+0000", 2024-03-11T21:35:25.113 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00019007799999999999 2024-03-11T21:35:25.113 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-03-11T21:35:25.113 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-03-11T21:35:25.113 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-03-11T21:35:25.113 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:35:25.113 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:35:25.113 INFO:teuthology.orchestra.run.smithi092.stdout: "description": "osd_op(client.34715.0:142044 7.e 7:798f10b5:::benchmark_data_smithi092_76040_object20133:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e124)", 2024-03-11T21:35:25.113 INFO:teuthology.orchestra.run.smithi092.stdout: "initiated_at": "2024-03-11T21:29:37.013602+0000", 2024-03-11T21:35:25.113 INFO:teuthology.orchestra.run.smithi092.stdout: "age": 347.433335021, 2024-03-11T21:35:25.113 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.084341611999999996, 2024-03-11T21:35:25.113 INFO:teuthology.orchestra.run.smithi092.stdout: "type_data": { 2024-03-11T21:35:25.113 INFO:teuthology.orchestra.run.smithi092.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-11T21:35:25.113 INFO:teuthology.orchestra.run.smithi092.stdout: "client_info": { 2024-03-11T21:35:25.114 INFO:teuthology.orchestra.run.smithi092.stdout: "client": "client.34715", 2024-03-11T21:35:25.114 INFO:teuthology.orchestra.run.smithi092.stdout: "client_addr": "v1:172.21.15.92:0/2403132573", 2024-03-11T21:35:25.114 INFO:teuthology.orchestra.run.smithi092.stdout: "tid": 142044 2024-03-11T21:35:25.114 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:35:25.114 INFO:teuthology.orchestra.run.smithi092.stdout: "events": [ 2024-03-11T21:35:25.114 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:35:25.114 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "initiated", 2024-03-11T21:35:25.114 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:29:37.013602+0000", 2024-03-11T21:35:25.114 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-03-11T21:35:25.114 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:35:25.114 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:35:25.114 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "header_read", 2024-03-11T21:35:25.114 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:29:37.013602+0000", 2024-03-11T21:35:25.114 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-03-11T21:35:25.115 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:35:25.115 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:35:25.115 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "throttled", 2024-03-11T21:35:25.115 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:29:37.013604+0000", 2024-03-11T21:35:25.115 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 2.3099999999999999e-06 2024-03-11T21:35:25.115 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:35:25.115 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:35:25.115 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "all_read", 2024-03-11T21:35:25.115 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:29:37.013611+0000", 2024-03-11T21:35:25.115 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 7.0500000000000003e-06 2024-03-11T21:35:25.115 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:35:25.115 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:35:25.115 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "dispatched", 2024-03-11T21:35:25.115 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:29:37.013624+0000", 2024-03-11T21:35:25.115 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.2858e-05 2024-03-11T21:35:25.116 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:35:25.116 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:35:25.116 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "queued_for_pg", 2024-03-11T21:35:25.116 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:29:37.013632+0000", 2024-03-11T21:35:25.116 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 7.9689999999999994e-06 2024-03-11T21:35:25.116 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:35:25.116 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:35:25.116 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-03-11T21:35:25.116 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:29:37.013690+0000", 2024-03-11T21:35:25.116 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 5.7472000000000001e-05 2024-03-11T21:35:25.116 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:35:25.116 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:35:25.116 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "started", 2024-03-11T21:35:25.116 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:29:37.013804+0000", 2024-03-11T21:35:25.117 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.000114225 2024-03-11T21:35:25.117 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:35:25.117 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:35:25.117 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "waiting for subops from 0,5", 2024-03-11T21:35:25.117 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:29:37.013905+0000", 2024-03-11T21:35:25.117 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.000101056 2024-03-11T21:35:25.117 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:35:25.117 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:35:25.117 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "op_commit", 2024-03-11T21:35:25.117 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:29:37.017257+0000", 2024-03-11T21:35:25.117 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.0033514679999999998 2024-03-11T21:35:25.117 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:35:25.118 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:35:25.118 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "sub_op_commit_rec", 2024-03-11T21:35:25.118 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:29:37.019214+0000", 2024-03-11T21:35:25.118 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.001957828 2024-03-11T21:35:25.118 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:35:25.118 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:35:25.118 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "sub_op_commit_rec", 2024-03-11T21:35:25.118 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:29:37.097839+0000", 2024-03-11T21:35:25.118 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.078624227000000005 2024-03-11T21:35:25.118 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:35:25.118 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:35:25.118 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "commit_sent", 2024-03-11T21:35:25.118 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:29:37.097895+0000", 2024-03-11T21:35:25.118 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 5.6149000000000001e-05 2024-03-11T21:35:25.119 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:35:25.119 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:35:25.119 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "done", 2024-03-11T21:35:25.119 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:29:37.097944+0000", 2024-03-11T21:35:25.119 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 4.8999999999999998e-05 2024-03-11T21:35:25.119 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-03-11T21:35:25.119 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-03-11T21:35:25.119 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-03-11T21:35:25.119 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:35:25.119 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:35:25.119 INFO:teuthology.orchestra.run.smithi092.stdout: "description": "osd_op(client.34715.0:157136 7.e 7:745b6815:::benchmark_data_smithi092_76040_object35225:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e127)", 2024-03-11T21:35:25.119 INFO:teuthology.orchestra.run.smithi092.stdout: "initiated_at": "2024-03-11T21:29:58.183594+0000", 2024-03-11T21:35:25.119 INFO:teuthology.orchestra.run.smithi092.stdout: "age": 326.26334364000002, 2024-03-11T21:35:25.119 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.095394920999999994, 2024-03-11T21:35:25.120 INFO:teuthology.orchestra.run.smithi092.stdout: "type_data": { 2024-03-11T21:35:25.120 INFO:teuthology.orchestra.run.smithi092.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-11T21:35:25.120 INFO:teuthology.orchestra.run.smithi092.stdout: "client_info": { 2024-03-11T21:35:25.120 INFO:teuthology.orchestra.run.smithi092.stdout: "client": "client.34715", 2024-03-11T21:35:25.120 INFO:teuthology.orchestra.run.smithi092.stdout: "client_addr": "v1:172.21.15.92:0/2403132573", 2024-03-11T21:35:25.120 INFO:teuthology.orchestra.run.smithi092.stdout: "tid": 157136 2024-03-11T21:35:25.120 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:35:25.120 INFO:teuthology.orchestra.run.smithi092.stdout: "events": [ 2024-03-11T21:35:25.120 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:35:25.120 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "initiated", 2024-03-11T21:35:25.120 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:29:58.183594+0000", 2024-03-11T21:35:25.120 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-03-11T21:35:25.120 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:35:25.121 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:35:25.121 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "header_read", 2024-03-11T21:35:25.121 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:29:58.183594+0000", 2024-03-11T21:35:25.121 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-03-11T21:35:25.121 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:35:25.121 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:35:25.121 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "throttled", 2024-03-11T21:35:25.121 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:29:58.183596+0000", 2024-03-11T21:35:25.121 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 2.4150000000000002e-06 2024-03-11T21:35:25.121 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:35:25.121 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:35:25.121 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "all_read", 2024-03-11T21:35:25.121 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:29:58.183604+0000", 2024-03-11T21:35:25.121 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 8.1950000000000003e-06 2024-03-11T21:35:25.121 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:35:25.121 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:35:25.122 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "dispatched", 2024-03-11T21:35:25.122 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:29:58.183622+0000", 2024-03-11T21:35:25.122 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.7969e-05 2024-03-11T21:35:25.122 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:35:25.122 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:35:25.122 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "queued_for_pg", 2024-03-11T21:35:25.122 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:29:58.183632+0000", 2024-03-11T21:35:25.122 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 9.7499999999999998e-06 2024-03-11T21:35:25.122 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:35:25.122 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:35:25.122 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-03-11T21:35:25.122 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:29:58.183813+0000", 2024-03-11T21:35:25.122 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00018102099999999999 2024-03-11T21:35:25.122 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:35:25.123 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:35:25.123 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "started", 2024-03-11T21:35:25.123 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:29:58.184140+0000", 2024-03-11T21:35:25.123 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00032760200000000003 2024-03-11T21:35:25.123 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:35:25.123 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:35:25.123 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "waiting for subops from 0,5", 2024-03-11T21:35:25.123 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:29:58.184404+0000", 2024-03-11T21:35:25.123 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00026328300000000001 2024-03-11T21:35:25.123 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:35:25.123 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:35:25.123 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "op_commit", 2024-03-11T21:35:25.123 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:29:58.191221+0000", 2024-03-11T21:35:25.123 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.0068170799999999997 2024-03-11T21:35:25.123 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:35:25.124 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:35:25.124 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "sub_op_commit_rec", 2024-03-11T21:35:25.124 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:29:58.191474+0000", 2024-03-11T21:35:25.124 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00025292199999999999 2024-03-11T21:35:25.124 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:35:25.124 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:35:25.124 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "sub_op_commit_rec", 2024-03-11T21:35:25.124 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:29:58.278663+0000", 2024-03-11T21:35:25.124 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.087188838000000005 2024-03-11T21:35:25.124 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:35:25.124 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:35:25.124 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "commit_sent", 2024-03-11T21:35:25.124 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:29:58.278856+0000", 2024-03-11T21:35:25.124 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00019299399999999999 2024-03-11T21:35:25.125 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:35:25.125 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:35:25.125 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "done", 2024-03-11T21:35:25.125 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:29:58.278988+0000", 2024-03-11T21:35:25.125 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00013285200000000001 2024-03-11T21:35:25.125 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-03-11T21:35:25.125 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-03-11T21:35:25.125 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-03-11T21:35:25.125 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:35:25.125 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:35:25.125 INFO:teuthology.orchestra.run.smithi092.stdout: "description": "osd_repop(client.15150.0:49973 8.4 e169/157 8:2d3557fd:::benchmark_data_smithi092_104865_object49972:head v 169'3101, mlcod=169'3099)", 2024-03-11T21:35:25.125 INFO:teuthology.orchestra.run.smithi092.stdout: "initiated_at": "2024-03-11T21:32:26.043596+0000", 2024-03-11T21:35:25.125 INFO:teuthology.orchestra.run.smithi092.stdout: "age": 178.40334127899999, 2024-03-11T21:35:25.125 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.098687574, 2024-03-11T21:35:25.126 INFO:teuthology.orchestra.run.smithi092.stdout: "type_data": { 2024-03-11T21:35:25.126 INFO:teuthology.orchestra.run.smithi092.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-11T21:35:25.126 INFO:teuthology.orchestra.run.smithi092.stdout: "events": [ 2024-03-11T21:35:25.126 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:35:25.126 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "initiated", 2024-03-11T21:35:25.126 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:32:26.043596+0000", 2024-03-11T21:35:25.126 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-03-11T21:35:25.126 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:35:25.126 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:35:25.126 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "header_read", 2024-03-11T21:35:25.126 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:32:26.043596+0000", 2024-03-11T21:35:25.126 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-03-11T21:35:25.126 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:35:25.126 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:35:25.126 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "throttled", 2024-03-11T21:35:25.127 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:32:26.043597+0000", 2024-03-11T21:35:25.127 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.5400000000000001e-06 2024-03-11T21:35:25.127 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:35:25.127 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:35:25.127 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "all_read", 2024-03-11T21:35:25.127 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:32:26.043713+0000", 2024-03-11T21:35:25.127 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.000115884 2024-03-11T21:35:25.127 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:35:25.127 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:35:25.127 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "dispatched", 2024-03-11T21:35:25.127 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:32:26.043722+0000", 2024-03-11T21:35:25.127 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 9.0920000000000004e-06 2024-03-11T21:35:25.127 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:35:25.127 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:35:25.128 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "queued_for_pg", 2024-03-11T21:35:25.128 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:32:26.043733+0000", 2024-03-11T21:35:25.128 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.0142e-05 2024-03-11T21:35:25.128 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:35:25.128 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:35:25.128 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-03-11T21:35:25.128 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:32:26.043784+0000", 2024-03-11T21:35:25.128 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 5.1076999999999999e-05 2024-03-11T21:35:25.128 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:35:25.128 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:35:25.128 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "started", 2024-03-11T21:35:25.128 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:32:26.140300+0000", 2024-03-11T21:35:25.128 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.096516770000000002 2024-03-11T21:35:25.128 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:35:25.128 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:35:25.129 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "commit_sent", 2024-03-11T21:35:25.129 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:32:26.142238+0000", 2024-03-11T21:35:25.129 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.0019378360000000001 2024-03-11T21:35:25.129 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:35:25.129 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:35:25.129 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "done", 2024-03-11T21:35:25.129 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:32:26.142283+0000", 2024-03-11T21:35:25.129 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 4.5232999999999999e-05 2024-03-11T21:35:25.129 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-03-11T21:35:25.129 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-03-11T21:35:25.129 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-03-11T21:35:25.129 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:35:25.129 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:35:25.129 INFO:teuthology.orchestra.run.smithi092.stdout: "description": "osd_repop(client.15150.0:49958 8.e e169/157 8:77e6f24a:::benchmark_data_smithi092_104865_object49957:head v 169'3228, mlcod=169'3226)", 2024-03-11T21:35:25.130 INFO:teuthology.orchestra.run.smithi092.stdout: "initiated_at": "2024-03-11T21:32:26.047074+0000", 2024-03-11T21:35:25.130 INFO:teuthology.orchestra.run.smithi092.stdout: "age": 178.399863057, 2024-03-11T21:35:25.130 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.097612361999999994, 2024-03-11T21:35:25.130 INFO:teuthology.orchestra.run.smithi092.stdout: "type_data": { 2024-03-11T21:35:25.130 INFO:teuthology.orchestra.run.smithi092.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-11T21:35:25.130 INFO:teuthology.orchestra.run.smithi092.stdout: "events": [ 2024-03-11T21:35:25.130 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:35:25.130 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "initiated", 2024-03-11T21:35:25.130 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:32:26.047074+0000", 2024-03-11T21:35:25.130 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-03-11T21:35:25.130 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:35:25.130 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:35:25.130 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "header_read", 2024-03-11T21:35:25.130 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:32:26.047074+0000", 2024-03-11T21:35:25.131 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-03-11T21:35:25.131 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:35:25.131 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:35:25.131 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "throttled", 2024-03-11T21:35:25.131 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:32:26.047076+0000", 2024-03-11T21:35:25.131 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 2.1270000000000001e-06 2024-03-11T21:35:25.131 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:35:25.131 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:35:25.131 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "all_read", 2024-03-11T21:35:25.131 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:32:26.047128+0000", 2024-03-11T21:35:25.131 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 5.1696000000000001e-05 2024-03-11T21:35:25.131 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:35:25.131 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:35:25.131 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "dispatched", 2024-03-11T21:35:25.132 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:32:26.140329+0000", 2024-03-11T21:35:25.132 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.093201488999999998 2024-03-11T21:35:25.132 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:35:25.132 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:35:25.132 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "queued_for_pg", 2024-03-11T21:35:25.132 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:32:26.140337+0000", 2024-03-11T21:35:25.132 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 8.0900000000000005e-06 2024-03-11T21:35:25.132 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:35:25.132 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:35:25.132 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-03-11T21:35:25.132 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:32:26.140452+0000", 2024-03-11T21:35:25.132 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.000114948 2024-03-11T21:35:25.132 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:35:25.132 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:35:25.132 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "started", 2024-03-11T21:35:25.133 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:32:26.140473+0000", 2024-03-11T21:35:25.133 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 2.048e-05 2024-03-11T21:35:25.133 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:35:25.133 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:35:25.133 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "commit_sent", 2024-03-11T21:35:25.133 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:32:26.144223+0000", 2024-03-11T21:35:25.133 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.0037504069999999999 2024-03-11T21:35:25.133 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:35:25.133 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:35:25.133 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "done", 2024-03-11T21:35:25.133 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:32:26.144686+0000", 2024-03-11T21:35:25.133 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.000463125 2024-03-11T21:35:25.133 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-03-11T21:35:25.133 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-03-11T21:35:25.133 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-03-11T21:35:25.134 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:35:25.134 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:35:25.134 INFO:teuthology.orchestra.run.smithi092.stdout: "description": "osd_repop(client.15150.0:49989 8.1 e169/157 8:87dddf73:::benchmark_data_smithi092_104865_object49988:head v 169'3117, mlcod=169'3115)", 2024-03-11T21:35:25.134 INFO:teuthology.orchestra.run.smithi092.stdout: "initiated_at": "2024-03-11T21:32:26.052509+0000", 2024-03-11T21:35:25.134 INFO:teuthology.orchestra.run.smithi092.stdout: "age": 178.39442821200001, 2024-03-11T21:35:25.134 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.090394098000000006, 2024-03-11T21:35:25.134 INFO:teuthology.orchestra.run.smithi092.stdout: "type_data": { 2024-03-11T21:35:25.134 INFO:teuthology.orchestra.run.smithi092.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-11T21:35:25.134 INFO:teuthology.orchestra.run.smithi092.stdout: "events": [ 2024-03-11T21:35:25.134 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:35:25.134 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "initiated", 2024-03-11T21:35:25.134 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:32:26.052509+0000", 2024-03-11T21:35:25.134 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-03-11T21:35:25.134 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:35:25.134 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:35:25.135 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "header_read", 2024-03-11T21:35:25.135 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:32:26.052509+0000", 2024-03-11T21:35:25.135 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-03-11T21:35:25.135 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:35:25.135 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:35:25.135 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "throttled", 2024-03-11T21:35:25.135 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:32:26.052511+0000", 2024-03-11T21:35:25.135 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 2.2000000000000001e-06 2024-03-11T21:35:25.135 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:35:25.135 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:35:25.135 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "all_read", 2024-03-11T21:35:25.135 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:32:26.052543+0000", 2024-03-11T21:35:25.135 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 3.1819000000000001e-05 2024-03-11T21:35:25.135 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:35:25.136 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:35:25.136 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "dispatched", 2024-03-11T21:35:25.136 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:32:26.140301+0000", 2024-03-11T21:35:25.136 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.087758117999999996 2024-03-11T21:35:25.136 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:35:25.136 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:35:25.136 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "queued_for_pg", 2024-03-11T21:35:25.136 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:32:26.140314+0000", 2024-03-11T21:35:25.136 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.2847000000000001e-05 2024-03-11T21:35:25.136 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:35:25.136 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:35:25.136 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-03-11T21:35:25.136 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:32:26.140379+0000", 2024-03-11T21:35:25.136 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 6.5271000000000006e-05 2024-03-11T21:35:25.136 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:35:25.137 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:35:25.137 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "started", 2024-03-11T21:35:25.137 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:32:26.140403+0000", 2024-03-11T21:35:25.137 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 2.4298e-05 2024-03-11T21:35:25.137 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:35:25.137 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:35:25.137 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "commit_sent", 2024-03-11T21:35:25.137 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:32:26.142841+0000", 2024-03-11T21:35:25.137 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.0024378590000000001 2024-03-11T21:35:25.137 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:35:25.137 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:35:25.137 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "done", 2024-03-11T21:35:25.137 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:32:26.142903+0000", 2024-03-11T21:35:25.137 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 6.1686000000000005e-05 2024-03-11T21:35:25.137 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-03-11T21:35:25.138 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-03-11T21:35:25.138 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-03-11T21:35:25.138 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:35:25.138 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:35:25.138 INFO:teuthology.orchestra.run.smithi092.stdout: "description": "osd_repop(client.15150.0:49999 8.f e169/157 8:fff2a1a0:::benchmark_data_smithi092_104865_object49998:head v 169'3086, mlcod=169'3084)", 2024-03-11T21:35:25.138 INFO:teuthology.orchestra.run.smithi092.stdout: "initiated_at": "2024-03-11T21:32:26.058512+0000", 2024-03-11T21:35:25.138 INFO:teuthology.orchestra.run.smithi092.stdout: "age": 178.388425154, 2024-03-11T21:35:25.138 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.084917562000000002, 2024-03-11T21:35:25.138 INFO:teuthology.orchestra.run.smithi092.stdout: "type_data": { 2024-03-11T21:35:25.138 INFO:teuthology.orchestra.run.smithi092.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-11T21:35:25.138 INFO:teuthology.orchestra.run.smithi092.stdout: "events": [ 2024-03-11T21:35:25.138 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:35:25.138 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "initiated", 2024-03-11T21:35:25.138 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:32:26.058512+0000", 2024-03-11T21:35:25.139 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-03-11T21:35:25.139 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:35:25.139 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:35:25.139 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "header_read", 2024-03-11T21:35:25.139 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:32:26.058512+0000", 2024-03-11T21:35:25.139 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-03-11T21:35:25.139 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:35:25.139 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:35:25.139 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "throttled", 2024-03-11T21:35:25.139 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:32:26.058514+0000", 2024-03-11T21:35:25.139 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 2.464e-06 2024-03-11T21:35:25.139 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:35:25.139 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:35:25.139 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "all_read", 2024-03-11T21:35:25.139 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:32:26.058540+0000", 2024-03-11T21:35:25.140 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 2.5712999999999999e-05 2024-03-11T21:35:25.140 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:35:25.140 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:35:25.140 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "dispatched", 2024-03-11T21:35:25.140 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:32:26.140327+0000", 2024-03-11T21:35:25.140 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.081786403999999993 2024-03-11T21:35:25.140 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:35:25.140 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:35:25.140 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "queued_for_pg", 2024-03-11T21:35:25.140 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:32:26.140337+0000", 2024-03-11T21:35:25.140 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.0838e-05 2024-03-11T21:35:25.140 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:35:25.140 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:35:25.140 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-03-11T21:35:25.141 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:32:26.140393+0000", 2024-03-11T21:35:25.141 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 5.5365000000000003e-05 2024-03-11T21:35:25.141 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:35:25.141 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:35:25.141 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "started", 2024-03-11T21:35:25.141 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:32:26.140413+0000", 2024-03-11T21:35:25.141 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 2.0613000000000001e-05 2024-03-11T21:35:25.141 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:35:25.141 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:35:25.141 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "commit_sent", 2024-03-11T21:35:25.141 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:32:26.143313+0000", 2024-03-11T21:35:25.141 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.002899263 2024-03-11T21:35:25.141 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:35:25.141 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:35:25.141 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "done", 2024-03-11T21:35:25.142 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:32:26.143430+0000", 2024-03-11T21:35:25.142 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.000116902 2024-03-11T21:35:25.142 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-03-11T21:35:25.142 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-03-11T21:35:25.142 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-03-11T21:35:25.142 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:35:25.142 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:35:25.142 INFO:teuthology.orchestra.run.smithi092.stdout: "description": "osd_repop(client.15150.0:88132 8.4 e180/157 8:2e4eda56:::benchmark_data_smithi092_104865_object88131:head v 180'5441, mlcod=180'5439)", 2024-03-11T21:35:25.142 INFO:teuthology.orchestra.run.smithi092.stdout: "initiated_at": "2024-03-11T21:32:56.039658+0000", 2024-03-11T21:35:25.142 INFO:teuthology.orchestra.run.smithi092.stdout: "age": 148.40727955, 2024-03-11T21:35:25.142 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.114835889, 2024-03-11T21:35:25.142 INFO:teuthology.orchestra.run.smithi092.stdout: "type_data": { 2024-03-11T21:35:25.142 INFO:teuthology.orchestra.run.smithi092.stdout: "flag_point": "started", 2024-03-11T21:35:25.142 INFO:teuthology.orchestra.run.smithi092.stdout: "events": [ 2024-03-11T21:35:25.143 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:35:25.143 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "initiated", 2024-03-11T21:35:25.143 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:32:56.039658+0000", 2024-03-11T21:35:25.143 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-03-11T21:35:25.143 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:35:25.143 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:35:25.143 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "header_read", 2024-03-11T21:35:25.143 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:32:56.039658+0000", 2024-03-11T21:35:25.143 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-03-11T21:35:25.143 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:35:25.143 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:35:25.143 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "throttled", 2024-03-11T21:35:25.143 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:32:56.039659+0000", 2024-03-11T21:35:25.143 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.514e-06 2024-03-11T21:35:25.144 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:35:25.144 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:35:25.144 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "all_read", 2024-03-11T21:35:25.144 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:32:56.039737+0000", 2024-03-11T21:35:25.144 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 7.8158000000000006e-05 2024-03-11T21:35:25.144 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:35:25.144 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:35:25.144 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "dispatched", 2024-03-11T21:35:25.144 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:32:56.039752+0000", 2024-03-11T21:35:25.144 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.4776999999999999e-05 2024-03-11T21:35:25.144 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:35:25.144 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:35:25.144 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "queued_for_pg", 2024-03-11T21:35:25.144 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:32:56.039761+0000", 2024-03-11T21:35:25.144 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 8.5639999999999997e-06 2024-03-11T21:35:25.145 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:35:25.145 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:35:25.145 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-03-11T21:35:25.145 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:32:56.039833+0000", 2024-03-11T21:35:25.145 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 7.2583999999999995e-05 2024-03-11T21:35:25.145 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:35:25.145 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:35:25.145 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "started", 2024-03-11T21:35:25.145 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:32:56.039863+0000", 2024-03-11T21:35:25.145 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 2.9757e-05 2024-03-11T21:35:25.145 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:35:25.145 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:35:25.145 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "done", 2024-03-11T21:35:25.145 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:32:56.154493+0000", 2024-03-11T21:35:25.146 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.11463053500000001 2024-03-11T21:35:25.146 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-03-11T21:35:25.146 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-03-11T21:35:25.146 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-03-11T21:35:25.146 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:35:25.146 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:35:25.146 INFO:teuthology.orchestra.run.smithi092.stdout: "description": "osd_repop(client.15150.0:88135 8.1 e180/157 8:8d497fa8:::benchmark_data_smithi092_104865_object88134:head v 180'5484, mlcod=180'5482)", 2024-03-11T21:35:25.146 INFO:teuthology.orchestra.run.smithi092.stdout: "initiated_at": "2024-03-11T21:32:56.043671+0000", 2024-03-11T21:35:25.146 INFO:teuthology.orchestra.run.smithi092.stdout: "age": 148.403266196, 2024-03-11T21:35:25.146 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.11037298399999999, 2024-03-11T21:35:25.146 INFO:teuthology.orchestra.run.smithi092.stdout: "type_data": { 2024-03-11T21:35:25.146 INFO:teuthology.orchestra.run.smithi092.stdout: "flag_point": "started", 2024-03-11T21:35:25.146 INFO:teuthology.orchestra.run.smithi092.stdout: "events": [ 2024-03-11T21:35:25.146 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:35:25.147 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "initiated", 2024-03-11T21:35:25.147 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:32:56.043671+0000", 2024-03-11T21:35:25.147 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-03-11T21:35:25.147 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:35:25.147 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:35:25.147 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "header_read", 2024-03-11T21:35:25.147 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:32:56.043671+0000", 2024-03-11T21:35:25.147 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-03-11T21:35:25.147 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:35:25.147 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:35:25.147 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "throttled", 2024-03-11T21:35:25.147 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:32:56.043673+0000", 2024-03-11T21:35:25.147 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.7379999999999999e-06 2024-03-11T21:35:25.147 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:35:25.147 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:35:25.148 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "all_read", 2024-03-11T21:35:25.148 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:32:56.043695+0000", 2024-03-11T21:35:25.148 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 2.2691e-05 2024-03-11T21:35:25.148 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:35:25.148 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:35:25.148 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "dispatched", 2024-03-11T21:35:25.148 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:32:56.043711+0000", 2024-03-11T21:35:25.148 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.5197999999999999e-05 2024-03-11T21:35:25.148 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:35:25.148 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:35:25.148 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "queued_for_pg", 2024-03-11T21:35:25.148 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:32:56.043718+0000", 2024-03-11T21:35:25.148 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 7.6939999999999995e-06 2024-03-11T21:35:25.148 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:35:25.149 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:35:25.149 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-03-11T21:35:25.149 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:32:56.043876+0000", 2024-03-11T21:35:25.149 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.000157534 2024-03-11T21:35:25.149 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:35:25.149 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:35:25.149 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "started", 2024-03-11T21:35:25.149 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:32:56.043921+0000", 2024-03-11T21:35:25.149 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 4.5349999999999998e-05 2024-03-11T21:35:25.149 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:35:25.149 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:35:25.149 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "done", 2024-03-11T21:35:25.149 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:32:56.154044+0000", 2024-03-11T21:35:25.149 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.110122779 2024-03-11T21:35:25.150 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-03-11T21:35:25.150 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-03-11T21:35:25.150 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-03-11T21:35:25.150 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:35:25.150 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:35:25.150 INFO:teuthology.orchestra.run.smithi092.stdout: "description": "osd_repop(client.15150.0:88139 8.4 e180/157)", 2024-03-11T21:35:25.150 INFO:teuthology.orchestra.run.smithi092.stdout: "initiated_at": "2024-03-11T21:32:56.046134+0000", 2024-03-11T21:35:25.150 INFO:teuthology.orchestra.run.smithi092.stdout: "age": 148.400803295, 2024-03-11T21:35:25.150 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.109597756, 2024-03-11T21:35:25.150 INFO:teuthology.orchestra.run.smithi092.stdout: "type_data": { 2024-03-11T21:35:25.150 INFO:teuthology.orchestra.run.smithi092.stdout: "flag_point": "reached pg", 2024-03-11T21:35:25.150 INFO:teuthology.orchestra.run.smithi092.stdout: "events": [ 2024-03-11T21:35:25.150 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:35:25.150 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "initiated", 2024-03-11T21:35:25.151 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:32:56.046134+0000", 2024-03-11T21:35:25.151 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-03-11T21:35:25.151 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:35:25.151 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:35:25.151 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "header_read", 2024-03-11T21:35:25.151 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:32:56.046134+0000", 2024-03-11T21:35:25.151 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-03-11T21:35:25.151 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:35:25.151 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:35:25.151 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "throttled", 2024-03-11T21:35:25.151 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:32:56.046136+0000", 2024-03-11T21:35:25.151 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.7e-06 2024-03-11T21:35:25.151 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:35:25.151 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:35:25.152 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "all_read", 2024-03-11T21:35:25.152 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:32:56.046173+0000", 2024-03-11T21:35:25.152 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 3.7635999999999997e-05 2024-03-11T21:35:25.152 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:35:25.152 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:35:25.152 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "dispatched", 2024-03-11T21:35:25.152 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:32:56.046186+0000", 2024-03-11T21:35:25.152 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.3164e-05 2024-03-11T21:35:25.152 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:35:25.152 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:35:25.152 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "queued_for_pg", 2024-03-11T21:35:25.152 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:32:56.046194+0000", 2024-03-11T21:35:25.152 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 7.4100000000000002e-06 2024-03-11T21:35:25.152 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:35:25.152 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:35:25.153 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-03-11T21:35:25.153 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:32:56.155713+0000", 2024-03-11T21:35:25.153 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.109519045 2024-03-11T21:35:25.153 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:35:25.153 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:35:25.153 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "done", 2024-03-11T21:35:25.153 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:32:56.155732+0000", 2024-03-11T21:35:25.153 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.8800999999999998e-05 2024-03-11T21:35:25.153 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-03-11T21:35:25.153 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-03-11T21:35:25.153 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-03-11T21:35:25.153 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:35:25.153 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:35:25.153 INFO:teuthology.orchestra.run.smithi092.stdout: "description": "osd_repop(client.15150.0:88143 8.1 e180/157)", 2024-03-11T21:35:25.153 INFO:teuthology.orchestra.run.smithi092.stdout: "initiated_at": "2024-03-11T21:32:56.059118+0000", 2024-03-11T21:35:25.154 INFO:teuthology.orchestra.run.smithi092.stdout: "age": 148.3878191, 2024-03-11T21:35:25.154 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.095333888000000006, 2024-03-11T21:35:25.154 INFO:teuthology.orchestra.run.smithi092.stdout: "type_data": { 2024-03-11T21:35:25.154 INFO:teuthology.orchestra.run.smithi092.stdout: "flag_point": "reached pg", 2024-03-11T21:35:25.154 INFO:teuthology.orchestra.run.smithi092.stdout: "events": [ 2024-03-11T21:35:25.154 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:35:25.154 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "initiated", 2024-03-11T21:35:25.154 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:32:56.059118+0000", 2024-03-11T21:35:25.154 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-03-11T21:35:25.154 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:35:25.154 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:35:25.154 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "header_read", 2024-03-11T21:35:25.154 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:32:56.059118+0000", 2024-03-11T21:35:25.154 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-03-11T21:35:25.154 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:35:25.155 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:35:25.155 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "throttled", 2024-03-11T21:35:25.155 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:32:56.059120+0000", 2024-03-11T21:35:25.155 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 2.2330000000000001e-06 2024-03-11T21:35:25.155 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:35:25.155 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:35:25.155 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "all_read", 2024-03-11T21:35:25.155 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:32:56.059148+0000", 2024-03-11T21:35:25.155 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 2.7450999999999999e-05 2024-03-11T21:35:25.155 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:35:25.155 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:35:25.155 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "dispatched", 2024-03-11T21:35:25.155 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:32:56.059161+0000", 2024-03-11T21:35:25.155 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.3338e-05 2024-03-11T21:35:25.156 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:35:25.156 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:35:25.156 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "queued_for_pg", 2024-03-11T21:35:25.156 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:32:56.059169+0000", 2024-03-11T21:35:25.156 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 8.2649999999999996e-06 2024-03-11T21:35:25.156 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:35:25.156 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:35:25.156 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-03-11T21:35:25.156 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:32:56.154430+0000", 2024-03-11T21:35:25.156 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.095260963000000004 2024-03-11T21:35:25.156 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:35:25.156 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:35:25.156 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "done", 2024-03-11T21:35:25.156 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:32:56.154452+0000", 2024-03-11T21:35:25.156 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 2.1637999999999998e-05 2024-03-11T21:35:25.157 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-03-11T21:35:25.157 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-03-11T21:35:25.157 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-03-11T21:35:25.157 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-03-11T21:35:25.157 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-03-11T21:35:25.157 INFO:teuthology.orchestra.run.smithi092.stdout:} 2024-03-11T21:35:25.157 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:35:25.187 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:35:25.188 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:35:25.188 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:35:25.289 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:35:25.328 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:35:25.329 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:35:25.329 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:35:25.430 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:35:25.458 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:35:25.459 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:35:25.460 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:35:25.561 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:35:25.591 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:35:25.591 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:35:25.592 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:35:25.693 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:35:25.733 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:35:25.733 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:35:25.734 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:35:25.806 INFO:tasks.radosbench.radosbench.0.smithi092.stdout:Removed 120498 objects 2024-03-11T21:35:25.806 INFO:tasks.radosbench.radosbench.0.smithi092.stdout:Clean up completed and total clean up time :113.359 2024-03-11T21:35:25.825 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:35:25 smithi097 ceph-mon[33317]: 6.a scrub starts 2024-03-11T21:35:25.825 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:35:25 smithi097 ceph-mon[33317]: 6.a scrub ok 2024-03-11T21:35:25.826 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:35:25 smithi097 ceph-mon[33317]: 5.4 scrub starts 2024-03-11T21:35:25.826 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:35:25 smithi097 ceph-mon[33317]: 5.4 scrub ok 2024-03-11T21:35:25.826 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:35:25 smithi097 ceph-mon[33317]: 8.b scrub starts 2024-03-11T21:35:25.826 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:35:25 smithi097 ceph-mon[33317]: 8.b scrub ok 2024-03-11T21:35:25.826 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:35:25 smithi097 ceph-mon[33317]: purged_snaps scrub starts 2024-03-11T21:35:25.826 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:35:25 smithi097 ceph-mon[33317]: purged_snaps scrub ok 2024-03-11T21:35:25.834 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:35:25.857 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:35:25.858 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:35:25.858 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:35:25.859 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:35:25 smithi092 ceph-mon[37563]: 6.a scrub starts 2024-03-11T21:35:25.859 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:35:25 smithi092 ceph-mon[37563]: 6.a scrub ok 2024-03-11T21:35:25.860 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:35:25 smithi092 ceph-mon[37563]: 5.4 scrub starts 2024-03-11T21:35:25.860 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:35:25 smithi092 ceph-mon[37563]: 5.4 scrub ok 2024-03-11T21:35:25.860 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:35:25 smithi092 ceph-mon[37563]: 8.b scrub starts 2024-03-11T21:35:25.860 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:35:25 smithi092 ceph-mon[37563]: 8.b scrub ok 2024-03-11T21:35:25.860 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:35:25 smithi092 ceph-mon[37563]: purged_snaps scrub starts 2024-03-11T21:35:25.860 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:35:25 smithi092 ceph-mon[37563]: purged_snaps scrub ok 2024-03-11T21:35:25.860 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:35:25 smithi092 ceph-mon[29572]: 6.a scrub starts 2024-03-11T21:35:25.860 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:35:25 smithi092 ceph-mon[29572]: 6.a scrub ok 2024-03-11T21:35:25.860 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:35:25 smithi092 ceph-mon[29572]: 5.4 scrub starts 2024-03-11T21:35:25.860 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:35:25 smithi092 ceph-mon[29572]: 5.4 scrub ok 2024-03-11T21:35:25.860 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:35:25 smithi092 ceph-mon[29572]: 8.b scrub starts 2024-03-11T21:35:25.860 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:35:25 smithi092 ceph-mon[29572]: 8.b scrub ok 2024-03-11T21:35:25.861 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:35:25 smithi092 ceph-mon[29572]: purged_snaps scrub starts 2024-03-11T21:35:25.861 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:35:25 smithi092 ceph-mon[29572]: purged_snaps scrub ok 2024-03-11T21:35:25.959 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:35:25.982 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:35:25.983 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:35:25.984 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:35:26.085 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:35:26.109 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:35:26.110 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:35:26.110 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:35:26.211 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:35:26.235 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:35:26.235 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:35:26.236 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:35:26.338 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:35:26.362 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:35:26.362 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:35:26.363 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:35:26.464 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:35:26.488 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:35:26.488 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:35:26.489 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:35:26.590 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:35:26.613 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:35:26.614 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:35:26.614 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:35:26.715 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:35:26.738 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:35:26.739 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:35:26.739 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:35:26.826 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:35:26 smithi097 ceph-mon[33317]: pgmap v437: 158 pgs: 1 active+remapped+backfill_toofull, 157 active+clean; 217 MiB data, 3.9 GiB used, 711 GiB / 715 GiB avail; 853 B/s rd, 0 B/s wr, 1.04k op/s; 144/10983 objects misplaced (1.311%) 2024-03-11T21:35:26.826 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:35:26 smithi097 ceph-mon[33317]: 5.1b scrub starts 2024-03-11T21:35:26.826 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:35:26 smithi097 ceph-mon[33317]: 5.1b scrub ok 2024-03-11T21:35:26.826 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:35:26 smithi097 ceph-mon[33317]: 4.d scrub starts 2024-03-11T21:35:26.826 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:35:26 smithi097 ceph-mon[33317]: 4.d scrub ok 2024-03-11T21:35:26.826 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:35:26 smithi097 ceph-mon[33317]: osdmap e250: 8 total, 8 up, 8 in 2024-03-11T21:35:26.826 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:35:26 smithi097 ceph-mon[33317]: from='mgr.14607 v1:172.21.15.92:0/1702717597' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T21:35:26.826 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:35:26 smithi097 ceph-mon[33317]: 5.14 scrub starts 2024-03-11T21:35:26.826 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:35:26 smithi097 ceph-mon[33317]: 5.14 scrub ok 2024-03-11T21:35:26.841 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:35:26.860 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:35:26 smithi092 ceph-mon[29572]: pgmap v437: 158 pgs: 1 active+remapped+backfill_toofull, 157 active+clean; 217 MiB data, 3.9 GiB used, 711 GiB / 715 GiB avail; 853 B/s rd, 0 B/s wr, 1.04k op/s; 144/10983 objects misplaced (1.311%) 2024-03-11T21:35:26.860 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:35:26 smithi092 ceph-mon[29572]: 5.1b scrub starts 2024-03-11T21:35:26.860 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:35:26 smithi092 ceph-mon[29572]: 5.1b scrub ok 2024-03-11T21:35:26.860 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:35:26 smithi092 ceph-mon[29572]: 4.d scrub starts 2024-03-11T21:35:26.860 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:35:26 smithi092 ceph-mon[29572]: 4.d scrub ok 2024-03-11T21:35:26.860 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:35:26 smithi092 ceph-mon[29572]: osdmap e250: 8 total, 8 up, 8 in 2024-03-11T21:35:26.860 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:35:26 smithi092 ceph-mon[29572]: from='mgr.14607 v1:172.21.15.92:0/1702717597' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T21:35:26.860 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:35:26 smithi092 ceph-mon[29572]: 5.14 scrub starts 2024-03-11T21:35:26.860 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:35:26 smithi092 ceph-mon[29572]: 5.14 scrub ok 2024-03-11T21:35:26.861 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:35:26 smithi092 ceph-mon[37563]: pgmap v437: 158 pgs: 1 active+remapped+backfill_toofull, 157 active+clean; 217 MiB data, 3.9 GiB used, 711 GiB / 715 GiB avail; 853 B/s rd, 0 B/s wr, 1.04k op/s; 144/10983 objects misplaced (1.311%) 2024-03-11T21:35:26.861 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:35:26 smithi092 ceph-mon[37563]: 5.1b scrub starts 2024-03-11T21:35:26.861 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:35:26 smithi092 ceph-mon[37563]: 5.1b scrub ok 2024-03-11T21:35:26.861 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:35:26 smithi092 ceph-mon[37563]: 4.d scrub starts 2024-03-11T21:35:26.861 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:35:26 smithi092 ceph-mon[37563]: 4.d scrub ok 2024-03-11T21:35:26.861 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:35:26 smithi092 ceph-mon[37563]: osdmap e250: 8 total, 8 up, 8 in 2024-03-11T21:35:26.861 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:35:26 smithi092 ceph-mon[37563]: from='mgr.14607 v1:172.21.15.92:0/1702717597' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T21:35:26.861 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:35:26 smithi092 ceph-mon[37563]: 5.14 scrub starts 2024-03-11T21:35:26.861 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:35:26 smithi092 ceph-mon[37563]: 5.14 scrub ok 2024-03-11T21:35:26.865 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:35:26.866 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:35:26.866 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:35:26.967 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:35:26.990 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:35:26.991 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:35:26.992 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:35:27.092 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:35:27.116 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:35:27.116 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:35:27.117 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:35:27.218 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:35:27.241 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:35:27.242 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:35:27.242 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:35:27.343 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:35:27.368 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:35:27.369 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:35:27.369 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:35:27.470 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:35:27.495 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:35:27.496 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:35:27.496 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:35:27.597 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:35:27.623 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:35:27.623 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:35:27.624 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:35:27.628 DEBUG:teuthology.orchestra.run.smithi097:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid 00db0016-dfed-11ee-95c8-87774f69a715 -- ceph daemon osd.4 dump_ops_in_flight 2024-03-11T21:35:27.725 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:35:27.749 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:35:27.749 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:35:27.750 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:35:27.850 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:35:27.860 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:35:27 smithi092 ceph-mon[29572]: 6.19 scrub starts 2024-03-11T21:35:27.860 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:35:27 smithi092 ceph-mon[29572]: 6.19 scrub ok 2024-03-11T21:35:27.860 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:35:27 smithi092 ceph-mon[29572]: 2.1d scrub starts 2024-03-11T21:35:27.860 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:35:27 smithi092 ceph-mon[29572]: 2.1d scrub ok 2024-03-11T21:35:27.860 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:35:27 smithi092 ceph-mon[37563]: 6.19 scrub starts 2024-03-11T21:35:27.860 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:35:27 smithi092 ceph-mon[37563]: 6.19 scrub ok 2024-03-11T21:35:27.860 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:35:27 smithi092 ceph-mon[37563]: 2.1d scrub starts 2024-03-11T21:35:27.860 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:35:27 smithi092 ceph-mon[37563]: 2.1d scrub ok 2024-03-11T21:35:27.873 INFO:teuthology.orchestra.run.smithi097.stderr:Inferring config /var/lib/ceph/00db0016-dfed-11ee-95c8-87774f69a715/mon.b/config 2024-03-11T21:35:27.876 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:35:27.876 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:35:27.877 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:35:27.977 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:35:27.995 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:35:27 smithi097 ceph-mon[33317]: 6.19 scrub starts 2024-03-11T21:35:27.995 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:35:27 smithi097 ceph-mon[33317]: 6.19 scrub ok 2024-03-11T21:35:27.995 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:35:27 smithi097 ceph-mon[33317]: 2.1d scrub starts 2024-03-11T21:35:27.995 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:35:27 smithi097 ceph-mon[33317]: 2.1d scrub ok 2024-03-11T21:35:28.002 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:35:28.002 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:35:28.003 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:35:28.103 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:35:28.128 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:35:28.129 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:35:28.129 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:35:28.230 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:35:28.257 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:35:28.258 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:35:28.259 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:35:28.359 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:35:28.360 INFO:journalctl@ceph.mgr.y.smithi092.stdout:Mar 11 21:35:27 smithi092 ceph-00db0016-dfed-11ee-95c8-87774f69a715-mgr-y[29801]: ::ffff:172.21.15.97 - - [11/Mar/2024:21:35:27] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-11T21:35:28.385 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:35:28.385 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:35:28.386 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:35:28.487 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:35:28.511 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:35:28.511 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:35:28.512 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:35:28.613 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:35:28.636 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:35:28.636 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:35:28.637 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:35:28.738 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:35:28.762 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:35:28.763 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:35:28.763 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:35:28.821 INFO:tasks.cephadm.ceph_manager.ceph:removing pool_name unique_pool_1 2024-03-11T21:35:28.821 DEBUG:teuthology.orchestra.run.smithi092:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid 00db0016-dfed-11ee-95c8-87774f69a715 -- ceph osd pool rm unique_pool_1 unique_pool_1 --yes-i-really-really-mean-it 2024-03-11T21:35:28.847 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:35:28 smithi092 ceph-mon[37563]: 2.12 scrub starts 2024-03-11T21:35:28.847 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:35:28 smithi092 ceph-mon[37563]: 2.12 scrub ok 2024-03-11T21:35:28.847 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:35:28 smithi092 ceph-mon[37563]: pgmap v439: 158 pgs: 1 active+remapped+backfill_toofull, 157 active+clean; 153 MiB data, 3.7 GiB used, 712 GiB / 715 GiB avail; 853 B/s rd, 0 B/s wr, 1.22k op/s; 144/7905 objects misplaced (1.822%); 72 B/s, 0 objects/s recovering 2024-03-11T21:35:28.847 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:35:28 smithi092 ceph-mon[37563]: 6.14 scrub starts 2024-03-11T21:35:28.847 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:35:28 smithi092 ceph-mon[37563]: 6.14 scrub ok 2024-03-11T21:35:28.847 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:35:28 smithi092 ceph-mon[37563]: 8.17 scrub starts 2024-03-11T21:35:28.847 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:35:28 smithi092 ceph-mon[37563]: 8.17 scrub ok 2024-03-11T21:35:28.847 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:35:28 smithi092 ceph-mon[37563]: 4.14 scrub starts 2024-03-11T21:35:28.848 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:35:28 smithi092 ceph-mon[37563]: 4.14 scrub ok 2024-03-11T21:35:28.848 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:35:28 smithi092 ceph-mon[37563]: osdmap e251: 8 total, 8 up, 8 in 2024-03-11T21:35:28.848 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:35:28 smithi092 ceph-mon[29572]: 2.12 scrub starts 2024-03-11T21:35:28.848 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:35:28 smithi092 ceph-mon[29572]: 2.12 scrub ok 2024-03-11T21:35:28.848 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:35:28 smithi092 ceph-mon[29572]: pgmap v439: 158 pgs: 1 active+remapped+backfill_toofull, 157 active+clean; 153 MiB data, 3.7 GiB used, 712 GiB / 715 GiB avail; 853 B/s rd, 0 B/s wr, 1.22k op/s; 144/7905 objects misplaced (1.822%); 72 B/s, 0 objects/s recovering 2024-03-11T21:35:28.848 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:35:28 smithi092 ceph-mon[29572]: 6.14 scrub starts 2024-03-11T21:35:28.848 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:35:28 smithi092 ceph-mon[29572]: 6.14 scrub ok 2024-03-11T21:35:28.848 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:35:28 smithi092 ceph-mon[29572]: 8.17 scrub starts 2024-03-11T21:35:28.848 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:35:28 smithi092 ceph-mon[29572]: 8.17 scrub ok 2024-03-11T21:35:28.848 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:35:28 smithi092 ceph-mon[29572]: 4.14 scrub starts 2024-03-11T21:35:28.848 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:35:28 smithi092 ceph-mon[29572]: 4.14 scrub ok 2024-03-11T21:35:28.849 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:35:28 smithi092 ceph-mon[29572]: osdmap e251: 8 total, 8 up, 8 in 2024-03-11T21:35:28.865 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:35:28.888 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:35:28.889 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:35:28.889 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:35:28.990 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:35:29.017 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:35:29.018 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:35:29.018 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:35:29.075 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:35:28 smithi097 ceph-mon[33317]: 2.12 scrub starts 2024-03-11T21:35:29.075 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:35:28 smithi097 ceph-mon[33317]: 2.12 scrub ok 2024-03-11T21:35:29.076 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:35:28 smithi097 ceph-mon[33317]: pgmap v439: 158 pgs: 1 active+remapped+backfill_toofull, 157 active+clean; 153 MiB data, 3.7 GiB used, 712 GiB / 715 GiB avail; 853 B/s rd, 0 B/s wr, 1.22k op/s; 144/7905 objects misplaced (1.822%); 72 B/s, 0 objects/s recovering 2024-03-11T21:35:29.076 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:35:28 smithi097 ceph-mon[33317]: 6.14 scrub starts 2024-03-11T21:35:29.076 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:35:28 smithi097 ceph-mon[33317]: 6.14 scrub ok 2024-03-11T21:35:29.076 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:35:28 smithi097 ceph-mon[33317]: 8.17 scrub starts 2024-03-11T21:35:29.076 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:35:28 smithi097 ceph-mon[33317]: 8.17 scrub ok 2024-03-11T21:35:29.076 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:35:28 smithi097 ceph-mon[33317]: 4.14 scrub starts 2024-03-11T21:35:29.076 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:35:28 smithi097 ceph-mon[33317]: 4.14 scrub ok 2024-03-11T21:35:29.076 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:35:28 smithi097 ceph-mon[33317]: osdmap e251: 8 total, 8 up, 8 in 2024-03-11T21:35:29.120 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:35:29.147 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:35:29.148 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:35:29.148 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:35:29.249 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:35:29.273 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:35:29.274 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:35:29.275 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:35:29.375 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:35:29.403 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:35:29.403 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:35:29.403 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:35:29.466 INFO:journalctl@ceph.osd.3.smithi092.stdout:Mar 11 21:35:29 smithi092 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd-3[59566]: 2024-03-11T21:35:29.310+0000 7f484845f700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.3 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-11T21:35:29.466 INFO:journalctl@ceph.osd.3.smithi092.stdout:Mar 11 21:35:29 smithi092 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd-3[59566]: 2024-03-11T21:35:29.311+0000 7f484845f700 -1 osd.3 251 *** Got signal Terminated *** 2024-03-11T21:35:29.466 INFO:journalctl@ceph.osd.3.smithi092.stdout:Mar 11 21:35:29 smithi092 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd-3[59566]: 2024-03-11T21:35:29.311+0000 7f484845f700 -1 osd.3 251 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-03-11T21:35:29.504 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:35:29.530 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:35:29.530 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:35:29.531 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:35:29.631 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:35:29.657 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:35:29.658 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:35:29.658 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:35:29.731 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:35:29 smithi092 ceph-mon[37563]: 4.b scrub starts 2024-03-11T21:35:29.732 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:35:29 smithi092 ceph-mon[37563]: 4.b scrub ok 2024-03-11T21:35:29.732 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:35:29 smithi092 ceph-mon[37563]: 8.a scrub starts 2024-03-11T21:35:29.732 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:35:29 smithi092 ceph-mon[37563]: 8.a scrub ok 2024-03-11T21:35:29.732 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:35:29 smithi092 ceph-mon[37563]: 2.1c scrub starts 2024-03-11T21:35:29.732 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:35:29 smithi092 ceph-mon[37563]: 2.1c scrub ok 2024-03-11T21:35:29.732 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:35:29 smithi092 ceph-mon[37563]: 8.5 scrub starts 2024-03-11T21:35:29.732 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:35:29 smithi092 ceph-mon[37563]: 8.5 scrub ok 2024-03-11T21:35:29.732 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:35:29 smithi092 ceph-mon[37563]: osd.3 marked itself down and dead 2024-03-11T21:35:29.732 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:35:29 smithi092 ceph-mon[37563]: from='mgr.14607 v1:172.21.15.92:0/1702717597' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T21:35:29.732 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:35:29 smithi092 ceph-mon[37563]: from='mgr.14607 v1:172.21.15.92:0/1702717597' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T21:35:29.732 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:35:29 smithi092 ceph-mon[37563]: from='mgr.14607 ' entity='mgr.y' 2024-03-11T21:35:29.733 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:35:29 smithi092 ceph-mon[29572]: 4.b scrub starts 2024-03-11T21:35:29.733 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:35:29 smithi092 ceph-mon[29572]: 4.b scrub ok 2024-03-11T21:35:29.733 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:35:29 smithi092 ceph-mon[29572]: 8.a scrub starts 2024-03-11T21:35:29.733 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:35:29 smithi092 ceph-mon[29572]: 8.a scrub ok 2024-03-11T21:35:29.733 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:35:29 smithi092 ceph-mon[29572]: 2.1c scrub starts 2024-03-11T21:35:29.733 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:35:29 smithi092 ceph-mon[29572]: 2.1c scrub ok 2024-03-11T21:35:29.733 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:35:29 smithi092 ceph-mon[29572]: 8.5 scrub starts 2024-03-11T21:35:29.733 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:35:29 smithi092 ceph-mon[29572]: 8.5 scrub ok 2024-03-11T21:35:29.733 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:35:29 smithi092 ceph-mon[29572]: osd.3 marked itself down and dead 2024-03-11T21:35:29.733 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:35:29 smithi092 ceph-mon[29572]: from='mgr.14607 v1:172.21.15.92:0/1702717597' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T21:35:29.733 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:35:29 smithi092 ceph-mon[29572]: from='mgr.14607 v1:172.21.15.92:0/1702717597' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T21:35:29.733 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:35:29 smithi092 ceph-mon[29572]: from='mgr.14607 ' entity='mgr.y' 2024-03-11T21:35:29.759 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:35:29.785 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:35:29.785 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:35:29.786 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:35:29.887 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:35:29.912 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:35:29.913 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:35:29.913 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:35:30.014 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:35:30.040 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:35:30.041 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:35:30.041 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:35:30.076 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:35:29 smithi097 ceph-mon[33317]: 4.b scrub starts 2024-03-11T21:35:30.076 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:35:29 smithi097 ceph-mon[33317]: 4.b scrub ok 2024-03-11T21:35:30.076 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:35:29 smithi097 ceph-mon[33317]: 8.a scrub starts 2024-03-11T21:35:30.076 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:35:29 smithi097 ceph-mon[33317]: 8.a scrub ok 2024-03-11T21:35:30.076 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:35:29 smithi097 ceph-mon[33317]: 2.1c scrub starts 2024-03-11T21:35:30.076 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:35:29 smithi097 ceph-mon[33317]: 2.1c scrub ok 2024-03-11T21:35:30.076 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:35:29 smithi097 ceph-mon[33317]: 8.5 scrub starts 2024-03-11T21:35:30.076 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:35:29 smithi097 ceph-mon[33317]: 8.5 scrub ok 2024-03-11T21:35:30.076 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:35:29 smithi097 ceph-mon[33317]: osd.3 marked itself down and dead 2024-03-11T21:35:30.076 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:35:29 smithi097 ceph-mon[33317]: from='mgr.14607 v1:172.21.15.92:0/1702717597' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T21:35:30.077 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:35:29 smithi097 ceph-mon[33317]: from='mgr.14607 v1:172.21.15.92:0/1702717597' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T21:35:30.077 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:35:29 smithi097 ceph-mon[33317]: from='mgr.14607 ' entity='mgr.y' 2024-03-11T21:35:30.142 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:35:30.734 INFO:teuthology.orchestra.run.smithi092.stdout:osd.0: {} 2024-03-11T21:35:30.735 INFO:teuthology.orchestra.run.smithi092.stdout:osd.1: {} 2024-03-11T21:35:30.735 INFO:teuthology.orchestra.run.smithi092.stdout:osd.2: {} 2024-03-11T21:35:30.735 INFO:teuthology.orchestra.run.smithi092.stdout:osd.4: {} 2024-03-11T21:35:30.735 INFO:teuthology.orchestra.run.smithi092.stdout:osd.5: {} 2024-03-11T21:35:30.735 INFO:teuthology.orchestra.run.smithi092.stdout:osd.6: {} 2024-03-11T21:35:30.735 INFO:teuthology.orchestra.run.smithi092.stdout:osd.7: {} 2024-03-11T21:35:30.736 INFO:teuthology.orchestra.run.smithi092.stderr:osd.0: osd_enable_op_tracker = '' 2024-03-11T21:35:30.736 INFO:teuthology.orchestra.run.smithi092.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-11T21:35:30.736 INFO:teuthology.orchestra.run.smithi092.stderr:osd.2: osd_enable_op_tracker = '' 2024-03-11T21:35:30.736 INFO:teuthology.orchestra.run.smithi092.stderr:Error ENXIO: problem getting command descriptions from osd.3 2024-03-11T21:35:30.737 INFO:teuthology.orchestra.run.smithi092.stderr:osd.4: osd_enable_op_tracker = '' 2024-03-11T21:35:30.737 INFO:teuthology.orchestra.run.smithi092.stderr:osd.5: osd_enable_op_tracker = '' 2024-03-11T21:35:30.737 INFO:teuthology.orchestra.run.smithi092.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-11T21:35:30.737 INFO:teuthology.orchestra.run.smithi092.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-11T21:35:30.737 INFO:journalctl@ceph.osd.3.smithi092.stdout:Mar 11 21:35:30 smithi092 podman[126083]: 2024-03-11 21:35:30.287507339 +0000 UTC m=+1.118531840 container died 5dd2cfb6f5fe5bcf95b15479740abc616d8035f6e80f21dae45192030965414b (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:8fe9635f3beb0247496d7e321f3068bd10c333c433ef4824d7e1c1a7d3d78379, name=ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd-3, ceph=True, RELEASE=main-2b0101e, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240307, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, GIT_CLEAN=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-03-11T21:35:30.738 INFO:teuthology.orchestra.run.smithi092.stderr:nodeep-scrub is unset 2024-03-11T21:35:30.763 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:35:30.763 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:35:30.764 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:35:30.806 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:35:30 smithi097 ceph-mon[33317]: pgmap v441: 158 pgs: 1 active+remapped+backfill_toofull, 157 active+clean; 582 KiB data, 3.0 GiB used, 712 GiB / 715 GiB avail; 853 B/s rd, 0 B/s wr, 784 op/s; 72 B/s, 0 objects/s recovering 2024-03-11T21:35:30.806 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:35:30 smithi097 ceph-mon[33317]: 4.15 scrub starts 2024-03-11T21:35:30.806 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:35:30 smithi097 ceph-mon[33317]: 4.15 scrub ok 2024-03-11T21:35:30.806 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:35:30 smithi097 ceph-mon[33317]: Health check failed: 1 osds down (OSD_DOWN) 2024-03-11T21:35:30.806 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:35:30 smithi097 ceph-mon[33317]: osdmap e252: 8 total, 7 up, 8 in 2024-03-11T21:35:30.806 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:35:30 smithi097 ceph-mon[33317]: from='client.? v1:172.21.15.92:0/689680006' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-11T21:35:30.806 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:35:30 smithi097 ceph-mon[33317]: from='client.? v1:172.21.15.92:0/2627357547' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-11T21:35:30.833 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:35:30 smithi092 ceph-mon[29572]: pgmap v441: 158 pgs: 1 active+remapped+backfill_toofull, 157 active+clean; 582 KiB data, 3.0 GiB used, 712 GiB / 715 GiB avail; 853 B/s rd, 0 B/s wr, 784 op/s; 72 B/s, 0 objects/s recovering 2024-03-11T21:35:30.833 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:35:30 smithi092 ceph-mon[29572]: 4.15 scrub starts 2024-03-11T21:35:30.833 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:35:30 smithi092 ceph-mon[29572]: 4.15 scrub ok 2024-03-11T21:35:30.833 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:35:30 smithi092 ceph-mon[29572]: Health check failed: 1 osds down (OSD_DOWN) 2024-03-11T21:35:30.833 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:35:30 smithi092 ceph-mon[29572]: osdmap e252: 8 total, 7 up, 8 in 2024-03-11T21:35:30.833 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:35:30 smithi092 ceph-mon[29572]: from='client.? v1:172.21.15.92:0/689680006' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-11T21:35:30.834 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:35:30 smithi092 ceph-mon[29572]: from='client.? v1:172.21.15.92:0/2627357547' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-11T21:35:30.834 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:35:30 smithi092 ceph-mon[37563]: pgmap v441: 158 pgs: 1 active+remapped+backfill_toofull, 157 active+clean; 582 KiB data, 3.0 GiB used, 712 GiB / 715 GiB avail; 853 B/s rd, 0 B/s wr, 784 op/s; 72 B/s, 0 objects/s recovering 2024-03-11T21:35:30.834 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:35:30 smithi092 ceph-mon[37563]: 4.15 scrub starts 2024-03-11T21:35:30.834 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:35:30 smithi092 ceph-mon[37563]: 4.15 scrub ok 2024-03-11T21:35:30.834 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:35:30 smithi092 ceph-mon[37563]: Health check failed: 1 osds down (OSD_DOWN) 2024-03-11T21:35:30.834 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:35:30 smithi092 ceph-mon[37563]: osdmap e252: 8 total, 7 up, 8 in 2024-03-11T21:35:30.834 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:35:30 smithi092 ceph-mon[37563]: from='client.? v1:172.21.15.92:0/689680006' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-11T21:35:30.834 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:35:30 smithi092 ceph-mon[37563]: from='client.? v1:172.21.15.92:0/2627357547' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-11T21:35:30.865 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:35:30.890 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:35:30.891 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:35:30.891 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:35:30.992 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:35:31.019 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:35:31.020 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:35:31.020 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:35:31.110 INFO:journalctl@ceph.osd.3.smithi092.stdout:Mar 11 21:35:30 smithi092 podman[126083]: 2024-03-11 21:35:30.852009106 +0000 UTC m=+1.683033613 container cleanup 5dd2cfb6f5fe5bcf95b15479740abc616d8035f6e80f21dae45192030965414b (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:8fe9635f3beb0247496d7e321f3068bd10c333c433ef4824d7e1c1a7d3d78379, name=ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd-3, GIT_CLEAN=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240307, ceph=True, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, RELEASE=main-2b0101e, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=) 2024-03-11T21:35:31.110 INFO:journalctl@ceph.osd.3.smithi092.stdout:Mar 11 21:35:30 smithi092 bash[126083]: ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd-3 2024-03-11T21:35:31.110 INFO:journalctl@ceph.osd.3.smithi092.stdout:Mar 11 21:35:31 smithi092 podman[126228]: 2024-03-11 21:35:31.011795981 +0000 UTC m=+0.778141210 container remove 5dd2cfb6f5fe5bcf95b15479740abc616d8035f6e80f21dae45192030965414b (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:8fe9635f3beb0247496d7e321f3068bd10c333c433ef4824d7e1c1a7d3d78379, name=ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd-3, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , RELEASE=main-2b0101e, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.license=GPLv2, ceph=True, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240307) 2024-03-11T21:35:31.121 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:35:31.147 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:35:31.164 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:35:31.165 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:35:31.266 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:35:31.289 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:35:31.290 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:35:31.290 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:35:31.377 INFO:teuthology.orchestra.run.smithi097.stdout:ERROR: (22) Invalid argument 2024-03-11T21:35:31.377 INFO:teuthology.orchestra.run.smithi097.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-11T21:35:31.392 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:35:31.417 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:35:31.417 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:35:31.418 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:35:31.518 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:35:31.542 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:35:31.543 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:35:31.543 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:35:31.644 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:35:31.656 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:35:31 smithi097 ceph-mon[33317]: 6.10 scrub starts 2024-03-11T21:35:31.656 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:35:31 smithi097 ceph-mon[33317]: 6.10 scrub ok 2024-03-11T21:35:31.656 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:35:31 smithi097 ceph-mon[33317]: 2.6 scrub starts 2024-03-11T21:35:31.656 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:35:31 smithi097 ceph-mon[33317]: 2.6 scrub ok 2024-03-11T21:35:31.656 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:35:31 smithi097 ceph-mon[33317]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-03-11T21:35:31.656 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:35:31 smithi097 ceph-mon[33317]: from='client.? v1:172.21.15.92:0/689680006' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-03-11T21:35:31.656 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:35:31 smithi097 ceph-mon[33317]: osdmap e253: 8 total, 7 up, 8 in 2024-03-11T21:35:31.656 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:35:31 smithi097 ceph-mon[33317]: 5.11 scrub starts 2024-03-11T21:35:31.656 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:35:31 smithi097 ceph-mon[33317]: 5.11 scrub ok 2024-03-11T21:35:31.668 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:35:31.669 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:35:31.669 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:35:31.770 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:35:31.795 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:35:31.795 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:35:31.795 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:35:31.815 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:35:31 smithi092 ceph-mon[29572]: 6.10 scrub starts 2024-03-11T21:35:31.815 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:35:31 smithi092 ceph-mon[29572]: 6.10 scrub ok 2024-03-11T21:35:31.815 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:35:31 smithi092 ceph-mon[29572]: 2.6 scrub starts 2024-03-11T21:35:31.815 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:35:31 smithi092 ceph-mon[29572]: 2.6 scrub ok 2024-03-11T21:35:31.815 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:35:31 smithi092 ceph-mon[29572]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-03-11T21:35:31.815 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:35:31 smithi092 ceph-mon[29572]: from='client.? v1:172.21.15.92:0/689680006' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-03-11T21:35:31.815 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:35:31 smithi092 ceph-mon[29572]: osdmap e253: 8 total, 7 up, 8 in 2024-03-11T21:35:31.815 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:35:31 smithi092 ceph-mon[29572]: 5.11 scrub starts 2024-03-11T21:35:31.815 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:35:31 smithi092 ceph-mon[29572]: 5.11 scrub ok 2024-03-11T21:35:31.816 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:35:31 smithi092 ceph-mon[37563]: 6.10 scrub starts 2024-03-11T21:35:31.816 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:35:31 smithi092 ceph-mon[37563]: 6.10 scrub ok 2024-03-11T21:35:31.816 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:35:31 smithi092 ceph-mon[37563]: 2.6 scrub starts 2024-03-11T21:35:31.816 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:35:31 smithi092 ceph-mon[37563]: 2.6 scrub ok 2024-03-11T21:35:31.816 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:35:31 smithi092 ceph-mon[37563]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-03-11T21:35:31.816 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:35:31 smithi092 ceph-mon[37563]: from='client.? v1:172.21.15.92:0/689680006' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-03-11T21:35:31.816 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:35:31 smithi092 ceph-mon[37563]: osdmap e253: 8 total, 7 up, 8 in 2024-03-11T21:35:31.816 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:35:31 smithi092 ceph-mon[37563]: 5.11 scrub starts 2024-03-11T21:35:31.816 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:35:31 smithi092 ceph-mon[37563]: 5.11 scrub ok 2024-03-11T21:35:31.896 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:35:31.919 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:35:31.920 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:35:31.920 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:35:32.021 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:35:32.045 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:35:32.045 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:35:32.046 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:35:32.088 DEBUG:teuthology.orchestra.run.smithi097:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid 00db0016-dfed-11ee-95c8-87774f69a715 -- ceph daemon osd.4 dump_blocked_ops 2024-03-11T21:35:32.146 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:35:32.169 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:35:32.170 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:35:32.170 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:35:32.271 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:35:32.297 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:35:32.298 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:35:32.298 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:35:32.303 INFO:teuthology.orchestra.run.smithi092.stderr:Inferring config /var/lib/ceph/00db0016-dfed-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T21:35:32.304 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-03-11T21:35:32.304 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-03-11T21:35:32.336 INFO:teuthology.orchestra.run.smithi097.stderr:Inferring config /var/lib/ceph/00db0016-dfed-11ee-95c8-87774f69a715/mon.b/config 2024-03-11T21:35:32.399 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:35:32.422 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:35:32.423 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:35:32.423 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:35:32.524 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:35:32.548 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:35:32.548 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:35:32.549 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:35:32.649 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:35:32.673 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:35:32.673 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:35:32.674 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:35:32.713 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:35:32 smithi092 ceph-mon[29572]: pgmap v444: 158 pgs: 27 stale+active+clean, 1 active+remapped+backfill_toofull, 130 active+clean; 582 KiB data, 3.0 GiB used, 712 GiB / 715 GiB avail; 917 B/s rd, 0 B/s wr, 627 op/s; 78 B/s, 0 objects/s recovering 2024-03-11T21:35:32.713 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:35:32 smithi092 ceph-mon[29572]: 2.7 scrub starts 2024-03-11T21:35:32.714 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:35:32 smithi092 ceph-mon[29572]: 2.7 scrub ok 2024-03-11T21:35:32.714 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:35:32 smithi092 ceph-mon[37563]: pgmap v444: 158 pgs: 27 stale+active+clean, 1 active+remapped+backfill_toofull, 130 active+clean; 582 KiB data, 3.0 GiB used, 712 GiB / 715 GiB avail; 917 B/s rd, 0 B/s wr, 627 op/s; 78 B/s, 0 objects/s recovering 2024-03-11T21:35:32.714 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:35:32 smithi092 ceph-mon[37563]: 2.7 scrub starts 2024-03-11T21:35:32.714 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:35:32 smithi092 ceph-mon[37563]: 2.7 scrub ok 2024-03-11T21:35:32.743 INFO:journalctl@ceph.iscsi.iscsi.a.smithi097.stdout:Mar 11 21:35:32 smithi097 ceph-00db0016-dfed-11ee-95c8-87774f69a715-iscsi-iscsi-a[60943]: debug there is no tcmu-runner data available 2024-03-11T21:35:32.743 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:35:32 smithi097 ceph-mon[33317]: pgmap v444: 158 pgs: 27 stale+active+clean, 1 active+remapped+backfill_toofull, 130 active+clean; 582 KiB data, 3.0 GiB used, 712 GiB / 715 GiB avail; 917 B/s rd, 0 B/s wr, 627 op/s; 78 B/s, 0 objects/s recovering 2024-03-11T21:35:32.743 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:35:32 smithi097 ceph-mon[33317]: 2.7 scrub starts 2024-03-11T21:35:32.743 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:35:32 smithi097 ceph-mon[33317]: 2.7 scrub ok 2024-03-11T21:35:32.775 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:35:32.798 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:35:32.798 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:35:32.799 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:35:32.899 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:35:32.924 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:35:32.925 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:35:32.925 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:35:33.026 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:35:33.050 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:35:33.051 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:35:33.051 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:35:33.152 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:35:33.175 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:35:33.176 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:35:33.176 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:35:33.277 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:35:33.301 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:35:33.301 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:35:33.302 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:35:33.402 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:35:33.426 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:35:33.427 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:35:33.427 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:35:33.528 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:35:33.552 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:35:33.553 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:35:33.554 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:35:33.654 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:35:33.681 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:35:33.682 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:35:33.682 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:35:33.783 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:35:33.807 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:35:33.808 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:35:33.808 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:35:33.817 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:35:33 smithi097 ceph-mon[33317]: 5.3 scrub starts 2024-03-11T21:35:33.817 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:35:33 smithi097 ceph-mon[33317]: 5.3 scrub ok 2024-03-11T21:35:33.817 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:35:33 smithi097 ceph-mon[33317]: from='client.24469 v1:172.21.15.97:0/404208310' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:35:33.817 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:35:33 smithi097 ceph-mon[33317]: 4.12 scrub starts 2024-03-11T21:35:33.818 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:35:33 smithi097 ceph-mon[33317]: 4.12 scrub ok 2024-03-11T21:35:33.818 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:35:33 smithi097 ceph-mon[33317]: 6.1e scrub starts 2024-03-11T21:35:33.818 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:35:33 smithi097 ceph-mon[33317]: 6.1e scrub ok 2024-03-11T21:35:33.818 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:35:33 smithi097 ceph-mon[33317]: 6.12 scrub starts 2024-03-11T21:35:33.818 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:35:33 smithi097 ceph-mon[33317]: 6.12 scrub ok 2024-03-11T21:35:33.818 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:35:33 smithi097 ceph-mon[33317]: osdmap e254: 8 total, 7 up, 8 in 2024-03-11T21:35:33.839 DEBUG:teuthology.orchestra.run.smithi092:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid 00db0016-dfed-11ee-95c8-87774f69a715 -- ceph osd set noscrub 2024-03-11T21:35:33.840 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:35:33 smithi092 ceph-mon[29572]: 5.3 scrub starts 2024-03-11T21:35:33.841 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:35:33 smithi092 ceph-mon[29572]: 5.3 scrub ok 2024-03-11T21:35:33.841 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:35:33 smithi092 ceph-mon[29572]: from='client.24469 v1:172.21.15.97:0/404208310' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:35:33.841 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:35:33 smithi092 ceph-mon[29572]: 4.12 scrub starts 2024-03-11T21:35:33.841 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:35:33 smithi092 ceph-mon[29572]: 4.12 scrub ok 2024-03-11T21:35:33.841 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:35:33 smithi092 ceph-mon[29572]: 6.1e scrub starts 2024-03-11T21:35:33.841 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:35:33 smithi092 ceph-mon[29572]: 6.1e scrub ok 2024-03-11T21:35:33.841 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:35:33 smithi092 ceph-mon[29572]: 6.12 scrub starts 2024-03-11T21:35:33.841 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:35:33 smithi092 ceph-mon[29572]: 6.12 scrub ok 2024-03-11T21:35:33.841 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:35:33 smithi092 ceph-mon[29572]: osdmap e254: 8 total, 7 up, 8 in 2024-03-11T21:35:33.841 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:35:33 smithi092 ceph-mon[37563]: 5.3 scrub starts 2024-03-11T21:35:33.841 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:35:33 smithi092 ceph-mon[37563]: 5.3 scrub ok 2024-03-11T21:35:33.842 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:35:33 smithi092 ceph-mon[37563]: from='client.24469 v1:172.21.15.97:0/404208310' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:35:33.842 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:35:33 smithi092 ceph-mon[37563]: 4.12 scrub starts 2024-03-11T21:35:33.842 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:35:33 smithi092 ceph-mon[37563]: 4.12 scrub ok 2024-03-11T21:35:33.842 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:35:33 smithi092 ceph-mon[37563]: 6.1e scrub starts 2024-03-11T21:35:33.842 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:35:33 smithi092 ceph-mon[37563]: 6.1e scrub ok 2024-03-11T21:35:33.842 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:35:33 smithi092 ceph-mon[37563]: 6.12 scrub starts 2024-03-11T21:35:33.842 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:35:33 smithi092 ceph-mon[37563]: 6.12 scrub ok 2024-03-11T21:35:33.842 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:35:33 smithi092 ceph-mon[37563]: osdmap e254: 8 total, 7 up, 8 in 2024-03-11T21:35:33.909 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:35:33.933 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:35:33.934 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:35:33.934 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:35:34.035 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:35:34.060 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:35:34.060 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:35:34.060 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:35:34.072 INFO:teuthology.orchestra.run.smithi097.stdout:ERROR: (22) Invalid argument 2024-03-11T21:35:34.072 INFO:teuthology.orchestra.run.smithi097.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-11T21:35:34.110 INFO:journalctl@ceph.osd.3.smithi092.stdout:Mar 11 21:35:33 smithi092 podman[126590]: 2024-03-11 21:35:33.762601159 +0000 UTC m=+0.019511745 image pull 4133f8bc41bb201212a28168427a2edb1f6f3795e694bf61031ea9e2f3ff5e87 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:8fe9635f3beb0247496d7e321f3068bd10c333c433ef4824d7e1c1a7d3d78379 2024-03-11T21:35:34.110 INFO:journalctl@ceph.osd.3.smithi092.stdout:Mar 11 21:35:33 smithi092 podman[126590]: 2024-03-11 21:35:33.939894786 +0000 UTC m=+0.196805371 container create f7cdbdab41f652f95a52058a5834d79428ac367a558a10ff69fb4011daf80df1 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:8fe9635f3beb0247496d7e321f3068bd10c333c433ef4824d7e1c1a7d3d78379, name=ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd-3-deactivate, GIT_BRANCH=HEAD, ceph=True, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , GIT_CLEAN=True, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, org.label-schema.build-date=20240307, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, RELEASE=main-2b0101e, org.label-schema.vendor=CentOS) 2024-03-11T21:35:34.162 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:35:34.186 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:35:34.187 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:35:34.187 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:35:34.288 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:35:34.305 DEBUG:teuthology.orchestra.run.smithi092:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid 00db0016-dfed-11ee-95c8-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-03-11T21:35:34.311 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:35:34.311 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:35:34.312 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:35:34.413 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:35:34.437 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:35:34.438 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:35:34.438 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:35:34.514 INFO:journalctl@ceph.osd.3.smithi092.stdout:Mar 11 21:35:34 smithi092 podman[126590]: 2024-03-11 21:35:34.2240602 +0000 UTC m=+0.480970785 container init f7cdbdab41f652f95a52058a5834d79428ac367a558a10ff69fb4011daf80df1 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:8fe9635f3beb0247496d7e321f3068bd10c333c433ef4824d7e1c1a7d3d78379, name=ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd-3-deactivate, org.label-schema.build-date=20240307, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, ceph=True, GIT_CLEAN=True, org.label-schema.license=GPLv2, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=main-2b0101e) 2024-03-11T21:35:34.515 INFO:journalctl@ceph.osd.3.smithi092.stdout:Mar 11 21:35:34 smithi092 podman[126590]: 2024-03-11 21:35:34.227155334 +0000 UTC m=+0.484065929 container start f7cdbdab41f652f95a52058a5834d79428ac367a558a10ff69fb4011daf80df1 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:8fe9635f3beb0247496d7e321f3068bd10c333c433ef4824d7e1c1a7d3d78379, name=ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd-3-deactivate, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, ceph=True, org.label-schema.build-date=20240307, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, RELEASE=main-2b0101e, GIT_CLEAN=True, io.buildah.version=1.33.5, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-03-11T21:35:34.515 INFO:journalctl@ceph.osd.3.smithi092.stdout:Mar 11 21:35:34 smithi092 podman[126590]: 2024-03-11 21:35:34.298715401 +0000 UTC m=+0.555625991 container attach f7cdbdab41f652f95a52058a5834d79428ac367a558a10ff69fb4011daf80df1 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:8fe9635f3beb0247496d7e321f3068bd10c333c433ef4824d7e1c1a7d3d78379, name=ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd-3-deactivate, GIT_BRANCH=HEAD, RELEASE=main-2b0101e, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, org.label-schema.build-date=20240307, org.label-schema.license=GPLv2, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, CEPH_POINT_RELEASE=, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5) 2024-03-11T21:35:34.539 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:35:34.562 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:35:34.563 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:35:34.563 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:35:34.664 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:35:34.688 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:35:34.688 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:35:34.689 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:35:34.722 DEBUG:teuthology.orchestra.run.smithi097:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid 00db0016-dfed-11ee-95c8-87774f69a715 -- ceph daemon osd.4 dump_historic_ops 2024-03-11T21:35:34.786 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:35:34 smithi097 ceph-mon[33317]: 5.5 scrub starts 2024-03-11T21:35:34.786 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:35:34 smithi097 ceph-mon[33317]: 5.5 scrub ok 2024-03-11T21:35:34.786 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:35:34 smithi097 ceph-mon[33317]: pgmap v445: 158 pgs: 20 active+undersized+degraded, 4 stale+active+clean, 1 active+remapped+backfill_toofull, 32 active+undersized, 101 active+clean; 582 KiB data, 2.9 GiB used, 712 GiB / 715 GiB avail; 0 B/s wr, 412 op/s; 85/594 objects degraded (14.310%) 2024-03-11T21:35:34.786 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:35:34 smithi097 ceph-mon[33317]: 2.4 scrub starts 2024-03-11T21:35:34.786 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:35:34 smithi097 ceph-mon[33317]: 2.4 scrub ok 2024-03-11T21:35:34.786 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:35:34 smithi097 ceph-mon[33317]: Health check failed: Degraded data redundancy: 85/594 objects degraded (14.310%), 20 pgs degraded (PG_DEGRADED) 2024-03-11T21:35:34.786 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:35:34 smithi097 ceph-mon[33317]: 4.1 scrub starts 2024-03-11T21:35:34.787 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:35:34 smithi097 ceph-mon[33317]: 4.1 scrub ok 2024-03-11T21:35:34.787 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:35:34 smithi097 ceph-mon[33317]: 2.5 scrub starts 2024-03-11T21:35:34.787 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:35:34 smithi097 ceph-mon[33317]: 2.5 scrub ok 2024-03-11T21:35:34.787 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:35:34 smithi097 ceph-mon[33317]: 4.a scrub starts 2024-03-11T21:35:34.787 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:35:34 smithi097 ceph-mon[33317]: 4.a scrub ok 2024-03-11T21:35:34.789 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:35:34.812 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:35:34.813 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:35:34.813 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:35:34.860 INFO:journalctl@ceph.osd.3.smithi092.stdout:Mar 11 21:35:34 smithi092 podman[126590]: 2024-03-11 21:35:34.534723931 +0000 UTC m=+0.791634517 container died f7cdbdab41f652f95a52058a5834d79428ac367a558a10ff69fb4011daf80df1 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:8fe9635f3beb0247496d7e321f3068bd10c333c433ef4824d7e1c1a7d3d78379, name=ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd-3-deactivate, GIT_BRANCH=HEAD, ceph=True, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240307, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, RELEASE=main-2b0101e, maintainer=Guillaume Abrioux , GIT_CLEAN=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5) 2024-03-11T21:35:34.861 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:35:34 smithi092 ceph-mon[29572]: 5.5 scrub starts 2024-03-11T21:35:34.861 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:35:34 smithi092 ceph-mon[29572]: 5.5 scrub ok 2024-03-11T21:35:34.861 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:35:34 smithi092 ceph-mon[29572]: pgmap v445: 158 pgs: 20 active+undersized+degraded, 4 stale+active+clean, 1 active+remapped+backfill_toofull, 32 active+undersized, 101 active+clean; 582 KiB data, 2.9 GiB used, 712 GiB / 715 GiB avail; 0 B/s wr, 412 op/s; 85/594 objects degraded (14.310%) 2024-03-11T21:35:34.861 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:35:34 smithi092 ceph-mon[29572]: 2.4 scrub starts 2024-03-11T21:35:34.861 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:35:34 smithi092 ceph-mon[29572]: 2.4 scrub ok 2024-03-11T21:35:34.861 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:35:34 smithi092 ceph-mon[29572]: Health check failed: Degraded data redundancy: 85/594 objects degraded (14.310%), 20 pgs degraded (PG_DEGRADED) 2024-03-11T21:35:34.861 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:35:34 smithi092 ceph-mon[29572]: 4.1 scrub starts 2024-03-11T21:35:34.862 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:35:34 smithi092 ceph-mon[29572]: 4.1 scrub ok 2024-03-11T21:35:34.862 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:35:34 smithi092 ceph-mon[29572]: 2.5 scrub starts 2024-03-11T21:35:34.862 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:35:34 smithi092 ceph-mon[29572]: 2.5 scrub ok 2024-03-11T21:35:34.862 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:35:34 smithi092 ceph-mon[29572]: 4.a scrub starts 2024-03-11T21:35:34.862 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:35:34 smithi092 ceph-mon[29572]: 4.a scrub ok 2024-03-11T21:35:34.862 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:35:34 smithi092 ceph-mon[37563]: 5.5 scrub starts 2024-03-11T21:35:34.863 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:35:34 smithi092 ceph-mon[37563]: 5.5 scrub ok 2024-03-11T21:35:34.863 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:35:34 smithi092 ceph-mon[37563]: pgmap v445: 158 pgs: 20 active+undersized+degraded, 4 stale+active+clean, 1 active+remapped+backfill_toofull, 32 active+undersized, 101 active+clean; 582 KiB data, 2.9 GiB used, 712 GiB / 715 GiB avail; 0 B/s wr, 412 op/s; 85/594 objects degraded (14.310%) 2024-03-11T21:35:34.863 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:35:34 smithi092 ceph-mon[37563]: 2.4 scrub starts 2024-03-11T21:35:34.863 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:35:34 smithi092 ceph-mon[37563]: 2.4 scrub ok 2024-03-11T21:35:34.863 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:35:34 smithi092 ceph-mon[37563]: Health check failed: Degraded data redundancy: 85/594 objects degraded (14.310%), 20 pgs degraded (PG_DEGRADED) 2024-03-11T21:35:34.863 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:35:34 smithi092 ceph-mon[37563]: 4.1 scrub starts 2024-03-11T21:35:34.863 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:35:34 smithi092 ceph-mon[37563]: 4.1 scrub ok 2024-03-11T21:35:34.863 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:35:34 smithi092 ceph-mon[37563]: 2.5 scrub starts 2024-03-11T21:35:34.863 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:35:34 smithi092 ceph-mon[37563]: 2.5 scrub ok 2024-03-11T21:35:34.864 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:35:34 smithi092 ceph-mon[37563]: 4.a scrub starts 2024-03-11T21:35:34.864 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:35:34 smithi092 ceph-mon[37563]: 4.a scrub ok 2024-03-11T21:35:34.914 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:35:34.939 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:35:34.940 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:35:34.941 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:35:35.003 INFO:teuthology.orchestra.run.smithi097.stderr:Inferring config /var/lib/ceph/00db0016-dfed-11ee-95c8-87774f69a715/mon.b/config 2024-03-11T21:35:35.042 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:35:35.066 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:35:35.067 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:35:35.067 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:35:35.131 DEBUG:teuthology.orchestra.run.smithi092:> sudo pkill -f 'journalctl -f -n 0 -u ceph-00db0016-dfed-11ee-95c8-87774f69a715@osd.3.service' 2024-03-11T21:35:35.168 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:35:35.182 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-11T21:35:35.182 INFO:tasks.cephadm.osd.3:Stopped osd.3 2024-03-11T21:35:35.182 INFO:tasks.thrashosds.thrasher:Testing ceph-objectstore-tool on down osd.3 2024-03-11T21:35:35.182 INFO:tasks.thrashosds.thrasher:remote for osd 3 is ubuntu@smithi092.front.sepia.ceph.com 2024-03-11T21:35:35.182 DEBUG:teuthology.orchestra.run.smithi092:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell -n osd.3 --fsid 00db0016-dfed-11ee-95c8-87774f69a715 -- ceph-objectstore-tool --err-to-stderr --no-mon-config '--log-file=/var/log/ceph/objectstore_tool.$pid.log' --data-path /var/lib/ceph/osd/ceph-3 --journal-path /var/lib/ceph/osd/ceph-3/journal --op list-pgs 2024-03-11T21:35:35.203 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:35:35.205 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:35:35.206 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:35:35.307 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:35:35.332 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:35:35.333 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:35:35.333 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:35:35.395 INFO:teuthology.orchestra.run.smithi092.stderr:Inferring config /var/lib/ceph/00db0016-dfed-11ee-95c8-87774f69a715/osd.3/config 2024-03-11T21:35:35.396 INFO:teuthology.orchestra.run.smithi092.stderr:Inferring config /var/lib/ceph/00db0016-dfed-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T21:35:35.434 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:35:35.438 INFO:teuthology.orchestra.run.smithi092.stderr:Inferring config /var/lib/ceph/00db0016-dfed-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T21:35:35.460 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:35:35.461 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:35:35.461 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:35:35.562 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:35:35.586 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:35:35.586 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:35:35.586 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:35:35.687 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:35:35.691 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:35:35 smithi092 ceph-mon[29572]: 4.e scrub starts 2024-03-11T21:35:35.691 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:35:35 smithi092 ceph-mon[29572]: 4.e scrub ok 2024-03-11T21:35:35.692 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:35:35 smithi092 ceph-mon[29572]: 2.8 scrub starts 2024-03-11T21:35:35.692 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:35:35 smithi092 ceph-mon[29572]: 2.8 scrub ok 2024-03-11T21:35:35.692 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:35:35 smithi092 ceph-mon[29572]: 8.12 scrub starts 2024-03-11T21:35:35.692 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:35:35 smithi092 ceph-mon[29572]: 8.12 scrub ok 2024-03-11T21:35:35.692 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:35:35 smithi092 ceph-mon[37563]: 4.e scrub starts 2024-03-11T21:35:35.692 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:35:35 smithi092 ceph-mon[37563]: 4.e scrub ok 2024-03-11T21:35:35.692 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:35:35 smithi092 ceph-mon[37563]: 2.8 scrub starts 2024-03-11T21:35:35.692 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:35:35 smithi092 ceph-mon[37563]: 2.8 scrub ok 2024-03-11T21:35:35.692 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:35:35 smithi092 ceph-mon[37563]: 8.12 scrub starts 2024-03-11T21:35:35.692 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:35:35 smithi092 ceph-mon[37563]: 8.12 scrub ok 2024-03-11T21:35:35.710 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:35:35.711 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:35:35.711 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:35:35.812 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:35:35.825 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:35:35 smithi097 ceph-mon[33317]: 4.e scrub starts 2024-03-11T21:35:35.825 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:35:35 smithi097 ceph-mon[33317]: 4.e scrub ok 2024-03-11T21:35:35.825 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:35:35 smithi097 ceph-mon[33317]: 2.8 scrub starts 2024-03-11T21:35:35.826 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:35:35 smithi097 ceph-mon[33317]: 2.8 scrub ok 2024-03-11T21:35:35.826 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:35:35 smithi097 ceph-mon[33317]: 8.12 scrub starts 2024-03-11T21:35:35.826 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:35:35 smithi097 ceph-mon[33317]: 8.12 scrub ok 2024-03-11T21:35:35.837 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:35:35.838 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:35:35.838 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:35:35.939 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:35:35.965 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:35:35.965 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:35:35.966 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:35:36.067 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:35:36.090 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:35:36.091 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:35:36.091 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:35:36.192 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:35:36.217 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:35:36.218 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:35:36.218 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:35:36.319 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:35:36.343 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:35:36.343 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:35:36.344 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:35:36.444 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:35:36.468 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:35:36.469 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:35:36.469 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:35:36.570 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:35:36.593 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:35:36.594 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:35:36.594 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:35:36.640 INFO:teuthology.orchestra.run.smithi092.stderr:pool 'unique_pool_1' removed 2024-03-11T21:35:36.695 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:35:36.722 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:35:36.722 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:35:36.723 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:35:36.752 INFO:teuthology.orchestra.run.smithi097.stdout:ERROR: (22) Invalid argument 2024-03-11T21:35:36.752 INFO:teuthology.orchestra.run.smithi097.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-11T21:35:36.766 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:35:36 smithi097 ceph-mon[33317]: pgmap v447: 158 pgs: 1 active+remapped+backfill_toofull, 22 active+undersized+degraded, 41 active+undersized, 94 active+clean; 582 KiB data, 2.9 GiB used, 712 GiB / 715 GiB avail; 94/594 objects degraded (15.825%) 2024-03-11T21:35:36.766 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:35:36 smithi097 ceph-mon[33317]: 8.2 scrub starts 2024-03-11T21:35:36.766 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:35:36 smithi097 ceph-mon[33317]: 8.2 scrub ok 2024-03-11T21:35:36.766 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:35:36 smithi097 ceph-mon[33317]: from='client.? v1:172.21.15.92:0/272807214' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "unique_pool_1", "pool2": "unique_pool_1", "yes_i_really_really_mean_it": true}]: dispatch 2024-03-11T21:35:36.766 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:35:36 smithi097 ceph-mon[33317]: purged_snaps scrub starts 2024-03-11T21:35:36.766 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:35:36 smithi097 ceph-mon[33317]: purged_snaps scrub ok 2024-03-11T21:35:36.824 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:35:36.847 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:35:36.848 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:35:36.848 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:35:36.949 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:35:36.972 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:35:36.973 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:35:36.973 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:35:36.993 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:35:36 smithi092 ceph-mon[29572]: pgmap v447: 158 pgs: 1 active+remapped+backfill_toofull, 22 active+undersized+degraded, 41 active+undersized, 94 active+clean; 582 KiB data, 2.9 GiB used, 712 GiB / 715 GiB avail; 94/594 objects degraded (15.825%) 2024-03-11T21:35:36.993 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:35:36 smithi092 ceph-mon[29572]: 8.2 scrub starts 2024-03-11T21:35:36.993 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:35:36 smithi092 ceph-mon[29572]: 8.2 scrub ok 2024-03-11T21:35:36.993 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:35:36 smithi092 ceph-mon[29572]: from='client.? v1:172.21.15.92:0/272807214' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "unique_pool_1", "pool2": "unique_pool_1", "yes_i_really_really_mean_it": true}]: dispatch 2024-03-11T21:35:36.994 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:35:36 smithi092 ceph-mon[29572]: purged_snaps scrub starts 2024-03-11T21:35:36.994 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:35:36 smithi092 ceph-mon[29572]: purged_snaps scrub ok 2024-03-11T21:35:36.994 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:35:36 smithi092 ceph-mon[37563]: pgmap v447: 158 pgs: 1 active+remapped+backfill_toofull, 22 active+undersized+degraded, 41 active+undersized, 94 active+clean; 582 KiB data, 2.9 GiB used, 712 GiB / 715 GiB avail; 94/594 objects degraded (15.825%) 2024-03-11T21:35:36.994 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:35:36 smithi092 ceph-mon[37563]: 8.2 scrub starts 2024-03-11T21:35:36.994 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:35:36 smithi092 ceph-mon[37563]: 8.2 scrub ok 2024-03-11T21:35:36.994 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:35:36 smithi092 ceph-mon[37563]: from='client.? v1:172.21.15.92:0/272807214' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "unique_pool_1", "pool2": "unique_pool_1", "yes_i_really_really_mean_it": true}]: dispatch 2024-03-11T21:35:36.994 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:35:36 smithi092 ceph-mon[37563]: purged_snaps scrub starts 2024-03-11T21:35:36.994 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:35:36 smithi092 ceph-mon[37563]: purged_snaps scrub ok 2024-03-11T21:35:37.074 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:35:37.098 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:35:37.099 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:35:37.099 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:35:37.200 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:35:37.224 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:35:37.225 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:35:37.225 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:35:37.326 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:35:37.349 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:35:37.350 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:35:37.350 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:35:37.394 INFO:teuthology.task.full_sequential:In full_sequential, running task radosbench... 2024-03-11T21:35:37.394 INFO:tasks.radosbench:Beginning radosbench... 2024-03-11T21:35:37.395 INFO:tasks.cephadm.ceph_manager.ceph:creating pool_name unique_pool_2 2024-03-11T21:35:37.395 DEBUG:teuthology.orchestra.run.smithi092:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid 00db0016-dfed-11ee-95c8-87774f69a715 -- ceph osd pool create unique_pool_2 16 2024-03-11T21:35:37.446 DEBUG:teuthology.orchestra.run.smithi097:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid 00db0016-dfed-11ee-95c8-87774f69a715 -- ceph daemon osd.5 dump_ops_in_flight 2024-03-11T21:35:37.451 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:35:37.474 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:35:37.474 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:35:37.475 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:35:37.575 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:35:37.601 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:35:37.601 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:35:37.602 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:35:37.701 INFO:teuthology.orchestra.run.smithi097.stderr:Inferring config /var/lib/ceph/00db0016-dfed-11ee-95c8-87774f69a715/mon.b/config 2024-03-11T21:35:37.703 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:35:37.728 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:35:37.728 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:35:37.728 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:35:37.823 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:35:37 smithi097 ceph-mon[33317]: 2.f scrub starts 2024-03-11T21:35:37.823 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:35:37 smithi097 ceph-mon[33317]: 2.f scrub ok 2024-03-11T21:35:37.823 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:35:37 smithi097 ceph-mon[33317]: from='client.? v1:172.21.15.92:0/272807214' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "unique_pool_1", "pool2": "unique_pool_1", "yes_i_really_really_mean_it": true}]': finished 2024-03-11T21:35:37.823 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:35:37 smithi097 ceph-mon[33317]: osdmap e255: 8 total, 7 up, 8 in 2024-03-11T21:35:37.824 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:35:37 smithi097 ceph-mon[33317]: from='mgr.14607 ' entity='mgr.y' 2024-03-11T21:35:37.824 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:35:37 smithi097 ceph-mon[33317]: from='mgr.14607 v1:172.21.15.92:0/1702717597' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T21:35:37.829 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:35:37.853 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:35:37.854 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:35:37.854 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:35:37.923 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:35:37 smithi092 ceph-mon[29572]: 2.f scrub starts 2024-03-11T21:35:37.923 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:35:37 smithi092 ceph-mon[29572]: 2.f scrub ok 2024-03-11T21:35:37.923 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:35:37 smithi092 ceph-mon[29572]: from='client.? v1:172.21.15.92:0/272807214' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "unique_pool_1", "pool2": "unique_pool_1", "yes_i_really_really_mean_it": true}]': finished 2024-03-11T21:35:37.923 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:35:37 smithi092 ceph-mon[29572]: osdmap e255: 8 total, 7 up, 8 in 2024-03-11T21:35:37.923 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:35:37 smithi092 ceph-mon[29572]: from='mgr.14607 ' entity='mgr.y' 2024-03-11T21:35:37.923 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:35:37 smithi092 ceph-mon[29572]: from='mgr.14607 v1:172.21.15.92:0/1702717597' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T21:35:37.923 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:35:37 smithi092 ceph-mon[37563]: 2.f scrub starts 2024-03-11T21:35:37.923 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:35:37 smithi092 ceph-mon[37563]: 2.f scrub ok 2024-03-11T21:35:37.924 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:35:37 smithi092 ceph-mon[37563]: from='client.? v1:172.21.15.92:0/272807214' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "unique_pool_1", "pool2": "unique_pool_1", "yes_i_really_really_mean_it": true}]': finished 2024-03-11T21:35:37.924 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:35:37 smithi092 ceph-mon[37563]: osdmap e255: 8 total, 7 up, 8 in 2024-03-11T21:35:37.924 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:35:37 smithi092 ceph-mon[37563]: from='mgr.14607 ' entity='mgr.y' 2024-03-11T21:35:37.924 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:35:37 smithi092 ceph-mon[37563]: from='mgr.14607 v1:172.21.15.92:0/1702717597' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T21:35:37.955 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:35:37.979 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:35:37.981 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:35:37.981 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:35:38.082 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:35:38.107 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:35:38.107 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:35:38.107 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:35:38.208 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:35:38.233 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:35:38.233 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:35:38.234 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:35:38.334 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:35:38.358 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:35:38.359 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:35:38.359 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:35:38.360 INFO:journalctl@ceph.mgr.y.smithi092.stdout:Mar 11 21:35:37 smithi092 ceph-00db0016-dfed-11ee-95c8-87774f69a715-mgr-y[29801]: ::ffff:172.21.15.97 - - [11/Mar/2024:21:35:37] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-11T21:35:38.460 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:35:38.486 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:35:38.487 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:35:38.487 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:35:38.588 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:35:38.613 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:35:38.613 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:35:38.613 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:35:38.714 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:35:38.739 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:35:38.739 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:35:38.740 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:35:38.826 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:35:38 smithi097 ceph-mon[33317]: 6.d deep-scrub starts 2024-03-11T21:35:38.826 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:35:38 smithi097 ceph-mon[33317]: pgmap v449: 132 pgs: 22 active+undersized+degraded, 32 active+undersized, 78 active+clean; 582 KiB data, 2.9 GiB used, 712 GiB / 715 GiB avail; 94/594 objects degraded (15.825%) 2024-03-11T21:35:38.826 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:35:38 smithi097 ceph-mon[33317]: 6.d deep-scrub ok 2024-03-11T21:35:38.826 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:35:38 smithi097 ceph-mon[33317]: 4.7 scrub starts 2024-03-11T21:35:38.826 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:35:38 smithi097 ceph-mon[33317]: 4.7 scrub ok 2024-03-11T21:35:38.826 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:35:38 smithi097 ceph-mon[33317]: 6.9 scrub starts 2024-03-11T21:35:38.826 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:35:38 smithi097 ceph-mon[33317]: 6.9 scrub ok 2024-03-11T21:35:38.826 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:35:38 smithi097 ceph-mon[33317]: 2.10 scrub starts 2024-03-11T21:35:38.826 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:35:38 smithi097 ceph-mon[33317]: 2.10 scrub ok 2024-03-11T21:35:38.827 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:35:38 smithi097 ceph-mon[33317]: Health check cleared: PG_BACKFILL_FULL (was: Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull) 2024-03-11T21:35:38.827 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:35:38 smithi097 ceph-mon[33317]: 4.1f deep-scrub starts 2024-03-11T21:35:38.827 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:35:38 smithi097 ceph-mon[33317]: 4.1f deep-scrub ok 2024-03-11T21:35:38.827 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:35:38 smithi097 ceph-mon[33317]: osdmap e256: 8 total, 7 up, 8 in 2024-03-11T21:35:38.840 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:35:38.860 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:35:38 smithi092 ceph-mon[29572]: 6.d deep-scrub starts 2024-03-11T21:35:38.860 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:35:38 smithi092 ceph-mon[29572]: pgmap v449: 132 pgs: 22 active+undersized+degraded, 32 active+undersized, 78 active+clean; 582 KiB data, 2.9 GiB used, 712 GiB / 715 GiB avail; 94/594 objects degraded (15.825%) 2024-03-11T21:35:38.860 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:35:38 smithi092 ceph-mon[29572]: 6.d deep-scrub ok 2024-03-11T21:35:38.860 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:35:38 smithi092 ceph-mon[29572]: 4.7 scrub starts 2024-03-11T21:35:38.860 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:35:38 smithi092 ceph-mon[29572]: 4.7 scrub ok 2024-03-11T21:35:38.860 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:35:38 smithi092 ceph-mon[29572]: 6.9 scrub starts 2024-03-11T21:35:38.860 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:35:38 smithi092 ceph-mon[29572]: 6.9 scrub ok 2024-03-11T21:35:38.860 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:35:38 smithi092 ceph-mon[29572]: 2.10 scrub starts 2024-03-11T21:35:38.860 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:35:38 smithi092 ceph-mon[29572]: 2.10 scrub ok 2024-03-11T21:35:38.860 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:35:38 smithi092 ceph-mon[29572]: Health check cleared: PG_BACKFILL_FULL (was: Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull) 2024-03-11T21:35:38.860 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:35:38 smithi092 ceph-mon[29572]: 4.1f deep-scrub starts 2024-03-11T21:35:38.861 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:35:38 smithi092 ceph-mon[29572]: 4.1f deep-scrub ok 2024-03-11T21:35:38.861 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:35:38 smithi092 ceph-mon[29572]: osdmap e256: 8 total, 7 up, 8 in 2024-03-11T21:35:38.861 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:35:38 smithi092 ceph-mon[37563]: 6.d deep-scrub starts 2024-03-11T21:35:38.861 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:35:38 smithi092 ceph-mon[37563]: pgmap v449: 132 pgs: 22 active+undersized+degraded, 32 active+undersized, 78 active+clean; 582 KiB data, 2.9 GiB used, 712 GiB / 715 GiB avail; 94/594 objects degraded (15.825%) 2024-03-11T21:35:38.861 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:35:38 smithi092 ceph-mon[37563]: 6.d deep-scrub ok 2024-03-11T21:35:38.861 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:35:38 smithi092 ceph-mon[37563]: 4.7 scrub starts 2024-03-11T21:35:38.861 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:35:38 smithi092 ceph-mon[37563]: 4.7 scrub ok 2024-03-11T21:35:38.861 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:35:38 smithi092 ceph-mon[37563]: 6.9 scrub starts 2024-03-11T21:35:38.861 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:35:38 smithi092 ceph-mon[37563]: 6.9 scrub ok 2024-03-11T21:35:38.861 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:35:38 smithi092 ceph-mon[37563]: 2.10 scrub starts 2024-03-11T21:35:38.861 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:35:38 smithi092 ceph-mon[37563]: 2.10 scrub ok 2024-03-11T21:35:38.861 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:35:38 smithi092 ceph-mon[37563]: Health check cleared: PG_BACKFILL_FULL (was: Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull) 2024-03-11T21:35:38.861 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:35:38 smithi092 ceph-mon[37563]: 4.1f deep-scrub starts 2024-03-11T21:35:38.862 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:35:38 smithi092 ceph-mon[37563]: 4.1f deep-scrub ok 2024-03-11T21:35:38.862 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:35:38 smithi092 ceph-mon[37563]: osdmap e256: 8 total, 7 up, 8 in 2024-03-11T21:35:38.864 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:35:38.865 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:35:38.865 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:35:38.966 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:35:38.990 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:35:38.991 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:35:38.991 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:35:39.092 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:35:39.116 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:35:39.116 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:35:39.117 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:35:39.218 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:35:39.241 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:35:39.242 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:35:39.242 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:35:39.343 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:35:39.367 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:35:39.367 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:35:39.368 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:35:39.400 INFO:teuthology.orchestra.run.smithi092.stderr:Creating an OSD daemon form without an OSD FSID value 2024-03-11T21:35:39.465 INFO:teuthology.orchestra.run.smithi097.stdout:ERROR: (22) Invalid argument 2024-03-11T21:35:39.465 INFO:teuthology.orchestra.run.smithi097.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-11T21:35:39.469 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:35:39.494 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:35:39.495 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:35:39.495 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:35:39.596 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:35:39.620 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:35:39.621 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:35:39.621 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:35:39.722 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:35:39.756 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:35:39.757 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:35:39.757 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:35:39.826 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:35:39 smithi097 ceph-mon[33317]: 3.0 scrub starts 2024-03-11T21:35:39.826 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:35:39 smithi097 ceph-mon[33317]: 3.0 scrub ok 2024-03-11T21:35:39.858 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:35:39.882 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:35:39.883 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:35:39.883 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:35:39.919 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:35:39 smithi092 ceph-mon[29572]: 3.0 scrub starts 2024-03-11T21:35:39.919 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:35:39 smithi092 ceph-mon[29572]: 3.0 scrub ok 2024-03-11T21:35:39.919 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:35:39 smithi092 ceph-mon[37563]: 3.0 scrub starts 2024-03-11T21:35:39.920 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:35:39 smithi092 ceph-mon[37563]: 3.0 scrub ok 2024-03-11T21:35:39.984 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:35:40.007 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:35:40.008 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:35:40.009 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:35:40.070 DEBUG:teuthology.orchestra.run.smithi097:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid 00db0016-dfed-11ee-95c8-87774f69a715 -- ceph daemon osd.5 dump_blocked_ops 2024-03-11T21:35:40.110 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:35:40.135 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:35:40.135 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:35:40.136 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:35:40.155 INFO:teuthology.orchestra.run.smithi092.stdout:osd.0: {} 2024-03-11T21:35:40.174 INFO:teuthology.orchestra.run.smithi092.stdout:osd.1: {} 2024-03-11T21:35:40.194 INFO:teuthology.orchestra.run.smithi092.stdout:osd.2: {} 2024-03-11T21:35:40.214 INFO:teuthology.orchestra.run.smithi092.stdout:osd.4: {} 2024-03-11T21:35:40.233 INFO:teuthology.orchestra.run.smithi092.stdout:osd.5: {} 2024-03-11T21:35:40.237 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:35:40.253 INFO:teuthology.orchestra.run.smithi092.stdout:osd.6: {} 2024-03-11T21:35:40.264 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:35:40.265 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:35:40.266 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:35:40.273 INFO:teuthology.orchestra.run.smithi092.stdout:osd.7: {} 2024-03-11T21:35:40.273 INFO:teuthology.orchestra.run.smithi092.stderr:osd.0: osd_enable_op_tracker = '' 2024-03-11T21:35:40.273 INFO:teuthology.orchestra.run.smithi092.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-11T21:35:40.273 INFO:teuthology.orchestra.run.smithi092.stderr:osd.2: osd_enable_op_tracker = '' 2024-03-11T21:35:40.273 INFO:teuthology.orchestra.run.smithi092.stderr:Error ENXIO: problem getting command descriptions from osd.3 2024-03-11T21:35:40.274 INFO:teuthology.orchestra.run.smithi092.stderr:osd.4: osd_enable_op_tracker = '' 2024-03-11T21:35:40.274 INFO:teuthology.orchestra.run.smithi092.stderr:osd.5: osd_enable_op_tracker = '' 2024-03-11T21:35:40.274 INFO:teuthology.orchestra.run.smithi092.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-11T21:35:40.274 INFO:teuthology.orchestra.run.smithi092.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-11T21:35:40.313 INFO:teuthology.orchestra.run.smithi097.stderr:Inferring config /var/lib/ceph/00db0016-dfed-11ee-95c8-87774f69a715/mon.b/config 2024-03-11T21:35:40.366 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:35:40.390 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:35:40.390 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:35:40.391 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:35:40.473 INFO:teuthology.orchestra.run.smithi092.stderr:Inferring config /var/lib/ceph/00db0016-dfed-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T21:35:40.491 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:35:40.515 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:35:40.515 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:35:40.516 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:35:40.617 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:35:40.642 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:35:40.643 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:35:40.643 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:35:40.649 INFO:teuthology.orchestra.run.smithi092.stderr:noscrub is set 2024-03-11T21:35:40.696 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:35:40 smithi097 ceph-mon[33317]: 2.16 scrub starts 2024-03-11T21:35:40.697 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:35:40 smithi097 ceph-mon[33317]: 2.16 scrub ok 2024-03-11T21:35:40.697 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:35:40 smithi097 ceph-mon[33317]: pgmap v451: 132 pgs: 22 active+undersized+degraded, 32 active+undersized, 78 active+clean; 582 KiB data, 2.9 GiB used, 712 GiB / 715 GiB avail; 682 B/s rd, 0 op/s; 94/594 objects degraded (15.825%) 2024-03-11T21:35:40.697 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:35:40 smithi097 ceph-mon[33317]: 6.18 scrub starts 2024-03-11T21:35:40.697 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:35:40 smithi097 ceph-mon[33317]: 6.18 scrub ok 2024-03-11T21:35:40.697 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:35:40 smithi097 ceph-mon[33317]: from='client.? v1:172.21.15.92:0/3844674442' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-11T21:35:40.697 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:35:40 smithi097 ceph-mon[33317]: from='client.? v1:172.21.15.92:0/3347510553' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-11T21:35:40.744 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:35:40.768 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:35:40.769 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:35:40.769 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:35:40.860 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:35:40 smithi092 ceph-mon[29572]: 2.16 scrub starts 2024-03-11T21:35:40.860 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:35:40 smithi092 ceph-mon[29572]: 2.16 scrub ok 2024-03-11T21:35:40.860 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:35:40 smithi092 ceph-mon[29572]: pgmap v451: 132 pgs: 22 active+undersized+degraded, 32 active+undersized, 78 active+clean; 582 KiB data, 2.9 GiB used, 712 GiB / 715 GiB avail; 682 B/s rd, 0 op/s; 94/594 objects degraded (15.825%) 2024-03-11T21:35:40.860 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:35:40 smithi092 ceph-mon[29572]: 6.18 scrub starts 2024-03-11T21:35:40.860 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:35:40 smithi092 ceph-mon[29572]: 6.18 scrub ok 2024-03-11T21:35:40.860 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:35:40 smithi092 ceph-mon[29572]: from='client.? v1:172.21.15.92:0/3844674442' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-11T21:35:40.860 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:35:40 smithi092 ceph-mon[29572]: from='client.? v1:172.21.15.92:0/3347510553' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-11T21:35:40.860 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:35:40 smithi092 ceph-mon[37563]: 2.16 scrub starts 2024-03-11T21:35:40.860 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:35:40 smithi092 ceph-mon[37563]: 2.16 scrub ok 2024-03-11T21:35:40.860 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:35:40 smithi092 ceph-mon[37563]: pgmap v451: 132 pgs: 22 active+undersized+degraded, 32 active+undersized, 78 active+clean; 582 KiB data, 2.9 GiB used, 712 GiB / 715 GiB avail; 682 B/s rd, 0 op/s; 94/594 objects degraded (15.825%) 2024-03-11T21:35:40.860 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:35:40 smithi092 ceph-mon[37563]: 6.18 scrub starts 2024-03-11T21:35:40.861 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:35:40 smithi092 ceph-mon[37563]: 6.18 scrub ok 2024-03-11T21:35:40.861 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:35:40 smithi092 ceph-mon[37563]: from='client.? v1:172.21.15.92:0/3844674442' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-11T21:35:40.861 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:35:40 smithi092 ceph-mon[37563]: from='client.? v1:172.21.15.92:0/3347510553' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-11T21:35:40.870 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:35:40.895 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:35:40.895 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:35:40.896 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:35:40.988 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-03-11T21:35:40.989 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-03-11T21:35:40.997 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:35:41.020 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:35:41.021 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:35:41.021 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:35:41.122 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:35:41.147 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:35:41.147 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:35:41.148 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:35:41.248 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:35:41.273 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:35:41.273 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:35:41.274 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:35:41.375 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:35:41.400 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:35:41.400 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:35:41.401 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:35:41.501 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:35:41.526 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:35:41.527 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:35:41.527 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:35:41.628 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:35:41.652 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:35:41.652 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:35:41.653 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:35:41.745 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:35:41 smithi097 ceph-mon[33317]: 2.2 scrub starts 2024-03-11T21:35:41.745 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:35:41 smithi097 ceph-mon[33317]: 2.2 scrub ok 2024-03-11T21:35:41.745 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:35:41 smithi097 ceph-mon[33317]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-03-11T21:35:41.746 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:35:41 smithi097 ceph-mon[33317]: from='client.? v1:172.21.15.92:0/3347510553' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-03-11T21:35:41.746 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:35:41 smithi097 ceph-mon[33317]: osdmap e257: 8 total, 7 up, 8 in 2024-03-11T21:35:41.754 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:35:41.777 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:35:41.778 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:35:41.779 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:35:41.860 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:35:41 smithi092 ceph-mon[29572]: 2.2 scrub starts 2024-03-11T21:35:41.860 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:35:41 smithi092 ceph-mon[29572]: 2.2 scrub ok 2024-03-11T21:35:41.860 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:35:41 smithi092 ceph-mon[29572]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-03-11T21:35:41.860 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:35:41 smithi092 ceph-mon[29572]: from='client.? v1:172.21.15.92:0/3347510553' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-03-11T21:35:41.860 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:35:41 smithi092 ceph-mon[29572]: osdmap e257: 8 total, 7 up, 8 in 2024-03-11T21:35:41.860 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:35:41 smithi092 ceph-mon[37563]: 2.2 scrub starts 2024-03-11T21:35:41.860 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:35:41 smithi092 ceph-mon[37563]: 2.2 scrub ok 2024-03-11T21:35:41.861 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:35:41 smithi092 ceph-mon[37563]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-03-11T21:35:41.861 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:35:41 smithi092 ceph-mon[37563]: from='client.? v1:172.21.15.92:0/3347510553' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-03-11T21:35:41.861 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:35:41 smithi092 ceph-mon[37563]: osdmap e257: 8 total, 7 up, 8 in 2024-03-11T21:35:41.879 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:35:41.903 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:35:41.903 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:35:41.904 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:35:42.003 INFO:teuthology.orchestra.run.smithi097.stdout:{ 2024-03-11T21:35:42.003 INFO:teuthology.orchestra.run.smithi097.stdout: "ops": [], 2024-03-11T21:35:42.003 INFO:teuthology.orchestra.run.smithi097.stdout: "complaint_time": 30, 2024-03-11T21:35:42.004 INFO:teuthology.orchestra.run.smithi097.stdout: "num_blocked_ops": 0 2024-03-11T21:35:42.004 INFO:teuthology.orchestra.run.smithi097.stdout:} 2024-03-11T21:35:42.005 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:35:42.029 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:35:42.030 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:35:42.030 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:35:42.131 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:35:42.158 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:35:42.158 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:35:42.159 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:35:42.260 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:35:42.284 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:35:42.285 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:35:42.285 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:35:42.386 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:35:42.409 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:35:42.410 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:35:42.410 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:35:42.511 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:35:42.536 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:35:42.536 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:35:42.537 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:35:42.638 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:35:42.646 INFO:journalctl@ceph.iscsi.iscsi.a.smithi097.stdout:Mar 11 21:35:42 smithi097 ceph-00db0016-dfed-11ee-95c8-87774f69a715-iscsi-iscsi-a[60943]: debug there is no tcmu-runner data available 2024-03-11T21:35:42.662 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:35:42.663 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:35:42.663 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:35:42.721 DEBUG:teuthology.orchestra.run.smithi097:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid 00db0016-dfed-11ee-95c8-87774f69a715 -- ceph daemon osd.5 dump_historic_ops 2024-03-11T21:35:42.764 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:35:42.787 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:35:42.788 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:35:42.789 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:35:42.859 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:35:42 smithi092 ceph-mon[29572]: pgmap v453: 132 pgs: 22 active+undersized+degraded, 32 active+undersized, 78 active+clean; 582 KiB data, 2.9 GiB used, 712 GiB / 715 GiB avail; 682 B/s rd, 0 op/s; 94/594 objects degraded (15.825%) 2024-03-11T21:35:42.860 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:35:42 smithi092 ceph-mon[37563]: pgmap v453: 132 pgs: 22 active+undersized+degraded, 32 active+undersized, 78 active+clean; 582 KiB data, 2.9 GiB used, 712 GiB / 715 GiB avail; 682 B/s rd, 0 op/s; 94/594 objects degraded (15.825%) 2024-03-11T21:35:42.889 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:35:42.914 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:35:42.914 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:35:42.915 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:35:42.962 INFO:teuthology.orchestra.run.smithi097.stderr:Inferring config /var/lib/ceph/00db0016-dfed-11ee-95c8-87774f69a715/mon.b/config 2024-03-11T21:35:42.990 DEBUG:teuthology.orchestra.run.smithi092:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid 00db0016-dfed-11ee-95c8-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-03-11T21:35:43.016 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:35:43.058 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:35:43.058 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:35:43.059 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:35:43.075 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:35:42 smithi097 ceph-mon[33317]: pgmap v453: 132 pgs: 22 active+undersized+degraded, 32 active+undersized, 78 active+clean; 582 KiB data, 2.9 GiB used, 712 GiB / 715 GiB avail; 682 B/s rd, 0 op/s; 94/594 objects degraded (15.825%) 2024-03-11T21:35:43.159 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:35:43.183 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:35:43.184 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:35:43.184 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:35:43.285 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:35:43.312 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:35:43.312 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:35:43.313 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:35:43.364 INFO:teuthology.orchestra.run.smithi092.stderr:Inferring config /var/lib/ceph/00db0016-dfed-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T21:35:43.414 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:35:43.439 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:35:43.439 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:35:43.439 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:35:43.540 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:35:43.564 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:35:43.564 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:35:43.565 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:35:43.594 DEBUG:teuthology.orchestra.run.smithi092:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid 00db0016-dfed-11ee-95c8-87774f69a715 -- ceph osd set nodeep-scrub 2024-03-11T21:35:43.666 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:35:43.690 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:35:43.690 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:35:43.691 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:35:43.792 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:35:43.815 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:35:43.816 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:35:43.816 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:35:43.859 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:35:43 smithi092 ceph-mon[37563]: from='client.24469 v1:172.21.15.97:0/404208310' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:35:43.860 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:35:43 smithi092 ceph-mon[37563]: Health check update: Degraded data redundancy: 94/594 objects degraded (15.825%), 22 pgs degraded (PG_DEGRADED) 2024-03-11T21:35:43.860 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:35:43 smithi092 ceph-mon[37563]: osdmap e258: 8 total, 7 up, 8 in 2024-03-11T21:35:43.860 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:35:43 smithi092 ceph-mon[29572]: from='client.24469 v1:172.21.15.97:0/404208310' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:35:43.860 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:35:43 smithi092 ceph-mon[29572]: Health check update: Degraded data redundancy: 94/594 objects degraded (15.825%), 22 pgs degraded (PG_DEGRADED) 2024-03-11T21:35:43.860 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:35:43 smithi092 ceph-mon[29572]: osdmap e258: 8 total, 7 up, 8 in 2024-03-11T21:35:43.918 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:35:43.942 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:35:43.943 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:35:43.944 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:35:43.952 INFO:teuthology.orchestra.run.smithi092.stderr:Inferring config /var/lib/ceph/00db0016-dfed-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T21:35:44.044 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:35:44.049 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:35:43 smithi097 ceph-mon[33317]: from='client.24469 v1:172.21.15.97:0/404208310' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:35:44.049 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:35:43 smithi097 ceph-mon[33317]: Health check update: Degraded data redundancy: 94/594 objects degraded (15.825%), 22 pgs degraded (PG_DEGRADED) 2024-03-11T21:35:44.049 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:35:43 smithi097 ceph-mon[33317]: osdmap e258: 8 total, 7 up, 8 in 2024-03-11T21:35:44.069 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:35:44.069 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:35:44.070 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:35:44.170 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:35:44.196 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:35:44.196 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:35:44.197 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:35:44.298 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:35:44.322 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:35:44.323 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:35:44.323 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:35:44.424 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:35:44.448 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:35:44.449 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:35:44.449 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:35:44.550 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:35:44.574 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:35:44.574 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:35:44.575 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:35:44.627 INFO:teuthology.orchestra.run.smithi097.stdout:{ 2024-03-11T21:35:44.627 INFO:teuthology.orchestra.run.smithi097.stdout: "size": 20, 2024-03-11T21:35:44.628 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 600, 2024-03-11T21:35:44.628 INFO:teuthology.orchestra.run.smithi097.stdout: "ops": [ 2024-03-11T21:35:44.628 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:35:44.628 INFO:teuthology.orchestra.run.smithi097.stdout: "description": "osd_op(client.15150.0:206884 8.8 8:168db5dc:::benchmark_data_smithi092_104865_object86383:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e236)", 2024-03-11T21:35:44.628 INFO:teuthology.orchestra.run.smithi097.stdout: "initiated_at": "2024-03-11T21:34:54.365485+0000", 2024-03-11T21:35:44.628 INFO:teuthology.orchestra.run.smithi097.stdout: "age": 50.278104452999997, 2024-03-11T21:35:44.628 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.99267883099999998, 2024-03-11T21:35:44.628 INFO:teuthology.orchestra.run.smithi097.stdout: "type_data": { 2024-03-11T21:35:44.628 INFO:teuthology.orchestra.run.smithi097.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-11T21:35:44.628 INFO:teuthology.orchestra.run.smithi097.stdout: "client_info": { 2024-03-11T21:35:44.628 INFO:teuthology.orchestra.run.smithi097.stdout: "client": "client.15150", 2024-03-11T21:35:44.628 INFO:teuthology.orchestra.run.smithi097.stdout: "client_addr": "v1:172.21.15.92:0/436606717", 2024-03-11T21:35:44.628 INFO:teuthology.orchestra.run.smithi097.stdout: "tid": 206884 2024-03-11T21:35:44.629 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:35:44.629 INFO:teuthology.orchestra.run.smithi097.stdout: "events": [ 2024-03-11T21:35:44.629 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:35:44.629 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "initiated", 2024-03-11T21:35:44.629 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:34:54.365485+0000", 2024-03-11T21:35:44.629 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-03-11T21:35:44.629 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:35:44.629 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:35:44.629 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "header_read", 2024-03-11T21:35:44.629 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:34:54.365485+0000", 2024-03-11T21:35:44.629 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-03-11T21:35:44.629 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:35:44.629 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:35:44.629 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "throttled", 2024-03-11T21:35:44.630 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:34:54.365486+0000", 2024-03-11T21:35:44.630 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.4160000000000001e-06 2024-03-11T21:35:44.630 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:35:44.630 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:35:44.630 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "all_read", 2024-03-11T21:35:44.630 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:34:54.365490+0000", 2024-03-11T21:35:44.630 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 4.0559999999999998e-06 2024-03-11T21:35:44.630 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:35:44.630 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:35:44.630 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "dispatched", 2024-03-11T21:35:44.630 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:34:54.365499+0000", 2024-03-11T21:35:44.630 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 8.7469999999999994e-06 2024-03-11T21:35:44.630 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:35:44.631 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:35:44.631 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "queued_for_pg", 2024-03-11T21:35:44.631 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:34:54.365504+0000", 2024-03-11T21:35:44.631 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 5.446e-06 2024-03-11T21:35:44.631 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:35:44.631 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:35:44.631 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "reached_pg", 2024-03-11T21:35:44.631 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:34:54.365546+0000", 2024-03-11T21:35:44.631 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 4.1978999999999999e-05 2024-03-11T21:35:44.631 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:35:44.631 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:35:44.631 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "waiting for peered", 2024-03-11T21:35:44.631 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:34:54.365552+0000", 2024-03-11T21:35:44.632 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 6.2720000000000003e-06 2024-03-11T21:35:44.632 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:35:44.632 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:35:44.632 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "reached_pg", 2024-03-11T21:35:44.632 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:34:55.351250+0000", 2024-03-11T21:35:44.632 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.98569733100000001 2024-03-11T21:35:44.632 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:35:44.632 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:35:44.632 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "started", 2024-03-11T21:35:44.632 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:34:55.351327+0000", 2024-03-11T21:35:44.632 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 7.6753000000000002e-05 2024-03-11T21:35:44.632 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:35:44.632 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:35:44.632 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "waiting for subops from 3,7", 2024-03-11T21:35:44.632 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:34:55.351685+0000", 2024-03-11T21:35:44.633 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.00035823799999999998 2024-03-11T21:35:44.633 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:35:44.633 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:35:44.633 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "sub_op_commit_rec", 2024-03-11T21:35:44.633 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:34:55.354118+0000", 2024-03-11T21:35:44.633 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.0024329680000000002 2024-03-11T21:35:44.633 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:35:44.633 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:35:44.633 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "op_commit", 2024-03-11T21:35:44.633 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:34:55.356229+0000", 2024-03-11T21:35:44.633 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.0021114549999999999 2024-03-11T21:35:44.633 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:35:44.633 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:35:44.633 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "sub_op_commit_rec", 2024-03-11T21:35:44.634 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:34:55.358065+0000", 2024-03-11T21:35:44.634 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.001836166 2024-03-11T21:35:44.634 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:35:44.634 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:35:44.634 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "commit_sent", 2024-03-11T21:35:44.634 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:34:55.358123+0000", 2024-03-11T21:35:44.634 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 5.7763999999999998e-05 2024-03-11T21:35:44.634 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:35:44.634 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:35:44.634 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "done", 2024-03-11T21:35:44.634 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:34:55.358163+0000", 2024-03-11T21:35:44.634 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 4.0240000000000001e-05 2024-03-11T21:35:44.634 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-03-11T21:35:44.635 INFO:teuthology.orchestra.run.smithi097.stdout: ] 2024-03-11T21:35:44.635 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-03-11T21:35:44.635 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:35:44.635 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:35:44.635 INFO:teuthology.orchestra.run.smithi097.stdout: "description": "osd_op(client.15150.0:206893 8.8 8:12ac760f:::benchmark_data_smithi092_104865_object86392:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e236)", 2024-03-11T21:35:44.635 INFO:teuthology.orchestra.run.smithi097.stdout: "initiated_at": "2024-03-11T21:34:54.372974+0000", 2024-03-11T21:35:44.635 INFO:teuthology.orchestra.run.smithi097.stdout: "age": 50.270614526999999, 2024-03-11T21:35:44.635 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.98534399299999997, 2024-03-11T21:35:44.635 INFO:teuthology.orchestra.run.smithi097.stdout: "type_data": { 2024-03-11T21:35:44.635 INFO:teuthology.orchestra.run.smithi097.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-11T21:35:44.635 INFO:teuthology.orchestra.run.smithi097.stdout: "client_info": { 2024-03-11T21:35:44.635 INFO:teuthology.orchestra.run.smithi097.stdout: "client": "client.15150", 2024-03-11T21:35:44.635 INFO:teuthology.orchestra.run.smithi097.stdout: "client_addr": "v1:172.21.15.92:0/436606717", 2024-03-11T21:35:44.636 INFO:teuthology.orchestra.run.smithi097.stdout: "tid": 206893 2024-03-11T21:35:44.636 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:35:44.636 INFO:teuthology.orchestra.run.smithi097.stdout: "events": [ 2024-03-11T21:35:44.636 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:35:44.636 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "initiated", 2024-03-11T21:35:44.636 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:34:54.372974+0000", 2024-03-11T21:35:44.636 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-03-11T21:35:44.636 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:35:44.636 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:35:44.636 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "header_read", 2024-03-11T21:35:44.636 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:34:54.372974+0000", 2024-03-11T21:35:44.636 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-03-11T21:35:44.636 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:35:44.636 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:35:44.637 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "throttled", 2024-03-11T21:35:44.637 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:34:54.372977+0000", 2024-03-11T21:35:44.637 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 2.1710000000000001e-06 2024-03-11T21:35:44.637 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:35:44.637 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:35:44.637 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "all_read", 2024-03-11T21:35:44.637 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:34:54.372982+0000", 2024-03-11T21:35:44.637 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 5.789e-06 2024-03-11T21:35:44.637 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:35:44.637 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:35:44.637 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "dispatched", 2024-03-11T21:35:44.637 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:34:54.372994+0000", 2024-03-11T21:35:44.637 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.1392e-05 2024-03-11T21:35:44.637 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:35:44.637 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:35:44.638 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "queued_for_pg", 2024-03-11T21:35:44.638 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:34:54.373012+0000", 2024-03-11T21:35:44.638 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.8196999999999999e-05 2024-03-11T21:35:44.638 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:35:44.638 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:35:44.638 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "reached_pg", 2024-03-11T21:35:44.638 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:34:54.373152+0000", 2024-03-11T21:35:44.638 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.00013991399999999999 2024-03-11T21:35:44.638 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:35:44.638 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:35:44.638 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "waiting for peered", 2024-03-11T21:35:44.638 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:34:54.373160+0000", 2024-03-11T21:35:44.638 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 8.4700000000000002e-06 2024-03-11T21:35:44.639 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:35:44.639 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:35:44.639 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "reached_pg", 2024-03-11T21:35:44.639 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:34:55.354162+0000", 2024-03-11T21:35:44.639 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.98100120700000004 2024-03-11T21:35:44.639 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:35:44.639 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:35:44.639 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "started", 2024-03-11T21:35:44.639 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:34:55.354284+0000", 2024-03-11T21:35:44.639 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.000122342 2024-03-11T21:35:44.639 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:35:44.639 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:35:44.639 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "waiting for subops from 3,7", 2024-03-11T21:35:44.639 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:34:55.354549+0000", 2024-03-11T21:35:44.639 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.00026506199999999999 2024-03-11T21:35:44.640 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:35:44.640 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:35:44.640 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "sub_op_commit_rec", 2024-03-11T21:35:44.640 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:34:55.358238+0000", 2024-03-11T21:35:44.640 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.0036888390000000002 2024-03-11T21:35:44.640 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:35:44.640 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:35:44.640 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "sub_op_commit_rec", 2024-03-11T21:35:44.640 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:34:55.358267+0000", 2024-03-11T21:35:44.640 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 2.8858999999999999e-05 2024-03-11T21:35:44.640 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:35:44.640 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:35:44.640 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "op_commit", 2024-03-11T21:35:44.640 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:34:55.358274+0000", 2024-03-11T21:35:44.641 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 7.3629999999999996e-06 2024-03-11T21:35:44.641 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:35:44.641 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:35:44.641 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "commit_sent", 2024-03-11T21:35:44.641 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:34:55.358299+0000", 2024-03-11T21:35:44.641 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 2.4612000000000001e-05 2024-03-11T21:35:44.641 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:35:44.641 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:35:44.641 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "done", 2024-03-11T21:35:44.641 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:34:55.358318+0000", 2024-03-11T21:35:44.641 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.9775999999999998e-05 2024-03-11T21:35:44.641 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-03-11T21:35:44.641 INFO:teuthology.orchestra.run.smithi097.stdout: ] 2024-03-11T21:35:44.641 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-03-11T21:35:44.642 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:35:44.642 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:35:44.642 INFO:teuthology.orchestra.run.smithi097.stdout: "description": "osd_op(client.15150.0:206901 8.8 8:1351d222:::benchmark_data_smithi092_104865_object86400:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e236)", 2024-03-11T21:35:44.642 INFO:teuthology.orchestra.run.smithi097.stdout: "initiated_at": "2024-03-11T21:34:54.379416+0000", 2024-03-11T21:35:44.642 INFO:teuthology.orchestra.run.smithi097.stdout: "age": 50.264173104999998, 2024-03-11T21:35:44.642 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.979003124, 2024-03-11T21:35:44.642 INFO:teuthology.orchestra.run.smithi097.stdout: "type_data": { 2024-03-11T21:35:44.642 INFO:teuthology.orchestra.run.smithi097.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-11T21:35:44.642 INFO:teuthology.orchestra.run.smithi097.stdout: "client_info": { 2024-03-11T21:35:44.642 INFO:teuthology.orchestra.run.smithi097.stdout: "client": "client.15150", 2024-03-11T21:35:44.642 INFO:teuthology.orchestra.run.smithi097.stdout: "client_addr": "v1:172.21.15.92:0/436606717", 2024-03-11T21:35:44.642 INFO:teuthology.orchestra.run.smithi097.stdout: "tid": 206901 2024-03-11T21:35:44.642 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:35:44.642 INFO:teuthology.orchestra.run.smithi097.stdout: "events": [ 2024-03-11T21:35:44.643 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:35:44.643 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "initiated", 2024-03-11T21:35:44.643 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:34:54.379416+0000", 2024-03-11T21:35:44.643 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-03-11T21:35:44.643 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:35:44.643 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:35:44.643 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "header_read", 2024-03-11T21:35:44.643 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:34:54.379416+0000", 2024-03-11T21:35:44.643 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-03-11T21:35:44.643 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:35:44.643 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:35:44.643 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "throttled", 2024-03-11T21:35:44.643 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:34:54.379418+0000", 2024-03-11T21:35:44.643 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.8160000000000001e-06 2024-03-11T21:35:44.643 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:35:44.644 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:35:44.644 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "all_read", 2024-03-11T21:35:44.644 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:34:54.379423+0000", 2024-03-11T21:35:44.644 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 5.4439999999999999e-06 2024-03-11T21:35:44.644 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:35:44.644 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:35:44.644 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "dispatched", 2024-03-11T21:35:44.644 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:34:54.379436+0000", 2024-03-11T21:35:44.644 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.3278000000000001e-05 2024-03-11T21:35:44.644 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:35:44.644 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:35:44.644 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "queued_for_pg", 2024-03-11T21:35:44.644 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:34:54.379444+0000", 2024-03-11T21:35:44.645 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 7.7719999999999994e-06 2024-03-11T21:35:44.645 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:35:44.645 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:35:44.645 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "reached_pg", 2024-03-11T21:35:44.645 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:34:54.379503+0000", 2024-03-11T21:35:44.645 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 5.9218000000000001e-05 2024-03-11T21:35:44.645 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:35:44.645 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:35:44.645 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "waiting for peered", 2024-03-11T21:35:44.645 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:34:54.379512+0000", 2024-03-11T21:35:44.645 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 8.1170000000000004e-06 2024-03-11T21:35:44.645 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:35:44.645 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:35:44.645 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "reached_pg", 2024-03-11T21:35:44.646 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:34:55.356273+0000", 2024-03-11T21:35:44.646 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.97676143999999998 2024-03-11T21:35:44.646 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:35:44.646 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:35:44.646 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "started", 2024-03-11T21:35:44.646 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:34:55.356383+0000", 2024-03-11T21:35:44.646 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.000109719 2024-03-11T21:35:44.646 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:35:44.646 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:35:44.646 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "waiting for subops from 3,7", 2024-03-11T21:35:44.646 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:34:55.356585+0000", 2024-03-11T21:35:44.646 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.000202645 2024-03-11T21:35:44.646 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:35:44.646 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:35:44.647 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "op_commit", 2024-03-11T21:35:44.647 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:34:55.358324+0000", 2024-03-11T21:35:44.647 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.001738442 2024-03-11T21:35:44.647 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:35:44.647 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:35:44.647 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "sub_op_commit_rec", 2024-03-11T21:35:44.647 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:34:55.358350+0000", 2024-03-11T21:35:44.647 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 2.5803999999999999e-05 2024-03-11T21:35:44.647 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:35:44.647 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:35:44.647 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "sub_op_commit_rec", 2024-03-11T21:35:44.647 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:34:55.358378+0000", 2024-03-11T21:35:44.647 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 2.8710000000000001e-05 2024-03-11T21:35:44.647 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:35:44.648 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:35:44.648 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "commit_sent", 2024-03-11T21:35:44.648 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:34:55.358402+0000", 2024-03-11T21:35:44.648 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 2.3473999999999999e-05 2024-03-11T21:35:44.648 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:35:44.648 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:35:44.648 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "done", 2024-03-11T21:35:44.648 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:34:55.358419+0000", 2024-03-11T21:35:44.648 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.7244999999999998e-05 2024-03-11T21:35:44.648 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-03-11T21:35:44.648 INFO:teuthology.orchestra.run.smithi097.stdout: ] 2024-03-11T21:35:44.648 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-03-11T21:35:44.648 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:35:44.648 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:35:44.649 INFO:teuthology.orchestra.run.smithi097.stdout: "description": "osd_op(client.15150.0:206908 8.8 8:15831f13:::benchmark_data_smithi092_104865_object86407:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e236)", 2024-03-11T21:35:44.649 INFO:teuthology.orchestra.run.smithi097.stdout: "initiated_at": "2024-03-11T21:34:54.388435+0000", 2024-03-11T21:35:44.649 INFO:teuthology.orchestra.run.smithi097.stdout: "age": 50.255153909999997, 2024-03-11T21:35:44.649 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.97410122300000002, 2024-03-11T21:35:44.649 INFO:teuthology.orchestra.run.smithi097.stdout: "type_data": { 2024-03-11T21:35:44.649 INFO:teuthology.orchestra.run.smithi097.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-11T21:35:44.649 INFO:teuthology.orchestra.run.smithi097.stdout: "client_info": { 2024-03-11T21:35:44.649 INFO:teuthology.orchestra.run.smithi097.stdout: "client": "client.15150", 2024-03-11T21:35:44.649 INFO:teuthology.orchestra.run.smithi097.stdout: "client_addr": "v1:172.21.15.92:0/436606717", 2024-03-11T21:35:44.649 INFO:teuthology.orchestra.run.smithi097.stdout: "tid": 206908 2024-03-11T21:35:44.649 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:35:44.649 INFO:teuthology.orchestra.run.smithi097.stdout: "events": [ 2024-03-11T21:35:44.649 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:35:44.649 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "initiated", 2024-03-11T21:35:44.650 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:34:54.388435+0000", 2024-03-11T21:35:44.650 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-03-11T21:35:44.650 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:35:44.650 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:35:44.650 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "header_read", 2024-03-11T21:35:44.650 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:34:54.388435+0000", 2024-03-11T21:35:44.650 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-03-11T21:35:44.650 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:35:44.650 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:35:44.650 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "throttled", 2024-03-11T21:35:44.650 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:34:54.388437+0000", 2024-03-11T21:35:44.650 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.469e-06 2024-03-11T21:35:44.650 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:35:44.650 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:35:44.651 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "all_read", 2024-03-11T21:35:44.651 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:34:54.388441+0000", 2024-03-11T21:35:44.651 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 4.6079999999999998e-06 2024-03-11T21:35:44.651 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:35:44.651 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:35:44.651 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "dispatched", 2024-03-11T21:35:44.651 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:34:54.388451+0000", 2024-03-11T21:35:44.651 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 9.7459999999999995e-06 2024-03-11T21:35:44.651 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:35:44.651 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:35:44.651 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "queued_for_pg", 2024-03-11T21:35:44.651 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:34:54.388459+0000", 2024-03-11T21:35:44.651 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 7.6240000000000002e-06 2024-03-11T21:35:44.651 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:35:44.652 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:35:44.652 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "reached_pg", 2024-03-11T21:35:44.652 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:34:54.388573+0000", 2024-03-11T21:35:44.652 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.00011455699999999999 2024-03-11T21:35:44.652 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:35:44.652 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:35:44.652 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "waiting for peered", 2024-03-11T21:35:44.652 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:34:54.388582+0000", 2024-03-11T21:35:44.652 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 8.6629999999999999e-06 2024-03-11T21:35:44.652 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:35:44.652 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:35:44.652 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "reached_pg", 2024-03-11T21:35:44.652 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:34:55.358443+0000", 2024-03-11T21:35:44.652 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.96986088999999998 2024-03-11T21:35:44.652 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:35:44.653 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:35:44.653 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "started", 2024-03-11T21:35:44.653 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:34:55.358510+0000", 2024-03-11T21:35:44.653 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 6.7407e-05 2024-03-11T21:35:44.653 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:35:44.653 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:35:44.653 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "waiting for subops from 3,7", 2024-03-11T21:35:44.653 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:34:55.358721+0000", 2024-03-11T21:35:44.653 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.00021080599999999999 2024-03-11T21:35:44.653 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:35:44.653 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:35:44.653 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "sub_op_commit_rec", 2024-03-11T21:35:44.653 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:34:55.362390+0000", 2024-03-11T21:35:44.653 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.0036691190000000002 2024-03-11T21:35:44.654 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:35:44.654 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:35:44.654 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "op_commit", 2024-03-11T21:35:44.654 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:34:55.362407+0000", 2024-03-11T21:35:44.654 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.6813999999999999e-05 2024-03-11T21:35:44.654 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:35:44.654 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:35:44.654 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "sub_op_commit_rec", 2024-03-11T21:35:44.654 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:34:55.362445+0000", 2024-03-11T21:35:44.654 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 3.8012999999999997e-05 2024-03-11T21:35:44.654 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:35:44.654 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:35:44.654 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "commit_sent", 2024-03-11T21:35:44.654 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:34:55.362493+0000", 2024-03-11T21:35:44.656 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 4.7759999999999997e-05 2024-03-11T21:35:44.656 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:35:44.656 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:35:44.656 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "done", 2024-03-11T21:35:44.656 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:34:55.362536+0000", 2024-03-11T21:35:44.656 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 4.3747e-05 2024-03-11T21:35:44.656 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-03-11T21:35:44.656 INFO:teuthology.orchestra.run.smithi097.stdout: ] 2024-03-11T21:35:44.656 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-03-11T21:35:44.656 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:35:44.657 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:35:44.657 INFO:teuthology.orchestra.run.smithi097.stdout: "description": "osd_op(client.15150.0:206922 8.8 8:17e36640:::benchmark_data_smithi092_104865_object86421:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e236)", 2024-03-11T21:35:44.657 INFO:teuthology.orchestra.run.smithi097.stdout: "initiated_at": "2024-03-11T21:34:54.400598+0000", 2024-03-11T21:35:44.657 INFO:teuthology.orchestra.run.smithi097.stdout: "age": 50.242990517999999, 2024-03-11T21:35:44.657 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.96768579600000004, 2024-03-11T21:35:44.657 INFO:teuthology.orchestra.run.smithi097.stdout: "type_data": { 2024-03-11T21:35:44.657 INFO:teuthology.orchestra.run.smithi097.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-11T21:35:44.657 INFO:teuthology.orchestra.run.smithi097.stdout: "client_info": { 2024-03-11T21:35:44.657 INFO:teuthology.orchestra.run.smithi097.stdout: "client": "client.15150", 2024-03-11T21:35:44.657 INFO:teuthology.orchestra.run.smithi097.stdout: "client_addr": "v1:172.21.15.92:0/436606717", 2024-03-11T21:35:44.657 INFO:teuthology.orchestra.run.smithi097.stdout: "tid": 206922 2024-03-11T21:35:44.657 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:35:44.658 INFO:teuthology.orchestra.run.smithi097.stdout: "events": [ 2024-03-11T21:35:44.658 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:35:44.658 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "initiated", 2024-03-11T21:35:44.658 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:34:54.400598+0000", 2024-03-11T21:35:44.658 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-03-11T21:35:44.658 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:35:44.658 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:35:44.658 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "header_read", 2024-03-11T21:35:44.658 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:34:54.400598+0000", 2024-03-11T21:35:44.658 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-03-11T21:35:44.658 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:35:44.658 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:35:44.658 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "throttled", 2024-03-11T21:35:44.659 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:34:54.400600+0000", 2024-03-11T21:35:44.659 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.4300000000000001e-06 2024-03-11T21:35:44.659 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:35:44.659 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:35:44.659 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "all_read", 2024-03-11T21:35:44.659 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:34:54.400605+0000", 2024-03-11T21:35:44.659 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 4.7720000000000002e-06 2024-03-11T21:35:44.659 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:35:44.659 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:35:44.659 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "dispatched", 2024-03-11T21:35:44.659 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:34:54.400619+0000", 2024-03-11T21:35:44.659 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.4229e-05 2024-03-11T21:35:44.659 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:35:44.660 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:35:44.660 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "queued_for_pg", 2024-03-11T21:35:44.660 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:34:54.400627+0000", 2024-03-11T21:35:44.660 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 7.8080000000000005e-06 2024-03-11T21:35:44.660 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:35:44.660 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:35:44.660 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "reached_pg", 2024-03-11T21:35:44.660 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:34:54.400721+0000", 2024-03-11T21:35:44.660 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 9.4081999999999998e-05 2024-03-11T21:35:44.660 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:35:44.660 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:35:44.660 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "waiting for peered", 2024-03-11T21:35:44.660 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:34:54.400730+0000", 2024-03-11T21:35:44.660 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 8.9479999999999997e-06 2024-03-11T21:35:44.661 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:35:44.661 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:35:44.661 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "reached_pg", 2024-03-11T21:35:44.661 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:34:55.360433+0000", 2024-03-11T21:35:44.661 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.959702995 2024-03-11T21:35:44.661 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:35:44.661 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:35:44.661 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "started", 2024-03-11T21:35:44.661 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:34:55.360519+0000", 2024-03-11T21:35:44.661 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 8.5859999999999994e-05 2024-03-11T21:35:44.661 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:35:44.661 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:35:44.662 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "waiting for subops from 3,7", 2024-03-11T21:35:44.662 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:34:55.360733+0000", 2024-03-11T21:35:44.662 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.00021452299999999999 2024-03-11T21:35:44.662 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:35:44.662 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:35:44.662 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "sub_op_commit_rec", 2024-03-11T21:35:44.662 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:34:55.368055+0000", 2024-03-11T21:35:44.662 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.0073219779999999998 2024-03-11T21:35:44.662 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:35:44.662 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:35:44.662 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "sub_op_commit_rec", 2024-03-11T21:35:44.662 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:34:55.368122+0000", 2024-03-11T21:35:44.662 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 6.6501000000000006e-05 2024-03-11T21:35:44.662 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:35:44.663 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:35:44.663 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "op_commit", 2024-03-11T21:35:44.663 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:34:55.368185+0000", 2024-03-11T21:35:44.663 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 6.3297000000000001e-05 2024-03-11T21:35:44.663 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:35:44.663 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:35:44.663 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "commit_sent", 2024-03-11T21:35:44.663 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:34:55.368241+0000", 2024-03-11T21:35:44.663 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 5.5813000000000002e-05 2024-03-11T21:35:44.663 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:35:44.663 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:35:44.663 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "done", 2024-03-11T21:35:44.663 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:34:55.368284+0000", 2024-03-11T21:35:44.663 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 4.3560000000000003e-05 2024-03-11T21:35:44.664 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-03-11T21:35:44.664 INFO:teuthology.orchestra.run.smithi097.stdout: ] 2024-03-11T21:35:44.664 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-03-11T21:35:44.664 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:35:44.664 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:35:44.664 INFO:teuthology.orchestra.run.smithi097.stdout: "description": "osd_op(client.15150.0:206925 8.8 8:166114d6:::benchmark_data_smithi092_104865_object86424:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e236)", 2024-03-11T21:35:44.664 INFO:teuthology.orchestra.run.smithi097.stdout: "initiated_at": "2024-03-11T21:34:54.403072+0000", 2024-03-11T21:35:44.664 INFO:teuthology.orchestra.run.smithi097.stdout: "age": 50.240516556999999, 2024-03-11T21:35:44.664 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.96542147899999997, 2024-03-11T21:35:44.664 INFO:teuthology.orchestra.run.smithi097.stdout: "type_data": { 2024-03-11T21:35:44.664 INFO:teuthology.orchestra.run.smithi097.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-11T21:35:44.664 INFO:teuthology.orchestra.run.smithi097.stdout: "client_info": { 2024-03-11T21:35:44.664 INFO:teuthology.orchestra.run.smithi097.stdout: "client": "client.15150", 2024-03-11T21:35:44.665 INFO:teuthology.orchestra.run.smithi097.stdout: "client_addr": "v1:172.21.15.92:0/436606717", 2024-03-11T21:35:44.665 INFO:teuthology.orchestra.run.smithi097.stdout: "tid": 206925 2024-03-11T21:35:44.665 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:35:44.665 INFO:teuthology.orchestra.run.smithi097.stdout: "events": [ 2024-03-11T21:35:44.665 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:35:44.665 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "initiated", 2024-03-11T21:35:44.665 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:34:54.403072+0000", 2024-03-11T21:35:44.665 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-03-11T21:35:44.665 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:35:44.665 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:35:44.665 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "header_read", 2024-03-11T21:35:44.665 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:34:54.403072+0000", 2024-03-11T21:35:44.665 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-03-11T21:35:44.666 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:35:44.666 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:35:44.666 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "throttled", 2024-03-11T21:35:44.666 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:34:54.403074+0000", 2024-03-11T21:35:44.666 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.4330000000000001e-06 2024-03-11T21:35:44.666 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:35:44.666 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:35:44.666 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "all_read", 2024-03-11T21:35:44.666 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:34:54.403078+0000", 2024-03-11T21:35:44.666 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 4.5909999999999998e-06 2024-03-11T21:35:44.666 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:35:44.666 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:35:44.666 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "dispatched", 2024-03-11T21:35:44.666 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:34:54.403091+0000", 2024-03-11T21:35:44.667 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.2405999999999999e-05 2024-03-11T21:35:44.667 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:35:44.667 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:35:44.667 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "queued_for_pg", 2024-03-11T21:35:44.667 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:34:54.403099+0000", 2024-03-11T21:35:44.667 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 8.2279999999999998e-06 2024-03-11T21:35:44.667 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:35:44.667 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:35:44.667 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "reached_pg", 2024-03-11T21:35:44.667 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:34:54.403163+0000", 2024-03-11T21:35:44.667 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 6.3998000000000006e-05 2024-03-11T21:35:44.667 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:35:44.667 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:35:44.667 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "waiting for peered", 2024-03-11T21:35:44.668 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:34:54.403171+0000", 2024-03-11T21:35:44.668 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 8.4239999999999993e-06 2024-03-11T21:35:44.668 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:35:44.668 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:35:44.668 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "reached_pg", 2024-03-11T21:35:44.668 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:34:55.362578+0000", 2024-03-11T21:35:44.668 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.95940687599999996 2024-03-11T21:35:44.668 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:35:44.668 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:35:44.668 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "started", 2024-03-11T21:35:44.668 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:34:55.362684+0000", 2024-03-11T21:35:44.668 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.00010527999999999999 2024-03-11T21:35:44.668 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:35:44.668 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:35:44.668 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "waiting for subops from 3,7", 2024-03-11T21:35:44.669 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:34:55.362934+0000", 2024-03-11T21:35:44.669 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.00025034100000000001 2024-03-11T21:35:44.669 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:35:44.669 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:35:44.669 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "sub_op_commit_rec", 2024-03-11T21:35:44.669 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:34:55.368171+0000", 2024-03-11T21:35:44.669 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.0052369269999999997 2024-03-11T21:35:44.669 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:35:44.669 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:35:44.669 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "op_commit", 2024-03-11T21:35:44.670 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:34:55.368294+0000", 2024-03-11T21:35:44.670 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.00012308499999999999 2024-03-11T21:35:44.670 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:35:44.670 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:35:44.670 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "sub_op_commit_rec", 2024-03-11T21:35:44.670 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:34:55.368395+0000", 2024-03-11T21:35:44.670 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.000101081 2024-03-11T21:35:44.670 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:35:44.670 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:35:44.670 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "commit_sent", 2024-03-11T21:35:44.670 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:34:55.368453+0000", 2024-03-11T21:35:44.670 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 5.8320999999999999e-05 2024-03-11T21:35:44.670 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:35:44.671 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:35:44.671 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "done", 2024-03-11T21:35:44.671 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:34:55.368494+0000", 2024-03-11T21:35:44.671 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 4.0488000000000003e-05 2024-03-11T21:35:44.671 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-03-11T21:35:44.671 INFO:teuthology.orchestra.run.smithi097.stdout: ] 2024-03-11T21:35:44.671 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-03-11T21:35:44.671 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:35:44.671 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:35:44.671 INFO:teuthology.orchestra.run.smithi097.stdout: "description": "osd_op(client.15150.0:206951 8.8 8:159749e8:::benchmark_data_smithi092_104865_object86450:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e236)", 2024-03-11T21:35:44.671 INFO:teuthology.orchestra.run.smithi097.stdout: "initiated_at": "2024-03-11T21:34:54.451988+0000", 2024-03-11T21:35:44.671 INFO:teuthology.orchestra.run.smithi097.stdout: "age": 50.19160136, 2024-03-11T21:35:44.671 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.91667804200000003, 2024-03-11T21:35:44.671 INFO:teuthology.orchestra.run.smithi097.stdout: "type_data": { 2024-03-11T21:35:44.672 INFO:teuthology.orchestra.run.smithi097.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-11T21:35:44.672 INFO:teuthology.orchestra.run.smithi097.stdout: "client_info": { 2024-03-11T21:35:44.672 INFO:teuthology.orchestra.run.smithi097.stdout: "client": "client.15150", 2024-03-11T21:35:44.672 INFO:teuthology.orchestra.run.smithi097.stdout: "client_addr": "v1:172.21.15.92:0/436606717", 2024-03-11T21:35:44.672 INFO:teuthology.orchestra.run.smithi097.stdout: "tid": 206951 2024-03-11T21:35:44.672 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:35:44.672 INFO:teuthology.orchestra.run.smithi097.stdout: "events": [ 2024-03-11T21:35:44.672 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:35:44.672 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "initiated", 2024-03-11T21:35:44.672 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:34:54.451988+0000", 2024-03-11T21:35:44.672 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-03-11T21:35:44.672 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:35:44.672 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:35:44.672 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "header_read", 2024-03-11T21:35:44.673 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:34:54.451988+0000", 2024-03-11T21:35:44.673 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-03-11T21:35:44.673 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:35:44.673 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:35:44.673 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "throttled", 2024-03-11T21:35:44.673 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:34:54.451989+0000", 2024-03-11T21:35:44.673 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.6419999999999999e-06 2024-03-11T21:35:44.673 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:35:44.673 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:35:44.673 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "all_read", 2024-03-11T21:35:44.673 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:34:54.451994+0000", 2024-03-11T21:35:44.673 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 5.186e-06 2024-03-11T21:35:44.673 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:35:44.673 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:35:44.674 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "dispatched", 2024-03-11T21:35:44.674 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:34:54.452003+0000", 2024-03-11T21:35:44.674 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 8.8720000000000008e-06 2024-03-11T21:35:44.674 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:35:44.674 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:35:44.674 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "queued_for_pg", 2024-03-11T21:35:44.674 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:34:54.452011+0000", 2024-03-11T21:35:44.674 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 7.4279999999999999e-06 2024-03-11T21:35:44.674 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:35:44.674 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:35:44.674 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "reached_pg", 2024-03-11T21:35:44.674 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:34:54.452090+0000", 2024-03-11T21:35:44.674 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 7.9476999999999998e-05 2024-03-11T21:35:44.674 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:35:44.675 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:35:44.675 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "waiting for peered", 2024-03-11T21:35:44.675 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:34:54.452097+0000", 2024-03-11T21:35:44.675 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 6.3600000000000001e-06 2024-03-11T21:35:44.675 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:35:44.675 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:35:44.675 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "reached_pg", 2024-03-11T21:35:44.675 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:34:55.365447+0000", 2024-03-11T21:35:44.675 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.91335046600000003 2024-03-11T21:35:44.675 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:35:44.675 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:35:44.675 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "started", 2024-03-11T21:35:44.675 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:34:55.365555+0000", 2024-03-11T21:35:44.676 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.00010751600000000001 2024-03-11T21:35:44.676 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:35:44.676 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:35:44.676 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "waiting for subops from 3,7", 2024-03-11T21:35:44.676 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:34:55.365890+0000", 2024-03-11T21:35:44.676 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.000335695 2024-03-11T21:35:44.676 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:35:44.676 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:35:44.676 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "op_commit", 2024-03-11T21:35:44.676 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:34:55.368300+0000", 2024-03-11T21:35:44.676 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.0024100620000000001 2024-03-11T21:35:44.676 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:35:44.676 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:35:44.677 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "sub_op_commit_rec", 2024-03-11T21:35:44.677 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:34:55.368347+0000", 2024-03-11T21:35:44.677 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 4.6412000000000002e-05 2024-03-11T21:35:44.677 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:35:44.677 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:35:44.677 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "sub_op_commit_rec", 2024-03-11T21:35:44.677 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:34:55.368543+0000", 2024-03-11T21:35:44.677 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.000196188 2024-03-11T21:35:44.677 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:35:44.677 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:35:44.677 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "commit_sent", 2024-03-11T21:35:44.677 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:34:55.368626+0000", 2024-03-11T21:35:44.677 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 8.3088000000000001e-05 2024-03-11T21:35:44.678 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:35:44.678 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:35:44.678 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "done", 2024-03-11T21:35:44.678 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:34:55.368666+0000", 2024-03-11T21:35:44.678 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 3.9650000000000002e-05 2024-03-11T21:35:44.678 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-03-11T21:35:44.678 INFO:teuthology.orchestra.run.smithi097.stdout: ] 2024-03-11T21:35:44.678 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-03-11T21:35:44.678 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:35:44.678 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:35:44.678 INFO:teuthology.orchestra.run.smithi097.stdout: "description": "osd_op(client.15150.0:208574 8.d 8:b4f0a13a:::benchmark_data_smithi092_104865_object88073:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e238)", 2024-03-11T21:35:44.678 INFO:teuthology.orchestra.run.smithi097.stdout: "initiated_at": "2024-03-11T21:34:57.352453+0000", 2024-03-11T21:35:44.678 INFO:teuthology.orchestra.run.smithi097.stdout: "age": 47.291135693000001, 2024-03-11T21:35:44.678 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.0319107139999999, 2024-03-11T21:35:44.679 INFO:teuthology.orchestra.run.smithi097.stdout: "type_data": { 2024-03-11T21:35:44.679 INFO:teuthology.orchestra.run.smithi097.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-11T21:35:44.679 INFO:teuthology.orchestra.run.smithi097.stdout: "client_info": { 2024-03-11T21:35:44.679 INFO:teuthology.orchestra.run.smithi097.stdout: "client": "client.15150", 2024-03-11T21:35:44.679 INFO:teuthology.orchestra.run.smithi097.stdout: "client_addr": "v1:172.21.15.92:0/436606717", 2024-03-11T21:35:44.679 INFO:teuthology.orchestra.run.smithi097.stdout: "tid": 208574 2024-03-11T21:35:44.679 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:35:44.679 INFO:teuthology.orchestra.run.smithi097.stdout: "events": [ 2024-03-11T21:35:44.679 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:35:44.679 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "initiated", 2024-03-11T21:35:44.679 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:34:57.352453+0000", 2024-03-11T21:35:44.679 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-03-11T21:35:44.679 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:35:44.680 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:35:44.680 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "header_read", 2024-03-11T21:35:44.680 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:34:57.352453+0000", 2024-03-11T21:35:44.680 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-03-11T21:35:44.680 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:35:44.680 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:35:44.680 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "throttled", 2024-03-11T21:35:44.680 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:34:57.352455+0000", 2024-03-11T21:35:44.680 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 2.0600000000000002e-06 2024-03-11T21:35:44.680 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:35:44.680 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:35:44.680 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "all_read", 2024-03-11T21:35:44.680 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:34:57.352462+0000", 2024-03-11T21:35:44.680 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 6.229e-06 2024-03-11T21:35:44.681 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:35:44.681 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:35:44.681 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "dispatched", 2024-03-11T21:35:44.681 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:34:57.353033+0000", 2024-03-11T21:35:44.681 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.00057142200000000003 2024-03-11T21:35:44.681 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:35:44.681 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:35:44.681 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "queued_for_pg", 2024-03-11T21:35:44.681 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:34:57.353041+0000", 2024-03-11T21:35:44.681 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 8.4379999999999995e-06 2024-03-11T21:35:44.681 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:35:44.681 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:35:44.681 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "reached_pg", 2024-03-11T21:35:44.681 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:34:57.353352+0000", 2024-03-11T21:35:44.681 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.00031082400000000002 2024-03-11T21:35:44.682 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:35:44.682 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:35:44.682 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "waiting for peered", 2024-03-11T21:35:44.682 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:34:57.353366+0000", 2024-03-11T21:35:44.682 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.4037e-05 2024-03-11T21:35:44.682 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:35:44.682 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:35:44.682 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "reached_pg", 2024-03-11T21:35:44.682 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:34:58.381460+0000", 2024-03-11T21:35:44.682 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.0280934779999999 2024-03-11T21:35:44.682 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:35:44.682 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:35:44.682 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "started", 2024-03-11T21:35:44.682 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:34:58.381562+0000", 2024-03-11T21:35:44.682 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.000102308 2024-03-11T21:35:44.683 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:35:44.683 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:35:44.683 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "waiting for subops from 2,7", 2024-03-11T21:35:44.683 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:34:58.381651+0000", 2024-03-11T21:35:44.683 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 8.9386000000000001e-05 2024-03-11T21:35:44.683 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:35:44.683 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:35:44.683 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "sub_op_commit_rec", 2024-03-11T21:35:44.683 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:34:58.384224+0000", 2024-03-11T21:35:44.683 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.0025727839999999998 2024-03-11T21:35:44.683 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:35:44.683 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:35:44.683 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "sub_op_commit_rec", 2024-03-11T21:35:44.683 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:34:58.384274+0000", 2024-03-11T21:35:44.684 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 4.994e-05 2024-03-11T21:35:44.684 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:35:44.684 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:35:44.684 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "op_commit", 2024-03-11T21:35:44.684 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:34:58.384287+0000", 2024-03-11T21:35:44.684 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.2376e-05 2024-03-11T21:35:44.684 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:35:44.684 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:35:44.684 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "commit_sent", 2024-03-11T21:35:44.684 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:34:58.384318+0000", 2024-03-11T21:35:44.684 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 3.1905e-05 2024-03-11T21:35:44.684 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:35:44.684 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:35:44.685 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "done", 2024-03-11T21:35:44.685 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:34:58.384364+0000", 2024-03-11T21:35:44.685 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 4.5527e-05 2024-03-11T21:35:44.685 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-03-11T21:35:44.685 INFO:teuthology.orchestra.run.smithi097.stdout: ] 2024-03-11T21:35:44.685 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-03-11T21:35:44.685 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:35:44.685 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:35:44.685 INFO:teuthology.orchestra.run.smithi097.stdout: "description": "osd_op(client.15150.0:208578 8.d 8:b8981014:::benchmark_data_smithi092_104865_object88077:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e239)", 2024-03-11T21:35:44.685 INFO:teuthology.orchestra.run.smithi097.stdout: "initiated_at": "2024-03-11T21:34:57.356717+0000", 2024-03-11T21:35:44.685 INFO:teuthology.orchestra.run.smithi097.stdout: "age": 47.286871579, 2024-03-11T21:35:44.685 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.0313246620000001, 2024-03-11T21:35:44.685 INFO:teuthology.orchestra.run.smithi097.stdout: "type_data": { 2024-03-11T21:35:44.685 INFO:teuthology.orchestra.run.smithi097.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-11T21:35:44.686 INFO:teuthology.orchestra.run.smithi097.stdout: "client_info": { 2024-03-11T21:35:44.686 INFO:teuthology.orchestra.run.smithi097.stdout: "client": "client.15150", 2024-03-11T21:35:44.686 INFO:teuthology.orchestra.run.smithi097.stdout: "client_addr": "v1:172.21.15.92:0/436606717", 2024-03-11T21:35:44.686 INFO:teuthology.orchestra.run.smithi097.stdout: "tid": 208578 2024-03-11T21:35:44.686 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:35:44.686 INFO:teuthology.orchestra.run.smithi097.stdout: "events": [ 2024-03-11T21:35:44.686 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:35:44.686 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "initiated", 2024-03-11T21:35:44.686 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:34:57.356717+0000", 2024-03-11T21:35:44.686 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-03-11T21:35:44.686 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:35:44.686 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:35:44.686 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "header_read", 2024-03-11T21:35:44.686 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:34:57.356717+0000", 2024-03-11T21:35:44.687 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-03-11T21:35:44.687 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:35:44.687 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:35:44.687 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "throttled", 2024-03-11T21:35:44.687 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:34:57.356719+0000", 2024-03-11T21:35:44.687 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.866e-06 2024-03-11T21:35:44.687 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:35:44.687 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:35:44.687 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "all_read", 2024-03-11T21:35:44.687 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:34:57.356724+0000", 2024-03-11T21:35:44.687 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 5.1909999999999999e-06 2024-03-11T21:35:44.687 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:35:44.687 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:35:44.687 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "dispatched", 2024-03-11T21:35:44.688 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:34:57.356736+0000", 2024-03-11T21:35:44.688 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.1231e-05 2024-03-11T21:35:44.688 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:35:44.688 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:35:44.688 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "queued_for_pg", 2024-03-11T21:35:44.688 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:34:57.356743+0000", 2024-03-11T21:35:44.688 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 7.5830000000000001e-06 2024-03-11T21:35:44.688 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:35:44.688 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:35:44.688 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "reached_pg", 2024-03-11T21:35:44.688 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:34:57.356802+0000", 2024-03-11T21:35:44.688 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 5.9107999999999997e-05 2024-03-11T21:35:44.688 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:35:44.689 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:35:44.689 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "waiting for peered", 2024-03-11T21:35:44.689 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:34:57.356811+0000", 2024-03-11T21:35:44.689 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 8.8869999999999998e-06 2024-03-11T21:35:44.689 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:35:44.689 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:35:44.689 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "reached_pg", 2024-03-11T21:35:44.689 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:34:58.384395+0000", 2024-03-11T21:35:44.689 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.0275842260000001 2024-03-11T21:35:44.689 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:35:44.689 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:35:44.689 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "started", 2024-03-11T21:35:44.689 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:34:58.384476+0000", 2024-03-11T21:35:44.689 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 8.0187999999999999e-05 2024-03-11T21:35:44.690 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:35:44.690 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:35:44.690 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "waiting for subops from 2,7", 2024-03-11T21:35:44.690 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:34:58.384553+0000", 2024-03-11T21:35:44.690 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 7.6897999999999995e-05 2024-03-11T21:35:44.690 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:35:44.690 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:35:44.690 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "sub_op_commit_rec", 2024-03-11T21:35:44.690 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:34:58.387913+0000", 2024-03-11T21:35:44.690 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.0033606370000000001 2024-03-11T21:35:44.690 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:35:44.690 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:35:44.690 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "sub_op_commit_rec", 2024-03-11T21:35:44.691 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:34:58.387977+0000", 2024-03-11T21:35:44.691 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 6.3377e-05 2024-03-11T21:35:44.691 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:35:44.691 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:35:44.691 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "op_commit", 2024-03-11T21:35:44.691 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:34:58.387987+0000", 2024-03-11T21:35:44.691 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.0173000000000001e-05 2024-03-11T21:35:44.691 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:35:44.691 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:35:44.691 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "commit_sent", 2024-03-11T21:35:44.691 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:34:58.388017+0000", 2024-03-11T21:35:44.691 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 3.0439000000000001e-05 2024-03-11T21:35:44.691 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:35:44.691 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:35:44.691 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "done", 2024-03-11T21:35:44.692 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:34:58.388042+0000", 2024-03-11T21:35:44.692 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 2.4858000000000002e-05 2024-03-11T21:35:44.692 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-03-11T21:35:44.692 INFO:teuthology.orchestra.run.smithi097.stdout: ] 2024-03-11T21:35:44.692 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-03-11T21:35:44.692 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:35:44.692 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:35:44.692 INFO:teuthology.orchestra.run.smithi097.stdout: "description": "osd_op(client.15150.0:208589 8.d 8:b0d80d0e:::benchmark_data_smithi092_104865_object88088:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e239)", 2024-03-11T21:35:44.692 INFO:teuthology.orchestra.run.smithi097.stdout: "initiated_at": "2024-03-11T21:34:57.366673+0000", 2024-03-11T21:35:44.692 INFO:teuthology.orchestra.run.smithi097.stdout: "age": 47.276915510000002, 2024-03-11T21:35:44.692 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.026816476, 2024-03-11T21:35:44.692 INFO:teuthology.orchestra.run.smithi097.stdout: "type_data": { 2024-03-11T21:35:44.692 INFO:teuthology.orchestra.run.smithi097.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-11T21:35:44.692 INFO:teuthology.orchestra.run.smithi097.stdout: "client_info": { 2024-03-11T21:35:44.693 INFO:teuthology.orchestra.run.smithi097.stdout: "client": "client.15150", 2024-03-11T21:35:44.693 INFO:teuthology.orchestra.run.smithi097.stdout: "client_addr": "v1:172.21.15.92:0/436606717", 2024-03-11T21:35:44.693 INFO:teuthology.orchestra.run.smithi097.stdout: "tid": 208589 2024-03-11T21:35:44.693 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:35:44.693 INFO:teuthology.orchestra.run.smithi097.stdout: "events": [ 2024-03-11T21:35:44.693 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:35:44.693 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "initiated", 2024-03-11T21:35:44.693 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:34:57.366673+0000", 2024-03-11T21:35:44.693 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-03-11T21:35:44.693 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:35:44.693 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:35:44.693 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "header_read", 2024-03-11T21:35:44.693 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:34:57.366673+0000", 2024-03-11T21:35:44.693 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-03-11T21:35:44.693 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:35:44.694 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:35:44.694 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "throttled", 2024-03-11T21:35:44.694 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:34:57.366675+0000", 2024-03-11T21:35:44.694 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.567e-06 2024-03-11T21:35:44.694 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:35:44.694 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:35:44.694 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "all_read", 2024-03-11T21:35:44.694 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:34:57.366679+0000", 2024-03-11T21:35:44.694 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 4.4299999999999999e-06 2024-03-11T21:35:44.694 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:35:44.694 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:35:44.694 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "dispatched", 2024-03-11T21:35:44.694 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:34:57.366690+0000", 2024-03-11T21:35:44.694 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.1028e-05 2024-03-11T21:35:44.695 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:35:44.695 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:35:44.695 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "queued_for_pg", 2024-03-11T21:35:44.695 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:34:57.366696+0000", 2024-03-11T21:35:44.695 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 5.8860000000000001e-06 2024-03-11T21:35:44.695 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:35:44.695 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:35:44.695 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "reached_pg", 2024-03-11T21:35:44.695 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:34:57.366752+0000", 2024-03-11T21:35:44.695 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 5.5566999999999998e-05 2024-03-11T21:35:44.695 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:35:44.695 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:35:44.695 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "waiting for peered", 2024-03-11T21:35:44.695 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:34:57.366758+0000", 2024-03-11T21:35:44.696 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 6.2269999999999998e-06 2024-03-11T21:35:44.696 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:35:44.696 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:35:44.696 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "reached_pg", 2024-03-11T21:35:44.696 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:34:58.386291+0000", 2024-03-11T21:35:44.696 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.019533238 2024-03-11T21:35:44.696 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:35:44.696 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:35:44.696 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "started", 2024-03-11T21:35:44.696 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:34:58.386368+0000", 2024-03-11T21:35:44.696 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 7.6126999999999998e-05 2024-03-11T21:35:44.696 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:35:44.696 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:35:44.696 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "waiting for subops from 2,7", 2024-03-11T21:35:44.696 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:34:58.386440+0000", 2024-03-11T21:35:44.697 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 7.2539999999999993e-05 2024-03-11T21:35:44.697 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:35:44.697 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:35:44.697 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "sub_op_commit_rec", 2024-03-11T21:35:44.697 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:34:58.388089+0000", 2024-03-11T21:35:44.697 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.001648722 2024-03-11T21:35:44.697 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:35:44.697 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:35:44.697 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "op_commit", 2024-03-11T21:35:44.697 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:34:58.390342+0000", 2024-03-11T21:35:44.697 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.0022533879999999998 2024-03-11T21:35:44.697 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:35:44.697 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:35:44.697 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "sub_op_commit_rec", 2024-03-11T21:35:44.698 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:34:58.393325+0000", 2024-03-11T21:35:44.698 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.0029828979999999999 2024-03-11T21:35:44.698 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:35:44.698 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:35:44.698 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "commit_sent", 2024-03-11T21:35:44.698 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:34:58.393440+0000", 2024-03-11T21:35:44.698 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.000115418 2024-03-11T21:35:44.698 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:35:44.698 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:35:44.698 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "done", 2024-03-11T21:35:44.698 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:34:58.393490+0000", 2024-03-11T21:35:44.698 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 4.9440000000000001e-05 2024-03-11T21:35:44.698 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-03-11T21:35:44.699 INFO:teuthology.orchestra.run.smithi097.stdout: ] 2024-03-11T21:35:44.699 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-03-11T21:35:44.699 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:35:44.699 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:35:44.699 INFO:teuthology.orchestra.run.smithi097.stdout: "description": "osd_op(client.15150.0:208601 8.d 8:b93ad02e:::benchmark_data_smithi092_104865_object88100:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e239)", 2024-03-11T21:35:44.699 INFO:teuthology.orchestra.run.smithi097.stdout: "initiated_at": "2024-03-11T21:34:57.372084+0000", 2024-03-11T21:35:44.699 INFO:teuthology.orchestra.run.smithi097.stdout: "age": 47.271504637, 2024-03-11T21:35:44.699 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.021649343, 2024-03-11T21:35:44.699 INFO:teuthology.orchestra.run.smithi097.stdout: "type_data": { 2024-03-11T21:35:44.699 INFO:teuthology.orchestra.run.smithi097.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-11T21:35:44.699 INFO:teuthology.orchestra.run.smithi097.stdout: "client_info": { 2024-03-11T21:35:44.699 INFO:teuthology.orchestra.run.smithi097.stdout: "client": "client.15150", 2024-03-11T21:35:44.699 INFO:teuthology.orchestra.run.smithi097.stdout: "client_addr": "v1:172.21.15.92:0/436606717", 2024-03-11T21:35:44.700 INFO:teuthology.orchestra.run.smithi097.stdout: "tid": 208601 2024-03-11T21:35:44.700 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:35:44.700 INFO:teuthology.orchestra.run.smithi097.stdout: "events": [ 2024-03-11T21:35:44.700 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:35:44.700 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "initiated", 2024-03-11T21:35:44.700 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:34:57.372084+0000", 2024-03-11T21:35:44.700 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-03-11T21:35:44.700 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:35:44.700 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:35:44.700 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "header_read", 2024-03-11T21:35:44.700 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:34:57.372084+0000", 2024-03-11T21:35:44.700 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-03-11T21:35:44.700 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:35:44.700 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:35:44.701 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "throttled", 2024-03-11T21:35:44.701 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:34:57.372086+0000", 2024-03-11T21:35:44.701 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.5209999999999999e-06 2024-03-11T21:35:44.701 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:35:44.701 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:35:44.701 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "all_read", 2024-03-11T21:35:44.701 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:34:57.372090+0000", 2024-03-11T21:35:44.701 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 4.5909999999999998e-06 2024-03-11T21:35:44.701 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:35:44.701 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:35:44.701 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "dispatched", 2024-03-11T21:35:44.701 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:34:57.372101+0000", 2024-03-11T21:35:44.701 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.0912e-05 2024-03-11T21:35:44.701 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:35:44.701 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:35:44.702 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "queued_for_pg", 2024-03-11T21:35:44.702 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:34:57.372117+0000", 2024-03-11T21:35:44.702 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.5247e-05 2024-03-11T21:35:44.702 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:35:44.702 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:35:44.702 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "reached_pg", 2024-03-11T21:35:44.702 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:34:57.372177+0000", 2024-03-11T21:35:44.702 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 5.9957999999999998e-05 2024-03-11T21:35:44.702 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:35:44.702 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:35:44.702 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "waiting for peered", 2024-03-11T21:35:44.702 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:34:57.372186+0000", 2024-03-11T21:35:44.702 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 9.5750000000000007e-06 2024-03-11T21:35:44.702 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:35:44.702 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:35:44.703 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "reached_pg", 2024-03-11T21:35:44.703 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:34:58.388134+0000", 2024-03-11T21:35:44.703 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.0159478159999999 2024-03-11T21:35:44.703 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:35:44.703 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:35:44.703 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "started", 2024-03-11T21:35:44.703 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:34:58.388202+0000", 2024-03-11T21:35:44.703 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 6.7717000000000002e-05 2024-03-11T21:35:44.703 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:35:44.703 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:35:44.703 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "waiting for subops from 2,7", 2024-03-11T21:35:44.703 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:34:58.388274+0000", 2024-03-11T21:35:44.703 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 7.2342999999999995e-05 2024-03-11T21:35:44.704 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:35:44.704 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:35:44.704 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "sub_op_commit_rec", 2024-03-11T21:35:44.704 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:34:58.393561+0000", 2024-03-11T21:35:44.704 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.0052871180000000004 2024-03-11T21:35:44.704 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:35:44.704 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:35:44.704 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "sub_op_commit_rec", 2024-03-11T21:35:44.704 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:34:58.393612+0000", 2024-03-11T21:35:44.704 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 5.0923e-05 2024-03-11T21:35:44.704 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:35:44.704 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:35:44.704 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "op_commit", 2024-03-11T21:35:44.704 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:34:58.393630+0000", 2024-03-11T21:35:44.705 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.7716000000000001e-05 2024-03-11T21:35:44.705 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:35:44.705 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:35:44.705 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "commit_sent", 2024-03-11T21:35:44.705 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:34:58.393683+0000", 2024-03-11T21:35:44.705 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 5.3656999999999998e-05 2024-03-11T21:35:44.705 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:35:44.705 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:35:44.705 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "done", 2024-03-11T21:35:44.705 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:34:58.393734+0000", 2024-03-11T21:35:44.705 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 5.0249e-05 2024-03-11T21:35:44.705 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-03-11T21:35:44.705 INFO:teuthology.orchestra.run.smithi097.stdout: ] 2024-03-11T21:35:44.705 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-03-11T21:35:44.706 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:35:44.706 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:35:44.706 INFO:teuthology.orchestra.run.smithi097.stdout: "description": "osd_op(client.15150.0:208608 8.d 8:b39a191f:::benchmark_data_smithi092_104865_object88107:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e239)", 2024-03-11T21:35:44.706 INFO:teuthology.orchestra.run.smithi097.stdout: "initiated_at": "2024-03-11T21:34:57.377729+0000", 2024-03-11T21:35:44.706 INFO:teuthology.orchestra.run.smithi097.stdout: "age": 47.265860177999997, 2024-03-11T21:35:44.706 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.016253541, 2024-03-11T21:35:44.706 INFO:teuthology.orchestra.run.smithi097.stdout: "type_data": { 2024-03-11T21:35:44.706 INFO:teuthology.orchestra.run.smithi097.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-11T21:35:44.706 INFO:teuthology.orchestra.run.smithi097.stdout: "client_info": { 2024-03-11T21:35:44.706 INFO:teuthology.orchestra.run.smithi097.stdout: "client": "client.15150", 2024-03-11T21:35:44.706 INFO:teuthology.orchestra.run.smithi097.stdout: "client_addr": "v1:172.21.15.92:0/436606717", 2024-03-11T21:35:44.706 INFO:teuthology.orchestra.run.smithi097.stdout: "tid": 208608 2024-03-11T21:35:44.706 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:35:44.706 INFO:teuthology.orchestra.run.smithi097.stdout: "events": [ 2024-03-11T21:35:44.707 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:35:44.707 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "initiated", 2024-03-11T21:35:44.707 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:34:57.377729+0000", 2024-03-11T21:35:44.707 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-03-11T21:35:44.707 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:35:44.707 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:35:44.707 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "header_read", 2024-03-11T21:35:44.707 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:34:57.377729+0000", 2024-03-11T21:35:44.707 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-03-11T21:35:44.707 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:35:44.707 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:35:44.707 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "throttled", 2024-03-11T21:35:44.707 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:34:57.377731+0000", 2024-03-11T21:35:44.707 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 2.1059999999999998e-06 2024-03-11T21:35:44.708 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:35:44.708 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:35:44.708 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "all_read", 2024-03-11T21:35:44.708 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:34:57.377739+0000", 2024-03-11T21:35:44.708 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 8.5110000000000003e-06 2024-03-11T21:35:44.708 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:35:44.708 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:35:44.708 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "dispatched", 2024-03-11T21:35:44.708 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:34:57.377752+0000", 2024-03-11T21:35:44.708 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.2523e-05 2024-03-11T21:35:44.708 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:35:44.708 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:35:44.708 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "queued_for_pg", 2024-03-11T21:35:44.709 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:34:57.377760+0000", 2024-03-11T21:35:44.709 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 7.8490000000000006e-06 2024-03-11T21:35:44.709 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:35:44.709 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:35:44.709 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "reached_pg", 2024-03-11T21:35:44.709 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:34:57.377823+0000", 2024-03-11T21:35:44.709 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 6.3337999999999996e-05 2024-03-11T21:35:44.709 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:35:44.709 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:35:44.709 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "waiting for peered", 2024-03-11T21:35:44.709 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:34:57.377833+0000", 2024-03-11T21:35:44.709 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 9.7070000000000004e-06 2024-03-11T21:35:44.709 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:35:44.709 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:35:44.709 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "reached_pg", 2024-03-11T21:35:44.710 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:34:58.390377+0000", 2024-03-11T21:35:44.710 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.0125443629999999 2024-03-11T21:35:44.710 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:35:44.710 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:35:44.710 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "started", 2024-03-11T21:35:44.710 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:34:58.390455+0000", 2024-03-11T21:35:44.710 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 7.7832999999999998e-05 2024-03-11T21:35:44.710 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:35:44.710 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:35:44.710 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "waiting for subops from 2,7", 2024-03-11T21:35:44.710 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:34:58.390541+0000", 2024-03-11T21:35:44.710 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 8.6014999999999996e-05 2024-03-11T21:35:44.710 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:35:44.711 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:35:44.711 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "sub_op_commit_rec", 2024-03-11T21:35:44.711 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:34:58.393817+0000", 2024-03-11T21:35:44.711 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.0032762799999999999 2024-03-11T21:35:44.711 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:35:44.711 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:35:44.711 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "op_commit", 2024-03-11T21:35:44.711 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:34:58.393832+0000", 2024-03-11T21:35:44.711 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.4773999999999999e-05 2024-03-11T21:35:44.711 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:35:44.711 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:35:44.711 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "sub_op_commit_rec", 2024-03-11T21:35:44.711 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:34:58.393890+0000", 2024-03-11T21:35:44.711 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 5.7969e-05 2024-03-11T21:35:44.712 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:35:44.712 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:35:44.712 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "commit_sent", 2024-03-11T21:35:44.712 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:34:58.393941+0000", 2024-03-11T21:35:44.712 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 5.1403999999999998e-05 2024-03-11T21:35:44.712 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:35:44.712 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:35:44.712 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "done", 2024-03-11T21:35:44.712 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:34:58.393982+0000", 2024-03-11T21:35:44.712 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 4.0868999999999998e-05 2024-03-11T21:35:44.712 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-03-11T21:35:44.712 INFO:teuthology.orchestra.run.smithi097.stdout: ] 2024-03-11T21:35:44.712 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-03-11T21:35:44.713 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:35:44.713 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:35:44.713 INFO:teuthology.orchestra.run.smithi097.stdout: "description": "osd_op(client.15150.0:208648 8.d 8:b7832715:::benchmark_data_smithi092_104865_object88147:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e239)", 2024-03-11T21:35:44.713 INFO:teuthology.orchestra.run.smithi097.stdout: "initiated_at": "2024-03-11T21:34:57.409741+0000", 2024-03-11T21:35:44.713 INFO:teuthology.orchestra.run.smithi097.stdout: "age": 47.233848348000002, 2024-03-11T21:35:44.713 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.99063108700000002, 2024-03-11T21:35:44.713 INFO:teuthology.orchestra.run.smithi097.stdout: "type_data": { 2024-03-11T21:35:44.713 INFO:teuthology.orchestra.run.smithi097.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-11T21:35:44.713 INFO:teuthology.orchestra.run.smithi097.stdout: "client_info": { 2024-03-11T21:35:44.713 INFO:teuthology.orchestra.run.smithi097.stdout: "client": "client.15150", 2024-03-11T21:35:44.713 INFO:teuthology.orchestra.run.smithi097.stdout: "client_addr": "v1:172.21.15.92:0/436606717", 2024-03-11T21:35:44.713 INFO:teuthology.orchestra.run.smithi097.stdout: "tid": 208648 2024-03-11T21:35:44.713 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:35:44.713 INFO:teuthology.orchestra.run.smithi097.stdout: "events": [ 2024-03-11T21:35:44.714 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:35:44.714 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "initiated", 2024-03-11T21:35:44.714 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:34:57.409741+0000", 2024-03-11T21:35:44.714 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-03-11T21:35:44.714 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:35:44.714 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:35:44.714 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "header_read", 2024-03-11T21:35:44.714 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:34:57.409741+0000", 2024-03-11T21:35:44.714 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-03-11T21:35:44.714 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:35:44.714 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:35:44.714 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "throttled", 2024-03-11T21:35:44.714 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:34:57.409743+0000", 2024-03-11T21:35:44.714 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 2.39e-06 2024-03-11T21:35:44.715 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:35:44.715 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:35:44.715 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "all_read", 2024-03-11T21:35:44.715 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:34:57.409749+0000", 2024-03-11T21:35:44.715 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 5.9200000000000001e-06 2024-03-11T21:35:44.715 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:35:44.715 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:35:44.715 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "dispatched", 2024-03-11T21:35:44.715 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:34:57.409764+0000", 2024-03-11T21:35:44.715 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.5427e-05 2024-03-11T21:35:44.715 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:35:44.715 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:35:44.715 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "queued_for_pg", 2024-03-11T21:35:44.715 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:34:57.409772+0000", 2024-03-11T21:35:44.716 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 7.7239999999999992e-06 2024-03-11T21:35:44.716 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:35:44.716 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:35:44.716 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "reached_pg", 2024-03-11T21:35:44.716 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:34:57.409829+0000", 2024-03-11T21:35:44.716 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 5.7169000000000001e-05 2024-03-11T21:35:44.716 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:35:44.716 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:35:44.716 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "waiting for peered", 2024-03-11T21:35:44.716 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:34:57.409837+0000", 2024-03-11T21:35:44.716 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 8.2379999999999997e-06 2024-03-11T21:35:44.716 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:35:44.716 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:35:44.716 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "reached_pg", 2024-03-11T21:35:44.716 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:34:58.394032+0000", 2024-03-11T21:35:44.717 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.98419469599999998 2024-03-11T21:35:44.717 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:35:44.717 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:35:44.717 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "started", 2024-03-11T21:35:44.717 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:34:58.394152+0000", 2024-03-11T21:35:44.717 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.00011963700000000001 2024-03-11T21:35:44.717 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:35:44.717 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:35:44.717 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "waiting for subops from 2,7", 2024-03-11T21:35:44.717 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:34:58.394262+0000", 2024-03-11T21:35:44.717 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.00010984600000000001 2024-03-11T21:35:44.717 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:35:44.717 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:35:44.717 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "sub_op_commit_rec", 2024-03-11T21:35:44.717 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:34:58.400153+0000", 2024-03-11T21:35:44.718 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.0058912019999999999 2024-03-11T21:35:44.718 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:35:44.718 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:35:44.718 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "sub_op_commit_rec", 2024-03-11T21:35:44.718 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:34:58.400199+0000", 2024-03-11T21:35:44.718 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 4.6400999999999998e-05 2024-03-11T21:35:44.718 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:35:44.718 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:35:44.718 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "op_commit", 2024-03-11T21:35:44.718 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:34:58.400212+0000", 2024-03-11T21:35:44.718 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.2258e-05 2024-03-11T21:35:44.718 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:35:44.718 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:35:44.718 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "commit_sent", 2024-03-11T21:35:44.719 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:34:58.400326+0000", 2024-03-11T21:35:44.719 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.000114855 2024-03-11T21:35:44.719 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:35:44.719 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:35:44.719 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "done", 2024-03-11T21:35:44.719 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:34:58.400372+0000", 2024-03-11T21:35:44.719 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 4.5324000000000002e-05 2024-03-11T21:35:44.719 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-03-11T21:35:44.719 INFO:teuthology.orchestra.run.smithi097.stdout: ] 2024-03-11T21:35:44.719 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-03-11T21:35:44.719 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:35:44.719 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:35:44.719 INFO:teuthology.orchestra.run.smithi097.stdout: "description": "osd_op(client.15150.0:208683 8.d 8:ba654bd9:::benchmark_data_smithi092_104865_object88182:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e239)", 2024-03-11T21:35:44.719 INFO:teuthology.orchestra.run.smithi097.stdout: "initiated_at": "2024-03-11T21:34:57.434209+0000", 2024-03-11T21:35:44.721 INFO:teuthology.orchestra.run.smithi097.stdout: "age": 47.209379738000003, 2024-03-11T21:35:44.721 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.97204510700000002, 2024-03-11T21:35:44.721 INFO:teuthology.orchestra.run.smithi097.stdout: "type_data": { 2024-03-11T21:35:44.721 INFO:teuthology.orchestra.run.smithi097.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-11T21:35:44.721 INFO:teuthology.orchestra.run.smithi097.stdout: "client_info": { 2024-03-11T21:35:44.721 INFO:teuthology.orchestra.run.smithi097.stdout: "client": "client.15150", 2024-03-11T21:35:44.721 INFO:teuthology.orchestra.run.smithi097.stdout: "client_addr": "v1:172.21.15.92:0/436606717", 2024-03-11T21:35:44.721 INFO:teuthology.orchestra.run.smithi097.stdout: "tid": 208683 2024-03-11T21:35:44.721 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:35:44.721 INFO:teuthology.orchestra.run.smithi097.stdout: "events": [ 2024-03-11T21:35:44.722 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:35:44.722 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "initiated", 2024-03-11T21:35:44.722 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:34:57.434209+0000", 2024-03-11T21:35:44.722 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-03-11T21:35:44.722 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:35:44.722 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:35:44.722 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "header_read", 2024-03-11T21:35:44.722 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:34:57.434209+0000", 2024-03-11T21:35:44.722 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-03-11T21:35:44.722 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:35:44.722 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:35:44.722 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "throttled", 2024-03-11T21:35:44.722 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:34:57.434211+0000", 2024-03-11T21:35:44.722 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 2.1579999999999999e-06 2024-03-11T21:35:44.723 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:35:44.723 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:35:44.723 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "all_read", 2024-03-11T21:35:44.723 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:34:57.434216+0000", 2024-03-11T21:35:44.723 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 4.9189999999999998e-06 2024-03-11T21:35:44.723 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:35:44.723 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:35:44.723 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "dispatched", 2024-03-11T21:35:44.723 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:34:57.434228+0000", 2024-03-11T21:35:44.723 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.1389000000000001e-05 2024-03-11T21:35:44.723 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:35:44.723 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:35:44.723 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "queued_for_pg", 2024-03-11T21:35:44.723 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:34:57.434235+0000", 2024-03-11T21:35:44.724 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 7.1130000000000003e-06 2024-03-11T21:35:44.724 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:35:44.724 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:35:44.724 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "reached_pg", 2024-03-11T21:35:44.724 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:34:57.434292+0000", 2024-03-11T21:35:44.724 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 5.7268999999999997e-05 2024-03-11T21:35:44.724 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:35:44.724 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:35:44.724 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "waiting for peered", 2024-03-11T21:35:44.724 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:34:57.434302+0000", 2024-03-11T21:35:44.724 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.0339e-05 2024-03-11T21:35:44.724 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:35:44.724 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:35:44.724 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "reached_pg", 2024-03-11T21:35:44.724 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:34:58.396996+0000", 2024-03-11T21:35:44.725 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.962694033 2024-03-11T21:35:44.725 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:35:44.725 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:35:44.725 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "started", 2024-03-11T21:35:44.725 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:34:58.397103+0000", 2024-03-11T21:35:44.725 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.000106118 2024-03-11T21:35:44.725 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:35:44.725 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:35:44.725 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "waiting for subops from 2,7", 2024-03-11T21:35:44.725 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:34:58.397243+0000", 2024-03-11T21:35:44.725 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.00014001399999999999 2024-03-11T21:35:44.725 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:35:44.725 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:35:44.726 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "sub_op_commit_rec", 2024-03-11T21:35:44.726 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:34:58.400415+0000", 2024-03-11T21:35:44.726 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.003172186 2024-03-11T21:35:44.726 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:35:44.726 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:35:44.726 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "sub_op_commit_rec", 2024-03-11T21:35:44.726 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:34:58.400496+0000", 2024-03-11T21:35:44.726 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 8.0828000000000006e-05 2024-03-11T21:35:44.726 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:35:44.726 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:35:44.726 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "op_commit", 2024-03-11T21:35:44.726 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:34:58.406159+0000", 2024-03-11T21:35:44.726 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.005663609 2024-03-11T21:35:44.727 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:35:44.727 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:35:44.727 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "commit_sent", 2024-03-11T21:35:44.727 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:34:58.406198+0000", 2024-03-11T21:35:44.727 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 3.9069000000000002e-05 2024-03-11T21:35:44.727 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:35:44.727 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:35:44.727 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "done", 2024-03-11T21:35:44.727 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:34:58.406254+0000", 2024-03-11T21:35:44.727 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 5.6063000000000002e-05 2024-03-11T21:35:44.727 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-03-11T21:35:44.727 INFO:teuthology.orchestra.run.smithi097.stdout: ] 2024-03-11T21:35:44.727 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-03-11T21:35:44.727 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:35:44.728 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:35:44.728 INFO:teuthology.orchestra.run.smithi097.stdout: "description": "osd_op(client.15150.0:208685 8.d 8:b7ae4ce9:::benchmark_data_smithi092_104865_object88184:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e239)", 2024-03-11T21:35:44.728 INFO:teuthology.orchestra.run.smithi097.stdout: "initiated_at": "2024-03-11T21:34:57.436752+0000", 2024-03-11T21:35:44.728 INFO:teuthology.orchestra.run.smithi097.stdout: "age": 47.206836494999997, 2024-03-11T21:35:44.728 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.969736033, 2024-03-11T21:35:44.728 INFO:teuthology.orchestra.run.smithi097.stdout: "type_data": { 2024-03-11T21:35:44.728 INFO:teuthology.orchestra.run.smithi097.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-11T21:35:44.728 INFO:teuthology.orchestra.run.smithi097.stdout: "client_info": { 2024-03-11T21:35:44.728 INFO:teuthology.orchestra.run.smithi097.stdout: "client": "client.15150", 2024-03-11T21:35:44.728 INFO:teuthology.orchestra.run.smithi097.stdout: "client_addr": "v1:172.21.15.92:0/436606717", 2024-03-11T21:35:44.728 INFO:teuthology.orchestra.run.smithi097.stdout: "tid": 208685 2024-03-11T21:35:44.728 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:35:44.728 INFO:teuthology.orchestra.run.smithi097.stdout: "events": [ 2024-03-11T21:35:44.728 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:35:44.729 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "initiated", 2024-03-11T21:35:44.729 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:34:57.436752+0000", 2024-03-11T21:35:44.729 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-03-11T21:35:44.729 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:35:44.729 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:35:44.729 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "header_read", 2024-03-11T21:35:44.729 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:34:57.436752+0000", 2024-03-11T21:35:44.729 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-03-11T21:35:44.729 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:35:44.729 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:35:44.729 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "throttled", 2024-03-11T21:35:44.729 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:34:57.436754+0000", 2024-03-11T21:35:44.729 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.319e-06 2024-03-11T21:35:44.729 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:35:44.730 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:35:44.730 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "all_read", 2024-03-11T21:35:44.730 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:34:57.436758+0000", 2024-03-11T21:35:44.730 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 4.6700000000000002e-06 2024-03-11T21:35:44.730 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:35:44.730 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:35:44.730 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "dispatched", 2024-03-11T21:35:44.730 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:34:57.436768+0000", 2024-03-11T21:35:44.730 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 9.8989999999999996e-06 2024-03-11T21:35:44.730 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:35:44.730 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:35:44.730 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "queued_for_pg", 2024-03-11T21:35:44.731 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:34:57.436776+0000", 2024-03-11T21:35:44.731 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 7.9279999999999993e-06 2024-03-11T21:35:44.731 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:35:44.731 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:35:44.731 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "reached_pg", 2024-03-11T21:35:44.731 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:34:57.436828+0000", 2024-03-11T21:35:44.731 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 5.2026e-05 2024-03-11T21:35:44.731 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:35:44.731 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:35:44.731 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "waiting for peered", 2024-03-11T21:35:44.731 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:34:57.436836+0000", 2024-03-11T21:35:44.731 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 7.9359999999999999e-06 2024-03-11T21:35:44.731 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:35:44.731 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:35:44.731 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "reached_pg", 2024-03-11T21:35:44.732 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:34:58.400540+0000", 2024-03-11T21:35:44.732 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.96370411899999997 2024-03-11T21:35:44.732 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:35:44.732 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:35:44.732 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "started", 2024-03-11T21:35:44.732 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:34:58.400635+0000", 2024-03-11T21:35:44.732 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 9.4242999999999999e-05 2024-03-11T21:35:44.732 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:35:44.732 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:35:44.732 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "waiting for subops from 2,7", 2024-03-11T21:35:44.732 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:34:58.400737+0000", 2024-03-11T21:35:44.732 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.00010212 2024-03-11T21:35:44.732 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:35:44.732 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:35:44.733 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "sub_op_commit_rec", 2024-03-11T21:35:44.733 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:34:58.406321+0000", 2024-03-11T21:35:44.733 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.0055847459999999998 2024-03-11T21:35:44.733 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:35:44.733 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:35:44.733 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "sub_op_commit_rec", 2024-03-11T21:35:44.733 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:34:58.406385+0000", 2024-03-11T21:35:44.733 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 6.4005999999999997e-05 2024-03-11T21:35:44.733 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:35:44.733 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:35:44.733 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "op_commit", 2024-03-11T21:35:44.733 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:34:58.406401+0000", 2024-03-11T21:35:44.733 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.5627000000000001e-05 2024-03-11T21:35:44.733 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:35:44.734 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:35:44.734 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "commit_sent", 2024-03-11T21:35:44.734 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:34:58.406448+0000", 2024-03-11T21:35:44.734 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 4.6703000000000003e-05 2024-03-11T21:35:44.734 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:35:44.734 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:35:44.734 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "done", 2024-03-11T21:35:44.734 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:34:58.406489+0000", 2024-03-11T21:35:44.734 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 4.0691000000000001e-05 2024-03-11T21:35:44.734 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-03-11T21:35:44.734 INFO:teuthology.orchestra.run.smithi097.stdout: ] 2024-03-11T21:35:44.734 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-03-11T21:35:44.734 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:35:44.734 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:35:44.735 INFO:teuthology.orchestra.run.smithi097.stdout: "description": "osd_op(client.15150.0:208705 8.d 8:bfee221f:::benchmark_data_smithi092_104865_object88204:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e239)", 2024-03-11T21:35:44.735 INFO:teuthology.orchestra.run.smithi097.stdout: "initiated_at": "2024-03-11T21:34:57.452387+0000", 2024-03-11T21:35:44.735 INFO:teuthology.orchestra.run.smithi097.stdout: "age": 47.191202306000001, 2024-03-11T21:35:44.735 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.95427008800000002, 2024-03-11T21:35:44.735 INFO:teuthology.orchestra.run.smithi097.stdout: "type_data": { 2024-03-11T21:35:44.735 INFO:teuthology.orchestra.run.smithi097.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-11T21:35:44.735 INFO:teuthology.orchestra.run.smithi097.stdout: "client_info": { 2024-03-11T21:35:44.735 INFO:teuthology.orchestra.run.smithi097.stdout: "client": "client.15150", 2024-03-11T21:35:44.735 INFO:teuthology.orchestra.run.smithi097.stdout: "client_addr": "v1:172.21.15.92:0/436606717", 2024-03-11T21:35:44.735 INFO:teuthology.orchestra.run.smithi097.stdout: "tid": 208705 2024-03-11T21:35:44.735 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:35:44.735 INFO:teuthology.orchestra.run.smithi097.stdout: "events": [ 2024-03-11T21:35:44.735 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:35:44.735 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "initiated", 2024-03-11T21:35:44.735 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:34:57.452387+0000", 2024-03-11T21:35:44.736 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-03-11T21:35:44.736 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:35:44.736 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:35:44.736 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "header_read", 2024-03-11T21:35:44.736 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:34:57.452387+0000", 2024-03-11T21:35:44.736 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-03-11T21:35:44.736 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:35:44.736 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:35:44.736 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "throttled", 2024-03-11T21:35:44.736 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:34:57.452388+0000", 2024-03-11T21:35:44.736 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.463e-06 2024-03-11T21:35:44.736 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:35:44.736 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:35:44.736 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "all_read", 2024-03-11T21:35:44.736 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:34:57.452392+0000", 2024-03-11T21:35:44.737 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 4.1779999999999996e-06 2024-03-11T21:35:44.737 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:35:44.737 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:35:44.737 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "dispatched", 2024-03-11T21:35:44.737 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:34:57.452402+0000", 2024-03-11T21:35:44.737 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.0029e-05 2024-03-11T21:35:44.737 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:35:44.737 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:35:44.737 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "queued_for_pg", 2024-03-11T21:35:44.737 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:34:57.452408+0000", 2024-03-11T21:35:44.737 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 5.5060000000000003e-06 2024-03-11T21:35:44.737 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:35:44.737 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:35:44.737 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "reached_pg", 2024-03-11T21:35:44.737 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:34:57.452463+0000", 2024-03-11T21:35:44.738 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 5.5161999999999998e-05 2024-03-11T21:35:44.738 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:35:44.738 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:35:44.738 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "waiting for peered", 2024-03-11T21:35:44.738 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:34:57.452469+0000", 2024-03-11T21:35:44.738 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 5.6339999999999998e-06 2024-03-11T21:35:44.738 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:35:44.738 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:35:44.738 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "reached_pg", 2024-03-11T21:35:44.738 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:34:58.403364+0000", 2024-03-11T21:35:44.738 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.95089549799999995 2024-03-11T21:35:44.738 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:35:44.738 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:35:44.738 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "started", 2024-03-11T21:35:44.739 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:34:58.403463+0000", 2024-03-11T21:35:44.739 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 9.8826999999999994e-05 2024-03-11T21:35:44.739 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:35:44.739 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:35:44.739 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "waiting for subops from 2,7", 2024-03-11T21:35:44.739 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:34:58.403571+0000", 2024-03-11T21:35:44.739 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.000108495 2024-03-11T21:35:44.739 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:35:44.739 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:35:44.739 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "sub_op_commit_rec", 2024-03-11T21:35:44.739 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:34:58.406528+0000", 2024-03-11T21:35:44.739 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.002956499 2024-03-11T21:35:44.739 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:35:44.739 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:35:44.739 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "op_commit", 2024-03-11T21:35:44.740 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:35:44.741 INFO:teuthology.orchestra.run.smithi092.stderr:pool 'unique_pool_2' created 2024-03-11T21:35:44.741 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:34:58.406540+0000", 2024-03-11T21:35:44.741 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.1873e-05 2024-03-11T21:35:44.741 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:35:44.741 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:35:44.742 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "sub_op_commit_rec", 2024-03-11T21:35:44.742 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:34:58.406592+0000", 2024-03-11T21:35:44.742 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 5.2373999999999997e-05 2024-03-11T21:35:44.742 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:35:44.742 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:35:44.742 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "commit_sent", 2024-03-11T21:35:44.742 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:34:58.406631+0000", 2024-03-11T21:35:44.742 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 3.8869999999999999e-05 2024-03-11T21:35:44.742 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:35:44.742 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:35:44.742 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "done", 2024-03-11T21:35:44.742 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:34:58.406657+0000", 2024-03-11T21:35:44.742 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 2.5680000000000001e-05 2024-03-11T21:35:44.742 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-03-11T21:35:44.743 INFO:teuthology.orchestra.run.smithi097.stdout: ] 2024-03-11T21:35:44.743 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-03-11T21:35:44.743 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:35:44.743 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:35:44.743 INFO:teuthology.orchestra.run.smithi097.stdout: "description": "osd_op(client.15150.0:208728 8.d 8:b9a1f7a4:::benchmark_data_smithi092_104865_object88227:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e239)", 2024-03-11T21:35:44.743 INFO:teuthology.orchestra.run.smithi097.stdout: "initiated_at": "2024-03-11T21:34:57.470992+0000", 2024-03-11T21:35:44.743 INFO:teuthology.orchestra.run.smithi097.stdout: "age": 47.172597166000003, 2024-03-11T21:35:44.743 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.94147666299999999, 2024-03-11T21:35:44.743 INFO:teuthology.orchestra.run.smithi097.stdout: "type_data": { 2024-03-11T21:35:44.743 INFO:teuthology.orchestra.run.smithi097.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-11T21:35:44.743 INFO:teuthology.orchestra.run.smithi097.stdout: "client_info": { 2024-03-11T21:35:44.743 INFO:teuthology.orchestra.run.smithi097.stdout: "client": "client.15150", 2024-03-11T21:35:44.743 INFO:teuthology.orchestra.run.smithi097.stdout: "client_addr": "v1:172.21.15.92:0/436606717", 2024-03-11T21:35:44.743 INFO:teuthology.orchestra.run.smithi097.stdout: "tid": 208728 2024-03-11T21:35:44.744 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:35:44.744 INFO:teuthology.orchestra.run.smithi097.stdout: "events": [ 2024-03-11T21:35:44.744 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:35:44.744 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "initiated", 2024-03-11T21:35:44.744 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:34:57.470992+0000", 2024-03-11T21:35:44.744 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-03-11T21:35:44.744 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:35:44.744 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:35:44.744 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "header_read", 2024-03-11T21:35:44.744 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:34:57.470992+0000", 2024-03-11T21:35:44.744 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-03-11T21:35:44.744 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:35:44.744 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:35:44.745 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "throttled", 2024-03-11T21:35:44.745 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:34:57.470993+0000", 2024-03-11T21:35:44.745 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.345e-06 2024-03-11T21:35:44.745 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:35:44.745 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:35:44.745 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "all_read", 2024-03-11T21:35:44.745 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:34:57.470998+0000", 2024-03-11T21:35:44.745 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 5.2209999999999996e-06 2024-03-11T21:35:44.745 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:35:44.745 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:35:44.745 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "dispatched", 2024-03-11T21:35:44.745 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:34:57.471013+0000", 2024-03-11T21:35:44.745 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.4208e-05 2024-03-11T21:35:44.745 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:35:44.745 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:35:44.746 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "queued_for_pg", 2024-03-11T21:35:44.746 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:34:57.471022+0000", 2024-03-11T21:35:44.746 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 8.9239999999999996e-06 2024-03-11T21:35:44.746 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:35:44.746 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:35:44.746 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "reached_pg", 2024-03-11T21:35:44.746 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:34:57.471088+0000", 2024-03-11T21:35:44.746 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 6.6729000000000004e-05 2024-03-11T21:35:44.746 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:35:44.746 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:35:44.746 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "waiting for peered", 2024-03-11T21:35:44.746 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:34:57.471097+0000", 2024-03-11T21:35:44.746 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 8.8689999999999993e-06 2024-03-11T21:35:44.747 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:35:44.747 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:35:44.747 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "reached_pg", 2024-03-11T21:35:44.747 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:34:58.406704+0000", 2024-03-11T21:35:44.747 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.93560714199999995 2024-03-11T21:35:44.747 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:35:44.747 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:35:44.747 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "started", 2024-03-11T21:35:44.747 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:34:58.406832+0000", 2024-03-11T21:35:44.747 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.000127274 2024-03-11T21:35:44.747 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:35:44.747 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:35:44.747 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "waiting for subops from 2,7", 2024-03-11T21:35:44.748 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:34:58.406945+0000", 2024-03-11T21:35:44.748 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.000113712 2024-03-11T21:35:44.748 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:35:44.748 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:35:44.748 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "sub_op_commit_rec", 2024-03-11T21:35:44.748 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:34:58.412224+0000", 2024-03-11T21:35:44.748 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.0052792100000000003 2024-03-11T21:35:44.748 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:35:44.748 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:35:44.748 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "sub_op_commit_rec", 2024-03-11T21:35:44.748 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:34:58.412355+0000", 2024-03-11T21:35:44.748 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.00013081 2024-03-11T21:35:44.748 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:35:44.748 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:35:44.749 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "op_commit", 2024-03-11T21:35:44.749 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:34:58.412370+0000", 2024-03-11T21:35:44.749 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.4664999999999999e-05 2024-03-11T21:35:44.749 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:35:44.749 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:35:44.749 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "commit_sent", 2024-03-11T21:35:44.749 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:34:58.412419+0000", 2024-03-11T21:35:44.749 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 4.8874e-05 2024-03-11T21:35:44.749 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:35:44.749 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:35:44.749 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "done", 2024-03-11T21:35:44.749 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:34:58.412468+0000", 2024-03-11T21:35:44.749 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 4.9679999999999999e-05 2024-03-11T21:35:44.749 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-03-11T21:35:44.749 INFO:teuthology.orchestra.run.smithi097.stdout: ] 2024-03-11T21:35:44.750 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-03-11T21:35:44.750 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:35:44.750 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:35:44.750 INFO:teuthology.orchestra.run.smithi097.stdout: "description": "osd_op(client.15150.0:208734 8.d 8:b921f92e:::benchmark_data_smithi092_104865_object88233:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e239)", 2024-03-11T21:35:44.750 INFO:teuthology.orchestra.run.smithi097.stdout: "initiated_at": "2024-03-11T21:34:57.475559+0000", 2024-03-11T21:35:44.750 INFO:teuthology.orchestra.run.smithi097.stdout: "age": 47.168030148, 2024-03-11T21:35:44.750 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.93958550399999996, 2024-03-11T21:35:44.750 INFO:teuthology.orchestra.run.smithi097.stdout: "type_data": { 2024-03-11T21:35:44.750 INFO:teuthology.orchestra.run.smithi097.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-11T21:35:44.750 INFO:teuthology.orchestra.run.smithi097.stdout: "client_info": { 2024-03-11T21:35:44.750 INFO:teuthology.orchestra.run.smithi097.stdout: "client": "client.15150", 2024-03-11T21:35:44.750 INFO:teuthology.orchestra.run.smithi097.stdout: "client_addr": "v1:172.21.15.92:0/436606717", 2024-03-11T21:35:44.750 INFO:teuthology.orchestra.run.smithi097.stdout: "tid": 208734 2024-03-11T21:35:44.751 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:35:44.751 INFO:teuthology.orchestra.run.smithi097.stdout: "events": [ 2024-03-11T21:35:44.751 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:35:44.751 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "initiated", 2024-03-11T21:35:44.751 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:34:57.475559+0000", 2024-03-11T21:35:44.751 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-03-11T21:35:44.751 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:35:44.751 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:35:44.751 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "header_read", 2024-03-11T21:35:44.751 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:34:57.475559+0000", 2024-03-11T21:35:44.751 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-03-11T21:35:44.751 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:35:44.751 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:35:44.751 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "throttled", 2024-03-11T21:35:44.752 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:34:57.475560+0000", 2024-03-11T21:35:44.752 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.637e-06 2024-03-11T21:35:44.752 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:35:44.752 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:35:44.752 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "all_read", 2024-03-11T21:35:44.752 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:34:57.475565+0000", 2024-03-11T21:35:44.752 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 4.7940000000000002e-06 2024-03-11T21:35:44.752 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:35:44.752 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:35:44.752 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "dispatched", 2024-03-11T21:35:44.752 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:34:57.475576+0000", 2024-03-11T21:35:44.752 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.0828e-05 2024-03-11T21:35:44.752 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:35:44.752 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:35:44.752 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "queued_for_pg", 2024-03-11T21:35:44.753 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:34:57.475584+0000", 2024-03-11T21:35:44.753 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 7.7330000000000003e-06 2024-03-11T21:35:44.753 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:35:44.753 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:35:44.753 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "reached_pg", 2024-03-11T21:35:44.753 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:34:57.475643+0000", 2024-03-11T21:35:44.753 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 5.9024999999999998e-05 2024-03-11T21:35:44.753 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:35:44.753 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:35:44.753 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "waiting for peered", 2024-03-11T21:35:44.753 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:34:57.475652+0000", 2024-03-11T21:35:44.753 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 9.3980000000000005e-06 2024-03-11T21:35:44.753 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:35:44.753 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:35:44.754 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "reached_pg", 2024-03-11T21:35:44.754 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:34:58.409488+0000", 2024-03-11T21:35:44.754 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.93383537000000005 2024-03-11T21:35:44.754 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:35:44.754 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:35:44.754 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "started", 2024-03-11T21:35:44.754 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:34:58.409573+0000", 2024-03-11T21:35:44.754 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 8.5628000000000001e-05 2024-03-11T21:35:44.754 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:35:44.754 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:35:44.754 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "waiting for subops from 2,7", 2024-03-11T21:35:44.754 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:34:58.409674+0000", 2024-03-11T21:35:44.754 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.000101148 2024-03-11T21:35:44.755 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:35:44.755 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:35:44.755 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "sub_op_commit_rec", 2024-03-11T21:35:44.755 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:34:58.412284+0000", 2024-03-11T21:35:44.755 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.002610064 2024-03-11T21:35:44.755 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:35:44.755 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:35:44.755 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "sub_op_commit_rec", 2024-03-11T21:35:44.755 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:34:58.415041+0000", 2024-03-11T21:35:44.755 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.0027566399999999999 2024-03-11T21:35:44.755 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:35:44.755 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:35:44.755 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "op_commit", 2024-03-11T21:35:44.755 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:34:58.415050+0000", 2024-03-11T21:35:44.756 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 8.4309999999999994e-06 2024-03-11T21:35:44.756 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:35:44.756 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:35:44.756 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "commit_sent", 2024-03-11T21:35:44.756 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:34:58.415091+0000", 2024-03-11T21:35:44.756 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 4.1628999999999997e-05 2024-03-11T21:35:44.756 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:35:44.756 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:35:44.756 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "done", 2024-03-11T21:35:44.756 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:34:58.415144+0000", 2024-03-11T21:35:44.756 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 5.3179e-05 2024-03-11T21:35:44.756 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-03-11T21:35:44.756 INFO:teuthology.orchestra.run.smithi097.stdout: ] 2024-03-11T21:35:44.756 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-03-11T21:35:44.757 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:35:44.757 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:35:44.757 INFO:teuthology.orchestra.run.smithi097.stdout: "description": "osd_op(client.15150.0:208746 8.d 8:b0e0235e:::benchmark_data_smithi092_104865_object88245:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e239)", 2024-03-11T21:35:44.757 INFO:teuthology.orchestra.run.smithi097.stdout: "initiated_at": "2024-03-11T21:34:57.486144+0000", 2024-03-11T21:35:44.757 INFO:teuthology.orchestra.run.smithi097.stdout: "age": 47.157444472999998, 2024-03-11T21:35:44.757 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.93540660200000003, 2024-03-11T21:35:44.757 INFO:teuthology.orchestra.run.smithi097.stdout: "type_data": { 2024-03-11T21:35:44.757 INFO:teuthology.orchestra.run.smithi097.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-11T21:35:44.757 INFO:teuthology.orchestra.run.smithi097.stdout: "client_info": { 2024-03-11T21:35:44.757 INFO:teuthology.orchestra.run.smithi097.stdout: "client": "client.15150", 2024-03-11T21:35:44.757 INFO:teuthology.orchestra.run.smithi097.stdout: "client_addr": "v1:172.21.15.92:0/436606717", 2024-03-11T21:35:44.757 INFO:teuthology.orchestra.run.smithi097.stdout: "tid": 208746 2024-03-11T21:35:44.757 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:35:44.757 INFO:teuthology.orchestra.run.smithi097.stdout: "events": [ 2024-03-11T21:35:44.758 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:35:44.758 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "initiated", 2024-03-11T21:35:44.758 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:34:57.486144+0000", 2024-03-11T21:35:44.758 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-03-11T21:35:44.758 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:35:44.758 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:35:44.758 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "header_read", 2024-03-11T21:35:44.758 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:34:57.486144+0000", 2024-03-11T21:35:44.758 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-03-11T21:35:44.758 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:35:44.758 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:35:44.758 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "throttled", 2024-03-11T21:35:44.758 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:34:57.486146+0000", 2024-03-11T21:35:44.758 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.147e-06 2024-03-11T21:35:44.759 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:35:44.759 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:35:44.759 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "all_read", 2024-03-11T21:35:44.759 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:34:57.486150+0000", 2024-03-11T21:35:44.759 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 3.9949999999999999e-06 2024-03-11T21:35:44.759 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:35:44.759 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:35:44.759 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "dispatched", 2024-03-11T21:35:44.759 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:34:57.486160+0000", 2024-03-11T21:35:44.759 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.0251000000000001e-05 2024-03-11T21:35:44.759 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:35:44.759 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:35:44.759 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "queued_for_pg", 2024-03-11T21:35:44.759 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:34:57.486166+0000", 2024-03-11T21:35:44.760 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 5.6790000000000002e-06 2024-03-11T21:35:44.760 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:35:44.760 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:35:44.760 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "reached_pg", 2024-03-11T21:35:44.760 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:34:57.486265+0000", 2024-03-11T21:35:44.760 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 9.9528e-05 2024-03-11T21:35:44.760 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:35:44.760 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:35:44.760 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "waiting for peered", 2024-03-11T21:35:44.760 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:34:57.486274+0000", 2024-03-11T21:35:44.760 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 9.1649999999999995e-06 2024-03-11T21:35:44.760 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:35:44.760 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:35:44.760 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "reached_pg", 2024-03-11T21:35:44.761 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:34:58.412511+0000", 2024-03-11T21:35:44.761 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.92623630099999998 2024-03-11T21:35:44.761 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:35:44.761 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:35:44.761 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "started", 2024-03-11T21:35:44.761 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:34:58.412602+0000", 2024-03-11T21:35:44.761 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 9.1260000000000004e-05 2024-03-11T21:35:44.761 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:35:44.761 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:35:44.761 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "waiting for subops from 2,7", 2024-03-11T21:35:44.761 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:34:58.412680+0000", 2024-03-11T21:35:44.761 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 7.8398000000000004e-05 2024-03-11T21:35:44.761 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:35:44.761 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:35:44.762 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "sub_op_commit_rec", 2024-03-11T21:35:44.762 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:34:58.415199+0000", 2024-03-11T21:35:44.762 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.0025187299999999998 2024-03-11T21:35:44.762 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:35:44.762 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:35:44.762 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "op_commit", 2024-03-11T21:35:44.762 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:34:58.417858+0000", 2024-03-11T21:35:44.762 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.0026595339999999999 2024-03-11T21:35:44.762 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:35:44.762 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:35:44.762 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "sub_op_commit_rec", 2024-03-11T21:35:44.762 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:34:58.421458+0000", 2024-03-11T21:35:44.762 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.0035995939999999998 2024-03-11T21:35:44.762 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:35:44.763 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:35:44.763 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "commit_sent", 2024-03-11T21:35:44.763 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:34:58.421509+0000", 2024-03-11T21:35:44.763 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 5.0988e-05 2024-03-11T21:35:44.763 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:35:44.763 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:35:44.763 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "done", 2024-03-11T21:35:44.763 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:34:58.421551+0000", 2024-03-11T21:35:44.763 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 4.2032e-05 2024-03-11T21:35:44.763 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-03-11T21:35:44.763 INFO:teuthology.orchestra.run.smithi097.stdout: ] 2024-03-11T21:35:44.763 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-03-11T21:35:44.763 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:35:44.763 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:35:44.763 INFO:teuthology.orchestra.run.smithi097.stdout: "description": "osd_op(client.15150.0:208771 8.d 8:b9357370:::benchmark_data_smithi092_104865_object88270:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e239)", 2024-03-11T21:35:44.764 INFO:teuthology.orchestra.run.smithi097.stdout: "initiated_at": "2024-03-11T21:34:57.511703+0000", 2024-03-11T21:35:44.764 INFO:teuthology.orchestra.run.smithi097.stdout: "age": 47.131886168000001, 2024-03-11T21:35:44.764 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.91534597799999995, 2024-03-11T21:35:44.764 INFO:teuthology.orchestra.run.smithi097.stdout: "type_data": { 2024-03-11T21:35:44.764 INFO:teuthology.orchestra.run.smithi097.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-11T21:35:44.764 INFO:teuthology.orchestra.run.smithi097.stdout: "client_info": { 2024-03-11T21:35:44.764 INFO:teuthology.orchestra.run.smithi097.stdout: "client": "client.15150", 2024-03-11T21:35:44.764 INFO:teuthology.orchestra.run.smithi097.stdout: "client_addr": "v1:172.21.15.92:0/436606717", 2024-03-11T21:35:44.764 INFO:teuthology.orchestra.run.smithi097.stdout: "tid": 208771 2024-03-11T21:35:44.764 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:35:44.764 INFO:teuthology.orchestra.run.smithi097.stdout: "events": [ 2024-03-11T21:35:44.764 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:35:44.764 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "initiated", 2024-03-11T21:35:44.764 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:34:57.511703+0000", 2024-03-11T21:35:44.764 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-03-11T21:35:44.765 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:35:44.765 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:35:44.765 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "header_read", 2024-03-11T21:35:44.765 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:34:57.511703+0000", 2024-03-11T21:35:44.765 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-03-11T21:35:44.765 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:35:44.765 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:35:44.765 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "throttled", 2024-03-11T21:35:44.765 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:34:57.511704+0000", 2024-03-11T21:35:44.765 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.423e-06 2024-03-11T21:35:44.765 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:35:44.765 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:35:44.765 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "all_read", 2024-03-11T21:35:44.765 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:34:57.511709+0000", 2024-03-11T21:35:44.766 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 4.7700000000000001e-06 2024-03-11T21:35:44.766 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:35:44.766 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:35:44.766 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "dispatched", 2024-03-11T21:35:44.766 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:34:57.511720+0000", 2024-03-11T21:35:44.766 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.0813e-05 2024-03-11T21:35:44.766 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:35:44.766 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:35:44.766 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "queued_for_pg", 2024-03-11T21:35:44.766 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:34:57.511728+0000", 2024-03-11T21:35:44.766 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 8.5520000000000005e-06 2024-03-11T21:35:44.766 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:35:44.766 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:35:44.766 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "reached_pg", 2024-03-11T21:35:44.766 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:34:57.511780+0000", 2024-03-11T21:35:44.767 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 5.2070000000000001e-05 2024-03-11T21:35:44.767 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:35:44.767 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:35:44.767 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "waiting for peered", 2024-03-11T21:35:44.767 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:34:57.511787+0000", 2024-03-11T21:35:44.767 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 6.9129999999999997e-06 2024-03-11T21:35:44.767 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:35:44.767 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:35:44.767 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "reached_pg", 2024-03-11T21:35:44.767 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:34:58.415235+0000", 2024-03-11T21:35:44.767 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.90344814799999995 2024-03-11T21:35:44.767 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:35:44.767 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:35:44.768 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "started", 2024-03-11T21:35:44.768 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:34:58.415330+0000", 2024-03-11T21:35:44.768 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 9.4048999999999993e-05 2024-03-11T21:35:44.768 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:35:44.768 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:35:44.768 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "waiting for subops from 2,7", 2024-03-11T21:35:44.768 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:34:58.415425+0000", 2024-03-11T21:35:44.768 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 9.5198000000000006e-05 2024-03-11T21:35:44.768 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:35:44.768 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:35:44.768 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "sub_op_commit_rec", 2024-03-11T21:35:44.768 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:34:58.421609+0000", 2024-03-11T21:35:44.768 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.0061840039999999999 2024-03-11T21:35:44.768 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:35:44.769 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:35:44.769 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "op_commit", 2024-03-11T21:35:44.769 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:34:58.426908+0000", 2024-03-11T21:35:44.769 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.0052995689999999996 2024-03-11T21:35:44.769 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:35:44.769 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:35:44.769 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "sub_op_commit_rec", 2024-03-11T21:35:44.769 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:34:58.426956+0000", 2024-03-11T21:35:44.769 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 4.7743e-05 2024-03-11T21:35:44.769 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:35:44.769 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:35:44.769 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "commit_sent", 2024-03-11T21:35:44.769 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:34:58.427006+0000", 2024-03-11T21:35:44.769 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 5.0365999999999999e-05 2024-03-11T21:35:44.770 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:35:44.770 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:35:44.770 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "done", 2024-03-11T21:35:44.770 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:34:58.427049+0000", 2024-03-11T21:35:44.770 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 4.2360000000000001e-05 2024-03-11T21:35:44.770 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-03-11T21:35:44.770 INFO:teuthology.orchestra.run.smithi097.stdout: ] 2024-03-11T21:35:44.770 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-03-11T21:35:44.770 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-03-11T21:35:44.770 INFO:teuthology.orchestra.run.smithi097.stdout: ] 2024-03-11T21:35:44.770 INFO:teuthology.orchestra.run.smithi097.stdout:} 2024-03-11T21:35:44.794 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:35:44.794 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:35:44.795 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:35:44.895 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:35:44.919 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:35:44.919 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:35:44.920 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:35:45.020 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:35:45.045 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:35:45.045 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:35:45.046 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:35:45.064 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:35:44 smithi092 ceph-mon[29572]: pgmap v454: 132 pgs: 22 active+undersized+degraded, 32 active+undersized, 78 active+clean; 582 KiB data, 2.8 GiB used, 712 GiB / 715 GiB avail; 1.5 KiB/s rd, 1 op/s; 94/594 objects degraded (15.825%) 2024-03-11T21:35:45.064 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:35:44 smithi092 ceph-mon[29572]: from='client.? v1:172.21.15.92:0/2316095079' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "unique_pool_2", "pg_num": 16}]: dispatch 2024-03-11T21:35:45.065 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:35:44 smithi092 ceph-mon[37563]: pgmap v454: 132 pgs: 22 active+undersized+degraded, 32 active+undersized, 78 active+clean; 582 KiB data, 2.8 GiB used, 712 GiB / 715 GiB avail; 1.5 KiB/s rd, 1 op/s; 94/594 objects degraded (15.825%) 2024-03-11T21:35:45.065 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:35:44 smithi092 ceph-mon[37563]: from='client.? v1:172.21.15.92:0/2316095079' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "unique_pool_2", "pg_num": 16}]: dispatch 2024-03-11T21:35:45.075 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:35:44 smithi097 ceph-mon[33317]: pgmap v454: 132 pgs: 22 active+undersized+degraded, 32 active+undersized, 78 active+clean; 582 KiB data, 2.8 GiB used, 712 GiB / 715 GiB avail; 1.5 KiB/s rd, 1 op/s; 94/594 objects degraded (15.825%) 2024-03-11T21:35:45.075 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:35:44 smithi097 ceph-mon[33317]: from='client.? v1:172.21.15.92:0/2316095079' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "unique_pool_2", "pg_num": 16}]: dispatch 2024-03-11T21:35:45.147 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:35:45.170 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:35:45.171 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:35:45.171 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:35:45.232 INFO:teuthology.orchestra.run.smithi092.stdout:8.f 2024-03-11T21:35:45.232 INFO:teuthology.orchestra.run.smithi092.stdout:8.c 2024-03-11T21:35:45.233 INFO:teuthology.orchestra.run.smithi092.stdout:8.b 2024-03-11T21:35:45.233 INFO:teuthology.orchestra.run.smithi092.stdout:8.8 2024-03-11T21:35:45.233 INFO:teuthology.orchestra.run.smithi092.stdout:8.7 2024-03-11T21:35:45.233 INFO:teuthology.orchestra.run.smithi092.stdout:8.18 2024-03-11T21:35:45.233 INFO:teuthology.orchestra.run.smithi092.stdout:8.16 2024-03-11T21:35:45.233 INFO:teuthology.orchestra.run.smithi092.stdout:8.13 2024-03-11T21:35:45.233 INFO:teuthology.orchestra.run.smithi092.stdout:6.f 2024-03-11T21:35:45.233 INFO:teuthology.orchestra.run.smithi092.stdout:6.a 2024-03-11T21:35:45.233 INFO:teuthology.orchestra.run.smithi092.stdout:6.6 2024-03-11T21:35:45.233 INFO:teuthology.orchestra.run.smithi092.stdout:6.c 2024-03-11T21:35:45.233 INFO:teuthology.orchestra.run.smithi092.stdout:6.4 2024-03-11T21:35:45.233 INFO:teuthology.orchestra.run.smithi092.stdout:6.1f 2024-03-11T21:35:45.233 INFO:teuthology.orchestra.run.smithi092.stdout:6.16 2024-03-11T21:35:45.234 INFO:teuthology.orchestra.run.smithi092.stdout:6.13 2024-03-11T21:35:45.234 INFO:teuthology.orchestra.run.smithi092.stdout:6.7 2024-03-11T21:35:45.234 INFO:teuthology.orchestra.run.smithi092.stdout:6.11 2024-03-11T21:35:45.234 INFO:teuthology.orchestra.run.smithi092.stdout:5.a 2024-03-11T21:35:45.234 INFO:teuthology.orchestra.run.smithi092.stdout:5.1c 2024-03-11T21:35:45.234 INFO:teuthology.orchestra.run.smithi092.stdout:4.11 2024-03-11T21:35:45.234 INFO:teuthology.orchestra.run.smithi092.stdout:4.9 2024-03-11T21:35:45.234 INFO:teuthology.orchestra.run.smithi092.stdout:6.b 2024-03-11T21:35:45.234 INFO:teuthology.orchestra.run.smithi092.stdout:3.2 2024-03-11T21:35:45.234 INFO:teuthology.orchestra.run.smithi092.stdout:2.d 2024-03-11T21:35:45.234 INFO:teuthology.orchestra.run.smithi092.stdout:6.0 2024-03-11T21:35:45.234 INFO:teuthology.orchestra.run.smithi092.stdout:5.13 2024-03-11T21:35:45.234 INFO:teuthology.orchestra.run.smithi092.stdout:5.16 2024-03-11T21:35:45.234 INFO:teuthology.orchestra.run.smithi092.stdout:4.4 2024-03-11T21:35:45.235 INFO:teuthology.orchestra.run.smithi092.stdout:2.a 2024-03-11T21:35:45.235 INFO:teuthology.orchestra.run.smithi092.stdout:2.9 2024-03-11T21:35:45.235 INFO:teuthology.orchestra.run.smithi092.stdout:2.1 2024-03-11T21:35:45.235 INFO:teuthology.orchestra.run.smithi092.stdout:8.0 2024-03-11T21:35:45.235 INFO:teuthology.orchestra.run.smithi092.stdout:2.13 2024-03-11T21:35:45.235 INFO:teuthology.orchestra.run.smithi092.stdout:2.12 2024-03-11T21:35:45.235 INFO:teuthology.orchestra.run.smithi092.stdout:4.10 2024-03-11T21:35:45.235 INFO:teuthology.orchestra.run.smithi092.stdout:2.14 2024-03-11T21:35:45.235 INFO:teuthology.orchestra.run.smithi092.stdout:2.e 2024-03-11T21:35:45.235 INFO:teuthology.orchestra.run.smithi092.stdout:5.1 2024-03-11T21:35:45.235 INFO:teuthology.orchestra.run.smithi092.stdout:2.1e 2024-03-11T21:35:45.235 INFO:teuthology.orchestra.run.smithi092.stdout:6.19 2024-03-11T21:35:45.235 INFO:teuthology.orchestra.run.smithi092.stdout:4.0 2024-03-11T21:35:45.235 INFO:teuthology.orchestra.run.smithi092.stdout:2.1b 2024-03-11T21:35:45.236 INFO:teuthology.orchestra.run.smithi092.stdout:4.17 2024-03-11T21:35:45.236 INFO:teuthology.orchestra.run.smithi092.stdout:4.14 2024-03-11T21:35:45.236 INFO:teuthology.orchestra.run.smithi092.stdout:6.5 2024-03-11T21:35:45.236 INFO:teuthology.orchestra.run.smithi092.stdout:4.16 2024-03-11T21:35:45.236 INFO:teuthology.orchestra.run.smithi092.stdout:4.1e 2024-03-11T21:35:45.236 INFO:teuthology.orchestra.run.smithi092.stdout:3.1 2024-03-11T21:35:45.236 INFO:teuthology.orchestra.run.smithi092.stdout:4.19 2024-03-11T21:35:45.236 INFO:teuthology.orchestra.run.smithi092.stdout:6.1b 2024-03-11T21:35:45.236 INFO:teuthology.orchestra.run.smithi092.stdout:2.19 2024-03-11T21:35:45.236 INFO:teuthology.orchestra.run.smithi092.stdout:4.1a 2024-03-11T21:35:45.236 INFO:teuthology.orchestra.run.smithi092.stdout:4.15 2024-03-11T21:35:45.236 INFO:teuthology.orchestra.run.smithi092.stdout:4.1c 2024-03-11T21:35:45.236 INFO:teuthology.orchestra.run.smithi092.stdout:4.1d 2024-03-11T21:35:45.236 INFO:teuthology.orchestra.run.smithi092.stdout:4.c 2024-03-11T21:35:45.237 INFO:teuthology.orchestra.run.smithi092.stdout:4.f 2024-03-11T21:35:45.237 INFO:teuthology.orchestra.run.smithi092.stdout:5.0 2024-03-11T21:35:45.237 INFO:teuthology.orchestra.run.smithi092.stdout:4.1b 2024-03-11T21:35:45.237 INFO:teuthology.orchestra.run.smithi092.stdout:5.17 2024-03-11T21:35:45.237 INFO:teuthology.orchestra.run.smithi092.stdout:5.12 2024-03-11T21:35:45.237 INFO:teuthology.orchestra.run.smithi092.stdout:5.14 2024-03-11T21:35:45.272 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:35:45.296 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:35:45.297 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:35:45.297 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:35:45.398 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:35:45.423 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:35:45.423 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:35:45.423 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:35:45.524 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:35:45.548 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:35:45.548 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:35:45.549 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:35:45.619 DEBUG:teuthology.orchestra.run.smithi097:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid 00db0016-dfed-11ee-95c8-87774f69a715 -- ceph daemon osd.6 dump_ops_in_flight 2024-03-11T21:35:45.649 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:35:45.678 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:35:45.679 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:35:45.680 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:35:45.778 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:35:45 smithi092 ceph-mon[29572]: from='client.? v1:172.21.15.92:0/2316095079' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "unique_pool_2", "pg_num": 16}]': finished 2024-03-11T21:35:45.778 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:35:45 smithi092 ceph-mon[29572]: osdmap e259: 8 total, 7 up, 8 in 2024-03-11T21:35:45.778 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:35:45 smithi092 ceph-mon[37563]: from='client.? v1:172.21.15.92:0/2316095079' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "unique_pool_2", "pg_num": 16}]': finished 2024-03-11T21:35:45.778 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:35:45 smithi092 ceph-mon[37563]: osdmap e259: 8 total, 7 up, 8 in 2024-03-11T21:35:45.780 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:35:45.805 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:35:45.805 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:35:45.806 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:35:45.868 INFO:teuthology.orchestra.run.smithi097.stderr:Inferring config /var/lib/ceph/00db0016-dfed-11ee-95c8-87774f69a715/mon.b/config 2024-03-11T21:35:45.907 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:35:45.931 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:35:45.932 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:35:45.932 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:35:45.989 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:35:45 smithi097 ceph-mon[33317]: from='client.? v1:172.21.15.92:0/2316095079' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "unique_pool_2", "pg_num": 16}]': finished 2024-03-11T21:35:45.989 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:35:45 smithi097 ceph-mon[33317]: osdmap e259: 8 total, 7 up, 8 in 2024-03-11T21:35:46.033 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:35:46.058 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:35:46.059 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:35:46.059 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:35:46.160 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:35:46.186 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:35:46.186 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:35:46.187 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:35:46.287 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:35:46.312 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:35:46.312 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:35:46.312 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:35:46.413 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:35:46.437 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:35:46.438 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:35:46.438 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:35:46.539 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:35:46.563 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:35:46.566 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:35:46.566 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:35:46.667 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:35:46.692 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:35:46.693 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:35:46.693 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:35:46.794 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:35:46.818 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:35:46.818 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:35:46.819 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:35:46.869 DEBUG:teuthology.orchestra.run.smithi092:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell -n osd.3 --fsid 00db0016-dfed-11ee-95c8-87774f69a715 -- ceph-objectstore-tool --err-to-stderr --no-mon-config '--log-file=/var/log/ceph/objectstore_tool.$pid.log' --data-path /var/lib/ceph/osd/ceph-3 --journal-path /var/lib/ceph/osd/ceph-3/journal --op export --pgid 8.8 --file /var/log/ceph/exp.8.8.3 2024-03-11T21:35:46.920 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:35:46.944 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:35:46.944 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:35:46.944 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:35:47.004 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:35:46 smithi092 ceph-mon[29572]: pgmap v457: 148 pgs: 16 unknown, 22 active+undersized+degraded, 32 active+undersized, 78 active+clean; 582 KiB data, 2.8 GiB used, 712 GiB / 715 GiB avail; 853 B/s rd, 0 op/s; 94/594 objects degraded (15.825%) 2024-03-11T21:35:47.004 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:35:46 smithi092 ceph-mon[29572]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-11T21:35:47.004 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:35:46 smithi092 ceph-mon[29572]: osdmap e260: 8 total, 7 up, 8 in 2024-03-11T21:35:47.005 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:35:46 smithi092 ceph-mon[37563]: pgmap v457: 148 pgs: 16 unknown, 22 active+undersized+degraded, 32 active+undersized, 78 active+clean; 582 KiB data, 2.8 GiB used, 712 GiB / 715 GiB avail; 853 B/s rd, 0 op/s; 94/594 objects degraded (15.825%) 2024-03-11T21:35:47.005 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:35:46 smithi092 ceph-mon[37563]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-11T21:35:47.005 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:35:46 smithi092 ceph-mon[37563]: osdmap e260: 8 total, 7 up, 8 in 2024-03-11T21:35:47.042 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:35:46 smithi097 ceph-mon[33317]: pgmap v457: 148 pgs: 16 unknown, 22 active+undersized+degraded, 32 active+undersized, 78 active+clean; 582 KiB data, 2.8 GiB used, 712 GiB / 715 GiB avail; 853 B/s rd, 0 op/s; 94/594 objects degraded (15.825%) 2024-03-11T21:35:47.042 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:35:46 smithi097 ceph-mon[33317]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-11T21:35:47.042 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:35:46 smithi097 ceph-mon[33317]: osdmap e260: 8 total, 7 up, 8 in 2024-03-11T21:35:47.046 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:35:47.069 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:35:47.070 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:35:47.070 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:35:47.131 INFO:teuthology.orchestra.run.smithi092.stdout:osd.0: {} 2024-03-11T21:35:47.131 INFO:teuthology.orchestra.run.smithi092.stderr:Inferring config /var/lib/ceph/00db0016-dfed-11ee-95c8-87774f69a715/osd.3/config 2024-03-11T21:35:47.157 INFO:teuthology.orchestra.run.smithi092.stdout:osd.1: {} 2024-03-11T21:35:47.171 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:35:47.177 INFO:teuthology.orchestra.run.smithi092.stdout:osd.2: {} 2024-03-11T21:35:47.195 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:35:47.195 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:35:47.196 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:35:47.197 INFO:teuthology.orchestra.run.smithi092.stdout:osd.4: {} 2024-03-11T21:35:47.216 INFO:teuthology.orchestra.run.smithi092.stdout:osd.5: {} 2024-03-11T21:35:47.236 INFO:teuthology.orchestra.run.smithi092.stdout:osd.6: {} 2024-03-11T21:35:47.255 INFO:teuthology.orchestra.run.smithi092.stdout:osd.7: {} 2024-03-11T21:35:47.255 INFO:teuthology.orchestra.run.smithi092.stderr:osd.0: osd_enable_op_tracker = '' 2024-03-11T21:35:47.255 INFO:teuthology.orchestra.run.smithi092.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-11T21:35:47.255 INFO:teuthology.orchestra.run.smithi092.stderr:osd.2: osd_enable_op_tracker = '' 2024-03-11T21:35:47.255 INFO:teuthology.orchestra.run.smithi092.stderr:Error ENXIO: problem getting command descriptions from osd.3 2024-03-11T21:35:47.255 INFO:teuthology.orchestra.run.smithi092.stderr:osd.4: osd_enable_op_tracker = '' 2024-03-11T21:35:47.255 INFO:teuthology.orchestra.run.smithi092.stderr:osd.5: osd_enable_op_tracker = '' 2024-03-11T21:35:47.256 INFO:teuthology.orchestra.run.smithi092.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-11T21:35:47.256 INFO:teuthology.orchestra.run.smithi092.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-11T21:35:47.297 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:35:47.321 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:35:47.322 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:35:47.322 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:35:47.423 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:35:47.447 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:35:47.447 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:35:47.447 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:35:47.548 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:35:47.572 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:35:47.572 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:35:47.573 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:35:47.638 INFO:teuthology.orchestra.run.smithi097.stdout:ERROR: (22) Invalid argument 2024-03-11T21:35:47.638 INFO:teuthology.orchestra.run.smithi097.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-11T21:35:47.673 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:35:47.687 DEBUG:teuthology.orchestra.run.smithi092:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid 00db0016-dfed-11ee-95c8-87774f69a715 -- ceph osd pool application enable unique_pool_2 rados --yes-i-really-mean-it || true 2024-03-11T21:35:47.697 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:35:47.697 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:35:47.698 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:35:47.799 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:35:47.823 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:35:47.824 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:35:47.824 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:35:47.923 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:35:47 smithi092 ceph-mon[29572]: osdmap e261: 8 total, 7 up, 8 in 2024-03-11T21:35:47.923 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:35:47 smithi092 ceph-mon[29572]: from='client.? v1:172.21.15.92:0/1432277736' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-11T21:35:47.923 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:35:47 smithi092 ceph-mon[37563]: osdmap e261: 8 total, 7 up, 8 in 2024-03-11T21:35:47.923 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:35:47 smithi092 ceph-mon[37563]: from='client.? v1:172.21.15.92:0/1432277736' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-11T21:35:47.925 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:35:47.949 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:35:47.949 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:35:47.949 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:35:48.050 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:35:48.075 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:35:48.075 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:35:48.076 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:35:48.076 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:35:47 smithi097 ceph-mon[33317]: osdmap e261: 8 total, 7 up, 8 in 2024-03-11T21:35:48.076 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:35:47 smithi097 ceph-mon[33317]: from='client.? v1:172.21.15.92:0/1432277736' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-11T21:35:48.176 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:35:48.187 INFO:journalctl@ceph.mgr.y.smithi092.stdout:Mar 11 21:35:47 smithi092 ceph-00db0016-dfed-11ee-95c8-87774f69a715-mgr-y[29801]: ::ffff:172.21.15.97 - - [11/Mar/2024:21:35:47] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-11T21:35:48.201 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:35:48.202 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:35:48.202 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:35:48.306 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:35:48.331 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:35:48.331 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:35:48.332 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:35:48.333 DEBUG:teuthology.orchestra.run.smithi097:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid 00db0016-dfed-11ee-95c8-87774f69a715 -- ceph daemon osd.6 dump_blocked_ops 2024-03-11T21:35:48.433 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:35:48.457 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:35:48.458 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:35:48.458 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:35:48.559 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:35:48.573 INFO:teuthology.orchestra.run.smithi097.stderr:Inferring config /var/lib/ceph/00db0016-dfed-11ee-95c8-87774f69a715/mon.b/config 2024-03-11T21:35:48.583 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:35:48.584 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:35:48.584 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:35:48.685 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:35:48.711 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:35:48.712 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:35:48.712 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:35:48.813 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:35:48.831 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-03-11T21:35:48.832 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-03-11T21:35:48.837 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:35:48.837 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:35:48.838 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:35:48.938 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:35:48.949 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:35:48 smithi092 ceph-mon[37563]: pgmap v460: 148 pgs: 8 unknown, 22 active+undersized+degraded, 36 active+undersized, 82 active+clean; 582 KiB data, 2.8 GiB used, 712 GiB / 715 GiB avail; 44 KiB/s rd, 0 B/s wr, 73 op/s; 94/594 objects degraded (15.825%) 2024-03-11T21:35:48.949 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:35:48 smithi092 ceph-mon[37563]: osdmap e262: 8 total, 7 up, 8 in 2024-03-11T21:35:48.949 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:35:48 smithi092 ceph-mon[29572]: pgmap v460: 148 pgs: 8 unknown, 22 active+undersized+degraded, 36 active+undersized, 82 active+clean; 582 KiB data, 2.8 GiB used, 712 GiB / 715 GiB avail; 44 KiB/s rd, 0 B/s wr, 73 op/s; 94/594 objects degraded (15.825%) 2024-03-11T21:35:48.949 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:35:48 smithi092 ceph-mon[29572]: osdmap e262: 8 total, 7 up, 8 in 2024-03-11T21:35:48.962 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:35:48.963 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:35:48.963 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:35:49.011 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:35:48 smithi097 ceph-mon[33317]: pgmap v460: 148 pgs: 8 unknown, 22 active+undersized+degraded, 36 active+undersized, 82 active+clean; 582 KiB data, 2.8 GiB used, 712 GiB / 715 GiB avail; 44 KiB/s rd, 0 B/s wr, 73 op/s; 94/594 objects degraded (15.825%) 2024-03-11T21:35:49.011 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:35:48 smithi097 ceph-mon[33317]: osdmap e262: 8 total, 7 up, 8 in 2024-03-11T21:35:49.063 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:35:49.087 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:35:49.087 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:35:49.088 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:35:49.188 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:35:49.212 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:35:49.212 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:35:49.213 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:35:49.314 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:35:49.337 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:35:49.338 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:35:49.338 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:35:49.439 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:35:49.464 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:35:49.465 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:35:49.465 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:35:49.566 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:35:49.590 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:35:49.590 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:35:49.591 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:35:49.691 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:35:49.716 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:35:49.717 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:35:49.717 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:35:49.818 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:35:49.842 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:35:49.842 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:35:49.845 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:35:49.946 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:35:49.970 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:35:49.971 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:35:49.971 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:35:50.024 INFO:teuthology.orchestra.run.smithi092.stderr:Creating an OSD daemon form without an OSD FSID value 2024-03-11T21:35:50.072 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:35:50.097 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:35:50.097 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:35:50.098 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:35:50.199 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:35:50.223 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:35:50.223 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:35:50.223 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:35:50.246 INFO:teuthology.orchestra.run.smithi097.stdout:ERROR: (22) Invalid argument 2024-03-11T21:35:50.246 INFO:teuthology.orchestra.run.smithi097.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-11T21:35:50.324 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:35:50.348 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:35:50.349 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:35:50.350 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:35:50.451 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:35:50.475 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:35:50.475 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:35:50.475 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:35:50.576 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:35:50.609 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:35:50.609 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:35:50.609 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:35:50.710 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:35:50.734 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:35:50.735 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:35:50.735 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:35:50.833 DEBUG:teuthology.orchestra.run.smithi092:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid 00db0016-dfed-11ee-95c8-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-03-11T21:35:50.836 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:35:50.899 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:35:50.900 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:35:50.900 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:35:50.905 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:35:50 smithi097 ceph-mon[33317]: pgmap v462: 148 pgs: 22 active+undersized+degraded, 40 active+undersized, 86 active+clean; 582 KiB data, 2.7 GiB used, 712 GiB / 715 GiB avail; 96 KiB/s rd, 0 B/s wr, 159 op/s; 94/594 objects degraded (15.825%) 2024-03-11T21:35:50.933 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:35:50 smithi092 ceph-mon[29572]: pgmap v462: 148 pgs: 22 active+undersized+degraded, 40 active+undersized, 86 active+clean; 582 KiB data, 2.7 GiB used, 712 GiB / 715 GiB avail; 96 KiB/s rd, 0 B/s wr, 159 op/s; 94/594 objects degraded (15.825%) 2024-03-11T21:35:50.933 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:35:50 smithi092 ceph-mon[37563]: pgmap v462: 148 pgs: 22 active+undersized+degraded, 40 active+undersized, 86 active+clean; 582 KiB data, 2.7 GiB used, 712 GiB / 715 GiB avail; 96 KiB/s rd, 0 B/s wr, 159 op/s; 94/594 objects degraded (15.825%) 2024-03-11T21:35:50.934 DEBUG:teuthology.orchestra.run.smithi097:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid 00db0016-dfed-11ee-95c8-87774f69a715 -- ceph daemon osd.6 dump_historic_ops 2024-03-11T21:35:51.002 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:35:51.017 INFO:teuthology.orchestra.run.smithi092.stderr:Inferring config /var/lib/ceph/00db0016-dfed-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T21:35:51.025 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:35:51.025 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:35:51.026 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:35:51.126 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:35:51.145 INFO:teuthology.orchestra.run.smithi092.stderr:Exporting 8.8 info 8.8( v 250'10472 (180'5587,250'10472] local-lis/les=236/237 n=0 ec=157/157 lis/c=236/236 les/c/f=237/237/0 sis=236) 2024-03-11T21:35:51.153 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:35:51.154 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:35:51.154 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:35:51.179 INFO:teuthology.orchestra.run.smithi097.stderr:Inferring config /var/lib/ceph/00db0016-dfed-11ee-95c8-87774f69a715/mon.b/config 2024-03-11T21:35:51.182 INFO:teuthology.orchestra.run.smithi092.stderr:Export successful 2024-03-11T21:35:51.255 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:35:51.279 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:35:51.280 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:35:51.280 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:35:51.381 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:35:51.405 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:35:51.405 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:35:51.406 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:35:51.506 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:35:51.532 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:35:51.532 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:35:51.532 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:35:51.634 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:35:51.658 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:35:51.659 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:35:51.659 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:35:51.691 INFO:teuthology.orchestra.run.smithi092.stderr:Inferring config /var/lib/ceph/00db0016-dfed-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T21:35:51.698 INFO:teuthology.orchestra.run.smithi092.stderr:nodeep-scrub is set 2024-03-11T21:35:51.760 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:35:51.784 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:35:51.784 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:35:51.784 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:35:51.826 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:35:51 smithi097 ceph-mon[33317]: from='client.? v1:172.21.15.92:0/691319752' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-11T21:35:51.826 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:35:51 smithi097 ceph-mon[33317]: from='client.35252 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-11T21:35:51.860 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:35:51 smithi092 ceph-mon[29572]: from='client.? v1:172.21.15.92:0/691319752' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-11T21:35:51.860 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:35:51 smithi092 ceph-mon[29572]: from='client.35252 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-11T21:35:51.860 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:35:51 smithi092 ceph-mon[37563]: from='client.? v1:172.21.15.92:0/691319752' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-11T21:35:51.860 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:35:51 smithi092 ceph-mon[37563]: from='client.35252 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-11T21:35:51.886 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:35:51.911 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:35:51.911 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:35:51.912 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:35:52.013 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:35:52.038 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:35:52.038 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:35:52.038 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:35:52.139 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:35:52.164 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:35:52.165 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:35:52.165 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:35:52.266 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:35:52.290 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:35:52.290 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:35:52.291 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:35:52.391 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:35:52.419 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:35:52.420 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:35:52.420 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:35:52.521 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:35:52.545 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:35:52.546 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:35:52.546 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:35:52.647 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:35:52.671 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:35:52.672 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:35:52.672 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:35:52.751 INFO:journalctl@ceph.iscsi.iscsi.a.smithi097.stdout:Mar 11 21:35:52 smithi097 ceph-00db0016-dfed-11ee-95c8-87774f69a715-iscsi-iscsi-a[60943]: debug there is no tcmu-runner data available 2024-03-11T21:35:52.752 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:35:52 smithi097 ceph-mon[33317]: pgmap v463: 148 pgs: 22 active+undersized+degraded, 40 active+undersized, 86 active+clean; 582 KiB data, 2.7 GiB used, 712 GiB / 715 GiB avail; 72 KiB/s rd, 0 B/s wr, 119 op/s; 94/594 objects degraded (15.825%) 2024-03-11T21:35:52.752 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:35:52 smithi097 ceph-mon[33317]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-11T21:35:52.752 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:35:52 smithi097 ceph-mon[33317]: from='client.35252 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-03-11T21:35:52.752 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:35:52 smithi097 ceph-mon[33317]: osdmap e263: 8 total, 7 up, 8 in 2024-03-11T21:35:52.752 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:35:52 smithi097 ceph-mon[33317]: from='mgr.14607 ' entity='mgr.y' 2024-03-11T21:35:52.752 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:35:52 smithi097 ceph-mon[33317]: from='mgr.14607 v1:172.21.15.92:0/1702717597' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T21:35:52.772 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:35:52.797 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:35:52.798 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:35:52.798 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:35:52.899 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:35:52.923 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:35:52.923 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:35:52.924 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:35:52.952 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:35:52 smithi092 ceph-mon[29572]: pgmap v463: 148 pgs: 22 active+undersized+degraded, 40 active+undersized, 86 active+clean; 582 KiB data, 2.7 GiB used, 712 GiB / 715 GiB avail; 72 KiB/s rd, 0 B/s wr, 119 op/s; 94/594 objects degraded (15.825%) 2024-03-11T21:35:52.952 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:35:52 smithi092 ceph-mon[29572]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-11T21:35:52.952 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:35:52 smithi092 ceph-mon[29572]: from='client.35252 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-03-11T21:35:52.952 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:35:52 smithi092 ceph-mon[29572]: osdmap e263: 8 total, 7 up, 8 in 2024-03-11T21:35:52.952 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:35:52 smithi092 ceph-mon[29572]: from='mgr.14607 ' entity='mgr.y' 2024-03-11T21:35:52.953 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:35:52 smithi092 ceph-mon[29572]: from='mgr.14607 v1:172.21.15.92:0/1702717597' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T21:35:52.953 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:35:52 smithi092 ceph-mon[37563]: pgmap v463: 148 pgs: 22 active+undersized+degraded, 40 active+undersized, 86 active+clean; 582 KiB data, 2.7 GiB used, 712 GiB / 715 GiB avail; 72 KiB/s rd, 0 B/s wr, 119 op/s; 94/594 objects degraded (15.825%) 2024-03-11T21:35:52.953 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:35:52 smithi092 ceph-mon[37563]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-11T21:35:52.953 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:35:52 smithi092 ceph-mon[37563]: from='client.35252 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-03-11T21:35:52.953 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:35:52 smithi092 ceph-mon[37563]: osdmap e263: 8 total, 7 up, 8 in 2024-03-11T21:35:52.953 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:35:52 smithi092 ceph-mon[37563]: from='mgr.14607 ' entity='mgr.y' 2024-03-11T21:35:52.953 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:35:52 smithi092 ceph-mon[37563]: from='mgr.14607 v1:172.21.15.92:0/1702717597' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T21:35:53.025 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:35:53.049 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:35:53.050 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:35:53.050 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:35:53.151 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:35:53.176 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:35:53.177 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:35:53.177 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:35:53.278 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:35:53.301 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:35:53.302 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:35:53.302 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:35:53.326 INFO:teuthology.orchestra.run.smithi097.stdout:ERROR: (22) Invalid argument 2024-03-11T21:35:53.326 INFO:teuthology.orchestra.run.smithi097.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-11T21:35:53.403 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:35:53.426 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:35:53.427 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:35:53.428 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:35:53.529 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:35:53.554 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:35:53.555 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:35:53.555 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:35:53.655 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:35:53.679 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:35:53.680 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:35:53.681 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:35:53.781 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:35:53.806 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:35:53.806 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:35:53.806 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:35:53.907 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:35:53.931 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:35:53.932 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:35:53.932 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:35:53.945 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:35:53 smithi092 ceph-mon[37563]: from='client.24469 v1:172.21.15.97:0/404208310' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:35:53.945 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:35:53 smithi092 ceph-mon[37563]: osdmap e264: 8 total, 7 up, 8 in 2024-03-11T21:35:53.945 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:35:53 smithi092 ceph-mon[29572]: from='client.24469 v1:172.21.15.97:0/404208310' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:35:53.945 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:35:53 smithi092 ceph-mon[29572]: osdmap e264: 8 total, 7 up, 8 in 2024-03-11T21:35:54.013 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:35:53 smithi097 ceph-mon[33317]: from='client.24469 v1:172.21.15.97:0/404208310' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:35:54.013 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:35:53 smithi097 ceph-mon[33317]: osdmap e264: 8 total, 7 up, 8 in 2024-03-11T21:35:54.033 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:35:54.038 DEBUG:teuthology.orchestra.run.smithi097:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid 00db0016-dfed-11ee-95c8-87774f69a715 -- ceph daemon osd.7 dump_ops_in_flight 2024-03-11T21:35:54.057 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:35:54.057 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:35:54.057 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:35:54.158 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:35:54.181 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:35:54.182 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:35:54.183 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:35:54.283 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:35:54.295 INFO:teuthology.orchestra.run.smithi097.stderr:Inferring config /var/lib/ceph/00db0016-dfed-11ee-95c8-87774f69a715/mon.b/config 2024-03-11T21:35:54.308 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:35:54.309 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:35:54.309 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:35:54.410 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:35:54.434 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:35:54.435 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:35:54.435 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:35:54.536 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:35:54.560 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:35:54.561 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:35:54.561 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:35:54.662 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:35:54.686 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:35:54.686 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:35:54.686 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:35:54.694 INFO:teuthology.orchestra.run.smithi092.stderr:enabled application 'rados' on pool 'unique_pool_2' 2024-03-11T21:35:54.787 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:35:54.810 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:35:54.811 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:35:54.812 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:35:54.860 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:35:54 smithi092 ceph-mon[29572]: pgmap v465: 148 pgs: 22 active+undersized+degraded, 40 active+undersized, 86 active+clean; 582 KiB data, 2.7 GiB used, 712 GiB / 715 GiB avail; 68 KiB/s rd, 0 B/s wr, 111 op/s; 94/594 objects degraded (15.825%) 2024-03-11T21:35:54.860 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:35:54 smithi092 ceph-mon[29572]: from='client.? v1:172.21.15.92:0/1563562688' entity='client.admin' cmd=[{"prefix": "osd pool application enable", "pool": "unique_pool_2", "app": "rados", "yes_i_really_mean_it": true}]: dispatch 2024-03-11T21:35:54.860 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:35:54 smithi092 ceph-mon[37563]: pgmap v465: 148 pgs: 22 active+undersized+degraded, 40 active+undersized, 86 active+clean; 582 KiB data, 2.7 GiB used, 712 GiB / 715 GiB avail; 68 KiB/s rd, 0 B/s wr, 111 op/s; 94/594 objects degraded (15.825%) 2024-03-11T21:35:54.860 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:35:54 smithi092 ceph-mon[37563]: from='client.? v1:172.21.15.92:0/1563562688' entity='client.admin' cmd=[{"prefix": "osd pool application enable", "pool": "unique_pool_2", "app": "rados", "yes_i_really_mean_it": true}]: dispatch 2024-03-11T21:35:54.860 DEBUG:teuthology.orchestra.run.smithi092:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid 00db0016-dfed-11ee-95c8-87774f69a715 -- ceph osd unset noscrub 2024-03-11T21:35:54.912 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:35:54.935 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:35:54.936 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:35:54.936 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:35:55.037 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:35:55.056 DEBUG:teuthology.orchestra.run.smithi092:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell -n osd.3 --fsid 00db0016-dfed-11ee-95c8-87774f69a715 -- ceph-objectstore-tool --err-to-stderr --no-mon-config '--log-file=/var/log/ceph/objectstore_tool.$pid.log' --data-path /var/lib/ceph/osd/ceph-3 --journal-path /var/lib/ceph/osd/ceph-3/journal --force --op remove --pgid 8.8 2024-03-11T21:35:55.062 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:35:55.062 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:35:55.063 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:35:55.075 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:35:54 smithi097 ceph-mon[33317]: pgmap v465: 148 pgs: 22 active+undersized+degraded, 40 active+undersized, 86 active+clean; 582 KiB data, 2.7 GiB used, 712 GiB / 715 GiB avail; 68 KiB/s rd, 0 B/s wr, 111 op/s; 94/594 objects degraded (15.825%) 2024-03-11T21:35:55.075 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:35:54 smithi097 ceph-mon[33317]: from='client.? v1:172.21.15.92:0/1563562688' entity='client.admin' cmd=[{"prefix": "osd pool application enable", "pool": "unique_pool_2", "app": "rados", "yes_i_really_mean_it": true}]: dispatch 2024-03-11T21:35:55.166 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:35:55.190 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:35:55.191 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:35:55.192 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:35:55.279 INFO:teuthology.orchestra.run.smithi092.stderr:Inferring config /var/lib/ceph/00db0016-dfed-11ee-95c8-87774f69a715/osd.3/config 2024-03-11T21:35:55.293 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:35:55.316 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:35:55.317 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:35:55.317 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:35:55.418 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:35:55.443 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:35:55.444 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:35:55.444 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:35:55.545 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:35:55.570 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:35:55.571 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:35:55.571 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:35:55.672 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:35:55.698 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:35:55.698 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:35:55.699 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:35:55.738 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:35:55 smithi097 ceph-mon[33317]: from='client.? v1:172.21.15.92:0/1563562688' entity='client.admin' cmd='[{"prefix": "osd pool application enable", "pool": "unique_pool_2", "app": "rados", "yes_i_really_mean_it": true}]': finished 2024-03-11T21:35:55.738 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:35:55 smithi097 ceph-mon[33317]: osdmap e265: 8 total, 7 up, 8 in 2024-03-11T21:35:55.799 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:35:55.823 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:35:55.823 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:35:55.824 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:35:55.925 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:35:55.949 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:35:55.949 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:35:55.950 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:35:56.051 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:35:56.064 INFO:teuthology.orchestra.run.smithi097.stdout:ERROR: (22) Invalid argument 2024-03-11T21:35:56.064 INFO:teuthology.orchestra.run.smithi097.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-11T21:35:56.074 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:35:56.075 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:35:56.075 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:35:56.109 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:35:55 smithi092 ceph-mon[37563]: from='client.? v1:172.21.15.92:0/1563562688' entity='client.admin' cmd='[{"prefix": "osd pool application enable", "pool": "unique_pool_2", "app": "rados", "yes_i_really_mean_it": true}]': finished 2024-03-11T21:35:56.110 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:35:55 smithi092 ceph-mon[37563]: osdmap e265: 8 total, 7 up, 8 in 2024-03-11T21:35:56.110 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:35:55 smithi092 ceph-mon[29572]: from='client.? v1:172.21.15.92:0/1563562688' entity='client.admin' cmd='[{"prefix": "osd pool application enable", "pool": "unique_pool_2", "app": "rados", "yes_i_really_mean_it": true}]': finished 2024-03-11T21:35:56.110 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:35:55 smithi092 ceph-mon[29572]: osdmap e265: 8 total, 7 up, 8 in 2024-03-11T21:35:56.176 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:35:56.199 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:35:56.200 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:35:56.200 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:35:56.301 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:35:56.325 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:35:56.326 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:35:56.326 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:35:56.427 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:35:56.449 INFO:teuthology.orchestra.run.smithi092.stderr:Inferring config /var/lib/ceph/00db0016-dfed-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T21:35:56.459 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:35:56.459 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:35:56.460 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:35:56.561 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:35:56.590 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:35:56.591 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:35:56.591 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:35:56.692 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:35:56.719 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:35:56.719 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:35:56.720 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:35:56.736 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:35:56 smithi097 ceph-mon[33317]: pgmap v468: 148 pgs: 22 active+undersized+degraded, 40 active+undersized, 86 active+clean; 582 KiB data, 2.7 GiB used, 712 GiB / 715 GiB avail; 853 B/s rd, 0 op/s; 94/594 objects degraded (15.825%) 2024-03-11T21:35:56.736 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:35:56 smithi097 ceph-mon[33317]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-03-11T21:35:56.737 DEBUG:teuthology.orchestra.run.smithi097:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid 00db0016-dfed-11ee-95c8-87774f69a715 -- ceph daemon osd.7 dump_blocked_ops 2024-03-11T21:35:56.820 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:35:56.848 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:35:56.849 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:35:56.849 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:35:56.950 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:35:56.975 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:35:56.976 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:35:56.976 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:35:56.977 INFO:teuthology.orchestra.run.smithi097.stderr:Inferring config /var/lib/ceph/00db0016-dfed-11ee-95c8-87774f69a715/mon.b/config 2024-03-11T21:35:57.077 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:35:57.104 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:35:57.104 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:35:57.105 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:35:57.109 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:35:56 smithi092 ceph-mon[29572]: pgmap v468: 148 pgs: 22 active+undersized+degraded, 40 active+undersized, 86 active+clean; 582 KiB data, 2.7 GiB used, 712 GiB / 715 GiB avail; 853 B/s rd, 0 op/s; 94/594 objects degraded (15.825%) 2024-03-11T21:35:57.110 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:35:56 smithi092 ceph-mon[29572]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-03-11T21:35:57.110 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:35:56 smithi092 ceph-mon[37563]: pgmap v468: 148 pgs: 22 active+undersized+degraded, 40 active+undersized, 86 active+clean; 582 KiB data, 2.7 GiB used, 712 GiB / 715 GiB avail; 853 B/s rd, 0 op/s; 94/594 objects degraded (15.825%) 2024-03-11T21:35:57.110 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:35:56 smithi092 ceph-mon[37563]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-03-11T21:35:57.205 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:35:57.229 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:35:57.229 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:35:57.230 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:35:57.330 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:35:57.354 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:35:57.355 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:35:57.355 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:35:57.456 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:35:57.469 DEBUG:teuthology.orchestra.run.smithi092:> /bin/sh -c 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados --no-log-to-stderr --name client.0 -b 65536 --object-size 65536 -p unique_pool_2 bench 90 write' 2024-03-11T21:35:57.471 INFO:tasks.radosbench:joining radosbench (timing out after 3000s) 2024-03-11T21:35:57.472 INFO:teuthology.orchestra.run:waiting for 3000 2024-03-11T21:35:57.480 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:35:57.481 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:35:57.481 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:35:57.517 INFO:tasks.radosbench.radosbench.0.smithi092.stdout:hints = 1 2024-03-11T21:35:57.517 INFO:tasks.radosbench.radosbench.0.smithi092.stdout:Maintaining 16 concurrent writes of 65536 bytes to objects of size 65536 for up to 90 seconds or 0 objects 2024-03-11T21:35:57.517 INFO:tasks.radosbench.radosbench.0.smithi092.stdout:Object prefix: benchmark_data_smithi092_130291 2024-03-11T21:35:57.517 INFO:tasks.radosbench.radosbench.0.smithi092.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-03-11T21:35:57.517 INFO:tasks.radosbench.radosbench.0.smithi092.stdout: 0 0 0 0 0 0 - 0 2024-03-11T21:35:57.582 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:35:57.619 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:35:57.620 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:35:57.620 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:35:57.721 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:35:57.757 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:35:57.758 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:35:57.758 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:35:57.859 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:35:57.897 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:35:57.898 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:35:57.898 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:35:57.999 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:35:58.034 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:35:58.034 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:35:58.035 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:35:58.136 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:35:58.173 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:35:58.174 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:35:58.174 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:35:58.237 INFO:teuthology.orchestra.run.smithi092.stderr:Creating an OSD daemon form without an OSD FSID value 2024-03-11T21:35:58.275 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:35:58.313 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:35:58.314 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:35:58.314 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:35:58.316 INFO:journalctl@ceph.mgr.y.smithi092.stdout:Mar 11 21:35:57 smithi092 ceph-00db0016-dfed-11ee-95c8-87774f69a715-mgr-y[29801]: ::ffff:172.21.15.97 - - [11/Mar/2024:21:35:57] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-11T21:35:58.415 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:35:58.458 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:35:58.458 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:35:58.458 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:35:58.518 INFO:tasks.radosbench.radosbench.0.smithi092.stdout: 1 16 4193 4177 261.047 261.062 0.00569831 0.00381577 2024-03-11T21:35:58.560 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:35:58.593 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:35:58.593 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:35:58.594 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:35:58.695 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:35:58.734 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:35:58.734 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:35:58.735 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:35:58.836 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:35:58.859 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:35:58 smithi092 ceph-mon[29572]: pgmap v469: 148 pgs: 22 active+undersized+degraded, 40 active+undersized, 86 active+clean; 582 KiB data, 2.7 GiB used, 712 GiB / 715 GiB avail; 853 B/s rd, 0 op/s; 94/594 objects degraded (15.825%) 2024-03-11T21:35:58.860 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:35:58 smithi092 ceph-mon[29572]: osdmap e266: 8 total, 7 up, 8 in 2024-03-11T21:35:58.860 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:35:58 smithi092 ceph-mon[37563]: pgmap v469: 148 pgs: 22 active+undersized+degraded, 40 active+undersized, 86 active+clean; 582 KiB data, 2.7 GiB used, 712 GiB / 715 GiB avail; 853 B/s rd, 0 op/s; 94/594 objects degraded (15.825%) 2024-03-11T21:35:58.860 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:35:58 smithi092 ceph-mon[37563]: osdmap e266: 8 total, 7 up, 8 in 2024-03-11T21:35:58.876 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:35:58.877 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:35:58.877 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:35:58.883 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:35:58 smithi097 ceph-mon[33317]: pgmap v469: 148 pgs: 22 active+undersized+degraded, 40 active+undersized, 86 active+clean; 582 KiB data, 2.7 GiB used, 712 GiB / 715 GiB avail; 853 B/s rd, 0 op/s; 94/594 objects degraded (15.825%) 2024-03-11T21:35:58.883 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:35:58 smithi097 ceph-mon[33317]: osdmap e266: 8 total, 7 up, 8 in 2024-03-11T21:35:58.978 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:35:59.020 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:35:59.021 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:35:59.021 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:35:59.079 INFO:teuthology.orchestra.run.smithi097.stdout:ERROR: (22) Invalid argument 2024-03-11T21:35:59.079 INFO:teuthology.orchestra.run.smithi097.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-11T21:35:59.123 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:35:59.159 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:35:59.160 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:35:59.161 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:35:59.262 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:35:59.299 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:35:59.299 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:35:59.299 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:35:59.400 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:35:59.425 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:35:59.426 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:35:59.426 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:35:59.517 INFO:tasks.radosbench.radosbench.0.smithi092.stdout: 2 15 7349 7334 229.171 197.312 0.00352242 0.0043544 2024-03-11T21:35:59.527 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:35:59.569 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:35:59.569 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:35:59.569 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:35:59.670 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:35:59.704 INFO:teuthology.orchestra.run.smithi092.stdout:osd.0: {} 2024-03-11T21:35:59.707 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:35:59.707 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:35:59.707 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:35:59.734 INFO:teuthology.orchestra.run.smithi092.stderr:noscrub is unset 2024-03-11T21:35:59.750 INFO:teuthology.orchestra.run.smithi092.stdout:osd.1: {} 2024-03-11T21:35:59.795 INFO:teuthology.orchestra.run.smithi092.stdout:osd.2: {} 2024-03-11T21:35:59.809 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:35:59.811 DEBUG:teuthology.orchestra.run.smithi097:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid 00db0016-dfed-11ee-95c8-87774f69a715 -- ceph daemon osd.7 dump_historic_ops 2024-03-11T21:35:59.831 INFO:teuthology.orchestra.run.smithi092.stdout:osd.4: {} 2024-03-11T21:35:59.846 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:35:59.847 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:35:59.847 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:35:59.859 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:35:59 smithi092 ceph-mon[29572]: from='client.? v1:172.21.15.92:0/943593669' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-11T21:35:59.860 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:35:59 smithi092 ceph-mon[29572]: from='client.? v1:172.21.15.92:0/802100576' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-11T21:35:59.860 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:35:59 smithi092 ceph-mon[37563]: from='client.? v1:172.21.15.92:0/943593669' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-11T21:35:59.860 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:35:59 smithi092 ceph-mon[37563]: from='client.? v1:172.21.15.92:0/802100576' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-11T21:35:59.873 INFO:teuthology.orchestra.run.smithi092.stdout:osd.5: {} 2024-03-11T21:35:59.911 INFO:teuthology.orchestra.run.smithi092.stdout:osd.6: {} 2024-03-11T21:35:59.945 INFO:teuthology.orchestra.run.smithi092.stdout:osd.7: {} 2024-03-11T21:35:59.946 INFO:teuthology.orchestra.run.smithi092.stderr:osd.0: osd_enable_op_tracker = '' 2024-03-11T21:35:59.946 INFO:teuthology.orchestra.run.smithi092.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-11T21:35:59.946 INFO:teuthology.orchestra.run.smithi092.stderr:osd.2: osd_enable_op_tracker = '' 2024-03-11T21:35:59.946 INFO:teuthology.orchestra.run.smithi092.stderr:Error ENXIO: problem getting command descriptions from osd.3 2024-03-11T21:35:59.946 INFO:teuthology.orchestra.run.smithi092.stderr:osd.4: osd_enable_op_tracker = '' 2024-03-11T21:35:59.946 INFO:teuthology.orchestra.run.smithi092.stderr:osd.5: osd_enable_op_tracker = '' 2024-03-11T21:35:59.946 INFO:teuthology.orchestra.run.smithi092.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-11T21:35:59.946 INFO:teuthology.orchestra.run.smithi092.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-11T21:35:59.949 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:35:59.991 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:35:59.992 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:35:59.992 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:36:00.044 INFO:teuthology.orchestra.run.smithi092.stdout: marking collection for removal 2024-03-11T21:36:00.046 INFO:teuthology.orchestra.run.smithi092.stdout:setting '_remove' omap key 2024-03-11T21:36:00.046 INFO:teuthology.orchestra.run.smithi092.stdout:finish_remove_pgs 8.8_head removing 8.8 2024-03-11T21:36:00.074 INFO:teuthology.orchestra.run.smithi092.stdout:Remove successful 2024-03-11T21:36:00.078 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:35:59 smithi097 ceph-mon[33317]: from='client.? v1:172.21.15.92:0/943593669' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-11T21:36:00.078 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:35:59 smithi097 ceph-mon[33317]: from='client.? v1:172.21.15.92:0/802100576' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-11T21:36:00.093 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:36:00.137 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:36:00.138 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:36:00.138 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:36:00.237 INFO:teuthology.orchestra.run.smithi097.stderr:Inferring config /var/lib/ceph/00db0016-dfed-11ee-95c8-87774f69a715/mon.b/config 2024-03-11T21:36:00.239 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:36:00.282 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:36:00.283 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:36:00.286 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:36:00.386 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:36:00.424 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:36:00.425 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:36:00.425 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:36:00.517 INFO:tasks.radosbench.radosbench.0.smithi092.stdout: 3 16 10801 10785 224.671 215.688 0.00326469 0.00444457 2024-03-11T21:36:00.526 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:36:00.566 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:36:00.567 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:36:00.568 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:36:00.668 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:36:00.712 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:36:00.712 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:36:00.713 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:36:00.814 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:36:00.841 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:36:00.842 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:36:00.843 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:36:00.859 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:36:00 smithi092 ceph-mon[29572]: pgmap v471: 148 pgs: 23 active+undersized+degraded, 39 active+undersized, 86 active+clean; 54 MiB data, 2.7 GiB used, 712 GiB / 715 GiB avail; 853 B/s rd, 9.0 MiB/s wr, 144 op/s; 311/3174 objects degraded (9.798%) 2024-03-11T21:36:00.860 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:36:00 smithi092 ceph-mon[29572]: purged_snaps scrub starts 2024-03-11T21:36:00.860 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:36:00 smithi092 ceph-mon[29572]: purged_snaps scrub ok 2024-03-11T21:36:00.860 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:36:00 smithi092 ceph-mon[29572]: Health check update: Degraded data redundancy: 311/3174 objects degraded (9.798%), 23 pgs degraded (PG_DEGRADED) 2024-03-11T21:36:00.860 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:36:00 smithi092 ceph-mon[29572]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-11T21:36:00.860 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:36:00 smithi092 ceph-mon[29572]: from='client.? v1:172.21.15.92:0/943593669' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-03-11T21:36:00.860 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:36:00 smithi092 ceph-mon[29572]: osdmap e267: 8 total, 7 up, 8 in 2024-03-11T21:36:00.860 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:36:00 smithi092 ceph-mon[37563]: pgmap v471: 148 pgs: 23 active+undersized+degraded, 39 active+undersized, 86 active+clean; 54 MiB data, 2.7 GiB used, 712 GiB / 715 GiB avail; 853 B/s rd, 9.0 MiB/s wr, 144 op/s; 311/3174 objects degraded (9.798%) 2024-03-11T21:36:00.860 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:36:00 smithi092 ceph-mon[37563]: purged_snaps scrub starts 2024-03-11T21:36:00.860 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:36:00 smithi092 ceph-mon[37563]: purged_snaps scrub ok 2024-03-11T21:36:00.860 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:36:00 smithi092 ceph-mon[37563]: Health check update: Degraded data redundancy: 311/3174 objects degraded (9.798%), 23 pgs degraded (PG_DEGRADED) 2024-03-11T21:36:00.861 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:36:00 smithi092 ceph-mon[37563]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-11T21:36:00.861 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:36:00 smithi092 ceph-mon[37563]: from='client.? v1:172.21.15.92:0/943593669' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-03-11T21:36:00.861 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:36:00 smithi092 ceph-mon[37563]: osdmap e267: 8 total, 7 up, 8 in 2024-03-11T21:36:00.943 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:36:00.979 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:36:00.980 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:36:00.980 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:36:01.002 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:36:00 smithi097 ceph-mon[33317]: pgmap v471: 148 pgs: 23 active+undersized+degraded, 39 active+undersized, 86 active+clean; 54 MiB data, 2.7 GiB used, 712 GiB / 715 GiB avail; 853 B/s rd, 9.0 MiB/s wr, 144 op/s; 311/3174 objects degraded (9.798%) 2024-03-11T21:36:01.003 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:36:00 smithi097 ceph-mon[33317]: purged_snaps scrub starts 2024-03-11T21:36:01.003 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:36:00 smithi097 ceph-mon[33317]: purged_snaps scrub ok 2024-03-11T21:36:01.003 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:36:00 smithi097 ceph-mon[33317]: Health check update: Degraded data redundancy: 311/3174 objects degraded (9.798%), 23 pgs degraded (PG_DEGRADED) 2024-03-11T21:36:01.003 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:36:00 smithi097 ceph-mon[33317]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-11T21:36:01.003 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:36:00 smithi097 ceph-mon[33317]: from='client.? v1:172.21.15.92:0/943593669' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-03-11T21:36:01.003 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:36:00 smithi097 ceph-mon[33317]: osdmap e267: 8 total, 7 up, 8 in 2024-03-11T21:36:01.082 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:36:01.116 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:36:01.117 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:36:01.117 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:36:01.218 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:36:01.245 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:36:01.245 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:36:01.246 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:36:01.346 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:36:01.385 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:36:01.385 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:36:01.385 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:36:01.486 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:36:01.517 INFO:tasks.radosbench.radosbench.0.smithi092.stdout: 4 16 13588 13572 212.046 174.188 0.00553296 0.00471044 2024-03-11T21:36:01.521 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:36:01.522 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:36:01.522 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:36:01.623 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:36:01.672 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:36:01.673 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:36:01.673 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:36:01.774 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:36:01.811 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:36:01.812 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:36:01.812 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:36:01.913 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:36:01.942 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:36:01.943 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:36:01.943 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:36:02.044 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:36:02.075 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:36:01 smithi097 ceph-mon[33317]: 5.18 scrub starts 2024-03-11T21:36:02.076 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:36:01 smithi097 ceph-mon[33317]: 5.18 scrub ok 2024-03-11T21:36:02.076 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:36:01 smithi097 ceph-mon[33317]: 2.1f scrub starts 2024-03-11T21:36:02.076 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:36:01 smithi097 ceph-mon[33317]: 2.1f scrub ok 2024-03-11T21:36:02.076 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:36:01 smithi097 ceph-mon[33317]: 2.c scrub starts 2024-03-11T21:36:02.076 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:36:01 smithi097 ceph-mon[33317]: 2.c scrub ok 2024-03-11T21:36:02.080 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:36:02.081 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:36:02.081 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:36:02.109 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:36:01 smithi092 ceph-mon[29572]: 5.18 scrub starts 2024-03-11T21:36:02.110 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:36:01 smithi092 ceph-mon[29572]: 5.18 scrub ok 2024-03-11T21:36:02.110 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:36:01 smithi092 ceph-mon[29572]: 2.1f scrub starts 2024-03-11T21:36:02.110 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:36:01 smithi092 ceph-mon[29572]: 2.1f scrub ok 2024-03-11T21:36:02.110 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:36:01 smithi092 ceph-mon[29572]: 2.c scrub starts 2024-03-11T21:36:02.110 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:36:01 smithi092 ceph-mon[29572]: 2.c scrub ok 2024-03-11T21:36:02.110 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:36:01 smithi092 ceph-mon[37563]: 5.18 scrub starts 2024-03-11T21:36:02.110 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:36:01 smithi092 ceph-mon[37563]: 5.18 scrub ok 2024-03-11T21:36:02.110 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:36:01 smithi092 ceph-mon[37563]: 2.1f scrub starts 2024-03-11T21:36:02.110 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:36:01 smithi092 ceph-mon[37563]: 2.1f scrub ok 2024-03-11T21:36:02.110 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:36:01 smithi092 ceph-mon[37563]: 2.c scrub starts 2024-03-11T21:36:02.110 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:36:01 smithi092 ceph-mon[37563]: 2.c scrub ok 2024-03-11T21:36:02.182 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:36:02.218 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:36:02.219 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:36:02.219 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:36:02.320 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:36:02.345 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:36:02.346 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:36:02.346 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:36:02.447 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:36:02.482 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:36:02.483 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:36:02.483 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:36:02.518 INFO:tasks.radosbench.radosbench.0.smithi092.stdout: 5 15 16122 16107 201.322 158.438 0.00393231 0.00496156 2024-03-11T21:36:02.584 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:36:02.621 DEBUG:teuthology.orchestra.run.smithi092:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid 00db0016-dfed-11ee-95c8-87774f69a715 -- ceph osd unset nodeep-scrub 2024-03-11T21:36:02.622 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:36:02.623 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:36:02.624 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:36:02.724 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:36:02.758 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:36:02.759 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:36:02.759 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:36:02.826 INFO:journalctl@ceph.iscsi.iscsi.a.smithi097.stdout:Mar 11 21:36:02 smithi097 ceph-00db0016-dfed-11ee-95c8-87774f69a715-iscsi-iscsi-a[60943]: debug there is no tcmu-runner data available 2024-03-11T21:36:02.826 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:36:02 smithi097 ceph-mon[33317]: pgmap v473: 148 pgs: 23 active+undersized+degraded, 39 active+undersized, 86 active+clean; 54 MiB data, 2.7 GiB used, 712 GiB / 715 GiB avail; 791 B/s rd, 8.3 MiB/s wr, 133 op/s; 311/3174 objects degraded (9.798%) 2024-03-11T21:36:02.860 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:36:02.897 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:36:02.898 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:36:02.899 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:36:02.999 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:36:03.010 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:36:02 smithi092 ceph-mon[37563]: pgmap v473: 148 pgs: 23 active+undersized+degraded, 39 active+undersized, 86 active+clean; 54 MiB data, 2.7 GiB used, 712 GiB / 715 GiB avail; 791 B/s rd, 8.3 MiB/s wr, 133 op/s; 311/3174 objects degraded (9.798%) 2024-03-11T21:36:03.011 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:36:02 smithi092 ceph-mon[29572]: pgmap v473: 148 pgs: 23 active+undersized+degraded, 39 active+undersized, 86 active+clean; 54 MiB data, 2.7 GiB used, 712 GiB / 715 GiB avail; 791 B/s rd, 8.3 MiB/s wr, 133 op/s; 311/3174 objects degraded (9.798%) 2024-03-11T21:36:03.045 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:36:03.046 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:36:03.047 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:36:03.089 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-03-11T21:36:03.089 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-03-11T21:36:03.148 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:36:03.181 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:36:03.182 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:36:03.183 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:36:03.283 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:36:03.323 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:36:03.324 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:36:03.324 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:36:03.425 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:36:03.461 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:36:03.462 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:36:03.462 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:36:03.518 INFO:tasks.radosbench.radosbench.0.smithi092.stdout: 6 16 19225 19209 200.078 193.875 0.00800268 0.00499225 2024-03-11T21:36:03.564 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:36:03.605 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:36:03.606 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:36:03.606 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:36:03.652 DEBUG:teuthology.orchestra.run.smithi092:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell -n osd.3 --fsid 00db0016-dfed-11ee-95c8-87774f69a715 -- ceph-objectstore-tool --err-to-stderr --data-path /var/lib/ceph/osd/ceph-3 --journal-path /var/lib/ceph/osd/ceph-3/journal '--log-file=/var/log/ceph/objectstore_tool.$pid.log' --op import --file /var/log/ceph/exp.8.8.3 2024-03-11T21:36:03.690 INFO:teuthology.orchestra.run.smithi092.stderr:Inferring config /var/lib/ceph/00db0016-dfed-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T21:36:03.707 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:36:03.747 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:36:03.747 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:36:03.748 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:36:03.848 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:36:03.859 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:36:03 smithi092 ceph-mon[29572]: from='client.24469 v1:172.21.15.97:0/404208310' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:36:03.860 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:36:03 smithi092 ceph-mon[29572]: osdmap e268: 8 total, 7 up, 8 in 2024-03-11T21:36:03.860 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:36:03 smithi092 ceph-mon[37563]: from='client.24469 v1:172.21.15.97:0/404208310' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:36:03.860 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:36:03 smithi092 ceph-mon[37563]: osdmap e268: 8 total, 7 up, 8 in 2024-03-11T21:36:03.880 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:36:03 smithi097 ceph-mon[33317]: from='client.24469 v1:172.21.15.97:0/404208310' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:36:03.880 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:36:03 smithi097 ceph-mon[33317]: osdmap e268: 8 total, 7 up, 8 in 2024-03-11T21:36:03.889 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:36:03.889 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:36:03.890 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:36:03.991 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:36:04.008 INFO:teuthology.orchestra.run.smithi092.stderr:Inferring config /var/lib/ceph/00db0016-dfed-11ee-95c8-87774f69a715/osd.3/config 2024-03-11T21:36:04.033 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:36:04.033 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:36:04.034 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:36:04.111 INFO:teuthology.orchestra.run.smithi097.stdout:{ 2024-03-11T21:36:04.112 INFO:teuthology.orchestra.run.smithi097.stdout: "size": 20, 2024-03-11T21:36:04.112 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 600, 2024-03-11T21:36:04.112 INFO:teuthology.orchestra.run.smithi097.stdout: "ops": [ 2024-03-11T21:36:04.112 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:36:04.112 INFO:teuthology.orchestra.run.smithi097.stdout: "description": "osd_op(client.15150.0:150448 8.15 8:abfa6e92:::benchmark_data_smithi092_104865_object29947:head [delete] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e213)", 2024-03-11T21:36:04.112 INFO:teuthology.orchestra.run.smithi097.stdout: "initiated_at": "2024-03-11T21:34:01.652028+0000", 2024-03-11T21:36:04.112 INFO:teuthology.orchestra.run.smithi097.stdout: "age": 122.472462534, 2024-03-11T21:36:04.112 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.011345704, 2024-03-11T21:36:04.112 INFO:teuthology.orchestra.run.smithi097.stdout: "type_data": { 2024-03-11T21:36:04.112 INFO:teuthology.orchestra.run.smithi097.stdout: "flag_point": "reached pg", 2024-03-11T21:36:04.112 INFO:teuthology.orchestra.run.smithi097.stdout: "client_info": { 2024-03-11T21:36:04.113 INFO:teuthology.orchestra.run.smithi097.stdout: "client": "client.15150", 2024-03-11T21:36:04.113 INFO:teuthology.orchestra.run.smithi097.stdout: "client_addr": "v1:172.21.15.92:0/436606717", 2024-03-11T21:36:04.113 INFO:teuthology.orchestra.run.smithi097.stdout: "tid": 150448 2024-03-11T21:36:04.113 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:36:04.113 INFO:teuthology.orchestra.run.smithi097.stdout: "events": [ 2024-03-11T21:36:04.113 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:36:04.113 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "initiated", 2024-03-11T21:36:04.113 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:34:01.652028+0000", 2024-03-11T21:36:04.113 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-03-11T21:36:04.113 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:36:04.113 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:36:04.113 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "header_read", 2024-03-11T21:36:04.113 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:34:01.652028+0000", 2024-03-11T21:36:04.114 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-03-11T21:36:04.114 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:36:04.114 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:36:04.114 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "throttled", 2024-03-11T21:36:04.114 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:34:01.652030+0000", 2024-03-11T21:36:04.114 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 2.1009999999999999e-06 2024-03-11T21:36:04.114 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:36:04.114 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:36:04.114 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "all_read", 2024-03-11T21:36:04.114 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:34:01.652044+0000", 2024-03-11T21:36:04.114 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.4307e-05 2024-03-11T21:36:04.114 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:36:04.114 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:36:04.114 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "dispatched", 2024-03-11T21:36:04.115 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:34:01.652062+0000", 2024-03-11T21:36:04.115 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.7530999999999999e-05 2024-03-11T21:36:04.115 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:36:04.115 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:36:04.115 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "queued_for_pg", 2024-03-11T21:36:04.115 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:34:01.652072+0000", 2024-03-11T21:36:04.115 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.0032e-05 2024-03-11T21:36:04.115 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:36:04.115 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:36:04.115 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "reached_pg", 2024-03-11T21:36:04.115 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:34:01.652145+0000", 2024-03-11T21:36:04.115 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 7.3454000000000005e-05 2024-03-11T21:36:04.115 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:36:04.116 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:36:04.116 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "waiting for peered", 2024-03-11T21:36:04.116 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:34:01.652157+0000", 2024-03-11T21:36:04.116 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.1571e-05 2024-03-11T21:36:04.116 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:36:04.116 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:36:04.116 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "reached_pg", 2024-03-11T21:36:04.116 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:34:02.663273+0000", 2024-03-11T21:36:04.116 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.0111165280000001 2024-03-11T21:36:04.116 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:36:04.116 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:36:04.116 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "done", 2024-03-11T21:36:04.116 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:34:02.663373+0000", 2024-03-11T21:36:04.117 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.00010018000000000001 2024-03-11T21:36:04.117 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-03-11T21:36:04.117 INFO:teuthology.orchestra.run.smithi097.stdout: ] 2024-03-11T21:36:04.117 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-03-11T21:36:04.117 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:36:04.117 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:36:04.117 INFO:teuthology.orchestra.run.smithi097.stdout: "description": "osd_op(client.15150.0:150458 8.15 8:ad4d3f8b:::benchmark_data_smithi092_104865_object29957:head [delete] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e213)", 2024-03-11T21:36:04.117 INFO:teuthology.orchestra.run.smithi097.stdout: "initiated_at": "2024-03-11T21:34:01.652091+0000", 2024-03-11T21:36:04.117 INFO:teuthology.orchestra.run.smithi097.stdout: "age": 122.472398979, 2024-03-11T21:36:04.117 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.0164343689999999, 2024-03-11T21:36:04.117 INFO:teuthology.orchestra.run.smithi097.stdout: "type_data": { 2024-03-11T21:36:04.117 INFO:teuthology.orchestra.run.smithi097.stdout: "flag_point": "reached pg", 2024-03-11T21:36:04.117 INFO:teuthology.orchestra.run.smithi097.stdout: "client_info": { 2024-03-11T21:36:04.117 INFO:teuthology.orchestra.run.smithi097.stdout: "client": "client.15150", 2024-03-11T21:36:04.118 INFO:teuthology.orchestra.run.smithi097.stdout: "client_addr": "v1:172.21.15.92:0/436606717", 2024-03-11T21:36:04.118 INFO:teuthology.orchestra.run.smithi097.stdout: "tid": 150458 2024-03-11T21:36:04.118 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:36:04.118 INFO:teuthology.orchestra.run.smithi097.stdout: "events": [ 2024-03-11T21:36:04.118 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:36:04.118 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "initiated", 2024-03-11T21:36:04.118 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:34:01.652091+0000", 2024-03-11T21:36:04.118 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-03-11T21:36:04.118 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:36:04.118 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:36:04.118 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "header_read", 2024-03-11T21:36:04.118 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:34:01.652091+0000", 2024-03-11T21:36:04.118 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-03-11T21:36:04.118 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:36:04.119 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:36:04.119 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "throttled", 2024-03-11T21:36:04.119 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:34:01.652092+0000", 2024-03-11T21:36:04.119 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 7.6899999999999996e-07 2024-03-11T21:36:04.119 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:36:04.119 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:36:04.119 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "all_read", 2024-03-11T21:36:04.119 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:34:01.652096+0000", 2024-03-11T21:36:04.119 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 3.456e-06 2024-03-11T21:36:04.119 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:36:04.119 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:36:04.119 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "dispatched", 2024-03-11T21:36:04.119 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:34:01.652103+0000", 2024-03-11T21:36:04.119 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 6.9970000000000001e-06 2024-03-11T21:36:04.119 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:36:04.120 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:36:04.120 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "queued_for_pg", 2024-03-11T21:36:04.120 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:34:01.652114+0000", 2024-03-11T21:36:04.120 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.1342000000000001e-05 2024-03-11T21:36:04.120 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:36:04.120 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:36:04.120 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "reached_pg", 2024-03-11T21:36:04.120 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:34:01.652188+0000", 2024-03-11T21:36:04.120 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 7.4315999999999998e-05 2024-03-11T21:36:04.120 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:36:04.120 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:36:04.120 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "waiting for peered", 2024-03-11T21:36:04.120 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:34:01.652196+0000", 2024-03-11T21:36:04.120 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 8.2600000000000005e-06 2024-03-11T21:36:04.120 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:36:04.121 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:36:04.121 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "reached_pg", 2024-03-11T21:36:04.121 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:34:02.663624+0000", 2024-03-11T21:36:04.121 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.0114278000000001 2024-03-11T21:36:04.121 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:36:04.121 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:36:04.121 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "waiting for missing object", 2024-03-11T21:36:04.121 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:34:02.663744+0000", 2024-03-11T21:36:04.121 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.000119999 2024-03-11T21:36:04.121 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:36:04.121 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:36:04.121 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "reached_pg", 2024-03-11T21:36:04.121 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:34:02.668438+0000", 2024-03-11T21:36:04.121 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.0046941680000000003 2024-03-11T21:36:04.121 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:36:04.122 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:36:04.122 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "done", 2024-03-11T21:36:04.122 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:34:02.668526+0000", 2024-03-11T21:36:04.122 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 8.7262000000000006e-05 2024-03-11T21:36:04.122 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-03-11T21:36:04.122 INFO:teuthology.orchestra.run.smithi097.stdout: ] 2024-03-11T21:36:04.122 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-03-11T21:36:04.122 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:36:04.122 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:36:04.122 INFO:teuthology.orchestra.run.smithi097.stdout: "description": "osd_op(client.15150.0:150469 8.15 8:aaea3dc1:::benchmark_data_smithi092_104865_object29968:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e213)", 2024-03-11T21:36:04.122 INFO:teuthology.orchestra.run.smithi097.stdout: "initiated_at": "2024-03-11T21:34:01.657742+0000", 2024-03-11T21:36:04.122 INFO:teuthology.orchestra.run.smithi097.stdout: "age": 122.466748338, 2024-03-11T21:36:04.122 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.0148719820000001, 2024-03-11T21:36:04.122 INFO:teuthology.orchestra.run.smithi097.stdout: "type_data": { 2024-03-11T21:36:04.123 INFO:teuthology.orchestra.run.smithi097.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-11T21:36:04.123 INFO:teuthology.orchestra.run.smithi097.stdout: "client_info": { 2024-03-11T21:36:04.123 INFO:teuthology.orchestra.run.smithi097.stdout: "client": "client.15150", 2024-03-11T21:36:04.123 INFO:teuthology.orchestra.run.smithi097.stdout: "client_addr": "v1:172.21.15.92:0/436606717", 2024-03-11T21:36:04.123 INFO:teuthology.orchestra.run.smithi097.stdout: "tid": 150469 2024-03-11T21:36:04.123 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:36:04.123 INFO:teuthology.orchestra.run.smithi097.stdout: "events": [ 2024-03-11T21:36:04.123 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:36:04.123 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "initiated", 2024-03-11T21:36:04.123 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:34:01.657742+0000", 2024-03-11T21:36:04.123 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-03-11T21:36:04.123 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:36:04.123 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:36:04.123 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "header_read", 2024-03-11T21:36:04.123 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:34:01.657742+0000", 2024-03-11T21:36:04.124 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-03-11T21:36:04.124 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:36:04.124 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:36:04.124 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "throttled", 2024-03-11T21:36:04.124 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:34:01.657744+0000", 2024-03-11T21:36:04.124 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 2.1959999999999998e-06 2024-03-11T21:36:04.124 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:36:04.124 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:36:04.124 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "all_read", 2024-03-11T21:36:04.124 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:34:01.657750+0000", 2024-03-11T21:36:04.124 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 6.2280000000000003e-06 2024-03-11T21:36:04.124 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:36:04.124 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:36:04.124 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "dispatched", 2024-03-11T21:36:04.124 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:34:01.657761+0000", 2024-03-11T21:36:04.125 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.0933e-05 2024-03-11T21:36:04.125 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:36:04.125 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:36:04.125 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "queued_for_pg", 2024-03-11T21:36:04.125 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:34:01.657771+0000", 2024-03-11T21:36:04.125 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 9.7489999999999993e-06 2024-03-11T21:36:04.125 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:36:04.125 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:36:04.125 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "reached_pg", 2024-03-11T21:36:04.125 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:34:01.660622+0000", 2024-03-11T21:36:04.125 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.0028506220000000001 2024-03-11T21:36:04.125 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:36:04.125 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:36:04.125 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "waiting for peered", 2024-03-11T21:36:04.126 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:34:01.660631+0000", 2024-03-11T21:36:04.126 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 9.4129999999999995e-06 2024-03-11T21:36:04.126 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:36:04.126 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:36:04.126 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "reached_pg", 2024-03-11T21:36:04.126 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:34:02.663808+0000", 2024-03-11T21:36:04.126 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.0031766390000001 2024-03-11T21:36:04.126 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:36:04.126 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:36:04.126 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "started", 2024-03-11T21:36:04.126 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:34:02.663929+0000", 2024-03-11T21:36:04.126 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.000121568 2024-03-11T21:36:04.126 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:36:04.126 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:36:04.127 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "waiting for subops from 1,6", 2024-03-11T21:36:04.127 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:34:02.664013+0000", 2024-03-11T21:36:04.127 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 8.4048999999999994e-05 2024-03-11T21:36:04.127 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:36:04.127 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:36:04.127 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "op_commit", 2024-03-11T21:36:04.127 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:34:02.668047+0000", 2024-03-11T21:36:04.127 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.0040336369999999996 2024-03-11T21:36:04.127 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:36:04.127 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:36:04.127 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "sub_op_commit_rec", 2024-03-11T21:36:04.127 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:34:02.668096+0000", 2024-03-11T21:36:04.127 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 4.8726000000000001e-05 2024-03-11T21:36:04.128 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:36:04.128 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:36:04.128 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "sub_op_commit_rec", 2024-03-11T21:36:04.128 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:34:02.672451+0000", 2024-03-11T21:36:04.128 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.0043555770000000002 2024-03-11T21:36:04.128 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:36:04.128 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:36:04.128 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "commit_sent", 2024-03-11T21:36:04.128 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:34:02.672509+0000", 2024-03-11T21:36:04.128 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 5.7725e-05 2024-03-11T21:36:04.128 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:36:04.128 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:36:04.128 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "done", 2024-03-11T21:36:04.129 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:34:02.672614+0000", 2024-03-11T21:36:04.129 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.00010492000000000001 2024-03-11T21:36:04.129 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-03-11T21:36:04.129 INFO:teuthology.orchestra.run.smithi097.stdout: ] 2024-03-11T21:36:04.129 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-03-11T21:36:04.129 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:36:04.129 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:36:04.129 INFO:teuthology.orchestra.run.smithi097.stdout: "description": "osd_op(client.15150.0:150474 8.15 8:ab344197:::benchmark_data_smithi092_104865_object29973:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e213)", 2024-03-11T21:36:04.129 INFO:teuthology.orchestra.run.smithi097.stdout: "initiated_at": "2024-03-11T21:34:01.664341+0000", 2024-03-11T21:36:04.129 INFO:teuthology.orchestra.run.smithi097.stdout: "age": 122.460149672, 2024-03-11T21:36:04.129 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.0085281740000001, 2024-03-11T21:36:04.129 INFO:teuthology.orchestra.run.smithi097.stdout: "type_data": { 2024-03-11T21:36:04.129 INFO:teuthology.orchestra.run.smithi097.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-11T21:36:04.129 INFO:teuthology.orchestra.run.smithi097.stdout: "client_info": { 2024-03-11T21:36:04.130 INFO:teuthology.orchestra.run.smithi097.stdout: "client": "client.15150", 2024-03-11T21:36:04.130 INFO:teuthology.orchestra.run.smithi097.stdout: "client_addr": "v1:172.21.15.92:0/436606717", 2024-03-11T21:36:04.130 INFO:teuthology.orchestra.run.smithi097.stdout: "tid": 150474 2024-03-11T21:36:04.130 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:36:04.130 INFO:teuthology.orchestra.run.smithi097.stdout: "events": [ 2024-03-11T21:36:04.130 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:36:04.130 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "initiated", 2024-03-11T21:36:04.130 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:34:01.664341+0000", 2024-03-11T21:36:04.130 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-03-11T21:36:04.130 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:36:04.130 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:36:04.130 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "header_read", 2024-03-11T21:36:04.130 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:34:01.664341+0000", 2024-03-11T21:36:04.130 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-03-11T21:36:04.131 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:36:04.131 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:36:04.131 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "throttled", 2024-03-11T21:36:04.131 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:34:01.664344+0000", 2024-03-11T21:36:04.131 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 2.937e-06 2024-03-11T21:36:04.131 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:36:04.131 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:36:04.131 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "all_read", 2024-03-11T21:36:04.131 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:34:01.664351+0000", 2024-03-11T21:36:04.131 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 7.097e-06 2024-03-11T21:36:04.131 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:36:04.131 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:36:04.131 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "dispatched", 2024-03-11T21:36:04.131 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:34:01.664365+0000", 2024-03-11T21:36:04.131 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.3933e-05 2024-03-11T21:36:04.132 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:36:04.132 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:36:04.132 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "queued_for_pg", 2024-03-11T21:36:04.132 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:34:01.664374+0000", 2024-03-11T21:36:04.132 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 9.3510000000000008e-06 2024-03-11T21:36:04.132 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:36:04.132 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:36:04.132 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "reached_pg", 2024-03-11T21:36:04.132 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:34:01.664436+0000", 2024-03-11T21:36:04.132 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 6.2026000000000006e-05 2024-03-11T21:36:04.132 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:36:04.132 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:36:04.132 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "waiting for peered", 2024-03-11T21:36:04.132 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:34:01.664447+0000", 2024-03-11T21:36:04.133 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.0781000000000001e-05 2024-03-11T21:36:04.133 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:36:04.133 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:36:04.133 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "reached_pg", 2024-03-11T21:36:04.133 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:34:02.665881+0000", 2024-03-11T21:36:04.133 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.001434283 2024-03-11T21:36:04.133 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:36:04.133 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:36:04.133 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "started", 2024-03-11T21:36:04.133 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:34:02.666004+0000", 2024-03-11T21:36:04.133 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.00012284899999999999 2024-03-11T21:36:04.133 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:36:04.133 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:36:04.133 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "waiting for subops from 1,6", 2024-03-11T21:36:04.133 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:34:02.666147+0000", 2024-03-11T21:36:04.134 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.00014334999999999999 2024-03-11T21:36:04.134 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:36:04.134 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:36:04.134 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "op_commit", 2024-03-11T21:36:04.134 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:34:02.668560+0000", 2024-03-11T21:36:04.134 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.002412544 2024-03-11T21:36:04.134 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:36:04.134 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:36:04.134 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "sub_op_commit_rec", 2024-03-11T21:36:04.134 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:34:02.668600+0000", 2024-03-11T21:36:04.134 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 4.0648000000000001e-05 2024-03-11T21:36:04.134 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:36:04.134 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:36:04.135 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "sub_op_commit_rec", 2024-03-11T21:36:04.135 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:34:02.672663+0000", 2024-03-11T21:36:04.135 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.0040630329999999997 2024-03-11T21:36:04.135 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:36:04.135 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:36:04.135 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "commit_sent", 2024-03-11T21:36:04.135 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:34:02.672749+0000", 2024-03-11T21:36:04.135 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 8.5470999999999995e-05 2024-03-11T21:36:04.135 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:36:04.135 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:36:04.135 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "done", 2024-03-11T21:36:04.135 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:34:02.672869+0000", 2024-03-11T21:36:04.135 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.000119871 2024-03-11T21:36:04.135 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-03-11T21:36:04.136 INFO:teuthology.orchestra.run.smithi097.stdout: ] 2024-03-11T21:36:04.136 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-03-11T21:36:04.136 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:36:04.136 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:36:04.136 INFO:teuthology.orchestra.run.smithi097.stdout: "description": "osd_op(client.15150.0:150552 8.15 8:adbf6385:::benchmark_data_smithi092_104865_object30051:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e213)", 2024-03-11T21:36:04.136 INFO:teuthology.orchestra.run.smithi097.stdout: "initiated_at": "2024-03-11T21:34:01.738436+0000", 2024-03-11T21:36:04.136 INFO:teuthology.orchestra.run.smithi097.stdout: "age": 122.386054364, 2024-03-11T21:36:04.136 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.93839203400000004, 2024-03-11T21:36:04.136 INFO:teuthology.orchestra.run.smithi097.stdout: "type_data": { 2024-03-11T21:36:04.136 INFO:teuthology.orchestra.run.smithi097.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-11T21:36:04.136 INFO:teuthology.orchestra.run.smithi097.stdout: "client_info": { 2024-03-11T21:36:04.136 INFO:teuthology.orchestra.run.smithi097.stdout: "client": "client.15150", 2024-03-11T21:36:04.136 INFO:teuthology.orchestra.run.smithi097.stdout: "client_addr": "v1:172.21.15.92:0/436606717", 2024-03-11T21:36:04.136 INFO:teuthology.orchestra.run.smithi097.stdout: "tid": 150552 2024-03-11T21:36:04.136 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:36:04.137 INFO:teuthology.orchestra.run.smithi097.stdout: "events": [ 2024-03-11T21:36:04.137 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:36:04.137 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "initiated", 2024-03-11T21:36:04.137 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:34:01.738436+0000", 2024-03-11T21:36:04.137 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-03-11T21:36:04.137 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:36:04.137 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:36:04.137 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "header_read", 2024-03-11T21:36:04.137 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:34:01.738436+0000", 2024-03-11T21:36:04.137 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-03-11T21:36:04.137 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:36:04.137 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:36:04.137 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "throttled", 2024-03-11T21:36:04.137 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:34:01.738439+0000", 2024-03-11T21:36:04.137 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 2.576e-06 2024-03-11T21:36:04.138 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:36:04.138 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:36:04.138 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "all_read", 2024-03-11T21:36:04.138 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:34:01.738447+0000", 2024-03-11T21:36:04.138 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 8.4330000000000004e-06 2024-03-11T21:36:04.138 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:36:04.138 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:36:04.138 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "dispatched", 2024-03-11T21:36:04.138 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:34:01.738463+0000", 2024-03-11T21:36:04.138 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.6019999999999999e-05 2024-03-11T21:36:04.138 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:36:04.138 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:36:04.138 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "queued_for_pg", 2024-03-11T21:36:04.138 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:34:01.738471+0000", 2024-03-11T21:36:04.139 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 8.388e-06 2024-03-11T21:36:04.139 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:36:04.139 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:36:04.139 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "reached_pg", 2024-03-11T21:36:04.139 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:36:04.140 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:34:01.738530+0000", 2024-03-11T21:36:04.140 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 5.8242000000000002e-05 2024-03-11T21:36:04.141 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:36:04.141 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:36:04.141 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "waiting for peered", 2024-03-11T21:36:04.141 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:34:01.738536+0000", 2024-03-11T21:36:04.141 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 6.8580000000000002e-06 2024-03-11T21:36:04.141 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:36:04.141 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:36:04.141 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "reached_pg", 2024-03-11T21:36:04.141 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:34:02.668651+0000", 2024-03-11T21:36:04.141 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.93011446600000003 2024-03-11T21:36:04.141 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:36:04.141 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:36:04.141 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "started", 2024-03-11T21:36:04.141 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:34:02.668773+0000", 2024-03-11T21:36:04.142 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.000122002 2024-03-11T21:36:04.142 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:36:04.142 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:36:04.142 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "waiting for subops from 1,6", 2024-03-11T21:36:04.142 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:34:02.668896+0000", 2024-03-11T21:36:04.142 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.000122652 2024-03-11T21:36:04.142 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:36:04.142 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:36:04.142 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "sub_op_commit_rec", 2024-03-11T21:36:04.142 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:34:02.672932+0000", 2024-03-11T21:36:04.142 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.0040368950000000004 2024-03-11T21:36:04.142 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:36:04.142 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:36:04.142 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "op_commit", 2024-03-11T21:36:04.143 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:34:02.673021+0000", 2024-03-11T21:36:04.143 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 8.8218000000000001e-05 2024-03-11T21:36:04.143 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:36:04.143 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:36:04.143 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "sub_op_commit_rec", 2024-03-11T21:36:04.143 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:34:02.676507+0000", 2024-03-11T21:36:04.143 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.0034864969999999999 2024-03-11T21:36:04.143 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:36:04.143 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:36:04.143 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "commit_sent", 2024-03-11T21:36:04.143 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:34:02.676567+0000", 2024-03-11T21:36:04.143 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 5.9914000000000003e-05 2024-03-11T21:36:04.143 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:36:04.143 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:36:04.144 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "done", 2024-03-11T21:36:04.144 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:34:02.676828+0000", 2024-03-11T21:36:04.144 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.00026087300000000001 2024-03-11T21:36:04.144 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-03-11T21:36:04.144 INFO:teuthology.orchestra.run.smithi097.stdout: ] 2024-03-11T21:36:04.144 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-03-11T21:36:04.144 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:36:04.144 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:36:04.144 INFO:teuthology.orchestra.run.smithi097.stdout: "description": "osd_op(client.15150.0:150558 8.15 8:aa3c4cf2:::benchmark_data_smithi092_104865_object30057:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e213)", 2024-03-11T21:36:04.144 INFO:teuthology.orchestra.run.smithi097.stdout: "initiated_at": "2024-03-11T21:34:01.740197+0000", 2024-03-11T21:36:04.144 INFO:teuthology.orchestra.run.smithi097.stdout: "age": 122.38429294700001, 2024-03-11T21:36:04.144 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.93882425199999997, 2024-03-11T21:36:04.144 INFO:teuthology.orchestra.run.smithi097.stdout: "type_data": { 2024-03-11T21:36:04.144 INFO:teuthology.orchestra.run.smithi097.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-11T21:36:04.145 INFO:teuthology.orchestra.run.smithi097.stdout: "client_info": { 2024-03-11T21:36:04.145 INFO:teuthology.orchestra.run.smithi097.stdout: "client": "client.15150", 2024-03-11T21:36:04.145 INFO:teuthology.orchestra.run.smithi097.stdout: "client_addr": "v1:172.21.15.92:0/436606717", 2024-03-11T21:36:04.145 INFO:teuthology.orchestra.run.smithi097.stdout: "tid": 150558 2024-03-11T21:36:04.145 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:36:04.145 INFO:teuthology.orchestra.run.smithi097.stdout: "events": [ 2024-03-11T21:36:04.145 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:36:04.145 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "initiated", 2024-03-11T21:36:04.145 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:34:01.740197+0000", 2024-03-11T21:36:04.145 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-03-11T21:36:04.145 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:36:04.145 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:36:04.145 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "header_read", 2024-03-11T21:36:04.145 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:34:01.740197+0000", 2024-03-11T21:36:04.146 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-03-11T21:36:04.146 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:36:04.146 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:36:04.146 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "throttled", 2024-03-11T21:36:04.146 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:34:01.740199+0000", 2024-03-11T21:36:04.146 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.7689999999999999e-06 2024-03-11T21:36:04.146 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:36:04.146 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:36:04.146 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "all_read", 2024-03-11T21:36:04.146 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:34:01.740205+0000", 2024-03-11T21:36:04.146 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 5.942e-06 2024-03-11T21:36:04.146 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:36:04.146 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:36:04.146 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "dispatched", 2024-03-11T21:36:04.147 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:34:01.740221+0000", 2024-03-11T21:36:04.147 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.6371e-05 2024-03-11T21:36:04.147 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:36:04.147 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:36:04.147 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "queued_for_pg", 2024-03-11T21:36:04.147 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:34:01.740230+0000", 2024-03-11T21:36:04.147 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 8.5290000000000008e-06 2024-03-11T21:36:04.147 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:36:04.147 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:36:04.147 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "reached_pg", 2024-03-11T21:36:04.147 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:34:01.740295+0000", 2024-03-11T21:36:04.147 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 6.5190000000000004e-05 2024-03-11T21:36:04.147 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:36:04.147 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:36:04.147 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "waiting for peered", 2024-03-11T21:36:04.148 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:34:01.740305+0000", 2024-03-11T21:36:04.148 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 9.4839999999999993e-06 2024-03-11T21:36:04.148 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:36:04.148 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:36:04.148 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "reached_pg", 2024-03-11T21:36:04.148 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:34:02.670688+0000", 2024-03-11T21:36:04.148 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.93038337299999996 2024-03-11T21:36:04.148 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:36:04.148 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:36:04.148 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "started", 2024-03-11T21:36:04.148 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:34:02.670790+0000", 2024-03-11T21:36:04.148 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.000102079 2024-03-11T21:36:04.148 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:36:04.148 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:36:04.148 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "waiting for subops from 1,6", 2024-03-11T21:36:04.149 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:34:02.670901+0000", 2024-03-11T21:36:04.149 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.0001106 2024-03-11T21:36:04.149 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:36:04.149 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:36:04.149 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "sub_op_commit_rec", 2024-03-11T21:36:04.149 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:34:02.673004+0000", 2024-03-11T21:36:04.149 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.0021035390000000002 2024-03-11T21:36:04.149 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:36:04.149 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:36:04.149 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "op_commit", 2024-03-11T21:36:04.149 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:34:02.676847+0000", 2024-03-11T21:36:04.149 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.0038423590000000001 2024-03-11T21:36:04.149 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:36:04.149 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:36:04.149 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "sub_op_commit_rec", 2024-03-11T21:36:04.150 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:34:02.678835+0000", 2024-03-11T21:36:04.150 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.0019883129999999998 2024-03-11T21:36:04.150 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:36:04.150 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:36:04.150 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "commit_sent", 2024-03-11T21:36:04.150 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:34:02.678918+0000", 2024-03-11T21:36:04.150 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 8.3443999999999996e-05 2024-03-11T21:36:04.150 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:36:04.150 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:36:04.150 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "done", 2024-03-11T21:36:04.150 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:34:02.679022+0000", 2024-03-11T21:36:04.150 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.00010326 2024-03-11T21:36:04.150 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-03-11T21:36:04.150 INFO:teuthology.orchestra.run.smithi097.stdout: ] 2024-03-11T21:36:04.150 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-03-11T21:36:04.151 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:36:04.151 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:36:04.151 INFO:teuthology.orchestra.run.smithi097.stdout: "description": "osd_op(client.15150.0:150566 8.15 8:a9de14ca:::benchmark_data_smithi092_104865_object30065:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e213)", 2024-03-11T21:36:04.151 INFO:teuthology.orchestra.run.smithi097.stdout: "initiated_at": "2024-03-11T21:34:01.751890+0000", 2024-03-11T21:36:04.151 INFO:teuthology.orchestra.run.smithi097.stdout: "age": 122.37260040300001, 2024-03-11T21:36:04.151 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.93812917100000004, 2024-03-11T21:36:04.151 INFO:teuthology.orchestra.run.smithi097.stdout: "type_data": { 2024-03-11T21:36:04.151 INFO:teuthology.orchestra.run.smithi097.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-11T21:36:04.151 INFO:teuthology.orchestra.run.smithi097.stdout: "client_info": { 2024-03-11T21:36:04.151 INFO:teuthology.orchestra.run.smithi097.stdout: "client": "client.15150", 2024-03-11T21:36:04.151 INFO:teuthology.orchestra.run.smithi097.stdout: "client_addr": "v1:172.21.15.92:0/436606717", 2024-03-11T21:36:04.151 INFO:teuthology.orchestra.run.smithi097.stdout: "tid": 150566 2024-03-11T21:36:04.151 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:36:04.152 INFO:teuthology.orchestra.run.smithi097.stdout: "events": [ 2024-03-11T21:36:04.152 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:36:04.152 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "initiated", 2024-03-11T21:36:04.152 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:34:01.751890+0000", 2024-03-11T21:36:04.152 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-03-11T21:36:04.152 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:36:04.152 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:36:04.152 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "header_read", 2024-03-11T21:36:04.152 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:34:01.751890+0000", 2024-03-11T21:36:04.152 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-03-11T21:36:04.152 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:36:04.152 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:36:04.152 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "throttled", 2024-03-11T21:36:04.152 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:34:01.751892+0000", 2024-03-11T21:36:04.152 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 2.2120000000000002e-06 2024-03-11T21:36:04.153 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:36:04.153 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:36:04.153 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "all_read", 2024-03-11T21:36:04.153 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:34:01.751899+0000", 2024-03-11T21:36:04.153 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 6.5660000000000003e-06 2024-03-11T21:36:04.153 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:36:04.153 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:36:04.153 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "dispatched", 2024-03-11T21:36:04.153 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:34:01.751913+0000", 2024-03-11T21:36:04.153 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.4419e-05 2024-03-11T21:36:04.153 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:36:04.153 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:36:04.153 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "queued_for_pg", 2024-03-11T21:36:04.153 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:34:01.751921+0000", 2024-03-11T21:36:04.153 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 7.6149999999999999e-06 2024-03-11T21:36:04.154 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:36:04.154 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:36:04.154 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "reached_pg", 2024-03-11T21:36:04.154 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:34:01.751974+0000", 2024-03-11T21:36:04.154 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 5.3646000000000001e-05 2024-03-11T21:36:04.154 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:36:04.154 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:36:04.154 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "waiting for peered", 2024-03-11T21:36:04.154 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:34:01.751981+0000", 2024-03-11T21:36:04.154 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 6.8759999999999999e-06 2024-03-11T21:36:04.154 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:36:04.154 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:36:04.155 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "reached_pg", 2024-03-11T21:36:04.155 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:34:02.673062+0000", 2024-03-11T21:36:04.155 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.92108067000000005 2024-03-11T21:36:04.155 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:36:04.155 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:36:04.155 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "started", 2024-03-11T21:36:04.155 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:34:02.673202+0000", 2024-03-11T21:36:04.155 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.000140147 2024-03-11T21:36:04.155 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:36:04.155 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:36:04.155 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "waiting for subops from 1,6", 2024-03-11T21:36:04.155 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:34:02.673336+0000", 2024-03-11T21:36:04.155 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.00013363200000000001 2024-03-11T21:36:04.155 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:36:04.156 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:36:04.156 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "sub_op_commit_rec", 2024-03-11T21:36:04.156 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:34:02.678731+0000", 2024-03-11T21:36:04.156 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.0053950439999999999 2024-03-11T21:36:04.156 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:36:04.156 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:36:04.156 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "op_commit", 2024-03-11T21:36:04.156 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:34:02.678745+0000", 2024-03-11T21:36:04.156 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.4314000000000001e-05 2024-03-11T21:36:04.156 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:36:04.156 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:36:04.156 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "sub_op_commit_rec", 2024-03-11T21:36:04.156 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:34:02.689835+0000", 2024-03-11T21:36:04.156 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.011089995 2024-03-11T21:36:04.156 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:36:04.156 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:36:04.156 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "commit_sent", 2024-03-11T21:36:04.157 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:34:02.689898+0000", 2024-03-11T21:36:04.157 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 6.2596999999999997e-05 2024-03-11T21:36:04.157 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:36:04.157 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:36:04.157 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "done", 2024-03-11T21:36:04.157 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:34:02.690019+0000", 2024-03-11T21:36:04.157 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.000121438 2024-03-11T21:36:04.157 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-03-11T21:36:04.157 INFO:teuthology.orchestra.run.smithi097.stdout: ] 2024-03-11T21:36:04.157 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-03-11T21:36:04.157 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:36:04.157 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:36:04.157 INFO:teuthology.orchestra.run.smithi097.stdout: "description": "osd_op(client.15150.0:150567 8.15 8:acd61c96:::benchmark_data_smithi092_104865_object30066:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e213)", 2024-03-11T21:36:04.157 INFO:teuthology.orchestra.run.smithi097.stdout: "initiated_at": "2024-03-11T21:34:01.751935+0000", 2024-03-11T21:36:04.157 INFO:teuthology.orchestra.run.smithi097.stdout: "age": 122.372555221, 2024-03-11T21:36:04.157 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.94653565699999997, 2024-03-11T21:36:04.158 INFO:teuthology.orchestra.run.smithi097.stdout: "type_data": { 2024-03-11T21:36:04.158 INFO:teuthology.orchestra.run.smithi097.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-11T21:36:04.158 INFO:teuthology.orchestra.run.smithi097.stdout: "client_info": { 2024-03-11T21:36:04.158 INFO:teuthology.orchestra.run.smithi097.stdout: "client": "client.15150", 2024-03-11T21:36:04.158 INFO:teuthology.orchestra.run.smithi097.stdout: "client_addr": "v1:172.21.15.92:0/436606717", 2024-03-11T21:36:04.158 INFO:teuthology.orchestra.run.smithi097.stdout: "tid": 150567 2024-03-11T21:36:04.158 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:36:04.158 INFO:teuthology.orchestra.run.smithi097.stdout: "events": [ 2024-03-11T21:36:04.158 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:36:04.158 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "initiated", 2024-03-11T21:36:04.158 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:34:01.751935+0000", 2024-03-11T21:36:04.158 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-03-11T21:36:04.158 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:36:04.158 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:36:04.159 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "header_read", 2024-03-11T21:36:04.159 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:34:01.751935+0000", 2024-03-11T21:36:04.159 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-03-11T21:36:04.159 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:36:04.159 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:36:04.159 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "throttled", 2024-03-11T21:36:04.159 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:34:01.751936+0000", 2024-03-11T21:36:04.159 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 4.3500000000000002e-07 2024-03-11T21:36:04.159 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:36:04.159 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:36:04.159 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "all_read", 2024-03-11T21:36:04.159 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:34:01.751938+0000", 2024-03-11T21:36:04.159 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 2.069e-06 2024-03-11T21:36:04.159 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:36:04.160 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:36:04.160 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "dispatched", 2024-03-11T21:36:04.160 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:34:01.751941+0000", 2024-03-11T21:36:04.160 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 3.5590000000000001e-06 2024-03-11T21:36:04.160 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:36:04.160 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:36:04.160 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "queued_for_pg", 2024-03-11T21:36:04.160 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:34:01.751944+0000", 2024-03-11T21:36:04.160 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 3.162e-06 2024-03-11T21:36:04.160 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:36:04.160 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:36:04.160 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "reached_pg", 2024-03-11T21:36:04.160 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:34:01.752002+0000", 2024-03-11T21:36:04.160 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 5.7822999999999998e-05 2024-03-11T21:36:04.160 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:36:04.161 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:36:04.161 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "waiting for peered", 2024-03-11T21:36:04.161 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:34:01.752009+0000", 2024-03-11T21:36:04.161 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 6.7560000000000002e-06 2024-03-11T21:36:04.161 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:36:04.161 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:36:04.161 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "reached_pg", 2024-03-11T21:36:04.161 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:34:02.674820+0000", 2024-03-11T21:36:04.161 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.92281079899999996 2024-03-11T21:36:04.161 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:36:04.161 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:36:04.161 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "started", 2024-03-11T21:36:04.161 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:34:02.674937+0000", 2024-03-11T21:36:04.161 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.000117217 2024-03-11T21:36:04.161 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:36:04.162 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:36:04.162 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "waiting for subops from 1,6", 2024-03-11T21:36:04.162 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:34:02.675002+0000", 2024-03-11T21:36:04.162 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 6.5345999999999994e-05 2024-03-11T21:36:04.162 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:36:04.162 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:36:04.162 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "op_commit", 2024-03-11T21:36:04.162 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:34:02.678751+0000", 2024-03-11T21:36:04.162 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.0037492250000000001 2024-03-11T21:36:04.162 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:36:04.162 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:36:04.162 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "sub_op_commit_rec", 2024-03-11T21:36:04.162 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:34:02.678784+0000", 2024-03-11T21:36:04.163 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 3.2539000000000001e-05 2024-03-11T21:36:04.163 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:36:04.163 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:36:04.163 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "sub_op_commit_rec", 2024-03-11T21:36:04.163 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:34:02.698408+0000", 2024-03-11T21:36:04.163 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.019623491999999999 2024-03-11T21:36:04.163 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:36:04.163 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:36:04.163 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "commit_sent", 2024-03-11T21:36:04.163 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:34:02.698444+0000", 2024-03-11T21:36:04.163 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 3.6634999999999997e-05 2024-03-11T21:36:04.163 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:36:04.163 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:36:04.163 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "done", 2024-03-11T21:36:04.163 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:34:02.698471+0000", 2024-03-11T21:36:04.164 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 2.6599999999999999e-05 2024-03-11T21:36:04.164 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-03-11T21:36:04.164 INFO:teuthology.orchestra.run.smithi097.stdout: ] 2024-03-11T21:36:04.166 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-03-11T21:36:04.166 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:36:04.166 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:36:04.166 INFO:teuthology.orchestra.run.smithi097.stdout: "description": "osd_op(client.15150.0:203653 8.17 8:ea1eb592:::benchmark_data_smithi092_104865_object83152:head [delete] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e233)", 2024-03-11T21:36:04.166 INFO:teuthology.orchestra.run.smithi097.stdout: "initiated_at": "2024-03-11T21:34:50.290471+0000", 2024-03-11T21:36:04.166 INFO:teuthology.orchestra.run.smithi097.stdout: "age": 73.834018974000003, 2024-03-11T21:36:04.166 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.0299367500000001, 2024-03-11T21:36:04.166 INFO:teuthology.orchestra.run.smithi097.stdout: "type_data": { 2024-03-11T21:36:04.166 INFO:teuthology.orchestra.run.smithi097.stdout: "flag_point": "reached pg", 2024-03-11T21:36:04.166 INFO:teuthology.orchestra.run.smithi097.stdout: "client_info": { 2024-03-11T21:36:04.166 INFO:teuthology.orchestra.run.smithi097.stdout: "client": "client.15150", 2024-03-11T21:36:04.166 INFO:teuthology.orchestra.run.smithi097.stdout: "client_addr": "v1:172.21.15.92:0/436606717", 2024-03-11T21:36:04.167 INFO:teuthology.orchestra.run.smithi097.stdout: "tid": 203653 2024-03-11T21:36:04.167 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:36:04.167 INFO:teuthology.orchestra.run.smithi097.stdout: "events": [ 2024-03-11T21:36:04.167 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:36:04.167 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "initiated", 2024-03-11T21:36:04.167 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:34:50.290471+0000", 2024-03-11T21:36:04.167 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-03-11T21:36:04.167 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:36:04.167 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:36:04.167 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "header_read", 2024-03-11T21:36:04.167 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:34:50.290471+0000", 2024-03-11T21:36:04.167 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-03-11T21:36:04.167 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:36:04.167 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:36:04.167 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "throttled", 2024-03-11T21:36:04.168 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:34:50.290473+0000", 2024-03-11T21:36:04.168 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 2.041e-06 2024-03-11T21:36:04.168 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:36:04.168 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:36:04.168 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "all_read", 2024-03-11T21:36:04.168 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:34:50.290479+0000", 2024-03-11T21:36:04.168 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 6.1099999999999999e-06 2024-03-11T21:36:04.168 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:36:04.168 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:36:04.168 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "dispatched", 2024-03-11T21:36:04.168 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:34:50.290496+0000", 2024-03-11T21:36:04.168 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.6069000000000001e-05 2024-03-11T21:36:04.168 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:36:04.168 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:36:04.169 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "queued_for_pg", 2024-03-11T21:36:04.169 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:34:50.290504+0000", 2024-03-11T21:36:04.169 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 8.0090000000000007e-06 2024-03-11T21:36:04.169 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:36:04.169 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:36:04.169 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "reached_pg", 2024-03-11T21:36:04.169 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:34:50.290577+0000", 2024-03-11T21:36:04.169 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 7.2954999999999995e-05 2024-03-11T21:36:04.169 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:36:04.169 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:36:04.169 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "op must wait for map", 2024-03-11T21:36:04.169 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:34:50.290584+0000", 2024-03-11T21:36:04.169 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 7.1420000000000004e-06 2024-03-11T21:36:04.170 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:36:04.170 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:36:04.170 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "reached_pg", 2024-03-11T21:36:04.170 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:34:50.297124+0000", 2024-03-11T21:36:04.170 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.0065406450000000003 2024-03-11T21:36:04.170 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:36:04.170 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:36:04.170 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "waiting for peered", 2024-03-11T21:36:04.170 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:34:50.297133+0000", 2024-03-11T21:36:04.170 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 8.9080000000000001e-06 2024-03-11T21:36:04.170 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:36:04.170 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:36:04.170 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "reached_pg", 2024-03-11T21:36:04.170 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:34:51.320325+0000", 2024-03-11T21:36:04.171 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.023191567 2024-03-11T21:36:04.171 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:36:04.171 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:36:04.171 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "done", 2024-03-11T21:36:04.171 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:34:51.320408+0000", 2024-03-11T21:36:04.171 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 8.3304000000000001e-05 2024-03-11T21:36:04.171 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-03-11T21:36:04.171 INFO:teuthology.orchestra.run.smithi097.stdout: ] 2024-03-11T21:36:04.171 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-03-11T21:36:04.171 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:36:04.171 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:36:04.171 INFO:teuthology.orchestra.run.smithi097.stdout: "description": "osd_op(client.15150.0:203682 8.17 8:e835ebdb:::benchmark_data_smithi092_104865_object83181:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e233)", 2024-03-11T21:36:04.171 INFO:teuthology.orchestra.run.smithi097.stdout: "initiated_at": "2024-03-11T21:34:50.309525+0000", 2024-03-11T21:36:04.171 INFO:teuthology.orchestra.run.smithi097.stdout: "age": 73.814965774000001, 2024-03-11T21:36:04.172 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.0163843560000001, 2024-03-11T21:36:04.172 INFO:teuthology.orchestra.run.smithi097.stdout: "type_data": { 2024-03-11T21:36:04.172 INFO:teuthology.orchestra.run.smithi097.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-11T21:36:04.172 INFO:teuthology.orchestra.run.smithi097.stdout: "client_info": { 2024-03-11T21:36:04.172 INFO:teuthology.orchestra.run.smithi097.stdout: "client": "client.15150", 2024-03-11T21:36:04.172 INFO:teuthology.orchestra.run.smithi097.stdout: "client_addr": "v1:172.21.15.92:0/436606717", 2024-03-11T21:36:04.172 INFO:teuthology.orchestra.run.smithi097.stdout: "tid": 203682 2024-03-11T21:36:04.172 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:36:04.172 INFO:teuthology.orchestra.run.smithi097.stdout: "events": [ 2024-03-11T21:36:04.172 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:36:04.172 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "initiated", 2024-03-11T21:36:04.172 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:34:50.309525+0000", 2024-03-11T21:36:04.172 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-03-11T21:36:04.172 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:36:04.172 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:36:04.173 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "header_read", 2024-03-11T21:36:04.173 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:34:50.309525+0000", 2024-03-11T21:36:04.173 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-03-11T21:36:04.173 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:36:04.173 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:36:04.173 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "throttled", 2024-03-11T21:36:04.173 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:34:50.309526+0000", 2024-03-11T21:36:04.173 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.762e-06 2024-03-11T21:36:04.173 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:36:04.173 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:36:04.173 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "all_read", 2024-03-11T21:36:04.173 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:34:50.309531+0000", 2024-03-11T21:36:04.173 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 4.8439999999999997e-06 2024-03-11T21:36:04.174 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:36:04.174 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:36:04.174 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "dispatched", 2024-03-11T21:36:04.174 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:34:50.309709+0000", 2024-03-11T21:36:04.174 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.00017786200000000001 2024-03-11T21:36:04.174 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:36:04.174 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:36:04.174 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "queued_for_pg", 2024-03-11T21:36:04.174 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:34:50.309718+0000", 2024-03-11T21:36:04.174 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 8.6060000000000003e-06 2024-03-11T21:36:04.174 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:36:04.174 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:36:04.174 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "reached_pg", 2024-03-11T21:36:04.174 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:34:50.309805+0000", 2024-03-11T21:36:04.174 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 8.7861999999999993e-05 2024-03-11T21:36:04.175 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:36:04.175 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:36:04.175 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "waiting for peered", 2024-03-11T21:36:04.175 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:34:50.309814+0000", 2024-03-11T21:36:04.175 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 8.9309999999999997e-06 2024-03-11T21:36:04.175 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:36:04.175 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:36:04.175 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "reached_pg", 2024-03-11T21:36:04.175 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:34:51.320819+0000", 2024-03-11T21:36:04.175 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.011005063 2024-03-11T21:36:04.175 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:36:04.175 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:36:04.175 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "started", 2024-03-11T21:36:04.176 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:34:51.320926+0000", 2024-03-11T21:36:04.176 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.000106252 2024-03-11T21:36:04.176 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:36:04.176 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:36:04.176 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "waiting for subops from 0,6", 2024-03-11T21:36:04.176 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:34:51.321218+0000", 2024-03-11T21:36:04.176 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.00029272 2024-03-11T21:36:04.176 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:36:04.176 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:36:04.176 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "sub_op_commit_rec", 2024-03-11T21:36:04.176 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:34:51.325634+0000", 2024-03-11T21:36:04.176 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.0044151559999999999 2024-03-11T21:36:04.176 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:36:04.176 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:36:04.176 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "op_commit", 2024-03-11T21:36:04.177 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:34:51.325752+0000", 2024-03-11T21:36:04.177 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.000118527 2024-03-11T21:36:04.177 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:36:04.177 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:36:04.177 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "sub_op_commit_rec", 2024-03-11T21:36:04.177 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:34:51.325811+0000", 2024-03-11T21:36:04.177 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 5.9212999999999997e-05 2024-03-11T21:36:04.177 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:36:04.177 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:36:04.177 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "commit_sent", 2024-03-11T21:36:04.177 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:34:51.325862+0000", 2024-03-11T21:36:04.177 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 5.0886e-05 2024-03-11T21:36:04.177 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:36:04.177 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:36:04.178 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "done", 2024-03-11T21:36:04.178 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:34:51.325909+0000", 2024-03-11T21:36:04.178 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 4.6672000000000003e-05 2024-03-11T21:36:04.178 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-03-11T21:36:04.178 INFO:teuthology.orchestra.run.smithi097.stdout: ] 2024-03-11T21:36:04.178 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-03-11T21:36:04.178 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:36:04.178 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:36:04.178 INFO:teuthology.orchestra.run.smithi097.stdout: "description": "osd_op(client.15150.0:203696 8.17 8:eddc2db3:::benchmark_data_smithi092_104865_object83195:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e233)", 2024-03-11T21:36:04.178 INFO:teuthology.orchestra.run.smithi097.stdout: "initiated_at": "2024-03-11T21:34:50.321937+0000", 2024-03-11T21:36:04.178 INFO:teuthology.orchestra.run.smithi097.stdout: "age": 73.802553672000002, 2024-03-11T21:36:04.178 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.0074158280000001, 2024-03-11T21:36:04.178 INFO:teuthology.orchestra.run.smithi097.stdout: "type_data": { 2024-03-11T21:36:04.178 INFO:teuthology.orchestra.run.smithi097.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-11T21:36:04.179 INFO:teuthology.orchestra.run.smithi097.stdout: "client_info": { 2024-03-11T21:36:04.179 INFO:teuthology.orchestra.run.smithi097.stdout: "client": "client.15150", 2024-03-11T21:36:04.179 INFO:teuthology.orchestra.run.smithi097.stdout: "client_addr": "v1:172.21.15.92:0/436606717", 2024-03-11T21:36:04.179 INFO:teuthology.orchestra.run.smithi097.stdout: "tid": 203696 2024-03-11T21:36:04.179 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:36:04.179 INFO:teuthology.orchestra.run.smithi097.stdout: "events": [ 2024-03-11T21:36:04.179 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:36:04.179 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "initiated", 2024-03-11T21:36:04.179 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:34:50.321937+0000", 2024-03-11T21:36:04.179 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-03-11T21:36:04.179 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:36:04.179 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:36:04.179 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "header_read", 2024-03-11T21:36:04.179 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:34:50.321937+0000", 2024-03-11T21:36:04.180 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-03-11T21:36:04.180 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:36:04.180 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:36:04.180 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "throttled", 2024-03-11T21:36:04.180 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:34:50.321939+0000", 2024-03-11T21:36:04.180 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 2.1459999999999999e-06 2024-03-11T21:36:04.180 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:36:04.180 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:36:04.180 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "all_read", 2024-03-11T21:36:04.180 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:34:50.321945+0000", 2024-03-11T21:36:04.180 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 6.3319999999999997e-06 2024-03-11T21:36:04.180 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:36:04.180 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:36:04.180 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "dispatched", 2024-03-11T21:36:04.180 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:34:50.321957+0000", 2024-03-11T21:36:04.180 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.2191000000000001e-05 2024-03-11T21:36:04.181 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:36:04.181 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:36:04.181 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "queued_for_pg", 2024-03-11T21:36:04.181 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:34:50.321965+0000", 2024-03-11T21:36:04.181 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 7.7789999999999996e-06 2024-03-11T21:36:04.181 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:36:04.181 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:36:04.181 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "reached_pg", 2024-03-11T21:36:04.181 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:34:50.322124+0000", 2024-03-11T21:36:04.181 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.00015893999999999999 2024-03-11T21:36:04.181 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:36:04.181 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:36:04.181 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "waiting for peered", 2024-03-11T21:36:04.181 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:34:50.322132+0000", 2024-03-11T21:36:04.182 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 8.2379999999999997e-06 2024-03-11T21:36:04.182 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:36:04.182 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:36:04.182 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "reached_pg", 2024-03-11T21:36:04.182 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:34:51.322497+0000", 2024-03-11T21:36:04.182 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.0003644279999999 2024-03-11T21:36:04.182 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:36:04.182 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:36:04.183 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "started", 2024-03-11T21:36:04.183 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:34:51.322604+0000", 2024-03-11T21:36:04.183 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.000107785 2024-03-11T21:36:04.183 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:36:04.183 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:36:04.184 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "waiting for subops from 0,6", 2024-03-11T21:36:04.184 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:34:51.322837+0000", 2024-03-11T21:36:04.184 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.000232592 2024-03-11T21:36:04.184 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:36:04.184 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:36:04.184 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "op_commit", 2024-03-11T21:36:04.184 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:34:51.327557+0000", 2024-03-11T21:36:04.184 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.0047198819999999999 2024-03-11T21:36:04.184 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:36:04.184 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:36:04.184 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "sub_op_commit_rec", 2024-03-11T21:36:04.184 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:34:51.329217+0000", 2024-03-11T21:36:04.184 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.0016600110000000001 2024-03-11T21:36:04.184 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:36:04.185 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:36:04.185 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "sub_op_commit_rec", 2024-03-11T21:36:04.185 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:34:51.329253+0000", 2024-03-11T21:36:04.185 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 3.5824000000000001e-05 2024-03-11T21:36:04.185 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:36:04.185 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:36:04.185 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "commit_sent", 2024-03-11T21:36:04.185 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:34:51.329306+0000", 2024-03-11T21:36:04.185 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 5.3241000000000001e-05 2024-03-11T21:36:04.185 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:36:04.185 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:36:04.185 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "done", 2024-03-11T21:36:04.185 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:34:51.329352+0000", 2024-03-11T21:36:04.185 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 4.6439e-05 2024-03-11T21:36:04.185 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-03-11T21:36:04.186 INFO:teuthology.orchestra.run.smithi097.stdout: ] 2024-03-11T21:36:04.186 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-03-11T21:36:04.186 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:36:04.186 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:36:04.186 INFO:teuthology.orchestra.run.smithi097.stdout: "description": "osd_op(client.15150.0:203717 8.17 8:e9113ebd:::benchmark_data_smithi092_104865_object83216:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e233)", 2024-03-11T21:36:04.186 INFO:teuthology.orchestra.run.smithi097.stdout: "initiated_at": "2024-03-11T21:34:50.337126+0000", 2024-03-11T21:36:04.186 INFO:teuthology.orchestra.run.smithi097.stdout: "age": 73.787363916000004, 2024-03-11T21:36:04.186 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.99565916300000001, 2024-03-11T21:36:04.186 INFO:teuthology.orchestra.run.smithi097.stdout: "type_data": { 2024-03-11T21:36:04.186 INFO:teuthology.orchestra.run.smithi097.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-11T21:36:04.186 INFO:teuthology.orchestra.run.smithi097.stdout: "client_info": { 2024-03-11T21:36:04.186 INFO:teuthology.orchestra.run.smithi097.stdout: "client": "client.15150", 2024-03-11T21:36:04.186 INFO:teuthology.orchestra.run.smithi097.stdout: "client_addr": "v1:172.21.15.92:0/436606717", 2024-03-11T21:36:04.186 INFO:teuthology.orchestra.run.smithi097.stdout: "tid": 203717 2024-03-11T21:36:04.186 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:36:04.187 INFO:teuthology.orchestra.run.smithi097.stdout: "events": [ 2024-03-11T21:36:04.187 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:36:04.187 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "initiated", 2024-03-11T21:36:04.187 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:34:50.337126+0000", 2024-03-11T21:36:04.187 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-03-11T21:36:04.187 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:36:04.187 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:36:04.187 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "header_read", 2024-03-11T21:36:04.187 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:34:50.337126+0000", 2024-03-11T21:36:04.187 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-03-11T21:36:04.187 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:36:04.187 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:36:04.187 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "throttled", 2024-03-11T21:36:04.187 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:34:50.337128+0000", 2024-03-11T21:36:04.188 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.871e-06 2024-03-11T21:36:04.188 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:36:04.188 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:36:04.188 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "all_read", 2024-03-11T21:36:04.188 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:34:50.337134+0000", 2024-03-11T21:36:04.188 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 5.5600000000000001e-06 2024-03-11T21:36:04.188 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:36:04.188 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:36:04.188 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "dispatched", 2024-03-11T21:36:04.188 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:34:50.337145+0000", 2024-03-11T21:36:04.188 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.1112e-05 2024-03-11T21:36:04.188 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:36:04.188 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:36:04.188 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "queued_for_pg", 2024-03-11T21:36:04.188 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:34:50.337153+0000", 2024-03-11T21:36:04.188 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 7.5889999999999997e-06 2024-03-11T21:36:04.189 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:36:04.189 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:36:04.189 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "reached_pg", 2024-03-11T21:36:04.189 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:34:50.337206+0000", 2024-03-11T21:36:04.189 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 5.3706999999999999e-05 2024-03-11T21:36:04.189 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:36:04.189 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:36:04.189 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "waiting for peered", 2024-03-11T21:36:04.189 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:34:50.337214+0000", 2024-03-11T21:36:04.189 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 7.294e-06 2024-03-11T21:36:04.189 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:36:04.189 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:36:04.189 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "reached_pg", 2024-03-11T21:36:04.190 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:34:51.324056+0000", 2024-03-11T21:36:04.190 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.98684222300000002 2024-03-11T21:36:04.190 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:36:04.190 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:36:04.190 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "started", 2024-03-11T21:36:04.190 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:34:51.324185+0000", 2024-03-11T21:36:04.190 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.00012944200000000001 2024-03-11T21:36:04.190 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:36:04.190 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:36:04.190 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "waiting for subops from 0,6", 2024-03-11T21:36:04.190 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:34:51.324400+0000", 2024-03-11T21:36:04.190 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.00021476899999999999 2024-03-11T21:36:04.190 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:36:04.190 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:36:04.191 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "op_commit", 2024-03-11T21:36:04.191 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:34:51.329178+0000", 2024-03-11T21:36:04.191 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.0047776019999999997 2024-03-11T21:36:04.191 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:36:04.191 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:36:04.191 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "sub_op_commit_rec", 2024-03-11T21:36:04.191 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:34:51.329400+0000", 2024-03-11T21:36:04.191 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.000222616 2024-03-11T21:36:04.191 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:36:04.191 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:36:04.191 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "sub_op_commit_rec", 2024-03-11T21:36:04.191 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:34:51.332703+0000", 2024-03-11T21:36:04.191 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.0033027870000000002 2024-03-11T21:36:04.191 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:36:04.191 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:36:04.192 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "commit_sent", 2024-03-11T21:36:04.192 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:34:51.332745+0000", 2024-03-11T21:36:04.192 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 4.2169000000000002e-05 2024-03-11T21:36:04.192 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:36:04.192 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:36:04.192 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "done", 2024-03-11T21:36:04.192 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:34:51.332786+0000", 2024-03-11T21:36:04.192 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 4.0422e-05 2024-03-11T21:36:04.192 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-03-11T21:36:04.192 INFO:teuthology.orchestra.run.smithi097.stdout: ] 2024-03-11T21:36:04.192 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-03-11T21:36:04.192 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:36:04.192 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:36:04.192 INFO:teuthology.orchestra.run.smithi097.stdout: "description": "osd_op(client.15150.0:203721 8.17 8:ea4988fc:::benchmark_data_smithi092_104865_object83220:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e233)", 2024-03-11T21:36:04.192 INFO:teuthology.orchestra.run.smithi097.stdout: "initiated_at": "2024-03-11T21:34:50.337910+0000", 2024-03-11T21:36:04.193 INFO:teuthology.orchestra.run.smithi097.stdout: "age": 73.786580534999999, 2024-03-11T21:36:04.193 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.99500893499999998, 2024-03-11T21:36:04.193 INFO:teuthology.orchestra.run.smithi097.stdout: "type_data": { 2024-03-11T21:36:04.193 INFO:teuthology.orchestra.run.smithi097.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-11T21:36:04.193 INFO:teuthology.orchestra.run.smithi097.stdout: "client_info": { 2024-03-11T21:36:04.193 INFO:teuthology.orchestra.run.smithi097.stdout: "client": "client.15150", 2024-03-11T21:36:04.193 INFO:teuthology.orchestra.run.smithi097.stdout: "client_addr": "v1:172.21.15.92:0/436606717", 2024-03-11T21:36:04.193 INFO:teuthology.orchestra.run.smithi097.stdout: "tid": 203721 2024-03-11T21:36:04.193 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:36:04.193 INFO:teuthology.orchestra.run.smithi097.stdout: "events": [ 2024-03-11T21:36:04.193 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:36:04.193 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "initiated", 2024-03-11T21:36:04.193 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:34:50.337910+0000", 2024-03-11T21:36:04.193 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-03-11T21:36:04.193 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:36:04.194 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:36:04.194 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "header_read", 2024-03-11T21:36:04.194 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:34:50.337910+0000", 2024-03-11T21:36:04.194 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-03-11T21:36:04.194 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:36:04.194 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:36:04.194 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "throttled", 2024-03-11T21:36:04.194 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:34:50.337911+0000", 2024-03-11T21:36:04.194 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.19e-06 2024-03-11T21:36:04.194 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:36:04.194 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:36:04.194 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "all_read", 2024-03-11T21:36:04.194 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:34:50.337918+0000", 2024-03-11T21:36:04.194 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 7.3590000000000002e-06 2024-03-11T21:36:04.194 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:36:04.195 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:36:04.195 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "dispatched", 2024-03-11T21:36:04.195 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:34:50.337927+0000", 2024-03-11T21:36:04.195 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 9.1719999999999996e-06 2024-03-11T21:36:04.195 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:36:04.195 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:36:04.195 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "queued_for_pg", 2024-03-11T21:36:04.195 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:34:50.337934+0000", 2024-03-11T21:36:04.195 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 6.9160000000000004e-06 2024-03-11T21:36:04.195 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:36:04.195 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:36:04.195 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "reached_pg", 2024-03-11T21:36:04.195 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:34:50.337988+0000", 2024-03-11T21:36:04.195 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 5.3919999999999999e-05 2024-03-11T21:36:04.195 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:36:04.196 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:36:04.196 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "waiting for peered", 2024-03-11T21:36:04.196 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:34:50.337996+0000", 2024-03-11T21:36:04.196 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 8.14e-06 2024-03-11T21:36:04.196 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:36:04.196 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:36:04.196 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "reached_pg", 2024-03-11T21:36:04.196 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:34:51.325950+0000", 2024-03-11T21:36:04.196 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.98795397299999999 2024-03-11T21:36:04.196 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:36:04.196 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:36:04.196 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "started", 2024-03-11T21:36:04.196 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:34:51.326051+0000", 2024-03-11T21:36:04.196 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.000100352 2024-03-11T21:36:04.196 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:36:04.197 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:36:04.197 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "waiting for subops from 0,6", 2024-03-11T21:36:04.197 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:34:51.326245+0000", 2024-03-11T21:36:04.197 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.000194604 2024-03-11T21:36:04.197 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:36:04.197 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:36:04.197 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "sub_op_commit_rec", 2024-03-11T21:36:04.197 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:34:51.329441+0000", 2024-03-11T21:36:04.197 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.003195472 2024-03-11T21:36:04.197 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:36:04.197 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:36:04.197 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "op_commit", 2024-03-11T21:36:04.197 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:34:51.330998+0000", 2024-03-11T21:36:04.197 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.0015573119999999999 2024-03-11T21:36:04.198 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:36:04.198 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:36:04.198 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "sub_op_commit_rec", 2024-03-11T21:36:04.198 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:34:51.332838+0000", 2024-03-11T21:36:04.198 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.001840221 2024-03-11T21:36:04.198 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:36:04.198 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:36:04.198 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "commit_sent", 2024-03-11T21:36:04.198 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:34:51.332879+0000", 2024-03-11T21:36:04.198 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 4.0445999999999999e-05 2024-03-11T21:36:04.198 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:36:04.198 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:36:04.198 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "done", 2024-03-11T21:36:04.199 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:34:51.332919+0000", 2024-03-11T21:36:04.199 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 3.9857999999999997e-05 2024-03-11T21:36:04.199 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-03-11T21:36:04.199 INFO:teuthology.orchestra.run.smithi097.stdout: ] 2024-03-11T21:36:04.199 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-03-11T21:36:04.199 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:36:04.199 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:36:04.199 INFO:teuthology.orchestra.run.smithi097.stdout: "description": "osd_op(client.15150.0:203735 8.17 8:ea4806f2:::benchmark_data_smithi092_104865_object83234:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e233)", 2024-03-11T21:36:04.199 INFO:teuthology.orchestra.run.smithi097.stdout: "initiated_at": "2024-03-11T21:34:50.349993+0000", 2024-03-11T21:36:04.199 INFO:teuthology.orchestra.run.smithi097.stdout: "age": 73.774497541000002, 2024-03-11T21:36:04.199 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.98641120400000004, 2024-03-11T21:36:04.199 INFO:teuthology.orchestra.run.smithi097.stdout: "type_data": { 2024-03-11T21:36:04.199 INFO:teuthology.orchestra.run.smithi097.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-11T21:36:04.200 INFO:teuthology.orchestra.run.smithi097.stdout: "client_info": { 2024-03-11T21:36:04.200 INFO:teuthology.orchestra.run.smithi097.stdout: "client": "client.15150", 2024-03-11T21:36:04.200 INFO:teuthology.orchestra.run.smithi097.stdout: "client_addr": "v1:172.21.15.92:0/436606717", 2024-03-11T21:36:04.200 INFO:teuthology.orchestra.run.smithi097.stdout: "tid": 203735 2024-03-11T21:36:04.200 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:36:04.200 INFO:teuthology.orchestra.run.smithi097.stdout: "events": [ 2024-03-11T21:36:04.200 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:36:04.200 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "initiated", 2024-03-11T21:36:04.200 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:34:50.349993+0000", 2024-03-11T21:36:04.200 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-03-11T21:36:04.200 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:36:04.200 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:36:04.200 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "header_read", 2024-03-11T21:36:04.200 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:34:50.349993+0000", 2024-03-11T21:36:04.201 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-03-11T21:36:04.201 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:36:04.201 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:36:04.201 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "throttled", 2024-03-11T21:36:04.201 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:34:50.349995+0000", 2024-03-11T21:36:04.201 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 2.0999999999999998e-06 2024-03-11T21:36:04.201 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:36:04.201 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:36:04.201 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "all_read", 2024-03-11T21:36:04.201 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:34:50.350000+0000", 2024-03-11T21:36:04.201 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 5.6319999999999996e-06 2024-03-11T21:36:04.201 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:36:04.201 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:36:04.201 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "dispatched", 2024-03-11T21:36:04.202 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:34:50.350016+0000", 2024-03-11T21:36:04.202 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.5406000000000001e-05 2024-03-11T21:36:04.202 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:36:04.202 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:36:04.202 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "queued_for_pg", 2024-03-11T21:36:04.202 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:34:50.350024+0000", 2024-03-11T21:36:04.202 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 8.4700000000000002e-06 2024-03-11T21:36:04.202 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:36:04.202 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:36:04.202 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "reached_pg", 2024-03-11T21:36:04.203 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:34:50.350076+0000", 2024-03-11T21:36:04.203 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 5.1838000000000001e-05 2024-03-11T21:36:04.203 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:36:04.203 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:36:04.203 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "waiting for peered", 2024-03-11T21:36:04.203 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:34:50.350084+0000", 2024-03-11T21:36:04.203 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 7.9740000000000002e-06 2024-03-11T21:36:04.203 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:36:04.203 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:36:04.203 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "reached_pg", 2024-03-11T21:36:04.203 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:34:51.327585+0000", 2024-03-11T21:36:04.203 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.97750038299999997 2024-03-11T21:36:04.203 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:36:04.203 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:36:04.204 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "started", 2024-03-11T21:36:04.204 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:34:51.327682+0000", 2024-03-11T21:36:04.204 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 9.7720000000000006e-05 2024-03-11T21:36:04.204 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:36:04.204 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:36:04.204 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "waiting for subops from 0,6", 2024-03-11T21:36:04.204 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:34:51.327889+0000", 2024-03-11T21:36:04.204 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.00020693900000000001 2024-03-11T21:36:04.204 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:36:04.204 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:36:04.204 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "sub_op_commit_rec", 2024-03-11T21:36:04.204 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:34:51.332647+0000", 2024-03-11T21:36:04.204 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.0047573939999999999 2024-03-11T21:36:04.204 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:36:04.205 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:36:04.205 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "op_commit", 2024-03-11T21:36:04.205 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:34:51.332659+0000", 2024-03-11T21:36:04.205 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.2488e-05 2024-03-11T21:36:04.205 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:36:04.205 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:36:04.205 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "sub_op_commit_rec", 2024-03-11T21:36:04.205 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:34:51.336312+0000", 2024-03-11T21:36:04.205 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.0036532610000000001 2024-03-11T21:36:04.205 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:36:04.205 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:36:04.205 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "commit_sent", 2024-03-11T21:36:04.205 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:34:51.336359+0000", 2024-03-11T21:36:04.205 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 4.7098999999999997e-05 2024-03-11T21:36:04.205 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:36:04.206 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:36:04.206 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "done", 2024-03-11T21:36:04.206 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:34:51.336404+0000", 2024-03-11T21:36:04.206 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 4.4499999999999997e-05 2024-03-11T21:36:04.206 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-03-11T21:36:04.206 INFO:teuthology.orchestra.run.smithi097.stdout: ] 2024-03-11T21:36:04.206 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-03-11T21:36:04.206 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:36:04.206 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:36:04.206 INFO:teuthology.orchestra.run.smithi097.stdout: "description": "osd_op(client.15150.0:203739 8.17 8:ec74d790:::benchmark_data_smithi092_104865_object83238:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e233)", 2024-03-11T21:36:04.206 INFO:teuthology.orchestra.run.smithi097.stdout: "initiated_at": "2024-03-11T21:34:50.352271+0000", 2024-03-11T21:36:04.206 INFO:teuthology.orchestra.run.smithi097.stdout: "age": 73.772219535999994, 2024-03-11T21:36:04.206 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.98429273100000003, 2024-03-11T21:36:04.206 INFO:teuthology.orchestra.run.smithi097.stdout: "type_data": { 2024-03-11T21:36:04.206 INFO:teuthology.orchestra.run.smithi097.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-11T21:36:04.207 INFO:teuthology.orchestra.run.smithi097.stdout: "client_info": { 2024-03-11T21:36:04.207 INFO:teuthology.orchestra.run.smithi097.stdout: "client": "client.15150", 2024-03-11T21:36:04.207 INFO:teuthology.orchestra.run.smithi097.stdout: "client_addr": "v1:172.21.15.92:0/436606717", 2024-03-11T21:36:04.207 INFO:teuthology.orchestra.run.smithi097.stdout: "tid": 203739 2024-03-11T21:36:04.207 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:36:04.207 INFO:teuthology.orchestra.run.smithi097.stdout: "events": [ 2024-03-11T21:36:04.207 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:36:04.207 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "initiated", 2024-03-11T21:36:04.207 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:34:50.352271+0000", 2024-03-11T21:36:04.207 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-03-11T21:36:04.207 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:36:04.207 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:36:04.207 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "header_read", 2024-03-11T21:36:04.207 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:34:50.352271+0000", 2024-03-11T21:36:04.208 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-03-11T21:36:04.208 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:36:04.208 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:36:04.208 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "throttled", 2024-03-11T21:36:04.208 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:34:50.352272+0000", 2024-03-11T21:36:04.208 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.3599999999999999e-06 2024-03-11T21:36:04.208 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:36:04.208 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:36:04.208 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "all_read", 2024-03-11T21:36:04.208 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:34:50.352281+0000", 2024-03-11T21:36:04.208 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 8.602e-06 2024-03-11T21:36:04.208 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:36:04.208 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:36:04.208 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "dispatched", 2024-03-11T21:36:04.208 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:34:50.352301+0000", 2024-03-11T21:36:04.209 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 2.0489999999999999e-05 2024-03-11T21:36:04.209 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:36:04.209 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:36:04.209 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "queued_for_pg", 2024-03-11T21:36:04.209 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:34:50.352308+0000", 2024-03-11T21:36:04.209 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 6.3600000000000001e-06 2024-03-11T21:36:04.209 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:36:04.209 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:36:04.209 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "reached_pg", 2024-03-11T21:36:04.209 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:34:50.352388+0000", 2024-03-11T21:36:04.209 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 8.0277000000000004e-05 2024-03-11T21:36:04.209 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:36:04.209 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:36:04.209 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "waiting for peered", 2024-03-11T21:36:04.209 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:34:50.352396+0000", 2024-03-11T21:36:04.210 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 8.5380000000000002e-06 2024-03-11T21:36:04.210 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:36:04.210 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:36:04.210 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "reached_pg", 2024-03-11T21:36:04.210 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:34:51.329474+0000", 2024-03-11T21:36:04.210 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.97707767300000004 2024-03-11T21:36:04.210 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:36:04.210 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:36:04.210 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "started", 2024-03-11T21:36:04.210 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:34:51.329570+0000", 2024-03-11T21:36:04.210 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 9.6329000000000006e-05 2024-03-11T21:36:04.210 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:36:04.210 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:36:04.210 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "waiting for subops from 0,6", 2024-03-11T21:36:04.211 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:34:51.329749+0000", 2024-03-11T21:36:04.211 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.00017819 2024-03-11T21:36:04.211 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:36:04.211 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:36:04.211 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "op_commit", 2024-03-11T21:36:04.211 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:34:51.332938+0000", 2024-03-11T21:36:04.211 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.0031892499999999998 2024-03-11T21:36:04.211 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:36:04.211 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:36:04.211 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "sub_op_commit_rec", 2024-03-11T21:36:04.211 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:34:51.336443+0000", 2024-03-11T21:36:04.211 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.003504783 2024-03-11T21:36:04.211 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:36:04.211 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:36:04.211 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "sub_op_commit_rec", 2024-03-11T21:36:04.212 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:34:51.336482+0000", 2024-03-11T21:36:04.212 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 3.9579000000000002e-05 2024-03-11T21:36:04.212 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:36:04.212 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:36:04.212 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "commit_sent", 2024-03-11T21:36:04.212 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:34:51.336527+0000", 2024-03-11T21:36:04.212 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 4.5151000000000002e-05 2024-03-11T21:36:04.212 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:36:04.212 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:36:04.212 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "done", 2024-03-11T21:36:04.212 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:34:51.336564+0000", 2024-03-11T21:36:04.212 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 3.6149000000000002e-05 2024-03-11T21:36:04.212 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-03-11T21:36:04.212 INFO:teuthology.orchestra.run.smithi097.stdout: ] 2024-03-11T21:36:04.213 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-03-11T21:36:04.213 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:36:04.213 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:36:04.213 INFO:teuthology.orchestra.run.smithi097.stdout: "description": "osd_op(client.15150.0:203799 8.17 8:eb7ac254:::benchmark_data_smithi092_104865_object83298:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e233)", 2024-03-11T21:36:04.213 INFO:teuthology.orchestra.run.smithi097.stdout: "initiated_at": "2024-03-11T21:34:50.410131+0000", 2024-03-11T21:36:04.213 INFO:teuthology.orchestra.run.smithi097.stdout: "age": 73.714358945000001, 2024-03-11T21:36:04.213 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.93847114499999995, 2024-03-11T21:36:04.213 INFO:teuthology.orchestra.run.smithi097.stdout: "type_data": { 2024-03-11T21:36:04.213 INFO:teuthology.orchestra.run.smithi097.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-11T21:36:04.213 INFO:teuthology.orchestra.run.smithi097.stdout: "client_info": { 2024-03-11T21:36:04.213 INFO:teuthology.orchestra.run.smithi097.stdout: "client": "client.15150", 2024-03-11T21:36:04.213 INFO:teuthology.orchestra.run.smithi097.stdout: "client_addr": "v1:172.21.15.92:0/436606717", 2024-03-11T21:36:04.213 INFO:teuthology.orchestra.run.smithi097.stdout: "tid": 203799 2024-03-11T21:36:04.213 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:36:04.214 INFO:teuthology.orchestra.run.smithi097.stdout: "events": [ 2024-03-11T21:36:04.214 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:36:04.214 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "initiated", 2024-03-11T21:36:04.214 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:34:50.410131+0000", 2024-03-11T21:36:04.214 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-03-11T21:36:04.214 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:36:04.214 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:36:04.214 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "header_read", 2024-03-11T21:36:04.214 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:34:50.410131+0000", 2024-03-11T21:36:04.214 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-03-11T21:36:04.214 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:36:04.214 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:36:04.214 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "throttled", 2024-03-11T21:36:04.214 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:34:50.410133+0000", 2024-03-11T21:36:04.215 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.5e-06 2024-03-11T21:36:04.215 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:36:04.215 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:36:04.215 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "all_read", 2024-03-11T21:36:04.215 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:34:50.410138+0000", 2024-03-11T21:36:04.215 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 5.1320000000000002e-06 2024-03-11T21:36:04.215 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:36:04.215 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:36:04.215 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "dispatched", 2024-03-11T21:36:04.215 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:34:50.410146+0000", 2024-03-11T21:36:04.215 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 7.9349999999999994e-06 2024-03-11T21:36:04.215 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:36:04.215 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:36:04.215 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "queued_for_pg", 2024-03-11T21:36:04.215 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:34:50.410151+0000", 2024-03-11T21:36:04.216 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 4.8810000000000004e-06 2024-03-11T21:36:04.216 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:36:04.216 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:36:04.216 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "reached_pg", 2024-03-11T21:36:04.216 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:34:50.410211+0000", 2024-03-11T21:36:04.216 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 5.9713999999999998e-05 2024-03-11T21:36:04.216 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:36:04.216 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:36:04.216 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "waiting for peered", 2024-03-11T21:36:04.216 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:34:50.410220+0000", 2024-03-11T21:36:04.216 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 9.1500000000000005e-06 2024-03-11T21:36:04.216 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:36:04.216 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:36:04.216 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "reached_pg", 2024-03-11T21:36:04.216 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:34:51.332969+0000", 2024-03-11T21:36:04.217 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.92274946700000005 2024-03-11T21:36:04.217 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:36:04.217 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:36:04.217 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "started", 2024-03-11T21:36:04.217 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:34:51.333079+0000", 2024-03-11T21:36:04.217 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.00010977800000000001 2024-03-11T21:36:04.217 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:36:04.217 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:36:04.217 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "waiting for subops from 0,6", 2024-03-11T21:36:04.217 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:34:51.333306+0000", 2024-03-11T21:36:04.217 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.000227555 2024-03-11T21:36:04.217 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:36:04.217 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:36:04.217 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "op_commit", 2024-03-11T21:36:04.217 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:34:51.341066+0000", 2024-03-11T21:36:04.218 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.0077596030000000003 2024-03-11T21:36:04.218 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:36:04.218 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:36:04.218 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "sub_op_commit_rec", 2024-03-11T21:36:04.218 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:34:51.343266+0000", 2024-03-11T21:36:04.218 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.0022003779999999998 2024-03-11T21:36:04.218 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:36:04.218 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:36:04.218 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "sub_op_commit_rec", 2024-03-11T21:36:04.218 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:34:51.348507+0000", 2024-03-11T21:36:04.218 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.0052403759999999997 2024-03-11T21:36:04.218 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:36:04.218 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:36:04.218 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "commit_sent", 2024-03-11T21:36:04.218 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:34:51.348556+0000", 2024-03-11T21:36:04.219 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 4.8961000000000001e-05 2024-03-11T21:36:04.219 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:36:04.219 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:36:04.219 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "done", 2024-03-11T21:36:04.219 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:34:51.348603+0000", 2024-03-11T21:36:04.219 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 4.6715000000000003e-05 2024-03-11T21:36:04.219 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-03-11T21:36:04.219 INFO:teuthology.orchestra.run.smithi097.stdout: ] 2024-03-11T21:36:04.219 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-03-11T21:36:04.219 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:36:04.219 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:36:04.219 INFO:teuthology.orchestra.run.smithi097.stdout: "description": "osd_op(client.15150.0:203989 8.10 8:0fc7a0ff:::benchmark_data_smithi092_104865_object83488:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e234)", 2024-03-11T21:36:04.219 INFO:teuthology.orchestra.run.smithi097.stdout: "initiated_at": "2024-03-11T21:34:51.357595+0000", 2024-03-11T21:36:04.219 INFO:teuthology.orchestra.run.smithi097.stdout: "age": 72.766895375000004, 2024-03-11T21:36:04.220 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.96940869299999999, 2024-03-11T21:36:04.220 INFO:teuthology.orchestra.run.smithi097.stdout: "type_data": { 2024-03-11T21:36:04.220 INFO:teuthology.orchestra.run.smithi097.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-11T21:36:04.220 INFO:teuthology.orchestra.run.smithi097.stdout: "client_info": { 2024-03-11T21:36:04.220 INFO:teuthology.orchestra.run.smithi097.stdout: "client": "client.15150", 2024-03-11T21:36:04.220 INFO:teuthology.orchestra.run.smithi097.stdout: "client_addr": "v1:172.21.15.92:0/436606717", 2024-03-11T21:36:04.220 INFO:teuthology.orchestra.run.smithi097.stdout: "tid": 203989 2024-03-11T21:36:04.220 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:36:04.220 INFO:teuthology.orchestra.run.smithi097.stdout: "events": [ 2024-03-11T21:36:04.220 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:36:04.220 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "initiated", 2024-03-11T21:36:04.220 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:34:51.357595+0000", 2024-03-11T21:36:04.220 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-03-11T21:36:04.220 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:36:04.220 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:36:04.220 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "header_read", 2024-03-11T21:36:04.221 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:34:51.357595+0000", 2024-03-11T21:36:04.221 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-03-11T21:36:04.221 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:36:04.221 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:36:04.221 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "throttled", 2024-03-11T21:36:04.221 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:34:51.357595+0000", 2024-03-11T21:36:04.221 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 5.13e-07 2024-03-11T21:36:04.221 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:36:04.221 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:36:04.221 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "all_read", 2024-03-11T21:36:04.221 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:34:51.357598+0000", 2024-03-11T21:36:04.221 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 2.356e-06 2024-03-11T21:36:04.221 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:36:04.221 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:36:04.221 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "dispatched", 2024-03-11T21:36:04.222 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:34:51.357603+0000", 2024-03-11T21:36:04.222 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 5.0119999999999996e-06 2024-03-11T21:36:04.222 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:36:04.222 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:36:04.222 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "queued_for_pg", 2024-03-11T21:36:04.222 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:34:51.357608+0000", 2024-03-11T21:36:04.222 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 4.8640000000000004e-06 2024-03-11T21:36:04.222 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:36:04.222 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:36:04.222 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "reached_pg", 2024-03-11T21:36:04.222 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:34:51.357663+0000", 2024-03-11T21:36:04.222 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 5.5458999999999999e-05 2024-03-11T21:36:04.222 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:36:04.222 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:36:04.223 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "waiting for peered", 2024-03-11T21:36:04.223 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:34:51.357671+0000", 2024-03-11T21:36:04.224 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 8.3469999999999999e-06 2024-03-11T21:36:04.224 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:36:04.224 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:36:04.224 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "reached_pg", 2024-03-11T21:36:04.224 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:34:52.319382+0000", 2024-03-11T21:36:04.224 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.961710858 2024-03-11T21:36:04.224 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:36:04.225 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:36:04.225 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "started", 2024-03-11T21:36:04.225 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:34:52.319469+0000", 2024-03-11T21:36:04.225 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 8.6674000000000004e-05 2024-03-11T21:36:04.225 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:36:04.225 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:36:04.225 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "waiting for subops from 4,5", 2024-03-11T21:36:04.225 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:34:52.319769+0000", 2024-03-11T21:36:04.225 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.00029961499999999998 2024-03-11T21:36:04.225 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:36:04.225 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:36:04.225 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "sub_op_commit_rec", 2024-03-11T21:36:04.225 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:34:52.326866+0000", 2024-03-11T21:36:04.225 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.0070977269999999999 2024-03-11T21:36:04.226 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:36:04.226 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:36:04.226 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "op_commit", 2024-03-11T21:36:04.226 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:34:52.326876+0000", 2024-03-11T21:36:04.226 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 9.6679999999999996e-06 2024-03-11T21:36:04.226 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:36:04.226 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:36:04.226 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "sub_op_commit_rec", 2024-03-11T21:36:04.226 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:34:52.326930+0000", 2024-03-11T21:36:04.226 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 5.4109e-05 2024-03-11T21:36:04.226 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:36:04.226 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:36:04.226 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "commit_sent", 2024-03-11T21:36:04.226 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:34:52.326963+0000", 2024-03-11T21:36:04.227 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 3.2583000000000003e-05 2024-03-11T21:36:04.227 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:36:04.227 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:36:04.227 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "done", 2024-03-11T21:36:04.227 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:34:52.327004+0000", 2024-03-11T21:36:04.227 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 4.0908000000000002e-05 2024-03-11T21:36:04.227 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-03-11T21:36:04.227 INFO:teuthology.orchestra.run.smithi097.stdout: ] 2024-03-11T21:36:04.227 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-03-11T21:36:04.227 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:36:04.227 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:36:04.227 INFO:teuthology.orchestra.run.smithi097.stdout: "description": "osd_op(client.15150.0:203991 8.10 8:08ac658d:::benchmark_data_smithi092_104865_object83490:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e234)", 2024-03-11T21:36:04.227 INFO:teuthology.orchestra.run.smithi097.stdout: "initiated_at": "2024-03-11T21:34:51.360667+0000", 2024-03-11T21:36:04.227 INFO:teuthology.orchestra.run.smithi097.stdout: "age": 72.763823439999996, 2024-03-11T21:36:04.228 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.97040149499999995, 2024-03-11T21:36:04.228 INFO:teuthology.orchestra.run.smithi097.stdout: "type_data": { 2024-03-11T21:36:04.228 INFO:teuthology.orchestra.run.smithi097.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-11T21:36:04.228 INFO:teuthology.orchestra.run.smithi097.stdout: "client_info": { 2024-03-11T21:36:04.228 INFO:teuthology.orchestra.run.smithi097.stdout: "client": "client.15150", 2024-03-11T21:36:04.228 INFO:teuthology.orchestra.run.smithi097.stdout: "client_addr": "v1:172.21.15.92:0/436606717", 2024-03-11T21:36:04.228 INFO:teuthology.orchestra.run.smithi097.stdout: "tid": 203991 2024-03-11T21:36:04.228 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:36:04.228 INFO:teuthology.orchestra.run.smithi097.stdout: "events": [ 2024-03-11T21:36:04.228 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:36:04.228 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "initiated", 2024-03-11T21:36:04.228 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:34:51.360667+0000", 2024-03-11T21:36:04.228 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-03-11T21:36:04.229 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:36:04.229 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:36:04.229 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "header_read", 2024-03-11T21:36:04.229 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:34:51.360667+0000", 2024-03-11T21:36:04.229 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-03-11T21:36:04.229 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:36:04.229 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:36:04.229 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "throttled", 2024-03-11T21:36:04.229 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:34:51.360669+0000", 2024-03-11T21:36:04.229 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.7090000000000001e-06 2024-03-11T21:36:04.229 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:36:04.229 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:36:04.229 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "all_read", 2024-03-11T21:36:04.229 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:34:51.360674+0000", 2024-03-11T21:36:04.230 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 5.113e-06 2024-03-11T21:36:04.230 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:36:04.230 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:36:04.230 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "dispatched", 2024-03-11T21:36:04.230 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:34:51.360685+0000", 2024-03-11T21:36:04.230 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.096e-05 2024-03-11T21:36:04.230 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:36:04.230 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:36:04.230 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "queued_for_pg", 2024-03-11T21:36:04.230 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:34:51.360692+0000", 2024-03-11T21:36:04.230 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 7.6260000000000003e-06 2024-03-11T21:36:04.230 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:36:04.230 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:36:04.230 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "reached_pg", 2024-03-11T21:36:04.230 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:34:51.360742+0000", 2024-03-11T21:36:04.231 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 5.0080999999999997e-05 2024-03-11T21:36:04.231 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:36:04.231 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:36:04.231 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "waiting for peered", 2024-03-11T21:36:04.231 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:34:51.360750+0000", 2024-03-11T21:36:04.231 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 8.0539999999999995e-06 2024-03-11T21:36:04.231 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:36:04.231 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:36:04.231 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "reached_pg", 2024-03-11T21:36:04.231 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:34:52.323279+0000", 2024-03-11T21:36:04.231 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.96252869900000004 2024-03-11T21:36:04.231 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:36:04.231 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:36:04.232 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "started", 2024-03-11T21:36:04.232 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:34:52.323413+0000", 2024-03-11T21:36:04.232 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.00013353599999999999 2024-03-11T21:36:04.232 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:36:04.232 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:36:04.232 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "waiting for subops from 4,5", 2024-03-11T21:36:04.232 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:34:52.323665+0000", 2024-03-11T21:36:04.232 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.00025266999999999998 2024-03-11T21:36:04.232 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:36:04.232 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:36:04.232 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "op_commit", 2024-03-11T21:36:04.232 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:34:52.330839+0000", 2024-03-11T21:36:04.232 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.0071737789999999999 2024-03-11T21:36:04.232 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:36:04.232 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:36:04.233 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "sub_op_commit_rec", 2024-03-11T21:36:04.233 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:34:52.330910+0000", 2024-03-11T21:36:04.233 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 7.1142000000000004e-05 2024-03-11T21:36:04.233 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:36:04.233 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:36:04.233 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "sub_op_commit_rec", 2024-03-11T21:36:04.233 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:34:52.330962+0000", 2024-03-11T21:36:04.233 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 5.1986e-05 2024-03-11T21:36:04.233 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:36:04.233 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:36:04.233 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "commit_sent", 2024-03-11T21:36:04.233 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:34:52.331021+0000", 2024-03-11T21:36:04.233 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 5.8325999999999997e-05 2024-03-11T21:36:04.233 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:36:04.234 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:36:04.234 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "done", 2024-03-11T21:36:04.234 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:34:52.331068+0000", 2024-03-11T21:36:04.234 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 4.7814e-05 2024-03-11T21:36:04.234 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-03-11T21:36:04.234 INFO:teuthology.orchestra.run.smithi097.stdout: ] 2024-03-11T21:36:04.234 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-03-11T21:36:04.234 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:36:04.234 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:36:04.234 INFO:teuthology.orchestra.run.smithi097.stdout: "description": "osd_op(client.15150.0:203993 8.10 8:0fcebf54:::benchmark_data_smithi092_104865_object83492:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e234)", 2024-03-11T21:36:04.234 INFO:teuthology.orchestra.run.smithi097.stdout: "initiated_at": "2024-03-11T21:34:51.361437+0000", 2024-03-11T21:36:04.234 INFO:teuthology.orchestra.run.smithi097.stdout: "age": 72.763053319999997, 2024-03-11T21:36:04.234 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.96990883999999999, 2024-03-11T21:36:04.234 INFO:teuthology.orchestra.run.smithi097.stdout: "type_data": { 2024-03-11T21:36:04.235 INFO:teuthology.orchestra.run.smithi097.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-11T21:36:04.235 INFO:teuthology.orchestra.run.smithi097.stdout: "client_info": { 2024-03-11T21:36:04.235 INFO:teuthology.orchestra.run.smithi097.stdout: "client": "client.15150", 2024-03-11T21:36:04.235 INFO:teuthology.orchestra.run.smithi097.stdout: "client_addr": "v1:172.21.15.92:0/436606717", 2024-03-11T21:36:04.235 INFO:teuthology.orchestra.run.smithi097.stdout: "tid": 203993 2024-03-11T21:36:04.235 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:36:04.235 INFO:teuthology.orchestra.run.smithi097.stdout: "events": [ 2024-03-11T21:36:04.235 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:36:04.235 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "initiated", 2024-03-11T21:36:04.235 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:34:51.361437+0000", 2024-03-11T21:36:04.235 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-03-11T21:36:04.235 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:36:04.235 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:36:04.235 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "header_read", 2024-03-11T21:36:04.235 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:34:51.361437+0000", 2024-03-11T21:36:04.236 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-03-11T21:36:04.236 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:36:04.236 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:36:04.236 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "throttled", 2024-03-11T21:36:04.236 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:34:51.361438+0000", 2024-03-11T21:36:04.236 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 8.7499999999999999e-07 2024-03-11T21:36:04.236 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:36:04.236 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:36:04.236 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "all_read", 2024-03-11T21:36:04.236 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:34:51.361441+0000", 2024-03-11T21:36:04.236 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 3.3639999999999999e-06 2024-03-11T21:36:04.236 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:36:04.236 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:36:04.236 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "dispatched", 2024-03-11T21:36:04.237 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:34:51.361448+0000", 2024-03-11T21:36:04.237 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 7.2529999999999998e-06 2024-03-11T21:36:04.237 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:36:04.237 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:36:04.237 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "queued_for_pg", 2024-03-11T21:36:04.237 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:34:51.361454+0000", 2024-03-11T21:36:04.237 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 5.823e-06 2024-03-11T21:36:04.237 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:36:04.237 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:36:04.237 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "reached_pg", 2024-03-11T21:36:04.237 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:34:51.361495+0000", 2024-03-11T21:36:04.237 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 4.0630999999999998e-05 2024-03-11T21:36:04.237 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:36:04.237 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:36:04.238 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "waiting for peered", 2024-03-11T21:36:04.238 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:34:51.361502+0000", 2024-03-11T21:36:04.238 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 7.029e-06 2024-03-11T21:36:04.238 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:36:04.238 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:36:04.238 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "reached_pg", 2024-03-11T21:36:04.238 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:34:52.325388+0000", 2024-03-11T21:36:04.238 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.96388603699999997 2024-03-11T21:36:04.238 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:36:04.238 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:36:04.238 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "started", 2024-03-11T21:36:04.238 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:34:52.325491+0000", 2024-03-11T21:36:04.238 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.00010323900000000001 2024-03-11T21:36:04.238 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:36:04.239 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:36:04.239 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "waiting for subops from 4,5", 2024-03-11T21:36:04.239 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:34:52.325704+0000", 2024-03-11T21:36:04.239 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.00021247799999999999 2024-03-11T21:36:04.239 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:36:04.239 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:36:04.239 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "sub_op_commit_rec", 2024-03-11T21:36:04.239 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:34:52.331132+0000", 2024-03-11T21:36:04.239 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.0054278909999999998 2024-03-11T21:36:04.239 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:36:04.239 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:36:04.239 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "op_commit", 2024-03-11T21:36:04.239 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:34:52.331149+0000", 2024-03-11T21:36:04.239 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.7629999999999999e-05 2024-03-11T21:36:04.239 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:36:04.240 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:36:04.240 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "sub_op_commit_rec", 2024-03-11T21:36:04.240 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:34:52.331254+0000", 2024-03-11T21:36:04.240 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.000105239 2024-03-11T21:36:04.240 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:36:04.240 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:36:04.240 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "commit_sent", 2024-03-11T21:36:04.240 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:34:52.331307+0000", 2024-03-11T21:36:04.240 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 5.2620000000000001e-05 2024-03-11T21:36:04.240 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:36:04.240 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:36:04.240 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "done", 2024-03-11T21:36:04.240 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:34:52.331346+0000", 2024-03-11T21:36:04.240 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 3.8730999999999999e-05 2024-03-11T21:36:04.240 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-03-11T21:36:04.241 INFO:teuthology.orchestra.run.smithi097.stdout: ] 2024-03-11T21:36:04.241 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-03-11T21:36:04.241 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:36:04.241 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:36:04.241 INFO:teuthology.orchestra.run.smithi097.stdout: "description": "osd_op(client.15150.0:204018 8.10 8:0d60ca10:::benchmark_data_smithi092_104865_object83517:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e234)", 2024-03-11T21:36:04.241 INFO:teuthology.orchestra.run.smithi097.stdout: "initiated_at": "2024-03-11T21:34:51.378792+0000", 2024-03-11T21:36:04.241 INFO:teuthology.orchestra.run.smithi097.stdout: "age": 72.745697995, 2024-03-11T21:36:04.241 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.95687534699999999, 2024-03-11T21:36:04.241 INFO:teuthology.orchestra.run.smithi097.stdout: "type_data": { 2024-03-11T21:36:04.241 INFO:teuthology.orchestra.run.smithi097.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-11T21:36:04.241 INFO:teuthology.orchestra.run.smithi097.stdout: "client_info": { 2024-03-11T21:36:04.241 INFO:teuthology.orchestra.run.smithi097.stdout: "client": "client.15150", 2024-03-11T21:36:04.241 INFO:teuthology.orchestra.run.smithi097.stdout: "client_addr": "v1:172.21.15.92:0/436606717", 2024-03-11T21:36:04.241 INFO:teuthology.orchestra.run.smithi097.stdout: "tid": 204018 2024-03-11T21:36:04.242 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:36:04.242 INFO:teuthology.orchestra.run.smithi097.stdout: "events": [ 2024-03-11T21:36:04.242 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:36:04.242 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "initiated", 2024-03-11T21:36:04.242 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:34:51.378792+0000", 2024-03-11T21:36:04.242 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-03-11T21:36:04.242 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:36:04.242 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:36:04.242 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "header_read", 2024-03-11T21:36:04.242 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:34:51.378792+0000", 2024-03-11T21:36:04.242 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-03-11T21:36:04.242 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:36:04.242 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:36:04.242 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "throttled", 2024-03-11T21:36:04.243 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:34:51.378794+0000", 2024-03-11T21:36:04.243 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.64e-06 2024-03-11T21:36:04.243 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:36:04.243 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:36:04.243 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "all_read", 2024-03-11T21:36:04.243 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:34:51.378800+0000", 2024-03-11T21:36:04.243 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 5.8690000000000001e-06 2024-03-11T21:36:04.243 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:36:04.243 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:36:04.243 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "dispatched", 2024-03-11T21:36:04.243 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:34:51.378808+0000", 2024-03-11T21:36:04.243 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 8.3399999999999998e-06 2024-03-11T21:36:04.243 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:36:04.243 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:36:04.243 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "queued_for_pg", 2024-03-11T21:36:04.244 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:34:51.378815+0000", 2024-03-11T21:36:04.244 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 6.6830000000000002e-06 2024-03-11T21:36:04.244 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:36:04.244 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:36:04.244 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "reached_pg", 2024-03-11T21:36:04.244 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:34:51.379938+0000", 2024-03-11T21:36:04.244 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.001123303 2024-03-11T21:36:04.244 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:36:04.244 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:36:04.244 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "waiting for peered", 2024-03-11T21:36:04.244 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:34:51.379946+0000", 2024-03-11T21:36:04.244 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 7.7370000000000006e-06 2024-03-11T21:36:04.244 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:36:04.244 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:36:04.244 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "reached_pg", 2024-03-11T21:36:04.245 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:34:52.327042+0000", 2024-03-11T21:36:04.245 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.94709584999999996 2024-03-11T21:36:04.245 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:36:04.245 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:36:04.245 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "started", 2024-03-11T21:36:04.245 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:34:52.327137+0000", 2024-03-11T21:36:04.245 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 9.5537000000000004e-05 2024-03-11T21:36:04.245 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:36:04.245 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:36:04.245 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "waiting for subops from 4,5", 2024-03-11T21:36:04.245 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:34:52.327320+0000", 2024-03-11T21:36:04.245 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.00018275799999999999 2024-03-11T21:36:04.245 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:36:04.245 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:36:04.245 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "op_commit", 2024-03-11T21:36:04.246 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:34:52.331156+0000", 2024-03-11T21:36:04.246 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.0038356190000000002 2024-03-11T21:36:04.246 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:36:04.246 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:36:04.246 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "sub_op_commit_rec", 2024-03-11T21:36:04.246 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:34:52.331404+0000", 2024-03-11T21:36:04.246 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.00024812 2024-03-11T21:36:04.246 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:36:04.246 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:36:04.246 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "sub_op_commit_rec", 2024-03-11T21:36:04.246 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:34:52.335569+0000", 2024-03-11T21:36:04.246 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.0041654910000000003 2024-03-11T21:36:04.246 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:36:04.246 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:36:04.247 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "commit_sent", 2024-03-11T21:36:04.247 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:34:52.335621+0000", 2024-03-11T21:36:04.247 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 5.1520000000000001e-05 2024-03-11T21:36:04.247 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:36:04.247 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:36:04.247 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "done", 2024-03-11T21:36:04.247 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:34:52.335668+0000", 2024-03-11T21:36:04.247 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 4.6879999999999998e-05 2024-03-11T21:36:04.247 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-03-11T21:36:04.247 INFO:teuthology.orchestra.run.smithi097.stdout: ] 2024-03-11T21:36:04.247 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-03-11T21:36:04.247 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-03-11T21:36:04.247 INFO:teuthology.orchestra.run.smithi097.stdout: ] 2024-03-11T21:36:04.247 INFO:teuthology.orchestra.run.smithi097.stdout:} 2024-03-11T21:36:04.248 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:36:04.248 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:36:04.248 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:36:04.349 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:36:04.388 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:36:04.389 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:36:04.389 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:36:04.490 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:36:04.518 INFO:tasks.radosbench.radosbench.0.smithi092.stdout: 7 16 21706 21690 193.645 155.062 0.00532814 0.00515908 2024-03-11T21:36:04.526 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:36:04.527 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:36:04.527 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:36:04.628 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:36:04.656 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:36:04.657 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:36:04.657 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:36:04.758 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:36:04.793 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:36:04.794 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:36:04.795 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:36:04.859 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:36:04 smithi092 ceph-mon[29572]: pgmap v474: 148 pgs: 30 active+undersized+degraded, 32 active+undersized, 86 active+clean; 768 MiB data, 4.7 GiB used, 711 GiB / 715 GiB avail; 1.2 KiB/s rd, 96 MiB/s wr, 1.53k op/s; 6974/37407 objects degraded (18.644%) 2024-03-11T21:36:04.860 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:36:04 smithi092 ceph-mon[29572]: 5.15 scrub starts 2024-03-11T21:36:04.860 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:36:04 smithi092 ceph-mon[29572]: 5.15 scrub ok 2024-03-11T21:36:04.860 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:36:04 smithi092 ceph-mon[37563]: pgmap v474: 148 pgs: 30 active+undersized+degraded, 32 active+undersized, 86 active+clean; 768 MiB data, 4.7 GiB used, 711 GiB / 715 GiB avail; 1.2 KiB/s rd, 96 MiB/s wr, 1.53k op/s; 6974/37407 objects degraded (18.644%) 2024-03-11T21:36:04.860 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:36:04 smithi092 ceph-mon[37563]: 5.15 scrub starts 2024-03-11T21:36:04.860 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:36:04 smithi092 ceph-mon[37563]: 5.15 scrub ok 2024-03-11T21:36:04.896 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:36:04.930 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:36:04.931 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:36:04.931 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:36:05.032 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:36:05.074 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:36:05.074 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:36:05.074 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:36:05.075 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:36:04 smithi097 ceph-mon[33317]: pgmap v474: 148 pgs: 30 active+undersized+degraded, 32 active+undersized, 86 active+clean; 768 MiB data, 4.7 GiB used, 711 GiB / 715 GiB avail; 1.2 KiB/s rd, 96 MiB/s wr, 1.53k op/s; 6974/37407 objects degraded (18.644%) 2024-03-11T21:36:05.075 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:36:04 smithi097 ceph-mon[33317]: 5.15 scrub starts 2024-03-11T21:36:05.075 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:36:04 smithi097 ceph-mon[33317]: 5.15 scrub ok 2024-03-11T21:36:05.091 DEBUG:teuthology.orchestra.run.smithi092:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid 00db0016-dfed-11ee-95c8-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-03-11T21:36:05.175 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:36:05.210 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:36:05.211 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:36:05.211 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:36:05.278 DEBUG:teuthology.orchestra.run.smithi092:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid 00db0016-dfed-11ee-95c8-87774f69a715 -- ceph daemon osd.0 dump_ops_in_flight 2024-03-11T21:36:05.312 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:36:05.353 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:36:05.354 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:36:05.354 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:36:05.455 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:36:05.493 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:36:05.494 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:36:05.494 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:36:05.518 INFO:tasks.radosbench.radosbench.0.smithi092.stdout: 8 16 24169 24153 188.68 153.938 0.00287399 0.00529374 2024-03-11T21:36:05.595 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:36:05.633 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:36:05.634 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:36:05.634 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:36:05.735 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:36:05.772 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:36:05.772 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:36:05.773 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:36:05.859 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:36:05 smithi092 ceph-mon[29572]: 6.17 scrub starts 2024-03-11T21:36:05.860 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:36:05 smithi092 ceph-mon[29572]: 6.17 scrub ok 2024-03-11T21:36:05.860 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:36:05 smithi092 ceph-mon[37563]: 6.17 scrub starts 2024-03-11T21:36:05.860 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:36:05 smithi092 ceph-mon[37563]: 6.17 scrub ok 2024-03-11T21:36:05.873 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:36:05.908 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:36:05.909 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:36:05.909 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:36:06.010 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:36:06.046 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:36:06.046 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:36:06.047 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:36:06.075 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:36:05 smithi097 ceph-mon[33317]: 6.17 scrub starts 2024-03-11T21:36:06.075 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:36:05 smithi097 ceph-mon[33317]: 6.17 scrub ok 2024-03-11T21:36:06.148 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:36:06.185 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:36:06.185 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:36:06.185 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:36:06.286 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:36:06.324 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:36:06.324 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:36:06.324 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:36:06.425 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:36:06.460 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:36:06.461 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:36:06.461 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:36:06.518 INFO:tasks.radosbench.radosbench.0.smithi092.stdout: 9 16 26904 26888 186.707 170.938 0.0038034 0.00535057 2024-03-11T21:36:06.552 INFO:teuthology.orchestra.run.smithi092.stderr:Creating an OSD daemon form without an OSD FSID value 2024-03-11T21:36:06.563 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:36:06.598 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:36:06.599 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:36:06.599 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:36:06.700 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:36:06.737 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:36:06.738 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:36:06.738 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:36:06.839 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:36:06.868 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:36:06.868 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:36:06.869 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:36:06.969 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:36:07.003 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:36:07.004 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:36:07.005 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:36:07.046 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:36:06 smithi092 ceph-mon[29572]: pgmap v476: 148 pgs: 30 active+undersized+degraded, 32 active+undersized, 86 active+clean; 1009 MiB data, 4.9 GiB used, 710 GiB / 715 GiB avail; 742 B/s rd, 146 MiB/s wr, 2.34k op/s; 7950/48993 objects degraded (16.227%) 2024-03-11T21:36:07.046 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:36:06 smithi092 ceph-mon[29572]: Health check update: Degraded data redundancy: 7950/48993 objects degraded (16.227%), 30 pgs degraded (PG_DEGRADED) 2024-03-11T21:36:07.046 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:36:06 smithi092 ceph-mon[37563]: pgmap v476: 148 pgs: 30 active+undersized+degraded, 32 active+undersized, 86 active+clean; 1009 MiB data, 4.9 GiB used, 710 GiB / 715 GiB avail; 742 B/s rd, 146 MiB/s wr, 2.34k op/s; 7950/48993 objects degraded (16.227%) 2024-03-11T21:36:07.047 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:36:06 smithi092 ceph-mon[37563]: Health check update: Degraded data redundancy: 7950/48993 objects degraded (16.227%), 30 pgs degraded (PG_DEGRADED) 2024-03-11T21:36:07.075 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:36:06 smithi097 ceph-mon[33317]: pgmap v476: 148 pgs: 30 active+undersized+degraded, 32 active+undersized, 86 active+clean; 1009 MiB data, 4.9 GiB used, 710 GiB / 715 GiB avail; 742 B/s rd, 146 MiB/s wr, 2.34k op/s; 7950/48993 objects degraded (16.227%) 2024-03-11T21:36:07.075 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:36:06 smithi097 ceph-mon[33317]: Health check update: Degraded data redundancy: 7950/48993 objects degraded (16.227%), 30 pgs degraded (PG_DEGRADED) 2024-03-11T21:36:07.106 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:36:07.149 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:36:07.150 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:36:07.151 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:36:07.251 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:36:07.286 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:36:07.287 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:36:07.287 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:36:07.388 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:36:07.416 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:36:07.416 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:36:07.417 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:36:07.518 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:36:07.518 INFO:tasks.radosbench.radosbench.0.smithi092.stdout: 10 16 29269 29253 182.816 147.812 0.00643192 0.00546507 2024-03-11T21:36:07.552 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:36:07.553 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:36:07.553 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:36:07.611 INFO:teuthology.orchestra.run.smithi092.stderr:Inferring config /var/lib/ceph/00db0016-dfed-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T21:36:07.613 INFO:teuthology.orchestra.run.smithi092.stderr:Inferring config /var/lib/ceph/00db0016-dfed-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T21:36:07.653 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:36:07.684 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:36:07.684 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:36:07.685 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:36:07.786 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:36:07.812 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:36:07.812 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:36:07.812 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:36:07.860 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:36:07 smithi092 ceph-mon[29572]: 2.15 scrub starts 2024-03-11T21:36:07.860 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:36:07 smithi092 ceph-mon[29572]: 2.15 scrub ok 2024-03-11T21:36:07.860 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:36:07 smithi092 ceph-mon[29572]: 6.8 scrub starts 2024-03-11T21:36:07.860 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:36:07 smithi092 ceph-mon[29572]: 6.8 scrub ok 2024-03-11T21:36:07.860 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:36:07 smithi092 ceph-mon[29572]: from='mgr.14607 ' entity='mgr.y' 2024-03-11T21:36:07.860 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:36:07 smithi092 ceph-mon[29572]: from='mgr.14607 v1:172.21.15.92:0/1702717597' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T21:36:07.860 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:36:07 smithi092 ceph-mon[29572]: from='client.? v1:172.21.15.92:0/1891486327' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-11T21:36:07.860 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:36:07 smithi092 ceph-mon[37563]: 2.15 scrub starts 2024-03-11T21:36:07.860 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:36:07 smithi092 ceph-mon[37563]: 2.15 scrub ok 2024-03-11T21:36:07.860 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:36:07 smithi092 ceph-mon[37563]: 6.8 scrub starts 2024-03-11T21:36:07.861 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:36:07 smithi092 ceph-mon[37563]: 6.8 scrub ok 2024-03-11T21:36:07.861 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:36:07 smithi092 ceph-mon[37563]: from='mgr.14607 ' entity='mgr.y' 2024-03-11T21:36:07.861 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:36:07 smithi092 ceph-mon[37563]: from='mgr.14607 v1:172.21.15.92:0/1702717597' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T21:36:07.861 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:36:07 smithi092 ceph-mon[37563]: from='client.? v1:172.21.15.92:0/1891486327' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-11T21:36:07.867 INFO:teuthology.orchestra.run.smithi092.stdout:get_pg_num_history pg_num_history pg_num_history(e251 pg_nums {1={20=1},2={66=32},3={66=3},4={68=32},5={70=32},6={72=32},8={157=16,181=26}} deleted_pools ) 2024-03-11T21:36:07.867 INFO:teuthology.orchestra.run.smithi092.stdout:Importing pgid 8.8 2024-03-11T21:36:07.881 INFO:teuthology.orchestra.run.smithi092.stdout:write_pg epoch 250 info 8.8( v 250'10472 (180'5587,250'10472] local-lis/les=236/237 n=0 ec=157/157 lis/c=236/236 les/c/f=237/237/0 sis=236) 2024-03-11T21:36:07.914 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:36:07.928 INFO:teuthology.orchestra.run.smithi092.stdout:Import successful 2024-03-11T21:36:07.952 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:36:07.952 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:36:07.953 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:36:08.054 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:36:08.075 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:36:07 smithi097 ceph-mon[33317]: 2.15 scrub starts 2024-03-11T21:36:08.075 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:36:07 smithi097 ceph-mon[33317]: 2.15 scrub ok 2024-03-11T21:36:08.075 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:36:07 smithi097 ceph-mon[33317]: 6.8 scrub starts 2024-03-11T21:36:08.075 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:36:07 smithi097 ceph-mon[33317]: 6.8 scrub ok 2024-03-11T21:36:08.075 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:36:07 smithi097 ceph-mon[33317]: from='mgr.14607 ' entity='mgr.y' 2024-03-11T21:36:08.076 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:36:07 smithi097 ceph-mon[33317]: from='mgr.14607 v1:172.21.15.92:0/1702717597' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T21:36:08.076 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:36:07 smithi097 ceph-mon[33317]: from='client.? v1:172.21.15.92:0/1891486327' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-11T21:36:08.079 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:36:08.080 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:36:08.080 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:36:08.126 INFO:journalctl@ceph.mgr.y.smithi092.stdout:Mar 11 21:36:07 smithi092 ceph-00db0016-dfed-11ee-95c8-87774f69a715-mgr-y[29801]: ::ffff:172.21.15.97 - - [11/Mar/2024:21:36:07] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-11T21:36:08.181 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:36:08.199 INFO:teuthology.orchestra.run.smithi092.stderr:nodeep-scrub is unset 2024-03-11T21:36:08.220 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:36:08.220 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:36:08.221 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:36:08.322 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:36:08.349 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:36:08.350 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:36:08.351 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:36:08.451 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:36:08.484 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:36:08.484 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:36:08.485 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:36:08.518 INFO:tasks.radosbench.radosbench.0.smithi092.stdout: 11 16 31354 31338 178.042 130.312 0.00543176 0.00561158 2024-03-11T21:36:08.586 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:36:08.613 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:36:08.613 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:36:08.614 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:36:08.714 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:36:08.747 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:36:08.747 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:36:08.748 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:36:08.849 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:36:08.883 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:36:08.883 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:36:08.884 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:36:08.985 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:36:09.022 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:36:09.026 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:36:09.026 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:36:09.075 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:36:08 smithi097 ceph-mon[33317]: pgmap v477: 148 pgs: 30 active+undersized+degraded, 32 active+undersized, 86 active+clean; 1.4 GiB data, 6.0 GiB used, 709 GiB / 715 GiB avail; 639 B/s rd, 171 MiB/s wr, 2.73k op/s; 11215/68832 objects degraded (16.293%) 2024-03-11T21:36:09.076 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:36:08 smithi097 ceph-mon[33317]: 5.7 scrub starts 2024-03-11T21:36:09.076 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:36:08 smithi097 ceph-mon[33317]: 5.7 scrub ok 2024-03-11T21:36:09.076 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:36:08 smithi097 ceph-mon[33317]: 5.1d scrub starts 2024-03-11T21:36:09.076 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:36:08 smithi097 ceph-mon[33317]: 5.1d scrub ok 2024-03-11T21:36:09.076 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:36:08 smithi097 ceph-mon[33317]: 5.e scrub starts 2024-03-11T21:36:09.076 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:36:08 smithi097 ceph-mon[33317]: 5.e scrub ok 2024-03-11T21:36:09.076 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:36:08 smithi097 ceph-mon[33317]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-03-11T21:36:09.076 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:36:08 smithi097 ceph-mon[33317]: from='client.? v1:172.21.15.92:0/1891486327' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-03-11T21:36:09.076 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:36:08 smithi097 ceph-mon[33317]: osdmap e269: 8 total, 7 up, 8 in 2024-03-11T21:36:09.076 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:36:08 smithi097 ceph-mon[33317]: osdmap e270: 8 total, 7 up, 8 in 2024-03-11T21:36:09.110 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:36:08 smithi092 ceph-mon[29572]: pgmap v477: 148 pgs: 30 active+undersized+degraded, 32 active+undersized, 86 active+clean; 1.4 GiB data, 6.0 GiB used, 709 GiB / 715 GiB avail; 639 B/s rd, 171 MiB/s wr, 2.73k op/s; 11215/68832 objects degraded (16.293%) 2024-03-11T21:36:09.110 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:36:08 smithi092 ceph-mon[29572]: 5.7 scrub starts 2024-03-11T21:36:09.110 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:36:08 smithi092 ceph-mon[29572]: 5.7 scrub ok 2024-03-11T21:36:09.110 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:36:08 smithi092 ceph-mon[29572]: 5.1d scrub starts 2024-03-11T21:36:09.110 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:36:08 smithi092 ceph-mon[29572]: 5.1d scrub ok 2024-03-11T21:36:09.110 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:36:08 smithi092 ceph-mon[29572]: 5.e scrub starts 2024-03-11T21:36:09.110 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:36:08 smithi092 ceph-mon[29572]: 5.e scrub ok 2024-03-11T21:36:09.111 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:36:08 smithi092 ceph-mon[29572]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-03-11T21:36:09.111 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:36:08 smithi092 ceph-mon[29572]: from='client.? v1:172.21.15.92:0/1891486327' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-03-11T21:36:09.111 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:36:08 smithi092 ceph-mon[29572]: osdmap e269: 8 total, 7 up, 8 in 2024-03-11T21:36:09.111 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:36:08 smithi092 ceph-mon[29572]: osdmap e270: 8 total, 7 up, 8 in 2024-03-11T21:36:09.111 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:36:08 smithi092 ceph-mon[37563]: pgmap v477: 148 pgs: 30 active+undersized+degraded, 32 active+undersized, 86 active+clean; 1.4 GiB data, 6.0 GiB used, 709 GiB / 715 GiB avail; 639 B/s rd, 171 MiB/s wr, 2.73k op/s; 11215/68832 objects degraded (16.293%) 2024-03-11T21:36:09.111 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:36:08 smithi092 ceph-mon[37563]: 5.7 scrub starts 2024-03-11T21:36:09.111 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:36:08 smithi092 ceph-mon[37563]: 5.7 scrub ok 2024-03-11T21:36:09.111 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:36:08 smithi092 ceph-mon[37563]: 5.1d scrub starts 2024-03-11T21:36:09.111 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:36:08 smithi092 ceph-mon[37563]: 5.1d scrub ok 2024-03-11T21:36:09.112 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:36:08 smithi092 ceph-mon[37563]: 5.e scrub starts 2024-03-11T21:36:09.112 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:36:08 smithi092 ceph-mon[37563]: 5.e scrub ok 2024-03-11T21:36:09.112 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:36:08 smithi092 ceph-mon[37563]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-03-11T21:36:09.112 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:36:08 smithi092 ceph-mon[37563]: from='client.? v1:172.21.15.92:0/1891486327' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-03-11T21:36:09.112 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:36:08 smithi092 ceph-mon[37563]: osdmap e269: 8 total, 7 up, 8 in 2024-03-11T21:36:09.112 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:36:08 smithi092 ceph-mon[37563]: osdmap e270: 8 total, 7 up, 8 in 2024-03-11T21:36:09.127 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:36:09.163 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:36:09.163 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:36:09.164 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:36:09.265 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:36:09.301 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:36:09.302 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:36:09.302 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:36:09.403 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:36:09.431 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:36:09.432 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:36:09.432 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:36:09.519 INFO:tasks.radosbench.radosbench.0.smithi092.stdout: 12 16 33395 33379 173.834 127.562 0.00296661 0.00570055 2024-03-11T21:36:09.534 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:36:09.568 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:36:09.569 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:36:09.569 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:36:09.670 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:36:09.707 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:36:09.707 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:36:09.707 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:36:09.808 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:36:09.844 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:36:09.844 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:36:09.845 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:36:09.945 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:36:09.977 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:36:09.978 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:36:09.979 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:36:10.075 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:36:09 smithi097 ceph-mon[33317]: 5.f scrub starts 2024-03-11T21:36:10.076 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:36:09 smithi097 ceph-mon[33317]: 5.f scrub ok 2024-03-11T21:36:10.076 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:36:09 smithi097 ceph-mon[33317]: 5.1f scrub starts 2024-03-11T21:36:10.076 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:36:09 smithi097 ceph-mon[33317]: 5.1f scrub ok 2024-03-11T21:36:10.079 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:36:10.109 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:36:09 smithi092 ceph-mon[29572]: 5.f scrub starts 2024-03-11T21:36:10.109 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:36:09 smithi092 ceph-mon[29572]: 5.f scrub ok 2024-03-11T21:36:10.109 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:36:09 smithi092 ceph-mon[29572]: 5.1f scrub starts 2024-03-11T21:36:10.110 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:36:09 smithi092 ceph-mon[29572]: 5.1f scrub ok 2024-03-11T21:36:10.110 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:36:09 smithi092 ceph-mon[37563]: 5.f scrub starts 2024-03-11T21:36:10.110 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:36:09 smithi092 ceph-mon[37563]: 5.f scrub ok 2024-03-11T21:36:10.110 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:36:09 smithi092 ceph-mon[37563]: 5.1f scrub starts 2024-03-11T21:36:10.110 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:36:09 smithi092 ceph-mon[37563]: 5.1f scrub ok 2024-03-11T21:36:10.118 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:36:10.118 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:36:10.119 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:36:10.220 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:36:10.248 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:36:10.249 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:36:10.249 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:36:10.350 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:36:10.384 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:36:10.385 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:36:10.385 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:36:10.486 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:36:10.519 INFO:tasks.radosbench.radosbench.0.smithi092.stdout: 13 16 35450 35434 170.341 128.438 0.00782171 0.0058657 2024-03-11T21:36:10.523 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:36:10.523 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:36:10.524 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:36:10.625 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:36:10.662 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:36:10.662 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:36:10.663 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:36:10.763 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:36:10.802 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:36:10.803 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:36:10.803 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:36:10.904 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:36:10.927 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:36:10.928 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:36:10.928 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:36:11.029 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:36:11.066 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:36:11.067 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:36:11.067 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:36:11.075 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:36:10 smithi097 ceph-mon[33317]: pgmap v480: 148 pgs: 30 active+undersized+degraded, 32 active+undersized, 86 active+clean; 1.8 GiB data, 7.0 GiB used, 708 GiB / 715 GiB avail; 852 B/s rd, 176 MiB/s wr, 2.82k op/s; 14517/88197 objects degraded (16.460%) 2024-03-11T21:36:11.075 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:36:10 smithi097 ceph-mon[33317]: 4.18 scrub starts 2024-03-11T21:36:11.076 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:36:10 smithi097 ceph-mon[33317]: 4.18 scrub ok 2024-03-11T21:36:11.109 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:36:10 smithi092 ceph-mon[29572]: pgmap v480: 148 pgs: 30 active+undersized+degraded, 32 active+undersized, 86 active+clean; 1.8 GiB data, 7.0 GiB used, 708 GiB / 715 GiB avail; 852 B/s rd, 176 MiB/s wr, 2.82k op/s; 14517/88197 objects degraded (16.460%) 2024-03-11T21:36:11.110 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:36:10 smithi092 ceph-mon[29572]: 4.18 scrub starts 2024-03-11T21:36:11.110 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:36:10 smithi092 ceph-mon[29572]: 4.18 scrub ok 2024-03-11T21:36:11.110 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:36:10 smithi092 ceph-mon[37563]: pgmap v480: 148 pgs: 30 active+undersized+degraded, 32 active+undersized, 86 active+clean; 1.8 GiB data, 7.0 GiB used, 708 GiB / 715 GiB avail; 852 B/s rd, 176 MiB/s wr, 2.82k op/s; 14517/88197 objects degraded (16.460%) 2024-03-11T21:36:11.110 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:36:10 smithi092 ceph-mon[37563]: 4.18 scrub starts 2024-03-11T21:36:11.111 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:36:10 smithi092 ceph-mon[37563]: 4.18 scrub ok 2024-03-11T21:36:11.168 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:36:11.205 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:36:11.206 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:36:11.206 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:36:11.307 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:36:11.344 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:36:11.344 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:36:11.345 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:36:11.445 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:36:11.481 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:36:11.481 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:36:11.482 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:36:11.518 INFO:tasks.radosbench.radosbench.0.smithi092.stdout: 14 15 37415 37400 166.95 122.875 0.00412456 0.00598462 2024-03-11T21:36:11.583 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:36:11.619 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:36:11.619 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:36:11.620 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:36:11.721 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:36:11.756 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:36:11.757 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:36:11.757 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:36:11.858 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:36:11.901 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:36:11.902 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:36:11.902 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:36:12.003 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:36:12.052 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:36:12.053 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:36:12.054 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:36:12.076 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:36:11 smithi097 ceph-mon[33317]: 2.18 scrub starts 2024-03-11T21:36:12.076 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:36:11 smithi097 ceph-mon[33317]: 2.18 scrub ok 2024-03-11T21:36:12.076 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:36:11 smithi097 ceph-mon[33317]: 5.c scrub starts 2024-03-11T21:36:12.076 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:36:11 smithi097 ceph-mon[33317]: 5.c scrub ok 2024-03-11T21:36:12.076 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:36:11 smithi097 ceph-mon[33317]: 6.e scrub starts 2024-03-11T21:36:12.076 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:36:11 smithi097 ceph-mon[33317]: 6.e scrub ok 2024-03-11T21:36:12.110 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:36:11 smithi092 ceph-mon[29572]: 2.18 scrub starts 2024-03-11T21:36:12.110 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:36:11 smithi092 ceph-mon[29572]: 2.18 scrub ok 2024-03-11T21:36:12.110 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:36:11 smithi092 ceph-mon[29572]: 5.c scrub starts 2024-03-11T21:36:12.110 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:36:11 smithi092 ceph-mon[29572]: 5.c scrub ok 2024-03-11T21:36:12.111 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:36:11 smithi092 ceph-mon[29572]: 6.e scrub starts 2024-03-11T21:36:12.111 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:36:11 smithi092 ceph-mon[29572]: 6.e scrub ok 2024-03-11T21:36:12.111 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:36:11 smithi092 ceph-mon[37563]: 2.18 scrub starts 2024-03-11T21:36:12.111 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:36:11 smithi092 ceph-mon[37563]: 2.18 scrub ok 2024-03-11T21:36:12.111 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:36:11 smithi092 ceph-mon[37563]: 5.c scrub starts 2024-03-11T21:36:12.111 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:36:11 smithi092 ceph-mon[37563]: 5.c scrub ok 2024-03-11T21:36:12.111 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:36:11 smithi092 ceph-mon[37563]: 6.e scrub starts 2024-03-11T21:36:12.111 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:36:11 smithi092 ceph-mon[37563]: 6.e scrub ok 2024-03-11T21:36:12.155 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:36:12.193 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:36:12.193 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:36:12.193 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:36:12.225 DEBUG:teuthology.orchestra.run.smithi092:> sudo rm -f /var/log/ceph/00db0016-dfed-11ee-95c8-87774f69a715/exp.8.8.3 2024-03-11T21:36:12.295 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:36:12.384 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:36:12.385 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:36:12.385 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:36:12.486 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:36:12.520 INFO:tasks.radosbench.radosbench.0.smithi092.stdout: 15 15 39803 39788 165.755 149.25 0.00476358 0.00602804 2024-03-11T21:36:12.523 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:36:12.524 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:36:12.524 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:36:12.625 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:36:12.659 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:36:12.660 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:36:12.660 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:36:12.761 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:36:12.788 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:36:12 smithi092 ceph-mon[29572]: pgmap v481: 148 pgs: 30 active+undersized+degraded, 32 active+undersized, 86 active+clean; 1.8 GiB data, 7.0 GiB used, 708 GiB / 715 GiB avail; 648 B/s rd, 134 MiB/s wr, 2.15k op/s; 14517/88197 objects degraded (16.460%) 2024-03-11T21:36:12.794 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:36:12.795 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:36:12.795 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:36:12.825 INFO:journalctl@ceph.iscsi.iscsi.a.smithi097.stdout:Mar 11 21:36:12 smithi097 ceph-00db0016-dfed-11ee-95c8-87774f69a715-iscsi-iscsi-a[60943]: debug there is no tcmu-runner data available 2024-03-11T21:36:12.826 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:36:12 smithi097 ceph-mon[33317]: pgmap v481: 148 pgs: 30 active+undersized+degraded, 32 active+undersized, 86 active+clean; 1.8 GiB data, 7.0 GiB used, 708 GiB / 715 GiB avail; 648 B/s rd, 134 MiB/s wr, 2.15k op/s; 14517/88197 objects degraded (16.460%) 2024-03-11T21:36:12.826 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:36:12 smithi097 ceph-mon[33317]: 6.1c scrub starts 2024-03-11T21:36:12.826 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:36:12 smithi097 ceph-mon[33317]: 6.1c scrub ok 2024-03-11T21:36:12.895 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:36:12.929 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:36:12.930 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:36:12.930 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:36:13.031 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:36:13.067 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:36:13.067 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:36:13.068 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:36:13.089 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:36:12 smithi092 ceph-mon[29572]: 6.1c scrub starts 2024-03-11T21:36:13.089 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:36:12 smithi092 ceph-mon[29572]: 6.1c scrub ok 2024-03-11T21:36:13.089 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:36:12 smithi092 ceph-mon[37563]: pgmap v481: 148 pgs: 30 active+undersized+degraded, 32 active+undersized, 86 active+clean; 1.8 GiB data, 7.0 GiB used, 708 GiB / 715 GiB avail; 648 B/s rd, 134 MiB/s wr, 2.15k op/s; 14517/88197 objects degraded (16.460%) 2024-03-11T21:36:13.089 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:36:12 smithi092 ceph-mon[37563]: 6.1c scrub starts 2024-03-11T21:36:13.089 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:36:12 smithi092 ceph-mon[37563]: 6.1c scrub ok 2024-03-11T21:36:13.169 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:36:13.203 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:36:13.204 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:36:13.204 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:36:13.305 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:36:13.340 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:36:13.341 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:36:13.341 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:36:13.442 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:36:13.477 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:36:13.478 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:36:13.478 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:36:13.520 INFO:tasks.radosbench.radosbench.0.smithi092.stdout: 16 16 42045 42029 164.149 140.062 0.00715046 0.00608691 2024-03-11T21:36:13.579 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:36:13.615 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:36:13.616 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:36:13.616 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:36:13.717 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:36:13.750 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:36:13.750 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:36:13.751 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:36:13.851 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:36:13.887 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:36:13.887 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:36:13.887 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:36:13.990 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:36:14.025 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:36:14.025 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:36:14.026 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:36:14.075 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:36:13 smithi097 ceph-mon[33317]: from='client.24469 v1:172.21.15.97:0/404208310' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:36:14.076 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:36:13 smithi097 ceph-mon[33317]: 2.1a scrub starts 2024-03-11T21:36:14.076 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:36:13 smithi097 ceph-mon[33317]: 2.1a scrub ok 2024-03-11T21:36:14.076 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:36:13 smithi097 ceph-mon[33317]: Health check update: Degraded data redundancy: 14517/88197 objects degraded (16.460%), 30 pgs degraded (PG_DEGRADED) 2024-03-11T21:36:14.084 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:36:13 smithi092 ceph-mon[29572]: from='client.24469 v1:172.21.15.97:0/404208310' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:36:14.084 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:36:13 smithi092 ceph-mon[29572]: 2.1a scrub starts 2024-03-11T21:36:14.084 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:36:13 smithi092 ceph-mon[29572]: 2.1a scrub ok 2024-03-11T21:36:14.084 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:36:13 smithi092 ceph-mon[29572]: Health check update: Degraded data redundancy: 14517/88197 objects degraded (16.460%), 30 pgs degraded (PG_DEGRADED) 2024-03-11T21:36:14.084 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:36:13 smithi092 ceph-mon[37563]: from='client.24469 v1:172.21.15.97:0/404208310' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:36:14.084 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:36:13 smithi092 ceph-mon[37563]: 2.1a scrub starts 2024-03-11T21:36:14.084 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:36:13 smithi092 ceph-mon[37563]: 2.1a scrub ok 2024-03-11T21:36:14.084 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:36:13 smithi092 ceph-mon[37563]: Health check update: Degraded data redundancy: 14517/88197 objects degraded (16.460%), 30 pgs degraded (PG_DEGRADED) 2024-03-11T21:36:14.127 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:36:14.162 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:36:14.162 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:36:14.163 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:36:14.263 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:36:14.299 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:36:14.300 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:36:14.300 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:36:14.401 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:36:14.444 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:36:14.445 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:36:14.445 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:36:14.520 INFO:tasks.radosbench.radosbench.0.smithi092.stdout: 17 16 44247 44231 162.588 137.625 0.00650996 0.00614588 2024-03-11T21:36:14.547 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:36:14.583 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:36:14.584 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:36:14.584 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:36:14.685 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:36:14.728 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:36:14.729 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:36:14.729 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:36:14.830 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:36:14.865 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:36:14.865 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:36:14.866 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:36:14.966 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:36:15.004 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:36:15.005 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:36:15.005 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:36:15.069 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:36:14 smithi097 ceph-mon[33317]: pgmap v482: 148 pgs: 30 active+undersized+degraded, 32 active+undersized, 86 active+clean; 2.3 GiB data, 8.5 GiB used, 707 GiB / 715 GiB avail; 1.2 KiB/s rd, 164 MiB/s wr, 2.62k op/s; 19124/111801 objects degraded (17.105%) 2024-03-11T21:36:15.069 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:36:14 smithi097 ceph-mon[33317]: 4.5 scrub starts 2024-03-11T21:36:15.069 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:36:14 smithi097 ceph-mon[33317]: 4.5 scrub ok 2024-03-11T21:36:15.106 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:36:15.109 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:36:14 smithi092 ceph-mon[29572]: pgmap v482: 148 pgs: 30 active+undersized+degraded, 32 active+undersized, 86 active+clean; 2.3 GiB data, 8.5 GiB used, 707 GiB / 715 GiB avail; 1.2 KiB/s rd, 164 MiB/s wr, 2.62k op/s; 19124/111801 objects degraded (17.105%) 2024-03-11T21:36:15.109 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:36:14 smithi092 ceph-mon[29572]: 4.5 scrub starts 2024-03-11T21:36:15.109 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:36:14 smithi092 ceph-mon[29572]: 4.5 scrub ok 2024-03-11T21:36:15.110 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:36:14 smithi092 ceph-mon[37563]: pgmap v482: 148 pgs: 30 active+undersized+degraded, 32 active+undersized, 86 active+clean; 2.3 GiB data, 8.5 GiB used, 707 GiB / 715 GiB avail; 1.2 KiB/s rd, 164 MiB/s wr, 2.62k op/s; 19124/111801 objects degraded (17.105%) 2024-03-11T21:36:15.110 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:36:14 smithi092 ceph-mon[37563]: 4.5 scrub starts 2024-03-11T21:36:15.110 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:36:14 smithi092 ceph-mon[37563]: 4.5 scrub ok 2024-03-11T21:36:15.141 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:36:15.142 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:36:15.142 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:36:15.243 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:36:15.279 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:36:15.279 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:36:15.280 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:36:15.381 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:36:15.422 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:36:15.422 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:36:15.423 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:36:15.460 INFO:teuthology.orchestra.run.smithi092.stdout:osd.0: {} 2024-03-11T21:36:15.520 INFO:tasks.radosbench.radosbench.0.smithi092.stdout: 18 16 46169 46153 160.228 120.125 0.00380823 0.00621996 2024-03-11T21:36:15.523 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:36:15.546 INFO:teuthology.orchestra.run.smithi092.stdout:osd.1: {} 2024-03-11T21:36:15.549 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:36:15.549 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:36:15.550 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:36:15.573 INFO:teuthology.orchestra.run.smithi092.stdout:osd.2: {} 2024-03-11T21:36:15.607 INFO:teuthology.orchestra.run.smithi092.stdout:osd.4: {} 2024-03-11T21:36:15.642 INFO:teuthology.orchestra.run.smithi092.stdout:osd.5: {} 2024-03-11T21:36:15.650 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:36:15.679 INFO:teuthology.orchestra.run.smithi092.stdout:osd.6: {} 2024-03-11T21:36:15.688 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:36:15.689 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:36:15.689 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:36:15.704 INFO:teuthology.orchestra.run.smithi092.stdout:osd.7: {} 2024-03-11T21:36:15.704 INFO:teuthology.orchestra.run.smithi092.stderr:osd.0: osd_enable_op_tracker = '' 2024-03-11T21:36:15.704 INFO:teuthology.orchestra.run.smithi092.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-11T21:36:15.704 INFO:teuthology.orchestra.run.smithi092.stderr:osd.2: osd_enable_op_tracker = '' 2024-03-11T21:36:15.704 INFO:teuthology.orchestra.run.smithi092.stderr:Error ENXIO: problem getting command descriptions from osd.3 2024-03-11T21:36:15.704 INFO:teuthology.orchestra.run.smithi092.stderr:osd.4: osd_enable_op_tracker = '' 2024-03-11T21:36:15.704 INFO:teuthology.orchestra.run.smithi092.stderr:osd.5: osd_enable_op_tracker = '' 2024-03-11T21:36:15.705 INFO:teuthology.orchestra.run.smithi092.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-11T21:36:15.705 INFO:teuthology.orchestra.run.smithi092.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-11T21:36:15.789 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:36:15.827 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:36:15.827 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:36:15.827 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:36:15.928 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:36:15.965 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:36:15.966 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:36:15.966 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:36:16.067 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:36:16.076 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:36:15 smithi097 ceph-mon[33317]: 6.15 scrub starts 2024-03-11T21:36:16.076 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:36:15 smithi097 ceph-mon[33317]: 6.15 scrub ok 2024-03-11T21:36:16.077 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:36:15 smithi097 ceph-mon[33317]: from='client.? v1:172.21.15.92:0/1506905993' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-11T21:36:16.102 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:36:16.102 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:36:16.103 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:36:16.109 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:36:15 smithi092 ceph-mon[29572]: 6.15 scrub starts 2024-03-11T21:36:16.109 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:36:15 smithi092 ceph-mon[29572]: 6.15 scrub ok 2024-03-11T21:36:16.109 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:36:15 smithi092 ceph-mon[29572]: from='client.? v1:172.21.15.92:0/1506905993' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-11T21:36:16.110 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:36:15 smithi092 ceph-mon[37563]: 6.15 scrub starts 2024-03-11T21:36:16.110 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:36:15 smithi092 ceph-mon[37563]: 6.15 scrub ok 2024-03-11T21:36:16.110 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:36:15 smithi092 ceph-mon[37563]: from='client.? v1:172.21.15.92:0/1506905993' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-11T21:36:16.203 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:36:16.242 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:36:16.243 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:36:16.243 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:36:16.344 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:36:16.369 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:36:16.370 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:36:16.370 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:36:16.471 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:36:16.479 DEBUG:teuthology.orchestra.run.smithi092:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid 00db0016-dfed-11ee-95c8-87774f69a715 -- ceph osd set noscrub 2024-03-11T21:36:16.504 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:36:16.505 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:36:16.505 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:36:16.520 INFO:tasks.radosbench.radosbench.0.smithi092.stdout: 19 16 48042 48026 157.956 117.062 0.00671705 0.0063259 2024-03-11T21:36:16.606 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:36:16.648 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:36:16.649 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:36:16.649 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:36:16.750 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:36:16.801 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:36:16.802 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:36:16.802 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:36:16.903 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:36:16.917 INFO:teuthology.orchestra.run.smithi092.stdout:ERROR: (22) Invalid argument 2024-03-11T21:36:16.918 INFO:teuthology.orchestra.run.smithi092.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-11T21:36:16.939 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:36:16.939 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:36:16.940 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:36:17.041 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:36:17.075 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:36:17.075 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:36:16 smithi097 ceph-mon[33317]: pgmap v483: 148 pgs: 30 active+undersized+degraded, 32 active+undersized, 86 active+clean; 2.4 GiB data, 8.6 GiB used, 707 GiB / 715 GiB avail; 1.2 KiB/s rd, 132 MiB/s wr, 2.12k op/s; 19772/119721 objects degraded (16.515%) 2024-03-11T21:36:17.076 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:36:16 smithi097 ceph-mon[33317]: 5.19 scrub starts 2024-03-11T21:36:17.076 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:36:16 smithi097 ceph-mon[33317]: 5.19 scrub ok 2024-03-11T21:36:17.076 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:36:17.076 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:36:17.110 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:36:16 smithi092 ceph-mon[29572]: pgmap v483: 148 pgs: 30 active+undersized+degraded, 32 active+undersized, 86 active+clean; 2.4 GiB data, 8.6 GiB used, 707 GiB / 715 GiB avail; 1.2 KiB/s rd, 132 MiB/s wr, 2.12k op/s; 19772/119721 objects degraded (16.515%) 2024-03-11T21:36:17.110 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:36:16 smithi092 ceph-mon[29572]: 5.19 scrub starts 2024-03-11T21:36:17.110 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:36:16 smithi092 ceph-mon[29572]: 5.19 scrub ok 2024-03-11T21:36:17.110 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:36:16 smithi092 ceph-mon[37563]: pgmap v483: 148 pgs: 30 active+undersized+degraded, 32 active+undersized, 86 active+clean; 2.4 GiB data, 8.6 GiB used, 707 GiB / 715 GiB avail; 1.2 KiB/s rd, 132 MiB/s wr, 2.12k op/s; 19772/119721 objects degraded (16.515%) 2024-03-11T21:36:17.110 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:36:16 smithi092 ceph-mon[37563]: 5.19 scrub starts 2024-03-11T21:36:17.110 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:36:16 smithi092 ceph-mon[37563]: 5.19 scrub ok 2024-03-11T21:36:17.177 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:36:17.212 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:36:17.213 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:36:17.214 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:36:17.306 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5, 6, 7] out_osds: [] dead_osds: [3] live_osds: [0, 1, 2, 4, 5, 6, 7] 2024-03-11T21:36:17.307 INFO:tasks.thrashosds.thrasher:Reviving osd 3 2024-03-11T21:36:17.307 INFO:tasks.cephadm.osd.3:Restarting osd.3 (starting--it wasn't running)... 2024-03-11T21:36:17.307 DEBUG:teuthology.orchestra.run.smithi092:osd.3> sudo journalctl -f -n 0 -u ceph-00db0016-dfed-11ee-95c8-87774f69a715@osd.3.service 2024-03-11T21:36:17.309 DEBUG:teuthology.orchestra.run.smithi092:> sudo systemctl start ceph-00db0016-dfed-11ee-95c8-87774f69a715@osd.3 2024-03-11T21:36:17.314 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:36:17.350 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:36:17.350 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:36:17.351 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:36:17.452 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:36:17.479 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-03-11T21:36:17.480 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-03-11T21:36:17.503 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:36:17.504 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:36:17.504 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:36:17.520 INFO:tasks.radosbench.radosbench.0.smithi092.stdout:2024-03-11T21:36:17.539767+0000 min lat: 0.00114249 max lat: 0.134206 avg lat: 0.00639265 2024-03-11T21:36:17.520 INFO:tasks.radosbench.radosbench.0.smithi092.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-03-11T21:36:17.520 INFO:tasks.radosbench.radosbench.0.smithi092.stdout: 20 16 50042 50026 156.308 125 0.00507358 0.00639265 2024-03-11T21:36:17.605 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:36:17.609 INFO:journalctl@ceph.osd.3.smithi092.stdout:Mar 11 21:36:17 smithi092 systemd[1]: Starting Ceph osd.3 for 00db0016-dfed-11ee-95c8-87774f69a715... 2024-03-11T21:36:17.639 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:36:17.639 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:36:17.640 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:36:17.697 INFO:teuthology.orchestra.run.smithi092.stderr:Inferring config /var/lib/ceph/00db0016-dfed-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T21:36:17.741 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:36:17.776 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:36:17.776 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:36:17.777 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:36:17.877 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:36:17.911 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:36:17.912 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:36:17.913 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:36:18.014 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:36:18.051 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:36:18.051 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:36:18.052 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:36:18.109 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:36:17 smithi092 ceph-mon[29572]: 2.b deep-scrub starts 2024-03-11T21:36:18.110 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:36:17 smithi092 ceph-mon[29572]: 2.b deep-scrub ok 2024-03-11T21:36:18.110 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:36:17 smithi092 ceph-mon[37563]: 2.b deep-scrub starts 2024-03-11T21:36:18.110 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:36:17 smithi092 ceph-mon[37563]: 2.b deep-scrub ok 2024-03-11T21:36:18.110 INFO:journalctl@ceph.mgr.y.smithi092.stdout:Mar 11 21:36:17 smithi092 ceph-00db0016-dfed-11ee-95c8-87774f69a715-mgr-y[29801]: ::ffff:172.21.15.97 - - [11/Mar/2024:21:36:17] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-11T21:36:18.153 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:36:18.186 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:36:18.187 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:36:18.187 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:36:18.198 DEBUG:teuthology.orchestra.run.smithi092:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid 00db0016-dfed-11ee-95c8-87774f69a715 -- ceph daemon osd.0 dump_blocked_ops 2024-03-11T21:36:18.288 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:36:18.325 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:36:18.325 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:36:17 smithi097 ceph-mon[33317]: 2.b deep-scrub starts 2024-03-11T21:36:18.325 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:36:17 smithi097 ceph-mon[33317]: 2.b deep-scrub ok 2024-03-11T21:36:18.326 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:36:18.326 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:36:18.427 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:36:18.462 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:36:18.462 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:36:18.463 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:36:18.520 INFO:tasks.radosbench.radosbench.0.smithi092.stdout: 21 15 51974 51959 154.617 120.812 0.00462614 0.00646291 2024-03-11T21:36:18.564 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:36:18.610 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:36:18.610 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:36:18.611 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:36:18.712 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:36:18.747 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:36:18.748 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:36:18.748 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:36:18.849 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:36:18.884 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:36:18.885 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:36:18.886 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:36:18.986 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:36:19.023 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:36:19.023 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:36:19.024 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:36:19.109 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:36:18 smithi092 ceph-mon[29572]: pgmap v484: 148 pgs: 30 active+undersized+degraded, 32 active+undersized, 86 active+clean; 2.6 GiB data, 9.1 GiB used, 706 GiB / 715 GiB avail; 570 B/s rd, 117 MiB/s wr, 1.88k op/s; 21678/129435 objects degraded (16.748%) 2024-03-11T21:36:19.110 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:36:18 smithi092 ceph-mon[29572]: Health check update: Degraded data redundancy: 21678/129435 objects degraded (16.748%), 30 pgs degraded (PG_DEGRADED) 2024-03-11T21:36:19.110 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:36:18 smithi092 ceph-mon[29572]: 6.1 scrub starts 2024-03-11T21:36:19.110 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:36:18 smithi092 ceph-mon[29572]: 6.1 scrub ok 2024-03-11T21:36:19.110 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:36:18 smithi092 ceph-mon[37563]: pgmap v484: 148 pgs: 30 active+undersized+degraded, 32 active+undersized, 86 active+clean; 2.6 GiB data, 9.1 GiB used, 706 GiB / 715 GiB avail; 570 B/s rd, 117 MiB/s wr, 1.88k op/s; 21678/129435 objects degraded (16.748%) 2024-03-11T21:36:19.110 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:36:18 smithi092 ceph-mon[37563]: Health check update: Degraded data redundancy: 21678/129435 objects degraded (16.748%), 30 pgs degraded (PG_DEGRADED) 2024-03-11T21:36:19.110 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:36:18 smithi092 ceph-mon[37563]: 6.1 scrub starts 2024-03-11T21:36:19.110 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:36:18 smithi092 ceph-mon[37563]: 6.1 scrub ok 2024-03-11T21:36:19.125 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:36:19.159 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:36:19.160 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:36:19.160 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:36:19.261 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:36:19.296 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:36:19.297 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:36:19.297 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:36:19.325 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:36:18 smithi097 ceph-mon[33317]: pgmap v484: 148 pgs: 30 active+undersized+degraded, 32 active+undersized, 86 active+clean; 2.6 GiB data, 9.1 GiB used, 706 GiB / 715 GiB avail; 570 B/s rd, 117 MiB/s wr, 1.88k op/s; 21678/129435 objects degraded (16.748%) 2024-03-11T21:36:19.325 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:36:18 smithi097 ceph-mon[33317]: Health check update: Degraded data redundancy: 21678/129435 objects degraded (16.748%), 30 pgs degraded (PG_DEGRADED) 2024-03-11T21:36:19.326 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:36:18 smithi097 ceph-mon[33317]: 6.1 scrub starts 2024-03-11T21:36:19.326 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:36:18 smithi097 ceph-mon[33317]: 6.1 scrub ok 2024-03-11T21:36:19.398 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:36:19.437 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:36:19.438 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:36:19.438 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:36:19.481 DEBUG:teuthology.orchestra.run.smithi092:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid 00db0016-dfed-11ee-95c8-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-03-11T21:36:19.522 INFO:tasks.radosbench.radosbench.0.smithi092.stdout: 22 16 53912 53896 153.082 121.062 0.00473661 0.00652776 2024-03-11T21:36:19.538 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:36:19.572 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:36:19.573 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:36:19.573 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:36:19.660 INFO:journalctl@ceph.osd.3.smithi092.stdout:Mar 11 21:36:19 smithi092 podman[133167]: 2024-03-11 21:36:19.433579745 +0000 UTC m=+0.464653746 image pull 4133f8bc41bb201212a28168427a2edb1f6f3795e694bf61031ea9e2f3ff5e87 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:8fe9635f3beb0247496d7e321f3068bd10c333c433ef4824d7e1c1a7d3d78379 2024-03-11T21:36:19.660 INFO:journalctl@ceph.osd.3.smithi092.stdout:Mar 11 21:36:19 smithi092 podman[133167]: 2024-03-11 21:36:19.565959791 +0000 UTC m=+0.597033783 container create b6c1fe4e4f5fb8ad16c48f728698fe7e156a0da5814ea98bbb50a42195cad602 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:8fe9635f3beb0247496d7e321f3068bd10c333c433ef4824d7e1c1a7d3d78379, name=ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd-3-activate, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240307, org.label-schema.vendor=CentOS, GIT_CLEAN=True, org.label-schema.license=GPLv2, ceph=True, io.buildah.version=1.33.5, RELEASE=main-2b0101e, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0) 2024-03-11T21:36:19.674 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:36:19.726 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:36:19.727 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:36:19.727 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:36:19.828 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:36:19.863 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:36:19.864 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:36:19.864 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:36:19.934 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:36:19 smithi092 ceph-mon[29572]: 5.9 scrub starts 2024-03-11T21:36:19.934 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:36:19 smithi092 ceph-mon[29572]: 5.9 scrub ok 2024-03-11T21:36:19.935 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:36:19 smithi092 ceph-mon[37563]: 5.9 scrub starts 2024-03-11T21:36:19.935 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:36:19 smithi092 ceph-mon[37563]: 5.9 scrub ok 2024-03-11T21:36:19.964 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:36:19.991 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:36:19.991 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:36:19.992 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:36:20.093 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:36:20.128 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:36:20.129 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:36:20.129 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:36:20.226 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:36:19 smithi097 ceph-mon[33317]: 5.9 scrub starts 2024-03-11T21:36:20.227 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:36:19 smithi097 ceph-mon[33317]: 5.9 scrub ok 2024-03-11T21:36:20.230 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:36:20.271 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:36:20.272 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:36:20.272 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:36:20.297 INFO:journalctl@ceph.osd.3.smithi092.stdout:Mar 11 21:36:20 smithi092 podman[133167]: 2024-03-11 21:36:20.05652935 +0000 UTC m=+1.087603343 container init b6c1fe4e4f5fb8ad16c48f728698fe7e156a0da5814ea98bbb50a42195cad602 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:8fe9635f3beb0247496d7e321f3068bd10c333c433ef4824d7e1c1a7d3d78379, name=ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd-3-activate, CEPH_POINT_RELEASE=, RELEASE=main-2b0101e, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, org.label-schema.license=GPLv2, org.label-schema.build-date=20240307, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, GIT_CLEAN=True) 2024-03-11T21:36:20.297 INFO:journalctl@ceph.osd.3.smithi092.stdout:Mar 11 21:36:20 smithi092 podman[133167]: 2024-03-11 21:36:20.064013913 +0000 UTC m=+1.095087897 container start b6c1fe4e4f5fb8ad16c48f728698fe7e156a0da5814ea98bbb50a42195cad602 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:8fe9635f3beb0247496d7e321f3068bd10c333c433ef4824d7e1c1a7d3d78379, name=ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd-3-activate, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, GIT_CLEAN=True, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240307, RELEASE=main-2b0101e) 2024-03-11T21:36:20.297 INFO:journalctl@ceph.osd.3.smithi092.stdout:Mar 11 21:36:20 smithi092 podman[133167]: 2024-03-11 21:36:20.155582311 +0000 UTC m=+1.186656311 container attach b6c1fe4e4f5fb8ad16c48f728698fe7e156a0da5814ea98bbb50a42195cad602 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:8fe9635f3beb0247496d7e321f3068bd10c333c433ef4824d7e1c1a7d3d78379, name=ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd-3-activate, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240307, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, ceph=True, RELEASE=main-2b0101e, org.label-schema.schema-version=1.0) 2024-03-11T21:36:20.374 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:36:20.421 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:36:20.423 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:36:20.423 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:36:20.514 INFO:teuthology.orchestra.run.smithi092.stderr:Inferring config /var/lib/ceph/00db0016-dfed-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T21:36:20.522 INFO:tasks.radosbench.radosbench.0.smithi092.stdout: 23 16 55650 55634 151.148 108.625 0.0223142 0.00661019 2024-03-11T21:36:20.524 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:36:20.602 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:36:20.602 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:36:20.603 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:36:20.632 INFO:teuthology.orchestra.run.smithi092.stderr:Inferring config /var/lib/ceph/00db0016-dfed-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T21:36:20.704 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:36:20.752 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:36:20.753 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:36:20.753 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:36:20.854 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:36:20.895 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:36:20.896 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:36:20.896 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:36:20.997 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:36:21.031 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:36:20 smithi092 ceph-mon[29572]: pgmap v485: 148 pgs: 30 active+undersized+degraded, 32 active+undersized, 86 active+clean; 3.1 GiB data, 10 GiB used, 705 GiB / 715 GiB avail; 940 B/s rd, 137 MiB/s wr, 2.20k op/s; 24929/150711 objects degraded (16.541%) 2024-03-11T21:36:21.032 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:36:20 smithi092 ceph-mon[29572]: purged_snaps scrub starts 2024-03-11T21:36:21.032 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:36:20 smithi092 ceph-mon[29572]: purged_snaps scrub ok 2024-03-11T21:36:21.032 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:36:20 smithi092 ceph-mon[37563]: pgmap v485: 148 pgs: 30 active+undersized+degraded, 32 active+undersized, 86 active+clean; 3.1 GiB data, 10 GiB used, 705 GiB / 715 GiB avail; 940 B/s rd, 137 MiB/s wr, 2.20k op/s; 24929/150711 objects degraded (16.541%) 2024-03-11T21:36:21.032 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:36:20 smithi092 ceph-mon[37563]: purged_snaps scrub starts 2024-03-11T21:36:21.032 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:36:20 smithi092 ceph-mon[37563]: purged_snaps scrub ok 2024-03-11T21:36:21.032 INFO:journalctl@ceph.osd.3.smithi092.stdout:Mar 11 21:36:20 smithi092 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd-3-activate[133252]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-3 2024-03-11T21:36:21.033 INFO:journalctl@ceph.osd.3.smithi092.stdout:Mar 11 21:36:20 smithi092 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd-3-activate[133252]: Running command: /usr/bin/ceph-bluestore-tool prime-osd-dir --path /var/lib/ceph/osd/ceph-3 --no-mon-config --dev /dev/mapper/vg_nvme-lv_1 2024-03-11T21:36:21.033 INFO:journalctl@ceph.osd.3.smithi092.stdout:Mar 11 21:36:20 smithi092 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd-3-activate[133252]: Running command: /usr/bin/chown -h ceph:ceph /dev/mapper/vg_nvme-lv_1 2024-03-11T21:36:21.033 INFO:journalctl@ceph.osd.3.smithi092.stdout:Mar 11 21:36:20 smithi092 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd-3-activate[133252]: Running command: /usr/bin/chown -R ceph:ceph /dev/dm-0 2024-03-11T21:36:21.033 INFO:journalctl@ceph.osd.3.smithi092.stdout:Mar 11 21:36:20 smithi092 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd-3-activate[133252]: Running command: /usr/bin/ln -s /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-3/block 2024-03-11T21:36:21.033 INFO:journalctl@ceph.osd.3.smithi092.stdout:Mar 11 21:36:20 smithi092 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd-3-activate[133252]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-3 2024-03-11T21:36:21.033 INFO:journalctl@ceph.osd.3.smithi092.stdout:Mar 11 21:36:20 smithi092 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd-3-activate[133252]: --> ceph-volume raw activate successful for osd ID: 3 2024-03-11T21:36:21.033 INFO:journalctl@ceph.osd.3.smithi092.stdout:Mar 11 21:36:20 smithi092 bash[133167]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-3 2024-03-11T21:36:21.033 INFO:journalctl@ceph.osd.3.smithi092.stdout:Mar 11 21:36:20 smithi092 bash[133167]: Running command: /usr/bin/ceph-bluestore-tool prime-osd-dir --path /var/lib/ceph/osd/ceph-3 --no-mon-config --dev /dev/mapper/vg_nvme-lv_1 2024-03-11T21:36:21.033 INFO:journalctl@ceph.osd.3.smithi092.stdout:Mar 11 21:36:20 smithi092 bash[133167]: Running command: /usr/bin/chown -h ceph:ceph /dev/mapper/vg_nvme-lv_1 2024-03-11T21:36:21.033 INFO:journalctl@ceph.osd.3.smithi092.stdout:Mar 11 21:36:20 smithi092 bash[133167]: Running command: /usr/bin/chown -R ceph:ceph /dev/dm-0 2024-03-11T21:36:21.033 INFO:journalctl@ceph.osd.3.smithi092.stdout:Mar 11 21:36:20 smithi092 bash[133167]: Running command: /usr/bin/ln -s /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-3/block 2024-03-11T21:36:21.033 INFO:journalctl@ceph.osd.3.smithi092.stdout:Mar 11 21:36:20 smithi092 bash[133167]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-3 2024-03-11T21:36:21.033 INFO:journalctl@ceph.osd.3.smithi092.stdout:Mar 11 21:36:20 smithi092 bash[133167]: --> ceph-volume raw activate successful for osd ID: 3 2024-03-11T21:36:21.035 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:36:21.036 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:36:21.037 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:36:21.138 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:36:21.176 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:36:21.176 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:36:21.177 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:36:21.277 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:36:21.301 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:36:20 smithi097 ceph-mon[33317]: pgmap v485: 148 pgs: 30 active+undersized+degraded, 32 active+undersized, 86 active+clean; 3.1 GiB data, 10 GiB used, 705 GiB / 715 GiB avail; 940 B/s rd, 137 MiB/s wr, 2.20k op/s; 24929/150711 objects degraded (16.541%) 2024-03-11T21:36:21.301 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:36:20 smithi097 ceph-mon[33317]: purged_snaps scrub starts 2024-03-11T21:36:21.301 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:36:20 smithi097 ceph-mon[33317]: purged_snaps scrub ok 2024-03-11T21:36:21.309 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:36:21.310 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:36:21.310 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:36:21.360 INFO:journalctl@ceph.osd.3.smithi092.stdout:Mar 11 21:36:21 smithi092 podman[133167]: 2024-03-11 21:36:21.17931095 +0000 UTC m=+2.210384947 container died b6c1fe4e4f5fb8ad16c48f728698fe7e156a0da5814ea98bbb50a42195cad602 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:8fe9635f3beb0247496d7e321f3068bd10c333c433ef4824d7e1c1a7d3d78379, name=ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd-3-activate, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, RELEASE=main-2b0101e, ceph=True, GIT_CLEAN=True, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240307) 2024-03-11T21:36:21.411 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:36:21.445 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:36:21.446 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:36:21.447 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:36:21.522 INFO:tasks.radosbench.radosbench.0.smithi092.stdout: 24 16 57266 57250 149.059 101 0.013552 0.00670372 2024-03-11T21:36:21.548 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:36:21.578 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:36:21.579 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:36:21.580 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:36:21.680 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:36:21.712 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:36:21.713 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:36:21.713 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:36:21.814 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:36:21.850 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:36:21.851 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:36:21.852 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:36:21.867 INFO:teuthology.orchestra.run.smithi092.stderr:noscrub is set 2024-03-11T21:36:21.953 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:36:21.993 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:36:21.993 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:36:21.993 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:36:22.094 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:36:22.129 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:36:22.130 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:36:22.130 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:36:22.231 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:36:22.263 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:36:22.263 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:36:22.263 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:36:22.325 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:36:21 smithi097 ceph-mon[33317]: osdmap e271: 8 total, 7 up, 8 in 2024-03-11T21:36:22.325 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:36:21 smithi097 ceph-mon[33317]: from='client.? v1:172.21.15.92:0/3847107272' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-11T21:36:22.359 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:36:21 smithi092 ceph-mon[29572]: osdmap e271: 8 total, 7 up, 8 in 2024-03-11T21:36:22.359 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:36:21 smithi092 ceph-mon[29572]: from='client.? v1:172.21.15.92:0/3847107272' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-11T21:36:22.359 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:36:21 smithi092 ceph-mon[37563]: osdmap e271: 8 total, 7 up, 8 in 2024-03-11T21:36:22.359 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:36:21 smithi092 ceph-mon[37563]: from='client.? v1:172.21.15.92:0/3847107272' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-11T21:36:22.364 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:36:22.408 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:36:22.410 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:36:22.411 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:36:22.513 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:36:22.522 INFO:tasks.radosbench.radosbench.0.smithi092.stdout: 25 16 58878 58862 147.126 100.75 0.00905252 0.00679151 2024-03-11T21:36:22.557 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:36:22.557 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:36:22.557 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:36:22.609 INFO:journalctl@ceph.osd.3.smithi092.stdout:Mar 11 21:36:22 smithi092 podman[133563]: 2024-03-11 21:36:22.377212885 +0000 UTC m=+1.514841957 container remove b6c1fe4e4f5fb8ad16c48f728698fe7e156a0da5814ea98bbb50a42195cad602 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:8fe9635f3beb0247496d7e321f3068bd10c333c433ef4824d7e1c1a7d3d78379, name=ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd-3-activate, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.build-date=20240307, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, ceph=True, GIT_CLEAN=True, RELEASE=main-2b0101e, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS) 2024-03-11T21:36:22.658 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:36:22.699 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:36:22.699 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:36:22.700 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:36:22.800 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:36:22.825 INFO:journalctl@ceph.iscsi.iscsi.a.smithi097.stdout:Mar 11 21:36:22 smithi097 ceph-00db0016-dfed-11ee-95c8-87774f69a715-iscsi-iscsi-a[60943]: debug there is no tcmu-runner data available 2024-03-11T21:36:22.836 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:36:22.836 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:36:22.837 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:36:22.937 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:36:22.974 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:36:22.975 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:36:22.975 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:36:23.075 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:36:23.115 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:36:23.115 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:36:23.116 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:36:23.216 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:36:23.251 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:36:23.252 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:36:23.252 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:36:23.325 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:36:22 smithi097 ceph-mon[33317]: pgmap v487: 148 pgs: 30 active+undersized+degraded, 32 active+undersized, 86 active+clean; 3.1 GiB data, 10 GiB used, 705 GiB / 715 GiB avail; 1023 B/s rd, 130 MiB/s wr, 2.08k op/s; 24929/150711 objects degraded (16.541%) 2024-03-11T21:36:23.326 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:36:22 smithi097 ceph-mon[33317]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-03-11T21:36:23.326 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:36:22 smithi097 ceph-mon[33317]: from='client.? v1:172.21.15.92:0/3847107272' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-03-11T21:36:23.326 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:36:22 smithi097 ceph-mon[33317]: osdmap e272: 8 total, 7 up, 8 in 2024-03-11T21:36:23.326 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:36:22 smithi097 ceph-mon[33317]: from='mgr.14607 v1:172.21.15.92:0/1702717597' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T21:36:23.326 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:36:22 smithi097 ceph-mon[33317]: purged_snaps scrub starts 2024-03-11T21:36:23.326 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:36:22 smithi097 ceph-mon[33317]: purged_snaps scrub ok 2024-03-11T21:36:23.352 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:36:23.359 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:36:22 smithi092 ceph-mon[29572]: pgmap v487: 148 pgs: 30 active+undersized+degraded, 32 active+undersized, 86 active+clean; 3.1 GiB data, 10 GiB used, 705 GiB / 715 GiB avail; 1023 B/s rd, 130 MiB/s wr, 2.08k op/s; 24929/150711 objects degraded (16.541%) 2024-03-11T21:36:23.359 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:36:22 smithi092 ceph-mon[29572]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-03-11T21:36:23.359 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:36:22 smithi092 ceph-mon[29572]: from='client.? v1:172.21.15.92:0/3847107272' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-03-11T21:36:23.359 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:36:22 smithi092 ceph-mon[29572]: osdmap e272: 8 total, 7 up, 8 in 2024-03-11T21:36:23.360 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:36:22 smithi092 ceph-mon[29572]: from='mgr.14607 v1:172.21.15.92:0/1702717597' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T21:36:23.360 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:36:22 smithi092 ceph-mon[29572]: purged_snaps scrub starts 2024-03-11T21:36:23.360 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:36:22 smithi092 ceph-mon[29572]: purged_snaps scrub ok 2024-03-11T21:36:23.360 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:36:22 smithi092 ceph-mon[37563]: pgmap v487: 148 pgs: 30 active+undersized+degraded, 32 active+undersized, 86 active+clean; 3.1 GiB data, 10 GiB used, 705 GiB / 715 GiB avail; 1023 B/s rd, 130 MiB/s wr, 2.08k op/s; 24929/150711 objects degraded (16.541%) 2024-03-11T21:36:23.360 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:36:22 smithi092 ceph-mon[37563]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-03-11T21:36:23.360 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:36:22 smithi092 ceph-mon[37563]: from='client.? v1:172.21.15.92:0/3847107272' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-03-11T21:36:23.360 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:36:22 smithi092 ceph-mon[37563]: osdmap e272: 8 total, 7 up, 8 in 2024-03-11T21:36:23.360 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:36:22 smithi092 ceph-mon[37563]: from='mgr.14607 v1:172.21.15.92:0/1702717597' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T21:36:23.360 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:36:22 smithi092 ceph-mon[37563]: purged_snaps scrub starts 2024-03-11T21:36:23.361 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:36:22 smithi092 ceph-mon[37563]: purged_snaps scrub ok 2024-03-11T21:36:23.383 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:36:23.383 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:36:23.384 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:36:23.485 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:36:23.517 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:36:23.518 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:36:23.519 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:36:23.522 INFO:tasks.radosbench.radosbench.0.smithi092.stdout: 26 16 60586 60570 145.573 106.75 0.00445415 0.00686502 2024-03-11T21:36:23.620 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:36:23.654 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:36:23.655 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:36:23.655 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:36:23.756 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:36:23.791 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:36:23.792 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:36:23.792 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:36:23.893 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:36:23.932 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:36:23.933 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:36:23.934 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:36:24.035 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:36:24.070 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:36:24.070 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:36:24.071 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:36:24.109 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:36:23 smithi092 ceph-mon[29572]: from='client.24469 v1:172.21.15.97:0/404208310' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:36:24.109 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:36:23 smithi092 ceph-mon[29572]: osdmap e273: 8 total, 7 up, 8 in 2024-03-11T21:36:24.110 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:36:23 smithi092 ceph-mon[29572]: Health check update: Degraded data redundancy: 24929/150711 objects degraded (16.541%), 30 pgs degraded (PG_DEGRADED) 2024-03-11T21:36:24.110 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:36:23 smithi092 ceph-mon[29572]: osdmap e274: 8 total, 7 up, 8 in 2024-03-11T21:36:24.110 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:36:23 smithi092 ceph-mon[37563]: from='client.24469 v1:172.21.15.97:0/404208310' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:36:24.110 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:36:23 smithi092 ceph-mon[37563]: osdmap e273: 8 total, 7 up, 8 in 2024-03-11T21:36:24.110 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:36:23 smithi092 ceph-mon[37563]: Health check update: Degraded data redundancy: 24929/150711 objects degraded (16.541%), 30 pgs degraded (PG_DEGRADED) 2024-03-11T21:36:24.110 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:36:23 smithi092 ceph-mon[37563]: osdmap e274: 8 total, 7 up, 8 in 2024-03-11T21:36:24.172 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:36:24.213 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:36:24.214 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:36:24.214 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:36:24.315 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:36:24.325 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:36:23 smithi097 ceph-mon[33317]: from='client.24469 v1:172.21.15.97:0/404208310' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:36:24.325 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:36:23 smithi097 ceph-mon[33317]: osdmap e273: 8 total, 7 up, 8 in 2024-03-11T21:36:24.326 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:36:23 smithi097 ceph-mon[33317]: Health check update: Degraded data redundancy: 24929/150711 objects degraded (16.541%), 30 pgs degraded (PG_DEGRADED) 2024-03-11T21:36:24.326 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:36:23 smithi097 ceph-mon[33317]: osdmap e274: 8 total, 7 up, 8 in 2024-03-11T21:36:24.350 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:36:24.351 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:36:24.352 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:36:24.452 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:36:24.492 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:36:24.492 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:36:24.493 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:36:24.522 INFO:tasks.radosbench.radosbench.0.smithi092.stdout: 27 15 62313 62298 144.181 108 0.0179847 0.00693137 2024-03-11T21:36:24.593 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:36:24.631 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:36:24.631 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:36:24.632 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:36:24.732 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:36:24.766 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:36:24.767 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:36:24.767 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:36:24.868 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:36:24.918 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:36:24.918 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:36:24.919 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:36:25.020 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:36:25.051 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:36:25.052 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:36:25.052 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:36:25.109 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:36:24 smithi092 ceph-mon[29572]: pgmap v490: 148 pgs: 30 active+undersized+degraded, 32 active+undersized, 86 active+clean; 3.5 GiB data, 12 GiB used, 704 GiB / 715 GiB avail; 1.7 KiB/s rd, 143 MiB/s wr, 2.29k op/s; 28852/170637 objects degraded (16.908%) 2024-03-11T21:36:25.110 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:36:24 smithi092 ceph-mon[37563]: pgmap v490: 148 pgs: 30 active+undersized+degraded, 32 active+undersized, 86 active+clean; 3.5 GiB data, 12 GiB used, 704 GiB / 715 GiB avail; 1.7 KiB/s rd, 143 MiB/s wr, 2.29k op/s; 28852/170637 objects degraded (16.908%) 2024-03-11T21:36:25.153 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:36:25.193 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:36:25.194 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:36:25.194 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:36:25.295 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:36:25.325 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:36:24 smithi097 ceph-mon[33317]: pgmap v490: 148 pgs: 30 active+undersized+degraded, 32 active+undersized, 86 active+clean; 3.5 GiB data, 12 GiB used, 704 GiB / 715 GiB avail; 1.7 KiB/s rd, 143 MiB/s wr, 2.29k op/s; 28852/170637 objects degraded (16.908%) 2024-03-11T21:36:25.330 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:36:25.331 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:36:25.331 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:36:25.432 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:36:25.433 INFO:journalctl@ceph.osd.3.smithi092.stdout:Mar 11 21:36:25 smithi092 podman[133931]: 2024-03-11 21:36:25.162199335 +0000 UTC m=+0.276696718 image pull 4133f8bc41bb201212a28168427a2edb1f6f3795e694bf61031ea9e2f3ff5e87 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:8fe9635f3beb0247496d7e321f3068bd10c333c433ef4824d7e1c1a7d3d78379 2024-03-11T21:36:25.476 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:36:25.477 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:36:25.477 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:36:25.522 INFO:tasks.radosbench.radosbench.0.smithi092.stdout: 28 16 63973 63957 142.735 103.688 0.00386426 0.00700153 2024-03-11T21:36:25.578 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:36:25.616 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:36:25.616 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:36:25.617 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:36:25.717 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:36:25.751 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:36:25.752 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:36:25.752 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:36:25.753 INFO:journalctl@ceph.osd.3.smithi092.stdout:Mar 11 21:36:25 smithi092 podman[133931]: 2024-03-11 21:36:25.56945539 +0000 UTC m=+0.683952761 container create ea196e969b3c874af0c911ce9d141e04fc33650a47e5e97222467872860add15 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:8fe9635f3beb0247496d7e321f3068bd10c333c433ef4824d7e1c1a7d3d78379, name=ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd-3, org.label-schema.license=GPLv2, org.label-schema.build-date=20240307, GIT_CLEAN=True, RELEASE=main-2b0101e, ceph=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD) 2024-03-11T21:36:25.853 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:36:25.868 INFO:teuthology.orchestra.run.smithi092.stdout:osd.0: {} 2024-03-11T21:36:25.871 DEBUG:teuthology.orchestra.run.smithi092:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid 00db0016-dfed-11ee-95c8-87774f69a715 -- ceph osd set nodeep-scrub 2024-03-11T21:36:25.890 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:36:25.891 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:36:25.891 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:36:25.908 INFO:teuthology.orchestra.run.smithi092.stdout:osd.1: {} 2024-03-11T21:36:25.946 INFO:teuthology.orchestra.run.smithi092.stdout:osd.2: {} 2024-03-11T21:36:25.986 INFO:teuthology.orchestra.run.smithi092.stdout:osd.4: {} 2024-03-11T21:36:25.992 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:36:26.025 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:36:26.026 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:36:26.027 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:36:26.027 INFO:teuthology.orchestra.run.smithi092.stdout:osd.5: {} 2024-03-11T21:36:26.028 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:36:25 smithi092 ceph-mon[29572]: from='client.? v1:172.21.15.92:0/3869295930' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-11T21:36:26.028 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:36:25 smithi092 ceph-mon[37563]: from='client.? v1:172.21.15.92:0/3869295930' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-11T21:36:26.081 INFO:teuthology.orchestra.run.smithi092.stdout:osd.6: {} 2024-03-11T21:36:26.119 INFO:teuthology.orchestra.run.smithi092.stdout:osd.7: {} 2024-03-11T21:36:26.119 INFO:teuthology.orchestra.run.smithi092.stderr:osd.0: osd_enable_op_tracker = '' 2024-03-11T21:36:26.119 INFO:teuthology.orchestra.run.smithi092.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-11T21:36:26.119 INFO:teuthology.orchestra.run.smithi092.stderr:osd.2: osd_enable_op_tracker = '' 2024-03-11T21:36:26.119 INFO:teuthology.orchestra.run.smithi092.stderr:Error ENXIO: problem getting command descriptions from osd.3 2024-03-11T21:36:26.119 INFO:teuthology.orchestra.run.smithi092.stderr:osd.4: osd_enable_op_tracker = '' 2024-03-11T21:36:26.120 INFO:teuthology.orchestra.run.smithi092.stderr:osd.5: osd_enable_op_tracker = '' 2024-03-11T21:36:26.120 INFO:teuthology.orchestra.run.smithi092.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-11T21:36:26.120 INFO:teuthology.orchestra.run.smithi092.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-11T21:36:26.127 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:36:26.192 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:36:26.193 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:36:26.193 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:36:26.263 INFO:teuthology.orchestra.run.smithi092.stdout:{ 2024-03-11T21:36:26.263 INFO:teuthology.orchestra.run.smithi092.stdout: "ops": [], 2024-03-11T21:36:26.263 INFO:teuthology.orchestra.run.smithi092.stdout: "complaint_time": 30, 2024-03-11T21:36:26.264 INFO:teuthology.orchestra.run.smithi092.stdout: "num_blocked_ops": 0 2024-03-11T21:36:26.264 INFO:teuthology.orchestra.run.smithi092.stdout:} 2024-03-11T21:36:26.294 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:36:26.325 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:36:25 smithi097 ceph-mon[33317]: from='client.? v1:172.21.15.92:0/3869295930' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-11T21:36:26.340 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:36:26.340 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:36:26.340 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:36:26.441 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:36:26.475 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:36:26.476 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:36:26.476 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:36:26.522 INFO:tasks.radosbench.radosbench.0.smithi092.stdout: 29 16 65449 65433 140.994 92.25 0.00722267 0.00708806 2024-03-11T21:36:26.577 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:36:26.614 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:36:26.615 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:36:26.615 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:36:26.716 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:36:26.754 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:36:26.754 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:36:26.755 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:36:26.856 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:36:26.882 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:36:26.882 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:36:26.883 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:36:26.984 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:36:27.018 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:36:27.018 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:36:27.018 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:36:27.119 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:36:27.158 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:36:27.159 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:36:27.159 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:36:27.260 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:36:27.293 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:36:27.294 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:36:27.294 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:36:27.304 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:36:26 smithi092 ceph-mon[29572]: pgmap v492: 148 pgs: 30 active+undersized+degraded, 32 active+undersized, 86 active+clean; 3.6 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 1.2 KiB/s rd, 127 MiB/s wr, 2.04k op/s; 29401/177108 objects degraded (16.601%) 2024-03-11T21:36:27.304 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:36:26 smithi092 ceph-mon[37563]: pgmap v492: 148 pgs: 30 active+undersized+degraded, 32 active+undersized, 86 active+clean; 3.6 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 1.2 KiB/s rd, 127 MiB/s wr, 2.04k op/s; 29401/177108 objects degraded (16.601%) 2024-03-11T21:36:27.325 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:36:26 smithi097 ceph-mon[33317]: pgmap v492: 148 pgs: 30 active+undersized+degraded, 32 active+undersized, 86 active+clean; 3.6 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 1.2 KiB/s rd, 127 MiB/s wr, 2.04k op/s; 29401/177108 objects degraded (16.601%) 2024-03-11T21:36:27.353 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-03-11T21:36:27.354 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-03-11T21:36:27.395 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:36:27.431 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:36:27.432 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:36:27.432 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:36:27.522 INFO:tasks.radosbench.radosbench.0.smithi092.stdout: 30 16 67063 67047 139.657 100.875 0.00527964 0.00715571 2024-03-11T21:36:27.533 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:36:27.566 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:36:27.567 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:36:27.567 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:36:27.668 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:36:27.704 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:36:27.704 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:36:27.704 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:36:27.805 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:36:27.840 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:36:27.841 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:36:27.841 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:36:27.941 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:36:27.979 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:36:27.980 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:36:27.980 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:36:28.081 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:36:28.117 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:36:28.117 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:36:28.118 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:36:28.124 DEBUG:teuthology.orchestra.run.smithi092:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid 00db0016-dfed-11ee-95c8-87774f69a715 -- ceph daemon osd.0 dump_historic_ops 2024-03-11T21:36:28.219 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:36:28.245 INFO:journalctl@ceph.mgr.y.smithi092.stdout:Mar 11 21:36:27 smithi092 ceph-00db0016-dfed-11ee-95c8-87774f69a715-mgr-y[29801]: ::ffff:172.21.15.97 - - [11/Mar/2024:21:36:27] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-11T21:36:28.253 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:36:28.253 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:36:28.254 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:36:28.355 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:36:28.372 DEBUG:teuthology.orchestra.run.smithi092:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid 00db0016-dfed-11ee-95c8-87774f69a715 -- ceph daemon osd.3 dump_ops_in_flight 2024-03-11T21:36:28.389 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:36:28.389 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:36:28.390 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:36:28.432 INFO:teuthology.orchestra.run.smithi092.stderr:Inferring config /var/lib/ceph/00db0016-dfed-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T21:36:28.490 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:36:28.522 INFO:tasks.radosbench.radosbench.0.smithi092.stdout: 31 16 68630 68614 138.311 97.9375 0.0088017 0.0072247 2024-03-11T21:36:28.543 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:36:28.544 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:36:28.544 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:36:28.586 INFO:journalctl@ceph.osd.3.smithi092.stdout:Mar 11 21:36:28 smithi092 podman[133931]: 2024-03-11 21:36:28.264255683 +0000 UTC m=+3.378753063 container init ea196e969b3c874af0c911ce9d141e04fc33650a47e5e97222467872860add15 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:8fe9635f3beb0247496d7e321f3068bd10c333c433ef4824d7e1c1a7d3d78379, name=ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd-3, io.buildah.version=1.33.5, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240307, org.label-schema.vendor=CentOS, RELEASE=main-2b0101e, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, GIT_CLEAN=True, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image) 2024-03-11T21:36:28.587 INFO:journalctl@ceph.osd.3.smithi092.stdout:Mar 11 21:36:28 smithi092 podman[133931]: 2024-03-11 21:36:28.27271198 +0000 UTC m=+3.387209345 container start ea196e969b3c874af0c911ce9d141e04fc33650a47e5e97222467872860add15 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:8fe9635f3beb0247496d7e321f3068bd10c333c433ef4824d7e1c1a7d3d78379, name=ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd-3, RELEASE=main-2b0101e, org.label-schema.build-date=20240307, CEPH_POINT_RELEASE=, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, GIT_CLEAN=True, org.label-schema.vendor=CentOS) 2024-03-11T21:36:28.587 INFO:journalctl@ceph.osd.3.smithi092.stdout:Mar 11 21:36:28 smithi092 ceph-osd[134300]: -- 172.21.15.92:0/1623826466 <== mon.2 v1:172.21.15.92:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 4011969818 0 0) 0x560eb1dcfd40 con 0x560eb1da0c00 2024-03-11T21:36:28.587 INFO:journalctl@ceph.osd.3.smithi092.stdout:Mar 11 21:36:28 smithi092 ceph-osd[134300]: -- 172.21.15.92:0/1623826466 <== mon.2 v1:172.21.15.92:6790/0 6 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (unknown 1881105576 0 0) 0x560eb1dd7680 con 0x560eb1da0c00 2024-03-11T21:36:28.587 INFO:journalctl@ceph.osd.3.smithi092.stdout:Mar 11 21:36:28 smithi092 bash[133931]: ea196e969b3c874af0c911ce9d141e04fc33650a47e5e97222467872860add15 2024-03-11T21:36:28.587 INFO:journalctl@ceph.osd.3.smithi092.stdout:Mar 11 21:36:28 smithi092 systemd[1]: Started Ceph osd.3 for 00db0016-dfed-11ee-95c8-87774f69a715. 2024-03-11T21:36:28.646 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:36:28.694 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:36:28.695 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:36:28.695 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:36:28.796 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:36:28.829 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:36:28.829 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:36:28.830 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:36:28.859 INFO:journalctl@ceph.osd.3.smithi092.stdout:Mar 11 21:36:28 smithi092 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd-3[134074]: 2024-03-11T21:36:28.605+0000 7f07ed2dd7c0 -1 Falling back to public interface 2024-03-11T21:36:28.931 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:36:28.969 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:36:28.969 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:36:28.970 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:36:29.071 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:36:29.108 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:36:29.109 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:36:29.109 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:36:29.210 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:36:29.244 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:36:29.244 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:36:29.245 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:36:29.250 INFO:teuthology.orchestra.run.smithi092.stderr:Inferring config /var/lib/ceph/00db0016-dfed-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T21:36:29.325 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:36:28 smithi097 ceph-mon[33317]: pgmap v493: 148 pgs: 30 active+undersized+degraded, 32 active+undersized, 86 active+clean; 3.7 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 853 B/s rd, 117 MiB/s wr, 1.88k op/s; 30846/184560 objects degraded (16.713%) 2024-03-11T21:36:29.325 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:36:28 smithi097 ceph-mon[33317]: Health check update: Degraded data redundancy: 30846/184560 objects degraded (16.713%), 30 pgs degraded (PG_DEGRADED) 2024-03-11T21:36:29.346 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:36:29.354 DEBUG:teuthology.orchestra.run.smithi092:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid 00db0016-dfed-11ee-95c8-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-03-11T21:36:29.359 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:36:28 smithi092 ceph-mon[29572]: pgmap v493: 148 pgs: 30 active+undersized+degraded, 32 active+undersized, 86 active+clean; 3.7 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 853 B/s rd, 117 MiB/s wr, 1.88k op/s; 30846/184560 objects degraded (16.713%) 2024-03-11T21:36:29.359 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:36:28 smithi092 ceph-mon[29572]: Health check update: Degraded data redundancy: 30846/184560 objects degraded (16.713%), 30 pgs degraded (PG_DEGRADED) 2024-03-11T21:36:29.359 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:36:28 smithi092 ceph-mon[37563]: pgmap v493: 148 pgs: 30 active+undersized+degraded, 32 active+undersized, 86 active+clean; 3.7 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 853 B/s rd, 117 MiB/s wr, 1.88k op/s; 30846/184560 objects degraded (16.713%) 2024-03-11T21:36:29.359 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:36:28 smithi092 ceph-mon[37563]: Health check update: Degraded data redundancy: 30846/184560 objects degraded (16.713%), 30 pgs degraded (PG_DEGRADED) 2024-03-11T21:36:29.381 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:36:29.382 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:36:29.382 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:36:29.483 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:36:29.523 INFO:tasks.radosbench.radosbench.0.smithi092.stdout: 32 16 70211 70195 137.075 98.8125 0.0145714 0.00729067 2024-03-11T21:36:29.526 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:36:29.526 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:36:29.527 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:36:29.627 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:36:29.677 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:36:29.678 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:36:29.679 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:36:29.779 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:36:29.821 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:36:29.822 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:36:29.822 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:36:29.923 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:36:29.956 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:36:29.957 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:36:29.957 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:36:30.058 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:36:30.098 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:36:30.099 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:36:30.099 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:36:30.190 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:36:29 smithi092 ceph-mon[37563]: from='mgr.14607 v1:172.21.15.92:0/1702717597' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T21:36:30.191 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:36:29 smithi092 ceph-mon[29572]: from='mgr.14607 v1:172.21.15.92:0/1702717597' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T21:36:30.200 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:36:30.241 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:36:30.242 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:36:30.243 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:36:30.325 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:36:29 smithi097 ceph-mon[33317]: from='mgr.14607 v1:172.21.15.92:0/1702717597' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T21:36:30.343 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:36:30.344 INFO:teuthology.orchestra.run.smithi092.stderr:Inferring config /var/lib/ceph/00db0016-dfed-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T21:36:30.380 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:36:30.380 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:36:30.381 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:36:30.482 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:36:30.515 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:36:30.516 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:36:30.516 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:36:30.523 INFO:tasks.radosbench.radosbench.0.smithi092.stdout: 33 16 71547 71531 135.452 83.5 0.00628405 0.00737836 2024-03-11T21:36:30.617 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:36:30.652 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:36:30.653 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:36:30.653 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:36:30.754 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:36:30.791 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:36:30.792 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:36:30.792 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:36:30.893 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:36:30.933 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:36:30.934 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:36:30.934 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:36:31.035 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:36:31.068 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:36:31.069 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:36:31.070 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:36:31.170 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:36:31.204 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:36:31.205 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:36:31.205 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:36:31.216 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:36:30 smithi092 ceph-mon[29572]: pgmap v494: 148 pgs: 30 active+undersized+degraded, 32 active+undersized, 86 active+clean; 4.1 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 1.4 KiB/s rd, 145 MiB/s wr, 2.32k op/s; 33438/201693 objects degraded (16.579%) 2024-03-11T21:36:31.216 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:36:30 smithi092 ceph-mon[29572]: from='mgr.14607 v1:172.21.15.92:0/1702717597' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-11T21:36:31.216 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:36:30 smithi092 ceph-mon[29572]: from='mgr.14607 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-11T21:36:31.216 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:36:30 smithi092 ceph-mon[29572]: from='mgr.14607 v1:172.21.15.92:0/1702717597' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-11T21:36:31.216 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:36:30 smithi092 ceph-mon[29572]: from='mgr.14607 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-11T21:36:31.216 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:36:30 smithi092 ceph-mon[29572]: from='mgr.14607 v1:172.21.15.92:0/1702717597' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-03-11T21:36:31.217 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:36:30 smithi092 ceph-mon[29572]: from='mgr.14607 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-03-11T21:36:31.217 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:36:30 smithi092 ceph-mon[29572]: from='mgr.14607 v1:172.21.15.92:0/1702717597' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-03-11T21:36:31.217 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:36:30 smithi092 ceph-mon[29572]: from='mgr.14607 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-03-11T21:36:31.217 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:36:30 smithi092 ceph-mon[29572]: from='mgr.14607 ' entity='mgr.y' 2024-03-11T21:36:31.217 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:36:30 smithi092 ceph-mon[37563]: pgmap v494: 148 pgs: 30 active+undersized+degraded, 32 active+undersized, 86 active+clean; 4.1 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 1.4 KiB/s rd, 145 MiB/s wr, 2.32k op/s; 33438/201693 objects degraded (16.579%) 2024-03-11T21:36:31.217 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:36:30 smithi092 ceph-mon[37563]: from='mgr.14607 v1:172.21.15.92:0/1702717597' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-11T21:36:31.217 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:36:30 smithi092 ceph-mon[37563]: from='mgr.14607 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-11T21:36:31.217 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:36:30 smithi092 ceph-mon[37563]: from='mgr.14607 v1:172.21.15.92:0/1702717597' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-11T21:36:31.217 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:36:30 smithi092 ceph-mon[37563]: from='mgr.14607 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-11T21:36:31.217 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:36:30 smithi092 ceph-mon[37563]: from='mgr.14607 v1:172.21.15.92:0/1702717597' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-03-11T21:36:31.217 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:36:30 smithi092 ceph-mon[37563]: from='mgr.14607 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-03-11T21:36:31.217 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:36:30 smithi092 ceph-mon[37563]: from='mgr.14607 v1:172.21.15.92:0/1702717597' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-03-11T21:36:31.218 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:36:30 smithi092 ceph-mon[37563]: from='mgr.14607 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-03-11T21:36:31.218 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:36:30 smithi092 ceph-mon[37563]: from='mgr.14607 ' entity='mgr.y' 2024-03-11T21:36:31.306 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:36:31.325 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:36:30 smithi097 ceph-mon[33317]: pgmap v494: 148 pgs: 30 active+undersized+degraded, 32 active+undersized, 86 active+clean; 4.1 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 1.4 KiB/s rd, 145 MiB/s wr, 2.32k op/s; 33438/201693 objects degraded (16.579%) 2024-03-11T21:36:31.326 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:36:30 smithi097 ceph-mon[33317]: from='mgr.14607 v1:172.21.15.92:0/1702717597' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-11T21:36:31.326 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:36:30 smithi097 ceph-mon[33317]: from='mgr.14607 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-11T21:36:31.326 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:36:30 smithi097 ceph-mon[33317]: from='mgr.14607 v1:172.21.15.92:0/1702717597' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-11T21:36:31.326 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:36:30 smithi097 ceph-mon[33317]: from='mgr.14607 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-11T21:36:31.326 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:36:30 smithi097 ceph-mon[33317]: from='mgr.14607 v1:172.21.15.92:0/1702717597' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-03-11T21:36:31.326 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:36:30 smithi097 ceph-mon[33317]: from='mgr.14607 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-03-11T21:36:31.326 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:36:30 smithi097 ceph-mon[33317]: from='mgr.14607 v1:172.21.15.92:0/1702717597' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-03-11T21:36:31.326 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:36:30 smithi097 ceph-mon[33317]: from='mgr.14607 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-03-11T21:36:31.326 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:36:30 smithi097 ceph-mon[33317]: from='mgr.14607 ' entity='mgr.y' 2024-03-11T21:36:31.340 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:36:31.341 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:36:31.341 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:36:31.442 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:36:31.474 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:36:31.475 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:36:31.476 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:36:31.523 INFO:tasks.radosbench.radosbench.0.smithi092.stdout: 34 16 72963 72947 134.071 88.5 0.00882184 0.00745398 2024-03-11T21:36:31.576 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:36:31.614 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:36:31.615 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:36:31.615 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:36:31.716 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:36:31.747 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:36:31.747 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:36:31.748 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:36:31.848 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:36:31.889 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:36:31.890 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:36:31.890 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:36:31.991 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:36:32.012 INFO:teuthology.orchestra.run.smithi092.stderr:nodeep-scrub is set 2024-03-11T21:36:32.035 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:36:32.036 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:36:32.036 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:36:32.137 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:36:32.175 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:36:32.176 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:36:32.176 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:36:32.226 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:36:31 smithi092 ceph-mon[29572]: Adjusting osd_memory_target on smithi097 to 3248M 2024-03-11T21:36:32.226 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:36:31 smithi092 ceph-mon[29572]: from='mgr.14607 v1:172.21.15.92:0/1702717597' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T21:36:32.226 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:36:31 smithi092 ceph-mon[29572]: from='mgr.14607 v1:172.21.15.92:0/1702717597' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T21:36:32.226 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:36:31 smithi092 ceph-mon[29572]: from='mgr.14607 ' entity='mgr.y' 2024-03-11T21:36:32.226 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:36:31 smithi092 ceph-mon[29572]: from='client.? v1:172.21.15.92:0/2642803740' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-11T21:36:32.227 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:36:31 smithi092 ceph-mon[37563]: Adjusting osd_memory_target on smithi097 to 3248M 2024-03-11T21:36:32.227 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:36:31 smithi092 ceph-mon[37563]: from='mgr.14607 v1:172.21.15.92:0/1702717597' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T21:36:32.227 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:36:31 smithi092 ceph-mon[37563]: from='mgr.14607 v1:172.21.15.92:0/1702717597' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T21:36:32.227 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:36:31 smithi092 ceph-mon[37563]: from='mgr.14607 ' entity='mgr.y' 2024-03-11T21:36:32.227 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:36:31 smithi092 ceph-mon[37563]: from='client.? v1:172.21.15.92:0/2642803740' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-11T21:36:32.276 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:36:32.314 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:36:32.315 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:36:32.315 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:36:32.325 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:36:31 smithi097 ceph-mon[33317]: Adjusting osd_memory_target on smithi097 to 3248M 2024-03-11T21:36:32.325 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:36:31 smithi097 ceph-mon[33317]: from='mgr.14607 v1:172.21.15.92:0/1702717597' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T21:36:32.325 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:36:31 smithi097 ceph-mon[33317]: from='mgr.14607 v1:172.21.15.92:0/1702717597' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T21:36:32.325 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:36:31 smithi097 ceph-mon[33317]: from='mgr.14607 ' entity='mgr.y' 2024-03-11T21:36:32.325 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:36:31 smithi097 ceph-mon[33317]: from='client.? v1:172.21.15.92:0/2642803740' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-11T21:36:32.416 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:36:32.450 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:36:32.451 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:36:32.451 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:36:32.523 INFO:tasks.radosbench.radosbench.0.smithi092.stdout: 35 16 74372 74356 132.756 88.0625 0.00929975 0.0075285 2024-03-11T21:36:32.552 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:36:32.560 INFO:journalctl@ceph.osd.3.smithi092.stdout:Mar 11 21:36:32 smithi092 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd-3[134074]: 2024-03-11T21:36:32.242+0000 7f07ed2dd7c0 -1 osd.3 251 log_to_monitors true 2024-03-11T21:36:32.587 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:36:32.587 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:36:32.588 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:36:32.688 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:36:32.723 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:36:32.723 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:36:32.724 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:36:32.824 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:36:32.870 INFO:journalctl@ceph.iscsi.iscsi.a.smithi097.stdout:Mar 11 21:36:32 smithi097 ceph-00db0016-dfed-11ee-95c8-87774f69a715-iscsi-iscsi-a[60943]: debug there is no tcmu-runner data available 2024-03-11T21:36:32.909 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:36:32.910 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:36:32.910 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:36:33.011 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:36:33.048 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:36:33.049 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:36:33.050 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:36:33.151 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:36:33.185 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:36:33.186 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:36:33.187 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:36:33.287 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:36:33.325 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:36:32 smithi097 ceph-mon[33317]: pgmap v495: 148 pgs: 30 active+undersized+degraded, 32 active+undersized, 86 active+clean; 4.1 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 616 B/s rd, 78 MiB/s wr, 1.25k op/s; 33438/201693 objects degraded (16.579%) 2024-03-11T21:36:33.325 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:36:32 smithi097 ceph-mon[33317]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-11T21:36:33.326 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:36:32 smithi097 ceph-mon[33317]: from='client.? v1:172.21.15.92:0/2642803740' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-03-11T21:36:33.326 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:36:32 smithi097 ceph-mon[33317]: osdmap e275: 8 total, 7 up, 8 in 2024-03-11T21:36:33.326 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:36:32 smithi097 ceph-mon[33317]: from='osd.3 v1:172.21.15.92:6813/3215041236' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-03-11T21:36:33.326 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:36:32 smithi097 ceph-mon[33317]: from='osd.3 ' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-03-11T21:36:33.329 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:36:33.330 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:36:33.330 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:36:33.359 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:36:32 smithi092 ceph-mon[29572]: pgmap v495: 148 pgs: 30 active+undersized+degraded, 32 active+undersized, 86 active+clean; 4.1 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 616 B/s rd, 78 MiB/s wr, 1.25k op/s; 33438/201693 objects degraded (16.579%) 2024-03-11T21:36:33.359 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:36:32 smithi092 ceph-mon[29572]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-11T21:36:33.359 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:36:32 smithi092 ceph-mon[29572]: from='client.? v1:172.21.15.92:0/2642803740' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-03-11T21:36:33.360 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:36:32 smithi092 ceph-mon[29572]: osdmap e275: 8 total, 7 up, 8 in 2024-03-11T21:36:33.360 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:36:32 smithi092 ceph-mon[29572]: from='osd.3 v1:172.21.15.92:6813/3215041236' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-03-11T21:36:33.360 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:36:32 smithi092 ceph-mon[29572]: from='osd.3 ' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-03-11T21:36:33.360 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:36:32 smithi092 ceph-mon[37563]: pgmap v495: 148 pgs: 30 active+undersized+degraded, 32 active+undersized, 86 active+clean; 4.1 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 616 B/s rd, 78 MiB/s wr, 1.25k op/s; 33438/201693 objects degraded (16.579%) 2024-03-11T21:36:33.360 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:36:32 smithi092 ceph-mon[37563]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-11T21:36:33.360 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:36:32 smithi092 ceph-mon[37563]: from='client.? v1:172.21.15.92:0/2642803740' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-03-11T21:36:33.360 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:36:32 smithi092 ceph-mon[37563]: osdmap e275: 8 total, 7 up, 8 in 2024-03-11T21:36:33.360 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:36:32 smithi092 ceph-mon[37563]: from='osd.3 v1:172.21.15.92:6813/3215041236' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-03-11T21:36:33.360 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:36:32 smithi092 ceph-mon[37563]: from='osd.3 ' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-03-11T21:36:33.361 INFO:journalctl@ceph.osd.3.smithi092.stdout:Mar 11 21:36:33 smithi092 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd-3[134074]: 2024-03-11T21:36:33.073+0000 7f07da72d700 -1 osd.3 251 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-11T21:36:33.431 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:36:33.468 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:36:33.469 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:36:33.469 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:36:33.523 INFO:tasks.radosbench.radosbench.0.smithi092.stdout: 36 16 75652 75636 131.29 80 0.00445791 0.00761268 2024-03-11T21:36:33.570 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:36:33.605 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:36:33.606 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:36:33.606 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:36:33.707 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:36:33.742 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:36:33.743 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:36:33.743 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:36:33.844 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:36:33.878 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:36:33.878 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:36:33.879 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:36:33.979 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:36:34.004 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:36:34.004 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:36:34.005 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:36:34.105 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:36:34.130 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:36:34.130 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:36:34.131 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:36:34.232 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:36:34.256 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:36:34.256 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:36:34.256 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:36:34.325 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:36:34 smithi097 ceph-mon[33317]: from='client.24469 v1:172.21.15.97:0/404208310' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:36:34.326 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:36:34 smithi097 ceph-mon[33317]: from='osd.3 ' entity='osd.3' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]': finished 2024-03-11T21:36:34.326 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:36:34 smithi097 ceph-mon[33317]: from='osd.3 v1:172.21.15.92:6813/3215041236' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi092", "root=default"]}]: dispatch 2024-03-11T21:36:34.326 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:36:34 smithi097 ceph-mon[33317]: osdmap e276: 8 total, 7 up, 8 in 2024-03-11T21:36:34.326 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:36:34 smithi097 ceph-mon[33317]: from='osd.3 ' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi092", "root=default"]}]: dispatch 2024-03-11T21:36:34.326 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:36:34 smithi097 ceph-mon[33317]: Health check update: Degraded data redundancy: 33438/201693 objects degraded (16.579%), 30 pgs degraded, 54 pgs undersized (PG_DEGRADED) 2024-03-11T21:36:34.358 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:36:34.359 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:36:34 smithi092 ceph-mon[29572]: from='client.24469 v1:172.21.15.97:0/404208310' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:36:34.359 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:36:34 smithi092 ceph-mon[29572]: from='osd.3 ' entity='osd.3' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]': finished 2024-03-11T21:36:34.359 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:36:34 smithi092 ceph-mon[29572]: from='osd.3 v1:172.21.15.92:6813/3215041236' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi092", "root=default"]}]: dispatch 2024-03-11T21:36:34.360 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:36:34 smithi092 ceph-mon[29572]: osdmap e276: 8 total, 7 up, 8 in 2024-03-11T21:36:34.360 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:36:34 smithi092 ceph-mon[29572]: from='osd.3 ' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi092", "root=default"]}]: dispatch 2024-03-11T21:36:34.360 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:36:34 smithi092 ceph-mon[29572]: Health check update: Degraded data redundancy: 33438/201693 objects degraded (16.579%), 30 pgs degraded, 54 pgs undersized (PG_DEGRADED) 2024-03-11T21:36:34.360 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:36:34 smithi092 ceph-mon[37563]: from='client.24469 v1:172.21.15.97:0/404208310' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:36:34.360 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:36:34 smithi092 ceph-mon[37563]: from='osd.3 ' entity='osd.3' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]': finished 2024-03-11T21:36:34.360 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:36:34 smithi092 ceph-mon[37563]: from='osd.3 v1:172.21.15.92:6813/3215041236' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi092", "root=default"]}]: dispatch 2024-03-11T21:36:34.360 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:36:34 smithi092 ceph-mon[37563]: osdmap e276: 8 total, 7 up, 8 in 2024-03-11T21:36:34.360 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:36:34 smithi092 ceph-mon[37563]: from='osd.3 ' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi092", "root=default"]}]: dispatch 2024-03-11T21:36:34.360 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:36:34 smithi092 ceph-mon[37563]: Health check update: Degraded data redundancy: 33438/201693 objects degraded (16.579%), 30 pgs degraded, 54 pgs undersized (PG_DEGRADED) 2024-03-11T21:36:34.382 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:36:34.383 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:36:34.383 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:36:34.484 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:36:34.509 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:36:34.509 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:36:34.510 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:36:34.523 INFO:tasks.radosbench.radosbench.0.smithi092.stdout: 37 16 76308 76292 128.85 41 0.0041213 0.00765512 2024-03-11T21:36:34.611 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:36:34.638 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:36:34.639 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:36:34.639 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:36:34.740 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:36:34.765 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:36:34.765 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:36:34.766 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:36:34.866 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:36:34.893 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:36:34.894 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:36:34.895 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:36:34.995 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:36:35.029 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:36:35.029 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:36:35.029 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:36:35.130 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:36:35.155 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:36:35.156 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:36:35.156 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:36:35.257 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:36:35.284 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:36:35.285 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:36:35.285 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:36:35.325 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:36:35 smithi097 ceph-mon[33317]: pgmap v498: 148 pgs: 30 active+undersized+degraded, 32 active+undersized, 86 active+clean; 4.4 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 1.2 KiB/s rd, 106 MiB/s wr, 1.70k op/s; 36593/217938 objects degraded (16.791%) 2024-03-11T21:36:35.326 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:36:35 smithi097 ceph-mon[33317]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-03-11T21:36:35.326 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:36:35 smithi097 ceph-mon[33317]: osd.3 v1:172.21.15.92:6813/3215041236 boot 2024-03-11T21:36:35.326 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:36:35 smithi097 ceph-mon[33317]: osdmap e277: 8 total, 8 up, 8 in 2024-03-11T21:36:35.326 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:36:35 smithi097 ceph-mon[33317]: from='mgr.14607 v1:172.21.15.92:0/1702717597' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-11T21:36:35.326 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:36:35 smithi097 ceph-mon[33317]: osdmap e278: 8 total, 8 up, 8 in 2024-03-11T21:36:35.359 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:36:35 smithi092 ceph-mon[29572]: pgmap v498: 148 pgs: 30 active+undersized+degraded, 32 active+undersized, 86 active+clean; 4.4 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 1.2 KiB/s rd, 106 MiB/s wr, 1.70k op/s; 36593/217938 objects degraded (16.791%) 2024-03-11T21:36:35.360 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:36:35 smithi092 ceph-mon[29572]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-03-11T21:36:35.360 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:36:35 smithi092 ceph-mon[29572]: osd.3 v1:172.21.15.92:6813/3215041236 boot 2024-03-11T21:36:35.360 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:36:35 smithi092 ceph-mon[29572]: osdmap e277: 8 total, 8 up, 8 in 2024-03-11T21:36:35.360 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:36:35 smithi092 ceph-mon[29572]: from='mgr.14607 v1:172.21.15.92:0/1702717597' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-11T21:36:35.360 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:36:35 smithi092 ceph-mon[29572]: osdmap e278: 8 total, 8 up, 8 in 2024-03-11T21:36:35.360 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:36:35 smithi092 ceph-mon[37563]: pgmap v498: 148 pgs: 30 active+undersized+degraded, 32 active+undersized, 86 active+clean; 4.4 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 1.2 KiB/s rd, 106 MiB/s wr, 1.70k op/s; 36593/217938 objects degraded (16.791%) 2024-03-11T21:36:35.360 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:36:35 smithi092 ceph-mon[37563]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-03-11T21:36:35.360 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:36:35 smithi092 ceph-mon[37563]: osd.3 v1:172.21.15.92:6813/3215041236 boot 2024-03-11T21:36:35.360 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:36:35 smithi092 ceph-mon[37563]: osdmap e277: 8 total, 8 up, 8 in 2024-03-11T21:36:35.360 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:36:35 smithi092 ceph-mon[37563]: from='mgr.14607 v1:172.21.15.92:0/1702717597' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-11T21:36:35.360 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:36:35 smithi092 ceph-mon[37563]: osdmap e278: 8 total, 8 up, 8 in 2024-03-11T21:36:35.386 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:36:35.413 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:36:35.414 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:36:35.414 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:36:35.515 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:36:35.523 INFO:tasks.radosbench.radosbench.0.smithi092.stdout: 38 16 76308 76292 125.459 0 - 0.00765512 2024-03-11T21:36:35.541 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:36:35.542 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:36:35.542 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:36:35.643 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:36:35.671 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:36:35.672 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:36:35.672 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:36:35.773 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:36:35.801 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:36:35.802 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:36:35.802 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:36:35.837 INFO:teuthology.orchestra.run.smithi092.stderr:Inferring config /var/lib/ceph/00db0016-dfed-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T21:36:35.903 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:36:35.928 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:36:35.929 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:36:35.929 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:36:36.030 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:36:36.067 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:36:36.068 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:36:36.068 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:36:36.169 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:36:36.198 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:36:36.199 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:36:36.199 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:36:36.300 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:36:36.325 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:36:36.326 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:36:36.326 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:36:36.427 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:36:36.463 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:36:36.464 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:36:36.465 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:36:36.523 INFO:tasks.radosbench.radosbench.0.smithi092.stdout: 39 16 76323 76307 122.267 0.46875 2.50397 0.00797818 2024-03-11T21:36:36.565 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:36:36.597 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:36:36.598 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:36:36.598 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:36:36.699 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:36:36.738 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:36:36.739 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:36:36.739 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:36:36.840 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:36:36.869 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:36:36.870 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:36:36.870 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:36:36.902 INFO:teuthology.orchestra.run.smithi092.stdout:{ 2024-03-11T21:36:36.902 INFO:teuthology.orchestra.run.smithi092.stdout: "size": 20, 2024-03-11T21:36:36.902 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 600, 2024-03-11T21:36:36.902 INFO:teuthology.orchestra.run.smithi092.stdout: "ops": [ 2024-03-11T21:36:36.902 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:36:36.902 INFO:teuthology.orchestra.run.smithi092.stdout: "description": "osd_op(client.34715.0:156527 7.9 7:9b920c90:::benchmark_data_smithi092_76040_object34616:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e126)", 2024-03-11T21:36:36.902 INFO:teuthology.orchestra.run.smithi092.stdout: "initiated_at": "2024-03-11T21:29:57.080353+0000", 2024-03-11T21:36:36.903 INFO:teuthology.orchestra.run.smithi092.stdout: "age": 399.83577687600001, 2024-03-11T21:36:36.903 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.27682804, 2024-03-11T21:36:36.903 INFO:teuthology.orchestra.run.smithi092.stdout: "type_data": { 2024-03-11T21:36:36.903 INFO:teuthology.orchestra.run.smithi092.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-11T21:36:36.903 INFO:teuthology.orchestra.run.smithi092.stdout: "client_info": { 2024-03-11T21:36:36.903 INFO:teuthology.orchestra.run.smithi092.stdout: "client": "client.34715", 2024-03-11T21:36:36.903 INFO:teuthology.orchestra.run.smithi092.stdout: "client_addr": "v1:172.21.15.92:0/2403132573", 2024-03-11T21:36:36.903 INFO:teuthology.orchestra.run.smithi092.stdout: "tid": 156527 2024-03-11T21:36:36.903 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:36:36.903 INFO:teuthology.orchestra.run.smithi092.stdout: "events": [ 2024-03-11T21:36:36.903 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:36:36.903 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "initiated", 2024-03-11T21:36:36.904 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:29:57.080353+0000", 2024-03-11T21:36:36.904 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-03-11T21:36:36.904 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:36:36.904 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:36:36.904 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "header_read", 2024-03-11T21:36:36.904 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:29:57.080353+0000", 2024-03-11T21:36:36.904 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-03-11T21:36:36.904 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:36:36.904 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:36:36.904 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "throttled", 2024-03-11T21:36:36.904 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:29:57.080355+0000", 2024-03-11T21:36:36.904 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.891e-06 2024-03-11T21:36:36.905 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:36:36.905 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:36:36.905 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "all_read", 2024-03-11T21:36:36.905 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:29:57.080361+0000", 2024-03-11T21:36:36.905 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 5.6339999999999998e-06 2024-03-11T21:36:36.905 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:36:36.905 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:36:36.905 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "dispatched", 2024-03-11T21:36:36.905 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:29:57.080372+0000", 2024-03-11T21:36:36.905 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.1574e-05 2024-03-11T21:36:36.905 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:36:36.905 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:36:36.906 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "queued_for_pg", 2024-03-11T21:36:36.906 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:29:57.080381+0000", 2024-03-11T21:36:36.906 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 8.2819999999999996e-06 2024-03-11T21:36:36.906 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:36:36.906 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:36:36.906 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-03-11T21:36:36.906 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:29:57.191188+0000", 2024-03-11T21:36:36.906 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.110807313 2024-03-11T21:36:36.906 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:36:36.907 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:36:36.907 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "started", 2024-03-11T21:36:36.907 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:29:57.191296+0000", 2024-03-11T21:36:36.907 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.000108043 2024-03-11T21:36:36.907 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:36:36.907 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:36:36.907 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "waiting for subops from 1,4", 2024-03-11T21:36:36.907 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:29:57.191404+0000", 2024-03-11T21:36:36.907 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.000108292 2024-03-11T21:36:36.907 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:36:36.907 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:36:36.907 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "sub_op_commit_rec", 2024-03-11T21:36:36.907 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:29:57.199597+0000", 2024-03-11T21:36:36.907 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.0081928209999999994 2024-03-11T21:36:36.907 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:36:36.908 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:36:36.908 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "sub_op_commit_rec", 2024-03-11T21:36:36.908 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:29:57.231100+0000", 2024-03-11T21:36:36.908 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.031503124 2024-03-11T21:36:36.908 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:36:36.908 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:36:36.908 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "op_commit", 2024-03-11T21:36:36.908 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:29:57.357067+0000", 2024-03-11T21:36:36.908 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.125966465 2024-03-11T21:36:36.908 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:36:36.908 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:36:36.908 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "commit_sent", 2024-03-11T21:36:36.908 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:29:57.357118+0000", 2024-03-11T21:36:36.908 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 5.1801000000000001e-05 2024-03-11T21:36:36.909 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:36:36.909 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:36:36.909 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "done", 2024-03-11T21:36:36.909 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:29:57.357181+0000", 2024-03-11T21:36:36.909 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 6.2799999999999995e-05 2024-03-11T21:36:36.909 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-03-11T21:36:36.909 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-03-11T21:36:36.909 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-03-11T21:36:36.909 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:36:36.909 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:36:36.909 INFO:teuthology.orchestra.run.smithi092.stdout: "description": "osd_op(client.34715.0:157074 7.9 7:9fc27db5:::benchmark_data_smithi092_76040_object35163:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e126)", 2024-03-11T21:36:36.909 INFO:teuthology.orchestra.run.smithi092.stdout: "initiated_at": "2024-03-11T21:29:58.058448+0000", 2024-03-11T21:36:36.909 INFO:teuthology.orchestra.run.smithi092.stdout: "age": 398.85768168099997, 2024-03-11T21:36:36.910 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.245258646, 2024-03-11T21:36:36.910 INFO:teuthology.orchestra.run.smithi092.stdout: "type_data": { 2024-03-11T21:36:36.910 INFO:teuthology.orchestra.run.smithi092.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-11T21:36:36.910 INFO:teuthology.orchestra.run.smithi092.stdout: "client_info": { 2024-03-11T21:36:36.910 INFO:teuthology.orchestra.run.smithi092.stdout: "client": "client.34715", 2024-03-11T21:36:36.910 INFO:teuthology.orchestra.run.smithi092.stdout: "client_addr": "v1:172.21.15.92:0/2403132573", 2024-03-11T21:36:36.910 INFO:teuthology.orchestra.run.smithi092.stdout: "tid": 157074 2024-03-11T21:36:36.910 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:36:36.910 INFO:teuthology.orchestra.run.smithi092.stdout: "events": [ 2024-03-11T21:36:36.910 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:36:36.910 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "initiated", 2024-03-11T21:36:36.910 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:29:58.058448+0000", 2024-03-11T21:36:36.910 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-03-11T21:36:36.910 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:36:36.911 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:36:36.911 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "header_read", 2024-03-11T21:36:36.911 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:29:58.058448+0000", 2024-03-11T21:36:36.911 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-03-11T21:36:36.911 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:36:36.911 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:36:36.911 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "throttled", 2024-03-11T21:36:36.911 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:29:58.058450+0000", 2024-03-11T21:36:36.911 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.5510000000000001e-06 2024-03-11T21:36:36.911 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:36:36.911 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:36:36.911 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "all_read", 2024-03-11T21:36:36.911 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:29:58.058455+0000", 2024-03-11T21:36:36.911 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 5.2530000000000004e-06 2024-03-11T21:36:36.912 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:36:36.912 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:36:36.912 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "dispatched", 2024-03-11T21:36:36.912 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:29:58.058467+0000", 2024-03-11T21:36:36.912 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.1576000000000001e-05 2024-03-11T21:36:36.912 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:36:36.912 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:36:36.912 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "queued_for_pg", 2024-03-11T21:36:36.912 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:29:58.058475+0000", 2024-03-11T21:36:36.912 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 8.2190000000000004e-06 2024-03-11T21:36:36.912 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:36:36.912 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:36:36.912 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-03-11T21:36:36.913 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:29:58.062438+0000", 2024-03-11T21:36:36.913 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.0039631179999999998 2024-03-11T21:36:36.913 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:36:36.913 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:36:36.913 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "started", 2024-03-11T21:36:36.913 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:29:58.062563+0000", 2024-03-11T21:36:36.913 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00012468700000000001 2024-03-11T21:36:36.913 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:36:36.913 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:36:36.913 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "waiting for subops from 1,4", 2024-03-11T21:36:36.913 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:29:58.062681+0000", 2024-03-11T21:36:36.913 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00011832000000000001 2024-03-11T21:36:36.913 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:36:36.913 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:36:36.914 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "sub_op_commit_rec", 2024-03-11T21:36:36.914 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:29:58.070382+0000", 2024-03-11T21:36:36.914 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.0077010239999999999 2024-03-11T21:36:36.914 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:36:36.914 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:36:36.914 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "op_commit", 2024-03-11T21:36:36.914 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:29:58.071367+0000", 2024-03-11T21:36:36.914 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.0009846290000000001 2024-03-11T21:36:36.914 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:36:36.914 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:36:36.914 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "sub_op_commit_rec", 2024-03-11T21:36:36.914 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:29:58.303597+0000", 2024-03-11T21:36:36.914 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.23222996000000001 2024-03-11T21:36:36.914 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:36:36.915 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:36:36.915 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "commit_sent", 2024-03-11T21:36:36.915 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:29:58.303650+0000", 2024-03-11T21:36:36.915 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 5.3690000000000003e-05 2024-03-11T21:36:36.915 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:36:36.915 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:36:36.915 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "done", 2024-03-11T21:36:36.915 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:29:58.303707+0000", 2024-03-11T21:36:36.915 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 5.6619000000000001e-05 2024-03-11T21:36:36.915 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-03-11T21:36:36.915 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-03-11T21:36:36.915 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-03-11T21:36:36.915 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:36:36.915 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:36:36.916 INFO:teuthology.orchestra.run.smithi092.stdout: "description": "osd_op(client.15150.0:207121 8.c 8:3b538f8b:::benchmark_data_smithi092_104865_object86620:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e237)", 2024-03-11T21:36:36.916 INFO:teuthology.orchestra.run.smithi092.stdout: "initiated_at": "2024-03-11T21:34:55.373083+0000", 2024-03-11T21:36:36.916 INFO:teuthology.orchestra.run.smithi092.stdout: "age": 101.543046723, 2024-03-11T21:36:36.916 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.98710652499999996, 2024-03-11T21:36:36.916 INFO:teuthology.orchestra.run.smithi092.stdout: "type_data": { 2024-03-11T21:36:36.916 INFO:teuthology.orchestra.run.smithi092.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-11T21:36:36.916 INFO:teuthology.orchestra.run.smithi092.stdout: "client_info": { 2024-03-11T21:36:36.916 INFO:teuthology.orchestra.run.smithi092.stdout: "client": "client.15150", 2024-03-11T21:36:36.916 INFO:teuthology.orchestra.run.smithi092.stdout: "client_addr": "v1:172.21.15.92:0/436606717", 2024-03-11T21:36:36.916 INFO:teuthology.orchestra.run.smithi092.stdout: "tid": 207121 2024-03-11T21:36:36.916 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:36:36.916 INFO:teuthology.orchestra.run.smithi092.stdout: "events": [ 2024-03-11T21:36:36.916 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:36:36.916 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "initiated", 2024-03-11T21:36:36.917 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:34:55.373083+0000", 2024-03-11T21:36:36.917 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-03-11T21:36:36.917 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:36:36.918 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:36:36.918 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "header_read", 2024-03-11T21:36:36.918 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:34:55.373083+0000", 2024-03-11T21:36:36.918 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-03-11T21:36:36.918 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:36:36.918 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:36:36.918 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "throttled", 2024-03-11T21:36:36.918 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:34:55.373085+0000", 2024-03-11T21:36:36.918 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.206e-06 2024-03-11T21:36:36.918 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:36:36.918 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:36:36.918 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "all_read", 2024-03-11T21:36:36.919 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:34:55.373090+0000", 2024-03-11T21:36:36.919 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 5.6760000000000004e-06 2024-03-11T21:36:36.919 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:36:36.919 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:36:36.919 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "dispatched", 2024-03-11T21:36:36.919 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:34:55.373107+0000", 2024-03-11T21:36:36.919 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.6362999999999999e-05 2024-03-11T21:36:36.919 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:36:36.919 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:36:36.919 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "queued_for_pg", 2024-03-11T21:36:36.919 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:34:55.373115+0000", 2024-03-11T21:36:36.919 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 8.2660000000000001e-06 2024-03-11T21:36:36.919 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:36:36.920 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:36:36.920 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-03-11T21:36:36.920 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:34:55.373173+0000", 2024-03-11T21:36:36.920 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 5.7991000000000001e-05 2024-03-11T21:36:36.920 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:36:36.920 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:36:36.920 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "waiting for peered", 2024-03-11T21:36:36.920 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:34:55.373182+0000", 2024-03-11T21:36:36.920 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 9.0049999999999993e-06 2024-03-11T21:36:36.920 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:36:36.920 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:36:36.920 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-03-11T21:36:36.920 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:34:56.354869+0000", 2024-03-11T21:36:36.920 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.98168716199999995 2024-03-11T21:36:36.921 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:36:36.921 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:36:36.921 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "started", 2024-03-11T21:36:36.921 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:34:56.355005+0000", 2024-03-11T21:36:36.921 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00013610799999999999 2024-03-11T21:36:36.921 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:36:36.921 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:36:36.921 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "waiting for subops from 3,7", 2024-03-11T21:36:36.921 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:34:56.355119+0000", 2024-03-11T21:36:36.921 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00011411 2024-03-11T21:36:36.921 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:36:36.921 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:36:36.921 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "sub_op_commit_rec", 2024-03-11T21:36:36.921 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:34:56.357333+0000", 2024-03-11T21:36:36.922 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.0022135430000000001 2024-03-11T21:36:36.922 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:36:36.922 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:36:36.922 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "op_commit", 2024-03-11T21:36:36.922 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:34:56.358863+0000", 2024-03-11T21:36:36.922 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.001530399 2024-03-11T21:36:36.922 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:36:36.922 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:36:36.922 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "sub_op_commit_rec", 2024-03-11T21:36:36.922 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:34:56.360128+0000", 2024-03-11T21:36:36.922 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.001264409 2024-03-11T21:36:36.922 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:36:36.922 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:36:36.922 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "commit_sent", 2024-03-11T21:36:36.923 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:34:56.360162+0000", 2024-03-11T21:36:36.923 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 3.4585999999999997e-05 2024-03-11T21:36:36.923 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:36:36.923 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:36:36.923 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "done", 2024-03-11T21:36:36.923 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:34:56.360190+0000", 2024-03-11T21:36:36.923 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 2.7701000000000001e-05 2024-03-11T21:36:36.923 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-03-11T21:36:36.923 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-03-11T21:36:36.923 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-03-11T21:36:36.923 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:36:36.923 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:36:36.923 INFO:teuthology.orchestra.run.smithi092.stdout: "description": "osd_op(client.15150.0:207122 8.c 8:3cecc2cc:::benchmark_data_smithi092_104865_object86621:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e237)", 2024-03-11T21:36:36.924 INFO:teuthology.orchestra.run.smithi092.stdout: "initiated_at": "2024-03-11T21:34:55.374081+0000", 2024-03-11T21:36:36.924 INFO:teuthology.orchestra.run.smithi092.stdout: "age": 101.542049316, 2024-03-11T21:36:36.924 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.98636186100000001, 2024-03-11T21:36:36.924 INFO:teuthology.orchestra.run.smithi092.stdout: "type_data": { 2024-03-11T21:36:36.924 INFO:teuthology.orchestra.run.smithi092.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-11T21:36:36.924 INFO:teuthology.orchestra.run.smithi092.stdout: "client_info": { 2024-03-11T21:36:36.924 INFO:teuthology.orchestra.run.smithi092.stdout: "client": "client.15150", 2024-03-11T21:36:36.924 INFO:teuthology.orchestra.run.smithi092.stdout: "client_addr": "v1:172.21.15.92:0/436606717", 2024-03-11T21:36:36.924 INFO:teuthology.orchestra.run.smithi092.stdout: "tid": 207122 2024-03-11T21:36:36.924 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:36:36.924 INFO:teuthology.orchestra.run.smithi092.stdout: "events": [ 2024-03-11T21:36:36.924 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:36:36.924 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "initiated", 2024-03-11T21:36:36.925 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:34:55.374081+0000", 2024-03-11T21:36:36.925 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-03-11T21:36:36.925 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:36:36.925 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:36:36.925 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "header_read", 2024-03-11T21:36:36.925 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:34:55.374081+0000", 2024-03-11T21:36:36.925 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-03-11T21:36:36.925 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:36:36.925 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:36:36.925 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "throttled", 2024-03-11T21:36:36.925 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:34:55.374082+0000", 2024-03-11T21:36:36.925 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.6020000000000001e-06 2024-03-11T21:36:36.925 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:36:36.925 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:36:36.926 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "all_read", 2024-03-11T21:36:36.926 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:34:55.374088+0000", 2024-03-11T21:36:36.926 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 5.4360000000000001e-06 2024-03-11T21:36:36.926 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:36:36.926 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:36:36.926 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "dispatched", 2024-03-11T21:36:36.926 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:34:55.374102+0000", 2024-03-11T21:36:36.926 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.4222e-05 2024-03-11T21:36:36.926 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:36:36.926 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:36:36.926 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "queued_for_pg", 2024-03-11T21:36:36.926 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:34:55.374110+0000", 2024-03-11T21:36:36.926 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 8.1200000000000002e-06 2024-03-11T21:36:36.926 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:36:36.927 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:36:36.927 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-03-11T21:36:36.927 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:34:55.374167+0000", 2024-03-11T21:36:36.927 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 5.6768999999999998e-05 2024-03-11T21:36:36.927 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:36:36.927 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:36:36.927 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "waiting for peered", 2024-03-11T21:36:36.927 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:34:55.374178+0000", 2024-03-11T21:36:36.927 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.0730000000000001e-05 2024-03-11T21:36:36.927 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:36:36.927 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:36:36.927 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-03-11T21:36:36.927 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:34:56.357391+0000", 2024-03-11T21:36:36.927 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.98321307000000002 2024-03-11T21:36:36.928 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:36:36.928 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:36:36.928 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "started", 2024-03-11T21:36:36.928 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:34:56.357504+0000", 2024-03-11T21:36:36.928 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.000113727 2024-03-11T21:36:36.928 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:36:36.928 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:36:36.928 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "waiting for subops from 3,7", 2024-03-11T21:36:36.928 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:34:56.357573+0000", 2024-03-11T21:36:36.928 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 6.8965000000000001e-05 2024-03-11T21:36:36.928 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:36:36.928 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:36:36.928 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "sub_op_commit_rec", 2024-03-11T21:36:36.928 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:34:56.360278+0000", 2024-03-11T21:36:36.929 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.0027041729999999998 2024-03-11T21:36:36.929 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:36:36.929 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:36:36.929 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "op_commit", 2024-03-11T21:36:36.929 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:34:56.360289+0000", 2024-03-11T21:36:36.929 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.1141000000000001e-05 2024-03-11T21:36:36.929 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:36:36.929 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:36:36.929 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "sub_op_commit_rec", 2024-03-11T21:36:36.929 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:34:56.360324+0000", 2024-03-11T21:36:36.929 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 3.5330999999999998e-05 2024-03-11T21:36:36.929 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:36:36.929 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:36:36.929 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "commit_sent", 2024-03-11T21:36:36.930 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:34:56.360404+0000", 2024-03-11T21:36:36.930 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 8.0290999999999994e-05 2024-03-11T21:36:36.930 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:36:36.930 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:36:36.930 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "done", 2024-03-11T21:36:36.930 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:34:56.360443+0000", 2024-03-11T21:36:36.930 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 3.8284000000000001e-05 2024-03-11T21:36:36.930 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-03-11T21:36:36.930 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-03-11T21:36:36.930 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-03-11T21:36:36.930 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:36:36.930 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:36:36.930 INFO:teuthology.orchestra.run.smithi092.stdout: "description": "osd_op(client.15150.0:207130 8.c 8:3e52addc:::benchmark_data_smithi092_104865_object86629:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e237)", 2024-03-11T21:36:36.931 INFO:teuthology.orchestra.run.smithi092.stdout: "initiated_at": "2024-03-11T21:34:55.382721+0000", 2024-03-11T21:36:36.931 INFO:teuthology.orchestra.run.smithi092.stdout: "age": 101.53340878500001, 2024-03-11T21:36:36.931 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.97792294199999996, 2024-03-11T21:36:36.931 INFO:teuthology.orchestra.run.smithi092.stdout: "type_data": { 2024-03-11T21:36:36.931 INFO:teuthology.orchestra.run.smithi092.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-11T21:36:36.931 INFO:teuthology.orchestra.run.smithi092.stdout: "client_info": { 2024-03-11T21:36:36.931 INFO:teuthology.orchestra.run.smithi092.stdout: "client": "client.15150", 2024-03-11T21:36:36.931 INFO:teuthology.orchestra.run.smithi092.stdout: "client_addr": "v1:172.21.15.92:0/436606717", 2024-03-11T21:36:36.931 INFO:teuthology.orchestra.run.smithi092.stdout: "tid": 207130 2024-03-11T21:36:36.931 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:36:36.931 INFO:teuthology.orchestra.run.smithi092.stdout: "events": [ 2024-03-11T21:36:36.931 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:36:36.931 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "initiated", 2024-03-11T21:36:36.931 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:34:55.382721+0000", 2024-03-11T21:36:36.931 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-03-11T21:36:36.932 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:36:36.932 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:36:36.932 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "header_read", 2024-03-11T21:36:36.932 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:34:55.382721+0000", 2024-03-11T21:36:36.932 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-03-11T21:36:36.932 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:36:36.932 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:36:36.932 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "throttled", 2024-03-11T21:36:36.932 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:34:55.382723+0000", 2024-03-11T21:36:36.932 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 2.1160000000000002e-06 2024-03-11T21:36:36.932 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:36:36.932 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:36:36.932 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "all_read", 2024-03-11T21:36:36.932 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:34:55.382729+0000", 2024-03-11T21:36:36.932 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 5.9240000000000004e-06 2024-03-11T21:36:36.933 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:36:36.933 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:36:36.933 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "dispatched", 2024-03-11T21:36:36.933 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:34:55.382742+0000", 2024-03-11T21:36:36.933 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.3161e-05 2024-03-11T21:36:36.933 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:36:36.933 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:36:36.933 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "queued_for_pg", 2024-03-11T21:36:36.933 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:34:55.382749+0000", 2024-03-11T21:36:36.933 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 6.9689999999999997e-06 2024-03-11T21:36:36.933 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:36:36.933 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:36:36.933 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-03-11T21:36:36.933 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:34:55.382797+0000", 2024-03-11T21:36:36.933 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 4.7277999999999997e-05 2024-03-11T21:36:36.934 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:36:36.934 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:36:36.934 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "waiting for peered", 2024-03-11T21:36:36.934 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:34:55.382804+0000", 2024-03-11T21:36:36.934 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 7.4800000000000004e-06 2024-03-11T21:36:36.934 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:36:36.934 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:36:36.934 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-03-11T21:36:36.934 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:34:56.358897+0000", 2024-03-11T21:36:36.934 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.97609253399999996 2024-03-11T21:36:36.934 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:36:36.934 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:36:36.934 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "started", 2024-03-11T21:36:36.935 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:34:56.358972+0000", 2024-03-11T21:36:36.935 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 7.4806000000000002e-05 2024-03-11T21:36:36.935 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:36:36.935 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:36:36.935 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "waiting for subops from 3,7", 2024-03-11T21:36:36.935 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:34:56.359055+0000", 2024-03-11T21:36:36.935 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 8.3790999999999998e-05 2024-03-11T21:36:36.935 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:36:36.935 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:36:36.935 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "op_commit", 2024-03-11T21:36:36.935 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:34:56.360293+0000", 2024-03-11T21:36:36.935 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.0012380970000000001 2024-03-11T21:36:36.935 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:36:36.935 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:36:36.935 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "sub_op_commit_rec", 2024-03-11T21:36:36.936 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:34:56.360502+0000", 2024-03-11T21:36:36.936 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.000208858 2024-03-11T21:36:36.936 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:36:36.936 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:36:36.936 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "sub_op_commit_rec", 2024-03-11T21:36:36.936 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:34:56.360563+0000", 2024-03-11T21:36:36.936 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 6.1173000000000005e-05 2024-03-11T21:36:36.936 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:36:36.936 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:36:36.936 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "commit_sent", 2024-03-11T21:36:36.936 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:34:56.360608+0000", 2024-03-11T21:36:36.936 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 4.4193999999999997e-05 2024-03-11T21:36:36.936 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:36:36.936 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:36:36.936 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "done", 2024-03-11T21:36:36.936 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:34:56.360644+0000", 2024-03-11T21:36:36.937 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 3.6560999999999998e-05 2024-03-11T21:36:36.937 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-03-11T21:36:36.937 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-03-11T21:36:36.937 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-03-11T21:36:36.937 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:36:36.937 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:36:36.937 INFO:teuthology.orchestra.run.smithi092.stdout: "description": "osd_op(client.15150.0:207239 8.c 8:3f0835ea:::benchmark_data_smithi092_104865_object86738:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e237)", 2024-03-11T21:36:36.937 INFO:teuthology.orchestra.run.smithi092.stdout: "initiated_at": "2024-03-11T21:34:55.465013+0000", 2024-03-11T21:36:36.937 INFO:teuthology.orchestra.run.smithi092.stdout: "age": 101.451116806, 2024-03-11T21:36:36.937 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.908045836, 2024-03-11T21:36:36.937 INFO:teuthology.orchestra.run.smithi092.stdout: "type_data": { 2024-03-11T21:36:36.937 INFO:teuthology.orchestra.run.smithi092.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-11T21:36:36.937 INFO:teuthology.orchestra.run.smithi092.stdout: "client_info": { 2024-03-11T21:36:36.938 INFO:teuthology.orchestra.run.smithi092.stdout: "client": "client.15150", 2024-03-11T21:36:36.938 INFO:teuthology.orchestra.run.smithi092.stdout: "client_addr": "v1:172.21.15.92:0/436606717", 2024-03-11T21:36:36.938 INFO:teuthology.orchestra.run.smithi092.stdout: "tid": 207239 2024-03-11T21:36:36.938 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:36:36.938 INFO:teuthology.orchestra.run.smithi092.stdout: "events": [ 2024-03-11T21:36:36.938 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:36:36.938 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "initiated", 2024-03-11T21:36:36.938 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:34:55.465013+0000", 2024-03-11T21:36:36.938 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-03-11T21:36:36.938 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:36:36.938 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:36:36.938 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "header_read", 2024-03-11T21:36:36.938 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:34:55.465013+0000", 2024-03-11T21:36:36.938 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-03-11T21:36:36.938 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:36:36.939 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:36:36.939 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "throttled", 2024-03-11T21:36:36.939 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:34:55.465016+0000", 2024-03-11T21:36:36.939 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 2.615e-06 2024-03-11T21:36:36.939 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:36:36.939 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:36:36.939 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "all_read", 2024-03-11T21:36:36.939 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:34:55.465021+0000", 2024-03-11T21:36:36.939 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 5.2889999999999997e-06 2024-03-11T21:36:36.939 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:36:36.939 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:36:36.939 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "dispatched", 2024-03-11T21:36:36.939 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:34:55.465036+0000", 2024-03-11T21:36:36.940 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.5023e-05 2024-03-11T21:36:36.940 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:36:36.940 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:36:36.940 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "queued_for_pg", 2024-03-11T21:36:36.940 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:34:55.465044+0000", 2024-03-11T21:36:36.940 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 7.9999999999999996e-06 2024-03-11T21:36:36.940 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:36:36.940 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:36:36.940 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-03-11T21:36:36.940 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:34:55.465100+0000", 2024-03-11T21:36:36.940 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 5.5337999999999998e-05 2024-03-11T21:36:36.940 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:36:36.940 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:36:36.941 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "waiting for peered", 2024-03-11T21:36:36.941 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:34:55.465108+0000", 2024-03-11T21:36:36.941 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 8.5259999999999993e-06 2024-03-11T21:36:36.941 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:36:36.941 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:36:36.941 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-03-11T21:36:36.941 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:34:56.360688+0000", 2024-03-11T21:36:36.941 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.89557976299999997 2024-03-11T21:36:36.941 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:36:36.941 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:36:36.941 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "started", 2024-03-11T21:36:36.941 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:34:56.360781+0000", 2024-03-11T21:36:36.941 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 9.3274000000000002e-05 2024-03-11T21:36:36.942 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:36:36.942 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:36:36.942 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "waiting for subops from 3,7", 2024-03-11T21:36:36.942 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:34:56.360842+0000", 2024-03-11T21:36:36.942 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 6.1334000000000006e-05 2024-03-11T21:36:36.942 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:36:36.942 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:36:36.942 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "sub_op_commit_rec", 2024-03-11T21:36:36.942 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:34:56.367917+0000", 2024-03-11T21:36:36.942 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.0070748219999999997 2024-03-11T21:36:36.942 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:36:36.942 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:36:36.943 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "sub_op_commit_rec", 2024-03-11T21:36:36.943 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:34:56.367996+0000", 2024-03-11T21:36:36.943 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 7.8614000000000003e-05 2024-03-11T21:36:36.943 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:36:36.943 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:36:36.943 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "op_commit", 2024-03-11T21:36:36.943 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:34:56.372955+0000", 2024-03-11T21:36:36.943 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.0049590809999999997 2024-03-11T21:36:36.943 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:36:36.943 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:36:36.943 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "commit_sent", 2024-03-11T21:36:36.943 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:34:56.373012+0000", 2024-03-11T21:36:36.943 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 5.7241000000000003e-05 2024-03-11T21:36:36.943 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:36:36.943 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:36:36.944 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "done", 2024-03-11T21:36:36.944 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:34:56.373059+0000", 2024-03-11T21:36:36.944 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 4.6916000000000003e-05 2024-03-11T21:36:36.944 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-03-11T21:36:36.944 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-03-11T21:36:36.944 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-03-11T21:36:36.944 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:36:36.944 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:36:36.944 INFO:teuthology.orchestra.run.smithi092.stdout: "description": "osd_op(client.15150.0:207246 8.c 8:36f7cf62:::benchmark_data_smithi092_104865_object86745:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e237)", 2024-03-11T21:36:36.944 INFO:teuthology.orchestra.run.smithi092.stdout: "initiated_at": "2024-03-11T21:34:55.472104+0000", 2024-03-11T21:36:36.944 INFO:teuthology.orchestra.run.smithi092.stdout: "age": 101.44402609300001, 2024-03-11T21:36:36.944 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.90107643000000004, 2024-03-11T21:36:36.944 INFO:teuthology.orchestra.run.smithi092.stdout: "type_data": { 2024-03-11T21:36:36.944 INFO:teuthology.orchestra.run.smithi092.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-11T21:36:36.944 INFO:teuthology.orchestra.run.smithi092.stdout: "client_info": { 2024-03-11T21:36:36.945 INFO:teuthology.orchestra.run.smithi092.stdout: "client": "client.15150", 2024-03-11T21:36:36.945 INFO:teuthology.orchestra.run.smithi092.stdout: "client_addr": "v1:172.21.15.92:0/436606717", 2024-03-11T21:36:36.945 INFO:teuthology.orchestra.run.smithi092.stdout: "tid": 207246 2024-03-11T21:36:36.945 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:36:36.945 INFO:teuthology.orchestra.run.smithi092.stdout: "events": [ 2024-03-11T21:36:36.945 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:36:36.945 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "initiated", 2024-03-11T21:36:36.945 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:34:55.472104+0000", 2024-03-11T21:36:36.945 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-03-11T21:36:36.945 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:36:36.945 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:36:36.945 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "header_read", 2024-03-11T21:36:36.945 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:34:55.472104+0000", 2024-03-11T21:36:36.945 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-03-11T21:36:36.946 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:36:36.946 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:36:36.946 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "throttled", 2024-03-11T21:36:36.946 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:34:55.472106+0000", 2024-03-11T21:36:36.946 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.9130000000000001e-06 2024-03-11T21:36:36.946 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:36:36.946 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:36:36.946 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "all_read", 2024-03-11T21:36:36.946 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:34:55.472110+0000", 2024-03-11T21:36:36.946 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 4.3769999999999998e-06 2024-03-11T21:36:36.946 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:36:36.946 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:36:36.946 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "dispatched", 2024-03-11T21:36:36.946 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:34:55.472120+0000", 2024-03-11T21:36:36.947 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.0010000000000001e-05 2024-03-11T21:36:36.947 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:36:36.947 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:36:36.947 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "queued_for_pg", 2024-03-11T21:36:36.947 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:34:55.472127+0000", 2024-03-11T21:36:36.947 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 6.8029999999999999e-06 2024-03-11T21:36:36.947 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:36:36.947 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:36:36.947 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-03-11T21:36:36.947 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:34:55.472219+0000", 2024-03-11T21:36:36.947 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 9.1914000000000001e-05 2024-03-11T21:36:36.947 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:36:36.947 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:36:36.947 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "waiting for peered", 2024-03-11T21:36:36.947 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:34:55.472250+0000", 2024-03-11T21:36:36.948 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 3.0528000000000003e-05 2024-03-11T21:36:36.948 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:36:36.948 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:36:36.948 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-03-11T21:36:36.948 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:34:56.362790+0000", 2024-03-11T21:36:36.948 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.89054004200000003 2024-03-11T21:36:36.948 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:36:36.948 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:36:36.948 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "started", 2024-03-11T21:36:36.948 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:34:56.362911+0000", 2024-03-11T21:36:36.948 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00012171999999999999 2024-03-11T21:36:36.948 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:36:36.948 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:36:36.948 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "waiting for subops from 3,7", 2024-03-11T21:36:36.948 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:34:56.363042+0000", 2024-03-11T21:36:36.948 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00013084699999999999 2024-03-11T21:36:36.949 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:36:36.949 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:36:36.949 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "sub_op_commit_rec", 2024-03-11T21:36:36.949 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:34:56.367956+0000", 2024-03-11T21:36:36.949 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.0049135979999999999 2024-03-11T21:36:36.949 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:36:36.949 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:36:36.949 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "sub_op_commit_rec", 2024-03-11T21:36:36.949 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:34:56.373103+0000", 2024-03-11T21:36:36.949 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.0051477320000000003 2024-03-11T21:36:36.949 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:36:36.949 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:36:36.949 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "op_commit", 2024-03-11T21:36:36.950 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:34:56.373117+0000", 2024-03-11T21:36:36.950 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.3678e-05 2024-03-11T21:36:36.950 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:36:36.950 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:36:36.950 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "commit_sent", 2024-03-11T21:36:36.950 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:34:56.373153+0000", 2024-03-11T21:36:36.950 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 3.5952999999999999e-05 2024-03-11T21:36:36.950 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:36:36.950 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:36:36.950 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "done", 2024-03-11T21:36:36.950 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:34:56.373180+0000", 2024-03-11T21:36:36.950 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 2.7314999999999999e-05 2024-03-11T21:36:36.950 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-03-11T21:36:36.950 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-03-11T21:36:36.950 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-03-11T21:36:36.951 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:36:36.951 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:36:36.951 INFO:teuthology.orchestra.run.smithi092.stdout: "description": "osd_op(client.15150.0:207247 8.c 8:3b696c26:::benchmark_data_smithi092_104865_object86746:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e237)", 2024-03-11T21:36:36.951 INFO:teuthology.orchestra.run.smithi092.stdout: "initiated_at": "2024-03-11T21:34:55.472328+0000", 2024-03-11T21:36:36.951 INFO:teuthology.orchestra.run.smithi092.stdout: "age": 101.44380253, 2024-03-11T21:36:36.951 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.90103873400000001, 2024-03-11T21:36:36.951 INFO:teuthology.orchestra.run.smithi092.stdout: "type_data": { 2024-03-11T21:36:36.951 INFO:teuthology.orchestra.run.smithi092.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-11T21:36:36.951 INFO:teuthology.orchestra.run.smithi092.stdout: "client_info": { 2024-03-11T21:36:36.951 INFO:teuthology.orchestra.run.smithi092.stdout: "client": "client.15150", 2024-03-11T21:36:36.951 INFO:teuthology.orchestra.run.smithi092.stdout: "client_addr": "v1:172.21.15.92:0/436606717", 2024-03-11T21:36:36.951 INFO:teuthology.orchestra.run.smithi092.stdout: "tid": 207247 2024-03-11T21:36:36.951 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:36:36.951 INFO:teuthology.orchestra.run.smithi092.stdout: "events": [ 2024-03-11T21:36:36.952 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:36:36.952 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "initiated", 2024-03-11T21:36:36.952 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:34:55.472328+0000", 2024-03-11T21:36:36.952 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-03-11T21:36:36.952 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:36:36.952 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:36:36.952 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "header_read", 2024-03-11T21:36:36.952 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:34:55.472328+0000", 2024-03-11T21:36:36.952 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-03-11T21:36:36.952 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:36:36.952 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:36:36.952 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "throttled", 2024-03-11T21:36:36.952 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:34:55.472329+0000", 2024-03-11T21:36:36.952 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.0020000000000001e-06 2024-03-11T21:36:36.952 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:36:36.953 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:36:36.953 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "all_read", 2024-03-11T21:36:36.953 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:34:55.472333+0000", 2024-03-11T21:36:36.953 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 4.0690000000000003e-06 2024-03-11T21:36:36.953 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:36:36.953 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:36:36.953 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "dispatched", 2024-03-11T21:36:36.953 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:34:55.472341+0000", 2024-03-11T21:36:36.953 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 8.3629999999999994e-06 2024-03-11T21:36:36.953 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:36:36.953 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:36:36.953 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "queued_for_pg", 2024-03-11T21:36:36.953 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:34:55.472347+0000", 2024-03-11T21:36:36.953 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 6.455e-06 2024-03-11T21:36:36.953 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:36:36.953 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:36:36.954 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-03-11T21:36:36.954 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:34:55.472409+0000", 2024-03-11T21:36:36.954 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 6.1939000000000004e-05 2024-03-11T21:36:36.954 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:36:36.954 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:36:36.954 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "waiting for peered", 2024-03-11T21:36:36.954 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:34:55.472418+0000", 2024-03-11T21:36:36.954 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 8.5299999999999996e-06 2024-03-11T21:36:36.954 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:36:36.954 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:36:36.954 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-03-11T21:36:36.954 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:34:56.365407+0000", 2024-03-11T21:36:36.955 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.89298929900000001 2024-03-11T21:36:36.955 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:36:36.955 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:36:36.955 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "started", 2024-03-11T21:36:36.955 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:34:56.365528+0000", 2024-03-11T21:36:36.955 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.000121049 2024-03-11T21:36:36.955 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:36:36.955 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:36:36.955 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "waiting for subops from 3,7", 2024-03-11T21:36:36.955 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:34:56.365636+0000", 2024-03-11T21:36:36.955 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.000107936 2024-03-11T21:36:36.955 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:36:36.955 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:36:36.956 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "sub_op_commit_rec", 2024-03-11T21:36:36.956 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:34:56.368039+0000", 2024-03-11T21:36:36.956 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.0024027969999999999 2024-03-11T21:36:36.956 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:36:36.956 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:36:36.956 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "op_commit", 2024-03-11T21:36:36.956 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:34:56.373189+0000", 2024-03-11T21:36:36.956 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.0051496520000000002 2024-03-11T21:36:36.956 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:36:36.956 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:36:36.956 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "sub_op_commit_rec", 2024-03-11T21:36:36.956 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:34:56.373254+0000", 2024-03-11T21:36:36.956 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 6.4906999999999994e-05 2024-03-11T21:36:36.956 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:36:36.956 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:36:36.957 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "commit_sent", 2024-03-11T21:36:36.957 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:34:56.373312+0000", 2024-03-11T21:36:36.957 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 5.8545000000000003e-05 2024-03-11T21:36:36.957 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:36:36.957 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:36:36.957 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "done", 2024-03-11T21:36:36.957 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:34:56.373366+0000", 2024-03-11T21:36:36.957 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 5.4190999999999997e-05 2024-03-11T21:36:36.957 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-03-11T21:36:36.957 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-03-11T21:36:36.957 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-03-11T21:36:36.957 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:36:36.957 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:36:36.957 INFO:teuthology.orchestra.run.smithi092.stdout: "description": "osd_op(client.15150.0:207254 8.c 8:33b1c360:::benchmark_data_smithi092_104865_object86753:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e237)", 2024-03-11T21:36:36.958 INFO:teuthology.orchestra.run.smithi092.stdout: "initiated_at": "2024-03-11T21:34:55.476305+0000", 2024-03-11T21:36:36.958 INFO:teuthology.orchestra.run.smithi092.stdout: "age": 101.439825267, 2024-03-11T21:36:36.958 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.897760224, 2024-03-11T21:36:36.958 INFO:teuthology.orchestra.run.smithi092.stdout: "type_data": { 2024-03-11T21:36:36.959 INFO:teuthology.orchestra.run.smithi092.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-11T21:36:36.959 INFO:teuthology.orchestra.run.smithi092.stdout: "client_info": { 2024-03-11T21:36:36.959 INFO:teuthology.orchestra.run.smithi092.stdout: "client": "client.15150", 2024-03-11T21:36:36.959 INFO:teuthology.orchestra.run.smithi092.stdout: "client_addr": "v1:172.21.15.92:0/436606717", 2024-03-11T21:36:36.959 INFO:teuthology.orchestra.run.smithi092.stdout: "tid": 207254 2024-03-11T21:36:36.959 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:36:36.959 INFO:teuthology.orchestra.run.smithi092.stdout: "events": [ 2024-03-11T21:36:36.959 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:36:36.959 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "initiated", 2024-03-11T21:36:36.959 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:34:55.476305+0000", 2024-03-11T21:36:36.959 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-03-11T21:36:36.960 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:36:36.960 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:36:36.960 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "header_read", 2024-03-11T21:36:36.960 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:34:55.476305+0000", 2024-03-11T21:36:36.960 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-03-11T21:36:36.960 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:36:36.960 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:36:36.960 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "throttled", 2024-03-11T21:36:36.960 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:34:55.476306+0000", 2024-03-11T21:36:36.960 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.491e-06 2024-03-11T21:36:36.960 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:36:36.960 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:36:36.960 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "all_read", 2024-03-11T21:36:36.961 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:34:55.476311+0000", 2024-03-11T21:36:36.961 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 4.6120000000000001e-06 2024-03-11T21:36:36.961 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:36:36.961 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:36:36.961 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "dispatched", 2024-03-11T21:36:36.961 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:34:55.476321+0000", 2024-03-11T21:36:36.961 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.0526999999999999e-05 2024-03-11T21:36:36.961 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:36:36.961 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:36:36.961 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "queued_for_pg", 2024-03-11T21:36:36.961 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:34:55.476329+0000", 2024-03-11T21:36:36.961 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 7.0899999999999999e-06 2024-03-11T21:36:36.961 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:36:36.961 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:36:36.962 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-03-11T21:36:36.962 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:34:55.476379+0000", 2024-03-11T21:36:36.962 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 5.0643999999999998e-05 2024-03-11T21:36:36.962 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:36:36.962 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:36:36.962 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "waiting for peered", 2024-03-11T21:36:36.962 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:34:55.476389+0000", 2024-03-11T21:36:36.962 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 9.465e-06 2024-03-11T21:36:36.962 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:36:36.962 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:36:36.962 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-03-11T21:36:36.962 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:34:56.368072+0000", 2024-03-11T21:36:36.962 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.89168363299999998 2024-03-11T21:36:36.962 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:36:36.962 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:36:36.963 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "started", 2024-03-11T21:36:36.963 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:34:56.368172+0000", 2024-03-11T21:36:36.963 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 9.9549000000000005e-05 2024-03-11T21:36:36.963 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:36:36.963 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:36:36.963 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "waiting for subops from 3,7", 2024-03-11T21:36:36.963 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:34:56.368536+0000", 2024-03-11T21:36:36.963 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00036459799999999999 2024-03-11T21:36:36.963 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:36:36.963 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:36:36.963 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "op_commit", 2024-03-11T21:36:36.963 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:34:56.373195+0000", 2024-03-11T21:36:36.963 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.0046583249999999996 2024-03-11T21:36:36.963 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:36:36.963 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:36:36.963 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "sub_op_commit_rec", 2024-03-11T21:36:36.964 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:34:56.373416+0000", 2024-03-11T21:36:36.964 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.000220955 2024-03-11T21:36:36.964 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:36:36.964 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:36:36.964 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "sub_op_commit_rec", 2024-03-11T21:36:36.964 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:34:56.373948+0000", 2024-03-11T21:36:36.964 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00053270499999999996 2024-03-11T21:36:36.964 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:36:36.964 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:36:36.964 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "commit_sent", 2024-03-11T21:36:36.964 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:34:56.374022+0000", 2024-03-11T21:36:36.964 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 7.3209999999999999e-05 2024-03-11T21:36:36.964 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:36:36.964 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:36:36.965 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "done", 2024-03-11T21:36:36.965 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:34:56.374065+0000", 2024-03-11T21:36:36.965 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 4.3420000000000001e-05 2024-03-11T21:36:36.965 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-03-11T21:36:36.965 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-03-11T21:36:36.965 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-03-11T21:36:36.965 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:36:36.965 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:36:36.965 INFO:teuthology.orchestra.run.smithi092.stdout: "description": "osd_op(client.15150.0:207261 8.c 8:3bacfc79:::benchmark_data_smithi092_104865_object86760:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e237)", 2024-03-11T21:36:36.965 INFO:teuthology.orchestra.run.smithi092.stdout: "initiated_at": "2024-03-11T21:34:55.481650+0000", 2024-03-11T21:36:36.965 INFO:teuthology.orchestra.run.smithi092.stdout: "age": 101.43448052700001, 2024-03-11T21:36:36.965 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.90050556400000004, 2024-03-11T21:36:36.965 INFO:teuthology.orchestra.run.smithi092.stdout: "type_data": { 2024-03-11T21:36:36.965 INFO:teuthology.orchestra.run.smithi092.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-11T21:36:36.965 INFO:teuthology.orchestra.run.smithi092.stdout: "client_info": { 2024-03-11T21:36:36.966 INFO:teuthology.orchestra.run.smithi092.stdout: "client": "client.15150", 2024-03-11T21:36:36.966 INFO:teuthology.orchestra.run.smithi092.stdout: "client_addr": "v1:172.21.15.92:0/436606717", 2024-03-11T21:36:36.966 INFO:teuthology.orchestra.run.smithi092.stdout: "tid": 207261 2024-03-11T21:36:36.966 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:36:36.966 INFO:teuthology.orchestra.run.smithi092.stdout: "events": [ 2024-03-11T21:36:36.966 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:36:36.966 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "initiated", 2024-03-11T21:36:36.966 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:34:55.481650+0000", 2024-03-11T21:36:36.966 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-03-11T21:36:36.966 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:36:36.966 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:36:36.966 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "header_read", 2024-03-11T21:36:36.966 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:34:55.481650+0000", 2024-03-11T21:36:36.966 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-03-11T21:36:36.966 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:36:36.967 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:36:36.967 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "throttled", 2024-03-11T21:36:36.967 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:34:55.481652+0000", 2024-03-11T21:36:36.967 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 2.069e-06 2024-03-11T21:36:36.967 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:36:36.967 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:36:36.967 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "all_read", 2024-03-11T21:36:36.967 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:34:55.481659+0000", 2024-03-11T21:36:36.967 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 6.9480000000000002e-06 2024-03-11T21:36:36.967 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:36:36.967 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:36:36.967 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "dispatched", 2024-03-11T21:36:36.967 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:34:55.481675+0000", 2024-03-11T21:36:36.967 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.6773e-05 2024-03-11T21:36:36.967 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:36:36.968 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:36:36.968 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "queued_for_pg", 2024-03-11T21:36:36.968 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:34:55.481685+0000", 2024-03-11T21:36:36.968 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 9.9839999999999996e-06 2024-03-11T21:36:36.968 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:36:36.968 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:36:36.968 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-03-11T21:36:36.968 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:34:55.481744+0000", 2024-03-11T21:36:36.968 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 5.8618e-05 2024-03-11T21:36:36.968 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:36:36.968 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:36:36.968 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "waiting for peered", 2024-03-11T21:36:36.968 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:34:55.481752+0000", 2024-03-11T21:36:36.968 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 8.0309999999999998e-06 2024-03-11T21:36:36.968 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:36:36.969 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:36:36.969 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-03-11T21:36:36.969 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:34:56.370719+0000", 2024-03-11T21:36:36.969 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.88896741000000001 2024-03-11T21:36:36.969 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:36:36.969 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:36:36.969 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "started", 2024-03-11T21:36:36.969 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:34:56.370828+0000", 2024-03-11T21:36:36.969 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00010908600000000001 2024-03-11T21:36:36.969 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:36:36.969 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:36:36.969 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "waiting for subops from 3,7", 2024-03-11T21:36:36.969 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:34:56.370943+0000", 2024-03-11T21:36:36.969 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.000114456 2024-03-11T21:36:36.969 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:36:36.970 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:36:36.970 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "op_commit", 2024-03-11T21:36:36.970 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:34:56.374079+0000", 2024-03-11T21:36:36.970 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.0031359790000000001 2024-03-11T21:36:36.970 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:36:36.970 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:36:36.970 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "sub_op_commit_rec", 2024-03-11T21:36:36.970 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:34:56.374111+0000", 2024-03-11T21:36:36.970 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 3.2277999999999998e-05 2024-03-11T21:36:36.970 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:36:36.970 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:36:36.970 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "sub_op_commit_rec", 2024-03-11T21:36:36.970 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:34:56.382056+0000", 2024-03-11T21:36:36.970 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.0079444860000000006 2024-03-11T21:36:36.971 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:36:36.971 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:36:36.971 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "commit_sent", 2024-03-11T21:36:36.971 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:34:56.382114+0000", 2024-03-11T21:36:36.971 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 5.8307999999999998e-05 2024-03-11T21:36:36.971 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:36:36.971 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:36:36.971 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "done", 2024-03-11T21:36:36.971 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:34:56.382155+0000", 2024-03-11T21:36:36.971 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 4.1137999999999998e-05 2024-03-11T21:36:36.971 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-03-11T21:36:36.971 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-03-11T21:36:36.971 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-03-11T21:36:36.971 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:36:36.971 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:36:36.972 INFO:teuthology.orchestra.run.smithi092.stdout: "description": "osd_op(client.15150.0:207274 8.c 8:35340fad:::benchmark_data_smithi092_104865_object86773:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e237)", 2024-03-11T21:36:36.972 INFO:teuthology.orchestra.run.smithi092.stdout: "initiated_at": "2024-03-11T21:34:55.489338+0000", 2024-03-11T21:36:36.972 INFO:teuthology.orchestra.run.smithi092.stdout: "age": 101.42679206, 2024-03-11T21:36:36.972 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.89598779799999995, 2024-03-11T21:36:36.972 INFO:teuthology.orchestra.run.smithi092.stdout: "type_data": { 2024-03-11T21:36:36.972 INFO:teuthology.orchestra.run.smithi092.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-11T21:36:36.972 INFO:teuthology.orchestra.run.smithi092.stdout: "client_info": { 2024-03-11T21:36:36.972 INFO:teuthology.orchestra.run.smithi092.stdout: "client": "client.15150", 2024-03-11T21:36:36.972 INFO:teuthology.orchestra.run.smithi092.stdout: "client_addr": "v1:172.21.15.92:0/436606717", 2024-03-11T21:36:36.972 INFO:teuthology.orchestra.run.smithi092.stdout: "tid": 207274 2024-03-11T21:36:36.972 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:36:36.972 INFO:teuthology.orchestra.run.smithi092.stdout: "events": [ 2024-03-11T21:36:36.972 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:36:36.973 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "initiated", 2024-03-11T21:36:36.973 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:34:55.489338+0000", 2024-03-11T21:36:36.973 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-03-11T21:36:36.973 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:36:36.973 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:36:36.973 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "header_read", 2024-03-11T21:36:36.973 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:34:55.489338+0000", 2024-03-11T21:36:36.973 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-03-11T21:36:36.973 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:36:36.973 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:36:36.973 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "throttled", 2024-03-11T21:36:36.973 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:34:55.489339+0000", 2024-03-11T21:36:36.973 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.2559999999999999e-06 2024-03-11T21:36:36.973 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:36:36.974 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:36:36.974 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "all_read", 2024-03-11T21:36:36.974 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:34:55.489344+0000", 2024-03-11T21:36:36.974 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 4.7049999999999998e-06 2024-03-11T21:36:36.974 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:36:36.974 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:36:36.974 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "dispatched", 2024-03-11T21:36:36.974 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:34:55.489353+0000", 2024-03-11T21:36:36.974 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 9.1419999999999999e-06 2024-03-11T21:36:36.974 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:36:36.974 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:36:36.974 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "queued_for_pg", 2024-03-11T21:36:36.974 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:34:55.489386+0000", 2024-03-11T21:36:36.975 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 3.2929999999999998e-05 2024-03-11T21:36:36.975 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:36:36.975 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:36:36.975 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-03-11T21:36:36.975 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:34:55.489435+0000", 2024-03-11T21:36:36.975 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 4.8609000000000002e-05 2024-03-11T21:36:36.975 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:36:36.975 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:36:36.975 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "waiting for peered", 2024-03-11T21:36:36.975 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:34:55.489443+0000", 2024-03-11T21:36:36.975 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 8.3580000000000003e-06 2024-03-11T21:36:36.975 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:36:36.975 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:36:36.976 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-03-11T21:36:36.976 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:34:56.374137+0000", 2024-03-11T21:36:36.976 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.884694325 2024-03-11T21:36:36.976 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:36:36.976 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:36:36.976 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "started", 2024-03-11T21:36:36.976 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:34:56.374240+0000", 2024-03-11T21:36:36.976 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.000102196 2024-03-11T21:36:36.976 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:36:36.976 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:36:36.976 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "waiting for subops from 3,7", 2024-03-11T21:36:36.976 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:34:56.374354+0000", 2024-03-11T21:36:36.976 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.000114065 2024-03-11T21:36:36.976 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:36:36.977 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:36:36.977 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "sub_op_commit_rec", 2024-03-11T21:36:36.977 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:34:56.382018+0000", 2024-03-11T21:36:36.977 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.0076644570000000004 2024-03-11T21:36:36.977 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:36:36.977 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:36:36.977 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "op_commit", 2024-03-11T21:36:36.977 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:34:56.385190+0000", 2024-03-11T21:36:36.977 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.0031721290000000001 2024-03-11T21:36:36.977 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:36:36.977 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:36:36.977 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "sub_op_commit_rec", 2024-03-11T21:36:36.977 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:34:56.385231+0000", 2024-03-11T21:36:36.977 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 4.0587000000000003e-05 2024-03-11T21:36:36.977 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:36:36.978 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:36:36.978 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "commit_sent", 2024-03-11T21:36:36.978 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:34:56.385282+0000", 2024-03-11T21:36:36.978 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 5.0899000000000002e-05 2024-03-11T21:36:36.978 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:36:36.978 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:36:36.978 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "done", 2024-03-11T21:36:36.978 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:34:56.385326+0000", 2024-03-11T21:36:36.978 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 4.4140000000000001e-05 2024-03-11T21:36:36.978 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-03-11T21:36:36.978 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-03-11T21:36:36.978 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-03-11T21:36:36.978 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:36:36.978 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:36:36.978 INFO:teuthology.orchestra.run.smithi092.stdout: "description": "osd_op(client.15150.0:207275 8.c 8:3a3c7648:::benchmark_data_smithi092_104865_object86774:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e237)", 2024-03-11T21:36:36.978 INFO:teuthology.orchestra.run.smithi092.stdout: "initiated_at": "2024-03-11T21:34:55.489720+0000", 2024-03-11T21:36:36.979 INFO:teuthology.orchestra.run.smithi092.stdout: "age": 101.426409656, 2024-03-11T21:36:36.979 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.89580858900000004, 2024-03-11T21:36:36.979 INFO:teuthology.orchestra.run.smithi092.stdout: "type_data": { 2024-03-11T21:36:36.979 INFO:teuthology.orchestra.run.smithi092.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-11T21:36:36.979 INFO:teuthology.orchestra.run.smithi092.stdout: "client_info": { 2024-03-11T21:36:36.979 INFO:teuthology.orchestra.run.smithi092.stdout: "client": "client.15150", 2024-03-11T21:36:36.979 INFO:teuthology.orchestra.run.smithi092.stdout: "client_addr": "v1:172.21.15.92:0/436606717", 2024-03-11T21:36:36.979 INFO:teuthology.orchestra.run.smithi092.stdout: "tid": 207275 2024-03-11T21:36:36.979 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:36:36.979 INFO:teuthology.orchestra.run.smithi092.stdout: "events": [ 2024-03-11T21:36:36.979 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:36:36.979 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "initiated", 2024-03-11T21:36:36.979 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:34:55.489720+0000", 2024-03-11T21:36:36.979 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-03-11T21:36:36.980 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:36:36.980 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:36:36.980 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "header_read", 2024-03-11T21:36:36.980 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:34:55.489720+0000", 2024-03-11T21:36:36.980 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-03-11T21:36:36.980 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:36:36.980 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:36:36.980 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "throttled", 2024-03-11T21:36:36.980 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:34:55.489722+0000", 2024-03-11T21:36:36.980 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.1370000000000001e-06 2024-03-11T21:36:36.980 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:36:36.980 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:36:36.980 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "all_read", 2024-03-11T21:36:36.980 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:34:55.489726+0000", 2024-03-11T21:36:36.980 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 4.6129999999999997e-06 2024-03-11T21:36:36.981 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:36:36.981 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:36:36.981 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "dispatched", 2024-03-11T21:36:36.981 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:34:55.489735+0000", 2024-03-11T21:36:36.981 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 9.3160000000000003e-06 2024-03-11T21:36:36.981 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:36:36.981 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:36:36.981 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "queued_for_pg", 2024-03-11T21:36:36.981 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:34:55.489742+0000", 2024-03-11T21:36:36.981 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 6.9720000000000003e-06 2024-03-11T21:36:36.981 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:36:36.981 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:36:36.981 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-03-11T21:36:36.981 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:34:55.489795+0000", 2024-03-11T21:36:36.981 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 5.2796000000000001e-05 2024-03-11T21:36:36.982 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:36:36.982 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:36:36.982 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "waiting for peered", 2024-03-11T21:36:36.982 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:34:55.489802+0000", 2024-03-11T21:36:36.982 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 7.2230000000000001e-06 2024-03-11T21:36:36.982 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:36:36.982 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:36:36.982 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-03-11T21:36:36.982 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:34:56.376733+0000", 2024-03-11T21:36:36.982 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.88693052100000003 2024-03-11T21:36:36.982 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:36:36.982 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:36:36.982 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "started", 2024-03-11T21:36:36.982 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:34:56.376845+0000", 2024-03-11T21:36:36.982 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.000111606 2024-03-11T21:36:36.982 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:36:36.983 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:36:36.983 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "waiting for subops from 3,7", 2024-03-11T21:36:36.983 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:34:56.376959+0000", 2024-03-11T21:36:36.983 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.000114636 2024-03-11T21:36:36.983 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:36:36.983 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:36:36.983 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "sub_op_commit_rec", 2024-03-11T21:36:36.983 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:34:56.382200+0000", 2024-03-11T21:36:36.983 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.0052405400000000001 2024-03-11T21:36:36.983 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:36:36.983 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:36:36.983 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "op_commit", 2024-03-11T21:36:36.983 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:34:56.385339+0000", 2024-03-11T21:36:36.983 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.0031393789999999999 2024-03-11T21:36:36.983 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:36:36.984 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:36:36.984 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "sub_op_commit_rec", 2024-03-11T21:36:36.984 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:34:56.385434+0000", 2024-03-11T21:36:36.984 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 9.4402999999999997e-05 2024-03-11T21:36:36.984 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:36:36.984 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:36:36.984 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "commit_sent", 2024-03-11T21:36:36.984 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:34:56.385491+0000", 2024-03-11T21:36:36.984 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 5.7373000000000001e-05 2024-03-11T21:36:36.984 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:36:36.984 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:36:36.984 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "done", 2024-03-11T21:36:36.984 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:34:56.385529+0000", 2024-03-11T21:36:36.984 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 3.8074000000000002e-05 2024-03-11T21:36:36.984 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-03-11T21:36:36.985 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-03-11T21:36:36.985 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-03-11T21:36:36.985 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:36:36.985 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:36:36.985 INFO:teuthology.orchestra.run.smithi092.stdout: "description": "osd_op(client.15150.0:207314 8.c 8:3b0993f7:::benchmark_data_smithi092_104865_object86813:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e237)", 2024-03-11T21:36:36.985 INFO:teuthology.orchestra.run.smithi092.stdout: "initiated_at": "2024-03-11T21:34:55.525379+0000", 2024-03-11T21:36:36.985 INFO:teuthology.orchestra.run.smithi092.stdout: "age": 101.390750584, 2024-03-11T21:36:36.985 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.86704656899999999, 2024-03-11T21:36:36.985 INFO:teuthology.orchestra.run.smithi092.stdout: "type_data": { 2024-03-11T21:36:36.985 INFO:teuthology.orchestra.run.smithi092.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-11T21:36:36.985 INFO:teuthology.orchestra.run.smithi092.stdout: "client_info": { 2024-03-11T21:36:36.985 INFO:teuthology.orchestra.run.smithi092.stdout: "client": "client.15150", 2024-03-11T21:36:36.985 INFO:teuthology.orchestra.run.smithi092.stdout: "client_addr": "v1:172.21.15.92:0/436606717", 2024-03-11T21:36:36.985 INFO:teuthology.orchestra.run.smithi092.stdout: "tid": 207314 2024-03-11T21:36:36.986 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:36:36.986 INFO:teuthology.orchestra.run.smithi092.stdout: "events": [ 2024-03-11T21:36:36.986 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:36:36.986 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "initiated", 2024-03-11T21:36:36.986 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:34:55.525379+0000", 2024-03-11T21:36:36.986 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-03-11T21:36:36.986 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:36:36.986 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:36:36.986 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "header_read", 2024-03-11T21:36:36.986 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:34:55.525379+0000", 2024-03-11T21:36:36.986 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-03-11T21:36:36.986 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:36:36.986 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:36:36.986 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "throttled", 2024-03-11T21:36:36.987 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:34:55.525381+0000", 2024-03-11T21:36:36.987 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.5239999999999999e-06 2024-03-11T21:36:36.987 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:36:36.987 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:36:36.987 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "all_read", 2024-03-11T21:36:36.987 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:34:55.525386+0000", 2024-03-11T21:36:36.987 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 4.549e-06 2024-03-11T21:36:36.987 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:36:36.987 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:36:36.987 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "dispatched", 2024-03-11T21:36:36.987 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:34:55.525394+0000", 2024-03-11T21:36:36.987 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 7.9689999999999994e-06 2024-03-11T21:36:36.987 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:36:36.987 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:36:36.987 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "queued_for_pg", 2024-03-11T21:36:36.987 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:34:55.525402+0000", 2024-03-11T21:36:36.988 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 8.0010000000000001e-06 2024-03-11T21:36:36.988 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:36:36.988 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:36:36.988 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-03-11T21:36:36.988 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:34:55.525468+0000", 2024-03-11T21:36:36.988 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 6.6648000000000003e-05 2024-03-11T21:36:36.988 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:36:36.988 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:36:36.988 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "waiting for peered", 2024-03-11T21:36:36.988 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:34:55.525476+0000", 2024-03-11T21:36:36.988 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 8.0409999999999998e-06 2024-03-11T21:36:36.988 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:36:36.988 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:36:36.988 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-03-11T21:36:36.989 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:34:56.379314+0000", 2024-03-11T21:36:36.989 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.85383820300000002 2024-03-11T21:36:36.989 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:36:36.989 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:36:36.989 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "started", 2024-03-11T21:36:36.989 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:34:56.379488+0000", 2024-03-11T21:36:36.989 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00017320899999999999 2024-03-11T21:36:36.989 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:36:36.989 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:36:36.989 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "waiting for subops from 3,7", 2024-03-11T21:36:36.989 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:34:56.379594+0000", 2024-03-11T21:36:36.990 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00010648499999999999 2024-03-11T21:36:36.990 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:36:36.990 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:36:36.990 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "op_commit", 2024-03-11T21:36:36.990 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:34:56.385344+0000", 2024-03-11T21:36:36.990 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.0057501740000000003 2024-03-11T21:36:36.990 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:36:36.990 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:36:36.990 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "sub_op_commit_rec", 2024-03-11T21:36:36.991 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:34:56.385571+0000", 2024-03-11T21:36:36.991 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00022667299999999999 2024-03-11T21:36:36.991 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:36:36.991 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:36:36.991 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "sub_op_commit_rec", 2024-03-11T21:36:36.991 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:34:56.392329+0000", 2024-03-11T21:36:36.991 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.0067575609999999996 2024-03-11T21:36:36.991 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:36:36.991 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:36:36.991 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "commit_sent", 2024-03-11T21:36:36.991 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:34:56.392382+0000", 2024-03-11T21:36:36.991 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 5.3282999999999998e-05 2024-03-11T21:36:36.992 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:36:36.992 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:36:36.992 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "done", 2024-03-11T21:36:36.992 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:34:56.392426+0000", 2024-03-11T21:36:36.992 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 4.4249000000000003e-05 2024-03-11T21:36:36.992 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-03-11T21:36:36.992 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-03-11T21:36:36.992 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-03-11T21:36:36.992 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:36:36.992 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:36:36.992 INFO:teuthology.orchestra.run.smithi092.stdout: "description": "osd_op(client.15150.0:207339 8.c 8:39421482:::benchmark_data_smithi092_104865_object86838:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e237)", 2024-03-11T21:36:36.992 INFO:teuthology.orchestra.run.smithi092.stdout: "initiated_at": "2024-03-11T21:34:55.550641+0000", 2024-03-11T21:36:36.992 INFO:teuthology.orchestra.run.smithi092.stdout: "age": 101.36548947, 2024-03-11T21:36:36.992 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.84199107500000003, 2024-03-11T21:36:36.992 INFO:teuthology.orchestra.run.smithi092.stdout: "type_data": { 2024-03-11T21:36:36.993 INFO:teuthology.orchestra.run.smithi092.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-11T21:36:36.993 INFO:teuthology.orchestra.run.smithi092.stdout: "client_info": { 2024-03-11T21:36:36.993 INFO:teuthology.orchestra.run.smithi092.stdout: "client": "client.15150", 2024-03-11T21:36:36.993 INFO:teuthology.orchestra.run.smithi092.stdout: "client_addr": "v1:172.21.15.92:0/436606717", 2024-03-11T21:36:36.993 INFO:teuthology.orchestra.run.smithi092.stdout: "tid": 207339 2024-03-11T21:36:36.993 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:36:36.993 INFO:teuthology.orchestra.run.smithi092.stdout: "events": [ 2024-03-11T21:36:36.993 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:36:36.993 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "initiated", 2024-03-11T21:36:36.993 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:34:55.550641+0000", 2024-03-11T21:36:36.993 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-03-11T21:36:36.993 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:36:36.993 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:36:36.994 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "header_read", 2024-03-11T21:36:36.994 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:34:55.550641+0000", 2024-03-11T21:36:36.994 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-03-11T21:36:36.994 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:36:36.994 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:36:36.994 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "throttled", 2024-03-11T21:36:36.994 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:34:55.550642+0000", 2024-03-11T21:36:36.994 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.474e-06 2024-03-11T21:36:36.994 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:36:36.994 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:36:36.994 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "all_read", 2024-03-11T21:36:36.994 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:34:55.550647+0000", 2024-03-11T21:36:36.994 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 4.639e-06 2024-03-11T21:36:36.995 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:36:36.995 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:36:36.995 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "dispatched", 2024-03-11T21:36:36.995 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:34:55.550656+0000", 2024-03-11T21:36:36.995 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 9.3109999999999995e-06 2024-03-11T21:36:36.995 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:36:36.995 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:36:36.995 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "queued_for_pg", 2024-03-11T21:36:36.995 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:34:55.550663+0000", 2024-03-11T21:36:36.995 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 7.2969999999999998e-06 2024-03-11T21:36:36.995 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:36:36.995 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:36:36.995 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-03-11T21:36:36.996 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:34:55.550752+0000", 2024-03-11T21:36:36.996 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 8.8441000000000002e-05 2024-03-11T21:36:36.996 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:36:36.996 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:36:36.996 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "waiting for peered", 2024-03-11T21:36:36.996 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:34:55.550762+0000", 2024-03-11T21:36:36.996 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.0685e-05 2024-03-11T21:36:36.996 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:36:36.996 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:36:36.996 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-03-11T21:36:36.996 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:34:56.382713+0000", 2024-03-11T21:36:36.996 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.83195014499999997 2024-03-11T21:36:36.996 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:36:36.996 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:36:36.997 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "started", 2024-03-11T21:36:36.997 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:34:56.382808+0000", 2024-03-11T21:36:36.997 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 9.5597e-05 2024-03-11T21:36:36.997 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:36:36.997 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:36:36.997 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "waiting for subops from 3,7", 2024-03-11T21:36:36.997 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:34:56.382885+0000", 2024-03-11T21:36:36.997 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 7.7191e-05 2024-03-11T21:36:36.997 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:36:36.997 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:36:36.997 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "sub_op_commit_rec", 2024-03-11T21:36:36.997 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:34:56.385611+0000", 2024-03-11T21:36:36.997 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.002726095 2024-03-11T21:36:36.997 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:36:36.997 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:36:36.998 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "op_commit", 2024-03-11T21:36:36.998 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:34:56.388598+0000", 2024-03-11T21:36:36.998 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.0029865439999999998 2024-03-11T21:36:36.998 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:36:36.998 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:36:36.998 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "sub_op_commit_rec", 2024-03-11T21:36:36.998 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:34:56.392542+0000", 2024-03-11T21:36:36.998 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.0039435980000000004 2024-03-11T21:36:36.998 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:36:36.998 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:36:36.998 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "commit_sent", 2024-03-11T21:36:36.998 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:34:56.392590+0000", 2024-03-11T21:36:36.998 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 4.8622999999999998e-05 2024-03-11T21:36:36.999 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:36:36.999 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:36:36.999 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "done", 2024-03-11T21:36:36.999 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:34:56.392632+0000", 2024-03-11T21:36:36.999 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 4.1434999999999999e-05 2024-03-11T21:36:36.999 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-03-11T21:36:36.999 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-03-11T21:36:36.999 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-03-11T21:36:36.999 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:36:36.999 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:36:36.999 INFO:teuthology.orchestra.run.smithi092.stdout: "description": "osd_op(client.15150.0:207361 8.c 8:3ecfd1aa:::benchmark_data_smithi092_104865_object86860:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e237)", 2024-03-11T21:36:36.999 INFO:teuthology.orchestra.run.smithi092.stdout: "initiated_at": "2024-03-11T21:34:55.573296+0000", 2024-03-11T21:36:36.999 INFO:teuthology.orchestra.run.smithi092.stdout: "age": 101.34283424, 2024-03-11T21:36:36.999 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.83335905099999996, 2024-03-11T21:36:37.000 INFO:teuthology.orchestra.run.smithi092.stdout: "type_data": { 2024-03-11T21:36:37.000 INFO:teuthology.orchestra.run.smithi092.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-11T21:36:37.000 INFO:teuthology.orchestra.run.smithi092.stdout: "client_info": { 2024-03-11T21:36:37.000 INFO:teuthology.orchestra.run.smithi092.stdout: "client": "client.15150", 2024-03-11T21:36:37.000 INFO:teuthology.orchestra.run.smithi092.stdout: "client_addr": "v1:172.21.15.92:0/436606717", 2024-03-11T21:36:37.000 INFO:teuthology.orchestra.run.smithi092.stdout: "tid": 207361 2024-03-11T21:36:37.000 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:36:37.000 INFO:teuthology.orchestra.run.smithi092.stdout: "events": [ 2024-03-11T21:36:37.000 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:36:37.000 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "initiated", 2024-03-11T21:36:37.000 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:34:55.573296+0000", 2024-03-11T21:36:37.000 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-03-11T21:36:37.000 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:36:37.000 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:36:37.000 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "header_read", 2024-03-11T21:36:37.001 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:34:55.573296+0000", 2024-03-11T21:36:37.001 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-03-11T21:36:37.001 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:36:37.001 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:36:37.001 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "throttled", 2024-03-11T21:36:37.001 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:34:55.573297+0000", 2024-03-11T21:36:37.001 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.547e-06 2024-03-11T21:36:37.001 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:36:37.001 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:36:37.001 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "all_read", 2024-03-11T21:36:37.001 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:34:55.573302+0000", 2024-03-11T21:36:37.001 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 4.2729999999999996e-06 2024-03-11T21:36:37.001 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:36:37.001 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:36:37.001 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "dispatched", 2024-03-11T21:36:37.002 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:34:55.573311+0000", 2024-03-11T21:36:37.002 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 9.7520000000000008e-06 2024-03-11T21:36:37.002 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:36:37.002 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:36:37.002 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "queued_for_pg", 2024-03-11T21:36:37.002 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:34:55.573319+0000", 2024-03-11T21:36:37.002 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 7.5270000000000001e-06 2024-03-11T21:36:37.002 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:36:37.002 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:36:37.002 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-03-11T21:36:37.002 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:34:55.573360+0000", 2024-03-11T21:36:37.002 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 4.1306000000000001e-05 2024-03-11T21:36:37.002 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:36:37.002 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:36:37.002 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "waiting for peered", 2024-03-11T21:36:37.003 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:34:55.573367+0000", 2024-03-11T21:36:37.003 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 6.3609999999999997e-06 2024-03-11T21:36:37.003 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:36:37.003 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:36:37.003 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-03-11T21:36:37.003 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:34:56.385737+0000", 2024-03-11T21:36:37.003 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.81237010300000001 2024-03-11T21:36:37.003 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:36:37.003 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:36:37.003 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "started", 2024-03-11T21:36:37.003 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:34:56.385845+0000", 2024-03-11T21:36:37.003 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.000108478 2024-03-11T21:36:37.003 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:36:37.003 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:36:37.004 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "waiting for subops from 3,7", 2024-03-11T21:36:37.004 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:34:56.385955+0000", 2024-03-11T21:36:37.004 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.000110135 2024-03-11T21:36:37.004 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:36:37.004 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:36:37.004 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "sub_op_commit_rec", 2024-03-11T21:36:37.004 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:34:56.392508+0000", 2024-03-11T21:36:37.004 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.0065524540000000001 2024-03-11T21:36:37.004 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:36:37.004 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:36:37.004 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "sub_op_commit_rec", 2024-03-11T21:36:37.004 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:34:56.395695+0000", 2024-03-11T21:36:37.004 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.003187685 2024-03-11T21:36:37.004 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:36:37.005 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:36:37.005 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "op_commit", 2024-03-11T21:36:37.005 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:34:56.406563+0000", 2024-03-11T21:36:37.005 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.010867523 2024-03-11T21:36:37.005 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:36:37.005 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:36:37.005 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "commit_sent", 2024-03-11T21:36:37.005 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:34:56.406610+0000", 2024-03-11T21:36:37.005 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 4.6808000000000003e-05 2024-03-11T21:36:37.005 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:36:37.005 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:36:37.005 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "done", 2024-03-11T21:36:37.005 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:34:56.406655+0000", 2024-03-11T21:36:37.005 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 4.5099000000000003e-05 2024-03-11T21:36:37.005 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-03-11T21:36:37.005 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-03-11T21:36:37.006 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-03-11T21:36:37.006 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:36:37.006 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:36:37.006 INFO:teuthology.orchestra.run.smithi092.stdout: "description": "osd_op(client.15150.0:207366 8.c 8:3b30730b:::benchmark_data_smithi092_104865_object86865:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e237)", 2024-03-11T21:36:37.006 INFO:teuthology.orchestra.run.smithi092.stdout: "initiated_at": "2024-03-11T21:34:55.578893+0000", 2024-03-11T21:36:37.006 INFO:teuthology.orchestra.run.smithi092.stdout: "age": 101.33723731000001, 2024-03-11T21:36:37.006 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.827840468, 2024-03-11T21:36:37.006 INFO:teuthology.orchestra.run.smithi092.stdout: "type_data": { 2024-03-11T21:36:37.006 INFO:teuthology.orchestra.run.smithi092.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-11T21:36:37.006 INFO:teuthology.orchestra.run.smithi092.stdout: "client_info": { 2024-03-11T21:36:37.006 INFO:teuthology.orchestra.run.smithi092.stdout: "client": "client.15150", 2024-03-11T21:36:37.006 INFO:teuthology.orchestra.run.smithi092.stdout: "client_addr": "v1:172.21.15.92:0/436606717", 2024-03-11T21:36:37.006 INFO:teuthology.orchestra.run.smithi092.stdout: "tid": 207366 2024-03-11T21:36:37.006 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:36:37.007 INFO:teuthology.orchestra.run.smithi092.stdout: "events": [ 2024-03-11T21:36:37.007 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:36:37.007 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "initiated", 2024-03-11T21:36:37.007 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:34:55.578893+0000", 2024-03-11T21:36:37.007 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-03-11T21:36:37.007 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:36:37.007 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:36:37.007 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "header_read", 2024-03-11T21:36:37.007 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:34:55.578893+0000", 2024-03-11T21:36:37.007 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-03-11T21:36:37.007 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:36:37.007 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:36:37.007 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "throttled", 2024-03-11T21:36:37.007 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:34:55.578894+0000", 2024-03-11T21:36:37.007 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.742e-06 2024-03-11T21:36:37.007 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:36:37.008 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:36:37.008 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "all_read", 2024-03-11T21:36:37.008 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:34:55.578901+0000", 2024-03-11T21:36:37.008 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 6.1369999999999999e-06 2024-03-11T21:36:37.008 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:36:37.008 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:36:37.008 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "dispatched", 2024-03-11T21:36:37.008 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:34:55.578912+0000", 2024-03-11T21:36:37.008 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.1402e-05 2024-03-11T21:36:37.008 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:36:37.008 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:36:37.008 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "queued_for_pg", 2024-03-11T21:36:37.008 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:34:55.578921+0000", 2024-03-11T21:36:37.008 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 8.5459999999999992e-06 2024-03-11T21:36:37.008 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:36:37.008 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:36:37.009 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-03-11T21:36:37.009 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:34:55.578972+0000", 2024-03-11T21:36:37.009 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 5.1138999999999999e-05 2024-03-11T21:36:37.009 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:36:37.009 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:36:37.009 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "waiting for peered", 2024-03-11T21:36:37.009 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:34:55.578978+0000", 2024-03-11T21:36:37.009 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 6.4219999999999997e-06 2024-03-11T21:36:37.009 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:36:37.009 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:36:37.009 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-03-11T21:36:37.009 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:34:56.389497+0000", 2024-03-11T21:36:37.009 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.81051906299999998 2024-03-11T21:36:37.009 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:36:37.009 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:36:37.009 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "started", 2024-03-11T21:36:37.010 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:34:56.389610+0000", 2024-03-11T21:36:37.010 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.000112972 2024-03-11T21:36:37.010 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:36:37.010 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:36:37.010 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "waiting for subops from 3,7", 2024-03-11T21:36:37.010 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:34:56.389751+0000", 2024-03-11T21:36:37.010 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.000140798 2024-03-11T21:36:37.010 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:36:37.010 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:36:37.010 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "sub_op_commit_rec", 2024-03-11T21:36:37.010 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:34:56.395736+0000", 2024-03-11T21:36:37.010 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.0059849680000000002 2024-03-11T21:36:37.010 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:36:37.010 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:36:37.011 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "sub_op_commit_rec", 2024-03-11T21:36:37.011 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:34:56.401487+0000", 2024-03-11T21:36:37.011 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.005750603 2024-03-11T21:36:37.011 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:36:37.011 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:36:37.011 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "op_commit", 2024-03-11T21:36:37.011 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:34:56.406662+0000", 2024-03-11T21:36:37.011 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.0051752480000000003 2024-03-11T21:36:37.011 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:36:37.011 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:36:37.011 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "commit_sent", 2024-03-11T21:36:37.011 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:34:56.406701+0000", 2024-03-11T21:36:37.011 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 3.9124e-05 2024-03-11T21:36:37.011 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:36:37.011 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:36:37.011 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "done", 2024-03-11T21:36:37.012 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:34:56.406733+0000", 2024-03-11T21:36:37.012 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 3.2304000000000001e-05 2024-03-11T21:36:37.012 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-03-11T21:36:37.012 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-03-11T21:36:37.012 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-03-11T21:36:37.012 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:36:37.012 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:36:37.012 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:36:37.013 INFO:teuthology.orchestra.run.smithi092.stdout: "description": "osd_op(client.15150.0:207382 8.c 8:31ac71bf:::benchmark_data_smithi092_104865_object86881:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e237)", 2024-03-11T21:36:37.013 INFO:teuthology.orchestra.run.smithi092.stdout: "initiated_at": "2024-03-11T21:34:55.605399+0000", 2024-03-11T21:36:37.013 INFO:teuthology.orchestra.run.smithi092.stdout: "age": 101.31073142, 2024-03-11T21:36:37.013 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.80149564600000001, 2024-03-11T21:36:37.014 INFO:teuthology.orchestra.run.smithi092.stdout: "type_data": { 2024-03-11T21:36:37.014 INFO:teuthology.orchestra.run.smithi092.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-11T21:36:37.014 INFO:teuthology.orchestra.run.smithi092.stdout: "client_info": { 2024-03-11T21:36:37.014 INFO:teuthology.orchestra.run.smithi092.stdout: "client": "client.15150", 2024-03-11T21:36:37.014 INFO:teuthology.orchestra.run.smithi092.stdout: "client_addr": "v1:172.21.15.92:0/436606717", 2024-03-11T21:36:37.014 INFO:teuthology.orchestra.run.smithi092.stdout: "tid": 207382 2024-03-11T21:36:37.014 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:36:37.014 INFO:teuthology.orchestra.run.smithi092.stdout: "events": [ 2024-03-11T21:36:37.014 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:36:37.014 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "initiated", 2024-03-11T21:36:37.014 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:34:55.605399+0000", 2024-03-11T21:36:37.014 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-03-11T21:36:37.014 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:36:37.015 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:36:37.015 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "header_read", 2024-03-11T21:36:37.015 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:34:55.605399+0000", 2024-03-11T21:36:37.015 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-03-11T21:36:37.015 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:36:37.015 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:36:37.015 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "throttled", 2024-03-11T21:36:37.015 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:34:55.605400+0000", 2024-03-11T21:36:37.015 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.778e-06 2024-03-11T21:36:37.015 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:36:37.015 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:36:37.015 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "all_read", 2024-03-11T21:36:37.015 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:34:55.605405+0000", 2024-03-11T21:36:37.015 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 5.0420000000000002e-06 2024-03-11T21:36:37.015 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:36:37.016 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:36:37.016 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "dispatched", 2024-03-11T21:36:37.016 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:34:55.605417+0000", 2024-03-11T21:36:37.016 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.1212e-05 2024-03-11T21:36:37.016 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:36:37.016 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:36:37.016 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "queued_for_pg", 2024-03-11T21:36:37.016 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:34:55.605426+0000", 2024-03-11T21:36:37.016 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 9.1889999999999996e-06 2024-03-11T21:36:37.016 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:36:37.016 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:36:37.016 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-03-11T21:36:37.016 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:34:55.605479+0000", 2024-03-11T21:36:37.016 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 5.3424999999999998e-05 2024-03-11T21:36:37.016 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:36:37.017 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:36:37.017 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "waiting for peered", 2024-03-11T21:36:37.017 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:34:55.605487+0000", 2024-03-11T21:36:37.017 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 7.8050000000000007e-06 2024-03-11T21:36:37.017 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:36:37.017 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:36:37.017 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-03-11T21:36:37.017 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:34:56.392659+0000", 2024-03-11T21:36:37.017 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.78717215399999996 2024-03-11T21:36:37.017 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:36:37.017 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:36:37.017 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "started", 2024-03-11T21:36:37.017 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:34:56.393047+0000", 2024-03-11T21:36:37.017 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00038755499999999999 2024-03-11T21:36:37.017 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:36:37.018 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:36:37.018 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "waiting for subops from 3,7", 2024-03-11T21:36:37.018 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:34:56.393166+0000", 2024-03-11T21:36:37.018 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00011882799999999999 2024-03-11T21:36:37.018 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:36:37.018 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:36:37.018 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "sub_op_commit_rec", 2024-03-11T21:36:37.018 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:34:56.401524+0000", 2024-03-11T21:36:37.018 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.0083586649999999995 2024-03-11T21:36:37.018 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:36:37.018 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:36:37.018 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "sub_op_commit_rec", 2024-03-11T21:36:37.018 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:34:56.401559+0000", 2024-03-11T21:36:37.019 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 3.4650000000000002e-05 2024-03-11T21:36:37.019 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:36:37.019 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:36:37.019 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "op_commit", 2024-03-11T21:36:37.019 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:34:56.406816+0000", 2024-03-11T21:36:37.019 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.0052568850000000002 2024-03-11T21:36:37.019 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:36:37.019 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:36:37.019 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "commit_sent", 2024-03-11T21:36:37.019 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:34:56.406859+0000", 2024-03-11T21:36:37.019 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 4.2892999999999997e-05 2024-03-11T21:36:37.019 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:36:37.019 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:36:37.019 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "done", 2024-03-11T21:36:37.019 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:34:56.406894+0000", 2024-03-11T21:36:37.020 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 3.5565000000000003e-05 2024-03-11T21:36:37.020 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-03-11T21:36:37.020 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-03-11T21:36:37.020 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-03-11T21:36:37.020 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:36:37.020 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:36:37.020 INFO:teuthology.orchestra.run.smithi092.stdout: "description": "osd_op(client.15150.0:207412 8.c 8:312392f7:::benchmark_data_smithi092_104865_object86911:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e237)", 2024-03-11T21:36:37.020 INFO:teuthology.orchestra.run.smithi092.stdout: "initiated_at": "2024-03-11T21:34:55.696161+0000", 2024-03-11T21:36:37.020 INFO:teuthology.orchestra.run.smithi092.stdout: "age": 101.219969456, 2024-03-11T21:36:37.020 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.71080487199999998, 2024-03-11T21:36:37.020 INFO:teuthology.orchestra.run.smithi092.stdout: "type_data": { 2024-03-11T21:36:37.020 INFO:teuthology.orchestra.run.smithi092.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-11T21:36:37.020 INFO:teuthology.orchestra.run.smithi092.stdout: "client_info": { 2024-03-11T21:36:37.020 INFO:teuthology.orchestra.run.smithi092.stdout: "client": "client.15150", 2024-03-11T21:36:37.021 INFO:teuthology.orchestra.run.smithi092.stdout: "client_addr": "v1:172.21.15.92:0/436606717", 2024-03-11T21:36:37.021 INFO:teuthology.orchestra.run.smithi092.stdout: "tid": 207412 2024-03-11T21:36:37.021 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:36:37.021 INFO:teuthology.orchestra.run.smithi092.stdout: "events": [ 2024-03-11T21:36:37.021 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:36:37.021 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "initiated", 2024-03-11T21:36:37.021 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:34:55.696161+0000", 2024-03-11T21:36:37.021 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-03-11T21:36:37.021 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:36:37.021 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:36:37.021 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "header_read", 2024-03-11T21:36:37.021 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:34:55.696161+0000", 2024-03-11T21:36:37.021 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-03-11T21:36:37.021 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:36:37.021 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:36:37.021 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "throttled", 2024-03-11T21:36:37.022 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:34:55.696163+0000", 2024-03-11T21:36:37.022 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.9309999999999998e-06 2024-03-11T21:36:37.022 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:36:37.022 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:36:37.022 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "all_read", 2024-03-11T21:36:37.022 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:34:55.696168+0000", 2024-03-11T21:36:37.022 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 5.0320000000000003e-06 2024-03-11T21:36:37.022 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:36:37.022 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:36:37.022 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "dispatched", 2024-03-11T21:36:37.022 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:34:55.696178+0000", 2024-03-11T21:36:37.022 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.0788000000000001e-05 2024-03-11T21:36:37.022 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:36:37.022 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:36:37.022 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "queued_for_pg", 2024-03-11T21:36:37.023 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:34:55.696184+0000", 2024-03-11T21:36:37.023 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 6.1380000000000004e-06 2024-03-11T21:36:37.023 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:36:37.023 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:36:37.023 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-03-11T21:36:37.023 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:34:55.696234+0000", 2024-03-11T21:36:37.023 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 4.9373000000000003e-05 2024-03-11T21:36:37.023 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:36:37.023 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:36:37.023 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "waiting for peered", 2024-03-11T21:36:37.023 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:34:55.696241+0000", 2024-03-11T21:36:37.023 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 6.6660000000000002e-06 2024-03-11T21:36:37.023 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:36:37.023 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:36:37.024 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-03-11T21:36:37.024 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:34:56.395765+0000", 2024-03-11T21:36:37.024 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.69952489699999998 2024-03-11T21:36:37.024 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:36:37.024 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:36:37.024 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "started", 2024-03-11T21:36:37.024 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:34:56.395874+0000", 2024-03-11T21:36:37.024 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00010840600000000001 2024-03-11T21:36:37.024 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:36:37.024 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:36:37.024 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "waiting for subops from 3,7", 2024-03-11T21:36:37.024 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:34:56.396221+0000", 2024-03-11T21:36:37.025 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.000346998 2024-03-11T21:36:37.025 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:36:37.025 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:36:37.025 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "sub_op_commit_rec", 2024-03-11T21:36:37.025 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:34:56.403836+0000", 2024-03-11T21:36:37.025 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.0076152470000000003 2024-03-11T21:36:37.025 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:36:37.025 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:36:37.025 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "sub_op_commit_rec", 2024-03-11T21:36:37.025 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:34:56.406771+0000", 2024-03-11T21:36:37.025 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.0029351730000000001 2024-03-11T21:36:37.026 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:36:37.026 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:36:37.026 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "op_commit", 2024-03-11T21:36:37.026 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:34:56.406900+0000", 2024-03-11T21:36:37.026 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.000128594 2024-03-11T21:36:37.026 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:36:37.026 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:36:37.026 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "commit_sent", 2024-03-11T21:36:37.026 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:34:56.406934+0000", 2024-03-11T21:36:37.026 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 3.4588000000000002e-05 2024-03-11T21:36:37.026 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:36:37.026 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:36:37.026 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "done", 2024-03-11T21:36:37.026 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:34:56.406965+0000", 2024-03-11T21:36:37.027 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 3.1041000000000003e-05 2024-03-11T21:36:37.027 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-03-11T21:36:37.027 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-03-11T21:36:37.027 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-03-11T21:36:37.027 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:36:37.027 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:36:37.027 INFO:teuthology.orchestra.run.smithi092.stdout: "description": "osd_op(client.25282.0:76299 9.8 9:17331e66:::benchmark_data_smithi092_130291_object76298:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e278)", 2024-03-11T21:36:37.027 INFO:teuthology.orchestra.run.smithi092.stdout: "initiated_at": "2024-03-11T21:36:35.048550+0000", 2024-03-11T21:36:37.027 INFO:teuthology.orchestra.run.smithi092.stdout: "age": 1.8675804579999999, 2024-03-11T21:36:37.027 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.4852829970000001, 2024-03-11T21:36:37.027 INFO:teuthology.orchestra.run.smithi092.stdout: "type_data": { 2024-03-11T21:36:37.027 INFO:teuthology.orchestra.run.smithi092.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-11T21:36:37.027 INFO:teuthology.orchestra.run.smithi092.stdout: "client_info": { 2024-03-11T21:36:37.027 INFO:teuthology.orchestra.run.smithi092.stdout: "client": "client.25282", 2024-03-11T21:36:37.028 INFO:teuthology.orchestra.run.smithi092.stdout: "client_addr": "v1:172.21.15.92:0/2489519821", 2024-03-11T21:36:37.028 INFO:teuthology.orchestra.run.smithi092.stdout: "tid": 76299 2024-03-11T21:36:37.028 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:36:37.028 INFO:teuthology.orchestra.run.smithi092.stdout: "events": [ 2024-03-11T21:36:37.028 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:36:37.028 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "initiated", 2024-03-11T21:36:37.028 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:36:35.048550+0000", 2024-03-11T21:36:37.028 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-03-11T21:36:37.028 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:36:37.028 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:36:37.028 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "header_read", 2024-03-11T21:36:37.028 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:36:35.048550+0000", 2024-03-11T21:36:37.028 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-03-11T21:36:37.029 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:36:37.029 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:36:37.029 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "throttled", 2024-03-11T21:36:37.029 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:36:35.048552+0000", 2024-03-11T21:36:37.029 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 2.0710000000000002e-06 2024-03-11T21:36:37.029 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:36:37.029 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:36:37.029 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "all_read", 2024-03-11T21:36:37.029 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:36:35.048584+0000", 2024-03-11T21:36:37.029 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 3.2676999999999999e-05 2024-03-11T21:36:37.029 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:36:37.029 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:36:37.029 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "dispatched", 2024-03-11T21:36:37.029 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:36:35.048597+0000", 2024-03-11T21:36:37.029 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.2845e-05 2024-03-11T21:36:37.030 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:36:37.030 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:36:37.030 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "queued_for_pg", 2024-03-11T21:36:37.030 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:36:35.048608+0000", 2024-03-11T21:36:37.030 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.1005000000000001e-05 2024-03-11T21:36:37.030 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:36:37.030 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:36:37.030 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-03-11T21:36:37.030 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:36:35.049475+0000", 2024-03-11T21:36:37.030 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00086655599999999999 2024-03-11T21:36:37.030 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:36:37.030 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:36:37.030 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "waiting for peered", 2024-03-11T21:36:37.031 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:36:35.049486+0000", 2024-03-11T21:36:37.031 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.0842000000000001e-05 2024-03-11T21:36:37.031 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:36:37.031 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:36:37.031 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-03-11T21:36:37.031 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:36:36.516700+0000", 2024-03-11T21:36:37.031 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.467214099 2024-03-11T21:36:37.031 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:36:37.031 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:36:37.031 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "started", 2024-03-11T21:36:37.031 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:36:36.516847+0000", 2024-03-11T21:36:37.031 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00014753699999999999 2024-03-11T21:36:37.031 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:36:37.031 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:36:37.031 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "waiting for subops from 3,5", 2024-03-11T21:36:37.032 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:36:36.517015+0000", 2024-03-11T21:36:37.032 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00016770700000000001 2024-03-11T21:36:37.032 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:36:37.032 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:36:37.032 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "sub_op_commit_rec", 2024-03-11T21:36:37.032 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:36:36.527882+0000", 2024-03-11T21:36:37.032 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.010866607 2024-03-11T21:36:37.032 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:36:37.032 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:36:37.032 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "sub_op_commit_rec", 2024-03-11T21:36:37.032 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:36:36.527930+0000", 2024-03-11T21:36:37.032 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 4.8575000000000002e-05 2024-03-11T21:36:37.032 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:36:37.032 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:36:37.032 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "op_commit", 2024-03-11T21:36:37.032 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:36:36.530142+0000", 2024-03-11T21:36:37.033 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.002212206 2024-03-11T21:36:37.033 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:36:37.033 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:36:37.033 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "commit_sent", 2024-03-11T21:36:37.033 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:36:36.530209+0000", 2024-03-11T21:36:37.033 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 6.6594999999999995e-05 2024-03-11T21:36:37.033 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:36:37.033 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:36:37.033 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "done", 2024-03-11T21:36:37.033 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:36:36.533833+0000", 2024-03-11T21:36:37.033 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.0036236749999999998 2024-03-11T21:36:37.033 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-03-11T21:36:37.033 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-03-11T21:36:37.033 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-03-11T21:36:37.033 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:36:37.034 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:36:37.034 INFO:teuthology.orchestra.run.smithi092.stdout: "description": "osd_op(client.25282.0:76301 9.8 9:1db40ce0:::benchmark_data_smithi092_130291_object76300:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e278)", 2024-03-11T21:36:37.034 INFO:teuthology.orchestra.run.smithi092.stdout: "initiated_at": "2024-03-11T21:36:35.048631+0000", 2024-03-11T21:36:37.034 INFO:teuthology.orchestra.run.smithi092.stdout: "age": 1.86749948, 2024-03-11T21:36:37.034 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.495054493, 2024-03-11T21:36:37.034 INFO:teuthology.orchestra.run.smithi092.stdout: "type_data": { 2024-03-11T21:36:37.034 INFO:teuthology.orchestra.run.smithi092.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-11T21:36:37.034 INFO:teuthology.orchestra.run.smithi092.stdout: "client_info": { 2024-03-11T21:36:37.034 INFO:teuthology.orchestra.run.smithi092.stdout: "client": "client.25282", 2024-03-11T21:36:37.034 INFO:teuthology.orchestra.run.smithi092.stdout: "client_addr": "v1:172.21.15.92:0/2489519821", 2024-03-11T21:36:37.034 INFO:teuthology.orchestra.run.smithi092.stdout: "tid": 76301 2024-03-11T21:36:37.034 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:36:37.034 INFO:teuthology.orchestra.run.smithi092.stdout: "events": [ 2024-03-11T21:36:37.034 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:36:37.034 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "initiated", 2024-03-11T21:36:37.035 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:36:35.048631+0000", 2024-03-11T21:36:37.035 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-03-11T21:36:37.035 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:36:37.035 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:36:37.035 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "header_read", 2024-03-11T21:36:37.035 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:36:35.048631+0000", 2024-03-11T21:36:37.035 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-03-11T21:36:37.035 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:36:37.035 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:36:37.035 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "throttled", 2024-03-11T21:36:37.035 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:36:35.048632+0000", 2024-03-11T21:36:37.035 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.1170000000000001e-06 2024-03-11T21:36:37.035 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:36:37.035 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:36:37.035 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "all_read", 2024-03-11T21:36:37.035 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:36:35.048660+0000", 2024-03-11T21:36:37.036 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 2.8243e-05 2024-03-11T21:36:37.036 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:36:37.036 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:36:37.036 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "dispatched", 2024-03-11T21:36:37.036 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:36:35.048669+0000", 2024-03-11T21:36:37.036 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 8.7579999999999998e-06 2024-03-11T21:36:37.036 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:36:37.036 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:36:37.036 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "queued_for_pg", 2024-03-11T21:36:37.036 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:36:35.048697+0000", 2024-03-11T21:36:37.036 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 2.8138999999999999e-05 2024-03-11T21:36:37.036 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:36:37.036 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:36:37.036 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-03-11T21:36:37.036 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:36:35.049509+0000", 2024-03-11T21:36:37.036 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00081172500000000001 2024-03-11T21:36:37.037 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:36:37.037 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:36:37.037 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "waiting for peered", 2024-03-11T21:36:37.037 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:36:35.049515+0000", 2024-03-11T21:36:37.037 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 5.9909999999999999e-06 2024-03-11T21:36:37.037 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:36:37.037 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:36:37.037 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-03-11T21:36:37.037 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:36:36.523893+0000", 2024-03-11T21:36:37.037 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.4743779799999999 2024-03-11T21:36:37.037 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:36:37.037 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:36:37.037 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "started", 2024-03-11T21:36:37.038 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:36:36.524187+0000", 2024-03-11T21:36:37.038 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00029435900000000001 2024-03-11T21:36:37.038 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:36:37.038 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:36:37.038 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "waiting for subops from 3,5", 2024-03-11T21:36:37.038 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:36:36.524376+0000", 2024-03-11T21:36:37.038 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00018891599999999999 2024-03-11T21:36:37.038 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:36:37.038 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:36:37.038 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "sub_op_commit_rec", 2024-03-11T21:36:37.038 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:36:36.530124+0000", 2024-03-11T21:36:37.038 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.0057478800000000004 2024-03-11T21:36:37.038 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:36:37.038 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:36:37.038 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "op_commit", 2024-03-11T21:36:37.039 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:36:36.533846+0000", 2024-03-11T21:36:37.039 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.0037218709999999999 2024-03-11T21:36:37.039 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:36:37.039 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:36:37.039 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "sub_op_commit_rec", 2024-03-11T21:36:37.039 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:36:36.538924+0000", 2024-03-11T21:36:37.039 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.0050780479999999999 2024-03-11T21:36:37.039 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:36:37.039 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:36:37.039 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "commit_sent", 2024-03-11T21:36:37.039 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:36:36.539008+0000", 2024-03-11T21:36:37.039 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 8.4528000000000001e-05 2024-03-11T21:36:37.039 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:36:37.040 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:36:37.040 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "done", 2024-03-11T21:36:37.040 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:36:36.543685+0000", 2024-03-11T21:36:37.040 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.0046769380000000003 2024-03-11T21:36:37.040 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-03-11T21:36:37.040 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-03-11T21:36:37.040 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-03-11T21:36:37.040 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-03-11T21:36:37.040 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-03-11T21:36:37.040 INFO:teuthology.orchestra.run.smithi092.stdout:} 2024-03-11T21:36:37.046 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:36:37.047 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:36:37.047 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:36:37.148 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:36:37.188 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:36:37.189 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:36:37.189 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:36:37.290 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:36:37.325 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:36:37.325 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:36:37 smithi097 ceph-mon[33317]: pgmap v501: 148 pgs: 30 active+undersized+degraded, 32 active+undersized, 86 active+clean; 4.5 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 1.2 KiB/s rd, 113 MiB/s wr, 1.81k op/s; 37033/223458 objects degraded (16.573%) 2024-03-11T21:36:37.325 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:36:37 smithi097 ceph-mon[33317]: osdmap e279: 8 total, 8 up, 8 in 2024-03-11T21:36:37.326 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:36:37.326 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:36:37.360 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:36:37 smithi092 ceph-mon[29572]: pgmap v501: 148 pgs: 30 active+undersized+degraded, 32 active+undersized, 86 active+clean; 4.5 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 1.2 KiB/s rd, 113 MiB/s wr, 1.81k op/s; 37033/223458 objects degraded (16.573%) 2024-03-11T21:36:37.360 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:36:37 smithi092 ceph-mon[29572]: osdmap e279: 8 total, 8 up, 8 in 2024-03-11T21:36:37.360 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:36:37 smithi092 ceph-mon[37563]: pgmap v501: 148 pgs: 30 active+undersized+degraded, 32 active+undersized, 86 active+clean; 4.5 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 1.2 KiB/s rd, 113 MiB/s wr, 1.81k op/s; 37033/223458 objects degraded (16.573%) 2024-03-11T21:36:37.360 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:36:37 smithi092 ceph-mon[37563]: osdmap e279: 8 total, 8 up, 8 in 2024-03-11T21:36:37.427 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:36:37.463 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:36:37.463 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:36:37.464 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:36:37.486 DEBUG:teuthology.orchestra.run.smithi092:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid 00db0016-dfed-11ee-95c8-87774f69a715 -- ceph osd unset noscrub 2024-03-11T21:36:37.524 INFO:tasks.radosbench.radosbench.0.smithi092.stdout:2024-03-11T21:36:37.543452+0000 min lat: 0.00114249 max lat: 2.50819 avg lat: 0.00829445 2024-03-11T21:36:37.524 INFO:tasks.radosbench.radosbench.0.smithi092.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-03-11T21:36:37.524 INFO:tasks.radosbench.radosbench.0.smithi092.stdout: 40 16 77127 77111 120.466 50.25 0.0467851 0.00829445 2024-03-11T21:36:37.565 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:36:37.600 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:36:37.600 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:36:37.601 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:36:37.702 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:36:37.756 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:36:37.757 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:36:37.757 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:36:37.858 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:36:37.886 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:36:37.887 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:36:37.887 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:36:37.988 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:36:38.023 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:36:38.023 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:36:38.023 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:36:38.124 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:36:38.159 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:36:38.159 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:36:38.160 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:36:38.242 INFO:journalctl@ceph.mgr.y.smithi092.stdout:Mar 11 21:36:37 smithi092 ceph-00db0016-dfed-11ee-95c8-87774f69a715-mgr-y[29801]: ::ffff:172.21.15.97 - - [11/Mar/2024:21:36:37] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-11T21:36:38.261 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:36:38.303 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:36:38.303 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:36:38.304 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:36:38.404 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:36:38.440 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:36:38.440 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:36:38.441 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:36:38.524 INFO:tasks.radosbench.radosbench.0.smithi092.stdout: 41 16 77927 77911 118.747 50 0.00688759 0.00841551 2024-03-11T21:36:38.542 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:36:38.575 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:36:38 smithi097 ceph-mon[33317]: pgmap v503: 148 pgs: 25 active+undersized+degraded, 1 active+recovering+undersized+degraded+remapped, 29 active+undersized, 93 active+clean; 4.5 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 30 MiB/s wr, 484 op/s; 37157/224001 objects degraded (16.588%); 291 KiB/s, 7 objects/s recovering 2024-03-11T21:36:38.576 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:36:38 smithi097 ceph-mon[33317]: from='mgr.14607 ' entity='mgr.y' 2024-03-11T21:36:38.576 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:36:38 smithi097 ceph-mon[33317]: from='mgr.14607 v1:172.21.15.92:0/1702717597' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T21:36:38.576 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:36:38.577 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:36:38.578 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:36:38.609 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:36:38 smithi092 ceph-mon[29572]: pgmap v503: 148 pgs: 25 active+undersized+degraded, 1 active+recovering+undersized+degraded+remapped, 29 active+undersized, 93 active+clean; 4.5 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 30 MiB/s wr, 484 op/s; 37157/224001 objects degraded (16.588%); 291 KiB/s, 7 objects/s recovering 2024-03-11T21:36:38.610 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:36:38 smithi092 ceph-mon[29572]: from='mgr.14607 ' entity='mgr.y' 2024-03-11T21:36:38.610 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:36:38 smithi092 ceph-mon[29572]: from='mgr.14607 v1:172.21.15.92:0/1702717597' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T21:36:38.610 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:36:38 smithi092 ceph-mon[37563]: pgmap v503: 148 pgs: 25 active+undersized+degraded, 1 active+recovering+undersized+degraded+remapped, 29 active+undersized, 93 active+clean; 4.5 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 30 MiB/s wr, 484 op/s; 37157/224001 objects degraded (16.588%); 291 KiB/s, 7 objects/s recovering 2024-03-11T21:36:38.610 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:36:38 smithi092 ceph-mon[37563]: from='mgr.14607 ' entity='mgr.y' 2024-03-11T21:36:38.610 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:36:38 smithi092 ceph-mon[37563]: from='mgr.14607 v1:172.21.15.92:0/1702717597' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T21:36:38.679 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:36:38.718 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:36:38.719 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:36:38.719 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:36:38.820 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:36:38.858 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:36:38.859 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:36:38.859 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:36:38.960 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:36:38.997 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:36:38.998 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:36:38.998 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:36:39.099 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:36:39.137 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:36:39.138 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:36:39.138 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:36:39.239 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:36:39.277 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:36:39.277 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:36:39.278 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:36:39.379 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:36:39.415 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:36:39.415 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:36:39.416 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:36:39.516 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:36:39.524 INFO:tasks.radosbench.radosbench.0.smithi092.stdout: 42 16 78779 78763 117.188 53.25 0.01347 0.00852794 2024-03-11T21:36:39.549 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:36:39.550 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:36:39.551 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:36:39.575 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:36:39 smithi097 ceph-mon[33317]: Health check update: Degraded data redundancy: 37157/224001 objects degraded (16.588%), 26 pgs degraded, 47 pgs undersized (PG_DEGRADED) 2024-03-11T21:36:39.575 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:36:39 smithi097 ceph-mon[33317]: osdmap e280: 8 total, 8 up, 8 in 2024-03-11T21:36:39.609 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:36:39 smithi092 ceph-mon[29572]: Health check update: Degraded data redundancy: 37157/224001 objects degraded (16.588%), 26 pgs degraded, 47 pgs undersized (PG_DEGRADED) 2024-03-11T21:36:39.609 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:36:39 smithi092 ceph-mon[29572]: osdmap e280: 8 total, 8 up, 8 in 2024-03-11T21:36:39.610 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:36:39 smithi092 ceph-mon[37563]: Health check update: Degraded data redundancy: 37157/224001 objects degraded (16.588%), 26 pgs degraded, 47 pgs undersized (PG_DEGRADED) 2024-03-11T21:36:39.610 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:36:39 smithi092 ceph-mon[37563]: osdmap e280: 8 total, 8 up, 8 in 2024-03-11T21:36:39.651 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:36:39.682 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:36:39.682 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:36:39.683 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:36:39.783 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:36:39.821 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:36:39.822 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:36:39.822 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:36:39.873 DEBUG:teuthology.orchestra.run.smithi092:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid 00db0016-dfed-11ee-95c8-87774f69a715 -- ceph daemon osd.1 dump_ops_in_flight 2024-03-11T21:36:39.923 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:36:39.965 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:36:39.966 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:36:39.966 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:36:40.067 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:36:40.106 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:36:40.107 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:36:40.107 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:36:40.208 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:36:40.248 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:36:40.249 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:36:40.249 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:36:40.350 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:36:40.392 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:36:40.393 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:36:40.393 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:36:40.494 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:36:40.526 INFO:tasks.radosbench.radosbench.0.smithi092.stdout: 43 16 79596 79580 115.644 51.0625 0.0329258 0.00864103 2024-03-11T21:36:40.549 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:36:40.549 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:36:40.550 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:36:40.575 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:36:40 smithi097 ceph-mon[33317]: pgmap v505: 148 pgs: 5 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+degraded+remapped, 140 active+clean; 4.7 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 198 B/s rd, 34 MiB/s wr, 541 op/s; 37960/231843 objects degraded (16.373%); 781 KiB/s, 25 objects/s recovering 2024-03-11T21:36:40.600 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:36:40 smithi092 ceph-mon[29572]: pgmap v505: 148 pgs: 5 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+degraded+remapped, 140 active+clean; 4.7 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 198 B/s rd, 34 MiB/s wr, 541 op/s; 37960/231843 objects degraded (16.373%); 781 KiB/s, 25 objects/s recovering 2024-03-11T21:36:40.600 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:36:40 smithi092 ceph-mon[37563]: pgmap v505: 148 pgs: 5 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+degraded+remapped, 140 active+clean; 4.7 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 198 B/s rd, 34 MiB/s wr, 541 op/s; 37960/231843 objects degraded (16.373%); 781 KiB/s, 25 objects/s recovering 2024-03-11T21:36:40.650 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:36:40.683 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:36:40.684 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:36:40.684 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:36:40.785 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:36:40.832 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:36:40.832 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:36:40.833 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:36:40.934 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:36:40.971 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:36:40.973 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:36:40.973 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:36:41.074 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:36:41.107 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:36:41.108 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:36:41.108 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:36:41.209 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:36:41.246 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:36:41.247 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:36:41.247 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:36:41.348 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:36:41.386 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:36:41.387 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:36:41.387 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:36:41.488 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:36:41.526 INFO:tasks.radosbench.radosbench.0.smithi092.stdout: 44 16 80439 80423 114.213 52.6875 0.0529807 0.00874941 2024-03-11T21:36:41.529 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:36:41.530 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:36:41.530 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:36:41.631 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:36:41.675 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:36:41.677 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:36:41.677 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:36:41.778 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:36:41.815 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:36:41.815 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:36:41.816 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:36:41.916 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:36:41.946 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:36:41.946 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:36:41.946 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:36:42.047 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:36:42.078 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:36:42.079 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:36:42.079 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:36:42.180 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:36:42.229 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:36:42.230 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:36:42.230 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:36:42.331 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:36:42.338 INFO:teuthology.orchestra.run.smithi092.stderr:Inferring config /var/lib/ceph/00db0016-dfed-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T21:36:42.341 INFO:teuthology.orchestra.run.smithi092.stderr:Inferring config /var/lib/ceph/00db0016-dfed-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T21:36:42.392 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:36:42.393 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:36:42.393 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:36:42.403 INFO:teuthology.orchestra.run.smithi092.stdout:{ 2024-03-11T21:36:42.403 INFO:teuthology.orchestra.run.smithi092.stdout: "ops": [ 2024-03-11T21:36:42.403 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:36:42.404 INFO:teuthology.orchestra.run.smithi092.stdout: "description": "osd_repop(client.25282.0:81102 9.c e282/278 9:3699e755:::benchmark_data_smithi092_130291_object81101:head v 282'4921, mlcod=265'170)", 2024-03-11T21:36:42.404 INFO:teuthology.orchestra.run.smithi092.stdout: "initiated_at": "2024-03-11T21:36:42.413417+0000", 2024-03-11T21:36:42.404 INFO:teuthology.orchestra.run.smithi092.stdout: "age": 0.0064499359999999999, 2024-03-11T21:36:42.404 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.0064910469999999998, 2024-03-11T21:36:42.404 INFO:teuthology.orchestra.run.smithi092.stdout: "type_data": { 2024-03-11T21:36:42.404 INFO:teuthology.orchestra.run.smithi092.stdout: "flag_point": "started", 2024-03-11T21:36:42.404 INFO:teuthology.orchestra.run.smithi092.stdout: "events": [ 2024-03-11T21:36:42.404 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:36:42.404 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "initiated", 2024-03-11T21:36:42.404 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:36:42.413417+0000", 2024-03-11T21:36:42.404 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-03-11T21:36:42.404 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:36:42.404 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:36:42.404 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "header_read", 2024-03-11T21:36:42.405 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:36:42.413417+0000", 2024-03-11T21:36:42.405 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-03-11T21:36:42.405 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:36:42.405 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:36:42.405 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "throttled", 2024-03-11T21:36:42.405 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:36:42.413418+0000", 2024-03-11T21:36:42.405 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.3540000000000001e-06 2024-03-11T21:36:42.405 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:36:42.405 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:36:42.405 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "all_read", 2024-03-11T21:36:42.405 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:36:42.413490+0000", 2024-03-11T21:36:42.405 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 7.1839000000000001e-05 2024-03-11T21:36:42.405 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:36:42.405 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:36:42.406 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "dispatched", 2024-03-11T21:36:42.406 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:36:42.413505+0000", 2024-03-11T21:36:42.406 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.5353e-05 2024-03-11T21:36:42.406 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:36:42.406 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:36:42.406 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "queued_for_pg", 2024-03-11T21:36:42.406 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:36:42.413514+0000", 2024-03-11T21:36:42.406 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 8.3259999999999996e-06 2024-03-11T21:36:42.406 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:36:42.406 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:36:42.406 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-03-11T21:36:42.406 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:36:42.413599+0000", 2024-03-11T21:36:42.406 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 8.5241000000000006e-05 2024-03-11T21:36:42.406 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:36:42.407 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:36:42.407 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "started", 2024-03-11T21:36:42.407 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:36:42.413623+0000", 2024-03-11T21:36:42.407 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 2.3615999999999999e-05 2024-03-11T21:36:42.407 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-03-11T21:36:42.407 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-03-11T21:36:42.407 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-03-11T21:36:42.407 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:36:42.407 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:36:42.407 INFO:teuthology.orchestra.run.smithi092.stdout: "description": "osd_repop(client.25282.0:81104 9.c e282/278)", 2024-03-11T21:36:42.407 INFO:teuthology.orchestra.run.smithi092.stdout: "initiated_at": "2024-03-11T21:36:42.415734+0000", 2024-03-11T21:36:42.407 INFO:teuthology.orchestra.run.smithi092.stdout: "age": 0.0041323890000000002, 2024-03-11T21:36:42.407 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.0042426979999999996, 2024-03-11T21:36:42.407 INFO:teuthology.orchestra.run.smithi092.stdout: "type_data": { 2024-03-11T21:36:42.407 INFO:teuthology.orchestra.run.smithi092.stdout: "flag_point": "queued for pg", 2024-03-11T21:36:42.408 INFO:teuthology.orchestra.run.smithi092.stdout: "events": [ 2024-03-11T21:36:42.408 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:36:42.408 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "initiated", 2024-03-11T21:36:42.408 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:36:42.415734+0000", 2024-03-11T21:36:42.408 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-03-11T21:36:42.408 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:36:42.408 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:36:42.408 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "header_read", 2024-03-11T21:36:42.408 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:36:42.415734+0000", 2024-03-11T21:36:42.408 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-03-11T21:36:42.408 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:36:42.408 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:36:42.408 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "throttled", 2024-03-11T21:36:42.408 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:36:42.415736+0000", 2024-03-11T21:36:42.408 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.869e-06 2024-03-11T21:36:42.409 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:36:42.409 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:36:42.409 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "all_read", 2024-03-11T21:36:42.409 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:36:42.415784+0000", 2024-03-11T21:36:42.409 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 4.8065000000000001e-05 2024-03-11T21:36:42.409 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:36:42.409 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:36:42.409 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "dispatched", 2024-03-11T21:36:42.409 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:36:42.415798+0000", 2024-03-11T21:36:42.409 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.3715e-05 2024-03-11T21:36:42.409 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:36:42.409 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:36:42.409 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "queued_for_pg", 2024-03-11T21:36:42.409 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:36:42.415808+0000", 2024-03-11T21:36:42.409 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 9.7200000000000001e-06 2024-03-11T21:36:42.410 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-03-11T21:36:42.410 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-03-11T21:36:42.410 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-03-11T21:36:42.410 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:36:42.410 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:36:42.410 INFO:teuthology.orchestra.run.smithi092.stdout: "description": "osd_repop(client.25282.0:81105 9.8 e282/278)", 2024-03-11T21:36:42.410 INFO:teuthology.orchestra.run.smithi092.stdout: "initiated_at": "2024-03-11T21:36:42.417464+0000", 2024-03-11T21:36:42.410 INFO:teuthology.orchestra.run.smithi092.stdout: "age": 0.0024022689999999998, 2024-03-11T21:36:42.410 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.0025656160000000002, 2024-03-11T21:36:42.410 INFO:teuthology.orchestra.run.smithi092.stdout: "type_data": { 2024-03-11T21:36:42.410 INFO:teuthology.orchestra.run.smithi092.stdout: "flag_point": "queued for pg", 2024-03-11T21:36:42.410 INFO:teuthology.orchestra.run.smithi092.stdout: "events": [ 2024-03-11T21:36:42.410 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:36:42.410 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "initiated", 2024-03-11T21:36:42.411 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:36:42.417464+0000", 2024-03-11T21:36:42.411 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-03-11T21:36:42.411 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:36:42.411 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:36:42.411 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "header_read", 2024-03-11T21:36:42.411 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:36:42.417464+0000", 2024-03-11T21:36:42.411 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-03-11T21:36:42.411 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:36:42.411 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:36:42.411 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "throttled", 2024-03-11T21:36:42.411 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:36:42.417467+0000", 2024-03-11T21:36:42.411 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 2.4719999999999998e-06 2024-03-11T21:36:42.411 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:36:42.411 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:36:42.411 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "all_read", 2024-03-11T21:36:42.412 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:36:42.417501+0000", 2024-03-11T21:36:42.412 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 3.3896999999999997e-05 2024-03-11T21:36:42.412 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:36:42.412 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:36:42.412 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "dispatched", 2024-03-11T21:36:42.412 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:36:42.417518+0000", 2024-03-11T21:36:42.412 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.7221999999999999e-05 2024-03-11T21:36:42.412 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:36:42.412 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:36:42.412 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "queued_for_pg", 2024-03-11T21:36:42.412 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:36:42.417528+0000", 2024-03-11T21:36:42.412 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 9.6209999999999999e-06 2024-03-11T21:36:42.412 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-03-11T21:36:42.412 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-03-11T21:36:42.413 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-03-11T21:36:42.413 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:36:42.413 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:36:42.413 INFO:teuthology.orchestra.run.smithi092.stdout: "description": "osd_repop(client.25282.0:81108 9.7 e282/278)", 2024-03-11T21:36:42.413 INFO:teuthology.orchestra.run.smithi092.stdout: "initiated_at": "2024-03-11T21:36:42.417929+0000", 2024-03-11T21:36:42.413 INFO:teuthology.orchestra.run.smithi092.stdout: "age": 0.001937334, 2024-03-11T21:36:42.413 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.002148568, 2024-03-11T21:36:42.413 INFO:teuthology.orchestra.run.smithi092.stdout: "type_data": { 2024-03-11T21:36:42.413 INFO:teuthology.orchestra.run.smithi092.stdout: "flag_point": "queued for pg", 2024-03-11T21:36:42.413 INFO:teuthology.orchestra.run.smithi092.stdout: "events": [ 2024-03-11T21:36:42.413 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:36:42.413 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "initiated", 2024-03-11T21:36:42.413 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:36:42.417929+0000", 2024-03-11T21:36:42.413 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-03-11T21:36:42.413 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:36:42.414 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:36:42.414 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "header_read", 2024-03-11T21:36:42.414 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:36:42.417929+0000", 2024-03-11T21:36:42.414 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-03-11T21:36:42.414 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:36:42.414 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:36:42.414 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "throttled", 2024-03-11T21:36:42.414 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:36:42.417931+0000", 2024-03-11T21:36:42.414 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.2249999999999999e-06 2024-03-11T21:36:42.414 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:36:42.414 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:36:42.414 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "all_read", 2024-03-11T21:36:42.414 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:36:42.417963+0000", 2024-03-11T21:36:42.414 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 3.2793999999999998e-05 2024-03-11T21:36:42.414 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:36:42.415 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:36:42.415 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "dispatched", 2024-03-11T21:36:42.415 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:36:42.417973+0000", 2024-03-11T21:36:42.415 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 9.7089999999999997e-06 2024-03-11T21:36:42.415 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:36:42.415 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:36:42.415 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "queued_for_pg", 2024-03-11T21:36:42.415 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:36:42.417980+0000", 2024-03-11T21:36:42.415 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 6.4840000000000001e-06 2024-03-11T21:36:42.415 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-03-11T21:36:42.415 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-03-11T21:36:42.415 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-03-11T21:36:42.415 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:36:42.415 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:36:42.416 INFO:teuthology.orchestra.run.smithi092.stdout: "description": "osd_repop(client.25282.0:81106 9.f e282/278)", 2024-03-11T21:36:42.416 INFO:teuthology.orchestra.run.smithi092.stdout: "initiated_at": "2024-03-11T21:36:42.417991+0000", 2024-03-11T21:36:42.416 INFO:teuthology.orchestra.run.smithi092.stdout: "age": 0.001875778, 2024-03-11T21:36:42.416 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.0021350219999999999, 2024-03-11T21:36:42.416 INFO:teuthology.orchestra.run.smithi092.stdout: "type_data": { 2024-03-11T21:36:42.416 INFO:teuthology.orchestra.run.smithi092.stdout: "flag_point": "queued for pg", 2024-03-11T21:36:42.416 INFO:teuthology.orchestra.run.smithi092.stdout: "events": [ 2024-03-11T21:36:42.416 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:36:42.416 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "initiated", 2024-03-11T21:36:42.416 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:36:42.417991+0000", 2024-03-11T21:36:42.416 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-03-11T21:36:42.416 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:36:42.416 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:36:42.416 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "header_read", 2024-03-11T21:36:42.416 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:36:42.417991+0000", 2024-03-11T21:36:42.417 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-03-11T21:36:42.417 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:36:42.417 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:36:42.417 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "throttled", 2024-03-11T21:36:42.417 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:36:42.417991+0000", 2024-03-11T21:36:42.417 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 4.9999999999999998e-07 2024-03-11T21:36:42.417 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:36:42.417 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:36:42.417 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "all_read", 2024-03-11T21:36:42.417 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:36:42.418027+0000", 2024-03-11T21:36:42.417 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 3.553e-05 2024-03-11T21:36:42.418 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:36:42.418 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:36:42.418 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "dispatched", 2024-03-11T21:36:42.418 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:36:42.418033+0000", 2024-03-11T21:36:42.418 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 5.7239999999999997e-06 2024-03-11T21:36:42.418 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:36:42.418 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:36:42.418 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "queued_for_pg", 2024-03-11T21:36:42.419 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:36:42.418037+0000", 2024-03-11T21:36:42.419 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 4.4209999999999997e-06 2024-03-11T21:36:42.419 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-03-11T21:36:42.419 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-03-11T21:36:42.419 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-03-11T21:36:42.419 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:36:42.419 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:36:42.419 INFO:teuthology.orchestra.run.smithi092.stdout: "description": "osd_repop(client.25282.0:81107 9.2 e282/278 9:45783e81:::benchmark_data_smithi092_130291_object81106:head v 282'5085, mlcod=0'0)", 2024-03-11T21:36:42.419 INFO:teuthology.orchestra.run.smithi092.stdout: "initiated_at": "2024-03-11T21:36:42.419786+0000", 2024-03-11T21:36:42.419 INFO:teuthology.orchestra.run.smithi092.stdout: "age": 8.1176999999999999e-05, 2024-03-11T21:36:42.419 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00039365, 2024-03-11T21:36:42.419 INFO:teuthology.orchestra.run.smithi092.stdout: "type_data": { 2024-03-11T21:36:42.419 INFO:teuthology.orchestra.run.smithi092.stdout: "flag_point": "started", 2024-03-11T21:36:42.419 INFO:teuthology.orchestra.run.smithi092.stdout: "events": [ 2024-03-11T21:36:42.420 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:36:42.420 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "initiated", 2024-03-11T21:36:42.420 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:36:42.419786+0000", 2024-03-11T21:36:42.420 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-03-11T21:36:42.420 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:36:42.420 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:36:42.420 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "header_read", 2024-03-11T21:36:42.420 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:36:42.419786+0000", 2024-03-11T21:36:42.420 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-03-11T21:36:42.420 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:36:42.420 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:36:42.420 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "throttled", 2024-03-11T21:36:42.420 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:36:42.419787+0000", 2024-03-11T21:36:42.420 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.1570000000000001e-06 2024-03-11T21:36:42.421 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:36:42.421 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:36:42.421 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "all_read", 2024-03-11T21:36:42.421 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:36:42.419808+0000", 2024-03-11T21:36:42.421 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 2.1732000000000001e-05 2024-03-11T21:36:42.421 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:36:42.421 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:36:42.421 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "dispatched", 2024-03-11T21:36:42.421 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:36:42.419820+0000", 2024-03-11T21:36:42.421 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.1072e-05 2024-03-11T21:36:42.421 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:36:42.421 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:36:42.421 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "queued_for_pg", 2024-03-11T21:36:42.421 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:36:42.419827+0000", 2024-03-11T21:36:42.422 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 7.0589999999999997e-06 2024-03-11T21:36:42.422 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:36:42.422 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:36:42.422 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-03-11T21:36:42.422 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:36:42.419918+0000", 2024-03-11T21:36:42.422 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 9.1119000000000006e-05 2024-03-11T21:36:42.422 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:36:42.422 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:36:42.422 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "started", 2024-03-11T21:36:42.422 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:36:42.419942+0000", 2024-03-11T21:36:42.422 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 2.4275000000000001e-05 2024-03-11T21:36:42.422 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-03-11T21:36:42.422 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-03-11T21:36:42.422 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-03-11T21:36:42.423 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:36:42.423 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:36:42.423 INFO:teuthology.orchestra.run.smithi092.stdout: "description": "MOSDPGPush(9.7 281/278 [PushOp(9:e56f1441:::benchmark_data_smithi092_130291_object2224:head, version: 265'128, data_included: [0~65536], data_size: 65536, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(9:e56f1441:::benchmark_data_smithi092_130291_object2224:head@265'128, size: 65536, copy_subset: [(0, 65536)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 65536, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-03-11T21:36:42.423 INFO:teuthology.orchestra.run.smithi092.stdout: "initiated_at": "2024-03-11T21:36:42.355104+0000", 2024-03-11T21:36:42.423 INFO:teuthology.orchestra.run.smithi092.stdout: "age": 0.064762261000000002, 2024-03-11T21:36:42.423 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.065162465000000003, 2024-03-11T21:36:42.423 INFO:teuthology.orchestra.run.smithi092.stdout: "type_data": { 2024-03-11T21:36:42.423 INFO:teuthology.orchestra.run.smithi092.stdout: "flag_point": "queued for pg", 2024-03-11T21:36:42.423 INFO:teuthology.orchestra.run.smithi092.stdout: "events": [ 2024-03-11T21:36:42.423 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:36:42.423 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "initiated", 2024-03-11T21:36:42.423 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:36:42.355104+0000", 2024-03-11T21:36:42.423 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-03-11T21:36:42.423 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:36:42.423 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:36:42.424 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "header_read", 2024-03-11T21:36:42.424 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:36:42.355104+0000", 2024-03-11T21:36:42.424 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-03-11T21:36:42.424 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:36:42.424 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:36:42.424 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "throttled", 2024-03-11T21:36:42.424 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:36:42.355106+0000", 2024-03-11T21:36:42.424 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.697e-06 2024-03-11T21:36:42.424 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:36:42.424 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:36:42.424 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "all_read", 2024-03-11T21:36:42.424 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:36:42.355159+0000", 2024-03-11T21:36:42.424 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 5.2510999999999999e-05 2024-03-11T21:36:42.424 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:36:42.425 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:36:42.425 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "dispatched", 2024-03-11T21:36:42.425 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:36:42.355181+0000", 2024-03-11T21:36:42.425 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 2.2036e-05 2024-03-11T21:36:42.425 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:36:42.425 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:36:42.425 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "queued_for_pg", 2024-03-11T21:36:42.425 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:36:42.355198+0000", 2024-03-11T21:36:42.425 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.7484000000000001e-05 2024-03-11T21:36:42.425 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-03-11T21:36:42.425 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-03-11T21:36:42.425 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-03-11T21:36:42.425 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:36:42.425 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:36:42.425 INFO:teuthology.orchestra.run.smithi092.stdout: "description": "MOSDPGPush(9.7 282/278 [PushOp(9:e25080d5:::benchmark_data_smithi092_130291_object2228:head, version: 265'129, data_included: [0~65536], data_size: 65536, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(9:e25080d5:::benchmark_data_smithi092_130291_object2228:head@265'129, size: 65536, copy_subset: [(0, 65536)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 65536, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-03-11T21:36:42.426 INFO:teuthology.orchestra.run.smithi092.stdout: "initiated_at": "2024-03-11T21:36:42.380949+0000", 2024-03-11T21:36:42.426 INFO:teuthology.orchestra.run.smithi092.stdout: "age": 0.038917602000000003, 2024-03-11T21:36:42.426 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.039389901999999997, 2024-03-11T21:36:42.426 INFO:teuthology.orchestra.run.smithi092.stdout: "type_data": { 2024-03-11T21:36:42.426 INFO:teuthology.orchestra.run.smithi092.stdout: "flag_point": "queued for pg", 2024-03-11T21:36:42.426 INFO:teuthology.orchestra.run.smithi092.stdout: "events": [ 2024-03-11T21:36:42.426 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:36:42.426 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "initiated", 2024-03-11T21:36:42.426 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:36:42.380949+0000", 2024-03-11T21:36:42.426 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-03-11T21:36:42.426 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:36:42.426 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:36:42.426 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "header_read", 2024-03-11T21:36:42.427 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:36:42.380949+0000", 2024-03-11T21:36:42.427 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-03-11T21:36:42.427 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:36:42.427 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:36:42.427 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "throttled", 2024-03-11T21:36:42.427 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:36:42.380951+0000", 2024-03-11T21:36:42.427 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 2.1220000000000002e-06 2024-03-11T21:36:42.427 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:36:42.427 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:36:42.427 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "all_read", 2024-03-11T21:36:42.427 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:36:42.381013+0000", 2024-03-11T21:36:42.427 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 6.1614999999999999e-05 2024-03-11T21:36:42.427 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:36:42.427 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:36:42.427 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "dispatched", 2024-03-11T21:36:42.428 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:36:42.381034+0000", 2024-03-11T21:36:42.428 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 2.1520000000000001e-05 2024-03-11T21:36:42.428 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:36:42.428 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:36:42.428 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "queued_for_pg", 2024-03-11T21:36:42.428 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:36:42.381052+0000", 2024-03-11T21:36:42.428 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.7875000000000001e-05 2024-03-11T21:36:42.428 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-03-11T21:36:42.428 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-03-11T21:36:42.428 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-03-11T21:36:42.428 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:36:42.428 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:36:42.428 INFO:teuthology.orchestra.run.smithi092.stdout: "description": "osd_repop(client.25282.0:81086 9.7 e282/278 9:e6690d3e:::benchmark_data_smithi092_130291_object81085:head v 282'5067, mlcod=265'127)", 2024-03-11T21:36:42.428 INFO:teuthology.orchestra.run.smithi092.stdout: "initiated_at": "2024-03-11T21:36:42.388917+0000", 2024-03-11T21:36:42.428 INFO:teuthology.orchestra.run.smithi092.stdout: "age": 0.030949586000000001, 2024-03-11T21:36:42.429 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.031473091000000002, 2024-03-11T21:36:42.429 INFO:teuthology.orchestra.run.smithi092.stdout: "type_data": { 2024-03-11T21:36:42.429 INFO:teuthology.orchestra.run.smithi092.stdout: "flag_point": "started", 2024-03-11T21:36:42.429 INFO:teuthology.orchestra.run.smithi092.stdout: "events": [ 2024-03-11T21:36:42.429 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:36:42.429 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "initiated", 2024-03-11T21:36:42.429 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:36:42.388917+0000", 2024-03-11T21:36:42.429 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-03-11T21:36:42.429 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:36:42.429 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:36:42.429 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "header_read", 2024-03-11T21:36:42.429 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:36:42.388917+0000", 2024-03-11T21:36:42.429 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-03-11T21:36:42.429 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:36:42.429 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:36:42.430 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "throttled", 2024-03-11T21:36:42.430 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:36:42.388919+0000", 2024-03-11T21:36:42.430 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.5510000000000001e-06 2024-03-11T21:36:42.430 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:36:42.430 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:36:42.430 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "all_read", 2024-03-11T21:36:42.430 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:36:42.388968+0000", 2024-03-11T21:36:42.430 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 4.9509000000000003e-05 2024-03-11T21:36:42.430 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:36:42.430 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:36:42.430 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "dispatched", 2024-03-11T21:36:42.430 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:36:42.388983+0000", 2024-03-11T21:36:42.430 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.4769999999999999e-05 2024-03-11T21:36:42.430 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:36:42.431 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:36:42.431 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "queued_for_pg", 2024-03-11T21:36:42.431 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:36:42.388992+0000", 2024-03-11T21:36:42.431 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 9.1549999999999996e-06 2024-03-11T21:36:42.431 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:36:42.431 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:36:42.431 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-03-11T21:36:42.431 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:36:42.407050+0000", 2024-03-11T21:36:42.431 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.018058026000000001 2024-03-11T21:36:42.431 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:36:42.431 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:36:42.431 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "started", 2024-03-11T21:36:42.431 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:36:42.407076+0000", 2024-03-11T21:36:42.431 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 2.6064e-05 2024-03-11T21:36:42.432 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-03-11T21:36:42.432 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-03-11T21:36:42.432 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-03-11T21:36:42.432 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:36:42.432 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:36:42.432 INFO:teuthology.orchestra.run.smithi092.stdout: "description": "osd_repop(client.25282.0:81090 9.4 e282/278 9:2095224b:::benchmark_data_smithi092_130291_object81089:head v 282'5119, mlcod=0'0)", 2024-03-11T21:36:42.432 INFO:teuthology.orchestra.run.smithi092.stdout: "initiated_at": "2024-03-11T21:36:42.394970+0000", 2024-03-11T21:36:42.432 INFO:teuthology.orchestra.run.smithi092.stdout: "age": 0.02489651, 2024-03-11T21:36:42.432 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.025484334000000001, 2024-03-11T21:36:42.432 INFO:teuthology.orchestra.run.smithi092.stdout: "type_data": { 2024-03-11T21:36:42.432 INFO:teuthology.orchestra.run.smithi092.stdout: "flag_point": "started", 2024-03-11T21:36:42.432 INFO:teuthology.orchestra.run.smithi092.stdout: "events": [ 2024-03-11T21:36:42.432 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:36:42.432 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "initiated", 2024-03-11T21:36:42.432 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:36:42.394970+0000", 2024-03-11T21:36:42.433 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-03-11T21:36:42.433 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:36:42.433 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:36:42.433 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "header_read", 2024-03-11T21:36:42.433 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:36:42.394970+0000", 2024-03-11T21:36:42.433 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-03-11T21:36:42.433 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:36:42.433 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:36:42.433 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "throttled", 2024-03-11T21:36:42.433 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:36:42.394971+0000", 2024-03-11T21:36:42.433 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 5.8100000000000003e-07 2024-03-11T21:36:42.433 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:36:42.433 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:36:42.433 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "all_read", 2024-03-11T21:36:42.434 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:36:42.394992+0000", 2024-03-11T21:36:42.434 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 2.1330000000000001e-05 2024-03-11T21:36:42.434 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:36:42.434 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:36:42.434 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "dispatched", 2024-03-11T21:36:42.434 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:36:42.394998+0000", 2024-03-11T21:36:42.434 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 5.8529999999999997e-06 2024-03-11T21:36:42.434 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:36:42.434 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:36:42.434 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "queued_for_pg", 2024-03-11T21:36:42.434 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:36:42.395012+0000", 2024-03-11T21:36:42.434 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.4005999999999999e-05 2024-03-11T21:36:42.434 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:36:42.435 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:36:42.435 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-03-11T21:36:42.435 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:36:42.403293+0000", 2024-03-11T21:36:42.435 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.0082806760000000007 2024-03-11T21:36:42.435 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:36:42.435 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:36:42.435 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "started", 2024-03-11T21:36:42.435 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:36:42.403323+0000", 2024-03-11T21:36:42.435 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 3.008e-05 2024-03-11T21:36:42.435 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-03-11T21:36:42.435 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-03-11T21:36:42.435 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-03-11T21:36:42.435 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:36:42.435 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:36:42.435 INFO:teuthology.orchestra.run.smithi092.stdout: "description": "osd_repop(client.25282.0:81094 9.7 e282/278)", 2024-03-11T21:36:42.436 INFO:teuthology.orchestra.run.smithi092.stdout: "initiated_at": "2024-03-11T21:36:42.403459+0000", 2024-03-11T21:36:42.436 INFO:teuthology.orchestra.run.smithi092.stdout: "age": 0.016407635, 2024-03-11T21:36:42.436 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.017053846000000001, 2024-03-11T21:36:42.436 INFO:teuthology.orchestra.run.smithi092.stdout: "type_data": { 2024-03-11T21:36:42.436 INFO:teuthology.orchestra.run.smithi092.stdout: "flag_point": "queued for pg", 2024-03-11T21:36:42.436 INFO:teuthology.orchestra.run.smithi092.stdout: "events": [ 2024-03-11T21:36:42.436 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:36:42.436 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "initiated", 2024-03-11T21:36:42.436 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:36:42.403459+0000", 2024-03-11T21:36:42.436 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-03-11T21:36:42.436 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:36:42.436 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:36:42.436 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "header_read", 2024-03-11T21:36:42.436 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:36:42.403459+0000", 2024-03-11T21:36:42.437 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-03-11T21:36:42.437 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:36:42.437 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:36:42.437 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "throttled", 2024-03-11T21:36:42.437 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:36:42.403461+0000", 2024-03-11T21:36:42.437 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.6050000000000001e-06 2024-03-11T21:36:42.437 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:36:42.437 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:36:42.437 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "all_read", 2024-03-11T21:36:42.437 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:36:42.403509+0000", 2024-03-11T21:36:42.437 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 4.7981e-05 2024-03-11T21:36:42.437 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:36:42.437 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:36:42.437 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "dispatched", 2024-03-11T21:36:42.437 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:36:42.403521+0000", 2024-03-11T21:36:42.438 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.2588e-05 2024-03-11T21:36:42.438 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:36:42.438 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:36:42.438 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "queued_for_pg", 2024-03-11T21:36:42.438 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:36:42.403530+0000", 2024-03-11T21:36:42.438 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 8.259e-06 2024-03-11T21:36:42.438 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-03-11T21:36:42.438 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-03-11T21:36:42.438 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-03-11T21:36:42.438 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:36:42.438 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:36:42.439 INFO:teuthology.orchestra.run.smithi092.stdout: "description": "osd_repop(client.25282.0:81092 9.8 e282/278 9:14fa328b:::benchmark_data_smithi092_130291_object81091:head v 282'5159, mlcod=0'0)", 2024-03-11T21:36:42.439 INFO:teuthology.orchestra.run.smithi092.stdout: "initiated_at": "2024-03-11T21:36:42.408625+0000", 2024-03-11T21:36:42.439 INFO:teuthology.orchestra.run.smithi092.stdout: "age": 0.011242106, 2024-03-11T21:36:42.439 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.011939747000000001, 2024-03-11T21:36:42.439 INFO:teuthology.orchestra.run.smithi092.stdout: "type_data": { 2024-03-11T21:36:42.439 INFO:teuthology.orchestra.run.smithi092.stdout: "flag_point": "started", 2024-03-11T21:36:42.439 INFO:teuthology.orchestra.run.smithi092.stdout: "events": [ 2024-03-11T21:36:42.439 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:36:42.439 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "initiated", 2024-03-11T21:36:42.439 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:36:42.408625+0000", 2024-03-11T21:36:42.439 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-03-11T21:36:42.439 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:36:42.439 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:36:42.440 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "header_read", 2024-03-11T21:36:42.440 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:36:42.408625+0000", 2024-03-11T21:36:42.440 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-03-11T21:36:42.440 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:36:42.440 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:36:42.440 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "throttled", 2024-03-11T21:36:42.440 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:36:42.408626+0000", 2024-03-11T21:36:42.440 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.5519999999999999e-06 2024-03-11T21:36:42.440 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:36:42.440 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:36:42.441 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "all_read", 2024-03-11T21:36:42.441 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:36:42.408657+0000", 2024-03-11T21:36:42.441 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 3.0738000000000003e-05 2024-03-11T21:36:42.441 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:36:42.441 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:36:42.441 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "dispatched", 2024-03-11T21:36:42.441 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:36:42.408670+0000", 2024-03-11T21:36:42.441 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.3083e-05 2024-03-11T21:36:42.441 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:36:42.441 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:36:42.441 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "queued_for_pg", 2024-03-11T21:36:42.441 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:36:42.408678+0000", 2024-03-11T21:36:42.441 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 7.5209999999999997e-06 2024-03-11T21:36:42.442 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:36:42.442 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:36:42.442 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-03-11T21:36:42.442 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:36:42.415978+0000", 2024-03-11T21:36:42.442 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.007300829 2024-03-11T21:36:42.442 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:36:42.442 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:36:42.442 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "started", 2024-03-11T21:36:42.442 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:36:42.416007+0000", 2024-03-11T21:36:42.442 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 2.8296000000000001e-05 2024-03-11T21:36:42.442 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-03-11T21:36:42.442 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-03-11T21:36:42.442 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-03-11T21:36:42.442 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:36:42.442 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:36:42.443 INFO:teuthology.orchestra.run.smithi092.stdout: "description": "osd_repop(client.25282.0:81095 9.7 e282/278)", 2024-03-11T21:36:42.443 INFO:teuthology.orchestra.run.smithi092.stdout: "initiated_at": "2024-03-11T21:36:42.408700+0000", 2024-03-11T21:36:42.443 INFO:teuthology.orchestra.run.smithi092.stdout: "age": 0.011166916000000001, 2024-03-11T21:36:42.443 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.011922554, 2024-03-11T21:36:42.443 INFO:teuthology.orchestra.run.smithi092.stdout: "type_data": { 2024-03-11T21:36:42.443 INFO:teuthology.orchestra.run.smithi092.stdout: "flag_point": "queued for pg", 2024-03-11T21:36:42.443 INFO:teuthology.orchestra.run.smithi092.stdout: "events": [ 2024-03-11T21:36:42.443 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:36:42.443 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "initiated", 2024-03-11T21:36:42.443 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:36:42.408700+0000", 2024-03-11T21:36:42.443 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-03-11T21:36:42.443 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:36:42.443 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:36:42.443 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "header_read", 2024-03-11T21:36:42.444 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:36:42.408700+0000", 2024-03-11T21:36:42.444 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-03-11T21:36:42.444 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:36:42.444 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:36:42.444 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "throttled", 2024-03-11T21:36:42.444 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:36:42.408700+0000", 2024-03-11T21:36:42.444 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 6.4199999999999995e-07 2024-03-11T21:36:42.444 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:36:42.444 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:36:42.444 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "all_read", 2024-03-11T21:36:42.444 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:36:42.408738+0000", 2024-03-11T21:36:42.444 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 3.7355999999999999e-05 2024-03-11T21:36:42.444 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:36:42.445 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:36:42.445 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "dispatched", 2024-03-11T21:36:42.445 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:36:42.408743+0000", 2024-03-11T21:36:42.445 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 5.553e-06 2024-03-11T21:36:42.445 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:36:42.445 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:36:42.445 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "queued_for_pg", 2024-03-11T21:36:42.445 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:36:42.408748+0000", 2024-03-11T21:36:42.445 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 4.2660000000000003e-06 2024-03-11T21:36:42.445 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-03-11T21:36:42.445 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-03-11T21:36:42.445 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-03-11T21:36:42.445 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-03-11T21:36:42.445 INFO:teuthology.orchestra.run.smithi092.stdout: ], 2024-03-11T21:36:42.445 INFO:teuthology.orchestra.run.smithi092.stdout: "num_ops": 13 2024-03-11T21:36:42.446 INFO:teuthology.orchestra.run.smithi092.stdout:} 2024-03-11T21:36:42.446 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:36:42 smithi092 ceph-mon[29572]: purged_snaps scrub starts 2024-03-11T21:36:42.446 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:36:42 smithi092 ceph-mon[29572]: purged_snaps scrub ok 2024-03-11T21:36:42.446 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:36:42 smithi092 ceph-mon[29572]: pgmap v506: 148 pgs: 5 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+degraded+remapped, 140 active+clean; 4.7 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 166 B/s rd, 28 MiB/s wr, 453 op/s; 37960/231843 objects degraded (16.373%); 654 KiB/s, 21 objects/s recovering 2024-03-11T21:36:42.446 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:36:42 smithi092 ceph-mon[29572]: osdmap e281: 8 total, 8 up, 8 in 2024-03-11T21:36:42.447 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:36:42 smithi092 ceph-mon[37563]: purged_snaps scrub starts 2024-03-11T21:36:42.447 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:36:42 smithi092 ceph-mon[37563]: purged_snaps scrub ok 2024-03-11T21:36:42.447 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:36:42 smithi092 ceph-mon[37563]: pgmap v506: 148 pgs: 5 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+degraded+remapped, 140 active+clean; 4.7 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 166 B/s rd, 28 MiB/s wr, 453 op/s; 37960/231843 objects degraded (16.373%); 654 KiB/s, 21 objects/s recovering 2024-03-11T21:36:42.447 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:36:42 smithi092 ceph-mon[37563]: osdmap e281: 8 total, 8 up, 8 in 2024-03-11T21:36:42.493 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:36:42.526 INFO:tasks.radosbench.radosbench.0.smithi092.stdout: 45 16 81203 81187 112.736 47.75 0.00754733 0.00886559 2024-03-11T21:36:42.533 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:36:42.534 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:36:42.534 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:36:42.574 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:36:42 smithi097 ceph-mon[33317]: purged_snaps scrub starts 2024-03-11T21:36:42.574 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:36:42 smithi097 ceph-mon[33317]: purged_snaps scrub ok 2024-03-11T21:36:42.574 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:36:42 smithi097 ceph-mon[33317]: pgmap v506: 148 pgs: 5 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+degraded+remapped, 140 active+clean; 4.7 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 166 B/s rd, 28 MiB/s wr, 453 op/s; 37960/231843 objects degraded (16.373%); 654 KiB/s, 21 objects/s recovering 2024-03-11T21:36:42.574 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:36:42 smithi097 ceph-mon[33317]: osdmap e281: 8 total, 8 up, 8 in 2024-03-11T21:36:42.635 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:36:42.674 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:36:42.675 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:36:42.675 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:36:42.776 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:36:42.816 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:36:42.817 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:36:42.817 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:36:42.825 INFO:journalctl@ceph.iscsi.iscsi.a.smithi097.stdout:Mar 11 21:36:42 smithi097 ceph-00db0016-dfed-11ee-95c8-87774f69a715-iscsi-iscsi-a[60943]: debug there is no tcmu-runner data available 2024-03-11T21:36:42.919 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:36:42.957 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:36:42.958 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:36:42.959 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:36:43.059 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:36:43.110 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:36:43.111 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:36:43.112 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:36:43.212 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:36:43.239 INFO:teuthology.orchestra.run.smithi092.stdout:osd.0: {} 2024-03-11T21:36:43.249 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:36:43.250 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:36:43.250 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:36:43.284 INFO:teuthology.orchestra.run.smithi092.stdout:osd.1: {} 2024-03-11T21:36:43.327 INFO:teuthology.orchestra.run.smithi092.stdout:osd.2: {} 2024-03-11T21:36:43.351 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:36:43.378 INFO:teuthology.orchestra.run.smithi092.stdout:osd.3: {} 2024-03-11T21:36:43.392 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:36:43.392 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:36:43.392 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:36:43.416 INFO:teuthology.orchestra.run.smithi092.stdout:osd.4: {} 2024-03-11T21:36:43.452 INFO:teuthology.orchestra.run.smithi092.stdout:osd.5: {} 2024-03-11T21:36:43.490 INFO:teuthology.orchestra.run.smithi092.stdout:osd.6: {} 2024-03-11T21:36:43.494 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:36:43.523 INFO:teuthology.orchestra.run.smithi092.stdout:osd.7: {} 2024-03-11T21:36:43.523 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:36:43 smithi092 ceph-mon[29572]: purged_snaps scrub starts 2024-03-11T21:36:43.523 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:36:43 smithi092 ceph-mon[29572]: purged_snaps scrub ok 2024-03-11T21:36:43.523 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:36:43 smithi092 ceph-mon[29572]: osdmap e282: 8 total, 8 up, 8 in 2024-03-11T21:36:43.524 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:36:43 smithi092 ceph-mon[29572]: from='client.24469 v1:172.21.15.97:0/404208310' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:36:43.524 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:36:43 smithi092 ceph-mon[29572]: from='client.? v1:172.21.15.92:0/942505522' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-11T21:36:43.524 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:36:43 smithi092 ceph-mon[29572]: Health check update: Degraded data redundancy: 37960/231843 objects degraded (16.373%), 8 pgs degraded (PG_DEGRADED) 2024-03-11T21:36:43.524 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:36:43 smithi092 ceph-mon[37563]: purged_snaps scrub starts 2024-03-11T21:36:43.524 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:36:43 smithi092 ceph-mon[37563]: purged_snaps scrub ok 2024-03-11T21:36:43.524 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:36:43 smithi092 ceph-mon[37563]: osdmap e282: 8 total, 8 up, 8 in 2024-03-11T21:36:43.524 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:36:43 smithi092 ceph-mon[37563]: from='client.24469 v1:172.21.15.97:0/404208310' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:36:43.524 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:36:43 smithi092 ceph-mon[37563]: from='client.? v1:172.21.15.92:0/942505522' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-11T21:36:43.524 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:36:43 smithi092 ceph-mon[37563]: Health check update: Degraded data redundancy: 37960/231843 objects degraded (16.373%), 8 pgs degraded (PG_DEGRADED) 2024-03-11T21:36:43.525 INFO:teuthology.orchestra.run.smithi092.stderr:osd.0: osd_enable_op_tracker = '' 2024-03-11T21:36:43.525 INFO:teuthology.orchestra.run.smithi092.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-11T21:36:43.525 INFO:teuthology.orchestra.run.smithi092.stderr:osd.2: osd_enable_op_tracker = '' 2024-03-11T21:36:43.526 INFO:teuthology.orchestra.run.smithi092.stderr:osd.3: osd_delete_sleep = '' osd_delete_sleep_hdd = '' osd_delete_sleep_hybrid = '' osd_delete_sleep_ssd = '' osd_enable_op_tracker = '' osd_recovery_max_active = '' osd_recovery_max_active_hdd = '' osd_recovery_max_active_ssd = '' osd_recovery_sleep = '' osd_recovery_sleep_hdd = '' osd_recovery_sleep_hybrid = '' osd_recovery_sleep_ssd = '' osd_scrub_sleep = '' osd_snap_trim_sleep = '' osd_snap_trim_sleep_hdd = '' osd_snap_trim_sleep_hybrid = '' osd_snap_trim_sleep_ssd = '' 2024-03-11T21:36:43.526 INFO:teuthology.orchestra.run.smithi092.stderr:osd.4: osd_enable_op_tracker = '' 2024-03-11T21:36:43.526 INFO:teuthology.orchestra.run.smithi092.stderr:osd.5: osd_enable_op_tracker = '' 2024-03-11T21:36:43.526 INFO:teuthology.orchestra.run.smithi092.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-11T21:36:43.526 INFO:teuthology.orchestra.run.smithi092.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-11T21:36:43.526 INFO:tasks.radosbench.radosbench.0.smithi092.stdout: 46 16 81974 81958 111.333 48.1875 0.0137823 0.00897669 2024-03-11T21:36:43.528 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:36:43.529 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:36:43.529 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:36:43.630 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:36:43.666 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:36:43.667 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:36:43.667 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:36:43.767 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:36:43.809 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:36:43.810 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:36:43.810 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:36:43.825 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:36:43 smithi097 ceph-mon[33317]: purged_snaps scrub starts 2024-03-11T21:36:43.825 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:36:43 smithi097 ceph-mon[33317]: purged_snaps scrub ok 2024-03-11T21:36:43.825 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:36:43 smithi097 ceph-mon[33317]: osdmap e282: 8 total, 8 up, 8 in 2024-03-11T21:36:43.826 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:36:43 smithi097 ceph-mon[33317]: from='client.24469 v1:172.21.15.97:0/404208310' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:36:43.826 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:36:43 smithi097 ceph-mon[33317]: from='client.? v1:172.21.15.92:0/942505522' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-11T21:36:43.826 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:36:43 smithi097 ceph-mon[33317]: Health check update: Degraded data redundancy: 37960/231843 objects degraded (16.373%), 8 pgs degraded (PG_DEGRADED) 2024-03-11T21:36:43.910 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:36:43.945 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:36:43.945 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:36:43.946 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:36:44.046 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:36:44.086 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:36:44.087 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:36:44.087 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:36:44.188 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:36:44.220 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:36:44.221 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:36:44.221 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:36:44.322 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:36:44.358 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:36:44.359 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:36:44.359 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:36:44.459 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:36:44.504 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:36:44.504 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:36:44.505 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:36:44.527 INFO:tasks.radosbench.radosbench.0.smithi092.stdout: 47 16 82831 82815 110.104 53.5625 0.0203177 0.00907647 2024-03-11T21:36:44.605 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:36:44.609 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:36:44 smithi092 ceph-mon[29572]: pgmap v509: 148 pgs: 5 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+degraded+remapped, 140 active+clean; 4.9 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 1023 B/s rd, 60 MiB/s wr, 955 op/s; 37546/241185 objects degraded (15.567%); 4.8 MiB/s, 85 objects/s recovering 2024-03-11T21:36:44.609 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:36:44 smithi092 ceph-mon[37563]: pgmap v509: 148 pgs: 5 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+degraded+remapped, 140 active+clean; 4.9 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 1023 B/s rd, 60 MiB/s wr, 955 op/s; 37546/241185 objects degraded (15.567%); 4.8 MiB/s, 85 objects/s recovering 2024-03-11T21:36:44.651 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:36:44.652 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:36:44.652 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:36:44.753 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:36:44.791 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:36:44.792 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:36:44.792 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:36:44.825 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:36:44 smithi097 ceph-mon[33317]: pgmap v509: 148 pgs: 5 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+degraded+remapped, 140 active+clean; 4.9 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 1023 B/s rd, 60 MiB/s wr, 955 op/s; 37546/241185 objects degraded (15.567%); 4.8 MiB/s, 85 objects/s recovering 2024-03-11T21:36:44.893 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:36:44.934 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:36:44.935 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:36:44.935 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:36:45.036 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:36:45.072 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:36:45.073 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:36:45.073 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:36:45.174 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:36:45.218 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:36:45.218 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:36:45.219 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:36:45.319 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:36:45.351 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:36:45.352 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:36:45.352 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:36:45.453 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:36:45.507 DEBUG:teuthology.orchestra.run.smithi092:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid 00db0016-dfed-11ee-95c8-87774f69a715 -- ceph osd reweight 0 1 2024-03-11T21:36:45.517 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:36:45.518 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:36:45.518 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:36:45.528 INFO:tasks.radosbench.radosbench.0.smithi092.stdout: 48 16 83672 83656 108.902 52.5625 0.0135401 0.009176 2024-03-11T21:36:45.619 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:36:45.653 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:36:45.653 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:36:45.654 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:36:45.755 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:36:45.799 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:36:45.800 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:36:45.800 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:36:45.901 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:36:45.945 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:36:45.946 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:36:45.946 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:36:46.047 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:36:46.085 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:36:46.086 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:36:46.086 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:36:46.187 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:36:46.226 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:36:46.227 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:36:46.227 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:36:46.328 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:36:46.361 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:36:46.361 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:36:46.362 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:36:46.406 DEBUG:teuthology.orchestra.run.smithi092:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid 00db0016-dfed-11ee-95c8-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-03-11T21:36:46.449 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:36:46 smithi092 ceph-mon[29572]: pgmap v510: 148 pgs: 5 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+degraded+remapped, 140 active+clean; 5.0 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 743 B/s rd, 43 MiB/s wr, 687 op/s; 37546/244245 objects degraded (15.372%); 3.8 MiB/s, 61 objects/s recovering 2024-03-11T21:36:46.449 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:36:46 smithi092 ceph-mon[29572]: purged_snaps scrub starts 2024-03-11T21:36:46.450 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:36:46 smithi092 ceph-mon[29572]: purged_snaps scrub ok 2024-03-11T21:36:46.450 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:36:46 smithi092 ceph-mon[37563]: pgmap v510: 148 pgs: 5 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+degraded+remapped, 140 active+clean; 5.0 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 743 B/s rd, 43 MiB/s wr, 687 op/s; 37546/244245 objects degraded (15.372%); 3.8 MiB/s, 61 objects/s recovering 2024-03-11T21:36:46.450 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:36:46 smithi092 ceph-mon[37563]: purged_snaps scrub starts 2024-03-11T21:36:46.450 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:36:46 smithi092 ceph-mon[37563]: purged_snaps scrub ok 2024-03-11T21:36:46.462 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:36:46.495 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:36:46.496 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:36:46.496 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:36:46.528 INFO:tasks.radosbench.radosbench.0.smithi092.stdout: 49 16 84466 84450 107.692 49.625 0.0384839 0.00928011 2024-03-11T21:36:46.597 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:36:46.626 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:36:46.627 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:36:46.627 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:36:46.728 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:36:46.774 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:36:46.774 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:36:46.775 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:36:46.825 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:36:46 smithi097 ceph-mon[33317]: pgmap v510: 148 pgs: 5 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+degraded+remapped, 140 active+clean; 5.0 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 743 B/s rd, 43 MiB/s wr, 687 op/s; 37546/244245 objects degraded (15.372%); 3.8 MiB/s, 61 objects/s recovering 2024-03-11T21:36:46.825 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:36:46 smithi097 ceph-mon[33317]: purged_snaps scrub starts 2024-03-11T21:36:46.825 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:36:46 smithi097 ceph-mon[33317]: purged_snaps scrub ok 2024-03-11T21:36:46.876 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:36:46.912 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:36:46.912 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:36:46.913 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:36:47.014 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:36:47.046 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:36:47.046 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:36:47.047 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:36:47.147 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:36:47.192 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:36:47.193 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:36:47.193 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:36:47.294 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:36:47.334 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:36:47.334 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:36:47.334 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:36:47.435 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:36:47.485 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:36:47.485 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:36:47.486 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:36:47.526 INFO:teuthology.orchestra.run.smithi092.stderr:Inferring config /var/lib/ceph/00db0016-dfed-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T21:36:47.529 INFO:tasks.radosbench.radosbench.0.smithi092.stdout: 50 16 85234 85218 106.496 48 0.0289098 0.00938433 2024-03-11T21:36:47.587 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:36:47.611 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:36:47 smithi092 ceph-mon[37563]: purged_snaps scrub starts 2024-03-11T21:36:47.611 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:36:47 smithi092 ceph-mon[37563]: purged_snaps scrub ok 2024-03-11T21:36:47.612 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:36:47 smithi092 ceph-mon[37563]: osdmap e283: 8 total, 8 up, 8 in 2024-03-11T21:36:47.612 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:36:47 smithi092 ceph-mon[29572]: purged_snaps scrub starts 2024-03-11T21:36:47.613 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:36:47 smithi092 ceph-mon[29572]: purged_snaps scrub ok 2024-03-11T21:36:47.613 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:36:47 smithi092 ceph-mon[29572]: osdmap e283: 8 total, 8 up, 8 in 2024-03-11T21:36:47.637 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:36:47.638 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:36:47.639 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:36:47.739 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:36:47.785 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:36:47.786 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:36:47.787 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:36:47.825 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:36:47 smithi097 ceph-mon[33317]: purged_snaps scrub starts 2024-03-11T21:36:47.826 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:36:47 smithi097 ceph-mon[33317]: purged_snaps scrub ok 2024-03-11T21:36:47.826 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:36:47 smithi097 ceph-mon[33317]: osdmap e283: 8 total, 8 up, 8 in 2024-03-11T21:36:47.887 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:36:47.932 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:36:47.932 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:36:47.933 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:36:48.033 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:36:48.069 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:36:48.069 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:36:48.070 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:36:48.094 INFO:journalctl@ceph.mgr.y.smithi092.stdout:Mar 11 21:36:47 smithi092 ceph-00db0016-dfed-11ee-95c8-87774f69a715-mgr-y[29801]: ::ffff:172.21.15.97 - - [11/Mar/2024:21:36:47] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-11T21:36:48.170 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:36:48.205 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:36:48.206 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:36:48.206 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:36:48.307 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:36:48.343 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:36:48.344 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:36:48.344 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:36:48.421 INFO:teuthology.orchestra.run.smithi092.stderr:noscrub is unset 2024-03-11T21:36:48.445 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:36:48.478 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:36:48.479 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:36:48.479 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:36:48.529 INFO:tasks.radosbench.radosbench.0.smithi092.stdout: 51 16 85981 85965 105.324 46.6875 0.00668163 0.00948962 2024-03-11T21:36:48.580 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:36:48.616 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:36:48.617 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:36:48.618 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:36:48.648 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:36:48 smithi092 ceph-mon[29572]: pgmap v512: 148 pgs: 5 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+degraded+remapped, 140 active+clean; 5.0 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 853 B/s rd, 46 MiB/s wr, 735 op/s; 37400/245070 objects degraded (15.261%); 5.8 MiB/s, 93 objects/s recovering 2024-03-11T21:36:48.648 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:36:48 smithi092 ceph-mon[29572]: purged_snaps scrub starts 2024-03-11T21:36:48.648 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:36:48 smithi092 ceph-mon[29572]: purged_snaps scrub ok 2024-03-11T21:36:48.648 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:36:48 smithi092 ceph-mon[29572]: osdmap e284: 8 total, 8 up, 8 in 2024-03-11T21:36:48.649 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:36:48 smithi092 ceph-mon[29572]: from='client.? v1:172.21.15.92:0/2208304566' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-11T21:36:48.649 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:36:48 smithi092 ceph-mon[29572]: from='client.35327 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-11T21:36:48.649 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:36:48 smithi092 ceph-mon[29572]: Health check update: Degraded data redundancy: 37400/245070 objects degraded (15.261%), 8 pgs degraded (PG_DEGRADED) 2024-03-11T21:36:48.649 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:36:48 smithi092 ceph-mon[37563]: pgmap v512: 148 pgs: 5 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+degraded+remapped, 140 active+clean; 5.0 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 853 B/s rd, 46 MiB/s wr, 735 op/s; 37400/245070 objects degraded (15.261%); 5.8 MiB/s, 93 objects/s recovering 2024-03-11T21:36:48.649 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:36:48 smithi092 ceph-mon[37563]: purged_snaps scrub starts 2024-03-11T21:36:48.649 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:36:48 smithi092 ceph-mon[37563]: purged_snaps scrub ok 2024-03-11T21:36:48.649 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:36:48 smithi092 ceph-mon[37563]: osdmap e284: 8 total, 8 up, 8 in 2024-03-11T21:36:48.649 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:36:48 smithi092 ceph-mon[37563]: from='client.? v1:172.21.15.92:0/2208304566' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-11T21:36:48.649 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:36:48 smithi092 ceph-mon[37563]: from='client.35327 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-11T21:36:48.649 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:36:48 smithi092 ceph-mon[37563]: Health check update: Degraded data redundancy: 37400/245070 objects degraded (15.261%), 8 pgs degraded (PG_DEGRADED) 2024-03-11T21:36:48.718 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:36:48.753 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:36:48.754 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:36:48.754 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:36:48.825 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:36:48 smithi097 ceph-mon[33317]: pgmap v512: 148 pgs: 5 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+degraded+remapped, 140 active+clean; 5.0 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 853 B/s rd, 46 MiB/s wr, 735 op/s; 37400/245070 objects degraded (15.261%); 5.8 MiB/s, 93 objects/s recovering 2024-03-11T21:36:48.825 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:36:48 smithi097 ceph-mon[33317]: purged_snaps scrub starts 2024-03-11T21:36:48.825 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:36:48 smithi097 ceph-mon[33317]: purged_snaps scrub ok 2024-03-11T21:36:48.826 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:36:48 smithi097 ceph-mon[33317]: osdmap e284: 8 total, 8 up, 8 in 2024-03-11T21:36:48.826 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:36:48 smithi097 ceph-mon[33317]: from='client.? v1:172.21.15.92:0/2208304566' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-11T21:36:48.826 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:36:48 smithi097 ceph-mon[33317]: from='client.35327 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-11T21:36:48.826 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:36:48 smithi097 ceph-mon[33317]: Health check update: Degraded data redundancy: 37400/245070 objects degraded (15.261%), 8 pgs degraded (PG_DEGRADED) 2024-03-11T21:36:48.854 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:36:48.887 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:36:48.888 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:36:48.888 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:36:48.989 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:36:49.031 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:36:49.031 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:36:49.032 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:36:49.132 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:36:49.166 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:36:49.167 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:36:49.167 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:36:49.268 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:36:49.303 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:36:49.303 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:36:49.304 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:36:49.405 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:36:49.439 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:36:49.439 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:36:49.440 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:36:49.530 INFO:tasks.radosbench.radosbench.0.smithi092.stdout: 52 16 86800 86784 104.283 51.1875 0.012596 0.00958287 2024-03-11T21:36:49.540 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:36:49.576 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:36:49.577 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:36:49.577 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:36:49.670 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:36:49 smithi092 ceph-mon[29572]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-11T21:36:49.670 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:36:49 smithi092 ceph-mon[29572]: from='client.35327 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-03-11T21:36:49.670 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:36:49 smithi092 ceph-mon[29572]: osdmap e285: 8 total, 8 up, 8 in 2024-03-11T21:36:49.670 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:36:49 smithi092 ceph-mon[37563]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-11T21:36:49.670 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:36:49 smithi092 ceph-mon[37563]: from='client.35327 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-03-11T21:36:49.670 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:36:49 smithi092 ceph-mon[37563]: osdmap e285: 8 total, 8 up, 8 in 2024-03-11T21:36:49.678 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:36:49.723 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:36:49.723 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:36:49.724 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:36:49.824 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:36:49.826 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:36:49 smithi097 ceph-mon[33317]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-11T21:36:49.826 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:36:49 smithi097 ceph-mon[33317]: from='client.35327 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-03-11T21:36:49.826 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:36:49 smithi097 ceph-mon[33317]: osdmap e285: 8 total, 8 up, 8 in 2024-03-11T21:36:49.856 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:36:49.857 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:36:49.857 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:36:49.958 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:36:49.992 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:36:49.993 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:36:49.993 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:36:50.094 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:36:50.118 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:36:50.118 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:36:50.119 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:36:50.219 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:36:50.253 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:36:50.254 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:36:50.255 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:36:50.355 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:36:50.382 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:36:50.383 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:36:50.383 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:36:50.484 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:36:50.527 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:36:50.527 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:36:50.528 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:36:50.529 INFO:tasks.radosbench.radosbench.0.smithi092.stdout: 53 16 87539 87523 103.187 46.1875 0.0168709 0.00968443 2024-03-11T21:36:50.629 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:36:50.667 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:36:50.668 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:36:50.668 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:36:50.726 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:36:50 smithi097 ceph-mon[33317]: pgmap v515: 148 pgs: 5 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+degraded+remapped, 140 active+clean; 5.2 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 853 B/s rd, 52 MiB/s wr, 839 op/s; 37129/256290 objects degraded (14.487%); 4.3 MiB/s, 69 objects/s recovering 2024-03-11T21:36:50.769 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:36:50.800 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:36:50.801 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:36:50.801 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:36:50.859 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:36:50 smithi092 ceph-mon[29572]: pgmap v515: 148 pgs: 5 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+degraded+remapped, 140 active+clean; 5.2 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 853 B/s rd, 52 MiB/s wr, 839 op/s; 37129/256290 objects degraded (14.487%); 4.3 MiB/s, 69 objects/s recovering 2024-03-11T21:36:50.860 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:36:50 smithi092 ceph-mon[37563]: pgmap v515: 148 pgs: 5 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+degraded+remapped, 140 active+clean; 5.2 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 853 B/s rd, 52 MiB/s wr, 839 op/s; 37129/256290 objects degraded (14.487%); 4.3 MiB/s, 69 objects/s recovering 2024-03-11T21:36:50.902 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:36:50.946 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:36:50.946 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:36:50.947 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:36:51.048 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:36:51.084 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:36:51.084 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:36:51.085 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:36:51.186 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:36:51.224 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:36:51.225 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:36:51.225 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:36:51.326 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:36:51.358 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:36:51.359 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:36:51.359 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:36:51.378 INFO:teuthology.orchestra.run.smithi092.stdout:ERROR: (22) Invalid argument 2024-03-11T21:36:51.378 INFO:teuthology.orchestra.run.smithi092.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-11T21:36:51.460 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:36:51.496 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:36:51.497 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:36:51.497 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:36:51.530 INFO:tasks.radosbench.radosbench.0.smithi092.stdout: 54 15 88320 88305 102.181 48.875 0.0196416 0.00978073 2024-03-11T21:36:51.598 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:36:51.631 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:36:51.632 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:36:51.632 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:36:51.733 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:36:51.770 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:36:51.771 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:36:51.771 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:36:51.872 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:36:51.907 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:36:51.908 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:36:51.908 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:36:52.009 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:36:52.046 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:36:52.046 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:36:52.047 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:36:52.148 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:36:52.180 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:36:52.181 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:36:52.181 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:36:52.282 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:36:52.323 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:36:52.323 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:36:52.324 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:36:52.405 DEBUG:teuthology.orchestra.run.smithi092:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid 00db0016-dfed-11ee-95c8-87774f69a715 -- ceph daemon osd.1 dump_blocked_ops 2024-03-11T21:36:52.425 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:36:52.463 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:36:52.464 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:36:52.464 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:36:52.530 INFO:tasks.radosbench.radosbench.0.smithi092.stdout: 55 16 89111 89095 101.221 49.375 0.0319265 0.00987282 2024-03-11T21:36:52.564 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:36:52.610 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:36:52 smithi092 ceph-mon[29572]: pgmap v516: 148 pgs: 5 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+degraded+remapped, 140 active+clean; 5.2 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 853 B/s rd, 42 MiB/s wr, 669 op/s; 37129/256290 objects degraded (14.487%); 4.3 MiB/s, 69 objects/s recovering 2024-03-11T21:36:52.610 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:36:52 smithi092 ceph-mon[29572]: from='mgr.14607 v1:172.21.15.92:0/1702717597' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T21:36:52.610 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:36:52 smithi092 ceph-mon[37563]: pgmap v516: 148 pgs: 5 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+degraded+remapped, 140 active+clean; 5.2 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 853 B/s rd, 42 MiB/s wr, 669 op/s; 37129/256290 objects degraded (14.487%); 4.3 MiB/s, 69 objects/s recovering 2024-03-11T21:36:52.610 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:36:52 smithi092 ceph-mon[37563]: from='mgr.14607 v1:172.21.15.92:0/1702717597' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T21:36:52.623 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:36:52.623 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:36:52.624 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:36:52.698 INFO:journalctl@ceph.iscsi.iscsi.a.smithi097.stdout:Mar 11 21:36:52 smithi097 ceph-00db0016-dfed-11ee-95c8-87774f69a715-iscsi-iscsi-a[60943]: debug there is no tcmu-runner data available 2024-03-11T21:36:52.699 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:36:52 smithi097 ceph-mon[33317]: pgmap v516: 148 pgs: 5 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+degraded+remapped, 140 active+clean; 5.2 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 853 B/s rd, 42 MiB/s wr, 669 op/s; 37129/256290 objects degraded (14.487%); 4.3 MiB/s, 69 objects/s recovering 2024-03-11T21:36:52.699 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:36:52 smithi097 ceph-mon[33317]: from='mgr.14607 v1:172.21.15.92:0/1702717597' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T21:36:52.724 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:36:52.759 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:36:52.759 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:36:52.759 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:36:52.860 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:36:52.893 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:36:52.894 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:36:52.894 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:36:52.995 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:36:53.032 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:36:53.032 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:36:53.033 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:36:53.133 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:36:53.173 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:36:53.173 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:36:53.174 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:36:53.184 INFO:teuthology.orchestra.run.smithi092.stderr:Inferring config /var/lib/ceph/00db0016-dfed-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T21:36:53.265 INFO:teuthology.orchestra.run.smithi092.stderr:Inferring config /var/lib/ceph/00db0016-dfed-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T21:36:53.274 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:36:53.312 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:36:53.313 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:36:53.313 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:36:53.414 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:36:53.448 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:36:53.449 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:36:53.449 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:36:53.530 INFO:tasks.radosbench.radosbench.0.smithi092.stdout: 56 16 89836 89820 100.222 45.3125 0.0182967 0.00997195 2024-03-11T21:36:53.550 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:36:53.587 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:36:53.587 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:36:53.587 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:36:53.598 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:36:53 smithi092 ceph-mon[29572]: from='client.24469 v1:172.21.15.97:0/404208310' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:36:53.598 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:36:53 smithi092 ceph-mon[29572]: Health check update: Degraded data redundancy: 37129/256290 objects degraded (14.487%), 8 pgs degraded (PG_DEGRADED) 2024-03-11T21:36:53.598 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:36:53 smithi092 ceph-mon[29572]: osdmap e286: 8 total, 8 up, 8 in 2024-03-11T21:36:53.598 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:36:53 smithi092 ceph-mon[37563]: from='client.24469 v1:172.21.15.97:0/404208310' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:36:53.598 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:36:53 smithi092 ceph-mon[37563]: Health check update: Degraded data redundancy: 37129/256290 objects degraded (14.487%), 8 pgs degraded (PG_DEGRADED) 2024-03-11T21:36:53.598 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:36:53 smithi092 ceph-mon[37563]: osdmap e286: 8 total, 8 up, 8 in 2024-03-11T21:36:53.688 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:36:53.726 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:36:53.727 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:36:53.727 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:36:53.826 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:36:53 smithi097 ceph-mon[33317]: from='client.24469 v1:172.21.15.97:0/404208310' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:36:53.826 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:36:53 smithi097 ceph-mon[33317]: Health check update: Degraded data redundancy: 37129/256290 objects degraded (14.487%), 8 pgs degraded (PG_DEGRADED) 2024-03-11T21:36:53.826 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:36:53 smithi097 ceph-mon[33317]: osdmap e286: 8 total, 8 up, 8 in 2024-03-11T21:36:53.828 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:36:53.874 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:36:53.874 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:36:53.875 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:36:53.975 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:36:54.017 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:36:54.018 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:36:54.019 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:36:54.119 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:36:54.157 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:36:54.157 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:36:54.158 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:36:54.259 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:36:54.291 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:36:54.291 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:36:54.291 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:36:54.392 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:36:54.435 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:36:54.435 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:36:54.436 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:36:54.451 INFO:teuthology.orchestra.run.smithi092.stderr:reweighted osd.0 to 1 (10000) 2024-03-11T21:36:54.454 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:36:54 smithi092 ceph-mon[29572]: pgmap v517: 148 pgs: 5 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+degraded+remapped, 140 active+clean; 5.4 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 1.5 KiB/s rd, 63 MiB/s wr, 1.01k op/s; 36689/265047 objects degraded (13.842%); 7.8 MiB/s, 125 objects/s recovering 2024-03-11T21:36:54.454 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:36:54 smithi092 ceph-mon[29572]: from='client.? v1:172.21.15.92:0/2744800296' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]: dispatch 2024-03-11T21:36:54.454 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:36:54 smithi092 ceph-mon[29572]: from='client.25339 ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]: dispatch 2024-03-11T21:36:54.454 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:36:54 smithi092 ceph-mon[37563]: pgmap v517: 148 pgs: 5 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+degraded+remapped, 140 active+clean; 5.4 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 1.5 KiB/s rd, 63 MiB/s wr, 1.01k op/s; 36689/265047 objects degraded (13.842%); 7.8 MiB/s, 125 objects/s recovering 2024-03-11T21:36:54.454 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:36:54 smithi092 ceph-mon[37563]: from='client.? v1:172.21.15.92:0/2744800296' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]: dispatch 2024-03-11T21:36:54.454 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:36:54 smithi092 ceph-mon[37563]: from='client.25339 ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]: dispatch 2024-03-11T21:36:54.531 INFO:tasks.radosbench.radosbench.0.smithi092.stdout: 57 16 90669 90653 99.3767 52.0625 0.00942595 0.0100569 2024-03-11T21:36:54.537 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:36:54.582 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:36:54.583 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:36:54.583 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:36:54.684 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:36:54.718 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:36:54.719 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:36:54.719 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:36:54.820 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:36:54.822 DEBUG:teuthology.orchestra.run.smithi092:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid 00db0016-dfed-11ee-95c8-87774f69a715 -- ceph osd unset nodeep-scrub 2024-03-11T21:36:54.826 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:36:54 smithi097 ceph-mon[33317]: pgmap v517: 148 pgs: 5 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+degraded+remapped, 140 active+clean; 5.4 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 1.5 KiB/s rd, 63 MiB/s wr, 1.01k op/s; 36689/265047 objects degraded (13.842%); 7.8 MiB/s, 125 objects/s recovering 2024-03-11T21:36:54.826 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:36:54 smithi097 ceph-mon[33317]: from='client.? v1:172.21.15.92:0/2744800296' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]: dispatch 2024-03-11T21:36:54.826 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:36:54 smithi097 ceph-mon[33317]: from='client.25339 ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]: dispatch 2024-03-11T21:36:54.853 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:36:54.854 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:36:54.854 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:36:54.955 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:36:54.983 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:36:54.984 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:36:54.984 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:36:55.085 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:36:55.115 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:36:55.116 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:36:55.116 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:36:55.217 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:36:55.258 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:36:55.258 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:36:55.259 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:36:55.359 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:36:55.389 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:36:55.390 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:36:55.390 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:36:55.491 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:36:55.521 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:36:55.522 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:36:55.522 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:36:55.530 INFO:tasks.radosbench.radosbench.0.smithi092.stdout: 58 15 91468 91453 98.5254 50 0.0184971 0.0101446 2024-03-11T21:36:55.623 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:36:55.669 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:36:55.669 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:36:55.669 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:36:55.703 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:36:55 smithi092 ceph-mon[29572]: from='client.25339 ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]': finished 2024-03-11T21:36:55.704 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:36:55 smithi092 ceph-mon[29572]: osdmap e287: 8 total, 8 up, 8 in 2024-03-11T21:36:55.704 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:36:55 smithi092 ceph-mon[37563]: from='client.25339 ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]': finished 2024-03-11T21:36:55.704 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:36:55 smithi092 ceph-mon[37563]: osdmap e287: 8 total, 8 up, 8 in 2024-03-11T21:36:55.705 DEBUG:teuthology.orchestra.run.smithi092:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid 00db0016-dfed-11ee-95c8-87774f69a715 -- ceph osd reweight 1 1 2024-03-11T21:36:55.770 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:36:55.825 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:36:55 smithi097 ceph-mon[33317]: from='client.25339 ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]': finished 2024-03-11T21:36:55.825 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:36:55 smithi097 ceph-mon[33317]: osdmap e287: 8 total, 8 up, 8 in 2024-03-11T21:36:55.842 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:36:55.842 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:36:55.843 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:36:55.943 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:36:55.985 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:36:55.986 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:36:55.986 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:36:56.087 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:36:56.124 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:36:56.125 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:36:56.125 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:36:56.226 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:36:56.294 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:36:56.295 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:36:56.295 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:36:56.396 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:36:56.441 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:36:56.442 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:36:56.442 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:36:56.531 INFO:tasks.radosbench.radosbench.0.smithi092.stdout: 59 16 92244 92228 97.6765 48.4375 0.023149 0.0102318 2024-03-11T21:36:56.543 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:36:56.587 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:36:56.588 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:36:56.588 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:36:56.689 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:36:56.722 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:36:56 smithi092 ceph-mon[29572]: pgmap v520: 148 pgs: 5 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+degraded+remapped, 140 active+clean; 5.4 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 753 B/s rd, 36 MiB/s wr, 573 op/s; 36689/267960 objects degraded (13.692%); 4.0 MiB/s, 64 objects/s recovering 2024-03-11T21:36:56.723 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:36:56 smithi092 ceph-mon[37563]: pgmap v520: 148 pgs: 5 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+degraded+remapped, 140 active+clean; 5.4 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 753 B/s rd, 36 MiB/s wr, 573 op/s; 36689/267960 objects degraded (13.692%); 4.0 MiB/s, 64 objects/s recovering 2024-03-11T21:36:56.734 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:36:56.735 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:36:56.735 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:36:56.825 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:36:56 smithi097 ceph-mon[33317]: pgmap v520: 148 pgs: 5 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+degraded+remapped, 140 active+clean; 5.4 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 753 B/s rd, 36 MiB/s wr, 573 op/s; 36689/267960 objects degraded (13.692%); 4.0 MiB/s, 64 objects/s recovering 2024-03-11T21:36:56.836 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:36:56.877 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:36:56.878 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:36:56.878 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:36:56.979 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:36:57.019 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:36:57.020 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:36:57.020 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:36:57.121 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:36:57.160 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:36:57.161 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:36:57.161 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:36:57.262 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:36:57.301 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:36:57.302 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:36:57.302 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:36:57.403 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:36:57.441 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:36:57.442 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:36:57.442 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:36:57.531 INFO:tasks.radosbench.radosbench.0.smithi092.stdout:2024-03-11T21:36:57.550549+0000 min lat: 0.00114249 max lat: 2.50819 avg lat: 0.0103106 2024-03-11T21:36:57.531 INFO:tasks.radosbench.radosbench.0.smithi092.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-03-11T21:36:57.531 INFO:tasks.radosbench.radosbench.0.smithi092.stdout: 60 16 93095 93079 96.9351 53.1875 0.0347812 0.0103106 2024-03-11T21:36:57.542 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:36:57.582 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:36:57.583 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:36:57.583 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:36:57.684 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:36:57.710 INFO:teuthology.orchestra.run.smithi092.stderr:Inferring config /var/lib/ceph/00db0016-dfed-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T21:36:57.731 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:36:57.732 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:36:57.732 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:36:57.786 INFO:teuthology.orchestra.run.smithi092.stderr:Inferring config /var/lib/ceph/00db0016-dfed-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T21:36:57.833 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:36:57.884 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:36:57.885 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:36:57.885 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:36:57.986 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:36:58.022 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:36:58.023 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:36:58.023 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:36:58.041 INFO:journalctl@ceph.mgr.y.smithi092.stdout:Mar 11 21:36:57 smithi092 ceph-00db0016-dfed-11ee-95c8-87774f69a715-mgr-y[29801]: ::ffff:172.21.15.97 - - [11/Mar/2024:21:36:57] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-11T21:36:58.124 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:36:58.158 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:36:58.159 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:36:58.159 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:36:58.260 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:36:58.296 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:36:58.296 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:36:58.297 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:36:58.397 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:36:58.455 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:36:58.456 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:36:58.457 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:36:58.531 INFO:tasks.radosbench.radosbench.0.smithi092.stdout: 61 16 93860 93844 96.1298 47.8125 0.00970045 0.0103956 2024-03-11T21:36:58.558 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:36:58.591 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:36:58.592 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:36:58.592 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:36:58.615 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:36:58 smithi092 ceph-mon[29572]: pgmap v521: 148 pgs: 5 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+degraded+remapped, 140 active+clean; 5.5 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 639 B/s rd, 32 MiB/s wr, 517 op/s; 36543/268719 objects degraded (13.599%); 4.6 MiB/s, 73 objects/s recovering 2024-03-11T21:36:58.615 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:36:58 smithi092 ceph-mon[29572]: Health check update: Degraded data redundancy: 36543/268719 objects degraded (13.599%), 8 pgs degraded (PG_DEGRADED) 2024-03-11T21:36:58.615 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:36:58 smithi092 ceph-mon[29572]: osdmap e288: 8 total, 8 up, 8 in 2024-03-11T21:36:58.615 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:36:58 smithi092 ceph-mon[37563]: pgmap v521: 148 pgs: 5 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+degraded+remapped, 140 active+clean; 5.5 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 639 B/s rd, 32 MiB/s wr, 517 op/s; 36543/268719 objects degraded (13.599%); 4.6 MiB/s, 73 objects/s recovering 2024-03-11T21:36:58.615 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:36:58 smithi092 ceph-mon[37563]: Health check update: Degraded data redundancy: 36543/268719 objects degraded (13.599%), 8 pgs degraded (PG_DEGRADED) 2024-03-11T21:36:58.615 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:36:58 smithi092 ceph-mon[37563]: osdmap e288: 8 total, 8 up, 8 in 2024-03-11T21:36:58.693 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:36:58.727 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:36:58.728 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:36:58.729 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:36:58.825 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:36:58 smithi097 ceph-mon[33317]: pgmap v521: 148 pgs: 5 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+degraded+remapped, 140 active+clean; 5.5 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 639 B/s rd, 32 MiB/s wr, 517 op/s; 36543/268719 objects degraded (13.599%); 4.6 MiB/s, 73 objects/s recovering 2024-03-11T21:36:58.826 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:36:58 smithi097 ceph-mon[33317]: Health check update: Degraded data redundancy: 36543/268719 objects degraded (13.599%), 8 pgs degraded (PG_DEGRADED) 2024-03-11T21:36:58.826 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:36:58 smithi097 ceph-mon[33317]: osdmap e288: 8 total, 8 up, 8 in 2024-03-11T21:36:58.830 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:36:58.879 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:36:58.879 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:36:58.880 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:36:58.934 INFO:teuthology.orchestra.run.smithi092.stdout:ERROR: (22) Invalid argument 2024-03-11T21:36:58.934 INFO:teuthology.orchestra.run.smithi092.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-11T21:36:58.980 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:36:59.027 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:36:59.028 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:36:59.028 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:36:59.129 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:36:59.162 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:36:59.162 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:36:59.162 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:36:59.263 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:36:59.303 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:36:59.303 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:36:59.304 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:36:59.404 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:36:59.437 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:36:59.438 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:36:59.438 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:36:59.531 INFO:tasks.radosbench.radosbench.0.smithi092.stdout: 62 16 94589 94573 95.3142 45.5625 0.0336467 0.010486 2024-03-11T21:36:59.539 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:36:59.573 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:36:59.574 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:36:59.574 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:36:59.675 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:36:59.686 DEBUG:teuthology.orchestra.run.smithi092:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid 00db0016-dfed-11ee-95c8-87774f69a715 -- ceph daemon osd.1 dump_historic_ops 2024-03-11T21:36:59.707 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:36:59.707 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:36:59.708 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:36:59.809 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:36:59.869 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:36:59.870 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:36:59.871 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:36:59.971 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:37:00.031 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:37:00.032 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:37:00.033 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:37:00.133 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:37:00.166 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:37:00.167 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:37:00.167 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:37:00.268 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:37:00.308 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:37:00.309 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:37:00.309 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:37:00.410 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:37:00.453 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:37:00.454 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:37:00.454 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:37:00.531 INFO:tasks.radosbench.radosbench.0.smithi092.stdout: 63 16 95313 95297 94.5196 45.25 0.00742426 0.0105743 2024-03-11T21:37:00.555 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:37:00.594 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:37:00.595 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:37:00.595 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:37:00.696 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:37:00.732 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:37:00.733 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:37:00.733 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:37:00.825 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:37:00 smithi097 ceph-mon[33317]: pgmap v523: 148 pgs: 5 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+degraded+remapped, 140 active+clean; 5.7 GiB data, 18 GiB used, 698 GiB / 715 GiB avail; 852 B/s rd, 52 MiB/s wr, 825 op/s; 36271/279897 objects degraded (12.959%); 4.4 MiB/s, 69 objects/s recovering 2024-03-11T21:37:00.834 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:37:00.860 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:37:00 smithi092 ceph-mon[29572]: pgmap v523: 148 pgs: 5 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+degraded+remapped, 140 active+clean; 5.7 GiB data, 18 GiB used, 698 GiB / 715 GiB avail; 852 B/s rd, 52 MiB/s wr, 825 op/s; 36271/279897 objects degraded (12.959%); 4.4 MiB/s, 69 objects/s recovering 2024-03-11T21:37:00.860 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:37:00 smithi092 ceph-mon[37563]: pgmap v523: 148 pgs: 5 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+degraded+remapped, 140 active+clean; 5.7 GiB data, 18 GiB used, 698 GiB / 715 GiB avail; 852 B/s rd, 52 MiB/s wr, 825 op/s; 36271/279897 objects degraded (12.959%); 4.4 MiB/s, 69 objects/s recovering 2024-03-11T21:37:00.874 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:37:00.875 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:37:00.875 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:37:00.976 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:37:01.021 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:37:01.022 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:37:01.022 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:37:01.123 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:37:01.156 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:37:01.156 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:37:01.157 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:37:01.257 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:37:01.297 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:37:01.298 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:37:01.298 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:37:01.399 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:37:01.429 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:37:01.430 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:37:01.430 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:37:01.531 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:37:01.532 INFO:tasks.radosbench.radosbench.0.smithi092.stdout: 64 16 96111 96095 93.822 49.875 0.0103712 0.0106528 2024-03-11T21:37:01.585 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:37:01.586 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:37:01.586 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:37:01.592 INFO:teuthology.orchestra.run.smithi092.stdout:osd.0: {} 2024-03-11T21:37:01.637 INFO:teuthology.orchestra.run.smithi092.stdout:osd.1: {} 2024-03-11T21:37:01.678 INFO:teuthology.orchestra.run.smithi092.stdout:osd.2: {} 2024-03-11T21:37:01.687 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:37:01.717 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:37:01.717 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:37:01.718 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:37:01.732 INFO:teuthology.orchestra.run.smithi092.stdout:osd.3: {} 2024-03-11T21:37:01.770 INFO:teuthology.orchestra.run.smithi092.stdout:osd.4: {} 2024-03-11T21:37:01.809 INFO:teuthology.orchestra.run.smithi092.stdout:osd.5: {} 2024-03-11T21:37:01.818 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:37:01.850 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:37:01.851 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:37:01.851 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:37:01.857 INFO:teuthology.orchestra.run.smithi092.stdout:osd.6: {} 2024-03-11T21:37:01.905 INFO:teuthology.orchestra.run.smithi092.stdout:osd.7: {} 2024-03-11T21:37:01.910 INFO:teuthology.orchestra.run.smithi092.stderr:osd.0: osd_enable_op_tracker = '' 2024-03-11T21:37:01.910 INFO:teuthology.orchestra.run.smithi092.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-11T21:37:01.910 INFO:teuthology.orchestra.run.smithi092.stderr:osd.2: osd_enable_op_tracker = '' 2024-03-11T21:37:01.911 INFO:teuthology.orchestra.run.smithi092.stderr:osd.3: osd_enable_op_tracker = '' 2024-03-11T21:37:01.911 INFO:teuthology.orchestra.run.smithi092.stderr:osd.4: osd_enable_op_tracker = '' 2024-03-11T21:37:01.911 INFO:teuthology.orchestra.run.smithi092.stderr:osd.5: osd_enable_op_tracker = '' 2024-03-11T21:37:01.911 INFO:teuthology.orchestra.run.smithi092.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-11T21:37:01.911 INFO:teuthology.orchestra.run.smithi092.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-11T21:37:01.952 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:37:01.985 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:37:01.986 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:37:01.986 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:37:02.087 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:37:02.125 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:37:02.126 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:37:02.126 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:37:02.227 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:37:02.262 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:37:02.263 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:37:02.263 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:37:02.364 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:37:02.393 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:37:02.394 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:37:02.394 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:37:02.495 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:37:02.525 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:37:02.526 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:37:02.526 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:37:02.531 INFO:tasks.radosbench.radosbench.0.smithi092.stdout: 65 16 96871 96855 93.1094 47.5 0.0346256 0.0107344 2024-03-11T21:37:02.627 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:37:02.661 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:37:02.661 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:37:02.662 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:37:02.762 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:37:02.802 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:37:02.803 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:37:02 smithi097 ceph-mon[33317]: pgmap v524: 148 pgs: 5 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+degraded+remapped, 140 active+clean; 5.7 GiB data, 18 GiB used, 698 GiB / 715 GiB avail; 648 B/s rd, 39 MiB/s wr, 627 op/s; 36271/279897 objects degraded (12.959%); 3.3 MiB/s, 52 objects/s recovering 2024-03-11T21:37:02.803 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:37:02 smithi097 ceph-mon[33317]: from='client.? v1:172.21.15.92:0/2549793393' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-11T21:37:02.803 INFO:journalctl@ceph.iscsi.iscsi.a.smithi097.stdout:Mar 11 21:37:02 smithi097 ceph-00db0016-dfed-11ee-95c8-87774f69a715-iscsi-iscsi-a[60943]: debug there is no tcmu-runner data available 2024-03-11T21:37:02.803 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:37:02.804 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:37:02.860 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:37:02 smithi092 ceph-mon[29572]: pgmap v524: 148 pgs: 5 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+degraded+remapped, 140 active+clean; 5.7 GiB data, 18 GiB used, 698 GiB / 715 GiB avail; 648 B/s rd, 39 MiB/s wr, 627 op/s; 36271/279897 objects degraded (12.959%); 3.3 MiB/s, 52 objects/s recovering 2024-03-11T21:37:02.860 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:37:02 smithi092 ceph-mon[29572]: from='client.? v1:172.21.15.92:0/2549793393' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-11T21:37:02.860 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:37:02 smithi092 ceph-mon[37563]: pgmap v524: 148 pgs: 5 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+degraded+remapped, 140 active+clean; 5.7 GiB data, 18 GiB used, 698 GiB / 715 GiB avail; 648 B/s rd, 39 MiB/s wr, 627 op/s; 36271/279897 objects degraded (12.959%); 3.3 MiB/s, 52 objects/s recovering 2024-03-11T21:37:02.860 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:37:02 smithi092 ceph-mon[37563]: from='client.? v1:172.21.15.92:0/2549793393' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-11T21:37:02.905 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:37:02.949 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:37:02.950 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:37:02.950 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:37:03.051 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:37:03.094 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:37:03.095 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:37:03.095 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:37:03.196 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:37:03.239 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:37:03.239 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:37:03.240 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:37:03.340 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:37:03.388 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:37:03.388 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:37:03.388 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:37:03.489 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:37:03.494 INFO:teuthology.orchestra.run.smithi092.stderr:nodeep-scrub is unset 2024-03-11T21:37:03.525 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:37:03.525 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:37:03.526 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:37:03.532 INFO:tasks.radosbench.radosbench.0.smithi092.stdout: 66 16 97625 97609 92.4127 47.125 0.00924143 0.0108139 2024-03-11T21:37:03.626 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:37:03.661 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:37:03.662 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:37:03.662 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:37:03.763 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:37:03.814 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:37:03.814 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:37:03.815 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:37:03.825 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:37:03 smithi097 ceph-mon[33317]: from='client.24469 v1:172.21.15.97:0/404208310' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:37:03.825 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:37:03 smithi097 ceph-mon[33317]: from='client.? v1:172.21.15.92:0/2962721728' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-11T21:37:03.826 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:37:03 smithi097 ceph-mon[33317]: from='client.35339 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-11T21:37:03.826 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:37:03 smithi097 ceph-mon[33317]: Health check update: Degraded data redundancy: 36271/279897 objects degraded (12.959%), 8 pgs degraded (PG_DEGRADED) 2024-03-11T21:37:03.860 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:37:03 smithi092 ceph-mon[29572]: from='client.24469 v1:172.21.15.97:0/404208310' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:37:03.860 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:37:03 smithi092 ceph-mon[29572]: from='client.? v1:172.21.15.92:0/2962721728' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-11T21:37:03.860 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:37:03 smithi092 ceph-mon[29572]: from='client.35339 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-11T21:37:03.860 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:37:03 smithi092 ceph-mon[29572]: Health check update: Degraded data redundancy: 36271/279897 objects degraded (12.959%), 8 pgs degraded (PG_DEGRADED) 2024-03-11T21:37:03.860 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:37:03 smithi092 ceph-mon[37563]: from='client.24469 v1:172.21.15.97:0/404208310' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:37:03.860 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:37:03 smithi092 ceph-mon[37563]: from='client.? v1:172.21.15.92:0/2962721728' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-11T21:37:03.861 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:37:03 smithi092 ceph-mon[37563]: from='client.35339 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-11T21:37:03.861 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:37:03 smithi092 ceph-mon[37563]: Health check update: Degraded data redundancy: 36271/279897 objects degraded (12.959%), 8 pgs degraded (PG_DEGRADED) 2024-03-11T21:37:03.916 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:37:03.956 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:37:03.957 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:37:03.957 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:37:04.058 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:37:04.116 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:37:04.116 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:37:04.117 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:37:04.218 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:37:04.254 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:37:04.254 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:37:04.254 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:37:04.355 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:37:04.400 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:37:04.401 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:37:04.401 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:37:04.502 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:37:04.531 INFO:tasks.radosbench.radosbench.0.smithi092.stdout: 67 16 98479 98463 91.8301 53.375 0.0216947 0.0108846 2024-03-11T21:37:04.545 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:37:04.545 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:37:04.546 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:37:04.610 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:37:04 smithi092 ceph-mon[29572]: pgmap v525: 148 pgs: 5 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+degraded+remapped, 140 active+clean; 5.9 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 1.1 KiB/s rd, 49 MiB/s wr, 779 op/s; 35835/288420 objects degraded (12.425%); 6.1 MiB/s, 97 objects/s recovering 2024-03-11T21:37:04.610 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:37:04 smithi092 ceph-mon[29572]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-03-11T21:37:04.610 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:37:04 smithi092 ceph-mon[29572]: from='client.35339 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-03-11T21:37:04.610 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:37:04 smithi092 ceph-mon[29572]: osdmap e289: 8 total, 8 up, 8 in 2024-03-11T21:37:04.610 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:37:04 smithi092 ceph-mon[37563]: pgmap v525: 148 pgs: 5 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+degraded+remapped, 140 active+clean; 5.9 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 1.1 KiB/s rd, 49 MiB/s wr, 779 op/s; 35835/288420 objects degraded (12.425%); 6.1 MiB/s, 97 objects/s recovering 2024-03-11T21:37:04.610 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:37:04 smithi092 ceph-mon[37563]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-03-11T21:37:04.610 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:37:04 smithi092 ceph-mon[37563]: from='client.35339 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-03-11T21:37:04.610 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:37:04 smithi092 ceph-mon[37563]: osdmap e289: 8 total, 8 up, 8 in 2024-03-11T21:37:04.650 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:37:04.686 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:37:04.686 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:37:04.687 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:37:04.788 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:37:04.825 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:37:04.826 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:37:04.826 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:37:04.827 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:37:04 smithi097 ceph-mon[33317]: pgmap v525: 148 pgs: 5 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+degraded+remapped, 140 active+clean; 5.9 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 1.1 KiB/s rd, 49 MiB/s wr, 779 op/s; 35835/288420 objects degraded (12.425%); 6.1 MiB/s, 97 objects/s recovering 2024-03-11T21:37:04.827 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:37:04 smithi097 ceph-mon[33317]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-03-11T21:37:04.827 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:37:04 smithi097 ceph-mon[33317]: from='client.35339 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-03-11T21:37:04.827 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:37:04 smithi097 ceph-mon[33317]: osdmap e289: 8 total, 8 up, 8 in 2024-03-11T21:37:04.927 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:37:04.965 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:37:04.966 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:37:04.966 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:37:05.067 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:37:05.104 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:37:05.105 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:37:05.105 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:37:05.206 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:37:05.237 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:37:05.237 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:37:05.238 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:37:05.339 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:37:05.373 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:37:05.373 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:37:05.374 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:37:05.474 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:37:05.512 INFO:teuthology.orchestra.run.smithi092.stderr:Inferring config /var/lib/ceph/00db0016-dfed-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T21:37:05.515 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:37:05.516 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:37:05.516 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:37:05.532 INFO:tasks.radosbench.radosbench.0.smithi092.stdout: 68 16 99257 99241 91.1937 48.625 0.0602733 0.0109595 2024-03-11T21:37:05.617 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:37:05.658 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:37:05.658 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:37:05.659 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:37:05.760 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:37:05.794 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:37:05.794 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:37:05.795 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:37:05.896 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:37:05.940 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:37:05.941 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:37:05.941 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:37:06.042 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:37:06.079 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:37:06.079 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:37:06.080 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:37:06.181 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:37:06.218 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:37:06.218 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:37:06.218 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:37:06.319 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:37:06.358 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:37:06.359 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:37:06.359 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:37:06.460 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:37:06.492 INFO:teuthology.orchestra.run.smithi092.stderr:reweighted osd.1 to 1 (10000) 2024-03-11T21:37:06.496 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:37:06.497 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:37:06.498 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:37:06.533 INFO:tasks.radosbench.radosbench.0.smithi092.stdout: 69 16 100025 100009 90.5677 48 0.00924531 0.0110357 2024-03-11T21:37:06.598 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:37:06.634 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:37:06.635 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:37:06.635 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:37:06.736 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:37:06.772 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:37:06.772 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:37:06.773 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:37:06.826 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:37:06 smithi097 ceph-mon[33317]: pgmap v527: 148 pgs: 5 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+degraded+remapped, 140 active+clean; 5.9 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 1.2 KiB/s rd, 59 MiB/s wr, 937 op/s; 35835/291204 objects degraded (12.306%); 5.5 MiB/s, 88 objects/s recovering 2024-03-11T21:37:06.826 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:37:06 smithi097 ceph-mon[33317]: from='client.? v1:172.21.15.92:0/2212364787' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]: dispatch 2024-03-11T21:37:06.826 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:37:06 smithi097 ceph-mon[33317]: from='client.35345 ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]: dispatch 2024-03-11T21:37:06.838 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:37:06 smithi092 ceph-mon[29572]: pgmap v527: 148 pgs: 5 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+degraded+remapped, 140 active+clean; 5.9 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 1.2 KiB/s rd, 59 MiB/s wr, 937 op/s; 35835/291204 objects degraded (12.306%); 5.5 MiB/s, 88 objects/s recovering 2024-03-11T21:37:06.838 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:37:06 smithi092 ceph-mon[29572]: from='client.? v1:172.21.15.92:0/2212364787' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]: dispatch 2024-03-11T21:37:06.838 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:37:06 smithi092 ceph-mon[29572]: from='client.35345 ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]: dispatch 2024-03-11T21:37:06.838 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:37:06 smithi092 ceph-mon[37563]: pgmap v527: 148 pgs: 5 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+degraded+remapped, 140 active+clean; 5.9 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 1.2 KiB/s rd, 59 MiB/s wr, 937 op/s; 35835/291204 objects degraded (12.306%); 5.5 MiB/s, 88 objects/s recovering 2024-03-11T21:37:06.839 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:37:06 smithi092 ceph-mon[37563]: from='client.? v1:172.21.15.92:0/2212364787' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]: dispatch 2024-03-11T21:37:06.839 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:37:06 smithi092 ceph-mon[37563]: from='client.35345 ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]: dispatch 2024-03-11T21:37:06.874 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:37:06.890 DEBUG:teuthology.orchestra.run.smithi092:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid 00db0016-dfed-11ee-95c8-87774f69a715 -- ceph osd set noscrub 2024-03-11T21:37:06.896 DEBUG:teuthology.orchestra.run.smithi092:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid 00db0016-dfed-11ee-95c8-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-03-11T21:37:06.914 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:37:06.915 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:37:06.916 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:37:07.016 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:37:07.113 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:37:07.113 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:37:07.114 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:37:07.215 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:37:07.250 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:37:07.250 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:37:07.251 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:37:07.351 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:37:07.385 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:37:07.385 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:37:07.386 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:37:07.486 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:37:07.528 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:37:07.529 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:37:07.529 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:37:07.533 INFO:tasks.radosbench.radosbench.0.smithi092.stdout: 70 16 100780 100764 89.9481 47.1875 0.0202782 0.0111125 2024-03-11T21:37:07.559 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:37:07 smithi092 ceph-mon[29572]: from='client.35345 ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]': finished 2024-03-11T21:37:07.559 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:37:07 smithi092 ceph-mon[29572]: osdmap e290: 8 total, 8 up, 8 in 2024-03-11T21:37:07.559 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:37:07 smithi092 ceph-mon[29572]: from='mgr.14607 ' entity='mgr.y' 2024-03-11T21:37:07.559 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:37:07 smithi092 ceph-mon[29572]: from='mgr.14607 v1:172.21.15.92:0/1702717597' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T21:37:07.559 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:37:07 smithi092 ceph-mon[37563]: from='client.35345 ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]': finished 2024-03-11T21:37:07.559 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:37:07 smithi092 ceph-mon[37563]: osdmap e290: 8 total, 8 up, 8 in 2024-03-11T21:37:07.559 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:37:07 smithi092 ceph-mon[37563]: from='mgr.14607 ' entity='mgr.y' 2024-03-11T21:37:07.559 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:37:07 smithi092 ceph-mon[37563]: from='mgr.14607 v1:172.21.15.92:0/1702717597' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T21:37:07.630 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:37:07.666 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:37:07.667 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:37:07.667 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:37:07.768 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:37:07.813 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:37:07.814 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:37:07.814 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:37:07.826 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:37:07 smithi097 ceph-mon[33317]: from='client.35345 ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]': finished 2024-03-11T21:37:07.826 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:37:07 smithi097 ceph-mon[33317]: osdmap e290: 8 total, 8 up, 8 in 2024-03-11T21:37:07.826 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:37:07 smithi097 ceph-mon[33317]: from='mgr.14607 ' entity='mgr.y' 2024-03-11T21:37:07.826 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:37:07 smithi097 ceph-mon[33317]: from='mgr.14607 v1:172.21.15.92:0/1702717597' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T21:37:07.914 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:37:07.949 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:37:07.950 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:37:07.950 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:37:08.042 DEBUG:teuthology.orchestra.run.smithi092:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid 00db0016-dfed-11ee-95c8-87774f69a715 -- ceph osd reweight 2 1 2024-03-11T21:37:08.051 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:37:08.082 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:37:08.083 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:37:08.083 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:37:08.184 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:37:08.235 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:37:08.235 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:37:08.236 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:37:08.334 INFO:journalctl@ceph.mgr.y.smithi092.stdout:Mar 11 21:37:07 smithi092 ceph-00db0016-dfed-11ee-95c8-87774f69a715-mgr-y[29801]: ::ffff:172.21.15.97 - - [11/Mar/2024:21:37:07] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-11T21:37:08.337 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:37:08.366 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:37:08.366 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:37:08.367 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:37:08.467 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:37:08.521 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:37:08.522 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:37:08.523 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:37:08.533 INFO:tasks.radosbench.radosbench.0.smithi092.stdout: 71 16 101551 101535 89.3599 48.1875 0.00995551 0.011184 2024-03-11T21:37:08.590 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:37:08 smithi092 ceph-mon[29572]: pgmap v529: 148 pgs: 5 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+degraded+remapped, 140 active+clean; 5.9 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 639 B/s rd, 31 MiB/s wr, 501 op/s; 35686/291927 objects degraded (12.224%); 4.6 MiB/s, 73 objects/s recovering 2024-03-11T21:37:08.590 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:37:08 smithi092 ceph-mon[29572]: Health check update: Degraded data redundancy: 35686/291927 objects degraded (12.224%), 8 pgs degraded (PG_DEGRADED) 2024-03-11T21:37:08.590 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:37:08 smithi092 ceph-mon[37563]: pgmap v529: 148 pgs: 5 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+degraded+remapped, 140 active+clean; 5.9 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 639 B/s rd, 31 MiB/s wr, 501 op/s; 35686/291927 objects degraded (12.224%); 4.6 MiB/s, 73 objects/s recovering 2024-03-11T21:37:08.590 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:37:08 smithi092 ceph-mon[37563]: Health check update: Degraded data redundancy: 35686/291927 objects degraded (12.224%), 8 pgs degraded (PG_DEGRADED) 2024-03-11T21:37:08.624 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:37:08.714 INFO:teuthology.orchestra.run.smithi092.stdout:{ 2024-03-11T21:37:08.714 INFO:teuthology.orchestra.run.smithi092.stdout: "size": 20, 2024-03-11T21:37:08.714 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 600, 2024-03-11T21:37:08.714 INFO:teuthology.orchestra.run.smithi092.stdout: "ops": [ 2024-03-11T21:37:08.714 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:37:08.714 INFO:teuthology.orchestra.run.smithi092.stdout: "description": "osd_op(client.15150.0:49214 8.1 8:88c6d4f2:::benchmark_data_smithi092_104865_object49213:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e169)", 2024-03-11T21:37:08.715 INFO:teuthology.orchestra.run.smithi092.stdout: "initiated_at": "2024-03-11T21:32:25.342252+0000", 2024-03-11T21:37:08.715 INFO:teuthology.orchestra.run.smithi092.stdout: "age": 283.38376969299998, 2024-03-11T21:37:08.715 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.13949117799999999, 2024-03-11T21:37:08.715 INFO:teuthology.orchestra.run.smithi092.stdout: "type_data": { 2024-03-11T21:37:08.715 INFO:teuthology.orchestra.run.smithi092.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-11T21:37:08.715 INFO:teuthology.orchestra.run.smithi092.stdout: "client_info": { 2024-03-11T21:37:08.715 INFO:teuthology.orchestra.run.smithi092.stdout: "client": "client.15150", 2024-03-11T21:37:08.715 INFO:teuthology.orchestra.run.smithi092.stdout: "client_addr": "v1:172.21.15.92:0/436606717", 2024-03-11T21:37:08.715 INFO:teuthology.orchestra.run.smithi092.stdout: "tid": 49214 2024-03-11T21:37:08.715 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:37:08.715 INFO:teuthology.orchestra.run.smithi092.stdout: "events": [ 2024-03-11T21:37:08.715 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:37:08.715 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "initiated", 2024-03-11T21:37:08.715 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:32:25.342252+0000", 2024-03-11T21:37:08.716 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-03-11T21:37:08.716 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:37:08.716 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:37:08.716 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "header_read", 2024-03-11T21:37:08.716 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:32:25.342252+0000", 2024-03-11T21:37:08.716 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-03-11T21:37:08.716 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:37:08.716 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:37:08.716 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "throttled", 2024-03-11T21:37:08.716 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:32:25.342253+0000", 2024-03-11T21:37:08.716 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.1540000000000001e-06 2024-03-11T21:37:08.716 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:37:08.716 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:37:08.716 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "all_read", 2024-03-11T21:37:08.716 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:32:25.342277+0000", 2024-03-11T21:37:08.717 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 2.3408999999999999e-05 2024-03-11T21:37:08.717 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:37:08.717 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:37:08.717 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "dispatched", 2024-03-11T21:37:08.717 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:32:25.342285+0000", 2024-03-11T21:37:08.717 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 8.7120000000000006e-06 2024-03-11T21:37:08.717 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:37:08.717 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:37:08.717 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "queued_for_pg", 2024-03-11T21:37:08.717 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:32:25.342292+0000", 2024-03-11T21:37:08.717 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 7.046e-06 2024-03-11T21:37:08.717 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:37:08.717 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:37:08.718 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-03-11T21:37:08.718 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:32:25.342335+0000", 2024-03-11T21:37:08.718 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 4.2398000000000003e-05 2024-03-11T21:37:08.718 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:37:08.718 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:37:08.718 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "started", 2024-03-11T21:37:08.718 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:32:25.342470+0000", 2024-03-11T21:37:08.718 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.000135067 2024-03-11T21:37:08.718 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:37:08.718 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:37:08.718 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "waiting for subops from 0,2", 2024-03-11T21:37:08.718 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:32:25.342551+0000", 2024-03-11T21:37:08.718 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 8.0766999999999994e-05 2024-03-11T21:37:08.718 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:37:08.719 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:37:08.719 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "sub_op_commit_rec", 2024-03-11T21:37:08.719 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:32:25.475372+0000", 2024-03-11T21:37:08.719 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.132821521 2024-03-11T21:37:08.719 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:37:08.719 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:37:08.719 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "op_commit", 2024-03-11T21:37:08.719 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:32:25.481321+0000", 2024-03-11T21:37:08.719 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.0059487450000000001 2024-03-11T21:37:08.719 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:37:08.719 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:37:08.719 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "sub_op_commit_rec", 2024-03-11T21:37:08.719 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:32:25.481631+0000", 2024-03-11T21:37:08.720 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00031029300000000001 2024-03-11T21:37:08.720 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:37:08.720 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:37:08.720 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "commit_sent", 2024-03-11T21:37:08.720 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:32:25.481693+0000", 2024-03-11T21:37:08.720 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 6.2070999999999996e-05 2024-03-11T21:37:08.720 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:37:08.720 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:37:08.720 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "done", 2024-03-11T21:37:08.720 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:32:25.481743+0000", 2024-03-11T21:37:08.720 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 4.9994999999999998e-05 2024-03-11T21:37:08.720 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-03-11T21:37:08.720 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-03-11T21:37:08.720 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-03-11T21:37:08.721 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:37:08.721 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:37:08.721 INFO:teuthology.orchestra.run.smithi092.stdout: "description": "osd_op(client.15150.0:49215 8.1 8:8b49bbc2:::benchmark_data_smithi092_104865_object49214:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e169)", 2024-03-11T21:37:08.721 INFO:teuthology.orchestra.run.smithi092.stdout: "initiated_at": "2024-03-11T21:32:25.342305+0000", 2024-03-11T21:37:08.721 INFO:teuthology.orchestra.run.smithi092.stdout: "age": 283.38371646299998, 2024-03-11T21:37:08.721 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.13963782199999999, 2024-03-11T21:37:08.721 INFO:teuthology.orchestra.run.smithi092.stdout: "type_data": { 2024-03-11T21:37:08.721 INFO:teuthology.orchestra.run.smithi092.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-11T21:37:08.721 INFO:teuthology.orchestra.run.smithi092.stdout: "client_info": { 2024-03-11T21:37:08.721 INFO:teuthology.orchestra.run.smithi092.stdout: "client": "client.15150", 2024-03-11T21:37:08.721 INFO:teuthology.orchestra.run.smithi092.stdout: "client_addr": "v1:172.21.15.92:0/436606717", 2024-03-11T21:37:08.721 INFO:teuthology.orchestra.run.smithi092.stdout: "tid": 49215 2024-03-11T21:37:08.721 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:37:08.721 INFO:teuthology.orchestra.run.smithi092.stdout: "events": [ 2024-03-11T21:37:08.721 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:37:08.722 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "initiated", 2024-03-11T21:37:08.722 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:32:25.342305+0000", 2024-03-11T21:37:08.722 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-03-11T21:37:08.722 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:37:08.722 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:37:08.722 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "header_read", 2024-03-11T21:37:08.722 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:32:25.342305+0000", 2024-03-11T21:37:08.722 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-03-11T21:37:08.722 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:37:08.722 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:37:08.722 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "throttled", 2024-03-11T21:37:08.722 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:32:25.342306+0000", 2024-03-11T21:37:08.722 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 6.9400000000000005e-07 2024-03-11T21:37:08.722 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:37:08.723 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:37:08.723 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "all_read", 2024-03-11T21:37:08.723 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:32:25.342326+0000", 2024-03-11T21:37:08.723 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.9698999999999999e-05 2024-03-11T21:37:08.723 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:37:08.723 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:37:08.723 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "dispatched", 2024-03-11T21:37:08.723 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:32:25.342331+0000", 2024-03-11T21:37:08.723 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 5.3489999999999999e-06 2024-03-11T21:37:08.723 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:37:08.723 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:37:08.723 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "queued_for_pg", 2024-03-11T21:37:08.723 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:32:25.342337+0000", 2024-03-11T21:37:08.723 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 5.6269999999999997e-06 2024-03-11T21:37:08.723 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:37:08.724 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:37:08.724 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-03-11T21:37:08.724 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:32:25.473100+0000", 2024-03-11T21:37:08.724 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.13076357199999999 2024-03-11T21:37:08.724 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:37:08.724 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:37:08.724 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "started", 2024-03-11T21:37:08.724 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:32:25.473243+0000", 2024-03-11T21:37:08.724 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00014294899999999999 2024-03-11T21:37:08.724 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:37:08.724 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:37:08.724 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "waiting for subops from 0,2", 2024-03-11T21:37:08.724 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:32:25.473426+0000", 2024-03-11T21:37:08.724 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.000183029 2024-03-11T21:37:08.724 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:37:08.725 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:37:08.725 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "op_commit", 2024-03-11T21:37:08.725 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:32:25.481755+0000", 2024-03-11T21:37:08.725 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.0083287629999999994 2024-03-11T21:37:08.725 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:37:08.725 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:37:08.725 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "sub_op_commit_rec", 2024-03-11T21:37:08.725 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:32:25.481829+0000", 2024-03-11T21:37:08.725 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 7.4548000000000005e-05 2024-03-11T21:37:08.725 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:37:08.725 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:37:08.725 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "sub_op_commit_rec", 2024-03-11T21:37:08.725 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:32:25.481864+0000", 2024-03-11T21:37:08.725 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 3.4443999999999998e-05 2024-03-11T21:37:08.725 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:37:08.726 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:37:08.726 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "commit_sent", 2024-03-11T21:37:08.726 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:32:25.481908+0000", 2024-03-11T21:37:08.726 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 4.4549000000000003e-05 2024-03-11T21:37:08.726 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:37:08.726 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:37:08.726 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "done", 2024-03-11T21:37:08.726 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:32:25.481943+0000", 2024-03-11T21:37:08.726 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 3.4598999999999999e-05 2024-03-11T21:37:08.726 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-03-11T21:37:08.726 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-03-11T21:37:08.726 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-03-11T21:37:08.726 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:37:08.726 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:37:08.727 INFO:teuthology.orchestra.run.smithi092.stdout: "description": "osd_repop_reply(client.15150.0:49214 8.1 e169/157 ondisk, result = 0)", 2024-03-11T21:37:08.727 INFO:teuthology.orchestra.run.smithi092.stdout: "initiated_at": "2024-03-11T21:32:25.344396+0000", 2024-03-11T21:37:08.727 INFO:teuthology.orchestra.run.smithi092.stdout: "age": 283.38162572900001, 2024-03-11T21:37:08.727 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.13098934100000001, 2024-03-11T21:37:08.727 INFO:teuthology.orchestra.run.smithi092.stdout: "type_data": { 2024-03-11T21:37:08.727 INFO:teuthology.orchestra.run.smithi092.stdout: "flag_point": "started", 2024-03-11T21:37:08.727 INFO:teuthology.orchestra.run.smithi092.stdout: "events": [ 2024-03-11T21:37:08.727 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:37:08.727 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "initiated", 2024-03-11T21:37:08.727 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:32:25.344396+0000", 2024-03-11T21:37:08.727 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-03-11T21:37:08.727 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:37:08.727 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:37:08.727 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "header_read", 2024-03-11T21:37:08.728 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:32:25.344396+0000", 2024-03-11T21:37:08.728 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-03-11T21:37:08.728 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:37:08.728 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:37:08.728 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "throttled", 2024-03-11T21:37:08.728 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:32:25.344397+0000", 2024-03-11T21:37:08.728 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.0869999999999999e-06 2024-03-11T21:37:08.728 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:37:08.728 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:37:08.728 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "all_read", 2024-03-11T21:37:08.728 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:32:25.344401+0000", 2024-03-11T21:37:08.728 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 3.9230000000000004e-06 2024-03-11T21:37:08.728 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:37:08.728 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:37:08.729 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "dispatched", 2024-03-11T21:37:08.729 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:32:25.472014+0000", 2024-03-11T21:37:08.729 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.127612693 2024-03-11T21:37:08.729 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:37:08.729 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:37:08.729 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "queued_for_pg", 2024-03-11T21:37:08.729 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:32:25.472022+0000", 2024-03-11T21:37:08.729 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 8.1170000000000004e-06 2024-03-11T21:37:08.729 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:37:08.729 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:37:08.729 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-03-11T21:37:08.729 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:32:25.475357+0000", 2024-03-11T21:37:08.729 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.0033354930000000001 2024-03-11T21:37:08.729 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:37:08.730 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:37:08.730 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "started", 2024-03-11T21:37:08.730 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:32:25.475365+0000", 2024-03-11T21:37:08.730 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 8.2169999999999994e-06 2024-03-11T21:37:08.730 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:37:08.730 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:37:08.730 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "done", 2024-03-11T21:37:08.730 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:32:25.475385+0000", 2024-03-11T21:37:08.730 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.9811000000000001e-05 2024-03-11T21:37:08.730 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-03-11T21:37:08.730 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-03-11T21:37:08.730 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-03-11T21:37:08.730 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:37:08.730 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:37:08.731 INFO:teuthology.orchestra.run.smithi092.stdout: "description": "osd_op(client.15150.0:49217 8.1 8:84bb6c69:::benchmark_data_smithi092_104865_object49216:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e169)", 2024-03-11T21:37:08.731 INFO:teuthology.orchestra.run.smithi092.stdout: "initiated_at": "2024-03-11T21:32:25.344882+0000", 2024-03-11T21:37:08.731 INFO:teuthology.orchestra.run.smithi092.stdout: "age": 283.381139808, 2024-03-11T21:37:08.731 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.13864465600000001, 2024-03-11T21:37:08.731 INFO:teuthology.orchestra.run.smithi092.stdout: "type_data": { 2024-03-11T21:37:08.731 INFO:teuthology.orchestra.run.smithi092.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-11T21:37:08.731 INFO:teuthology.orchestra.run.smithi092.stdout: "client_info": { 2024-03-11T21:37:08.731 INFO:teuthology.orchestra.run.smithi092.stdout: "client": "client.15150", 2024-03-11T21:37:08.731 INFO:teuthology.orchestra.run.smithi092.stdout: "client_addr": "v1:172.21.15.92:0/436606717", 2024-03-11T21:37:08.731 INFO:teuthology.orchestra.run.smithi092.stdout: "tid": 49217 2024-03-11T21:37:08.731 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:37:08.731 INFO:teuthology.orchestra.run.smithi092.stdout: "events": [ 2024-03-11T21:37:08.731 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:37:08.731 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "initiated", 2024-03-11T21:37:08.731 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:32:25.344882+0000", 2024-03-11T21:37:08.732 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-03-11T21:37:08.732 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:37:08.732 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:37:08.732 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "header_read", 2024-03-11T21:37:08.732 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:32:25.344882+0000", 2024-03-11T21:37:08.732 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-03-11T21:37:08.732 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:37:08.732 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:37:08.732 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "throttled", 2024-03-11T21:37:08.732 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:32:25.344883+0000", 2024-03-11T21:37:08.732 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.2500000000000001e-06 2024-03-11T21:37:08.732 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:37:08.732 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:37:08.732 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "all_read", 2024-03-11T21:37:08.733 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:32:25.344902+0000", 2024-03-11T21:37:08.733 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.8488e-05 2024-03-11T21:37:08.733 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:37:08.733 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:37:08.733 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "dispatched", 2024-03-11T21:37:08.733 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:32:25.471202+0000", 2024-03-11T21:37:08.733 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.12630037899999999 2024-03-11T21:37:08.733 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:37:08.733 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:37:08.733 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "queued_for_pg", 2024-03-11T21:37:08.733 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:32:25.471219+0000", 2024-03-11T21:37:08.733 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.6909999999999999e-05 2024-03-11T21:37:08.733 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:37:08.733 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:37:08.733 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-03-11T21:37:08.733 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:32:25.475416+0000", 2024-03-11T21:37:08.733 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.0041970030000000004 2024-03-11T21:37:08.734 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:37:08.734 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:37:08.734 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "started", 2024-03-11T21:37:08.734 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:32:25.475549+0000", 2024-03-11T21:37:08.734 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00013357199999999999 2024-03-11T21:37:08.734 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:37:08.734 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:37:08.734 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "waiting for subops from 0,2", 2024-03-11T21:37:08.734 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:32:25.475737+0000", 2024-03-11T21:37:08.734 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.000187935 2024-03-11T21:37:08.734 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:37:08.734 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:37:08.734 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "op_commit", 2024-03-11T21:37:08.734 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:32:25.481795+0000", 2024-03-11T21:37:08.735 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.0060577749999999996 2024-03-11T21:37:08.735 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:37:08.735 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:37:08.735 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "sub_op_commit_rec", 2024-03-11T21:37:08.735 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:32:25.482847+0000", 2024-03-11T21:37:08.735 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.0010517219999999999 2024-03-11T21:37:08.735 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:37:08.735 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:37:08.735 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "sub_op_commit_rec", 2024-03-11T21:37:08.735 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:32:25.483180+0000", 2024-03-11T21:37:08.735 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.000333436 2024-03-11T21:37:08.735 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:37:08.735 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:37:08.735 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "commit_sent", 2024-03-11T21:37:08.735 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:32:25.483342+0000", 2024-03-11T21:37:08.736 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00016119600000000001 2024-03-11T21:37:08.736 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:37:08.736 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:37:08.736 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "done", 2024-03-11T21:37:08.736 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:32:25.483526+0000", 2024-03-11T21:37:08.736 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00018499 2024-03-11T21:37:08.736 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-03-11T21:37:08.736 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-03-11T21:37:08.736 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-03-11T21:37:08.736 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:37:08.736 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:37:08.736 INFO:teuthology.orchestra.run.smithi092.stdout: "description": "osd_op(client.15150.0:88135 8.1 8:8d497fa8:::benchmark_data_smithi092_104865_object88134:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e180)", 2024-03-11T21:37:08.736 INFO:teuthology.orchestra.run.smithi092.stdout: "initiated_at": "2024-03-11T21:32:56.040513+0000", 2024-03-11T21:37:08.736 INFO:teuthology.orchestra.run.smithi092.stdout: "age": 252.68550877499999, 2024-03-11T21:37:08.736 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.17163281, 2024-03-11T21:37:08.736 INFO:teuthology.orchestra.run.smithi092.stdout: "type_data": { 2024-03-11T21:37:08.737 INFO:teuthology.orchestra.run.smithi092.stdout: "flag_point": "reached pg", 2024-03-11T21:37:08.737 INFO:teuthology.orchestra.run.smithi092.stdout: "client_info": { 2024-03-11T21:37:08.737 INFO:teuthology.orchestra.run.smithi092.stdout: "client": "client.15150", 2024-03-11T21:37:08.737 INFO:teuthology.orchestra.run.smithi092.stdout: "client_addr": "v1:172.21.15.92:0/436606717", 2024-03-11T21:37:08.737 INFO:teuthology.orchestra.run.smithi092.stdout: "tid": 88135 2024-03-11T21:37:08.737 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:37:08.737 INFO:teuthology.orchestra.run.smithi092.stdout: "events": [ 2024-03-11T21:37:08.737 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:37:08.737 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "initiated", 2024-03-11T21:37:08.737 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:32:56.040513+0000", 2024-03-11T21:37:08.737 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-03-11T21:37:08.737 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:37:08.737 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:37:08.737 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "header_read", 2024-03-11T21:37:08.737 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:32:56.040513+0000", 2024-03-11T21:37:08.737 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-03-11T21:37:08.738 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:37:08.738 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:37:08.738 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "throttled", 2024-03-11T21:37:08.738 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:32:56.040515+0000", 2024-03-11T21:37:08.738 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.767e-06 2024-03-11T21:37:08.738 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:37:08.738 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:37:08.738 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "all_read", 2024-03-11T21:37:08.738 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:32:56.040539+0000", 2024-03-11T21:37:08.738 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 2.4485e-05 2024-03-11T21:37:08.738 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:37:08.738 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:37:08.738 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "dispatched", 2024-03-11T21:37:08.738 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:32:56.040554+0000", 2024-03-11T21:37:08.738 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.5034e-05 2024-03-11T21:37:08.739 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:37:08.739 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:37:08.739 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "queued_for_pg", 2024-03-11T21:37:08.739 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:32:56.040562+0000", 2024-03-11T21:37:08.739 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 8.0220000000000004e-06 2024-03-11T21:37:08.739 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:37:08.739 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:37:08.741 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-03-11T21:37:08.742 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:32:56.040611+0000", 2024-03-11T21:37:08.742 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 4.8692000000000001e-05 2024-03-11T21:37:08.742 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:37:08.742 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:37:08.742 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "started", 2024-03-11T21:37:08.742 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:32:56.040724+0000", 2024-03-11T21:37:08.742 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.000113192 2024-03-11T21:37:08.742 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:37:08.742 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:37:08.742 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "waiting for subops from 0,2", 2024-03-11T21:37:08.742 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:32:56.040878+0000", 2024-03-11T21:37:08.742 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00015365999999999999 2024-03-11T21:37:08.742 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:37:08.742 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:37:08.743 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "op_commit", 2024-03-11T21:37:08.743 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:32:56.061039+0000", 2024-03-11T21:37:08.744 INFO:teuthology.orchestra.run.smithi092.stderr:Inferring config /var/lib/ceph/00db0016-dfed-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T21:37:08.744 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:37:08.745 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:37:08.745 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:37:08.745 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.020160919999999999 2024-03-11T21:37:08.745 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:37:08.745 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:37:08.745 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-03-11T21:37:08.745 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:32:56.212128+0000", 2024-03-11T21:37:08.745 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.151089433 2024-03-11T21:37:08.746 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:37:08.746 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:37:08.746 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "done", 2024-03-11T21:37:08.746 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:32:56.212146+0000", 2024-03-11T21:37:08.746 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.7604999999999998e-05 2024-03-11T21:37:08.746 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-03-11T21:37:08.746 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-03-11T21:37:08.746 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-03-11T21:37:08.746 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:37:08.746 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:37:08.746 INFO:teuthology.orchestra.run.smithi092.stdout: "description": "osd_op(client.15150.0:88143 8.1 8:88a030b3:::benchmark_data_smithi092_104865_object88142:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e180)", 2024-03-11T21:37:08.746 INFO:teuthology.orchestra.run.smithi092.stdout: "initiated_at": "2024-03-11T21:32:56.054816+0000", 2024-03-11T21:37:08.746 INFO:teuthology.orchestra.run.smithi092.stdout: "age": 252.67120541400001, 2024-03-11T21:37:08.746 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.16188037399999999, 2024-03-11T21:37:08.747 INFO:teuthology.orchestra.run.smithi092.stdout: "type_data": { 2024-03-11T21:37:08.747 INFO:teuthology.orchestra.run.smithi092.stdout: "flag_point": "reached pg", 2024-03-11T21:37:08.747 INFO:teuthology.orchestra.run.smithi092.stdout: "client_info": { 2024-03-11T21:37:08.747 INFO:teuthology.orchestra.run.smithi092.stdout: "client": "client.15150", 2024-03-11T21:37:08.747 INFO:teuthology.orchestra.run.smithi092.stdout: "client_addr": "v1:172.21.15.92:0/436606717", 2024-03-11T21:37:08.747 INFO:teuthology.orchestra.run.smithi092.stdout: "tid": 88143 2024-03-11T21:37:08.747 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:37:08.747 INFO:teuthology.orchestra.run.smithi092.stdout: "events": [ 2024-03-11T21:37:08.747 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:37:08.747 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "initiated", 2024-03-11T21:37:08.747 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:32:56.054816+0000", 2024-03-11T21:37:08.747 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-03-11T21:37:08.747 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:37:08.747 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:37:08.747 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "header_read", 2024-03-11T21:37:08.748 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:32:56.054816+0000", 2024-03-11T21:37:08.748 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-03-11T21:37:08.748 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:37:08.748 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:37:08.748 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "throttled", 2024-03-11T21:37:08.748 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:32:56.054819+0000", 2024-03-11T21:37:08.748 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 2.915e-06 2024-03-11T21:37:08.748 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:37:08.748 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:37:08.748 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "all_read", 2024-03-11T21:37:08.748 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:32:56.054863+0000", 2024-03-11T21:37:08.748 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 4.3383000000000001e-05 2024-03-11T21:37:08.748 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:37:08.748 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:37:08.749 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "dispatched", 2024-03-11T21:37:08.749 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:32:56.054885+0000", 2024-03-11T21:37:08.749 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 2.2507999999999999e-05 2024-03-11T21:37:08.749 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:37:08.749 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:37:08.749 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "queued_for_pg", 2024-03-11T21:37:08.749 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:32:56.054896+0000", 2024-03-11T21:37:08.749 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.1450999999999999e-05 2024-03-11T21:37:08.749 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:37:08.749 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:37:08.749 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-03-11T21:37:08.749 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:32:56.058263+0000", 2024-03-11T21:37:08.749 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.003366399 2024-03-11T21:37:08.749 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:37:08.750 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:37:08.750 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "started", 2024-03-11T21:37:08.750 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:32:56.058391+0000", 2024-03-11T21:37:08.750 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.000128477 2024-03-11T21:37:08.750 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:37:08.750 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:37:08.750 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "waiting for subops from 0,2", 2024-03-11T21:37:08.750 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:32:56.058527+0000", 2024-03-11T21:37:08.750 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00013525700000000001 2024-03-11T21:37:08.750 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:37:08.750 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:37:08.750 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "op_commit", 2024-03-11T21:37:08.750 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:32:56.061984+0000", 2024-03-11T21:37:08.750 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00345729 2024-03-11T21:37:08.750 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:37:08.751 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:37:08.751 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-03-11T21:37:08.751 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:32:56.216682+0000", 2024-03-11T21:37:08.751 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.15469781699999999 2024-03-11T21:37:08.751 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:37:08.751 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:37:08.751 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "done", 2024-03-11T21:37:08.751 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:32:56.216697+0000", 2024-03-11T21:37:08.751 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.4877e-05 2024-03-11T21:37:08.751 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-03-11T21:37:08.751 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-03-11T21:37:08.751 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-03-11T21:37:08.751 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:37:08.751 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:37:08.752 INFO:teuthology.orchestra.run.smithi092.stdout: "description": "osd_op(client.15150.0:88144 8.3 8:c2cccf37:::benchmark_data_smithi092_104865_object88143:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e180)", 2024-03-11T21:37:08.752 INFO:teuthology.orchestra.run.smithi092.stdout: "initiated_at": "2024-03-11T21:32:56.054918+0000", 2024-03-11T21:37:08.752 INFO:teuthology.orchestra.run.smithi092.stdout: "age": 252.67110376799999, 2024-03-11T21:37:08.752 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.16754129500000001, 2024-03-11T21:37:08.752 INFO:teuthology.orchestra.run.smithi092.stdout: "type_data": { 2024-03-11T21:37:08.752 INFO:teuthology.orchestra.run.smithi092.stdout: "flag_point": "reached pg", 2024-03-11T21:37:08.752 INFO:teuthology.orchestra.run.smithi092.stdout: "client_info": { 2024-03-11T21:37:08.752 INFO:teuthology.orchestra.run.smithi092.stdout: "client": "client.15150", 2024-03-11T21:37:08.752 INFO:teuthology.orchestra.run.smithi092.stdout: "client_addr": "v1:172.21.15.92:0/436606717", 2024-03-11T21:37:08.752 INFO:teuthology.orchestra.run.smithi092.stdout: "tid": 88144 2024-03-11T21:37:08.752 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:37:08.752 INFO:teuthology.orchestra.run.smithi092.stdout: "events": [ 2024-03-11T21:37:08.752 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:37:08.752 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "initiated", 2024-03-11T21:37:08.752 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:32:56.054918+0000", 2024-03-11T21:37:08.753 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-03-11T21:37:08.753 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:37:08.753 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:37:08.753 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "header_read", 2024-03-11T21:37:08.753 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:32:56.054918+0000", 2024-03-11T21:37:08.753 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-03-11T21:37:08.753 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:37:08.753 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:37:08.753 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "throttled", 2024-03-11T21:37:08.753 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:32:56.054919+0000", 2024-03-11T21:37:08.753 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 7.4700000000000001e-07 2024-03-11T21:37:08.753 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:37:08.753 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:37:08.753 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "all_read", 2024-03-11T21:37:08.754 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:32:56.054941+0000", 2024-03-11T21:37:08.754 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 2.2838999999999999e-05 2024-03-11T21:37:08.754 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:37:08.754 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:37:08.754 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "dispatched", 2024-03-11T21:37:08.754 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:32:56.054951+0000", 2024-03-11T21:37:08.754 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 9.7499999999999998e-06 2024-03-11T21:37:08.754 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:37:08.754 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:37:08.754 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "queued_for_pg", 2024-03-11T21:37:08.754 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:32:56.054957+0000", 2024-03-11T21:37:08.754 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 5.7729999999999996e-06 2024-03-11T21:37:08.754 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:37:08.754 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:37:08.755 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-03-11T21:37:08.755 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:32:56.055030+0000", 2024-03-11T21:37:08.755 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 7.3035999999999997e-05 2024-03-11T21:37:08.755 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:37:08.755 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:37:08.755 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "started", 2024-03-11T21:37:08.755 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:32:56.055180+0000", 2024-03-11T21:37:08.755 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00014947800000000001 2024-03-11T21:37:08.755 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:37:08.755 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:37:08.755 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "waiting for subops from 5,6", 2024-03-11T21:37:08.755 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:32:56.055328+0000", 2024-03-11T21:37:08.755 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00014811800000000001 2024-03-11T21:37:08.755 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:37:08.755 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:37:08.756 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "op_commit", 2024-03-11T21:37:08.756 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:32:56.060662+0000", 2024-03-11T21:37:08.756 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.005333979 2024-03-11T21:37:08.756 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:37:08.756 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:37:08.756 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "sub_op_commit_rec", 2024-03-11T21:37:08.756 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:32:56.065170+0000", 2024-03-11T21:37:08.756 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.0045079669999999999 2024-03-11T21:37:08.756 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:37:08.756 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:37:08.756 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-03-11T21:37:08.756 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:32:56.222444+0000", 2024-03-11T21:37:08.756 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.157274264 2024-03-11T21:37:08.756 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:37:08.756 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:37:08.757 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "done", 2024-03-11T21:37:08.757 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:32:56.222459+0000", 2024-03-11T21:37:08.757 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.5344000000000001e-05 2024-03-11T21:37:08.757 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-03-11T21:37:08.757 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-03-11T21:37:08.757 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-03-11T21:37:08.757 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:37:08.757 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:37:08.757 INFO:teuthology.orchestra.run.smithi092.stdout: "description": "osd_op(client.15150.0:103690 8.13 8:c8430642:::benchmark_data_smithi092_104865_object103689:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e191)", 2024-03-11T21:37:08.757 INFO:teuthology.orchestra.run.smithi092.stdout: "initiated_at": "2024-03-11T21:33:12.941320+0000", 2024-03-11T21:37:08.757 INFO:teuthology.orchestra.run.smithi092.stdout: "age": 235.78470206399999, 2024-03-11T21:37:08.757 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.133193281, 2024-03-11T21:37:08.757 INFO:teuthology.orchestra.run.smithi092.stdout: "type_data": { 2024-03-11T21:37:08.757 INFO:teuthology.orchestra.run.smithi092.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-11T21:37:08.757 INFO:teuthology.orchestra.run.smithi092.stdout: "client_info": { 2024-03-11T21:37:08.758 INFO:teuthology.orchestra.run.smithi092.stdout: "client": "client.15150", 2024-03-11T21:37:08.758 INFO:teuthology.orchestra.run.smithi092.stdout: "client_addr": "v1:172.21.15.92:0/436606717", 2024-03-11T21:37:08.758 INFO:teuthology.orchestra.run.smithi092.stdout: "tid": 103690 2024-03-11T21:37:08.758 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:37:08.758 INFO:teuthology.orchestra.run.smithi092.stdout: "events": [ 2024-03-11T21:37:08.758 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:37:08.758 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "initiated", 2024-03-11T21:37:08.758 INFO:teuthology.orchestra.run.smithi092.stderr:Inferring config /var/lib/ceph/00db0016-dfed-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T21:37:08.759 INFO:teuthology.orchestra.run.smithi092.stderr:Inferring config /var/lib/ceph/00db0016-dfed-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T21:37:08.759 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:33:12.941320+0000", 2024-03-11T21:37:08.759 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-03-11T21:37:08.759 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:37:08.759 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:37:08.759 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "header_read", 2024-03-11T21:37:08.759 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:33:12.941320+0000", 2024-03-11T21:37:08.760 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-03-11T21:37:08.760 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:37:08.760 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:37:08.760 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "throttled", 2024-03-11T21:37:08.760 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:33:12.941322+0000", 2024-03-11T21:37:08.760 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 2.063e-06 2024-03-11T21:37:08.760 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:37:08.760 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:37:08.760 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "all_read", 2024-03-11T21:37:08.760 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:33:12.941356+0000", 2024-03-11T21:37:08.760 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 3.4261000000000003e-05 2024-03-11T21:37:08.760 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:37:08.760 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:37:08.760 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "dispatched", 2024-03-11T21:37:08.760 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:33:12.941374+0000", 2024-03-11T21:37:08.761 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.7663999999999999e-05 2024-03-11T21:37:08.761 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:37:08.761 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:37:08.761 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "queued_for_pg", 2024-03-11T21:37:08.761 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:33:12.941382+0000", 2024-03-11T21:37:08.761 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 8.7290000000000006e-06 2024-03-11T21:37:08.761 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:37:08.761 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:37:08.761 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-03-11T21:37:08.761 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:33:12.941987+0000", 2024-03-11T21:37:08.761 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00060477399999999998 2024-03-11T21:37:08.761 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:37:08.761 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:37:08.761 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "started", 2024-03-11T21:37:08.762 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:33:12.942170+0000", 2024-03-11T21:37:08.762 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00018326400000000001 2024-03-11T21:37:08.762 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:37:08.762 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:37:08.762 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "waiting for subops from 2,3,5,6", 2024-03-11T21:37:08.762 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:33:12.942539+0000", 2024-03-11T21:37:08.762 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00036884500000000002 2024-03-11T21:37:08.762 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:37:08.762 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:37:08.762 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "op_commit", 2024-03-11T21:37:08.762 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:33:12.944328+0000", 2024-03-11T21:37:08.762 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.0017892520000000001 2024-03-11T21:37:08.762 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:37:08.762 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:37:08.762 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "sub_op_commit_rec", 2024-03-11T21:37:08.763 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:33:12.948234+0000", 2024-03-11T21:37:08.763 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.0039054139999999998 2024-03-11T21:37:08.763 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:37:08.763 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:37:08.763 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "sub_op_commit_rec", 2024-03-11T21:37:08.763 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:33:12.948366+0000", 2024-03-11T21:37:08.763 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00013263700000000001 2024-03-11T21:37:08.763 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:37:08.763 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:37:08.763 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "sub_op_commit_rec", 2024-03-11T21:37:08.763 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:33:12.954113+0000", 2024-03-11T21:37:08.763 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.0057465989999999998 2024-03-11T21:37:08.763 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:37:08.763 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:37:08.763 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "sub_op_commit_rec", 2024-03-11T21:37:08.764 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:33:13.074257+0000", 2024-03-11T21:37:08.764 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.120143577 2024-03-11T21:37:08.764 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:37:08.764 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:37:08.764 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "commit_sent", 2024-03-11T21:37:08.764 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:33:13.074339+0000", 2024-03-11T21:37:08.764 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 8.2218000000000005e-05 2024-03-11T21:37:08.764 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:37:08.764 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:37:08.764 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "done", 2024-03-11T21:37:08.764 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:33:13.074513+0000", 2024-03-11T21:37:08.764 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00017398399999999999 2024-03-11T21:37:08.764 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-03-11T21:37:08.764 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-03-11T21:37:08.764 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-03-11T21:37:08.765 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:37:08.765 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:37:08.765 INFO:teuthology.orchestra.run.smithi092.stdout: "description": "osd_op(client.15150.0:105925 8.17 8:ea318f53:::benchmark_data_smithi092_104865_object105924:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e192)", 2024-03-11T21:37:08.765 INFO:teuthology.orchestra.run.smithi092.stdout: "initiated_at": "2024-03-11T21:33:15.867731+0000", 2024-03-11T21:37:08.765 INFO:teuthology.orchestra.run.smithi092.stdout: "age": 232.858290493, 2024-03-11T21:37:08.765 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.145738176, 2024-03-11T21:37:08.765 INFO:teuthology.orchestra.run.smithi092.stdout: "type_data": { 2024-03-11T21:37:08.765 INFO:teuthology.orchestra.run.smithi092.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-11T21:37:08.765 INFO:teuthology.orchestra.run.smithi092.stdout: "client_info": { 2024-03-11T21:37:08.765 INFO:teuthology.orchestra.run.smithi092.stdout: "client": "client.15150", 2024-03-11T21:37:08.765 INFO:teuthology.orchestra.run.smithi092.stdout: "client_addr": "v1:172.21.15.92:0/436606717", 2024-03-11T21:37:08.765 INFO:teuthology.orchestra.run.smithi092.stdout: "tid": 105925 2024-03-11T21:37:08.765 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:37:08.765 INFO:teuthology.orchestra.run.smithi092.stdout: "events": [ 2024-03-11T21:37:08.766 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:37:08.766 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "initiated", 2024-03-11T21:37:08.766 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:33:15.867731+0000", 2024-03-11T21:37:08.766 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-03-11T21:37:08.766 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:37:08.766 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:37:08.766 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "header_read", 2024-03-11T21:37:08.766 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:33:15.867731+0000", 2024-03-11T21:37:08.766 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-03-11T21:37:08.766 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:37:08.766 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:37:08.766 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "throttled", 2024-03-11T21:37:08.766 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:33:15.867733+0000", 2024-03-11T21:37:08.766 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.7290000000000001e-06 2024-03-11T21:37:08.766 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:37:08.767 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:37:08.767 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "all_read", 2024-03-11T21:37:08.767 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:33:15.867757+0000", 2024-03-11T21:37:08.767 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 2.385e-05 2024-03-11T21:37:08.767 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:37:08.767 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:37:08.767 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "dispatched", 2024-03-11T21:37:08.767 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:33:15.867767+0000", 2024-03-11T21:37:08.767 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.0319000000000001e-05 2024-03-11T21:37:08.767 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:37:08.767 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:37:08.767 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "queued_for_pg", 2024-03-11T21:37:08.767 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:33:15.867775+0000", 2024-03-11T21:37:08.767 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 7.6290000000000001e-06 2024-03-11T21:37:08.768 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:37:08.768 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:37:08.768 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-03-11T21:37:08.768 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:33:15.868047+0000", 2024-03-11T21:37:08.768 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.000271821 2024-03-11T21:37:08.768 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:37:08.768 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:37:08.768 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "started", 2024-03-11T21:37:08.768 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:33:15.868186+0000", 2024-03-11T21:37:08.768 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.000139486 2024-03-11T21:37:08.768 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:37:08.768 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:37:08.768 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "waiting for subops from 0,2,3,6", 2024-03-11T21:37:08.768 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:33:15.868590+0000", 2024-03-11T21:37:08.768 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00040403299999999999 2024-03-11T21:37:08.769 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:37:08.769 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:37:08.769 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "sub_op_commit_rec", 2024-03-11T21:37:08.769 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:33:15.870545+0000", 2024-03-11T21:37:08.769 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.0019545180000000001 2024-03-11T21:37:08.769 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:37:08.769 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:37:08.769 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "sub_op_commit_rec", 2024-03-11T21:37:08.769 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:33:15.871503+0000", 2024-03-11T21:37:08.769 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00095892000000000004 2024-03-11T21:37:08.769 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:37:08.769 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:37:08.769 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "op_commit", 2024-03-11T21:37:08.769 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:33:15.873053+0000", 2024-03-11T21:37:08.769 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00154922 2024-03-11T21:37:08.770 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:37:08.770 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:37:08.770 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "sub_op_commit_rec", 2024-03-11T21:37:08.770 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:33:15.874401+0000", 2024-03-11T21:37:08.770 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.0013484829999999999 2024-03-11T21:37:08.770 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:37:08.770 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:37:08.770 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "sub_op_commit_rec", 2024-03-11T21:37:08.770 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:33:16.013260+0000", 2024-03-11T21:37:08.770 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.13885871599999999 2024-03-11T21:37:08.770 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:37:08.770 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:37:08.770 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "commit_sent", 2024-03-11T21:37:08.770 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:33:16.013351+0000", 2024-03-11T21:37:08.770 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 9.1459999999999995e-05 2024-03-11T21:37:08.771 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:37:08.771 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:37:08.771 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "done", 2024-03-11T21:37:08.771 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:33:16.013469+0000", 2024-03-11T21:37:08.771 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.000117992 2024-03-11T21:37:08.771 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-03-11T21:37:08.771 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-03-11T21:37:08.771 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-03-11T21:37:08.771 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:37:08.771 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:37:08.771 INFO:teuthology.orchestra.run.smithi092.stdout: "description": "osd_op(client.15150.0:105926 8.7 8:e3368c86:::benchmark_data_smithi092_104865_object105925:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e192)", 2024-03-11T21:37:08.771 INFO:teuthology.orchestra.run.smithi092.stdout: "initiated_at": "2024-03-11T21:33:15.868886+0000", 2024-03-11T21:37:08.771 INFO:teuthology.orchestra.run.smithi092.stdout: "age": 232.857135484, 2024-03-11T21:37:08.771 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.14424000000000001, 2024-03-11T21:37:08.771 INFO:teuthology.orchestra.run.smithi092.stdout: "type_data": { 2024-03-11T21:37:08.772 INFO:teuthology.orchestra.run.smithi092.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-11T21:37:08.772 INFO:teuthology.orchestra.run.smithi092.stdout: "client_info": { 2024-03-11T21:37:08.772 INFO:teuthology.orchestra.run.smithi092.stdout: "client": "client.15150", 2024-03-11T21:37:08.772 INFO:teuthology.orchestra.run.smithi092.stdout: "client_addr": "v1:172.21.15.92:0/436606717", 2024-03-11T21:37:08.772 INFO:teuthology.orchestra.run.smithi092.stdout: "tid": 105926 2024-03-11T21:37:08.772 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:37:08.772 INFO:teuthology.orchestra.run.smithi092.stdout: "events": [ 2024-03-11T21:37:08.772 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:37:08.772 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "initiated", 2024-03-11T21:37:08.772 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:33:15.868886+0000", 2024-03-11T21:37:08.772 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-03-11T21:37:08.772 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:37:08.772 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:37:08.772 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "header_read", 2024-03-11T21:37:08.773 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:33:15.868886+0000", 2024-03-11T21:37:08.773 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-03-11T21:37:08.773 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:37:08.773 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:37:08.773 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "throttled", 2024-03-11T21:37:08.773 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:33:15.868888+0000", 2024-03-11T21:37:08.773 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.6300000000000001e-06 2024-03-11T21:37:08.773 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:37:08.773 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:37:08.773 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "all_read", 2024-03-11T21:37:08.773 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:33:15.868979+0000", 2024-03-11T21:37:08.773 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 9.0876000000000002e-05 2024-03-11T21:37:08.773 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:37:08.773 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:37:08.773 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "dispatched", 2024-03-11T21:37:08.774 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:33:15.868989+0000", 2024-03-11T21:37:08.774 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.0453e-05 2024-03-11T21:37:08.774 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:37:08.774 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:37:08.774 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "queued_for_pg", 2024-03-11T21:37:08.774 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:33:15.868997+0000", 2024-03-11T21:37:08.774 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 7.8650000000000001e-06 2024-03-11T21:37:08.774 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:37:08.774 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:37:08.774 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-03-11T21:37:08.774 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:33:15.869218+0000", 2024-03-11T21:37:08.774 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00022079900000000001 2024-03-11T21:37:08.774 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:37:08.774 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:37:08.775 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "started", 2024-03-11T21:37:08.775 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:33:15.869338+0000", 2024-03-11T21:37:08.775 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.000120003 2024-03-11T21:37:08.775 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:37:08.775 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:37:08.775 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "waiting for subops from 3,6", 2024-03-11T21:37:08.775 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:33:15.869647+0000", 2024-03-11T21:37:08.775 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00030924400000000002 2024-03-11T21:37:08.775 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:37:08.775 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:37:08.775 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "sub_op_commit_rec", 2024-03-11T21:37:08.775 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:33:15.872513+0000", 2024-03-11T21:37:08.775 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.0028658009999999999 2024-03-11T21:37:08.775 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:37:08.776 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:37:08.776 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "op_commit", 2024-03-11T21:37:08.776 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:33:15.873060+0000", 2024-03-11T21:37:08.776 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00054739700000000003 2024-03-11T21:37:08.776 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:37:08.776 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:37:08.776 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "sub_op_commit_rec", 2024-03-11T21:37:08.776 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:33:15.875877+0000", 2024-03-11T21:37:08.776 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.002816813 2024-03-11T21:37:08.776 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:37:08.776 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:37:08.776 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "commit_sent", 2024-03-11T21:37:08.776 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:33:16.013082+0000", 2024-03-11T21:37:08.776 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.13720491800000001 2024-03-11T21:37:08.776 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:37:08.777 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:37:08.777 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "done", 2024-03-11T21:37:08.777 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:33:16.013126+0000", 2024-03-11T21:37:08.777 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 4.4200999999999999e-05 2024-03-11T21:37:08.777 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-03-11T21:37:08.777 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-03-11T21:37:08.777 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-03-11T21:37:08.777 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:37:08.777 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:37:08.777 INFO:teuthology.orchestra.run.smithi092.stdout: "description": "osd_op(client.15150.0:105927 8.a 8:5cfca5c5:::benchmark_data_smithi092_104865_object105926:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e192)", 2024-03-11T21:37:08.777 INFO:teuthology.orchestra.run.smithi092.stdout: "initiated_at": "2024-03-11T21:33:15.869018+0000", 2024-03-11T21:37:08.777 INFO:teuthology.orchestra.run.smithi092.stdout: "age": 232.857003709, 2024-03-11T21:37:08.777 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.14413793499999999, 2024-03-11T21:37:08.777 INFO:teuthology.orchestra.run.smithi092.stdout: "type_data": { 2024-03-11T21:37:08.778 INFO:teuthology.orchestra.run.smithi092.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-11T21:37:08.778 INFO:teuthology.orchestra.run.smithi092.stdout: "client_info": { 2024-03-11T21:37:08.778 INFO:teuthology.orchestra.run.smithi092.stdout: "client": "client.15150", 2024-03-11T21:37:08.778 INFO:teuthology.orchestra.run.smithi092.stdout: "client_addr": "v1:172.21.15.92:0/436606717", 2024-03-11T21:37:08.778 INFO:teuthology.orchestra.run.smithi092.stdout: "tid": 105927 2024-03-11T21:37:08.778 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:37:08.778 INFO:teuthology.orchestra.run.smithi092.stdout: "events": [ 2024-03-11T21:37:08.778 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:37:08.778 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "initiated", 2024-03-11T21:37:08.778 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:33:15.869018+0000", 2024-03-11T21:37:08.778 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-03-11T21:37:08.778 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:37:08.778 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:37:08.778 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "header_read", 2024-03-11T21:37:08.778 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:33:15.869018+0000", 2024-03-11T21:37:08.779 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-03-11T21:37:08.779 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:37:08.779 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:37:08.779 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "throttled", 2024-03-11T21:37:08.779 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:33:15.869019+0000", 2024-03-11T21:37:08.779 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 9.2999999999999999e-07 2024-03-11T21:37:08.779 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:37:08.779 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:37:08.779 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "all_read", 2024-03-11T21:37:08.779 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:33:15.869041+0000", 2024-03-11T21:37:08.779 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 2.207e-05 2024-03-11T21:37:08.779 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:37:08.779 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:37:08.779 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "dispatched", 2024-03-11T21:37:08.779 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:33:15.869048+0000", 2024-03-11T21:37:08.780 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 7.2300000000000002e-06 2024-03-11T21:37:08.780 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:37:08.780 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:37:08.780 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "queued_for_pg", 2024-03-11T21:37:08.780 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:33:15.869054+0000", 2024-03-11T21:37:08.780 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 6.1340000000000001e-06 2024-03-11T21:37:08.780 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:37:08.780 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:37:08.780 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-03-11T21:37:08.780 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:33:15.870171+0000", 2024-03-11T21:37:08.780 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.001117095 2024-03-11T21:37:08.780 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:37:08.780 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:37:08.780 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "started", 2024-03-11T21:37:08.780 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:33:15.870290+0000", 2024-03-11T21:37:08.781 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.000118319 2024-03-11T21:37:08.781 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:37:08.781 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:37:08.781 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "waiting for subops from 2,5", 2024-03-11T21:37:08.781 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:33:15.870421+0000", 2024-03-11T21:37:08.781 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00013092600000000001 2024-03-11T21:37:08.781 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:37:08.781 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:37:08.781 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "op_commit", 2024-03-11T21:37:08.781 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:33:15.873077+0000", 2024-03-11T21:37:08.781 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.0026562930000000001 2024-03-11T21:37:08.781 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:37:08.781 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:37:08.781 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "sub_op_commit_rec", 2024-03-11T21:37:08.781 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:33:15.874033+0000", 2024-03-11T21:37:08.781 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00095567700000000005 2024-03-11T21:37:08.782 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:37:08.782 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:37:08.782 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "sub_op_commit_rec", 2024-03-11T21:37:08.782 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:33:16.012990+0000", 2024-03-11T21:37:08.782 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.13895734300000001 2024-03-11T21:37:08.782 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:37:08.782 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:37:08.782 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "commit_sent", 2024-03-11T21:37:08.782 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:33:16.013121+0000", 2024-03-11T21:37:08.782 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00013089800000000001 2024-03-11T21:37:08.782 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:37:08.782 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:37:08.782 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "done", 2024-03-11T21:37:08.782 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:33:16.013156+0000", 2024-03-11T21:37:08.782 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 3.502e-05 2024-03-11T21:37:08.783 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-03-11T21:37:08.783 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-03-11T21:37:08.783 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-03-11T21:37:08.783 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:37:08.783 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:37:08.783 INFO:teuthology.orchestra.run.smithi092.stdout: "description": "osd_repop(client.15150.0:105934 8.b e192/157 8:d3c779ec:::benchmark_data_smithi092_104865_object105933:head v 192'6695, mlcod=192'6693)", 2024-03-11T21:37:08.783 INFO:teuthology.orchestra.run.smithi092.stdout: "initiated_at": "2024-03-11T21:33:15.874970+0000", 2024-03-11T21:37:08.783 INFO:teuthology.orchestra.run.smithi092.stdout: "age": 232.851051953, 2024-03-11T21:37:08.783 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.13726905, 2024-03-11T21:37:08.783 INFO:teuthology.orchestra.run.smithi092.stdout: "type_data": { 2024-03-11T21:37:08.783 INFO:teuthology.orchestra.run.smithi092.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-11T21:37:08.783 INFO:teuthology.orchestra.run.smithi092.stdout: "events": [ 2024-03-11T21:37:08.783 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:37:08.783 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "initiated", 2024-03-11T21:37:08.783 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:33:15.874970+0000", 2024-03-11T21:37:08.784 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-03-11T21:37:08.784 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:37:08.784 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:37:08.784 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "header_read", 2024-03-11T21:37:08.784 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:33:15.874970+0000", 2024-03-11T21:37:08.784 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-03-11T21:37:08.784 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:37:08.784 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:37:08.784 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "throttled", 2024-03-11T21:37:08.784 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:33:15.874971+0000", 2024-03-11T21:37:08.784 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.2139999999999999e-06 2024-03-11T21:37:08.784 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:37:08.784 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:37:08.784 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "all_read", 2024-03-11T21:37:08.784 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:33:15.874995+0000", 2024-03-11T21:37:08.785 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 2.4119999999999999e-05 2024-03-11T21:37:08.785 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:37:08.785 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:37:08.786 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "dispatched", 2024-03-11T21:37:08.786 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:33:15.875018+0000", 2024-03-11T21:37:08.786 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 2.2495000000000001e-05 2024-03-11T21:37:08.786 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:37:08.786 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:37:08.786 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "queued_for_pg", 2024-03-11T21:37:08.786 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:33:15.875025+0000", 2024-03-11T21:37:08.786 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 7.3529999999999997e-06 2024-03-11T21:37:08.786 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:37:08.786 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:37:08.786 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-03-11T21:37:08.786 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:33:15.875224+0000", 2024-03-11T21:37:08.787 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.000198743 2024-03-11T21:37:08.787 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:37:08.787 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:37:08.787 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "started", 2024-03-11T21:37:08.787 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:33:15.875244+0000", 2024-03-11T21:37:08.787 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 2.0027e-05 2024-03-11T21:37:08.787 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:37:08.787 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:37:08.787 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "commit_sent", 2024-03-11T21:37:08.787 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:33:16.012192+0000", 2024-03-11T21:37:08.787 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.13694829 2024-03-11T21:37:08.787 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:37:08.787 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:37:08.787 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "done", 2024-03-11T21:37:08.788 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:33:16.012239+0000", 2024-03-11T21:37:08.788 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 4.6808000000000003e-05 2024-03-11T21:37:08.788 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-03-11T21:37:08.788 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-03-11T21:37:08.788 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-03-11T21:37:08.788 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:37:08.788 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:37:08.788 INFO:teuthology.orchestra.run.smithi092.stdout: "description": "MOSDPGPushReply(8.13 192/184 [PushReplyOp(8:cc53b3b0:::benchmark_data_smithi092_104865_object68064:head)])", 2024-03-11T21:37:08.788 INFO:teuthology.orchestra.run.smithi092.stdout: "initiated_at": "2024-03-11T21:33:15.875722+0000", 2024-03-11T21:37:08.788 INFO:teuthology.orchestra.run.smithi092.stdout: "age": 232.85029936000001, 2024-03-11T21:37:08.788 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.13666853200000001, 2024-03-11T21:37:08.788 INFO:teuthology.orchestra.run.smithi092.stdout: "type_data": { 2024-03-11T21:37:08.788 INFO:teuthology.orchestra.run.smithi092.stdout: "flag_point": "reached pg", 2024-03-11T21:37:08.788 INFO:teuthology.orchestra.run.smithi092.stdout: "events": [ 2024-03-11T21:37:08.789 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:37:08.789 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "initiated", 2024-03-11T21:37:08.789 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:33:15.875722+0000", 2024-03-11T21:37:08.789 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-03-11T21:37:08.789 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:37:08.789 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:37:08.789 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "header_read", 2024-03-11T21:37:08.789 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:33:15.875722+0000", 2024-03-11T21:37:08.789 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-03-11T21:37:08.789 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:37:08.789 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:37:08.789 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "throttled", 2024-03-11T21:37:08.789 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:33:15.875723+0000", 2024-03-11T21:37:08.789 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 4.6100000000000001e-07 2024-03-11T21:37:08.790 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:37:08.790 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:37:08.790 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "all_read", 2024-03-11T21:37:08.790 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:33:15.875726+0000", 2024-03-11T21:37:08.790 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 3.0970000000000002e-06 2024-03-11T21:37:08.790 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:37:08.790 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:37:08.790 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "dispatched", 2024-03-11T21:37:08.790 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:33:15.875732+0000", 2024-03-11T21:37:08.790 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 5.9020000000000004e-06 2024-03-11T21:37:08.790 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:37:08.790 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:37:08.790 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "queued_for_pg", 2024-03-11T21:37:08.790 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:33:15.875739+0000", 2024-03-11T21:37:08.791 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 7.216e-06 2024-03-11T21:37:08.791 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:37:08.791 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:37:08.791 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-03-11T21:37:08.791 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:33:15.875814+0000", 2024-03-11T21:37:08.791 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 7.4584000000000003e-05 2024-03-11T21:37:08.791 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:37:08.791 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:37:08.791 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "done", 2024-03-11T21:37:08.791 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:33:16.012391+0000", 2024-03-11T21:37:08.791 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.136577272 2024-03-11T21:37:08.791 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-03-11T21:37:08.791 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-03-11T21:37:08.791 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-03-11T21:37:08.791 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:37:08.792 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:37:08.792 INFO:teuthology.orchestra.run.smithi092.stdout: "description": "osd_repop_reply(client.15150.0:105926 8.7 e192/181 ondisk, result = 0)", 2024-03-11T21:37:08.792 INFO:teuthology.orchestra.run.smithi092.stdout: "initiated_at": "2024-03-11T21:33:15.875800+0000", 2024-03-11T21:37:08.792 INFO:teuthology.orchestra.run.smithi092.stdout: "age": 232.85022137000001, 2024-03-11T21:37:08.792 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.13733347100000001, 2024-03-11T21:37:08.792 INFO:teuthology.orchestra.run.smithi092.stdout: "type_data": { 2024-03-11T21:37:08.792 INFO:teuthology.orchestra.run.smithi092.stdout: "flag_point": "started", 2024-03-11T21:37:08.792 INFO:teuthology.orchestra.run.smithi092.stdout: "events": [ 2024-03-11T21:37:08.792 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:37:08.792 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "initiated", 2024-03-11T21:37:08.792 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:33:15.875800+0000", 2024-03-11T21:37:08.792 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-03-11T21:37:08.792 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:37:08.792 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:37:08.792 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "header_read", 2024-03-11T21:37:08.793 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:33:15.875800+0000", 2024-03-11T21:37:08.793 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-03-11T21:37:08.793 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:37:08.793 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:37:08.793 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "throttled", 2024-03-11T21:37:08.793 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:33:15.875801+0000", 2024-03-11T21:37:08.793 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.0979999999999999e-06 2024-03-11T21:37:08.793 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:37:08.793 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:37:08.793 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "all_read", 2024-03-11T21:37:08.793 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:33:15.875806+0000", 2024-03-11T21:37:08.793 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 4.8489999999999997e-06 2024-03-11T21:37:08.793 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:37:08.793 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:37:08.794 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "dispatched", 2024-03-11T21:37:08.794 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:33:15.875813+0000", 2024-03-11T21:37:08.794 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 7.1080000000000004e-06 2024-03-11T21:37:08.794 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:37:08.794 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:37:08.794 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "queued_for_pg", 2024-03-11T21:37:08.794 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:33:15.875818+0000", 2024-03-11T21:37:08.794 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 4.9540000000000003e-06 2024-03-11T21:37:08.794 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:37:08.794 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:37:08.794 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-03-11T21:37:08.794 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:33:15.875864+0000", 2024-03-11T21:37:08.794 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 4.6094999999999998e-05 2024-03-11T21:37:08.794 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:37:08.795 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:37:08.795 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "started", 2024-03-11T21:37:08.795 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:33:15.875872+0000", 2024-03-11T21:37:08.795 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 7.1500000000000002e-06 2024-03-11T21:37:08.795 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:37:08.795 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:37:08.795 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "done", 2024-03-11T21:37:08.795 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:33:16.013134+0000", 2024-03-11T21:37:08.795 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.13726221699999999 2024-03-11T21:37:08.795 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-03-11T21:37:08.795 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-03-11T21:37:08.795 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-03-11T21:37:08.795 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:37:08.795 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:37:08.796 INFO:teuthology.orchestra.run.smithi092.stdout: "description": "osd_op(client.15150.0:107411 8.1 8:85752a3c:::benchmark_data_smithi092_104865_object107410:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e192)", 2024-03-11T21:37:08.796 INFO:teuthology.orchestra.run.smithi092.stdout: "initiated_at": "2024-03-11T21:33:17.604654+0000", 2024-03-11T21:37:08.796 INFO:teuthology.orchestra.run.smithi092.stdout: "age": 231.12136729700001, 2024-03-11T21:37:08.796 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.14808184699999999, 2024-03-11T21:37:08.796 INFO:teuthology.orchestra.run.smithi092.stdout: "type_data": { 2024-03-11T21:37:08.796 INFO:teuthology.orchestra.run.smithi092.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-11T21:37:08.796 INFO:teuthology.orchestra.run.smithi092.stdout: "client_info": { 2024-03-11T21:37:08.796 INFO:teuthology.orchestra.run.smithi092.stdout: "client": "client.15150", 2024-03-11T21:37:08.796 INFO:teuthology.orchestra.run.smithi092.stdout: "client_addr": "v1:172.21.15.92:0/436606717", 2024-03-11T21:37:08.796 INFO:teuthology.orchestra.run.smithi092.stdout: "tid": 107411 2024-03-11T21:37:08.796 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:37:08.796 INFO:teuthology.orchestra.run.smithi092.stdout: "events": [ 2024-03-11T21:37:08.796 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:37:08.796 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "initiated", 2024-03-11T21:37:08.796 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:33:17.604654+0000", 2024-03-11T21:37:08.796 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-03-11T21:37:08.796 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:37:08.797 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:37:08.797 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "header_read", 2024-03-11T21:37:08.797 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:33:17.604654+0000", 2024-03-11T21:37:08.797 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-03-11T21:37:08.797 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:37:08.797 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:37:08.797 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "throttled", 2024-03-11T21:37:08.797 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:33:17.604656+0000", 2024-03-11T21:37:08.797 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.812e-06 2024-03-11T21:37:08.797 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:37:08.797 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:37:08.797 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "all_read", 2024-03-11T21:37:08.797 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:33:17.604681+0000", 2024-03-11T21:37:08.797 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 2.5219999999999999e-05 2024-03-11T21:37:08.798 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:37:08.798 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:37:08.798 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "dispatched", 2024-03-11T21:37:08.798 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:33:17.604693+0000", 2024-03-11T21:37:08.798 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.1678999999999999e-05 2024-03-11T21:37:08.798 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:37:08.798 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:37:08.798 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "queued_for_pg", 2024-03-11T21:37:08.798 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:33:17.604701+0000", 2024-03-11T21:37:08.798 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 7.4479999999999997e-06 2024-03-11T21:37:08.798 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:37:08.798 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:37:08.798 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-03-11T21:37:08.798 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:33:17.604757+0000", 2024-03-11T21:37:08.798 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 5.6030999999999999e-05 2024-03-11T21:37:08.799 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:37:08.799 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:37:08.799 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "started", 2024-03-11T21:37:08.799 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:33:17.604889+0000", 2024-03-11T21:37:08.799 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.000132297 2024-03-11T21:37:08.799 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:37:08.799 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:37:08.799 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "waiting for subops from 0,2", 2024-03-11T21:37:08.799 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:33:17.605200+0000", 2024-03-11T21:37:08.799 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00031093899999999999 2024-03-11T21:37:08.799 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:37:08.799 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:37:08.799 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "op_commit", 2024-03-11T21:37:08.799 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:33:17.606932+0000", 2024-03-11T21:37:08.799 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.001732253 2024-03-11T21:37:08.800 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:37:08.800 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:37:08.800 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "sub_op_commit_rec", 2024-03-11T21:37:08.800 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:33:17.608901+0000", 2024-03-11T21:37:08.800 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.0019693549999999999 2024-03-11T21:37:08.800 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:37:08.800 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:37:08.800 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "sub_op_commit_rec", 2024-03-11T21:37:08.800 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:33:17.752511+0000", 2024-03-11T21:37:08.800 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.14360931699999999 2024-03-11T21:37:08.800 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:37:08.800 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:37:08.800 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "commit_sent", 2024-03-11T21:37:08.800 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:33:17.752688+0000", 2024-03-11T21:37:08.800 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.000177596 2024-03-11T21:37:08.801 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:37:08.801 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:37:08.801 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "done", 2024-03-11T21:37:08.801 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:33:17.752736+0000", 2024-03-11T21:37:08.801 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 4.7899999999999999e-05 2024-03-11T21:37:08.801 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-03-11T21:37:08.801 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-03-11T21:37:08.801 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-03-11T21:37:08.801 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:37:08.801 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:37:08.801 INFO:teuthology.orchestra.run.smithi092.stdout: "description": "osd_op(client.15150.0:107417 8.17 8:ed093fcf:::benchmark_data_smithi092_104865_object107416:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e192)", 2024-03-11T21:37:08.801 INFO:teuthology.orchestra.run.smithi092.stdout: "initiated_at": "2024-03-11T21:33:17.608265+0000", 2024-03-11T21:37:08.801 INFO:teuthology.orchestra.run.smithi092.stdout: "age": 231.11775635699999, 2024-03-11T21:37:08.801 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.14778530400000001, 2024-03-11T21:37:08.801 INFO:teuthology.orchestra.run.smithi092.stdout: "type_data": { 2024-03-11T21:37:08.802 INFO:teuthology.orchestra.run.smithi092.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-11T21:37:08.802 INFO:teuthology.orchestra.run.smithi092.stdout: "client_info": { 2024-03-11T21:37:08.802 INFO:teuthology.orchestra.run.smithi092.stdout: "client": "client.15150", 2024-03-11T21:37:08.802 INFO:teuthology.orchestra.run.smithi092.stdout: "client_addr": "v1:172.21.15.92:0/436606717", 2024-03-11T21:37:08.802 INFO:teuthology.orchestra.run.smithi092.stdout: "tid": 107417 2024-03-11T21:37:08.802 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:37:08.802 INFO:teuthology.orchestra.run.smithi092.stdout: "events": [ 2024-03-11T21:37:08.802 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:37:08.802 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "initiated", 2024-03-11T21:37:08.802 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:33:17.608265+0000", 2024-03-11T21:37:08.802 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-03-11T21:37:08.802 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:37:08.802 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:37:08.802 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "header_read", 2024-03-11T21:37:08.803 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:33:17.608265+0000", 2024-03-11T21:37:08.803 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-03-11T21:37:08.803 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:37:08.803 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:37:08.803 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "throttled", 2024-03-11T21:37:08.803 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:33:17.608267+0000", 2024-03-11T21:37:08.803 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.254e-06 2024-03-11T21:37:08.803 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:37:08.803 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:37:08.803 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "all_read", 2024-03-11T21:37:08.803 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:33:17.608287+0000", 2024-03-11T21:37:08.803 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 2.0678000000000001e-05 2024-03-11T21:37:08.803 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:37:08.803 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:37:08.803 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "dispatched", 2024-03-11T21:37:08.804 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:33:17.608297+0000", 2024-03-11T21:37:08.804 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 9.7580000000000004e-06 2024-03-11T21:37:08.804 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:37:08.804 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:37:08.804 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "queued_for_pg", 2024-03-11T21:37:08.804 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:33:17.608304+0000", 2024-03-11T21:37:08.804 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 7.1439999999999997e-06 2024-03-11T21:37:08.804 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:37:08.804 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:37:08.804 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-03-11T21:37:08.804 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:33:17.608364+0000", 2024-03-11T21:37:08.804 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 5.9524999999999997e-05 2024-03-11T21:37:08.804 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:37:08.804 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:37:08.804 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "started", 2024-03-11T21:37:08.804 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:33:17.608508+0000", 2024-03-11T21:37:08.805 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00014461699999999999 2024-03-11T21:37:08.805 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:37:08.805 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:37:08.805 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "waiting for subops from 0,2,3,6", 2024-03-11T21:37:08.805 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:33:17.608969+0000", 2024-03-11T21:37:08.805 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00046111699999999998 2024-03-11T21:37:08.805 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:37:08.805 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:37:08.805 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "op_commit", 2024-03-11T21:37:08.805 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:33:17.610759+0000", 2024-03-11T21:37:08.805 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.001789971 2024-03-11T21:37:08.805 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:37:08.805 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:37:08.805 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "sub_op_commit_rec", 2024-03-11T21:37:08.806 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:33:17.610972+0000", 2024-03-11T21:37:08.806 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.000213103 2024-03-11T21:37:08.806 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:37:08.806 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:37:08.806 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "sub_op_commit_rec", 2024-03-11T21:37:08.806 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:33:17.611916+0000", 2024-03-11T21:37:08.806 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00094380699999999996 2024-03-11T21:37:08.806 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:37:08.806 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:37:08.806 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "sub_op_commit_rec", 2024-03-11T21:37:08.806 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:33:17.612145+0000", 2024-03-11T21:37:08.806 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00022870400000000001 2024-03-11T21:37:08.806 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:37:08.806 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:37:08.806 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "sub_op_commit_rec", 2024-03-11T21:37:08.807 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:33:17.755847+0000", 2024-03-11T21:37:08.807 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.14370169099999999 2024-03-11T21:37:08.807 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:37:08.807 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:37:08.807 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "commit_sent", 2024-03-11T21:37:08.807 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:33:17.755910+0000", 2024-03-11T21:37:08.807 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 6.3010999999999997e-05 2024-03-11T21:37:08.807 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:37:08.807 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:37:08.807 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "done", 2024-03-11T21:37:08.807 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:33:17.756051+0000", 2024-03-11T21:37:08.807 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00014092400000000001 2024-03-11T21:37:08.807 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-03-11T21:37:08.807 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-03-11T21:37:08.807 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-03-11T21:37:08.808 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:37:08.808 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:37:08.808 INFO:teuthology.orchestra.run.smithi092.stdout: "description": "osd_op(client.25282.0:76305 9.4 9:24afab6b:::benchmark_data_smithi092_130291_object76304:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e277)", 2024-03-11T21:37:08.808 INFO:teuthology.orchestra.run.smithi092.stdout: "initiated_at": "2024-03-11T21:36:34.066675+0000", 2024-03-11T21:37:08.808 INFO:teuthology.orchestra.run.smithi092.stdout: "age": 34.659346540999998, 2024-03-11T21:37:08.808 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 2.4656751789999998, 2024-03-11T21:37:08.808 INFO:teuthology.orchestra.run.smithi092.stdout: "type_data": { 2024-03-11T21:37:08.808 INFO:teuthology.orchestra.run.smithi092.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-11T21:37:08.808 INFO:teuthology.orchestra.run.smithi092.stdout: "client_info": { 2024-03-11T21:37:08.808 INFO:teuthology.orchestra.run.smithi092.stdout: "client": "client.25282", 2024-03-11T21:37:08.808 INFO:teuthology.orchestra.run.smithi092.stdout: "client_addr": "v1:172.21.15.92:0/2489519821", 2024-03-11T21:37:08.808 INFO:teuthology.orchestra.run.smithi092.stdout: "tid": 76305 2024-03-11T21:37:08.808 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:37:08.808 INFO:teuthology.orchestra.run.smithi092.stdout: "events": [ 2024-03-11T21:37:08.809 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:37:08.809 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "initiated", 2024-03-11T21:37:08.809 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:36:34.066675+0000", 2024-03-11T21:37:08.809 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-03-11T21:37:08.809 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:37:08.809 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:37:08.809 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "header_read", 2024-03-11T21:37:08.809 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:36:34.066675+0000", 2024-03-11T21:37:08.809 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-03-11T21:37:08.809 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:37:08.809 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:37:08.809 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "throttled", 2024-03-11T21:37:08.809 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:36:34.066677+0000", 2024-03-11T21:37:08.809 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 2.148e-06 2024-03-11T21:37:08.809 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:37:08.810 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:37:08.810 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "all_read", 2024-03-11T21:37:08.810 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:36:34.066702+0000", 2024-03-11T21:37:08.810 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 2.4715999999999998e-05 2024-03-11T21:37:08.810 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:37:08.810 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:37:08.810 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "dispatched", 2024-03-11T21:37:08.810 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:36:34.066716+0000", 2024-03-11T21:37:08.810 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.3918e-05 2024-03-11T21:37:08.810 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:37:08.810 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:37:08.810 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "queued_for_pg", 2024-03-11T21:37:08.810 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:36:34.066725+0000", 2024-03-11T21:37:08.810 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 9.1849999999999993e-06 2024-03-11T21:37:08.811 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:37:08.811 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:37:08.811 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-03-11T21:37:08.811 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:36:34.066790+0000", 2024-03-11T21:37:08.811 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 6.4807999999999993e-05 2024-03-11T21:37:08.811 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:37:08.811 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:37:08.811 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "waiting for peered", 2024-03-11T21:37:08.811 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:36:34.066800+0000", 2024-03-11T21:37:08.811 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.0501e-05 2024-03-11T21:37:08.811 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:37:08.811 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:37:08.811 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-03-11T21:37:08.811 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:36:35.049363+0000", 2024-03-11T21:37:08.811 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.98256274399999999 2024-03-11T21:37:08.811 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:37:08.811 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:37:08.812 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "waiting for peered", 2024-03-11T21:37:08.812 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:36:35.049377+0000", 2024-03-11T21:37:08.812 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.4369000000000001e-05 2024-03-11T21:37:08.812 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:37:08.812 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:37:08.812 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-03-11T21:37:08.812 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:36:36.513910+0000", 2024-03-11T21:37:08.812 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.4645325759999999 2024-03-11T21:37:08.812 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:37:08.812 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:37:08.812 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "started", 2024-03-11T21:37:08.812 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:36:36.514088+0000", 2024-03-11T21:37:08.812 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00017830499999999999 2024-03-11T21:37:08.812 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:37:08.813 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:37:08.813 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "waiting for subops from 3,4", 2024-03-11T21:37:08.813 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:36:36.514279+0000", 2024-03-11T21:37:08.813 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.000191032 2024-03-11T21:37:08.813 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:37:08.813 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:37:08.813 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "op_commit", 2024-03-11T21:37:08.813 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:36:36.520031+0000", 2024-03-11T21:37:08.813 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.0057516420000000004 2024-03-11T21:37:08.813 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:37:08.813 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:37:08.813 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "sub_op_commit_rec", 2024-03-11T21:37:08.813 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:36:36.527515+0000", 2024-03-11T21:37:08.813 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.0074835020000000004 2024-03-11T21:37:08.813 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:37:08.813 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:37:08.814 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "sub_op_commit_rec", 2024-03-11T21:37:08.814 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:36:36.528312+0000", 2024-03-11T21:37:08.814 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00079732100000000003 2024-03-11T21:37:08.814 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:37:08.814 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:37:08.814 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "commit_sent", 2024-03-11T21:37:08.814 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:36:36.529068+0000", 2024-03-11T21:37:08.814 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00075650100000000003 2024-03-11T21:37:08.814 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:37:08.814 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:37:08.814 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "done", 2024-03-11T21:37:08.814 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:36:36.532350+0000", 2024-03-11T21:37:08.814 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.0032819110000000002 2024-03-11T21:37:08.814 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-03-11T21:37:08.815 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-03-11T21:37:08.815 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-03-11T21:37:08.815 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:37:08.815 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:37:08.815 INFO:teuthology.orchestra.run.smithi092.stdout: "description": "osd_op(client.25282.0:76293 9.2 9:470137e4:::benchmark_data_smithi092_130291_object76292:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e278)", 2024-03-11T21:37:08.815 INFO:teuthology.orchestra.run.smithi092.stdout: "initiated_at": "2024-03-11T21:36:35.047541+0000", 2024-03-11T21:37:08.815 INFO:teuthology.orchestra.run.smithi092.stdout: "age": 33.678480981999996, 2024-03-11T21:37:08.815 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.466307545, 2024-03-11T21:37:08.815 INFO:teuthology.orchestra.run.smithi092.stdout: "type_data": { 2024-03-11T21:37:08.815 INFO:teuthology.orchestra.run.smithi092.stdout: "flag_point": "reached pg", 2024-03-11T21:37:08.815 INFO:teuthology.orchestra.run.smithi092.stdout: "client_info": { 2024-03-11T21:37:08.815 INFO:teuthology.orchestra.run.smithi092.stdout: "client": "client.25282", 2024-03-11T21:37:08.815 INFO:teuthology.orchestra.run.smithi092.stdout: "client_addr": "v1:172.21.15.92:0/2489519821", 2024-03-11T21:37:08.815 INFO:teuthology.orchestra.run.smithi092.stdout: "tid": 76293 2024-03-11T21:37:08.815 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:37:08.815 INFO:teuthology.orchestra.run.smithi092.stdout: "events": [ 2024-03-11T21:37:08.816 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:37:08.816 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "initiated", 2024-03-11T21:37:08.816 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:36:35.047541+0000", 2024-03-11T21:37:08.816 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-03-11T21:37:08.816 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:37:08.816 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:37:08.816 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "header_read", 2024-03-11T21:37:08.816 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:36:35.047541+0000", 2024-03-11T21:37:08.816 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-03-11T21:37:08.816 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:37:08.816 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:37:08.816 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "throttled", 2024-03-11T21:37:08.816 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:36:35.047544+0000", 2024-03-11T21:37:08.816 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 3.0000000000000001e-06 2024-03-11T21:37:08.816 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:37:08.816 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:37:08.817 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "all_read", 2024-03-11T21:37:08.817 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:36:35.047590+0000", 2024-03-11T21:37:08.817 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 4.6313000000000002e-05 2024-03-11T21:37:08.817 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:37:08.817 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:37:08.817 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "dispatched", 2024-03-11T21:37:08.817 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:36:35.047605+0000", 2024-03-11T21:37:08.817 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.5004000000000001e-05 2024-03-11T21:37:08.817 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:37:08.817 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:37:08.817 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "queued_for_pg", 2024-03-11T21:37:08.817 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:36:35.047625+0000", 2024-03-11T21:37:08.817 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 2.0290000000000001e-05 2024-03-11T21:37:08.817 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:37:08.817 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:37:08.817 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-03-11T21:37:08.818 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:36:35.052040+0000", 2024-03-11T21:37:08.818 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.0044150500000000002 2024-03-11T21:37:08.818 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:37:08.818 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:37:08.818 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "waiting for peered", 2024-03-11T21:37:08.818 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:36:35.052052+0000", 2024-03-11T21:37:08.818 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.1242e-05 2024-03-11T21:37:08.818 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:37:08.818 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:37:08.818 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-03-11T21:37:08.818 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:36:36.513737+0000", 2024-03-11T21:37:08.818 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.4616853000000001 2024-03-11T21:37:08.818 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:37:08.819 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:37:08.819 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "done", 2024-03-11T21:37:08.819 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:36:36.513848+0000", 2024-03-11T21:37:08.819 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.000111346 2024-03-11T21:37:08.819 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-03-11T21:37:08.819 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-03-11T21:37:08.819 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-03-11T21:37:08.819 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:37:08.819 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:37:08.819 INFO:teuthology.orchestra.run.smithi092.stdout: "description": "osd_op(client.25282.0:76303 9.2 9:4fecfdb0:::benchmark_data_smithi092_130291_object76302:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e278)", 2024-03-11T21:37:08.819 INFO:teuthology.orchestra.run.smithi092.stdout: "initiated_at": "2024-03-11T21:36:35.047641+0000", 2024-03-11T21:37:08.819 INFO:teuthology.orchestra.run.smithi092.stdout: "age": 33.678381080000001, 2024-03-11T21:37:08.819 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.486106173, 2024-03-11T21:37:08.819 INFO:teuthology.orchestra.run.smithi092.stdout: "type_data": { 2024-03-11T21:37:08.819 INFO:teuthology.orchestra.run.smithi092.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-11T21:37:08.820 INFO:teuthology.orchestra.run.smithi092.stdout: "client_info": { 2024-03-11T21:37:08.820 INFO:teuthology.orchestra.run.smithi092.stdout: "client": "client.25282", 2024-03-11T21:37:08.820 INFO:teuthology.orchestra.run.smithi092.stdout: "client_addr": "v1:172.21.15.92:0/2489519821", 2024-03-11T21:37:08.820 INFO:teuthology.orchestra.run.smithi092.stdout: "tid": 76303 2024-03-11T21:37:08.820 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:37:08.820 INFO:teuthology.orchestra.run.smithi092.stdout: "events": [ 2024-03-11T21:37:08.820 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:37:08.820 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "initiated", 2024-03-11T21:37:08.820 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:36:35.047641+0000", 2024-03-11T21:37:08.820 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-03-11T21:37:08.820 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:37:08.820 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:37:08.820 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "header_read", 2024-03-11T21:37:08.820 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:36:35.047641+0000", 2024-03-11T21:37:08.820 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-03-11T21:37:08.821 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:37:08.821 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:37:08.821 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "throttled", 2024-03-11T21:37:08.821 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:36:35.047642+0000", 2024-03-11T21:37:08.821 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.02e-06 2024-03-11T21:37:08.821 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:37:08.821 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:37:08.821 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "all_read", 2024-03-11T21:37:08.821 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:36:35.047672+0000", 2024-03-11T21:37:08.821 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 3.0846000000000002e-05 2024-03-11T21:37:08.821 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:37:08.821 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:37:08.821 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "dispatched", 2024-03-11T21:37:08.821 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:36:35.047681+0000", 2024-03-11T21:37:08.822 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 8.123e-06 2024-03-11T21:37:08.822 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:37:08.822 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:37:08.822 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "queued_for_pg", 2024-03-11T21:37:08.822 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:36:35.047687+0000", 2024-03-11T21:37:08.822 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 6.2639999999999997e-06 2024-03-11T21:37:08.822 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:37:08.822 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:37:08.822 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-03-11T21:37:08.822 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:36:35.052079+0000", 2024-03-11T21:37:08.822 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.0043926099999999999 2024-03-11T21:37:08.822 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:37:08.822 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:37:08.822 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "waiting for peered", 2024-03-11T21:37:08.823 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:36:35.052087+0000", 2024-03-11T21:37:08.823 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 7.3440000000000003e-06 2024-03-11T21:37:08.823 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:37:08.823 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:37:08.823 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-03-11T21:37:08.823 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:36:36.516340+0000", 2024-03-11T21:37:08.823 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.4642529550000001 2024-03-11T21:37:08.823 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:37:08.823 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:37:08.823 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "started", 2024-03-11T21:37:08.823 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:36:36.516472+0000", 2024-03-11T21:37:08.823 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.000132256 2024-03-11T21:37:08.823 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:37:08.823 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:37:08.823 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "waiting for subops from 2,3", 2024-03-11T21:37:08.823 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:36:36.516588+0000", 2024-03-11T21:37:08.824 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00011624399999999999 2024-03-11T21:37:08.824 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:37:08.824 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:37:08.824 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "op_commit", 2024-03-11T21:37:08.824 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:36:36.523219+0000", 2024-03-11T21:37:08.824 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.0066303819999999998 2024-03-11T21:37:08.824 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:37:08.824 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:37:08.824 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "sub_op_commit_rec", 2024-03-11T21:37:08.824 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:36:36.527823+0000", 2024-03-11T21:37:08.825 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.0046046430000000003 2024-03-11T21:37:08.825 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:37:08.825 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:37:08.825 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "sub_op_commit_rec", 2024-03-11T21:37:08.825 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:36:36.527903+0000", 2024-03-11T21:37:08.825 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 7.9942000000000001e-05 2024-03-11T21:37:08.825 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:37:08.825 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:37:08.825 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "commit_sent", 2024-03-11T21:37:08.825 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:36:36.528983+0000", 2024-03-11T21:37:08.825 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.0010797770000000001 2024-03-11T21:37:08.825 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:37:08.825 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:37:08.825 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "done", 2024-03-11T21:37:08.825 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:36:36.533747+0000", 2024-03-11T21:37:08.826 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.0047637670000000003 2024-03-11T21:37:08.826 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-03-11T21:37:08.826 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-03-11T21:37:08.826 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-03-11T21:37:08.826 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:37:08.826 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:37:08.826 INFO:teuthology.orchestra.run.smithi092.stdout: "description": "osd_op(client.25282.0:76304 9.2 9:40dc0b9f:::benchmark_data_smithi092_130291_object76303:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e278)", 2024-03-11T21:37:08.826 INFO:teuthology.orchestra.run.smithi092.stdout: "initiated_at": "2024-03-11T21:36:35.047696+0000", 2024-03-11T21:37:08.826 INFO:teuthology.orchestra.run.smithi092.stdout: "age": 33.678325997000002, 2024-03-11T21:37:08.826 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.4932335409999999, 2024-03-11T21:37:08.826 INFO:teuthology.orchestra.run.smithi092.stdout: "type_data": { 2024-03-11T21:37:08.826 INFO:teuthology.orchestra.run.smithi092.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-11T21:37:08.826 INFO:teuthology.orchestra.run.smithi092.stdout: "client_info": { 2024-03-11T21:37:08.826 INFO:teuthology.orchestra.run.smithi092.stdout: "client": "client.25282", 2024-03-11T21:37:08.826 INFO:teuthology.orchestra.run.smithi092.stdout: "client_addr": "v1:172.21.15.92:0/2489519821", 2024-03-11T21:37:08.827 INFO:teuthology.orchestra.run.smithi092.stdout: "tid": 76304 2024-03-11T21:37:08.827 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:37:08.827 INFO:teuthology.orchestra.run.smithi092.stdout: "events": [ 2024-03-11T21:37:08.827 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:37:08.827 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "initiated", 2024-03-11T21:37:08.827 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:36:35.047696+0000", 2024-03-11T21:37:08.827 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-03-11T21:37:08.827 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:37:08.827 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:37:08.827 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "header_read", 2024-03-11T21:37:08.827 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:36:35.047696+0000", 2024-03-11T21:37:08.827 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-03-11T21:37:08.827 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:37:08.827 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:37:08.827 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "throttled", 2024-03-11T21:37:08.828 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:36:35.047696+0000", 2024-03-11T21:37:08.828 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 7.23e-07 2024-03-11T21:37:08.828 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:37:08.828 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:37:08.828 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "all_read", 2024-03-11T21:37:08.828 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:36:35.047723+0000", 2024-03-11T21:37:08.828 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 2.6811000000000001e-05 2024-03-11T21:37:08.828 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:37:08.828 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:37:08.828 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "dispatched", 2024-03-11T21:37:08.828 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:36:35.047731+0000", 2024-03-11T21:37:08.828 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 7.7700000000000001e-06 2024-03-11T21:37:08.828 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:37:08.828 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:37:08.829 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "queued_for_pg", 2024-03-11T21:37:08.829 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:36:35.047737+0000", 2024-03-11T21:37:08.829 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 6.2260000000000002e-06 2024-03-11T21:37:08.829 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:37:08.829 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:37:08.829 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-03-11T21:37:08.829 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:36:35.052114+0000", 2024-03-11T21:37:08.829 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.004376698 2024-03-11T21:37:08.829 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:37:08.829 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:37:08.829 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "waiting for peered", 2024-03-11T21:37:08.829 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:36:35.052121+0000", 2024-03-11T21:37:08.829 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 6.832e-06 2024-03-11T21:37:08.829 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:37:08.830 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:37:08.830 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-03-11T21:37:08.830 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:36:36.519831+0000", 2024-03-11T21:37:08.830 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.467709943 2024-03-11T21:37:08.830 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:37:08.830 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:37:08.830 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "started", 2024-03-11T21:37:08.830 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:36:36.520062+0000", 2024-03-11T21:37:08.830 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00023085900000000001 2024-03-11T21:37:08.830 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:37:08.830 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:37:08.830 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "waiting for subops from 2,3", 2024-03-11T21:37:08.830 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:36:36.520266+0000", 2024-03-11T21:37:08.830 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00020451599999999999 2024-03-11T21:37:08.830 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:37:08.831 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:37:08.831 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "op_commit", 2024-03-11T21:37:08.831 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:36:36.526472+0000", 2024-03-11T21:37:08.831 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.0062056050000000003 2024-03-11T21:37:08.831 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:37:08.831 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:37:08.831 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "sub_op_commit_rec", 2024-03-11T21:37:08.831 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:36:36.534867+0000", 2024-03-11T21:37:08.831 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.0083950359999999998 2024-03-11T21:37:08.831 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:37:08.831 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:37:08.831 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "sub_op_commit_rec", 2024-03-11T21:37:08.831 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:36:36.537251+0000", 2024-03-11T21:37:08.831 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.0023839130000000001 2024-03-11T21:37:08.831 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:37:08.831 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:37:08.831 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "commit_sent", 2024-03-11T21:37:08.832 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:36:36.537314+0000", 2024-03-11T21:37:08.832 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 6.2983000000000003e-05 2024-03-11T21:37:08.832 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:37:08.832 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:37:08.832 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "done", 2024-03-11T21:37:08.832 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:36:36.540929+0000", 2024-03-11T21:37:08.832 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.0036156259999999998 2024-03-11T21:37:08.832 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-03-11T21:37:08.832 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-03-11T21:37:08.832 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-03-11T21:37:08.832 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-03-11T21:37:08.832 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-03-11T21:37:08.832 INFO:teuthology.orchestra.run.smithi092.stdout:} 2024-03-11T21:37:08.833 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:37:08 smithi097 ceph-mon[33317]: pgmap v529: 148 pgs: 5 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+degraded+remapped, 140 active+clean; 5.9 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 639 B/s rd, 31 MiB/s wr, 501 op/s; 35686/291927 objects degraded (12.224%); 4.6 MiB/s, 73 objects/s recovering 2024-03-11T21:37:08.833 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:37:08 smithi097 ceph-mon[33317]: Health check update: Degraded data redundancy: 35686/291927 objects degraded (12.224%), 8 pgs degraded (PG_DEGRADED) 2024-03-11T21:37:08.846 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:37:08.889 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:37:08.890 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:37:08.890 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:37:08.991 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:37:09.033 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:37:09.034 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:37:09.034 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:37:09.135 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:37:09.173 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:37:09.174 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:37:09.174 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:37:09.275 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:37:09.313 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:37:09.314 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:37:09.314 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:37:09.415 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:37:09.450 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:37:09.451 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:37:09.452 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:37:09.533 INFO:tasks.radosbench.radosbench.0.smithi092.stdout: 72 16 102301 102285 88.7698 46.875 0.0204978 0.0112596 2024-03-11T21:37:09.552 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:37:09.587 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:37:09.587 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:37:09.587 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:37:09.688 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:37:09.728 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:37:09.728 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:37:09.729 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:37:09.829 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:37:09.867 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:37:09.867 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:37:09.868 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:37:09.969 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:37:10.010 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:37:10.010 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:37:10.011 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:37:10.111 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:37:10.154 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:37:10.154 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:37:10.155 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:37:10.256 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:37:10.294 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:37:10.295 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:37:10.295 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:37:10.396 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:37:10.431 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:37:10.432 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:37:10.432 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:37:10.533 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:37:10.534 INFO:tasks.radosbench.radosbench.0.smithi092.stdout: 73 16 103107 103091 88.2439 50.375 0.0247116 0.0113265 2024-03-11T21:37:10.569 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:37:10.569 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:37:10.570 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:37:10.670 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:37:10.702 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:37:10.703 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:37:10.703 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:37:10.804 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:37:10.825 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:37:10 smithi097 ceph-mon[33317]: pgmap v530: 148 pgs: 5 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+degraded+remapped, 140 active+clean; 6.2 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 1.2 KiB/s rd, 60 MiB/s wr, 965 op/s; 35392/303054 objects degraded (11.678%); 6.9 MiB/s, 109 objects/s recovering 2024-03-11T21:37:10.841 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:37:10 smithi092 ceph-mon[29572]: pgmap v530: 148 pgs: 5 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+degraded+remapped, 140 active+clean; 6.2 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 1.2 KiB/s rd, 60 MiB/s wr, 965 op/s; 35392/303054 objects degraded (11.678%); 6.9 MiB/s, 109 objects/s recovering 2024-03-11T21:37:10.842 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:37:10 smithi092 ceph-mon[37563]: pgmap v530: 148 pgs: 5 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+degraded+remapped, 140 active+clean; 6.2 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 1.2 KiB/s rd, 60 MiB/s wr, 965 op/s; 35392/303054 objects degraded (11.678%); 6.9 MiB/s, 109 objects/s recovering 2024-03-11T21:37:10.845 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:37:10.846 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:37:10.847 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:37:10.947 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:37:10.988 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:37:10.988 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:37:10.989 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:37:11.089 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:37:11.122 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:37:11.122 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:37:11.123 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:37:11.223 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:37:11.258 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:37:11.259 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:37:11.259 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:37:11.360 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:37:11.395 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:37:11.396 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:37:11.396 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:37:11.497 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:37:11.532 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:37:11.533 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:37:11.533 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:37:11.534 INFO:tasks.radosbench.radosbench.0.smithi092.stdout: 74 16 103948 103932 87.7617 52.5625 0.0102198 0.0113895 2024-03-11T21:37:11.634 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:37:11.683 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:37:11.684 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:37:11.684 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:37:11.785 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:37:11.821 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:37:11.822 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:37:11.822 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:37:11.923 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:37:11.961 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:37:11.962 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:37:11.962 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:37:12.063 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:37:12.103 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:37:12.103 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:37:12.104 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:37:12.204 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:37:12.253 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:37:12.253 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:37:12.253 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:37:12.354 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:37:12.387 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:37:12.388 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:37:12.388 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:37:12.489 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:37:12.523 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:37:12.524 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:37:12.524 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:37:12.533 INFO:tasks.radosbench.radosbench.0.smithi092.stdout: 75 16 104776 104760 87.2816 51.75 0.0149805 0.0114523 2024-03-11T21:37:12.625 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:37:12.663 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:37:12.664 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:37:12.664 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:37:12.765 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:37:12.800 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:37:12.801 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:37:12.801 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:37:12.825 INFO:journalctl@ceph.iscsi.iscsi.a.smithi097.stdout:Mar 11 21:37:12 smithi097 ceph-00db0016-dfed-11ee-95c8-87774f69a715-iscsi-iscsi-a[60943]: debug there is no tcmu-runner data available 2024-03-11T21:37:12.826 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:37:12 smithi097 ceph-mon[33317]: pgmap v531: 148 pgs: 5 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+degraded+remapped, 140 active+clean; 6.2 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 639 B/s rd, 38 MiB/s wr, 610 op/s; 35392/303054 objects degraded (11.678%); 3.5 MiB/s, 55 objects/s recovering 2024-03-11T21:37:12.828 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:37:12 smithi092 ceph-mon[29572]: pgmap v531: 148 pgs: 5 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+degraded+remapped, 140 active+clean; 6.2 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 639 B/s rd, 38 MiB/s wr, 610 op/s; 35392/303054 objects degraded (11.678%); 3.5 MiB/s, 55 objects/s recovering 2024-03-11T21:37:12.829 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:37:12 smithi092 ceph-mon[37563]: pgmap v531: 148 pgs: 5 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+degraded+remapped, 140 active+clean; 6.2 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 639 B/s rd, 38 MiB/s wr, 610 op/s; 35392/303054 objects degraded (11.678%); 3.5 MiB/s, 55 objects/s recovering 2024-03-11T21:37:12.902 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:37:12.941 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:37:12.942 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:37:12.942 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:37:13.043 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:37:13.059 INFO:teuthology.orchestra.run.smithi092.stdout:osd.0: {} 2024-03-11T21:37:13.092 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:37:13.092 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:37:13.093 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:37:13.100 INFO:teuthology.orchestra.run.smithi092.stdout:osd.1: {} 2024-03-11T21:37:13.139 INFO:teuthology.orchestra.run.smithi092.stdout:osd.2: {} 2024-03-11T21:37:13.177 INFO:teuthology.orchestra.run.smithi092.stdout:osd.3: {} 2024-03-11T21:37:13.194 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:37:13.209 INFO:teuthology.orchestra.run.smithi092.stdout:osd.4: {} 2024-03-11T21:37:13.239 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:37:13.239 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:37:13.240 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:37:13.252 INFO:teuthology.orchestra.run.smithi092.stdout:osd.5: {} 2024-03-11T21:37:13.289 INFO:teuthology.orchestra.run.smithi092.stdout:osd.6: {} 2024-03-11T21:37:13.331 INFO:teuthology.orchestra.run.smithi092.stdout:osd.7: {} 2024-03-11T21:37:13.336 INFO:teuthology.orchestra.run.smithi092.stderr:osd.0: osd_enable_op_tracker = '' 2024-03-11T21:37:13.336 INFO:teuthology.orchestra.run.smithi092.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-11T21:37:13.336 INFO:teuthology.orchestra.run.smithi092.stderr:osd.2: osd_enable_op_tracker = '' 2024-03-11T21:37:13.337 INFO:teuthology.orchestra.run.smithi092.stderr:osd.3: osd_enable_op_tracker = '' 2024-03-11T21:37:13.337 INFO:teuthology.orchestra.run.smithi092.stderr:osd.4: osd_enable_op_tracker = '' 2024-03-11T21:37:13.337 INFO:teuthology.orchestra.run.smithi092.stderr:osd.5: osd_enable_op_tracker = '' 2024-03-11T21:37:13.337 INFO:teuthology.orchestra.run.smithi092.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-11T21:37:13.337 INFO:teuthology.orchestra.run.smithi092.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-11T21:37:13.341 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:37:13.373 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:37:13.373 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:37:13.373 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:37:13.463 DEBUG:teuthology.orchestra.run.smithi092:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid 00db0016-dfed-11ee-95c8-87774f69a715 -- ceph daemon osd.2 dump_ops_in_flight 2024-03-11T21:37:13.474 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:37:13.509 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:37:13.510 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:37:13.510 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:37:13.533 INFO:tasks.radosbench.radosbench.0.smithi092.stdout: 76 16 105559 105543 86.777 48.9375 0.0220951 0.0115177 2024-03-11T21:37:13.610 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:37:13 smithi092 ceph-mon[29572]: from='client.24469 v1:172.21.15.97:0/404208310' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:37:13.610 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:37:13 smithi092 ceph-mon[29572]: from='client.? v1:172.21.15.92:0/518945249' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-11T21:37:13.610 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:37:13 smithi092 ceph-mon[29572]: Health check update: Degraded data redundancy: 35392/303054 objects degraded (11.678%), 8 pgs degraded (PG_DEGRADED) 2024-03-11T21:37:13.611 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:37:13 smithi092 ceph-mon[37563]: from='client.24469 v1:172.21.15.97:0/404208310' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:37:13.611 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:37:13 smithi092 ceph-mon[37563]: from='client.? v1:172.21.15.92:0/518945249' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-11T21:37:13.611 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:37:13 smithi092 ceph-mon[37563]: Health check update: Degraded data redundancy: 35392/303054 objects degraded (11.678%), 8 pgs degraded (PG_DEGRADED) 2024-03-11T21:37:13.611 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:37:13.642 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:37:13.643 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:37:13.643 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:37:13.744 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:37:13.786 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:37:13.786 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:37:13.786 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:37:13.825 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:37:13 smithi097 ceph-mon[33317]: from='client.24469 v1:172.21.15.97:0/404208310' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:37:13.826 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:37:13 smithi097 ceph-mon[33317]: from='client.? v1:172.21.15.92:0/518945249' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-11T21:37:13.826 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:37:13 smithi097 ceph-mon[33317]: Health check update: Degraded data redundancy: 35392/303054 objects degraded (11.678%), 8 pgs degraded (PG_DEGRADED) 2024-03-11T21:37:13.887 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:37:13.971 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:37:13.971 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:37:13.972 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:37:14.072 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:37:14.116 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:37:14.117 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:37:14.117 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:37:14.218 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:37:14.237 INFO:teuthology.orchestra.run.smithi092.stderr:Inferring config /var/lib/ceph/00db0016-dfed-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T21:37:14.250 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:37:14.251 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:37:14.251 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:37:14.352 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:37:14.404 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:37:14.404 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:37:14.405 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:37:14.505 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:37:14.534 INFO:tasks.radosbench.radosbench.0.smithi092.stdout: 77 16 106321 106305 86.2686 47.625 0.013873 0.0115859 2024-03-11T21:37:14.536 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:37:14.536 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:37:14.537 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:37:14.599 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:37:14 smithi092 ceph-mon[29572]: pgmap v532: 148 pgs: 5 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+degraded+remapped, 140 active+clean; 6.3 GiB data, 20 GiB used, 696 GiB / 715 GiB avail; 1.0 KiB/s rd, 44 MiB/s wr, 710 op/s; 34936/311910 objects degraded (11.201%); 5.8 MiB/s, 92 objects/s recovering 2024-03-11T21:37:14.600 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:37:14 smithi092 ceph-mon[37563]: pgmap v532: 148 pgs: 5 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+degraded+remapped, 140 active+clean; 6.3 GiB data, 20 GiB used, 696 GiB / 715 GiB avail; 1.0 KiB/s rd, 44 MiB/s wr, 710 op/s; 34936/311910 objects degraded (11.201%); 5.8 MiB/s, 92 objects/s recovering 2024-03-11T21:37:14.637 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:37:14.673 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:37:14.673 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:37:14.674 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:37:14.774 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:37:14.808 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:37:14.809 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:37:14.809 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:37:14.825 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:37:14 smithi097 ceph-mon[33317]: pgmap v532: 148 pgs: 5 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+degraded+remapped, 140 active+clean; 6.3 GiB data, 20 GiB used, 696 GiB / 715 GiB avail; 1.0 KiB/s rd, 44 MiB/s wr, 710 op/s; 34936/311910 objects degraded (11.201%); 5.8 MiB/s, 92 objects/s recovering 2024-03-11T21:37:14.910 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:37:14.961 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:37:14.962 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:37:14.962 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:37:15.063 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:37:15.108 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:37:15.109 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:37:15.109 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:37:15.210 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:37:15.247 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:37:15.247 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:37:15.248 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:37:15.349 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:37:15.376 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:37:15.377 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:37:15.377 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:37:15.478 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:37:15.514 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:37:15.515 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:37:15.515 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:37:15.534 INFO:tasks.radosbench.radosbench.0.smithi092.stdout: 78 16 107057 107041 85.7523 46 0.00849766 0.0116556 2024-03-11T21:37:15.561 INFO:teuthology.orchestra.run.smithi092.stderr:reweighted osd.2 to 1 (10000) 2024-03-11T21:37:15.617 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:37:15.654 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:37:15.654 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:37:15.655 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:37:15.756 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:37:15.795 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:37:15.796 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:37:15.796 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:37:15.825 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:37:15 smithi097 ceph-mon[33317]: from='client.? v1:172.21.15.92:0/1742188243' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]: dispatch 2024-03-11T21:37:15.826 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:37:15 smithi097 ceph-mon[33317]: from='client.25366 ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]: dispatch 2024-03-11T21:37:15.860 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:37:15 smithi092 ceph-mon[29572]: from='client.? v1:172.21.15.92:0/1742188243' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]: dispatch 2024-03-11T21:37:15.860 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:37:15 smithi092 ceph-mon[29572]: from='client.25366 ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]: dispatch 2024-03-11T21:37:15.860 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:37:15 smithi092 ceph-mon[37563]: from='client.? v1:172.21.15.92:0/1742188243' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]: dispatch 2024-03-11T21:37:15.860 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:37:15 smithi092 ceph-mon[37563]: from='client.25366 ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]: dispatch 2024-03-11T21:37:15.898 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:37:15.941 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:37:15.942 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:37:15.942 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:37:16.043 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:37:16.073 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:37:16.074 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:37:16.074 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:37:16.175 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:37:16.213 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:37:16.213 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:37:16.214 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:37:16.315 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:37:16.350 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:37:16.351 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:37:16.351 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:37:16.452 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:37:16.486 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:37:16.486 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:37:16.487 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:37:16.534 INFO:tasks.radosbench.radosbench.0.smithi092.stdout: 79 16 107827 107811 85.276 48.125 0.0122452 0.0117217 2024-03-11T21:37:16.580 INFO:teuthology.orchestra.run.smithi092.stderr:noscrub is set 2024-03-11T21:37:16.588 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:37:16.620 DEBUG:teuthology.orchestra.run.smithi092:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid 00db0016-dfed-11ee-95c8-87774f69a715 -- ceph osd reweight 3 1 2024-03-11T21:37:16.646 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:37:16.646 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:37:16.647 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:37:16.748 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:37:16.784 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:37:16.785 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:37:16.785 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:37:16.826 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:37:16 smithi097 ceph-mon[33317]: pgmap v533: 148 pgs: 5 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+degraded+remapped, 140 active+clean; 6.4 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 1023 B/s rd, 50 MiB/s wr, 795 op/s; 34936/315042 objects degraded (11.089%); 5.6 MiB/s, 89 objects/s recovering 2024-03-11T21:37:16.826 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:37:16 smithi097 ceph-mon[33317]: from='client.25366 ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]': finished 2024-03-11T21:37:16.826 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:37:16 smithi097 ceph-mon[33317]: osdmap e291: 8 total, 8 up, 8 in 2024-03-11T21:37:16.826 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:37:16 smithi097 ceph-mon[33317]: from='client.? v1:172.21.15.92:0/2031480128' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-11T21:37:16.826 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:37:16 smithi097 ceph-mon[33317]: from='client.35357 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-11T21:37:16.860 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:37:16 smithi092 ceph-mon[29572]: pgmap v533: 148 pgs: 5 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+degraded+remapped, 140 active+clean; 6.4 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 1023 B/s rd, 50 MiB/s wr, 795 op/s; 34936/315042 objects degraded (11.089%); 5.6 MiB/s, 89 objects/s recovering 2024-03-11T21:37:16.860 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:37:16 smithi092 ceph-mon[29572]: from='client.25366 ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]': finished 2024-03-11T21:37:16.860 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:37:16 smithi092 ceph-mon[29572]: osdmap e291: 8 total, 8 up, 8 in 2024-03-11T21:37:16.860 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:37:16 smithi092 ceph-mon[29572]: from='client.? v1:172.21.15.92:0/2031480128' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-11T21:37:16.860 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:37:16 smithi092 ceph-mon[29572]: from='client.35357 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-11T21:37:16.860 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:37:16 smithi092 ceph-mon[37563]: pgmap v533: 148 pgs: 5 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+degraded+remapped, 140 active+clean; 6.4 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 1023 B/s rd, 50 MiB/s wr, 795 op/s; 34936/315042 objects degraded (11.089%); 5.6 MiB/s, 89 objects/s recovering 2024-03-11T21:37:16.860 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:37:16 smithi092 ceph-mon[37563]: from='client.25366 ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]': finished 2024-03-11T21:37:16.860 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:37:16 smithi092 ceph-mon[37563]: osdmap e291: 8 total, 8 up, 8 in 2024-03-11T21:37:16.861 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:37:16 smithi092 ceph-mon[37563]: from='client.? v1:172.21.15.92:0/2031480128' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-11T21:37:16.861 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:37:16 smithi092 ceph-mon[37563]: from='client.35357 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-11T21:37:16.886 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:37:16.939 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:37:16.939 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:37:16.940 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:37:17.041 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:37:17.072 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:37:17.073 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:37:17.073 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:37:17.076 DEBUG:teuthology.orchestra.run.smithi092:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid 00db0016-dfed-11ee-95c8-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-03-11T21:37:17.174 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:37:17.228 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:37:17.228 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:37:17.229 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:37:17.330 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:37:17.359 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:37:17.359 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:37:17.359 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:37:17.460 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:37:17.507 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:37:17.508 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:37:17.508 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:37:17.534 INFO:tasks.radosbench.radosbench.0.smithi092.stdout:2024-03-11T21:37:17.553071+0000 min lat: 0.00114249 max lat: 2.50819 avg lat: 0.011791 2024-03-11T21:37:17.534 INFO:tasks.radosbench.radosbench.0.smithi092.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-03-11T21:37:17.534 INFO:tasks.radosbench.radosbench.0.smithi092.stdout: 80 16 108546 108530 84.7718 44.9375 0.00872049 0.011791 2024-03-11T21:37:17.609 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:37:17.643 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:37:17.644 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:37:17.644 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:37:17.745 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:37:17.783 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:37:17.783 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:37:17.784 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:37:17.826 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:37:17 smithi097 ceph-mon[33317]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-03-11T21:37:17.826 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:37:17 smithi097 ceph-mon[33317]: from='client.35357 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-03-11T21:37:17.826 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:37:17 smithi097 ceph-mon[33317]: osdmap e292: 8 total, 8 up, 8 in 2024-03-11T21:37:17.860 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:37:17 smithi092 ceph-mon[29572]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-03-11T21:37:17.860 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:37:17 smithi092 ceph-mon[29572]: from='client.35357 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-03-11T21:37:17.860 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:37:17 smithi092 ceph-mon[29572]: osdmap e292: 8 total, 8 up, 8 in 2024-03-11T21:37:17.860 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:37:17 smithi092 ceph-mon[37563]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-03-11T21:37:17.860 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:37:17 smithi092 ceph-mon[37563]: from='client.35357 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-03-11T21:37:17.860 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:37:17 smithi092 ceph-mon[37563]: osdmap e292: 8 total, 8 up, 8 in 2024-03-11T21:37:17.885 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:37:17.917 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:37:17.918 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:37:17.918 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:37:18.019 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:37:18.055 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:37:18.055 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:37:18.056 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:37:18.157 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:37:18.194 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:37:18.195 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:37:18.195 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:37:18.291 INFO:journalctl@ceph.mgr.y.smithi092.stdout:Mar 11 21:37:17 smithi092 ceph-00db0016-dfed-11ee-95c8-87774f69a715-mgr-y[29801]: ::ffff:172.21.15.97 - - [11/Mar/2024:21:37:17] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-11T21:37:18.296 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:37:18.342 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:37:18.342 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:37:18.343 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:37:18.443 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:37:18.479 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:37:18.480 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:37:18.480 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:37:18.534 INFO:tasks.radosbench.radosbench.0.smithi092.stdout: 81 16 109359 109343 84.3525 50.8125 0.0155655 0.01185 2024-03-11T21:37:18.581 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:37:18.619 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:37:18.619 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:37:18.620 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:37:18.720 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:37:18.759 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:37:18.759 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:37:18.760 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:37:18.826 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:37:18 smithi097 ceph-mon[33317]: pgmap v536: 148 pgs: 5 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+degraded+remapped, 140 active+clean; 6.4 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 639 B/s rd, 33 MiB/s wr, 527 op/s; 34788/315708 objects degraded (11.019%); 4.7 MiB/s, 75 objects/s recovering 2024-03-11T21:37:18.826 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:37:18 smithi097 ceph-mon[33317]: Health check update: Degraded data redundancy: 34788/315708 objects degraded (11.019%), 8 pgs degraded (PG_DEGRADED) 2024-03-11T21:37:18.860 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:37:18 smithi092 ceph-mon[29572]: pgmap v536: 148 pgs: 5 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+degraded+remapped, 140 active+clean; 6.4 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 639 B/s rd, 33 MiB/s wr, 527 op/s; 34788/315708 objects degraded (11.019%); 4.7 MiB/s, 75 objects/s recovering 2024-03-11T21:37:18.860 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:37:18 smithi092 ceph-mon[29572]: Health check update: Degraded data redundancy: 34788/315708 objects degraded (11.019%), 8 pgs degraded (PG_DEGRADED) 2024-03-11T21:37:18.860 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:37:18 smithi092 ceph-mon[37563]: pgmap v536: 148 pgs: 5 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+degraded+remapped, 140 active+clean; 6.4 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 639 B/s rd, 33 MiB/s wr, 527 op/s; 34788/315708 objects degraded (11.019%); 4.7 MiB/s, 75 objects/s recovering 2024-03-11T21:37:18.860 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:37:18 smithi092 ceph-mon[37563]: Health check update: Degraded data redundancy: 34788/315708 objects degraded (11.019%), 8 pgs degraded (PG_DEGRADED) 2024-03-11T21:37:18.861 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:37:18.894 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:37:18.895 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:37:18.895 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:37:18.996 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:37:19.025 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:37:19.025 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:37:19.026 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:37:19.127 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:37:19.162 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:37:19.162 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:37:19.163 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:37:19.263 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:37:19.293 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:37:19.294 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:37:19.294 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:37:19.395 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:37:19.426 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:37:19.427 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:37:19.427 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:37:19.528 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:37:19.534 INFO:tasks.radosbench.radosbench.0.smithi092.stdout: 82 16 110148 110132 83.9246 49.3125 0.0315588 0.011908 2024-03-11T21:37:19.599 INFO:teuthology.orchestra.run.smithi092.stderr:Inferring config /var/lib/ceph/00db0016-dfed-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T21:37:19.601 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:37:19.601 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:37:19.602 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:37:19.612 INFO:teuthology.orchestra.run.smithi092.stderr:Inferring config /var/lib/ceph/00db0016-dfed-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T21:37:19.631 INFO:teuthology.orchestra.run.smithi092.stdout:ERROR: (22) Invalid argument 2024-03-11T21:37:19.631 INFO:teuthology.orchestra.run.smithi092.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-11T21:37:19.703 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:37:19.736 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:37:19.736 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:37:19.737 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:37:19.837 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:37:19.872 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:37:19.873 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:37:19.873 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:37:19.974 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:37:20.018 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:37:20.019 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:37:20.019 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:37:20.120 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:37:20.149 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:37:20.150 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:37:20.150 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:37:20.251 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:37:20.287 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:37:20.288 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:37:20.289 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:37:20.389 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:37:20.434 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:37:20.435 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:37:20.435 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:37:20.535 INFO:tasks.radosbench.radosbench.0.smithi092.stdout: 83 16 110955 110939 83.5211 50.4375 0.0349998 0.0119676 2024-03-11T21:37:20.537 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:37:20.570 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:37:20.571 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:37:20.571 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:37:20.672 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:37:20.708 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:37:20.708 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:37:20.709 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:37:20.809 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:37:20.826 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:37:20 smithi097 ceph-mon[33317]: pgmap v537: 148 pgs: 5 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+degraded+remapped, 140 active+clean; 6.6 GiB data, 21 GiB used, 695 GiB / 715 GiB avail; 1.2 KiB/s rd, 61 MiB/s wr, 973 op/s; 34486/326406 objects degraded (10.565%); 7.1 MiB/s, 113 objects/s recovering 2024-03-11T21:37:20.848 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:37:20.849 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:37:20.850 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:37:20.860 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:37:20 smithi092 ceph-mon[29572]: pgmap v537: 148 pgs: 5 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+degraded+remapped, 140 active+clean; 6.6 GiB data, 21 GiB used, 695 GiB / 715 GiB avail; 1.2 KiB/s rd, 61 MiB/s wr, 973 op/s; 34486/326406 objects degraded (10.565%); 7.1 MiB/s, 113 objects/s recovering 2024-03-11T21:37:20.861 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:37:20 smithi092 ceph-mon[37563]: pgmap v537: 148 pgs: 5 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+degraded+remapped, 140 active+clean; 6.6 GiB data, 21 GiB used, 695 GiB / 715 GiB avail; 1.2 KiB/s rd, 61 MiB/s wr, 973 op/s; 34486/326406 objects degraded (10.565%); 7.1 MiB/s, 113 objects/s recovering 2024-03-11T21:37:20.950 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:37:20.985 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:37:20.985 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:37:20.986 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:37:21.034 DEBUG:teuthology.orchestra.run.smithi092:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid 00db0016-dfed-11ee-95c8-87774f69a715 -- ceph osd set nodeep-scrub 2024-03-11T21:37:21.087 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:37:21.125 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:37:21.126 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:37:21.126 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:37:21.227 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:37:21.260 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:37:21.261 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:37:21.261 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:37:21.362 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:37:21.397 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:37:21.398 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:37:21.398 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:37:21.474 DEBUG:teuthology.orchestra.run.smithi092:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid 00db0016-dfed-11ee-95c8-87774f69a715 -- ceph daemon osd.2 dump_blocked_ops 2024-03-11T21:37:21.500 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:37:21.534 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:37:21.535 INFO:tasks.radosbench.radosbench.0.smithi092.stdout: 84 16 111761 111745 83.1265 50.375 0.00910076 0.0120235 2024-03-11T21:37:21.535 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:37:21.536 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:37:21.637 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:37:21.698 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:37:21.699 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:37:21.699 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:37:21.734 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:37:21 smithi092 ceph-mon[29572]: purged_snaps scrub starts 2024-03-11T21:37:21.735 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:37:21 smithi092 ceph-mon[29572]: purged_snaps scrub ok 2024-03-11T21:37:21.735 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:37:21 smithi092 ceph-mon[37563]: purged_snaps scrub starts 2024-03-11T21:37:21.735 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:37:21 smithi092 ceph-mon[37563]: purged_snaps scrub ok 2024-03-11T21:37:21.801 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:37:21.843 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:37:21.844 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:37:21.845 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:37:21.946 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:37:21.987 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:37:21.988 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:37:21.988 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:37:22.076 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:37:21 smithi097 ceph-mon[33317]: purged_snaps scrub starts 2024-03-11T21:37:22.076 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:37:21 smithi097 ceph-mon[33317]: purged_snaps scrub ok 2024-03-11T21:37:22.088 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:37:22.124 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:37:22.125 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:37:22.126 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:37:22.226 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:37:22.265 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:37:22.266 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:37:22.266 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:37:22.367 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:37:22.412 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:37:22.413 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:37:22.413 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:37:22.514 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:37:22.535 INFO:tasks.radosbench.radosbench.0.smithi092.stdout: 85 16 112588 112572 82.7567 51.6875 0.0348486 0.0120783 2024-03-11T21:37:22.559 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:37:22.560 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:37:22.560 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:37:22.661 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:37:22.705 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:37:22.705 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:37:22.706 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:37:22.806 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:37:22.847 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:37:22.848 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:37:22.848 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:37:22.860 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:37:22 smithi092 ceph-mon[29572]: pgmap v538: 148 pgs: 5 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+degraded+remapped, 140 active+clean; 6.6 GiB data, 21 GiB used, 695 GiB / 715 GiB avail; 639 B/s rd, 38 MiB/s wr, 604 op/s; 34486/326406 objects degraded (10.565%); 3.5 MiB/s, 56 objects/s recovering 2024-03-11T21:37:22.860 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:37:22 smithi092 ceph-mon[29572]: osdmap e293: 8 total, 8 up, 8 in 2024-03-11T21:37:22.860 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:37:22 smithi092 ceph-mon[29572]: from='mgr.14607 ' entity='mgr.y' 2024-03-11T21:37:22.860 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:37:22 smithi092 ceph-mon[29572]: from='mgr.14607 v1:172.21.15.92:0/1702717597' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T21:37:22.860 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:37:22 smithi092 ceph-mon[37563]: pgmap v538: 148 pgs: 5 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+degraded+remapped, 140 active+clean; 6.6 GiB data, 21 GiB used, 695 GiB / 715 GiB avail; 639 B/s rd, 38 MiB/s wr, 604 op/s; 34486/326406 objects degraded (10.565%); 3.5 MiB/s, 56 objects/s recovering 2024-03-11T21:37:22.860 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:37:22 smithi092 ceph-mon[37563]: osdmap e293: 8 total, 8 up, 8 in 2024-03-11T21:37:22.860 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:37:22 smithi092 ceph-mon[37563]: from='mgr.14607 ' entity='mgr.y' 2024-03-11T21:37:22.860 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:37:22 smithi092 ceph-mon[37563]: from='mgr.14607 v1:172.21.15.92:0/1702717597' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T21:37:22.948 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:37:22.977 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:37:22.977 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:37:22.978 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:37:23.076 INFO:journalctl@ceph.iscsi.iscsi.a.smithi097.stdout:Mar 11 21:37:22 smithi097 ceph-00db0016-dfed-11ee-95c8-87774f69a715-iscsi-iscsi-a[60943]: debug there is no tcmu-runner data available 2024-03-11T21:37:23.076 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:37:22 smithi097 ceph-mon[33317]: pgmap v538: 148 pgs: 5 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+degraded+remapped, 140 active+clean; 6.6 GiB data, 21 GiB used, 695 GiB / 715 GiB avail; 639 B/s rd, 38 MiB/s wr, 604 op/s; 34486/326406 objects degraded (10.565%); 3.5 MiB/s, 56 objects/s recovering 2024-03-11T21:37:23.076 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:37:22 smithi097 ceph-mon[33317]: osdmap e293: 8 total, 8 up, 8 in 2024-03-11T21:37:23.076 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:37:22 smithi097 ceph-mon[33317]: from='mgr.14607 ' entity='mgr.y' 2024-03-11T21:37:23.077 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:37:22 smithi097 ceph-mon[33317]: from='mgr.14607 v1:172.21.15.92:0/1702717597' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T21:37:23.079 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:37:23.110 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:37:23.110 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:37:23.110 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:37:23.211 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:37:23.252 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:37:23.252 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:37:23.253 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:37:23.354 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:37:23.384 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:37:23.385 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:37:23.385 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:37:23.486 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:37:23.525 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:37:23.526 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:37:23.526 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:37:23.535 INFO:tasks.radosbench.radosbench.0.smithi092.stdout: 86 16 113281 113265 82.298 43.3125 0.022589 0.012144 2024-03-11T21:37:23.627 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:37:23.665 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:37:23.665 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:37:23.666 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:37:23.766 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:37:23.808 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:37:23.808 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:37:23.808 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:37:23.860 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:37:23 smithi092 ceph-mon[29572]: from='client.24469 v1:172.21.15.97:0/404208310' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:37:23.860 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:37:23 smithi092 ceph-mon[29572]: Health check update: Degraded data redundancy: 34486/326406 objects degraded (10.565%), 8 pgs degraded (PG_DEGRADED) 2024-03-11T21:37:23.860 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:37:23 smithi092 ceph-mon[29572]: osdmap e294: 8 total, 8 up, 8 in 2024-03-11T21:37:23.860 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:37:23 smithi092 ceph-mon[37563]: from='client.24469 v1:172.21.15.97:0/404208310' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:37:23.861 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:37:23 smithi092 ceph-mon[37563]: Health check update: Degraded data redundancy: 34486/326406 objects degraded (10.565%), 8 pgs degraded (PG_DEGRADED) 2024-03-11T21:37:23.861 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:37:23 smithi092 ceph-mon[37563]: osdmap e294: 8 total, 8 up, 8 in 2024-03-11T21:37:23.910 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:37:23.945 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:37:23.945 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:37:23.946 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:37:24.046 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:37:24.076 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:37:23 smithi097 ceph-mon[33317]: from='client.24469 v1:172.21.15.97:0/404208310' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:37:24.076 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:37:23 smithi097 ceph-mon[33317]: Health check update: Degraded data redundancy: 34486/326406 objects degraded (10.565%), 8 pgs degraded (PG_DEGRADED) 2024-03-11T21:37:24.076 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:37:23 smithi097 ceph-mon[33317]: osdmap e294: 8 total, 8 up, 8 in 2024-03-11T21:37:24.083 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:37:24.083 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:37:24.084 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:37:24.185 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:37:24.227 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:37:24.227 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:37:24.228 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:37:24.329 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:37:24.360 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:37:24.361 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:37:24.361 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:37:24.462 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:37:24.498 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:37:24.499 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:37:24.500 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:37:24.535 INFO:tasks.radosbench.radosbench.0.smithi092.stdout: 87 16 114109 114093 81.9469 51.75 0.00990922 0.0121977 2024-03-11T21:37:24.601 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:37:24.643 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:37:24.644 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:37:24.644 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:37:24.745 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:37:24.782 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:37:24.783 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:37:24.784 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:37:24.860 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:37:24 smithi092 ceph-mon[29572]: pgmap v540: 148 pgs: 5 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+degraded+remapped, 140 active+clean; 6.8 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 1.3 KiB/s rd, 55 MiB/s wr, 880 op/s; 33999/335280 objects degraded (10.140%); 7.6 MiB/s, 122 objects/s recovering 2024-03-11T21:37:24.860 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:37:24 smithi092 ceph-mon[37563]: pgmap v540: 148 pgs: 5 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+degraded+remapped, 140 active+clean; 6.8 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 1.3 KiB/s rd, 55 MiB/s wr, 880 op/s; 33999/335280 objects degraded (10.140%); 7.6 MiB/s, 122 objects/s recovering 2024-03-11T21:37:24.885 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:37:24.920 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:37:24.921 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:37:24.921 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:37:25.022 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:37:25.058 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:37:25.059 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:37:25.059 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:37:25.076 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:37:24 smithi097 ceph-mon[33317]: pgmap v540: 148 pgs: 5 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+degraded+remapped, 140 active+clean; 6.8 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 1.3 KiB/s rd, 55 MiB/s wr, 880 op/s; 33999/335280 objects degraded (10.140%); 7.6 MiB/s, 122 objects/s recovering 2024-03-11T21:37:25.160 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:37:25.190 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:37:25.191 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:37:25.191 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:37:25.292 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:37:25.329 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:37:25.330 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:37:25.330 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:37:25.431 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:37:25.476 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:37:25.477 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:37:25.478 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:37:25.535 INFO:tasks.radosbench.radosbench.0.smithi092.stdout: 88 16 114876 114860 81.5604 47.9375 0.00752818 0.0122553 2024-03-11T21:37:25.579 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:37:25.627 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:37:25.628 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:37:25.628 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:37:25.729 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:37:25.759 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:37:25.760 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:37:25.760 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:37:25.861 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:37:25.901 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:37:25.901 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:37:25.902 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:37:26.003 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:37:26.036 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:37:26.037 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:37:26.037 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:37:26.049 INFO:teuthology.orchestra.run.smithi092.stderr:Inferring config /var/lib/ceph/00db0016-dfed-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T21:37:26.138 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:37:26.177 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:37:26.178 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:37:26.178 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:37:26.279 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:37:26.306 INFO:teuthology.orchestra.run.smithi092.stderr:Inferring config /var/lib/ceph/00db0016-dfed-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T21:37:26.333 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:37:26.334 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:37:26.334 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:37:26.398 INFO:teuthology.orchestra.run.smithi092.stdout:osd.0: {} 2024-03-11T21:37:26.436 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:37:26.440 INFO:teuthology.orchestra.run.smithi092.stdout:osd.1: {} 2024-03-11T21:37:26.474 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:37:26.475 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:37:26.475 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:37:26.481 INFO:teuthology.orchestra.run.smithi092.stdout:osd.2: {} 2024-03-11T21:37:26.522 INFO:teuthology.orchestra.run.smithi092.stdout:osd.3: {} 2024-03-11T21:37:26.535 INFO:tasks.radosbench.radosbench.0.smithi092.stdout: 89 16 115586 115570 81.1426 44.375 0.0109922 0.0123187 2024-03-11T21:37:26.562 INFO:teuthology.orchestra.run.smithi092.stdout:osd.4: {} 2024-03-11T21:37:26.577 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:37:26.600 INFO:teuthology.orchestra.run.smithi092.stdout:osd.5: {} 2024-03-11T21:37:26.614 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:37:26.614 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:37:26.615 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:37:26.636 INFO:teuthology.orchestra.run.smithi092.stderr:reweighted osd.3 to 1 (10000) 2024-03-11T21:37:26.646 INFO:teuthology.orchestra.run.smithi092.stdout:osd.6: {} 2024-03-11T21:37:26.699 INFO:teuthology.orchestra.run.smithi092.stdout:osd.7: {} 2024-03-11T21:37:26.705 INFO:teuthology.orchestra.run.smithi092.stderr:osd.0: osd_enable_op_tracker = '' 2024-03-11T21:37:26.705 INFO:teuthology.orchestra.run.smithi092.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-11T21:37:26.705 INFO:teuthology.orchestra.run.smithi092.stderr:osd.2: osd_enable_op_tracker = '' 2024-03-11T21:37:26.705 INFO:teuthology.orchestra.run.smithi092.stderr:osd.3: osd_enable_op_tracker = '' 2024-03-11T21:37:26.705 INFO:teuthology.orchestra.run.smithi092.stderr:osd.4: osd_enable_op_tracker = '' 2024-03-11T21:37:26.705 INFO:teuthology.orchestra.run.smithi092.stderr:osd.5: osd_enable_op_tracker = '' 2024-03-11T21:37:26.705 INFO:teuthology.orchestra.run.smithi092.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-11T21:37:26.705 INFO:teuthology.orchestra.run.smithi092.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-11T21:37:26.716 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:37:26.744 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:37:26.745 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:37:26.745 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:37:26.846 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:37:26.860 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:37:26 smithi092 ceph-mon[29572]: pgmap v542: 148 pgs: 5 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+degraded+remapped, 140 active+clean; 6.9 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 1.2 KiB/s rd, 59 MiB/s wr, 946 op/s; 33999/338394 objects degraded (10.047%); 6.2 MiB/s, 98 objects/s recovering 2024-03-11T21:37:26.860 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:37:26 smithi092 ceph-mon[29572]: from='client.? v1:172.21.15.92:0/2335977967' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]: dispatch 2024-03-11T21:37:26.860 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:37:26 smithi092 ceph-mon[29572]: from='client.35363 ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]: dispatch 2024-03-11T21:37:26.860 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:37:26 smithi092 ceph-mon[29572]: from='client.? v1:172.21.15.92:0/1271534889' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-11T21:37:26.860 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:37:26 smithi092 ceph-mon[37563]: pgmap v542: 148 pgs: 5 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+degraded+remapped, 140 active+clean; 6.9 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 1.2 KiB/s rd, 59 MiB/s wr, 946 op/s; 33999/338394 objects degraded (10.047%); 6.2 MiB/s, 98 objects/s recovering 2024-03-11T21:37:26.861 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:37:26 smithi092 ceph-mon[37563]: from='client.? v1:172.21.15.92:0/2335977967' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]: dispatch 2024-03-11T21:37:26.861 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:37:26 smithi092 ceph-mon[37563]: from='client.35363 ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]: dispatch 2024-03-11T21:37:26.861 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:37:26 smithi092 ceph-mon[37563]: from='client.? v1:172.21.15.92:0/1271534889' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-11T21:37:26.889 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:37:26.889 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:37:26.890 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:37:26.991 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:37:27.028 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:37:27.029 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:37:27.029 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:37:27.076 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:37:26 smithi097 ceph-mon[33317]: pgmap v542: 148 pgs: 5 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+degraded+remapped, 140 active+clean; 6.9 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 1.2 KiB/s rd, 59 MiB/s wr, 946 op/s; 33999/338394 objects degraded (10.047%); 6.2 MiB/s, 98 objects/s recovering 2024-03-11T21:37:27.076 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:37:26 smithi097 ceph-mon[33317]: from='client.? v1:172.21.15.92:0/2335977967' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]: dispatch 2024-03-11T21:37:27.076 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:37:26 smithi097 ceph-mon[33317]: from='client.35363 ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]: dispatch 2024-03-11T21:37:27.076 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:37:26 smithi097 ceph-mon[33317]: from='client.? v1:172.21.15.92:0/1271534889' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-11T21:37:27.130 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:37:27.170 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:37:27.171 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:37:27.171 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:37:27.272 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:37:27.311 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:37:27.312 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:37:27.312 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:37:27.412 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:37:27.448 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:37:27.449 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:37:27.449 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:37:27.536 INFO:tasks.radosbench.radosbench.0.smithi092.stdout:Total time run: 90.0159 2024-03-11T21:37:27.536 INFO:tasks.radosbench.radosbench.0.smithi092.stdout:Total writes made: 116381 2024-03-11T21:37:27.536 INFO:tasks.radosbench.radosbench.0.smithi092.stdout:Write size: 65536 2024-03-11T21:37:27.536 INFO:tasks.radosbench.radosbench.0.smithi092.stdout:Object size: 65536 2024-03-11T21:37:27.536 INFO:tasks.radosbench.radosbench.0.smithi092.stdout:Bandwidth (MB/sec): 80.8059 2024-03-11T21:37:27.536 INFO:tasks.radosbench.radosbench.0.smithi092.stdout:Stddev Bandwidth: 49.2445 2024-03-11T21:37:27.536 INFO:tasks.radosbench.radosbench.0.smithi092.stdout:Max bandwidth (MB/sec): 261.062 2024-03-11T21:37:27.536 INFO:tasks.radosbench.radosbench.0.smithi092.stdout:Min bandwidth (MB/sec): 0 2024-03-11T21:37:27.536 INFO:tasks.radosbench.radosbench.0.smithi092.stdout:Average IOPS: 1292 2024-03-11T21:37:27.536 INFO:tasks.radosbench.radosbench.0.smithi092.stdout:Stddev IOPS: 787.921 2024-03-11T21:37:27.536 INFO:tasks.radosbench.radosbench.0.smithi092.stdout:Max IOPS: 4177 2024-03-11T21:37:27.537 INFO:tasks.radosbench.radosbench.0.smithi092.stdout:Min IOPS: 0 2024-03-11T21:37:27.537 INFO:tasks.radosbench.radosbench.0.smithi092.stdout:Average Latency(s): 0.0123702 2024-03-11T21:37:27.537 INFO:tasks.radosbench.radosbench.0.smithi092.stdout:Stddev Latency(s): 0.0311025 2024-03-11T21:37:27.537 INFO:tasks.radosbench.radosbench.0.smithi092.stdout:Max latency(s): 2.50819 2024-03-11T21:37:27.537 INFO:tasks.radosbench.radosbench.0.smithi092.stdout:Min latency(s): 0.00114249 2024-03-11T21:37:27.541 INFO:tasks.radosbench.radosbench.0.smithi092.stdout:Cleaning up (deleting benchmark objects) 2024-03-11T21:37:27.550 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:37:27.587 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:37:27.588 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:37:27.588 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:37:27.689 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:37:27.723 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:37:27.724 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:37:27.724 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:37:27.825 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:37:27.866 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:37:27.867 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:37:27.867 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:37:27.924 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:37:27 smithi092 ceph-mon[29572]: from='client.35363 ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]': finished 2024-03-11T21:37:27.924 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:37:27 smithi092 ceph-mon[29572]: osdmap e295: 8 total, 8 up, 8 in 2024-03-11T21:37:27.924 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:37:27 smithi092 ceph-mon[37563]: from='client.35363 ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]': finished 2024-03-11T21:37:27.925 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:37:27 smithi092 ceph-mon[37563]: osdmap e295: 8 total, 8 up, 8 in 2024-03-11T21:37:27.968 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:37:28.008 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:37:28.008 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:37:28.009 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:37:28.076 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:37:27 smithi097 ceph-mon[33317]: from='client.35363 ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]': finished 2024-03-11T21:37:28.076 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:37:27 smithi097 ceph-mon[33317]: osdmap e295: 8 total, 8 up, 8 in 2024-03-11T21:37:28.110 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:37:28.150 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:37:28.151 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:37:28.151 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:37:28.252 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:37:28.285 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:37:28.286 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:37:28.286 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:37:28.360 INFO:journalctl@ceph.mgr.y.smithi092.stdout:Mar 11 21:37:27 smithi092 ceph-00db0016-dfed-11ee-95c8-87774f69a715-mgr-y[29801]: ::ffff:172.21.15.97 - - [11/Mar/2024:21:37:27] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-11T21:37:28.387 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:37:28.429 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:37:28.430 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:37:28.430 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:37:28.531 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:37:28.570 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:37:28.570 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:37:28.571 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:37:28.672 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:37:28.706 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:37:28.707 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:37:28.707 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:37:28.808 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:37:28.832 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:37:28 smithi092 ceph-mon[29572]: pgmap v544: 148 pgs: 5 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+degraded+remapped, 140 active+clean; 6.9 GiB data, 22 GiB used, 694 GiB / 715 GiB avail; 852 B/s rd, 44 MiB/s wr, 707 op/s; 33812/339141 objects degraded (9.970%); 7.0 MiB/s, 112 objects/s recovering 2024-03-11T21:37:28.833 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:37:28 smithi092 ceph-mon[29572]: Health check update: Degraded data redundancy: 33812/339141 objects degraded (9.970%), 8 pgs degraded (PG_DEGRADED) 2024-03-11T21:37:28.833 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:37:28 smithi092 ceph-mon[29572]: osdmap e296: 8 total, 8 up, 8 in 2024-03-11T21:37:28.833 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:37:28 smithi092 ceph-mon[37563]: pgmap v544: 148 pgs: 5 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+degraded+remapped, 140 active+clean; 6.9 GiB data, 22 GiB used, 694 GiB / 715 GiB avail; 852 B/s rd, 44 MiB/s wr, 707 op/s; 33812/339141 objects degraded (9.970%); 7.0 MiB/s, 112 objects/s recovering 2024-03-11T21:37:28.833 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:37:28 smithi092 ceph-mon[37563]: Health check update: Degraded data redundancy: 33812/339141 objects degraded (9.970%), 8 pgs degraded (PG_DEGRADED) 2024-03-11T21:37:28.833 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:37:28 smithi092 ceph-mon[37563]: osdmap e296: 8 total, 8 up, 8 in 2024-03-11T21:37:28.843 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:37:28.844 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:37:28.844 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:37:28.887 DEBUG:teuthology.orchestra.run.smithi092:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid 00db0016-dfed-11ee-95c8-87774f69a715 -- ceph osd reweight 4 1 2024-03-11T21:37:28.945 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:37:28.976 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:37:28.976 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:37:28.977 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:37:29.076 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:37:28 smithi097 ceph-mon[33317]: pgmap v544: 148 pgs: 5 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+degraded+remapped, 140 active+clean; 6.9 GiB data, 22 GiB used, 694 GiB / 715 GiB avail; 852 B/s rd, 44 MiB/s wr, 707 op/s; 33812/339141 objects degraded (9.970%); 7.0 MiB/s, 112 objects/s recovering 2024-03-11T21:37:29.076 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:37:28 smithi097 ceph-mon[33317]: Health check update: Degraded data redundancy: 33812/339141 objects degraded (9.970%), 8 pgs degraded (PG_DEGRADED) 2024-03-11T21:37:29.076 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:37:28 smithi097 ceph-mon[33317]: osdmap e296: 8 total, 8 up, 8 in 2024-03-11T21:37:29.078 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:37:29.107 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:37:29.108 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:37:29.108 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:37:29.209 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:37:29.258 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:37:29.259 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:37:29.259 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:37:29.360 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:37:29.393 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:37:29.393 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:37:29.394 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:37:29.495 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:37:29.534 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:37:29.534 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:37:29.535 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:37:29.636 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:37:29.686 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:37:29.686 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:37:29.687 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:37:29.788 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:37:29.818 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:37:29.819 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:37:29.819 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:37:29.920 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:37:29.967 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:37:29.967 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:37:29.968 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:37:30.068 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:37:30.097 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:37:30.098 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:37:30.099 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:37:30.199 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:37:30.230 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:37:30.230 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:37:30.231 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:37:30.332 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:37:30.368 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:37:30.369 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:37:30.369 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:37:30.470 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:37:30.514 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:37:30.514 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:37:30.515 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:37:30.616 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:37:30.646 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:37:30.647 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:37:30.647 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:37:30.748 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:37:30.777 INFO:teuthology.orchestra.run.smithi092.stderr:Inferring config /var/lib/ceph/00db0016-dfed-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T21:37:30.778 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:37:30.779 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:37:30.779 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:37:30.880 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:37:30.895 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:37:30 smithi092 ceph-mon[29572]: pgmap v546: 148 pgs: 5 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+degraded+remapped, 140 active+clean; 7.1 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 852 B/s rd, 49 MiB/s wr, 814 op/s; 33473/348801 objects degraded (9.597%); 5.5 MiB/s, 87 objects/s recovering 2024-03-11T21:37:30.895 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:37:30 smithi092 ceph-mon[37563]: pgmap v546: 148 pgs: 5 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+degraded+remapped, 140 active+clean; 7.1 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 852 B/s rd, 49 MiB/s wr, 814 op/s; 33473/348801 objects degraded (9.597%); 5.5 MiB/s, 87 objects/s recovering 2024-03-11T21:37:30.933 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:37:30.934 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:37:30.934 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:37:31.035 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:37:31.065 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:37:31.066 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:37:31.066 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:37:31.076 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:37:30 smithi097 ceph-mon[33317]: pgmap v546: 148 pgs: 5 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+degraded+remapped, 140 active+clean; 7.1 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 852 B/s rd, 49 MiB/s wr, 814 op/s; 33473/348801 objects degraded (9.597%); 5.5 MiB/s, 87 objects/s recovering 2024-03-11T21:37:31.167 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:37:31.217 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:37:31.217 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:37:31.218 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:37:31.318 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:37:31.352 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:37:31.352 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:37:31.353 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:37:31.453 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:37:31.520 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:37:31.521 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:37:31.521 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:37:31.622 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:37:31.665 DEBUG:teuthology.orchestra.run.smithi092:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid 00db0016-dfed-11ee-95c8-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-03-11T21:37:31.666 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:37:31.667 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:37:31.667 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:37:31.679 INFO:teuthology.orchestra.run.smithi092.stderr:nodeep-scrub is set 2024-03-11T21:37:31.768 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:37:31.784 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:37:31 smithi092 ceph-mon[29572]: from='client.? v1:172.21.15.92:0/1726726988' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-11T21:37:31.784 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:37:31 smithi092 ceph-mon[29572]: from='client.35372 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-11T21:37:31.784 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:37:31 smithi092 ceph-mon[29572]: from='mgr.14607 v1:172.21.15.92:0/1702717597' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T21:37:31.784 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:37:31 smithi092 ceph-mon[37563]: from='client.? v1:172.21.15.92:0/1726726988' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-11T21:37:31.784 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:37:31 smithi092 ceph-mon[37563]: from='client.35372 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-11T21:37:31.785 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:37:31 smithi092 ceph-mon[37563]: from='mgr.14607 v1:172.21.15.92:0/1702717597' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T21:37:31.811 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:37:31.812 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:37:31.813 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:37:31.913 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:37:31.972 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:37:31.972 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:37:31.972 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:37:32.073 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:37:32.076 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:37:31 smithi097 ceph-mon[33317]: from='client.? v1:172.21.15.92:0/1726726988' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-11T21:37:32.076 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:37:31 smithi097 ceph-mon[33317]: from='client.35372 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-11T21:37:32.076 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:37:31 smithi097 ceph-mon[33317]: from='mgr.14607 v1:172.21.15.92:0/1702717597' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T21:37:32.148 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:37:32.149 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:37:32.149 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:37:32.250 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:37:32.287 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:37:32.287 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:37:32.288 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:37:32.388 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:37:32.431 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:37:32.431 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:37:32.431 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:37:32.532 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:37:32.562 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:37:32.562 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:37:32.563 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:37:32.664 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:37:32.695 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:37:32.696 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:37:32.696 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:37:32.797 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:37:32.841 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:37:32.842 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:37:32.842 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:37:32.943 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:37:32.980 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:37:32.981 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:37:32.981 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:37:33.077 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:37:32 smithi097 ceph-mon[33317]: pgmap v547: 148 pgs: 5 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+degraded+remapped, 140 active+clean; 7.1 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 647 B/s rd, 37 MiB/s wr, 618 op/s; 33473/348801 objects degraded (9.597%); 4.2 MiB/s, 66 objects/s recovering 2024-03-11T21:37:33.077 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:37:32 smithi097 ceph-mon[33317]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-11T21:37:33.077 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:37:32 smithi097 ceph-mon[33317]: from='client.35372 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-03-11T21:37:33.077 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:37:32 smithi097 ceph-mon[33317]: osdmap e297: 8 total, 8 up, 8 in 2024-03-11T21:37:33.077 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:37:32 smithi097 ceph-mon[33317]: from='mgr.14607 v1:172.21.15.92:0/1702717597' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-11T21:37:33.077 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:37:32 smithi097 ceph-mon[33317]: from='mgr.14607 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-11T21:37:33.077 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:37:32 smithi097 ceph-mon[33317]: from='mgr.14607 v1:172.21.15.92:0/1702717597' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-11T21:37:33.077 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:37:32 smithi097 ceph-mon[33317]: from='mgr.14607 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-11T21:37:33.077 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:37:32 smithi097 ceph-mon[33317]: from='mgr.14607 v1:172.21.15.92:0/1702717597' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-11T21:37:33.077 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:37:32 smithi097 ceph-mon[33317]: from='mgr.14607 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-11T21:37:33.077 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:37:32 smithi097 ceph-mon[33317]: from='mgr.14607 v1:172.21.15.92:0/1702717597' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-03-11T21:37:33.077 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:37:32 smithi097 ceph-mon[33317]: from='mgr.14607 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-03-11T21:37:33.077 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:37:32 smithi097 ceph-mon[33317]: from='mgr.14607 ' entity='mgr.y' 2024-03-11T21:37:33.077 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:37:32 smithi097 ceph-mon[33317]: from='mgr.14607 v1:172.21.15.92:0/1702717597' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T21:37:33.078 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:37:32 smithi097 ceph-mon[33317]: from='mgr.14607 v1:172.21.15.92:0/1702717597' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T21:37:33.078 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:37:32 smithi097 ceph-mon[33317]: from='mgr.14607 ' entity='mgr.y' 2024-03-11T21:37:33.078 INFO:journalctl@ceph.iscsi.iscsi.a.smithi097.stdout:Mar 11 21:37:32 smithi097 ceph-00db0016-dfed-11ee-95c8-87774f69a715-iscsi-iscsi-a[60943]: debug there is no tcmu-runner data available 2024-03-11T21:37:33.082 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:37:33.110 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:37:32 smithi092 ceph-mon[29572]: pgmap v547: 148 pgs: 5 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+degraded+remapped, 140 active+clean; 7.1 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 647 B/s rd, 37 MiB/s wr, 618 op/s; 33473/348801 objects degraded (9.597%); 4.2 MiB/s, 66 objects/s recovering 2024-03-11T21:37:33.111 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:37:32 smithi092 ceph-mon[29572]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-11T21:37:33.111 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:37:32 smithi092 ceph-mon[29572]: from='client.35372 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-03-11T21:37:33.111 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:37:32 smithi092 ceph-mon[29572]: osdmap e297: 8 total, 8 up, 8 in 2024-03-11T21:37:33.111 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:37:32 smithi092 ceph-mon[29572]: from='mgr.14607 v1:172.21.15.92:0/1702717597' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-11T21:37:33.111 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:37:32 smithi092 ceph-mon[29572]: from='mgr.14607 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-11T21:37:33.111 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:37:32 smithi092 ceph-mon[29572]: from='mgr.14607 v1:172.21.15.92:0/1702717597' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-11T21:37:33.111 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:37:32 smithi092 ceph-mon[29572]: from='mgr.14607 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-11T21:37:33.111 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:37:32 smithi092 ceph-mon[29572]: from='mgr.14607 v1:172.21.15.92:0/1702717597' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-11T21:37:33.111 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:37:32 smithi092 ceph-mon[29572]: from='mgr.14607 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-11T21:37:33.111 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:37:32 smithi092 ceph-mon[29572]: from='mgr.14607 v1:172.21.15.92:0/1702717597' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-03-11T21:37:33.111 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:37:32 smithi092 ceph-mon[29572]: from='mgr.14607 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-03-11T21:37:33.111 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:37:32 smithi092 ceph-mon[29572]: from='mgr.14607 ' entity='mgr.y' 2024-03-11T21:37:33.111 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:37:32 smithi092 ceph-mon[29572]: from='mgr.14607 v1:172.21.15.92:0/1702717597' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T21:37:33.111 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:37:32 smithi092 ceph-mon[29572]: from='mgr.14607 v1:172.21.15.92:0/1702717597' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T21:37:33.112 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:37:32 smithi092 ceph-mon[29572]: from='mgr.14607 ' entity='mgr.y' 2024-03-11T21:37:33.112 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:37:32 smithi092 ceph-mon[37563]: pgmap v547: 148 pgs: 5 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+degraded+remapped, 140 active+clean; 7.1 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 647 B/s rd, 37 MiB/s wr, 618 op/s; 33473/348801 objects degraded (9.597%); 4.2 MiB/s, 66 objects/s recovering 2024-03-11T21:37:33.112 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:37:32 smithi092 ceph-mon[37563]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-11T21:37:33.112 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:37:32 smithi092 ceph-mon[37563]: from='client.35372 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-03-11T21:37:33.112 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:37:32 smithi092 ceph-mon[37563]: osdmap e297: 8 total, 8 up, 8 in 2024-03-11T21:37:33.112 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:37:32 smithi092 ceph-mon[37563]: from='mgr.14607 v1:172.21.15.92:0/1702717597' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-11T21:37:33.112 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:37:32 smithi092 ceph-mon[37563]: from='mgr.14607 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-11T21:37:33.112 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:37:32 smithi092 ceph-mon[37563]: from='mgr.14607 v1:172.21.15.92:0/1702717597' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-11T21:37:33.112 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:37:32 smithi092 ceph-mon[37563]: from='mgr.14607 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-11T21:37:33.112 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:37:32 smithi092 ceph-mon[37563]: from='mgr.14607 v1:172.21.15.92:0/1702717597' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-11T21:37:33.112 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:37:32 smithi092 ceph-mon[37563]: from='mgr.14607 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-11T21:37:33.112 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:37:32 smithi092 ceph-mon[37563]: from='mgr.14607 v1:172.21.15.92:0/1702717597' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-03-11T21:37:33.112 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:37:32 smithi092 ceph-mon[37563]: from='mgr.14607 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-03-11T21:37:33.113 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:37:32 smithi092 ceph-mon[37563]: from='mgr.14607 ' entity='mgr.y' 2024-03-11T21:37:33.113 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:37:32 smithi092 ceph-mon[37563]: from='mgr.14607 v1:172.21.15.92:0/1702717597' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T21:37:33.113 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:37:32 smithi092 ceph-mon[37563]: from='mgr.14607 v1:172.21.15.92:0/1702717597' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T21:37:33.113 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:37:32 smithi092 ceph-mon[37563]: from='mgr.14607 ' entity='mgr.y' 2024-03-11T21:37:33.122 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:37:33.122 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:37:33.123 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:37:33.223 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:37:33.260 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:37:33.261 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:37:33.261 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:37:33.362 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:37:33.406 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:37:33.407 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:37:33.407 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:37:33.508 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:37:33.558 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:37:33.559 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:37:33.559 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:37:33.660 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:37:33.702 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:37:33.703 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:37:33.703 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:37:33.804 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:37:33.837 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:37:33.838 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:37:33.838 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:37:33.939 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:37:33.970 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:37:33.971 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:37:33.971 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:37:34.072 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:37:34.076 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:37:33 smithi097 ceph-mon[33317]: Adjusting osd_memory_target on smithi092 to 3248M 2024-03-11T21:37:34.076 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:37:33 smithi097 ceph-mon[33317]: from='client.24469 v1:172.21.15.97:0/404208310' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:37:34.076 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:37:33 smithi097 ceph-mon[33317]: Health check update: Degraded data redundancy: 33473/348801 objects degraded (9.597%), 8 pgs degraded (PG_DEGRADED) 2024-03-11T21:37:34.076 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:37:33 smithi097 ceph-mon[33317]: osdmap e298: 8 total, 8 up, 8 in 2024-03-11T21:37:34.110 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:37:33 smithi092 ceph-mon[29572]: Adjusting osd_memory_target on smithi092 to 3248M 2024-03-11T21:37:34.110 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:37:33 smithi092 ceph-mon[29572]: from='client.24469 v1:172.21.15.97:0/404208310' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:37:34.111 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:37:33 smithi092 ceph-mon[29572]: Health check update: Degraded data redundancy: 33473/348801 objects degraded (9.597%), 8 pgs degraded (PG_DEGRADED) 2024-03-11T21:37:34.111 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:37:33 smithi092 ceph-mon[29572]: osdmap e298: 8 total, 8 up, 8 in 2024-03-11T21:37:34.111 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:37:33 smithi092 ceph-mon[37563]: Adjusting osd_memory_target on smithi092 to 3248M 2024-03-11T21:37:34.111 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:37:33 smithi092 ceph-mon[37563]: from='client.24469 v1:172.21.15.97:0/404208310' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:37:34.111 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:37:33 smithi092 ceph-mon[37563]: Health check update: Degraded data redundancy: 33473/348801 objects degraded (9.597%), 8 pgs degraded (PG_DEGRADED) 2024-03-11T21:37:34.111 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:37:33 smithi092 ceph-mon[37563]: osdmap e298: 8 total, 8 up, 8 in 2024-03-11T21:37:34.114 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:37:34.115 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:37:34.115 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:37:34.216 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:37:34.257 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:37:34.258 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:37:34.258 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:37:34.359 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:37:34.404 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:37:34.405 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:37:34.405 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:37:34.506 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:37:34.533 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:37:34.534 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:37:34.535 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:37:34.635 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:37:34.664 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:37:34.665 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:37:34.665 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:37:34.766 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:37:34.801 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:37:34.801 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:37:34.802 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:37:34.902 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:37:34.947 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:37:34.948 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:37:34.949 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:37:35.015 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:37:34 smithi092 ceph-mon[29572]: pgmap v549: 148 pgs: 5 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+degraded+remapped, 140 active+clean; 6.9 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 1.7 KiB/s rd, 36 MiB/s wr, 1.01k op/s; 32960/341130 objects degraded (9.662%); 9.8 MiB/s, 157 objects/s recovering 2024-03-11T21:37:35.015 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:37:34 smithi092 ceph-mon[37563]: pgmap v549: 148 pgs: 5 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+degraded+remapped, 140 active+clean; 6.9 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 1.7 KiB/s rd, 36 MiB/s wr, 1.01k op/s; 32960/341130 objects degraded (9.662%); 9.8 MiB/s, 157 objects/s recovering 2024-03-11T21:37:35.050 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:37:35.076 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:37:34 smithi097 ceph-mon[33317]: pgmap v549: 148 pgs: 5 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+degraded+remapped, 140 active+clean; 6.9 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 1.7 KiB/s rd, 36 MiB/s wr, 1.01k op/s; 32960/341130 objects degraded (9.662%); 9.8 MiB/s, 157 objects/s recovering 2024-03-11T21:37:35.087 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:37:35.088 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:37:35.088 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:37:35.189 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:37:35.227 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:37:35.227 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:37:35.227 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:37:35.328 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:37:35.366 DEBUG:teuthology.orchestra.run.smithi092:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid 00db0016-dfed-11ee-95c8-87774f69a715 -- ceph osd unset noscrub 2024-03-11T21:37:35.371 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:37:35.372 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:37:35.372 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:37:35.473 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:37:35.505 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:37:35.505 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:37:35.506 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:37:35.607 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:37:35.641 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:37:35.642 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:37:35.643 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:37:35.743 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:37:35.798 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:37:35.799 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:37:35.799 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:37:35.833 INFO:teuthology.orchestra.run.smithi092.stdout:{ 2024-03-11T21:37:35.833 INFO:teuthology.orchestra.run.smithi092.stdout: "ops": [], 2024-03-11T21:37:35.833 INFO:teuthology.orchestra.run.smithi092.stdout: "complaint_time": 30, 2024-03-11T21:37:35.834 INFO:teuthology.orchestra.run.smithi092.stdout: "num_blocked_ops": 0 2024-03-11T21:37:35.834 INFO:teuthology.orchestra.run.smithi092.stdout:} 2024-03-11T21:37:35.899 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:37:35.930 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:37:35.930 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:37:35.931 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:37:36.031 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:37:36.079 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:37:36.080 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:37:36.080 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:37:36.109 INFO:teuthology.orchestra.run.smithi092.stderr:Inferring config /var/lib/ceph/00db0016-dfed-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T21:37:36.181 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:37:36.227 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:37:36.228 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:37:36.228 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:37:36.329 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:37:36.358 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:37:36.359 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:37:36.359 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:37:36.460 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:37:36.503 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:37:36.504 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:37:36.504 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:37:36.605 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:37:36.655 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:37:36.655 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:37:36.655 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:37:36.677 INFO:teuthology.orchestra.run.smithi092.stderr:reweighted osd.4 to 1 (10000) 2024-03-11T21:37:36.757 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:37:36.770 DEBUG:teuthology.orchestra.run.smithi092:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid 00db0016-dfed-11ee-95c8-87774f69a715 -- ceph daemon osd.2 dump_historic_ops 2024-03-11T21:37:36.800 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:37:36.801 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:37:36.802 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:37:36.902 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:37:36.942 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:37:36.942 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:37:36.943 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:37:37.023 INFO:teuthology.orchestra.run.smithi092.stderr:Inferring config /var/lib/ceph/00db0016-dfed-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T21:37:37.043 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:37:37.076 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:37:36 smithi097 ceph-mon[33317]: pgmap v551: 148 pgs: 5 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+degraded+remapped, 140 active+clean; 6.9 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 889 B/s rd, 8.1 MiB/s wr, 681 op/s; 32960/338298 objects degraded (9.743%); 4.6 MiB/s, 74 objects/s recovering 2024-03-11T21:37:37.076 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:37:36 smithi097 ceph-mon[33317]: from='client.? v1:172.21.15.92:0/3099891229' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]: dispatch 2024-03-11T21:37:37.103 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:37:37.104 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:37:37.104 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:37:37.110 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:37:36 smithi092 ceph-mon[29572]: pgmap v551: 148 pgs: 5 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+degraded+remapped, 140 active+clean; 6.9 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 889 B/s rd, 8.1 MiB/s wr, 681 op/s; 32960/338298 objects degraded (9.743%); 4.6 MiB/s, 74 objects/s recovering 2024-03-11T21:37:37.110 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:37:36 smithi092 ceph-mon[29572]: from='client.? v1:172.21.15.92:0/3099891229' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]: dispatch 2024-03-11T21:37:37.110 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:37:36 smithi092 ceph-mon[37563]: pgmap v551: 148 pgs: 5 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+degraded+remapped, 140 active+clean; 6.9 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 889 B/s rd, 8.1 MiB/s wr, 681 op/s; 32960/338298 objects degraded (9.743%); 4.6 MiB/s, 74 objects/s recovering 2024-03-11T21:37:37.110 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:37:36 smithi092 ceph-mon[37563]: from='client.? v1:172.21.15.92:0/3099891229' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]: dispatch 2024-03-11T21:37:37.205 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:37:37.244 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:37:37.245 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:37:37.245 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:37:37.346 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:37:37.372 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:37:37.373 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:37:37.373 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:37:37.474 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:37:37.514 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:37:37.515 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:37:37.515 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:37:37.616 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:37:37.653 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:37:37.654 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:37:37.655 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:37:37.755 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:37:37.789 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:37:37.789 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:37:37.790 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:37:37.861 DEBUG:teuthology.orchestra.run.smithi092:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid 00db0016-dfed-11ee-95c8-87774f69a715 -- ceph osd reweight 5 1 2024-03-11T21:37:37.890 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:37:37.920 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:37:37.921 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:37:37.921 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:37:38.022 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:37:38.051 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:37:38.051 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:37:38.052 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:37:38.076 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:37:37 smithi097 ceph-mon[33317]: from='client.? v1:172.21.15.92:0/3099891229' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]': finished 2024-03-11T21:37:38.076 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:37:37 smithi097 ceph-mon[33317]: osdmap e299: 8 total, 8 up, 8 in 2024-03-11T21:37:38.076 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:37:37 smithi097 ceph-mon[33317]: from='mgr.14607 v1:172.21.15.92:0/1702717597' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T21:37:38.110 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:37:37 smithi092 ceph-mon[29572]: from='client.? v1:172.21.15.92:0/3099891229' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]': finished 2024-03-11T21:37:38.110 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:37:37 smithi092 ceph-mon[29572]: osdmap e299: 8 total, 8 up, 8 in 2024-03-11T21:37:38.110 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:37:37 smithi092 ceph-mon[29572]: from='mgr.14607 v1:172.21.15.92:0/1702717597' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T21:37:38.111 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:37:37 smithi092 ceph-mon[37563]: from='client.? v1:172.21.15.92:0/3099891229' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]': finished 2024-03-11T21:37:38.111 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:37:37 smithi092 ceph-mon[37563]: osdmap e299: 8 total, 8 up, 8 in 2024-03-11T21:37:38.111 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:37:37 smithi092 ceph-mon[37563]: from='mgr.14607 v1:172.21.15.92:0/1702717597' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T21:37:38.111 INFO:journalctl@ceph.mgr.y.smithi092.stdout:Mar 11 21:37:37 smithi092 ceph-00db0016-dfed-11ee-95c8-87774f69a715-mgr-y[29801]: ::ffff:172.21.15.97 - - [11/Mar/2024:21:37:37] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-11T21:37:38.152 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:37:38.207 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:37:38.207 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:37:38.208 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:37:38.308 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:37:38.358 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:37:38.359 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:37:38.359 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:37:38.460 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:37:38.502 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:37:38.503 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:37:38.503 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:37:38.604 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:37:38.637 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:37:38.638 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:37:38.638 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:37:38.739 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:37:38.783 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:37:38.783 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:37:38.784 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:37:38.884 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:37:38.916 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:37:38.917 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:37:38.917 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:37:38.954 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:37:38 smithi092 ceph-mon[29572]: pgmap v553: 148 pgs: 5 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+degraded+remapped, 140 active+clean; 6.9 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 1022 B/s rd, 1.3 MiB/s wr, 669 op/s; 32779/337509 objects degraded (9.712%); 7.2 MiB/s, 115 objects/s recovering 2024-03-11T21:37:38.954 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:37:38 smithi092 ceph-mon[29572]: Health check update: Degraded data redundancy: 32779/337509 objects degraded (9.712%), 8 pgs degraded, 8 pgs undersized (PG_DEGRADED) 2024-03-11T21:37:38.954 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:37:38 smithi092 ceph-mon[29572]: osdmap e300: 8 total, 8 up, 8 in 2024-03-11T21:37:38.954 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:37:38 smithi092 ceph-mon[37563]: pgmap v553: 148 pgs: 5 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+degraded+remapped, 140 active+clean; 6.9 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 1022 B/s rd, 1.3 MiB/s wr, 669 op/s; 32779/337509 objects degraded (9.712%); 7.2 MiB/s, 115 objects/s recovering 2024-03-11T21:37:38.954 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:37:38 smithi092 ceph-mon[37563]: Health check update: Degraded data redundancy: 32779/337509 objects degraded (9.712%), 8 pgs degraded, 8 pgs undersized (PG_DEGRADED) 2024-03-11T21:37:38.955 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:37:38 smithi092 ceph-mon[37563]: osdmap e300: 8 total, 8 up, 8 in 2024-03-11T21:37:39.018 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:37:39.057 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:37:39.058 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:37:39.058 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:37:39.076 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:37:38 smithi097 ceph-mon[33317]: pgmap v553: 148 pgs: 5 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+degraded+remapped, 140 active+clean; 6.9 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 1022 B/s rd, 1.3 MiB/s wr, 669 op/s; 32779/337509 objects degraded (9.712%); 7.2 MiB/s, 115 objects/s recovering 2024-03-11T21:37:39.076 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:37:38 smithi097 ceph-mon[33317]: Health check update: Degraded data redundancy: 32779/337509 objects degraded (9.712%), 8 pgs degraded, 8 pgs undersized (PG_DEGRADED) 2024-03-11T21:37:39.076 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:37:38 smithi097 ceph-mon[33317]: osdmap e300: 8 total, 8 up, 8 in 2024-03-11T21:37:39.159 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:37:39.196 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:37:39.197 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:37:39.197 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:37:39.298 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:37:39.337 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:37:39.338 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:37:39.338 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:37:39.439 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:37:39.477 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:37:39.478 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:37:39.478 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:37:39.579 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:37:39.618 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:37:39.618 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:37:39.619 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:37:39.719 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:37:39.755 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:37:39.755 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:37:39.756 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:37:39.826 INFO:journalctl@ceph.grafana.a.smithi097.stdout:Mar 11 21:37:39 smithi097 ceph-00db0016-dfed-11ee-95c8-87774f69a715-grafana-a[65126]: logger=cleanup t=2024-03-11T21:37:39.529053851Z level=info msg="Completed cleanup jobs" duration=3.517141ms 2024-03-11T21:37:39.857 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:37:39.892 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:37:39.893 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:37:39.893 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:37:39.994 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:37:40.028 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:37:40.029 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:37:40.029 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:37:40.130 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:37:40.176 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:37:40.177 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:37:40.177 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:37:40.236 INFO:teuthology.orchestra.run.smithi092.stderr:Inferring config /var/lib/ceph/00db0016-dfed-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T21:37:40.278 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:37:40.317 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:37:40.317 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:37:40.318 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:37:40.418 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:37:40.461 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:37:40.462 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:37:40.462 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:37:40.563 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:37:40.606 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:37:40.606 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:37:40.607 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:37:40.707 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:37:40.747 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:37:40.747 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:37:40.747 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:37:40.848 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:37:40.880 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:37:40.881 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:37:40.881 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:37:40.982 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:37:40.999 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:37:40 smithi092 ceph-mon[37563]: purged_snaps scrub starts 2024-03-11T21:37:40.999 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:37:40 smithi092 ceph-mon[37563]: purged_snaps scrub ok 2024-03-11T21:37:40.999 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:37:40 smithi092 ceph-mon[37563]: pgmap v555: 148 pgs: 5 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+degraded+remapped, 140 active+clean; 6.6 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 852 B/s rd, 0 B/s wr, 828 op/s; 32439/326208 objects degraded (9.944%); 5.4 MiB/s, 86 objects/s recovering 2024-03-11T21:37:40.999 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:37:40 smithi092 ceph-mon[37563]: osdmap e301: 8 total, 8 up, 8 in 2024-03-11T21:37:41.000 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:37:40 smithi092 ceph-mon[29572]: purged_snaps scrub starts 2024-03-11T21:37:41.000 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:37:40 smithi092 ceph-mon[29572]: purged_snaps scrub ok 2024-03-11T21:37:41.000 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:37:40 smithi092 ceph-mon[29572]: pgmap v555: 148 pgs: 5 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+degraded+remapped, 140 active+clean; 6.6 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 852 B/s rd, 0 B/s wr, 828 op/s; 32439/326208 objects degraded (9.944%); 5.4 MiB/s, 86 objects/s recovering 2024-03-11T21:37:41.000 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:37:40 smithi092 ceph-mon[29572]: osdmap e301: 8 total, 8 up, 8 in 2024-03-11T21:37:41.028 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:37:41.028 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:37:41.029 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:37:41.076 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:37:40 smithi097 ceph-mon[33317]: purged_snaps scrub starts 2024-03-11T21:37:41.076 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:37:40 smithi097 ceph-mon[33317]: purged_snaps scrub ok 2024-03-11T21:37:41.076 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:37:40 smithi097 ceph-mon[33317]: pgmap v555: 148 pgs: 5 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+degraded+remapped, 140 active+clean; 6.6 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 852 B/s rd, 0 B/s wr, 828 op/s; 32439/326208 objects degraded (9.944%); 5.4 MiB/s, 86 objects/s recovering 2024-03-11T21:37:41.077 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:37:40 smithi097 ceph-mon[33317]: osdmap e301: 8 total, 8 up, 8 in 2024-03-11T21:37:41.129 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:37:41.158 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:37:41.159 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:37:41.159 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:37:41.260 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:37:41.301 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:37:41.301 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:37:41.302 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:37:41.402 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:37:41.436 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:37:41.437 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:37:41.437 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:37:41.538 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:37:41.582 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:37:41.583 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:37:41.583 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:37:41.684 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:37:41.712 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:37:41.713 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:37:41.714 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:37:41.814 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:37:41.863 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:37:41.864 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:37:41.864 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:37:41.942 INFO:teuthology.orchestra.run.smithi092.stderr:Inferring config /var/lib/ceph/00db0016-dfed-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T21:37:41.965 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:37:42.015 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:37:42.015 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:37:42.016 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:37:42.066 INFO:teuthology.orchestra.run.smithi092.stdout:osd.0: {} 2024-03-11T21:37:42.109 INFO:teuthology.orchestra.run.smithi092.stdout:osd.1: {} 2024-03-11T21:37:42.116 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:37:42.145 INFO:teuthology.orchestra.run.smithi092.stdout:osd.2: {} 2024-03-11T21:37:42.152 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:37:42.152 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:37:42.153 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:37:42.182 INFO:teuthology.orchestra.run.smithi092.stdout:osd.3: {} 2024-03-11T21:37:42.221 INFO:teuthology.orchestra.run.smithi092.stdout:osd.4: {} 2024-03-11T21:37:42.253 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:37:42.261 INFO:teuthology.orchestra.run.smithi092.stdout:osd.5: {} 2024-03-11T21:37:42.298 INFO:teuthology.orchestra.run.smithi092.stdout:osd.6: {} 2024-03-11T21:37:42.299 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:37:42.299 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:37:42.299 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:37:42.338 INFO:teuthology.orchestra.run.smithi092.stdout:osd.7: {} 2024-03-11T21:37:42.340 INFO:teuthology.orchestra.run.smithi092.stderr:osd.0: osd_enable_op_tracker = '' 2024-03-11T21:37:42.340 INFO:teuthology.orchestra.run.smithi092.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-11T21:37:42.341 INFO:teuthology.orchestra.run.smithi092.stderr:osd.2: osd_enable_op_tracker = '' 2024-03-11T21:37:42.341 INFO:teuthology.orchestra.run.smithi092.stderr:osd.3: osd_enable_op_tracker = '' 2024-03-11T21:37:42.341 INFO:teuthology.orchestra.run.smithi092.stderr:osd.4: osd_enable_op_tracker = '' 2024-03-11T21:37:42.341 INFO:teuthology.orchestra.run.smithi092.stderr:osd.5: osd_enable_op_tracker = '' 2024-03-11T21:37:42.341 INFO:teuthology.orchestra.run.smithi092.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-11T21:37:42.341 INFO:teuthology.orchestra.run.smithi092.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-11T21:37:42.401 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:37:42.442 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:37:42.443 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:37:42.443 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:37:42.544 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:37:42.578 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:37:42.579 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:37:42.579 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:37:42.680 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:37:42.714 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:37:42.715 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:37:42.715 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:37:42.816 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:37:42.850 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:37:42.851 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:37:42.851 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:37:42.952 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:37:42.979 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:37:42 smithi092 ceph-mon[29572]: pgmap v557: 148 pgs: 5 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+degraded+remapped, 140 active+clean; 6.6 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 853 B/s rd, 0 B/s wr, 672 op/s; 32439/326208 objects degraded (9.944%); 5.4 MiB/s, 86 objects/s recovering 2024-03-11T21:37:42.979 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:37:42 smithi092 ceph-mon[29572]: from='client.? v1:172.21.15.92:0/2185213593' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-11T21:37:42.979 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:37:42 smithi092 ceph-mon[37563]: pgmap v557: 148 pgs: 5 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+degraded+remapped, 140 active+clean; 6.6 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 853 B/s rd, 0 B/s wr, 672 op/s; 32439/326208 objects degraded (9.944%); 5.4 MiB/s, 86 objects/s recovering 2024-03-11T21:37:42.980 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:37:42 smithi092 ceph-mon[37563]: from='client.? v1:172.21.15.92:0/2185213593' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-11T21:37:42.980 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:37:42.980 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:37:42.981 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:37:43.076 INFO:journalctl@ceph.iscsi.iscsi.a.smithi097.stdout:Mar 11 21:37:42 smithi097 ceph-00db0016-dfed-11ee-95c8-87774f69a715-iscsi-iscsi-a[60943]: debug there is no tcmu-runner data available 2024-03-11T21:37:43.077 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:37:42 smithi097 ceph-mon[33317]: pgmap v557: 148 pgs: 5 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+degraded+remapped, 140 active+clean; 6.6 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 853 B/s rd, 0 B/s wr, 672 op/s; 32439/326208 objects degraded (9.944%); 5.4 MiB/s, 86 objects/s recovering 2024-03-11T21:37:43.077 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:37:42 smithi097 ceph-mon[33317]: from='client.? v1:172.21.15.92:0/2185213593' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-11T21:37:43.081 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:37:43.123 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:37:43.124 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:37:43.125 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:37:43.225 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:37:43.265 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:37:43.266 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:37:43.266 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:37:43.366 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:37:43.396 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:37:43.396 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:37:43.397 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:37:43.497 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:37:43.549 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:37:43.549 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:37:43.549 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:37:43.650 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:37:43.690 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:37:43.690 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:37:43.691 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:37:43.791 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:37:43.820 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:37:43.821 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:37:43.821 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:37:43.922 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:37:43.959 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:37:43.960 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:37:43.960 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:37:44.016 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:37:43 smithi092 ceph-mon[29572]: from='client.24469 v1:172.21.15.97:0/404208310' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:37:44.016 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:37:43 smithi092 ceph-mon[29572]: Health check update: Degraded data redundancy: 32439/326208 objects degraded (9.944%), 8 pgs degraded, 8 pgs undersized (PG_DEGRADED) 2024-03-11T21:37:44.017 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:37:43 smithi092 ceph-mon[29572]: osdmap e302: 8 total, 8 up, 8 in 2024-03-11T21:37:44.017 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:37:43 smithi092 ceph-mon[37563]: from='client.24469 v1:172.21.15.97:0/404208310' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:37:44.017 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:37:43 smithi092 ceph-mon[37563]: Health check update: Degraded data redundancy: 32439/326208 objects degraded (9.944%), 8 pgs degraded, 8 pgs undersized (PG_DEGRADED) 2024-03-11T21:37:44.017 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:37:43 smithi092 ceph-mon[37563]: osdmap e302: 8 total, 8 up, 8 in 2024-03-11T21:37:44.061 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:37:44.076 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:37:43 smithi097 ceph-mon[33317]: from='client.24469 v1:172.21.15.97:0/404208310' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:37:44.076 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:37:43 smithi097 ceph-mon[33317]: Health check update: Degraded data redundancy: 32439/326208 objects degraded (9.944%), 8 pgs degraded, 8 pgs undersized (PG_DEGRADED) 2024-03-11T21:37:44.076 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:37:43 smithi097 ceph-mon[33317]: osdmap e302: 8 total, 8 up, 8 in 2024-03-11T21:37:44.099 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:37:44.100 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:37:44.101 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:37:44.201 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:37:44.245 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:37:44.246 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:37:44.247 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:37:44.347 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:37:44.380 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:37:44.381 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:37:44.381 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:37:44.482 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:37:44.513 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:37:44.514 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:37:44.514 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:37:44.541 INFO:teuthology.orchestra.run.smithi092.stdout:ERROR: (22) Invalid argument 2024-03-11T21:37:44.541 INFO:teuthology.orchestra.run.smithi092.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-11T21:37:44.615 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:37:44.654 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:37:44.655 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:37:44.656 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:37:44.756 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:37:44.795 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:37:44.796 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:37:44.796 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:37:44.897 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:37:44.936 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:37:44.937 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:37:44.937 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:37:44.942 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:37:44 smithi092 ceph-mon[29572]: pgmap v558: 148 pgs: 5 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+degraded+remapped, 140 active+clean; 6.4 GiB data, 21 GiB used, 695 GiB / 715 GiB avail; 1.5 KiB/s rd, 0 B/s wr, 1.09k op/s; 31880/316815 objects degraded (10.063%); 10 MiB/s, 164 objects/s recovering 2024-03-11T21:37:44.942 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:37:44 smithi092 ceph-mon[37563]: pgmap v558: 148 pgs: 5 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+degraded+remapped, 140 active+clean; 6.4 GiB data, 21 GiB used, 695 GiB / 715 GiB avail; 1.5 KiB/s rd, 0 B/s wr, 1.09k op/s; 31880/316815 objects degraded (10.063%); 10 MiB/s, 164 objects/s recovering 2024-03-11T21:37:45.038 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:37:45.071 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:37:45.072 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:37:45.073 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:37:45.076 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:37:44 smithi097 ceph-mon[33317]: pgmap v558: 148 pgs: 5 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+degraded+remapped, 140 active+clean; 6.4 GiB data, 21 GiB used, 695 GiB / 715 GiB avail; 1.5 KiB/s rd, 0 B/s wr, 1.09k op/s; 31880/316815 objects degraded (10.063%); 10 MiB/s, 164 objects/s recovering 2024-03-11T21:37:45.174 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:37:45.207 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:37:45.208 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:37:45.208 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:37:45.309 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:37:45.358 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:37:45.359 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:37:45.359 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:37:45.460 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:37:45.493 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:37:45.494 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:37:45.494 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:37:45.522 DEBUG:teuthology.orchestra.run.smithi092:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid 00db0016-dfed-11ee-95c8-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-03-11T21:37:45.596 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:37:45.647 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:37:45.648 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:37:45.648 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:37:45.749 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:37:45.770 INFO:teuthology.orchestra.run.smithi092.stderr:noscrub is unset 2024-03-11T21:37:45.801 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:37:45.802 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:37:45.802 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:37:45.860 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:37:45 smithi092 ceph-mon[29572]: from='client.? v1:172.21.15.92:0/1241484778' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-11T21:37:45.861 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:37:45 smithi092 ceph-mon[29572]: from='client.25402 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-11T21:37:45.861 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:37:45 smithi092 ceph-mon[37563]: from='client.? v1:172.21.15.92:0/1241484778' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-11T21:37:45.861 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:37:45 smithi092 ceph-mon[37563]: from='client.25402 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-11T21:37:45.904 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:37:45.955 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:37:45.956 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:37:45.956 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:37:46.057 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:37:46.076 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:37:45 smithi097 ceph-mon[33317]: from='client.? v1:172.21.15.92:0/1241484778' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-11T21:37:46.076 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:37:45 smithi097 ceph-mon[33317]: from='client.25402 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-11T21:37:46.081 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:37:46.081 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:37:46.081 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:37:46.182 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:37:46.214 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:37:46.215 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:37:46.215 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:37:46.316 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:37:46.344 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:37:46.344 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:37:46.345 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:37:46.446 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:37:46.486 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:37:46.486 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:37:46.487 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:37:46.588 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:37:46.625 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:37:46.625 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:37:46.625 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:37:46.726 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:37:46.770 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:37:46.771 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:37:46.771 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:37:46.872 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:37:46.908 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:37:46.909 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:37:46.910 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:37:47.010 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:37:47.050 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:37:47.050 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:37:47.050 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:37:47.076 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:37:46 smithi097 ceph-mon[33317]: pgmap v560: 148 pgs: 5 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+degraded+remapped, 140 active+clean; 6.4 GiB data, 21 GiB used, 695 GiB / 715 GiB avail; 741 B/s rd, 0 B/s wr, 755 op/s; 31880/313548 objects degraded (10.168%); 5.1 MiB/s, 80 objects/s recovering 2024-03-11T21:37:47.077 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:37:46 smithi097 ceph-mon[33317]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-11T21:37:47.077 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:37:46 smithi097 ceph-mon[33317]: from='client.25402 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-03-11T21:37:47.077 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:37:46 smithi097 ceph-mon[33317]: osdmap e303: 8 total, 8 up, 8 in 2024-03-11T21:37:47.110 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:37:46 smithi092 ceph-mon[29572]: pgmap v560: 148 pgs: 5 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+degraded+remapped, 140 active+clean; 6.4 GiB data, 21 GiB used, 695 GiB / 715 GiB avail; 741 B/s rd, 0 B/s wr, 755 op/s; 31880/313548 objects degraded (10.168%); 5.1 MiB/s, 80 objects/s recovering 2024-03-11T21:37:47.111 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:37:46 smithi092 ceph-mon[29572]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-11T21:37:47.111 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:37:46 smithi092 ceph-mon[29572]: from='client.25402 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-03-11T21:37:47.111 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:37:46 smithi092 ceph-mon[29572]: osdmap e303: 8 total, 8 up, 8 in 2024-03-11T21:37:47.111 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:37:46 smithi092 ceph-mon[37563]: pgmap v560: 148 pgs: 5 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+degraded+remapped, 140 active+clean; 6.4 GiB data, 21 GiB used, 695 GiB / 715 GiB avail; 741 B/s rd, 0 B/s wr, 755 op/s; 31880/313548 objects degraded (10.168%); 5.1 MiB/s, 80 objects/s recovering 2024-03-11T21:37:47.111 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:37:46 smithi092 ceph-mon[37563]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-11T21:37:47.111 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:37:46 smithi092 ceph-mon[37563]: from='client.25402 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-03-11T21:37:47.111 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:37:46 smithi092 ceph-mon[37563]: osdmap e303: 8 total, 8 up, 8 in 2024-03-11T21:37:47.152 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:37:47.190 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:37:47.191 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:37:47.191 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:37:47.292 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:37:47.299 DEBUG:teuthology.orchestra.run.smithi097:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid 00db0016-dfed-11ee-95c8-87774f69a715 -- ceph daemon osd.4 dump_ops_in_flight 2024-03-11T21:37:47.347 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:37:47.347 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:37:47.347 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:37:47.448 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:37:47.496 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:37:47.496 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:37:47.497 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:37:47.597 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:37:47.641 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:37:47.642 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:37:47.642 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:37:47.684 INFO:teuthology.orchestra.run.smithi097.stderr:Inferring config /var/lib/ceph/00db0016-dfed-11ee-95c8-87774f69a715/mon.b/config 2024-03-11T21:37:47.743 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:37:47.782 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:37:47.783 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:37:47.783 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:37:47.884 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:37:47.926 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:37:47.927 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:37:47.927 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:37:48.028 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:37:48.042 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:37:47 smithi097 ceph-mon[33317]: purged_snaps scrub starts 2024-03-11T21:37:48.042 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:37:47 smithi097 ceph-mon[33317]: purged_snaps scrub ok 2024-03-11T21:37:48.042 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:37:47 smithi097 ceph-mon[33317]: osdmap e304: 8 total, 8 up, 8 in 2024-03-11T21:37:48.074 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:37:48.074 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:37:48.075 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:37:48.110 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:37:47 smithi092 ceph-mon[29572]: purged_snaps scrub starts 2024-03-11T21:37:48.111 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:37:47 smithi092 ceph-mon[29572]: purged_snaps scrub ok 2024-03-11T21:37:48.111 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:37:47 smithi092 ceph-mon[29572]: osdmap e304: 8 total, 8 up, 8 in 2024-03-11T21:37:48.111 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:37:47 smithi092 ceph-mon[37563]: purged_snaps scrub starts 2024-03-11T21:37:48.111 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:37:47 smithi092 ceph-mon[37563]: purged_snaps scrub ok 2024-03-11T21:37:48.111 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:37:47 smithi092 ceph-mon[37563]: osdmap e304: 8 total, 8 up, 8 in 2024-03-11T21:37:48.111 INFO:journalctl@ceph.mgr.y.smithi092.stdout:Mar 11 21:37:47 smithi092 ceph-00db0016-dfed-11ee-95c8-87774f69a715-mgr-y[29801]: ::ffff:172.21.15.97 - - [11/Mar/2024:21:37:47] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-11T21:37:48.176 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:37:48.219 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:37:48.220 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:37:48.220 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:37:48.321 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:37:48.366 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:37:48.366 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:37:48.366 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:37:48.467 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:37:48.500 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:37:48.501 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:37:48.501 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:37:48.571 INFO:teuthology.orchestra.run.smithi092.stderr:Inferring config /var/lib/ceph/00db0016-dfed-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T21:37:48.601 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:37:48.671 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:37:48.672 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:37:48.673 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:37:48.774 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:37:48.807 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:37:48.808 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:37:48.809 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:37:48.909 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:37:48.951 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:37:48.951 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:37:48.951 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:37:49.052 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:37:49.074 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:37:48 smithi092 ceph-mon[29572]: pgmap v563: 148 pgs: 5 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+degraded+remapped, 140 active+clean; 6.4 GiB data, 21 GiB used, 695 GiB / 715 GiB avail; 853 B/s rd, 0 B/s wr, 747 op/s; 31693/312759 objects degraded (10.133%); 7.8 MiB/s, 124 objects/s recovering 2024-03-11T21:37:49.074 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:37:48 smithi092 ceph-mon[29572]: Health check update: Degraded data redundancy: 31693/312759 objects degraded (10.133%), 8 pgs degraded, 8 pgs undersized (PG_DEGRADED) 2024-03-11T21:37:49.074 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:37:48 smithi092 ceph-mon[37563]: pgmap v563: 148 pgs: 5 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+degraded+remapped, 140 active+clean; 6.4 GiB data, 21 GiB used, 695 GiB / 715 GiB avail; 853 B/s rd, 0 B/s wr, 747 op/s; 31693/312759 objects degraded (10.133%); 7.8 MiB/s, 124 objects/s recovering 2024-03-11T21:37:49.074 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:37:48 smithi092 ceph-mon[37563]: Health check update: Degraded data redundancy: 31693/312759 objects degraded (10.133%), 8 pgs degraded, 8 pgs undersized (PG_DEGRADED) 2024-03-11T21:37:49.076 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:37:48 smithi097 ceph-mon[33317]: pgmap v563: 148 pgs: 5 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+degraded+remapped, 140 active+clean; 6.4 GiB data, 21 GiB used, 695 GiB / 715 GiB avail; 853 B/s rd, 0 B/s wr, 747 op/s; 31693/312759 objects degraded (10.133%); 7.8 MiB/s, 124 objects/s recovering 2024-03-11T21:37:49.076 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:37:48 smithi097 ceph-mon[33317]: Health check update: Degraded data redundancy: 31693/312759 objects degraded (10.133%), 8 pgs degraded, 8 pgs undersized (PG_DEGRADED) 2024-03-11T21:37:49.085 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:37:49.086 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:37:49.086 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:37:49.187 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:37:49.224 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:37:49.224 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:37:49.225 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:37:49.325 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:37:49.365 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:37:49.365 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:37:49.366 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:37:49.469 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:37:49.512 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:37:49.513 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:37:49.513 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:37:49.614 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:37:49.643 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:37:49.644 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:37:49.644 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:37:49.745 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:37:49.772 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:37:49.773 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:37:49.774 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:37:49.830 INFO:teuthology.orchestra.run.smithi092.stderr:reweighted osd.5 to 1 (10000) 2024-03-11T21:37:49.840 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:37:49 smithi092 ceph-mon[29572]: from='client.? v1:172.21.15.92:0/2261708935' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]: dispatch 2024-03-11T21:37:49.841 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:37:49 smithi092 ceph-mon[29572]: from='client.35390 ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]: dispatch 2024-03-11T21:37:49.841 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:37:49 smithi092 ceph-mon[37563]: from='client.? v1:172.21.15.92:0/2261708935' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]: dispatch 2024-03-11T21:37:49.841 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:37:49 smithi092 ceph-mon[37563]: from='client.35390 ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]: dispatch 2024-03-11T21:37:49.853 DEBUG:teuthology.orchestra.run.smithi092:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid 00db0016-dfed-11ee-95c8-87774f69a715 -- ceph osd unset nodeep-scrub 2024-03-11T21:37:49.875 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:37:49.941 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:37:49.942 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:37:49.942 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:37:50.043 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:37:50.077 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:37:49 smithi097 ceph-mon[33317]: from='client.? v1:172.21.15.92:0/2261708935' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]: dispatch 2024-03-11T21:37:50.077 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:37:49 smithi097 ceph-mon[33317]: from='client.35390 ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]: dispatch 2024-03-11T21:37:50.098 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:37:50.099 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:37:50.100 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:37:50.201 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:37:50.233 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:37:50.233 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:37:50.234 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:37:50.334 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:37:50.374 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:37:50.375 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:37:50.375 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:37:50.476 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:37:50.527 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:37:50.527 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:37:50.527 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:37:50.628 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:37:50.666 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:37:50.666 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:37:50.667 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:37:50.768 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:37:50.794 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:37:50.795 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:37:50.796 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:37:50.861 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:37:50 smithi092 ceph-mon[29572]: pgmap v564: 148 pgs: 5 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+degraded+remapped, 140 active+clean; 6.1 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 853 B/s rd, 0 B/s wr, 855 op/s; 31289/301434 objects degraded (10.380%); 6.2 MiB/s, 98 objects/s recovering 2024-03-11T21:37:50.861 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:37:50 smithi092 ceph-mon[29572]: from='client.35390 ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]': finished 2024-03-11T21:37:50.861 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:37:50 smithi092 ceph-mon[29572]: osdmap e305: 8 total, 8 up, 8 in 2024-03-11T21:37:50.861 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:37:50 smithi092 ceph-mon[37563]: pgmap v564: 148 pgs: 5 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+degraded+remapped, 140 active+clean; 6.1 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 853 B/s rd, 0 B/s wr, 855 op/s; 31289/301434 objects degraded (10.380%); 6.2 MiB/s, 98 objects/s recovering 2024-03-11T21:37:50.861 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:37:50 smithi092 ceph-mon[37563]: from='client.35390 ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]': finished 2024-03-11T21:37:50.861 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:37:50 smithi092 ceph-mon[37563]: osdmap e305: 8 total, 8 up, 8 in 2024-03-11T21:37:50.897 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:37:50.946 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:37:50.947 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:37:50.947 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:37:50.967 DEBUG:teuthology.orchestra.run.smithi092:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid 00db0016-dfed-11ee-95c8-87774f69a715 -- ceph osd reweight 6 1 2024-03-11T21:37:51.048 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:37:51.076 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:37:50 smithi097 ceph-mon[33317]: pgmap v564: 148 pgs: 5 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+degraded+remapped, 140 active+clean; 6.1 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 853 B/s rd, 0 B/s wr, 855 op/s; 31289/301434 objects degraded (10.380%); 6.2 MiB/s, 98 objects/s recovering 2024-03-11T21:37:51.077 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:37:50 smithi097 ceph-mon[33317]: from='client.35390 ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]': finished 2024-03-11T21:37:51.077 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:37:50 smithi097 ceph-mon[33317]: osdmap e305: 8 total, 8 up, 8 in 2024-03-11T21:37:51.096 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:37:51.097 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:37:51.098 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:37:51.198 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:37:51.230 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:37:51.230 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:37:51.231 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:37:51.332 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:37:51.368 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:37:51.369 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:37:51.369 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:37:51.470 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:37:51.507 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:37:51.508 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:37:51.508 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:37:51.520 INFO:teuthology.orchestra.run.smithi092.stderr:Inferring config /var/lib/ceph/00db0016-dfed-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T21:37:51.609 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:37:51.633 INFO:teuthology.orchestra.run.smithi092.stdout:osd.0: {} 2024-03-11T21:37:51.662 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:37:51.665 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:37:51.665 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:37:51.675 INFO:teuthology.orchestra.run.smithi092.stdout:osd.1: {} 2024-03-11T21:37:51.711 INFO:teuthology.orchestra.run.smithi092.stderr:Inferring config /var/lib/ceph/00db0016-dfed-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T21:37:51.711 INFO:teuthology.orchestra.run.smithi092.stdout:osd.2: {} 2024-03-11T21:37:51.748 INFO:teuthology.orchestra.run.smithi092.stdout:osd.3: {} 2024-03-11T21:37:51.766 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:37:51.775 INFO:teuthology.orchestra.run.smithi092.stdout:osd.4: {} 2024-03-11T21:37:51.813 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:37:51.813 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:37:51.814 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:37:51.818 INFO:teuthology.orchestra.run.smithi092.stdout:osd.5: {} 2024-03-11T21:37:51.861 INFO:teuthology.orchestra.run.smithi092.stdout:osd.6: {} 2024-03-11T21:37:51.902 INFO:teuthology.orchestra.run.smithi092.stdout:osd.7: {} 2024-03-11T21:37:51.906 INFO:teuthology.orchestra.run.smithi092.stderr:osd.0: osd_enable_op_tracker = '' 2024-03-11T21:37:51.906 INFO:teuthology.orchestra.run.smithi092.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-11T21:37:51.906 INFO:teuthology.orchestra.run.smithi092.stderr:osd.2: osd_enable_op_tracker = '' 2024-03-11T21:37:51.907 INFO:teuthology.orchestra.run.smithi092.stderr:osd.3: osd_enable_op_tracker = '' 2024-03-11T21:37:51.907 INFO:teuthology.orchestra.run.smithi092.stderr:osd.4: osd_enable_op_tracker = '' 2024-03-11T21:37:51.907 INFO:teuthology.orchestra.run.smithi092.stderr:osd.5: osd_enable_op_tracker = '' 2024-03-11T21:37:51.907 INFO:teuthology.orchestra.run.smithi092.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-11T21:37:51.907 INFO:teuthology.orchestra.run.smithi092.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-11T21:37:51.914 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:37:51.962 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:37:51.963 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:37:51.963 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:37:52.064 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:37:52.067 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:37:51 smithi092 ceph-mon[29572]: from='client.? v1:172.21.15.92:0/3289624770' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-11T21:37:52.067 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:37:51 smithi092 ceph-mon[37563]: from='client.? v1:172.21.15.92:0/3289624770' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-11T21:37:52.077 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:37:51 smithi097 ceph-mon[33317]: from='client.? v1:172.21.15.92:0/3289624770' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-11T21:37:52.101 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:37:52.101 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:37:52.102 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:37:52.202 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:37:52.236 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:37:52.237 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:37:52.237 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:37:52.339 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:37:52.373 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:37:52.374 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:37:52.374 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:37:52.475 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:37:52.507 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:37:52.507 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:37:52.508 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:37:52.609 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:37:52.642 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:37:52.643 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:37:52.643 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:37:52.744 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:37:52.785 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:37:52.786 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:37:52.786 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:37:52.821 INFO:journalctl@ceph.iscsi.iscsi.a.smithi097.stdout:Mar 11 21:37:52 smithi097 ceph-00db0016-dfed-11ee-95c8-87774f69a715-iscsi-iscsi-a[60943]: debug there is no tcmu-runner data available 2024-03-11T21:37:52.887 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:37:52.927 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:37:52.927 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:37:52.927 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:37:53.028 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:37:53.066 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:37:53.066 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:37:53.066 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:37:53.076 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:37:52 smithi097 ceph-mon[33317]: pgmap v566: 148 pgs: 5 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+degraded+remapped, 140 active+clean; 6.1 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 852 B/s rd, 0 B/s wr, 673 op/s; 31289/301434 objects degraded (10.380%); 6.2 MiB/s, 98 objects/s recovering 2024-03-11T21:37:53.076 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:37:52 smithi097 ceph-mon[33317]: from='mgr.14607 v1:172.21.15.92:0/1702717597' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T21:37:53.111 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:37:52 smithi092 ceph-mon[29572]: pgmap v566: 148 pgs: 5 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+degraded+remapped, 140 active+clean; 6.1 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 852 B/s rd, 0 B/s wr, 673 op/s; 31289/301434 objects degraded (10.380%); 6.2 MiB/s, 98 objects/s recovering 2024-03-11T21:37:53.111 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:37:52 smithi092 ceph-mon[29572]: from='mgr.14607 v1:172.21.15.92:0/1702717597' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T21:37:53.111 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:37:52 smithi092 ceph-mon[37563]: pgmap v566: 148 pgs: 5 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+degraded+remapped, 140 active+clean; 6.1 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 852 B/s rd, 0 B/s wr, 673 op/s; 31289/301434 objects degraded (10.380%); 6.2 MiB/s, 98 objects/s recovering 2024-03-11T21:37:53.111 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:37:52 smithi092 ceph-mon[37563]: from='mgr.14607 v1:172.21.15.92:0/1702717597' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T21:37:53.168 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:37:53.210 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:37:53.211 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:37:53.211 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:37:53.312 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:37:53.356 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:37:53.357 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:37:53.357 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:37:53.388 INFO:teuthology.orchestra.run.smithi097.stdout:{ 2024-03-11T21:37:53.389 INFO:teuthology.orchestra.run.smithi097.stdout: "ops": [ 2024-03-11T21:37:53.389 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:37:53.389 INFO:teuthology.orchestra.run.smithi097.stdout: "description": "osd_op(client.25282.0:136898 9.c 9:3df5c01d:::benchmark_data_smithi092_130291_object20514:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e306)", 2024-03-11T21:37:53.389 INFO:teuthology.orchestra.run.smithi097.stdout: "initiated_at": "2024-03-11T21:37:53.403091+0000", 2024-03-11T21:37:53.389 INFO:teuthology.orchestra.run.smithi097.stdout: "age": 0.001826002, 2024-03-11T21:37:53.389 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.001857168, 2024-03-11T21:37:53.389 INFO:teuthology.orchestra.run.smithi097.stdout: "type_data": { 2024-03-11T21:37:53.389 INFO:teuthology.orchestra.run.smithi097.stdout: "flag_point": "waiting for sub ops", 2024-03-11T21:37:53.389 INFO:teuthology.orchestra.run.smithi097.stdout: "client_info": { 2024-03-11T21:37:53.389 INFO:teuthology.orchestra.run.smithi097.stdout: "client": "client.25282", 2024-03-11T21:37:53.389 INFO:teuthology.orchestra.run.smithi097.stdout: "client_addr": "v1:172.21.15.92:0/2489519821", 2024-03-11T21:37:53.389 INFO:teuthology.orchestra.run.smithi097.stdout: "tid": 136898 2024-03-11T21:37:53.390 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:37:53.390 INFO:teuthology.orchestra.run.smithi097.stdout: "events": [ 2024-03-11T21:37:53.390 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:37:53.390 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "initiated", 2024-03-11T21:37:53.390 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:37:53.403091+0000", 2024-03-11T21:37:53.390 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-03-11T21:37:53.390 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:37:53.390 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:37:53.390 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "header_read", 2024-03-11T21:37:53.390 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:37:53.403091+0000", 2024-03-11T21:37:53.390 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-03-11T21:37:53.390 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:37:53.390 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:37:53.390 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "throttled", 2024-03-11T21:37:53.391 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:37:53.403093+0000", 2024-03-11T21:37:53.391 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 2.1289999999999999e-06 2024-03-11T21:37:53.391 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:37:53.391 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:37:53.391 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "all_read", 2024-03-11T21:37:53.391 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:37:53.403101+0000", 2024-03-11T21:37:53.391 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 7.3209999999999999e-06 2024-03-11T21:37:53.391 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:37:53.391 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:37:53.391 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "dispatched", 2024-03-11T21:37:53.391 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:37:53.403128+0000", 2024-03-11T21:37:53.391 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 2.7137000000000001e-05 2024-03-11T21:37:53.391 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:37:53.391 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:37:53.392 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "queued_for_pg", 2024-03-11T21:37:53.392 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:37:53.403138+0000", 2024-03-11T21:37:53.392 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.0223e-05 2024-03-11T21:37:53.392 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:37:53.392 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:37:53.392 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "reached_pg", 2024-03-11T21:37:53.392 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:37:53.403202+0000", 2024-03-11T21:37:53.392 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 6.4416000000000001e-05 2024-03-11T21:37:53.392 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:37:53.392 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:37:53.392 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "started", 2024-03-11T21:37:53.392 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:37:53.403345+0000", 2024-03-11T21:37:53.392 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.000142881 2024-03-11T21:37:53.392 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:37:53.392 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:37:53.393 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "waiting for subops from 3,7", 2024-03-11T21:37:53.393 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:37:53.403562+0000", 2024-03-11T21:37:53.393 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.000217128 2024-03-11T21:37:53.393 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-03-11T21:37:53.393 INFO:teuthology.orchestra.run.smithi097.stdout: ] 2024-03-11T21:37:53.393 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-03-11T21:37:53.393 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:37:53.393 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:37:53.393 INFO:teuthology.orchestra.run.smithi097.stdout: "description": "osd_repop(client.25282.0:136896 9.4 e306/278)", 2024-03-11T21:37:53.393 INFO:teuthology.orchestra.run.smithi097.stdout: "initiated_at": "2024-03-11T21:37:53.403394+0000", 2024-03-11T21:37:53.394 INFO:teuthology.orchestra.run.smithi097.stdout: "age": 0.0015230339999999999, 2024-03-11T21:37:53.394 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.0016311889999999999, 2024-03-11T21:37:53.394 INFO:teuthology.orchestra.run.smithi097.stdout: "type_data": { 2024-03-11T21:37:53.394 INFO:teuthology.orchestra.run.smithi097.stdout: "flag_point": "queued for pg", 2024-03-11T21:37:53.394 INFO:teuthology.orchestra.run.smithi097.stdout: "events": [ 2024-03-11T21:37:53.394 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:37:53.394 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "initiated", 2024-03-11T21:37:53.394 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:37:53.403394+0000", 2024-03-11T21:37:53.394 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-03-11T21:37:53.394 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:37:53.394 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:37:53.394 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "header_read", 2024-03-11T21:37:53.395 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:37:53.403394+0000", 2024-03-11T21:37:53.395 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-03-11T21:37:53.395 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:37:53.395 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:37:53.395 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "throttled", 2024-03-11T21:37:53.395 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:37:53.403395+0000", 2024-03-11T21:37:53.396 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.164e-06 2024-03-11T21:37:53.396 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:37:53.396 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:37:53.396 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "all_read", 2024-03-11T21:37:53.396 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:37:53.403405+0000", 2024-03-11T21:37:53.397 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 9.329e-06 2024-03-11T21:37:53.397 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:37:53.397 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:37:53.397 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "dispatched", 2024-03-11T21:37:53.397 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:37:53.403412+0000", 2024-03-11T21:37:53.397 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 7.729e-06 2024-03-11T21:37:53.398 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:37:53.398 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:37:53.398 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "queued_for_pg", 2024-03-11T21:37:53.398 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:37:53.403419+0000", 2024-03-11T21:37:53.398 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 6.3770000000000001e-06 2024-03-11T21:37:53.398 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-03-11T21:37:53.398 INFO:teuthology.orchestra.run.smithi097.stdout: ] 2024-03-11T21:37:53.398 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-03-11T21:37:53.398 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:37:53.398 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:37:53.399 INFO:teuthology.orchestra.run.smithi097.stdout: "description": "osd_repop(client.25282.0:136897 9.e e306/259 9:7a86879b:::benchmark_data_smithi092_130291_object20513:head v 306'8414, mlcod=305'8412)", 2024-03-11T21:37:53.399 INFO:teuthology.orchestra.run.smithi097.stdout: "initiated_at": "2024-03-11T21:37:53.404572+0000", 2024-03-11T21:37:53.399 INFO:teuthology.orchestra.run.smithi097.stdout: "age": 0.000345645, 2024-03-11T21:37:53.399 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.00050421999999999999, 2024-03-11T21:37:53.399 INFO:teuthology.orchestra.run.smithi097.stdout: "type_data": { 2024-03-11T21:37:53.399 INFO:teuthology.orchestra.run.smithi097.stdout: "flag_point": "started", 2024-03-11T21:37:53.399 INFO:teuthology.orchestra.run.smithi097.stdout: "events": [ 2024-03-11T21:37:53.399 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:37:53.399 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "initiated", 2024-03-11T21:37:53.399 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:37:53.404572+0000", 2024-03-11T21:37:53.399 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-03-11T21:37:53.400 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:37:53.400 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:37:53.400 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "header_read", 2024-03-11T21:37:53.400 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:37:53.404572+0000", 2024-03-11T21:37:53.400 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-03-11T21:37:53.400 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:37:53.400 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:37:53.400 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "throttled", 2024-03-11T21:37:53.400 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:37:53.404573+0000", 2024-03-11T21:37:53.400 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.5260000000000001e-06 2024-03-11T21:37:53.400 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:37:53.400 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:37:53.401 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "all_read", 2024-03-11T21:37:53.401 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:37:53.404580+0000", 2024-03-11T21:37:53.401 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 7.0249999999999997e-06 2024-03-11T21:37:53.401 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:37:53.401 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:37:53.401 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "dispatched", 2024-03-11T21:37:53.402 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:37:53.404593+0000", 2024-03-11T21:37:53.402 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.3329000000000001e-05 2024-03-11T21:37:53.402 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:37:53.402 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:37:53.402 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "queued_for_pg", 2024-03-11T21:37:53.402 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:37:53.404601+0000", 2024-03-11T21:37:53.403 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 7.6760000000000007e-06 2024-03-11T21:37:53.403 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:37:53.403 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:37:53.403 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "reached_pg", 2024-03-11T21:37:53.403 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:37:53.404674+0000", 2024-03-11T21:37:53.403 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 7.2760000000000001e-05 2024-03-11T21:37:53.403 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:37:53.403 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:37:53.404 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "started", 2024-03-11T21:37:53.404 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:37:53.404693+0000", 2024-03-11T21:37:53.404 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.9454e-05 2024-03-11T21:37:53.404 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-03-11T21:37:53.404 INFO:teuthology.orchestra.run.smithi097.stdout: ] 2024-03-11T21:37:53.404 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-03-11T21:37:53.404 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-03-11T21:37:53.404 INFO:teuthology.orchestra.run.smithi097.stdout: ], 2024-03-11T21:37:53.404 INFO:teuthology.orchestra.run.smithi097.stdout: "num_ops": 3 2024-03-11T21:37:53.404 INFO:teuthology.orchestra.run.smithi097.stdout:} 2024-03-11T21:37:53.458 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:37:53.505 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:37:53.505 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:37:53.506 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:37:53.607 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:37:53.638 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:37:53.639 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:37:53.639 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:37:53.740 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:37:53.768 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:37:53.769 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:37:53.769 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:37:53.870 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:37:53.908 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:37:53.909 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:37:53.909 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:37:54.010 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:37:54.042 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:37:54.043 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:37:54.043 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:37:54.111 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:37:53 smithi092 ceph-mon[29572]: from='client.24469 v1:172.21.15.97:0/404208310' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:37:54.111 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:37:53 smithi092 ceph-mon[29572]: Health check update: Degraded data redundancy: 31289/301434 objects degraded (10.380%), 8 pgs degraded, 8 pgs undersized (PG_DEGRADED) 2024-03-11T21:37:54.111 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:37:53 smithi092 ceph-mon[29572]: osdmap e306: 8 total, 8 up, 8 in 2024-03-11T21:37:54.112 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:37:53 smithi092 ceph-mon[37563]: from='client.24469 v1:172.21.15.97:0/404208310' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:37:54.112 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:37:53 smithi092 ceph-mon[37563]: Health check update: Degraded data redundancy: 31289/301434 objects degraded (10.380%), 8 pgs degraded, 8 pgs undersized (PG_DEGRADED) 2024-03-11T21:37:54.112 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:37:53 smithi092 ceph-mon[37563]: osdmap e306: 8 total, 8 up, 8 in 2024-03-11T21:37:54.142 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:37:53 smithi097 ceph-mon[33317]: from='client.24469 v1:172.21.15.97:0/404208310' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:37:54.143 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:37:53 smithi097 ceph-mon[33317]: Health check update: Degraded data redundancy: 31289/301434 objects degraded (10.380%), 8 pgs degraded, 8 pgs undersized (PG_DEGRADED) 2024-03-11T21:37:54.143 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:37:53 smithi097 ceph-mon[33317]: osdmap e306: 8 total, 8 up, 8 in 2024-03-11T21:37:54.144 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:37:54.177 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:37:54.178 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:37:54.178 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:37:54.189 DEBUG:teuthology.orchestra.run.smithi097:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid 00db0016-dfed-11ee-95c8-87774f69a715 -- ceph daemon osd.4 dump_blocked_ops 2024-03-11T21:37:54.279 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:37:54.302 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:37:54.302 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:37:54.303 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:37:54.404 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:37:54.447 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:37:54.447 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:37:54.447 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:37:54.548 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:37:54.571 INFO:teuthology.orchestra.run.smithi097.stderr:Inferring config /var/lib/ceph/00db0016-dfed-11ee-95c8-87774f69a715/mon.b/config 2024-03-11T21:37:54.589 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:37:54.589 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:37:54.590 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:37:54.691 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:37:54.728 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:37:54.729 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:37:54.730 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:37:54.830 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:37:54.868 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:37:54.869 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:37:54.869 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:37:54.969 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:37:55.006 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:37:55.007 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:37:55.007 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:37:55.077 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:37:54 smithi097 ceph-mon[33317]: pgmap v567: 148 pgs: 5 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+degraded+remapped, 140 active+clean; 5.9 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 1.3 KiB/s rd, 0 B/s wr, 918 op/s; 30670/292968 objects degraded (10.469%); 10 MiB/s, 161 objects/s recovering 2024-03-11T21:37:55.108 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:37:55.111 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:37:54 smithi092 ceph-mon[29572]: pgmap v567: 148 pgs: 5 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+degraded+remapped, 140 active+clean; 5.9 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 1.3 KiB/s rd, 0 B/s wr, 918 op/s; 30670/292968 objects degraded (10.469%); 10 MiB/s, 161 objects/s recovering 2024-03-11T21:37:55.111 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:37:54 smithi092 ceph-mon[37563]: pgmap v567: 148 pgs: 5 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+degraded+remapped, 140 active+clean; 5.9 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 1.3 KiB/s rd, 0 B/s wr, 918 op/s; 30670/292968 objects degraded (10.469%); 10 MiB/s, 161 objects/s recovering 2024-03-11T21:37:55.143 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:37:55.144 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:37:55.144 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:37:55.245 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:37:55.296 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:37:55.296 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:37:55.297 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:37:55.398 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:37:55.656 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:37:55.657 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:37:55.658 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:37:55.758 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:37:55.791 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:37:55.792 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:37:55.792 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:37:55.893 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:37:55.927 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:37:55.928 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:37:55.928 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:37:56.029 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:37:56.069 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:37:56.070 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:37:56.070 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:37:56.171 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:37:56.204 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:37:56.204 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:37:56.204 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:37:56.305 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:37:56.337 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:37:56.338 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:37:56.338 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:37:56.439 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:37:56.483 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:37:56.484 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:37:56.484 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:37:56.585 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:37:56.621 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:37:56.621 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:37:56.622 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:37:56.723 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:37:56.762 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:37:56.762 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:37:56.763 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:37:56.827 DEBUG:teuthology.orchestra.run.smithi092:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid 00db0016-dfed-11ee-95c8-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-03-11T21:37:56.864 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:37:56.889 INFO:teuthology.orchestra.run.smithi092.stderr:reweighted osd.6 to 1 (10000) 2024-03-11T21:37:56.921 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:37:56.922 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:37:56.922 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:37:57.023 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:37:57.060 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:37:57.060 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:37:57.061 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:37:57.076 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:37:56 smithi097 ceph-mon[33317]: pgmap v569: 148 pgs: 5 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+degraded+remapped, 140 active+clean; 5.9 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 940 op/s; 30670/290202 objects degraded (10.569%); 8.0 MiB/s, 127 objects/s recovering 2024-03-11T21:37:57.077 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:37:56 smithi097 ceph-mon[33317]: from='client.? v1:172.21.15.92:0/2906589109' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]: dispatch 2024-03-11T21:37:57.111 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:37:56 smithi092 ceph-mon[37563]: pgmap v569: 148 pgs: 5 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+degraded+remapped, 140 active+clean; 5.9 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 940 op/s; 30670/290202 objects degraded (10.569%); 8.0 MiB/s, 127 objects/s recovering 2024-03-11T21:37:57.111 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:37:56 smithi092 ceph-mon[37563]: from='client.? v1:172.21.15.92:0/2906589109' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]: dispatch 2024-03-11T21:37:57.111 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:37:56 smithi092 ceph-mon[29572]: pgmap v569: 148 pgs: 5 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+degraded+remapped, 140 active+clean; 5.9 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 940 op/s; 30670/290202 objects degraded (10.569%); 8.0 MiB/s, 127 objects/s recovering 2024-03-11T21:37:57.111 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:37:56 smithi092 ceph-mon[29572]: from='client.? v1:172.21.15.92:0/2906589109' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]: dispatch 2024-03-11T21:37:57.161 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:37:57.204 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:37:57.204 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:37:57.205 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:37:57.306 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:37:57.315 INFO:teuthology.orchestra.run.smithi097.stdout:{ 2024-03-11T21:37:57.315 INFO:teuthology.orchestra.run.smithi097.stdout: "ops": [], 2024-03-11T21:37:57.315 INFO:teuthology.orchestra.run.smithi097.stdout: "complaint_time": 30, 2024-03-11T21:37:57.315 INFO:teuthology.orchestra.run.smithi097.stdout: "num_blocked_ops": 0 2024-03-11T21:37:57.315 INFO:teuthology.orchestra.run.smithi097.stdout:} 2024-03-11T21:37:57.336 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:37:57.337 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:37:57.337 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:37:57.438 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:37:57.462 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:37:57.462 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:37:57.463 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:37:57.563 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:37:57.604 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:37:57.604 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:37:57.605 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:37:57.706 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:37:57.741 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:37:57.741 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:37:57.742 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:37:57.843 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:37:57.872 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:37:57.872 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:37:57.872 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:37:57.908 INFO:teuthology.orchestra.run.smithi092.stderr:nodeep-scrub is unset 2024-03-11T21:37:57.973 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:37:58.006 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:37:58.007 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:37:58.007 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:37:58.108 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:37:58.140 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:37:58.141 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:37:58.141 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:37:58.242 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:37:58.325 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:37:57 smithi092 ceph-mon[29572]: from='client.? v1:172.21.15.92:0/2906589109' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]': finished 2024-03-11T21:37:58.325 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:37:57 smithi092 ceph-mon[29572]: osdmap e307: 8 total, 8 up, 8 in 2024-03-11T21:37:58.325 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:37:57 smithi092 ceph-mon[29572]: from='client.? v1:172.21.15.92:0/4281593102' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-11T21:37:58.325 INFO:journalctl@ceph.mgr.y.smithi092.stdout:Mar 11 21:37:57 smithi092 ceph-00db0016-dfed-11ee-95c8-87774f69a715-mgr-y[29801]: ::ffff:172.21.15.97 - - [11/Mar/2024:21:37:57] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-11T21:37:58.326 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:37:57 smithi092 ceph-mon[37563]: from='client.? v1:172.21.15.92:0/2906589109' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]': finished 2024-03-11T21:37:58.326 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:37:57 smithi092 ceph-mon[37563]: osdmap e307: 8 total, 8 up, 8 in 2024-03-11T21:37:58.326 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:37:57 smithi092 ceph-mon[37563]: from='client.? v1:172.21.15.92:0/4281593102' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-11T21:37:58.326 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:37:57 smithi097 ceph-mon[33317]: from='client.? v1:172.21.15.92:0/2906589109' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]': finished 2024-03-11T21:37:58.326 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:37:57 smithi097 ceph-mon[33317]: osdmap e307: 8 total, 8 up, 8 in 2024-03-11T21:37:58.327 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:37:57 smithi097 ceph-mon[33317]: from='client.? v1:172.21.15.92:0/4281593102' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-11T21:37:58.376 DEBUG:teuthology.orchestra.run.smithi092:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid 00db0016-dfed-11ee-95c8-87774f69a715 -- ceph osd reweight 7 1 2024-03-11T21:37:58.458 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:37:58.459 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:37:58.459 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:37:58.560 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:37:58.590 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:37:58.591 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:37:58.591 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:37:58.692 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:37:58.750 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:37:58.750 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:37:58.751 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:37:58.852 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:37:58.893 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:37:58.894 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:37:58.895 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:37:58.995 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:37:59.031 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:37:59.032 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:37:59.032 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:37:59.039 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:37:58 smithi092 ceph-mon[29572]: pgmap v571: 148 pgs: 5 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+degraded+remapped, 140 active+clean; 5.9 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 689 B/s rd, 0 B/s wr, 539 op/s; 30470/289431 objects degraded (10.528%); 6.9 MiB/s, 110 objects/s recovering 2024-03-11T21:37:59.039 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:37:58 smithi092 ceph-mon[29572]: purged_snaps scrub starts 2024-03-11T21:37:59.039 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:37:58 smithi092 ceph-mon[29572]: purged_snaps scrub ok 2024-03-11T21:37:59.039 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:37:58 smithi092 ceph-mon[29572]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-03-11T21:37:59.039 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:37:58 smithi092 ceph-mon[29572]: from='client.? v1:172.21.15.92:0/4281593102' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-03-11T21:37:59.039 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:37:58 smithi092 ceph-mon[29572]: osdmap e308: 8 total, 8 up, 8 in 2024-03-11T21:37:59.039 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:37:58 smithi092 ceph-mon[29572]: Health check update: Degraded data redundancy: 30470/289431 objects degraded (10.528%), 8 pgs degraded, 8 pgs undersized (PG_DEGRADED) 2024-03-11T21:37:59.041 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:37:58 smithi092 ceph-mon[37563]: pgmap v571: 148 pgs: 5 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+degraded+remapped, 140 active+clean; 5.9 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 689 B/s rd, 0 B/s wr, 539 op/s; 30470/289431 objects degraded (10.528%); 6.9 MiB/s, 110 objects/s recovering 2024-03-11T21:37:59.041 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:37:58 smithi092 ceph-mon[37563]: purged_snaps scrub starts 2024-03-11T21:37:59.041 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:37:58 smithi092 ceph-mon[37563]: purged_snaps scrub ok 2024-03-11T21:37:59.041 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:37:58 smithi092 ceph-mon[37563]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-03-11T21:37:59.041 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:37:58 smithi092 ceph-mon[37563]: from='client.? v1:172.21.15.92:0/4281593102' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-03-11T21:37:59.041 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:37:58 smithi092 ceph-mon[37563]: osdmap e308: 8 total, 8 up, 8 in 2024-03-11T21:37:59.041 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:37:58 smithi092 ceph-mon[37563]: Health check update: Degraded data redundancy: 30470/289431 objects degraded (10.528%), 8 pgs degraded, 8 pgs undersized (PG_DEGRADED) 2024-03-11T21:37:59.118 INFO:teuthology.orchestra.run.smithi092.stderr:Inferring config /var/lib/ceph/00db0016-dfed-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T21:37:59.133 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:37:59.199 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:37:59.200 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:37:59.200 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:37:59.301 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:37:59.327 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:37:58 smithi097 ceph-mon[33317]: pgmap v571: 148 pgs: 5 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+degraded+remapped, 140 active+clean; 5.9 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 689 B/s rd, 0 B/s wr, 539 op/s; 30470/289431 objects degraded (10.528%); 6.9 MiB/s, 110 objects/s recovering 2024-03-11T21:37:59.327 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:37:58 smithi097 ceph-mon[33317]: purged_snaps scrub starts 2024-03-11T21:37:59.327 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:37:58 smithi097 ceph-mon[33317]: purged_snaps scrub ok 2024-03-11T21:37:59.327 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:37:58 smithi097 ceph-mon[33317]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-03-11T21:37:59.327 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:37:58 smithi097 ceph-mon[33317]: from='client.? v1:172.21.15.92:0/4281593102' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-03-11T21:37:59.327 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:37:58 smithi097 ceph-mon[33317]: osdmap e308: 8 total, 8 up, 8 in 2024-03-11T21:37:59.327 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:37:58 smithi097 ceph-mon[33317]: Health check update: Degraded data redundancy: 30470/289431 objects degraded (10.528%), 8 pgs degraded, 8 pgs undersized (PG_DEGRADED) 2024-03-11T21:37:59.339 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:37:59.340 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:37:59.340 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:37:59.440 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:37:59.473 INFO:teuthology.orchestra.run.smithi092.stderr:Inferring config /var/lib/ceph/00db0016-dfed-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T21:37:59.495 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:37:59.495 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:37:59.495 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:37:59.596 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:37:59.627 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:37:59.628 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:37:59.629 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:37:59.718 DEBUG:teuthology.orchestra.run.smithi097:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid 00db0016-dfed-11ee-95c8-87774f69a715 -- ceph daemon osd.4 dump_historic_ops 2024-03-11T21:37:59.730 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:37:59.777 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:37:59.778 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:37:59.778 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:37:59.879 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:37:59.916 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:37:59.917 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:37:59.917 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:38:00.018 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:38:00.057 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:38:00.058 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:38:00.058 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:38:00.159 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:38:00.202 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:38:00.203 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:38:00.203 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:38:00.210 INFO:teuthology.orchestra.run.smithi097.stderr:Inferring config /var/lib/ceph/00db0016-dfed-11ee-95c8-87774f69a715/mon.b/config 2024-03-11T21:38:00.304 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:38:00.346 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:38:00.346 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:38:00.347 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:38:00.447 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:38:00.483 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:38:00.484 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:38:00.484 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:38:00.585 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:38:00.622 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:38:00.623 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:38:00.623 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:38:00.724 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:38:00.759 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:38:00.759 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:38:00.760 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:38:00.860 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:38:00.892 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:38:00.892 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:38:00.893 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:38:00.993 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:38:01.024 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:38:01.024 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:38:01.025 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:38:01.098 DEBUG:teuthology.orchestra.run.smithi092:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid 00db0016-dfed-11ee-95c8-87774f69a715 -- ceph osd set noscrub 2024-03-11T21:38:01.126 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:38:01.137 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:38:00 smithi092 ceph-mon[37563]: pgmap v573: 148 pgs: 5 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+degraded+remapped, 140 active+clean; 5.6 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 853 B/s rd, 0 B/s wr, 836 op/s; 30075/277920 objects degraded (10.821%); 6.2 MiB/s, 99 objects/s recovering 2024-03-11T21:38:01.137 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:38:00 smithi092 ceph-mon[37563]: purged_snaps scrub starts 2024-03-11T21:38:01.137 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:38:00 smithi092 ceph-mon[37563]: purged_snaps scrub ok 2024-03-11T21:38:01.138 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:38:00 smithi092 ceph-mon[29572]: pgmap v573: 148 pgs: 5 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+degraded+remapped, 140 active+clean; 5.6 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 853 B/s rd, 0 B/s wr, 836 op/s; 30075/277920 objects degraded (10.821%); 6.2 MiB/s, 99 objects/s recovering 2024-03-11T21:38:01.138 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:38:00 smithi092 ceph-mon[29572]: purged_snaps scrub starts 2024-03-11T21:38:01.138 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:38:00 smithi092 ceph-mon[29572]: purged_snaps scrub ok 2024-03-11T21:38:01.160 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:38:01.160 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:38:01.161 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:38:01.262 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:38:01.303 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:38:01.303 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:38:01.304 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:38:01.326 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:38:00 smithi097 ceph-mon[33317]: pgmap v573: 148 pgs: 5 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+degraded+remapped, 140 active+clean; 5.6 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 853 B/s rd, 0 B/s wr, 836 op/s; 30075/277920 objects degraded (10.821%); 6.2 MiB/s, 99 objects/s recovering 2024-03-11T21:38:01.327 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:38:00 smithi097 ceph-mon[33317]: purged_snaps scrub starts 2024-03-11T21:38:01.327 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:38:00 smithi097 ceph-mon[33317]: purged_snaps scrub ok 2024-03-11T21:38:01.405 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:38:01.445 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:38:01.446 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:38:01.447 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:38:01.547 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:38:01.573 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:38:01.574 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:38:01.575 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:38:01.676 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:38:01.711 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:38:01.712 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:38:01.712 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:38:01.813 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:38:01.851 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:38:01.852 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:38:01.852 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:38:01.953 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:38:02.016 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:38:02.017 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:38:02.017 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:38:02.118 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:38:02.148 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:38:02.149 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:38:02.149 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:38:02.168 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:38:01 smithi097 ceph-mon[33317]: osdmap e309: 8 total, 8 up, 8 in 2024-03-11T21:38:02.243 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:38:01 smithi092 ceph-mon[29572]: osdmap e309: 8 total, 8 up, 8 in 2024-03-11T21:38:02.243 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:38:01 smithi092 ceph-mon[37563]: osdmap e309: 8 total, 8 up, 8 in 2024-03-11T21:38:02.250 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:38:02.315 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:38:02.316 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:38:02.316 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:38:02.417 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:38:02.452 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:38:02.452 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:38:02.453 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:38:02.553 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:38:02.586 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:38:02.587 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:38:02.588 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:38:02.689 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:38:02.723 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:38:02.724 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:38:02.724 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:38:02.825 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:38:02.853 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:38:02.854 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:38:02.855 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:38:02.956 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:38:02.997 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:38:02.997 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:38:02.998 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:38:03.055 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:38:02 smithi092 ceph-mon[29572]: pgmap v575: 148 pgs: 5 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+degraded+remapped, 140 active+clean; 5.6 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 853 B/s rd, 0 B/s wr, 682 op/s; 30075/277920 objects degraded (10.821%); 6.2 MiB/s, 99 objects/s recovering 2024-03-11T21:38:03.055 INFO:journalctl@ceph.iscsi.iscsi.a.smithi097.stdout:Mar 11 21:38:02 smithi097 ceph-00db0016-dfed-11ee-95c8-87774f69a715-iscsi-iscsi-a[60943]: debug there is no tcmu-runner data available 2024-03-11T21:38:03.055 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:38:02 smithi092 ceph-mon[37563]: pgmap v575: 148 pgs: 5 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+degraded+remapped, 140 active+clean; 5.6 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 853 B/s rd, 0 B/s wr, 682 op/s; 30075/277920 objects degraded (10.821%); 6.2 MiB/s, 99 objects/s recovering 2024-03-11T21:38:03.055 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:38:02 smithi097 ceph-mon[33317]: pgmap v575: 148 pgs: 5 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+degraded+remapped, 140 active+clean; 5.6 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 853 B/s rd, 0 B/s wr, 682 op/s; 30075/277920 objects degraded (10.821%); 6.2 MiB/s, 99 objects/s recovering 2024-03-11T21:38:03.099 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:38:03.132 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:38:03.132 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:38:03.133 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:38:03.233 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:38:03.266 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:38:03.266 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:38:03.266 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:38:03.367 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:38:03.397 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:38:03.398 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:38:03.398 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:38:03.499 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:38:03.529 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:38:03.530 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:38:03.530 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:38:03.631 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:38:03.660 INFO:teuthology.orchestra.run.smithi092.stdout:osd.0: {} 2024-03-11T21:38:03.670 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:38:03.671 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:38:03.671 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:38:03.707 INFO:teuthology.orchestra.run.smithi092.stdout:osd.1: {} 2024-03-11T21:38:03.746 INFO:teuthology.orchestra.run.smithi092.stdout:osd.2: {} 2024-03-11T21:38:03.771 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:38:03.780 INFO:teuthology.orchestra.run.smithi092.stdout:osd.3: {} 2024-03-11T21:38:03.795 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:38:03.795 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:38:03.796 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:38:03.843 INFO:teuthology.orchestra.run.smithi092.stdout:osd.4: {} 2024-03-11T21:38:03.884 INFO:teuthology.orchestra.run.smithi092.stderr:Inferring config /var/lib/ceph/00db0016-dfed-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T21:38:03.890 INFO:teuthology.orchestra.run.smithi092.stdout:osd.5: {} 2024-03-11T21:38:03.897 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:38:03.926 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:38:03.927 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:38:03.928 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:38:03.934 INFO:teuthology.orchestra.run.smithi092.stdout:osd.6: {} 2024-03-11T21:38:03.981 INFO:teuthology.orchestra.run.smithi092.stdout:osd.7: {} 2024-03-11T21:38:04.005 INFO:teuthology.orchestra.run.smithi092.stderr:osd.0: osd_enable_op_tracker = '' 2024-03-11T21:38:04.006 INFO:teuthology.orchestra.run.smithi092.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-11T21:38:04.006 INFO:teuthology.orchestra.run.smithi092.stderr:osd.2: osd_enable_op_tracker = '' 2024-03-11T21:38:04.006 INFO:teuthology.orchestra.run.smithi092.stderr:osd.3: osd_enable_op_tracker = '' 2024-03-11T21:38:04.006 INFO:teuthology.orchestra.run.smithi092.stderr:osd.4: osd_enable_op_tracker = '' 2024-03-11T21:38:04.006 INFO:teuthology.orchestra.run.smithi092.stderr:osd.5: osd_enable_op_tracker = '' 2024-03-11T21:38:04.006 INFO:teuthology.orchestra.run.smithi092.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-11T21:38:04.006 INFO:teuthology.orchestra.run.smithi092.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-11T21:38:04.028 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:38:04.057 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:38:04.057 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:38:04.058 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:38:04.111 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:38:03 smithi092 ceph-mon[37563]: from='client.24469 v1:172.21.15.97:0/404208310' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:38:04.111 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:38:03 smithi092 ceph-mon[37563]: Health check update: Degraded data redundancy: 30075/277920 objects degraded (10.821%), 8 pgs degraded, 8 pgs undersized (PG_DEGRADED) 2024-03-11T21:38:04.111 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:38:03 smithi092 ceph-mon[37563]: osdmap e310: 8 total, 8 up, 8 in 2024-03-11T21:38:04.111 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:38:03 smithi092 ceph-mon[37563]: purged_snaps scrub starts 2024-03-11T21:38:04.111 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:38:03 smithi092 ceph-mon[37563]: purged_snaps scrub ok 2024-03-11T21:38:04.111 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:38:03 smithi092 ceph-mon[37563]: from='client.? v1:172.21.15.92:0/4278256681' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-11T21:38:04.111 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:38:03 smithi092 ceph-mon[29572]: from='client.24469 v1:172.21.15.97:0/404208310' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:38:04.111 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:38:03 smithi092 ceph-mon[29572]: Health check update: Degraded data redundancy: 30075/277920 objects degraded (10.821%), 8 pgs degraded, 8 pgs undersized (PG_DEGRADED) 2024-03-11T21:38:04.111 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:38:03 smithi092 ceph-mon[29572]: osdmap e310: 8 total, 8 up, 8 in 2024-03-11T21:38:04.111 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:38:03 smithi092 ceph-mon[29572]: purged_snaps scrub starts 2024-03-11T21:38:04.112 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:38:03 smithi092 ceph-mon[29572]: purged_snaps scrub ok 2024-03-11T21:38:04.112 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:38:03 smithi092 ceph-mon[29572]: from='client.? v1:172.21.15.92:0/4278256681' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-11T21:38:04.158 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:38:04.197 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:38:04.197 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:38:04.198 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:38:04.299 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:38:04.321 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:38:03 smithi097 ceph-mon[33317]: from='client.24469 v1:172.21.15.97:0/404208310' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:38:04.321 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:38:03 smithi097 ceph-mon[33317]: Health check update: Degraded data redundancy: 30075/277920 objects degraded (10.821%), 8 pgs degraded, 8 pgs undersized (PG_DEGRADED) 2024-03-11T21:38:04.321 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:38:03 smithi097 ceph-mon[33317]: osdmap e310: 8 total, 8 up, 8 in 2024-03-11T21:38:04.321 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:38:03 smithi097 ceph-mon[33317]: purged_snaps scrub starts 2024-03-11T21:38:04.322 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:38:03 smithi097 ceph-mon[33317]: purged_snaps scrub ok 2024-03-11T21:38:04.322 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:38:03 smithi097 ceph-mon[33317]: from='client.? v1:172.21.15.92:0/4278256681' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-11T21:38:04.339 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:38:04.340 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:38:04.340 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:38:04.348 INFO:teuthology.orchestra.run.smithi092.stderr:reweighted osd.7 to 1 (10000) 2024-03-11T21:38:04.441 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:38:04.481 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:38:04.482 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:38:04.482 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:38:04.583 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:38:04.628 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:38:04.628 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:38:04.628 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:38:04.632 INFO:teuthology.orchestra.run.smithi097.stdout:ERROR: (22) Invalid argument 2024-03-11T21:38:04.632 INFO:teuthology.orchestra.run.smithi097.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-11T21:38:04.729 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:38:04.773 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:38:04.774 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:38:04.774 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:38:04.875 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:38:04.908 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:38:04.908 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:38:04.908 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:38:05.009 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:38:05.053 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:38:05.053 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:38:05.054 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:38:05.155 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:38:05.188 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:38:05.188 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:38:05.189 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:38:05.239 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:38:04 smithi092 ceph-mon[29572]: pgmap v576: 148 pgs: 5 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+degraded+remapped, 140 active+clean; 5.5 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 1.6 KiB/s rd, 0 B/s wr, 1.10k op/s; 29381/269115 objects degraded (10.918%); 13 MiB/s, 202 objects/s recovering 2024-03-11T21:38:05.239 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:38:04 smithi092 ceph-mon[29572]: from='client.? v1:172.21.15.92:0/1374763199' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]: dispatch 2024-03-11T21:38:05.239 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:38:04 smithi092 ceph-mon[29572]: from='client.25423 ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]: dispatch 2024-03-11T21:38:05.239 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:38:04 smithi092 ceph-mon[29572]: from='client.25423 ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]': finished 2024-03-11T21:38:05.240 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:38:04 smithi092 ceph-mon[29572]: osdmap e311: 8 total, 8 up, 8 in 2024-03-11T21:38:05.240 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:38:04 smithi092 ceph-mon[37563]: pgmap v576: 148 pgs: 5 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+degraded+remapped, 140 active+clean; 5.5 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 1.6 KiB/s rd, 0 B/s wr, 1.10k op/s; 29381/269115 objects degraded (10.918%); 13 MiB/s, 202 objects/s recovering 2024-03-11T21:38:05.240 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:38:04 smithi092 ceph-mon[37563]: from='client.? v1:172.21.15.92:0/1374763199' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]: dispatch 2024-03-11T21:38:05.240 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:38:04 smithi092 ceph-mon[37563]: from='client.25423 ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]: dispatch 2024-03-11T21:38:05.240 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:38:04 smithi092 ceph-mon[37563]: from='client.25423 ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]': finished 2024-03-11T21:38:05.240 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:38:04 smithi092 ceph-mon[37563]: osdmap e311: 8 total, 8 up, 8 in 2024-03-11T21:38:05.290 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:38:05.327 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:38:05.328 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:38:04 smithi097 ceph-mon[33317]: pgmap v576: 148 pgs: 5 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+degraded+remapped, 140 active+clean; 5.5 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 1.6 KiB/s rd, 0 B/s wr, 1.10k op/s; 29381/269115 objects degraded (10.918%); 13 MiB/s, 202 objects/s recovering 2024-03-11T21:38:05.328 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:38:04 smithi097 ceph-mon[33317]: from='client.? v1:172.21.15.92:0/1374763199' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]: dispatch 2024-03-11T21:38:05.328 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:38:04 smithi097 ceph-mon[33317]: from='client.25423 ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]: dispatch 2024-03-11T21:38:05.328 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:38:04 smithi097 ceph-mon[33317]: from='client.25423 ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]': finished 2024-03-11T21:38:05.328 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:38:04 smithi097 ceph-mon[33317]: osdmap e311: 8 total, 8 up, 8 in 2024-03-11T21:38:05.329 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:38:05.329 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:38:05.430 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:38:05.469 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:38:05.469 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:38:05.470 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:38:05.550 DEBUG:teuthology.orchestra.run.smithi097:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid 00db0016-dfed-11ee-95c8-87774f69a715 -- ceph daemon osd.5 dump_ops_in_flight 2024-03-11T21:38:05.570 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:38:05.609 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:38:05.610 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:38:05.610 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:38:05.711 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:38:05.747 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:38:05.747 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:38:05.748 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:38:05.850 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:38:05.884 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:38:05.885 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:38:05.886 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:38:05.912 INFO:teuthology.orchestra.run.smithi097.stderr:Inferring config /var/lib/ceph/00db0016-dfed-11ee-95c8-87774f69a715/mon.b/config 2024-03-11T21:38:05.987 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:38:06.030 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:38:06.031 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:38:06.031 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:38:06.132 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:38:06.175 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:38:06.176 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:38:06.176 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:38:06.277 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:38:06.316 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:38:06.317 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:38:06.317 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:38:06.418 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:38:06.474 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:38:06.474 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:38:06.475 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:38:06.484 INFO:tasks.cephadm.ceph_manager.ceph:waiting for recovery to complete 2024-03-11T21:38:06.484 DEBUG:teuthology.orchestra.run.smithi092:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid 00db0016-dfed-11ee-95c8-87774f69a715 -- ceph pg dump --format=json 2024-03-11T21:38:06.575 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:38:06.633 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:38:06.633 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:38:06.634 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:38:06.735 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:38:06.769 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:38:06.770 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:38:06.770 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:38:06.871 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:38:06.900 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:38:06.900 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:38:06.901 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:38:07.002 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:38:07.030 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:38:07.030 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:38:07.031 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:38:07.111 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:38:06 smithi092 ceph-mon[37563]: pgmap v579: 148 pgs: 5 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+degraded+remapped, 140 active+clean; 5.4 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 853 B/s rd, 0 B/s wr, 660 op/s; 29381/266046 objects degraded (11.044%); 7.2 MiB/s, 115 objects/s recovering 2024-03-11T21:38:07.111 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:38:06 smithi092 ceph-mon[29572]: pgmap v579: 148 pgs: 5 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+degraded+remapped, 140 active+clean; 5.4 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 853 B/s rd, 0 B/s wr, 660 op/s; 29381/266046 objects degraded (11.044%); 7.2 MiB/s, 115 objects/s recovering 2024-03-11T21:38:07.131 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:38:07.161 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:38:07.161 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:38:07.162 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:38:07.262 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:38:07.307 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:38:07.308 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:38:07.308 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:38:07.327 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:38:06 smithi097 ceph-mon[33317]: pgmap v579: 148 pgs: 5 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+degraded+remapped, 140 active+clean; 5.4 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 853 B/s rd, 0 B/s wr, 660 op/s; 29381/266046 objects degraded (11.044%); 7.2 MiB/s, 115 objects/s recovering 2024-03-11T21:38:07.409 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:38:07.437 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:38:07.438 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:38:07.438 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:38:07.539 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:38:07.582 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:38:07.583 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:38:07.583 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:38:07.684 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:38:07.698 INFO:teuthology.orchestra.run.smithi092.stderr:Inferring config /var/lib/ceph/00db0016-dfed-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T21:38:07.730 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:38:07.731 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:38:07.731 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:38:07.832 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:38:07.876 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:38:07.877 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:38:07.877 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:38:07.978 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:38:08.033 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:38:08.037 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:38:08.037 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:38:08.111 INFO:journalctl@ceph.mgr.y.smithi092.stdout:Mar 11 21:38:07 smithi092 ceph-00db0016-dfed-11ee-95c8-87774f69a715-mgr-y[29801]: ::ffff:172.21.15.97 - - [11/Mar/2024:21:38:07] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-11T21:38:08.138 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:38:08.177 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:38:08.177 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:38:08.178 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:38:08.233 INFO:teuthology.orchestra.run.smithi097.stdout:ERROR: (22) Invalid argument 2024-03-11T21:38:08.234 INFO:teuthology.orchestra.run.smithi097.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-11T21:38:08.279 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:38:08.313 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:38:08.313 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:38:08.314 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:38:08.334 INFO:teuthology.orchestra.run.smithi092.stderr:noscrub is set 2024-03-11T21:38:08.415 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:38:08.448 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:38:08.448 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:38:08.449 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:38:08.458 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:38:08 smithi092 ceph-mon[29572]: pgmap v580: 148 pgs: 5 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+degraded+remapped, 140 active+clean; 5.4 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 805 B/s rd, 0 B/s wr, 657 op/s; 29137/265401 objects degraded (10.978%); 9.2 MiB/s, 147 objects/s recovering 2024-03-11T21:38:08.458 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:38:08 smithi092 ceph-mon[29572]: from='mgr.14607 ' entity='mgr.y' 2024-03-11T21:38:08.458 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:38:08 smithi092 ceph-mon[29572]: from='mgr.14607 v1:172.21.15.92:0/1702717597' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T21:38:08.458 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:38:08 smithi092 ceph-mon[29572]: from='client.? v1:172.21.15.92:0/2115639945' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-11T21:38:08.458 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:38:08 smithi092 ceph-mon[29572]: Health check update: Degraded data redundancy: 29137/265401 objects degraded (10.978%), 8 pgs degraded, 8 pgs undersized (PG_DEGRADED) 2024-03-11T21:38:08.458 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:38:08 smithi092 ceph-mon[29572]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-03-11T21:38:08.458 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:38:08 smithi092 ceph-mon[29572]: from='client.? v1:172.21.15.92:0/2115639945' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-03-11T21:38:08.458 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:38:08 smithi092 ceph-mon[29572]: osdmap e312: 8 total, 8 up, 8 in 2024-03-11T21:38:08.458 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:38:08 smithi092 ceph-mon[37563]: pgmap v580: 148 pgs: 5 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+degraded+remapped, 140 active+clean; 5.4 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 805 B/s rd, 0 B/s wr, 657 op/s; 29137/265401 objects degraded (10.978%); 9.2 MiB/s, 147 objects/s recovering 2024-03-11T21:38:08.458 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:38:08 smithi092 ceph-mon[37563]: from='mgr.14607 ' entity='mgr.y' 2024-03-11T21:38:08.458 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:38:08 smithi092 ceph-mon[37563]: from='mgr.14607 v1:172.21.15.92:0/1702717597' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T21:38:08.459 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:38:08 smithi092 ceph-mon[37563]: from='client.? v1:172.21.15.92:0/2115639945' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-11T21:38:08.459 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:38:08 smithi092 ceph-mon[37563]: Health check update: Degraded data redundancy: 29137/265401 objects degraded (10.978%), 8 pgs degraded, 8 pgs undersized (PG_DEGRADED) 2024-03-11T21:38:08.459 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:38:08 smithi092 ceph-mon[37563]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-03-11T21:38:08.459 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:38:08 smithi092 ceph-mon[37563]: from='client.? v1:172.21.15.92:0/2115639945' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-03-11T21:38:08.459 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:38:08 smithi092 ceph-mon[37563]: osdmap e312: 8 total, 8 up, 8 in 2024-03-11T21:38:08.549 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:38:08.577 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:38:08 smithi097 ceph-mon[33317]: pgmap v580: 148 pgs: 5 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+degraded+remapped, 140 active+clean; 5.4 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 805 B/s rd, 0 B/s wr, 657 op/s; 29137/265401 objects degraded (10.978%); 9.2 MiB/s, 147 objects/s recovering 2024-03-11T21:38:08.577 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:38:08 smithi097 ceph-mon[33317]: from='mgr.14607 ' entity='mgr.y' 2024-03-11T21:38:08.577 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:38:08 smithi097 ceph-mon[33317]: from='mgr.14607 v1:172.21.15.92:0/1702717597' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T21:38:08.577 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:38:08 smithi097 ceph-mon[33317]: from='client.? v1:172.21.15.92:0/2115639945' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-11T21:38:08.577 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:38:08 smithi097 ceph-mon[33317]: Health check update: Degraded data redundancy: 29137/265401 objects degraded (10.978%), 8 pgs degraded, 8 pgs undersized (PG_DEGRADED) 2024-03-11T21:38:08.577 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:38:08 smithi097 ceph-mon[33317]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-03-11T21:38:08.577 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:38:08 smithi097 ceph-mon[33317]: from='client.? v1:172.21.15.92:0/2115639945' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-03-11T21:38:08.577 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:38:08 smithi097 ceph-mon[33317]: osdmap e312: 8 total, 8 up, 8 in 2024-03-11T21:38:08.587 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:38:08.588 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:38:08.588 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:38:08.689 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:38:08.721 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:38:08.722 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:38:08.722 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:38:08.823 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:38:08.853 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:38:08.854 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:38:08.854 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:38:08.955 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:38:08.990 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:38:08.991 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:38:08.991 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:38:09.035 DEBUG:teuthology.orchestra.run.smithi097:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid 00db0016-dfed-11ee-95c8-87774f69a715 -- ceph daemon osd.5 dump_blocked_ops 2024-03-11T21:38:09.092 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:38:09.146 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:38:09.147 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:38:09.147 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:38:09.248 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:38:09.292 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:38:09.293 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:38:09.293 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:38:09.394 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:38:09.441 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:38:09.442 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:38:09.443 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:38:09.516 INFO:teuthology.orchestra.run.smithi097.stderr:Inferring config /var/lib/ceph/00db0016-dfed-11ee-95c8-87774f69a715/mon.b/config 2024-03-11T21:38:09.544 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:38:09.591 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:38:09.592 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:38:09.593 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:38:09.633 DEBUG:teuthology.orchestra.run.smithi092:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid 00db0016-dfed-11ee-95c8-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-03-11T21:38:09.694 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:38:09.754 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:38:09.755 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:38:09.755 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:38:09.856 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:38:09.886 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:38:09.887 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:38:09.887 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:38:09.988 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:38:10.014 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:38:10.014 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:38:10.015 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:38:10.115 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:38:10.151 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:38:10.152 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:38:10.152 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:38:10.253 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:38:10.286 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:38:10.286 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:38:10.287 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:38:10.388 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:38:10.432 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:38:10.433 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:38:10.433 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:38:10.530 INFO:teuthology.orchestra.run.smithi092.stderr:Inferring config /var/lib/ceph/00db0016-dfed-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T21:38:10.534 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:38:10.565 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:38:10.566 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:38:10.566 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:38:10.611 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:38:10 smithi092 ceph-mon[29572]: pgmap v582: 148 pgs: 5 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+degraded+remapped, 140 active+clean; 5.2 GiB data, 18 GiB used, 698 GiB / 715 GiB avail; 853 B/s rd, 0 B/s wr, 825 op/s; 28688/254274 objects degraded (11.282%); 7.2 MiB/s, 115 objects/s recovering 2024-03-11T21:38:10.611 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:38:10 smithi092 ceph-mon[37563]: pgmap v582: 148 pgs: 5 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+degraded+remapped, 140 active+clean; 5.2 GiB data, 18 GiB used, 698 GiB / 715 GiB avail; 853 B/s rd, 0 B/s wr, 825 op/s; 28688/254274 objects degraded (11.282%); 7.2 MiB/s, 115 objects/s recovering 2024-03-11T21:38:10.667 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:38:10.679 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:38:10 smithi097 ceph-mon[33317]: pgmap v582: 148 pgs: 5 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+degraded+remapped, 140 active+clean; 5.2 GiB data, 18 GiB used, 698 GiB / 715 GiB avail; 853 B/s rd, 0 B/s wr, 825 op/s; 28688/254274 objects degraded (11.282%); 7.2 MiB/s, 115 objects/s recovering 2024-03-11T21:38:10.704 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:38:10.704 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:38:10.705 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:38:10.797 INFO:teuthology.orchestra.run.smithi092.stdout: 2024-03-11T21:38:10.800 INFO:teuthology.orchestra.run.smithi092.stderr:dumped all 2024-03-11T21:38:10.805 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:38:10.852 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:38:10.853 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:38:10.853 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:38:10.954 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:38:10.984 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:38:10.984 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:38:10.985 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:38:11.086 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:38:11.128 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:38:11.128 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:38:11.129 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:38:11.167 DEBUG:teuthology.orchestra.run.smithi092:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid 00db0016-dfed-11ee-95c8-87774f69a715 -- ceph osd set nodeep-scrub 2024-03-11T21:38:11.230 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:38:11.270 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:38:11.270 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:38:11.271 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:38:11.372 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:38:11.427 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:38:11.428 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:38:11.428 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:38:11.529 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:38:11.554 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:38:11.555 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:38:11.556 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:38:11.572 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:38:11 smithi092 ceph-mon[29572]: from='client.25429 v1:172.21.15.92:0/3811565470' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-11T21:38:11.572 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:38:11 smithi092 ceph-mon[37563]: from='client.25429 v1:172.21.15.92:0/3811565470' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-11T21:38:11.627 INFO:teuthology.orchestra.run.smithi092.stdout:{"pg_ready":true,"pg_map":{"version":582,"stamp":"2024-03-11T21:38:09.250849+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":5542255040,"num_objects":84758,"num_object_clones":0,"num_object_copies":254274,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":28688,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":84758,"num_whiteouts":0,"num_read":2078,"num_read_kb":1926,"num_write":149105,"num_write_kb":7450846,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9538,"num_bytes_recovered":614734278,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":85187,"ondisk_log_size":85187,"up":444,"acting":436,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":444,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":749961216,"kb_used":18483528,"kb_used_data":15138504,"kb_used_omap":15,"kb_used_meta":3344944,"kb_avail":731477688,"statfs":{"total":767960285184,"available":749033152512,"internally_reserved":0,"allocated":15501828096,"data_stored":15495839056,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15601,"internal_metadata":3425223439},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-324206592,"num_objects":-4947,"num_object_clones":0,"num_object_copies":-14841,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-693,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-4947,"num_whiteouts":0,"num_read":5,"num_read_kb":5,"num_write":4947,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":693,"num_bytes_recovered":45416448,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.002038"},"pg_stats":[{"pgid":"6.1b","version":"0'0","reported_seq":473,"reported_epoch":312,"state":"active+clean","last_fresh":"2024-03-11T21:38:08.367544+0000","last_change":"2024-03-11T21:36:35.074931+0000","last_active":"2024-03-11T21:38:08.367544+0000","last_peered":"2024-03-11T21:38:08.367544+0000","last_clean":"2024-03-11T21:38:08.367544+0000","last_became_active":"2024-03-11T21:36:35.072636+0000","last_became_peered":"2024-03-11T21:36:35.072636+0000","last_unstale":"2024-03-11T21:38:08.367544+0000","last_undegraded":"2024-03-11T21:38:08.367544+0000","last_fullsized":"2024-03-11T21:38:08.367544+0000","mapping_epoch":277,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":278,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:35:07.021520+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:43.559452+0000","last_clean_scrub_stamp":"2024-03-11T21:35:07.021520+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":536,"reported_epoch":311,"state":"active+clean","last_fresh":"2024-03-11T21:38:04.368133+0000","last_change":"2024-03-11T21:36:00.720089+0000","last_active":"2024-03-11T21:38:04.368133+0000","last_peered":"2024-03-11T21:38:04.368133+0000","last_clean":"2024-03-11T21:38:04.368133+0000","last_became_active":"2024-03-11T21:25:42.578893+0000","last_became_peered":"2024-03-11T21:25:42.578893+0000","last_unstale":"2024-03-11T21:38:04.368133+0000","last_undegraded":"2024-03-11T21:38:04.368133+0000","last_fullsized":"2024-03-11T21:38:04.368133+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:36:00.719956+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:41.544550+0000","last_clean_scrub_stamp":"2024-03-11T21:36:00.719956+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1f","version":"0'0","reported_seq":567,"reported_epoch":312,"state":"active+clean","last_fresh":"2024-03-11T21:38:08.349363+0000","last_change":"2024-03-11T21:36:00.843838+0000","last_active":"2024-03-11T21:38:08.349363+0000","last_peered":"2024-03-11T21:38:08.349363+0000","last_clean":"2024-03-11T21:38:08.349363+0000","last_became_active":"2024-03-11T21:34:41.144945+0000","last_became_peered":"2024-03-11T21:34:41.144945+0000","last_unstale":"2024-03-11T21:38:08.349363+0000","last_undegraded":"2024-03-11T21:38:08.349363+0000","last_fullsized":"2024-03-11T21:38:08.349363+0000","mapping_epoch":226,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":227,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:36:00.843718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:37.512018+0000","last_clean_scrub_stamp":"2024-03-11T21:36:00.843718+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,4],"acting":[0,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.19","version":"260'27","reported_seq":520,"reported_epoch":312,"state":"active+clean","last_fresh":"2024-03-11T21:38:08.364531+0000","last_change":"2024-03-11T21:36:35.089901+0000","last_active":"2024-03-11T21:38:08.364531+0000","last_peered":"2024-03-11T21:38:08.364531+0000","last_clean":"2024-03-11T21:38:08.364531+0000","last_became_active":"2024-03-11T21:36:35.067652+0000","last_became_peered":"2024-03-11T21:36:35.067652+0000","last_unstale":"2024-03-11T21:38:08.364531+0000","last_undegraded":"2024-03-11T21:38:08.364531+0000","last_fullsized":"2024-03-11T21:38:08.364531+0000","mapping_epoch":277,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":278,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'15","last_scrub_stamp":"2024-03-11T21:35:22.099374+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:39.525554+0000","last_clean_scrub_stamp":"2024-03-11T21:35:22.099374+0000","objects_scrubbed":0,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":42,"num_read_kb":33,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.1a","version":"0'0","reported_seq":523,"reported_epoch":311,"state":"active+clean","last_fresh":"2024-03-11T21:38:04.368182+0000","last_change":"2024-03-11T21:35:16.410927+0000","last_active":"2024-03-11T21:38:04.368182+0000","last_peered":"2024-03-11T21:38:04.368182+0000","last_clean":"2024-03-11T21:38:04.368182+0000","last_became_active":"2024-03-11T21:25:44.590923+0000","last_became_peered":"2024-03-11T21:25:44.590923+0000","last_unstale":"2024-03-11T21:38:04.368182+0000","last_undegraded":"2024-03-11T21:38:04.368182+0000","last_fullsized":"2024-03-11T21:38:04.368182+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:35:16.410816+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:43.559452+0000","last_clean_scrub_stamp":"2024-03-11T21:35:16.410816+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":24,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,1],"acting":[4,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":548,"reported_epoch":311,"state":"active+clean","last_fresh":"2024-03-11T21:38:04.363049+0000","last_change":"2024-03-11T21:36:15.576138+0000","last_active":"2024-03-11T21:38:04.363049+0000","last_peered":"2024-03-11T21:38:04.363049+0000","last_clean":"2024-03-11T21:38:04.363049+0000","last_became_active":"2024-03-11T21:34:41.154216+0000","last_became_peered":"2024-03-11T21:34:41.154216+0000","last_unstale":"2024-03-11T21:38:04.363049+0000","last_undegraded":"2024-03-11T21:38:04.363049+0000","last_fullsized":"2024-03-11T21:38:04.363049+0000","mapping_epoch":226,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":227,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:36:15.575887+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:41.544550+0000","last_clean_scrub_stamp":"2024-03-11T21:36:15.575887+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,7],"acting":[1,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1e","version":"0'0","reported_seq":486,"reported_epoch":312,"state":"active+clean","last_fresh":"2024-03-11T21:38:08.357486+0000","last_change":"2024-03-11T21:36:35.062413+0000","last_active":"2024-03-11T21:38:08.357486+0000","last_peered":"2024-03-11T21:38:08.357486+0000","last_clean":"2024-03-11T21:38:08.357486+0000","last_became_active":"2024-03-11T21:36:35.061786+0000","last_became_peered":"2024-03-11T21:36:35.061786+0000","last_unstale":"2024-03-11T21:38:08.357486+0000","last_undegraded":"2024-03-11T21:38:08.357486+0000","last_fullsized":"2024-03-11T21:38:08.357486+0000","mapping_epoch":277,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":278,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:32:18.578038+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:37.512018+0000","last_clean_scrub_stamp":"2024-03-11T21:32:18.578038+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.18","version":"260'15","reported_seq":573,"reported_epoch":311,"state":"active+clean","last_fresh":"2024-03-11T21:38:04.367937+0000","last_change":"2024-03-11T21:36:09.662765+0000","last_active":"2024-03-11T21:38:04.367937+0000","last_peered":"2024-03-11T21:38:04.367937+0000","last_clean":"2024-03-11T21:38:04.367937+0000","last_became_active":"2024-03-11T21:25:40.552216+0000","last_became_peered":"2024-03-11T21:25:40.552216+0000","last_unstale":"2024-03-11T21:38:04.367937+0000","last_undegraded":"2024-03-11T21:38:04.367937+0000","last_fullsized":"2024-03-11T21:38:04.367937+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"260'15","last_scrub_stamp":"2024-03-11T21:36:09.662615+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:39.525554+0000","last_clean_scrub_stamp":"2024-03-11T21:36:09.662615+0000","objects_scrubbed":4,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":540,"reported_epoch":311,"state":"active+clean","last_fresh":"2024-03-11T21:38:04.392424+0000","last_change":"2024-03-11T21:36:35.068915+0000","last_active":"2024-03-11T21:38:04.392424+0000","last_peered":"2024-03-11T21:38:04.392424+0000","last_clean":"2024-03-11T21:38:04.392424+0000","last_became_active":"2024-03-11T21:36:35.068453+0000","last_became_peered":"2024-03-11T21:36:35.068453+0000","last_unstale":"2024-03-11T21:38:04.392424+0000","last_undegraded":"2024-03-11T21:38:04.392424+0000","last_fullsized":"2024-03-11T21:38:04.392424+0000","mapping_epoch":277,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":278,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:35:26.190634+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:32:12.654573+0000","last_clean_scrub_stamp":"2024-03-11T21:35:26.190634+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3],"acting":[5,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":188,"reported_epoch":311,"state":"active+clean","last_fresh":"2024-03-11T21:38:04.394881+0000","last_change":"2024-03-11T21:35:11.902920+0000","last_active":"2024-03-11T21:38:04.394881+0000","last_peered":"2024-03-11T21:38:04.394881+0000","last_clean":"2024-03-11T21:38:04.394881+0000","last_became_active":"2024-03-11T21:34:41.145299+0000","last_became_peered":"2024-03-11T21:34:41.145299+0000","last_unstale":"2024-03-11T21:38:04.394881+0000","last_undegraded":"2024-03-11T21:38:04.394881+0000","last_fullsized":"2024-03-11T21:38:04.394881+0000","mapping_epoch":226,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":227,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:35:11.902818+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:41.544550+0000","last_clean_scrub_stamp":"2024-03-11T21:35:11.902818+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.1d","version":"0'0","reported_seq":550,"reported_epoch":311,"state":"active+clean","last_fresh":"2024-03-11T21:38:04.394960+0000","last_change":"2024-03-11T21:35:26.855588+0000","last_active":"2024-03-11T21:38:04.394960+0000","last_peered":"2024-03-11T21:38:04.394960+0000","last_clean":"2024-03-11T21:38:04.394960+0000","last_became_active":"2024-03-11T21:25:38.542628+0000","last_became_peered":"2024-03-11T21:25:38.542628+0000","last_unstale":"2024-03-11T21:38:04.394960+0000","last_undegraded":"2024-03-11T21:38:04.394960+0000","last_fullsized":"2024-03-11T21:38:04.394960+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:35:26.855511+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:37.512018+0000","last_clean_scrub_stamp":"2024-03-11T21:35:26.855511+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,0],"acting":[7,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.1b","version":"260'7","reported_seq":558,"reported_epoch":311,"state":"active+clean","last_fresh":"2024-03-11T21:38:04.368408+0000","last_change":"2024-03-11T21:36:35.087276+0000","last_active":"2024-03-11T21:38:04.368408+0000","last_peered":"2024-03-11T21:38:04.368408+0000","last_clean":"2024-03-11T21:38:04.368408+0000","last_became_active":"2024-03-11T21:36:35.057678+0000","last_became_peered":"2024-03-11T21:36:35.057678+0000","last_unstale":"2024-03-11T21:38:04.368408+0000","last_undegraded":"2024-03-11T21:38:04.368408+0000","last_fullsized":"2024-03-11T21:38:04.368408+0000","mapping_epoch":277,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":278,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'5","last_scrub_stamp":"2024-03-11T21:35:09.427562+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:39.525554+0000","last_clean_scrub_stamp":"2024-03-11T21:35:09.427562+0000","objects_scrubbed":2,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":14,"num_read_kb":10,"num_write":8,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,1],"acting":[4,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":534,"reported_epoch":312,"state":"active+clean","last_fresh":"2024-03-11T21:38:08.351083+0000","last_change":"2024-03-11T21:35:39.691825+0000","last_active":"2024-03-11T21:38:08.351083+0000","last_peered":"2024-03-11T21:38:08.351083+0000","last_clean":"2024-03-11T21:38:08.351083+0000","last_became_active":"2024-03-11T21:25:44.589786+0000","last_became_peered":"2024-03-11T21:25:44.589786+0000","last_unstale":"2024-03-11T21:38:08.351083+0000","last_undegraded":"2024-03-11T21:38:08.351083+0000","last_fullsized":"2024-03-11T21:38:08.351083+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:35:39.691690+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:43.559452+0000","last_clean_scrub_stamp":"2024-03-11T21:35:39.691690+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":18,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,7],"acting":[0,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":546,"reported_epoch":311,"state":"active+clean","last_fresh":"2024-03-11T21:38:04.393669+0000","last_change":"2024-03-11T21:35:25.194240+0000","last_active":"2024-03-11T21:38:04.393669+0000","last_peered":"2024-03-11T21:38:04.393669+0000","last_clean":"2024-03-11T21:38:04.393669+0000","last_became_active":"2024-03-11T21:34:41.147185+0000","last_became_peered":"2024-03-11T21:34:41.147185+0000","last_unstale":"2024-03-11T21:38:04.393669+0000","last_undegraded":"2024-03-11T21:38:04.393669+0000","last_fullsized":"2024-03-11T21:38:04.393669+0000","mapping_epoch":226,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":227,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:35:25.194115+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:32:21.535034+0000","last_clean_scrub_stamp":"2024-03-11T21:35:25.194115+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,7],"acting":[5,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1c","version":"0'0","reported_seq":550,"reported_epoch":311,"state":"active+clean","last_fresh":"2024-03-11T21:38:04.394561+0000","last_change":"2024-03-11T21:35:28.789693+0000","last_active":"2024-03-11T21:38:04.394561+0000","last_peered":"2024-03-11T21:38:04.394561+0000","last_clean":"2024-03-11T21:38:04.394561+0000","last_became_active":"2024-03-11T21:25:38.541364+0000","last_became_peered":"2024-03-11T21:25:38.541364+0000","last_unstale":"2024-03-11T21:38:04.394561+0000","last_undegraded":"2024-03-11T21:38:04.394561+0000","last_fullsized":"2024-03-11T21:38:04.394561+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:35:28.789592+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:28:32.566437+0000","last_clean_scrub_stamp":"2024-03-11T21:35:28.789592+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.1a","version":"260'15","reported_seq":583,"reported_epoch":311,"state":"active+clean","last_fresh":"2024-03-11T21:38:04.368345+0000","last_change":"2024-03-11T21:36:35.085358+0000","last_active":"2024-03-11T21:38:04.368345+0000","last_peered":"2024-03-11T21:38:04.368345+0000","last_clean":"2024-03-11T21:38:04.368345+0000","last_became_active":"2024-03-11T21:36:35.057029+0000","last_became_peered":"2024-03-11T21:36:35.057029+0000","last_unstale":"2024-03-11T21:38:04.368345+0000","last_undegraded":"2024-03-11T21:38:04.368345+0000","last_fullsized":"2024-03-11T21:38:04.368345+0000","mapping_epoch":277,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":278,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'9","last_scrub_stamp":"2024-03-11T21:35:05.451780+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:39.525554+0000","last_clean_scrub_stamp":"2024-03-11T21:35:05.451780+0000","objects_scrubbed":4,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0],"acting":[4,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1f","version":"0'0","reported_seq":409,"reported_epoch":312,"state":"active+clean","last_fresh":"2024-03-11T21:38:08.367217+0000","last_change":"2024-03-11T21:36:35.074675+0000","last_active":"2024-03-11T21:38:08.367217+0000","last_peered":"2024-03-11T21:38:08.367217+0000","last_clean":"2024-03-11T21:38:08.367217+0000","last_became_active":"2024-03-11T21:36:35.071755+0000","last_became_peered":"2024-03-11T21:36:35.071755+0000","last_unstale":"2024-03-11T21:38:08.367217+0000","last_undegraded":"2024-03-11T21:38:08.367217+0000","last_fullsized":"2024-03-11T21:38:08.367217+0000","mapping_epoch":277,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":278,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:34:05.862891+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:43.559452+0000","last_clean_scrub_stamp":"2024-03-11T21:34:05.862891+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1c","version":"0'0","reported_seq":536,"reported_epoch":311,"state":"active+clean","last_fresh":"2024-03-11T21:38:04.369887+0000","last_change":"2024-03-11T21:36:35.067469+0000","last_active":"2024-03-11T21:38:04.369887+0000","last_peered":"2024-03-11T21:38:04.369887+0000","last_clean":"2024-03-11T21:38:04.369887+0000","last_became_active":"2024-03-11T21:36:35.067127+0000","last_became_peered":"2024-03-11T21:36:35.067127+0000","last_unstale":"2024-03-11T21:38:04.369887+0000","last_undegraded":"2024-03-11T21:38:04.369887+0000","last_fullsized":"2024-03-11T21:38:04.369887+0000","mapping_epoch":277,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":278,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:34:11.766066+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:41.544550+0000","last_clean_scrub_stamp":"2024-03-11T21:34:11.766066+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1b","version":"0'0","reported_seq":458,"reported_epoch":312,"state":"active+clean","last_fresh":"2024-03-11T21:38:08.367637+0000","last_change":"2024-03-11T21:36:35.075385+0000","last_active":"2024-03-11T21:38:08.367637+0000","last_peered":"2024-03-11T21:38:08.367637+0000","last_clean":"2024-03-11T21:38:08.367637+0000","last_became_active":"2024-03-11T21:36:35.073959+0000","last_became_peered":"2024-03-11T21:36:35.073959+0000","last_unstale":"2024-03-11T21:38:08.367637+0000","last_undegraded":"2024-03-11T21:38:08.367637+0000","last_fullsized":"2024-03-11T21:38:08.367637+0000","mapping_epoch":277,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":278,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:34:27.998724+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:34:27.998724+0000","last_clean_scrub_stamp":"2024-03-11T21:34:27.998724+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1d","version":"260'18","reported_seq":502,"reported_epoch":312,"state":"active+clean","last_fresh":"2024-03-11T21:38:08.357040+0000","last_change":"2024-03-11T21:36:35.080207+0000","last_active":"2024-03-11T21:38:08.357040+0000","last_peered":"2024-03-11T21:38:08.357040+0000","last_clean":"2024-03-11T21:38:08.357040+0000","last_became_active":"2024-03-11T21:36:35.055880+0000","last_became_peered":"2024-03-11T21:36:35.055880+0000","last_unstale":"2024-03-11T21:38:08.357040+0000","last_undegraded":"2024-03-11T21:38:08.357040+0000","last_fullsized":"2024-03-11T21:38:08.357040+0000","mapping_epoch":277,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":278,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'12","last_scrub_stamp":"2024-03-11T21:35:07.971581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:39.525554+0000","last_clean_scrub_stamp":"2024-03-11T21:35:07.971581+0000","objects_scrubbed":0,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":33,"num_read_kb":24,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.1e","version":"0'0","reported_seq":525,"reported_epoch":311,"state":"active+clean","last_fresh":"2024-03-11T21:38:04.368845+0000","last_change":"2024-03-11T21:35:32.664451+0000","last_active":"2024-03-11T21:38:04.368845+0000","last_peered":"2024-03-11T21:38:04.368845+0000","last_clean":"2024-03-11T21:38:04.368845+0000","last_became_active":"2024-03-11T21:25:45.300355+0000","last_became_peered":"2024-03-11T21:25:45.300355+0000","last_unstale":"2024-03-11T21:38:04.368845+0000","last_undegraded":"2024-03-11T21:38:04.368845+0000","last_fullsized":"2024-03-11T21:38:04.368845+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:35:32.664371+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:43.559452+0000","last_clean_scrub_stamp":"2024-03-11T21:35:32.664371+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5],"acting":[4,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1d","version":"0'0","reported_seq":535,"reported_epoch":311,"state":"active+clean","last_fresh":"2024-03-11T21:38:04.367743+0000","last_change":"2024-03-11T21:36:07.624196+0000","last_active":"2024-03-11T21:38:04.367743+0000","last_peered":"2024-03-11T21:38:04.367743+0000","last_clean":"2024-03-11T21:38:04.367743+0000","last_became_active":"2024-03-11T21:25:42.579407+0000","last_became_peered":"2024-03-11T21:25:42.579407+0000","last_unstale":"2024-03-11T21:38:04.367743+0000","last_undegraded":"2024-03-11T21:38:04.367743+0000","last_fullsized":"2024-03-11T21:38:04.367743+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:36:07.624075+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:30:18.678531+0000","last_clean_scrub_stamp":"2024-03-11T21:36:07.624075+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1a","version":"0'0","reported_seq":563,"reported_epoch":311,"state":"active+clean","last_fresh":"2024-03-11T21:38:04.363501+0000","last_change":"2024-03-11T21:36:12.993542+0000","last_active":"2024-03-11T21:38:04.363501+0000","last_peered":"2024-03-11T21:38:04.363501+0000","last_clean":"2024-03-11T21:38:04.363501+0000","last_became_active":"2024-03-11T21:34:41.145014+0000","last_became_peered":"2024-03-11T21:34:41.145014+0000","last_unstale":"2024-03-11T21:38:04.363501+0000","last_undegraded":"2024-03-11T21:38:04.363501+0000","last_fullsized":"2024-03-11T21:38:04.363501+0000","mapping_epoch":226,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":227,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:36:12.993437+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:37.512018+0000","last_clean_scrub_stamp":"2024-03-11T21:36:12.993437+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.1c","version":"260'27","reported_seq":597,"reported_epoch":311,"state":"active+clean","last_fresh":"2024-03-11T21:38:04.366239+0000","last_change":"2024-03-11T21:36:35.097208+0000","last_active":"2024-03-11T21:38:04.366239+0000","last_peered":"2024-03-11T21:38:04.366239+0000","last_clean":"2024-03-11T21:38:04.366239+0000","last_became_active":"2024-03-11T21:36:35.074579+0000","last_became_peered":"2024-03-11T21:36:35.074579+0000","last_unstale":"2024-03-11T21:38:04.366239+0000","last_undegraded":"2024-03-11T21:38:04.366239+0000","last_fullsized":"2024-03-11T21:38:04.366239+0000","mapping_epoch":277,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":278,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'15","last_scrub_stamp":"2024-03-11T21:34:10.311181+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:39.525554+0000","last_clean_scrub_stamp":"2024-03-11T21:34:10.311181+0000","objects_scrubbed":7,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":44,"num_read_kb":35,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.1d","version":"0'0","reported_seq":523,"reported_epoch":311,"state":"active+clean","last_fresh":"2024-03-11T21:38:04.367249+0000","last_change":"2024-03-11T21:35:21.744646+0000","last_active":"2024-03-11T21:38:04.367249+0000","last_peered":"2024-03-11T21:38:04.367249+0000","last_clean":"2024-03-11T21:38:04.367249+0000","last_became_active":"2024-03-11T21:25:44.588845+0000","last_became_peered":"2024-03-11T21:25:44.588845+0000","last_unstale":"2024-03-11T21:38:04.367249+0000","last_undegraded":"2024-03-11T21:38:04.367249+0000","last_fullsized":"2024-03-11T21:38:04.367249+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:35:21.744223+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:43.559452+0000","last_clean_scrub_stamp":"2024-03-11T21:35:21.744223+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":99,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.1e","version":"0'0","reported_seq":539,"reported_epoch":312,"state":"active+clean","last_fresh":"2024-03-11T21:38:08.349273+0000","last_change":"2024-03-11T21:35:10.728691+0000","last_active":"2024-03-11T21:38:08.349273+0000","last_peered":"2024-03-11T21:38:08.349273+0000","last_clean":"2024-03-11T21:38:08.349273+0000","last_became_active":"2024-03-11T21:25:42.582959+0000","last_became_peered":"2024-03-11T21:25:42.582959+0000","last_unstale":"2024-03-11T21:38:08.349273+0000","last_undegraded":"2024-03-11T21:38:08.349273+0000","last_fullsized":"2024-03-11T21:38:08.349273+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:35:10.728569+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:26:58.335431+0000","last_clean_scrub_stamp":"2024-03-11T21:35:10.728569+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":18,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.19","version":"67'2","reported_seq":484,"reported_epoch":312,"state":"active+clean","last_fresh":"2024-03-11T21:38:08.364422+0000","last_change":"2024-03-11T21:36:35.069729+0000","last_active":"2024-03-11T21:38:08.364422+0000","last_peered":"2024-03-11T21:38:08.364422+0000","last_clean":"2024-03-11T21:38:08.364422+0000","last_became_active":"2024-03-11T21:36:35.064249+0000","last_became_peered":"2024-03-11T21:36:35.064249+0000","last_unstale":"2024-03-11T21:38:08.364422+0000","last_undegraded":"2024-03-11T21:38:08.364422+0000","last_fullsized":"2024-03-11T21:38:08.364422+0000","mapping_epoch":277,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":278,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'2","last_scrub_stamp":"2024-03-11T21:32:15.509542+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:37.512018+0000","last_clean_scrub_stamp":"2024-03-11T21:32:15.509542+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":957,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,0],"acting":[3,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1f","version":"260'24","reported_seq":602,"reported_epoch":311,"state":"active+clean","last_fresh":"2024-03-11T21:38:04.386771+0000","last_change":"2024-03-11T21:35:38.173937+0000","last_active":"2024-03-11T21:38:04.386771+0000","last_peered":"2024-03-11T21:38:04.386771+0000","last_clean":"2024-03-11T21:38:04.386771+0000","last_became_active":"2024-03-11T21:32:22.798242+0000","last_became_peered":"2024-03-11T21:32:22.798242+0000","last_unstale":"2024-03-11T21:38:04.386771+0000","last_undegraded":"2024-03-11T21:38:04.386771+0000","last_fullsized":"2024-03-11T21:38:04.386771+0000","mapping_epoch":166,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":167,"parent":"0.0","parent_split_bits":0,"last_scrub":"124'16","last_scrub_stamp":"2024-03-11T21:35:38.173841+0000","last_deep_scrub":"124'16","last_deep_scrub_stamp":"2024-03-11T21:35:38.173841+0000","last_clean_scrub_stamp":"2024-03-11T21:35:38.173841+0000","objects_scrubbed":5,"log_size":24,"log_dups_size":0,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":32,"num_read_kb":25,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,0],"acting":[6,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.1c","version":"74'1","reported_seq":192,"reported_epoch":311,"state":"active+clean","last_fresh":"2024-03-11T21:38:04.394446+0000","last_change":"2024-03-11T21:36:11.732043+0000","last_active":"2024-03-11T21:38:04.394446+0000","last_peered":"2024-03-11T21:38:04.394446+0000","last_clean":"2024-03-11T21:38:04.394446+0000","last_became_active":"2024-03-11T21:34:41.144839+0000","last_became_peered":"2024-03-11T21:34:41.144839+0000","last_unstale":"2024-03-11T21:38:04.394446+0000","last_undegraded":"2024-03-11T21:38:04.394446+0000","last_fullsized":"2024-03-11T21:38:04.394446+0000","mapping_epoch":226,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":227,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'1","last_scrub_stamp":"2024-03-11T21:36:11.731914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:43.559452+0000","last_clean_scrub_stamp":"2024-03-11T21:36:11.731914+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":370,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":370,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.1f","version":"229'10","reported_seq":733,"reported_epoch":311,"state":"active+clean","last_fresh":"2024-03-11T21:38:04.386650+0000","last_change":"2024-03-11T21:36:08.871358+0000","last_active":"2024-03-11T21:38:04.386650+0000","last_peered":"2024-03-11T21:38:04.386650+0000","last_clean":"2024-03-11T21:38:04.386650+0000","last_became_active":"2024-03-11T21:34:41.145862+0000","last_became_peered":"2024-03-11T21:34:41.145862+0000","last_unstale":"2024-03-11T21:38:04.386650+0000","last_undegraded":"2024-03-11T21:38:04.386650+0000","last_fullsized":"2024-03-11T21:38:04.386650+0000","mapping_epoch":226,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":227,"parent":"0.0","parent_split_bits":0,"last_scrub":"229'10","last_scrub_stamp":"2024-03-11T21:36:08.871213+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:41.544550+0000","last_clean_scrub_stamp":"2024-03-11T21:36:08.871213+0000","objects_scrubbed":1,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.18","version":"67'1","reported_seq":587,"reported_epoch":311,"state":"active+clean","last_fresh":"2024-03-11T21:38:04.392168+0000","last_change":"2024-03-11T21:36:10.458150+0000","last_active":"2024-03-11T21:38:04.392168+0000","last_peered":"2024-03-11T21:38:04.392168+0000","last_clean":"2024-03-11T21:38:04.392168+0000","last_became_active":"2024-03-11T21:35:23.310241+0000","last_became_peered":"2024-03-11T21:35:23.310241+0000","last_unstale":"2024-03-11T21:38:04.392168+0000","last_undegraded":"2024-03-11T21:38:04.392168+0000","last_fullsized":"2024-03-11T21:38:04.392168+0000","mapping_epoch":248,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":249,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'1","last_scrub_stamp":"2024-03-11T21:36:10.458028+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:37.512018+0000","last_clean_scrub_stamp":"2024-03-11T21:36:10.458028+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":436,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":7,"num_read_kb":7,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":1308,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,2],"acting":[5,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1e","version":"260'20","reported_seq":631,"reported_epoch":312,"state":"active+clean","last_fresh":"2024-03-11T21:38:08.349170+0000","last_change":"2024-03-11T21:36:35.083447+0000","last_active":"2024-03-11T21:38:08.349170+0000","last_peered":"2024-03-11T21:38:08.349170+0000","last_clean":"2024-03-11T21:38:08.349170+0000","last_became_active":"2024-03-11T21:36:35.057843+0000","last_became_peered":"2024-03-11T21:36:35.057843+0000","last_unstale":"2024-03-11T21:38:08.349170+0000","last_undegraded":"2024-03-11T21:38:08.349170+0000","last_fullsized":"2024-03-11T21:38:08.349170+0000","mapping_epoch":277,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":278,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'10","last_scrub_stamp":"2024-03-11T21:35:16.825146+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:39.525554+0000","last_clean_scrub_stamp":"2024-03-11T21:35:16.825146+0000","objects_scrubbed":5,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":44,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":25,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.13","version":"0'0","reported_seq":417,"reported_epoch":312,"state":"active+clean","last_fresh":"2024-03-11T21:38:08.367850+0000","last_change":"2024-03-11T21:36:35.063744+0000","last_active":"2024-03-11T21:38:08.367850+0000","last_peered":"2024-03-11T21:38:08.367850+0000","last_clean":"2024-03-11T21:38:08.367850+0000","last_became_active":"2024-03-11T21:36:35.062618+0000","last_became_peered":"2024-03-11T21:36:35.062618+0000","last_unstale":"2024-03-11T21:38:08.367850+0000","last_undegraded":"2024-03-11T21:38:08.367850+0000","last_fullsized":"2024-03-11T21:38:08.367850+0000","mapping_epoch":277,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":278,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:34:13.822803+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:43.559452+0000","last_clean_scrub_stamp":"2024-03-11T21:34:13.822803+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":188,"reported_epoch":311,"state":"active+clean","last_fresh":"2024-03-11T21:38:04.393406+0000","last_change":"2024-03-11T21:35:16.917648+0000","last_active":"2024-03-11T21:38:04.393406+0000","last_peered":"2024-03-11T21:38:04.393406+0000","last_clean":"2024-03-11T21:38:04.393406+0000","last_became_active":"2024-03-11T21:34:41.142569+0000","last_became_peered":"2024-03-11T21:34:41.142569+0000","last_unstale":"2024-03-11T21:38:04.393406+0000","last_undegraded":"2024-03-11T21:38:04.393406+0000","last_fullsized":"2024-03-11T21:38:04.393406+0000","mapping_epoch":226,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":227,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:35:16.917532+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:41.544550+0000","last_clean_scrub_stamp":"2024-03-11T21:35:16.917532+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.17","version":"0'0","reported_seq":544,"reported_epoch":311,"state":"active+clean","last_fresh":"2024-03-11T21:38:04.386488+0000","last_change":"2024-03-11T21:35:06.186953+0000","last_active":"2024-03-11T21:38:04.386488+0000","last_peered":"2024-03-11T21:38:04.386488+0000","last_clean":"2024-03-11T21:38:04.386488+0000","last_became_active":"2024-03-11T21:25:38.540349+0000","last_became_peered":"2024-03-11T21:25:38.540349+0000","last_unstale":"2024-03-11T21:38:04.386488+0000","last_undegraded":"2024-03-11T21:38:04.386488+0000","last_fullsized":"2024-03-11T21:38:04.386488+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:35:06.186840+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:28:25.918101+0000","last_clean_scrub_stamp":"2024-03-11T21:35:06.186840+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,2],"acting":[6,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.11","version":"260'19","reported_seq":543,"reported_epoch":312,"state":"active+clean","last_fresh":"2024-03-11T21:38:08.364880+0000","last_change":"2024-03-11T21:36:35.096720+0000","last_active":"2024-03-11T21:38:08.364880+0000","last_peered":"2024-03-11T21:38:08.364880+0000","last_clean":"2024-03-11T21:38:08.364880+0000","last_became_active":"2024-03-11T21:36:35.073689+0000","last_became_peered":"2024-03-11T21:36:35.073689+0000","last_unstale":"2024-03-11T21:38:08.364880+0000","last_undegraded":"2024-03-11T21:38:08.364880+0000","last_fullsized":"2024-03-11T21:38:08.364880+0000","mapping_epoch":277,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":278,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'11","last_scrub_stamp":"2024-03-11T21:35:16.044397+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:39.525554+0000","last_clean_scrub_stamp":"2024-03-11T21:35:16.044397+0000","objects_scrubbed":5,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":31,"num_read_kb":24,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":188,"reported_epoch":311,"state":"active+clean","last_fresh":"2024-03-11T21:38:04.394749+0000","last_change":"2024-03-11T21:35:32.719869+0000","last_active":"2024-03-11T21:38:04.394749+0000","last_peered":"2024-03-11T21:38:04.394749+0000","last_clean":"2024-03-11T21:38:04.394749+0000","last_became_active":"2024-03-11T21:34:41.137235+0000","last_became_peered":"2024-03-11T21:34:41.137235+0000","last_unstale":"2024-03-11T21:38:04.394749+0000","last_undegraded":"2024-03-11T21:38:04.394749+0000","last_fullsized":"2024-03-11T21:38:04.394749+0000","mapping_epoch":226,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":227,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:35:32.719789+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:43.559452+0000","last_clean_scrub_stamp":"2024-03-11T21:35:32.719789+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,4],"acting":[7,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":556,"reported_epoch":311,"state":"active+clean","last_fresh":"2024-03-11T21:38:04.386031+0000","last_change":"2024-03-11T21:35:31.111662+0000","last_active":"2024-03-11T21:38:04.386031+0000","last_peered":"2024-03-11T21:38:04.386031+0000","last_clean":"2024-03-11T21:38:04.386031+0000","last_became_active":"2024-03-11T21:34:41.156036+0000","last_became_peered":"2024-03-11T21:34:41.156036+0000","last_unstale":"2024-03-11T21:38:04.386031+0000","last_undegraded":"2024-03-11T21:38:04.386031+0000","last_fullsized":"2024-03-11T21:38:04.386031+0000","mapping_epoch":226,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":227,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:35:31.111559+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:29:39.923895+0000","last_clean_scrub_stamp":"2024-03-11T21:35:31.111559+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.16","version":"0'0","reported_seq":551,"reported_epoch":311,"state":"active+clean","last_fresh":"2024-03-11T21:38:04.397032+0000","last_change":"2024-03-11T21:35:39.108623+0000","last_active":"2024-03-11T21:38:04.397032+0000","last_peered":"2024-03-11T21:38:04.397032+0000","last_clean":"2024-03-11T21:38:04.397032+0000","last_became_active":"2024-03-11T21:25:38.539103+0000","last_became_peered":"2024-03-11T21:25:38.539103+0000","last_unstale":"2024-03-11T21:38:04.397032+0000","last_undegraded":"2024-03-11T21:38:04.397032+0000","last_fullsized":"2024-03-11T21:38:04.397032+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:35:39.108546+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:27:11.938909+0000","last_clean_scrub_stamp":"2024-03-11T21:35:39.108546+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,2],"acting":[5,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.10","version":"260'10","reported_seq":425,"reported_epoch":312,"state":"active+clean","last_fresh":"2024-03-11T21:38:08.356845+0000","last_change":"2024-03-11T21:36:35.084268+0000","last_active":"2024-03-11T21:38:08.356845+0000","last_peered":"2024-03-11T21:38:08.356845+0000","last_clean":"2024-03-11T21:38:08.356845+0000","last_became_active":"2024-03-11T21:36:35.067500+0000","last_became_peered":"2024-03-11T21:36:35.067500+0000","last_unstale":"2024-03-11T21:38:08.356845+0000","last_undegraded":"2024-03-11T21:38:08.356845+0000","last_fullsized":"2024-03-11T21:38:08.356845+0000","mapping_epoch":277,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":278,"parent":"0.0","parent_split_bits":0,"last_scrub":"94'6","last_scrub_stamp":"2024-03-11T21:34:00.727416+0000","last_deep_scrub":"94'6","last_deep_scrub_stamp":"2024-03-11T21:28:24.603974+0000","last_clean_scrub_stamp":"2024-03-11T21:34:00.727416+0000","objects_scrubbed":2,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":12,"num_read_kb":10,"num_write":8,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6],"acting":[3,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":470,"reported_epoch":312,"state":"active+clean","last_fresh":"2024-03-11T21:38:08.364713+0000","last_change":"2024-03-11T21:36:35.070468+0000","last_active":"2024-03-11T21:38:08.364713+0000","last_peered":"2024-03-11T21:38:08.364713+0000","last_clean":"2024-03-11T21:38:08.364713+0000","last_became_active":"2024-03-11T21:36:35.070176+0000","last_became_peered":"2024-03-11T21:36:35.070176+0000","last_unstale":"2024-03-11T21:38:08.364713+0000","last_undegraded":"2024-03-11T21:38:08.364713+0000","last_fullsized":"2024-03-11T21:38:08.364713+0000","mapping_epoch":277,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":278,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:35:18.062272+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:43.559452+0000","last_clean_scrub_stamp":"2024-03-11T21:35:18.062272+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":537,"reported_epoch":311,"state":"active+clean","last_fresh":"2024-03-11T21:38:04.368326+0000","last_change":"2024-03-11T21:36:35.058551+0000","last_active":"2024-03-11T21:38:04.368326+0000","last_peered":"2024-03-11T21:38:04.368326+0000","last_clean":"2024-03-11T21:38:04.368326+0000","last_became_active":"2024-03-11T21:36:35.057906+0000","last_became_peered":"2024-03-11T21:36:35.057906+0000","last_unstale":"2024-03-11T21:38:04.368326+0000","last_undegraded":"2024-03-11T21:38:04.368326+0000","last_fullsized":"2024-03-11T21:38:04.368326+0000","mapping_epoch":277,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":278,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:34:15.516278+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:27:21.207303+0000","last_clean_scrub_stamp":"2024-03-11T21:34:15.516278+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.15","version":"0'0","reported_seq":544,"reported_epoch":311,"state":"active+clean","last_fresh":"2024-03-11T21:38:04.367450+0000","last_change":"2024-03-11T21:36:06.579754+0000","last_active":"2024-03-11T21:38:04.367450+0000","last_peered":"2024-03-11T21:38:04.367450+0000","last_clean":"2024-03-11T21:38:04.367450+0000","last_became_active":"2024-03-11T21:25:38.541705+0000","last_became_peered":"2024-03-11T21:25:38.541705+0000","last_unstale":"2024-03-11T21:38:04.367450+0000","last_undegraded":"2024-03-11T21:38:04.367450+0000","last_fullsized":"2024-03-11T21:38:04.367450+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:36:06.579636+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:26:44.860499+0000","last_clean_scrub_stamp":"2024-03-11T21:36:06.579636+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.13","version":"260'24","reported_seq":594,"reported_epoch":311,"state":"active+clean","last_fresh":"2024-03-11T21:38:04.368303+0000","last_change":"2024-03-11T21:35:14.453053+0000","last_active":"2024-03-11T21:38:04.368303+0000","last_peered":"2024-03-11T21:38:04.368303+0000","last_clean":"2024-03-11T21:38:04.368303+0000","last_became_active":"2024-03-11T21:26:11.214879+0000","last_became_peered":"2024-03-11T21:26:11.214879+0000","last_unstale":"2024-03-11T21:38:04.368303+0000","last_undegraded":"2024-03-11T21:38:04.368303+0000","last_fullsized":"2024-03-11T21:38:04.368303+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"109'16","last_scrub_stamp":"2024-03-11T21:35:14.452891+0000","last_deep_scrub":"109'16","last_deep_scrub_stamp":"2024-03-11T21:32:24.657490+0000","last_clean_scrub_stamp":"2024-03-11T21:35:14.452891+0000","objects_scrubbed":5,"log_size":24,"log_dups_size":0,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":32,"num_read_kb":25,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,2],"acting":[4,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.10","version":"0'0","reported_seq":534,"reported_epoch":312,"state":"active+clean","last_fresh":"2024-03-11T21:38:08.346332+0000","last_change":"2024-03-11T21:35:29.712658+0000","last_active":"2024-03-11T21:38:08.346332+0000","last_peered":"2024-03-11T21:38:08.346332+0000","last_clean":"2024-03-11T21:38:08.346332+0000","last_became_active":"2024-03-11T21:25:44.591189+0000","last_became_peered":"2024-03-11T21:25:44.591189+0000","last_unstale":"2024-03-11T21:38:08.346332+0000","last_undegraded":"2024-03-11T21:38:08.346332+0000","last_fullsized":"2024-03-11T21:38:08.346332+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:35:29.712557+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:27:25.417450+0000","last_clean_scrub_stamp":"2024-03-11T21:35:29.712557+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,1],"acting":[0,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":468,"reported_epoch":312,"state":"active+clean","last_fresh":"2024-03-11T21:38:08.367929+0000","last_change":"2024-03-11T21:36:35.062276+0000","last_active":"2024-03-11T21:38:08.367929+0000","last_peered":"2024-03-11T21:38:08.367929+0000","last_clean":"2024-03-11T21:38:08.367929+0000","last_became_active":"2024-03-11T21:36:35.056619+0000","last_became_peered":"2024-03-11T21:36:35.056619+0000","last_unstale":"2024-03-11T21:38:08.367929+0000","last_undegraded":"2024-03-11T21:38:08.367929+0000","last_fullsized":"2024-03-11T21:38:08.367929+0000","mapping_epoch":277,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":278,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:35:13.060487+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:31:15.367898+0000","last_clean_scrub_stamp":"2024-03-11T21:35:13.060487+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.14","version":"0'0","reported_seq":561,"reported_epoch":311,"state":"active+clean","last_fresh":"2024-03-11T21:38:04.385985+0000","last_change":"2024-03-11T21:36:35.068386+0000","last_active":"2024-03-11T21:38:04.385985+0000","last_peered":"2024-03-11T21:38:04.385985+0000","last_clean":"2024-03-11T21:38:04.385985+0000","last_became_active":"2024-03-11T21:36:35.068038+0000","last_became_peered":"2024-03-11T21:36:35.068038+0000","last_unstale":"2024-03-11T21:38:04.385985+0000","last_undegraded":"2024-03-11T21:38:04.385985+0000","last_fullsized":"2024-03-11T21:38:04.385985+0000","mapping_epoch":277,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":278,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:35:09.122761+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:37.512018+0000","last_clean_scrub_stamp":"2024-03-11T21:35:09.122761+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,5],"acting":[6,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.12","version":"260'15","reported_seq":565,"reported_epoch":311,"state":"active+clean","last_fresh":"2024-03-11T21:38:04.368144+0000","last_change":"2024-03-11T21:35:32.590439+0000","last_active":"2024-03-11T21:38:04.368144+0000","last_peered":"2024-03-11T21:38:04.368144+0000","last_clean":"2024-03-11T21:38:04.368144+0000","last_became_active":"2024-03-11T21:25:40.547082+0000","last_became_peered":"2024-03-11T21:25:40.547082+0000","last_unstale":"2024-03-11T21:38:04.368144+0000","last_undegraded":"2024-03-11T21:38:04.368144+0000","last_fullsized":"2024-03-11T21:38:04.368144+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'9","last_scrub_stamp":"2024-03-11T21:35:32.590318+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:39.525554+0000","last_clean_scrub_stamp":"2024-03-11T21:35:32.590318+0000","objects_scrubbed":4,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":532,"reported_epoch":311,"state":"active+clean","last_fresh":"2024-03-11T21:38:04.368938+0000","last_change":"2024-03-11T21:36:04.754168+0000","last_active":"2024-03-11T21:38:04.368938+0000","last_peered":"2024-03-11T21:38:04.368938+0000","last_clean":"2024-03-11T21:38:04.368938+0000","last_became_active":"2024-03-11T21:25:44.589586+0000","last_became_peered":"2024-03-11T21:25:44.589586+0000","last_unstale":"2024-03-11T21:38:04.368938+0000","last_undegraded":"2024-03-11T21:38:04.368938+0000","last_fullsized":"2024-03-11T21:38:04.368938+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:36:04.753979+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:27:09.801138+0000","last_clean_scrub_stamp":"2024-03-11T21:36:04.753979+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":113,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,5],"acting":[4,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.14","version":"280'12","reported_seq":648,"reported_epoch":312,"state":"active+clean","last_fresh":"2024-03-11T21:38:08.365510+0000","last_change":"2024-03-11T21:36:35.074550+0000","last_active":"2024-03-11T21:38:08.365510+0000","last_peered":"2024-03-11T21:38:08.365510+0000","last_clean":"2024-03-11T21:38:08.365510+0000","last_became_active":"2024-03-11T21:36:35.073912+0000","last_became_peered":"2024-03-11T21:36:35.073912+0000","last_unstale":"2024-03-11T21:38:08.365510+0000","last_undegraded":"2024-03-11T21:38:08.365510+0000","last_fullsized":"2024-03-11T21:38:08.365510+0000","mapping_epoch":277,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":278,"parent":"0.0","parent_split_bits":0,"last_scrub":"229'10","last_scrub_stamp":"2024-03-11T21:35:26.182107+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:41.544550+0000","last_clean_scrub_stamp":"2024-03-11T21:35:26.182107+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,2],"acting":[3,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.13","version":"0'0","reported_seq":547,"reported_epoch":312,"state":"active+clean","last_fresh":"2024-03-11T21:38:08.349518+0000","last_change":"2024-03-11T21:36:35.057688+0000","last_active":"2024-03-11T21:38:08.349518+0000","last_peered":"2024-03-11T21:38:08.349518+0000","last_clean":"2024-03-11T21:38:08.349518+0000","last_became_active":"2024-03-11T21:36:35.057132+0000","last_became_peered":"2024-03-11T21:36:35.057132+0000","last_unstale":"2024-03-11T21:38:08.349518+0000","last_undegraded":"2024-03-11T21:38:08.349518+0000","last_fullsized":"2024-03-11T21:38:08.349518+0000","mapping_epoch":277,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":278,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:34:08.650371+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:37.512018+0000","last_clean_scrub_stamp":"2024-03-11T21:34:08.650371+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.15","version":"260'19","reported_seq":618,"reported_epoch":311,"state":"active+clean","last_fresh":"2024-03-11T21:38:04.397487+0000","last_change":"2024-03-11T21:36:35.088986+0000","last_active":"2024-03-11T21:38:04.397487+0000","last_peered":"2024-03-11T21:38:04.397487+0000","last_clean":"2024-03-11T21:38:04.397487+0000","last_became_active":"2024-03-11T21:36:35.057285+0000","last_became_peered":"2024-03-11T21:36:35.057285+0000","last_unstale":"2024-03-11T21:38:04.397487+0000","last_undegraded":"2024-03-11T21:38:04.397487+0000","last_fullsized":"2024-03-11T21:38:04.397487+0000","mapping_epoch":277,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":278,"parent":"0.0","parent_split_bits":0,"last_scrub":"94'13","last_scrub_stamp":"2024-03-11T21:35:29.178543+0000","last_deep_scrub":"94'13","last_deep_scrub_stamp":"2024-03-11T21:29:41.853158+0000","last_clean_scrub_stamp":"2024-03-11T21:35:29.178543+0000","objects_scrubbed":4,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":11,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.16","version":"0'0","reported_seq":555,"reported_epoch":312,"state":"active+clean","last_fresh":"2024-03-11T21:38:08.348228+0000","last_change":"2024-03-11T21:36:35.058372+0000","last_active":"2024-03-11T21:38:08.348228+0000","last_peered":"2024-03-11T21:38:08.348228+0000","last_clean":"2024-03-11T21:38:08.348228+0000","last_became_active":"2024-03-11T21:36:35.057355+0000","last_became_peered":"2024-03-11T21:36:35.057355+0000","last_unstale":"2024-03-11T21:38:08.348228+0000","last_undegraded":"2024-03-11T21:38:08.348228+0000","last_fullsized":"2024-03-11T21:38:08.348228+0000","mapping_epoch":277,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":278,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:35:07.665075+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:43.559452+0000","last_clean_scrub_stamp":"2024-03-11T21:35:07.665075+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":35,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.15","version":"98'9","reported_seq":694,"reported_epoch":311,"state":"active+clean","last_fresh":"2024-03-11T21:38:04.397648+0000","last_change":"2024-03-11T21:36:03.366581+0000","last_active":"2024-03-11T21:38:04.397648+0000","last_peered":"2024-03-11T21:38:04.397648+0000","last_clean":"2024-03-11T21:38:04.397648+0000","last_became_active":"2024-03-11T21:25:42.572555+0000","last_became_peered":"2024-03-11T21:25:42.572555+0000","last_unstale":"2024-03-11T21:38:04.397648+0000","last_undegraded":"2024-03-11T21:38:04.397648+0000","last_fullsized":"2024-03-11T21:38:04.397648+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"98'9","last_scrub_stamp":"2024-03-11T21:36:03.366450+0000","last_deep_scrub":"98'9","last_deep_scrub_stamp":"2024-03-11T21:31:20.624923+0000","last_clean_scrub_stamp":"2024-03-11T21:36:03.366450+0000","objects_scrubbed":1,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.12","version":"0'0","reported_seq":564,"reported_epoch":311,"state":"active+clean","last_fresh":"2024-03-11T21:38:04.392731+0000","last_change":"2024-03-11T21:36:35.058369+0000","last_active":"2024-03-11T21:38:04.392731+0000","last_peered":"2024-03-11T21:38:04.392731+0000","last_clean":"2024-03-11T21:38:04.392731+0000","last_became_active":"2024-03-11T21:36:35.058033+0000","last_became_peered":"2024-03-11T21:36:35.058033+0000","last_unstale":"2024-03-11T21:38:04.392731+0000","last_undegraded":"2024-03-11T21:38:04.392731+0000","last_fullsized":"2024-03-11T21:38:04.392731+0000","mapping_epoch":277,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":278,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:35:27.158468+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:37.512018+0000","last_clean_scrub_stamp":"2024-03-11T21:35:27.158468+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.14","version":"260'20","reported_seq":551,"reported_epoch":312,"state":"active+clean","last_fresh":"2024-03-11T21:38:08.365202+0000","last_change":"2024-03-11T21:36:35.095382+0000","last_active":"2024-03-11T21:38:08.365202+0000","last_peered":"2024-03-11T21:38:08.365202+0000","last_clean":"2024-03-11T21:38:08.365202+0000","last_became_active":"2024-03-11T21:36:35.069638+0000","last_became_peered":"2024-03-11T21:36:35.069638+0000","last_unstale":"2024-03-11T21:38:08.365202+0000","last_undegraded":"2024-03-11T21:38:08.365202+0000","last_fullsized":"2024-03-11T21:38:08.365202+0000","mapping_epoch":277,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":278,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'10","last_scrub_stamp":"2024-03-11T21:35:28.189296+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:39.525554+0000","last_clean_scrub_stamp":"2024-03-11T21:35:28.189296+0000","objects_scrubbed":0,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":29,"num_read_kb":24,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.15","version":"0'0","reported_seq":188,"reported_epoch":311,"state":"active+clean","last_fresh":"2024-03-11T21:38:04.394850+0000","last_change":"2024-03-11T21:36:14.736240+0000","last_active":"2024-03-11T21:38:04.394850+0000","last_peered":"2024-03-11T21:38:04.394850+0000","last_clean":"2024-03-11T21:38:04.394850+0000","last_became_active":"2024-03-11T21:34:41.142760+0000","last_became_peered":"2024-03-11T21:34:41.142760+0000","last_unstale":"2024-03-11T21:38:04.394850+0000","last_undegraded":"2024-03-11T21:38:04.394850+0000","last_fullsized":"2024-03-11T21:38:04.394850+0000","mapping_epoch":226,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":227,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:36:14.735985+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:43.559452+0000","last_clean_scrub_stamp":"2024-03-11T21:36:14.735985+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":36,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":536,"reported_epoch":311,"state":"active+clean","last_fresh":"2024-03-11T21:38:04.396944+0000","last_change":"2024-03-11T21:36:35.060666+0000","last_active":"2024-03-11T21:38:04.396944+0000","last_peered":"2024-03-11T21:38:04.396944+0000","last_clean":"2024-03-11T21:38:04.396944+0000","last_became_active":"2024-03-11T21:36:35.060342+0000","last_became_peered":"2024-03-11T21:36:35.060342+0000","last_unstale":"2024-03-11T21:38:04.396944+0000","last_undegraded":"2024-03-11T21:38:04.396944+0000","last_fullsized":"2024-03-11T21:38:04.396944+0000","mapping_epoch":277,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":278,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:34:09.244852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:41.544550+0000","last_clean_scrub_stamp":"2024-03-11T21:34:09.244852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,1],"acting":[5,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.11","version":"0'0","reported_seq":544,"reported_epoch":311,"state":"active+clean","last_fresh":"2024-03-11T21:38:04.386240+0000","last_change":"2024-03-11T21:35:08.152853+0000","last_active":"2024-03-11T21:38:04.386240+0000","last_peered":"2024-03-11T21:38:04.386240+0000","last_clean":"2024-03-11T21:38:04.386240+0000","last_became_active":"2024-03-11T21:25:38.537415+0000","last_became_peered":"2024-03-11T21:25:38.537415+0000","last_unstale":"2024-03-11T21:38:04.386240+0000","last_undegraded":"2024-03-11T21:38:04.386240+0000","last_fullsized":"2024-03-11T21:38:04.386240+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:35:08.152740+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:37.512018+0000","last_clean_scrub_stamp":"2024-03-11T21:35:08.152740+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.17","version":"260'15","reported_seq":496,"reported_epoch":312,"state":"active+clean","last_fresh":"2024-03-11T21:38:08.367295+0000","last_change":"2024-03-11T21:36:35.099451+0000","last_active":"2024-03-11T21:38:08.367295+0000","last_peered":"2024-03-11T21:38:08.367295+0000","last_clean":"2024-03-11T21:38:08.367295+0000","last_became_active":"2024-03-11T21:36:35.071346+0000","last_became_peered":"2024-03-11T21:36:35.071346+0000","last_unstale":"2024-03-11T21:38:08.367295+0000","last_undegraded":"2024-03-11T21:38:08.367295+0000","last_fullsized":"2024-03-11T21:38:08.367295+0000","mapping_epoch":277,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":278,"parent":"0.0","parent_split_bits":0,"last_scrub":"97'9","last_scrub_stamp":"2024-03-11T21:32:08.495352+0000","last_deep_scrub":"97'9","last_deep_scrub_stamp":"2024-03-11T21:27:25.803132+0000","last_clean_scrub_stamp":"2024-03-11T21:32:08.495352+0000","objects_scrubbed":3,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":17,"num_read_kb":14,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,5],"acting":[3,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.14","version":"0'0","reported_seq":543,"reported_epoch":311,"state":"active+clean","last_fresh":"2024-03-11T21:38:04.365887+0000","last_change":"2024-03-11T21:35:27.606664+0000","last_active":"2024-03-11T21:38:04.365887+0000","last_peered":"2024-03-11T21:38:04.365887+0000","last_clean":"2024-03-11T21:38:04.365887+0000","last_became_active":"2024-03-11T21:34:41.136946+0000","last_became_peered":"2024-03-11T21:34:41.136946+0000","last_unstale":"2024-03-11T21:38:04.365887+0000","last_undegraded":"2024-03-11T21:38:04.365887+0000","last_fullsized":"2024-03-11T21:38:04.365887+0000","mapping_epoch":226,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":227,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:35:27.606551+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:43.559452+0000","last_clean_scrub_stamp":"2024-03-11T21:35:27.606551+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,7],"acting":[2,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":450,"reported_epoch":312,"state":"active+clean","last_fresh":"2024-03-11T21:38:08.367021+0000","last_change":"2024-03-11T21:36:35.076782+0000","last_active":"2024-03-11T21:38:08.367021+0000","last_peered":"2024-03-11T21:38:08.367021+0000","last_clean":"2024-03-11T21:38:08.367021+0000","last_became_active":"2024-03-11T21:36:35.076335+0000","last_became_peered":"2024-03-11T21:36:35.076335+0000","last_unstale":"2024-03-11T21:38:08.367021+0000","last_undegraded":"2024-03-11T21:38:08.367021+0000","last_fullsized":"2024-03-11T21:38:08.367021+0000","mapping_epoch":277,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":278,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:34:33.082577+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:41.544550+0000","last_clean_scrub_stamp":"2024-03-11T21:34:33.082577+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":35,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.10","version":"0'0","reported_seq":543,"reported_epoch":311,"state":"active+clean","last_fresh":"2024-03-11T21:38:04.365139+0000","last_change":"2024-03-11T21:35:37.694623+0000","last_active":"2024-03-11T21:38:04.365139+0000","last_peered":"2024-03-11T21:38:04.365139+0000","last_clean":"2024-03-11T21:38:04.365139+0000","last_became_active":"2024-03-11T21:25:38.533138+0000","last_became_peered":"2024-03-11T21:25:38.533138+0000","last_unstale":"2024-03-11T21:38:04.365139+0000","last_undegraded":"2024-03-11T21:38:04.365139+0000","last_fullsized":"2024-03-11T21:38:04.365139+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:35:37.694533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:37.512018+0000","last_clean_scrub_stamp":"2024-03-11T21:35:37.694533+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,0],"acting":[2,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.16","version":"260'15","reported_seq":617,"reported_epoch":312,"state":"active+clean","last_fresh":"2024-03-11T21:38:08.348423+0000","last_change":"2024-03-11T21:36:35.080158+0000","last_active":"2024-03-11T21:38:08.348423+0000","last_peered":"2024-03-11T21:38:08.348423+0000","last_clean":"2024-03-11T21:38:08.348423+0000","last_became_active":"2024-03-11T21:36:35.056862+0000","last_became_peered":"2024-03-11T21:36:35.056862+0000","last_unstale":"2024-03-11T21:38:08.348423+0000","last_undegraded":"2024-03-11T21:38:08.348423+0000","last_fullsized":"2024-03-11T21:38:08.348423+0000","mapping_epoch":277,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":278,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'9","last_scrub_stamp":"2024-03-11T21:35:12.741225+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:39.525554+0000","last_clean_scrub_stamp":"2024-03-11T21:35:12.741225+0000","objects_scrubbed":4,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":11,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.4","version":"311'9410","reported_seq":9525,"reported_epoch":311,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-03-11T21:38:07.720729+0000","last_change":"2024-03-11T21:36:36.516780+0000","last_active":"2024-03-11T21:38:07.720729+0000","last_peered":"2024-03-11T21:38:07.720729+0000","last_clean":"2024-03-11T21:35:44.669493+0000","last_became_active":"2024-03-11T21:36:36.506468+0000","last_became_peered":"2024-03-11T21:36:36.506468+0000","last_unstale":"2024-03-11T21:38:07.720729+0000","last_undegraded":"2024-03-11T21:36:36.069183+0000","last_fullsized":"2024-03-11T21:36:36.038484+0000","mapping_epoch":278,"log_start":"274'4400","ondisk_log_start":"274'4400","created":259,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:35:44.669493+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:35:44.669493+0000","last_clean_scrub_stamp":"2024-03-11T21:35:44.669493+0000","objects_scrubbed":0,"log_size":5010,"log_dups_size":0,"ondisk_log_size":5010,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":350355456,"num_objects":5346,"num_object_clones":0,"num_object_copies":16038,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4816,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5346,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9410,"num_write_kb":472192,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,3],"acting":[1,4],"avail_no_missing":["1","4"],"object_location_counts":[{"shards":"1,4","objects":530}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.b","version":"0'0","reported_seq":438,"reported_epoch":312,"state":"active+clean","last_fresh":"2024-03-11T21:38:08.367717+0000","last_change":"2024-03-11T21:36:35.074815+0000","last_active":"2024-03-11T21:38:08.367717+0000","last_peered":"2024-03-11T21:38:08.367717+0000","last_clean":"2024-03-11T21:38:08.367717+0000","last_became_active":"2024-03-11T21:36:35.072147+0000","last_became_peered":"2024-03-11T21:36:35.072147+0000","last_unstale":"2024-03-11T21:38:08.367717+0000","last_undegraded":"2024-03-11T21:38:08.367717+0000","last_fullsized":"2024-03-11T21:38:08.367717+0000","mapping_epoch":277,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":278,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:32:22.598150+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:43.559452+0000","last_clean_scrub_stamp":"2024-03-11T21:32:22.598150+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":57,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.f","version":"67'2","reported_seq":556,"reported_epoch":311,"state":"active+clean","last_fresh":"2024-03-11T21:38:04.368837+0000","last_change":"2024-03-11T21:35:36.613667+0000","last_active":"2024-03-11T21:38:04.368837+0000","last_peered":"2024-03-11T21:38:04.368837+0000","last_clean":"2024-03-11T21:38:04.368837+0000","last_became_active":"2024-03-11T21:25:38.537711+0000","last_became_peered":"2024-03-11T21:25:38.537711+0000","last_unstale":"2024-03-11T21:38:04.368837+0000","last_undegraded":"2024-03-11T21:38:04.368837+0000","last_fullsized":"2024-03-11T21:38:04.368837+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'2","last_scrub_stamp":"2024-03-11T21:35:36.613516+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:37.512018+0000","last_clean_scrub_stamp":"2024-03-11T21:35:36.613516+0000","objects_scrubbed":2,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":92,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":10,"num_read_kb":10,"num_write":4,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,7],"acting":[4,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.9","version":"260'23","reported_seq":587,"reported_epoch":311,"state":"active+clean","last_fresh":"2024-03-11T21:38:04.368180+0000","last_change":"2024-03-11T21:36:35.089034+0000","last_active":"2024-03-11T21:38:04.368180+0000","last_peered":"2024-03-11T21:38:04.368180+0000","last_clean":"2024-03-11T21:38:04.368180+0000","last_became_active":"2024-03-11T21:36:35.059930+0000","last_became_peered":"2024-03-11T21:36:35.059930+0000","last_unstale":"2024-03-11T21:38:04.368180+0000","last_undegraded":"2024-03-11T21:38:04.368180+0000","last_fullsized":"2024-03-11T21:38:04.368180+0000","mapping_epoch":277,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":278,"parent":"0.0","parent_split_bits":0,"last_scrub":"146'17","last_scrub_stamp":"2024-03-11T21:34:26.765075+0000","last_deep_scrub":"146'17","last_deep_scrub_stamp":"2024-03-11T21:34:26.765075+0000","last_clean_scrub_stamp":"2024-03-11T21:34:26.765075+0000","objects_scrubbed":5,"log_size":23,"log_dups_size":0,"ondisk_log_size":23,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":34,"num_read_kb":25,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,3],"acting":[4,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":535,"reported_epoch":311,"state":"active+clean","last_fresh":"2024-03-11T21:38:04.365240+0000","last_change":"2024-03-11T21:35:16.643421+0000","last_active":"2024-03-11T21:38:04.365240+0000","last_peered":"2024-03-11T21:38:04.365240+0000","last_clean":"2024-03-11T21:38:04.365240+0000","last_became_active":"2024-03-11T21:25:42.573987+0000","last_became_peered":"2024-03-11T21:25:42.573987+0000","last_unstale":"2024-03-11T21:38:04.365240+0000","last_undegraded":"2024-03-11T21:38:04.365240+0000","last_fullsized":"2024-03-11T21:38:04.365240+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:35:16.643311+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:41.544550+0000","last_clean_scrub_stamp":"2024-03-11T21:35:16.643311+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"9.f","version":"311'9212","reported_seq":9329,"reported_epoch":311,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-03-11T21:38:07.383283+0000","last_change":"2024-03-11T21:36:36.519822+0000","last_active":"2024-03-11T21:38:07.383283+0000","last_peered":"2024-03-11T21:38:07.383283+0000","last_clean":"2024-03-11T21:35:44.669493+0000","last_became_active":"2024-03-11T21:36:36.508836+0000","last_became_peered":"2024-03-11T21:36:36.508836+0000","last_unstale":"2024-03-11T21:38:07.383283+0000","last_undegraded":"2024-03-11T21:36:36.176770+0000","last_fullsized":"2024-03-11T21:36:36.038651+0000","mapping_epoch":278,"log_start":"274'4000","ondisk_log_start":"274'4000","created":259,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:35:44.669493+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:35:44.669493+0000","last_clean_scrub_stamp":"2024-03-11T21:35:44.669493+0000","objects_scrubbed":0,"log_size":5212,"log_dups_size":0,"ondisk_log_size":5212,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":338165760,"num_objects":5160,"num_object_clones":0,"num_object_copies":15480,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4738,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5160,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9212,"num_write_kb":459904,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":65536,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3],"acting":[5,1],"avail_no_missing":["5","1"],"object_location_counts":[{"shards":"1,5","objects":422}],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.0","version":"0'0","reported_seq":536,"reported_epoch":312,"state":"active+clean","last_fresh":"2024-03-11T21:38:08.346428+0000","last_change":"2024-03-11T21:36:35.058390+0000","last_active":"2024-03-11T21:38:08.346428+0000","last_peered":"2024-03-11T21:38:08.346428+0000","last_clean":"2024-03-11T21:38:08.346428+0000","last_became_active":"2024-03-11T21:36:35.058054+0000","last_became_peered":"2024-03-11T21:36:35.058054+0000","last_unstale":"2024-03-11T21:38:08.346428+0000","last_undegraded":"2024-03-11T21:38:08.346428+0000","last_fullsized":"2024-03-11T21:38:08.346428+0000","mapping_epoch":277,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":278,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:35:04.626668+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:29:39.498784+0000","last_clean_scrub_stamp":"2024-03-11T21:35:04.626668+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":39,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":557,"reported_epoch":311,"state":"active+clean","last_fresh":"2024-03-11T21:38:04.369499+0000","last_change":"2024-03-11T21:35:33.608063+0000","last_active":"2024-03-11T21:38:04.369499+0000","last_peered":"2024-03-11T21:38:04.369499+0000","last_clean":"2024-03-11T21:38:04.369499+0000","last_became_active":"2024-03-11T21:34:41.150913+0000","last_became_peered":"2024-03-11T21:34:41.150913+0000","last_unstale":"2024-03-11T21:38:04.369499+0000","last_undegraded":"2024-03-11T21:38:04.369499+0000","last_fullsized":"2024-03-11T21:38:04.369499+0000","mapping_epoch":226,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":227,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:35:33.607969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:37.512018+0000","last_clean_scrub_stamp":"2024-03-11T21:35:33.607969+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,7],"acting":[1,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.2","version":"260'25","reported_seq":575,"reported_epoch":311,"state":"active+clean","last_fresh":"2024-03-11T21:38:04.368234+0000","last_change":"2024-03-11T21:35:17.666749+0000","last_active":"2024-03-11T21:38:04.368234+0000","last_peered":"2024-03-11T21:38:04.368234+0000","last_clean":"2024-03-11T21:38:04.368234+0000","last_became_active":"2024-03-11T21:25:40.548833+0000","last_became_peered":"2024-03-11T21:25:40.548833+0000","last_unstale":"2024-03-11T21:38:04.368234+0000","last_undegraded":"2024-03-11T21:38:04.368234+0000","last_fullsized":"2024-03-11T21:38:04.368234+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"94'15","last_scrub_stamp":"2024-03-11T21:35:17.666498+0000","last_deep_scrub":"94'15","last_deep_scrub_stamp":"2024-03-11T21:26:49.993060+0000","last_clean_scrub_stamp":"2024-03-11T21:35:17.666498+0000","objects_scrubbed":5,"log_size":25,"log_dups_size":0,"ondisk_log_size":25,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":20,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":25,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.3","version":"74'8","reported_seq":696,"reported_epoch":312,"state":"active+clean","last_fresh":"2024-03-11T21:38:08.349724+0000","last_change":"2024-03-11T21:35:31.695369+0000","last_active":"2024-03-11T21:38:08.349724+0000","last_peered":"2024-03-11T21:38:08.349724+0000","last_clean":"2024-03-11T21:38:08.349724+0000","last_became_active":"2024-03-11T21:25:42.578161+0000","last_became_peered":"2024-03-11T21:25:42.578161+0000","last_unstale":"2024-03-11T21:38:08.349724+0000","last_undegraded":"2024-03-11T21:38:08.349724+0000","last_fullsized":"2024-03-11T21:38:08.349724+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'8","last_scrub_stamp":"2024-03-11T21:35:31.695263+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:41.544550+0000","last_clean_scrub_stamp":"2024-03-11T21:35:31.695263+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,5],"acting":[0,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.9","version":"312'9469","reported_seq":9586,"reported_epoch":312,"state":"active+clean","last_fresh":"2024-03-11T21:38:08.511211+0000","last_change":"2024-03-11T21:35:45.696767+0000","last_active":"2024-03-11T21:38:08.511211+0000","last_peered":"2024-03-11T21:38:08.511211+0000","last_clean":"2024-03-11T21:38:08.511211+0000","last_became_active":"2024-03-11T21:35:45.696379+0000","last_became_peered":"2024-03-11T21:35:45.696379+0000","last_unstale":"2024-03-11T21:38:08.511211+0000","last_undegraded":"2024-03-11T21:38:08.511211+0000","last_fullsized":"2024-03-11T21:38:08.511211+0000","mapping_epoch":259,"log_start":"274'3900","ondisk_log_start":"274'3900","created":259,"last_epoch_clean":260,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:35:44.669493+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:35:44.669493+0000","last_clean_scrub_stamp":"2024-03-11T21:35:44.669493+0000","objects_scrubbed":0,"log_size":5569,"log_dups_size":0,"ondisk_log_size":5569,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":348717056,"num_objects":5321,"num_object_clones":0,"num_object_copies":15963,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5321,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9469,"num_write_kb":473280,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,5],"acting":[0,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":476,"reported_epoch":312,"state":"active+clean","last_fresh":"2024-03-11T21:38:08.357401+0000","last_change":"2024-03-11T21:36:35.068977+0000","last_active":"2024-03-11T21:38:08.357401+0000","last_peered":"2024-03-11T21:38:08.357401+0000","last_clean":"2024-03-11T21:38:08.357401+0000","last_became_active":"2024-03-11T21:36:35.067288+0000","last_became_peered":"2024-03-11T21:36:35.067288+0000","last_unstale":"2024-03-11T21:38:08.357401+0000","last_undegraded":"2024-03-11T21:38:08.357401+0000","last_fullsized":"2024-03-11T21:38:08.357401+0000","mapping_epoch":277,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":278,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:35:09.985377+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:28:30.563712+0000","last_clean_scrub_stamp":"2024-03-11T21:35:09.985377+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,7],"acting":[3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":551,"reported_epoch":311,"state":"active+clean","last_fresh":"2024-03-11T21:38:04.392820+0000","last_change":"2024-03-11T21:35:40.116979+0000","last_active":"2024-03-11T21:38:04.392820+0000","last_peered":"2024-03-11T21:38:04.392820+0000","last_clean":"2024-03-11T21:38:04.392820+0000","last_became_active":"2024-03-11T21:25:38.538540+0000","last_became_peered":"2024-03-11T21:25:38.538540+0000","last_unstale":"2024-03-11T21:38:04.392820+0000","last_undegraded":"2024-03-11T21:38:04.392820+0000","last_fullsized":"2024-03-11T21:38:04.392820+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:35:40.116877+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:37.512018+0000","last_clean_scrub_stamp":"2024-03-11T21:35:40.116877+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.4","version":"280'42","reported_seq":773,"reported_epoch":311,"state":"active+clean","last_fresh":"2024-03-11T21:38:04.369366+0000","last_change":"2024-03-11T21:36:35.095935+0000","last_active":"2024-03-11T21:38:04.369366+0000","last_peered":"2024-03-11T21:38:04.369366+0000","last_clean":"2024-03-11T21:38:04.369366+0000","last_became_active":"2024-03-11T21:36:35.073430+0000","last_became_peered":"2024-03-11T21:36:35.073430+0000","last_unstale":"2024-03-11T21:38:04.369366+0000","last_undegraded":"2024-03-11T21:38:04.369366+0000","last_fullsized":"2024-03-11T21:38:04.369366+0000","mapping_epoch":277,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":278,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'28","last_scrub_stamp":"2024-03-11T21:34:19.513252+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:39.525554+0000","last_clean_scrub_stamp":"2024-03-11T21:34:19.513252+0000","objects_scrubbed":10,"log_size":42,"log_dups_size":0,"ondisk_log_size":42,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":358,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":66,"num_read_kb":51,"num_write":38,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3],"acting":[1,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":538,"reported_epoch":312,"state":"active+clean","last_fresh":"2024-03-11T21:38:08.348456+0000","last_change":"2024-03-11T21:35:32.651731+0000","last_active":"2024-03-11T21:38:08.348456+0000","last_peered":"2024-03-11T21:38:08.348456+0000","last_clean":"2024-03-11T21:38:08.348456+0000","last_became_active":"2024-03-11T21:25:42.572673+0000","last_became_peered":"2024-03-11T21:25:42.572673+0000","last_unstale":"2024-03-11T21:38:08.348456+0000","last_undegraded":"2024-03-11T21:38:08.348456+0000","last_fullsized":"2024-03-11T21:38:08.348456+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:35:32.651652+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:41.544550+0000","last_clean_scrub_stamp":"2024-03-11T21:35:32.651652+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.a","version":"311'9327","reported_seq":9442,"reported_epoch":311,"state":"active+clean","last_fresh":"2024-03-11T21:38:07.438511+0000","last_change":"2024-03-11T21:35:45.698558+0000","last_active":"2024-03-11T21:38:07.438511+0000","last_peered":"2024-03-11T21:38:07.438511+0000","last_clean":"2024-03-11T21:38:07.438511+0000","last_became_active":"2024-03-11T21:35:45.698238+0000","last_became_peered":"2024-03-11T21:35:45.698238+0000","last_unstale":"2024-03-11T21:38:07.438511+0000","last_undegraded":"2024-03-11T21:38:07.438511+0000","last_fullsized":"2024-03-11T21:38:07.438511+0000","mapping_epoch":259,"log_start":"274'4100","ondisk_log_start":"274'4100","created":259,"last_epoch_clean":260,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:35:44.669493+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:35:44.669493+0000","last_clean_scrub_stamp":"2024-03-11T21:35:44.669493+0000","objects_scrubbed":0,"log_size":5227,"log_dups_size":0,"ondisk_log_size":5227,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":357892096,"num_objects":5461,"num_object_clones":0,"num_object_copies":16383,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5461,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9327,"num_write_kb":473216,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,7],"acting":[5,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.5","version":"0'0","reported_seq":197,"reported_epoch":311,"state":"active+clean","last_fresh":"2024-03-11T21:38:04.394690+0000","last_change":"2024-03-11T21:36:35.069989+0000","last_active":"2024-03-11T21:38:04.394690+0000","last_peered":"2024-03-11T21:38:04.394690+0000","last_clean":"2024-03-11T21:38:04.394690+0000","last_became_active":"2024-03-11T21:36:35.069358+0000","last_became_peered":"2024-03-11T21:36:35.069358+0000","last_unstale":"2024-03-11T21:38:04.394690+0000","last_undegraded":"2024-03-11T21:38:04.394690+0000","last_fullsized":"2024-03-11T21:38:04.394690+0000","mapping_epoch":277,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":278,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:35:19.841632+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:43.559452+0000","last_clean_scrub_stamp":"2024-03-11T21:35:19.841632+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,3],"acting":[7,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.0","version":"164'6","reported_seq":1298,"reported_epoch":311,"state":"active+clean","last_fresh":"2024-03-11T21:38:06.506957+0000","last_change":"2024-03-11T21:35:38.518243+0000","last_active":"2024-03-11T21:38:06.506957+0000","last_peered":"2024-03-11T21:38:06.506957+0000","last_clean":"2024-03-11T21:38:06.506957+0000","last_became_active":"2024-03-11T21:25:38.536890+0000","last_became_peered":"2024-03-11T21:25:38.536890+0000","last_unstale":"2024-03-11T21:38:06.506957+0000","last_undegraded":"2024-03-11T21:38:06.506957+0000","last_fullsized":"2024-03-11T21:38:06.506957+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"164'6","last_scrub_stamp":"2024-03-11T21:35:38.518097+0000","last_deep_scrub":"164'6","last_deep_scrub_stamp":"2024-03-11T21:32:17.597666+0000","last_clean_scrub_stamp":"2024-03-11T21:35:38.518097+0000","objects_scrubbed":1,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0.000753559,"stat_sum":{"num_bytes":389,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":742,"num_read_kb":737,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,6],"acting":[1,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":553,"reported_epoch":311,"state":"active+clean","last_fresh":"2024-03-11T21:38:04.366669+0000","last_change":"2024-03-11T21:36:35.057287+0000","last_active":"2024-03-11T21:38:04.366669+0000","last_peered":"2024-03-11T21:38:04.366669+0000","last_clean":"2024-03-11T21:38:04.366669+0000","last_became_active":"2024-03-11T21:36:35.056966+0000","last_became_peered":"2024-03-11T21:36:35.056966+0000","last_unstale":"2024-03-11T21:38:04.366669+0000","last_undegraded":"2024-03-11T21:38:04.366669+0000","last_fullsized":"2024-03-11T21:38:04.366669+0000","mapping_epoch":277,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":278,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:34:18.424010+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:26:48.997454+0000","last_clean_scrub_stamp":"2024-03-11T21:34:18.424010+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.7","version":"260'22","reported_seq":581,"reported_epoch":311,"state":"active+clean","last_fresh":"2024-03-11T21:38:04.367014+0000","last_change":"2024-03-11T21:35:37.508679+0000","last_active":"2024-03-11T21:38:04.367014+0000","last_peered":"2024-03-11T21:38:04.367014+0000","last_clean":"2024-03-11T21:38:04.367014+0000","last_became_active":"2024-03-11T21:25:40.541769+0000","last_became_peered":"2024-03-11T21:25:40.541769+0000","last_unstale":"2024-03-11T21:38:04.367014+0000","last_undegraded":"2024-03-11T21:38:04.367014+0000","last_fullsized":"2024-03-11T21:38:04.367014+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"97'18","last_scrub_stamp":"2024-03-11T21:35:37.508567+0000","last_deep_scrub":"97'18","last_deep_scrub_stamp":"2024-03-11T21:27:19.239404+0000","last_clean_scrub_stamp":"2024-03-11T21:35:37.508567+0000","objects_scrubbed":5,"log_size":22,"log_dups_size":0,"ondisk_log_size":22,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":36,"num_read_kb":25,"num_write":20,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":547,"reported_epoch":311,"state":"active+clean","last_fresh":"2024-03-11T21:38:04.368657+0000","last_change":"2024-03-11T21:35:22.632822+0000","last_active":"2024-03-11T21:38:04.368657+0000","last_peered":"2024-03-11T21:38:04.368657+0000","last_clean":"2024-03-11T21:38:04.368657+0000","last_became_active":"2024-03-11T21:34:41.143660+0000","last_became_peered":"2024-03-11T21:34:41.143660+0000","last_unstale":"2024-03-11T21:38:04.368657+0000","last_undegraded":"2024-03-11T21:38:04.368657+0000","last_fullsized":"2024-03-11T21:38:04.368657+0000","mapping_epoch":226,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":227,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:35:22.632707+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:41.544550+0000","last_clean_scrub_stamp":"2024-03-11T21:35:22.632707+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":18,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7],"acting":[2,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"9.8","version":"312'9465","reported_seq":9579,"reported_epoch":312,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-03-11T21:38:08.506675+0000","last_change":"2024-03-11T21:36:36.520071+0000","last_active":"2024-03-11T21:38:08.506675+0000","last_peered":"2024-03-11T21:38:08.506675+0000","last_clean":"2024-03-11T21:35:44.669493+0000","last_became_active":"2024-03-11T21:36:36.507652+0000","last_became_peered":"2024-03-11T21:36:36.507652+0000","last_unstale":"2024-03-11T21:38:08.506675+0000","last_undegraded":"2024-03-11T21:36:36.064628+0000","last_fullsized":"2024-03-11T21:36:36.036628+0000","mapping_epoch":278,"log_start":"274'3900","ondisk_log_start":"274'3900","created":259,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:35:44.669493+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:35:44.669493+0000","last_clean_scrub_stamp":"2024-03-11T21:35:44.669493+0000","objects_scrubbed":0,"log_size":5565,"log_dups_size":0,"ondisk_log_size":5565,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":346357760,"num_objects":5285,"num_object_clones":0,"num_object_copies":15855,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4846,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5285,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9465,"num_write_kb":472000,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[0,5],"avail_no_missing":["0","5"],"object_location_counts":[{"shards":"0,5","objects":439}],"blocked_by":[],"up_primary":3,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.7","version":"124'2","reported_seq":534,"reported_epoch":311,"state":"active+clean","last_fresh":"2024-03-11T21:38:04.412886+0000","last_change":"2024-03-11T21:36:35.059190+0000","last_active":"2024-03-11T21:38:04.412886+0000","last_peered":"2024-03-11T21:38:04.412886+0000","last_clean":"2024-03-11T21:38:04.412886+0000","last_became_active":"2024-03-11T21:36:35.058786+0000","last_became_peered":"2024-03-11T21:36:35.058786+0000","last_unstale":"2024-03-11T21:38:04.412886+0000","last_undegraded":"2024-03-11T21:38:04.412886+0000","last_fullsized":"2024-03-11T21:38:04.412886+0000","mapping_epoch":277,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":278,"parent":"0.0","parent_split_bits":0,"last_scrub":"124'2","last_scrub_stamp":"2024-03-11T21:35:16.414595+0000","last_deep_scrub":"124'2","last_deep_scrub_stamp":"2024-03-11T21:29:34.956908+0000","last_clean_scrub_stamp":"2024-03-11T21:35:16.414595+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":42,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,4],"acting":[5,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2","version":"71'2","reported_seq":497,"reported_epoch":312,"state":"active+clean","last_fresh":"2024-03-11T21:38:08.374463+0000","last_change":"2024-03-11T21:36:35.071218+0000","last_active":"2024-03-11T21:38:08.374463+0000","last_peered":"2024-03-11T21:38:08.374463+0000","last_clean":"2024-03-11T21:38:08.374463+0000","last_became_active":"2024-03-11T21:36:35.070783+0000","last_became_peered":"2024-03-11T21:36:35.070783+0000","last_unstale":"2024-03-11T21:38:08.374463+0000","last_undegraded":"2024-03-11T21:38:08.374463+0000","last_fullsized":"2024-03-11T21:38:08.374463+0000","mapping_epoch":277,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":278,"parent":"0.0","parent_split_bits":0,"last_scrub":"71'2","last_scrub_stamp":"2024-03-11T21:32:21.543053+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:37.512018+0000","last_clean_scrub_stamp":"2024-03-11T21:32:21.543053+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014206449999999999,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":561,"reported_epoch":311,"state":"active+clean","last_fresh":"2024-03-11T21:38:04.393765+0000","last_change":"2024-03-11T21:35:08.391825+0000","last_active":"2024-03-11T21:38:04.393765+0000","last_peered":"2024-03-11T21:38:04.393765+0000","last_clean":"2024-03-11T21:38:04.393765+0000","last_became_active":"2024-03-11T21:34:41.146672+0000","last_became_peered":"2024-03-11T21:34:41.146672+0000","last_unstale":"2024-03-11T21:38:04.393765+0000","last_undegraded":"2024-03-11T21:38:04.393765+0000","last_fullsized":"2024-03-11T21:38:04.393765+0000","mapping_epoch":226,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":227,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:35:08.391713+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:37.512018+0000","last_clean_scrub_stamp":"2024-03-11T21:35:08.391713+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"87'128","reported_seq":659,"reported_epoch":311,"state":"active+clean","last_fresh":"2024-03-11T21:38:04.393157+0000","last_change":"2024-03-11T21:35:21.798272+0000","last_active":"2024-03-11T21:38:04.393157+0000","last_peered":"2024-03-11T21:38:04.393157+0000","last_clean":"2024-03-11T21:38:04.393157+0000","last_became_active":"2024-03-11T21:34:44.164569+0000","last_became_peered":"2024-03-11T21:34:44.164569+0000","last_unstale":"2024-03-11T21:38:04.393157+0000","last_undegraded":"2024-03-11T21:38:04.393157+0000","last_fullsized":"2024-03-11T21:38:04.393157+0000","mapping_epoch":229,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":230,"parent":"0.0","parent_split_bits":0,"last_scrub":"87'128","last_scrub_stamp":"2024-03-11T21:35:21.798144+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:24.261974+0000","last_clean_scrub_stamp":"2024-03-11T21:35:21.798144+0000","objects_scrubbed":2,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":198,"num_read_kb":294,"num_write":179,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":2951840,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.5","version":"260'28","reported_seq":738,"reported_epoch":311,"state":"active+clean","last_fresh":"2024-03-11T21:38:04.385758+0000","last_change":"2024-03-11T21:36:14.026933+0000","last_active":"2024-03-11T21:38:04.385758+0000","last_peered":"2024-03-11T21:38:04.385758+0000","last_clean":"2024-03-11T21:38:04.385758+0000","last_became_active":"2024-03-11T21:25:40.761457+0000","last_became_peered":"2024-03-11T21:25:40.761457+0000","last_unstale":"2024-03-11T21:38:04.385758+0000","last_undegraded":"2024-03-11T21:38:04.385758+0000","last_fullsized":"2024-03-11T21:38:04.385758+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"260'28","last_scrub_stamp":"2024-03-11T21:36:14.026775+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:39.525554+0000","last_clean_scrub_stamp":"2024-03-11T21:36:14.026775+0000","objects_scrubbed":8,"log_size":28,"log_dups_size":0,"ondisk_log_size":28,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":154,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":43,"num_read_kb":33,"num_write":25,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":188,"reported_epoch":311,"state":"active+clean","last_fresh":"2024-03-11T21:38:04.394365+0000","last_change":"2024-03-11T21:35:24.841170+0000","last_active":"2024-03-11T21:38:04.394365+0000","last_peered":"2024-03-11T21:38:04.394365+0000","last_clean":"2024-03-11T21:38:04.394365+0000","last_became_active":"2024-03-11T21:34:41.139782+0000","last_became_peered":"2024-03-11T21:34:41.139782+0000","last_unstale":"2024-03-11T21:38:04.394365+0000","last_undegraded":"2024-03-11T21:38:04.394365+0000","last_fullsized":"2024-03-11T21:38:04.394365+0000","mapping_epoch":226,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":227,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:35:24.841041+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:32:21.266672+0000","last_clean_scrub_stamp":"2024-03-11T21:35:24.841041+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,5],"acting":[7,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.b","version":"312'9266","reported_seq":9383,"reported_epoch":312,"state":"active+clean","last_fresh":"2024-03-11T21:38:08.468451+0000","last_change":"2024-03-11T21:35:45.697213+0000","last_active":"2024-03-11T21:38:08.468451+0000","last_peered":"2024-03-11T21:38:08.468451+0000","last_clean":"2024-03-11T21:38:08.468451+0000","last_became_active":"2024-03-11T21:35:45.696869+0000","last_became_peered":"2024-03-11T21:35:45.696869+0000","last_unstale":"2024-03-11T21:38:08.468451+0000","last_undegraded":"2024-03-11T21:38:08.468451+0000","last_fullsized":"2024-03-11T21:38:08.468451+0000","mapping_epoch":259,"log_start":"272'3700","ondisk_log_start":"272'3700","created":259,"last_epoch_clean":260,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:35:44.669493+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:35:44.669493+0000","last_clean_scrub_stamp":"2024-03-11T21:35:44.669493+0000","objects_scrubbed":0,"log_size":5566,"log_dups_size":0,"ondisk_log_size":5566,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":339476480,"num_objects":5180,"num_object_clones":0,"num_object_copies":15540,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5180,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9266,"num_write_kb":462272,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,5],"acting":[0,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.4","version":"0'0","reported_seq":533,"reported_epoch":311,"state":"active+clean","last_fresh":"2024-03-11T21:38:04.369699+0000","last_change":"2024-03-11T21:36:35.073305+0000","last_active":"2024-03-11T21:38:04.369699+0000","last_peered":"2024-03-11T21:38:04.369699+0000","last_clean":"2024-03-11T21:38:04.369699+0000","last_became_active":"2024-03-11T21:36:35.072824+0000","last_became_peered":"2024-03-11T21:36:35.072824+0000","last_unstale":"2024-03-11T21:38:04.369699+0000","last_undegraded":"2024-03-11T21:38:04.369699+0000","last_fullsized":"2024-03-11T21:38:04.369699+0000","mapping_epoch":277,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":278,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:34:23.531064+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:43.559452+0000","last_clean_scrub_stamp":"2024-03-11T21:34:23.531064+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":563,"reported_epoch":312,"state":"active+clean","last_fresh":"2024-03-11T21:38:08.349996+0000","last_change":"2024-03-11T21:36:35.070626+0000","last_active":"2024-03-11T21:38:08.349996+0000","last_peered":"2024-03-11T21:38:08.349996+0000","last_clean":"2024-03-11T21:38:08.349996+0000","last_became_active":"2024-03-11T21:36:35.069452+0000","last_became_peered":"2024-03-11T21:36:35.069452+0000","last_unstale":"2024-03-11T21:38:08.349996+0000","last_undegraded":"2024-03-11T21:38:08.349996+0000","last_fullsized":"2024-03-11T21:38:08.349996+0000","mapping_epoch":277,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":278,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:35:13.741947+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:37.512018+0000","last_clean_scrub_stamp":"2024-03-11T21:35:13.741947+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0.00089342499999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":180,"reported_epoch":311,"state":"active+clean","last_fresh":"2024-03-11T21:38:04.393051+0000","last_change":"2024-03-11T21:34:41.150606+0000","last_active":"2024-03-11T21:38:04.393051+0000","last_peered":"2024-03-11T21:38:04.393051+0000","last_clean":"2024-03-11T21:38:04.393051+0000","last_became_active":"2024-03-11T21:34:41.149219+0000","last_became_peered":"2024-03-11T21:34:41.149219+0000","last_unstale":"2024-03-11T21:38:04.393051+0000","last_undegraded":"2024-03-11T21:38:04.393051+0000","last_fullsized":"2024-03-11T21:38:04.393051+0000","mapping_epoch":226,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":227,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:34:31.628014+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:37.512018+0000","last_clean_scrub_stamp":"2024-03-11T21:34:31.628014+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,0],"acting":[7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.6","version":"260'24","reported_seq":598,"reported_epoch":312,"state":"active+clean","last_fresh":"2024-03-11T21:38:08.348332+0000","last_change":"2024-03-11T21:35:19.753433+0000","last_active":"2024-03-11T21:38:08.348332+0000","last_peered":"2024-03-11T21:38:08.348332+0000","last_clean":"2024-03-11T21:38:08.348332+0000","last_became_active":"2024-03-11T21:32:22.796486+0000","last_became_peered":"2024-03-11T21:32:22.796486+0000","last_unstale":"2024-03-11T21:38:08.348332+0000","last_undegraded":"2024-03-11T21:38:08.348332+0000","last_fullsized":"2024-03-11T21:38:08.348332+0000","mapping_epoch":166,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":167,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'12","last_scrub_stamp":"2024-03-11T21:35:19.753224+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:39.525554+0000","last_clean_scrub_stamp":"2024-03-11T21:35:19.753224+0000","objects_scrubbed":6,"log_size":24,"log_dups_size":0,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":36,"num_read_kb":30,"num_write":24,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":535,"reported_epoch":311,"state":"active+clean","last_fresh":"2024-03-11T21:38:04.412791+0000","last_change":"2024-03-11T21:36:07.451482+0000","last_active":"2024-03-11T21:38:04.412791+0000","last_peered":"2024-03-11T21:38:04.412791+0000","last_clean":"2024-03-11T21:38:04.412791+0000","last_became_active":"2024-03-11T21:25:42.572324+0000","last_became_peered":"2024-03-11T21:25:42.572324+0000","last_unstale":"2024-03-11T21:38:04.412791+0000","last_undegraded":"2024-03-11T21:38:04.412791+0000","last_fullsized":"2024-03-11T21:38:04.412791+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:36:07.451102+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:41.544550+0000","last_clean_scrub_stamp":"2024-03-11T21:36:07.451102+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":54,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.e","version":"311'9039","reported_seq":9154,"reported_epoch":311,"state":"active+clean","last_fresh":"2024-03-11T21:38:07.260923+0000","last_change":"2024-03-11T21:35:45.697331+0000","last_active":"2024-03-11T21:38:07.260923+0000","last_peered":"2024-03-11T21:38:07.260923+0000","last_clean":"2024-03-11T21:38:07.260923+0000","last_became_active":"2024-03-11T21:35:45.696948+0000","last_became_peered":"2024-03-11T21:35:45.696948+0000","last_unstale":"2024-03-11T21:38:07.260923+0000","last_undegraded":"2024-03-11T21:38:07.260923+0000","last_fullsized":"2024-03-11T21:38:07.260923+0000","mapping_epoch":259,"log_start":"274'4400","ondisk_log_start":"274'4400","created":259,"last_epoch_clean":260,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:35:44.669493+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:35:44.669493+0000","last_clean_scrub_stamp":"2024-03-11T21:35:44.669493+0000","objects_scrubbed":0,"log_size":4639,"log_dups_size":0,"ondisk_log_size":4639,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":343474176,"num_objects":5241,"num_object_clones":0,"num_object_copies":15723,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5241,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9039,"num_write_kb":456960,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":540,"reported_epoch":311,"state":"active+clean","last_fresh":"2024-03-11T21:38:04.363182+0000","last_change":"2024-03-11T21:36:18.529506+0000","last_active":"2024-03-11T21:38:04.363182+0000","last_peered":"2024-03-11T21:38:04.363182+0000","last_clean":"2024-03-11T21:38:04.363182+0000","last_became_active":"2024-03-11T21:25:45.301416+0000","last_became_peered":"2024-03-11T21:25:45.301416+0000","last_unstale":"2024-03-11T21:38:04.363182+0000","last_undegraded":"2024-03-11T21:38:04.363182+0000","last_fullsized":"2024-03-11T21:38:04.363182+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:36:18.529370+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:43.559452+0000","last_clean_scrub_stamp":"2024-03-11T21:36:18.529370+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":54,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":188,"reported_epoch":311,"state":"active+clean","last_fresh":"2024-03-11T21:38:04.394775+0000","last_change":"2024-03-11T21:35:33.740615+0000","last_active":"2024-03-11T21:38:04.394775+0000","last_peered":"2024-03-11T21:38:04.394775+0000","last_clean":"2024-03-11T21:38:04.394775+0000","last_became_active":"2024-03-11T21:34:41.152721+0000","last_became_peered":"2024-03-11T21:34:41.152721+0000","last_unstale":"2024-03-11T21:38:04.394775+0000","last_undegraded":"2024-03-11T21:38:04.394775+0000","last_fullsized":"2024-03-11T21:38:04.394775+0000","mapping_epoch":226,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":227,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:35:33.740538+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:26:57.333669+0000","last_clean_scrub_stamp":"2024-03-11T21:35:33.740538+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.3","version":"260'37","reported_seq":649,"reported_epoch":312,"state":"active+clean","last_fresh":"2024-03-11T21:38:08.349421+0000","last_change":"2024-03-11T21:35:11.697314+0000","last_active":"2024-03-11T21:38:08.349421+0000","last_peered":"2024-03-11T21:38:08.349421+0000","last_clean":"2024-03-11T21:38:08.349421+0000","last_became_active":"2024-03-11T21:34:41.145607+0000","last_became_peered":"2024-03-11T21:34:41.145607+0000","last_unstale":"2024-03-11T21:38:08.349421+0000","last_undegraded":"2024-03-11T21:38:08.349421+0000","last_fullsized":"2024-03-11T21:38:08.349421+0000","mapping_epoch":226,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":227,"parent":"0.0","parent_split_bits":0,"last_scrub":"165'27","last_scrub_stamp":"2024-03-11T21:35:11.697140+0000","last_deep_scrub":"165'27","last_deep_scrub_stamp":"2024-03-11T21:32:21.576732+0000","last_clean_scrub_stamp":"2024-03-11T21:35:11.697140+0000","objects_scrubbed":8,"log_size":37,"log_dups_size":0,"ondisk_log_size":37,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":54,"num_read_kb":40,"num_write":32,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":660,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":544,"reported_epoch":311,"state":"active+clean","last_fresh":"2024-03-11T21:38:04.385751+0000","last_change":"2024-03-11T21:35:12.091317+0000","last_active":"2024-03-11T21:38:04.385751+0000","last_peered":"2024-03-11T21:38:04.385751+0000","last_clean":"2024-03-11T21:38:04.385751+0000","last_became_active":"2024-03-11T21:25:42.579045+0000","last_became_peered":"2024-03-11T21:25:42.579045+0000","last_unstale":"2024-03-11T21:38:04.385751+0000","last_undegraded":"2024-03-11T21:38:04.385751+0000","last_fullsized":"2024-03-11T21:38:04.385751+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:35:12.091185+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:29:36.918936+0000","last_clean_scrub_stamp":"2024-03-11T21:35:12.091185+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.d","version":"311'9178","reported_seq":9294,"reported_epoch":311,"state":"active+clean","last_fresh":"2024-03-11T21:38:07.256168+0000","last_change":"2024-03-11T21:35:45.700060+0000","last_active":"2024-03-11T21:38:07.256168+0000","last_peered":"2024-03-11T21:38:07.256168+0000","last_clean":"2024-03-11T21:38:07.256168+0000","last_became_active":"2024-03-11T21:35:45.699674+0000","last_became_peered":"2024-03-11T21:35:45.699674+0000","last_unstale":"2024-03-11T21:38:07.256168+0000","last_undegraded":"2024-03-11T21:38:07.256168+0000","last_fullsized":"2024-03-11T21:38:07.256168+0000","mapping_epoch":259,"log_start":"274'4500","ondisk_log_start":"274'4500","created":259,"last_epoch_clean":260,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:35:44.669493+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:35:44.669493+0000","last_clean_scrub_stamp":"2024-03-11T21:35:44.669493+0000","objects_scrubbed":0,"log_size":4678,"log_dups_size":0,"ondisk_log_size":4678,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":346488856,"num_objects":5288,"num_object_clones":0,"num_object_copies":15864,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5288,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":9178,"num_write_kb":462849,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,5],"acting":[7,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":538,"reported_epoch":311,"state":"active+clean","last_fresh":"2024-03-11T21:38:04.368507+0000","last_change":"2024-03-11T21:35:08.437783+0000","last_active":"2024-03-11T21:38:04.368507+0000","last_peered":"2024-03-11T21:38:04.368507+0000","last_clean":"2024-03-11T21:38:04.368507+0000","last_became_active":"2024-03-11T21:32:22.791439+0000","last_became_peered":"2024-03-11T21:32:22.791439+0000","last_unstale":"2024-03-11T21:38:04.368507+0000","last_undegraded":"2024-03-11T21:38:04.368507+0000","last_fullsized":"2024-03-11T21:38:04.368507+0000","mapping_epoch":166,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":167,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:35:08.437697+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:43.559452+0000","last_clean_scrub_stamp":"2024-03-11T21:35:08.437697+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":33,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,2],"acting":[4,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.6","version":"67'1","reported_seq":545,"reported_epoch":311,"state":"active+clean","last_fresh":"2024-03-11T21:38:04.368112+0000","last_change":"2024-03-11T21:35:30.581783+0000","last_active":"2024-03-11T21:38:04.368112+0000","last_peered":"2024-03-11T21:38:04.368112+0000","last_clean":"2024-03-11T21:38:04.368112+0000","last_became_active":"2024-03-11T21:25:38.536142+0000","last_became_peered":"2024-03-11T21:25:38.536142+0000","last_unstale":"2024-03-11T21:38:04.368112+0000","last_undegraded":"2024-03-11T21:38:04.368112+0000","last_fullsized":"2024-03-11T21:38:04.368112+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'1","last_scrub_stamp":"2024-03-11T21:35:30.581701+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:37.512018+0000","last_clean_scrub_stamp":"2024-03-11T21:35:30.581701+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,4],"acting":[1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.0","version":"260'38","reported_seq":545,"reported_epoch":312,"state":"active+clean","last_fresh":"2024-03-11T21:38:08.356934+0000","last_change":"2024-03-11T21:36:35.087771+0000","last_active":"2024-03-11T21:38:08.356934+0000","last_peered":"2024-03-11T21:38:08.356934+0000","last_clean":"2024-03-11T21:38:08.356934+0000","last_became_active":"2024-03-11T21:36:35.068071+0000","last_became_peered":"2024-03-11T21:36:35.068071+0000","last_unstale":"2024-03-11T21:38:08.356934+0000","last_undegraded":"2024-03-11T21:38:08.356934+0000","last_fullsized":"2024-03-11T21:38:08.356934+0000","mapping_epoch":277,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":278,"parent":"0.0","parent_split_bits":0,"last_scrub":"93'26","last_scrub_stamp":"2024-03-11T21:34:16.863984+0000","last_deep_scrub":"93'26","last_deep_scrub_stamp":"2024-03-11T21:26:40.680324+0000","last_clean_scrub_stamp":"2024-03-11T21:34:16.863984+0000","objects_scrubbed":8,"log_size":38,"log_dups_size":0,"ondisk_log_size":38,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":22,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":51,"num_read_kb":39,"num_write":32,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":557,"reported_epoch":311,"state":"active+clean","last_fresh":"2024-03-11T21:38:04.368238+0000","last_change":"2024-03-11T21:36:35.057830+0000","last_active":"2024-03-11T21:38:04.368238+0000","last_peered":"2024-03-11T21:38:04.368238+0000","last_clean":"2024-03-11T21:38:04.368238+0000","last_became_active":"2024-03-11T21:36:35.057517+0000","last_became_peered":"2024-03-11T21:36:35.057517+0000","last_unstale":"2024-03-11T21:38:04.368238+0000","last_undegraded":"2024-03-11T21:38:04.368238+0000","last_fullsized":"2024-03-11T21:38:04.368238+0000","mapping_epoch":277,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":278,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:35:12.411136+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:26:51.863621+0000","last_clean_scrub_stamp":"2024-03-11T21:35:12.411136+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,7],"acting":[4,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.c","version":"311'9011","reported_seq":15574,"reported_epoch":311,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-03-11T21:38:07.159809+0000","last_change":"2024-03-11T21:36:36.491614+0000","last_active":"2024-03-11T21:38:07.159809+0000","last_peered":"2024-03-11T21:38:07.159809+0000","last_clean":"2024-03-11T21:35:44.669493+0000","last_became_active":"2024-03-11T21:36:36.482095+0000","last_became_peered":"2024-03-11T21:36:36.482095+0000","last_unstale":"2024-03-11T21:38:07.159809+0000","last_undegraded":"2024-03-11T21:36:36.076624+0000","last_fullsized":"2024-03-11T21:36:36.041449+0000","mapping_epoch":278,"log_start":"273'3600","ondisk_log_start":"273'3600","created":259,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:35:44.669493+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:35:44.669493+0000","last_clean_scrub_stamp":"2024-03-11T21:35:44.669493+0000","objects_scrubbed":0,"log_size":5411,"log_dups_size":0,"ondisk_log_size":5411,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":341508096,"num_objects":5211,"num_object_clones":0,"num_object_copies":15633,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1404,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5211,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9011,"num_write_kb":455104,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3224,"num_bytes_recovered":211288064,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,7],"acting":[4,7],"avail_no_missing":["4","7"],"object_location_counts":[{"shards":"4,7","objects":3807}],"blocked_by":[],"up_primary":3,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.3","version":"0'0","reported_seq":180,"reported_epoch":311,"state":"active+clean","last_fresh":"2024-03-11T21:38:04.397916+0000","last_change":"2024-03-11T21:34:41.143973+0000","last_active":"2024-03-11T21:38:04.397916+0000","last_peered":"2024-03-11T21:38:04.397916+0000","last_clean":"2024-03-11T21:38:04.397916+0000","last_became_active":"2024-03-11T21:34:41.142683+0000","last_became_peered":"2024-03-11T21:34:41.142683+0000","last_unstale":"2024-03-11T21:38:04.397916+0000","last_undegraded":"2024-03-11T21:38:04.397916+0000","last_fullsized":"2024-03-11T21:38:04.397916+0000","mapping_epoch":226,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":227,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:34:27.353700+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:29:42.485740+0000","last_clean_scrub_stamp":"2024-03-11T21:34:27.353700+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,2],"acting":[7,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":564,"reported_epoch":311,"state":"active+clean","last_fresh":"2024-03-11T21:38:04.386565+0000","last_change":"2024-03-11T21:35:32.155983+0000","last_active":"2024-03-11T21:38:04.386565+0000","last_peered":"2024-03-11T21:38:04.386565+0000","last_clean":"2024-03-11T21:38:04.386565+0000","last_became_active":"2024-03-11T21:34:41.145628+0000","last_became_peered":"2024-03-11T21:34:41.145628+0000","last_unstale":"2024-03-11T21:38:04.386565+0000","last_undegraded":"2024-03-11T21:38:04.386565+0000","last_fullsized":"2024-03-11T21:38:04.386565+0000","mapping_epoch":226,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":227,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:35:32.155853+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:37.512018+0000","last_clean_scrub_stamp":"2024-03-11T21:35:32.155853+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,2],"acting":[6,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.1","version":"260'28","reported_seq":583,"reported_epoch":311,"state":"active+clean","last_fresh":"2024-03-11T21:38:04.368142+0000","last_change":"2024-03-11T21:35:33.644654+0000","last_active":"2024-03-11T21:38:04.368142+0000","last_peered":"2024-03-11T21:38:04.368142+0000","last_clean":"2024-03-11T21:38:04.368142+0000","last_became_active":"2024-03-11T21:25:40.546219+0000","last_became_peered":"2024-03-11T21:25:40.546219+0000","last_unstale":"2024-03-11T21:38:04.368142+0000","last_undegraded":"2024-03-11T21:38:04.368142+0000","last_fullsized":"2024-03-11T21:38:04.368142+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'14","last_scrub_stamp":"2024-03-11T21:35:33.644503+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:39.525554+0000","last_clean_scrub_stamp":"2024-03-11T21:35:33.644503+0000","objects_scrubbed":7,"log_size":28,"log_dups_size":0,"ondisk_log_size":28,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":42,"num_read_kb":35,"num_write":28,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6],"acting":[4,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.0","version":"280'11","reported_seq":622,"reported_epoch":312,"state":"active+clean","last_fresh":"2024-03-11T21:38:08.356682+0000","last_change":"2024-03-11T21:36:35.064085+0000","last_active":"2024-03-11T21:38:08.356682+0000","last_peered":"2024-03-11T21:38:08.356682+0000","last_clean":"2024-03-11T21:38:08.356682+0000","last_became_active":"2024-03-11T21:36:35.055250+0000","last_became_peered":"2024-03-11T21:36:35.055250+0000","last_unstale":"2024-03-11T21:38:08.356682+0000","last_undegraded":"2024-03-11T21:38:08.356682+0000","last_fullsized":"2024-03-11T21:38:08.356682+0000","mapping_epoch":277,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":278,"parent":"0.0","parent_split_bits":0,"last_scrub":"98'9","last_scrub_stamp":"2024-03-11T21:35:21.143578+0000","last_deep_scrub":"98'9","last_deep_scrub_stamp":"2024-03-11T21:27:29.745711+0000","last_clean_scrub_stamp":"2024-03-11T21:35:21.143578+0000","objects_scrubbed":1,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.3","version":"311'9074","reported_seq":9198,"reported_epoch":311,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-03-11T21:38:07.242822+0000","last_change":"2024-03-11T21:36:36.544317+0000","last_active":"2024-03-11T21:38:07.242822+0000","last_peered":"2024-03-11T21:38:07.242822+0000","last_clean":"2024-03-11T21:35:44.669493+0000","last_became_active":"2024-03-11T21:36:36.470537+0000","last_became_peered":"2024-03-11T21:36:36.470537+0000","last_unstale":"2024-03-11T21:38:07.242822+0000","last_undegraded":"2024-03-11T21:36:36.071501+0000","last_fullsized":"2024-03-11T21:36:36.040233+0000","mapping_epoch":278,"log_start":"274'4400","ondisk_log_start":"274'4400","created":259,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:35:44.669493+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:35:44.669493+0000","last_clean_scrub_stamp":"2024-03-11T21:35:44.669493+0000","objects_scrubbed":0,"log_size":4674,"log_dups_size":0,"ondisk_log_size":4674,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":343801856,"num_objects":5246,"num_object_clones":0,"num_object_copies":15738,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4641,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5246,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9074,"num_write_kb":458240,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":196608,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,1],"acting":[7,1],"avail_no_missing":["7","1"],"object_location_counts":[{"shards":"1,7","objects":605}],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.c","version":"0'0","reported_seq":466,"reported_epoch":312,"state":"active+clean","last_fresh":"2024-03-11T21:38:08.365290+0000","last_change":"2024-03-11T21:36:35.074112+0000","last_active":"2024-03-11T21:38:08.365290+0000","last_peered":"2024-03-11T21:38:08.365290+0000","last_clean":"2024-03-11T21:38:08.365290+0000","last_became_active":"2024-03-11T21:36:35.073778+0000","last_became_peered":"2024-03-11T21:36:35.073778+0000","last_unstale":"2024-03-11T21:38:08.365290+0000","last_undegraded":"2024-03-11T21:38:08.365290+0000","last_fullsized":"2024-03-11T21:38:08.365290+0000","mapping_epoch":277,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":278,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:32:17.600213+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:43.559452+0000","last_clean_scrub_stamp":"2024-03-11T21:32:17.600213+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.f","version":"0'0","reported_seq":543,"reported_epoch":311,"state":"active+clean","last_fresh":"2024-03-11T21:38:04.412551+0000","last_change":"2024-03-11T21:36:08.439013+0000","last_active":"2024-03-11T21:38:04.412551+0000","last_peered":"2024-03-11T21:38:04.412551+0000","last_clean":"2024-03-11T21:38:04.412551+0000","last_became_active":"2024-03-11T21:25:42.575935+0000","last_became_peered":"2024-03-11T21:25:42.575935+0000","last_unstale":"2024-03-11T21:38:04.412551+0000","last_undegraded":"2024-03-11T21:38:04.412551+0000","last_fullsized":"2024-03-11T21:38:04.412551+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:36:08.438895+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:41.544550+0000","last_clean_scrub_stamp":"2024-03-11T21:36:08.438895+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.8","version":"0'0","reported_seq":188,"reported_epoch":311,"state":"active+clean","last_fresh":"2024-03-11T21:38:04.393494+0000","last_change":"2024-03-11T21:35:34.733595+0000","last_active":"2024-03-11T21:38:04.393494+0000","last_peered":"2024-03-11T21:38:04.393494+0000","last_clean":"2024-03-11T21:38:04.393494+0000","last_became_active":"2024-03-11T21:34:41.144681+0000","last_became_peered":"2024-03-11T21:34:41.144681+0000","last_unstale":"2024-03-11T21:38:04.393494+0000","last_undegraded":"2024-03-11T21:38:04.393494+0000","last_fullsized":"2024-03-11T21:38:04.393494+0000","mapping_epoch":226,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":227,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:35:34.733491+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:37.512018+0000","last_clean_scrub_stamp":"2024-03-11T21:35:34.733491+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,1],"acting":[7,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.e","version":"260'24","reported_seq":576,"reported_epoch":311,"state":"active+clean","last_fresh":"2024-03-11T21:38:04.368679+0000","last_change":"2024-03-11T21:35:34.679906+0000","last_active":"2024-03-11T21:38:04.368679+0000","last_peered":"2024-03-11T21:38:04.368679+0000","last_clean":"2024-03-11T21:38:04.368679+0000","last_became_active":"2024-03-11T21:25:40.552178+0000","last_became_peered":"2024-03-11T21:25:40.552178+0000","last_unstale":"2024-03-11T21:38:04.368679+0000","last_undegraded":"2024-03-11T21:38:04.368679+0000","last_fullsized":"2024-03-11T21:38:04.368679+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"109'16","last_scrub_stamp":"2024-03-11T21:35:34.679817+0000","last_deep_scrub":"109'16","last_deep_scrub_stamp":"2024-03-11T21:28:24.739395+0000","last_clean_scrub_stamp":"2024-03-11T21:35:34.679817+0000","objects_scrubbed":5,"log_size":24,"log_dups_size":0,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":32,"num_read_kb":25,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.2","version":"311'9201","reported_seq":9314,"reported_epoch":311,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-03-11T21:38:07.729013+0000","last_change":"2024-03-11T21:36:36.513966+0000","last_active":"2024-03-11T21:38:07.729013+0000","last_peered":"2024-03-11T21:38:07.729013+0000","last_clean":"2024-03-11T21:35:44.669493+0000","last_became_active":"2024-03-11T21:36:36.506474+0000","last_became_peered":"2024-03-11T21:36:36.506474+0000","last_unstale":"2024-03-11T21:38:07.729013+0000","last_undegraded":"2024-03-11T21:36:36.068182+0000","last_fullsized":"2024-03-11T21:36:36.037781+0000","mapping_epoch":278,"log_start":"274'4200","ondisk_log_start":"274'4200","created":259,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:35:44.669493+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:35:44.669493+0000","last_clean_scrub_stamp":"2024-03-11T21:35:44.669493+0000","objects_scrubbed":0,"log_size":5001,"log_dups_size":0,"ondisk_log_size":5001,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":349241344,"num_objects":5329,"num_object_clones":0,"num_object_copies":15987,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4773,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5329,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9201,"num_write_kb":464960,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[1,2],"avail_no_missing":["1","2"],"object_location_counts":[{"shards":"1,2","objects":556}],"blocked_by":[],"up_primary":3,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.d","version":"0'0","reported_seq":531,"reported_epoch":311,"state":"active+clean","last_fresh":"2024-03-11T21:38:04.397801+0000","last_change":"2024-03-11T21:35:37.185210+0000","last_active":"2024-03-11T21:38:04.397801+0000","last_peered":"2024-03-11T21:38:04.397801+0000","last_clean":"2024-03-11T21:38:04.397801+0000","last_became_active":"2024-03-11T21:25:44.590333+0000","last_became_peered":"2024-03-11T21:25:44.590333+0000","last_unstale":"2024-03-11T21:38:04.397801+0000","last_undegraded":"2024-03-11T21:38:04.397801+0000","last_fullsized":"2024-03-11T21:38:04.397801+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:35:37.182850+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:35:37.182850+0000","last_clean_scrub_stamp":"2024-03-11T21:35:37.182850+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":40,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.e","version":"74'8","reported_seq":694,"reported_epoch":311,"state":"active+clean","last_fresh":"2024-03-11T21:38:04.368985+0000","last_change":"2024-03-11T21:36:07.734369+0000","last_active":"2024-03-11T21:38:04.368985+0000","last_peered":"2024-03-11T21:38:04.368985+0000","last_clean":"2024-03-11T21:38:04.368985+0000","last_became_active":"2024-03-11T21:25:42.576742+0000","last_became_peered":"2024-03-11T21:25:42.576742+0000","last_unstale":"2024-03-11T21:38:04.368985+0000","last_undegraded":"2024-03-11T21:38:04.368985+0000","last_fullsized":"2024-03-11T21:38:04.368985+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'8","last_scrub_stamp":"2024-03-11T21:36:07.734115+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:41.544550+0000","last_clean_scrub_stamp":"2024-03-11T21:36:07.734115+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,0],"acting":[4,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.9","version":"0'0","reported_seq":557,"reported_epoch":311,"state":"active+clean","last_fresh":"2024-03-11T21:38:04.362957+0000","last_change":"2024-03-11T21:36:35.066093+0000","last_active":"2024-03-11T21:38:04.362957+0000","last_peered":"2024-03-11T21:38:04.362957+0000","last_clean":"2024-03-11T21:38:04.362957+0000","last_became_active":"2024-03-11T21:36:35.065733+0000","last_became_peered":"2024-03-11T21:36:35.065733+0000","last_unstale":"2024-03-11T21:38:04.362957+0000","last_undegraded":"2024-03-11T21:38:04.362957+0000","last_fullsized":"2024-03-11T21:38:04.362957+0000","mapping_epoch":277,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":278,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:32:02.506579+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:26:50.991712+0000","last_clean_scrub_stamp":"2024-03-11T21:32:02.506579+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,3],"acting":[1,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.f","version":"260'27","reported_seq":559,"reported_epoch":311,"state":"active+clean","last_fresh":"2024-03-11T21:38:04.368644+0000","last_change":"2024-03-11T21:36:35.084069+0000","last_active":"2024-03-11T21:38:04.368644+0000","last_peered":"2024-03-11T21:38:04.368644+0000","last_clean":"2024-03-11T21:38:04.368644+0000","last_became_active":"2024-03-11T21:36:35.058272+0000","last_became_peered":"2024-03-11T21:36:35.058272+0000","last_unstale":"2024-03-11T21:38:04.368644+0000","last_undegraded":"2024-03-11T21:38:04.368644+0000","last_fullsized":"2024-03-11T21:38:04.368644+0000","mapping_epoch":277,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":278,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'15","last_scrub_stamp":"2024-03-11T21:34:21.517894+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:39.525554+0000","last_clean_scrub_stamp":"2024-03-11T21:34:21.517894+0000","objects_scrubbed":7,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":44,"num_read_kb":35,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,4],"acting":[2,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"9.1","version":"311'9215","reported_seq":16786,"reported_epoch":311,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-03-11T21:38:07.296489+0000","last_change":"2024-03-11T21:36:36.545694+0000","last_active":"2024-03-11T21:38:07.296489+0000","last_peered":"2024-03-11T21:38:07.296489+0000","last_clean":"2024-03-11T21:35:44.669493+0000","last_became_active":"2024-03-11T21:36:36.509431+0000","last_became_peered":"2024-03-11T21:36:36.509431+0000","last_unstale":"2024-03-11T21:38:07.296489+0000","last_undegraded":"2024-03-11T21:36:36.165469+0000","last_fullsized":"2024-03-11T21:36:36.044027+0000","mapping_epoch":278,"log_start":"270'2600","ondisk_log_start":"270'2600","created":259,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:35:44.669493+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:35:44.669493+0000","last_clean_scrub_stamp":"2024-03-11T21:35:44.669493+0000","objects_scrubbed":0,"log_size":6615,"log_dups_size":0,"ondisk_log_size":6615,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":351862784,"num_objects":5369,"num_object_clones":0,"num_object_copies":16107,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1097,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5369,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9215,"num_write_kb":466688,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3728,"num_bytes_recovered":244318208,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,7],"acting":[6,7],"avail_no_missing":["6","7"],"object_location_counts":[{"shards":"6,7","objects":4272}],"blocked_by":[],"up_primary":3,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.e","version":"0'0","reported_seq":532,"reported_epoch":311,"state":"active+clean","last_fresh":"2024-03-11T21:38:04.369072+0000","last_change":"2024-03-11T21:36:10.769557+0000","last_active":"2024-03-11T21:38:04.369072+0000","last_peered":"2024-03-11T21:38:04.369072+0000","last_clean":"2024-03-11T21:38:04.369072+0000","last_became_active":"2024-03-11T21:25:44.590198+0000","last_became_peered":"2024-03-11T21:25:44.590198+0000","last_unstale":"2024-03-11T21:38:04.369072+0000","last_undegraded":"2024-03-11T21:38:04.369072+0000","last_fullsized":"2024-03-11T21:38:04.369072+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:36:10.769460+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:43.559452+0000","last_clean_scrub_stamp":"2024-03-11T21:36:10.769460+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":75,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.d","version":"109'9","reported_seq":694,"reported_epoch":311,"state":"active+clean","last_fresh":"2024-03-11T21:38:04.366522+0000","last_change":"2024-03-11T21:35:07.562523+0000","last_active":"2024-03-11T21:38:04.366522+0000","last_peered":"2024-03-11T21:38:04.366522+0000","last_clean":"2024-03-11T21:38:04.366522+0000","last_became_active":"2024-03-11T21:25:42.575583+0000","last_became_peered":"2024-03-11T21:25:42.575583+0000","last_unstale":"2024-03-11T21:38:04.366522+0000","last_undegraded":"2024-03-11T21:38:04.366522+0000","last_fullsized":"2024-03-11T21:38:04.366522+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"109'9","last_scrub_stamp":"2024-03-11T21:35:07.562385+0000","last_deep_scrub":"109'9","last_deep_scrub_stamp":"2024-03-11T21:28:27.082975+0000","last_clean_scrub_stamp":"2024-03-11T21:35:07.562385+0000","objects_scrubbed":1,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,5],"acting":[2,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.a","version":"0'0","reported_seq":565,"reported_epoch":311,"state":"active+clean","last_fresh":"2024-03-11T21:38:04.368060+0000","last_change":"2024-03-11T21:36:35.067262+0000","last_active":"2024-03-11T21:38:04.368060+0000","last_peered":"2024-03-11T21:38:04.368060+0000","last_clean":"2024-03-11T21:38:04.368060+0000","last_became_active":"2024-03-11T21:36:35.066832+0000","last_became_peered":"2024-03-11T21:36:35.066832+0000","last_unstale":"2024-03-11T21:38:04.368060+0000","last_undegraded":"2024-03-11T21:38:04.368060+0000","last_fullsized":"2024-03-11T21:38:04.368060+0000","mapping_epoch":277,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":278,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:34:22.526916+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:37.512018+0000","last_clean_scrub_stamp":"2024-03-11T21:34:22.526916+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,7],"acting":[1,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.c","version":"260'20","reported_seq":584,"reported_epoch":311,"state":"active+clean","last_fresh":"2024-03-11T21:38:04.369721+0000","last_change":"2024-03-11T21:36:35.095574+0000","last_active":"2024-03-11T21:38:04.369721+0000","last_peered":"2024-03-11T21:38:04.369721+0000","last_clean":"2024-03-11T21:38:04.369721+0000","last_became_active":"2024-03-11T21:36:35.067269+0000","last_became_peered":"2024-03-11T21:36:35.067269+0000","last_unstale":"2024-03-11T21:38:04.369721+0000","last_undegraded":"2024-03-11T21:38:04.369721+0000","last_fullsized":"2024-03-11T21:38:04.369721+0000","mapping_epoch":277,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":278,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'10","last_scrub_stamp":"2024-03-11T21:32:54.846584+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:39.525554+0000","last_clean_scrub_stamp":"2024-03-11T21:32:54.846584+0000","objects_scrubbed":5,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":25,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.0","version":"311'9302","reported_seq":9417,"reported_epoch":311,"state":"active+clean","last_fresh":"2024-03-11T21:38:07.662263+0000","last_change":"2024-03-11T21:35:45.703065+0000","last_active":"2024-03-11T21:38:07.662263+0000","last_peered":"2024-03-11T21:38:07.662263+0000","last_clean":"2024-03-11T21:38:07.662263+0000","last_became_active":"2024-03-11T21:35:45.702739+0000","last_became_peered":"2024-03-11T21:35:45.702739+0000","last_unstale":"2024-03-11T21:38:07.662263+0000","last_undegraded":"2024-03-11T21:38:07.662263+0000","last_fullsized":"2024-03-11T21:38:07.662263+0000","mapping_epoch":259,"log_start":"274'4300","ondisk_log_start":"274'4300","created":259,"last_epoch_clean":260,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:35:44.669493+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:35:44.669493+0000","last_clean_scrub_stamp":"2024-03-11T21:35:44.669493+0000","objects_scrubbed":0,"log_size":5002,"log_dups_size":0,"ondisk_log_size":5002,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":346161152,"num_objects":5282,"num_object_clones":0,"num_object_copies":15846,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5282,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9302,"num_write_kb":466688,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.f","version":"0'0","reported_seq":540,"reported_epoch":311,"state":"active+clean","last_fresh":"2024-03-11T21:38:04.368489+0000","last_change":"2024-03-11T21:36:35.058630+0000","last_active":"2024-03-11T21:38:04.368489+0000","last_peered":"2024-03-11T21:38:04.368489+0000","last_clean":"2024-03-11T21:38:04.368489+0000","last_became_active":"2024-03-11T21:36:35.057800+0000","last_became_peered":"2024-03-11T21:36:35.057800+0000","last_unstale":"2024-03-11T21:38:04.368489+0000","last_undegraded":"2024-03-11T21:38:04.368489+0000","last_fullsized":"2024-03-11T21:38:04.368489+0000","mapping_epoch":277,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":278,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:35:10.608981+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:27:03.010969+0000","last_clean_scrub_stamp":"2024-03-11T21:35:10.608981+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":94,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,4],"acting":[2,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":536,"reported_epoch":311,"state":"active+clean","last_fresh":"2024-03-11T21:38:04.369595+0000","last_change":"2024-03-11T21:36:10.608789+0000","last_active":"2024-03-11T21:38:04.369595+0000","last_peered":"2024-03-11T21:38:04.369595+0000","last_clean":"2024-03-11T21:38:04.369595+0000","last_became_active":"2024-03-11T21:25:42.579409+0000","last_became_peered":"2024-03-11T21:25:42.579409+0000","last_unstale":"2024-03-11T21:38:04.369595+0000","last_undegraded":"2024-03-11T21:38:04.369595+0000","last_fullsized":"2024-03-11T21:38:04.369595+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:36:10.608552+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:30:22.750053+0000","last_clean_scrub_stamp":"2024-03-11T21:36:10.608552+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.b","version":"0'0","reported_seq":188,"reported_epoch":311,"state":"active+clean","last_fresh":"2024-03-11T21:38:04.398014+0000","last_change":"2024-03-11T21:36:16.719432+0000","last_active":"2024-03-11T21:38:04.398014+0000","last_peered":"2024-03-11T21:38:04.398014+0000","last_clean":"2024-03-11T21:38:04.398014+0000","last_became_active":"2024-03-11T21:34:41.134987+0000","last_became_peered":"2024-03-11T21:34:41.134987+0000","last_unstale":"2024-03-11T21:38:04.398014+0000","last_undegraded":"2024-03-11T21:38:04.398014+0000","last_fullsized":"2024-03-11T21:38:04.398014+0000","mapping_epoch":226,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":227,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:36:16.719309+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:36:16.719309+0000","last_clean_scrub_stamp":"2024-03-11T21:36:16.719309+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,5],"acting":[7,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.d","version":"260'39","reported_seq":597,"reported_epoch":311,"state":"active+clean","last_fresh":"2024-03-11T21:38:04.368682+0000","last_change":"2024-03-11T21:35:25.543769+0000","last_active":"2024-03-11T21:38:04.368682+0000","last_peered":"2024-03-11T21:38:04.368682+0000","last_clean":"2024-03-11T21:38:04.368682+0000","last_became_active":"2024-03-11T21:25:40.553454+0000","last_became_peered":"2024-03-11T21:25:40.553454+0000","last_unstale":"2024-03-11T21:38:04.368682+0000","last_undegraded":"2024-03-11T21:38:04.368682+0000","last_fullsized":"2024-03-11T21:38:04.368682+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"165'25","last_scrub_stamp":"2024-03-11T21:35:25.543628+0000","last_deep_scrub":"165'25","last_deep_scrub_stamp":"2024-03-11T21:32:21.678328+0000","last_clean_scrub_stamp":"2024-03-11T21:35:25.543628+0000","objects_scrubbed":8,"log_size":39,"log_dups_size":0,"ondisk_log_size":39,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":50,"num_read_kb":40,"num_write":32,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,1],"acting":[4,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.7","version":"311'9281","reported_seq":14155,"reported_epoch":311,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-03-11T21:38:07.453881+0000","last_change":"2024-03-11T21:36:36.539792+0000","last_active":"2024-03-11T21:38:07.453881+0000","last_peered":"2024-03-11T21:38:07.453881+0000","last_clean":"2024-03-11T21:35:44.669493+0000","last_became_active":"2024-03-11T21:36:36.508974+0000","last_became_peered":"2024-03-11T21:36:36.508974+0000","last_unstale":"2024-03-11T21:38:07.453881+0000","last_undegraded":"2024-03-11T21:36:36.183555+0000","last_fullsized":"2024-03-11T21:36:36.038772+0000","mapping_epoch":278,"log_start":"274'4100","ondisk_log_start":"274'4100","created":259,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:35:44.669493+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:35:44.669493+0000","last_clean_scrub_stamp":"2024-03-11T21:35:44.669493+0000","objects_scrubbed":0,"log_size":5181,"log_dups_size":0,"ondisk_log_size":5181,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":343867392,"num_objects":5247,"num_object_clones":0,"num_object_copies":15741,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2373,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5247,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9281,"num_write_kb":464896,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2379,"num_bytes_recovered":155910144,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,7],"avail_no_missing":["5","7"],"object_location_counts":[{"shards":"5,7","objects":2874}],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":188,"reported_epoch":311,"state":"active+clean","last_fresh":"2024-03-11T21:38:04.393253+0000","last_change":"2024-03-11T21:36:06.735035+0000","last_active":"2024-03-11T21:38:04.393253+0000","last_peered":"2024-03-11T21:38:04.393253+0000","last_clean":"2024-03-11T21:38:04.393253+0000","last_became_active":"2024-03-11T21:34:41.145012+0000","last_became_peered":"2024-03-11T21:34:41.145012+0000","last_unstale":"2024-03-11T21:38:04.393253+0000","last_undegraded":"2024-03-11T21:38:04.393253+0000","last_fullsized":"2024-03-11T21:38:04.393253+0000","mapping_epoch":226,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":227,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:36:06.734921+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:28:37.561826+0000","last_clean_scrub_stamp":"2024-03-11T21:36:06.734921+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":23,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,0],"acting":[7,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":535,"reported_epoch":311,"state":"active+clean","last_fresh":"2024-03-11T21:38:04.365666+0000","last_change":"2024-03-11T21:35:14.636561+0000","last_active":"2024-03-11T21:38:04.365666+0000","last_peered":"2024-03-11T21:38:04.365666+0000","last_clean":"2024-03-11T21:38:04.365666+0000","last_became_active":"2024-03-11T21:25:42.574515+0000","last_became_peered":"2024-03-11T21:25:42.574515+0000","last_unstale":"2024-03-11T21:38:04.365666+0000","last_undegraded":"2024-03-11T21:38:04.365666+0000","last_fullsized":"2024-03-11T21:38:04.365666+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:35:14.636450+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:41.544550+0000","last_clean_scrub_stamp":"2024-03-11T21:35:14.636450+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,5],"acting":[2,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.c","version":"0'0","reported_seq":543,"reported_epoch":311,"state":"active+clean","last_fresh":"2024-03-11T21:38:04.366153+0000","last_change":"2024-03-11T21:36:00.849272+0000","last_active":"2024-03-11T21:38:04.366153+0000","last_peered":"2024-03-11T21:38:04.366153+0000","last_clean":"2024-03-11T21:38:04.366153+0000","last_became_active":"2024-03-11T21:25:38.533240+0000","last_became_peered":"2024-03-11T21:25:38.533240+0000","last_unstale":"2024-03-11T21:38:04.366153+0000","last_undegraded":"2024-03-11T21:38:04.366153+0000","last_fullsized":"2024-03-11T21:38:04.366153+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:36:00.849153+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:29:30.279910+0000","last_clean_scrub_stamp":"2024-03-11T21:36:00.849153+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,0],"acting":[2,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.a","version":"260'44","reported_seq":664,"reported_epoch":311,"state":"active+clean","last_fresh":"2024-03-11T21:38:04.363452+0000","last_change":"2024-03-11T21:35:34.138400+0000","last_active":"2024-03-11T21:38:04.363452+0000","last_peered":"2024-03-11T21:38:04.363452+0000","last_clean":"2024-03-11T21:38:04.363452+0000","last_became_active":"2024-03-11T21:34:41.148863+0000","last_became_peered":"2024-03-11T21:34:41.148863+0000","last_unstale":"2024-03-11T21:38:04.363452+0000","last_undegraded":"2024-03-11T21:38:04.363452+0000","last_fullsized":"2024-03-11T21:38:04.363452+0000","mapping_epoch":226,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":227,"parent":"0.0","parent_split_bits":0,"last_scrub":"109'28","last_scrub_stamp":"2024-03-11T21:35:34.138261+0000","last_deep_scrub":"109'28","last_deep_scrub_stamp":"2024-03-11T21:28:24.016389+0000","last_clean_scrub_stamp":"2024-03-11T21:35:34.138261+0000","objects_scrubbed":9,"log_size":44,"log_dups_size":0,"ondisk_log_size":44,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":56,"num_read_kb":45,"num_write":36,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,7],"acting":[6,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.6","version":"311'9349","reported_seq":9464,"reported_epoch":311,"state":"active+clean","last_fresh":"2024-03-11T21:38:07.432126+0000","last_change":"2024-03-11T21:35:45.703533+0000","last_active":"2024-03-11T21:38:07.432126+0000","last_peered":"2024-03-11T21:38:07.432126+0000","last_clean":"2024-03-11T21:38:07.432126+0000","last_became_active":"2024-03-11T21:35:45.703184+0000","last_became_peered":"2024-03-11T21:35:45.703184+0000","last_unstale":"2024-03-11T21:38:07.432126+0000","last_undegraded":"2024-03-11T21:38:07.432126+0000","last_fullsized":"2024-03-11T21:38:07.432126+0000","mapping_epoch":259,"log_start":"274'4100","ondisk_log_start":"274'4100","created":259,"last_epoch_clean":260,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:35:44.669493+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:35:44.669493+0000","last_clean_scrub_stamp":"2024-03-11T21:35:44.669493+0000","objects_scrubbed":0,"log_size":5249,"log_dups_size":0,"ondisk_log_size":5249,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":340590592,"num_objects":5197,"num_object_clones":0,"num_object_copies":15591,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5197,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9349,"num_write_kb":465472,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,1],"acting":[5,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":546,"reported_epoch":312,"state":"active+clean","last_fresh":"2024-03-11T21:38:08.350291+0000","last_change":"2024-03-11T21:35:37.687281+0000","last_active":"2024-03-11T21:38:08.350291+0000","last_peered":"2024-03-11T21:38:08.350291+0000","last_clean":"2024-03-11T21:38:08.350291+0000","last_became_active":"2024-03-11T21:34:41.154507+0000","last_became_peered":"2024-03-11T21:34:41.154507+0000","last_unstale":"2024-03-11T21:38:08.350291+0000","last_undegraded":"2024-03-11T21:38:08.350291+0000","last_fullsized":"2024-03-11T21:38:08.350291+0000","mapping_epoch":226,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":227,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:35:37.687187+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:27:11.361913+0000","last_clean_scrub_stamp":"2024-03-11T21:35:37.687187+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":39,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":536,"reported_epoch":311,"state":"active+clean","last_fresh":"2024-03-11T21:38:04.365883+0000","last_change":"2024-03-11T21:36:35.059662+0000","last_active":"2024-03-11T21:38:04.365883+0000","last_peered":"2024-03-11T21:38:04.365883+0000","last_clean":"2024-03-11T21:38:04.365883+0000","last_became_active":"2024-03-11T21:36:35.059192+0000","last_became_peered":"2024-03-11T21:36:35.059192+0000","last_unstale":"2024-03-11T21:38:04.365883+0000","last_undegraded":"2024-03-11T21:38:04.365883+0000","last_fullsized":"2024-03-11T21:38:04.365883+0000","mapping_epoch":277,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":278,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:34:17.412687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:41.544550+0000","last_clean_scrub_stamp":"2024-03-11T21:34:17.412687+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,3],"acting":[2,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.d","version":"0'0","reported_seq":545,"reported_epoch":311,"state":"active+clean","last_fresh":"2024-03-11T21:38:04.367599+0000","last_change":"2024-03-11T21:36:35.058635+0000","last_active":"2024-03-11T21:38:04.367599+0000","last_peered":"2024-03-11T21:38:04.367599+0000","last_clean":"2024-03-11T21:38:04.367599+0000","last_became_active":"2024-03-11T21:36:35.058296+0000","last_became_peered":"2024-03-11T21:36:35.058296+0000","last_unstale":"2024-03-11T21:38:04.367599+0000","last_undegraded":"2024-03-11T21:38:04.367599+0000","last_fullsized":"2024-03-11T21:38:04.367599+0000","mapping_epoch":277,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":278,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:31:55.464145+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:37.512018+0000","last_clean_scrub_stamp":"2024-03-11T21:31:55.464145+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,3],"acting":[1,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.b","version":"260'15","reported_seq":567,"reported_epoch":312,"state":"active+clean","last_fresh":"2024-03-11T21:38:08.349604+0000","last_change":"2024-03-11T21:35:27.760318+0000","last_active":"2024-03-11T21:38:08.349604+0000","last_peered":"2024-03-11T21:38:08.349604+0000","last_clean":"2024-03-11T21:38:08.349604+0000","last_became_active":"2024-03-11T21:25:40.549612+0000","last_became_peered":"2024-03-11T21:25:40.549612+0000","last_unstale":"2024-03-11T21:38:08.349604+0000","last_undegraded":"2024-03-11T21:38:08.349604+0000","last_fullsized":"2024-03-11T21:38:08.349604+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'9","last_scrub_stamp":"2024-03-11T21:35:27.760208+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:39.525554+0000","last_clean_scrub_stamp":"2024-03-11T21:35:27.760208+0000","objects_scrubbed":4,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.5","version":"312'9405","reported_seq":9522,"reported_epoch":312,"state":"active+clean","last_fresh":"2024-03-11T21:38:08.501053+0000","last_change":"2024-03-11T21:35:45.697209+0000","last_active":"2024-03-11T21:38:08.501053+0000","last_peered":"2024-03-11T21:38:08.501053+0000","last_clean":"2024-03-11T21:38:08.501053+0000","last_became_active":"2024-03-11T21:35:45.696788+0000","last_became_peered":"2024-03-11T21:35:45.696788+0000","last_unstale":"2024-03-11T21:38:08.501053+0000","last_undegraded":"2024-03-11T21:38:08.501053+0000","last_fullsized":"2024-03-11T21:38:08.501053+0000","mapping_epoch":259,"log_start":"274'3800","ondisk_log_start":"274'3800","created":259,"last_epoch_clean":260,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:35:44.669493+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:35:44.669493+0000","last_clean_scrub_stamp":"2024-03-11T21:35:44.669493+0000","objects_scrubbed":0,"log_size":5605,"log_dups_size":0,"ondisk_log_size":5605,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":353697792,"num_objects":5397,"num_object_clones":0,"num_object_copies":16191,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5397,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9405,"num_write_kb":473664,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":446,"reported_epoch":311,"state":"active+clean","last_fresh":"2024-03-11T21:38:04.368067+0000","last_change":"2024-03-11T21:36:35.057118+0000","last_active":"2024-03-11T21:38:04.368067+0000","last_peered":"2024-03-11T21:38:04.368067+0000","last_clean":"2024-03-11T21:38:04.368067+0000","last_became_active":"2024-03-11T21:36:35.056518+0000","last_became_peered":"2024-03-11T21:36:35.056518+0000","last_unstale":"2024-03-11T21:38:04.368067+0000","last_undegraded":"2024-03-11T21:38:04.368067+0000","last_fullsized":"2024-03-11T21:38:04.368067+0000","mapping_epoch":277,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":278,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:35:24.514584+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:43.559452+0000","last_clean_scrub_stamp":"2024-03-11T21:35:24.514584+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,3],"acting":[4,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.e","version":"0'0","reported_seq":564,"reported_epoch":311,"state":"active+clean","last_fresh":"2024-03-11T21:38:04.368473+0000","last_change":"2024-03-11T21:36:35.057187+0000","last_active":"2024-03-11T21:38:04.368473+0000","last_peered":"2024-03-11T21:38:04.368473+0000","last_clean":"2024-03-11T21:38:04.368473+0000","last_became_active":"2024-03-11T21:36:35.056777+0000","last_became_peered":"2024-03-11T21:36:35.056777+0000","last_unstale":"2024-03-11T21:38:04.368473+0000","last_undegraded":"2024-03-11T21:38:04.368473+0000","last_fullsized":"2024-03-11T21:38:04.368473+0000","mapping_epoch":277,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":278,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:34:19.476172+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:37.512018+0000","last_clean_scrub_stamp":"2024-03-11T21:34:19.476172+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":17,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,7],"acting":[2,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.8","version":"260'34","reported_seq":633,"reported_epoch":311,"state":"active+clean","last_fresh":"2024-03-11T21:38:04.392046+0000","last_change":"2024-03-11T21:35:20.304345+0000","last_active":"2024-03-11T21:38:04.392046+0000","last_peered":"2024-03-11T21:38:04.392046+0000","last_clean":"2024-03-11T21:38:04.392046+0000","last_became_active":"2024-03-11T21:34:41.146631+0000","last_became_peered":"2024-03-11T21:34:41.146631+0000","last_unstale":"2024-03-11T21:38:04.392046+0000","last_undegraded":"2024-03-11T21:38:04.392046+0000","last_fullsized":"2024-03-11T21:38:04.392046+0000","mapping_epoch":226,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":227,"parent":"0.0","parent_split_bits":0,"last_scrub":"109'22","last_scrub_stamp":"2024-03-11T21:35:20.304114+0000","last_deep_scrub":"109'22","last_deep_scrub_stamp":"2024-03-11T21:32:07.573197+0000","last_clean_scrub_stamp":"2024-03-11T21:35:20.304114+0000","objects_scrubbed":7,"log_size":34,"log_dups_size":0,"ondisk_log_size":34,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":44,"num_read_kb":35,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.9","version":"229'11","reported_seq":233,"reported_epoch":311,"state":"active+clean","last_fresh":"2024-03-11T21:38:04.393682+0000","last_change":"2024-03-11T21:36:18.726876+0000","last_active":"2024-03-11T21:38:04.393682+0000","last_peered":"2024-03-11T21:38:04.393682+0000","last_clean":"2024-03-11T21:38:04.393682+0000","last_became_active":"2024-03-11T21:34:41.156892+0000","last_became_peered":"2024-03-11T21:34:41.156892+0000","last_unstale":"2024-03-11T21:38:04.393682+0000","last_undegraded":"2024-03-11T21:38:04.393682+0000","last_fullsized":"2024-03-11T21:38:04.393682+0000","mapping_epoch":226,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":227,"parent":"0.0","parent_split_bits":0,"last_scrub":"229'11","last_scrub_stamp":"2024-03-11T21:36:18.726763+0000","last_deep_scrub":"94'9","last_deep_scrub_stamp":"2024-03-11T21:26:53.422552+0000","last_clean_scrub_stamp":"2024-03-11T21:36:18.726763+0000","objects_scrubbed":1,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":9,"num_pg":16,"stat_sum":{"num_bytes":5541658648,"num_objects":84560,"num_object_clones":0,"num_object_copies":253680,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":28688,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":84560,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":148204,"num_write_kb":7448385,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9335,"num_bytes_recovered":611778560,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":15424434176,"data_stored":15424421960,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":84204,"ondisk_log_size":84204,"up":48,"acting":40,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":1531,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":17,"num_read_kb":17,"num_write":10,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":1308,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":4593,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":6,"ondisk_log_size":6,"up":96,"acting":96,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":198,"num_read_kb":294,"num_write":179,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":2951840,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":2375680,"data_stored":2361472,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":128,"ondisk_log_size":128,"up":3,"acting":3,"num_store_stats":4},{"poolid":3,"num_pg":3,"stat_sum":{"num_bytes":408,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":742,"num_read_kb":737,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1224,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":8,"ondisk_log_size":8,"up":9,"acting":9,"num_store_stats":7},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":3702,"num_objects":178,"num_object_clones":0,"num_object_copies":534,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":178,"num_whiteouts":0,"num_read":1120,"num_read_kb":877,"num_write":703,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":186,"num_bytes_recovered":2200,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":417792,"data_stored":11106,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":760,"ondisk_log_size":760,"up":96,"acting":96,"num_store_stats":8},{"poolid":5,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":78,"ondisk_log_size":78,"up":96,"acting":96,"num_store_stats":8},{"poolid":6,"num_pg":32,"stat_sum":{"num_bytes":383,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":370,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1149,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":96,"acting":96,"num_store_stats":8}],"osd_stats":[{"osd":7,"up_from":60,"seq":257698037913,"num_pgs":65,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":3538940,"kb_used_data":3092660,"kb_used_omap":1,"kb_used_meta":446270,"kb_avail":90206212,"statfs":{"total":95995035648,"available":92371161088,"internally_reserved":0,"allocated":3166883840,"data_stored":3166140142,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":456980938},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":50,"seq":214748364959,"num_pgs":46,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1395960,"kb_used_data":1021872,"kb_used_omap":1,"kb_used_meta":374078,"kb_avail":92349192,"statfs":{"total":95995035648,"available":94565572608,"internally_reserved":0,"allocated":1046396928,"data_stored":1045658031,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":383056326},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834575033,"num_pgs":60,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":3587540,"kb_used_data":3042700,"kb_used_omap":1,"kb_used_meta":544830,"kb_avail":90157612,"statfs":{"total":95995035648,"available":92321394688,"internally_reserved":0,"allocated":3115724800,"data_stored":3114956709,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":557906373},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738558,"num_pgs":54,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1858500,"kb_used_data":1367804,"kb_used_omap":1,"kb_used_meta":490686,"kb_avail":91886652,"statfs":{"total":95995035648,"available":94091931648,"internally_reserved":0,"allocated":1400631296,"data_stored":1399880145,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":502462917},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411507,"num_pgs":43,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1444080,"kb_used_data":1020008,"kb_used_omap":1,"kb_used_meta":424062,"kb_avail":92301072,"statfs":{"total":95995035648,"available":94516297728,"internally_reserved":0,"allocated":1044488192,"data_stored":1043756711,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":434239941},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":277,"seq":1189705941012,"num_pgs":62,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1692564,"kb_used_data":1506188,"kb_used_omap":4,"kb_used_meta":186363,"kb_avail":92052588,"statfs":{"total":95995035648,"available":94261850112,"internally_reserved":0,"allocated":1542336512,"data_stored":1541568841,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4442,"internal_metadata":190836390},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":4,"up_from":32,"seq":137438953640,"num_pgs":56,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1388932,"kb_used_data":1019004,"kb_used_omap":1,"kb_used_meta":369918,"kb_avail":92356220,"statfs":{"total":95995035648,"available":94572769280,"internally_reserved":0,"allocated":1043460096,"data_stored":1042707983,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":378796485},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":41,"seq":176093659299,"num_pgs":58,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":3577012,"kb_used_data":3068268,"kb_used_omap":1,"kb_used_meta":508734,"kb_avail":90168140,"statfs":{"total":95995035648,"available":92332175360,"internally_reserved":0,"allocated":3141906432,"data_stored":3141170494,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":520944069},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1049,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":46,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":436,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":957,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":574,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":436,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1003,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":92,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":1430,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":77824,"data_stored":2008,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":1018,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":1898,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":1474,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":990,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":1034,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":1254,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":370,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":383,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":370,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":1388249088,"data_stored":1388249088,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":3109294080,"data_stored":3109290008,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":1037959168,"data_stored":1037959168,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":1539833856,"data_stored":1539833856,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":1037762560,"data_stored":1037762560,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":3129020416,"data_stored":3129016344,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":1043005440,"data_stored":1043005440,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":3139309568,"data_stored":3139305496,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-11T21:38:11.634 DEBUG:teuthology.orchestra.run.smithi092:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid 00db0016-dfed-11ee-95c8-87774f69a715 -- ceph pg dump --format=json 2024-03-11T21:38:11.657 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:38:11.689 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:38:11.690 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:38:11.690 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:38:11.791 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:38:11.823 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:38:11.824 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:38:11.824 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:38:11.827 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:38:11 smithi097 ceph-mon[33317]: from='client.25429 v1:172.21.15.92:0/3811565470' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-11T21:38:11.926 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:38:11.962 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:38:11.962 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:38:11.963 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:38:12.064 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:38:12.096 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:38:12.096 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:38:12.097 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:38:12.198 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:38:12.245 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:38:12.246 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:38:12.247 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:38:12.347 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:38:12.746 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:38:12 smithi092 ceph-mon[29572]: pgmap v583: 148 pgs: 5 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+degraded+remapped, 140 active+clean; 5.2 GiB data, 18 GiB used, 698 GiB / 715 GiB avail; 647 B/s rd, 0 B/s wr, 625 op/s; 28688/254274 objects degraded (11.282%); 5.5 MiB/s, 87 objects/s recovering 2024-03-11T21:38:12.746 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:38:12 smithi092 ceph-mon[37563]: pgmap v583: 148 pgs: 5 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+degraded+remapped, 140 active+clean; 5.2 GiB data, 18 GiB used, 698 GiB / 715 GiB avail; 647 B/s rd, 0 B/s wr, 625 op/s; 28688/254274 objects degraded (11.282%); 5.5 MiB/s, 87 objects/s recovering 2024-03-11T21:38:12.811 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:38:12.812 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:38:12.812 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:38:12.827 INFO:journalctl@ceph.iscsi.iscsi.a.smithi097.stdout:Mar 11 21:38:12 smithi097 ceph-00db0016-dfed-11ee-95c8-87774f69a715-iscsi-iscsi-a[60943]: debug there is no tcmu-runner data available 2024-03-11T21:38:12.827 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:38:12 smithi097 ceph-mon[33317]: pgmap v583: 148 pgs: 5 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+degraded+remapped, 140 active+clean; 5.2 GiB data, 18 GiB used, 698 GiB / 715 GiB avail; 647 B/s rd, 0 B/s wr, 625 op/s; 28688/254274 objects degraded (11.282%); 5.5 MiB/s, 87 objects/s recovering 2024-03-11T21:38:12.913 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:38:12.943 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:38:12.944 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:38:12.945 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:38:13.045 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:38:13.070 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:38:13.071 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:38:13.071 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:38:13.172 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:38:13.234 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:38:13.234 INFO:teuthology.orchestra.run.smithi092.stderr:Inferring config /var/lib/ceph/00db0016-dfed-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T21:38:13.234 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:38:13.235 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:38:13.272 INFO:teuthology.orchestra.run.smithi092.stderr:Inferring config /var/lib/ceph/00db0016-dfed-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T21:38:13.336 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:38:13.384 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:38:13.385 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:38:13.385 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:38:13.486 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:38:13.525 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:38:13.525 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:38:13.526 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:38:13.627 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:38:13.661 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:38:13.662 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:38:13.662 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:38:13.740 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:38:13 smithi092 ceph-mon[29572]: from='client.24469 v1:172.21.15.97:0/404208310' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:38:13.740 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:38:13 smithi092 ceph-mon[29572]: Health check update: Degraded data redundancy: 28688/254274 objects degraded (11.282%), 8 pgs degraded, 8 pgs undersized (PG_DEGRADED) 2024-03-11T21:38:13.740 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:38:13 smithi092 ceph-mon[37563]: from='client.24469 v1:172.21.15.97:0/404208310' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:38:13.740 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:38:13 smithi092 ceph-mon[37563]: Health check update: Degraded data redundancy: 28688/254274 objects degraded (11.282%), 8 pgs degraded, 8 pgs undersized (PG_DEGRADED) 2024-03-11T21:38:13.763 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:38:13.801 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:38:13.801 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:38:13.801 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:38:13.827 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:38:13 smithi097 ceph-mon[33317]: from='client.24469 v1:172.21.15.97:0/404208310' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:38:13.827 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:38:13 smithi097 ceph-mon[33317]: Health check update: Degraded data redundancy: 28688/254274 objects degraded (11.282%), 8 pgs degraded, 8 pgs undersized (PG_DEGRADED) 2024-03-11T21:38:13.903 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:38:13.930 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:38:13.931 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:38:13.932 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:38:14.032 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:38:14.072 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:38:14.072 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:38:14.073 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:38:14.174 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:38:14.206 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:38:14.206 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:38:14.207 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:38:14.307 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:38:14.335 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:38:14.336 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:38:14.336 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:38:14.437 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:38:14.479 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:38:14.479 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:38:14.480 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:38:14.524 INFO:teuthology.orchestra.run.smithi092.stdout:osd.0: {} 2024-03-11T21:38:14.563 INFO:teuthology.orchestra.run.smithi092.stdout:osd.1: {} 2024-03-11T21:38:14.577 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:38:14 smithi097 ceph-mon[33317]: pgmap v584: 148 pgs: 5 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+degraded+remapped, 140 active+clean; 5.0 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 1.1 KiB/s rd, 0 B/s wr, 768 op/s; 27936/245574 objects degraded (11.376%); 10 MiB/s, 162 objects/s recovering 2024-03-11T21:38:14.580 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:38:14.607 INFO:teuthology.orchestra.run.smithi092.stdout:osd.2: {} 2024-03-11T21:38:14.632 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:38:14.632 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:38:14.632 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:38:14.646 INFO:teuthology.orchestra.run.smithi092.stdout:osd.3: {} 2024-03-11T21:38:14.689 INFO:teuthology.orchestra.run.smithi092.stdout:osd.4: {} 2024-03-11T21:38:14.690 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:38:14 smithi092 ceph-mon[29572]: pgmap v584: 148 pgs: 5 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+degraded+remapped, 140 active+clean; 5.0 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 1.1 KiB/s rd, 0 B/s wr, 768 op/s; 27936/245574 objects degraded (11.376%); 10 MiB/s, 162 objects/s recovering 2024-03-11T21:38:14.690 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:38:14 smithi092 ceph-mon[37563]: pgmap v584: 148 pgs: 5 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+degraded+remapped, 140 active+clean; 5.0 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 1.1 KiB/s rd, 0 B/s wr, 768 op/s; 27936/245574 objects degraded (11.376%); 10 MiB/s, 162 objects/s recovering 2024-03-11T21:38:14.726 INFO:teuthology.orchestra.run.smithi092.stdout:osd.5: {} 2024-03-11T21:38:14.733 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:38:14.768 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:38:14.768 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:38:14.769 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:38:14.769 INFO:teuthology.orchestra.run.smithi092.stdout:osd.6: {} 2024-03-11T21:38:14.807 INFO:teuthology.orchestra.run.smithi092.stdout:osd.7: {} 2024-03-11T21:38:14.812 INFO:teuthology.orchestra.run.smithi092.stderr:osd.0: osd_enable_op_tracker = '' 2024-03-11T21:38:14.812 INFO:teuthology.orchestra.run.smithi092.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-11T21:38:14.812 INFO:teuthology.orchestra.run.smithi092.stderr:osd.2: osd_enable_op_tracker = '' 2024-03-11T21:38:14.812 INFO:teuthology.orchestra.run.smithi092.stderr:osd.3: osd_enable_op_tracker = '' 2024-03-11T21:38:14.812 INFO:teuthology.orchestra.run.smithi092.stderr:osd.4: osd_enable_op_tracker = '' 2024-03-11T21:38:14.812 INFO:teuthology.orchestra.run.smithi092.stderr:osd.5: osd_enable_op_tracker = '' 2024-03-11T21:38:14.813 INFO:teuthology.orchestra.run.smithi092.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-11T21:38:14.813 INFO:teuthology.orchestra.run.smithi092.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-11T21:38:14.869 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:38:14.905 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:38:14.905 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:38:14.905 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:38:15.006 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:38:15.034 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:38:15.035 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:38:15.035 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:38:15.136 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:38:15.168 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:38:15.169 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:38:15.169 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:38:15.270 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:38:15.318 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:38:15.318 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:38:15.319 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:38:15.419 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:38:15.449 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:38:15.450 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:38:15.450 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:38:15.551 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:38:15.589 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:38:15.590 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:38:15.590 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:38:15.691 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:38:15.725 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:38:15.726 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:38:15.726 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:38:15.754 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:38:15 smithi092 ceph-mon[29572]: from='client.? v1:172.21.15.92:0/2435079237' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-11T21:38:15.754 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:38:15 smithi092 ceph-mon[37563]: from='client.? v1:172.21.15.92:0/2435079237' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-11T21:38:15.774 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:38:15 smithi097 ceph-mon[33317]: from='client.? v1:172.21.15.92:0/2435079237' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-11T21:38:15.827 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:38:15.857 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:38:15.858 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:38:15.858 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:38:15.959 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:38:15.996 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:38:15.996 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:38:15.997 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:38:16.098 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:38:16.135 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:38:16.136 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:38:16.136 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:38:16.237 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:38:16.274 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:38:16.274 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:38:16.275 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:38:16.376 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:38:16.378 INFO:teuthology.orchestra.run.smithi097.stdout:{ 2024-03-11T21:38:16.378 INFO:teuthology.orchestra.run.smithi097.stdout: "ops": [], 2024-03-11T21:38:16.378 INFO:teuthology.orchestra.run.smithi097.stdout: "complaint_time": 30, 2024-03-11T21:38:16.378 INFO:teuthology.orchestra.run.smithi097.stdout: "num_blocked_ops": 0 2024-03-11T21:38:16.378 INFO:teuthology.orchestra.run.smithi097.stdout:} 2024-03-11T21:38:16.422 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:38:16.423 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:38:16.423 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:38:16.524 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:38:16.552 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:38:16.552 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:38:16.553 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:38:16.577 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:38:16 smithi097 ceph-mon[33317]: purged_snaps scrub starts 2024-03-11T21:38:16.577 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:38:16 smithi097 ceph-mon[33317]: purged_snaps scrub ok 2024-03-11T21:38:16.577 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:38:16 smithi097 ceph-mon[33317]: pgmap v585: 148 pgs: 5 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+degraded+remapped, 140 active+clean; 4.9 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 776 op/s; 27936/242784 objects degraded (11.507%); 9.0 MiB/s, 144 objects/s recovering 2024-03-11T21:38:16.577 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:38:16 smithi097 ceph-mon[33317]: osdmap e313: 8 total, 8 up, 8 in 2024-03-11T21:38:16.577 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:38:16 smithi097 ceph-mon[33317]: purged_snaps scrub starts 2024-03-11T21:38:16.577 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:38:16 smithi097 ceph-mon[33317]: purged_snaps scrub ok 2024-03-11T21:38:16.654 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:38:16.687 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:38:16.688 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:38:16.688 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:38:16.768 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:38:16 smithi092 ceph-mon[29572]: purged_snaps scrub starts 2024-03-11T21:38:16.769 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:38:16 smithi092 ceph-mon[29572]: purged_snaps scrub ok 2024-03-11T21:38:16.769 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:38:16 smithi092 ceph-mon[29572]: pgmap v585: 148 pgs: 5 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+degraded+remapped, 140 active+clean; 4.9 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 776 op/s; 27936/242784 objects degraded (11.507%); 9.0 MiB/s, 144 objects/s recovering 2024-03-11T21:38:16.769 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:38:16 smithi092 ceph-mon[29572]: osdmap e313: 8 total, 8 up, 8 in 2024-03-11T21:38:16.769 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:38:16 smithi092 ceph-mon[29572]: purged_snaps scrub starts 2024-03-11T21:38:16.769 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:38:16 smithi092 ceph-mon[29572]: purged_snaps scrub ok 2024-03-11T21:38:16.769 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:38:16 smithi092 ceph-mon[37563]: purged_snaps scrub starts 2024-03-11T21:38:16.769 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:38:16 smithi092 ceph-mon[37563]: purged_snaps scrub ok 2024-03-11T21:38:16.769 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:38:16 smithi092 ceph-mon[37563]: pgmap v585: 148 pgs: 5 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+degraded+remapped, 140 active+clean; 4.9 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 776 op/s; 27936/242784 objects degraded (11.507%); 9.0 MiB/s, 144 objects/s recovering 2024-03-11T21:38:16.769 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:38:16 smithi092 ceph-mon[37563]: osdmap e313: 8 total, 8 up, 8 in 2024-03-11T21:38:16.770 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:38:16 smithi092 ceph-mon[37563]: purged_snaps scrub starts 2024-03-11T21:38:16.770 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:38:16 smithi092 ceph-mon[37563]: purged_snaps scrub ok 2024-03-11T21:38:16.788 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:38:16.816 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:38:16.817 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:38:16.817 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:38:16.917 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:38:16.948 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:38:16.949 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:38:16.949 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:38:17.050 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:38:17.054 DEBUG:teuthology.orchestra.run.smithi097:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid 00db0016-dfed-11ee-95c8-87774f69a715 -- ceph daemon osd.5 dump_historic_ops 2024-03-11T21:38:17.105 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:38:17.105 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:38:17.105 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:38:17.206 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:38:17.242 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:38:17.242 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:38:17.243 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:38:17.344 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:38:17.389 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:38:17.390 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:38:17.391 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:38:17.492 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:38:17.533 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:38:17.534 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:38:17.534 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:38:17.604 INFO:teuthology.orchestra.run.smithi097.stderr:Inferring config /var/lib/ceph/00db0016-dfed-11ee-95c8-87774f69a715/mon.b/config 2024-03-11T21:38:17.635 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:38:17.687 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:38:17.687 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:38:17.687 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:38:17.691 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:38:17 smithi092 ceph-mon[29572]: osdmap e314: 8 total, 8 up, 8 in 2024-03-11T21:38:17.692 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:38:17 smithi092 ceph-mon[37563]: osdmap e314: 8 total, 8 up, 8 in 2024-03-11T21:38:17.738 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:38:17 smithi097 ceph-mon[33317]: osdmap e314: 8 total, 8 up, 8 in 2024-03-11T21:38:17.789 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:38:17.825 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:38:17.826 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:38:17.826 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:38:17.927 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:38:17.955 INFO:journalctl@ceph.mgr.y.smithi092.stdout:Mar 11 21:38:17 smithi092 ceph-00db0016-dfed-11ee-95c8-87774f69a715-mgr-y[29801]: ::ffff:172.21.15.97 - - [11/Mar/2024:21:38:17] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-11T21:38:17.963 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:38:17.963 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:38:17.964 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:38:18.064 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:38:18.110 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:38:18.111 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:38:18.111 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:38:18.212 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:38:18.244 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:38:18.244 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:38:18.245 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:38:18.346 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:38:18.401 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:38:18.402 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:38:18.402 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:38:18.441 INFO:teuthology.orchestra.run.smithi092.stdout: 2024-03-11T21:38:18.446 INFO:teuthology.orchestra.run.smithi092.stderr:dumped all 2024-03-11T21:38:18.503 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:38:18.542 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:38:18.542 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:38:18.542 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:38:18.643 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:38:18.680 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:38:18.681 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:38:18.681 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:38:18.782 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:38:18.818 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:38:18.819 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:38:18.819 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:38:18.828 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:38:18 smithi097 ceph-mon[33317]: pgmap v588: 148 pgs: 5 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+degraded+remapped, 140 active+clean; 4.9 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 639 B/s rd, 0 B/s wr, 508 op/s; 27615/242073 objects degraded (11.408%); 8.4 MiB/s, 134 objects/s recovering 2024-03-11T21:38:18.828 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:38:18 smithi097 ceph-mon[33317]: Health check update: Degraded data redundancy: 27615/242073 objects degraded (11.408%), 8 pgs degraded, 8 pgs undersized (PG_DEGRADED) 2024-03-11T21:38:18.861 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:38:18 smithi092 ceph-mon[29572]: pgmap v588: 148 pgs: 5 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+degraded+remapped, 140 active+clean; 4.9 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 639 B/s rd, 0 B/s wr, 508 op/s; 27615/242073 objects degraded (11.408%); 8.4 MiB/s, 134 objects/s recovering 2024-03-11T21:38:18.861 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:38:18 smithi092 ceph-mon[29572]: Health check update: Degraded data redundancy: 27615/242073 objects degraded (11.408%), 8 pgs degraded, 8 pgs undersized (PG_DEGRADED) 2024-03-11T21:38:18.861 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:38:18 smithi092 ceph-mon[37563]: pgmap v588: 148 pgs: 5 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+degraded+remapped, 140 active+clean; 4.9 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 639 B/s rd, 0 B/s wr, 508 op/s; 27615/242073 objects degraded (11.408%); 8.4 MiB/s, 134 objects/s recovering 2024-03-11T21:38:18.862 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:38:18 smithi092 ceph-mon[37563]: Health check update: Degraded data redundancy: 27615/242073 objects degraded (11.408%), 8 pgs degraded, 8 pgs undersized (PG_DEGRADED) 2024-03-11T21:38:18.920 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:38:18.943 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:38:18.944 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:38:18.944 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:38:19.045 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:38:19.085 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:38:19.086 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:38:19.087 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:38:19.145 DEBUG:teuthology.orchestra.run.smithi092:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid 00db0016-dfed-11ee-95c8-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-03-11T21:38:19.188 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:38:19.250 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:38:19.251 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:38:19.251 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:38:19.352 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:38:19.395 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:38:19.395 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:38:19.396 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:38:19.495 INFO:teuthology.orchestra.run.smithi092.stderr:nodeep-scrub is set 2024-03-11T21:38:19.496 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:38:19.557 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:38:19.558 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:38:19.558 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:38:19.659 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:38:19.689 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:38:19.690 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:38:19.691 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:38:19.791 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:38:19.820 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:38:19.820 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:38:19.821 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:38:19.827 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:38:19 smithi097 ceph-mon[33317]: from='client.15591 v1:172.21.15.92:0/1469377727' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-11T21:38:19.827 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:38:19 smithi097 ceph-mon[33317]: from='client.? v1:172.21.15.92:0/3104707477' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-11T21:38:19.827 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:38:19 smithi097 ceph-mon[33317]: from='client.25447 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-11T21:38:19.861 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:38:19 smithi092 ceph-mon[29572]: from='client.15591 v1:172.21.15.92:0/1469377727' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-11T21:38:19.861 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:38:19 smithi092 ceph-mon[29572]: from='client.? v1:172.21.15.92:0/3104707477' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-11T21:38:19.861 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:38:19 smithi092 ceph-mon[29572]: from='client.25447 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-11T21:38:19.862 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:38:19 smithi092 ceph-mon[37563]: from='client.15591 v1:172.21.15.92:0/1469377727' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-11T21:38:19.862 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:38:19 smithi092 ceph-mon[37563]: from='client.? v1:172.21.15.92:0/3104707477' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-11T21:38:19.862 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:38:19 smithi092 ceph-mon[37563]: from='client.25447 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-11T21:38:19.922 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:38:19.954 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:38:19.955 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:38:19.955 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:38:20.056 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:38:20.098 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:38:20.098 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:38:20.099 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:38:20.173 INFO:teuthology.orchestra.run.smithi092.stdout:{"pg_ready":true,"pg_map":{"version":588,"stamp":"2024-03-11T21:38:17.253366+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":5275720128,"num_objects":80691,"num_object_clones":0,"num_object_copies":242073,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":27615,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":80691,"num_whiteouts":0,"num_read":2083,"num_read_kb":1931,"num_write":153172,"num_write_kb":7450846,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10611,"num_bytes_recovered":685054406,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":85354,"ondisk_log_size":85354,"up":444,"acting":436,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":444,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":749961216,"kb_used":17833160,"kb_used_data":14470016,"kb_used_omap":15,"kb_used_meta":3362992,"kb_avail":732128056,"statfs":{"total":767960285184,"available":749699129344,"internally_reserved":0,"allocated":14817296384,"data_stored":14811256904,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15601,"internal_metadata":3443704591},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-266534912,"num_objects":-4067,"num_object_clones":0,"num_object_copies":-12201,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-1073,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-4067,"num_whiteouts":0,"num_read":5,"num_read_kb":5,"num_write":4067,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1073,"num_bytes_recovered":70320128,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.002516"},"pg_stats":[{"pgid":"6.1b","version":"0'0","reported_seq":473,"reported_epoch":312,"state":"active+clean","last_fresh":"2024-03-11T21:38:08.367544+0000","last_change":"2024-03-11T21:36:35.074931+0000","last_active":"2024-03-11T21:38:08.367544+0000","last_peered":"2024-03-11T21:38:08.367544+0000","last_clean":"2024-03-11T21:38:08.367544+0000","last_became_active":"2024-03-11T21:36:35.072636+0000","last_became_peered":"2024-03-11T21:36:35.072636+0000","last_unstale":"2024-03-11T21:38:08.367544+0000","last_undegraded":"2024-03-11T21:38:08.367544+0000","last_fullsized":"2024-03-11T21:38:08.367544+0000","mapping_epoch":277,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":278,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:35:07.021520+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:43.559452+0000","last_clean_scrub_stamp":"2024-03-11T21:35:07.021520+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":542,"reported_epoch":314,"state":"active+clean","last_fresh":"2024-03-11T21:38:16.475826+0000","last_change":"2024-03-11T21:36:00.720089+0000","last_active":"2024-03-11T21:38:16.475826+0000","last_peered":"2024-03-11T21:38:16.475826+0000","last_clean":"2024-03-11T21:38:16.475826+0000","last_became_active":"2024-03-11T21:25:42.578893+0000","last_became_peered":"2024-03-11T21:25:42.578893+0000","last_unstale":"2024-03-11T21:38:16.475826+0000","last_undegraded":"2024-03-11T21:38:16.475826+0000","last_fullsized":"2024-03-11T21:38:16.475826+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:36:00.719956+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:41.544550+0000","last_clean_scrub_stamp":"2024-03-11T21:36:00.719956+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1f","version":"0'0","reported_seq":567,"reported_epoch":312,"state":"active+clean","last_fresh":"2024-03-11T21:38:08.349363+0000","last_change":"2024-03-11T21:36:00.843838+0000","last_active":"2024-03-11T21:38:08.349363+0000","last_peered":"2024-03-11T21:38:08.349363+0000","last_clean":"2024-03-11T21:38:08.349363+0000","last_became_active":"2024-03-11T21:34:41.144945+0000","last_became_peered":"2024-03-11T21:34:41.144945+0000","last_unstale":"2024-03-11T21:38:08.349363+0000","last_undegraded":"2024-03-11T21:38:08.349363+0000","last_fullsized":"2024-03-11T21:38:08.349363+0000","mapping_epoch":226,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":227,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:36:00.843718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:37.512018+0000","last_clean_scrub_stamp":"2024-03-11T21:36:00.843718+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,4],"acting":[0,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.19","version":"260'27","reported_seq":520,"reported_epoch":312,"state":"active+clean","last_fresh":"2024-03-11T21:38:08.364531+0000","last_change":"2024-03-11T21:36:35.089901+0000","last_active":"2024-03-11T21:38:08.364531+0000","last_peered":"2024-03-11T21:38:08.364531+0000","last_clean":"2024-03-11T21:38:08.364531+0000","last_became_active":"2024-03-11T21:36:35.067652+0000","last_became_peered":"2024-03-11T21:36:35.067652+0000","last_unstale":"2024-03-11T21:38:08.364531+0000","last_undegraded":"2024-03-11T21:38:08.364531+0000","last_fullsized":"2024-03-11T21:38:08.364531+0000","mapping_epoch":277,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":278,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'15","last_scrub_stamp":"2024-03-11T21:35:22.099374+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:39.525554+0000","last_clean_scrub_stamp":"2024-03-11T21:35:22.099374+0000","objects_scrubbed":0,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":42,"num_read_kb":33,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.1a","version":"0'0","reported_seq":530,"reported_epoch":314,"state":"active+clean","last_fresh":"2024-03-11T21:38:16.476389+0000","last_change":"2024-03-11T21:35:16.410927+0000","last_active":"2024-03-11T21:38:16.476389+0000","last_peered":"2024-03-11T21:38:16.476389+0000","last_clean":"2024-03-11T21:38:16.476389+0000","last_became_active":"2024-03-11T21:25:44.590923+0000","last_became_peered":"2024-03-11T21:25:44.590923+0000","last_unstale":"2024-03-11T21:38:16.476389+0000","last_undegraded":"2024-03-11T21:38:16.476389+0000","last_fullsized":"2024-03-11T21:38:16.476389+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:35:16.410816+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:43.559452+0000","last_clean_scrub_stamp":"2024-03-11T21:35:16.410816+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":24,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,1],"acting":[4,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":550,"reported_epoch":312,"state":"active+clean","last_fresh":"2024-03-11T21:38:08.351539+0000","last_change":"2024-03-11T21:36:15.576138+0000","last_active":"2024-03-11T21:38:08.351539+0000","last_peered":"2024-03-11T21:38:08.351539+0000","last_clean":"2024-03-11T21:38:08.351539+0000","last_became_active":"2024-03-11T21:34:41.154216+0000","last_became_peered":"2024-03-11T21:34:41.154216+0000","last_unstale":"2024-03-11T21:38:08.351539+0000","last_undegraded":"2024-03-11T21:38:08.351539+0000","last_fullsized":"2024-03-11T21:38:08.351539+0000","mapping_epoch":226,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":227,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:36:15.575887+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:41.544550+0000","last_clean_scrub_stamp":"2024-03-11T21:36:15.575887+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,7],"acting":[1,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1e","version":"0'0","reported_seq":486,"reported_epoch":312,"state":"active+clean","last_fresh":"2024-03-11T21:38:08.357486+0000","last_change":"2024-03-11T21:36:35.062413+0000","last_active":"2024-03-11T21:38:08.357486+0000","last_peered":"2024-03-11T21:38:08.357486+0000","last_clean":"2024-03-11T21:38:08.357486+0000","last_became_active":"2024-03-11T21:36:35.061786+0000","last_became_peered":"2024-03-11T21:36:35.061786+0000","last_unstale":"2024-03-11T21:38:08.357486+0000","last_undegraded":"2024-03-11T21:38:08.357486+0000","last_fullsized":"2024-03-11T21:38:08.357486+0000","mapping_epoch":277,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":278,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:32:18.578038+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:37.512018+0000","last_clean_scrub_stamp":"2024-03-11T21:32:18.578038+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.18","version":"260'15","reported_seq":579,"reported_epoch":314,"state":"active+clean","last_fresh":"2024-03-11T21:38:16.475697+0000","last_change":"2024-03-11T21:36:09.662765+0000","last_active":"2024-03-11T21:38:16.475697+0000","last_peered":"2024-03-11T21:38:16.475697+0000","last_clean":"2024-03-11T21:38:16.475697+0000","last_became_active":"2024-03-11T21:25:40.552216+0000","last_became_peered":"2024-03-11T21:25:40.552216+0000","last_unstale":"2024-03-11T21:38:16.475697+0000","last_undegraded":"2024-03-11T21:38:16.475697+0000","last_fullsized":"2024-03-11T21:38:16.475697+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"260'15","last_scrub_stamp":"2024-03-11T21:36:09.662615+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:39.525554+0000","last_clean_scrub_stamp":"2024-03-11T21:36:09.662615+0000","objects_scrubbed":4,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":542,"reported_epoch":312,"state":"active+clean","last_fresh":"2024-03-11T21:38:08.352705+0000","last_change":"2024-03-11T21:36:35.068915+0000","last_active":"2024-03-11T21:38:08.352705+0000","last_peered":"2024-03-11T21:38:08.352705+0000","last_clean":"2024-03-11T21:38:08.352705+0000","last_became_active":"2024-03-11T21:36:35.068453+0000","last_became_peered":"2024-03-11T21:36:35.068453+0000","last_unstale":"2024-03-11T21:38:08.352705+0000","last_undegraded":"2024-03-11T21:38:08.352705+0000","last_fullsized":"2024-03-11T21:38:08.352705+0000","mapping_epoch":277,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":278,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:35:26.190634+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:32:12.654573+0000","last_clean_scrub_stamp":"2024-03-11T21:35:26.190634+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3],"acting":[5,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":190,"reported_epoch":312,"state":"active+clean","last_fresh":"2024-03-11T21:38:08.351535+0000","last_change":"2024-03-11T21:35:11.902920+0000","last_active":"2024-03-11T21:38:08.351535+0000","last_peered":"2024-03-11T21:38:08.351535+0000","last_clean":"2024-03-11T21:38:08.351535+0000","last_became_active":"2024-03-11T21:34:41.145299+0000","last_became_peered":"2024-03-11T21:34:41.145299+0000","last_unstale":"2024-03-11T21:38:08.351535+0000","last_undegraded":"2024-03-11T21:38:08.351535+0000","last_fullsized":"2024-03-11T21:38:08.351535+0000","mapping_epoch":226,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":227,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:35:11.902818+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:41.544550+0000","last_clean_scrub_stamp":"2024-03-11T21:35:11.902818+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.1d","version":"0'0","reported_seq":553,"reported_epoch":312,"state":"active+clean","last_fresh":"2024-03-11T21:38:08.360132+0000","last_change":"2024-03-11T21:35:26.855588+0000","last_active":"2024-03-11T21:38:08.360132+0000","last_peered":"2024-03-11T21:38:08.360132+0000","last_clean":"2024-03-11T21:38:08.360132+0000","last_became_active":"2024-03-11T21:25:38.542628+0000","last_became_peered":"2024-03-11T21:25:38.542628+0000","last_unstale":"2024-03-11T21:38:08.360132+0000","last_undegraded":"2024-03-11T21:38:08.360132+0000","last_fullsized":"2024-03-11T21:38:08.360132+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:35:26.855511+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:37.512018+0000","last_clean_scrub_stamp":"2024-03-11T21:35:26.855511+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,0],"acting":[7,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.1b","version":"260'7","reported_seq":565,"reported_epoch":314,"state":"active+clean","last_fresh":"2024-03-11T21:38:16.476161+0000","last_change":"2024-03-11T21:36:35.087276+0000","last_active":"2024-03-11T21:38:16.476161+0000","last_peered":"2024-03-11T21:38:16.476161+0000","last_clean":"2024-03-11T21:38:16.476161+0000","last_became_active":"2024-03-11T21:36:35.057678+0000","last_became_peered":"2024-03-11T21:36:35.057678+0000","last_unstale":"2024-03-11T21:38:16.476161+0000","last_undegraded":"2024-03-11T21:38:16.476161+0000","last_fullsized":"2024-03-11T21:38:16.476161+0000","mapping_epoch":277,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":278,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'5","last_scrub_stamp":"2024-03-11T21:35:09.427562+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:39.525554+0000","last_clean_scrub_stamp":"2024-03-11T21:35:09.427562+0000","objects_scrubbed":2,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":14,"num_read_kb":10,"num_write":8,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,1],"acting":[4,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":534,"reported_epoch":312,"state":"active+clean","last_fresh":"2024-03-11T21:38:08.351083+0000","last_change":"2024-03-11T21:35:39.691825+0000","last_active":"2024-03-11T21:38:08.351083+0000","last_peered":"2024-03-11T21:38:08.351083+0000","last_clean":"2024-03-11T21:38:08.351083+0000","last_became_active":"2024-03-11T21:25:44.589786+0000","last_became_peered":"2024-03-11T21:25:44.589786+0000","last_unstale":"2024-03-11T21:38:08.351083+0000","last_undegraded":"2024-03-11T21:38:08.351083+0000","last_fullsized":"2024-03-11T21:38:08.351083+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:35:39.691690+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:43.559452+0000","last_clean_scrub_stamp":"2024-03-11T21:35:39.691690+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":18,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,7],"acting":[0,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":548,"reported_epoch":312,"state":"active+clean","last_fresh":"2024-03-11T21:38:08.350436+0000","last_change":"2024-03-11T21:35:25.194240+0000","last_active":"2024-03-11T21:38:08.350436+0000","last_peered":"2024-03-11T21:38:08.350436+0000","last_clean":"2024-03-11T21:38:08.350436+0000","last_became_active":"2024-03-11T21:34:41.147185+0000","last_became_peered":"2024-03-11T21:34:41.147185+0000","last_unstale":"2024-03-11T21:38:08.350436+0000","last_undegraded":"2024-03-11T21:38:08.350436+0000","last_fullsized":"2024-03-11T21:38:08.350436+0000","mapping_epoch":226,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":227,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:35:25.194115+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:32:21.535034+0000","last_clean_scrub_stamp":"2024-03-11T21:35:25.194115+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,7],"acting":[5,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1c","version":"0'0","reported_seq":553,"reported_epoch":312,"state":"active+clean","last_fresh":"2024-03-11T21:38:08.355416+0000","last_change":"2024-03-11T21:35:28.789693+0000","last_active":"2024-03-11T21:38:08.355416+0000","last_peered":"2024-03-11T21:38:08.355416+0000","last_clean":"2024-03-11T21:38:08.355416+0000","last_became_active":"2024-03-11T21:25:38.541364+0000","last_became_peered":"2024-03-11T21:25:38.541364+0000","last_unstale":"2024-03-11T21:38:08.355416+0000","last_undegraded":"2024-03-11T21:38:08.355416+0000","last_fullsized":"2024-03-11T21:38:08.355416+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:35:28.789592+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:28:32.566437+0000","last_clean_scrub_stamp":"2024-03-11T21:35:28.789592+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.1a","version":"260'15","reported_seq":589,"reported_epoch":314,"state":"active+clean","last_fresh":"2024-03-11T21:38:16.475961+0000","last_change":"2024-03-11T21:36:35.085358+0000","last_active":"2024-03-11T21:38:16.475961+0000","last_peered":"2024-03-11T21:38:16.475961+0000","last_clean":"2024-03-11T21:38:16.475961+0000","last_became_active":"2024-03-11T21:36:35.057029+0000","last_became_peered":"2024-03-11T21:36:35.057029+0000","last_unstale":"2024-03-11T21:38:16.475961+0000","last_undegraded":"2024-03-11T21:38:16.475961+0000","last_fullsized":"2024-03-11T21:38:16.475961+0000","mapping_epoch":277,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":278,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'9","last_scrub_stamp":"2024-03-11T21:35:05.451780+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:39.525554+0000","last_clean_scrub_stamp":"2024-03-11T21:35:05.451780+0000","objects_scrubbed":4,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0],"acting":[4,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1f","version":"0'0","reported_seq":409,"reported_epoch":312,"state":"active+clean","last_fresh":"2024-03-11T21:38:08.367217+0000","last_change":"2024-03-11T21:36:35.074675+0000","last_active":"2024-03-11T21:38:08.367217+0000","last_peered":"2024-03-11T21:38:08.367217+0000","last_clean":"2024-03-11T21:38:08.367217+0000","last_became_active":"2024-03-11T21:36:35.071755+0000","last_became_peered":"2024-03-11T21:36:35.071755+0000","last_unstale":"2024-03-11T21:38:08.367217+0000","last_undegraded":"2024-03-11T21:38:08.367217+0000","last_fullsized":"2024-03-11T21:38:08.367217+0000","mapping_epoch":277,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":278,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:34:05.862891+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:43.559452+0000","last_clean_scrub_stamp":"2024-03-11T21:34:05.862891+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1c","version":"0'0","reported_seq":543,"reported_epoch":314,"state":"active+clean","last_fresh":"2024-03-11T21:38:16.476866+0000","last_change":"2024-03-11T21:36:35.067469+0000","last_active":"2024-03-11T21:38:16.476866+0000","last_peered":"2024-03-11T21:38:16.476866+0000","last_clean":"2024-03-11T21:38:16.476866+0000","last_became_active":"2024-03-11T21:36:35.067127+0000","last_became_peered":"2024-03-11T21:36:35.067127+0000","last_unstale":"2024-03-11T21:38:16.476866+0000","last_undegraded":"2024-03-11T21:38:16.476866+0000","last_fullsized":"2024-03-11T21:38:16.476866+0000","mapping_epoch":277,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":278,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:34:11.766066+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:41.544550+0000","last_clean_scrub_stamp":"2024-03-11T21:34:11.766066+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1b","version":"0'0","reported_seq":458,"reported_epoch":312,"state":"active+clean","last_fresh":"2024-03-11T21:38:08.367637+0000","last_change":"2024-03-11T21:36:35.075385+0000","last_active":"2024-03-11T21:38:08.367637+0000","last_peered":"2024-03-11T21:38:08.367637+0000","last_clean":"2024-03-11T21:38:08.367637+0000","last_became_active":"2024-03-11T21:36:35.073959+0000","last_became_peered":"2024-03-11T21:36:35.073959+0000","last_unstale":"2024-03-11T21:38:08.367637+0000","last_undegraded":"2024-03-11T21:38:08.367637+0000","last_fullsized":"2024-03-11T21:38:08.367637+0000","mapping_epoch":277,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":278,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:34:27.998724+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:34:27.998724+0000","last_clean_scrub_stamp":"2024-03-11T21:34:27.998724+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1d","version":"260'18","reported_seq":502,"reported_epoch":312,"state":"active+clean","last_fresh":"2024-03-11T21:38:08.357040+0000","last_change":"2024-03-11T21:36:35.080207+0000","last_active":"2024-03-11T21:38:08.357040+0000","last_peered":"2024-03-11T21:38:08.357040+0000","last_clean":"2024-03-11T21:38:08.357040+0000","last_became_active":"2024-03-11T21:36:35.055880+0000","last_became_peered":"2024-03-11T21:36:35.055880+0000","last_unstale":"2024-03-11T21:38:08.357040+0000","last_undegraded":"2024-03-11T21:38:08.357040+0000","last_fullsized":"2024-03-11T21:38:08.357040+0000","mapping_epoch":277,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":278,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'12","last_scrub_stamp":"2024-03-11T21:35:07.971581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:39.525554+0000","last_clean_scrub_stamp":"2024-03-11T21:35:07.971581+0000","objects_scrubbed":0,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":33,"num_read_kb":24,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.1e","version":"0'0","reported_seq":531,"reported_epoch":314,"state":"active+clean","last_fresh":"2024-03-11T21:38:16.477054+0000","last_change":"2024-03-11T21:35:32.664451+0000","last_active":"2024-03-11T21:38:16.477054+0000","last_peered":"2024-03-11T21:38:16.477054+0000","last_clean":"2024-03-11T21:38:16.477054+0000","last_became_active":"2024-03-11T21:25:45.300355+0000","last_became_peered":"2024-03-11T21:25:45.300355+0000","last_unstale":"2024-03-11T21:38:16.477054+0000","last_undegraded":"2024-03-11T21:38:16.477054+0000","last_fullsized":"2024-03-11T21:38:16.477054+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:35:32.664371+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:43.559452+0000","last_clean_scrub_stamp":"2024-03-11T21:35:32.664371+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5],"acting":[4,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1d","version":"0'0","reported_seq":538,"reported_epoch":312,"state":"active+clean","last_fresh":"2024-03-11T21:38:08.356299+0000","last_change":"2024-03-11T21:36:07.624196+0000","last_active":"2024-03-11T21:38:08.356299+0000","last_peered":"2024-03-11T21:38:08.356299+0000","last_clean":"2024-03-11T21:38:08.356299+0000","last_became_active":"2024-03-11T21:25:42.579407+0000","last_became_peered":"2024-03-11T21:25:42.579407+0000","last_unstale":"2024-03-11T21:38:08.356299+0000","last_undegraded":"2024-03-11T21:38:08.356299+0000","last_fullsized":"2024-03-11T21:38:08.356299+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:36:07.624075+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:30:18.678531+0000","last_clean_scrub_stamp":"2024-03-11T21:36:07.624075+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1a","version":"0'0","reported_seq":566,"reported_epoch":312,"state":"active+clean","last_fresh":"2024-03-11T21:38:08.346590+0000","last_change":"2024-03-11T21:36:12.993542+0000","last_active":"2024-03-11T21:38:08.346590+0000","last_peered":"2024-03-11T21:38:08.346590+0000","last_clean":"2024-03-11T21:38:08.346590+0000","last_became_active":"2024-03-11T21:34:41.145014+0000","last_became_peered":"2024-03-11T21:34:41.145014+0000","last_unstale":"2024-03-11T21:38:08.346590+0000","last_undegraded":"2024-03-11T21:38:08.346590+0000","last_fullsized":"2024-03-11T21:38:08.346590+0000","mapping_epoch":226,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":227,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:36:12.993437+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:37.512018+0000","last_clean_scrub_stamp":"2024-03-11T21:36:12.993437+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.1c","version":"260'27","reported_seq":600,"reported_epoch":312,"state":"active+clean","last_fresh":"2024-03-11T21:38:08.366760+0000","last_change":"2024-03-11T21:36:35.097208+0000","last_active":"2024-03-11T21:38:08.366760+0000","last_peered":"2024-03-11T21:38:08.366760+0000","last_clean":"2024-03-11T21:38:08.366760+0000","last_became_active":"2024-03-11T21:36:35.074579+0000","last_became_peered":"2024-03-11T21:36:35.074579+0000","last_unstale":"2024-03-11T21:38:08.366760+0000","last_undegraded":"2024-03-11T21:38:08.366760+0000","last_fullsized":"2024-03-11T21:38:08.366760+0000","mapping_epoch":277,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":278,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'15","last_scrub_stamp":"2024-03-11T21:34:10.311181+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:39.525554+0000","last_clean_scrub_stamp":"2024-03-11T21:34:10.311181+0000","objects_scrubbed":7,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":44,"num_read_kb":35,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.1d","version":"0'0","reported_seq":525,"reported_epoch":312,"state":"active+clean","last_fresh":"2024-03-11T21:38:08.355823+0000","last_change":"2024-03-11T21:35:21.744646+0000","last_active":"2024-03-11T21:38:08.355823+0000","last_peered":"2024-03-11T21:38:08.355823+0000","last_clean":"2024-03-11T21:38:08.355823+0000","last_became_active":"2024-03-11T21:25:44.588845+0000","last_became_peered":"2024-03-11T21:25:44.588845+0000","last_unstale":"2024-03-11T21:38:08.355823+0000","last_undegraded":"2024-03-11T21:38:08.355823+0000","last_fullsized":"2024-03-11T21:38:08.355823+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:35:21.744223+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:43.559452+0000","last_clean_scrub_stamp":"2024-03-11T21:35:21.744223+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":99,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.1e","version":"0'0","reported_seq":539,"reported_epoch":312,"state":"active+clean","last_fresh":"2024-03-11T21:38:08.349273+0000","last_change":"2024-03-11T21:35:10.728691+0000","last_active":"2024-03-11T21:38:08.349273+0000","last_peered":"2024-03-11T21:38:08.349273+0000","last_clean":"2024-03-11T21:38:08.349273+0000","last_became_active":"2024-03-11T21:25:42.582959+0000","last_became_peered":"2024-03-11T21:25:42.582959+0000","last_unstale":"2024-03-11T21:38:08.349273+0000","last_undegraded":"2024-03-11T21:38:08.349273+0000","last_fullsized":"2024-03-11T21:38:08.349273+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:35:10.728569+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:26:58.335431+0000","last_clean_scrub_stamp":"2024-03-11T21:35:10.728569+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":18,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.19","version":"67'2","reported_seq":484,"reported_epoch":312,"state":"active+clean","last_fresh":"2024-03-11T21:38:08.364422+0000","last_change":"2024-03-11T21:36:35.069729+0000","last_active":"2024-03-11T21:38:08.364422+0000","last_peered":"2024-03-11T21:38:08.364422+0000","last_clean":"2024-03-11T21:38:08.364422+0000","last_became_active":"2024-03-11T21:36:35.064249+0000","last_became_peered":"2024-03-11T21:36:35.064249+0000","last_unstale":"2024-03-11T21:38:08.364422+0000","last_undegraded":"2024-03-11T21:38:08.364422+0000","last_fullsized":"2024-03-11T21:38:08.364422+0000","mapping_epoch":277,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":278,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'2","last_scrub_stamp":"2024-03-11T21:32:15.509542+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:37.512018+0000","last_clean_scrub_stamp":"2024-03-11T21:32:15.509542+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":957,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,0],"acting":[3,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1f","version":"260'24","reported_seq":605,"reported_epoch":312,"state":"active+clean","last_fresh":"2024-03-11T21:38:08.347936+0000","last_change":"2024-03-11T21:35:38.173937+0000","last_active":"2024-03-11T21:38:08.347936+0000","last_peered":"2024-03-11T21:38:08.347936+0000","last_clean":"2024-03-11T21:38:08.347936+0000","last_became_active":"2024-03-11T21:32:22.798242+0000","last_became_peered":"2024-03-11T21:32:22.798242+0000","last_unstale":"2024-03-11T21:38:08.347936+0000","last_undegraded":"2024-03-11T21:38:08.347936+0000","last_fullsized":"2024-03-11T21:38:08.347936+0000","mapping_epoch":166,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":167,"parent":"0.0","parent_split_bits":0,"last_scrub":"124'16","last_scrub_stamp":"2024-03-11T21:35:38.173841+0000","last_deep_scrub":"124'16","last_deep_scrub_stamp":"2024-03-11T21:35:38.173841+0000","last_clean_scrub_stamp":"2024-03-11T21:35:38.173841+0000","objects_scrubbed":5,"log_size":24,"log_dups_size":0,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":32,"num_read_kb":25,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,0],"acting":[6,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.1c","version":"74'1","reported_seq":194,"reported_epoch":312,"state":"active+clean","last_fresh":"2024-03-11T21:38:08.355293+0000","last_change":"2024-03-11T21:36:11.732043+0000","last_active":"2024-03-11T21:38:08.355293+0000","last_peered":"2024-03-11T21:38:08.355293+0000","last_clean":"2024-03-11T21:38:08.355293+0000","last_became_active":"2024-03-11T21:34:41.144839+0000","last_became_peered":"2024-03-11T21:34:41.144839+0000","last_unstale":"2024-03-11T21:38:08.355293+0000","last_undegraded":"2024-03-11T21:38:08.355293+0000","last_fullsized":"2024-03-11T21:38:08.355293+0000","mapping_epoch":226,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":227,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'1","last_scrub_stamp":"2024-03-11T21:36:11.731914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:43.559452+0000","last_clean_scrub_stamp":"2024-03-11T21:36:11.731914+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":370,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":370,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.1f","version":"229'10","reported_seq":737,"reported_epoch":312,"state":"active+clean","last_fresh":"2024-03-11T21:38:08.751938+0000","last_change":"2024-03-11T21:36:08.871358+0000","last_active":"2024-03-11T21:38:08.751938+0000","last_peered":"2024-03-11T21:38:08.751938+0000","last_clean":"2024-03-11T21:38:08.751938+0000","last_became_active":"2024-03-11T21:34:41.145862+0000","last_became_peered":"2024-03-11T21:34:41.145862+0000","last_unstale":"2024-03-11T21:38:08.751938+0000","last_undegraded":"2024-03-11T21:38:08.751938+0000","last_fullsized":"2024-03-11T21:38:08.751938+0000","mapping_epoch":226,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":227,"parent":"0.0","parent_split_bits":0,"last_scrub":"229'10","last_scrub_stamp":"2024-03-11T21:36:08.871213+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:41.544550+0000","last_clean_scrub_stamp":"2024-03-11T21:36:08.871213+0000","objects_scrubbed":1,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.18","version":"67'1","reported_seq":589,"reported_epoch":312,"state":"active+clean","last_fresh":"2024-03-11T21:38:08.352248+0000","last_change":"2024-03-11T21:36:10.458150+0000","last_active":"2024-03-11T21:38:08.352248+0000","last_peered":"2024-03-11T21:38:08.352248+0000","last_clean":"2024-03-11T21:38:08.352248+0000","last_became_active":"2024-03-11T21:35:23.310241+0000","last_became_peered":"2024-03-11T21:35:23.310241+0000","last_unstale":"2024-03-11T21:38:08.352248+0000","last_undegraded":"2024-03-11T21:38:08.352248+0000","last_fullsized":"2024-03-11T21:38:08.352248+0000","mapping_epoch":248,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":249,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'1","last_scrub_stamp":"2024-03-11T21:36:10.458028+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:37.512018+0000","last_clean_scrub_stamp":"2024-03-11T21:36:10.458028+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":436,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":7,"num_read_kb":7,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":1308,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,2],"acting":[5,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1e","version":"260'20","reported_seq":631,"reported_epoch":312,"state":"active+clean","last_fresh":"2024-03-11T21:38:08.349170+0000","last_change":"2024-03-11T21:36:35.083447+0000","last_active":"2024-03-11T21:38:08.349170+0000","last_peered":"2024-03-11T21:38:08.349170+0000","last_clean":"2024-03-11T21:38:08.349170+0000","last_became_active":"2024-03-11T21:36:35.057843+0000","last_became_peered":"2024-03-11T21:36:35.057843+0000","last_unstale":"2024-03-11T21:38:08.349170+0000","last_undegraded":"2024-03-11T21:38:08.349170+0000","last_fullsized":"2024-03-11T21:38:08.349170+0000","mapping_epoch":277,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":278,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'10","last_scrub_stamp":"2024-03-11T21:35:16.825146+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:39.525554+0000","last_clean_scrub_stamp":"2024-03-11T21:35:16.825146+0000","objects_scrubbed":5,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":44,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":25,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.13","version":"0'0","reported_seq":417,"reported_epoch":312,"state":"active+clean","last_fresh":"2024-03-11T21:38:08.367850+0000","last_change":"2024-03-11T21:36:35.063744+0000","last_active":"2024-03-11T21:38:08.367850+0000","last_peered":"2024-03-11T21:38:08.367850+0000","last_clean":"2024-03-11T21:38:08.367850+0000","last_became_active":"2024-03-11T21:36:35.062618+0000","last_became_peered":"2024-03-11T21:36:35.062618+0000","last_unstale":"2024-03-11T21:38:08.367850+0000","last_undegraded":"2024-03-11T21:38:08.367850+0000","last_fullsized":"2024-03-11T21:38:08.367850+0000","mapping_epoch":277,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":278,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:34:13.822803+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:43.559452+0000","last_clean_scrub_stamp":"2024-03-11T21:34:13.822803+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":190,"reported_epoch":312,"state":"active+clean","last_fresh":"2024-03-11T21:38:08.350863+0000","last_change":"2024-03-11T21:35:16.917648+0000","last_active":"2024-03-11T21:38:08.350863+0000","last_peered":"2024-03-11T21:38:08.350863+0000","last_clean":"2024-03-11T21:38:08.350863+0000","last_became_active":"2024-03-11T21:34:41.142569+0000","last_became_peered":"2024-03-11T21:34:41.142569+0000","last_unstale":"2024-03-11T21:38:08.350863+0000","last_undegraded":"2024-03-11T21:38:08.350863+0000","last_fullsized":"2024-03-11T21:38:08.350863+0000","mapping_epoch":226,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":227,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:35:16.917532+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:41.544550+0000","last_clean_scrub_stamp":"2024-03-11T21:35:16.917532+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.17","version":"0'0","reported_seq":546,"reported_epoch":312,"state":"active+clean","last_fresh":"2024-03-11T21:38:08.347663+0000","last_change":"2024-03-11T21:35:06.186953+0000","last_active":"2024-03-11T21:38:08.347663+0000","last_peered":"2024-03-11T21:38:08.347663+0000","last_clean":"2024-03-11T21:38:08.347663+0000","last_became_active":"2024-03-11T21:25:38.540349+0000","last_became_peered":"2024-03-11T21:25:38.540349+0000","last_unstale":"2024-03-11T21:38:08.347663+0000","last_undegraded":"2024-03-11T21:38:08.347663+0000","last_fullsized":"2024-03-11T21:38:08.347663+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:35:06.186840+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:28:25.918101+0000","last_clean_scrub_stamp":"2024-03-11T21:35:06.186840+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,2],"acting":[6,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.11","version":"260'19","reported_seq":543,"reported_epoch":312,"state":"active+clean","last_fresh":"2024-03-11T21:38:08.364880+0000","last_change":"2024-03-11T21:36:35.096720+0000","last_active":"2024-03-11T21:38:08.364880+0000","last_peered":"2024-03-11T21:38:08.364880+0000","last_clean":"2024-03-11T21:38:08.364880+0000","last_became_active":"2024-03-11T21:36:35.073689+0000","last_became_peered":"2024-03-11T21:36:35.073689+0000","last_unstale":"2024-03-11T21:38:08.364880+0000","last_undegraded":"2024-03-11T21:38:08.364880+0000","last_fullsized":"2024-03-11T21:38:08.364880+0000","mapping_epoch":277,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":278,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'11","last_scrub_stamp":"2024-03-11T21:35:16.044397+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:39.525554+0000","last_clean_scrub_stamp":"2024-03-11T21:35:16.044397+0000","objects_scrubbed":5,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":31,"num_read_kb":24,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":190,"reported_epoch":312,"state":"active+clean","last_fresh":"2024-03-11T21:38:08.351384+0000","last_change":"2024-03-11T21:35:32.719869+0000","last_active":"2024-03-11T21:38:08.351384+0000","last_peered":"2024-03-11T21:38:08.351384+0000","last_clean":"2024-03-11T21:38:08.351384+0000","last_became_active":"2024-03-11T21:34:41.137235+0000","last_became_peered":"2024-03-11T21:34:41.137235+0000","last_unstale":"2024-03-11T21:38:08.351384+0000","last_undegraded":"2024-03-11T21:38:08.351384+0000","last_fullsized":"2024-03-11T21:38:08.351384+0000","mapping_epoch":226,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":227,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:35:32.719789+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:43.559452+0000","last_clean_scrub_stamp":"2024-03-11T21:35:32.719789+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,4],"acting":[7,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":558,"reported_epoch":312,"state":"active+clean","last_fresh":"2024-03-11T21:38:08.345263+0000","last_change":"2024-03-11T21:35:31.111662+0000","last_active":"2024-03-11T21:38:08.345263+0000","last_peered":"2024-03-11T21:38:08.345263+0000","last_clean":"2024-03-11T21:38:08.345263+0000","last_became_active":"2024-03-11T21:34:41.156036+0000","last_became_peered":"2024-03-11T21:34:41.156036+0000","last_unstale":"2024-03-11T21:38:08.345263+0000","last_undegraded":"2024-03-11T21:38:08.345263+0000","last_fullsized":"2024-03-11T21:38:08.345263+0000","mapping_epoch":226,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":227,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:35:31.111559+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:29:39.923895+0000","last_clean_scrub_stamp":"2024-03-11T21:35:31.111559+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.16","version":"0'0","reported_seq":553,"reported_epoch":312,"state":"active+clean","last_fresh":"2024-03-11T21:38:08.352473+0000","last_change":"2024-03-11T21:35:39.108623+0000","last_active":"2024-03-11T21:38:08.352473+0000","last_peered":"2024-03-11T21:38:08.352473+0000","last_clean":"2024-03-11T21:38:08.352473+0000","last_became_active":"2024-03-11T21:25:38.539103+0000","last_became_peered":"2024-03-11T21:25:38.539103+0000","last_unstale":"2024-03-11T21:38:08.352473+0000","last_undegraded":"2024-03-11T21:38:08.352473+0000","last_fullsized":"2024-03-11T21:38:08.352473+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:35:39.108546+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:27:11.938909+0000","last_clean_scrub_stamp":"2024-03-11T21:35:39.108546+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,2],"acting":[5,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.10","version":"260'10","reported_seq":425,"reported_epoch":312,"state":"active+clean","last_fresh":"2024-03-11T21:38:08.356845+0000","last_change":"2024-03-11T21:36:35.084268+0000","last_active":"2024-03-11T21:38:08.356845+0000","last_peered":"2024-03-11T21:38:08.356845+0000","last_clean":"2024-03-11T21:38:08.356845+0000","last_became_active":"2024-03-11T21:36:35.067500+0000","last_became_peered":"2024-03-11T21:36:35.067500+0000","last_unstale":"2024-03-11T21:38:08.356845+0000","last_undegraded":"2024-03-11T21:38:08.356845+0000","last_fullsized":"2024-03-11T21:38:08.356845+0000","mapping_epoch":277,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":278,"parent":"0.0","parent_split_bits":0,"last_scrub":"94'6","last_scrub_stamp":"2024-03-11T21:34:00.727416+0000","last_deep_scrub":"94'6","last_deep_scrub_stamp":"2024-03-11T21:28:24.603974+0000","last_clean_scrub_stamp":"2024-03-11T21:34:00.727416+0000","objects_scrubbed":2,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":12,"num_read_kb":10,"num_write":8,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6],"acting":[3,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":470,"reported_epoch":312,"state":"active+clean","last_fresh":"2024-03-11T21:38:08.364713+0000","last_change":"2024-03-11T21:36:35.070468+0000","last_active":"2024-03-11T21:38:08.364713+0000","last_peered":"2024-03-11T21:38:08.364713+0000","last_clean":"2024-03-11T21:38:08.364713+0000","last_became_active":"2024-03-11T21:36:35.070176+0000","last_became_peered":"2024-03-11T21:36:35.070176+0000","last_unstale":"2024-03-11T21:38:08.364713+0000","last_undegraded":"2024-03-11T21:38:08.364713+0000","last_fullsized":"2024-03-11T21:38:08.364713+0000","mapping_epoch":277,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":278,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:35:18.062272+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:43.559452+0000","last_clean_scrub_stamp":"2024-03-11T21:35:18.062272+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":539,"reported_epoch":312,"state":"active+clean","last_fresh":"2024-03-11T21:38:08.360253+0000","last_change":"2024-03-11T21:36:35.058551+0000","last_active":"2024-03-11T21:38:08.360253+0000","last_peered":"2024-03-11T21:38:08.360253+0000","last_clean":"2024-03-11T21:38:08.360253+0000","last_became_active":"2024-03-11T21:36:35.057906+0000","last_became_peered":"2024-03-11T21:36:35.057906+0000","last_unstale":"2024-03-11T21:38:08.360253+0000","last_undegraded":"2024-03-11T21:38:08.360253+0000","last_fullsized":"2024-03-11T21:38:08.360253+0000","mapping_epoch":277,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":278,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:34:15.516278+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:27:21.207303+0000","last_clean_scrub_stamp":"2024-03-11T21:34:15.516278+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.15","version":"0'0","reported_seq":546,"reported_epoch":312,"state":"active+clean","last_fresh":"2024-03-11T21:38:08.355994+0000","last_change":"2024-03-11T21:36:06.579754+0000","last_active":"2024-03-11T21:38:08.355994+0000","last_peered":"2024-03-11T21:38:08.355994+0000","last_clean":"2024-03-11T21:38:08.355994+0000","last_became_active":"2024-03-11T21:25:38.541705+0000","last_became_peered":"2024-03-11T21:25:38.541705+0000","last_unstale":"2024-03-11T21:38:08.355994+0000","last_undegraded":"2024-03-11T21:38:08.355994+0000","last_fullsized":"2024-03-11T21:38:08.355994+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:36:06.579636+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:26:44.860499+0000","last_clean_scrub_stamp":"2024-03-11T21:36:06.579636+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.13","version":"260'24","reported_seq":600,"reported_epoch":314,"state":"active+clean","last_fresh":"2024-03-11T21:38:16.476312+0000","last_change":"2024-03-11T21:35:14.453053+0000","last_active":"2024-03-11T21:38:16.476312+0000","last_peered":"2024-03-11T21:38:16.476312+0000","last_clean":"2024-03-11T21:38:16.476312+0000","last_became_active":"2024-03-11T21:26:11.214879+0000","last_became_peered":"2024-03-11T21:26:11.214879+0000","last_unstale":"2024-03-11T21:38:16.476312+0000","last_undegraded":"2024-03-11T21:38:16.476312+0000","last_fullsized":"2024-03-11T21:38:16.476312+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"109'16","last_scrub_stamp":"2024-03-11T21:35:14.452891+0000","last_deep_scrub":"109'16","last_deep_scrub_stamp":"2024-03-11T21:32:24.657490+0000","last_clean_scrub_stamp":"2024-03-11T21:35:14.452891+0000","objects_scrubbed":5,"log_size":24,"log_dups_size":0,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":32,"num_read_kb":25,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,2],"acting":[4,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.10","version":"0'0","reported_seq":534,"reported_epoch":312,"state":"active+clean","last_fresh":"2024-03-11T21:38:08.346332+0000","last_change":"2024-03-11T21:35:29.712658+0000","last_active":"2024-03-11T21:38:08.346332+0000","last_peered":"2024-03-11T21:38:08.346332+0000","last_clean":"2024-03-11T21:38:08.346332+0000","last_became_active":"2024-03-11T21:25:44.591189+0000","last_became_peered":"2024-03-11T21:25:44.591189+0000","last_unstale":"2024-03-11T21:38:08.346332+0000","last_undegraded":"2024-03-11T21:38:08.346332+0000","last_fullsized":"2024-03-11T21:38:08.346332+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:35:29.712557+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:27:25.417450+0000","last_clean_scrub_stamp":"2024-03-11T21:35:29.712557+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,1],"acting":[0,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":468,"reported_epoch":312,"state":"active+clean","last_fresh":"2024-03-11T21:38:08.367929+0000","last_change":"2024-03-11T21:36:35.062276+0000","last_active":"2024-03-11T21:38:08.367929+0000","last_peered":"2024-03-11T21:38:08.367929+0000","last_clean":"2024-03-11T21:38:08.367929+0000","last_became_active":"2024-03-11T21:36:35.056619+0000","last_became_peered":"2024-03-11T21:36:35.056619+0000","last_unstale":"2024-03-11T21:38:08.367929+0000","last_undegraded":"2024-03-11T21:38:08.367929+0000","last_fullsized":"2024-03-11T21:38:08.367929+0000","mapping_epoch":277,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":278,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:35:13.060487+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:31:15.367898+0000","last_clean_scrub_stamp":"2024-03-11T21:35:13.060487+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.14","version":"0'0","reported_seq":563,"reported_epoch":312,"state":"active+clean","last_fresh":"2024-03-11T21:38:08.347264+0000","last_change":"2024-03-11T21:36:35.068386+0000","last_active":"2024-03-11T21:38:08.347264+0000","last_peered":"2024-03-11T21:38:08.347264+0000","last_clean":"2024-03-11T21:38:08.347264+0000","last_became_active":"2024-03-11T21:36:35.068038+0000","last_became_peered":"2024-03-11T21:36:35.068038+0000","last_unstale":"2024-03-11T21:38:08.347264+0000","last_undegraded":"2024-03-11T21:38:08.347264+0000","last_fullsized":"2024-03-11T21:38:08.347264+0000","mapping_epoch":277,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":278,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:35:09.122761+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:37.512018+0000","last_clean_scrub_stamp":"2024-03-11T21:35:09.122761+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,5],"acting":[6,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.12","version":"260'15","reported_seq":567,"reported_epoch":312,"state":"active+clean","last_fresh":"2024-03-11T21:38:08.360065+0000","last_change":"2024-03-11T21:35:32.590439+0000","last_active":"2024-03-11T21:38:08.360065+0000","last_peered":"2024-03-11T21:38:08.360065+0000","last_clean":"2024-03-11T21:38:08.360065+0000","last_became_active":"2024-03-11T21:25:40.547082+0000","last_became_peered":"2024-03-11T21:25:40.547082+0000","last_unstale":"2024-03-11T21:38:08.360065+0000","last_undegraded":"2024-03-11T21:38:08.360065+0000","last_fullsized":"2024-03-11T21:38:08.360065+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'9","last_scrub_stamp":"2024-03-11T21:35:32.590318+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:39.525554+0000","last_clean_scrub_stamp":"2024-03-11T21:35:32.590318+0000","objects_scrubbed":4,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":538,"reported_epoch":314,"state":"active+clean","last_fresh":"2024-03-11T21:38:16.476727+0000","last_change":"2024-03-11T21:36:04.754168+0000","last_active":"2024-03-11T21:38:16.476727+0000","last_peered":"2024-03-11T21:38:16.476727+0000","last_clean":"2024-03-11T21:38:16.476727+0000","last_became_active":"2024-03-11T21:25:44.589586+0000","last_became_peered":"2024-03-11T21:25:44.589586+0000","last_unstale":"2024-03-11T21:38:16.476727+0000","last_undegraded":"2024-03-11T21:38:16.476727+0000","last_fullsized":"2024-03-11T21:38:16.476727+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:36:04.753979+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:27:09.801138+0000","last_clean_scrub_stamp":"2024-03-11T21:36:04.753979+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":113,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,5],"acting":[4,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.14","version":"280'12","reported_seq":649,"reported_epoch":312,"state":"active+clean","last_fresh":"2024-03-11T21:38:08.365548+0000","last_change":"2024-03-11T21:36:35.074550+0000","last_active":"2024-03-11T21:38:08.365548+0000","last_peered":"2024-03-11T21:38:08.365548+0000","last_clean":"2024-03-11T21:38:08.365548+0000","last_became_active":"2024-03-11T21:36:35.073912+0000","last_became_peered":"2024-03-11T21:36:35.073912+0000","last_unstale":"2024-03-11T21:38:08.365548+0000","last_undegraded":"2024-03-11T21:38:08.365548+0000","last_fullsized":"2024-03-11T21:38:08.365548+0000","mapping_epoch":277,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":278,"parent":"0.0","parent_split_bits":0,"last_scrub":"229'10","last_scrub_stamp":"2024-03-11T21:35:26.182107+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:41.544550+0000","last_clean_scrub_stamp":"2024-03-11T21:35:26.182107+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,2],"acting":[3,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.13","version":"0'0","reported_seq":547,"reported_epoch":312,"state":"active+clean","last_fresh":"2024-03-11T21:38:08.349518+0000","last_change":"2024-03-11T21:36:35.057688+0000","last_active":"2024-03-11T21:38:08.349518+0000","last_peered":"2024-03-11T21:38:08.349518+0000","last_clean":"2024-03-11T21:38:08.349518+0000","last_became_active":"2024-03-11T21:36:35.057132+0000","last_became_peered":"2024-03-11T21:36:35.057132+0000","last_unstale":"2024-03-11T21:38:08.349518+0000","last_undegraded":"2024-03-11T21:38:08.349518+0000","last_fullsized":"2024-03-11T21:38:08.349518+0000","mapping_epoch":277,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":278,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:34:08.650371+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:37.512018+0000","last_clean_scrub_stamp":"2024-03-11T21:34:08.650371+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.15","version":"260'19","reported_seq":620,"reported_epoch":312,"state":"active+clean","last_fresh":"2024-03-11T21:38:08.351395+0000","last_change":"2024-03-11T21:36:35.088986+0000","last_active":"2024-03-11T21:38:08.351395+0000","last_peered":"2024-03-11T21:38:08.351395+0000","last_clean":"2024-03-11T21:38:08.351395+0000","last_became_active":"2024-03-11T21:36:35.057285+0000","last_became_peered":"2024-03-11T21:36:35.057285+0000","last_unstale":"2024-03-11T21:38:08.351395+0000","last_undegraded":"2024-03-11T21:38:08.351395+0000","last_fullsized":"2024-03-11T21:38:08.351395+0000","mapping_epoch":277,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":278,"parent":"0.0","parent_split_bits":0,"last_scrub":"94'13","last_scrub_stamp":"2024-03-11T21:35:29.178543+0000","last_deep_scrub":"94'13","last_deep_scrub_stamp":"2024-03-11T21:29:41.853158+0000","last_clean_scrub_stamp":"2024-03-11T21:35:29.178543+0000","objects_scrubbed":4,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":11,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.16","version":"0'0","reported_seq":555,"reported_epoch":312,"state":"active+clean","last_fresh":"2024-03-11T21:38:08.348228+0000","last_change":"2024-03-11T21:36:35.058372+0000","last_active":"2024-03-11T21:38:08.348228+0000","last_peered":"2024-03-11T21:38:08.348228+0000","last_clean":"2024-03-11T21:38:08.348228+0000","last_became_active":"2024-03-11T21:36:35.057355+0000","last_became_peered":"2024-03-11T21:36:35.057355+0000","last_unstale":"2024-03-11T21:38:08.348228+0000","last_undegraded":"2024-03-11T21:38:08.348228+0000","last_fullsized":"2024-03-11T21:38:08.348228+0000","mapping_epoch":277,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":278,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:35:07.665075+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:43.559452+0000","last_clean_scrub_stamp":"2024-03-11T21:35:07.665075+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":35,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.15","version":"98'9","reported_seq":697,"reported_epoch":312,"state":"active+clean","last_fresh":"2024-03-11T21:38:08.351566+0000","last_change":"2024-03-11T21:36:03.366581+0000","last_active":"2024-03-11T21:38:08.351566+0000","last_peered":"2024-03-11T21:38:08.351566+0000","last_clean":"2024-03-11T21:38:08.351566+0000","last_became_active":"2024-03-11T21:25:42.572555+0000","last_became_peered":"2024-03-11T21:25:42.572555+0000","last_unstale":"2024-03-11T21:38:08.351566+0000","last_undegraded":"2024-03-11T21:38:08.351566+0000","last_fullsized":"2024-03-11T21:38:08.351566+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"98'9","last_scrub_stamp":"2024-03-11T21:36:03.366450+0000","last_deep_scrub":"98'9","last_deep_scrub_stamp":"2024-03-11T21:31:20.624923+0000","last_clean_scrub_stamp":"2024-03-11T21:36:03.366450+0000","objects_scrubbed":1,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.12","version":"0'0","reported_seq":566,"reported_epoch":312,"state":"active+clean","last_fresh":"2024-03-11T21:38:08.350913+0000","last_change":"2024-03-11T21:36:35.058369+0000","last_active":"2024-03-11T21:38:08.350913+0000","last_peered":"2024-03-11T21:38:08.350913+0000","last_clean":"2024-03-11T21:38:08.350913+0000","last_became_active":"2024-03-11T21:36:35.058033+0000","last_became_peered":"2024-03-11T21:36:35.058033+0000","last_unstale":"2024-03-11T21:38:08.350913+0000","last_undegraded":"2024-03-11T21:38:08.350913+0000","last_fullsized":"2024-03-11T21:38:08.350913+0000","mapping_epoch":277,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":278,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:35:27.158468+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:37.512018+0000","last_clean_scrub_stamp":"2024-03-11T21:35:27.158468+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.14","version":"260'20","reported_seq":551,"reported_epoch":312,"state":"active+clean","last_fresh":"2024-03-11T21:38:08.365202+0000","last_change":"2024-03-11T21:36:35.095382+0000","last_active":"2024-03-11T21:38:08.365202+0000","last_peered":"2024-03-11T21:38:08.365202+0000","last_clean":"2024-03-11T21:38:08.365202+0000","last_became_active":"2024-03-11T21:36:35.069638+0000","last_became_peered":"2024-03-11T21:36:35.069638+0000","last_unstale":"2024-03-11T21:38:08.365202+0000","last_undegraded":"2024-03-11T21:38:08.365202+0000","last_fullsized":"2024-03-11T21:38:08.365202+0000","mapping_epoch":277,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":278,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'10","last_scrub_stamp":"2024-03-11T21:35:28.189296+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:39.525554+0000","last_clean_scrub_stamp":"2024-03-11T21:35:28.189296+0000","objects_scrubbed":0,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":29,"num_read_kb":24,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.15","version":"0'0","reported_seq":190,"reported_epoch":312,"state":"active+clean","last_fresh":"2024-03-11T21:38:08.358446+0000","last_change":"2024-03-11T21:36:14.736240+0000","last_active":"2024-03-11T21:38:08.358446+0000","last_peered":"2024-03-11T21:38:08.358446+0000","last_clean":"2024-03-11T21:38:08.358446+0000","last_became_active":"2024-03-11T21:34:41.142760+0000","last_became_peered":"2024-03-11T21:34:41.142760+0000","last_unstale":"2024-03-11T21:38:08.358446+0000","last_undegraded":"2024-03-11T21:38:08.358446+0000","last_fullsized":"2024-03-11T21:38:08.358446+0000","mapping_epoch":226,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":227,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:36:14.735985+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:43.559452+0000","last_clean_scrub_stamp":"2024-03-11T21:36:14.735985+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":36,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":538,"reported_epoch":312,"state":"active+clean","last_fresh":"2024-03-11T21:38:08.352398+0000","last_change":"2024-03-11T21:36:35.060666+0000","last_active":"2024-03-11T21:38:08.352398+0000","last_peered":"2024-03-11T21:38:08.352398+0000","last_clean":"2024-03-11T21:38:08.352398+0000","last_became_active":"2024-03-11T21:36:35.060342+0000","last_became_peered":"2024-03-11T21:36:35.060342+0000","last_unstale":"2024-03-11T21:38:08.352398+0000","last_undegraded":"2024-03-11T21:38:08.352398+0000","last_fullsized":"2024-03-11T21:38:08.352398+0000","mapping_epoch":277,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":278,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:34:09.244852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:41.544550+0000","last_clean_scrub_stamp":"2024-03-11T21:34:09.244852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,1],"acting":[5,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.11","version":"0'0","reported_seq":546,"reported_epoch":312,"state":"active+clean","last_fresh":"2024-03-11T21:38:08.345343+0000","last_change":"2024-03-11T21:35:08.152853+0000","last_active":"2024-03-11T21:38:08.345343+0000","last_peered":"2024-03-11T21:38:08.345343+0000","last_clean":"2024-03-11T21:38:08.345343+0000","last_became_active":"2024-03-11T21:25:38.537415+0000","last_became_peered":"2024-03-11T21:25:38.537415+0000","last_unstale":"2024-03-11T21:38:08.345343+0000","last_undegraded":"2024-03-11T21:38:08.345343+0000","last_fullsized":"2024-03-11T21:38:08.345343+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:35:08.152740+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:37.512018+0000","last_clean_scrub_stamp":"2024-03-11T21:35:08.152740+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.17","version":"260'15","reported_seq":496,"reported_epoch":312,"state":"active+clean","last_fresh":"2024-03-11T21:38:08.367295+0000","last_change":"2024-03-11T21:36:35.099451+0000","last_active":"2024-03-11T21:38:08.367295+0000","last_peered":"2024-03-11T21:38:08.367295+0000","last_clean":"2024-03-11T21:38:08.367295+0000","last_became_active":"2024-03-11T21:36:35.071346+0000","last_became_peered":"2024-03-11T21:36:35.071346+0000","last_unstale":"2024-03-11T21:38:08.367295+0000","last_undegraded":"2024-03-11T21:38:08.367295+0000","last_fullsized":"2024-03-11T21:38:08.367295+0000","mapping_epoch":277,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":278,"parent":"0.0","parent_split_bits":0,"last_scrub":"97'9","last_scrub_stamp":"2024-03-11T21:32:08.495352+0000","last_deep_scrub":"97'9","last_deep_scrub_stamp":"2024-03-11T21:27:25.803132+0000","last_clean_scrub_stamp":"2024-03-11T21:32:08.495352+0000","objects_scrubbed":3,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":17,"num_read_kb":14,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,5],"acting":[3,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.14","version":"0'0","reported_seq":545,"reported_epoch":312,"state":"active+clean","last_fresh":"2024-03-11T21:38:08.366390+0000","last_change":"2024-03-11T21:35:27.606664+0000","last_active":"2024-03-11T21:38:08.366390+0000","last_peered":"2024-03-11T21:38:08.366390+0000","last_clean":"2024-03-11T21:38:08.366390+0000","last_became_active":"2024-03-11T21:34:41.136946+0000","last_became_peered":"2024-03-11T21:34:41.136946+0000","last_unstale":"2024-03-11T21:38:08.366390+0000","last_undegraded":"2024-03-11T21:38:08.366390+0000","last_fullsized":"2024-03-11T21:38:08.366390+0000","mapping_epoch":226,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":227,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:35:27.606551+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:43.559452+0000","last_clean_scrub_stamp":"2024-03-11T21:35:27.606551+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,7],"acting":[2,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":450,"reported_epoch":312,"state":"active+clean","last_fresh":"2024-03-11T21:38:08.367021+0000","last_change":"2024-03-11T21:36:35.076782+0000","last_active":"2024-03-11T21:38:08.367021+0000","last_peered":"2024-03-11T21:38:08.367021+0000","last_clean":"2024-03-11T21:38:08.367021+0000","last_became_active":"2024-03-11T21:36:35.076335+0000","last_became_peered":"2024-03-11T21:36:35.076335+0000","last_unstale":"2024-03-11T21:38:08.367021+0000","last_undegraded":"2024-03-11T21:38:08.367021+0000","last_fullsized":"2024-03-11T21:38:08.367021+0000","mapping_epoch":277,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":278,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:34:33.082577+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:41.544550+0000","last_clean_scrub_stamp":"2024-03-11T21:34:33.082577+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":35,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.10","version":"0'0","reported_seq":545,"reported_epoch":312,"state":"active+clean","last_fresh":"2024-03-11T21:38:08.365853+0000","last_change":"2024-03-11T21:35:37.694623+0000","last_active":"2024-03-11T21:38:08.365853+0000","last_peered":"2024-03-11T21:38:08.365853+0000","last_clean":"2024-03-11T21:38:08.365853+0000","last_became_active":"2024-03-11T21:25:38.533138+0000","last_became_peered":"2024-03-11T21:25:38.533138+0000","last_unstale":"2024-03-11T21:38:08.365853+0000","last_undegraded":"2024-03-11T21:38:08.365853+0000","last_fullsized":"2024-03-11T21:38:08.365853+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:35:37.694533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:37.512018+0000","last_clean_scrub_stamp":"2024-03-11T21:35:37.694533+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,0],"acting":[2,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.16","version":"260'15","reported_seq":617,"reported_epoch":312,"state":"active+clean","last_fresh":"2024-03-11T21:38:08.348423+0000","last_change":"2024-03-11T21:36:35.080158+0000","last_active":"2024-03-11T21:38:08.348423+0000","last_peered":"2024-03-11T21:38:08.348423+0000","last_clean":"2024-03-11T21:38:08.348423+0000","last_became_active":"2024-03-11T21:36:35.056862+0000","last_became_peered":"2024-03-11T21:36:35.056862+0000","last_unstale":"2024-03-11T21:38:08.348423+0000","last_undegraded":"2024-03-11T21:38:08.348423+0000","last_fullsized":"2024-03-11T21:38:08.348423+0000","mapping_epoch":277,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":278,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'9","last_scrub_stamp":"2024-03-11T21:35:12.741225+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:39.525554+0000","last_clean_scrub_stamp":"2024-03-11T21:35:12.741225+0000","objects_scrubbed":4,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":11,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.4","version":"312'9642","reported_seq":9759,"reported_epoch":312,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-03-11T21:38:12.616396+0000","last_change":"2024-03-11T21:36:36.516780+0000","last_active":"2024-03-11T21:38:12.616396+0000","last_peered":"2024-03-11T21:38:12.616396+0000","last_clean":"2024-03-11T21:35:44.669493+0000","last_became_active":"2024-03-11T21:36:36.506468+0000","last_became_peered":"2024-03-11T21:36:36.506468+0000","last_unstale":"2024-03-11T21:38:12.616396+0000","last_undegraded":"2024-03-11T21:36:36.069183+0000","last_fullsized":"2024-03-11T21:36:36.038484+0000","mapping_epoch":278,"log_start":"274'4600","ondisk_log_start":"274'4600","created":259,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:35:44.669493+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:35:44.669493+0000","last_clean_scrub_stamp":"2024-03-11T21:35:44.669493+0000","objects_scrubbed":0,"log_size":5042,"log_dups_size":0,"ondisk_log_size":5042,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":335151104,"num_objects":5114,"num_object_clones":0,"num_object_copies":15342,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4816,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5114,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9642,"num_write_kb":472192,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,3],"acting":[1,4],"avail_no_missing":["1","4"],"object_location_counts":[{"shards":"1,4","objects":298}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.b","version":"0'0","reported_seq":438,"reported_epoch":312,"state":"active+clean","last_fresh":"2024-03-11T21:38:08.367717+0000","last_change":"2024-03-11T21:36:35.074815+0000","last_active":"2024-03-11T21:38:08.367717+0000","last_peered":"2024-03-11T21:38:08.367717+0000","last_clean":"2024-03-11T21:38:08.367717+0000","last_became_active":"2024-03-11T21:36:35.072147+0000","last_became_peered":"2024-03-11T21:36:35.072147+0000","last_unstale":"2024-03-11T21:38:08.367717+0000","last_undegraded":"2024-03-11T21:38:08.367717+0000","last_fullsized":"2024-03-11T21:38:08.367717+0000","mapping_epoch":277,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":278,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:32:22.598150+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:43.559452+0000","last_clean_scrub_stamp":"2024-03-11T21:32:22.598150+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":57,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.f","version":"67'2","reported_seq":562,"reported_epoch":314,"state":"active+clean","last_fresh":"2024-03-11T21:38:16.476522+0000","last_change":"2024-03-11T21:35:36.613667+0000","last_active":"2024-03-11T21:38:16.476522+0000","last_peered":"2024-03-11T21:38:16.476522+0000","last_clean":"2024-03-11T21:38:16.476522+0000","last_became_active":"2024-03-11T21:25:38.537711+0000","last_became_peered":"2024-03-11T21:25:38.537711+0000","last_unstale":"2024-03-11T21:38:16.476522+0000","last_undegraded":"2024-03-11T21:38:16.476522+0000","last_fullsized":"2024-03-11T21:38:16.476522+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'2","last_scrub_stamp":"2024-03-11T21:35:36.613516+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:37.512018+0000","last_clean_scrub_stamp":"2024-03-11T21:35:36.613516+0000","objects_scrubbed":2,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":92,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":10,"num_read_kb":10,"num_write":4,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,7],"acting":[4,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.9","version":"260'23","reported_seq":593,"reported_epoch":314,"state":"active+clean","last_fresh":"2024-03-11T21:38:16.476007+0000","last_change":"2024-03-11T21:36:35.089034+0000","last_active":"2024-03-11T21:38:16.476007+0000","last_peered":"2024-03-11T21:38:16.476007+0000","last_clean":"2024-03-11T21:38:16.476007+0000","last_became_active":"2024-03-11T21:36:35.059930+0000","last_became_peered":"2024-03-11T21:36:35.059930+0000","last_unstale":"2024-03-11T21:38:16.476007+0000","last_undegraded":"2024-03-11T21:38:16.476007+0000","last_fullsized":"2024-03-11T21:38:16.476007+0000","mapping_epoch":277,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":278,"parent":"0.0","parent_split_bits":0,"last_scrub":"146'17","last_scrub_stamp":"2024-03-11T21:34:26.765075+0000","last_deep_scrub":"146'17","last_deep_scrub_stamp":"2024-03-11T21:34:26.765075+0000","last_clean_scrub_stamp":"2024-03-11T21:34:26.765075+0000","objects_scrubbed":5,"log_size":23,"log_dups_size":0,"ondisk_log_size":23,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":34,"num_read_kb":25,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,3],"acting":[4,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":537,"reported_epoch":312,"state":"active+clean","last_fresh":"2024-03-11T21:38:08.365944+0000","last_change":"2024-03-11T21:35:16.643421+0000","last_active":"2024-03-11T21:38:08.365944+0000","last_peered":"2024-03-11T21:38:08.365944+0000","last_clean":"2024-03-11T21:38:08.365944+0000","last_became_active":"2024-03-11T21:25:42.573987+0000","last_became_peered":"2024-03-11T21:25:42.573987+0000","last_unstale":"2024-03-11T21:38:08.365944+0000","last_undegraded":"2024-03-11T21:38:08.365944+0000","last_fullsized":"2024-03-11T21:38:08.365944+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:35:16.643311+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:41.544550+0000","last_clean_scrub_stamp":"2024-03-11T21:35:16.643311+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"9.f","version":"312'9428","reported_seq":9547,"reported_epoch":312,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-03-11T21:38:12.424693+0000","last_change":"2024-03-11T21:36:36.519822+0000","last_active":"2024-03-11T21:38:12.424693+0000","last_peered":"2024-03-11T21:38:12.424693+0000","last_clean":"2024-03-11T21:35:44.669493+0000","last_became_active":"2024-03-11T21:36:36.508836+0000","last_became_peered":"2024-03-11T21:36:36.508836+0000","last_unstale":"2024-03-11T21:38:12.424693+0000","last_undegraded":"2024-03-11T21:36:36.176770+0000","last_fullsized":"2024-03-11T21:36:36.038651+0000","mapping_epoch":278,"log_start":"274'4200","ondisk_log_start":"274'4200","created":259,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:35:44.669493+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:35:44.669493+0000","last_clean_scrub_stamp":"2024-03-11T21:35:44.669493+0000","objects_scrubbed":0,"log_size":5228,"log_dups_size":0,"ondisk_log_size":5228,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":324009984,"num_objects":4944,"num_object_clones":0,"num_object_copies":14832,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4738,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4944,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9428,"num_write_kb":459904,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":65536,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3],"acting":[5,1],"avail_no_missing":["5","1"],"object_location_counts":[{"shards":"1,5","objects":206}],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.0","version":"0'0","reported_seq":536,"reported_epoch":312,"state":"active+clean","last_fresh":"2024-03-11T21:38:08.346428+0000","last_change":"2024-03-11T21:36:35.058390+0000","last_active":"2024-03-11T21:38:08.346428+0000","last_peered":"2024-03-11T21:38:08.346428+0000","last_clean":"2024-03-11T21:38:08.346428+0000","last_became_active":"2024-03-11T21:36:35.058054+0000","last_became_peered":"2024-03-11T21:36:35.058054+0000","last_unstale":"2024-03-11T21:38:08.346428+0000","last_undegraded":"2024-03-11T21:38:08.346428+0000","last_fullsized":"2024-03-11T21:38:08.346428+0000","mapping_epoch":277,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":278,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:35:04.626668+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:29:39.498784+0000","last_clean_scrub_stamp":"2024-03-11T21:35:04.626668+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":39,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":559,"reported_epoch":312,"state":"active+clean","last_fresh":"2024-03-11T21:38:08.382292+0000","last_change":"2024-03-11T21:35:33.608063+0000","last_active":"2024-03-11T21:38:08.382292+0000","last_peered":"2024-03-11T21:38:08.382292+0000","last_clean":"2024-03-11T21:38:08.382292+0000","last_became_active":"2024-03-11T21:34:41.150913+0000","last_became_peered":"2024-03-11T21:34:41.150913+0000","last_unstale":"2024-03-11T21:38:08.382292+0000","last_undegraded":"2024-03-11T21:38:08.382292+0000","last_fullsized":"2024-03-11T21:38:08.382292+0000","mapping_epoch":226,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":227,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:35:33.607969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:37.512018+0000","last_clean_scrub_stamp":"2024-03-11T21:35:33.607969+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,7],"acting":[1,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.2","version":"260'25","reported_seq":578,"reported_epoch":312,"state":"active+clean","last_fresh":"2024-03-11T21:38:08.360203+0000","last_change":"2024-03-11T21:35:17.666749+0000","last_active":"2024-03-11T21:38:08.360203+0000","last_peered":"2024-03-11T21:38:08.360203+0000","last_clean":"2024-03-11T21:38:08.360203+0000","last_became_active":"2024-03-11T21:25:40.548833+0000","last_became_peered":"2024-03-11T21:25:40.548833+0000","last_unstale":"2024-03-11T21:38:08.360203+0000","last_undegraded":"2024-03-11T21:38:08.360203+0000","last_fullsized":"2024-03-11T21:38:08.360203+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"94'15","last_scrub_stamp":"2024-03-11T21:35:17.666498+0000","last_deep_scrub":"94'15","last_deep_scrub_stamp":"2024-03-11T21:26:49.993060+0000","last_clean_scrub_stamp":"2024-03-11T21:35:17.666498+0000","objects_scrubbed":5,"log_size":25,"log_dups_size":0,"ondisk_log_size":25,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":20,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":25,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.3","version":"74'8","reported_seq":698,"reported_epoch":312,"state":"active+clean","last_fresh":"2024-03-11T21:38:08.755511+0000","last_change":"2024-03-11T21:35:31.695369+0000","last_active":"2024-03-11T21:38:08.755511+0000","last_peered":"2024-03-11T21:38:08.755511+0000","last_clean":"2024-03-11T21:38:08.755511+0000","last_became_active":"2024-03-11T21:25:42.578161+0000","last_became_peered":"2024-03-11T21:25:42.578161+0000","last_unstale":"2024-03-11T21:38:08.755511+0000","last_undegraded":"2024-03-11T21:38:08.755511+0000","last_fullsized":"2024-03-11T21:38:08.755511+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'8","last_scrub_stamp":"2024-03-11T21:35:31.695263+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:41.544550+0000","last_clean_scrub_stamp":"2024-03-11T21:35:31.695263+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,5],"acting":[0,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.9","version":"312'9690","reported_seq":9807,"reported_epoch":312,"state":"active+clean","last_fresh":"2024-03-11T21:38:13.438774+0000","last_change":"2024-03-11T21:35:45.696767+0000","last_active":"2024-03-11T21:38:13.438774+0000","last_peered":"2024-03-11T21:38:13.438774+0000","last_clean":"2024-03-11T21:38:13.438774+0000","last_became_active":"2024-03-11T21:35:45.696379+0000","last_became_peered":"2024-03-11T21:35:45.696379+0000","last_unstale":"2024-03-11T21:38:13.438774+0000","last_undegraded":"2024-03-11T21:38:13.438774+0000","last_fullsized":"2024-03-11T21:38:13.438774+0000","mapping_epoch":259,"log_start":"274'4100","ondisk_log_start":"274'4100","created":259,"last_epoch_clean":260,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:35:44.669493+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:35:44.669493+0000","last_clean_scrub_stamp":"2024-03-11T21:35:44.669493+0000","objects_scrubbed":0,"log_size":5590,"log_dups_size":0,"ondisk_log_size":5590,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":334233600,"num_objects":5100,"num_object_clones":0,"num_object_copies":15300,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5100,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9690,"num_write_kb":473280,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,5],"acting":[0,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":476,"reported_epoch":312,"state":"active+clean","last_fresh":"2024-03-11T21:38:08.357401+0000","last_change":"2024-03-11T21:36:35.068977+0000","last_active":"2024-03-11T21:38:08.357401+0000","last_peered":"2024-03-11T21:38:08.357401+0000","last_clean":"2024-03-11T21:38:08.357401+0000","last_became_active":"2024-03-11T21:36:35.067288+0000","last_became_peered":"2024-03-11T21:36:35.067288+0000","last_unstale":"2024-03-11T21:38:08.357401+0000","last_undegraded":"2024-03-11T21:38:08.357401+0000","last_fullsized":"2024-03-11T21:38:08.357401+0000","mapping_epoch":277,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":278,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:35:09.985377+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:28:30.563712+0000","last_clean_scrub_stamp":"2024-03-11T21:35:09.985377+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,7],"acting":[3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":554,"reported_epoch":312,"state":"active+clean","last_fresh":"2024-03-11T21:38:08.351022+0000","last_change":"2024-03-11T21:35:40.116979+0000","last_active":"2024-03-11T21:38:08.351022+0000","last_peered":"2024-03-11T21:38:08.351022+0000","last_clean":"2024-03-11T21:38:08.351022+0000","last_became_active":"2024-03-11T21:25:38.538540+0000","last_became_peered":"2024-03-11T21:25:38.538540+0000","last_unstale":"2024-03-11T21:38:08.351022+0000","last_undegraded":"2024-03-11T21:38:08.351022+0000","last_fullsized":"2024-03-11T21:38:08.351022+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:35:40.116877+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:37.512018+0000","last_clean_scrub_stamp":"2024-03-11T21:35:40.116877+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.4","version":"280'42","reported_seq":777,"reported_epoch":312,"state":"active+clean","last_fresh":"2024-03-11T21:38:08.751246+0000","last_change":"2024-03-11T21:36:35.095935+0000","last_active":"2024-03-11T21:38:08.751246+0000","last_peered":"2024-03-11T21:38:08.751246+0000","last_clean":"2024-03-11T21:38:08.751246+0000","last_became_active":"2024-03-11T21:36:35.073430+0000","last_became_peered":"2024-03-11T21:36:35.073430+0000","last_unstale":"2024-03-11T21:38:08.751246+0000","last_undegraded":"2024-03-11T21:38:08.751246+0000","last_fullsized":"2024-03-11T21:38:08.751246+0000","mapping_epoch":277,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":278,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'28","last_scrub_stamp":"2024-03-11T21:34:19.513252+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:39.525554+0000","last_clean_scrub_stamp":"2024-03-11T21:34:19.513252+0000","objects_scrubbed":10,"log_size":42,"log_dups_size":0,"ondisk_log_size":42,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":358,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":66,"num_read_kb":51,"num_write":38,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3],"acting":[1,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":538,"reported_epoch":312,"state":"active+clean","last_fresh":"2024-03-11T21:38:08.348456+0000","last_change":"2024-03-11T21:35:32.651731+0000","last_active":"2024-03-11T21:38:08.348456+0000","last_peered":"2024-03-11T21:38:08.348456+0000","last_clean":"2024-03-11T21:38:08.348456+0000","last_became_active":"2024-03-11T21:25:42.572673+0000","last_became_peered":"2024-03-11T21:25:42.572673+0000","last_unstale":"2024-03-11T21:38:08.348456+0000","last_undegraded":"2024-03-11T21:38:08.348456+0000","last_fullsized":"2024-03-11T21:38:08.348456+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:35:32.651652+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:41.544550+0000","last_clean_scrub_stamp":"2024-03-11T21:35:32.651652+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.a","version":"312'9601","reported_seq":9717,"reported_epoch":312,"state":"active+clean","last_fresh":"2024-03-11T21:38:12.416342+0000","last_change":"2024-03-11T21:35:45.698558+0000","last_active":"2024-03-11T21:38:12.416342+0000","last_peered":"2024-03-11T21:38:12.416342+0000","last_clean":"2024-03-11T21:38:12.416342+0000","last_became_active":"2024-03-11T21:35:45.698238+0000","last_became_peered":"2024-03-11T21:35:45.698238+0000","last_unstale":"2024-03-11T21:38:12.416342+0000","last_undegraded":"2024-03-11T21:38:12.416342+0000","last_fullsized":"2024-03-11T21:38:12.416342+0000","mapping_epoch":259,"log_start":"274'4400","ondisk_log_start":"274'4400","created":259,"last_epoch_clean":260,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:35:44.669493+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:35:44.669493+0000","last_clean_scrub_stamp":"2024-03-11T21:35:44.669493+0000","objects_scrubbed":0,"log_size":5201,"log_dups_size":0,"ondisk_log_size":5201,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":339935232,"num_objects":5187,"num_object_clones":0,"num_object_copies":15561,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5187,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9601,"num_write_kb":473216,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,7],"acting":[5,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.5","version":"0'0","reported_seq":199,"reported_epoch":312,"state":"active+clean","last_fresh":"2024-03-11T21:38:08.358286+0000","last_change":"2024-03-11T21:36:35.069989+0000","last_active":"2024-03-11T21:38:08.358286+0000","last_peered":"2024-03-11T21:38:08.358286+0000","last_clean":"2024-03-11T21:38:08.358286+0000","last_became_active":"2024-03-11T21:36:35.069358+0000","last_became_peered":"2024-03-11T21:36:35.069358+0000","last_unstale":"2024-03-11T21:38:08.358286+0000","last_undegraded":"2024-03-11T21:38:08.358286+0000","last_fullsized":"2024-03-11T21:38:08.358286+0000","mapping_epoch":277,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":278,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:35:19.841632+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:43.559452+0000","last_clean_scrub_stamp":"2024-03-11T21:35:19.841632+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,3],"acting":[7,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.0","version":"164'6","reported_seq":1305,"reported_epoch":312,"state":"active+clean","last_fresh":"2024-03-11T21:38:11.532420+0000","last_change":"2024-03-11T21:35:38.518243+0000","last_active":"2024-03-11T21:38:11.532420+0000","last_peered":"2024-03-11T21:38:11.532420+0000","last_clean":"2024-03-11T21:38:11.532420+0000","last_became_active":"2024-03-11T21:25:38.536890+0000","last_became_peered":"2024-03-11T21:25:38.536890+0000","last_unstale":"2024-03-11T21:38:11.532420+0000","last_undegraded":"2024-03-11T21:38:11.532420+0000","last_fullsized":"2024-03-11T21:38:11.532420+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"164'6","last_scrub_stamp":"2024-03-11T21:35:38.518097+0000","last_deep_scrub":"164'6","last_deep_scrub_stamp":"2024-03-11T21:32:17.597666+0000","last_clean_scrub_stamp":"2024-03-11T21:35:38.518097+0000","objects_scrubbed":1,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0.000753559,"stat_sum":{"num_bytes":389,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":747,"num_read_kb":742,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,6],"acting":[1,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":555,"reported_epoch":312,"state":"active+clean","last_fresh":"2024-03-11T21:38:08.365699+0000","last_change":"2024-03-11T21:36:35.057287+0000","last_active":"2024-03-11T21:38:08.365699+0000","last_peered":"2024-03-11T21:38:08.365699+0000","last_clean":"2024-03-11T21:38:08.365699+0000","last_became_active":"2024-03-11T21:36:35.056966+0000","last_became_peered":"2024-03-11T21:36:35.056966+0000","last_unstale":"2024-03-11T21:38:08.365699+0000","last_undegraded":"2024-03-11T21:38:08.365699+0000","last_fullsized":"2024-03-11T21:38:08.365699+0000","mapping_epoch":277,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":278,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:34:18.424010+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:26:48.997454+0000","last_clean_scrub_stamp":"2024-03-11T21:34:18.424010+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.7","version":"260'22","reported_seq":583,"reported_epoch":312,"state":"active+clean","last_fresh":"2024-03-11T21:38:08.354429+0000","last_change":"2024-03-11T21:35:37.508679+0000","last_active":"2024-03-11T21:38:08.354429+0000","last_peered":"2024-03-11T21:38:08.354429+0000","last_clean":"2024-03-11T21:38:08.354429+0000","last_became_active":"2024-03-11T21:25:40.541769+0000","last_became_peered":"2024-03-11T21:25:40.541769+0000","last_unstale":"2024-03-11T21:38:08.354429+0000","last_undegraded":"2024-03-11T21:38:08.354429+0000","last_fullsized":"2024-03-11T21:38:08.354429+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"97'18","last_scrub_stamp":"2024-03-11T21:35:37.508567+0000","last_deep_scrub":"97'18","last_deep_scrub_stamp":"2024-03-11T21:27:19.239404+0000","last_clean_scrub_stamp":"2024-03-11T21:35:37.508567+0000","objects_scrubbed":5,"log_size":22,"log_dups_size":0,"ondisk_log_size":22,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":36,"num_read_kb":25,"num_write":20,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":549,"reported_epoch":312,"state":"active+clean","last_fresh":"2024-03-11T21:38:08.366691+0000","last_change":"2024-03-11T21:35:22.632822+0000","last_active":"2024-03-11T21:38:08.366691+0000","last_peered":"2024-03-11T21:38:08.366691+0000","last_clean":"2024-03-11T21:38:08.366691+0000","last_became_active":"2024-03-11T21:34:41.143660+0000","last_became_peered":"2024-03-11T21:34:41.143660+0000","last_unstale":"2024-03-11T21:38:08.366691+0000","last_undegraded":"2024-03-11T21:38:08.366691+0000","last_fullsized":"2024-03-11T21:38:08.366691+0000","mapping_epoch":226,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":227,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:35:22.632707+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:41.544550+0000","last_clean_scrub_stamp":"2024-03-11T21:35:22.632707+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":18,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7],"acting":[2,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"9.8","version":"312'9704","reported_seq":9820,"reported_epoch":312,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-03-11T21:38:13.487091+0000","last_change":"2024-03-11T21:36:36.520071+0000","last_active":"2024-03-11T21:38:13.487091+0000","last_peered":"2024-03-11T21:38:13.487091+0000","last_clean":"2024-03-11T21:35:44.669493+0000","last_became_active":"2024-03-11T21:36:36.507652+0000","last_became_peered":"2024-03-11T21:36:36.507652+0000","last_unstale":"2024-03-11T21:38:13.487091+0000","last_undegraded":"2024-03-11T21:36:36.064628+0000","last_fullsized":"2024-03-11T21:36:36.036628+0000","mapping_epoch":278,"log_start":"274'4100","ondisk_log_start":"274'4100","created":259,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:35:44.669493+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:35:44.669493+0000","last_clean_scrub_stamp":"2024-03-11T21:35:44.669493+0000","objects_scrubbed":0,"log_size":5604,"log_dups_size":0,"ondisk_log_size":5604,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330694656,"num_objects":5046,"num_object_clones":0,"num_object_copies":15138,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4846,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5046,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9704,"num_write_kb":472000,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[0,5],"avail_no_missing":["0","5"],"object_location_counts":[{"shards":"0,5","objects":200}],"blocked_by":[],"up_primary":3,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.7","version":"124'2","reported_seq":536,"reported_epoch":312,"state":"active+clean","last_fresh":"2024-03-11T21:38:08.367332+0000","last_change":"2024-03-11T21:36:35.059190+0000","last_active":"2024-03-11T21:38:08.367332+0000","last_peered":"2024-03-11T21:38:08.367332+0000","last_clean":"2024-03-11T21:38:08.367332+0000","last_became_active":"2024-03-11T21:36:35.058786+0000","last_became_peered":"2024-03-11T21:36:35.058786+0000","last_unstale":"2024-03-11T21:38:08.367332+0000","last_undegraded":"2024-03-11T21:38:08.367332+0000","last_fullsized":"2024-03-11T21:38:08.367332+0000","mapping_epoch":277,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":278,"parent":"0.0","parent_split_bits":0,"last_scrub":"124'2","last_scrub_stamp":"2024-03-11T21:35:16.414595+0000","last_deep_scrub":"124'2","last_deep_scrub_stamp":"2024-03-11T21:29:34.956908+0000","last_clean_scrub_stamp":"2024-03-11T21:35:16.414595+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":42,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,4],"acting":[5,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2","version":"71'2","reported_seq":497,"reported_epoch":312,"state":"active+clean","last_fresh":"2024-03-11T21:38:08.374463+0000","last_change":"2024-03-11T21:36:35.071218+0000","last_active":"2024-03-11T21:38:08.374463+0000","last_peered":"2024-03-11T21:38:08.374463+0000","last_clean":"2024-03-11T21:38:08.374463+0000","last_became_active":"2024-03-11T21:36:35.070783+0000","last_became_peered":"2024-03-11T21:36:35.070783+0000","last_unstale":"2024-03-11T21:38:08.374463+0000","last_undegraded":"2024-03-11T21:38:08.374463+0000","last_fullsized":"2024-03-11T21:38:08.374463+0000","mapping_epoch":277,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":278,"parent":"0.0","parent_split_bits":0,"last_scrub":"71'2","last_scrub_stamp":"2024-03-11T21:32:21.543053+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:37.512018+0000","last_clean_scrub_stamp":"2024-03-11T21:32:21.543053+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014206449999999999,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":564,"reported_epoch":312,"state":"active+clean","last_fresh":"2024-03-11T21:38:08.350559+0000","last_change":"2024-03-11T21:35:08.391825+0000","last_active":"2024-03-11T21:38:08.350559+0000","last_peered":"2024-03-11T21:38:08.350559+0000","last_clean":"2024-03-11T21:38:08.350559+0000","last_became_active":"2024-03-11T21:34:41.146672+0000","last_became_peered":"2024-03-11T21:34:41.146672+0000","last_unstale":"2024-03-11T21:38:08.350559+0000","last_undegraded":"2024-03-11T21:38:08.350559+0000","last_fullsized":"2024-03-11T21:38:08.350559+0000","mapping_epoch":226,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":227,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:35:08.391713+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:37.512018+0000","last_clean_scrub_stamp":"2024-03-11T21:35:08.391713+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"87'128","reported_seq":661,"reported_epoch":312,"state":"active+clean","last_fresh":"2024-03-11T21:38:08.350645+0000","last_change":"2024-03-11T21:35:21.798272+0000","last_active":"2024-03-11T21:38:08.350645+0000","last_peered":"2024-03-11T21:38:08.350645+0000","last_clean":"2024-03-11T21:38:08.350645+0000","last_became_active":"2024-03-11T21:34:44.164569+0000","last_became_peered":"2024-03-11T21:34:44.164569+0000","last_unstale":"2024-03-11T21:38:08.350645+0000","last_undegraded":"2024-03-11T21:38:08.350645+0000","last_fullsized":"2024-03-11T21:38:08.350645+0000","mapping_epoch":229,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":230,"parent":"0.0","parent_split_bits":0,"last_scrub":"87'128","last_scrub_stamp":"2024-03-11T21:35:21.798144+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:24.261974+0000","last_clean_scrub_stamp":"2024-03-11T21:35:21.798144+0000","objects_scrubbed":2,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":198,"num_read_kb":294,"num_write":179,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":2951840,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.5","version":"260'28","reported_seq":742,"reported_epoch":312,"state":"active+clean","last_fresh":"2024-03-11T21:38:08.751847+0000","last_change":"2024-03-11T21:36:14.026933+0000","last_active":"2024-03-11T21:38:08.751847+0000","last_peered":"2024-03-11T21:38:08.751847+0000","last_clean":"2024-03-11T21:38:08.751847+0000","last_became_active":"2024-03-11T21:25:40.761457+0000","last_became_peered":"2024-03-11T21:25:40.761457+0000","last_unstale":"2024-03-11T21:38:08.751847+0000","last_undegraded":"2024-03-11T21:38:08.751847+0000","last_fullsized":"2024-03-11T21:38:08.751847+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"260'28","last_scrub_stamp":"2024-03-11T21:36:14.026775+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:39.525554+0000","last_clean_scrub_stamp":"2024-03-11T21:36:14.026775+0000","objects_scrubbed":8,"log_size":28,"log_dups_size":0,"ondisk_log_size":28,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":154,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":43,"num_read_kb":33,"num_write":25,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":190,"reported_epoch":312,"state":"active+clean","last_fresh":"2024-03-11T21:38:08.355203+0000","last_change":"2024-03-11T21:35:24.841170+0000","last_active":"2024-03-11T21:38:08.355203+0000","last_peered":"2024-03-11T21:38:08.355203+0000","last_clean":"2024-03-11T21:38:08.355203+0000","last_became_active":"2024-03-11T21:34:41.139782+0000","last_became_peered":"2024-03-11T21:34:41.139782+0000","last_unstale":"2024-03-11T21:38:08.355203+0000","last_undegraded":"2024-03-11T21:38:08.355203+0000","last_fullsized":"2024-03-11T21:38:08.355203+0000","mapping_epoch":226,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":227,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:35:24.841041+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:32:21.266672+0000","last_clean_scrub_stamp":"2024-03-11T21:35:24.841041+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,5],"acting":[7,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.b","version":"312'9496","reported_seq":9613,"reported_epoch":312,"state":"active+clean","last_fresh":"2024-03-11T21:38:13.522870+0000","last_change":"2024-03-11T21:35:45.697213+0000","last_active":"2024-03-11T21:38:13.522870+0000","last_peered":"2024-03-11T21:38:13.522870+0000","last_clean":"2024-03-11T21:38:13.522870+0000","last_became_active":"2024-03-11T21:35:45.696869+0000","last_became_peered":"2024-03-11T21:35:45.696869+0000","last_unstale":"2024-03-11T21:38:13.522870+0000","last_undegraded":"2024-03-11T21:38:13.522870+0000","last_fullsized":"2024-03-11T21:38:13.522870+0000","mapping_epoch":259,"log_start":"274'3900","ondisk_log_start":"274'3900","created":259,"last_epoch_clean":260,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:35:44.669493+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:35:44.669493+0000","last_clean_scrub_stamp":"2024-03-11T21:35:44.669493+0000","objects_scrubbed":0,"log_size":5596,"log_dups_size":0,"ondisk_log_size":5596,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":324403200,"num_objects":4950,"num_object_clones":0,"num_object_copies":14850,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4950,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9496,"num_write_kb":462272,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,5],"acting":[0,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.4","version":"0'0","reported_seq":535,"reported_epoch":312,"state":"active+clean","last_fresh":"2024-03-11T21:38:08.382625+0000","last_change":"2024-03-11T21:36:35.073305+0000","last_active":"2024-03-11T21:38:08.382625+0000","last_peered":"2024-03-11T21:38:08.382625+0000","last_clean":"2024-03-11T21:38:08.382625+0000","last_became_active":"2024-03-11T21:36:35.072824+0000","last_became_peered":"2024-03-11T21:36:35.072824+0000","last_unstale":"2024-03-11T21:38:08.382625+0000","last_undegraded":"2024-03-11T21:38:08.382625+0000","last_fullsized":"2024-03-11T21:38:08.382625+0000","mapping_epoch":277,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":278,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:34:23.531064+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:43.559452+0000","last_clean_scrub_stamp":"2024-03-11T21:34:23.531064+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":563,"reported_epoch":312,"state":"active+clean","last_fresh":"2024-03-11T21:38:08.349996+0000","last_change":"2024-03-11T21:36:35.070626+0000","last_active":"2024-03-11T21:38:08.349996+0000","last_peered":"2024-03-11T21:38:08.349996+0000","last_clean":"2024-03-11T21:38:08.349996+0000","last_became_active":"2024-03-11T21:36:35.069452+0000","last_became_peered":"2024-03-11T21:36:35.069452+0000","last_unstale":"2024-03-11T21:38:08.349996+0000","last_undegraded":"2024-03-11T21:38:08.349996+0000","last_fullsized":"2024-03-11T21:38:08.349996+0000","mapping_epoch":277,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":278,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:35:13.741947+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:37.512018+0000","last_clean_scrub_stamp":"2024-03-11T21:35:13.741947+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0.00089342499999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":182,"reported_epoch":312,"state":"active+clean","last_fresh":"2024-03-11T21:38:08.350554+0000","last_change":"2024-03-11T21:34:41.150606+0000","last_active":"2024-03-11T21:38:08.350554+0000","last_peered":"2024-03-11T21:38:08.350554+0000","last_clean":"2024-03-11T21:38:08.350554+0000","last_became_active":"2024-03-11T21:34:41.149219+0000","last_became_peered":"2024-03-11T21:34:41.149219+0000","last_unstale":"2024-03-11T21:38:08.350554+0000","last_undegraded":"2024-03-11T21:38:08.350554+0000","last_fullsized":"2024-03-11T21:38:08.350554+0000","mapping_epoch":226,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":227,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:34:31.628014+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:37.512018+0000","last_clean_scrub_stamp":"2024-03-11T21:34:31.628014+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,0],"acting":[7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.6","version":"260'24","reported_seq":598,"reported_epoch":312,"state":"active+clean","last_fresh":"2024-03-11T21:38:08.348332+0000","last_change":"2024-03-11T21:35:19.753433+0000","last_active":"2024-03-11T21:38:08.348332+0000","last_peered":"2024-03-11T21:38:08.348332+0000","last_clean":"2024-03-11T21:38:08.348332+0000","last_became_active":"2024-03-11T21:32:22.796486+0000","last_became_peered":"2024-03-11T21:32:22.796486+0000","last_unstale":"2024-03-11T21:38:08.348332+0000","last_undegraded":"2024-03-11T21:38:08.348332+0000","last_fullsized":"2024-03-11T21:38:08.348332+0000","mapping_epoch":166,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":167,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'12","last_scrub_stamp":"2024-03-11T21:35:19.753224+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:39.525554+0000","last_clean_scrub_stamp":"2024-03-11T21:35:19.753224+0000","objects_scrubbed":6,"log_size":24,"log_dups_size":0,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":36,"num_read_kb":30,"num_write":24,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":537,"reported_epoch":312,"state":"active+clean","last_fresh":"2024-03-11T21:38:08.367305+0000","last_change":"2024-03-11T21:36:07.451482+0000","last_active":"2024-03-11T21:38:08.367305+0000","last_peered":"2024-03-11T21:38:08.367305+0000","last_clean":"2024-03-11T21:38:08.367305+0000","last_became_active":"2024-03-11T21:25:42.572324+0000","last_became_peered":"2024-03-11T21:25:42.572324+0000","last_unstale":"2024-03-11T21:38:08.367305+0000","last_undegraded":"2024-03-11T21:38:08.367305+0000","last_fullsized":"2024-03-11T21:38:08.367305+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:36:07.451102+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:41.544550+0000","last_clean_scrub_stamp":"2024-03-11T21:36:07.451102+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":54,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.e","version":"312'9261","reported_seq":9378,"reported_epoch":312,"state":"active+clean","last_fresh":"2024-03-11T21:38:12.239828+0000","last_change":"2024-03-11T21:35:45.697331+0000","last_active":"2024-03-11T21:38:12.239828+0000","last_peered":"2024-03-11T21:38:12.239828+0000","last_clean":"2024-03-11T21:38:12.239828+0000","last_became_active":"2024-03-11T21:35:45.696948+0000","last_became_peered":"2024-03-11T21:35:45.696948+0000","last_unstale":"2024-03-11T21:38:12.239828+0000","last_undegraded":"2024-03-11T21:38:12.239828+0000","last_fullsized":"2024-03-11T21:38:12.239828+0000","mapping_epoch":259,"log_start":"276'4600","ondisk_log_start":"276'4600","created":259,"last_epoch_clean":260,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:35:44.669493+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:35:44.669493+0000","last_clean_scrub_stamp":"2024-03-11T21:35:44.669493+0000","objects_scrubbed":0,"log_size":4661,"log_dups_size":0,"ondisk_log_size":4661,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328925184,"num_objects":5019,"num_object_clones":0,"num_object_copies":15057,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5019,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9261,"num_write_kb":456960,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":542,"reported_epoch":312,"state":"active+clean","last_fresh":"2024-03-11T21:38:08.351618+0000","last_change":"2024-03-11T21:36:18.529506+0000","last_active":"2024-03-11T21:38:08.351618+0000","last_peered":"2024-03-11T21:38:08.351618+0000","last_clean":"2024-03-11T21:38:08.351618+0000","last_became_active":"2024-03-11T21:25:45.301416+0000","last_became_peered":"2024-03-11T21:25:45.301416+0000","last_unstale":"2024-03-11T21:38:08.351618+0000","last_undegraded":"2024-03-11T21:38:08.351618+0000","last_fullsized":"2024-03-11T21:38:08.351618+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:36:18.529370+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:43.559452+0000","last_clean_scrub_stamp":"2024-03-11T21:36:18.529370+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":54,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":190,"reported_epoch":312,"state":"active+clean","last_fresh":"2024-03-11T21:38:08.358372+0000","last_change":"2024-03-11T21:35:33.740615+0000","last_active":"2024-03-11T21:38:08.358372+0000","last_peered":"2024-03-11T21:38:08.358372+0000","last_clean":"2024-03-11T21:38:08.358372+0000","last_became_active":"2024-03-11T21:34:41.152721+0000","last_became_peered":"2024-03-11T21:34:41.152721+0000","last_unstale":"2024-03-11T21:38:08.358372+0000","last_undegraded":"2024-03-11T21:38:08.358372+0000","last_fullsized":"2024-03-11T21:38:08.358372+0000","mapping_epoch":226,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":227,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:35:33.740538+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:26:57.333669+0000","last_clean_scrub_stamp":"2024-03-11T21:35:33.740538+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.3","version":"260'37","reported_seq":650,"reported_epoch":312,"state":"active+clean","last_fresh":"2024-03-11T21:38:08.349469+0000","last_change":"2024-03-11T21:35:11.697314+0000","last_active":"2024-03-11T21:38:08.349469+0000","last_peered":"2024-03-11T21:38:08.349469+0000","last_clean":"2024-03-11T21:38:08.349469+0000","last_became_active":"2024-03-11T21:34:41.145607+0000","last_became_peered":"2024-03-11T21:34:41.145607+0000","last_unstale":"2024-03-11T21:38:08.349469+0000","last_undegraded":"2024-03-11T21:38:08.349469+0000","last_fullsized":"2024-03-11T21:38:08.349469+0000","mapping_epoch":226,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":227,"parent":"0.0","parent_split_bits":0,"last_scrub":"165'27","last_scrub_stamp":"2024-03-11T21:35:11.697140+0000","last_deep_scrub":"165'27","last_deep_scrub_stamp":"2024-03-11T21:32:21.576732+0000","last_clean_scrub_stamp":"2024-03-11T21:35:11.697140+0000","objects_scrubbed":8,"log_size":37,"log_dups_size":0,"ondisk_log_size":37,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":54,"num_read_kb":40,"num_write":32,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":660,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":547,"reported_epoch":312,"state":"active+clean","last_fresh":"2024-03-11T21:38:08.346669+0000","last_change":"2024-03-11T21:35:12.091317+0000","last_active":"2024-03-11T21:38:08.346669+0000","last_peered":"2024-03-11T21:38:08.346669+0000","last_clean":"2024-03-11T21:38:08.346669+0000","last_became_active":"2024-03-11T21:25:42.579045+0000","last_became_peered":"2024-03-11T21:25:42.579045+0000","last_unstale":"2024-03-11T21:38:08.346669+0000","last_undegraded":"2024-03-11T21:38:08.346669+0000","last_fullsized":"2024-03-11T21:38:08.346669+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:35:12.091185+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:29:36.918936+0000","last_clean_scrub_stamp":"2024-03-11T21:35:12.091185+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.d","version":"312'9437","reported_seq":9555,"reported_epoch":312,"state":"active+clean","last_fresh":"2024-03-11T21:38:12.255812+0000","last_change":"2024-03-11T21:35:45.700060+0000","last_active":"2024-03-11T21:38:12.255812+0000","last_peered":"2024-03-11T21:38:12.255812+0000","last_clean":"2024-03-11T21:38:12.255812+0000","last_became_active":"2024-03-11T21:35:45.699674+0000","last_became_peered":"2024-03-11T21:35:45.699674+0000","last_unstale":"2024-03-11T21:38:12.255812+0000","last_undegraded":"2024-03-11T21:38:12.255812+0000","last_fullsized":"2024-03-11T21:38:12.255812+0000","mapping_epoch":259,"log_start":"279'4800","ondisk_log_start":"279'4800","created":259,"last_epoch_clean":260,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:35:44.669493+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:35:44.669493+0000","last_clean_scrub_stamp":"2024-03-11T21:35:44.669493+0000","objects_scrubbed":0,"log_size":4637,"log_dups_size":0,"ondisk_log_size":4637,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":329515032,"num_objects":5029,"num_object_clones":0,"num_object_copies":15087,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5029,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":9437,"num_write_kb":462849,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,5],"acting":[7,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":545,"reported_epoch":314,"state":"active+clean","last_fresh":"2024-03-11T21:38:16.476343+0000","last_change":"2024-03-11T21:35:08.437783+0000","last_active":"2024-03-11T21:38:16.476343+0000","last_peered":"2024-03-11T21:38:16.476343+0000","last_clean":"2024-03-11T21:38:16.476343+0000","last_became_active":"2024-03-11T21:32:22.791439+0000","last_became_peered":"2024-03-11T21:32:22.791439+0000","last_unstale":"2024-03-11T21:38:16.476343+0000","last_undegraded":"2024-03-11T21:38:16.476343+0000","last_fullsized":"2024-03-11T21:38:16.476343+0000","mapping_epoch":166,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":167,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:35:08.437697+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:43.559452+0000","last_clean_scrub_stamp":"2024-03-11T21:35:08.437697+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":33,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,2],"acting":[4,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.6","version":"67'1","reported_seq":547,"reported_epoch":312,"state":"active+clean","last_fresh":"2024-03-11T21:38:08.355632+0000","last_change":"2024-03-11T21:35:30.581783+0000","last_active":"2024-03-11T21:38:08.355632+0000","last_peered":"2024-03-11T21:38:08.355632+0000","last_clean":"2024-03-11T21:38:08.355632+0000","last_became_active":"2024-03-11T21:25:38.536142+0000","last_became_peered":"2024-03-11T21:25:38.536142+0000","last_unstale":"2024-03-11T21:38:08.355632+0000","last_undegraded":"2024-03-11T21:38:08.355632+0000","last_fullsized":"2024-03-11T21:38:08.355632+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'1","last_scrub_stamp":"2024-03-11T21:35:30.581701+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:37.512018+0000","last_clean_scrub_stamp":"2024-03-11T21:35:30.581701+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,4],"acting":[1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.0","version":"260'38","reported_seq":545,"reported_epoch":312,"state":"active+clean","last_fresh":"2024-03-11T21:38:08.356934+0000","last_change":"2024-03-11T21:36:35.087771+0000","last_active":"2024-03-11T21:38:08.356934+0000","last_peered":"2024-03-11T21:38:08.356934+0000","last_clean":"2024-03-11T21:38:08.356934+0000","last_became_active":"2024-03-11T21:36:35.068071+0000","last_became_peered":"2024-03-11T21:36:35.068071+0000","last_unstale":"2024-03-11T21:38:08.356934+0000","last_undegraded":"2024-03-11T21:38:08.356934+0000","last_fullsized":"2024-03-11T21:38:08.356934+0000","mapping_epoch":277,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":278,"parent":"0.0","parent_split_bits":0,"last_scrub":"93'26","last_scrub_stamp":"2024-03-11T21:34:16.863984+0000","last_deep_scrub":"93'26","last_deep_scrub_stamp":"2024-03-11T21:26:40.680324+0000","last_clean_scrub_stamp":"2024-03-11T21:34:16.863984+0000","objects_scrubbed":8,"log_size":38,"log_dups_size":0,"ondisk_log_size":38,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":22,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":51,"num_read_kb":39,"num_write":32,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":564,"reported_epoch":314,"state":"active+clean","last_fresh":"2024-03-11T21:38:16.476033+0000","last_change":"2024-03-11T21:36:35.057830+0000","last_active":"2024-03-11T21:38:16.476033+0000","last_peered":"2024-03-11T21:38:16.476033+0000","last_clean":"2024-03-11T21:38:16.476033+0000","last_became_active":"2024-03-11T21:36:35.057517+0000","last_became_peered":"2024-03-11T21:36:35.057517+0000","last_unstale":"2024-03-11T21:38:16.476033+0000","last_undegraded":"2024-03-11T21:38:16.476033+0000","last_fullsized":"2024-03-11T21:38:16.476033+0000","mapping_epoch":277,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":278,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:35:12.411136+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:26:51.863621+0000","last_clean_scrub_stamp":"2024-03-11T21:35:12.411136+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,7],"acting":[4,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.c","version":"314'9504","reported_seq":17237,"reported_epoch":314,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-03-11T21:38:17.197810+0000","last_change":"2024-03-11T21:36:36.491614+0000","last_active":"2024-03-11T21:38:17.197810+0000","last_peered":"2024-03-11T21:38:17.197810+0000","last_clean":"2024-03-11T21:35:44.669493+0000","last_became_active":"2024-03-11T21:36:36.482095+0000","last_became_peered":"2024-03-11T21:36:36.482095+0000","last_unstale":"2024-03-11T21:38:17.197810+0000","last_undegraded":"2024-03-11T21:36:36.076624+0000","last_fullsized":"2024-03-11T21:36:36.041449+0000","mapping_epoch":278,"log_start":"274'4100","ondisk_log_start":"274'4100","created":259,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:35:44.669493+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:35:44.669493+0000","last_clean_scrub_stamp":"2024-03-11T21:35:44.669493+0000","objects_scrubbed":0,"log_size":5404,"log_dups_size":0,"ondisk_log_size":5404,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":309198848,"num_objects":4718,"num_object_clones":0,"num_object_copies":14154,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":822,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4718,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9504,"num_write_kb":455104,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3806,"num_bytes_recovered":249430016,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,7],"acting":[4,7],"avail_no_missing":["4","7"],"object_location_counts":[{"shards":"4,7","objects":3896}],"blocked_by":[],"up_primary":3,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.3","version":"0'0","reported_seq":182,"reported_epoch":312,"state":"active+clean","last_fresh":"2024-03-11T21:38:08.358124+0000","last_change":"2024-03-11T21:34:41.143973+0000","last_active":"2024-03-11T21:38:08.358124+0000","last_peered":"2024-03-11T21:38:08.358124+0000","last_clean":"2024-03-11T21:38:08.358124+0000","last_became_active":"2024-03-11T21:34:41.142683+0000","last_became_peered":"2024-03-11T21:34:41.142683+0000","last_unstale":"2024-03-11T21:38:08.358124+0000","last_undegraded":"2024-03-11T21:38:08.358124+0000","last_fullsized":"2024-03-11T21:38:08.358124+0000","mapping_epoch":226,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":227,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:34:27.353700+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:29:42.485740+0000","last_clean_scrub_stamp":"2024-03-11T21:34:27.353700+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,2],"acting":[7,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":566,"reported_epoch":312,"state":"active+clean","last_fresh":"2024-03-11T21:38:08.347738+0000","last_change":"2024-03-11T21:35:32.155983+0000","last_active":"2024-03-11T21:38:08.347738+0000","last_peered":"2024-03-11T21:38:08.347738+0000","last_clean":"2024-03-11T21:38:08.347738+0000","last_became_active":"2024-03-11T21:34:41.145628+0000","last_became_peered":"2024-03-11T21:34:41.145628+0000","last_unstale":"2024-03-11T21:38:08.347738+0000","last_undegraded":"2024-03-11T21:38:08.347738+0000","last_fullsized":"2024-03-11T21:38:08.347738+0000","mapping_epoch":226,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":227,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:35:32.155853+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:37.512018+0000","last_clean_scrub_stamp":"2024-03-11T21:35:32.155853+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,2],"acting":[6,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.1","version":"260'28","reported_seq":589,"reported_epoch":314,"state":"active+clean","last_fresh":"2024-03-11T21:38:16.475835+0000","last_change":"2024-03-11T21:35:33.644654+0000","last_active":"2024-03-11T21:38:16.475835+0000","last_peered":"2024-03-11T21:38:16.475835+0000","last_clean":"2024-03-11T21:38:16.475835+0000","last_became_active":"2024-03-11T21:25:40.546219+0000","last_became_peered":"2024-03-11T21:25:40.546219+0000","last_unstale":"2024-03-11T21:38:16.475835+0000","last_undegraded":"2024-03-11T21:38:16.475835+0000","last_fullsized":"2024-03-11T21:38:16.475835+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'14","last_scrub_stamp":"2024-03-11T21:35:33.644503+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:39.525554+0000","last_clean_scrub_stamp":"2024-03-11T21:35:33.644503+0000","objects_scrubbed":7,"log_size":28,"log_dups_size":0,"ondisk_log_size":28,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":42,"num_read_kb":35,"num_write":28,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6],"acting":[4,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.0","version":"280'11","reported_seq":623,"reported_epoch":312,"state":"active+clean","last_fresh":"2024-03-11T21:38:08.356734+0000","last_change":"2024-03-11T21:36:35.064085+0000","last_active":"2024-03-11T21:38:08.356734+0000","last_peered":"2024-03-11T21:38:08.356734+0000","last_clean":"2024-03-11T21:38:08.356734+0000","last_became_active":"2024-03-11T21:36:35.055250+0000","last_became_peered":"2024-03-11T21:36:35.055250+0000","last_unstale":"2024-03-11T21:38:08.356734+0000","last_undegraded":"2024-03-11T21:38:08.356734+0000","last_fullsized":"2024-03-11T21:38:08.356734+0000","mapping_epoch":277,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":278,"parent":"0.0","parent_split_bits":0,"last_scrub":"98'9","last_scrub_stamp":"2024-03-11T21:35:21.143578+0000","last_deep_scrub":"98'9","last_deep_scrub_stamp":"2024-03-11T21:27:29.745711+0000","last_clean_scrub_stamp":"2024-03-11T21:35:21.143578+0000","objects_scrubbed":1,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.3","version":"312'9319","reported_seq":9445,"reported_epoch":312,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-03-11T21:38:12.263227+0000","last_change":"2024-03-11T21:36:36.544317+0000","last_active":"2024-03-11T21:38:12.263227+0000","last_peered":"2024-03-11T21:38:12.263227+0000","last_clean":"2024-03-11T21:35:44.669493+0000","last_became_active":"2024-03-11T21:36:36.470537+0000","last_became_peered":"2024-03-11T21:36:36.470537+0000","last_unstale":"2024-03-11T21:38:12.263227+0000","last_undegraded":"2024-03-11T21:36:36.071501+0000","last_fullsized":"2024-03-11T21:36:36.040233+0000","mapping_epoch":278,"log_start":"279'4700","ondisk_log_start":"279'4700","created":259,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:35:44.669493+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:35:44.669493+0000","last_clean_scrub_stamp":"2024-03-11T21:35:44.669493+0000","objects_scrubbed":0,"log_size":4619,"log_dups_size":0,"ondisk_log_size":4619,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":327745536,"num_objects":5001,"num_object_clones":0,"num_object_copies":15003,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4641,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5001,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9319,"num_write_kb":458240,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":196608,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,1],"acting":[7,1],"avail_no_missing":["7","1"],"object_location_counts":[{"shards":"1,7","objects":360}],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.c","version":"0'0","reported_seq":466,"reported_epoch":312,"state":"active+clean","last_fresh":"2024-03-11T21:38:08.365290+0000","last_change":"2024-03-11T21:36:35.074112+0000","last_active":"2024-03-11T21:38:08.365290+0000","last_peered":"2024-03-11T21:38:08.365290+0000","last_clean":"2024-03-11T21:38:08.365290+0000","last_became_active":"2024-03-11T21:36:35.073778+0000","last_became_peered":"2024-03-11T21:36:35.073778+0000","last_unstale":"2024-03-11T21:38:08.365290+0000","last_undegraded":"2024-03-11T21:38:08.365290+0000","last_fullsized":"2024-03-11T21:38:08.365290+0000","mapping_epoch":277,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":278,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:32:17.600213+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:43.559452+0000","last_clean_scrub_stamp":"2024-03-11T21:32:17.600213+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.f","version":"0'0","reported_seq":545,"reported_epoch":312,"state":"active+clean","last_fresh":"2024-03-11T21:38:08.367182+0000","last_change":"2024-03-11T21:36:08.439013+0000","last_active":"2024-03-11T21:38:08.367182+0000","last_peered":"2024-03-11T21:38:08.367182+0000","last_clean":"2024-03-11T21:38:08.367182+0000","last_became_active":"2024-03-11T21:25:42.575935+0000","last_became_peered":"2024-03-11T21:25:42.575935+0000","last_unstale":"2024-03-11T21:38:08.367182+0000","last_undegraded":"2024-03-11T21:38:08.367182+0000","last_fullsized":"2024-03-11T21:38:08.367182+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:36:08.438895+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:41.544550+0000","last_clean_scrub_stamp":"2024-03-11T21:36:08.438895+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.8","version":"0'0","reported_seq":190,"reported_epoch":312,"state":"active+clean","last_fresh":"2024-03-11T21:38:08.350938+0000","last_change":"2024-03-11T21:35:34.733595+0000","last_active":"2024-03-11T21:38:08.350938+0000","last_peered":"2024-03-11T21:38:08.350938+0000","last_clean":"2024-03-11T21:38:08.350938+0000","last_became_active":"2024-03-11T21:34:41.144681+0000","last_became_peered":"2024-03-11T21:34:41.144681+0000","last_unstale":"2024-03-11T21:38:08.350938+0000","last_undegraded":"2024-03-11T21:38:08.350938+0000","last_fullsized":"2024-03-11T21:38:08.350938+0000","mapping_epoch":226,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":227,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:35:34.733491+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:37.512018+0000","last_clean_scrub_stamp":"2024-03-11T21:35:34.733491+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,1],"acting":[7,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.e","version":"260'24","reported_seq":582,"reported_epoch":314,"state":"active+clean","last_fresh":"2024-03-11T21:38:16.476917+0000","last_change":"2024-03-11T21:35:34.679906+0000","last_active":"2024-03-11T21:38:16.476917+0000","last_peered":"2024-03-11T21:38:16.476917+0000","last_clean":"2024-03-11T21:38:16.476917+0000","last_became_active":"2024-03-11T21:25:40.552178+0000","last_became_peered":"2024-03-11T21:25:40.552178+0000","last_unstale":"2024-03-11T21:38:16.476917+0000","last_undegraded":"2024-03-11T21:38:16.476917+0000","last_fullsized":"2024-03-11T21:38:16.476917+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"109'16","last_scrub_stamp":"2024-03-11T21:35:34.679817+0000","last_deep_scrub":"109'16","last_deep_scrub_stamp":"2024-03-11T21:28:24.739395+0000","last_clean_scrub_stamp":"2024-03-11T21:35:34.679817+0000","objects_scrubbed":5,"log_size":24,"log_dups_size":0,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":32,"num_read_kb":25,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.2","version":"312'9470","reported_seq":9585,"reported_epoch":312,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-03-11T21:38:12.622144+0000","last_change":"2024-03-11T21:36:36.513966+0000","last_active":"2024-03-11T21:38:12.622144+0000","last_peered":"2024-03-11T21:38:12.622144+0000","last_clean":"2024-03-11T21:35:44.669493+0000","last_became_active":"2024-03-11T21:36:36.506474+0000","last_became_peered":"2024-03-11T21:36:36.506474+0000","last_unstale":"2024-03-11T21:38:12.622144+0000","last_undegraded":"2024-03-11T21:36:36.068182+0000","last_fullsized":"2024-03-11T21:36:36.037781+0000","mapping_epoch":278,"log_start":"274'4400","ondisk_log_start":"274'4400","created":259,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:35:44.669493+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:35:44.669493+0000","last_clean_scrub_stamp":"2024-03-11T21:35:44.669493+0000","objects_scrubbed":0,"log_size":5070,"log_dups_size":0,"ondisk_log_size":5070,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":331612160,"num_objects":5060,"num_object_clones":0,"num_object_copies":15180,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4773,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5060,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9470,"num_write_kb":464960,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[1,2],"avail_no_missing":["1","2"],"object_location_counts":[{"shards":"1,2","objects":287}],"blocked_by":[],"up_primary":3,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.d","version":"0'0","reported_seq":533,"reported_epoch":312,"state":"active+clean","last_fresh":"2024-03-11T21:38:08.351662+0000","last_change":"2024-03-11T21:35:37.185210+0000","last_active":"2024-03-11T21:38:08.351662+0000","last_peered":"2024-03-11T21:38:08.351662+0000","last_clean":"2024-03-11T21:38:08.351662+0000","last_became_active":"2024-03-11T21:25:44.590333+0000","last_became_peered":"2024-03-11T21:25:44.590333+0000","last_unstale":"2024-03-11T21:38:08.351662+0000","last_undegraded":"2024-03-11T21:38:08.351662+0000","last_fullsized":"2024-03-11T21:38:08.351662+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:35:37.182850+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:35:37.182850+0000","last_clean_scrub_stamp":"2024-03-11T21:35:37.182850+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":40,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.e","version":"74'8","reported_seq":702,"reported_epoch":314,"state":"active+clean","last_fresh":"2024-03-11T21:38:16.477190+0000","last_change":"2024-03-11T21:36:07.734369+0000","last_active":"2024-03-11T21:38:16.477190+0000","last_peered":"2024-03-11T21:38:16.477190+0000","last_clean":"2024-03-11T21:38:16.477190+0000","last_became_active":"2024-03-11T21:25:42.576742+0000","last_became_peered":"2024-03-11T21:25:42.576742+0000","last_unstale":"2024-03-11T21:38:16.477190+0000","last_undegraded":"2024-03-11T21:38:16.477190+0000","last_fullsized":"2024-03-11T21:38:16.477190+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'8","last_scrub_stamp":"2024-03-11T21:36:07.734115+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:41.544550+0000","last_clean_scrub_stamp":"2024-03-11T21:36:07.734115+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,0],"acting":[4,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.9","version":"0'0","reported_seq":559,"reported_epoch":312,"state":"active+clean","last_fresh":"2024-03-11T21:38:08.351450+0000","last_change":"2024-03-11T21:36:35.066093+0000","last_active":"2024-03-11T21:38:08.351450+0000","last_peered":"2024-03-11T21:38:08.351450+0000","last_clean":"2024-03-11T21:38:08.351450+0000","last_became_active":"2024-03-11T21:36:35.065733+0000","last_became_peered":"2024-03-11T21:36:35.065733+0000","last_unstale":"2024-03-11T21:38:08.351450+0000","last_undegraded":"2024-03-11T21:38:08.351450+0000","last_fullsized":"2024-03-11T21:38:08.351450+0000","mapping_epoch":277,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":278,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:32:02.506579+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:26:50.991712+0000","last_clean_scrub_stamp":"2024-03-11T21:32:02.506579+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,3],"acting":[1,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.f","version":"260'27","reported_seq":561,"reported_epoch":312,"state":"active+clean","last_fresh":"2024-03-11T21:38:08.366284+0000","last_change":"2024-03-11T21:36:35.084069+0000","last_active":"2024-03-11T21:38:08.366284+0000","last_peered":"2024-03-11T21:38:08.366284+0000","last_clean":"2024-03-11T21:38:08.366284+0000","last_became_active":"2024-03-11T21:36:35.058272+0000","last_became_peered":"2024-03-11T21:36:35.058272+0000","last_unstale":"2024-03-11T21:38:08.366284+0000","last_undegraded":"2024-03-11T21:38:08.366284+0000","last_fullsized":"2024-03-11T21:38:08.366284+0000","mapping_epoch":277,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":278,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'15","last_scrub_stamp":"2024-03-11T21:34:21.517894+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:39.525554+0000","last_clean_scrub_stamp":"2024-03-11T21:34:21.517894+0000","objects_scrubbed":7,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":44,"num_read_kb":35,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,4],"acting":[2,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"9.1","version":"312'9464","reported_seq":17713,"reported_epoch":312,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-03-11T21:38:12.302330+0000","last_change":"2024-03-11T21:36:36.545694+0000","last_active":"2024-03-11T21:38:12.302330+0000","last_peered":"2024-03-11T21:38:12.302330+0000","last_clean":"2024-03-11T21:35:44.669493+0000","last_became_active":"2024-03-11T21:36:36.509431+0000","last_became_peered":"2024-03-11T21:36:36.509431+0000","last_unstale":"2024-03-11T21:38:12.302330+0000","last_undegraded":"2024-03-11T21:36:36.165469+0000","last_fullsized":"2024-03-11T21:36:36.044027+0000","mapping_epoch":278,"log_start":"270'2900","ondisk_log_start":"270'2900","created":259,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:35:44.669493+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:35:44.669493+0000","last_clean_scrub_stamp":"2024-03-11T21:35:44.669493+0000","objects_scrubbed":0,"log_size":6564,"log_dups_size":0,"ondisk_log_size":6564,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":335544320,"num_objects":5120,"num_object_clones":0,"num_object_copies":15360,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":759,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5120,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9464,"num_write_kb":466688,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4066,"num_bytes_recovered":266469376,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,7],"acting":[6,7],"avail_no_missing":["6","7"],"object_location_counts":[{"shards":"6,7","objects":4361}],"blocked_by":[],"up_primary":3,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.e","version":"0'0","reported_seq":538,"reported_epoch":314,"state":"active+clean","last_fresh":"2024-03-11T21:38:16.477272+0000","last_change":"2024-03-11T21:36:10.769557+0000","last_active":"2024-03-11T21:38:16.477272+0000","last_peered":"2024-03-11T21:38:16.477272+0000","last_clean":"2024-03-11T21:38:16.477272+0000","last_became_active":"2024-03-11T21:25:44.590198+0000","last_became_peered":"2024-03-11T21:25:44.590198+0000","last_unstale":"2024-03-11T21:38:16.477272+0000","last_undegraded":"2024-03-11T21:38:16.477272+0000","last_fullsized":"2024-03-11T21:38:16.477272+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:36:10.769460+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:43.559452+0000","last_clean_scrub_stamp":"2024-03-11T21:36:10.769460+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":75,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.d","version":"109'9","reported_seq":697,"reported_epoch":312,"state":"active+clean","last_fresh":"2024-03-11T21:38:08.366259+0000","last_change":"2024-03-11T21:35:07.562523+0000","last_active":"2024-03-11T21:38:08.366259+0000","last_peered":"2024-03-11T21:38:08.366259+0000","last_clean":"2024-03-11T21:38:08.366259+0000","last_became_active":"2024-03-11T21:25:42.575583+0000","last_became_peered":"2024-03-11T21:25:42.575583+0000","last_unstale":"2024-03-11T21:38:08.366259+0000","last_undegraded":"2024-03-11T21:38:08.366259+0000","last_fullsized":"2024-03-11T21:38:08.366259+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"109'9","last_scrub_stamp":"2024-03-11T21:35:07.562385+0000","last_deep_scrub":"109'9","last_deep_scrub_stamp":"2024-03-11T21:28:27.082975+0000","last_clean_scrub_stamp":"2024-03-11T21:35:07.562385+0000","objects_scrubbed":1,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,5],"acting":[2,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.a","version":"0'0","reported_seq":567,"reported_epoch":312,"state":"active+clean","last_fresh":"2024-03-11T21:38:08.359955+0000","last_change":"2024-03-11T21:36:35.067262+0000","last_active":"2024-03-11T21:38:08.359955+0000","last_peered":"2024-03-11T21:38:08.359955+0000","last_clean":"2024-03-11T21:38:08.359955+0000","last_became_active":"2024-03-11T21:36:35.066832+0000","last_became_peered":"2024-03-11T21:36:35.066832+0000","last_unstale":"2024-03-11T21:38:08.359955+0000","last_undegraded":"2024-03-11T21:38:08.359955+0000","last_fullsized":"2024-03-11T21:38:08.359955+0000","mapping_epoch":277,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":278,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:34:22.526916+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:37.512018+0000","last_clean_scrub_stamp":"2024-03-11T21:34:22.526916+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,7],"acting":[1,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.c","version":"260'20","reported_seq":590,"reported_epoch":314,"state":"active+clean","last_fresh":"2024-03-11T21:38:16.476690+0000","last_change":"2024-03-11T21:36:35.095574+0000","last_active":"2024-03-11T21:38:16.476690+0000","last_peered":"2024-03-11T21:38:16.476690+0000","last_clean":"2024-03-11T21:38:16.476690+0000","last_became_active":"2024-03-11T21:36:35.067269+0000","last_became_peered":"2024-03-11T21:36:35.067269+0000","last_unstale":"2024-03-11T21:38:16.476690+0000","last_undegraded":"2024-03-11T21:38:16.476690+0000","last_fullsized":"2024-03-11T21:38:16.476690+0000","mapping_epoch":277,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":278,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'10","last_scrub_stamp":"2024-03-11T21:32:54.846584+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:39.525554+0000","last_clean_scrub_stamp":"2024-03-11T21:32:54.846584+0000","objects_scrubbed":5,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":25,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.0","version":"312'9529","reported_seq":9646,"reported_epoch":312,"state":"active+clean","last_fresh":"2024-03-11T21:38:12.735465+0000","last_change":"2024-03-11T21:35:45.703065+0000","last_active":"2024-03-11T21:38:12.735465+0000","last_peered":"2024-03-11T21:38:12.735465+0000","last_clean":"2024-03-11T21:38:12.735465+0000","last_became_active":"2024-03-11T21:35:45.702739+0000","last_became_peered":"2024-03-11T21:35:45.702739+0000","last_unstale":"2024-03-11T21:38:12.735465+0000","last_undegraded":"2024-03-11T21:38:12.735465+0000","last_fullsized":"2024-03-11T21:38:12.735465+0000","mapping_epoch":259,"log_start":"274'4500","ondisk_log_start":"274'4500","created":259,"last_epoch_clean":260,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:35:44.669493+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:35:44.669493+0000","last_clean_scrub_stamp":"2024-03-11T21:35:44.669493+0000","objects_scrubbed":0,"log_size":5029,"log_dups_size":0,"ondisk_log_size":5029,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":331284480,"num_objects":5055,"num_object_clones":0,"num_object_copies":15165,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5055,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9529,"num_write_kb":466688,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.f","version":"0'0","reported_seq":542,"reported_epoch":312,"state":"active+clean","last_fresh":"2024-03-11T21:38:08.366144+0000","last_change":"2024-03-11T21:36:35.058630+0000","last_active":"2024-03-11T21:38:08.366144+0000","last_peered":"2024-03-11T21:38:08.366144+0000","last_clean":"2024-03-11T21:38:08.366144+0000","last_became_active":"2024-03-11T21:36:35.057800+0000","last_became_peered":"2024-03-11T21:36:35.057800+0000","last_unstale":"2024-03-11T21:38:08.366144+0000","last_undegraded":"2024-03-11T21:38:08.366144+0000","last_fullsized":"2024-03-11T21:38:08.366144+0000","mapping_epoch":277,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":278,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:35:10.608981+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:27:03.010969+0000","last_clean_scrub_stamp":"2024-03-11T21:35:10.608981+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":94,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,4],"acting":[2,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":538,"reported_epoch":312,"state":"active+clean","last_fresh":"2024-03-11T21:38:08.382517+0000","last_change":"2024-03-11T21:36:10.608789+0000","last_active":"2024-03-11T21:38:08.382517+0000","last_peered":"2024-03-11T21:38:08.382517+0000","last_clean":"2024-03-11T21:38:08.382517+0000","last_became_active":"2024-03-11T21:25:42.579409+0000","last_became_peered":"2024-03-11T21:25:42.579409+0000","last_unstale":"2024-03-11T21:38:08.382517+0000","last_undegraded":"2024-03-11T21:38:08.382517+0000","last_fullsized":"2024-03-11T21:38:08.382517+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:36:10.608552+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:30:22.750053+0000","last_clean_scrub_stamp":"2024-03-11T21:36:10.608552+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.b","version":"0'0","reported_seq":190,"reported_epoch":312,"state":"active+clean","last_fresh":"2024-03-11T21:38:08.358212+0000","last_change":"2024-03-11T21:36:16.719432+0000","last_active":"2024-03-11T21:38:08.358212+0000","last_peered":"2024-03-11T21:38:08.358212+0000","last_clean":"2024-03-11T21:38:08.358212+0000","last_became_active":"2024-03-11T21:34:41.134987+0000","last_became_peered":"2024-03-11T21:34:41.134987+0000","last_unstale":"2024-03-11T21:38:08.358212+0000","last_undegraded":"2024-03-11T21:38:08.358212+0000","last_fullsized":"2024-03-11T21:38:08.358212+0000","mapping_epoch":226,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":227,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:36:16.719309+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:36:16.719309+0000","last_clean_scrub_stamp":"2024-03-11T21:36:16.719309+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,5],"acting":[7,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.d","version":"260'39","reported_seq":603,"reported_epoch":314,"state":"active+clean","last_fresh":"2024-03-11T21:38:16.476375+0000","last_change":"2024-03-11T21:35:25.543769+0000","last_active":"2024-03-11T21:38:16.476375+0000","last_peered":"2024-03-11T21:38:16.476375+0000","last_clean":"2024-03-11T21:38:16.476375+0000","last_became_active":"2024-03-11T21:25:40.553454+0000","last_became_peered":"2024-03-11T21:25:40.553454+0000","last_unstale":"2024-03-11T21:38:16.476375+0000","last_undegraded":"2024-03-11T21:38:16.476375+0000","last_fullsized":"2024-03-11T21:38:16.476375+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"165'25","last_scrub_stamp":"2024-03-11T21:35:25.543628+0000","last_deep_scrub":"165'25","last_deep_scrub_stamp":"2024-03-11T21:32:21.678328+0000","last_clean_scrub_stamp":"2024-03-11T21:35:25.543628+0000","objects_scrubbed":8,"log_size":39,"log_dups_size":0,"ondisk_log_size":39,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":50,"num_read_kb":40,"num_write":32,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,1],"acting":[4,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.7","version":"312'9511","reported_seq":14694,"reported_epoch":312,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-03-11T21:38:12.450661+0000","last_change":"2024-03-11T21:36:36.539792+0000","last_active":"2024-03-11T21:38:12.450661+0000","last_peered":"2024-03-11T21:38:12.450661+0000","last_clean":"2024-03-11T21:35:44.669493+0000","last_became_active":"2024-03-11T21:36:36.508974+0000","last_became_peered":"2024-03-11T21:36:36.508974+0000","last_unstale":"2024-03-11T21:38:12.450661+0000","last_undegraded":"2024-03-11T21:36:36.183555+0000","last_fullsized":"2024-03-11T21:36:36.038772+0000","mapping_epoch":278,"log_start":"274'4300","ondisk_log_start":"274'4300","created":259,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:35:44.669493+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:35:44.669493+0000","last_clean_scrub_stamp":"2024-03-11T21:35:44.669493+0000","objects_scrubbed":0,"log_size":5211,"log_dups_size":0,"ondisk_log_size":5211,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328794112,"num_objects":5017,"num_object_clones":0,"num_object_copies":15051,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2220,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5017,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9511,"num_write_kb":464896,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2532,"num_bytes_recovered":165937152,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,7],"avail_no_missing":["5","7"],"object_location_counts":[{"shards":"5,7","objects":2797}],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":190,"reported_epoch":312,"state":"active+clean","last_fresh":"2024-03-11T21:38:08.350730+0000","last_change":"2024-03-11T21:36:06.735035+0000","last_active":"2024-03-11T21:38:08.350730+0000","last_peered":"2024-03-11T21:38:08.350730+0000","last_clean":"2024-03-11T21:38:08.350730+0000","last_became_active":"2024-03-11T21:34:41.145012+0000","last_became_peered":"2024-03-11T21:34:41.145012+0000","last_unstale":"2024-03-11T21:38:08.350730+0000","last_undegraded":"2024-03-11T21:38:08.350730+0000","last_fullsized":"2024-03-11T21:38:08.350730+0000","mapping_epoch":226,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":227,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:36:06.734921+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:28:37.561826+0000","last_clean_scrub_stamp":"2024-03-11T21:36:06.734921+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":23,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,0],"acting":[7,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":537,"reported_epoch":312,"state":"active+clean","last_fresh":"2024-03-11T21:38:08.365989+0000","last_change":"2024-03-11T21:35:14.636561+0000","last_active":"2024-03-11T21:38:08.365989+0000","last_peered":"2024-03-11T21:38:08.365989+0000","last_clean":"2024-03-11T21:38:08.365989+0000","last_became_active":"2024-03-11T21:25:42.574515+0000","last_became_peered":"2024-03-11T21:25:42.574515+0000","last_unstale":"2024-03-11T21:38:08.365989+0000","last_undegraded":"2024-03-11T21:38:08.365989+0000","last_fullsized":"2024-03-11T21:38:08.365989+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:35:14.636450+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:41.544550+0000","last_clean_scrub_stamp":"2024-03-11T21:35:14.636450+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,5],"acting":[2,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.c","version":"0'0","reported_seq":545,"reported_epoch":312,"state":"active+clean","last_fresh":"2024-03-11T21:38:08.366628+0000","last_change":"2024-03-11T21:36:00.849272+0000","last_active":"2024-03-11T21:38:08.366628+0000","last_peered":"2024-03-11T21:38:08.366628+0000","last_clean":"2024-03-11T21:38:08.366628+0000","last_became_active":"2024-03-11T21:25:38.533240+0000","last_became_peered":"2024-03-11T21:25:38.533240+0000","last_unstale":"2024-03-11T21:38:08.366628+0000","last_undegraded":"2024-03-11T21:38:08.366628+0000","last_fullsized":"2024-03-11T21:38:08.366628+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:36:00.849153+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:29:30.279910+0000","last_clean_scrub_stamp":"2024-03-11T21:36:00.849153+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,0],"acting":[2,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.a","version":"260'44","reported_seq":666,"reported_epoch":312,"state":"active+clean","last_fresh":"2024-03-11T21:38:08.346452+0000","last_change":"2024-03-11T21:35:34.138400+0000","last_active":"2024-03-11T21:38:08.346452+0000","last_peered":"2024-03-11T21:38:08.346452+0000","last_clean":"2024-03-11T21:38:08.346452+0000","last_became_active":"2024-03-11T21:34:41.148863+0000","last_became_peered":"2024-03-11T21:34:41.148863+0000","last_unstale":"2024-03-11T21:38:08.346452+0000","last_undegraded":"2024-03-11T21:38:08.346452+0000","last_fullsized":"2024-03-11T21:38:08.346452+0000","mapping_epoch":226,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":227,"parent":"0.0","parent_split_bits":0,"last_scrub":"109'28","last_scrub_stamp":"2024-03-11T21:35:34.138261+0000","last_deep_scrub":"109'28","last_deep_scrub_stamp":"2024-03-11T21:28:24.016389+0000","last_clean_scrub_stamp":"2024-03-11T21:35:34.138261+0000","objects_scrubbed":9,"log_size":44,"log_dups_size":0,"ondisk_log_size":44,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":56,"num_read_kb":45,"num_write":36,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,7],"acting":[6,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.6","version":"312'9570","reported_seq":9687,"reported_epoch":312,"state":"active+clean","last_fresh":"2024-03-11T21:38:12.451632+0000","last_change":"2024-03-11T21:35:45.703533+0000","last_active":"2024-03-11T21:38:12.451632+0000","last_peered":"2024-03-11T21:38:12.451632+0000","last_clean":"2024-03-11T21:38:12.451632+0000","last_became_active":"2024-03-11T21:35:45.703184+0000","last_became_peered":"2024-03-11T21:35:45.703184+0000","last_unstale":"2024-03-11T21:38:12.451632+0000","last_undegraded":"2024-03-11T21:38:12.451632+0000","last_fullsized":"2024-03-11T21:38:12.451632+0000","mapping_epoch":259,"log_start":"274'4300","ondisk_log_start":"274'4300","created":259,"last_epoch_clean":260,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:35:44.669493+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:35:44.669493+0000","last_clean_scrub_stamp":"2024-03-11T21:35:44.669493+0000","objects_scrubbed":0,"log_size":5270,"log_dups_size":0,"ondisk_log_size":5270,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":326107136,"num_objects":4976,"num_object_clones":0,"num_object_copies":14928,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4976,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9570,"num_write_kb":465472,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,1],"acting":[5,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":546,"reported_epoch":312,"state":"active+clean","last_fresh":"2024-03-11T21:38:08.350291+0000","last_change":"2024-03-11T21:35:37.687281+0000","last_active":"2024-03-11T21:38:08.350291+0000","last_peered":"2024-03-11T21:38:08.350291+0000","last_clean":"2024-03-11T21:38:08.350291+0000","last_became_active":"2024-03-11T21:34:41.154507+0000","last_became_peered":"2024-03-11T21:34:41.154507+0000","last_unstale":"2024-03-11T21:38:08.350291+0000","last_undegraded":"2024-03-11T21:38:08.350291+0000","last_fullsized":"2024-03-11T21:38:08.350291+0000","mapping_epoch":226,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":227,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:35:37.687187+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:27:11.361913+0000","last_clean_scrub_stamp":"2024-03-11T21:35:37.687187+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":39,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":538,"reported_epoch":312,"state":"active+clean","last_fresh":"2024-03-11T21:38:08.368214+0000","last_change":"2024-03-11T21:36:35.059662+0000","last_active":"2024-03-11T21:38:08.368214+0000","last_peered":"2024-03-11T21:38:08.368214+0000","last_clean":"2024-03-11T21:38:08.368214+0000","last_became_active":"2024-03-11T21:36:35.059192+0000","last_became_peered":"2024-03-11T21:36:35.059192+0000","last_unstale":"2024-03-11T21:38:08.368214+0000","last_undegraded":"2024-03-11T21:38:08.368214+0000","last_fullsized":"2024-03-11T21:38:08.368214+0000","mapping_epoch":277,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":278,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:34:17.412687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:41.544550+0000","last_clean_scrub_stamp":"2024-03-11T21:34:17.412687+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,3],"acting":[2,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.d","version":"0'0","reported_seq":547,"reported_epoch":312,"state":"active+clean","last_fresh":"2024-03-11T21:38:08.356124+0000","last_change":"2024-03-11T21:36:35.058635+0000","last_active":"2024-03-11T21:38:08.356124+0000","last_peered":"2024-03-11T21:38:08.356124+0000","last_clean":"2024-03-11T21:38:08.356124+0000","last_became_active":"2024-03-11T21:36:35.058296+0000","last_became_peered":"2024-03-11T21:36:35.058296+0000","last_unstale":"2024-03-11T21:38:08.356124+0000","last_undegraded":"2024-03-11T21:38:08.356124+0000","last_fullsized":"2024-03-11T21:38:08.356124+0000","mapping_epoch":277,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":278,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:31:55.464145+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:37.512018+0000","last_clean_scrub_stamp":"2024-03-11T21:31:55.464145+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,3],"acting":[1,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.b","version":"260'15","reported_seq":567,"reported_epoch":312,"state":"active+clean","last_fresh":"2024-03-11T21:38:08.349604+0000","last_change":"2024-03-11T21:35:27.760318+0000","last_active":"2024-03-11T21:38:08.349604+0000","last_peered":"2024-03-11T21:38:08.349604+0000","last_clean":"2024-03-11T21:38:08.349604+0000","last_became_active":"2024-03-11T21:25:40.549612+0000","last_became_peered":"2024-03-11T21:25:40.549612+0000","last_unstale":"2024-03-11T21:38:08.349604+0000","last_undegraded":"2024-03-11T21:38:08.349604+0000","last_fullsized":"2024-03-11T21:38:08.349604+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'9","last_scrub_stamp":"2024-03-11T21:35:27.760208+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:39.525554+0000","last_clean_scrub_stamp":"2024-03-11T21:35:27.760208+0000","objects_scrubbed":4,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.5","version":"312'9645","reported_seq":9762,"reported_epoch":312,"state":"active+clean","last_fresh":"2024-03-11T21:38:13.464429+0000","last_change":"2024-03-11T21:35:45.697209+0000","last_active":"2024-03-11T21:38:13.464429+0000","last_peered":"2024-03-11T21:38:13.464429+0000","last_clean":"2024-03-11T21:38:13.464429+0000","last_became_active":"2024-03-11T21:35:45.696788+0000","last_became_peered":"2024-03-11T21:35:45.696788+0000","last_unstale":"2024-03-11T21:38:13.464429+0000","last_undegraded":"2024-03-11T21:38:13.464429+0000","last_fullsized":"2024-03-11T21:38:13.464429+0000","mapping_epoch":259,"log_start":"274'4000","ondisk_log_start":"274'4000","created":259,"last_epoch_clean":260,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:35:44.669493+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:35:44.669493+0000","last_clean_scrub_stamp":"2024-03-11T21:35:44.669493+0000","objects_scrubbed":0,"log_size":5645,"log_dups_size":0,"ondisk_log_size":5645,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":337969152,"num_objects":5157,"num_object_clones":0,"num_object_copies":15471,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5157,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9645,"num_write_kb":473664,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":452,"reported_epoch":314,"state":"active+clean","last_fresh":"2024-03-11T21:38:16.475773+0000","last_change":"2024-03-11T21:36:35.057118+0000","last_active":"2024-03-11T21:38:16.475773+0000","last_peered":"2024-03-11T21:38:16.475773+0000","last_clean":"2024-03-11T21:38:16.475773+0000","last_became_active":"2024-03-11T21:36:35.056518+0000","last_became_peered":"2024-03-11T21:36:35.056518+0000","last_unstale":"2024-03-11T21:38:16.475773+0000","last_undegraded":"2024-03-11T21:38:16.475773+0000","last_fullsized":"2024-03-11T21:38:16.475773+0000","mapping_epoch":277,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":278,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:35:24.514584+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:43.559452+0000","last_clean_scrub_stamp":"2024-03-11T21:35:24.514584+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,3],"acting":[4,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.e","version":"0'0","reported_seq":566,"reported_epoch":312,"state":"active+clean","last_fresh":"2024-03-11T21:38:08.366500+0000","last_change":"2024-03-11T21:36:35.057187+0000","last_active":"2024-03-11T21:38:08.366500+0000","last_peered":"2024-03-11T21:38:08.366500+0000","last_clean":"2024-03-11T21:38:08.366500+0000","last_became_active":"2024-03-11T21:36:35.056777+0000","last_became_peered":"2024-03-11T21:36:35.056777+0000","last_unstale":"2024-03-11T21:38:08.366500+0000","last_undegraded":"2024-03-11T21:38:08.366500+0000","last_fullsized":"2024-03-11T21:38:08.366500+0000","mapping_epoch":277,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":278,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:34:19.476172+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:37.512018+0000","last_clean_scrub_stamp":"2024-03-11T21:34:19.476172+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":17,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,7],"acting":[2,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.8","version":"260'34","reported_seq":635,"reported_epoch":312,"state":"active+clean","last_fresh":"2024-03-11T21:38:08.352136+0000","last_change":"2024-03-11T21:35:20.304345+0000","last_active":"2024-03-11T21:38:08.352136+0000","last_peered":"2024-03-11T21:38:08.352136+0000","last_clean":"2024-03-11T21:38:08.352136+0000","last_became_active":"2024-03-11T21:34:41.146631+0000","last_became_peered":"2024-03-11T21:34:41.146631+0000","last_unstale":"2024-03-11T21:38:08.352136+0000","last_undegraded":"2024-03-11T21:38:08.352136+0000","last_fullsized":"2024-03-11T21:38:08.352136+0000","mapping_epoch":226,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":227,"parent":"0.0","parent_split_bits":0,"last_scrub":"109'22","last_scrub_stamp":"2024-03-11T21:35:20.304114+0000","last_deep_scrub":"109'22","last_deep_scrub_stamp":"2024-03-11T21:32:07.573197+0000","last_clean_scrub_stamp":"2024-03-11T21:35:20.304114+0000","objects_scrubbed":7,"log_size":34,"log_dups_size":0,"ondisk_log_size":34,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":44,"num_read_kb":35,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.9","version":"229'11","reported_seq":236,"reported_epoch":312,"state":"active+clean","last_fresh":"2024-03-11T21:38:08.352978+0000","last_change":"2024-03-11T21:36:18.726876+0000","last_active":"2024-03-11T21:38:08.352978+0000","last_peered":"2024-03-11T21:38:08.352978+0000","last_clean":"2024-03-11T21:38:08.352978+0000","last_became_active":"2024-03-11T21:34:41.156892+0000","last_became_peered":"2024-03-11T21:34:41.156892+0000","last_unstale":"2024-03-11T21:38:08.352978+0000","last_undegraded":"2024-03-11T21:38:08.352978+0000","last_fullsized":"2024-03-11T21:38:08.352978+0000","mapping_epoch":226,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":227,"parent":"0.0","parent_split_bits":0,"last_scrub":"229'11","last_scrub_stamp":"2024-03-11T21:36:18.726763+0000","last_deep_scrub":"94'9","last_deep_scrub_stamp":"2024-03-11T21:26:53.422552+0000","last_clean_scrub_stamp":"2024-03-11T21:36:18.726763+0000","objects_scrubbed":1,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":9,"num_pg":16,"stat_sum":{"num_bytes":5275123736,"num_objects":80493,"num_object_clones":0,"num_object_copies":241479,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":27615,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":80493,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":152271,"num_write_kb":7448385,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10408,"num_bytes_recovered":682098688,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":14750986240,"data_stored":14750974024,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":84371,"ondisk_log_size":84371,"up":48,"acting":40,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":1531,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":17,"num_read_kb":17,"num_write":10,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":1308,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":4593,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":6,"ondisk_log_size":6,"up":96,"acting":96,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":198,"num_read_kb":294,"num_write":179,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":2951840,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":2375680,"data_stored":2361472,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":128,"ondisk_log_size":128,"up":3,"acting":3,"num_store_stats":4},{"poolid":3,"num_pg":3,"stat_sum":{"num_bytes":408,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":747,"num_read_kb":742,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1224,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":8,"ondisk_log_size":8,"up":9,"acting":9,"num_store_stats":7},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":3702,"num_objects":178,"num_object_clones":0,"num_object_copies":534,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":178,"num_whiteouts":0,"num_read":1120,"num_read_kb":877,"num_write":703,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":186,"num_bytes_recovered":2200,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":417792,"data_stored":11106,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":760,"ondisk_log_size":760,"up":96,"acting":96,"num_store_stats":8},{"poolid":5,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":78,"ondisk_log_size":78,"up":96,"acting":96,"num_store_stats":8},{"poolid":6,"num_pg":32,"stat_sum":{"num_bytes":383,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":370,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1149,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":96,"acting":96,"num_store_stats":8}],"osd_stats":[{"osd":7,"up_from":60,"seq":257698037914,"num_pgs":65,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":3405128,"kb_used_data":2950528,"kb_used_omap":1,"kb_used_meta":454590,"kb_avail":90340024,"statfs":{"total":95995035648,"available":92508184576,"internally_reserved":0,"allocated":3021340672,"data_stored":3020591920,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":465500618},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":50,"seq":214748364960,"num_pgs":46,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1354052,"kb_used_data":975804,"kb_used_omap":1,"kb_used_meta":378238,"kb_avail":92391100,"statfs":{"total":95995035648,"available":94608486400,"internally_reserved":0,"allocated":999223296,"data_stored":998479345,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":387316166},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834575034,"num_pgs":60,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":3442464,"kb_used_data":2905880,"kb_used_omap":1,"kb_used_meta":536510,"kb_avail":90302688,"statfs":{"total":95995035648,"available":92469952512,"internally_reserved":0,"allocated":2975621120,"data_stored":2974847975,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":549386693},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738559,"num_pgs":54,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1801040,"kb_used_data":1306184,"kb_used_omap":1,"kb_used_meta":494846,"kb_avail":91944112,"statfs":{"total":95995035648,"available":94150770688,"internally_reserved":0,"allocated":1337532416,"data_stored":1336776211,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":506722757},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411508,"num_pgs":43,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1397820,"kb_used_data":973748,"kb_used_omap":1,"kb_used_meta":424062,"kb_avail":92347332,"statfs":{"total":95995035648,"available":94563667968,"internally_reserved":0,"allocated":997117952,"data_stored":996381417,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":434239941},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":277,"seq":1189705941013,"num_pgs":62,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1707744,"kb_used_data":1507480,"kb_used_omap":4,"kb_used_meta":200251,"kb_avail":92037408,"statfs":{"total":95995035648,"available":94246305792,"internally_reserved":0,"allocated":1543659520,"data_stored":1542886795,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4442,"internal_metadata":205057702},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":4,"up_from":32,"seq":137438953642,"num_pgs":56,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1299056,"kb_used_data":924960,"kb_used_omap":1,"kb_used_meta":374078,"kb_avail":92446096,"statfs":{"total":95995035648,"available":94664802304,"internally_reserved":0,"allocated":947159040,"data_stored":946391865,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":383056325},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":41,"seq":176093659300,"num_pgs":58,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":3425856,"kb_used_data":2925432,"kb_used_omap":1,"kb_used_meta":500414,"kb_avail":90319296,"statfs":{"total":95995035648,"available":92486959104,"internally_reserved":0,"allocated":2995642368,"data_stored":2994901376,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":512424389},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1049,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":46,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":436,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":957,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":574,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":436,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1003,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":92,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":1430,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":77824,"data_stored":2008,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":1018,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":1898,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":1474,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":990,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":1034,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":1254,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":370,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":383,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":370,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":1327169536,"data_stored":1327169536,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":2967015424,"data_stored":2967011352,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":990838784,"data_stored":990838784,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":1548091392,"data_stored":1548091392,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":942604288,"data_stored":942604288,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":2985103360,"data_stored":2985099288,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":996081664,"data_stored":996081664,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":2994081792,"data_stored":2994077720,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-11T21:38:20.179 DEBUG:teuthology.orchestra.run.smithi092:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid 00db0016-dfed-11ee-95c8-87774f69a715 -- ceph status --format=json 2024-03-11T21:38:20.200 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:38:20.238 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:38:20.238 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:38:20.239 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:38:20.339 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:38:20.384 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:38:20.384 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:38:20.385 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:38:20.486 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:38:20.515 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:38:20 smithi097 ceph-mon[33317]: pgmap v589: 148 pgs: 5 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+degraded+remapped, 140 active+clean; 4.7 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 988 op/s; 27048/230574 objects degraded (11.731%); 13 MiB/s, 204 objects/s recovering 2024-03-11T21:38:20.515 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:38:20 smithi097 ceph-mon[33317]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-11T21:38:20.515 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:38:20 smithi097 ceph-mon[33317]: from='client.25447 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-03-11T21:38:20.515 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:38:20 smithi097 ceph-mon[33317]: osdmap e315: 8 total, 8 up, 8 in 2024-03-11T21:38:20.526 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:38:20.527 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:38:20.527 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:38:20.628 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:38:20.667 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:38:20.667 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:38:20.668 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:38:20.768 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:38:20.832 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:38:20.833 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:38:20.833 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:38:20.861 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:38:20 smithi092 ceph-mon[29572]: pgmap v589: 148 pgs: 5 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+degraded+remapped, 140 active+clean; 4.7 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 988 op/s; 27048/230574 objects degraded (11.731%); 13 MiB/s, 204 objects/s recovering 2024-03-11T21:38:20.861 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:38:20 smithi092 ceph-mon[29572]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-11T21:38:20.861 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:38:20 smithi092 ceph-mon[29572]: from='client.25447 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-03-11T21:38:20.861 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:38:20 smithi092 ceph-mon[29572]: osdmap e315: 8 total, 8 up, 8 in 2024-03-11T21:38:20.862 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:38:20 smithi092 ceph-mon[37563]: pgmap v589: 148 pgs: 5 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+degraded+remapped, 140 active+clean; 4.7 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 988 op/s; 27048/230574 objects degraded (11.731%); 13 MiB/s, 204 objects/s recovering 2024-03-11T21:38:20.862 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:38:20 smithi092 ceph-mon[37563]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-11T21:38:20.862 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:38:20 smithi092 ceph-mon[37563]: from='client.25447 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-03-11T21:38:20.862 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:38:20 smithi092 ceph-mon[37563]: osdmap e315: 8 total, 8 up, 8 in 2024-03-11T21:38:20.934 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:38:20.968 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:38:20.969 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:38:20.969 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:38:21.070 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:38:21.099 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:38:21.100 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:38:21.100 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:38:21.201 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:38:21.245 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:38:21.246 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:38:21.246 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:38:21.347 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:38:21.386 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:38:21.387 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:38:21.387 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:38:21.488 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:38:21.532 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:38:21.533 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:38:21.534 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:38:21.634 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:38:21.664 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:38:21.665 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:38:21.665 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:38:21.767 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:38:21.793 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:38:21.794 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:38:21.794 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:38:21.895 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:38:21.921 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:38:21.922 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:38:21.922 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:38:21.999 INFO:teuthology.orchestra.run.smithi092.stderr:Inferring config /var/lib/ceph/00db0016-dfed-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T21:38:22.023 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:38:22.040 INFO:teuthology.orchestra.run.smithi092.stderr:Inferring config /var/lib/ceph/00db0016-dfed-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T21:38:22.049 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:38:22.050 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:38:22.051 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:38:22.152 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:38:22.177 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:38:22.177 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:38:22.178 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:38:22.279 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:38:22.307 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:38:22.307 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:38:22.308 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:38:22.408 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:38:22.435 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:38:22.436 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:38:22.436 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:38:22.537 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:38:22.568 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:38:22.568 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:38:22.569 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:38:22.577 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:38:22 smithi097 ceph-mon[33317]: pgmap v591: 148 pgs: 5 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+degraded+remapped, 140 active+clean; 4.7 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 853 B/s rd, 0 B/s wr, 678 op/s; 27048/230574 objects degraded (11.731%); 9.2 MiB/s, 147 objects/s recovering 2024-03-11T21:38:22.577 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:38:22 smithi097 ceph-mon[33317]: osdmap e316: 8 total, 8 up, 8 in 2024-03-11T21:38:22.577 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:38:22 smithi097 ceph-mon[33317]: from='mgr.14607 ' entity='mgr.y' 2024-03-11T21:38:22.577 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:38:22 smithi097 ceph-mon[33317]: from='mgr.14607 v1:172.21.15.92:0/1702717597' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T21:38:22.670 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:38:22.698 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:38:22.699 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:38:22.699 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:38:22.779 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:38:22 smithi092 ceph-mon[29572]: pgmap v591: 148 pgs: 5 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+degraded+remapped, 140 active+clean; 4.7 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 853 B/s rd, 0 B/s wr, 678 op/s; 27048/230574 objects degraded (11.731%); 9.2 MiB/s, 147 objects/s recovering 2024-03-11T21:38:22.779 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:38:22 smithi092 ceph-mon[29572]: osdmap e316: 8 total, 8 up, 8 in 2024-03-11T21:38:22.779 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:38:22 smithi092 ceph-mon[29572]: from='mgr.14607 ' entity='mgr.y' 2024-03-11T21:38:22.779 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:38:22 smithi092 ceph-mon[29572]: from='mgr.14607 v1:172.21.15.92:0/1702717597' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T21:38:22.779 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:38:22 smithi092 ceph-mon[37563]: pgmap v591: 148 pgs: 5 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+degraded+remapped, 140 active+clean; 4.7 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 853 B/s rd, 0 B/s wr, 678 op/s; 27048/230574 objects degraded (11.731%); 9.2 MiB/s, 147 objects/s recovering 2024-03-11T21:38:22.779 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:38:22 smithi092 ceph-mon[37563]: osdmap e316: 8 total, 8 up, 8 in 2024-03-11T21:38:22.780 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:38:22 smithi092 ceph-mon[37563]: from='mgr.14607 ' entity='mgr.y' 2024-03-11T21:38:22.780 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:38:22 smithi092 ceph-mon[37563]: from='mgr.14607 v1:172.21.15.92:0/1702717597' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T21:38:22.800 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:38:22.837 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:38:22.838 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:38:22.838 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:38:22.902 INFO:teuthology.orchestra.run.smithi097.stdout:{ 2024-03-11T21:38:22.902 INFO:teuthology.orchestra.run.smithi097.stdout: "size": 20, 2024-03-11T21:38:22.902 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 600, 2024-03-11T21:38:22.902 INFO:teuthology.orchestra.run.smithi097.stdout: "ops": [ 2024-03-11T21:38:22.902 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:38:22.902 INFO:teuthology.orchestra.run.smithi097.stdout: "description": "osd_op(client.15150.0:206884 8.8 8:168db5dc:::benchmark_data_smithi092_104865_object86383:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e236)", 2024-03-11T21:38:22.902 INFO:teuthology.orchestra.run.smithi097.stdout: "initiated_at": "2024-03-11T21:34:54.365485+0000", 2024-03-11T21:38:22.903 INFO:teuthology.orchestra.run.smithi097.stdout: "age": 207.894710374, 2024-03-11T21:38:22.903 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.99267883099999998, 2024-03-11T21:38:22.903 INFO:teuthology.orchestra.run.smithi097.stdout: "type_data": { 2024-03-11T21:38:22.903 INFO:teuthology.orchestra.run.smithi097.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-11T21:38:22.903 INFO:teuthology.orchestra.run.smithi097.stdout: "client_info": { 2024-03-11T21:38:22.903 INFO:teuthology.orchestra.run.smithi097.stdout: "client": "client.15150", 2024-03-11T21:38:22.903 INFO:teuthology.orchestra.run.smithi097.stdout: "client_addr": "v1:172.21.15.92:0/436606717", 2024-03-11T21:38:22.903 INFO:teuthology.orchestra.run.smithi097.stdout: "tid": 206884 2024-03-11T21:38:22.903 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:38:22.903 INFO:teuthology.orchestra.run.smithi097.stdout: "events": [ 2024-03-11T21:38:22.903 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:38:22.903 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "initiated", 2024-03-11T21:38:22.903 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:34:54.365485+0000", 2024-03-11T21:38:22.903 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-03-11T21:38:22.904 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:38:22.904 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:38:22.904 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "header_read", 2024-03-11T21:38:22.904 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:34:54.365485+0000", 2024-03-11T21:38:22.904 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-03-11T21:38:22.904 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:38:22.904 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:38:22.904 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "throttled", 2024-03-11T21:38:22.904 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:34:54.365486+0000", 2024-03-11T21:38:22.904 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.4160000000000001e-06 2024-03-11T21:38:22.904 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:38:22.904 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:38:22.904 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "all_read", 2024-03-11T21:38:22.904 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:34:54.365490+0000", 2024-03-11T21:38:22.904 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 4.0559999999999998e-06 2024-03-11T21:38:22.905 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:38:22.905 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:38:22.905 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "dispatched", 2024-03-11T21:38:22.905 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:34:54.365499+0000", 2024-03-11T21:38:22.905 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 8.7469999999999994e-06 2024-03-11T21:38:22.905 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:38:22.905 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:38:22.905 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "queued_for_pg", 2024-03-11T21:38:22.905 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:34:54.365504+0000", 2024-03-11T21:38:22.905 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 5.446e-06 2024-03-11T21:38:22.905 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:38:22.905 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:38:22.905 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "reached_pg", 2024-03-11T21:38:22.905 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:34:54.365546+0000", 2024-03-11T21:38:22.905 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 4.1978999999999999e-05 2024-03-11T21:38:22.905 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:38:22.906 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:38:22.906 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "waiting for peered", 2024-03-11T21:38:22.906 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:34:54.365552+0000", 2024-03-11T21:38:22.906 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 6.2720000000000003e-06 2024-03-11T21:38:22.906 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:38:22.906 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:38:22.906 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "reached_pg", 2024-03-11T21:38:22.906 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:34:55.351250+0000", 2024-03-11T21:38:22.906 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.98569733100000001 2024-03-11T21:38:22.906 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:38:22.906 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:38:22.906 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "started", 2024-03-11T21:38:22.906 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:34:55.351327+0000", 2024-03-11T21:38:22.906 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 7.6753000000000002e-05 2024-03-11T21:38:22.907 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:38:22.907 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:38:22.907 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "waiting for subops from 3,7", 2024-03-11T21:38:22.907 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:34:55.351685+0000", 2024-03-11T21:38:22.907 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.00035823799999999998 2024-03-11T21:38:22.907 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:38:22.907 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:38:22.907 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "sub_op_commit_rec", 2024-03-11T21:38:22.907 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:34:55.354118+0000", 2024-03-11T21:38:22.907 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.0024329680000000002 2024-03-11T21:38:22.907 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:38:22.907 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:38:22.907 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "op_commit", 2024-03-11T21:38:22.907 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:34:55.356229+0000", 2024-03-11T21:38:22.907 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.0021114549999999999 2024-03-11T21:38:22.907 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:38:22.908 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:38:22.908 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "sub_op_commit_rec", 2024-03-11T21:38:22.908 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:34:55.358065+0000", 2024-03-11T21:38:22.908 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.001836166 2024-03-11T21:38:22.908 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:38:22.908 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:38:22.908 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "commit_sent", 2024-03-11T21:38:22.908 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:34:55.358123+0000", 2024-03-11T21:38:22.908 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 5.7763999999999998e-05 2024-03-11T21:38:22.908 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:38:22.908 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:38:22.908 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "done", 2024-03-11T21:38:22.908 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:34:55.358163+0000", 2024-03-11T21:38:22.908 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 4.0240000000000001e-05 2024-03-11T21:38:22.908 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-03-11T21:38:22.908 INFO:teuthology.orchestra.run.smithi097.stdout: ] 2024-03-11T21:38:22.909 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-03-11T21:38:22.909 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:38:22.909 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:38:22.909 INFO:teuthology.orchestra.run.smithi097.stdout: "description": "osd_op(client.15150.0:206893 8.8 8:12ac760f:::benchmark_data_smithi092_104865_object86392:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e236)", 2024-03-11T21:38:22.909 INFO:teuthology.orchestra.run.smithi097.stdout: "initiated_at": "2024-03-11T21:34:54.372974+0000", 2024-03-11T21:38:22.909 INFO:teuthology.orchestra.run.smithi097.stdout: "age": 207.88722044799999, 2024-03-11T21:38:22.909 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.98534399299999997, 2024-03-11T21:38:22.909 INFO:teuthology.orchestra.run.smithi097.stdout: "type_data": { 2024-03-11T21:38:22.909 INFO:teuthology.orchestra.run.smithi097.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-11T21:38:22.909 INFO:teuthology.orchestra.run.smithi097.stdout: "client_info": { 2024-03-11T21:38:22.909 INFO:teuthology.orchestra.run.smithi097.stdout: "client": "client.15150", 2024-03-11T21:38:22.909 INFO:teuthology.orchestra.run.smithi097.stdout: "client_addr": "v1:172.21.15.92:0/436606717", 2024-03-11T21:38:22.909 INFO:teuthology.orchestra.run.smithi097.stdout: "tid": 206893 2024-03-11T21:38:22.909 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:38:22.909 INFO:teuthology.orchestra.run.smithi097.stdout: "events": [ 2024-03-11T21:38:22.909 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:38:22.910 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "initiated", 2024-03-11T21:38:22.910 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:34:54.372974+0000", 2024-03-11T21:38:22.910 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-03-11T21:38:22.910 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:38:22.910 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:38:22.910 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "header_read", 2024-03-11T21:38:22.910 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:34:54.372974+0000", 2024-03-11T21:38:22.910 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-03-11T21:38:22.910 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:38:22.910 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:38:22.910 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "throttled", 2024-03-11T21:38:22.910 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:34:54.372977+0000", 2024-03-11T21:38:22.910 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 2.1710000000000001e-06 2024-03-11T21:38:22.911 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:38:22.911 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:38:22.911 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "all_read", 2024-03-11T21:38:22.911 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:34:54.372982+0000", 2024-03-11T21:38:22.911 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 5.789e-06 2024-03-11T21:38:22.911 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:38:22.911 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:38:22.911 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "dispatched", 2024-03-11T21:38:22.911 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:34:54.372994+0000", 2024-03-11T21:38:22.911 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.1392e-05 2024-03-11T21:38:22.911 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:38:22.911 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:38:22.911 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "queued_for_pg", 2024-03-11T21:38:22.911 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:34:54.373012+0000", 2024-03-11T21:38:22.912 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.8196999999999999e-05 2024-03-11T21:38:22.912 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:38:22.912 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:38:22.912 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "reached_pg", 2024-03-11T21:38:22.912 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:34:54.373152+0000", 2024-03-11T21:38:22.912 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.00013991399999999999 2024-03-11T21:38:22.912 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:38:22.912 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:38:22.912 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "waiting for peered", 2024-03-11T21:38:22.912 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:34:54.373160+0000", 2024-03-11T21:38:22.912 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 8.4700000000000002e-06 2024-03-11T21:38:22.912 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:38:22.912 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:38:22.912 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "reached_pg", 2024-03-11T21:38:22.912 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:34:55.354162+0000", 2024-03-11T21:38:22.913 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.98100120700000004 2024-03-11T21:38:22.913 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:38:22.913 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:38:22.913 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "started", 2024-03-11T21:38:22.913 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:34:55.354284+0000", 2024-03-11T21:38:22.913 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.000122342 2024-03-11T21:38:22.913 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:38:22.913 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:38:22.913 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "waiting for subops from 3,7", 2024-03-11T21:38:22.913 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:34:55.354549+0000", 2024-03-11T21:38:22.913 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.00026506199999999999 2024-03-11T21:38:22.913 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:38:22.913 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:38:22.913 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "sub_op_commit_rec", 2024-03-11T21:38:22.913 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:34:55.358238+0000", 2024-03-11T21:38:22.914 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.0036888390000000002 2024-03-11T21:38:22.914 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:38:22.914 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:38:22.914 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "sub_op_commit_rec", 2024-03-11T21:38:22.914 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:34:55.358267+0000", 2024-03-11T21:38:22.914 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 2.8858999999999999e-05 2024-03-11T21:38:22.914 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:38:22.914 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:38:22.914 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "op_commit", 2024-03-11T21:38:22.914 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:34:55.358274+0000", 2024-03-11T21:38:22.914 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 7.3629999999999996e-06 2024-03-11T21:38:22.914 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:38:22.914 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:38:22.914 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "commit_sent", 2024-03-11T21:38:22.915 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:34:55.358299+0000", 2024-03-11T21:38:22.915 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 2.4612000000000001e-05 2024-03-11T21:38:22.915 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:38:22.915 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:38:22.915 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "done", 2024-03-11T21:38:22.915 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:34:55.358318+0000", 2024-03-11T21:38:22.915 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.9775999999999998e-05 2024-03-11T21:38:22.915 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-03-11T21:38:22.915 INFO:teuthology.orchestra.run.smithi097.stdout: ] 2024-03-11T21:38:22.915 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-03-11T21:38:22.915 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:38:22.915 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:38:22.915 INFO:teuthology.orchestra.run.smithi097.stdout: "description": "osd_op(client.15150.0:206901 8.8 8:1351d222:::benchmark_data_smithi092_104865_object86400:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e236)", 2024-03-11T21:38:22.915 INFO:teuthology.orchestra.run.smithi097.stdout: "initiated_at": "2024-03-11T21:34:54.379416+0000", 2024-03-11T21:38:22.916 INFO:teuthology.orchestra.run.smithi097.stdout: "age": 207.880779026, 2024-03-11T21:38:22.916 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.979003124, 2024-03-11T21:38:22.916 INFO:teuthology.orchestra.run.smithi097.stdout: "type_data": { 2024-03-11T21:38:22.916 INFO:teuthology.orchestra.run.smithi097.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-11T21:38:22.916 INFO:teuthology.orchestra.run.smithi097.stdout: "client_info": { 2024-03-11T21:38:22.916 INFO:teuthology.orchestra.run.smithi097.stdout: "client": "client.15150", 2024-03-11T21:38:22.916 INFO:teuthology.orchestra.run.smithi097.stdout: "client_addr": "v1:172.21.15.92:0/436606717", 2024-03-11T21:38:22.916 INFO:teuthology.orchestra.run.smithi097.stdout: "tid": 206901 2024-03-11T21:38:22.916 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:38:22.916 INFO:teuthology.orchestra.run.smithi097.stdout: "events": [ 2024-03-11T21:38:22.916 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:38:22.916 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "initiated", 2024-03-11T21:38:22.916 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:34:54.379416+0000", 2024-03-11T21:38:22.916 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-03-11T21:38:22.916 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:38:22.917 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:38:22.917 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "header_read", 2024-03-11T21:38:22.917 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:34:54.379416+0000", 2024-03-11T21:38:22.917 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-03-11T21:38:22.917 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:38:22.917 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:38:22.917 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "throttled", 2024-03-11T21:38:22.917 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:34:54.379418+0000", 2024-03-11T21:38:22.917 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.8160000000000001e-06 2024-03-11T21:38:22.917 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:38:22.917 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:38:22.917 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "all_read", 2024-03-11T21:38:22.917 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:34:54.379423+0000", 2024-03-11T21:38:22.917 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 5.4439999999999999e-06 2024-03-11T21:38:22.917 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:38:22.917 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:38:22.918 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "dispatched", 2024-03-11T21:38:22.918 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:34:54.379436+0000", 2024-03-11T21:38:22.918 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.3278000000000001e-05 2024-03-11T21:38:22.918 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:38:22.918 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:38:22.918 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "queued_for_pg", 2024-03-11T21:38:22.918 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:34:54.379444+0000", 2024-03-11T21:38:22.918 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 7.7719999999999994e-06 2024-03-11T21:38:22.918 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:38:22.918 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:38:22.918 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "reached_pg", 2024-03-11T21:38:22.918 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:34:54.379503+0000", 2024-03-11T21:38:22.918 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 5.9218000000000001e-05 2024-03-11T21:38:22.918 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:38:22.919 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:38:22.919 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "waiting for peered", 2024-03-11T21:38:22.919 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:34:54.379512+0000", 2024-03-11T21:38:22.919 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 8.1170000000000004e-06 2024-03-11T21:38:22.919 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:38:22.919 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:38:22.919 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "reached_pg", 2024-03-11T21:38:22.919 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:34:55.356273+0000", 2024-03-11T21:38:22.919 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.97676143999999998 2024-03-11T21:38:22.919 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:38:22.919 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:38:22.919 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "started", 2024-03-11T21:38:22.919 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:34:55.356383+0000", 2024-03-11T21:38:22.919 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.000109719 2024-03-11T21:38:22.920 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:38:22.920 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:38:22.920 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "waiting for subops from 3,7", 2024-03-11T21:38:22.920 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:34:55.356585+0000", 2024-03-11T21:38:22.920 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.000202645 2024-03-11T21:38:22.920 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:38:22.920 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:38:22.920 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "op_commit", 2024-03-11T21:38:22.920 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:34:55.358324+0000", 2024-03-11T21:38:22.920 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.001738442 2024-03-11T21:38:22.920 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:38:22.920 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:38:22.920 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "sub_op_commit_rec", 2024-03-11T21:38:22.921 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:34:55.358350+0000", 2024-03-11T21:38:22.921 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 2.5803999999999999e-05 2024-03-11T21:38:22.921 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:38:22.921 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:38:22.921 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "sub_op_commit_rec", 2024-03-11T21:38:22.921 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:34:55.358378+0000", 2024-03-11T21:38:22.921 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 2.8710000000000001e-05 2024-03-11T21:38:22.921 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:38:22.921 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:38:22.921 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "commit_sent", 2024-03-11T21:38:22.921 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:34:55.358402+0000", 2024-03-11T21:38:22.921 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 2.3473999999999999e-05 2024-03-11T21:38:22.921 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:38:22.921 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:38:22.922 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "done", 2024-03-11T21:38:22.922 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:34:55.358419+0000", 2024-03-11T21:38:22.922 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.7244999999999998e-05 2024-03-11T21:38:22.922 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-03-11T21:38:22.922 INFO:teuthology.orchestra.run.smithi097.stdout: ] 2024-03-11T21:38:22.922 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-03-11T21:38:22.922 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:38:22.922 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:38:22.922 INFO:teuthology.orchestra.run.smithi097.stdout: "description": "osd_op(client.15150.0:206908 8.8 8:15831f13:::benchmark_data_smithi092_104865_object86407:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e236)", 2024-03-11T21:38:22.922 INFO:teuthology.orchestra.run.smithi097.stdout: "initiated_at": "2024-03-11T21:34:54.388435+0000", 2024-03-11T21:38:22.922 INFO:teuthology.orchestra.run.smithi097.stdout: "age": 207.87175983099999, 2024-03-11T21:38:22.922 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.97410122300000002, 2024-03-11T21:38:22.922 INFO:teuthology.orchestra.run.smithi097.stdout: "type_data": { 2024-03-11T21:38:22.922 INFO:teuthology.orchestra.run.smithi097.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-11T21:38:22.923 INFO:teuthology.orchestra.run.smithi097.stdout: "client_info": { 2024-03-11T21:38:22.923 INFO:teuthology.orchestra.run.smithi097.stdout: "client": "client.15150", 2024-03-11T21:38:22.923 INFO:teuthology.orchestra.run.smithi097.stdout: "client_addr": "v1:172.21.15.92:0/436606717", 2024-03-11T21:38:22.923 INFO:teuthology.orchestra.run.smithi097.stdout: "tid": 206908 2024-03-11T21:38:22.923 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:38:22.923 INFO:teuthology.orchestra.run.smithi097.stdout: "events": [ 2024-03-11T21:38:22.923 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:38:22.923 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "initiated", 2024-03-11T21:38:22.923 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:34:54.388435+0000", 2024-03-11T21:38:22.923 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-03-11T21:38:22.923 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:38:22.923 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:38:22.923 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "header_read", 2024-03-11T21:38:22.923 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:34:54.388435+0000", 2024-03-11T21:38:22.923 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-03-11T21:38:22.924 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:38:22.924 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:38:22.924 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "throttled", 2024-03-11T21:38:22.924 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:34:54.388437+0000", 2024-03-11T21:38:22.924 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.469e-06 2024-03-11T21:38:22.924 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:38:22.924 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:38:22.924 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "all_read", 2024-03-11T21:38:22.924 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:34:54.388441+0000", 2024-03-11T21:38:22.924 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 4.6079999999999998e-06 2024-03-11T21:38:22.924 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:38:22.924 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:38:22.924 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "dispatched", 2024-03-11T21:38:22.924 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:34:54.388451+0000", 2024-03-11T21:38:22.924 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 9.7459999999999995e-06 2024-03-11T21:38:22.925 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:38:22.925 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:38:22.925 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "queued_for_pg", 2024-03-11T21:38:22.925 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:34:54.388459+0000", 2024-03-11T21:38:22.925 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 7.6240000000000002e-06 2024-03-11T21:38:22.925 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:38:22.925 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:38:22.925 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "reached_pg", 2024-03-11T21:38:22.925 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:34:54.388573+0000", 2024-03-11T21:38:22.925 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.00011455699999999999 2024-03-11T21:38:22.925 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:38:22.925 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:38:22.925 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "waiting for peered", 2024-03-11T21:38:22.925 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:34:54.388582+0000", 2024-03-11T21:38:22.926 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 8.6629999999999999e-06 2024-03-11T21:38:22.926 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:38:22.926 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:38:22.926 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "reached_pg", 2024-03-11T21:38:22.926 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:34:55.358443+0000", 2024-03-11T21:38:22.926 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.96986088999999998 2024-03-11T21:38:22.926 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:38:22.926 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:38:22.926 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "started", 2024-03-11T21:38:22.926 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:34:55.358510+0000", 2024-03-11T21:38:22.926 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 6.7407e-05 2024-03-11T21:38:22.926 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:38:22.926 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:38:22.926 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "waiting for subops from 3,7", 2024-03-11T21:38:22.926 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:34:55.358721+0000", 2024-03-11T21:38:22.927 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.00021080599999999999 2024-03-11T21:38:22.927 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:38:22.927 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:38:22.927 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "sub_op_commit_rec", 2024-03-11T21:38:22.927 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:34:55.362390+0000", 2024-03-11T21:38:22.927 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.0036691190000000002 2024-03-11T21:38:22.927 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:38:22.927 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:38:22.927 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "op_commit", 2024-03-11T21:38:22.927 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:34:55.362407+0000", 2024-03-11T21:38:22.927 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.6813999999999999e-05 2024-03-11T21:38:22.927 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:38:22.927 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:38:22.927 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "sub_op_commit_rec", 2024-03-11T21:38:22.927 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:34:55.362445+0000", 2024-03-11T21:38:22.928 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 3.8012999999999997e-05 2024-03-11T21:38:22.928 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:38:22.928 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:38:22.928 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "commit_sent", 2024-03-11T21:38:22.928 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:34:55.362493+0000", 2024-03-11T21:38:22.928 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 4.7759999999999997e-05 2024-03-11T21:38:22.928 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:38:22.928 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:38:22.928 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "done", 2024-03-11T21:38:22.928 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:34:55.362536+0000", 2024-03-11T21:38:22.928 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 4.3747e-05 2024-03-11T21:38:22.928 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-03-11T21:38:22.928 INFO:teuthology.orchestra.run.smithi097.stdout: ] 2024-03-11T21:38:22.928 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-03-11T21:38:22.928 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:38:22.929 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:38:22.929 INFO:teuthology.orchestra.run.smithi097.stdout: "description": "osd_op(client.15150.0:206922 8.8 8:17e36640:::benchmark_data_smithi092_104865_object86421:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e236)", 2024-03-11T21:38:22.929 INFO:teuthology.orchestra.run.smithi097.stdout: "initiated_at": "2024-03-11T21:34:54.400598+0000", 2024-03-11T21:38:22.929 INFO:teuthology.orchestra.run.smithi097.stdout: "age": 207.859596439, 2024-03-11T21:38:22.929 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.96768579600000004, 2024-03-11T21:38:22.929 INFO:teuthology.orchestra.run.smithi097.stdout: "type_data": { 2024-03-11T21:38:22.929 INFO:teuthology.orchestra.run.smithi097.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-11T21:38:22.929 INFO:teuthology.orchestra.run.smithi097.stdout: "client_info": { 2024-03-11T21:38:22.929 INFO:teuthology.orchestra.run.smithi097.stdout: "client": "client.15150", 2024-03-11T21:38:22.929 INFO:teuthology.orchestra.run.smithi097.stdout: "client_addr": "v1:172.21.15.92:0/436606717", 2024-03-11T21:38:22.929 INFO:teuthology.orchestra.run.smithi097.stdout: "tid": 206922 2024-03-11T21:38:22.929 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:38:22.929 INFO:teuthology.orchestra.run.smithi097.stdout: "events": [ 2024-03-11T21:38:22.929 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:38:22.930 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "initiated", 2024-03-11T21:38:22.930 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:34:54.400598+0000", 2024-03-11T21:38:22.930 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-03-11T21:38:22.930 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:38:22.930 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:38:22.930 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "header_read", 2024-03-11T21:38:22.930 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:34:54.400598+0000", 2024-03-11T21:38:22.930 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-03-11T21:38:22.930 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:38:22.930 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:38:22.930 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "throttled", 2024-03-11T21:38:22.930 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:34:54.400600+0000", 2024-03-11T21:38:22.930 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.4300000000000001e-06 2024-03-11T21:38:22.931 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:38:22.931 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:38:22.931 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "all_read", 2024-03-11T21:38:22.931 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:34:54.400605+0000", 2024-03-11T21:38:22.931 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 4.7720000000000002e-06 2024-03-11T21:38:22.931 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:38:22.931 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:38:22.931 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "dispatched", 2024-03-11T21:38:22.931 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:34:54.400619+0000", 2024-03-11T21:38:22.931 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.4229e-05 2024-03-11T21:38:22.931 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:38:22.931 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:38:22.931 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "queued_for_pg", 2024-03-11T21:38:22.931 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:34:54.400627+0000", 2024-03-11T21:38:22.931 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 7.8080000000000005e-06 2024-03-11T21:38:22.932 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:38:22.932 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:38:22.932 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "reached_pg", 2024-03-11T21:38:22.932 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:34:54.400721+0000", 2024-03-11T21:38:22.932 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 9.4081999999999998e-05 2024-03-11T21:38:22.932 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:38:22.932 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:38:22.932 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "waiting for peered", 2024-03-11T21:38:22.932 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:34:54.400730+0000", 2024-03-11T21:38:22.932 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 8.9479999999999997e-06 2024-03-11T21:38:22.932 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:38:22.932 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:38:22.932 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "reached_pg", 2024-03-11T21:38:22.932 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:34:55.360433+0000", 2024-03-11T21:38:22.933 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.959702995 2024-03-11T21:38:22.933 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:38:22.933 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:38:22.933 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "started", 2024-03-11T21:38:22.933 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:34:55.360519+0000", 2024-03-11T21:38:22.933 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 8.5859999999999994e-05 2024-03-11T21:38:22.933 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:38:22.933 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:38:22.933 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "waiting for subops from 3,7", 2024-03-11T21:38:22.933 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:34:55.360733+0000", 2024-03-11T21:38:22.933 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.00021452299999999999 2024-03-11T21:38:22.933 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:38:22.933 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:38:22.933 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "sub_op_commit_rec", 2024-03-11T21:38:22.934 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:34:55.368055+0000", 2024-03-11T21:38:22.934 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.0073219779999999998 2024-03-11T21:38:22.934 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:38:22.934 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:38:22.934 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "sub_op_commit_rec", 2024-03-11T21:38:22.934 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:34:55.368122+0000", 2024-03-11T21:38:22.934 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 6.6501000000000006e-05 2024-03-11T21:38:22.934 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:38:22.934 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:38:22.934 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "op_commit", 2024-03-11T21:38:22.934 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:34:55.368185+0000", 2024-03-11T21:38:22.934 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 6.3297000000000001e-05 2024-03-11T21:38:22.934 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:38:22.934 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:38:22.934 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "commit_sent", 2024-03-11T21:38:22.935 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:34:55.368241+0000", 2024-03-11T21:38:22.935 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 5.5813000000000002e-05 2024-03-11T21:38:22.935 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:38:22.935 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:38:22.935 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "done", 2024-03-11T21:38:22.935 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:34:55.368284+0000", 2024-03-11T21:38:22.935 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 4.3560000000000003e-05 2024-03-11T21:38:22.935 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-03-11T21:38:22.935 INFO:teuthology.orchestra.run.smithi097.stdout: ] 2024-03-11T21:38:22.935 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-03-11T21:38:22.935 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:38:22.935 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:38:22.935 INFO:teuthology.orchestra.run.smithi097.stdout: "description": "osd_op(client.15150.0:208574 8.d 8:b4f0a13a:::benchmark_data_smithi092_104865_object88073:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e238)", 2024-03-11T21:38:22.935 INFO:teuthology.orchestra.run.smithi097.stdout: "initiated_at": "2024-03-11T21:34:57.352453+0000", 2024-03-11T21:38:22.936 INFO:teuthology.orchestra.run.smithi097.stdout: "age": 204.907741614, 2024-03-11T21:38:22.936 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.0319107139999999, 2024-03-11T21:38:22.936 INFO:teuthology.orchestra.run.smithi097.stdout: "type_data": { 2024-03-11T21:38:22.936 INFO:teuthology.orchestra.run.smithi097.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-11T21:38:22.936 INFO:teuthology.orchestra.run.smithi097.stdout: "client_info": { 2024-03-11T21:38:22.936 INFO:teuthology.orchestra.run.smithi097.stdout: "client": "client.15150", 2024-03-11T21:38:22.936 INFO:teuthology.orchestra.run.smithi097.stdout: "client_addr": "v1:172.21.15.92:0/436606717", 2024-03-11T21:38:22.936 INFO:teuthology.orchestra.run.smithi097.stdout: "tid": 208574 2024-03-11T21:38:22.936 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:38:22.936 INFO:teuthology.orchestra.run.smithi097.stdout: "events": [ 2024-03-11T21:38:22.936 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:38:22.936 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "initiated", 2024-03-11T21:38:22.936 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:34:57.352453+0000", 2024-03-11T21:38:22.936 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-03-11T21:38:22.936 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:38:22.937 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:38:22.937 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "header_read", 2024-03-11T21:38:22.937 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:34:57.352453+0000", 2024-03-11T21:38:22.937 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-03-11T21:38:22.937 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:38:22.937 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:38:22.937 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "throttled", 2024-03-11T21:38:22.937 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:34:57.352455+0000", 2024-03-11T21:38:22.937 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 2.0600000000000002e-06 2024-03-11T21:38:22.937 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:38:22.937 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:38:22.937 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "all_read", 2024-03-11T21:38:22.937 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:34:57.352462+0000", 2024-03-11T21:38:22.937 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 6.229e-06 2024-03-11T21:38:22.937 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:38:22.938 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:38:22.938 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "dispatched", 2024-03-11T21:38:22.938 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:34:57.353033+0000", 2024-03-11T21:38:22.938 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.00057142200000000003 2024-03-11T21:38:22.938 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:38:22.938 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:38:22.938 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "queued_for_pg", 2024-03-11T21:38:22.938 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:34:57.353041+0000", 2024-03-11T21:38:22.938 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 8.4379999999999995e-06 2024-03-11T21:38:22.938 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:38:22.938 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:38:22.938 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "reached_pg", 2024-03-11T21:38:22.938 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:34:57.353352+0000", 2024-03-11T21:38:22.938 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.00031082400000000002 2024-03-11T21:38:22.939 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:38:22.939 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:38:22.939 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "waiting for peered", 2024-03-11T21:38:22.939 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:34:57.353366+0000", 2024-03-11T21:38:22.939 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.4037e-05 2024-03-11T21:38:22.939 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:38:22.939 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:38:22.939 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "reached_pg", 2024-03-11T21:38:22.939 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:34:58.381460+0000", 2024-03-11T21:38:22.939 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.0280934779999999 2024-03-11T21:38:22.939 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:38:22.939 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:38:22.939 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "started", 2024-03-11T21:38:22.939 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:34:58.381562+0000", 2024-03-11T21:38:22.939 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.000102308 2024-03-11T21:38:22.940 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:38:22.940 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:38:22.940 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "waiting for subops from 2,7", 2024-03-11T21:38:22.940 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:34:58.381651+0000", 2024-03-11T21:38:22.940 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 8.9386000000000001e-05 2024-03-11T21:38:22.940 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:38:22.940 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:38:22.940 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "sub_op_commit_rec", 2024-03-11T21:38:22.940 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:34:58.384224+0000", 2024-03-11T21:38:22.940 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.0025727839999999998 2024-03-11T21:38:22.940 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:38:22.940 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:38:22.940 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "sub_op_commit_rec", 2024-03-11T21:38:22.940 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:34:58.384274+0000", 2024-03-11T21:38:22.941 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 4.994e-05 2024-03-11T21:38:22.941 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:38:22.941 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:38:22.941 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "op_commit", 2024-03-11T21:38:22.941 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:34:58.384287+0000", 2024-03-11T21:38:22.941 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.2376e-05 2024-03-11T21:38:22.941 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:38:22.941 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:38:22.941 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "commit_sent", 2024-03-11T21:38:22.941 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:34:58.384318+0000", 2024-03-11T21:38:22.941 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 3.1905e-05 2024-03-11T21:38:22.941 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:38:22.941 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:38:22.941 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "done", 2024-03-11T21:38:22.942 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:34:58.384364+0000", 2024-03-11T21:38:22.942 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 4.5527e-05 2024-03-11T21:38:22.942 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-03-11T21:38:22.942 INFO:teuthology.orchestra.run.smithi097.stdout: ] 2024-03-11T21:38:22.942 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-03-11T21:38:22.942 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:38:22.942 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:38:22.942 INFO:teuthology.orchestra.run.smithi097.stdout: "description": "osd_op(client.15150.0:208578 8.d 8:b8981014:::benchmark_data_smithi092_104865_object88077:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e239)", 2024-03-11T21:38:22.942 INFO:teuthology.orchestra.run.smithi097.stdout: "initiated_at": "2024-03-11T21:34:57.356717+0000", 2024-03-11T21:38:22.942 INFO:teuthology.orchestra.run.smithi097.stdout: "age": 204.90347750000001, 2024-03-11T21:38:22.942 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.0313246620000001, 2024-03-11T21:38:22.942 INFO:teuthology.orchestra.run.smithi097.stdout: "type_data": { 2024-03-11T21:38:22.942 INFO:teuthology.orchestra.run.smithi097.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-11T21:38:22.943 INFO:teuthology.orchestra.run.smithi097.stdout: "client_info": { 2024-03-11T21:38:22.943 INFO:teuthology.orchestra.run.smithi097.stdout: "client": "client.15150", 2024-03-11T21:38:22.943 INFO:teuthology.orchestra.run.smithi097.stdout: "client_addr": "v1:172.21.15.92:0/436606717", 2024-03-11T21:38:22.943 INFO:teuthology.orchestra.run.smithi097.stdout: "tid": 208578 2024-03-11T21:38:22.943 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:38:22.943 INFO:teuthology.orchestra.run.smithi097.stdout: "events": [ 2024-03-11T21:38:22.943 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:38:22.943 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "initiated", 2024-03-11T21:38:22.943 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:34:57.356717+0000", 2024-03-11T21:38:22.943 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-03-11T21:38:22.943 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:38:22.943 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:38:22.943 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "header_read", 2024-03-11T21:38:22.943 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:34:57.356717+0000", 2024-03-11T21:38:22.944 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-03-11T21:38:22.944 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:38:22.944 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:38:22.944 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "throttled", 2024-03-11T21:38:22.944 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:34:57.356719+0000", 2024-03-11T21:38:22.944 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.866e-06 2024-03-11T21:38:22.944 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:38:22.944 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:38:22.944 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "all_read", 2024-03-11T21:38:22.944 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:34:57.356724+0000", 2024-03-11T21:38:22.944 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 5.1909999999999999e-06 2024-03-11T21:38:22.944 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:38:22.944 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:38:22.944 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "dispatched", 2024-03-11T21:38:22.945 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:34:57.356736+0000", 2024-03-11T21:38:22.945 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.1231e-05 2024-03-11T21:38:22.945 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:38:22.945 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:38:22.945 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "queued_for_pg", 2024-03-11T21:38:22.945 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:34:57.356743+0000", 2024-03-11T21:38:22.945 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 7.5830000000000001e-06 2024-03-11T21:38:22.945 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:38:22.945 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:38:22.945 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "reached_pg", 2024-03-11T21:38:22.945 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:34:57.356802+0000", 2024-03-11T21:38:22.945 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 5.9107999999999997e-05 2024-03-11T21:38:22.945 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:38:22.945 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:38:22.945 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "waiting for peered", 2024-03-11T21:38:22.946 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:34:57.356811+0000", 2024-03-11T21:38:22.946 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 8.8869999999999998e-06 2024-03-11T21:38:22.946 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:38:22.946 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:38:22.946 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "reached_pg", 2024-03-11T21:38:22.946 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:34:58.384395+0000", 2024-03-11T21:38:22.946 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.0275842260000001 2024-03-11T21:38:22.946 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:38:22.946 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:38:22.946 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "started", 2024-03-11T21:38:22.946 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:34:58.384476+0000", 2024-03-11T21:38:22.946 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 8.0187999999999999e-05 2024-03-11T21:38:22.946 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:38:22.946 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:38:22.946 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "waiting for subops from 2,7", 2024-03-11T21:38:22.947 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:34:58.384553+0000", 2024-03-11T21:38:22.947 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 7.6897999999999995e-05 2024-03-11T21:38:22.947 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:38:22.947 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:38:22.947 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "sub_op_commit_rec", 2024-03-11T21:38:22.947 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:34:58.387913+0000", 2024-03-11T21:38:22.947 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.0033606370000000001 2024-03-11T21:38:22.947 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:38:22.947 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:38:22.947 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "sub_op_commit_rec", 2024-03-11T21:38:22.947 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:34:58.387977+0000", 2024-03-11T21:38:22.947 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 6.3377e-05 2024-03-11T21:38:22.947 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:38:22.947 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:38:22.947 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "op_commit", 2024-03-11T21:38:22.948 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:34:58.387987+0000", 2024-03-11T21:38:22.948 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.0173000000000001e-05 2024-03-11T21:38:22.948 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:38:22.948 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:38:22.948 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "commit_sent", 2024-03-11T21:38:22.948 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:34:58.388017+0000", 2024-03-11T21:38:22.948 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 3.0439000000000001e-05 2024-03-11T21:38:22.948 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:38:22.948 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:38:22.948 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "done", 2024-03-11T21:38:22.948 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:34:58.388042+0000", 2024-03-11T21:38:22.948 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 2.4858000000000002e-05 2024-03-11T21:38:22.948 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-03-11T21:38:22.948 INFO:teuthology.orchestra.run.smithi097.stdout: ] 2024-03-11T21:38:22.949 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-03-11T21:38:22.949 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:38:22.949 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:38:22.949 INFO:teuthology.orchestra.run.smithi097.stdout: "description": "osd_op(client.15150.0:208589 8.d 8:b0d80d0e:::benchmark_data_smithi092_104865_object88088:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e239)", 2024-03-11T21:38:22.949 INFO:teuthology.orchestra.run.smithi097.stdout: "initiated_at": "2024-03-11T21:34:57.366673+0000", 2024-03-11T21:38:22.949 INFO:teuthology.orchestra.run.smithi097.stdout: "age": 204.89352143100001, 2024-03-11T21:38:22.949 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.026816476, 2024-03-11T21:38:22.949 INFO:teuthology.orchestra.run.smithi097.stdout: "type_data": { 2024-03-11T21:38:22.949 INFO:teuthology.orchestra.run.smithi097.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-11T21:38:22.949 INFO:teuthology.orchestra.run.smithi097.stdout: "client_info": { 2024-03-11T21:38:22.949 INFO:teuthology.orchestra.run.smithi097.stdout: "client": "client.15150", 2024-03-11T21:38:22.949 INFO:teuthology.orchestra.run.smithi097.stdout: "client_addr": "v1:172.21.15.92:0/436606717", 2024-03-11T21:38:22.949 INFO:teuthology.orchestra.run.smithi097.stdout: "tid": 208589 2024-03-11T21:38:22.949 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:38:22.950 INFO:teuthology.orchestra.run.smithi097.stdout: "events": [ 2024-03-11T21:38:22.950 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:38:22.950 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "initiated", 2024-03-11T21:38:22.950 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:34:57.366673+0000", 2024-03-11T21:38:22.950 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-03-11T21:38:22.950 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:38:22.950 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:38:22.950 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "header_read", 2024-03-11T21:38:22.950 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:34:57.366673+0000", 2024-03-11T21:38:22.950 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-03-11T21:38:22.950 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:38:22.950 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:38:22.950 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "throttled", 2024-03-11T21:38:22.951 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:34:57.366675+0000", 2024-03-11T21:38:22.951 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.567e-06 2024-03-11T21:38:22.951 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:38:22.951 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:38:22.951 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "all_read", 2024-03-11T21:38:22.951 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:34:57.366679+0000", 2024-03-11T21:38:22.951 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 4.4299999999999999e-06 2024-03-11T21:38:22.951 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:38:22.951 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:38:22.951 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "dispatched", 2024-03-11T21:38:22.951 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:34:57.366690+0000", 2024-03-11T21:38:22.951 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.1028e-05 2024-03-11T21:38:22.951 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:38:22.952 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:38:22.952 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "queued_for_pg", 2024-03-11T21:38:22.952 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:34:57.366696+0000", 2024-03-11T21:38:22.952 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 5.8860000000000001e-06 2024-03-11T21:38:22.952 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:38:22.952 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:38:22.952 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "reached_pg", 2024-03-11T21:38:22.952 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:34:57.366752+0000", 2024-03-11T21:38:22.952 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 5.5566999999999998e-05 2024-03-11T21:38:22.952 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:38:22.952 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:38:22.952 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "waiting for peered", 2024-03-11T21:38:22.952 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:34:57.366758+0000", 2024-03-11T21:38:22.953 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 6.2269999999999998e-06 2024-03-11T21:38:22.953 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:38:22.953 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:38:22.953 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "reached_pg", 2024-03-11T21:38:22.953 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:34:58.386291+0000", 2024-03-11T21:38:22.953 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.019533238 2024-03-11T21:38:22.953 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:38:22.953 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:38:22.953 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "started", 2024-03-11T21:38:22.953 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:34:58.386368+0000", 2024-03-11T21:38:22.953 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 7.6126999999999998e-05 2024-03-11T21:38:22.953 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:38:22.953 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:38:22.953 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "waiting for subops from 2,7", 2024-03-11T21:38:22.954 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:34:58.386440+0000", 2024-03-11T21:38:22.954 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 7.2539999999999993e-05 2024-03-11T21:38:22.954 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:38:22.954 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:38:22.954 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "sub_op_commit_rec", 2024-03-11T21:38:22.954 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:34:58.388089+0000", 2024-03-11T21:38:22.954 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.001648722 2024-03-11T21:38:22.954 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:38:22.954 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:38:22.954 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "op_commit", 2024-03-11T21:38:22.954 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:34:58.390342+0000", 2024-03-11T21:38:22.954 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.0022533879999999998 2024-03-11T21:38:22.954 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:38:22.954 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:38:22.955 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "sub_op_commit_rec", 2024-03-11T21:38:22.955 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:34:58.393325+0000", 2024-03-11T21:38:22.955 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.0029828979999999999 2024-03-11T21:38:22.955 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:38:22.955 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:38:22.955 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "commit_sent", 2024-03-11T21:38:22.955 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:34:58.393440+0000", 2024-03-11T21:38:22.955 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.000115418 2024-03-11T21:38:22.955 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:38:22.955 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:38:22.955 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "done", 2024-03-11T21:38:22.955 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:34:58.393490+0000", 2024-03-11T21:38:22.955 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 4.9440000000000001e-05 2024-03-11T21:38:22.956 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-03-11T21:38:22.956 INFO:teuthology.orchestra.run.smithi097.stdout: ] 2024-03-11T21:38:22.956 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-03-11T21:38:22.956 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:38:22.956 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:38:22.956 INFO:teuthology.orchestra.run.smithi097.stdout: "description": "osd_op(client.15150.0:208601 8.d 8:b93ad02e:::benchmark_data_smithi092_104865_object88100:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e239)", 2024-03-11T21:38:22.956 INFO:teuthology.orchestra.run.smithi097.stdout: "initiated_at": "2024-03-11T21:34:57.372084+0000", 2024-03-11T21:38:22.956 INFO:teuthology.orchestra.run.smithi097.stdout: "age": 204.88811055799999, 2024-03-11T21:38:22.956 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.021649343, 2024-03-11T21:38:22.956 INFO:teuthology.orchestra.run.smithi097.stdout: "type_data": { 2024-03-11T21:38:22.956 INFO:teuthology.orchestra.run.smithi097.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-11T21:38:22.956 INFO:teuthology.orchestra.run.smithi097.stdout: "client_info": { 2024-03-11T21:38:22.956 INFO:teuthology.orchestra.run.smithi097.stdout: "client": "client.15150", 2024-03-11T21:38:22.956 INFO:teuthology.orchestra.run.smithi097.stdout: "client_addr": "v1:172.21.15.92:0/436606717", 2024-03-11T21:38:22.956 INFO:teuthology.orchestra.run.smithi097.stdout: "tid": 208601 2024-03-11T21:38:22.957 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:38:22.957 INFO:teuthology.orchestra.run.smithi097.stdout: "events": [ 2024-03-11T21:38:22.957 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:38:22.957 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "initiated", 2024-03-11T21:38:22.957 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:34:57.372084+0000", 2024-03-11T21:38:22.957 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-03-11T21:38:22.957 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:38:22.957 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:38:22.957 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "header_read", 2024-03-11T21:38:22.957 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:34:57.372084+0000", 2024-03-11T21:38:22.957 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-03-11T21:38:22.957 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:38:22.957 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:38:22.958 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "throttled", 2024-03-11T21:38:22.958 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:34:57.372086+0000", 2024-03-11T21:38:22.958 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.5209999999999999e-06 2024-03-11T21:38:22.958 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:38:22.958 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:38:22.958 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "all_read", 2024-03-11T21:38:22.958 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:34:57.372090+0000", 2024-03-11T21:38:22.958 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 4.5909999999999998e-06 2024-03-11T21:38:22.958 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:38:22.958 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:38:22.958 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "dispatched", 2024-03-11T21:38:22.958 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:34:57.372101+0000", 2024-03-11T21:38:22.958 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.0912e-05 2024-03-11T21:38:22.958 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:38:22.959 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:38:22.959 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "queued_for_pg", 2024-03-11T21:38:22.959 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:34:57.372117+0000", 2024-03-11T21:38:22.959 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.5247e-05 2024-03-11T21:38:22.959 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:38:22.959 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:38:22.959 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "reached_pg", 2024-03-11T21:38:22.959 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:34:57.372177+0000", 2024-03-11T21:38:22.959 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 5.9957999999999998e-05 2024-03-11T21:38:22.959 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:38:22.959 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:38:22.959 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "waiting for peered", 2024-03-11T21:38:22.959 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:34:57.372186+0000", 2024-03-11T21:38:22.959 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 9.5750000000000007e-06 2024-03-11T21:38:22.959 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:38:22.959 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:38:22.960 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "reached_pg", 2024-03-11T21:38:22.960 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:34:58.388134+0000", 2024-03-11T21:38:22.960 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.0159478159999999 2024-03-11T21:38:22.960 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:38:22.960 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:38:22.960 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "started", 2024-03-11T21:38:22.960 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:34:58.388202+0000", 2024-03-11T21:38:22.960 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 6.7717000000000002e-05 2024-03-11T21:38:22.960 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:38:22.960 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:38:22.960 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "waiting for subops from 2,7", 2024-03-11T21:38:22.960 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:34:58.388274+0000", 2024-03-11T21:38:22.960 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 7.2342999999999995e-05 2024-03-11T21:38:22.960 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:38:22.960 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:38:22.961 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "sub_op_commit_rec", 2024-03-11T21:38:22.961 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:34:58.393561+0000", 2024-03-11T21:38:22.961 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.0052871180000000004 2024-03-11T21:38:22.961 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:38:22.961 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:38:22.961 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "sub_op_commit_rec", 2024-03-11T21:38:22.961 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:34:58.393612+0000", 2024-03-11T21:38:22.961 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 5.0923e-05 2024-03-11T21:38:22.961 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:38:22.961 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:38:22.961 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "op_commit", 2024-03-11T21:38:22.961 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:34:58.393630+0000", 2024-03-11T21:38:22.961 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.7716000000000001e-05 2024-03-11T21:38:22.961 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:38:22.962 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:38:22.962 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "commit_sent", 2024-03-11T21:38:22.962 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:34:58.393683+0000", 2024-03-11T21:38:22.962 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 5.3656999999999998e-05 2024-03-11T21:38:22.962 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:38:22.962 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:38:22.962 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "done", 2024-03-11T21:38:22.962 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:34:58.393734+0000", 2024-03-11T21:38:22.962 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 5.0249e-05 2024-03-11T21:38:22.962 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-03-11T21:38:22.962 INFO:teuthology.orchestra.run.smithi097.stdout: ] 2024-03-11T21:38:22.962 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-03-11T21:38:22.962 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:38:22.962 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:38:22.962 INFO:teuthology.orchestra.run.smithi097.stdout: "description": "osd_op(client.15150.0:208608 8.d 8:b39a191f:::benchmark_data_smithi092_104865_object88107:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e239)", 2024-03-11T21:38:22.963 INFO:teuthology.orchestra.run.smithi097.stdout: "initiated_at": "2024-03-11T21:34:57.377729+0000", 2024-03-11T21:38:22.963 INFO:teuthology.orchestra.run.smithi097.stdout: "age": 204.882466099, 2024-03-11T21:38:22.963 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.016253541, 2024-03-11T21:38:22.963 INFO:teuthology.orchestra.run.smithi097.stdout: "type_data": { 2024-03-11T21:38:22.963 INFO:teuthology.orchestra.run.smithi097.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-11T21:38:22.963 INFO:teuthology.orchestra.run.smithi097.stdout: "client_info": { 2024-03-11T21:38:22.963 INFO:teuthology.orchestra.run.smithi097.stdout: "client": "client.15150", 2024-03-11T21:38:22.963 INFO:teuthology.orchestra.run.smithi097.stdout: "client_addr": "v1:172.21.15.92:0/436606717", 2024-03-11T21:38:22.963 INFO:teuthology.orchestra.run.smithi097.stdout: "tid": 208608 2024-03-11T21:38:22.963 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:38:22.963 INFO:teuthology.orchestra.run.smithi097.stdout: "events": [ 2024-03-11T21:38:22.963 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:38:22.963 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "initiated", 2024-03-11T21:38:22.963 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:34:57.377729+0000", 2024-03-11T21:38:22.963 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-03-11T21:38:22.964 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:38:22.964 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:38:22.964 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "header_read", 2024-03-11T21:38:22.964 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:34:57.377729+0000", 2024-03-11T21:38:22.964 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-03-11T21:38:22.964 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:38:22.964 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:38:22.964 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "throttled", 2024-03-11T21:38:22.964 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:34:57.377731+0000", 2024-03-11T21:38:22.964 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 2.1059999999999998e-06 2024-03-11T21:38:22.964 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:38:22.964 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:38:22.964 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "all_read", 2024-03-11T21:38:22.964 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:34:57.377739+0000", 2024-03-11T21:38:22.964 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 8.5110000000000003e-06 2024-03-11T21:38:22.964 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:38:22.965 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:38:22.965 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "dispatched", 2024-03-11T21:38:22.965 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:34:57.377752+0000", 2024-03-11T21:38:22.965 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.2523e-05 2024-03-11T21:38:22.965 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:38:22.965 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:38:22.965 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "queued_for_pg", 2024-03-11T21:38:22.965 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:34:57.377760+0000", 2024-03-11T21:38:22.965 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 7.8490000000000006e-06 2024-03-11T21:38:22.965 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:38:22.965 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:38:22.965 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "reached_pg", 2024-03-11T21:38:22.965 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:34:57.377823+0000", 2024-03-11T21:38:22.965 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 6.3337999999999996e-05 2024-03-11T21:38:22.965 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:38:22.966 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:38:22.966 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "waiting for peered", 2024-03-11T21:38:22.966 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:34:57.377833+0000", 2024-03-11T21:38:22.966 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 9.7070000000000004e-06 2024-03-11T21:38:22.966 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:38:22.966 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:38:22.966 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "reached_pg", 2024-03-11T21:38:22.966 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:34:58.390377+0000", 2024-03-11T21:38:22.966 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.0125443629999999 2024-03-11T21:38:22.966 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:38:22.966 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:38:22.966 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "started", 2024-03-11T21:38:22.966 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:34:58.390455+0000", 2024-03-11T21:38:22.966 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 7.7832999999999998e-05 2024-03-11T21:38:22.966 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:38:22.967 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:38:22.967 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "waiting for subops from 2,7", 2024-03-11T21:38:22.967 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:34:58.390541+0000", 2024-03-11T21:38:22.967 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 8.6014999999999996e-05 2024-03-11T21:38:22.967 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:38:22.967 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:38:22.967 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "sub_op_commit_rec", 2024-03-11T21:38:22.967 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:34:58.393817+0000", 2024-03-11T21:38:22.967 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.0032762799999999999 2024-03-11T21:38:22.967 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:38:22.967 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:38:22.967 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "op_commit", 2024-03-11T21:38:22.967 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:34:58.393832+0000", 2024-03-11T21:38:22.967 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.4773999999999999e-05 2024-03-11T21:38:22.967 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:38:22.967 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:38:22.967 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "sub_op_commit_rec", 2024-03-11T21:38:22.968 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:34:58.393890+0000", 2024-03-11T21:38:22.968 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 5.7969e-05 2024-03-11T21:38:22.968 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:38:22.968 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:38:22.968 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "commit_sent", 2024-03-11T21:38:22.968 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:34:58.393941+0000", 2024-03-11T21:38:22.968 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 5.1403999999999998e-05 2024-03-11T21:38:22.968 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:38:22.968 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:38:22.968 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "done", 2024-03-11T21:38:22.968 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:34:58.393982+0000", 2024-03-11T21:38:22.968 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 4.0868999999999998e-05 2024-03-11T21:38:22.968 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-03-11T21:38:22.968 INFO:teuthology.orchestra.run.smithi097.stdout: ] 2024-03-11T21:38:22.968 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-03-11T21:38:22.969 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:38:22.969 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:38:22.969 INFO:teuthology.orchestra.run.smithi097.stdout: "description": "osd_op(client.15150.0:208648 8.d 8:b7832715:::benchmark_data_smithi092_104865_object88147:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e239)", 2024-03-11T21:38:22.969 INFO:teuthology.orchestra.run.smithi097.stdout: "initiated_at": "2024-03-11T21:34:57.409741+0000", 2024-03-11T21:38:22.969 INFO:teuthology.orchestra.run.smithi097.stdout: "age": 204.85045426900001, 2024-03-11T21:38:22.969 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.99063108700000002, 2024-03-11T21:38:22.969 INFO:teuthology.orchestra.run.smithi097.stdout: "type_data": { 2024-03-11T21:38:22.969 INFO:teuthology.orchestra.run.smithi097.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-11T21:38:22.969 INFO:teuthology.orchestra.run.smithi097.stdout: "client_info": { 2024-03-11T21:38:22.969 INFO:teuthology.orchestra.run.smithi097.stdout: "client": "client.15150", 2024-03-11T21:38:22.969 INFO:teuthology.orchestra.run.smithi097.stdout: "client_addr": "v1:172.21.15.92:0/436606717", 2024-03-11T21:38:22.969 INFO:teuthology.orchestra.run.smithi097.stdout: "tid": 208648 2024-03-11T21:38:22.969 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:38:22.969 INFO:teuthology.orchestra.run.smithi097.stdout: "events": [ 2024-03-11T21:38:22.969 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:38:22.969 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "initiated", 2024-03-11T21:38:22.970 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:34:57.409741+0000", 2024-03-11T21:38:22.970 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-03-11T21:38:22.970 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:38:22.970 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:38:22.970 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "header_read", 2024-03-11T21:38:22.970 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:34:57.409741+0000", 2024-03-11T21:38:22.970 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-03-11T21:38:22.970 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:38:22.970 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:38:22.970 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "throttled", 2024-03-11T21:38:22.970 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:34:57.409743+0000", 2024-03-11T21:38:22.970 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 2.39e-06 2024-03-11T21:38:22.970 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:38:22.970 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:38:22.970 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "all_read", 2024-03-11T21:38:22.971 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:34:57.409749+0000", 2024-03-11T21:38:22.971 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 5.9200000000000001e-06 2024-03-11T21:38:22.971 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:38:22.971 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:38:22.971 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "dispatched", 2024-03-11T21:38:22.971 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:34:57.409764+0000", 2024-03-11T21:38:22.971 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.5427e-05 2024-03-11T21:38:22.971 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:38:22.971 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:38:22.971 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "queued_for_pg", 2024-03-11T21:38:22.971 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:34:57.409772+0000", 2024-03-11T21:38:22.971 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 7.7239999999999992e-06 2024-03-11T21:38:22.971 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:38:22.971 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:38:22.972 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "reached_pg", 2024-03-11T21:38:22.972 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:34:57.409829+0000", 2024-03-11T21:38:22.972 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 5.7169000000000001e-05 2024-03-11T21:38:22.972 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:38:22.972 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:38:22.972 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "waiting for peered", 2024-03-11T21:38:22.972 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:34:57.409837+0000", 2024-03-11T21:38:22.972 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 8.2379999999999997e-06 2024-03-11T21:38:22.972 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:38:22.972 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:38:22.972 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "reached_pg", 2024-03-11T21:38:22.972 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:34:58.394032+0000", 2024-03-11T21:38:22.972 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.98419469599999998 2024-03-11T21:38:22.972 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:38:22.972 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:38:22.972 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "started", 2024-03-11T21:38:22.973 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:34:58.394152+0000", 2024-03-11T21:38:22.973 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.00011963700000000001 2024-03-11T21:38:22.973 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:38:22.973 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:38:22.973 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "waiting for subops from 2,7", 2024-03-11T21:38:22.973 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:34:58.394262+0000", 2024-03-11T21:38:22.973 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.00010984600000000001 2024-03-11T21:38:22.973 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:38:22.973 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:38:22.973 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "sub_op_commit_rec", 2024-03-11T21:38:22.973 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:34:58.400153+0000", 2024-03-11T21:38:22.973 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.0058912019999999999 2024-03-11T21:38:22.973 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:38:22.973 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:38:22.973 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "sub_op_commit_rec", 2024-03-11T21:38:22.973 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:34:58.400199+0000", 2024-03-11T21:38:22.974 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 4.6400999999999998e-05 2024-03-11T21:38:22.974 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:38:22.974 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:38:22.974 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "op_commit", 2024-03-11T21:38:22.974 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:34:58.400212+0000", 2024-03-11T21:38:22.974 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.2258e-05 2024-03-11T21:38:22.974 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:38:22.974 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:38:22.974 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "commit_sent", 2024-03-11T21:38:22.974 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:34:58.400326+0000", 2024-03-11T21:38:22.974 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.000114855 2024-03-11T21:38:22.974 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:38:22.974 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:38:22.974 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "done", 2024-03-11T21:38:22.975 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:34:58.400372+0000", 2024-03-11T21:38:22.975 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 4.5324000000000002e-05 2024-03-11T21:38:22.975 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-03-11T21:38:22.975 INFO:teuthology.orchestra.run.smithi097.stdout: ] 2024-03-11T21:38:22.975 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-03-11T21:38:22.975 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:38:22.975 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:38:22.975 INFO:teuthology.orchestra.run.smithi097.stdout: "description": "osd_op(client.15150.0:208683 8.d 8:ba654bd9:::benchmark_data_smithi092_104865_object88182:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e239)", 2024-03-11T21:38:22.975 INFO:teuthology.orchestra.run.smithi097.stdout: "initiated_at": "2024-03-11T21:34:57.434209+0000", 2024-03-11T21:38:22.975 INFO:teuthology.orchestra.run.smithi097.stdout: "age": 204.825985659, 2024-03-11T21:38:22.975 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.97204510700000002, 2024-03-11T21:38:22.975 INFO:teuthology.orchestra.run.smithi097.stdout: "type_data": { 2024-03-11T21:38:22.975 INFO:teuthology.orchestra.run.smithi097.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-11T21:38:22.975 INFO:teuthology.orchestra.run.smithi097.stdout: "client_info": { 2024-03-11T21:38:22.976 INFO:teuthology.orchestra.run.smithi097.stdout: "client": "client.15150", 2024-03-11T21:38:22.976 INFO:teuthology.orchestra.run.smithi097.stdout: "client_addr": "v1:172.21.15.92:0/436606717", 2024-03-11T21:38:22.976 INFO:teuthology.orchestra.run.smithi097.stdout: "tid": 208683 2024-03-11T21:38:22.976 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:38:22.976 INFO:teuthology.orchestra.run.smithi097.stdout: "events": [ 2024-03-11T21:38:22.976 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:38:22.976 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "initiated", 2024-03-11T21:38:22.976 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:34:57.434209+0000", 2024-03-11T21:38:22.976 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-03-11T21:38:22.976 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:38:22.976 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:38:22.976 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "header_read", 2024-03-11T21:38:22.976 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:34:57.434209+0000", 2024-03-11T21:38:22.976 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-03-11T21:38:22.976 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:38:22.976 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:38:22.977 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "throttled", 2024-03-11T21:38:22.977 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:34:57.434211+0000", 2024-03-11T21:38:22.977 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 2.1579999999999999e-06 2024-03-11T21:38:22.977 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:38:22.977 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:38:22.977 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "all_read", 2024-03-11T21:38:22.977 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:34:57.434216+0000", 2024-03-11T21:38:22.977 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 4.9189999999999998e-06 2024-03-11T21:38:22.977 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:38:22.977 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:38:22.977 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "dispatched", 2024-03-11T21:38:22.977 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:34:57.434228+0000", 2024-03-11T21:38:22.977 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.1389000000000001e-05 2024-03-11T21:38:22.977 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:38:22.977 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:38:22.978 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "queued_for_pg", 2024-03-11T21:38:22.978 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:34:57.434235+0000", 2024-03-11T21:38:22.978 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 7.1130000000000003e-06 2024-03-11T21:38:22.978 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:38:22.978 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:38:22.978 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "reached_pg", 2024-03-11T21:38:22.978 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:34:57.434292+0000", 2024-03-11T21:38:22.978 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 5.7268999999999997e-05 2024-03-11T21:38:22.978 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:38:22.978 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:38:22.978 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "waiting for peered", 2024-03-11T21:38:22.978 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:34:57.434302+0000", 2024-03-11T21:38:22.978 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.0339e-05 2024-03-11T21:38:22.978 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:38:22.979 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:38:22.979 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "reached_pg", 2024-03-11T21:38:22.979 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:34:58.396996+0000", 2024-03-11T21:38:22.979 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.962694033 2024-03-11T21:38:22.979 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:38:22.979 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:38:22.979 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "started", 2024-03-11T21:38:22.979 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:34:58.397103+0000", 2024-03-11T21:38:22.979 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.000106118 2024-03-11T21:38:22.979 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:38:22.979 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:38:22.979 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "waiting for subops from 2,7", 2024-03-11T21:38:22.979 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:34:58.397243+0000", 2024-03-11T21:38:22.979 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.00014001399999999999 2024-03-11T21:38:22.979 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:38:22.980 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:38:22.980 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "sub_op_commit_rec", 2024-03-11T21:38:22.980 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:34:58.400415+0000", 2024-03-11T21:38:22.980 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.003172186 2024-03-11T21:38:22.980 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:38:22.980 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:38:22.980 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "sub_op_commit_rec", 2024-03-11T21:38:22.980 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:34:58.400496+0000", 2024-03-11T21:38:22.980 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 8.0828000000000006e-05 2024-03-11T21:38:22.980 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:38:22.980 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:38:22.980 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "op_commit", 2024-03-11T21:38:22.980 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:34:58.406159+0000", 2024-03-11T21:38:22.980 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.005663609 2024-03-11T21:38:22.980 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:38:22.980 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:38:22.981 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "commit_sent", 2024-03-11T21:38:22.981 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:34:58.406198+0000", 2024-03-11T21:38:22.981 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 3.9069000000000002e-05 2024-03-11T21:38:22.981 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:38:22.981 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:38:22.981 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "done", 2024-03-11T21:38:22.981 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:34:58.406254+0000", 2024-03-11T21:38:22.981 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 5.6063000000000002e-05 2024-03-11T21:38:22.981 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-03-11T21:38:22.981 INFO:teuthology.orchestra.run.smithi097.stdout: ] 2024-03-11T21:38:22.981 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-03-11T21:38:22.981 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:38:22.982 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:38:22.982 INFO:teuthology.orchestra.run.smithi097.stdout: "description": "osd_op(client.15150.0:208685 8.d 8:b7ae4ce9:::benchmark_data_smithi092_104865_object88184:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e239)", 2024-03-11T21:38:22.982 INFO:teuthology.orchestra.run.smithi097.stdout: "initiated_at": "2024-03-11T21:34:57.436752+0000", 2024-03-11T21:38:22.982 INFO:teuthology.orchestra.run.smithi097.stdout: "age": 204.82344241600001, 2024-03-11T21:38:22.982 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.969736033, 2024-03-11T21:38:22.982 INFO:teuthology.orchestra.run.smithi097.stdout: "type_data": { 2024-03-11T21:38:22.982 INFO:teuthology.orchestra.run.smithi097.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-11T21:38:22.982 INFO:teuthology.orchestra.run.smithi097.stdout: "client_info": { 2024-03-11T21:38:22.982 INFO:teuthology.orchestra.run.smithi097.stdout: "client": "client.15150", 2024-03-11T21:38:22.982 INFO:teuthology.orchestra.run.smithi097.stdout: "client_addr": "v1:172.21.15.92:0/436606717", 2024-03-11T21:38:22.982 INFO:teuthology.orchestra.run.smithi097.stdout: "tid": 208685 2024-03-11T21:38:22.982 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:38:22.982 INFO:teuthology.orchestra.run.smithi097.stdout: "events": [ 2024-03-11T21:38:22.982 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:38:22.983 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "initiated", 2024-03-11T21:38:22.983 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:34:57.436752+0000", 2024-03-11T21:38:22.983 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-03-11T21:38:22.983 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:38:22.983 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:38:22.983 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "header_read", 2024-03-11T21:38:22.983 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:34:57.436752+0000", 2024-03-11T21:38:22.983 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-03-11T21:38:22.983 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:38:22.983 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:38:22.983 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "throttled", 2024-03-11T21:38:22.983 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:34:57.436754+0000", 2024-03-11T21:38:22.983 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.319e-06 2024-03-11T21:38:22.983 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:38:22.983 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:38:22.983 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "all_read", 2024-03-11T21:38:22.984 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:34:57.436758+0000", 2024-03-11T21:38:22.984 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 4.6700000000000002e-06 2024-03-11T21:38:22.984 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:38:22.984 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:38:22.984 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "dispatched", 2024-03-11T21:38:22.984 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:34:57.436768+0000", 2024-03-11T21:38:22.984 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 9.8989999999999996e-06 2024-03-11T21:38:22.984 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:38:22.984 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:38:22.984 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "queued_for_pg", 2024-03-11T21:38:22.984 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:34:57.436776+0000", 2024-03-11T21:38:22.984 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 7.9279999999999993e-06 2024-03-11T21:38:22.984 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:38:22.984 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:38:22.984 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "reached_pg", 2024-03-11T21:38:22.984 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:34:57.436828+0000", 2024-03-11T21:38:22.985 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 5.2026e-05 2024-03-11T21:38:22.985 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:38:22.985 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:38:22.985 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "waiting for peered", 2024-03-11T21:38:22.985 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:34:57.436836+0000", 2024-03-11T21:38:22.985 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 7.9359999999999999e-06 2024-03-11T21:38:22.985 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:38:22.985 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:38:22.985 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "reached_pg", 2024-03-11T21:38:22.985 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:34:58.400540+0000", 2024-03-11T21:38:22.985 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.96370411899999997 2024-03-11T21:38:22.985 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:38:22.985 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:38:22.985 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "started", 2024-03-11T21:38:22.985 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:34:58.400635+0000", 2024-03-11T21:38:22.986 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 9.4242999999999999e-05 2024-03-11T21:38:22.986 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:38:22.986 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:38:22.986 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "waiting for subops from 2,7", 2024-03-11T21:38:22.986 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:34:58.400737+0000", 2024-03-11T21:38:22.986 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.00010212 2024-03-11T21:38:22.986 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:38:22.986 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:38:22.986 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "sub_op_commit_rec", 2024-03-11T21:38:22.986 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:34:58.406321+0000", 2024-03-11T21:38:22.986 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.0055847459999999998 2024-03-11T21:38:22.986 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:38:22.986 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:38:22.986 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "sub_op_commit_rec", 2024-03-11T21:38:22.987 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:34:58.406385+0000", 2024-03-11T21:38:22.987 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 6.4005999999999997e-05 2024-03-11T21:38:22.987 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:38:22.987 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:38:22.987 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "op_commit", 2024-03-11T21:38:22.987 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:34:58.406401+0000", 2024-03-11T21:38:22.987 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.5627000000000001e-05 2024-03-11T21:38:22.987 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:38:22.987 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:38:22.987 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "commit_sent", 2024-03-11T21:38:22.987 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:34:58.406448+0000", 2024-03-11T21:38:22.987 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 4.6703000000000003e-05 2024-03-11T21:38:22.987 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:38:22.987 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:38:22.987 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "done", 2024-03-11T21:38:22.987 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:34:58.406489+0000", 2024-03-11T21:38:22.988 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 4.0691000000000001e-05 2024-03-11T21:38:22.988 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-03-11T21:38:22.988 INFO:teuthology.orchestra.run.smithi097.stdout: ] 2024-03-11T21:38:22.988 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-03-11T21:38:22.988 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:38:22.988 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:38:22.988 INFO:teuthology.orchestra.run.smithi097.stdout: "description": "osd_op(client.25282.0:76283 9.f 9:fcbba6a0:::benchmark_data_smithi092_130291_object76282:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e276)", 2024-03-11T21:38:22.988 INFO:teuthology.orchestra.run.smithi097.stdout: "initiated_at": "2024-03-11T21:36:34.033579+0000", 2024-03-11T21:38:22.988 INFO:teuthology.orchestra.run.smithi097.stdout: "age": 108.226616079, 2024-03-11T21:38:22.988 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 2.490014623, 2024-03-11T21:38:22.988 INFO:teuthology.orchestra.run.smithi097.stdout: "type_data": { 2024-03-11T21:38:22.988 INFO:teuthology.orchestra.run.smithi097.stdout: "flag_point": "reached pg", 2024-03-11T21:38:22.988 INFO:teuthology.orchestra.run.smithi097.stdout: "client_info": { 2024-03-11T21:38:22.988 INFO:teuthology.orchestra.run.smithi097.stdout: "client": "client.25282", 2024-03-11T21:38:22.988 INFO:teuthology.orchestra.run.smithi097.stdout: "client_addr": "v1:172.21.15.92:0/2489519821", 2024-03-11T21:38:22.988 INFO:teuthology.orchestra.run.smithi097.stdout: "tid": 76283 2024-03-11T21:38:22.989 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:38:22.989 INFO:teuthology.orchestra.run.smithi097.stdout: "events": [ 2024-03-11T21:38:22.989 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:38:22.989 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "initiated", 2024-03-11T21:38:22.989 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:36:34.033579+0000", 2024-03-11T21:38:22.989 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-03-11T21:38:22.989 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:38:22.989 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:38:22.989 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "header_read", 2024-03-11T21:38:22.989 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:36:34.033579+0000", 2024-03-11T21:38:22.989 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-03-11T21:38:22.989 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:38:22.989 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:38:22.989 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "throttled", 2024-03-11T21:38:22.989 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:36:34.033580+0000", 2024-03-11T21:38:22.989 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 7.3200000000000004e-07 2024-03-11T21:38:22.990 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:38:22.990 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:38:22.990 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "all_read", 2024-03-11T21:38:22.990 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:36:34.033612+0000", 2024-03-11T21:38:22.990 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 3.2396e-05 2024-03-11T21:38:22.990 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:38:22.990 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:38:22.990 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "dispatched", 2024-03-11T21:38:22.990 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:36:34.033620+0000", 2024-03-11T21:38:22.990 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 7.7039999999999994e-06 2024-03-11T21:38:22.990 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:38:22.990 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:38:22.990 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "queued_for_pg", 2024-03-11T21:38:22.990 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:36:34.033625+0000", 2024-03-11T21:38:22.991 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 5.5779999999999998e-06 2024-03-11T21:38:22.991 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:38:22.991 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:38:22.991 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "reached_pg", 2024-03-11T21:38:22.991 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:36:34.033753+0000", 2024-03-11T21:38:22.991 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.00012819 2024-03-11T21:38:22.991 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:38:22.991 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:38:22.991 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "started", 2024-03-11T21:38:22.991 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:36:34.034007+0000", 2024-03-11T21:38:22.991 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.00025316200000000003 2024-03-11T21:38:22.991 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:38:22.991 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:38:22.991 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "waiting for subops from 1", 2024-03-11T21:38:22.991 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:36:34.034136+0000", 2024-03-11T21:38:22.992 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.00012985399999999999 2024-03-11T21:38:22.992 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:38:22.992 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:38:22.992 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "op_commit", 2024-03-11T21:38:22.992 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:36:34.046308+0000", 2024-03-11T21:38:22.992 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.012171131 2024-03-11T21:38:22.992 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:38:22.992 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:38:22.992 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "reached_pg", 2024-03-11T21:38:22.992 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:36:34.058375+0000", 2024-03-11T21:38:22.992 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.012067791 2024-03-11T21:38:22.992 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:38:22.992 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:38:22.992 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "waiting for peered", 2024-03-11T21:38:22.992 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:36:34.058387+0000", 2024-03-11T21:38:22.993 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.1623999999999999e-05 2024-03-11T21:38:22.993 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:38:22.993 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:38:22.993 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "reached_pg", 2024-03-11T21:38:22.993 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:36:35.046799+0000", 2024-03-11T21:38:22.993 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.98841167900000004 2024-03-11T21:38:22.993 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:38:22.993 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:38:22.993 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "waiting for peered", 2024-03-11T21:38:22.993 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:36:35.046813+0000", 2024-03-11T21:38:22.993 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.4258e-05 2024-03-11T21:38:22.993 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:38:22.993 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:38:22.993 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "reached_pg", 2024-03-11T21:38:22.993 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:36:36.523493+0000", 2024-03-11T21:38:22.994 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.476680408 2024-03-11T21:38:22.994 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:38:22.994 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:38:22.994 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "done", 2024-03-11T21:38:22.994 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:36:36.523593+0000", 2024-03-11T21:38:22.994 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.000100116 2024-03-11T21:38:22.994 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-03-11T21:38:22.994 INFO:teuthology.orchestra.run.smithi097.stdout: ] 2024-03-11T21:38:22.994 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-03-11T21:38:22.994 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:38:22.994 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:38:22.994 INFO:teuthology.orchestra.run.smithi097.stdout: "description": "osd_op(client.25282.0:76287 9.7 9:eeb358ff:::benchmark_data_smithi092_130291_object76286:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e276)", 2024-03-11T21:38:22.994 INFO:teuthology.orchestra.run.smithi097.stdout: "initiated_at": "2024-03-11T21:36:34.036194+0000", 2024-03-11T21:38:22.994 INFO:teuthology.orchestra.run.smithi097.stdout: "age": 108.224000851, 2024-03-11T21:38:22.995 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 2.5035221519999999, 2024-03-11T21:38:22.995 INFO:teuthology.orchestra.run.smithi097.stdout: "type_data": { 2024-03-11T21:38:22.995 INFO:teuthology.orchestra.run.smithi097.stdout: "flag_point": "reached pg", 2024-03-11T21:38:22.995 INFO:teuthology.orchestra.run.smithi097.stdout: "client_info": { 2024-03-11T21:38:22.995 INFO:teuthology.orchestra.run.smithi097.stdout: "client": "client.25282", 2024-03-11T21:38:22.995 INFO:teuthology.orchestra.run.smithi097.stdout: "client_addr": "v1:172.21.15.92:0/2489519821", 2024-03-11T21:38:22.995 INFO:teuthology.orchestra.run.smithi097.stdout: "tid": 76287 2024-03-11T21:38:22.995 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:38:22.995 INFO:teuthology.orchestra.run.smithi097.stdout: "events": [ 2024-03-11T21:38:22.995 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:38:22.995 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "initiated", 2024-03-11T21:38:22.995 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:36:34.036194+0000", 2024-03-11T21:38:22.995 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-03-11T21:38:22.995 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:38:22.995 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:38:22.995 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "header_read", 2024-03-11T21:38:22.996 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:36:34.036194+0000", 2024-03-11T21:38:22.996 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-03-11T21:38:22.996 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:38:22.996 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:38:22.996 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "throttled", 2024-03-11T21:38:22.996 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:36:34.036196+0000", 2024-03-11T21:38:22.996 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 2.317e-06 2024-03-11T21:38:22.996 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:38:22.996 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:38:22.996 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "all_read", 2024-03-11T21:38:22.996 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:36:34.036261+0000", 2024-03-11T21:38:22.996 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 6.4343000000000004e-05 2024-03-11T21:38:22.996 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:38:22.996 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:38:22.997 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "dispatched", 2024-03-11T21:38:22.997 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:36:34.036272+0000", 2024-03-11T21:38:22.997 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.1790000000000001e-05 2024-03-11T21:38:22.997 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:38:22.997 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:38:22.997 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "queued_for_pg", 2024-03-11T21:38:22.997 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:36:34.036281+0000", 2024-03-11T21:38:22.997 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 8.3110000000000006e-06 2024-03-11T21:38:22.997 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:38:22.997 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:38:22.997 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "reached_pg", 2024-03-11T21:38:22.997 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:36:34.037543+0000", 2024-03-11T21:38:22.997 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.001262631 2024-03-11T21:38:22.997 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:38:22.997 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:38:22.997 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "started", 2024-03-11T21:38:22.998 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:36:34.037890+0000", 2024-03-11T21:38:22.998 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.00034664199999999998 2024-03-11T21:38:22.998 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:38:22.998 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:38:22.998 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "waiting for subops from 7", 2024-03-11T21:38:22.998 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:36:34.038099+0000", 2024-03-11T21:38:22.998 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.000208438 2024-03-11T21:38:22.998 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:38:22.998 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:38:22.998 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "reached_pg", 2024-03-11T21:38:22.998 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:36:34.058425+0000", 2024-03-11T21:38:22.998 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.020326765 2024-03-11T21:38:22.998 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:38:22.998 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:38:22.999 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "waiting for peered", 2024-03-11T21:38:22.999 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:36:34.058434+0000", 2024-03-11T21:38:22.999 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 8.9099999999999994e-06 2024-03-11T21:38:22.999 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:38:22.999 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:38:22.999 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "reached_pg", 2024-03-11T21:38:22.999 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:36:35.046759+0000", 2024-03-11T21:38:22.999 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.98832484399999998 2024-03-11T21:38:22.999 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:38:22.999 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:38:22.999 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "waiting for peered", 2024-03-11T21:38:22.999 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:36:35.046795+0000", 2024-03-11T21:38:22.999 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 3.6155999999999997e-05 2024-03-11T21:38:22.999 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:38:22.999 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:38:22.999 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "reached_pg", 2024-03-11T21:38:23.000 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:36:36.523236+0000", 2024-03-11T21:38:23.000 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.476440669 2024-03-11T21:38:23.000 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:38:23.000 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:38:23.000 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "waiting for degraded object", 2024-03-11T21:38:23.000 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:36:36.523912+0000", 2024-03-11T21:38:23.000 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.00067594499999999995 2024-03-11T21:38:23.000 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:38:23.000 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:38:23.000 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "reached_pg", 2024-03-11T21:38:23.000 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:36:36.539590+0000", 2024-03-11T21:38:23.000 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.015678455000000001 2024-03-11T21:38:23.000 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:38:23.000 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:38:23.000 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "done", 2024-03-11T21:38:23.001 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:36:36.539716+0000", 2024-03-11T21:38:23.001 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.00012593599999999999 2024-03-11T21:38:23.001 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-03-11T21:38:23.001 INFO:teuthology.orchestra.run.smithi097.stdout: ] 2024-03-11T21:38:23.001 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-03-11T21:38:23.001 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:38:23.001 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:38:23.001 INFO:teuthology.orchestra.run.smithi097.stdout: "description": "osd_op(client.25282.0:76292 9.f 9:f3aa237a:::benchmark_data_smithi092_130291_object76291:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e276)", 2024-03-11T21:38:23.001 INFO:teuthology.orchestra.run.smithi097.stdout: "initiated_at": "2024-03-11T21:36:34.038523+0000", 2024-03-11T21:38:23.001 INFO:teuthology.orchestra.run.smithi097.stdout: "age": 108.221672325, 2024-03-11T21:38:23.001 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 2.5077632140000001, 2024-03-11T21:38:23.001 INFO:teuthology.orchestra.run.smithi097.stdout: "type_data": { 2024-03-11T21:38:23.001 INFO:teuthology.orchestra.run.smithi097.stdout: "flag_point": "reached pg", 2024-03-11T21:38:23.001 INFO:teuthology.orchestra.run.smithi097.stdout: "client_info": { 2024-03-11T21:38:23.002 INFO:teuthology.orchestra.run.smithi097.stdout: "client": "client.25282", 2024-03-11T21:38:23.002 INFO:teuthology.orchestra.run.smithi097.stdout: "client_addr": "v1:172.21.15.92:0/2489519821", 2024-03-11T21:38:23.002 INFO:teuthology.orchestra.run.smithi097.stdout: "tid": 76292 2024-03-11T21:38:23.002 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:38:23.002 INFO:teuthology.orchestra.run.smithi097.stdout: "events": [ 2024-03-11T21:38:23.002 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:38:23.002 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "initiated", 2024-03-11T21:38:23.002 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:36:34.038523+0000", 2024-03-11T21:38:23.002 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-03-11T21:38:23.002 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:38:23.002 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:38:23.002 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "header_read", 2024-03-11T21:38:23.002 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:36:34.038523+0000", 2024-03-11T21:38:23.002 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-03-11T21:38:23.003 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:38:23.003 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:38:23.003 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "throttled", 2024-03-11T21:38:23.003 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:36:34.038524+0000", 2024-03-11T21:38:23.003 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.206e-06 2024-03-11T21:38:23.003 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:38:23.003 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:38:23.003 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "all_read", 2024-03-11T21:38:23.003 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:36:34.038578+0000", 2024-03-11T21:38:23.003 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 5.3983000000000001e-05 2024-03-11T21:38:23.003 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:38:23.003 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:38:23.003 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "dispatched", 2024-03-11T21:38:23.003 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:36:34.038586+0000", 2024-03-11T21:38:23.004 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 8.1610000000000003e-06 2024-03-11T21:38:23.004 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:38:23.004 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:38:23.004 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "queued_for_pg", 2024-03-11T21:38:23.004 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:36:34.038593+0000", 2024-03-11T21:38:23.004 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 6.6220000000000003e-06 2024-03-11T21:38:23.004 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:38:23.004 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:38:23.004 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "reached_pg", 2024-03-11T21:38:23.004 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:36:34.040758+0000", 2024-03-11T21:38:23.004 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.0021650860000000001 2024-03-11T21:38:23.004 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:38:23.004 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:38:23.004 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "started", 2024-03-11T21:38:23.004 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:36:34.040879+0000", 2024-03-11T21:38:23.005 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.000121019 2024-03-11T21:38:23.005 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:38:23.005 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:38:23.005 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "waiting for subops from 1", 2024-03-11T21:38:23.005 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:36:34.041015+0000", 2024-03-11T21:38:23.005 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.00013673799999999999 2024-03-11T21:38:23.005 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:38:23.005 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:38:23.005 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "reached_pg", 2024-03-11T21:38:23.005 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:36:34.058475+0000", 2024-03-11T21:38:23.005 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.017459204999999998 2024-03-11T21:38:23.005 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:38:23.005 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:38:23.005 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "waiting for peered", 2024-03-11T21:38:23.005 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:36:34.058485+0000", 2024-03-11T21:38:23.006 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.0876999999999999e-05 2024-03-11T21:38:23.006 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:38:23.006 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:38:23.006 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "reached_pg", 2024-03-11T21:38:23.006 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:36:35.046859+0000", 2024-03-11T21:38:23.006 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.98837370899999999 2024-03-11T21:38:23.006 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:38:23.006 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:38:23.006 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "waiting for peered", 2024-03-11T21:38:23.006 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:36:35.046870+0000", 2024-03-11T21:38:23.006 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.0985000000000001e-05 2024-03-11T21:38:23.006 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:38:23.006 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:38:23.007 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "reached_pg", 2024-03-11T21:38:23.007 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:36:36.523712+0000", 2024-03-11T21:38:23.007 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.4768413869999999 2024-03-11T21:38:23.007 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:38:23.007 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:38:23.007 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "waiting for degraded object", 2024-03-11T21:38:23.007 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:36:36.524491+0000", 2024-03-11T21:38:23.007 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.00077957099999999995 2024-03-11T21:38:23.007 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:38:23.007 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:38:23.007 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "reached_pg", 2024-03-11T21:38:23.007 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:36:36.546182+0000", 2024-03-11T21:38:23.007 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.021691298000000001 2024-03-11T21:38:23.007 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:38:23.007 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:38:23.008 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "done", 2024-03-11T21:38:23.008 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:36:36.546286+0000", 2024-03-11T21:38:23.008 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.000103367 2024-03-11T21:38:23.008 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-03-11T21:38:23.008 INFO:teuthology.orchestra.run.smithi097.stdout: ] 2024-03-11T21:38:23.008 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-03-11T21:38:23.008 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:38:23.008 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:38:23.008 INFO:teuthology.orchestra.run.smithi097.stdout: "description": "osd_op(client.25282.0:76296 9.f 9:f93c9389:::benchmark_data_smithi092_130291_object76295:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e277)", 2024-03-11T21:38:23.008 INFO:teuthology.orchestra.run.smithi097.stdout: "initiated_at": "2024-03-11T21:36:34.055957+0000", 2024-03-11T21:38:23.008 INFO:teuthology.orchestra.run.smithi097.stdout: "age": 108.204237494, 2024-03-11T21:38:23.008 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 2.4942122050000002, 2024-03-11T21:38:23.008 INFO:teuthology.orchestra.run.smithi097.stdout: "type_data": { 2024-03-11T21:38:23.008 INFO:teuthology.orchestra.run.smithi097.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-11T21:38:23.008 INFO:teuthology.orchestra.run.smithi097.stdout: "client_info": { 2024-03-11T21:38:23.009 INFO:teuthology.orchestra.run.smithi097.stdout: "client": "client.25282", 2024-03-11T21:38:23.009 INFO:teuthology.orchestra.run.smithi097.stdout: "client_addr": "v1:172.21.15.92:0/2489519821", 2024-03-11T21:38:23.009 INFO:teuthology.orchestra.run.smithi097.stdout: "tid": 76296 2024-03-11T21:38:23.009 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:38:23.009 INFO:teuthology.orchestra.run.smithi097.stdout: "events": [ 2024-03-11T21:38:23.009 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:38:23.009 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "initiated", 2024-03-11T21:38:23.009 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:36:34.055957+0000", 2024-03-11T21:38:23.009 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-03-11T21:38:23.009 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:38:23.009 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:38:23.009 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "header_read", 2024-03-11T21:38:23.009 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:36:34.055957+0000", 2024-03-11T21:38:23.009 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-03-11T21:38:23.010 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:38:23.010 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:38:23.010 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "throttled", 2024-03-11T21:38:23.010 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:36:34.055960+0000", 2024-03-11T21:38:23.010 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 2.2800000000000002e-06 2024-03-11T21:38:23.010 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:38:23.010 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:38:23.010 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "all_read", 2024-03-11T21:38:23.010 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:36:34.055993+0000", 2024-03-11T21:38:23.010 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 3.2874999999999999e-05 2024-03-11T21:38:23.010 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:38:23.010 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:38:23.010 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "dispatched", 2024-03-11T21:38:23.010 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:36:34.056006+0000", 2024-03-11T21:38:23.011 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.2979999999999999e-05 2024-03-11T21:38:23.011 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:38:23.011 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:38:23.011 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "queued_for_pg", 2024-03-11T21:38:23.011 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:36:34.056015+0000", 2024-03-11T21:38:23.011 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 9.3009999999999995e-06 2024-03-11T21:38:23.011 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:38:23.011 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:38:23.011 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "reached_pg", 2024-03-11T21:38:23.011 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:36:34.058531+0000", 2024-03-11T21:38:23.011 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.0025166609999999999 2024-03-11T21:38:23.011 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:38:23.011 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:38:23.012 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "waiting for peered", 2024-03-11T21:38:23.012 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:36:34.058540+0000", 2024-03-11T21:38:23.012 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 8.9390000000000003e-06 2024-03-11T21:38:23.012 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:38:23.012 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:38:23.012 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "reached_pg", 2024-03-11T21:38:23.012 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:36:35.046903+0000", 2024-03-11T21:38:23.012 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.98836246699999997 2024-03-11T21:38:23.012 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:38:23.012 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:38:23.012 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "waiting for peered", 2024-03-11T21:38:23.012 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:36:35.046911+0000", 2024-03-11T21:38:23.012 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 8.5900000000000008e-06 2024-03-11T21:38:23.012 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:38:23.013 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:38:23.013 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "reached_pg", 2024-03-11T21:38:23.013 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:36:36.524565+0000", 2024-03-11T21:38:23.013 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.4776533839999999 2024-03-11T21:38:23.013 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:38:23.013 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:38:23.013 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "started", 2024-03-11T21:38:23.013 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:36:36.524737+0000", 2024-03-11T21:38:23.013 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.00017203499999999999 2024-03-11T21:38:23.013 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:38:23.013 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:38:23.013 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "waiting for subops from 1,3", 2024-03-11T21:38:23.013 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:36:36.524977+0000", 2024-03-11T21:38:23.014 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.00023978000000000001 2024-03-11T21:38:23.014 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:38:23.014 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:38:23.014 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "op_commit", 2024-03-11T21:38:23.014 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:36:36.532361+0000", 2024-03-11T21:38:23.014 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.0073842630000000003 2024-03-11T21:38:23.014 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:38:23.014 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:38:23.014 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "sub_op_commit_rec", 2024-03-11T21:38:23.014 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:36:36.546393+0000", 2024-03-11T21:38:23.014 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.014032282 2024-03-11T21:38:23.014 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:38:23.014 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:38:23.015 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "sub_op_commit_rec", 2024-03-11T21:38:23.015 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:36:36.546479+0000", 2024-03-11T21:38:23.015 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 8.5855999999999999e-05 2024-03-11T21:38:23.015 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:38:23.015 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:38:23.015 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "commit_sent", 2024-03-11T21:38:23.015 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:36:36.546554+0000", 2024-03-11T21:38:23.015 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 7.4998000000000003e-05 2024-03-11T21:38:23.015 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:38:23.015 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:38:23.015 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "done", 2024-03-11T21:38:23.015 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:36:36.550170+0000", 2024-03-11T21:38:23.015 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.0036155139999999998 2024-03-11T21:38:23.015 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-03-11T21:38:23.015 INFO:teuthology.orchestra.run.smithi097.stdout: ] 2024-03-11T21:38:23.016 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-03-11T21:38:23.016 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:38:23.016 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:38:23.016 INFO:teuthology.orchestra.run.smithi097.stdout: "description": "osd_op(client.25282.0:76298 9.f 9:f515a6f8:::benchmark_data_smithi092_130291_object76297:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e277)", 2024-03-11T21:38:23.016 INFO:teuthology.orchestra.run.smithi097.stdout: "initiated_at": "2024-03-11T21:36:34.059383+0000", 2024-03-11T21:38:23.016 INFO:teuthology.orchestra.run.smithi097.stdout: "age": 108.200811887, 2024-03-11T21:38:23.016 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 2.4975940599999999, 2024-03-11T21:38:23.016 INFO:teuthology.orchestra.run.smithi097.stdout: "type_data": { 2024-03-11T21:38:23.016 INFO:teuthology.orchestra.run.smithi097.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-11T21:38:23.016 INFO:teuthology.orchestra.run.smithi097.stdout: "client_info": { 2024-03-11T21:38:23.016 INFO:teuthology.orchestra.run.smithi097.stdout: "client": "client.25282", 2024-03-11T21:38:23.016 INFO:teuthology.orchestra.run.smithi097.stdout: "client_addr": "v1:172.21.15.92:0/2489519821", 2024-03-11T21:38:23.016 INFO:teuthology.orchestra.run.smithi097.stdout: "tid": 76298 2024-03-11T21:38:23.016 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:38:23.016 INFO:teuthology.orchestra.run.smithi097.stdout: "events": [ 2024-03-11T21:38:23.016 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:38:23.017 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "initiated", 2024-03-11T21:38:23.017 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:36:34.059383+0000", 2024-03-11T21:38:23.017 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-03-11T21:38:23.017 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:38:23.017 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:38:23.017 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "header_read", 2024-03-11T21:38:23.017 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:36:34.059383+0000", 2024-03-11T21:38:23.017 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-03-11T21:38:23.017 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:38:23.017 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:38:23.017 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "throttled", 2024-03-11T21:38:23.017 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:36:34.059385+0000", 2024-03-11T21:38:23.017 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.7099999999999999e-06 2024-03-11T21:38:23.017 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:38:23.018 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:38:23.018 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "all_read", 2024-03-11T21:38:23.018 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:36:34.059449+0000", 2024-03-11T21:38:23.018 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 6.4513999999999999e-05 2024-03-11T21:38:23.018 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:38:23.018 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:38:23.018 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "dispatched", 2024-03-11T21:38:23.018 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:36:34.059459+0000", 2024-03-11T21:38:23.018 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.0175e-05 2024-03-11T21:38:23.018 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:38:23.018 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:38:23.018 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "queued_for_pg", 2024-03-11T21:38:23.018 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:36:34.059467+0000", 2024-03-11T21:38:23.018 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 7.2790000000000001e-06 2024-03-11T21:38:23.019 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:38:23.019 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:38:23.019 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "reached_pg", 2024-03-11T21:38:23.019 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:36:34.059525+0000", 2024-03-11T21:38:23.019 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 5.8496999999999999e-05 2024-03-11T21:38:23.019 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:38:23.019 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:38:23.019 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "waiting for peered", 2024-03-11T21:38:23.019 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:36:34.059534+0000", 2024-03-11T21:38:23.019 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 9.1519999999999998e-06 2024-03-11T21:38:23.019 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:38:23.019 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:38:23.019 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "reached_pg", 2024-03-11T21:38:23.019 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:36:35.046940+0000", 2024-03-11T21:38:23.019 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.98740518300000002 2024-03-11T21:38:23.019 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:38:23.020 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:38:23.020 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "waiting for peered", 2024-03-11T21:38:23.020 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:36:35.046947+0000", 2024-03-11T21:38:23.020 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 7.9759999999999995e-06 2024-03-11T21:38:23.020 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:38:23.020 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:38:23.020 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "reached_pg", 2024-03-11T21:38:23.020 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:36:36.528284+0000", 2024-03-11T21:38:23.020 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.481336177 2024-03-11T21:38:23.020 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:38:23.020 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:38:23.020 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "started", 2024-03-11T21:38:23.020 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:36:36.528443+0000", 2024-03-11T21:38:23.020 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.000159209 2024-03-11T21:38:23.020 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:38:23.021 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:38:23.021 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "waiting for subops from 1,3", 2024-03-11T21:38:23.021 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:36:36.528658+0000", 2024-03-11T21:38:23.021 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.00021467900000000001 2024-03-11T21:38:23.021 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:38:23.021 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:38:23.021 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "op_commit", 2024-03-11T21:38:23.021 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:36:36.532370+0000", 2024-03-11T21:38:23.021 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.0037121099999999998 2024-03-11T21:38:23.021 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:38:23.021 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:38:23.021 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "sub_op_commit_rec", 2024-03-11T21:38:23.021 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:36:36.546446+0000", 2024-03-11T21:38:23.021 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.014075966000000001 2024-03-11T21:38:23.022 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:38:23.022 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:38:23.022 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "sub_op_commit_rec", 2024-03-11T21:38:23.022 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:36:36.553366+0000", 2024-03-11T21:38:23.022 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.0069203570000000002 2024-03-11T21:38:23.022 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:38:23.022 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:38:23.022 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "commit_sent", 2024-03-11T21:38:23.022 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:36:36.553421+0000", 2024-03-11T21:38:23.022 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 5.5148000000000001e-05 2024-03-11T21:38:23.022 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:38:23.022 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:38:23.022 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "done", 2024-03-11T21:38:23.023 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:36:36.556977+0000", 2024-03-11T21:38:23.023 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.0035559279999999999 2024-03-11T21:38:23.023 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-03-11T21:38:23.023 INFO:teuthology.orchestra.run.smithi097.stdout: ] 2024-03-11T21:38:23.023 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-03-11T21:38:23.023 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:38:23.023 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:38:23.023 INFO:teuthology.orchestra.run.smithi097.stdout: "description": "osd_op(client.25282.0:76300 9.f 9:f8b776ca:::benchmark_data_smithi092_130291_object76299:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e277)", 2024-03-11T21:38:23.023 INFO:teuthology.orchestra.run.smithi097.stdout: "initiated_at": "2024-03-11T21:36:34.064758+0000", 2024-03-11T21:38:23.023 INFO:teuthology.orchestra.run.smithi097.stdout: "age": 108.195436953, 2024-03-11T21:38:23.023 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 2.5087678960000002, 2024-03-11T21:38:23.023 INFO:teuthology.orchestra.run.smithi097.stdout: "type_data": { 2024-03-11T21:38:23.023 INFO:teuthology.orchestra.run.smithi097.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-11T21:38:23.023 INFO:teuthology.orchestra.run.smithi097.stdout: "client_info": { 2024-03-11T21:38:23.023 INFO:teuthology.orchestra.run.smithi097.stdout: "client": "client.25282", 2024-03-11T21:38:23.024 INFO:teuthology.orchestra.run.smithi097.stdout: "client_addr": "v1:172.21.15.92:0/2489519821", 2024-03-11T21:38:23.024 INFO:teuthology.orchestra.run.smithi097.stdout: "tid": 76300 2024-03-11T21:38:23.024 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:38:23.024 INFO:teuthology.orchestra.run.smithi097.stdout: "events": [ 2024-03-11T21:38:23.024 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:38:23.024 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "initiated", 2024-03-11T21:38:23.024 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:36:34.064758+0000", 2024-03-11T21:38:23.024 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-03-11T21:38:23.024 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:38:23.024 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:38:23.024 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "header_read", 2024-03-11T21:38:23.024 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:36:34.064758+0000", 2024-03-11T21:38:23.024 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-03-11T21:38:23.024 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:38:23.025 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:38:23.025 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "throttled", 2024-03-11T21:38:23.025 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:36:34.064759+0000", 2024-03-11T21:38:23.025 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.012e-06 2024-03-11T21:38:23.025 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:38:23.025 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:38:23.025 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "all_read", 2024-03-11T21:38:23.025 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:36:34.064790+0000", 2024-03-11T21:38:23.025 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 3.0901000000000001e-05 2024-03-11T21:38:23.025 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:38:23.025 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:38:23.025 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "dispatched", 2024-03-11T21:38:23.025 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:36:34.064799+0000", 2024-03-11T21:38:23.025 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 9.1190000000000003e-06 2024-03-11T21:38:23.026 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:38:23.026 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:38:23.026 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "queued_for_pg", 2024-03-11T21:38:23.026 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:36:34.064807+0000", 2024-03-11T21:38:23.026 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 8.3610000000000001e-06 2024-03-11T21:38:23.026 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:38:23.026 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:38:23.026 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "reached_pg", 2024-03-11T21:38:23.026 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:36:34.064853+0000", 2024-03-11T21:38:23.026 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 4.5867e-05 2024-03-11T21:38:23.026 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:38:23.026 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:38:23.026 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "waiting for peered", 2024-03-11T21:38:23.026 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:36:34.064862+0000", 2024-03-11T21:38:23.026 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 8.8599999999999999e-06 2024-03-11T21:38:23.027 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:38:23.027 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:38:23.027 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "reached_pg", 2024-03-11T21:38:23.027 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:36:35.046976+0000", 2024-03-11T21:38:23.027 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.98211411100000001 2024-03-11T21:38:23.027 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:38:23.027 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:38:23.027 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "waiting for peered", 2024-03-11T21:38:23.027 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:36:35.046984+0000", 2024-03-11T21:38:23.027 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 7.9449999999999993e-06 2024-03-11T21:38:23.027 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:38:23.027 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:38:23.027 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "reached_pg", 2024-03-11T21:38:23.027 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:36:36.550220+0000", 2024-03-11T21:38:23.028 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.5032355509999999 2024-03-11T21:38:23.028 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:38:23.028 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:38:23.028 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "started", 2024-03-11T21:38:23.028 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:36:36.550382+0000", 2024-03-11T21:38:23.028 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.00016266499999999999 2024-03-11T21:38:23.028 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:38:23.028 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:38:23.028 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "waiting for subops from 1,3", 2024-03-11T21:38:23.028 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:36:36.550548+0000", 2024-03-11T21:38:23.028 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.00016551599999999999 2024-03-11T21:38:23.028 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:38:23.028 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:38:23.028 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "sub_op_commit_rec", 2024-03-11T21:38:23.028 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:36:36.560198+0000", 2024-03-11T21:38:23.029 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.009649899 2024-03-11T21:38:23.029 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:38:23.029 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:38:23.029 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "sub_op_commit_rec", 2024-03-11T21:38:23.029 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:36:36.560301+0000", 2024-03-11T21:38:23.029 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.000103558 2024-03-11T21:38:23.029 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:38:23.029 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:38:23.029 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "op_commit", 2024-03-11T21:38:23.029 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:36:36.569928+0000", 2024-03-11T21:38:23.029 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.0096262550000000002 2024-03-11T21:38:23.029 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:38:23.029 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:38:23.029 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "commit_sent", 2024-03-11T21:38:23.029 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:36:36.569984+0000", 2024-03-11T21:38:23.030 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 5.6273999999999997e-05 2024-03-11T21:38:23.030 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:38:23.030 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:38:23.030 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "done", 2024-03-11T21:38:23.030 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:36:36.573526+0000", 2024-03-11T21:38:23.030 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.0035420019999999998 2024-03-11T21:38:23.030 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-03-11T21:38:23.030 INFO:teuthology.orchestra.run.smithi097.stdout: ] 2024-03-11T21:38:23.030 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-03-11T21:38:23.030 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:38:23.030 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:38:23.030 INFO:teuthology.orchestra.run.smithi097.stdout: "description": "osd_op(client.25282.0:76308 9.f 9:fb5a14a2:::benchmark_data_smithi092_130291_object76307:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e277)", 2024-03-11T21:38:23.030 INFO:teuthology.orchestra.run.smithi097.stdout: "initiated_at": "2024-03-11T21:36:34.074082+0000", 2024-03-11T21:38:23.031 INFO:teuthology.orchestra.run.smithi097.stdout: "age": 108.186112474, 2024-03-11T21:38:23.031 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 2.503648359, 2024-03-11T21:38:23.031 INFO:teuthology.orchestra.run.smithi097.stdout: "type_data": { 2024-03-11T21:38:23.031 INFO:teuthology.orchestra.run.smithi097.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-11T21:38:23.031 INFO:teuthology.orchestra.run.smithi097.stdout: "client_info": { 2024-03-11T21:38:23.031 INFO:teuthology.orchestra.run.smithi097.stdout: "client": "client.25282", 2024-03-11T21:38:23.031 INFO:teuthology.orchestra.run.smithi097.stdout: "client_addr": "v1:172.21.15.92:0/2489519821", 2024-03-11T21:38:23.031 INFO:teuthology.orchestra.run.smithi097.stdout: "tid": 76308 2024-03-11T21:38:23.031 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:38:23.031 INFO:teuthology.orchestra.run.smithi097.stdout: "events": [ 2024-03-11T21:38:23.031 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:38:23.031 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "initiated", 2024-03-11T21:38:23.031 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:36:34.074082+0000", 2024-03-11T21:38:23.031 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-03-11T21:38:23.032 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:38:23.032 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:38:23.032 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "header_read", 2024-03-11T21:38:23.032 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:36:34.074082+0000", 2024-03-11T21:38:23.032 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-03-11T21:38:23.032 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:38:23.032 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:38:23.032 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "throttled", 2024-03-11T21:38:23.032 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:36:34.074085+0000", 2024-03-11T21:38:23.032 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 2.323e-06 2024-03-11T21:38:23.032 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:38:23.032 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:38:23.032 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "all_read", 2024-03-11T21:38:23.032 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:36:34.074145+0000", 2024-03-11T21:38:23.032 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 6.0176000000000001e-05 2024-03-11T21:38:23.033 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:38:23.033 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:38:23.033 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "dispatched", 2024-03-11T21:38:23.033 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:36:34.074158+0000", 2024-03-11T21:38:23.033 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.2911000000000001e-05 2024-03-11T21:38:23.033 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:38:23.033 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:38:23.033 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "queued_for_pg", 2024-03-11T21:38:23.033 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:36:34.074164+0000", 2024-03-11T21:38:23.033 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 5.9490000000000001e-06 2024-03-11T21:38:23.033 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:38:23.033 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:38:23.033 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "reached_pg", 2024-03-11T21:38:23.033 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:36:34.074208+0000", 2024-03-11T21:38:23.033 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 4.4642999999999999e-05 2024-03-11T21:38:23.033 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:38:23.033 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:38:23.034 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "waiting for peered", 2024-03-11T21:38:23.034 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:36:34.074215+0000", 2024-03-11T21:38:23.034 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 7.0010000000000004e-06 2024-03-11T21:38:23.034 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:38:23.034 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:38:23.034 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "reached_pg", 2024-03-11T21:38:23.034 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:36:35.047052+0000", 2024-03-11T21:38:23.034 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.97283687399999996 2024-03-11T21:38:23.034 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:38:23.034 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:38:23.034 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "waiting for peered", 2024-03-11T21:38:23.034 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:36:35.047063+0000", 2024-03-11T21:38:23.034 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.1052e-05 2024-03-11T21:38:23.035 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:38:23.035 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:38:23.035 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "reached_pg", 2024-03-11T21:38:23.035 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:36:36.557028+0000", 2024-03-11T21:38:23.035 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.5099643540000001 2024-03-11T21:38:23.035 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:38:23.035 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:38:23.035 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "started", 2024-03-11T21:38:23.035 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:36:36.557189+0000", 2024-03-11T21:38:23.035 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.00016171499999999999 2024-03-11T21:38:23.035 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:38:23.035 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:38:23.035 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "waiting for subops from 1,3", 2024-03-11T21:38:23.035 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:36:36.557367+0000", 2024-03-11T21:38:23.035 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.000177238 2024-03-11T21:38:23.036 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:38:23.036 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:38:23.036 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "sub_op_commit_rec", 2024-03-11T21:38:23.036 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:36:36.567005+0000", 2024-03-11T21:38:23.036 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.0096378829999999999 2024-03-11T21:38:23.036 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:38:23.036 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:38:23.036 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "op_commit", 2024-03-11T21:38:23.036 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:36:36.573545+0000", 2024-03-11T21:38:23.036 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.0065399819999999997 2024-03-11T21:38:23.036 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:38:23.036 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:38:23.036 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "sub_op_commit_rec", 2024-03-11T21:38:23.036 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:36:36.574036+0000", 2024-03-11T21:38:23.037 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.00049146699999999999 2024-03-11T21:38:23.037 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:38:23.037 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:38:23.037 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "commit_sent", 2024-03-11T21:38:23.037 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:36:36.574095+0000", 2024-03-11T21:38:23.037 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 5.8768999999999999e-05 2024-03-11T21:38:23.037 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:38:23.037 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:38:23.037 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "done", 2024-03-11T21:38:23.037 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:36:36.577731+0000", 2024-03-11T21:38:23.037 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.003636022 2024-03-11T21:38:23.037 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-03-11T21:38:23.037 INFO:teuthology.orchestra.run.smithi097.stdout: ] 2024-03-11T21:38:23.037 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-03-11T21:38:23.037 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-03-11T21:38:23.038 INFO:teuthology.orchestra.run.smithi097.stdout: ] 2024-03-11T21:38:23.038 INFO:teuthology.orchestra.run.smithi097.stdout:} 2024-03-11T21:38:23.038 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:38:23.068 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:38:23.069 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:38:23.069 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:38:23.077 INFO:journalctl@ceph.iscsi.iscsi.a.smithi097.stdout:Mar 11 21:38:22 smithi097 ceph-00db0016-dfed-11ee-95c8-87774f69a715-iscsi-iscsi-a[60943]: debug there is no tcmu-runner data available 2024-03-11T21:38:23.170 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:38:23.201 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:38:23.201 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:38:23.202 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:38:23.303 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:38:23.328 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:38:23.328 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:38:23.329 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:38:23.430 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:38:23.461 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:38:23.462 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:38:23.462 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:38:23.563 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:38:23.623 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:38:23.624 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:38:23.624 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:38:23.725 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:38:23.769 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:38:23.770 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:38:23.770 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:38:23.827 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:38:23 smithi097 ceph-mon[33317]: osdmap e317: 8 total, 8 up, 8 in 2024-03-11T21:38:23.827 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:38:23 smithi097 ceph-mon[33317]: from='client.24469 v1:172.21.15.97:0/404208310' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:38:23.827 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:38:23 smithi097 ceph-mon[33317]: Health check update: Degraded data redundancy: 27048/230574 objects degraded (11.731%), 8 pgs degraded, 8 pgs undersized (PG_DEGRADED) 2024-03-11T21:38:23.827 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:38:23 smithi097 ceph-mon[33317]: osdmap e318: 8 total, 8 up, 8 in 2024-03-11T21:38:23.861 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:38:23 smithi092 ceph-mon[29572]: osdmap e317: 8 total, 8 up, 8 in 2024-03-11T21:38:23.861 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:38:23 smithi092 ceph-mon[29572]: from='client.24469 v1:172.21.15.97:0/404208310' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:38:23.861 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:38:23 smithi092 ceph-mon[29572]: Health check update: Degraded data redundancy: 27048/230574 objects degraded (11.731%), 8 pgs degraded, 8 pgs undersized (PG_DEGRADED) 2024-03-11T21:38:23.862 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:38:23 smithi092 ceph-mon[29572]: osdmap e318: 8 total, 8 up, 8 in 2024-03-11T21:38:23.862 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:38:23 smithi092 ceph-mon[37563]: osdmap e317: 8 total, 8 up, 8 in 2024-03-11T21:38:23.862 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:38:23 smithi092 ceph-mon[37563]: from='client.24469 v1:172.21.15.97:0/404208310' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:38:23.862 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:38:23 smithi092 ceph-mon[37563]: Health check update: Degraded data redundancy: 27048/230574 objects degraded (11.731%), 8 pgs degraded, 8 pgs undersized (PG_DEGRADED) 2024-03-11T21:38:23.862 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:38:23 smithi092 ceph-mon[37563]: osdmap e318: 8 total, 8 up, 8 in 2024-03-11T21:38:23.871 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:38:23.905 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:38:23.906 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:38:23.906 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:38:23.918 DEBUG:teuthology.orchestra.run.smithi092:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid 00db0016-dfed-11ee-95c8-87774f69a715 -- ceph osd unset noscrub 2024-03-11T21:38:24.010 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:38:24.075 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:38:24.076 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:38:24.076 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:38:24.177 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:38:24.209 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:38:24.210 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:38:24.210 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:38:24.311 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:38:24.355 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:38:24.356 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:38:24.356 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:38:24.457 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:38:24.487 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:38:24.488 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:38:24.488 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:38:24.589 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:38:24.623 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:38:24.623 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:38:24.623 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:38:24.724 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:38:24.756 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:38:24.756 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:38:24.756 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:38:24.794 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:38:24 smithi092 ceph-mon[29572]: pgmap v594: 148 pgs: 4 active+recovery_wait+undersized+degraded+remapped, 4 active+recovering+undersized+degraded+remapped, 140 active+clean; 4.5 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 1.7 KiB/s rd, 0 B/s wr, 1.03k op/s; 26347/223467 objects degraded (11.790%); 13 MiB/s, 211 objects/s recovering 2024-03-11T21:38:24.794 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:38:24 smithi092 ceph-mon[37563]: pgmap v594: 148 pgs: 4 active+recovery_wait+undersized+degraded+remapped, 4 active+recovering+undersized+degraded+remapped, 140 active+clean; 4.5 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 1.7 KiB/s rd, 0 B/s wr, 1.03k op/s; 26347/223467 objects degraded (11.790%); 13 MiB/s, 211 objects/s recovering 2024-03-11T21:38:24.827 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:38:24 smithi097 ceph-mon[33317]: pgmap v594: 148 pgs: 4 active+recovery_wait+undersized+degraded+remapped, 4 active+recovering+undersized+degraded+remapped, 140 active+clean; 4.5 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 1.7 KiB/s rd, 0 B/s wr, 1.03k op/s; 26347/223467 objects degraded (11.790%); 13 MiB/s, 211 objects/s recovering 2024-03-11T21:38:24.858 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:38:24.890 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:38:24.891 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:38:24.891 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:38:24.992 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:38:25.042 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:38:25.042 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:38:25.043 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:38:25.144 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:38:25.170 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:38:25.170 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:38:25.171 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:38:25.271 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:38:25.277 DEBUG:teuthology.orchestra.run.smithi097:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid 00db0016-dfed-11ee-95c8-87774f69a715 -- ceph daemon osd.6 dump_ops_in_flight 2024-03-11T21:38:25.304 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:38:25.304 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:38:25.304 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:38:25.405 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:38:25.434 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:38:25.438 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:38:25.439 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:38:25.485 INFO:teuthology.orchestra.run.smithi092.stdout:osd.0: {} 2024-03-11T21:38:25.536 INFO:teuthology.orchestra.run.smithi092.stdout:osd.1: {} 2024-03-11T21:38:25.540 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:38:25.572 INFO:teuthology.orchestra.run.smithi092.stdout:osd.2: {} 2024-03-11T21:38:25.589 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:38:25.589 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:38:25.590 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:38:25.615 INFO:teuthology.orchestra.run.smithi092.stdout:osd.3: {} 2024-03-11T21:38:25.665 INFO:teuthology.orchestra.run.smithi092.stdout:osd.4: {} 2024-03-11T21:38:25.666 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:38:25 smithi092 ceph-mon[29572]: from='client.? v1:172.21.15.92:0/3129905650' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-11T21:38:25.666 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:38:25 smithi092 ceph-mon[37563]: from='client.? v1:172.21.15.92:0/3129905650' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-11T21:38:25.672 INFO:teuthology.orchestra.run.smithi092.stderr:Inferring config /var/lib/ceph/00db0016-dfed-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T21:38:25.691 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:38:25.707 INFO:teuthology.orchestra.run.smithi092.stdout:osd.5: {} 2024-03-11T21:38:25.753 INFO:teuthology.orchestra.run.smithi092.stdout:osd.6: {} 2024-03-11T21:38:25.763 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:38:25.763 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:38:25.763 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:38:25.801 INFO:teuthology.orchestra.run.smithi092.stdout:osd.7: {} 2024-03-11T21:38:25.806 INFO:teuthology.orchestra.run.smithi092.stderr:osd.0: osd_enable_op_tracker = '' 2024-03-11T21:38:25.806 INFO:teuthology.orchestra.run.smithi092.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-11T21:38:25.806 INFO:teuthology.orchestra.run.smithi092.stderr:osd.2: osd_enable_op_tracker = '' 2024-03-11T21:38:25.806 INFO:teuthology.orchestra.run.smithi092.stderr:osd.3: osd_enable_op_tracker = '' 2024-03-11T21:38:25.806 INFO:teuthology.orchestra.run.smithi092.stderr:osd.4: osd_enable_op_tracker = '' 2024-03-11T21:38:25.807 INFO:teuthology.orchestra.run.smithi092.stderr:osd.5: osd_enable_op_tracker = '' 2024-03-11T21:38:25.807 INFO:teuthology.orchestra.run.smithi092.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-11T21:38:25.807 INFO:teuthology.orchestra.run.smithi092.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-11T21:38:25.827 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:38:25 smithi097 ceph-mon[33317]: from='client.? v1:172.21.15.92:0/3129905650' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-11T21:38:25.865 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:38:25.893 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:38:25.894 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:38:25.895 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:38:25.995 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:38:26.051 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:38:26.051 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:38:26.052 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:38:26.153 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:38:26.184 INFO:teuthology.orchestra.run.smithi097.stderr:Inferring config /var/lib/ceph/00db0016-dfed-11ee-95c8-87774f69a715/mon.b/config 2024-03-11T21:38:26.193 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:38:26.194 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:38:26.194 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:38:26.242 INFO:teuthology.orchestra.run.smithi092.stdout: 2024-03-11T21:38:26.242 INFO:teuthology.orchestra.run.smithi092.stdout:{"fsid":"00db0016-dfed-11ee-95c8-87774f69a715","health":{"status":"HEALTH_WARN","checks":{"OSDMAP_FLAGS":{"severity":"HEALTH_WARN","summary":{"message":"noscrub,nodeep-scrub flag(s) set","count":20},"muted":false},"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 26008/220062 objects degraded (11.818%), 7 pgs degraded, 7 pgs undersized","count":14},"muted":false}},"mutes":[]},"election_epoch":14,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":988,"monmap":{"epoch":3,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":318,"num_osds":8,"num_up_osds":8,"osd_up_since":1710192994,"num_in_osds":8,"osd_in_since":1710192308,"num_remapped_pgs":7},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":141},{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":4},{"state_name":"active+recovering+undersized+degraded+remapped","count":3}],"num_pgs":148,"num_pools":7,"num_objects":73354,"data_bytes":4794882496,"bytes_used":17134202880,"bytes_avail":750826082304,"bytes_total":767960285184,"degraded_objects":26008,"degraded_total":220062,"degraded_ratio":0.11818487517154257,"recovering_objects_per_sec":180,"recovering_bytes_per_sec":11837755,"recovering_keys_per_sec":0,"num_objects_recovered":1040,"num_bytes_recovered":68157440,"num_keys_recovered":0,"read_bytes_sec":889,"write_bytes_sec":0,"read_op_per_sec":0,"write_op_per_sec":608},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":1,"modules":["cephadm","dashboard","iostat","nfs","prometheus","restful"],"services":{"dashboard":"https://172.21.15.92:8443/","prometheus":"http://172.21.15.92:9283/"}},"servicemap":{"epoch":15,"modified":"2024-03-11T21:37:33.231049+0000","services":{"rgw":{"daemons":{"summary":"","34352":{"start_epoch":12,"start_stamp":"2024-03-11T21:25:45.672905+0000","gid":34352,"addr":"172.21.15.92:0/372197479","metadata":{"arch":"x86_64","ceph_release":"squid","ceph_version":"ceph version 19.0.0-1979-g2b0101eb (2b0101eb74d60610d544cb8a473e23a35ac1cc0b) squid (dev)","ceph_version_short":"19.0.0-1979-g2b0101eb","container_hostname":"smithi092","container_image":"quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:8fe9635f3beb0247496d7e321f3068bd10c333c433ef4824d7e1c1a7d3d78379","cpu":"Intel(R) Xeon(R) CPU E5-1620 v4 @ 3.50GHz","distro":"centos","distro_description":"CentOS Stream 8","distro_version":"8","frontend_config#0":"beast port=80","frontend_type#0":"beast","hostname":"smithi092","id":"foo.a","kernel_description":"#1 SMP PREEMPT_DYNAMIC Fri Feb 23 04:45:07 UTC 2024","kernel_version":"5.14.0-427.el9.x86_64","mem_swap_kb":"0","mem_total_kb":"32488440","num_handles":"1","os":"Linux","pid":"2","realm_id":"","realm_name":"","zone_id":"bf7aa506-b210-476a-b68a-1feac2caf199","zone_name":"default","zonegroup_id":"33e24078-41e3-4494-bd9e-6ff3741afaba","zonegroup_name":"default"},"task_status":{}}}}}},"progress_events":{}} 2024-03-11T21:38:26.294 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:38:26.340 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:38:26.341 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:38:26.341 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:38:26.442 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:38:26.484 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:38:26.484 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:38:26.484 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:38:26.585 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:38:26.617 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:38:26.618 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:38:26.618 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:38:26.719 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:38:26.759 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:38:26.759 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:38:26.760 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:38:26.808 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:38:26 smithi097 ceph-mon[33317]: pgmap v596: 148 pgs: 4 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+degraded+remapped, 141 active+clean; 4.5 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 889 B/s rd, 0 B/s wr, 609 op/s; 26008/220062 objects degraded (11.818%); 11 MiB/s, 180 objects/s recovering 2024-03-11T21:38:26.808 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:38:26 smithi097 ceph-mon[33317]: purged_snaps scrub starts 2024-03-11T21:38:26.809 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:38:26 smithi097 ceph-mon[33317]: purged_snaps scrub ok 2024-03-11T21:38:26.809 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:38:26 smithi097 ceph-mon[33317]: from='client.? v1:172.21.15.92:0/1960534869' entity='client.admin' cmd=[{"prefix": "status", "format": "json"}]: dispatch 2024-03-11T21:38:26.861 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:38:26.866 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:38:26 smithi092 ceph-mon[29572]: pgmap v596: 148 pgs: 4 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+degraded+remapped, 141 active+clean; 4.5 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 889 B/s rd, 0 B/s wr, 609 op/s; 26008/220062 objects degraded (11.818%); 11 MiB/s, 180 objects/s recovering 2024-03-11T21:38:26.866 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:38:26 smithi092 ceph-mon[29572]: purged_snaps scrub starts 2024-03-11T21:38:26.866 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:38:26 smithi092 ceph-mon[29572]: purged_snaps scrub ok 2024-03-11T21:38:26.866 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:38:26 smithi092 ceph-mon[29572]: from='client.? v1:172.21.15.92:0/1960534869' entity='client.admin' cmd=[{"prefix": "status", "format": "json"}]: dispatch 2024-03-11T21:38:26.867 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:38:26 smithi092 ceph-mon[37563]: pgmap v596: 148 pgs: 4 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+degraded+remapped, 141 active+clean; 4.5 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 889 B/s rd, 0 B/s wr, 609 op/s; 26008/220062 objects degraded (11.818%); 11 MiB/s, 180 objects/s recovering 2024-03-11T21:38:26.867 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:38:26 smithi092 ceph-mon[37563]: purged_snaps scrub starts 2024-03-11T21:38:26.867 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:38:26 smithi092 ceph-mon[37563]: purged_snaps scrub ok 2024-03-11T21:38:26.867 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:38:26 smithi092 ceph-mon[37563]: from='client.? v1:172.21.15.92:0/1960534869' entity='client.admin' cmd=[{"prefix": "status", "format": "json"}]: dispatch 2024-03-11T21:38:26.898 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:38:26.898 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:38:26.898 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:38:26.999 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:38:27.028 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:38:27.029 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:38:27.029 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:38:27.130 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:38:27.160 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:38:27.160 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:38:27.160 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:38:27.261 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:38:27.292 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:38:27.293 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:38:27.293 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:38:27.394 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:38:27.427 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:38:27.428 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:38:27.428 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:38:27.529 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:38:27.566 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:38:27.566 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:38:27.567 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:38:27.668 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:38:27.698 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:38:27.698 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:38:27.698 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:38:27.754 INFO:tasks.cephadm.ceph_manager.ceph:making progress, resetting timeout 2024-03-11T21:38:27.755 DEBUG:teuthology.orchestra.run.smithi092:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid 00db0016-dfed-11ee-95c8-87774f69a715 -- ceph pg dump --format=json 2024-03-11T21:38:27.799 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:38:27.827 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:38:27 smithi097 ceph-mon[33317]: osdmap e319: 8 total, 8 up, 8 in 2024-03-11T21:38:27.836 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:38:27.837 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:38:27.837 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:38:27.861 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:38:27 smithi092 ceph-mon[29572]: osdmap e319: 8 total, 8 up, 8 in 2024-03-11T21:38:27.862 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:38:27 smithi092 ceph-mon[37563]: osdmap e319: 8 total, 8 up, 8 in 2024-03-11T21:38:27.938 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:38:27.979 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:38:27.980 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:38:27.980 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:38:28.081 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:38:28.121 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:38:28.122 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:38:28.123 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:38:28.223 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:38:28.261 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:38:28.261 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:38:28.262 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:38:28.361 INFO:journalctl@ceph.mgr.y.smithi092.stdout:Mar 11 21:38:27 smithi092 ceph-00db0016-dfed-11ee-95c8-87774f69a715-mgr-y[29801]: ::ffff:172.21.15.97 - - [11/Mar/2024:21:38:27] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-11T21:38:28.363 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:38:28.401 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:38:28.402 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:38:28.402 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:38:28.503 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:38:28.542 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:38:28.542 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:38:28.543 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:38:28.644 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:38:28.663 DEBUG:teuthology.orchestra.run.smithi092:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid 00db0016-dfed-11ee-95c8-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-03-11T21:38:28.684 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:38:28.685 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:38:28.685 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:38:28.720 INFO:teuthology.orchestra.run.smithi097.stdout:ERROR: (22) Invalid argument 2024-03-11T21:38:28.720 INFO:teuthology.orchestra.run.smithi097.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-11T21:38:28.786 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:38:28.827 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:38:28 smithi097 ceph-mon[33317]: pgmap v598: 148 pgs: 4 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+degraded+remapped, 141 active+clean; 4.5 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 891 B/s rd, 0 B/s wr, 649 op/s; 25662/219402 objects degraded (11.696%); 15 MiB/s, 241 objects/s recovering 2024-03-11T21:38:28.827 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:38:28 smithi097 ceph-mon[33317]: Health check update: Degraded data redundancy: 25662/219402 objects degraded (11.696%), 7 pgs degraded, 7 pgs undersized (PG_DEGRADED) 2024-03-11T21:38:28.827 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:38:28 smithi097 ceph-mon[33317]: osdmap e320: 8 total, 8 up, 8 in 2024-03-11T21:38:28.861 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:38:28 smithi092 ceph-mon[29572]: pgmap v598: 148 pgs: 4 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+degraded+remapped, 141 active+clean; 4.5 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 891 B/s rd, 0 B/s wr, 649 op/s; 25662/219402 objects degraded (11.696%); 15 MiB/s, 241 objects/s recovering 2024-03-11T21:38:28.861 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:38:28 smithi092 ceph-mon[29572]: Health check update: Degraded data redundancy: 25662/219402 objects degraded (11.696%), 7 pgs degraded, 7 pgs undersized (PG_DEGRADED) 2024-03-11T21:38:28.861 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:38:28 smithi092 ceph-mon[29572]: osdmap e320: 8 total, 8 up, 8 in 2024-03-11T21:38:28.862 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:38:28 smithi092 ceph-mon[37563]: pgmap v598: 148 pgs: 4 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+degraded+remapped, 141 active+clean; 4.5 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 891 B/s rd, 0 B/s wr, 649 op/s; 25662/219402 objects degraded (11.696%); 15 MiB/s, 241 objects/s recovering 2024-03-11T21:38:28.862 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:38:28 smithi092 ceph-mon[37563]: Health check update: Degraded data redundancy: 25662/219402 objects degraded (11.696%), 7 pgs degraded, 7 pgs undersized (PG_DEGRADED) 2024-03-11T21:38:28.862 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:38:28 smithi092 ceph-mon[37563]: osdmap e320: 8 total, 8 up, 8 in 2024-03-11T21:38:28.862 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:38:28.863 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:38:28.863 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:38:28.964 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:38:28.995 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:38:28.996 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:38:28.996 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:38:29.097 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:38:29.123 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:38:29.124 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:38:29.124 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:38:29.225 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:38:29.253 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:38:29.253 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:38:29.254 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:38:29.354 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:38:29.379 INFO:teuthology.orchestra.run.smithi092.stderr:Inferring config /var/lib/ceph/00db0016-dfed-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T21:38:29.408 DEBUG:teuthology.orchestra.run.smithi097:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid 00db0016-dfed-11ee-95c8-87774f69a715 -- ceph daemon osd.6 dump_blocked_ops 2024-03-11T21:38:29.409 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:38:29.410 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:38:29.410 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:38:29.483 INFO:teuthology.orchestra.run.smithi092.stderr:Inferring config /var/lib/ceph/00db0016-dfed-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T21:38:29.511 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:38:29.565 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:38:29.566 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:38:29.566 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:38:29.667 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:38:29.705 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:38:29.706 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:38:29.706 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:38:29.747 INFO:teuthology.orchestra.run.smithi097.stderr:Inferring config /var/lib/ceph/00db0016-dfed-11ee-95c8-87774f69a715/mon.b/config 2024-03-11T21:38:29.807 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:38:29.833 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:38:29.834 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:38:29.834 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:38:29.935 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:38:29.960 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:38:29.961 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:38:29.961 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:38:30.062 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:38:30.087 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:38:30.088 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:38:30.088 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:38:30.189 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:38:30.511 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:38:30.512 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:38:30.512 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:38:30.593 INFO:teuthology.orchestra.run.smithi092.stderr:noscrub is unset 2024-03-11T21:38:30.613 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:38:30.646 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:38:30.646 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:38:30.647 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:38:30.748 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:38:30.786 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:38:30.787 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:38:30.787 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:38:30.862 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:38:30 smithi092 ceph-mon[29572]: pgmap v600: 148 pgs: 4 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+degraded+remapped, 141 active+clean; 4.2 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 851 B/s rd, 0 B/s wr, 864 op/s; 25355/207894 objects degraded (12.196%); 10 MiB/s, 164 objects/s recovering 2024-03-11T21:38:30.862 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:38:30 smithi092 ceph-mon[29572]: osdmap e321: 8 total, 8 up, 8 in 2024-03-11T21:38:30.862 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:38:30 smithi092 ceph-mon[29572]: from='client.? v1:172.21.15.92:0/2292093308' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-11T21:38:30.862 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:38:30 smithi092 ceph-mon[29572]: from='client.25456 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-11T21:38:30.862 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:38:30 smithi092 ceph-mon[37563]: pgmap v600: 148 pgs: 4 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+degraded+remapped, 141 active+clean; 4.2 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 851 B/s rd, 0 B/s wr, 864 op/s; 25355/207894 objects degraded (12.196%); 10 MiB/s, 164 objects/s recovering 2024-03-11T21:38:30.862 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:38:30 smithi092 ceph-mon[37563]: osdmap e321: 8 total, 8 up, 8 in 2024-03-11T21:38:30.862 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:38:30 smithi092 ceph-mon[37563]: from='client.? v1:172.21.15.92:0/2292093308' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-11T21:38:30.862 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:38:30 smithi092 ceph-mon[37563]: from='client.25456 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-11T21:38:30.888 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:38:30.931 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:38:30.932 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:38:30.933 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:38:30.973 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:38:30 smithi097 ceph-mon[33317]: pgmap v600: 148 pgs: 4 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+degraded+remapped, 141 active+clean; 4.2 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 851 B/s rd, 0 B/s wr, 864 op/s; 25355/207894 objects degraded (12.196%); 10 MiB/s, 164 objects/s recovering 2024-03-11T21:38:30.973 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:38:30 smithi097 ceph-mon[33317]: osdmap e321: 8 total, 8 up, 8 in 2024-03-11T21:38:30.973 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:38:30 smithi097 ceph-mon[33317]: from='client.? v1:172.21.15.92:0/2292093308' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-11T21:38:30.973 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:38:30 smithi097 ceph-mon[33317]: from='client.25456 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-11T21:38:31.033 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:38:31.078 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:38:31.078 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:38:31.079 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:38:31.180 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:38:31.227 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:38:31.227 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:38:31.228 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:38:31.329 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:38:31.362 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:38:31.363 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:38:31.363 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:38:31.464 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:38:31.494 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:38:31.495 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:38:31.495 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:38:31.596 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:38:31.627 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:38:31.628 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:38:31.628 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:38:31.729 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:38:31.730 INFO:teuthology.orchestra.run.smithi097.stdout:ERROR: (22) Invalid argument 2024-03-11T21:38:31.730 INFO:teuthology.orchestra.run.smithi097.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-11T21:38:31.761 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:38:31.762 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:38:31.762 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:38:31.827 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:38:31 smithi097 ceph-mon[33317]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-11T21:38:31.828 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:38:31 smithi097 ceph-mon[33317]: from='client.25456 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-03-11T21:38:31.828 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:38:31 smithi097 ceph-mon[33317]: osdmap e322: 8 total, 8 up, 8 in 2024-03-11T21:38:31.861 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:38:31 smithi092 ceph-mon[29572]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-11T21:38:31.862 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:38:31 smithi092 ceph-mon[29572]: from='client.25456 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-03-11T21:38:31.862 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:38:31 smithi092 ceph-mon[29572]: osdmap e322: 8 total, 8 up, 8 in 2024-03-11T21:38:31.862 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:38:31 smithi092 ceph-mon[37563]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-11T21:38:31.862 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:38:31 smithi092 ceph-mon[37563]: from='client.25456 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-03-11T21:38:31.862 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:38:31 smithi092 ceph-mon[37563]: osdmap e322: 8 total, 8 up, 8 in 2024-03-11T21:38:31.862 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:38:31.891 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:38:31.892 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:38:31.892 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:38:31.992 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:38:32.050 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:38:32.050 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:38:32.050 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:38:32.151 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:38:32.181 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:38:32.182 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:38:32.182 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:38:32.283 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:38:32.318 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:38:32.319 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:38:32.319 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:38:32.420 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:38:32.475 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:38:32.478 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:38:32.478 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:38:32.579 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:38:32.586 DEBUG:teuthology.orchestra.run.smithi097:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid 00db0016-dfed-11ee-95c8-87774f69a715 -- ceph daemon osd.6 dump_historic_ops 2024-03-11T21:38:32.638 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:38:32.638 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:38:32.639 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:38:32.740 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:38:32.781 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:38:32.782 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:38:32.782 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:38:32.827 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:38:32 smithi097 ceph-mon[33317]: pgmap v603: 148 pgs: 4 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+degraded+remapped, 141 active+clean; 4.2 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 1.1 KiB/s rd, 0 B/s wr, 864 op/s; 25355/207894 objects degraded (12.196%); 8.7 MiB/s, 139 objects/s recovering 2024-03-11T21:38:32.827 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:38:32 smithi097 ceph-mon[33317]: from='mgr.14607 v1:172.21.15.92:0/1702717597' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T21:38:32.828 INFO:journalctl@ceph.iscsi.iscsi.a.smithi097.stdout:Mar 11 21:38:32 smithi097 ceph-00db0016-dfed-11ee-95c8-87774f69a715-iscsi-iscsi-a[60943]: debug there is no tcmu-runner data available 2024-03-11T21:38:32.867 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:38:32 smithi092 ceph-mon[29572]: pgmap v603: 148 pgs: 4 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+degraded+remapped, 141 active+clean; 4.2 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 1.1 KiB/s rd, 0 B/s wr, 864 op/s; 25355/207894 objects degraded (12.196%); 8.7 MiB/s, 139 objects/s recovering 2024-03-11T21:38:32.867 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:38:32 smithi092 ceph-mon[29572]: from='mgr.14607 v1:172.21.15.92:0/1702717597' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T21:38:32.873 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:38:32 smithi092 ceph-mon[37563]: pgmap v603: 148 pgs: 4 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+degraded+remapped, 141 active+clean; 4.2 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 1.1 KiB/s rd, 0 B/s wr, 864 op/s; 25355/207894 objects degraded (12.196%); 8.7 MiB/s, 139 objects/s recovering 2024-03-11T21:38:32.873 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:38:32 smithi092 ceph-mon[37563]: from='mgr.14607 v1:172.21.15.92:0/1702717597' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T21:38:32.884 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:38:32.920 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:38:32.921 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:38:32.921 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:38:32.933 INFO:teuthology.orchestra.run.smithi097.stderr:Inferring config /var/lib/ceph/00db0016-dfed-11ee-95c8-87774f69a715/mon.b/config 2024-03-11T21:38:33.022 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:38:33.051 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:38:33.052 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:38:33.052 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:38:33.153 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:38:33.207 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:38:33.208 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:38:33.208 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:38:33.309 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:38:33.335 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:38:33.336 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:38:33.336 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:38:33.437 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:38:33.489 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:38:33.489 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:38:33.489 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:38:33.590 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:38:33.619 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:38:33.620 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:38:33.620 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:38:33.721 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:38:33.768 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:38:33.769 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:38:33.769 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:38:33.870 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:38:33.912 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:38:33.913 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:38:33.913 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:38:33.926 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:38:33 smithi092 ceph-mon[29572]: from='client.24469 v1:172.21.15.97:0/404208310' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:38:33.927 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:38:33 smithi092 ceph-mon[29572]: Health check update: Degraded data redundancy: 25355/207894 objects degraded (12.196%), 7 pgs degraded, 7 pgs undersized (PG_DEGRADED) 2024-03-11T21:38:33.927 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:38:33 smithi092 ceph-mon[37563]: from='client.24469 v1:172.21.15.97:0/404208310' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:38:33.927 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:38:33 smithi092 ceph-mon[37563]: Health check update: Degraded data redundancy: 25355/207894 objects degraded (12.196%), 7 pgs degraded, 7 pgs undersized (PG_DEGRADED) 2024-03-11T21:38:33.932 INFO:teuthology.orchestra.run.smithi092.stdout: 2024-03-11T21:38:33.942 INFO:teuthology.orchestra.run.smithi092.stderr:dumped all 2024-03-11T21:38:34.014 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:38:34.049 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:38:34.050 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:38:34.050 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:38:34.078 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:38:33 smithi097 ceph-mon[33317]: from='client.24469 v1:172.21.15.97:0/404208310' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:38:34.078 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:38:33 smithi097 ceph-mon[33317]: Health check update: Degraded data redundancy: 25355/207894 objects degraded (12.196%), 7 pgs degraded, 7 pgs undersized (PG_DEGRADED) 2024-03-11T21:38:34.151 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:38:34.194 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:38:34.194 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:38:34.195 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:38:34.295 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:38:34.329 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:38:34.330 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:38:34.330 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:38:34.430 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:38:34.477 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:38:34.478 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:38:34.478 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:38:34.579 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:38:34.611 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:38:34.612 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:38:34.612 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:38:34.713 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:38:34.743 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:38:34.743 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:38:34.744 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:38:34.769 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:38:34 smithi092 ceph-mon[29572]: pgmap v604: 148 pgs: 3 active+recovery_wait+undersized+degraded+remapped, 4 active+recovering+undersized+degraded+remapped, 141 active+clean; 4.1 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 1.7 KiB/s rd, 0 B/s wr, 983 op/s; 24747/201696 objects degraded (12.269%); 9.5 MiB/s, 152 objects/s recovering 2024-03-11T21:38:34.770 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:38:34 smithi092 ceph-mon[29572]: from='mgr.14607 v1:172.21.15.92:0/1702717597' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T21:38:34.770 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:38:34 smithi092 ceph-mon[29572]: from='mgr.14607 v1:172.21.15.92:0/1702717597' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T21:38:34.770 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:38:34 smithi092 ceph-mon[29572]: from='mgr.14607 ' entity='mgr.y' 2024-03-11T21:38:34.770 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:38:34 smithi092 ceph-mon[29572]: from='client.25462 v1:172.21.15.92:0/1099051884' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-11T21:38:34.770 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:38:34 smithi092 ceph-mon[37563]: pgmap v604: 148 pgs: 3 active+recovery_wait+undersized+degraded+remapped, 4 active+recovering+undersized+degraded+remapped, 141 active+clean; 4.1 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 1.7 KiB/s rd, 0 B/s wr, 983 op/s; 24747/201696 objects degraded (12.269%); 9.5 MiB/s, 152 objects/s recovering 2024-03-11T21:38:34.770 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:38:34 smithi092 ceph-mon[37563]: from='mgr.14607 v1:172.21.15.92:0/1702717597' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T21:38:34.770 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:38:34 smithi092 ceph-mon[37563]: from='mgr.14607 v1:172.21.15.92:0/1702717597' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T21:38:34.770 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:38:34 smithi092 ceph-mon[37563]: from='mgr.14607 ' entity='mgr.y' 2024-03-11T21:38:34.770 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:38:34 smithi092 ceph-mon[37563]: from='client.25462 v1:172.21.15.92:0/1099051884' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-11T21:38:34.828 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:38:34 smithi097 ceph-mon[33317]: pgmap v604: 148 pgs: 3 active+recovery_wait+undersized+degraded+remapped, 4 active+recovering+undersized+degraded+remapped, 141 active+clean; 4.1 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 1.7 KiB/s rd, 0 B/s wr, 983 op/s; 24747/201696 objects degraded (12.269%); 9.5 MiB/s, 152 objects/s recovering 2024-03-11T21:38:34.828 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:38:34 smithi097 ceph-mon[33317]: from='mgr.14607 v1:172.21.15.92:0/1702717597' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T21:38:34.828 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:38:34 smithi097 ceph-mon[33317]: from='mgr.14607 v1:172.21.15.92:0/1702717597' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T21:38:34.828 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:38:34 smithi097 ceph-mon[33317]: from='mgr.14607 ' entity='mgr.y' 2024-03-11T21:38:34.828 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:38:34 smithi097 ceph-mon[33317]: from='client.25462 v1:172.21.15.92:0/1099051884' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-11T21:38:34.844 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:38:34.883 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:38:34.884 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:38:34.884 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:38:34.985 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:38:35.022 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:38:35.023 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:38:35.023 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:38:35.124 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:38:35.156 INFO:teuthology.orchestra.run.smithi092.stdout:{"pg_ready":true,"pg_map":{"version":604,"stamp":"2024-03-11T21:38:33.269404+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":4393671104,"num_objects":67232,"num_object_clones":0,"num_object_copies":201696,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":24747,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":67232,"num_whiteouts":0,"num_read":2103,"num_read_kb":1951,"num_write":166631,"num_write_kb":7450846,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13479,"num_bytes_recovered":873011654,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":83718,"ondisk_log_size":83718,"up":444,"acting":437,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":444,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":749961216,"kb_used":15641076,"kb_used_data":12211232,"kb_used_omap":15,"kb_used_meta":3429616,"kb_avail":734320140,"statfs":{"total":767960285184,"available":751943823360,"internally_reserved":0,"allocated":12504301568,"data_stored":12497898020,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15601,"internal_metadata":3511927567},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,3,2,0,2],"upper_bound":32},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-386793472,"num_objects":-5902,"num_object_clones":0,"num_object_copies":-17706,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-915,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-5902,"num_whiteouts":0,"num_read":10,"num_read_kb":10,"num_write":5902,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":915,"num_bytes_recovered":59965440,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.012332"},"pg_stats":[{"pgid":"6.1b","version":"0'0","reported_seq":489,"reported_epoch":320,"state":"active+clean","last_fresh":"2024-03-11T21:38:28.363875+0000","last_change":"2024-03-11T21:36:35.074931+0000","last_active":"2024-03-11T21:38:28.363875+0000","last_peered":"2024-03-11T21:38:28.363875+0000","last_clean":"2024-03-11T21:38:28.363875+0000","last_became_active":"2024-03-11T21:36:35.072636+0000","last_became_peered":"2024-03-11T21:36:35.072636+0000","last_unstale":"2024-03-11T21:38:28.363875+0000","last_undegraded":"2024-03-11T21:38:28.363875+0000","last_fullsized":"2024-03-11T21:38:28.363875+0000","mapping_epoch":277,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":278,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:35:07.021520+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:43.559452+0000","last_clean_scrub_stamp":"2024-03-11T21:35:07.021520+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":558,"reported_epoch":322,"state":"active+clean","last_fresh":"2024-03-11T21:38:30.601247+0000","last_change":"2024-03-11T21:36:00.720089+0000","last_active":"2024-03-11T21:38:30.601247+0000","last_peered":"2024-03-11T21:38:30.601247+0000","last_clean":"2024-03-11T21:38:30.601247+0000","last_became_active":"2024-03-11T21:25:42.578893+0000","last_became_peered":"2024-03-11T21:25:42.578893+0000","last_unstale":"2024-03-11T21:38:30.601247+0000","last_undegraded":"2024-03-11T21:38:30.601247+0000","last_fullsized":"2024-03-11T21:38:30.601247+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:36:00.719956+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:41.544550+0000","last_clean_scrub_stamp":"2024-03-11T21:36:00.719956+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1f","version":"0'0","reported_seq":583,"reported_epoch":320,"state":"active+clean","last_fresh":"2024-03-11T21:38:28.354073+0000","last_change":"2024-03-11T21:36:00.843838+0000","last_active":"2024-03-11T21:38:28.354073+0000","last_peered":"2024-03-11T21:38:28.354073+0000","last_clean":"2024-03-11T21:38:28.354073+0000","last_became_active":"2024-03-11T21:34:41.144945+0000","last_became_peered":"2024-03-11T21:34:41.144945+0000","last_unstale":"2024-03-11T21:38:28.354073+0000","last_undegraded":"2024-03-11T21:38:28.354073+0000","last_fullsized":"2024-03-11T21:38:28.354073+0000","mapping_epoch":226,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":227,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:36:00.843718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:37.512018+0000","last_clean_scrub_stamp":"2024-03-11T21:36:00.843718+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,4],"acting":[0,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.19","version":"260'27","reported_seq":536,"reported_epoch":320,"state":"active+clean","last_fresh":"2024-03-11T21:38:28.362560+0000","last_change":"2024-03-11T21:36:35.089901+0000","last_active":"2024-03-11T21:38:28.362560+0000","last_peered":"2024-03-11T21:38:28.362560+0000","last_clean":"2024-03-11T21:38:28.362560+0000","last_became_active":"2024-03-11T21:36:35.067652+0000","last_became_peered":"2024-03-11T21:36:35.067652+0000","last_unstale":"2024-03-11T21:38:28.362560+0000","last_undegraded":"2024-03-11T21:38:28.362560+0000","last_fullsized":"2024-03-11T21:38:28.362560+0000","mapping_epoch":277,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":278,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'15","last_scrub_stamp":"2024-03-11T21:35:22.099374+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:39.525554+0000","last_clean_scrub_stamp":"2024-03-11T21:35:22.099374+0000","objects_scrubbed":0,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":42,"num_read_kb":33,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.1a","version":"0'0","reported_seq":546,"reported_epoch":322,"state":"active+clean","last_fresh":"2024-03-11T21:38:30.601342+0000","last_change":"2024-03-11T21:35:16.410927+0000","last_active":"2024-03-11T21:38:30.601342+0000","last_peered":"2024-03-11T21:38:30.601342+0000","last_clean":"2024-03-11T21:38:30.601342+0000","last_became_active":"2024-03-11T21:25:44.590923+0000","last_became_peered":"2024-03-11T21:25:44.590923+0000","last_unstale":"2024-03-11T21:38:30.601342+0000","last_undegraded":"2024-03-11T21:38:30.601342+0000","last_fullsized":"2024-03-11T21:38:30.601342+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:35:16.410816+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:43.559452+0000","last_clean_scrub_stamp":"2024-03-11T21:35:16.410816+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":24,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,1],"acting":[4,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":570,"reported_epoch":322,"state":"active+clean","last_fresh":"2024-03-11T21:38:30.602061+0000","last_change":"2024-03-11T21:36:15.576138+0000","last_active":"2024-03-11T21:38:30.602061+0000","last_peered":"2024-03-11T21:38:30.602061+0000","last_clean":"2024-03-11T21:38:30.602061+0000","last_became_active":"2024-03-11T21:34:41.154216+0000","last_became_peered":"2024-03-11T21:34:41.154216+0000","last_unstale":"2024-03-11T21:38:30.602061+0000","last_undegraded":"2024-03-11T21:38:30.602061+0000","last_fullsized":"2024-03-11T21:38:30.602061+0000","mapping_epoch":226,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":227,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:36:15.575887+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:41.544550+0000","last_clean_scrub_stamp":"2024-03-11T21:36:15.575887+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,7],"acting":[1,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1e","version":"0'0","reported_seq":502,"reported_epoch":320,"state":"active+clean","last_fresh":"2024-03-11T21:38:28.363432+0000","last_change":"2024-03-11T21:36:35.062413+0000","last_active":"2024-03-11T21:38:28.363432+0000","last_peered":"2024-03-11T21:38:28.363432+0000","last_clean":"2024-03-11T21:38:28.363432+0000","last_became_active":"2024-03-11T21:36:35.061786+0000","last_became_peered":"2024-03-11T21:36:35.061786+0000","last_unstale":"2024-03-11T21:38:28.363432+0000","last_undegraded":"2024-03-11T21:38:28.363432+0000","last_fullsized":"2024-03-11T21:38:28.363432+0000","mapping_epoch":277,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":278,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:32:18.578038+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:37.512018+0000","last_clean_scrub_stamp":"2024-03-11T21:32:18.578038+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.18","version":"260'15","reported_seq":595,"reported_epoch":322,"state":"active+clean","last_fresh":"2024-03-11T21:38:30.601091+0000","last_change":"2024-03-11T21:36:09.662765+0000","last_active":"2024-03-11T21:38:30.601091+0000","last_peered":"2024-03-11T21:38:30.601091+0000","last_clean":"2024-03-11T21:38:30.601091+0000","last_became_active":"2024-03-11T21:25:40.552216+0000","last_became_peered":"2024-03-11T21:25:40.552216+0000","last_unstale":"2024-03-11T21:38:30.601091+0000","last_undegraded":"2024-03-11T21:38:30.601091+0000","last_fullsized":"2024-03-11T21:38:30.601091+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"260'15","last_scrub_stamp":"2024-03-11T21:36:09.662615+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:39.525554+0000","last_clean_scrub_stamp":"2024-03-11T21:36:09.662615+0000","objects_scrubbed":4,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":562,"reported_epoch":322,"state":"active+clean","last_fresh":"2024-03-11T21:38:30.615369+0000","last_change":"2024-03-11T21:36:35.068915+0000","last_active":"2024-03-11T21:38:30.615369+0000","last_peered":"2024-03-11T21:38:30.615369+0000","last_clean":"2024-03-11T21:38:30.615369+0000","last_became_active":"2024-03-11T21:36:35.068453+0000","last_became_peered":"2024-03-11T21:36:35.068453+0000","last_unstale":"2024-03-11T21:38:30.615369+0000","last_undegraded":"2024-03-11T21:38:30.615369+0000","last_fullsized":"2024-03-11T21:38:30.615369+0000","mapping_epoch":277,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":278,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:35:26.190634+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:32:12.654573+0000","last_clean_scrub_stamp":"2024-03-11T21:35:26.190634+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3],"acting":[5,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":210,"reported_epoch":322,"state":"active+clean","last_fresh":"2024-03-11T21:38:30.607565+0000","last_change":"2024-03-11T21:35:11.902920+0000","last_active":"2024-03-11T21:38:30.607565+0000","last_peered":"2024-03-11T21:38:30.607565+0000","last_clean":"2024-03-11T21:38:30.607565+0000","last_became_active":"2024-03-11T21:34:41.145299+0000","last_became_peered":"2024-03-11T21:34:41.145299+0000","last_unstale":"2024-03-11T21:38:30.607565+0000","last_undegraded":"2024-03-11T21:38:30.607565+0000","last_fullsized":"2024-03-11T21:38:30.607565+0000","mapping_epoch":226,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":227,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:35:11.902818+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:41.544550+0000","last_clean_scrub_stamp":"2024-03-11T21:35:11.902818+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.1d","version":"0'0","reported_seq":573,"reported_epoch":322,"state":"active+clean","last_fresh":"2024-03-11T21:38:30.607926+0000","last_change":"2024-03-11T21:35:26.855588+0000","last_active":"2024-03-11T21:38:30.607926+0000","last_peered":"2024-03-11T21:38:30.607926+0000","last_clean":"2024-03-11T21:38:30.607926+0000","last_became_active":"2024-03-11T21:25:38.542628+0000","last_became_peered":"2024-03-11T21:25:38.542628+0000","last_unstale":"2024-03-11T21:38:30.607926+0000","last_undegraded":"2024-03-11T21:38:30.607926+0000","last_fullsized":"2024-03-11T21:38:30.607926+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:35:26.855511+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:37.512018+0000","last_clean_scrub_stamp":"2024-03-11T21:35:26.855511+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,0],"acting":[7,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.1b","version":"260'7","reported_seq":581,"reported_epoch":322,"state":"active+clean","last_fresh":"2024-03-11T21:38:30.601478+0000","last_change":"2024-03-11T21:36:35.087276+0000","last_active":"2024-03-11T21:38:30.601478+0000","last_peered":"2024-03-11T21:38:30.601478+0000","last_clean":"2024-03-11T21:38:30.601478+0000","last_became_active":"2024-03-11T21:36:35.057678+0000","last_became_peered":"2024-03-11T21:36:35.057678+0000","last_unstale":"2024-03-11T21:38:30.601478+0000","last_undegraded":"2024-03-11T21:38:30.601478+0000","last_fullsized":"2024-03-11T21:38:30.601478+0000","mapping_epoch":277,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":278,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'5","last_scrub_stamp":"2024-03-11T21:35:09.427562+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:39.525554+0000","last_clean_scrub_stamp":"2024-03-11T21:35:09.427562+0000","objects_scrubbed":2,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":14,"num_read_kb":10,"num_write":8,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,1],"acting":[4,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":550,"reported_epoch":320,"state":"active+clean","last_fresh":"2024-03-11T21:38:28.353832+0000","last_change":"2024-03-11T21:35:39.691825+0000","last_active":"2024-03-11T21:38:28.353832+0000","last_peered":"2024-03-11T21:38:28.353832+0000","last_clean":"2024-03-11T21:38:28.353832+0000","last_became_active":"2024-03-11T21:25:44.589786+0000","last_became_peered":"2024-03-11T21:25:44.589786+0000","last_unstale":"2024-03-11T21:38:28.353832+0000","last_undegraded":"2024-03-11T21:38:28.353832+0000","last_fullsized":"2024-03-11T21:38:28.353832+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:35:39.691690+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:43.559452+0000","last_clean_scrub_stamp":"2024-03-11T21:35:39.691690+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":18,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,7],"acting":[0,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":569,"reported_epoch":322,"state":"active+clean","last_fresh":"2024-03-11T21:38:30.615481+0000","last_change":"2024-03-11T21:35:25.194240+0000","last_active":"2024-03-11T21:38:30.615481+0000","last_peered":"2024-03-11T21:38:30.615481+0000","last_clean":"2024-03-11T21:38:30.615481+0000","last_became_active":"2024-03-11T21:34:41.147185+0000","last_became_peered":"2024-03-11T21:34:41.147185+0000","last_unstale":"2024-03-11T21:38:30.615481+0000","last_undegraded":"2024-03-11T21:38:30.615481+0000","last_fullsized":"2024-03-11T21:38:30.615481+0000","mapping_epoch":226,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":227,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:35:25.194115+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:32:21.535034+0000","last_clean_scrub_stamp":"2024-03-11T21:35:25.194115+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,7],"acting":[5,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1c","version":"0'0","reported_seq":573,"reported_epoch":322,"state":"active+clean","last_fresh":"2024-03-11T21:38:30.607963+0000","last_change":"2024-03-11T21:35:28.789693+0000","last_active":"2024-03-11T21:38:30.607963+0000","last_peered":"2024-03-11T21:38:30.607963+0000","last_clean":"2024-03-11T21:38:30.607963+0000","last_became_active":"2024-03-11T21:25:38.541364+0000","last_became_peered":"2024-03-11T21:25:38.541364+0000","last_unstale":"2024-03-11T21:38:30.607963+0000","last_undegraded":"2024-03-11T21:38:30.607963+0000","last_fullsized":"2024-03-11T21:38:30.607963+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:35:28.789592+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:28:32.566437+0000","last_clean_scrub_stamp":"2024-03-11T21:35:28.789592+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.1a","version":"260'15","reported_seq":605,"reported_epoch":322,"state":"active+clean","last_fresh":"2024-03-11T21:38:30.601347+0000","last_change":"2024-03-11T21:36:35.085358+0000","last_active":"2024-03-11T21:38:30.601347+0000","last_peered":"2024-03-11T21:38:30.601347+0000","last_clean":"2024-03-11T21:38:30.601347+0000","last_became_active":"2024-03-11T21:36:35.057029+0000","last_became_peered":"2024-03-11T21:36:35.057029+0000","last_unstale":"2024-03-11T21:38:30.601347+0000","last_undegraded":"2024-03-11T21:38:30.601347+0000","last_fullsized":"2024-03-11T21:38:30.601347+0000","mapping_epoch":277,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":278,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'9","last_scrub_stamp":"2024-03-11T21:35:05.451780+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:39.525554+0000","last_clean_scrub_stamp":"2024-03-11T21:35:05.451780+0000","objects_scrubbed":4,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0],"acting":[4,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1f","version":"0'0","reported_seq":425,"reported_epoch":320,"state":"active+clean","last_fresh":"2024-03-11T21:38:28.363420+0000","last_change":"2024-03-11T21:36:35.074675+0000","last_active":"2024-03-11T21:38:28.363420+0000","last_peered":"2024-03-11T21:38:28.363420+0000","last_clean":"2024-03-11T21:38:28.363420+0000","last_became_active":"2024-03-11T21:36:35.071755+0000","last_became_peered":"2024-03-11T21:36:35.071755+0000","last_unstale":"2024-03-11T21:38:28.363420+0000","last_undegraded":"2024-03-11T21:38:28.363420+0000","last_fullsized":"2024-03-11T21:38:28.363420+0000","mapping_epoch":277,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":278,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:34:05.862891+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:43.559452+0000","last_clean_scrub_stamp":"2024-03-11T21:34:05.862891+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1c","version":"0'0","reported_seq":559,"reported_epoch":322,"state":"active+clean","last_fresh":"2024-03-11T21:38:30.601805+0000","last_change":"2024-03-11T21:36:35.067469+0000","last_active":"2024-03-11T21:38:30.601805+0000","last_peered":"2024-03-11T21:38:30.601805+0000","last_clean":"2024-03-11T21:38:30.601805+0000","last_became_active":"2024-03-11T21:36:35.067127+0000","last_became_peered":"2024-03-11T21:36:35.067127+0000","last_unstale":"2024-03-11T21:38:30.601805+0000","last_undegraded":"2024-03-11T21:38:30.601805+0000","last_fullsized":"2024-03-11T21:38:30.601805+0000","mapping_epoch":277,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":278,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:34:11.766066+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:41.544550+0000","last_clean_scrub_stamp":"2024-03-11T21:34:11.766066+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1b","version":"0'0","reported_seq":474,"reported_epoch":320,"state":"active+clean","last_fresh":"2024-03-11T21:38:28.363962+0000","last_change":"2024-03-11T21:36:35.075385+0000","last_active":"2024-03-11T21:38:28.363962+0000","last_peered":"2024-03-11T21:38:28.363962+0000","last_clean":"2024-03-11T21:38:28.363962+0000","last_became_active":"2024-03-11T21:36:35.073959+0000","last_became_peered":"2024-03-11T21:36:35.073959+0000","last_unstale":"2024-03-11T21:38:28.363962+0000","last_undegraded":"2024-03-11T21:38:28.363962+0000","last_fullsized":"2024-03-11T21:38:28.363962+0000","mapping_epoch":277,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":278,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:34:27.998724+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:34:27.998724+0000","last_clean_scrub_stamp":"2024-03-11T21:34:27.998724+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1d","version":"260'18","reported_seq":518,"reported_epoch":320,"state":"active+clean","last_fresh":"2024-03-11T21:38:28.363922+0000","last_change":"2024-03-11T21:36:35.080207+0000","last_active":"2024-03-11T21:38:28.363922+0000","last_peered":"2024-03-11T21:38:28.363922+0000","last_clean":"2024-03-11T21:38:28.363922+0000","last_became_active":"2024-03-11T21:36:35.055880+0000","last_became_peered":"2024-03-11T21:36:35.055880+0000","last_unstale":"2024-03-11T21:38:28.363922+0000","last_undegraded":"2024-03-11T21:38:28.363922+0000","last_fullsized":"2024-03-11T21:38:28.363922+0000","mapping_epoch":277,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":278,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'12","last_scrub_stamp":"2024-03-11T21:35:07.971581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:39.525554+0000","last_clean_scrub_stamp":"2024-03-11T21:35:07.971581+0000","objects_scrubbed":0,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":33,"num_read_kb":24,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.1e","version":"0'0","reported_seq":547,"reported_epoch":322,"state":"active+clean","last_fresh":"2024-03-11T21:38:30.601801+0000","last_change":"2024-03-11T21:35:32.664451+0000","last_active":"2024-03-11T21:38:30.601801+0000","last_peered":"2024-03-11T21:38:30.601801+0000","last_clean":"2024-03-11T21:38:30.601801+0000","last_became_active":"2024-03-11T21:25:45.300355+0000","last_became_peered":"2024-03-11T21:25:45.300355+0000","last_unstale":"2024-03-11T21:38:30.601801+0000","last_undegraded":"2024-03-11T21:38:30.601801+0000","last_fullsized":"2024-03-11T21:38:30.601801+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:35:32.664371+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:43.559452+0000","last_clean_scrub_stamp":"2024-03-11T21:35:32.664371+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5],"acting":[4,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1d","version":"0'0","reported_seq":558,"reported_epoch":322,"state":"active+clean","last_fresh":"2024-03-11T21:38:30.602432+0000","last_change":"2024-03-11T21:36:07.624196+0000","last_active":"2024-03-11T21:38:30.602432+0000","last_peered":"2024-03-11T21:38:30.602432+0000","last_clean":"2024-03-11T21:38:30.602432+0000","last_became_active":"2024-03-11T21:25:42.579407+0000","last_became_peered":"2024-03-11T21:25:42.579407+0000","last_unstale":"2024-03-11T21:38:30.602432+0000","last_undegraded":"2024-03-11T21:38:30.602432+0000","last_fullsized":"2024-03-11T21:38:30.602432+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:36:07.624075+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:30:18.678531+0000","last_clean_scrub_stamp":"2024-03-11T21:36:07.624075+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1a","version":"0'0","reported_seq":586,"reported_epoch":322,"state":"active+clean","last_fresh":"2024-03-11T21:38:30.669294+0000","last_change":"2024-03-11T21:36:12.993542+0000","last_active":"2024-03-11T21:38:30.669294+0000","last_peered":"2024-03-11T21:38:30.669294+0000","last_clean":"2024-03-11T21:38:30.669294+0000","last_became_active":"2024-03-11T21:34:41.145014+0000","last_became_peered":"2024-03-11T21:34:41.145014+0000","last_unstale":"2024-03-11T21:38:30.669294+0000","last_undegraded":"2024-03-11T21:38:30.669294+0000","last_fullsized":"2024-03-11T21:38:30.669294+0000","mapping_epoch":226,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":227,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:36:12.993437+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:37.512018+0000","last_clean_scrub_stamp":"2024-03-11T21:36:12.993437+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:38:32.120542+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.1c","version":"260'27","reported_seq":620,"reported_epoch":322,"state":"active+clean","last_fresh":"2024-03-11T21:38:30.658857+0000","last_change":"2024-03-11T21:36:35.097208+0000","last_active":"2024-03-11T21:38:30.658857+0000","last_peered":"2024-03-11T21:38:30.658857+0000","last_clean":"2024-03-11T21:38:30.658857+0000","last_became_active":"2024-03-11T21:36:35.074579+0000","last_became_peered":"2024-03-11T21:36:35.074579+0000","last_unstale":"2024-03-11T21:38:30.658857+0000","last_undegraded":"2024-03-11T21:38:30.658857+0000","last_fullsized":"2024-03-11T21:38:30.658857+0000","mapping_epoch":277,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":278,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'15","last_scrub_stamp":"2024-03-11T21:34:10.311181+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:39.525554+0000","last_clean_scrub_stamp":"2024-03-11T21:34:10.311181+0000","objects_scrubbed":7,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":44,"num_read_kb":35,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.1d","version":"0'0","reported_seq":546,"reported_epoch":322,"state":"active+clean","last_fresh":"2024-03-11T21:38:30.602029+0000","last_change":"2024-03-11T21:35:21.744646+0000","last_active":"2024-03-11T21:38:30.602029+0000","last_peered":"2024-03-11T21:38:30.602029+0000","last_clean":"2024-03-11T21:38:30.602029+0000","last_became_active":"2024-03-11T21:25:44.588845+0000","last_became_peered":"2024-03-11T21:25:44.588845+0000","last_unstale":"2024-03-11T21:38:30.602029+0000","last_undegraded":"2024-03-11T21:38:30.602029+0000","last_fullsized":"2024-03-11T21:38:30.602029+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:35:21.744223+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:43.559452+0000","last_clean_scrub_stamp":"2024-03-11T21:35:21.744223+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":99,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.1e","version":"0'0","reported_seq":555,"reported_epoch":320,"state":"active+clean","last_fresh":"2024-03-11T21:38:28.354485+0000","last_change":"2024-03-11T21:35:10.728691+0000","last_active":"2024-03-11T21:38:28.354485+0000","last_peered":"2024-03-11T21:38:28.354485+0000","last_clean":"2024-03-11T21:38:28.354485+0000","last_became_active":"2024-03-11T21:25:42.582959+0000","last_became_peered":"2024-03-11T21:25:42.582959+0000","last_unstale":"2024-03-11T21:38:28.354485+0000","last_undegraded":"2024-03-11T21:38:28.354485+0000","last_fullsized":"2024-03-11T21:38:28.354485+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:35:10.728569+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:26:58.335431+0000","last_clean_scrub_stamp":"2024-03-11T21:35:10.728569+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":18,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.19","version":"67'2","reported_seq":500,"reported_epoch":320,"state":"active+clean","last_fresh":"2024-03-11T21:38:28.362456+0000","last_change":"2024-03-11T21:36:35.069729+0000","last_active":"2024-03-11T21:38:28.362456+0000","last_peered":"2024-03-11T21:38:28.362456+0000","last_clean":"2024-03-11T21:38:28.362456+0000","last_became_active":"2024-03-11T21:36:35.064249+0000","last_became_peered":"2024-03-11T21:36:35.064249+0000","last_unstale":"2024-03-11T21:38:28.362456+0000","last_undegraded":"2024-03-11T21:38:28.362456+0000","last_fullsized":"2024-03-11T21:38:28.362456+0000","mapping_epoch":277,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":278,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'2","last_scrub_stamp":"2024-03-11T21:32:15.509542+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:37.512018+0000","last_clean_scrub_stamp":"2024-03-11T21:32:15.509542+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":957,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,0],"acting":[3,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1f","version":"260'24","reported_seq":625,"reported_epoch":322,"state":"active+clean","last_fresh":"2024-03-11T21:38:30.670030+0000","last_change":"2024-03-11T21:35:38.173937+0000","last_active":"2024-03-11T21:38:30.670030+0000","last_peered":"2024-03-11T21:38:30.670030+0000","last_clean":"2024-03-11T21:38:30.670030+0000","last_became_active":"2024-03-11T21:32:22.798242+0000","last_became_peered":"2024-03-11T21:32:22.798242+0000","last_unstale":"2024-03-11T21:38:30.670030+0000","last_undegraded":"2024-03-11T21:38:30.670030+0000","last_fullsized":"2024-03-11T21:38:30.670030+0000","mapping_epoch":166,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":167,"parent":"0.0","parent_split_bits":0,"last_scrub":"124'16","last_scrub_stamp":"2024-03-11T21:35:38.173841+0000","last_deep_scrub":"124'16","last_deep_scrub_stamp":"2024-03-11T21:35:38.173841+0000","last_clean_scrub_stamp":"2024-03-11T21:35:38.173841+0000","objects_scrubbed":5,"log_size":24,"log_dups_size":0,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:38:32.120379+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":32,"num_read_kb":25,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,0],"acting":[6,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.1c","version":"74'1","reported_seq":214,"reported_epoch":322,"state":"active+clean","last_fresh":"2024-03-11T21:38:30.607942+0000","last_change":"2024-03-11T21:36:11.732043+0000","last_active":"2024-03-11T21:38:30.607942+0000","last_peered":"2024-03-11T21:38:30.607942+0000","last_clean":"2024-03-11T21:38:30.607942+0000","last_became_active":"2024-03-11T21:34:41.144839+0000","last_became_peered":"2024-03-11T21:34:41.144839+0000","last_unstale":"2024-03-11T21:38:30.607942+0000","last_undegraded":"2024-03-11T21:38:30.607942+0000","last_fullsized":"2024-03-11T21:38:30.607942+0000","mapping_epoch":226,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":227,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'1","last_scrub_stamp":"2024-03-11T21:36:11.731914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:43.559452+0000","last_clean_scrub_stamp":"2024-03-11T21:36:11.731914+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":370,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":370,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.1f","version":"229'10","reported_seq":761,"reported_epoch":322,"state":"active+clean","last_fresh":"2024-03-11T21:38:30.669996+0000","last_change":"2024-03-11T21:36:08.871358+0000","last_active":"2024-03-11T21:38:30.669996+0000","last_peered":"2024-03-11T21:38:30.669996+0000","last_clean":"2024-03-11T21:38:30.669996+0000","last_became_active":"2024-03-11T21:34:41.145862+0000","last_became_peered":"2024-03-11T21:34:41.145862+0000","last_unstale":"2024-03-11T21:38:30.669996+0000","last_undegraded":"2024-03-11T21:38:30.669996+0000","last_fullsized":"2024-03-11T21:38:30.669996+0000","mapping_epoch":226,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":227,"parent":"0.0","parent_split_bits":0,"last_scrub":"229'10","last_scrub_stamp":"2024-03-11T21:36:08.871213+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:41.544550+0000","last_clean_scrub_stamp":"2024-03-11T21:36:08.871213+0000","objects_scrubbed":1,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:38:32.120479+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.18","version":"67'1","reported_seq":609,"reported_epoch":322,"state":"active+clean","last_fresh":"2024-03-11T21:38:30.615282+0000","last_change":"2024-03-11T21:36:10.458150+0000","last_active":"2024-03-11T21:38:30.615282+0000","last_peered":"2024-03-11T21:38:30.615282+0000","last_clean":"2024-03-11T21:38:30.615282+0000","last_became_active":"2024-03-11T21:35:23.310241+0000","last_became_peered":"2024-03-11T21:35:23.310241+0000","last_unstale":"2024-03-11T21:38:30.615282+0000","last_undegraded":"2024-03-11T21:38:30.615282+0000","last_fullsized":"2024-03-11T21:38:30.615282+0000","mapping_epoch":248,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":249,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'1","last_scrub_stamp":"2024-03-11T21:36:10.458028+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:37.512018+0000","last_clean_scrub_stamp":"2024-03-11T21:36:10.458028+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":436,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":7,"num_read_kb":7,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":1308,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,2],"acting":[5,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1e","version":"260'20","reported_seq":647,"reported_epoch":320,"state":"active+clean","last_fresh":"2024-03-11T21:38:28.354389+0000","last_change":"2024-03-11T21:36:35.083447+0000","last_active":"2024-03-11T21:38:28.354389+0000","last_peered":"2024-03-11T21:38:28.354389+0000","last_clean":"2024-03-11T21:38:28.354389+0000","last_became_active":"2024-03-11T21:36:35.057843+0000","last_became_peered":"2024-03-11T21:36:35.057843+0000","last_unstale":"2024-03-11T21:38:28.354389+0000","last_undegraded":"2024-03-11T21:38:28.354389+0000","last_fullsized":"2024-03-11T21:38:28.354389+0000","mapping_epoch":277,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":278,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'10","last_scrub_stamp":"2024-03-11T21:35:16.825146+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:39.525554+0000","last_clean_scrub_stamp":"2024-03-11T21:35:16.825146+0000","objects_scrubbed":5,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":44,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":25,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.13","version":"0'0","reported_seq":433,"reported_epoch":320,"state":"active+clean","last_fresh":"2024-03-11T21:38:28.364244+0000","last_change":"2024-03-11T21:36:35.063744+0000","last_active":"2024-03-11T21:38:28.364244+0000","last_peered":"2024-03-11T21:38:28.364244+0000","last_clean":"2024-03-11T21:38:28.364244+0000","last_became_active":"2024-03-11T21:36:35.062618+0000","last_became_peered":"2024-03-11T21:36:35.062618+0000","last_unstale":"2024-03-11T21:38:28.364244+0000","last_undegraded":"2024-03-11T21:38:28.364244+0000","last_fullsized":"2024-03-11T21:38:28.364244+0000","mapping_epoch":277,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":278,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:34:13.822803+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:43.559452+0000","last_clean_scrub_stamp":"2024-03-11T21:34:13.822803+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":210,"reported_epoch":322,"state":"active+clean","last_fresh":"2024-03-11T21:38:30.607225+0000","last_change":"2024-03-11T21:35:16.917648+0000","last_active":"2024-03-11T21:38:30.607225+0000","last_peered":"2024-03-11T21:38:30.607225+0000","last_clean":"2024-03-11T21:38:30.607225+0000","last_became_active":"2024-03-11T21:34:41.142569+0000","last_became_peered":"2024-03-11T21:34:41.142569+0000","last_unstale":"2024-03-11T21:38:30.607225+0000","last_undegraded":"2024-03-11T21:38:30.607225+0000","last_fullsized":"2024-03-11T21:38:30.607225+0000","mapping_epoch":226,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":227,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:35:16.917532+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:41.544550+0000","last_clean_scrub_stamp":"2024-03-11T21:35:16.917532+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.17","version":"0'0","reported_seq":566,"reported_epoch":322,"state":"active+clean","last_fresh":"2024-03-11T21:38:30.669915+0000","last_change":"2024-03-11T21:35:06.186953+0000","last_active":"2024-03-11T21:38:30.669915+0000","last_peered":"2024-03-11T21:38:30.669915+0000","last_clean":"2024-03-11T21:38:30.669915+0000","last_became_active":"2024-03-11T21:25:38.540349+0000","last_became_peered":"2024-03-11T21:25:38.540349+0000","last_unstale":"2024-03-11T21:38:30.669915+0000","last_undegraded":"2024-03-11T21:38:30.669915+0000","last_fullsized":"2024-03-11T21:38:30.669915+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:35:06.186840+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:28:25.918101+0000","last_clean_scrub_stamp":"2024-03-11T21:35:06.186840+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:38:32.120255+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,2],"acting":[6,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.11","version":"260'19","reported_seq":559,"reported_epoch":320,"state":"active+clean","last_fresh":"2024-03-11T21:38:28.362912+0000","last_change":"2024-03-11T21:36:35.096720+0000","last_active":"2024-03-11T21:38:28.362912+0000","last_peered":"2024-03-11T21:38:28.362912+0000","last_clean":"2024-03-11T21:38:28.362912+0000","last_became_active":"2024-03-11T21:36:35.073689+0000","last_became_peered":"2024-03-11T21:36:35.073689+0000","last_unstale":"2024-03-11T21:38:28.362912+0000","last_undegraded":"2024-03-11T21:38:28.362912+0000","last_fullsized":"2024-03-11T21:38:28.362912+0000","mapping_epoch":277,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":278,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'11","last_scrub_stamp":"2024-03-11T21:35:16.044397+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:39.525554+0000","last_clean_scrub_stamp":"2024-03-11T21:35:16.044397+0000","objects_scrubbed":5,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":31,"num_read_kb":24,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":210,"reported_epoch":322,"state":"active+clean","last_fresh":"2024-03-11T21:38:30.607339+0000","last_change":"2024-03-11T21:35:32.719869+0000","last_active":"2024-03-11T21:38:30.607339+0000","last_peered":"2024-03-11T21:38:30.607339+0000","last_clean":"2024-03-11T21:38:30.607339+0000","last_became_active":"2024-03-11T21:34:41.137235+0000","last_became_peered":"2024-03-11T21:34:41.137235+0000","last_unstale":"2024-03-11T21:38:30.607339+0000","last_undegraded":"2024-03-11T21:38:30.607339+0000","last_fullsized":"2024-03-11T21:38:30.607339+0000","mapping_epoch":226,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":227,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:35:32.719789+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:43.559452+0000","last_clean_scrub_stamp":"2024-03-11T21:35:32.719789+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,4],"acting":[7,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":578,"reported_epoch":322,"state":"active+clean","last_fresh":"2024-03-11T21:38:30.669409+0000","last_change":"2024-03-11T21:35:31.111662+0000","last_active":"2024-03-11T21:38:30.669409+0000","last_peered":"2024-03-11T21:38:30.669409+0000","last_clean":"2024-03-11T21:38:30.669409+0000","last_became_active":"2024-03-11T21:34:41.156036+0000","last_became_peered":"2024-03-11T21:34:41.156036+0000","last_unstale":"2024-03-11T21:38:30.669409+0000","last_undegraded":"2024-03-11T21:38:30.669409+0000","last_fullsized":"2024-03-11T21:38:30.669409+0000","mapping_epoch":226,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":227,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:35:31.111559+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:29:39.923895+0000","last_clean_scrub_stamp":"2024-03-11T21:35:31.111559+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:38:32.120316+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.16","version":"0'0","reported_seq":573,"reported_epoch":322,"state":"active+clean","last_fresh":"2024-03-11T21:38:30.615820+0000","last_change":"2024-03-11T21:35:39.108623+0000","last_active":"2024-03-11T21:38:30.615820+0000","last_peered":"2024-03-11T21:38:30.615820+0000","last_clean":"2024-03-11T21:38:30.615820+0000","last_became_active":"2024-03-11T21:25:38.539103+0000","last_became_peered":"2024-03-11T21:25:38.539103+0000","last_unstale":"2024-03-11T21:38:30.615820+0000","last_undegraded":"2024-03-11T21:38:30.615820+0000","last_fullsized":"2024-03-11T21:38:30.615820+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:35:39.108546+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:27:11.938909+0000","last_clean_scrub_stamp":"2024-03-11T21:35:39.108546+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,2],"acting":[5,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.10","version":"260'10","reported_seq":441,"reported_epoch":320,"state":"active+clean","last_fresh":"2024-03-11T21:38:28.385492+0000","last_change":"2024-03-11T21:36:35.084268+0000","last_active":"2024-03-11T21:38:28.385492+0000","last_peered":"2024-03-11T21:38:28.385492+0000","last_clean":"2024-03-11T21:38:28.385492+0000","last_became_active":"2024-03-11T21:36:35.067500+0000","last_became_peered":"2024-03-11T21:36:35.067500+0000","last_unstale":"2024-03-11T21:38:28.385492+0000","last_undegraded":"2024-03-11T21:38:28.385492+0000","last_fullsized":"2024-03-11T21:38:28.385492+0000","mapping_epoch":277,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":278,"parent":"0.0","parent_split_bits":0,"last_scrub":"94'6","last_scrub_stamp":"2024-03-11T21:34:00.727416+0000","last_deep_scrub":"94'6","last_deep_scrub_stamp":"2024-03-11T21:28:24.603974+0000","last_clean_scrub_stamp":"2024-03-11T21:34:00.727416+0000","objects_scrubbed":2,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":12,"num_read_kb":10,"num_write":8,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6],"acting":[3,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":486,"reported_epoch":320,"state":"active+clean","last_fresh":"2024-03-11T21:38:28.362742+0000","last_change":"2024-03-11T21:36:35.070468+0000","last_active":"2024-03-11T21:38:28.362742+0000","last_peered":"2024-03-11T21:38:28.362742+0000","last_clean":"2024-03-11T21:38:28.362742+0000","last_became_active":"2024-03-11T21:36:35.070176+0000","last_became_peered":"2024-03-11T21:36:35.070176+0000","last_unstale":"2024-03-11T21:38:28.362742+0000","last_undegraded":"2024-03-11T21:38:28.362742+0000","last_fullsized":"2024-03-11T21:38:28.362742+0000","mapping_epoch":277,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":278,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:35:18.062272+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:43.559452+0000","last_clean_scrub_stamp":"2024-03-11T21:35:18.062272+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":559,"reported_epoch":322,"state":"active+clean","last_fresh":"2024-03-11T21:38:30.606855+0000","last_change":"2024-03-11T21:36:35.058551+0000","last_active":"2024-03-11T21:38:30.606855+0000","last_peered":"2024-03-11T21:38:30.606855+0000","last_clean":"2024-03-11T21:38:30.606855+0000","last_became_active":"2024-03-11T21:36:35.057906+0000","last_became_peered":"2024-03-11T21:36:35.057906+0000","last_unstale":"2024-03-11T21:38:30.606855+0000","last_undegraded":"2024-03-11T21:38:30.606855+0000","last_fullsized":"2024-03-11T21:38:30.606855+0000","mapping_epoch":277,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":278,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:34:15.516278+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:27:21.207303+0000","last_clean_scrub_stamp":"2024-03-11T21:34:15.516278+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.15","version":"0'0","reported_seq":566,"reported_epoch":322,"state":"active+clean","last_fresh":"2024-03-11T21:38:30.602202+0000","last_change":"2024-03-11T21:36:06.579754+0000","last_active":"2024-03-11T21:38:30.602202+0000","last_peered":"2024-03-11T21:38:30.602202+0000","last_clean":"2024-03-11T21:38:30.602202+0000","last_became_active":"2024-03-11T21:25:38.541705+0000","last_became_peered":"2024-03-11T21:25:38.541705+0000","last_unstale":"2024-03-11T21:38:30.602202+0000","last_undegraded":"2024-03-11T21:38:30.602202+0000","last_fullsized":"2024-03-11T21:38:30.602202+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:36:06.579636+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:26:44.860499+0000","last_clean_scrub_stamp":"2024-03-11T21:36:06.579636+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.13","version":"260'24","reported_seq":616,"reported_epoch":322,"state":"active+clean","last_fresh":"2024-03-11T21:38:30.601568+0000","last_change":"2024-03-11T21:35:14.453053+0000","last_active":"2024-03-11T21:38:30.601568+0000","last_peered":"2024-03-11T21:38:30.601568+0000","last_clean":"2024-03-11T21:38:30.601568+0000","last_became_active":"2024-03-11T21:26:11.214879+0000","last_became_peered":"2024-03-11T21:26:11.214879+0000","last_unstale":"2024-03-11T21:38:30.601568+0000","last_undegraded":"2024-03-11T21:38:30.601568+0000","last_fullsized":"2024-03-11T21:38:30.601568+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"109'16","last_scrub_stamp":"2024-03-11T21:35:14.452891+0000","last_deep_scrub":"109'16","last_deep_scrub_stamp":"2024-03-11T21:32:24.657490+0000","last_clean_scrub_stamp":"2024-03-11T21:35:14.452891+0000","objects_scrubbed":5,"log_size":24,"log_dups_size":0,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":32,"num_read_kb":25,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,2],"acting":[4,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.10","version":"0'0","reported_seq":550,"reported_epoch":320,"state":"active+clean","last_fresh":"2024-03-11T21:38:28.353402+0000","last_change":"2024-03-11T21:35:29.712658+0000","last_active":"2024-03-11T21:38:28.353402+0000","last_peered":"2024-03-11T21:38:28.353402+0000","last_clean":"2024-03-11T21:38:28.353402+0000","last_became_active":"2024-03-11T21:25:44.591189+0000","last_became_peered":"2024-03-11T21:25:44.591189+0000","last_unstale":"2024-03-11T21:38:28.353402+0000","last_undegraded":"2024-03-11T21:38:28.353402+0000","last_fullsized":"2024-03-11T21:38:28.353402+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:35:29.712557+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:27:25.417450+0000","last_clean_scrub_stamp":"2024-03-11T21:35:29.712557+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,1],"acting":[0,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":484,"reported_epoch":320,"state":"active+clean","last_fresh":"2024-03-11T21:38:28.364325+0000","last_change":"2024-03-11T21:36:35.062276+0000","last_active":"2024-03-11T21:38:28.364325+0000","last_peered":"2024-03-11T21:38:28.364325+0000","last_clean":"2024-03-11T21:38:28.364325+0000","last_became_active":"2024-03-11T21:36:35.056619+0000","last_became_peered":"2024-03-11T21:36:35.056619+0000","last_unstale":"2024-03-11T21:38:28.364325+0000","last_undegraded":"2024-03-11T21:38:28.364325+0000","last_fullsized":"2024-03-11T21:38:28.364325+0000","mapping_epoch":277,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":278,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:35:13.060487+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:31:15.367898+0000","last_clean_scrub_stamp":"2024-03-11T21:35:13.060487+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.14","version":"0'0","reported_seq":583,"reported_epoch":322,"state":"active+clean","last_fresh":"2024-03-11T21:38:30.669657+0000","last_change":"2024-03-11T21:36:35.068386+0000","last_active":"2024-03-11T21:38:30.669657+0000","last_peered":"2024-03-11T21:38:30.669657+0000","last_clean":"2024-03-11T21:38:30.669657+0000","last_became_active":"2024-03-11T21:36:35.068038+0000","last_became_peered":"2024-03-11T21:36:35.068038+0000","last_unstale":"2024-03-11T21:38:30.669657+0000","last_undegraded":"2024-03-11T21:38:30.669657+0000","last_fullsized":"2024-03-11T21:38:30.669657+0000","mapping_epoch":277,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":278,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:35:09.122761+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:37.512018+0000","last_clean_scrub_stamp":"2024-03-11T21:35:09.122761+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:38:32.120223+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,5],"acting":[6,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.12","version":"260'15","reported_seq":587,"reported_epoch":322,"state":"active+clean","last_fresh":"2024-03-11T21:38:30.606662+0000","last_change":"2024-03-11T21:35:32.590439+0000","last_active":"2024-03-11T21:38:30.606662+0000","last_peered":"2024-03-11T21:38:30.606662+0000","last_clean":"2024-03-11T21:38:30.606662+0000","last_became_active":"2024-03-11T21:25:40.547082+0000","last_became_peered":"2024-03-11T21:25:40.547082+0000","last_unstale":"2024-03-11T21:38:30.606662+0000","last_undegraded":"2024-03-11T21:38:30.606662+0000","last_fullsized":"2024-03-11T21:38:30.606662+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'9","last_scrub_stamp":"2024-03-11T21:35:32.590318+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:39.525554+0000","last_clean_scrub_stamp":"2024-03-11T21:35:32.590318+0000","objects_scrubbed":4,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":554,"reported_epoch":322,"state":"active+clean","last_fresh":"2024-03-11T21:38:30.601995+0000","last_change":"2024-03-11T21:36:04.754168+0000","last_active":"2024-03-11T21:38:30.601995+0000","last_peered":"2024-03-11T21:38:30.601995+0000","last_clean":"2024-03-11T21:38:30.601995+0000","last_became_active":"2024-03-11T21:25:44.589586+0000","last_became_peered":"2024-03-11T21:25:44.589586+0000","last_unstale":"2024-03-11T21:38:30.601995+0000","last_undegraded":"2024-03-11T21:38:30.601995+0000","last_fullsized":"2024-03-11T21:38:30.601995+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:36:04.753979+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:27:09.801138+0000","last_clean_scrub_stamp":"2024-03-11T21:36:04.753979+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":113,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,5],"acting":[4,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.14","version":"280'12","reported_seq":668,"reported_epoch":320,"state":"active+clean","last_fresh":"2024-03-11T21:38:28.377645+0000","last_change":"2024-03-11T21:36:35.074550+0000","last_active":"2024-03-11T21:38:28.377645+0000","last_peered":"2024-03-11T21:38:28.377645+0000","last_clean":"2024-03-11T21:38:28.377645+0000","last_became_active":"2024-03-11T21:36:35.073912+0000","last_became_peered":"2024-03-11T21:36:35.073912+0000","last_unstale":"2024-03-11T21:38:28.377645+0000","last_undegraded":"2024-03-11T21:38:28.377645+0000","last_fullsized":"2024-03-11T21:38:28.377645+0000","mapping_epoch":277,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":278,"parent":"0.0","parent_split_bits":0,"last_scrub":"229'10","last_scrub_stamp":"2024-03-11T21:35:26.182107+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:41.544550+0000","last_clean_scrub_stamp":"2024-03-11T21:35:26.182107+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,2],"acting":[3,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.13","version":"0'0","reported_seq":563,"reported_epoch":320,"state":"active+clean","last_fresh":"2024-03-11T21:38:28.356453+0000","last_change":"2024-03-11T21:36:35.057688+0000","last_active":"2024-03-11T21:38:28.356453+0000","last_peered":"2024-03-11T21:38:28.356453+0000","last_clean":"2024-03-11T21:38:28.356453+0000","last_became_active":"2024-03-11T21:36:35.057132+0000","last_became_peered":"2024-03-11T21:36:35.057132+0000","last_unstale":"2024-03-11T21:38:28.356453+0000","last_undegraded":"2024-03-11T21:38:28.356453+0000","last_fullsized":"2024-03-11T21:38:28.356453+0000","mapping_epoch":277,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":278,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:34:08.650371+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:37.512018+0000","last_clean_scrub_stamp":"2024-03-11T21:34:08.650371+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.15","version":"260'19","reported_seq":640,"reported_epoch":322,"state":"active+clean","last_fresh":"2024-03-11T21:38:30.615765+0000","last_change":"2024-03-11T21:36:35.088986+0000","last_active":"2024-03-11T21:38:30.615765+0000","last_peered":"2024-03-11T21:38:30.615765+0000","last_clean":"2024-03-11T21:38:30.615765+0000","last_became_active":"2024-03-11T21:36:35.057285+0000","last_became_peered":"2024-03-11T21:36:35.057285+0000","last_unstale":"2024-03-11T21:38:30.615765+0000","last_undegraded":"2024-03-11T21:38:30.615765+0000","last_fullsized":"2024-03-11T21:38:30.615765+0000","mapping_epoch":277,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":278,"parent":"0.0","parent_split_bits":0,"last_scrub":"94'13","last_scrub_stamp":"2024-03-11T21:35:29.178543+0000","last_deep_scrub":"94'13","last_deep_scrub_stamp":"2024-03-11T21:29:41.853158+0000","last_clean_scrub_stamp":"2024-03-11T21:35:29.178543+0000","objects_scrubbed":4,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":11,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.16","version":"0'0","reported_seq":571,"reported_epoch":320,"state":"active+clean","last_fresh":"2024-03-11T21:38:28.353637+0000","last_change":"2024-03-11T21:36:35.058372+0000","last_active":"2024-03-11T21:38:28.353637+0000","last_peered":"2024-03-11T21:38:28.353637+0000","last_clean":"2024-03-11T21:38:28.353637+0000","last_became_active":"2024-03-11T21:36:35.057355+0000","last_became_peered":"2024-03-11T21:36:35.057355+0000","last_unstale":"2024-03-11T21:38:28.353637+0000","last_undegraded":"2024-03-11T21:38:28.353637+0000","last_fullsized":"2024-03-11T21:38:28.353637+0000","mapping_epoch":277,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":278,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:35:07.665075+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:43.559452+0000","last_clean_scrub_stamp":"2024-03-11T21:35:07.665075+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":35,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.15","version":"98'9","reported_seq":721,"reported_epoch":322,"state":"active+clean","last_fresh":"2024-03-11T21:38:30.615874+0000","last_change":"2024-03-11T21:36:03.366581+0000","last_active":"2024-03-11T21:38:30.615874+0000","last_peered":"2024-03-11T21:38:30.615874+0000","last_clean":"2024-03-11T21:38:30.615874+0000","last_became_active":"2024-03-11T21:25:42.572555+0000","last_became_peered":"2024-03-11T21:25:42.572555+0000","last_unstale":"2024-03-11T21:38:30.615874+0000","last_undegraded":"2024-03-11T21:38:30.615874+0000","last_fullsized":"2024-03-11T21:38:30.615874+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"98'9","last_scrub_stamp":"2024-03-11T21:36:03.366450+0000","last_deep_scrub":"98'9","last_deep_scrub_stamp":"2024-03-11T21:31:20.624923+0000","last_clean_scrub_stamp":"2024-03-11T21:36:03.366450+0000","objects_scrubbed":1,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.12","version":"0'0","reported_seq":586,"reported_epoch":322,"state":"active+clean","last_fresh":"2024-03-11T21:38:30.615451+0000","last_change":"2024-03-11T21:36:35.058369+0000","last_active":"2024-03-11T21:38:30.615451+0000","last_peered":"2024-03-11T21:38:30.615451+0000","last_clean":"2024-03-11T21:38:30.615451+0000","last_became_active":"2024-03-11T21:36:35.058033+0000","last_became_peered":"2024-03-11T21:36:35.058033+0000","last_unstale":"2024-03-11T21:38:30.615451+0000","last_undegraded":"2024-03-11T21:38:30.615451+0000","last_fullsized":"2024-03-11T21:38:30.615451+0000","mapping_epoch":277,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":278,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:35:27.158468+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:37.512018+0000","last_clean_scrub_stamp":"2024-03-11T21:35:27.158468+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.14","version":"260'20","reported_seq":567,"reported_epoch":320,"state":"active+clean","last_fresh":"2024-03-11T21:38:28.377332+0000","last_change":"2024-03-11T21:36:35.095382+0000","last_active":"2024-03-11T21:38:28.377332+0000","last_peered":"2024-03-11T21:38:28.377332+0000","last_clean":"2024-03-11T21:38:28.377332+0000","last_became_active":"2024-03-11T21:36:35.069638+0000","last_became_peered":"2024-03-11T21:36:35.069638+0000","last_unstale":"2024-03-11T21:38:28.377332+0000","last_undegraded":"2024-03-11T21:38:28.377332+0000","last_fullsized":"2024-03-11T21:38:28.377332+0000","mapping_epoch":277,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":278,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'10","last_scrub_stamp":"2024-03-11T21:35:28.189296+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:39.525554+0000","last_clean_scrub_stamp":"2024-03-11T21:35:28.189296+0000","objects_scrubbed":0,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":29,"num_read_kb":24,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.15","version":"0'0","reported_seq":210,"reported_epoch":322,"state":"active+clean","last_fresh":"2024-03-11T21:38:30.607826+0000","last_change":"2024-03-11T21:36:14.736240+0000","last_active":"2024-03-11T21:38:30.607826+0000","last_peered":"2024-03-11T21:38:30.607826+0000","last_clean":"2024-03-11T21:38:30.607826+0000","last_became_active":"2024-03-11T21:34:41.142760+0000","last_became_peered":"2024-03-11T21:34:41.142760+0000","last_unstale":"2024-03-11T21:38:30.607826+0000","last_undegraded":"2024-03-11T21:38:30.607826+0000","last_fullsized":"2024-03-11T21:38:30.607826+0000","mapping_epoch":226,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":227,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:36:14.735985+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:43.559452+0000","last_clean_scrub_stamp":"2024-03-11T21:36:14.735985+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":36,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":558,"reported_epoch":322,"state":"active+clean","last_fresh":"2024-03-11T21:38:30.615812+0000","last_change":"2024-03-11T21:36:35.060666+0000","last_active":"2024-03-11T21:38:30.615812+0000","last_peered":"2024-03-11T21:38:30.615812+0000","last_clean":"2024-03-11T21:38:30.615812+0000","last_became_active":"2024-03-11T21:36:35.060342+0000","last_became_peered":"2024-03-11T21:36:35.060342+0000","last_unstale":"2024-03-11T21:38:30.615812+0000","last_undegraded":"2024-03-11T21:38:30.615812+0000","last_fullsized":"2024-03-11T21:38:30.615812+0000","mapping_epoch":277,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":278,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:34:09.244852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:41.544550+0000","last_clean_scrub_stamp":"2024-03-11T21:34:09.244852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,1],"acting":[5,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.11","version":"0'0","reported_seq":566,"reported_epoch":322,"state":"active+clean","last_fresh":"2024-03-11T21:38:30.669515+0000","last_change":"2024-03-11T21:35:08.152853+0000","last_active":"2024-03-11T21:38:30.669515+0000","last_peered":"2024-03-11T21:38:30.669515+0000","last_clean":"2024-03-11T21:38:30.669515+0000","last_became_active":"2024-03-11T21:25:38.537415+0000","last_became_peered":"2024-03-11T21:25:38.537415+0000","last_unstale":"2024-03-11T21:38:30.669515+0000","last_undegraded":"2024-03-11T21:38:30.669515+0000","last_fullsized":"2024-03-11T21:38:30.669515+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:35:08.152740+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:37.512018+0000","last_clean_scrub_stamp":"2024-03-11T21:35:08.152740+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:38:32.120284+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.17","version":"260'15","reported_seq":512,"reported_epoch":320,"state":"active+clean","last_fresh":"2024-03-11T21:38:28.363502+0000","last_change":"2024-03-11T21:36:35.099451+0000","last_active":"2024-03-11T21:38:28.363502+0000","last_peered":"2024-03-11T21:38:28.363502+0000","last_clean":"2024-03-11T21:38:28.363502+0000","last_became_active":"2024-03-11T21:36:35.071346+0000","last_became_peered":"2024-03-11T21:36:35.071346+0000","last_unstale":"2024-03-11T21:38:28.363502+0000","last_undegraded":"2024-03-11T21:38:28.363502+0000","last_fullsized":"2024-03-11T21:38:28.363502+0000","mapping_epoch":277,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":278,"parent":"0.0","parent_split_bits":0,"last_scrub":"97'9","last_scrub_stamp":"2024-03-11T21:32:08.495352+0000","last_deep_scrub":"97'9","last_deep_scrub_stamp":"2024-03-11T21:27:25.803132+0000","last_clean_scrub_stamp":"2024-03-11T21:32:08.495352+0000","objects_scrubbed":3,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":17,"num_read_kb":14,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,5],"acting":[3,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.14","version":"0'0","reported_seq":565,"reported_epoch":322,"state":"active+clean","last_fresh":"2024-03-11T21:38:30.658705+0000","last_change":"2024-03-11T21:35:27.606664+0000","last_active":"2024-03-11T21:38:30.658705+0000","last_peered":"2024-03-11T21:38:30.658705+0000","last_clean":"2024-03-11T21:38:30.658705+0000","last_became_active":"2024-03-11T21:34:41.136946+0000","last_became_peered":"2024-03-11T21:34:41.136946+0000","last_unstale":"2024-03-11T21:38:30.658705+0000","last_undegraded":"2024-03-11T21:38:30.658705+0000","last_fullsized":"2024-03-11T21:38:30.658705+0000","mapping_epoch":226,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":227,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:35:27.606551+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:43.559452+0000","last_clean_scrub_stamp":"2024-03-11T21:35:27.606551+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,7],"acting":[2,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":466,"reported_epoch":320,"state":"active+clean","last_fresh":"2024-03-11T21:38:28.363598+0000","last_change":"2024-03-11T21:36:35.076782+0000","last_active":"2024-03-11T21:38:28.363598+0000","last_peered":"2024-03-11T21:38:28.363598+0000","last_clean":"2024-03-11T21:38:28.363598+0000","last_became_active":"2024-03-11T21:36:35.076335+0000","last_became_peered":"2024-03-11T21:36:35.076335+0000","last_unstale":"2024-03-11T21:38:28.363598+0000","last_undegraded":"2024-03-11T21:38:28.363598+0000","last_fullsized":"2024-03-11T21:38:28.363598+0000","mapping_epoch":277,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":278,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:34:33.082577+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:41.544550+0000","last_clean_scrub_stamp":"2024-03-11T21:34:33.082577+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":35,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.10","version":"0'0","reported_seq":565,"reported_epoch":322,"state":"active+clean","last_fresh":"2024-03-11T21:38:30.658275+0000","last_change":"2024-03-11T21:35:37.694623+0000","last_active":"2024-03-11T21:38:30.658275+0000","last_peered":"2024-03-11T21:38:30.658275+0000","last_clean":"2024-03-11T21:38:30.658275+0000","last_became_active":"2024-03-11T21:25:38.533138+0000","last_became_peered":"2024-03-11T21:25:38.533138+0000","last_unstale":"2024-03-11T21:38:30.658275+0000","last_undegraded":"2024-03-11T21:38:30.658275+0000","last_fullsized":"2024-03-11T21:38:30.658275+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:35:37.694533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:37.512018+0000","last_clean_scrub_stamp":"2024-03-11T21:35:37.694533+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,0],"acting":[2,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.16","version":"260'15","reported_seq":633,"reported_epoch":320,"state":"active+clean","last_fresh":"2024-03-11T21:38:28.353820+0000","last_change":"2024-03-11T21:36:35.080158+0000","last_active":"2024-03-11T21:38:28.353820+0000","last_peered":"2024-03-11T21:38:28.353820+0000","last_clean":"2024-03-11T21:38:28.353820+0000","last_became_active":"2024-03-11T21:36:35.056862+0000","last_became_peered":"2024-03-11T21:36:35.056862+0000","last_unstale":"2024-03-11T21:38:28.353820+0000","last_undegraded":"2024-03-11T21:38:28.353820+0000","last_fullsized":"2024-03-11T21:38:28.353820+0000","mapping_epoch":277,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":278,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'9","last_scrub_stamp":"2024-03-11T21:35:12.741225+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:39.525554+0000","last_clean_scrub_stamp":"2024-03-11T21:35:12.741225+0000","objects_scrubbed":4,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":11,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.4","version":"322'10582","reported_seq":11399,"reported_epoch":322,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-03-11T21:38:32.811465+0000","last_change":"2024-03-11T21:38:21.285644+0000","last_active":"2024-03-11T21:38:32.811465+0000","last_peered":"2024-03-11T21:38:32.811465+0000","last_clean":"2024-03-11T21:35:44.669493+0000","last_became_active":"2024-03-11T21:36:36.506468+0000","last_became_peered":"2024-03-11T21:36:36.506468+0000","last_unstale":"2024-03-11T21:38:32.811465+0000","last_undegraded":"2024-03-11T21:36:36.069183+0000","last_fullsized":"2024-03-11T21:36:36.038484+0000","mapping_epoch":278,"log_start":"285'5500","ondisk_log_start":"285'5500","created":259,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:35:44.669493+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:35:44.669493+0000","last_clean_scrub_stamp":"2024-03-11T21:35:44.669493+0000","objects_scrubbed":0,"log_size":5082,"log_dups_size":0,"ondisk_log_size":5082,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":273547264,"num_objects":4174,"num_object_clones":0,"num_object_copies":12522,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4475,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4174,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10582,"num_write_kb":472192,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":341,"num_bytes_recovered":22347776,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,3],"acting":[1,4],"avail_no_missing":["1","4"],"object_location_counts":[{"shards":"1,4","objects":-301}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.b","version":"0'0","reported_seq":454,"reported_epoch":320,"state":"active+clean","last_fresh":"2024-03-11T21:38:28.364109+0000","last_change":"2024-03-11T21:36:35.074815+0000","last_active":"2024-03-11T21:38:28.364109+0000","last_peered":"2024-03-11T21:38:28.364109+0000","last_clean":"2024-03-11T21:38:28.364109+0000","last_became_active":"2024-03-11T21:36:35.072147+0000","last_became_peered":"2024-03-11T21:36:35.072147+0000","last_unstale":"2024-03-11T21:38:28.364109+0000","last_undegraded":"2024-03-11T21:38:28.364109+0000","last_fullsized":"2024-03-11T21:38:28.364109+0000","mapping_epoch":277,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":278,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:32:22.598150+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:43.559452+0000","last_clean_scrub_stamp":"2024-03-11T21:32:22.598150+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":57,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.f","version":"67'2","reported_seq":578,"reported_epoch":322,"state":"active+clean","last_fresh":"2024-03-11T21:38:30.601817+0000","last_change":"2024-03-11T21:35:36.613667+0000","last_active":"2024-03-11T21:38:30.601817+0000","last_peered":"2024-03-11T21:38:30.601817+0000","last_clean":"2024-03-11T21:38:30.601817+0000","last_became_active":"2024-03-11T21:25:38.537711+0000","last_became_peered":"2024-03-11T21:25:38.537711+0000","last_unstale":"2024-03-11T21:38:30.601817+0000","last_undegraded":"2024-03-11T21:38:30.601817+0000","last_fullsized":"2024-03-11T21:38:30.601817+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'2","last_scrub_stamp":"2024-03-11T21:35:36.613516+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:37.512018+0000","last_clean_scrub_stamp":"2024-03-11T21:35:36.613516+0000","objects_scrubbed":2,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":92,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":10,"num_read_kb":10,"num_write":4,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,7],"acting":[4,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.9","version":"260'23","reported_seq":609,"reported_epoch":322,"state":"active+clean","last_fresh":"2024-03-11T21:38:30.601363+0000","last_change":"2024-03-11T21:36:35.089034+0000","last_active":"2024-03-11T21:38:30.601363+0000","last_peered":"2024-03-11T21:38:30.601363+0000","last_clean":"2024-03-11T21:38:30.601363+0000","last_became_active":"2024-03-11T21:36:35.059930+0000","last_became_peered":"2024-03-11T21:36:35.059930+0000","last_unstale":"2024-03-11T21:38:30.601363+0000","last_undegraded":"2024-03-11T21:38:30.601363+0000","last_fullsized":"2024-03-11T21:38:30.601363+0000","mapping_epoch":277,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":278,"parent":"0.0","parent_split_bits":0,"last_scrub":"146'17","last_scrub_stamp":"2024-03-11T21:34:26.765075+0000","last_deep_scrub":"146'17","last_deep_scrub_stamp":"2024-03-11T21:34:26.765075+0000","last_clean_scrub_stamp":"2024-03-11T21:34:26.765075+0000","objects_scrubbed":5,"log_size":23,"log_dups_size":0,"ondisk_log_size":23,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":34,"num_read_kb":25,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,3],"acting":[4,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":557,"reported_epoch":322,"state":"active+clean","last_fresh":"2024-03-11T21:38:30.658371+0000","last_change":"2024-03-11T21:35:16.643421+0000","last_active":"2024-03-11T21:38:30.658371+0000","last_peered":"2024-03-11T21:38:30.658371+0000","last_clean":"2024-03-11T21:38:30.658371+0000","last_became_active":"2024-03-11T21:25:42.573987+0000","last_became_peered":"2024-03-11T21:25:42.573987+0000","last_unstale":"2024-03-11T21:38:30.658371+0000","last_undegraded":"2024-03-11T21:38:30.658371+0000","last_fullsized":"2024-03-11T21:38:30.658371+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:35:16.643311+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:41.544550+0000","last_clean_scrub_stamp":"2024-03-11T21:35:16.643311+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"9.f","version":"322'10317","reported_seq":10456,"reported_epoch":322,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-03-11T21:38:32.455547+0000","last_change":"2024-03-11T21:36:36.519822+0000","last_active":"2024-03-11T21:38:32.455547+0000","last_peered":"2024-03-11T21:38:32.455547+0000","last_clean":"2024-03-11T21:35:44.669493+0000","last_became_active":"2024-03-11T21:36:36.508836+0000","last_became_peered":"2024-03-11T21:36:36.508836+0000","last_unstale":"2024-03-11T21:38:32.455547+0000","last_undegraded":"2024-03-11T21:36:36.176770+0000","last_fullsized":"2024-03-11T21:36:36.038651+0000","mapping_epoch":278,"log_start":"282'5100","ondisk_log_start":"282'5100","created":259,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:35:44.669493+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:35:44.669493+0000","last_clean_scrub_stamp":"2024-03-11T21:35:44.669493+0000","objects_scrubbed":0,"log_size":5217,"log_dups_size":0,"ondisk_log_size":5217,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":265748480,"num_objects":4055,"num_object_clones":0,"num_object_copies":12165,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4738,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4055,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10317,"num_write_kb":459904,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":65536,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3],"acting":[5,1],"avail_no_missing":["5","1"],"object_location_counts":[{"shards":"1,5","objects":-683}],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.0","version":"0'0","reported_seq":552,"reported_epoch":320,"state":"active+clean","last_fresh":"2024-03-11T21:38:28.353502+0000","last_change":"2024-03-11T21:36:35.058390+0000","last_active":"2024-03-11T21:38:28.353502+0000","last_peered":"2024-03-11T21:38:28.353502+0000","last_clean":"2024-03-11T21:38:28.353502+0000","last_became_active":"2024-03-11T21:36:35.058054+0000","last_became_peered":"2024-03-11T21:36:35.058054+0000","last_unstale":"2024-03-11T21:38:28.353502+0000","last_undegraded":"2024-03-11T21:38:28.353502+0000","last_fullsized":"2024-03-11T21:38:28.353502+0000","mapping_epoch":277,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":278,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:35:04.626668+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:29:39.498784+0000","last_clean_scrub_stamp":"2024-03-11T21:35:04.626668+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":39,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":579,"reported_epoch":322,"state":"active+clean","last_fresh":"2024-03-11T21:38:30.605218+0000","last_change":"2024-03-11T21:35:33.608063+0000","last_active":"2024-03-11T21:38:30.605218+0000","last_peered":"2024-03-11T21:38:30.605218+0000","last_clean":"2024-03-11T21:38:30.605218+0000","last_became_active":"2024-03-11T21:34:41.150913+0000","last_became_peered":"2024-03-11T21:34:41.150913+0000","last_unstale":"2024-03-11T21:38:30.605218+0000","last_undegraded":"2024-03-11T21:38:30.605218+0000","last_fullsized":"2024-03-11T21:38:30.605218+0000","mapping_epoch":226,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":227,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:35:33.607969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:37.512018+0000","last_clean_scrub_stamp":"2024-03-11T21:35:33.607969+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,7],"acting":[1,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.2","version":"260'25","reported_seq":598,"reported_epoch":322,"state":"active+clean","last_fresh":"2024-03-11T21:38:30.606759+0000","last_change":"2024-03-11T21:35:17.666749+0000","last_active":"2024-03-11T21:38:30.606759+0000","last_peered":"2024-03-11T21:38:30.606759+0000","last_clean":"2024-03-11T21:38:30.606759+0000","last_became_active":"2024-03-11T21:25:40.548833+0000","last_became_peered":"2024-03-11T21:25:40.548833+0000","last_unstale":"2024-03-11T21:38:30.606759+0000","last_undegraded":"2024-03-11T21:38:30.606759+0000","last_fullsized":"2024-03-11T21:38:30.606759+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"94'15","last_scrub_stamp":"2024-03-11T21:35:17.666498+0000","last_deep_scrub":"94'15","last_deep_scrub_stamp":"2024-03-11T21:26:49.993060+0000","last_clean_scrub_stamp":"2024-03-11T21:35:17.666498+0000","objects_scrubbed":5,"log_size":25,"log_dups_size":0,"ondisk_log_size":25,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":20,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":25,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.3","version":"74'8","reported_seq":717,"reported_epoch":320,"state":"active+clean","last_fresh":"2024-03-11T21:38:28.356826+0000","last_change":"2024-03-11T21:35:31.695369+0000","last_active":"2024-03-11T21:38:28.356826+0000","last_peered":"2024-03-11T21:38:28.356826+0000","last_clean":"2024-03-11T21:38:28.356826+0000","last_became_active":"2024-03-11T21:25:42.578161+0000","last_became_peered":"2024-03-11T21:25:42.578161+0000","last_unstale":"2024-03-11T21:38:28.356826+0000","last_undegraded":"2024-03-11T21:38:28.356826+0000","last_fullsized":"2024-03-11T21:38:28.356826+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'8","last_scrub_stamp":"2024-03-11T21:35:31.695263+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:41.544550+0000","last_clean_scrub_stamp":"2024-03-11T21:35:31.695263+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,5],"acting":[0,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.9","version":"320'10400","reported_seq":10533,"reported_epoch":320,"state":"active+clean","last_fresh":"2024-03-11T21:38:28.516413+0000","last_change":"2024-03-11T21:35:45.696767+0000","last_active":"2024-03-11T21:38:28.516413+0000","last_peered":"2024-03-11T21:38:28.516413+0000","last_clean":"2024-03-11T21:38:28.516413+0000","last_became_active":"2024-03-11T21:35:45.696379+0000","last_became_peered":"2024-03-11T21:35:45.696379+0000","last_unstale":"2024-03-11T21:38:28.516413+0000","last_undegraded":"2024-03-11T21:38:28.516413+0000","last_fullsized":"2024-03-11T21:38:28.516413+0000","mapping_epoch":259,"log_start":"276'4800","ondisk_log_start":"276'4800","created":259,"last_epoch_clean":260,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:35:44.669493+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:35:44.669493+0000","last_clean_scrub_stamp":"2024-03-11T21:35:44.669493+0000","objects_scrubbed":0,"log_size":5600,"log_dups_size":0,"ondisk_log_size":5600,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":287703040,"num_objects":4390,"num_object_clones":0,"num_object_copies":13170,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4390,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10400,"num_write_kb":473280,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,5],"acting":[0,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":492,"reported_epoch":320,"state":"active+clean","last_fresh":"2024-03-11T21:38:28.363343+0000","last_change":"2024-03-11T21:36:35.068977+0000","last_active":"2024-03-11T21:38:28.363343+0000","last_peered":"2024-03-11T21:38:28.363343+0000","last_clean":"2024-03-11T21:38:28.363343+0000","last_became_active":"2024-03-11T21:36:35.067288+0000","last_became_peered":"2024-03-11T21:36:35.067288+0000","last_unstale":"2024-03-11T21:38:28.363343+0000","last_undegraded":"2024-03-11T21:38:28.363343+0000","last_fullsized":"2024-03-11T21:38:28.363343+0000","mapping_epoch":277,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":278,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:35:09.985377+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:28:30.563712+0000","last_clean_scrub_stamp":"2024-03-11T21:35:09.985377+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,7],"acting":[3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":574,"reported_epoch":322,"state":"active+clean","last_fresh":"2024-03-11T21:38:30.615485+0000","last_change":"2024-03-11T21:35:40.116979+0000","last_active":"2024-03-11T21:38:30.615485+0000","last_peered":"2024-03-11T21:38:30.615485+0000","last_clean":"2024-03-11T21:38:30.615485+0000","last_became_active":"2024-03-11T21:25:38.538540+0000","last_became_peered":"2024-03-11T21:25:38.538540+0000","last_unstale":"2024-03-11T21:38:30.615485+0000","last_undegraded":"2024-03-11T21:38:30.615485+0000","last_fullsized":"2024-03-11T21:38:30.615485+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:35:40.116877+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:37.512018+0000","last_clean_scrub_stamp":"2024-03-11T21:35:40.116877+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.4","version":"280'42","reported_seq":801,"reported_epoch":322,"state":"active+clean","last_fresh":"2024-03-11T21:38:30.605079+0000","last_change":"2024-03-11T21:36:35.095935+0000","last_active":"2024-03-11T21:38:30.605079+0000","last_peered":"2024-03-11T21:38:30.605079+0000","last_clean":"2024-03-11T21:38:30.605079+0000","last_became_active":"2024-03-11T21:36:35.073430+0000","last_became_peered":"2024-03-11T21:36:35.073430+0000","last_unstale":"2024-03-11T21:38:30.605079+0000","last_undegraded":"2024-03-11T21:38:30.605079+0000","last_fullsized":"2024-03-11T21:38:30.605079+0000","mapping_epoch":277,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":278,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'28","last_scrub_stamp":"2024-03-11T21:34:19.513252+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:39.525554+0000","last_clean_scrub_stamp":"2024-03-11T21:34:19.513252+0000","objects_scrubbed":10,"log_size":42,"log_dups_size":0,"ondisk_log_size":42,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":358,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":66,"num_read_kb":51,"num_write":38,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3],"acting":[1,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":555,"reported_epoch":320,"state":"active+clean","last_fresh":"2024-03-11T21:38:28.357344+0000","last_change":"2024-03-11T21:35:32.651731+0000","last_active":"2024-03-11T21:38:28.357344+0000","last_peered":"2024-03-11T21:38:28.357344+0000","last_clean":"2024-03-11T21:38:28.357344+0000","last_became_active":"2024-03-11T21:25:42.572673+0000","last_became_peered":"2024-03-11T21:25:42.572673+0000","last_unstale":"2024-03-11T21:38:28.357344+0000","last_undegraded":"2024-03-11T21:38:28.357344+0000","last_fullsized":"2024-03-11T21:38:28.357344+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:35:32.651652+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:41.544550+0000","last_clean_scrub_stamp":"2024-03-11T21:35:32.651652+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.a","version":"322'10554","reported_seq":10691,"reported_epoch":322,"state":"active+clean","last_fresh":"2024-03-11T21:38:32.436475+0000","last_change":"2024-03-11T21:35:45.698558+0000","last_active":"2024-03-11T21:38:32.436475+0000","last_peered":"2024-03-11T21:38:32.436475+0000","last_clean":"2024-03-11T21:38:32.436475+0000","last_became_active":"2024-03-11T21:35:45.698238+0000","last_became_peered":"2024-03-11T21:35:45.698238+0000","last_unstale":"2024-03-11T21:38:32.436475+0000","last_undegraded":"2024-03-11T21:38:32.436475+0000","last_fullsized":"2024-03-11T21:38:32.436475+0000","mapping_epoch":259,"log_start":"282'5300","ondisk_log_start":"282'5300","created":259,"last_epoch_clean":260,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:35:44.669493+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:35:44.669493+0000","last_clean_scrub_stamp":"2024-03-11T21:35:44.669493+0000","objects_scrubbed":0,"log_size":5254,"log_dups_size":0,"ondisk_log_size":5254,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":277479424,"num_objects":4234,"num_object_clones":0,"num_object_copies":12702,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4234,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10554,"num_write_kb":473216,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,7],"acting":[5,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.5","version":"0'0","reported_seq":219,"reported_epoch":322,"state":"active+clean","last_fresh":"2024-03-11T21:38:30.607716+0000","last_change":"2024-03-11T21:36:35.069989+0000","last_active":"2024-03-11T21:38:30.607716+0000","last_peered":"2024-03-11T21:38:30.607716+0000","last_clean":"2024-03-11T21:38:30.607716+0000","last_became_active":"2024-03-11T21:36:35.069358+0000","last_became_peered":"2024-03-11T21:36:35.069358+0000","last_unstale":"2024-03-11T21:38:30.607716+0000","last_undegraded":"2024-03-11T21:38:30.607716+0000","last_fullsized":"2024-03-11T21:38:30.607716+0000","mapping_epoch":277,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":278,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:35:19.841632+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:43.559452+0000","last_clean_scrub_stamp":"2024-03-11T21:35:19.841632+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,3],"acting":[7,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.0","version":"164'6","reported_seq":1345,"reported_epoch":322,"state":"active+clean","last_fresh":"2024-03-11T21:38:31.584530+0000","last_change":"2024-03-11T21:35:38.518243+0000","last_active":"2024-03-11T21:38:31.584530+0000","last_peered":"2024-03-11T21:38:31.584530+0000","last_clean":"2024-03-11T21:38:31.584530+0000","last_became_active":"2024-03-11T21:25:38.536890+0000","last_became_peered":"2024-03-11T21:25:38.536890+0000","last_unstale":"2024-03-11T21:38:31.584530+0000","last_undegraded":"2024-03-11T21:38:31.584530+0000","last_fullsized":"2024-03-11T21:38:31.584530+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"164'6","last_scrub_stamp":"2024-03-11T21:35:38.518097+0000","last_deep_scrub":"164'6","last_deep_scrub_stamp":"2024-03-11T21:32:17.597666+0000","last_clean_scrub_stamp":"2024-03-11T21:35:38.518097+0000","objects_scrubbed":1,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0.000753559,"stat_sum":{"num_bytes":389,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":767,"num_read_kb":762,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,6],"acting":[1,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":575,"reported_epoch":322,"state":"active+clean","last_fresh":"2024-03-11T21:38:30.658479+0000","last_change":"2024-03-11T21:36:35.057287+0000","last_active":"2024-03-11T21:38:30.658479+0000","last_peered":"2024-03-11T21:38:30.658479+0000","last_clean":"2024-03-11T21:38:30.658479+0000","last_became_active":"2024-03-11T21:36:35.056966+0000","last_became_peered":"2024-03-11T21:36:35.056966+0000","last_unstale":"2024-03-11T21:38:30.658479+0000","last_undegraded":"2024-03-11T21:38:30.658479+0000","last_fullsized":"2024-03-11T21:38:30.658479+0000","mapping_epoch":277,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":278,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:34:18.424010+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:26:48.997454+0000","last_clean_scrub_stamp":"2024-03-11T21:34:18.424010+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.7","version":"260'22","reported_seq":603,"reported_epoch":322,"state":"active+clean","last_fresh":"2024-03-11T21:38:30.603255+0000","last_change":"2024-03-11T21:35:37.508679+0000","last_active":"2024-03-11T21:38:30.603255+0000","last_peered":"2024-03-11T21:38:30.603255+0000","last_clean":"2024-03-11T21:38:30.603255+0000","last_became_active":"2024-03-11T21:25:40.541769+0000","last_became_peered":"2024-03-11T21:25:40.541769+0000","last_unstale":"2024-03-11T21:38:30.603255+0000","last_undegraded":"2024-03-11T21:38:30.603255+0000","last_fullsized":"2024-03-11T21:38:30.603255+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"97'18","last_scrub_stamp":"2024-03-11T21:35:37.508567+0000","last_deep_scrub":"97'18","last_deep_scrub_stamp":"2024-03-11T21:27:19.239404+0000","last_clean_scrub_stamp":"2024-03-11T21:35:37.508567+0000","objects_scrubbed":5,"log_size":22,"log_dups_size":0,"ondisk_log_size":22,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":36,"num_read_kb":25,"num_write":20,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":569,"reported_epoch":322,"state":"active+clean","last_fresh":"2024-03-11T21:38:30.660013+0000","last_change":"2024-03-11T21:35:22.632822+0000","last_active":"2024-03-11T21:38:30.660013+0000","last_peered":"2024-03-11T21:38:30.660013+0000","last_clean":"2024-03-11T21:38:30.660013+0000","last_became_active":"2024-03-11T21:34:41.143660+0000","last_became_peered":"2024-03-11T21:34:41.143660+0000","last_unstale":"2024-03-11T21:38:30.660013+0000","last_undegraded":"2024-03-11T21:38:30.660013+0000","last_fullsized":"2024-03-11T21:38:30.660013+0000","mapping_epoch":226,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":227,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:35:22.632707+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:41.544550+0000","last_clean_scrub_stamp":"2024-03-11T21:35:22.632707+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":18,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7],"acting":[2,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"9.8","version":"320'10403","reported_seq":10535,"reported_epoch":320,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-03-11T21:38:28.509629+0000","last_change":"2024-03-11T21:36:36.520071+0000","last_active":"2024-03-11T21:38:28.509629+0000","last_peered":"2024-03-11T21:38:28.509629+0000","last_clean":"2024-03-11T21:35:44.669493+0000","last_became_active":"2024-03-11T21:36:36.507652+0000","last_became_peered":"2024-03-11T21:36:36.507652+0000","last_unstale":"2024-03-11T21:38:28.509629+0000","last_undegraded":"2024-03-11T21:36:36.064628+0000","last_fullsized":"2024-03-11T21:36:36.036628+0000","mapping_epoch":278,"log_start":"276'4800","ondisk_log_start":"276'4800","created":259,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:35:44.669493+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:35:44.669493+0000","last_clean_scrub_stamp":"2024-03-11T21:35:44.669493+0000","objects_scrubbed":0,"log_size":5603,"log_dups_size":0,"ondisk_log_size":5603,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":284884992,"num_objects":4347,"num_object_clones":0,"num_object_copies":13041,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4846,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4347,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10403,"num_write_kb":472000,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[0,5],"avail_no_missing":["0","5"],"object_location_counts":[{"shards":"0,5","objects":-499}],"blocked_by":[],"up_primary":3,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.7","version":"124'2","reported_seq":556,"reported_epoch":322,"state":"active+clean","last_fresh":"2024-03-11T21:38:30.618724+0000","last_change":"2024-03-11T21:36:35.059190+0000","last_active":"2024-03-11T21:38:30.618724+0000","last_peered":"2024-03-11T21:38:30.618724+0000","last_clean":"2024-03-11T21:38:30.618724+0000","last_became_active":"2024-03-11T21:36:35.058786+0000","last_became_peered":"2024-03-11T21:36:35.058786+0000","last_unstale":"2024-03-11T21:38:30.618724+0000","last_undegraded":"2024-03-11T21:38:30.618724+0000","last_fullsized":"2024-03-11T21:38:30.618724+0000","mapping_epoch":277,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":278,"parent":"0.0","parent_split_bits":0,"last_scrub":"124'2","last_scrub_stamp":"2024-03-11T21:35:16.414595+0000","last_deep_scrub":"124'2","last_deep_scrub_stamp":"2024-03-11T21:29:34.956908+0000","last_clean_scrub_stamp":"2024-03-11T21:35:16.414595+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":42,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,4],"acting":[5,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2","version":"71'2","reported_seq":513,"reported_epoch":320,"state":"active+clean","last_fresh":"2024-03-11T21:38:28.376755+0000","last_change":"2024-03-11T21:36:35.071218+0000","last_active":"2024-03-11T21:38:28.376755+0000","last_peered":"2024-03-11T21:38:28.376755+0000","last_clean":"2024-03-11T21:38:28.376755+0000","last_became_active":"2024-03-11T21:36:35.070783+0000","last_became_peered":"2024-03-11T21:36:35.070783+0000","last_unstale":"2024-03-11T21:38:28.376755+0000","last_undegraded":"2024-03-11T21:38:28.376755+0000","last_fullsized":"2024-03-11T21:38:28.376755+0000","mapping_epoch":277,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":278,"parent":"0.0","parent_split_bits":0,"last_scrub":"71'2","last_scrub_stamp":"2024-03-11T21:32:21.543053+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:37.512018+0000","last_clean_scrub_stamp":"2024-03-11T21:32:21.543053+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014206449999999999,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":584,"reported_epoch":322,"state":"active+clean","last_fresh":"2024-03-11T21:38:30.615504+0000","last_change":"2024-03-11T21:35:08.391825+0000","last_active":"2024-03-11T21:38:30.615504+0000","last_peered":"2024-03-11T21:38:30.615504+0000","last_clean":"2024-03-11T21:38:30.615504+0000","last_became_active":"2024-03-11T21:34:41.146672+0000","last_became_peered":"2024-03-11T21:34:41.146672+0000","last_unstale":"2024-03-11T21:38:30.615504+0000","last_undegraded":"2024-03-11T21:38:30.615504+0000","last_fullsized":"2024-03-11T21:38:30.615504+0000","mapping_epoch":226,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":227,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:35:08.391713+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:37.512018+0000","last_clean_scrub_stamp":"2024-03-11T21:35:08.391713+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"87'128","reported_seq":681,"reported_epoch":322,"state":"active+clean","last_fresh":"2024-03-11T21:38:30.607017+0000","last_change":"2024-03-11T21:35:21.798272+0000","last_active":"2024-03-11T21:38:30.607017+0000","last_peered":"2024-03-11T21:38:30.607017+0000","last_clean":"2024-03-11T21:38:30.607017+0000","last_became_active":"2024-03-11T21:34:44.164569+0000","last_became_peered":"2024-03-11T21:34:44.164569+0000","last_unstale":"2024-03-11T21:38:30.607017+0000","last_undegraded":"2024-03-11T21:38:30.607017+0000","last_fullsized":"2024-03-11T21:38:30.607017+0000","mapping_epoch":229,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":230,"parent":"0.0","parent_split_bits":0,"last_scrub":"87'128","last_scrub_stamp":"2024-03-11T21:35:21.798144+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:24.261974+0000","last_clean_scrub_stamp":"2024-03-11T21:35:21.798144+0000","objects_scrubbed":2,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":198,"num_read_kb":294,"num_write":179,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":2951840,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.5","version":"260'28","reported_seq":766,"reported_epoch":322,"state":"active+clean","last_fresh":"2024-03-11T21:38:30.669533+0000","last_change":"2024-03-11T21:36:14.026933+0000","last_active":"2024-03-11T21:38:30.669533+0000","last_peered":"2024-03-11T21:38:30.669533+0000","last_clean":"2024-03-11T21:38:30.669533+0000","last_became_active":"2024-03-11T21:25:40.761457+0000","last_became_peered":"2024-03-11T21:25:40.761457+0000","last_unstale":"2024-03-11T21:38:30.669533+0000","last_undegraded":"2024-03-11T21:38:30.669533+0000","last_fullsized":"2024-03-11T21:38:30.669533+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"260'28","last_scrub_stamp":"2024-03-11T21:36:14.026775+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:39.525554+0000","last_clean_scrub_stamp":"2024-03-11T21:36:14.026775+0000","objects_scrubbed":8,"log_size":28,"log_dups_size":0,"ondisk_log_size":28,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:38:32.120515+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":154,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":43,"num_read_kb":33,"num_write":25,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":210,"reported_epoch":322,"state":"active+clean","last_fresh":"2024-03-11T21:38:30.607851+0000","last_change":"2024-03-11T21:35:24.841170+0000","last_active":"2024-03-11T21:38:30.607851+0000","last_peered":"2024-03-11T21:38:30.607851+0000","last_clean":"2024-03-11T21:38:30.607851+0000","last_became_active":"2024-03-11T21:34:41.139782+0000","last_became_peered":"2024-03-11T21:34:41.139782+0000","last_unstale":"2024-03-11T21:38:30.607851+0000","last_undegraded":"2024-03-11T21:38:30.607851+0000","last_fullsized":"2024-03-11T21:38:30.607851+0000","mapping_epoch":226,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":227,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:35:24.841041+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:32:21.266672+0000","last_clean_scrub_stamp":"2024-03-11T21:35:24.841041+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,5],"acting":[7,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.b","version":"320'10199","reported_seq":10332,"reported_epoch":320,"state":"active+clean","last_fresh":"2024-03-11T21:38:28.496955+0000","last_change":"2024-03-11T21:35:45.697213+0000","last_active":"2024-03-11T21:38:28.496955+0000","last_peered":"2024-03-11T21:38:28.496955+0000","last_clean":"2024-03-11T21:38:28.496955+0000","last_became_active":"2024-03-11T21:35:45.696869+0000","last_became_peered":"2024-03-11T21:35:45.696869+0000","last_unstale":"2024-03-11T21:38:28.496955+0000","last_undegraded":"2024-03-11T21:38:28.496955+0000","last_fullsized":"2024-03-11T21:38:28.496955+0000","mapping_epoch":259,"log_start":"274'4600","ondisk_log_start":"274'4600","created":259,"last_epoch_clean":260,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:35:44.669493+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:35:44.669493+0000","last_clean_scrub_stamp":"2024-03-11T21:35:44.669493+0000","objects_scrubbed":0,"log_size":5599,"log_dups_size":0,"ondisk_log_size":5599,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":278331392,"num_objects":4247,"num_object_clones":0,"num_object_copies":12741,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4247,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10199,"num_write_kb":462272,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,5],"acting":[0,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.4","version":"0'0","reported_seq":556,"reported_epoch":322,"state":"active+clean","last_fresh":"2024-03-11T21:38:30.605459+0000","last_change":"2024-03-11T21:36:35.073305+0000","last_active":"2024-03-11T21:38:30.605459+0000","last_peered":"2024-03-11T21:38:30.605459+0000","last_clean":"2024-03-11T21:38:30.605459+0000","last_became_active":"2024-03-11T21:36:35.072824+0000","last_became_peered":"2024-03-11T21:36:35.072824+0000","last_unstale":"2024-03-11T21:38:30.605459+0000","last_undegraded":"2024-03-11T21:38:30.605459+0000","last_fullsized":"2024-03-11T21:38:30.605459+0000","mapping_epoch":277,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":278,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:34:23.531064+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:43.559452+0000","last_clean_scrub_stamp":"2024-03-11T21:34:23.531064+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":579,"reported_epoch":320,"state":"active+clean","last_fresh":"2024-03-11T21:38:28.355689+0000","last_change":"2024-03-11T21:36:35.070626+0000","last_active":"2024-03-11T21:38:28.355689+0000","last_peered":"2024-03-11T21:38:28.355689+0000","last_clean":"2024-03-11T21:38:28.355689+0000","last_became_active":"2024-03-11T21:36:35.069452+0000","last_became_peered":"2024-03-11T21:36:35.069452+0000","last_unstale":"2024-03-11T21:38:28.355689+0000","last_undegraded":"2024-03-11T21:38:28.355689+0000","last_fullsized":"2024-03-11T21:38:28.355689+0000","mapping_epoch":277,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":278,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:35:13.741947+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:37.512018+0000","last_clean_scrub_stamp":"2024-03-11T21:35:13.741947+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0.00089342499999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":202,"reported_epoch":322,"state":"active+clean","last_fresh":"2024-03-11T21:38:30.606992+0000","last_change":"2024-03-11T21:34:41.150606+0000","last_active":"2024-03-11T21:38:30.606992+0000","last_peered":"2024-03-11T21:38:30.606992+0000","last_clean":"2024-03-11T21:38:30.606992+0000","last_became_active":"2024-03-11T21:34:41.149219+0000","last_became_peered":"2024-03-11T21:34:41.149219+0000","last_unstale":"2024-03-11T21:38:30.606992+0000","last_undegraded":"2024-03-11T21:38:30.606992+0000","last_fullsized":"2024-03-11T21:38:30.606992+0000","mapping_epoch":226,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":227,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:34:31.628014+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:37.512018+0000","last_clean_scrub_stamp":"2024-03-11T21:34:31.628014+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,0],"acting":[7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.6","version":"260'24","reported_seq":614,"reported_epoch":320,"state":"active+clean","last_fresh":"2024-03-11T21:38:28.353727+0000","last_change":"2024-03-11T21:35:19.753433+0000","last_active":"2024-03-11T21:38:28.353727+0000","last_peered":"2024-03-11T21:38:28.353727+0000","last_clean":"2024-03-11T21:38:28.353727+0000","last_became_active":"2024-03-11T21:32:22.796486+0000","last_became_peered":"2024-03-11T21:32:22.796486+0000","last_unstale":"2024-03-11T21:38:28.353727+0000","last_undegraded":"2024-03-11T21:38:28.353727+0000","last_fullsized":"2024-03-11T21:38:28.353727+0000","mapping_epoch":166,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":167,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'12","last_scrub_stamp":"2024-03-11T21:35:19.753224+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:39.525554+0000","last_clean_scrub_stamp":"2024-03-11T21:35:19.753224+0000","objects_scrubbed":6,"log_size":24,"log_dups_size":0,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":36,"num_read_kb":30,"num_write":24,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":557,"reported_epoch":322,"state":"active+clean","last_fresh":"2024-03-11T21:38:30.618632+0000","last_change":"2024-03-11T21:36:07.451482+0000","last_active":"2024-03-11T21:38:30.618632+0000","last_peered":"2024-03-11T21:38:30.618632+0000","last_clean":"2024-03-11T21:38:30.618632+0000","last_became_active":"2024-03-11T21:25:42.572324+0000","last_became_peered":"2024-03-11T21:25:42.572324+0000","last_unstale":"2024-03-11T21:38:30.618632+0000","last_undegraded":"2024-03-11T21:38:30.618632+0000","last_fullsized":"2024-03-11T21:38:30.618632+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:36:07.451102+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:41.544550+0000","last_clean_scrub_stamp":"2024-03-11T21:36:07.451102+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":54,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.e","version":"322'10167","reported_seq":10304,"reported_epoch":322,"state":"active+clean","last_fresh":"2024-03-11T21:38:32.289009+0000","last_change":"2024-03-11T21:35:45.697331+0000","last_active":"2024-03-11T21:38:32.289009+0000","last_peered":"2024-03-11T21:38:32.289009+0000","last_clean":"2024-03-11T21:38:32.289009+0000","last_became_active":"2024-03-11T21:35:45.696948+0000","last_became_peered":"2024-03-11T21:35:45.696948+0000","last_unstale":"2024-03-11T21:38:32.289009+0000","last_undegraded":"2024-03-11T21:38:32.289009+0000","last_fullsized":"2024-03-11T21:38:32.289009+0000","mapping_epoch":259,"log_start":"286'5500","ondisk_log_start":"286'5500","created":259,"last_epoch_clean":260,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:35:44.669493+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:35:44.669493+0000","last_clean_scrub_stamp":"2024-03-11T21:35:44.669493+0000","objects_scrubbed":0,"log_size":4667,"log_dups_size":0,"ondisk_log_size":4667,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":269549568,"num_objects":4113,"num_object_clones":0,"num_object_copies":12339,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4113,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10167,"num_write_kb":456960,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":563,"reported_epoch":322,"state":"active+clean","last_fresh":"2024-03-11T21:38:30.602148+0000","last_change":"2024-03-11T21:36:18.529506+0000","last_active":"2024-03-11T21:38:30.602148+0000","last_peered":"2024-03-11T21:38:30.602148+0000","last_clean":"2024-03-11T21:38:30.602148+0000","last_became_active":"2024-03-11T21:25:45.301416+0000","last_became_peered":"2024-03-11T21:25:45.301416+0000","last_unstale":"2024-03-11T21:38:30.602148+0000","last_undegraded":"2024-03-11T21:38:30.602148+0000","last_fullsized":"2024-03-11T21:38:30.602148+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:36:18.529370+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:43.559452+0000","last_clean_scrub_stamp":"2024-03-11T21:36:18.529370+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":54,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":210,"reported_epoch":322,"state":"active+clean","last_fresh":"2024-03-11T21:38:30.607822+0000","last_change":"2024-03-11T21:35:33.740615+0000","last_active":"2024-03-11T21:38:30.607822+0000","last_peered":"2024-03-11T21:38:30.607822+0000","last_clean":"2024-03-11T21:38:30.607822+0000","last_became_active":"2024-03-11T21:34:41.152721+0000","last_became_peered":"2024-03-11T21:34:41.152721+0000","last_unstale":"2024-03-11T21:38:30.607822+0000","last_undegraded":"2024-03-11T21:38:30.607822+0000","last_fullsized":"2024-03-11T21:38:30.607822+0000","mapping_epoch":226,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":227,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:35:33.740538+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:26:57.333669+0000","last_clean_scrub_stamp":"2024-03-11T21:35:33.740538+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.3","version":"260'37","reported_seq":666,"reported_epoch":320,"state":"active+clean","last_fresh":"2024-03-11T21:38:28.356221+0000","last_change":"2024-03-11T21:35:11.697314+0000","last_active":"2024-03-11T21:38:28.356221+0000","last_peered":"2024-03-11T21:38:28.356221+0000","last_clean":"2024-03-11T21:38:28.356221+0000","last_became_active":"2024-03-11T21:34:41.145607+0000","last_became_peered":"2024-03-11T21:34:41.145607+0000","last_unstale":"2024-03-11T21:38:28.356221+0000","last_undegraded":"2024-03-11T21:38:28.356221+0000","last_fullsized":"2024-03-11T21:38:28.356221+0000","mapping_epoch":226,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":227,"parent":"0.0","parent_split_bits":0,"last_scrub":"165'27","last_scrub_stamp":"2024-03-11T21:35:11.697140+0000","last_deep_scrub":"165'27","last_deep_scrub_stamp":"2024-03-11T21:32:21.576732+0000","last_clean_scrub_stamp":"2024-03-11T21:35:11.697140+0000","objects_scrubbed":8,"log_size":37,"log_dups_size":0,"ondisk_log_size":37,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":54,"num_read_kb":40,"num_write":32,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":660,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":567,"reported_epoch":322,"state":"active+clean","last_fresh":"2024-03-11T21:38:30.669074+0000","last_change":"2024-03-11T21:35:12.091317+0000","last_active":"2024-03-11T21:38:30.669074+0000","last_peered":"2024-03-11T21:38:30.669074+0000","last_clean":"2024-03-11T21:38:30.669074+0000","last_became_active":"2024-03-11T21:25:42.579045+0000","last_became_peered":"2024-03-11T21:25:42.579045+0000","last_unstale":"2024-03-11T21:38:30.669074+0000","last_undegraded":"2024-03-11T21:38:30.669074+0000","last_fullsized":"2024-03-11T21:38:30.669074+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:35:12.091185+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:29:36.918936+0000","last_clean_scrub_stamp":"2024-03-11T21:35:12.091185+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:38:32.120346+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.d","version":"322'10364","reported_seq":10502,"reported_epoch":322,"state":"active+clean","last_fresh":"2024-03-11T21:38:32.297214+0000","last_change":"2024-03-11T21:35:45.700060+0000","last_active":"2024-03-11T21:38:32.297214+0000","last_peered":"2024-03-11T21:38:32.297214+0000","last_clean":"2024-03-11T21:38:32.297214+0000","last_became_active":"2024-03-11T21:35:45.699674+0000","last_became_peered":"2024-03-11T21:35:45.699674+0000","last_unstale":"2024-03-11T21:38:32.297214+0000","last_undegraded":"2024-03-11T21:38:32.297214+0000","last_fullsized":"2024-03-11T21:38:32.297214+0000","mapping_epoch":259,"log_start":"287'5700","ondisk_log_start":"287'5700","created":259,"last_epoch_clean":260,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:35:44.669493+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:35:44.669493+0000","last_clean_scrub_stamp":"2024-03-11T21:35:44.669493+0000","objects_scrubbed":0,"log_size":4664,"log_dups_size":0,"ondisk_log_size":4664,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":268763160,"num_objects":4102,"num_object_clones":0,"num_object_copies":12306,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4102,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":10364,"num_write_kb":462849,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,5],"acting":[7,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":561,"reported_epoch":322,"state":"active+clean","last_fresh":"2024-03-11T21:38:30.601703+0000","last_change":"2024-03-11T21:35:08.437783+0000","last_active":"2024-03-11T21:38:30.601703+0000","last_peered":"2024-03-11T21:38:30.601703+0000","last_clean":"2024-03-11T21:38:30.601703+0000","last_became_active":"2024-03-11T21:32:22.791439+0000","last_became_peered":"2024-03-11T21:32:22.791439+0000","last_unstale":"2024-03-11T21:38:30.601703+0000","last_undegraded":"2024-03-11T21:38:30.601703+0000","last_fullsized":"2024-03-11T21:38:30.601703+0000","mapping_epoch":166,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":167,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:35:08.437697+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:43.559452+0000","last_clean_scrub_stamp":"2024-03-11T21:35:08.437697+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":33,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,2],"acting":[4,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.6","version":"67'1","reported_seq":567,"reported_epoch":322,"state":"active+clean","last_fresh":"2024-03-11T21:38:30.602964+0000","last_change":"2024-03-11T21:35:30.581783+0000","last_active":"2024-03-11T21:38:30.602964+0000","last_peered":"2024-03-11T21:38:30.602964+0000","last_clean":"2024-03-11T21:38:30.602964+0000","last_became_active":"2024-03-11T21:25:38.536142+0000","last_became_peered":"2024-03-11T21:25:38.536142+0000","last_unstale":"2024-03-11T21:38:30.602964+0000","last_undegraded":"2024-03-11T21:38:30.602964+0000","last_fullsized":"2024-03-11T21:38:30.602964+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'1","last_scrub_stamp":"2024-03-11T21:35:30.581701+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:37.512018+0000","last_clean_scrub_stamp":"2024-03-11T21:35:30.581701+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,4],"acting":[1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.0","version":"260'38","reported_seq":561,"reported_epoch":320,"state":"active+clean","last_fresh":"2024-03-11T21:38:28.385574+0000","last_change":"2024-03-11T21:36:35.087771+0000","last_active":"2024-03-11T21:38:28.385574+0000","last_peered":"2024-03-11T21:38:28.385574+0000","last_clean":"2024-03-11T21:38:28.385574+0000","last_became_active":"2024-03-11T21:36:35.068071+0000","last_became_peered":"2024-03-11T21:36:35.068071+0000","last_unstale":"2024-03-11T21:38:28.385574+0000","last_undegraded":"2024-03-11T21:38:28.385574+0000","last_fullsized":"2024-03-11T21:38:28.385574+0000","mapping_epoch":277,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":278,"parent":"0.0","parent_split_bits":0,"last_scrub":"93'26","last_scrub_stamp":"2024-03-11T21:34:16.863984+0000","last_deep_scrub":"93'26","last_deep_scrub_stamp":"2024-03-11T21:26:40.680324+0000","last_clean_scrub_stamp":"2024-03-11T21:34:16.863984+0000","objects_scrubbed":8,"log_size":38,"log_dups_size":0,"ondisk_log_size":38,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":22,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":51,"num_read_kb":39,"num_write":32,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":580,"reported_epoch":322,"state":"active+clean","last_fresh":"2024-03-11T21:38:30.601382+0000","last_change":"2024-03-11T21:36:35.057830+0000","last_active":"2024-03-11T21:38:30.601382+0000","last_peered":"2024-03-11T21:38:30.601382+0000","last_clean":"2024-03-11T21:38:30.601382+0000","last_became_active":"2024-03-11T21:36:35.057517+0000","last_became_peered":"2024-03-11T21:36:35.057517+0000","last_unstale":"2024-03-11T21:38:30.601382+0000","last_undegraded":"2024-03-11T21:38:30.601382+0000","last_fullsized":"2024-03-11T21:38:30.601382+0000","mapping_epoch":277,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":278,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:35:12.411136+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:26:51.863621+0000","last_clean_scrub_stamp":"2024-03-11T21:35:12.411136+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,7],"acting":[4,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.c","version":"319'9968","reported_seq":19135,"reported_epoch":319,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-03-11T21:38:27.193188+0000","last_change":"2024-03-11T21:36:36.491614+0000","last_active":"2024-03-11T21:38:27.193188+0000","last_peered":"2024-03-11T21:38:27.193188+0000","last_clean":"2024-03-11T21:35:44.669493+0000","last_became_active":"2024-03-11T21:36:36.482095+0000","last_became_peered":"2024-03-11T21:36:36.482095+0000","last_unstale":"2024-03-11T21:38:27.193188+0000","last_undegraded":"2024-03-11T21:36:36.076624+0000","last_fullsized":"2024-03-11T21:36:36.041449+0000","mapping_epoch":278,"log_start":"276'4600","ondisk_log_start":"276'4600","created":259,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:35:44.669493+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:35:44.669493+0000","last_clean_scrub_stamp":"2024-03-11T21:35:44.669493+0000","objects_scrubbed":0,"log_size":5368,"log_dups_size":0,"ondisk_log_size":5368,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":278790144,"num_objects":4254,"num_object_clones":0,"num_object_copies":12762,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":110,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4254,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9968,"num_write_kb":455104,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4518,"num_bytes_recovered":296091648,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,7],"acting":[4,7],"avail_no_missing":["4","7"],"object_location_counts":[{"shards":"4,7","objects":4144}],"blocked_by":[],"up_primary":3,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.3","version":"0'0","reported_seq":202,"reported_epoch":322,"state":"active+clean","last_fresh":"2024-03-11T21:38:30.608463+0000","last_change":"2024-03-11T21:34:41.143973+0000","last_active":"2024-03-11T21:38:30.608463+0000","last_peered":"2024-03-11T21:38:30.608463+0000","last_clean":"2024-03-11T21:38:30.608463+0000","last_became_active":"2024-03-11T21:34:41.142683+0000","last_became_peered":"2024-03-11T21:34:41.142683+0000","last_unstale":"2024-03-11T21:38:30.608463+0000","last_undegraded":"2024-03-11T21:38:30.608463+0000","last_fullsized":"2024-03-11T21:38:30.608463+0000","mapping_epoch":226,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":227,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:34:27.353700+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:29:42.485740+0000","last_clean_scrub_stamp":"2024-03-11T21:34:27.353700+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,2],"acting":[7,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":586,"reported_epoch":322,"state":"active+clean","last_fresh":"2024-03-11T21:38:30.669917+0000","last_change":"2024-03-11T21:35:32.155983+0000","last_active":"2024-03-11T21:38:30.669917+0000","last_peered":"2024-03-11T21:38:30.669917+0000","last_clean":"2024-03-11T21:38:30.669917+0000","last_became_active":"2024-03-11T21:34:41.145628+0000","last_became_peered":"2024-03-11T21:34:41.145628+0000","last_unstale":"2024-03-11T21:38:30.669917+0000","last_undegraded":"2024-03-11T21:38:30.669917+0000","last_fullsized":"2024-03-11T21:38:30.669917+0000","mapping_epoch":226,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":227,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:35:32.155853+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:37.512018+0000","last_clean_scrub_stamp":"2024-03-11T21:35:32.155853+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:38:32.120408+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,2],"acting":[6,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.1","version":"260'28","reported_seq":605,"reported_epoch":322,"state":"active+clean","last_fresh":"2024-03-11T21:38:30.601258+0000","last_change":"2024-03-11T21:35:33.644654+0000","last_active":"2024-03-11T21:38:30.601258+0000","last_peered":"2024-03-11T21:38:30.601258+0000","last_clean":"2024-03-11T21:38:30.601258+0000","last_became_active":"2024-03-11T21:25:40.546219+0000","last_became_peered":"2024-03-11T21:25:40.546219+0000","last_unstale":"2024-03-11T21:38:30.601258+0000","last_undegraded":"2024-03-11T21:38:30.601258+0000","last_fullsized":"2024-03-11T21:38:30.601258+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'14","last_scrub_stamp":"2024-03-11T21:35:33.644503+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:39.525554+0000","last_clean_scrub_stamp":"2024-03-11T21:35:33.644503+0000","objects_scrubbed":7,"log_size":28,"log_dups_size":0,"ondisk_log_size":28,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":42,"num_read_kb":35,"num_write":28,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6],"acting":[4,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.0","version":"280'11","reported_seq":643,"reported_epoch":320,"state":"active+clean","last_fresh":"2024-03-11T21:38:28.385390+0000","last_change":"2024-03-11T21:36:35.064085+0000","last_active":"2024-03-11T21:38:28.385390+0000","last_peered":"2024-03-11T21:38:28.385390+0000","last_clean":"2024-03-11T21:38:28.385390+0000","last_became_active":"2024-03-11T21:36:35.055250+0000","last_became_peered":"2024-03-11T21:36:35.055250+0000","last_unstale":"2024-03-11T21:38:28.385390+0000","last_undegraded":"2024-03-11T21:38:28.385390+0000","last_fullsized":"2024-03-11T21:38:28.385390+0000","mapping_epoch":277,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":278,"parent":"0.0","parent_split_bits":0,"last_scrub":"98'9","last_scrub_stamp":"2024-03-11T21:35:21.143578+0000","last_deep_scrub":"98'9","last_deep_scrub_stamp":"2024-03-11T21:27:29.745711+0000","last_clean_scrub_stamp":"2024-03-11T21:35:21.143578+0000","objects_scrubbed":1,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.3","version":"322'10231","reported_seq":10377,"reported_epoch":322,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-03-11T21:38:32.306231+0000","last_change":"2024-03-11T21:36:36.544317+0000","last_active":"2024-03-11T21:38:32.306231+0000","last_peered":"2024-03-11T21:38:32.306231+0000","last_clean":"2024-03-11T21:35:44.669493+0000","last_became_active":"2024-03-11T21:36:36.470537+0000","last_became_peered":"2024-03-11T21:36:36.470537+0000","last_unstale":"2024-03-11T21:38:32.306231+0000","last_undegraded":"2024-03-11T21:36:36.071501+0000","last_fullsized":"2024-03-11T21:36:36.040233+0000","mapping_epoch":278,"log_start":"287'5600","ondisk_log_start":"287'5600","created":259,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:35:44.669493+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:35:44.669493+0000","last_clean_scrub_stamp":"2024-03-11T21:35:44.669493+0000","objects_scrubbed":0,"log_size":4631,"log_dups_size":0,"ondisk_log_size":4631,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":267976704,"num_objects":4089,"num_object_clones":0,"num_object_copies":12267,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4641,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4089,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10231,"num_write_kb":458240,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":196608,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,1],"acting":[7,1],"avail_no_missing":["7","1"],"object_location_counts":[{"shards":"1,7","objects":-552}],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.c","version":"0'0","reported_seq":482,"reported_epoch":320,"state":"active+clean","last_fresh":"2024-03-11T21:38:28.377421+0000","last_change":"2024-03-11T21:36:35.074112+0000","last_active":"2024-03-11T21:38:28.377421+0000","last_peered":"2024-03-11T21:38:28.377421+0000","last_clean":"2024-03-11T21:38:28.377421+0000","last_became_active":"2024-03-11T21:36:35.073778+0000","last_became_peered":"2024-03-11T21:36:35.073778+0000","last_unstale":"2024-03-11T21:38:28.377421+0000","last_undegraded":"2024-03-11T21:38:28.377421+0000","last_fullsized":"2024-03-11T21:38:28.377421+0000","mapping_epoch":277,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":278,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:32:17.600213+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:43.559452+0000","last_clean_scrub_stamp":"2024-03-11T21:32:17.600213+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.f","version":"0'0","reported_seq":565,"reported_epoch":322,"state":"active+clean","last_fresh":"2024-03-11T21:38:30.618412+0000","last_change":"2024-03-11T21:36:08.439013+0000","last_active":"2024-03-11T21:38:30.618412+0000","last_peered":"2024-03-11T21:38:30.618412+0000","last_clean":"2024-03-11T21:38:30.618412+0000","last_became_active":"2024-03-11T21:25:42.575935+0000","last_became_peered":"2024-03-11T21:25:42.575935+0000","last_unstale":"2024-03-11T21:38:30.618412+0000","last_undegraded":"2024-03-11T21:38:30.618412+0000","last_fullsized":"2024-03-11T21:38:30.618412+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:36:08.438895+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:41.544550+0000","last_clean_scrub_stamp":"2024-03-11T21:36:08.438895+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.8","version":"0'0","reported_seq":210,"reported_epoch":322,"state":"active+clean","last_fresh":"2024-03-11T21:38:30.607300+0000","last_change":"2024-03-11T21:35:34.733595+0000","last_active":"2024-03-11T21:38:30.607300+0000","last_peered":"2024-03-11T21:38:30.607300+0000","last_clean":"2024-03-11T21:38:30.607300+0000","last_became_active":"2024-03-11T21:34:41.144681+0000","last_became_peered":"2024-03-11T21:34:41.144681+0000","last_unstale":"2024-03-11T21:38:30.607300+0000","last_undegraded":"2024-03-11T21:38:30.607300+0000","last_fullsized":"2024-03-11T21:38:30.607300+0000","mapping_epoch":226,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":227,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:35:34.733491+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:37.512018+0000","last_clean_scrub_stamp":"2024-03-11T21:35:34.733491+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,1],"acting":[7,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.e","version":"260'24","reported_seq":598,"reported_epoch":322,"state":"active+clean","last_fresh":"2024-03-11T21:38:30.602089+0000","last_change":"2024-03-11T21:35:34.679906+0000","last_active":"2024-03-11T21:38:30.602089+0000","last_peered":"2024-03-11T21:38:30.602089+0000","last_clean":"2024-03-11T21:38:30.602089+0000","last_became_active":"2024-03-11T21:25:40.552178+0000","last_became_peered":"2024-03-11T21:25:40.552178+0000","last_unstale":"2024-03-11T21:38:30.602089+0000","last_undegraded":"2024-03-11T21:38:30.602089+0000","last_fullsized":"2024-03-11T21:38:30.602089+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"109'16","last_scrub_stamp":"2024-03-11T21:35:34.679817+0000","last_deep_scrub":"109'16","last_deep_scrub_stamp":"2024-03-11T21:28:24.739395+0000","last_clean_scrub_stamp":"2024-03-11T21:35:34.679817+0000","objects_scrubbed":5,"log_size":24,"log_dups_size":0,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":32,"num_read_kb":25,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.2","version":"322'10425","reported_seq":10829,"reported_epoch":322,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-03-11T21:38:32.827975+0000","last_change":"2024-03-11T21:38:29.073541+0000","last_active":"2024-03-11T21:38:32.827975+0000","last_peered":"2024-03-11T21:38:32.827975+0000","last_clean":"2024-03-11T21:35:44.669493+0000","last_became_active":"2024-03-11T21:36:36.506474+0000","last_became_peered":"2024-03-11T21:36:36.506474+0000","last_unstale":"2024-03-11T21:38:32.827975+0000","last_undegraded":"2024-03-11T21:36:36.068182+0000","last_fullsized":"2024-03-11T21:36:36.037781+0000","mapping_epoch":278,"log_start":"285'5400","ondisk_log_start":"285'5400","created":259,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:35:44.669493+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:35:44.669493+0000","last_clean_scrub_stamp":"2024-03-11T21:35:44.669493+0000","objects_scrubbed":0,"log_size":5025,"log_dups_size":0,"ondisk_log_size":5025,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":269025280,"num_objects":4105,"num_object_clones":0,"num_object_copies":12315,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4639,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4105,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10425,"num_write_kb":464960,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":134,"num_bytes_recovered":8781824,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[1,2],"avail_no_missing":["1","2"],"object_location_counts":[{"shards":"1,2","objects":-534}],"blocked_by":[],"up_primary":3,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.d","version":"0'0","reported_seq":553,"reported_epoch":322,"state":"active+clean","last_fresh":"2024-03-11T21:38:30.615943+0000","last_change":"2024-03-11T21:35:37.185210+0000","last_active":"2024-03-11T21:38:30.615943+0000","last_peered":"2024-03-11T21:38:30.615943+0000","last_clean":"2024-03-11T21:38:30.615943+0000","last_became_active":"2024-03-11T21:25:44.590333+0000","last_became_peered":"2024-03-11T21:25:44.590333+0000","last_unstale":"2024-03-11T21:38:30.615943+0000","last_undegraded":"2024-03-11T21:38:30.615943+0000","last_fullsized":"2024-03-11T21:38:30.615943+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:35:37.182850+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:35:37.182850+0000","last_clean_scrub_stamp":"2024-03-11T21:35:37.182850+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":40,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.e","version":"74'8","reported_seq":721,"reported_epoch":322,"state":"active+clean","last_fresh":"2024-03-11T21:38:30.601947+0000","last_change":"2024-03-11T21:36:07.734369+0000","last_active":"2024-03-11T21:38:30.601947+0000","last_peered":"2024-03-11T21:38:30.601947+0000","last_clean":"2024-03-11T21:38:30.601947+0000","last_became_active":"2024-03-11T21:25:42.576742+0000","last_became_peered":"2024-03-11T21:25:42.576742+0000","last_unstale":"2024-03-11T21:38:30.601947+0000","last_undegraded":"2024-03-11T21:38:30.601947+0000","last_fullsized":"2024-03-11T21:38:30.601947+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'8","last_scrub_stamp":"2024-03-11T21:36:07.734115+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:41.544550+0000","last_clean_scrub_stamp":"2024-03-11T21:36:07.734115+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,0],"acting":[4,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.9","version":"0'0","reported_seq":579,"reported_epoch":322,"state":"active+clean","last_fresh":"2024-03-11T21:38:30.601968+0000","last_change":"2024-03-11T21:36:35.066093+0000","last_active":"2024-03-11T21:38:30.601968+0000","last_peered":"2024-03-11T21:38:30.601968+0000","last_clean":"2024-03-11T21:38:30.601968+0000","last_became_active":"2024-03-11T21:36:35.065733+0000","last_became_peered":"2024-03-11T21:36:35.065733+0000","last_unstale":"2024-03-11T21:38:30.601968+0000","last_undegraded":"2024-03-11T21:38:30.601968+0000","last_fullsized":"2024-03-11T21:38:30.601968+0000","mapping_epoch":277,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":278,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:32:02.506579+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:26:50.991712+0000","last_clean_scrub_stamp":"2024-03-11T21:32:02.506579+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,3],"acting":[1,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.f","version":"260'27","reported_seq":581,"reported_epoch":322,"state":"active+clean","last_fresh":"2024-03-11T21:38:30.659150+0000","last_change":"2024-03-11T21:36:35.084069+0000","last_active":"2024-03-11T21:38:30.659150+0000","last_peered":"2024-03-11T21:38:30.659150+0000","last_clean":"2024-03-11T21:38:30.659150+0000","last_became_active":"2024-03-11T21:36:35.058272+0000","last_became_peered":"2024-03-11T21:36:35.058272+0000","last_unstale":"2024-03-11T21:38:30.659150+0000","last_undegraded":"2024-03-11T21:38:30.659150+0000","last_fullsized":"2024-03-11T21:38:30.659150+0000","mapping_epoch":277,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":278,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'15","last_scrub_stamp":"2024-03-11T21:34:21.517894+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:39.525554+0000","last_clean_scrub_stamp":"2024-03-11T21:34:21.517894+0000","objects_scrubbed":7,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":44,"num_read_kb":35,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,4],"acting":[2,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"9.1","version":"320'10288","reported_seq":20078,"reported_epoch":320,"state":"active+clean","last_fresh":"2024-03-11T21:38:28.731276+0000","last_change":"2024-03-11T21:38:22.558103+0000","last_active":"2024-03-11T21:38:28.731276+0000","last_peered":"2024-03-11T21:38:28.731276+0000","last_clean":"2024-03-11T21:38:28.731276+0000","last_became_active":"2024-03-11T21:38:22.540816+0000","last_became_peered":"2024-03-11T21:38:22.540816+0000","last_unstale":"2024-03-11T21:38:28.731276+0000","last_undegraded":"2024-03-11T21:38:28.731276+0000","last_fullsized":"2024-03-11T21:38:28.731276+0000","mapping_epoch":316,"log_start":"284'5395","ondisk_log_start":"284'5395","created":259,"last_epoch_clean":317,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:35:44.669493+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:35:44.669493+0000","last_clean_scrub_stamp":"2024-03-11T21:35:44.669493+0000","objects_scrubbed":0,"log_size":4893,"log_dups_size":0,"ondisk_log_size":4893,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":281542656,"num_objects":4296,"num_object_clones":0,"num_object_copies":12888,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4296,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10288,"num_write_kb":466688,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4825,"num_bytes_recovered":316211200,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,7],"acting":[3,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.e","version":"0'0","reported_seq":554,"reported_epoch":322,"state":"active+clean","last_fresh":"2024-03-11T21:38:30.602000+0000","last_change":"2024-03-11T21:36:10.769557+0000","last_active":"2024-03-11T21:38:30.602000+0000","last_peered":"2024-03-11T21:38:30.602000+0000","last_clean":"2024-03-11T21:38:30.602000+0000","last_became_active":"2024-03-11T21:25:44.590198+0000","last_became_peered":"2024-03-11T21:25:44.590198+0000","last_unstale":"2024-03-11T21:38:30.602000+0000","last_undegraded":"2024-03-11T21:38:30.602000+0000","last_fullsized":"2024-03-11T21:38:30.602000+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:36:10.769460+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:43.559452+0000","last_clean_scrub_stamp":"2024-03-11T21:36:10.769460+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":75,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.d","version":"109'9","reported_seq":721,"reported_epoch":322,"state":"active+clean","last_fresh":"2024-03-11T21:38:30.658993+0000","last_change":"2024-03-11T21:35:07.562523+0000","last_active":"2024-03-11T21:38:30.658993+0000","last_peered":"2024-03-11T21:38:30.658993+0000","last_clean":"2024-03-11T21:38:30.658993+0000","last_became_active":"2024-03-11T21:25:42.575583+0000","last_became_peered":"2024-03-11T21:25:42.575583+0000","last_unstale":"2024-03-11T21:38:30.658993+0000","last_undegraded":"2024-03-11T21:38:30.658993+0000","last_fullsized":"2024-03-11T21:38:30.658993+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"109'9","last_scrub_stamp":"2024-03-11T21:35:07.562385+0000","last_deep_scrub":"109'9","last_deep_scrub_stamp":"2024-03-11T21:28:27.082975+0000","last_clean_scrub_stamp":"2024-03-11T21:35:07.562385+0000","objects_scrubbed":1,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,5],"acting":[2,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.a","version":"0'0","reported_seq":587,"reported_epoch":322,"state":"active+clean","last_fresh":"2024-03-11T21:38:30.606439+0000","last_change":"2024-03-11T21:36:35.067262+0000","last_active":"2024-03-11T21:38:30.606439+0000","last_peered":"2024-03-11T21:38:30.606439+0000","last_clean":"2024-03-11T21:38:30.606439+0000","last_became_active":"2024-03-11T21:36:35.066832+0000","last_became_peered":"2024-03-11T21:36:35.066832+0000","last_unstale":"2024-03-11T21:38:30.606439+0000","last_undegraded":"2024-03-11T21:38:30.606439+0000","last_fullsized":"2024-03-11T21:38:30.606439+0000","mapping_epoch":277,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":278,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:34:22.526916+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:37.512018+0000","last_clean_scrub_stamp":"2024-03-11T21:34:22.526916+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,7],"acting":[1,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.c","version":"260'20","reported_seq":606,"reported_epoch":322,"state":"active+clean","last_fresh":"2024-03-11T21:38:30.601664+0000","last_change":"2024-03-11T21:36:35.095574+0000","last_active":"2024-03-11T21:38:30.601664+0000","last_peered":"2024-03-11T21:38:30.601664+0000","last_clean":"2024-03-11T21:38:30.601664+0000","last_became_active":"2024-03-11T21:36:35.067269+0000","last_became_peered":"2024-03-11T21:36:35.067269+0000","last_unstale":"2024-03-11T21:38:30.601664+0000","last_undegraded":"2024-03-11T21:38:30.601664+0000","last_fullsized":"2024-03-11T21:38:30.601664+0000","mapping_epoch":277,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":278,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'10","last_scrub_stamp":"2024-03-11T21:32:54.846584+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:39.525554+0000","last_clean_scrub_stamp":"2024-03-11T21:32:54.846584+0000","objects_scrubbed":5,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":25,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.0","version":"322'10487","reported_seq":10624,"reported_epoch":322,"state":"active+clean","last_fresh":"2024-03-11T21:38:32.823176+0000","last_change":"2024-03-11T21:35:45.703065+0000","last_active":"2024-03-11T21:38:32.823176+0000","last_peered":"2024-03-11T21:38:32.823176+0000","last_clean":"2024-03-11T21:38:32.823176+0000","last_became_active":"2024-03-11T21:35:45.702739+0000","last_became_peered":"2024-03-11T21:35:45.702739+0000","last_unstale":"2024-03-11T21:38:32.823176+0000","last_undegraded":"2024-03-11T21:38:32.823176+0000","last_fullsized":"2024-03-11T21:38:32.823176+0000","mapping_epoch":259,"log_start":"285'5400","ondisk_log_start":"285'5400","created":259,"last_epoch_clean":260,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:35:44.669493+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:35:44.669493+0000","last_clean_scrub_stamp":"2024-03-11T21:35:44.669493+0000","objects_scrubbed":0,"log_size":5087,"log_dups_size":0,"ondisk_log_size":5087,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":268500992,"num_objects":4097,"num_object_clones":0,"num_object_copies":12291,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4097,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10487,"num_write_kb":466688,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.f","version":"0'0","reported_seq":562,"reported_epoch":322,"state":"active+clean","last_fresh":"2024-03-11T21:38:30.659078+0000","last_change":"2024-03-11T21:36:35.058630+0000","last_active":"2024-03-11T21:38:30.659078+0000","last_peered":"2024-03-11T21:38:30.659078+0000","last_clean":"2024-03-11T21:38:30.659078+0000","last_became_active":"2024-03-11T21:36:35.057800+0000","last_became_peered":"2024-03-11T21:36:35.057800+0000","last_unstale":"2024-03-11T21:38:30.659078+0000","last_undegraded":"2024-03-11T21:38:30.659078+0000","last_fullsized":"2024-03-11T21:38:30.659078+0000","mapping_epoch":277,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":278,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:35:10.608981+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:27:03.010969+0000","last_clean_scrub_stamp":"2024-03-11T21:35:10.608981+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":94,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,4],"acting":[2,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":558,"reported_epoch":322,"state":"active+clean","last_fresh":"2024-03-11T21:38:30.605371+0000","last_change":"2024-03-11T21:36:10.608789+0000","last_active":"2024-03-11T21:38:30.605371+0000","last_peered":"2024-03-11T21:38:30.605371+0000","last_clean":"2024-03-11T21:38:30.605371+0000","last_became_active":"2024-03-11T21:25:42.579409+0000","last_became_peered":"2024-03-11T21:25:42.579409+0000","last_unstale":"2024-03-11T21:38:30.605371+0000","last_undegraded":"2024-03-11T21:38:30.605371+0000","last_fullsized":"2024-03-11T21:38:30.605371+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:36:10.608552+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:30:22.750053+0000","last_clean_scrub_stamp":"2024-03-11T21:36:10.608552+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.b","version":"0'0","reported_seq":210,"reported_epoch":322,"state":"active+clean","last_fresh":"2024-03-11T21:38:30.608518+0000","last_change":"2024-03-11T21:36:16.719432+0000","last_active":"2024-03-11T21:38:30.608518+0000","last_peered":"2024-03-11T21:38:30.608518+0000","last_clean":"2024-03-11T21:38:30.608518+0000","last_became_active":"2024-03-11T21:34:41.134987+0000","last_became_peered":"2024-03-11T21:34:41.134987+0000","last_unstale":"2024-03-11T21:38:30.608518+0000","last_undegraded":"2024-03-11T21:38:30.608518+0000","last_fullsized":"2024-03-11T21:38:30.608518+0000","mapping_epoch":226,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":227,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:36:16.719309+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:36:16.719309+0000","last_clean_scrub_stamp":"2024-03-11T21:36:16.719309+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,5],"acting":[7,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.d","version":"260'39","reported_seq":619,"reported_epoch":322,"state":"active+clean","last_fresh":"2024-03-11T21:38:30.601814+0000","last_change":"2024-03-11T21:35:25.543769+0000","last_active":"2024-03-11T21:38:30.601814+0000","last_peered":"2024-03-11T21:38:30.601814+0000","last_clean":"2024-03-11T21:38:30.601814+0000","last_became_active":"2024-03-11T21:25:40.553454+0000","last_became_peered":"2024-03-11T21:25:40.553454+0000","last_unstale":"2024-03-11T21:38:30.601814+0000","last_undegraded":"2024-03-11T21:38:30.601814+0000","last_fullsized":"2024-03-11T21:38:30.601814+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"165'25","last_scrub_stamp":"2024-03-11T21:35:25.543628+0000","last_deep_scrub":"165'25","last_deep_scrub_stamp":"2024-03-11T21:32:21.678328+0000","last_clean_scrub_stamp":"2024-03-11T21:35:25.543628+0000","objects_scrubbed":8,"log_size":39,"log_dups_size":0,"ondisk_log_size":39,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":50,"num_read_kb":40,"num_write":32,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,1],"acting":[4,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.7","version":"322'10439","reported_seq":17486,"reported_epoch":322,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-03-11T21:38:32.477658+0000","last_change":"2024-03-11T21:36:36.539792+0000","last_active":"2024-03-11T21:38:32.477658+0000","last_peered":"2024-03-11T21:38:32.477658+0000","last_clean":"2024-03-11T21:35:44.669493+0000","last_became_active":"2024-03-11T21:36:36.508974+0000","last_became_peered":"2024-03-11T21:36:36.508974+0000","last_unstale":"2024-03-11T21:38:32.477658+0000","last_undegraded":"2024-03-11T21:36:36.183555+0000","last_fullsized":"2024-03-11T21:36:36.038772+0000","mapping_epoch":278,"log_start":"282'5200","ondisk_log_start":"282'5200","created":259,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:35:44.669493+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:35:44.669493+0000","last_clean_scrub_stamp":"2024-03-11T21:35:44.669493+0000","objects_scrubbed":0,"log_size":5239,"log_dups_size":0,"ondisk_log_size":5239,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":267976704,"num_objects":4089,"num_object_clones":0,"num_object_copies":12267,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1298,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4089,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10439,"num_write_kb":464896,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3454,"num_bytes_recovered":226361344,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,7],"avail_no_missing":["5","7"],"object_location_counts":[{"shards":"5,7","objects":2791}],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":210,"reported_epoch":322,"state":"active+clean","last_fresh":"2024-03-11T21:38:30.607094+0000","last_change":"2024-03-11T21:36:06.735035+0000","last_active":"2024-03-11T21:38:30.607094+0000","last_peered":"2024-03-11T21:38:30.607094+0000","last_clean":"2024-03-11T21:38:30.607094+0000","last_became_active":"2024-03-11T21:34:41.145012+0000","last_became_peered":"2024-03-11T21:34:41.145012+0000","last_unstale":"2024-03-11T21:38:30.607094+0000","last_undegraded":"2024-03-11T21:38:30.607094+0000","last_fullsized":"2024-03-11T21:38:30.607094+0000","mapping_epoch":226,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":227,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:36:06.734921+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:28:37.561826+0000","last_clean_scrub_stamp":"2024-03-11T21:36:06.734921+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":23,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,0],"acting":[7,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":557,"reported_epoch":322,"state":"active+clean","last_fresh":"2024-03-11T21:38:30.658175+0000","last_change":"2024-03-11T21:35:14.636561+0000","last_active":"2024-03-11T21:38:30.658175+0000","last_peered":"2024-03-11T21:38:30.658175+0000","last_clean":"2024-03-11T21:38:30.658175+0000","last_became_active":"2024-03-11T21:25:42.574515+0000","last_became_peered":"2024-03-11T21:25:42.574515+0000","last_unstale":"2024-03-11T21:38:30.658175+0000","last_undegraded":"2024-03-11T21:38:30.658175+0000","last_fullsized":"2024-03-11T21:38:30.658175+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:35:14.636450+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:41.544550+0000","last_clean_scrub_stamp":"2024-03-11T21:35:14.636450+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,5],"acting":[2,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.c","version":"0'0","reported_seq":565,"reported_epoch":322,"state":"active+clean","last_fresh":"2024-03-11T21:38:30.658834+0000","last_change":"2024-03-11T21:36:00.849272+0000","last_active":"2024-03-11T21:38:30.658834+0000","last_peered":"2024-03-11T21:38:30.658834+0000","last_clean":"2024-03-11T21:38:30.658834+0000","last_became_active":"2024-03-11T21:25:38.533240+0000","last_became_peered":"2024-03-11T21:25:38.533240+0000","last_unstale":"2024-03-11T21:38:30.658834+0000","last_undegraded":"2024-03-11T21:38:30.658834+0000","last_fullsized":"2024-03-11T21:38:30.658834+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:36:00.849153+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:29:30.279910+0000","last_clean_scrub_stamp":"2024-03-11T21:36:00.849153+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,0],"acting":[2,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.a","version":"260'44","reported_seq":686,"reported_epoch":322,"state":"active+clean","last_fresh":"2024-03-11T21:38:30.669237+0000","last_change":"2024-03-11T21:35:34.138400+0000","last_active":"2024-03-11T21:38:30.669237+0000","last_peered":"2024-03-11T21:38:30.669237+0000","last_clean":"2024-03-11T21:38:30.669237+0000","last_became_active":"2024-03-11T21:34:41.148863+0000","last_became_peered":"2024-03-11T21:34:41.148863+0000","last_unstale":"2024-03-11T21:38:30.669237+0000","last_undegraded":"2024-03-11T21:38:30.669237+0000","last_fullsized":"2024-03-11T21:38:30.669237+0000","mapping_epoch":226,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":227,"parent":"0.0","parent_split_bits":0,"last_scrub":"109'28","last_scrub_stamp":"2024-03-11T21:35:34.138261+0000","last_deep_scrub":"109'28","last_deep_scrub_stamp":"2024-03-11T21:28:24.016389+0000","last_clean_scrub_stamp":"2024-03-11T21:35:34.138261+0000","objects_scrubbed":9,"log_size":44,"log_dups_size":0,"ondisk_log_size":44,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:38:32.120441+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":56,"num_read_kb":45,"num_write":36,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,7],"acting":[6,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.6","version":"322'10521","reported_seq":10658,"reported_epoch":322,"state":"active+clean","last_fresh":"2024-03-11T21:38:32.458772+0000","last_change":"2024-03-11T21:35:45.703533+0000","last_active":"2024-03-11T21:38:32.458772+0000","last_peered":"2024-03-11T21:38:32.458772+0000","last_clean":"2024-03-11T21:38:32.458772+0000","last_became_active":"2024-03-11T21:35:45.703184+0000","last_became_peered":"2024-03-11T21:35:45.703184+0000","last_unstale":"2024-03-11T21:38:32.458772+0000","last_undegraded":"2024-03-11T21:38:32.458772+0000","last_fullsized":"2024-03-11T21:38:32.458772+0000","mapping_epoch":259,"log_start":"283'5300","ondisk_log_start":"283'5300","created":259,"last_epoch_clean":260,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:35:44.669493+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:35:44.669493+0000","last_clean_scrub_stamp":"2024-03-11T21:35:44.669493+0000","objects_scrubbed":0,"log_size":5221,"log_dups_size":0,"ondisk_log_size":5221,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":263782400,"num_objects":4025,"num_object_clones":0,"num_object_copies":12075,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4025,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10521,"num_write_kb":465472,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,1],"acting":[5,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":562,"reported_epoch":320,"state":"active+clean","last_fresh":"2024-03-11T21:38:28.356012+0000","last_change":"2024-03-11T21:35:37.687281+0000","last_active":"2024-03-11T21:38:28.356012+0000","last_peered":"2024-03-11T21:38:28.356012+0000","last_clean":"2024-03-11T21:38:28.356012+0000","last_became_active":"2024-03-11T21:34:41.154507+0000","last_became_peered":"2024-03-11T21:34:41.154507+0000","last_unstale":"2024-03-11T21:38:28.356012+0000","last_undegraded":"2024-03-11T21:38:28.356012+0000","last_fullsized":"2024-03-11T21:38:28.356012+0000","mapping_epoch":226,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":227,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:35:37.687187+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:27:11.361913+0000","last_clean_scrub_stamp":"2024-03-11T21:35:37.687187+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":39,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":558,"reported_epoch":322,"state":"active+clean","last_fresh":"2024-03-11T21:38:30.658681+0000","last_change":"2024-03-11T21:36:35.059662+0000","last_active":"2024-03-11T21:38:30.658681+0000","last_peered":"2024-03-11T21:38:30.658681+0000","last_clean":"2024-03-11T21:38:30.658681+0000","last_became_active":"2024-03-11T21:36:35.059192+0000","last_became_peered":"2024-03-11T21:36:35.059192+0000","last_unstale":"2024-03-11T21:38:30.658681+0000","last_undegraded":"2024-03-11T21:38:30.658681+0000","last_fullsized":"2024-03-11T21:38:30.658681+0000","mapping_epoch":277,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":278,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:34:17.412687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:41.544550+0000","last_clean_scrub_stamp":"2024-03-11T21:34:17.412687+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,3],"acting":[2,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.d","version":"0'0","reported_seq":567,"reported_epoch":322,"state":"active+clean","last_fresh":"2024-03-11T21:38:30.602318+0000","last_change":"2024-03-11T21:36:35.058635+0000","last_active":"2024-03-11T21:38:30.602318+0000","last_peered":"2024-03-11T21:38:30.602318+0000","last_clean":"2024-03-11T21:38:30.602318+0000","last_became_active":"2024-03-11T21:36:35.058296+0000","last_became_peered":"2024-03-11T21:36:35.058296+0000","last_unstale":"2024-03-11T21:38:30.602318+0000","last_undegraded":"2024-03-11T21:38:30.602318+0000","last_fullsized":"2024-03-11T21:38:30.602318+0000","mapping_epoch":277,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":278,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:31:55.464145+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:37.512018+0000","last_clean_scrub_stamp":"2024-03-11T21:31:55.464145+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,3],"acting":[1,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.b","version":"260'15","reported_seq":583,"reported_epoch":320,"state":"active+clean","last_fresh":"2024-03-11T21:38:28.356730+0000","last_change":"2024-03-11T21:35:27.760318+0000","last_active":"2024-03-11T21:38:28.356730+0000","last_peered":"2024-03-11T21:38:28.356730+0000","last_clean":"2024-03-11T21:38:28.356730+0000","last_became_active":"2024-03-11T21:25:40.549612+0000","last_became_peered":"2024-03-11T21:25:40.549612+0000","last_unstale":"2024-03-11T21:38:28.356730+0000","last_undegraded":"2024-03-11T21:38:28.356730+0000","last_fullsized":"2024-03-11T21:38:28.356730+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'9","last_scrub_stamp":"2024-03-11T21:35:27.760208+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:39.525554+0000","last_clean_scrub_stamp":"2024-03-11T21:35:27.760208+0000","objects_scrubbed":4,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.5","version":"320'10385","reported_seq":10518,"reported_epoch":320,"state":"active+clean","last_fresh":"2024-03-11T21:38:28.506491+0000","last_change":"2024-03-11T21:35:45.697209+0000","last_active":"2024-03-11T21:38:28.506491+0000","last_peered":"2024-03-11T21:38:28.506491+0000","last_clean":"2024-03-11T21:38:28.506491+0000","last_became_active":"2024-03-11T21:35:45.696788+0000","last_became_peered":"2024-03-11T21:35:45.696788+0000","last_unstale":"2024-03-11T21:38:28.506491+0000","last_undegraded":"2024-03-11T21:38:28.506491+0000","last_fullsized":"2024-03-11T21:38:28.506491+0000","mapping_epoch":259,"log_start":"276'4800","ondisk_log_start":"276'4800","created":259,"last_epoch_clean":260,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:35:44.669493+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:35:44.669493+0000","last_clean_scrub_stamp":"2024-03-11T21:35:44.669493+0000","objects_scrubbed":0,"log_size":5585,"log_dups_size":0,"ondisk_log_size":5585,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":289472512,"num_objects":4417,"num_object_clones":0,"num_object_copies":13251,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4417,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10385,"num_write_kb":473664,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":468,"reported_epoch":322,"state":"active+clean","last_fresh":"2024-03-11T21:38:30.601218+0000","last_change":"2024-03-11T21:36:35.057118+0000","last_active":"2024-03-11T21:38:30.601218+0000","last_peered":"2024-03-11T21:38:30.601218+0000","last_clean":"2024-03-11T21:38:30.601218+0000","last_became_active":"2024-03-11T21:36:35.056518+0000","last_became_peered":"2024-03-11T21:36:35.056518+0000","last_unstale":"2024-03-11T21:38:30.601218+0000","last_undegraded":"2024-03-11T21:38:30.601218+0000","last_fullsized":"2024-03-11T21:38:30.601218+0000","mapping_epoch":277,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":278,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:35:24.514584+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:43.559452+0000","last_clean_scrub_stamp":"2024-03-11T21:35:24.514584+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,3],"acting":[4,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.e","version":"0'0","reported_seq":586,"reported_epoch":322,"state":"active+clean","last_fresh":"2024-03-11T21:38:30.659833+0000","last_change":"2024-03-11T21:36:35.057187+0000","last_active":"2024-03-11T21:38:30.659833+0000","last_peered":"2024-03-11T21:38:30.659833+0000","last_clean":"2024-03-11T21:38:30.659833+0000","last_became_active":"2024-03-11T21:36:35.056777+0000","last_became_peered":"2024-03-11T21:36:35.056777+0000","last_unstale":"2024-03-11T21:38:30.659833+0000","last_undegraded":"2024-03-11T21:38:30.659833+0000","last_fullsized":"2024-03-11T21:38:30.659833+0000","mapping_epoch":277,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":278,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:34:19.476172+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:37.512018+0000","last_clean_scrub_stamp":"2024-03-11T21:34:19.476172+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":17,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,7],"acting":[2,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.8","version":"260'34","reported_seq":655,"reported_epoch":322,"state":"active+clean","last_fresh":"2024-03-11T21:38:30.615242+0000","last_change":"2024-03-11T21:35:20.304345+0000","last_active":"2024-03-11T21:38:30.615242+0000","last_peered":"2024-03-11T21:38:30.615242+0000","last_clean":"2024-03-11T21:38:30.615242+0000","last_became_active":"2024-03-11T21:34:41.146631+0000","last_became_peered":"2024-03-11T21:34:41.146631+0000","last_unstale":"2024-03-11T21:38:30.615242+0000","last_undegraded":"2024-03-11T21:38:30.615242+0000","last_fullsized":"2024-03-11T21:38:30.615242+0000","mapping_epoch":226,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":227,"parent":"0.0","parent_split_bits":0,"last_scrub":"109'22","last_scrub_stamp":"2024-03-11T21:35:20.304114+0000","last_deep_scrub":"109'22","last_deep_scrub_stamp":"2024-03-11T21:32:07.573197+0000","last_clean_scrub_stamp":"2024-03-11T21:35:20.304114+0000","objects_scrubbed":7,"log_size":34,"log_dups_size":0,"ondisk_log_size":34,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":44,"num_read_kb":35,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.9","version":"229'11","reported_seq":260,"reported_epoch":322,"state":"active+clean","last_fresh":"2024-03-11T21:38:30.607689+0000","last_change":"2024-03-11T21:36:18.726876+0000","last_active":"2024-03-11T21:38:30.607689+0000","last_peered":"2024-03-11T21:38:30.607689+0000","last_clean":"2024-03-11T21:38:30.607689+0000","last_became_active":"2024-03-11T21:34:41.156892+0000","last_became_peered":"2024-03-11T21:34:41.156892+0000","last_unstale":"2024-03-11T21:38:30.607689+0000","last_undegraded":"2024-03-11T21:38:30.607689+0000","last_fullsized":"2024-03-11T21:38:30.607689+0000","mapping_epoch":226,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":227,"parent":"0.0","parent_split_bits":0,"last_scrub":"229'11","last_scrub_stamp":"2024-03-11T21:36:18.726763+0000","last_deep_scrub":"94'9","last_deep_scrub_stamp":"2024-03-11T21:26:53.422552+0000","last_clean_scrub_stamp":"2024-03-11T21:36:18.726763+0000","objects_scrubbed":1,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":9,"num_pg":16,"stat_sum":{"num_bytes":4393074712,"num_objects":67034,"num_object_clones":0,"num_object_copies":201102,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":24747,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":67034,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":165730,"num_write_kb":7448385,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13276,"num_bytes_recovered":870055936,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12426686464,"data_stored":12426674248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":82735,"ondisk_log_size":82735,"up":48,"acting":41,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":1531,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":17,"num_read_kb":17,"num_write":10,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":1308,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":4593,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":6,"ondisk_log_size":6,"up":96,"acting":96,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":198,"num_read_kb":294,"num_write":179,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":2951840,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":2375680,"data_stored":2361472,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":128,"ondisk_log_size":128,"up":3,"acting":3,"num_store_stats":4},{"poolid":3,"num_pg":3,"stat_sum":{"num_bytes":408,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":767,"num_read_kb":762,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1224,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":8,"ondisk_log_size":8,"up":9,"acting":9,"num_store_stats":7},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":3702,"num_objects":178,"num_object_clones":0,"num_object_copies":534,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":178,"num_whiteouts":0,"num_read":1120,"num_read_kb":877,"num_write":703,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":186,"num_bytes_recovered":2200,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":417792,"data_stored":11106,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":760,"ondisk_log_size":760,"up":96,"acting":96,"num_store_stats":8},{"poolid":5,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":78,"ondisk_log_size":78,"up":96,"acting":96,"num_store_stats":8},{"poolid":6,"num_pg":32,"stat_sum":{"num_bytes":383,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":370,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1149,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":96,"acting":96,"num_store_stats":8}],"osd_stats":[{"osd":7,"up_from":60,"seq":257698037918,"num_pgs":65,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2893320,"kb_used_data":2420664,"kb_used_omap":1,"kb_used_meta":472638,"kb_avail":90851832,"statfs":{"total":95995035648,"available":93032275968,"internally_reserved":0,"allocated":2478759936,"data_stored":2477960607,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":483981770},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":50,"seq":214748364964,"num_pgs":46,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1171708,"kb_used_data":797620,"kb_used_omap":1,"kb_used_meta":374078,"kb_avail":92573444,"statfs":{"total":95995035648,"available":94795206656,"internally_reserved":0,"allocated":816762880,"data_stored":815968352,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":383056326},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834575038,"num_pgs":60,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2896280,"kb_used_data":2363792,"kb_used_omap":1,"kb_used_meta":532350,"kb_avail":90848872,"statfs":{"total":95995035648,"available":93029244928,"internally_reserved":0,"allocated":2420523008,"data_stored":2419699286,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":545126853},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738562,"num_pgs":54,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1631720,"kb_used_data":1127132,"kb_used_omap":1,"kb_used_meta":504574,"kb_avail":92113432,"statfs":{"total":95995035648,"available":94324154368,"internally_reserved":0,"allocated":1154183168,"data_stored":1153391656,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":516684229},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411512,"num_pgs":43,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1229748,"kb_used_data":793196,"kb_used_omap":1,"kb_used_meta":436542,"kb_avail":92515404,"statfs":{"total":95995035648,"available":94735773696,"internally_reserved":0,"allocated":812232704,"data_stored":811445592,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":447019461},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":277,"seq":1189705941016,"num_pgs":62,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1732852,"kb_used_data":1520108,"kb_used_omap":4,"kb_used_meta":212731,"kb_avail":92012300,"statfs":{"total":95995035648,"available":94220595200,"internally_reserved":0,"allocated":1556590592,"data_stored":1555782560,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4442,"internal_metadata":217837222},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,3,2,0,2],"upper_bound":32},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":4,"up_from":32,"seq":137438953645,"num_pgs":56,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1179536,"kb_used_data":797120,"kb_used_omap":1,"kb_used_meta":382398,"kb_avail":92565616,"statfs":{"total":95995035648,"available":94787190784,"internally_reserved":0,"allocated":816250880,"data_stored":815443136,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":391576005},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":41,"seq":176093659304,"num_pgs":58,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2905912,"kb_used_data":2391600,"kb_used_omap":1,"kb_used_meta":514302,"kb_avail":90839240,"statfs":{"total":95995035648,"available":93019381760,"internally_reserved":0,"allocated":2448998400,"data_stored":2448206831,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":526645701},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1049,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":46,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":436,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":957,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":574,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":436,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1003,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":92,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":1430,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":77824,"data_stored":2008,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":1018,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":1898,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":1474,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":990,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":1034,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":1254,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":370,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":383,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":370,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":1140064256,"data_stored":1140064256,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":2415792128,"data_stored":2415788056,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":803405824,"data_stored":803405824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":1551630336,"data_stored":1551630336,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":812187648,"data_stored":812187648,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":2442203136,"data_stored":2442199064,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":811401216,"data_stored":811401216,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":2450001920,"data_stored":2449997848,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-11T21:38:35.166 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:38:35.166 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:38:35.167 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:38:35.256 DEBUG:teuthology.orchestra.run.smithi092:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid 00db0016-dfed-11ee-95c8-87774f69a715 -- ceph osd unset nodeep-scrub 2024-03-11T21:38:35.268 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:38:35.340 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:38:35.341 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:38:35.341 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:38:35.421 INFO:teuthology.orchestra.run.smithi097.stdout:ERROR: (22) Invalid argument 2024-03-11T21:38:35.421 INFO:teuthology.orchestra.run.smithi097.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-11T21:38:35.442 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:38:35.482 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:38:35.483 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:38:35.483 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:38:35.584 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:38:35.623 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:38:35.624 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:38:35.624 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:38:35.725 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:38:35.753 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:38:35.754 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:38:35.754 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:38:35.855 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:38:35.882 INFO:teuthology.orchestra.run.smithi092.stdout:osd.0: {} 2024-03-11T21:38:35.909 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:38:35.909 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:38:35.910 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:38:35.923 INFO:teuthology.orchestra.run.smithi092.stdout:osd.1: {} 2024-03-11T21:38:35.961 INFO:teuthology.orchestra.run.smithi092.stdout:osd.2: {} 2024-03-11T21:38:35.972 INFO:teuthology.orchestra.run.smithi092.stderr:Inferring config /var/lib/ceph/00db0016-dfed-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T21:38:36.001 INFO:teuthology.orchestra.run.smithi092.stdout:osd.3: {} 2024-03-11T21:38:36.011 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:38:36.044 INFO:teuthology.orchestra.run.smithi092.stdout:osd.4: {} 2024-03-11T21:38:36.073 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:38:36.073 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:38:36.074 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:38:36.088 INFO:teuthology.orchestra.run.smithi092.stdout:osd.5: {} 2024-03-11T21:38:36.101 DEBUG:teuthology.orchestra.run.smithi097:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid 00db0016-dfed-11ee-95c8-87774f69a715 -- ceph daemon osd.7 dump_ops_in_flight 2024-03-11T21:38:36.123 INFO:teuthology.orchestra.run.smithi092.stdout:osd.6: {} 2024-03-11T21:38:36.161 INFO:teuthology.orchestra.run.smithi092.stdout:osd.7: {} 2024-03-11T21:38:36.168 INFO:teuthology.orchestra.run.smithi092.stderr:osd.0: osd_enable_op_tracker = '' 2024-03-11T21:38:36.168 INFO:teuthology.orchestra.run.smithi092.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-11T21:38:36.169 INFO:teuthology.orchestra.run.smithi092.stderr:osd.2: osd_enable_op_tracker = '' 2024-03-11T21:38:36.169 INFO:teuthology.orchestra.run.smithi092.stderr:osd.3: osd_enable_op_tracker = '' 2024-03-11T21:38:36.169 INFO:teuthology.orchestra.run.smithi092.stderr:osd.4: osd_enable_op_tracker = '' 2024-03-11T21:38:36.169 INFO:teuthology.orchestra.run.smithi092.stderr:osd.5: osd_enable_op_tracker = '' 2024-03-11T21:38:36.169 INFO:teuthology.orchestra.run.smithi092.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-11T21:38:36.169 INFO:teuthology.orchestra.run.smithi092.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-11T21:38:36.174 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:38:36.206 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:38:36.206 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:38:36.206 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:38:36.307 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:38:36.343 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:38:36.344 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:38:36.344 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:38:36.445 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:38:36.450 INFO:teuthology.orchestra.run.smithi097.stderr:Inferring config /var/lib/ceph/00db0016-dfed-11ee-95c8-87774f69a715/mon.b/config 2024-03-11T21:38:36.480 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:38:36.480 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:38:36.480 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:38:36.581 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:38:36.620 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:38:36.621 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:38:36.621 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:38:36.722 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:38:36.768 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:38:36.769 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:38:36.769 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:38:36.870 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:38:36.880 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:38:36 smithi097 ceph-mon[33317]: pgmap v605: 148 pgs: 3 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+degraded+remapped, 142 active+clean; 4.0 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 739 B/s rd, 0 B/s wr, 668 op/s; 24637/197877 objects degraded (12.451%); 6.5 MiB/s, 103 objects/s recovering 2024-03-11T21:38:36.880 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:38:36 smithi097 ceph-mon[33317]: from='client.? v1:172.21.15.92:0/2872158326' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-11T21:38:36.880 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:38:36 smithi097 ceph-mon[33317]: 2.14 scrub starts 2024-03-11T21:38:36.880 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:38:36 smithi097 ceph-mon[33317]: 2.14 scrub ok 2024-03-11T21:38:36.906 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:38:36.906 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:38:36.907 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:38:36.948 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:38:36 smithi092 ceph-mon[29572]: pgmap v605: 148 pgs: 3 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+degraded+remapped, 142 active+clean; 4.0 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 739 B/s rd, 0 B/s wr, 668 op/s; 24637/197877 objects degraded (12.451%); 6.5 MiB/s, 103 objects/s recovering 2024-03-11T21:38:36.948 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:38:36 smithi092 ceph-mon[29572]: from='client.? v1:172.21.15.92:0/2872158326' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-11T21:38:36.948 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:38:36 smithi092 ceph-mon[29572]: 2.14 scrub starts 2024-03-11T21:38:36.948 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:38:36 smithi092 ceph-mon[29572]: 2.14 scrub ok 2024-03-11T21:38:36.948 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:38:36 smithi092 ceph-mon[37563]: pgmap v605: 148 pgs: 3 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+degraded+remapped, 142 active+clean; 4.0 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 739 B/s rd, 0 B/s wr, 668 op/s; 24637/197877 objects degraded (12.451%); 6.5 MiB/s, 103 objects/s recovering 2024-03-11T21:38:36.949 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:38:36 smithi092 ceph-mon[37563]: from='client.? v1:172.21.15.92:0/2872158326' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-11T21:38:36.949 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:38:36 smithi092 ceph-mon[37563]: 2.14 scrub starts 2024-03-11T21:38:36.949 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:38:36 smithi092 ceph-mon[37563]: 2.14 scrub ok 2024-03-11T21:38:37.007 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:38:37.049 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:38:37.050 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:38:37.050 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:38:37.151 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:38:37.191 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:38:37.191 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:38:37.192 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:38:37.292 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:38:37.320 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:38:37.321 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:38:37.321 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:38:37.422 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:38:37.453 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:38:37.453 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:38:37.454 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:38:37.554 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:38:37.584 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:38:37.585 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:38:37.585 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:38:37.686 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:38:37.716 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:38:37.717 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:38:37.717 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:38:37.818 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:38:37.871 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:38:37.872 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:38:37.872 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:38:37.926 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:38:37 smithi092 ceph-mon[29572]: from='mgr.14607 v1:172.21.15.92:0/1702717597' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T21:38:37.927 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:38:37 smithi092 ceph-mon[37563]: from='mgr.14607 v1:172.21.15.92:0/1702717597' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T21:38:37.972 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:38:37.998 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:38:37 smithi097 ceph-mon[33317]: from='mgr.14607 v1:172.21.15.92:0/1702717597' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T21:38:38.024 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:38:38.024 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:38:38.024 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:38:38.125 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:38:38.163 DEBUG:teuthology.orchestra.run.smithi092:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid 00db0016-dfed-11ee-95c8-87774f69a715 -- ceph pg dump --format=json 2024-03-11T21:38:38.167 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:38:38.168 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:38:38.168 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:38:38.244 INFO:journalctl@ceph.mgr.y.smithi092.stdout:Mar 11 21:38:37 smithi092 ceph-00db0016-dfed-11ee-95c8-87774f69a715-mgr-y[29801]: ::ffff:172.21.15.97 - - [11/Mar/2024:21:38:37] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-11T21:38:38.269 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:38:38.301 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:38:38.301 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:38:38.302 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:38:38.403 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:38:38.435 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:38:38.436 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:38:38.436 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:38:38.493 INFO:teuthology.orchestra.run.smithi097.stdout:{ 2024-03-11T21:38:38.493 INFO:teuthology.orchestra.run.smithi097.stdout: "ops": [ 2024-03-11T21:38:38.493 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:38:38.493 INFO:teuthology.orchestra.run.smithi097.stdout: "description": "osd_op(client.25282.0:171252 9.3 9:cdda949f:::benchmark_data_smithi092_130291_object54868:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e322)", 2024-03-11T21:38:38.493 INFO:teuthology.orchestra.run.smithi097.stdout: "initiated_at": "2024-03-11T21:38:38.492771+0000", 2024-03-11T21:38:38.494 INFO:teuthology.orchestra.run.smithi097.stdout: "age": 0.016417971, 2024-03-11T21:38:38.494 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.016441113, 2024-03-11T21:38:38.494 INFO:teuthology.orchestra.run.smithi097.stdout: "type_data": { 2024-03-11T21:38:38.494 INFO:teuthology.orchestra.run.smithi097.stdout: "flag_point": "waiting for sub ops", 2024-03-11T21:38:38.494 INFO:teuthology.orchestra.run.smithi097.stdout: "client_info": { 2024-03-11T21:38:38.494 INFO:teuthology.orchestra.run.smithi097.stdout: "client": "client.25282", 2024-03-11T21:38:38.494 INFO:teuthology.orchestra.run.smithi097.stdout: "client_addr": "v1:172.21.15.92:0/2489519821", 2024-03-11T21:38:38.494 INFO:teuthology.orchestra.run.smithi097.stdout: "tid": 171252 2024-03-11T21:38:38.494 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:38:38.494 INFO:teuthology.orchestra.run.smithi097.stdout: "events": [ 2024-03-11T21:38:38.494 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:38:38.494 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "initiated", 2024-03-11T21:38:38.494 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:38:38.492771+0000", 2024-03-11T21:38:38.494 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-03-11T21:38:38.495 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:38:38.495 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:38:38.495 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "header_read", 2024-03-11T21:38:38.495 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:38:38.492771+0000", 2024-03-11T21:38:38.495 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-03-11T21:38:38.495 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:38:38.495 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:38:38.495 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "throttled", 2024-03-11T21:38:38.495 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:38:38.492772+0000", 2024-03-11T21:38:38.495 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.302e-06 2024-03-11T21:38:38.495 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:38:38.495 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:38:38.495 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "all_read", 2024-03-11T21:38:38.495 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:38:38.492777+0000", 2024-03-11T21:38:38.495 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 4.7219999999999999e-06 2024-03-11T21:38:38.496 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:38:38.496 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:38:38.496 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "dispatched", 2024-03-11T21:38:38.496 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:38:38.492790+0000", 2024-03-11T21:38:38.496 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.3281e-05 2024-03-11T21:38:38.496 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:38:38.496 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:38:38.496 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "queued_for_pg", 2024-03-11T21:38:38.496 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:38:38.492798+0000", 2024-03-11T21:38:38.496 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 7.6159999999999996e-06 2024-03-11T21:38:38.496 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:38:38.496 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:38:38.496 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "reached_pg", 2024-03-11T21:38:38.496 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:38:38.492853+0000", 2024-03-11T21:38:38.497 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 5.5077999999999997e-05 2024-03-11T21:38:38.497 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:38:38.497 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:38:38.497 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "started", 2024-03-11T21:38:38.497 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:38:38.492994+0000", 2024-03-11T21:38:38.497 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.000141058 2024-03-11T21:38:38.497 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:38:38.497 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:38:38.497 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "waiting for subops from 1,3", 2024-03-11T21:38:38.497 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:38:38.493145+0000", 2024-03-11T21:38:38.497 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.000150972 2024-03-11T21:38:38.497 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:38:38.497 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:38:38.497 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "op_commit", 2024-03-11T21:38:38.497 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:38:38.495381+0000", 2024-03-11T21:38:38.498 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.0022360169999999999 2024-03-11T21:38:38.498 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:38:38.498 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:38:38.498 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "sub_op_commit_rec", 2024-03-11T21:38:38.498 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:38:38.496283+0000", 2024-03-11T21:38:38.498 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.00090208499999999995 2024-03-11T21:38:38.498 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-03-11T21:38:38.498 INFO:teuthology.orchestra.run.smithi097.stdout: ] 2024-03-11T21:38:38.498 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-03-11T21:38:38.498 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:38:38.498 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:38:38.498 INFO:teuthology.orchestra.run.smithi097.stdout: "description": "osd_op(client.25282.0:171255 9.3 9:ca28cfe7:::benchmark_data_smithi092_130291_object54871:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e322)", 2024-03-11T21:38:38.498 INFO:teuthology.orchestra.run.smithi097.stdout: "initiated_at": "2024-03-11T21:38:38.497016+0000", 2024-03-11T21:38:38.498 INFO:teuthology.orchestra.run.smithi097.stdout: "age": 0.012172397, 2024-03-11T21:38:38.498 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.012251523, 2024-03-11T21:38:38.499 INFO:teuthology.orchestra.run.smithi097.stdout: "type_data": { 2024-03-11T21:38:38.499 INFO:teuthology.orchestra.run.smithi097.stdout: "flag_point": "waiting for sub ops", 2024-03-11T21:38:38.499 INFO:teuthology.orchestra.run.smithi097.stdout: "client_info": { 2024-03-11T21:38:38.499 INFO:teuthology.orchestra.run.smithi097.stdout: "client": "client.25282", 2024-03-11T21:38:38.499 INFO:teuthology.orchestra.run.smithi097.stdout: "client_addr": "v1:172.21.15.92:0/2489519821", 2024-03-11T21:38:38.499 INFO:teuthology.orchestra.run.smithi097.stdout: "tid": 171255 2024-03-11T21:38:38.499 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:38:38.499 INFO:teuthology.orchestra.run.smithi097.stdout: "events": [ 2024-03-11T21:38:38.499 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:38:38.499 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "initiated", 2024-03-11T21:38:38.499 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:38:38.497016+0000", 2024-03-11T21:38:38.499 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-03-11T21:38:38.499 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:38:38.499 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:38:38.501 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "header_read", 2024-03-11T21:38:38.501 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:38:38.497016+0000", 2024-03-11T21:38:38.501 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-03-11T21:38:38.501 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:38:38.501 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:38:38.502 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "throttled", 2024-03-11T21:38:38.502 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:38:38.497017+0000", 2024-03-11T21:38:38.502 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 3.4999999999999998e-07 2024-03-11T21:38:38.502 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:38:38.502 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:38:38.502 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "all_read", 2024-03-11T21:38:38.502 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:38:38.497018+0000", 2024-03-11T21:38:38.502 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.823e-06 2024-03-11T21:38:38.502 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:38:38.502 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:38:38.502 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "dispatched", 2024-03-11T21:38:38.502 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:38:38.497022+0000", 2024-03-11T21:38:38.502 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 3.123e-06 2024-03-11T21:38:38.502 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:38:38.503 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:38:38.503 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "queued_for_pg", 2024-03-11T21:38:38.503 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:38:38.497025+0000", 2024-03-11T21:38:38.503 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 3.4450000000000001e-06 2024-03-11T21:38:38.503 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:38:38.503 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:38:38.503 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "reached_pg", 2024-03-11T21:38:38.503 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:38:38.497067+0000", 2024-03-11T21:38:38.503 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 4.1974000000000002e-05 2024-03-11T21:38:38.503 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:38:38.503 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:38:38.503 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "started", 2024-03-11T21:38:38.503 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:38:38.497183+0000", 2024-03-11T21:38:38.503 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.00011565600000000001 2024-03-11T21:38:38.503 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:38:38.503 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:38:38.504 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "waiting for subops from 1,3", 2024-03-11T21:38:38.504 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:38:38.497275+0000", 2024-03-11T21:38:38.504 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 9.1998999999999998e-05 2024-03-11T21:38:38.504 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:38:38.504 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:38:38.504 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "op_commit", 2024-03-11T21:38:38.504 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:38:38.500891+0000", 2024-03-11T21:38:38.504 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.0036161909999999999 2024-03-11T21:38:38.504 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:38:38.504 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:38:38.504 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "sub_op_commit_rec", 2024-03-11T21:38:38.504 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:38:38.504347+0000", 2024-03-11T21:38:38.504 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.0034558309999999999 2024-03-11T21:38:38.504 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-03-11T21:38:38.504 INFO:teuthology.orchestra.run.smithi097.stdout: ] 2024-03-11T21:38:38.505 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-03-11T21:38:38.505 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:38:38.505 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:38:38.505 INFO:teuthology.orchestra.run.smithi097.stdout: "description": "osd_op(client.25282.0:171256 9.3 9:c0634b9b:::benchmark_data_smithi092_130291_object54872:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e322)", 2024-03-11T21:38:38.505 INFO:teuthology.orchestra.run.smithi097.stdout: "initiated_at": "2024-03-11T21:38:38.497031+0000", 2024-03-11T21:38:38.505 INFO:teuthology.orchestra.run.smithi097.stdout: "age": 0.012157292, 2024-03-11T21:38:38.505 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.012285758000000001, 2024-03-11T21:38:38.505 INFO:teuthology.orchestra.run.smithi097.stdout: "type_data": { 2024-03-11T21:38:38.505 INFO:teuthology.orchestra.run.smithi097.stdout: "flag_point": "waiting for sub ops", 2024-03-11T21:38:38.505 INFO:teuthology.orchestra.run.smithi097.stdout: "client_info": { 2024-03-11T21:38:38.505 INFO:teuthology.orchestra.run.smithi097.stdout: "client": "client.25282", 2024-03-11T21:38:38.505 INFO:teuthology.orchestra.run.smithi097.stdout: "client_addr": "v1:172.21.15.92:0/2489519821", 2024-03-11T21:38:38.505 INFO:teuthology.orchestra.run.smithi097.stdout: "tid": 171256 2024-03-11T21:38:38.505 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:38:38.505 INFO:teuthology.orchestra.run.smithi097.stdout: "events": [ 2024-03-11T21:38:38.506 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:38:38.506 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "initiated", 2024-03-11T21:38:38.506 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:38:38.497031+0000", 2024-03-11T21:38:38.506 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-03-11T21:38:38.506 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:38:38.506 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:38:38.506 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "header_read", 2024-03-11T21:38:38.506 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:38:38.497031+0000", 2024-03-11T21:38:38.506 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-03-11T21:38:38.506 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:38:38.506 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:38:38.506 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "throttled", 2024-03-11T21:38:38.506 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:38:38.497032+0000", 2024-03-11T21:38:38.506 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 4.4799999999999999e-07 2024-03-11T21:38:38.506 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:38:38.507 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:38:38.507 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "all_read", 2024-03-11T21:38:38.507 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:38:38.497034+0000", 2024-03-11T21:38:38.507 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 2.198e-06 2024-03-11T21:38:38.507 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:38:38.507 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:38:38.507 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "dispatched", 2024-03-11T21:38:38.507 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:38:38.497038+0000", 2024-03-11T21:38:38.507 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 4.3789999999999999e-06 2024-03-11T21:38:38.507 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:38:38.507 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:38:38.507 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "queued_for_pg", 2024-03-11T21:38:38.507 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:38:38.497044+0000", 2024-03-11T21:38:38.507 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 5.0950000000000003e-06 2024-03-11T21:38:38.507 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:38:38.508 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:38:38.508 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "reached_pg", 2024-03-11T21:38:38.508 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:38:38.498953+0000", 2024-03-11T21:38:38.508 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.0019098299999999999 2024-03-11T21:38:38.508 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:38:38.508 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:38:38.508 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "started", 2024-03-11T21:38:38.508 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:38:38.499122+0000", 2024-03-11T21:38:38.508 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.000168367 2024-03-11T21:38:38.509 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:38:38.509 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:38:38.509 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "waiting for subops from 1,3", 2024-03-11T21:38:38.509 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:38:38.499277+0000", 2024-03-11T21:38:38.509 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.000155117 2024-03-11T21:38:38.509 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:38:38.509 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:38:38.509 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "op_commit", 2024-03-11T21:38:38.509 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:38:38.501018+0000", 2024-03-11T21:38:38.509 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.001740716 2024-03-11T21:38:38.509 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:38:38.509 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:38:38.509 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "sub_op_commit_rec", 2024-03-11T21:38:38.509 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:38:38.504389+0000", 2024-03-11T21:38:38.509 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.0033710070000000001 2024-03-11T21:38:38.510 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-03-11T21:38:38.510 INFO:teuthology.orchestra.run.smithi097.stdout: ] 2024-03-11T21:38:38.510 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-03-11T21:38:38.510 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:38:38.510 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:38:38.510 INFO:teuthology.orchestra.run.smithi097.stdout: "description": "osd_repop(client.25282.0:171262 9.5 e322/259 9:ab28e00f:::benchmark_data_smithi092_130291_object54878:head v 322'10835, mlcod=322'10833)", 2024-03-11T21:38:38.510 INFO:teuthology.orchestra.run.smithi097.stdout: "initiated_at": "2024-03-11T21:38:38.508530+0000", 2024-03-11T21:38:38.510 INFO:teuthology.orchestra.run.smithi097.stdout: "age": 0.00065830100000000003, 2024-03-11T21:38:38.510 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.00083559499999999996, 2024-03-11T21:38:38.510 INFO:teuthology.orchestra.run.smithi097.stdout: "type_data": { 2024-03-11T21:38:38.510 INFO:teuthology.orchestra.run.smithi097.stdout: "flag_point": "started", 2024-03-11T21:38:38.510 INFO:teuthology.orchestra.run.smithi097.stdout: "events": [ 2024-03-11T21:38:38.510 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:38:38.510 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "initiated", 2024-03-11T21:38:38.510 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:38:38.508530+0000", 2024-03-11T21:38:38.511 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-03-11T21:38:38.511 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:38:38.511 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:38:38.511 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "header_read", 2024-03-11T21:38:38.511 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:38:38.508530+0000", 2024-03-11T21:38:38.511 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-03-11T21:38:38.511 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:38:38.511 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:38:38.511 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "throttled", 2024-03-11T21:38:38.511 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:38:38.508535+0000", 2024-03-11T21:38:38.511 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 4.5140000000000003e-06 2024-03-11T21:38:38.511 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:38:38.511 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:38:38.511 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "all_read", 2024-03-11T21:38:38.511 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:38:38.508541+0000", 2024-03-11T21:38:38.511 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 6.1090000000000003e-06 2024-03-11T21:38:38.512 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:38:38.512 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:38:38.512 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "dispatched", 2024-03-11T21:38:38.512 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:38:38.508550+0000", 2024-03-11T21:38:38.512 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 9.4089999999999992e-06 2024-03-11T21:38:38.512 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:38:38.512 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:38:38.512 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "queued_for_pg", 2024-03-11T21:38:38.512 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:38:38.508557+0000", 2024-03-11T21:38:38.512 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 6.7889999999999997e-06 2024-03-11T21:38:38.512 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:38:38.512 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:38:38.512 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "reached_pg", 2024-03-11T21:38:38.512 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:38:38.508620+0000", 2024-03-11T21:38:38.512 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 6.2521999999999995e-05 2024-03-11T21:38:38.513 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:38:38.513 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:38:38.513 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "started", 2024-03-11T21:38:38.513 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:38:38.508643+0000", 2024-03-11T21:38:38.513 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 2.3436999999999999e-05 2024-03-11T21:38:38.513 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-03-11T21:38:38.513 INFO:teuthology.orchestra.run.smithi097.stdout: ] 2024-03-11T21:38:38.513 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-03-11T21:38:38.513 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:38:38.513 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:38:38.513 INFO:teuthology.orchestra.run.smithi097.stdout: "description": "osd_repop(client.25282.0:171261 9.1 e322/316 9:860887dd:::benchmark_data_smithi092_130291_object54877:head v 322'10742, mlcod=322'10740)", 2024-03-11T21:38:38.513 INFO:teuthology.orchestra.run.smithi097.stdout: "initiated_at": "2024-03-11T21:38:38.508559+0000", 2024-03-11T21:38:38.513 INFO:teuthology.orchestra.run.smithi097.stdout: "age": 0.00062938000000000002, 2024-03-11T21:38:38.513 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.00084015000000000001, 2024-03-11T21:38:38.513 INFO:teuthology.orchestra.run.smithi097.stdout: "type_data": { 2024-03-11T21:38:38.514 INFO:teuthology.orchestra.run.smithi097.stdout: "flag_point": "started", 2024-03-11T21:38:38.514 INFO:teuthology.orchestra.run.smithi097.stdout: "events": [ 2024-03-11T21:38:38.514 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:38:38.514 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "initiated", 2024-03-11T21:38:38.514 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:38:38.508559+0000", 2024-03-11T21:38:38.514 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-03-11T21:38:38.514 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:38:38.514 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:38:38.514 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "header_read", 2024-03-11T21:38:38.514 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:38:38.508559+0000", 2024-03-11T21:38:38.514 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-03-11T21:38:38.514 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:38:38.514 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:38:38.514 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "throttled", 2024-03-11T21:38:38.514 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:38:38.508561+0000", 2024-03-11T21:38:38.515 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.2669999999999999e-06 2024-03-11T21:38:38.515 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:38:38.515 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:38:38.515 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "all_read", 2024-03-11T21:38:38.515 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:38:38.508567+0000", 2024-03-11T21:38:38.515 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 6.6019999999999996e-06 2024-03-11T21:38:38.515 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:38:38.515 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:38:38.515 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "dispatched", 2024-03-11T21:38:38.515 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:38:38.508576+0000", 2024-03-11T21:38:38.515 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 9.2399999999999996e-06 2024-03-11T21:38:38.515 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:38:38.515 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:38:38.515 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "queued_for_pg", 2024-03-11T21:38:38.515 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:38:38.508583+0000", 2024-03-11T21:38:38.516 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 7.0729999999999999e-06 2024-03-11T21:38:38.516 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:38:38.516 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:38:38.516 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "reached_pg", 2024-03-11T21:38:38.516 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:38:38.508649+0000", 2024-03-11T21:38:38.516 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 6.5025999999999997e-05 2024-03-11T21:38:38.516 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:38:38.516 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:38:38.516 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "started", 2024-03-11T21:38:38.516 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:38:38.508669+0000", 2024-03-11T21:38:38.516 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 2.0582e-05 2024-03-11T21:38:38.516 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-03-11T21:38:38.516 INFO:teuthology.orchestra.run.smithi097.stdout: ] 2024-03-11T21:38:38.516 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-03-11T21:38:38.516 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-03-11T21:38:38.517 INFO:teuthology.orchestra.run.smithi097.stdout: ], 2024-03-11T21:38:38.517 INFO:teuthology.orchestra.run.smithi097.stdout: "num_ops": 5 2024-03-11T21:38:38.517 INFO:teuthology.orchestra.run.smithi097.stdout:} 2024-03-11T21:38:38.537 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:38:38.599 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:38:38.599 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:38:38.600 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:38:38.700 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:38:38.733 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:38:38.733 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:38:38.734 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:38:38.834 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:38:38.929 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:38:38.930 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:38:38.930 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:38:38.998 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:38:38 smithi092 ceph-mon[29572]: pgmap v606: 148 pgs: 3 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+degraded+remapped, 142 active+clean; 4.0 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 639 B/s rd, 0 B/s wr, 417 op/s; 24637/197877 objects degraded (12.451%); 5.6 MiB/s, 89 objects/s recovering 2024-03-11T21:38:38.998 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:38:38 smithi092 ceph-mon[29572]: Health check update: Degraded data redundancy: 24637/197877 objects degraded (12.451%), 6 pgs degraded, 6 pgs undersized (PG_DEGRADED) 2024-03-11T21:38:38.998 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:38:38 smithi092 ceph-mon[37563]: pgmap v606: 148 pgs: 3 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+degraded+remapped, 142 active+clean; 4.0 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 639 B/s rd, 0 B/s wr, 417 op/s; 24637/197877 objects degraded (12.451%); 5.6 MiB/s, 89 objects/s recovering 2024-03-11T21:38:38.998 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:38:38 smithi092 ceph-mon[37563]: Health check update: Degraded data redundancy: 24637/197877 objects degraded (12.451%), 6 pgs degraded, 6 pgs undersized (PG_DEGRADED) 2024-03-11T21:38:39.032 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:38:39.077 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:38:38 smithi097 ceph-mon[33317]: pgmap v606: 148 pgs: 3 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+degraded+remapped, 142 active+clean; 4.0 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 639 B/s rd, 0 B/s wr, 417 op/s; 24637/197877 objects degraded (12.451%); 5.6 MiB/s, 89 objects/s recovering 2024-03-11T21:38:39.078 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:38:38 smithi097 ceph-mon[33317]: Health check update: Degraded data redundancy: 24637/197877 objects degraded (12.451%), 6 pgs degraded, 6 pgs undersized (PG_DEGRADED) 2024-03-11T21:38:39.083 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:38:39.083 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:38:39.084 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:38:39.184 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:38:39.203 INFO:teuthology.orchestra.run.smithi092.stderr:Inferring config /var/lib/ceph/00db0016-dfed-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T21:38:39.241 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:38:39.243 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:38:39.243 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:38:39.245 DEBUG:teuthology.orchestra.run.smithi097:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid 00db0016-dfed-11ee-95c8-87774f69a715 -- ceph daemon osd.7 dump_blocked_ops 2024-03-11T21:38:39.344 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:38:39.372 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:38:39.372 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:38:39.373 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:38:39.473 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:38:39.515 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:38:39.515 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:38:39.516 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:38:39.559 INFO:teuthology.orchestra.run.smithi097.stderr:Inferring config /var/lib/ceph/00db0016-dfed-11ee-95c8-87774f69a715/mon.b/config 2024-03-11T21:38:39.616 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:38:39.644 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:38:39.645 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:38:39.645 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:38:39.678 INFO:teuthology.orchestra.run.smithi092.stderr:nodeep-scrub is unset 2024-03-11T21:38:39.746 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:38:39.788 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:38:39.788 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:38:39.789 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:38:39.889 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:38:39.935 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:38:39.936 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:38:39.936 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:38:39.941 DEBUG:teuthology.orchestra.run.smithi092:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid 00db0016-dfed-11ee-95c8-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-03-11T21:38:39.975 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:38:39 smithi097 ceph-mon[33317]: from='client.? v1:172.21.15.92:0/648545035' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-11T21:38:39.976 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:38:39 smithi097 ceph-mon[33317]: from='client.25474 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-11T21:38:39.987 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:38:39 smithi092 ceph-mon[29572]: from='client.? v1:172.21.15.92:0/648545035' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-11T21:38:39.987 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:38:39 smithi092 ceph-mon[29572]: from='client.25474 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-11T21:38:39.988 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:38:39 smithi092 ceph-mon[37563]: from='client.? v1:172.21.15.92:0/648545035' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-11T21:38:39.988 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:38:39 smithi092 ceph-mon[37563]: from='client.25474 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-11T21:38:40.037 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:38:40.072 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:38:40.073 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:38:40.073 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:38:40.174 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:38:40.219 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:38:40.220 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:38:40.220 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:38:40.321 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:38:40.362 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:38:40.363 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:38:40.363 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:38:40.464 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:38:40.492 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:38:40.493 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:38:40.493 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:38:40.594 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:38:40.639 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:38:40.640 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:38:40.640 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:38:40.741 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:38:40.773 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:38:40.773 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:38:40.774 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:38:40.875 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:38:40.910 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:38:40.911 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:38:40.912 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:38:41.012 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:38:41.046 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:38:41.046 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:38:41.047 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:38:41.078 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:38:40 smithi097 ceph-mon[33317]: pgmap v607: 148 pgs: 3 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+degraded+remapped, 142 active+clean; 3.8 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 1.0 KiB/s rd, 0 B/s wr, 740 op/s; 24190/186429 objects degraded (12.975%); 7.5 MiB/s, 120 objects/s recovering 2024-03-11T21:38:41.078 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:38:40 smithi097 ceph-mon[33317]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-03-11T21:38:41.078 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:38:40 smithi097 ceph-mon[33317]: from='client.25474 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-03-11T21:38:41.078 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:38:40 smithi097 ceph-mon[33317]: osdmap e323: 8 total, 8 up, 8 in 2024-03-11T21:38:41.112 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:38:40 smithi092 ceph-mon[29572]: pgmap v607: 148 pgs: 3 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+degraded+remapped, 142 active+clean; 3.8 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 1.0 KiB/s rd, 0 B/s wr, 740 op/s; 24190/186429 objects degraded (12.975%); 7.5 MiB/s, 120 objects/s recovering 2024-03-11T21:38:41.112 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:38:40 smithi092 ceph-mon[29572]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-03-11T21:38:41.112 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:38:40 smithi092 ceph-mon[29572]: from='client.25474 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-03-11T21:38:41.112 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:38:40 smithi092 ceph-mon[29572]: osdmap e323: 8 total, 8 up, 8 in 2024-03-11T21:38:41.112 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:38:40 smithi092 ceph-mon[37563]: pgmap v607: 148 pgs: 3 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+degraded+remapped, 142 active+clean; 3.8 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 1.0 KiB/s rd, 0 B/s wr, 740 op/s; 24190/186429 objects degraded (12.975%); 7.5 MiB/s, 120 objects/s recovering 2024-03-11T21:38:41.112 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:38:40 smithi092 ceph-mon[37563]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-03-11T21:38:41.112 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:38:40 smithi092 ceph-mon[37563]: from='client.25474 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-03-11T21:38:41.113 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:38:40 smithi092 ceph-mon[37563]: osdmap e323: 8 total, 8 up, 8 in 2024-03-11T21:38:41.148 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:38:41.194 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:38:41.195 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:38:41.195 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:38:41.296 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:38:41.323 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:38:41.323 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:38:41.324 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:38:41.425 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:38:41.457 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:38:41.459 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:38:41.459 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:38:41.560 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:38:41.595 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:38:41.596 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:38:41.596 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:38:41.697 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:38:41.733 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:38:41.733 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:38:41.734 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:38:41.835 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:38:41.867 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:38:41.868 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:38:41.868 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:38:41.969 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:38:41.999 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:38:42.000 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:38:42.000 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:38:42.101 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:38:42.140 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:38:42.141 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:38:42.141 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:38:42.242 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:38:42.273 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:38:42.274 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:38:42.274 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:38:42.375 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:38:42.412 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:38:42.413 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:38:42.413 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:38:42.514 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:38:42.546 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:38:42.546 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:38:42.547 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:38:42.547 INFO:teuthology.orchestra.run.smithi097.stdout:{ 2024-03-11T21:38:42.547 INFO:teuthology.orchestra.run.smithi097.stdout: "ops": [], 2024-03-11T21:38:42.547 INFO:teuthology.orchestra.run.smithi097.stdout: "complaint_time": 30, 2024-03-11T21:38:42.547 INFO:teuthology.orchestra.run.smithi097.stdout: "num_blocked_ops": 0 2024-03-11T21:38:42.547 INFO:teuthology.orchestra.run.smithi097.stdout:} 2024-03-11T21:38:42.647 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:38:42.691 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:38:42.692 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:38:42.692 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:38:42.793 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:38:42.828 INFO:journalctl@ceph.iscsi.iscsi.a.smithi097.stdout:Mar 11 21:38:42 smithi097 ceph-00db0016-dfed-11ee-95c8-87774f69a715-iscsi-iscsi-a[60943]: debug there is no tcmu-runner data available 2024-03-11T21:38:42.828 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:38:42 smithi097 ceph-mon[33317]: pgmap v609: 148 pgs: 3 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+degraded+remapped, 142 active+clean; 3.8 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 716 op/s; 24190/186429 objects degraded (12.975%); 7.3 MiB/s, 116 objects/s recovering 2024-03-11T21:38:42.830 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:38:42.830 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:38:42.831 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:38:42.932 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:38:42.956 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:38:42.957 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:38:42.957 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:38:43.058 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:38:43.087 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:38:43.087 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:38:43.088 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:38:43.112 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:38:42 smithi092 ceph-mon[29572]: pgmap v609: 148 pgs: 3 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+degraded+remapped, 142 active+clean; 3.8 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 716 op/s; 24190/186429 objects degraded (12.975%); 7.3 MiB/s, 116 objects/s recovering 2024-03-11T21:38:43.112 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:38:42 smithi092 ceph-mon[37563]: pgmap v609: 148 pgs: 3 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+degraded+remapped, 142 active+clean; 3.8 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 716 op/s; 24190/186429 objects degraded (12.975%); 7.3 MiB/s, 116 objects/s recovering 2024-03-11T21:38:43.188 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:38:43.226 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:38:43.226 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:38:43.227 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:38:43.252 DEBUG:teuthology.orchestra.run.smithi097:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid 00db0016-dfed-11ee-95c8-87774f69a715 -- ceph daemon osd.7 dump_historic_ops 2024-03-11T21:38:43.327 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:38:43.355 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:38:43.356 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:38:43.356 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:38:43.457 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:38:43.496 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:38:43.497 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:38:43.497 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:38:43.577 INFO:teuthology.orchestra.run.smithi097.stderr:Inferring config /var/lib/ceph/00db0016-dfed-11ee-95c8-87774f69a715/mon.b/config 2024-03-11T21:38:43.597 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:38:43.634 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:38:43.635 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:38:43.636 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:38:43.736 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:38:43.774 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:38:43.775 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:38:43.776 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:38:43.876 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:38:43.906 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:38:43.907 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:38:43.907 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:38:43.997 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:38:43 smithi097 ceph-mon[33317]: from='client.24469 v1:172.21.15.97:0/404208310' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:38:43.997 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:38:43 smithi097 ceph-mon[33317]: Health check update: Degraded data redundancy: 24190/186429 objects degraded (12.975%), 6 pgs degraded, 6 pgs undersized (PG_DEGRADED) 2024-03-11T21:38:43.997 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:38:43 smithi097 ceph-mon[33317]: osdmap e324: 8 total, 8 up, 8 in 2024-03-11T21:38:44.008 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:38:44.041 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:38:44.042 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:38:44.042 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:38:44.111 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:38:43 smithi092 ceph-mon[29572]: from='client.24469 v1:172.21.15.97:0/404208310' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:38:44.112 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:38:43 smithi092 ceph-mon[29572]: Health check update: Degraded data redundancy: 24190/186429 objects degraded (12.975%), 6 pgs degraded, 6 pgs undersized (PG_DEGRADED) 2024-03-11T21:38:44.112 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:38:43 smithi092 ceph-mon[29572]: osdmap e324: 8 total, 8 up, 8 in 2024-03-11T21:38:44.112 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:38:43 smithi092 ceph-mon[37563]: from='client.24469 v1:172.21.15.97:0/404208310' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:38:44.112 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:38:43 smithi092 ceph-mon[37563]: Health check update: Degraded data redundancy: 24190/186429 objects degraded (12.975%), 6 pgs degraded, 6 pgs undersized (PG_DEGRADED) 2024-03-11T21:38:44.112 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:38:43 smithi092 ceph-mon[37563]: osdmap e324: 8 total, 8 up, 8 in 2024-03-11T21:38:44.142 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:38:44.175 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:38:44.176 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:38:44.176 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:38:44.277 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:38:44.310 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:38:44.310 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:38:44.311 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:38:44.412 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:38:44.445 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:38:44.446 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:38:44.447 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:38:44.547 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:38:44.585 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:38:44.585 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:38:44.586 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:38:44.686 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:38:44.716 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:38:44.717 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:38:44.717 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:38:44.818 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:38:44.821 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:38:44 smithi097 ceph-mon[33317]: pgmap v610: 148 pgs: 3 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+degraded+remapped, 142 active+clean; 3.6 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 761 op/s; 23713/178866 objects degraded (13.257%); 6.5 MiB/s, 103 objects/s recovering 2024-03-11T21:38:44.851 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:38:44.852 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:38:44.852 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:38:44.953 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:38:44.960 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:38:44 smithi092 ceph-mon[29572]: pgmap v610: 148 pgs: 3 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+degraded+remapped, 142 active+clean; 3.6 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 761 op/s; 23713/178866 objects degraded (13.257%); 6.5 MiB/s, 103 objects/s recovering 2024-03-11T21:38:44.961 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:38:44 smithi092 ceph-mon[37563]: pgmap v610: 148 pgs: 3 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+degraded+remapped, 142 active+clean; 3.6 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 761 op/s; 23713/178866 objects degraded (13.257%); 6.5 MiB/s, 103 objects/s recovering 2024-03-11T21:38:44.996 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:38:44.997 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:38:44.997 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:38:45.098 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:38:45.133 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:38:45.133 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:38:45.134 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:38:45.234 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:38:45.258 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:38:45.259 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:38:45.259 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:38:45.360 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:38:45.394 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:38:45.395 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:38:45.395 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:38:45.496 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:38:45.522 INFO:teuthology.orchestra.run.smithi097.stdout:{ 2024-03-11T21:38:45.522 INFO:teuthology.orchestra.run.smithi097.stdout: "size": 20, 2024-03-11T21:38:45.522 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 600, 2024-03-11T21:38:45.522 INFO:teuthology.orchestra.run.smithi097.stdout: "ops": [ 2024-03-11T21:38:45.523 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:38:45.523 INFO:teuthology.orchestra.run.smithi097.stdout: "description": "osd_op(client.15150.0:150448 8.15 8:abfa6e92:::benchmark_data_smithi092_104865_object29947:head [delete] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e213)", 2024-03-11T21:38:45.523 INFO:teuthology.orchestra.run.smithi097.stdout: "initiated_at": "2024-03-11T21:34:01.652028+0000", 2024-03-11T21:38:45.523 INFO:teuthology.orchestra.run.smithi097.stdout: "age": 283.883138755, 2024-03-11T21:38:45.523 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.011345704, 2024-03-11T21:38:45.523 INFO:teuthology.orchestra.run.smithi097.stdout: "type_data": { 2024-03-11T21:38:45.523 INFO:teuthology.orchestra.run.smithi097.stdout: "flag_point": "reached pg", 2024-03-11T21:38:45.523 INFO:teuthology.orchestra.run.smithi097.stdout: "client_info": { 2024-03-11T21:38:45.523 INFO:teuthology.orchestra.run.smithi097.stdout: "client": "client.15150", 2024-03-11T21:38:45.523 INFO:teuthology.orchestra.run.smithi097.stdout: "client_addr": "v1:172.21.15.92:0/436606717", 2024-03-11T21:38:45.523 INFO:teuthology.orchestra.run.smithi097.stdout: "tid": 150448 2024-03-11T21:38:45.523 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:38:45.523 INFO:teuthology.orchestra.run.smithi097.stdout: "events": [ 2024-03-11T21:38:45.523 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:38:45.524 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "initiated", 2024-03-11T21:38:45.524 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:34:01.652028+0000", 2024-03-11T21:38:45.524 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-03-11T21:38:45.524 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:38:45.524 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:38:45.524 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "header_read", 2024-03-11T21:38:45.524 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:34:01.652028+0000", 2024-03-11T21:38:45.524 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-03-11T21:38:45.524 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:38:45.524 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:38:45.524 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "throttled", 2024-03-11T21:38:45.524 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:34:01.652030+0000", 2024-03-11T21:38:45.524 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 2.1009999999999999e-06 2024-03-11T21:38:45.524 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:38:45.524 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:38:45.525 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "all_read", 2024-03-11T21:38:45.525 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:34:01.652044+0000", 2024-03-11T21:38:45.525 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.4307e-05 2024-03-11T21:38:45.525 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:38:45.525 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:38:45.525 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "dispatched", 2024-03-11T21:38:45.525 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:34:01.652062+0000", 2024-03-11T21:38:45.525 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.7530999999999999e-05 2024-03-11T21:38:45.525 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:38:45.525 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:38:45.525 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "queued_for_pg", 2024-03-11T21:38:45.525 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:34:01.652072+0000", 2024-03-11T21:38:45.525 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.0032e-05 2024-03-11T21:38:45.525 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:38:45.525 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:38:45.526 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "reached_pg", 2024-03-11T21:38:45.526 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:34:01.652145+0000", 2024-03-11T21:38:45.526 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 7.3454000000000005e-05 2024-03-11T21:38:45.526 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:38:45.526 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:38:45.526 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "waiting for peered", 2024-03-11T21:38:45.526 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:34:01.652157+0000", 2024-03-11T21:38:45.526 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.1571e-05 2024-03-11T21:38:45.526 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:38:45.526 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:38:45.526 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "reached_pg", 2024-03-11T21:38:45.526 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:34:02.663273+0000", 2024-03-11T21:38:45.526 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.0111165280000001 2024-03-11T21:38:45.526 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:38:45.527 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:38:45.527 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "done", 2024-03-11T21:38:45.527 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:34:02.663373+0000", 2024-03-11T21:38:45.527 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.00010018000000000001 2024-03-11T21:38:45.527 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-03-11T21:38:45.527 INFO:teuthology.orchestra.run.smithi097.stdout: ] 2024-03-11T21:38:45.527 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-03-11T21:38:45.527 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:38:45.527 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:38:45.527 INFO:teuthology.orchestra.run.smithi097.stdout: "description": "osd_op(client.15150.0:150458 8.15 8:ad4d3f8b:::benchmark_data_smithi092_104865_object29957:head [delete] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e213)", 2024-03-11T21:38:45.527 INFO:teuthology.orchestra.run.smithi097.stdout: "initiated_at": "2024-03-11T21:34:01.652091+0000", 2024-03-11T21:38:45.527 INFO:teuthology.orchestra.run.smithi097.stdout: "age": 283.88307520000001, 2024-03-11T21:38:45.527 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.0164343689999999, 2024-03-11T21:38:45.527 INFO:teuthology.orchestra.run.smithi097.stdout: "type_data": { 2024-03-11T21:38:45.527 INFO:teuthology.orchestra.run.smithi097.stdout: "flag_point": "reached pg", 2024-03-11T21:38:45.527 INFO:teuthology.orchestra.run.smithi097.stdout: "client_info": { 2024-03-11T21:38:45.528 INFO:teuthology.orchestra.run.smithi097.stdout: "client": "client.15150", 2024-03-11T21:38:45.528 INFO:teuthology.orchestra.run.smithi097.stdout: "client_addr": "v1:172.21.15.92:0/436606717", 2024-03-11T21:38:45.528 INFO:teuthology.orchestra.run.smithi097.stdout: "tid": 150458 2024-03-11T21:38:45.528 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:38:45.528 INFO:teuthology.orchestra.run.smithi097.stdout: "events": [ 2024-03-11T21:38:45.528 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:38:45.528 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "initiated", 2024-03-11T21:38:45.528 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:34:01.652091+0000", 2024-03-11T21:38:45.528 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-03-11T21:38:45.528 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:38:45.528 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:38:45.528 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "header_read", 2024-03-11T21:38:45.528 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:34:01.652091+0000", 2024-03-11T21:38:45.528 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-03-11T21:38:45.528 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:38:45.528 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:38:45.529 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "throttled", 2024-03-11T21:38:45.529 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:34:01.652092+0000", 2024-03-11T21:38:45.529 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 7.6899999999999996e-07 2024-03-11T21:38:45.529 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:38:45.529 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:38:45.529 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "all_read", 2024-03-11T21:38:45.529 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:34:01.652096+0000", 2024-03-11T21:38:45.529 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 3.456e-06 2024-03-11T21:38:45.529 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:38:45.529 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:38:45.529 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "dispatched", 2024-03-11T21:38:45.529 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:34:01.652103+0000", 2024-03-11T21:38:45.529 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 6.9970000000000001e-06 2024-03-11T21:38:45.529 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:38:45.529 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:38:45.530 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "queued_for_pg", 2024-03-11T21:38:45.530 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:34:01.652114+0000", 2024-03-11T21:38:45.530 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.1342000000000001e-05 2024-03-11T21:38:45.530 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:38:45.530 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:38:45.530 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "reached_pg", 2024-03-11T21:38:45.530 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:34:01.652188+0000", 2024-03-11T21:38:45.530 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 7.4315999999999998e-05 2024-03-11T21:38:45.530 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:38:45.530 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:38:45.530 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "waiting for peered", 2024-03-11T21:38:45.530 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:34:01.652196+0000", 2024-03-11T21:38:45.530 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 8.2600000000000005e-06 2024-03-11T21:38:45.530 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:38:45.531 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:38:45.531 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "reached_pg", 2024-03-11T21:38:45.531 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:34:02.663624+0000", 2024-03-11T21:38:45.531 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.0114278000000001 2024-03-11T21:38:45.531 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:38:45.531 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:38:45.531 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "waiting for missing object", 2024-03-11T21:38:45.531 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:34:02.663744+0000", 2024-03-11T21:38:45.531 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.000119999 2024-03-11T21:38:45.531 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:38:45.531 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:38:45.531 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "reached_pg", 2024-03-11T21:38:45.531 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:34:02.668438+0000", 2024-03-11T21:38:45.531 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.0046941680000000003 2024-03-11T21:38:45.531 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:38:45.532 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:38:45.532 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "done", 2024-03-11T21:38:45.532 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:34:02.668526+0000", 2024-03-11T21:38:45.532 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 8.7262000000000006e-05 2024-03-11T21:38:45.532 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-03-11T21:38:45.532 INFO:teuthology.orchestra.run.smithi097.stdout: ] 2024-03-11T21:38:45.532 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-03-11T21:38:45.532 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:38:45.532 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:38:45.532 INFO:teuthology.orchestra.run.smithi097.stdout: "description": "osd_op(client.15150.0:150469 8.15 8:aaea3dc1:::benchmark_data_smithi092_104865_object29968:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e213)", 2024-03-11T21:38:45.532 INFO:teuthology.orchestra.run.smithi097.stdout: "initiated_at": "2024-03-11T21:34:01.657742+0000", 2024-03-11T21:38:45.532 INFO:teuthology.orchestra.run.smithi097.stdout: "age": 283.87742455900002, 2024-03-11T21:38:45.532 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.0148719820000001, 2024-03-11T21:38:45.532 INFO:teuthology.orchestra.run.smithi097.stdout: "type_data": { 2024-03-11T21:38:45.532 INFO:teuthology.orchestra.run.smithi097.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-11T21:38:45.532 INFO:teuthology.orchestra.run.smithi097.stdout: "client_info": { 2024-03-11T21:38:45.533 INFO:teuthology.orchestra.run.smithi097.stdout: "client": "client.15150", 2024-03-11T21:38:45.533 INFO:teuthology.orchestra.run.smithi097.stdout: "client_addr": "v1:172.21.15.92:0/436606717", 2024-03-11T21:38:45.533 INFO:teuthology.orchestra.run.smithi097.stdout: "tid": 150469 2024-03-11T21:38:45.533 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:38:45.533 INFO:teuthology.orchestra.run.smithi097.stdout: "events": [ 2024-03-11T21:38:45.533 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:38:45.533 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "initiated", 2024-03-11T21:38:45.533 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:34:01.657742+0000", 2024-03-11T21:38:45.533 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-03-11T21:38:45.533 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:38:45.533 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:38:45.533 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "header_read", 2024-03-11T21:38:45.533 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:34:01.657742+0000", 2024-03-11T21:38:45.533 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-03-11T21:38:45.533 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:38:45.534 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:38:45.534 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "throttled", 2024-03-11T21:38:45.534 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:34:01.657744+0000", 2024-03-11T21:38:45.534 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 2.1959999999999998e-06 2024-03-11T21:38:45.534 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:38:45.534 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:38:45.534 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "all_read", 2024-03-11T21:38:45.534 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:34:01.657750+0000", 2024-03-11T21:38:45.534 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 6.2280000000000003e-06 2024-03-11T21:38:45.534 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:38:45.534 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:38:45.534 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "dispatched", 2024-03-11T21:38:45.534 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:34:01.657761+0000", 2024-03-11T21:38:45.534 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.0933e-05 2024-03-11T21:38:45.534 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:38:45.534 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:38:45.535 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "queued_for_pg", 2024-03-11T21:38:45.535 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:34:01.657771+0000", 2024-03-11T21:38:45.535 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 9.7489999999999993e-06 2024-03-11T21:38:45.535 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:38:45.535 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:38:45.535 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "reached_pg", 2024-03-11T21:38:45.535 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:34:01.660622+0000", 2024-03-11T21:38:45.535 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.0028506220000000001 2024-03-11T21:38:45.535 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:38:45.536 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:38:45.536 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "waiting for peered", 2024-03-11T21:38:45.536 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:34:01.660631+0000", 2024-03-11T21:38:45.536 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 9.4129999999999995e-06 2024-03-11T21:38:45.536 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:38:45.536 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:38:45.536 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "reached_pg", 2024-03-11T21:38:45.536 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:34:02.663808+0000", 2024-03-11T21:38:45.536 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.0031766390000001 2024-03-11T21:38:45.536 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:38:45.536 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:38:45.536 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "started", 2024-03-11T21:38:45.536 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:34:02.663929+0000", 2024-03-11T21:38:45.537 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.000121568 2024-03-11T21:38:45.537 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:38:45.537 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:38:45.537 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "waiting for subops from 1,6", 2024-03-11T21:38:45.537 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:34:02.664013+0000", 2024-03-11T21:38:45.537 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 8.4048999999999994e-05 2024-03-11T21:38:45.537 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:38:45.537 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:38:45.537 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "op_commit", 2024-03-11T21:38:45.537 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:34:02.668047+0000", 2024-03-11T21:38:45.537 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.0040336369999999996 2024-03-11T21:38:45.537 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:38:45.537 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:38:45.537 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "sub_op_commit_rec", 2024-03-11T21:38:45.537 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:34:02.668096+0000", 2024-03-11T21:38:45.538 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 4.8726000000000001e-05 2024-03-11T21:38:45.538 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:38:45.538 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:38:45.538 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "sub_op_commit_rec", 2024-03-11T21:38:45.538 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:34:02.672451+0000", 2024-03-11T21:38:45.538 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.0043555770000000002 2024-03-11T21:38:45.538 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:38:45.538 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:38:45.538 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "commit_sent", 2024-03-11T21:38:45.538 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:34:02.672509+0000", 2024-03-11T21:38:45.538 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 5.7725e-05 2024-03-11T21:38:45.538 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:38:45.538 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:38:45.538 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "done", 2024-03-11T21:38:45.539 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:34:02.672614+0000", 2024-03-11T21:38:45.539 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.00010492000000000001 2024-03-11T21:38:45.539 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-03-11T21:38:45.539 INFO:teuthology.orchestra.run.smithi097.stdout: ] 2024-03-11T21:38:45.539 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-03-11T21:38:45.539 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:38:45.539 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:38:45.539 INFO:teuthology.orchestra.run.smithi097.stdout: "description": "osd_op(client.15150.0:150474 8.15 8:ab344197:::benchmark_data_smithi092_104865_object29973:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e213)", 2024-03-11T21:38:45.539 INFO:teuthology.orchestra.run.smithi097.stdout: "initiated_at": "2024-03-11T21:34:01.664341+0000", 2024-03-11T21:38:45.539 INFO:teuthology.orchestra.run.smithi097.stdout: "age": 283.87082589300002, 2024-03-11T21:38:45.539 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.0085281740000001, 2024-03-11T21:38:45.539 INFO:teuthology.orchestra.run.smithi097.stdout: "type_data": { 2024-03-11T21:38:45.539 INFO:teuthology.orchestra.run.smithi097.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-11T21:38:45.539 INFO:teuthology.orchestra.run.smithi097.stdout: "client_info": { 2024-03-11T21:38:45.539 INFO:teuthology.orchestra.run.smithi097.stdout: "client": "client.15150", 2024-03-11T21:38:45.539 INFO:teuthology.orchestra.run.smithi097.stdout: "client_addr": "v1:172.21.15.92:0/436606717", 2024-03-11T21:38:45.540 INFO:teuthology.orchestra.run.smithi097.stdout: "tid": 150474 2024-03-11T21:38:45.540 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:38:45.540 INFO:teuthology.orchestra.run.smithi097.stdout: "events": [ 2024-03-11T21:38:45.540 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:38:45.540 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "initiated", 2024-03-11T21:38:45.540 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:34:01.664341+0000", 2024-03-11T21:38:45.540 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-03-11T21:38:45.540 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:38:45.540 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:38:45.540 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "header_read", 2024-03-11T21:38:45.540 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:34:01.664341+0000", 2024-03-11T21:38:45.540 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-03-11T21:38:45.540 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:38:45.540 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:38:45.541 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "throttled", 2024-03-11T21:38:45.541 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:34:01.664344+0000", 2024-03-11T21:38:45.541 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 2.937e-06 2024-03-11T21:38:45.541 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:38:45.541 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:38:45.541 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "all_read", 2024-03-11T21:38:45.541 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:34:01.664351+0000", 2024-03-11T21:38:45.541 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 7.097e-06 2024-03-11T21:38:45.541 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:38:45.541 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:38:45.541 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "dispatched", 2024-03-11T21:38:45.541 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:34:01.664365+0000", 2024-03-11T21:38:45.541 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.3933e-05 2024-03-11T21:38:45.541 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:38:45.541 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:38:45.542 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "queued_for_pg", 2024-03-11T21:38:45.542 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:34:01.664374+0000", 2024-03-11T21:38:45.542 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 9.3510000000000008e-06 2024-03-11T21:38:45.542 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:38:45.542 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:38:45.542 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "reached_pg", 2024-03-11T21:38:45.542 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:34:01.664436+0000", 2024-03-11T21:38:45.542 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 6.2026000000000006e-05 2024-03-11T21:38:45.542 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:38:45.542 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:38:45.542 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "waiting for peered", 2024-03-11T21:38:45.542 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:34:01.664447+0000", 2024-03-11T21:38:45.542 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.0781000000000001e-05 2024-03-11T21:38:45.542 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:38:45.542 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:38:45.542 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "reached_pg", 2024-03-11T21:38:45.543 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:34:02.665881+0000", 2024-03-11T21:38:45.543 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.001434283 2024-03-11T21:38:45.543 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:38:45.543 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:38:45.543 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "started", 2024-03-11T21:38:45.543 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:34:02.666004+0000", 2024-03-11T21:38:45.543 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.00012284899999999999 2024-03-11T21:38:45.543 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:38:45.543 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:38:45.543 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "waiting for subops from 1,6", 2024-03-11T21:38:45.543 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:34:02.666147+0000", 2024-03-11T21:38:45.543 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.00014334999999999999 2024-03-11T21:38:45.543 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:38:45.543 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:38:45.544 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "op_commit", 2024-03-11T21:38:45.544 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:34:02.668560+0000", 2024-03-11T21:38:45.544 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.002412544 2024-03-11T21:38:45.544 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:38:45.544 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:38:45.544 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "sub_op_commit_rec", 2024-03-11T21:38:45.544 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:34:02.668600+0000", 2024-03-11T21:38:45.544 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 4.0648000000000001e-05 2024-03-11T21:38:45.544 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:38:45.544 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:38:45.544 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "sub_op_commit_rec", 2024-03-11T21:38:45.544 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:34:02.672663+0000", 2024-03-11T21:38:45.544 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.0040630329999999997 2024-03-11T21:38:45.545 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:38:45.545 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:38:45.545 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "commit_sent", 2024-03-11T21:38:45.545 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:34:02.672749+0000", 2024-03-11T21:38:45.545 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 8.5470999999999995e-05 2024-03-11T21:38:45.545 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:38:45.545 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:38:45.545 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "done", 2024-03-11T21:38:45.545 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:34:02.672869+0000", 2024-03-11T21:38:45.545 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.000119871 2024-03-11T21:38:45.545 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-03-11T21:38:45.545 INFO:teuthology.orchestra.run.smithi097.stdout: ] 2024-03-11T21:38:45.545 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-03-11T21:38:45.545 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:38:45.545 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:38:45.545 INFO:teuthology.orchestra.run.smithi097.stdout: "description": "osd_op(client.15150.0:150552 8.15 8:adbf6385:::benchmark_data_smithi092_104865_object30051:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e213)", 2024-03-11T21:38:45.546 INFO:teuthology.orchestra.run.smithi097.stdout: "initiated_at": "2024-03-11T21:34:01.738436+0000", 2024-03-11T21:38:45.546 INFO:teuthology.orchestra.run.smithi097.stdout: "age": 283.79673058499998, 2024-03-11T21:38:45.546 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.93839203400000004, 2024-03-11T21:38:45.546 INFO:teuthology.orchestra.run.smithi097.stdout: "type_data": { 2024-03-11T21:38:45.546 INFO:teuthology.orchestra.run.smithi097.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-11T21:38:45.546 INFO:teuthology.orchestra.run.smithi097.stdout: "client_info": { 2024-03-11T21:38:45.546 INFO:teuthology.orchestra.run.smithi097.stdout: "client": "client.15150", 2024-03-11T21:38:45.546 INFO:teuthology.orchestra.run.smithi097.stdout: "client_addr": "v1:172.21.15.92:0/436606717", 2024-03-11T21:38:45.546 INFO:teuthology.orchestra.run.smithi097.stdout: "tid": 150552 2024-03-11T21:38:45.546 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:38:45.546 INFO:teuthology.orchestra.run.smithi097.stdout: "events": [ 2024-03-11T21:38:45.546 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:38:45.546 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "initiated", 2024-03-11T21:38:45.546 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:34:01.738436+0000", 2024-03-11T21:38:45.547 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-03-11T21:38:45.547 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:38:45.547 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:38:45.547 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "header_read", 2024-03-11T21:38:45.547 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:34:01.738436+0000", 2024-03-11T21:38:45.547 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-03-11T21:38:45.547 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:38:45.547 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:38:45.547 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "throttled", 2024-03-11T21:38:45.547 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:34:01.738439+0000", 2024-03-11T21:38:45.547 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 2.576e-06 2024-03-11T21:38:45.547 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:38:45.547 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:38:45.547 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "all_read", 2024-03-11T21:38:45.547 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:34:01.738447+0000", 2024-03-11T21:38:45.547 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 8.4330000000000004e-06 2024-03-11T21:38:45.548 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:38:45.548 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:38:45.548 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "dispatched", 2024-03-11T21:38:45.548 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:34:01.738463+0000", 2024-03-11T21:38:45.548 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.6019999999999999e-05 2024-03-11T21:38:45.548 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:38:45.548 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:38:45.548 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "queued_for_pg", 2024-03-11T21:38:45.548 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:34:01.738471+0000", 2024-03-11T21:38:45.548 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 8.388e-06 2024-03-11T21:38:45.548 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:38:45.548 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:38:45.548 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "reached_pg", 2024-03-11T21:38:45.548 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:34:01.738530+0000", 2024-03-11T21:38:45.549 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 5.8242000000000002e-05 2024-03-11T21:38:45.549 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:38:45.549 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:38:45.549 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "waiting for peered", 2024-03-11T21:38:45.549 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:34:01.738536+0000", 2024-03-11T21:38:45.549 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 6.8580000000000002e-06 2024-03-11T21:38:45.549 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:38:45.549 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:38:45.549 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "reached_pg", 2024-03-11T21:38:45.549 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:34:02.668651+0000", 2024-03-11T21:38:45.549 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.93011446600000003 2024-03-11T21:38:45.549 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:38:45.549 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:38:45.549 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "started", 2024-03-11T21:38:45.549 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:34:02.668773+0000", 2024-03-11T21:38:45.549 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.000122002 2024-03-11T21:38:45.550 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:38:45.550 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:38:45.550 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "waiting for subops from 1,6", 2024-03-11T21:38:45.550 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:34:02.668896+0000", 2024-03-11T21:38:45.550 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.000122652 2024-03-11T21:38:45.550 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:38:45.550 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:38:45.550 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "sub_op_commit_rec", 2024-03-11T21:38:45.550 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:34:02.672932+0000", 2024-03-11T21:38:45.550 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.0040368950000000004 2024-03-11T21:38:45.550 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:38:45.550 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:38:45.550 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "op_commit", 2024-03-11T21:38:45.550 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:34:02.673021+0000", 2024-03-11T21:38:45.551 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 8.8218000000000001e-05 2024-03-11T21:38:45.551 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:38:45.551 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:38:45.551 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "sub_op_commit_rec", 2024-03-11T21:38:45.551 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:34:02.676507+0000", 2024-03-11T21:38:45.551 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.0034864969999999999 2024-03-11T21:38:45.551 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:38:45.551 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:38:45.551 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "commit_sent", 2024-03-11T21:38:45.551 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:34:02.676567+0000", 2024-03-11T21:38:45.551 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 5.9914000000000003e-05 2024-03-11T21:38:45.551 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:38:45.551 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:38:45.551 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "done", 2024-03-11T21:38:45.551 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:34:02.676828+0000", 2024-03-11T21:38:45.552 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.00026087300000000001 2024-03-11T21:38:45.552 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-03-11T21:38:45.552 INFO:teuthology.orchestra.run.smithi097.stdout: ] 2024-03-11T21:38:45.552 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-03-11T21:38:45.552 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:38:45.552 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:38:45.552 INFO:teuthology.orchestra.run.smithi097.stdout: "description": "osd_op(client.15150.0:150558 8.15 8:aa3c4cf2:::benchmark_data_smithi092_104865_object30057:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e213)", 2024-03-11T21:38:45.552 INFO:teuthology.orchestra.run.smithi097.stdout: "initiated_at": "2024-03-11T21:34:01.740197+0000", 2024-03-11T21:38:45.552 INFO:teuthology.orchestra.run.smithi097.stdout: "age": 283.79496916800002, 2024-03-11T21:38:45.552 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.93882425199999997, 2024-03-11T21:38:45.552 INFO:teuthology.orchestra.run.smithi097.stdout: "type_data": { 2024-03-11T21:38:45.552 INFO:teuthology.orchestra.run.smithi097.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-11T21:38:45.552 INFO:teuthology.orchestra.run.smithi097.stdout: "client_info": { 2024-03-11T21:38:45.552 INFO:teuthology.orchestra.run.smithi097.stdout: "client": "client.15150", 2024-03-11T21:38:45.552 INFO:teuthology.orchestra.run.smithi097.stdout: "client_addr": "v1:172.21.15.92:0/436606717", 2024-03-11T21:38:45.553 INFO:teuthology.orchestra.run.smithi097.stdout: "tid": 150558 2024-03-11T21:38:45.553 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:38:45.553 INFO:teuthology.orchestra.run.smithi097.stdout: "events": [ 2024-03-11T21:38:45.553 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:38:45.553 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "initiated", 2024-03-11T21:38:45.553 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:34:01.740197+0000", 2024-03-11T21:38:45.553 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-03-11T21:38:45.553 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:38:45.553 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:38:45.553 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "header_read", 2024-03-11T21:38:45.553 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:34:01.740197+0000", 2024-03-11T21:38:45.553 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-03-11T21:38:45.553 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:38:45.553 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:38:45.554 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "throttled", 2024-03-11T21:38:45.554 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:34:01.740199+0000", 2024-03-11T21:38:45.554 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.7689999999999999e-06 2024-03-11T21:38:45.554 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:38:45.554 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:38:45.554 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "all_read", 2024-03-11T21:38:45.554 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:34:01.740205+0000", 2024-03-11T21:38:45.554 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 5.942e-06 2024-03-11T21:38:45.554 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:38:45.554 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:38:45.554 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "dispatched", 2024-03-11T21:38:45.554 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:34:01.740221+0000", 2024-03-11T21:38:45.554 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.6371e-05 2024-03-11T21:38:45.554 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:38:45.554 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:38:45.555 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "queued_for_pg", 2024-03-11T21:38:45.555 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:34:01.740230+0000", 2024-03-11T21:38:45.555 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 8.5290000000000008e-06 2024-03-11T21:38:45.555 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:38:45.555 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:38:45.555 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "reached_pg", 2024-03-11T21:38:45.555 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:34:01.740295+0000", 2024-03-11T21:38:45.555 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 6.5190000000000004e-05 2024-03-11T21:38:45.555 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:38:45.555 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:38:45.555 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "waiting for peered", 2024-03-11T21:38:45.555 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:34:01.740305+0000", 2024-03-11T21:38:45.555 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 9.4839999999999993e-06 2024-03-11T21:38:45.555 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:38:45.555 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:38:45.556 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "reached_pg", 2024-03-11T21:38:45.556 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:34:02.670688+0000", 2024-03-11T21:38:45.556 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.93038337299999996 2024-03-11T21:38:45.556 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:38:45.556 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:38:45.556 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "started", 2024-03-11T21:38:45.556 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:34:02.670790+0000", 2024-03-11T21:38:45.556 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.000102079 2024-03-11T21:38:45.556 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:38:45.556 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:38:45.556 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "waiting for subops from 1,6", 2024-03-11T21:38:45.556 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:34:02.670901+0000", 2024-03-11T21:38:45.556 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.0001106 2024-03-11T21:38:45.556 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:38:45.556 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:38:45.557 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "sub_op_commit_rec", 2024-03-11T21:38:45.557 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:34:02.673004+0000", 2024-03-11T21:38:45.557 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.0021035390000000002 2024-03-11T21:38:45.557 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:38:45.557 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:38:45.557 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "op_commit", 2024-03-11T21:38:45.557 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:34:02.676847+0000", 2024-03-11T21:38:45.557 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.0038423590000000001 2024-03-11T21:38:45.557 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:38:45.557 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:38:45.557 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "sub_op_commit_rec", 2024-03-11T21:38:45.557 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:34:02.678835+0000", 2024-03-11T21:38:45.557 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.0019883129999999998 2024-03-11T21:38:45.557 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:38:45.558 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:38:45.558 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "commit_sent", 2024-03-11T21:38:45.558 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:34:02.678918+0000", 2024-03-11T21:38:45.558 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 8.3443999999999996e-05 2024-03-11T21:38:45.558 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:38:45.558 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:38:45.558 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "done", 2024-03-11T21:38:45.558 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:34:02.679022+0000", 2024-03-11T21:38:45.558 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.00010326 2024-03-11T21:38:45.558 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-03-11T21:38:45.558 INFO:teuthology.orchestra.run.smithi097.stdout: ] 2024-03-11T21:38:45.558 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-03-11T21:38:45.558 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:38:45.558 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:38:45.558 INFO:teuthology.orchestra.run.smithi097.stdout: "description": "osd_op(client.15150.0:150567 8.15 8:acd61c96:::benchmark_data_smithi092_104865_object30066:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e213)", 2024-03-11T21:38:45.559 INFO:teuthology.orchestra.run.smithi097.stdout: "initiated_at": "2024-03-11T21:34:01.751935+0000", 2024-03-11T21:38:45.559 INFO:teuthology.orchestra.run.smithi097.stdout: "age": 283.78323144199999, 2024-03-11T21:38:45.559 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.94653565699999997, 2024-03-11T21:38:45.559 INFO:teuthology.orchestra.run.smithi097.stdout: "type_data": { 2024-03-11T21:38:45.559 INFO:teuthology.orchestra.run.smithi097.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-11T21:38:45.559 INFO:teuthology.orchestra.run.smithi097.stdout: "client_info": { 2024-03-11T21:38:45.559 INFO:teuthology.orchestra.run.smithi097.stdout: "client": "client.15150", 2024-03-11T21:38:45.559 INFO:teuthology.orchestra.run.smithi097.stdout: "client_addr": "v1:172.21.15.92:0/436606717", 2024-03-11T21:38:45.559 INFO:teuthology.orchestra.run.smithi097.stdout: "tid": 150567 2024-03-11T21:38:45.559 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:38:45.559 INFO:teuthology.orchestra.run.smithi097.stdout: "events": [ 2024-03-11T21:38:45.559 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:38:45.559 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "initiated", 2024-03-11T21:38:45.559 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:34:01.751935+0000", 2024-03-11T21:38:45.559 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-03-11T21:38:45.560 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:38:45.560 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:38:45.560 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "header_read", 2024-03-11T21:38:45.560 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:34:01.751935+0000", 2024-03-11T21:38:45.560 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-03-11T21:38:45.560 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:38:45.560 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:38:45.560 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "throttled", 2024-03-11T21:38:45.560 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:34:01.751936+0000", 2024-03-11T21:38:45.560 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 4.3500000000000002e-07 2024-03-11T21:38:45.560 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:38:45.560 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:38:45.560 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "all_read", 2024-03-11T21:38:45.560 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:34:01.751938+0000", 2024-03-11T21:38:45.560 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 2.069e-06 2024-03-11T21:38:45.561 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:38:45.561 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:38:45.561 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "dispatched", 2024-03-11T21:38:45.561 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:34:01.751941+0000", 2024-03-11T21:38:45.561 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 3.5590000000000001e-06 2024-03-11T21:38:45.561 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:38:45.561 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:38:45.561 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "queued_for_pg", 2024-03-11T21:38:45.561 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:34:01.751944+0000", 2024-03-11T21:38:45.561 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 3.162e-06 2024-03-11T21:38:45.562 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:38:45.563 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:38:45.563 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "reached_pg", 2024-03-11T21:38:45.563 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:34:01.752002+0000", 2024-03-11T21:38:45.563 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 5.7822999999999998e-05 2024-03-11T21:38:45.563 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:38:45.563 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:38:45.563 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "waiting for peered", 2024-03-11T21:38:45.563 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:34:01.752009+0000", 2024-03-11T21:38:45.563 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 6.7560000000000002e-06 2024-03-11T21:38:45.563 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:38:45.563 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:38:45.563 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "reached_pg", 2024-03-11T21:38:45.563 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:34:02.674820+0000", 2024-03-11T21:38:45.563 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.92281079899999996 2024-03-11T21:38:45.563 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:38:45.564 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:38:45.564 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "started", 2024-03-11T21:38:45.564 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:34:02.674937+0000", 2024-03-11T21:38:45.564 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.000117217 2024-03-11T21:38:45.564 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:38:45.564 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:38:45.564 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "waiting for subops from 1,6", 2024-03-11T21:38:45.564 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:34:02.675002+0000", 2024-03-11T21:38:45.564 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 6.5345999999999994e-05 2024-03-11T21:38:45.564 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:38:45.564 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:38:45.564 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "op_commit", 2024-03-11T21:38:45.564 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:34:02.678751+0000", 2024-03-11T21:38:45.564 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.0037492250000000001 2024-03-11T21:38:45.564 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:38:45.565 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:38:45.565 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "sub_op_commit_rec", 2024-03-11T21:38:45.565 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:34:02.678784+0000", 2024-03-11T21:38:45.565 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 3.2539000000000001e-05 2024-03-11T21:38:45.565 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:38:45.565 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:38:45.565 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "sub_op_commit_rec", 2024-03-11T21:38:45.565 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:34:02.698408+0000", 2024-03-11T21:38:45.565 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.019623491999999999 2024-03-11T21:38:45.565 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:38:45.565 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:38:45.565 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "commit_sent", 2024-03-11T21:38:45.565 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:34:02.698444+0000", 2024-03-11T21:38:45.565 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 3.6634999999999997e-05 2024-03-11T21:38:45.565 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:38:45.566 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:38:45.566 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "done", 2024-03-11T21:38:45.566 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:34:02.698471+0000", 2024-03-11T21:38:45.566 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 2.6599999999999999e-05 2024-03-11T21:38:45.566 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-03-11T21:38:45.566 INFO:teuthology.orchestra.run.smithi097.stdout: ] 2024-03-11T21:38:45.566 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-03-11T21:38:45.566 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:38:45.566 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:38:45.566 INFO:teuthology.orchestra.run.smithi097.stdout: "description": "osd_op(client.15150.0:203653 8.17 8:ea1eb592:::benchmark_data_smithi092_104865_object83152:head [delete] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e233)", 2024-03-11T21:38:45.566 INFO:teuthology.orchestra.run.smithi097.stdout: "initiated_at": "2024-03-11T21:34:50.290471+0000", 2024-03-11T21:38:45.566 INFO:teuthology.orchestra.run.smithi097.stdout: "age": 235.24469519499999, 2024-03-11T21:38:45.566 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.0299367500000001, 2024-03-11T21:38:45.566 INFO:teuthology.orchestra.run.smithi097.stdout: "type_data": { 2024-03-11T21:38:45.567 INFO:teuthology.orchestra.run.smithi097.stdout: "flag_point": "reached pg", 2024-03-11T21:38:45.567 INFO:teuthology.orchestra.run.smithi097.stdout: "client_info": { 2024-03-11T21:38:45.567 INFO:teuthology.orchestra.run.smithi097.stdout: "client": "client.15150", 2024-03-11T21:38:45.567 INFO:teuthology.orchestra.run.smithi097.stdout: "client_addr": "v1:172.21.15.92:0/436606717", 2024-03-11T21:38:45.567 INFO:teuthology.orchestra.run.smithi097.stdout: "tid": 203653 2024-03-11T21:38:45.567 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:38:45.567 INFO:teuthology.orchestra.run.smithi097.stdout: "events": [ 2024-03-11T21:38:45.567 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:38:45.567 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "initiated", 2024-03-11T21:38:45.567 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:34:50.290471+0000", 2024-03-11T21:38:45.567 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-03-11T21:38:45.567 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:38:45.567 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:38:45.567 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "header_read", 2024-03-11T21:38:45.567 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:34:50.290471+0000", 2024-03-11T21:38:45.567 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-03-11T21:38:45.568 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:38:45.568 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:38:45.568 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "throttled", 2024-03-11T21:38:45.568 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:34:50.290473+0000", 2024-03-11T21:38:45.568 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 2.041e-06 2024-03-11T21:38:45.568 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:38:45.568 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:38:45.568 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "all_read", 2024-03-11T21:38:45.568 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:34:50.290479+0000", 2024-03-11T21:38:45.568 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 6.1099999999999999e-06 2024-03-11T21:38:45.568 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:38:45.568 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:38:45.568 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "dispatched", 2024-03-11T21:38:45.568 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:34:50.290496+0000", 2024-03-11T21:38:45.568 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.6069000000000001e-05 2024-03-11T21:38:45.569 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:38:45.569 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:38:45.569 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "queued_for_pg", 2024-03-11T21:38:45.569 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:34:50.290504+0000", 2024-03-11T21:38:45.569 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 8.0090000000000007e-06 2024-03-11T21:38:45.569 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:38:45.569 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:38:45.569 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "reached_pg", 2024-03-11T21:38:45.569 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:34:50.290577+0000", 2024-03-11T21:38:45.569 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 7.2954999999999995e-05 2024-03-11T21:38:45.569 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:38:45.569 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:38:45.569 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "op must wait for map", 2024-03-11T21:38:45.569 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:34:50.290584+0000", 2024-03-11T21:38:45.569 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 7.1420000000000004e-06 2024-03-11T21:38:45.569 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:38:45.570 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:38:45.570 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "reached_pg", 2024-03-11T21:38:45.570 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:34:50.297124+0000", 2024-03-11T21:38:45.570 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.0065406450000000003 2024-03-11T21:38:45.570 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:38:45.570 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:38:45.570 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "waiting for peered", 2024-03-11T21:38:45.570 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:34:50.297133+0000", 2024-03-11T21:38:45.570 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 8.9080000000000001e-06 2024-03-11T21:38:45.570 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:38:45.570 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:38:45.570 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "reached_pg", 2024-03-11T21:38:45.570 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:34:51.320325+0000", 2024-03-11T21:38:45.570 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.023191567 2024-03-11T21:38:45.571 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:38:45.571 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:38:45.571 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "done", 2024-03-11T21:38:45.571 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:34:51.320408+0000", 2024-03-11T21:38:45.571 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 8.3304000000000001e-05 2024-03-11T21:38:45.571 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-03-11T21:38:45.571 INFO:teuthology.orchestra.run.smithi097.stdout: ] 2024-03-11T21:38:45.571 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-03-11T21:38:45.571 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:38:45.571 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:38:45.571 INFO:teuthology.orchestra.run.smithi097.stdout: "description": "osd_op(client.15150.0:203682 8.17 8:e835ebdb:::benchmark_data_smithi092_104865_object83181:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e233)", 2024-03-11T21:38:45.571 INFO:teuthology.orchestra.run.smithi097.stdout: "initiated_at": "2024-03-11T21:34:50.309525+0000", 2024-03-11T21:38:45.571 INFO:teuthology.orchestra.run.smithi097.stdout: "age": 235.22564199499999, 2024-03-11T21:38:45.571 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.0163843560000001, 2024-03-11T21:38:45.571 INFO:teuthology.orchestra.run.smithi097.stdout: "type_data": { 2024-03-11T21:38:45.571 INFO:teuthology.orchestra.run.smithi097.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-11T21:38:45.572 INFO:teuthology.orchestra.run.smithi097.stdout: "client_info": { 2024-03-11T21:38:45.572 INFO:teuthology.orchestra.run.smithi097.stdout: "client": "client.15150", 2024-03-11T21:38:45.572 INFO:teuthology.orchestra.run.smithi097.stdout: "client_addr": "v1:172.21.15.92:0/436606717", 2024-03-11T21:38:45.572 INFO:teuthology.orchestra.run.smithi097.stdout: "tid": 203682 2024-03-11T21:38:45.572 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:38:45.572 INFO:teuthology.orchestra.run.smithi097.stdout: "events": [ 2024-03-11T21:38:45.572 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:38:45.572 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "initiated", 2024-03-11T21:38:45.572 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:34:50.309525+0000", 2024-03-11T21:38:45.572 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-03-11T21:38:45.572 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:38:45.572 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:38:45.572 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "header_read", 2024-03-11T21:38:45.572 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:34:50.309525+0000", 2024-03-11T21:38:45.573 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-03-11T21:38:45.573 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:38:45.573 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:38:45.573 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "throttled", 2024-03-11T21:38:45.573 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:34:50.309526+0000", 2024-03-11T21:38:45.573 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.762e-06 2024-03-11T21:38:45.573 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:38:45.573 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:38:45.573 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "all_read", 2024-03-11T21:38:45.573 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:34:50.309531+0000", 2024-03-11T21:38:45.573 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 4.8439999999999997e-06 2024-03-11T21:38:45.573 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:38:45.573 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:38:45.573 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "dispatched", 2024-03-11T21:38:45.573 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:34:50.309709+0000", 2024-03-11T21:38:45.573 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.00017786200000000001 2024-03-11T21:38:45.574 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:38:45.574 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:38:45.574 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "queued_for_pg", 2024-03-11T21:38:45.574 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:34:50.309718+0000", 2024-03-11T21:38:45.574 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 8.6060000000000003e-06 2024-03-11T21:38:45.574 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:38:45.574 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:38:45.574 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "reached_pg", 2024-03-11T21:38:45.574 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:34:50.309805+0000", 2024-03-11T21:38:45.574 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 8.7861999999999993e-05 2024-03-11T21:38:45.574 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:38:45.574 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:38:45.574 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "waiting for peered", 2024-03-11T21:38:45.574 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:34:50.309814+0000", 2024-03-11T21:38:45.575 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 8.9309999999999997e-06 2024-03-11T21:38:45.575 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:38:45.575 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:38:45.575 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "reached_pg", 2024-03-11T21:38:45.575 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:34:51.320819+0000", 2024-03-11T21:38:45.575 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.011005063 2024-03-11T21:38:45.575 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:38:45.575 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:38:45.575 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "started", 2024-03-11T21:38:45.575 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:34:51.320926+0000", 2024-03-11T21:38:45.575 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.000106252 2024-03-11T21:38:45.575 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:38:45.575 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:38:45.575 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "waiting for subops from 0,6", 2024-03-11T21:38:45.575 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:34:51.321218+0000", 2024-03-11T21:38:45.576 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.00029272 2024-03-11T21:38:45.576 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:38:45.576 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:38:45.576 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "sub_op_commit_rec", 2024-03-11T21:38:45.576 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:34:51.325634+0000", 2024-03-11T21:38:45.576 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.0044151559999999999 2024-03-11T21:38:45.576 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:38:45.576 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:38:45.576 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "op_commit", 2024-03-11T21:38:45.576 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:34:51.325752+0000", 2024-03-11T21:38:45.576 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.000118527 2024-03-11T21:38:45.576 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:38:45.576 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:38:45.576 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "sub_op_commit_rec", 2024-03-11T21:38:45.577 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:34:51.325811+0000", 2024-03-11T21:38:45.577 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 5.9212999999999997e-05 2024-03-11T21:38:45.577 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:38:45.577 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:38:45.577 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "commit_sent", 2024-03-11T21:38:45.577 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:34:51.325862+0000", 2024-03-11T21:38:45.577 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 5.0886e-05 2024-03-11T21:38:45.577 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:38:45.577 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:38:45.577 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "done", 2024-03-11T21:38:45.577 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:34:51.325909+0000", 2024-03-11T21:38:45.577 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 4.6672000000000003e-05 2024-03-11T21:38:45.577 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-03-11T21:38:45.578 INFO:teuthology.orchestra.run.smithi097.stdout: ] 2024-03-11T21:38:45.578 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-03-11T21:38:45.578 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:38:45.578 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:38:45.578 INFO:teuthology.orchestra.run.smithi097.stdout: "description": "osd_op(client.15150.0:203696 8.17 8:eddc2db3:::benchmark_data_smithi092_104865_object83195:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e233)", 2024-03-11T21:38:45.578 INFO:teuthology.orchestra.run.smithi097.stdout: "initiated_at": "2024-03-11T21:34:50.321937+0000", 2024-03-11T21:38:45.578 INFO:teuthology.orchestra.run.smithi097.stdout: "age": 235.213229893, 2024-03-11T21:38:45.578 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.0074158280000001, 2024-03-11T21:38:45.578 INFO:teuthology.orchestra.run.smithi097.stdout: "type_data": { 2024-03-11T21:38:45.578 INFO:teuthology.orchestra.run.smithi097.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-11T21:38:45.578 INFO:teuthology.orchestra.run.smithi097.stdout: "client_info": { 2024-03-11T21:38:45.578 INFO:teuthology.orchestra.run.smithi097.stdout: "client": "client.15150", 2024-03-11T21:38:45.578 INFO:teuthology.orchestra.run.smithi097.stdout: "client_addr": "v1:172.21.15.92:0/436606717", 2024-03-11T21:38:45.578 INFO:teuthology.orchestra.run.smithi097.stdout: "tid": 203696 2024-03-11T21:38:45.579 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:38:45.579 INFO:teuthology.orchestra.run.smithi097.stdout: "events": [ 2024-03-11T21:38:45.579 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:38:45.579 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "initiated", 2024-03-11T21:38:45.579 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:34:50.321937+0000", 2024-03-11T21:38:45.579 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-03-11T21:38:45.579 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:38:45.579 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:38:45.579 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "header_read", 2024-03-11T21:38:45.579 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:34:50.321937+0000", 2024-03-11T21:38:45.579 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-03-11T21:38:45.579 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:38:45.579 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:38:45.579 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "throttled", 2024-03-11T21:38:45.580 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:34:50.321939+0000", 2024-03-11T21:38:45.580 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 2.1459999999999999e-06 2024-03-11T21:38:45.580 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:38:45.580 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:38:45.580 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "all_read", 2024-03-11T21:38:45.580 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:34:50.321945+0000", 2024-03-11T21:38:45.580 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 6.3319999999999997e-06 2024-03-11T21:38:45.580 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:38:45.580 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:38:45.580 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "dispatched", 2024-03-11T21:38:45.580 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:34:50.321957+0000", 2024-03-11T21:38:45.580 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.2191000000000001e-05 2024-03-11T21:38:45.580 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:38:45.580 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:38:45.580 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "queued_for_pg", 2024-03-11T21:38:45.581 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:34:50.321965+0000", 2024-03-11T21:38:45.581 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 7.7789999999999996e-06 2024-03-11T21:38:45.581 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:38:45.581 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:38:45.581 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "reached_pg", 2024-03-11T21:38:45.581 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:34:50.322124+0000", 2024-03-11T21:38:45.581 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.00015893999999999999 2024-03-11T21:38:45.581 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:38:45.581 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:38:45.581 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "waiting for peered", 2024-03-11T21:38:45.581 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:34:50.322132+0000", 2024-03-11T21:38:45.581 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 8.2379999999999997e-06 2024-03-11T21:38:45.581 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:38:45.581 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:38:45.581 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "reached_pg", 2024-03-11T21:38:45.582 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:34:51.322497+0000", 2024-03-11T21:38:45.582 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.0003644279999999 2024-03-11T21:38:45.582 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:38:45.582 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:38:45.582 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "started", 2024-03-11T21:38:45.582 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:34:51.322604+0000", 2024-03-11T21:38:45.582 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.000107785 2024-03-11T21:38:45.582 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:38:45.582 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:38:45.582 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "waiting for subops from 0,6", 2024-03-11T21:38:45.582 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:34:51.322837+0000", 2024-03-11T21:38:45.582 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.000232592 2024-03-11T21:38:45.582 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:38:45.582 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:38:45.583 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "op_commit", 2024-03-11T21:38:45.583 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:34:51.327557+0000", 2024-03-11T21:38:45.583 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.0047198819999999999 2024-03-11T21:38:45.583 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:38:45.583 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:38:45.583 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "sub_op_commit_rec", 2024-03-11T21:38:45.583 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:34:51.329217+0000", 2024-03-11T21:38:45.583 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.0016600110000000001 2024-03-11T21:38:45.583 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:38:45.583 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:38:45.583 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "sub_op_commit_rec", 2024-03-11T21:38:45.583 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:34:51.329253+0000", 2024-03-11T21:38:45.583 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 3.5824000000000001e-05 2024-03-11T21:38:45.583 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:38:45.583 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:38:45.584 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "commit_sent", 2024-03-11T21:38:45.584 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:34:51.329306+0000", 2024-03-11T21:38:45.584 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 5.3241000000000001e-05 2024-03-11T21:38:45.584 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:38:45.584 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:38:45.584 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "done", 2024-03-11T21:38:45.584 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:34:51.329352+0000", 2024-03-11T21:38:45.584 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 4.6439e-05 2024-03-11T21:38:45.584 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-03-11T21:38:45.584 INFO:teuthology.orchestra.run.smithi097.stdout: ] 2024-03-11T21:38:45.584 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-03-11T21:38:45.584 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:38:45.584 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:38:45.584 INFO:teuthology.orchestra.run.smithi097.stdout: "description": "osd_op(client.15150.0:203717 8.17 8:e9113ebd:::benchmark_data_smithi092_104865_object83216:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e233)", 2024-03-11T21:38:45.584 INFO:teuthology.orchestra.run.smithi097.stdout: "initiated_at": "2024-03-11T21:34:50.337126+0000", 2024-03-11T21:38:45.585 INFO:teuthology.orchestra.run.smithi097.stdout: "age": 235.19804013699999, 2024-03-11T21:38:45.585 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.99565916300000001, 2024-03-11T21:38:45.585 INFO:teuthology.orchestra.run.smithi097.stdout: "type_data": { 2024-03-11T21:38:45.585 INFO:teuthology.orchestra.run.smithi097.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-11T21:38:45.585 INFO:teuthology.orchestra.run.smithi097.stdout: "client_info": { 2024-03-11T21:38:45.585 INFO:teuthology.orchestra.run.smithi097.stdout: "client": "client.15150", 2024-03-11T21:38:45.585 INFO:teuthology.orchestra.run.smithi097.stdout: "client_addr": "v1:172.21.15.92:0/436606717", 2024-03-11T21:38:45.585 INFO:teuthology.orchestra.run.smithi097.stdout: "tid": 203717 2024-03-11T21:38:45.585 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:38:45.585 INFO:teuthology.orchestra.run.smithi097.stdout: "events": [ 2024-03-11T21:38:45.585 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:38:45.585 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "initiated", 2024-03-11T21:38:45.585 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:34:50.337126+0000", 2024-03-11T21:38:45.585 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-03-11T21:38:45.585 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:38:45.586 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:38:45.586 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "header_read", 2024-03-11T21:38:45.586 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:34:50.337126+0000", 2024-03-11T21:38:45.586 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-03-11T21:38:45.586 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:38:45.586 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:38:45.586 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "throttled", 2024-03-11T21:38:45.586 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:34:50.337128+0000", 2024-03-11T21:38:45.586 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.871e-06 2024-03-11T21:38:45.586 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:38:45.586 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:38:45.586 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "all_read", 2024-03-11T21:38:45.586 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:34:50.337134+0000", 2024-03-11T21:38:45.586 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 5.5600000000000001e-06 2024-03-11T21:38:45.587 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:38:45.587 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:38:45.587 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "dispatched", 2024-03-11T21:38:45.587 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:34:50.337145+0000", 2024-03-11T21:38:45.587 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.1112e-05 2024-03-11T21:38:45.587 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:38:45.587 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:38:45.587 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "queued_for_pg", 2024-03-11T21:38:45.587 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:34:50.337153+0000", 2024-03-11T21:38:45.587 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 7.5889999999999997e-06 2024-03-11T21:38:45.587 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:38:45.587 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:38:45.587 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "reached_pg", 2024-03-11T21:38:45.587 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:34:50.337206+0000", 2024-03-11T21:38:45.587 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 5.3706999999999999e-05 2024-03-11T21:38:45.587 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:38:45.588 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:38:45.588 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "waiting for peered", 2024-03-11T21:38:45.588 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:34:50.337214+0000", 2024-03-11T21:38:45.588 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 7.294e-06 2024-03-11T21:38:45.588 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:38:45.588 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:38:45.588 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "reached_pg", 2024-03-11T21:38:45.588 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:34:51.324056+0000", 2024-03-11T21:38:45.588 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.98684222300000002 2024-03-11T21:38:45.588 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:38:45.588 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:38:45.588 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "started", 2024-03-11T21:38:45.588 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:34:51.324185+0000", 2024-03-11T21:38:45.588 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.00012944200000000001 2024-03-11T21:38:45.588 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:38:45.589 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:38:45.589 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "waiting for subops from 0,6", 2024-03-11T21:38:45.589 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:34:51.324400+0000", 2024-03-11T21:38:45.589 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.00021476899999999999 2024-03-11T21:38:45.589 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:38:45.589 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:38:45.589 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "op_commit", 2024-03-11T21:38:45.589 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:34:51.329178+0000", 2024-03-11T21:38:45.589 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.0047776019999999997 2024-03-11T21:38:45.589 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:38:45.589 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:38:45.589 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "sub_op_commit_rec", 2024-03-11T21:38:45.589 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:34:51.329400+0000", 2024-03-11T21:38:45.589 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.000222616 2024-03-11T21:38:45.589 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:38:45.590 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:38:45.590 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "sub_op_commit_rec", 2024-03-11T21:38:45.590 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:34:51.332703+0000", 2024-03-11T21:38:45.590 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.0033027870000000002 2024-03-11T21:38:45.590 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:38:45.590 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:38:45.590 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "commit_sent", 2024-03-11T21:38:45.590 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:34:51.332745+0000", 2024-03-11T21:38:45.590 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 4.2169000000000002e-05 2024-03-11T21:38:45.590 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:38:45.590 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:38:45.590 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "done", 2024-03-11T21:38:45.590 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:34:51.332786+0000", 2024-03-11T21:38:45.590 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 4.0422e-05 2024-03-11T21:38:45.591 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-03-11T21:38:45.591 INFO:teuthology.orchestra.run.smithi097.stdout: ] 2024-03-11T21:38:45.591 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-03-11T21:38:45.591 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:38:45.591 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:38:45.591 INFO:teuthology.orchestra.run.smithi097.stdout: "description": "osd_op(client.15150.0:203721 8.17 8:ea4988fc:::benchmark_data_smithi092_104865_object83220:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e233)", 2024-03-11T21:38:45.591 INFO:teuthology.orchestra.run.smithi097.stdout: "initiated_at": "2024-03-11T21:34:50.337910+0000", 2024-03-11T21:38:45.591 INFO:teuthology.orchestra.run.smithi097.stdout: "age": 235.197256756, 2024-03-11T21:38:45.591 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.99500893499999998, 2024-03-11T21:38:45.591 INFO:teuthology.orchestra.run.smithi097.stdout: "type_data": { 2024-03-11T21:38:45.591 INFO:teuthology.orchestra.run.smithi097.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-11T21:38:45.591 INFO:teuthology.orchestra.run.smithi097.stdout: "client_info": { 2024-03-11T21:38:45.591 INFO:teuthology.orchestra.run.smithi097.stdout: "client": "client.15150", 2024-03-11T21:38:45.592 INFO:teuthology.orchestra.run.smithi097.stdout: "client_addr": "v1:172.21.15.92:0/436606717", 2024-03-11T21:38:45.592 INFO:teuthology.orchestra.run.smithi097.stdout: "tid": 203721 2024-03-11T21:38:45.592 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:38:45.592 INFO:teuthology.orchestra.run.smithi097.stdout: "events": [ 2024-03-11T21:38:45.592 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:38:45.592 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "initiated", 2024-03-11T21:38:45.592 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:34:50.337910+0000", 2024-03-11T21:38:45.592 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-03-11T21:38:45.592 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:38:45.592 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:38:45.592 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "header_read", 2024-03-11T21:38:45.592 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:34:50.337910+0000", 2024-03-11T21:38:45.592 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-03-11T21:38:45.592 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:38:45.593 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:38:45.593 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "throttled", 2024-03-11T21:38:45.593 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:34:50.337911+0000", 2024-03-11T21:38:45.593 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.19e-06 2024-03-11T21:38:45.593 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:38:45.593 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:38:45.593 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "all_read", 2024-03-11T21:38:45.593 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:34:50.337918+0000", 2024-03-11T21:38:45.593 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 7.3590000000000002e-06 2024-03-11T21:38:45.593 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:38:45.593 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:38:45.593 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "dispatched", 2024-03-11T21:38:45.593 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:34:50.337927+0000", 2024-03-11T21:38:45.593 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 9.1719999999999996e-06 2024-03-11T21:38:45.593 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:38:45.594 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:38:45.594 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "queued_for_pg", 2024-03-11T21:38:45.594 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:34:50.337934+0000", 2024-03-11T21:38:45.594 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 6.9160000000000004e-06 2024-03-11T21:38:45.594 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:38:45.594 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:38:45.594 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "reached_pg", 2024-03-11T21:38:45.594 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:34:50.337988+0000", 2024-03-11T21:38:45.594 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 5.3919999999999999e-05 2024-03-11T21:38:45.594 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:38:45.594 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:38:45.594 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "waiting for peered", 2024-03-11T21:38:45.594 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:34:50.337996+0000", 2024-03-11T21:38:45.594 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 8.14e-06 2024-03-11T21:38:45.595 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:38:45.595 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:38:45.595 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "reached_pg", 2024-03-11T21:38:45.595 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:34:51.325950+0000", 2024-03-11T21:38:45.595 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.98795397299999999 2024-03-11T21:38:45.595 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:38:45.595 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:38:45.595 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "started", 2024-03-11T21:38:45.595 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:34:51.326051+0000", 2024-03-11T21:38:45.595 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.000100352 2024-03-11T21:38:45.595 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:38:45.595 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:38:45.595 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "waiting for subops from 0,6", 2024-03-11T21:38:45.595 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:34:51.326245+0000", 2024-03-11T21:38:45.596 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.000194604 2024-03-11T21:38:45.596 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:38:45.596 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:38:45.596 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "sub_op_commit_rec", 2024-03-11T21:38:45.596 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:34:51.329441+0000", 2024-03-11T21:38:45.596 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.003195472 2024-03-11T21:38:45.596 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:38:45.596 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:38:45.596 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "op_commit", 2024-03-11T21:38:45.596 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:34:51.330998+0000", 2024-03-11T21:38:45.596 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.0015573119999999999 2024-03-11T21:38:45.596 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:38:45.596 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:38:45.596 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "sub_op_commit_rec", 2024-03-11T21:38:45.596 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:34:51.332838+0000", 2024-03-11T21:38:45.597 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.001840221 2024-03-11T21:38:45.597 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:38:45.597 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:38:45.597 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "commit_sent", 2024-03-11T21:38:45.597 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:34:51.332879+0000", 2024-03-11T21:38:45.597 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 4.0445999999999999e-05 2024-03-11T21:38:45.597 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:38:45.597 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:38:45.597 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "done", 2024-03-11T21:38:45.597 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:34:51.332919+0000", 2024-03-11T21:38:45.597 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 3.9857999999999997e-05 2024-03-11T21:38:45.597 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-03-11T21:38:45.597 INFO:teuthology.orchestra.run.smithi097.stdout: ] 2024-03-11T21:38:45.597 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-03-11T21:38:45.597 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:38:45.597 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:38:45.598 INFO:teuthology.orchestra.run.smithi097.stdout: "description": "osd_op(client.15150.0:203735 8.17 8:ea4806f2:::benchmark_data_smithi092_104865_object83234:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e233)", 2024-03-11T21:38:45.598 INFO:teuthology.orchestra.run.smithi097.stdout: "initiated_at": "2024-03-11T21:34:50.349993+0000", 2024-03-11T21:38:45.598 INFO:teuthology.orchestra.run.smithi097.stdout: "age": 235.18517376200001, 2024-03-11T21:38:45.598 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.98641120400000004, 2024-03-11T21:38:45.598 INFO:teuthology.orchestra.run.smithi097.stdout: "type_data": { 2024-03-11T21:38:45.598 INFO:teuthology.orchestra.run.smithi097.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-11T21:38:45.598 INFO:teuthology.orchestra.run.smithi097.stdout: "client_info": { 2024-03-11T21:38:45.598 INFO:teuthology.orchestra.run.smithi097.stdout: "client": "client.15150", 2024-03-11T21:38:45.598 INFO:teuthology.orchestra.run.smithi097.stdout: "client_addr": "v1:172.21.15.92:0/436606717", 2024-03-11T21:38:45.598 INFO:teuthology.orchestra.run.smithi097.stdout: "tid": 203735 2024-03-11T21:38:45.598 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:38:45.598 INFO:teuthology.orchestra.run.smithi097.stdout: "events": [ 2024-03-11T21:38:45.598 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:38:45.599 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "initiated", 2024-03-11T21:38:45.599 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:34:50.349993+0000", 2024-03-11T21:38:45.599 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-03-11T21:38:45.599 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:38:45.599 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:38:45.599 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "header_read", 2024-03-11T21:38:45.599 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:34:50.349993+0000", 2024-03-11T21:38:45.599 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-03-11T21:38:45.599 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:38:45.599 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:38:45.599 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "throttled", 2024-03-11T21:38:45.599 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:34:50.349995+0000", 2024-03-11T21:38:45.599 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 2.0999999999999998e-06 2024-03-11T21:38:45.599 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:38:45.599 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:38:45.600 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "all_read", 2024-03-11T21:38:45.600 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:34:50.350000+0000", 2024-03-11T21:38:45.600 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 5.6319999999999996e-06 2024-03-11T21:38:45.600 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:38:45.600 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:38:45.600 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "dispatched", 2024-03-11T21:38:45.600 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:34:50.350016+0000", 2024-03-11T21:38:45.600 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.5406000000000001e-05 2024-03-11T21:38:45.600 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:38:45.600 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:38:45.600 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "queued_for_pg", 2024-03-11T21:38:45.600 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:34:50.350024+0000", 2024-03-11T21:38:45.600 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 8.4700000000000002e-06 2024-03-11T21:38:45.600 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:38:45.600 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:38:45.601 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "reached_pg", 2024-03-11T21:38:45.601 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:34:50.350076+0000", 2024-03-11T21:38:45.601 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 5.1838000000000001e-05 2024-03-11T21:38:45.601 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:38:45.601 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:38:45.601 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "waiting for peered", 2024-03-11T21:38:45.601 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:34:50.350084+0000", 2024-03-11T21:38:45.601 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 7.9740000000000002e-06 2024-03-11T21:38:45.601 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:38:45.601 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:38:45.601 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "reached_pg", 2024-03-11T21:38:45.601 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:34:51.327585+0000", 2024-03-11T21:38:45.602 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:38:45.602 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:38:45.602 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:38:45.603 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.97750038299999997 2024-03-11T21:38:45.603 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:38:45.603 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:38:45.603 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "started", 2024-03-11T21:38:45.604 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:34:51.327682+0000", 2024-03-11T21:38:45.604 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 9.7720000000000006e-05 2024-03-11T21:38:45.604 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:38:45.604 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:38:45.604 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "waiting for subops from 0,6", 2024-03-11T21:38:45.604 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:34:51.327889+0000", 2024-03-11T21:38:45.604 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.00020693900000000001 2024-03-11T21:38:45.604 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:38:45.604 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:38:45.604 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "sub_op_commit_rec", 2024-03-11T21:38:45.604 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:34:51.332647+0000", 2024-03-11T21:38:45.604 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.0047573939999999999 2024-03-11T21:38:45.604 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:38:45.604 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:38:45.604 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "op_commit", 2024-03-11T21:38:45.605 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:34:51.332659+0000", 2024-03-11T21:38:45.605 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.2488e-05 2024-03-11T21:38:45.605 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:38:45.605 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:38:45.605 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "sub_op_commit_rec", 2024-03-11T21:38:45.605 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:34:51.336312+0000", 2024-03-11T21:38:45.605 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.0036532610000000001 2024-03-11T21:38:45.605 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:38:45.605 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:38:45.605 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "commit_sent", 2024-03-11T21:38:45.605 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:34:51.336359+0000", 2024-03-11T21:38:45.605 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 4.7098999999999997e-05 2024-03-11T21:38:45.605 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:38:45.605 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:38:45.606 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "done", 2024-03-11T21:38:45.606 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:34:51.336404+0000", 2024-03-11T21:38:45.606 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 4.4499999999999997e-05 2024-03-11T21:38:45.606 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-03-11T21:38:45.606 INFO:teuthology.orchestra.run.smithi097.stdout: ] 2024-03-11T21:38:45.606 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-03-11T21:38:45.606 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:38:45.606 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:38:45.606 INFO:teuthology.orchestra.run.smithi097.stdout: "description": "osd_op(client.15150.0:203739 8.17 8:ec74d790:::benchmark_data_smithi092_104865_object83238:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e233)", 2024-03-11T21:38:45.606 INFO:teuthology.orchestra.run.smithi097.stdout: "initiated_at": "2024-03-11T21:34:50.352271+0000", 2024-03-11T21:38:45.606 INFO:teuthology.orchestra.run.smithi097.stdout: "age": 235.18289575700001, 2024-03-11T21:38:45.606 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.98429273100000003, 2024-03-11T21:38:45.606 INFO:teuthology.orchestra.run.smithi097.stdout: "type_data": { 2024-03-11T21:38:45.606 INFO:teuthology.orchestra.run.smithi097.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-11T21:38:45.607 INFO:teuthology.orchestra.run.smithi097.stdout: "client_info": { 2024-03-11T21:38:45.607 INFO:teuthology.orchestra.run.smithi097.stdout: "client": "client.15150", 2024-03-11T21:38:45.607 INFO:teuthology.orchestra.run.smithi097.stdout: "client_addr": "v1:172.21.15.92:0/436606717", 2024-03-11T21:38:45.607 INFO:teuthology.orchestra.run.smithi097.stdout: "tid": 203739 2024-03-11T21:38:45.607 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:38:45.607 INFO:teuthology.orchestra.run.smithi097.stdout: "events": [ 2024-03-11T21:38:45.607 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:38:45.607 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "initiated", 2024-03-11T21:38:45.607 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:34:50.352271+0000", 2024-03-11T21:38:45.607 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-03-11T21:38:45.607 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:38:45.607 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:38:45.607 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "header_read", 2024-03-11T21:38:45.607 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:34:50.352271+0000", 2024-03-11T21:38:45.608 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-03-11T21:38:45.608 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:38:45.608 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:38:45.608 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "throttled", 2024-03-11T21:38:45.608 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:34:50.352272+0000", 2024-03-11T21:38:45.608 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.3599999999999999e-06 2024-03-11T21:38:45.608 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:38:45.608 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:38:45.608 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "all_read", 2024-03-11T21:38:45.608 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:34:50.352281+0000", 2024-03-11T21:38:45.608 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 8.602e-06 2024-03-11T21:38:45.608 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:38:45.608 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:38:45.608 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "dispatched", 2024-03-11T21:38:45.608 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:34:50.352301+0000", 2024-03-11T21:38:45.609 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 2.0489999999999999e-05 2024-03-11T21:38:45.609 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:38:45.609 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:38:45.609 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "queued_for_pg", 2024-03-11T21:38:45.609 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:34:50.352308+0000", 2024-03-11T21:38:45.609 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 6.3600000000000001e-06 2024-03-11T21:38:45.609 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:38:45.609 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:38:45.609 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "reached_pg", 2024-03-11T21:38:45.609 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:34:50.352388+0000", 2024-03-11T21:38:45.609 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 8.0277000000000004e-05 2024-03-11T21:38:45.609 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:38:45.609 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:38:45.609 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "waiting for peered", 2024-03-11T21:38:45.610 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:34:50.352396+0000", 2024-03-11T21:38:45.610 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 8.5380000000000002e-06 2024-03-11T21:38:45.610 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:38:45.610 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:38:45.610 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "reached_pg", 2024-03-11T21:38:45.610 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:34:51.329474+0000", 2024-03-11T21:38:45.610 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.97707767300000004 2024-03-11T21:38:45.610 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:38:45.610 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:38:45.610 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "started", 2024-03-11T21:38:45.610 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:34:51.329570+0000", 2024-03-11T21:38:45.610 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 9.6329000000000006e-05 2024-03-11T21:38:45.610 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:38:45.610 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:38:45.611 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "waiting for subops from 0,6", 2024-03-11T21:38:45.611 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:34:51.329749+0000", 2024-03-11T21:38:45.611 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.00017819 2024-03-11T21:38:45.611 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:38:45.611 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:38:45.611 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "op_commit", 2024-03-11T21:38:45.611 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:34:51.332938+0000", 2024-03-11T21:38:45.611 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.0031892499999999998 2024-03-11T21:38:45.611 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:38:45.611 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:38:45.611 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "sub_op_commit_rec", 2024-03-11T21:38:45.611 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:34:51.336443+0000", 2024-03-11T21:38:45.611 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.003504783 2024-03-11T21:38:45.611 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:38:45.611 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:38:45.612 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "sub_op_commit_rec", 2024-03-11T21:38:45.612 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:34:51.336482+0000", 2024-03-11T21:38:45.612 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 3.9579000000000002e-05 2024-03-11T21:38:45.612 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:38:45.612 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:38:45.612 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "commit_sent", 2024-03-11T21:38:45.612 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:34:51.336527+0000", 2024-03-11T21:38:45.612 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 4.5151000000000002e-05 2024-03-11T21:38:45.612 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:38:45.612 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:38:45.612 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "done", 2024-03-11T21:38:45.612 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:34:51.336564+0000", 2024-03-11T21:38:45.612 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 3.6149000000000002e-05 2024-03-11T21:38:45.612 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-03-11T21:38:45.612 INFO:teuthology.orchestra.run.smithi097.stdout: ] 2024-03-11T21:38:45.613 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-03-11T21:38:45.613 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:38:45.613 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:38:45.613 INFO:teuthology.orchestra.run.smithi097.stdout: "description": "osd_op(client.15150.0:203799 8.17 8:eb7ac254:::benchmark_data_smithi092_104865_object83298:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e233)", 2024-03-11T21:38:45.613 INFO:teuthology.orchestra.run.smithi097.stdout: "initiated_at": "2024-03-11T21:34:50.410131+0000", 2024-03-11T21:38:45.613 INFO:teuthology.orchestra.run.smithi097.stdout: "age": 235.125035166, 2024-03-11T21:38:45.613 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.93847114499999995, 2024-03-11T21:38:45.613 INFO:teuthology.orchestra.run.smithi097.stdout: "type_data": { 2024-03-11T21:38:45.613 INFO:teuthology.orchestra.run.smithi097.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-11T21:38:45.613 INFO:teuthology.orchestra.run.smithi097.stdout: "client_info": { 2024-03-11T21:38:45.613 INFO:teuthology.orchestra.run.smithi097.stdout: "client": "client.15150", 2024-03-11T21:38:45.613 INFO:teuthology.orchestra.run.smithi097.stdout: "client_addr": "v1:172.21.15.92:0/436606717", 2024-03-11T21:38:45.613 INFO:teuthology.orchestra.run.smithi097.stdout: "tid": 203799 2024-03-11T21:38:45.613 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:38:45.613 INFO:teuthology.orchestra.run.smithi097.stdout: "events": [ 2024-03-11T21:38:45.614 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:38:45.614 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "initiated", 2024-03-11T21:38:45.614 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:34:50.410131+0000", 2024-03-11T21:38:45.614 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-03-11T21:38:45.614 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:38:45.614 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:38:45.614 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "header_read", 2024-03-11T21:38:45.614 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:34:50.410131+0000", 2024-03-11T21:38:45.614 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-03-11T21:38:45.614 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:38:45.614 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:38:45.614 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "throttled", 2024-03-11T21:38:45.614 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:34:50.410133+0000", 2024-03-11T21:38:45.614 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.5e-06 2024-03-11T21:38:45.615 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:38:45.615 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:38:45.615 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "all_read", 2024-03-11T21:38:45.615 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:34:50.410138+0000", 2024-03-11T21:38:45.615 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 5.1320000000000002e-06 2024-03-11T21:38:45.615 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:38:45.615 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:38:45.615 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "dispatched", 2024-03-11T21:38:45.615 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:34:50.410146+0000", 2024-03-11T21:38:45.615 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 7.9349999999999994e-06 2024-03-11T21:38:45.615 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:38:45.615 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:38:45.615 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "queued_for_pg", 2024-03-11T21:38:45.615 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:34:50.410151+0000", 2024-03-11T21:38:45.615 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 4.8810000000000004e-06 2024-03-11T21:38:45.616 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:38:45.616 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:38:45.616 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "reached_pg", 2024-03-11T21:38:45.616 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:34:50.410211+0000", 2024-03-11T21:38:45.616 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 5.9713999999999998e-05 2024-03-11T21:38:45.616 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:38:45.616 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:38:45.616 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "waiting for peered", 2024-03-11T21:38:45.616 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:34:50.410220+0000", 2024-03-11T21:38:45.616 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 9.1500000000000005e-06 2024-03-11T21:38:45.616 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:38:45.616 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:38:45.616 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "reached_pg", 2024-03-11T21:38:45.616 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:34:51.332969+0000", 2024-03-11T21:38:45.616 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.92274946700000005 2024-03-11T21:38:45.617 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:38:45.617 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:38:45.617 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "started", 2024-03-11T21:38:45.617 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:34:51.333079+0000", 2024-03-11T21:38:45.617 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.00010977800000000001 2024-03-11T21:38:45.617 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:38:45.617 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:38:45.617 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "waiting for subops from 0,6", 2024-03-11T21:38:45.617 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:34:51.333306+0000", 2024-03-11T21:38:45.617 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.000227555 2024-03-11T21:38:45.617 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:38:45.617 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:38:45.617 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "op_commit", 2024-03-11T21:38:45.617 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:34:51.341066+0000", 2024-03-11T21:38:45.617 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.0077596030000000003 2024-03-11T21:38:45.618 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:38:45.618 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:38:45.618 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "sub_op_commit_rec", 2024-03-11T21:38:45.618 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:34:51.343266+0000", 2024-03-11T21:38:45.618 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.0022003779999999998 2024-03-11T21:38:45.618 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:38:45.618 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:38:45.618 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "sub_op_commit_rec", 2024-03-11T21:38:45.618 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:34:51.348507+0000", 2024-03-11T21:38:45.618 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.0052403759999999997 2024-03-11T21:38:45.618 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:38:45.618 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:38:45.618 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "commit_sent", 2024-03-11T21:38:45.618 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:34:51.348556+0000", 2024-03-11T21:38:45.619 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 4.8961000000000001e-05 2024-03-11T21:38:45.619 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:38:45.619 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:38:45.619 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "done", 2024-03-11T21:38:45.619 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:34:51.348603+0000", 2024-03-11T21:38:45.619 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 4.6715000000000003e-05 2024-03-11T21:38:45.619 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-03-11T21:38:45.619 INFO:teuthology.orchestra.run.smithi097.stdout: ] 2024-03-11T21:38:45.619 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-03-11T21:38:45.619 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:38:45.619 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:38:45.619 INFO:teuthology.orchestra.run.smithi097.stdout: "description": "osd_op(client.15150.0:203989 8.10 8:0fc7a0ff:::benchmark_data_smithi092_104865_object83488:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e234)", 2024-03-11T21:38:45.619 INFO:teuthology.orchestra.run.smithi097.stdout: "initiated_at": "2024-03-11T21:34:51.357595+0000", 2024-03-11T21:38:45.619 INFO:teuthology.orchestra.run.smithi097.stdout: "age": 234.17757159600001, 2024-03-11T21:38:45.619 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.96940869299999999, 2024-03-11T21:38:45.620 INFO:teuthology.orchestra.run.smithi097.stdout: "type_data": { 2024-03-11T21:38:45.620 INFO:teuthology.orchestra.run.smithi097.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-11T21:38:45.620 INFO:teuthology.orchestra.run.smithi097.stdout: "client_info": { 2024-03-11T21:38:45.620 INFO:teuthology.orchestra.run.smithi097.stdout: "client": "client.15150", 2024-03-11T21:38:45.620 INFO:teuthology.orchestra.run.smithi097.stdout: "client_addr": "v1:172.21.15.92:0/436606717", 2024-03-11T21:38:45.620 INFO:teuthology.orchestra.run.smithi097.stdout: "tid": 203989 2024-03-11T21:38:45.620 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:38:45.620 INFO:teuthology.orchestra.run.smithi097.stdout: "events": [ 2024-03-11T21:38:45.620 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:38:45.620 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "initiated", 2024-03-11T21:38:45.620 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:34:51.357595+0000", 2024-03-11T21:38:45.620 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-03-11T21:38:45.620 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:38:45.620 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:38:45.620 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "header_read", 2024-03-11T21:38:45.620 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:34:51.357595+0000", 2024-03-11T21:38:45.621 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-03-11T21:38:45.621 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:38:45.621 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:38:45.621 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "throttled", 2024-03-11T21:38:45.621 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:34:51.357595+0000", 2024-03-11T21:38:45.621 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 5.13e-07 2024-03-11T21:38:45.621 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:38:45.621 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:38:45.621 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "all_read", 2024-03-11T21:38:45.621 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:34:51.357598+0000", 2024-03-11T21:38:45.621 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 2.356e-06 2024-03-11T21:38:45.621 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:38:45.621 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:38:45.621 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "dispatched", 2024-03-11T21:38:45.621 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:34:51.357603+0000", 2024-03-11T21:38:45.622 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 5.0119999999999996e-06 2024-03-11T21:38:45.622 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:38:45.622 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:38:45.622 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "queued_for_pg", 2024-03-11T21:38:45.622 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:34:51.357608+0000", 2024-03-11T21:38:45.622 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 4.8640000000000004e-06 2024-03-11T21:38:45.622 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:38:45.622 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:38:45.622 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "reached_pg", 2024-03-11T21:38:45.622 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:34:51.357663+0000", 2024-03-11T21:38:45.622 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 5.5458999999999999e-05 2024-03-11T21:38:45.622 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:38:45.622 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:38:45.622 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "waiting for peered", 2024-03-11T21:38:45.623 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:34:51.357671+0000", 2024-03-11T21:38:45.623 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 8.3469999999999999e-06 2024-03-11T21:38:45.623 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:38:45.623 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:38:45.623 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "reached_pg", 2024-03-11T21:38:45.623 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:34:52.319382+0000", 2024-03-11T21:38:45.623 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.961710858 2024-03-11T21:38:45.623 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:38:45.623 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:38:45.623 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "started", 2024-03-11T21:38:45.623 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:34:52.319469+0000", 2024-03-11T21:38:45.623 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 8.6674000000000004e-05 2024-03-11T21:38:45.623 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:38:45.623 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:38:45.623 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "waiting for subops from 4,5", 2024-03-11T21:38:45.624 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:34:52.319769+0000", 2024-03-11T21:38:45.624 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.00029961499999999998 2024-03-11T21:38:45.624 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:38:45.624 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:38:45.624 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "sub_op_commit_rec", 2024-03-11T21:38:45.624 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:34:52.326866+0000", 2024-03-11T21:38:45.624 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.0070977269999999999 2024-03-11T21:38:45.624 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:38:45.624 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:38:45.624 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "op_commit", 2024-03-11T21:38:45.624 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:34:52.326876+0000", 2024-03-11T21:38:45.624 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 9.6679999999999996e-06 2024-03-11T21:38:45.624 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:38:45.624 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:38:45.624 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "sub_op_commit_rec", 2024-03-11T21:38:45.625 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:34:52.326930+0000", 2024-03-11T21:38:45.625 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 5.4109e-05 2024-03-11T21:38:45.625 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:38:45.625 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:38:45.625 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "commit_sent", 2024-03-11T21:38:45.625 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:34:52.326963+0000", 2024-03-11T21:38:45.625 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 3.2583000000000003e-05 2024-03-11T21:38:45.625 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:38:45.625 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:38:45.625 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "done", 2024-03-11T21:38:45.625 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:34:52.327004+0000", 2024-03-11T21:38:45.625 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 4.0908000000000002e-05 2024-03-11T21:38:45.625 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-03-11T21:38:45.625 INFO:teuthology.orchestra.run.smithi097.stdout: ] 2024-03-11T21:38:45.625 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-03-11T21:38:45.625 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:38:45.626 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:38:45.626 INFO:teuthology.orchestra.run.smithi097.stdout: "description": "osd_op(client.15150.0:203991 8.10 8:08ac658d:::benchmark_data_smithi092_104865_object83490:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e234)", 2024-03-11T21:38:45.626 INFO:teuthology.orchestra.run.smithi097.stdout: "initiated_at": "2024-03-11T21:34:51.360667+0000", 2024-03-11T21:38:45.626 INFO:teuthology.orchestra.run.smithi097.stdout: "age": 234.174499661, 2024-03-11T21:38:45.626 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.97040149499999995, 2024-03-11T21:38:45.626 INFO:teuthology.orchestra.run.smithi097.stdout: "type_data": { 2024-03-11T21:38:45.626 INFO:teuthology.orchestra.run.smithi097.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-11T21:38:45.626 INFO:teuthology.orchestra.run.smithi097.stdout: "client_info": { 2024-03-11T21:38:45.626 INFO:teuthology.orchestra.run.smithi097.stdout: "client": "client.15150", 2024-03-11T21:38:45.626 INFO:teuthology.orchestra.run.smithi097.stdout: "client_addr": "v1:172.21.15.92:0/436606717", 2024-03-11T21:38:45.626 INFO:teuthology.orchestra.run.smithi097.stdout: "tid": 203991 2024-03-11T21:38:45.626 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:38:45.626 INFO:teuthology.orchestra.run.smithi097.stdout: "events": [ 2024-03-11T21:38:45.626 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:38:45.627 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "initiated", 2024-03-11T21:38:45.627 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:34:51.360667+0000", 2024-03-11T21:38:45.627 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-03-11T21:38:45.627 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:38:45.627 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:38:45.627 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "header_read", 2024-03-11T21:38:45.627 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:34:51.360667+0000", 2024-03-11T21:38:45.627 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-03-11T21:38:45.627 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:38:45.627 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:38:45.627 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "throttled", 2024-03-11T21:38:45.627 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:34:51.360669+0000", 2024-03-11T21:38:45.627 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.7090000000000001e-06 2024-03-11T21:38:45.627 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:38:45.627 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:38:45.628 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "all_read", 2024-03-11T21:38:45.628 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:34:51.360674+0000", 2024-03-11T21:38:45.628 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 5.113e-06 2024-03-11T21:38:45.628 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:38:45.628 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:38:45.628 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "dispatched", 2024-03-11T21:38:45.628 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:34:51.360685+0000", 2024-03-11T21:38:45.628 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.096e-05 2024-03-11T21:38:45.628 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:38:45.628 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:38:45.628 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "queued_for_pg", 2024-03-11T21:38:45.628 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:34:51.360692+0000", 2024-03-11T21:38:45.628 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 7.6260000000000003e-06 2024-03-11T21:38:45.628 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:38:45.628 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:38:45.628 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "reached_pg", 2024-03-11T21:38:45.629 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:34:51.360742+0000", 2024-03-11T21:38:45.629 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 5.0080999999999997e-05 2024-03-11T21:38:45.629 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:38:45.629 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:38:45.629 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "waiting for peered", 2024-03-11T21:38:45.629 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:34:51.360750+0000", 2024-03-11T21:38:45.629 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 8.0539999999999995e-06 2024-03-11T21:38:45.629 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:38:45.629 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:38:45.629 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "reached_pg", 2024-03-11T21:38:45.629 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:34:52.323279+0000", 2024-03-11T21:38:45.629 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.96252869900000004 2024-03-11T21:38:45.629 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:38:45.629 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:38:45.630 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "started", 2024-03-11T21:38:45.630 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:34:52.323413+0000", 2024-03-11T21:38:45.630 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.00013353599999999999 2024-03-11T21:38:45.630 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:38:45.630 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:38:45.630 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "waiting for subops from 4,5", 2024-03-11T21:38:45.630 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:34:52.323665+0000", 2024-03-11T21:38:45.630 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.00025266999999999998 2024-03-11T21:38:45.630 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:38:45.630 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:38:45.630 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "op_commit", 2024-03-11T21:38:45.630 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:34:52.330839+0000", 2024-03-11T21:38:45.630 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.0071737789999999999 2024-03-11T21:38:45.630 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:38:45.630 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:38:45.631 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "sub_op_commit_rec", 2024-03-11T21:38:45.631 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:34:52.330910+0000", 2024-03-11T21:38:45.631 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 7.1142000000000004e-05 2024-03-11T21:38:45.631 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:38:45.631 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:38:45.631 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "sub_op_commit_rec", 2024-03-11T21:38:45.631 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:34:52.330962+0000", 2024-03-11T21:38:45.631 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 5.1986e-05 2024-03-11T21:38:45.631 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:38:45.631 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:38:45.631 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "commit_sent", 2024-03-11T21:38:45.631 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:34:52.331021+0000", 2024-03-11T21:38:45.631 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 5.8325999999999997e-05 2024-03-11T21:38:45.631 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:38:45.632 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:38:45.632 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "done", 2024-03-11T21:38:45.632 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:34:52.331068+0000", 2024-03-11T21:38:45.632 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 4.7814e-05 2024-03-11T21:38:45.632 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-03-11T21:38:45.632 INFO:teuthology.orchestra.run.smithi097.stdout: ] 2024-03-11T21:38:45.632 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-03-11T21:38:45.632 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:38:45.632 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:38:45.632 INFO:teuthology.orchestra.run.smithi097.stdout: "description": "osd_op(client.15150.0:203993 8.10 8:0fcebf54:::benchmark_data_smithi092_104865_object83492:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e234)", 2024-03-11T21:38:45.632 INFO:teuthology.orchestra.run.smithi097.stdout: "initiated_at": "2024-03-11T21:34:51.361437+0000", 2024-03-11T21:38:45.632 INFO:teuthology.orchestra.run.smithi097.stdout: "age": 234.173729541, 2024-03-11T21:38:45.632 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.96990883999999999, 2024-03-11T21:38:45.632 INFO:teuthology.orchestra.run.smithi097.stdout: "type_data": { 2024-03-11T21:38:45.633 INFO:teuthology.orchestra.run.smithi097.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-11T21:38:45.633 INFO:teuthology.orchestra.run.smithi097.stdout: "client_info": { 2024-03-11T21:38:45.633 INFO:teuthology.orchestra.run.smithi097.stdout: "client": "client.15150", 2024-03-11T21:38:45.633 INFO:teuthology.orchestra.run.smithi097.stdout: "client_addr": "v1:172.21.15.92:0/436606717", 2024-03-11T21:38:45.633 INFO:teuthology.orchestra.run.smithi097.stdout: "tid": 203993 2024-03-11T21:38:45.633 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:38:45.633 INFO:teuthology.orchestra.run.smithi097.stdout: "events": [ 2024-03-11T21:38:45.633 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:38:45.633 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "initiated", 2024-03-11T21:38:45.633 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:34:51.361437+0000", 2024-03-11T21:38:45.633 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-03-11T21:38:45.633 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:38:45.633 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:38:45.633 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "header_read", 2024-03-11T21:38:45.633 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:34:51.361437+0000", 2024-03-11T21:38:45.633 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-03-11T21:38:45.634 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:38:45.634 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:38:45.634 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "throttled", 2024-03-11T21:38:45.634 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:34:51.361438+0000", 2024-03-11T21:38:45.634 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 8.7499999999999999e-07 2024-03-11T21:38:45.634 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:38:45.634 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:38:45.634 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "all_read", 2024-03-11T21:38:45.634 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:34:51.361441+0000", 2024-03-11T21:38:45.634 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 3.3639999999999999e-06 2024-03-11T21:38:45.634 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:38:45.634 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:38:45.634 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "dispatched", 2024-03-11T21:38:45.634 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:34:51.361448+0000", 2024-03-11T21:38:45.635 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 7.2529999999999998e-06 2024-03-11T21:38:45.635 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:38:45.635 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:38:45.635 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "queued_for_pg", 2024-03-11T21:38:45.635 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:34:51.361454+0000", 2024-03-11T21:38:45.635 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 5.823e-06 2024-03-11T21:38:45.635 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:38:45.635 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:38:45.635 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "reached_pg", 2024-03-11T21:38:45.635 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:34:51.361495+0000", 2024-03-11T21:38:45.635 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 4.0630999999999998e-05 2024-03-11T21:38:45.635 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:38:45.635 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:38:45.635 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "waiting for peered", 2024-03-11T21:38:45.635 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:34:51.361502+0000", 2024-03-11T21:38:45.636 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 7.029e-06 2024-03-11T21:38:45.636 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:38:45.636 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:38:45.636 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "reached_pg", 2024-03-11T21:38:45.636 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:34:52.325388+0000", 2024-03-11T21:38:45.636 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.96388603699999997 2024-03-11T21:38:45.636 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:38:45.636 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:38:45.636 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "started", 2024-03-11T21:38:45.636 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:34:52.325491+0000", 2024-03-11T21:38:45.636 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.00010323900000000001 2024-03-11T21:38:45.636 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:38:45.636 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:38:45.636 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "waiting for subops from 4,5", 2024-03-11T21:38:45.636 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:34:52.325704+0000", 2024-03-11T21:38:45.637 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.00021247799999999999 2024-03-11T21:38:45.637 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:38:45.637 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:38:45.637 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "sub_op_commit_rec", 2024-03-11T21:38:45.637 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:34:52.331132+0000", 2024-03-11T21:38:45.637 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.0054278909999999998 2024-03-11T21:38:45.637 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:38:45.637 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:38:45.637 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "op_commit", 2024-03-11T21:38:45.637 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:34:52.331149+0000", 2024-03-11T21:38:45.637 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.7629999999999999e-05 2024-03-11T21:38:45.637 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:38:45.637 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:38:45.637 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "sub_op_commit_rec", 2024-03-11T21:38:45.637 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:34:52.331254+0000", 2024-03-11T21:38:45.637 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.000105239 2024-03-11T21:38:45.638 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:38:45.638 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:38:45.638 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "commit_sent", 2024-03-11T21:38:45.638 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:34:52.331307+0000", 2024-03-11T21:38:45.638 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 5.2620000000000001e-05 2024-03-11T21:38:45.638 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:38:45.638 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:38:45.638 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "done", 2024-03-11T21:38:45.638 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:34:52.331346+0000", 2024-03-11T21:38:45.638 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 3.8730999999999999e-05 2024-03-11T21:38:45.638 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-03-11T21:38:45.638 INFO:teuthology.orchestra.run.smithi097.stdout: ] 2024-03-11T21:38:45.638 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-03-11T21:38:45.638 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:38:45.638 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:38:45.639 INFO:teuthology.orchestra.run.smithi097.stdout: "description": "osd_op(client.15150.0:204018 8.10 8:0d60ca10:::benchmark_data_smithi092_104865_object83517:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e234)", 2024-03-11T21:38:45.639 INFO:teuthology.orchestra.run.smithi097.stdout: "initiated_at": "2024-03-11T21:34:51.378792+0000", 2024-03-11T21:38:45.639 INFO:teuthology.orchestra.run.smithi097.stdout: "age": 234.15637421599999, 2024-03-11T21:38:45.639 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.95687534699999999, 2024-03-11T21:38:45.639 INFO:teuthology.orchestra.run.smithi097.stdout: "type_data": { 2024-03-11T21:38:45.639 INFO:teuthology.orchestra.run.smithi097.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-11T21:38:45.639 INFO:teuthology.orchestra.run.smithi097.stdout: "client_info": { 2024-03-11T21:38:45.639 INFO:teuthology.orchestra.run.smithi097.stdout: "client": "client.15150", 2024-03-11T21:38:45.639 INFO:teuthology.orchestra.run.smithi097.stdout: "client_addr": "v1:172.21.15.92:0/436606717", 2024-03-11T21:38:45.639 INFO:teuthology.orchestra.run.smithi097.stdout: "tid": 204018 2024-03-11T21:38:45.639 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:38:45.639 INFO:teuthology.orchestra.run.smithi097.stdout: "events": [ 2024-03-11T21:38:45.639 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:38:45.639 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "initiated", 2024-03-11T21:38:45.640 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:34:51.378792+0000", 2024-03-11T21:38:45.640 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-03-11T21:38:45.640 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:38:45.640 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:38:45.640 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "header_read", 2024-03-11T21:38:45.640 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:34:51.378792+0000", 2024-03-11T21:38:45.640 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-03-11T21:38:45.640 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:38:45.640 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:38:45.640 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "throttled", 2024-03-11T21:38:45.640 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:34:51.378794+0000", 2024-03-11T21:38:45.640 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.64e-06 2024-03-11T21:38:45.640 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:38:45.640 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:38:45.640 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "all_read", 2024-03-11T21:38:45.640 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:34:51.378800+0000", 2024-03-11T21:38:45.641 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 5.8690000000000001e-06 2024-03-11T21:38:45.641 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:38:45.641 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:38:45.641 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "dispatched", 2024-03-11T21:38:45.641 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:34:51.378808+0000", 2024-03-11T21:38:45.641 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 8.3399999999999998e-06 2024-03-11T21:38:45.641 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:38:45.641 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:38:45.641 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "queued_for_pg", 2024-03-11T21:38:45.641 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:34:51.378815+0000", 2024-03-11T21:38:45.641 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 6.6830000000000002e-06 2024-03-11T21:38:45.641 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:38:45.641 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:38:45.641 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "reached_pg", 2024-03-11T21:38:45.642 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:34:51.379938+0000", 2024-03-11T21:38:45.642 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.001123303 2024-03-11T21:38:45.642 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:38:45.642 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:38:45.642 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "waiting for peered", 2024-03-11T21:38:45.642 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:34:51.379946+0000", 2024-03-11T21:38:45.642 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 7.7370000000000006e-06 2024-03-11T21:38:45.642 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:38:45.642 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:38:45.642 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "reached_pg", 2024-03-11T21:38:45.642 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:34:52.327042+0000", 2024-03-11T21:38:45.642 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.94709584999999996 2024-03-11T21:38:45.642 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:38:45.642 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:38:45.643 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "started", 2024-03-11T21:38:45.643 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:34:52.327137+0000", 2024-03-11T21:38:45.643 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 9.5537000000000004e-05 2024-03-11T21:38:45.643 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:38:45.643 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:38:45.643 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "waiting for subops from 4,5", 2024-03-11T21:38:45.643 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:34:52.327320+0000", 2024-03-11T21:38:45.643 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.00018275799999999999 2024-03-11T21:38:45.643 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:38:45.643 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:38:45.643 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "op_commit", 2024-03-11T21:38:45.643 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:34:52.331156+0000", 2024-03-11T21:38:45.643 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.0038356190000000002 2024-03-11T21:38:45.643 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:38:45.644 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:38:45.644 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "sub_op_commit_rec", 2024-03-11T21:38:45.644 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:34:52.331404+0000", 2024-03-11T21:38:45.644 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.00024812 2024-03-11T21:38:45.644 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:38:45.644 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:38:45.644 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "sub_op_commit_rec", 2024-03-11T21:38:45.644 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:34:52.335569+0000", 2024-03-11T21:38:45.644 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.0041654910000000003 2024-03-11T21:38:45.644 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:38:45.644 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:38:45.644 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "commit_sent", 2024-03-11T21:38:45.644 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:34:52.335621+0000", 2024-03-11T21:38:45.644 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 5.1520000000000001e-05 2024-03-11T21:38:45.644 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:38:45.645 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:38:45.645 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "done", 2024-03-11T21:38:45.645 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:34:52.335668+0000", 2024-03-11T21:38:45.645 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 4.6879999999999998e-05 2024-03-11T21:38:45.645 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-03-11T21:38:45.645 INFO:teuthology.orchestra.run.smithi097.stdout: ] 2024-03-11T21:38:45.645 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-03-11T21:38:45.645 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:38:45.645 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:38:45.645 INFO:teuthology.orchestra.run.smithi097.stdout: "description": "osd_op(client.25282.0:76306 9.3 9:c88b17d3:::benchmark_data_smithi092_130291_object76305:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e277)", 2024-03-11T21:38:45.645 INFO:teuthology.orchestra.run.smithi097.stdout: "initiated_at": "2024-03-11T21:36:34.067025+0000", 2024-03-11T21:38:45.645 INFO:teuthology.orchestra.run.smithi097.stdout: "age": 131.46814129500001, 2024-03-11T21:38:45.645 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 2.4375083210000001, 2024-03-11T21:38:45.645 INFO:teuthology.orchestra.run.smithi097.stdout: "type_data": { 2024-03-11T21:38:45.645 INFO:teuthology.orchestra.run.smithi097.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-11T21:38:45.646 INFO:teuthology.orchestra.run.smithi097.stdout: "client_info": { 2024-03-11T21:38:45.646 INFO:teuthology.orchestra.run.smithi097.stdout: "client": "client.25282", 2024-03-11T21:38:45.646 INFO:teuthology.orchestra.run.smithi097.stdout: "client_addr": "v1:172.21.15.92:0/2489519821", 2024-03-11T21:38:45.646 INFO:teuthology.orchestra.run.smithi097.stdout: "tid": 76306 2024-03-11T21:38:45.646 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:38:45.646 INFO:teuthology.orchestra.run.smithi097.stdout: "events": [ 2024-03-11T21:38:45.646 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:38:45.646 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "initiated", 2024-03-11T21:38:45.646 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:36:34.067025+0000", 2024-03-11T21:38:45.646 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-03-11T21:38:45.646 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:38:45.646 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:38:45.646 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "header_read", 2024-03-11T21:38:45.646 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:36:34.067025+0000", 2024-03-11T21:38:45.646 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-03-11T21:38:45.647 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:38:45.647 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:38:45.647 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "throttled", 2024-03-11T21:38:45.647 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:36:34.067026+0000", 2024-03-11T21:38:45.647 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.1000000000000001e-06 2024-03-11T21:38:45.647 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:38:45.647 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:38:45.647 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "all_read", 2024-03-11T21:38:45.647 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:36:34.067088+0000", 2024-03-11T21:38:45.647 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 6.2080000000000002e-05 2024-03-11T21:38:45.647 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:38:45.647 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:38:45.647 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "dispatched", 2024-03-11T21:38:45.647 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:36:34.067098+0000", 2024-03-11T21:38:45.647 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 9.4120000000000007e-06 2024-03-11T21:38:45.648 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:38:45.648 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:38:45.648 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "queued_for_pg", 2024-03-11T21:38:45.648 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:36:34.067110+0000", 2024-03-11T21:38:45.648 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.2520999999999999e-05 2024-03-11T21:38:45.648 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:38:45.648 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:38:45.648 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "reached_pg", 2024-03-11T21:38:45.648 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:36:34.067169+0000", 2024-03-11T21:38:45.648 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 5.8616000000000003e-05 2024-03-11T21:38:45.648 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:38:45.648 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:38:45.648 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "waiting for peered", 2024-03-11T21:38:45.648 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:36:34.067180+0000", 2024-03-11T21:38:45.648 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.1211e-05 2024-03-11T21:38:45.648 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:38:45.649 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:38:45.649 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "reached_pg", 2024-03-11T21:38:45.649 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:36:35.042554+0000", 2024-03-11T21:38:45.649 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.97537341399999999 2024-03-11T21:38:45.649 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:38:45.649 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:38:45.649 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "waiting for peered", 2024-03-11T21:38:45.649 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:36:35.042568+0000", 2024-03-11T21:38:45.649 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.438e-05 2024-03-11T21:38:45.649 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:38:45.649 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:38:45.649 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "reached_pg", 2024-03-11T21:38:45.649 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:36:36.476295+0000", 2024-03-11T21:38:45.649 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.4337273480000001 2024-03-11T21:38:45.649 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:38:45.649 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:38:45.650 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "started", 2024-03-11T21:38:45.650 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:36:36.476497+0000", 2024-03-11T21:38:45.650 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.000201349 2024-03-11T21:38:45.650 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:38:45.650 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:38:45.650 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "waiting for subops from 1,3", 2024-03-11T21:38:45.650 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:36:36.476642+0000", 2024-03-11T21:38:45.650 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.00014531099999999999 2024-03-11T21:38:45.650 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:38:45.650 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:38:45.650 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "op_commit", 2024-03-11T21:38:45.650 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:36:36.481252+0000", 2024-03-11T21:38:45.650 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.0046099909999999999 2024-03-11T21:38:45.650 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:38:45.651 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:38:45.651 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "sub_op_commit_rec", 2024-03-11T21:38:45.651 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:36:36.481301+0000", 2024-03-11T21:38:45.651 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 4.9075e-05 2024-03-11T21:38:45.651 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:38:45.651 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:38:45.651 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "sub_op_commit_rec", 2024-03-11T21:38:45.651 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:36:36.501199+0000", 2024-03-11T21:38:45.651 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.019898223 2024-03-11T21:38:45.651 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:38:45.651 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:38:45.651 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "commit_sent", 2024-03-11T21:38:45.651 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:36:36.501283+0000", 2024-03-11T21:38:45.651 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 8.3523999999999995e-05 2024-03-11T21:38:45.651 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:38:45.652 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:38:45.652 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "done", 2024-03-11T21:38:45.652 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:36:36.504534+0000", 2024-03-11T21:38:45.652 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.003250766 2024-03-11T21:38:45.652 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-03-11T21:38:45.652 INFO:teuthology.orchestra.run.smithi097.stdout: ] 2024-03-11T21:38:45.652 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-03-11T21:38:45.652 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-03-11T21:38:45.652 INFO:teuthology.orchestra.run.smithi097.stdout: ] 2024-03-11T21:38:45.652 INFO:teuthology.orchestra.run.smithi097.stdout:} 2024-03-11T21:38:45.704 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:38:45.732 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:38:45.732 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:38:45.733 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:38:45.833 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:38:45.880 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:38:45.880 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:38:45.881 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:38:45.981 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:38:46.021 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:38:46.021 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:38:46.022 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:38:46.123 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:38:46.159 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:38:46.159 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:38:46.160 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:38:46.260 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:38:46.290 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:38:46.291 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:38:46.291 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:38:46.322 INFO:teuthology.orchestra.run.smithi092.stderr:Inferring config /var/lib/ceph/00db0016-dfed-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T21:38:46.391 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:38:46.429 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:38:46.429 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:38:46.429 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:38:46.530 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:38:46.535 DEBUG:teuthology.orchestra.run.smithi092:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid 00db0016-dfed-11ee-95c8-87774f69a715 -- ceph daemon osd.3 dump_ops_in_flight 2024-03-11T21:38:46.556 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:38:46.556 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:38:46.557 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:38:46.658 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:38:46.685 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:38:46.686 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:38:46.687 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:38:46.787 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:38:46.819 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:38:46.819 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:38:46.820 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:38:46.827 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:38:46 smithi097 ceph-mon[33317]: pgmap v612: 148 pgs: 3 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+degraded+remapped, 142 active+clean; 3.5 GiB data, 14 GiB used, 702 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 973 op/s; 23713/174546 objects degraded (13.586%); 7.2 MiB/s, 115 objects/s recovering 2024-03-11T21:38:46.862 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:38:46 smithi092 ceph-mon[29572]: pgmap v612: 148 pgs: 3 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+degraded+remapped, 142 active+clean; 3.5 GiB data, 14 GiB used, 702 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 973 op/s; 23713/174546 objects degraded (13.586%); 7.2 MiB/s, 115 objects/s recovering 2024-03-11T21:38:46.862 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:38:46 smithi092 ceph-mon[37563]: pgmap v612: 148 pgs: 3 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+degraded+remapped, 142 active+clean; 3.5 GiB data, 14 GiB used, 702 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 973 op/s; 23713/174546 objects degraded (13.586%); 7.2 MiB/s, 115 objects/s recovering 2024-03-11T21:38:46.920 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:38:46.948 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:38:46.949 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:38:46.949 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:38:47.050 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:38:47.081 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:38:47.082 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:38:47.082 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:38:47.183 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:38:47.212 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:38:47.213 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:38:47.213 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:38:47.314 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:38:47.351 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:38:47.351 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:38:47.351 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:38:47.452 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:38:47.497 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:38:47.498 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:38:47.499 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:38:47.599 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:38:47.643 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:38:47.644 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:38:47.644 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:38:47.745 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:38:47.792 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:38:47.793 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:38:47.793 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:38:47.894 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:38:47.917 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:38:47.918 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:38:47.918 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:38:48.019 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:38:48.066 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:38:48.067 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:38:48.067 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:38:48.167 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:38:48.192 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:38:48.192 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:38:48.193 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:38:48.195 INFO:journalctl@ceph.mgr.y.smithi092.stdout:Mar 11 21:38:47 smithi092 ceph-00db0016-dfed-11ee-95c8-87774f69a715-mgr-y[29801]: ::ffff:172.21.15.97 - - [11/Mar/2024:21:38:47] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-11T21:38:48.213 DEBUG:teuthology.orchestra.run.smithi092:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid 00db0016-dfed-11ee-95c8-87774f69a715 -- ceph osd set noscrub 2024-03-11T21:38:48.294 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:38:48.322 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:38:48.323 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:38:48.323 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:38:48.424 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:38:48.451 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:38:48.452 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:38:48.452 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:38:48.553 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:38:48.604 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:38:48.605 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:38:48.605 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:38:48.706 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:38:48.732 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:38:48.733 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:38:48.733 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:38:48.834 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:38:48.878 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:38:48.879 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:38:48.879 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:38:48.980 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:38:49.008 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:38:49.008 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:38:49.009 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:38:49.078 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:38:48 smithi097 ceph-mon[33317]: pgmap v613: 148 pgs: 3 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+degraded+remapped, 142 active+clean; 3.5 GiB data, 14 GiB used, 702 GiB / 715 GiB avail; 639 B/s rd, 0 B/s wr, 495 op/s; 23713/174546 objects degraded (13.586%); 3.7 MiB/s, 59 objects/s recovering 2024-03-11T21:38:49.078 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:38:48 smithi097 ceph-mon[33317]: purged_snaps scrub starts 2024-03-11T21:38:49.078 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:38:48 smithi097 ceph-mon[33317]: purged_snaps scrub ok 2024-03-11T21:38:49.078 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:38:48 smithi097 ceph-mon[33317]: 2.17 scrub starts 2024-03-11T21:38:49.078 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:38:48 smithi097 ceph-mon[33317]: 2.17 scrub ok 2024-03-11T21:38:49.078 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:38:48 smithi097 ceph-mon[33317]: Health check update: Degraded data redundancy: 23713/174546 objects degraded (13.586%), 6 pgs degraded, 6 pgs undersized (PG_DEGRADED) 2024-03-11T21:38:49.110 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:38:49.113 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:38:48 smithi092 ceph-mon[29572]: pgmap v613: 148 pgs: 3 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+degraded+remapped, 142 active+clean; 3.5 GiB data, 14 GiB used, 702 GiB / 715 GiB avail; 639 B/s rd, 0 B/s wr, 495 op/s; 23713/174546 objects degraded (13.586%); 3.7 MiB/s, 59 objects/s recovering 2024-03-11T21:38:49.113 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:38:48 smithi092 ceph-mon[29572]: purged_snaps scrub starts 2024-03-11T21:38:49.113 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:38:48 smithi092 ceph-mon[29572]: purged_snaps scrub ok 2024-03-11T21:38:49.113 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:38:48 smithi092 ceph-mon[29572]: 2.17 scrub starts 2024-03-11T21:38:49.113 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:38:48 smithi092 ceph-mon[29572]: 2.17 scrub ok 2024-03-11T21:38:49.113 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:38:48 smithi092 ceph-mon[29572]: Health check update: Degraded data redundancy: 23713/174546 objects degraded (13.586%), 6 pgs degraded, 6 pgs undersized (PG_DEGRADED) 2024-03-11T21:38:49.113 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:38:48 smithi092 ceph-mon[37563]: pgmap v613: 148 pgs: 3 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+degraded+remapped, 142 active+clean; 3.5 GiB data, 14 GiB used, 702 GiB / 715 GiB avail; 639 B/s rd, 0 B/s wr, 495 op/s; 23713/174546 objects degraded (13.586%); 3.7 MiB/s, 59 objects/s recovering 2024-03-11T21:38:49.113 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:38:48 smithi092 ceph-mon[37563]: purged_snaps scrub starts 2024-03-11T21:38:49.113 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:38:48 smithi092 ceph-mon[37563]: purged_snaps scrub ok 2024-03-11T21:38:49.113 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:38:48 smithi092 ceph-mon[37563]: 2.17 scrub starts 2024-03-11T21:38:49.114 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:38:48 smithi092 ceph-mon[37563]: 2.17 scrub ok 2024-03-11T21:38:49.114 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:38:48 smithi092 ceph-mon[37563]: Health check update: Degraded data redundancy: 23713/174546 objects degraded (13.586%), 6 pgs degraded, 6 pgs undersized (PG_DEGRADED) 2024-03-11T21:38:49.153 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:38:49.153 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:38:49.154 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:38:49.255 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:38:49.284 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:38:49.285 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:38:49.286 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:38:49.386 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:38:49.418 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:38:49.419 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:38:49.419 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:38:49.520 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:38:49.560 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:38:49.561 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:38:49.561 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:38:49.662 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:38:49.685 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:38:49.686 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:38:49.687 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:38:49.787 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:38:49.828 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:38:49.829 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:38:49.829 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:38:49.930 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:38:49.958 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:38:49.959 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:38:49.959 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:38:50.041 INFO:teuthology.orchestra.run.smithi092.stdout: 2024-03-11T21:38:50.041 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:38:49 smithi092 ceph-mon[29572]: osdmap e325: 8 total, 8 up, 8 in 2024-03-11T21:38:50.041 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:38:49 smithi092 ceph-mon[29572]: 2.11 scrub starts 2024-03-11T21:38:50.041 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:38:49 smithi092 ceph-mon[29572]: 2.11 scrub ok 2024-03-11T21:38:50.043 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:38:49 smithi092 ceph-mon[37563]: osdmap e325: 8 total, 8 up, 8 in 2024-03-11T21:38:50.043 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:38:49 smithi092 ceph-mon[37563]: 2.11 scrub starts 2024-03-11T21:38:50.043 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:38:49 smithi092 ceph-mon[37563]: 2.11 scrub ok 2024-03-11T21:38:50.058 INFO:teuthology.orchestra.run.smithi092.stderr:dumped all 2024-03-11T21:38:50.059 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:38:50.077 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:38:49 smithi097 ceph-mon[33317]: osdmap e325: 8 total, 8 up, 8 in 2024-03-11T21:38:50.078 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:38:49 smithi097 ceph-mon[33317]: 2.11 scrub starts 2024-03-11T21:38:50.078 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:38:49 smithi097 ceph-mon[33317]: 2.11 scrub ok 2024-03-11T21:38:50.108 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:38:50.109 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:38:50.109 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:38:50.210 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:38:50.240 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:38:50.240 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:38:50.241 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:38:50.342 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:38:50.376 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:38:50.377 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:38:50.377 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:38:50.478 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:38:50.501 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:38:50.502 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:38:50.502 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:38:50.603 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:38:50.638 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:38:50.639 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:38:50.639 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:38:50.740 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:38:50.768 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:38:50.768 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:38:50.769 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:38:50.869 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:38:50.900 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:38:50.901 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:38:50.901 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:38:50.993 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:38:50 smithi097 ceph-mon[33317]: pgmap v615: 148 pgs: 3 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+degraded+remapped, 142 active+clean; 3.3 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 968 op/s; 23123/163182 objects degraded (14.170%); 8.3 MiB/s, 133 objects/s recovering 2024-03-11T21:38:50.993 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:38:50 smithi097 ceph-mon[33317]: from='client.25480 v1:172.21.15.92:0/3288140539' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-11T21:38:51.002 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:38:51.035 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:38:51.035 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:38:51.036 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:38:51.112 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:38:50 smithi092 ceph-mon[29572]: pgmap v615: 148 pgs: 3 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+degraded+remapped, 142 active+clean; 3.3 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 968 op/s; 23123/163182 objects degraded (14.170%); 8.3 MiB/s, 133 objects/s recovering 2024-03-11T21:38:51.112 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:38:50 smithi092 ceph-mon[29572]: from='client.25480 v1:172.21.15.92:0/3288140539' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-11T21:38:51.112 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:38:50 smithi092 ceph-mon[37563]: pgmap v615: 148 pgs: 3 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+degraded+remapped, 142 active+clean; 3.3 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 968 op/s; 23123/163182 objects degraded (14.170%); 8.3 MiB/s, 133 objects/s recovering 2024-03-11T21:38:51.113 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:38:50 smithi092 ceph-mon[37563]: from='client.25480 v1:172.21.15.92:0/3288140539' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-11T21:38:51.136 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:38:51.177 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:38:51.178 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:38:51.179 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:38:51.279 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:38:51.317 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:38:51.318 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:38:51.318 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:38:51.419 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:38:51.457 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:38:51.458 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:38:51.459 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:38:51.559 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:38:51.602 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:38:51.603 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:38:51.603 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:38:51.704 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:38:51.732 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:38:51.732 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:38:51.733 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:38:51.834 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:38:51.876 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:38:51.877 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:38:51.877 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:38:51.978 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:38:52.012 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:38:52.013 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:38:52.013 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:38:52.064 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:38:51 smithi092 ceph-mon[29572]: 5.11 deep-scrub starts 2024-03-11T21:38:52.064 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:38:51 smithi092 ceph-mon[29572]: 5.11 deep-scrub ok 2024-03-11T21:38:52.064 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:38:51 smithi092 ceph-mon[37563]: 5.11 deep-scrub starts 2024-03-11T21:38:52.064 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:38:51 smithi092 ceph-mon[37563]: 5.11 deep-scrub ok 2024-03-11T21:38:52.077 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:38:51 smithi097 ceph-mon[33317]: 5.11 deep-scrub starts 2024-03-11T21:38:52.078 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:38:51 smithi097 ceph-mon[33317]: 5.11 deep-scrub ok 2024-03-11T21:38:52.114 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:38:52.146 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:38:52.146 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:38:52.147 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:38:52.248 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:38:52.283 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:38:52.284 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:38:52.284 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:38:52.385 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:38:52.423 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:38:52.424 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:38:52.424 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:38:52.525 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:38:52.555 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:38:52.555 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:38:52.556 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:38:52.657 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:38:52.689 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:38:52.690 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:38:52.690 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:38:52.791 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:38:52.819 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:38:52.820 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:38:52.820 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:38:52.850 INFO:teuthology.orchestra.run.smithi092.stdout:{"pg_ready":true,"pg_map":{"version":615,"stamp":"2024-03-11T21:38:49.280066+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":3552319936,"num_objects":54394,"num_object_clones":0,"num_object_copies":163182,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":23123,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":54394,"num_whiteouts":0,"num_read":2118,"num_read_kb":1966,"num_write":179469,"num_write_kb":7450846,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15103,"num_bytes_recovered":979442118,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":83112,"ondisk_log_size":83112,"up":444,"acting":438,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":444,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":749961216,"kb_used":13689320,"kb_used_data":10204008,"kb_used_omap":15,"kb_used_meta":3485168,"kb_avail":736271896,"statfs":{"total":767960285184,"available":753942421504,"internally_reserved":0,"allocated":10448904192,"data_stored":10442388312,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15601,"internal_metadata":3568812815},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,12,3,2,4,5,4],"upper_bound":128},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-507838464,"num_objects":-7749,"num_object_clones":0,"num_object_copies":-23247,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-1067,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-7749,"num_whiteouts":0,"num_read":10,"num_read_kb":10,"num_write":7749,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1067,"num_bytes_recovered":69926912,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.007518"},"pg_stats":[{"pgid":"6.1b","version":"0'0","reported_seq":499,"reported_epoch":325,"state":"active+clean","last_fresh":"2024-03-11T21:38:48.758080+0000","last_change":"2024-03-11T21:36:35.074931+0000","last_active":"2024-03-11T21:38:48.758080+0000","last_peered":"2024-03-11T21:38:48.758080+0000","last_clean":"2024-03-11T21:38:48.758080+0000","last_became_active":"2024-03-11T21:36:35.072636+0000","last_became_peered":"2024-03-11T21:36:35.072636+0000","last_unstale":"2024-03-11T21:38:48.758080+0000","last_undegraded":"2024-03-11T21:38:48.758080+0000","last_fullsized":"2024-03-11T21:38:48.758080+0000","mapping_epoch":277,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":278,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:35:07.021520+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:43.559452+0000","last_clean_scrub_stamp":"2024-03-11T21:35:07.021520+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":562,"reported_epoch":324,"state":"active+clean","last_fresh":"2024-03-11T21:38:43.364570+0000","last_change":"2024-03-11T21:36:00.720089+0000","last_active":"2024-03-11T21:38:43.364570+0000","last_peered":"2024-03-11T21:38:43.364570+0000","last_clean":"2024-03-11T21:38:43.364570+0000","last_became_active":"2024-03-11T21:25:42.578893+0000","last_became_peered":"2024-03-11T21:25:42.578893+0000","last_unstale":"2024-03-11T21:38:43.364570+0000","last_undegraded":"2024-03-11T21:38:43.364570+0000","last_fullsized":"2024-03-11T21:38:43.364570+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:36:00.719956+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:41.544550+0000","last_clean_scrub_stamp":"2024-03-11T21:36:00.719956+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1f","version":"0'0","reported_seq":591,"reported_epoch":324,"state":"active+clean","last_fresh":"2024-03-11T21:38:43.408372+0000","last_change":"2024-03-11T21:36:00.843838+0000","last_active":"2024-03-11T21:38:43.408372+0000","last_peered":"2024-03-11T21:38:43.408372+0000","last_clean":"2024-03-11T21:38:43.408372+0000","last_became_active":"2024-03-11T21:34:41.144945+0000","last_became_peered":"2024-03-11T21:34:41.144945+0000","last_unstale":"2024-03-11T21:38:43.408372+0000","last_undegraded":"2024-03-11T21:38:43.408372+0000","last_fullsized":"2024-03-11T21:38:43.408372+0000","mapping_epoch":226,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":227,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:36:00.843718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:37.512018+0000","last_clean_scrub_stamp":"2024-03-11T21:36:00.843718+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,4],"acting":[0,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.19","version":"260'27","reported_seq":546,"reported_epoch":325,"state":"active+clean","last_fresh":"2024-03-11T21:38:48.756381+0000","last_change":"2024-03-11T21:36:35.089901+0000","last_active":"2024-03-11T21:38:48.756381+0000","last_peered":"2024-03-11T21:38:48.756381+0000","last_clean":"2024-03-11T21:38:48.756381+0000","last_became_active":"2024-03-11T21:36:35.067652+0000","last_became_peered":"2024-03-11T21:36:35.067652+0000","last_unstale":"2024-03-11T21:38:48.756381+0000","last_undegraded":"2024-03-11T21:38:48.756381+0000","last_fullsized":"2024-03-11T21:38:48.756381+0000","mapping_epoch":277,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":278,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'15","last_scrub_stamp":"2024-03-11T21:35:22.099374+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:39.525554+0000","last_clean_scrub_stamp":"2024-03-11T21:35:22.099374+0000","objects_scrubbed":0,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":42,"num_read_kb":33,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.1a","version":"0'0","reported_seq":550,"reported_epoch":324,"state":"active+clean","last_fresh":"2024-03-11T21:38:43.362985+0000","last_change":"2024-03-11T21:35:16.410927+0000","last_active":"2024-03-11T21:38:43.362985+0000","last_peered":"2024-03-11T21:38:43.362985+0000","last_clean":"2024-03-11T21:38:43.362985+0000","last_became_active":"2024-03-11T21:25:44.590923+0000","last_became_peered":"2024-03-11T21:25:44.590923+0000","last_unstale":"2024-03-11T21:38:43.362985+0000","last_undegraded":"2024-03-11T21:38:43.362985+0000","last_fullsized":"2024-03-11T21:38:43.362985+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:35:16.410816+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:43.559452+0000","last_clean_scrub_stamp":"2024-03-11T21:35:16.410816+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":24,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,1],"acting":[4,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":574,"reported_epoch":324,"state":"active+clean","last_fresh":"2024-03-11T21:38:43.418226+0000","last_change":"2024-03-11T21:36:15.576138+0000","last_active":"2024-03-11T21:38:43.418226+0000","last_peered":"2024-03-11T21:38:43.418226+0000","last_clean":"2024-03-11T21:38:43.418226+0000","last_became_active":"2024-03-11T21:34:41.154216+0000","last_became_peered":"2024-03-11T21:34:41.154216+0000","last_unstale":"2024-03-11T21:38:43.418226+0000","last_undegraded":"2024-03-11T21:38:43.418226+0000","last_fullsized":"2024-03-11T21:38:43.418226+0000","mapping_epoch":226,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":227,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:36:15.575887+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:41.544550+0000","last_clean_scrub_stamp":"2024-03-11T21:36:15.575887+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,7],"acting":[1,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1e","version":"0'0","reported_seq":513,"reported_epoch":325,"state":"active+clean","last_fresh":"2024-03-11T21:38:48.756983+0000","last_change":"2024-03-11T21:36:35.062413+0000","last_active":"2024-03-11T21:38:48.756983+0000","last_peered":"2024-03-11T21:38:48.756983+0000","last_clean":"2024-03-11T21:38:48.756983+0000","last_became_active":"2024-03-11T21:36:35.061786+0000","last_became_peered":"2024-03-11T21:36:35.061786+0000","last_unstale":"2024-03-11T21:38:48.756983+0000","last_undegraded":"2024-03-11T21:38:48.756983+0000","last_fullsized":"2024-03-11T21:38:48.756983+0000","mapping_epoch":277,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":278,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:32:18.578038+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:37.512018+0000","last_clean_scrub_stamp":"2024-03-11T21:32:18.578038+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.18","version":"260'15","reported_seq":599,"reported_epoch":324,"state":"active+clean","last_fresh":"2024-03-11T21:38:43.362456+0000","last_change":"2024-03-11T21:36:09.662765+0000","last_active":"2024-03-11T21:38:43.362456+0000","last_peered":"2024-03-11T21:38:43.362456+0000","last_clean":"2024-03-11T21:38:43.362456+0000","last_became_active":"2024-03-11T21:25:40.552216+0000","last_became_peered":"2024-03-11T21:25:40.552216+0000","last_unstale":"2024-03-11T21:38:43.362456+0000","last_undegraded":"2024-03-11T21:38:43.362456+0000","last_fullsized":"2024-03-11T21:38:43.362456+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"260'15","last_scrub_stamp":"2024-03-11T21:36:09.662615+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:39.525554+0000","last_clean_scrub_stamp":"2024-03-11T21:36:09.662615+0000","objects_scrubbed":4,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":566,"reported_epoch":324,"state":"active+clean","last_fresh":"2024-03-11T21:38:43.357880+0000","last_change":"2024-03-11T21:36:35.068915+0000","last_active":"2024-03-11T21:38:43.357880+0000","last_peered":"2024-03-11T21:38:43.357880+0000","last_clean":"2024-03-11T21:38:43.357880+0000","last_became_active":"2024-03-11T21:36:35.068453+0000","last_became_peered":"2024-03-11T21:36:35.068453+0000","last_unstale":"2024-03-11T21:38:43.357880+0000","last_undegraded":"2024-03-11T21:38:43.357880+0000","last_fullsized":"2024-03-11T21:38:43.357880+0000","mapping_epoch":277,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":278,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:35:26.190634+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:32:12.654573+0000","last_clean_scrub_stamp":"2024-03-11T21:35:26.190634+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3],"acting":[5,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":214,"reported_epoch":324,"state":"active+clean","last_fresh":"2024-03-11T21:38:43.354291+0000","last_change":"2024-03-11T21:35:11.902920+0000","last_active":"2024-03-11T21:38:43.354291+0000","last_peered":"2024-03-11T21:38:43.354291+0000","last_clean":"2024-03-11T21:38:43.354291+0000","last_became_active":"2024-03-11T21:34:41.145299+0000","last_became_peered":"2024-03-11T21:34:41.145299+0000","last_unstale":"2024-03-11T21:38:43.354291+0000","last_undegraded":"2024-03-11T21:38:43.354291+0000","last_fullsized":"2024-03-11T21:38:43.354291+0000","mapping_epoch":226,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":227,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:35:11.902818+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:41.544550+0000","last_clean_scrub_stamp":"2024-03-11T21:35:11.902818+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.1d","version":"0'0","reported_seq":577,"reported_epoch":324,"state":"active+clean","last_fresh":"2024-03-11T21:38:43.354562+0000","last_change":"2024-03-11T21:35:26.855588+0000","last_active":"2024-03-11T21:38:43.354562+0000","last_peered":"2024-03-11T21:38:43.354562+0000","last_clean":"2024-03-11T21:38:43.354562+0000","last_became_active":"2024-03-11T21:25:38.542628+0000","last_became_peered":"2024-03-11T21:25:38.542628+0000","last_unstale":"2024-03-11T21:38:43.354562+0000","last_undegraded":"2024-03-11T21:38:43.354562+0000","last_fullsized":"2024-03-11T21:38:43.354562+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:35:26.855511+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:37.512018+0000","last_clean_scrub_stamp":"2024-03-11T21:35:26.855511+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,0],"acting":[7,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.1b","version":"260'7","reported_seq":585,"reported_epoch":324,"state":"active+clean","last_fresh":"2024-03-11T21:38:43.363688+0000","last_change":"2024-03-11T21:36:35.087276+0000","last_active":"2024-03-11T21:38:43.363688+0000","last_peered":"2024-03-11T21:38:43.363688+0000","last_clean":"2024-03-11T21:38:43.363688+0000","last_became_active":"2024-03-11T21:36:35.057678+0000","last_became_peered":"2024-03-11T21:36:35.057678+0000","last_unstale":"2024-03-11T21:38:43.363688+0000","last_undegraded":"2024-03-11T21:38:43.363688+0000","last_fullsized":"2024-03-11T21:38:43.363688+0000","mapping_epoch":277,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":278,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'5","last_scrub_stamp":"2024-03-11T21:35:09.427562+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:39.525554+0000","last_clean_scrub_stamp":"2024-03-11T21:35:09.427562+0000","objects_scrubbed":2,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":14,"num_read_kb":10,"num_write":8,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,1],"acting":[4,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":558,"reported_epoch":324,"state":"active+clean","last_fresh":"2024-03-11T21:38:43.406489+0000","last_change":"2024-03-11T21:35:39.691825+0000","last_active":"2024-03-11T21:38:43.406489+0000","last_peered":"2024-03-11T21:38:43.406489+0000","last_clean":"2024-03-11T21:38:43.406489+0000","last_became_active":"2024-03-11T21:25:44.589786+0000","last_became_peered":"2024-03-11T21:25:44.589786+0000","last_unstale":"2024-03-11T21:38:43.406489+0000","last_undegraded":"2024-03-11T21:38:43.406489+0000","last_fullsized":"2024-03-11T21:38:43.406489+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:35:39.691690+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:43.559452+0000","last_clean_scrub_stamp":"2024-03-11T21:35:39.691690+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":18,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,7],"acting":[0,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":573,"reported_epoch":324,"state":"active+clean","last_fresh":"2024-03-11T21:38:43.358757+0000","last_change":"2024-03-11T21:35:25.194240+0000","last_active":"2024-03-11T21:38:43.358757+0000","last_peered":"2024-03-11T21:38:43.358757+0000","last_clean":"2024-03-11T21:38:43.358757+0000","last_became_active":"2024-03-11T21:34:41.147185+0000","last_became_peered":"2024-03-11T21:34:41.147185+0000","last_unstale":"2024-03-11T21:38:43.358757+0000","last_undegraded":"2024-03-11T21:38:43.358757+0000","last_fullsized":"2024-03-11T21:38:43.358757+0000","mapping_epoch":226,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":227,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:35:25.194115+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:32:21.535034+0000","last_clean_scrub_stamp":"2024-03-11T21:35:25.194115+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,7],"acting":[5,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1c","version":"0'0","reported_seq":577,"reported_epoch":324,"state":"active+clean","last_fresh":"2024-03-11T21:38:43.355023+0000","last_change":"2024-03-11T21:35:28.789693+0000","last_active":"2024-03-11T21:38:43.355023+0000","last_peered":"2024-03-11T21:38:43.355023+0000","last_clean":"2024-03-11T21:38:43.355023+0000","last_became_active":"2024-03-11T21:25:38.541364+0000","last_became_peered":"2024-03-11T21:25:38.541364+0000","last_unstale":"2024-03-11T21:38:43.355023+0000","last_undegraded":"2024-03-11T21:38:43.355023+0000","last_fullsized":"2024-03-11T21:38:43.355023+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:35:28.789592+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:28:32.566437+0000","last_clean_scrub_stamp":"2024-03-11T21:35:28.789592+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.1a","version":"260'15","reported_seq":609,"reported_epoch":324,"state":"active+clean","last_fresh":"2024-03-11T21:38:43.363079+0000","last_change":"2024-03-11T21:36:35.085358+0000","last_active":"2024-03-11T21:38:43.363079+0000","last_peered":"2024-03-11T21:38:43.363079+0000","last_clean":"2024-03-11T21:38:43.363079+0000","last_became_active":"2024-03-11T21:36:35.057029+0000","last_became_peered":"2024-03-11T21:36:35.057029+0000","last_unstale":"2024-03-11T21:38:43.363079+0000","last_undegraded":"2024-03-11T21:38:43.363079+0000","last_fullsized":"2024-03-11T21:38:43.363079+0000","mapping_epoch":277,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":278,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'9","last_scrub_stamp":"2024-03-11T21:35:05.451780+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:39.525554+0000","last_clean_scrub_stamp":"2024-03-11T21:35:05.451780+0000","objects_scrubbed":4,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0],"acting":[4,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1f","version":"0'0","reported_seq":436,"reported_epoch":325,"state":"active+clean","last_fresh":"2024-03-11T21:38:48.759090+0000","last_change":"2024-03-11T21:36:35.074675+0000","last_active":"2024-03-11T21:38:48.759090+0000","last_peered":"2024-03-11T21:38:48.759090+0000","last_clean":"2024-03-11T21:38:48.759090+0000","last_became_active":"2024-03-11T21:36:35.071755+0000","last_became_peered":"2024-03-11T21:36:35.071755+0000","last_unstale":"2024-03-11T21:38:48.759090+0000","last_undegraded":"2024-03-11T21:38:48.759090+0000","last_fullsized":"2024-03-11T21:38:48.759090+0000","mapping_epoch":277,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":278,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:34:05.862891+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:43.559452+0000","last_clean_scrub_stamp":"2024-03-11T21:34:05.862891+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1c","version":"0'0","reported_seq":563,"reported_epoch":324,"state":"active+clean","last_fresh":"2024-03-11T21:38:43.363753+0000","last_change":"2024-03-11T21:36:35.067469+0000","last_active":"2024-03-11T21:38:43.363753+0000","last_peered":"2024-03-11T21:38:43.363753+0000","last_clean":"2024-03-11T21:38:43.363753+0000","last_became_active":"2024-03-11T21:36:35.067127+0000","last_became_peered":"2024-03-11T21:36:35.067127+0000","last_unstale":"2024-03-11T21:38:43.363753+0000","last_undegraded":"2024-03-11T21:38:43.363753+0000","last_fullsized":"2024-03-11T21:38:43.363753+0000","mapping_epoch":277,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":278,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:34:11.766066+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:41.544550+0000","last_clean_scrub_stamp":"2024-03-11T21:34:11.766066+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1b","version":"0'0","reported_seq":484,"reported_epoch":325,"state":"active+clean","last_fresh":"2024-03-11T21:38:48.758176+0000","last_change":"2024-03-11T21:36:35.075385+0000","last_active":"2024-03-11T21:38:48.758176+0000","last_peered":"2024-03-11T21:38:48.758176+0000","last_clean":"2024-03-11T21:38:48.758176+0000","last_became_active":"2024-03-11T21:36:35.073959+0000","last_became_peered":"2024-03-11T21:36:35.073959+0000","last_unstale":"2024-03-11T21:38:48.758176+0000","last_undegraded":"2024-03-11T21:38:48.758176+0000","last_fullsized":"2024-03-11T21:38:48.758176+0000","mapping_epoch":277,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":278,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:34:27.998724+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:34:27.998724+0000","last_clean_scrub_stamp":"2024-03-11T21:34:27.998724+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1d","version":"260'18","reported_seq":528,"reported_epoch":325,"state":"active+clean","last_fresh":"2024-03-11T21:38:48.756522+0000","last_change":"2024-03-11T21:36:35.080207+0000","last_active":"2024-03-11T21:38:48.756522+0000","last_peered":"2024-03-11T21:38:48.756522+0000","last_clean":"2024-03-11T21:38:48.756522+0000","last_became_active":"2024-03-11T21:36:35.055880+0000","last_became_peered":"2024-03-11T21:36:35.055880+0000","last_unstale":"2024-03-11T21:38:48.756522+0000","last_undegraded":"2024-03-11T21:38:48.756522+0000","last_fullsized":"2024-03-11T21:38:48.756522+0000","mapping_epoch":277,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":278,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'12","last_scrub_stamp":"2024-03-11T21:35:07.971581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:39.525554+0000","last_clean_scrub_stamp":"2024-03-11T21:35:07.971581+0000","objects_scrubbed":0,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":33,"num_read_kb":24,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.1e","version":"0'0","reported_seq":551,"reported_epoch":324,"state":"active+clean","last_fresh":"2024-03-11T21:38:43.363557+0000","last_change":"2024-03-11T21:35:32.664451+0000","last_active":"2024-03-11T21:38:43.363557+0000","last_peered":"2024-03-11T21:38:43.363557+0000","last_clean":"2024-03-11T21:38:43.363557+0000","last_became_active":"2024-03-11T21:25:45.300355+0000","last_became_peered":"2024-03-11T21:25:45.300355+0000","last_unstale":"2024-03-11T21:38:43.363557+0000","last_undegraded":"2024-03-11T21:38:43.363557+0000","last_fullsized":"2024-03-11T21:38:43.363557+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:35:32.664371+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:43.559452+0000","last_clean_scrub_stamp":"2024-03-11T21:35:32.664371+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5],"acting":[4,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1d","version":"0'0","reported_seq":562,"reported_epoch":324,"state":"active+clean","last_fresh":"2024-03-11T21:38:43.421640+0000","last_change":"2024-03-11T21:36:07.624196+0000","last_active":"2024-03-11T21:38:43.421640+0000","last_peered":"2024-03-11T21:38:43.421640+0000","last_clean":"2024-03-11T21:38:43.421640+0000","last_became_active":"2024-03-11T21:25:42.579407+0000","last_became_peered":"2024-03-11T21:25:42.579407+0000","last_unstale":"2024-03-11T21:38:43.421640+0000","last_undegraded":"2024-03-11T21:38:43.421640+0000","last_fullsized":"2024-03-11T21:38:43.421640+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:36:07.624075+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:30:18.678531+0000","last_clean_scrub_stamp":"2024-03-11T21:36:07.624075+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1a","version":"0'0","reported_seq":590,"reported_epoch":324,"state":"active+clean","last_fresh":"2024-03-11T21:38:43.362666+0000","last_change":"2024-03-11T21:36:12.993542+0000","last_active":"2024-03-11T21:38:43.362666+0000","last_peered":"2024-03-11T21:38:43.362666+0000","last_clean":"2024-03-11T21:38:43.362666+0000","last_became_active":"2024-03-11T21:34:41.145014+0000","last_became_peered":"2024-03-11T21:34:41.145014+0000","last_unstale":"2024-03-11T21:38:43.362666+0000","last_undegraded":"2024-03-11T21:38:43.362666+0000","last_fullsized":"2024-03-11T21:38:43.362666+0000","mapping_epoch":226,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":227,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:36:12.993437+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:37.512018+0000","last_clean_scrub_stamp":"2024-03-11T21:36:12.993437+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.1c","version":"260'27","reported_seq":624,"reported_epoch":324,"state":"active+clean","last_fresh":"2024-03-11T21:38:43.408836+0000","last_change":"2024-03-11T21:36:35.097208+0000","last_active":"2024-03-11T21:38:43.408836+0000","last_peered":"2024-03-11T21:38:43.408836+0000","last_clean":"2024-03-11T21:38:43.408836+0000","last_became_active":"2024-03-11T21:36:35.074579+0000","last_became_peered":"2024-03-11T21:36:35.074579+0000","last_unstale":"2024-03-11T21:38:43.408836+0000","last_undegraded":"2024-03-11T21:38:43.408836+0000","last_fullsized":"2024-03-11T21:38:43.408836+0000","mapping_epoch":277,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":278,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'15","last_scrub_stamp":"2024-03-11T21:34:10.311181+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:39.525554+0000","last_clean_scrub_stamp":"2024-03-11T21:34:10.311181+0000","objects_scrubbed":7,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":44,"num_read_kb":35,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.1d","version":"0'0","reported_seq":550,"reported_epoch":324,"state":"active+clean","last_fresh":"2024-03-11T21:38:43.421455+0000","last_change":"2024-03-11T21:35:21.744646+0000","last_active":"2024-03-11T21:38:43.421455+0000","last_peered":"2024-03-11T21:38:43.421455+0000","last_clean":"2024-03-11T21:38:43.421455+0000","last_became_active":"2024-03-11T21:25:44.588845+0000","last_became_peered":"2024-03-11T21:25:44.588845+0000","last_unstale":"2024-03-11T21:38:43.421455+0000","last_undegraded":"2024-03-11T21:38:43.421455+0000","last_fullsized":"2024-03-11T21:38:43.421455+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:35:21.744223+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:43.559452+0000","last_clean_scrub_stamp":"2024-03-11T21:35:21.744223+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":99,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.1e","version":"0'0","reported_seq":563,"reported_epoch":324,"state":"active+clean","last_fresh":"2024-03-11T21:38:43.407500+0000","last_change":"2024-03-11T21:35:10.728691+0000","last_active":"2024-03-11T21:38:43.407500+0000","last_peered":"2024-03-11T21:38:43.407500+0000","last_clean":"2024-03-11T21:38:43.407500+0000","last_became_active":"2024-03-11T21:25:42.582959+0000","last_became_peered":"2024-03-11T21:25:42.582959+0000","last_unstale":"2024-03-11T21:38:43.407500+0000","last_undegraded":"2024-03-11T21:38:43.407500+0000","last_fullsized":"2024-03-11T21:38:43.407500+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:35:10.728569+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:26:58.335431+0000","last_clean_scrub_stamp":"2024-03-11T21:35:10.728569+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":18,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.19","version":"67'2","reported_seq":510,"reported_epoch":325,"state":"active+clean","last_fresh":"2024-03-11T21:38:48.756279+0000","last_change":"2024-03-11T21:36:35.069729+0000","last_active":"2024-03-11T21:38:48.756279+0000","last_peered":"2024-03-11T21:38:48.756279+0000","last_clean":"2024-03-11T21:38:48.756279+0000","last_became_active":"2024-03-11T21:36:35.064249+0000","last_became_peered":"2024-03-11T21:36:35.064249+0000","last_unstale":"2024-03-11T21:38:48.756279+0000","last_undegraded":"2024-03-11T21:38:48.756279+0000","last_fullsized":"2024-03-11T21:38:48.756279+0000","mapping_epoch":277,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":278,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'2","last_scrub_stamp":"2024-03-11T21:32:15.509542+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:37.512018+0000","last_clean_scrub_stamp":"2024-03-11T21:32:15.509542+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":957,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,0],"acting":[3,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1f","version":"260'24","reported_seq":629,"reported_epoch":324,"state":"active+clean","last_fresh":"2024-03-11T21:38:43.363373+0000","last_change":"2024-03-11T21:35:38.173937+0000","last_active":"2024-03-11T21:38:43.363373+0000","last_peered":"2024-03-11T21:38:43.363373+0000","last_clean":"2024-03-11T21:38:43.363373+0000","last_became_active":"2024-03-11T21:32:22.798242+0000","last_became_peered":"2024-03-11T21:32:22.798242+0000","last_unstale":"2024-03-11T21:38:43.363373+0000","last_undegraded":"2024-03-11T21:38:43.363373+0000","last_fullsized":"2024-03-11T21:38:43.363373+0000","mapping_epoch":166,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":167,"parent":"0.0","parent_split_bits":0,"last_scrub":"124'16","last_scrub_stamp":"2024-03-11T21:35:38.173841+0000","last_deep_scrub":"124'16","last_deep_scrub_stamp":"2024-03-11T21:35:38.173841+0000","last_clean_scrub_stamp":"2024-03-11T21:35:38.173841+0000","objects_scrubbed":5,"log_size":24,"log_dups_size":0,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":32,"num_read_kb":25,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,0],"acting":[6,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.1c","version":"74'1","reported_seq":218,"reported_epoch":324,"state":"active+clean","last_fresh":"2024-03-11T21:38:43.354870+0000","last_change":"2024-03-11T21:36:11.732043+0000","last_active":"2024-03-11T21:38:43.354870+0000","last_peered":"2024-03-11T21:38:43.354870+0000","last_clean":"2024-03-11T21:38:43.354870+0000","last_became_active":"2024-03-11T21:34:41.144839+0000","last_became_peered":"2024-03-11T21:34:41.144839+0000","last_unstale":"2024-03-11T21:38:43.354870+0000","last_undegraded":"2024-03-11T21:38:43.354870+0000","last_fullsized":"2024-03-11T21:38:43.354870+0000","mapping_epoch":226,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":227,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'1","last_scrub_stamp":"2024-03-11T21:36:11.731914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:43.559452+0000","last_clean_scrub_stamp":"2024-03-11T21:36:11.731914+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":370,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":370,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.1f","version":"229'10","reported_seq":768,"reported_epoch":324,"state":"active+clean","last_fresh":"2024-03-11T21:38:43.363318+0000","last_change":"2024-03-11T21:36:08.871358+0000","last_active":"2024-03-11T21:38:43.363318+0000","last_peered":"2024-03-11T21:38:43.363318+0000","last_clean":"2024-03-11T21:38:43.363318+0000","last_became_active":"2024-03-11T21:34:41.145862+0000","last_became_peered":"2024-03-11T21:34:41.145862+0000","last_unstale":"2024-03-11T21:38:43.363318+0000","last_undegraded":"2024-03-11T21:38:43.363318+0000","last_fullsized":"2024-03-11T21:38:43.363318+0000","mapping_epoch":226,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":227,"parent":"0.0","parent_split_bits":0,"last_scrub":"229'10","last_scrub_stamp":"2024-03-11T21:36:08.871213+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:41.544550+0000","last_clean_scrub_stamp":"2024-03-11T21:36:08.871213+0000","objects_scrubbed":1,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.18","version":"67'1","reported_seq":613,"reported_epoch":324,"state":"active+clean","last_fresh":"2024-03-11T21:38:43.357748+0000","last_change":"2024-03-11T21:36:10.458150+0000","last_active":"2024-03-11T21:38:43.357748+0000","last_peered":"2024-03-11T21:38:43.357748+0000","last_clean":"2024-03-11T21:38:43.357748+0000","last_became_active":"2024-03-11T21:35:23.310241+0000","last_became_peered":"2024-03-11T21:35:23.310241+0000","last_unstale":"2024-03-11T21:38:43.357748+0000","last_undegraded":"2024-03-11T21:38:43.357748+0000","last_fullsized":"2024-03-11T21:38:43.357748+0000","mapping_epoch":248,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":249,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'1","last_scrub_stamp":"2024-03-11T21:36:10.458028+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:37.512018+0000","last_clean_scrub_stamp":"2024-03-11T21:36:10.458028+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":436,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":7,"num_read_kb":7,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":1308,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,2],"acting":[5,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1e","version":"260'20","reported_seq":655,"reported_epoch":324,"state":"active+clean","last_fresh":"2024-03-11T21:38:43.407426+0000","last_change":"2024-03-11T21:36:35.083447+0000","last_active":"2024-03-11T21:38:43.407426+0000","last_peered":"2024-03-11T21:38:43.407426+0000","last_clean":"2024-03-11T21:38:43.407426+0000","last_became_active":"2024-03-11T21:36:35.057843+0000","last_became_peered":"2024-03-11T21:36:35.057843+0000","last_unstale":"2024-03-11T21:38:43.407426+0000","last_undegraded":"2024-03-11T21:38:43.407426+0000","last_fullsized":"2024-03-11T21:38:43.407426+0000","mapping_epoch":277,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":278,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'10","last_scrub_stamp":"2024-03-11T21:35:16.825146+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:39.525554+0000","last_clean_scrub_stamp":"2024-03-11T21:35:16.825146+0000","objects_scrubbed":5,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":44,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":25,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.13","version":"0'0","reported_seq":443,"reported_epoch":325,"state":"active+clean","last_fresh":"2024-03-11T21:38:48.758394+0000","last_change":"2024-03-11T21:36:35.063744+0000","last_active":"2024-03-11T21:38:48.758394+0000","last_peered":"2024-03-11T21:38:48.758394+0000","last_clean":"2024-03-11T21:38:48.758394+0000","last_became_active":"2024-03-11T21:36:35.062618+0000","last_became_peered":"2024-03-11T21:36:35.062618+0000","last_unstale":"2024-03-11T21:38:48.758394+0000","last_undegraded":"2024-03-11T21:38:48.758394+0000","last_fullsized":"2024-03-11T21:38:48.758394+0000","mapping_epoch":277,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":278,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:34:13.822803+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:43.559452+0000","last_clean_scrub_stamp":"2024-03-11T21:34:13.822803+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":214,"reported_epoch":324,"state":"active+clean","last_fresh":"2024-03-11T21:38:43.353723+0000","last_change":"2024-03-11T21:35:16.917648+0000","last_active":"2024-03-11T21:38:43.353723+0000","last_peered":"2024-03-11T21:38:43.353723+0000","last_clean":"2024-03-11T21:38:43.353723+0000","last_became_active":"2024-03-11T21:34:41.142569+0000","last_became_peered":"2024-03-11T21:34:41.142569+0000","last_unstale":"2024-03-11T21:38:43.353723+0000","last_undegraded":"2024-03-11T21:38:43.353723+0000","last_fullsized":"2024-03-11T21:38:43.353723+0000","mapping_epoch":226,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":227,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:35:16.917532+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:41.544550+0000","last_clean_scrub_stamp":"2024-03-11T21:35:16.917532+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.17","version":"0'0","reported_seq":570,"reported_epoch":324,"state":"active+clean","last_fresh":"2024-03-11T21:38:43.363023+0000","last_change":"2024-03-11T21:35:06.186953+0000","last_active":"2024-03-11T21:38:43.363023+0000","last_peered":"2024-03-11T21:38:43.363023+0000","last_clean":"2024-03-11T21:38:43.363023+0000","last_became_active":"2024-03-11T21:25:38.540349+0000","last_became_peered":"2024-03-11T21:25:38.540349+0000","last_unstale":"2024-03-11T21:38:43.363023+0000","last_undegraded":"2024-03-11T21:38:43.363023+0000","last_fullsized":"2024-03-11T21:38:43.363023+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:35:06.186840+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:28:25.918101+0000","last_clean_scrub_stamp":"2024-03-11T21:35:06.186840+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,2],"acting":[6,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.11","version":"260'19","reported_seq":569,"reported_epoch":325,"state":"active+clean","last_fresh":"2024-03-11T21:38:48.756786+0000","last_change":"2024-03-11T21:36:35.096720+0000","last_active":"2024-03-11T21:38:48.756786+0000","last_peered":"2024-03-11T21:38:48.756786+0000","last_clean":"2024-03-11T21:38:48.756786+0000","last_became_active":"2024-03-11T21:36:35.073689+0000","last_became_peered":"2024-03-11T21:36:35.073689+0000","last_unstale":"2024-03-11T21:38:48.756786+0000","last_undegraded":"2024-03-11T21:38:48.756786+0000","last_fullsized":"2024-03-11T21:38:48.756786+0000","mapping_epoch":277,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":278,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'11","last_scrub_stamp":"2024-03-11T21:35:16.044397+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:39.525554+0000","last_clean_scrub_stamp":"2024-03-11T21:35:16.044397+0000","objects_scrubbed":5,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":31,"num_read_kb":24,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":214,"reported_epoch":324,"state":"active+clean","last_fresh":"2024-03-11T21:38:43.354190+0000","last_change":"2024-03-11T21:35:32.719869+0000","last_active":"2024-03-11T21:38:43.354190+0000","last_peered":"2024-03-11T21:38:43.354190+0000","last_clean":"2024-03-11T21:38:43.354190+0000","last_became_active":"2024-03-11T21:34:41.137235+0000","last_became_peered":"2024-03-11T21:34:41.137235+0000","last_unstale":"2024-03-11T21:38:43.354190+0000","last_undegraded":"2024-03-11T21:38:43.354190+0000","last_fullsized":"2024-03-11T21:38:43.354190+0000","mapping_epoch":226,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":227,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:35:32.719789+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:43.559452+0000","last_clean_scrub_stamp":"2024-03-11T21:35:32.719789+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,4],"acting":[7,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":582,"reported_epoch":324,"state":"active+clean","last_fresh":"2024-03-11T21:38:43.363875+0000","last_change":"2024-03-11T21:35:31.111662+0000","last_active":"2024-03-11T21:38:43.363875+0000","last_peered":"2024-03-11T21:38:43.363875+0000","last_clean":"2024-03-11T21:38:43.363875+0000","last_became_active":"2024-03-11T21:34:41.156036+0000","last_became_peered":"2024-03-11T21:34:41.156036+0000","last_unstale":"2024-03-11T21:38:43.363875+0000","last_undegraded":"2024-03-11T21:38:43.363875+0000","last_fullsized":"2024-03-11T21:38:43.363875+0000","mapping_epoch":226,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":227,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:35:31.111559+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:29:39.923895+0000","last_clean_scrub_stamp":"2024-03-11T21:35:31.111559+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.16","version":"0'0","reported_seq":577,"reported_epoch":324,"state":"active+clean","last_fresh":"2024-03-11T21:38:43.358751+0000","last_change":"2024-03-11T21:35:39.108623+0000","last_active":"2024-03-11T21:38:43.358751+0000","last_peered":"2024-03-11T21:38:43.358751+0000","last_clean":"2024-03-11T21:38:43.358751+0000","last_became_active":"2024-03-11T21:25:38.539103+0000","last_became_peered":"2024-03-11T21:25:38.539103+0000","last_unstale":"2024-03-11T21:38:43.358751+0000","last_undegraded":"2024-03-11T21:38:43.358751+0000","last_fullsized":"2024-03-11T21:38:43.358751+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:35:39.108546+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:27:11.938909+0000","last_clean_scrub_stamp":"2024-03-11T21:35:39.108546+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,2],"acting":[5,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.10","version":"260'10","reported_seq":451,"reported_epoch":325,"state":"active+clean","last_fresh":"2024-03-11T21:38:48.777505+0000","last_change":"2024-03-11T21:36:35.084268+0000","last_active":"2024-03-11T21:38:48.777505+0000","last_peered":"2024-03-11T21:38:48.777505+0000","last_clean":"2024-03-11T21:38:48.777505+0000","last_became_active":"2024-03-11T21:36:35.067500+0000","last_became_peered":"2024-03-11T21:36:35.067500+0000","last_unstale":"2024-03-11T21:38:48.777505+0000","last_undegraded":"2024-03-11T21:38:48.777505+0000","last_fullsized":"2024-03-11T21:38:48.777505+0000","mapping_epoch":277,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":278,"parent":"0.0","parent_split_bits":0,"last_scrub":"94'6","last_scrub_stamp":"2024-03-11T21:34:00.727416+0000","last_deep_scrub":"94'6","last_deep_scrub_stamp":"2024-03-11T21:28:24.603974+0000","last_clean_scrub_stamp":"2024-03-11T21:34:00.727416+0000","objects_scrubbed":2,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":12,"num_read_kb":10,"num_write":8,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6],"acting":[3,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":496,"reported_epoch":325,"state":"active+clean","last_fresh":"2024-03-11T21:38:48.756589+0000","last_change":"2024-03-11T21:36:35.070468+0000","last_active":"2024-03-11T21:38:48.756589+0000","last_peered":"2024-03-11T21:38:48.756589+0000","last_clean":"2024-03-11T21:38:48.756589+0000","last_became_active":"2024-03-11T21:36:35.070176+0000","last_became_peered":"2024-03-11T21:36:35.070176+0000","last_unstale":"2024-03-11T21:38:48.756589+0000","last_undegraded":"2024-03-11T21:38:48.756589+0000","last_fullsized":"2024-03-11T21:38:48.756589+0000","mapping_epoch":277,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":278,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:35:18.062272+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:43.559452+0000","last_clean_scrub_stamp":"2024-03-11T21:35:18.062272+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":563,"reported_epoch":324,"state":"active+clean","last_fresh":"2024-03-11T21:38:43.418605+0000","last_change":"2024-03-11T21:36:35.058551+0000","last_active":"2024-03-11T21:38:43.418605+0000","last_peered":"2024-03-11T21:38:43.418605+0000","last_clean":"2024-03-11T21:38:43.418605+0000","last_became_active":"2024-03-11T21:36:35.057906+0000","last_became_peered":"2024-03-11T21:36:35.057906+0000","last_unstale":"2024-03-11T21:38:43.418605+0000","last_undegraded":"2024-03-11T21:38:43.418605+0000","last_fullsized":"2024-03-11T21:38:43.418605+0000","mapping_epoch":277,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":278,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:34:15.516278+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:27:21.207303+0000","last_clean_scrub_stamp":"2024-03-11T21:34:15.516278+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.15","version":"0'0","reported_seq":570,"reported_epoch":324,"state":"active+clean","last_fresh":"2024-03-11T21:38:43.421536+0000","last_change":"2024-03-11T21:36:06.579754+0000","last_active":"2024-03-11T21:38:43.421536+0000","last_peered":"2024-03-11T21:38:43.421536+0000","last_clean":"2024-03-11T21:38:43.421536+0000","last_became_active":"2024-03-11T21:25:38.541705+0000","last_became_peered":"2024-03-11T21:25:38.541705+0000","last_unstale":"2024-03-11T21:38:43.421536+0000","last_undegraded":"2024-03-11T21:38:43.421536+0000","last_fullsized":"2024-03-11T21:38:43.421536+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:36:06.579636+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:26:44.860499+0000","last_clean_scrub_stamp":"2024-03-11T21:36:06.579636+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.13","version":"260'24","reported_seq":620,"reported_epoch":324,"state":"active+clean","last_fresh":"2024-03-11T21:38:43.363516+0000","last_change":"2024-03-11T21:35:14.453053+0000","last_active":"2024-03-11T21:38:43.363516+0000","last_peered":"2024-03-11T21:38:43.363516+0000","last_clean":"2024-03-11T21:38:43.363516+0000","last_became_active":"2024-03-11T21:26:11.214879+0000","last_became_peered":"2024-03-11T21:26:11.214879+0000","last_unstale":"2024-03-11T21:38:43.363516+0000","last_undegraded":"2024-03-11T21:38:43.363516+0000","last_fullsized":"2024-03-11T21:38:43.363516+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"109'16","last_scrub_stamp":"2024-03-11T21:35:14.452891+0000","last_deep_scrub":"109'16","last_deep_scrub_stamp":"2024-03-11T21:32:24.657490+0000","last_clean_scrub_stamp":"2024-03-11T21:35:14.452891+0000","objects_scrubbed":5,"log_size":24,"log_dups_size":0,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":32,"num_read_kb":25,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,2],"acting":[4,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.10","version":"0'0","reported_seq":558,"reported_epoch":324,"state":"active+clean","last_fresh":"2024-03-11T21:38:43.354539+0000","last_change":"2024-03-11T21:35:29.712658+0000","last_active":"2024-03-11T21:38:43.354539+0000","last_peered":"2024-03-11T21:38:43.354539+0000","last_clean":"2024-03-11T21:38:43.354539+0000","last_became_active":"2024-03-11T21:25:44.591189+0000","last_became_peered":"2024-03-11T21:25:44.591189+0000","last_unstale":"2024-03-11T21:38:43.354539+0000","last_undegraded":"2024-03-11T21:38:43.354539+0000","last_fullsized":"2024-03-11T21:38:43.354539+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:35:29.712557+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:27:25.417450+0000","last_clean_scrub_stamp":"2024-03-11T21:35:29.712557+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,1],"acting":[0,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":494,"reported_epoch":325,"state":"active+clean","last_fresh":"2024-03-11T21:38:48.758511+0000","last_change":"2024-03-11T21:36:35.062276+0000","last_active":"2024-03-11T21:38:48.758511+0000","last_peered":"2024-03-11T21:38:48.758511+0000","last_clean":"2024-03-11T21:38:48.758511+0000","last_became_active":"2024-03-11T21:36:35.056619+0000","last_became_peered":"2024-03-11T21:36:35.056619+0000","last_unstale":"2024-03-11T21:38:48.758511+0000","last_undegraded":"2024-03-11T21:38:48.758511+0000","last_fullsized":"2024-03-11T21:38:48.758511+0000","mapping_epoch":277,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":278,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:35:13.060487+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:31:15.367898+0000","last_clean_scrub_stamp":"2024-03-11T21:35:13.060487+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.14","version":"0'0","reported_seq":595,"reported_epoch":324,"state":"active+clean","last_fresh":"2024-03-11T21:38:43.363244+0000","last_change":"2024-03-11T21:38:35.999365+0000","last_active":"2024-03-11T21:38:43.363244+0000","last_peered":"2024-03-11T21:38:43.363244+0000","last_clean":"2024-03-11T21:38:43.363244+0000","last_became_active":"2024-03-11T21:36:35.068038+0000","last_became_peered":"2024-03-11T21:36:35.068038+0000","last_unstale":"2024-03-11T21:38:43.363244+0000","last_undegraded":"2024-03-11T21:38:43.363244+0000","last_fullsized":"2024-03-11T21:38:43.363244+0000","mapping_epoch":277,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":278,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:38:35.999231+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:37.512018+0000","last_clean_scrub_stamp":"2024-03-11T21:38:35.999231+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:40:00.642531+0000","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,5],"acting":[6,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.12","version":"260'15","reported_seq":591,"reported_epoch":324,"state":"active+clean","last_fresh":"2024-03-11T21:38:43.418442+0000","last_change":"2024-03-11T21:35:32.590439+0000","last_active":"2024-03-11T21:38:43.418442+0000","last_peered":"2024-03-11T21:38:43.418442+0000","last_clean":"2024-03-11T21:38:43.418442+0000","last_became_active":"2024-03-11T21:25:40.547082+0000","last_became_peered":"2024-03-11T21:25:40.547082+0000","last_unstale":"2024-03-11T21:38:43.418442+0000","last_undegraded":"2024-03-11T21:38:43.418442+0000","last_fullsized":"2024-03-11T21:38:43.418442+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'9","last_scrub_stamp":"2024-03-11T21:35:32.590318+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:39.525554+0000","last_clean_scrub_stamp":"2024-03-11T21:35:32.590318+0000","objects_scrubbed":4,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":558,"reported_epoch":324,"state":"active+clean","last_fresh":"2024-03-11T21:38:43.363275+0000","last_change":"2024-03-11T21:36:04.754168+0000","last_active":"2024-03-11T21:38:43.363275+0000","last_peered":"2024-03-11T21:38:43.363275+0000","last_clean":"2024-03-11T21:38:43.363275+0000","last_became_active":"2024-03-11T21:25:44.589586+0000","last_became_peered":"2024-03-11T21:25:44.589586+0000","last_unstale":"2024-03-11T21:38:43.363275+0000","last_undegraded":"2024-03-11T21:38:43.363275+0000","last_fullsized":"2024-03-11T21:38:43.363275+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:36:04.753979+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:27:09.801138+0000","last_clean_scrub_stamp":"2024-03-11T21:36:04.753979+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":113,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,5],"acting":[4,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.14","version":"280'12","reported_seq":683,"reported_epoch":325,"state":"active+clean","last_fresh":"2024-03-11T21:38:48.757600+0000","last_change":"2024-03-11T21:36:35.074550+0000","last_active":"2024-03-11T21:38:48.757600+0000","last_peered":"2024-03-11T21:38:48.757600+0000","last_clean":"2024-03-11T21:38:48.757600+0000","last_became_active":"2024-03-11T21:36:35.073912+0000","last_became_peered":"2024-03-11T21:36:35.073912+0000","last_unstale":"2024-03-11T21:38:48.757600+0000","last_undegraded":"2024-03-11T21:38:48.757600+0000","last_fullsized":"2024-03-11T21:38:48.757600+0000","mapping_epoch":277,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":278,"parent":"0.0","parent_split_bits":0,"last_scrub":"229'10","last_scrub_stamp":"2024-03-11T21:35:26.182107+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:41.544550+0000","last_clean_scrub_stamp":"2024-03-11T21:35:26.182107+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,2],"acting":[3,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.13","version":"0'0","reported_seq":571,"reported_epoch":324,"state":"active+clean","last_fresh":"2024-03-11T21:38:43.406577+0000","last_change":"2024-03-11T21:36:35.057688+0000","last_active":"2024-03-11T21:38:43.406577+0000","last_peered":"2024-03-11T21:38:43.406577+0000","last_clean":"2024-03-11T21:38:43.406577+0000","last_became_active":"2024-03-11T21:36:35.057132+0000","last_became_peered":"2024-03-11T21:36:35.057132+0000","last_unstale":"2024-03-11T21:38:43.406577+0000","last_undegraded":"2024-03-11T21:38:43.406577+0000","last_fullsized":"2024-03-11T21:38:43.406577+0000","mapping_epoch":277,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":278,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:34:08.650371+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:37.512018+0000","last_clean_scrub_stamp":"2024-03-11T21:34:08.650371+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.15","version":"260'19","reported_seq":644,"reported_epoch":324,"state":"active+clean","last_fresh":"2024-03-11T21:38:43.359194+0000","last_change":"2024-03-11T21:36:35.088986+0000","last_active":"2024-03-11T21:38:43.359194+0000","last_peered":"2024-03-11T21:38:43.359194+0000","last_clean":"2024-03-11T21:38:43.359194+0000","last_became_active":"2024-03-11T21:36:35.057285+0000","last_became_peered":"2024-03-11T21:36:35.057285+0000","last_unstale":"2024-03-11T21:38:43.359194+0000","last_undegraded":"2024-03-11T21:38:43.359194+0000","last_fullsized":"2024-03-11T21:38:43.359194+0000","mapping_epoch":277,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":278,"parent":"0.0","parent_split_bits":0,"last_scrub":"94'13","last_scrub_stamp":"2024-03-11T21:35:29.178543+0000","last_deep_scrub":"94'13","last_deep_scrub_stamp":"2024-03-11T21:29:41.853158+0000","last_clean_scrub_stamp":"2024-03-11T21:35:29.178543+0000","objects_scrubbed":4,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":11,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.16","version":"0'0","reported_seq":579,"reported_epoch":324,"state":"active+clean","last_fresh":"2024-03-11T21:38:43.407059+0000","last_change":"2024-03-11T21:36:35.058372+0000","last_active":"2024-03-11T21:38:43.407059+0000","last_peered":"2024-03-11T21:38:43.407059+0000","last_clean":"2024-03-11T21:38:43.407059+0000","last_became_active":"2024-03-11T21:36:35.057355+0000","last_became_peered":"2024-03-11T21:36:35.057355+0000","last_unstale":"2024-03-11T21:38:43.407059+0000","last_undegraded":"2024-03-11T21:38:43.407059+0000","last_fullsized":"2024-03-11T21:38:43.407059+0000","mapping_epoch":277,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":278,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:35:07.665075+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:43.559452+0000","last_clean_scrub_stamp":"2024-03-11T21:35:07.665075+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":35,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.15","version":"98'9","reported_seq":728,"reported_epoch":324,"state":"active+clean","last_fresh":"2024-03-11T21:38:43.359322+0000","last_change":"2024-03-11T21:36:03.366581+0000","last_active":"2024-03-11T21:38:43.359322+0000","last_peered":"2024-03-11T21:38:43.359322+0000","last_clean":"2024-03-11T21:38:43.359322+0000","last_became_active":"2024-03-11T21:25:42.572555+0000","last_became_peered":"2024-03-11T21:25:42.572555+0000","last_unstale":"2024-03-11T21:38:43.359322+0000","last_undegraded":"2024-03-11T21:38:43.359322+0000","last_fullsized":"2024-03-11T21:38:43.359322+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"98'9","last_scrub_stamp":"2024-03-11T21:36:03.366450+0000","last_deep_scrub":"98'9","last_deep_scrub_stamp":"2024-03-11T21:31:20.624923+0000","last_clean_scrub_stamp":"2024-03-11T21:36:03.366450+0000","objects_scrubbed":1,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.12","version":"0'0","reported_seq":590,"reported_epoch":324,"state":"active+clean","last_fresh":"2024-03-11T21:38:43.357996+0000","last_change":"2024-03-11T21:36:35.058369+0000","last_active":"2024-03-11T21:38:43.357996+0000","last_peered":"2024-03-11T21:38:43.357996+0000","last_clean":"2024-03-11T21:38:43.357996+0000","last_became_active":"2024-03-11T21:36:35.058033+0000","last_became_peered":"2024-03-11T21:36:35.058033+0000","last_unstale":"2024-03-11T21:38:43.357996+0000","last_undegraded":"2024-03-11T21:38:43.357996+0000","last_fullsized":"2024-03-11T21:38:43.357996+0000","mapping_epoch":277,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":278,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:35:27.158468+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:37.512018+0000","last_clean_scrub_stamp":"2024-03-11T21:35:27.158468+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.14","version":"260'20","reported_seq":577,"reported_epoch":325,"state":"active+clean","last_fresh":"2024-03-11T21:38:48.757197+0000","last_change":"2024-03-11T21:36:35.095382+0000","last_active":"2024-03-11T21:38:48.757197+0000","last_peered":"2024-03-11T21:38:48.757197+0000","last_clean":"2024-03-11T21:38:48.757197+0000","last_became_active":"2024-03-11T21:36:35.069638+0000","last_became_peered":"2024-03-11T21:36:35.069638+0000","last_unstale":"2024-03-11T21:38:48.757197+0000","last_undegraded":"2024-03-11T21:38:48.757197+0000","last_fullsized":"2024-03-11T21:38:48.757197+0000","mapping_epoch":277,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":278,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'10","last_scrub_stamp":"2024-03-11T21:35:28.189296+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:39.525554+0000","last_clean_scrub_stamp":"2024-03-11T21:35:28.189296+0000","objects_scrubbed":0,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":29,"num_read_kb":24,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.15","version":"0'0","reported_seq":214,"reported_epoch":324,"state":"active+clean","last_fresh":"2024-03-11T21:38:43.354818+0000","last_change":"2024-03-11T21:36:14.736240+0000","last_active":"2024-03-11T21:38:43.354818+0000","last_peered":"2024-03-11T21:38:43.354818+0000","last_clean":"2024-03-11T21:38:43.354818+0000","last_became_active":"2024-03-11T21:34:41.142760+0000","last_became_peered":"2024-03-11T21:34:41.142760+0000","last_unstale":"2024-03-11T21:38:43.354818+0000","last_undegraded":"2024-03-11T21:38:43.354818+0000","last_fullsized":"2024-03-11T21:38:43.354818+0000","mapping_epoch":226,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":227,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:36:14.735985+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:43.559452+0000","last_clean_scrub_stamp":"2024-03-11T21:36:14.735985+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":36,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":562,"reported_epoch":324,"state":"active+clean","last_fresh":"2024-03-11T21:38:43.358835+0000","last_change":"2024-03-11T21:36:35.060666+0000","last_active":"2024-03-11T21:38:43.358835+0000","last_peered":"2024-03-11T21:38:43.358835+0000","last_clean":"2024-03-11T21:38:43.358835+0000","last_became_active":"2024-03-11T21:36:35.060342+0000","last_became_peered":"2024-03-11T21:36:35.060342+0000","last_unstale":"2024-03-11T21:38:43.358835+0000","last_undegraded":"2024-03-11T21:38:43.358835+0000","last_fullsized":"2024-03-11T21:38:43.358835+0000","mapping_epoch":277,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":278,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:34:09.244852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:41.544550+0000","last_clean_scrub_stamp":"2024-03-11T21:34:09.244852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,1],"acting":[5,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.11","version":"0'0","reported_seq":570,"reported_epoch":324,"state":"active+clean","last_fresh":"2024-03-11T21:38:43.363968+0000","last_change":"2024-03-11T21:35:08.152853+0000","last_active":"2024-03-11T21:38:43.363968+0000","last_peered":"2024-03-11T21:38:43.363968+0000","last_clean":"2024-03-11T21:38:43.363968+0000","last_became_active":"2024-03-11T21:25:38.537415+0000","last_became_peered":"2024-03-11T21:25:38.537415+0000","last_unstale":"2024-03-11T21:38:43.363968+0000","last_undegraded":"2024-03-11T21:38:43.363968+0000","last_fullsized":"2024-03-11T21:38:43.363968+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:35:08.152740+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:37.512018+0000","last_clean_scrub_stamp":"2024-03-11T21:35:08.152740+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.17","version":"260'15","reported_seq":522,"reported_epoch":325,"state":"active+clean","last_fresh":"2024-03-11T21:38:48.759348+0000","last_change":"2024-03-11T21:36:35.099451+0000","last_active":"2024-03-11T21:38:48.759348+0000","last_peered":"2024-03-11T21:38:48.759348+0000","last_clean":"2024-03-11T21:38:48.759348+0000","last_became_active":"2024-03-11T21:36:35.071346+0000","last_became_peered":"2024-03-11T21:36:35.071346+0000","last_unstale":"2024-03-11T21:38:48.759348+0000","last_undegraded":"2024-03-11T21:38:48.759348+0000","last_fullsized":"2024-03-11T21:38:48.759348+0000","mapping_epoch":277,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":278,"parent":"0.0","parent_split_bits":0,"last_scrub":"97'9","last_scrub_stamp":"2024-03-11T21:32:08.495352+0000","last_deep_scrub":"97'9","last_deep_scrub_stamp":"2024-03-11T21:27:25.803132+0000","last_clean_scrub_stamp":"2024-03-11T21:32:08.495352+0000","objects_scrubbed":3,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":17,"num_read_kb":14,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,5],"acting":[3,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.14","version":"0'0","reported_seq":569,"reported_epoch":324,"state":"active+clean","last_fresh":"2024-03-11T21:38:43.362452+0000","last_change":"2024-03-11T21:35:27.606664+0000","last_active":"2024-03-11T21:38:43.362452+0000","last_peered":"2024-03-11T21:38:43.362452+0000","last_clean":"2024-03-11T21:38:43.362452+0000","last_became_active":"2024-03-11T21:34:41.136946+0000","last_became_peered":"2024-03-11T21:34:41.136946+0000","last_unstale":"2024-03-11T21:38:43.362452+0000","last_undegraded":"2024-03-11T21:38:43.362452+0000","last_fullsized":"2024-03-11T21:38:43.362452+0000","mapping_epoch":226,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":227,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:35:27.606551+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:43.559452+0000","last_clean_scrub_stamp":"2024-03-11T21:35:27.606551+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,7],"acting":[2,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":476,"reported_epoch":325,"state":"active+clean","last_fresh":"2024-03-11T21:38:48.758625+0000","last_change":"2024-03-11T21:36:35.076782+0000","last_active":"2024-03-11T21:38:48.758625+0000","last_peered":"2024-03-11T21:38:48.758625+0000","last_clean":"2024-03-11T21:38:48.758625+0000","last_became_active":"2024-03-11T21:36:35.076335+0000","last_became_peered":"2024-03-11T21:36:35.076335+0000","last_unstale":"2024-03-11T21:38:48.758625+0000","last_undegraded":"2024-03-11T21:38:48.758625+0000","last_fullsized":"2024-03-11T21:38:48.758625+0000","mapping_epoch":277,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":278,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:34:33.082577+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:41.544550+0000","last_clean_scrub_stamp":"2024-03-11T21:34:33.082577+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":35,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.10","version":"0'0","reported_seq":569,"reported_epoch":324,"state":"active+clean","last_fresh":"2024-03-11T21:38:43.362285+0000","last_change":"2024-03-11T21:35:37.694623+0000","last_active":"2024-03-11T21:38:43.362285+0000","last_peered":"2024-03-11T21:38:43.362285+0000","last_clean":"2024-03-11T21:38:43.362285+0000","last_became_active":"2024-03-11T21:25:38.533138+0000","last_became_peered":"2024-03-11T21:25:38.533138+0000","last_unstale":"2024-03-11T21:38:43.362285+0000","last_undegraded":"2024-03-11T21:38:43.362285+0000","last_fullsized":"2024-03-11T21:38:43.362285+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:35:37.694533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:37.512018+0000","last_clean_scrub_stamp":"2024-03-11T21:35:37.694533+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,0],"acting":[2,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.16","version":"260'15","reported_seq":641,"reported_epoch":324,"state":"active+clean","last_fresh":"2024-03-11T21:38:43.407245+0000","last_change":"2024-03-11T21:36:35.080158+0000","last_active":"2024-03-11T21:38:43.407245+0000","last_peered":"2024-03-11T21:38:43.407245+0000","last_clean":"2024-03-11T21:38:43.407245+0000","last_became_active":"2024-03-11T21:36:35.056862+0000","last_became_peered":"2024-03-11T21:36:35.056862+0000","last_unstale":"2024-03-11T21:38:43.407245+0000","last_undegraded":"2024-03-11T21:38:43.407245+0000","last_fullsized":"2024-03-11T21:38:43.407245+0000","mapping_epoch":277,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":278,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'9","last_scrub_stamp":"2024-03-11T21:35:12.741225+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:39.525554+0000","last_clean_scrub_stamp":"2024-03-11T21:35:12.741225+0000","objects_scrubbed":4,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":11,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.4","version":"324'11306","reported_seq":12917,"reported_epoch":324,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-03-11T21:38:48.003799+0000","last_change":"2024-03-11T21:38:21.285644+0000","last_active":"2024-03-11T21:38:48.003799+0000","last_peered":"2024-03-11T21:38:48.003799+0000","last_clean":"2024-03-11T21:35:44.669493+0000","last_became_active":"2024-03-11T21:36:36.506468+0000","last_became_peered":"2024-03-11T21:36:36.506468+0000","last_unstale":"2024-03-11T21:38:48.003799+0000","last_undegraded":"2024-03-11T21:36:36.069183+0000","last_fullsized":"2024-03-11T21:36:36.038484+0000","mapping_epoch":278,"log_start":"289'6300","ondisk_log_start":"289'6300","created":259,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:35:44.669493+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:35:44.669493+0000","last_clean_scrub_stamp":"2024-03-11T21:35:44.669493+0000","objects_scrubbed":0,"log_size":5006,"log_dups_size":0,"ondisk_log_size":5006,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":226099200,"num_objects":3450,"num_object_clones":0,"num_object_copies":10350,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4080,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3450,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11306,"num_write_kb":472192,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":736,"num_bytes_recovered":48234496,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,3],"acting":[1,4],"avail_no_missing":["1","4"],"object_location_counts":[{"shards":"1,4","objects":-630}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.b","version":"0'0","reported_seq":465,"reported_epoch":325,"state":"active+clean","last_fresh":"2024-03-11T21:38:48.758296+0000","last_change":"2024-03-11T21:36:35.074815+0000","last_active":"2024-03-11T21:38:48.758296+0000","last_peered":"2024-03-11T21:38:48.758296+0000","last_clean":"2024-03-11T21:38:48.758296+0000","last_became_active":"2024-03-11T21:36:35.072147+0000","last_became_peered":"2024-03-11T21:36:35.072147+0000","last_unstale":"2024-03-11T21:38:48.758296+0000","last_undegraded":"2024-03-11T21:38:48.758296+0000","last_fullsized":"2024-03-11T21:38:48.758296+0000","mapping_epoch":277,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":278,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:32:22.598150+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:43.559452+0000","last_clean_scrub_stamp":"2024-03-11T21:32:22.598150+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":57,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.f","version":"67'2","reported_seq":582,"reported_epoch":324,"state":"active+clean","last_fresh":"2024-03-11T21:38:43.363048+0000","last_change":"2024-03-11T21:35:36.613667+0000","last_active":"2024-03-11T21:38:43.363048+0000","last_peered":"2024-03-11T21:38:43.363048+0000","last_clean":"2024-03-11T21:38:43.363048+0000","last_became_active":"2024-03-11T21:25:38.537711+0000","last_became_peered":"2024-03-11T21:25:38.537711+0000","last_unstale":"2024-03-11T21:38:43.363048+0000","last_undegraded":"2024-03-11T21:38:43.363048+0000","last_fullsized":"2024-03-11T21:38:43.363048+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'2","last_scrub_stamp":"2024-03-11T21:35:36.613516+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:37.512018+0000","last_clean_scrub_stamp":"2024-03-11T21:35:36.613516+0000","objects_scrubbed":2,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":92,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":10,"num_read_kb":10,"num_write":4,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,7],"acting":[4,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.9","version":"260'23","reported_seq":614,"reported_epoch":324,"state":"active+clean","last_fresh":"2024-03-11T21:38:43.364699+0000","last_change":"2024-03-11T21:36:35.089034+0000","last_active":"2024-03-11T21:38:43.364699+0000","last_peered":"2024-03-11T21:38:43.364699+0000","last_clean":"2024-03-11T21:38:43.364699+0000","last_became_active":"2024-03-11T21:36:35.059930+0000","last_became_peered":"2024-03-11T21:36:35.059930+0000","last_unstale":"2024-03-11T21:38:43.364699+0000","last_undegraded":"2024-03-11T21:38:43.364699+0000","last_fullsized":"2024-03-11T21:38:43.364699+0000","mapping_epoch":277,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":278,"parent":"0.0","parent_split_bits":0,"last_scrub":"146'17","last_scrub_stamp":"2024-03-11T21:34:26.765075+0000","last_deep_scrub":"146'17","last_deep_scrub_stamp":"2024-03-11T21:34:26.765075+0000","last_clean_scrub_stamp":"2024-03-11T21:34:26.765075+0000","objects_scrubbed":5,"log_size":23,"log_dups_size":0,"ondisk_log_size":23,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":34,"num_read_kb":25,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,3],"acting":[4,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":562,"reported_epoch":324,"state":"active+clean","last_fresh":"2024-03-11T21:38:43.362355+0000","last_change":"2024-03-11T21:35:16.643421+0000","last_active":"2024-03-11T21:38:43.362355+0000","last_peered":"2024-03-11T21:38:43.362355+0000","last_clean":"2024-03-11T21:38:43.362355+0000","last_became_active":"2024-03-11T21:25:42.573987+0000","last_became_peered":"2024-03-11T21:25:42.573987+0000","last_unstale":"2024-03-11T21:38:43.362355+0000","last_undegraded":"2024-03-11T21:38:43.362355+0000","last_fullsized":"2024-03-11T21:38:43.362355+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:35:16.643311+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:41.544550+0000","last_clean_scrub_stamp":"2024-03-11T21:35:16.643311+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"9.f","version":"324'11044","reported_seq":11187,"reported_epoch":324,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-03-11T21:38:47.429575+0000","last_change":"2024-03-11T21:36:36.519822+0000","last_active":"2024-03-11T21:38:47.429575+0000","last_peered":"2024-03-11T21:38:47.429575+0000","last_clean":"2024-03-11T21:35:44.669493+0000","last_became_active":"2024-03-11T21:36:36.508836+0000","last_became_peered":"2024-03-11T21:36:36.508836+0000","last_unstale":"2024-03-11T21:38:47.429575+0000","last_undegraded":"2024-03-11T21:36:36.176770+0000","last_fullsized":"2024-03-11T21:36:36.038651+0000","mapping_epoch":278,"log_start":"287'5800","ondisk_log_start":"287'5800","created":259,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:35:44.669493+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:35:44.669493+0000","last_clean_scrub_stamp":"2024-03-11T21:35:44.669493+0000","objects_scrubbed":0,"log_size":5244,"log_dups_size":0,"ondisk_log_size":5244,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":218103808,"num_objects":3328,"num_object_clones":0,"num_object_copies":9984,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4738,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3328,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11044,"num_write_kb":459904,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":65536,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3],"acting":[5,1],"avail_no_missing":["5","1"],"object_location_counts":[{"shards":"1,5","objects":-1410}],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.0","version":"0'0","reported_seq":560,"reported_epoch":324,"state":"active+clean","last_fresh":"2024-03-11T21:38:43.354817+0000","last_change":"2024-03-11T21:36:35.058390+0000","last_active":"2024-03-11T21:38:43.354817+0000","last_peered":"2024-03-11T21:38:43.354817+0000","last_clean":"2024-03-11T21:38:43.354817+0000","last_became_active":"2024-03-11T21:36:35.058054+0000","last_became_peered":"2024-03-11T21:36:35.058054+0000","last_unstale":"2024-03-11T21:38:43.354817+0000","last_undegraded":"2024-03-11T21:38:43.354817+0000","last_fullsized":"2024-03-11T21:38:43.354817+0000","mapping_epoch":277,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":278,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:35:04.626668+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:29:39.498784+0000","last_clean_scrub_stamp":"2024-03-11T21:35:04.626668+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":39,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":583,"reported_epoch":324,"state":"active+clean","last_fresh":"2024-03-11T21:38:43.419886+0000","last_change":"2024-03-11T21:35:33.608063+0000","last_active":"2024-03-11T21:38:43.419886+0000","last_peered":"2024-03-11T21:38:43.419886+0000","last_clean":"2024-03-11T21:38:43.419886+0000","last_became_active":"2024-03-11T21:34:41.150913+0000","last_became_peered":"2024-03-11T21:34:41.150913+0000","last_unstale":"2024-03-11T21:38:43.419886+0000","last_undegraded":"2024-03-11T21:38:43.419886+0000","last_fullsized":"2024-03-11T21:38:43.419886+0000","mapping_epoch":226,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":227,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:35:33.607969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:37.512018+0000","last_clean_scrub_stamp":"2024-03-11T21:35:33.607969+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,7],"acting":[1,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.2","version":"260'25","reported_seq":602,"reported_epoch":324,"state":"active+clean","last_fresh":"2024-03-11T21:38:43.418524+0000","last_change":"2024-03-11T21:35:17.666749+0000","last_active":"2024-03-11T21:38:43.418524+0000","last_peered":"2024-03-11T21:38:43.418524+0000","last_clean":"2024-03-11T21:38:43.418524+0000","last_became_active":"2024-03-11T21:25:40.548833+0000","last_became_peered":"2024-03-11T21:25:40.548833+0000","last_unstale":"2024-03-11T21:38:43.418524+0000","last_undegraded":"2024-03-11T21:38:43.418524+0000","last_fullsized":"2024-03-11T21:38:43.418524+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"94'15","last_scrub_stamp":"2024-03-11T21:35:17.666498+0000","last_deep_scrub":"94'15","last_deep_scrub_stamp":"2024-03-11T21:26:49.993060+0000","last_clean_scrub_stamp":"2024-03-11T21:35:17.666498+0000","objects_scrubbed":5,"log_size":25,"log_dups_size":0,"ondisk_log_size":25,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":20,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":25,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.3","version":"74'8","reported_seq":729,"reported_epoch":324,"state":"active+clean","last_fresh":"2024-03-11T21:38:43.406805+0000","last_change":"2024-03-11T21:35:31.695369+0000","last_active":"2024-03-11T21:38:43.406805+0000","last_peered":"2024-03-11T21:38:43.406805+0000","last_clean":"2024-03-11T21:38:43.406805+0000","last_became_active":"2024-03-11T21:25:42.578161+0000","last_became_peered":"2024-03-11T21:25:42.578161+0000","last_unstale":"2024-03-11T21:38:43.406805+0000","last_undegraded":"2024-03-11T21:38:43.406805+0000","last_fullsized":"2024-03-11T21:38:43.406805+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'8","last_scrub_stamp":"2024-03-11T21:35:31.695263+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:41.544550+0000","last_clean_scrub_stamp":"2024-03-11T21:35:31.695263+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,5],"acting":[0,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.9","version":"324'11361","reported_seq":11502,"reported_epoch":324,"state":"active+clean","last_fresh":"2024-03-11T21:38:48.543208+0000","last_change":"2024-03-11T21:35:45.696767+0000","last_active":"2024-03-11T21:38:48.543208+0000","last_peered":"2024-03-11T21:38:48.543208+0000","last_clean":"2024-03-11T21:38:48.543208+0000","last_became_active":"2024-03-11T21:35:45.696379+0000","last_became_peered":"2024-03-11T21:35:45.696379+0000","last_unstale":"2024-03-11T21:38:48.543208+0000","last_undegraded":"2024-03-11T21:38:48.543208+0000","last_fullsized":"2024-03-11T21:38:48.543208+0000","mapping_epoch":259,"log_start":"287'5800","ondisk_log_start":"287'5800","created":259,"last_epoch_clean":260,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:35:44.669493+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:35:44.669493+0000","last_clean_scrub_stamp":"2024-03-11T21:35:44.669493+0000","objects_scrubbed":0,"log_size":5561,"log_dups_size":0,"ondisk_log_size":5561,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":224722944,"num_objects":3429,"num_object_clones":0,"num_object_copies":10287,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3429,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11361,"num_write_kb":473280,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,5],"acting":[0,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":503,"reported_epoch":325,"state":"active+clean","last_fresh":"2024-03-11T21:38:48.756906+0000","last_change":"2024-03-11T21:36:35.068977+0000","last_active":"2024-03-11T21:38:48.756906+0000","last_peered":"2024-03-11T21:38:48.756906+0000","last_clean":"2024-03-11T21:38:48.756906+0000","last_became_active":"2024-03-11T21:36:35.067288+0000","last_became_peered":"2024-03-11T21:36:35.067288+0000","last_unstale":"2024-03-11T21:38:48.756906+0000","last_undegraded":"2024-03-11T21:38:48.756906+0000","last_fullsized":"2024-03-11T21:38:48.756906+0000","mapping_epoch":277,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":278,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:35:09.985377+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:28:30.563712+0000","last_clean_scrub_stamp":"2024-03-11T21:35:09.985377+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,7],"acting":[3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":578,"reported_epoch":324,"state":"active+clean","last_fresh":"2024-03-11T21:38:43.358042+0000","last_change":"2024-03-11T21:35:40.116979+0000","last_active":"2024-03-11T21:38:43.358042+0000","last_peered":"2024-03-11T21:38:43.358042+0000","last_clean":"2024-03-11T21:38:43.358042+0000","last_became_active":"2024-03-11T21:25:38.538540+0000","last_became_peered":"2024-03-11T21:25:38.538540+0000","last_unstale":"2024-03-11T21:38:43.358042+0000","last_undegraded":"2024-03-11T21:38:43.358042+0000","last_fullsized":"2024-03-11T21:38:43.358042+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:35:40.116877+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:37.512018+0000","last_clean_scrub_stamp":"2024-03-11T21:35:40.116877+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.4","version":"280'42","reported_seq":808,"reported_epoch":324,"state":"active+clean","last_fresh":"2024-03-11T21:38:43.419835+0000","last_change":"2024-03-11T21:36:35.095935+0000","last_active":"2024-03-11T21:38:43.419835+0000","last_peered":"2024-03-11T21:38:43.419835+0000","last_clean":"2024-03-11T21:38:43.419835+0000","last_became_active":"2024-03-11T21:36:35.073430+0000","last_became_peered":"2024-03-11T21:36:35.073430+0000","last_unstale":"2024-03-11T21:38:43.419835+0000","last_undegraded":"2024-03-11T21:38:43.419835+0000","last_fullsized":"2024-03-11T21:38:43.419835+0000","mapping_epoch":277,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":278,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'28","last_scrub_stamp":"2024-03-11T21:34:19.513252+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:39.525554+0000","last_clean_scrub_stamp":"2024-03-11T21:34:19.513252+0000","objects_scrubbed":10,"log_size":42,"log_dups_size":0,"ondisk_log_size":42,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":358,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":66,"num_read_kb":51,"num_write":38,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3],"acting":[1,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":563,"reported_epoch":324,"state":"active+clean","last_fresh":"2024-03-11T21:38:43.407289+0000","last_change":"2024-03-11T21:35:32.651731+0000","last_active":"2024-03-11T21:38:43.407289+0000","last_peered":"2024-03-11T21:38:43.407289+0000","last_clean":"2024-03-11T21:38:43.407289+0000","last_became_active":"2024-03-11T21:25:42.572673+0000","last_became_peered":"2024-03-11T21:25:42.572673+0000","last_unstale":"2024-03-11T21:38:43.407289+0000","last_undegraded":"2024-03-11T21:38:43.407289+0000","last_fullsized":"2024-03-11T21:38:43.407289+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:35:32.651652+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:41.544550+0000","last_clean_scrub_stamp":"2024-03-11T21:35:32.651652+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.a","version":"324'11275","reported_seq":11416,"reported_epoch":324,"state":"active+clean","last_fresh":"2024-03-11T21:38:47.498241+0000","last_change":"2024-03-11T21:35:45.698558+0000","last_active":"2024-03-11T21:38:47.498241+0000","last_peered":"2024-03-11T21:38:47.498241+0000","last_clean":"2024-03-11T21:38:47.498241+0000","last_became_active":"2024-03-11T21:35:45.698238+0000","last_became_peered":"2024-03-11T21:35:45.698238+0000","last_unstale":"2024-03-11T21:38:47.498241+0000","last_undegraded":"2024-03-11T21:38:47.498241+0000","last_fullsized":"2024-03-11T21:38:47.498241+0000","mapping_epoch":259,"log_start":"288'6100","ondisk_log_start":"288'6100","created":259,"last_epoch_clean":260,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:35:44.669493+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:35:44.669493+0000","last_clean_scrub_stamp":"2024-03-11T21:35:44.669493+0000","objects_scrubbed":0,"log_size":5175,"log_dups_size":0,"ondisk_log_size":5175,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":230227968,"num_objects":3513,"num_object_clones":0,"num_object_copies":10539,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3513,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11275,"num_write_kb":473216,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,7],"acting":[5,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.5","version":"0'0","reported_seq":223,"reported_epoch":324,"state":"active+clean","last_fresh":"2024-03-11T21:38:43.354302+0000","last_change":"2024-03-11T21:36:35.069989+0000","last_active":"2024-03-11T21:38:43.354302+0000","last_peered":"2024-03-11T21:38:43.354302+0000","last_clean":"2024-03-11T21:38:43.354302+0000","last_became_active":"2024-03-11T21:36:35.069358+0000","last_became_peered":"2024-03-11T21:36:35.069358+0000","last_unstale":"2024-03-11T21:38:43.354302+0000","last_undegraded":"2024-03-11T21:38:43.354302+0000","last_fullsized":"2024-03-11T21:38:43.354302+0000","mapping_epoch":277,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":278,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:35:19.841632+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:43.559452+0000","last_clean_scrub_stamp":"2024-03-11T21:35:19.841632+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,3],"acting":[7,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.0","version":"164'6","reported_seq":1364,"reported_epoch":324,"state":"active+clean","last_fresh":"2024-03-11T21:38:46.639454+0000","last_change":"2024-03-11T21:35:38.518243+0000","last_active":"2024-03-11T21:38:46.639454+0000","last_peered":"2024-03-11T21:38:46.639454+0000","last_clean":"2024-03-11T21:38:46.639454+0000","last_became_active":"2024-03-11T21:25:38.536890+0000","last_became_peered":"2024-03-11T21:25:38.536890+0000","last_unstale":"2024-03-11T21:38:46.639454+0000","last_undegraded":"2024-03-11T21:38:46.639454+0000","last_fullsized":"2024-03-11T21:38:46.639454+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"164'6","last_scrub_stamp":"2024-03-11T21:35:38.518097+0000","last_deep_scrub":"164'6","last_deep_scrub_stamp":"2024-03-11T21:32:17.597666+0000","last_clean_scrub_stamp":"2024-03-11T21:35:38.518097+0000","objects_scrubbed":1,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0.000753559,"stat_sum":{"num_bytes":389,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":782,"num_read_kb":777,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,6],"acting":[1,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":579,"reported_epoch":324,"state":"active+clean","last_fresh":"2024-03-11T21:38:43.362440+0000","last_change":"2024-03-11T21:36:35.057287+0000","last_active":"2024-03-11T21:38:43.362440+0000","last_peered":"2024-03-11T21:38:43.362440+0000","last_clean":"2024-03-11T21:38:43.362440+0000","last_became_active":"2024-03-11T21:36:35.056966+0000","last_became_peered":"2024-03-11T21:36:35.056966+0000","last_unstale":"2024-03-11T21:38:43.362440+0000","last_undegraded":"2024-03-11T21:38:43.362440+0000","last_fullsized":"2024-03-11T21:38:43.362440+0000","mapping_epoch":277,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":278,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:34:18.424010+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:26:48.997454+0000","last_clean_scrub_stamp":"2024-03-11T21:34:18.424010+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.7","version":"260'22","reported_seq":608,"reported_epoch":324,"state":"active+clean","last_fresh":"2024-03-11T21:38:43.419147+0000","last_change":"2024-03-11T21:35:37.508679+0000","last_active":"2024-03-11T21:38:43.419147+0000","last_peered":"2024-03-11T21:38:43.419147+0000","last_clean":"2024-03-11T21:38:43.419147+0000","last_became_active":"2024-03-11T21:25:40.541769+0000","last_became_peered":"2024-03-11T21:25:40.541769+0000","last_unstale":"2024-03-11T21:38:43.419147+0000","last_undegraded":"2024-03-11T21:38:43.419147+0000","last_fullsized":"2024-03-11T21:38:43.419147+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"97'18","last_scrub_stamp":"2024-03-11T21:35:37.508567+0000","last_deep_scrub":"97'18","last_deep_scrub_stamp":"2024-03-11T21:27:19.239404+0000","last_clean_scrub_stamp":"2024-03-11T21:35:37.508567+0000","objects_scrubbed":5,"log_size":22,"log_dups_size":0,"ondisk_log_size":22,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":36,"num_read_kb":25,"num_write":20,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":574,"reported_epoch":324,"state":"active+clean","last_fresh":"2024-03-11T21:38:43.408098+0000","last_change":"2024-03-11T21:35:22.632822+0000","last_active":"2024-03-11T21:38:43.408098+0000","last_peered":"2024-03-11T21:38:43.408098+0000","last_clean":"2024-03-11T21:38:43.408098+0000","last_became_active":"2024-03-11T21:34:41.143660+0000","last_became_peered":"2024-03-11T21:34:41.143660+0000","last_unstale":"2024-03-11T21:38:43.408098+0000","last_undegraded":"2024-03-11T21:38:43.408098+0000","last_fullsized":"2024-03-11T21:38:43.408098+0000","mapping_epoch":226,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":227,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:35:22.632707+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:41.544550+0000","last_clean_scrub_stamp":"2024-03-11T21:35:22.632707+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":18,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7],"acting":[2,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"9.8","version":"324'11357","reported_seq":11497,"reported_epoch":324,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-03-11T21:38:48.556272+0000","last_change":"2024-03-11T21:36:36.520071+0000","last_active":"2024-03-11T21:38:48.556272+0000","last_peered":"2024-03-11T21:38:48.556272+0000","last_clean":"2024-03-11T21:35:44.669493+0000","last_became_active":"2024-03-11T21:36:36.507652+0000","last_became_peered":"2024-03-11T21:36:36.507652+0000","last_unstale":"2024-03-11T21:38:48.556272+0000","last_undegraded":"2024-03-11T21:36:36.064628+0000","last_fullsized":"2024-03-11T21:36:36.036628+0000","mapping_epoch":278,"log_start":"287'5800","ondisk_log_start":"287'5800","created":259,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:35:44.669493+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:35:44.669493+0000","last_clean_scrub_stamp":"2024-03-11T21:35:44.669493+0000","objects_scrubbed":0,"log_size":5557,"log_dups_size":0,"ondisk_log_size":5557,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":222363648,"num_objects":3393,"num_object_clones":0,"num_object_copies":10179,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4846,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3393,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11357,"num_write_kb":472000,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[0,5],"avail_no_missing":["0","5"],"object_location_counts":[{"shards":"0,5","objects":-1453}],"blocked_by":[],"up_primary":3,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.7","version":"124'2","reported_seq":561,"reported_epoch":324,"state":"active+clean","last_fresh":"2024-03-11T21:38:43.363289+0000","last_change":"2024-03-11T21:36:35.059190+0000","last_active":"2024-03-11T21:38:43.363289+0000","last_peered":"2024-03-11T21:38:43.363289+0000","last_clean":"2024-03-11T21:38:43.363289+0000","last_became_active":"2024-03-11T21:36:35.058786+0000","last_became_peered":"2024-03-11T21:36:35.058786+0000","last_unstale":"2024-03-11T21:38:43.363289+0000","last_undegraded":"2024-03-11T21:38:43.363289+0000","last_fullsized":"2024-03-11T21:38:43.363289+0000","mapping_epoch":277,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":278,"parent":"0.0","parent_split_bits":0,"last_scrub":"124'2","last_scrub_stamp":"2024-03-11T21:35:16.414595+0000","last_deep_scrub":"124'2","last_deep_scrub_stamp":"2024-03-11T21:29:34.956908+0000","last_clean_scrub_stamp":"2024-03-11T21:35:16.414595+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":42,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,4],"acting":[5,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2","version":"71'2","reported_seq":524,"reported_epoch":325,"state":"active+clean","last_fresh":"2024-03-11T21:38:48.763324+0000","last_change":"2024-03-11T21:36:35.071218+0000","last_active":"2024-03-11T21:38:48.763324+0000","last_peered":"2024-03-11T21:38:48.763324+0000","last_clean":"2024-03-11T21:38:48.763324+0000","last_became_active":"2024-03-11T21:36:35.070783+0000","last_became_peered":"2024-03-11T21:36:35.070783+0000","last_unstale":"2024-03-11T21:38:48.763324+0000","last_undegraded":"2024-03-11T21:38:48.763324+0000","last_fullsized":"2024-03-11T21:38:48.763324+0000","mapping_epoch":277,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":278,"parent":"0.0","parent_split_bits":0,"last_scrub":"71'2","last_scrub_stamp":"2024-03-11T21:32:21.543053+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:37.512018+0000","last_clean_scrub_stamp":"2024-03-11T21:32:21.543053+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014206449999999999,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":588,"reported_epoch":324,"state":"active+clean","last_fresh":"2024-03-11T21:38:43.358324+0000","last_change":"2024-03-11T21:35:08.391825+0000","last_active":"2024-03-11T21:38:43.358324+0000","last_peered":"2024-03-11T21:38:43.358324+0000","last_clean":"2024-03-11T21:38:43.358324+0000","last_became_active":"2024-03-11T21:34:41.146672+0000","last_became_peered":"2024-03-11T21:34:41.146672+0000","last_unstale":"2024-03-11T21:38:43.358324+0000","last_undegraded":"2024-03-11T21:38:43.358324+0000","last_fullsized":"2024-03-11T21:38:43.358324+0000","mapping_epoch":226,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":227,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:35:08.391713+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:37.512018+0000","last_clean_scrub_stamp":"2024-03-11T21:35:08.391713+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"87'128","reported_seq":686,"reported_epoch":324,"state":"active+clean","last_fresh":"2024-03-11T21:38:43.353612+0000","last_change":"2024-03-11T21:35:21.798272+0000","last_active":"2024-03-11T21:38:43.353612+0000","last_peered":"2024-03-11T21:38:43.353612+0000","last_clean":"2024-03-11T21:38:43.353612+0000","last_became_active":"2024-03-11T21:34:44.164569+0000","last_became_peered":"2024-03-11T21:34:44.164569+0000","last_unstale":"2024-03-11T21:38:43.353612+0000","last_undegraded":"2024-03-11T21:38:43.353612+0000","last_fullsized":"2024-03-11T21:38:43.353612+0000","mapping_epoch":229,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":230,"parent":"0.0","parent_split_bits":0,"last_scrub":"87'128","last_scrub_stamp":"2024-03-11T21:35:21.798144+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:24.261974+0000","last_clean_scrub_stamp":"2024-03-11T21:35:21.798144+0000","objects_scrubbed":2,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":198,"num_read_kb":294,"num_write":179,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":2951840,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.5","version":"260'28","reported_seq":773,"reported_epoch":324,"state":"active+clean","last_fresh":"2024-03-11T21:38:43.362945+0000","last_change":"2024-03-11T21:36:14.026933+0000","last_active":"2024-03-11T21:38:43.362945+0000","last_peered":"2024-03-11T21:38:43.362945+0000","last_clean":"2024-03-11T21:38:43.362945+0000","last_became_active":"2024-03-11T21:25:40.761457+0000","last_became_peered":"2024-03-11T21:25:40.761457+0000","last_unstale":"2024-03-11T21:38:43.362945+0000","last_undegraded":"2024-03-11T21:38:43.362945+0000","last_fullsized":"2024-03-11T21:38:43.362945+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"260'28","last_scrub_stamp":"2024-03-11T21:36:14.026775+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:39.525554+0000","last_clean_scrub_stamp":"2024-03-11T21:36:14.026775+0000","objects_scrubbed":8,"log_size":28,"log_dups_size":0,"ondisk_log_size":28,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":154,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":43,"num_read_kb":33,"num_write":25,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":214,"reported_epoch":324,"state":"active+clean","last_fresh":"2024-03-11T21:38:43.354933+0000","last_change":"2024-03-11T21:35:24.841170+0000","last_active":"2024-03-11T21:38:43.354933+0000","last_peered":"2024-03-11T21:38:43.354933+0000","last_clean":"2024-03-11T21:38:43.354933+0000","last_became_active":"2024-03-11T21:34:41.139782+0000","last_became_peered":"2024-03-11T21:34:41.139782+0000","last_unstale":"2024-03-11T21:38:43.354933+0000","last_undegraded":"2024-03-11T21:38:43.354933+0000","last_fullsized":"2024-03-11T21:38:43.354933+0000","mapping_epoch":226,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":227,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:35:24.841041+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:32:21.266672+0000","last_clean_scrub_stamp":"2024-03-11T21:35:24.841041+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,5],"acting":[7,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.b","version":"324'11115","reported_seq":11256,"reported_epoch":324,"state":"active+clean","last_fresh":"2024-03-11T21:38:48.528773+0000","last_change":"2024-03-11T21:35:45.697213+0000","last_active":"2024-03-11T21:38:48.528773+0000","last_peered":"2024-03-11T21:38:48.528773+0000","last_clean":"2024-03-11T21:38:48.528773+0000","last_became_active":"2024-03-11T21:35:45.696869+0000","last_became_peered":"2024-03-11T21:35:45.696869+0000","last_unstale":"2024-03-11T21:38:48.528773+0000","last_undegraded":"2024-03-11T21:38:48.528773+0000","last_fullsized":"2024-03-11T21:38:48.528773+0000","mapping_epoch":259,"log_start":"285'5500","ondisk_log_start":"285'5500","created":259,"last_epoch_clean":260,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:35:44.669493+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:35:44.669493+0000","last_clean_scrub_stamp":"2024-03-11T21:35:44.669493+0000","objects_scrubbed":0,"log_size":5615,"log_dups_size":0,"ondisk_log_size":5615,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":218300416,"num_objects":3331,"num_object_clones":0,"num_object_copies":9993,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3331,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11115,"num_write_kb":462272,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,5],"acting":[0,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.4","version":"0'0","reported_seq":560,"reported_epoch":324,"state":"active+clean","last_fresh":"2024-03-11T21:38:43.420128+0000","last_change":"2024-03-11T21:36:35.073305+0000","last_active":"2024-03-11T21:38:43.420128+0000","last_peered":"2024-03-11T21:38:43.420128+0000","last_clean":"2024-03-11T21:38:43.420128+0000","last_became_active":"2024-03-11T21:36:35.072824+0000","last_became_peered":"2024-03-11T21:36:35.072824+0000","last_unstale":"2024-03-11T21:38:43.420128+0000","last_undegraded":"2024-03-11T21:38:43.420128+0000","last_fullsized":"2024-03-11T21:38:43.420128+0000","mapping_epoch":277,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":278,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:34:23.531064+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:43.559452+0000","last_clean_scrub_stamp":"2024-03-11T21:34:23.531064+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":587,"reported_epoch":324,"state":"active+clean","last_fresh":"2024-03-11T21:38:43.406740+0000","last_change":"2024-03-11T21:36:35.070626+0000","last_active":"2024-03-11T21:38:43.406740+0000","last_peered":"2024-03-11T21:38:43.406740+0000","last_clean":"2024-03-11T21:38:43.406740+0000","last_became_active":"2024-03-11T21:36:35.069452+0000","last_became_peered":"2024-03-11T21:36:35.069452+0000","last_unstale":"2024-03-11T21:38:43.406740+0000","last_undegraded":"2024-03-11T21:38:43.406740+0000","last_fullsized":"2024-03-11T21:38:43.406740+0000","mapping_epoch":277,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":278,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:35:13.741947+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:37.512018+0000","last_clean_scrub_stamp":"2024-03-11T21:35:13.741947+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0.00089342499999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":207,"reported_epoch":324,"state":"active+clean","last_fresh":"2024-03-11T21:38:43.353539+0000","last_change":"2024-03-11T21:34:41.150606+0000","last_active":"2024-03-11T21:38:43.353539+0000","last_peered":"2024-03-11T21:38:43.353539+0000","last_clean":"2024-03-11T21:38:43.353539+0000","last_became_active":"2024-03-11T21:34:41.149219+0000","last_became_peered":"2024-03-11T21:34:41.149219+0000","last_unstale":"2024-03-11T21:38:43.353539+0000","last_undegraded":"2024-03-11T21:38:43.353539+0000","last_fullsized":"2024-03-11T21:38:43.353539+0000","mapping_epoch":226,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":227,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:34:31.628014+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:37.512018+0000","last_clean_scrub_stamp":"2024-03-11T21:34:31.628014+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,0],"acting":[7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.6","version":"260'24","reported_seq":623,"reported_epoch":324,"state":"active+clean","last_fresh":"2024-03-11T21:38:43.407196+0000","last_change":"2024-03-11T21:35:19.753433+0000","last_active":"2024-03-11T21:38:43.407196+0000","last_peered":"2024-03-11T21:38:43.407196+0000","last_clean":"2024-03-11T21:38:43.407196+0000","last_became_active":"2024-03-11T21:32:22.796486+0000","last_became_peered":"2024-03-11T21:32:22.796486+0000","last_unstale":"2024-03-11T21:38:43.407196+0000","last_undegraded":"2024-03-11T21:38:43.407196+0000","last_fullsized":"2024-03-11T21:38:43.407196+0000","mapping_epoch":166,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":167,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'12","last_scrub_stamp":"2024-03-11T21:35:19.753224+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:39.525554+0000","last_clean_scrub_stamp":"2024-03-11T21:35:19.753224+0000","objects_scrubbed":6,"log_size":24,"log_dups_size":0,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":36,"num_read_kb":30,"num_write":24,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":562,"reported_epoch":324,"state":"active+clean","last_fresh":"2024-03-11T21:38:43.363186+0000","last_change":"2024-03-11T21:36:07.451482+0000","last_active":"2024-03-11T21:38:43.363186+0000","last_peered":"2024-03-11T21:38:43.363186+0000","last_clean":"2024-03-11T21:38:43.363186+0000","last_became_active":"2024-03-11T21:25:42.572324+0000","last_became_peered":"2024-03-11T21:25:42.572324+0000","last_unstale":"2024-03-11T21:38:43.363186+0000","last_undegraded":"2024-03-11T21:38:43.363186+0000","last_fullsized":"2024-03-11T21:38:43.363186+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:36:07.451102+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:41.544550+0000","last_clean_scrub_stamp":"2024-03-11T21:36:07.451102+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":54,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.e","version":"324'10904","reported_seq":11045,"reported_epoch":324,"state":"active+clean","last_fresh":"2024-03-11T21:38:47.339893+0000","last_change":"2024-03-11T21:35:45.697331+0000","last_active":"2024-03-11T21:38:47.339893+0000","last_peered":"2024-03-11T21:38:47.339893+0000","last_clean":"2024-03-11T21:38:47.339893+0000","last_became_active":"2024-03-11T21:35:45.696948+0000","last_became_peered":"2024-03-11T21:35:45.696948+0000","last_unstale":"2024-03-11T21:38:47.339893+0000","last_undegraded":"2024-03-11T21:38:47.339893+0000","last_fullsized":"2024-03-11T21:38:47.339893+0000","mapping_epoch":259,"log_start":"290'6200","ondisk_log_start":"290'6200","created":259,"last_epoch_clean":260,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:35:44.669493+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:35:44.669493+0000","last_clean_scrub_stamp":"2024-03-11T21:35:44.669493+0000","objects_scrubbed":0,"log_size":4704,"log_dups_size":0,"ondisk_log_size":4704,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":221249536,"num_objects":3376,"num_object_clones":0,"num_object_copies":10128,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3376,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10904,"num_write_kb":456960,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":567,"reported_epoch":324,"state":"active+clean","last_fresh":"2024-03-11T21:38:43.418322+0000","last_change":"2024-03-11T21:36:18.529506+0000","last_active":"2024-03-11T21:38:43.418322+0000","last_peered":"2024-03-11T21:38:43.418322+0000","last_clean":"2024-03-11T21:38:43.418322+0000","last_became_active":"2024-03-11T21:25:45.301416+0000","last_became_peered":"2024-03-11T21:25:45.301416+0000","last_unstale":"2024-03-11T21:38:43.418322+0000","last_undegraded":"2024-03-11T21:38:43.418322+0000","last_fullsized":"2024-03-11T21:38:43.418322+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:36:18.529370+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:43.559452+0000","last_clean_scrub_stamp":"2024-03-11T21:36:18.529370+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":54,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":214,"reported_epoch":324,"state":"active+clean","last_fresh":"2024-03-11T21:38:43.354384+0000","last_change":"2024-03-11T21:35:33.740615+0000","last_active":"2024-03-11T21:38:43.354384+0000","last_peered":"2024-03-11T21:38:43.354384+0000","last_clean":"2024-03-11T21:38:43.354384+0000","last_became_active":"2024-03-11T21:34:41.152721+0000","last_became_peered":"2024-03-11T21:34:41.152721+0000","last_unstale":"2024-03-11T21:38:43.354384+0000","last_undegraded":"2024-03-11T21:38:43.354384+0000","last_fullsized":"2024-03-11T21:38:43.354384+0000","mapping_epoch":226,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":227,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:35:33.740538+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:26:57.333669+0000","last_clean_scrub_stamp":"2024-03-11T21:35:33.740538+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.3","version":"260'37","reported_seq":674,"reported_epoch":324,"state":"active+clean","last_fresh":"2024-03-11T21:38:43.406483+0000","last_change":"2024-03-11T21:35:11.697314+0000","last_active":"2024-03-11T21:38:43.406483+0000","last_peered":"2024-03-11T21:38:43.406483+0000","last_clean":"2024-03-11T21:38:43.406483+0000","last_became_active":"2024-03-11T21:34:41.145607+0000","last_became_peered":"2024-03-11T21:34:41.145607+0000","last_unstale":"2024-03-11T21:38:43.406483+0000","last_undegraded":"2024-03-11T21:38:43.406483+0000","last_fullsized":"2024-03-11T21:38:43.406483+0000","mapping_epoch":226,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":227,"parent":"0.0","parent_split_bits":0,"last_scrub":"165'27","last_scrub_stamp":"2024-03-11T21:35:11.697140+0000","last_deep_scrub":"165'27","last_deep_scrub_stamp":"2024-03-11T21:32:21.576732+0000","last_clean_scrub_stamp":"2024-03-11T21:35:11.697140+0000","objects_scrubbed":8,"log_size":37,"log_dups_size":0,"ondisk_log_size":37,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":54,"num_read_kb":40,"num_write":32,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":660,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":571,"reported_epoch":324,"state":"active+clean","last_fresh":"2024-03-11T21:38:43.362758+0000","last_change":"2024-03-11T21:35:12.091317+0000","last_active":"2024-03-11T21:38:43.362758+0000","last_peered":"2024-03-11T21:38:43.362758+0000","last_clean":"2024-03-11T21:38:43.362758+0000","last_became_active":"2024-03-11T21:25:42.579045+0000","last_became_peered":"2024-03-11T21:25:42.579045+0000","last_unstale":"2024-03-11T21:38:43.362758+0000","last_undegraded":"2024-03-11T21:38:43.362758+0000","last_fullsized":"2024-03-11T21:38:43.362758+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:35:12.091185+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:29:36.918936+0000","last_clean_scrub_stamp":"2024-03-11T21:35:12.091185+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.d","version":"324'11067","reported_seq":11209,"reported_epoch":324,"state":"active+clean","last_fresh":"2024-03-11T21:38:47.343953+0000","last_change":"2024-03-11T21:35:45.700060+0000","last_active":"2024-03-11T21:38:47.343953+0000","last_peered":"2024-03-11T21:38:47.343953+0000","last_clean":"2024-03-11T21:38:47.343953+0000","last_became_active":"2024-03-11T21:35:45.699674+0000","last_became_peered":"2024-03-11T21:35:45.699674+0000","last_unstale":"2024-03-11T21:38:47.343953+0000","last_undegraded":"2024-03-11T21:38:47.343953+0000","last_fullsized":"2024-03-11T21:38:47.343953+0000","mapping_epoch":259,"log_start":"290'6400","ondisk_log_start":"290'6400","created":259,"last_epoch_clean":260,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:35:44.669493+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:35:44.669493+0000","last_clean_scrub_stamp":"2024-03-11T21:35:44.669493+0000","objects_scrubbed":0,"log_size":4667,"log_dups_size":0,"ondisk_log_size":4667,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":222691352,"num_objects":3399,"num_object_clones":0,"num_object_copies":10197,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3399,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":11067,"num_write_kb":462849,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,5],"acting":[7,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":565,"reported_epoch":324,"state":"active+clean","last_fresh":"2024-03-11T21:38:43.363417+0000","last_change":"2024-03-11T21:35:08.437783+0000","last_active":"2024-03-11T21:38:43.363417+0000","last_peered":"2024-03-11T21:38:43.363417+0000","last_clean":"2024-03-11T21:38:43.363417+0000","last_became_active":"2024-03-11T21:32:22.791439+0000","last_became_peered":"2024-03-11T21:32:22.791439+0000","last_unstale":"2024-03-11T21:38:43.363417+0000","last_undegraded":"2024-03-11T21:38:43.363417+0000","last_fullsized":"2024-03-11T21:38:43.363417+0000","mapping_epoch":166,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":167,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:35:08.437697+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:43.559452+0000","last_clean_scrub_stamp":"2024-03-11T21:35:08.437697+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":33,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,2],"acting":[4,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.6","version":"67'1","reported_seq":572,"reported_epoch":324,"state":"active+clean","last_fresh":"2024-03-11T21:38:43.419661+0000","last_change":"2024-03-11T21:35:30.581783+0000","last_active":"2024-03-11T21:38:43.419661+0000","last_peered":"2024-03-11T21:38:43.419661+0000","last_clean":"2024-03-11T21:38:43.419661+0000","last_became_active":"2024-03-11T21:25:38.536142+0000","last_became_peered":"2024-03-11T21:25:38.536142+0000","last_unstale":"2024-03-11T21:38:43.419661+0000","last_undegraded":"2024-03-11T21:38:43.419661+0000","last_fullsized":"2024-03-11T21:38:43.419661+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'1","last_scrub_stamp":"2024-03-11T21:35:30.581701+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:37.512018+0000","last_clean_scrub_stamp":"2024-03-11T21:35:30.581701+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,4],"acting":[1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.0","version":"260'38","reported_seq":572,"reported_epoch":325,"state":"active+clean","last_fresh":"2024-03-11T21:38:48.777592+0000","last_change":"2024-03-11T21:36:35.087771+0000","last_active":"2024-03-11T21:38:48.777592+0000","last_peered":"2024-03-11T21:38:48.777592+0000","last_clean":"2024-03-11T21:38:48.777592+0000","last_became_active":"2024-03-11T21:36:35.068071+0000","last_became_peered":"2024-03-11T21:36:35.068071+0000","last_unstale":"2024-03-11T21:38:48.777592+0000","last_undegraded":"2024-03-11T21:38:48.777592+0000","last_fullsized":"2024-03-11T21:38:48.777592+0000","mapping_epoch":277,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":278,"parent":"0.0","parent_split_bits":0,"last_scrub":"93'26","last_scrub_stamp":"2024-03-11T21:34:16.863984+0000","last_deep_scrub":"93'26","last_deep_scrub_stamp":"2024-03-11T21:26:40.680324+0000","last_clean_scrub_stamp":"2024-03-11T21:34:16.863984+0000","objects_scrubbed":8,"log_size":38,"log_dups_size":0,"ondisk_log_size":38,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":22,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":51,"num_read_kb":39,"num_write":32,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":584,"reported_epoch":324,"state":"active+clean","last_fresh":"2024-03-11T21:38:43.364737+0000","last_change":"2024-03-11T21:36:35.057830+0000","last_active":"2024-03-11T21:38:43.364737+0000","last_peered":"2024-03-11T21:38:43.364737+0000","last_clean":"2024-03-11T21:38:43.364737+0000","last_became_active":"2024-03-11T21:36:35.057517+0000","last_became_peered":"2024-03-11T21:36:35.057517+0000","last_unstale":"2024-03-11T21:38:43.364737+0000","last_undegraded":"2024-03-11T21:38:43.364737+0000","last_fullsized":"2024-03-11T21:38:43.364737+0000","mapping_epoch":277,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":278,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:35:12.411136+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:26:51.863621+0000","last_clean_scrub_stamp":"2024-03-11T21:35:12.411136+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,7],"acting":[4,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.c","version":"325'10886","reported_seq":20293,"reported_epoch":325,"state":"active+clean","last_fresh":"2024-03-11T21:38:48.768845+0000","last_change":"2024-03-11T21:38:30.644319+0000","last_active":"2024-03-11T21:38:48.768845+0000","last_peered":"2024-03-11T21:38:48.768845+0000","last_clean":"2024-03-11T21:38:48.768845+0000","last_became_active":"2024-03-11T21:38:30.610971+0000","last_became_peered":"2024-03-11T21:38:30.610971+0000","last_unstale":"2024-03-11T21:38:48.768845+0000","last_undegraded":"2024-03-11T21:38:48.768845+0000","last_fullsized":"2024-03-11T21:38:48.768845+0000","mapping_epoch":321,"log_start":"289'6044","ondisk_log_start":"289'6044","created":259,"last_epoch_clean":322,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:35:44.669493+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:35:44.669493+0000","last_clean_scrub_stamp":"2024-03-11T21:35:44.669493+0000","objects_scrubbed":0,"log_size":4842,"log_dups_size":0,"ondisk_log_size":4842,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":218628096,"num_objects":3336,"num_object_clones":0,"num_object_copies":10008,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3336,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10886,"num_write_kb":455104,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4628,"num_bytes_recovered":303300608,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,7],"acting":[3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.3","version":"0'0","reported_seq":206,"reported_epoch":324,"state":"active+clean","last_fresh":"2024-03-11T21:38:43.355993+0000","last_change":"2024-03-11T21:34:41.143973+0000","last_active":"2024-03-11T21:38:43.355993+0000","last_peered":"2024-03-11T21:38:43.355993+0000","last_clean":"2024-03-11T21:38:43.355993+0000","last_became_active":"2024-03-11T21:34:41.142683+0000","last_became_peered":"2024-03-11T21:34:41.142683+0000","last_unstale":"2024-03-11T21:38:43.355993+0000","last_undegraded":"2024-03-11T21:38:43.355993+0000","last_fullsized":"2024-03-11T21:38:43.355993+0000","mapping_epoch":226,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":227,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:34:27.353700+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:29:42.485740+0000","last_clean_scrub_stamp":"2024-03-11T21:34:27.353700+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,2],"acting":[7,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":591,"reported_epoch":324,"state":"active+clean","last_fresh":"2024-03-11T21:38:43.363184+0000","last_change":"2024-03-11T21:35:32.155983+0000","last_active":"2024-03-11T21:38:43.363184+0000","last_peered":"2024-03-11T21:38:43.363184+0000","last_clean":"2024-03-11T21:38:43.363184+0000","last_became_active":"2024-03-11T21:34:41.145628+0000","last_became_peered":"2024-03-11T21:34:41.145628+0000","last_unstale":"2024-03-11T21:38:43.363184+0000","last_undegraded":"2024-03-11T21:38:43.363184+0000","last_fullsized":"2024-03-11T21:38:43.363184+0000","mapping_epoch":226,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":227,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:35:32.155853+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:37.512018+0000","last_clean_scrub_stamp":"2024-03-11T21:35:32.155853+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,2],"acting":[6,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.1","version":"260'28","reported_seq":609,"reported_epoch":324,"state":"active+clean","last_fresh":"2024-03-11T21:38:43.364518+0000","last_change":"2024-03-11T21:35:33.644654+0000","last_active":"2024-03-11T21:38:43.364518+0000","last_peered":"2024-03-11T21:38:43.364518+0000","last_clean":"2024-03-11T21:38:43.364518+0000","last_became_active":"2024-03-11T21:25:40.546219+0000","last_became_peered":"2024-03-11T21:25:40.546219+0000","last_unstale":"2024-03-11T21:38:43.364518+0000","last_undegraded":"2024-03-11T21:38:43.364518+0000","last_fullsized":"2024-03-11T21:38:43.364518+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'14","last_scrub_stamp":"2024-03-11T21:35:33.644503+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:39.525554+0000","last_clean_scrub_stamp":"2024-03-11T21:35:33.644503+0000","objects_scrubbed":7,"log_size":28,"log_dups_size":0,"ondisk_log_size":28,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":42,"num_read_kb":35,"num_write":28,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6],"acting":[4,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.0","version":"280'11","reported_seq":658,"reported_epoch":325,"state":"active+clean","last_fresh":"2024-03-11T21:38:48.777329+0000","last_change":"2024-03-11T21:36:35.064085+0000","last_active":"2024-03-11T21:38:48.777329+0000","last_peered":"2024-03-11T21:38:48.777329+0000","last_clean":"2024-03-11T21:38:48.777329+0000","last_became_active":"2024-03-11T21:36:35.055250+0000","last_became_peered":"2024-03-11T21:36:35.055250+0000","last_unstale":"2024-03-11T21:38:48.777329+0000","last_undegraded":"2024-03-11T21:38:48.777329+0000","last_fullsized":"2024-03-11T21:38:48.777329+0000","mapping_epoch":277,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":278,"parent":"0.0","parent_split_bits":0,"last_scrub":"98'9","last_scrub_stamp":"2024-03-11T21:35:21.143578+0000","last_deep_scrub":"98'9","last_deep_scrub_stamp":"2024-03-11T21:27:29.745711+0000","last_clean_scrub_stamp":"2024-03-11T21:35:21.143578+0000","objects_scrubbed":1,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.3","version":"324'10950","reported_seq":11100,"reported_epoch":324,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-03-11T21:38:47.333766+0000","last_change":"2024-03-11T21:36:36.544317+0000","last_active":"2024-03-11T21:38:47.333766+0000","last_peered":"2024-03-11T21:38:47.333766+0000","last_clean":"2024-03-11T21:35:44.669493+0000","last_became_active":"2024-03-11T21:36:36.470537+0000","last_became_peered":"2024-03-11T21:36:36.470537+0000","last_unstale":"2024-03-11T21:38:47.333766+0000","last_undegraded":"2024-03-11T21:36:36.071501+0000","last_fullsized":"2024-03-11T21:36:36.040233+0000","mapping_epoch":278,"log_start":"290'6300","ondisk_log_start":"290'6300","created":259,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:35:44.669493+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:35:44.669493+0000","last_clean_scrub_stamp":"2024-03-11T21:35:44.669493+0000","objects_scrubbed":0,"log_size":4650,"log_dups_size":0,"ondisk_log_size":4650,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220856320,"num_objects":3370,"num_object_clones":0,"num_object_copies":10110,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4641,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3370,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10950,"num_write_kb":458240,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":196608,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,1],"acting":[7,1],"avail_no_missing":["7","1"],"object_location_counts":[{"shards":"1,7","objects":-1271}],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.c","version":"0'0","reported_seq":493,"reported_epoch":325,"state":"active+clean","last_fresh":"2024-03-11T21:38:48.757316+0000","last_change":"2024-03-11T21:36:35.074112+0000","last_active":"2024-03-11T21:38:48.757316+0000","last_peered":"2024-03-11T21:38:48.757316+0000","last_clean":"2024-03-11T21:38:48.757316+0000","last_became_active":"2024-03-11T21:36:35.073778+0000","last_became_peered":"2024-03-11T21:36:35.073778+0000","last_unstale":"2024-03-11T21:38:48.757316+0000","last_undegraded":"2024-03-11T21:38:48.757316+0000","last_fullsized":"2024-03-11T21:38:48.757316+0000","mapping_epoch":277,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":278,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:32:17.600213+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:43.559452+0000","last_clean_scrub_stamp":"2024-03-11T21:32:17.600213+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.f","version":"0'0","reported_seq":569,"reported_epoch":324,"state":"active+clean","last_fresh":"2024-03-11T21:38:43.362930+0000","last_change":"2024-03-11T21:36:08.439013+0000","last_active":"2024-03-11T21:38:43.362930+0000","last_peered":"2024-03-11T21:38:43.362930+0000","last_clean":"2024-03-11T21:38:43.362930+0000","last_became_active":"2024-03-11T21:25:42.575935+0000","last_became_peered":"2024-03-11T21:25:42.575935+0000","last_unstale":"2024-03-11T21:38:43.362930+0000","last_undegraded":"2024-03-11T21:38:43.362930+0000","last_fullsized":"2024-03-11T21:38:43.362930+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:36:08.438895+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:41.544550+0000","last_clean_scrub_stamp":"2024-03-11T21:36:08.438895+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.8","version":"0'0","reported_seq":214,"reported_epoch":324,"state":"active+clean","last_fresh":"2024-03-11T21:38:43.353745+0000","last_change":"2024-03-11T21:35:34.733595+0000","last_active":"2024-03-11T21:38:43.353745+0000","last_peered":"2024-03-11T21:38:43.353745+0000","last_clean":"2024-03-11T21:38:43.353745+0000","last_became_active":"2024-03-11T21:34:41.144681+0000","last_became_peered":"2024-03-11T21:34:41.144681+0000","last_unstale":"2024-03-11T21:38:43.353745+0000","last_undegraded":"2024-03-11T21:38:43.353745+0000","last_fullsized":"2024-03-11T21:38:43.353745+0000","mapping_epoch":226,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":227,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:35:34.733491+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:37.512018+0000","last_clean_scrub_stamp":"2024-03-11T21:35:34.733491+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,1],"acting":[7,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.e","version":"260'24","reported_seq":602,"reported_epoch":324,"state":"active+clean","last_fresh":"2024-03-11T21:38:43.363507+0000","last_change":"2024-03-11T21:35:34.679906+0000","last_active":"2024-03-11T21:38:43.363507+0000","last_peered":"2024-03-11T21:38:43.363507+0000","last_clean":"2024-03-11T21:38:43.363507+0000","last_became_active":"2024-03-11T21:25:40.552178+0000","last_became_peered":"2024-03-11T21:25:40.552178+0000","last_unstale":"2024-03-11T21:38:43.363507+0000","last_undegraded":"2024-03-11T21:38:43.363507+0000","last_fullsized":"2024-03-11T21:38:43.363507+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"109'16","last_scrub_stamp":"2024-03-11T21:35:34.679817+0000","last_deep_scrub":"109'16","last_deep_scrub_stamp":"2024-03-11T21:28:24.739395+0000","last_clean_scrub_stamp":"2024-03-11T21:35:34.679817+0000","objects_scrubbed":5,"log_size":24,"log_dups_size":0,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":32,"num_read_kb":25,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.2","version":"324'11161","reported_seq":12345,"reported_epoch":324,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-03-11T21:38:48.014400+0000","last_change":"2024-03-11T21:38:29.073541+0000","last_active":"2024-03-11T21:38:48.014400+0000","last_peered":"2024-03-11T21:38:48.014400+0000","last_clean":"2024-03-11T21:35:44.669493+0000","last_became_active":"2024-03-11T21:36:36.506474+0000","last_became_peered":"2024-03-11T21:36:36.506474+0000","last_unstale":"2024-03-11T21:38:48.014400+0000","last_undegraded":"2024-03-11T21:36:36.068182+0000","last_fullsized":"2024-03-11T21:36:36.037781+0000","mapping_epoch":278,"log_start":"288'6100","ondisk_log_start":"288'6100","created":259,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:35:44.669493+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:35:44.669493+0000","last_clean_scrub_stamp":"2024-03-11T21:35:44.669493+0000","objects_scrubbed":0,"log_size":5061,"log_dups_size":0,"ondisk_log_size":5061,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220790784,"num_objects":3369,"num_object_clones":0,"num_object_copies":10107,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4251,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3369,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11161,"num_write_kb":464960,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":522,"num_bytes_recovered":34209792,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[1,2],"avail_no_missing":["1","2"],"object_location_counts":[{"shards":"1,2","objects":-882}],"blocked_by":[],"up_primary":3,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.d","version":"0'0","reported_seq":557,"reported_epoch":324,"state":"active+clean","last_fresh":"2024-03-11T21:38:43.359367+0000","last_change":"2024-03-11T21:35:37.185210+0000","last_active":"2024-03-11T21:38:43.359367+0000","last_peered":"2024-03-11T21:38:43.359367+0000","last_clean":"2024-03-11T21:38:43.359367+0000","last_became_active":"2024-03-11T21:25:44.590333+0000","last_became_peered":"2024-03-11T21:25:44.590333+0000","last_unstale":"2024-03-11T21:38:43.359367+0000","last_undegraded":"2024-03-11T21:38:43.359367+0000","last_fullsized":"2024-03-11T21:38:43.359367+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:35:37.182850+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:35:37.182850+0000","last_clean_scrub_stamp":"2024-03-11T21:35:37.182850+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":40,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.e","version":"74'8","reported_seq":728,"reported_epoch":324,"state":"active+clean","last_fresh":"2024-03-11T21:38:43.363643+0000","last_change":"2024-03-11T21:36:07.734369+0000","last_active":"2024-03-11T21:38:43.363643+0000","last_peered":"2024-03-11T21:38:43.363643+0000","last_clean":"2024-03-11T21:38:43.363643+0000","last_became_active":"2024-03-11T21:25:42.576742+0000","last_became_peered":"2024-03-11T21:25:42.576742+0000","last_unstale":"2024-03-11T21:38:43.363643+0000","last_undegraded":"2024-03-11T21:38:43.363643+0000","last_fullsized":"2024-03-11T21:38:43.363643+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'8","last_scrub_stamp":"2024-03-11T21:36:07.734115+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:41.544550+0000","last_clean_scrub_stamp":"2024-03-11T21:36:07.734115+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,0],"acting":[4,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.9","version":"0'0","reported_seq":584,"reported_epoch":324,"state":"active+clean","last_fresh":"2024-03-11T21:38:43.418137+0000","last_change":"2024-03-11T21:36:35.066093+0000","last_active":"2024-03-11T21:38:43.418137+0000","last_peered":"2024-03-11T21:38:43.418137+0000","last_clean":"2024-03-11T21:38:43.418137+0000","last_became_active":"2024-03-11T21:36:35.065733+0000","last_became_peered":"2024-03-11T21:36:35.065733+0000","last_unstale":"2024-03-11T21:38:43.418137+0000","last_undegraded":"2024-03-11T21:38:43.418137+0000","last_fullsized":"2024-03-11T21:38:43.418137+0000","mapping_epoch":277,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":278,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:32:02.506579+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:26:50.991712+0000","last_clean_scrub_stamp":"2024-03-11T21:32:02.506579+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,3],"acting":[1,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.f","version":"260'27","reported_seq":585,"reported_epoch":324,"state":"active+clean","last_fresh":"2024-03-11T21:38:43.407285+0000","last_change":"2024-03-11T21:36:35.084069+0000","last_active":"2024-03-11T21:38:43.407285+0000","last_peered":"2024-03-11T21:38:43.407285+0000","last_clean":"2024-03-11T21:38:43.407285+0000","last_became_active":"2024-03-11T21:36:35.058272+0000","last_became_peered":"2024-03-11T21:36:35.058272+0000","last_unstale":"2024-03-11T21:38:43.407285+0000","last_undegraded":"2024-03-11T21:38:43.407285+0000","last_fullsized":"2024-03-11T21:38:43.407285+0000","mapping_epoch":277,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":278,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'15","last_scrub_stamp":"2024-03-11T21:34:21.517894+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:39.525554+0000","last_clean_scrub_stamp":"2024-03-11T21:34:21.517894+0000","objects_scrubbed":7,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":44,"num_read_kb":35,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,4],"acting":[2,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"9.1","version":"325'11282","reported_seq":21082,"reported_epoch":325,"state":"active+clean","last_fresh":"2024-03-11T21:38:48.803016+0000","last_change":"2024-03-11T21:38:22.558103+0000","last_active":"2024-03-11T21:38:48.803016+0000","last_peered":"2024-03-11T21:38:48.803016+0000","last_clean":"2024-03-11T21:38:48.803016+0000","last_became_active":"2024-03-11T21:38:22.540816+0000","last_became_peered":"2024-03-11T21:38:22.540816+0000","last_unstale":"2024-03-11T21:38:48.803016+0000","last_undegraded":"2024-03-11T21:38:48.803016+0000","last_fullsized":"2024-03-11T21:38:48.803016+0000","mapping_epoch":316,"log_start":"290'6395","ondisk_log_start":"290'6395","created":259,"last_epoch_clean":317,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:35:44.669493+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:35:44.669493+0000","last_clean_scrub_stamp":"2024-03-11T21:35:44.669493+0000","objects_scrubbed":0,"log_size":4887,"log_dups_size":0,"ondisk_log_size":4887,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":216399872,"num_objects":3302,"num_object_clones":0,"num_object_copies":9906,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3302,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11282,"num_write_kb":466688,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4825,"num_bytes_recovered":316211200,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,7],"acting":[3,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.e","version":"0'0","reported_seq":558,"reported_epoch":324,"state":"active+clean","last_fresh":"2024-03-11T21:38:43.363658+0000","last_change":"2024-03-11T21:36:10.769557+0000","last_active":"2024-03-11T21:38:43.363658+0000","last_peered":"2024-03-11T21:38:43.363658+0000","last_clean":"2024-03-11T21:38:43.363658+0000","last_became_active":"2024-03-11T21:25:44.590198+0000","last_became_peered":"2024-03-11T21:25:44.590198+0000","last_unstale":"2024-03-11T21:38:43.363658+0000","last_undegraded":"2024-03-11T21:38:43.363658+0000","last_fullsized":"2024-03-11T21:38:43.363658+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:36:10.769460+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:43.559452+0000","last_clean_scrub_stamp":"2024-03-11T21:36:10.769460+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":75,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.d","version":"109'9","reported_seq":728,"reported_epoch":324,"state":"active+clean","last_fresh":"2024-03-11T21:38:43.406857+0000","last_change":"2024-03-11T21:35:07.562523+0000","last_active":"2024-03-11T21:38:43.406857+0000","last_peered":"2024-03-11T21:38:43.406857+0000","last_clean":"2024-03-11T21:38:43.406857+0000","last_became_active":"2024-03-11T21:25:42.575583+0000","last_became_peered":"2024-03-11T21:25:42.575583+0000","last_unstale":"2024-03-11T21:38:43.406857+0000","last_undegraded":"2024-03-11T21:38:43.406857+0000","last_fullsized":"2024-03-11T21:38:43.406857+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"109'9","last_scrub_stamp":"2024-03-11T21:35:07.562385+0000","last_deep_scrub":"109'9","last_deep_scrub_stamp":"2024-03-11T21:28:27.082975+0000","last_clean_scrub_stamp":"2024-03-11T21:35:07.562385+0000","objects_scrubbed":1,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,5],"acting":[2,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.a","version":"0'0","reported_seq":592,"reported_epoch":324,"state":"active+clean","last_fresh":"2024-03-11T21:38:43.418353+0000","last_change":"2024-03-11T21:36:35.067262+0000","last_active":"2024-03-11T21:38:43.418353+0000","last_peered":"2024-03-11T21:38:43.418353+0000","last_clean":"2024-03-11T21:38:43.418353+0000","last_became_active":"2024-03-11T21:36:35.066832+0000","last_became_peered":"2024-03-11T21:36:35.066832+0000","last_unstale":"2024-03-11T21:38:43.418353+0000","last_undegraded":"2024-03-11T21:38:43.418353+0000","last_fullsized":"2024-03-11T21:38:43.418353+0000","mapping_epoch":277,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":278,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:34:22.526916+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:37.512018+0000","last_clean_scrub_stamp":"2024-03-11T21:34:22.526916+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,7],"acting":[1,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.c","version":"260'20","reported_seq":610,"reported_epoch":324,"state":"active+clean","last_fresh":"2024-03-11T21:38:43.363626+0000","last_change":"2024-03-11T21:36:35.095574+0000","last_active":"2024-03-11T21:38:43.363626+0000","last_peered":"2024-03-11T21:38:43.363626+0000","last_clean":"2024-03-11T21:38:43.363626+0000","last_became_active":"2024-03-11T21:36:35.067269+0000","last_became_peered":"2024-03-11T21:36:35.067269+0000","last_unstale":"2024-03-11T21:38:43.363626+0000","last_undegraded":"2024-03-11T21:38:43.363626+0000","last_fullsized":"2024-03-11T21:38:43.363626+0000","mapping_epoch":277,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":278,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'10","last_scrub_stamp":"2024-03-11T21:32:54.846584+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:39.525554+0000","last_clean_scrub_stamp":"2024-03-11T21:32:54.846584+0000","objects_scrubbed":5,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":25,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.0","version":"324'11199","reported_seq":11340,"reported_epoch":324,"state":"active+clean","last_fresh":"2024-03-11T21:38:47.990939+0000","last_change":"2024-03-11T21:35:45.703065+0000","last_active":"2024-03-11T21:38:47.990939+0000","last_peered":"2024-03-11T21:38:47.990939+0000","last_clean":"2024-03-11T21:38:47.990939+0000","last_became_active":"2024-03-11T21:35:45.702739+0000","last_became_peered":"2024-03-11T21:35:45.702739+0000","last_unstale":"2024-03-11T21:38:47.990939+0000","last_undegraded":"2024-03-11T21:38:47.990939+0000","last_fullsized":"2024-03-11T21:38:47.990939+0000","mapping_epoch":259,"log_start":"288'6100","ondisk_log_start":"288'6100","created":259,"last_epoch_clean":260,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:35:44.669493+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:35:44.669493+0000","last_clean_scrub_stamp":"2024-03-11T21:35:44.669493+0000","objects_scrubbed":0,"log_size":5099,"log_dups_size":0,"ondisk_log_size":5099,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":221839360,"num_objects":3385,"num_object_clones":0,"num_object_copies":10155,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3385,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11199,"num_write_kb":466688,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.f","version":"0'0","reported_seq":566,"reported_epoch":324,"state":"active+clean","last_fresh":"2024-03-11T21:38:43.362626+0000","last_change":"2024-03-11T21:36:35.058630+0000","last_active":"2024-03-11T21:38:43.362626+0000","last_peered":"2024-03-11T21:38:43.362626+0000","last_clean":"2024-03-11T21:38:43.362626+0000","last_became_active":"2024-03-11T21:36:35.057800+0000","last_became_peered":"2024-03-11T21:36:35.057800+0000","last_unstale":"2024-03-11T21:38:43.362626+0000","last_undegraded":"2024-03-11T21:38:43.362626+0000","last_fullsized":"2024-03-11T21:38:43.362626+0000","mapping_epoch":277,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":278,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:35:10.608981+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:27:03.010969+0000","last_clean_scrub_stamp":"2024-03-11T21:35:10.608981+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":94,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,4],"acting":[2,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":562,"reported_epoch":324,"state":"active+clean","last_fresh":"2024-03-11T21:38:43.420042+0000","last_change":"2024-03-11T21:36:10.608789+0000","last_active":"2024-03-11T21:38:43.420042+0000","last_peered":"2024-03-11T21:38:43.420042+0000","last_clean":"2024-03-11T21:38:43.420042+0000","last_became_active":"2024-03-11T21:25:42.579409+0000","last_became_peered":"2024-03-11T21:25:42.579409+0000","last_unstale":"2024-03-11T21:38:43.420042+0000","last_undegraded":"2024-03-11T21:38:43.420042+0000","last_fullsized":"2024-03-11T21:38:43.420042+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:36:10.608552+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:30:22.750053+0000","last_clean_scrub_stamp":"2024-03-11T21:36:10.608552+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.b","version":"0'0","reported_seq":214,"reported_epoch":324,"state":"active+clean","last_fresh":"2024-03-11T21:38:43.356087+0000","last_change":"2024-03-11T21:36:16.719432+0000","last_active":"2024-03-11T21:38:43.356087+0000","last_peered":"2024-03-11T21:38:43.356087+0000","last_clean":"2024-03-11T21:38:43.356087+0000","last_became_active":"2024-03-11T21:34:41.134987+0000","last_became_peered":"2024-03-11T21:34:41.134987+0000","last_unstale":"2024-03-11T21:38:43.356087+0000","last_undegraded":"2024-03-11T21:38:43.356087+0000","last_fullsized":"2024-03-11T21:38:43.356087+0000","mapping_epoch":226,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":227,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:36:16.719309+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:36:16.719309+0000","last_clean_scrub_stamp":"2024-03-11T21:36:16.719309+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,5],"acting":[7,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.d","version":"260'39","reported_seq":623,"reported_epoch":324,"state":"active+clean","last_fresh":"2024-03-11T21:38:43.364147+0000","last_change":"2024-03-11T21:35:25.543769+0000","last_active":"2024-03-11T21:38:43.364147+0000","last_peered":"2024-03-11T21:38:43.364147+0000","last_clean":"2024-03-11T21:38:43.364147+0000","last_became_active":"2024-03-11T21:25:40.553454+0000","last_became_peered":"2024-03-11T21:25:40.553454+0000","last_unstale":"2024-03-11T21:38:43.364147+0000","last_undegraded":"2024-03-11T21:38:43.364147+0000","last_fullsized":"2024-03-11T21:38:43.364147+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"165'25","last_scrub_stamp":"2024-03-11T21:35:25.543628+0000","last_deep_scrub":"165'25","last_deep_scrub_stamp":"2024-03-11T21:32:21.678328+0000","last_clean_scrub_stamp":"2024-03-11T21:35:25.543628+0000","objects_scrubbed":8,"log_size":39,"log_dups_size":0,"ondisk_log_size":39,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":50,"num_read_kb":40,"num_write":32,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,1],"acting":[4,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.7","version":"324'11134","reported_seq":19647,"reported_epoch":324,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-03-11T21:38:47.474293+0000","last_change":"2024-03-11T21:36:36.539792+0000","last_active":"2024-03-11T21:38:47.474293+0000","last_peered":"2024-03-11T21:38:47.474293+0000","last_clean":"2024-03-11T21:35:44.669493+0000","last_became_active":"2024-03-11T21:36:36.508974+0000","last_became_peered":"2024-03-11T21:36:36.508974+0000","last_unstale":"2024-03-11T21:38:47.474293+0000","last_undegraded":"2024-03-11T21:36:36.183555+0000","last_fullsized":"2024-03-11T21:36:36.038772+0000","mapping_epoch":278,"log_start":"288'5900","ondisk_log_start":"288'5900","created":259,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:35:44.669493+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:35:44.669493+0000","last_clean_scrub_stamp":"2024-03-11T21:35:44.669493+0000","objects_scrubbed":0,"log_size":5234,"log_dups_size":0,"ondisk_log_size":5234,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":222429184,"num_objects":3394,"num_object_clones":0,"num_object_copies":10182,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":567,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3394,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11134,"num_write_kb":464896,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4185,"num_bytes_recovered":274268160,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,7],"avail_no_missing":["5","7"],"object_location_counts":[{"shards":"5,7","objects":2827}],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":214,"reported_epoch":324,"state":"active+clean","last_fresh":"2024-03-11T21:38:43.353640+0000","last_change":"2024-03-11T21:36:06.735035+0000","last_active":"2024-03-11T21:38:43.353640+0000","last_peered":"2024-03-11T21:38:43.353640+0000","last_clean":"2024-03-11T21:38:43.353640+0000","last_became_active":"2024-03-11T21:34:41.145012+0000","last_became_peered":"2024-03-11T21:34:41.145012+0000","last_unstale":"2024-03-11T21:38:43.353640+0000","last_undegraded":"2024-03-11T21:38:43.353640+0000","last_fullsized":"2024-03-11T21:38:43.353640+0000","mapping_epoch":226,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":227,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:36:06.734921+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:28:37.561826+0000","last_clean_scrub_stamp":"2024-03-11T21:36:06.734921+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":23,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,0],"acting":[7,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":561,"reported_epoch":324,"state":"active+clean","last_fresh":"2024-03-11T21:38:43.407401+0000","last_change":"2024-03-11T21:35:14.636561+0000","last_active":"2024-03-11T21:38:43.407401+0000","last_peered":"2024-03-11T21:38:43.407401+0000","last_clean":"2024-03-11T21:38:43.407401+0000","last_became_active":"2024-03-11T21:25:42.574515+0000","last_became_peered":"2024-03-11T21:25:42.574515+0000","last_unstale":"2024-03-11T21:38:43.407401+0000","last_undegraded":"2024-03-11T21:38:43.407401+0000","last_fullsized":"2024-03-11T21:38:43.407401+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:35:14.636450+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:41.544550+0000","last_clean_scrub_stamp":"2024-03-11T21:35:14.636450+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,5],"acting":[2,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.c","version":"0'0","reported_seq":569,"reported_epoch":324,"state":"active+clean","last_fresh":"2024-03-11T21:38:43.408891+0000","last_change":"2024-03-11T21:36:00.849272+0000","last_active":"2024-03-11T21:38:43.408891+0000","last_peered":"2024-03-11T21:38:43.408891+0000","last_clean":"2024-03-11T21:38:43.408891+0000","last_became_active":"2024-03-11T21:25:38.533240+0000","last_became_peered":"2024-03-11T21:25:38.533240+0000","last_unstale":"2024-03-11T21:38:43.408891+0000","last_undegraded":"2024-03-11T21:38:43.408891+0000","last_fullsized":"2024-03-11T21:38:43.408891+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:36:00.849153+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:29:30.279910+0000","last_clean_scrub_stamp":"2024-03-11T21:36:00.849153+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,0],"acting":[2,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.a","version":"260'44","reported_seq":691,"reported_epoch":324,"state":"active+clean","last_fresh":"2024-03-11T21:38:43.362529+0000","last_change":"2024-03-11T21:35:34.138400+0000","last_active":"2024-03-11T21:38:43.362529+0000","last_peered":"2024-03-11T21:38:43.362529+0000","last_clean":"2024-03-11T21:38:43.362529+0000","last_became_active":"2024-03-11T21:34:41.148863+0000","last_became_peered":"2024-03-11T21:34:41.148863+0000","last_unstale":"2024-03-11T21:38:43.362529+0000","last_undegraded":"2024-03-11T21:38:43.362529+0000","last_fullsized":"2024-03-11T21:38:43.362529+0000","mapping_epoch":226,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":227,"parent":"0.0","parent_split_bits":0,"last_scrub":"109'28","last_scrub_stamp":"2024-03-11T21:35:34.138261+0000","last_deep_scrub":"109'28","last_deep_scrub_stamp":"2024-03-11T21:28:24.016389+0000","last_clean_scrub_stamp":"2024-03-11T21:35:34.138261+0000","objects_scrubbed":9,"log_size":44,"log_dups_size":0,"ondisk_log_size":44,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":56,"num_read_kb":45,"num_write":36,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,7],"acting":[6,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.6","version":"324'11196","reported_seq":11337,"reported_epoch":324,"state":"active+clean","last_fresh":"2024-03-11T21:38:47.495771+0000","last_change":"2024-03-11T21:35:45.703533+0000","last_active":"2024-03-11T21:38:47.495771+0000","last_peered":"2024-03-11T21:38:47.495771+0000","last_clean":"2024-03-11T21:38:47.495771+0000","last_became_active":"2024-03-11T21:35:45.703184+0000","last_became_peered":"2024-03-11T21:35:45.703184+0000","last_unstale":"2024-03-11T21:38:47.495771+0000","last_undegraded":"2024-03-11T21:38:47.495771+0000","last_fullsized":"2024-03-11T21:38:47.495771+0000","mapping_epoch":259,"log_start":"288'6000","ondisk_log_start":"288'6000","created":259,"last_epoch_clean":260,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:35:44.669493+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:35:44.669493+0000","last_clean_scrub_stamp":"2024-03-11T21:35:44.669493+0000","objects_scrubbed":0,"log_size":5196,"log_dups_size":0,"ondisk_log_size":5196,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":219545600,"num_objects":3350,"num_object_clones":0,"num_object_copies":10050,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3350,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11196,"num_write_kb":465472,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,1],"acting":[5,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":570,"reported_epoch":324,"state":"active+clean","last_fresh":"2024-03-11T21:38:43.407808+0000","last_change":"2024-03-11T21:35:37.687281+0000","last_active":"2024-03-11T21:38:43.407808+0000","last_peered":"2024-03-11T21:38:43.407808+0000","last_clean":"2024-03-11T21:38:43.407808+0000","last_became_active":"2024-03-11T21:34:41.154507+0000","last_became_peered":"2024-03-11T21:34:41.154507+0000","last_unstale":"2024-03-11T21:38:43.407808+0000","last_undegraded":"2024-03-11T21:38:43.407808+0000","last_fullsized":"2024-03-11T21:38:43.407808+0000","mapping_epoch":226,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":227,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:35:37.687187+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:27:11.361913+0000","last_clean_scrub_stamp":"2024-03-11T21:35:37.687187+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":39,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":563,"reported_epoch":324,"state":"active+clean","last_fresh":"2024-03-11T21:38:43.407681+0000","last_change":"2024-03-11T21:36:35.059662+0000","last_active":"2024-03-11T21:38:43.407681+0000","last_peered":"2024-03-11T21:38:43.407681+0000","last_clean":"2024-03-11T21:38:43.407681+0000","last_became_active":"2024-03-11T21:36:35.059192+0000","last_became_peered":"2024-03-11T21:36:35.059192+0000","last_unstale":"2024-03-11T21:38:43.407681+0000","last_undegraded":"2024-03-11T21:38:43.407681+0000","last_fullsized":"2024-03-11T21:38:43.407681+0000","mapping_epoch":277,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":278,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:34:17.412687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:41.544550+0000","last_clean_scrub_stamp":"2024-03-11T21:34:17.412687+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,3],"acting":[2,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.d","version":"0'0","reported_seq":571,"reported_epoch":324,"state":"active+clean","last_fresh":"2024-03-11T21:38:43.421590+0000","last_change":"2024-03-11T21:36:35.058635+0000","last_active":"2024-03-11T21:38:43.421590+0000","last_peered":"2024-03-11T21:38:43.421590+0000","last_clean":"2024-03-11T21:38:43.421590+0000","last_became_active":"2024-03-11T21:36:35.058296+0000","last_became_peered":"2024-03-11T21:36:35.058296+0000","last_unstale":"2024-03-11T21:38:43.421590+0000","last_undegraded":"2024-03-11T21:38:43.421590+0000","last_fullsized":"2024-03-11T21:38:43.421590+0000","mapping_epoch":277,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":278,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:31:55.464145+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:37.512018+0000","last_clean_scrub_stamp":"2024-03-11T21:31:55.464145+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,3],"acting":[1,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.b","version":"260'15","reported_seq":591,"reported_epoch":324,"state":"active+clean","last_fresh":"2024-03-11T21:38:43.406660+0000","last_change":"2024-03-11T21:35:27.760318+0000","last_active":"2024-03-11T21:38:43.406660+0000","last_peered":"2024-03-11T21:38:43.406660+0000","last_clean":"2024-03-11T21:38:43.406660+0000","last_became_active":"2024-03-11T21:25:40.549612+0000","last_became_peered":"2024-03-11T21:25:40.549612+0000","last_unstale":"2024-03-11T21:38:43.406660+0000","last_undegraded":"2024-03-11T21:38:43.406660+0000","last_fullsized":"2024-03-11T21:38:43.406660+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'9","last_scrub_stamp":"2024-03-11T21:35:27.760208+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:39.525554+0000","last_clean_scrub_stamp":"2024-03-11T21:35:27.760208+0000","objects_scrubbed":4,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.5","version":"324'11331","reported_seq":11472,"reported_epoch":324,"state":"active+clean","last_fresh":"2024-03-11T21:38:48.535222+0000","last_change":"2024-03-11T21:35:45.697209+0000","last_active":"2024-03-11T21:38:48.535222+0000","last_peered":"2024-03-11T21:38:48.535222+0000","last_clean":"2024-03-11T21:38:48.535222+0000","last_became_active":"2024-03-11T21:35:45.696788+0000","last_became_peered":"2024-03-11T21:35:45.696788+0000","last_unstale":"2024-03-11T21:38:48.535222+0000","last_undegraded":"2024-03-11T21:38:48.535222+0000","last_fullsized":"2024-03-11T21:38:48.535222+0000","mapping_epoch":259,"log_start":"286'5700","ondisk_log_start":"286'5700","created":259,"last_epoch_clean":260,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:35:44.669493+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:35:44.669493+0000","last_clean_scrub_stamp":"2024-03-11T21:35:44.669493+0000","objects_scrubbed":0,"log_size":5631,"log_dups_size":0,"ondisk_log_size":5631,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":227475456,"num_objects":3471,"num_object_clones":0,"num_object_copies":10413,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3471,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11331,"num_write_kb":473664,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":472,"reported_epoch":324,"state":"active+clean","last_fresh":"2024-03-11T21:38:43.362828+0000","last_change":"2024-03-11T21:36:35.057118+0000","last_active":"2024-03-11T21:38:43.362828+0000","last_peered":"2024-03-11T21:38:43.362828+0000","last_clean":"2024-03-11T21:38:43.362828+0000","last_became_active":"2024-03-11T21:36:35.056518+0000","last_became_peered":"2024-03-11T21:36:35.056518+0000","last_unstale":"2024-03-11T21:38:43.362828+0000","last_undegraded":"2024-03-11T21:38:43.362828+0000","last_fullsized":"2024-03-11T21:38:43.362828+0000","mapping_epoch":277,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":278,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:35:24.514584+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:43.559452+0000","last_clean_scrub_stamp":"2024-03-11T21:35:24.514584+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,3],"acting":[4,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.e","version":"0'0","reported_seq":590,"reported_epoch":324,"state":"active+clean","last_fresh":"2024-03-11T21:38:43.407970+0000","last_change":"2024-03-11T21:36:35.057187+0000","last_active":"2024-03-11T21:38:43.407970+0000","last_peered":"2024-03-11T21:38:43.407970+0000","last_clean":"2024-03-11T21:38:43.407970+0000","last_became_active":"2024-03-11T21:36:35.056777+0000","last_became_peered":"2024-03-11T21:36:35.056777+0000","last_unstale":"2024-03-11T21:38:43.407970+0000","last_undegraded":"2024-03-11T21:38:43.407970+0000","last_fullsized":"2024-03-11T21:38:43.407970+0000","mapping_epoch":277,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":278,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:34:19.476172+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:37.512018+0000","last_clean_scrub_stamp":"2024-03-11T21:34:19.476172+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":17,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,7],"acting":[2,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.8","version":"260'34","reported_seq":660,"reported_epoch":324,"state":"active+clean","last_fresh":"2024-03-11T21:38:43.357653+0000","last_change":"2024-03-11T21:35:20.304345+0000","last_active":"2024-03-11T21:38:43.357653+0000","last_peered":"2024-03-11T21:38:43.357653+0000","last_clean":"2024-03-11T21:38:43.357653+0000","last_became_active":"2024-03-11T21:34:41.146631+0000","last_became_peered":"2024-03-11T21:34:41.146631+0000","last_unstale":"2024-03-11T21:38:43.357653+0000","last_undegraded":"2024-03-11T21:38:43.357653+0000","last_fullsized":"2024-03-11T21:38:43.357653+0000","mapping_epoch":226,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":227,"parent":"0.0","parent_split_bits":0,"last_scrub":"109'22","last_scrub_stamp":"2024-03-11T21:35:20.304114+0000","last_deep_scrub":"109'22","last_deep_scrub_stamp":"2024-03-11T21:32:07.573197+0000","last_clean_scrub_stamp":"2024-03-11T21:35:20.304114+0000","objects_scrubbed":7,"log_size":34,"log_dups_size":0,"ondisk_log_size":34,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":44,"num_read_kb":35,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.9","version":"229'11","reported_seq":267,"reported_epoch":324,"state":"active+clean","last_fresh":"2024-03-11T21:38:43.354052+0000","last_change":"2024-03-11T21:36:18.726876+0000","last_active":"2024-03-11T21:38:43.354052+0000","last_peered":"2024-03-11T21:38:43.354052+0000","last_clean":"2024-03-11T21:38:43.354052+0000","last_became_active":"2024-03-11T21:34:41.156892+0000","last_became_peered":"2024-03-11T21:34:41.156892+0000","last_unstale":"2024-03-11T21:38:43.354052+0000","last_undegraded":"2024-03-11T21:38:43.354052+0000","last_fullsized":"2024-03-11T21:38:43.354052+0000","mapping_epoch":226,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":227,"parent":"0.0","parent_split_bits":0,"last_scrub":"229'11","last_scrub_stamp":"2024-03-11T21:36:18.726763+0000","last_deep_scrub":"94'9","last_deep_scrub_stamp":"2024-03-11T21:26:53.422552+0000","last_clean_scrub_stamp":"2024-03-11T21:36:18.726763+0000","objects_scrubbed":1,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":9,"num_pg":16,"stat_sum":{"num_bytes":3551723544,"num_objects":54196,"num_object_clones":0,"num_object_copies":162588,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":23123,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":54196,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":178568,"num_write_kb":7448385,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14900,"num_bytes_recovered":976486400,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":10371674112,"data_stored":10371661896,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":82129,"ondisk_log_size":82129,"up":48,"acting":42,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":1531,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":17,"num_read_kb":17,"num_write":10,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":1308,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":4593,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":6,"ondisk_log_size":6,"up":96,"acting":96,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":198,"num_read_kb":294,"num_write":179,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":2951840,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":2375680,"data_stored":2361472,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":128,"ondisk_log_size":128,"up":3,"acting":3,"num_store_stats":4},{"poolid":3,"num_pg":3,"stat_sum":{"num_bytes":408,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":782,"num_read_kb":777,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1224,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":8,"ondisk_log_size":8,"up":9,"acting":9,"num_store_stats":7},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":3702,"num_objects":178,"num_object_clones":0,"num_object_copies":534,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":178,"num_whiteouts":0,"num_read":1120,"num_read_kb":877,"num_write":703,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":186,"num_bytes_recovered":2200,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":417792,"data_stored":11106,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":760,"ondisk_log_size":760,"up":96,"acting":96,"num_store_stats":8},{"poolid":5,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":78,"ondisk_log_size":78,"up":96,"acting":96,"num_store_stats":8},{"poolid":6,"num_pg":32,"stat_sum":{"num_bytes":383,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":370,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1149,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":96,"acting":96,"num_store_stats":8}],"osd_stats":[{"osd":7,"up_from":60,"seq":257698037921,"num_pgs":65,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2460760,"kb_used_data":1996432,"kb_used_omap":1,"kb_used_meta":464318,"kb_avail":91284392,"statfs":{"total":95995035648,"available":93475217408,"internally_reserved":0,"allocated":2044346368,"data_stored":2043536815,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":475462090},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,3,1,2,1,3],"upper_bound":64},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":50,"seq":214748364967,"num_pgs":46,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1037716,"kb_used_data":655308,"kb_used_omap":1,"kb_used_meta":382398,"kb_avail":92707436,"statfs":{"total":95995035648,"available":94932414464,"internally_reserved":0,"allocated":671035392,"data_stored":670230640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":391576006},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1],"upper_bound":4},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834575041,"num_pgs":60,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2511472,"kb_used_data":1956904,"kb_used_omap":1,"kb_used_meta":554558,"kb_avail":91233680,"statfs":{"total":95995035648,"available":93423288320,"internally_reserved":0,"allocated":2003869696,"data_stored":2003035750,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":567867845},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,6,0,0,1,1,4],"upper_bound":128},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738566,"num_pgs":54,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1422688,"kb_used_data":881944,"kb_used_omap":1,"kb_used_meta":540670,"kb_avail":92322464,"statfs":{"total":95995035648,"available":94538203136,"internally_reserved":0,"allocated":903110656,"data_stored":902293650,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":553646533},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,0,0,1],"upper_bound":64},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411515,"num_pgs":43,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1086348,"kb_used_data":658116,"kb_used_omap":1,"kb_used_meta":428222,"kb_avail":92658804,"statfs":{"total":95995035648,"available":94882615296,"internally_reserved":0,"allocated":673910784,"data_stored":673113448,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":438499781},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1],"upper_bound":8},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":277,"seq":1189705941020,"num_pgs":62,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1626416,"kb_used_data":1412264,"kb_used_omap":4,"kb_used_meta":214139,"kb_avail":92118736,"statfs":{"total":95995035648,"available":94329585664,"internally_reserved":0,"allocated":1446158336,"data_stored":1445324810,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4442,"internal_metadata":219279014},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1],"upper_bound":4},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":4,"up_from":32,"seq":137438953648,"num_pgs":56,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1036256,"kb_used_data":662168,"kb_used_omap":1,"kb_used_meta":374078,"kb_avail":92708896,"statfs":{"total":95995035648,"available":94933909504,"internally_reserved":0,"allocated":678060032,"data_stored":677242064,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":383056325},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":41,"seq":176093659307,"num_pgs":58,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2507664,"kb_used_data":1980872,"kb_used_omap":1,"kb_used_meta":526782,"kb_avail":91237488,"statfs":{"total":95995035648,"available":93427187712,"internally_reserved":0,"allocated":2028412928,"data_stored":2027611135,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":539425221},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,0,0,2],"upper_bound":32},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1049,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":46,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":436,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":957,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":574,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":436,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1003,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":92,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":1430,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":77824,"data_stored":2008,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":1018,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":1898,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":1474,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":990,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":1034,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":1254,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":370,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":383,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":370,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":892600320,"data_stored":892600320,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":1990856704,"data_stored":1990852632,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":665452544,"data_stored":665452544,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":1444020224,"data_stored":1444020224,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":673054720,"data_stored":673054720,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":2017333248,"data_stored":2017329176,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":666632192,"data_stored":666632192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":2021724160,"data_stored":2021720088,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-11T21:38:52.858 DEBUG:teuthology.orchestra.run.smithi092:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid 00db0016-dfed-11ee-95c8-87774f69a715 -- ceph status --format=json 2024-03-11T21:38:52.921 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:38:52.981 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:38:52.982 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:38:52.982 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:38:53.078 INFO:journalctl@ceph.iscsi.iscsi.a.smithi097.stdout:Mar 11 21:38:52 smithi097 ceph-00db0016-dfed-11ee-95c8-87774f69a715-iscsi-iscsi-a[60943]: debug there is no tcmu-runner data available 2024-03-11T21:38:53.078 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:38:52 smithi097 ceph-mon[33317]: pgmap v616: 148 pgs: 3 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+degraded+remapped, 142 active+clean; 3.3 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 639 B/s rd, 0 B/s wr, 653 op/s; 23123/163182 objects degraded (14.170%); 4.6 MiB/s, 73 objects/s recovering 2024-03-11T21:38:53.078 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:38:52 smithi097 ceph-mon[33317]: from='mgr.14607 ' entity='mgr.y' 2024-03-11T21:38:53.078 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:38:52 smithi097 ceph-mon[33317]: from='mgr.14607 v1:172.21.15.92:0/1702717597' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T21:38:53.083 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:38:53.112 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:38:52 smithi092 ceph-mon[29572]: pgmap v616: 148 pgs: 3 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+degraded+remapped, 142 active+clean; 3.3 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 639 B/s rd, 0 B/s wr, 653 op/s; 23123/163182 objects degraded (14.170%); 4.6 MiB/s, 73 objects/s recovering 2024-03-11T21:38:53.113 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:38:52 smithi092 ceph-mon[29572]: from='mgr.14607 ' entity='mgr.y' 2024-03-11T21:38:53.113 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:38:52 smithi092 ceph-mon[29572]: from='mgr.14607 v1:172.21.15.92:0/1702717597' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T21:38:53.113 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:38:52 smithi092 ceph-mon[37563]: pgmap v616: 148 pgs: 3 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+degraded+remapped, 142 active+clean; 3.3 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 639 B/s rd, 0 B/s wr, 653 op/s; 23123/163182 objects degraded (14.170%); 4.6 MiB/s, 73 objects/s recovering 2024-03-11T21:38:53.113 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:38:52 smithi092 ceph-mon[37563]: from='mgr.14607 ' entity='mgr.y' 2024-03-11T21:38:53.113 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:38:52 smithi092 ceph-mon[37563]: from='mgr.14607 v1:172.21.15.92:0/1702717597' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T21:38:53.135 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:38:53.136 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:38:53.136 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:38:53.237 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:38:53.265 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:38:53.266 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:38:53.267 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:38:53.368 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:38:53.438 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:38:53.439 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:38:53.439 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:38:53.540 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:38:53.588 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:38:53.589 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:38:53.589 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:38:53.690 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:38:53.747 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:38:53.748 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:38:53.749 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:38:53.849 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:38:53.897 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:38:53.897 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:38:53.898 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:38:53.998 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:38:54.048 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:38:54.049 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:38:54.050 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:38:54.078 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:38:53 smithi097 ceph-mon[33317]: purged_snaps scrub starts 2024-03-11T21:38:54.078 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:38:53 smithi097 ceph-mon[33317]: purged_snaps scrub ok 2024-03-11T21:38:54.078 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:38:53 smithi097 ceph-mon[33317]: from='client.24469 v1:172.21.15.97:0/404208310' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:38:54.078 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:38:53 smithi097 ceph-mon[33317]: osdmap e326: 8 total, 8 up, 8 in 2024-03-11T21:38:54.078 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:38:53 smithi097 ceph-mon[33317]: 5.2 deep-scrub starts 2024-03-11T21:38:54.078 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:38:53 smithi097 ceph-mon[33317]: 5.2 deep-scrub ok 2024-03-11T21:38:54.078 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:38:53 smithi097 ceph-mon[33317]: Health check update: Degraded data redundancy: 23123/163182 objects degraded (14.170%), 6 pgs degraded, 6 pgs undersized (PG_DEGRADED) 2024-03-11T21:38:54.112 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:38:53 smithi092 ceph-mon[29572]: purged_snaps scrub starts 2024-03-11T21:38:54.112 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:38:53 smithi092 ceph-mon[29572]: purged_snaps scrub ok 2024-03-11T21:38:54.113 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:38:53 smithi092 ceph-mon[29572]: from='client.24469 v1:172.21.15.97:0/404208310' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:38:54.113 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:38:53 smithi092 ceph-mon[29572]: osdmap e326: 8 total, 8 up, 8 in 2024-03-11T21:38:54.113 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:38:53 smithi092 ceph-mon[29572]: 5.2 deep-scrub starts 2024-03-11T21:38:54.113 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:38:53 smithi092 ceph-mon[29572]: 5.2 deep-scrub ok 2024-03-11T21:38:54.113 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:38:53 smithi092 ceph-mon[29572]: Health check update: Degraded data redundancy: 23123/163182 objects degraded (14.170%), 6 pgs degraded, 6 pgs undersized (PG_DEGRADED) 2024-03-11T21:38:54.113 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:38:53 smithi092 ceph-mon[37563]: purged_snaps scrub starts 2024-03-11T21:38:54.113 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:38:53 smithi092 ceph-mon[37563]: purged_snaps scrub ok 2024-03-11T21:38:54.113 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:38:53 smithi092 ceph-mon[37563]: from='client.24469 v1:172.21.15.97:0/404208310' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:38:54.113 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:38:53 smithi092 ceph-mon[37563]: osdmap e326: 8 total, 8 up, 8 in 2024-03-11T21:38:54.113 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:38:53 smithi092 ceph-mon[37563]: 5.2 deep-scrub starts 2024-03-11T21:38:54.113 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:38:53 smithi092 ceph-mon[37563]: 5.2 deep-scrub ok 2024-03-11T21:38:54.113 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:38:53 smithi092 ceph-mon[37563]: Health check update: Degraded data redundancy: 23123/163182 objects degraded (14.170%), 6 pgs degraded, 6 pgs undersized (PG_DEGRADED) 2024-03-11T21:38:54.151 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:38:54.172 INFO:teuthology.orchestra.run.smithi092.stderr:Inferring config /var/lib/ceph/00db0016-dfed-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T21:38:54.177 INFO:teuthology.orchestra.run.smithi092.stderr:Inferring config /var/lib/ceph/00db0016-dfed-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T21:38:54.255 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:38:54.255 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:38:54.255 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:38:54.356 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:38:54.385 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:38:54.386 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:38:54.386 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:38:54.487 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:38:54.513 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:38:54.514 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:38:54.514 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:38:54.615 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:38:54.647 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:38:54.647 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:38:54.648 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:38:54.748 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:38:54.780 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:38:54.781 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:38:54.782 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:38:54.882 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:38:54.911 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:38:54.912 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:38:54.912 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:38:55.013 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:38:55.039 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:38:55.040 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:38:55.040 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:38:55.078 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:38:54 smithi097 ceph-mon[33317]: pgmap v618: 148 pgs: 3 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+degraded+remapped, 142 active+clean; 3.2 GiB data, 13 GiB used, 703 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 773 op/s; 22607/156003 objects degraded (14.491%); 8.6 MiB/s, 138 objects/s recovering 2024-03-11T21:38:55.078 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:38:54 smithi097 ceph-mon[33317]: 4.1f scrub starts 2024-03-11T21:38:55.078 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:38:54 smithi097 ceph-mon[33317]: 4.1f scrub ok 2024-03-11T21:38:55.109 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:38:54 smithi092 ceph-mon[37563]: pgmap v618: 148 pgs: 3 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+degraded+remapped, 142 active+clean; 3.2 GiB data, 13 GiB used, 703 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 773 op/s; 22607/156003 objects degraded (14.491%); 8.6 MiB/s, 138 objects/s recovering 2024-03-11T21:38:55.110 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:38:54 smithi092 ceph-mon[37563]: 4.1f scrub starts 2024-03-11T21:38:55.110 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:38:54 smithi092 ceph-mon[37563]: 4.1f scrub ok 2024-03-11T21:38:55.110 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:38:54 smithi092 ceph-mon[29572]: pgmap v618: 148 pgs: 3 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+degraded+remapped, 142 active+clean; 3.2 GiB data, 13 GiB used, 703 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 773 op/s; 22607/156003 objects degraded (14.491%); 8.6 MiB/s, 138 objects/s recovering 2024-03-11T21:38:55.110 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:38:54 smithi092 ceph-mon[29572]: 4.1f scrub starts 2024-03-11T21:38:55.110 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:38:54 smithi092 ceph-mon[29572]: 4.1f scrub ok 2024-03-11T21:38:55.141 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:38:55.171 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:38:55.172 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:38:55.172 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:38:55.273 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:38:55.304 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:38:55.305 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:38:55.305 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:38:55.406 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:38:55.446 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:38:55.447 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:38:55.447 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:38:55.548 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:38:55.588 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:38:55.588 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:38:55.589 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:38:55.689 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:38:55.702 INFO:teuthology.orchestra.run.smithi092.stderr:Inferring config /var/lib/ceph/00db0016-dfed-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T21:38:55.772 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:38:55.772 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:38:55.773 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:38:55.873 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:38:55.904 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:38:55.905 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:38:55.905 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:38:56.006 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:38:56.035 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:38:56.036 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:38:56.036 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:38:56.078 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:38:55 smithi097 ceph-mon[33317]: 2.7 scrub starts 2024-03-11T21:38:56.078 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:38:55 smithi097 ceph-mon[33317]: 2.7 scrub ok 2024-03-11T21:38:56.096 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:38:55 smithi092 ceph-mon[29572]: 2.7 scrub starts 2024-03-11T21:38:56.096 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:38:55 smithi092 ceph-mon[29572]: 2.7 scrub ok 2024-03-11T21:38:56.096 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:38:55 smithi092 ceph-mon[37563]: 2.7 scrub starts 2024-03-11T21:38:56.096 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:38:55 smithi092 ceph-mon[37563]: 2.7 scrub ok 2024-03-11T21:38:56.137 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:38:56.166 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:38:56.166 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:38:56.167 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:38:56.268 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:38:56.308 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:38:56.308 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:38:56.309 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:38:56.409 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:38:56.434 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:38:56.435 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:38:56.435 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:38:56.536 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:38:56.589 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:38:56.590 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:38:56.590 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:38:56.691 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:38:56.734 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:38:56.735 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:38:56.736 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:38:56.836 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:38:56.863 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:38:56.863 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:38:56.864 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:38:56.965 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:38:56.991 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:38:56.992 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:38:56.993 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:38:57.078 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:38:56 smithi097 ceph-mon[33317]: pgmap v619: 148 pgs: 3 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+degraded+remapped, 142 active+clean; 3.1 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 965 op/s; 22607/151371 objects degraded (14.935%); 8.6 MiB/s, 138 objects/s recovering 2024-03-11T21:38:57.084 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:38:56 smithi092 ceph-mon[29572]: pgmap v619: 148 pgs: 3 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+degraded+remapped, 142 active+clean; 3.1 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 965 op/s; 22607/151371 objects degraded (14.935%); 8.6 MiB/s, 138 objects/s recovering 2024-03-11T21:38:57.085 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:38:56 smithi092 ceph-mon[37563]: pgmap v619: 148 pgs: 3 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+degraded+remapped, 142 active+clean; 3.1 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 965 op/s; 22607/151371 objects degraded (14.935%); 8.6 MiB/s, 138 objects/s recovering 2024-03-11T21:38:57.093 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:38:57.121 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:38:57.121 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:38:57.121 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:38:57.222 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:38:57.272 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:38:57.272 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:38:57.273 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:38:57.374 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:38:57.402 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:38:57.403 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:38:57.403 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:38:57.504 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:38:57.535 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:38:57.535 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:38:57.536 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:38:57.636 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:38:57.667 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:38:57.667 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:38:57.667 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:38:57.770 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:38:57.816 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:38:57.817 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:38:57.817 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:38:57.917 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:38:57.957 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:38:57.957 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:38:57.958 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:38:58.059 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:38:58.078 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:38:57 smithi097 ceph-mon[33317]: 4.a scrub starts 2024-03-11T21:38:58.078 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:38:57 smithi097 ceph-mon[33317]: 4.a scrub ok 2024-03-11T21:38:58.112 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:38:57 smithi092 ceph-mon[29572]: 4.a scrub starts 2024-03-11T21:38:58.112 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:38:57 smithi092 ceph-mon[29572]: 4.a scrub ok 2024-03-11T21:38:58.112 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:38:57 smithi092 ceph-mon[37563]: 4.a scrub starts 2024-03-11T21:38:58.112 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:38:57 smithi092 ceph-mon[37563]: 4.a scrub ok 2024-03-11T21:38:58.112 INFO:journalctl@ceph.mgr.y.smithi092.stdout:Mar 11 21:38:57 smithi092 ceph-00db0016-dfed-11ee-95c8-87774f69a715-mgr-y[29801]: ::ffff:172.21.15.97 - - [11/Mar/2024:21:38:57] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-11T21:38:58.114 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:38:58.115 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:38:58.115 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:38:58.216 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:38:58.240 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:38:58.241 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:38:58.241 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:38:58.342 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:38:58.371 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:38:58.371 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:38:58.371 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:38:58.472 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:38:58.516 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:38:58.517 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:38:58.517 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:38:58.598 INFO:teuthology.orchestra.run.smithi092.stdout:osd.0: {} 2024-03-11T21:38:58.618 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:38:58.657 INFO:teuthology.orchestra.run.smithi092.stdout:osd.1: {} 2024-03-11T21:38:58.671 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:38:58.672 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:38:58.672 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:38:58.696 INFO:teuthology.orchestra.run.smithi092.stdout:osd.2: {} 2024-03-11T21:38:58.736 INFO:teuthology.orchestra.run.smithi092.stdout:osd.3: {} 2024-03-11T21:38:58.769 INFO:teuthology.orchestra.run.smithi092.stdout:osd.4: {} 2024-03-11T21:38:58.774 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:38:58.804 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:38:58.804 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:38:58.805 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:38:58.807 INFO:teuthology.orchestra.run.smithi092.stdout:osd.5: {} 2024-03-11T21:38:58.846 INFO:teuthology.orchestra.run.smithi092.stdout:osd.6: {} 2024-03-11T21:38:58.888 INFO:teuthology.orchestra.run.smithi092.stdout:osd.7: {} 2024-03-11T21:38:58.889 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:38:58 smithi092 ceph-mon[29572]: pgmap v620: 148 pgs: 3 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+degraded+remapped, 142 active+clean; 3.1 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 599 B/s rd, 0 B/s wr, 517 op/s; 22607/151371 objects degraded (14.935%); 3.8 MiB/s, 60 objects/s recovering 2024-03-11T21:38:58.889 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:38:58 smithi092 ceph-mon[29572]: osdmap e327: 8 total, 8 up, 8 in 2024-03-11T21:38:58.889 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:38:58 smithi092 ceph-mon[29572]: Health check update: Degraded data redundancy: 22607/151371 objects degraded (14.935%), 6 pgs degraded, 6 pgs undersized (PG_DEGRADED) 2024-03-11T21:38:58.889 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:38:58 smithi092 ceph-mon[29572]: osdmap e328: 8 total, 8 up, 8 in 2024-03-11T21:38:58.889 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:38:58 smithi092 ceph-mon[29572]: from='client.? v1:172.21.15.92:0/1796134137' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-11T21:38:58.896 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:38:58 smithi092 ceph-mon[37563]: pgmap v620: 148 pgs: 3 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+degraded+remapped, 142 active+clean; 3.1 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 599 B/s rd, 0 B/s wr, 517 op/s; 22607/151371 objects degraded (14.935%); 3.8 MiB/s, 60 objects/s recovering 2024-03-11T21:38:58.897 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:38:58 smithi092 ceph-mon[37563]: osdmap e327: 8 total, 8 up, 8 in 2024-03-11T21:38:58.897 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:38:58 smithi092 ceph-mon[37563]: Health check update: Degraded data redundancy: 22607/151371 objects degraded (14.935%), 6 pgs degraded, 6 pgs undersized (PG_DEGRADED) 2024-03-11T21:38:58.897 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:38:58 smithi092 ceph-mon[37563]: osdmap e328: 8 total, 8 up, 8 in 2024-03-11T21:38:58.897 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:38:58 smithi092 ceph-mon[37563]: from='client.? v1:172.21.15.92:0/1796134137' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-11T21:38:58.899 INFO:teuthology.orchestra.run.smithi092.stderr:osd.0: osd_enable_op_tracker = '' 2024-03-11T21:38:58.899 INFO:teuthology.orchestra.run.smithi092.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-11T21:38:58.899 INFO:teuthology.orchestra.run.smithi092.stderr:osd.2: osd_enable_op_tracker = '' 2024-03-11T21:38:58.899 INFO:teuthology.orchestra.run.smithi092.stderr:osd.3: osd_enable_op_tracker = '' 2024-03-11T21:38:58.900 INFO:teuthology.orchestra.run.smithi092.stderr:osd.4: osd_enable_op_tracker = '' 2024-03-11T21:38:58.900 INFO:teuthology.orchestra.run.smithi092.stderr:osd.5: osd_enable_op_tracker = '' 2024-03-11T21:38:58.900 INFO:teuthology.orchestra.run.smithi092.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-11T21:38:58.900 INFO:teuthology.orchestra.run.smithi092.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-11T21:38:58.905 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:38:58.965 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:38:58.966 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:38:58.966 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:38:59.067 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:38:59.078 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:38:58 smithi097 ceph-mon[33317]: pgmap v620: 148 pgs: 3 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+degraded+remapped, 142 active+clean; 3.1 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 599 B/s rd, 0 B/s wr, 517 op/s; 22607/151371 objects degraded (14.935%); 3.8 MiB/s, 60 objects/s recovering 2024-03-11T21:38:59.078 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:38:58 smithi097 ceph-mon[33317]: osdmap e327: 8 total, 8 up, 8 in 2024-03-11T21:38:59.078 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:38:58 smithi097 ceph-mon[33317]: Health check update: Degraded data redundancy: 22607/151371 objects degraded (14.935%), 6 pgs degraded, 6 pgs undersized (PG_DEGRADED) 2024-03-11T21:38:59.078 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:38:58 smithi097 ceph-mon[33317]: osdmap e328: 8 total, 8 up, 8 in 2024-03-11T21:38:59.078 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:38:58 smithi097 ceph-mon[33317]: from='client.? v1:172.21.15.92:0/1796134137' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-11T21:38:59.093 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:38:59.094 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:38:59.094 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:38:59.197 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:38:59.227 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:38:59.227 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:38:59.227 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:38:59.328 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:38:59.367 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:38:59.367 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:38:59.368 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:38:59.469 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:38:59.495 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:38:59.495 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:38:59.496 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:38:59.597 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:38:59.671 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:38:59.671 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:38:59.672 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:38:59.773 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:38:59.799 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:38:59.800 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:38:59.800 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:38:59.901 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:38:59.929 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:38:59.930 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:38:59.931 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:39:00.032 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:39:00.059 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:39:00.060 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:39:00.060 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:39:00.161 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:39:00.192 INFO:teuthology.orchestra.run.smithi092.stdout:ERROR: (22) Invalid argument 2024-03-11T21:39:00.192 INFO:teuthology.orchestra.run.smithi092.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-11T21:39:00.217 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:39:00.218 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:39:00.218 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:39:00.319 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:39:00.347 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:39:00.348 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:39:00.348 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:39:00.449 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:39:00.491 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:39:00.491 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:39:00.492 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:39:00.593 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:39:00.635 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:39:00.636 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:39:00.636 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:39:00.737 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:39:00.766 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:39:00.767 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:39:00.767 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:39:00.857 INFO:teuthology.orchestra.run.smithi092.stderr:noscrub is set 2024-03-11T21:39:00.868 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:39:00.895 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:39:00.895 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:39:00.896 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:39:00.984 DEBUG:teuthology.orchestra.run.smithi092:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid 00db0016-dfed-11ee-95c8-87774f69a715 -- ceph daemon osd.3 dump_blocked_ops 2024-03-11T21:39:00.996 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:39:01.025 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:39:01.026 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:39:01.026 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:39:01.112 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:39:00 smithi092 ceph-mon[29572]: pgmap v623: 148 pgs: 1 active+undersized+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+degraded+remapped, 142 active+clean; 2.8 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 1.5 KiB/s rd, 0 B/s wr, 987 op/s; 22005/139002 objects degraded (15.831%); 8.1 MiB/s, 130 objects/s recovering 2024-03-11T21:39:01.112 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:39:00 smithi092 ceph-mon[29572]: from='client.? v1:172.21.15.92:0/2271506950' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-11T21:39:01.112 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:39:00 smithi092 ceph-mon[29572]: from='client.35474 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-11T21:39:01.113 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:39:00 smithi092 ceph-mon[37563]: pgmap v623: 148 pgs: 1 active+undersized+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+degraded+remapped, 142 active+clean; 2.8 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 1.5 KiB/s rd, 0 B/s wr, 987 op/s; 22005/139002 objects degraded (15.831%); 8.1 MiB/s, 130 objects/s recovering 2024-03-11T21:39:01.113 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:39:00 smithi092 ceph-mon[37563]: from='client.? v1:172.21.15.92:0/2271506950' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-11T21:39:01.113 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:39:00 smithi092 ceph-mon[37563]: from='client.35474 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-11T21:39:01.127 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:39:01.169 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:39:01.169 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:39:01.169 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:39:01.270 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:39:01.328 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:39:00 smithi097 ceph-mon[33317]: pgmap v623: 148 pgs: 1 active+undersized+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+degraded+remapped, 142 active+clean; 2.8 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 1.5 KiB/s rd, 0 B/s wr, 987 op/s; 22005/139002 objects degraded (15.831%); 8.1 MiB/s, 130 objects/s recovering 2024-03-11T21:39:01.328 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:39:00 smithi097 ceph-mon[33317]: from='client.? v1:172.21.15.92:0/2271506950' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-11T21:39:01.328 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:39:00 smithi097 ceph-mon[33317]: from='client.35474 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-11T21:39:01.330 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:39:01.330 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:39:01.331 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:39:01.431 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:39:01.485 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:39:01.486 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:39:01.486 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:39:01.587 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:39:01.628 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:39:01.628 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:39:01.628 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:39:01.729 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:39:01.774 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:39:01.774 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:39:01.775 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:39:01.876 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:39:01.903 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:39:01.904 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:39:01.904 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:39:02.005 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:39:02.052 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:39:02.052 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:39:02.052 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:39:02.112 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:39:01 smithi092 ceph-mon[29572]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-03-11T21:39:02.112 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:39:01 smithi092 ceph-mon[29572]: from='client.35474 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-03-11T21:39:02.112 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:39:01 smithi092 ceph-mon[29572]: osdmap e329: 8 total, 8 up, 8 in 2024-03-11T21:39:02.113 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:39:01 smithi092 ceph-mon[37563]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-03-11T21:39:02.113 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:39:01 smithi092 ceph-mon[37563]: from='client.35474 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-03-11T21:39:02.113 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:39:01 smithi092 ceph-mon[37563]: osdmap e329: 8 total, 8 up, 8 in 2024-03-11T21:39:02.153 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:39:02.202 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:39:02.203 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:39:02.203 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:39:02.304 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:39:02.328 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:39:01 smithi097 ceph-mon[33317]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-03-11T21:39:02.328 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:39:01 smithi097 ceph-mon[33317]: from='client.35474 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-03-11T21:39:02.328 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:39:01 smithi097 ceph-mon[33317]: osdmap e329: 8 total, 8 up, 8 in 2024-03-11T21:39:02.341 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:39:02.342 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:39:02.343 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:39:02.444 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:39:02.488 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:39:02.489 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:39:02.489 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:39:02.556 INFO:teuthology.orchestra.run.smithi092.stderr:Inferring config /var/lib/ceph/00db0016-dfed-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T21:39:02.590 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:39:02.657 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:39:02.657 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:39:02.658 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:39:02.759 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:39:02.803 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:39:02.804 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:39:02.804 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:39:02.905 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:39:02.933 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:39:02.934 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:39:02.935 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:39:03.033 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:39:02 smithi092 ceph-mon[29572]: pgmap v625: 148 pgs: 1 active+undersized+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+degraded+remapped, 142 active+clean; 2.8 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 853 B/s rd, 0 B/s wr, 687 op/s; 22005/139002 objects degraded (15.831%); 6.3 MiB/s, 100 objects/s recovering 2024-03-11T21:39:03.033 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:39:02 smithi092 ceph-mon[37563]: pgmap v625: 148 pgs: 1 active+undersized+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+degraded+remapped, 142 active+clean; 2.8 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 853 B/s rd, 0 B/s wr, 687 op/s; 22005/139002 objects degraded (15.831%); 6.3 MiB/s, 100 objects/s recovering 2024-03-11T21:39:03.035 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:39:03.078 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:39:03.078 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:39:03.078 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:39:03.079 INFO:journalctl@ceph.iscsi.iscsi.a.smithi097.stdout:Mar 11 21:39:02 smithi097 ceph-00db0016-dfed-11ee-95c8-87774f69a715-iscsi-iscsi-a[60943]: debug there is no tcmu-runner data available 2024-03-11T21:39:03.079 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:39:02 smithi097 ceph-mon[33317]: pgmap v625: 148 pgs: 1 active+undersized+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+degraded+remapped, 142 active+clean; 2.8 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 853 B/s rd, 0 B/s wr, 687 op/s; 22005/139002 objects degraded (15.831%); 6.3 MiB/s, 100 objects/s recovering 2024-03-11T21:39:03.112 INFO:teuthology.orchestra.run.smithi092.stdout: 2024-03-11T21:39:03.113 INFO:teuthology.orchestra.run.smithi092.stdout:{"fsid":"00db0016-dfed-11ee-95c8-87774f69a715","health":{"status":"HEALTH_WARN","checks":{"OSDMAP_FLAGS":{"severity":"HEALTH_WARN","summary":{"message":"noscrub flag(s) set","count":7},"muted":false},"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 22005/139002 objects degraded (15.831%), 5 pgs degraded, 6 pgs undersized","count":11},"muted":false}},"mutes":[]},"election_epoch":14,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":1025,"monmap":{"epoch":3,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":329,"num_osds":8,"num_up_osds":8,"osd_up_since":1710192994,"num_in_osds":8,"osd_in_since":1710192308,"num_remapped_pgs":5},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":142},{"state_name":"active+recovering+undersized+degraded+remapped","count":3},{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":2},{"state_name":"active+undersized+remapped","count":1}],"num_pgs":148,"num_pools":7,"num_objects":46334,"data_bytes":3024099776,"bytes_used":12654288896,"bytes_avail":755305996288,"bytes_total":767960285184,"degraded_objects":22005,"degraded_total":139002,"degraded_ratio":0.15830707471834937,"recovering_objects_per_sec":100,"recovering_bytes_per_sec":6573300,"recovering_keys_per_sec":0,"num_objects_recovered":602,"num_bytes_recovered":39452672,"num_keys_recovered":0,"read_bytes_sec":853,"write_bytes_sec":0,"read_op_per_sec":0,"write_op_per_sec":686},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":1,"modules":["cephadm","dashboard","iostat","nfs","prometheus","restful"],"services":{"dashboard":"https://172.21.15.92:8443/","prometheus":"http://172.21.15.92:9283/"}},"servicemap":{"epoch":15,"modified":"2024-03-11T21:37:33.231049+0000","services":{"rgw":{"daemons":{"summary":"","34352":{"start_epoch":12,"start_stamp":"2024-03-11T21:25:45.672905+0000","gid":34352,"addr":"172.21.15.92:0/372197479","metadata":{"arch":"x86_64","ceph_release":"squid","ceph_version":"ceph version 19.0.0-1979-g2b0101eb (2b0101eb74d60610d544cb8a473e23a35ac1cc0b) squid (dev)","ceph_version_short":"19.0.0-1979-g2b0101eb","container_hostname":"smithi092","container_image":"quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:8fe9635f3beb0247496d7e321f3068bd10c333c433ef4824d7e1c1a7d3d78379","cpu":"Intel(R) Xeon(R) CPU E5-1620 v4 @ 3.50GHz","distro":"centos","distro_description":"CentOS Stream 8","distro_version":"8","frontend_config#0":"beast port=80","frontend_type#0":"beast","hostname":"smithi092","id":"foo.a","kernel_description":"#1 SMP PREEMPT_DYNAMIC Fri Feb 23 04:45:07 UTC 2024","kernel_version":"5.14.0-427.el9.x86_64","mem_swap_kb":"0","mem_total_kb":"32488440","num_handles":"1","os":"Linux","pid":"2","realm_id":"","realm_name":"","zone_id":"bf7aa506-b210-476a-b68a-1feac2caf199","zone_name":"default","zonegroup_id":"33e24078-41e3-4494-bd9e-6ff3741afaba","zonegroup_name":"default"},"task_status":{}}}}}},"progress_events":{}} 2024-03-11T21:39:03.179 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:39:03.209 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:39:03.209 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:39:03.210 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:39:03.310 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:39:03.337 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:39:03.338 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:39:03.339 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:39:03.439 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:39:03.468 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:39:03.469 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:39:03.469 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:39:03.570 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:39:03.802 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:39:03.803 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:39:03.803 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:39:03.841 DEBUG:teuthology.orchestra.run.smithi092:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid 00db0016-dfed-11ee-95c8-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-03-11T21:39:03.904 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:39:03.931 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:39:03.931 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:39:03.932 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:39:04.033 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:39:04.063 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:39:04.063 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:39:04.064 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:39:04.165 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:39:04.192 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:39:04.193 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:39:04.193 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:39:04.274 DEBUG:teuthology.orchestra.run.smithi092:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid 00db0016-dfed-11ee-95c8-87774f69a715 -- ceph osd set nodeep-scrub 2024-03-11T21:39:04.294 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:39:04.317 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:39:03 smithi092 ceph-mon[29572]: from='client.24469 v1:172.21.15.97:0/404208310' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:39:04.317 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:39:03 smithi092 ceph-mon[29572]: from='client.? v1:172.21.15.92:0/610863027' entity='client.admin' cmd=[{"prefix": "status", "format": "json"}]: dispatch 2024-03-11T21:39:04.317 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:39:03 smithi092 ceph-mon[29572]: Health check update: Degraded data redundancy: 22005/139002 objects degraded (15.831%), 5 pgs degraded, 6 pgs undersized (PG_DEGRADED) 2024-03-11T21:39:04.317 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:39:03 smithi092 ceph-mon[29572]: osdmap e330: 8 total, 8 up, 8 in 2024-03-11T21:39:04.317 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:39:03 smithi092 ceph-mon[37563]: from='client.24469 v1:172.21.15.97:0/404208310' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:39:04.318 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:39:03 smithi092 ceph-mon[37563]: from='client.? v1:172.21.15.92:0/610863027' entity='client.admin' cmd=[{"prefix": "status", "format": "json"}]: dispatch 2024-03-11T21:39:04.318 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:39:03 smithi092 ceph-mon[37563]: Health check update: Degraded data redundancy: 22005/139002 objects degraded (15.831%), 5 pgs degraded, 6 pgs undersized (PG_DEGRADED) 2024-03-11T21:39:04.318 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:39:03 smithi092 ceph-mon[37563]: osdmap e330: 8 total, 8 up, 8 in 2024-03-11T21:39:04.323 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:39:04.324 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:39:04.324 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:39:04.328 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:39:03 smithi097 ceph-mon[33317]: from='client.24469 v1:172.21.15.97:0/404208310' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:39:04.328 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:39:03 smithi097 ceph-mon[33317]: from='client.? v1:172.21.15.92:0/610863027' entity='client.admin' cmd=[{"prefix": "status", "format": "json"}]: dispatch 2024-03-11T21:39:04.328 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:39:03 smithi097 ceph-mon[33317]: Health check update: Degraded data redundancy: 22005/139002 objects degraded (15.831%), 5 pgs degraded, 6 pgs undersized (PG_DEGRADED) 2024-03-11T21:39:04.328 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:39:03 smithi097 ceph-mon[33317]: osdmap e330: 8 total, 8 up, 8 in 2024-03-11T21:39:04.425 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:39:04.453 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:39:04.453 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:39:04.454 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:39:04.555 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:39:04.580 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:39:04.580 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:39:04.581 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:39:04.681 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:39:04.712 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:39:04.713 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:39:04.713 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:39:04.814 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:39:04.842 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:39:04.843 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:39:04.843 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:39:04.944 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:39:04.971 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:39:04.971 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:39:04.972 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:39:05.073 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:39:05.104 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:39:05.105 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:39:05.105 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:39:05.206 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:39:05.234 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:39:05.235 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:39:05.235 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:39:05.328 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:39:04 smithi097 ceph-mon[33317]: pgmap v626: 148 pgs: 2 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+degraded+remapped, 143 active+clean; 2.7 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 1.7 KiB/s rd, 0 B/s wr, 1.07k op/s; 21718/132141 objects degraded (16.435%); 9.3 MiB/s, 148 objects/s recovering 2024-03-11T21:39:05.335 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:39:05.360 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:39:05.360 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:39:05.361 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:39:05.362 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:39:04 smithi092 ceph-mon[29572]: pgmap v626: 148 pgs: 2 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+degraded+remapped, 143 active+clean; 2.7 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 1.7 KiB/s rd, 0 B/s wr, 1.07k op/s; 21718/132141 objects degraded (16.435%); 9.3 MiB/s, 148 objects/s recovering 2024-03-11T21:39:05.362 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:39:04 smithi092 ceph-mon[37563]: pgmap v626: 148 pgs: 2 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+degraded+remapped, 143 active+clean; 2.7 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 1.7 KiB/s rd, 0 B/s wr, 1.07k op/s; 21718/132141 objects degraded (16.435%); 9.3 MiB/s, 148 objects/s recovering 2024-03-11T21:39:05.462 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:39:05.485 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:39:05.486 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:39:05.486 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:39:05.587 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:39:05.618 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:39:05.619 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:39:05.619 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:39:05.720 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:39:05.743 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:39:05.744 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:39:05.744 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:39:05.845 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:39:05.872 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:39:05.873 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:39:05.873 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:39:05.974 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:39:06.011 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:39:06.012 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:39:06.012 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:39:06.113 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:39:06.178 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:39:06.179 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:39:06.179 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:39:06.280 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:39:06.330 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:39:06.330 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:39:06.330 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:39:06.434 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:39:06.486 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:39:06.487 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:39:06.487 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:39:06.588 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:39:06.618 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:39:06.618 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:39:06.619 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:39:06.719 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:39:06.766 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:39:06.766 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:39:06.767 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:39:06.868 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:39:06.898 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:39:06.899 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:39:06.899 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:39:07.000 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:39:07.041 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:39:07.042 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:39:07.042 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:39:07.112 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:39:06 smithi092 ceph-mon[29572]: pgmap v628: 148 pgs: 2 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+degraded+remapped, 143 active+clean; 2.6 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 738 B/s rd, 0 B/s wr, 772 op/s; 21597/127416 objects degraded (16.950%); 4.1 MiB/s, 65 objects/s recovering 2024-03-11T21:39:07.113 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:39:06 smithi092 ceph-mon[37563]: pgmap v628: 148 pgs: 2 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+degraded+remapped, 143 active+clean; 2.6 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 738 B/s rd, 0 B/s wr, 772 op/s; 21597/127416 objects degraded (16.950%); 4.1 MiB/s, 65 objects/s recovering 2024-03-11T21:39:07.143 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:39:07.185 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:39:07.185 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:39:07.186 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:39:07.287 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:39:07.328 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:39:06 smithi097 ceph-mon[33317]: pgmap v628: 148 pgs: 2 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+degraded+remapped, 143 active+clean; 2.6 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 738 B/s rd, 0 B/s wr, 772 op/s; 21597/127416 objects degraded (16.950%); 4.1 MiB/s, 65 objects/s recovering 2024-03-11T21:39:07.330 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:39:07.331 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:39:07.331 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:39:07.432 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:39:07.459 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:39:07.460 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:39:07.460 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:39:07.561 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:39:07.603 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:39:07.604 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:39:07.604 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:39:07.705 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:39:07.743 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:39:07.743 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:39:07.744 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:39:07.844 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:39:07.872 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:39:07.872 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:39:07.873 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:39:07.973 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:39:08.002 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:39:08.002 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:39:08.003 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:39:08.035 INFO:tasks.cephadm.ceph_manager.ceph:making progress, resetting timeout 2024-03-11T21:39:08.035 DEBUG:teuthology.orchestra.run.smithi092:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid 00db0016-dfed-11ee-95c8-87774f69a715 -- ceph pg dump --format=json 2024-03-11T21:39:08.080 INFO:journalctl@ceph.mgr.y.smithi092.stdout:Mar 11 21:39:07 smithi092 ceph-00db0016-dfed-11ee-95c8-87774f69a715-mgr-y[29801]: ::ffff:172.21.15.97 - - [11/Mar/2024:21:39:07] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-11T21:39:08.104 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:39:08.127 INFO:teuthology.orchestra.run.smithi092.stdout:ERROR: (22) Invalid argument 2024-03-11T21:39:08.127 INFO:teuthology.orchestra.run.smithi092.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-11T21:39:08.132 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:39:08.133 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:39:08.134 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:39:08.234 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:39:08.265 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:39:08.265 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:39:08.266 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:39:08.366 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:39:08.392 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:39:08.392 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:39:08.393 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:39:08.417 INFO:teuthology.orchestra.run.smithi092.stderr:Inferring config /var/lib/ceph/00db0016-dfed-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T21:39:08.494 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:39:08.547 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:39:08.547 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:39:08.548 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:39:08.649 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:39:08.705 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:39:08.705 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:39:08.705 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:39:08.806 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:39:08.828 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:39:08 smithi097 ceph-mon[33317]: pgmap v629: 148 pgs: 2 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+degraded+remapped, 143 active+clean; 2.6 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 639 B/s rd, 0 B/s wr, 483 op/s; 21597/127416 objects degraded (16.950%); 3.2 MiB/s, 50 objects/s recovering 2024-03-11T21:39:08.828 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:39:08 smithi097 ceph-mon[33317]: from='mgr.14607 ' entity='mgr.y' 2024-03-11T21:39:08.828 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:39:08 smithi097 ceph-mon[33317]: from='mgr.14607 v1:172.21.15.92:0/1702717597' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T21:39:08.828 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:39:08 smithi097 ceph-mon[33317]: Health check update: Degraded data redundancy: 21597/127416 objects degraded (16.950%), 5 pgs degraded, 5 pgs undersized (PG_DEGRADED) 2024-03-11T21:39:08.835 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:39:08.836 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:39:08.836 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:39:08.863 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:39:08 smithi092 ceph-mon[29572]: pgmap v629: 148 pgs: 2 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+degraded+remapped, 143 active+clean; 2.6 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 639 B/s rd, 0 B/s wr, 483 op/s; 21597/127416 objects degraded (16.950%); 3.2 MiB/s, 50 objects/s recovering 2024-03-11T21:39:08.863 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:39:08 smithi092 ceph-mon[29572]: from='mgr.14607 ' entity='mgr.y' 2024-03-11T21:39:08.863 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:39:08 smithi092 ceph-mon[29572]: from='mgr.14607 v1:172.21.15.92:0/1702717597' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T21:39:08.863 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:39:08 smithi092 ceph-mon[29572]: Health check update: Degraded data redundancy: 21597/127416 objects degraded (16.950%), 5 pgs degraded, 5 pgs undersized (PG_DEGRADED) 2024-03-11T21:39:08.863 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:39:08 smithi092 ceph-mon[37563]: pgmap v629: 148 pgs: 2 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+degraded+remapped, 143 active+clean; 2.6 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 639 B/s rd, 0 B/s wr, 483 op/s; 21597/127416 objects degraded (16.950%); 3.2 MiB/s, 50 objects/s recovering 2024-03-11T21:39:08.863 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:39:08 smithi092 ceph-mon[37563]: from='mgr.14607 ' entity='mgr.y' 2024-03-11T21:39:08.863 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:39:08 smithi092 ceph-mon[37563]: from='mgr.14607 v1:172.21.15.92:0/1702717597' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T21:39:08.863 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:39:08 smithi092 ceph-mon[37563]: Health check update: Degraded data redundancy: 21597/127416 objects degraded (16.950%), 5 pgs degraded, 5 pgs undersized (PG_DEGRADED) 2024-03-11T21:39:08.937 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:39:08.967 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:39:08.969 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:39:08.970 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:39:09.068 DEBUG:teuthology.orchestra.run.smithi092:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid 00db0016-dfed-11ee-95c8-87774f69a715 -- ceph daemon osd.3 dump_historic_ops 2024-03-11T21:39:09.070 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:39:09.140 INFO:teuthology.orchestra.run.smithi092.stderr:Inferring config /var/lib/ceph/00db0016-dfed-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T21:39:09.202 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:39:09.202 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:39:09.202 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:39:09.303 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:39:09.357 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:39:09.357 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:39:09.358 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:39:09.458 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:39:09.501 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:39:09.502 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:39:09.502 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:39:09.603 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:39:09.641 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:39:09.642 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:39:09.642 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:39:09.743 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:39:09.790 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:39:09.791 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:39:09.791 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:39:09.892 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:39:09.920 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:39:09.921 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:39:09.921 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:39:10.022 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:39:10.050 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:39:10.051 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:39:10.051 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:39:10.152 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:39:10.178 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:39:10.178 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:39:10.179 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:39:10.279 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:39:10.319 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:39:10.320 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:39:10.320 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:39:10.421 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:39:10.446 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:39:10.447 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:39:10.447 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:39:10.548 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:39:10.571 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:39:10.571 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:39:10.572 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:39:10.612 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:39:10 smithi092 ceph-mon[29572]: pgmap v630: 148 pgs: 2 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+degraded+remapped, 143 active+clean; 2.3 GiB data, 10 GiB used, 705 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 923 op/s; 21196/115680 objects degraded (18.323%); 4.5 MiB/s, 95 objects/s recovering 2024-03-11T21:39:10.612 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:39:10 smithi092 ceph-mon[37563]: pgmap v630: 148 pgs: 2 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+degraded+remapped, 143 active+clean; 2.3 GiB data, 10 GiB used, 705 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 923 op/s; 21196/115680 objects degraded (18.323%); 4.5 MiB/s, 95 objects/s recovering 2024-03-11T21:39:10.673 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:39:10.709 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:39:10.709 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:39:10.710 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:39:10.810 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:39:10.828 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:39:10 smithi097 ceph-mon[33317]: pgmap v630: 148 pgs: 2 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+degraded+remapped, 143 active+clean; 2.3 GiB data, 10 GiB used, 705 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 923 op/s; 21196/115680 objects degraded (18.323%); 4.5 MiB/s, 95 objects/s recovering 2024-03-11T21:39:10.837 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:39:10.838 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:39:10.838 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:39:10.939 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:39:10.979 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:39:10.980 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:39:10.980 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:39:11.081 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:39:11.113 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:39:11.113 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:39:11.113 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:39:11.214 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:39:11.251 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:39:11.252 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:39:11.252 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:39:11.353 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:39:11.392 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:39:11.393 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:39:11.393 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:39:11.494 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:39:11.522 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:39:11.522 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:39:11.522 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:39:11.623 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:39:11.654 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:39:11.654 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:39:11.654 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:39:11.755 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:39:11.782 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:39:11.782 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:39:11.783 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:39:11.884 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:39:11.913 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:39:11.913 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:39:11.914 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:39:12.015 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:39:12.045 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:39:12.046 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:39:12.046 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:39:12.147 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:39:12.186 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:39:12.187 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:39:12.187 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:39:12.288 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:39:12.328 INFO:teuthology.orchestra.run.smithi092.stderr:Inferring config /var/lib/ceph/00db0016-dfed-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T21:39:12.330 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:39:12.330 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:39:12.330 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:39:12.386 INFO:teuthology.orchestra.run.smithi092.stderr:Inferring config /var/lib/ceph/00db0016-dfed-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T21:39:12.431 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:39:12.457 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:39:12.458 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:39:12.458 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:39:12.559 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:39:12.588 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:39:12.589 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:39:12.589 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:39:12.612 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:39:12 smithi092 ceph-mon[29572]: pgmap v631: 148 pgs: 2 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+degraded+remapped, 143 active+clean; 2.3 GiB data, 10 GiB used, 705 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 778 op/s; 21196/115680 objects degraded (18.323%); 3.8 MiB/s, 80 objects/s recovering 2024-03-11T21:39:12.613 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:39:12 smithi092 ceph-mon[37563]: pgmap v631: 148 pgs: 2 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+degraded+remapped, 143 active+clean; 2.3 GiB data, 10 GiB used, 705 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 778 op/s; 21196/115680 objects degraded (18.323%); 3.8 MiB/s, 80 objects/s recovering 2024-03-11T21:39:12.690 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:39:12.731 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:39:12.731 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:39:12.731 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:39:12.828 INFO:journalctl@ceph.iscsi.iscsi.a.smithi097.stdout:Mar 11 21:39:12 smithi097 ceph-00db0016-dfed-11ee-95c8-87774f69a715-iscsi-iscsi-a[60943]: debug there is no tcmu-runner data available 2024-03-11T21:39:12.828 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:39:12 smithi097 ceph-mon[33317]: pgmap v631: 148 pgs: 2 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+degraded+remapped, 143 active+clean; 2.3 GiB data, 10 GiB used, 705 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 778 op/s; 21196/115680 objects degraded (18.323%); 3.8 MiB/s, 80 objects/s recovering 2024-03-11T21:39:12.832 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:39:12.861 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:39:12.862 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:39:12.862 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:39:12.963 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:39:12.993 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:39:12.994 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:39:12.994 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:39:13.095 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:39:13.121 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:39:13.122 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:39:13.122 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:39:13.223 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:39:13.263 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:39:13.264 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:39:13.264 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:39:13.365 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:39:13.392 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:39:13.393 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:39:13.393 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:39:13.494 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:39:13.525 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:39:13.525 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:39:13.526 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:39:13.626 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:39:13.656 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:39:13.657 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:39:13.657 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:39:13.758 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:39:13.782 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:39:13.783 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:39:13.783 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:39:13.828 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:39:13 smithi097 ceph-mon[33317]: from='client.24469 v1:172.21.15.97:0/404208310' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:39:13.828 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:39:13 smithi097 ceph-mon[33317]: Health check update: Degraded data redundancy: 21196/115680 objects degraded (18.323%), 5 pgs degraded, 5 pgs undersized (PG_DEGRADED) 2024-03-11T21:39:13.862 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:39:13 smithi092 ceph-mon[29572]: from='client.24469 v1:172.21.15.97:0/404208310' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:39:13.862 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:39:13 smithi092 ceph-mon[29572]: Health check update: Degraded data redundancy: 21196/115680 objects degraded (18.323%), 5 pgs degraded, 5 pgs undersized (PG_DEGRADED) 2024-03-11T21:39:13.862 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:39:13 smithi092 ceph-mon[37563]: from='client.24469 v1:172.21.15.97:0/404208310' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:39:13.863 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:39:13 smithi092 ceph-mon[37563]: Health check update: Degraded data redundancy: 21196/115680 objects degraded (18.323%), 5 pgs degraded, 5 pgs undersized (PG_DEGRADED) 2024-03-11T21:39:13.884 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:39:13.913 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:39:13.914 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:39:13.914 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:39:14.015 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:39:14.062 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:39:14.062 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:39:14.063 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:39:14.164 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:39:14.200 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:39:14.201 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:39:14.201 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:39:14.302 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:39:14.335 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:39:14.335 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:39:14.336 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:39:14.437 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:39:14.477 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:39:14.478 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:39:14.478 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:39:14.579 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:39:14.623 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:39:14.623 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:39:14.623 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:39:14.724 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:39:14.762 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:39:14.762 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:39:14.763 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:39:14.828 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:39:14 smithi097 ceph-mon[33317]: pgmap v632: 148 pgs: 2 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+degraded+remapped, 143 active+clean; 2.2 GiB data, 10 GiB used, 705 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 811 op/s; 20899/107805 objects degraded (19.386%); 2.0 MiB/s, 81 objects/s recovering 2024-03-11T21:39:14.862 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:39:14 smithi092 ceph-mon[29572]: pgmap v632: 148 pgs: 2 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+degraded+remapped, 143 active+clean; 2.2 GiB data, 10 GiB used, 705 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 811 op/s; 20899/107805 objects degraded (19.386%); 2.0 MiB/s, 81 objects/s recovering 2024-03-11T21:39:14.863 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:39:14 smithi092 ceph-mon[37563]: pgmap v632: 148 pgs: 2 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+degraded+remapped, 143 active+clean; 2.2 GiB data, 10 GiB used, 705 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 811 op/s; 20899/107805 objects degraded (19.386%); 2.0 MiB/s, 81 objects/s recovering 2024-03-11T21:39:14.863 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:39:14.891 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:39:14.891 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:39:14.892 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:39:14.993 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:39:15.023 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:39:15.024 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:39:15.024 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:39:15.125 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:39:15.153 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:39:15.153 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:39:15.154 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:39:15.254 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:39:15.283 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:39:15.284 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:39:15.284 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:39:15.385 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:39:15.420 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:39:15.420 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:39:15.421 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:39:15.521 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:39:15.550 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:39:15.551 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:39:15.551 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:39:15.652 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:39:15.678 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:39:15.679 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:39:15.679 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:39:15.780 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:39:15.831 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:39:15.835 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:39:15.836 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:39:15.936 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:39:15.967 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:39:15.967 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:39:15.967 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:39:16.069 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:39:16.113 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:39:16.114 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:39:16.114 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:39:16.215 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:39:16.258 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:39:16.258 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:39:16.258 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:39:16.359 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:39:16.388 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:39:16.389 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:39:16.389 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:39:16.491 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:39:16.531 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:39:16.531 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:39:16.532 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:39:16.632 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:39:16.661 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:39:16.662 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:39:16.662 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:39:16.763 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:39:16.792 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:39:16.792 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:39:16.793 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:39:16.828 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:39:16 smithi097 ceph-mon[33317]: pgmap v633: 148 pgs: 2 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+degraded+remapped, 143 active+clean; 2.1 GiB data, 9.8 GiB used, 705 GiB / 715 GiB avail; 858 B/s rd, 0 B/s wr, 816 op/s; 20783/102933 objects degraded (20.191%); 1.7 MiB/s, 78 objects/s recovering 2024-03-11T21:39:16.862 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:39:16 smithi092 ceph-mon[29572]: pgmap v633: 148 pgs: 2 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+degraded+remapped, 143 active+clean; 2.1 GiB data, 9.8 GiB used, 705 GiB / 715 GiB avail; 858 B/s rd, 0 B/s wr, 816 op/s; 20783/102933 objects degraded (20.191%); 1.7 MiB/s, 78 objects/s recovering 2024-03-11T21:39:16.863 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:39:16 smithi092 ceph-mon[37563]: pgmap v633: 148 pgs: 2 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+degraded+remapped, 143 active+clean; 2.1 GiB data, 9.8 GiB used, 705 GiB / 715 GiB avail; 858 B/s rd, 0 B/s wr, 816 op/s; 20783/102933 objects degraded (20.191%); 1.7 MiB/s, 78 objects/s recovering 2024-03-11T21:39:16.894 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:39:16.934 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:39:16.935 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:39:16.935 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:39:17.036 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:39:17.065 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:39:17.065 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:39:17.066 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:39:17.167 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:39:17.203 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:39:17.204 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:39:17.204 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:39:17.305 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:39:17.342 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:39:17.343 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:39:17.343 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:39:17.444 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:39:17.485 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:39:17.486 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:39:17.487 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:39:17.587 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:39:17.615 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:39:17.615 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:39:17.615 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:39:17.716 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:39:17.753 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:39:17.753 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:39:17.754 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:39:17.854 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:39:17.885 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:39:17.886 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:39:17.886 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:39:17.987 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:39:18.023 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:39:18.023 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:39:18.023 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:39:18.124 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:39:18.150 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:39:18.151 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:39:18.151 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:39:18.252 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:39:18.276 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:39:18.277 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:39:18.277 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:39:18.362 INFO:journalctl@ceph.mgr.y.smithi092.stdout:Mar 11 21:39:17 smithi092 ceph-00db0016-dfed-11ee-95c8-87774f69a715-mgr-y[29801]: ::ffff:172.21.15.97 - - [11/Mar/2024:21:39:17] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-11T21:39:18.377 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:39:18.420 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:39:18.421 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:39:18.421 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:39:18.522 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:39:18.562 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:39:18.563 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:39:18.564 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:39:18.665 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:39:18.706 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:39:18.706 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:39:18.707 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:39:18.807 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:39:18.828 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:39:18 smithi097 ceph-mon[33317]: pgmap v634: 148 pgs: 2 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+degraded+remapped, 143 active+clean; 2.1 GiB data, 9.8 GiB used, 705 GiB / 715 GiB avail; 853 B/s rd, 0 B/s wr, 680 op/s; 20783/102933 objects degraded (20.191%); 1.1 MiB/s, 67 objects/s recovering 2024-03-11T21:39:18.828 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:39:18 smithi097 ceph-mon[33317]: Health check update: Degraded data redundancy: 20783/102933 objects degraded (20.191%), 5 pgs degraded, 5 pgs undersized (PG_DEGRADED) 2024-03-11T21:39:18.845 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:39:18.845 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:39:18.846 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:39:18.862 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:39:18 smithi092 ceph-mon[29572]: pgmap v634: 148 pgs: 2 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+degraded+remapped, 143 active+clean; 2.1 GiB data, 9.8 GiB used, 705 GiB / 715 GiB avail; 853 B/s rd, 0 B/s wr, 680 op/s; 20783/102933 objects degraded (20.191%); 1.1 MiB/s, 67 objects/s recovering 2024-03-11T21:39:18.863 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:39:18 smithi092 ceph-mon[29572]: Health check update: Degraded data redundancy: 20783/102933 objects degraded (20.191%), 5 pgs degraded, 5 pgs undersized (PG_DEGRADED) 2024-03-11T21:39:18.863 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:39:18 smithi092 ceph-mon[37563]: pgmap v634: 148 pgs: 2 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+degraded+remapped, 143 active+clean; 2.1 GiB data, 9.8 GiB used, 705 GiB / 715 GiB avail; 853 B/s rd, 0 B/s wr, 680 op/s; 20783/102933 objects degraded (20.191%); 1.1 MiB/s, 67 objects/s recovering 2024-03-11T21:39:18.863 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:39:18 smithi092 ceph-mon[37563]: Health check update: Degraded data redundancy: 20783/102933 objects degraded (20.191%), 5 pgs degraded, 5 pgs undersized (PG_DEGRADED) 2024-03-11T21:39:18.947 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:39:18.977 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:39:18.977 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:39:18.978 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:39:19.078 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:39:19.122 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:39:19.123 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:39:19.123 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:39:19.224 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:39:19.251 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:39:19.252 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:39:19.252 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:39:19.353 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:39:19.386 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:39:19.386 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:39:19.387 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:39:19.488 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:39:19.527 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:39:19.527 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:39:19.527 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:39:19.628 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:39:19.667 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:39:19.668 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:39:19.668 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:39:19.769 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:39:19.799 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:39:19.800 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:39:19.800 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:39:19.901 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:39:19.941 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:39:19.942 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:39:19.942 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:39:20.043 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:39:20.067 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:39:20.068 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:39:20.068 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:39:20.169 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:39:20.197 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:39:20.197 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:39:20.198 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:39:20.299 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:39:20.335 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:39:20.335 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:39:20.336 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:39:20.436 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:39:20.461 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:39:20.461 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:39:20.462 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:39:20.562 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:39:20.589 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:39:20.590 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:39:20.591 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:39:20.682 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:39:20 smithi092 ceph-mon[29572]: pgmap v635: 148 pgs: 2 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+degraded+remapped, 143 active+clean; 1.8 GiB data, 9.0 GiB used, 706 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.03k op/s; 20382/90159 objects degraded (22.607%); 1.1 MiB/s, 101 objects/s recovering 2024-03-11T21:39:20.683 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:39:20 smithi092 ceph-mon[29572]: from='client.? v1:172.21.15.92:0/875393691' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-11T21:39:20.683 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:39:20 smithi092 ceph-mon[37563]: pgmap v635: 148 pgs: 2 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+degraded+remapped, 143 active+clean; 1.8 GiB data, 9.0 GiB used, 706 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.03k op/s; 20382/90159 objects degraded (22.607%); 1.1 MiB/s, 101 objects/s recovering 2024-03-11T21:39:20.683 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:39:20 smithi092 ceph-mon[37563]: from='client.? v1:172.21.15.92:0/875393691' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-11T21:39:20.691 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:39:20.723 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:39:20.724 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:39:20.724 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:39:20.825 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:39:20.828 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:39:20 smithi097 ceph-mon[33317]: pgmap v635: 148 pgs: 2 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+degraded+remapped, 143 active+clean; 1.8 GiB data, 9.0 GiB used, 706 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.03k op/s; 20382/90159 objects degraded (22.607%); 1.1 MiB/s, 101 objects/s recovering 2024-03-11T21:39:20.828 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:39:20 smithi097 ceph-mon[33317]: from='client.? v1:172.21.15.92:0/875393691' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-11T21:39:20.853 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:39:20.854 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:39:20.854 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:39:20.912 INFO:teuthology.orchestra.run.smithi092.stdout:osd.0: {} 2024-03-11T21:39:20.913 INFO:teuthology.orchestra.run.smithi092.stdout:osd.1: {} 2024-03-11T21:39:20.913 INFO:teuthology.orchestra.run.smithi092.stdout:osd.2: {} 2024-03-11T21:39:20.913 INFO:teuthology.orchestra.run.smithi092.stdout:osd.3: {} 2024-03-11T21:39:20.913 INFO:teuthology.orchestra.run.smithi092.stdout:osd.4: {} 2024-03-11T21:39:20.913 INFO:teuthology.orchestra.run.smithi092.stdout:osd.5: {} 2024-03-11T21:39:20.913 INFO:teuthology.orchestra.run.smithi092.stdout:osd.6: {} 2024-03-11T21:39:20.913 INFO:teuthology.orchestra.run.smithi092.stdout:osd.7: {} 2024-03-11T21:39:20.913 INFO:teuthology.orchestra.run.smithi092.stderr:osd.0: osd_enable_op_tracker = '' 2024-03-11T21:39:20.913 INFO:teuthology.orchestra.run.smithi092.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-11T21:39:20.913 INFO:teuthology.orchestra.run.smithi092.stderr:osd.2: osd_enable_op_tracker = '' 2024-03-11T21:39:20.914 INFO:teuthology.orchestra.run.smithi092.stderr:osd.3: osd_enable_op_tracker = '' 2024-03-11T21:39:20.914 INFO:teuthology.orchestra.run.smithi092.stderr:osd.4: osd_enable_op_tracker = '' 2024-03-11T21:39:20.914 INFO:teuthology.orchestra.run.smithi092.stderr:osd.5: osd_enable_op_tracker = '' 2024-03-11T21:39:20.914 INFO:teuthology.orchestra.run.smithi092.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-11T21:39:20.914 INFO:teuthology.orchestra.run.smithi092.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-11T21:39:20.955 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:39:20.978 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:39:20.979 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:39:20.979 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:39:21.080 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:39:21.111 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:39:21.112 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:39:21.113 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:39:21.213 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:39:21.248 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:39:21.249 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:39:21.249 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:39:21.350 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:39:21.389 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:39:21.390 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:39:21.390 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:39:21.491 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:39:21.534 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:39:21.535 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:39:21.535 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:39:21.636 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:39:21.678 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:39:21.678 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:39:21.679 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:39:21.780 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:39:21.820 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:39:21.820 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:39:21.821 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:39:21.922 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:39:21.946 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:39:21.946 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:39:21.947 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:39:22.047 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:39:22.085 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:39:22.085 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:39:22.086 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:39:22.187 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:39:22.227 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:39:22.228 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:39:22.228 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:39:22.329 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:39:22.376 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:39:22.377 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:39:22.377 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:39:22.478 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:39:22.509 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:39:22.510 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:39:22.510 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:39:22.611 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:39:22.641 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:39:22.641 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:39:22.642 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:39:22.743 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:39:22.786 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:39:22.787 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:39:22.787 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:39:22.828 INFO:journalctl@ceph.iscsi.iscsi.a.smithi097.stdout:Mar 11 21:39:22 smithi097 ceph-00db0016-dfed-11ee-95c8-87774f69a715-iscsi-iscsi-a[60943]: debug there is no tcmu-runner data available 2024-03-11T21:39:22.829 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:39:22 smithi097 ceph-mon[33317]: pgmap v636: 148 pgs: 2 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+degraded+remapped, 143 active+clean; 1.8 GiB data, 9.0 GiB used, 706 GiB / 715 GiB avail; 853 B/s rd, 0 B/s wr, 709 op/s; 20382/90159 objects degraded (22.607%); 0 B/s, 67 objects/s recovering 2024-03-11T21:39:22.829 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:39:22 smithi097 ceph-mon[33317]: from='mgr.14607 v1:172.21.15.92:0/1702717597' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T21:39:22.862 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:39:22 smithi092 ceph-mon[29572]: pgmap v636: 148 pgs: 2 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+degraded+remapped, 143 active+clean; 1.8 GiB data, 9.0 GiB used, 706 GiB / 715 GiB avail; 853 B/s rd, 0 B/s wr, 709 op/s; 20382/90159 objects degraded (22.607%); 0 B/s, 67 objects/s recovering 2024-03-11T21:39:22.863 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:39:22 smithi092 ceph-mon[29572]: from='mgr.14607 v1:172.21.15.92:0/1702717597' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T21:39:22.863 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:39:22 smithi092 ceph-mon[37563]: pgmap v636: 148 pgs: 2 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+degraded+remapped, 143 active+clean; 1.8 GiB data, 9.0 GiB used, 706 GiB / 715 GiB avail; 853 B/s rd, 0 B/s wr, 709 op/s; 20382/90159 objects degraded (22.607%); 0 B/s, 67 objects/s recovering 2024-03-11T21:39:22.863 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:39:22 smithi092 ceph-mon[37563]: from='mgr.14607 v1:172.21.15.92:0/1702717597' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T21:39:22.888 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:39:22.924 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:39:22.924 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:39:22.925 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:39:23.025 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:39:23.065 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:39:23.066 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:39:23.066 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:39:23.167 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:39:23.191 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:39:23.191 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:39:23.191 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:39:23.292 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:39:23.332 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:39:23.333 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:39:23.333 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:39:23.434 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:39:23.472 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:39:23.473 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:39:23.473 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:39:23.575 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:39:23.613 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:39:23.614 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:39:23.614 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:39:23.715 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:39:23.744 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:39:23.745 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:39:23.745 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:39:23.828 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:39:23 smithi097 ceph-mon[33317]: from='client.24469 v1:172.21.15.97:0/404208310' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:39:23.828 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:39:23 smithi097 ceph-mon[33317]: Health check update: Degraded data redundancy: 20382/90159 objects degraded (22.607%), 5 pgs degraded, 5 pgs undersized (PG_DEGRADED) 2024-03-11T21:39:23.846 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:39:23.862 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:39:23 smithi092 ceph-mon[29572]: from='client.24469 v1:172.21.15.97:0/404208310' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:39:23.863 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:39:23 smithi092 ceph-mon[29572]: Health check update: Degraded data redundancy: 20382/90159 objects degraded (22.607%), 5 pgs degraded, 5 pgs undersized (PG_DEGRADED) 2024-03-11T21:39:23.863 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:39:23 smithi092 ceph-mon[37563]: from='client.24469 v1:172.21.15.97:0/404208310' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:39:23.863 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:39:23 smithi092 ceph-mon[37563]: Health check update: Degraded data redundancy: 20382/90159 objects degraded (22.607%), 5 pgs degraded, 5 pgs undersized (PG_DEGRADED) 2024-03-11T21:39:23.899 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:39:23.900 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:39:23.900 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:39:24.001 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:39:24.030 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:39:24.031 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:39:24.031 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:39:24.132 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:39:24.157 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:39:24.157 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:39:24.158 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:39:24.258 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:39:24.296 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:39:24.296 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:39:24.297 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:39:24.397 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:39:24.426 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:39:24.427 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:39:24.427 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:39:24.528 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:39:24.565 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:39:24.566 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:39:24.566 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:39:24.599 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:39:24 smithi092 ceph-mon[29572]: pgmap v637: 148 pgs: 2 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+degraded+remapped, 143 active+clean; 1.7 GiB data, 8.5 GiB used, 707 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 915 op/s; 20092/82755 objects degraded (24.279%); 0 B/s, 91 objects/s recovering 2024-03-11T21:39:24.599 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:39:24 smithi092 ceph-mon[37563]: pgmap v637: 148 pgs: 2 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+degraded+remapped, 143 active+clean; 1.7 GiB data, 8.5 GiB used, 707 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 915 op/s; 20092/82755 objects degraded (24.279%); 0 B/s, 91 objects/s recovering 2024-03-11T21:39:24.667 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:39:24.702 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:39:24.703 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:39:24.704 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:39:24.804 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:39:24.828 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:39:24 smithi097 ceph-mon[33317]: pgmap v637: 148 pgs: 2 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+degraded+remapped, 143 active+clean; 1.7 GiB data, 8.5 GiB used, 707 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 915 op/s; 20092/82755 objects degraded (24.279%); 0 B/s, 91 objects/s recovering 2024-03-11T21:39:24.849 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:39:24.850 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:39:24.851 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:39:24.952 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:39:24.996 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:39:24.997 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:39:24.997 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:39:25.098 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:39:25.128 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:39:25.129 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:39:25.129 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:39:25.230 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:39:25.255 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:39:25.256 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:39:25.256 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:39:25.357 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:39:25.395 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:39:25.396 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:39:25.396 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:39:25.497 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:39:25.547 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:39:25.548 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:39:25.548 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:39:25.649 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:39:25.693 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:39:25.694 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:39:25.694 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:39:25.795 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:39:25.822 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:39:25.822 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:39:25.823 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:39:25.923 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:39:25.962 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:39:25.962 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:39:25.963 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:39:26.064 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:39:26.103 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:39:26.103 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:39:26.104 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:39:26.205 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:39:26.233 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:39:26.234 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:39:26.234 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:39:26.335 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:39:26.363 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:39:26.364 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:39:26.364 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:39:26.464 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:39:26.497 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:39:26.498 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:39:26.498 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:39:26.595 INFO:teuthology.orchestra.run.smithi092.stderr:nodeep-scrub is set 2024-03-11T21:39:26.599 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:39:26.639 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:39:26.640 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:39:26.640 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:39:26.741 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:39:26.809 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:39:26 smithi092 ceph-mon[29572]: pgmap v638: 148 pgs: 2 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+degraded+remapped, 143 active+clean; 1.6 GiB data, 8.3 GiB used, 707 GiB / 715 GiB avail; 853 B/s rd, 0 B/s wr, 817 op/s; 19975/78384 objects degraded (25.484%); 0 B/s, 76 objects/s recovering 2024-03-11T21:39:26.810 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:39:26 smithi092 ceph-mon[29572]: purged_snaps scrub starts 2024-03-11T21:39:26.810 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:39:26 smithi092 ceph-mon[29572]: purged_snaps scrub ok 2024-03-11T21:39:26.810 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:39:26 smithi092 ceph-mon[29572]: from='client.? v1:172.21.15.92:0/4020693071' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-11T21:39:26.810 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:39:26 smithi092 ceph-mon[29572]: from='client.35489 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-11T21:39:26.810 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:39:26 smithi092 ceph-mon[37563]: pgmap v638: 148 pgs: 2 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+degraded+remapped, 143 active+clean; 1.6 GiB data, 8.3 GiB used, 707 GiB / 715 GiB avail; 853 B/s rd, 0 B/s wr, 817 op/s; 19975/78384 objects degraded (25.484%); 0 B/s, 76 objects/s recovering 2024-03-11T21:39:26.810 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:39:26 smithi092 ceph-mon[37563]: purged_snaps scrub starts 2024-03-11T21:39:26.810 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:39:26 smithi092 ceph-mon[37563]: purged_snaps scrub ok 2024-03-11T21:39:26.810 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:39:26 smithi092 ceph-mon[37563]: from='client.? v1:172.21.15.92:0/4020693071' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-11T21:39:26.810 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:39:26 smithi092 ceph-mon[37563]: from='client.35489 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-11T21:39:26.813 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:39:26.813 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:39:26.813 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:39:26.828 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:39:26 smithi097 ceph-mon[33317]: pgmap v638: 148 pgs: 2 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+degraded+remapped, 143 active+clean; 1.6 GiB data, 8.3 GiB used, 707 GiB / 715 GiB avail; 853 B/s rd, 0 B/s wr, 817 op/s; 19975/78384 objects degraded (25.484%); 0 B/s, 76 objects/s recovering 2024-03-11T21:39:26.829 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:39:26 smithi097 ceph-mon[33317]: purged_snaps scrub starts 2024-03-11T21:39:26.829 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:39:26 smithi097 ceph-mon[33317]: purged_snaps scrub ok 2024-03-11T21:39:26.829 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:39:26 smithi097 ceph-mon[33317]: from='client.? v1:172.21.15.92:0/4020693071' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-11T21:39:26.829 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:39:26 smithi097 ceph-mon[33317]: from='client.35489 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-11T21:39:26.914 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:39:26.946 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:39:26.947 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:39:26.947 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:39:27.048 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:39:27.082 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:39:27.082 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:39:27.083 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:39:27.183 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:39:27.215 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:39:27.216 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:39:27.217 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:39:27.317 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:39:27.347 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:39:27.348 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:39:27.348 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:39:27.449 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:39:27.474 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:39:27.475 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:39:27.476 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:39:27.577 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:39:27.608 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:39:27.608 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:39:27.609 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:39:27.710 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:39:27.755 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:39:27.756 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:39:27.756 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:39:27.756 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:39:27 smithi092 ceph-mon[29572]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-11T21:39:27.757 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:39:27 smithi092 ceph-mon[29572]: from='client.35489 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-03-11T21:39:27.757 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:39:27 smithi092 ceph-mon[29572]: osdmap e331: 8 total, 8 up, 8 in 2024-03-11T21:39:27.757 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:39:27 smithi092 ceph-mon[29572]: purged_snaps scrub starts 2024-03-11T21:39:27.757 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:39:27 smithi092 ceph-mon[29572]: purged_snaps scrub ok 2024-03-11T21:39:27.757 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:39:27 smithi092 ceph-mon[37563]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-11T21:39:27.757 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:39:27 smithi092 ceph-mon[37563]: from='client.35489 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-03-11T21:39:27.757 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:39:27 smithi092 ceph-mon[37563]: osdmap e331: 8 total, 8 up, 8 in 2024-03-11T21:39:27.757 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:39:27 smithi092 ceph-mon[37563]: purged_snaps scrub starts 2024-03-11T21:39:27.757 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:39:27 smithi092 ceph-mon[37563]: purged_snaps scrub ok 2024-03-11T21:39:27.828 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:39:27 smithi097 ceph-mon[33317]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-11T21:39:27.829 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:39:27 smithi097 ceph-mon[33317]: from='client.35489 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-03-11T21:39:27.829 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:39:27 smithi097 ceph-mon[33317]: osdmap e331: 8 total, 8 up, 8 in 2024-03-11T21:39:27.829 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:39:27 smithi097 ceph-mon[33317]: purged_snaps scrub starts 2024-03-11T21:39:27.829 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:39:27 smithi097 ceph-mon[33317]: purged_snaps scrub ok 2024-03-11T21:39:27.857 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:39:27.899 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:39:27.899 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:39:27.900 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:39:28.001 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:39:28.028 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:39:28.029 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:39:28.029 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:39:28.113 INFO:journalctl@ceph.mgr.y.smithi092.stdout:Mar 11 21:39:27 smithi092 ceph-00db0016-dfed-11ee-95c8-87774f69a715-mgr-y[29801]: ::ffff:172.21.15.97 - - [11/Mar/2024:21:39:27] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-11T21:39:28.131 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:39:28.160 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:39:28.161 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:39:28.161 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:39:28.262 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:39:28.303 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:39:28.304 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:39:28.304 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:39:28.405 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:39:28.432 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:39:28.434 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:39:28.434 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:39:28.535 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:39:28.567 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:39:28.567 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:39:28.567 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:39:28.668 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:39:28.694 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:39:28.695 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:39:28.695 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:39:28.796 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:39:28.822 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:39:28.823 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:39:28.823 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:39:28.863 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:39:28 smithi092 ceph-mon[29572]: pgmap v640: 148 pgs: 2 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+degraded+remapped, 143 active+clean; 1.6 GiB data, 8.2 GiB used, 707 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 819 op/s; 19975/78384 objects degraded (25.484%); 0 B/s, 80 objects/s recovering 2024-03-11T21:39:28.863 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:39:28 smithi092 ceph-mon[29572]: osdmap e332: 8 total, 8 up, 8 in 2024-03-11T21:39:28.863 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:39:28 smithi092 ceph-mon[29572]: Health check update: Degraded data redundancy: 19975/78384 objects degraded (25.484%), 5 pgs degraded, 5 pgs undersized (PG_DEGRADED) 2024-03-11T21:39:28.863 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:39:28 smithi092 ceph-mon[37563]: pgmap v640: 148 pgs: 2 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+degraded+remapped, 143 active+clean; 1.6 GiB data, 8.2 GiB used, 707 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 819 op/s; 19975/78384 objects degraded (25.484%); 0 B/s, 80 objects/s recovering 2024-03-11T21:39:28.863 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:39:28 smithi092 ceph-mon[37563]: osdmap e332: 8 total, 8 up, 8 in 2024-03-11T21:39:28.863 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:39:28 smithi092 ceph-mon[37563]: Health check update: Degraded data redundancy: 19975/78384 objects degraded (25.484%), 5 pgs degraded, 5 pgs undersized (PG_DEGRADED) 2024-03-11T21:39:28.924 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:39:28.954 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:39:28.955 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:39:28.956 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:39:29.056 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:39:29.078 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:39:28 smithi097 ceph-mon[33317]: pgmap v640: 148 pgs: 2 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+degraded+remapped, 143 active+clean; 1.6 GiB data, 8.2 GiB used, 707 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 819 op/s; 19975/78384 objects degraded (25.484%); 0 B/s, 80 objects/s recovering 2024-03-11T21:39:29.078 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:39:28 smithi097 ceph-mon[33317]: osdmap e332: 8 total, 8 up, 8 in 2024-03-11T21:39:29.079 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:39:28 smithi097 ceph-mon[33317]: Health check update: Degraded data redundancy: 19975/78384 objects degraded (25.484%), 5 pgs degraded, 5 pgs undersized (PG_DEGRADED) 2024-03-11T21:39:29.087 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:39:29.088 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:39:29.089 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:39:29.189 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:39:29.245 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:39:29.246 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:39:29.246 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:39:29.347 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:39:29.386 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:39:29.387 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:39:29.387 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:39:29.484 DEBUG:teuthology.orchestra.run.smithi092:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid 00db0016-dfed-11ee-95c8-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-03-11T21:39:29.488 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:39:29.515 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:39:29.515 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:39:29.516 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:39:29.617 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:39:29.644 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:39:29.644 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:39:29.644 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:39:29.745 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:39:29.786 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:39:29.786 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:39:29.787 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:39:29.887 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:39:29.916 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:39:29.917 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:39:29.917 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:39:30.018 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:39:30.061 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:39:30.062 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:39:30.062 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:39:30.163 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:39:30.216 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:39:30.216 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:39:30.217 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:39:30.237 DEBUG:teuthology.orchestra.run.smithi092:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid 00db0016-dfed-11ee-95c8-87774f69a715 -- ceph osd unset noscrub 2024-03-11T21:39:30.318 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:39:30.366 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:39:30.367 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:39:30.367 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:39:30.468 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:39:30.497 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:39:30.498 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:39:30.498 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:39:30.599 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:39:30.637 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:39:30.638 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:39:30.638 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:39:30.739 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:39:30.785 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:39:30.785 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:39:30.786 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:39:30.863 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:39:30 smithi092 ceph-mon[37563]: pgmap v642: 148 pgs: 2 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+degraded+remapped, 143 active+clean; 1.3 GiB data, 7.5 GiB used, 708 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.02k op/s; 19530/65661 objects degraded (29.744%); 0 B/s, 106 objects/s recovering 2024-03-11T21:39:30.863 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:39:30 smithi092 ceph-mon[29572]: pgmap v642: 148 pgs: 2 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+degraded+remapped, 143 active+clean; 1.3 GiB data, 7.5 GiB used, 708 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.02k op/s; 19530/65661 objects degraded (29.744%); 0 B/s, 106 objects/s recovering 2024-03-11T21:39:30.886 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:39:30.910 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:39:30.910 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:39:30.911 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:39:31.012 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:39:31.041 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:39:31.042 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:39:31.042 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:39:31.078 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:39:30 smithi097 ceph-mon[33317]: pgmap v642: 148 pgs: 2 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+degraded+remapped, 143 active+clean; 1.3 GiB data, 7.5 GiB used, 708 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.02k op/s; 19530/65661 objects degraded (29.744%); 0 B/s, 106 objects/s recovering 2024-03-11T21:39:31.143 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:39:31.171 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:39:31.172 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:39:31.172 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:39:31.273 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:39:31.325 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:39:31.326 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:39:31.326 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:39:31.427 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:39:31.459 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:39:31.460 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:39:31.460 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:39:31.561 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:39:31.618 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:39:31.619 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:39:31.619 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:39:31.720 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:39:31.765 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:39:31.766 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:39:31.766 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:39:31.863 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:39:31 smithi092 ceph-mon[29572]: purged_snaps scrub starts 2024-03-11T21:39:31.863 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:39:31 smithi092 ceph-mon[29572]: purged_snaps scrub ok 2024-03-11T21:39:31.863 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:39:31 smithi092 ceph-mon[37563]: purged_snaps scrub starts 2024-03-11T21:39:31.863 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:39:31 smithi092 ceph-mon[37563]: purged_snaps scrub ok 2024-03-11T21:39:31.867 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:39:31.894 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:39:31.895 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:39:31.895 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:39:31.996 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:39:32.038 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:39:32.039 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:39:32.039 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:39:32.078 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:39:31 smithi097 ceph-mon[33317]: purged_snaps scrub starts 2024-03-11T21:39:32.079 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:39:31 smithi097 ceph-mon[33317]: purged_snaps scrub ok 2024-03-11T21:39:32.108 INFO:teuthology.orchestra.run.smithi092.stderr:Inferring config /var/lib/ceph/00db0016-dfed-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T21:39:32.116 INFO:teuthology.orchestra.run.smithi092.stderr:Inferring config /var/lib/ceph/00db0016-dfed-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T21:39:32.141 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:39:32.142 INFO:teuthology.orchestra.run.smithi092.stdout:{ 2024-03-11T21:39:32.142 INFO:teuthology.orchestra.run.smithi092.stdout: "size": 20, 2024-03-11T21:39:32.142 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 600, 2024-03-11T21:39:32.143 INFO:teuthology.orchestra.run.smithi092.stdout: "ops": [ 2024-03-11T21:39:32.143 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:39:32.143 INFO:teuthology.orchestra.run.smithi092.stdout: "description": "osd_op(client.25282.0:76286 9.1 9.68546201 (undecoded) ondisk+retry+write+known_if_redirected+supports_pool_eio e277)", 2024-03-11T21:39:32.143 INFO:teuthology.orchestra.run.smithi092.stdout: "initiated_at": "2024-03-11T21:36:34.048064+0000", 2024-03-11T21:39:32.143 INFO:teuthology.orchestra.run.smithi092.stdout: "age": 178.106704949, 2024-03-11T21:39:32.143 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.0080588479999999, 2024-03-11T21:39:32.143 INFO:teuthology.orchestra.run.smithi092.stdout: "type_data": { 2024-03-11T21:39:32.143 INFO:teuthology.orchestra.run.smithi092.stdout: "flag_point": "reached pg", 2024-03-11T21:39:32.143 INFO:teuthology.orchestra.run.smithi092.stdout: "client_info": { 2024-03-11T21:39:32.143 INFO:teuthology.orchestra.run.smithi092.stdout: "client": "client.25282", 2024-03-11T21:39:32.143 INFO:teuthology.orchestra.run.smithi092.stdout: "client_addr": "v1:172.21.15.92:0/2489519821", 2024-03-11T21:39:32.143 INFO:teuthology.orchestra.run.smithi092.stdout: "tid": 76286 2024-03-11T21:39:32.143 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:39:32.144 INFO:teuthology.orchestra.run.smithi092.stdout: "events": [ 2024-03-11T21:39:32.144 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:39:32.144 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "initiated", 2024-03-11T21:39:32.144 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:36:34.048064+0000", 2024-03-11T21:39:32.144 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-03-11T21:39:32.144 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:39:32.144 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:39:32.144 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "header_read", 2024-03-11T21:39:32.144 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:36:34.048064+0000", 2024-03-11T21:39:32.144 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-03-11T21:39:32.144 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:39:32.144 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:39:32.144 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "throttled", 2024-03-11T21:39:32.145 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:36:34.048066+0000", 2024-03-11T21:39:32.145 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 2.1210000000000001e-06 2024-03-11T21:39:32.145 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:39:32.145 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:39:32.145 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "all_read", 2024-03-11T21:39:32.145 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:36:34.048123+0000", 2024-03-11T21:39:32.145 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 5.7192999999999999e-05 2024-03-11T21:39:32.145 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:39:32.145 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:39:32.145 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "dispatched", 2024-03-11T21:39:32.145 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:36:34.048137+0000", 2024-03-11T21:39:32.145 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.3623e-05 2024-03-11T21:39:32.146 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:39:32.146 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:39:32.146 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "queued_for_pg", 2024-03-11T21:39:32.146 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:36:34.048149+0000", 2024-03-11T21:39:32.146 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.1939999999999999e-05 2024-03-11T21:39:32.146 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:39:32.146 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:39:32.146 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-03-11T21:39:32.146 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:36:34.067528+0000", 2024-03-11T21:39:32.146 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.019379156000000002 2024-03-11T21:39:32.146 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:39:32.146 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:39:32.146 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "waiting for peered", 2024-03-11T21:39:32.146 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:36:34.067535+0000", 2024-03-11T21:39:32.147 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 7.255e-06 2024-03-11T21:39:32.147 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:39:32.147 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:39:32.147 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-03-11T21:39:32.147 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:36:35.056113+0000", 2024-03-11T21:39:32.147 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.98857784000000004 2024-03-11T21:39:32.147 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:39:32.147 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:39:32.147 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "done", 2024-03-11T21:39:32.147 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:36:35.056123+0000", 2024-03-11T21:39:32.147 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 9.7200000000000001e-06 2024-03-11T21:39:32.147 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-03-11T21:39:32.147 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-03-11T21:39:32.147 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-03-11T21:39:32.148 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:39:32.148 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:39:32.148 INFO:teuthology.orchestra.run.smithi092.stdout: "description": "osd_op(client.25282.0:76290 9.1 9.a7705091 (undecoded) ondisk+retry+write+known_if_redirected+supports_pool_eio e277)", 2024-03-11T21:39:32.148 INFO:teuthology.orchestra.run.smithi092.stdout: "initiated_at": "2024-03-11T21:36:34.048167+0000", 2024-03-11T21:39:32.148 INFO:teuthology.orchestra.run.smithi092.stdout: "age": 178.106601536, 2024-03-11T21:39:32.148 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.007992564, 2024-03-11T21:39:32.148 INFO:teuthology.orchestra.run.smithi092.stdout: "type_data": { 2024-03-11T21:39:32.148 INFO:teuthology.orchestra.run.smithi092.stdout: "flag_point": "reached pg", 2024-03-11T21:39:32.148 INFO:teuthology.orchestra.run.smithi092.stdout: "client_info": { 2024-03-11T21:39:32.148 INFO:teuthology.orchestra.run.smithi092.stdout: "client": "client.25282", 2024-03-11T21:39:32.148 INFO:teuthology.orchestra.run.smithi092.stdout: "client_addr": "v1:172.21.15.92:0/2489519821", 2024-03-11T21:39:32.148 INFO:teuthology.orchestra.run.smithi092.stdout: "tid": 76290 2024-03-11T21:39:32.148 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:39:32.148 INFO:teuthology.orchestra.run.smithi092.stdout: "events": [ 2024-03-11T21:39:32.149 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:39:32.149 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "initiated", 2024-03-11T21:39:32.149 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:36:34.048167+0000", 2024-03-11T21:39:32.149 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-03-11T21:39:32.149 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:39:32.149 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:39:32.149 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "header_read", 2024-03-11T21:39:32.149 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:36:34.048167+0000", 2024-03-11T21:39:32.149 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-03-11T21:39:32.149 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:39:32.149 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:39:32.149 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "throttled", 2024-03-11T21:39:32.149 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:36:34.048168+0000", 2024-03-11T21:39:32.149 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.0100000000000001e-06 2024-03-11T21:39:32.150 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:39:32.150 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:39:32.150 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "all_read", 2024-03-11T21:39:32.150 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:36:34.048194+0000", 2024-03-11T21:39:32.150 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 2.5534e-05 2024-03-11T21:39:32.150 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:39:32.150 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:39:32.150 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "dispatched", 2024-03-11T21:39:32.150 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:36:34.048201+0000", 2024-03-11T21:39:32.150 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 7.1250000000000004e-06 2024-03-11T21:39:32.150 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:39:32.150 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:39:32.150 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "queued_for_pg", 2024-03-11T21:39:32.150 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:36:34.048209+0000", 2024-03-11T21:39:32.151 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 8.1159999999999999e-06 2024-03-11T21:39:32.151 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:39:32.151 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:39:32.151 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-03-11T21:39:32.151 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:36:34.067563+0000", 2024-03-11T21:39:32.151 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.019353750999999999 2024-03-11T21:39:32.151 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:39:32.151 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:39:32.153 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "waiting for peered", 2024-03-11T21:39:32.153 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:36:34.067569+0000", 2024-03-11T21:39:32.153 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 6.2430000000000002e-06 2024-03-11T21:39:32.153 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:39:32.153 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:39:32.153 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-03-11T21:39:32.153 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:36:35.056151+0000", 2024-03-11T21:39:32.153 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.98858233299999998 2024-03-11T21:39:32.153 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:39:32.153 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:39:32.153 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "done", 2024-03-11T21:39:32.153 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:36:35.056160+0000", 2024-03-11T21:39:32.154 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 8.4519999999999997e-06 2024-03-11T21:39:32.154 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-03-11T21:39:32.154 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-03-11T21:39:32.154 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-03-11T21:39:32.154 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:39:32.154 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:39:32.154 INFO:teuthology.orchestra.run.smithi092.stdout: "description": "osd_op(client.25282.0:76293 9.2 9.27ec80e2 (undecoded) ondisk+retry+write+known_if_redirected+supports_pool_eio e277)", 2024-03-11T21:39:32.154 INFO:teuthology.orchestra.run.smithi092.stdout: "initiated_at": "2024-03-11T21:36:34.048217+0000", 2024-03-11T21:39:32.154 INFO:teuthology.orchestra.run.smithi092.stdout: "age": 178.106551387, 2024-03-11T21:39:32.154 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.0056144440000001, 2024-03-11T21:39:32.154 INFO:teuthology.orchestra.run.smithi092.stdout: "type_data": { 2024-03-11T21:39:32.154 INFO:teuthology.orchestra.run.smithi092.stdout: "flag_point": "reached pg", 2024-03-11T21:39:32.154 INFO:teuthology.orchestra.run.smithi092.stdout: "client_info": { 2024-03-11T21:39:32.155 INFO:teuthology.orchestra.run.smithi092.stdout: "client": "client.25282", 2024-03-11T21:39:32.155 INFO:teuthology.orchestra.run.smithi092.stdout: "client_addr": "v1:172.21.15.92:0/2489519821", 2024-03-11T21:39:32.155 INFO:teuthology.orchestra.run.smithi092.stdout: "tid": 76293 2024-03-11T21:39:32.155 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:39:32.155 INFO:teuthology.orchestra.run.smithi092.stdout: "events": [ 2024-03-11T21:39:32.155 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:39:32.155 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "initiated", 2024-03-11T21:39:32.155 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:36:34.048217+0000", 2024-03-11T21:39:32.155 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-03-11T21:39:32.155 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:39:32.155 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:39:32.155 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "header_read", 2024-03-11T21:39:32.155 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:36:34.048217+0000", 2024-03-11T21:39:32.155 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-03-11T21:39:32.155 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:39:32.156 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:39:32.156 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "throttled", 2024-03-11T21:39:32.156 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:36:34.048218+0000", 2024-03-11T21:39:32.156 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 7.5300000000000003e-07 2024-03-11T21:39:32.156 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:39:32.156 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:39:32.156 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "all_read", 2024-03-11T21:39:32.156 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:36:34.048243+0000", 2024-03-11T21:39:32.156 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 2.5358e-05 2024-03-11T21:39:32.156 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:39:32.156 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:39:32.156 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "dispatched", 2024-03-11T21:39:32.156 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:36:34.048250+0000", 2024-03-11T21:39:32.156 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 6.5390000000000004e-06 2024-03-11T21:39:32.157 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:39:32.157 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:39:32.157 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "queued_for_pg", 2024-03-11T21:39:32.157 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:36:34.048256+0000", 2024-03-11T21:39:32.157 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 5.6690000000000003e-06 2024-03-11T21:39:32.157 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:39:32.157 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:39:32.157 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-03-11T21:39:32.157 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:36:34.065536+0000", 2024-03-11T21:39:32.157 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.01728037 2024-03-11T21:39:32.157 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:39:32.157 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:39:32.157 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "waiting for peered", 2024-03-11T21:39:32.157 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:36:34.065552+0000", 2024-03-11T21:39:32.157 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.5730999999999999e-05 2024-03-11T21:39:32.158 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:39:32.158 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:39:32.158 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-03-11T21:39:32.158 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:36:35.053820+0000", 2024-03-11T21:39:32.158 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.98826853400000003 2024-03-11T21:39:32.158 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:39:32.158 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:39:32.158 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "done", 2024-03-11T21:39:32.158 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:36:35.053832+0000", 2024-03-11T21:39:32.159 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.149e-05 2024-03-11T21:39:32.159 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-03-11T21:39:32.160 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-03-11T21:39:32.160 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-03-11T21:39:32.160 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:39:32.160 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:39:32.160 INFO:teuthology.orchestra.run.smithi092.stdout: "description": "osd_op(client.25282.0:76299 9.8 9.6678cce8 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e277)", 2024-03-11T21:39:32.160 INFO:teuthology.orchestra.run.smithi092.stdout: "initiated_at": "2024-03-11T21:36:34.065550+0000", 2024-03-11T21:39:32.160 INFO:teuthology.orchestra.run.smithi092.stdout: "age": 178.089218794, 2024-03-11T21:39:32.160 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.98870970999999996, 2024-03-11T21:39:32.160 INFO:teuthology.orchestra.run.smithi092.stdout: "type_data": { 2024-03-11T21:39:32.160 INFO:teuthology.orchestra.run.smithi092.stdout: "flag_point": "reached pg", 2024-03-11T21:39:32.160 INFO:teuthology.orchestra.run.smithi092.stdout: "client_info": { 2024-03-11T21:39:32.160 INFO:teuthology.orchestra.run.smithi092.stdout: "client": "client.25282", 2024-03-11T21:39:32.160 INFO:teuthology.orchestra.run.smithi092.stdout: "client_addr": "v1:172.21.15.92:0/2489519821", 2024-03-11T21:39:32.160 INFO:teuthology.orchestra.run.smithi092.stdout: "tid": 76299 2024-03-11T21:39:32.161 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:39:32.161 INFO:teuthology.orchestra.run.smithi092.stdout: "events": [ 2024-03-11T21:39:32.161 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:39:32.161 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "initiated", 2024-03-11T21:39:32.161 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:36:34.065550+0000", 2024-03-11T21:39:32.161 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-03-11T21:39:32.161 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:39:32.161 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:39:32.161 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "header_read", 2024-03-11T21:39:32.161 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:36:34.065550+0000", 2024-03-11T21:39:32.161 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-03-11T21:39:32.161 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:39:32.161 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:39:32.161 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "throttled", 2024-03-11T21:39:32.162 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:36:34.065552+0000", 2024-03-11T21:39:32.162 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.767e-06 2024-03-11T21:39:32.162 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:39:32.162 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:39:32.162 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "all_read", 2024-03-11T21:39:32.162 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:36:34.065577+0000", 2024-03-11T21:39:32.162 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 2.4964999999999999e-05 2024-03-11T21:39:32.162 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:39:32.162 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:39:32.162 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "dispatched", 2024-03-11T21:39:32.162 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:36:34.065585+0000", 2024-03-11T21:39:32.162 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 8.3329999999999997e-06 2024-03-11T21:39:32.162 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:39:32.163 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:39:32.163 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "queued_for_pg", 2024-03-11T21:39:32.163 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:36:34.065594+0000", 2024-03-11T21:39:32.163 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 9.3000000000000007e-06 2024-03-11T21:39:32.163 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:39:32.163 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:39:32.163 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-03-11T21:39:32.163 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:36:34.066995+0000", 2024-03-11T21:39:32.163 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.0014003170000000001 2024-03-11T21:39:32.163 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:39:32.163 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:39:32.163 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "waiting for peered", 2024-03-11T21:39:32.163 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:36:34.067015+0000", 2024-03-11T21:39:32.164 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 2.0738999999999999e-05 2024-03-11T21:39:32.164 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:39:32.164 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:39:32.164 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-03-11T21:39:32.164 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:36:35.054251+0000", 2024-03-11T21:39:32.164 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.98723527499999997 2024-03-11T21:39:32.164 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:39:32.164 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:39:32.164 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "done", 2024-03-11T21:39:32.164 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:36:35.054260+0000", 2024-03-11T21:39:32.164 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 9.0140000000000004e-06 2024-03-11T21:39:32.164 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-03-11T21:39:32.164 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-03-11T21:39:32.164 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-03-11T21:39:32.165 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:39:32.165 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:39:32.165 INFO:teuthology.orchestra.run.smithi092.stdout: "description": "osd_op(client.25282.0:76301 9.8 9.7302db8 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e277)", 2024-03-11T21:39:32.165 INFO:teuthology.orchestra.run.smithi092.stdout: "initiated_at": "2024-03-11T21:36:34.066382+0000", 2024-03-11T21:39:32.165 INFO:teuthology.orchestra.run.smithi092.stdout: "age": 178.08838656, 2024-03-11T21:39:32.165 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.98874176000000003, 2024-03-11T21:39:32.165 INFO:teuthology.orchestra.run.smithi092.stdout: "type_data": { 2024-03-11T21:39:32.165 INFO:teuthology.orchestra.run.smithi092.stdout: "flag_point": "reached pg", 2024-03-11T21:39:32.165 INFO:teuthology.orchestra.run.smithi092.stdout: "client_info": { 2024-03-11T21:39:32.165 INFO:teuthology.orchestra.run.smithi092.stdout: "client": "client.25282", 2024-03-11T21:39:32.165 INFO:teuthology.orchestra.run.smithi092.stdout: "client_addr": "v1:172.21.15.92:0/2489519821", 2024-03-11T21:39:32.165 INFO:teuthology.orchestra.run.smithi092.stdout: "tid": 76301 2024-03-11T21:39:32.165 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:39:32.165 INFO:teuthology.orchestra.run.smithi092.stdout: "events": [ 2024-03-11T21:39:32.165 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:39:32.166 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "initiated", 2024-03-11T21:39:32.166 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:36:34.066382+0000", 2024-03-11T21:39:32.166 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-03-11T21:39:32.166 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:39:32.166 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:39:32.166 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "header_read", 2024-03-11T21:39:32.166 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:36:34.066382+0000", 2024-03-11T21:39:32.166 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-03-11T21:39:32.166 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:39:32.166 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:39:32.166 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "throttled", 2024-03-11T21:39:32.166 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:36:34.066384+0000", 2024-03-11T21:39:32.166 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.5239999999999999e-06 2024-03-11T21:39:32.167 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:39:32.167 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:39:32.167 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "all_read", 2024-03-11T21:39:32.167 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:36:34.066411+0000", 2024-03-11T21:39:32.167 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 2.6942999999999999e-05 2024-03-11T21:39:32.167 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:39:32.167 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:39:32.167 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "dispatched", 2024-03-11T21:39:32.167 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:36:34.066826+0000", 2024-03-11T21:39:32.167 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00041499100000000003 2024-03-11T21:39:32.167 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:39:32.167 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:39:32.167 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "queued_for_pg", 2024-03-11T21:39:32.167 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:36:34.069731+0000", 2024-03-11T21:39:32.167 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.0029053939999999999 2024-03-11T21:39:32.168 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:39:32.168 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:39:32.168 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-03-11T21:39:32.168 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:36:34.069797+0000", 2024-03-11T21:39:32.168 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 6.6271000000000003e-05 2024-03-11T21:39:32.168 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:39:32.168 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:39:32.168 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "waiting for peered", 2024-03-11T21:39:32.168 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:36:34.069805+0000", 2024-03-11T21:39:32.168 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 7.6669999999999996e-06 2024-03-11T21:39:32.168 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:39:32.168 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:39:32.168 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-03-11T21:39:32.168 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:36:35.054284+0000", 2024-03-11T21:39:32.169 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.98447862399999997 2024-03-11T21:39:32.169 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:39:32.169 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:39:32.169 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "done", 2024-03-11T21:39:32.169 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:36:35.055124+0000", 2024-03-11T21:39:32.169 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00084034599999999995 2024-03-11T21:39:32.169 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-03-11T21:39:32.169 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-03-11T21:39:32.169 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-03-11T21:39:32.169 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:39:32.169 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:39:32.169 INFO:teuthology.orchestra.run.smithi092.stdout: "description": "osd_op(client.25282.0:76303 9.2 9.dbf37f2 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e277)", 2024-03-11T21:39:32.169 INFO:teuthology.orchestra.run.smithi092.stdout: "initiated_at": "2024-03-11T21:36:34.069755+0000", 2024-03-11T21:39:32.169 INFO:teuthology.orchestra.run.smithi092.stdout: "age": 178.08501359900001, 2024-03-11T21:39:32.169 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.984125782, 2024-03-11T21:39:32.170 INFO:teuthology.orchestra.run.smithi092.stdout: "type_data": { 2024-03-11T21:39:32.170 INFO:teuthology.orchestra.run.smithi092.stdout: "flag_point": "reached pg", 2024-03-11T21:39:32.170 INFO:teuthology.orchestra.run.smithi092.stdout: "client_info": { 2024-03-11T21:39:32.170 INFO:teuthology.orchestra.run.smithi092.stdout: "client": "client.25282", 2024-03-11T21:39:32.170 INFO:teuthology.orchestra.run.smithi092.stdout: "client_addr": "v1:172.21.15.92:0/2489519821", 2024-03-11T21:39:32.170 INFO:teuthology.orchestra.run.smithi092.stdout: "tid": 76303 2024-03-11T21:39:32.170 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:39:32.170 INFO:teuthology.orchestra.run.smithi092.stdout: "events": [ 2024-03-11T21:39:32.170 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:39:32.170 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "initiated", 2024-03-11T21:39:32.170 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:36:34.069755+0000", 2024-03-11T21:39:32.170 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-03-11T21:39:32.170 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:39:32.170 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:39:32.171 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "header_read", 2024-03-11T21:39:32.171 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:36:34.069755+0000", 2024-03-11T21:39:32.171 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-03-11T21:39:32.171 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:39:32.171 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:39:32.171 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "throttled", 2024-03-11T21:39:32.171 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:36:34.069757+0000", 2024-03-11T21:39:32.171 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.5710000000000001e-06 2024-03-11T21:39:32.171 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:39:32.171 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:39:32.171 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "all_read", 2024-03-11T21:39:32.171 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:36:34.069807+0000", 2024-03-11T21:39:32.171 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 5.0180999999999999e-05 2024-03-11T21:39:32.171 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:39:32.172 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:39:32.172 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "dispatched", 2024-03-11T21:39:32.172 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:36:34.069813+0000", 2024-03-11T21:39:32.172 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 6.3810000000000004e-06 2024-03-11T21:39:32.172 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:39:32.172 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:39:32.172 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "queued_for_pg", 2024-03-11T21:39:32.172 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:36:34.069818+0000", 2024-03-11T21:39:32.172 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 4.2989999999999998e-06 2024-03-11T21:39:32.172 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:39:32.172 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:39:32.172 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-03-11T21:39:32.172 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:36:34.069993+0000", 2024-03-11T21:39:32.172 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00017577900000000001 2024-03-11T21:39:32.172 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:39:32.173 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:39:32.173 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "waiting for peered", 2024-03-11T21:39:32.173 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:36:34.070009+0000", 2024-03-11T21:39:32.173 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.6203000000000001e-05 2024-03-11T21:39:32.173 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:39:32.173 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:39:32.173 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-03-11T21:39:32.173 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:36:35.053872+0000", 2024-03-11T21:39:32.173 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.98386298999999999 2024-03-11T21:39:32.173 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:39:32.173 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:39:32.173 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "done", 2024-03-11T21:39:32.173 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:36:35.053881+0000", 2024-03-11T21:39:32.173 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 8.3780000000000001e-06 2024-03-11T21:39:32.174 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-03-11T21:39:32.174 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-03-11T21:39:32.174 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-03-11T21:39:32.174 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:39:32.174 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:39:32.174 INFO:teuthology.orchestra.run.smithi092.stdout: "description": "osd_op(client.25282.0:76304 9.2 9.f9d03b02 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e277)", 2024-03-11T21:39:32.174 INFO:teuthology.orchestra.run.smithi092.stdout: "initiated_at": "2024-03-11T21:36:34.069825+0000", 2024-03-11T21:39:32.174 INFO:teuthology.orchestra.run.smithi092.stdout: "age": 178.08494390600001, 2024-03-11T21:39:32.174 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.98408725200000002, 2024-03-11T21:39:32.174 INFO:teuthology.orchestra.run.smithi092.stdout: "type_data": { 2024-03-11T21:39:32.174 INFO:teuthology.orchestra.run.smithi092.stdout: "flag_point": "reached pg", 2024-03-11T21:39:32.174 INFO:teuthology.orchestra.run.smithi092.stdout: "client_info": { 2024-03-11T21:39:32.174 INFO:teuthology.orchestra.run.smithi092.stdout: "client": "client.25282", 2024-03-11T21:39:32.174 INFO:teuthology.orchestra.run.smithi092.stdout: "client_addr": "v1:172.21.15.92:0/2489519821", 2024-03-11T21:39:32.175 INFO:teuthology.orchestra.run.smithi092.stdout: "tid": 76304 2024-03-11T21:39:32.175 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:39:32.175 INFO:teuthology.orchestra.run.smithi092.stdout: "events": [ 2024-03-11T21:39:32.175 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:39:32.175 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "initiated", 2024-03-11T21:39:32.175 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:36:34.069825+0000", 2024-03-11T21:39:32.175 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-03-11T21:39:32.175 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:39:32.175 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:39:32.175 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "header_read", 2024-03-11T21:39:32.175 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:36:34.069825+0000", 2024-03-11T21:39:32.175 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-03-11T21:39:32.175 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:39:32.175 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:39:32.175 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "throttled", 2024-03-11T21:39:32.176 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:36:34.069825+0000", 2024-03-11T21:39:32.176 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 5.4000000000000002e-07 2024-03-11T21:39:32.176 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:39:32.176 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:39:32.176 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "all_read", 2024-03-11T21:39:32.176 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:36:34.069865+0000", 2024-03-11T21:39:32.176 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 3.9755000000000001e-05 2024-03-11T21:39:32.176 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:39:32.176 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:39:32.176 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "dispatched", 2024-03-11T21:39:32.176 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:36:34.069871+0000", 2024-03-11T21:39:32.176 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 6.3829999999999997e-06 2024-03-11T21:39:32.176 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:39:32.176 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:39:32.177 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "queued_for_pg", 2024-03-11T21:39:32.177 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:36:34.069876+0000", 2024-03-11T21:39:32.177 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 4.1339999999999997e-06 2024-03-11T21:39:32.177 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:39:32.177 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:39:32.177 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-03-11T21:39:32.177 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:36:34.070356+0000", 2024-03-11T21:39:32.177 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00048021799999999998 2024-03-11T21:39:32.177 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:39:32.177 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:39:32.177 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "waiting for peered", 2024-03-11T21:39:32.177 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:36:34.070363+0000", 2024-03-11T21:39:32.177 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 7.5050000000000001e-06 2024-03-11T21:39:32.177 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:39:32.178 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:39:32.178 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-03-11T21:39:32.178 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:36:35.053904+0000", 2024-03-11T21:39:32.178 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.98354064699999999 2024-03-11T21:39:32.178 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:39:32.178 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:39:32.178 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "done", 2024-03-11T21:39:32.178 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:36:35.053912+0000", 2024-03-11T21:39:32.178 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 8.0700000000000007e-06 2024-03-11T21:39:32.178 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-03-11T21:39:32.178 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-03-11T21:39:32.178 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-03-11T21:39:32.178 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:39:32.179 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:39:32.179 INFO:teuthology.orchestra.run.smithi092.stdout: "description": "osd_op(client.25282.0:159139 9.1 9:8edc64a8:::benchmark_data_smithi092_130291_object42755:head [delete] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e316)", 2024-03-11T21:39:32.179 INFO:teuthology.orchestra.run.smithi092.stdout: "initiated_at": "2024-03-11T21:38:21.530077+0000", 2024-03-11T21:39:32.179 INFO:teuthology.orchestra.run.smithi092.stdout: "age": 70.624691597999998, 2024-03-11T21:39:32.179 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.027682604, 2024-03-11T21:39:32.179 INFO:teuthology.orchestra.run.smithi092.stdout: "type_data": { 2024-03-11T21:39:32.179 INFO:teuthology.orchestra.run.smithi092.stdout: "flag_point": "reached pg", 2024-03-11T21:39:32.179 INFO:teuthology.orchestra.run.smithi092.stdout: "client_info": { 2024-03-11T21:39:32.179 INFO:teuthology.orchestra.run.smithi092.stdout: "client": "client.25282", 2024-03-11T21:39:32.179 INFO:teuthology.orchestra.run.smithi092.stdout: "client_addr": "v1:172.21.15.92:0/2489519821", 2024-03-11T21:39:32.179 INFO:teuthology.orchestra.run.smithi092.stdout: "tid": 159139 2024-03-11T21:39:32.179 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:39:32.179 INFO:teuthology.orchestra.run.smithi092.stdout: "events": [ 2024-03-11T21:39:32.180 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:39:32.180 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "initiated", 2024-03-11T21:39:32.180 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:38:21.530077+0000", 2024-03-11T21:39:32.180 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-03-11T21:39:32.180 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:39:32.180 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:39:32.180 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "header_read", 2024-03-11T21:39:32.180 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:38:21.530077+0000", 2024-03-11T21:39:32.180 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-03-11T21:39:32.180 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:39:32.180 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:39:32.180 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "throttled", 2024-03-11T21:39:32.180 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:38:21.530079+0000", 2024-03-11T21:39:32.180 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.917e-06 2024-03-11T21:39:32.180 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:39:32.181 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:39:32.181 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "all_read", 2024-03-11T21:39:32.181 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:38:21.530085+0000", 2024-03-11T21:39:32.181 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 6.2559999999999999e-06 2024-03-11T21:39:32.181 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:39:32.181 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:39:32.181 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "dispatched", 2024-03-11T21:39:32.181 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:38:21.530097+0000", 2024-03-11T21:39:32.181 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.1837e-05 2024-03-11T21:39:32.181 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:39:32.181 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:39:32.181 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "queued_for_pg", 2024-03-11T21:39:32.181 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:38:21.530109+0000", 2024-03-11T21:39:32.181 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.1841999999999999e-05 2024-03-11T21:39:32.181 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:39:32.182 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:39:32.182 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-03-11T21:39:32.182 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:38:21.531789+0000", 2024-03-11T21:39:32.182 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.001680394 2024-03-11T21:39:32.182 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:39:32.182 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:39:32.182 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "waiting for peered", 2024-03-11T21:39:32.182 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:38:21.531798+0000", 2024-03-11T21:39:32.182 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 8.8959999999999992e-06 2024-03-11T21:39:32.182 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:39:32.182 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:39:32.182 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-03-11T21:39:32.182 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:38:22.557675+0000", 2024-03-11T21:39:32.182 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.0258772599999999 2024-03-11T21:39:32.183 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:39:32.183 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:39:32.183 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "done", 2024-03-11T21:39:32.183 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:38:22.557760+0000", 2024-03-11T21:39:32.183 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 8.4202000000000005e-05 2024-03-11T21:39:32.183 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-03-11T21:39:32.183 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-03-11T21:39:32.183 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-03-11T21:39:32.183 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:39:32.183 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:39:32.183 INFO:teuthology.orchestra.run.smithi092.stdout: "description": "osd_op(client.25282.0:159149 9.1 9:8f30385f:::benchmark_data_smithi092_130291_object42765:head [delete] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e316)", 2024-03-11T21:39:32.183 INFO:teuthology.orchestra.run.smithi092.stdout: "initiated_at": "2024-03-11T21:38:21.530123+0000", 2024-03-11T21:39:32.183 INFO:teuthology.orchestra.run.smithi092.stdout: "age": 70.624645669000003, 2024-03-11T21:39:32.183 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.028215455, 2024-03-11T21:39:32.184 INFO:teuthology.orchestra.run.smithi092.stdout: "type_data": { 2024-03-11T21:39:32.184 INFO:teuthology.orchestra.run.smithi092.stdout: "flag_point": "reached pg", 2024-03-11T21:39:32.184 INFO:teuthology.orchestra.run.smithi092.stdout: "client_info": { 2024-03-11T21:39:32.184 INFO:teuthology.orchestra.run.smithi092.stdout: "client": "client.25282", 2024-03-11T21:39:32.184 INFO:teuthology.orchestra.run.smithi092.stdout: "client_addr": "v1:172.21.15.92:0/2489519821", 2024-03-11T21:39:32.184 INFO:teuthology.orchestra.run.smithi092.stdout: "tid": 159149 2024-03-11T21:39:32.184 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:39:32.184 INFO:teuthology.orchestra.run.smithi092.stdout: "events": [ 2024-03-11T21:39:32.184 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:39:32.184 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "initiated", 2024-03-11T21:39:32.184 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:38:21.530123+0000", 2024-03-11T21:39:32.184 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-03-11T21:39:32.184 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:39:32.184 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:39:32.184 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "header_read", 2024-03-11T21:39:32.185 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:38:21.530123+0000", 2024-03-11T21:39:32.185 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-03-11T21:39:32.185 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:39:32.185 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:39:32.185 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "throttled", 2024-03-11T21:39:32.185 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:38:21.530124+0000", 2024-03-11T21:39:32.185 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 7.2900000000000003e-07 2024-03-11T21:39:32.185 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:39:32.185 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:39:32.185 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "all_read", 2024-03-11T21:39:32.185 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:38:21.530127+0000", 2024-03-11T21:39:32.185 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 2.8509999999999999e-06 2024-03-11T21:39:32.185 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:39:32.185 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:39:32.186 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "dispatched", 2024-03-11T21:39:32.186 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:38:21.530132+0000", 2024-03-11T21:39:32.186 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 5.147e-06 2024-03-11T21:39:32.186 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:39:32.186 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:39:32.186 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "queued_for_pg", 2024-03-11T21:39:32.186 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:38:21.530137+0000", 2024-03-11T21:39:32.186 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 4.8930000000000004e-06 2024-03-11T21:39:32.186 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:39:32.186 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:39:32.186 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-03-11T21:39:32.186 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:38:21.531819+0000", 2024-03-11T21:39:32.186 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.0016821710000000001 2024-03-11T21:39:32.186 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:39:32.187 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:39:32.187 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "waiting for peered", 2024-03-11T21:39:32.187 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:38:21.531825+0000", 2024-03-11T21:39:32.187 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 6.1709999999999999e-06 2024-03-11T21:39:32.187 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:39:32.187 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:39:32.188 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-03-11T21:39:32.188 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:38:22.558262+0000", 2024-03-11T21:39:32.188 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.026436685 2024-03-11T21:39:32.188 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:39:32.188 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:39:32.188 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "done", 2024-03-11T21:39:32.188 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:38:22.558338+0000", 2024-03-11T21:39:32.188 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 7.6808000000000001e-05 2024-03-11T21:39:32.188 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-03-11T21:39:32.188 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-03-11T21:39:32.189 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-03-11T21:39:32.189 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:39:32.189 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:39:32.189 INFO:teuthology.orchestra.run.smithi092.stdout: "description": "osd_op(client.25282.0:159152 9.1 9:8a20b019:::benchmark_data_smithi092_130291_object42768:head [delete] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e316)", 2024-03-11T21:39:32.189 INFO:teuthology.orchestra.run.smithi092.stdout: "initiated_at": "2024-03-11T21:38:21.530142+0000", 2024-03-11T21:39:32.189 INFO:teuthology.orchestra.run.smithi092.stdout: "age": 70.624626448000001, 2024-03-11T21:39:32.189 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.0283078560000001, 2024-03-11T21:39:32.189 INFO:teuthology.orchestra.run.smithi092.stdout: "type_data": { 2024-03-11T21:39:32.189 INFO:teuthology.orchestra.run.smithi092.stdout: "flag_point": "reached pg", 2024-03-11T21:39:32.189 INFO:teuthology.orchestra.run.smithi092.stdout: "client_info": { 2024-03-11T21:39:32.189 INFO:teuthology.orchestra.run.smithi092.stdout: "client": "client.25282", 2024-03-11T21:39:32.189 INFO:teuthology.orchestra.run.smithi092.stdout: "client_addr": "v1:172.21.15.92:0/2489519821", 2024-03-11T21:39:32.189 INFO:teuthology.orchestra.run.smithi092.stdout: "tid": 159152 2024-03-11T21:39:32.189 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:39:32.190 INFO:teuthology.orchestra.run.smithi092.stdout: "events": [ 2024-03-11T21:39:32.190 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:39:32.190 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "initiated", 2024-03-11T21:39:32.190 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:38:21.530142+0000", 2024-03-11T21:39:32.190 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-03-11T21:39:32.190 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:39:32.190 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:39:32.190 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "header_read", 2024-03-11T21:39:32.190 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:38:21.530142+0000", 2024-03-11T21:39:32.190 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-03-11T21:39:32.190 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:39:32.190 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:39:32.190 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "throttled", 2024-03-11T21:39:32.190 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:38:21.530143+0000", 2024-03-11T21:39:32.191 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 4.7100000000000002e-07 2024-03-11T21:39:32.191 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:39:32.191 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:39:32.191 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "all_read", 2024-03-11T21:39:32.191 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:38:21.530153+0000", 2024-03-11T21:39:32.191 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.0614e-05 2024-03-11T21:39:32.191 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:39:32.191 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:39:32.191 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "dispatched", 2024-03-11T21:39:32.191 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:38:21.530158+0000", 2024-03-11T21:39:32.191 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 4.9559999999999996e-06 2024-03-11T21:39:32.191 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:39:32.191 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:39:32.191 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "queued_for_pg", 2024-03-11T21:39:32.191 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:38:21.530163+0000", 2024-03-11T21:39:32.192 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 4.5850000000000001e-06 2024-03-11T21:39:32.192 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:39:32.192 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:39:32.192 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-03-11T21:39:32.192 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:38:21.531844+0000", 2024-03-11T21:39:32.192 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.0016812890000000001 2024-03-11T21:39:32.192 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:39:32.192 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:39:32.192 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "waiting for peered", 2024-03-11T21:39:32.192 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:38:21.531851+0000", 2024-03-11T21:39:32.192 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 6.494e-06 2024-03-11T21:39:32.192 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:39:32.192 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:39:32.192 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-03-11T21:39:32.192 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:38:22.558382+0000", 2024-03-11T21:39:32.193 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.0265312740000001 2024-03-11T21:39:32.193 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:39:32.193 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:39:32.193 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "done", 2024-03-11T21:39:32.193 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:38:22.558450+0000", 2024-03-11T21:39:32.193 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 6.8172999999999999e-05 2024-03-11T21:39:32.193 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-03-11T21:39:32.193 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-03-11T21:39:32.193 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-03-11T21:39:32.193 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:39:32.193 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:39:32.193 INFO:teuthology.orchestra.run.smithi092.stdout: "description": "osd_op(client.25282.0:159159 9.1 9:8503bfb3:::benchmark_data_smithi092_130291_object42775:head [delete] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e316)", 2024-03-11T21:39:32.193 INFO:teuthology.orchestra.run.smithi092.stdout: "initiated_at": "2024-03-11T21:38:21.530168+0000", 2024-03-11T21:39:32.193 INFO:teuthology.orchestra.run.smithi092.stdout: "age": 70.624600830000006, 2024-03-11T21:39:32.194 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.0547834810000001, 2024-03-11T21:39:32.194 INFO:teuthology.orchestra.run.smithi092.stdout: "type_data": { 2024-03-11T21:39:32.194 INFO:teuthology.orchestra.run.smithi092.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-11T21:39:32.194 INFO:teuthology.orchestra.run.smithi092.stdout: "client_info": { 2024-03-11T21:39:32.194 INFO:teuthology.orchestra.run.smithi092.stdout: "client": "client.25282", 2024-03-11T21:39:32.194 INFO:teuthology.orchestra.run.smithi092.stdout: "client_addr": "v1:172.21.15.92:0/2489519821", 2024-03-11T21:39:32.194 INFO:teuthology.orchestra.run.smithi092.stdout: "tid": 159159 2024-03-11T21:39:32.194 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:39:32.194 INFO:teuthology.orchestra.run.smithi092.stdout: "events": [ 2024-03-11T21:39:32.194 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:39:32.194 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "initiated", 2024-03-11T21:39:32.194 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:38:21.530168+0000", 2024-03-11T21:39:32.194 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-03-11T21:39:32.194 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:39:32.195 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:39:32.195 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "header_read", 2024-03-11T21:39:32.195 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:38:21.530168+0000", 2024-03-11T21:39:32.195 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-03-11T21:39:32.195 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:39:32.195 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:39:32.195 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "throttled", 2024-03-11T21:39:32.195 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:38:21.530168+0000", 2024-03-11T21:39:32.195 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 4.6199999999999998e-07 2024-03-11T21:39:32.195 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:39:32.195 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:39:32.195 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "all_read", 2024-03-11T21:39:32.195 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:38:21.530170+0000", 2024-03-11T21:39:32.195 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.8870000000000001e-06 2024-03-11T21:39:32.196 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:39:32.196 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:39:32.196 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "dispatched", 2024-03-11T21:39:32.196 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:38:21.530174+0000", 2024-03-11T21:39:32.196 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 4.1119999999999998e-06 2024-03-11T21:39:32.196 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:39:32.196 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:39:32.196 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "queued_for_pg", 2024-03-11T21:39:32.196 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:38:21.530178+0000", 2024-03-11T21:39:32.196 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 4.126e-06 2024-03-11T21:39:32.196 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:39:32.196 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:39:32.196 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-03-11T21:39:32.196 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:38:21.531871+0000", 2024-03-11T21:39:32.197 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.001692834 2024-03-11T21:39:32.197 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:39:32.197 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:39:32.197 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "waiting for peered", 2024-03-11T21:39:32.197 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:38:21.531877+0000", 2024-03-11T21:39:32.197 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 5.896e-06 2024-03-11T21:39:32.197 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:39:32.197 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:39:32.197 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-03-11T21:39:32.197 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:38:22.558477+0000", 2024-03-11T21:39:32.197 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.0266000749999999 2024-03-11T21:39:32.197 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:39:32.197 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:39:32.197 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "started", 2024-03-11T21:39:32.197 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:38:22.558572+0000", 2024-03-11T21:39:32.198 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 9.4343000000000001e-05 2024-03-11T21:39:32.198 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:39:32.198 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:39:32.198 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "waiting for subops from 6,7", 2024-03-11T21:39:32.198 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:38:22.558973+0000", 2024-03-11T21:39:32.198 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00040125000000000002 2024-03-11T21:39:32.198 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:39:32.198 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:39:32.198 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "op_commit", 2024-03-11T21:39:32.198 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:38:22.576806+0000", 2024-03-11T21:39:32.198 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.017832877 2024-03-11T21:39:32.198 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:39:32.198 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:39:32.198 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "sub_op_commit_rec", 2024-03-11T21:39:32.199 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:38:22.579332+0000", 2024-03-11T21:39:32.199 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.0025260590000000002 2024-03-11T21:39:32.199 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:39:32.199 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:39:32.199 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "sub_op_commit_rec", 2024-03-11T21:39:32.199 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:38:22.584864+0000", 2024-03-11T21:39:32.199 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.0055325610000000001 2024-03-11T21:39:32.199 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:39:32.199 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:39:32.199 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "commit_sent", 2024-03-11T21:39:32.199 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:38:22.584909+0000", 2024-03-11T21:39:32.199 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 4.4598000000000002e-05 2024-03-11T21:39:32.199 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:39:32.199 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:39:32.200 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "done", 2024-03-11T21:39:32.200 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:38:22.584951+0000", 2024-03-11T21:39:32.200 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 4.2401000000000003e-05 2024-03-11T21:39:32.200 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-03-11T21:39:32.200 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-03-11T21:39:32.200 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-03-11T21:39:32.200 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:39:32.200 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:39:32.200 INFO:teuthology.orchestra.run.smithi092.stdout: "description": "osd_op(client.25282.0:159177 9.1 9:87394de3:::benchmark_data_smithi092_130291_object42793:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e316)", 2024-03-11T21:39:32.200 INFO:teuthology.orchestra.run.smithi092.stdout: "initiated_at": "2024-03-11T21:38:21.552627+0000", 2024-03-11T21:39:32.200 INFO:teuthology.orchestra.run.smithi092.stdout: "age": 70.602141420999999, 2024-03-11T21:39:32.200 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.032448735, 2024-03-11T21:39:32.200 INFO:teuthology.orchestra.run.smithi092.stdout: "type_data": { 2024-03-11T21:39:32.200 INFO:teuthology.orchestra.run.smithi092.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-11T21:39:32.200 INFO:teuthology.orchestra.run.smithi092.stdout: "client_info": { 2024-03-11T21:39:32.201 INFO:teuthology.orchestra.run.smithi092.stdout: "client": "client.25282", 2024-03-11T21:39:32.201 INFO:teuthology.orchestra.run.smithi092.stdout: "client_addr": "v1:172.21.15.92:0/2489519821", 2024-03-11T21:39:32.201 INFO:teuthology.orchestra.run.smithi092.stdout: "tid": 159177 2024-03-11T21:39:32.201 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:39:32.201 INFO:teuthology.orchestra.run.smithi092.stdout: "events": [ 2024-03-11T21:39:32.201 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:39:32.201 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "initiated", 2024-03-11T21:39:32.201 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:38:21.552627+0000", 2024-03-11T21:39:32.201 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-03-11T21:39:32.201 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:39:32.201 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:39:32.201 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "header_read", 2024-03-11T21:39:32.201 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:38:21.552627+0000", 2024-03-11T21:39:32.201 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-03-11T21:39:32.202 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:39:32.202 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:39:32.202 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "throttled", 2024-03-11T21:39:32.202 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:38:21.552629+0000", 2024-03-11T21:39:32.202 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 2.092e-06 2024-03-11T21:39:32.202 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:39:32.202 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:39:32.202 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "all_read", 2024-03-11T21:39:32.202 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:38:21.552635+0000", 2024-03-11T21:39:32.202 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 5.429e-06 2024-03-11T21:39:32.202 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:39:32.202 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:39:32.202 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "dispatched", 2024-03-11T21:39:32.202 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:38:21.552646+0000", 2024-03-11T21:39:32.203 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.1337e-05 2024-03-11T21:39:32.203 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:39:32.203 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:39:32.203 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "queued_for_pg", 2024-03-11T21:39:32.203 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:38:21.552654+0000", 2024-03-11T21:39:32.203 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 7.7330000000000003e-06 2024-03-11T21:39:32.203 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:39:32.203 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:39:32.203 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-03-11T21:39:32.203 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:38:21.552718+0000", 2024-03-11T21:39:32.203 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 6.4197999999999998e-05 2024-03-11T21:39:32.203 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:39:32.203 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:39:32.203 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "waiting for peered", 2024-03-11T21:39:32.203 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:38:21.552728+0000", 2024-03-11T21:39:32.203 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 9.4979999999999995e-06 2024-03-11T21:39:32.204 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:39:32.204 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:39:32.204 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-03-11T21:39:32.204 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:38:22.571674+0000", 2024-03-11T21:39:32.204 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.018946339 2024-03-11T21:39:32.204 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:39:32.204 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:39:32.204 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "started", 2024-03-11T21:39:32.204 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:38:22.571801+0000", 2024-03-11T21:39:32.204 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00012707800000000001 2024-03-11T21:39:32.204 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:39:32.204 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:39:32.204 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "waiting for subops from 6,7", 2024-03-11T21:39:32.204 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:38:22.571929+0000", 2024-03-11T21:39:32.204 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00012757600000000001 2024-03-11T21:39:32.205 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:39:32.205 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:39:32.205 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "op_commit", 2024-03-11T21:39:32.205 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:38:22.579270+0000", 2024-03-11T21:39:32.205 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.0073415099999999999 2024-03-11T21:39:32.205 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:39:32.205 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:39:32.205 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "sub_op_commit_rec", 2024-03-11T21:39:32.205 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:38:22.579415+0000", 2024-03-11T21:39:32.205 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.000144506 2024-03-11T21:39:32.205 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:39:32.205 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:39:32.205 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "sub_op_commit_rec", 2024-03-11T21:39:32.206 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:38:22.584989+0000", 2024-03-11T21:39:32.206 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.005574351 2024-03-11T21:39:32.206 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:39:32.206 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:39:32.206 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "commit_sent", 2024-03-11T21:39:32.206 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:38:22.585046+0000", 2024-03-11T21:39:32.206 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 5.6932999999999999e-05 2024-03-11T21:39:32.206 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:39:32.206 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:39:32.206 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "done", 2024-03-11T21:39:32.206 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:38:22.585076+0000", 2024-03-11T21:39:32.206 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 3.0154999999999998e-05 2024-03-11T21:39:32.206 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-03-11T21:39:32.206 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-03-11T21:39:32.207 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-03-11T21:39:32.207 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:39:32.207 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:39:32.207 INFO:teuthology.orchestra.run.smithi092.stdout: "description": "osd_op(client.25282.0:159188 9.1 9:8d26acf2:::benchmark_data_smithi092_130291_object42804:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e316)", 2024-03-11T21:39:32.207 INFO:teuthology.orchestra.run.smithi092.stdout: "initiated_at": "2024-03-11T21:38:21.566457+0000", 2024-03-11T21:39:32.207 INFO:teuthology.orchestra.run.smithi092.stdout: "age": 70.588311841000007, 2024-03-11T21:39:32.207 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.03005539, 2024-03-11T21:39:32.207 INFO:teuthology.orchestra.run.smithi092.stdout: "type_data": { 2024-03-11T21:39:32.207 INFO:teuthology.orchestra.run.smithi092.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-11T21:39:32.207 INFO:teuthology.orchestra.run.smithi092.stdout: "client_info": { 2024-03-11T21:39:32.207 INFO:teuthology.orchestra.run.smithi092.stdout: "client": "client.25282", 2024-03-11T21:39:32.207 INFO:teuthology.orchestra.run.smithi092.stdout: "client_addr": "v1:172.21.15.92:0/2489519821", 2024-03-11T21:39:32.207 INFO:teuthology.orchestra.run.smithi092.stdout: "tid": 159188 2024-03-11T21:39:32.207 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:39:32.208 INFO:teuthology.orchestra.run.smithi092.stdout: "events": [ 2024-03-11T21:39:32.208 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:39:32.208 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "initiated", 2024-03-11T21:39:32.208 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:38:21.566457+0000", 2024-03-11T21:39:32.208 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-03-11T21:39:32.208 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:39:32.208 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:39:32.208 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "header_read", 2024-03-11T21:39:32.208 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:38:21.566457+0000", 2024-03-11T21:39:32.208 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-03-11T21:39:32.208 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:39:32.208 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:39:32.208 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "throttled", 2024-03-11T21:39:32.208 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:38:21.566459+0000", 2024-03-11T21:39:32.208 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 2.1280000000000002e-06 2024-03-11T21:39:32.209 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:39:32.209 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:39:32.209 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "all_read", 2024-03-11T21:39:32.209 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:38:21.566467+0000", 2024-03-11T21:39:32.209 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 8.5310000000000001e-06 2024-03-11T21:39:32.209 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:39:32.209 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:39:32.209 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "dispatched", 2024-03-11T21:39:32.209 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:38:21.566483+0000", 2024-03-11T21:39:32.209 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.5580999999999999e-05 2024-03-11T21:39:32.209 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:39:32.209 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:39:32.209 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "queued_for_pg", 2024-03-11T21:39:32.209 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:38:21.566491+0000", 2024-03-11T21:39:32.210 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 8.1489999999999994e-06 2024-03-11T21:39:32.210 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:39:32.210 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:39:32.210 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-03-11T21:39:32.210 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:38:21.566563+0000", 2024-03-11T21:39:32.210 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 7.1996000000000007e-05 2024-03-11T21:39:32.210 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:39:32.210 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:39:32.210 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "waiting for peered", 2024-03-11T21:39:32.210 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:38:21.566574+0000", 2024-03-11T21:39:32.210 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.0533000000000001e-05 2024-03-11T21:39:32.210 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:39:32.210 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:39:32.210 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-03-11T21:39:32.211 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:38:22.574277+0000", 2024-03-11T21:39:32.211 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.0077027890000001 2024-03-11T21:39:32.211 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:39:32.211 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:39:32.211 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "started", 2024-03-11T21:39:32.211 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:38:22.574374+0000", 2024-03-11T21:39:32.211 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 9.7377e-05 2024-03-11T21:39:32.211 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:39:32.211 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:39:32.211 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "waiting for subops from 6,7", 2024-03-11T21:39:32.211 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:38:22.574461+0000", 2024-03-11T21:39:32.211 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 8.7429000000000006e-05 2024-03-11T21:39:32.211 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:39:32.211 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:39:32.211 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "op_commit", 2024-03-11T21:39:32.212 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:38:22.579433+0000", 2024-03-11T21:39:32.212 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.0049713680000000003 2024-03-11T21:39:32.212 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:39:32.212 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:39:32.212 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "sub_op_commit_rec", 2024-03-11T21:39:32.212 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:38:22.584824+0000", 2024-03-11T21:39:32.212 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.0053917230000000002 2024-03-11T21:39:32.212 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:39:32.212 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:39:32.212 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "sub_op_commit_rec", 2024-03-11T21:39:32.212 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:38:22.596430+0000", 2024-03-11T21:39:32.212 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.011606053999999999 2024-03-11T21:39:32.212 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:39:32.212 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:39:32.213 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "commit_sent", 2024-03-11T21:39:32.213 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:38:22.596463+0000", 2024-03-11T21:39:32.213 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 3.2662e-05 2024-03-11T21:39:32.213 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:39:32.213 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:39:32.213 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "done", 2024-03-11T21:39:32.213 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:38:22.596512+0000", 2024-03-11T21:39:32.213 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 4.9070000000000003e-05 2024-03-11T21:39:32.213 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-03-11T21:39:32.213 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-03-11T21:39:32.213 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-03-11T21:39:32.213 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:39:32.214 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:39:32.214 INFO:teuthology.orchestra.run.smithi092.stdout: "description": "osd_op(client.25282.0:159192 9.1 9:8431376f:::benchmark_data_smithi092_130291_object42808:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e316)", 2024-03-11T21:39:32.214 INFO:teuthology.orchestra.run.smithi092.stdout: "initiated_at": "2024-03-11T21:38:21.572637+0000", 2024-03-11T21:39:32.214 INFO:teuthology.orchestra.run.smithi092.stdout: "age": 70.582132113, 2024-03-11T21:39:32.214 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.02405857, 2024-03-11T21:39:32.214 INFO:teuthology.orchestra.run.smithi092.stdout: "type_data": { 2024-03-11T21:39:32.214 INFO:teuthology.orchestra.run.smithi092.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-11T21:39:32.214 INFO:teuthology.orchestra.run.smithi092.stdout: "client_info": { 2024-03-11T21:39:32.214 INFO:teuthology.orchestra.run.smithi092.stdout: "client": "client.25282", 2024-03-11T21:39:32.214 INFO:teuthology.orchestra.run.smithi092.stdout: "client_addr": "v1:172.21.15.92:0/2489519821", 2024-03-11T21:39:32.214 INFO:teuthology.orchestra.run.smithi092.stdout: "tid": 159192 2024-03-11T21:39:32.214 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:39:32.214 INFO:teuthology.orchestra.run.smithi092.stdout: "events": [ 2024-03-11T21:39:32.214 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:39:32.215 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "initiated", 2024-03-11T21:39:32.215 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:39:32.215 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:39:32.215 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:39:32.216 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:38:21.572637+0000", 2024-03-11T21:39:32.216 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-03-11T21:39:32.216 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:39:32.216 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:39:32.216 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "header_read", 2024-03-11T21:39:32.216 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:38:21.572637+0000", 2024-03-11T21:39:32.216 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-03-11T21:39:32.216 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:39:32.216 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:39:32.216 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "throttled", 2024-03-11T21:39:32.217 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:38:21.572639+0000", 2024-03-11T21:39:32.217 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.9989999999999998e-06 2024-03-11T21:39:32.217 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:39:32.217 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:39:32.217 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "all_read", 2024-03-11T21:39:32.217 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:38:21.572645+0000", 2024-03-11T21:39:32.217 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 6.0549999999999996e-06 2024-03-11T21:39:32.217 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:39:32.217 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:39:32.217 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "dispatched", 2024-03-11T21:39:32.217 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:38:21.572658+0000", 2024-03-11T21:39:32.217 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.3521e-05 2024-03-11T21:39:32.217 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:39:32.217 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:39:32.217 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "queued_for_pg", 2024-03-11T21:39:32.218 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:38:21.572666+0000", 2024-03-11T21:39:32.218 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 7.3969999999999997e-06 2024-03-11T21:39:32.218 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:39:32.218 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:39:32.218 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-03-11T21:39:32.218 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:38:21.572733+0000", 2024-03-11T21:39:32.218 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 6.7527999999999994e-05 2024-03-11T21:39:32.218 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:39:32.218 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:39:32.219 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "waiting for peered", 2024-03-11T21:39:32.219 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:38:21.572745+0000", 2024-03-11T21:39:32.219 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.1839e-05 2024-03-11T21:39:32.219 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:39:32.219 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:39:32.219 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-03-11T21:39:32.219 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:38:22.576835+0000", 2024-03-11T21:39:32.219 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.004090052 2024-03-11T21:39:32.219 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:39:32.219 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:39:32.219 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "started", 2024-03-11T21:39:32.219 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:38:22.576949+0000", 2024-03-11T21:39:32.219 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00011376500000000001 2024-03-11T21:39:32.219 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:39:32.220 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:39:32.220 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "waiting for subops from 6,7", 2024-03-11T21:39:32.220 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:38:22.577086+0000", 2024-03-11T21:39:32.220 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.000136821 2024-03-11T21:39:32.220 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:39:32.220 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:39:32.220 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "op_commit", 2024-03-11T21:39:32.220 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:38:22.584836+0000", 2024-03-11T21:39:32.220 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.0077503770000000001 2024-03-11T21:39:32.220 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:39:32.220 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:39:32.220 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "sub_op_commit_rec", 2024-03-11T21:39:32.220 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:38:22.585109+0000", 2024-03-11T21:39:32.220 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00027282000000000001 2024-03-11T21:39:32.221 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:39:32.221 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:39:32.221 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "sub_op_commit_rec", 2024-03-11T21:39:32.221 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:38:22.596603+0000", 2024-03-11T21:39:32.221 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.011494519999999999 2024-03-11T21:39:32.221 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:39:32.221 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:39:32.221 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "commit_sent", 2024-03-11T21:39:32.221 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:38:22.596658+0000", 2024-03-11T21:39:32.221 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 5.5234e-05 2024-03-11T21:39:32.221 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:39:32.221 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:39:32.221 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "done", 2024-03-11T21:39:32.222 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:38:22.596695+0000", 2024-03-11T21:39:32.222 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 3.6641999999999999e-05 2024-03-11T21:39:32.222 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-03-11T21:39:32.222 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-03-11T21:39:32.222 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-03-11T21:39:32.222 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:39:32.222 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:39:32.222 INFO:teuthology.orchestra.run.smithi092.stdout: "description": "osd_op(client.25282.0:159208 9.1 9:818b13f8:::benchmark_data_smithi092_130291_object42824:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e316)", 2024-03-11T21:39:32.223 INFO:teuthology.orchestra.run.smithi092.stdout: "initiated_at": "2024-03-11T21:38:21.607504+0000", 2024-03-11T21:39:32.223 INFO:teuthology.orchestra.run.smithi092.stdout: "age": 70.547264613999999, 2024-03-11T21:39:32.223 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.98933565300000004, 2024-03-11T21:39:32.223 INFO:teuthology.orchestra.run.smithi092.stdout: "type_data": { 2024-03-11T21:39:32.223 INFO:teuthology.orchestra.run.smithi092.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-11T21:39:32.223 INFO:teuthology.orchestra.run.smithi092.stdout: "client_info": { 2024-03-11T21:39:32.223 INFO:teuthology.orchestra.run.smithi092.stdout: "client": "client.25282", 2024-03-11T21:39:32.223 INFO:teuthology.orchestra.run.smithi092.stdout: "client_addr": "v1:172.21.15.92:0/2489519821", 2024-03-11T21:39:32.223 INFO:teuthology.orchestra.run.smithi092.stdout: "tid": 159208 2024-03-11T21:39:32.223 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:39:32.223 INFO:teuthology.orchestra.run.smithi092.stdout: "events": [ 2024-03-11T21:39:32.223 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:39:32.223 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "initiated", 2024-03-11T21:39:32.224 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:38:21.607504+0000", 2024-03-11T21:39:32.224 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-03-11T21:39:32.224 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:39:32.224 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:39:32.224 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "header_read", 2024-03-11T21:39:32.224 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:38:21.607504+0000", 2024-03-11T21:39:32.224 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-03-11T21:39:32.224 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:39:32.224 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:39:32.224 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "throttled", 2024-03-11T21:39:32.224 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:38:21.607507+0000", 2024-03-11T21:39:32.224 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 3.2509999999999999e-06 2024-03-11T21:39:32.224 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:39:32.224 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:39:32.224 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "all_read", 2024-03-11T21:39:32.225 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:38:21.607519+0000", 2024-03-11T21:39:32.225 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.1201e-05 2024-03-11T21:39:32.225 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:39:32.225 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:39:32.225 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "dispatched", 2024-03-11T21:39:32.225 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:38:21.607540+0000", 2024-03-11T21:39:32.225 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 2.1967000000000002e-05 2024-03-11T21:39:32.225 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:39:32.225 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:39:32.225 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "queued_for_pg", 2024-03-11T21:39:32.225 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:38:21.607554+0000", 2024-03-11T21:39:32.225 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.323e-05 2024-03-11T21:39:32.225 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:39:32.225 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:39:32.225 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-03-11T21:39:32.226 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:38:21.607636+0000", 2024-03-11T21:39:32.226 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 8.2688000000000005e-05 2024-03-11T21:39:32.226 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:39:32.226 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:39:32.226 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "waiting for peered", 2024-03-11T21:39:32.226 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:38:21.607649+0000", 2024-03-11T21:39:32.226 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.2587e-05 2024-03-11T21:39:32.226 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:39:32.226 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:39:32.226 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-03-11T21:39:32.226 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:38:22.579475+0000", 2024-03-11T21:39:32.226 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.97182567900000005 2024-03-11T21:39:32.227 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:39:32.227 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:39:32.227 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "started", 2024-03-11T21:39:32.227 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:38:22.579614+0000", 2024-03-11T21:39:32.227 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00013914099999999999 2024-03-11T21:39:32.227 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:39:32.227 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:39:32.227 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "waiting for subops from 6,7", 2024-03-11T21:39:32.227 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:38:22.579754+0000", 2024-03-11T21:39:32.227 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.000140099 2024-03-11T21:39:32.227 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:39:32.227 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:39:32.227 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "op_commit", 2024-03-11T21:39:32.227 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:38:22.584961+0000", 2024-03-11T21:39:32.227 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.0052068820000000004 2024-03-11T21:39:32.228 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:39:32.228 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:39:32.228 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "sub_op_commit_rec", 2024-03-11T21:39:32.228 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:38:22.590679+0000", 2024-03-11T21:39:32.228 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.0057186939999999999 2024-03-11T21:39:32.228 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:39:32.228 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:39:32.228 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "sub_op_commit_rec", 2024-03-11T21:39:32.228 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:38:22.596746+0000", 2024-03-11T21:39:32.228 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.0060667910000000002 2024-03-11T21:39:32.228 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:39:32.228 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:39:32.228 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "commit_sent", 2024-03-11T21:39:32.228 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:38:22.596798+0000", 2024-03-11T21:39:32.228 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 5.1773e-05 2024-03-11T21:39:32.229 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:39:32.229 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:39:32.229 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "done", 2024-03-11T21:39:32.229 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:38:22.596840+0000", 2024-03-11T21:39:32.229 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 4.1669999999999999e-05 2024-03-11T21:39:32.229 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-03-11T21:39:32.229 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-03-11T21:39:32.229 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-03-11T21:39:32.229 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:39:32.229 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:39:32.229 INFO:teuthology.orchestra.run.smithi092.stdout: "description": "osd_op(client.25282.0:159230 9.1 9:84b7046d:::benchmark_data_smithi092_130291_object42846:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e316)", 2024-03-11T21:39:32.229 INFO:teuthology.orchestra.run.smithi092.stdout: "initiated_at": "2024-03-11T21:38:21.634583+0000", 2024-03-11T21:39:32.229 INFO:teuthology.orchestra.run.smithi092.stdout: "age": 70.520185440999995, 2024-03-11T21:39:32.229 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.96240426300000004, 2024-03-11T21:39:32.229 INFO:teuthology.orchestra.run.smithi092.stdout: "type_data": { 2024-03-11T21:39:32.230 INFO:teuthology.orchestra.run.smithi092.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-11T21:39:32.230 INFO:teuthology.orchestra.run.smithi092.stdout: "client_info": { 2024-03-11T21:39:32.230 INFO:teuthology.orchestra.run.smithi092.stdout: "client": "client.25282", 2024-03-11T21:39:32.230 INFO:teuthology.orchestra.run.smithi092.stdout: "client_addr": "v1:172.21.15.92:0/2489519821", 2024-03-11T21:39:32.230 INFO:teuthology.orchestra.run.smithi092.stdout: "tid": 159230 2024-03-11T21:39:32.230 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:39:32.230 INFO:teuthology.orchestra.run.smithi092.stdout: "events": [ 2024-03-11T21:39:32.230 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:39:32.230 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "initiated", 2024-03-11T21:39:32.230 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:38:21.634583+0000", 2024-03-11T21:39:32.230 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-03-11T21:39:32.230 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:39:32.230 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:39:32.230 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "header_read", 2024-03-11T21:39:32.231 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:38:21.634583+0000", 2024-03-11T21:39:32.231 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-03-11T21:39:32.231 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:39:32.231 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:39:32.231 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "throttled", 2024-03-11T21:39:32.231 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:38:21.634585+0000", 2024-03-11T21:39:32.231 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.9690000000000001e-06 2024-03-11T21:39:32.231 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:39:32.231 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:39:32.231 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "all_read", 2024-03-11T21:39:32.231 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:38:21.634591+0000", 2024-03-11T21:39:32.231 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 5.5940000000000001e-06 2024-03-11T21:39:32.231 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:39:32.231 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:39:32.232 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "dispatched", 2024-03-11T21:39:32.232 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:38:21.634603+0000", 2024-03-11T21:39:32.232 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.2223e-05 2024-03-11T21:39:32.232 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:39:32.232 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:39:32.232 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "queued_for_pg", 2024-03-11T21:39:32.232 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:38:21.634611+0000", 2024-03-11T21:39:32.232 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 7.6839999999999996e-06 2024-03-11T21:39:32.232 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:39:32.232 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:39:32.232 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-03-11T21:39:32.232 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:38:21.634673+0000", 2024-03-11T21:39:32.232 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 6.2286999999999995e-05 2024-03-11T21:39:32.232 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:39:32.233 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:39:32.233 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "waiting for peered", 2024-03-11T21:39:32.233 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:38:21.634683+0000", 2024-03-11T21:39:32.233 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 9.8230000000000006e-06 2024-03-11T21:39:32.233 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:39:32.233 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:39:32.233 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-03-11T21:39:32.233 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:38:22.582217+0000", 2024-03-11T21:39:32.233 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.94753467999999996 2024-03-11T21:39:32.233 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:39:32.233 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:39:32.233 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "started", 2024-03-11T21:39:32.233 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:38:22.582330+0000", 2024-03-11T21:39:32.233 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.000112267 2024-03-11T21:39:32.234 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:39:32.234 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:39:32.234 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "waiting for subops from 6,7", 2024-03-11T21:39:32.234 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:38:22.582438+0000", 2024-03-11T21:39:32.234 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00010788700000000001 2024-03-11T21:39:32.234 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:39:32.234 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:39:32.234 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "op_commit", 2024-03-11T21:39:32.234 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:38:22.590695+0000", 2024-03-11T21:39:32.234 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.0082569800000000006 2024-03-11T21:39:32.234 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:39:32.234 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:39:32.234 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "sub_op_commit_rec", 2024-03-11T21:39:32.234 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:38:22.590729+0000", 2024-03-11T21:39:32.235 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 3.4139e-05 2024-03-11T21:39:32.235 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:39:32.235 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:39:32.235 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "sub_op_commit_rec", 2024-03-11T21:39:32.235 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:38:22.596897+0000", 2024-03-11T21:39:32.235 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.0061681549999999998 2024-03-11T21:39:32.235 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:39:32.235 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:39:32.235 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "commit_sent", 2024-03-11T21:39:32.235 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:38:22.596948+0000", 2024-03-11T21:39:32.235 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 5.0967999999999997e-05 2024-03-11T21:39:32.235 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:39:32.235 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:39:32.235 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "done", 2024-03-11T21:39:32.235 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:38:22.596987+0000", 2024-03-11T21:39:32.236 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 3.9607000000000002e-05 2024-03-11T21:39:32.236 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-03-11T21:39:32.236 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-03-11T21:39:32.236 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-03-11T21:39:32.236 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:39:32.236 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:39:32.236 INFO:teuthology.orchestra.run.smithi092.stdout: "description": "osd_op(client.25282.0:159243 9.1 9:808c8193:::benchmark_data_smithi092_130291_object42859:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e316)", 2024-03-11T21:39:32.236 INFO:teuthology.orchestra.run.smithi092.stdout: "initiated_at": "2024-03-11T21:38:21.661345+0000", 2024-03-11T21:39:32.236 INFO:teuthology.orchestra.run.smithi092.stdout: "age": 70.493424028999996, 2024-03-11T21:39:32.236 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.93583571700000001, 2024-03-11T21:39:32.236 INFO:teuthology.orchestra.run.smithi092.stdout: "type_data": { 2024-03-11T21:39:32.236 INFO:teuthology.orchestra.run.smithi092.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-11T21:39:32.236 INFO:teuthology.orchestra.run.smithi092.stdout: "client_info": { 2024-03-11T21:39:32.237 INFO:teuthology.orchestra.run.smithi092.stdout: "client": "client.25282", 2024-03-11T21:39:32.237 INFO:teuthology.orchestra.run.smithi092.stdout: "client_addr": "v1:172.21.15.92:0/2489519821", 2024-03-11T21:39:32.237 INFO:teuthology.orchestra.run.smithi092.stdout: "tid": 159243 2024-03-11T21:39:32.237 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:39:32.237 INFO:teuthology.orchestra.run.smithi092.stdout: "events": [ 2024-03-11T21:39:32.237 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:39:32.237 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "initiated", 2024-03-11T21:39:32.237 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:38:21.661345+0000", 2024-03-11T21:39:32.237 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-03-11T21:39:32.237 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:39:32.237 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:39:32.237 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "header_read", 2024-03-11T21:39:32.237 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:38:21.661345+0000", 2024-03-11T21:39:32.237 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-03-11T21:39:32.237 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:39:32.238 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:39:32.238 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "throttled", 2024-03-11T21:39:32.238 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:38:21.661347+0000", 2024-03-11T21:39:32.238 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 2.5459999999999998e-06 2024-03-11T21:39:32.238 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:39:32.238 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:39:32.238 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "all_read", 2024-03-11T21:39:32.238 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:38:21.661358+0000", 2024-03-11T21:39:32.238 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.0711e-05 2024-03-11T21:39:32.238 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:39:32.238 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:39:32.238 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "dispatched", 2024-03-11T21:39:32.238 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:38:21.661375+0000", 2024-03-11T21:39:32.239 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.6779e-05 2024-03-11T21:39:32.239 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:39:32.239 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:39:32.239 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "queued_for_pg", 2024-03-11T21:39:32.239 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:38:21.661385+0000", 2024-03-11T21:39:32.239 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.0200999999999999e-05 2024-03-11T21:39:32.239 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:39:32.239 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:39:32.239 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-03-11T21:39:32.239 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:38:21.661463+0000", 2024-03-11T21:39:32.239 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 7.8042999999999998e-05 2024-03-11T21:39:32.239 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:39:32.239 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:39:32.239 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "waiting for peered", 2024-03-11T21:39:32.239 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:38:21.661476+0000", 2024-03-11T21:39:32.240 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.2864000000000001e-05 2024-03-11T21:39:32.240 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:39:32.240 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:39:32.240 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-03-11T21:39:32.240 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:38:22.585132+0000", 2024-03-11T21:39:32.240 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.92365648899999997 2024-03-11T21:39:32.240 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:39:32.240 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:39:32.240 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "started", 2024-03-11T21:39:32.240 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:38:22.585240+0000", 2024-03-11T21:39:32.240 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.000107878 2024-03-11T21:39:32.240 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:39:32.240 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:39:32.240 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "waiting for subops from 6,7", 2024-03-11T21:39:32.241 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:38:22.585347+0000", 2024-03-11T21:39:32.241 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.000107236 2024-03-11T21:39:32.241 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:39:32.241 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:39:32.241 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "op_commit", 2024-03-11T21:39:32.241 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:38:22.590738+0000", 2024-03-11T21:39:32.241 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.0053908619999999997 2024-03-11T21:39:32.241 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:39:32.241 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:39:32.241 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "sub_op_commit_rec", 2024-03-11T21:39:32.241 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:38:22.596395+0000", 2024-03-11T21:39:32.241 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.0056570309999999999 2024-03-11T21:39:32.241 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:39:32.241 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:39:32.242 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "sub_op_commit_rec", 2024-03-11T21:39:32.242 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:38:22.597055+0000", 2024-03-11T21:39:32.242 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00065976800000000003 2024-03-11T21:39:32.242 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:39:32.242 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:39:32.242 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "commit_sent", 2024-03-11T21:39:32.242 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:38:22.597142+0000", 2024-03-11T21:39:32.242 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 8.6792999999999994e-05 2024-03-11T21:39:32.242 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:39:32.242 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:39:32.242 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "done", 2024-03-11T21:39:32.242 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:38:22.597180+0000", 2024-03-11T21:39:32.242 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 3.8516000000000002e-05 2024-03-11T21:39:32.242 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-03-11T21:39:32.243 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-03-11T21:39:32.243 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-03-11T21:39:32.243 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:39:32.243 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:39:32.243 INFO:teuthology.orchestra.run.smithi092.stdout: "description": "osd_op(client.25282.0:159244 9.1 9:8c82037c:::benchmark_data_smithi092_130291_object42860:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e316)", 2024-03-11T21:39:32.243 INFO:teuthology.orchestra.run.smithi092.stdout: "initiated_at": "2024-03-11T21:38:21.661718+0000", 2024-03-11T21:39:32.243 INFO:teuthology.orchestra.run.smithi092.stdout: "age": 70.493050350999994, 2024-03-11T21:39:32.243 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.94180018899999995, 2024-03-11T21:39:32.243 INFO:teuthology.orchestra.run.smithi092.stdout: "type_data": { 2024-03-11T21:39:32.243 INFO:teuthology.orchestra.run.smithi092.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-11T21:39:32.243 INFO:teuthology.orchestra.run.smithi092.stdout: "client_info": { 2024-03-11T21:39:32.243 INFO:teuthology.orchestra.run.smithi092.stdout: "client": "client.25282", 2024-03-11T21:39:32.244 INFO:teuthology.orchestra.run.smithi092.stdout: "client_addr": "v1:172.21.15.92:0/2489519821", 2024-03-11T21:39:32.244 INFO:teuthology.orchestra.run.smithi092.stdout: "tid": 159244 2024-03-11T21:39:32.244 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:39:32.244 INFO:teuthology.orchestra.run.smithi092.stdout: "events": [ 2024-03-11T21:39:32.244 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:39:32.244 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "initiated", 2024-03-11T21:39:32.244 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:38:21.661718+0000", 2024-03-11T21:39:32.244 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-03-11T21:39:32.244 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:39:32.245 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:39:32.245 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "header_read", 2024-03-11T21:39:32.245 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:38:21.661718+0000", 2024-03-11T21:39:32.245 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-03-11T21:39:32.245 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:39:32.245 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:39:32.245 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "throttled", 2024-03-11T21:39:32.245 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:38:21.661720+0000", 2024-03-11T21:39:32.245 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.3570000000000001e-06 2024-03-11T21:39:32.245 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:39:32.245 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:39:32.245 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "all_read", 2024-03-11T21:39:32.245 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:38:21.661724+0000", 2024-03-11T21:39:32.245 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 4.729e-06 2024-03-11T21:39:32.246 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:39:32.246 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:39:32.246 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "dispatched", 2024-03-11T21:39:32.246 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:38:21.661734+0000", 2024-03-11T21:39:32.246 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 9.978e-06 2024-03-11T21:39:32.246 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:39:32.246 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:39:32.246 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "queued_for_pg", 2024-03-11T21:39:32.246 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:38:21.661740+0000", 2024-03-11T21:39:32.246 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 6.0460000000000002e-06 2024-03-11T21:39:32.246 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:39:32.246 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:39:32.246 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-03-11T21:39:32.246 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:38:21.661802+0000", 2024-03-11T21:39:32.247 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 6.2003999999999998e-05 2024-03-11T21:39:32.247 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:39:32.247 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:39:32.247 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "waiting for peered", 2024-03-11T21:39:32.247 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:38:21.661814+0000", 2024-03-11T21:39:32.247 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.1292e-05 2024-03-11T21:39:32.247 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:39:32.247 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:39:32.247 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-03-11T21:39:32.247 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:38:22.587531+0000", 2024-03-11T21:39:32.247 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.92571720999999996 2024-03-11T21:39:32.247 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:39:32.247 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:39:32.247 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "started", 2024-03-11T21:39:32.248 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:38:22.587615+0000", 2024-03-11T21:39:32.248 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 8.4394000000000005e-05 2024-03-11T21:39:32.248 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:39:32.248 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:39:32.248 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "waiting for subops from 6,7", 2024-03-11T21:39:32.248 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:38:22.587690+0000", 2024-03-11T21:39:32.248 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 7.5152000000000001e-05 2024-03-11T21:39:32.248 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:39:32.248 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:39:32.248 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "op_commit", 2024-03-11T21:39:32.248 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:38:22.590743+0000", 2024-03-11T21:39:32.248 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.0030522909999999999 2024-03-11T21:39:32.248 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:39:32.248 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:39:32.249 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "sub_op_commit_rec", 2024-03-11T21:39:32.249 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:38:22.596567+0000", 2024-03-11T21:39:32.249 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.005823801 2024-03-11T21:39:32.249 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:39:32.249 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:39:32.249 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "sub_op_commit_rec", 2024-03-11T21:39:32.249 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:38:22.603413+0000", 2024-03-11T21:39:32.249 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.0068468119999999999 2024-03-11T21:39:32.249 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:39:32.249 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:39:32.249 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "commit_sent", 2024-03-11T21:39:32.249 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:38:22.603465+0000", 2024-03-11T21:39:32.249 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 5.1399e-05 2024-03-11T21:39:32.249 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:39:32.249 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:39:32.250 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "done", 2024-03-11T21:39:32.250 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:38:22.603519+0000", 2024-03-11T21:39:32.250 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 5.3724000000000003e-05 2024-03-11T21:39:32.250 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-03-11T21:39:32.250 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-03-11T21:39:32.250 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-03-11T21:39:32.250 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:39:32.250 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:39:32.250 INFO:teuthology.orchestra.run.smithi092.stdout: "description": "osd_op(client.25282.0:159273 9.1 9:81844339:::benchmark_data_smithi092_130291_object42889:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e316)", 2024-03-11T21:39:32.250 INFO:teuthology.orchestra.run.smithi092.stdout: "initiated_at": "2024-03-11T21:38:21.709414+0000", 2024-03-11T21:39:32.250 INFO:teuthology.orchestra.run.smithi092.stdout: "age": 70.445354961999996, 2024-03-11T21:39:32.250 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.89421969400000001, 2024-03-11T21:39:32.250 INFO:teuthology.orchestra.run.smithi092.stdout: "type_data": { 2024-03-11T21:39:32.250 INFO:teuthology.orchestra.run.smithi092.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-11T21:39:32.251 INFO:teuthology.orchestra.run.smithi092.stdout: "client_info": { 2024-03-11T21:39:32.251 INFO:teuthology.orchestra.run.smithi092.stdout: "client": "client.25282", 2024-03-11T21:39:32.251 INFO:teuthology.orchestra.run.smithi092.stdout: "client_addr": "v1:172.21.15.92:0/2489519821", 2024-03-11T21:39:32.251 INFO:teuthology.orchestra.run.smithi092.stdout: "tid": 159273 2024-03-11T21:39:32.251 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:39:32.251 INFO:teuthology.orchestra.run.smithi092.stdout: "events": [ 2024-03-11T21:39:32.251 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:39:32.251 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "initiated", 2024-03-11T21:39:32.251 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:38:21.709414+0000", 2024-03-11T21:39:32.251 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-03-11T21:39:32.251 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:39:32.251 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:39:32.251 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "header_read", 2024-03-11T21:39:32.251 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:38:21.709414+0000", 2024-03-11T21:39:32.252 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-03-11T21:39:32.252 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:39:32.252 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:39:32.252 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "throttled", 2024-03-11T21:39:32.252 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:38:21.709416+0000", 2024-03-11T21:39:32.252 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 2.2570000000000002e-06 2024-03-11T21:39:32.252 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:39:32.252 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:39:32.252 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "all_read", 2024-03-11T21:39:32.252 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:38:21.709422+0000", 2024-03-11T21:39:32.252 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 6.1240000000000002e-06 2024-03-11T21:39:32.252 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:39:32.252 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:39:32.252 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "dispatched", 2024-03-11T21:39:32.252 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:38:21.709437+0000", 2024-03-11T21:39:32.253 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.5359999999999999e-05 2024-03-11T21:39:32.253 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:39:32.253 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:39:32.253 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "queued_for_pg", 2024-03-11T21:39:32.253 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:38:21.709446+0000", 2024-03-11T21:39:32.253 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 8.5180000000000004e-06 2024-03-11T21:39:32.253 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:39:32.253 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:39:32.253 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-03-11T21:39:32.253 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:38:21.709515+0000", 2024-03-11T21:39:32.253 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 6.8748e-05 2024-03-11T21:39:32.253 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:39:32.253 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:39:32.253 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "waiting for peered", 2024-03-11T21:39:32.253 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:38:21.709525+0000", 2024-03-11T21:39:32.254 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.0614999999999999e-05 2024-03-11T21:39:32.254 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:39:32.254 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:39:32.254 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-03-11T21:39:32.254 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:38:22.590766+0000", 2024-03-11T21:39:32.254 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.88124031400000002 2024-03-11T21:39:32.254 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:39:32.254 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:39:32.254 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "started", 2024-03-11T21:39:32.254 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:38:22.590884+0000", 2024-03-11T21:39:32.254 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00011848199999999999 2024-03-11T21:39:32.254 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:39:32.254 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:39:32.254 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "waiting for subops from 6,7", 2024-03-11T21:39:32.255 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:38:22.590997+0000", 2024-03-11T21:39:32.255 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.000112977 2024-03-11T21:39:32.255 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:39:32.255 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:39:32.255 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "op_commit", 2024-03-11T21:39:32.255 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:38:22.596523+0000", 2024-03-11T21:39:32.255 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.0055256929999999999 2024-03-11T21:39:32.255 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:39:32.255 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:39:32.255 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "sub_op_commit_rec", 2024-03-11T21:39:32.255 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:38:22.603340+0000", 2024-03-11T21:39:32.255 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.0068174539999999997 2024-03-11T21:39:32.255 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:39:32.255 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:39:32.256 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "sub_op_commit_rec", 2024-03-11T21:39:32.256 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:38:22.603555+0000", 2024-03-11T21:39:32.256 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00021494099999999999 2024-03-11T21:39:32.256 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:39:32.256 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:39:32.256 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "commit_sent", 2024-03-11T21:39:32.256 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:38:22.603601+0000", 2024-03-11T21:39:32.256 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 4.5640999999999999e-05 2024-03-11T21:39:32.256 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:39:32.256 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:39:32.256 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "done", 2024-03-11T21:39:32.256 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:38:22.603633+0000", 2024-03-11T21:39:32.256 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 3.2570000000000002e-05 2024-03-11T21:39:32.256 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-03-11T21:39:32.256 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-03-11T21:39:32.257 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-03-11T21:39:32.257 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:39:32.257 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:39:32.257 INFO:teuthology.orchestra.run.smithi092.stdout: "description": "osd_op(client.25282.0:159291 9.1 9:807ee25e:::benchmark_data_smithi092_130291_object42907:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e316)", 2024-03-11T21:39:32.257 INFO:teuthology.orchestra.run.smithi092.stdout: "initiated_at": "2024-03-11T21:38:21.736587+0000", 2024-03-11T21:39:32.257 INFO:teuthology.orchestra.run.smithi092.stdout: "age": 70.418181426999993, 2024-03-11T21:39:32.257 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.88000051099999999, 2024-03-11T21:39:32.257 INFO:teuthology.orchestra.run.smithi092.stdout: "type_data": { 2024-03-11T21:39:32.257 INFO:teuthology.orchestra.run.smithi092.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-11T21:39:32.257 INFO:teuthology.orchestra.run.smithi092.stdout: "client_info": { 2024-03-11T21:39:32.257 INFO:teuthology.orchestra.run.smithi092.stdout: "client": "client.25282", 2024-03-11T21:39:32.257 INFO:teuthology.orchestra.run.smithi092.stdout: "client_addr": "v1:172.21.15.92:0/2489519821", 2024-03-11T21:39:32.257 INFO:teuthology.orchestra.run.smithi092.stdout: "tid": 159291 2024-03-11T21:39:32.257 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:39:32.258 INFO:teuthology.orchestra.run.smithi092.stdout: "events": [ 2024-03-11T21:39:32.258 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:39:32.258 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "initiated", 2024-03-11T21:39:32.258 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:38:21.736587+0000", 2024-03-11T21:39:32.258 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-03-11T21:39:32.258 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:39:32.258 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:39:32.258 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "header_read", 2024-03-11T21:39:32.258 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:38:21.736587+0000", 2024-03-11T21:39:32.258 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-03-11T21:39:32.258 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:39:32.258 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:39:32.258 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "throttled", 2024-03-11T21:39:32.258 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:38:21.736590+0000", 2024-03-11T21:39:32.259 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 2.266e-06 2024-03-11T21:39:32.259 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:39:32.259 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:39:32.259 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "all_read", 2024-03-11T21:39:32.259 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:38:21.736597+0000", 2024-03-11T21:39:32.259 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 7.9200000000000004e-06 2024-03-11T21:39:32.259 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:39:32.259 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:39:32.259 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "dispatched", 2024-03-11T21:39:32.259 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:38:21.736612+0000", 2024-03-11T21:39:32.259 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.4729000000000001e-05 2024-03-11T21:39:32.259 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:39:32.259 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:39:32.259 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "queued_for_pg", 2024-03-11T21:39:32.260 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:38:21.736620+0000", 2024-03-11T21:39:32.260 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 7.5989999999999996e-06 2024-03-11T21:39:32.260 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:39:32.260 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:39:32.260 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-03-11T21:39:32.260 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:38:21.736684+0000", 2024-03-11T21:39:32.260 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 6.3830000000000004e-05 2024-03-11T21:39:32.260 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:39:32.260 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:39:32.260 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "waiting for peered", 2024-03-11T21:39:32.260 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:38:21.736694+0000", 2024-03-11T21:39:32.260 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.0227999999999999e-05 2024-03-11T21:39:32.260 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:39:32.260 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:39:32.261 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-03-11T21:39:32.261 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:38:22.597213+0000", 2024-03-11T21:39:32.261 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.86051923900000005 2024-03-11T21:39:32.261 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:39:32.261 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:39:32.261 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "started", 2024-03-11T21:39:32.261 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:38:22.597311+0000", 2024-03-11T21:39:32.261 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 9.8121999999999993e-05 2024-03-11T21:39:32.261 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:39:32.261 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:39:32.261 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "waiting for subops from 6,7", 2024-03-11T21:39:32.261 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:38:22.597441+0000", 2024-03-11T21:39:32.261 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00012948500000000001 2024-03-11T21:39:32.261 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:39:32.262 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:39:32.262 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "sub_op_commit_rec", 2024-03-11T21:39:32.262 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:38:22.606649+0000", 2024-03-11T21:39:32.262 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.0092086919999999992 2024-03-11T21:39:32.262 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:39:32.262 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:39:32.262 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "sub_op_commit_rec", 2024-03-11T21:39:32.262 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:38:22.606718+0000", 2024-03-11T21:39:32.262 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 6.8763000000000006e-05 2024-03-11T21:39:32.262 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:39:32.262 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:39:32.262 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "op_commit", 2024-03-11T21:39:32.262 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:38:22.616436+0000", 2024-03-11T21:39:32.262 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.0097176869999999992 2024-03-11T21:39:32.263 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:39:32.263 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:39:32.263 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "commit_sent", 2024-03-11T21:39:32.263 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:38:22.616541+0000", 2024-03-11T21:39:32.263 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.000105028 2024-03-11T21:39:32.263 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:39:32.263 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:39:32.263 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "done", 2024-03-11T21:39:32.263 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:38:22.616588+0000", 2024-03-11T21:39:32.263 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 4.6922999999999998e-05 2024-03-11T21:39:32.263 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-03-11T21:39:32.263 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-03-11T21:39:32.263 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-03-11T21:39:32.263 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-03-11T21:39:32.263 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-03-11T21:39:32.264 INFO:teuthology.orchestra.run.smithi092.stdout:} 2024-03-11T21:39:32.285 INFO:teuthology.orchestra.run.smithi092.stdout: 2024-03-11T21:39:32.291 INFO:teuthology.orchestra.run.smithi092.stderr:dumped all 2024-03-11T21:39:32.316 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:39:32.344 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:39:32.345 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:39:32.345 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:39:32.446 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:39:32.474 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:39:32.475 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:39:32.476 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:39:32.576 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:39:32.620 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:39:32.621 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:39:32.621 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:39:32.722 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:39:32.761 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:39:32.762 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:39:32.762 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:39:32.863 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:39:32 smithi092 ceph-mon[37563]: pgmap v643: 148 pgs: 2 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+degraded+remapped, 143 active+clean; 1.3 GiB data, 7.5 GiB used, 708 GiB / 715 GiB avail; 638 B/s rd, 0 B/s wr, 711 op/s; 19530/65661 objects degraded (29.744%); 0 B/s, 70 objects/s recovering 2024-03-11T21:39:32.863 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:39:32 smithi092 ceph-mon[37563]: purged_snaps scrub starts 2024-03-11T21:39:32.863 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:39:32 smithi092 ceph-mon[37563]: purged_snaps scrub ok 2024-03-11T21:39:32.863 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:39:32 smithi092 ceph-mon[37563]: osdmap e333: 8 total, 8 up, 8 in 2024-03-11T21:39:32.863 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:39:32 smithi092 ceph-mon[37563]: purged_snaps scrub starts 2024-03-11T21:39:32.863 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:39:32 smithi092 ceph-mon[37563]: purged_snaps scrub ok 2024-03-11T21:39:32.864 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:39:32 smithi092 ceph-mon[29572]: pgmap v643: 148 pgs: 2 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+degraded+remapped, 143 active+clean; 1.3 GiB data, 7.5 GiB used, 708 GiB / 715 GiB avail; 638 B/s rd, 0 B/s wr, 711 op/s; 19530/65661 objects degraded (29.744%); 0 B/s, 70 objects/s recovering 2024-03-11T21:39:32.864 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:39:32 smithi092 ceph-mon[29572]: purged_snaps scrub starts 2024-03-11T21:39:32.864 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:39:32 smithi092 ceph-mon[29572]: purged_snaps scrub ok 2024-03-11T21:39:32.864 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:39:32 smithi092 ceph-mon[29572]: osdmap e333: 8 total, 8 up, 8 in 2024-03-11T21:39:32.864 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:39:32 smithi092 ceph-mon[29572]: purged_snaps scrub starts 2024-03-11T21:39:32.864 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:39:32 smithi092 ceph-mon[29572]: purged_snaps scrub ok 2024-03-11T21:39:32.865 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:39:32.893 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:39:32.894 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:39:32.894 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:39:32.995 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:39:33.023 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:39:33.024 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:39:33.025 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:39:33.078 INFO:journalctl@ceph.iscsi.iscsi.a.smithi097.stdout:Mar 11 21:39:32 smithi097 ceph-00db0016-dfed-11ee-95c8-87774f69a715-iscsi-iscsi-a[60943]: debug there is no tcmu-runner data available 2024-03-11T21:39:33.079 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:39:32 smithi097 ceph-mon[33317]: pgmap v643: 148 pgs: 2 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+degraded+remapped, 143 active+clean; 1.3 GiB data, 7.5 GiB used, 708 GiB / 715 GiB avail; 638 B/s rd, 0 B/s wr, 711 op/s; 19530/65661 objects degraded (29.744%); 0 B/s, 70 objects/s recovering 2024-03-11T21:39:33.079 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:39:32 smithi097 ceph-mon[33317]: purged_snaps scrub starts 2024-03-11T21:39:33.079 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:39:32 smithi097 ceph-mon[33317]: purged_snaps scrub ok 2024-03-11T21:39:33.079 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:39:32 smithi097 ceph-mon[33317]: osdmap e333: 8 total, 8 up, 8 in 2024-03-11T21:39:33.079 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:39:32 smithi097 ceph-mon[33317]: purged_snaps scrub starts 2024-03-11T21:39:33.079 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:39:32 smithi097 ceph-mon[33317]: purged_snaps scrub ok 2024-03-11T21:39:33.126 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:39:33.153 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:39:33.154 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:39:33.154 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:39:33.255 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:39:33.284 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:39:33.285 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:39:33.285 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:39:33.386 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:39:33.434 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:39:33.435 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:39:33.435 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:39:33.536 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:39:33.565 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:39:33.565 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:39:33.566 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:39:33.666 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:39:33.710 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:39:33.711 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:39:33.712 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:39:33.812 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:39:33.842 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:39:33.843 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:39:33.843 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:39:33.924 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:39:33 smithi092 ceph-mon[29572]: from='client.15660 v1:172.21.15.92:0/2148840832' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-11T21:39:33.924 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:39:33 smithi092 ceph-mon[29572]: osdmap e334: 8 total, 8 up, 8 in 2024-03-11T21:39:33.924 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:39:33 smithi092 ceph-mon[29572]: from='client.24469 v1:172.21.15.97:0/404208310' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:39:33.924 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:39:33 smithi092 ceph-mon[29572]: Health check update: Degraded data redundancy: 19530/65661 objects degraded (29.744%), 5 pgs degraded, 5 pgs undersized (PG_DEGRADED) 2024-03-11T21:39:33.925 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:39:33 smithi092 ceph-mon[37563]: from='client.15660 v1:172.21.15.92:0/2148840832' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-11T21:39:33.925 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:39:33 smithi092 ceph-mon[37563]: osdmap e334: 8 total, 8 up, 8 in 2024-03-11T21:39:33.925 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:39:33 smithi092 ceph-mon[37563]: from='client.24469 v1:172.21.15.97:0/404208310' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:39:33.925 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:39:33 smithi092 ceph-mon[37563]: Health check update: Degraded data redundancy: 19530/65661 objects degraded (29.744%), 5 pgs degraded, 5 pgs undersized (PG_DEGRADED) 2024-03-11T21:39:33.944 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:39:33.976 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:39:33.977 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:39:33.978 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:39:34.064 DEBUG:teuthology.orchestra.run.smithi092:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid 00db0016-dfed-11ee-95c8-87774f69a715 -- ceph daemon osd.0 dump_ops_in_flight 2024-03-11T21:39:34.078 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:39:34.079 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:39:33 smithi097 ceph-mon[33317]: from='client.15660 v1:172.21.15.92:0/2148840832' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-11T21:39:34.079 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:39:33 smithi097 ceph-mon[33317]: osdmap e334: 8 total, 8 up, 8 in 2024-03-11T21:39:34.079 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:39:33 smithi097 ceph-mon[33317]: from='client.24469 v1:172.21.15.97:0/404208310' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:39:34.079 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:39:33 smithi097 ceph-mon[33317]: Health check update: Degraded data redundancy: 19530/65661 objects degraded (29.744%), 5 pgs degraded, 5 pgs undersized (PG_DEGRADED) 2024-03-11T21:39:34.158 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:39:34.159 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:39:34.159 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:39:34.261 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:39:34.305 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:39:34.305 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:39:34.306 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:39:34.406 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:39:34.473 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:39:34.474 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:39:34.474 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:39:34.575 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:39:34.616 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:39:34.617 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:39:34.617 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:39:34.718 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:39:34.757 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:39:34.758 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:39:34.758 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:39:34.860 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:39:34.902 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:39:34.902 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:39:34.902 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:39:34.928 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:39:34 smithi092 ceph-mon[29572]: pgmap v646: 148 pgs: 2 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+degraded+remapped, 143 active+clean; 1.2 GiB data, 7.0 GiB used, 708 GiB / 715 GiB avail; 1.7 KiB/s rd, 0 B/s wr, 1.14k op/s; 19243/57885 objects degraded (33.244%); 0 B/s, 121 objects/s recovering 2024-03-11T21:39:34.928 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:39:34 smithi092 ceph-mon[29572]: from='mgr.14607 v1:172.21.15.92:0/1702717597' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T21:39:34.928 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:39:34 smithi092 ceph-mon[37563]: pgmap v646: 148 pgs: 2 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+degraded+remapped, 143 active+clean; 1.2 GiB data, 7.0 GiB used, 708 GiB / 715 GiB avail; 1.7 KiB/s rd, 0 B/s wr, 1.14k op/s; 19243/57885 objects degraded (33.244%); 0 B/s, 121 objects/s recovering 2024-03-11T21:39:34.928 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:39:34 smithi092 ceph-mon[37563]: from='mgr.14607 v1:172.21.15.92:0/1702717597' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T21:39:35.003 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:39:35.040 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:39:35.040 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:39:35.040 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:39:35.080 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:39:34 smithi097 ceph-mon[33317]: pgmap v646: 148 pgs: 2 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+degraded+remapped, 143 active+clean; 1.2 GiB data, 7.0 GiB used, 708 GiB / 715 GiB avail; 1.7 KiB/s rd, 0 B/s wr, 1.14k op/s; 19243/57885 objects degraded (33.244%); 0 B/s, 121 objects/s recovering 2024-03-11T21:39:35.080 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:39:34 smithi097 ceph-mon[33317]: from='mgr.14607 v1:172.21.15.92:0/1702717597' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T21:39:35.142 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:39:35.191 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:39:35.191 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:39:35.192 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:39:35.272 INFO:teuthology.orchestra.run.smithi092.stdout:{"pg_ready":true,"pg_map":{"version":644,"stamp":"2024-03-11T21:39:31.632292+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":1421941184,"num_objects":21887,"num_object_clones":0,"num_object_copies":65661,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19530,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21887,"num_whiteouts":0,"num_read":2158,"num_read_kb":2006,"num_write":211976,"num_write_kb":7450846,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18696,"num_bytes_recovered":1092753862,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":83419,"ondisk_log_size":83419,"up":444,"acting":439,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":444,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":749961216,"kb_used":7912876,"kb_used_data":4279060,"kb_used_omap":15,"kb_used_meta":3633712,"kb_avail":742048340,"statfs":{"total":767960285184,"available":759857500160,"internally_reserved":0,"allocated":4381757440,"data_stored":4374950533,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15601,"internal_metadata":3720921871},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,3,4,3,8,4,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-277938176,"num_objects":-4241,"num_object_clones":0,"num_object_copies":-12723,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-445,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-4241,"num_whiteouts":0,"num_read":5,"num_read_kb":5,"num_write":4241,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":445,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.340012"},"pg_stats":[{"pgid":"6.1b","version":"0'0","reported_seq":514,"reported_epoch":332,"state":"active+clean","last_fresh":"2024-03-11T21:39:27.610955+0000","last_change":"2024-03-11T21:36:35.074931+0000","last_active":"2024-03-11T21:39:27.610955+0000","last_peered":"2024-03-11T21:39:27.610955+0000","last_clean":"2024-03-11T21:39:27.610955+0000","last_became_active":"2024-03-11T21:36:35.072636+0000","last_became_peered":"2024-03-11T21:36:35.072636+0000","last_unstale":"2024-03-11T21:39:27.610955+0000","last_undegraded":"2024-03-11T21:39:27.610955+0000","last_fullsized":"2024-03-11T21:39:27.610955+0000","mapping_epoch":277,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":278,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:35:07.021520+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:43.559452+0000","last_clean_scrub_stamp":"2024-03-11T21:35:07.021520+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":576,"reported_epoch":331,"state":"active+clean","last_fresh":"2024-03-11T21:39:26.584856+0000","last_change":"2024-03-11T21:36:00.720089+0000","last_active":"2024-03-11T21:39:26.584856+0000","last_peered":"2024-03-11T21:39:26.584856+0000","last_clean":"2024-03-11T21:39:26.584856+0000","last_became_active":"2024-03-11T21:25:42.578893+0000","last_became_peered":"2024-03-11T21:25:42.578893+0000","last_unstale":"2024-03-11T21:39:26.584856+0000","last_undegraded":"2024-03-11T21:39:26.584856+0000","last_fullsized":"2024-03-11T21:39:26.584856+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:36:00.719956+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:41.544550+0000","last_clean_scrub_stamp":"2024-03-11T21:36:00.719956+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1f","version":"0'0","reported_seq":607,"reported_epoch":332,"state":"active+clean","last_fresh":"2024-03-11T21:39:27.603128+0000","last_change":"2024-03-11T21:36:00.843838+0000","last_active":"2024-03-11T21:39:27.603128+0000","last_peered":"2024-03-11T21:39:27.603128+0000","last_clean":"2024-03-11T21:39:27.603128+0000","last_became_active":"2024-03-11T21:34:41.144945+0000","last_became_peered":"2024-03-11T21:34:41.144945+0000","last_unstale":"2024-03-11T21:39:27.603128+0000","last_undegraded":"2024-03-11T21:39:27.603128+0000","last_fullsized":"2024-03-11T21:39:27.603128+0000","mapping_epoch":226,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":227,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:36:00.843718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:37.512018+0000","last_clean_scrub_stamp":"2024-03-11T21:36:00.843718+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,4],"acting":[0,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.19","version":"260'27","reported_seq":561,"reported_epoch":332,"state":"active+clean","last_fresh":"2024-03-11T21:39:27.613121+0000","last_change":"2024-03-11T21:36:35.089901+0000","last_active":"2024-03-11T21:39:27.613121+0000","last_peered":"2024-03-11T21:39:27.613121+0000","last_clean":"2024-03-11T21:39:27.613121+0000","last_became_active":"2024-03-11T21:36:35.067652+0000","last_became_peered":"2024-03-11T21:36:35.067652+0000","last_unstale":"2024-03-11T21:39:27.613121+0000","last_undegraded":"2024-03-11T21:39:27.613121+0000","last_fullsized":"2024-03-11T21:39:27.613121+0000","mapping_epoch":277,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":278,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'15","last_scrub_stamp":"2024-03-11T21:35:22.099374+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:39.525554+0000","last_clean_scrub_stamp":"2024-03-11T21:35:22.099374+0000","objects_scrubbed":0,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":42,"num_read_kb":33,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.1a","version":"0'0","reported_seq":564,"reported_epoch":331,"state":"active+clean","last_fresh":"2024-03-11T21:39:26.585453+0000","last_change":"2024-03-11T21:35:16.410927+0000","last_active":"2024-03-11T21:39:26.585453+0000","last_peered":"2024-03-11T21:39:26.585453+0000","last_clean":"2024-03-11T21:39:26.585453+0000","last_became_active":"2024-03-11T21:25:44.590923+0000","last_became_peered":"2024-03-11T21:25:44.590923+0000","last_unstale":"2024-03-11T21:39:26.585453+0000","last_undegraded":"2024-03-11T21:39:26.585453+0000","last_fullsized":"2024-03-11T21:39:26.585453+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:35:16.410816+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:43.559452+0000","last_clean_scrub_stamp":"2024-03-11T21:35:16.410816+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":24,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,1],"acting":[4,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":590,"reported_epoch":332,"state":"active+clean","last_fresh":"2024-03-11T21:39:27.597153+0000","last_change":"2024-03-11T21:36:15.576138+0000","last_active":"2024-03-11T21:39:27.597153+0000","last_peered":"2024-03-11T21:39:27.597153+0000","last_clean":"2024-03-11T21:39:27.597153+0000","last_became_active":"2024-03-11T21:34:41.154216+0000","last_became_peered":"2024-03-11T21:34:41.154216+0000","last_unstale":"2024-03-11T21:39:27.597153+0000","last_undegraded":"2024-03-11T21:39:27.597153+0000","last_fullsized":"2024-03-11T21:39:27.597153+0000","mapping_epoch":226,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":227,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:36:15.575887+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:41.544550+0000","last_clean_scrub_stamp":"2024-03-11T21:36:15.575887+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,7],"acting":[1,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1e","version":"0'0","reported_seq":527,"reported_epoch":332,"state":"active+clean","last_fresh":"2024-03-11T21:39:27.610195+0000","last_change":"2024-03-11T21:36:35.062413+0000","last_active":"2024-03-11T21:39:27.610195+0000","last_peered":"2024-03-11T21:39:27.610195+0000","last_clean":"2024-03-11T21:39:27.610195+0000","last_became_active":"2024-03-11T21:36:35.061786+0000","last_became_peered":"2024-03-11T21:36:35.061786+0000","last_unstale":"2024-03-11T21:39:27.610195+0000","last_undegraded":"2024-03-11T21:39:27.610195+0000","last_fullsized":"2024-03-11T21:39:27.610195+0000","mapping_epoch":277,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":278,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:32:18.578038+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:37.512018+0000","last_clean_scrub_stamp":"2024-03-11T21:32:18.578038+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.18","version":"260'15","reported_seq":613,"reported_epoch":331,"state":"active+clean","last_fresh":"2024-03-11T21:39:26.584869+0000","last_change":"2024-03-11T21:36:09.662765+0000","last_active":"2024-03-11T21:39:26.584869+0000","last_peered":"2024-03-11T21:39:26.584869+0000","last_clean":"2024-03-11T21:39:26.584869+0000","last_became_active":"2024-03-11T21:25:40.552216+0000","last_became_peered":"2024-03-11T21:25:40.552216+0000","last_unstale":"2024-03-11T21:39:26.584869+0000","last_undegraded":"2024-03-11T21:39:26.584869+0000","last_fullsized":"2024-03-11T21:39:26.584869+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"260'15","last_scrub_stamp":"2024-03-11T21:36:09.662615+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:39.525554+0000","last_clean_scrub_stamp":"2024-03-11T21:36:09.662615+0000","objects_scrubbed":4,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":581,"reported_epoch":331,"state":"active+clean","last_fresh":"2024-03-11T21:39:26.602836+0000","last_change":"2024-03-11T21:36:35.068915+0000","last_active":"2024-03-11T21:39:26.602836+0000","last_peered":"2024-03-11T21:39:26.602836+0000","last_clean":"2024-03-11T21:39:26.602836+0000","last_became_active":"2024-03-11T21:36:35.068453+0000","last_became_peered":"2024-03-11T21:36:35.068453+0000","last_unstale":"2024-03-11T21:39:26.602836+0000","last_undegraded":"2024-03-11T21:39:26.602836+0000","last_fullsized":"2024-03-11T21:39:26.602836+0000","mapping_epoch":277,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":278,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:35:26.190634+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:32:12.654573+0000","last_clean_scrub_stamp":"2024-03-11T21:35:26.190634+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3],"acting":[5,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":228,"reported_epoch":331,"state":"active+clean","last_fresh":"2024-03-11T21:39:26.601412+0000","last_change":"2024-03-11T21:35:11.902920+0000","last_active":"2024-03-11T21:39:26.601412+0000","last_peered":"2024-03-11T21:39:26.601412+0000","last_clean":"2024-03-11T21:39:26.601412+0000","last_became_active":"2024-03-11T21:34:41.145299+0000","last_became_peered":"2024-03-11T21:34:41.145299+0000","last_unstale":"2024-03-11T21:39:26.601412+0000","last_undegraded":"2024-03-11T21:39:26.601412+0000","last_fullsized":"2024-03-11T21:39:26.601412+0000","mapping_epoch":226,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":227,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:35:11.902818+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:41.544550+0000","last_clean_scrub_stamp":"2024-03-11T21:35:11.902818+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.1d","version":"0'0","reported_seq":591,"reported_epoch":331,"state":"active+clean","last_fresh":"2024-03-11T21:39:26.602192+0000","last_change":"2024-03-11T21:35:26.855588+0000","last_active":"2024-03-11T21:39:26.602192+0000","last_peered":"2024-03-11T21:39:26.602192+0000","last_clean":"2024-03-11T21:39:26.602192+0000","last_became_active":"2024-03-11T21:25:38.542628+0000","last_became_peered":"2024-03-11T21:25:38.542628+0000","last_unstale":"2024-03-11T21:39:26.602192+0000","last_undegraded":"2024-03-11T21:39:26.602192+0000","last_fullsized":"2024-03-11T21:39:26.602192+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:35:26.855511+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:37.512018+0000","last_clean_scrub_stamp":"2024-03-11T21:35:26.855511+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,0],"acting":[7,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.1b","version":"260'7","reported_seq":599,"reported_epoch":331,"state":"active+clean","last_fresh":"2024-03-11T21:39:26.586071+0000","last_change":"2024-03-11T21:36:35.087276+0000","last_active":"2024-03-11T21:39:26.586071+0000","last_peered":"2024-03-11T21:39:26.586071+0000","last_clean":"2024-03-11T21:39:26.586071+0000","last_became_active":"2024-03-11T21:36:35.057678+0000","last_became_peered":"2024-03-11T21:36:35.057678+0000","last_unstale":"2024-03-11T21:39:26.586071+0000","last_undegraded":"2024-03-11T21:39:26.586071+0000","last_fullsized":"2024-03-11T21:39:26.586071+0000","mapping_epoch":277,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":278,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'5","last_scrub_stamp":"2024-03-11T21:35:09.427562+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:39.525554+0000","last_clean_scrub_stamp":"2024-03-11T21:35:09.427562+0000","objects_scrubbed":2,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":14,"num_read_kb":10,"num_write":8,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,1],"acting":[4,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":574,"reported_epoch":332,"state":"active+clean","last_fresh":"2024-03-11T21:39:27.606182+0000","last_change":"2024-03-11T21:35:39.691825+0000","last_active":"2024-03-11T21:39:27.606182+0000","last_peered":"2024-03-11T21:39:27.606182+0000","last_clean":"2024-03-11T21:39:27.606182+0000","last_became_active":"2024-03-11T21:25:44.589786+0000","last_became_peered":"2024-03-11T21:25:44.589786+0000","last_unstale":"2024-03-11T21:39:27.606182+0000","last_undegraded":"2024-03-11T21:39:27.606182+0000","last_fullsized":"2024-03-11T21:39:27.606182+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:35:39.691690+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:43.559452+0000","last_clean_scrub_stamp":"2024-03-11T21:35:39.691690+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":18,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,7],"acting":[0,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":587,"reported_epoch":331,"state":"active+clean","last_fresh":"2024-03-11T21:39:26.603651+0000","last_change":"2024-03-11T21:35:25.194240+0000","last_active":"2024-03-11T21:39:26.603651+0000","last_peered":"2024-03-11T21:39:26.603651+0000","last_clean":"2024-03-11T21:39:26.603651+0000","last_became_active":"2024-03-11T21:34:41.147185+0000","last_became_peered":"2024-03-11T21:34:41.147185+0000","last_unstale":"2024-03-11T21:39:26.603651+0000","last_undegraded":"2024-03-11T21:39:26.603651+0000","last_fullsized":"2024-03-11T21:39:26.603651+0000","mapping_epoch":226,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":227,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:35:25.194115+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:32:21.535034+0000","last_clean_scrub_stamp":"2024-03-11T21:35:25.194115+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,7],"acting":[5,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1c","version":"0'0","reported_seq":591,"reported_epoch":331,"state":"active+clean","last_fresh":"2024-03-11T21:39:26.601987+0000","last_change":"2024-03-11T21:35:28.789693+0000","last_active":"2024-03-11T21:39:26.601987+0000","last_peered":"2024-03-11T21:39:26.601987+0000","last_clean":"2024-03-11T21:39:26.601987+0000","last_became_active":"2024-03-11T21:25:38.541364+0000","last_became_peered":"2024-03-11T21:25:38.541364+0000","last_unstale":"2024-03-11T21:39:26.601987+0000","last_undegraded":"2024-03-11T21:39:26.601987+0000","last_fullsized":"2024-03-11T21:39:26.601987+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:35:28.789592+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:28:32.566437+0000","last_clean_scrub_stamp":"2024-03-11T21:35:28.789592+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.1a","version":"260'15","reported_seq":623,"reported_epoch":331,"state":"active+clean","last_fresh":"2024-03-11T21:39:26.585542+0000","last_change":"2024-03-11T21:36:35.085358+0000","last_active":"2024-03-11T21:39:26.585542+0000","last_peered":"2024-03-11T21:39:26.585542+0000","last_clean":"2024-03-11T21:39:26.585542+0000","last_became_active":"2024-03-11T21:36:35.057029+0000","last_became_peered":"2024-03-11T21:36:35.057029+0000","last_unstale":"2024-03-11T21:39:26.585542+0000","last_undegraded":"2024-03-11T21:39:26.585542+0000","last_fullsized":"2024-03-11T21:39:26.585542+0000","mapping_epoch":277,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":278,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'9","last_scrub_stamp":"2024-03-11T21:35:05.451780+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:39.525554+0000","last_clean_scrub_stamp":"2024-03-11T21:35:05.451780+0000","objects_scrubbed":4,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0],"acting":[4,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1f","version":"0'0","reported_seq":450,"reported_epoch":332,"state":"active+clean","last_fresh":"2024-03-11T21:39:27.610778+0000","last_change":"2024-03-11T21:36:35.074675+0000","last_active":"2024-03-11T21:39:27.610778+0000","last_peered":"2024-03-11T21:39:27.610778+0000","last_clean":"2024-03-11T21:39:27.610778+0000","last_became_active":"2024-03-11T21:36:35.071755+0000","last_became_peered":"2024-03-11T21:36:35.071755+0000","last_unstale":"2024-03-11T21:39:27.610778+0000","last_undegraded":"2024-03-11T21:39:27.610778+0000","last_fullsized":"2024-03-11T21:39:27.610778+0000","mapping_epoch":277,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":278,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:34:05.862891+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:43.559452+0000","last_clean_scrub_stamp":"2024-03-11T21:34:05.862891+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1c","version":"0'0","reported_seq":577,"reported_epoch":331,"state":"active+clean","last_fresh":"2024-03-11T21:39:26.586547+0000","last_change":"2024-03-11T21:36:35.067469+0000","last_active":"2024-03-11T21:39:26.586547+0000","last_peered":"2024-03-11T21:39:26.586547+0000","last_clean":"2024-03-11T21:39:26.586547+0000","last_became_active":"2024-03-11T21:36:35.067127+0000","last_became_peered":"2024-03-11T21:36:35.067127+0000","last_unstale":"2024-03-11T21:39:26.586547+0000","last_undegraded":"2024-03-11T21:39:26.586547+0000","last_fullsized":"2024-03-11T21:39:26.586547+0000","mapping_epoch":277,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":278,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:34:11.766066+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:41.544550+0000","last_clean_scrub_stamp":"2024-03-11T21:34:11.766066+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1b","version":"0'0","reported_seq":499,"reported_epoch":332,"state":"active+clean","last_fresh":"2024-03-11T21:39:27.611050+0000","last_change":"2024-03-11T21:36:35.075385+0000","last_active":"2024-03-11T21:39:27.611050+0000","last_peered":"2024-03-11T21:39:27.611050+0000","last_clean":"2024-03-11T21:39:27.611050+0000","last_became_active":"2024-03-11T21:36:35.073959+0000","last_became_peered":"2024-03-11T21:36:35.073959+0000","last_unstale":"2024-03-11T21:39:27.611050+0000","last_undegraded":"2024-03-11T21:39:27.611050+0000","last_fullsized":"2024-03-11T21:39:27.611050+0000","mapping_epoch":277,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":278,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:34:27.998724+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:34:27.998724+0000","last_clean_scrub_stamp":"2024-03-11T21:34:27.998724+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1d","version":"260'18","reported_seq":543,"reported_epoch":332,"state":"active+clean","last_fresh":"2024-03-11T21:39:27.609833+0000","last_change":"2024-03-11T21:36:35.080207+0000","last_active":"2024-03-11T21:39:27.609833+0000","last_peered":"2024-03-11T21:39:27.609833+0000","last_clean":"2024-03-11T21:39:27.609833+0000","last_became_active":"2024-03-11T21:36:35.055880+0000","last_became_peered":"2024-03-11T21:36:35.055880+0000","last_unstale":"2024-03-11T21:39:27.609833+0000","last_undegraded":"2024-03-11T21:39:27.609833+0000","last_fullsized":"2024-03-11T21:39:27.609833+0000","mapping_epoch":277,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":278,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'12","last_scrub_stamp":"2024-03-11T21:35:07.971581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:39.525554+0000","last_clean_scrub_stamp":"2024-03-11T21:35:07.971581+0000","objects_scrubbed":0,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":33,"num_read_kb":24,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.1e","version":"0'0","reported_seq":565,"reported_epoch":331,"state":"active+clean","last_fresh":"2024-03-11T21:39:26.586900+0000","last_change":"2024-03-11T21:35:32.664451+0000","last_active":"2024-03-11T21:39:26.586900+0000","last_peered":"2024-03-11T21:39:26.586900+0000","last_clean":"2024-03-11T21:39:26.586900+0000","last_became_active":"2024-03-11T21:25:45.300355+0000","last_became_peered":"2024-03-11T21:25:45.300355+0000","last_unstale":"2024-03-11T21:39:26.586900+0000","last_undegraded":"2024-03-11T21:39:26.586900+0000","last_fullsized":"2024-03-11T21:39:26.586900+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:35:32.664371+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:43.559452+0000","last_clean_scrub_stamp":"2024-03-11T21:35:32.664371+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5],"acting":[4,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1d","version":"0'0","reported_seq":578,"reported_epoch":332,"state":"active+clean","last_fresh":"2024-03-11T21:39:27.597761+0000","last_change":"2024-03-11T21:36:07.624196+0000","last_active":"2024-03-11T21:39:27.597761+0000","last_peered":"2024-03-11T21:39:27.597761+0000","last_clean":"2024-03-11T21:39:27.597761+0000","last_became_active":"2024-03-11T21:25:42.579407+0000","last_became_peered":"2024-03-11T21:25:42.579407+0000","last_unstale":"2024-03-11T21:39:27.597761+0000","last_undegraded":"2024-03-11T21:39:27.597761+0000","last_fullsized":"2024-03-11T21:39:27.597761+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:36:07.624075+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:30:18.678531+0000","last_clean_scrub_stamp":"2024-03-11T21:36:07.624075+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1a","version":"0'0","reported_seq":604,"reported_epoch":331,"state":"active+clean","last_fresh":"2024-03-11T21:39:26.614426+0000","last_change":"2024-03-11T21:36:12.993542+0000","last_active":"2024-03-11T21:39:26.614426+0000","last_peered":"2024-03-11T21:39:26.614426+0000","last_clean":"2024-03-11T21:39:26.614426+0000","last_became_active":"2024-03-11T21:34:41.145014+0000","last_became_peered":"2024-03-11T21:34:41.145014+0000","last_unstale":"2024-03-11T21:39:26.614426+0000","last_undegraded":"2024-03-11T21:39:26.614426+0000","last_fullsized":"2024-03-11T21:39:26.614426+0000","mapping_epoch":226,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":227,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:36:12.993437+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:37.512018+0000","last_clean_scrub_stamp":"2024-03-11T21:36:12.993437+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.1c","version":"260'27","reported_seq":638,"reported_epoch":331,"state":"active+clean","last_fresh":"2024-03-11T21:39:26.625524+0000","last_change":"2024-03-11T21:36:35.097208+0000","last_active":"2024-03-11T21:39:26.625524+0000","last_peered":"2024-03-11T21:39:26.625524+0000","last_clean":"2024-03-11T21:39:26.625524+0000","last_became_active":"2024-03-11T21:36:35.074579+0000","last_became_peered":"2024-03-11T21:36:35.074579+0000","last_unstale":"2024-03-11T21:39:26.625524+0000","last_undegraded":"2024-03-11T21:39:26.625524+0000","last_fullsized":"2024-03-11T21:39:26.625524+0000","mapping_epoch":277,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":278,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'15","last_scrub_stamp":"2024-03-11T21:34:10.311181+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:39.525554+0000","last_clean_scrub_stamp":"2024-03-11T21:34:10.311181+0000","objects_scrubbed":7,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":44,"num_read_kb":35,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.1d","version":"0'0","reported_seq":566,"reported_epoch":332,"state":"active+clean","last_fresh":"2024-03-11T21:39:27.597441+0000","last_change":"2024-03-11T21:35:21.744646+0000","last_active":"2024-03-11T21:39:27.597441+0000","last_peered":"2024-03-11T21:39:27.597441+0000","last_clean":"2024-03-11T21:39:27.597441+0000","last_became_active":"2024-03-11T21:25:44.588845+0000","last_became_peered":"2024-03-11T21:25:44.588845+0000","last_unstale":"2024-03-11T21:39:27.597441+0000","last_undegraded":"2024-03-11T21:39:27.597441+0000","last_fullsized":"2024-03-11T21:39:27.597441+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:35:21.744223+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:43.559452+0000","last_clean_scrub_stamp":"2024-03-11T21:35:21.744223+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":99,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.1e","version":"0'0","reported_seq":579,"reported_epoch":332,"state":"active+clean","last_fresh":"2024-03-11T21:39:27.602921+0000","last_change":"2024-03-11T21:35:10.728691+0000","last_active":"2024-03-11T21:39:27.602921+0000","last_peered":"2024-03-11T21:39:27.602921+0000","last_clean":"2024-03-11T21:39:27.602921+0000","last_became_active":"2024-03-11T21:25:42.582959+0000","last_became_peered":"2024-03-11T21:25:42.582959+0000","last_unstale":"2024-03-11T21:39:27.602921+0000","last_undegraded":"2024-03-11T21:39:27.602921+0000","last_fullsized":"2024-03-11T21:39:27.602921+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:35:10.728569+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:26:58.335431+0000","last_clean_scrub_stamp":"2024-03-11T21:35:10.728569+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":18,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.19","version":"67'2","reported_seq":525,"reported_epoch":332,"state":"active+clean","last_fresh":"2024-03-11T21:39:27.609984+0000","last_change":"2024-03-11T21:36:35.069729+0000","last_active":"2024-03-11T21:39:27.609984+0000","last_peered":"2024-03-11T21:39:27.609984+0000","last_clean":"2024-03-11T21:39:27.609984+0000","last_became_active":"2024-03-11T21:36:35.064249+0000","last_became_peered":"2024-03-11T21:36:35.064249+0000","last_unstale":"2024-03-11T21:39:27.609984+0000","last_undegraded":"2024-03-11T21:39:27.609984+0000","last_fullsized":"2024-03-11T21:39:27.609984+0000","mapping_epoch":277,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":278,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'2","last_scrub_stamp":"2024-03-11T21:32:15.509542+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:37.512018+0000","last_clean_scrub_stamp":"2024-03-11T21:32:15.509542+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":957,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,0],"acting":[3,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1f","version":"260'24","reported_seq":651,"reported_epoch":331,"state":"active+clean","last_fresh":"2024-03-11T21:39:26.614754+0000","last_change":"2024-03-11T21:38:54.049460+0000","last_active":"2024-03-11T21:39:26.614754+0000","last_peered":"2024-03-11T21:39:26.614754+0000","last_clean":"2024-03-11T21:39:26.614754+0000","last_became_active":"2024-03-11T21:32:22.798242+0000","last_became_peered":"2024-03-11T21:32:22.798242+0000","last_unstale":"2024-03-11T21:39:26.614754+0000","last_undegraded":"2024-03-11T21:39:26.614754+0000","last_fullsized":"2024-03-11T21:39:26.614754+0000","mapping_epoch":166,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":167,"parent":"0.0","parent_split_bits":0,"last_scrub":"260'24","last_scrub_stamp":"2024-03-11T21:38:54.049368+0000","last_deep_scrub":"124'16","last_deep_scrub_stamp":"2024-03-11T21:35:38.173841+0000","last_clean_scrub_stamp":"2024-03-11T21:38:54.049368+0000","objects_scrubbed":5,"log_size":24,"log_dups_size":0,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:40:23.463882+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":32,"num_read_kb":25,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,0],"acting":[6,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.1c","version":"74'1","reported_seq":232,"reported_epoch":331,"state":"active+clean","last_fresh":"2024-03-11T21:39:26.601888+0000","last_change":"2024-03-11T21:36:11.732043+0000","last_active":"2024-03-11T21:39:26.601888+0000","last_peered":"2024-03-11T21:39:26.601888+0000","last_clean":"2024-03-11T21:39:26.601888+0000","last_became_active":"2024-03-11T21:34:41.144839+0000","last_became_peered":"2024-03-11T21:34:41.144839+0000","last_unstale":"2024-03-11T21:39:26.601888+0000","last_undegraded":"2024-03-11T21:39:26.601888+0000","last_fullsized":"2024-03-11T21:39:26.601888+0000","mapping_epoch":226,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":227,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'1","last_scrub_stamp":"2024-03-11T21:36:11.731914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:43.559452+0000","last_clean_scrub_stamp":"2024-03-11T21:36:11.731914+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":370,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":370,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.1f","version":"229'10","reported_seq":790,"reported_epoch":331,"state":"active+clean","last_fresh":"2024-03-11T21:39:26.614660+0000","last_change":"2024-03-11T21:36:08.871358+0000","last_active":"2024-03-11T21:39:26.614660+0000","last_peered":"2024-03-11T21:39:26.614660+0000","last_clean":"2024-03-11T21:39:26.614660+0000","last_became_active":"2024-03-11T21:34:41.145862+0000","last_became_peered":"2024-03-11T21:34:41.145862+0000","last_unstale":"2024-03-11T21:39:26.614660+0000","last_undegraded":"2024-03-11T21:39:26.614660+0000","last_fullsized":"2024-03-11T21:39:26.614660+0000","mapping_epoch":226,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":227,"parent":"0.0","parent_split_bits":0,"last_scrub":"229'10","last_scrub_stamp":"2024-03-11T21:36:08.871213+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:41.544550+0000","last_clean_scrub_stamp":"2024-03-11T21:36:08.871213+0000","objects_scrubbed":1,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.18","version":"67'1","reported_seq":627,"reported_epoch":331,"state":"active+clean","last_fresh":"2024-03-11T21:39:26.602426+0000","last_change":"2024-03-11T21:36:10.458150+0000","last_active":"2024-03-11T21:39:26.602426+0000","last_peered":"2024-03-11T21:39:26.602426+0000","last_clean":"2024-03-11T21:39:26.602426+0000","last_became_active":"2024-03-11T21:35:23.310241+0000","last_became_peered":"2024-03-11T21:35:23.310241+0000","last_unstale":"2024-03-11T21:39:26.602426+0000","last_undegraded":"2024-03-11T21:39:26.602426+0000","last_fullsized":"2024-03-11T21:39:26.602426+0000","mapping_epoch":248,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":249,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'1","last_scrub_stamp":"2024-03-11T21:36:10.458028+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:37.512018+0000","last_clean_scrub_stamp":"2024-03-11T21:36:10.458028+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":436,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":7,"num_read_kb":7,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":1308,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,2],"acting":[5,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1e","version":"260'20","reported_seq":671,"reported_epoch":332,"state":"active+clean","last_fresh":"2024-03-11T21:39:27.602839+0000","last_change":"2024-03-11T21:36:35.083447+0000","last_active":"2024-03-11T21:39:27.602839+0000","last_peered":"2024-03-11T21:39:27.602839+0000","last_clean":"2024-03-11T21:39:27.602839+0000","last_became_active":"2024-03-11T21:36:35.057843+0000","last_became_peered":"2024-03-11T21:36:35.057843+0000","last_unstale":"2024-03-11T21:39:27.602839+0000","last_undegraded":"2024-03-11T21:39:27.602839+0000","last_fullsized":"2024-03-11T21:39:27.602839+0000","mapping_epoch":277,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":278,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'10","last_scrub_stamp":"2024-03-11T21:35:16.825146+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:39.525554+0000","last_clean_scrub_stamp":"2024-03-11T21:35:16.825146+0000","objects_scrubbed":5,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":44,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":25,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.13","version":"0'0","reported_seq":458,"reported_epoch":332,"state":"active+clean","last_fresh":"2024-03-11T21:39:27.611258+0000","last_change":"2024-03-11T21:36:35.063744+0000","last_active":"2024-03-11T21:39:27.611258+0000","last_peered":"2024-03-11T21:39:27.611258+0000","last_clean":"2024-03-11T21:39:27.611258+0000","last_became_active":"2024-03-11T21:36:35.062618+0000","last_became_peered":"2024-03-11T21:36:35.062618+0000","last_unstale":"2024-03-11T21:39:27.611258+0000","last_undegraded":"2024-03-11T21:39:27.611258+0000","last_fullsized":"2024-03-11T21:39:27.611258+0000","mapping_epoch":277,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":278,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:34:13.822803+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:43.559452+0000","last_clean_scrub_stamp":"2024-03-11T21:34:13.822803+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":229,"reported_epoch":331,"state":"active+clean","last_fresh":"2024-03-11T21:39:26.601143+0000","last_change":"2024-03-11T21:35:16.917648+0000","last_active":"2024-03-11T21:39:26.601143+0000","last_peered":"2024-03-11T21:39:26.601143+0000","last_clean":"2024-03-11T21:39:26.601143+0000","last_became_active":"2024-03-11T21:34:41.142569+0000","last_became_peered":"2024-03-11T21:34:41.142569+0000","last_unstale":"2024-03-11T21:39:26.601143+0000","last_undegraded":"2024-03-11T21:39:26.601143+0000","last_fullsized":"2024-03-11T21:39:26.601143+0000","mapping_epoch":226,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":227,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:35:16.917532+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:41.544550+0000","last_clean_scrub_stamp":"2024-03-11T21:35:16.917532+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.17","version":"0'0","reported_seq":592,"reported_epoch":331,"state":"active+clean","last_fresh":"2024-03-11T21:39:26.614506+0000","last_change":"2024-03-11T21:38:48.084622+0000","last_active":"2024-03-11T21:39:26.614506+0000","last_peered":"2024-03-11T21:39:26.614506+0000","last_clean":"2024-03-11T21:39:26.614506+0000","last_became_active":"2024-03-11T21:25:38.540349+0000","last_became_peered":"2024-03-11T21:25:38.540349+0000","last_unstale":"2024-03-11T21:39:26.614506+0000","last_undegraded":"2024-03-11T21:39:26.614506+0000","last_fullsized":"2024-03-11T21:39:26.614506+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:38:48.084549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:28:25.918101+0000","last_clean_scrub_stamp":"2024-03-11T21:38:48.084549+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:40:17.652132+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,2],"acting":[6,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.11","version":"260'19","reported_seq":584,"reported_epoch":332,"state":"active+clean","last_fresh":"2024-03-11T21:39:27.613607+0000","last_change":"2024-03-11T21:36:35.096720+0000","last_active":"2024-03-11T21:39:27.613607+0000","last_peered":"2024-03-11T21:39:27.613607+0000","last_clean":"2024-03-11T21:39:27.613607+0000","last_became_active":"2024-03-11T21:36:35.073689+0000","last_became_peered":"2024-03-11T21:36:35.073689+0000","last_unstale":"2024-03-11T21:39:27.613607+0000","last_undegraded":"2024-03-11T21:39:27.613607+0000","last_fullsized":"2024-03-11T21:39:27.613607+0000","mapping_epoch":277,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":278,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'11","last_scrub_stamp":"2024-03-11T21:35:16.044397+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:39.525554+0000","last_clean_scrub_stamp":"2024-03-11T21:35:16.044397+0000","objects_scrubbed":5,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":31,"num_read_kb":24,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":229,"reported_epoch":331,"state":"active+clean","last_fresh":"2024-03-11T21:39:26.601307+0000","last_change":"2024-03-11T21:35:32.719869+0000","last_active":"2024-03-11T21:39:26.601307+0000","last_peered":"2024-03-11T21:39:26.601307+0000","last_clean":"2024-03-11T21:39:26.601307+0000","last_became_active":"2024-03-11T21:34:41.137235+0000","last_became_peered":"2024-03-11T21:34:41.137235+0000","last_unstale":"2024-03-11T21:39:26.601307+0000","last_undegraded":"2024-03-11T21:39:26.601307+0000","last_fullsized":"2024-03-11T21:39:26.601307+0000","mapping_epoch":226,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":227,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:35:32.719789+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:43.559452+0000","last_clean_scrub_stamp":"2024-03-11T21:35:32.719789+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,4],"acting":[7,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":604,"reported_epoch":331,"state":"active+clean","last_fresh":"2024-03-11T21:39:26.616055+0000","last_change":"2024-03-11T21:38:51.036994+0000","last_active":"2024-03-11T21:39:26.616055+0000","last_peered":"2024-03-11T21:39:26.616055+0000","last_clean":"2024-03-11T21:39:26.616055+0000","last_became_active":"2024-03-11T21:34:41.156036+0000","last_became_peered":"2024-03-11T21:34:41.156036+0000","last_unstale":"2024-03-11T21:39:26.616055+0000","last_undegraded":"2024-03-11T21:39:26.616055+0000","last_fullsized":"2024-03-11T21:39:26.616055+0000","mapping_epoch":226,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":227,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:38:51.036736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:38:51.036736+0000","last_clean_scrub_stamp":"2024-03-11T21:38:51.036736+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:40:02.840388+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.16","version":"0'0","reported_seq":591,"reported_epoch":331,"state":"active+clean","last_fresh":"2024-03-11T21:39:26.603514+0000","last_change":"2024-03-11T21:35:39.108623+0000","last_active":"2024-03-11T21:39:26.603514+0000","last_peered":"2024-03-11T21:39:26.603514+0000","last_clean":"2024-03-11T21:39:26.603514+0000","last_became_active":"2024-03-11T21:25:38.539103+0000","last_became_peered":"2024-03-11T21:25:38.539103+0000","last_unstale":"2024-03-11T21:39:26.603514+0000","last_undegraded":"2024-03-11T21:39:26.603514+0000","last_fullsized":"2024-03-11T21:39:26.603514+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:35:39.108546+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:27:11.938909+0000","last_clean_scrub_stamp":"2024-03-11T21:35:39.108546+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,2],"acting":[5,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.10","version":"260'10","reported_seq":466,"reported_epoch":332,"state":"active+clean","last_fresh":"2024-03-11T21:39:27.610181+0000","last_change":"2024-03-11T21:36:35.084268+0000","last_active":"2024-03-11T21:39:27.610181+0000","last_peered":"2024-03-11T21:39:27.610181+0000","last_clean":"2024-03-11T21:39:27.610181+0000","last_became_active":"2024-03-11T21:36:35.067500+0000","last_became_peered":"2024-03-11T21:36:35.067500+0000","last_unstale":"2024-03-11T21:39:27.610181+0000","last_undegraded":"2024-03-11T21:39:27.610181+0000","last_fullsized":"2024-03-11T21:39:27.610181+0000","mapping_epoch":277,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":278,"parent":"0.0","parent_split_bits":0,"last_scrub":"94'6","last_scrub_stamp":"2024-03-11T21:34:00.727416+0000","last_deep_scrub":"94'6","last_deep_scrub_stamp":"2024-03-11T21:28:24.603974+0000","last_clean_scrub_stamp":"2024-03-11T21:34:00.727416+0000","objects_scrubbed":2,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":12,"num_read_kb":10,"num_write":8,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6],"acting":[3,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":511,"reported_epoch":332,"state":"active+clean","last_fresh":"2024-03-11T21:39:27.613419+0000","last_change":"2024-03-11T21:36:35.070468+0000","last_active":"2024-03-11T21:39:27.613419+0000","last_peered":"2024-03-11T21:39:27.613419+0000","last_clean":"2024-03-11T21:39:27.613419+0000","last_became_active":"2024-03-11T21:36:35.070176+0000","last_became_peered":"2024-03-11T21:36:35.070176+0000","last_unstale":"2024-03-11T21:39:27.613419+0000","last_undegraded":"2024-03-11T21:39:27.613419+0000","last_fullsized":"2024-03-11T21:39:27.613419+0000","mapping_epoch":277,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":278,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:35:18.062272+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:43.559452+0000","last_clean_scrub_stamp":"2024-03-11T21:35:18.062272+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":579,"reported_epoch":332,"state":"active+clean","last_fresh":"2024-03-11T21:39:27.599872+0000","last_change":"2024-03-11T21:36:35.058551+0000","last_active":"2024-03-11T21:39:27.599872+0000","last_peered":"2024-03-11T21:39:27.599872+0000","last_clean":"2024-03-11T21:39:27.599872+0000","last_became_active":"2024-03-11T21:36:35.057906+0000","last_became_peered":"2024-03-11T21:36:35.057906+0000","last_unstale":"2024-03-11T21:39:27.599872+0000","last_undegraded":"2024-03-11T21:39:27.599872+0000","last_fullsized":"2024-03-11T21:39:27.599872+0000","mapping_epoch":277,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":278,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:34:15.516278+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:27:21.207303+0000","last_clean_scrub_stamp":"2024-03-11T21:34:15.516278+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.15","version":"0'0","reported_seq":586,"reported_epoch":332,"state":"active+clean","last_fresh":"2024-03-11T21:39:27.597532+0000","last_change":"2024-03-11T21:36:06.579754+0000","last_active":"2024-03-11T21:39:27.597532+0000","last_peered":"2024-03-11T21:39:27.597532+0000","last_clean":"2024-03-11T21:39:27.597532+0000","last_became_active":"2024-03-11T21:25:38.541705+0000","last_became_peered":"2024-03-11T21:25:38.541705+0000","last_unstale":"2024-03-11T21:39:27.597532+0000","last_undegraded":"2024-03-11T21:39:27.597532+0000","last_fullsized":"2024-03-11T21:39:27.597532+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:36:06.579636+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:26:44.860499+0000","last_clean_scrub_stamp":"2024-03-11T21:36:06.579636+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.13","version":"260'24","reported_seq":634,"reported_epoch":331,"state":"active+clean","last_fresh":"2024-03-11T21:39:26.585942+0000","last_change":"2024-03-11T21:35:14.453053+0000","last_active":"2024-03-11T21:39:26.585942+0000","last_peered":"2024-03-11T21:39:26.585942+0000","last_clean":"2024-03-11T21:39:26.585942+0000","last_became_active":"2024-03-11T21:26:11.214879+0000","last_became_peered":"2024-03-11T21:26:11.214879+0000","last_unstale":"2024-03-11T21:39:26.585942+0000","last_undegraded":"2024-03-11T21:39:26.585942+0000","last_fullsized":"2024-03-11T21:39:26.585942+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"109'16","last_scrub_stamp":"2024-03-11T21:35:14.452891+0000","last_deep_scrub":"109'16","last_deep_scrub_stamp":"2024-03-11T21:32:24.657490+0000","last_clean_scrub_stamp":"2024-03-11T21:35:14.452891+0000","objects_scrubbed":5,"log_size":24,"log_dups_size":0,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":32,"num_read_kb":25,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,2],"acting":[4,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.10","version":"0'0","reported_seq":574,"reported_epoch":332,"state":"active+clean","last_fresh":"2024-03-11T21:39:27.601777+0000","last_change":"2024-03-11T21:35:29.712658+0000","last_active":"2024-03-11T21:39:27.601777+0000","last_peered":"2024-03-11T21:39:27.601777+0000","last_clean":"2024-03-11T21:39:27.601777+0000","last_became_active":"2024-03-11T21:25:44.591189+0000","last_became_peered":"2024-03-11T21:25:44.591189+0000","last_unstale":"2024-03-11T21:39:27.601777+0000","last_undegraded":"2024-03-11T21:39:27.601777+0000","last_fullsized":"2024-03-11T21:39:27.601777+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:35:29.712557+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:27:25.417450+0000","last_clean_scrub_stamp":"2024-03-11T21:35:29.712557+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,1],"acting":[0,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":509,"reported_epoch":332,"state":"active+clean","last_fresh":"2024-03-11T21:39:27.611334+0000","last_change":"2024-03-11T21:36:35.062276+0000","last_active":"2024-03-11T21:39:27.611334+0000","last_peered":"2024-03-11T21:39:27.611334+0000","last_clean":"2024-03-11T21:39:27.611334+0000","last_became_active":"2024-03-11T21:36:35.056619+0000","last_became_peered":"2024-03-11T21:36:35.056619+0000","last_unstale":"2024-03-11T21:39:27.611334+0000","last_undegraded":"2024-03-11T21:39:27.611334+0000","last_fullsized":"2024-03-11T21:39:27.611334+0000","mapping_epoch":277,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":278,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:35:13.060487+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:31:15.367898+0000","last_clean_scrub_stamp":"2024-03-11T21:35:13.060487+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.14","version":"0'0","reported_seq":609,"reported_epoch":331,"state":"active+clean","last_fresh":"2024-03-11T21:39:26.614442+0000","last_change":"2024-03-11T21:38:35.999365+0000","last_active":"2024-03-11T21:39:26.614442+0000","last_peered":"2024-03-11T21:39:26.614442+0000","last_clean":"2024-03-11T21:39:26.614442+0000","last_became_active":"2024-03-11T21:36:35.068038+0000","last_became_peered":"2024-03-11T21:36:35.068038+0000","last_unstale":"2024-03-11T21:39:26.614442+0000","last_undegraded":"2024-03-11T21:39:26.614442+0000","last_fullsized":"2024-03-11T21:39:26.614442+0000","mapping_epoch":277,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":278,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:38:35.999231+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:37.512018+0000","last_clean_scrub_stamp":"2024-03-11T21:38:35.999231+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:40:00.642531+0000","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,5],"acting":[6,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.12","version":"260'15","reported_seq":607,"reported_epoch":332,"state":"active+clean","last_fresh":"2024-03-11T21:39:27.599721+0000","last_change":"2024-03-11T21:35:32.590439+0000","last_active":"2024-03-11T21:39:27.599721+0000","last_peered":"2024-03-11T21:39:27.599721+0000","last_clean":"2024-03-11T21:39:27.599721+0000","last_became_active":"2024-03-11T21:25:40.547082+0000","last_became_peered":"2024-03-11T21:25:40.547082+0000","last_unstale":"2024-03-11T21:39:27.599721+0000","last_undegraded":"2024-03-11T21:39:27.599721+0000","last_fullsized":"2024-03-11T21:39:27.599721+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'9","last_scrub_stamp":"2024-03-11T21:35:32.590318+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:39.525554+0000","last_clean_scrub_stamp":"2024-03-11T21:35:32.590318+0000","objects_scrubbed":4,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":572,"reported_epoch":331,"state":"active+clean","last_fresh":"2024-03-11T21:39:26.587370+0000","last_change":"2024-03-11T21:36:04.754168+0000","last_active":"2024-03-11T21:39:26.587370+0000","last_peered":"2024-03-11T21:39:26.587370+0000","last_clean":"2024-03-11T21:39:26.587370+0000","last_became_active":"2024-03-11T21:25:44.589586+0000","last_became_peered":"2024-03-11T21:25:44.589586+0000","last_unstale":"2024-03-11T21:39:26.587370+0000","last_undegraded":"2024-03-11T21:39:26.587370+0000","last_fullsized":"2024-03-11T21:39:26.587370+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:36:04.753979+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:27:09.801138+0000","last_clean_scrub_stamp":"2024-03-11T21:36:04.753979+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":113,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,5],"acting":[4,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.14","version":"280'12","reported_seq":706,"reported_epoch":332,"state":"active+clean","last_fresh":"2024-03-11T21:39:27.612063+0000","last_change":"2024-03-11T21:36:35.074550+0000","last_active":"2024-03-11T21:39:27.612063+0000","last_peered":"2024-03-11T21:39:27.612063+0000","last_clean":"2024-03-11T21:39:27.612063+0000","last_became_active":"2024-03-11T21:36:35.073912+0000","last_became_peered":"2024-03-11T21:36:35.073912+0000","last_unstale":"2024-03-11T21:39:27.612063+0000","last_undegraded":"2024-03-11T21:39:27.612063+0000","last_fullsized":"2024-03-11T21:39:27.612063+0000","mapping_epoch":277,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":278,"parent":"0.0","parent_split_bits":0,"last_scrub":"229'10","last_scrub_stamp":"2024-03-11T21:35:26.182107+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:41.544550+0000","last_clean_scrub_stamp":"2024-03-11T21:35:26.182107+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,2],"acting":[3,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.13","version":"0'0","reported_seq":587,"reported_epoch":332,"state":"active+clean","last_fresh":"2024-03-11T21:39:27.606539+0000","last_change":"2024-03-11T21:36:35.057688+0000","last_active":"2024-03-11T21:39:27.606539+0000","last_peered":"2024-03-11T21:39:27.606539+0000","last_clean":"2024-03-11T21:39:27.606539+0000","last_became_active":"2024-03-11T21:36:35.057132+0000","last_became_peered":"2024-03-11T21:36:35.057132+0000","last_unstale":"2024-03-11T21:39:27.606539+0000","last_undegraded":"2024-03-11T21:39:27.606539+0000","last_fullsized":"2024-03-11T21:39:27.606539+0000","mapping_epoch":277,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":278,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:34:08.650371+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:37.512018+0000","last_clean_scrub_stamp":"2024-03-11T21:34:08.650371+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.15","version":"260'19","reported_seq":658,"reported_epoch":331,"state":"active+clean","last_fresh":"2024-03-11T21:39:26.602909+0000","last_change":"2024-03-11T21:36:35.088986+0000","last_active":"2024-03-11T21:39:26.602909+0000","last_peered":"2024-03-11T21:39:26.602909+0000","last_clean":"2024-03-11T21:39:26.602909+0000","last_became_active":"2024-03-11T21:36:35.057285+0000","last_became_peered":"2024-03-11T21:36:35.057285+0000","last_unstale":"2024-03-11T21:39:26.602909+0000","last_undegraded":"2024-03-11T21:39:26.602909+0000","last_fullsized":"2024-03-11T21:39:26.602909+0000","mapping_epoch":277,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":278,"parent":"0.0","parent_split_bits":0,"last_scrub":"94'13","last_scrub_stamp":"2024-03-11T21:35:29.178543+0000","last_deep_scrub":"94'13","last_deep_scrub_stamp":"2024-03-11T21:29:41.853158+0000","last_clean_scrub_stamp":"2024-03-11T21:35:29.178543+0000","objects_scrubbed":4,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":11,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.16","version":"0'0","reported_seq":595,"reported_epoch":332,"state":"active+clean","last_fresh":"2024-03-11T21:39:27.602487+0000","last_change":"2024-03-11T21:36:35.058372+0000","last_active":"2024-03-11T21:39:27.602487+0000","last_peered":"2024-03-11T21:39:27.602487+0000","last_clean":"2024-03-11T21:39:27.602487+0000","last_became_active":"2024-03-11T21:36:35.057355+0000","last_became_peered":"2024-03-11T21:36:35.057355+0000","last_unstale":"2024-03-11T21:39:27.602487+0000","last_undegraded":"2024-03-11T21:39:27.602487+0000","last_fullsized":"2024-03-11T21:39:27.602487+0000","mapping_epoch":277,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":278,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:35:07.665075+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:43.559452+0000","last_clean_scrub_stamp":"2024-03-11T21:35:07.665075+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":35,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.15","version":"98'9","reported_seq":750,"reported_epoch":331,"state":"active+clean","last_fresh":"2024-03-11T21:39:26.603081+0000","last_change":"2024-03-11T21:36:03.366581+0000","last_active":"2024-03-11T21:39:26.603081+0000","last_peered":"2024-03-11T21:39:26.603081+0000","last_clean":"2024-03-11T21:39:26.603081+0000","last_became_active":"2024-03-11T21:25:42.572555+0000","last_became_peered":"2024-03-11T21:25:42.572555+0000","last_unstale":"2024-03-11T21:39:26.603081+0000","last_undegraded":"2024-03-11T21:39:26.603081+0000","last_fullsized":"2024-03-11T21:39:26.603081+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"98'9","last_scrub_stamp":"2024-03-11T21:36:03.366450+0000","last_deep_scrub":"98'9","last_deep_scrub_stamp":"2024-03-11T21:31:20.624923+0000","last_clean_scrub_stamp":"2024-03-11T21:36:03.366450+0000","objects_scrubbed":1,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.12","version":"0'0","reported_seq":604,"reported_epoch":331,"state":"active+clean","last_fresh":"2024-03-11T21:39:26.602814+0000","last_change":"2024-03-11T21:36:35.058369+0000","last_active":"2024-03-11T21:39:26.602814+0000","last_peered":"2024-03-11T21:39:26.602814+0000","last_clean":"2024-03-11T21:39:26.602814+0000","last_became_active":"2024-03-11T21:36:35.058033+0000","last_became_peered":"2024-03-11T21:36:35.058033+0000","last_unstale":"2024-03-11T21:39:26.602814+0000","last_undegraded":"2024-03-11T21:39:26.602814+0000","last_fullsized":"2024-03-11T21:39:26.602814+0000","mapping_epoch":277,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":278,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:35:27.158468+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:37.512018+0000","last_clean_scrub_stamp":"2024-03-11T21:35:27.158468+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.14","version":"260'20","reported_seq":592,"reported_epoch":332,"state":"active+clean","last_fresh":"2024-03-11T21:39:27.611656+0000","last_change":"2024-03-11T21:36:35.095382+0000","last_active":"2024-03-11T21:39:27.611656+0000","last_peered":"2024-03-11T21:39:27.611656+0000","last_clean":"2024-03-11T21:39:27.611656+0000","last_became_active":"2024-03-11T21:36:35.069638+0000","last_became_peered":"2024-03-11T21:36:35.069638+0000","last_unstale":"2024-03-11T21:39:27.611656+0000","last_undegraded":"2024-03-11T21:39:27.611656+0000","last_fullsized":"2024-03-11T21:39:27.611656+0000","mapping_epoch":277,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":278,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'10","last_scrub_stamp":"2024-03-11T21:35:28.189296+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:39.525554+0000","last_clean_scrub_stamp":"2024-03-11T21:35:28.189296+0000","objects_scrubbed":0,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":29,"num_read_kb":24,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.15","version":"0'0","reported_seq":228,"reported_epoch":331,"state":"active+clean","last_fresh":"2024-03-11T21:39:26.602372+0000","last_change":"2024-03-11T21:36:14.736240+0000","last_active":"2024-03-11T21:39:26.602372+0000","last_peered":"2024-03-11T21:39:26.602372+0000","last_clean":"2024-03-11T21:39:26.602372+0000","last_became_active":"2024-03-11T21:34:41.142760+0000","last_became_peered":"2024-03-11T21:34:41.142760+0000","last_unstale":"2024-03-11T21:39:26.602372+0000","last_undegraded":"2024-03-11T21:39:26.602372+0000","last_fullsized":"2024-03-11T21:39:26.602372+0000","mapping_epoch":226,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":227,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:36:14.735985+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:43.559452+0000","last_clean_scrub_stamp":"2024-03-11T21:36:14.735985+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":36,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":576,"reported_epoch":331,"state":"active+clean","last_fresh":"2024-03-11T21:39:26.603491+0000","last_change":"2024-03-11T21:36:35.060666+0000","last_active":"2024-03-11T21:39:26.603491+0000","last_peered":"2024-03-11T21:39:26.603491+0000","last_clean":"2024-03-11T21:39:26.603491+0000","last_became_active":"2024-03-11T21:36:35.060342+0000","last_became_peered":"2024-03-11T21:36:35.060342+0000","last_unstale":"2024-03-11T21:39:26.603491+0000","last_undegraded":"2024-03-11T21:39:26.603491+0000","last_fullsized":"2024-03-11T21:39:26.603491+0000","mapping_epoch":277,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":278,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:34:09.244852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:41.544550+0000","last_clean_scrub_stamp":"2024-03-11T21:34:09.244852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,1],"acting":[5,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.11","version":"0'0","reported_seq":592,"reported_epoch":331,"state":"active+clean","last_fresh":"2024-03-11T21:39:26.616116+0000","last_change":"2024-03-11T21:38:49.044802+0000","last_active":"2024-03-11T21:39:26.616116+0000","last_peered":"2024-03-11T21:39:26.616116+0000","last_clean":"2024-03-11T21:39:26.616116+0000","last_became_active":"2024-03-11T21:25:38.537415+0000","last_became_peered":"2024-03-11T21:25:38.537415+0000","last_unstale":"2024-03-11T21:39:26.616116+0000","last_undegraded":"2024-03-11T21:39:26.616116+0000","last_fullsized":"2024-03-11T21:39:26.616116+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:38:49.044722+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:37.512018+0000","last_clean_scrub_stamp":"2024-03-11T21:38:49.044722+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:40:07.269648+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.17","version":"260'15","reported_seq":537,"reported_epoch":332,"state":"active+clean","last_fresh":"2024-03-11T21:39:27.611490+0000","last_change":"2024-03-11T21:36:35.099451+0000","last_active":"2024-03-11T21:39:27.611490+0000","last_peered":"2024-03-11T21:39:27.611490+0000","last_clean":"2024-03-11T21:39:27.611490+0000","last_became_active":"2024-03-11T21:36:35.071346+0000","last_became_peered":"2024-03-11T21:36:35.071346+0000","last_unstale":"2024-03-11T21:39:27.611490+0000","last_undegraded":"2024-03-11T21:39:27.611490+0000","last_fullsized":"2024-03-11T21:39:27.611490+0000","mapping_epoch":277,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":278,"parent":"0.0","parent_split_bits":0,"last_scrub":"97'9","last_scrub_stamp":"2024-03-11T21:32:08.495352+0000","last_deep_scrub":"97'9","last_deep_scrub_stamp":"2024-03-11T21:27:25.803132+0000","last_clean_scrub_stamp":"2024-03-11T21:32:08.495352+0000","objects_scrubbed":3,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":17,"num_read_kb":14,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,5],"acting":[3,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.14","version":"0'0","reported_seq":583,"reported_epoch":331,"state":"active+clean","last_fresh":"2024-03-11T21:39:26.625175+0000","last_change":"2024-03-11T21:35:27.606664+0000","last_active":"2024-03-11T21:39:26.625175+0000","last_peered":"2024-03-11T21:39:26.625175+0000","last_clean":"2024-03-11T21:39:26.625175+0000","last_became_active":"2024-03-11T21:34:41.136946+0000","last_became_peered":"2024-03-11T21:34:41.136946+0000","last_unstale":"2024-03-11T21:39:26.625175+0000","last_undegraded":"2024-03-11T21:39:26.625175+0000","last_fullsized":"2024-03-11T21:39:26.625175+0000","mapping_epoch":226,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":227,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:35:27.606551+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:43.559452+0000","last_clean_scrub_stamp":"2024-03-11T21:35:27.606551+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,7],"acting":[2,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":491,"reported_epoch":332,"state":"active+clean","last_fresh":"2024-03-11T21:39:27.610601+0000","last_change":"2024-03-11T21:36:35.076782+0000","last_active":"2024-03-11T21:39:27.610601+0000","last_peered":"2024-03-11T21:39:27.610601+0000","last_clean":"2024-03-11T21:39:27.610601+0000","last_became_active":"2024-03-11T21:36:35.076335+0000","last_became_peered":"2024-03-11T21:36:35.076335+0000","last_unstale":"2024-03-11T21:39:27.610601+0000","last_undegraded":"2024-03-11T21:39:27.610601+0000","last_fullsized":"2024-03-11T21:39:27.610601+0000","mapping_epoch":277,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":278,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:34:33.082577+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:41.544550+0000","last_clean_scrub_stamp":"2024-03-11T21:34:33.082577+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":35,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.10","version":"0'0","reported_seq":583,"reported_epoch":331,"state":"active+clean","last_fresh":"2024-03-11T21:39:26.624659+0000","last_change":"2024-03-11T21:35:37.694623+0000","last_active":"2024-03-11T21:39:26.624659+0000","last_peered":"2024-03-11T21:39:26.624659+0000","last_clean":"2024-03-11T21:39:26.624659+0000","last_became_active":"2024-03-11T21:25:38.533138+0000","last_became_peered":"2024-03-11T21:25:38.533138+0000","last_unstale":"2024-03-11T21:39:26.624659+0000","last_undegraded":"2024-03-11T21:39:26.624659+0000","last_fullsized":"2024-03-11T21:39:26.624659+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:35:37.694533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:37.512018+0000","last_clean_scrub_stamp":"2024-03-11T21:35:37.694533+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,0],"acting":[2,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.16","version":"260'15","reported_seq":657,"reported_epoch":332,"state":"active+clean","last_fresh":"2024-03-11T21:39:27.602661+0000","last_change":"2024-03-11T21:36:35.080158+0000","last_active":"2024-03-11T21:39:27.602661+0000","last_peered":"2024-03-11T21:39:27.602661+0000","last_clean":"2024-03-11T21:39:27.602661+0000","last_became_active":"2024-03-11T21:36:35.056862+0000","last_became_peered":"2024-03-11T21:36:35.056862+0000","last_unstale":"2024-03-11T21:39:27.602661+0000","last_undegraded":"2024-03-11T21:39:27.602661+0000","last_fullsized":"2024-03-11T21:39:27.602661+0000","mapping_epoch":277,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":278,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'9","last_scrub_stamp":"2024-03-11T21:35:12.741225+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:39.525554+0000","last_clean_scrub_stamp":"2024-03-11T21:35:12.741225+0000","objects_scrubbed":4,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":11,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.4","version":"332'13361","reported_seq":17197,"reported_epoch":332,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-03-11T21:39:28.233674+0000","last_change":"2024-03-11T21:38:21.285644+0000","last_active":"2024-03-11T21:39:28.233674+0000","last_peered":"2024-03-11T21:39:28.233674+0000","last_clean":"2024-03-11T21:35:44.669493+0000","last_became_active":"2024-03-11T21:36:36.506468+0000","last_became_peered":"2024-03-11T21:36:36.506468+0000","last_unstale":"2024-03-11T21:39:28.233674+0000","last_undegraded":"2024-03-11T21:36:36.069183+0000","last_fullsized":"2024-03-11T21:36:36.038484+0000","mapping_epoch":278,"log_start":"302'8300","ondisk_log_start":"302'8300","created":259,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:35:44.669493+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:35:44.669493+0000","last_clean_scrub_stamp":"2024-03-11T21:35:44.669493+0000","objects_scrubbed":0,"log_size":5061,"log_dups_size":0,"ondisk_log_size":5061,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":91422720,"num_objects":1395,"num_object_clones":0,"num_object_copies":4185,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2977,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1395,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13361,"num_write_kb":472192,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1839,"num_bytes_recovered":78577664,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,3],"acting":[1,4],"avail_no_missing":["1","4"],"object_location_counts":[{"shards":"1,4","objects":-1582}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.b","version":"0'0","reported_seq":479,"reported_epoch":332,"state":"active+clean","last_fresh":"2024-03-11T21:39:27.611135+0000","last_change":"2024-03-11T21:36:35.074815+0000","last_active":"2024-03-11T21:39:27.611135+0000","last_peered":"2024-03-11T21:39:27.611135+0000","last_clean":"2024-03-11T21:39:27.611135+0000","last_became_active":"2024-03-11T21:36:35.072147+0000","last_became_peered":"2024-03-11T21:36:35.072147+0000","last_unstale":"2024-03-11T21:39:27.611135+0000","last_undegraded":"2024-03-11T21:39:27.611135+0000","last_fullsized":"2024-03-11T21:39:27.611135+0000","mapping_epoch":277,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":278,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:32:22.598150+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:43.559452+0000","last_clean_scrub_stamp":"2024-03-11T21:32:22.598150+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":57,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.f","version":"67'2","reported_seq":597,"reported_epoch":331,"state":"active+clean","last_fresh":"2024-03-11T21:39:26.587175+0000","last_change":"2024-03-11T21:35:36.613667+0000","last_active":"2024-03-11T21:39:26.587175+0000","last_peered":"2024-03-11T21:39:26.587175+0000","last_clean":"2024-03-11T21:39:26.587175+0000","last_became_active":"2024-03-11T21:25:38.537711+0000","last_became_peered":"2024-03-11T21:25:38.537711+0000","last_unstale":"2024-03-11T21:39:26.587175+0000","last_undegraded":"2024-03-11T21:39:26.587175+0000","last_fullsized":"2024-03-11T21:39:26.587175+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'2","last_scrub_stamp":"2024-03-11T21:35:36.613516+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:37.512018+0000","last_clean_scrub_stamp":"2024-03-11T21:35:36.613516+0000","objects_scrubbed":2,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":92,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":10,"num_read_kb":10,"num_write":4,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,7],"acting":[4,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.9","version":"260'23","reported_seq":628,"reported_epoch":331,"state":"active+clean","last_fresh":"2024-03-11T21:39:26.585199+0000","last_change":"2024-03-11T21:36:35.089034+0000","last_active":"2024-03-11T21:39:26.585199+0000","last_peered":"2024-03-11T21:39:26.585199+0000","last_clean":"2024-03-11T21:39:26.585199+0000","last_became_active":"2024-03-11T21:36:35.059930+0000","last_became_peered":"2024-03-11T21:36:35.059930+0000","last_unstale":"2024-03-11T21:39:26.585199+0000","last_undegraded":"2024-03-11T21:39:26.585199+0000","last_fullsized":"2024-03-11T21:39:26.585199+0000","mapping_epoch":277,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":278,"parent":"0.0","parent_split_bits":0,"last_scrub":"146'17","last_scrub_stamp":"2024-03-11T21:34:26.765075+0000","last_deep_scrub":"146'17","last_deep_scrub_stamp":"2024-03-11T21:34:26.765075+0000","last_clean_scrub_stamp":"2024-03-11T21:34:26.765075+0000","objects_scrubbed":5,"log_size":23,"log_dups_size":0,"ondisk_log_size":23,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":34,"num_read_kb":25,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,3],"acting":[4,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":576,"reported_epoch":331,"state":"active+clean","last_fresh":"2024-03-11T21:39:26.624758+0000","last_change":"2024-03-11T21:35:16.643421+0000","last_active":"2024-03-11T21:39:26.624758+0000","last_peered":"2024-03-11T21:39:26.624758+0000","last_clean":"2024-03-11T21:39:26.624758+0000","last_became_active":"2024-03-11T21:25:42.573987+0000","last_became_peered":"2024-03-11T21:25:42.573987+0000","last_unstale":"2024-03-11T21:39:26.624758+0000","last_undegraded":"2024-03-11T21:39:26.624758+0000","last_fullsized":"2024-03-11T21:39:26.624758+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:35:16.643311+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:41.544550+0000","last_clean_scrub_stamp":"2024-03-11T21:35:16.643311+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"9.f","version":"331'13011","reported_seq":13168,"reported_epoch":331,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-03-11T21:39:27.469030+0000","last_change":"2024-03-11T21:36:36.519822+0000","last_active":"2024-03-11T21:39:27.469030+0000","last_peered":"2024-03-11T21:39:27.469030+0000","last_clean":"2024-03-11T21:35:44.669493+0000","last_became_active":"2024-03-11T21:36:36.508836+0000","last_became_peered":"2024-03-11T21:36:36.508836+0000","last_unstale":"2024-03-11T21:39:27.469030+0000","last_undegraded":"2024-03-11T21:36:36.176770+0000","last_fullsized":"2024-03-11T21:36:36.038651+0000","mapping_epoch":278,"log_start":"301'7800","ondisk_log_start":"301'7800","created":259,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:35:44.669493+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:35:44.669493+0000","last_clean_scrub_stamp":"2024-03-11T21:35:44.669493+0000","objects_scrubbed":0,"log_size":5211,"log_dups_size":0,"ondisk_log_size":5211,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":89194496,"num_objects":1361,"num_object_clones":0,"num_object_copies":4083,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4738,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1361,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13011,"num_write_kb":459904,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":65536,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3],"acting":[5,1],"avail_no_missing":["5","1"],"object_location_counts":[{"shards":"1,5","objects":-3377}],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.0","version":"0'0","reported_seq":576,"reported_epoch":332,"state":"active+clean","last_fresh":"2024-03-11T21:39:27.601902+0000","last_change":"2024-03-11T21:36:35.058390+0000","last_active":"2024-03-11T21:39:27.601902+0000","last_peered":"2024-03-11T21:39:27.601902+0000","last_clean":"2024-03-11T21:39:27.601902+0000","last_became_active":"2024-03-11T21:36:35.058054+0000","last_became_peered":"2024-03-11T21:36:35.058054+0000","last_unstale":"2024-03-11T21:39:27.601902+0000","last_undegraded":"2024-03-11T21:39:27.601902+0000","last_fullsized":"2024-03-11T21:39:27.601902+0000","mapping_epoch":277,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":278,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:35:04.626668+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:29:39.498784+0000","last_clean_scrub_stamp":"2024-03-11T21:35:04.626668+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":39,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":599,"reported_epoch":332,"state":"active+clean","last_fresh":"2024-03-11T21:39:27.603288+0000","last_change":"2024-03-11T21:35:33.608063+0000","last_active":"2024-03-11T21:39:27.603288+0000","last_peered":"2024-03-11T21:39:27.603288+0000","last_clean":"2024-03-11T21:39:27.603288+0000","last_became_active":"2024-03-11T21:34:41.150913+0000","last_became_peered":"2024-03-11T21:34:41.150913+0000","last_unstale":"2024-03-11T21:39:27.603288+0000","last_undegraded":"2024-03-11T21:39:27.603288+0000","last_fullsized":"2024-03-11T21:39:27.603288+0000","mapping_epoch":226,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":227,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:35:33.607969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:37.512018+0000","last_clean_scrub_stamp":"2024-03-11T21:35:33.607969+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,7],"acting":[1,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.2","version":"260'25","reported_seq":618,"reported_epoch":332,"state":"active+clean","last_fresh":"2024-03-11T21:39:27.599795+0000","last_change":"2024-03-11T21:35:17.666749+0000","last_active":"2024-03-11T21:39:27.599795+0000","last_peered":"2024-03-11T21:39:27.599795+0000","last_clean":"2024-03-11T21:39:27.599795+0000","last_became_active":"2024-03-11T21:25:40.548833+0000","last_became_peered":"2024-03-11T21:25:40.548833+0000","last_unstale":"2024-03-11T21:39:27.599795+0000","last_undegraded":"2024-03-11T21:39:27.599795+0000","last_fullsized":"2024-03-11T21:39:27.599795+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"94'15","last_scrub_stamp":"2024-03-11T21:35:17.666498+0000","last_deep_scrub":"94'15","last_deep_scrub_stamp":"2024-03-11T21:26:49.993060+0000","last_clean_scrub_stamp":"2024-03-11T21:35:17.666498+0000","objects_scrubbed":5,"log_size":25,"log_dups_size":0,"ondisk_log_size":25,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":20,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":25,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.3","version":"74'8","reported_seq":753,"reported_epoch":332,"state":"active+clean","last_fresh":"2024-03-11T21:39:27.606874+0000","last_change":"2024-03-11T21:35:31.695369+0000","last_active":"2024-03-11T21:39:27.606874+0000","last_peered":"2024-03-11T21:39:27.606874+0000","last_clean":"2024-03-11T21:39:27.606874+0000","last_became_active":"2024-03-11T21:25:42.578161+0000","last_became_peered":"2024-03-11T21:25:42.578161+0000","last_unstale":"2024-03-11T21:39:27.606874+0000","last_undegraded":"2024-03-11T21:39:27.606874+0000","last_fullsized":"2024-03-11T21:39:27.606874+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'8","last_scrub_stamp":"2024-03-11T21:35:31.695263+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:41.544550+0000","last_clean_scrub_stamp":"2024-03-11T21:35:31.695263+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,5],"acting":[0,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.9","version":"332'13441","reported_seq":13598,"reported_epoch":332,"state":"active+clean","last_fresh":"2024-03-11T21:39:28.713922+0000","last_change":"2024-03-11T21:35:45.696767+0000","last_active":"2024-03-11T21:39:28.713922+0000","last_peered":"2024-03-11T21:39:28.713922+0000","last_clean":"2024-03-11T21:39:28.713922+0000","last_became_active":"2024-03-11T21:35:45.696379+0000","last_became_peered":"2024-03-11T21:35:45.696379+0000","last_unstale":"2024-03-11T21:39:28.713922+0000","last_undegraded":"2024-03-11T21:39:28.713922+0000","last_fullsized":"2024-03-11T21:39:28.713922+0000","mapping_epoch":259,"log_start":"298'7800","ondisk_log_start":"298'7800","created":259,"last_epoch_clean":260,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:35:44.669493+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:35:44.669493+0000","last_clean_scrub_stamp":"2024-03-11T21:35:44.669493+0000","objects_scrubbed":0,"log_size":5641,"log_dups_size":0,"ondisk_log_size":5641,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":88408064,"num_objects":1349,"num_object_clones":0,"num_object_copies":4047,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1349,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13441,"num_write_kb":473280,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,5],"acting":[0,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":517,"reported_epoch":332,"state":"active+clean","last_fresh":"2024-03-11T21:39:27.610218+0000","last_change":"2024-03-11T21:36:35.068977+0000","last_active":"2024-03-11T21:39:27.610218+0000","last_peered":"2024-03-11T21:39:27.610218+0000","last_clean":"2024-03-11T21:39:27.610218+0000","last_became_active":"2024-03-11T21:36:35.067288+0000","last_became_peered":"2024-03-11T21:36:35.067288+0000","last_unstale":"2024-03-11T21:39:27.610218+0000","last_undegraded":"2024-03-11T21:39:27.610218+0000","last_fullsized":"2024-03-11T21:39:27.610218+0000","mapping_epoch":277,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":278,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:35:09.985377+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:28:30.563712+0000","last_clean_scrub_stamp":"2024-03-11T21:35:09.985377+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,7],"acting":[3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":592,"reported_epoch":331,"state":"active+clean","last_fresh":"2024-03-11T21:39:26.602905+0000","last_change":"2024-03-11T21:35:40.116979+0000","last_active":"2024-03-11T21:39:26.602905+0000","last_peered":"2024-03-11T21:39:26.602905+0000","last_clean":"2024-03-11T21:39:26.602905+0000","last_became_active":"2024-03-11T21:25:38.538540+0000","last_became_peered":"2024-03-11T21:25:38.538540+0000","last_unstale":"2024-03-11T21:39:26.602905+0000","last_undegraded":"2024-03-11T21:39:26.602905+0000","last_fullsized":"2024-03-11T21:39:26.602905+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:35:40.116877+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:37.512018+0000","last_clean_scrub_stamp":"2024-03-11T21:35:40.116877+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.4","version":"280'42","reported_seq":832,"reported_epoch":332,"state":"active+clean","last_fresh":"2024-03-11T21:39:27.603193+0000","last_change":"2024-03-11T21:36:35.095935+0000","last_active":"2024-03-11T21:39:27.603193+0000","last_peered":"2024-03-11T21:39:27.603193+0000","last_clean":"2024-03-11T21:39:27.603193+0000","last_became_active":"2024-03-11T21:36:35.073430+0000","last_became_peered":"2024-03-11T21:36:35.073430+0000","last_unstale":"2024-03-11T21:39:27.603193+0000","last_undegraded":"2024-03-11T21:39:27.603193+0000","last_fullsized":"2024-03-11T21:39:27.603193+0000","mapping_epoch":277,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":278,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'28","last_scrub_stamp":"2024-03-11T21:34:19.513252+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:39.525554+0000","last_clean_scrub_stamp":"2024-03-11T21:34:19.513252+0000","objects_scrubbed":10,"log_size":42,"log_dups_size":0,"ondisk_log_size":42,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":358,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":66,"num_read_kb":51,"num_write":38,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3],"acting":[1,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":579,"reported_epoch":332,"state":"active+clean","last_fresh":"2024-03-11T21:39:27.609137+0000","last_change":"2024-03-11T21:35:32.651731+0000","last_active":"2024-03-11T21:39:27.609137+0000","last_peered":"2024-03-11T21:39:27.609137+0000","last_clean":"2024-03-11T21:39:27.609137+0000","last_became_active":"2024-03-11T21:25:42.572673+0000","last_became_peered":"2024-03-11T21:25:42.572673+0000","last_unstale":"2024-03-11T21:39:27.609137+0000","last_undegraded":"2024-03-11T21:39:27.609137+0000","last_fullsized":"2024-03-11T21:39:27.609137+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:35:32.651652+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:41.544550+0000","last_clean_scrub_stamp":"2024-03-11T21:35:32.651652+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.a","version":"331'13385","reported_seq":13539,"reported_epoch":331,"state":"active+clean","last_fresh":"2024-03-11T21:39:27.463931+0000","last_change":"2024-03-11T21:35:45.698558+0000","last_active":"2024-03-11T21:39:27.463931+0000","last_peered":"2024-03-11T21:39:27.463931+0000","last_clean":"2024-03-11T21:39:27.463931+0000","last_became_active":"2024-03-11T21:35:45.698238+0000","last_became_peered":"2024-03-11T21:35:45.698238+0000","last_unstale":"2024-03-11T21:39:27.463931+0000","last_undegraded":"2024-03-11T21:39:27.463931+0000","last_fullsized":"2024-03-11T21:39:27.463931+0000","mapping_epoch":259,"log_start":"302'8200","ondisk_log_start":"302'8200","created":259,"last_epoch_clean":260,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:35:44.669493+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:35:44.669493+0000","last_clean_scrub_stamp":"2024-03-11T21:35:44.669493+0000","objects_scrubbed":0,"log_size":5185,"log_dups_size":0,"ondisk_log_size":5185,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":91947008,"num_objects":1403,"num_object_clones":0,"num_object_copies":4209,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1403,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13385,"num_write_kb":473216,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,7],"acting":[5,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.5","version":"0'0","reported_seq":238,"reported_epoch":331,"state":"active+clean","last_fresh":"2024-03-11T21:39:26.602028+0000","last_change":"2024-03-11T21:36:35.069989+0000","last_active":"2024-03-11T21:39:26.602028+0000","last_peered":"2024-03-11T21:39:26.602028+0000","last_clean":"2024-03-11T21:39:26.602028+0000","last_became_active":"2024-03-11T21:36:35.069358+0000","last_became_peered":"2024-03-11T21:36:35.069358+0000","last_unstale":"2024-03-11T21:39:26.602028+0000","last_undegraded":"2024-03-11T21:39:26.602028+0000","last_fullsized":"2024-03-11T21:39:26.602028+0000","mapping_epoch":277,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":278,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:35:19.841632+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:43.559452+0000","last_clean_scrub_stamp":"2024-03-11T21:35:19.841632+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,3],"acting":[7,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.0","version":"164'6","reported_seq":1420,"reported_epoch":332,"state":"active+clean","last_fresh":"2024-03-11T21:39:27.598379+0000","last_change":"2024-03-11T21:35:38.518243+0000","last_active":"2024-03-11T21:39:27.598379+0000","last_peered":"2024-03-11T21:39:27.598379+0000","last_clean":"2024-03-11T21:39:27.598379+0000","last_became_active":"2024-03-11T21:25:38.536890+0000","last_became_peered":"2024-03-11T21:25:38.536890+0000","last_unstale":"2024-03-11T21:39:27.598379+0000","last_undegraded":"2024-03-11T21:39:27.598379+0000","last_fullsized":"2024-03-11T21:39:27.598379+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"164'6","last_scrub_stamp":"2024-03-11T21:35:38.518097+0000","last_deep_scrub":"164'6","last_deep_scrub_stamp":"2024-03-11T21:32:17.597666+0000","last_clean_scrub_stamp":"2024-03-11T21:35:38.518097+0000","objects_scrubbed":1,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0.000753559,"stat_sum":{"num_bytes":389,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":822,"num_read_kb":817,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,6],"acting":[1,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":593,"reported_epoch":331,"state":"active+clean","last_fresh":"2024-03-11T21:39:26.624716+0000","last_change":"2024-03-11T21:36:35.057287+0000","last_active":"2024-03-11T21:39:26.624716+0000","last_peered":"2024-03-11T21:39:26.624716+0000","last_clean":"2024-03-11T21:39:26.624716+0000","last_became_active":"2024-03-11T21:36:35.056966+0000","last_became_peered":"2024-03-11T21:36:35.056966+0000","last_unstale":"2024-03-11T21:39:26.624716+0000","last_undegraded":"2024-03-11T21:39:26.624716+0000","last_fullsized":"2024-03-11T21:39:26.624716+0000","mapping_epoch":277,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":278,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:34:18.424010+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:26:48.997454+0000","last_clean_scrub_stamp":"2024-03-11T21:34:18.424010+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.7","version":"260'22","reported_seq":624,"reported_epoch":332,"state":"active+clean","last_fresh":"2024-03-11T21:39:27.597340+0000","last_change":"2024-03-11T21:35:37.508679+0000","last_active":"2024-03-11T21:39:27.597340+0000","last_peered":"2024-03-11T21:39:27.597340+0000","last_clean":"2024-03-11T21:39:27.597340+0000","last_became_active":"2024-03-11T21:25:40.541769+0000","last_became_peered":"2024-03-11T21:25:40.541769+0000","last_unstale":"2024-03-11T21:39:27.597340+0000","last_undegraded":"2024-03-11T21:39:27.597340+0000","last_fullsized":"2024-03-11T21:39:27.597340+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"97'18","last_scrub_stamp":"2024-03-11T21:35:37.508567+0000","last_deep_scrub":"97'18","last_deep_scrub_stamp":"2024-03-11T21:27:19.239404+0000","last_clean_scrub_stamp":"2024-03-11T21:35:37.508567+0000","objects_scrubbed":5,"log_size":22,"log_dups_size":0,"ondisk_log_size":22,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":36,"num_read_kb":25,"num_write":20,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":588,"reported_epoch":331,"state":"active+clean","last_fresh":"2024-03-11T21:39:26.628395+0000","last_change":"2024-03-11T21:35:22.632822+0000","last_active":"2024-03-11T21:39:26.628395+0000","last_peered":"2024-03-11T21:39:26.628395+0000","last_clean":"2024-03-11T21:39:26.628395+0000","last_became_active":"2024-03-11T21:34:41.143660+0000","last_became_peered":"2024-03-11T21:34:41.143660+0000","last_unstale":"2024-03-11T21:39:26.628395+0000","last_undegraded":"2024-03-11T21:39:26.628395+0000","last_fullsized":"2024-03-11T21:39:26.628395+0000","mapping_epoch":226,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":227,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:35:22.632707+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:41.544550+0000","last_clean_scrub_stamp":"2024-03-11T21:35:22.632707+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":18,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7],"acting":[2,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"9.8","version":"332'13427","reported_seq":15144,"reported_epoch":332,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-03-11T21:39:28.740462+0000","last_change":"2024-03-11T21:38:57.479443+0000","last_active":"2024-03-11T21:39:28.740462+0000","last_peered":"2024-03-11T21:39:28.740462+0000","last_clean":"2024-03-11T21:35:44.669493+0000","last_became_active":"2024-03-11T21:36:36.507652+0000","last_became_peered":"2024-03-11T21:36:36.507652+0000","last_unstale":"2024-03-11T21:39:28.740462+0000","last_undegraded":"2024-03-11T21:36:36.064628+0000","last_fullsized":"2024-03-11T21:36:36.036628+0000","mapping_epoch":278,"log_start":"298'7800","ondisk_log_start":"298'7800","created":259,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:35:44.669493+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:35:44.669493+0000","last_clean_scrub_stamp":"2024-03-11T21:35:44.669493+0000","objects_scrubbed":0,"log_size":5627,"log_dups_size":0,"ondisk_log_size":5627,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":86704128,"num_objects":1323,"num_object_clones":0,"num_object_copies":3969,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4066,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1323,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13427,"num_write_kb":472000,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":780,"num_bytes_recovered":14483456,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[0,5],"avail_no_missing":["0","5"],"object_location_counts":[{"shards":"0,5","objects":-2743}],"blocked_by":[],"up_primary":3,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.7","version":"124'2","reported_seq":575,"reported_epoch":331,"state":"active+clean","last_fresh":"2024-03-11T21:39:26.608442+0000","last_change":"2024-03-11T21:36:35.059190+0000","last_active":"2024-03-11T21:39:26.608442+0000","last_peered":"2024-03-11T21:39:26.608442+0000","last_clean":"2024-03-11T21:39:26.608442+0000","last_became_active":"2024-03-11T21:36:35.058786+0000","last_became_peered":"2024-03-11T21:36:35.058786+0000","last_unstale":"2024-03-11T21:39:26.608442+0000","last_undegraded":"2024-03-11T21:39:26.608442+0000","last_fullsized":"2024-03-11T21:39:26.608442+0000","mapping_epoch":277,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":278,"parent":"0.0","parent_split_bits":0,"last_scrub":"124'2","last_scrub_stamp":"2024-03-11T21:35:16.414595+0000","last_deep_scrub":"124'2","last_deep_scrub_stamp":"2024-03-11T21:29:34.956908+0000","last_clean_scrub_stamp":"2024-03-11T21:35:16.414595+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":42,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,4],"acting":[5,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2","version":"71'2","reported_seq":538,"reported_epoch":332,"state":"active+clean","last_fresh":"2024-03-11T21:39:27.609771+0000","last_change":"2024-03-11T21:36:35.071218+0000","last_active":"2024-03-11T21:39:27.609771+0000","last_peered":"2024-03-11T21:39:27.609771+0000","last_clean":"2024-03-11T21:39:27.609771+0000","last_became_active":"2024-03-11T21:36:35.070783+0000","last_became_peered":"2024-03-11T21:36:35.070783+0000","last_unstale":"2024-03-11T21:39:27.609771+0000","last_undegraded":"2024-03-11T21:39:27.609771+0000","last_fullsized":"2024-03-11T21:39:27.609771+0000","mapping_epoch":277,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":278,"parent":"0.0","parent_split_bits":0,"last_scrub":"71'2","last_scrub_stamp":"2024-03-11T21:32:21.543053+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:37.512018+0000","last_clean_scrub_stamp":"2024-03-11T21:32:21.543053+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014206449999999999,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":602,"reported_epoch":331,"state":"active+clean","last_fresh":"2024-03-11T21:39:26.603255+0000","last_change":"2024-03-11T21:35:08.391825+0000","last_active":"2024-03-11T21:39:26.603255+0000","last_peered":"2024-03-11T21:39:26.603255+0000","last_clean":"2024-03-11T21:39:26.603255+0000","last_became_active":"2024-03-11T21:34:41.146672+0000","last_became_peered":"2024-03-11T21:34:41.146672+0000","last_unstale":"2024-03-11T21:39:26.603255+0000","last_undegraded":"2024-03-11T21:39:26.603255+0000","last_fullsized":"2024-03-11T21:39:26.603255+0000","mapping_epoch":226,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":227,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:35:08.391713+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:37.512018+0000","last_clean_scrub_stamp":"2024-03-11T21:35:08.391713+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"87'128","reported_seq":700,"reported_epoch":331,"state":"active+clean","last_fresh":"2024-03-11T21:39:26.600977+0000","last_change":"2024-03-11T21:35:21.798272+0000","last_active":"2024-03-11T21:39:26.600977+0000","last_peered":"2024-03-11T21:39:26.600977+0000","last_clean":"2024-03-11T21:39:26.600977+0000","last_became_active":"2024-03-11T21:34:44.164569+0000","last_became_peered":"2024-03-11T21:34:44.164569+0000","last_unstale":"2024-03-11T21:39:26.600977+0000","last_undegraded":"2024-03-11T21:39:26.600977+0000","last_fullsized":"2024-03-11T21:39:26.600977+0000","mapping_epoch":229,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":230,"parent":"0.0","parent_split_bits":0,"last_scrub":"87'128","last_scrub_stamp":"2024-03-11T21:35:21.798144+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:24.261974+0000","last_clean_scrub_stamp":"2024-03-11T21:35:21.798144+0000","objects_scrubbed":2,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":198,"num_read_kb":294,"num_write":179,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":2951840,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.5","version":"260'28","reported_seq":795,"reported_epoch":331,"state":"active+clean","last_fresh":"2024-03-11T21:39:26.614618+0000","last_change":"2024-03-11T21:36:14.026933+0000","last_active":"2024-03-11T21:39:26.614618+0000","last_peered":"2024-03-11T21:39:26.614618+0000","last_clean":"2024-03-11T21:39:26.614618+0000","last_became_active":"2024-03-11T21:25:40.761457+0000","last_became_peered":"2024-03-11T21:25:40.761457+0000","last_unstale":"2024-03-11T21:39:26.614618+0000","last_undegraded":"2024-03-11T21:39:26.614618+0000","last_fullsized":"2024-03-11T21:39:26.614618+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"260'28","last_scrub_stamp":"2024-03-11T21:36:14.026775+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:39.525554+0000","last_clean_scrub_stamp":"2024-03-11T21:36:14.026775+0000","objects_scrubbed":8,"log_size":28,"log_dups_size":0,"ondisk_log_size":28,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":154,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":43,"num_read_kb":33,"num_write":25,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":229,"reported_epoch":331,"state":"active+clean","last_fresh":"2024-03-11T21:39:26.601686+0000","last_change":"2024-03-11T21:35:24.841170+0000","last_active":"2024-03-11T21:39:26.601686+0000","last_peered":"2024-03-11T21:39:26.601686+0000","last_clean":"2024-03-11T21:39:26.601686+0000","last_became_active":"2024-03-11T21:34:41.139782+0000","last_became_peered":"2024-03-11T21:34:41.139782+0000","last_unstale":"2024-03-11T21:39:26.601686+0000","last_undegraded":"2024-03-11T21:39:26.601686+0000","last_fullsized":"2024-03-11T21:39:26.601686+0000","mapping_epoch":226,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":227,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:35:24.841041+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:32:21.266672+0000","last_clean_scrub_stamp":"2024-03-11T21:35:24.841041+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,5],"acting":[7,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.b","version":"332'13141","reported_seq":13297,"reported_epoch":332,"state":"active+clean","last_fresh":"2024-03-11T21:39:28.760204+0000","last_change":"2024-03-11T21:35:45.697213+0000","last_active":"2024-03-11T21:39:28.760204+0000","last_peered":"2024-03-11T21:39:28.760204+0000","last_clean":"2024-03-11T21:39:28.760204+0000","last_became_active":"2024-03-11T21:35:45.696869+0000","last_became_peered":"2024-03-11T21:35:45.696869+0000","last_unstale":"2024-03-11T21:39:28.760204+0000","last_undegraded":"2024-03-11T21:39:28.760204+0000","last_fullsized":"2024-03-11T21:39:28.760204+0000","mapping_epoch":259,"log_start":"298'7500","ondisk_log_start":"298'7500","created":259,"last_epoch_clean":260,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:35:44.669493+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:35:44.669493+0000","last_clean_scrub_stamp":"2024-03-11T21:35:44.669493+0000","objects_scrubbed":0,"log_size":5641,"log_dups_size":0,"ondisk_log_size":5641,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":85524480,"num_objects":1305,"num_object_clones":0,"num_object_copies":3915,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1305,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13141,"num_write_kb":462272,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,5],"acting":[0,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.4","version":"0'0","reported_seq":576,"reported_epoch":332,"state":"active+clean","last_fresh":"2024-03-11T21:39:27.603505+0000","last_change":"2024-03-11T21:36:35.073305+0000","last_active":"2024-03-11T21:39:27.603505+0000","last_peered":"2024-03-11T21:39:27.603505+0000","last_clean":"2024-03-11T21:39:27.603505+0000","last_became_active":"2024-03-11T21:36:35.072824+0000","last_became_peered":"2024-03-11T21:36:35.072824+0000","last_unstale":"2024-03-11T21:39:27.603505+0000","last_undegraded":"2024-03-11T21:39:27.603505+0000","last_fullsized":"2024-03-11T21:39:27.603505+0000","mapping_epoch":277,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":278,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:34:23.531064+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:43.559452+0000","last_clean_scrub_stamp":"2024-03-11T21:34:23.531064+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":603,"reported_epoch":332,"state":"active+clean","last_fresh":"2024-03-11T21:39:27.603491+0000","last_change":"2024-03-11T21:36:35.070626+0000","last_active":"2024-03-11T21:39:27.603491+0000","last_peered":"2024-03-11T21:39:27.603491+0000","last_clean":"2024-03-11T21:39:27.603491+0000","last_became_active":"2024-03-11T21:36:35.069452+0000","last_became_peered":"2024-03-11T21:36:35.069452+0000","last_unstale":"2024-03-11T21:39:27.603491+0000","last_undegraded":"2024-03-11T21:39:27.603491+0000","last_fullsized":"2024-03-11T21:39:27.603491+0000","mapping_epoch":277,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":278,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:35:13.741947+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:37.512018+0000","last_clean_scrub_stamp":"2024-03-11T21:35:13.741947+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0.00089342499999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":221,"reported_epoch":331,"state":"active+clean","last_fresh":"2024-03-11T21:39:26.600921+0000","last_change":"2024-03-11T21:34:41.150606+0000","last_active":"2024-03-11T21:39:26.600921+0000","last_peered":"2024-03-11T21:39:26.600921+0000","last_clean":"2024-03-11T21:39:26.600921+0000","last_became_active":"2024-03-11T21:34:41.149219+0000","last_became_peered":"2024-03-11T21:34:41.149219+0000","last_unstale":"2024-03-11T21:39:26.600921+0000","last_undegraded":"2024-03-11T21:39:26.600921+0000","last_fullsized":"2024-03-11T21:39:26.600921+0000","mapping_epoch":226,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":227,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:34:31.628014+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:37.512018+0000","last_clean_scrub_stamp":"2024-03-11T21:34:31.628014+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,0],"acting":[7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.6","version":"260'24","reported_seq":639,"reported_epoch":332,"state":"active+clean","last_fresh":"2024-03-11T21:39:27.602574+0000","last_change":"2024-03-11T21:35:19.753433+0000","last_active":"2024-03-11T21:39:27.602574+0000","last_peered":"2024-03-11T21:39:27.602574+0000","last_clean":"2024-03-11T21:39:27.602574+0000","last_became_active":"2024-03-11T21:32:22.796486+0000","last_became_peered":"2024-03-11T21:32:22.796486+0000","last_unstale":"2024-03-11T21:39:27.602574+0000","last_undegraded":"2024-03-11T21:39:27.602574+0000","last_fullsized":"2024-03-11T21:39:27.602574+0000","mapping_epoch":166,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":167,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'12","last_scrub_stamp":"2024-03-11T21:35:19.753224+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:39.525554+0000","last_clean_scrub_stamp":"2024-03-11T21:35:19.753224+0000","objects_scrubbed":6,"log_size":24,"log_dups_size":0,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":36,"num_read_kb":30,"num_write":24,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":576,"reported_epoch":331,"state":"active+clean","last_fresh":"2024-03-11T21:39:26.608426+0000","last_change":"2024-03-11T21:36:07.451482+0000","last_active":"2024-03-11T21:39:26.608426+0000","last_peered":"2024-03-11T21:39:26.608426+0000","last_clean":"2024-03-11T21:39:26.608426+0000","last_became_active":"2024-03-11T21:25:42.572324+0000","last_became_peered":"2024-03-11T21:25:42.572324+0000","last_unstale":"2024-03-11T21:39:26.608426+0000","last_undegraded":"2024-03-11T21:39:26.608426+0000","last_fullsized":"2024-03-11T21:39:26.608426+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:36:07.451102+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:41.544550+0000","last_clean_scrub_stamp":"2024-03-11T21:36:07.451102+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":54,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.e","version":"331'12881","reported_seq":13036,"reported_epoch":331,"state":"active+clean","last_fresh":"2024-03-11T21:39:27.337616+0000","last_change":"2024-03-11T21:35:45.697331+0000","last_active":"2024-03-11T21:39:27.337616+0000","last_peered":"2024-03-11T21:39:27.337616+0000","last_clean":"2024-03-11T21:39:27.337616+0000","last_became_active":"2024-03-11T21:35:45.696948+0000","last_became_peered":"2024-03-11T21:35:45.696948+0000","last_unstale":"2024-03-11T21:39:27.337616+0000","last_undegraded":"2024-03-11T21:39:27.337616+0000","last_fullsized":"2024-03-11T21:39:27.337616+0000","mapping_epoch":259,"log_start":"304'8200","ondisk_log_start":"304'8200","created":259,"last_epoch_clean":260,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:35:44.669493+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:35:44.669493+0000","last_clean_scrub_stamp":"2024-03-11T21:35:44.669493+0000","objects_scrubbed":0,"log_size":4681,"log_dups_size":0,"ondisk_log_size":4681,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":91684864,"num_objects":1399,"num_object_clones":0,"num_object_copies":4197,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1399,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12881,"num_write_kb":456960,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":583,"reported_epoch":332,"state":"active+clean","last_fresh":"2024-03-11T21:39:27.597243+0000","last_change":"2024-03-11T21:36:18.529506+0000","last_active":"2024-03-11T21:39:27.597243+0000","last_peered":"2024-03-11T21:39:27.597243+0000","last_clean":"2024-03-11T21:39:27.597243+0000","last_became_active":"2024-03-11T21:25:45.301416+0000","last_became_peered":"2024-03-11T21:25:45.301416+0000","last_unstale":"2024-03-11T21:39:27.597243+0000","last_undegraded":"2024-03-11T21:39:27.597243+0000","last_fullsized":"2024-03-11T21:39:27.597243+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:36:18.529370+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:43.559452+0000","last_clean_scrub_stamp":"2024-03-11T21:36:18.529370+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":54,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":229,"reported_epoch":331,"state":"active+clean","last_fresh":"2024-03-11T21:39:26.602114+0000","last_change":"2024-03-11T21:35:33.740615+0000","last_active":"2024-03-11T21:39:26.602114+0000","last_peered":"2024-03-11T21:39:26.602114+0000","last_clean":"2024-03-11T21:39:26.602114+0000","last_became_active":"2024-03-11T21:34:41.152721+0000","last_became_peered":"2024-03-11T21:34:41.152721+0000","last_unstale":"2024-03-11T21:39:26.602114+0000","last_undegraded":"2024-03-11T21:39:26.602114+0000","last_fullsized":"2024-03-11T21:39:26.602114+0000","mapping_epoch":226,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":227,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:35:33.740538+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:26:57.333669+0000","last_clean_scrub_stamp":"2024-03-11T21:35:33.740538+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.3","version":"260'37","reported_seq":690,"reported_epoch":332,"state":"active+clean","last_fresh":"2024-03-11T21:39:27.606438+0000","last_change":"2024-03-11T21:35:11.697314+0000","last_active":"2024-03-11T21:39:27.606438+0000","last_peered":"2024-03-11T21:39:27.606438+0000","last_clean":"2024-03-11T21:39:27.606438+0000","last_became_active":"2024-03-11T21:34:41.145607+0000","last_became_peered":"2024-03-11T21:34:41.145607+0000","last_unstale":"2024-03-11T21:39:27.606438+0000","last_undegraded":"2024-03-11T21:39:27.606438+0000","last_fullsized":"2024-03-11T21:39:27.606438+0000","mapping_epoch":226,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":227,"parent":"0.0","parent_split_bits":0,"last_scrub":"165'27","last_scrub_stamp":"2024-03-11T21:35:11.697140+0000","last_deep_scrub":"165'27","last_deep_scrub_stamp":"2024-03-11T21:32:21.576732+0000","last_clean_scrub_stamp":"2024-03-11T21:35:11.697140+0000","objects_scrubbed":8,"log_size":37,"log_dups_size":0,"ondisk_log_size":37,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":54,"num_read_kb":40,"num_write":32,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":660,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":593,"reported_epoch":331,"state":"active+clean","last_fresh":"2024-03-11T21:39:26.614055+0000","last_change":"2024-03-11T21:38:53.027632+0000","last_active":"2024-03-11T21:39:26.614055+0000","last_peered":"2024-03-11T21:39:26.614055+0000","last_clean":"2024-03-11T21:39:26.614055+0000","last_became_active":"2024-03-11T21:25:42.579045+0000","last_became_peered":"2024-03-11T21:25:42.579045+0000","last_unstale":"2024-03-11T21:39:26.614055+0000","last_undegraded":"2024-03-11T21:39:26.614055+0000","last_fullsized":"2024-03-11T21:39:26.614055+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:38:53.027520+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:38:53.027520+0000","last_clean_scrub_stamp":"2024-03-11T21:38:53.027520+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:40:12.157404+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.d","version":"331'13104","reported_seq":13260,"reported_epoch":331,"state":"active+clean","last_fresh":"2024-03-11T21:39:27.272971+0000","last_change":"2024-03-11T21:35:45.700060+0000","last_active":"2024-03-11T21:39:27.272971+0000","last_peered":"2024-03-11T21:39:27.272971+0000","last_clean":"2024-03-11T21:39:27.272971+0000","last_became_active":"2024-03-11T21:35:45.699674+0000","last_became_peered":"2024-03-11T21:35:45.699674+0000","last_unstale":"2024-03-11T21:39:27.272971+0000","last_undegraded":"2024-03-11T21:39:27.272971+0000","last_fullsized":"2024-03-11T21:39:27.272971+0000","mapping_epoch":259,"log_start":"305'8400","ondisk_log_start":"305'8400","created":259,"last_epoch_clean":260,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:35:44.669493+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:35:44.669493+0000","last_clean_scrub_stamp":"2024-03-11T21:35:44.669493+0000","objects_scrubbed":0,"log_size":4704,"log_dups_size":0,"ondisk_log_size":4704,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":89194520,"num_objects":1362,"num_object_clones":0,"num_object_copies":4086,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1362,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":13104,"num_write_kb":462849,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,5],"acting":[7,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":579,"reported_epoch":331,"state":"active+clean","last_fresh":"2024-03-11T21:39:26.585805+0000","last_change":"2024-03-11T21:35:08.437783+0000","last_active":"2024-03-11T21:39:26.585805+0000","last_peered":"2024-03-11T21:39:26.585805+0000","last_clean":"2024-03-11T21:39:26.585805+0000","last_became_active":"2024-03-11T21:32:22.791439+0000","last_became_peered":"2024-03-11T21:32:22.791439+0000","last_unstale":"2024-03-11T21:39:26.585805+0000","last_undegraded":"2024-03-11T21:39:26.585805+0000","last_fullsized":"2024-03-11T21:39:26.585805+0000","mapping_epoch":166,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":167,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:35:08.437697+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:43.559452+0000","last_clean_scrub_stamp":"2024-03-11T21:35:08.437697+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":33,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,2],"acting":[4,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.6","version":"67'1","reported_seq":588,"reported_epoch":332,"state":"active+clean","last_fresh":"2024-03-11T21:39:27.597403+0000","last_change":"2024-03-11T21:35:30.581783+0000","last_active":"2024-03-11T21:39:27.597403+0000","last_peered":"2024-03-11T21:39:27.597403+0000","last_clean":"2024-03-11T21:39:27.597403+0000","last_became_active":"2024-03-11T21:25:38.536142+0000","last_became_peered":"2024-03-11T21:25:38.536142+0000","last_unstale":"2024-03-11T21:39:27.597403+0000","last_undegraded":"2024-03-11T21:39:27.597403+0000","last_fullsized":"2024-03-11T21:39:27.597403+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'1","last_scrub_stamp":"2024-03-11T21:35:30.581701+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:37.512018+0000","last_clean_scrub_stamp":"2024-03-11T21:35:30.581701+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,4],"acting":[1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.0","version":"260'38","reported_seq":586,"reported_epoch":332,"state":"active+clean","last_fresh":"2024-03-11T21:39:27.610279+0000","last_change":"2024-03-11T21:36:35.087771+0000","last_active":"2024-03-11T21:39:27.610279+0000","last_peered":"2024-03-11T21:39:27.610279+0000","last_clean":"2024-03-11T21:39:27.610279+0000","last_became_active":"2024-03-11T21:36:35.068071+0000","last_became_peered":"2024-03-11T21:36:35.068071+0000","last_unstale":"2024-03-11T21:39:27.610279+0000","last_undegraded":"2024-03-11T21:39:27.610279+0000","last_fullsized":"2024-03-11T21:39:27.610279+0000","mapping_epoch":277,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":278,"parent":"0.0","parent_split_bits":0,"last_scrub":"93'26","last_scrub_stamp":"2024-03-11T21:34:16.863984+0000","last_deep_scrub":"93'26","last_deep_scrub_stamp":"2024-03-11T21:26:40.680324+0000","last_clean_scrub_stamp":"2024-03-11T21:34:16.863984+0000","objects_scrubbed":8,"log_size":38,"log_dups_size":0,"ondisk_log_size":38,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":22,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":51,"num_read_kb":39,"num_write":32,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":598,"reported_epoch":331,"state":"active+clean","last_fresh":"2024-03-11T21:39:26.585278+0000","last_change":"2024-03-11T21:36:35.057830+0000","last_active":"2024-03-11T21:39:26.585278+0000","last_peered":"2024-03-11T21:39:26.585278+0000","last_clean":"2024-03-11T21:39:26.585278+0000","last_became_active":"2024-03-11T21:36:35.057517+0000","last_became_peered":"2024-03-11T21:36:35.057517+0000","last_unstale":"2024-03-11T21:39:26.585278+0000","last_undegraded":"2024-03-11T21:39:26.585278+0000","last_fullsized":"2024-03-11T21:39:26.585278+0000","mapping_epoch":277,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":278,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:35:12.411136+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:26:51.863621+0000","last_clean_scrub_stamp":"2024-03-11T21:35:12.411136+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,7],"acting":[4,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.c","version":"332'12942","reported_seq":22363,"reported_epoch":332,"state":"active+clean","last_fresh":"2024-03-11T21:39:29.063909+0000","last_change":"2024-03-11T21:38:30.644319+0000","last_active":"2024-03-11T21:39:29.063909+0000","last_peered":"2024-03-11T21:39:29.063909+0000","last_clean":"2024-03-11T21:39:29.063909+0000","last_became_active":"2024-03-11T21:38:30.610971+0000","last_became_peered":"2024-03-11T21:38:30.610971+0000","last_unstale":"2024-03-11T21:39:29.063909+0000","last_undegraded":"2024-03-11T21:39:29.063909+0000","last_fullsized":"2024-03-11T21:39:29.063909+0000","mapping_epoch":321,"log_start":"303'8044","ondisk_log_start":"303'8044","created":259,"last_epoch_clean":322,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:35:44.669493+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:35:44.669493+0000","last_clean_scrub_stamp":"2024-03-11T21:35:44.669493+0000","objects_scrubbed":0,"log_size":4898,"log_dups_size":0,"ondisk_log_size":4898,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":83886080,"num_objects":1280,"num_object_clones":0,"num_object_copies":3840,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1280,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12942,"num_write_kb":455104,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4628,"num_bytes_recovered":303300608,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,7],"acting":[3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.3","version":"0'0","reported_seq":221,"reported_epoch":331,"state":"active+clean","last_fresh":"2024-03-11T21:39:26.602758+0000","last_change":"2024-03-11T21:34:41.143973+0000","last_active":"2024-03-11T21:39:26.602758+0000","last_peered":"2024-03-11T21:39:26.602758+0000","last_clean":"2024-03-11T21:39:26.602758+0000","last_became_active":"2024-03-11T21:34:41.142683+0000","last_became_peered":"2024-03-11T21:34:41.142683+0000","last_unstale":"2024-03-11T21:39:26.602758+0000","last_undegraded":"2024-03-11T21:39:26.602758+0000","last_fullsized":"2024-03-11T21:39:26.602758+0000","mapping_epoch":226,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":227,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:34:27.353700+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:29:42.485740+0000","last_clean_scrub_stamp":"2024-03-11T21:34:27.353700+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,2],"acting":[7,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":613,"reported_epoch":331,"state":"active+clean","last_fresh":"2024-03-11T21:39:26.614595+0000","last_change":"2024-03-11T21:38:55.048335+0000","last_active":"2024-03-11T21:39:26.614595+0000","last_peered":"2024-03-11T21:39:26.614595+0000","last_clean":"2024-03-11T21:39:26.614595+0000","last_became_active":"2024-03-11T21:34:41.145628+0000","last_became_peered":"2024-03-11T21:34:41.145628+0000","last_unstale":"2024-03-11T21:39:26.614595+0000","last_undegraded":"2024-03-11T21:39:26.614595+0000","last_fullsized":"2024-03-11T21:39:26.614595+0000","mapping_epoch":226,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":227,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:38:55.048231+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:37.512018+0000","last_clean_scrub_stamp":"2024-03-11T21:38:55.048231+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:40:07.136440+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,2],"acting":[6,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.1","version":"260'28","reported_seq":623,"reported_epoch":331,"state":"active+clean","last_fresh":"2024-03-11T21:39:26.585029+0000","last_change":"2024-03-11T21:35:33.644654+0000","last_active":"2024-03-11T21:39:26.585029+0000","last_peered":"2024-03-11T21:39:26.585029+0000","last_clean":"2024-03-11T21:39:26.585029+0000","last_became_active":"2024-03-11T21:25:40.546219+0000","last_became_peered":"2024-03-11T21:25:40.546219+0000","last_unstale":"2024-03-11T21:39:26.585029+0000","last_undegraded":"2024-03-11T21:39:26.585029+0000","last_fullsized":"2024-03-11T21:39:26.585029+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'14","last_scrub_stamp":"2024-03-11T21:35:33.644503+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:39.525554+0000","last_clean_scrub_stamp":"2024-03-11T21:35:33.644503+0000","objects_scrubbed":7,"log_size":28,"log_dups_size":0,"ondisk_log_size":28,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":42,"num_read_kb":35,"num_write":28,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6],"acting":[4,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.0","version":"280'11","reported_seq":680,"reported_epoch":332,"state":"active+clean","last_fresh":"2024-03-11T21:39:27.612902+0000","last_change":"2024-03-11T21:36:35.064085+0000","last_active":"2024-03-11T21:39:27.612902+0000","last_peered":"2024-03-11T21:39:27.612902+0000","last_clean":"2024-03-11T21:39:27.612902+0000","last_became_active":"2024-03-11T21:36:35.055250+0000","last_became_peered":"2024-03-11T21:36:35.055250+0000","last_unstale":"2024-03-11T21:39:27.612902+0000","last_undegraded":"2024-03-11T21:39:27.612902+0000","last_fullsized":"2024-03-11T21:39:27.612902+0000","mapping_epoch":277,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":278,"parent":"0.0","parent_split_bits":0,"last_scrub":"98'9","last_scrub_stamp":"2024-03-11T21:35:21.143578+0000","last_deep_scrub":"98'9","last_deep_scrub_stamp":"2024-03-11T21:27:29.745711+0000","last_clean_scrub_stamp":"2024-03-11T21:35:21.143578+0000","objects_scrubbed":1,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.3","version":"331'12918","reported_seq":13078,"reported_epoch":331,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-03-11T21:39:27.357953+0000","last_change":"2024-03-11T21:36:36.544317+0000","last_active":"2024-03-11T21:39:27.357953+0000","last_peered":"2024-03-11T21:39:27.357953+0000","last_clean":"2024-03-11T21:35:44.669493+0000","last_became_active":"2024-03-11T21:36:36.470537+0000","last_became_peered":"2024-03-11T21:36:36.470537+0000","last_unstale":"2024-03-11T21:39:27.357953+0000","last_undegraded":"2024-03-11T21:36:36.071501+0000","last_fullsized":"2024-03-11T21:36:36.040233+0000","mapping_epoch":278,"log_start":"305'8300","ondisk_log_start":"305'8300","created":259,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:35:44.669493+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:35:44.669493+0000","last_clean_scrub_stamp":"2024-03-11T21:35:44.669493+0000","objects_scrubbed":0,"log_size":4618,"log_dups_size":0,"ondisk_log_size":4618,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":91881472,"num_objects":1402,"num_object_clones":0,"num_object_copies":4206,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4641,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1402,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12918,"num_write_kb":458240,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":196608,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,1],"acting":[7,1],"avail_no_missing":["7","1"],"object_location_counts":[{"shards":"1,7","objects":-3239}],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.c","version":"0'0","reported_seq":507,"reported_epoch":332,"state":"active+clean","last_fresh":"2024-03-11T21:39:27.611754+0000","last_change":"2024-03-11T21:36:35.074112+0000","last_active":"2024-03-11T21:39:27.611754+0000","last_peered":"2024-03-11T21:39:27.611754+0000","last_clean":"2024-03-11T21:39:27.611754+0000","last_became_active":"2024-03-11T21:36:35.073778+0000","last_became_peered":"2024-03-11T21:36:35.073778+0000","last_unstale":"2024-03-11T21:39:27.611754+0000","last_undegraded":"2024-03-11T21:39:27.611754+0000","last_fullsized":"2024-03-11T21:39:27.611754+0000","mapping_epoch":277,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":278,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:32:17.600213+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:43.559452+0000","last_clean_scrub_stamp":"2024-03-11T21:32:17.600213+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.f","version":"0'0","reported_seq":584,"reported_epoch":331,"state":"active+clean","last_fresh":"2024-03-11T21:39:26.608350+0000","last_change":"2024-03-11T21:36:08.439013+0000","last_active":"2024-03-11T21:39:26.608350+0000","last_peered":"2024-03-11T21:39:26.608350+0000","last_clean":"2024-03-11T21:39:26.608350+0000","last_became_active":"2024-03-11T21:25:42.575935+0000","last_became_peered":"2024-03-11T21:25:42.575935+0000","last_unstale":"2024-03-11T21:39:26.608350+0000","last_undegraded":"2024-03-11T21:39:26.608350+0000","last_fullsized":"2024-03-11T21:39:26.608350+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:36:08.438895+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:41.544550+0000","last_clean_scrub_stamp":"2024-03-11T21:36:08.438895+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.8","version":"0'0","reported_seq":229,"reported_epoch":331,"state":"active+clean","last_fresh":"2024-03-11T21:39:26.601146+0000","last_change":"2024-03-11T21:35:34.733595+0000","last_active":"2024-03-11T21:39:26.601146+0000","last_peered":"2024-03-11T21:39:26.601146+0000","last_clean":"2024-03-11T21:39:26.601146+0000","last_became_active":"2024-03-11T21:34:41.144681+0000","last_became_peered":"2024-03-11T21:34:41.144681+0000","last_unstale":"2024-03-11T21:39:26.601146+0000","last_undegraded":"2024-03-11T21:39:26.601146+0000","last_fullsized":"2024-03-11T21:39:26.601146+0000","mapping_epoch":226,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":227,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:35:34.733491+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:37.512018+0000","last_clean_scrub_stamp":"2024-03-11T21:35:34.733491+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,1],"acting":[7,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.e","version":"260'24","reported_seq":617,"reported_epoch":331,"state":"active+clean","last_fresh":"2024-03-11T21:39:26.586745+0000","last_change":"2024-03-11T21:35:34.679906+0000","last_active":"2024-03-11T21:39:26.586745+0000","last_peered":"2024-03-11T21:39:26.586745+0000","last_clean":"2024-03-11T21:39:26.586745+0000","last_became_active":"2024-03-11T21:25:40.552178+0000","last_became_peered":"2024-03-11T21:25:40.552178+0000","last_unstale":"2024-03-11T21:39:26.586745+0000","last_undegraded":"2024-03-11T21:39:26.586745+0000","last_fullsized":"2024-03-11T21:39:26.586745+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"109'16","last_scrub_stamp":"2024-03-11T21:35:34.679817+0000","last_deep_scrub":"109'16","last_deep_scrub_stamp":"2024-03-11T21:28:24.739395+0000","last_clean_scrub_stamp":"2024-03-11T21:35:34.679817+0000","objects_scrubbed":5,"log_size":24,"log_dups_size":0,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":32,"num_read_kb":25,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.2","version":"332'13187","reported_seq":16673,"reported_epoch":332,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-03-11T21:39:28.223298+0000","last_change":"2024-03-11T21:38:29.073541+0000","last_active":"2024-03-11T21:39:28.223298+0000","last_peered":"2024-03-11T21:39:28.223298+0000","last_clean":"2024-03-11T21:35:44.669493+0000","last_became_active":"2024-03-11T21:36:36.506474+0000","last_became_peered":"2024-03-11T21:36:36.506474+0000","last_unstale":"2024-03-11T21:39:28.223298+0000","last_undegraded":"2024-03-11T21:36:36.068182+0000","last_fullsized":"2024-03-11T21:36:36.037781+0000","mapping_epoch":278,"log_start":"302'8100","ondisk_log_start":"302'8100","created":259,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:35:44.669493+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:35:44.669493+0000","last_clean_scrub_stamp":"2024-03-11T21:35:44.669493+0000","objects_scrubbed":0,"log_size":5087,"log_dups_size":0,"ondisk_log_size":5087,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":88014848,"num_objects":1343,"num_object_clones":0,"num_object_copies":4029,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3108,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1343,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13187,"num_write_kb":464960,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1665,"num_bytes_recovered":65536000,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[1,2],"avail_no_missing":["1","2"],"object_location_counts":[{"shards":"1,2","objects":-1765}],"blocked_by":[],"up_primary":3,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.d","version":"0'0","reported_seq":572,"reported_epoch":331,"state":"active+clean","last_fresh":"2024-03-11T21:39:26.603686+0000","last_change":"2024-03-11T21:35:37.185210+0000","last_active":"2024-03-11T21:39:26.603686+0000","last_peered":"2024-03-11T21:39:26.603686+0000","last_clean":"2024-03-11T21:39:26.603686+0000","last_became_active":"2024-03-11T21:25:44.590333+0000","last_became_peered":"2024-03-11T21:25:44.590333+0000","last_unstale":"2024-03-11T21:39:26.603686+0000","last_undegraded":"2024-03-11T21:39:26.603686+0000","last_fullsized":"2024-03-11T21:39:26.603686+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:35:37.182850+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:35:37.182850+0000","last_clean_scrub_stamp":"2024-03-11T21:35:37.182850+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":40,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.e","version":"74'8","reported_seq":751,"reported_epoch":331,"state":"active+clean","last_fresh":"2024-03-11T21:39:26.587038+0000","last_change":"2024-03-11T21:36:07.734369+0000","last_active":"2024-03-11T21:39:26.587038+0000","last_peered":"2024-03-11T21:39:26.587038+0000","last_clean":"2024-03-11T21:39:26.587038+0000","last_became_active":"2024-03-11T21:25:42.576742+0000","last_became_peered":"2024-03-11T21:25:42.576742+0000","last_unstale":"2024-03-11T21:39:26.587038+0000","last_undegraded":"2024-03-11T21:39:26.587038+0000","last_fullsized":"2024-03-11T21:39:26.587038+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'8","last_scrub_stamp":"2024-03-11T21:36:07.734115+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:41.544550+0000","last_clean_scrub_stamp":"2024-03-11T21:36:07.734115+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,0],"acting":[4,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.9","version":"0'0","reported_seq":600,"reported_epoch":332,"state":"active+clean","last_fresh":"2024-03-11T21:39:27.597048+0000","last_change":"2024-03-11T21:36:35.066093+0000","last_active":"2024-03-11T21:39:27.597048+0000","last_peered":"2024-03-11T21:39:27.597048+0000","last_clean":"2024-03-11T21:39:27.597048+0000","last_became_active":"2024-03-11T21:36:35.065733+0000","last_became_peered":"2024-03-11T21:36:35.065733+0000","last_unstale":"2024-03-11T21:39:27.597048+0000","last_undegraded":"2024-03-11T21:39:27.597048+0000","last_fullsized":"2024-03-11T21:39:27.597048+0000","mapping_epoch":277,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":278,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:32:02.506579+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:26:50.991712+0000","last_clean_scrub_stamp":"2024-03-11T21:32:02.506579+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,3],"acting":[1,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.f","version":"260'27","reported_seq":599,"reported_epoch":331,"state":"active+clean","last_fresh":"2024-03-11T21:39:26.626245+0000","last_change":"2024-03-11T21:36:35.084069+0000","last_active":"2024-03-11T21:39:26.626245+0000","last_peered":"2024-03-11T21:39:26.626245+0000","last_clean":"2024-03-11T21:39:26.626245+0000","last_became_active":"2024-03-11T21:36:35.058272+0000","last_became_peered":"2024-03-11T21:36:35.058272+0000","last_unstale":"2024-03-11T21:39:26.626245+0000","last_undegraded":"2024-03-11T21:39:26.626245+0000","last_fullsized":"2024-03-11T21:39:26.626245+0000","mapping_epoch":277,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":278,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'15","last_scrub_stamp":"2024-03-11T21:34:21.517894+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:39.525554+0000","last_clean_scrub_stamp":"2024-03-11T21:34:21.517894+0000","objects_scrubbed":7,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":44,"num_read_kb":35,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,4],"acting":[2,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"9.1","version":"332'13310","reported_seq":23124,"reported_epoch":332,"state":"active+clean","last_fresh":"2024-03-11T21:39:29.022460+0000","last_change":"2024-03-11T21:38:22.558103+0000","last_active":"2024-03-11T21:39:29.022460+0000","last_peered":"2024-03-11T21:39:29.022460+0000","last_clean":"2024-03-11T21:39:29.022460+0000","last_became_active":"2024-03-11T21:38:22.540816+0000","last_became_peered":"2024-03-11T21:38:22.540816+0000","last_unstale":"2024-03-11T21:39:29.022460+0000","last_undegraded":"2024-03-11T21:39:29.022460+0000","last_fullsized":"2024-03-11T21:39:29.022460+0000","mapping_epoch":316,"log_start":"305'8395","ondisk_log_start":"305'8395","created":259,"last_epoch_clean":317,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:35:44.669493+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:35:44.669493+0000","last_clean_scrub_stamp":"2024-03-11T21:35:44.669493+0000","objects_scrubbed":0,"log_size":4915,"log_dups_size":0,"ondisk_log_size":4915,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":83492864,"num_objects":1274,"num_object_clones":0,"num_object_copies":3822,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1274,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13310,"num_write_kb":466688,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4825,"num_bytes_recovered":316211200,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,7],"acting":[3,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.e","version":"0'0","reported_seq":573,"reported_epoch":331,"state":"active+clean","last_fresh":"2024-03-11T21:39:26.587132+0000","last_change":"2024-03-11T21:36:10.769557+0000","last_active":"2024-03-11T21:39:26.587132+0000","last_peered":"2024-03-11T21:39:26.587132+0000","last_clean":"2024-03-11T21:39:26.587132+0000","last_became_active":"2024-03-11T21:25:44.590198+0000","last_became_peered":"2024-03-11T21:25:44.590198+0000","last_unstale":"2024-03-11T21:39:26.587132+0000","last_undegraded":"2024-03-11T21:39:26.587132+0000","last_fullsized":"2024-03-11T21:39:26.587132+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:36:10.769460+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:43.559452+0000","last_clean_scrub_stamp":"2024-03-11T21:36:10.769460+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":75,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.d","version":"109'9","reported_seq":751,"reported_epoch":331,"state":"active+clean","last_fresh":"2024-03-11T21:39:26.625794+0000","last_change":"2024-03-11T21:35:07.562523+0000","last_active":"2024-03-11T21:39:26.625794+0000","last_peered":"2024-03-11T21:39:26.625794+0000","last_clean":"2024-03-11T21:39:26.625794+0000","last_became_active":"2024-03-11T21:25:42.575583+0000","last_became_peered":"2024-03-11T21:25:42.575583+0000","last_unstale":"2024-03-11T21:39:26.625794+0000","last_undegraded":"2024-03-11T21:39:26.625794+0000","last_fullsized":"2024-03-11T21:39:26.625794+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"109'9","last_scrub_stamp":"2024-03-11T21:35:07.562385+0000","last_deep_scrub":"109'9","last_deep_scrub_stamp":"2024-03-11T21:28:27.082975+0000","last_clean_scrub_stamp":"2024-03-11T21:35:07.562385+0000","objects_scrubbed":1,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,5],"acting":[2,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.a","version":"0'0","reported_seq":608,"reported_epoch":332,"state":"active+clean","last_fresh":"2024-03-11T21:39:27.599648+0000","last_change":"2024-03-11T21:36:35.067262+0000","last_active":"2024-03-11T21:39:27.599648+0000","last_peered":"2024-03-11T21:39:27.599648+0000","last_clean":"2024-03-11T21:39:27.599648+0000","last_became_active":"2024-03-11T21:36:35.066832+0000","last_became_peered":"2024-03-11T21:36:35.066832+0000","last_unstale":"2024-03-11T21:39:27.599648+0000","last_undegraded":"2024-03-11T21:39:27.599648+0000","last_fullsized":"2024-03-11T21:39:27.599648+0000","mapping_epoch":277,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":278,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:34:22.526916+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:37.512018+0000","last_clean_scrub_stamp":"2024-03-11T21:34:22.526916+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,7],"acting":[1,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.c","version":"260'20","reported_seq":625,"reported_epoch":331,"state":"active+clean","last_fresh":"2024-03-11T21:39:26.586388+0000","last_change":"2024-03-11T21:36:35.095574+0000","last_active":"2024-03-11T21:39:26.586388+0000","last_peered":"2024-03-11T21:39:26.586388+0000","last_clean":"2024-03-11T21:39:26.586388+0000","last_became_active":"2024-03-11T21:36:35.067269+0000","last_became_peered":"2024-03-11T21:36:35.067269+0000","last_unstale":"2024-03-11T21:39:26.586388+0000","last_undegraded":"2024-03-11T21:39:26.586388+0000","last_fullsized":"2024-03-11T21:39:26.586388+0000","mapping_epoch":277,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":278,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'10","last_scrub_stamp":"2024-03-11T21:32:54.846584+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:39.525554+0000","last_clean_scrub_stamp":"2024-03-11T21:32:54.846584+0000","objects_scrubbed":5,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":25,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.0","version":"332'13227","reported_seq":13384,"reported_epoch":332,"state":"active+clean","last_fresh":"2024-03-11T21:39:28.166672+0000","last_change":"2024-03-11T21:35:45.703065+0000","last_active":"2024-03-11T21:39:28.166672+0000","last_peered":"2024-03-11T21:39:28.166672+0000","last_clean":"2024-03-11T21:39:28.166672+0000","last_became_active":"2024-03-11T21:35:45.702739+0000","last_became_peered":"2024-03-11T21:35:45.702739+0000","last_unstale":"2024-03-11T21:39:28.166672+0000","last_undegraded":"2024-03-11T21:39:28.166672+0000","last_fullsized":"2024-03-11T21:39:28.166672+0000","mapping_epoch":259,"log_start":"302'8200","ondisk_log_start":"302'8200","created":259,"last_epoch_clean":260,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:35:44.669493+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:35:44.669493+0000","last_clean_scrub_stamp":"2024-03-11T21:35:44.669493+0000","objects_scrubbed":0,"log_size":5027,"log_dups_size":0,"ondisk_log_size":5027,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":88932352,"num_objects":1357,"num_object_clones":0,"num_object_copies":4071,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1357,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13227,"num_write_kb":466688,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.f","version":"0'0","reported_seq":581,"reported_epoch":331,"state":"active+clean","last_fresh":"2024-03-11T21:39:26.626108+0000","last_change":"2024-03-11T21:36:35.058630+0000","last_active":"2024-03-11T21:39:26.626108+0000","last_peered":"2024-03-11T21:39:26.626108+0000","last_clean":"2024-03-11T21:39:26.626108+0000","last_became_active":"2024-03-11T21:36:35.057800+0000","last_became_peered":"2024-03-11T21:36:35.057800+0000","last_unstale":"2024-03-11T21:39:26.626108+0000","last_undegraded":"2024-03-11T21:39:26.626108+0000","last_fullsized":"2024-03-11T21:39:26.626108+0000","mapping_epoch":277,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":278,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:35:10.608981+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:27:03.010969+0000","last_clean_scrub_stamp":"2024-03-11T21:35:10.608981+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":94,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,4],"acting":[2,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":579,"reported_epoch":332,"state":"active+clean","last_fresh":"2024-03-11T21:39:27.603428+0000","last_change":"2024-03-11T21:36:10.608789+0000","last_active":"2024-03-11T21:39:27.603428+0000","last_peered":"2024-03-11T21:39:27.603428+0000","last_clean":"2024-03-11T21:39:27.603428+0000","last_became_active":"2024-03-11T21:25:42.579409+0000","last_became_peered":"2024-03-11T21:25:42.579409+0000","last_unstale":"2024-03-11T21:39:27.603428+0000","last_undegraded":"2024-03-11T21:39:27.603428+0000","last_fullsized":"2024-03-11T21:39:27.603428+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:36:10.608552+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:30:22.750053+0000","last_clean_scrub_stamp":"2024-03-11T21:36:10.608552+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.b","version":"0'0","reported_seq":229,"reported_epoch":331,"state":"active+clean","last_fresh":"2024-03-11T21:39:26.602836+0000","last_change":"2024-03-11T21:36:16.719432+0000","last_active":"2024-03-11T21:39:26.602836+0000","last_peered":"2024-03-11T21:39:26.602836+0000","last_clean":"2024-03-11T21:39:26.602836+0000","last_became_active":"2024-03-11T21:34:41.134987+0000","last_became_peered":"2024-03-11T21:34:41.134987+0000","last_unstale":"2024-03-11T21:39:26.602836+0000","last_undegraded":"2024-03-11T21:39:26.602836+0000","last_fullsized":"2024-03-11T21:39:26.602836+0000","mapping_epoch":226,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":227,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:36:16.719309+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:36:16.719309+0000","last_clean_scrub_stamp":"2024-03-11T21:36:16.719309+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,5],"acting":[7,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.d","version":"260'39","reported_seq":638,"reported_epoch":331,"state":"active+clean","last_fresh":"2024-03-11T21:39:26.586810+0000","last_change":"2024-03-11T21:35:25.543769+0000","last_active":"2024-03-11T21:39:26.586810+0000","last_peered":"2024-03-11T21:39:26.586810+0000","last_clean":"2024-03-11T21:39:26.586810+0000","last_became_active":"2024-03-11T21:25:40.553454+0000","last_became_peered":"2024-03-11T21:25:40.553454+0000","last_unstale":"2024-03-11T21:39:26.586810+0000","last_undegraded":"2024-03-11T21:39:26.586810+0000","last_fullsized":"2024-03-11T21:39:26.586810+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"165'25","last_scrub_stamp":"2024-03-11T21:35:25.543628+0000","last_deep_scrub":"165'25","last_deep_scrub_stamp":"2024-03-11T21:32:21.678328+0000","last_clean_scrub_stamp":"2024-03-11T21:35:25.543628+0000","objects_scrubbed":8,"log_size":39,"log_dups_size":0,"ondisk_log_size":39,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":50,"num_read_kb":40,"num_write":32,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,1],"acting":[4,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.7","version":"331'13141","reported_seq":22807,"reported_epoch":331,"state":"active+clean","last_fresh":"2024-03-11T21:39:27.486162+0000","last_change":"2024-03-11T21:38:58.387549+0000","last_active":"2024-03-11T21:39:27.486162+0000","last_peered":"2024-03-11T21:39:27.486162+0000","last_clean":"2024-03-11T21:39:27.486162+0000","last_became_active":"2024-03-11T21:38:58.365063+0000","last_became_peered":"2024-03-11T21:38:58.365063+0000","last_unstale":"2024-03-11T21:39:27.486162+0000","last_undegraded":"2024-03-11T21:39:27.486162+0000","last_fullsized":"2024-03-11T21:39:27.486162+0000","mapping_epoch":327,"log_start":"300'7900","ondisk_log_start":"300'7900","created":259,"last_epoch_clean":328,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:35:44.669493+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:35:44.669493+0000","last_clean_scrub_stamp":"2024-03-11T21:35:44.669493+0000","objects_scrubbed":0,"log_size":5241,"log_dups_size":0,"ondisk_log_size":5241,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90898432,"num_objects":1387,"num_object_clones":0,"num_object_copies":4161,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1387,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13141,"num_write_kb":464896,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4752,"num_bytes_recovered":311427072,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":229,"reported_epoch":331,"state":"active+clean","last_fresh":"2024-03-11T21:39:26.601031+0000","last_change":"2024-03-11T21:36:06.735035+0000","last_active":"2024-03-11T21:39:26.601031+0000","last_peered":"2024-03-11T21:39:26.601031+0000","last_clean":"2024-03-11T21:39:26.601031+0000","last_became_active":"2024-03-11T21:34:41.145012+0000","last_became_peered":"2024-03-11T21:34:41.145012+0000","last_unstale":"2024-03-11T21:39:26.601031+0000","last_undegraded":"2024-03-11T21:39:26.601031+0000","last_fullsized":"2024-03-11T21:39:26.601031+0000","mapping_epoch":226,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":227,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:36:06.734921+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:28:37.561826+0000","last_clean_scrub_stamp":"2024-03-11T21:36:06.734921+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":23,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,0],"acting":[7,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":576,"reported_epoch":331,"state":"active+clean","last_fresh":"2024-03-11T21:39:26.625021+0000","last_change":"2024-03-11T21:35:14.636561+0000","last_active":"2024-03-11T21:39:26.625021+0000","last_peered":"2024-03-11T21:39:26.625021+0000","last_clean":"2024-03-11T21:39:26.625021+0000","last_became_active":"2024-03-11T21:25:42.574515+0000","last_became_peered":"2024-03-11T21:25:42.574515+0000","last_unstale":"2024-03-11T21:39:26.625021+0000","last_undegraded":"2024-03-11T21:39:26.625021+0000","last_fullsized":"2024-03-11T21:39:26.625021+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:35:14.636450+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:41.544550+0000","last_clean_scrub_stamp":"2024-03-11T21:35:14.636450+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,5],"acting":[2,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.c","version":"0'0","reported_seq":584,"reported_epoch":331,"state":"active+clean","last_fresh":"2024-03-11T21:39:26.625442+0000","last_change":"2024-03-11T21:36:00.849272+0000","last_active":"2024-03-11T21:39:26.625442+0000","last_peered":"2024-03-11T21:39:26.625442+0000","last_clean":"2024-03-11T21:39:26.625442+0000","last_became_active":"2024-03-11T21:25:38.533240+0000","last_became_peered":"2024-03-11T21:25:38.533240+0000","last_unstale":"2024-03-11T21:39:26.625442+0000","last_undegraded":"2024-03-11T21:39:26.625442+0000","last_fullsized":"2024-03-11T21:39:26.625442+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:36:00.849153+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:29:30.279910+0000","last_clean_scrub_stamp":"2024-03-11T21:36:00.849153+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,0],"acting":[2,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.a","version":"260'44","reported_seq":713,"reported_epoch":331,"state":"active+clean","last_fresh":"2024-03-11T21:39:26.614322+0000","last_change":"2024-03-11T21:38:57.101011+0000","last_active":"2024-03-11T21:39:26.614322+0000","last_peered":"2024-03-11T21:39:26.614322+0000","last_clean":"2024-03-11T21:39:26.614322+0000","last_became_active":"2024-03-11T21:34:41.148863+0000","last_became_peered":"2024-03-11T21:34:41.148863+0000","last_unstale":"2024-03-11T21:39:26.614322+0000","last_undegraded":"2024-03-11T21:39:26.614322+0000","last_fullsized":"2024-03-11T21:39:26.614322+0000","mapping_epoch":226,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":227,"parent":"0.0","parent_split_bits":0,"last_scrub":"260'44","last_scrub_stamp":"2024-03-11T21:38:57.100873+0000","last_deep_scrub":"109'28","last_deep_scrub_stamp":"2024-03-11T21:28:24.016389+0000","last_clean_scrub_stamp":"2024-03-11T21:38:57.100873+0000","objects_scrubbed":9,"log_size":44,"log_dups_size":0,"ondisk_log_size":44,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:40:01.805926+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":56,"num_read_kb":45,"num_write":36,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,7],"acting":[6,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.6","version":"331'13162","reported_seq":13317,"reported_epoch":331,"state":"active+clean","last_fresh":"2024-03-11T21:39:27.434356+0000","last_change":"2024-03-11T21:35:45.703533+0000","last_active":"2024-03-11T21:39:27.434356+0000","last_peered":"2024-03-11T21:39:27.434356+0000","last_clean":"2024-03-11T21:39:27.434356+0000","last_became_active":"2024-03-11T21:35:45.703184+0000","last_became_peered":"2024-03-11T21:35:45.703184+0000","last_unstale":"2024-03-11T21:39:27.434356+0000","last_undegraded":"2024-03-11T21:39:27.434356+0000","last_fullsized":"2024-03-11T21:39:27.434356+0000","mapping_epoch":259,"log_start":"301'7900","ondisk_log_start":"301'7900","created":259,"last_epoch_clean":260,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:35:44.669493+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:35:44.669493+0000","last_clean_scrub_stamp":"2024-03-11T21:35:44.669493+0000","objects_scrubbed":0,"log_size":5262,"log_dups_size":0,"ondisk_log_size":5262,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90701824,"num_objects":1384,"num_object_clones":0,"num_object_copies":4152,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1384,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13162,"num_write_kb":465472,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,1],"acting":[5,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":587,"reported_epoch":332,"state":"active+clean","last_fresh":"2024-03-11T21:39:27.603755+0000","last_change":"2024-03-11T21:35:37.687281+0000","last_active":"2024-03-11T21:39:27.603755+0000","last_peered":"2024-03-11T21:39:27.603755+0000","last_clean":"2024-03-11T21:39:27.603755+0000","last_became_active":"2024-03-11T21:34:41.154507+0000","last_became_peered":"2024-03-11T21:34:41.154507+0000","last_unstale":"2024-03-11T21:39:27.603755+0000","last_undegraded":"2024-03-11T21:39:27.603755+0000","last_fullsized":"2024-03-11T21:39:27.603755+0000","mapping_epoch":226,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":227,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:35:37.687187+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:27:11.361913+0000","last_clean_scrub_stamp":"2024-03-11T21:35:37.687187+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":39,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":577,"reported_epoch":331,"state":"active+clean","last_fresh":"2024-03-11T21:39:26.625299+0000","last_change":"2024-03-11T21:36:35.059662+0000","last_active":"2024-03-11T21:39:26.625299+0000","last_peered":"2024-03-11T21:39:26.625299+0000","last_clean":"2024-03-11T21:39:26.625299+0000","last_became_active":"2024-03-11T21:36:35.059192+0000","last_became_peered":"2024-03-11T21:36:35.059192+0000","last_unstale":"2024-03-11T21:39:26.625299+0000","last_undegraded":"2024-03-11T21:39:26.625299+0000","last_fullsized":"2024-03-11T21:39:26.625299+0000","mapping_epoch":277,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":278,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:34:17.412687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:41.544550+0000","last_clean_scrub_stamp":"2024-03-11T21:34:17.412687+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,3],"acting":[2,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.d","version":"0'0","reported_seq":588,"reported_epoch":332,"state":"active+clean","last_fresh":"2024-03-11T21:39:27.597622+0000","last_change":"2024-03-11T21:36:35.058635+0000","last_active":"2024-03-11T21:39:27.597622+0000","last_peered":"2024-03-11T21:39:27.597622+0000","last_clean":"2024-03-11T21:39:27.597622+0000","last_became_active":"2024-03-11T21:36:35.058296+0000","last_became_peered":"2024-03-11T21:36:35.058296+0000","last_unstale":"2024-03-11T21:39:27.597622+0000","last_undegraded":"2024-03-11T21:39:27.597622+0000","last_fullsized":"2024-03-11T21:39:27.597622+0000","mapping_epoch":277,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":278,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:31:55.464145+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:37.512018+0000","last_clean_scrub_stamp":"2024-03-11T21:31:55.464145+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,3],"acting":[1,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.b","version":"260'15","reported_seq":608,"reported_epoch":332,"state":"active+clean","last_fresh":"2024-03-11T21:39:27.606778+0000","last_change":"2024-03-11T21:35:27.760318+0000","last_active":"2024-03-11T21:39:27.606778+0000","last_peered":"2024-03-11T21:39:27.606778+0000","last_clean":"2024-03-11T21:39:27.606778+0000","last_became_active":"2024-03-11T21:25:40.549612+0000","last_became_peered":"2024-03-11T21:25:40.549612+0000","last_unstale":"2024-03-11T21:39:27.606778+0000","last_undegraded":"2024-03-11T21:39:27.606778+0000","last_fullsized":"2024-03-11T21:39:27.606778+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'9","last_scrub_stamp":"2024-03-11T21:35:27.760208+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:39.525554+0000","last_clean_scrub_stamp":"2024-03-11T21:35:27.760208+0000","objects_scrubbed":4,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.5","version":"332'13437","reported_seq":13594,"reported_epoch":332,"state":"active+clean","last_fresh":"2024-03-11T21:39:28.732925+0000","last_change":"2024-03-11T21:35:45.697209+0000","last_active":"2024-03-11T21:39:28.732925+0000","last_peered":"2024-03-11T21:39:28.732925+0000","last_clean":"2024-03-11T21:39:28.732925+0000","last_became_active":"2024-03-11T21:35:45.696788+0000","last_became_peered":"2024-03-11T21:35:45.696788+0000","last_unstale":"2024-03-11T21:39:28.732925+0000","last_undegraded":"2024-03-11T21:39:28.732925+0000","last_fullsized":"2024-03-11T21:39:28.732925+0000","mapping_epoch":259,"log_start":"298'7800","ondisk_log_start":"298'7800","created":259,"last_epoch_clean":260,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:35:44.669493+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:35:44.669493+0000","last_clean_scrub_stamp":"2024-03-11T21:35:44.669493+0000","objects_scrubbed":0,"log_size":5637,"log_dups_size":0,"ondisk_log_size":5637,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":89456640,"num_objects":1365,"num_object_clones":0,"num_object_copies":4095,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1365,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13437,"num_write_kb":473664,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":486,"reported_epoch":331,"state":"active+clean","last_fresh":"2024-03-11T21:39:26.584705+0000","last_change":"2024-03-11T21:36:35.057118+0000","last_active":"2024-03-11T21:39:26.584705+0000","last_peered":"2024-03-11T21:39:26.584705+0000","last_clean":"2024-03-11T21:39:26.584705+0000","last_became_active":"2024-03-11T21:36:35.056518+0000","last_became_peered":"2024-03-11T21:36:35.056518+0000","last_unstale":"2024-03-11T21:39:26.584705+0000","last_undegraded":"2024-03-11T21:39:26.584705+0000","last_fullsized":"2024-03-11T21:39:26.584705+0000","mapping_epoch":277,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":278,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:35:24.514584+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:43.559452+0000","last_clean_scrub_stamp":"2024-03-11T21:35:24.514584+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,3],"acting":[4,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.e","version":"0'0","reported_seq":605,"reported_epoch":331,"state":"active+clean","last_fresh":"2024-03-11T21:39:26.628199+0000","last_change":"2024-03-11T21:36:35.057187+0000","last_active":"2024-03-11T21:39:26.628199+0000","last_peered":"2024-03-11T21:39:26.628199+0000","last_clean":"2024-03-11T21:39:26.628199+0000","last_became_active":"2024-03-11T21:36:35.056777+0000","last_became_peered":"2024-03-11T21:36:35.056777+0000","last_unstale":"2024-03-11T21:39:26.628199+0000","last_undegraded":"2024-03-11T21:39:26.628199+0000","last_fullsized":"2024-03-11T21:39:26.628199+0000","mapping_epoch":277,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":278,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:34:19.476172+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:37.512018+0000","last_clean_scrub_stamp":"2024-03-11T21:34:19.476172+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":17,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,7],"acting":[2,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.8","version":"260'34","reported_seq":674,"reported_epoch":331,"state":"active+clean","last_fresh":"2024-03-11T21:39:26.602305+0000","last_change":"2024-03-11T21:35:20.304345+0000","last_active":"2024-03-11T21:39:26.602305+0000","last_peered":"2024-03-11T21:39:26.602305+0000","last_clean":"2024-03-11T21:39:26.602305+0000","last_became_active":"2024-03-11T21:34:41.146631+0000","last_became_peered":"2024-03-11T21:34:41.146631+0000","last_unstale":"2024-03-11T21:39:26.602305+0000","last_undegraded":"2024-03-11T21:39:26.602305+0000","last_fullsized":"2024-03-11T21:39:26.602305+0000","mapping_epoch":226,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":227,"parent":"0.0","parent_split_bits":0,"last_scrub":"109'22","last_scrub_stamp":"2024-03-11T21:35:20.304114+0000","last_deep_scrub":"109'22","last_deep_scrub_stamp":"2024-03-11T21:32:07.573197+0000","last_clean_scrub_stamp":"2024-03-11T21:35:20.304114+0000","objects_scrubbed":7,"log_size":34,"log_dups_size":0,"ondisk_log_size":34,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":44,"num_read_kb":35,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.9","version":"229'11","reported_seq":290,"reported_epoch":331,"state":"active+clean","last_fresh":"2024-03-11T21:39:26.601313+0000","last_change":"2024-03-11T21:36:18.726876+0000","last_active":"2024-03-11T21:39:26.601313+0000","last_peered":"2024-03-11T21:39:26.601313+0000","last_clean":"2024-03-11T21:39:26.601313+0000","last_became_active":"2024-03-11T21:34:41.156892+0000","last_became_peered":"2024-03-11T21:34:41.156892+0000","last_unstale":"2024-03-11T21:39:26.601313+0000","last_undegraded":"2024-03-11T21:39:26.601313+0000","last_fullsized":"2024-03-11T21:39:26.601313+0000","mapping_epoch":226,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":227,"parent":"0.0","parent_split_bits":0,"last_scrub":"229'11","last_scrub_stamp":"2024-03-11T21:36:18.726763+0000","last_deep_scrub":"94'9","last_deep_scrub_stamp":"2024-03-11T21:26:53.422552+0000","last_clean_scrub_stamp":"2024-03-11T21:36:18.726763+0000","objects_scrubbed":1,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":9,"num_pg":16,"stat_sum":{"num_bytes":1421344792,"num_objects":21689,"num_object_clones":0,"num_object_copies":65067,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19530,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21689,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":211075,"num_write_kb":7448385,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18493,"num_bytes_recovered":1089798144,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":4294914048,"data_stored":4294901832,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":82436,"ondisk_log_size":82436,"up":48,"acting":43,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":1531,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":17,"num_read_kb":17,"num_write":10,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":1308,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":4593,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":6,"ondisk_log_size":6,"up":96,"acting":96,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":198,"num_read_kb":294,"num_write":179,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":2951840,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":2375680,"data_stored":2361472,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":128,"ondisk_log_size":128,"up":3,"acting":3,"num_store_stats":4},{"poolid":3,"num_pg":3,"stat_sum":{"num_bytes":408,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":822,"num_read_kb":817,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1224,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":8,"ondisk_log_size":8,"up":9,"acting":9,"num_store_stats":7},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":3702,"num_objects":178,"num_object_clones":0,"num_object_copies":534,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":178,"num_whiteouts":0,"num_read":1120,"num_read_kb":877,"num_write":703,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":186,"num_bytes_recovered":2200,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":417792,"data_stored":11106,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":760,"ondisk_log_size":760,"up":96,"acting":96,"num_store_stats":8},{"poolid":5,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":78,"ondisk_log_size":78,"up":96,"acting":96,"num_store_stats":8},{"poolid":6,"num_pg":32,"stat_sum":{"num_bytes":383,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":370,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1149,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":96,"acting":96,"num_store_stats":8}],"osd_stats":[{"osd":7,"up_from":60,"seq":257698037929,"num_pgs":65,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1322912,"kb_used_data":836376,"kb_used_omap":1,"kb_used_meta":486526,"kb_avail":92422240,"statfs":{"total":95995035648,"available":94640373760,"internally_reserved":0,"allocated":856449024,"data_stored":855608820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":498203082},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,1,0,1],"upper_bound":32},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":50,"seq":214748364975,"num_pgs":46,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":663916,"kb_used_data":271776,"kb_used_omap":1,"kb_used_meta":392126,"kb_avail":93081236,"statfs":{"total":95995035648,"available":95315185664,"internally_reserved":0,"allocated":278298624,"data_stored":277458130,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":401537478},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1],"upper_bound":8},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834575049,"num_pgs":60,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1368520,"kb_used_data":800060,"kb_used_omap":1,"kb_used_meta":568446,"kb_avail":92376632,"statfs":{"total":95995035648,"available":94593671168,"internally_reserved":0,"allocated":819261440,"data_stored":818391752,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":582089157},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,0,1,1],"upper_bound":32},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738574,"num_pgs":54,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":903364,"kb_used_data":344632,"kb_used_omap":1,"kb_used_meta":558718,"kb_avail":92841788,"statfs":{"total":95995035648,"available":95069990912,"internally_reserved":0,"allocated":352903168,"data_stored":352045329,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":572127685},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,1,0,3,1,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411523,"num_pgs":43,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":706276,"kb_used_data":269720,"kb_used_omap":1,"kb_used_meta":436542,"kb_avail":93038876,"statfs":{"total":95995035648,"available":95271809024,"internally_reserved":0,"allocated":276193280,"data_stored":275360202,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":447019461},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1],"upper_bound":8},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":277,"seq":1189705941028,"num_pgs":62,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":909392,"kb_used_data":668872,"kb_used_omap":4,"kb_used_meta":240507,"kb_avail":92835760,"statfs":{"total":95995035648,"available":95063818240,"internally_reserved":0,"allocated":684924928,"data_stored":684050569,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4442,"internal_metadata":246279846},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,0,1,2,3],"upper_bound":64},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":4,"up_from":32,"seq":137438953656,"num_pgs":56,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":696632,"kb_used_data":278124,"kb_used_omap":1,"kb_used_meta":418494,"kb_avail":93048520,"statfs":{"total":95995035648,"available":95281684480,"internally_reserved":0,"allocated":284798976,"data_stored":283945266,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":428538309},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1],"upper_bound":2},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":41,"seq":176093659315,"num_pgs":58,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1341864,"kb_used_data":809500,"kb_used_omap":1,"kb_used_meta":532350,"kb_avail":92403288,"statfs":{"total":95995035648,"available":94620966912,"internally_reserved":0,"allocated":828928000,"data_stored":828090465,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":545126853},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,1,1],"upper_bound":32},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1049,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":46,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":436,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":957,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":574,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":436,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1003,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":92,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":1430,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":77824,"data_stored":2008,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":1018,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":1898,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":1474,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":990,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":1034,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":1254,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":370,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":383,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":370,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":350027776,"data_stored":350027776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":796463104,"data_stored":796459032,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":274268160,"data_stored":274268160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":680853504,"data_stored":680853504,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":276168704,"data_stored":276168704,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":820580352,"data_stored":820576280,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":272302080,"data_stored":272302080,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":824250368,"data_stored":824246296,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-11T21:39:35.293 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:39:35.342 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:39:35.343 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:39:35.343 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:39:35.444 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:39:35.474 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:39:35.474 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:39:35.475 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:39:35.575 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:39:35.604 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:39:35.604 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:39:35.605 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:39:35.705 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:39:35.734 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:39:35.734 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:39:35.735 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:39:35.837 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:39:35.876 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:39:35.876 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:39:35.877 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:39:35.977 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:39:36.005 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:39:36.006 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:39:36.006 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:39:36.078 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:39:35 smithi097 ceph-mon[33317]: from='mgr.14607 v1:172.21.15.92:0/1702717597' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T21:39:36.079 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:39:35 smithi097 ceph-mon[33317]: from='mgr.14607 v1:172.21.15.92:0/1702717597' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T21:39:36.079 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:39:35 smithi097 ceph-mon[33317]: from='mgr.14607 ' entity='mgr.y' 2024-03-11T21:39:36.107 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:39:36.115 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:39:35 smithi092 ceph-mon[29572]: from='mgr.14607 v1:172.21.15.92:0/1702717597' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T21:39:36.115 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:39:35 smithi092 ceph-mon[29572]: from='mgr.14607 v1:172.21.15.92:0/1702717597' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T21:39:36.115 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:39:35 smithi092 ceph-mon[29572]: from='mgr.14607 ' entity='mgr.y' 2024-03-11T21:39:36.115 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:39:35 smithi092 ceph-mon[37563]: from='mgr.14607 v1:172.21.15.92:0/1702717597' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T21:39:36.115 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:39:35 smithi092 ceph-mon[37563]: from='mgr.14607 v1:172.21.15.92:0/1702717597' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T21:39:36.115 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:39:35 smithi092 ceph-mon[37563]: from='mgr.14607 ' entity='mgr.y' 2024-03-11T21:39:36.178 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:39:36.179 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:39:36.180 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:39:36.280 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:39:36.331 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:39:36.331 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:39:36.331 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:39:36.432 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:39:36.470 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:39:36.470 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:39:36.471 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:39:36.571 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:39:36.615 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:39:36.615 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:39:36.615 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:39:36.716 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:39:36.742 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:39:36.743 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:39:36.743 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:39:36.844 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:39:36.873 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:39:36.874 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:39:36.874 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:39:36.975 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:39:37.016 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:39:37.017 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:39:37.017 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:39:37.019 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:39:36 smithi092 ceph-mon[29572]: pgmap v647: 148 pgs: 2 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+degraded+remapped, 143 active+clean; 1.1 GiB data, 6.8 GiB used, 708 GiB / 715 GiB avail; 1.3 KiB/s rd, 0 B/s wr, 859 op/s; 19114/53226 objects degraded (35.911%); 0 B/s, 111 objects/s recovering 2024-03-11T21:39:37.019 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:39:36 smithi092 ceph-mon[37563]: pgmap v647: 148 pgs: 2 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+degraded+remapped, 143 active+clean; 1.1 GiB data, 6.8 GiB used, 708 GiB / 715 GiB avail; 1.3 KiB/s rd, 0 B/s wr, 859 op/s; 19114/53226 objects degraded (35.911%); 0 B/s, 111 objects/s recovering 2024-03-11T21:39:37.078 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:39:36 smithi097 ceph-mon[33317]: pgmap v647: 148 pgs: 2 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+degraded+remapped, 143 active+clean; 1.1 GiB data, 6.8 GiB used, 708 GiB / 715 GiB avail; 1.3 KiB/s rd, 0 B/s wr, 859 op/s; 19114/53226 objects degraded (35.911%); 0 B/s, 111 objects/s recovering 2024-03-11T21:39:37.118 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:39:37.165 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:39:37.165 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:39:37.166 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:39:37.267 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:39:37.298 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:39:37.299 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:39:37.299 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:39:37.400 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:39:37.430 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:39:37.430 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:39:37.431 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:39:37.531 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:39:37.561 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:39:37.561 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:39:37.562 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:39:37.663 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:39:37.691 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:39:37.691 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:39:37.692 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:39:37.793 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:39:37.821 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:39:37.822 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:39:37.822 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:39:37.863 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:39:37 smithi092 ceph-mon[29572]: from='mgr.14607 v1:172.21.15.92:0/1702717597' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T21:39:37.863 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:39:37 smithi092 ceph-mon[37563]: from='mgr.14607 v1:172.21.15.92:0/1702717597' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T21:39:37.923 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:39:37.964 INFO:teuthology.orchestra.run.smithi092.stdout:osd.0: {} 2024-03-11T21:39:37.983 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:39:37.983 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:39:37.984 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:39:38.012 INFO:teuthology.orchestra.run.smithi092.stdout:osd.1: {} 2024-03-11T21:39:38.057 INFO:teuthology.orchestra.run.smithi092.stdout:osd.2: {} 2024-03-11T21:39:38.078 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:39:37 smithi097 ceph-mon[33317]: from='mgr.14607 v1:172.21.15.92:0/1702717597' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T21:39:38.085 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:39:38.106 INFO:teuthology.orchestra.run.smithi092.stdout:osd.3: {} 2024-03-11T21:39:38.111 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:39:38.111 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:39:38.112 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:39:38.118 INFO:teuthology.orchestra.run.smithi092.stderr:Inferring config /var/lib/ceph/00db0016-dfed-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T21:39:38.156 INFO:teuthology.orchestra.run.smithi092.stdout:osd.4: {} 2024-03-11T21:39:38.205 INFO:teuthology.orchestra.run.smithi092.stdout:osd.5: {} 2024-03-11T21:39:38.207 INFO:journalctl@ceph.mgr.y.smithi092.stdout:Mar 11 21:39:37 smithi092 ceph-00db0016-dfed-11ee-95c8-87774f69a715-mgr-y[29801]: ::ffff:172.21.15.97 - - [11/Mar/2024:21:39:37] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-11T21:39:38.213 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:39:38.239 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:39:38.240 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:39:38.241 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:39:38.251 INFO:teuthology.orchestra.run.smithi092.stdout:osd.6: {} 2024-03-11T21:39:38.280 DEBUG:teuthology.orchestra.run.smithi092:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid 00db0016-dfed-11ee-95c8-87774f69a715 -- ceph pg dump --format=json 2024-03-11T21:39:38.300 INFO:teuthology.orchestra.run.smithi092.stdout:osd.7: {} 2024-03-11T21:39:38.312 INFO:teuthology.orchestra.run.smithi092.stderr:osd.0: osd_enable_op_tracker = '' 2024-03-11T21:39:38.312 INFO:teuthology.orchestra.run.smithi092.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-11T21:39:38.312 INFO:teuthology.orchestra.run.smithi092.stderr:osd.2: osd_enable_op_tracker = '' 2024-03-11T21:39:38.313 INFO:teuthology.orchestra.run.smithi092.stderr:osd.3: osd_enable_op_tracker = '' 2024-03-11T21:39:38.313 INFO:teuthology.orchestra.run.smithi092.stderr:osd.4: osd_enable_op_tracker = '' 2024-03-11T21:39:38.313 INFO:teuthology.orchestra.run.smithi092.stderr:osd.5: osd_enable_op_tracker = '' 2024-03-11T21:39:38.313 INFO:teuthology.orchestra.run.smithi092.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-11T21:39:38.313 INFO:teuthology.orchestra.run.smithi092.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-11T21:39:38.341 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:39:38.367 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:39:38.367 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:39:38.368 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:39:38.468 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:39:38.496 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:39:38.496 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:39:38.496 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:39:38.597 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:39:38.624 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:39:38.625 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:39:38.625 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:39:38.726 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:39:38.740 INFO:teuthology.orchestra.run.smithi092.stderr:noscrub is unset 2024-03-11T21:39:38.754 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:39:38.755 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:39:38.755 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:39:38.856 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:39:38.886 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:39:38.886 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:39:38.887 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:39:38.988 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:39:39.015 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:39:39.016 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:39:39.016 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:39:39.079 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:39:38 smithi097 ceph-mon[33317]: pgmap v648: 148 pgs: 2 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+degraded+remapped, 143 active+clean; 1.1 GiB data, 6.8 GiB used, 708 GiB / 715 GiB avail; 638 B/s rd, 0 B/s wr, 517 op/s; 19114/53226 objects degraded (35.911%); 0 B/s, 51 objects/s recovering 2024-03-11T21:39:39.079 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:39:38 smithi097 ceph-mon[33317]: from='client.? v1:172.21.15.92:0/899044086' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-11T21:39:39.079 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:39:38 smithi097 ceph-mon[33317]: Health check update: Degraded data redundancy: 19114/53226 objects degraded (35.911%), 5 pgs degraded, 5 pgs undersized (PG_DEGRADED) 2024-03-11T21:39:39.079 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:39:38 smithi097 ceph-mon[33317]: from='client.? v1:172.21.15.92:0/2428957595' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-11T21:39:39.079 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:39:38 smithi097 ceph-mon[33317]: from='client.35507 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-11T21:39:39.113 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:39:38 smithi092 ceph-mon[29572]: pgmap v648: 148 pgs: 2 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+degraded+remapped, 143 active+clean; 1.1 GiB data, 6.8 GiB used, 708 GiB / 715 GiB avail; 638 B/s rd, 0 B/s wr, 517 op/s; 19114/53226 objects degraded (35.911%); 0 B/s, 51 objects/s recovering 2024-03-11T21:39:39.113 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:39:38 smithi092 ceph-mon[29572]: from='client.? v1:172.21.15.92:0/899044086' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-11T21:39:39.113 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:39:38 smithi092 ceph-mon[29572]: Health check update: Degraded data redundancy: 19114/53226 objects degraded (35.911%), 5 pgs degraded, 5 pgs undersized (PG_DEGRADED) 2024-03-11T21:39:39.114 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:39:38 smithi092 ceph-mon[29572]: from='client.? v1:172.21.15.92:0/2428957595' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-11T21:39:39.114 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:39:38 smithi092 ceph-mon[29572]: from='client.35507 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-11T21:39:39.114 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:39:38 smithi092 ceph-mon[37563]: pgmap v648: 148 pgs: 2 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+degraded+remapped, 143 active+clean; 1.1 GiB data, 6.8 GiB used, 708 GiB / 715 GiB avail; 638 B/s rd, 0 B/s wr, 517 op/s; 19114/53226 objects degraded (35.911%); 0 B/s, 51 objects/s recovering 2024-03-11T21:39:39.114 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:39:38 smithi092 ceph-mon[37563]: from='client.? v1:172.21.15.92:0/899044086' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-11T21:39:39.114 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:39:38 smithi092 ceph-mon[37563]: Health check update: Degraded data redundancy: 19114/53226 objects degraded (35.911%), 5 pgs degraded, 5 pgs undersized (PG_DEGRADED) 2024-03-11T21:39:39.114 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:39:38 smithi092 ceph-mon[37563]: from='client.? v1:172.21.15.92:0/2428957595' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-11T21:39:39.114 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:39:38 smithi092 ceph-mon[37563]: from='client.35507 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-11T21:39:39.118 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:39:39.147 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:39:39.148 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:39:39.148 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:39:39.249 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:39:39.290 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:39:39.290 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:39:39.291 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:39:39.391 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:39:39.433 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:39:39.433 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:39:39.433 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:39:39.534 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:39:39.576 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:39:39.577 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:39:39.577 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:39:39.678 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:39:39.709 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:39:39.709 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:39:39.710 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:39:39.810 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:39:39.840 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:39:39.841 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:39:39.841 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:39:39.863 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:39:39 smithi092 ceph-mon[29572]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-11T21:39:39.863 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:39:39 smithi092 ceph-mon[29572]: from='client.35507 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-03-11T21:39:39.863 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:39:39 smithi092 ceph-mon[29572]: osdmap e335: 8 total, 8 up, 8 in 2024-03-11T21:39:39.863 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:39:39 smithi092 ceph-mon[37563]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-11T21:39:39.863 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:39:39 smithi092 ceph-mon[37563]: from='client.35507 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-03-11T21:39:39.863 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:39:39 smithi092 ceph-mon[37563]: osdmap e335: 8 total, 8 up, 8 in 2024-03-11T21:39:39.942 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:39:39.995 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:39:39.995 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:39:39.996 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:39:40.079 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:39:39 smithi097 ceph-mon[33317]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-11T21:39:40.079 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:39:39 smithi097 ceph-mon[33317]: from='client.35507 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-03-11T21:39:40.079 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:39:39 smithi097 ceph-mon[33317]: osdmap e335: 8 total, 8 up, 8 in 2024-03-11T21:39:40.096 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:39:40.125 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:39:40.126 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:39:40.126 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:39:40.227 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:39:40.255 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:39:40.256 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:39:40.256 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:39:40.357 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:39:40.403 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:39:40.404 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:39:40.404 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:39:40.505 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:39:40.546 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:39:40.547 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:39:40.548 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:39:40.648 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:39:40.693 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:39:40.693 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:39:40.694 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:39:40.795 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:39:40.836 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:39:40.837 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:39:40.837 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:39:40.938 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:39:40.969 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:39:40.969 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:39:40.970 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:39:41.071 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:39:41.078 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:39:40 smithi097 ceph-mon[33317]: pgmap v650: 148 pgs: 2 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+degraded+remapped, 143 active+clean; 836 MiB data, 6.0 GiB used, 709 GiB / 715 GiB avail; 1.3 KiB/s rd, 0 B/s wr, 1.09k op/s; 18672/40680 objects degraded (45.900%); 0 B/s, 111 objects/s recovering 2024-03-11T21:39:41.110 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:39:41.110 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:39:41.110 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:39:41.113 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:39:40 smithi092 ceph-mon[29572]: pgmap v650: 148 pgs: 2 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+degraded+remapped, 143 active+clean; 836 MiB data, 6.0 GiB used, 709 GiB / 715 GiB avail; 1.3 KiB/s rd, 0 B/s wr, 1.09k op/s; 18672/40680 objects degraded (45.900%); 0 B/s, 111 objects/s recovering 2024-03-11T21:39:41.113 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:39:40 smithi092 ceph-mon[37563]: pgmap v650: 148 pgs: 2 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+degraded+remapped, 143 active+clean; 836 MiB data, 6.0 GiB used, 709 GiB / 715 GiB avail; 1.3 KiB/s rd, 0 B/s wr, 1.09k op/s; 18672/40680 objects degraded (45.900%); 0 B/s, 111 objects/s recovering 2024-03-11T21:39:41.212 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:39:41.238 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:39:41.239 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:39:41.240 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:39:41.341 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:39:41.384 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:39:41.385 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:39:41.385 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:39:41.486 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:39:41.516 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:39:41.517 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:39:41.518 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:39:41.618 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:39:41.658 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:39:41.659 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:39:41.659 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:39:41.760 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:39:41.788 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:39:41.789 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:39:41.790 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:39:41.890 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:39:41.900 INFO:teuthology.orchestra.run.smithi092.stderr:Inferring config /var/lib/ceph/00db0016-dfed-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T21:39:41.917 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:39:41.917 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:39:41.918 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:39:42.019 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:39:42.031 DEBUG:teuthology.orchestra.run.smithi092:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid 00db0016-dfed-11ee-95c8-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-03-11T21:39:42.049 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:39:42.050 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:39:42.050 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:39:42.151 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:39:42.180 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:39:42.180 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:39:42.181 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:39:42.282 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:39:42.305 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:39:42.306 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:39:42.306 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:39:42.407 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:39:42.435 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:39:42.436 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:39:42.436 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:39:42.537 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:39:42.572 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:39:42.573 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:39:42.573 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:39:42.674 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:39:42.703 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:39:42.704 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:39:42.704 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:39:42.805 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:39:42.843 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:39:42.844 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:39:42.844 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:39:42.945 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:39:42.998 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:39:42.998 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:39:42.999 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:39:43.079 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:39:42 smithi097 ceph-mon[33317]: pgmap v651: 148 pgs: 2 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+degraded+remapped, 143 active+clean; 836 MiB data, 6.0 GiB used, 709 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 758 op/s; 18672/40680 objects degraded (45.900%); 0 B/s, 99 objects/s recovering 2024-03-11T21:39:43.079 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:39:42 smithi097 ceph-mon[33317]: 5.1f scrub starts 2024-03-11T21:39:43.079 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:39:42 smithi097 ceph-mon[33317]: 5.1f scrub ok 2024-03-11T21:39:43.080 INFO:journalctl@ceph.iscsi.iscsi.a.smithi097.stdout:Mar 11 21:39:42 smithi097 ceph-00db0016-dfed-11ee-95c8-87774f69a715-iscsi-iscsi-a[60943]: debug there is no tcmu-runner data available 2024-03-11T21:39:43.099 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:39:43.114 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:39:42 smithi092 ceph-mon[29572]: pgmap v651: 148 pgs: 2 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+degraded+remapped, 143 active+clean; 836 MiB data, 6.0 GiB used, 709 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 758 op/s; 18672/40680 objects degraded (45.900%); 0 B/s, 99 objects/s recovering 2024-03-11T21:39:43.114 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:39:42 smithi092 ceph-mon[29572]: 5.1f scrub starts 2024-03-11T21:39:43.114 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:39:42 smithi092 ceph-mon[29572]: 5.1f scrub ok 2024-03-11T21:39:43.114 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:39:42 smithi092 ceph-mon[37563]: pgmap v651: 148 pgs: 2 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+degraded+remapped, 143 active+clean; 836 MiB data, 6.0 GiB used, 709 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 758 op/s; 18672/40680 objects degraded (45.900%); 0 B/s, 99 objects/s recovering 2024-03-11T21:39:43.114 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:39:42 smithi092 ceph-mon[37563]: 5.1f scrub starts 2024-03-11T21:39:43.114 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:39:42 smithi092 ceph-mon[37563]: 5.1f scrub ok 2024-03-11T21:39:43.139 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:39:43.140 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:39:43.140 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:39:43.241 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:39:43.262 DEBUG:teuthology.orchestra.run.smithi092:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid 00db0016-dfed-11ee-95c8-87774f69a715 -- ceph osd unset nodeep-scrub 2024-03-11T21:39:43.267 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:39:43.267 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:39:43.268 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:39:43.368 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:39:43.411 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:39:43.411 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:39:43.411 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:39:43.512 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:39:43.569 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:39:43.569 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:39:43.569 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:39:43.670 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:39:43.714 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:39:43.715 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:39:43.715 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:39:43.817 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:39:43.862 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:39:43.863 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:39:43.863 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:39:43.964 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:39:43.994 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:39:43.995 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:39:43.995 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:39:44.079 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:39:43 smithi097 ceph-mon[33317]: from='client.24469 v1:172.21.15.97:0/404208310' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:39:44.079 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:39:43 smithi097 ceph-mon[33317]: Health check update: Degraded data redundancy: 18672/40680 objects degraded (45.900%), 5 pgs degraded, 5 pgs undersized (PG_DEGRADED) 2024-03-11T21:39:44.079 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:39:43 smithi097 ceph-mon[33317]: osdmap e336: 8 total, 8 up, 8 in 2024-03-11T21:39:44.096 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:39:44.113 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:39:43 smithi092 ceph-mon[37563]: from='client.24469 v1:172.21.15.97:0/404208310' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:39:44.113 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:39:43 smithi092 ceph-mon[37563]: Health check update: Degraded data redundancy: 18672/40680 objects degraded (45.900%), 5 pgs degraded, 5 pgs undersized (PG_DEGRADED) 2024-03-11T21:39:44.113 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:39:43 smithi092 ceph-mon[37563]: osdmap e336: 8 total, 8 up, 8 in 2024-03-11T21:39:44.113 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:39:43 smithi092 ceph-mon[29572]: from='client.24469 v1:172.21.15.97:0/404208310' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:39:44.113 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:39:43 smithi092 ceph-mon[29572]: Health check update: Degraded data redundancy: 18672/40680 objects degraded (45.900%), 5 pgs degraded, 5 pgs undersized (PG_DEGRADED) 2024-03-11T21:39:44.113 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:39:43 smithi092 ceph-mon[29572]: osdmap e336: 8 total, 8 up, 8 in 2024-03-11T21:39:44.129 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:39:44.130 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:39:44.130 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:39:44.231 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:39:44.259 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:39:44.260 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:39:44.260 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:39:44.361 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:39:44.420 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:39:44.420 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:39:44.421 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:39:44.521 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:39:44.550 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:39:44.550 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:39:44.551 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:39:44.652 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:39:44.681 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:39:44.682 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:39:44.682 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:39:44.783 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:39:44.812 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:39:44.813 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:39:44.813 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:39:44.914 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:39:44.955 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:39:44.956 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:39:44.956 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:39:45.057 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:39:45.080 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:39:44 smithi097 ceph-mon[33317]: pgmap v652: 148 pgs: 2 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+degraded+remapped, 143 active+clean; 667 MiB data, 5.5 GiB used, 710 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 844 op/s; 18352/32574 objects degraded (56.339%); 0 B/s, 89 objects/s recovering 2024-03-11T21:39:45.086 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:39:45.086 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:39:45.086 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:39:45.105 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:39:44 smithi092 ceph-mon[37563]: pgmap v652: 148 pgs: 2 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+degraded+remapped, 143 active+clean; 667 MiB data, 5.5 GiB used, 710 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 844 op/s; 18352/32574 objects degraded (56.339%); 0 B/s, 89 objects/s recovering 2024-03-11T21:39:45.105 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:39:44 smithi092 ceph-mon[29572]: pgmap v652: 148 pgs: 2 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+degraded+remapped, 143 active+clean; 667 MiB data, 5.5 GiB used, 710 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 844 op/s; 18352/32574 objects degraded (56.339%); 0 B/s, 89 objects/s recovering 2024-03-11T21:39:45.189 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:39:45.220 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:39:45.221 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:39:45.221 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:39:45.322 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:39:45.343 INFO:teuthology.orchestra.run.smithi092.stdout:ERROR: (22) Invalid argument 2024-03-11T21:39:45.344 INFO:teuthology.orchestra.run.smithi092.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-11T21:39:45.353 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:39:45.354 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:39:45.354 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:39:45.455 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:39:45.487 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:39:45.488 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:39:45.488 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:39:45.589 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:39:45.620 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:39:45.621 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:39:45.621 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:39:45.722 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:39:45.754 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:39:45.754 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:39:45.754 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:39:45.855 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:39:45.895 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:39:45.896 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:39:45.896 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:39:45.997 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:39:46.022 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:39:46.023 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:39:46.024 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:39:46.066 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:39:45 smithi092 ceph-mon[29572]: purged_snaps scrub starts 2024-03-11T21:39:46.066 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:39:45 smithi092 ceph-mon[29572]: purged_snaps scrub ok 2024-03-11T21:39:46.066 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:39:45 smithi092 ceph-mon[29572]: 4.5 scrub starts 2024-03-11T21:39:46.066 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:39:45 smithi092 ceph-mon[29572]: 4.5 scrub ok 2024-03-11T21:39:46.066 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:39:45 smithi092 ceph-mon[37563]: purged_snaps scrub starts 2024-03-11T21:39:46.067 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:39:45 smithi092 ceph-mon[37563]: purged_snaps scrub ok 2024-03-11T21:39:46.067 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:39:45 smithi092 ceph-mon[37563]: 4.5 scrub starts 2024-03-11T21:39:46.067 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:39:45 smithi092 ceph-mon[37563]: 4.5 scrub ok 2024-03-11T21:39:46.079 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:39:45 smithi097 ceph-mon[33317]: purged_snaps scrub starts 2024-03-11T21:39:46.079 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:39:45 smithi097 ceph-mon[33317]: purged_snaps scrub ok 2024-03-11T21:39:46.079 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:39:45 smithi097 ceph-mon[33317]: 4.5 scrub starts 2024-03-11T21:39:46.079 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:39:45 smithi097 ceph-mon[33317]: 4.5 scrub ok 2024-03-11T21:39:46.125 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:39:46.168 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:39:46.168 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:39:46.169 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:39:46.270 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:39:46.311 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:39:46.311 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:39:46.312 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:39:46.412 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:39:46.452 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:39:46.453 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:39:46.453 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:39:46.554 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:39:46.595 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:39:46.596 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:39:46.596 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:39:46.697 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:39:46.733 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:39:46.734 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:39:46.735 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:39:46.835 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:39:46.868 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:39:46.868 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:39:46.868 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:39:46.969 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:39:46.992 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:39:46.993 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:39:46.993 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:39:47.079 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:39:46 smithi097 ceph-mon[33317]: pgmap v654: 148 pgs: 2 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+degraded+remapped, 143 active+clean; 567 MiB data, 5.3 GiB used, 710 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.06k op/s; 18208/27789 objects degraded (65.522%); 0 B/s, 113 objects/s recovering 2024-03-11T21:39:47.079 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:39:46 smithi097 ceph-mon[33317]: osdmap e337: 8 total, 8 up, 8 in 2024-03-11T21:39:47.093 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:39:47.113 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:39:46 smithi092 ceph-mon[29572]: pgmap v654: 148 pgs: 2 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+degraded+remapped, 143 active+clean; 567 MiB data, 5.3 GiB used, 710 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.06k op/s; 18208/27789 objects degraded (65.522%); 0 B/s, 113 objects/s recovering 2024-03-11T21:39:47.113 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:39:46 smithi092 ceph-mon[29572]: osdmap e337: 8 total, 8 up, 8 in 2024-03-11T21:39:47.113 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:39:46 smithi092 ceph-mon[37563]: pgmap v654: 148 pgs: 2 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+degraded+remapped, 143 active+clean; 567 MiB data, 5.3 GiB used, 710 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.06k op/s; 18208/27789 objects degraded (65.522%); 0 B/s, 113 objects/s recovering 2024-03-11T21:39:47.114 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:39:46 smithi092 ceph-mon[37563]: osdmap e337: 8 total, 8 up, 8 in 2024-03-11T21:39:47.132 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:39:47.133 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:39:47.133 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:39:47.234 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:39:47.261 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:39:47.262 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:39:47.262 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:39:47.363 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:39:47.395 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:39:47.396 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:39:47.396 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:39:47.498 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:39:47.540 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:39:47.540 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:39:47.541 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:39:47.642 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:39:47.691 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:39:47.692 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:39:47.692 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:39:47.771 INFO:teuthology.orchestra.run.smithi092.stderr:Inferring config /var/lib/ceph/00db0016-dfed-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T21:39:47.793 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:39:47.820 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:39:47.820 DEBUG:teuthology.orchestra.run.smithi092:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid 00db0016-dfed-11ee-95c8-87774f69a715 -- ceph daemon osd.0 dump_blocked_ops 2024-03-11T21:39:47.821 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:39:47.821 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:39:47.922 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:39:47.949 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:39:47.950 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:39:47.950 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:39:48.051 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:39:48.116 INFO:journalctl@ceph.mgr.y.smithi092.stdout:Mar 11 21:39:47 smithi092 ceph-00db0016-dfed-11ee-95c8-87774f69a715-mgr-y[29801]: ::ffff:172.21.15.97 - - [11/Mar/2024:21:39:47] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-11T21:39:48.123 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:39:48.123 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:39:48.124 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:39:48.204 INFO:teuthology.orchestra.run.smithi092.stderr:Inferring config /var/lib/ceph/00db0016-dfed-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T21:39:48.225 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:39:48.289 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:39:48.290 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:39:48.290 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:39:48.391 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:39:48.416 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:39:48.417 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:39:48.417 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:39:48.518 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:39:48.545 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:39:48.545 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:39:48.546 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:39:48.646 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:39:48.689 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:39:48.690 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:39:48.690 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:39:48.791 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:39:48.821 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:39:48.822 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:39:48.822 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:39:48.915 INFO:teuthology.orchestra.run.smithi092.stdout: 2024-03-11T21:39:48.923 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:39:48.924 INFO:teuthology.orchestra.run.smithi092.stderr:dumped all 2024-03-11T21:39:48.970 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:39:48.971 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:39:48.971 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:39:49.072 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:39:49.079 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:39:48 smithi097 ceph-mon[33317]: pgmap v656: 148 pgs: 2 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+degraded+remapped, 143 active+clean; 567 MiB data, 5.2 GiB used, 710 GiB / 715 GiB avail; 639 B/s rd, 0 B/s wr, 537 op/s; 18208/27789 objects degraded (65.522%); 0 B/s, 57 objects/s recovering 2024-03-11T21:39:49.079 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:39:48 smithi097 ceph-mon[33317]: Health check update: Degraded data redundancy: 18208/27789 objects degraded (65.522%), 5 pgs degraded, 5 pgs undersized (PG_DEGRADED) 2024-03-11T21:39:49.079 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:39:48 smithi097 ceph-mon[33317]: osdmap e338: 8 total, 8 up, 8 in 2024-03-11T21:39:49.100 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:39:49.101 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:39:49.101 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:39:49.113 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:39:48 smithi092 ceph-mon[29572]: pgmap v656: 148 pgs: 2 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+degraded+remapped, 143 active+clean; 567 MiB data, 5.2 GiB used, 710 GiB / 715 GiB avail; 639 B/s rd, 0 B/s wr, 537 op/s; 18208/27789 objects degraded (65.522%); 0 B/s, 57 objects/s recovering 2024-03-11T21:39:49.113 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:39:48 smithi092 ceph-mon[29572]: Health check update: Degraded data redundancy: 18208/27789 objects degraded (65.522%), 5 pgs degraded, 5 pgs undersized (PG_DEGRADED) 2024-03-11T21:39:49.113 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:39:48 smithi092 ceph-mon[29572]: osdmap e338: 8 total, 8 up, 8 in 2024-03-11T21:39:49.113 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:39:48 smithi092 ceph-mon[37563]: pgmap v656: 148 pgs: 2 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+degraded+remapped, 143 active+clean; 567 MiB data, 5.2 GiB used, 710 GiB / 715 GiB avail; 639 B/s rd, 0 B/s wr, 537 op/s; 18208/27789 objects degraded (65.522%); 0 B/s, 57 objects/s recovering 2024-03-11T21:39:49.113 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:39:48 smithi092 ceph-mon[37563]: Health check update: Degraded data redundancy: 18208/27789 objects degraded (65.522%), 5 pgs degraded, 5 pgs undersized (PG_DEGRADED) 2024-03-11T21:39:49.113 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:39:48 smithi092 ceph-mon[37563]: osdmap e338: 8 total, 8 up, 8 in 2024-03-11T21:39:49.202 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:39:49.247 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:39:49.247 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:39:49.248 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:39:49.349 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:39:49.378 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:39:49.379 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:39:49.379 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:39:49.480 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:39:49.519 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:39:49.520 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:39:49.520 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:39:49.621 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:39:49.650 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:39:49.651 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:39:49.651 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:39:49.752 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:39:49.788 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:39:49.789 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:39:49.789 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:39:49.890 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:39:49.918 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:39:49.919 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:39:49.919 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:39:50.020 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:39:50.038 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:39:49 smithi092 ceph-mon[29572]: from='client.15672 v1:172.21.15.92:0/3439413056' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-11T21:39:50.038 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:39:49 smithi092 ceph-mon[37563]: from='client.15672 v1:172.21.15.92:0/3439413056' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-11T21:39:50.067 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:39:50.067 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:39:50.068 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:39:50.079 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:39:49 smithi097 ceph-mon[33317]: from='client.15672 v1:172.21.15.92:0/3439413056' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-11T21:39:50.168 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:39:50.207 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:39:50.207 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:39:50.208 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:39:50.309 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:39:50.353 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:39:50.354 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:39:50.354 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:39:50.455 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:39:50.502 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:39:50.503 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:39:50.503 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:39:50.604 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:39:50.644 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:39:50.645 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:39:50.645 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:39:50.746 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:39:50.788 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:39:50.789 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:39:50.789 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:39:50.890 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:39:50.937 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:39:50.938 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:39:50.938 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:39:51.039 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:39:51.079 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:39:50 smithi097 ceph-mon[33317]: pgmap v658: 148 pgs: 2 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+degraded+remapped, 143 active+clean; 301 MiB data, 4.5 GiB used, 711 GiB / 715 GiB avail; 853 B/s rd, 0 B/s wr, 974 op/s; 17728/15036 objects degraded (117.904%); 0 B/s, 103 objects/s recovering 2024-03-11T21:39:51.096 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:39:51.096 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:39:51.097 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:39:51.114 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:39:50 smithi092 ceph-mon[29572]: pgmap v658: 148 pgs: 2 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+degraded+remapped, 143 active+clean; 301 MiB data, 4.5 GiB used, 711 GiB / 715 GiB avail; 853 B/s rd, 0 B/s wr, 974 op/s; 17728/15036 objects degraded (117.904%); 0 B/s, 103 objects/s recovering 2024-03-11T21:39:51.115 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:39:50 smithi092 ceph-mon[37563]: pgmap v658: 148 pgs: 2 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+degraded+remapped, 143 active+clean; 301 MiB data, 4.5 GiB used, 711 GiB / 715 GiB avail; 853 B/s rd, 0 B/s wr, 974 op/s; 17728/15036 objects degraded (117.904%); 0 B/s, 103 objects/s recovering 2024-03-11T21:39:51.197 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:39:51.241 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:39:51.241 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:39:51.242 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:39:51.343 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:39:51.383 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:39:51.383 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:39:51.384 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:39:51.485 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:39:51.504 INFO:teuthology.orchestra.run.smithi092.stdout:{"pg_ready":true,"pg_map":{"version":657,"stamp":"2024-03-11T21:39:48.377910+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":422975936,"num_objects":6644,"num_object_clones":0,"num_object_copies":19932,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":17862,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6644,"num_whiteouts":0,"num_read":2178,"num_read_kb":2026,"num_write":227219,"num_write_kb":7450846,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20364,"num_bytes_recovered":1092753862,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":83262,"ondisk_log_size":83262,"up":444,"acting":439,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":444,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":749961216,"kb_used":4921720,"kb_used_data":1263908,"kb_used_omap":15,"kb_used_meta":3657328,"kb_avail":745039496,"statfs":{"total":767960285184,"available":762920443904,"internally_reserved":0,"allocated":1294241792,"data_stored":1287210531,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15601,"internal_metadata":3745104655},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-453246976,"num_objects":-6916,"num_object_clones":0,"num_object_copies":-20748,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-810,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-6916,"num_whiteouts":0,"num_read":10,"num_read_kb":10,"num_write":6916,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":810,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"7.069233"},"pg_stats":[{"pgid":"6.1b","version":"0'0","reported_seq":522,"reported_epoch":336,"state":"active+clean","last_fresh":"2024-03-11T21:39:43.429890+0000","last_change":"2024-03-11T21:36:35.074931+0000","last_active":"2024-03-11T21:39:43.429890+0000","last_peered":"2024-03-11T21:39:43.429890+0000","last_clean":"2024-03-11T21:39:43.429890+0000","last_became_active":"2024-03-11T21:36:35.072636+0000","last_became_peered":"2024-03-11T21:36:35.072636+0000","last_unstale":"2024-03-11T21:39:43.429890+0000","last_undegraded":"2024-03-11T21:39:43.429890+0000","last_fullsized":"2024-03-11T21:39:43.429890+0000","mapping_epoch":277,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":278,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:35:07.021520+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:43.559452+0000","last_clean_scrub_stamp":"2024-03-11T21:35:07.021520+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":588,"reported_epoch":337,"state":"active+clean","last_fresh":"2024-03-11T21:39:45.752509+0000","last_change":"2024-03-11T21:36:00.720089+0000","last_active":"2024-03-11T21:39:45.752509+0000","last_peered":"2024-03-11T21:39:45.752509+0000","last_clean":"2024-03-11T21:39:45.752509+0000","last_became_active":"2024-03-11T21:25:42.578893+0000","last_became_peered":"2024-03-11T21:25:42.578893+0000","last_unstale":"2024-03-11T21:39:45.752509+0000","last_undegraded":"2024-03-11T21:39:45.752509+0000","last_fullsized":"2024-03-11T21:39:45.752509+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:36:00.719956+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:41.544550+0000","last_clean_scrub_stamp":"2024-03-11T21:36:00.719956+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1f","version":"0'0","reported_seq":615,"reported_epoch":336,"state":"active+clean","last_fresh":"2024-03-11T21:39:43.382327+0000","last_change":"2024-03-11T21:36:00.843838+0000","last_active":"2024-03-11T21:39:43.382327+0000","last_peered":"2024-03-11T21:39:43.382327+0000","last_clean":"2024-03-11T21:39:43.382327+0000","last_became_active":"2024-03-11T21:34:41.144945+0000","last_became_peered":"2024-03-11T21:34:41.144945+0000","last_unstale":"2024-03-11T21:39:43.382327+0000","last_undegraded":"2024-03-11T21:39:43.382327+0000","last_fullsized":"2024-03-11T21:39:43.382327+0000","mapping_epoch":226,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":227,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:36:00.843718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:37.512018+0000","last_clean_scrub_stamp":"2024-03-11T21:36:00.843718+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,4],"acting":[0,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.19","version":"260'27","reported_seq":569,"reported_epoch":336,"state":"active+clean","last_fresh":"2024-03-11T21:39:43.407299+0000","last_change":"2024-03-11T21:36:35.089901+0000","last_active":"2024-03-11T21:39:43.407299+0000","last_peered":"2024-03-11T21:39:43.407299+0000","last_clean":"2024-03-11T21:39:43.407299+0000","last_became_active":"2024-03-11T21:36:35.067652+0000","last_became_peered":"2024-03-11T21:36:35.067652+0000","last_unstale":"2024-03-11T21:39:43.407299+0000","last_undegraded":"2024-03-11T21:39:43.407299+0000","last_fullsized":"2024-03-11T21:39:43.407299+0000","mapping_epoch":277,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":278,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'15","last_scrub_stamp":"2024-03-11T21:35:22.099374+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:39.525554+0000","last_clean_scrub_stamp":"2024-03-11T21:35:22.099374+0000","objects_scrubbed":0,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":42,"num_read_kb":33,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.1a","version":"0'0","reported_seq":576,"reported_epoch":337,"state":"active+clean","last_fresh":"2024-03-11T21:39:45.752794+0000","last_change":"2024-03-11T21:35:16.410927+0000","last_active":"2024-03-11T21:39:45.752794+0000","last_peered":"2024-03-11T21:39:45.752794+0000","last_clean":"2024-03-11T21:39:45.752794+0000","last_became_active":"2024-03-11T21:25:44.590923+0000","last_became_peered":"2024-03-11T21:25:44.590923+0000","last_unstale":"2024-03-11T21:39:45.752794+0000","last_undegraded":"2024-03-11T21:39:45.752794+0000","last_fullsized":"2024-03-11T21:39:45.752794+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:35:16.410816+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:43.559452+0000","last_clean_scrub_stamp":"2024-03-11T21:35:16.410816+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":24,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,1],"acting":[4,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":600,"reported_epoch":337,"state":"active+clean","last_fresh":"2024-03-11T21:39:45.765655+0000","last_change":"2024-03-11T21:36:15.576138+0000","last_active":"2024-03-11T21:39:45.765655+0000","last_peered":"2024-03-11T21:39:45.765655+0000","last_clean":"2024-03-11T21:39:45.765655+0000","last_became_active":"2024-03-11T21:34:41.154216+0000","last_became_peered":"2024-03-11T21:34:41.154216+0000","last_unstale":"2024-03-11T21:39:45.765655+0000","last_undegraded":"2024-03-11T21:39:45.765655+0000","last_fullsized":"2024-03-11T21:39:45.765655+0000","mapping_epoch":226,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":227,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:36:15.575887+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:41.544550+0000","last_clean_scrub_stamp":"2024-03-11T21:36:15.575887+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,7],"acting":[1,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1e","version":"0'0","reported_seq":535,"reported_epoch":336,"state":"active+clean","last_fresh":"2024-03-11T21:39:43.406722+0000","last_change":"2024-03-11T21:36:35.062413+0000","last_active":"2024-03-11T21:39:43.406722+0000","last_peered":"2024-03-11T21:39:43.406722+0000","last_clean":"2024-03-11T21:39:43.406722+0000","last_became_active":"2024-03-11T21:36:35.061786+0000","last_became_peered":"2024-03-11T21:36:35.061786+0000","last_unstale":"2024-03-11T21:39:43.406722+0000","last_undegraded":"2024-03-11T21:39:43.406722+0000","last_fullsized":"2024-03-11T21:39:43.406722+0000","mapping_epoch":277,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":278,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:32:18.578038+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:37.512018+0000","last_clean_scrub_stamp":"2024-03-11T21:32:18.578038+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.18","version":"260'15","reported_seq":625,"reported_epoch":337,"state":"active+clean","last_fresh":"2024-03-11T21:39:45.752219+0000","last_change":"2024-03-11T21:36:09.662765+0000","last_active":"2024-03-11T21:39:45.752219+0000","last_peered":"2024-03-11T21:39:45.752219+0000","last_clean":"2024-03-11T21:39:45.752219+0000","last_became_active":"2024-03-11T21:25:40.552216+0000","last_became_peered":"2024-03-11T21:25:40.552216+0000","last_unstale":"2024-03-11T21:39:45.752219+0000","last_undegraded":"2024-03-11T21:39:45.752219+0000","last_fullsized":"2024-03-11T21:39:45.752219+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"260'15","last_scrub_stamp":"2024-03-11T21:36:09.662615+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:39.525554+0000","last_clean_scrub_stamp":"2024-03-11T21:36:09.662615+0000","objects_scrubbed":4,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":593,"reported_epoch":337,"state":"active+clean","last_fresh":"2024-03-11T21:39:45.764258+0000","last_change":"2024-03-11T21:36:35.068915+0000","last_active":"2024-03-11T21:39:45.764258+0000","last_peered":"2024-03-11T21:39:45.764258+0000","last_clean":"2024-03-11T21:39:45.764258+0000","last_became_active":"2024-03-11T21:36:35.068453+0000","last_became_peered":"2024-03-11T21:36:35.068453+0000","last_unstale":"2024-03-11T21:39:45.764258+0000","last_undegraded":"2024-03-11T21:39:45.764258+0000","last_fullsized":"2024-03-11T21:39:45.764258+0000","mapping_epoch":277,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":278,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:35:26.190634+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:32:12.654573+0000","last_clean_scrub_stamp":"2024-03-11T21:35:26.190634+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3],"acting":[5,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":241,"reported_epoch":337,"state":"active+clean","last_fresh":"2024-03-11T21:39:45.757422+0000","last_change":"2024-03-11T21:35:11.902920+0000","last_active":"2024-03-11T21:39:45.757422+0000","last_peered":"2024-03-11T21:39:45.757422+0000","last_clean":"2024-03-11T21:39:45.757422+0000","last_became_active":"2024-03-11T21:34:41.145299+0000","last_became_peered":"2024-03-11T21:34:41.145299+0000","last_unstale":"2024-03-11T21:39:45.757422+0000","last_undegraded":"2024-03-11T21:39:45.757422+0000","last_fullsized":"2024-03-11T21:39:45.757422+0000","mapping_epoch":226,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":227,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:35:11.902818+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:41.544550+0000","last_clean_scrub_stamp":"2024-03-11T21:35:11.902818+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.1d","version":"0'0","reported_seq":603,"reported_epoch":337,"state":"active+clean","last_fresh":"2024-03-11T21:39:45.757804+0000","last_change":"2024-03-11T21:35:26.855588+0000","last_active":"2024-03-11T21:39:45.757804+0000","last_peered":"2024-03-11T21:39:45.757804+0000","last_clean":"2024-03-11T21:39:45.757804+0000","last_became_active":"2024-03-11T21:25:38.542628+0000","last_became_peered":"2024-03-11T21:25:38.542628+0000","last_unstale":"2024-03-11T21:39:45.757804+0000","last_undegraded":"2024-03-11T21:39:45.757804+0000","last_fullsized":"2024-03-11T21:39:45.757804+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:35:26.855511+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:37.512018+0000","last_clean_scrub_stamp":"2024-03-11T21:35:26.855511+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,0],"acting":[7,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.1b","version":"260'7","reported_seq":611,"reported_epoch":337,"state":"active+clean","last_fresh":"2024-03-11T21:39:45.753760+0000","last_change":"2024-03-11T21:36:35.087276+0000","last_active":"2024-03-11T21:39:45.753760+0000","last_peered":"2024-03-11T21:39:45.753760+0000","last_clean":"2024-03-11T21:39:45.753760+0000","last_became_active":"2024-03-11T21:36:35.057678+0000","last_became_peered":"2024-03-11T21:36:35.057678+0000","last_unstale":"2024-03-11T21:39:45.753760+0000","last_undegraded":"2024-03-11T21:39:45.753760+0000","last_fullsized":"2024-03-11T21:39:45.753760+0000","mapping_epoch":277,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":278,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'5","last_scrub_stamp":"2024-03-11T21:35:09.427562+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:39.525554+0000","last_clean_scrub_stamp":"2024-03-11T21:35:09.427562+0000","objects_scrubbed":2,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":14,"num_read_kb":10,"num_write":8,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,1],"acting":[4,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":582,"reported_epoch":336,"state":"active+clean","last_fresh":"2024-03-11T21:39:43.399523+0000","last_change":"2024-03-11T21:35:39.691825+0000","last_active":"2024-03-11T21:39:43.399523+0000","last_peered":"2024-03-11T21:39:43.399523+0000","last_clean":"2024-03-11T21:39:43.399523+0000","last_became_active":"2024-03-11T21:25:44.589786+0000","last_became_peered":"2024-03-11T21:25:44.589786+0000","last_unstale":"2024-03-11T21:39:43.399523+0000","last_undegraded":"2024-03-11T21:39:43.399523+0000","last_fullsized":"2024-03-11T21:39:43.399523+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:35:39.691690+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:43.559452+0000","last_clean_scrub_stamp":"2024-03-11T21:35:39.691690+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":18,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,7],"acting":[0,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":599,"reported_epoch":337,"state":"active+clean","last_fresh":"2024-03-11T21:39:45.764468+0000","last_change":"2024-03-11T21:35:25.194240+0000","last_active":"2024-03-11T21:39:45.764468+0000","last_peered":"2024-03-11T21:39:45.764468+0000","last_clean":"2024-03-11T21:39:45.764468+0000","last_became_active":"2024-03-11T21:34:41.147185+0000","last_became_peered":"2024-03-11T21:34:41.147185+0000","last_unstale":"2024-03-11T21:39:45.764468+0000","last_undegraded":"2024-03-11T21:39:45.764468+0000","last_fullsized":"2024-03-11T21:39:45.764468+0000","mapping_epoch":226,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":227,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:35:25.194115+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:32:21.535034+0000","last_clean_scrub_stamp":"2024-03-11T21:35:25.194115+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,7],"acting":[5,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1c","version":"0'0","reported_seq":603,"reported_epoch":337,"state":"active+clean","last_fresh":"2024-03-11T21:39:45.757628+0000","last_change":"2024-03-11T21:35:28.789693+0000","last_active":"2024-03-11T21:39:45.757628+0000","last_peered":"2024-03-11T21:39:45.757628+0000","last_clean":"2024-03-11T21:39:45.757628+0000","last_became_active":"2024-03-11T21:25:38.541364+0000","last_became_peered":"2024-03-11T21:25:38.541364+0000","last_unstale":"2024-03-11T21:39:45.757628+0000","last_undegraded":"2024-03-11T21:39:45.757628+0000","last_fullsized":"2024-03-11T21:39:45.757628+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:35:28.789592+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:28:32.566437+0000","last_clean_scrub_stamp":"2024-03-11T21:35:28.789592+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.1a","version":"260'15","reported_seq":635,"reported_epoch":337,"state":"active+clean","last_fresh":"2024-03-11T21:39:45.753257+0000","last_change":"2024-03-11T21:36:35.085358+0000","last_active":"2024-03-11T21:39:45.753257+0000","last_peered":"2024-03-11T21:39:45.753257+0000","last_clean":"2024-03-11T21:39:45.753257+0000","last_became_active":"2024-03-11T21:36:35.057029+0000","last_became_peered":"2024-03-11T21:36:35.057029+0000","last_unstale":"2024-03-11T21:39:45.753257+0000","last_undegraded":"2024-03-11T21:39:45.753257+0000","last_fullsized":"2024-03-11T21:39:45.753257+0000","mapping_epoch":277,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":278,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'9","last_scrub_stamp":"2024-03-11T21:35:05.451780+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:39.525554+0000","last_clean_scrub_stamp":"2024-03-11T21:35:05.451780+0000","objects_scrubbed":4,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0],"acting":[4,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1f","version":"0'0","reported_seq":458,"reported_epoch":336,"state":"active+clean","last_fresh":"2024-03-11T21:39:43.418704+0000","last_change":"2024-03-11T21:36:35.074675+0000","last_active":"2024-03-11T21:39:43.418704+0000","last_peered":"2024-03-11T21:39:43.418704+0000","last_clean":"2024-03-11T21:39:43.418704+0000","last_became_active":"2024-03-11T21:36:35.071755+0000","last_became_peered":"2024-03-11T21:36:35.071755+0000","last_unstale":"2024-03-11T21:39:43.418704+0000","last_undegraded":"2024-03-11T21:39:43.418704+0000","last_fullsized":"2024-03-11T21:39:43.418704+0000","mapping_epoch":277,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":278,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:34:05.862891+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:43.559452+0000","last_clean_scrub_stamp":"2024-03-11T21:34:05.862891+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1c","version":"0'0","reported_seq":589,"reported_epoch":337,"state":"active+clean","last_fresh":"2024-03-11T21:39:45.754070+0000","last_change":"2024-03-11T21:36:35.067469+0000","last_active":"2024-03-11T21:39:45.754070+0000","last_peered":"2024-03-11T21:39:45.754070+0000","last_clean":"2024-03-11T21:39:45.754070+0000","last_became_active":"2024-03-11T21:36:35.067127+0000","last_became_peered":"2024-03-11T21:36:35.067127+0000","last_unstale":"2024-03-11T21:39:45.754070+0000","last_undegraded":"2024-03-11T21:39:45.754070+0000","last_fullsized":"2024-03-11T21:39:45.754070+0000","mapping_epoch":277,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":278,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:34:11.766066+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:41.544550+0000","last_clean_scrub_stamp":"2024-03-11T21:34:11.766066+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1b","version":"0'0","reported_seq":507,"reported_epoch":336,"state":"active+clean","last_fresh":"2024-03-11T21:39:43.429989+0000","last_change":"2024-03-11T21:36:35.075385+0000","last_active":"2024-03-11T21:39:43.429989+0000","last_peered":"2024-03-11T21:39:43.429989+0000","last_clean":"2024-03-11T21:39:43.429989+0000","last_became_active":"2024-03-11T21:36:35.073959+0000","last_became_peered":"2024-03-11T21:36:35.073959+0000","last_unstale":"2024-03-11T21:39:43.429989+0000","last_undegraded":"2024-03-11T21:39:43.429989+0000","last_fullsized":"2024-03-11T21:39:43.429989+0000","mapping_epoch":277,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":278,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:34:27.998724+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:34:27.998724+0000","last_clean_scrub_stamp":"2024-03-11T21:34:27.998724+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1d","version":"260'18","reported_seq":551,"reported_epoch":336,"state":"active+clean","last_fresh":"2024-03-11T21:39:43.409612+0000","last_change":"2024-03-11T21:36:35.080207+0000","last_active":"2024-03-11T21:39:43.409612+0000","last_peered":"2024-03-11T21:39:43.409612+0000","last_clean":"2024-03-11T21:39:43.409612+0000","last_became_active":"2024-03-11T21:36:35.055880+0000","last_became_peered":"2024-03-11T21:36:35.055880+0000","last_unstale":"2024-03-11T21:39:43.409612+0000","last_undegraded":"2024-03-11T21:39:43.409612+0000","last_fullsized":"2024-03-11T21:39:43.409612+0000","mapping_epoch":277,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":278,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'12","last_scrub_stamp":"2024-03-11T21:35:07.971581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:39.525554+0000","last_clean_scrub_stamp":"2024-03-11T21:35:07.971581+0000","objects_scrubbed":0,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":33,"num_read_kb":24,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.1e","version":"0'0","reported_seq":578,"reported_epoch":337,"state":"active+clean","last_fresh":"2024-03-11T21:39:45.754998+0000","last_change":"2024-03-11T21:35:32.664451+0000","last_active":"2024-03-11T21:39:45.754998+0000","last_peered":"2024-03-11T21:39:45.754998+0000","last_clean":"2024-03-11T21:39:45.754998+0000","last_became_active":"2024-03-11T21:25:45.300355+0000","last_became_peered":"2024-03-11T21:25:45.300355+0000","last_unstale":"2024-03-11T21:39:45.754998+0000","last_undegraded":"2024-03-11T21:39:45.754998+0000","last_fullsized":"2024-03-11T21:39:45.754998+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:35:32.664371+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:43.559452+0000","last_clean_scrub_stamp":"2024-03-11T21:35:32.664371+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5],"acting":[4,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1d","version":"0'0","reported_seq":588,"reported_epoch":337,"state":"active+clean","last_fresh":"2024-03-11T21:39:45.765545+0000","last_change":"2024-03-11T21:36:07.624196+0000","last_active":"2024-03-11T21:39:45.765545+0000","last_peered":"2024-03-11T21:39:45.765545+0000","last_clean":"2024-03-11T21:39:45.765545+0000","last_became_active":"2024-03-11T21:25:42.579407+0000","last_became_peered":"2024-03-11T21:25:42.579407+0000","last_unstale":"2024-03-11T21:39:45.765545+0000","last_undegraded":"2024-03-11T21:39:45.765545+0000","last_fullsized":"2024-03-11T21:39:45.765545+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:36:07.624075+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:30:18.678531+0000","last_clean_scrub_stamp":"2024-03-11T21:36:07.624075+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1a","version":"0'0","reported_seq":616,"reported_epoch":337,"state":"active+clean","last_fresh":"2024-03-11T21:39:45.766442+0000","last_change":"2024-03-11T21:36:12.993542+0000","last_active":"2024-03-11T21:39:45.766442+0000","last_peered":"2024-03-11T21:39:45.766442+0000","last_clean":"2024-03-11T21:39:45.766442+0000","last_became_active":"2024-03-11T21:34:41.145014+0000","last_became_peered":"2024-03-11T21:34:41.145014+0000","last_unstale":"2024-03-11T21:39:45.766442+0000","last_undegraded":"2024-03-11T21:39:45.766442+0000","last_fullsized":"2024-03-11T21:39:45.766442+0000","mapping_epoch":226,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":227,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:36:12.993437+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:37.512018+0000","last_clean_scrub_stamp":"2024-03-11T21:36:12.993437+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.1c","version":"260'27","reported_seq":650,"reported_epoch":337,"state":"active+clean","last_fresh":"2024-03-11T21:39:45.770175+0000","last_change":"2024-03-11T21:36:35.097208+0000","last_active":"2024-03-11T21:39:45.770175+0000","last_peered":"2024-03-11T21:39:45.770175+0000","last_clean":"2024-03-11T21:39:45.770175+0000","last_became_active":"2024-03-11T21:36:35.074579+0000","last_became_peered":"2024-03-11T21:36:35.074579+0000","last_unstale":"2024-03-11T21:39:45.770175+0000","last_undegraded":"2024-03-11T21:39:45.770175+0000","last_fullsized":"2024-03-11T21:39:45.770175+0000","mapping_epoch":277,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":278,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'15","last_scrub_stamp":"2024-03-11T21:34:10.311181+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:39.525554+0000","last_clean_scrub_stamp":"2024-03-11T21:34:10.311181+0000","objects_scrubbed":7,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":44,"num_read_kb":35,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.1d","version":"0'0","reported_seq":576,"reported_epoch":337,"state":"active+clean","last_fresh":"2024-03-11T21:39:45.765043+0000","last_change":"2024-03-11T21:35:21.744646+0000","last_active":"2024-03-11T21:39:45.765043+0000","last_peered":"2024-03-11T21:39:45.765043+0000","last_clean":"2024-03-11T21:39:45.765043+0000","last_became_active":"2024-03-11T21:25:44.588845+0000","last_became_peered":"2024-03-11T21:25:44.588845+0000","last_unstale":"2024-03-11T21:39:45.765043+0000","last_undegraded":"2024-03-11T21:39:45.765043+0000","last_fullsized":"2024-03-11T21:39:45.765043+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:35:21.744223+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:43.559452+0000","last_clean_scrub_stamp":"2024-03-11T21:35:21.744223+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":99,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.1e","version":"0'0","reported_seq":587,"reported_epoch":336,"state":"active+clean","last_fresh":"2024-03-11T21:39:43.382069+0000","last_change":"2024-03-11T21:35:10.728691+0000","last_active":"2024-03-11T21:39:43.382069+0000","last_peered":"2024-03-11T21:39:43.382069+0000","last_clean":"2024-03-11T21:39:43.382069+0000","last_became_active":"2024-03-11T21:25:42.582959+0000","last_became_peered":"2024-03-11T21:25:42.582959+0000","last_unstale":"2024-03-11T21:39:43.382069+0000","last_undegraded":"2024-03-11T21:39:43.382069+0000","last_fullsized":"2024-03-11T21:39:43.382069+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:35:10.728569+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:26:58.335431+0000","last_clean_scrub_stamp":"2024-03-11T21:35:10.728569+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":18,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.19","version":"67'2","reported_seq":533,"reported_epoch":336,"state":"active+clean","last_fresh":"2024-03-11T21:39:43.407192+0000","last_change":"2024-03-11T21:36:35.069729+0000","last_active":"2024-03-11T21:39:43.407192+0000","last_peered":"2024-03-11T21:39:43.407192+0000","last_clean":"2024-03-11T21:39:43.407192+0000","last_became_active":"2024-03-11T21:36:35.064249+0000","last_became_peered":"2024-03-11T21:36:35.064249+0000","last_unstale":"2024-03-11T21:39:43.407192+0000","last_undegraded":"2024-03-11T21:39:43.407192+0000","last_fullsized":"2024-03-11T21:39:43.407192+0000","mapping_epoch":277,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":278,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'2","last_scrub_stamp":"2024-03-11T21:32:15.509542+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:37.512018+0000","last_clean_scrub_stamp":"2024-03-11T21:32:15.509542+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":957,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,0],"acting":[3,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1f","version":"260'24","reported_seq":663,"reported_epoch":337,"state":"active+clean","last_fresh":"2024-03-11T21:39:45.767558+0000","last_change":"2024-03-11T21:38:54.049460+0000","last_active":"2024-03-11T21:39:45.767558+0000","last_peered":"2024-03-11T21:39:45.767558+0000","last_clean":"2024-03-11T21:39:45.767558+0000","last_became_active":"2024-03-11T21:32:22.798242+0000","last_became_peered":"2024-03-11T21:32:22.798242+0000","last_unstale":"2024-03-11T21:39:45.767558+0000","last_undegraded":"2024-03-11T21:39:45.767558+0000","last_fullsized":"2024-03-11T21:39:45.767558+0000","mapping_epoch":166,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":167,"parent":"0.0","parent_split_bits":0,"last_scrub":"260'24","last_scrub_stamp":"2024-03-11T21:38:54.049368+0000","last_deep_scrub":"124'16","last_deep_scrub_stamp":"2024-03-11T21:35:38.173841+0000","last_clean_scrub_stamp":"2024-03-11T21:38:54.049368+0000","objects_scrubbed":5,"log_size":24,"log_dups_size":0,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:40:23.463882+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":32,"num_read_kb":25,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,0],"acting":[6,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.1c","version":"74'1","reported_seq":245,"reported_epoch":337,"state":"active+clean","last_fresh":"2024-03-11T21:39:45.757525+0000","last_change":"2024-03-11T21:36:11.732043+0000","last_active":"2024-03-11T21:39:45.757525+0000","last_peered":"2024-03-11T21:39:45.757525+0000","last_clean":"2024-03-11T21:39:45.757525+0000","last_became_active":"2024-03-11T21:34:41.144839+0000","last_became_peered":"2024-03-11T21:34:41.144839+0000","last_unstale":"2024-03-11T21:39:45.757525+0000","last_undegraded":"2024-03-11T21:39:45.757525+0000","last_fullsized":"2024-03-11T21:39:45.757525+0000","mapping_epoch":226,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":227,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'1","last_scrub_stamp":"2024-03-11T21:36:11.731914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:43.559452+0000","last_clean_scrub_stamp":"2024-03-11T21:36:11.731914+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":370,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":370,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.1f","version":"229'10","reported_seq":814,"reported_epoch":337,"state":"active+clean","last_fresh":"2024-03-11T21:39:45.767520+0000","last_change":"2024-03-11T21:39:42.162545+0000","last_active":"2024-03-11T21:39:45.767520+0000","last_peered":"2024-03-11T21:39:45.767520+0000","last_clean":"2024-03-11T21:39:45.767520+0000","last_became_active":"2024-03-11T21:34:41.145862+0000","last_became_peered":"2024-03-11T21:34:41.145862+0000","last_unstale":"2024-03-11T21:39:45.767520+0000","last_undegraded":"2024-03-11T21:39:45.767520+0000","last_fullsized":"2024-03-11T21:39:45.767520+0000","mapping_epoch":226,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":227,"parent":"0.0","parent_split_bits":0,"last_scrub":"229'10","last_scrub_stamp":"2024-03-11T21:39:42.162463+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:41.544550+0000","last_clean_scrub_stamp":"2024-03-11T21:39:42.162463+0000","objects_scrubbed":1,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:41:06.487514+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.18","version":"67'1","reported_seq":639,"reported_epoch":337,"state":"active+clean","last_fresh":"2024-03-11T21:39:45.763998+0000","last_change":"2024-03-11T21:36:10.458150+0000","last_active":"2024-03-11T21:39:45.763998+0000","last_peered":"2024-03-11T21:39:45.763998+0000","last_clean":"2024-03-11T21:39:45.763998+0000","last_became_active":"2024-03-11T21:35:23.310241+0000","last_became_peered":"2024-03-11T21:35:23.310241+0000","last_unstale":"2024-03-11T21:39:45.763998+0000","last_undegraded":"2024-03-11T21:39:45.763998+0000","last_fullsized":"2024-03-11T21:39:45.763998+0000","mapping_epoch":248,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":249,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'1","last_scrub_stamp":"2024-03-11T21:36:10.458028+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:37.512018+0000","last_clean_scrub_stamp":"2024-03-11T21:36:10.458028+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":436,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":7,"num_read_kb":7,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":1308,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,2],"acting":[5,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1e","version":"260'20","reported_seq":679,"reported_epoch":336,"state":"active+clean","last_fresh":"2024-03-11T21:39:43.381952+0000","last_change":"2024-03-11T21:36:35.083447+0000","last_active":"2024-03-11T21:39:43.381952+0000","last_peered":"2024-03-11T21:39:43.381952+0000","last_clean":"2024-03-11T21:39:43.381952+0000","last_became_active":"2024-03-11T21:36:35.057843+0000","last_became_peered":"2024-03-11T21:36:35.057843+0000","last_unstale":"2024-03-11T21:39:43.381952+0000","last_undegraded":"2024-03-11T21:39:43.381952+0000","last_fullsized":"2024-03-11T21:39:43.381952+0000","mapping_epoch":277,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":278,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'10","last_scrub_stamp":"2024-03-11T21:35:16.825146+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:39.525554+0000","last_clean_scrub_stamp":"2024-03-11T21:35:16.825146+0000","objects_scrubbed":5,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":44,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":25,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.13","version":"0'0","reported_seq":466,"reported_epoch":336,"state":"active+clean","last_fresh":"2024-03-11T21:39:43.430205+0000","last_change":"2024-03-11T21:36:35.063744+0000","last_active":"2024-03-11T21:39:43.430205+0000","last_peered":"2024-03-11T21:39:43.430205+0000","last_clean":"2024-03-11T21:39:43.430205+0000","last_became_active":"2024-03-11T21:36:35.062618+0000","last_became_peered":"2024-03-11T21:36:35.062618+0000","last_unstale":"2024-03-11T21:39:43.430205+0000","last_undegraded":"2024-03-11T21:39:43.430205+0000","last_fullsized":"2024-03-11T21:39:43.430205+0000","mapping_epoch":277,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":278,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:34:13.822803+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:43.559452+0000","last_clean_scrub_stamp":"2024-03-11T21:34:13.822803+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":241,"reported_epoch":337,"state":"active+clean","last_fresh":"2024-03-11T21:39:45.756973+0000","last_change":"2024-03-11T21:35:16.917648+0000","last_active":"2024-03-11T21:39:45.756973+0000","last_peered":"2024-03-11T21:39:45.756973+0000","last_clean":"2024-03-11T21:39:45.756973+0000","last_became_active":"2024-03-11T21:34:41.142569+0000","last_became_peered":"2024-03-11T21:34:41.142569+0000","last_unstale":"2024-03-11T21:39:45.756973+0000","last_undegraded":"2024-03-11T21:39:45.756973+0000","last_fullsized":"2024-03-11T21:39:45.756973+0000","mapping_epoch":226,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":227,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:35:16.917532+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:41.544550+0000","last_clean_scrub_stamp":"2024-03-11T21:35:16.917532+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.17","version":"0'0","reported_seq":604,"reported_epoch":337,"state":"active+clean","last_fresh":"2024-03-11T21:39:45.767325+0000","last_change":"2024-03-11T21:38:48.084622+0000","last_active":"2024-03-11T21:39:45.767325+0000","last_peered":"2024-03-11T21:39:45.767325+0000","last_clean":"2024-03-11T21:39:45.767325+0000","last_became_active":"2024-03-11T21:25:38.540349+0000","last_became_peered":"2024-03-11T21:25:38.540349+0000","last_unstale":"2024-03-11T21:39:45.767325+0000","last_undegraded":"2024-03-11T21:39:45.767325+0000","last_fullsized":"2024-03-11T21:39:45.767325+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:38:48.084549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:28:25.918101+0000","last_clean_scrub_stamp":"2024-03-11T21:38:48.084549+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:40:17.652132+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,2],"acting":[6,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.11","version":"260'19","reported_seq":592,"reported_epoch":336,"state":"active+clean","last_fresh":"2024-03-11T21:39:43.407709+0000","last_change":"2024-03-11T21:36:35.096720+0000","last_active":"2024-03-11T21:39:43.407709+0000","last_peered":"2024-03-11T21:39:43.407709+0000","last_clean":"2024-03-11T21:39:43.407709+0000","last_became_active":"2024-03-11T21:36:35.073689+0000","last_became_peered":"2024-03-11T21:36:35.073689+0000","last_unstale":"2024-03-11T21:39:43.407709+0000","last_undegraded":"2024-03-11T21:39:43.407709+0000","last_fullsized":"2024-03-11T21:39:43.407709+0000","mapping_epoch":277,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":278,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'11","last_scrub_stamp":"2024-03-11T21:35:16.044397+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:39.525554+0000","last_clean_scrub_stamp":"2024-03-11T21:35:16.044397+0000","objects_scrubbed":5,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":31,"num_read_kb":24,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":241,"reported_epoch":337,"state":"active+clean","last_fresh":"2024-03-11T21:39:45.757143+0000","last_change":"2024-03-11T21:35:32.719869+0000","last_active":"2024-03-11T21:39:45.757143+0000","last_peered":"2024-03-11T21:39:45.757143+0000","last_clean":"2024-03-11T21:39:45.757143+0000","last_became_active":"2024-03-11T21:34:41.137235+0000","last_became_peered":"2024-03-11T21:34:41.137235+0000","last_unstale":"2024-03-11T21:39:45.757143+0000","last_undegraded":"2024-03-11T21:39:45.757143+0000","last_fullsized":"2024-03-11T21:39:45.757143+0000","mapping_epoch":226,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":227,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:35:32.719789+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:43.559452+0000","last_clean_scrub_stamp":"2024-03-11T21:35:32.719789+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,4],"acting":[7,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":617,"reported_epoch":337,"state":"active+clean","last_fresh":"2024-03-11T21:39:45.767004+0000","last_change":"2024-03-11T21:38:51.036994+0000","last_active":"2024-03-11T21:39:45.767004+0000","last_peered":"2024-03-11T21:39:45.767004+0000","last_clean":"2024-03-11T21:39:45.767004+0000","last_became_active":"2024-03-11T21:34:41.156036+0000","last_became_peered":"2024-03-11T21:34:41.156036+0000","last_unstale":"2024-03-11T21:39:45.767004+0000","last_undegraded":"2024-03-11T21:39:45.767004+0000","last_fullsized":"2024-03-11T21:39:45.767004+0000","mapping_epoch":226,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":227,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:38:51.036736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:38:51.036736+0000","last_clean_scrub_stamp":"2024-03-11T21:38:51.036736+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:40:02.840388+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.16","version":"0'0","reported_seq":603,"reported_epoch":337,"state":"active+clean","last_fresh":"2024-03-11T21:39:45.764857+0000","last_change":"2024-03-11T21:35:39.108623+0000","last_active":"2024-03-11T21:39:45.764857+0000","last_peered":"2024-03-11T21:39:45.764857+0000","last_clean":"2024-03-11T21:39:45.764857+0000","last_became_active":"2024-03-11T21:25:38.539103+0000","last_became_peered":"2024-03-11T21:25:38.539103+0000","last_unstale":"2024-03-11T21:39:45.764857+0000","last_undegraded":"2024-03-11T21:39:45.764857+0000","last_fullsized":"2024-03-11T21:39:45.764857+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:35:39.108546+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:27:11.938909+0000","last_clean_scrub_stamp":"2024-03-11T21:35:39.108546+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,2],"acting":[5,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.10","version":"260'10","reported_seq":474,"reported_epoch":336,"state":"active+clean","last_fresh":"2024-03-11T21:39:43.407206+0000","last_change":"2024-03-11T21:36:35.084268+0000","last_active":"2024-03-11T21:39:43.407206+0000","last_peered":"2024-03-11T21:39:43.407206+0000","last_clean":"2024-03-11T21:39:43.407206+0000","last_became_active":"2024-03-11T21:36:35.067500+0000","last_became_peered":"2024-03-11T21:36:35.067500+0000","last_unstale":"2024-03-11T21:39:43.407206+0000","last_undegraded":"2024-03-11T21:39:43.407206+0000","last_fullsized":"2024-03-11T21:39:43.407206+0000","mapping_epoch":277,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":278,"parent":"0.0","parent_split_bits":0,"last_scrub":"94'6","last_scrub_stamp":"2024-03-11T21:34:00.727416+0000","last_deep_scrub":"94'6","last_deep_scrub_stamp":"2024-03-11T21:28:24.603974+0000","last_clean_scrub_stamp":"2024-03-11T21:34:00.727416+0000","objects_scrubbed":2,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":12,"num_read_kb":10,"num_write":8,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6],"acting":[3,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":519,"reported_epoch":336,"state":"active+clean","last_fresh":"2024-03-11T21:39:43.407515+0000","last_change":"2024-03-11T21:36:35.070468+0000","last_active":"2024-03-11T21:39:43.407515+0000","last_peered":"2024-03-11T21:39:43.407515+0000","last_clean":"2024-03-11T21:39:43.407515+0000","last_became_active":"2024-03-11T21:36:35.070176+0000","last_became_peered":"2024-03-11T21:36:35.070176+0000","last_unstale":"2024-03-11T21:39:43.407515+0000","last_undegraded":"2024-03-11T21:39:43.407515+0000","last_fullsized":"2024-03-11T21:39:43.407515+0000","mapping_epoch":277,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":278,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:35:18.062272+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:43.559452+0000","last_clean_scrub_stamp":"2024-03-11T21:35:18.062272+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":590,"reported_epoch":337,"state":"active+clean","last_fresh":"2024-03-11T21:39:45.768948+0000","last_change":"2024-03-11T21:36:35.058551+0000","last_active":"2024-03-11T21:39:45.768948+0000","last_peered":"2024-03-11T21:39:45.768948+0000","last_clean":"2024-03-11T21:39:45.768948+0000","last_became_active":"2024-03-11T21:36:35.057906+0000","last_became_peered":"2024-03-11T21:36:35.057906+0000","last_unstale":"2024-03-11T21:39:45.768948+0000","last_undegraded":"2024-03-11T21:39:45.768948+0000","last_fullsized":"2024-03-11T21:39:45.768948+0000","mapping_epoch":277,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":278,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:34:15.516278+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:27:21.207303+0000","last_clean_scrub_stamp":"2024-03-11T21:34:15.516278+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.15","version":"0'0","reported_seq":596,"reported_epoch":337,"state":"active+clean","last_fresh":"2024-03-11T21:39:45.765256+0000","last_change":"2024-03-11T21:36:06.579754+0000","last_active":"2024-03-11T21:39:45.765256+0000","last_peered":"2024-03-11T21:39:45.765256+0000","last_clean":"2024-03-11T21:39:45.765256+0000","last_became_active":"2024-03-11T21:25:38.541705+0000","last_became_peered":"2024-03-11T21:25:38.541705+0000","last_unstale":"2024-03-11T21:39:45.765256+0000","last_undegraded":"2024-03-11T21:39:45.765256+0000","last_fullsized":"2024-03-11T21:39:45.765256+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:36:06.579636+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:26:44.860499+0000","last_clean_scrub_stamp":"2024-03-11T21:36:06.579636+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.13","version":"260'24","reported_seq":647,"reported_epoch":337,"state":"active+clean","last_fresh":"2024-03-11T21:39:45.752792+0000","last_change":"2024-03-11T21:35:14.453053+0000","last_active":"2024-03-11T21:39:45.752792+0000","last_peered":"2024-03-11T21:39:45.752792+0000","last_clean":"2024-03-11T21:39:45.752792+0000","last_became_active":"2024-03-11T21:26:11.214879+0000","last_became_peered":"2024-03-11T21:26:11.214879+0000","last_unstale":"2024-03-11T21:39:45.752792+0000","last_undegraded":"2024-03-11T21:39:45.752792+0000","last_fullsized":"2024-03-11T21:39:45.752792+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"109'16","last_scrub_stamp":"2024-03-11T21:35:14.452891+0000","last_deep_scrub":"109'16","last_deep_scrub_stamp":"2024-03-11T21:32:24.657490+0000","last_clean_scrub_stamp":"2024-03-11T21:35:14.452891+0000","objects_scrubbed":5,"log_size":24,"log_dups_size":0,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":32,"num_read_kb":25,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,2],"acting":[4,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.10","version":"0'0","reported_seq":582,"reported_epoch":336,"state":"active+clean","last_fresh":"2024-03-11T21:39:43.380007+0000","last_change":"2024-03-11T21:35:29.712658+0000","last_active":"2024-03-11T21:39:43.380007+0000","last_peered":"2024-03-11T21:39:43.380007+0000","last_clean":"2024-03-11T21:39:43.380007+0000","last_became_active":"2024-03-11T21:25:44.591189+0000","last_became_peered":"2024-03-11T21:25:44.591189+0000","last_unstale":"2024-03-11T21:39:43.380007+0000","last_undegraded":"2024-03-11T21:39:43.380007+0000","last_fullsized":"2024-03-11T21:39:43.380007+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:35:29.712557+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:27:25.417450+0000","last_clean_scrub_stamp":"2024-03-11T21:35:29.712557+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,1],"acting":[0,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":517,"reported_epoch":336,"state":"active+clean","last_fresh":"2024-03-11T21:39:43.430293+0000","last_change":"2024-03-11T21:36:35.062276+0000","last_active":"2024-03-11T21:39:43.430293+0000","last_peered":"2024-03-11T21:39:43.430293+0000","last_clean":"2024-03-11T21:39:43.430293+0000","last_became_active":"2024-03-11T21:36:35.056619+0000","last_became_peered":"2024-03-11T21:36:35.056619+0000","last_unstale":"2024-03-11T21:39:43.430293+0000","last_undegraded":"2024-03-11T21:39:43.430293+0000","last_fullsized":"2024-03-11T21:39:43.430293+0000","mapping_epoch":277,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":278,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:35:13.060487+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:31:15.367898+0000","last_clean_scrub_stamp":"2024-03-11T21:35:13.060487+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.14","version":"0'0","reported_seq":622,"reported_epoch":337,"state":"active+clean","last_fresh":"2024-03-11T21:39:45.766746+0000","last_change":"2024-03-11T21:38:35.999365+0000","last_active":"2024-03-11T21:39:45.766746+0000","last_peered":"2024-03-11T21:39:45.766746+0000","last_clean":"2024-03-11T21:39:45.766746+0000","last_became_active":"2024-03-11T21:36:35.068038+0000","last_became_peered":"2024-03-11T21:36:35.068038+0000","last_unstale":"2024-03-11T21:39:45.766746+0000","last_undegraded":"2024-03-11T21:39:45.766746+0000","last_fullsized":"2024-03-11T21:39:45.766746+0000","mapping_epoch":277,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":278,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:38:35.999231+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:37.512018+0000","last_clean_scrub_stamp":"2024-03-11T21:38:35.999231+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:40:00.642531+0000","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,5],"acting":[6,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.12","version":"260'15","reported_seq":618,"reported_epoch":337,"state":"active+clean","last_fresh":"2024-03-11T21:39:45.768772+0000","last_change":"2024-03-11T21:35:32.590439+0000","last_active":"2024-03-11T21:39:45.768772+0000","last_peered":"2024-03-11T21:39:45.768772+0000","last_clean":"2024-03-11T21:39:45.768772+0000","last_became_active":"2024-03-11T21:25:40.547082+0000","last_became_peered":"2024-03-11T21:25:40.547082+0000","last_unstale":"2024-03-11T21:39:45.768772+0000","last_undegraded":"2024-03-11T21:39:45.768772+0000","last_fullsized":"2024-03-11T21:39:45.768772+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'9","last_scrub_stamp":"2024-03-11T21:35:32.590318+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:39.525554+0000","last_clean_scrub_stamp":"2024-03-11T21:35:32.590318+0000","objects_scrubbed":4,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":584,"reported_epoch":337,"state":"active+clean","last_fresh":"2024-03-11T21:39:45.754875+0000","last_change":"2024-03-11T21:36:04.754168+0000","last_active":"2024-03-11T21:39:45.754875+0000","last_peered":"2024-03-11T21:39:45.754875+0000","last_clean":"2024-03-11T21:39:45.754875+0000","last_became_active":"2024-03-11T21:25:44.589586+0000","last_became_peered":"2024-03-11T21:25:44.589586+0000","last_unstale":"2024-03-11T21:39:45.754875+0000","last_undegraded":"2024-03-11T21:39:45.754875+0000","last_fullsized":"2024-03-11T21:39:45.754875+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:36:04.753979+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:27:09.801138+0000","last_clean_scrub_stamp":"2024-03-11T21:36:04.753979+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":113,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,5],"acting":[4,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.14","version":"280'12","reported_seq":717,"reported_epoch":336,"state":"active+clean","last_fresh":"2024-03-11T21:39:43.422862+0000","last_change":"2024-03-11T21:36:35.074550+0000","last_active":"2024-03-11T21:39:43.422862+0000","last_peered":"2024-03-11T21:39:43.422862+0000","last_clean":"2024-03-11T21:39:43.422862+0000","last_became_active":"2024-03-11T21:36:35.073912+0000","last_became_peered":"2024-03-11T21:36:35.073912+0000","last_unstale":"2024-03-11T21:39:43.422862+0000","last_undegraded":"2024-03-11T21:39:43.422862+0000","last_fullsized":"2024-03-11T21:39:43.422862+0000","mapping_epoch":277,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":278,"parent":"0.0","parent_split_bits":0,"last_scrub":"229'10","last_scrub_stamp":"2024-03-11T21:35:26.182107+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:41.544550+0000","last_clean_scrub_stamp":"2024-03-11T21:35:26.182107+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,2],"acting":[3,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.13","version":"0'0","reported_seq":596,"reported_epoch":336,"state":"active+clean","last_fresh":"2024-03-11T21:39:43.381917+0000","last_change":"2024-03-11T21:36:35.057688+0000","last_active":"2024-03-11T21:39:43.381917+0000","last_peered":"2024-03-11T21:39:43.381917+0000","last_clean":"2024-03-11T21:39:43.381917+0000","last_became_active":"2024-03-11T21:36:35.057132+0000","last_became_peered":"2024-03-11T21:36:35.057132+0000","last_unstale":"2024-03-11T21:39:43.381917+0000","last_undegraded":"2024-03-11T21:39:43.381917+0000","last_fullsized":"2024-03-11T21:39:43.381917+0000","mapping_epoch":277,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":278,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:34:08.650371+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:37.512018+0000","last_clean_scrub_stamp":"2024-03-11T21:34:08.650371+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.15","version":"260'19","reported_seq":670,"reported_epoch":337,"state":"active+clean","last_fresh":"2024-03-11T21:39:45.765236+0000","last_change":"2024-03-11T21:36:35.088986+0000","last_active":"2024-03-11T21:39:45.765236+0000","last_peered":"2024-03-11T21:39:45.765236+0000","last_clean":"2024-03-11T21:39:45.765236+0000","last_became_active":"2024-03-11T21:36:35.057285+0000","last_became_peered":"2024-03-11T21:36:35.057285+0000","last_unstale":"2024-03-11T21:39:45.765236+0000","last_undegraded":"2024-03-11T21:39:45.765236+0000","last_fullsized":"2024-03-11T21:39:45.765236+0000","mapping_epoch":277,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":278,"parent":"0.0","parent_split_bits":0,"last_scrub":"94'13","last_scrub_stamp":"2024-03-11T21:35:29.178543+0000","last_deep_scrub":"94'13","last_deep_scrub_stamp":"2024-03-11T21:29:41.853158+0000","last_clean_scrub_stamp":"2024-03-11T21:35:29.178543+0000","objects_scrubbed":4,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":11,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.16","version":"0'0","reported_seq":603,"reported_epoch":336,"state":"active+clean","last_fresh":"2024-03-11T21:39:43.381576+0000","last_change":"2024-03-11T21:36:35.058372+0000","last_active":"2024-03-11T21:39:43.381576+0000","last_peered":"2024-03-11T21:39:43.381576+0000","last_clean":"2024-03-11T21:39:43.381576+0000","last_became_active":"2024-03-11T21:36:35.057355+0000","last_became_peered":"2024-03-11T21:36:35.057355+0000","last_unstale":"2024-03-11T21:39:43.381576+0000","last_undegraded":"2024-03-11T21:39:43.381576+0000","last_fullsized":"2024-03-11T21:39:43.381576+0000","mapping_epoch":277,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":278,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:35:07.665075+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:43.559452+0000","last_clean_scrub_stamp":"2024-03-11T21:35:07.665075+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":35,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.15","version":"98'9","reported_seq":766,"reported_epoch":337,"state":"active+clean","last_fresh":"2024-03-11T21:39:45.765352+0000","last_change":"2024-03-11T21:36:03.366581+0000","last_active":"2024-03-11T21:39:45.765352+0000","last_peered":"2024-03-11T21:39:45.765352+0000","last_clean":"2024-03-11T21:39:45.765352+0000","last_became_active":"2024-03-11T21:25:42.572555+0000","last_became_peered":"2024-03-11T21:25:42.572555+0000","last_unstale":"2024-03-11T21:39:45.765352+0000","last_undegraded":"2024-03-11T21:39:45.765352+0000","last_fullsized":"2024-03-11T21:39:45.765352+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"98'9","last_scrub_stamp":"2024-03-11T21:36:03.366450+0000","last_deep_scrub":"98'9","last_deep_scrub_stamp":"2024-03-11T21:31:20.624923+0000","last_clean_scrub_stamp":"2024-03-11T21:36:03.366450+0000","objects_scrubbed":1,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.12","version":"0'0","reported_seq":617,"reported_epoch":337,"state":"active+clean","last_fresh":"2024-03-11T21:39:45.764583+0000","last_change":"2024-03-11T21:36:35.058369+0000","last_active":"2024-03-11T21:39:45.764583+0000","last_peered":"2024-03-11T21:39:45.764583+0000","last_clean":"2024-03-11T21:39:45.764583+0000","last_became_active":"2024-03-11T21:36:35.058033+0000","last_became_peered":"2024-03-11T21:36:35.058033+0000","last_unstale":"2024-03-11T21:39:45.764583+0000","last_undegraded":"2024-03-11T21:39:45.764583+0000","last_fullsized":"2024-03-11T21:39:45.764583+0000","mapping_epoch":277,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":278,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:35:27.158468+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:37.512018+0000","last_clean_scrub_stamp":"2024-03-11T21:35:27.158468+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.14","version":"260'20","reported_seq":600,"reported_epoch":336,"state":"active+clean","last_fresh":"2024-03-11T21:39:43.422502+0000","last_change":"2024-03-11T21:36:35.095382+0000","last_active":"2024-03-11T21:39:43.422502+0000","last_peered":"2024-03-11T21:39:43.422502+0000","last_clean":"2024-03-11T21:39:43.422502+0000","last_became_active":"2024-03-11T21:36:35.069638+0000","last_became_peered":"2024-03-11T21:36:35.069638+0000","last_unstale":"2024-03-11T21:39:43.422502+0000","last_undegraded":"2024-03-11T21:39:43.422502+0000","last_fullsized":"2024-03-11T21:39:43.422502+0000","mapping_epoch":277,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":278,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'10","last_scrub_stamp":"2024-03-11T21:35:28.189296+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:39.525554+0000","last_clean_scrub_stamp":"2024-03-11T21:35:28.189296+0000","objects_scrubbed":0,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":29,"num_read_kb":24,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.15","version":"0'0","reported_seq":241,"reported_epoch":337,"state":"active+clean","last_fresh":"2024-03-11T21:39:45.757718+0000","last_change":"2024-03-11T21:36:14.736240+0000","last_active":"2024-03-11T21:39:45.757718+0000","last_peered":"2024-03-11T21:39:45.757718+0000","last_clean":"2024-03-11T21:39:45.757718+0000","last_became_active":"2024-03-11T21:34:41.142760+0000","last_became_peered":"2024-03-11T21:34:41.142760+0000","last_unstale":"2024-03-11T21:39:45.757718+0000","last_undegraded":"2024-03-11T21:39:45.757718+0000","last_fullsized":"2024-03-11T21:39:45.757718+0000","mapping_epoch":226,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":227,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:36:14.735985+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:43.559452+0000","last_clean_scrub_stamp":"2024-03-11T21:36:14.735985+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":36,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":588,"reported_epoch":337,"state":"active+clean","last_fresh":"2024-03-11T21:39:45.764774+0000","last_change":"2024-03-11T21:36:35.060666+0000","last_active":"2024-03-11T21:39:45.764774+0000","last_peered":"2024-03-11T21:39:45.764774+0000","last_clean":"2024-03-11T21:39:45.764774+0000","last_became_active":"2024-03-11T21:36:35.060342+0000","last_became_peered":"2024-03-11T21:36:35.060342+0000","last_unstale":"2024-03-11T21:39:45.764774+0000","last_undegraded":"2024-03-11T21:39:45.764774+0000","last_fullsized":"2024-03-11T21:39:45.764774+0000","mapping_epoch":277,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":278,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:34:09.244852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:41.544550+0000","last_clean_scrub_stamp":"2024-03-11T21:34:09.244852+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,1],"acting":[5,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.11","version":"0'0","reported_seq":605,"reported_epoch":337,"state":"active+clean","last_fresh":"2024-03-11T21:39:45.767091+0000","last_change":"2024-03-11T21:38:49.044802+0000","last_active":"2024-03-11T21:39:45.767091+0000","last_peered":"2024-03-11T21:39:45.767091+0000","last_clean":"2024-03-11T21:39:45.767091+0000","last_became_active":"2024-03-11T21:25:38.537415+0000","last_became_peered":"2024-03-11T21:25:38.537415+0000","last_unstale":"2024-03-11T21:39:45.767091+0000","last_undegraded":"2024-03-11T21:39:45.767091+0000","last_fullsized":"2024-03-11T21:39:45.767091+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:38:49.044722+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:37.512018+0000","last_clean_scrub_stamp":"2024-03-11T21:38:49.044722+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:40:07.269648+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.17","version":"260'15","reported_seq":545,"reported_epoch":336,"state":"active+clean","last_fresh":"2024-03-11T21:39:43.418779+0000","last_change":"2024-03-11T21:36:35.099451+0000","last_active":"2024-03-11T21:39:43.418779+0000","last_peered":"2024-03-11T21:39:43.418779+0000","last_clean":"2024-03-11T21:39:43.418779+0000","last_became_active":"2024-03-11T21:36:35.071346+0000","last_became_peered":"2024-03-11T21:36:35.071346+0000","last_unstale":"2024-03-11T21:39:43.418779+0000","last_undegraded":"2024-03-11T21:39:43.418779+0000","last_fullsized":"2024-03-11T21:39:43.418779+0000","mapping_epoch":277,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":278,"parent":"0.0","parent_split_bits":0,"last_scrub":"97'9","last_scrub_stamp":"2024-03-11T21:32:08.495352+0000","last_deep_scrub":"97'9","last_deep_scrub_stamp":"2024-03-11T21:27:25.803132+0000","last_clean_scrub_stamp":"2024-03-11T21:32:08.495352+0000","objects_scrubbed":3,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":17,"num_read_kb":14,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,5],"acting":[3,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.14","version":"0'0","reported_seq":595,"reported_epoch":337,"state":"active+clean","last_fresh":"2024-03-11T21:39:45.769860+0000","last_change":"2024-03-11T21:35:27.606664+0000","last_active":"2024-03-11T21:39:45.769860+0000","last_peered":"2024-03-11T21:39:45.769860+0000","last_clean":"2024-03-11T21:39:45.769860+0000","last_became_active":"2024-03-11T21:34:41.136946+0000","last_became_peered":"2024-03-11T21:34:41.136946+0000","last_unstale":"2024-03-11T21:39:45.769860+0000","last_undegraded":"2024-03-11T21:39:45.769860+0000","last_fullsized":"2024-03-11T21:39:45.769860+0000","mapping_epoch":226,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":227,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:35:27.606551+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:43.559452+0000","last_clean_scrub_stamp":"2024-03-11T21:35:27.606551+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,7],"acting":[2,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":499,"reported_epoch":336,"state":"active+clean","last_fresh":"2024-03-11T21:39:43.418517+0000","last_change":"2024-03-11T21:36:35.076782+0000","last_active":"2024-03-11T21:39:43.418517+0000","last_peered":"2024-03-11T21:39:43.418517+0000","last_clean":"2024-03-11T21:39:43.418517+0000","last_became_active":"2024-03-11T21:36:35.076335+0000","last_became_peered":"2024-03-11T21:36:35.076335+0000","last_unstale":"2024-03-11T21:39:43.418517+0000","last_undegraded":"2024-03-11T21:39:43.418517+0000","last_fullsized":"2024-03-11T21:39:43.418517+0000","mapping_epoch":277,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":278,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:34:33.082577+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:41.544550+0000","last_clean_scrub_stamp":"2024-03-11T21:34:33.082577+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":35,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.10","version":"0'0","reported_seq":596,"reported_epoch":337,"state":"active+clean","last_fresh":"2024-03-11T21:39:45.770615+0000","last_change":"2024-03-11T21:35:37.694623+0000","last_active":"2024-03-11T21:39:45.770615+0000","last_peered":"2024-03-11T21:39:45.770615+0000","last_clean":"2024-03-11T21:39:45.770615+0000","last_became_active":"2024-03-11T21:25:38.533138+0000","last_became_peered":"2024-03-11T21:25:38.533138+0000","last_unstale":"2024-03-11T21:39:45.770615+0000","last_undegraded":"2024-03-11T21:39:45.770615+0000","last_fullsized":"2024-03-11T21:39:45.770615+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:35:37.694533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:37.512018+0000","last_clean_scrub_stamp":"2024-03-11T21:35:37.694533+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,0],"acting":[2,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.16","version":"260'15","reported_seq":665,"reported_epoch":336,"state":"active+clean","last_fresh":"2024-03-11T21:39:43.381760+0000","last_change":"2024-03-11T21:36:35.080158+0000","last_active":"2024-03-11T21:39:43.381760+0000","last_peered":"2024-03-11T21:39:43.381760+0000","last_clean":"2024-03-11T21:39:43.381760+0000","last_became_active":"2024-03-11T21:36:35.056862+0000","last_became_peered":"2024-03-11T21:36:35.056862+0000","last_unstale":"2024-03-11T21:39:43.381760+0000","last_undegraded":"2024-03-11T21:39:43.381760+0000","last_fullsized":"2024-03-11T21:39:43.381760+0000","mapping_epoch":277,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":278,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'9","last_scrub_stamp":"2024-03-11T21:35:12.741225+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:39.525554+0000","last_clean_scrub_stamp":"2024-03-11T21:35:12.741225+0000","objects_scrubbed":4,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":11,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.4","version":"337'14465","reported_seq":19532,"reported_epoch":337,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-03-11T21:39:48.269428+0000","last_change":"2024-03-11T21:38:21.285644+0000","last_active":"2024-03-11T21:39:48.269428+0000","last_peered":"2024-03-11T21:39:48.269428+0000","last_clean":"2024-03-11T21:35:44.669493+0000","last_became_active":"2024-03-11T21:36:36.506468+0000","last_became_peered":"2024-03-11T21:36:36.506468+0000","last_unstale":"2024-03-11T21:39:48.269428+0000","last_undegraded":"2024-03-11T21:36:36.069183+0000","last_fullsized":"2024-03-11T21:36:36.038484+0000","mapping_epoch":278,"log_start":"311'9400","ondisk_log_start":"311'9400","created":259,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:35:44.669493+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:35:44.669493+0000","last_clean_scrub_stamp":"2024-03-11T21:35:44.669493+0000","objects_scrubbed":0,"log_size":5065,"log_dups_size":0,"ondisk_log_size":5065,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19070976,"num_objects":291,"num_object_clones":0,"num_object_copies":873,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2366,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":291,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14465,"num_write_kb":472192,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2450,"num_bytes_recovered":78577664,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,3],"acting":[1,4],"avail_no_missing":["1","4"],"object_location_counts":[{"shards":"1,4","objects":-2075}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.b","version":"0'0","reported_seq":487,"reported_epoch":336,"state":"active+clean","last_fresh":"2024-03-11T21:39:43.430074+0000","last_change":"2024-03-11T21:36:35.074815+0000","last_active":"2024-03-11T21:39:43.430074+0000","last_peered":"2024-03-11T21:39:43.430074+0000","last_clean":"2024-03-11T21:39:43.430074+0000","last_became_active":"2024-03-11T21:36:35.072147+0000","last_became_peered":"2024-03-11T21:36:35.072147+0000","last_unstale":"2024-03-11T21:39:43.430074+0000","last_undegraded":"2024-03-11T21:39:43.430074+0000","last_fullsized":"2024-03-11T21:39:43.430074+0000","mapping_epoch":277,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":278,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:32:22.598150+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:43.559452+0000","last_clean_scrub_stamp":"2024-03-11T21:32:22.598150+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":57,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.f","version":"67'2","reported_seq":609,"reported_epoch":337,"state":"active+clean","last_fresh":"2024-03-11T21:39:45.754683+0000","last_change":"2024-03-11T21:35:36.613667+0000","last_active":"2024-03-11T21:39:45.754683+0000","last_peered":"2024-03-11T21:39:45.754683+0000","last_clean":"2024-03-11T21:39:45.754683+0000","last_became_active":"2024-03-11T21:25:38.537711+0000","last_became_peered":"2024-03-11T21:25:38.537711+0000","last_unstale":"2024-03-11T21:39:45.754683+0000","last_undegraded":"2024-03-11T21:39:45.754683+0000","last_fullsized":"2024-03-11T21:39:45.754683+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'2","last_scrub_stamp":"2024-03-11T21:35:36.613516+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:37.512018+0000","last_clean_scrub_stamp":"2024-03-11T21:35:36.613516+0000","objects_scrubbed":2,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":92,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":10,"num_read_kb":10,"num_write":4,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,7],"acting":[4,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.9","version":"260'23","reported_seq":640,"reported_epoch":337,"state":"active+clean","last_fresh":"2024-03-11T21:39:45.753013+0000","last_change":"2024-03-11T21:36:35.089034+0000","last_active":"2024-03-11T21:39:45.753013+0000","last_peered":"2024-03-11T21:39:45.753013+0000","last_clean":"2024-03-11T21:39:45.753013+0000","last_became_active":"2024-03-11T21:36:35.059930+0000","last_became_peered":"2024-03-11T21:36:35.059930+0000","last_unstale":"2024-03-11T21:39:45.753013+0000","last_undegraded":"2024-03-11T21:39:45.753013+0000","last_fullsized":"2024-03-11T21:39:45.753013+0000","mapping_epoch":277,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":278,"parent":"0.0","parent_split_bits":0,"last_scrub":"146'17","last_scrub_stamp":"2024-03-11T21:34:26.765075+0000","last_deep_scrub":"146'17","last_deep_scrub_stamp":"2024-03-11T21:34:26.765075+0000","last_clean_scrub_stamp":"2024-03-11T21:34:26.765075+0000","objects_scrubbed":5,"log_size":23,"log_dups_size":0,"ondisk_log_size":23,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":34,"num_read_kb":25,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,3],"acting":[4,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":588,"reported_epoch":337,"state":"active+clean","last_fresh":"2024-03-11T21:39:45.770733+0000","last_change":"2024-03-11T21:35:16.643421+0000","last_active":"2024-03-11T21:39:45.770733+0000","last_peered":"2024-03-11T21:39:45.770733+0000","last_clean":"2024-03-11T21:39:45.770733+0000","last_became_active":"2024-03-11T21:25:42.573987+0000","last_became_peered":"2024-03-11T21:25:42.573987+0000","last_unstale":"2024-03-11T21:39:45.770733+0000","last_undegraded":"2024-03-11T21:39:45.770733+0000","last_fullsized":"2024-03-11T21:39:45.770733+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:35:16.643311+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:41.544550+0000","last_clean_scrub_stamp":"2024-03-11T21:35:16.643311+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"9.f","version":"337'14047","reported_seq":14216,"reported_epoch":337,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-03-11T21:39:47.494938+0000","last_change":"2024-03-11T21:36:36.519822+0000","last_active":"2024-03-11T21:39:47.494938+0000","last_peered":"2024-03-11T21:39:47.494938+0000","last_clean":"2024-03-11T21:35:44.669493+0000","last_became_active":"2024-03-11T21:36:36.508836+0000","last_became_peered":"2024-03-11T21:36:36.508836+0000","last_unstale":"2024-03-11T21:39:47.494938+0000","last_undegraded":"2024-03-11T21:36:36.176770+0000","last_fullsized":"2024-03-11T21:36:36.038651+0000","mapping_epoch":278,"log_start":"308'8800","ondisk_log_start":"308'8800","created":259,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:35:44.669493+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:35:44.669493+0000","last_clean_scrub_stamp":"2024-03-11T21:35:44.669493+0000","objects_scrubbed":0,"log_size":5247,"log_dups_size":0,"ondisk_log_size":5247,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":21299200,"num_objects":325,"num_object_clones":0,"num_object_copies":975,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4738,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":325,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14047,"num_write_kb":459904,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":65536,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3],"acting":[5,1],"avail_no_missing":["5","1"],"object_location_counts":[{"shards":"1,5","objects":-4413}],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.0","version":"0'0","reported_seq":584,"reported_epoch":336,"state":"active+clean","last_fresh":"2024-03-11T21:39:43.380868+0000","last_change":"2024-03-11T21:36:35.058390+0000","last_active":"2024-03-11T21:39:43.380868+0000","last_peered":"2024-03-11T21:39:43.380868+0000","last_clean":"2024-03-11T21:39:43.380868+0000","last_became_active":"2024-03-11T21:36:35.058054+0000","last_became_peered":"2024-03-11T21:36:35.058054+0000","last_unstale":"2024-03-11T21:39:43.380868+0000","last_undegraded":"2024-03-11T21:39:43.380868+0000","last_fullsized":"2024-03-11T21:39:43.380868+0000","mapping_epoch":277,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":278,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:35:04.626668+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:29:39.498784+0000","last_clean_scrub_stamp":"2024-03-11T21:35:04.626668+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":39,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":609,"reported_epoch":337,"state":"active+clean","last_fresh":"2024-03-11T21:39:45.773332+0000","last_change":"2024-03-11T21:35:33.608063+0000","last_active":"2024-03-11T21:39:45.773332+0000","last_peered":"2024-03-11T21:39:45.773332+0000","last_clean":"2024-03-11T21:39:45.773332+0000","last_became_active":"2024-03-11T21:34:41.150913+0000","last_became_peered":"2024-03-11T21:34:41.150913+0000","last_unstale":"2024-03-11T21:39:45.773332+0000","last_undegraded":"2024-03-11T21:39:45.773332+0000","last_fullsized":"2024-03-11T21:39:45.773332+0000","mapping_epoch":226,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":227,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:35:33.607969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:37.512018+0000","last_clean_scrub_stamp":"2024-03-11T21:35:33.607969+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,7],"acting":[1,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.2","version":"260'25","reported_seq":628,"reported_epoch":337,"state":"active+clean","last_fresh":"2024-03-11T21:39:45.768862+0000","last_change":"2024-03-11T21:35:17.666749+0000","last_active":"2024-03-11T21:39:45.768862+0000","last_peered":"2024-03-11T21:39:45.768862+0000","last_clean":"2024-03-11T21:39:45.768862+0000","last_became_active":"2024-03-11T21:25:40.548833+0000","last_became_peered":"2024-03-11T21:25:40.548833+0000","last_unstale":"2024-03-11T21:39:45.768862+0000","last_undegraded":"2024-03-11T21:39:45.768862+0000","last_fullsized":"2024-03-11T21:39:45.768862+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"94'15","last_scrub_stamp":"2024-03-11T21:35:17.666498+0000","last_deep_scrub":"94'15","last_deep_scrub_stamp":"2024-03-11T21:26:49.993060+0000","last_clean_scrub_stamp":"2024-03-11T21:35:17.666498+0000","objects_scrubbed":5,"log_size":25,"log_dups_size":0,"ondisk_log_size":25,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":20,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":25,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.3","version":"74'8","reported_seq":765,"reported_epoch":336,"state":"active+clean","last_fresh":"2024-03-11T21:39:43.382226+0000","last_change":"2024-03-11T21:35:31.695369+0000","last_active":"2024-03-11T21:39:43.382226+0000","last_peered":"2024-03-11T21:39:43.382226+0000","last_clean":"2024-03-11T21:39:43.382226+0000","last_became_active":"2024-03-11T21:25:42.578161+0000","last_became_peered":"2024-03-11T21:25:42.578161+0000","last_unstale":"2024-03-11T21:39:43.382226+0000","last_undegraded":"2024-03-11T21:39:43.382226+0000","last_fullsized":"2024-03-11T21:39:43.382226+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'8","last_scrub_stamp":"2024-03-11T21:35:31.695263+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:41.544550+0000","last_clean_scrub_stamp":"2024-03-11T21:35:31.695263+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,5],"acting":[0,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.9","version":"336'14211","reported_seq":14376,"reported_epoch":336,"state":"active+clean","last_fresh":"2024-03-11T21:39:43.711362+0000","last_change":"2024-03-11T21:35:45.696767+0000","last_active":"2024-03-11T21:39:43.711362+0000","last_peered":"2024-03-11T21:39:43.711362+0000","last_clean":"2024-03-11T21:39:43.711362+0000","last_became_active":"2024-03-11T21:35:45.696379+0000","last_became_peered":"2024-03-11T21:35:45.696379+0000","last_unstale":"2024-03-11T21:39:43.711362+0000","last_undegraded":"2024-03-11T21:39:43.711362+0000","last_fullsized":"2024-03-11T21:39:43.711362+0000","mapping_epoch":259,"log_start":"305'8600","ondisk_log_start":"305'8600","created":259,"last_epoch_clean":260,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:35:44.669493+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:35:44.669493+0000","last_clean_scrub_stamp":"2024-03-11T21:35:44.669493+0000","objects_scrubbed":0,"log_size":5611,"log_dups_size":0,"ondisk_log_size":5611,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":37945344,"num_objects":579,"num_object_clones":0,"num_object_copies":1737,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":579,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14211,"num_write_kb":473280,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,5],"acting":[0,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":525,"reported_epoch":336,"state":"active+clean","last_fresh":"2024-03-11T21:39:43.406600+0000","last_change":"2024-03-11T21:36:35.068977+0000","last_active":"2024-03-11T21:39:43.406600+0000","last_peered":"2024-03-11T21:39:43.406600+0000","last_clean":"2024-03-11T21:39:43.406600+0000","last_became_active":"2024-03-11T21:36:35.067288+0000","last_became_peered":"2024-03-11T21:36:35.067288+0000","last_unstale":"2024-03-11T21:39:43.406600+0000","last_undegraded":"2024-03-11T21:39:43.406600+0000","last_fullsized":"2024-03-11T21:39:43.406600+0000","mapping_epoch":277,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":278,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:35:09.985377+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:28:30.563712+0000","last_clean_scrub_stamp":"2024-03-11T21:35:09.985377+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,7],"acting":[3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":604,"reported_epoch":337,"state":"active+clean","last_fresh":"2024-03-11T21:39:45.764674+0000","last_change":"2024-03-11T21:35:40.116979+0000","last_active":"2024-03-11T21:39:45.764674+0000","last_peered":"2024-03-11T21:39:45.764674+0000","last_clean":"2024-03-11T21:39:45.764674+0000","last_became_active":"2024-03-11T21:25:38.538540+0000","last_became_peered":"2024-03-11T21:25:38.538540+0000","last_unstale":"2024-03-11T21:39:45.764674+0000","last_undegraded":"2024-03-11T21:39:45.764674+0000","last_fullsized":"2024-03-11T21:39:45.764674+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:35:40.116877+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:37.512018+0000","last_clean_scrub_stamp":"2024-03-11T21:35:40.116877+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.4","version":"280'42","reported_seq":846,"reported_epoch":337,"state":"active+clean","last_fresh":"2024-03-11T21:39:45.772645+0000","last_change":"2024-03-11T21:36:35.095935+0000","last_active":"2024-03-11T21:39:45.772645+0000","last_peered":"2024-03-11T21:39:45.772645+0000","last_clean":"2024-03-11T21:39:45.772645+0000","last_became_active":"2024-03-11T21:36:35.073430+0000","last_became_peered":"2024-03-11T21:36:35.073430+0000","last_unstale":"2024-03-11T21:39:45.772645+0000","last_undegraded":"2024-03-11T21:39:45.772645+0000","last_fullsized":"2024-03-11T21:39:45.772645+0000","mapping_epoch":277,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":278,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'28","last_scrub_stamp":"2024-03-11T21:34:19.513252+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:39.525554+0000","last_clean_scrub_stamp":"2024-03-11T21:34:19.513252+0000","objects_scrubbed":10,"log_size":42,"log_dups_size":0,"ondisk_log_size":42,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":358,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":66,"num_read_kb":51,"num_write":38,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3],"acting":[1,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":587,"reported_epoch":336,"state":"active+clean","last_fresh":"2024-03-11T21:39:43.382894+0000","last_change":"2024-03-11T21:35:32.651731+0000","last_active":"2024-03-11T21:39:43.382894+0000","last_peered":"2024-03-11T21:39:43.382894+0000","last_clean":"2024-03-11T21:39:43.382894+0000","last_became_active":"2024-03-11T21:25:42.572673+0000","last_became_peered":"2024-03-11T21:25:42.572673+0000","last_unstale":"2024-03-11T21:39:43.382894+0000","last_undegraded":"2024-03-11T21:39:43.382894+0000","last_fullsized":"2024-03-11T21:39:43.382894+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:35:32.651652+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:41.544550+0000","last_clean_scrub_stamp":"2024-03-11T21:35:32.651652+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.a","version":"337'14465","reported_seq":14631,"reported_epoch":337,"state":"active+clean","last_fresh":"2024-03-11T21:39:47.519451+0000","last_change":"2024-03-11T21:35:45.698558+0000","last_active":"2024-03-11T21:39:47.519451+0000","last_peered":"2024-03-11T21:39:47.519451+0000","last_clean":"2024-03-11T21:39:47.519451+0000","last_became_active":"2024-03-11T21:35:45.698238+0000","last_became_peered":"2024-03-11T21:35:45.698238+0000","last_unstale":"2024-03-11T21:39:47.519451+0000","last_undegraded":"2024-03-11T21:39:47.519451+0000","last_fullsized":"2024-03-11T21:39:47.519451+0000","mapping_epoch":259,"log_start":"311'9200","ondisk_log_start":"311'9200","created":259,"last_epoch_clean":260,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:35:44.669493+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:35:44.669493+0000","last_clean_scrub_stamp":"2024-03-11T21:35:44.669493+0000","objects_scrubbed":0,"log_size":5265,"log_dups_size":0,"ondisk_log_size":5265,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":21168128,"num_objects":323,"num_object_clones":0,"num_object_copies":969,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":323,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14465,"num_write_kb":473216,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,7],"acting":[5,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.5","version":"0'0","reported_seq":250,"reported_epoch":337,"state":"active+clean","last_fresh":"2024-03-11T21:39:45.757540+0000","last_change":"2024-03-11T21:36:35.069989+0000","last_active":"2024-03-11T21:39:45.757540+0000","last_peered":"2024-03-11T21:39:45.757540+0000","last_clean":"2024-03-11T21:39:45.757540+0000","last_became_active":"2024-03-11T21:36:35.069358+0000","last_became_peered":"2024-03-11T21:36:35.069358+0000","last_unstale":"2024-03-11T21:39:45.757540+0000","last_undegraded":"2024-03-11T21:39:45.757540+0000","last_fullsized":"2024-03-11T21:39:45.757540+0000","mapping_epoch":277,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":278,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:35:19.841632+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:43.559452+0000","last_clean_scrub_stamp":"2024-03-11T21:35:19.841632+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,3],"acting":[7,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.0","version":"164'6","reported_seq":1451,"reported_epoch":337,"state":"active+clean","last_fresh":"2024-03-11T21:39:46.951601+0000","last_change":"2024-03-11T21:35:38.518243+0000","last_active":"2024-03-11T21:39:46.951601+0000","last_peered":"2024-03-11T21:39:46.951601+0000","last_clean":"2024-03-11T21:39:46.951601+0000","last_became_active":"2024-03-11T21:25:38.536890+0000","last_became_peered":"2024-03-11T21:25:38.536890+0000","last_unstale":"2024-03-11T21:39:46.951601+0000","last_undegraded":"2024-03-11T21:39:46.951601+0000","last_fullsized":"2024-03-11T21:39:46.951601+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"164'6","last_scrub_stamp":"2024-03-11T21:35:38.518097+0000","last_deep_scrub":"164'6","last_deep_scrub_stamp":"2024-03-11T21:32:17.597666+0000","last_clean_scrub_stamp":"2024-03-11T21:35:38.518097+0000","objects_scrubbed":1,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0.000753559,"stat_sum":{"num_bytes":389,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":842,"num_read_kb":837,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,6],"acting":[1,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":605,"reported_epoch":337,"state":"active+clean","last_fresh":"2024-03-11T21:39:45.769257+0000","last_change":"2024-03-11T21:36:35.057287+0000","last_active":"2024-03-11T21:39:45.769257+0000","last_peered":"2024-03-11T21:39:45.769257+0000","last_clean":"2024-03-11T21:39:45.769257+0000","last_became_active":"2024-03-11T21:36:35.056966+0000","last_became_peered":"2024-03-11T21:36:35.056966+0000","last_unstale":"2024-03-11T21:39:45.769257+0000","last_undegraded":"2024-03-11T21:39:45.769257+0000","last_fullsized":"2024-03-11T21:39:45.769257+0000","mapping_epoch":277,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":278,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:34:18.424010+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:26:48.997454+0000","last_clean_scrub_stamp":"2024-03-11T21:34:18.424010+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.7","version":"260'22","reported_seq":634,"reported_epoch":337,"state":"active+clean","last_fresh":"2024-03-11T21:39:45.766518+0000","last_change":"2024-03-11T21:35:37.508679+0000","last_active":"2024-03-11T21:39:45.766518+0000","last_peered":"2024-03-11T21:39:45.766518+0000","last_clean":"2024-03-11T21:39:45.766518+0000","last_became_active":"2024-03-11T21:25:40.541769+0000","last_became_peered":"2024-03-11T21:25:40.541769+0000","last_unstale":"2024-03-11T21:39:45.766518+0000","last_undegraded":"2024-03-11T21:39:45.766518+0000","last_fullsized":"2024-03-11T21:39:45.766518+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"97'18","last_scrub_stamp":"2024-03-11T21:35:37.508567+0000","last_deep_scrub":"97'18","last_deep_scrub_stamp":"2024-03-11T21:27:19.239404+0000","last_clean_scrub_stamp":"2024-03-11T21:35:37.508567+0000","objects_scrubbed":5,"log_size":22,"log_dups_size":0,"ondisk_log_size":22,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":36,"num_read_kb":25,"num_write":20,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":600,"reported_epoch":337,"state":"active+clean","last_fresh":"2024-03-11T21:39:45.771275+0000","last_change":"2024-03-11T21:35:22.632822+0000","last_active":"2024-03-11T21:39:45.771275+0000","last_peered":"2024-03-11T21:39:45.771275+0000","last_clean":"2024-03-11T21:39:45.771275+0000","last_became_active":"2024-03-11T21:34:41.143660+0000","last_became_peered":"2024-03-11T21:34:41.143660+0000","last_unstale":"2024-03-11T21:39:45.771275+0000","last_undegraded":"2024-03-11T21:39:45.771275+0000","last_fullsized":"2024-03-11T21:39:45.771275+0000","mapping_epoch":226,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":227,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:35:22.632707+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:41.544550+0000","last_clean_scrub_stamp":"2024-03-11T21:35:22.632707+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":18,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7],"acting":[2,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"9.8","version":"336'14244","reported_seq":16775,"reported_epoch":336,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-03-11T21:39:43.712044+0000","last_change":"2024-03-11T21:38:57.479443+0000","last_active":"2024-03-11T21:39:43.712044+0000","last_peered":"2024-03-11T21:39:43.712044+0000","last_clean":"2024-03-11T21:35:44.669493+0000","last_became_active":"2024-03-11T21:36:36.507652+0000","last_became_peered":"2024-03-11T21:36:36.507652+0000","last_unstale":"2024-03-11T21:39:43.712044+0000","last_undegraded":"2024-03-11T21:36:36.064628+0000","last_fullsized":"2024-03-11T21:36:36.036628+0000","mapping_epoch":278,"log_start":"305'8600","ondisk_log_start":"305'8600","created":259,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:35:44.669493+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:35:44.669493+0000","last_clean_scrub_stamp":"2024-03-11T21:35:44.669493+0000","objects_scrubbed":0,"log_size":5644,"log_dups_size":0,"ondisk_log_size":5644,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":33161216,"num_objects":506,"num_object_clones":0,"num_object_copies":1518,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3663,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":506,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14244,"num_write_kb":472000,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1183,"num_bytes_recovered":14483456,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[0,5],"avail_no_missing":["0","5"],"object_location_counts":[{"shards":"0,5","objects":-3157}],"blocked_by":[],"up_primary":3,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.7","version":"124'2","reported_seq":587,"reported_epoch":337,"state":"active+clean","last_fresh":"2024-03-11T21:39:45.766150+0000","last_change":"2024-03-11T21:36:35.059190+0000","last_active":"2024-03-11T21:39:45.766150+0000","last_peered":"2024-03-11T21:39:45.766150+0000","last_clean":"2024-03-11T21:39:45.766150+0000","last_became_active":"2024-03-11T21:36:35.058786+0000","last_became_peered":"2024-03-11T21:36:35.058786+0000","last_unstale":"2024-03-11T21:39:45.766150+0000","last_undegraded":"2024-03-11T21:39:45.766150+0000","last_fullsized":"2024-03-11T21:39:45.766150+0000","mapping_epoch":277,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":278,"parent":"0.0","parent_split_bits":0,"last_scrub":"124'2","last_scrub_stamp":"2024-03-11T21:35:16.414595+0000","last_deep_scrub":"124'2","last_deep_scrub_stamp":"2024-03-11T21:29:34.956908+0000","last_clean_scrub_stamp":"2024-03-11T21:35:16.414595+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":42,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,4],"acting":[5,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2","version":"71'2","reported_seq":546,"reported_epoch":336,"state":"active+clean","last_fresh":"2024-03-11T21:39:43.417602+0000","last_change":"2024-03-11T21:36:35.071218+0000","last_active":"2024-03-11T21:39:43.417602+0000","last_peered":"2024-03-11T21:39:43.417602+0000","last_clean":"2024-03-11T21:39:43.417602+0000","last_became_active":"2024-03-11T21:36:35.070783+0000","last_became_peered":"2024-03-11T21:36:35.070783+0000","last_unstale":"2024-03-11T21:39:43.417602+0000","last_undegraded":"2024-03-11T21:39:43.417602+0000","last_fullsized":"2024-03-11T21:39:43.417602+0000","mapping_epoch":277,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":278,"parent":"0.0","parent_split_bits":0,"last_scrub":"71'2","last_scrub_stamp":"2024-03-11T21:32:21.543053+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:37.512018+0000","last_clean_scrub_stamp":"2024-03-11T21:32:21.543053+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014206449999999999,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":614,"reported_epoch":337,"state":"active+clean","last_fresh":"2024-03-11T21:39:45.764505+0000","last_change":"2024-03-11T21:35:08.391825+0000","last_active":"2024-03-11T21:39:45.764505+0000","last_peered":"2024-03-11T21:39:45.764505+0000","last_clean":"2024-03-11T21:39:45.764505+0000","last_became_active":"2024-03-11T21:34:41.146672+0000","last_became_peered":"2024-03-11T21:34:41.146672+0000","last_unstale":"2024-03-11T21:39:45.764505+0000","last_undegraded":"2024-03-11T21:39:45.764505+0000","last_fullsized":"2024-03-11T21:39:45.764505+0000","mapping_epoch":226,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":227,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:35:08.391713+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:37.512018+0000","last_clean_scrub_stamp":"2024-03-11T21:35:08.391713+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"87'128","reported_seq":712,"reported_epoch":337,"state":"active+clean","last_fresh":"2024-03-11T21:39:45.756829+0000","last_change":"2024-03-11T21:35:21.798272+0000","last_active":"2024-03-11T21:39:45.756829+0000","last_peered":"2024-03-11T21:39:45.756829+0000","last_clean":"2024-03-11T21:39:45.756829+0000","last_became_active":"2024-03-11T21:34:44.164569+0000","last_became_peered":"2024-03-11T21:34:44.164569+0000","last_unstale":"2024-03-11T21:39:45.756829+0000","last_undegraded":"2024-03-11T21:39:45.756829+0000","last_fullsized":"2024-03-11T21:39:45.756829+0000","mapping_epoch":229,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":230,"parent":"0.0","parent_split_bits":0,"last_scrub":"87'128","last_scrub_stamp":"2024-03-11T21:35:21.798144+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:24.261974+0000","last_clean_scrub_stamp":"2024-03-11T21:35:21.798144+0000","objects_scrubbed":2,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":198,"num_read_kb":294,"num_write":179,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":2951840,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.5","version":"260'28","reported_seq":819,"reported_epoch":337,"state":"active+clean","last_fresh":"2024-03-11T21:39:45.766692+0000","last_change":"2024-03-11T21:39:45.216634+0000","last_active":"2024-03-11T21:39:45.766692+0000","last_peered":"2024-03-11T21:39:45.766692+0000","last_clean":"2024-03-11T21:39:45.766692+0000","last_became_active":"2024-03-11T21:25:40.761457+0000","last_became_peered":"2024-03-11T21:25:40.761457+0000","last_unstale":"2024-03-11T21:39:45.766692+0000","last_undegraded":"2024-03-11T21:39:45.766692+0000","last_fullsized":"2024-03-11T21:39:45.766692+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"260'28","last_scrub_stamp":"2024-03-11T21:39:45.216468+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:39.525554+0000","last_clean_scrub_stamp":"2024-03-11T21:39:45.216468+0000","objects_scrubbed":8,"log_size":28,"log_dups_size":0,"ondisk_log_size":28,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:40:57.658763+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":154,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":43,"num_read_kb":33,"num_write":25,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":241,"reported_epoch":337,"state":"active+clean","last_fresh":"2024-03-11T21:39:45.757356+0000","last_change":"2024-03-11T21:35:24.841170+0000","last_active":"2024-03-11T21:39:45.757356+0000","last_peered":"2024-03-11T21:39:45.757356+0000","last_clean":"2024-03-11T21:39:45.757356+0000","last_became_active":"2024-03-11T21:34:41.139782+0000","last_became_peered":"2024-03-11T21:34:41.139782+0000","last_unstale":"2024-03-11T21:39:45.757356+0000","last_undegraded":"2024-03-11T21:39:45.757356+0000","last_fullsized":"2024-03-11T21:39:45.757356+0000","mapping_epoch":226,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":227,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:35:24.841041+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:32:21.266672+0000","last_clean_scrub_stamp":"2024-03-11T21:35:24.841041+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,5],"acting":[7,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.b","version":"336'13899","reported_seq":14064,"reported_epoch":336,"state":"active+clean","last_fresh":"2024-03-11T21:39:43.717827+0000","last_change":"2024-03-11T21:35:45.697213+0000","last_active":"2024-03-11T21:39:43.717827+0000","last_peered":"2024-03-11T21:39:43.717827+0000","last_clean":"2024-03-11T21:39:43.717827+0000","last_became_active":"2024-03-11T21:35:45.696869+0000","last_became_peered":"2024-03-11T21:35:45.696869+0000","last_unstale":"2024-03-11T21:39:43.717827+0000","last_undegraded":"2024-03-11T21:39:43.717827+0000","last_fullsized":"2024-03-11T21:39:43.717827+0000","mapping_epoch":259,"log_start":"304'8300","ondisk_log_start":"304'8300","created":259,"last_epoch_clean":260,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:35:44.669493+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:35:44.669493+0000","last_clean_scrub_stamp":"2024-03-11T21:35:44.669493+0000","objects_scrubbed":0,"log_size":5599,"log_dups_size":0,"ondisk_log_size":5599,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":35848192,"num_objects":547,"num_object_clones":0,"num_object_copies":1641,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":547,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13899,"num_write_kb":462272,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,5],"acting":[0,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.4","version":"0'0","reported_seq":586,"reported_epoch":337,"state":"active+clean","last_fresh":"2024-03-11T21:39:45.773563+0000","last_change":"2024-03-11T21:36:35.073305+0000","last_active":"2024-03-11T21:39:45.773563+0000","last_peered":"2024-03-11T21:39:45.773563+0000","last_clean":"2024-03-11T21:39:45.773563+0000","last_became_active":"2024-03-11T21:36:35.072824+0000","last_became_peered":"2024-03-11T21:36:35.072824+0000","last_unstale":"2024-03-11T21:39:45.773563+0000","last_undegraded":"2024-03-11T21:39:45.773563+0000","last_fullsized":"2024-03-11T21:39:45.773563+0000","mapping_epoch":277,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":278,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:34:23.531064+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:43.559452+0000","last_clean_scrub_stamp":"2024-03-11T21:34:23.531064+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":611,"reported_epoch":336,"state":"active+clean","last_fresh":"2024-03-11T21:39:43.380246+0000","last_change":"2024-03-11T21:36:35.070626+0000","last_active":"2024-03-11T21:39:43.380246+0000","last_peered":"2024-03-11T21:39:43.380246+0000","last_clean":"2024-03-11T21:39:43.380246+0000","last_became_active":"2024-03-11T21:36:35.069452+0000","last_became_peered":"2024-03-11T21:36:35.069452+0000","last_unstale":"2024-03-11T21:39:43.380246+0000","last_undegraded":"2024-03-11T21:39:43.380246+0000","last_fullsized":"2024-03-11T21:39:43.380246+0000","mapping_epoch":277,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":278,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:35:13.741947+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:37.512018+0000","last_clean_scrub_stamp":"2024-03-11T21:35:13.741947+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0.00089342499999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":233,"reported_epoch":337,"state":"active+clean","last_fresh":"2024-03-11T21:39:45.756762+0000","last_change":"2024-03-11T21:34:41.150606+0000","last_active":"2024-03-11T21:39:45.756762+0000","last_peered":"2024-03-11T21:39:45.756762+0000","last_clean":"2024-03-11T21:39:45.756762+0000","last_became_active":"2024-03-11T21:34:41.149219+0000","last_became_peered":"2024-03-11T21:34:41.149219+0000","last_unstale":"2024-03-11T21:39:45.756762+0000","last_undegraded":"2024-03-11T21:39:45.756762+0000","last_fullsized":"2024-03-11T21:39:45.756762+0000","mapping_epoch":226,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":227,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:34:31.628014+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:37.512018+0000","last_clean_scrub_stamp":"2024-03-11T21:34:31.628014+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,0],"acting":[7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.6","version":"260'24","reported_seq":647,"reported_epoch":336,"state":"active+clean","last_fresh":"2024-03-11T21:39:43.381671+0000","last_change":"2024-03-11T21:35:19.753433+0000","last_active":"2024-03-11T21:39:43.381671+0000","last_peered":"2024-03-11T21:39:43.381671+0000","last_clean":"2024-03-11T21:39:43.381671+0000","last_became_active":"2024-03-11T21:32:22.796486+0000","last_became_peered":"2024-03-11T21:32:22.796486+0000","last_unstale":"2024-03-11T21:39:43.381671+0000","last_undegraded":"2024-03-11T21:39:43.381671+0000","last_fullsized":"2024-03-11T21:39:43.381671+0000","mapping_epoch":166,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":167,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'12","last_scrub_stamp":"2024-03-11T21:35:19.753224+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:39.525554+0000","last_clean_scrub_stamp":"2024-03-11T21:35:19.753224+0000","objects_scrubbed":6,"log_size":24,"log_dups_size":0,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":36,"num_read_kb":30,"num_write":24,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":588,"reported_epoch":337,"state":"active+clean","last_fresh":"2024-03-11T21:39:45.766074+0000","last_change":"2024-03-11T21:36:07.451482+0000","last_active":"2024-03-11T21:39:45.766074+0000","last_peered":"2024-03-11T21:39:45.766074+0000","last_clean":"2024-03-11T21:39:45.766074+0000","last_became_active":"2024-03-11T21:25:42.572324+0000","last_became_peered":"2024-03-11T21:25:42.572324+0000","last_unstale":"2024-03-11T21:39:45.766074+0000","last_undegraded":"2024-03-11T21:39:45.766074+0000","last_fullsized":"2024-03-11T21:39:45.766074+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:36:07.451102+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:41.544550+0000","last_clean_scrub_stamp":"2024-03-11T21:36:07.451102+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":54,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.e","version":"337'13927","reported_seq":14094,"reported_epoch":337,"state":"active+clean","last_fresh":"2024-03-11T21:39:47.351795+0000","last_change":"2024-03-11T21:35:45.697331+0000","last_active":"2024-03-11T21:39:47.351795+0000","last_peered":"2024-03-11T21:39:47.351795+0000","last_clean":"2024-03-11T21:39:47.351795+0000","last_became_active":"2024-03-11T21:35:45.696948+0000","last_became_peered":"2024-03-11T21:35:45.696948+0000","last_unstale":"2024-03-11T21:39:47.351795+0000","last_undegraded":"2024-03-11T21:39:47.351795+0000","last_fullsized":"2024-03-11T21:39:47.351795+0000","mapping_epoch":259,"log_start":"312'9300","ondisk_log_start":"312'9300","created":259,"last_epoch_clean":260,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:35:44.669493+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:35:44.669493+0000","last_clean_scrub_stamp":"2024-03-11T21:35:44.669493+0000","objects_scrubbed":0,"log_size":4627,"log_dups_size":0,"ondisk_log_size":4627,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":23134208,"num_objects":353,"num_object_clones":0,"num_object_copies":1059,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":353,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13927,"num_write_kb":456960,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":593,"reported_epoch":337,"state":"active+clean","last_fresh":"2024-03-11T21:39:45.765736+0000","last_change":"2024-03-11T21:36:18.529506+0000","last_active":"2024-03-11T21:39:45.765736+0000","last_peered":"2024-03-11T21:39:45.765736+0000","last_clean":"2024-03-11T21:39:45.765736+0000","last_became_active":"2024-03-11T21:25:45.301416+0000","last_became_peered":"2024-03-11T21:25:45.301416+0000","last_unstale":"2024-03-11T21:39:45.765736+0000","last_undegraded":"2024-03-11T21:39:45.765736+0000","last_fullsized":"2024-03-11T21:39:45.765736+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:36:18.529370+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:43.559452+0000","last_clean_scrub_stamp":"2024-03-11T21:36:18.529370+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":54,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":241,"reported_epoch":337,"state":"active+clean","last_fresh":"2024-03-11T21:39:45.757632+0000","last_change":"2024-03-11T21:35:33.740615+0000","last_active":"2024-03-11T21:39:45.757632+0000","last_peered":"2024-03-11T21:39:45.757632+0000","last_clean":"2024-03-11T21:39:45.757632+0000","last_became_active":"2024-03-11T21:34:41.152721+0000","last_became_peered":"2024-03-11T21:34:41.152721+0000","last_unstale":"2024-03-11T21:39:45.757632+0000","last_undegraded":"2024-03-11T21:39:45.757632+0000","last_fullsized":"2024-03-11T21:39:45.757632+0000","mapping_epoch":226,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":227,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:35:33.740538+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:26:57.333669+0000","last_clean_scrub_stamp":"2024-03-11T21:35:33.740538+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.3","version":"260'37","reported_seq":698,"reported_epoch":336,"state":"active+clean","last_fresh":"2024-03-11T21:39:43.381808+0000","last_change":"2024-03-11T21:35:11.697314+0000","last_active":"2024-03-11T21:39:43.381808+0000","last_peered":"2024-03-11T21:39:43.381808+0000","last_clean":"2024-03-11T21:39:43.381808+0000","last_became_active":"2024-03-11T21:34:41.145607+0000","last_became_peered":"2024-03-11T21:34:41.145607+0000","last_unstale":"2024-03-11T21:39:43.381808+0000","last_undegraded":"2024-03-11T21:39:43.381808+0000","last_fullsized":"2024-03-11T21:39:43.381808+0000","mapping_epoch":226,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":227,"parent":"0.0","parent_split_bits":0,"last_scrub":"165'27","last_scrub_stamp":"2024-03-11T21:35:11.697140+0000","last_deep_scrub":"165'27","last_deep_scrub_stamp":"2024-03-11T21:32:21.576732+0000","last_clean_scrub_stamp":"2024-03-11T21:35:11.697140+0000","objects_scrubbed":8,"log_size":37,"log_dups_size":0,"ondisk_log_size":37,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":54,"num_read_kb":40,"num_write":32,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":660,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":605,"reported_epoch":337,"state":"active+clean","last_fresh":"2024-03-11T21:39:45.766463+0000","last_change":"2024-03-11T21:38:53.027632+0000","last_active":"2024-03-11T21:39:45.766463+0000","last_peered":"2024-03-11T21:39:45.766463+0000","last_clean":"2024-03-11T21:39:45.766463+0000","last_became_active":"2024-03-11T21:25:42.579045+0000","last_became_peered":"2024-03-11T21:25:42.579045+0000","last_unstale":"2024-03-11T21:39:45.766463+0000","last_undegraded":"2024-03-11T21:39:45.766463+0000","last_fullsized":"2024-03-11T21:39:45.766463+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:38:53.027520+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:38:53.027520+0000","last_clean_scrub_stamp":"2024-03-11T21:38:53.027520+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:40:12.157404+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.d","version":"337'14157","reported_seq":14324,"reported_epoch":337,"state":"active+clean","last_fresh":"2024-03-11T21:39:47.367212+0000","last_change":"2024-03-11T21:35:45.700060+0000","last_active":"2024-03-11T21:39:47.367212+0000","last_peered":"2024-03-11T21:39:47.367212+0000","last_clean":"2024-03-11T21:39:47.367212+0000","last_became_active":"2024-03-11T21:35:45.699674+0000","last_became_peered":"2024-03-11T21:35:45.699674+0000","last_unstale":"2024-03-11T21:39:47.367212+0000","last_undegraded":"2024-03-11T21:39:47.367212+0000","last_fullsized":"2024-03-11T21:39:47.367212+0000","mapping_epoch":259,"log_start":"312'9500","ondisk_log_start":"312'9500","created":259,"last_epoch_clean":260,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:35:44.669493+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:35:44.669493+0000","last_clean_scrub_stamp":"2024-03-11T21:35:44.669493+0000","objects_scrubbed":0,"log_size":4657,"log_dups_size":0,"ondisk_log_size":4657,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20185112,"num_objects":309,"num_object_clones":0,"num_object_copies":927,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":309,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":14157,"num_write_kb":462849,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,5],"acting":[7,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":591,"reported_epoch":337,"state":"active+clean","last_fresh":"2024-03-11T21:39:45.753514+0000","last_change":"2024-03-11T21:35:08.437783+0000","last_active":"2024-03-11T21:39:45.753514+0000","last_peered":"2024-03-11T21:39:45.753514+0000","last_clean":"2024-03-11T21:39:45.753514+0000","last_became_active":"2024-03-11T21:32:22.791439+0000","last_became_peered":"2024-03-11T21:32:22.791439+0000","last_unstale":"2024-03-11T21:39:45.753514+0000","last_undegraded":"2024-03-11T21:39:45.753514+0000","last_fullsized":"2024-03-11T21:39:45.753514+0000","mapping_epoch":166,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":167,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:35:08.437697+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:43.559452+0000","last_clean_scrub_stamp":"2024-03-11T21:35:08.437697+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":33,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,2],"acting":[4,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.6","version":"67'1","reported_seq":598,"reported_epoch":337,"state":"active+clean","last_fresh":"2024-03-11T21:39:45.766344+0000","last_change":"2024-03-11T21:35:30.581783+0000","last_active":"2024-03-11T21:39:45.766344+0000","last_peered":"2024-03-11T21:39:45.766344+0000","last_clean":"2024-03-11T21:39:45.766344+0000","last_became_active":"2024-03-11T21:25:38.536142+0000","last_became_peered":"2024-03-11T21:25:38.536142+0000","last_unstale":"2024-03-11T21:39:45.766344+0000","last_undegraded":"2024-03-11T21:39:45.766344+0000","last_fullsized":"2024-03-11T21:39:45.766344+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'1","last_scrub_stamp":"2024-03-11T21:35:30.581701+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:37.512018+0000","last_clean_scrub_stamp":"2024-03-11T21:35:30.581701+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,4],"acting":[1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.0","version":"260'38","reported_seq":594,"reported_epoch":336,"state":"active+clean","last_fresh":"2024-03-11T21:39:43.407321+0000","last_change":"2024-03-11T21:36:35.087771+0000","last_active":"2024-03-11T21:39:43.407321+0000","last_peered":"2024-03-11T21:39:43.407321+0000","last_clean":"2024-03-11T21:39:43.407321+0000","last_became_active":"2024-03-11T21:36:35.068071+0000","last_became_peered":"2024-03-11T21:36:35.068071+0000","last_unstale":"2024-03-11T21:39:43.407321+0000","last_undegraded":"2024-03-11T21:39:43.407321+0000","last_fullsized":"2024-03-11T21:39:43.407321+0000","mapping_epoch":277,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":278,"parent":"0.0","parent_split_bits":0,"last_scrub":"93'26","last_scrub_stamp":"2024-03-11T21:34:16.863984+0000","last_deep_scrub":"93'26","last_deep_scrub_stamp":"2024-03-11T21:26:40.680324+0000","last_clean_scrub_stamp":"2024-03-11T21:34:16.863984+0000","objects_scrubbed":8,"log_size":38,"log_dups_size":0,"ondisk_log_size":38,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":22,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":51,"num_read_kb":39,"num_write":32,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":610,"reported_epoch":337,"state":"active+clean","last_fresh":"2024-03-11T21:39:45.753096+0000","last_change":"2024-03-11T21:36:35.057830+0000","last_active":"2024-03-11T21:39:45.753096+0000","last_peered":"2024-03-11T21:39:45.753096+0000","last_clean":"2024-03-11T21:39:45.753096+0000","last_became_active":"2024-03-11T21:36:35.057517+0000","last_became_peered":"2024-03-11T21:36:35.057517+0000","last_unstale":"2024-03-11T21:39:45.753096+0000","last_undegraded":"2024-03-11T21:39:45.753096+0000","last_fullsized":"2024-03-11T21:39:45.753096+0000","mapping_epoch":277,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":278,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:35:12.411136+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:26:51.863621+0000","last_clean_scrub_stamp":"2024-03-11T21:35:12.411136+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,7],"acting":[4,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.c","version":"336'13711","reported_seq":23140,"reported_epoch":336,"state":"active+clean","last_fresh":"2024-03-11T21:39:44.128047+0000","last_change":"2024-03-11T21:38:30.644319+0000","last_active":"2024-03-11T21:39:44.128047+0000","last_peered":"2024-03-11T21:39:44.128047+0000","last_clean":"2024-03-11T21:39:44.128047+0000","last_became_active":"2024-03-11T21:38:30.610971+0000","last_became_peered":"2024-03-11T21:38:30.610971+0000","last_unstale":"2024-03-11T21:39:44.128047+0000","last_undegraded":"2024-03-11T21:39:44.128047+0000","last_fullsized":"2024-03-11T21:39:44.128047+0000","mapping_epoch":321,"log_start":"310'8844","ondisk_log_start":"310'8844","created":259,"last_epoch_clean":322,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:35:44.669493+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:35:44.669493+0000","last_clean_scrub_stamp":"2024-03-11T21:35:44.669493+0000","objects_scrubbed":0,"log_size":4867,"log_dups_size":0,"ondisk_log_size":4867,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":33488896,"num_objects":511,"num_object_clones":0,"num_object_copies":1533,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":511,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13711,"num_write_kb":455104,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4628,"num_bytes_recovered":303300608,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,7],"acting":[3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.3","version":"0'0","reported_seq":233,"reported_epoch":337,"state":"active+clean","last_fresh":"2024-03-11T21:39:45.757752+0000","last_change":"2024-03-11T21:34:41.143973+0000","last_active":"2024-03-11T21:39:45.757752+0000","last_peered":"2024-03-11T21:39:45.757752+0000","last_clean":"2024-03-11T21:39:45.757752+0000","last_became_active":"2024-03-11T21:34:41.142683+0000","last_became_peered":"2024-03-11T21:34:41.142683+0000","last_unstale":"2024-03-11T21:39:45.757752+0000","last_undegraded":"2024-03-11T21:39:45.757752+0000","last_fullsized":"2024-03-11T21:39:45.757752+0000","mapping_epoch":226,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":227,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:34:27.353700+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:29:42.485740+0000","last_clean_scrub_stamp":"2024-03-11T21:34:27.353700+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,2],"acting":[7,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":625,"reported_epoch":337,"state":"active+clean","last_fresh":"2024-03-11T21:39:45.767409+0000","last_change":"2024-03-11T21:38:55.048335+0000","last_active":"2024-03-11T21:39:45.767409+0000","last_peered":"2024-03-11T21:39:45.767409+0000","last_clean":"2024-03-11T21:39:45.767409+0000","last_became_active":"2024-03-11T21:34:41.145628+0000","last_became_peered":"2024-03-11T21:34:41.145628+0000","last_unstale":"2024-03-11T21:39:45.767409+0000","last_undegraded":"2024-03-11T21:39:45.767409+0000","last_fullsized":"2024-03-11T21:39:45.767409+0000","mapping_epoch":226,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":227,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:38:55.048231+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:37.512018+0000","last_clean_scrub_stamp":"2024-03-11T21:38:55.048231+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:40:07.136440+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,2],"acting":[6,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.1","version":"260'28","reported_seq":635,"reported_epoch":337,"state":"active+clean","last_fresh":"2024-03-11T21:39:45.752838+0000","last_change":"2024-03-11T21:35:33.644654+0000","last_active":"2024-03-11T21:39:45.752838+0000","last_peered":"2024-03-11T21:39:45.752838+0000","last_clean":"2024-03-11T21:39:45.752838+0000","last_became_active":"2024-03-11T21:25:40.546219+0000","last_became_peered":"2024-03-11T21:25:40.546219+0000","last_unstale":"2024-03-11T21:39:45.752838+0000","last_undegraded":"2024-03-11T21:39:45.752838+0000","last_fullsized":"2024-03-11T21:39:45.752838+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'14","last_scrub_stamp":"2024-03-11T21:35:33.644503+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:39.525554+0000","last_clean_scrub_stamp":"2024-03-11T21:35:33.644503+0000","objects_scrubbed":7,"log_size":28,"log_dups_size":0,"ondisk_log_size":28,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":42,"num_read_kb":35,"num_write":28,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6],"acting":[4,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.0","version":"280'11","reported_seq":691,"reported_epoch":336,"state":"active+clean","last_fresh":"2024-03-11T21:39:43.398553+0000","last_change":"2024-03-11T21:36:35.064085+0000","last_active":"2024-03-11T21:39:43.398553+0000","last_peered":"2024-03-11T21:39:43.398553+0000","last_clean":"2024-03-11T21:39:43.398553+0000","last_became_active":"2024-03-11T21:36:35.055250+0000","last_became_peered":"2024-03-11T21:36:35.055250+0000","last_unstale":"2024-03-11T21:39:43.398553+0000","last_undegraded":"2024-03-11T21:39:43.398553+0000","last_fullsized":"2024-03-11T21:39:43.398553+0000","mapping_epoch":277,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":278,"parent":"0.0","parent_split_bits":0,"last_scrub":"98'9","last_scrub_stamp":"2024-03-11T21:35:21.143578+0000","last_deep_scrub":"98'9","last_deep_scrub_stamp":"2024-03-11T21:27:29.745711+0000","last_clean_scrub_stamp":"2024-03-11T21:35:21.143578+0000","objects_scrubbed":1,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.3","version":"337'13969","reported_seq":14144,"reported_epoch":337,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-03-11T21:39:47.328903+0000","last_change":"2024-03-11T21:36:36.544317+0000","last_active":"2024-03-11T21:39:47.328903+0000","last_peered":"2024-03-11T21:39:47.328903+0000","last_clean":"2024-03-11T21:35:44.669493+0000","last_became_active":"2024-03-11T21:36:36.470537+0000","last_became_peered":"2024-03-11T21:36:36.470537+0000","last_unstale":"2024-03-11T21:39:47.328903+0000","last_undegraded":"2024-03-11T21:36:36.071501+0000","last_fullsized":"2024-03-11T21:36:36.040233+0000","mapping_epoch":278,"log_start":"312'9300","ondisk_log_start":"312'9300","created":259,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:35:44.669493+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:35:44.669493+0000","last_clean_scrub_stamp":"2024-03-11T21:35:44.669493+0000","objects_scrubbed":0,"log_size":4669,"log_dups_size":0,"ondisk_log_size":4669,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":23003136,"num_objects":351,"num_object_clones":0,"num_object_copies":1053,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4641,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":351,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13969,"num_write_kb":458240,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":196608,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,1],"acting":[7,1],"avail_no_missing":["7","1"],"object_location_counts":[{"shards":"1,7","objects":-4290}],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.c","version":"0'0","reported_seq":515,"reported_epoch":336,"state":"active+clean","last_fresh":"2024-03-11T21:39:43.422596+0000","last_change":"2024-03-11T21:36:35.074112+0000","last_active":"2024-03-11T21:39:43.422596+0000","last_peered":"2024-03-11T21:39:43.422596+0000","last_clean":"2024-03-11T21:39:43.422596+0000","last_became_active":"2024-03-11T21:36:35.073778+0000","last_became_peered":"2024-03-11T21:36:35.073778+0000","last_unstale":"2024-03-11T21:39:43.422596+0000","last_undegraded":"2024-03-11T21:39:43.422596+0000","last_fullsized":"2024-03-11T21:39:43.422596+0000","mapping_epoch":277,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":278,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:32:17.600213+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:43.559452+0000","last_clean_scrub_stamp":"2024-03-11T21:32:17.600213+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.f","version":"0'0","reported_seq":596,"reported_epoch":337,"state":"active+clean","last_fresh":"2024-03-11T21:39:45.765919+0000","last_change":"2024-03-11T21:36:08.439013+0000","last_active":"2024-03-11T21:39:45.765919+0000","last_peered":"2024-03-11T21:39:45.765919+0000","last_clean":"2024-03-11T21:39:45.765919+0000","last_became_active":"2024-03-11T21:25:42.575935+0000","last_became_peered":"2024-03-11T21:25:42.575935+0000","last_unstale":"2024-03-11T21:39:45.765919+0000","last_undegraded":"2024-03-11T21:39:45.765919+0000","last_fullsized":"2024-03-11T21:39:45.765919+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:36:08.438895+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:41.544550+0000","last_clean_scrub_stamp":"2024-03-11T21:36:08.438895+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.8","version":"0'0","reported_seq":241,"reported_epoch":337,"state":"active+clean","last_fresh":"2024-03-11T21:39:45.757002+0000","last_change":"2024-03-11T21:35:34.733595+0000","last_active":"2024-03-11T21:39:45.757002+0000","last_peered":"2024-03-11T21:39:45.757002+0000","last_clean":"2024-03-11T21:39:45.757002+0000","last_became_active":"2024-03-11T21:34:41.144681+0000","last_became_peered":"2024-03-11T21:34:41.144681+0000","last_unstale":"2024-03-11T21:39:45.757002+0000","last_undegraded":"2024-03-11T21:39:45.757002+0000","last_fullsized":"2024-03-11T21:39:45.757002+0000","mapping_epoch":226,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":227,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:35:34.733491+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:37.512018+0000","last_clean_scrub_stamp":"2024-03-11T21:35:34.733491+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,1],"acting":[7,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.e","version":"260'24","reported_seq":629,"reported_epoch":337,"state":"active+clean","last_fresh":"2024-03-11T21:39:45.754809+0000","last_change":"2024-03-11T21:35:34.679906+0000","last_active":"2024-03-11T21:39:45.754809+0000","last_peered":"2024-03-11T21:39:45.754809+0000","last_clean":"2024-03-11T21:39:45.754809+0000","last_became_active":"2024-03-11T21:25:40.552178+0000","last_became_peered":"2024-03-11T21:25:40.552178+0000","last_unstale":"2024-03-11T21:39:45.754809+0000","last_undegraded":"2024-03-11T21:39:45.754809+0000","last_fullsized":"2024-03-11T21:39:45.754809+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"109'16","last_scrub_stamp":"2024-03-11T21:35:34.679817+0000","last_deep_scrub":"109'16","last_deep_scrub_stamp":"2024-03-11T21:28:24.739395+0000","last_clean_scrub_stamp":"2024-03-11T21:35:34.679817+0000","objects_scrubbed":5,"log_size":24,"log_dups_size":0,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":32,"num_read_kb":25,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.2","version":"337'14228","reported_seq":19032,"reported_epoch":337,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-03-11T21:39:48.301260+0000","last_change":"2024-03-11T21:38:29.073541+0000","last_active":"2024-03-11T21:39:48.301260+0000","last_peered":"2024-03-11T21:39:48.301260+0000","last_clean":"2024-03-11T21:35:44.669493+0000","last_became_active":"2024-03-11T21:36:36.506474+0000","last_became_peered":"2024-03-11T21:36:36.506474+0000","last_unstale":"2024-03-11T21:39:48.301260+0000","last_undegraded":"2024-03-11T21:36:36.068182+0000","last_fullsized":"2024-03-11T21:36:36.037781+0000","mapping_epoch":278,"log_start":"311'9200","ondisk_log_start":"311'9200","created":259,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:35:44.669493+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:35:44.669493+0000","last_clean_scrub_stamp":"2024-03-11T21:35:44.669493+0000","objects_scrubbed":0,"log_size":5028,"log_dups_size":0,"ondisk_log_size":5028,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19791872,"num_objects":302,"num_object_clones":0,"num_object_copies":906,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2454,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":302,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14228,"num_write_kb":464960,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2319,"num_bytes_recovered":65536000,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[1,2],"avail_no_missing":["1","2"],"object_location_counts":[{"shards":"1,2","objects":-2152}],"blocked_by":[],"up_primary":3,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.d","version":"0'0","reported_seq":584,"reported_epoch":337,"state":"active+clean","last_fresh":"2024-03-11T21:39:45.765580+0000","last_change":"2024-03-11T21:35:37.185210+0000","last_active":"2024-03-11T21:39:45.765580+0000","last_peered":"2024-03-11T21:39:45.765580+0000","last_clean":"2024-03-11T21:39:45.765580+0000","last_became_active":"2024-03-11T21:25:44.590333+0000","last_became_peered":"2024-03-11T21:25:44.590333+0000","last_unstale":"2024-03-11T21:39:45.765580+0000","last_undegraded":"2024-03-11T21:39:45.765580+0000","last_fullsized":"2024-03-11T21:39:45.765580+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:35:37.182850+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:35:37.182850+0000","last_clean_scrub_stamp":"2024-03-11T21:35:37.182850+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":40,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.e","version":"74'8","reported_seq":767,"reported_epoch":337,"state":"active+clean","last_fresh":"2024-03-11T21:39:45.755100+0000","last_change":"2024-03-11T21:36:07.734369+0000","last_active":"2024-03-11T21:39:45.755100+0000","last_peered":"2024-03-11T21:39:45.755100+0000","last_clean":"2024-03-11T21:39:45.755100+0000","last_became_active":"2024-03-11T21:25:42.576742+0000","last_became_peered":"2024-03-11T21:25:42.576742+0000","last_unstale":"2024-03-11T21:39:45.755100+0000","last_undegraded":"2024-03-11T21:39:45.755100+0000","last_fullsized":"2024-03-11T21:39:45.755100+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'8","last_scrub_stamp":"2024-03-11T21:36:07.734115+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:41.544550+0000","last_clean_scrub_stamp":"2024-03-11T21:36:07.734115+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,0],"acting":[4,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.9","version":"0'0","reported_seq":610,"reported_epoch":337,"state":"active+clean","last_fresh":"2024-03-11T21:39:45.765565+0000","last_change":"2024-03-11T21:36:35.066093+0000","last_active":"2024-03-11T21:39:45.765565+0000","last_peered":"2024-03-11T21:39:45.765565+0000","last_clean":"2024-03-11T21:39:45.765565+0000","last_became_active":"2024-03-11T21:36:35.065733+0000","last_became_peered":"2024-03-11T21:36:35.065733+0000","last_unstale":"2024-03-11T21:39:45.765565+0000","last_undegraded":"2024-03-11T21:39:45.765565+0000","last_fullsized":"2024-03-11T21:39:45.765565+0000","mapping_epoch":277,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":278,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:32:02.506579+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:26:50.991712+0000","last_clean_scrub_stamp":"2024-03-11T21:32:02.506579+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,3],"acting":[1,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.f","version":"260'27","reported_seq":612,"reported_epoch":337,"state":"active+clean","last_fresh":"2024-03-11T21:39:45.771469+0000","last_change":"2024-03-11T21:36:35.084069+0000","last_active":"2024-03-11T21:39:45.771469+0000","last_peered":"2024-03-11T21:39:45.771469+0000","last_clean":"2024-03-11T21:39:45.771469+0000","last_became_active":"2024-03-11T21:36:35.058272+0000","last_became_peered":"2024-03-11T21:36:35.058272+0000","last_unstale":"2024-03-11T21:39:45.771469+0000","last_undegraded":"2024-03-11T21:39:45.771469+0000","last_fullsized":"2024-03-11T21:39:45.771469+0000","mapping_epoch":277,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":278,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'15","last_scrub_stamp":"2024-03-11T21:34:21.517894+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:39.525554+0000","last_clean_scrub_stamp":"2024-03-11T21:34:21.517894+0000","objects_scrubbed":7,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":44,"num_read_kb":35,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,4],"acting":[2,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"9.1","version":"336'14077","reported_seq":23899,"reported_epoch":336,"state":"active+clean","last_fresh":"2024-03-11T21:39:44.069901+0000","last_change":"2024-03-11T21:38:22.558103+0000","last_active":"2024-03-11T21:39:44.069901+0000","last_peered":"2024-03-11T21:39:44.069901+0000","last_clean":"2024-03-11T21:39:44.069901+0000","last_became_active":"2024-03-11T21:38:22.540816+0000","last_became_peered":"2024-03-11T21:38:22.540816+0000","last_unstale":"2024-03-11T21:39:44.069901+0000","last_undegraded":"2024-03-11T21:39:44.069901+0000","last_fullsized":"2024-03-11T21:39:44.069901+0000","mapping_epoch":316,"log_start":"311'9195","ondisk_log_start":"311'9195","created":259,"last_epoch_clean":317,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:35:44.669493+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:35:44.669493+0000","last_clean_scrub_stamp":"2024-03-11T21:35:44.669493+0000","objects_scrubbed":0,"log_size":4882,"log_dups_size":0,"ondisk_log_size":4882,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":33226752,"num_objects":507,"num_object_clones":0,"num_object_copies":1521,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":507,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14077,"num_write_kb":466688,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4825,"num_bytes_recovered":316211200,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,7],"acting":[3,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.e","version":"0'0","reported_seq":585,"reported_epoch":337,"state":"active+clean","last_fresh":"2024-03-11T21:39:45.755209+0000","last_change":"2024-03-11T21:36:10.769557+0000","last_active":"2024-03-11T21:39:45.755209+0000","last_peered":"2024-03-11T21:39:45.755209+0000","last_clean":"2024-03-11T21:39:45.755209+0000","last_became_active":"2024-03-11T21:25:44.590198+0000","last_became_peered":"2024-03-11T21:25:44.590198+0000","last_unstale":"2024-03-11T21:39:45.755209+0000","last_undegraded":"2024-03-11T21:39:45.755209+0000","last_fullsized":"2024-03-11T21:39:45.755209+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:36:10.769460+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:43.559452+0000","last_clean_scrub_stamp":"2024-03-11T21:36:10.769460+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":75,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.d","version":"109'9","reported_seq":767,"reported_epoch":337,"state":"active+clean","last_fresh":"2024-03-11T21:39:45.770438+0000","last_change":"2024-03-11T21:35:07.562523+0000","last_active":"2024-03-11T21:39:45.770438+0000","last_peered":"2024-03-11T21:39:45.770438+0000","last_clean":"2024-03-11T21:39:45.770438+0000","last_became_active":"2024-03-11T21:25:42.575583+0000","last_became_peered":"2024-03-11T21:25:42.575583+0000","last_unstale":"2024-03-11T21:39:45.770438+0000","last_undegraded":"2024-03-11T21:39:45.770438+0000","last_fullsized":"2024-03-11T21:39:45.770438+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"109'9","last_scrub_stamp":"2024-03-11T21:35:07.562385+0000","last_deep_scrub":"109'9","last_deep_scrub_stamp":"2024-03-11T21:28:27.082975+0000","last_clean_scrub_stamp":"2024-03-11T21:35:07.562385+0000","objects_scrubbed":1,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,5],"acting":[2,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.a","version":"0'0","reported_seq":618,"reported_epoch":337,"state":"active+clean","last_fresh":"2024-03-11T21:39:45.768693+0000","last_change":"2024-03-11T21:36:35.067262+0000","last_active":"2024-03-11T21:39:45.768693+0000","last_peered":"2024-03-11T21:39:45.768693+0000","last_clean":"2024-03-11T21:39:45.768693+0000","last_became_active":"2024-03-11T21:36:35.066832+0000","last_became_peered":"2024-03-11T21:36:35.066832+0000","last_unstale":"2024-03-11T21:39:45.768693+0000","last_undegraded":"2024-03-11T21:39:45.768693+0000","last_fullsized":"2024-03-11T21:39:45.768693+0000","mapping_epoch":277,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":278,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:34:22.526916+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:37.512018+0000","last_clean_scrub_stamp":"2024-03-11T21:34:22.526916+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,7],"acting":[1,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.c","version":"260'20","reported_seq":637,"reported_epoch":337,"state":"active+clean","last_fresh":"2024-03-11T21:39:45.753927+0000","last_change":"2024-03-11T21:36:35.095574+0000","last_active":"2024-03-11T21:39:45.753927+0000","last_peered":"2024-03-11T21:39:45.753927+0000","last_clean":"2024-03-11T21:39:45.753927+0000","last_became_active":"2024-03-11T21:36:35.067269+0000","last_became_peered":"2024-03-11T21:36:35.067269+0000","last_unstale":"2024-03-11T21:39:45.753927+0000","last_undegraded":"2024-03-11T21:39:45.753927+0000","last_fullsized":"2024-03-11T21:39:45.753927+0000","mapping_epoch":277,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":278,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'10","last_scrub_stamp":"2024-03-11T21:32:54.846584+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:39.525554+0000","last_clean_scrub_stamp":"2024-03-11T21:32:54.846584+0000","objects_scrubbed":5,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":25,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.0","version":"337'14288","reported_seq":14455,"reported_epoch":337,"state":"active+clean","last_fresh":"2024-03-11T21:39:48.282450+0000","last_change":"2024-03-11T21:35:45.703065+0000","last_active":"2024-03-11T21:39:48.282450+0000","last_peered":"2024-03-11T21:39:48.282450+0000","last_clean":"2024-03-11T21:39:48.282450+0000","last_became_active":"2024-03-11T21:35:45.702739+0000","last_became_peered":"2024-03-11T21:35:45.702739+0000","last_unstale":"2024-03-11T21:39:48.282450+0000","last_undegraded":"2024-03-11T21:39:48.282450+0000","last_fullsized":"2024-03-11T21:39:48.282450+0000","mapping_epoch":259,"log_start":"311'9200","ondisk_log_start":"311'9200","created":259,"last_epoch_clean":260,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:35:44.669493+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:35:44.669493+0000","last_clean_scrub_stamp":"2024-03-11T21:35:44.669493+0000","objects_scrubbed":0,"log_size":5088,"log_dups_size":0,"ondisk_log_size":5088,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19398656,"num_objects":296,"num_object_clones":0,"num_object_copies":888,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":296,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14288,"num_write_kb":466688,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.f","version":"0'0","reported_seq":593,"reported_epoch":337,"state":"active+clean","last_fresh":"2024-03-11T21:39:45.771292+0000","last_change":"2024-03-11T21:36:35.058630+0000","last_active":"2024-03-11T21:39:45.771292+0000","last_peered":"2024-03-11T21:39:45.771292+0000","last_clean":"2024-03-11T21:39:45.771292+0000","last_became_active":"2024-03-11T21:36:35.057800+0000","last_became_peered":"2024-03-11T21:36:35.057800+0000","last_unstale":"2024-03-11T21:39:45.771292+0000","last_undegraded":"2024-03-11T21:39:45.771292+0000","last_fullsized":"2024-03-11T21:39:45.771292+0000","mapping_epoch":277,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":278,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:35:10.608981+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:27:03.010969+0000","last_clean_scrub_stamp":"2024-03-11T21:35:10.608981+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":94,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,4],"acting":[2,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":589,"reported_epoch":337,"state":"active+clean","last_fresh":"2024-03-11T21:39:45.773470+0000","last_change":"2024-03-11T21:36:10.608789+0000","last_active":"2024-03-11T21:39:45.773470+0000","last_peered":"2024-03-11T21:39:45.773470+0000","last_clean":"2024-03-11T21:39:45.773470+0000","last_became_active":"2024-03-11T21:25:42.579409+0000","last_became_peered":"2024-03-11T21:25:42.579409+0000","last_unstale":"2024-03-11T21:39:45.773470+0000","last_undegraded":"2024-03-11T21:39:45.773470+0000","last_fullsized":"2024-03-11T21:39:45.773470+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:36:10.608552+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:30:22.750053+0000","last_clean_scrub_stamp":"2024-03-11T21:36:10.608552+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.b","version":"0'0","reported_seq":241,"reported_epoch":337,"state":"active+clean","last_fresh":"2024-03-11T21:39:45.757847+0000","last_change":"2024-03-11T21:36:16.719432+0000","last_active":"2024-03-11T21:39:45.757847+0000","last_peered":"2024-03-11T21:39:45.757847+0000","last_clean":"2024-03-11T21:39:45.757847+0000","last_became_active":"2024-03-11T21:34:41.134987+0000","last_became_peered":"2024-03-11T21:34:41.134987+0000","last_unstale":"2024-03-11T21:39:45.757847+0000","last_undegraded":"2024-03-11T21:39:45.757847+0000","last_fullsized":"2024-03-11T21:39:45.757847+0000","mapping_epoch":226,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":227,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:36:16.719309+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:36:16.719309+0000","last_clean_scrub_stamp":"2024-03-11T21:36:16.719309+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,5],"acting":[7,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.d","version":"260'39","reported_seq":650,"reported_epoch":337,"state":"active+clean","last_fresh":"2024-03-11T21:39:45.754340+0000","last_change":"2024-03-11T21:35:25.543769+0000","last_active":"2024-03-11T21:39:45.754340+0000","last_peered":"2024-03-11T21:39:45.754340+0000","last_clean":"2024-03-11T21:39:45.754340+0000","last_became_active":"2024-03-11T21:25:40.553454+0000","last_became_peered":"2024-03-11T21:25:40.553454+0000","last_unstale":"2024-03-11T21:39:45.754340+0000","last_undegraded":"2024-03-11T21:39:45.754340+0000","last_fullsized":"2024-03-11T21:39:45.754340+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"165'25","last_scrub_stamp":"2024-03-11T21:35:25.543628+0000","last_deep_scrub":"165'25","last_deep_scrub_stamp":"2024-03-11T21:32:21.678328+0000","last_clean_scrub_stamp":"2024-03-11T21:35:25.543628+0000","objects_scrubbed":8,"log_size":39,"log_dups_size":0,"ondisk_log_size":39,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":50,"num_read_kb":40,"num_write":32,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,1],"acting":[4,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.7","version":"337'14177","reported_seq":23855,"reported_epoch":337,"state":"active+clean","last_fresh":"2024-03-11T21:39:47.500994+0000","last_change":"2024-03-11T21:38:58.387549+0000","last_active":"2024-03-11T21:39:47.500994+0000","last_peered":"2024-03-11T21:39:47.500994+0000","last_clean":"2024-03-11T21:39:47.500994+0000","last_became_active":"2024-03-11T21:38:58.365063+0000","last_became_peered":"2024-03-11T21:38:58.365063+0000","last_unstale":"2024-03-11T21:39:47.500994+0000","last_undegraded":"2024-03-11T21:39:47.500994+0000","last_fullsized":"2024-03-11T21:39:47.500994+0000","mapping_epoch":327,"log_start":"309'9000","ondisk_log_start":"309'9000","created":259,"last_epoch_clean":328,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:35:44.669493+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:35:44.669493+0000","last_clean_scrub_stamp":"2024-03-11T21:35:44.669493+0000","objects_scrubbed":0,"log_size":5177,"log_dups_size":0,"ondisk_log_size":5177,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":23003136,"num_objects":351,"num_object_clones":0,"num_object_copies":1053,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":351,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14177,"num_write_kb":464896,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4752,"num_bytes_recovered":311427072,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":241,"reported_epoch":337,"state":"active+clean","last_fresh":"2024-03-11T21:39:45.756864+0000","last_change":"2024-03-11T21:36:06.735035+0000","last_active":"2024-03-11T21:39:45.756864+0000","last_peered":"2024-03-11T21:39:45.756864+0000","last_clean":"2024-03-11T21:39:45.756864+0000","last_became_active":"2024-03-11T21:34:41.145012+0000","last_became_peered":"2024-03-11T21:34:41.145012+0000","last_unstale":"2024-03-11T21:39:45.756864+0000","last_undegraded":"2024-03-11T21:39:45.756864+0000","last_fullsized":"2024-03-11T21:39:45.756864+0000","mapping_epoch":226,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":227,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:36:06.734921+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:28:37.561826+0000","last_clean_scrub_stamp":"2024-03-11T21:36:06.734921+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":23,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,0],"acting":[7,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":588,"reported_epoch":337,"state":"active+clean","last_fresh":"2024-03-11T21:39:45.769708+0000","last_change":"2024-03-11T21:35:14.636561+0000","last_active":"2024-03-11T21:39:45.769708+0000","last_peered":"2024-03-11T21:39:45.769708+0000","last_clean":"2024-03-11T21:39:45.769708+0000","last_became_active":"2024-03-11T21:25:42.574515+0000","last_became_peered":"2024-03-11T21:25:42.574515+0000","last_unstale":"2024-03-11T21:39:45.769708+0000","last_undegraded":"2024-03-11T21:39:45.769708+0000","last_fullsized":"2024-03-11T21:39:45.769708+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:35:14.636450+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:41.544550+0000","last_clean_scrub_stamp":"2024-03-11T21:35:14.636450+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,5],"acting":[2,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.c","version":"0'0","reported_seq":596,"reported_epoch":337,"state":"active+clean","last_fresh":"2024-03-11T21:39:45.770095+0000","last_change":"2024-03-11T21:36:00.849272+0000","last_active":"2024-03-11T21:39:45.770095+0000","last_peered":"2024-03-11T21:39:45.770095+0000","last_clean":"2024-03-11T21:39:45.770095+0000","last_became_active":"2024-03-11T21:25:38.533240+0000","last_became_peered":"2024-03-11T21:25:38.533240+0000","last_unstale":"2024-03-11T21:39:45.770095+0000","last_undegraded":"2024-03-11T21:39:45.770095+0000","last_fullsized":"2024-03-11T21:39:45.770095+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:36:00.849153+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:29:30.279910+0000","last_clean_scrub_stamp":"2024-03-11T21:36:00.849153+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,0],"acting":[2,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.a","version":"260'44","reported_seq":725,"reported_epoch":337,"state":"active+clean","last_fresh":"2024-03-11T21:39:45.766316+0000","last_change":"2024-03-11T21:38:57.101011+0000","last_active":"2024-03-11T21:39:45.766316+0000","last_peered":"2024-03-11T21:39:45.766316+0000","last_clean":"2024-03-11T21:39:45.766316+0000","last_became_active":"2024-03-11T21:34:41.148863+0000","last_became_peered":"2024-03-11T21:34:41.148863+0000","last_unstale":"2024-03-11T21:39:45.766316+0000","last_undegraded":"2024-03-11T21:39:45.766316+0000","last_fullsized":"2024-03-11T21:39:45.766316+0000","mapping_epoch":226,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":227,"parent":"0.0","parent_split_bits":0,"last_scrub":"260'44","last_scrub_stamp":"2024-03-11T21:38:57.100873+0000","last_deep_scrub":"109'28","last_deep_scrub_stamp":"2024-03-11T21:28:24.016389+0000","last_clean_scrub_stamp":"2024-03-11T21:38:57.100873+0000","objects_scrubbed":9,"log_size":44,"log_dups_size":0,"ondisk_log_size":44,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:40:01.805926+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":56,"num_read_kb":45,"num_write":36,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,7],"acting":[6,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.6","version":"337'14199","reported_seq":14366,"reported_epoch":337,"state":"active+clean","last_fresh":"2024-03-11T21:39:47.479285+0000","last_change":"2024-03-11T21:35:45.703533+0000","last_active":"2024-03-11T21:39:47.479285+0000","last_peered":"2024-03-11T21:39:47.479285+0000","last_clean":"2024-03-11T21:39:47.479285+0000","last_became_active":"2024-03-11T21:35:45.703184+0000","last_became_peered":"2024-03-11T21:35:45.703184+0000","last_unstale":"2024-03-11T21:39:47.479285+0000","last_undegraded":"2024-03-11T21:39:47.479285+0000","last_fullsized":"2024-03-11T21:39:47.479285+0000","mapping_epoch":259,"log_start":"308'9000","ondisk_log_start":"308'9000","created":259,"last_epoch_clean":260,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:35:44.669493+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:35:44.669493+0000","last_clean_scrub_stamp":"2024-03-11T21:35:44.669493+0000","objects_scrubbed":0,"log_size":5199,"log_dups_size":0,"ondisk_log_size":5199,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22740992,"num_objects":347,"num_object_clones":0,"num_object_copies":1041,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":347,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14199,"num_write_kb":465472,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,1],"acting":[5,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":595,"reported_epoch":336,"state":"active+clean","last_fresh":"2024-03-11T21:39:43.380721+0000","last_change":"2024-03-11T21:35:37.687281+0000","last_active":"2024-03-11T21:39:43.380721+0000","last_peered":"2024-03-11T21:39:43.380721+0000","last_clean":"2024-03-11T21:39:43.380721+0000","last_became_active":"2024-03-11T21:34:41.154507+0000","last_became_peered":"2024-03-11T21:34:41.154507+0000","last_unstale":"2024-03-11T21:39:43.380721+0000","last_undegraded":"2024-03-11T21:39:43.380721+0000","last_fullsized":"2024-03-11T21:39:43.380721+0000","mapping_epoch":226,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":227,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:35:37.687187+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:27:11.361913+0000","last_clean_scrub_stamp":"2024-03-11T21:35:37.687187+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":39,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":589,"reported_epoch":337,"state":"active+clean","last_fresh":"2024-03-11T21:39:45.769238+0000","last_change":"2024-03-11T21:36:35.059662+0000","last_active":"2024-03-11T21:39:45.769238+0000","last_peered":"2024-03-11T21:39:45.769238+0000","last_clean":"2024-03-11T21:39:45.769238+0000","last_became_active":"2024-03-11T21:36:35.059192+0000","last_became_peered":"2024-03-11T21:36:35.059192+0000","last_unstale":"2024-03-11T21:39:45.769238+0000","last_undegraded":"2024-03-11T21:39:45.769238+0000","last_fullsized":"2024-03-11T21:39:45.769238+0000","mapping_epoch":277,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":278,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:34:17.412687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:41.544550+0000","last_clean_scrub_stamp":"2024-03-11T21:34:17.412687+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,3],"acting":[2,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.d","version":"0'0","reported_seq":598,"reported_epoch":337,"state":"active+clean","last_fresh":"2024-03-11T21:39:45.765397+0000","last_change":"2024-03-11T21:36:35.058635+0000","last_active":"2024-03-11T21:39:45.765397+0000","last_peered":"2024-03-11T21:39:45.765397+0000","last_clean":"2024-03-11T21:39:45.765397+0000","last_became_active":"2024-03-11T21:36:35.058296+0000","last_became_peered":"2024-03-11T21:36:35.058296+0000","last_unstale":"2024-03-11T21:39:45.765397+0000","last_undegraded":"2024-03-11T21:39:45.765397+0000","last_fullsized":"2024-03-11T21:39:45.765397+0000","mapping_epoch":277,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":278,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:31:55.464145+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:37.512018+0000","last_clean_scrub_stamp":"2024-03-11T21:31:55.464145+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,3],"acting":[1,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.b","version":"260'15","reported_seq":616,"reported_epoch":336,"state":"active+clean","last_fresh":"2024-03-11T21:39:43.381958+0000","last_change":"2024-03-11T21:35:27.760318+0000","last_active":"2024-03-11T21:39:43.381958+0000","last_peered":"2024-03-11T21:39:43.381958+0000","last_clean":"2024-03-11T21:39:43.381958+0000","last_became_active":"2024-03-11T21:25:40.549612+0000","last_became_peered":"2024-03-11T21:25:40.549612+0000","last_unstale":"2024-03-11T21:39:43.381958+0000","last_undegraded":"2024-03-11T21:39:43.381958+0000","last_fullsized":"2024-03-11T21:39:43.381958+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'9","last_scrub_stamp":"2024-03-11T21:35:27.760208+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:39.525554+0000","last_clean_scrub_stamp":"2024-03-11T21:35:27.760208+0000","objects_scrubbed":4,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.5","version":"336'14254","reported_seq":14418,"reported_epoch":336,"state":"active+clean","last_fresh":"2024-03-11T21:39:43.702040+0000","last_change":"2024-03-11T21:35:45.697209+0000","last_active":"2024-03-11T21:39:43.702040+0000","last_peered":"2024-03-11T21:39:43.702040+0000","last_clean":"2024-03-11T21:39:43.702040+0000","last_became_active":"2024-03-11T21:35:45.696788+0000","last_became_peered":"2024-03-11T21:35:45.696788+0000","last_unstale":"2024-03-11T21:39:43.702040+0000","last_undegraded":"2024-03-11T21:39:43.702040+0000","last_fullsized":"2024-03-11T21:39:43.702040+0000","mapping_epoch":259,"log_start":"305'8600","ondisk_log_start":"305'8600","created":259,"last_epoch_clean":260,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:35:44.669493+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:35:44.669493+0000","last_clean_scrub_stamp":"2024-03-11T21:35:44.669493+0000","objects_scrubbed":0,"log_size":5654,"log_dups_size":0,"ondisk_log_size":5654,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":35913728,"num_objects":548,"num_object_clones":0,"num_object_copies":1644,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":548,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14254,"num_write_kb":473664,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":499,"reported_epoch":337,"state":"active+clean","last_fresh":"2024-03-11T21:39:45.753178+0000","last_change":"2024-03-11T21:36:35.057118+0000","last_active":"2024-03-11T21:39:45.753178+0000","last_peered":"2024-03-11T21:39:45.753178+0000","last_clean":"2024-03-11T21:39:45.753178+0000","last_became_active":"2024-03-11T21:36:35.056518+0000","last_became_peered":"2024-03-11T21:36:35.056518+0000","last_unstale":"2024-03-11T21:39:45.753178+0000","last_undegraded":"2024-03-11T21:39:45.753178+0000","last_fullsized":"2024-03-11T21:39:45.753178+0000","mapping_epoch":277,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":278,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:35:24.514584+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:43.559452+0000","last_clean_scrub_stamp":"2024-03-11T21:35:24.514584+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,3],"acting":[4,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.e","version":"0'0","reported_seq":617,"reported_epoch":337,"state":"active+clean","last_fresh":"2024-03-11T21:39:45.771531+0000","last_change":"2024-03-11T21:36:35.057187+0000","last_active":"2024-03-11T21:39:45.771531+0000","last_peered":"2024-03-11T21:39:45.771531+0000","last_clean":"2024-03-11T21:39:45.771531+0000","last_became_active":"2024-03-11T21:36:35.056777+0000","last_became_peered":"2024-03-11T21:36:35.056777+0000","last_unstale":"2024-03-11T21:39:45.771531+0000","last_undegraded":"2024-03-11T21:39:45.771531+0000","last_fullsized":"2024-03-11T21:39:45.771531+0000","mapping_epoch":277,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":278,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:34:19.476172+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:37.512018+0000","last_clean_scrub_stamp":"2024-03-11T21:34:19.476172+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":17,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,7],"acting":[2,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.8","version":"260'34","reported_seq":686,"reported_epoch":337,"state":"active+clean","last_fresh":"2024-03-11T21:39:45.763964+0000","last_change":"2024-03-11T21:35:20.304345+0000","last_active":"2024-03-11T21:39:45.763964+0000","last_peered":"2024-03-11T21:39:45.763964+0000","last_clean":"2024-03-11T21:39:45.763964+0000","last_became_active":"2024-03-11T21:34:41.146631+0000","last_became_peered":"2024-03-11T21:34:41.146631+0000","last_unstale":"2024-03-11T21:39:45.763964+0000","last_undegraded":"2024-03-11T21:39:45.763964+0000","last_fullsized":"2024-03-11T21:39:45.763964+0000","mapping_epoch":226,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":227,"parent":"0.0","parent_split_bits":0,"last_scrub":"109'22","last_scrub_stamp":"2024-03-11T21:35:20.304114+0000","last_deep_scrub":"109'22","last_deep_scrub_stamp":"2024-03-11T21:32:07.573197+0000","last_clean_scrub_stamp":"2024-03-11T21:35:20.304114+0000","objects_scrubbed":7,"log_size":34,"log_dups_size":0,"ondisk_log_size":34,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":44,"num_read_kb":35,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.9","version":"229'11","reported_seq":306,"reported_epoch":337,"state":"active+clean","last_fresh":"2024-03-11T21:39:45.757324+0000","last_change":"2024-03-11T21:36:18.726876+0000","last_active":"2024-03-11T21:39:45.757324+0000","last_peered":"2024-03-11T21:39:45.757324+0000","last_clean":"2024-03-11T21:39:45.757324+0000","last_became_active":"2024-03-11T21:34:41.156892+0000","last_became_peered":"2024-03-11T21:34:41.156892+0000","last_unstale":"2024-03-11T21:39:45.757324+0000","last_undegraded":"2024-03-11T21:39:45.757324+0000","last_fullsized":"2024-03-11T21:39:45.757324+0000","mapping_epoch":226,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":227,"parent":"0.0","parent_split_bits":0,"last_scrub":"229'11","last_scrub_stamp":"2024-03-11T21:36:18.726763+0000","last_deep_scrub":"94'9","last_deep_scrub_stamp":"2024-03-11T21:26:53.422552+0000","last_clean_scrub_stamp":"2024-03-11T21:36:18.726763+0000","objects_scrubbed":1,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":9,"num_pg":16,"stat_sum":{"num_bytes":422379544,"num_objects":6446,"num_object_clones":0,"num_object_copies":19338,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":17862,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6446,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":226318,"num_write_kb":7448385,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20161,"num_bytes_recovered":1089798144,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1186082816,"data_stored":1186070600,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":82279,"ondisk_log_size":82279,"up":48,"acting":43,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":1531,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":17,"num_read_kb":17,"num_write":10,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":1308,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":4593,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":6,"ondisk_log_size":6,"up":96,"acting":96,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":198,"num_read_kb":294,"num_write":179,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":2951840,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":2375680,"data_stored":2361472,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":128,"ondisk_log_size":128,"up":3,"acting":3,"num_store_stats":4},{"poolid":3,"num_pg":3,"stat_sum":{"num_bytes":408,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":842,"num_read_kb":837,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1224,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":8,"ondisk_log_size":8,"up":9,"acting":9,"num_store_stats":7},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":3702,"num_objects":178,"num_object_clones":0,"num_object_copies":534,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":178,"num_whiteouts":0,"num_read":1120,"num_read_kb":877,"num_write":703,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":186,"num_bytes_recovered":2200,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":417792,"data_stored":11106,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":760,"ondisk_log_size":760,"up":96,"acting":96,"num_store_stats":8},{"poolid":5,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":78,"ondisk_log_size":78,"up":96,"acting":96,"num_store_stats":8},{"poolid":6,"num_pg":32,"stat_sum":{"num_bytes":383,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":370,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1149,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":96,"acting":96,"num_store_stats":8}],"osd_stats":[{"osd":7,"up_from":60,"seq":257698037933,"num_pgs":65,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":716088,"kb_used_data":225388,"kb_used_omap":1,"kb_used_meta":490686,"kb_avail":93029064,"statfs":{"total":95995035648,"available":95261761536,"internally_reserved":0,"allocated":230797312,"data_stored":229921421,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":502462922},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":50,"seq":214748364979,"num_pgs":46,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":474928,"kb_used_data":68904,"kb_used_omap":1,"kb_used_meta":406014,"kb_avail":93270224,"statfs":{"total":95995035648,"available":95508709376,"internally_reserved":0,"allocated":70557696,"data_stored":69686606,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":415758790},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834575053,"num_pgs":60,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":758156,"kb_used_data":189380,"kb_used_omap":1,"kb_used_meta":568446,"kb_avail":92986996,"statfs":{"total":95995035648,"available":95218683904,"internally_reserved":0,"allocated":193925120,"data_stored":193024836,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":582089157},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738577,"num_pgs":54,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":698356,"kb_used_data":143784,"kb_used_omap":1,"kb_used_meta":554558,"kb_avail":93046796,"statfs":{"total":95995035648,"available":95279919104,"internally_reserved":0,"allocated":147234816,"data_stored":146356563,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":567867845},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411527,"num_pgs":43,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":507564,"kb_used_data":75168,"kb_used_omap":1,"kb_used_meta":432382,"kb_avail":93237588,"statfs":{"total":95995035648,"available":95475290112,"internally_reserved":0,"allocated":76972032,"data_stored":76108358,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":442759621},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":277,"seq":1189705941031,"num_pgs":62,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":540084,"kb_used_data":285612,"kb_used_omap":4,"kb_used_meta":254395,"kb_avail":93205068,"statfs":{"total":95995035648,"available":95441989632,"internally_reserved":0,"allocated":292466688,"data_stored":291577056,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4442,"internal_metadata":260501158},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":4,"up_from":32,"seq":137438953660,"num_pgs":56,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":491456,"kb_used_data":77108,"kb_used_omap":1,"kb_used_meta":414334,"kb_avail":93253696,"statfs":{"total":95995035648,"available":95491784704,"internally_reserved":0,"allocated":78958592,"data_stored":78074286,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":424278469},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":41,"seq":176093659319,"num_pgs":58,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":735088,"kb_used_data":198564,"kb_used_omap":1,"kb_used_meta":536510,"kb_avail":93010064,"statfs":{"total":95995035648,"available":95242305536,"internally_reserved":0,"allocated":203329536,"data_stored":202461405,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":549386693},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1049,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":46,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":436,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":957,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":574,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":436,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1003,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":92,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":1430,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":77824,"data_stored":2008,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":1018,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":1898,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":1474,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":990,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":1034,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":1254,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":370,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":383,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":370,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":142737408,"data_stored":142737408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":172888064,"data_stored":172883992,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":67829760,"data_stored":67829760,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":268042240,"data_stored":268042240,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":70778880,"data_stored":70778880,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":194842624,"data_stored":194838552,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":67174400,"data_stored":67174400,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":201789440,"data_stored":201785368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-11T21:39:51.511 DEBUG:teuthology.orchestra.run.smithi092:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid 00db0016-dfed-11ee-95c8-87774f69a715 -- ceph status --format=json 2024-03-11T21:39:51.537 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:39:51.538 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:39:51.538 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:39:51.639 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:39:51.689 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:39:51.690 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:39:51.690 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:39:51.791 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:39:51.858 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:39:51.859 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:39:51.859 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:39:51.960 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:39:51.988 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:39:51.989 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:39:51.989 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:39:52.055 INFO:teuthology.orchestra.run.smithi092.stderr:Inferring config /var/lib/ceph/00db0016-dfed-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T21:39:52.090 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:39:52.147 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:39:52.147 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:39:52.148 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:39:52.248 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:39:52.296 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:39:52.297 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:39:52.297 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:39:52.398 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:39:52.447 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:39:52.447 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:39:52.447 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:39:52.524 INFO:teuthology.orchestra.run.smithi092.stderr:Inferring config /var/lib/ceph/00db0016-dfed-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T21:39:52.549 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:39:52.611 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:39:52.611 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:39:52.612 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:39:52.713 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:39:52.740 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:39:52.740 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:39:52.741 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:39:52.763 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:39:52 smithi092 ceph-mon[37563]: pgmap v659: 148 pgs: 2 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+degraded+remapped, 143 active+clean; 301 MiB data, 4.5 GiB used, 711 GiB / 715 GiB avail; 645 B/s rd, 0 B/s wr, 737 op/s; 17728/15036 objects degraded (117.904%); 0 B/s, 78 objects/s recovering 2024-03-11T21:39:52.763 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:39:52 smithi092 ceph-mon[37563]: from='client.? v1:172.21.15.92:0/3475312558' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-11T21:39:52.763 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:39:52 smithi092 ceph-mon[37563]: from='client.25528 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-11T21:39:52.763 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:39:52 smithi092 ceph-mon[37563]: from='mgr.14607 v1:172.21.15.92:0/1702717597' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T21:39:52.788 INFO:teuthology.orchestra.run.smithi092.stderr:nodeep-scrub is unset 2024-03-11T21:39:52.842 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:39:52.879 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:39:52.880 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:39:52.880 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:39:52.981 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:39:53.007 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:39:53.008 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:39:53.008 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:39:53.079 INFO:journalctl@ceph.iscsi.iscsi.a.smithi097.stdout:Mar 11 21:39:52 smithi097 ceph-00db0016-dfed-11ee-95c8-87774f69a715-iscsi-iscsi-a[60943]: debug there is no tcmu-runner data available 2024-03-11T21:39:53.079 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:39:52 smithi097 ceph-mon[33317]: pgmap v659: 148 pgs: 2 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+degraded+remapped, 143 active+clean; 301 MiB data, 4.5 GiB used, 711 GiB / 715 GiB avail; 645 B/s rd, 0 B/s wr, 737 op/s; 17728/15036 objects degraded (117.904%); 0 B/s, 78 objects/s recovering 2024-03-11T21:39:53.079 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:39:52 smithi097 ceph-mon[33317]: from='client.? v1:172.21.15.92:0/3475312558' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-11T21:39:53.079 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:39:52 smithi097 ceph-mon[33317]: from='client.25528 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-11T21:39:53.080 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:39:52 smithi097 ceph-mon[33317]: from='mgr.14607 v1:172.21.15.92:0/1702717597' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T21:39:53.109 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:39:53.113 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:39:52 smithi092 ceph-mon[29572]: pgmap v659: 148 pgs: 2 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+degraded+remapped, 143 active+clean; 301 MiB data, 4.5 GiB used, 711 GiB / 715 GiB avail; 645 B/s rd, 0 B/s wr, 737 op/s; 17728/15036 objects degraded (117.904%); 0 B/s, 78 objects/s recovering 2024-03-11T21:39:53.113 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:39:52 smithi092 ceph-mon[29572]: from='client.? v1:172.21.15.92:0/3475312558' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-11T21:39:53.113 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:39:52 smithi092 ceph-mon[29572]: from='client.25528 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-11T21:39:53.113 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:39:52 smithi092 ceph-mon[29572]: from='mgr.14607 v1:172.21.15.92:0/1702717597' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T21:39:53.138 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:39:53.139 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:39:53.140 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:39:53.240 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:39:53.272 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:39:53.273 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:39:53.273 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:39:53.374 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:39:53.406 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:39:53.407 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:39:53.407 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:39:53.508 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:39:53.535 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:39:53.535 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:39:53.536 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:39:53.636 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:39:53.660 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:39:53.660 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:39:53.661 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:39:53.707 INFO:tasks.radosbench.radosbench.0.smithi092.stdout:Removed 116381 objects 2024-03-11T21:39:53.708 INFO:tasks.radosbench.radosbench.0.smithi092.stdout:Clean up completed and total clean up time :146.163 2024-03-11T21:39:53.761 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:39:53.790 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:39:53.790 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:39:53.791 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:39:53.863 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:39:53 smithi092 ceph-mon[29572]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-03-11T21:39:53.863 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:39:53 smithi092 ceph-mon[29572]: from='client.25528 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-03-11T21:39:53.863 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:39:53 smithi092 ceph-mon[29572]: from='client.24469 v1:172.21.15.97:0/404208310' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:39:53.863 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:39:53 smithi092 ceph-mon[29572]: osdmap e339: 8 total, 8 up, 8 in 2024-03-11T21:39:53.863 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:39:53 smithi092 ceph-mon[29572]: 2.1a scrub starts 2024-03-11T21:39:53.863 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:39:53 smithi092 ceph-mon[29572]: 2.1a scrub ok 2024-03-11T21:39:53.864 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:39:53 smithi092 ceph-mon[29572]: Health check update: Degraded data redundancy: 17728/15036 objects degraded (117.904%), 5 pgs degraded, 5 pgs undersized (PG_DEGRADED) 2024-03-11T21:39:53.864 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:39:53 smithi092 ceph-mon[29572]: osdmap e340: 8 total, 8 up, 8 in 2024-03-11T21:39:53.864 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:39:53 smithi092 ceph-mon[37563]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-03-11T21:39:53.864 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:39:53 smithi092 ceph-mon[37563]: from='client.25528 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-03-11T21:39:53.864 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:39:53 smithi092 ceph-mon[37563]: from='client.24469 v1:172.21.15.97:0/404208310' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:39:53.864 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:39:53 smithi092 ceph-mon[37563]: osdmap e339: 8 total, 8 up, 8 in 2024-03-11T21:39:53.864 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:39:53 smithi092 ceph-mon[37563]: 2.1a scrub starts 2024-03-11T21:39:53.864 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:39:53 smithi092 ceph-mon[37563]: 2.1a scrub ok 2024-03-11T21:39:53.864 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:39:53 smithi092 ceph-mon[37563]: Health check update: Degraded data redundancy: 17728/15036 objects degraded (117.904%), 5 pgs degraded, 5 pgs undersized (PG_DEGRADED) 2024-03-11T21:39:53.864 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:39:53 smithi092 ceph-mon[37563]: osdmap e340: 8 total, 8 up, 8 in 2024-03-11T21:39:53.892 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:39:53.917 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:39:53.918 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:39:53.918 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:39:54.019 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:39:54.047 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:39:54.050 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:39:54.050 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:39:54.079 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:39:53 smithi097 ceph-mon[33317]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-03-11T21:39:54.079 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:39:53 smithi097 ceph-mon[33317]: from='client.25528 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-03-11T21:39:54.079 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:39:53 smithi097 ceph-mon[33317]: from='client.24469 v1:172.21.15.97:0/404208310' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:39:54.079 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:39:53 smithi097 ceph-mon[33317]: osdmap e339: 8 total, 8 up, 8 in 2024-03-11T21:39:54.079 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:39:53 smithi097 ceph-mon[33317]: 2.1a scrub starts 2024-03-11T21:39:54.080 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:39:53 smithi097 ceph-mon[33317]: 2.1a scrub ok 2024-03-11T21:39:54.080 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:39:53 smithi097 ceph-mon[33317]: Health check update: Degraded data redundancy: 17728/15036 objects degraded (117.904%), 5 pgs degraded, 5 pgs undersized (PG_DEGRADED) 2024-03-11T21:39:54.080 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:39:53 smithi097 ceph-mon[33317]: osdmap e340: 8 total, 8 up, 8 in 2024-03-11T21:39:54.151 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:39:54.177 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:39:54.178 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:39:54.179 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:39:54.279 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:39:54.305 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:39:54.306 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:39:54.306 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:39:54.407 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:39:54.437 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:39:54.438 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:39:54.438 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:39:54.539 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:39:54.565 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:39:54.566 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:39:54.566 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:39:54.667 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:39:54.693 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:39:54.694 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:39:54.694 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:39:54.795 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:39:55.080 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:39:54 smithi097 ceph-mon[33317]: pgmap v661: 148 pgs: 2 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+degraded+remapped, 143 active+clean; 183 MiB data, 4.1 GiB used, 711 GiB / 715 GiB avail; 677 B/s rd, 0 B/s wr, 812 op/s; 17728/9372 objects degraded (189.159%); 0 B/s, 63 objects/s recovering 2024-03-11T21:39:55.080 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:39:54 smithi097 ceph-mon[33317]: purged_snaps scrub starts 2024-03-11T21:39:55.080 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:39:54 smithi097 ceph-mon[33317]: purged_snaps scrub ok 2024-03-11T21:39:55.113 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:39:54 smithi092 ceph-mon[29572]: pgmap v661: 148 pgs: 2 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+degraded+remapped, 143 active+clean; 183 MiB data, 4.1 GiB used, 711 GiB / 715 GiB avail; 677 B/s rd, 0 B/s wr, 812 op/s; 17728/9372 objects degraded (189.159%); 0 B/s, 63 objects/s recovering 2024-03-11T21:39:55.113 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:39:54 smithi092 ceph-mon[29572]: purged_snaps scrub starts 2024-03-11T21:39:55.113 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:39:54 smithi092 ceph-mon[29572]: purged_snaps scrub ok 2024-03-11T21:39:55.113 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:39:54 smithi092 ceph-mon[37563]: pgmap v661: 148 pgs: 2 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+degraded+remapped, 143 active+clean; 183 MiB data, 4.1 GiB used, 711 GiB / 715 GiB avail; 677 B/s rd, 0 B/s wr, 812 op/s; 17728/9372 objects degraded (189.159%); 0 B/s, 63 objects/s recovering 2024-03-11T21:39:55.113 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:39:54 smithi092 ceph-mon[37563]: purged_snaps scrub starts 2024-03-11T21:39:55.114 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:39:54 smithi092 ceph-mon[37563]: purged_snaps scrub ok 2024-03-11T21:39:55.403 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:39:55.404 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:39:55.404 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:39:55.429 INFO:tasks.cephadm.ceph_manager.ceph:removing pool_name unique_pool_2 2024-03-11T21:39:55.429 DEBUG:teuthology.orchestra.run.smithi092:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid 00db0016-dfed-11ee-95c8-87774f69a715 -- ceph osd pool rm unique_pool_2 unique_pool_2 --yes-i-really-really-mean-it 2024-03-11T21:39:55.506 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:39:55.534 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:39:55.535 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:39:55.535 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:39:55.636 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:39:55.669 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:39:55.669 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:39:55.669 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:39:55.770 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:39:55.806 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:39:55.806 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:39:55.806 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:39:55.907 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:39:55.939 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:39:55.940 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:39:55.940 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:39:56.041 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:39:56.071 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:39:56.072 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:39:56.072 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:39:56.079 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:39:55 smithi097 ceph-mon[33317]: osdmap e341: 8 total, 8 up, 8 in 2024-03-11T21:39:56.113 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:39:55 smithi092 ceph-mon[29572]: osdmap e341: 8 total, 8 up, 8 in 2024-03-11T21:39:56.113 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:39:55 smithi092 ceph-mon[37563]: osdmap e341: 8 total, 8 up, 8 in 2024-03-11T21:39:56.173 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:39:56.203 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:39:56.204 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:39:56.204 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:39:56.306 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:39:56.336 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:39:56.336 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:39:56.337 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:39:56.438 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:39:56.465 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:39:56.466 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:39:56.466 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:39:56.567 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:39:56.621 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:39:56.622 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:39:56.622 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:39:56.723 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:39:56.751 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:39:56.751 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:39:56.752 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:39:56.852 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:39:56.880 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:39:56.881 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:39:56.882 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:39:56.982 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:39:57.010 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:39:57.011 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:39:57.011 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:39:57.079 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:39:56 smithi097 ceph-mon[33317]: pgmap v664: 148 pgs: 2 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+degraded+remapped, 143 active+clean; 34 MiB data, 3.8 GiB used, 711 GiB / 715 GiB avail; 853 B/s rd, 0 B/s wr, 712 op/s; 17239/2220 objects degraded (776.532%); 0 B/s, 81 objects/s recovering 2024-03-11T21:39:57.112 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:39:57.113 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:39:56 smithi092 ceph-mon[29572]: pgmap v664: 148 pgs: 2 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+degraded+remapped, 143 active+clean; 34 MiB data, 3.8 GiB used, 711 GiB / 715 GiB avail; 853 B/s rd, 0 B/s wr, 712 op/s; 17239/2220 objects degraded (776.532%); 0 B/s, 81 objects/s recovering 2024-03-11T21:39:57.113 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:39:56 smithi092 ceph-mon[37563]: pgmap v664: 148 pgs: 2 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+degraded+remapped, 143 active+clean; 34 MiB data, 3.8 GiB used, 711 GiB / 715 GiB avail; 853 B/s rd, 0 B/s wr, 712 op/s; 17239/2220 objects degraded (776.532%); 0 B/s, 81 objects/s recovering 2024-03-11T21:39:57.140 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:39:57.140 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:39:57.141 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:39:57.242 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:39:57.268 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:39:57.268 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:39:57.269 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:39:57.370 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:39:57.399 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:39:57.400 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:39:57.400 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:39:57.501 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:39:57.534 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:39:57.535 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:39:57.535 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:39:57.636 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:39:57.665 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:39:57.665 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:39:57.666 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:39:57.767 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:39:57.795 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:39:57.795 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:39:57.796 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:39:57.897 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:39:57.926 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:39:57.926 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:39:57.927 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:39:58.027 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:39:58.055 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:39:58.056 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:39:58.056 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:39:58.157 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:39:58.185 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:39:58.186 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:39:58.186 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:39:58.287 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:39:58.314 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:39:58.315 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:39:58.315 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:39:58.363 INFO:journalctl@ceph.mgr.y.smithi092.stdout:Mar 11 21:39:57 smithi092 ceph-00db0016-dfed-11ee-95c8-87774f69a715-mgr-y[29801]: ::ffff:172.21.15.97 - - [11/Mar/2024:21:39:57] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-11T21:39:58.416 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:39:58.442 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:39:58.443 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:39:58.443 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:39:58.544 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:39:58.570 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:39:58.570 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:39:58.571 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:39:58.672 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:39:58.697 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:39:58.698 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:39:58.698 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:39:58.799 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:39:58.824 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:39:58.824 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:39:58.825 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:39:58.925 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:39:58.989 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:39:58.989 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:39:58.989 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:39:59.079 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:39:58 smithi097 ceph-mon[33317]: pgmap v665: 148 pgs: 2 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+degraded+remapped, 143 active+clean; 34 MiB data, 3.7 GiB used, 711 GiB / 715 GiB avail; 853 B/s rd, 0 B/s wr, 712 op/s; 17239/2220 objects degraded (776.532%); 0 B/s, 81 objects/s recovering 2024-03-11T21:39:59.079 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:39:58 smithi097 ceph-mon[33317]: Health check update: Degraded data redundancy: 17239/2220 objects degraded (776.532%), 5 pgs degraded, 5 pgs undersized (PG_DEGRADED) 2024-03-11T21:39:59.079 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:39:58 smithi097 ceph-mon[33317]: osdmap e342: 8 total, 8 up, 8 in 2024-03-11T21:39:59.090 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:39:59.113 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:39:58 smithi092 ceph-mon[29572]: pgmap v665: 148 pgs: 2 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+degraded+remapped, 143 active+clean; 34 MiB data, 3.7 GiB used, 711 GiB / 715 GiB avail; 853 B/s rd, 0 B/s wr, 712 op/s; 17239/2220 objects degraded (776.532%); 0 B/s, 81 objects/s recovering 2024-03-11T21:39:59.113 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:39:58 smithi092 ceph-mon[29572]: Health check update: Degraded data redundancy: 17239/2220 objects degraded (776.532%), 5 pgs degraded, 5 pgs undersized (PG_DEGRADED) 2024-03-11T21:39:59.113 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:39:58 smithi092 ceph-mon[29572]: osdmap e342: 8 total, 8 up, 8 in 2024-03-11T21:39:59.113 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:39:58 smithi092 ceph-mon[37563]: pgmap v665: 148 pgs: 2 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+degraded+remapped, 143 active+clean; 34 MiB data, 3.7 GiB used, 711 GiB / 715 GiB avail; 853 B/s rd, 0 B/s wr, 712 op/s; 17239/2220 objects degraded (776.532%); 0 B/s, 81 objects/s recovering 2024-03-11T21:39:59.113 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:39:58 smithi092 ceph-mon[37563]: Health check update: Degraded data redundancy: 17239/2220 objects degraded (776.532%), 5 pgs degraded, 5 pgs undersized (PG_DEGRADED) 2024-03-11T21:39:59.113 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:39:58 smithi092 ceph-mon[37563]: osdmap e342: 8 total, 8 up, 8 in 2024-03-11T21:39:59.116 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:39:59.116 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:39:59.117 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:39:59.218 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:39:59.246 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:39:59.246 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:39:59.247 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:39:59.347 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:39:59.375 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:39:59.376 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:39:59.376 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:39:59.477 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:39:59.502 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:39:59.503 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:39:59.503 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:39:59.604 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:39:59.630 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:39:59.631 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:39:59.631 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:39:59.732 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:39:59.760 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:39:59.760 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:39:59.761 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:39:59.861 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:39:59.889 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:39:59.889 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:39:59.889 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:39:59.990 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:40:00.017 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:40:00.018 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:40:00.018 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:40:00.119 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:40:00.145 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:40:00.146 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:40:00.146 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:40:00.247 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:40:00.273 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:40:00.274 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:40:00.275 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:40:00.375 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:40:00.402 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:40:00.402 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:40:00.403 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:40:00.504 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:40:00.531 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:40:00.532 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:40:00.532 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:40:00.633 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:40:00.661 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:40:00.662 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:40:00.662 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:40:00.763 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:40:00.790 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:40:00.791 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:40:00.791 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:40:00.892 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:40:00.920 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:40:00.921 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:40:00.921 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:40:01.022 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:40:01.050 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:40:01.051 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:40:01.051 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:40:01.079 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:40:00 smithi097 ceph-mon[33317]: pgmap v667: 148 pgs: 2 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+degraded+remapped, 143 active+clean; 582 KiB data, 3.5 GiB used, 712 GiB / 715 GiB avail; 1.7 KiB/s rd, 0 B/s wr, 489 op/s; 14740/594 objects degraded (2481.481%); 0 B/s, 497 objects/s recovering 2024-03-11T21:40:01.079 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:40:00 smithi097 ceph-mon[33317]: Health detail: HEALTH_WARN Degraded data redundancy: 14740/594 objects degraded (2481.481%), 5 pgs degraded, 5 pgs undersized 2024-03-11T21:40:01.079 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:40:00 smithi097 ceph-mon[33317]: [WRN] PG_DEGRADED: Degraded data redundancy: 14740/594 objects degraded (2481.481%), 5 pgs degraded, 5 pgs undersized 2024-03-11T21:40:01.079 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:40:00 smithi097 ceph-mon[33317]: pg 9.2 is stuck undersized for 3m, current state active+recovering+undersized+degraded+remapped, last acting [1,2] 2024-03-11T21:40:01.079 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:40:00 smithi097 ceph-mon[33317]: pg 9.3 is stuck undersized for 3m, current state active+recovery_wait+undersized+degraded+remapped, last acting [7,1] 2024-03-11T21:40:01.079 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:40:00 smithi097 ceph-mon[33317]: pg 9.4 is stuck undersized for 3m, current state active+recovering+undersized+degraded+remapped, last acting [1,4] 2024-03-11T21:40:01.079 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:40:00 smithi097 ceph-mon[33317]: pg 9.8 is stuck undersized for 3m, current state active+recovering+undersized+degraded+remapped, last acting [0,5] 2024-03-11T21:40:01.079 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:40:00 smithi097 ceph-mon[33317]: pg 9.f is stuck undersized for 3m, current state active+recovery_wait+undersized+degraded+remapped, last acting [5,1] 2024-03-11T21:40:01.113 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:40:00 smithi092 ceph-mon[29572]: pgmap v667: 148 pgs: 2 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+degraded+remapped, 143 active+clean; 582 KiB data, 3.5 GiB used, 712 GiB / 715 GiB avail; 1.7 KiB/s rd, 0 B/s wr, 489 op/s; 14740/594 objects degraded (2481.481%); 0 B/s, 497 objects/s recovering 2024-03-11T21:40:01.113 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:40:00 smithi092 ceph-mon[29572]: Health detail: HEALTH_WARN Degraded data redundancy: 14740/594 objects degraded (2481.481%), 5 pgs degraded, 5 pgs undersized 2024-03-11T21:40:01.113 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:40:00 smithi092 ceph-mon[29572]: [WRN] PG_DEGRADED: Degraded data redundancy: 14740/594 objects degraded (2481.481%), 5 pgs degraded, 5 pgs undersized 2024-03-11T21:40:01.113 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:40:00 smithi092 ceph-mon[29572]: pg 9.2 is stuck undersized for 3m, current state active+recovering+undersized+degraded+remapped, last acting [1,2] 2024-03-11T21:40:01.113 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:40:00 smithi092 ceph-mon[29572]: pg 9.3 is stuck undersized for 3m, current state active+recovery_wait+undersized+degraded+remapped, last acting [7,1] 2024-03-11T21:40:01.113 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:40:00 smithi092 ceph-mon[29572]: pg 9.4 is stuck undersized for 3m, current state active+recovering+undersized+degraded+remapped, last acting [1,4] 2024-03-11T21:40:01.114 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:40:00 smithi092 ceph-mon[29572]: pg 9.8 is stuck undersized for 3m, current state active+recovering+undersized+degraded+remapped, last acting [0,5] 2024-03-11T21:40:01.114 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:40:00 smithi092 ceph-mon[29572]: pg 9.f is stuck undersized for 3m, current state active+recovery_wait+undersized+degraded+remapped, last acting [5,1] 2024-03-11T21:40:01.114 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:40:00 smithi092 ceph-mon[37563]: pgmap v667: 148 pgs: 2 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+degraded+remapped, 143 active+clean; 582 KiB data, 3.5 GiB used, 712 GiB / 715 GiB avail; 1.7 KiB/s rd, 0 B/s wr, 489 op/s; 14740/594 objects degraded (2481.481%); 0 B/s, 497 objects/s recovering 2024-03-11T21:40:01.114 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:40:00 smithi092 ceph-mon[37563]: Health detail: HEALTH_WARN Degraded data redundancy: 14740/594 objects degraded (2481.481%), 5 pgs degraded, 5 pgs undersized 2024-03-11T21:40:01.114 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:40:00 smithi092 ceph-mon[37563]: [WRN] PG_DEGRADED: Degraded data redundancy: 14740/594 objects degraded (2481.481%), 5 pgs degraded, 5 pgs undersized 2024-03-11T21:40:01.114 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:40:00 smithi092 ceph-mon[37563]: pg 9.2 is stuck undersized for 3m, current state active+recovering+undersized+degraded+remapped, last acting [1,2] 2024-03-11T21:40:01.114 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:40:00 smithi092 ceph-mon[37563]: pg 9.3 is stuck undersized for 3m, current state active+recovery_wait+undersized+degraded+remapped, last acting [7,1] 2024-03-11T21:40:01.114 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:40:00 smithi092 ceph-mon[37563]: pg 9.4 is stuck undersized for 3m, current state active+recovering+undersized+degraded+remapped, last acting [1,4] 2024-03-11T21:40:01.114 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:40:00 smithi092 ceph-mon[37563]: pg 9.8 is stuck undersized for 3m, current state active+recovering+undersized+degraded+remapped, last acting [0,5] 2024-03-11T21:40:01.114 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:40:00 smithi092 ceph-mon[37563]: pg 9.f is stuck undersized for 3m, current state active+recovery_wait+undersized+degraded+remapped, last acting [5,1] 2024-03-11T21:40:01.152 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:40:01.180 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:40:01.180 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:40:01.180 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:40:01.281 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:40:01.309 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:40:01.310 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:40:01.310 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:40:01.411 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:40:01.440 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:40:01.440 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:40:01.441 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:40:01.541 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:40:01.570 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:40:01.571 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:40:01.571 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:40:01.672 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:40:01.700 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:40:01.702 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:40:01.702 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:40:01.803 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:40:01.834 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:40:01.835 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:40:01.835 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:40:01.936 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:40:01.965 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:40:01.966 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:40:01.966 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:40:02.067 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:40:02.079 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:40:01 smithi097 ceph-mon[33317]: 2.14 scrub starts 2024-03-11T21:40:02.079 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:40:01 smithi097 ceph-mon[33317]: 2.14 scrub ok 2024-03-11T21:40:02.094 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:40:02.095 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:40:02.095 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:40:02.113 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:40:01 smithi092 ceph-mon[29572]: 2.14 scrub starts 2024-03-11T21:40:02.113 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:40:01 smithi092 ceph-mon[29572]: 2.14 scrub ok 2024-03-11T21:40:02.113 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:40:01 smithi092 ceph-mon[37563]: 2.14 scrub starts 2024-03-11T21:40:02.113 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:40:01 smithi092 ceph-mon[37563]: 2.14 scrub ok 2024-03-11T21:40:02.196 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:40:02.224 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:40:02.224 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:40:02.225 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:40:02.325 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:40:02.352 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:40:02.352 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:40:02.353 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:40:02.454 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:40:02.480 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:40:02.481 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:40:02.481 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:40:02.582 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:40:02.617 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:40:02.617 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:40:02.617 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:40:02.718 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:40:02.743 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:40:02.743 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:40:02.744 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:40:02.844 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:40:02.873 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:40:02.873 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:40:02.874 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:40:02.975 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:40:03.001 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:40:03.001 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:40:03.001 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:40:03.079 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:40:02 smithi097 ceph-mon[33317]: pgmap v668: 148 pgs: 2 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+degraded+remapped, 143 active+clean; 582 KiB data, 3.5 GiB used, 712 GiB / 715 GiB avail; 645 B/s rd, 0 B/s wr, 266 op/s; 14740/594 objects degraded (2481.481%); 0 B/s, 335 objects/s recovering 2024-03-11T21:40:03.079 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:40:02 smithi097 ceph-mon[33317]: 4.a scrub starts 2024-03-11T21:40:03.079 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:40:02 smithi097 ceph-mon[33317]: 4.a scrub ok 2024-03-11T21:40:03.080 INFO:journalctl@ceph.iscsi.iscsi.a.smithi097.stdout:Mar 11 21:40:02 smithi097 ceph-00db0016-dfed-11ee-95c8-87774f69a715-iscsi-iscsi-a[60943]: debug there is no tcmu-runner data available 2024-03-11T21:40:03.102 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:40:03.113 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:40:02 smithi092 ceph-mon[29572]: pgmap v668: 148 pgs: 2 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+degraded+remapped, 143 active+clean; 582 KiB data, 3.5 GiB used, 712 GiB / 715 GiB avail; 645 B/s rd, 0 B/s wr, 266 op/s; 14740/594 objects degraded (2481.481%); 0 B/s, 335 objects/s recovering 2024-03-11T21:40:03.113 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:40:02 smithi092 ceph-mon[29572]: 4.a scrub starts 2024-03-11T21:40:03.113 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:40:02 smithi092 ceph-mon[29572]: 4.a scrub ok 2024-03-11T21:40:03.113 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:40:02 smithi092 ceph-mon[37563]: pgmap v668: 148 pgs: 2 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+degraded+remapped, 143 active+clean; 582 KiB data, 3.5 GiB used, 712 GiB / 715 GiB avail; 645 B/s rd, 0 B/s wr, 266 op/s; 14740/594 objects degraded (2481.481%); 0 B/s, 335 objects/s recovering 2024-03-11T21:40:03.113 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:40:02 smithi092 ceph-mon[37563]: 4.a scrub starts 2024-03-11T21:40:03.113 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:40:02 smithi092 ceph-mon[37563]: 4.a scrub ok 2024-03-11T21:40:03.130 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:40:03.131 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:40:03.131 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:40:03.232 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:40:03.261 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:40:03.262 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:40:03.262 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:40:03.366 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:40:03.400 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:40:03.401 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:40:03.401 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:40:03.502 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:40:03.532 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:40:03.533 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:40:03.533 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:40:03.634 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:40:03.658 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:40:03.659 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:40:03.659 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:40:03.760 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:40:03.785 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:40:03.786 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:40:03.786 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:40:03.887 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:40:03.913 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:40:03.914 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:40:03.914 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:40:04.015 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:40:04.041 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:40:04.041 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:40:04.042 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:40:04.082 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:40:03 smithi097 ceph-mon[33317]: 4.c scrub starts 2024-03-11T21:40:04.082 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:40:03 smithi097 ceph-mon[33317]: 4.c scrub ok 2024-03-11T21:40:04.082 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:40:03 smithi097 ceph-mon[33317]: from='client.24469 v1:172.21.15.97:0/404208310' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:40:04.082 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:40:03 smithi097 ceph-mon[33317]: osdmap e343: 8 total, 8 up, 8 in 2024-03-11T21:40:04.082 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:40:03 smithi097 ceph-mon[33317]: 5.11 scrub starts 2024-03-11T21:40:04.082 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:40:03 smithi097 ceph-mon[33317]: 5.11 scrub ok 2024-03-11T21:40:04.082 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:40:03 smithi097 ceph-mon[33317]: Health check update: Degraded data redundancy: 14740/594 objects degraded (2481.481%), 5 pgs degraded, 5 pgs undersized (PG_DEGRADED) 2024-03-11T21:40:04.082 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:40:03 smithi097 ceph-mon[33317]: osdmap e344: 8 total, 8 up, 8 in 2024-03-11T21:40:04.113 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:40:03 smithi092 ceph-mon[29572]: 4.c scrub starts 2024-03-11T21:40:04.113 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:40:03 smithi092 ceph-mon[29572]: 4.c scrub ok 2024-03-11T21:40:04.113 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:40:03 smithi092 ceph-mon[29572]: from='client.24469 v1:172.21.15.97:0/404208310' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:40:04.114 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:40:03 smithi092 ceph-mon[29572]: osdmap e343: 8 total, 8 up, 8 in 2024-03-11T21:40:04.114 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:40:03 smithi092 ceph-mon[29572]: 5.11 scrub starts 2024-03-11T21:40:04.114 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:40:03 smithi092 ceph-mon[29572]: 5.11 scrub ok 2024-03-11T21:40:04.114 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:40:03 smithi092 ceph-mon[29572]: Health check update: Degraded data redundancy: 14740/594 objects degraded (2481.481%), 5 pgs degraded, 5 pgs undersized (PG_DEGRADED) 2024-03-11T21:40:04.114 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:40:03 smithi092 ceph-mon[29572]: osdmap e344: 8 total, 8 up, 8 in 2024-03-11T21:40:04.114 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:40:03 smithi092 ceph-mon[37563]: 4.c scrub starts 2024-03-11T21:40:04.114 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:40:03 smithi092 ceph-mon[37563]: 4.c scrub ok 2024-03-11T21:40:04.114 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:40:03 smithi092 ceph-mon[37563]: from='client.24469 v1:172.21.15.97:0/404208310' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:40:04.114 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:40:03 smithi092 ceph-mon[37563]: osdmap e343: 8 total, 8 up, 8 in 2024-03-11T21:40:04.114 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:40:03 smithi092 ceph-mon[37563]: 5.11 scrub starts 2024-03-11T21:40:04.114 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:40:03 smithi092 ceph-mon[37563]: 5.11 scrub ok 2024-03-11T21:40:04.115 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:40:03 smithi092 ceph-mon[37563]: Health check update: Degraded data redundancy: 14740/594 objects degraded (2481.481%), 5 pgs degraded, 5 pgs undersized (PG_DEGRADED) 2024-03-11T21:40:04.115 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:40:03 smithi092 ceph-mon[37563]: osdmap e344: 8 total, 8 up, 8 in 2024-03-11T21:40:04.143 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:40:04.170 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:40:04.170 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:40:04.171 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:40:04.271 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:40:04.296 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:40:04.296 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:40:04.297 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:40:04.397 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:40:04.424 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:40:04.425 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:40:04.425 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:40:04.526 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:40:04.553 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:40:04.553 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:40:04.554 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:40:04.655 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:40:04.681 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:40:04.682 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:40:04.682 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:40:04.783 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:40:04.807 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:40:04.808 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:40:04.808 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:40:04.909 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:40:04.933 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:40:04.934 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:40:04.934 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:40:05.035 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:40:05.058 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:40:05.059 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:40:05.059 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:40:05.079 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:40:04 smithi097 ceph-mon[33317]: pgmap v670: 148 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 4 active+recovering+undersized+degraded+remapped, 143 active+clean; 582 KiB data, 3.5 GiB used, 712 GiB / 715 GiB avail; 639 B/s rd, 0 B/s wr, 68 op/s; 14706/594 objects degraded (2475.758%); 0 B/s, 316 objects/s recovering 2024-03-11T21:40:05.113 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:40:04 smithi092 ceph-mon[29572]: pgmap v670: 148 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 4 active+recovering+undersized+degraded+remapped, 143 active+clean; 582 KiB data, 3.5 GiB used, 712 GiB / 715 GiB avail; 639 B/s rd, 0 B/s wr, 68 op/s; 14706/594 objects degraded (2475.758%); 0 B/s, 316 objects/s recovering 2024-03-11T21:40:05.113 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:40:04 smithi092 ceph-mon[37563]: pgmap v670: 148 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 4 active+recovering+undersized+degraded+remapped, 143 active+clean; 582 KiB data, 3.5 GiB used, 712 GiB / 715 GiB avail; 639 B/s rd, 0 B/s wr, 68 op/s; 14706/594 objects degraded (2475.758%); 0 B/s, 316 objects/s recovering 2024-03-11T21:40:05.160 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:40:05.185 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:40:05.186 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:40:05.186 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:40:05.287 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:40:05.430 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:40:05.430 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:40:05.431 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:40:05.531 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:40:05.557 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:40:05.558 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:40:05.558 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:40:05.659 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:40:05.683 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:40:05.684 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:40:05.684 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:40:05.785 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:40:05.809 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:40:05.810 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:40:05.810 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:40:05.911 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:40:05.936 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:40:05.937 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:40:05.937 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:40:06.038 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:40:06.063 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:40:06.063 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:40:06.064 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:40:06.165 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:40:06.190 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:40:06.191 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:40:06.191 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:40:06.292 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:40:06.318 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:40:06.319 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:40:06.319 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:40:06.419 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:40:06.445 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:40:06.445 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:40:06.445 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:40:06.546 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:40:06.571 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:40:06.571 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:40:06.572 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:40:06.673 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:40:06.699 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:40:06.699 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:40:06.700 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:40:06.801 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:40:06.826 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:40:06.827 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:40:06.827 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:40:06.928 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:40:06.953 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:40:06.954 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:40:06.954 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:40:07.055 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:40:07.081 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:40:07.081 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:40:07.081 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:40:07.113 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:40:06 smithi092 ceph-mon[29572]: pgmap v672: 148 pgs: 1 peering, 1 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+degraded+remapped, 143 active+clean; 582 KiB data, 3.5 GiB used, 712 GiB / 715 GiB avail; 738 B/s rd, 0 op/s; 11981/594 objects degraded (2017.003%); 0 B/s, 481 objects/s recovering 2024-03-11T21:40:07.113 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:40:06 smithi092 ceph-mon[29572]: osdmap e345: 8 total, 8 up, 8 in 2024-03-11T21:40:07.114 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:40:06 smithi092 ceph-mon[37563]: pgmap v672: 148 pgs: 1 peering, 1 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+degraded+remapped, 143 active+clean; 582 KiB data, 3.5 GiB used, 712 GiB / 715 GiB avail; 738 B/s rd, 0 op/s; 11981/594 objects degraded (2017.003%); 0 B/s, 481 objects/s recovering 2024-03-11T21:40:07.114 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:40:06 smithi092 ceph-mon[37563]: osdmap e345: 8 total, 8 up, 8 in 2024-03-11T21:40:07.182 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:40:07.208 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:40:07.208 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:40:07.209 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:40:07.309 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:40:07.329 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:40:06 smithi097 ceph-mon[33317]: pgmap v672: 148 pgs: 1 peering, 1 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+degraded+remapped, 143 active+clean; 582 KiB data, 3.5 GiB used, 712 GiB / 715 GiB avail; 738 B/s rd, 0 op/s; 11981/594 objects degraded (2017.003%); 0 B/s, 481 objects/s recovering 2024-03-11T21:40:07.330 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:40:06 smithi097 ceph-mon[33317]: osdmap e345: 8 total, 8 up, 8 in 2024-03-11T21:40:07.336 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:40:07.337 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:40:07.338 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:40:07.439 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:40:07.463 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:40:07.464 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:40:07.464 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:40:07.565 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:40:07.593 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:40:07.594 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:40:07.594 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:40:07.695 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:40:07.719 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:40:07.720 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:40:07.720 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:40:07.821 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:40:07.845 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:40:07.846 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:40:07.846 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:40:07.947 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:40:07.971 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:40:07.972 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:40:07.972 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:40:08.073 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:40:08.098 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:40:08.098 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:40:08.099 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:40:08.113 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:40:07 smithi092 ceph-mon[29572]: osdmap e346: 8 total, 8 up, 8 in 2024-03-11T21:40:08.114 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:40:07 smithi092 ceph-mon[29572]: 2.7 scrub starts 2024-03-11T21:40:08.114 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:40:07 smithi092 ceph-mon[29572]: 2.7 scrub ok 2024-03-11T21:40:08.114 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:40:07 smithi092 ceph-mon[29572]: from='mgr.14607 ' entity='mgr.y' 2024-03-11T21:40:08.114 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:40:07 smithi092 ceph-mon[29572]: from='mgr.14607 v1:172.21.15.92:0/1702717597' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T21:40:08.114 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:40:07 smithi092 ceph-mon[37563]: osdmap e346: 8 total, 8 up, 8 in 2024-03-11T21:40:08.115 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:40:07 smithi092 ceph-mon[37563]: 2.7 scrub starts 2024-03-11T21:40:08.115 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:40:07 smithi092 ceph-mon[37563]: 2.7 scrub ok 2024-03-11T21:40:08.115 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:40:07 smithi092 ceph-mon[37563]: from='mgr.14607 ' entity='mgr.y' 2024-03-11T21:40:08.115 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:40:07 smithi092 ceph-mon[37563]: from='mgr.14607 v1:172.21.15.92:0/1702717597' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T21:40:08.116 INFO:journalctl@ceph.mgr.y.smithi092.stdout:Mar 11 21:40:07 smithi092 ceph-00db0016-dfed-11ee-95c8-87774f69a715-mgr-y[29801]: ::ffff:172.21.15.97 - - [11/Mar/2024:21:40:07] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-11T21:40:08.199 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:40:08.223 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:40:08.224 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:40:08.225 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:40:08.325 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:40:08.329 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:40:07 smithi097 ceph-mon[33317]: osdmap e346: 8 total, 8 up, 8 in 2024-03-11T21:40:08.329 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:40:07 smithi097 ceph-mon[33317]: 2.7 scrub starts 2024-03-11T21:40:08.329 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:40:07 smithi097 ceph-mon[33317]: 2.7 scrub ok 2024-03-11T21:40:08.329 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:40:07 smithi097 ceph-mon[33317]: from='mgr.14607 ' entity='mgr.y' 2024-03-11T21:40:08.329 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:40:07 smithi097 ceph-mon[33317]: from='mgr.14607 v1:172.21.15.92:0/1702717597' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T21:40:08.349 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:40:08.350 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:40:08.350 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:40:08.451 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:40:08.475 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:40:08.476 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:40:08.476 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:40:08.577 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:40:08.601 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:40:08.602 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:40:08.602 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:40:08.703 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:40:08.729 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:40:08.730 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:40:08.730 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:40:08.831 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:40:08.857 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:40:08.857 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:40:08.857 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:40:08.958 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:40:08.982 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:40:08.983 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:40:08.984 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:40:09.085 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:40:09.110 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:40:09.111 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:40:09.111 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:40:09.113 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:40:08 smithi092 ceph-mon[29572]: pgmap v675: 148 pgs: 1 peering, 1 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+degraded+remapped, 143 active+clean; 582 KiB data, 3.5 GiB used, 712 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s; 11981/594 objects degraded (2017.003%); 0 B/s, 607 objects/s recovering 2024-03-11T21:40:09.113 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:40:08 smithi092 ceph-mon[29572]: 5.1c deep-scrub starts 2024-03-11T21:40:09.113 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:40:08 smithi092 ceph-mon[29572]: 5.1c deep-scrub ok 2024-03-11T21:40:09.113 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:40:08 smithi092 ceph-mon[29572]: Health check update: Degraded data redundancy: 11981/594 objects degraded (2017.003%), 4 pgs degraded, 4 pgs undersized (PG_DEGRADED) 2024-03-11T21:40:09.113 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:40:08 smithi092 ceph-mon[37563]: pgmap v675: 148 pgs: 1 peering, 1 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+degraded+remapped, 143 active+clean; 582 KiB data, 3.5 GiB used, 712 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s; 11981/594 objects degraded (2017.003%); 0 B/s, 607 objects/s recovering 2024-03-11T21:40:09.113 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:40:08 smithi092 ceph-mon[37563]: 5.1c deep-scrub starts 2024-03-11T21:40:09.114 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:40:08 smithi092 ceph-mon[37563]: 5.1c deep-scrub ok 2024-03-11T21:40:09.114 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:40:08 smithi092 ceph-mon[37563]: Health check update: Degraded data redundancy: 11981/594 objects degraded (2017.003%), 4 pgs degraded, 4 pgs undersized (PG_DEGRADED) 2024-03-11T21:40:09.212 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:40:09.238 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:40:09.238 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:40:09.239 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:40:09.329 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:40:08 smithi097 ceph-mon[33317]: pgmap v675: 148 pgs: 1 peering, 1 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+degraded+remapped, 143 active+clean; 582 KiB data, 3.5 GiB used, 712 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s; 11981/594 objects degraded (2017.003%); 0 B/s, 607 objects/s recovering 2024-03-11T21:40:09.329 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:40:08 smithi097 ceph-mon[33317]: 5.1c deep-scrub starts 2024-03-11T21:40:09.329 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:40:08 smithi097 ceph-mon[33317]: 5.1c deep-scrub ok 2024-03-11T21:40:09.329 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:40:08 smithi097 ceph-mon[33317]: Health check update: Degraded data redundancy: 11981/594 objects degraded (2017.003%), 4 pgs degraded, 4 pgs undersized (PG_DEGRADED) 2024-03-11T21:40:09.340 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:40:09.364 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:40:09.365 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:40:09.365 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:40:09.466 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:40:09.493 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:40:09.493 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:40:09.493 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:40:09.594 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:40:09.619 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:40:09.619 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:40:09.620 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:40:09.720 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:40:09.745 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:40:09.746 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:40:09.746 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:40:09.847 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:40:09.873 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:40:09.874 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:40:09.874 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:40:09.975 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:40:09.999 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:40:10.000 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:40:10.001 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:40:10.101 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:40:10.113 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:40:09 smithi092 ceph-mon[29572]: 2.11 scrub starts 2024-03-11T21:40:10.113 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:40:09 smithi092 ceph-mon[29572]: 2.11 scrub ok 2024-03-11T21:40:10.113 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:40:09 smithi092 ceph-mon[37563]: 2.11 scrub starts 2024-03-11T21:40:10.113 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:40:09 smithi092 ceph-mon[37563]: 2.11 scrub ok 2024-03-11T21:40:10.125 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:40:10.126 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:40:10.126 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:40:10.227 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:40:10.253 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:40:10.254 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:40:10.254 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:40:10.329 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:40:09 smithi097 ceph-mon[33317]: 2.11 scrub starts 2024-03-11T21:40:10.329 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:40:09 smithi097 ceph-mon[33317]: 2.11 scrub ok 2024-03-11T21:40:10.356 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:40:10.382 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:40:10.383 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:40:10.383 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:40:10.484 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:40:10.509 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:40:10.510 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:40:10.510 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:40:10.611 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:40:10.635 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:40:10.635 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:40:10.636 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:40:10.737 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:40:10.761 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:40:10.761 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:40:10.761 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:40:10.862 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:40:10.892 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:40:10.893 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:40:10.893 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:40:10.994 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:40:11.018 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:40:11.019 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:40:11.019 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:40:11.113 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:40:10 smithi092 ceph-mon[29572]: purged_snaps scrub starts 2024-03-11T21:40:11.114 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:40:10 smithi092 ceph-mon[29572]: purged_snaps scrub ok 2024-03-11T21:40:11.114 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:40:10 smithi092 ceph-mon[29572]: pgmap v676: 148 pgs: 3 active+recovering+undersized+degraded+remapped, 145 active+clean; 582 KiB data, 3.6 GiB used, 712 GiB / 715 GiB avail; 1.7 KiB/s rd, 1 op/s; 9380/594 objects degraded (1579.125%); 0 B/s, 887 objects/s recovering 2024-03-11T21:40:11.114 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:40:10 smithi092 ceph-mon[29572]: osdmap e347: 8 total, 8 up, 8 in 2024-03-11T21:40:11.114 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:40:10 smithi092 ceph-mon[29572]: 2.1 deep-scrub starts 2024-03-11T21:40:11.114 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:40:10 smithi092 ceph-mon[29572]: 2.1 deep-scrub ok 2024-03-11T21:40:11.114 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:40:10 smithi092 ceph-mon[37563]: purged_snaps scrub starts 2024-03-11T21:40:11.114 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:40:10 smithi092 ceph-mon[37563]: purged_snaps scrub ok 2024-03-11T21:40:11.114 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:40:10 smithi092 ceph-mon[37563]: pgmap v676: 148 pgs: 3 active+recovering+undersized+degraded+remapped, 145 active+clean; 582 KiB data, 3.6 GiB used, 712 GiB / 715 GiB avail; 1.7 KiB/s rd, 1 op/s; 9380/594 objects degraded (1579.125%); 0 B/s, 887 objects/s recovering 2024-03-11T21:40:11.114 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:40:10 smithi092 ceph-mon[37563]: osdmap e347: 8 total, 8 up, 8 in 2024-03-11T21:40:11.114 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:40:10 smithi092 ceph-mon[37563]: 2.1 deep-scrub starts 2024-03-11T21:40:11.114 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:40:10 smithi092 ceph-mon[37563]: 2.1 deep-scrub ok 2024-03-11T21:40:11.120 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:40:11.144 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:40:11.145 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:40:11.145 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:40:11.246 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:40:11.283 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:40:11.284 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:40:11.284 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:40:11.330 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:40:10 smithi097 ceph-mon[33317]: purged_snaps scrub starts 2024-03-11T21:40:11.331 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:40:10 smithi097 ceph-mon[33317]: purged_snaps scrub ok 2024-03-11T21:40:11.331 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:40:10 smithi097 ceph-mon[33317]: pgmap v676: 148 pgs: 3 active+recovering+undersized+degraded+remapped, 145 active+clean; 582 KiB data, 3.6 GiB used, 712 GiB / 715 GiB avail; 1.7 KiB/s rd, 1 op/s; 9380/594 objects degraded (1579.125%); 0 B/s, 887 objects/s recovering 2024-03-11T21:40:11.331 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:40:10 smithi097 ceph-mon[33317]: osdmap e347: 8 total, 8 up, 8 in 2024-03-11T21:40:11.331 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:40:10 smithi097 ceph-mon[33317]: 2.1 deep-scrub starts 2024-03-11T21:40:11.331 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:40:10 smithi097 ceph-mon[33317]: 2.1 deep-scrub ok 2024-03-11T21:40:11.385 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:40:11.413 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:40:11.413 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:40:11.414 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:40:11.515 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:40:11.539 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:40:11.539 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:40:11.540 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:40:11.641 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:40:11.667 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:40:11.667 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:40:11.668 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:40:11.768 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:40:11.793 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:40:11.793 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:40:11.794 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:40:11.894 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:40:11.920 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:40:11.920 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:40:11.921 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:40:12.021 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:40:12.046 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:40:12.047 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:40:12.047 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:40:12.148 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:40:12.172 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:40:12.172 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:40:12.173 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:40:12.274 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:40:12.299 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:40:12.299 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:40:12.300 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:40:12.329 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:40:11 smithi097 ceph-mon[33317]: osdmap e348: 8 total, 8 up, 8 in 2024-03-11T21:40:12.329 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:40:11 smithi097 ceph-mon[33317]: 2.e scrub starts 2024-03-11T21:40:12.363 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:40:11 smithi092 ceph-mon[29572]: osdmap e348: 8 total, 8 up, 8 in 2024-03-11T21:40:12.363 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:40:11 smithi092 ceph-mon[29572]: 2.e scrub starts 2024-03-11T21:40:12.364 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:40:11 smithi092 ceph-mon[37563]: osdmap e348: 8 total, 8 up, 8 in 2024-03-11T21:40:12.364 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:40:11 smithi092 ceph-mon[37563]: 2.e scrub starts 2024-03-11T21:40:12.401 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:40:12.426 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:40:12.427 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:40:12.427 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:40:12.528 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:40:12.551 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:40:12.552 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:40:12.552 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:40:12.653 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:40:12.679 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:40:12.680 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:40:12.680 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:40:12.781 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:40:12.805 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:40:12.806 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:40:12.806 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:40:12.907 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:40:12.931 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:40:12.932 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:40:12.932 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:40:13.033 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:40:13.057 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:40:13.058 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:40:13.058 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:40:13.079 INFO:journalctl@ceph.iscsi.iscsi.a.smithi097.stdout:Mar 11 21:40:12 smithi097 ceph-00db0016-dfed-11ee-95c8-87774f69a715-iscsi-iscsi-a[60943]: debug there is no tcmu-runner data available 2024-03-11T21:40:13.079 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:40:12 smithi097 ceph-mon[33317]: 2.e scrub ok 2024-03-11T21:40:13.079 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:40:12 smithi097 ceph-mon[33317]: 2.13 scrub starts 2024-03-11T21:40:13.079 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:40:12 smithi097 ceph-mon[33317]: 2.13 scrub ok 2024-03-11T21:40:13.079 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:40:12 smithi097 ceph-mon[33317]: pgmap v679: 148 pgs: 3 active+recovering+undersized+degraded+remapped, 145 active+clean; 582 KiB data, 3.6 GiB used, 712 GiB / 715 GiB avail; 935 B/s rd, 0 op/s; 9380/594 objects degraded (1579.125%); 0 B/s, 475 objects/s recovering 2024-03-11T21:40:13.080 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:40:12 smithi097 ceph-mon[33317]: osdmap e349: 8 total, 8 up, 8 in 2024-03-11T21:40:13.113 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:40:12 smithi092 ceph-mon[29572]: 2.e scrub ok 2024-03-11T21:40:13.113 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:40:12 smithi092 ceph-mon[29572]: 2.13 scrub starts 2024-03-11T21:40:13.113 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:40:12 smithi092 ceph-mon[29572]: 2.13 scrub ok 2024-03-11T21:40:13.114 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:40:12 smithi092 ceph-mon[29572]: pgmap v679: 148 pgs: 3 active+recovering+undersized+degraded+remapped, 145 active+clean; 582 KiB data, 3.6 GiB used, 712 GiB / 715 GiB avail; 935 B/s rd, 0 op/s; 9380/594 objects degraded (1579.125%); 0 B/s, 475 objects/s recovering 2024-03-11T21:40:13.114 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:40:12 smithi092 ceph-mon[29572]: osdmap e349: 8 total, 8 up, 8 in 2024-03-11T21:40:13.114 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:40:12 smithi092 ceph-mon[37563]: 2.e scrub ok 2024-03-11T21:40:13.114 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:40:12 smithi092 ceph-mon[37563]: 2.13 scrub starts 2024-03-11T21:40:13.114 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:40:12 smithi092 ceph-mon[37563]: 2.13 scrub ok 2024-03-11T21:40:13.114 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:40:12 smithi092 ceph-mon[37563]: pgmap v679: 148 pgs: 3 active+recovering+undersized+degraded+remapped, 145 active+clean; 582 KiB data, 3.6 GiB used, 712 GiB / 715 GiB avail; 935 B/s rd, 0 op/s; 9380/594 objects degraded (1579.125%); 0 B/s, 475 objects/s recovering 2024-03-11T21:40:13.114 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:40:12 smithi092 ceph-mon[37563]: osdmap e349: 8 total, 8 up, 8 in 2024-03-11T21:40:13.160 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:40:13.183 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:40:13.184 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:40:13.184 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:40:13.285 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:40:13.310 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:40:13.310 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:40:13.311 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:40:13.412 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:40:13.435 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:40:13.436 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:40:13.436 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:40:13.537 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:40:13.560 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:40:13.561 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:40:13.561 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:40:13.662 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:40:13.685 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:40:13.686 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:40:13.686 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:40:13.787 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:40:13.810 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:40:13.811 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:40:13.811 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:40:13.912 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:40:13.935 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:40:13.936 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:40:13.936 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:40:14.037 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:40:14.060 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:40:14.061 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:40:14.061 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:40:14.162 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:40:14.186 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:40:14.186 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:40:14.187 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:40:14.288 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:40:14.312 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:40:14.312 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:40:14.313 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:40:14.331 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:40:13 smithi097 ceph-mon[33317]: from='client.24469 v1:172.21.15.97:0/404208310' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:40:14.331 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:40:13 smithi097 ceph-mon[33317]: 4.f scrub starts 2024-03-11T21:40:14.331 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:40:13 smithi097 ceph-mon[33317]: 4.f scrub ok 2024-03-11T21:40:14.331 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:40:13 smithi097 ceph-mon[33317]: Health check update: Degraded data redundancy: 9380/594 objects degraded (1579.125%), 3 pgs degraded, 3 pgs undersized (PG_DEGRADED) 2024-03-11T21:40:14.331 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:40:13 smithi097 ceph-mon[33317]: osdmap e350: 8 total, 8 up, 8 in 2024-03-11T21:40:14.363 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:40:13 smithi092 ceph-mon[29572]: from='client.24469 v1:172.21.15.97:0/404208310' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:40:14.364 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:40:13 smithi092 ceph-mon[29572]: 4.f scrub starts 2024-03-11T21:40:14.364 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:40:13 smithi092 ceph-mon[29572]: 4.f scrub ok 2024-03-11T21:40:14.364 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:40:13 smithi092 ceph-mon[29572]: Health check update: Degraded data redundancy: 9380/594 objects degraded (1579.125%), 3 pgs degraded, 3 pgs undersized (PG_DEGRADED) 2024-03-11T21:40:14.364 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:40:13 smithi092 ceph-mon[29572]: osdmap e350: 8 total, 8 up, 8 in 2024-03-11T21:40:14.364 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:40:13 smithi092 ceph-mon[37563]: from='client.24469 v1:172.21.15.97:0/404208310' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:40:14.364 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:40:13 smithi092 ceph-mon[37563]: 4.f scrub starts 2024-03-11T21:40:14.364 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:40:13 smithi092 ceph-mon[37563]: 4.f scrub ok 2024-03-11T21:40:14.364 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:40:13 smithi092 ceph-mon[37563]: Health check update: Degraded data redundancy: 9380/594 objects degraded (1579.125%), 3 pgs degraded, 3 pgs undersized (PG_DEGRADED) 2024-03-11T21:40:14.364 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:40:13 smithi092 ceph-mon[37563]: osdmap e350: 8 total, 8 up, 8 in 2024-03-11T21:40:14.414 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:40:14.438 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:40:14.439 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:40:14.439 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:40:14.540 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:40:14.564 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:40:14.564 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:40:14.565 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:40:14.666 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:40:14.690 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:40:14.691 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:40:14.691 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:40:14.792 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:40:14.816 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:40:14.817 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:40:14.818 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:40:14.918 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:40:14.942 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:40:14.943 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:40:14.946 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:40:15.047 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:40:15.072 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:40:15.072 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:40:15.073 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:40:15.174 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:40:15.197 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:40:15.198 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:40:15.199 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:40:15.217 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:40:14 smithi092 ceph-mon[29572]: pgmap v681: 148 pgs: 3 active+recovering+undersized+degraded+remapped, 145 active+clean; 582 KiB data, 3.6 GiB used, 712 GiB / 715 GiB avail; 853 B/s rd, 0 op/s; 8245/594 objects degraded (1388.047%); 0 B/s, 622 objects/s recovering 2024-03-11T21:40:15.218 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:40:14 smithi092 ceph-mon[37563]: pgmap v681: 148 pgs: 3 active+recovering+undersized+degraded+remapped, 145 active+clean; 582 KiB data, 3.6 GiB used, 712 GiB / 715 GiB avail; 853 B/s rd, 0 op/s; 8245/594 objects degraded (1388.047%); 0 B/s, 622 objects/s recovering 2024-03-11T21:40:15.299 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:40:15.324 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:40:15.325 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:40:15.325 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:40:15.329 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:40:14 smithi097 ceph-mon[33317]: pgmap v681: 148 pgs: 3 active+recovering+undersized+degraded+remapped, 145 active+clean; 582 KiB data, 3.6 GiB used, 712 GiB / 715 GiB avail; 853 B/s rd, 0 op/s; 8245/594 objects degraded (1388.047%); 0 B/s, 622 objects/s recovering 2024-03-11T21:40:15.426 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:40:15.451 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:40:15.451 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:40:15.452 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:40:15.552 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:40:15.577 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:40:15.577 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:40:15.577 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:40:15.678 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:40:15.703 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:40:15.703 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:40:15.703 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:40:15.804 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:40:15.810 INFO:teuthology.orchestra.run.smithi092.stdout:osd.0: {} 2024-03-11T21:40:15.810 INFO:teuthology.orchestra.run.smithi092.stdout:osd.1: {} 2024-03-11T21:40:15.810 INFO:teuthology.orchestra.run.smithi092.stdout:osd.2: {} 2024-03-11T21:40:15.810 INFO:teuthology.orchestra.run.smithi092.stdout:osd.3: {} 2024-03-11T21:40:15.810 INFO:teuthology.orchestra.run.smithi092.stdout:osd.4: {} 2024-03-11T21:40:15.810 INFO:teuthology.orchestra.run.smithi092.stdout:osd.5: {} 2024-03-11T21:40:15.810 INFO:teuthology.orchestra.run.smithi092.stdout:osd.6: {} 2024-03-11T21:40:15.811 INFO:teuthology.orchestra.run.smithi092.stdout:osd.7: {} 2024-03-11T21:40:15.811 INFO:teuthology.orchestra.run.smithi092.stderr:osd.0: osd_enable_op_tracker = '' 2024-03-11T21:40:15.811 INFO:teuthology.orchestra.run.smithi092.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-11T21:40:15.811 INFO:teuthology.orchestra.run.smithi092.stderr:osd.2: osd_enable_op_tracker = '' 2024-03-11T21:40:15.811 INFO:teuthology.orchestra.run.smithi092.stderr:osd.3: osd_enable_op_tracker = '' 2024-03-11T21:40:15.811 INFO:teuthology.orchestra.run.smithi092.stderr:osd.4: osd_enable_op_tracker = '' 2024-03-11T21:40:15.811 INFO:teuthology.orchestra.run.smithi092.stderr:osd.5: osd_enable_op_tracker = '' 2024-03-11T21:40:15.811 INFO:teuthology.orchestra.run.smithi092.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-11T21:40:15.811 INFO:teuthology.orchestra.run.smithi092.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-11T21:40:16.113 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:40:15 smithi092 ceph-mon[29572]: 4.9 scrub starts 2024-03-11T21:40:16.113 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:40:15 smithi092 ceph-mon[29572]: 4.9 scrub ok 2024-03-11T21:40:16.114 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:40:15 smithi092 ceph-mon[29572]: 5.2 deep-scrub starts 2024-03-11T21:40:16.114 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:40:15 smithi092 ceph-mon[29572]: 5.2 deep-scrub ok 2024-03-11T21:40:16.114 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:40:15 smithi092 ceph-mon[29572]: from='client.? v1:172.21.15.92:0/3306003536' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-11T21:40:16.114 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:40:15 smithi092 ceph-mon[37563]: 4.9 scrub starts 2024-03-11T21:40:16.114 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:40:15 smithi092 ceph-mon[37563]: 4.9 scrub ok 2024-03-11T21:40:16.114 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:40:15 smithi092 ceph-mon[37563]: 5.2 deep-scrub starts 2024-03-11T21:40:16.114 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:40:15 smithi092 ceph-mon[37563]: 5.2 deep-scrub ok 2024-03-11T21:40:16.114 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:40:15 smithi092 ceph-mon[37563]: from='client.? v1:172.21.15.92:0/3306003536' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-11T21:40:16.329 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:40:15 smithi097 ceph-mon[33317]: 4.9 scrub starts 2024-03-11T21:40:16.329 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:40:15 smithi097 ceph-mon[33317]: 4.9 scrub ok 2024-03-11T21:40:16.330 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:40:15 smithi097 ceph-mon[33317]: 5.2 deep-scrub starts 2024-03-11T21:40:16.330 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:40:15 smithi097 ceph-mon[33317]: 5.2 deep-scrub ok 2024-03-11T21:40:16.330 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:40:15 smithi097 ceph-mon[33317]: from='client.? v1:172.21.15.92:0/3306003536' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-11T21:40:16.346 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:40:16.346 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:40:16.347 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:40:16.448 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:40:16.472 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:40:16.473 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:40:16.474 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:40:16.574 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:40:16.602 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:40:16.603 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:40:16.604 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:40:16.704 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:40:16.728 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:40:16.729 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:40:16.729 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:40:16.830 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:40:16.853 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:40:16.854 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:40:16.854 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:40:16.955 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:40:16.978 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:40:16.979 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:40:16.979 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:40:17.080 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:40:17.105 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:40:17.106 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:40:17.106 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:40:17.207 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:40:17.329 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:40:16 smithi097 ceph-mon[33317]: pgmap v683: 148 pgs: 2 active+recovering+undersized+degraded+remapped, 146 active+clean; 582 KiB data, 3.6 GiB used, 712 GiB / 715 GiB avail; 939 B/s rd, 0 op/s; 7443/594 objects degraded (1253.030%); 0 B/s, 355 objects/s recovering 2024-03-11T21:40:17.329 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:40:16 smithi097 ceph-mon[33317]: 4.1c scrub starts 2024-03-11T21:40:17.329 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:40:16 smithi097 ceph-mon[33317]: 4.1c scrub ok 2024-03-11T21:40:17.363 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:40:16 smithi092 ceph-mon[29572]: pgmap v683: 148 pgs: 2 active+recovering+undersized+degraded+remapped, 146 active+clean; 582 KiB data, 3.6 GiB used, 712 GiB / 715 GiB avail; 939 B/s rd, 0 op/s; 7443/594 objects degraded (1253.030%); 0 B/s, 355 objects/s recovering 2024-03-11T21:40:17.363 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:40:16 smithi092 ceph-mon[29572]: 4.1c scrub starts 2024-03-11T21:40:17.363 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:40:16 smithi092 ceph-mon[29572]: 4.1c scrub ok 2024-03-11T21:40:17.364 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:40:16 smithi092 ceph-mon[37563]: pgmap v683: 148 pgs: 2 active+recovering+undersized+degraded+remapped, 146 active+clean; 582 KiB data, 3.6 GiB used, 712 GiB / 715 GiB avail; 939 B/s rd, 0 op/s; 7443/594 objects degraded (1253.030%); 0 B/s, 355 objects/s recovering 2024-03-11T21:40:17.364 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:40:16 smithi092 ceph-mon[37563]: 4.1c scrub starts 2024-03-11T21:40:17.364 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:40:16 smithi092 ceph-mon[37563]: 4.1c scrub ok 2024-03-11T21:40:17.398 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:40:17.399 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:40:17.399 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:40:17.500 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:40:17.525 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:40:17.526 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:40:17.526 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:40:17.627 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:40:17.652 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:40:17.652 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:40:17.653 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:40:17.754 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:40:17.778 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:40:17.779 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:40:17.780 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:40:17.880 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:40:17.905 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:40:17.906 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:40:17.906 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:40:18.007 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:40:18.031 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:40:18.032 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:40:18.032 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:40:18.133 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:40:18.158 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:40:18.159 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:40:18.159 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:40:18.261 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:40:18.285 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:40:18.285 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:40:18.286 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:40:18.363 INFO:journalctl@ceph.mgr.y.smithi092.stdout:Mar 11 21:40:17 smithi092 ceph-00db0016-dfed-11ee-95c8-87774f69a715-mgr-y[29801]: ::ffff:172.21.15.97 - - [11/Mar/2024:21:40:17] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-11T21:40:18.387 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:40:18.412 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:40:18.412 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:40:18.413 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:40:18.513 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:40:18.537 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:40:18.538 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:40:18.538 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:40:18.639 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:40:18.665 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:40:18.666 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:40:18.666 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:40:18.767 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:40:18.791 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:40:18.792 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:40:18.792 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:40:18.893 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:40:18.917 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:40:18.918 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:40:18.918 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:40:19.019 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:40:19.044 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:40:19.045 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:40:19.045 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:40:19.146 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:40:19.170 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:40:19.171 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:40:19.171 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:40:19.272 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:40:19.329 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:40:18 smithi097 ceph-mon[33317]: pgmap v684: 148 pgs: 2 active+recovering+undersized+degraded+remapped, 146 active+clean; 582 KiB data, 3.6 GiB used, 712 GiB / 715 GiB avail; 794 B/s rd, 0 op/s; 7443/594 objects degraded (1253.030%); 0 B/s, 300 objects/s recovering 2024-03-11T21:40:19.329 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:40:18 smithi097 ceph-mon[33317]: 2.17 scrub starts 2024-03-11T21:40:19.330 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:40:18 smithi097 ceph-mon[33317]: 2.17 scrub ok 2024-03-11T21:40:19.330 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:40:18 smithi097 ceph-mon[33317]: Health check update: Degraded data redundancy: 7443/594 objects degraded (1253.030%), 2 pgs degraded, 2 pgs undersized (PG_DEGRADED) 2024-03-11T21:40:19.363 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:40:18 smithi092 ceph-mon[29572]: pgmap v684: 148 pgs: 2 active+recovering+undersized+degraded+remapped, 146 active+clean; 582 KiB data, 3.6 GiB used, 712 GiB / 715 GiB avail; 794 B/s rd, 0 op/s; 7443/594 objects degraded (1253.030%); 0 B/s, 300 objects/s recovering 2024-03-11T21:40:19.364 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:40:18 smithi092 ceph-mon[29572]: 2.17 scrub starts 2024-03-11T21:40:19.364 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:40:18 smithi092 ceph-mon[29572]: 2.17 scrub ok 2024-03-11T21:40:19.364 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:40:18 smithi092 ceph-mon[29572]: Health check update: Degraded data redundancy: 7443/594 objects degraded (1253.030%), 2 pgs degraded, 2 pgs undersized (PG_DEGRADED) 2024-03-11T21:40:19.364 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:40:18 smithi092 ceph-mon[37563]: pgmap v684: 148 pgs: 2 active+recovering+undersized+degraded+remapped, 146 active+clean; 582 KiB data, 3.6 GiB used, 712 GiB / 715 GiB avail; 794 B/s rd, 0 op/s; 7443/594 objects degraded (1253.030%); 0 B/s, 300 objects/s recovering 2024-03-11T21:40:19.364 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:40:18 smithi092 ceph-mon[37563]: 2.17 scrub starts 2024-03-11T21:40:19.364 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:40:18 smithi092 ceph-mon[37563]: 2.17 scrub ok 2024-03-11T21:40:19.364 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:40:18 smithi092 ceph-mon[37563]: Health check update: Degraded data redundancy: 7443/594 objects degraded (1253.030%), 2 pgs degraded, 2 pgs undersized (PG_DEGRADED) 2024-03-11T21:40:19.495 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:40:19.495 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:40:19.496 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:40:19.597 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:40:19.622 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:40:19.622 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:40:19.623 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:40:19.723 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:40:19.748 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:40:19.748 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:40:19.749 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:40:19.849 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:40:19.873 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:40:19.874 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:40:19.874 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:40:19.975 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:40:19.999 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:40:20.000 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:40:20.000 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:40:20.063 DEBUG:teuthology.orchestra.run.smithi092:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid 00db0016-dfed-11ee-95c8-87774f69a715 -- ceph osd set noscrub 2024-03-11T21:40:20.101 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:40:20.125 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:40:20.126 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:40:20.127 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:40:20.203 INFO:teuthology.orchestra.run.smithi092.stderr:Inferring config /var/lib/ceph/00db0016-dfed-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T21:40:20.227 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:40:20.252 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:40:20.252 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:40:20.253 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:40:20.353 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:40:20.378 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:40:20.379 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:40:20.379 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:40:20.480 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:40:20.504 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:40:20.505 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:40:20.505 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:40:20.606 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:40:20.630 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:40:20.630 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:40:20.631 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:40:20.731 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:40:20.756 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:40:20.757 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:40:20.757 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:40:20.858 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:40:20.883 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:40:20.884 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:40:20.884 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:40:20.985 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:40:21.010 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:40:21.011 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:40:21.012 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:40:21.089 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:40:20 smithi092 ceph-mon[29572]: pgmap v685: 148 pgs: 2 active+recovering+undersized+degraded+remapped, 146 active+clean; 582 KiB data, 3.6 GiB used, 712 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s; 6053/594 objects degraded (1019.024%); 0 B/s, 415 objects/s recovering 2024-03-11T21:40:21.090 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:40:20 smithi092 ceph-mon[37563]: pgmap v685: 148 pgs: 2 active+recovering+undersized+degraded+remapped, 146 active+clean; 582 KiB data, 3.6 GiB used, 712 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s; 6053/594 objects degraded (1019.024%); 0 B/s, 415 objects/s recovering 2024-03-11T21:40:21.112 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:40:21.140 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:40:21.141 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:40:21.141 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:40:21.242 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:40:21.267 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:40:21.268 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:40:21.268 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:40:21.330 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:40:20 smithi097 ceph-mon[33317]: pgmap v685: 148 pgs: 2 active+recovering+undersized+degraded+remapped, 146 active+clean; 582 KiB data, 3.6 GiB used, 712 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s; 6053/594 objects degraded (1019.024%); 0 B/s, 415 objects/s recovering 2024-03-11T21:40:21.368 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:40:21.394 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:40:21.395 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:40:21.395 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:40:21.496 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:40:21.522 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:40:21.523 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:40:21.523 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:40:21.624 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:40:21.648 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:40:21.649 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:40:21.649 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:40:21.750 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:40:21.775 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:40:21.776 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:40:21.776 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:40:21.877 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:40:21.902 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:40:21.903 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:40:21.903 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:40:22.004 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:40:22.028 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:40:22.029 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:40:22.029 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:40:22.130 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:40:22.155 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:40:22.156 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:40:22.156 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:40:22.163 DEBUG:teuthology.orchestra.run.smithi092:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid 00db0016-dfed-11ee-95c8-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-03-11T21:40:22.195 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:40:21 smithi092 ceph-mon[29572]: 4.1a deep-scrub starts 2024-03-11T21:40:22.195 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:40:21 smithi092 ceph-mon[29572]: 4.1a deep-scrub ok 2024-03-11T21:40:22.195 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:40:21 smithi092 ceph-mon[37563]: 4.1a deep-scrub starts 2024-03-11T21:40:22.195 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:40:21 smithi092 ceph-mon[37563]: 4.1a deep-scrub ok 2024-03-11T21:40:22.217 INFO:teuthology.orchestra.run.smithi092.stdout:{ 2024-03-11T21:40:22.217 INFO:teuthology.orchestra.run.smithi092.stdout: "ops": [], 2024-03-11T21:40:22.217 INFO:teuthology.orchestra.run.smithi092.stdout: "complaint_time": 30, 2024-03-11T21:40:22.217 INFO:teuthology.orchestra.run.smithi092.stdout: "num_blocked_ops": 0 2024-03-11T21:40:22.217 INFO:teuthology.orchestra.run.smithi092.stdout:} 2024-03-11T21:40:22.258 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:40:22.283 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:40:22.283 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:40:22.284 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:40:22.329 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:40:21 smithi097 ceph-mon[33317]: 4.1a deep-scrub starts 2024-03-11T21:40:22.329 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:40:21 smithi097 ceph-mon[33317]: 4.1a deep-scrub ok 2024-03-11T21:40:22.385 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:40:22.415 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:40:22.416 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:40:22.416 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:40:22.517 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:40:22.542 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:40:22.543 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:40:22.543 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:40:22.644 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:40:22.671 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:40:22.672 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:40:22.672 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:40:22.773 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:40:22.797 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:40:22.798 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:40:22.798 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:40:22.899 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:40:22.923 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:40:22.923 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:40:22.923 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:40:23.024 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:40:23.048 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:40:23.049 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:40:23.049 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:40:23.079 INFO:journalctl@ceph.iscsi.iscsi.a.smithi097.stdout:Mar 11 21:40:22 smithi097 ceph-00db0016-dfed-11ee-95c8-87774f69a715-iscsi-iscsi-a[60943]: debug there is no tcmu-runner data available 2024-03-11T21:40:23.080 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:40:22 smithi097 ceph-mon[33317]: pgmap v686: 148 pgs: 2 active+recovering+undersized+degraded+remapped, 146 active+clean; 582 KiB data, 3.6 GiB used, 712 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s; 6053/594 objects degraded (1019.024%); 0 B/s, 352 objects/s recovering 2024-03-11T21:40:23.080 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:40:22 smithi097 ceph-mon[33317]: 5.1 deep-scrub starts 2024-03-11T21:40:23.080 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:40:22 smithi097 ceph-mon[33317]: 5.1 deep-scrub ok 2024-03-11T21:40:23.080 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:40:22 smithi097 ceph-mon[33317]: from='mgr.14607 v1:172.21.15.92:0/1702717597' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T21:40:23.150 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:40:23.174 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:40:23.175 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:40:23.175 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:40:23.266 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:40:22 smithi092 ceph-mon[29572]: pgmap v686: 148 pgs: 2 active+recovering+undersized+degraded+remapped, 146 active+clean; 582 KiB data, 3.6 GiB used, 712 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s; 6053/594 objects degraded (1019.024%); 0 B/s, 352 objects/s recovering 2024-03-11T21:40:23.267 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:40:22 smithi092 ceph-mon[29572]: 5.1 deep-scrub starts 2024-03-11T21:40:23.267 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:40:22 smithi092 ceph-mon[29572]: 5.1 deep-scrub ok 2024-03-11T21:40:23.267 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:40:22 smithi092 ceph-mon[29572]: from='mgr.14607 v1:172.21.15.92:0/1702717597' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T21:40:23.267 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:40:22 smithi092 ceph-mon[37563]: pgmap v686: 148 pgs: 2 active+recovering+undersized+degraded+remapped, 146 active+clean; 582 KiB data, 3.6 GiB used, 712 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s; 6053/594 objects degraded (1019.024%); 0 B/s, 352 objects/s recovering 2024-03-11T21:40:23.267 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:40:22 smithi092 ceph-mon[37563]: 5.1 deep-scrub starts 2024-03-11T21:40:23.267 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:40:22 smithi092 ceph-mon[37563]: 5.1 deep-scrub ok 2024-03-11T21:40:23.267 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:40:22 smithi092 ceph-mon[37563]: from='mgr.14607 v1:172.21.15.92:0/1702717597' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T21:40:23.277 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:40:23.301 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:40:23.302 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:40:23.302 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:40:23.403 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:40:23.427 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:40:23.428 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:40:23.428 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:40:23.516 DEBUG:teuthology.orchestra.run.smithi092:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid 00db0016-dfed-11ee-95c8-87774f69a715 -- ceph daemon osd.0 dump_historic_ops 2024-03-11T21:40:23.529 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:40:23.556 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:40:23.557 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:40:23.558 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:40:23.658 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:40:23.685 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:40:23.685 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:40:23.686 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:40:23.786 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:40:23.812 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:40:23.812 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:40:23.812 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:40:23.913 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:40:23.962 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:40:23.963 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:40:23.963 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:40:24.064 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:40:24.089 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:40:24.090 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:40:24.091 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:40:24.143 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:40:23 smithi092 ceph-mon[29572]: from='client.24469 v1:172.21.15.97:0/404208310' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:40:24.143 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:40:23 smithi092 ceph-mon[29572]: Health check update: Degraded data redundancy: 6053/594 objects degraded (1019.024%), 2 pgs degraded, 2 pgs undersized (PG_DEGRADED) 2024-03-11T21:40:24.143 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:40:23 smithi092 ceph-mon[37563]: from='client.24469 v1:172.21.15.97:0/404208310' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:40:24.143 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:40:23 smithi092 ceph-mon[37563]: Health check update: Degraded data redundancy: 6053/594 objects degraded (1019.024%), 2 pgs degraded, 2 pgs undersized (PG_DEGRADED) 2024-03-11T21:40:24.192 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:40:24.216 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:40:24.217 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:40:24.217 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:40:24.318 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:40:24.329 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:40:23 smithi097 ceph-mon[33317]: from='client.24469 v1:172.21.15.97:0/404208310' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:40:24.329 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:40:23 smithi097 ceph-mon[33317]: Health check update: Degraded data redundancy: 6053/594 objects degraded (1019.024%), 2 pgs degraded, 2 pgs undersized (PG_DEGRADED) 2024-03-11T21:40:24.343 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:40:24.343 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:40:24.343 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:40:24.444 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:40:24.469 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:40:24.470 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:40:24.470 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:40:24.493 INFO:teuthology.orchestra.run.smithi092.stderr:Inferring config /var/lib/ceph/00db0016-dfed-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T21:40:24.493 INFO:teuthology.orchestra.run.smithi092.stderr:Inferring config /var/lib/ceph/00db0016-dfed-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T21:40:24.497 INFO:teuthology.orchestra.run.smithi092.stderr:Inferring config /var/lib/ceph/00db0016-dfed-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T21:40:24.571 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:40:24.597 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:40:24.598 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:40:24.598 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:40:24.674 INFO:teuthology.orchestra.run.smithi092.stdout: 2024-03-11T21:40:24.674 INFO:teuthology.orchestra.run.smithi092.stdout:{"fsid":"00db0016-dfed-11ee-95c8-87774f69a715","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 4340/594 objects degraded (730.640%), 2 pgs degraded, 2 pgs undersized","count":4},"muted":false}},"mutes":[]},"election_epoch":14,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":1107,"monmap":{"epoch":3,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":350,"num_osds":8,"num_up_osds":8,"osd_up_since":1710192994,"num_in_osds":8,"osd_in_since":1710192308,"num_remapped_pgs":2},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":146},{"state_name":"active+recovering+undersized+degraded+remapped","count":2}],"num_pgs":148,"num_pools":7,"num_objects":198,"data_bytes":596392,"bytes_used":3812667392,"bytes_avail":764147617792,"bytes_total":767960285184,"degraded_objects":4340,"degraded_total":594,"degraded_ratio":7.3063973063973062,"recovering_objects_per_sec":390,"recovering_bytes_per_sec":0,"recovering_keys_per_sec":0,"num_objects_recovered":3905,"num_bytes_recovered":0,"num_keys_recovered":0,"read_bytes_sec":1023,"read_op_per_sec":0,"write_op_per_sec":0},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":1,"modules":["cephadm","dashboard","iostat","nfs","prometheus","restful"],"services":{"dashboard":"https://172.21.15.92:8443/","prometheus":"http://172.21.15.92:9283/"}},"servicemap":{"epoch":15,"modified":"2024-03-11T21:37:33.231049+0000","services":{"rgw":{"daemons":{"summary":"","34352":{"start_epoch":12,"start_stamp":"2024-03-11T21:25:45.672905+0000","gid":34352,"addr":"172.21.15.92:0/372197479","metadata":{"arch":"x86_64","ceph_release":"squid","ceph_version":"ceph version 19.0.0-1979-g2b0101eb (2b0101eb74d60610d544cb8a473e23a35ac1cc0b) squid (dev)","ceph_version_short":"19.0.0-1979-g2b0101eb","container_hostname":"smithi092","container_image":"quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:8fe9635f3beb0247496d7e321f3068bd10c333c433ef4824d7e1c1a7d3d78379","cpu":"Intel(R) Xeon(R) CPU E5-1620 v4 @ 3.50GHz","distro":"centos","distro_description":"CentOS Stream 8","distro_version":"8","frontend_config#0":"beast port=80","frontend_type#0":"beast","hostname":"smithi092","id":"foo.a","kernel_description":"#1 SMP PREEMPT_DYNAMIC Fri Feb 23 04:45:07 UTC 2024","kernel_version":"5.14.0-427.el9.x86_64","mem_swap_kb":"0","mem_total_kb":"32488440","num_handles":"1","os":"Linux","pid":"2","realm_id":"","realm_name":"","zone_id":"bf7aa506-b210-476a-b68a-1feac2caf199","zone_name":"default","zonegroup_id":"33e24078-41e3-4494-bd9e-6ff3741afaba","zonegroup_name":"default"},"task_status":{}}}}}},"progress_events":{}} 2024-03-11T21:40:24.699 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:40:24.724 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:40:24.725 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:40:24.725 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:40:24.826 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:40:24.850 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:40:24.851 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:40:24.851 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:40:24.911 INFO:teuthology.orchestra.run.smithi092.stderr:pool 'unique_pool_2' removed 2024-03-11T21:40:24.927 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:40:24 smithi092 ceph-mon[29572]: pgmap v687: 148 pgs: 2 active+recovering+undersized+degraded+remapped, 146 active+clean; 582 KiB data, 3.6 GiB used, 712 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s; 4340/594 objects degraded (730.640%); 0 B/s, 390 objects/s recovering 2024-03-11T21:40:24.928 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:40:24 smithi092 ceph-mon[29572]: from='client.? v1:172.21.15.92:0/114675945' entity='client.admin' cmd=[{"prefix": "status", "format": "json"}]: dispatch 2024-03-11T21:40:24.928 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:40:24 smithi092 ceph-mon[29572]: from='client.? v1:172.21.15.92:0/1181248713' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "unique_pool_2", "pool2": "unique_pool_2", "yes_i_really_really_mean_it": true}]: dispatch 2024-03-11T21:40:24.928 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:40:24 smithi092 ceph-mon[29572]: from='client.35537 ' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "unique_pool_2", "pool2": "unique_pool_2", "yes_i_really_really_mean_it": true}]: dispatch 2024-03-11T21:40:24.928 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:40:24 smithi092 ceph-mon[37563]: pgmap v687: 148 pgs: 2 active+recovering+undersized+degraded+remapped, 146 active+clean; 582 KiB data, 3.6 GiB used, 712 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s; 4340/594 objects degraded (730.640%); 0 B/s, 390 objects/s recovering 2024-03-11T21:40:24.928 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:40:24 smithi092 ceph-mon[37563]: from='client.? v1:172.21.15.92:0/114675945' entity='client.admin' cmd=[{"prefix": "status", "format": "json"}]: dispatch 2024-03-11T21:40:24.928 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:40:24 smithi092 ceph-mon[37563]: from='client.? v1:172.21.15.92:0/1181248713' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "unique_pool_2", "pool2": "unique_pool_2", "yes_i_really_really_mean_it": true}]: dispatch 2024-03-11T21:40:24.929 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:40:24 smithi092 ceph-mon[37563]: from='client.35537 ' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "unique_pool_2", "pool2": "unique_pool_2", "yes_i_really_really_mean_it": true}]: dispatch 2024-03-11T21:40:24.952 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:40:24.993 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:40:24.993 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:40:24.994 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:40:25.095 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:40:25.121 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:40:25.122 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:40:25.122 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:40:25.223 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:40:25.246 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:40:25.247 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:40:25.248 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:40:25.329 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:40:24 smithi097 ceph-mon[33317]: pgmap v687: 148 pgs: 2 active+recovering+undersized+degraded+remapped, 146 active+clean; 582 KiB data, 3.6 GiB used, 712 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s; 4340/594 objects degraded (730.640%); 0 B/s, 390 objects/s recovering 2024-03-11T21:40:25.330 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:40:24 smithi097 ceph-mon[33317]: from='client.? v1:172.21.15.92:0/114675945' entity='client.admin' cmd=[{"prefix": "status", "format": "json"}]: dispatch 2024-03-11T21:40:25.330 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:40:24 smithi097 ceph-mon[33317]: from='client.? v1:172.21.15.92:0/1181248713' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "unique_pool_2", "pool2": "unique_pool_2", "yes_i_really_really_mean_it": true}]: dispatch 2024-03-11T21:40:25.330 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:40:24 smithi097 ceph-mon[33317]: from='client.35537 ' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "unique_pool_2", "pool2": "unique_pool_2", "yes_i_really_really_mean_it": true}]: dispatch 2024-03-11T21:40:25.348 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:40:25.373 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:40:25.374 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:40:25.374 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:40:25.475 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:40:25.499 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:40:25.500 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:40:25.500 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:40:25.600 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:40:25.626 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:40:25.626 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:40:25.627 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:40:25.639 INFO:tasks.cephadm.ceph_manager.ceph:making progress, resetting timeout 2024-03-11T21:40:25.639 DEBUG:teuthology.orchestra.run.smithi092:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid 00db0016-dfed-11ee-95c8-87774f69a715 -- ceph pg dump --format=json 2024-03-11T21:40:25.727 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:40:25.752 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:40:25.753 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:40:25.753 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:40:25.854 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:40:25.878 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:40:25.878 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:40:25.879 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:40:25.981 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:40:26.005 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:40:26.005 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:40:26.005 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:40:26.106 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:40:26.131 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:40:26.132 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:40:26.132 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:40:26.233 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:40:26.260 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:40:26.261 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:40:26.261 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:40:26.330 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:40:25 smithi097 ceph-mon[33317]: from='client.35537 ' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "unique_pool_2", "pool2": "unique_pool_2", "yes_i_really_really_mean_it": true}]': finished 2024-03-11T21:40:26.330 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:40:25 smithi097 ceph-mon[33317]: osdmap e351: 8 total, 8 up, 8 in 2024-03-11T21:40:26.330 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:40:25 smithi097 ceph-mon[33317]: 4.1f scrub starts 2024-03-11T21:40:26.330 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:40:25 smithi097 ceph-mon[33317]: 4.1f scrub ok 2024-03-11T21:40:26.330 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:40:25 smithi097 ceph-mon[33317]: 6.3 scrub starts 2024-03-11T21:40:26.330 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:40:25 smithi097 ceph-mon[33317]: 6.3 scrub ok 2024-03-11T21:40:26.362 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:40:26.364 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:40:25 smithi092 ceph-mon[29572]: from='client.35537 ' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "unique_pool_2", "pool2": "unique_pool_2", "yes_i_really_really_mean_it": true}]': finished 2024-03-11T21:40:26.364 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:40:25 smithi092 ceph-mon[29572]: osdmap e351: 8 total, 8 up, 8 in 2024-03-11T21:40:26.364 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:40:25 smithi092 ceph-mon[29572]: 4.1f scrub starts 2024-03-11T21:40:26.364 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:40:25 smithi092 ceph-mon[29572]: 4.1f scrub ok 2024-03-11T21:40:26.364 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:40:25 smithi092 ceph-mon[29572]: 6.3 scrub starts 2024-03-11T21:40:26.364 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:40:25 smithi092 ceph-mon[29572]: 6.3 scrub ok 2024-03-11T21:40:26.364 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:40:25 smithi092 ceph-mon[37563]: from='client.35537 ' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "unique_pool_2", "pool2": "unique_pool_2", "yes_i_really_really_mean_it": true}]': finished 2024-03-11T21:40:26.364 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:40:25 smithi092 ceph-mon[37563]: osdmap e351: 8 total, 8 up, 8 in 2024-03-11T21:40:26.364 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:40:25 smithi092 ceph-mon[37563]: 4.1f scrub starts 2024-03-11T21:40:26.364 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:40:25 smithi092 ceph-mon[37563]: 4.1f scrub ok 2024-03-11T21:40:26.365 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:40:25 smithi092 ceph-mon[37563]: 6.3 scrub starts 2024-03-11T21:40:26.365 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:40:25 smithi092 ceph-mon[37563]: 6.3 scrub ok 2024-03-11T21:40:26.390 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:40:26.390 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:40:26.390 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:40:26.491 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:40:26.515 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:40:26.516 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:40:26.516 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:40:26.617 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:40:26.642 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:40:26.642 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:40:26.643 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:40:26.743 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:40:26.768 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:40:26.768 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:40:26.768 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:40:26.869 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:40:26.893 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:40:26.893 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:40:26.894 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:40:26.994 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:40:27.020 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:40:27.021 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:40:27.021 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:40:27.105 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:40:26 smithi092 ceph-mon[29572]: pgmap v689: 132 pgs: 132 active+clean; 582 KiB data, 3.5 GiB used, 712 GiB / 715 GiB avail; 921 B/s rd, 0 op/s 2024-03-11T21:40:27.105 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:40:26 smithi092 ceph-mon[29572]: 4.13 deep-scrub starts 2024-03-11T21:40:27.106 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:40:26 smithi092 ceph-mon[29572]: 4.13 deep-scrub ok 2024-03-11T21:40:27.106 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:40:26 smithi092 ceph-mon[29572]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 4340/594 objects degraded (730.640%), 2 pgs degraded, 2 pgs undersized) 2024-03-11T21:40:27.106 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:40:26 smithi092 ceph-mon[29572]: Cluster is now healthy 2024-03-11T21:40:27.106 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:40:26 smithi092 ceph-mon[37563]: pgmap v689: 132 pgs: 132 active+clean; 582 KiB data, 3.5 GiB used, 712 GiB / 715 GiB avail; 921 B/s rd, 0 op/s 2024-03-11T21:40:27.106 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:40:26 smithi092 ceph-mon[37563]: 4.13 deep-scrub starts 2024-03-11T21:40:27.106 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:40:26 smithi092 ceph-mon[37563]: 4.13 deep-scrub ok 2024-03-11T21:40:27.106 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:40:26 smithi092 ceph-mon[37563]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 4340/594 objects degraded (730.640%), 2 pgs degraded, 2 pgs undersized) 2024-03-11T21:40:27.106 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:40:26 smithi092 ceph-mon[37563]: Cluster is now healthy 2024-03-11T21:40:27.123 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:40:27.146 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:40:27.147 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:40:27.147 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:40:27.248 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:40:27.275 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:40:27.275 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:40:27.276 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:40:27.330 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:40:26 smithi097 ceph-mon[33317]: pgmap v689: 132 pgs: 132 active+clean; 582 KiB data, 3.5 GiB used, 712 GiB / 715 GiB avail; 921 B/s rd, 0 op/s 2024-03-11T21:40:27.330 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:40:26 smithi097 ceph-mon[33317]: 4.13 deep-scrub starts 2024-03-11T21:40:27.330 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:40:26 smithi097 ceph-mon[33317]: 4.13 deep-scrub ok 2024-03-11T21:40:27.330 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:40:26 smithi097 ceph-mon[33317]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 4340/594 objects degraded (730.640%), 2 pgs degraded, 2 pgs undersized) 2024-03-11T21:40:27.330 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:40:26 smithi097 ceph-mon[33317]: Cluster is now healthy 2024-03-11T21:40:27.377 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:40:27.404 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:40:27.404 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:40:27.404 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:40:27.505 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:40:27.529 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:40:27.529 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:40:27.530 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:40:27.631 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:40:27.656 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:40:27.656 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:40:27.657 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:40:27.758 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:40:27.783 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:40:27.784 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:40:27.784 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:40:27.885 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:40:27.909 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:40:27.909 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:40:27.909 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:40:28.010 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:40:28.038 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:40:28.039 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:40:28.039 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:40:28.101 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:40:27 smithi092 ceph-mon[29572]: 5.16 scrub starts 2024-03-11T21:40:28.101 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:40:27 smithi092 ceph-mon[29572]: 5.16 scrub ok 2024-03-11T21:40:28.101 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:40:27 smithi092 ceph-mon[29572]: 2.0 scrub starts 2024-03-11T21:40:28.101 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:40:27 smithi092 ceph-mon[29572]: 2.0 scrub ok 2024-03-11T21:40:28.101 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:40:27 smithi092 ceph-mon[29572]: 6.0 scrub starts 2024-03-11T21:40:28.101 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:40:27 smithi092 ceph-mon[29572]: 6.0 scrub ok 2024-03-11T21:40:28.101 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:40:27 smithi092 ceph-mon[29572]: 6.b scrub starts 2024-03-11T21:40:28.102 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:40:27 smithi092 ceph-mon[37563]: 5.16 scrub starts 2024-03-11T21:40:28.102 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:40:27 smithi092 ceph-mon[37563]: 5.16 scrub ok 2024-03-11T21:40:28.102 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:40:27 smithi092 ceph-mon[37563]: 2.0 scrub starts 2024-03-11T21:40:28.102 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:40:27 smithi092 ceph-mon[37563]: 2.0 scrub ok 2024-03-11T21:40:28.102 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:40:27 smithi092 ceph-mon[37563]: 6.0 scrub starts 2024-03-11T21:40:28.102 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:40:27 smithi092 ceph-mon[37563]: 6.0 scrub ok 2024-03-11T21:40:28.102 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:40:27 smithi092 ceph-mon[37563]: 6.b scrub starts 2024-03-11T21:40:28.102 INFO:journalctl@ceph.mgr.y.smithi092.stdout:Mar 11 21:40:27 smithi092 ceph-00db0016-dfed-11ee-95c8-87774f69a715-mgr-y[29801]: ::ffff:172.21.15.97 - - [11/Mar/2024:21:40:27] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-11T21:40:28.140 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:40:28.168 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:40:28.169 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:40:28.169 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:40:28.270 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:40:28.295 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:40:28.296 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:40:28.296 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:40:28.329 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:40:27 smithi097 ceph-mon[33317]: 5.16 scrub starts 2024-03-11T21:40:28.330 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:40:27 smithi097 ceph-mon[33317]: 5.16 scrub ok 2024-03-11T21:40:28.330 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:40:27 smithi097 ceph-mon[33317]: 2.0 scrub starts 2024-03-11T21:40:28.330 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:40:27 smithi097 ceph-mon[33317]: 2.0 scrub ok 2024-03-11T21:40:28.330 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:40:27 smithi097 ceph-mon[33317]: 6.0 scrub starts 2024-03-11T21:40:28.330 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:40:27 smithi097 ceph-mon[33317]: 6.0 scrub ok 2024-03-11T21:40:28.330 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:40:27 smithi097 ceph-mon[33317]: 6.b scrub starts 2024-03-11T21:40:28.397 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:40:28.423 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:40:28.424 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:40:28.424 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:40:28.525 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:40:28.553 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:40:28.553 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:40:28.554 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:40:28.655 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:40:28.682 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:40:28.683 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:40:28.683 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:40:28.722 INFO:teuthology.task.full_sequential:In full_sequential, running task radosbench... 2024-03-11T21:40:28.723 INFO:tasks.radosbench:Beginning radosbench... 2024-03-11T21:40:28.723 INFO:tasks.cephadm.ceph_manager.ceph:creating pool_name unique_pool_3 2024-03-11T21:40:28.724 DEBUG:teuthology.orchestra.run.smithi092:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid 00db0016-dfed-11ee-95c8-87774f69a715 -- ceph osd pool create unique_pool_3 16 2024-03-11T21:40:28.784 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:40:28.812 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:40:28.813 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:40:28.813 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:40:28.914 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:40:28.941 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:40:28.942 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:40:28.943 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:40:29.043 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:40:29.071 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:40:29.072 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:40:29.072 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:40:29.114 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:40:28 smithi092 ceph-mon[29572]: pgmap v690: 132 pgs: 132 active+clean; 582 KiB data, 3.5 GiB used, 712 GiB / 715 GiB avail; 921 B/s rd, 0 op/s 2024-03-11T21:40:29.114 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:40:28 smithi092 ceph-mon[29572]: 5.1b scrub starts 2024-03-11T21:40:29.114 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:40:28 smithi092 ceph-mon[29572]: 5.1b scrub ok 2024-03-11T21:40:29.114 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:40:28 smithi092 ceph-mon[29572]: 6.b scrub ok 2024-03-11T21:40:29.114 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:40:28 smithi092 ceph-mon[29572]: 5.a scrub starts 2024-03-11T21:40:29.114 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:40:28 smithi092 ceph-mon[29572]: 5.a scrub ok 2024-03-11T21:40:29.114 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:40:28 smithi092 ceph-mon[29572]: 5.1e scrub starts 2024-03-11T21:40:29.114 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:40:28 smithi092 ceph-mon[29572]: 5.1e scrub ok 2024-03-11T21:40:29.114 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:40:28 smithi092 ceph-mon[29572]: osdmap e352: 8 total, 8 up, 8 in 2024-03-11T21:40:29.115 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:40:28 smithi092 ceph-mon[37563]: pgmap v690: 132 pgs: 132 active+clean; 582 KiB data, 3.5 GiB used, 712 GiB / 715 GiB avail; 921 B/s rd, 0 op/s 2024-03-11T21:40:29.115 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:40:28 smithi092 ceph-mon[37563]: 5.1b scrub starts 2024-03-11T21:40:29.115 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:40:28 smithi092 ceph-mon[37563]: 5.1b scrub ok 2024-03-11T21:40:29.115 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:40:28 smithi092 ceph-mon[37563]: 6.b scrub ok 2024-03-11T21:40:29.115 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:40:28 smithi092 ceph-mon[37563]: 5.a scrub starts 2024-03-11T21:40:29.115 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:40:28 smithi092 ceph-mon[37563]: 5.a scrub ok 2024-03-11T21:40:29.115 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:40:28 smithi092 ceph-mon[37563]: 5.1e scrub starts 2024-03-11T21:40:29.115 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:40:28 smithi092 ceph-mon[37563]: 5.1e scrub ok 2024-03-11T21:40:29.115 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:40:28 smithi092 ceph-mon[37563]: osdmap e352: 8 total, 8 up, 8 in 2024-03-11T21:40:29.173 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:40:29.198 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:40:29.198 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:40:29.199 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:40:29.300 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:40:29.327 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:40:29.328 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:40:29.329 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:40:29.329 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:40:28 smithi097 ceph-mon[33317]: pgmap v690: 132 pgs: 132 active+clean; 582 KiB data, 3.5 GiB used, 712 GiB / 715 GiB avail; 921 B/s rd, 0 op/s 2024-03-11T21:40:29.330 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:40:28 smithi097 ceph-mon[33317]: 5.1b scrub starts 2024-03-11T21:40:29.330 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:40:28 smithi097 ceph-mon[33317]: 5.1b scrub ok 2024-03-11T21:40:29.330 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:40:28 smithi097 ceph-mon[33317]: 6.b scrub ok 2024-03-11T21:40:29.330 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:40:28 smithi097 ceph-mon[33317]: 5.a scrub starts 2024-03-11T21:40:29.330 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:40:28 smithi097 ceph-mon[33317]: 5.a scrub ok 2024-03-11T21:40:29.330 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:40:28 smithi097 ceph-mon[33317]: 5.1e scrub starts 2024-03-11T21:40:29.330 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:40:28 smithi097 ceph-mon[33317]: 5.1e scrub ok 2024-03-11T21:40:29.330 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:40:28 smithi097 ceph-mon[33317]: osdmap e352: 8 total, 8 up, 8 in 2024-03-11T21:40:29.430 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:40:29.457 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:40:29.458 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:40:29.459 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:40:29.543 INFO:teuthology.orchestra.run.smithi092.stderr:Inferring config /var/lib/ceph/00db0016-dfed-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T21:40:29.559 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:40:29.590 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:40:29.591 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:40:29.591 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:40:29.692 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:40:29.720 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:40:29.721 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:40:29.722 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:40:29.823 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:40:29.851 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:40:29.852 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:40:29.852 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:40:29.953 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:40:29.981 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:40:29.982 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:40:29.982 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:40:30.005 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:40:29 smithi092 ceph-mon[29572]: 6.2 scrub starts 2024-03-11T21:40:30.005 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:40:29 smithi092 ceph-mon[29572]: 6.2 scrub ok 2024-03-11T21:40:30.006 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:40:29 smithi092 ceph-mon[29572]: 5.10 deep-scrub starts 2024-03-11T21:40:30.006 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:40:29 smithi092 ceph-mon[29572]: 5.10 deep-scrub ok 2024-03-11T21:40:30.006 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:40:29 smithi092 ceph-mon[37563]: 6.2 scrub starts 2024-03-11T21:40:30.006 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:40:29 smithi092 ceph-mon[37563]: 6.2 scrub ok 2024-03-11T21:40:30.007 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:40:29 smithi092 ceph-mon[37563]: 5.10 deep-scrub starts 2024-03-11T21:40:30.007 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:40:29 smithi092 ceph-mon[37563]: 5.10 deep-scrub ok 2024-03-11T21:40:30.083 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:40:30.112 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:40:30.112 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:40:30.113 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:40:30.214 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:40:30.240 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:40:30.241 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:40:30.241 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:40:30.330 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:40:29 smithi097 ceph-mon[33317]: 6.2 scrub starts 2024-03-11T21:40:30.330 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:40:29 smithi097 ceph-mon[33317]: 6.2 scrub ok 2024-03-11T21:40:30.330 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:40:29 smithi097 ceph-mon[33317]: 5.10 deep-scrub starts 2024-03-11T21:40:30.330 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:40:29 smithi097 ceph-mon[33317]: 5.10 deep-scrub ok 2024-03-11T21:40:30.343 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:40:30.371 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:40:30.372 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:40:30.373 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:40:30.444 INFO:teuthology.orchestra.run.smithi092.stdout:{ 2024-03-11T21:40:30.444 INFO:teuthology.orchestra.run.smithi092.stdout: "size": 20, 2024-03-11T21:40:30.444 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 600, 2024-03-11T21:40:30.444 INFO:teuthology.orchestra.run.smithi092.stdout: "ops": [ 2024-03-11T21:40:30.444 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:40:30.444 INFO:teuthology.orchestra.run.smithi092.stdout: "description": "osd_op(client.15150.0:207121 8.c 8:3b538f8b:::benchmark_data_smithi092_104865_object86620:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e237)", 2024-03-11T21:40:30.445 INFO:teuthology.orchestra.run.smithi092.stdout: "initiated_at": "2024-03-11T21:34:55.373083+0000", 2024-03-11T21:40:30.445 INFO:teuthology.orchestra.run.smithi092.stdout: "age": 335.08302544700001, 2024-03-11T21:40:30.445 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.98710652499999996, 2024-03-11T21:40:30.445 INFO:teuthology.orchestra.run.smithi092.stdout: "type_data": { 2024-03-11T21:40:30.445 INFO:teuthology.orchestra.run.smithi092.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-11T21:40:30.445 INFO:teuthology.orchestra.run.smithi092.stdout: "client_info": { 2024-03-11T21:40:30.445 INFO:teuthology.orchestra.run.smithi092.stdout: "client": "client.15150", 2024-03-11T21:40:30.445 INFO:teuthology.orchestra.run.smithi092.stdout: "client_addr": "v1:172.21.15.92:0/436606717", 2024-03-11T21:40:30.445 INFO:teuthology.orchestra.run.smithi092.stdout: "tid": 207121 2024-03-11T21:40:30.445 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:40:30.445 INFO:teuthology.orchestra.run.smithi092.stdout: "events": [ 2024-03-11T21:40:30.445 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:40:30.445 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "initiated", 2024-03-11T21:40:30.446 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:34:55.373083+0000", 2024-03-11T21:40:30.446 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-03-11T21:40:30.446 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:40:30.446 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:40:30.446 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "header_read", 2024-03-11T21:40:30.446 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:34:55.373083+0000", 2024-03-11T21:40:30.446 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-03-11T21:40:30.446 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:40:30.447 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:40:30.447 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "throttled", 2024-03-11T21:40:30.447 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:34:55.373085+0000", 2024-03-11T21:40:30.447 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.206e-06 2024-03-11T21:40:30.447 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:40:30.447 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:40:30.447 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "all_read", 2024-03-11T21:40:30.447 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:34:55.373090+0000", 2024-03-11T21:40:30.448 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 5.6760000000000004e-06 2024-03-11T21:40:30.448 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:40:30.448 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:40:30.448 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "dispatched", 2024-03-11T21:40:30.448 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:34:55.373107+0000", 2024-03-11T21:40:30.448 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.6362999999999999e-05 2024-03-11T21:40:30.448 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:40:30.448 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:40:30.448 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "queued_for_pg", 2024-03-11T21:40:30.448 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:34:55.373115+0000", 2024-03-11T21:40:30.448 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 8.2660000000000001e-06 2024-03-11T21:40:30.448 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:40:30.449 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:40:30.449 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-03-11T21:40:30.449 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:34:55.373173+0000", 2024-03-11T21:40:30.449 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 5.7991000000000001e-05 2024-03-11T21:40:30.449 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:40:30.449 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:40:30.449 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "waiting for peered", 2024-03-11T21:40:30.449 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:34:55.373182+0000", 2024-03-11T21:40:30.449 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 9.0049999999999993e-06 2024-03-11T21:40:30.449 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:40:30.449 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:40:30.449 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-03-11T21:40:30.449 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:34:56.354869+0000", 2024-03-11T21:40:30.450 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.98168716199999995 2024-03-11T21:40:30.450 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:40:30.450 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:40:30.450 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "started", 2024-03-11T21:40:30.450 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:34:56.355005+0000", 2024-03-11T21:40:30.450 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00013610799999999999 2024-03-11T21:40:30.450 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:40:30.450 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:40:30.450 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "waiting for subops from 3,7", 2024-03-11T21:40:30.450 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:34:56.355119+0000", 2024-03-11T21:40:30.450 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00011411 2024-03-11T21:40:30.450 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:40:30.451 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:40:30.451 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "sub_op_commit_rec", 2024-03-11T21:40:30.451 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:34:56.357333+0000", 2024-03-11T21:40:30.451 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.0022135430000000001 2024-03-11T21:40:30.451 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:40:30.451 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:40:30.451 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "op_commit", 2024-03-11T21:40:30.451 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:34:56.358863+0000", 2024-03-11T21:40:30.451 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.001530399 2024-03-11T21:40:30.451 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:40:30.451 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:40:30.451 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "sub_op_commit_rec", 2024-03-11T21:40:30.451 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:34:56.360128+0000", 2024-03-11T21:40:30.451 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.001264409 2024-03-11T21:40:30.452 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:40:30.452 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:40:30.452 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "commit_sent", 2024-03-11T21:40:30.452 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:34:56.360162+0000", 2024-03-11T21:40:30.452 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 3.4585999999999997e-05 2024-03-11T21:40:30.452 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:40:30.452 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:40:30.452 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "done", 2024-03-11T21:40:30.452 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:34:56.360190+0000", 2024-03-11T21:40:30.452 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 2.7701000000000001e-05 2024-03-11T21:40:30.452 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-03-11T21:40:30.452 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-03-11T21:40:30.452 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-03-11T21:40:30.453 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:40:30.453 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:40:30.453 INFO:teuthology.orchestra.run.smithi092.stdout: "description": "osd_op(client.15150.0:207122 8.c 8:3cecc2cc:::benchmark_data_smithi092_104865_object86621:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e237)", 2024-03-11T21:40:30.453 INFO:teuthology.orchestra.run.smithi092.stdout: "initiated_at": "2024-03-11T21:34:55.374081+0000", 2024-03-11T21:40:30.453 INFO:teuthology.orchestra.run.smithi092.stdout: "age": 335.08202804000001, 2024-03-11T21:40:30.453 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.98636186100000001, 2024-03-11T21:40:30.453 INFO:teuthology.orchestra.run.smithi092.stdout: "type_data": { 2024-03-11T21:40:30.453 INFO:teuthology.orchestra.run.smithi092.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-11T21:40:30.453 INFO:teuthology.orchestra.run.smithi092.stdout: "client_info": { 2024-03-11T21:40:30.453 INFO:teuthology.orchestra.run.smithi092.stdout: "client": "client.15150", 2024-03-11T21:40:30.453 INFO:teuthology.orchestra.run.smithi092.stdout: "client_addr": "v1:172.21.15.92:0/436606717", 2024-03-11T21:40:30.454 INFO:teuthology.orchestra.run.smithi092.stdout: "tid": 207122 2024-03-11T21:40:30.454 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:40:30.454 INFO:teuthology.orchestra.run.smithi092.stdout: "events": [ 2024-03-11T21:40:30.454 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:40:30.454 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "initiated", 2024-03-11T21:40:30.454 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:34:55.374081+0000", 2024-03-11T21:40:30.454 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-03-11T21:40:30.454 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:40:30.454 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:40:30.454 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "header_read", 2024-03-11T21:40:30.454 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:34:55.374081+0000", 2024-03-11T21:40:30.454 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-03-11T21:40:30.455 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:40:30.455 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:40:30.455 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "throttled", 2024-03-11T21:40:30.455 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:34:55.374082+0000", 2024-03-11T21:40:30.455 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.6020000000000001e-06 2024-03-11T21:40:30.455 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:40:30.455 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:40:30.455 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "all_read", 2024-03-11T21:40:30.455 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:34:55.374088+0000", 2024-03-11T21:40:30.456 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 5.4360000000000001e-06 2024-03-11T21:40:30.456 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:40:30.456 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:40:30.456 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "dispatched", 2024-03-11T21:40:30.456 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:34:55.374102+0000", 2024-03-11T21:40:30.456 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.4222e-05 2024-03-11T21:40:30.457 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:40:30.457 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:40:30.457 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "queued_for_pg", 2024-03-11T21:40:30.457 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:34:55.374110+0000", 2024-03-11T21:40:30.457 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 8.1200000000000002e-06 2024-03-11T21:40:30.457 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:40:30.458 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:40:30.458 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-03-11T21:40:30.458 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:34:55.374167+0000", 2024-03-11T21:40:30.458 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 5.6768999999999998e-05 2024-03-11T21:40:30.458 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:40:30.458 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:40:30.459 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "waiting for peered", 2024-03-11T21:40:30.459 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:34:55.374178+0000", 2024-03-11T21:40:30.459 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.0730000000000001e-05 2024-03-11T21:40:30.459 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:40:30.459 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:40:30.459 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-03-11T21:40:30.460 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:34:56.357391+0000", 2024-03-11T21:40:30.460 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.98321307000000002 2024-03-11T21:40:30.460 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:40:30.460 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:40:30.460 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "started", 2024-03-11T21:40:30.460 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:34:56.357504+0000", 2024-03-11T21:40:30.460 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.000113727 2024-03-11T21:40:30.461 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:40:30.461 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:40:30.461 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "waiting for subops from 3,7", 2024-03-11T21:40:30.461 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:34:56.357573+0000", 2024-03-11T21:40:30.461 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 6.8965000000000001e-05 2024-03-11T21:40:30.461 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:40:30.462 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:40:30.462 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "sub_op_commit_rec", 2024-03-11T21:40:30.462 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:34:56.360278+0000", 2024-03-11T21:40:30.462 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.0027041729999999998 2024-03-11T21:40:30.462 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:40:30.462 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:40:30.463 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "op_commit", 2024-03-11T21:40:30.463 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:34:56.360289+0000", 2024-03-11T21:40:30.463 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.1141000000000001e-05 2024-03-11T21:40:30.463 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:40:30.463 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:40:30.463 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "sub_op_commit_rec", 2024-03-11T21:40:30.464 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:34:56.360324+0000", 2024-03-11T21:40:30.464 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 3.5330999999999998e-05 2024-03-11T21:40:30.464 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:40:30.464 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:40:30.464 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "commit_sent", 2024-03-11T21:40:30.464 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:34:56.360404+0000", 2024-03-11T21:40:30.464 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 8.0290999999999994e-05 2024-03-11T21:40:30.465 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:40:30.465 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:40:30.465 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "done", 2024-03-11T21:40:30.467 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:34:56.360443+0000", 2024-03-11T21:40:30.467 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 3.8284000000000001e-05 2024-03-11T21:40:30.467 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-03-11T21:40:30.467 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-03-11T21:40:30.467 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-03-11T21:40:30.467 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:40:30.468 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:40:30.468 INFO:teuthology.orchestra.run.smithi092.stdout: "description": "osd_op(client.15150.0:207130 8.c 8:3e52addc:::benchmark_data_smithi092_104865_object86629:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e237)", 2024-03-11T21:40:30.468 INFO:teuthology.orchestra.run.smithi092.stdout: "initiated_at": "2024-03-11T21:34:55.382721+0000", 2024-03-11T21:40:30.468 INFO:teuthology.orchestra.run.smithi092.stdout: "age": 335.07338750899999, 2024-03-11T21:40:30.468 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.97792294199999996, 2024-03-11T21:40:30.468 INFO:teuthology.orchestra.run.smithi092.stdout: "type_data": { 2024-03-11T21:40:30.469 INFO:teuthology.orchestra.run.smithi092.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-11T21:40:30.469 INFO:teuthology.orchestra.run.smithi092.stdout: "client_info": { 2024-03-11T21:40:30.469 INFO:teuthology.orchestra.run.smithi092.stdout: "client": "client.15150", 2024-03-11T21:40:30.469 INFO:teuthology.orchestra.run.smithi092.stdout: "client_addr": "v1:172.21.15.92:0/436606717", 2024-03-11T21:40:30.469 INFO:teuthology.orchestra.run.smithi092.stdout: "tid": 207130 2024-03-11T21:40:30.469 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:40:30.470 INFO:teuthology.orchestra.run.smithi092.stdout: "events": [ 2024-03-11T21:40:30.470 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:40:30.470 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "initiated", 2024-03-11T21:40:30.470 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:34:55.382721+0000", 2024-03-11T21:40:30.470 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-03-11T21:40:30.470 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:40:30.471 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:40:30.471 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "header_read", 2024-03-11T21:40:30.471 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:34:55.382721+0000", 2024-03-11T21:40:30.471 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-03-11T21:40:30.471 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:40:30.472 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:40:30.472 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "throttled", 2024-03-11T21:40:30.472 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:34:55.382723+0000", 2024-03-11T21:40:30.472 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 2.1160000000000002e-06 2024-03-11T21:40:30.472 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:40:30.472 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:40:30.473 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "all_read", 2024-03-11T21:40:30.473 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:34:55.382729+0000", 2024-03-11T21:40:30.473 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 5.9240000000000004e-06 2024-03-11T21:40:30.473 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:40:30.473 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:40:30.473 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "dispatched", 2024-03-11T21:40:30.474 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:34:55.382742+0000", 2024-03-11T21:40:30.474 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.3161e-05 2024-03-11T21:40:30.474 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:40:30.474 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:40:30.474 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "queued_for_pg", 2024-03-11T21:40:30.475 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:34:55.382749+0000", 2024-03-11T21:40:30.475 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 6.9689999999999997e-06 2024-03-11T21:40:30.475 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:40:30.475 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:40:30.477 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-03-11T21:40:30.477 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:34:55.382797+0000", 2024-03-11T21:40:30.477 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 4.7277999999999997e-05 2024-03-11T21:40:30.477 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:40:30.478 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:40:30.478 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "waiting for peered", 2024-03-11T21:40:30.478 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:34:55.382804+0000", 2024-03-11T21:40:30.478 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 7.4800000000000004e-06 2024-03-11T21:40:30.478 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:40:30.478 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:40:30.479 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-03-11T21:40:30.479 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:34:56.358897+0000", 2024-03-11T21:40:30.479 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.97609253399999996 2024-03-11T21:40:30.479 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:40:30.479 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:40:30.479 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "started", 2024-03-11T21:40:30.480 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:34:56.358972+0000", 2024-03-11T21:40:30.480 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 7.4806000000000002e-05 2024-03-11T21:40:30.480 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:40:30.480 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:40:30.480 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "waiting for subops from 3,7", 2024-03-11T21:40:30.480 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:34:56.359055+0000", 2024-03-11T21:40:30.481 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 8.3790999999999998e-05 2024-03-11T21:40:30.481 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:40:30.481 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:40:30.481 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "op_commit", 2024-03-11T21:40:30.481 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:34:56.360293+0000", 2024-03-11T21:40:30.481 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.0012380970000000001 2024-03-11T21:40:30.481 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:40:30.482 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:40:30.482 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "sub_op_commit_rec", 2024-03-11T21:40:30.482 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:34:56.360502+0000", 2024-03-11T21:40:30.482 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.000208858 2024-03-11T21:40:30.482 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:40:30.482 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:40:30.483 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "sub_op_commit_rec", 2024-03-11T21:40:30.483 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:34:56.360563+0000", 2024-03-11T21:40:30.483 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 6.1173000000000005e-05 2024-03-11T21:40:30.483 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:40:30.483 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:40:30.484 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "commit_sent", 2024-03-11T21:40:30.484 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:34:56.360608+0000", 2024-03-11T21:40:30.484 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 4.4193999999999997e-05 2024-03-11T21:40:30.484 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:40:30.484 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:40:30.484 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "done", 2024-03-11T21:40:30.484 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:34:56.360644+0000", 2024-03-11T21:40:30.484 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 3.6560999999999998e-05 2024-03-11T21:40:30.485 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-03-11T21:40:30.485 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-03-11T21:40:30.485 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-03-11T21:40:30.485 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:40:30.485 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:40:30.485 INFO:teuthology.orchestra.run.smithi092.stdout: "description": "osd_op(client.15150.0:207239 8.c 8:3f0835ea:::benchmark_data_smithi092_104865_object86738:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e237)", 2024-03-11T21:40:30.485 INFO:teuthology.orchestra.run.smithi092.stdout: "initiated_at": "2024-03-11T21:34:55.465013+0000", 2024-03-11T21:40:30.486 INFO:teuthology.orchestra.run.smithi092.stdout: "age": 334.99109553, 2024-03-11T21:40:30.486 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.908045836, 2024-03-11T21:40:30.486 INFO:teuthology.orchestra.run.smithi092.stdout: "type_data": { 2024-03-11T21:40:30.486 INFO:teuthology.orchestra.run.smithi092.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-11T21:40:30.486 INFO:teuthology.orchestra.run.smithi092.stdout: "client_info": { 2024-03-11T21:40:30.486 INFO:teuthology.orchestra.run.smithi092.stdout: "client": "client.15150", 2024-03-11T21:40:30.487 INFO:teuthology.orchestra.run.smithi092.stdout: "client_addr": "v1:172.21.15.92:0/436606717", 2024-03-11T21:40:30.487 INFO:teuthology.orchestra.run.smithi092.stdout: "tid": 207239 2024-03-11T21:40:30.487 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:40:30.487 INFO:teuthology.orchestra.run.smithi092.stdout: "events": [ 2024-03-11T21:40:30.487 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:40:30.487 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "initiated", 2024-03-11T21:40:30.487 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:34:55.465013+0000", 2024-03-11T21:40:30.487 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-03-11T21:40:30.488 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:40:30.488 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:40:30.488 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "header_read", 2024-03-11T21:40:30.488 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:34:55.465013+0000", 2024-03-11T21:40:30.488 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-03-11T21:40:30.488 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:40:30.488 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:40:30.489 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "throttled", 2024-03-11T21:40:30.489 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:34:55.465016+0000", 2024-03-11T21:40:30.489 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 2.615e-06 2024-03-11T21:40:30.489 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:40:30.489 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:40:30.489 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "all_read", 2024-03-11T21:40:30.489 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:34:55.465021+0000", 2024-03-11T21:40:30.490 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 5.2889999999999997e-06 2024-03-11T21:40:30.490 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:40:30.490 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:40:30.490 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "dispatched", 2024-03-11T21:40:30.490 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:34:55.465036+0000", 2024-03-11T21:40:30.490 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.5023e-05 2024-03-11T21:40:30.490 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:40:30.491 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:40:30.491 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "queued_for_pg", 2024-03-11T21:40:30.491 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:34:55.465044+0000", 2024-03-11T21:40:30.491 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 7.9999999999999996e-06 2024-03-11T21:40:30.491 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:40:30.491 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:40:30.491 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-03-11T21:40:30.491 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:34:55.465100+0000", 2024-03-11T21:40:30.492 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 5.5337999999999998e-05 2024-03-11T21:40:30.492 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:40:30.492 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:40:30.492 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "waiting for peered", 2024-03-11T21:40:30.492 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:34:55.465108+0000", 2024-03-11T21:40:30.492 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 8.5259999999999993e-06 2024-03-11T21:40:30.492 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:40:30.492 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:40:30.492 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-03-11T21:40:30.492 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:34:56.360688+0000", 2024-03-11T21:40:30.493 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.89557976299999997 2024-03-11T21:40:30.493 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:40:30.493 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:40:30.493 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "started", 2024-03-11T21:40:30.493 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:34:56.360781+0000", 2024-03-11T21:40:30.493 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 9.3274000000000002e-05 2024-03-11T21:40:30.493 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:40:30.493 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:40:30.494 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "waiting for subops from 3,7", 2024-03-11T21:40:30.494 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:34:56.360842+0000", 2024-03-11T21:40:30.494 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 6.1334000000000006e-05 2024-03-11T21:40:30.494 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:40:30.494 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:40:30.494 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "sub_op_commit_rec", 2024-03-11T21:40:30.494 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:34:56.367917+0000", 2024-03-11T21:40:30.495 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.0070748219999999997 2024-03-11T21:40:30.495 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:40:30.495 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:40:30.495 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "sub_op_commit_rec", 2024-03-11T21:40:30.495 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:34:56.367996+0000", 2024-03-11T21:40:30.495 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 7.8614000000000003e-05 2024-03-11T21:40:30.495 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:40:30.495 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:40:30.495 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "op_commit", 2024-03-11T21:40:30.495 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:34:56.372955+0000", 2024-03-11T21:40:30.495 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.0049590809999999997 2024-03-11T21:40:30.495 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:40:30.495 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:40:30.495 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "commit_sent", 2024-03-11T21:40:30.495 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:34:56.373012+0000", 2024-03-11T21:40:30.496 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 5.7241000000000003e-05 2024-03-11T21:40:30.496 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:40:30.496 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:40:30.496 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "done", 2024-03-11T21:40:30.496 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:34:56.373059+0000", 2024-03-11T21:40:30.496 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 4.6916000000000003e-05 2024-03-11T21:40:30.496 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-03-11T21:40:30.496 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-03-11T21:40:30.496 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-03-11T21:40:30.496 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:40:30.496 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:40:30.496 INFO:teuthology.orchestra.run.smithi092.stdout: "description": "osd_op(client.15150.0:207246 8.c 8:36f7cf62:::benchmark_data_smithi092_104865_object86745:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e237)", 2024-03-11T21:40:30.496 INFO:teuthology.orchestra.run.smithi092.stdout: "initiated_at": "2024-03-11T21:34:55.472104+0000", 2024-03-11T21:40:30.496 INFO:teuthology.orchestra.run.smithi092.stdout: "age": 334.98400481700003, 2024-03-11T21:40:30.497 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.90107643000000004, 2024-03-11T21:40:30.497 INFO:teuthology.orchestra.run.smithi092.stdout: "type_data": { 2024-03-11T21:40:30.497 INFO:teuthology.orchestra.run.smithi092.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-11T21:40:30.497 INFO:teuthology.orchestra.run.smithi092.stdout: "client_info": { 2024-03-11T21:40:30.497 INFO:teuthology.orchestra.run.smithi092.stdout: "client": "client.15150", 2024-03-11T21:40:30.497 INFO:teuthology.orchestra.run.smithi092.stdout: "client_addr": "v1:172.21.15.92:0/436606717", 2024-03-11T21:40:30.497 INFO:teuthology.orchestra.run.smithi092.stdout: "tid": 207246 2024-03-11T21:40:30.497 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:40:30.497 INFO:teuthology.orchestra.run.smithi092.stdout: "events": [ 2024-03-11T21:40:30.497 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:40:30.497 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "initiated", 2024-03-11T21:40:30.497 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:34:55.472104+0000", 2024-03-11T21:40:30.497 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-03-11T21:40:30.497 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:40:30.497 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:40:30.498 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "header_read", 2024-03-11T21:40:30.498 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:34:55.472104+0000", 2024-03-11T21:40:30.498 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-03-11T21:40:30.498 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:40:30.498 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:40:30.498 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "throttled", 2024-03-11T21:40:30.498 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:34:55.472106+0000", 2024-03-11T21:40:30.498 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.9130000000000001e-06 2024-03-11T21:40:30.498 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:40:30.498 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:40:30.498 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "all_read", 2024-03-11T21:40:30.498 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:34:55.472110+0000", 2024-03-11T21:40:30.498 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 4.3769999999999998e-06 2024-03-11T21:40:30.499 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:40:30.499 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:40:30.499 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "dispatched", 2024-03-11T21:40:30.499 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:34:55.472120+0000", 2024-03-11T21:40:30.499 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.0010000000000001e-05 2024-03-11T21:40:30.499 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:40:30.499 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:40:30.499 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "queued_for_pg", 2024-03-11T21:40:30.499 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:34:55.472127+0000", 2024-03-11T21:40:30.499 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 6.8029999999999999e-06 2024-03-11T21:40:30.499 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:40:30.499 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:40:30.499 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-03-11T21:40:30.499 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:34:55.472219+0000", 2024-03-11T21:40:30.500 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 9.1914000000000001e-05 2024-03-11T21:40:30.500 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:40:30.500 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:40:30.500 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "waiting for peered", 2024-03-11T21:40:30.500 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:34:55.472250+0000", 2024-03-11T21:40:30.500 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 3.0528000000000003e-05 2024-03-11T21:40:30.500 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:40:30.500 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:40:30.500 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-03-11T21:40:30.500 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:34:56.362790+0000", 2024-03-11T21:40:30.500 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.89054004200000003 2024-03-11T21:40:30.500 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:40:30.500 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:40:30.500 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "started", 2024-03-11T21:40:30.501 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:34:56.362911+0000", 2024-03-11T21:40:30.501 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00012171999999999999 2024-03-11T21:40:30.501 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:40:30.501 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:40:30.501 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "waiting for subops from 3,7", 2024-03-11T21:40:30.501 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:34:56.363042+0000", 2024-03-11T21:40:30.501 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00013084699999999999 2024-03-11T21:40:30.501 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:40:30.501 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:40:30.501 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "sub_op_commit_rec", 2024-03-11T21:40:30.501 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:34:56.367956+0000", 2024-03-11T21:40:30.501 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.0049135979999999999 2024-03-11T21:40:30.501 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:40:30.501 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:40:30.501 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "sub_op_commit_rec", 2024-03-11T21:40:30.502 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:34:56.373103+0000", 2024-03-11T21:40:30.502 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.0051477320000000003 2024-03-11T21:40:30.502 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:40:30.502 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:40:30.502 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "op_commit", 2024-03-11T21:40:30.502 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:34:56.373117+0000", 2024-03-11T21:40:30.502 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.3678e-05 2024-03-11T21:40:30.502 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:40:30.502 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:40:30.502 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "commit_sent", 2024-03-11T21:40:30.502 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:34:56.373153+0000", 2024-03-11T21:40:30.502 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 3.5952999999999999e-05 2024-03-11T21:40:30.503 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:40:30.503 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:40:30.503 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "done", 2024-03-11T21:40:30.503 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:34:56.373180+0000", 2024-03-11T21:40:30.503 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 2.7314999999999999e-05 2024-03-11T21:40:30.503 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-03-11T21:40:30.503 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-03-11T21:40:30.503 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-03-11T21:40:30.503 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:40:30.503 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:40:30.503 INFO:teuthology.orchestra.run.smithi092.stdout: "description": "osd_op(client.15150.0:207247 8.c 8:3b696c26:::benchmark_data_smithi092_104865_object86746:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e237)", 2024-03-11T21:40:30.503 INFO:teuthology.orchestra.run.smithi092.stdout: "initiated_at": "2024-03-11T21:34:55.472328+0000", 2024-03-11T21:40:30.503 INFO:teuthology.orchestra.run.smithi092.stdout: "age": 334.98378125400001, 2024-03-11T21:40:30.503 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.90103873400000001, 2024-03-11T21:40:30.504 INFO:teuthology.orchestra.run.smithi092.stdout: "type_data": { 2024-03-11T21:40:30.504 INFO:teuthology.orchestra.run.smithi092.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-11T21:40:30.504 INFO:teuthology.orchestra.run.smithi092.stdout: "client_info": { 2024-03-11T21:40:30.504 INFO:teuthology.orchestra.run.smithi092.stdout: "client": "client.15150", 2024-03-11T21:40:30.504 INFO:teuthology.orchestra.run.smithi092.stdout: "client_addr": "v1:172.21.15.92:0/436606717", 2024-03-11T21:40:30.504 INFO:teuthology.orchestra.run.smithi092.stdout: "tid": 207247 2024-03-11T21:40:30.504 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:40:30.504 INFO:teuthology.orchestra.run.smithi092.stdout: "events": [ 2024-03-11T21:40:30.504 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:40:30.504 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "initiated", 2024-03-11T21:40:30.504 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:34:55.472328+0000", 2024-03-11T21:40:30.504 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-03-11T21:40:30.504 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:40:30.504 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:40:30.505 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "header_read", 2024-03-11T21:40:30.505 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:34:55.472328+0000", 2024-03-11T21:40:30.505 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-03-11T21:40:30.505 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:40:30.505 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:40:30.505 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "throttled", 2024-03-11T21:40:30.505 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:34:55.472329+0000", 2024-03-11T21:40:30.505 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.0020000000000001e-06 2024-03-11T21:40:30.505 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:40:30.505 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:40:30.505 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "all_read", 2024-03-11T21:40:30.505 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:34:55.472333+0000", 2024-03-11T21:40:30.505 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 4.0690000000000003e-06 2024-03-11T21:40:30.505 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:40:30.506 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:40:30.506 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "dispatched", 2024-03-11T21:40:30.506 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:34:55.472341+0000", 2024-03-11T21:40:30.506 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 8.3629999999999994e-06 2024-03-11T21:40:30.506 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:40:30.506 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:40:30.506 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "queued_for_pg", 2024-03-11T21:40:30.506 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:34:55.472347+0000", 2024-03-11T21:40:30.506 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 6.455e-06 2024-03-11T21:40:30.506 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:40:30.506 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:40:30.506 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-03-11T21:40:30.506 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:34:55.472409+0000", 2024-03-11T21:40:30.507 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 6.1939000000000004e-05 2024-03-11T21:40:30.507 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:40:30.507 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:40:30.507 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "waiting for peered", 2024-03-11T21:40:30.507 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:34:55.472418+0000", 2024-03-11T21:40:30.507 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 8.5299999999999996e-06 2024-03-11T21:40:30.507 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:40:30.507 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:40:30.507 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-03-11T21:40:30.507 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:34:56.365407+0000", 2024-03-11T21:40:30.507 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.89298929900000001 2024-03-11T21:40:30.507 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:40:30.507 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:40:30.507 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "started", 2024-03-11T21:40:30.507 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:34:56.365528+0000", 2024-03-11T21:40:30.507 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.000121049 2024-03-11T21:40:30.507 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:40:30.508 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:40:30.508 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "waiting for subops from 3,7", 2024-03-11T21:40:30.508 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:34:56.365636+0000", 2024-03-11T21:40:30.508 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.000107936 2024-03-11T21:40:30.508 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:40:30.508 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:40:30.508 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "sub_op_commit_rec", 2024-03-11T21:40:30.508 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:34:56.368039+0000", 2024-03-11T21:40:30.508 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.0024027969999999999 2024-03-11T21:40:30.508 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:40:30.508 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:40:30.508 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "op_commit", 2024-03-11T21:40:30.508 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:34:56.373189+0000", 2024-03-11T21:40:30.508 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.0051496520000000002 2024-03-11T21:40:30.508 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:40:30.509 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:40:30.509 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "sub_op_commit_rec", 2024-03-11T21:40:30.509 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:34:56.373254+0000", 2024-03-11T21:40:30.509 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 6.4906999999999994e-05 2024-03-11T21:40:30.509 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:40:30.509 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:40:30.509 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "commit_sent", 2024-03-11T21:40:30.509 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:34:56.373312+0000", 2024-03-11T21:40:30.509 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 5.8545000000000003e-05 2024-03-11T21:40:30.509 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:40:30.509 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:40:30.509 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "done", 2024-03-11T21:40:30.509 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:34:56.373366+0000", 2024-03-11T21:40:30.509 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 5.4190999999999997e-05 2024-03-11T21:40:30.509 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-03-11T21:40:30.509 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-03-11T21:40:30.510 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-03-11T21:40:30.510 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:40:30.510 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:40:30.510 INFO:teuthology.orchestra.run.smithi092.stdout: "description": "osd_op(client.15150.0:207254 8.c 8:33b1c360:::benchmark_data_smithi092_104865_object86753:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e237)", 2024-03-11T21:40:30.510 INFO:teuthology.orchestra.run.smithi092.stdout: "initiated_at": "2024-03-11T21:34:55.476305+0000", 2024-03-11T21:40:30.510 INFO:teuthology.orchestra.run.smithi092.stdout: "age": 334.97980399099998, 2024-03-11T21:40:30.510 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.897760224, 2024-03-11T21:40:30.510 INFO:teuthology.orchestra.run.smithi092.stdout: "type_data": { 2024-03-11T21:40:30.510 INFO:teuthology.orchestra.run.smithi092.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-11T21:40:30.510 INFO:teuthology.orchestra.run.smithi092.stdout: "client_info": { 2024-03-11T21:40:30.510 INFO:teuthology.orchestra.run.smithi092.stdout: "client": "client.15150", 2024-03-11T21:40:30.510 INFO:teuthology.orchestra.run.smithi092.stdout: "client_addr": "v1:172.21.15.92:0/436606717", 2024-03-11T21:40:30.510 INFO:teuthology.orchestra.run.smithi092.stdout: "tid": 207254 2024-03-11T21:40:30.511 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:40:30.511 INFO:teuthology.orchestra.run.smithi092.stdout: "events": [ 2024-03-11T21:40:30.511 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:40:30.511 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "initiated", 2024-03-11T21:40:30.511 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:34:55.476305+0000", 2024-03-11T21:40:30.511 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-03-11T21:40:30.511 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:40:30.511 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:40:30.511 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "header_read", 2024-03-11T21:40:30.511 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:34:55.476305+0000", 2024-03-11T21:40:30.511 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-03-11T21:40:30.511 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:40:30.511 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:40:30.511 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "throttled", 2024-03-11T21:40:30.512 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:34:55.476306+0000", 2024-03-11T21:40:30.512 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.491e-06 2024-03-11T21:40:30.512 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:40:30.512 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:40:30.512 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "all_read", 2024-03-11T21:40:30.512 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:34:55.476311+0000", 2024-03-11T21:40:30.512 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 4.6120000000000001e-06 2024-03-11T21:40:30.512 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:40:30.512 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:40:30.512 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "dispatched", 2024-03-11T21:40:30.512 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:34:55.476321+0000", 2024-03-11T21:40:30.512 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.0526999999999999e-05 2024-03-11T21:40:30.512 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:40:30.513 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:40:30.513 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "queued_for_pg", 2024-03-11T21:40:30.513 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:34:55.476329+0000", 2024-03-11T21:40:30.513 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 7.0899999999999999e-06 2024-03-11T21:40:30.513 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:40:30.513 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:40:30.513 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-03-11T21:40:30.513 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:34:55.476379+0000", 2024-03-11T21:40:30.513 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 5.0643999999999998e-05 2024-03-11T21:40:30.513 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:40:30.513 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:40:30.513 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "waiting for peered", 2024-03-11T21:40:30.514 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:34:55.476389+0000", 2024-03-11T21:40:30.514 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 9.465e-06 2024-03-11T21:40:30.514 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:40:30.514 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:40:30.514 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-03-11T21:40:30.514 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:34:56.368072+0000", 2024-03-11T21:40:30.514 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.89168363299999998 2024-03-11T21:40:30.514 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:40:30.514 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:40:30.514 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "started", 2024-03-11T21:40:30.514 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:34:56.368172+0000", 2024-03-11T21:40:30.514 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 9.9549000000000005e-05 2024-03-11T21:40:30.515 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:40:30.515 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:40:30.515 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "waiting for subops from 3,7", 2024-03-11T21:40:30.515 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:34:56.368536+0000", 2024-03-11T21:40:30.515 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00036459799999999999 2024-03-11T21:40:30.515 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:40:30.515 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:40:30.515 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "op_commit", 2024-03-11T21:40:30.515 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:34:56.373195+0000", 2024-03-11T21:40:30.515 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.0046583249999999996 2024-03-11T21:40:30.515 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:40:30.515 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:40:30.515 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "sub_op_commit_rec", 2024-03-11T21:40:30.515 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:34:56.373416+0000", 2024-03-11T21:40:30.516 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.000220955 2024-03-11T21:40:30.516 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:40:30.516 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:40:30.516 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "sub_op_commit_rec", 2024-03-11T21:40:30.516 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:34:56.373948+0000", 2024-03-11T21:40:30.516 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00053270499999999996 2024-03-11T21:40:30.516 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:40:30.516 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:40:30.516 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "commit_sent", 2024-03-11T21:40:30.516 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:34:56.374022+0000", 2024-03-11T21:40:30.516 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 7.3209999999999999e-05 2024-03-11T21:40:30.516 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:40:30.516 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:40:30.517 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "done", 2024-03-11T21:40:30.517 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:34:56.374065+0000", 2024-03-11T21:40:30.517 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 4.3420000000000001e-05 2024-03-11T21:40:30.517 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-03-11T21:40:30.517 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-03-11T21:40:30.517 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-03-11T21:40:30.517 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:40:30.517 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:40:30.517 INFO:teuthology.orchestra.run.smithi092.stdout: "description": "osd_op(client.15150.0:207261 8.c 8:3bacfc79:::benchmark_data_smithi092_104865_object86760:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e237)", 2024-03-11T21:40:30.517 INFO:teuthology.orchestra.run.smithi092.stdout: "initiated_at": "2024-03-11T21:34:55.481650+0000", 2024-03-11T21:40:30.517 INFO:teuthology.orchestra.run.smithi092.stdout: "age": 334.97445925099998, 2024-03-11T21:40:30.517 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.90050556400000004, 2024-03-11T21:40:30.517 INFO:teuthology.orchestra.run.smithi092.stdout: "type_data": { 2024-03-11T21:40:30.517 INFO:teuthology.orchestra.run.smithi092.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-11T21:40:30.518 INFO:teuthology.orchestra.run.smithi092.stdout: "client_info": { 2024-03-11T21:40:30.518 INFO:teuthology.orchestra.run.smithi092.stdout: "client": "client.15150", 2024-03-11T21:40:30.518 INFO:teuthology.orchestra.run.smithi092.stdout: "client_addr": "v1:172.21.15.92:0/436606717", 2024-03-11T21:40:30.518 INFO:teuthology.orchestra.run.smithi092.stdout: "tid": 207261 2024-03-11T21:40:30.518 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:40:30.518 INFO:teuthology.orchestra.run.smithi092.stdout: "events": [ 2024-03-11T21:40:30.518 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:40:30.518 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "initiated", 2024-03-11T21:40:30.518 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:34:55.481650+0000", 2024-03-11T21:40:30.518 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-03-11T21:40:30.518 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:40:30.518 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:40:30.519 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "header_read", 2024-03-11T21:40:30.519 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:34:55.481650+0000", 2024-03-11T21:40:30.519 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-03-11T21:40:30.519 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:40:30.519 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:40:30.519 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "throttled", 2024-03-11T21:40:30.519 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:34:55.481652+0000", 2024-03-11T21:40:30.519 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 2.069e-06 2024-03-11T21:40:30.519 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:40:30.519 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:40:30.519 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "all_read", 2024-03-11T21:40:30.519 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:34:55.481659+0000", 2024-03-11T21:40:30.520 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 6.9480000000000002e-06 2024-03-11T21:40:30.520 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:40:30.520 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:40:30.520 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "dispatched", 2024-03-11T21:40:30.520 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:34:55.481675+0000", 2024-03-11T21:40:30.520 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.6773e-05 2024-03-11T21:40:30.520 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:40:30.520 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:40:30.520 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "queued_for_pg", 2024-03-11T21:40:30.520 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:34:55.481685+0000", 2024-03-11T21:40:30.520 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 9.9839999999999996e-06 2024-03-11T21:40:30.520 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:40:30.520 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:40:30.520 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-03-11T21:40:30.521 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:34:55.481744+0000", 2024-03-11T21:40:30.521 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 5.8618e-05 2024-03-11T21:40:30.521 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:40:30.521 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:40:30.521 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "waiting for peered", 2024-03-11T21:40:30.521 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:34:55.481752+0000", 2024-03-11T21:40:30.521 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 8.0309999999999998e-06 2024-03-11T21:40:30.521 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:40:30.521 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:40:30.521 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-03-11T21:40:30.521 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:34:56.370719+0000", 2024-03-11T21:40:30.522 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.88896741000000001 2024-03-11T21:40:30.522 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:40:30.522 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:40:30.522 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "started", 2024-03-11T21:40:30.522 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:34:56.370828+0000", 2024-03-11T21:40:30.522 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00010908600000000001 2024-03-11T21:40:30.522 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:40:30.522 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:40:30.522 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "waiting for subops from 3,7", 2024-03-11T21:40:30.523 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:34:56.370943+0000", 2024-03-11T21:40:30.523 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.000114456 2024-03-11T21:40:30.523 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:40:30.523 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:40:30.523 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "op_commit", 2024-03-11T21:40:30.523 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:34:56.374079+0000", 2024-03-11T21:40:30.523 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.0031359790000000001 2024-03-11T21:40:30.523 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:40:30.524 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:40:30.524 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "sub_op_commit_rec", 2024-03-11T21:40:30.524 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:34:56.374111+0000", 2024-03-11T21:40:30.524 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 3.2277999999999998e-05 2024-03-11T21:40:30.524 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:40:30.524 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:40:30.524 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "sub_op_commit_rec", 2024-03-11T21:40:30.524 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:34:56.382056+0000", 2024-03-11T21:40:30.524 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.0079444860000000006 2024-03-11T21:40:30.525 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:40:30.525 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:40:30.525 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "commit_sent", 2024-03-11T21:40:30.525 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:34:56.382114+0000", 2024-03-11T21:40:30.525 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 5.8307999999999998e-05 2024-03-11T21:40:30.525 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:40:30.526 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:40:30.526 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "done", 2024-03-11T21:40:30.526 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:34:56.382155+0000", 2024-03-11T21:40:30.526 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 4.1137999999999998e-05 2024-03-11T21:40:30.526 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-03-11T21:40:30.526 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-03-11T21:40:30.527 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-03-11T21:40:30.527 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:40:30.527 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:40:30.527 INFO:teuthology.orchestra.run.smithi092.stdout: "description": "osd_op(client.15150.0:207274 8.c 8:35340fad:::benchmark_data_smithi092_104865_object86773:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e237)", 2024-03-11T21:40:30.527 INFO:teuthology.orchestra.run.smithi092.stdout: "initiated_at": "2024-03-11T21:34:55.489338+0000", 2024-03-11T21:40:30.527 INFO:teuthology.orchestra.run.smithi092.stdout: "age": 334.966770784, 2024-03-11T21:40:30.528 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.89598779799999995, 2024-03-11T21:40:30.528 INFO:teuthology.orchestra.run.smithi092.stdout: "type_data": { 2024-03-11T21:40:30.528 INFO:teuthology.orchestra.run.smithi092.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-11T21:40:30.528 INFO:teuthology.orchestra.run.smithi092.stdout: "client_info": { 2024-03-11T21:40:30.528 INFO:teuthology.orchestra.run.smithi092.stdout: "client": "client.15150", 2024-03-11T21:40:30.528 INFO:teuthology.orchestra.run.smithi092.stdout: "client_addr": "v1:172.21.15.92:0/436606717", 2024-03-11T21:40:30.528 INFO:teuthology.orchestra.run.smithi092.stdout: "tid": 207274 2024-03-11T21:40:30.529 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:40:30.529 INFO:teuthology.orchestra.run.smithi092.stdout: "events": [ 2024-03-11T21:40:30.529 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:40:30.529 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "initiated", 2024-03-11T21:40:30.529 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:34:55.489338+0000", 2024-03-11T21:40:30.529 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-03-11T21:40:30.529 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:40:30.529 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:40:30.530 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "header_read", 2024-03-11T21:40:30.530 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:34:55.489338+0000", 2024-03-11T21:40:30.530 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-03-11T21:40:30.530 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:40:30.530 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:40:30.530 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "throttled", 2024-03-11T21:40:30.530 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:34:55.489339+0000", 2024-03-11T21:40:30.531 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.2559999999999999e-06 2024-03-11T21:40:30.531 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:40:30.531 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:40:30.531 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "all_read", 2024-03-11T21:40:30.531 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:34:55.489344+0000", 2024-03-11T21:40:30.531 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 4.7049999999999998e-06 2024-03-11T21:40:30.531 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:40:30.532 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:40:30.532 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "dispatched", 2024-03-11T21:40:30.532 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:34:55.489353+0000", 2024-03-11T21:40:30.532 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 9.1419999999999999e-06 2024-03-11T21:40:30.532 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:40:30.532 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:40:30.532 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "queued_for_pg", 2024-03-11T21:40:30.533 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:34:55.489386+0000", 2024-03-11T21:40:30.533 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 3.2929999999999998e-05 2024-03-11T21:40:30.533 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:40:30.533 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:40:30.533 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-03-11T21:40:30.533 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:34:55.489435+0000", 2024-03-11T21:40:30.534 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 4.8609000000000002e-05 2024-03-11T21:40:30.534 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:40:30.534 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:40:30.534 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "waiting for peered", 2024-03-11T21:40:30.534 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:34:55.489443+0000", 2024-03-11T21:40:30.534 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 8.3580000000000003e-06 2024-03-11T21:40:30.534 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:40:30.534 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:40:30.534 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-03-11T21:40:30.534 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:34:56.374137+0000", 2024-03-11T21:40:30.534 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.884694325 2024-03-11T21:40:30.534 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:40:30.535 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:40:30.535 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "started", 2024-03-11T21:40:30.535 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:34:56.374240+0000", 2024-03-11T21:40:30.535 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.000102196 2024-03-11T21:40:30.535 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:40:30.535 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:40:30.535 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "waiting for subops from 3,7", 2024-03-11T21:40:30.535 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:34:56.374354+0000", 2024-03-11T21:40:30.535 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.000114065 2024-03-11T21:40:30.535 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:40:30.535 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:40:30.535 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "sub_op_commit_rec", 2024-03-11T21:40:30.535 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:34:56.382018+0000", 2024-03-11T21:40:30.535 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.0076644570000000004 2024-03-11T21:40:30.535 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:40:30.535 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:40:30.536 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "op_commit", 2024-03-11T21:40:30.536 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:34:56.385190+0000", 2024-03-11T21:40:30.536 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.0031721290000000001 2024-03-11T21:40:30.536 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:40:30.536 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:40:30.536 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "sub_op_commit_rec", 2024-03-11T21:40:30.536 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:34:56.385231+0000", 2024-03-11T21:40:30.536 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 4.0587000000000003e-05 2024-03-11T21:40:30.536 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:40:30.536 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:40:30.536 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "commit_sent", 2024-03-11T21:40:30.536 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:34:56.385282+0000", 2024-03-11T21:40:30.536 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 5.0899000000000002e-05 2024-03-11T21:40:30.536 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:40:30.537 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:40:30.537 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "done", 2024-03-11T21:40:30.537 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:34:56.385326+0000", 2024-03-11T21:40:30.537 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 4.4140000000000001e-05 2024-03-11T21:40:30.537 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-03-11T21:40:30.537 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-03-11T21:40:30.537 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-03-11T21:40:30.537 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:40:30.537 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:40:30.537 INFO:teuthology.orchestra.run.smithi092.stdout: "description": "osd_op(client.15150.0:207275 8.c 8:3a3c7648:::benchmark_data_smithi092_104865_object86774:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e237)", 2024-03-11T21:40:30.537 INFO:teuthology.orchestra.run.smithi092.stdout: "initiated_at": "2024-03-11T21:34:55.489720+0000", 2024-03-11T21:40:30.537 INFO:teuthology.orchestra.run.smithi092.stdout: "age": 334.96638838000001, 2024-03-11T21:40:30.537 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.89580858900000004, 2024-03-11T21:40:30.537 INFO:teuthology.orchestra.run.smithi092.stdout: "type_data": { 2024-03-11T21:40:30.537 INFO:teuthology.orchestra.run.smithi092.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-11T21:40:30.538 INFO:teuthology.orchestra.run.smithi092.stdout: "client_info": { 2024-03-11T21:40:30.538 INFO:teuthology.orchestra.run.smithi092.stdout: "client": "client.15150", 2024-03-11T21:40:30.538 INFO:teuthology.orchestra.run.smithi092.stdout: "client_addr": "v1:172.21.15.92:0/436606717", 2024-03-11T21:40:30.538 INFO:teuthology.orchestra.run.smithi092.stdout: "tid": 207275 2024-03-11T21:40:30.538 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:40:30.538 INFO:teuthology.orchestra.run.smithi092.stdout: "events": [ 2024-03-11T21:40:30.538 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:40:30.538 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "initiated", 2024-03-11T21:40:30.538 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:34:55.489720+0000", 2024-03-11T21:40:30.538 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-03-11T21:40:30.538 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:40:30.538 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:40:30.538 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "header_read", 2024-03-11T21:40:30.539 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:34:55.489720+0000", 2024-03-11T21:40:30.539 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-03-11T21:40:30.539 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:40:30.539 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:40:30.539 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "throttled", 2024-03-11T21:40:30.539 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:34:55.489722+0000", 2024-03-11T21:40:30.539 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.1370000000000001e-06 2024-03-11T21:40:30.539 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:40:30.539 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:40:30.539 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "all_read", 2024-03-11T21:40:30.539 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:34:55.489726+0000", 2024-03-11T21:40:30.539 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 4.6129999999999997e-06 2024-03-11T21:40:30.539 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:40:30.539 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:40:30.540 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "dispatched", 2024-03-11T21:40:30.540 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:34:55.489735+0000", 2024-03-11T21:40:30.540 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 9.3160000000000003e-06 2024-03-11T21:40:30.540 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:40:30.540 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:40:30.540 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "queued_for_pg", 2024-03-11T21:40:30.540 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:34:55.489742+0000", 2024-03-11T21:40:30.540 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 6.9720000000000003e-06 2024-03-11T21:40:30.540 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:40:30.540 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:40:30.540 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-03-11T21:40:30.540 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:34:55.489795+0000", 2024-03-11T21:40:30.540 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 5.2796000000000001e-05 2024-03-11T21:40:30.540 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:40:30.540 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:40:30.541 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "waiting for peered", 2024-03-11T21:40:30.541 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:34:55.489802+0000", 2024-03-11T21:40:30.541 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 7.2230000000000001e-06 2024-03-11T21:40:30.541 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:40:30.541 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:40:30.541 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-03-11T21:40:30.541 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:34:56.376733+0000", 2024-03-11T21:40:30.541 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.88693052100000003 2024-03-11T21:40:30.541 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:40:30.541 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:40:30.541 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "started", 2024-03-11T21:40:30.541 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:34:56.376845+0000", 2024-03-11T21:40:30.541 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.000111606 2024-03-11T21:40:30.541 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:40:30.542 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:40:30.542 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "waiting for subops from 3,7", 2024-03-11T21:40:30.542 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:34:56.376959+0000", 2024-03-11T21:40:30.542 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.000114636 2024-03-11T21:40:30.542 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:40:30.542 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:40:30.542 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "sub_op_commit_rec", 2024-03-11T21:40:30.542 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:34:56.382200+0000", 2024-03-11T21:40:30.542 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.0052405400000000001 2024-03-11T21:40:30.542 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:40:30.543 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:40:30.543 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "op_commit", 2024-03-11T21:40:30.543 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:34:56.385339+0000", 2024-03-11T21:40:30.543 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.0031393789999999999 2024-03-11T21:40:30.543 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:40:30.543 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:40:30.543 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "sub_op_commit_rec", 2024-03-11T21:40:30.543 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:34:56.385434+0000", 2024-03-11T21:40:30.544 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 9.4402999999999997e-05 2024-03-11T21:40:30.544 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:40:30.544 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:40:30.544 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "commit_sent", 2024-03-11T21:40:30.544 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:34:56.385491+0000", 2024-03-11T21:40:30.544 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 5.7373000000000001e-05 2024-03-11T21:40:30.544 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:40:30.544 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:40:30.545 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "done", 2024-03-11T21:40:30.545 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:34:56.385529+0000", 2024-03-11T21:40:30.545 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 3.8074000000000002e-05 2024-03-11T21:40:30.545 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-03-11T21:40:30.545 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-03-11T21:40:30.545 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-03-11T21:40:30.545 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:40:30.545 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:40:30.545 INFO:teuthology.orchestra.run.smithi092.stdout: "description": "osd_op(client.15150.0:207314 8.c 8:3b0993f7:::benchmark_data_smithi092_104865_object86813:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e237)", 2024-03-11T21:40:30.546 INFO:teuthology.orchestra.run.smithi092.stdout: "initiated_at": "2024-03-11T21:34:55.525379+0000", 2024-03-11T21:40:30.546 INFO:teuthology.orchestra.run.smithi092.stdout: "age": 334.93072930800002, 2024-03-11T21:40:30.546 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.86704656899999999, 2024-03-11T21:40:30.546 INFO:teuthology.orchestra.run.smithi092.stdout: "type_data": { 2024-03-11T21:40:30.546 INFO:teuthology.orchestra.run.smithi092.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-11T21:40:30.546 INFO:teuthology.orchestra.run.smithi092.stdout: "client_info": { 2024-03-11T21:40:30.546 INFO:teuthology.orchestra.run.smithi092.stdout: "client": "client.15150", 2024-03-11T21:40:30.546 INFO:teuthology.orchestra.run.smithi092.stdout: "client_addr": "v1:172.21.15.92:0/436606717", 2024-03-11T21:40:30.547 INFO:teuthology.orchestra.run.smithi092.stdout: "tid": 207314 2024-03-11T21:40:30.547 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:40:30.547 INFO:teuthology.orchestra.run.smithi092.stdout: "events": [ 2024-03-11T21:40:30.547 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:40:30.547 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "initiated", 2024-03-11T21:40:30.547 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:34:55.525379+0000", 2024-03-11T21:40:30.547 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-03-11T21:40:30.547 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:40:30.547 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:40:30.548 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "header_read", 2024-03-11T21:40:30.548 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:34:55.525379+0000", 2024-03-11T21:40:30.548 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-03-11T21:40:30.548 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:40:30.548 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:40:30.548 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "throttled", 2024-03-11T21:40:30.548 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:34:55.525381+0000", 2024-03-11T21:40:30.548 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.5239999999999999e-06 2024-03-11T21:40:30.549 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:40:30.549 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:40:30.549 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "all_read", 2024-03-11T21:40:30.549 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:34:55.525386+0000", 2024-03-11T21:40:30.549 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 4.549e-06 2024-03-11T21:40:30.549 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:40:30.549 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:40:30.549 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "dispatched", 2024-03-11T21:40:30.549 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:34:55.525394+0000", 2024-03-11T21:40:30.550 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 7.9689999999999994e-06 2024-03-11T21:40:30.550 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:40:30.550 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:40:30.550 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "queued_for_pg", 2024-03-11T21:40:30.550 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:34:55.525402+0000", 2024-03-11T21:40:30.550 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 8.0010000000000001e-06 2024-03-11T21:40:30.550 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:40:30.550 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:40:30.551 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-03-11T21:40:30.551 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:34:55.525468+0000", 2024-03-11T21:40:30.551 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 6.6648000000000003e-05 2024-03-11T21:40:30.551 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:40:30.551 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:40:30.551 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "waiting for peered", 2024-03-11T21:40:30.551 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:34:55.525476+0000", 2024-03-11T21:40:30.551 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 8.0409999999999998e-06 2024-03-11T21:40:30.551 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:40:30.552 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:40:30.552 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-03-11T21:40:30.552 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:34:56.379314+0000", 2024-03-11T21:40:30.552 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.85383820300000002 2024-03-11T21:40:30.552 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:40:30.552 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:40:30.552 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "started", 2024-03-11T21:40:30.552 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:34:56.379488+0000", 2024-03-11T21:40:30.552 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00017320899999999999 2024-03-11T21:40:30.553 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:40:30.553 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:40:30.553 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "waiting for subops from 3,7", 2024-03-11T21:40:30.553 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:34:56.379594+0000", 2024-03-11T21:40:30.553 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00010648499999999999 2024-03-11T21:40:30.553 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:40:30.553 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:40:30.553 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "op_commit", 2024-03-11T21:40:30.554 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:34:56.385344+0000", 2024-03-11T21:40:30.554 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.0057501740000000003 2024-03-11T21:40:30.554 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:40:30.554 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:40:30.554 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "sub_op_commit_rec", 2024-03-11T21:40:30.554 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:34:56.385571+0000", 2024-03-11T21:40:30.554 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00022667299999999999 2024-03-11T21:40:30.554 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:40:30.554 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:40:30.555 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "sub_op_commit_rec", 2024-03-11T21:40:30.555 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:34:56.392329+0000", 2024-03-11T21:40:30.555 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.0067575609999999996 2024-03-11T21:40:30.555 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:40:30.555 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:40:30.555 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "commit_sent", 2024-03-11T21:40:30.555 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:34:56.392382+0000", 2024-03-11T21:40:30.555 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 5.3282999999999998e-05 2024-03-11T21:40:30.555 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:40:30.556 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:40:30.556 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "done", 2024-03-11T21:40:30.556 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:34:56.392426+0000", 2024-03-11T21:40:30.556 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 4.4249000000000003e-05 2024-03-11T21:40:30.556 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-03-11T21:40:30.556 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-03-11T21:40:30.556 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-03-11T21:40:30.556 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:40:30.556 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:40:30.557 INFO:teuthology.orchestra.run.smithi092.stdout: "description": "osd_op(client.15150.0:207339 8.c 8:39421482:::benchmark_data_smithi092_104865_object86838:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e237)", 2024-03-11T21:40:30.557 INFO:teuthology.orchestra.run.smithi092.stdout: "initiated_at": "2024-03-11T21:34:55.550641+0000", 2024-03-11T21:40:30.557 INFO:teuthology.orchestra.run.smithi092.stdout: "age": 334.90546819399998, 2024-03-11T21:40:30.557 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.84199107500000003, 2024-03-11T21:40:30.557 INFO:teuthology.orchestra.run.smithi092.stdout: "type_data": { 2024-03-11T21:40:30.557 INFO:teuthology.orchestra.run.smithi092.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-11T21:40:30.557 INFO:teuthology.orchestra.run.smithi092.stdout: "client_info": { 2024-03-11T21:40:30.557 INFO:teuthology.orchestra.run.smithi092.stdout: "client": "client.15150", 2024-03-11T21:40:30.557 INFO:teuthology.orchestra.run.smithi092.stdout: "client_addr": "v1:172.21.15.92:0/436606717", 2024-03-11T21:40:30.557 INFO:teuthology.orchestra.run.smithi092.stdout: "tid": 207339 2024-03-11T21:40:30.557 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:40:30.558 INFO:teuthology.orchestra.run.smithi092.stdout: "events": [ 2024-03-11T21:40:30.558 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:40:30.558 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "initiated", 2024-03-11T21:40:30.558 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:34:55.550641+0000", 2024-03-11T21:40:30.558 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-03-11T21:40:30.558 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:40:30.558 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:40:30.558 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "header_read", 2024-03-11T21:40:30.558 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:34:55.550641+0000", 2024-03-11T21:40:30.559 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-03-11T21:40:30.559 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:40:30.559 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:40:30.559 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "throttled", 2024-03-11T21:40:30.559 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:34:55.550642+0000", 2024-03-11T21:40:30.559 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.474e-06 2024-03-11T21:40:30.559 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:40:30.559 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:40:30.559 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "all_read", 2024-03-11T21:40:30.559 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:34:55.550647+0000", 2024-03-11T21:40:30.559 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 4.639e-06 2024-03-11T21:40:30.559 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:40:30.559 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:40:30.559 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "dispatched", 2024-03-11T21:40:30.560 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:34:55.550656+0000", 2024-03-11T21:40:30.560 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 9.3109999999999995e-06 2024-03-11T21:40:30.560 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:40:30.560 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:40:30.560 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "queued_for_pg", 2024-03-11T21:40:30.560 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:34:55.550663+0000", 2024-03-11T21:40:30.560 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 7.2969999999999998e-06 2024-03-11T21:40:30.560 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:40:30.560 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:40:30.560 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-03-11T21:40:30.560 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:34:55.550752+0000", 2024-03-11T21:40:30.561 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 8.8441000000000002e-05 2024-03-11T21:40:30.561 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:40:30.561 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:40:30.561 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "waiting for peered", 2024-03-11T21:40:30.561 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:34:55.550762+0000", 2024-03-11T21:40:30.561 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.0685e-05 2024-03-11T21:40:30.561 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:40:30.561 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:40:30.562 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-03-11T21:40:30.562 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:34:56.382713+0000", 2024-03-11T21:40:30.562 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.83195014499999997 2024-03-11T21:40:30.562 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:40:30.562 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:40:30.562 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "started", 2024-03-11T21:40:30.562 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:34:56.382808+0000", 2024-03-11T21:40:30.562 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 9.5597e-05 2024-03-11T21:40:30.562 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:40:30.562 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:40:30.562 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "waiting for subops from 3,7", 2024-03-11T21:40:30.562 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:34:56.382885+0000", 2024-03-11T21:40:30.563 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 7.7191e-05 2024-03-11T21:40:30.563 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:40:30.563 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:40:30.563 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "sub_op_commit_rec", 2024-03-11T21:40:30.563 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:34:56.385611+0000", 2024-03-11T21:40:30.563 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.002726095 2024-03-11T21:40:30.563 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:40:30.563 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:40:30.563 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "op_commit", 2024-03-11T21:40:30.563 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:34:56.388598+0000", 2024-03-11T21:40:30.563 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:40:30.565 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.0029865439999999998 2024-03-11T21:40:30.565 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:40:30.565 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:40:30.565 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "sub_op_commit_rec", 2024-03-11T21:40:30.565 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:34:56.392542+0000", 2024-03-11T21:40:30.565 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.0039435980000000004 2024-03-11T21:40:30.565 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:40:30.566 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:40:30.566 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "commit_sent", 2024-03-11T21:40:30.566 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:34:56.392590+0000", 2024-03-11T21:40:30.566 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 4.8622999999999998e-05 2024-03-11T21:40:30.566 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:40:30.566 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:40:30.566 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "done", 2024-03-11T21:40:30.566 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:34:56.392632+0000", 2024-03-11T21:40:30.566 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 4.1434999999999999e-05 2024-03-11T21:40:30.566 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-03-11T21:40:30.567 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-03-11T21:40:30.567 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-03-11T21:40:30.567 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:40:30.567 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:40:30.567 INFO:teuthology.orchestra.run.smithi092.stdout: "description": "osd_op(client.15150.0:207361 8.c 8:3ecfd1aa:::benchmark_data_smithi092_104865_object86860:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e237)", 2024-03-11T21:40:30.567 INFO:teuthology.orchestra.run.smithi092.stdout: "initiated_at": "2024-03-11T21:34:55.573296+0000", 2024-03-11T21:40:30.567 INFO:teuthology.orchestra.run.smithi092.stdout: "age": 334.88281296399998, 2024-03-11T21:40:30.567 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.83335905099999996, 2024-03-11T21:40:30.567 INFO:teuthology.orchestra.run.smithi092.stdout: "type_data": { 2024-03-11T21:40:30.567 INFO:teuthology.orchestra.run.smithi092.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-11T21:40:30.567 INFO:teuthology.orchestra.run.smithi092.stdout: "client_info": { 2024-03-11T21:40:30.567 INFO:teuthology.orchestra.run.smithi092.stdout: "client": "client.15150", 2024-03-11T21:40:30.567 INFO:teuthology.orchestra.run.smithi092.stdout: "client_addr": "v1:172.21.15.92:0/436606717", 2024-03-11T21:40:30.568 INFO:teuthology.orchestra.run.smithi092.stdout: "tid": 207361 2024-03-11T21:40:30.568 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:40:30.568 INFO:teuthology.orchestra.run.smithi092.stdout: "events": [ 2024-03-11T21:40:30.568 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:40:30.568 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "initiated", 2024-03-11T21:40:30.568 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:34:55.573296+0000", 2024-03-11T21:40:30.569 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-03-11T21:40:30.569 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:40:30.569 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:40:30.569 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "header_read", 2024-03-11T21:40:30.569 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:34:55.573296+0000", 2024-03-11T21:40:30.569 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-03-11T21:40:30.570 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:40:30.570 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:40:30.570 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "throttled", 2024-03-11T21:40:30.570 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:34:55.573297+0000", 2024-03-11T21:40:30.570 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.547e-06 2024-03-11T21:40:30.570 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:40:30.571 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:40:30.571 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "all_read", 2024-03-11T21:40:30.571 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:34:55.573302+0000", 2024-03-11T21:40:30.571 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 4.2729999999999996e-06 2024-03-11T21:40:30.571 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:40:30.571 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:40:30.571 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "dispatched", 2024-03-11T21:40:30.571 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:34:55.573311+0000", 2024-03-11T21:40:30.572 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 9.7520000000000008e-06 2024-03-11T21:40:30.572 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:40:30.572 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:40:30.572 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "queued_for_pg", 2024-03-11T21:40:30.572 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:34:55.573319+0000", 2024-03-11T21:40:30.572 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 7.5270000000000001e-06 2024-03-11T21:40:30.572 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:40:30.573 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:40:30.573 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-03-11T21:40:30.573 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:34:55.573360+0000", 2024-03-11T21:40:30.573 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 4.1306000000000001e-05 2024-03-11T21:40:30.573 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:40:30.573 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:40:30.573 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "waiting for peered", 2024-03-11T21:40:30.574 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:34:55.573367+0000", 2024-03-11T21:40:30.574 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 6.3609999999999997e-06 2024-03-11T21:40:30.574 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:40:30.574 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:40:30.574 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-03-11T21:40:30.574 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:34:56.385737+0000", 2024-03-11T21:40:30.574 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.81237010300000001 2024-03-11T21:40:30.574 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:40:30.575 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:40:30.575 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "started", 2024-03-11T21:40:30.575 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:34:56.385845+0000", 2024-03-11T21:40:30.575 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.000108478 2024-03-11T21:40:30.575 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:40:30.575 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:40:30.575 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "waiting for subops from 3,7", 2024-03-11T21:40:30.575 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:34:56.385955+0000", 2024-03-11T21:40:30.575 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.000110135 2024-03-11T21:40:30.575 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:40:30.575 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:40:30.575 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "sub_op_commit_rec", 2024-03-11T21:40:30.575 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:34:56.392508+0000", 2024-03-11T21:40:30.575 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.0065524540000000001 2024-03-11T21:40:30.576 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:40:30.576 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:40:30.576 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "sub_op_commit_rec", 2024-03-11T21:40:30.576 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:34:56.395695+0000", 2024-03-11T21:40:30.576 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.003187685 2024-03-11T21:40:30.576 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:40:30.576 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:40:30.576 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "op_commit", 2024-03-11T21:40:30.576 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:34:56.406563+0000", 2024-03-11T21:40:30.576 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.010867523 2024-03-11T21:40:30.576 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:40:30.576 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:40:30.576 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "commit_sent", 2024-03-11T21:40:30.576 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:34:56.406610+0000", 2024-03-11T21:40:30.576 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 4.6808000000000003e-05 2024-03-11T21:40:30.577 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:40:30.577 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:40:30.577 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "done", 2024-03-11T21:40:30.577 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:34:56.406655+0000", 2024-03-11T21:40:30.577 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 4.5099000000000003e-05 2024-03-11T21:40:30.577 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-03-11T21:40:30.577 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-03-11T21:40:30.577 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-03-11T21:40:30.577 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:40:30.577 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:40:30.577 INFO:teuthology.orchestra.run.smithi092.stdout: "description": "osd_op(client.15150.0:207366 8.c 8:3b30730b:::benchmark_data_smithi092_104865_object86865:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e237)", 2024-03-11T21:40:30.577 INFO:teuthology.orchestra.run.smithi092.stdout: "initiated_at": "2024-03-11T21:34:55.578893+0000", 2024-03-11T21:40:30.577 INFO:teuthology.orchestra.run.smithi092.stdout: "age": 334.87721603400001, 2024-03-11T21:40:30.577 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.827840468, 2024-03-11T21:40:30.578 INFO:teuthology.orchestra.run.smithi092.stdout: "type_data": { 2024-03-11T21:40:30.578 INFO:teuthology.orchestra.run.smithi092.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-11T21:40:30.578 INFO:teuthology.orchestra.run.smithi092.stdout: "client_info": { 2024-03-11T21:40:30.578 INFO:teuthology.orchestra.run.smithi092.stdout: "client": "client.15150", 2024-03-11T21:40:30.578 INFO:teuthology.orchestra.run.smithi092.stdout: "client_addr": "v1:172.21.15.92:0/436606717", 2024-03-11T21:40:30.578 INFO:teuthology.orchestra.run.smithi092.stdout: "tid": 207366 2024-03-11T21:40:30.578 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:40:30.578 INFO:teuthology.orchestra.run.smithi092.stdout: "events": [ 2024-03-11T21:40:30.578 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:40:30.578 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "initiated", 2024-03-11T21:40:30.578 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:34:55.578893+0000", 2024-03-11T21:40:30.578 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-03-11T21:40:30.578 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:40:30.578 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:40:30.579 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "header_read", 2024-03-11T21:40:30.579 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:34:55.578893+0000", 2024-03-11T21:40:30.579 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-03-11T21:40:30.579 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:40:30.579 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:40:30.579 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "throttled", 2024-03-11T21:40:30.579 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:34:55.578894+0000", 2024-03-11T21:40:30.579 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.742e-06 2024-03-11T21:40:30.579 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:40:30.579 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:40:30.579 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "all_read", 2024-03-11T21:40:30.579 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:34:55.578901+0000", 2024-03-11T21:40:30.579 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 6.1369999999999999e-06 2024-03-11T21:40:30.579 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:40:30.580 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:40:30.580 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "dispatched", 2024-03-11T21:40:30.580 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:34:55.578912+0000", 2024-03-11T21:40:30.580 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.1402e-05 2024-03-11T21:40:30.580 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:40:30.580 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:40:30.580 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "queued_for_pg", 2024-03-11T21:40:30.580 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:34:55.578921+0000", 2024-03-11T21:40:30.580 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 8.5459999999999992e-06 2024-03-11T21:40:30.580 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:40:30.580 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:40:30.580 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-03-11T21:40:30.580 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:34:55.578972+0000", 2024-03-11T21:40:30.580 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 5.1138999999999999e-05 2024-03-11T21:40:30.581 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:40:30.581 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:40:30.581 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "waiting for peered", 2024-03-11T21:40:30.581 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:34:55.578978+0000", 2024-03-11T21:40:30.581 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 6.4219999999999997e-06 2024-03-11T21:40:30.581 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:40:30.581 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:40:30.581 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-03-11T21:40:30.581 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:34:56.389497+0000", 2024-03-11T21:40:30.581 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.81051906299999998 2024-03-11T21:40:30.581 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:40:30.581 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:40:30.581 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "started", 2024-03-11T21:40:30.581 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:34:56.389610+0000", 2024-03-11T21:40:30.581 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.000112972 2024-03-11T21:40:30.582 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:40:30.582 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:40:30.582 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "waiting for subops from 3,7", 2024-03-11T21:40:30.582 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:34:56.389751+0000", 2024-03-11T21:40:30.582 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.000140798 2024-03-11T21:40:30.582 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:40:30.582 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:40:30.582 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "sub_op_commit_rec", 2024-03-11T21:40:30.582 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:34:56.395736+0000", 2024-03-11T21:40:30.582 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.0059849680000000002 2024-03-11T21:40:30.582 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:40:30.582 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:40:30.582 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "sub_op_commit_rec", 2024-03-11T21:40:30.583 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:34:56.401487+0000", 2024-03-11T21:40:30.583 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.005750603 2024-03-11T21:40:30.583 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:40:30.583 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:40:30.583 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "op_commit", 2024-03-11T21:40:30.583 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:34:56.406662+0000", 2024-03-11T21:40:30.583 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.0051752480000000003 2024-03-11T21:40:30.583 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:40:30.583 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:40:30.583 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "commit_sent", 2024-03-11T21:40:30.583 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:34:56.406701+0000", 2024-03-11T21:40:30.583 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 3.9124e-05 2024-03-11T21:40:30.583 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:40:30.584 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:40:30.584 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "done", 2024-03-11T21:40:30.584 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:34:56.406733+0000", 2024-03-11T21:40:30.584 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 3.2304000000000001e-05 2024-03-11T21:40:30.584 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-03-11T21:40:30.584 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-03-11T21:40:30.584 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-03-11T21:40:30.584 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:40:30.584 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:40:30.584 INFO:teuthology.orchestra.run.smithi092.stdout: "description": "osd_op(client.15150.0:207382 8.c 8:31ac71bf:::benchmark_data_smithi092_104865_object86881:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e237)", 2024-03-11T21:40:30.584 INFO:teuthology.orchestra.run.smithi092.stdout: "initiated_at": "2024-03-11T21:34:55.605399+0000", 2024-03-11T21:40:30.584 INFO:teuthology.orchestra.run.smithi092.stdout: "age": 334.850710144, 2024-03-11T21:40:30.584 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.80149564600000001, 2024-03-11T21:40:30.585 INFO:teuthology.orchestra.run.smithi092.stdout: "type_data": { 2024-03-11T21:40:30.585 INFO:teuthology.orchestra.run.smithi092.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-11T21:40:30.585 INFO:teuthology.orchestra.run.smithi092.stdout: "client_info": { 2024-03-11T21:40:30.585 INFO:teuthology.orchestra.run.smithi092.stdout: "client": "client.15150", 2024-03-11T21:40:30.585 INFO:teuthology.orchestra.run.smithi092.stdout: "client_addr": "v1:172.21.15.92:0/436606717", 2024-03-11T21:40:30.585 INFO:teuthology.orchestra.run.smithi092.stdout: "tid": 207382 2024-03-11T21:40:30.585 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:40:30.585 INFO:teuthology.orchestra.run.smithi092.stdout: "events": [ 2024-03-11T21:40:30.585 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:40:30.585 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "initiated", 2024-03-11T21:40:30.585 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:34:55.605399+0000", 2024-03-11T21:40:30.585 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-03-11T21:40:30.585 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:40:30.585 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:40:30.586 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "header_read", 2024-03-11T21:40:30.586 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:34:55.605399+0000", 2024-03-11T21:40:30.586 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-03-11T21:40:30.586 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:40:30.586 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:40:30.586 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "throttled", 2024-03-11T21:40:30.586 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:34:55.605400+0000", 2024-03-11T21:40:30.586 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.778e-06 2024-03-11T21:40:30.586 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:40:30.586 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:40:30.586 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "all_read", 2024-03-11T21:40:30.586 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:34:55.605405+0000", 2024-03-11T21:40:30.586 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 5.0420000000000002e-06 2024-03-11T21:40:30.587 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:40:30.587 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:40:30.587 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "dispatched", 2024-03-11T21:40:30.587 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:34:55.605417+0000", 2024-03-11T21:40:30.587 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.1212e-05 2024-03-11T21:40:30.587 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:40:30.587 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:40:30.587 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "queued_for_pg", 2024-03-11T21:40:30.587 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:34:55.605426+0000", 2024-03-11T21:40:30.587 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 9.1889999999999996e-06 2024-03-11T21:40:30.587 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:40:30.587 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:40:30.587 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-03-11T21:40:30.587 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:34:55.605479+0000", 2024-03-11T21:40:30.588 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 5.3424999999999998e-05 2024-03-11T21:40:30.588 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:40:30.588 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:40:30.588 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "waiting for peered", 2024-03-11T21:40:30.588 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:34:55.605487+0000", 2024-03-11T21:40:30.588 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 7.8050000000000007e-06 2024-03-11T21:40:30.588 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:40:30.588 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:40:30.588 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-03-11T21:40:30.588 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:34:56.392659+0000", 2024-03-11T21:40:30.588 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.78717215399999996 2024-03-11T21:40:30.588 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:40:30.588 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:40:30.588 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "started", 2024-03-11T21:40:30.588 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:34:56.393047+0000", 2024-03-11T21:40:30.589 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00038755499999999999 2024-03-11T21:40:30.589 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:40:30.589 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:40:30.589 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "waiting for subops from 3,7", 2024-03-11T21:40:30.589 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:34:56.393166+0000", 2024-03-11T21:40:30.589 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00011882799999999999 2024-03-11T21:40:30.589 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:40:30.589 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:40:30.589 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "sub_op_commit_rec", 2024-03-11T21:40:30.589 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:34:56.401524+0000", 2024-03-11T21:40:30.589 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.0083586649999999995 2024-03-11T21:40:30.589 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:40:30.589 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:40:30.589 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "sub_op_commit_rec", 2024-03-11T21:40:30.589 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:34:56.401559+0000", 2024-03-11T21:40:30.590 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 3.4650000000000002e-05 2024-03-11T21:40:30.590 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:40:30.590 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:40:30.590 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "op_commit", 2024-03-11T21:40:30.590 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:34:56.406816+0000", 2024-03-11T21:40:30.590 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.0052568850000000002 2024-03-11T21:40:30.590 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:40:30.590 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:40:30.590 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "commit_sent", 2024-03-11T21:40:30.590 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:34:56.406859+0000", 2024-03-11T21:40:30.590 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 4.2892999999999997e-05 2024-03-11T21:40:30.590 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:40:30.590 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:40:30.591 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "done", 2024-03-11T21:40:30.591 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:34:56.406894+0000", 2024-03-11T21:40:30.591 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 3.5565000000000003e-05 2024-03-11T21:40:30.591 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-03-11T21:40:30.591 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-03-11T21:40:30.591 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-03-11T21:40:30.591 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:40:30.591 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:40:30.591 INFO:teuthology.orchestra.run.smithi092.stdout: "description": "osd_op(client.15150.0:207412 8.c 8:312392f7:::benchmark_data_smithi092_104865_object86911:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e237)", 2024-03-11T21:40:30.591 INFO:teuthology.orchestra.run.smithi092.stdout: "initiated_at": "2024-03-11T21:34:55.696161+0000", 2024-03-11T21:40:30.591 INFO:teuthology.orchestra.run.smithi092.stdout: "age": 334.75994817999998, 2024-03-11T21:40:30.591 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.71080487199999998, 2024-03-11T21:40:30.591 INFO:teuthology.orchestra.run.smithi092.stdout: "type_data": { 2024-03-11T21:40:30.591 INFO:teuthology.orchestra.run.smithi092.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-11T21:40:30.592 INFO:teuthology.orchestra.run.smithi092.stdout: "client_info": { 2024-03-11T21:40:30.592 INFO:teuthology.orchestra.run.smithi092.stdout: "client": "client.15150", 2024-03-11T21:40:30.592 INFO:teuthology.orchestra.run.smithi092.stdout: "client_addr": "v1:172.21.15.92:0/436606717", 2024-03-11T21:40:30.592 INFO:teuthology.orchestra.run.smithi092.stdout: "tid": 207412 2024-03-11T21:40:30.592 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:40:30.592 INFO:teuthology.orchestra.run.smithi092.stdout: "events": [ 2024-03-11T21:40:30.592 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:40:30.592 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "initiated", 2024-03-11T21:40:30.592 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:34:55.696161+0000", 2024-03-11T21:40:30.592 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-03-11T21:40:30.592 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:40:30.592 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:40:30.592 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "header_read", 2024-03-11T21:40:30.592 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:34:55.696161+0000", 2024-03-11T21:40:30.593 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-03-11T21:40:30.593 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:40:30.593 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:40:30.593 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "throttled", 2024-03-11T21:40:30.593 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:34:55.696163+0000", 2024-03-11T21:40:30.593 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.9309999999999998e-06 2024-03-11T21:40:30.593 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:40:30.593 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:40:30.593 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "all_read", 2024-03-11T21:40:30.593 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:34:55.696168+0000", 2024-03-11T21:40:30.593 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 5.0320000000000003e-06 2024-03-11T21:40:30.593 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:40:30.594 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:40:30.594 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "dispatched", 2024-03-11T21:40:30.594 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:34:55.696178+0000", 2024-03-11T21:40:30.594 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.0788000000000001e-05 2024-03-11T21:40:30.594 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:40:30.594 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:40:30.594 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "queued_for_pg", 2024-03-11T21:40:30.594 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:34:55.696184+0000", 2024-03-11T21:40:30.594 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 6.1380000000000004e-06 2024-03-11T21:40:30.594 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:40:30.594 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:40:30.594 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-03-11T21:40:30.594 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:34:55.696234+0000", 2024-03-11T21:40:30.595 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 4.9373000000000003e-05 2024-03-11T21:40:30.595 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:40:30.595 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:40:30.595 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "waiting for peered", 2024-03-11T21:40:30.595 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:34:55.696241+0000", 2024-03-11T21:40:30.595 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 6.6660000000000002e-06 2024-03-11T21:40:30.595 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:40:30.595 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:40:30.595 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-03-11T21:40:30.595 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:34:56.395765+0000", 2024-03-11T21:40:30.595 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.69952489699999998 2024-03-11T21:40:30.595 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:40:30.595 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:40:30.595 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "started", 2024-03-11T21:40:30.595 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:34:56.395874+0000", 2024-03-11T21:40:30.596 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00010840600000000001 2024-03-11T21:40:30.596 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:40:30.596 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:40:30.596 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "waiting for subops from 3,7", 2024-03-11T21:40:30.596 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:34:56.396221+0000", 2024-03-11T21:40:30.596 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.000346998 2024-03-11T21:40:30.596 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:40:30.596 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:40:30.596 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "sub_op_commit_rec", 2024-03-11T21:40:30.596 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:34:56.403836+0000", 2024-03-11T21:40:30.596 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.0076152470000000003 2024-03-11T21:40:30.596 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:40:30.596 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:40:30.596 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "sub_op_commit_rec", 2024-03-11T21:40:30.597 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:34:56.406771+0000", 2024-03-11T21:40:30.597 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.0029351730000000001 2024-03-11T21:40:30.597 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:40:30.597 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:40:30.597 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "op_commit", 2024-03-11T21:40:30.597 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:34:56.406900+0000", 2024-03-11T21:40:30.597 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.000128594 2024-03-11T21:40:30.597 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:40:30.597 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:40:30.597 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "commit_sent", 2024-03-11T21:40:30.597 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:34:56.406934+0000", 2024-03-11T21:40:30.597 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 3.4588000000000002e-05 2024-03-11T21:40:30.597 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:40:30.597 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:40:30.597 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "done", 2024-03-11T21:40:30.598 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:34:56.406965+0000", 2024-03-11T21:40:30.598 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 3.1041000000000003e-05 2024-03-11T21:40:30.598 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-03-11T21:40:30.598 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-03-11T21:40:30.598 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-03-11T21:40:30.598 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:40:30.598 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:40:30.598 INFO:teuthology.orchestra.run.smithi092.stdout: "description": "osd_op(client.25282.0:76299 9.8 9:17331e66:::benchmark_data_smithi092_130291_object76298:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e278)", 2024-03-11T21:40:30.598 INFO:teuthology.orchestra.run.smithi092.stdout: "initiated_at": "2024-03-11T21:36:35.048550+0000", 2024-03-11T21:40:30.598 INFO:teuthology.orchestra.run.smithi092.stdout: "age": 235.407559182, 2024-03-11T21:40:30.598 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.4852829970000001, 2024-03-11T21:40:30.598 INFO:teuthology.orchestra.run.smithi092.stdout: "type_data": { 2024-03-11T21:40:30.599 INFO:teuthology.orchestra.run.smithi092.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-11T21:40:30.599 INFO:teuthology.orchestra.run.smithi092.stdout: "client_info": { 2024-03-11T21:40:30.599 INFO:teuthology.orchestra.run.smithi092.stdout: "client": "client.25282", 2024-03-11T21:40:30.599 INFO:teuthology.orchestra.run.smithi092.stdout: "client_addr": "v1:172.21.15.92:0/2489519821", 2024-03-11T21:40:30.599 INFO:teuthology.orchestra.run.smithi092.stdout: "tid": 76299 2024-03-11T21:40:30.599 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:40:30.599 INFO:teuthology.orchestra.run.smithi092.stdout: "events": [ 2024-03-11T21:40:30.599 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:40:30.599 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "initiated", 2024-03-11T21:40:30.599 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:36:35.048550+0000", 2024-03-11T21:40:30.599 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-03-11T21:40:30.599 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:40:30.599 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:40:30.599 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "header_read", 2024-03-11T21:40:30.600 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:36:35.048550+0000", 2024-03-11T21:40:30.600 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-03-11T21:40:30.600 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:40:30.600 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:40:30.600 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "throttled", 2024-03-11T21:40:30.600 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:36:35.048552+0000", 2024-03-11T21:40:30.600 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 2.0710000000000002e-06 2024-03-11T21:40:30.601 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:40:30.601 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:40:30.601 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "all_read", 2024-03-11T21:40:30.601 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:36:35.048584+0000", 2024-03-11T21:40:30.601 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 3.2676999999999999e-05 2024-03-11T21:40:30.601 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:40:30.602 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:40:30.602 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "dispatched", 2024-03-11T21:40:30.602 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:36:35.048597+0000", 2024-03-11T21:40:30.602 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.2845e-05 2024-03-11T21:40:30.602 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:40:30.602 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:40:30.603 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "queued_for_pg", 2024-03-11T21:40:30.603 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:36:35.048608+0000", 2024-03-11T21:40:30.603 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.1005000000000001e-05 2024-03-11T21:40:30.603 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:40:30.603 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:40:30.603 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-03-11T21:40:30.603 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:36:35.049475+0000", 2024-03-11T21:40:30.604 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00086655599999999999 2024-03-11T21:40:30.604 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:40:30.604 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:40:30.604 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "waiting for peered", 2024-03-11T21:40:30.604 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:36:35.049486+0000", 2024-03-11T21:40:30.604 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.0842000000000001e-05 2024-03-11T21:40:30.604 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:40:30.604 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:40:30.604 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-03-11T21:40:30.605 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:36:36.516700+0000", 2024-03-11T21:40:30.605 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.467214099 2024-03-11T21:40:30.605 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:40:30.605 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:40:30.605 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "started", 2024-03-11T21:40:30.605 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:36:36.516847+0000", 2024-03-11T21:40:30.605 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00014753699999999999 2024-03-11T21:40:30.605 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:40:30.606 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:40:30.606 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "waiting for subops from 3,5", 2024-03-11T21:40:30.606 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:36:36.517015+0000", 2024-03-11T21:40:30.606 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00016770700000000001 2024-03-11T21:40:30.606 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:40:30.606 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:40:30.606 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "sub_op_commit_rec", 2024-03-11T21:40:30.606 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:36:36.527882+0000", 2024-03-11T21:40:30.607 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.010866607 2024-03-11T21:40:30.607 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:40:30.607 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:40:30.607 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "sub_op_commit_rec", 2024-03-11T21:40:30.607 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:36:36.527930+0000", 2024-03-11T21:40:30.607 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 4.8575000000000002e-05 2024-03-11T21:40:30.607 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:40:30.607 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:40:30.608 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "op_commit", 2024-03-11T21:40:30.608 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:36:36.530142+0000", 2024-03-11T21:40:30.608 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.002212206 2024-03-11T21:40:30.608 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:40:30.608 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:40:30.608 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "commit_sent", 2024-03-11T21:40:30.608 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:36:36.530209+0000", 2024-03-11T21:40:30.608 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 6.6594999999999995e-05 2024-03-11T21:40:30.608 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:40:30.608 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:40:30.608 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "done", 2024-03-11T21:40:30.608 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:36:36.533833+0000", 2024-03-11T21:40:30.608 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.0036236749999999998 2024-03-11T21:40:30.608 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-03-11T21:40:30.608 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-03-11T21:40:30.609 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-03-11T21:40:30.609 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:40:30.609 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:40:30.609 INFO:teuthology.orchestra.run.smithi092.stdout: "description": "osd_op(client.25282.0:76301 9.8 9:1db40ce0:::benchmark_data_smithi092_130291_object76300:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e278)", 2024-03-11T21:40:30.609 INFO:teuthology.orchestra.run.smithi092.stdout: "initiated_at": "2024-03-11T21:36:35.048631+0000", 2024-03-11T21:40:30.609 INFO:teuthology.orchestra.run.smithi092.stdout: "age": 235.407478204, 2024-03-11T21:40:30.609 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.495054493, 2024-03-11T21:40:30.609 INFO:teuthology.orchestra.run.smithi092.stdout: "type_data": { 2024-03-11T21:40:30.609 INFO:teuthology.orchestra.run.smithi092.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-11T21:40:30.609 INFO:teuthology.orchestra.run.smithi092.stdout: "client_info": { 2024-03-11T21:40:30.609 INFO:teuthology.orchestra.run.smithi092.stdout: "client": "client.25282", 2024-03-11T21:40:30.609 INFO:teuthology.orchestra.run.smithi092.stdout: "client_addr": "v1:172.21.15.92:0/2489519821", 2024-03-11T21:40:30.609 INFO:teuthology.orchestra.run.smithi092.stdout: "tid": 76301 2024-03-11T21:40:30.609 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:40:30.609 INFO:teuthology.orchestra.run.smithi092.stdout: "events": [ 2024-03-11T21:40:30.610 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:40:30.610 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "initiated", 2024-03-11T21:40:30.610 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:36:35.048631+0000", 2024-03-11T21:40:30.610 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-03-11T21:40:30.610 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:40:30.610 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:40:30.610 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "header_read", 2024-03-11T21:40:30.610 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:36:35.048631+0000", 2024-03-11T21:40:30.610 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-03-11T21:40:30.610 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:40:30.610 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:40:30.610 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "throttled", 2024-03-11T21:40:30.610 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:36:35.048632+0000", 2024-03-11T21:40:30.611 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.1170000000000001e-06 2024-03-11T21:40:30.611 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:40:30.611 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:40:30.611 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "all_read", 2024-03-11T21:40:30.611 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:36:35.048660+0000", 2024-03-11T21:40:30.611 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 2.8243e-05 2024-03-11T21:40:30.611 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:40:30.611 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:40:30.611 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "dispatched", 2024-03-11T21:40:30.611 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:36:35.048669+0000", 2024-03-11T21:40:30.611 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 8.7579999999999998e-06 2024-03-11T21:40:30.611 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:40:30.611 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:40:30.611 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "queued_for_pg", 2024-03-11T21:40:30.611 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:36:35.048697+0000", 2024-03-11T21:40:30.612 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 2.8138999999999999e-05 2024-03-11T21:40:30.612 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:40:30.612 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:40:30.612 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-03-11T21:40:30.612 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:36:35.049509+0000", 2024-03-11T21:40:30.612 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00081172500000000001 2024-03-11T21:40:30.612 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:40:30.612 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:40:30.612 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "waiting for peered", 2024-03-11T21:40:30.613 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:36:35.049515+0000", 2024-03-11T21:40:30.613 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 5.9909999999999999e-06 2024-03-11T21:40:30.613 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:40:30.613 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:40:30.613 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-03-11T21:40:30.613 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:36:36.523893+0000", 2024-03-11T21:40:30.613 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.4743779799999999 2024-03-11T21:40:30.613 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:40:30.614 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:40:30.614 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "started", 2024-03-11T21:40:30.614 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:36:36.524187+0000", 2024-03-11T21:40:30.614 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00029435900000000001 2024-03-11T21:40:30.614 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:40:30.614 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:40:30.614 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "waiting for subops from 3,5", 2024-03-11T21:40:30.615 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:36:36.524376+0000", 2024-03-11T21:40:30.615 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00018891599999999999 2024-03-11T21:40:30.615 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:40:30.615 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:40:30.615 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "sub_op_commit_rec", 2024-03-11T21:40:30.616 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:36:36.530124+0000", 2024-03-11T21:40:30.616 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.0057478800000000004 2024-03-11T21:40:30.616 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:40:30.616 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:40:30.616 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "op_commit", 2024-03-11T21:40:30.616 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:36:36.533846+0000", 2024-03-11T21:40:30.616 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.0037218709999999999 2024-03-11T21:40:30.617 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:40:30.617 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:40:30.617 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "sub_op_commit_rec", 2024-03-11T21:40:30.617 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:36:36.538924+0000", 2024-03-11T21:40:30.617 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.0050780479999999999 2024-03-11T21:40:30.617 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:40:30.617 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:40:30.618 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "commit_sent", 2024-03-11T21:40:30.618 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:36:36.539008+0000", 2024-03-11T21:40:30.618 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 8.4528000000000001e-05 2024-03-11T21:40:30.618 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:40:30.618 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:40:30.618 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "done", 2024-03-11T21:40:30.618 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:36:36.543685+0000", 2024-03-11T21:40:30.618 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.0046769380000000003 2024-03-11T21:40:30.619 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-03-11T21:40:30.619 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-03-11T21:40:30.619 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-03-11T21:40:30.619 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:40:30.619 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:40:30.619 INFO:teuthology.orchestra.run.smithi092.stdout: "description": "osd_op(client.25282.0:177577 9.5 9:a81d75f3:::benchmark_data_smithi092_130291_object61193:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e324)", 2024-03-11T21:40:30.619 INFO:teuthology.orchestra.run.smithi092.stdout: "initiated_at": "2024-03-11T21:38:46.446161+0000", 2024-03-11T21:40:30.619 INFO:teuthology.orchestra.run.smithi092.stdout: "age": 104.009948005, 2024-03-11T21:40:30.620 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.29837460999999998, 2024-03-11T21:40:30.620 INFO:teuthology.orchestra.run.smithi092.stdout: "type_data": { 2024-03-11T21:40:30.620 INFO:teuthology.orchestra.run.smithi092.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-11T21:40:30.620 INFO:teuthology.orchestra.run.smithi092.stdout: "client_info": { 2024-03-11T21:40:30.620 INFO:teuthology.orchestra.run.smithi092.stdout: "client": "client.25282", 2024-03-11T21:40:30.620 INFO:teuthology.orchestra.run.smithi092.stdout: "client_addr": "v1:172.21.15.92:0/2489519821", 2024-03-11T21:40:30.621 INFO:teuthology.orchestra.run.smithi092.stdout: "tid": 177577 2024-03-11T21:40:30.621 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:40:30.621 INFO:teuthology.orchestra.run.smithi092.stdout: "events": [ 2024-03-11T21:40:30.621 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:40:30.621 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "initiated", 2024-03-11T21:40:30.621 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:38:46.446161+0000", 2024-03-11T21:40:30.622 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-03-11T21:40:30.622 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:40:30.622 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:40:30.622 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "header_read", 2024-03-11T21:40:30.622 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:38:46.446161+0000", 2024-03-11T21:40:30.622 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-03-11T21:40:30.622 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:40:30.622 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:40:30.623 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "throttled", 2024-03-11T21:40:30.623 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:38:46.446164+0000", 2024-03-11T21:40:30.623 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 2.8100000000000002e-06 2024-03-11T21:40:30.623 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:40:30.623 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:40:30.623 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "all_read", 2024-03-11T21:40:30.623 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:38:46.446172+0000", 2024-03-11T21:40:30.624 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 7.9880000000000004e-06 2024-03-11T21:40:30.624 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:40:30.624 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:40:30.624 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "dispatched", 2024-03-11T21:40:30.624 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:38:46.446187+0000", 2024-03-11T21:40:30.624 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.5048999999999999e-05 2024-03-11T21:40:30.624 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:40:30.624 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:40:30.624 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "queued_for_pg", 2024-03-11T21:40:30.625 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:38:46.446197+0000", 2024-03-11T21:40:30.625 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.0193e-05 2024-03-11T21:40:30.625 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:40:30.625 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:40:30.625 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-03-11T21:40:30.625 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:38:46.446261+0000", 2024-03-11T21:40:30.625 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 6.4320999999999996e-05 2024-03-11T21:40:30.625 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:40:30.625 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:40:30.625 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "started", 2024-03-11T21:40:30.625 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:38:46.446394+0000", 2024-03-11T21:40:30.625 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00013324699999999999 2024-03-11T21:40:30.625 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:40:30.625 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:40:30.625 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "waiting for subops from 5,7", 2024-03-11T21:40:30.626 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:38:46.446525+0000", 2024-03-11T21:40:30.626 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00013073199999999999 2024-03-11T21:40:30.626 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:40:30.626 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:40:30.626 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "sub_op_commit_rec", 2024-03-11T21:40:30.626 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:38:46.740814+0000", 2024-03-11T21:40:30.626 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.294288934 2024-03-11T21:40:30.626 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:40:30.626 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:40:30.626 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "sub_op_commit_rec", 2024-03-11T21:40:30.626 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:38:46.740862+0000", 2024-03-11T21:40:30.627 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 4.7756000000000002e-05 2024-03-11T21:40:30.627 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:40:30.627 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:40:30.627 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "op_commit", 2024-03-11T21:40:30.627 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:38:46.744429+0000", 2024-03-11T21:40:30.627 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.0035676509999999998 2024-03-11T21:40:30.627 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:40:30.627 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:40:30.627 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "commit_sent", 2024-03-11T21:40:30.627 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:38:46.744490+0000", 2024-03-11T21:40:30.627 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 6.0763000000000001e-05 2024-03-11T21:40:30.628 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:40:30.628 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:40:30.628 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "done", 2024-03-11T21:40:30.628 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:38:46.744535+0000", 2024-03-11T21:40:30.628 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 4.5166000000000001e-05 2024-03-11T21:40:30.628 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-03-11T21:40:30.628 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-03-11T21:40:30.628 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-03-11T21:40:30.628 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:40:30.628 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:40:30.628 INFO:teuthology.orchestra.run.smithi092.stdout: "description": "osd_op(client.25282.0:177578 9.b 9:d58a6f8a:::benchmark_data_smithi092_130291_object61194:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e324)", 2024-03-11T21:40:30.628 INFO:teuthology.orchestra.run.smithi092.stdout: "initiated_at": "2024-03-11T21:38:46.446217+0000", 2024-03-11T21:40:30.628 INFO:teuthology.orchestra.run.smithi092.stdout: "age": 104.009891954, 2024-03-11T21:40:30.628 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.30095150799999998, 2024-03-11T21:40:30.629 INFO:teuthology.orchestra.run.smithi092.stdout: "type_data": { 2024-03-11T21:40:30.629 INFO:teuthology.orchestra.run.smithi092.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-11T21:40:30.629 INFO:teuthology.orchestra.run.smithi092.stdout: "client_info": { 2024-03-11T21:40:30.629 INFO:teuthology.orchestra.run.smithi092.stdout: "client": "client.25282", 2024-03-11T21:40:30.629 INFO:teuthology.orchestra.run.smithi092.stdout: "client_addr": "v1:172.21.15.92:0/2489519821", 2024-03-11T21:40:30.629 INFO:teuthology.orchestra.run.smithi092.stdout: "tid": 177578 2024-03-11T21:40:30.629 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:40:30.629 INFO:teuthology.orchestra.run.smithi092.stdout: "events": [ 2024-03-11T21:40:30.629 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:40:30.630 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "initiated", 2024-03-11T21:40:30.630 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:38:46.446217+0000", 2024-03-11T21:40:30.630 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-03-11T21:40:30.630 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:40:30.630 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:40:30.630 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "header_read", 2024-03-11T21:40:30.630 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:38:46.446217+0000", 2024-03-11T21:40:30.630 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-03-11T21:40:30.630 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:40:30.630 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:40:30.630 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "throttled", 2024-03-11T21:40:30.630 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:38:46.446218+0000", 2024-03-11T21:40:30.631 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 8.6700000000000002e-07 2024-03-11T21:40:30.631 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:40:30.631 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:40:30.631 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "all_read", 2024-03-11T21:40:30.631 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:38:46.446222+0000", 2024-03-11T21:40:30.631 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 4.1810000000000003e-06 2024-03-11T21:40:30.631 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:40:30.631 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:40:30.631 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "dispatched", 2024-03-11T21:40:30.631 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:38:46.446228+0000", 2024-03-11T21:40:30.631 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 5.9889999999999997e-06 2024-03-11T21:40:30.632 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:40:30.632 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:40:30.632 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "queued_for_pg", 2024-03-11T21:40:30.632 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:38:46.446233+0000", 2024-03-11T21:40:30.632 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 5.3149999999999999e-06 2024-03-11T21:40:30.632 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:40:30.632 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:40:30.632 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-03-11T21:40:30.632 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:38:46.446296+0000", 2024-03-11T21:40:30.632 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 6.2323000000000006e-05 2024-03-11T21:40:30.632 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:40:30.632 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:40:30.632 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "started", 2024-03-11T21:40:30.632 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:38:46.446413+0000", 2024-03-11T21:40:30.633 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.000117768 2024-03-11T21:40:30.633 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:40:30.633 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:40:30.633 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "waiting for subops from 5,6", 2024-03-11T21:40:30.633 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:38:46.446532+0000", 2024-03-11T21:40:30.633 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.000118918 2024-03-11T21:40:30.633 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:40:30.633 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:40:30.633 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "sub_op_commit_rec", 2024-03-11T21:40:30.633 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:38:46.740653+0000", 2024-03-11T21:40:30.634 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.29412064500000001 2024-03-11T21:40:30.634 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:40:30.634 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:40:30.634 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "sub_op_commit_rec", 2024-03-11T21:40:30.634 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:38:46.740702+0000", 2024-03-11T21:40:30.634 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 4.9354999999999998e-05 2024-03-11T21:40:30.634 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:40:30.634 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:40:30.634 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "op_commit", 2024-03-11T21:40:30.634 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:38:46.747074+0000", 2024-03-11T21:40:30.634 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.006371659 2024-03-11T21:40:30.635 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:40:30.635 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:40:30.635 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "commit_sent", 2024-03-11T21:40:30.635 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:38:46.747123+0000", 2024-03-11T21:40:30.635 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 4.9502000000000001e-05 2024-03-11T21:40:30.635 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:40:30.635 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:40:30.635 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "done", 2024-03-11T21:40:30.635 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:38:46.747168+0000", 2024-03-11T21:40:30.635 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 4.4985999999999999e-05 2024-03-11T21:40:30.635 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-03-11T21:40:30.635 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-03-11T21:40:30.635 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-03-11T21:40:30.635 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-03-11T21:40:30.636 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-03-11T21:40:30.636 INFO:teuthology.orchestra.run.smithi092.stdout:} 2024-03-11T21:40:30.659 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:40:30.660 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:40:30.660 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:40:30.761 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:40:30.790 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:40:30.790 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:40:30.790 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:40:30.891 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:40:30.916 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:40:30.917 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:40:30.917 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:40:31.018 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:40:31.042 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:40:31.043 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:40:31.043 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:40:31.144 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:40:31.169 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:40:31.170 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:40:31.170 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:40:31.271 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:40:31.294 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:40:31.295 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:40:31.295 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:40:31.308 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:40:30 smithi092 ceph-mon[29572]: pgmap v692: 132 pgs: 132 active+clean; 582 KiB data, 3.6 GiB used, 712 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-03-11T21:40:31.308 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:40:30 smithi092 ceph-mon[29572]: 4.1b scrub starts 2024-03-11T21:40:31.308 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:40:30 smithi092 ceph-mon[29572]: 4.1b scrub ok 2024-03-11T21:40:31.308 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:40:30 smithi092 ceph-mon[29572]: 2.9 scrub starts 2024-03-11T21:40:31.308 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:40:30 smithi092 ceph-mon[29572]: 2.9 scrub ok 2024-03-11T21:40:31.308 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:40:30 smithi092 ceph-mon[29572]: 6.16 deep-scrub starts 2024-03-11T21:40:31.308 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:40:30 smithi092 ceph-mon[29572]: 6.16 deep-scrub ok 2024-03-11T21:40:31.309 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:40:30 smithi092 ceph-mon[37563]: pgmap v692: 132 pgs: 132 active+clean; 582 KiB data, 3.6 GiB used, 712 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-03-11T21:40:31.309 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:40:30 smithi092 ceph-mon[37563]: 4.1b scrub starts 2024-03-11T21:40:31.309 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:40:30 smithi092 ceph-mon[37563]: 4.1b scrub ok 2024-03-11T21:40:31.309 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:40:30 smithi092 ceph-mon[37563]: 2.9 scrub starts 2024-03-11T21:40:31.309 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:40:30 smithi092 ceph-mon[37563]: 2.9 scrub ok 2024-03-11T21:40:31.309 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:40:30 smithi092 ceph-mon[37563]: 6.16 deep-scrub starts 2024-03-11T21:40:31.309 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:40:30 smithi092 ceph-mon[37563]: 6.16 deep-scrub ok 2024-03-11T21:40:31.330 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:40:30 smithi097 ceph-mon[33317]: pgmap v692: 132 pgs: 132 active+clean; 582 KiB data, 3.6 GiB used, 712 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-03-11T21:40:31.330 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:40:30 smithi097 ceph-mon[33317]: 4.1b scrub starts 2024-03-11T21:40:31.330 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:40:30 smithi097 ceph-mon[33317]: 4.1b scrub ok 2024-03-11T21:40:31.330 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:40:30 smithi097 ceph-mon[33317]: 2.9 scrub starts 2024-03-11T21:40:31.330 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:40:30 smithi097 ceph-mon[33317]: 2.9 scrub ok 2024-03-11T21:40:31.330 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:40:30 smithi097 ceph-mon[33317]: 6.16 deep-scrub starts 2024-03-11T21:40:31.330 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:40:30 smithi097 ceph-mon[33317]: 6.16 deep-scrub ok 2024-03-11T21:40:31.396 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:40:31.420 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:40:31.420 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:40:31.421 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:40:31.522 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:40:31.545 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:40:31.546 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:40:31.547 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:40:31.647 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:40:31.676 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:40:31.676 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:40:31.677 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:40:31.777 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:40:31.803 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:40:31.803 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:40:31.804 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:40:31.905 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:40:31.931 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:40:31.932 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:40:31.932 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:40:31.941 INFO:teuthology.orchestra.run.smithi092.stderr:noscrub is set 2024-03-11T21:40:32.033 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:40:32.057 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:40:32.057 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:40:32.058 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:40:32.158 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:40:32.182 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:40:32.183 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:40:32.184 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:40:32.193 INFO:teuthology.orchestra.run.smithi092.stdout:osd.0: {} 2024-03-11T21:40:32.194 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:40:31 smithi092 ceph-mon[29572]: 6.c scrub starts 2024-03-11T21:40:32.194 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:40:31 smithi092 ceph-mon[29572]: 6.c scrub ok 2024-03-11T21:40:32.194 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:40:31 smithi092 ceph-mon[29572]: 4.6 deep-scrub starts 2024-03-11T21:40:32.194 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:40:31 smithi092 ceph-mon[29572]: 4.6 deep-scrub ok 2024-03-11T21:40:32.194 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:40:31 smithi092 ceph-mon[29572]: from='client.? v1:172.21.15.92:0/1463256568' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-11T21:40:32.194 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:40:31 smithi092 ceph-mon[29572]: from='client.35543 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-11T21:40:32.194 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:40:31 smithi092 ceph-mon[37563]: 6.c scrub starts 2024-03-11T21:40:32.194 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:40:31 smithi092 ceph-mon[37563]: 6.c scrub ok 2024-03-11T21:40:32.194 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:40:31 smithi092 ceph-mon[37563]: 4.6 deep-scrub starts 2024-03-11T21:40:32.195 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:40:31 smithi092 ceph-mon[37563]: 4.6 deep-scrub ok 2024-03-11T21:40:32.195 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:40:31 smithi092 ceph-mon[37563]: from='client.? v1:172.21.15.92:0/1463256568' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-11T21:40:32.195 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:40:31 smithi092 ceph-mon[37563]: from='client.35543 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-11T21:40:32.213 INFO:teuthology.orchestra.run.smithi092.stdout:osd.1: {} 2024-03-11T21:40:32.232 INFO:teuthology.orchestra.run.smithi092.stdout:osd.2: {} 2024-03-11T21:40:32.251 INFO:teuthology.orchestra.run.smithi092.stdout:osd.3: {} 2024-03-11T21:40:32.272 INFO:teuthology.orchestra.run.smithi092.stdout:osd.4: {} 2024-03-11T21:40:32.285 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:40:32.292 INFO:teuthology.orchestra.run.smithi092.stdout:osd.5: {} 2024-03-11T21:40:32.309 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:40:32.309 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:40:32.309 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:40:32.313 INFO:teuthology.orchestra.run.smithi092.stdout:osd.6: {} 2024-03-11T21:40:32.330 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:40:31 smithi097 ceph-mon[33317]: 6.c scrub starts 2024-03-11T21:40:32.330 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:40:31 smithi097 ceph-mon[33317]: 6.c scrub ok 2024-03-11T21:40:32.330 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:40:31 smithi097 ceph-mon[33317]: 4.6 deep-scrub starts 2024-03-11T21:40:32.330 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:40:31 smithi097 ceph-mon[33317]: 4.6 deep-scrub ok 2024-03-11T21:40:32.330 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:40:31 smithi097 ceph-mon[33317]: from='client.? v1:172.21.15.92:0/1463256568' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-11T21:40:32.330 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:40:31 smithi097 ceph-mon[33317]: from='client.35543 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-11T21:40:32.333 INFO:teuthology.orchestra.run.smithi092.stdout:osd.7: {} 2024-03-11T21:40:32.334 INFO:teuthology.orchestra.run.smithi092.stderr:osd.0: osd_enable_op_tracker = '' 2024-03-11T21:40:32.334 INFO:teuthology.orchestra.run.smithi092.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-11T21:40:32.334 INFO:teuthology.orchestra.run.smithi092.stderr:osd.2: osd_enable_op_tracker = '' 2024-03-11T21:40:32.334 INFO:teuthology.orchestra.run.smithi092.stderr:osd.3: osd_enable_op_tracker = '' 2024-03-11T21:40:32.334 INFO:teuthology.orchestra.run.smithi092.stderr:osd.4: osd_enable_op_tracker = '' 2024-03-11T21:40:32.334 INFO:teuthology.orchestra.run.smithi092.stderr:osd.5: osd_enable_op_tracker = '' 2024-03-11T21:40:32.334 INFO:teuthology.orchestra.run.smithi092.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-11T21:40:32.334 INFO:teuthology.orchestra.run.smithi092.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-11T21:40:32.410 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:40:32.438 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:40:32.439 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:40:32.439 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:40:32.540 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:40:32.568 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:40:32.568 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:40:32.568 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:40:32.669 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:40:32.676 DEBUG:teuthology.orchestra.run.smithi092:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid 00db0016-dfed-11ee-95c8-87774f69a715 -- ceph daemon osd.1 dump_ops_in_flight 2024-03-11T21:40:32.697 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:40:32.697 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:40:32.698 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:40:32.799 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:40:32.826 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:40:32.827 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:40:32.828 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:40:32.929 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:40:32.954 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:40:32.954 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:40:32.955 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:40:33.056 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:40:33.075 INFO:teuthology.orchestra.run.smithi092.stderr:Inferring config /var/lib/ceph/00db0016-dfed-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T21:40:33.079 INFO:journalctl@ceph.iscsi.iscsi.a.smithi097.stdout:Mar 11 21:40:32 smithi097 ceph-00db0016-dfed-11ee-95c8-87774f69a715-iscsi-iscsi-a[60943]: debug there is no tcmu-runner data available 2024-03-11T21:40:33.080 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:40:32 smithi097 ceph-mon[33317]: pgmap v693: 132 pgs: 132 active+clean; 582 KiB data, 3.6 GiB used, 712 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-03-11T21:40:33.080 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:40:32 smithi097 ceph-mon[33317]: 6.7 scrub starts 2024-03-11T21:40:33.080 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:40:32 smithi097 ceph-mon[33317]: 6.7 scrub ok 2024-03-11T21:40:33.080 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:40:32 smithi097 ceph-mon[33317]: 2.d scrub starts 2024-03-11T21:40:33.080 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:40:32 smithi097 ceph-mon[33317]: 2.d scrub ok 2024-03-11T21:40:33.080 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:40:32 smithi097 ceph-mon[33317]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-03-11T21:40:33.080 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:40:32 smithi097 ceph-mon[33317]: from='client.35543 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-03-11T21:40:33.080 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:40:32 smithi097 ceph-mon[33317]: osdmap e353: 8 total, 8 up, 8 in 2024-03-11T21:40:33.080 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:40:32 smithi097 ceph-mon[33317]: from='client.? v1:172.21.15.92:0/492654019' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-11T21:40:33.084 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:40:33.085 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:40:33.085 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:40:33.186 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:40:33.214 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:40:33.214 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:40:33.214 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:40:33.315 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:40:33.343 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:40:33.343 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:40:33.343 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:40:33.364 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:40:32 smithi092 ceph-mon[29572]: pgmap v693: 132 pgs: 132 active+clean; 582 KiB data, 3.6 GiB used, 712 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-03-11T21:40:33.364 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:40:32 smithi092 ceph-mon[29572]: 6.7 scrub starts 2024-03-11T21:40:33.364 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:40:32 smithi092 ceph-mon[29572]: 6.7 scrub ok 2024-03-11T21:40:33.364 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:40:32 smithi092 ceph-mon[29572]: 2.d scrub starts 2024-03-11T21:40:33.364 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:40:32 smithi092 ceph-mon[29572]: 2.d scrub ok 2024-03-11T21:40:33.364 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:40:32 smithi092 ceph-mon[29572]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-03-11T21:40:33.365 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:40:32 smithi092 ceph-mon[29572]: from='client.35543 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-03-11T21:40:33.365 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:40:32 smithi092 ceph-mon[29572]: osdmap e353: 8 total, 8 up, 8 in 2024-03-11T21:40:33.365 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:40:32 smithi092 ceph-mon[29572]: from='client.? v1:172.21.15.92:0/492654019' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-11T21:40:33.365 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:40:32 smithi092 ceph-mon[37563]: pgmap v693: 132 pgs: 132 active+clean; 582 KiB data, 3.6 GiB used, 712 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-03-11T21:40:33.365 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:40:32 smithi092 ceph-mon[37563]: 6.7 scrub starts 2024-03-11T21:40:33.365 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:40:32 smithi092 ceph-mon[37563]: 6.7 scrub ok 2024-03-11T21:40:33.365 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:40:32 smithi092 ceph-mon[37563]: 2.d scrub starts 2024-03-11T21:40:33.365 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:40:32 smithi092 ceph-mon[37563]: 2.d scrub ok 2024-03-11T21:40:33.365 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:40:32 smithi092 ceph-mon[37563]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-03-11T21:40:33.365 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:40:32 smithi092 ceph-mon[37563]: from='client.35543 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-03-11T21:40:33.365 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:40:32 smithi092 ceph-mon[37563]: osdmap e353: 8 total, 8 up, 8 in 2024-03-11T21:40:33.365 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:40:32 smithi092 ceph-mon[37563]: from='client.? v1:172.21.15.92:0/492654019' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-11T21:40:33.444 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:40:33.471 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:40:33.472 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:40:33.472 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:40:33.573 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:40:33.599 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:40:33.599 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:40:33.600 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:40:33.701 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:40:33.727 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:40:33.727 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:40:33.728 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:40:33.829 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:40:33.853 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:40:33.854 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:40:33.854 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:40:33.955 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:40:33.978 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:40:33.979 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:40:33.979 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:40:34.079 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:40:34.107 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:40:34.108 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:40:34.108 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:40:34.209 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:40:34.229 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:40:33 smithi092 ceph-mon[29572]: from='client.24469 v1:172.21.15.97:0/404208310' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:40:34.229 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:40:33 smithi092 ceph-mon[29572]: osdmap e354: 8 total, 8 up, 8 in 2024-03-11T21:40:34.229 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:40:33 smithi092 ceph-mon[37563]: from='client.24469 v1:172.21.15.97:0/404208310' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:40:34.229 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:40:33 smithi092 ceph-mon[37563]: osdmap e354: 8 total, 8 up, 8 in 2024-03-11T21:40:34.236 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:40:34.237 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:40:34.237 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:40:34.330 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:40:33 smithi097 ceph-mon[33317]: from='client.24469 v1:172.21.15.97:0/404208310' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:40:34.330 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:40:33 smithi097 ceph-mon[33317]: osdmap e354: 8 total, 8 up, 8 in 2024-03-11T21:40:34.338 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:40:34.364 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:40:34.364 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:40:34.364 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:40:34.465 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:40:34.493 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:40:34.494 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:40:34.494 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:40:34.595 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:40:34.623 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:40:34.624 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:40:34.624 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:40:34.725 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:40:34.749 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:40:34.750 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:40:34.750 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:40:34.851 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:40:34.878 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:40:34.879 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:40:34.879 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:40:34.980 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:40:35.005 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:40:35.006 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:40:35.006 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:40:35.107 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:40:35.113 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:40:34 smithi092 ceph-mon[29572]: pgmap v695: 132 pgs: 132 active+clean; 582 KiB data, 3.6 GiB used, 712 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-03-11T21:40:35.114 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:40:34 smithi092 ceph-mon[37563]: pgmap v695: 132 pgs: 132 active+clean; 582 KiB data, 3.6 GiB used, 712 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-03-11T21:40:35.133 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:40:35.134 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:40:35.135 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:40:35.235 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:40:35.264 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:40:35.265 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:40:35.265 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:40:35.330 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:40:34 smithi097 ceph-mon[33317]: pgmap v695: 132 pgs: 132 active+clean; 582 KiB data, 3.6 GiB used, 712 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-03-11T21:40:35.367 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:40:35.396 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:40:35.397 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:40:35.397 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:40:35.498 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:40:35.517 DEBUG:teuthology.orchestra.run.smithi092:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid 00db0016-dfed-11ee-95c8-87774f69a715 -- ceph osd set nodeep-scrub 2024-03-11T21:40:35.522 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:40:35.522 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:40:35.523 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:40:35.624 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:40:35.652 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:40:35.653 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:40:35.653 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:40:35.754 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:40:35.780 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:40:35.781 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:40:35.782 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:40:35.883 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:40:35.908 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:40:35.909 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:40:35.910 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:40:35.989 INFO:teuthology.orchestra.run.smithi092.stdout: 2024-03-11T21:40:35.991 INFO:teuthology.orchestra.run.smithi092.stderr:dumped all 2024-03-11T21:40:36.011 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:40:36.038 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:40:36.039 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:40:36.039 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:40:36.091 DEBUG:teuthology.orchestra.run.smithi092:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid 00db0016-dfed-11ee-95c8-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-03-11T21:40:36.141 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:40:36.165 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:40:36.165 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:40:36.166 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:40:36.260 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:40:35 smithi092 ceph-mon[37563]: purged_snaps scrub starts 2024-03-11T21:40:36.260 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:40:35 smithi092 ceph-mon[37563]: purged_snaps scrub ok 2024-03-11T21:40:36.260 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:40:35 smithi092 ceph-mon[37563]: from='mgr.14607 v1:172.21.15.92:0/1702717597' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T21:40:36.260 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:40:35 smithi092 ceph-mon[29572]: purged_snaps scrub starts 2024-03-11T21:40:36.261 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:40:35 smithi092 ceph-mon[29572]: purged_snaps scrub ok 2024-03-11T21:40:36.261 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:40:35 smithi092 ceph-mon[29572]: from='mgr.14607 v1:172.21.15.92:0/1702717597' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T21:40:36.266 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:40:36.290 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:40:36.291 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:40:36.291 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:40:36.329 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:40:35 smithi097 ceph-mon[33317]: purged_snaps scrub starts 2024-03-11T21:40:36.330 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:40:35 smithi097 ceph-mon[33317]: purged_snaps scrub ok 2024-03-11T21:40:36.330 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:40:35 smithi097 ceph-mon[33317]: from='mgr.14607 v1:172.21.15.92:0/1702717597' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T21:40:36.391 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:40:36.415 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:40:36.418 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:40:36.419 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:40:36.520 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:40:36.544 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:40:36.545 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:40:36.545 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:40:36.646 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:40:36.669 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:40:36.670 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:40:36.670 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:40:36.771 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:40:36.795 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:40:36.796 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:40:36.796 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:40:36.897 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:40:36.921 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:40:36.922 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:40:36.922 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:40:36.941 INFO:teuthology.orchestra.run.smithi092.stderr:Inferring config /var/lib/ceph/00db0016-dfed-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T21:40:37.023 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:40:37.046 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:40:37.047 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:40:37.047 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:40:37.148 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:40:37.171 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:40:37.172 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:40:37.172 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:40:37.229 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:40:36 smithi092 ceph-mon[29572]: pgmap v697: 132 pgs: 132 active+clean; 582 KiB data, 3.6 GiB used, 712 GiB / 715 GiB avail; 738 B/s rd, 0 op/s 2024-03-11T21:40:37.230 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:40:36 smithi092 ceph-mon[29572]: osdmap e355: 8 total, 8 up, 8 in 2024-03-11T21:40:37.230 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:40:36 smithi092 ceph-mon[29572]: from='client.35555 v1:172.21.15.92:0/1452754422' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-11T21:40:37.230 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:40:36 smithi092 ceph-mon[37563]: pgmap v697: 132 pgs: 132 active+clean; 582 KiB data, 3.6 GiB used, 712 GiB / 715 GiB avail; 738 B/s rd, 0 op/s 2024-03-11T21:40:37.230 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:40:36 smithi092 ceph-mon[37563]: osdmap e355: 8 total, 8 up, 8 in 2024-03-11T21:40:37.230 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:40:36 smithi092 ceph-mon[37563]: from='client.35555 v1:172.21.15.92:0/1452754422' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-11T21:40:37.273 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:40:37.296 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:40:37.297 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:40:37.297 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:40:37.330 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:40:36 smithi097 ceph-mon[33317]: pgmap v697: 132 pgs: 132 active+clean; 582 KiB data, 3.6 GiB used, 712 GiB / 715 GiB avail; 738 B/s rd, 0 op/s 2024-03-11T21:40:37.330 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:40:36 smithi097 ceph-mon[33317]: osdmap e355: 8 total, 8 up, 8 in 2024-03-11T21:40:37.330 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:40:36 smithi097 ceph-mon[33317]: from='client.35555 v1:172.21.15.92:0/1452754422' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-11T21:40:37.398 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:40:37.422 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:40:37.423 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:40:37.423 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:40:37.524 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:40:37.548 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:40:37.548 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:40:37.549 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:40:37.649 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:40:37.673 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:40:37.674 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:40:37.674 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:40:37.775 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:40:37.800 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:40:37.800 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:40:37.801 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:40:37.902 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:40:37.925 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:40:37.926 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:40:37.926 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:40:38.027 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:40:38.052 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:40:38.052 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:40:38.053 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:40:38.154 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:40:38.160 INFO:teuthology.orchestra.run.smithi092.stdout:{"pg_ready":true,"pg_map":{"version":698,"stamp":"2024-03-11T21:40:35.966604+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":596392,"num_objects":198,"num_object_clones":0,"num_object_copies":594,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":198,"num_whiteouts":0,"num_read":2222,"num_read_kb":2070,"num_write":901,"num_write_kb":2461,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":203,"num_bytes_recovered":2955718,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":993,"ondisk_log_size":993,"up":396,"acting":396,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":396,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":749961216,"kb_used":3763824,"kb_used_data":18976,"kb_used_omap":15,"kb_used_meta":3744752,"kb_avail":746197392,"statfs":{"total":767960285184,"available":764106129408,"internally_reserved":0,"allocated":19431424,"data_stored":11723319,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15601,"internal_metadata":3834626831},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":5,"num_read_kb":5,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.645010"},"pg_stats":[{"pgid":"6.1b","version":"0'0","reported_seq":558,"reported_epoch":354,"state":"active+clean","last_fresh":"2024-03-11T21:40:33.758388+0000","last_change":"2024-03-11T21:36:35.074931+0000","last_active":"2024-03-11T21:40:33.758388+0000","last_peered":"2024-03-11T21:40:33.758388+0000","last_clean":"2024-03-11T21:40:33.758388+0000","last_became_active":"2024-03-11T21:36:35.072636+0000","last_became_peered":"2024-03-11T21:36:35.072636+0000","last_unstale":"2024-03-11T21:40:33.758388+0000","last_undegraded":"2024-03-11T21:40:33.758388+0000","last_fullsized":"2024-03-11T21:40:33.758388+0000","mapping_epoch":277,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":278,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:35:07.021520+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:43.559452+0000","last_clean_scrub_stamp":"2024-03-11T21:35:07.021520+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":621,"reported_epoch":353,"state":"active+clean","last_fresh":"2024-03-11T21:40:31.954834+0000","last_change":"2024-03-11T21:36:00.720089+0000","last_active":"2024-03-11T21:40:31.954834+0000","last_peered":"2024-03-11T21:40:31.954834+0000","last_clean":"2024-03-11T21:40:31.954834+0000","last_became_active":"2024-03-11T21:25:42.578893+0000","last_became_peered":"2024-03-11T21:25:42.578893+0000","last_unstale":"2024-03-11T21:40:31.954834+0000","last_undegraded":"2024-03-11T21:40:31.954834+0000","last_fullsized":"2024-03-11T21:40:31.954834+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:36:00.719956+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:41.544550+0000","last_clean_scrub_stamp":"2024-03-11T21:36:00.719956+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1f","version":"0'0","reported_seq":652,"reported_epoch":354,"state":"active+clean","last_fresh":"2024-03-11T21:40:33.390641+0000","last_change":"2024-03-11T21:36:00.843838+0000","last_active":"2024-03-11T21:40:33.390641+0000","last_peered":"2024-03-11T21:40:33.390641+0000","last_clean":"2024-03-11T21:40:33.390641+0000","last_became_active":"2024-03-11T21:34:41.144945+0000","last_became_peered":"2024-03-11T21:34:41.144945+0000","last_unstale":"2024-03-11T21:40:33.390641+0000","last_undegraded":"2024-03-11T21:40:33.390641+0000","last_fullsized":"2024-03-11T21:40:33.390641+0000","mapping_epoch":226,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":227,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:36:00.843718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:37.512018+0000","last_clean_scrub_stamp":"2024-03-11T21:36:00.843718+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:40:37.147375+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,4],"acting":[0,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.19","version":"260'27","reported_seq":605,"reported_epoch":354,"state":"active+clean","last_fresh":"2024-03-11T21:40:33.757784+0000","last_change":"2024-03-11T21:36:35.089901+0000","last_active":"2024-03-11T21:40:33.757784+0000","last_peered":"2024-03-11T21:40:33.757784+0000","last_clean":"2024-03-11T21:40:33.757784+0000","last_became_active":"2024-03-11T21:36:35.067652+0000","last_became_peered":"2024-03-11T21:36:35.067652+0000","last_unstale":"2024-03-11T21:40:33.757784+0000","last_undegraded":"2024-03-11T21:40:33.757784+0000","last_fullsized":"2024-03-11T21:40:33.757784+0000","mapping_epoch":277,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":278,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'15","last_scrub_stamp":"2024-03-11T21:35:22.099374+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:39.525554+0000","last_clean_scrub_stamp":"2024-03-11T21:35:22.099374+0000","objects_scrubbed":0,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":42,"num_read_kb":33,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.1a","version":"0'0","reported_seq":608,"reported_epoch":353,"state":"active+clean","last_fresh":"2024-03-11T21:40:31.954961+0000","last_change":"2024-03-11T21:35:16.410927+0000","last_active":"2024-03-11T21:40:31.954961+0000","last_peered":"2024-03-11T21:40:31.954961+0000","last_clean":"2024-03-11T21:40:31.954961+0000","last_became_active":"2024-03-11T21:25:44.590923+0000","last_became_peered":"2024-03-11T21:25:44.590923+0000","last_unstale":"2024-03-11T21:40:31.954961+0000","last_undegraded":"2024-03-11T21:40:31.954961+0000","last_fullsized":"2024-03-11T21:40:31.954961+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:35:16.410816+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:43.559452+0000","last_clean_scrub_stamp":"2024-03-11T21:35:16.410816+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":24,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,1],"acting":[4,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":633,"reported_epoch":353,"state":"active+clean","last_fresh":"2024-03-11T21:40:31.950134+0000","last_change":"2024-03-11T21:36:15.576138+0000","last_active":"2024-03-11T21:40:31.950134+0000","last_peered":"2024-03-11T21:40:31.950134+0000","last_clean":"2024-03-11T21:40:31.950134+0000","last_became_active":"2024-03-11T21:34:41.154216+0000","last_became_peered":"2024-03-11T21:34:41.154216+0000","last_unstale":"2024-03-11T21:40:31.950134+0000","last_undegraded":"2024-03-11T21:40:31.950134+0000","last_fullsized":"2024-03-11T21:40:31.950134+0000","mapping_epoch":226,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":227,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:36:15.575887+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:41.544550+0000","last_clean_scrub_stamp":"2024-03-11T21:36:15.575887+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,7],"acting":[1,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1e","version":"0'0","reported_seq":572,"reported_epoch":354,"state":"active+clean","last_fresh":"2024-03-11T21:40:33.758461+0000","last_change":"2024-03-11T21:36:35.062413+0000","last_active":"2024-03-11T21:40:33.758461+0000","last_peered":"2024-03-11T21:40:33.758461+0000","last_clean":"2024-03-11T21:40:33.758461+0000","last_became_active":"2024-03-11T21:36:35.061786+0000","last_became_peered":"2024-03-11T21:36:35.061786+0000","last_unstale":"2024-03-11T21:40:33.758461+0000","last_undegraded":"2024-03-11T21:40:33.758461+0000","last_fullsized":"2024-03-11T21:40:33.758461+0000","mapping_epoch":277,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":278,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:32:18.578038+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:37.512018+0000","last_clean_scrub_stamp":"2024-03-11T21:32:18.578038+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.18","version":"260'15","reported_seq":658,"reported_epoch":353,"state":"active+clean","last_fresh":"2024-03-11T21:40:31.954637+0000","last_change":"2024-03-11T21:36:09.662765+0000","last_active":"2024-03-11T21:40:31.954637+0000","last_peered":"2024-03-11T21:40:31.954637+0000","last_clean":"2024-03-11T21:40:31.954637+0000","last_became_active":"2024-03-11T21:25:40.552216+0000","last_became_peered":"2024-03-11T21:25:40.552216+0000","last_unstale":"2024-03-11T21:40:31.954637+0000","last_undegraded":"2024-03-11T21:40:31.954637+0000","last_fullsized":"2024-03-11T21:40:31.954637+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"260'15","last_scrub_stamp":"2024-03-11T21:36:09.662615+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:39.525554+0000","last_clean_scrub_stamp":"2024-03-11T21:36:09.662615+0000","objects_scrubbed":4,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":623,"reported_epoch":352,"state":"active+clean","last_fresh":"2024-03-11T21:40:28.555931+0000","last_change":"2024-03-11T21:36:35.068915+0000","last_active":"2024-03-11T21:40:28.555931+0000","last_peered":"2024-03-11T21:40:28.555931+0000","last_clean":"2024-03-11T21:40:28.555931+0000","last_became_active":"2024-03-11T21:36:35.068453+0000","last_became_peered":"2024-03-11T21:36:35.068453+0000","last_unstale":"2024-03-11T21:40:28.555931+0000","last_undegraded":"2024-03-11T21:40:28.555931+0000","last_fullsized":"2024-03-11T21:40:28.555931+0000","mapping_epoch":277,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":278,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:35:26.190634+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:32:12.654573+0000","last_clean_scrub_stamp":"2024-03-11T21:35:26.190634+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3],"acting":[5,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":271,"reported_epoch":352,"state":"active+clean","last_fresh":"2024-03-11T21:40:28.457492+0000","last_change":"2024-03-11T21:35:11.902920+0000","last_active":"2024-03-11T21:40:28.457492+0000","last_peered":"2024-03-11T21:40:28.457492+0000","last_clean":"2024-03-11T21:40:28.457492+0000","last_became_active":"2024-03-11T21:34:41.145299+0000","last_became_peered":"2024-03-11T21:34:41.145299+0000","last_unstale":"2024-03-11T21:40:28.457492+0000","last_undegraded":"2024-03-11T21:40:28.457492+0000","last_fullsized":"2024-03-11T21:40:28.457492+0000","mapping_epoch":226,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":227,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:35:11.902818+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:41.544550+0000","last_clean_scrub_stamp":"2024-03-11T21:35:11.902818+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.1d","version":"0'0","reported_seq":633,"reported_epoch":352,"state":"active+clean","last_fresh":"2024-03-11T21:40:28.457967+0000","last_change":"2024-03-11T21:35:26.855588+0000","last_active":"2024-03-11T21:40:28.457967+0000","last_peered":"2024-03-11T21:40:28.457967+0000","last_clean":"2024-03-11T21:40:28.457967+0000","last_became_active":"2024-03-11T21:25:38.542628+0000","last_became_peered":"2024-03-11T21:25:38.542628+0000","last_unstale":"2024-03-11T21:40:28.457967+0000","last_undegraded":"2024-03-11T21:40:28.457967+0000","last_fullsized":"2024-03-11T21:40:28.457967+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:35:26.855511+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:37.512018+0000","last_clean_scrub_stamp":"2024-03-11T21:35:26.855511+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,0],"acting":[7,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.1b","version":"260'7","reported_seq":651,"reported_epoch":353,"state":"active+clean","last_fresh":"2024-03-11T21:40:31.955199+0000","last_change":"2024-03-11T21:40:29.597326+0000","last_active":"2024-03-11T21:40:31.955199+0000","last_peered":"2024-03-11T21:40:31.955199+0000","last_clean":"2024-03-11T21:40:31.955199+0000","last_became_active":"2024-03-11T21:36:35.057678+0000","last_became_peered":"2024-03-11T21:36:35.057678+0000","last_unstale":"2024-03-11T21:40:31.955199+0000","last_undegraded":"2024-03-11T21:40:31.955199+0000","last_fullsized":"2024-03-11T21:40:31.955199+0000","mapping_epoch":277,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":278,"parent":"0.0","parent_split_bits":0,"last_scrub":"260'7","last_scrub_stamp":"2024-03-11T21:40:29.597219+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:39.525554+0000","last_clean_scrub_stamp":"2024-03-11T21:40:29.597219+0000","objects_scrubbed":2,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:41:33.394843+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":14,"num_read_kb":10,"num_write":8,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,1],"acting":[4,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":619,"reported_epoch":354,"state":"active+clean","last_fresh":"2024-03-11T21:40:33.391419+0000","last_change":"2024-03-11T21:35:39.691825+0000","last_active":"2024-03-11T21:40:33.391419+0000","last_peered":"2024-03-11T21:40:33.391419+0000","last_clean":"2024-03-11T21:40:33.391419+0000","last_became_active":"2024-03-11T21:25:44.589786+0000","last_became_peered":"2024-03-11T21:25:44.589786+0000","last_unstale":"2024-03-11T21:40:33.391419+0000","last_undegraded":"2024-03-11T21:40:33.391419+0000","last_fullsized":"2024-03-11T21:40:33.391419+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:35:39.691690+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:43.559452+0000","last_clean_scrub_stamp":"2024-03-11T21:35:39.691690+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:40:37.147324+0000","scrub_duration":18,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,7],"acting":[0,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":637,"reported_epoch":352,"state":"active+clean","last_fresh":"2024-03-11T21:40:28.556016+0000","last_change":"2024-03-11T21:40:27.604550+0000","last_active":"2024-03-11T21:40:28.556016+0000","last_peered":"2024-03-11T21:40:28.556016+0000","last_clean":"2024-03-11T21:40:28.556016+0000","last_became_active":"2024-03-11T21:34:41.147185+0000","last_became_peered":"2024-03-11T21:34:41.147185+0000","last_unstale":"2024-03-11T21:40:28.556016+0000","last_undegraded":"2024-03-11T21:40:28.556016+0000","last_fullsized":"2024-03-11T21:40:28.556016+0000","mapping_epoch":226,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":227,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:40:27.604453+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:32:21.535034+0000","last_clean_scrub_stamp":"2024-03-11T21:40:27.604453+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:41:27.648926+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,7],"acting":[5,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1c","version":"0'0","reported_seq":633,"reported_epoch":352,"state":"active+clean","last_fresh":"2024-03-11T21:40:28.457933+0000","last_change":"2024-03-11T21:35:28.789693+0000","last_active":"2024-03-11T21:40:28.457933+0000","last_peered":"2024-03-11T21:40:28.457933+0000","last_clean":"2024-03-11T21:40:28.457933+0000","last_became_active":"2024-03-11T21:25:38.541364+0000","last_became_peered":"2024-03-11T21:25:38.541364+0000","last_unstale":"2024-03-11T21:40:28.457933+0000","last_undegraded":"2024-03-11T21:40:28.457933+0000","last_fullsized":"2024-03-11T21:40:28.457933+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:35:28.789592+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:28:32.566437+0000","last_clean_scrub_stamp":"2024-03-11T21:35:28.789592+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.1a","version":"350'19","reported_seq":679,"reported_epoch":353,"state":"active+clean","last_fresh":"2024-03-11T21:40:31.954983+0000","last_change":"2024-03-11T21:40:20.484449+0000","last_active":"2024-03-11T21:40:31.954983+0000","last_peered":"2024-03-11T21:40:31.954983+0000","last_clean":"2024-03-11T21:40:31.954983+0000","last_became_active":"2024-03-11T21:36:35.057029+0000","last_became_peered":"2024-03-11T21:36:35.057029+0000","last_unstale":"2024-03-11T21:40:31.954983+0000","last_undegraded":"2024-03-11T21:40:31.954983+0000","last_fullsized":"2024-03-11T21:40:31.954983+0000","mapping_epoch":277,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":278,"parent":"0.0","parent_split_bits":0,"last_scrub":"350'19","last_scrub_stamp":"2024-03-11T21:40:20.484308+0000","last_deep_scrub":"350'19","last_deep_scrub_stamp":"2024-03-11T21:40:20.484308+0000","last_clean_scrub_stamp":"2024-03-11T21:40:20.484308+0000","objects_scrubbed":4,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:41:24.916676+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0],"acting":[4,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1f","version":"0'0","reported_seq":494,"reported_epoch":354,"state":"active+clean","last_fresh":"2024-03-11T21:40:33.758537+0000","last_change":"2024-03-11T21:36:35.074675+0000","last_active":"2024-03-11T21:40:33.758537+0000","last_peered":"2024-03-11T21:40:33.758537+0000","last_clean":"2024-03-11T21:40:33.758537+0000","last_became_active":"2024-03-11T21:36:35.071755+0000","last_became_peered":"2024-03-11T21:36:35.071755+0000","last_unstale":"2024-03-11T21:40:33.758537+0000","last_undegraded":"2024-03-11T21:40:33.758537+0000","last_fullsized":"2024-03-11T21:40:33.758537+0000","mapping_epoch":277,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":278,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:34:05.862891+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:43.559452+0000","last_clean_scrub_stamp":"2024-03-11T21:34:05.862891+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1c","version":"0'0","reported_seq":629,"reported_epoch":353,"state":"active+clean","last_fresh":"2024-03-11T21:40:31.955505+0000","last_change":"2024-03-11T21:40:07.678642+0000","last_active":"2024-03-11T21:40:31.955505+0000","last_peered":"2024-03-11T21:40:31.955505+0000","last_clean":"2024-03-11T21:40:31.955505+0000","last_became_active":"2024-03-11T21:36:35.067127+0000","last_became_peered":"2024-03-11T21:36:35.067127+0000","last_unstale":"2024-03-11T21:40:31.955505+0000","last_undegraded":"2024-03-11T21:40:31.955505+0000","last_fullsized":"2024-03-11T21:40:31.955505+0000","mapping_epoch":277,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":278,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:40:07.678561+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:40:07.678561+0000","last_clean_scrub_stamp":"2024-03-11T21:40:07.678561+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:41:27.033148+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1b","version":"0'0","reported_seq":543,"reported_epoch":354,"state":"active+clean","last_fresh":"2024-03-11T21:40:33.758492+0000","last_change":"2024-03-11T21:36:35.075385+0000","last_active":"2024-03-11T21:40:33.758492+0000","last_peered":"2024-03-11T21:40:33.758492+0000","last_clean":"2024-03-11T21:40:33.758492+0000","last_became_active":"2024-03-11T21:36:35.073959+0000","last_became_peered":"2024-03-11T21:36:35.073959+0000","last_unstale":"2024-03-11T21:40:33.758492+0000","last_undegraded":"2024-03-11T21:40:33.758492+0000","last_fullsized":"2024-03-11T21:40:33.758492+0000","mapping_epoch":277,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":278,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:34:27.998724+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:34:27.998724+0000","last_clean_scrub_stamp":"2024-03-11T21:34:27.998724+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1d","version":"260'18","reported_seq":587,"reported_epoch":354,"state":"active+clean","last_fresh":"2024-03-11T21:40:33.758187+0000","last_change":"2024-03-11T21:36:35.080207+0000","last_active":"2024-03-11T21:40:33.758187+0000","last_peered":"2024-03-11T21:40:33.758187+0000","last_clean":"2024-03-11T21:40:33.758187+0000","last_became_active":"2024-03-11T21:36:35.055880+0000","last_became_peered":"2024-03-11T21:36:35.055880+0000","last_unstale":"2024-03-11T21:40:33.758187+0000","last_undegraded":"2024-03-11T21:40:33.758187+0000","last_fullsized":"2024-03-11T21:40:33.758187+0000","mapping_epoch":277,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":278,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'12","last_scrub_stamp":"2024-03-11T21:35:07.971581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:39.525554+0000","last_clean_scrub_stamp":"2024-03-11T21:35:07.971581+0000","objects_scrubbed":0,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":33,"num_read_kb":24,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.1e","version":"0'0","reported_seq":610,"reported_epoch":353,"state":"active+clean","last_fresh":"2024-03-11T21:40:31.955584+0000","last_change":"2024-03-11T21:35:32.664451+0000","last_active":"2024-03-11T21:40:31.955584+0000","last_peered":"2024-03-11T21:40:31.955584+0000","last_clean":"2024-03-11T21:40:31.955584+0000","last_became_active":"2024-03-11T21:25:45.300355+0000","last_became_peered":"2024-03-11T21:25:45.300355+0000","last_unstale":"2024-03-11T21:40:31.955584+0000","last_undegraded":"2024-03-11T21:40:31.955584+0000","last_fullsized":"2024-03-11T21:40:31.955584+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:35:32.664371+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:43.559452+0000","last_clean_scrub_stamp":"2024-03-11T21:35:32.664371+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5],"acting":[4,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1d","version":"0'0","reported_seq":620,"reported_epoch":353,"state":"active+clean","last_fresh":"2024-03-11T21:40:31.950969+0000","last_change":"2024-03-11T21:36:07.624196+0000","last_active":"2024-03-11T21:40:31.950969+0000","last_peered":"2024-03-11T21:40:31.950969+0000","last_clean":"2024-03-11T21:40:31.950969+0000","last_became_active":"2024-03-11T21:25:42.579407+0000","last_became_peered":"2024-03-11T21:25:42.579407+0000","last_unstale":"2024-03-11T21:40:31.950969+0000","last_undegraded":"2024-03-11T21:40:31.950969+0000","last_fullsized":"2024-03-11T21:40:31.950969+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:36:07.624075+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:30:18.678531+0000","last_clean_scrub_stamp":"2024-03-11T21:36:07.624075+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1a","version":"0'0","reported_seq":654,"reported_epoch":352,"state":"active+clean","last_fresh":"2024-03-11T21:40:28.556492+0000","last_change":"2024-03-11T21:39:53.299021+0000","last_active":"2024-03-11T21:40:28.556492+0000","last_peered":"2024-03-11T21:40:28.556492+0000","last_clean":"2024-03-11T21:40:28.556492+0000","last_became_active":"2024-03-11T21:34:41.145014+0000","last_became_peered":"2024-03-11T21:34:41.145014+0000","last_unstale":"2024-03-11T21:40:28.556492+0000","last_undegraded":"2024-03-11T21:40:28.556492+0000","last_fullsized":"2024-03-11T21:40:28.556492+0000","mapping_epoch":226,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":227,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:39:53.298917+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:37.512018+0000","last_clean_scrub_stamp":"2024-03-11T21:39:53.298917+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:41:21.887727+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.1c","version":"260'27","reported_seq":687,"reported_epoch":352,"state":"active+clean","last_fresh":"2024-03-11T21:40:28.388577+0000","last_change":"2024-03-11T21:40:15.885360+0000","last_active":"2024-03-11T21:40:28.388577+0000","last_peered":"2024-03-11T21:40:28.388577+0000","last_clean":"2024-03-11T21:40:28.388577+0000","last_became_active":"2024-03-11T21:36:35.074579+0000","last_became_peered":"2024-03-11T21:36:35.074579+0000","last_unstale":"2024-03-11T21:40:28.388577+0000","last_undegraded":"2024-03-11T21:40:28.388577+0000","last_fullsized":"2024-03-11T21:40:28.388577+0000","mapping_epoch":277,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":278,"parent":"0.0","parent_split_bits":0,"last_scrub":"260'27","last_scrub_stamp":"2024-03-11T21:40:15.885265+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:39.525554+0000","last_clean_scrub_stamp":"2024-03-11T21:40:15.885265+0000","objects_scrubbed":7,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:41:39.688604+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":44,"num_read_kb":35,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.1d","version":"0'0","reported_seq":608,"reported_epoch":353,"state":"active+clean","last_fresh":"2024-03-11T21:40:31.950558+0000","last_change":"2024-03-11T21:35:21.744646+0000","last_active":"2024-03-11T21:40:31.950558+0000","last_peered":"2024-03-11T21:40:31.950558+0000","last_clean":"2024-03-11T21:40:31.950558+0000","last_became_active":"2024-03-11T21:25:44.588845+0000","last_became_peered":"2024-03-11T21:25:44.588845+0000","last_unstale":"2024-03-11T21:40:31.950558+0000","last_undegraded":"2024-03-11T21:40:31.950558+0000","last_fullsized":"2024-03-11T21:40:31.950558+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:35:21.744223+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:43.559452+0000","last_clean_scrub_stamp":"2024-03-11T21:35:21.744223+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":99,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.1e","version":"0'0","reported_seq":632,"reported_epoch":354,"state":"active+clean","last_fresh":"2024-03-11T21:40:33.390932+0000","last_change":"2024-03-11T21:40:28.163223+0000","last_active":"2024-03-11T21:40:33.390932+0000","last_peered":"2024-03-11T21:40:33.390932+0000","last_clean":"2024-03-11T21:40:33.390932+0000","last_became_active":"2024-03-11T21:25:42.582959+0000","last_became_peered":"2024-03-11T21:25:42.582959+0000","last_unstale":"2024-03-11T21:40:33.390932+0000","last_undegraded":"2024-03-11T21:40:33.390932+0000","last_fullsized":"2024-03-11T21:40:33.390932+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:40:28.163142+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:26:58.335431+0000","last_clean_scrub_stamp":"2024-03-11T21:40:28.163142+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:41:55.681758+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.19","version":"67'2","reported_seq":569,"reported_epoch":354,"state":"active+clean","last_fresh":"2024-03-11T21:40:33.757758+0000","last_change":"2024-03-11T21:36:35.069729+0000","last_active":"2024-03-11T21:40:33.757758+0000","last_peered":"2024-03-11T21:40:33.757758+0000","last_clean":"2024-03-11T21:40:33.757758+0000","last_became_active":"2024-03-11T21:36:35.064249+0000","last_became_peered":"2024-03-11T21:36:35.064249+0000","last_unstale":"2024-03-11T21:40:33.757758+0000","last_undegraded":"2024-03-11T21:40:33.757758+0000","last_fullsized":"2024-03-11T21:40:33.757758+0000","mapping_epoch":277,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":278,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'2","last_scrub_stamp":"2024-03-11T21:32:15.509542+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:37.512018+0000","last_clean_scrub_stamp":"2024-03-11T21:32:15.509542+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":957,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,0],"acting":[3,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1f","version":"260'24","reported_seq":702,"reported_epoch":352,"state":"active+clean","last_fresh":"2024-03-11T21:40:28.557383+0000","last_change":"2024-03-11T21:40:25.079945+0000","last_active":"2024-03-11T21:40:28.557383+0000","last_peered":"2024-03-11T21:40:28.557383+0000","last_clean":"2024-03-11T21:40:28.557383+0000","last_became_active":"2024-03-11T21:32:22.798242+0000","last_became_peered":"2024-03-11T21:32:22.798242+0000","last_unstale":"2024-03-11T21:40:28.557383+0000","last_undegraded":"2024-03-11T21:40:28.557383+0000","last_fullsized":"2024-03-11T21:40:28.557383+0000","mapping_epoch":166,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":167,"parent":"0.0","parent_split_bits":0,"last_scrub":"260'24","last_scrub_stamp":"2024-03-11T21:40:25.079808+0000","last_deep_scrub":"124'16","last_deep_scrub_stamp":"2024-03-11T21:35:38.173841+0000","last_clean_scrub_stamp":"2024-03-11T21:40:25.079808+0000","objects_scrubbed":5,"log_size":24,"log_dups_size":0,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:41:47.993849+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":32,"num_read_kb":25,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,0],"acting":[6,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.1c","version":"74'1","reported_seq":275,"reported_epoch":352,"state":"active+clean","last_fresh":"2024-03-11T21:40:28.457843+0000","last_change":"2024-03-11T21:36:11.732043+0000","last_active":"2024-03-11T21:40:28.457843+0000","last_peered":"2024-03-11T21:40:28.457843+0000","last_clean":"2024-03-11T21:40:28.457843+0000","last_became_active":"2024-03-11T21:34:41.144839+0000","last_became_peered":"2024-03-11T21:34:41.144839+0000","last_unstale":"2024-03-11T21:40:28.457843+0000","last_undegraded":"2024-03-11T21:40:28.457843+0000","last_fullsized":"2024-03-11T21:40:28.457843+0000","mapping_epoch":226,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":227,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'1","last_scrub_stamp":"2024-03-11T21:36:11.731914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:43.559452+0000","last_clean_scrub_stamp":"2024-03-11T21:36:11.731914+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":370,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":370,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.1f","version":"229'10","reported_seq":854,"reported_epoch":352,"state":"active+clean","last_fresh":"2024-03-11T21:40:28.557374+0000","last_change":"2024-03-11T21:39:42.162545+0000","last_active":"2024-03-11T21:40:28.557374+0000","last_peered":"2024-03-11T21:40:28.557374+0000","last_clean":"2024-03-11T21:40:28.557374+0000","last_became_active":"2024-03-11T21:34:41.145862+0000","last_became_peered":"2024-03-11T21:34:41.145862+0000","last_unstale":"2024-03-11T21:40:28.557374+0000","last_undegraded":"2024-03-11T21:40:28.557374+0000","last_fullsized":"2024-03-11T21:40:28.557374+0000","mapping_epoch":226,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":227,"parent":"0.0","parent_split_bits":0,"last_scrub":"229'10","last_scrub_stamp":"2024-03-11T21:39:42.162463+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:41.544550+0000","last_clean_scrub_stamp":"2024-03-11T21:39:42.162463+0000","objects_scrubbed":1,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:41:06.487514+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.18","version":"67'1","reported_seq":670,"reported_epoch":352,"state":"active+clean","last_fresh":"2024-03-11T21:40:28.555588+0000","last_change":"2024-03-11T21:36:10.458150+0000","last_active":"2024-03-11T21:40:28.555588+0000","last_peered":"2024-03-11T21:40:28.555588+0000","last_clean":"2024-03-11T21:40:28.555588+0000","last_became_active":"2024-03-11T21:35:23.310241+0000","last_became_peered":"2024-03-11T21:35:23.310241+0000","last_unstale":"2024-03-11T21:40:28.555588+0000","last_undegraded":"2024-03-11T21:40:28.555588+0000","last_fullsized":"2024-03-11T21:40:28.555588+0000","mapping_epoch":248,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":249,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'1","last_scrub_stamp":"2024-03-11T21:36:10.458028+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:37.512018+0000","last_clean_scrub_stamp":"2024-03-11T21:36:10.458028+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":436,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":7,"num_read_kb":7,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":1308,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,2],"acting":[5,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1e","version":"260'20","reported_seq":716,"reported_epoch":354,"state":"active+clean","last_fresh":"2024-03-11T21:40:33.390817+0000","last_change":"2024-03-11T21:36:35.083447+0000","last_active":"2024-03-11T21:40:33.390817+0000","last_peered":"2024-03-11T21:40:33.390817+0000","last_clean":"2024-03-11T21:40:33.390817+0000","last_became_active":"2024-03-11T21:36:35.057843+0000","last_became_peered":"2024-03-11T21:36:35.057843+0000","last_unstale":"2024-03-11T21:40:33.390817+0000","last_undegraded":"2024-03-11T21:40:33.390817+0000","last_fullsized":"2024-03-11T21:40:33.390817+0000","mapping_epoch":277,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":278,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'10","last_scrub_stamp":"2024-03-11T21:35:16.825146+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:39.525554+0000","last_clean_scrub_stamp":"2024-03-11T21:35:16.825146+0000","objects_scrubbed":5,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:40:37.147236+0000","scrub_duration":44,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":25,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.13","version":"0'0","reported_seq":502,"reported_epoch":354,"state":"active+clean","last_fresh":"2024-03-11T21:40:33.758632+0000","last_change":"2024-03-11T21:36:35.063744+0000","last_active":"2024-03-11T21:40:33.758632+0000","last_peered":"2024-03-11T21:40:33.758632+0000","last_clean":"2024-03-11T21:40:33.758632+0000","last_became_active":"2024-03-11T21:36:35.062618+0000","last_became_peered":"2024-03-11T21:36:35.062618+0000","last_unstale":"2024-03-11T21:40:33.758632+0000","last_undegraded":"2024-03-11T21:40:33.758632+0000","last_fullsized":"2024-03-11T21:40:33.758632+0000","mapping_epoch":277,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":278,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:34:13.822803+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:43.559452+0000","last_clean_scrub_stamp":"2024-03-11T21:34:13.822803+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":279,"reported_epoch":352,"state":"active+clean","last_fresh":"2024-03-11T21:40:29.084155+0000","last_change":"2024-03-11T21:40:29.084178+0000","last_active":"2024-03-11T21:40:29.084155+0000","last_peered":"2024-03-11T21:40:29.084155+0000","last_clean":"2024-03-11T21:40:29.084155+0000","last_became_active":"2024-03-11T21:34:41.142569+0000","last_became_peered":"2024-03-11T21:34:41.142569+0000","last_unstale":"2024-03-11T21:40:29.084155+0000","last_undegraded":"2024-03-11T21:40:29.084155+0000","last_fullsized":"2024-03-11T21:40:29.084155+0000","mapping_epoch":226,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":227,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:40:29.084062+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:40:29.084062+0000","last_clean_scrub_stamp":"2024-03-11T21:40:29.084062+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:41:31.066228+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.17","version":"0'0","reported_seq":643,"reported_epoch":352,"state":"active+clean","last_fresh":"2024-03-11T21:40:28.557196+0000","last_change":"2024-03-11T21:40:18.112951+0000","last_active":"2024-03-11T21:40:28.557196+0000","last_peered":"2024-03-11T21:40:28.557196+0000","last_clean":"2024-03-11T21:40:28.557196+0000","last_became_active":"2024-03-11T21:25:38.540349+0000","last_became_peered":"2024-03-11T21:25:38.540349+0000","last_unstale":"2024-03-11T21:40:28.557196+0000","last_undegraded":"2024-03-11T21:40:28.557196+0000","last_fullsized":"2024-03-11T21:40:28.557196+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:40:18.112829+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:28:25.918101+0000","last_clean_scrub_stamp":"2024-03-11T21:40:18.112829+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:41:20.570429+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,2],"acting":[6,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.11","version":"260'19","reported_seq":628,"reported_epoch":354,"state":"active+clean","last_fresh":"2024-03-11T21:40:33.758055+0000","last_change":"2024-03-11T21:36:35.096720+0000","last_active":"2024-03-11T21:40:33.758055+0000","last_peered":"2024-03-11T21:40:33.758055+0000","last_clean":"2024-03-11T21:40:33.758055+0000","last_became_active":"2024-03-11T21:36:35.073689+0000","last_became_peered":"2024-03-11T21:36:35.073689+0000","last_unstale":"2024-03-11T21:40:33.758055+0000","last_undegraded":"2024-03-11T21:40:33.758055+0000","last_fullsized":"2024-03-11T21:40:33.758055+0000","mapping_epoch":277,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":278,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'11","last_scrub_stamp":"2024-03-11T21:35:16.044397+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:39.525554+0000","last_clean_scrub_stamp":"2024-03-11T21:35:16.044397+0000","objects_scrubbed":5,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":31,"num_read_kb":24,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":271,"reported_epoch":352,"state":"active+clean","last_fresh":"2024-03-11T21:40:28.457378+0000","last_change":"2024-03-11T21:35:32.719869+0000","last_active":"2024-03-11T21:40:28.457378+0000","last_peered":"2024-03-11T21:40:28.457378+0000","last_clean":"2024-03-11T21:40:28.457378+0000","last_became_active":"2024-03-11T21:34:41.137235+0000","last_became_peered":"2024-03-11T21:34:41.137235+0000","last_unstale":"2024-03-11T21:40:28.457378+0000","last_undegraded":"2024-03-11T21:40:28.457378+0000","last_fullsized":"2024-03-11T21:40:28.457378+0000","mapping_epoch":226,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":227,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:35:32.719789+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:43.559452+0000","last_clean_scrub_stamp":"2024-03-11T21:35:32.719789+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,4],"acting":[7,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":655,"reported_epoch":352,"state":"active+clean","last_fresh":"2024-03-11T21:40:28.556124+0000","last_change":"2024-03-11T21:40:03.181212+0000","last_active":"2024-03-11T21:40:28.556124+0000","last_peered":"2024-03-11T21:40:28.556124+0000","last_clean":"2024-03-11T21:40:28.556124+0000","last_became_active":"2024-03-11T21:34:41.156036+0000","last_became_peered":"2024-03-11T21:34:41.156036+0000","last_unstale":"2024-03-11T21:40:28.556124+0000","last_undegraded":"2024-03-11T21:40:28.556124+0000","last_fullsized":"2024-03-11T21:40:28.556124+0000","mapping_epoch":226,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":227,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:40:03.181074+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:38:51.036736+0000","last_clean_scrub_stamp":"2024-03-11T21:40:03.181074+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:41:28.087147+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.16","version":"0'0","reported_seq":634,"reported_epoch":352,"state":"active+clean","last_fresh":"2024-03-11T21:40:28.556851+0000","last_change":"2024-03-11T21:35:39.108623+0000","last_active":"2024-03-11T21:40:28.556851+0000","last_peered":"2024-03-11T21:40:28.556851+0000","last_clean":"2024-03-11T21:40:28.556851+0000","last_became_active":"2024-03-11T21:25:38.539103+0000","last_became_peered":"2024-03-11T21:25:38.539103+0000","last_unstale":"2024-03-11T21:40:28.556851+0000","last_undegraded":"2024-03-11T21:40:28.556851+0000","last_fullsized":"2024-03-11T21:40:28.556851+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:35:39.108546+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:27:11.938909+0000","last_clean_scrub_stamp":"2024-03-11T21:35:39.108546+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,2],"acting":[5,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.10","version":"260'10","reported_seq":510,"reported_epoch":354,"state":"active+clean","last_fresh":"2024-03-11T21:40:33.757804+0000","last_change":"2024-03-11T21:36:35.084268+0000","last_active":"2024-03-11T21:40:33.757804+0000","last_peered":"2024-03-11T21:40:33.757804+0000","last_clean":"2024-03-11T21:40:33.757804+0000","last_became_active":"2024-03-11T21:36:35.067500+0000","last_became_peered":"2024-03-11T21:36:35.067500+0000","last_unstale":"2024-03-11T21:40:33.757804+0000","last_undegraded":"2024-03-11T21:40:33.757804+0000","last_fullsized":"2024-03-11T21:40:33.757804+0000","mapping_epoch":277,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":278,"parent":"0.0","parent_split_bits":0,"last_scrub":"94'6","last_scrub_stamp":"2024-03-11T21:34:00.727416+0000","last_deep_scrub":"94'6","last_deep_scrub_stamp":"2024-03-11T21:28:24.603974+0000","last_clean_scrub_stamp":"2024-03-11T21:34:00.727416+0000","objects_scrubbed":2,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":12,"num_read_kb":10,"num_write":8,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6],"acting":[3,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":555,"reported_epoch":354,"state":"active+clean","last_fresh":"2024-03-11T21:40:33.757937+0000","last_change":"2024-03-11T21:36:35.070468+0000","last_active":"2024-03-11T21:40:33.757937+0000","last_peered":"2024-03-11T21:40:33.757937+0000","last_clean":"2024-03-11T21:40:33.757937+0000","last_became_active":"2024-03-11T21:36:35.070176+0000","last_became_peered":"2024-03-11T21:36:35.070176+0000","last_unstale":"2024-03-11T21:40:33.757937+0000","last_undegraded":"2024-03-11T21:40:33.757937+0000","last_fullsized":"2024-03-11T21:40:33.757937+0000","mapping_epoch":277,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":278,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:35:18.062272+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:43.559452+0000","last_clean_scrub_stamp":"2024-03-11T21:35:18.062272+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":622,"reported_epoch":353,"state":"active+clean","last_fresh":"2024-03-11T21:40:31.950757+0000","last_change":"2024-03-11T21:36:35.058551+0000","last_active":"2024-03-11T21:40:31.950757+0000","last_peered":"2024-03-11T21:40:31.950757+0000","last_clean":"2024-03-11T21:40:31.950757+0000","last_became_active":"2024-03-11T21:36:35.057906+0000","last_became_peered":"2024-03-11T21:36:35.057906+0000","last_unstale":"2024-03-11T21:40:31.950757+0000","last_undegraded":"2024-03-11T21:40:31.950757+0000","last_fullsized":"2024-03-11T21:40:31.950757+0000","mapping_epoch":277,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":278,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:34:15.516278+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:27:21.207303+0000","last_clean_scrub_stamp":"2024-03-11T21:34:15.516278+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.15","version":"0'0","reported_seq":629,"reported_epoch":353,"state":"active+clean","last_fresh":"2024-03-11T21:40:31.950791+0000","last_change":"2024-03-11T21:36:06.579754+0000","last_active":"2024-03-11T21:40:31.950791+0000","last_peered":"2024-03-11T21:40:31.950791+0000","last_clean":"2024-03-11T21:40:31.950791+0000","last_became_active":"2024-03-11T21:25:38.541705+0000","last_became_peered":"2024-03-11T21:25:38.541705+0000","last_unstale":"2024-03-11T21:40:31.950791+0000","last_undegraded":"2024-03-11T21:40:31.950791+0000","last_fullsized":"2024-03-11T21:40:31.950791+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:36:06.579636+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:26:44.860499+0000","last_clean_scrub_stamp":"2024-03-11T21:36:06.579636+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.13","version":"260'24","reported_seq":687,"reported_epoch":353,"state":"active+clean","last_fresh":"2024-03-11T21:40:31.954981+0000","last_change":"2024-03-11T21:40:25.595320+0000","last_active":"2024-03-11T21:40:31.954981+0000","last_peered":"2024-03-11T21:40:31.954981+0000","last_clean":"2024-03-11T21:40:31.954981+0000","last_became_active":"2024-03-11T21:26:11.214879+0000","last_became_peered":"2024-03-11T21:26:11.214879+0000","last_unstale":"2024-03-11T21:40:31.954981+0000","last_undegraded":"2024-03-11T21:40:31.954981+0000","last_fullsized":"2024-03-11T21:40:31.954981+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"260'24","last_scrub_stamp":"2024-03-11T21:40:25.595220+0000","last_deep_scrub":"260'24","last_deep_scrub_stamp":"2024-03-11T21:40:25.595220+0000","last_clean_scrub_stamp":"2024-03-11T21:40:25.595220+0000","objects_scrubbed":5,"log_size":24,"log_dups_size":0,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:41:39.345264+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":32,"num_read_kb":25,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,2],"acting":[4,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.10","version":"0'0","reported_seq":619,"reported_epoch":354,"state":"active+clean","last_fresh":"2024-03-11T21:40:33.389349+0000","last_change":"2024-03-11T21:35:29.712658+0000","last_active":"2024-03-11T21:40:33.389349+0000","last_peered":"2024-03-11T21:40:33.389349+0000","last_clean":"2024-03-11T21:40:33.389349+0000","last_became_active":"2024-03-11T21:25:44.591189+0000","last_became_peered":"2024-03-11T21:25:44.591189+0000","last_unstale":"2024-03-11T21:40:33.389349+0000","last_undegraded":"2024-03-11T21:40:33.389349+0000","last_fullsized":"2024-03-11T21:40:33.389349+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:35:29.712557+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:27:25.417450+0000","last_clean_scrub_stamp":"2024-03-11T21:35:29.712557+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:40:37.147340+0000","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,1],"acting":[0,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":553,"reported_epoch":354,"state":"active+clean","last_fresh":"2024-03-11T21:40:33.758653+0000","last_change":"2024-03-11T21:36:35.062276+0000","last_active":"2024-03-11T21:40:33.758653+0000","last_peered":"2024-03-11T21:40:33.758653+0000","last_clean":"2024-03-11T21:40:33.758653+0000","last_became_active":"2024-03-11T21:36:35.056619+0000","last_became_peered":"2024-03-11T21:36:35.056619+0000","last_unstale":"2024-03-11T21:40:33.758653+0000","last_undegraded":"2024-03-11T21:40:33.758653+0000","last_fullsized":"2024-03-11T21:40:33.758653+0000","mapping_epoch":277,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":278,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:35:13.060487+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:31:15.367898+0000","last_clean_scrub_stamp":"2024-03-11T21:35:13.060487+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.14","version":"0'0","reported_seq":660,"reported_epoch":352,"state":"active+clean","last_fresh":"2024-03-11T21:40:28.557123+0000","last_change":"2024-03-11T21:40:01.212243+0000","last_active":"2024-03-11T21:40:28.557123+0000","last_peered":"2024-03-11T21:40:28.557123+0000","last_clean":"2024-03-11T21:40:28.557123+0000","last_became_active":"2024-03-11T21:36:35.068038+0000","last_became_peered":"2024-03-11T21:36:35.068038+0000","last_unstale":"2024-03-11T21:40:28.557123+0000","last_undegraded":"2024-03-11T21:40:28.557123+0000","last_fullsized":"2024-03-11T21:40:28.557123+0000","mapping_epoch":277,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":278,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:40:01.212145+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:37.512018+0000","last_clean_scrub_stamp":"2024-03-11T21:40:01.212145+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:41:09.997136+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,5],"acting":[6,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.12","version":"260'15","reported_seq":650,"reported_epoch":353,"state":"active+clean","last_fresh":"2024-03-11T21:40:31.950602+0000","last_change":"2024-03-11T21:35:32.590439+0000","last_active":"2024-03-11T21:40:31.950602+0000","last_peered":"2024-03-11T21:40:31.950602+0000","last_clean":"2024-03-11T21:40:31.950602+0000","last_became_active":"2024-03-11T21:25:40.547082+0000","last_became_peered":"2024-03-11T21:25:40.547082+0000","last_unstale":"2024-03-11T21:40:31.950602+0000","last_undegraded":"2024-03-11T21:40:31.950602+0000","last_fullsized":"2024-03-11T21:40:31.950602+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'9","last_scrub_stamp":"2024-03-11T21:35:32.590318+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:39.525554+0000","last_clean_scrub_stamp":"2024-03-11T21:35:32.590318+0000","objects_scrubbed":4,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":617,"reported_epoch":353,"state":"active+clean","last_fresh":"2024-03-11T21:40:31.955845+0000","last_change":"2024-03-11T21:36:04.754168+0000","last_active":"2024-03-11T21:40:31.955845+0000","last_peered":"2024-03-11T21:40:31.955845+0000","last_clean":"2024-03-11T21:40:31.955845+0000","last_became_active":"2024-03-11T21:25:44.589586+0000","last_became_peered":"2024-03-11T21:25:44.589586+0000","last_unstale":"2024-03-11T21:40:31.955845+0000","last_undegraded":"2024-03-11T21:40:31.955845+0000","last_fullsized":"2024-03-11T21:40:31.955845+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:36:04.753979+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:27:09.801138+0000","last_clean_scrub_stamp":"2024-03-11T21:36:04.753979+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":113,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,5],"acting":[4,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.14","version":"280'12","reported_seq":763,"reported_epoch":354,"state":"active+clean","last_fresh":"2024-03-11T21:40:33.759144+0000","last_change":"2024-03-11T21:36:35.074550+0000","last_active":"2024-03-11T21:40:33.759144+0000","last_peered":"2024-03-11T21:40:33.759144+0000","last_clean":"2024-03-11T21:40:33.759144+0000","last_became_active":"2024-03-11T21:36:35.073912+0000","last_became_peered":"2024-03-11T21:36:35.073912+0000","last_unstale":"2024-03-11T21:40:33.759144+0000","last_undegraded":"2024-03-11T21:40:33.759144+0000","last_fullsized":"2024-03-11T21:40:33.759144+0000","mapping_epoch":277,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":278,"parent":"0.0","parent_split_bits":0,"last_scrub":"229'10","last_scrub_stamp":"2024-03-11T21:35:26.182107+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:41.544550+0000","last_clean_scrub_stamp":"2024-03-11T21:35:26.182107+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,2],"acting":[3,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.13","version":"0'0","reported_seq":640,"reported_epoch":354,"state":"active+clean","last_fresh":"2024-03-11T21:40:33.391485+0000","last_change":"2024-03-11T21:40:11.027382+0000","last_active":"2024-03-11T21:40:33.391485+0000","last_peered":"2024-03-11T21:40:33.391485+0000","last_clean":"2024-03-11T21:40:33.391485+0000","last_became_active":"2024-03-11T21:36:35.057132+0000","last_became_peered":"2024-03-11T21:36:35.057132+0000","last_unstale":"2024-03-11T21:40:33.391485+0000","last_undegraded":"2024-03-11T21:40:33.391485+0000","last_fullsized":"2024-03-11T21:40:33.391485+0000","mapping_epoch":277,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":278,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:40:11.027268+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:37.512018+0000","last_clean_scrub_stamp":"2024-03-11T21:40:11.027268+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:41:40.995455+0000","scrub_duration":24,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.15","version":"260'19","reported_seq":701,"reported_epoch":352,"state":"active+clean","last_fresh":"2024-03-11T21:40:28.556598+0000","last_change":"2024-03-11T21:36:35.088986+0000","last_active":"2024-03-11T21:40:28.556598+0000","last_peered":"2024-03-11T21:40:28.556598+0000","last_clean":"2024-03-11T21:40:28.556598+0000","last_became_active":"2024-03-11T21:36:35.057285+0000","last_became_peered":"2024-03-11T21:36:35.057285+0000","last_unstale":"2024-03-11T21:40:28.556598+0000","last_undegraded":"2024-03-11T21:40:28.556598+0000","last_fullsized":"2024-03-11T21:40:28.556598+0000","mapping_epoch":277,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":278,"parent":"0.0","parent_split_bits":0,"last_scrub":"94'13","last_scrub_stamp":"2024-03-11T21:35:29.178543+0000","last_deep_scrub":"94'13","last_deep_scrub_stamp":"2024-03-11T21:29:41.853158+0000","last_clean_scrub_stamp":"2024-03-11T21:35:29.178543+0000","objects_scrubbed":4,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":11,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.16","version":"0'0","reported_seq":648,"reported_epoch":354,"state":"active+clean","last_fresh":"2024-03-11T21:40:33.390423+0000","last_change":"2024-03-11T21:40:30.249810+0000","last_active":"2024-03-11T21:40:33.390423+0000","last_peered":"2024-03-11T21:40:33.390423+0000","last_clean":"2024-03-11T21:40:33.390423+0000","last_became_active":"2024-03-11T21:36:35.057355+0000","last_became_peered":"2024-03-11T21:36:35.057355+0000","last_unstale":"2024-03-11T21:40:33.390423+0000","last_undegraded":"2024-03-11T21:40:33.390423+0000","last_fullsized":"2024-03-11T21:40:33.390423+0000","mapping_epoch":277,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":278,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:40:30.249679+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:40:30.249679+0000","last_clean_scrub_stamp":"2024-03-11T21:40:30.249679+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:41:59.927156+0000","scrub_duration":40,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.15","version":"98'9","reported_seq":806,"reported_epoch":352,"state":"active+clean","last_fresh":"2024-03-11T21:40:28.556867+0000","last_change":"2024-03-11T21:36:03.366581+0000","last_active":"2024-03-11T21:40:28.556867+0000","last_peered":"2024-03-11T21:40:28.556867+0000","last_clean":"2024-03-11T21:40:28.556867+0000","last_became_active":"2024-03-11T21:25:42.572555+0000","last_became_peered":"2024-03-11T21:25:42.572555+0000","last_unstale":"2024-03-11T21:40:28.556867+0000","last_undegraded":"2024-03-11T21:40:28.556867+0000","last_fullsized":"2024-03-11T21:40:28.556867+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"98'9","last_scrub_stamp":"2024-03-11T21:36:03.366450+0000","last_deep_scrub":"98'9","last_deep_scrub_stamp":"2024-03-11T21:31:20.624923+0000","last_clean_scrub_stamp":"2024-03-11T21:36:03.366450+0000","objects_scrubbed":1,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.12","version":"0'0","reported_seq":647,"reported_epoch":352,"state":"active+clean","last_fresh":"2024-03-11T21:40:28.556177+0000","last_change":"2024-03-11T21:36:35.058369+0000","last_active":"2024-03-11T21:40:28.556177+0000","last_peered":"2024-03-11T21:40:28.556177+0000","last_clean":"2024-03-11T21:40:28.556177+0000","last_became_active":"2024-03-11T21:36:35.058033+0000","last_became_peered":"2024-03-11T21:36:35.058033+0000","last_unstale":"2024-03-11T21:40:28.556177+0000","last_undegraded":"2024-03-11T21:40:28.556177+0000","last_fullsized":"2024-03-11T21:40:28.556177+0000","mapping_epoch":277,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":278,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:35:27.158468+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:37.512018+0000","last_clean_scrub_stamp":"2024-03-11T21:35:27.158468+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.14","version":"260'20","reported_seq":636,"reported_epoch":354,"state":"active+clean","last_fresh":"2024-03-11T21:40:33.758823+0000","last_change":"2024-03-11T21:36:35.095382+0000","last_active":"2024-03-11T21:40:33.758823+0000","last_peered":"2024-03-11T21:40:33.758823+0000","last_clean":"2024-03-11T21:40:33.758823+0000","last_became_active":"2024-03-11T21:36:35.069638+0000","last_became_peered":"2024-03-11T21:36:35.069638+0000","last_unstale":"2024-03-11T21:40:33.758823+0000","last_undegraded":"2024-03-11T21:40:33.758823+0000","last_fullsized":"2024-03-11T21:40:33.758823+0000","mapping_epoch":277,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":278,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'10","last_scrub_stamp":"2024-03-11T21:35:28.189296+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:39.525554+0000","last_clean_scrub_stamp":"2024-03-11T21:35:28.189296+0000","objects_scrubbed":0,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":29,"num_read_kb":24,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.15","version":"0'0","reported_seq":271,"reported_epoch":352,"state":"active+clean","last_fresh":"2024-03-11T21:40:28.457938+0000","last_change":"2024-03-11T21:36:14.736240+0000","last_active":"2024-03-11T21:40:28.457938+0000","last_peered":"2024-03-11T21:40:28.457938+0000","last_clean":"2024-03-11T21:40:28.457938+0000","last_became_active":"2024-03-11T21:34:41.142760+0000","last_became_peered":"2024-03-11T21:34:41.142760+0000","last_unstale":"2024-03-11T21:40:28.457938+0000","last_undegraded":"2024-03-11T21:40:28.457938+0000","last_fullsized":"2024-03-11T21:40:28.457938+0000","mapping_epoch":226,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":227,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:36:14.735985+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:43.559452+0000","last_clean_scrub_stamp":"2024-03-11T21:36:14.735985+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":36,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":627,"reported_epoch":352,"state":"active+clean","last_fresh":"2024-03-11T21:40:28.556696+0000","last_change":"2024-03-11T21:40:26.587777+0000","last_active":"2024-03-11T21:40:28.556696+0000","last_peered":"2024-03-11T21:40:28.556696+0000","last_clean":"2024-03-11T21:40:28.556696+0000","last_became_active":"2024-03-11T21:36:35.060342+0000","last_became_peered":"2024-03-11T21:36:35.060342+0000","last_unstale":"2024-03-11T21:40:28.556696+0000","last_undegraded":"2024-03-11T21:40:28.556696+0000","last_fullsized":"2024-03-11T21:40:28.556696+0000","mapping_epoch":277,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":278,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:40:26.587672+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:41.544550+0000","last_clean_scrub_stamp":"2024-03-11T21:40:26.587672+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:41:33.380076+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,1],"acting":[5,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.11","version":"0'0","reported_seq":643,"reported_epoch":352,"state":"active+clean","last_fresh":"2024-03-11T21:40:28.556255+0000","last_change":"2024-03-11T21:40:09.218006+0000","last_active":"2024-03-11T21:40:28.556255+0000","last_peered":"2024-03-11T21:40:28.556255+0000","last_clean":"2024-03-11T21:40:28.556255+0000","last_became_active":"2024-03-11T21:25:38.537415+0000","last_became_peered":"2024-03-11T21:25:38.537415+0000","last_unstale":"2024-03-11T21:40:28.556255+0000","last_undegraded":"2024-03-11T21:40:28.556255+0000","last_fullsized":"2024-03-11T21:40:28.556255+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:40:09.217927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:37.512018+0000","last_clean_scrub_stamp":"2024-03-11T21:40:09.217927+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:41:24.546509+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.17","version":"260'15","reported_seq":581,"reported_epoch":354,"state":"active+clean","last_fresh":"2024-03-11T21:40:33.758637+0000","last_change":"2024-03-11T21:36:35.099451+0000","last_active":"2024-03-11T21:40:33.758637+0000","last_peered":"2024-03-11T21:40:33.758637+0000","last_clean":"2024-03-11T21:40:33.758637+0000","last_became_active":"2024-03-11T21:36:35.071346+0000","last_became_peered":"2024-03-11T21:36:35.071346+0000","last_unstale":"2024-03-11T21:40:33.758637+0000","last_undegraded":"2024-03-11T21:40:33.758637+0000","last_fullsized":"2024-03-11T21:40:33.758637+0000","mapping_epoch":277,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":278,"parent":"0.0","parent_split_bits":0,"last_scrub":"97'9","last_scrub_stamp":"2024-03-11T21:32:08.495352+0000","last_deep_scrub":"97'9","last_deep_scrub_stamp":"2024-03-11T21:27:25.803132+0000","last_clean_scrub_stamp":"2024-03-11T21:32:08.495352+0000","objects_scrubbed":3,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":17,"num_read_kb":14,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,5],"acting":[3,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.14","version":"0'0","reported_seq":625,"reported_epoch":352,"state":"active+clean","last_fresh":"2024-03-11T21:40:28.388375+0000","last_change":"2024-03-11T21:35:27.606664+0000","last_active":"2024-03-11T21:40:28.388375+0000","last_peered":"2024-03-11T21:40:28.388375+0000","last_clean":"2024-03-11T21:40:28.388375+0000","last_became_active":"2024-03-11T21:34:41.136946+0000","last_became_peered":"2024-03-11T21:34:41.136946+0000","last_unstale":"2024-03-11T21:40:28.388375+0000","last_undegraded":"2024-03-11T21:40:28.388375+0000","last_fullsized":"2024-03-11T21:40:28.388375+0000","mapping_epoch":226,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":227,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:35:27.606551+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:43.559452+0000","last_clean_scrub_stamp":"2024-03-11T21:35:27.606551+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,7],"acting":[2,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":535,"reported_epoch":354,"state":"active+clean","last_fresh":"2024-03-11T21:40:33.758317+0000","last_change":"2024-03-11T21:36:35.076782+0000","last_active":"2024-03-11T21:40:33.758317+0000","last_peered":"2024-03-11T21:40:33.758317+0000","last_clean":"2024-03-11T21:40:33.758317+0000","last_became_active":"2024-03-11T21:36:35.076335+0000","last_became_peered":"2024-03-11T21:36:35.076335+0000","last_unstale":"2024-03-11T21:40:33.758317+0000","last_undegraded":"2024-03-11T21:40:33.758317+0000","last_fullsized":"2024-03-11T21:40:33.758317+0000","mapping_epoch":277,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":278,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:34:33.082577+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:41.544550+0000","last_clean_scrub_stamp":"2024-03-11T21:34:33.082577+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":35,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.10","version":"0'0","reported_seq":625,"reported_epoch":352,"state":"active+clean","last_fresh":"2024-03-11T21:40:28.387608+0000","last_change":"2024-03-11T21:35:37.694623+0000","last_active":"2024-03-11T21:40:28.387608+0000","last_peered":"2024-03-11T21:40:28.387608+0000","last_clean":"2024-03-11T21:40:28.387608+0000","last_became_active":"2024-03-11T21:25:38.533138+0000","last_became_peered":"2024-03-11T21:25:38.533138+0000","last_unstale":"2024-03-11T21:40:28.387608+0000","last_undegraded":"2024-03-11T21:40:28.387608+0000","last_fullsized":"2024-03-11T21:40:28.387608+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:35:37.694533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:37.512018+0000","last_clean_scrub_stamp":"2024-03-11T21:35:37.694533+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,0],"acting":[2,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.16","version":"260'15","reported_seq":702,"reported_epoch":354,"state":"active+clean","last_fresh":"2024-03-11T21:40:33.391035+0000","last_change":"2024-03-11T21:36:35.080158+0000","last_active":"2024-03-11T21:40:33.391035+0000","last_peered":"2024-03-11T21:40:33.391035+0000","last_clean":"2024-03-11T21:40:33.391035+0000","last_became_active":"2024-03-11T21:36:35.056862+0000","last_became_peered":"2024-03-11T21:36:35.056862+0000","last_unstale":"2024-03-11T21:40:33.391035+0000","last_undegraded":"2024-03-11T21:40:33.391035+0000","last_fullsized":"2024-03-11T21:40:33.391035+0000","mapping_epoch":277,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":278,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'9","last_scrub_stamp":"2024-03-11T21:35:12.741225+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:39.525554+0000","last_clean_scrub_stamp":"2024-03-11T21:35:12.741225+0000","objects_scrubbed":4,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:40:37.147254+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":11,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.b","version":"0'0","reported_seq":531,"reported_epoch":354,"state":"active+clean","last_fresh":"2024-03-11T21:40:33.758531+0000","last_change":"2024-03-11T21:40:27.611092+0000","last_active":"2024-03-11T21:40:33.758531+0000","last_peered":"2024-03-11T21:40:33.758531+0000","last_clean":"2024-03-11T21:40:33.758531+0000","last_became_active":"2024-03-11T21:36:35.072147+0000","last_became_peered":"2024-03-11T21:36:35.072147+0000","last_unstale":"2024-03-11T21:40:33.758531+0000","last_undegraded":"2024-03-11T21:40:33.758531+0000","last_fullsized":"2024-03-11T21:40:33.758531+0000","mapping_epoch":277,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":278,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:40:27.610987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:43.559452+0000","last_clean_scrub_stamp":"2024-03-11T21:40:27.610987+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:41:47.989897+0000","scrub_duration":60,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.f","version":"67'2","reported_seq":641,"reported_epoch":353,"state":"active+clean","last_fresh":"2024-03-11T21:40:31.955718+0000","last_change":"2024-03-11T21:35:36.613667+0000","last_active":"2024-03-11T21:40:31.955718+0000","last_peered":"2024-03-11T21:40:31.955718+0000","last_clean":"2024-03-11T21:40:31.955718+0000","last_became_active":"2024-03-11T21:25:38.537711+0000","last_became_peered":"2024-03-11T21:25:38.537711+0000","last_unstale":"2024-03-11T21:40:31.955718+0000","last_undegraded":"2024-03-11T21:40:31.955718+0000","last_fullsized":"2024-03-11T21:40:31.955718+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'2","last_scrub_stamp":"2024-03-11T21:35:36.613516+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:37.512018+0000","last_clean_scrub_stamp":"2024-03-11T21:35:36.613516+0000","objects_scrubbed":2,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":92,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":10,"num_read_kb":10,"num_write":4,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,7],"acting":[4,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.9","version":"260'23","reported_seq":680,"reported_epoch":353,"state":"active+clean","last_fresh":"2024-03-11T21:40:31.954988+0000","last_change":"2024-03-11T21:40:14.543701+0000","last_active":"2024-03-11T21:40:31.954988+0000","last_peered":"2024-03-11T21:40:31.954988+0000","last_clean":"2024-03-11T21:40:31.954988+0000","last_became_active":"2024-03-11T21:36:35.059930+0000","last_became_peered":"2024-03-11T21:36:35.059930+0000","last_unstale":"2024-03-11T21:40:31.954988+0000","last_undegraded":"2024-03-11T21:40:31.954988+0000","last_fullsized":"2024-03-11T21:40:31.954988+0000","mapping_epoch":277,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":278,"parent":"0.0","parent_split_bits":0,"last_scrub":"260'23","last_scrub_stamp":"2024-03-11T21:40:14.543596+0000","last_deep_scrub":"146'17","last_deep_scrub_stamp":"2024-03-11T21:34:26.765075+0000","last_clean_scrub_stamp":"2024-03-11T21:40:14.543596+0000","objects_scrubbed":5,"log_size":23,"log_dups_size":0,"ondisk_log_size":23,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:41:34.525219+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":34,"num_read_kb":25,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,3],"acting":[4,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":617,"reported_epoch":352,"state":"active+clean","last_fresh":"2024-03-11T21:40:28.387705+0000","last_change":"2024-03-11T21:35:16.643421+0000","last_active":"2024-03-11T21:40:28.387705+0000","last_peered":"2024-03-11T21:40:28.387705+0000","last_clean":"2024-03-11T21:40:28.387705+0000","last_became_active":"2024-03-11T21:25:42.573987+0000","last_became_peered":"2024-03-11T21:25:42.573987+0000","last_unstale":"2024-03-11T21:40:28.387705+0000","last_undegraded":"2024-03-11T21:40:28.387705+0000","last_fullsized":"2024-03-11T21:40:28.387705+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:35:16.643311+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:41.544550+0000","last_clean_scrub_stamp":"2024-03-11T21:35:16.643311+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.0","version":"0'0","reported_seq":629,"reported_epoch":354,"state":"active+clean","last_fresh":"2024-03-11T21:40:33.389406+0000","last_change":"2024-03-11T21:40:27.230013+0000","last_active":"2024-03-11T21:40:33.389406+0000","last_peered":"2024-03-11T21:40:33.389406+0000","last_clean":"2024-03-11T21:40:33.389406+0000","last_became_active":"2024-03-11T21:36:35.058054+0000","last_became_peered":"2024-03-11T21:36:35.058054+0000","last_unstale":"2024-03-11T21:40:33.389406+0000","last_undegraded":"2024-03-11T21:40:33.389406+0000","last_fullsized":"2024-03-11T21:40:33.389406+0000","mapping_epoch":277,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":278,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:40:27.229904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:29:39.498784+0000","last_clean_scrub_stamp":"2024-03-11T21:40:27.229904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:41:49.392723+0000","scrub_duration":38,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":642,"reported_epoch":353,"state":"active+clean","last_fresh":"2024-03-11T21:40:31.950241+0000","last_change":"2024-03-11T21:35:33.608063+0000","last_active":"2024-03-11T21:40:31.950241+0000","last_peered":"2024-03-11T21:40:31.950241+0000","last_clean":"2024-03-11T21:40:31.950241+0000","last_became_active":"2024-03-11T21:34:41.150913+0000","last_became_peered":"2024-03-11T21:34:41.150913+0000","last_unstale":"2024-03-11T21:40:31.950241+0000","last_undegraded":"2024-03-11T21:40:31.950241+0000","last_fullsized":"2024-03-11T21:40:31.950241+0000","mapping_epoch":226,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":227,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:35:33.607969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:37.512018+0000","last_clean_scrub_stamp":"2024-03-11T21:35:33.607969+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,7],"acting":[1,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.2","version":"260'25","reported_seq":661,"reported_epoch":353,"state":"active+clean","last_fresh":"2024-03-11T21:40:31.950698+0000","last_change":"2024-03-11T21:35:17.666749+0000","last_active":"2024-03-11T21:40:31.950698+0000","last_peered":"2024-03-11T21:40:31.950698+0000","last_clean":"2024-03-11T21:40:31.950698+0000","last_became_active":"2024-03-11T21:25:40.548833+0000","last_became_peered":"2024-03-11T21:25:40.548833+0000","last_unstale":"2024-03-11T21:40:31.950698+0000","last_undegraded":"2024-03-11T21:40:31.950698+0000","last_fullsized":"2024-03-11T21:40:31.950698+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"94'15","last_scrub_stamp":"2024-03-11T21:35:17.666498+0000","last_deep_scrub":"94'15","last_deep_scrub_stamp":"2024-03-11T21:26:49.993060+0000","last_clean_scrub_stamp":"2024-03-11T21:35:17.666498+0000","objects_scrubbed":5,"log_size":25,"log_dups_size":0,"ondisk_log_size":25,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":20,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":25,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.3","version":"74'8","reported_seq":812,"reported_epoch":354,"state":"active+clean","last_fresh":"2024-03-11T21:40:33.391611+0000","last_change":"2024-03-11T21:35:31.695369+0000","last_active":"2024-03-11T21:40:33.391611+0000","last_peered":"2024-03-11T21:40:33.391611+0000","last_clean":"2024-03-11T21:40:33.391611+0000","last_became_active":"2024-03-11T21:25:42.578161+0000","last_became_peered":"2024-03-11T21:25:42.578161+0000","last_unstale":"2024-03-11T21:40:33.391611+0000","last_undegraded":"2024-03-11T21:40:33.391611+0000","last_fullsized":"2024-03-11T21:40:33.391611+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'8","last_scrub_stamp":"2024-03-11T21:35:31.695263+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:41.544550+0000","last_clean_scrub_stamp":"2024-03-11T21:35:31.695263+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:40:37.147290+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,5],"acting":[0,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":561,"reported_epoch":354,"state":"active+clean","last_fresh":"2024-03-11T21:40:33.758326+0000","last_change":"2024-03-11T21:36:35.068977+0000","last_active":"2024-03-11T21:40:33.758326+0000","last_peered":"2024-03-11T21:40:33.758326+0000","last_clean":"2024-03-11T21:40:33.758326+0000","last_became_active":"2024-03-11T21:36:35.067288+0000","last_became_peered":"2024-03-11T21:36:35.067288+0000","last_unstale":"2024-03-11T21:40:33.758326+0000","last_undegraded":"2024-03-11T21:40:33.758326+0000","last_fullsized":"2024-03-11T21:40:33.758326+0000","mapping_epoch":277,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":278,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:35:09.985377+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:28:30.563712+0000","last_clean_scrub_stamp":"2024-03-11T21:35:09.985377+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,7],"acting":[3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":635,"reported_epoch":352,"state":"active+clean","last_fresh":"2024-03-11T21:40:28.556272+0000","last_change":"2024-03-11T21:35:40.116979+0000","last_active":"2024-03-11T21:40:28.556272+0000","last_peered":"2024-03-11T21:40:28.556272+0000","last_clean":"2024-03-11T21:40:28.556272+0000","last_became_active":"2024-03-11T21:25:38.538540+0000","last_became_peered":"2024-03-11T21:25:38.538540+0000","last_unstale":"2024-03-11T21:40:28.556272+0000","last_undegraded":"2024-03-11T21:40:28.556272+0000","last_fullsized":"2024-03-11T21:40:28.556272+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:35:40.116877+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:37.512018+0000","last_clean_scrub_stamp":"2024-03-11T21:35:40.116877+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.4","version":"280'42","reported_seq":888,"reported_epoch":353,"state":"active+clean","last_fresh":"2024-03-11T21:40:31.950141+0000","last_change":"2024-03-11T21:36:35.095935+0000","last_active":"2024-03-11T21:40:31.950141+0000","last_peered":"2024-03-11T21:40:31.950141+0000","last_clean":"2024-03-11T21:40:31.950141+0000","last_became_active":"2024-03-11T21:36:35.073430+0000","last_became_peered":"2024-03-11T21:36:35.073430+0000","last_unstale":"2024-03-11T21:40:31.950141+0000","last_undegraded":"2024-03-11T21:40:31.950141+0000","last_fullsized":"2024-03-11T21:40:31.950141+0000","mapping_epoch":277,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":278,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'28","last_scrub_stamp":"2024-03-11T21:34:19.513252+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:39.525554+0000","last_clean_scrub_stamp":"2024-03-11T21:34:19.513252+0000","objects_scrubbed":10,"log_size":42,"log_dups_size":0,"ondisk_log_size":42,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":358,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":66,"num_read_kb":51,"num_write":38,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3],"acting":[1,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":624,"reported_epoch":354,"state":"active+clean","last_fresh":"2024-03-11T21:40:33.390620+0000","last_change":"2024-03-11T21:35:32.651731+0000","last_active":"2024-03-11T21:40:33.390620+0000","last_peered":"2024-03-11T21:40:33.390620+0000","last_clean":"2024-03-11T21:40:33.390620+0000","last_became_active":"2024-03-11T21:25:42.572673+0000","last_became_peered":"2024-03-11T21:25:42.572673+0000","last_unstale":"2024-03-11T21:40:33.390620+0000","last_undegraded":"2024-03-11T21:40:33.390620+0000","last_fullsized":"2024-03-11T21:40:33.390620+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:35:32.651652+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:41.544550+0000","last_clean_scrub_stamp":"2024-03-11T21:35:32.651652+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:40:37.147358+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.5","version":"0'0","reported_seq":280,"reported_epoch":352,"state":"active+clean","last_fresh":"2024-03-11T21:40:28.457794+0000","last_change":"2024-03-11T21:36:35.069989+0000","last_active":"2024-03-11T21:40:28.457794+0000","last_peered":"2024-03-11T21:40:28.457794+0000","last_clean":"2024-03-11T21:40:28.457794+0000","last_became_active":"2024-03-11T21:36:35.069358+0000","last_became_peered":"2024-03-11T21:36:35.069358+0000","last_unstale":"2024-03-11T21:40:28.457794+0000","last_undegraded":"2024-03-11T21:40:28.457794+0000","last_fullsized":"2024-03-11T21:40:28.457794+0000","mapping_epoch":277,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":278,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:35:19.841632+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:43.559452+0000","last_clean_scrub_stamp":"2024-03-11T21:35:19.841632+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,3],"acting":[7,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.0","version":"164'6","reported_seq":1528,"reported_epoch":353,"state":"active+clean","last_fresh":"2024-03-11T21:40:32.066109+0000","last_change":"2024-03-11T21:35:38.518243+0000","last_active":"2024-03-11T21:40:32.066109+0000","last_peered":"2024-03-11T21:40:32.066109+0000","last_clean":"2024-03-11T21:40:32.066109+0000","last_became_active":"2024-03-11T21:25:38.536890+0000","last_became_peered":"2024-03-11T21:25:38.536890+0000","last_unstale":"2024-03-11T21:40:32.066109+0000","last_undegraded":"2024-03-11T21:40:32.066109+0000","last_fullsized":"2024-03-11T21:40:32.066109+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"164'6","last_scrub_stamp":"2024-03-11T21:35:38.518097+0000","last_deep_scrub":"164'6","last_deep_scrub_stamp":"2024-03-11T21:32:17.597666+0000","last_clean_scrub_stamp":"2024-03-11T21:35:38.518097+0000","objects_scrubbed":1,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0.000753559,"stat_sum":{"num_bytes":389,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":887,"num_read_kb":882,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,6],"acting":[1,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":643,"reported_epoch":352,"state":"active+clean","last_fresh":"2024-03-11T21:40:28.387405+0000","last_change":"2024-03-11T21:40:09.915304+0000","last_active":"2024-03-11T21:40:28.387405+0000","last_peered":"2024-03-11T21:40:28.387405+0000","last_clean":"2024-03-11T21:40:28.387405+0000","last_became_active":"2024-03-11T21:36:35.056966+0000","last_became_peered":"2024-03-11T21:36:35.056966+0000","last_unstale":"2024-03-11T21:40:28.387405+0000","last_undegraded":"2024-03-11T21:40:28.387405+0000","last_fullsized":"2024-03-11T21:40:28.387405+0000","mapping_epoch":277,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":278,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:40:09.915179+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:40:09.915179+0000","last_clean_scrub_stamp":"2024-03-11T21:40:09.915179+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:41:27.096360+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.7","version":"260'22","reported_seq":666,"reported_epoch":353,"state":"active+clean","last_fresh":"2024-03-11T21:40:31.950776+0000","last_change":"2024-03-11T21:35:37.508679+0000","last_active":"2024-03-11T21:40:31.950776+0000","last_peered":"2024-03-11T21:40:31.950776+0000","last_clean":"2024-03-11T21:40:31.950776+0000","last_became_active":"2024-03-11T21:25:40.541769+0000","last_became_peered":"2024-03-11T21:25:40.541769+0000","last_unstale":"2024-03-11T21:40:31.950776+0000","last_undegraded":"2024-03-11T21:40:31.950776+0000","last_fullsized":"2024-03-11T21:40:31.950776+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"97'18","last_scrub_stamp":"2024-03-11T21:35:37.508567+0000","last_deep_scrub":"97'18","last_deep_scrub_stamp":"2024-03-11T21:27:19.239404+0000","last_clean_scrub_stamp":"2024-03-11T21:35:37.508567+0000","objects_scrubbed":5,"log_size":22,"log_dups_size":0,"ondisk_log_size":22,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":36,"num_read_kb":25,"num_write":20,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":629,"reported_epoch":352,"state":"active+clean","last_fresh":"2024-03-11T21:40:28.388564+0000","last_change":"2024-03-11T21:35:22.632822+0000","last_active":"2024-03-11T21:40:28.388564+0000","last_peered":"2024-03-11T21:40:28.388564+0000","last_clean":"2024-03-11T21:40:28.388564+0000","last_became_active":"2024-03-11T21:34:41.143660+0000","last_became_peered":"2024-03-11T21:34:41.143660+0000","last_unstale":"2024-03-11T21:40:28.388564+0000","last_undegraded":"2024-03-11T21:40:28.388564+0000","last_fullsized":"2024-03-11T21:40:28.388564+0000","mapping_epoch":226,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":227,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:35:22.632707+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:41.544550+0000","last_clean_scrub_stamp":"2024-03-11T21:35:22.632707+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":18,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7],"acting":[2,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.7","version":"124'2","reported_seq":625,"reported_epoch":352,"state":"active+clean","last_fresh":"2024-03-11T21:40:31.722212+0000","last_change":"2024-03-11T21:40:31.722243+0000","last_active":"2024-03-11T21:40:31.722212+0000","last_peered":"2024-03-11T21:40:31.722212+0000","last_clean":"2024-03-11T21:40:31.722212+0000","last_became_active":"2024-03-11T21:36:35.058786+0000","last_became_peered":"2024-03-11T21:36:35.058786+0000","last_unstale":"2024-03-11T21:40:31.722212+0000","last_undegraded":"2024-03-11T21:40:31.722212+0000","last_fullsized":"2024-03-11T21:40:31.722212+0000","mapping_epoch":277,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":278,"parent":"0.0","parent_split_bits":0,"last_scrub":"124'2","last_scrub_stamp":"2024-03-11T21:40:31.722100+0000","last_deep_scrub":"124'2","last_deep_scrub_stamp":"2024-03-11T21:29:34.956908+0000","last_clean_scrub_stamp":"2024-03-11T21:40:31.722100+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:41:45.441808+0000","scrub_duration":87,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,4],"acting":[5,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2","version":"71'2","reported_seq":582,"reported_epoch":354,"state":"active+clean","last_fresh":"2024-03-11T21:40:33.758420+0000","last_change":"2024-03-11T21:36:35.071218+0000","last_active":"2024-03-11T21:40:33.758420+0000","last_peered":"2024-03-11T21:40:33.758420+0000","last_clean":"2024-03-11T21:40:33.758420+0000","last_became_active":"2024-03-11T21:36:35.070783+0000","last_became_peered":"2024-03-11T21:36:35.070783+0000","last_unstale":"2024-03-11T21:40:33.758420+0000","last_undegraded":"2024-03-11T21:40:33.758420+0000","last_fullsized":"2024-03-11T21:40:33.758420+0000","mapping_epoch":277,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":278,"parent":"0.0","parent_split_bits":0,"last_scrub":"71'2","last_scrub_stamp":"2024-03-11T21:32:21.543053+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:37.512018+0000","last_clean_scrub_stamp":"2024-03-11T21:32:21.543053+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014206449999999999,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":645,"reported_epoch":352,"state":"active+clean","last_fresh":"2024-03-11T21:40:28.556190+0000","last_change":"2024-03-11T21:35:08.391825+0000","last_active":"2024-03-11T21:40:28.556190+0000","last_peered":"2024-03-11T21:40:28.556190+0000","last_clean":"2024-03-11T21:40:28.556190+0000","last_became_active":"2024-03-11T21:34:41.146672+0000","last_became_peered":"2024-03-11T21:34:41.146672+0000","last_unstale":"2024-03-11T21:40:28.556190+0000","last_undegraded":"2024-03-11T21:40:28.556190+0000","last_fullsized":"2024-03-11T21:40:28.556190+0000","mapping_epoch":226,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":227,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:35:08.391713+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:37.512018+0000","last_clean_scrub_stamp":"2024-03-11T21:35:08.391713+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"87'128","reported_seq":742,"reported_epoch":352,"state":"active+clean","last_fresh":"2024-03-11T21:40:28.457199+0000","last_change":"2024-03-11T21:35:21.798272+0000","last_active":"2024-03-11T21:40:28.457199+0000","last_peered":"2024-03-11T21:40:28.457199+0000","last_clean":"2024-03-11T21:40:28.457199+0000","last_became_active":"2024-03-11T21:34:44.164569+0000","last_became_peered":"2024-03-11T21:34:44.164569+0000","last_unstale":"2024-03-11T21:40:28.457199+0000","last_undegraded":"2024-03-11T21:40:28.457199+0000","last_fullsized":"2024-03-11T21:40:28.457199+0000","mapping_epoch":229,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":230,"parent":"0.0","parent_split_bits":0,"last_scrub":"87'128","last_scrub_stamp":"2024-03-11T21:35:21.798144+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:24.261974+0000","last_clean_scrub_stamp":"2024-03-11T21:35:21.798144+0000","objects_scrubbed":2,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":198,"num_read_kb":294,"num_write":179,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":2951840,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.5","version":"260'28","reported_seq":859,"reported_epoch":352,"state":"active+clean","last_fresh":"2024-03-11T21:40:28.556620+0000","last_change":"2024-03-11T21:39:45.216634+0000","last_active":"2024-03-11T21:40:28.556620+0000","last_peered":"2024-03-11T21:40:28.556620+0000","last_clean":"2024-03-11T21:40:28.556620+0000","last_became_active":"2024-03-11T21:25:40.761457+0000","last_became_peered":"2024-03-11T21:25:40.761457+0000","last_unstale":"2024-03-11T21:40:28.556620+0000","last_undegraded":"2024-03-11T21:40:28.556620+0000","last_fullsized":"2024-03-11T21:40:28.556620+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"260'28","last_scrub_stamp":"2024-03-11T21:39:45.216468+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:39.525554+0000","last_clean_scrub_stamp":"2024-03-11T21:39:45.216468+0000","objects_scrubbed":8,"log_size":28,"log_dups_size":0,"ondisk_log_size":28,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:40:57.658763+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":154,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":43,"num_read_kb":33,"num_write":25,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":271,"reported_epoch":352,"state":"active+clean","last_fresh":"2024-03-11T21:40:28.457827+0000","last_change":"2024-03-11T21:35:24.841170+0000","last_active":"2024-03-11T21:40:28.457827+0000","last_peered":"2024-03-11T21:40:28.457827+0000","last_clean":"2024-03-11T21:40:28.457827+0000","last_became_active":"2024-03-11T21:34:41.139782+0000","last_became_peered":"2024-03-11T21:34:41.139782+0000","last_unstale":"2024-03-11T21:40:28.457827+0000","last_undegraded":"2024-03-11T21:40:28.457827+0000","last_fullsized":"2024-03-11T21:40:28.457827+0000","mapping_epoch":226,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":227,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:35:24.841041+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:32:21.266672+0000","last_clean_scrub_stamp":"2024-03-11T21:35:24.841041+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,5],"acting":[7,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.4","version":"0'0","reported_seq":619,"reported_epoch":353,"state":"active+clean","last_fresh":"2024-03-11T21:40:31.950581+0000","last_change":"2024-03-11T21:36:35.073305+0000","last_active":"2024-03-11T21:40:31.950581+0000","last_peered":"2024-03-11T21:40:31.950581+0000","last_clean":"2024-03-11T21:40:31.950581+0000","last_became_active":"2024-03-11T21:36:35.072824+0000","last_became_peered":"2024-03-11T21:36:35.072824+0000","last_unstale":"2024-03-11T21:40:31.950581+0000","last_undegraded":"2024-03-11T21:40:31.950581+0000","last_fullsized":"2024-03-11T21:40:31.950581+0000","mapping_epoch":277,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":278,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:34:23.531064+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:43.559452+0000","last_clean_scrub_stamp":"2024-03-11T21:34:23.531064+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":648,"reported_epoch":354,"state":"active+clean","last_fresh":"2024-03-11T21:40:33.389496+0000","last_change":"2024-03-11T21:36:35.070626+0000","last_active":"2024-03-11T21:40:33.389496+0000","last_peered":"2024-03-11T21:40:33.389496+0000","last_clean":"2024-03-11T21:40:33.389496+0000","last_became_active":"2024-03-11T21:36:35.069452+0000","last_became_peered":"2024-03-11T21:36:35.069452+0000","last_unstale":"2024-03-11T21:40:33.389496+0000","last_undegraded":"2024-03-11T21:40:33.389496+0000","last_fullsized":"2024-03-11T21:40:33.389496+0000","mapping_epoch":277,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":278,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:35:13.741947+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:37.512018+0000","last_clean_scrub_stamp":"2024-03-11T21:35:13.741947+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:40:37.147197+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0.00089342499999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":271,"reported_epoch":352,"state":"active+clean","last_fresh":"2024-03-11T21:40:28.457095+0000","last_change":"2024-03-11T21:40:27.117606+0000","last_active":"2024-03-11T21:40:28.457095+0000","last_peered":"2024-03-11T21:40:28.457095+0000","last_clean":"2024-03-11T21:40:28.457095+0000","last_became_active":"2024-03-11T21:34:41.149219+0000","last_became_peered":"2024-03-11T21:34:41.149219+0000","last_unstale":"2024-03-11T21:40:28.457095+0000","last_undegraded":"2024-03-11T21:40:28.457095+0000","last_fullsized":"2024-03-11T21:40:28.457095+0000","mapping_epoch":226,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":227,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:40:27.117515+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:37.512018+0000","last_clean_scrub_stamp":"2024-03-11T21:40:27.117515+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:41:35.677145+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,0],"acting":[7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.6","version":"352'30","reported_seq":697,"reported_epoch":354,"state":"active+clean","last_fresh":"2024-03-11T21:40:33.390528+0000","last_change":"2024-03-11T21:40:31.175417+0000","last_active":"2024-03-11T21:40:33.390528+0000","last_peered":"2024-03-11T21:40:33.390528+0000","last_clean":"2024-03-11T21:40:33.390528+0000","last_became_active":"2024-03-11T21:32:22.796486+0000","last_became_peered":"2024-03-11T21:32:22.796486+0000","last_unstale":"2024-03-11T21:40:33.390528+0000","last_undegraded":"2024-03-11T21:40:33.390528+0000","last_fullsized":"2024-03-11T21:40:33.390528+0000","mapping_epoch":166,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":167,"parent":"0.0","parent_split_bits":0,"last_scrub":"352'30","last_scrub_stamp":"2024-03-11T21:40:31.175323+0000","last_deep_scrub":"352'30","last_deep_scrub_stamp":"2024-03-11T21:40:31.175323+0000","last_clean_scrub_stamp":"2024-03-11T21:40:31.175323+0000","objects_scrubbed":6,"log_size":30,"log_dups_size":0,"ondisk_log_size":30,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:41:40.436187+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":36,"num_read_kb":30,"num_write":24,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":618,"reported_epoch":352,"state":"active+clean","last_fresh":"2024-03-11T21:40:28.557647+0000","last_change":"2024-03-11T21:36:07.451482+0000","last_active":"2024-03-11T21:40:28.557647+0000","last_peered":"2024-03-11T21:40:28.557647+0000","last_clean":"2024-03-11T21:40:28.557647+0000","last_became_active":"2024-03-11T21:25:42.572324+0000","last_became_peered":"2024-03-11T21:25:42.572324+0000","last_unstale":"2024-03-11T21:40:28.557647+0000","last_undegraded":"2024-03-11T21:40:28.557647+0000","last_fullsized":"2024-03-11T21:40:28.557647+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:36:07.451102+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:41.544550+0000","last_clean_scrub_stamp":"2024-03-11T21:36:07.451102+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":54,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":626,"reported_epoch":353,"state":"active+clean","last_fresh":"2024-03-11T21:40:31.950343+0000","last_change":"2024-03-11T21:36:18.529506+0000","last_active":"2024-03-11T21:40:31.950343+0000","last_peered":"2024-03-11T21:40:31.950343+0000","last_clean":"2024-03-11T21:40:31.950343+0000","last_became_active":"2024-03-11T21:25:45.301416+0000","last_became_peered":"2024-03-11T21:25:45.301416+0000","last_unstale":"2024-03-11T21:40:31.950343+0000","last_undegraded":"2024-03-11T21:40:31.950343+0000","last_fullsized":"2024-03-11T21:40:31.950343+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:36:18.529370+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:43.559452+0000","last_clean_scrub_stamp":"2024-03-11T21:36:18.529370+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":54,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":271,"reported_epoch":352,"state":"active+clean","last_fresh":"2024-03-11T21:40:28.457859+0000","last_change":"2024-03-11T21:35:33.740615+0000","last_active":"2024-03-11T21:40:28.457859+0000","last_peered":"2024-03-11T21:40:28.457859+0000","last_clean":"2024-03-11T21:40:28.457859+0000","last_became_active":"2024-03-11T21:34:41.152721+0000","last_became_peered":"2024-03-11T21:34:41.152721+0000","last_unstale":"2024-03-11T21:40:28.457859+0000","last_undegraded":"2024-03-11T21:40:28.457859+0000","last_fullsized":"2024-03-11T21:40:28.457859+0000","mapping_epoch":226,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":227,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:35:33.740538+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:26:57.333669+0000","last_clean_scrub_stamp":"2024-03-11T21:35:33.740538+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.3","version":"260'37","reported_seq":735,"reported_epoch":354,"state":"active+clean","last_fresh":"2024-03-11T21:40:33.391389+0000","last_change":"2024-03-11T21:35:11.697314+0000","last_active":"2024-03-11T21:40:33.391389+0000","last_peered":"2024-03-11T21:40:33.391389+0000","last_clean":"2024-03-11T21:40:33.391389+0000","last_became_active":"2024-03-11T21:34:41.145607+0000","last_became_peered":"2024-03-11T21:34:41.145607+0000","last_unstale":"2024-03-11T21:40:33.391389+0000","last_undegraded":"2024-03-11T21:40:33.391389+0000","last_fullsized":"2024-03-11T21:40:33.391389+0000","mapping_epoch":226,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":227,"parent":"0.0","parent_split_bits":0,"last_scrub":"165'27","last_scrub_stamp":"2024-03-11T21:35:11.697140+0000","last_deep_scrub":"165'27","last_deep_scrub_stamp":"2024-03-11T21:32:21.576732+0000","last_clean_scrub_stamp":"2024-03-11T21:35:11.697140+0000","objects_scrubbed":8,"log_size":37,"log_dups_size":0,"ondisk_log_size":37,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:40:37.147219+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":54,"num_read_kb":40,"num_write":32,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":660,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":644,"reported_epoch":352,"state":"active+clean","last_fresh":"2024-03-11T21:40:28.556616+0000","last_change":"2024-03-11T21:40:15.172829+0000","last_active":"2024-03-11T21:40:28.556616+0000","last_peered":"2024-03-11T21:40:28.556616+0000","last_clean":"2024-03-11T21:40:28.556616+0000","last_became_active":"2024-03-11T21:25:42.579045+0000","last_became_peered":"2024-03-11T21:25:42.579045+0000","last_unstale":"2024-03-11T21:40:28.556616+0000","last_undegraded":"2024-03-11T21:40:28.556616+0000","last_fullsized":"2024-03-11T21:40:28.556616+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:40:15.172720+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:40:15.172720+0000","last_clean_scrub_stamp":"2024-03-11T21:40:15.172720+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:41:19.672684+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":632,"reported_epoch":353,"state":"active+clean","last_fresh":"2024-03-11T21:40:31.955330+0000","last_change":"2024-03-11T21:40:28.582452+0000","last_active":"2024-03-11T21:40:31.955330+0000","last_peered":"2024-03-11T21:40:31.955330+0000","last_clean":"2024-03-11T21:40:31.955330+0000","last_became_active":"2024-03-11T21:32:22.791439+0000","last_became_peered":"2024-03-11T21:32:22.791439+0000","last_unstale":"2024-03-11T21:40:31.955330+0000","last_undegraded":"2024-03-11T21:40:31.955330+0000","last_fullsized":"2024-03-11T21:40:31.955330+0000","mapping_epoch":166,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":167,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:40:28.582339+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:43.559452+0000","last_clean_scrub_stamp":"2024-03-11T21:40:28.582339+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:41:49.291632+0000","scrub_duration":20,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,2],"acting":[4,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.6","version":"67'1","reported_seq":630,"reported_epoch":353,"state":"active+clean","last_fresh":"2024-03-11T21:40:31.950790+0000","last_change":"2024-03-11T21:35:30.581783+0000","last_active":"2024-03-11T21:40:31.950790+0000","last_peered":"2024-03-11T21:40:31.950790+0000","last_clean":"2024-03-11T21:40:31.950790+0000","last_became_active":"2024-03-11T21:25:38.536142+0000","last_became_peered":"2024-03-11T21:25:38.536142+0000","last_unstale":"2024-03-11T21:40:31.950790+0000","last_undegraded":"2024-03-11T21:40:31.950790+0000","last_fullsized":"2024-03-11T21:40:31.950790+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'1","last_scrub_stamp":"2024-03-11T21:35:30.581701+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:37.512018+0000","last_clean_scrub_stamp":"2024-03-11T21:35:30.581701+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,4],"acting":[1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.0","version":"260'38","reported_seq":631,"reported_epoch":354,"state":"active+clean","last_fresh":"2024-03-11T21:40:33.757863+0000","last_change":"2024-03-11T21:36:35.087771+0000","last_active":"2024-03-11T21:40:33.757863+0000","last_peered":"2024-03-11T21:40:33.757863+0000","last_clean":"2024-03-11T21:40:33.757863+0000","last_became_active":"2024-03-11T21:36:35.068071+0000","last_became_peered":"2024-03-11T21:36:35.068071+0000","last_unstale":"2024-03-11T21:40:33.757863+0000","last_undegraded":"2024-03-11T21:40:33.757863+0000","last_fullsized":"2024-03-11T21:40:33.757863+0000","mapping_epoch":277,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":278,"parent":"0.0","parent_split_bits":0,"last_scrub":"93'26","last_scrub_stamp":"2024-03-11T21:34:16.863984+0000","last_deep_scrub":"93'26","last_deep_scrub_stamp":"2024-03-11T21:26:40.680324+0000","last_clean_scrub_stamp":"2024-03-11T21:34:16.863984+0000","objects_scrubbed":8,"log_size":38,"log_dups_size":0,"ondisk_log_size":38,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":22,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":51,"num_read_kb":39,"num_write":32,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":651,"reported_epoch":353,"state":"active+clean","last_fresh":"2024-03-11T21:40:31.955035+0000","last_change":"2024-03-11T21:40:21.518921+0000","last_active":"2024-03-11T21:40:31.955035+0000","last_peered":"2024-03-11T21:40:31.955035+0000","last_clean":"2024-03-11T21:40:31.955035+0000","last_became_active":"2024-03-11T21:36:35.057517+0000","last_became_peered":"2024-03-11T21:36:35.057517+0000","last_unstale":"2024-03-11T21:40:31.955035+0000","last_undegraded":"2024-03-11T21:40:31.955035+0000","last_fullsized":"2024-03-11T21:40:31.955035+0000","mapping_epoch":277,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":278,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:40:21.518840+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:40:21.518840+0000","last_clean_scrub_stamp":"2024-03-11T21:40:21.518840+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:41:26.549696+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,7],"acting":[4,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.3","version":"0'0","reported_seq":271,"reported_epoch":352,"state":"active+clean","last_fresh":"2024-03-11T21:40:28.457632+0000","last_change":"2024-03-11T21:40:25.234980+0000","last_active":"2024-03-11T21:40:28.457632+0000","last_peered":"2024-03-11T21:40:28.457632+0000","last_clean":"2024-03-11T21:40:28.457632+0000","last_became_active":"2024-03-11T21:34:41.142683+0000","last_became_peered":"2024-03-11T21:34:41.142683+0000","last_unstale":"2024-03-11T21:40:28.457632+0000","last_undegraded":"2024-03-11T21:40:28.457632+0000","last_fullsized":"2024-03-11T21:40:28.457632+0000","mapping_epoch":226,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":227,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:40:25.213541+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:29:42.485740+0000","last_clean_scrub_stamp":"2024-03-11T21:40:25.213541+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:41:30.275517+0000","scrub_duration":55,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,2],"acting":[7,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":663,"reported_epoch":352,"state":"active+clean","last_fresh":"2024-03-11T21:40:28.557292+0000","last_change":"2024-03-11T21:40:07.187684+0000","last_active":"2024-03-11T21:40:28.557292+0000","last_peered":"2024-03-11T21:40:28.557292+0000","last_clean":"2024-03-11T21:40:28.557292+0000","last_became_active":"2024-03-11T21:34:41.145628+0000","last_became_peered":"2024-03-11T21:34:41.145628+0000","last_unstale":"2024-03-11T21:40:28.557292+0000","last_undegraded":"2024-03-11T21:40:28.557292+0000","last_fullsized":"2024-03-11T21:40:28.557292+0000","mapping_epoch":226,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":227,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:40:07.187595+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:37.512018+0000","last_clean_scrub_stamp":"2024-03-11T21:40:07.187595+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:41:20.424112+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,2],"acting":[6,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.1","version":"260'28","reported_seq":668,"reported_epoch":353,"state":"active+clean","last_fresh":"2024-03-11T21:40:31.954842+0000","last_change":"2024-03-11T21:35:33.644654+0000","last_active":"2024-03-11T21:40:31.954842+0000","last_peered":"2024-03-11T21:40:31.954842+0000","last_clean":"2024-03-11T21:40:31.954842+0000","last_became_active":"2024-03-11T21:25:40.546219+0000","last_became_peered":"2024-03-11T21:25:40.546219+0000","last_unstale":"2024-03-11T21:40:31.954842+0000","last_undegraded":"2024-03-11T21:40:31.954842+0000","last_fullsized":"2024-03-11T21:40:31.954842+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'14","last_scrub_stamp":"2024-03-11T21:35:33.644503+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:39.525554+0000","last_clean_scrub_stamp":"2024-03-11T21:35:33.644503+0000","objects_scrubbed":7,"log_size":28,"log_dups_size":0,"ondisk_log_size":28,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":42,"num_read_kb":35,"num_write":28,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6],"acting":[4,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.0","version":"280'11","reported_seq":738,"reported_epoch":354,"state":"active+clean","last_fresh":"2024-03-11T21:40:33.775594+0000","last_change":"2024-03-11T21:36:35.064085+0000","last_active":"2024-03-11T21:40:33.775594+0000","last_peered":"2024-03-11T21:40:33.775594+0000","last_clean":"2024-03-11T21:40:33.775594+0000","last_became_active":"2024-03-11T21:36:35.055250+0000","last_became_peered":"2024-03-11T21:36:35.055250+0000","last_unstale":"2024-03-11T21:40:33.775594+0000","last_undegraded":"2024-03-11T21:40:33.775594+0000","last_fullsized":"2024-03-11T21:40:33.775594+0000","mapping_epoch":277,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":278,"parent":"0.0","parent_split_bits":0,"last_scrub":"98'9","last_scrub_stamp":"2024-03-11T21:35:21.143578+0000","last_deep_scrub":"98'9","last_deep_scrub_stamp":"2024-03-11T21:27:29.745711+0000","last_clean_scrub_stamp":"2024-03-11T21:35:21.143578+0000","objects_scrubbed":1,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.c","version":"0'0","reported_seq":559,"reported_epoch":354,"state":"active+clean","last_fresh":"2024-03-11T21:40:33.758914+0000","last_change":"2024-03-11T21:40:30.659272+0000","last_active":"2024-03-11T21:40:33.758914+0000","last_peered":"2024-03-11T21:40:33.758914+0000","last_clean":"2024-03-11T21:40:33.758914+0000","last_became_active":"2024-03-11T21:36:35.073778+0000","last_became_peered":"2024-03-11T21:36:35.073778+0000","last_unstale":"2024-03-11T21:40:33.758914+0000","last_undegraded":"2024-03-11T21:40:33.758914+0000","last_fullsized":"2024-03-11T21:40:33.758914+0000","mapping_epoch":277,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":278,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:40:30.659160+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:43.559452+0000","last_clean_scrub_stamp":"2024-03-11T21:40:30.659160+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:41:38.577899+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.f","version":"0'0","reported_seq":626,"reported_epoch":352,"state":"active+clean","last_fresh":"2024-03-11T21:40:28.557440+0000","last_change":"2024-03-11T21:36:08.439013+0000","last_active":"2024-03-11T21:40:28.557440+0000","last_peered":"2024-03-11T21:40:28.557440+0000","last_clean":"2024-03-11T21:40:28.557440+0000","last_became_active":"2024-03-11T21:25:42.575935+0000","last_became_peered":"2024-03-11T21:25:42.575935+0000","last_unstale":"2024-03-11T21:40:28.557440+0000","last_undegraded":"2024-03-11T21:40:28.557440+0000","last_fullsized":"2024-03-11T21:40:28.557440+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:36:08.438895+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:41.544550+0000","last_clean_scrub_stamp":"2024-03-11T21:36:08.438895+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.8","version":"0'0","reported_seq":271,"reported_epoch":352,"state":"active+clean","last_fresh":"2024-03-11T21:40:28.457483+0000","last_change":"2024-03-11T21:35:34.733595+0000","last_active":"2024-03-11T21:40:28.457483+0000","last_peered":"2024-03-11T21:40:28.457483+0000","last_clean":"2024-03-11T21:40:28.457483+0000","last_became_active":"2024-03-11T21:34:41.144681+0000","last_became_peered":"2024-03-11T21:34:41.144681+0000","last_unstale":"2024-03-11T21:40:28.457483+0000","last_undegraded":"2024-03-11T21:40:28.457483+0000","last_fullsized":"2024-03-11T21:40:28.457483+0000","mapping_epoch":226,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":227,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:35:34.733491+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:37.512018+0000","last_clean_scrub_stamp":"2024-03-11T21:35:34.733491+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,1],"acting":[7,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.e","version":"260'24","reported_seq":661,"reported_epoch":353,"state":"active+clean","last_fresh":"2024-03-11T21:40:31.955530+0000","last_change":"2024-03-11T21:35:34.679906+0000","last_active":"2024-03-11T21:40:31.955530+0000","last_peered":"2024-03-11T21:40:31.955530+0000","last_clean":"2024-03-11T21:40:31.955530+0000","last_became_active":"2024-03-11T21:25:40.552178+0000","last_became_peered":"2024-03-11T21:25:40.552178+0000","last_unstale":"2024-03-11T21:40:31.955530+0000","last_undegraded":"2024-03-11T21:40:31.955530+0000","last_fullsized":"2024-03-11T21:40:31.955530+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"109'16","last_scrub_stamp":"2024-03-11T21:35:34.679817+0000","last_deep_scrub":"109'16","last_deep_scrub_stamp":"2024-03-11T21:28:24.739395+0000","last_clean_scrub_stamp":"2024-03-11T21:35:34.679817+0000","objects_scrubbed":5,"log_size":24,"log_dups_size":0,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":32,"num_read_kb":25,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.d","version":"0'0","reported_seq":614,"reported_epoch":352,"state":"active+clean","last_fresh":"2024-03-11T21:40:28.556980+0000","last_change":"2024-03-11T21:35:37.185210+0000","last_active":"2024-03-11T21:40:28.556980+0000","last_peered":"2024-03-11T21:40:28.556980+0000","last_clean":"2024-03-11T21:40:28.556980+0000","last_became_active":"2024-03-11T21:25:44.590333+0000","last_became_peered":"2024-03-11T21:25:44.590333+0000","last_unstale":"2024-03-11T21:40:28.556980+0000","last_undegraded":"2024-03-11T21:40:28.556980+0000","last_fullsized":"2024-03-11T21:40:28.556980+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:35:37.182850+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:35:37.182850+0000","last_clean_scrub_stamp":"2024-03-11T21:35:37.182850+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":40,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.e","version":"74'8","reported_seq":808,"reported_epoch":353,"state":"active+clean","last_fresh":"2024-03-11T21:40:31.955730+0000","last_change":"2024-03-11T21:36:07.734369+0000","last_active":"2024-03-11T21:40:31.955730+0000","last_peered":"2024-03-11T21:40:31.955730+0000","last_clean":"2024-03-11T21:40:31.955730+0000","last_became_active":"2024-03-11T21:25:42.576742+0000","last_became_peered":"2024-03-11T21:25:42.576742+0000","last_unstale":"2024-03-11T21:40:31.955730+0000","last_undegraded":"2024-03-11T21:40:31.955730+0000","last_fullsized":"2024-03-11T21:40:31.955730+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'8","last_scrub_stamp":"2024-03-11T21:36:07.734115+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:41.544550+0000","last_clean_scrub_stamp":"2024-03-11T21:36:07.734115+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,0],"acting":[4,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.9","version":"0'0","reported_seq":650,"reported_epoch":353,"state":"active+clean","last_fresh":"2024-03-11T21:40:31.950024+0000","last_change":"2024-03-11T21:40:29.915912+0000","last_active":"2024-03-11T21:40:31.950024+0000","last_peered":"2024-03-11T21:40:31.950024+0000","last_clean":"2024-03-11T21:40:31.950024+0000","last_became_active":"2024-03-11T21:36:35.065733+0000","last_became_peered":"2024-03-11T21:36:35.065733+0000","last_unstale":"2024-03-11T21:40:31.950024+0000","last_undegraded":"2024-03-11T21:40:31.950024+0000","last_fullsized":"2024-03-11T21:40:31.950024+0000","mapping_epoch":277,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":278,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:40:29.915835+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:26:50.991712+0000","last_clean_scrub_stamp":"2024-03-11T21:40:29.915835+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:41:46.722143+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,3],"acting":[1,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.f","version":"260'27","reported_seq":649,"reported_epoch":352,"state":"active+clean","last_fresh":"2024-03-11T21:40:28.388326+0000","last_change":"2024-03-11T21:40:12.838694+0000","last_active":"2024-03-11T21:40:28.388326+0000","last_peered":"2024-03-11T21:40:28.388326+0000","last_clean":"2024-03-11T21:40:28.388326+0000","last_became_active":"2024-03-11T21:36:35.058272+0000","last_became_peered":"2024-03-11T21:36:35.058272+0000","last_unstale":"2024-03-11T21:40:28.388326+0000","last_undegraded":"2024-03-11T21:40:28.388326+0000","last_fullsized":"2024-03-11T21:40:28.388326+0000","mapping_epoch":277,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":278,"parent":"0.0","parent_split_bits":0,"last_scrub":"260'27","last_scrub_stamp":"2024-03-11T21:40:12.838597+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:39.525554+0000","last_clean_scrub_stamp":"2024-03-11T21:40:12.838597+0000","objects_scrubbed":7,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:41:21.967430+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":44,"num_read_kb":35,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,4],"acting":[2,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.e","version":"0'0","reported_seq":617,"reported_epoch":353,"state":"active+clean","last_fresh":"2024-03-11T21:40:31.955816+0000","last_change":"2024-03-11T21:36:10.769557+0000","last_active":"2024-03-11T21:40:31.955816+0000","last_peered":"2024-03-11T21:40:31.955816+0000","last_clean":"2024-03-11T21:40:31.955816+0000","last_became_active":"2024-03-11T21:25:44.590198+0000","last_became_peered":"2024-03-11T21:25:44.590198+0000","last_unstale":"2024-03-11T21:40:31.955816+0000","last_undegraded":"2024-03-11T21:40:31.955816+0000","last_fullsized":"2024-03-11T21:40:31.955816+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:36:10.769460+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:43.559452+0000","last_clean_scrub_stamp":"2024-03-11T21:36:10.769460+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":75,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.d","version":"109'9","reported_seq":805,"reported_epoch":352,"state":"active+clean","last_fresh":"2024-03-11T21:40:28.388436+0000","last_change":"2024-03-11T21:35:07.562523+0000","last_active":"2024-03-11T21:40:28.388436+0000","last_peered":"2024-03-11T21:40:28.388436+0000","last_clean":"2024-03-11T21:40:28.388436+0000","last_became_active":"2024-03-11T21:25:42.575583+0000","last_became_peered":"2024-03-11T21:25:42.575583+0000","last_unstale":"2024-03-11T21:40:28.388436+0000","last_undegraded":"2024-03-11T21:40:28.388436+0000","last_fullsized":"2024-03-11T21:40:28.388436+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"109'9","last_scrub_stamp":"2024-03-11T21:35:07.562385+0000","last_deep_scrub":"109'9","last_deep_scrub_stamp":"2024-03-11T21:28:27.082975+0000","last_clean_scrub_stamp":"2024-03-11T21:35:07.562385+0000","objects_scrubbed":1,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,5],"acting":[2,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.a","version":"0'0","reported_seq":650,"reported_epoch":353,"state":"active+clean","last_fresh":"2024-03-11T21:40:31.950513+0000","last_change":"2024-03-11T21:36:35.067262+0000","last_active":"2024-03-11T21:40:31.950513+0000","last_peered":"2024-03-11T21:40:31.950513+0000","last_clean":"2024-03-11T21:40:31.950513+0000","last_became_active":"2024-03-11T21:36:35.066832+0000","last_became_peered":"2024-03-11T21:36:35.066832+0000","last_unstale":"2024-03-11T21:40:31.950513+0000","last_undegraded":"2024-03-11T21:40:31.950513+0000","last_fullsized":"2024-03-11T21:40:31.950513+0000","mapping_epoch":277,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":278,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:34:22.526916+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:37.512018+0000","last_clean_scrub_stamp":"2024-03-11T21:34:22.526916+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,7],"acting":[1,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.c","version":"260'20","reported_seq":677,"reported_epoch":353,"state":"active+clean","last_fresh":"2024-03-11T21:40:31.955359+0000","last_change":"2024-03-11T21:40:02.706928+0000","last_active":"2024-03-11T21:40:31.955359+0000","last_peered":"2024-03-11T21:40:31.955359+0000","last_clean":"2024-03-11T21:40:31.955359+0000","last_became_active":"2024-03-11T21:36:35.067269+0000","last_became_peered":"2024-03-11T21:36:35.067269+0000","last_unstale":"2024-03-11T21:40:31.955359+0000","last_undegraded":"2024-03-11T21:40:31.955359+0000","last_fullsized":"2024-03-11T21:40:31.955359+0000","mapping_epoch":277,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":278,"parent":"0.0","parent_split_bits":0,"last_scrub":"260'20","last_scrub_stamp":"2024-03-11T21:40:02.706832+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:39.525554+0000","last_clean_scrub_stamp":"2024-03-11T21:40:02.706832+0000","objects_scrubbed":5,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:41:04.628603+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":25,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.f","version":"0'0","reported_seq":622,"reported_epoch":352,"state":"active+clean","last_fresh":"2024-03-11T21:40:28.388160+0000","last_change":"2024-03-11T21:36:35.058630+0000","last_active":"2024-03-11T21:40:28.388160+0000","last_peered":"2024-03-11T21:40:28.388160+0000","last_clean":"2024-03-11T21:40:28.388160+0000","last_became_active":"2024-03-11T21:36:35.057800+0000","last_became_peered":"2024-03-11T21:36:35.057800+0000","last_unstale":"2024-03-11T21:40:28.388160+0000","last_undegraded":"2024-03-11T21:40:28.388160+0000","last_fullsized":"2024-03-11T21:40:28.388160+0000","mapping_epoch":277,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":278,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:35:10.608981+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:27:03.010969+0000","last_clean_scrub_stamp":"2024-03-11T21:35:10.608981+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":94,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,4],"acting":[2,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":621,"reported_epoch":353,"state":"active+clean","last_fresh":"2024-03-11T21:40:31.950486+0000","last_change":"2024-03-11T21:36:10.608789+0000","last_active":"2024-03-11T21:40:31.950486+0000","last_peered":"2024-03-11T21:40:31.950486+0000","last_clean":"2024-03-11T21:40:31.950486+0000","last_became_active":"2024-03-11T21:25:42.579409+0000","last_became_peered":"2024-03-11T21:25:42.579409+0000","last_unstale":"2024-03-11T21:40:31.950486+0000","last_undegraded":"2024-03-11T21:40:31.950486+0000","last_fullsized":"2024-03-11T21:40:31.950486+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:36:10.608552+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:30:22.750053+0000","last_clean_scrub_stamp":"2024-03-11T21:36:10.608552+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.b","version":"0'0","reported_seq":271,"reported_epoch":352,"state":"active+clean","last_fresh":"2024-03-11T21:40:28.457688+0000","last_change":"2024-03-11T21:36:16.719432+0000","last_active":"2024-03-11T21:40:28.457688+0000","last_peered":"2024-03-11T21:40:28.457688+0000","last_clean":"2024-03-11T21:40:28.457688+0000","last_became_active":"2024-03-11T21:34:41.134987+0000","last_became_peered":"2024-03-11T21:34:41.134987+0000","last_unstale":"2024-03-11T21:40:28.457688+0000","last_undegraded":"2024-03-11T21:40:28.457688+0000","last_fullsized":"2024-03-11T21:40:28.457688+0000","mapping_epoch":226,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":227,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:36:16.719309+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:36:16.719309+0000","last_clean_scrub_stamp":"2024-03-11T21:36:16.719309+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,5],"acting":[7,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.d","version":"260'39","reported_seq":682,"reported_epoch":353,"state":"active+clean","last_fresh":"2024-03-11T21:40:31.955598+0000","last_change":"2024-03-11T21:35:25.543769+0000","last_active":"2024-03-11T21:40:31.955598+0000","last_peered":"2024-03-11T21:40:31.955598+0000","last_clean":"2024-03-11T21:40:31.955598+0000","last_became_active":"2024-03-11T21:25:40.553454+0000","last_became_peered":"2024-03-11T21:25:40.553454+0000","last_unstale":"2024-03-11T21:40:31.955598+0000","last_undegraded":"2024-03-11T21:40:31.955598+0000","last_fullsized":"2024-03-11T21:40:31.955598+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"165'25","last_scrub_stamp":"2024-03-11T21:35:25.543628+0000","last_deep_scrub":"165'25","last_deep_scrub_stamp":"2024-03-11T21:32:21.678328+0000","last_clean_scrub_stamp":"2024-03-11T21:35:25.543628+0000","objects_scrubbed":8,"log_size":39,"log_dups_size":0,"ondisk_log_size":39,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":50,"num_read_kb":40,"num_write":32,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,1],"acting":[4,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":271,"reported_epoch":352,"state":"active+clean","last_fresh":"2024-03-11T21:40:28.457204+0000","last_change":"2024-03-11T21:36:06.735035+0000","last_active":"2024-03-11T21:40:28.457204+0000","last_peered":"2024-03-11T21:40:28.457204+0000","last_clean":"2024-03-11T21:40:28.457204+0000","last_became_active":"2024-03-11T21:34:41.145012+0000","last_became_peered":"2024-03-11T21:34:41.145012+0000","last_unstale":"2024-03-11T21:40:28.457204+0000","last_undegraded":"2024-03-11T21:40:28.457204+0000","last_fullsized":"2024-03-11T21:40:28.457204+0000","mapping_epoch":226,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":227,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:36:06.734921+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:28:37.561826+0000","last_clean_scrub_stamp":"2024-03-11T21:36:06.734921+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":23,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,0],"acting":[7,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":617,"reported_epoch":352,"state":"active+clean","last_fresh":"2024-03-11T21:40:28.387796+0000","last_change":"2024-03-11T21:35:14.636561+0000","last_active":"2024-03-11T21:40:28.387796+0000","last_peered":"2024-03-11T21:40:28.387796+0000","last_clean":"2024-03-11T21:40:28.387796+0000","last_became_active":"2024-03-11T21:25:42.574515+0000","last_became_peered":"2024-03-11T21:25:42.574515+0000","last_unstale":"2024-03-11T21:40:28.387796+0000","last_undegraded":"2024-03-11T21:40:28.387796+0000","last_fullsized":"2024-03-11T21:40:28.387796+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:35:14.636450+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:41.544550+0000","last_clean_scrub_stamp":"2024-03-11T21:35:14.636450+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,5],"acting":[2,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.c","version":"0'0","reported_seq":625,"reported_epoch":352,"state":"active+clean","last_fresh":"2024-03-11T21:40:28.388529+0000","last_change":"2024-03-11T21:36:00.849272+0000","last_active":"2024-03-11T21:40:28.388529+0000","last_peered":"2024-03-11T21:40:28.388529+0000","last_clean":"2024-03-11T21:40:28.388529+0000","last_became_active":"2024-03-11T21:25:38.533240+0000","last_became_peered":"2024-03-11T21:25:38.533240+0000","last_unstale":"2024-03-11T21:40:28.388529+0000","last_undegraded":"2024-03-11T21:40:28.388529+0000","last_fullsized":"2024-03-11T21:40:28.388529+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:36:00.849153+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:29:30.279910+0000","last_clean_scrub_stamp":"2024-03-11T21:36:00.849153+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,0],"acting":[2,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.a","version":"260'44","reported_seq":763,"reported_epoch":352,"state":"active+clean","last_fresh":"2024-03-11T21:40:28.556441+0000","last_change":"2024-03-11T21:40:02.183543+0000","last_active":"2024-03-11T21:40:28.556441+0000","last_peered":"2024-03-11T21:40:28.556441+0000","last_clean":"2024-03-11T21:40:28.556441+0000","last_became_active":"2024-03-11T21:34:41.148863+0000","last_became_peered":"2024-03-11T21:34:41.148863+0000","last_unstale":"2024-03-11T21:40:28.556441+0000","last_undegraded":"2024-03-11T21:40:28.556441+0000","last_fullsized":"2024-03-11T21:40:28.556441+0000","mapping_epoch":226,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":227,"parent":"0.0","parent_split_bits":0,"last_scrub":"260'44","last_scrub_stamp":"2024-03-11T21:40:02.183417+0000","last_deep_scrub":"109'28","last_deep_scrub_stamp":"2024-03-11T21:28:24.016389+0000","last_clean_scrub_stamp":"2024-03-11T21:40:02.183417+0000","objects_scrubbed":9,"log_size":44,"log_dups_size":0,"ondisk_log_size":44,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:41:31.089360+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":56,"num_read_kb":45,"num_write":36,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,7],"acting":[6,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":631,"reported_epoch":354,"state":"active+clean","last_fresh":"2024-03-11T21:40:33.391271+0000","last_change":"2024-03-11T21:35:37.687281+0000","last_active":"2024-03-11T21:40:33.391271+0000","last_peered":"2024-03-11T21:40:33.391271+0000","last_clean":"2024-03-11T21:40:33.391271+0000","last_became_active":"2024-03-11T21:34:41.154507+0000","last_became_peered":"2024-03-11T21:34:41.154507+0000","last_unstale":"2024-03-11T21:40:33.391271+0000","last_undegraded":"2024-03-11T21:40:33.391271+0000","last_fullsized":"2024-03-11T21:40:33.391271+0000","mapping_epoch":226,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":227,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:35:37.687187+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:27:11.361913+0000","last_clean_scrub_stamp":"2024-03-11T21:35:37.687187+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:40:37.147272+0000","scrub_duration":39,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":626,"reported_epoch":352,"state":"active+clean","last_fresh":"2024-03-11T21:40:28.387939+0000","last_change":"2024-03-11T21:40:28.015350+0000","last_active":"2024-03-11T21:40:28.387939+0000","last_peered":"2024-03-11T21:40:28.387939+0000","last_clean":"2024-03-11T21:40:28.387939+0000","last_became_active":"2024-03-11T21:36:35.059192+0000","last_became_peered":"2024-03-11T21:36:35.059192+0000","last_unstale":"2024-03-11T21:40:28.387939+0000","last_undegraded":"2024-03-11T21:40:28.387939+0000","last_fullsized":"2024-03-11T21:40:28.387939+0000","mapping_epoch":277,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":278,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:40:28.015231+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:41.544550+0000","last_clean_scrub_stamp":"2024-03-11T21:40:28.015231+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:41:54.304353+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,3],"acting":[2,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.d","version":"0'0","reported_seq":638,"reported_epoch":353,"state":"active+clean","last_fresh":"2024-03-11T21:40:31.950887+0000","last_change":"2024-03-11T21:40:31.873781+0000","last_active":"2024-03-11T21:40:31.950887+0000","last_peered":"2024-03-11T21:40:31.950887+0000","last_clean":"2024-03-11T21:40:31.950887+0000","last_became_active":"2024-03-11T21:36:35.058296+0000","last_became_peered":"2024-03-11T21:36:35.058296+0000","last_unstale":"2024-03-11T21:40:31.950887+0000","last_undegraded":"2024-03-11T21:40:31.950887+0000","last_fullsized":"2024-03-11T21:40:31.950887+0000","mapping_epoch":277,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":278,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:40:31.873695+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:37.512018+0000","last_clean_scrub_stamp":"2024-03-11T21:40:31.873695+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:41:53.890729+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,3],"acting":[1,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.b","version":"260'15","reported_seq":652,"reported_epoch":354,"state":"active+clean","last_fresh":"2024-03-11T21:40:33.391565+0000","last_change":"2024-03-11T21:35:27.760318+0000","last_active":"2024-03-11T21:40:33.391565+0000","last_peered":"2024-03-11T21:40:33.391565+0000","last_clean":"2024-03-11T21:40:33.391565+0000","last_became_active":"2024-03-11T21:25:40.549612+0000","last_became_peered":"2024-03-11T21:25:40.549612+0000","last_unstale":"2024-03-11T21:40:33.391565+0000","last_undegraded":"2024-03-11T21:40:33.391565+0000","last_fullsized":"2024-03-11T21:40:33.391565+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'9","last_scrub_stamp":"2024-03-11T21:35:27.760208+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:39.525554+0000","last_clean_scrub_stamp":"2024-03-11T21:35:27.760208+0000","objects_scrubbed":4,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:40:37.147307+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":531,"reported_epoch":353,"state":"active+clean","last_fresh":"2024-03-11T21:40:31.954839+0000","last_change":"2024-03-11T21:36:35.057118+0000","last_active":"2024-03-11T21:40:31.954839+0000","last_peered":"2024-03-11T21:40:31.954839+0000","last_clean":"2024-03-11T21:40:31.954839+0000","last_became_active":"2024-03-11T21:36:35.056518+0000","last_became_peered":"2024-03-11T21:36:35.056518+0000","last_unstale":"2024-03-11T21:40:31.954839+0000","last_undegraded":"2024-03-11T21:40:31.954839+0000","last_fullsized":"2024-03-11T21:40:31.954839+0000","mapping_epoch":277,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":278,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:35:24.514584+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:43.559452+0000","last_clean_scrub_stamp":"2024-03-11T21:35:24.514584+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,3],"acting":[4,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.e","version":"0'0","reported_seq":654,"reported_epoch":352,"state":"active+clean","last_fresh":"2024-03-11T21:40:28.388333+0000","last_change":"2024-03-11T21:40:11.026129+0000","last_active":"2024-03-11T21:40:28.388333+0000","last_peered":"2024-03-11T21:40:28.388333+0000","last_clean":"2024-03-11T21:40:28.388333+0000","last_became_active":"2024-03-11T21:36:35.056777+0000","last_became_peered":"2024-03-11T21:36:35.056777+0000","last_unstale":"2024-03-11T21:40:28.388333+0000","last_undegraded":"2024-03-11T21:40:28.388333+0000","last_fullsized":"2024-03-11T21:40:28.388333+0000","mapping_epoch":277,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":278,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:40:11.025986+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:37.512018+0000","last_clean_scrub_stamp":"2024-03-11T21:40:11.025986+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:41:21.568389+0000","scrub_duration":114,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,7],"acting":[2,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.8","version":"260'34","reported_seq":716,"reported_epoch":352,"state":"active+clean","last_fresh":"2024-03-11T21:40:28.555588+0000","last_change":"2024-03-11T21:35:20.304345+0000","last_active":"2024-03-11T21:40:28.555588+0000","last_peered":"2024-03-11T21:40:28.555588+0000","last_clean":"2024-03-11T21:40:28.555588+0000","last_became_active":"2024-03-11T21:34:41.146631+0000","last_became_peered":"2024-03-11T21:34:41.146631+0000","last_unstale":"2024-03-11T21:40:28.555588+0000","last_undegraded":"2024-03-11T21:40:28.555588+0000","last_fullsized":"2024-03-11T21:40:28.555588+0000","mapping_epoch":226,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":227,"parent":"0.0","parent_split_bits":0,"last_scrub":"109'22","last_scrub_stamp":"2024-03-11T21:35:20.304114+0000","last_deep_scrub":"109'22","last_deep_scrub_stamp":"2024-03-11T21:32:07.573197+0000","last_clean_scrub_stamp":"2024-03-11T21:35:20.304114+0000","objects_scrubbed":7,"log_size":34,"log_dups_size":0,"ondisk_log_size":34,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":44,"num_read_kb":35,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.9","version":"229'11","reported_seq":345,"reported_epoch":352,"state":"active+clean","last_fresh":"2024-03-11T21:40:28.457542+0000","last_change":"2024-03-11T21:36:18.726876+0000","last_active":"2024-03-11T21:40:28.457542+0000","last_peered":"2024-03-11T21:40:28.457542+0000","last_clean":"2024-03-11T21:40:28.457542+0000","last_became_active":"2024-03-11T21:34:41.156892+0000","last_became_peered":"2024-03-11T21:34:41.156892+0000","last_unstale":"2024-03-11T21:40:28.457542+0000","last_undegraded":"2024-03-11T21:40:28.457542+0000","last_fullsized":"2024-03-11T21:40:28.457542+0000","mapping_epoch":226,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":227,"parent":"0.0","parent_split_bits":0,"last_scrub":"229'11","last_scrub_stamp":"2024-03-11T21:36:18.726763+0000","last_deep_scrub":"94'9","last_deep_scrub_stamp":"2024-03-11T21:26:53.422552+0000","last_clean_scrub_stamp":"2024-03-11T21:36:18.726763+0000","objects_scrubbed":1,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":1531,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":17,"num_read_kb":17,"num_write":10,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":1308,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":4593,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":6,"ondisk_log_size":6,"up":96,"acting":96,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":198,"num_read_kb":294,"num_write":179,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":2951840,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":2375680,"data_stored":2361472,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":128,"ondisk_log_size":128,"up":3,"acting":3,"num_store_stats":4},{"poolid":3,"num_pg":3,"stat_sum":{"num_bytes":408,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":887,"num_read_kb":882,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1224,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":8,"ondisk_log_size":8,"up":9,"acting":9,"num_store_stats":7},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":3702,"num_objects":178,"num_object_clones":0,"num_object_copies":534,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":178,"num_whiteouts":0,"num_read":1120,"num_read_kb":877,"num_write":703,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":186,"num_bytes_recovered":2200,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":417792,"data_stored":11106,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":770,"ondisk_log_size":770,"up":96,"acting":96,"num_store_stats":8},{"poolid":5,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":78,"ondisk_log_size":78,"up":96,"acting":96,"num_store_stats":8},{"poolid":6,"num_pg":32,"stat_sum":{"num_bytes":383,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":370,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1149,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":96,"acting":96,"num_store_stats":8}],"osd_stats":[{"osd":7,"up_from":60,"seq":257698037942,"num_pgs":56,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":511464,"kb_used_data":2720,"kb_used_omap":1,"kb_used_meta":508734,"kb_avail":93233688,"statfs":{"total":95995035648,"available":95471296512,"internally_reserved":0,"allocated":2785280,"data_stored":1831363,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":520944074},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":50,"seq":214748364988,"num_pgs":43,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":412908,"kb_used_data":2720,"kb_used_omap":1,"kb_used_meta":410174,"kb_avail":93332244,"statfs":{"total":95995035648,"available":95572217856,"internally_reserved":0,"allocated":2785280,"data_stored":1832092,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":420018630},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834575062,"num_pgs":51,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":578960,"kb_used_data":2180,"kb_used_omap":1,"kb_used_meta":576766,"kb_avail":93166192,"statfs":{"total":95995035648,"available":95402180608,"internally_reserved":0,"allocated":2232320,"data_stored":1248359,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":590608837},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738587,"num_pgs":50,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":565636,"kb_used_data":2744,"kb_used_omap":1,"kb_used_meta":562878,"kb_avail":93179516,"statfs":{"total":95995035648,"available":95415824384,"internally_reserved":0,"allocated":2809856,"data_stored":1838763,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":576387525},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411536,"num_pgs":40,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":438688,"kb_used_data":2136,"kb_used_omap":1,"kb_used_meta":436542,"kb_avail":93306464,"statfs":{"total":95995035648,"available":95545819136,"internally_reserved":0,"allocated":2187264,"data_stored":1241492,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":447019461},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":277,"seq":1189705941041,"num_pgs":54,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":278804,"kb_used_data":2184,"kb_used_omap":4,"kb_used_meta":276603,"kb_avail":93466348,"statfs":{"total":95995035648,"available":95709540352,"internally_reserved":0,"allocated":2236416,"data_stored":1248803,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4442,"internal_metadata":283242150},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":32,"seq":137438953669,"num_pgs":53,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":424820,"kb_used_data":2156,"kb_used_omap":1,"kb_used_meta":422654,"kb_avail":93320332,"statfs":{"total":95995035648,"available":95560019968,"internally_reserved":0,"allocated":2207744,"data_stored":1241340,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":432798149},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":41,"seq":176093659328,"num_pgs":49,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":552544,"kb_used_data":2136,"kb_used_omap":1,"kb_used_meta":550398,"kb_avail":93192608,"statfs":{"total":95995035648,"available":95429230592,"internally_reserved":0,"allocated":2187264,"data_stored":1241107,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":563608005},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1049,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":46,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":436,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":957,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":574,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":436,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1003,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":92,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":1430,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":77824,"data_stored":2008,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":1018,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":1898,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":1474,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":990,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":1034,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":1254,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":370,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":383,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":370,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-11T21:40:38.178 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:40:38.179 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:40:38.179 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:40:38.216 INFO:teuthology.orchestra.run.smithi092.stderr:pool 'unique_pool_3' created 2024-03-11T21:40:38.217 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:40:37 smithi092 ceph-mon[29572]: from='mgr.14607 v1:172.21.15.92:0/1702717597' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T21:40:38.217 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:40:37 smithi092 ceph-mon[29572]: from='mgr.14607 v1:172.21.15.92:0/1702717597' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T21:40:38.217 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:40:37 smithi092 ceph-mon[29572]: from='mgr.14607 ' entity='mgr.y' 2024-03-11T21:40:38.217 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:40:37 smithi092 ceph-mon[29572]: purged_snaps scrub starts 2024-03-11T21:40:38.217 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:40:37 smithi092 ceph-mon[29572]: purged_snaps scrub ok 2024-03-11T21:40:38.217 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:40:37 smithi092 ceph-mon[29572]: from='mgr.14607 ' entity='mgr.y' 2024-03-11T21:40:38.217 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:40:37 smithi092 ceph-mon[29572]: from='mgr.14607 v1:172.21.15.92:0/1702717597' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T21:40:38.217 INFO:journalctl@ceph.mgr.y.smithi092.stdout:Mar 11 21:40:37 smithi092 ceph-00db0016-dfed-11ee-95c8-87774f69a715-mgr-y[29801]: ::ffff:172.21.15.97 - - [11/Mar/2024:21:40:37] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-11T21:40:38.218 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:40:37 smithi092 ceph-mon[37563]: from='mgr.14607 v1:172.21.15.92:0/1702717597' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T21:40:38.218 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:40:37 smithi092 ceph-mon[37563]: from='mgr.14607 v1:172.21.15.92:0/1702717597' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T21:40:38.218 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:40:37 smithi092 ceph-mon[37563]: from='mgr.14607 ' entity='mgr.y' 2024-03-11T21:40:38.218 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:40:37 smithi092 ceph-mon[37563]: purged_snaps scrub starts 2024-03-11T21:40:38.218 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:40:37 smithi092 ceph-mon[37563]: purged_snaps scrub ok 2024-03-11T21:40:38.218 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:40:37 smithi092 ceph-mon[37563]: from='mgr.14607 ' entity='mgr.y' 2024-03-11T21:40:38.218 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:40:37 smithi092 ceph-mon[37563]: from='mgr.14607 v1:172.21.15.92:0/1702717597' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T21:40:38.280 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:40:38.304 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:40:38.304 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:40:38.304 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:40:38.330 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:40:37 smithi097 ceph-mon[33317]: from='mgr.14607 v1:172.21.15.92:0/1702717597' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T21:40:38.330 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:40:37 smithi097 ceph-mon[33317]: from='mgr.14607 v1:172.21.15.92:0/1702717597' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T21:40:38.330 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:40:37 smithi097 ceph-mon[33317]: from='mgr.14607 ' entity='mgr.y' 2024-03-11T21:40:38.330 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:40:37 smithi097 ceph-mon[33317]: purged_snaps scrub starts 2024-03-11T21:40:38.330 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:40:37 smithi097 ceph-mon[33317]: purged_snaps scrub ok 2024-03-11T21:40:38.330 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:40:37 smithi097 ceph-mon[33317]: from='mgr.14607 ' entity='mgr.y' 2024-03-11T21:40:38.330 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:40:37 smithi097 ceph-mon[33317]: from='mgr.14607 v1:172.21.15.92:0/1702717597' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T21:40:38.405 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:40:38.429 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:40:38.429 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:40:38.430 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:40:38.530 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:40:38.555 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:40:38.556 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:40:38.556 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:40:38.640 INFO:teuthology.orchestra.run.smithi092.stderr:Inferring config /var/lib/ceph/00db0016-dfed-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T21:40:38.657 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:40:38.681 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:40:38.682 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:40:38.683 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:40:38.783 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:40:38.807 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:40:38.808 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:40:38.809 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:40:38.909 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:40:38.935 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:40:38.936 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:40:38.936 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:40:39.037 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:40:39.061 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:40:39.061 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:40:39.062 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:40:39.092 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:40:38 smithi092 ceph-mon[37563]: pgmap v699: 132 pgs: 132 active+clean; 582 KiB data, 3.6 GiB used, 712 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-11T21:40:39.092 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:40:38 smithi092 ceph-mon[37563]: from='client.? v1:172.21.15.92:0/3615972364' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "unique_pool_3", "pg_num": 16}]: dispatch 2024-03-11T21:40:39.092 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:40:38 smithi092 ceph-mon[37563]: from='client.35561 ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "unique_pool_3", "pg_num": 16}]: dispatch 2024-03-11T21:40:39.092 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:40:38 smithi092 ceph-mon[37563]: from='client.35561 ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "unique_pool_3", "pg_num": 16}]': finished 2024-03-11T21:40:39.092 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:40:38 smithi092 ceph-mon[37563]: osdmap e356: 8 total, 8 up, 8 in 2024-03-11T21:40:39.092 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:40:38 smithi092 ceph-mon[29572]: pgmap v699: 132 pgs: 132 active+clean; 582 KiB data, 3.6 GiB used, 712 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-11T21:40:39.092 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:40:38 smithi092 ceph-mon[29572]: from='client.? v1:172.21.15.92:0/3615972364' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "unique_pool_3", "pg_num": 16}]: dispatch 2024-03-11T21:40:39.092 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:40:38 smithi092 ceph-mon[29572]: from='client.35561 ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "unique_pool_3", "pg_num": 16}]: dispatch 2024-03-11T21:40:39.092 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:40:38 smithi092 ceph-mon[29572]: from='client.35561 ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "unique_pool_3", "pg_num": 16}]': finished 2024-03-11T21:40:39.093 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:40:38 smithi092 ceph-mon[29572]: osdmap e356: 8 total, 8 up, 8 in 2024-03-11T21:40:39.162 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:40:39.187 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:40:39.187 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:40:39.187 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:40:39.290 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:40:39.313 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:40:39.314 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:40:39.314 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:40:39.330 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:40:38 smithi097 ceph-mon[33317]: pgmap v699: 132 pgs: 132 active+clean; 582 KiB data, 3.6 GiB used, 712 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-11T21:40:39.330 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:40:38 smithi097 ceph-mon[33317]: from='client.? v1:172.21.15.92:0/3615972364' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "unique_pool_3", "pg_num": 16}]: dispatch 2024-03-11T21:40:39.330 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:40:38 smithi097 ceph-mon[33317]: from='client.35561 ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "unique_pool_3", "pg_num": 16}]: dispatch 2024-03-11T21:40:39.330 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:40:38 smithi097 ceph-mon[33317]: from='client.35561 ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "unique_pool_3", "pg_num": 16}]': finished 2024-03-11T21:40:39.330 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:40:38 smithi097 ceph-mon[33317]: osdmap e356: 8 total, 8 up, 8 in 2024-03-11T21:40:39.413 DEBUG:teuthology.orchestra.run.smithi092:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid 00db0016-dfed-11ee-95c8-87774f69a715 -- ceph osd pool application enable unique_pool_3 rados --yes-i-really-mean-it || true 2024-03-11T21:40:39.415 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:40:39.444 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:40:39.444 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:40:39.445 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:40:39.545 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:40:39.574 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:40:39.575 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:40:39.575 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:40:39.676 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:40:39.704 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:40:39.705 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:40:39.705 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:40:39.806 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:40:39.831 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:40:39.831 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:40:39.832 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:40:39.933 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:40:39.956 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:40:39.957 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:40:39.958 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:40:40.049 INFO:teuthology.orchestra.run.smithi092.stderr:Inferring config /var/lib/ceph/00db0016-dfed-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T21:40:40.058 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:40:40.086 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:40:40.087 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:40:40.087 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:40:40.188 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:40:40.217 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:40:40.217 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:40:40.218 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:40:40.318 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:40:40.343 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:40:40.343 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:40:40.344 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:40:40.364 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:40:40 smithi092 ceph-mon[37563]: osdmap e357: 8 total, 8 up, 8 in 2024-03-11T21:40:40.364 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:40:40 smithi092 ceph-mon[37563]: pgmap v702: 148 pgs: 3 creating+peering, 13 unknown, 132 active+clean; 582 KiB data, 3.6 GiB used, 712 GiB / 715 GiB avail; 863 B/s rd, 0 op/s 2024-03-11T21:40:40.364 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:40:40 smithi092 ceph-mon[29572]: osdmap e357: 8 total, 8 up, 8 in 2024-03-11T21:40:40.364 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:40:40 smithi092 ceph-mon[29572]: pgmap v702: 148 pgs: 3 creating+peering, 13 unknown, 132 active+clean; 582 KiB data, 3.6 GiB used, 712 GiB / 715 GiB avail; 863 B/s rd, 0 op/s 2024-03-11T21:40:40.444 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:40:40.472 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:40:40.473 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:40:40.473 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:40:40.574 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:40:40.579 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:40:40 smithi097 ceph-mon[33317]: osdmap e357: 8 total, 8 up, 8 in 2024-03-11T21:40:40.579 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:40:40 smithi097 ceph-mon[33317]: pgmap v702: 148 pgs: 3 creating+peering, 13 unknown, 132 active+clean; 582 KiB data, 3.6 GiB used, 712 GiB / 715 GiB avail; 863 B/s rd, 0 op/s 2024-03-11T21:40:40.602 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:40:40.603 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:40:40.603 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:40:40.704 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:40:40.728 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:40:40.729 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:40:40.729 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:40:40.830 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:40:40.858 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:40:40.859 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:40:40.859 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:40:40.960 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:40:40.989 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:40:40.989 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:40:40.990 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:40:41.091 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:40:41.115 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:40:41.116 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:40:41.116 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:40:41.130 INFO:teuthology.orchestra.run.smithi092.stdout:ERROR: (22) Invalid argument 2024-03-11T21:40:41.130 INFO:teuthology.orchestra.run.smithi092.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-11T21:40:41.167 DEBUG:teuthology.orchestra.run.smithi092:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid 00db0016-dfed-11ee-95c8-87774f69a715 -- ceph pg dump --format=json 2024-03-11T21:40:41.217 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:40:41.241 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:40:41.241 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:40:41.242 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:40:41.343 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:40:41.371 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:40:41.371 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:40:41.372 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:40:41.473 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:40:41.496 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:40:41.497 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:40:41.498 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:40:41.580 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:40:41 smithi097 ceph-mon[33317]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-11T21:40:41.580 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:40:41 smithi097 ceph-mon[33317]: osdmap e358: 8 total, 8 up, 8 in 2024-03-11T21:40:41.598 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:40:41.614 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:40:41 smithi092 ceph-mon[29572]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-11T21:40:41.614 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:40:41 smithi092 ceph-mon[29572]: osdmap e358: 8 total, 8 up, 8 in 2024-03-11T21:40:41.614 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:40:41 smithi092 ceph-mon[37563]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-11T21:40:41.614 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:40:41 smithi092 ceph-mon[37563]: osdmap e358: 8 total, 8 up, 8 in 2024-03-11T21:40:41.626 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:40:41.627 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:40:41.627 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:40:41.728 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:40:41.756 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:40:41.756 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:40:41.757 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:40:41.858 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:40:41.885 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:40:41.885 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:40:41.886 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:40:41.987 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:40:42.016 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:40:42.017 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:40:42.017 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:40:42.038 DEBUG:teuthology.orchestra.run.smithi092:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid 00db0016-dfed-11ee-95c8-87774f69a715 -- ceph daemon osd.1 dump_blocked_ops 2024-03-11T21:40:42.119 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:40:42.145 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:40:42.175 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:40:42.176 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:40:42.277 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:40:42.305 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:40:42.306 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:40:42.306 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:40:42.365 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:40:42 smithi092 ceph-mon[29572]: pgmap v704: 148 pgs: 3 creating+peering, 13 unknown, 132 active+clean; 582 KiB data, 3.6 GiB used, 712 GiB / 715 GiB avail; 955 B/s rd, 0 op/s 2024-03-11T21:40:42.366 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:40:42 smithi092 ceph-mon[37563]: pgmap v704: 148 pgs: 3 creating+peering, 13 unknown, 132 active+clean; 582 KiB data, 3.6 GiB used, 712 GiB / 715 GiB avail; 955 B/s rd, 0 op/s 2024-03-11T21:40:42.407 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:40:42.431 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:40:42.431 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:40:42.432 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:40:42.532 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:40:42.561 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:40:42.561 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:40:42.562 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:40:42.580 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:40:42 smithi097 ceph-mon[33317]: pgmap v704: 148 pgs: 3 creating+peering, 13 unknown, 132 active+clean; 582 KiB data, 3.6 GiB used, 712 GiB / 715 GiB avail; 955 B/s rd, 0 op/s 2024-03-11T21:40:42.662 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:40:42.690 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:40:42.690 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:40:42.690 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:40:42.791 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:40:42.815 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:40:42.816 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:40:42.816 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:40:42.917 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:40:42.944 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:40:42.945 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:40:42.946 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:40:43.007 INFO:teuthology.orchestra.run.smithi092.stderr:Inferring config /var/lib/ceph/00db0016-dfed-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T21:40:43.046 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:40:43.074 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:40:43.074 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:40:43.075 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:40:43.079 INFO:journalctl@ceph.iscsi.iscsi.a.smithi097.stdout:Mar 11 21:40:42 smithi097 ceph-00db0016-dfed-11ee-95c8-87774f69a715-iscsi-iscsi-a[60943]: debug there is no tcmu-runner data available 2024-03-11T21:40:43.176 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:40:43.204 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:40:43.204 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:40:43.205 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:40:43.290 INFO:teuthology.orchestra.run.smithi092.stderr:Inferring config /var/lib/ceph/00db0016-dfed-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T21:40:43.305 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:40:43.329 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:40:43.330 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:40:43.330 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:40:43.405 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:40:43 smithi092 ceph-mon[29572]: purged_snaps scrub starts 2024-03-11T21:40:43.405 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:40:43 smithi092 ceph-mon[29572]: purged_snaps scrub ok 2024-03-11T21:40:43.405 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:40:43 smithi092 ceph-mon[29572]: osdmap e359: 8 total, 8 up, 8 in 2024-03-11T21:40:43.405 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:40:43 smithi092 ceph-mon[29572]: from='client.24469 v1:172.21.15.97:0/404208310' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:40:43.405 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:40:43 smithi092 ceph-mon[37563]: purged_snaps scrub starts 2024-03-11T21:40:43.405 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:40:43 smithi092 ceph-mon[37563]: purged_snaps scrub ok 2024-03-11T21:40:43.405 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:40:43 smithi092 ceph-mon[37563]: osdmap e359: 8 total, 8 up, 8 in 2024-03-11T21:40:43.406 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:40:43 smithi092 ceph-mon[37563]: from='client.24469 v1:172.21.15.97:0/404208310' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:40:43.431 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:40:43.455 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:40:43.456 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:40:43.456 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:40:43.557 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:40:43.580 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:40:43 smithi097 ceph-mon[33317]: purged_snaps scrub starts 2024-03-11T21:40:43.580 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:40:43 smithi097 ceph-mon[33317]: purged_snaps scrub ok 2024-03-11T21:40:43.580 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:40:43 smithi097 ceph-mon[33317]: osdmap e359: 8 total, 8 up, 8 in 2024-03-11T21:40:43.580 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:40:43 smithi097 ceph-mon[33317]: from='client.24469 v1:172.21.15.97:0/404208310' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:40:43.581 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:40:43.581 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:40:43.582 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:40:43.683 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:40:43.711 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:40:43.711 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:40:43.712 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:40:43.813 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:40:43.841 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:40:43.841 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:40:43.842 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:40:43.943 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:40:43.971 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:40:43.971 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:40:43.972 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:40:44.072 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:40:44.097 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:40:44.098 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:40:44.098 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:40:44.199 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:40:44.223 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:40:44.224 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:40:44.225 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:40:44.325 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:40:44.350 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:40:44.351 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:40:44.351 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:40:44.452 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:40:44.480 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:40:44.481 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:40:44.481 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:40:44.582 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:40:44.609 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:40:44.609 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:40:44.610 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:40:44.614 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:40:44 smithi092 ceph-mon[29572]: pgmap v706: 148 pgs: 3 creating+peering, 145 active+clean; 582 KiB data, 3.6 GiB used, 712 GiB / 715 GiB avail; 1004 B/s rd, 0 op/s 2024-03-11T21:40:44.614 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:40:44 smithi092 ceph-mon[29572]: osdmap e360: 8 total, 8 up, 8 in 2024-03-11T21:40:44.614 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:40:44 smithi092 ceph-mon[37563]: pgmap v706: 148 pgs: 3 creating+peering, 145 active+clean; 582 KiB data, 3.6 GiB used, 712 GiB / 715 GiB avail; 1004 B/s rd, 0 op/s 2024-03-11T21:40:44.614 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:40:44 smithi092 ceph-mon[37563]: osdmap e360: 8 total, 8 up, 8 in 2024-03-11T21:40:44.710 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:40:44.739 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:40:44.739 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:40:44.740 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:40:44.746 INFO:teuthology.orchestra.run.smithi092.stderr:Inferring config /var/lib/ceph/00db0016-dfed-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T21:40:44.830 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:40:44 smithi097 ceph-mon[33317]: pgmap v706: 148 pgs: 3 creating+peering, 145 active+clean; 582 KiB data, 3.6 GiB used, 712 GiB / 715 GiB avail; 1004 B/s rd, 0 op/s 2024-03-11T21:40:44.830 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:40:44 smithi097 ceph-mon[33317]: osdmap e360: 8 total, 8 up, 8 in 2024-03-11T21:40:44.840 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:40:44.866 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:40:44.866 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:40:44.866 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:40:44.968 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:40:44.993 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:40:44.994 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:40:44.994 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:40:45.095 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:40:45.124 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:40:45.124 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:40:45.125 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:40:45.225 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:40:45.249 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:40:45.250 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:40:45.250 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:40:45.351 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:40:45.379 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:40:45.380 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:40:45.380 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:40:45.398 INFO:teuthology.orchestra.run.smithi092.stderr:nodeep-scrub is set 2024-03-11T21:40:45.481 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:40:45.505 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:40:45.506 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:40:45.506 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:40:45.607 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:40:45.639 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:40:45.639 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:40:45.640 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:40:45.741 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:40:45.765 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:40:45.765 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:40:45.766 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:40:45.821 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:40:45 smithi092 ceph-mon[29572]: purged_snaps scrub starts 2024-03-11T21:40:45.821 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:40:45 smithi092 ceph-mon[29572]: purged_snaps scrub ok 2024-03-11T21:40:45.821 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:40:45 smithi092 ceph-mon[29572]: osdmap e361: 8 total, 8 up, 8 in 2024-03-11T21:40:45.822 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:40:45 smithi092 ceph-mon[29572]: from='client.? v1:172.21.15.92:0/174839744' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-11T21:40:45.822 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:40:45 smithi092 ceph-mon[37563]: purged_snaps scrub starts 2024-03-11T21:40:45.822 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:40:45 smithi092 ceph-mon[37563]: purged_snaps scrub ok 2024-03-11T21:40:45.822 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:40:45 smithi092 ceph-mon[37563]: osdmap e361: 8 total, 8 up, 8 in 2024-03-11T21:40:45.822 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:40:45 smithi092 ceph-mon[37563]: from='client.? v1:172.21.15.92:0/174839744' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-11T21:40:45.830 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:40:45 smithi097 ceph-mon[33317]: purged_snaps scrub starts 2024-03-11T21:40:45.830 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:40:45 smithi097 ceph-mon[33317]: purged_snaps scrub ok 2024-03-11T21:40:45.830 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:40:45 smithi097 ceph-mon[33317]: osdmap e361: 8 total, 8 up, 8 in 2024-03-11T21:40:45.830 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:40:45 smithi097 ceph-mon[33317]: from='client.? v1:172.21.15.92:0/174839744' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-11T21:40:45.866 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:40:45.894 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:40:45.895 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:40:45.895 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:40:45.996 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:40:46.020 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:40:46.021 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:40:46.021 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:40:46.122 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:40:46.145 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:40:46.146 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:40:46.146 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:40:46.247 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:40:46.271 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:40:46.272 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:40:46.273 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:40:46.374 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:40:46.401 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:40:46.401 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:40:46.401 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:40:46.502 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:40:46.529 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:40:46.529 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:40:46.530 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:40:46.630 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:40:46.655 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:40:46.655 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:40:46.656 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:40:46.685 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:40:46 smithi092 ceph-mon[29572]: pgmap v709: 148 pgs: 148 active+clean; 582 KiB data, 3.6 GiB used, 712 GiB / 715 GiB avail; 1004 B/s rd, 0 op/s 2024-03-11T21:40:46.686 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:40:46 smithi092 ceph-mon[29572]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-11T21:40:46.686 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:40:46 smithi092 ceph-mon[29572]: from='client.? v1:172.21.15.92:0/174839744' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-03-11T21:40:46.686 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:40:46 smithi092 ceph-mon[29572]: osdmap e362: 8 total, 8 up, 8 in 2024-03-11T21:40:46.686 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:40:46 smithi092 ceph-mon[37563]: pgmap v709: 148 pgs: 148 active+clean; 582 KiB data, 3.6 GiB used, 712 GiB / 715 GiB avail; 1004 B/s rd, 0 op/s 2024-03-11T21:40:46.686 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:40:46 smithi092 ceph-mon[37563]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-11T21:40:46.686 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:40:46 smithi092 ceph-mon[37563]: from='client.? v1:172.21.15.92:0/174839744' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-03-11T21:40:46.686 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:40:46 smithi092 ceph-mon[37563]: osdmap e362: 8 total, 8 up, 8 in 2024-03-11T21:40:46.757 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:40:46.781 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:40:46.782 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:40:46.782 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:40:46.830 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:40:46 smithi097 ceph-mon[33317]: pgmap v709: 148 pgs: 148 active+clean; 582 KiB data, 3.6 GiB used, 712 GiB / 715 GiB avail; 1004 B/s rd, 0 op/s 2024-03-11T21:40:46.830 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:40:46 smithi097 ceph-mon[33317]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-11T21:40:46.830 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:40:46 smithi097 ceph-mon[33317]: from='client.? v1:172.21.15.92:0/174839744' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-03-11T21:40:46.830 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:40:46 smithi097 ceph-mon[33317]: osdmap e362: 8 total, 8 up, 8 in 2024-03-11T21:40:46.883 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:40:46.906 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:40:46.906 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:40:46.907 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:40:47.008 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:40:47.032 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:40:47.032 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:40:47.033 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:40:47.133 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:40:47.158 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:40:47.158 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:40:47.159 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:40:47.260 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:40:47.264 INFO:teuthology.orchestra.run.smithi092.stdout:osd.0: {} 2024-03-11T21:40:47.283 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:40:47.284 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:40:47.284 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:40:47.285 INFO:teuthology.orchestra.run.smithi092.stdout:osd.1: {} 2024-03-11T21:40:47.303 INFO:teuthology.orchestra.run.smithi092.stdout:osd.2: {} 2024-03-11T21:40:47.323 INFO:teuthology.orchestra.run.smithi092.stdout:osd.3: {} 2024-03-11T21:40:47.343 INFO:teuthology.orchestra.run.smithi092.stdout:osd.4: {} 2024-03-11T21:40:47.364 INFO:teuthology.orchestra.run.smithi092.stdout:osd.5: {} 2024-03-11T21:40:47.385 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:40:47.386 INFO:teuthology.orchestra.run.smithi092.stdout:osd.6: {} 2024-03-11T21:40:47.407 INFO:teuthology.orchestra.run.smithi092.stdout:osd.7: {} 2024-03-11T21:40:47.408 INFO:teuthology.orchestra.run.smithi092.stderr:osd.0: osd_enable_op_tracker = '' 2024-03-11T21:40:47.408 INFO:teuthology.orchestra.run.smithi092.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-11T21:40:47.408 INFO:teuthology.orchestra.run.smithi092.stderr:osd.2: osd_enable_op_tracker = '' 2024-03-11T21:40:47.408 INFO:teuthology.orchestra.run.smithi092.stderr:osd.3: osd_enable_op_tracker = '' 2024-03-11T21:40:47.408 INFO:teuthology.orchestra.run.smithi092.stderr:osd.4: osd_enable_op_tracker = '' 2024-03-11T21:40:47.408 INFO:teuthology.orchestra.run.smithi092.stderr:osd.5: osd_enable_op_tracker = '' 2024-03-11T21:40:47.408 INFO:teuthology.orchestra.run.smithi092.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-11T21:40:47.408 INFO:teuthology.orchestra.run.smithi092.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-11T21:40:47.412 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:40:47.413 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:40:47.413 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:40:47.502 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:40:47 smithi092 ceph-mon[29572]: from='client.? v1:172.21.15.92:0/1729977980' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-11T21:40:47.502 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:40:47 smithi092 ceph-mon[37563]: from='client.? v1:172.21.15.92:0/1729977980' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-11T21:40:47.514 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:40:47.542 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:40:47.542 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:40:47.543 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:40:47.644 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:40:47.670 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:40:47.671 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:40:47.672 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:40:47.772 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:40:47.799 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:40:47.799 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:40:47.800 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:40:47.830 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:40:47 smithi097 ceph-mon[33317]: from='client.? v1:172.21.15.92:0/1729977980' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-11T21:40:47.900 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:40:47.925 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:40:47.926 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:40:47.926 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:40:48.027 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:40:48.054 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:40:48.055 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:40:48.055 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:40:48.114 INFO:journalctl@ceph.mgr.y.smithi092.stdout:Mar 11 21:40:47 smithi092 ceph-00db0016-dfed-11ee-95c8-87774f69a715-mgr-y[29801]: ::ffff:172.21.15.97 - - [11/Mar/2024:21:40:47] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-11T21:40:48.129 DEBUG:teuthology.orchestra.run.smithi092:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid 00db0016-dfed-11ee-95c8-87774f69a715 -- ceph osd unset noscrub 2024-03-11T21:40:48.156 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:40:48.182 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:40:48.183 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:40:48.183 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:40:48.284 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:40:48.310 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:40:48.311 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:40:48.311 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:40:48.412 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:40:48.437 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:40:48.438 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:40:48.438 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:40:48.533 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:40:48 smithi092 ceph-mon[29572]: pgmap v711: 148 pgs: 148 active+clean; 582 KiB data, 3.6 GiB used, 712 GiB / 715 GiB avail; 1005 B/s rd, 0 op/s 2024-03-11T21:40:48.533 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:40:48 smithi092 ceph-mon[29572]: purged_snaps scrub starts 2024-03-11T21:40:48.533 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:40:48 smithi092 ceph-mon[29572]: purged_snaps scrub ok 2024-03-11T21:40:48.534 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:40:48 smithi092 ceph-mon[37563]: pgmap v711: 148 pgs: 148 active+clean; 582 KiB data, 3.6 GiB used, 712 GiB / 715 GiB avail; 1005 B/s rd, 0 op/s 2024-03-11T21:40:48.534 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:40:48 smithi092 ceph-mon[37563]: purged_snaps scrub starts 2024-03-11T21:40:48.534 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:40:48 smithi092 ceph-mon[37563]: purged_snaps scrub ok 2024-03-11T21:40:48.539 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:40:48.564 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:40:48.564 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:40:48.565 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:40:48.666 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:40:48.693 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:40:48.693 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:40:48.694 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:40:48.795 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:40:48.820 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:40:48.821 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:40:48.821 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:40:48.830 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:40:48 smithi097 ceph-mon[33317]: pgmap v711: 148 pgs: 148 active+clean; 582 KiB data, 3.6 GiB used, 712 GiB / 715 GiB avail; 1005 B/s rd, 0 op/s 2024-03-11T21:40:48.830 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:40:48 smithi097 ceph-mon[33317]: purged_snaps scrub starts 2024-03-11T21:40:48.830 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:40:48 smithi097 ceph-mon[33317]: purged_snaps scrub ok 2024-03-11T21:40:48.922 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:40:48.948 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:40:48.948 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:40:48.949 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:40:49.049 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:40:49.075 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:40:49.076 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:40:49.076 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:40:49.177 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:40:49.202 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:40:49.202 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:40:49.203 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:40:49.303 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:40:49.327 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:40:49.328 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:40:49.328 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:40:49.429 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:40:49.454 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:40:49.454 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:40:49.454 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:40:49.467 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:40:49 smithi092 ceph-mon[29572]: osdmap e363: 8 total, 8 up, 8 in 2024-03-11T21:40:49.468 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:40:49 smithi092 ceph-mon[37563]: osdmap e363: 8 total, 8 up, 8 in 2024-03-11T21:40:49.555 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:40:49.579 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:40:49.579 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:40:49.580 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:40:49.681 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:40:49.707 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:40:49.708 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:40:49.709 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:40:49.810 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:40:49.830 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:40:49 smithi097 ceph-mon[33317]: osdmap e363: 8 total, 8 up, 8 in 2024-03-11T21:40:49.836 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:40:49.837 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:40:49.837 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:40:49.938 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:40:49.961 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:40:49.962 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:40:49.962 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:40:50.063 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:40:50.087 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:40:50.088 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:40:50.088 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:40:50.189 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:40:50.217 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:40:50.217 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:40:50.218 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:40:50.261 INFO:teuthology.orchestra.run.smithi092.stdout:{ 2024-03-11T21:40:50.261 INFO:teuthology.orchestra.run.smithi092.stdout: "ops": [], 2024-03-11T21:40:50.261 INFO:teuthology.orchestra.run.smithi092.stdout: "complaint_time": 30, 2024-03-11T21:40:50.261 INFO:teuthology.orchestra.run.smithi092.stdout: "num_blocked_ops": 0 2024-03-11T21:40:50.261 INFO:teuthology.orchestra.run.smithi092.stdout:} 2024-03-11T21:40:50.318 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:40:50.346 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:40:50.347 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:40:50.347 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:40:50.407 INFO:teuthology.orchestra.run.smithi092.stderr:enabled application 'rados' on pool 'unique_pool_3' 2024-03-11T21:40:50.448 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:40:50.472 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:40:50.472 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:40:50.473 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:40:50.574 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:40:50.602 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:40:50.603 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:40:50.603 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:40:50.614 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:40:50 smithi092 ceph-mon[29572]: pgmap v713: 148 pgs: 148 active+clean; 582 KiB data, 3.6 GiB used, 712 GiB / 715 GiB avail; 862 B/s rd, 0 op/s 2024-03-11T21:40:50.614 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:40:50 smithi092 ceph-mon[29572]: from='client.? v1:172.21.15.92:0/968664088' entity='client.admin' cmd=[{"prefix": "osd pool application enable", "pool": "unique_pool_3", "app": "rados", "yes_i_really_mean_it": true}]: dispatch 2024-03-11T21:40:50.614 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:40:50 smithi092 ceph-mon[29572]: from='client.35579 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable", "pool": "unique_pool_3", "app": "rados", "yes_i_really_mean_it": true}]: dispatch 2024-03-11T21:40:50.614 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:40:50 smithi092 ceph-mon[37563]: pgmap v713: 148 pgs: 148 active+clean; 582 KiB data, 3.6 GiB used, 712 GiB / 715 GiB avail; 862 B/s rd, 0 op/s 2024-03-11T21:40:50.614 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:40:50 smithi092 ceph-mon[37563]: from='client.? v1:172.21.15.92:0/968664088' entity='client.admin' cmd=[{"prefix": "osd pool application enable", "pool": "unique_pool_3", "app": "rados", "yes_i_really_mean_it": true}]: dispatch 2024-03-11T21:40:50.614 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:40:50 smithi092 ceph-mon[37563]: from='client.35579 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable", "pool": "unique_pool_3", "app": "rados", "yes_i_really_mean_it": true}]: dispatch 2024-03-11T21:40:50.704 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:40:50.731 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:40:50.732 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:40:50.732 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:40:50.830 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:40:50 smithi097 ceph-mon[33317]: pgmap v713: 148 pgs: 148 active+clean; 582 KiB data, 3.6 GiB used, 712 GiB / 715 GiB avail; 862 B/s rd, 0 op/s 2024-03-11T21:40:50.830 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:40:50 smithi097 ceph-mon[33317]: from='client.? v1:172.21.15.92:0/968664088' entity='client.admin' cmd=[{"prefix": "osd pool application enable", "pool": "unique_pool_3", "app": "rados", "yes_i_really_mean_it": true}]: dispatch 2024-03-11T21:40:50.830 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:40:50 smithi097 ceph-mon[33317]: from='client.35579 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable", "pool": "unique_pool_3", "app": "rados", "yes_i_really_mean_it": true}]: dispatch 2024-03-11T21:40:50.834 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:40:50.861 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:40:50.861 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:40:50.862 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:40:50.963 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:40:50.988 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:40:50.988 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:40:50.989 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:40:51.089 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:40:51.118 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:40:51.119 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:40:51.119 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:40:51.220 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:40:51.248 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:40:51.249 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:40:51.250 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:40:51.350 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:40:51.378 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:40:51.378 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:40:51.379 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:40:51.479 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:40:51.506 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:40:51.507 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:40:51.507 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:40:51.608 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:40:51.633 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:40:51.633 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:40:51.634 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:40:51.717 INFO:teuthology.orchestra.run.smithi092.stderr:Inferring config /var/lib/ceph/00db0016-dfed-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T21:40:51.731 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:40:51 smithi097 ceph-mon[33317]: from='client.35579 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable", "pool": "unique_pool_3", "app": "rados", "yes_i_really_mean_it": true}]': finished 2024-03-11T21:40:51.732 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:40:51 smithi097 ceph-mon[33317]: osdmap e364: 8 total, 8 up, 8 in 2024-03-11T21:40:51.735 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:40:51.759 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:40:51.759 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:40:51.760 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:40:51.813 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:40:51 smithi092 ceph-mon[29572]: from='client.35579 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable", "pool": "unique_pool_3", "app": "rados", "yes_i_really_mean_it": true}]': finished 2024-03-11T21:40:51.813 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:40:51 smithi092 ceph-mon[29572]: osdmap e364: 8 total, 8 up, 8 in 2024-03-11T21:40:51.813 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:40:51 smithi092 ceph-mon[37563]: from='client.35579 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable", "pool": "unique_pool_3", "app": "rados", "yes_i_really_mean_it": true}]': finished 2024-03-11T21:40:51.813 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:40:51 smithi092 ceph-mon[37563]: osdmap e364: 8 total, 8 up, 8 in 2024-03-11T21:40:51.860 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:40:51.885 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:40:51.885 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:40:51.885 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:40:51.986 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:40:52.011 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:40:52.012 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:40:52.012 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:40:52.026 DEBUG:teuthology.orchestra.run.smithi092:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid 00db0016-dfed-11ee-95c8-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-03-11T21:40:52.100 DEBUG:teuthology.orchestra.run.smithi092:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid 00db0016-dfed-11ee-95c8-87774f69a715 -- ceph daemon osd.1 dump_historic_ops 2024-03-11T21:40:52.113 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:40:52.166 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:40:52.167 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:40:52.167 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:40:52.268 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:40:52.294 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:40:52.294 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:40:52.294 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:40:52.395 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:40:52.423 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:40:52.424 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:40:52.424 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:40:52.511 DEBUG:teuthology.orchestra.run.smithi092:> /bin/sh -c 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados --no-log-to-stderr --name client.0 -b 65536 --object-size 65536 -p unique_pool_3 bench 90 write' 2024-03-11T21:40:52.513 INFO:tasks.radosbench:joining radosbench (timing out after 3000s) 2024-03-11T21:40:52.514 INFO:teuthology.orchestra.run:waiting for 3000 2024-03-11T21:40:52.526 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:40:52.553 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:40:52.554 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:40:52.554 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:40:52.655 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:40:52.680 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:40:52.681 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:40:52.681 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:40:52.742 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:40:52 smithi092 ceph-mon[29572]: pgmap v715: 148 pgs: 148 active+clean; 582 KiB data, 3.6 GiB used, 712 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-11T21:40:52.742 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:40:52 smithi092 ceph-mon[29572]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-03-11T21:40:52.743 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:40:52 smithi092 ceph-mon[37563]: pgmap v715: 148 pgs: 148 active+clean; 582 KiB data, 3.6 GiB used, 712 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-11T21:40:52.743 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:40:52 smithi092 ceph-mon[37563]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-03-11T21:40:52.782 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:40:52.810 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:40:52.811 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:40:52.811 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:40:52.828 INFO:tasks.radosbench.radosbench.0.smithi092.stdout:hints = 1 2024-03-11T21:40:52.828 INFO:tasks.radosbench.radosbench.0.smithi092.stdout:Maintaining 16 concurrent writes of 65536 bytes to objects of size 65536 for up to 90 seconds or 0 objects 2024-03-11T21:40:52.828 INFO:tasks.radosbench.radosbench.0.smithi092.stdout:Object prefix: benchmark_data_smithi092_161462 2024-03-11T21:40:52.829 INFO:tasks.radosbench.radosbench.0.smithi092.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-03-11T21:40:52.829 INFO:tasks.radosbench.radosbench.0.smithi092.stdout: 0 0 0 0 0 0 - 0 2024-03-11T21:40:52.830 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:40:52 smithi097 ceph-mon[33317]: pgmap v715: 148 pgs: 148 active+clean; 582 KiB data, 3.6 GiB used, 712 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-11T21:40:52.830 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:40:52 smithi097 ceph-mon[33317]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-03-11T21:40:52.912 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:40:52.956 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:40:52.957 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:40:52.957 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:40:53.050 INFO:teuthology.orchestra.run.smithi092.stdout: 2024-03-11T21:40:53.055 INFO:teuthology.orchestra.run.smithi092.stderr:dumped all 2024-03-11T21:40:53.058 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:40:53.095 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:40:53.095 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:40:53.096 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:40:53.196 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:40:53.238 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:40:53.238 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:40:53.238 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:40:53.330 INFO:journalctl@ceph.iscsi.iscsi.a.smithi097.stdout:Mar 11 21:40:52 smithi097 ceph-00db0016-dfed-11ee-95c8-87774f69a715-iscsi-iscsi-a[60943]: debug there is no tcmu-runner data available 2024-03-11T21:40:53.340 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:40:53.378 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:40:53.379 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:40:53.379 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:40:53.480 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:40:53.516 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:40:53.516 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:40:53.517 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:40:53.618 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:40:53.657 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:40:53.657 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:40:53.658 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:40:53.742 INFO:teuthology.orchestra.run.smithi092.stderr:Inferring config /var/lib/ceph/00db0016-dfed-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T21:40:53.758 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:40:53.803 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:40:53.803 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:40:53.804 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:40:53.829 INFO:tasks.radosbench.radosbench.0.smithi092.stdout: 1 16 3655 3639 227.424 227.438 0.00405008 0.00437823 2024-03-11T21:40:53.830 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:40:53 smithi097 ceph-mon[33317]: from='mgr.14607 v1:172.21.15.92:0/1702717597' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T21:40:53.830 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:40:53 smithi097 ceph-mon[33317]: from='client.24469 v1:172.21.15.97:0/404208310' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:40:53.830 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:40:53 smithi097 ceph-mon[33317]: from='client.15735 v1:172.21.15.92:0/3795057743' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-11T21:40:53.857 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:40:53 smithi092 ceph-mon[29572]: from='mgr.14607 v1:172.21.15.92:0/1702717597' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T21:40:53.857 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:40:53 smithi092 ceph-mon[29572]: from='client.24469 v1:172.21.15.97:0/404208310' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:40:53.858 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:40:53 smithi092 ceph-mon[29572]: from='client.15735 v1:172.21.15.92:0/3795057743' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-11T21:40:53.858 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:40:53 smithi092 ceph-mon[37563]: from='mgr.14607 v1:172.21.15.92:0/1702717597' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T21:40:53.858 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:40:53 smithi092 ceph-mon[37563]: from='client.24469 v1:172.21.15.97:0/404208310' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:40:53.858 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:40:53 smithi092 ceph-mon[37563]: from='client.15735 v1:172.21.15.92:0/3795057743' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-11T21:40:53.904 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:40:53.938 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:40:53.939 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:40:53.939 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:40:54.040 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:40:54.080 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:40:54.081 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:40:54.081 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:40:54.182 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:40:54.218 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:40:54.219 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:40:54.219 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:40:54.320 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:40:54.357 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:40:54.358 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:40:54.358 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:40:54.413 INFO:teuthology.orchestra.run.smithi092.stderr:Inferring config /var/lib/ceph/00db0016-dfed-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T21:40:54.444 INFO:teuthology.orchestra.run.smithi092.stdout:{"pg_ready":true,"pg_map":{"version":715,"stamp":"2024-03-11T21:40:51.326307+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":596392,"num_objects":198,"num_object_clones":0,"num_object_copies":594,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":198,"num_whiteouts":0,"num_read":2237,"num_read_kb":2085,"num_write":901,"num_write_kb":2461,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":203,"num_bytes_recovered":2955718,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":993,"ondisk_log_size":993,"up":444,"acting":444,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":444,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":749961216,"kb_used":3748160,"kb_used_data":19900,"kb_used_omap":15,"kb_used_meta":3728112,"kb_avail":746213056,"statfs":{"total":767960285184,"available":764122169344,"internally_reserved":0,"allocated":20377600,"data_stored":12261720,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15601,"internal_metadata":3817587471},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":5,"num_read_kb":5,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.001187"},"pg_stats":[{"pgid":"6.1b","version":"0'0","reported_seq":576,"reported_epoch":363,"state":"active+clean","last_fresh":"2024-03-11T21:40:48.881666+0000","last_change":"2024-03-11T21:36:35.074931+0000","last_active":"2024-03-11T21:40:48.881666+0000","last_peered":"2024-03-11T21:40:48.881666+0000","last_clean":"2024-03-11T21:40:48.881666+0000","last_became_active":"2024-03-11T21:36:35.072636+0000","last_became_peered":"2024-03-11T21:36:35.072636+0000","last_unstale":"2024-03-11T21:40:48.881666+0000","last_undegraded":"2024-03-11T21:40:48.881666+0000","last_fullsized":"2024-03-11T21:40:48.881666+0000","mapping_epoch":277,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":278,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:35:07.021520+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:43.559452+0000","last_clean_scrub_stamp":"2024-03-11T21:35:07.021520+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:40:52.711278+0000","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":639,"reported_epoch":362,"state":"active+clean","last_fresh":"2024-03-11T21:40:45.410620+0000","last_change":"2024-03-11T21:36:00.720089+0000","last_active":"2024-03-11T21:40:45.410620+0000","last_peered":"2024-03-11T21:40:45.410620+0000","last_clean":"2024-03-11T21:40:45.410620+0000","last_became_active":"2024-03-11T21:25:42.578893+0000","last_became_peered":"2024-03-11T21:25:42.578893+0000","last_unstale":"2024-03-11T21:40:45.410620+0000","last_undegraded":"2024-03-11T21:40:45.410620+0000","last_fullsized":"2024-03-11T21:40:45.410620+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:36:00.719956+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:41.544550+0000","last_clean_scrub_stamp":"2024-03-11T21:36:00.719956+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1f","version":"0'0","reported_seq":670,"reported_epoch":363,"state":"active+clean","last_fresh":"2024-03-11T21:40:48.411441+0000","last_change":"2024-03-11T21:36:00.843838+0000","last_active":"2024-03-11T21:40:48.411441+0000","last_peered":"2024-03-11T21:40:48.411441+0000","last_clean":"2024-03-11T21:40:48.411441+0000","last_became_active":"2024-03-11T21:34:41.144945+0000","last_became_peered":"2024-03-11T21:34:41.144945+0000","last_unstale":"2024-03-11T21:40:48.411441+0000","last_undegraded":"2024-03-11T21:40:48.411441+0000","last_fullsized":"2024-03-11T21:40:48.411441+0000","mapping_epoch":226,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":227,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:36:00.843718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:37.512018+0000","last_clean_scrub_stamp":"2024-03-11T21:36:00.843718+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:40:49.118447+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,4],"acting":[0,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.19","version":"260'27","reported_seq":623,"reported_epoch":363,"state":"active+clean","last_fresh":"2024-03-11T21:40:48.881364+0000","last_change":"2024-03-11T21:36:35.089901+0000","last_active":"2024-03-11T21:40:48.881364+0000","last_peered":"2024-03-11T21:40:48.881364+0000","last_clean":"2024-03-11T21:40:48.881364+0000","last_became_active":"2024-03-11T21:36:35.067652+0000","last_became_peered":"2024-03-11T21:36:35.067652+0000","last_unstale":"2024-03-11T21:40:48.881364+0000","last_undegraded":"2024-03-11T21:40:48.881364+0000","last_fullsized":"2024-03-11T21:40:48.881364+0000","mapping_epoch":277,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":278,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'15","last_scrub_stamp":"2024-03-11T21:35:22.099374+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:39.525554+0000","last_clean_scrub_stamp":"2024-03-11T21:35:22.099374+0000","objects_scrubbed":0,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:40:52.711211+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":42,"num_read_kb":33,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.1a","version":"0'0","reported_seq":627,"reported_epoch":362,"state":"active+clean","last_fresh":"2024-03-11T21:40:45.410888+0000","last_change":"2024-03-11T21:35:16.410927+0000","last_active":"2024-03-11T21:40:45.410888+0000","last_peered":"2024-03-11T21:40:45.410888+0000","last_clean":"2024-03-11T21:40:45.410888+0000","last_became_active":"2024-03-11T21:25:44.590923+0000","last_became_peered":"2024-03-11T21:25:44.590923+0000","last_unstale":"2024-03-11T21:40:45.410888+0000","last_undegraded":"2024-03-11T21:40:45.410888+0000","last_fullsized":"2024-03-11T21:40:45.410888+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:35:16.410816+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:43.559452+0000","last_clean_scrub_stamp":"2024-03-11T21:35:16.410816+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":24,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,1],"acting":[4,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":651,"reported_epoch":362,"state":"active+clean","last_fresh":"2024-03-11T21:40:45.410969+0000","last_change":"2024-03-11T21:36:15.576138+0000","last_active":"2024-03-11T21:40:45.410969+0000","last_peered":"2024-03-11T21:40:45.410969+0000","last_clean":"2024-03-11T21:40:45.410969+0000","last_became_active":"2024-03-11T21:34:41.154216+0000","last_became_peered":"2024-03-11T21:34:41.154216+0000","last_unstale":"2024-03-11T21:40:45.410969+0000","last_undegraded":"2024-03-11T21:40:45.410969+0000","last_fullsized":"2024-03-11T21:40:45.410969+0000","mapping_epoch":226,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":227,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:36:15.575887+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:41.544550+0000","last_clean_scrub_stamp":"2024-03-11T21:36:15.575887+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:40:50.059622+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,7],"acting":[1,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1e","version":"0'0","reported_seq":590,"reported_epoch":363,"state":"active+clean","last_fresh":"2024-03-11T21:40:48.882400+0000","last_change":"2024-03-11T21:36:35.062413+0000","last_active":"2024-03-11T21:40:48.882400+0000","last_peered":"2024-03-11T21:40:48.882400+0000","last_clean":"2024-03-11T21:40:48.882400+0000","last_became_active":"2024-03-11T21:36:35.061786+0000","last_became_peered":"2024-03-11T21:36:35.061786+0000","last_unstale":"2024-03-11T21:40:48.882400+0000","last_undegraded":"2024-03-11T21:40:48.882400+0000","last_fullsized":"2024-03-11T21:40:48.882400+0000","mapping_epoch":277,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":278,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:32:18.578038+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:37.512018+0000","last_clean_scrub_stamp":"2024-03-11T21:32:18.578038+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:40:52.711043+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.18","version":"260'15","reported_seq":676,"reported_epoch":362,"state":"active+clean","last_fresh":"2024-03-11T21:40:45.410451+0000","last_change":"2024-03-11T21:36:09.662765+0000","last_active":"2024-03-11T21:40:45.410451+0000","last_peered":"2024-03-11T21:40:45.410451+0000","last_clean":"2024-03-11T21:40:45.410451+0000","last_became_active":"2024-03-11T21:25:40.552216+0000","last_became_peered":"2024-03-11T21:25:40.552216+0000","last_unstale":"2024-03-11T21:40:45.410451+0000","last_undegraded":"2024-03-11T21:40:45.410451+0000","last_fullsized":"2024-03-11T21:40:45.410451+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"260'15","last_scrub_stamp":"2024-03-11T21:36:09.662615+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:39.525554+0000","last_clean_scrub_stamp":"2024-03-11T21:36:09.662615+0000","objects_scrubbed":4,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":643,"reported_epoch":362,"state":"active+clean","last_fresh":"2024-03-11T21:40:46.104015+0000","last_change":"2024-03-11T21:36:35.068915+0000","last_active":"2024-03-11T21:40:46.104015+0000","last_peered":"2024-03-11T21:40:46.104015+0000","last_clean":"2024-03-11T21:40:46.104015+0000","last_became_active":"2024-03-11T21:36:35.068453+0000","last_became_peered":"2024-03-11T21:36:35.068453+0000","last_unstale":"2024-03-11T21:40:46.104015+0000","last_undegraded":"2024-03-11T21:40:46.104015+0000","last_fullsized":"2024-03-11T21:40:46.104015+0000","mapping_epoch":277,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":278,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:35:26.190634+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:32:12.654573+0000","last_clean_scrub_stamp":"2024-03-11T21:35:26.190634+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3],"acting":[5,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":291,"reported_epoch":362,"state":"active+clean","last_fresh":"2024-03-11T21:40:46.061465+0000","last_change":"2024-03-11T21:35:11.902920+0000","last_active":"2024-03-11T21:40:46.061465+0000","last_peered":"2024-03-11T21:40:46.061465+0000","last_clean":"2024-03-11T21:40:46.061465+0000","last_became_active":"2024-03-11T21:34:41.145299+0000","last_became_peered":"2024-03-11T21:34:41.145299+0000","last_unstale":"2024-03-11T21:40:46.061465+0000","last_undegraded":"2024-03-11T21:40:46.061465+0000","last_fullsized":"2024-03-11T21:40:46.061465+0000","mapping_epoch":226,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":227,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:35:11.902818+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:41.544550+0000","last_clean_scrub_stamp":"2024-03-11T21:35:11.902818+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.1d","version":"0'0","reported_seq":654,"reported_epoch":362,"state":"active+clean","last_fresh":"2024-03-11T21:40:46.061830+0000","last_change":"2024-03-11T21:35:26.855588+0000","last_active":"2024-03-11T21:40:46.061830+0000","last_peered":"2024-03-11T21:40:46.061830+0000","last_clean":"2024-03-11T21:40:46.061830+0000","last_became_active":"2024-03-11T21:25:38.542628+0000","last_became_peered":"2024-03-11T21:25:38.542628+0000","last_unstale":"2024-03-11T21:40:46.061830+0000","last_undegraded":"2024-03-11T21:40:46.061830+0000","last_fullsized":"2024-03-11T21:40:46.061830+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:35:26.855511+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:37.512018+0000","last_clean_scrub_stamp":"2024-03-11T21:35:26.855511+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,0],"acting":[7,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.1b","version":"260'7","reported_seq":670,"reported_epoch":362,"state":"active+clean","last_fresh":"2024-03-11T21:40:45.411001+0000","last_change":"2024-03-11T21:40:29.597326+0000","last_active":"2024-03-11T21:40:45.411001+0000","last_peered":"2024-03-11T21:40:45.411001+0000","last_clean":"2024-03-11T21:40:45.411001+0000","last_became_active":"2024-03-11T21:36:35.057678+0000","last_became_peered":"2024-03-11T21:36:35.057678+0000","last_unstale":"2024-03-11T21:40:45.411001+0000","last_undegraded":"2024-03-11T21:40:45.411001+0000","last_fullsized":"2024-03-11T21:40:45.411001+0000","mapping_epoch":277,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":278,"parent":"0.0","parent_split_bits":0,"last_scrub":"260'7","last_scrub_stamp":"2024-03-11T21:40:29.597219+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:39.525554+0000","last_clean_scrub_stamp":"2024-03-11T21:40:29.597219+0000","objects_scrubbed":2,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:41:33.394843+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":14,"num_read_kb":10,"num_write":8,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,1],"acting":[4,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":637,"reported_epoch":363,"state":"active+clean","last_fresh":"2024-03-11T21:40:48.411100+0000","last_change":"2024-03-11T21:35:39.691825+0000","last_active":"2024-03-11T21:40:48.411100+0000","last_peered":"2024-03-11T21:40:48.411100+0000","last_clean":"2024-03-11T21:40:48.411100+0000","last_became_active":"2024-03-11T21:25:44.589786+0000","last_became_peered":"2024-03-11T21:25:44.589786+0000","last_unstale":"2024-03-11T21:40:48.411100+0000","last_undegraded":"2024-03-11T21:40:48.411100+0000","last_fullsized":"2024-03-11T21:40:48.411100+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:35:39.691690+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:43.559452+0000","last_clean_scrub_stamp":"2024-03-11T21:35:39.691690+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:40:49.118399+0000","scrub_duration":18,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,7],"acting":[0,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":658,"reported_epoch":362,"state":"active+clean","last_fresh":"2024-03-11T21:40:46.104379+0000","last_change":"2024-03-11T21:40:27.604550+0000","last_active":"2024-03-11T21:40:46.104379+0000","last_peered":"2024-03-11T21:40:46.104379+0000","last_clean":"2024-03-11T21:40:46.104379+0000","last_became_active":"2024-03-11T21:34:41.147185+0000","last_became_peered":"2024-03-11T21:34:41.147185+0000","last_unstale":"2024-03-11T21:40:46.104379+0000","last_undegraded":"2024-03-11T21:40:46.104379+0000","last_fullsized":"2024-03-11T21:40:46.104379+0000","mapping_epoch":226,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":227,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:40:27.604453+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:32:21.535034+0000","last_clean_scrub_stamp":"2024-03-11T21:40:27.604453+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:41:27.648926+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,7],"acting":[5,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1c","version":"0'0","reported_seq":654,"reported_epoch":362,"state":"active+clean","last_fresh":"2024-03-11T21:40:46.061848+0000","last_change":"2024-03-11T21:35:28.789693+0000","last_active":"2024-03-11T21:40:46.061848+0000","last_peered":"2024-03-11T21:40:46.061848+0000","last_clean":"2024-03-11T21:40:46.061848+0000","last_became_active":"2024-03-11T21:25:38.541364+0000","last_became_peered":"2024-03-11T21:25:38.541364+0000","last_unstale":"2024-03-11T21:40:46.061848+0000","last_undegraded":"2024-03-11T21:40:46.061848+0000","last_fullsized":"2024-03-11T21:40:46.061848+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:35:28.789592+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:28:32.566437+0000","last_clean_scrub_stamp":"2024-03-11T21:35:28.789592+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.1a","version":"350'19","reported_seq":698,"reported_epoch":362,"state":"active+clean","last_fresh":"2024-03-11T21:40:45.410840+0000","last_change":"2024-03-11T21:40:20.484449+0000","last_active":"2024-03-11T21:40:45.410840+0000","last_peered":"2024-03-11T21:40:45.410840+0000","last_clean":"2024-03-11T21:40:45.410840+0000","last_became_active":"2024-03-11T21:36:35.057029+0000","last_became_peered":"2024-03-11T21:36:35.057029+0000","last_unstale":"2024-03-11T21:40:45.410840+0000","last_undegraded":"2024-03-11T21:40:45.410840+0000","last_fullsized":"2024-03-11T21:40:45.410840+0000","mapping_epoch":277,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":278,"parent":"0.0","parent_split_bits":0,"last_scrub":"350'19","last_scrub_stamp":"2024-03-11T21:40:20.484308+0000","last_deep_scrub":"350'19","last_deep_scrub_stamp":"2024-03-11T21:40:20.484308+0000","last_clean_scrub_stamp":"2024-03-11T21:40:20.484308+0000","objects_scrubbed":4,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:41:24.916676+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0],"acting":[4,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1f","version":"0'0","reported_seq":513,"reported_epoch":363,"state":"active+clean","last_fresh":"2024-03-11T21:40:48.882624+0000","last_change":"2024-03-11T21:36:35.074675+0000","last_active":"2024-03-11T21:40:48.882624+0000","last_peered":"2024-03-11T21:40:48.882624+0000","last_clean":"2024-03-11T21:40:48.882624+0000","last_became_active":"2024-03-11T21:36:35.071755+0000","last_became_peered":"2024-03-11T21:36:35.071755+0000","last_unstale":"2024-03-11T21:40:48.882624+0000","last_undegraded":"2024-03-11T21:40:48.882624+0000","last_fullsized":"2024-03-11T21:40:48.882624+0000","mapping_epoch":277,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":278,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:34:05.862891+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:43.559452+0000","last_clean_scrub_stamp":"2024-03-11T21:34:05.862891+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:40:52.711076+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1c","version":"0'0","reported_seq":648,"reported_epoch":362,"state":"active+clean","last_fresh":"2024-03-11T21:40:45.411435+0000","last_change":"2024-03-11T21:40:07.678642+0000","last_active":"2024-03-11T21:40:45.411435+0000","last_peered":"2024-03-11T21:40:45.411435+0000","last_clean":"2024-03-11T21:40:45.411435+0000","last_became_active":"2024-03-11T21:36:35.067127+0000","last_became_peered":"2024-03-11T21:36:35.067127+0000","last_unstale":"2024-03-11T21:40:45.411435+0000","last_undegraded":"2024-03-11T21:40:45.411435+0000","last_fullsized":"2024-03-11T21:40:45.411435+0000","mapping_epoch":277,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":278,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:40:07.678561+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:40:07.678561+0000","last_clean_scrub_stamp":"2024-03-11T21:40:07.678561+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:41:27.033148+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1b","version":"0'0","reported_seq":561,"reported_epoch":363,"state":"active+clean","last_fresh":"2024-03-11T21:40:48.881721+0000","last_change":"2024-03-11T21:36:35.075385+0000","last_active":"2024-03-11T21:40:48.881721+0000","last_peered":"2024-03-11T21:40:48.881721+0000","last_clean":"2024-03-11T21:40:48.881721+0000","last_became_active":"2024-03-11T21:36:35.073959+0000","last_became_peered":"2024-03-11T21:36:35.073959+0000","last_unstale":"2024-03-11T21:40:48.881721+0000","last_undegraded":"2024-03-11T21:40:48.881721+0000","last_fullsized":"2024-03-11T21:40:48.881721+0000","mapping_epoch":277,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":278,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:34:27.998724+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:34:27.998724+0000","last_clean_scrub_stamp":"2024-03-11T21:34:27.998724+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:40:52.711109+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1d","version":"260'18","reported_seq":605,"reported_epoch":363,"state":"active+clean","last_fresh":"2024-03-11T21:40:48.881694+0000","last_change":"2024-03-11T21:36:35.080207+0000","last_active":"2024-03-11T21:40:48.881694+0000","last_peered":"2024-03-11T21:40:48.881694+0000","last_clean":"2024-03-11T21:40:48.881694+0000","last_became_active":"2024-03-11T21:36:35.055880+0000","last_became_peered":"2024-03-11T21:36:35.055880+0000","last_unstale":"2024-03-11T21:40:48.881694+0000","last_undegraded":"2024-03-11T21:40:48.881694+0000","last_fullsized":"2024-03-11T21:40:48.881694+0000","mapping_epoch":277,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":278,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'12","last_scrub_stamp":"2024-03-11T21:35:07.971581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:39.525554+0000","last_clean_scrub_stamp":"2024-03-11T21:35:07.971581+0000","objects_scrubbed":0,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:40:52.711247+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":33,"num_read_kb":24,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.1e","version":"0'0","reported_seq":628,"reported_epoch":362,"state":"active+clean","last_fresh":"2024-03-11T21:40:45.411217+0000","last_change":"2024-03-11T21:35:32.664451+0000","last_active":"2024-03-11T21:40:45.411217+0000","last_peered":"2024-03-11T21:40:45.411217+0000","last_clean":"2024-03-11T21:40:45.411217+0000","last_became_active":"2024-03-11T21:25:45.300355+0000","last_became_peered":"2024-03-11T21:25:45.300355+0000","last_unstale":"2024-03-11T21:40:45.411217+0000","last_undegraded":"2024-03-11T21:40:45.411217+0000","last_fullsized":"2024-03-11T21:40:45.411217+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:35:32.664371+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:43.559452+0000","last_clean_scrub_stamp":"2024-03-11T21:35:32.664371+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5],"acting":[4,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1d","version":"0'0","reported_seq":639,"reported_epoch":362,"state":"active+clean","last_fresh":"2024-03-11T21:40:45.412554+0000","last_change":"2024-03-11T21:36:07.624196+0000","last_active":"2024-03-11T21:40:45.412554+0000","last_peered":"2024-03-11T21:40:45.412554+0000","last_clean":"2024-03-11T21:40:45.412554+0000","last_became_active":"2024-03-11T21:25:42.579407+0000","last_became_peered":"2024-03-11T21:25:42.579407+0000","last_unstale":"2024-03-11T21:40:45.412554+0000","last_undegraded":"2024-03-11T21:40:45.412554+0000","last_fullsized":"2024-03-11T21:40:45.412554+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:36:07.624075+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:30:18.678531+0000","last_clean_scrub_stamp":"2024-03-11T21:36:07.624075+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:40:50.059566+0000","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1a","version":"0'0","reported_seq":675,"reported_epoch":362,"state":"active+clean","last_fresh":"2024-03-11T21:40:46.104186+0000","last_change":"2024-03-11T21:39:53.299021+0000","last_active":"2024-03-11T21:40:46.104186+0000","last_peered":"2024-03-11T21:40:46.104186+0000","last_clean":"2024-03-11T21:40:46.104186+0000","last_became_active":"2024-03-11T21:34:41.145014+0000","last_became_peered":"2024-03-11T21:34:41.145014+0000","last_unstale":"2024-03-11T21:40:46.104186+0000","last_undegraded":"2024-03-11T21:40:46.104186+0000","last_fullsized":"2024-03-11T21:40:46.104186+0000","mapping_epoch":226,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":227,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:39:53.298917+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:37.512018+0000","last_clean_scrub_stamp":"2024-03-11T21:39:53.298917+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:41:21.887727+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.1c","version":"260'27","reported_seq":708,"reported_epoch":362,"state":"active+clean","last_fresh":"2024-03-11T21:40:46.104939+0000","last_change":"2024-03-11T21:40:15.885360+0000","last_active":"2024-03-11T21:40:46.104939+0000","last_peered":"2024-03-11T21:40:46.104939+0000","last_clean":"2024-03-11T21:40:46.104939+0000","last_became_active":"2024-03-11T21:36:35.074579+0000","last_became_peered":"2024-03-11T21:36:35.074579+0000","last_unstale":"2024-03-11T21:40:46.104939+0000","last_undegraded":"2024-03-11T21:40:46.104939+0000","last_fullsized":"2024-03-11T21:40:46.104939+0000","mapping_epoch":277,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":278,"parent":"0.0","parent_split_bits":0,"last_scrub":"260'27","last_scrub_stamp":"2024-03-11T21:40:15.885265+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:39.525554+0000","last_clean_scrub_stamp":"2024-03-11T21:40:15.885265+0000","objects_scrubbed":7,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:41:39.688604+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":44,"num_read_kb":35,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.1d","version":"0'0","reported_seq":627,"reported_epoch":362,"state":"active+clean","last_fresh":"2024-03-11T21:40:45.412232+0000","last_change":"2024-03-11T21:35:21.744646+0000","last_active":"2024-03-11T21:40:45.412232+0000","last_peered":"2024-03-11T21:40:45.412232+0000","last_clean":"2024-03-11T21:40:45.412232+0000","last_became_active":"2024-03-11T21:25:44.588845+0000","last_became_peered":"2024-03-11T21:25:44.588845+0000","last_unstale":"2024-03-11T21:40:45.412232+0000","last_undegraded":"2024-03-11T21:40:45.412232+0000","last_fullsized":"2024-03-11T21:40:45.412232+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:35:21.744223+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:43.559452+0000","last_clean_scrub_stamp":"2024-03-11T21:35:21.744223+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:40:50.059297+0000","scrub_duration":99,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.1e","version":"0'0","reported_seq":650,"reported_epoch":363,"state":"active+clean","last_fresh":"2024-03-11T21:40:48.411512+0000","last_change":"2024-03-11T21:40:28.163223+0000","last_active":"2024-03-11T21:40:48.411512+0000","last_peered":"2024-03-11T21:40:48.411512+0000","last_clean":"2024-03-11T21:40:48.411512+0000","last_became_active":"2024-03-11T21:25:42.582959+0000","last_became_peered":"2024-03-11T21:25:42.582959+0000","last_unstale":"2024-03-11T21:40:48.411512+0000","last_undegraded":"2024-03-11T21:40:48.411512+0000","last_fullsized":"2024-03-11T21:40:48.411512+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:40:28.163142+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:26:58.335431+0000","last_clean_scrub_stamp":"2024-03-11T21:40:28.163142+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:41:55.681758+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.19","version":"67'2","reported_seq":587,"reported_epoch":363,"state":"active+clean","last_fresh":"2024-03-11T21:40:48.881287+0000","last_change":"2024-03-11T21:36:35.069729+0000","last_active":"2024-03-11T21:40:48.881287+0000","last_peered":"2024-03-11T21:40:48.881287+0000","last_clean":"2024-03-11T21:40:48.881287+0000","last_became_active":"2024-03-11T21:36:35.064249+0000","last_became_peered":"2024-03-11T21:36:35.064249+0000","last_unstale":"2024-03-11T21:40:48.881287+0000","last_undegraded":"2024-03-11T21:40:48.881287+0000","last_fullsized":"2024-03-11T21:40:48.881287+0000","mapping_epoch":277,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":278,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'2","last_scrub_stamp":"2024-03-11T21:32:15.509542+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:37.512018+0000","last_clean_scrub_stamp":"2024-03-11T21:32:15.509542+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:40:52.711007+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":957,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,0],"acting":[3,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1f","version":"260'24","reported_seq":722,"reported_epoch":362,"state":"active+clean","last_fresh":"2024-03-11T21:40:46.105043+0000","last_change":"2024-03-11T21:40:25.079945+0000","last_active":"2024-03-11T21:40:46.105043+0000","last_peered":"2024-03-11T21:40:46.105043+0000","last_clean":"2024-03-11T21:40:46.105043+0000","last_became_active":"2024-03-11T21:32:22.798242+0000","last_became_peered":"2024-03-11T21:32:22.798242+0000","last_unstale":"2024-03-11T21:40:46.105043+0000","last_undegraded":"2024-03-11T21:40:46.105043+0000","last_fullsized":"2024-03-11T21:40:46.105043+0000","mapping_epoch":166,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":167,"parent":"0.0","parent_split_bits":0,"last_scrub":"260'24","last_scrub_stamp":"2024-03-11T21:40:25.079808+0000","last_deep_scrub":"124'16","last_deep_scrub_stamp":"2024-03-11T21:35:38.173841+0000","last_clean_scrub_stamp":"2024-03-11T21:40:25.079808+0000","objects_scrubbed":5,"log_size":24,"log_dups_size":0,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:41:47.993849+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":32,"num_read_kb":25,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,0],"acting":[6,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.1c","version":"74'1","reported_seq":295,"reported_epoch":362,"state":"active+clean","last_fresh":"2024-03-11T21:40:46.061783+0000","last_change":"2024-03-11T21:36:11.732043+0000","last_active":"2024-03-11T21:40:46.061783+0000","last_peered":"2024-03-11T21:40:46.061783+0000","last_clean":"2024-03-11T21:40:46.061783+0000","last_became_active":"2024-03-11T21:34:41.144839+0000","last_became_peered":"2024-03-11T21:34:41.144839+0000","last_unstale":"2024-03-11T21:40:46.061783+0000","last_undegraded":"2024-03-11T21:40:46.061783+0000","last_fullsized":"2024-03-11T21:40:46.061783+0000","mapping_epoch":226,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":227,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'1","last_scrub_stamp":"2024-03-11T21:36:11.731914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:43.559452+0000","last_clean_scrub_stamp":"2024-03-11T21:36:11.731914+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":370,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":370,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.1f","version":"229'10","reported_seq":877,"reported_epoch":362,"state":"active+clean","last_fresh":"2024-03-11T21:40:46.104996+0000","last_change":"2024-03-11T21:39:42.162545+0000","last_active":"2024-03-11T21:40:46.104996+0000","last_peered":"2024-03-11T21:40:46.104996+0000","last_clean":"2024-03-11T21:40:46.104996+0000","last_became_active":"2024-03-11T21:34:41.145862+0000","last_became_peered":"2024-03-11T21:34:41.145862+0000","last_unstale":"2024-03-11T21:40:46.104996+0000","last_undegraded":"2024-03-11T21:40:46.104996+0000","last_fullsized":"2024-03-11T21:40:46.104996+0000","mapping_epoch":226,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":227,"parent":"0.0","parent_split_bits":0,"last_scrub":"229'10","last_scrub_stamp":"2024-03-11T21:39:42.162463+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:41.544550+0000","last_clean_scrub_stamp":"2024-03-11T21:39:42.162463+0000","objects_scrubbed":1,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:41:06.487514+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.18","version":"67'1","reported_seq":690,"reported_epoch":362,"state":"active+clean","last_fresh":"2024-03-11T21:40:46.103644+0000","last_change":"2024-03-11T21:36:10.458150+0000","last_active":"2024-03-11T21:40:46.103644+0000","last_peered":"2024-03-11T21:40:46.103644+0000","last_clean":"2024-03-11T21:40:46.103644+0000","last_became_active":"2024-03-11T21:35:23.310241+0000","last_became_peered":"2024-03-11T21:35:23.310241+0000","last_unstale":"2024-03-11T21:40:46.103644+0000","last_undegraded":"2024-03-11T21:40:46.103644+0000","last_fullsized":"2024-03-11T21:40:46.103644+0000","mapping_epoch":248,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":249,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'1","last_scrub_stamp":"2024-03-11T21:36:10.458028+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:37.512018+0000","last_clean_scrub_stamp":"2024-03-11T21:36:10.458028+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":436,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":7,"num_read_kb":7,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":1308,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,2],"acting":[5,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1e","version":"260'20","reported_seq":734,"reported_epoch":363,"state":"active+clean","last_fresh":"2024-03-11T21:40:48.411460+0000","last_change":"2024-03-11T21:36:35.083447+0000","last_active":"2024-03-11T21:40:48.411460+0000","last_peered":"2024-03-11T21:40:48.411460+0000","last_clean":"2024-03-11T21:40:48.411460+0000","last_became_active":"2024-03-11T21:36:35.057843+0000","last_became_peered":"2024-03-11T21:36:35.057843+0000","last_unstale":"2024-03-11T21:40:48.411460+0000","last_undegraded":"2024-03-11T21:40:48.411460+0000","last_fullsized":"2024-03-11T21:40:48.411460+0000","mapping_epoch":277,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":278,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'10","last_scrub_stamp":"2024-03-11T21:35:16.825146+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:39.525554+0000","last_clean_scrub_stamp":"2024-03-11T21:35:16.825146+0000","objects_scrubbed":5,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:40:49.118310+0000","scrub_duration":44,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":25,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.13","version":"0'0","reported_seq":520,"reported_epoch":363,"state":"active+clean","last_fresh":"2024-03-11T21:40:48.881865+0000","last_change":"2024-03-11T21:36:35.063744+0000","last_active":"2024-03-11T21:40:48.881865+0000","last_peered":"2024-03-11T21:40:48.881865+0000","last_clean":"2024-03-11T21:40:48.881865+0000","last_became_active":"2024-03-11T21:36:35.062618+0000","last_became_peered":"2024-03-11T21:36:35.062618+0000","last_unstale":"2024-03-11T21:40:48.881865+0000","last_undegraded":"2024-03-11T21:40:48.881865+0000","last_fullsized":"2024-03-11T21:40:48.881865+0000","mapping_epoch":277,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":278,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:34:13.822803+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:43.559452+0000","last_clean_scrub_stamp":"2024-03-11T21:34:13.822803+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:40:52.711126+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":299,"reported_epoch":362,"state":"active+clean","last_fresh":"2024-03-11T21:40:46.061166+0000","last_change":"2024-03-11T21:40:29.084178+0000","last_active":"2024-03-11T21:40:46.061166+0000","last_peered":"2024-03-11T21:40:46.061166+0000","last_clean":"2024-03-11T21:40:46.061166+0000","last_became_active":"2024-03-11T21:34:41.142569+0000","last_became_peered":"2024-03-11T21:34:41.142569+0000","last_unstale":"2024-03-11T21:40:46.061166+0000","last_undegraded":"2024-03-11T21:40:46.061166+0000","last_fullsized":"2024-03-11T21:40:46.061166+0000","mapping_epoch":226,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":227,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:40:29.084062+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:40:29.084062+0000","last_clean_scrub_stamp":"2024-03-11T21:40:29.084062+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:41:31.066228+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.17","version":"0'0","reported_seq":663,"reported_epoch":362,"state":"active+clean","last_fresh":"2024-03-11T21:40:46.104836+0000","last_change":"2024-03-11T21:40:18.112951+0000","last_active":"2024-03-11T21:40:46.104836+0000","last_peered":"2024-03-11T21:40:46.104836+0000","last_clean":"2024-03-11T21:40:46.104836+0000","last_became_active":"2024-03-11T21:25:38.540349+0000","last_became_peered":"2024-03-11T21:25:38.540349+0000","last_unstale":"2024-03-11T21:40:46.104836+0000","last_undegraded":"2024-03-11T21:40:46.104836+0000","last_fullsized":"2024-03-11T21:40:46.104836+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:40:18.112829+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:28:25.918101+0000","last_clean_scrub_stamp":"2024-03-11T21:40:18.112829+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:41:20.570429+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,2],"acting":[6,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.11","version":"260'19","reported_seq":646,"reported_epoch":363,"state":"active+clean","last_fresh":"2024-03-11T21:40:48.881618+0000","last_change":"2024-03-11T21:36:35.096720+0000","last_active":"2024-03-11T21:40:48.881618+0000","last_peered":"2024-03-11T21:40:48.881618+0000","last_clean":"2024-03-11T21:40:48.881618+0000","last_became_active":"2024-03-11T21:36:35.073689+0000","last_became_peered":"2024-03-11T21:36:35.073689+0000","last_unstale":"2024-03-11T21:40:48.881618+0000","last_undegraded":"2024-03-11T21:40:48.881618+0000","last_fullsized":"2024-03-11T21:40:48.881618+0000","mapping_epoch":277,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":278,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'11","last_scrub_stamp":"2024-03-11T21:35:16.044397+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:39.525554+0000","last_clean_scrub_stamp":"2024-03-11T21:35:16.044397+0000","objects_scrubbed":5,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:40:52.711175+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":31,"num_read_kb":24,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":291,"reported_epoch":362,"state":"active+clean","last_fresh":"2024-03-11T21:40:46.061095+0000","last_change":"2024-03-11T21:35:32.719869+0000","last_active":"2024-03-11T21:40:46.061095+0000","last_peered":"2024-03-11T21:40:46.061095+0000","last_clean":"2024-03-11T21:40:46.061095+0000","last_became_active":"2024-03-11T21:34:41.137235+0000","last_became_peered":"2024-03-11T21:34:41.137235+0000","last_unstale":"2024-03-11T21:40:46.061095+0000","last_undegraded":"2024-03-11T21:40:46.061095+0000","last_fullsized":"2024-03-11T21:40:46.061095+0000","mapping_epoch":226,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":227,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:35:32.719789+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:43.559452+0000","last_clean_scrub_stamp":"2024-03-11T21:35:32.719789+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,4],"acting":[7,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":675,"reported_epoch":362,"state":"active+clean","last_fresh":"2024-03-11T21:40:46.103883+0000","last_change":"2024-03-11T21:40:03.181212+0000","last_active":"2024-03-11T21:40:46.103883+0000","last_peered":"2024-03-11T21:40:46.103883+0000","last_clean":"2024-03-11T21:40:46.103883+0000","last_became_active":"2024-03-11T21:34:41.156036+0000","last_became_peered":"2024-03-11T21:34:41.156036+0000","last_unstale":"2024-03-11T21:40:46.103883+0000","last_undegraded":"2024-03-11T21:40:46.103883+0000","last_fullsized":"2024-03-11T21:40:46.103883+0000","mapping_epoch":226,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":227,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:40:03.181074+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:38:51.036736+0000","last_clean_scrub_stamp":"2024-03-11T21:40:03.181074+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:41:28.087147+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.16","version":"0'0","reported_seq":654,"reported_epoch":362,"state":"active+clean","last_fresh":"2024-03-11T21:40:46.104942+0000","last_change":"2024-03-11T21:35:39.108623+0000","last_active":"2024-03-11T21:40:46.104942+0000","last_peered":"2024-03-11T21:40:46.104942+0000","last_clean":"2024-03-11T21:40:46.104942+0000","last_became_active":"2024-03-11T21:25:38.539103+0000","last_became_peered":"2024-03-11T21:25:38.539103+0000","last_unstale":"2024-03-11T21:40:46.104942+0000","last_undegraded":"2024-03-11T21:40:46.104942+0000","last_fullsized":"2024-03-11T21:40:46.104942+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:35:39.108546+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:27:11.938909+0000","last_clean_scrub_stamp":"2024-03-11T21:35:39.108546+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,2],"acting":[5,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.10","version":"260'10","reported_seq":528,"reported_epoch":363,"state":"active+clean","last_fresh":"2024-03-11T21:40:48.881121+0000","last_change":"2024-03-11T21:36:35.084268+0000","last_active":"2024-03-11T21:40:48.881121+0000","last_peered":"2024-03-11T21:40:48.881121+0000","last_clean":"2024-03-11T21:40:48.881121+0000","last_became_active":"2024-03-11T21:36:35.067500+0000","last_became_peered":"2024-03-11T21:36:35.067500+0000","last_unstale":"2024-03-11T21:40:48.881121+0000","last_undegraded":"2024-03-11T21:40:48.881121+0000","last_fullsized":"2024-03-11T21:40:48.881121+0000","mapping_epoch":277,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":278,"parent":"0.0","parent_split_bits":0,"last_scrub":"94'6","last_scrub_stamp":"2024-03-11T21:34:00.727416+0000","last_deep_scrub":"94'6","last_deep_scrub_stamp":"2024-03-11T21:28:24.603974+0000","last_clean_scrub_stamp":"2024-03-11T21:34:00.727416+0000","objects_scrubbed":2,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:40:52.711093+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":12,"num_read_kb":10,"num_write":8,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6],"acting":[3,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":573,"reported_epoch":363,"state":"active+clean","last_fresh":"2024-03-11T21:40:48.881495+0000","last_change":"2024-03-11T21:36:35.070468+0000","last_active":"2024-03-11T21:40:48.881495+0000","last_peered":"2024-03-11T21:40:48.881495+0000","last_clean":"2024-03-11T21:40:48.881495+0000","last_became_active":"2024-03-11T21:36:35.070176+0000","last_became_peered":"2024-03-11T21:36:35.070176+0000","last_unstale":"2024-03-11T21:40:48.881495+0000","last_undegraded":"2024-03-11T21:40:48.881495+0000","last_fullsized":"2024-03-11T21:40:48.881495+0000","mapping_epoch":277,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":278,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:35:18.062272+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:43.559452+0000","last_clean_scrub_stamp":"2024-03-11T21:35:18.062272+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:40:52.711192+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":640,"reported_epoch":362,"state":"active+clean","last_fresh":"2024-03-11T21:40:45.411810+0000","last_change":"2024-03-11T21:36:35.058551+0000","last_active":"2024-03-11T21:40:45.411810+0000","last_peered":"2024-03-11T21:40:45.411810+0000","last_clean":"2024-03-11T21:40:45.411810+0000","last_became_active":"2024-03-11T21:36:35.057906+0000","last_became_peered":"2024-03-11T21:36:35.057906+0000","last_unstale":"2024-03-11T21:40:45.411810+0000","last_undegraded":"2024-03-11T21:40:45.411810+0000","last_fullsized":"2024-03-11T21:40:45.411810+0000","mapping_epoch":277,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":278,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:34:15.516278+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:27:21.207303+0000","last_clean_scrub_stamp":"2024-03-11T21:34:15.516278+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:40:50.059201+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.15","version":"0'0","reported_seq":647,"reported_epoch":362,"state":"active+clean","last_fresh":"2024-03-11T21:40:45.412382+0000","last_change":"2024-03-11T21:36:06.579754+0000","last_active":"2024-03-11T21:40:45.412382+0000","last_peered":"2024-03-11T21:40:45.412382+0000","last_clean":"2024-03-11T21:40:45.412382+0000","last_became_active":"2024-03-11T21:25:38.541705+0000","last_became_peered":"2024-03-11T21:25:38.541705+0000","last_unstale":"2024-03-11T21:40:45.412382+0000","last_undegraded":"2024-03-11T21:40:45.412382+0000","last_fullsized":"2024-03-11T21:40:45.412382+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:36:06.579636+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:26:44.860499+0000","last_clean_scrub_stamp":"2024-03-11T21:36:06.579636+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:40:50.059537+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.13","version":"260'24","reported_seq":705,"reported_epoch":362,"state":"active+clean","last_fresh":"2024-03-11T21:40:45.410766+0000","last_change":"2024-03-11T21:40:25.595320+0000","last_active":"2024-03-11T21:40:45.410766+0000","last_peered":"2024-03-11T21:40:45.410766+0000","last_clean":"2024-03-11T21:40:45.410766+0000","last_became_active":"2024-03-11T21:26:11.214879+0000","last_became_peered":"2024-03-11T21:26:11.214879+0000","last_unstale":"2024-03-11T21:40:45.410766+0000","last_undegraded":"2024-03-11T21:40:45.410766+0000","last_fullsized":"2024-03-11T21:40:45.410766+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"260'24","last_scrub_stamp":"2024-03-11T21:40:25.595220+0000","last_deep_scrub":"260'24","last_deep_scrub_stamp":"2024-03-11T21:40:25.595220+0000","last_clean_scrub_stamp":"2024-03-11T21:40:25.595220+0000","objects_scrubbed":5,"log_size":24,"log_dups_size":0,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:41:39.345264+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":32,"num_read_kb":25,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,2],"acting":[4,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.10","version":"0'0","reported_seq":637,"reported_epoch":363,"state":"active+clean","last_fresh":"2024-03-11T21:40:48.410734+0000","last_change":"2024-03-11T21:35:29.712658+0000","last_active":"2024-03-11T21:40:48.410734+0000","last_peered":"2024-03-11T21:40:48.410734+0000","last_clean":"2024-03-11T21:40:48.410734+0000","last_became_active":"2024-03-11T21:25:44.591189+0000","last_became_peered":"2024-03-11T21:25:44.591189+0000","last_unstale":"2024-03-11T21:40:48.410734+0000","last_undegraded":"2024-03-11T21:40:48.410734+0000","last_fullsized":"2024-03-11T21:40:48.410734+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:35:29.712557+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:27:25.417450+0000","last_clean_scrub_stamp":"2024-03-11T21:35:29.712557+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:40:49.118415+0000","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,1],"acting":[0,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":571,"reported_epoch":363,"state":"active+clean","last_fresh":"2024-03-11T21:40:48.881890+0000","last_change":"2024-03-11T21:36:35.062276+0000","last_active":"2024-03-11T21:40:48.881890+0000","last_peered":"2024-03-11T21:40:48.881890+0000","last_clean":"2024-03-11T21:40:48.881890+0000","last_became_active":"2024-03-11T21:36:35.056619+0000","last_became_peered":"2024-03-11T21:36:35.056619+0000","last_unstale":"2024-03-11T21:40:48.881890+0000","last_undegraded":"2024-03-11T21:40:48.881890+0000","last_fullsized":"2024-03-11T21:40:48.881890+0000","mapping_epoch":277,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":278,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:35:13.060487+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:31:15.367898+0000","last_clean_scrub_stamp":"2024-03-11T21:35:13.060487+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:40:52.711262+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.14","version":"0'0","reported_seq":680,"reported_epoch":362,"state":"active+clean","last_fresh":"2024-03-11T21:40:46.103966+0000","last_change":"2024-03-11T21:40:01.212243+0000","last_active":"2024-03-11T21:40:46.103966+0000","last_peered":"2024-03-11T21:40:46.103966+0000","last_clean":"2024-03-11T21:40:46.103966+0000","last_became_active":"2024-03-11T21:36:35.068038+0000","last_became_peered":"2024-03-11T21:36:35.068038+0000","last_unstale":"2024-03-11T21:40:46.103966+0000","last_undegraded":"2024-03-11T21:40:46.103966+0000","last_fullsized":"2024-03-11T21:40:46.103966+0000","mapping_epoch":277,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":278,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:40:01.212145+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:37.512018+0000","last_clean_scrub_stamp":"2024-03-11T21:40:01.212145+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:41:09.997136+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,5],"acting":[6,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.12","version":"260'15","reported_seq":668,"reported_epoch":362,"state":"active+clean","last_fresh":"2024-03-11T21:40:45.410755+0000","last_change":"2024-03-11T21:35:32.590439+0000","last_active":"2024-03-11T21:40:45.410755+0000","last_peered":"2024-03-11T21:40:45.410755+0000","last_clean":"2024-03-11T21:40:45.410755+0000","last_became_active":"2024-03-11T21:25:40.547082+0000","last_became_peered":"2024-03-11T21:25:40.547082+0000","last_unstale":"2024-03-11T21:40:45.410755+0000","last_undegraded":"2024-03-11T21:40:45.410755+0000","last_fullsized":"2024-03-11T21:40:45.410755+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'9","last_scrub_stamp":"2024-03-11T21:35:32.590318+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:39.525554+0000","last_clean_scrub_stamp":"2024-03-11T21:35:32.590318+0000","objects_scrubbed":4,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:40:50.059345+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":635,"reported_epoch":362,"state":"active+clean","last_fresh":"2024-03-11T21:40:45.411515+0000","last_change":"2024-03-11T21:36:04.754168+0000","last_active":"2024-03-11T21:40:45.411515+0000","last_peered":"2024-03-11T21:40:45.411515+0000","last_clean":"2024-03-11T21:40:45.411515+0000","last_became_active":"2024-03-11T21:25:44.589586+0000","last_became_peered":"2024-03-11T21:25:44.589586+0000","last_unstale":"2024-03-11T21:40:45.411515+0000","last_undegraded":"2024-03-11T21:40:45.411515+0000","last_fullsized":"2024-03-11T21:40:45.411515+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:36:04.753979+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:27:09.801138+0000","last_clean_scrub_stamp":"2024-03-11T21:36:04.753979+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":113,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,5],"acting":[4,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.14","version":"280'12","reported_seq":784,"reported_epoch":363,"state":"active+clean","last_fresh":"2024-03-11T21:40:48.882144+0000","last_change":"2024-03-11T21:36:35.074550+0000","last_active":"2024-03-11T21:40:48.882144+0000","last_peered":"2024-03-11T21:40:48.882144+0000","last_clean":"2024-03-11T21:40:48.882144+0000","last_became_active":"2024-03-11T21:36:35.073912+0000","last_became_peered":"2024-03-11T21:36:35.073912+0000","last_unstale":"2024-03-11T21:40:48.882144+0000","last_undegraded":"2024-03-11T21:40:48.882144+0000","last_fullsized":"2024-03-11T21:40:48.882144+0000","mapping_epoch":277,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":278,"parent":"0.0","parent_split_bits":0,"last_scrub":"229'10","last_scrub_stamp":"2024-03-11T21:35:26.182107+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:41.544550+0000","last_clean_scrub_stamp":"2024-03-11T21:35:26.182107+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:40:52.711328+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,2],"acting":[3,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.13","version":"0'0","reported_seq":658,"reported_epoch":363,"state":"active+clean","last_fresh":"2024-03-11T21:40:48.411458+0000","last_change":"2024-03-11T21:40:11.027382+0000","last_active":"2024-03-11T21:40:48.411458+0000","last_peered":"2024-03-11T21:40:48.411458+0000","last_clean":"2024-03-11T21:40:48.411458+0000","last_became_active":"2024-03-11T21:36:35.057132+0000","last_became_peered":"2024-03-11T21:36:35.057132+0000","last_unstale":"2024-03-11T21:40:48.411458+0000","last_undegraded":"2024-03-11T21:40:48.411458+0000","last_fullsized":"2024-03-11T21:40:48.411458+0000","mapping_epoch":277,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":278,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:40:11.027268+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:37.512018+0000","last_clean_scrub_stamp":"2024-03-11T21:40:11.027268+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:41:40.995455+0000","scrub_duration":24,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.15","version":"260'19","reported_seq":721,"reported_epoch":362,"state":"active+clean","last_fresh":"2024-03-11T21:40:46.103941+0000","last_change":"2024-03-11T21:36:35.088986+0000","last_active":"2024-03-11T21:40:46.103941+0000","last_peered":"2024-03-11T21:40:46.103941+0000","last_clean":"2024-03-11T21:40:46.103941+0000","last_became_active":"2024-03-11T21:36:35.057285+0000","last_became_peered":"2024-03-11T21:36:35.057285+0000","last_unstale":"2024-03-11T21:40:46.103941+0000","last_undegraded":"2024-03-11T21:40:46.103941+0000","last_fullsized":"2024-03-11T21:40:46.103941+0000","mapping_epoch":277,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":278,"parent":"0.0","parent_split_bits":0,"last_scrub":"94'13","last_scrub_stamp":"2024-03-11T21:35:29.178543+0000","last_deep_scrub":"94'13","last_deep_scrub_stamp":"2024-03-11T21:29:41.853158+0000","last_clean_scrub_stamp":"2024-03-11T21:35:29.178543+0000","objects_scrubbed":4,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":11,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.16","version":"0'0","reported_seq":666,"reported_epoch":363,"state":"active+clean","last_fresh":"2024-03-11T21:40:48.411227+0000","last_change":"2024-03-11T21:40:30.249810+0000","last_active":"2024-03-11T21:40:48.411227+0000","last_peered":"2024-03-11T21:40:48.411227+0000","last_clean":"2024-03-11T21:40:48.411227+0000","last_became_active":"2024-03-11T21:36:35.057355+0000","last_became_peered":"2024-03-11T21:36:35.057355+0000","last_unstale":"2024-03-11T21:40:48.411227+0000","last_undegraded":"2024-03-11T21:40:48.411227+0000","last_fullsized":"2024-03-11T21:40:48.411227+0000","mapping_epoch":277,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":278,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:40:30.249679+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:40:30.249679+0000","last_clean_scrub_stamp":"2024-03-11T21:40:30.249679+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:41:59.927156+0000","scrub_duration":40,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.15","version":"98'9","reported_seq":829,"reported_epoch":362,"state":"active+clean","last_fresh":"2024-03-11T21:40:46.104861+0000","last_change":"2024-03-11T21:36:03.366581+0000","last_active":"2024-03-11T21:40:46.104861+0000","last_peered":"2024-03-11T21:40:46.104861+0000","last_clean":"2024-03-11T21:40:46.104861+0000","last_became_active":"2024-03-11T21:25:42.572555+0000","last_became_peered":"2024-03-11T21:25:42.572555+0000","last_unstale":"2024-03-11T21:40:46.104861+0000","last_undegraded":"2024-03-11T21:40:46.104861+0000","last_fullsized":"2024-03-11T21:40:46.104861+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"98'9","last_scrub_stamp":"2024-03-11T21:36:03.366450+0000","last_deep_scrub":"98'9","last_deep_scrub_stamp":"2024-03-11T21:31:20.624923+0000","last_clean_scrub_stamp":"2024-03-11T21:36:03.366450+0000","objects_scrubbed":1,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.12","version":"0'0","reported_seq":667,"reported_epoch":362,"state":"active+clean","last_fresh":"2024-03-11T21:40:46.104175+0000","last_change":"2024-03-11T21:36:35.058369+0000","last_active":"2024-03-11T21:40:46.104175+0000","last_peered":"2024-03-11T21:40:46.104175+0000","last_clean":"2024-03-11T21:40:46.104175+0000","last_became_active":"2024-03-11T21:36:35.058033+0000","last_became_peered":"2024-03-11T21:36:35.058033+0000","last_unstale":"2024-03-11T21:40:46.104175+0000","last_undegraded":"2024-03-11T21:40:46.104175+0000","last_fullsized":"2024-03-11T21:40:46.104175+0000","mapping_epoch":277,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":278,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:35:27.158468+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:37.512018+0000","last_clean_scrub_stamp":"2024-03-11T21:35:27.158468+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.14","version":"260'20","reported_seq":654,"reported_epoch":363,"state":"active+clean","last_fresh":"2024-03-11T21:40:48.881896+0000","last_change":"2024-03-11T21:36:35.095382+0000","last_active":"2024-03-11T21:40:48.881896+0000","last_peered":"2024-03-11T21:40:48.881896+0000","last_clean":"2024-03-11T21:40:48.881896+0000","last_became_active":"2024-03-11T21:36:35.069638+0000","last_became_peered":"2024-03-11T21:36:35.069638+0000","last_unstale":"2024-03-11T21:40:48.881896+0000","last_undegraded":"2024-03-11T21:40:48.881896+0000","last_fullsized":"2024-03-11T21:40:48.881896+0000","mapping_epoch":277,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":278,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'10","last_scrub_stamp":"2024-03-11T21:35:28.189296+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:39.525554+0000","last_clean_scrub_stamp":"2024-03-11T21:35:28.189296+0000","objects_scrubbed":0,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:40:52.711345+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":29,"num_read_kb":24,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.15","version":"0'0","reported_seq":291,"reported_epoch":362,"state":"active+clean","last_fresh":"2024-03-11T21:40:46.061798+0000","last_change":"2024-03-11T21:36:14.736240+0000","last_active":"2024-03-11T21:40:46.061798+0000","last_peered":"2024-03-11T21:40:46.061798+0000","last_clean":"2024-03-11T21:40:46.061798+0000","last_became_active":"2024-03-11T21:34:41.142760+0000","last_became_peered":"2024-03-11T21:34:41.142760+0000","last_unstale":"2024-03-11T21:40:46.061798+0000","last_undegraded":"2024-03-11T21:40:46.061798+0000","last_fullsized":"2024-03-11T21:40:46.061798+0000","mapping_epoch":226,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":227,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:36:14.735985+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:43.559452+0000","last_clean_scrub_stamp":"2024-03-11T21:36:14.735985+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":36,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":647,"reported_epoch":362,"state":"active+clean","last_fresh":"2024-03-11T21:40:46.104937+0000","last_change":"2024-03-11T21:40:26.587777+0000","last_active":"2024-03-11T21:40:46.104937+0000","last_peered":"2024-03-11T21:40:46.104937+0000","last_clean":"2024-03-11T21:40:46.104937+0000","last_became_active":"2024-03-11T21:36:35.060342+0000","last_became_peered":"2024-03-11T21:36:35.060342+0000","last_unstale":"2024-03-11T21:40:46.104937+0000","last_undegraded":"2024-03-11T21:40:46.104937+0000","last_fullsized":"2024-03-11T21:40:46.104937+0000","mapping_epoch":277,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":278,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:40:26.587672+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:41.544550+0000","last_clean_scrub_stamp":"2024-03-11T21:40:26.587672+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:41:33.380076+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,1],"acting":[5,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.11","version":"0'0","reported_seq":663,"reported_epoch":362,"state":"active+clean","last_fresh":"2024-03-11T21:40:46.103991+0000","last_change":"2024-03-11T21:40:09.218006+0000","last_active":"2024-03-11T21:40:46.103991+0000","last_peered":"2024-03-11T21:40:46.103991+0000","last_clean":"2024-03-11T21:40:46.103991+0000","last_became_active":"2024-03-11T21:25:38.537415+0000","last_became_peered":"2024-03-11T21:25:38.537415+0000","last_unstale":"2024-03-11T21:40:46.103991+0000","last_undegraded":"2024-03-11T21:40:46.103991+0000","last_fullsized":"2024-03-11T21:40:46.103991+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:40:09.217927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:37.512018+0000","last_clean_scrub_stamp":"2024-03-11T21:40:09.217927+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:41:24.546509+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.17","version":"260'15","reported_seq":599,"reported_epoch":363,"state":"active+clean","last_fresh":"2024-03-11T21:40:48.882622+0000","last_change":"2024-03-11T21:36:35.099451+0000","last_active":"2024-03-11T21:40:48.882622+0000","last_peered":"2024-03-11T21:40:48.882622+0000","last_clean":"2024-03-11T21:40:48.882622+0000","last_became_active":"2024-03-11T21:36:35.071346+0000","last_became_peered":"2024-03-11T21:36:35.071346+0000","last_unstale":"2024-03-11T21:40:48.882622+0000","last_undegraded":"2024-03-11T21:40:48.882622+0000","last_fullsized":"2024-03-11T21:40:48.882622+0000","mapping_epoch":277,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":278,"parent":"0.0","parent_split_bits":0,"last_scrub":"97'9","last_scrub_stamp":"2024-03-11T21:32:08.495352+0000","last_deep_scrub":"97'9","last_deep_scrub_stamp":"2024-03-11T21:27:25.803132+0000","last_clean_scrub_stamp":"2024-03-11T21:32:08.495352+0000","objects_scrubbed":3,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:40:52.710939+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":17,"num_read_kb":14,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,5],"acting":[3,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.14","version":"0'0","reported_seq":645,"reported_epoch":362,"state":"active+clean","last_fresh":"2024-03-11T21:40:46.104619+0000","last_change":"2024-03-11T21:35:27.606664+0000","last_active":"2024-03-11T21:40:46.104619+0000","last_peered":"2024-03-11T21:40:46.104619+0000","last_clean":"2024-03-11T21:40:46.104619+0000","last_became_active":"2024-03-11T21:34:41.136946+0000","last_became_peered":"2024-03-11T21:34:41.136946+0000","last_unstale":"2024-03-11T21:40:46.104619+0000","last_undegraded":"2024-03-11T21:40:46.104619+0000","last_fullsized":"2024-03-11T21:40:46.104619+0000","mapping_epoch":226,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":227,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:35:27.606551+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:43.559452+0000","last_clean_scrub_stamp":"2024-03-11T21:35:27.606551+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,7],"acting":[2,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":553,"reported_epoch":363,"state":"active+clean","last_fresh":"2024-03-11T21:40:48.882528+0000","last_change":"2024-03-11T21:36:35.076782+0000","last_active":"2024-03-11T21:40:48.882528+0000","last_peered":"2024-03-11T21:40:48.882528+0000","last_clean":"2024-03-11T21:40:48.882528+0000","last_became_active":"2024-03-11T21:36:35.076335+0000","last_became_peered":"2024-03-11T21:36:35.076335+0000","last_unstale":"2024-03-11T21:40:48.882528+0000","last_undegraded":"2024-03-11T21:40:48.882528+0000","last_fullsized":"2024-03-11T21:40:48.882528+0000","mapping_epoch":277,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":278,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:34:33.082577+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:41.544550+0000","last_clean_scrub_stamp":"2024-03-11T21:34:33.082577+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:40:52.711158+0000","scrub_duration":35,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.10","version":"0'0","reported_seq":645,"reported_epoch":362,"state":"active+clean","last_fresh":"2024-03-11T21:40:46.103145+0000","last_change":"2024-03-11T21:35:37.694623+0000","last_active":"2024-03-11T21:40:46.103145+0000","last_peered":"2024-03-11T21:40:46.103145+0000","last_clean":"2024-03-11T21:40:46.103145+0000","last_became_active":"2024-03-11T21:25:38.533138+0000","last_became_peered":"2024-03-11T21:25:38.533138+0000","last_unstale":"2024-03-11T21:40:46.103145+0000","last_undegraded":"2024-03-11T21:40:46.103145+0000","last_fullsized":"2024-03-11T21:40:46.103145+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:35:37.694533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:37.512018+0000","last_clean_scrub_stamp":"2024-03-11T21:35:37.694533+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,0],"acting":[2,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.16","version":"260'15","reported_seq":720,"reported_epoch":363,"state":"active+clean","last_fresh":"2024-03-11T21:40:48.411337+0000","last_change":"2024-03-11T21:36:35.080158+0000","last_active":"2024-03-11T21:40:48.411337+0000","last_peered":"2024-03-11T21:40:48.411337+0000","last_clean":"2024-03-11T21:40:48.411337+0000","last_became_active":"2024-03-11T21:36:35.056862+0000","last_became_peered":"2024-03-11T21:36:35.056862+0000","last_unstale":"2024-03-11T21:40:48.411337+0000","last_undegraded":"2024-03-11T21:40:48.411337+0000","last_fullsized":"2024-03-11T21:40:48.411337+0000","mapping_epoch":277,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":278,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'9","last_scrub_stamp":"2024-03-11T21:35:12.741225+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:39.525554+0000","last_clean_scrub_stamp":"2024-03-11T21:35:12.741225+0000","objects_scrubbed":4,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:40:49.118330+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":11,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.7","version":"0'0","reported_seq":21,"reported_epoch":362,"state":"active+clean","last_fresh":"2024-03-11T21:40:46.104524+0000","last_change":"2024-03-11T21:40:39.541210+0000","last_active":"2024-03-11T21:40:46.104524+0000","last_peered":"2024-03-11T21:40:46.104524+0000","last_clean":"2024-03-11T21:40:46.104524+0000","last_became_active":"2024-03-11T21:40:39.540781+0000","last_became_peered":"2024-03-11T21:40:39.540781+0000","last_unstale":"2024-03-11T21:40:46.104524+0000","last_undegraded":"2024-03-11T21:40:46.104524+0000","last_fullsized":"2024-03-11T21:40:46.104524+0000","mapping_epoch":356,"log_start":"0'0","ondisk_log_start":"0'0","created":356,"last_epoch_clean":357,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:40:38.219602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:40:38.219602+0000","last_clean_scrub_stamp":"2024-03-11T21:40:38.219602+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:41:43.847787+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,5],"acting":[6,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.b","version":"0'0","reported_seq":549,"reported_epoch":363,"state":"active+clean","last_fresh":"2024-03-11T21:40:48.881768+0000","last_change":"2024-03-11T21:40:27.611092+0000","last_active":"2024-03-11T21:40:48.881768+0000","last_peered":"2024-03-11T21:40:48.881768+0000","last_clean":"2024-03-11T21:40:48.881768+0000","last_became_active":"2024-03-11T21:36:35.072147+0000","last_became_peered":"2024-03-11T21:36:35.072147+0000","last_unstale":"2024-03-11T21:40:48.881768+0000","last_undegraded":"2024-03-11T21:40:48.881768+0000","last_fullsized":"2024-03-11T21:40:48.881768+0000","mapping_epoch":277,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":278,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:40:27.610987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:43.559452+0000","last_clean_scrub_stamp":"2024-03-11T21:40:27.610987+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:41:47.989897+0000","scrub_duration":60,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.f","version":"67'2","reported_seq":659,"reported_epoch":362,"state":"active+clean","last_fresh":"2024-03-11T21:40:45.411353+0000","last_change":"2024-03-11T21:35:36.613667+0000","last_active":"2024-03-11T21:40:45.411353+0000","last_peered":"2024-03-11T21:40:45.411353+0000","last_clean":"2024-03-11T21:40:45.411353+0000","last_became_active":"2024-03-11T21:25:38.537711+0000","last_became_peered":"2024-03-11T21:25:38.537711+0000","last_unstale":"2024-03-11T21:40:45.411353+0000","last_undegraded":"2024-03-11T21:40:45.411353+0000","last_fullsized":"2024-03-11T21:40:45.411353+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'2","last_scrub_stamp":"2024-03-11T21:35:36.613516+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:37.512018+0000","last_clean_scrub_stamp":"2024-03-11T21:35:36.613516+0000","objects_scrubbed":2,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":92,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":10,"num_read_kb":10,"num_write":4,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,7],"acting":[4,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.9","version":"260'23","reported_seq":698,"reported_epoch":362,"state":"active+clean","last_fresh":"2024-03-11T21:40:45.410919+0000","last_change":"2024-03-11T21:40:14.543701+0000","last_active":"2024-03-11T21:40:45.410919+0000","last_peered":"2024-03-11T21:40:45.410919+0000","last_clean":"2024-03-11T21:40:45.410919+0000","last_became_active":"2024-03-11T21:36:35.059930+0000","last_became_peered":"2024-03-11T21:36:35.059930+0000","last_unstale":"2024-03-11T21:40:45.410919+0000","last_undegraded":"2024-03-11T21:40:45.410919+0000","last_fullsized":"2024-03-11T21:40:45.410919+0000","mapping_epoch":277,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":278,"parent":"0.0","parent_split_bits":0,"last_scrub":"260'23","last_scrub_stamp":"2024-03-11T21:40:14.543596+0000","last_deep_scrub":"146'17","last_deep_scrub_stamp":"2024-03-11T21:34:26.765075+0000","last_clean_scrub_stamp":"2024-03-11T21:40:14.543596+0000","objects_scrubbed":5,"log_size":23,"log_dups_size":0,"ondisk_log_size":23,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:41:34.525219+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":34,"num_read_kb":25,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,3],"acting":[4,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":637,"reported_epoch":362,"state":"active+clean","last_fresh":"2024-03-11T21:40:46.103176+0000","last_change":"2024-03-11T21:35:16.643421+0000","last_active":"2024-03-11T21:40:46.103176+0000","last_peered":"2024-03-11T21:40:46.103176+0000","last_clean":"2024-03-11T21:40:46.103176+0000","last_became_active":"2024-03-11T21:25:42.573987+0000","last_became_peered":"2024-03-11T21:25:42.573987+0000","last_unstale":"2024-03-11T21:40:46.103176+0000","last_undegraded":"2024-03-11T21:40:46.103176+0000","last_fullsized":"2024-03-11T21:40:46.103176+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:35:16.643311+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:41.544550+0000","last_clean_scrub_stamp":"2024-03-11T21:35:16.643311+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"10.c","version":"0'0","reported_seq":21,"reported_epoch":362,"state":"active+clean","last_fresh":"2024-03-11T21:40:46.061511+0000","last_change":"2024-03-11T21:40:39.242008+0000","last_active":"2024-03-11T21:40:46.061511+0000","last_peered":"2024-03-11T21:40:46.061511+0000","last_clean":"2024-03-11T21:40:46.061511+0000","last_became_active":"2024-03-11T21:40:39.241699+0000","last_became_peered":"2024-03-11T21:40:39.241699+0000","last_unstale":"2024-03-11T21:40:46.061511+0000","last_undegraded":"2024-03-11T21:40:46.061511+0000","last_fullsized":"2024-03-11T21:40:46.061511+0000","mapping_epoch":356,"log_start":"0'0","ondisk_log_start":"0'0","created":356,"last_epoch_clean":357,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:40:38.219602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:40:38.219602+0000","last_clean_scrub_stamp":"2024-03-11T21:40:38.219602+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:41:39.794513+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,0],"acting":[7,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.0","version":"0'0","reported_seq":647,"reported_epoch":363,"state":"active+clean","last_fresh":"2024-03-11T21:40:48.410771+0000","last_change":"2024-03-11T21:40:27.230013+0000","last_active":"2024-03-11T21:40:48.410771+0000","last_peered":"2024-03-11T21:40:48.410771+0000","last_clean":"2024-03-11T21:40:48.410771+0000","last_became_active":"2024-03-11T21:36:35.058054+0000","last_became_peered":"2024-03-11T21:36:35.058054+0000","last_unstale":"2024-03-11T21:40:48.410771+0000","last_undegraded":"2024-03-11T21:40:48.410771+0000","last_fullsized":"2024-03-11T21:40:48.410771+0000","mapping_epoch":277,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":278,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:40:27.229904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:29:39.498784+0000","last_clean_scrub_stamp":"2024-03-11T21:40:27.229904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:41:49.392723+0000","scrub_duration":38,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":660,"reported_epoch":362,"state":"active+clean","last_fresh":"2024-03-11T21:40:45.412105+0000","last_change":"2024-03-11T21:35:33.608063+0000","last_active":"2024-03-11T21:40:45.412105+0000","last_peered":"2024-03-11T21:40:45.412105+0000","last_clean":"2024-03-11T21:40:45.412105+0000","last_became_active":"2024-03-11T21:34:41.150913+0000","last_became_peered":"2024-03-11T21:34:41.150913+0000","last_unstale":"2024-03-11T21:40:45.412105+0000","last_undegraded":"2024-03-11T21:40:45.412105+0000","last_fullsized":"2024-03-11T21:40:45.412105+0000","mapping_epoch":226,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":227,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:35:33.607969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:37.512018+0000","last_clean_scrub_stamp":"2024-03-11T21:35:33.607969+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:40:50.059408+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,7],"acting":[1,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.2","version":"260'25","reported_seq":679,"reported_epoch":362,"state":"active+clean","last_fresh":"2024-03-11T21:40:45.410941+0000","last_change":"2024-03-11T21:35:17.666749+0000","last_active":"2024-03-11T21:40:45.410941+0000","last_peered":"2024-03-11T21:40:45.410941+0000","last_clean":"2024-03-11T21:40:45.410941+0000","last_became_active":"2024-03-11T21:25:40.548833+0000","last_became_peered":"2024-03-11T21:25:40.548833+0000","last_unstale":"2024-03-11T21:40:45.410941+0000","last_undegraded":"2024-03-11T21:40:45.410941+0000","last_fullsized":"2024-03-11T21:40:45.410941+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"94'15","last_scrub_stamp":"2024-03-11T21:35:17.666498+0000","last_deep_scrub":"94'15","last_deep_scrub_stamp":"2024-03-11T21:26:49.993060+0000","last_clean_scrub_stamp":"2024-03-11T21:35:17.666498+0000","objects_scrubbed":5,"log_size":25,"log_dups_size":0,"ondisk_log_size":25,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:40:50.059269+0000","scrub_duration":20,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":25,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.3","version":"74'8","reported_seq":833,"reported_epoch":363,"state":"active+clean","last_fresh":"2024-03-11T21:40:48.411693+0000","last_change":"2024-03-11T21:35:31.695369+0000","last_active":"2024-03-11T21:40:48.411693+0000","last_peered":"2024-03-11T21:40:48.411693+0000","last_clean":"2024-03-11T21:40:48.411693+0000","last_became_active":"2024-03-11T21:25:42.578161+0000","last_became_peered":"2024-03-11T21:25:42.578161+0000","last_unstale":"2024-03-11T21:40:48.411693+0000","last_undegraded":"2024-03-11T21:40:48.411693+0000","last_fullsized":"2024-03-11T21:40:48.411693+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'8","last_scrub_stamp":"2024-03-11T21:35:31.695263+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:41.544550+0000","last_clean_scrub_stamp":"2024-03-11T21:35:31.695263+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:40:49.118365+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,5],"acting":[0,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.a","version":"0'0","reported_seq":21,"reported_epoch":362,"state":"active+clean","last_fresh":"2024-03-11T21:40:46.103816+0000","last_change":"2024-03-11T21:40:39.243111+0000","last_active":"2024-03-11T21:40:46.103816+0000","last_peered":"2024-03-11T21:40:46.103816+0000","last_clean":"2024-03-11T21:40:46.103816+0000","last_became_active":"2024-03-11T21:40:39.242809+0000","last_became_peered":"2024-03-11T21:40:39.242809+0000","last_unstale":"2024-03-11T21:40:46.103816+0000","last_undegraded":"2024-03-11T21:40:46.103816+0000","last_fullsized":"2024-03-11T21:40:46.103816+0000","mapping_epoch":356,"log_start":"0'0","ondisk_log_start":"0'0","created":356,"last_epoch_clean":357,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:40:38.219602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:40:38.219602+0000","last_clean_scrub_stamp":"2024-03-11T21:40:38.219602+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:41:52.103793+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":579,"reported_epoch":363,"state":"active+clean","last_fresh":"2024-03-11T21:40:48.882394+0000","last_change":"2024-03-11T21:36:35.068977+0000","last_active":"2024-03-11T21:40:48.882394+0000","last_peered":"2024-03-11T21:40:48.882394+0000","last_clean":"2024-03-11T21:40:48.882394+0000","last_became_active":"2024-03-11T21:36:35.067288+0000","last_became_peered":"2024-03-11T21:36:35.067288+0000","last_unstale":"2024-03-11T21:40:48.882394+0000","last_undegraded":"2024-03-11T21:40:48.882394+0000","last_fullsized":"2024-03-11T21:40:48.882394+0000","mapping_epoch":277,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":278,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:35:09.985377+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:28:30.563712+0000","last_clean_scrub_stamp":"2024-03-11T21:35:09.985377+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:40:52.711294+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,7],"acting":[3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":655,"reported_epoch":362,"state":"active+clean","last_fresh":"2024-03-11T21:40:46.104264+0000","last_change":"2024-03-11T21:35:40.116979+0000","last_active":"2024-03-11T21:40:46.104264+0000","last_peered":"2024-03-11T21:40:46.104264+0000","last_clean":"2024-03-11T21:40:46.104264+0000","last_became_active":"2024-03-11T21:25:38.538540+0000","last_became_peered":"2024-03-11T21:25:38.538540+0000","last_unstale":"2024-03-11T21:40:46.104264+0000","last_undegraded":"2024-03-11T21:40:46.104264+0000","last_fullsized":"2024-03-11T21:40:46.104264+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:35:40.116877+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:37.512018+0000","last_clean_scrub_stamp":"2024-03-11T21:35:40.116877+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.4","version":"280'42","reported_seq":909,"reported_epoch":362,"state":"active+clean","last_fresh":"2024-03-11T21:40:45.412102+0000","last_change":"2024-03-11T21:36:35.095935+0000","last_active":"2024-03-11T21:40:45.412102+0000","last_peered":"2024-03-11T21:40:45.412102+0000","last_clean":"2024-03-11T21:40:45.412102+0000","last_became_active":"2024-03-11T21:36:35.073430+0000","last_became_peered":"2024-03-11T21:36:35.073430+0000","last_unstale":"2024-03-11T21:40:45.412102+0000","last_undegraded":"2024-03-11T21:40:45.412102+0000","last_fullsized":"2024-03-11T21:40:45.412102+0000","mapping_epoch":277,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":278,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'28","last_scrub_stamp":"2024-03-11T21:34:19.513252+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:39.525554+0000","last_clean_scrub_stamp":"2024-03-11T21:34:19.513252+0000","objects_scrubbed":10,"log_size":42,"log_dups_size":0,"ondisk_log_size":42,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:40:50.059234+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":358,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":66,"num_read_kb":51,"num_write":38,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3],"acting":[1,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":642,"reported_epoch":363,"state":"active+clean","last_fresh":"2024-03-11T21:40:48.411427+0000","last_change":"2024-03-11T21:35:32.651731+0000","last_active":"2024-03-11T21:40:48.411427+0000","last_peered":"2024-03-11T21:40:48.411427+0000","last_clean":"2024-03-11T21:40:48.411427+0000","last_became_active":"2024-03-11T21:25:42.572673+0000","last_became_peered":"2024-03-11T21:25:42.572673+0000","last_unstale":"2024-03-11T21:40:48.411427+0000","last_undegraded":"2024-03-11T21:40:48.411427+0000","last_fullsized":"2024-03-11T21:40:48.411427+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:35:32.651652+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:41.544550+0000","last_clean_scrub_stamp":"2024-03-11T21:35:32.651652+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:40:49.118431+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.9","version":"0'0","reported_seq":21,"reported_epoch":362,"state":"active+clean","last_fresh":"2024-03-11T21:40:46.103886+0000","last_change":"2024-03-11T21:40:39.241786+0000","last_active":"2024-03-11T21:40:46.103886+0000","last_peered":"2024-03-11T21:40:46.103886+0000","last_clean":"2024-03-11T21:40:46.103886+0000","last_became_active":"2024-03-11T21:40:39.241438+0000","last_became_peered":"2024-03-11T21:40:39.241438+0000","last_unstale":"2024-03-11T21:40:46.103886+0000","last_undegraded":"2024-03-11T21:40:46.103886+0000","last_fullsized":"2024-03-11T21:40:46.103886+0000","mapping_epoch":356,"log_start":"0'0","ondisk_log_start":"0'0","created":356,"last_epoch_clean":357,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:40:38.219602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:40:38.219602+0000","last_clean_scrub_stamp":"2024-03-11T21:40:38.219602+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:42:00.129587+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.5","version":"0'0","reported_seq":300,"reported_epoch":362,"state":"active+clean","last_fresh":"2024-03-11T21:40:46.061664+0000","last_change":"2024-03-11T21:36:35.069989+0000","last_active":"2024-03-11T21:40:46.061664+0000","last_peered":"2024-03-11T21:40:46.061664+0000","last_clean":"2024-03-11T21:40:46.061664+0000","last_became_active":"2024-03-11T21:36:35.069358+0000","last_became_peered":"2024-03-11T21:36:35.069358+0000","last_unstale":"2024-03-11T21:40:46.061664+0000","last_undegraded":"2024-03-11T21:40:46.061664+0000","last_fullsized":"2024-03-11T21:40:46.061664+0000","mapping_epoch":277,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":278,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:35:19.841632+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:43.559452+0000","last_clean_scrub_stamp":"2024-03-11T21:35:19.841632+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,3],"acting":[7,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.0","version":"164'6","reported_seq":1561,"reported_epoch":362,"state":"active+clean","last_fresh":"2024-03-11T21:40:47.087289+0000","last_change":"2024-03-11T21:35:38.518243+0000","last_active":"2024-03-11T21:40:47.087289+0000","last_peered":"2024-03-11T21:40:47.087289+0000","last_clean":"2024-03-11T21:40:47.087289+0000","last_became_active":"2024-03-11T21:25:38.536890+0000","last_became_peered":"2024-03-11T21:25:38.536890+0000","last_unstale":"2024-03-11T21:40:47.087289+0000","last_undegraded":"2024-03-11T21:40:47.087289+0000","last_fullsized":"2024-03-11T21:40:47.087289+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"164'6","last_scrub_stamp":"2024-03-11T21:35:38.518097+0000","last_deep_scrub":"164'6","last_deep_scrub_stamp":"2024-03-11T21:32:17.597666+0000","last_clean_scrub_stamp":"2024-03-11T21:35:38.518097+0000","objects_scrubbed":1,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:40:50.059480+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0.000753559,"stat_sum":{"num_bytes":389,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":902,"num_read_kb":897,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,6],"acting":[1,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":663,"reported_epoch":362,"state":"active+clean","last_fresh":"2024-03-11T21:40:46.103396+0000","last_change":"2024-03-11T21:40:09.915304+0000","last_active":"2024-03-11T21:40:46.103396+0000","last_peered":"2024-03-11T21:40:46.103396+0000","last_clean":"2024-03-11T21:40:46.103396+0000","last_became_active":"2024-03-11T21:36:35.056966+0000","last_became_peered":"2024-03-11T21:36:35.056966+0000","last_unstale":"2024-03-11T21:40:46.103396+0000","last_undegraded":"2024-03-11T21:40:46.103396+0000","last_fullsized":"2024-03-11T21:40:46.103396+0000","mapping_epoch":277,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":278,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:40:09.915179+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:40:09.915179+0000","last_clean_scrub_stamp":"2024-03-11T21:40:09.915179+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:41:27.096360+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.7","version":"260'22","reported_seq":684,"reported_epoch":362,"state":"active+clean","last_fresh":"2024-03-11T21:40:45.412398+0000","last_change":"2024-03-11T21:35:37.508679+0000","last_active":"2024-03-11T21:40:45.412398+0000","last_peered":"2024-03-11T21:40:45.412398+0000","last_clean":"2024-03-11T21:40:45.412398+0000","last_became_active":"2024-03-11T21:25:40.541769+0000","last_became_peered":"2024-03-11T21:25:40.541769+0000","last_unstale":"2024-03-11T21:40:45.412398+0000","last_undegraded":"2024-03-11T21:40:45.412398+0000","last_fullsized":"2024-03-11T21:40:45.412398+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"97'18","last_scrub_stamp":"2024-03-11T21:35:37.508567+0000","last_deep_scrub":"97'18","last_deep_scrub_stamp":"2024-03-11T21:27:19.239404+0000","last_clean_scrub_stamp":"2024-03-11T21:35:37.508567+0000","objects_scrubbed":5,"log_size":22,"log_dups_size":0,"ondisk_log_size":22,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:40:50.059444+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":36,"num_read_kb":25,"num_write":20,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":649,"reported_epoch":362,"state":"active+clean","last_fresh":"2024-03-11T21:40:46.105144+0000","last_change":"2024-03-11T21:35:22.632822+0000","last_active":"2024-03-11T21:40:46.105144+0000","last_peered":"2024-03-11T21:40:46.105144+0000","last_clean":"2024-03-11T21:40:46.105144+0000","last_became_active":"2024-03-11T21:34:41.143660+0000","last_became_peered":"2024-03-11T21:34:41.143660+0000","last_unstale":"2024-03-11T21:40:46.105144+0000","last_undegraded":"2024-03-11T21:40:46.105144+0000","last_fullsized":"2024-03-11T21:40:46.105144+0000","mapping_epoch":226,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":227,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:35:22.632707+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:41.544550+0000","last_clean_scrub_stamp":"2024-03-11T21:35:22.632707+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":18,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7],"acting":[2,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"10.b","version":"0'0","reported_seq":21,"reported_epoch":362,"state":"active+clean","last_fresh":"2024-03-11T21:40:46.105187+0000","last_change":"2024-03-11T21:40:39.241086+0000","last_active":"2024-03-11T21:40:46.105187+0000","last_peered":"2024-03-11T21:40:46.105187+0000","last_clean":"2024-03-11T21:40:46.105187+0000","last_became_active":"2024-03-11T21:40:39.240775+0000","last_became_peered":"2024-03-11T21:40:39.240775+0000","last_unstale":"2024-03-11T21:40:46.105187+0000","last_undegraded":"2024-03-11T21:40:46.105187+0000","last_fullsized":"2024-03-11T21:40:46.105187+0000","mapping_epoch":356,"log_start":"0'0","ondisk_log_start":"0'0","created":356,"last_epoch_clean":357,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:40:38.219602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:40:38.219602+0000","last_clean_scrub_stamp":"2024-03-11T21:40:38.219602+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:41:46.894434+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,0],"acting":[6,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.7","version":"124'2","reported_seq":645,"reported_epoch":362,"state":"active+clean","last_fresh":"2024-03-11T21:40:46.105517+0000","last_change":"2024-03-11T21:40:31.722243+0000","last_active":"2024-03-11T21:40:46.105517+0000","last_peered":"2024-03-11T21:40:46.105517+0000","last_clean":"2024-03-11T21:40:46.105517+0000","last_became_active":"2024-03-11T21:36:35.058786+0000","last_became_peered":"2024-03-11T21:36:35.058786+0000","last_unstale":"2024-03-11T21:40:46.105517+0000","last_undegraded":"2024-03-11T21:40:46.105517+0000","last_fullsized":"2024-03-11T21:40:46.105517+0000","mapping_epoch":277,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":278,"parent":"0.0","parent_split_bits":0,"last_scrub":"124'2","last_scrub_stamp":"2024-03-11T21:40:31.722100+0000","last_deep_scrub":"124'2","last_deep_scrub_stamp":"2024-03-11T21:29:34.956908+0000","last_clean_scrub_stamp":"2024-03-11T21:40:31.722100+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:41:45.441808+0000","scrub_duration":87,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,4],"acting":[5,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2","version":"71'2","reported_seq":601,"reported_epoch":363,"state":"active+clean","last_fresh":"2024-03-11T21:40:48.881315+0000","last_change":"2024-03-11T21:36:35.071218+0000","last_active":"2024-03-11T21:40:48.881315+0000","last_peered":"2024-03-11T21:40:48.881315+0000","last_clean":"2024-03-11T21:40:48.881315+0000","last_became_active":"2024-03-11T21:36:35.070783+0000","last_became_peered":"2024-03-11T21:36:35.070783+0000","last_unstale":"2024-03-11T21:40:48.881315+0000","last_undegraded":"2024-03-11T21:40:48.881315+0000","last_fullsized":"2024-03-11T21:40:48.881315+0000","mapping_epoch":277,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":278,"parent":"0.0","parent_split_bits":0,"last_scrub":"71'2","last_scrub_stamp":"2024-03-11T21:32:21.543053+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:37.512018+0000","last_clean_scrub_stamp":"2024-03-11T21:32:21.543053+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:40:52.711061+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014206449999999999,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":665,"reported_epoch":362,"state":"active+clean","last_fresh":"2024-03-11T21:40:46.104377+0000","last_change":"2024-03-11T21:35:08.391825+0000","last_active":"2024-03-11T21:40:46.104377+0000","last_peered":"2024-03-11T21:40:46.104377+0000","last_clean":"2024-03-11T21:40:46.104377+0000","last_became_active":"2024-03-11T21:34:41.146672+0000","last_became_peered":"2024-03-11T21:34:41.146672+0000","last_unstale":"2024-03-11T21:40:46.104377+0000","last_undegraded":"2024-03-11T21:40:46.104377+0000","last_fullsized":"2024-03-11T21:40:46.104377+0000","mapping_epoch":226,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":227,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:35:08.391713+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:37.512018+0000","last_clean_scrub_stamp":"2024-03-11T21:35:08.391713+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"87'128","reported_seq":763,"reported_epoch":362,"state":"active+clean","last_fresh":"2024-03-11T21:40:46.060989+0000","last_change":"2024-03-11T21:35:21.798272+0000","last_active":"2024-03-11T21:40:46.060989+0000","last_peered":"2024-03-11T21:40:46.060989+0000","last_clean":"2024-03-11T21:40:46.060989+0000","last_became_active":"2024-03-11T21:34:44.164569+0000","last_became_peered":"2024-03-11T21:34:44.164569+0000","last_unstale":"2024-03-11T21:40:46.060989+0000","last_undegraded":"2024-03-11T21:40:46.060989+0000","last_fullsized":"2024-03-11T21:40:46.060989+0000","mapping_epoch":229,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":230,"parent":"0.0","parent_split_bits":0,"last_scrub":"87'128","last_scrub_stamp":"2024-03-11T21:35:21.798144+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:24.261974+0000","last_clean_scrub_stamp":"2024-03-11T21:35:21.798144+0000","objects_scrubbed":2,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:40:51.069046+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":198,"num_read_kb":294,"num_write":179,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":2951840,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.5","version":"260'28","reported_seq":882,"reported_epoch":362,"state":"active+clean","last_fresh":"2024-03-11T21:40:46.104288+0000","last_change":"2024-03-11T21:39:45.216634+0000","last_active":"2024-03-11T21:40:46.104288+0000","last_peered":"2024-03-11T21:40:46.104288+0000","last_clean":"2024-03-11T21:40:46.104288+0000","last_became_active":"2024-03-11T21:25:40.761457+0000","last_became_peered":"2024-03-11T21:25:40.761457+0000","last_unstale":"2024-03-11T21:40:46.104288+0000","last_undegraded":"2024-03-11T21:40:46.104288+0000","last_fullsized":"2024-03-11T21:40:46.104288+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"260'28","last_scrub_stamp":"2024-03-11T21:39:45.216468+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:39.525554+0000","last_clean_scrub_stamp":"2024-03-11T21:39:45.216468+0000","objects_scrubbed":8,"log_size":28,"log_dups_size":0,"ondisk_log_size":28,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:40:57.658763+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":154,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":43,"num_read_kb":33,"num_write":25,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":291,"reported_epoch":362,"state":"active+clean","last_fresh":"2024-03-11T21:40:46.061710+0000","last_change":"2024-03-11T21:35:24.841170+0000","last_active":"2024-03-11T21:40:46.061710+0000","last_peered":"2024-03-11T21:40:46.061710+0000","last_clean":"2024-03-11T21:40:46.061710+0000","last_became_active":"2024-03-11T21:34:41.139782+0000","last_became_peered":"2024-03-11T21:34:41.139782+0000","last_unstale":"2024-03-11T21:40:46.061710+0000","last_undegraded":"2024-03-11T21:40:46.061710+0000","last_fullsized":"2024-03-11T21:40:46.061710+0000","mapping_epoch":226,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":227,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:35:24.841041+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:32:21.266672+0000","last_clean_scrub_stamp":"2024-03-11T21:35:24.841041+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,5],"acting":[7,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.8","version":"0'0","reported_seq":23,"reported_epoch":363,"state":"active+clean","last_fresh":"2024-03-11T21:40:48.411197+0000","last_change":"2024-03-11T21:40:39.237338+0000","last_active":"2024-03-11T21:40:48.411197+0000","last_peered":"2024-03-11T21:40:48.411197+0000","last_clean":"2024-03-11T21:40:48.411197+0000","last_became_active":"2024-03-11T21:40:39.237027+0000","last_became_peered":"2024-03-11T21:40:39.237027+0000","last_unstale":"2024-03-11T21:40:48.411197+0000","last_undegraded":"2024-03-11T21:40:48.411197+0000","last_fullsized":"2024-03-11T21:40:48.411197+0000","mapping_epoch":356,"log_start":"0'0","ondisk_log_start":"0'0","created":356,"last_epoch_clean":357,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:40:38.219602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:40:38.219602+0000","last_clean_scrub_stamp":"2024-03-11T21:40:38.219602+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:42:04.131130+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,7],"acting":[0,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.4","version":"0'0","reported_seq":637,"reported_epoch":362,"state":"active+clean","last_fresh":"2024-03-11T21:40:45.412387+0000","last_change":"2024-03-11T21:36:35.073305+0000","last_active":"2024-03-11T21:40:45.412387+0000","last_peered":"2024-03-11T21:40:45.412387+0000","last_clean":"2024-03-11T21:40:45.412387+0000","last_became_active":"2024-03-11T21:36:35.072824+0000","last_became_peered":"2024-03-11T21:36:35.072824+0000","last_unstale":"2024-03-11T21:40:45.412387+0000","last_undegraded":"2024-03-11T21:40:45.412387+0000","last_fullsized":"2024-03-11T21:40:45.412387+0000","mapping_epoch":277,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":278,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:34:23.531064+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:43.559452+0000","last_clean_scrub_stamp":"2024-03-11T21:34:23.531064+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:40:50.059142+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":666,"reported_epoch":363,"state":"active+clean","last_fresh":"2024-03-11T21:40:48.410800+0000","last_change":"2024-03-11T21:36:35.070626+0000","last_active":"2024-03-11T21:40:48.410800+0000","last_peered":"2024-03-11T21:40:48.410800+0000","last_clean":"2024-03-11T21:40:48.410800+0000","last_became_active":"2024-03-11T21:36:35.069452+0000","last_became_peered":"2024-03-11T21:36:35.069452+0000","last_unstale":"2024-03-11T21:40:48.410800+0000","last_undegraded":"2024-03-11T21:40:48.410800+0000","last_fullsized":"2024-03-11T21:40:48.410800+0000","mapping_epoch":277,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":278,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:35:13.741947+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:37.512018+0000","last_clean_scrub_stamp":"2024-03-11T21:35:13.741947+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:40:49.118269+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0.00089342499999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":292,"reported_epoch":362,"state":"active+clean","last_fresh":"2024-03-11T21:40:46.060846+0000","last_change":"2024-03-11T21:40:27.117606+0000","last_active":"2024-03-11T21:40:46.060846+0000","last_peered":"2024-03-11T21:40:46.060846+0000","last_clean":"2024-03-11T21:40:46.060846+0000","last_became_active":"2024-03-11T21:34:41.149219+0000","last_became_peered":"2024-03-11T21:34:41.149219+0000","last_unstale":"2024-03-11T21:40:46.060846+0000","last_undegraded":"2024-03-11T21:40:46.060846+0000","last_fullsized":"2024-03-11T21:40:46.060846+0000","mapping_epoch":226,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":227,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:40:27.117515+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:37.512018+0000","last_clean_scrub_stamp":"2024-03-11T21:40:27.117515+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:41:35.677145+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,0],"acting":[7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.6","version":"352'30","reported_seq":715,"reported_epoch":363,"state":"active+clean","last_fresh":"2024-03-11T21:40:48.411297+0000","last_change":"2024-03-11T21:40:31.175417+0000","last_active":"2024-03-11T21:40:48.411297+0000","last_peered":"2024-03-11T21:40:48.411297+0000","last_clean":"2024-03-11T21:40:48.411297+0000","last_became_active":"2024-03-11T21:32:22.796486+0000","last_became_peered":"2024-03-11T21:32:22.796486+0000","last_unstale":"2024-03-11T21:40:48.411297+0000","last_undegraded":"2024-03-11T21:40:48.411297+0000","last_fullsized":"2024-03-11T21:40:48.411297+0000","mapping_epoch":166,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":167,"parent":"0.0","parent_split_bits":0,"last_scrub":"352'30","last_scrub_stamp":"2024-03-11T21:40:31.175323+0000","last_deep_scrub":"352'30","last_deep_scrub_stamp":"2024-03-11T21:40:31.175323+0000","last_clean_scrub_stamp":"2024-03-11T21:40:31.175323+0000","objects_scrubbed":6,"log_size":30,"log_dups_size":0,"ondisk_log_size":30,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:41:40.436187+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":36,"num_read_kb":30,"num_write":24,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":638,"reported_epoch":362,"state":"active+clean","last_fresh":"2024-03-11T21:40:46.105534+0000","last_change":"2024-03-11T21:36:07.451482+0000","last_active":"2024-03-11T21:40:46.105534+0000","last_peered":"2024-03-11T21:40:46.105534+0000","last_clean":"2024-03-11T21:40:46.105534+0000","last_became_active":"2024-03-11T21:25:42.572324+0000","last_became_peered":"2024-03-11T21:25:42.572324+0000","last_unstale":"2024-03-11T21:40:46.105534+0000","last_undegraded":"2024-03-11T21:40:46.105534+0000","last_fullsized":"2024-03-11T21:40:46.105534+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:36:07.451102+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:41.544550+0000","last_clean_scrub_stamp":"2024-03-11T21:36:07.451102+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":54,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.d","version":"0'0","reported_seq":21,"reported_epoch":362,"state":"active+clean","last_fresh":"2024-03-11T21:40:46.104052+0000","last_change":"2024-03-11T21:40:39.242068+0000","last_active":"2024-03-11T21:40:46.104052+0000","last_peered":"2024-03-11T21:40:46.104052+0000","last_clean":"2024-03-11T21:40:46.104052+0000","last_became_active":"2024-03-11T21:40:39.241769+0000","last_became_peered":"2024-03-11T21:40:39.241769+0000","last_unstale":"2024-03-11T21:40:46.104052+0000","last_undegraded":"2024-03-11T21:40:46.104052+0000","last_fullsized":"2024-03-11T21:40:46.104052+0000","mapping_epoch":356,"log_start":"0'0","ondisk_log_start":"0'0","created":356,"last_epoch_clean":357,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:40:38.219602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:40:38.219602+0000","last_clean_scrub_stamp":"2024-03-11T21:40:38.219602+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:41:55.963946+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,0],"acting":[6,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":644,"reported_epoch":362,"state":"active+clean","last_fresh":"2024-03-11T21:40:45.411738+0000","last_change":"2024-03-11T21:36:18.529506+0000","last_active":"2024-03-11T21:40:45.411738+0000","last_peered":"2024-03-11T21:40:45.411738+0000","last_clean":"2024-03-11T21:40:45.411738+0000","last_became_active":"2024-03-11T21:25:45.301416+0000","last_became_peered":"2024-03-11T21:25:45.301416+0000","last_unstale":"2024-03-11T21:40:45.411738+0000","last_undegraded":"2024-03-11T21:40:45.411738+0000","last_fullsized":"2024-03-11T21:40:45.411738+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:36:18.529370+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:43.559452+0000","last_clean_scrub_stamp":"2024-03-11T21:36:18.529370+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:40:50.059594+0000","scrub_duration":54,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":291,"reported_epoch":362,"state":"active+clean","last_fresh":"2024-03-11T21:40:46.061675+0000","last_change":"2024-03-11T21:35:33.740615+0000","last_active":"2024-03-11T21:40:46.061675+0000","last_peered":"2024-03-11T21:40:46.061675+0000","last_clean":"2024-03-11T21:40:46.061675+0000","last_became_active":"2024-03-11T21:34:41.152721+0000","last_became_peered":"2024-03-11T21:34:41.152721+0000","last_unstale":"2024-03-11T21:40:46.061675+0000","last_undegraded":"2024-03-11T21:40:46.061675+0000","last_fullsized":"2024-03-11T21:40:46.061675+0000","mapping_epoch":226,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":227,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:35:33.740538+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:26:57.333669+0000","last_clean_scrub_stamp":"2024-03-11T21:35:33.740538+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.3","version":"260'37","reported_seq":753,"reported_epoch":363,"state":"active+clean","last_fresh":"2024-03-11T21:40:48.411324+0000","last_change":"2024-03-11T21:35:11.697314+0000","last_active":"2024-03-11T21:40:48.411324+0000","last_peered":"2024-03-11T21:40:48.411324+0000","last_clean":"2024-03-11T21:40:48.411324+0000","last_became_active":"2024-03-11T21:34:41.145607+0000","last_became_peered":"2024-03-11T21:34:41.145607+0000","last_unstale":"2024-03-11T21:40:48.411324+0000","last_undegraded":"2024-03-11T21:40:48.411324+0000","last_fullsized":"2024-03-11T21:40:48.411324+0000","mapping_epoch":226,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":227,"parent":"0.0","parent_split_bits":0,"last_scrub":"165'27","last_scrub_stamp":"2024-03-11T21:35:11.697140+0000","last_deep_scrub":"165'27","last_deep_scrub_stamp":"2024-03-11T21:32:21.576732+0000","last_clean_scrub_stamp":"2024-03-11T21:35:11.697140+0000","objects_scrubbed":8,"log_size":37,"log_dups_size":0,"ondisk_log_size":37,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:40:49.118291+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":54,"num_read_kb":40,"num_write":32,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":660,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":664,"reported_epoch":362,"state":"active+clean","last_fresh":"2024-03-11T21:40:46.104286+0000","last_change":"2024-03-11T21:40:15.172829+0000","last_active":"2024-03-11T21:40:46.104286+0000","last_peered":"2024-03-11T21:40:46.104286+0000","last_clean":"2024-03-11T21:40:46.104286+0000","last_became_active":"2024-03-11T21:25:42.579045+0000","last_became_peered":"2024-03-11T21:25:42.579045+0000","last_unstale":"2024-03-11T21:40:46.104286+0000","last_undegraded":"2024-03-11T21:40:46.104286+0000","last_fullsized":"2024-03-11T21:40:46.104286+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:40:15.172720+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:40:15.172720+0000","last_clean_scrub_stamp":"2024-03-11T21:40:15.172720+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:41:19.672684+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"10.e","version":"0'0","reported_seq":21,"reported_epoch":362,"state":"active+clean","last_fresh":"2024-03-11T21:40:45.411337+0000","last_change":"2024-03-11T21:40:39.239887+0000","last_active":"2024-03-11T21:40:45.411337+0000","last_peered":"2024-03-11T21:40:45.411337+0000","last_clean":"2024-03-11T21:40:45.411337+0000","last_became_active":"2024-03-11T21:40:39.239590+0000","last_became_peered":"2024-03-11T21:40:39.239590+0000","last_unstale":"2024-03-11T21:40:45.411337+0000","last_undegraded":"2024-03-11T21:40:45.411337+0000","last_fullsized":"2024-03-11T21:40:45.411337+0000","mapping_epoch":356,"log_start":"0'0","ondisk_log_start":"0'0","created":356,"last_epoch_clean":357,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:40:38.219602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:40:38.219602+0000","last_clean_scrub_stamp":"2024-03-11T21:40:38.219602+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:41:42.443121+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,2],"acting":[4,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":650,"reported_epoch":362,"state":"active+clean","last_fresh":"2024-03-11T21:40:45.411432+0000","last_change":"2024-03-11T21:40:28.582452+0000","last_active":"2024-03-11T21:40:45.411432+0000","last_peered":"2024-03-11T21:40:45.411432+0000","last_clean":"2024-03-11T21:40:45.411432+0000","last_became_active":"2024-03-11T21:32:22.791439+0000","last_became_peered":"2024-03-11T21:32:22.791439+0000","last_unstale":"2024-03-11T21:40:45.411432+0000","last_undegraded":"2024-03-11T21:40:45.411432+0000","last_fullsized":"2024-03-11T21:40:45.411432+0000","mapping_epoch":166,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":167,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:40:28.582339+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:43.559452+0000","last_clean_scrub_stamp":"2024-03-11T21:40:28.582339+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:41:49.291632+0000","scrub_duration":20,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,2],"acting":[4,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.6","version":"67'1","reported_seq":649,"reported_epoch":362,"state":"active+clean","last_fresh":"2024-03-11T21:40:45.412394+0000","last_change":"2024-03-11T21:35:30.581783+0000","last_active":"2024-03-11T21:40:45.412394+0000","last_peered":"2024-03-11T21:40:45.412394+0000","last_clean":"2024-03-11T21:40:45.412394+0000","last_became_active":"2024-03-11T21:25:38.536142+0000","last_became_peered":"2024-03-11T21:25:38.536142+0000","last_unstale":"2024-03-11T21:40:45.412394+0000","last_undegraded":"2024-03-11T21:40:45.412394+0000","last_fullsized":"2024-03-11T21:40:45.412394+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'1","last_scrub_stamp":"2024-03-11T21:35:30.581701+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:37.512018+0000","last_clean_scrub_stamp":"2024-03-11T21:35:30.581701+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:40:50.059379+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,4],"acting":[1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.0","version":"260'38","reported_seq":649,"reported_epoch":363,"state":"active+clean","last_fresh":"2024-03-11T21:40:48.881188+0000","last_change":"2024-03-11T21:36:35.087771+0000","last_active":"2024-03-11T21:40:48.881188+0000","last_peered":"2024-03-11T21:40:48.881188+0000","last_clean":"2024-03-11T21:40:48.881188+0000","last_became_active":"2024-03-11T21:36:35.068071+0000","last_became_peered":"2024-03-11T21:36:35.068071+0000","last_unstale":"2024-03-11T21:40:48.881188+0000","last_undegraded":"2024-03-11T21:40:48.881188+0000","last_fullsized":"2024-03-11T21:40:48.881188+0000","mapping_epoch":277,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":278,"parent":"0.0","parent_split_bits":0,"last_scrub":"93'26","last_scrub_stamp":"2024-03-11T21:34:16.863984+0000","last_deep_scrub":"93'26","last_deep_scrub_stamp":"2024-03-11T21:26:40.680324+0000","last_clean_scrub_stamp":"2024-03-11T21:34:16.863984+0000","objects_scrubbed":8,"log_size":38,"log_dups_size":0,"ondisk_log_size":38,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:40:52.711143+0000","scrub_duration":22,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":51,"num_read_kb":39,"num_write":32,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":669,"reported_epoch":362,"state":"active+clean","last_fresh":"2024-03-11T21:40:45.411047+0000","last_change":"2024-03-11T21:40:21.518921+0000","last_active":"2024-03-11T21:40:45.411047+0000","last_peered":"2024-03-11T21:40:45.411047+0000","last_clean":"2024-03-11T21:40:45.411047+0000","last_became_active":"2024-03-11T21:36:35.057517+0000","last_became_peered":"2024-03-11T21:36:35.057517+0000","last_unstale":"2024-03-11T21:40:45.411047+0000","last_undegraded":"2024-03-11T21:40:45.411047+0000","last_fullsized":"2024-03-11T21:40:45.411047+0000","mapping_epoch":277,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":278,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:40:21.518840+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:40:21.518840+0000","last_clean_scrub_stamp":"2024-03-11T21:40:21.518840+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:41:26.549696+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,7],"acting":[4,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"10.f","version":"0'0","reported_seq":21,"reported_epoch":362,"state":"active+clean","last_fresh":"2024-03-11T21:40:46.104893+0000","last_change":"2024-03-11T21:40:39.541796+0000","last_active":"2024-03-11T21:40:46.104893+0000","last_peered":"2024-03-11T21:40:46.104893+0000","last_clean":"2024-03-11T21:40:46.104893+0000","last_became_active":"2024-03-11T21:40:39.541420+0000","last_became_peered":"2024-03-11T21:40:39.541420+0000","last_unstale":"2024-03-11T21:40:46.104893+0000","last_undegraded":"2024-03-11T21:40:46.104893+0000","last_fullsized":"2024-03-11T21:40:46.104893+0000","mapping_epoch":356,"log_start":"0'0","ondisk_log_start":"0'0","created":356,"last_epoch_clean":357,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:40:38.219602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:40:38.219602+0000","last_clean_scrub_stamp":"2024-03-11T21:40:38.219602+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:42:07.694933+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,2],"acting":[6,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.3","version":"0'0","reported_seq":291,"reported_epoch":362,"state":"active+clean","last_fresh":"2024-03-11T21:40:46.061561+0000","last_change":"2024-03-11T21:40:25.234980+0000","last_active":"2024-03-11T21:40:46.061561+0000","last_peered":"2024-03-11T21:40:46.061561+0000","last_clean":"2024-03-11T21:40:46.061561+0000","last_became_active":"2024-03-11T21:34:41.142683+0000","last_became_peered":"2024-03-11T21:34:41.142683+0000","last_unstale":"2024-03-11T21:40:46.061561+0000","last_undegraded":"2024-03-11T21:40:46.061561+0000","last_fullsized":"2024-03-11T21:40:46.061561+0000","mapping_epoch":226,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":227,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:40:25.213541+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:29:42.485740+0000","last_clean_scrub_stamp":"2024-03-11T21:40:25.213541+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:41:30.275517+0000","scrub_duration":55,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,2],"acting":[7,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":683,"reported_epoch":362,"state":"active+clean","last_fresh":"2024-03-11T21:40:46.104939+0000","last_change":"2024-03-11T21:40:07.187684+0000","last_active":"2024-03-11T21:40:46.104939+0000","last_peered":"2024-03-11T21:40:46.104939+0000","last_clean":"2024-03-11T21:40:46.104939+0000","last_became_active":"2024-03-11T21:34:41.145628+0000","last_became_peered":"2024-03-11T21:34:41.145628+0000","last_unstale":"2024-03-11T21:40:46.104939+0000","last_undegraded":"2024-03-11T21:40:46.104939+0000","last_fullsized":"2024-03-11T21:40:46.104939+0000","mapping_epoch":226,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":227,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:40:07.187595+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:37.512018+0000","last_clean_scrub_stamp":"2024-03-11T21:40:07.187595+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:41:20.424112+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,2],"acting":[6,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.1","version":"260'28","reported_seq":686,"reported_epoch":362,"state":"active+clean","last_fresh":"2024-03-11T21:40:45.410656+0000","last_change":"2024-03-11T21:35:33.644654+0000","last_active":"2024-03-11T21:40:45.410656+0000","last_peered":"2024-03-11T21:40:45.410656+0000","last_clean":"2024-03-11T21:40:45.410656+0000","last_became_active":"2024-03-11T21:25:40.546219+0000","last_became_peered":"2024-03-11T21:25:40.546219+0000","last_unstale":"2024-03-11T21:40:45.410656+0000","last_undegraded":"2024-03-11T21:40:45.410656+0000","last_fullsized":"2024-03-11T21:40:45.410656+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'14","last_scrub_stamp":"2024-03-11T21:35:33.644503+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:39.525554+0000","last_clean_scrub_stamp":"2024-03-11T21:35:33.644503+0000","objects_scrubbed":7,"log_size":28,"log_dups_size":0,"ondisk_log_size":28,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":42,"num_read_kb":35,"num_write":28,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6],"acting":[4,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.0","version":"280'11","reported_seq":759,"reported_epoch":363,"state":"active+clean","last_fresh":"2024-03-11T21:40:48.880937+0000","last_change":"2024-03-11T21:36:35.064085+0000","last_active":"2024-03-11T21:40:48.880937+0000","last_peered":"2024-03-11T21:40:48.880937+0000","last_clean":"2024-03-11T21:40:48.880937+0000","last_became_active":"2024-03-11T21:36:35.055250+0000","last_became_peered":"2024-03-11T21:36:35.055250+0000","last_unstale":"2024-03-11T21:40:48.880937+0000","last_undegraded":"2024-03-11T21:40:48.880937+0000","last_fullsized":"2024-03-11T21:40:48.880937+0000","mapping_epoch":277,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":278,"parent":"0.0","parent_split_bits":0,"last_scrub":"98'9","last_scrub_stamp":"2024-03-11T21:35:21.143578+0000","last_deep_scrub":"98'9","last_deep_scrub_stamp":"2024-03-11T21:27:29.745711+0000","last_clean_scrub_stamp":"2024-03-11T21:35:21.143578+0000","objects_scrubbed":1,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:40:52.711229+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.0","version":"0'0","reported_seq":21,"reported_epoch":362,"state":"active+clean","last_fresh":"2024-03-11T21:40:46.103548+0000","last_change":"2024-03-11T21:40:39.242132+0000","last_active":"2024-03-11T21:40:46.103548+0000","last_peered":"2024-03-11T21:40:46.103548+0000","last_clean":"2024-03-11T21:40:46.103548+0000","last_became_active":"2024-03-11T21:40:39.241802+0000","last_became_peered":"2024-03-11T21:40:39.241802+0000","last_unstale":"2024-03-11T21:40:46.103548+0000","last_undegraded":"2024-03-11T21:40:46.103548+0000","last_fullsized":"2024-03-11T21:40:46.103548+0000","mapping_epoch":356,"log_start":"0'0","ondisk_log_start":"0'0","created":356,"last_epoch_clean":357,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:40:38.219602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:40:38.219602+0000","last_clean_scrub_stamp":"2024-03-11T21:40:38.219602+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:41:52.888642+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,4],"acting":[6,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.c","version":"0'0","reported_seq":577,"reported_epoch":363,"state":"active+clean","last_fresh":"2024-03-11T21:40:48.882023+0000","last_change":"2024-03-11T21:40:30.659272+0000","last_active":"2024-03-11T21:40:48.882023+0000","last_peered":"2024-03-11T21:40:48.882023+0000","last_clean":"2024-03-11T21:40:48.882023+0000","last_became_active":"2024-03-11T21:36:35.073778+0000","last_became_peered":"2024-03-11T21:36:35.073778+0000","last_unstale":"2024-03-11T21:40:48.882023+0000","last_undegraded":"2024-03-11T21:40:48.882023+0000","last_fullsized":"2024-03-11T21:40:48.882023+0000","mapping_epoch":277,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":278,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:40:30.659160+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:43.559452+0000","last_clean_scrub_stamp":"2024-03-11T21:40:30.659160+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:41:38.577899+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.f","version":"0'0","reported_seq":646,"reported_epoch":362,"state":"active+clean","last_fresh":"2024-03-11T21:40:46.105417+0000","last_change":"2024-03-11T21:36:08.439013+0000","last_active":"2024-03-11T21:40:46.105417+0000","last_peered":"2024-03-11T21:40:46.105417+0000","last_clean":"2024-03-11T21:40:46.105417+0000","last_became_active":"2024-03-11T21:25:42.575935+0000","last_became_peered":"2024-03-11T21:25:42.575935+0000","last_unstale":"2024-03-11T21:40:46.105417+0000","last_undegraded":"2024-03-11T21:40:46.105417+0000","last_fullsized":"2024-03-11T21:40:46.105417+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:36:08.438895+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:41.544550+0000","last_clean_scrub_stamp":"2024-03-11T21:36:08.438895+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.8","version":"0'0","reported_seq":291,"reported_epoch":362,"state":"active+clean","last_fresh":"2024-03-11T21:40:46.061221+0000","last_change":"2024-03-11T21:35:34.733595+0000","last_active":"2024-03-11T21:40:46.061221+0000","last_peered":"2024-03-11T21:40:46.061221+0000","last_clean":"2024-03-11T21:40:46.061221+0000","last_became_active":"2024-03-11T21:34:41.144681+0000","last_became_peered":"2024-03-11T21:34:41.144681+0000","last_unstale":"2024-03-11T21:40:46.061221+0000","last_undegraded":"2024-03-11T21:40:46.061221+0000","last_fullsized":"2024-03-11T21:40:46.061221+0000","mapping_epoch":226,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":227,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:35:34.733491+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:37.512018+0000","last_clean_scrub_stamp":"2024-03-11T21:35:34.733491+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,1],"acting":[7,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.e","version":"260'24","reported_seq":679,"reported_epoch":362,"state":"active+clean","last_fresh":"2024-03-11T21:40:45.411091+0000","last_change":"2024-03-11T21:35:34.679906+0000","last_active":"2024-03-11T21:40:45.411091+0000","last_peered":"2024-03-11T21:40:45.411091+0000","last_clean":"2024-03-11T21:40:45.411091+0000","last_became_active":"2024-03-11T21:25:40.552178+0000","last_became_peered":"2024-03-11T21:25:40.552178+0000","last_unstale":"2024-03-11T21:40:45.411091+0000","last_undegraded":"2024-03-11T21:40:45.411091+0000","last_fullsized":"2024-03-11T21:40:45.411091+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"109'16","last_scrub_stamp":"2024-03-11T21:35:34.679817+0000","last_deep_scrub":"109'16","last_deep_scrub_stamp":"2024-03-11T21:28:24.739395+0000","last_clean_scrub_stamp":"2024-03-11T21:35:34.679817+0000","objects_scrubbed":5,"log_size":24,"log_dups_size":0,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":32,"num_read_kb":25,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"10.1","version":"0'0","reported_seq":21,"reported_epoch":362,"state":"active+clean","last_fresh":"2024-03-11T21:40:45.410527+0000","last_change":"2024-03-11T21:40:39.236538+0000","last_active":"2024-03-11T21:40:45.410527+0000","last_peered":"2024-03-11T21:40:45.410527+0000","last_clean":"2024-03-11T21:40:45.410527+0000","last_became_active":"2024-03-11T21:40:39.236129+0000","last_became_peered":"2024-03-11T21:40:39.236129+0000","last_unstale":"2024-03-11T21:40:45.410527+0000","last_undegraded":"2024-03-11T21:40:45.410527+0000","last_fullsized":"2024-03-11T21:40:45.410527+0000","mapping_epoch":356,"log_start":"0'0","ondisk_log_start":"0'0","created":356,"last_epoch_clean":357,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:40:38.219602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:40:38.219602+0000","last_clean_scrub_stamp":"2024-03-11T21:40:38.219602+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:42:05.193065+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,0],"acting":[4,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.d","version":"0'0","reported_seq":634,"reported_epoch":362,"state":"active+clean","last_fresh":"2024-03-11T21:40:46.104956+0000","last_change":"2024-03-11T21:35:37.185210+0000","last_active":"2024-03-11T21:40:46.104956+0000","last_peered":"2024-03-11T21:40:46.104956+0000","last_clean":"2024-03-11T21:40:46.104956+0000","last_became_active":"2024-03-11T21:25:44.590333+0000","last_became_peered":"2024-03-11T21:25:44.590333+0000","last_unstale":"2024-03-11T21:40:46.104956+0000","last_undegraded":"2024-03-11T21:40:46.104956+0000","last_fullsized":"2024-03-11T21:40:46.104956+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:35:37.182850+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:35:37.182850+0000","last_clean_scrub_stamp":"2024-03-11T21:35:37.182850+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":40,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.e","version":"74'8","reported_seq":829,"reported_epoch":362,"state":"active+clean","last_fresh":"2024-03-11T21:40:45.411564+0000","last_change":"2024-03-11T21:36:07.734369+0000","last_active":"2024-03-11T21:40:45.411564+0000","last_peered":"2024-03-11T21:40:45.411564+0000","last_clean":"2024-03-11T21:40:45.411564+0000","last_became_active":"2024-03-11T21:25:42.576742+0000","last_became_peered":"2024-03-11T21:25:42.576742+0000","last_unstale":"2024-03-11T21:40:45.411564+0000","last_undegraded":"2024-03-11T21:40:45.411564+0000","last_fullsized":"2024-03-11T21:40:45.411564+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'8","last_scrub_stamp":"2024-03-11T21:36:07.734115+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:41.544550+0000","last_clean_scrub_stamp":"2024-03-11T21:36:07.734115+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,0],"acting":[4,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.9","version":"0'0","reported_seq":669,"reported_epoch":362,"state":"active+clean","last_fresh":"2024-03-11T21:40:45.410918+0000","last_change":"2024-03-11T21:40:29.915912+0000","last_active":"2024-03-11T21:40:45.410918+0000","last_peered":"2024-03-11T21:40:45.410918+0000","last_clean":"2024-03-11T21:40:45.410918+0000","last_became_active":"2024-03-11T21:36:35.065733+0000","last_became_peered":"2024-03-11T21:36:35.065733+0000","last_unstale":"2024-03-11T21:40:45.410918+0000","last_undegraded":"2024-03-11T21:40:45.410918+0000","last_fullsized":"2024-03-11T21:40:45.410918+0000","mapping_epoch":277,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":278,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:40:29.915835+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:26:50.991712+0000","last_clean_scrub_stamp":"2024-03-11T21:40:29.915835+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:41:46.722143+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,3],"acting":[1,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.f","version":"260'27","reported_seq":669,"reported_epoch":362,"state":"active+clean","last_fresh":"2024-03-11T21:40:46.104099+0000","last_change":"2024-03-11T21:40:12.838694+0000","last_active":"2024-03-11T21:40:46.104099+0000","last_peered":"2024-03-11T21:40:46.104099+0000","last_clean":"2024-03-11T21:40:46.104099+0000","last_became_active":"2024-03-11T21:36:35.058272+0000","last_became_peered":"2024-03-11T21:36:35.058272+0000","last_unstale":"2024-03-11T21:40:46.104099+0000","last_undegraded":"2024-03-11T21:40:46.104099+0000","last_fullsized":"2024-03-11T21:40:46.104099+0000","mapping_epoch":277,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":278,"parent":"0.0","parent_split_bits":0,"last_scrub":"260'27","last_scrub_stamp":"2024-03-11T21:40:12.838597+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:39.525554+0000","last_clean_scrub_stamp":"2024-03-11T21:40:12.838597+0000","objects_scrubbed":7,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:41:21.967430+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":44,"num_read_kb":35,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,4],"acting":[2,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"10.2","version":"0'0","reported_seq":21,"reported_epoch":362,"state":"active+clean","last_fresh":"2024-03-11T21:40:45.410568+0000","last_change":"2024-03-11T21:40:39.233862+0000","last_active":"2024-03-11T21:40:45.410568+0000","last_peered":"2024-03-11T21:40:45.410568+0000","last_clean":"2024-03-11T21:40:45.410568+0000","last_became_active":"2024-03-11T21:40:39.233499+0000","last_became_peered":"2024-03-11T21:40:39.233499+0000","last_unstale":"2024-03-11T21:40:45.410568+0000","last_undegraded":"2024-03-11T21:40:45.410568+0000","last_fullsized":"2024-03-11T21:40:45.410568+0000","mapping_epoch":356,"log_start":"0'0","ondisk_log_start":"0'0","created":356,"last_epoch_clean":357,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:40:38.219602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:40:38.219602+0000","last_clean_scrub_stamp":"2024-03-11T21:40:38.219602+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:42:02.053869+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.e","version":"0'0","reported_seq":635,"reported_epoch":362,"state":"active+clean","last_fresh":"2024-03-11T21:40:45.411575+0000","last_change":"2024-03-11T21:36:10.769557+0000","last_active":"2024-03-11T21:40:45.411575+0000","last_peered":"2024-03-11T21:40:45.411575+0000","last_clean":"2024-03-11T21:40:45.411575+0000","last_became_active":"2024-03-11T21:25:44.590198+0000","last_became_peered":"2024-03-11T21:25:44.590198+0000","last_unstale":"2024-03-11T21:40:45.411575+0000","last_undegraded":"2024-03-11T21:40:45.411575+0000","last_fullsized":"2024-03-11T21:40:45.411575+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:36:10.769460+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:43.559452+0000","last_clean_scrub_stamp":"2024-03-11T21:36:10.769460+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":75,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.d","version":"109'9","reported_seq":828,"reported_epoch":362,"state":"active+clean","last_fresh":"2024-03-11T21:40:46.104400+0000","last_change":"2024-03-11T21:35:07.562523+0000","last_active":"2024-03-11T21:40:46.104400+0000","last_peered":"2024-03-11T21:40:46.104400+0000","last_clean":"2024-03-11T21:40:46.104400+0000","last_became_active":"2024-03-11T21:25:42.575583+0000","last_became_peered":"2024-03-11T21:25:42.575583+0000","last_unstale":"2024-03-11T21:40:46.104400+0000","last_undegraded":"2024-03-11T21:40:46.104400+0000","last_fullsized":"2024-03-11T21:40:46.104400+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"109'9","last_scrub_stamp":"2024-03-11T21:35:07.562385+0000","last_deep_scrub":"109'9","last_deep_scrub_stamp":"2024-03-11T21:28:27.082975+0000","last_clean_scrub_stamp":"2024-03-11T21:35:07.562385+0000","objects_scrubbed":1,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,5],"acting":[2,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.a","version":"0'0","reported_seq":668,"reported_epoch":362,"state":"active+clean","last_fresh":"2024-03-11T21:40:45.410664+0000","last_change":"2024-03-11T21:36:35.067262+0000","last_active":"2024-03-11T21:40:45.410664+0000","last_peered":"2024-03-11T21:40:45.410664+0000","last_clean":"2024-03-11T21:40:45.410664+0000","last_became_active":"2024-03-11T21:36:35.066832+0000","last_became_peered":"2024-03-11T21:36:35.066832+0000","last_unstale":"2024-03-11T21:40:45.410664+0000","last_undegraded":"2024-03-11T21:40:45.410664+0000","last_fullsized":"2024-03-11T21:40:45.410664+0000","mapping_epoch":277,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":278,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:34:22.526916+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:37.512018+0000","last_clean_scrub_stamp":"2024-03-11T21:34:22.526916+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:40:50.059174+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,7],"acting":[1,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.c","version":"260'20","reported_seq":695,"reported_epoch":362,"state":"active+clean","last_fresh":"2024-03-11T21:40:45.411296+0000","last_change":"2024-03-11T21:40:02.706928+0000","last_active":"2024-03-11T21:40:45.411296+0000","last_peered":"2024-03-11T21:40:45.411296+0000","last_clean":"2024-03-11T21:40:45.411296+0000","last_became_active":"2024-03-11T21:36:35.067269+0000","last_became_peered":"2024-03-11T21:36:35.067269+0000","last_unstale":"2024-03-11T21:40:45.411296+0000","last_undegraded":"2024-03-11T21:40:45.411296+0000","last_fullsized":"2024-03-11T21:40:45.411296+0000","mapping_epoch":277,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":278,"parent":"0.0","parent_split_bits":0,"last_scrub":"260'20","last_scrub_stamp":"2024-03-11T21:40:02.706832+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:39.525554+0000","last_clean_scrub_stamp":"2024-03-11T21:40:02.706832+0000","objects_scrubbed":5,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:41:04.628603+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":25,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"10.3","version":"0'0","reported_seq":21,"reported_epoch":362,"state":"active+clean","last_fresh":"2024-03-11T21:40:46.104587+0000","last_change":"2024-03-11T21:40:39.541091+0000","last_active":"2024-03-11T21:40:46.104587+0000","last_peered":"2024-03-11T21:40:46.104587+0000","last_clean":"2024-03-11T21:40:46.104587+0000","last_became_active":"2024-03-11T21:40:39.540740+0000","last_became_peered":"2024-03-11T21:40:39.540740+0000","last_unstale":"2024-03-11T21:40:46.104587+0000","last_undegraded":"2024-03-11T21:40:46.104587+0000","last_fullsized":"2024-03-11T21:40:46.104587+0000","mapping_epoch":356,"log_start":"0'0","ondisk_log_start":"0'0","created":356,"last_epoch_clean":357,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:40:38.219602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:40:38.219602+0000","last_clean_scrub_stamp":"2024-03-11T21:40:38.219602+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:42:07.518784+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,3],"acting":[2,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.f","version":"0'0","reported_seq":642,"reported_epoch":362,"state":"active+clean","last_fresh":"2024-03-11T21:40:46.103934+0000","last_change":"2024-03-11T21:36:35.058630+0000","last_active":"2024-03-11T21:40:46.103934+0000","last_peered":"2024-03-11T21:40:46.103934+0000","last_clean":"2024-03-11T21:40:46.103934+0000","last_became_active":"2024-03-11T21:36:35.057800+0000","last_became_peered":"2024-03-11T21:36:35.057800+0000","last_unstale":"2024-03-11T21:40:46.103934+0000","last_undegraded":"2024-03-11T21:40:46.103934+0000","last_fullsized":"2024-03-11T21:40:46.103934+0000","mapping_epoch":277,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":278,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:35:10.608981+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:27:03.010969+0000","last_clean_scrub_stamp":"2024-03-11T21:35:10.608981+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":94,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,4],"acting":[2,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":639,"reported_epoch":362,"state":"active+clean","last_fresh":"2024-03-11T21:40:45.412256+0000","last_change":"2024-03-11T21:36:10.608789+0000","last_active":"2024-03-11T21:40:45.412256+0000","last_peered":"2024-03-11T21:40:45.412256+0000","last_clean":"2024-03-11T21:40:45.412256+0000","last_became_active":"2024-03-11T21:25:42.579409+0000","last_became_peered":"2024-03-11T21:25:42.579409+0000","last_unstale":"2024-03-11T21:40:45.412256+0000","last_undegraded":"2024-03-11T21:40:45.412256+0000","last_fullsized":"2024-03-11T21:40:45.412256+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:36:10.608552+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:30:22.750053+0000","last_clean_scrub_stamp":"2024-03-11T21:36:10.608552+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:40:50.059509+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.b","version":"0'0","reported_seq":291,"reported_epoch":362,"state":"active+clean","last_fresh":"2024-03-11T21:40:46.061622+0000","last_change":"2024-03-11T21:36:16.719432+0000","last_active":"2024-03-11T21:40:46.061622+0000","last_peered":"2024-03-11T21:40:46.061622+0000","last_clean":"2024-03-11T21:40:46.061622+0000","last_became_active":"2024-03-11T21:34:41.134987+0000","last_became_peered":"2024-03-11T21:34:41.134987+0000","last_unstale":"2024-03-11T21:40:46.061622+0000","last_undegraded":"2024-03-11T21:40:46.061622+0000","last_fullsized":"2024-03-11T21:40:46.061622+0000","mapping_epoch":226,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":227,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:36:16.719309+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:36:16.719309+0000","last_clean_scrub_stamp":"2024-03-11T21:36:16.719309+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,5],"acting":[7,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.d","version":"260'39","reported_seq":700,"reported_epoch":362,"state":"active+clean","last_fresh":"2024-03-11T21:40:45.411153+0000","last_change":"2024-03-11T21:35:25.543769+0000","last_active":"2024-03-11T21:40:45.411153+0000","last_peered":"2024-03-11T21:40:45.411153+0000","last_clean":"2024-03-11T21:40:45.411153+0000","last_became_active":"2024-03-11T21:25:40.553454+0000","last_became_peered":"2024-03-11T21:25:40.553454+0000","last_unstale":"2024-03-11T21:40:45.411153+0000","last_undegraded":"2024-03-11T21:40:45.411153+0000","last_fullsized":"2024-03-11T21:40:45.411153+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"165'25","last_scrub_stamp":"2024-03-11T21:35:25.543628+0000","last_deep_scrub":"165'25","last_deep_scrub_stamp":"2024-03-11T21:32:21.678328+0000","last_clean_scrub_stamp":"2024-03-11T21:35:25.543628+0000","objects_scrubbed":8,"log_size":39,"log_dups_size":0,"ondisk_log_size":39,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":50,"num_read_kb":40,"num_write":32,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,1],"acting":[4,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"10.4","version":"0'0","reported_seq":23,"reported_epoch":363,"state":"active+clean","last_fresh":"2024-03-11T21:40:48.411033+0000","last_change":"2024-03-11T21:40:39.540876+0000","last_active":"2024-03-11T21:40:48.411033+0000","last_peered":"2024-03-11T21:40:48.411033+0000","last_clean":"2024-03-11T21:40:48.411033+0000","last_became_active":"2024-03-11T21:40:39.540563+0000","last_became_peered":"2024-03-11T21:40:39.540563+0000","last_unstale":"2024-03-11T21:40:48.411033+0000","last_undegraded":"2024-03-11T21:40:48.411033+0000","last_fullsized":"2024-03-11T21:40:48.411033+0000","mapping_epoch":356,"log_start":"0'0","ondisk_log_start":"0'0","created":356,"last_epoch_clean":357,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:40:38.219602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:40:38.219602+0000","last_clean_scrub_stamp":"2024-03-11T21:40:38.219602+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:41:55.091611+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":291,"reported_epoch":362,"state":"active+clean","last_fresh":"2024-03-11T21:40:46.060940+0000","last_change":"2024-03-11T21:36:06.735035+0000","last_active":"2024-03-11T21:40:46.060940+0000","last_peered":"2024-03-11T21:40:46.060940+0000","last_clean":"2024-03-11T21:40:46.060940+0000","last_became_active":"2024-03-11T21:34:41.145012+0000","last_became_peered":"2024-03-11T21:34:41.145012+0000","last_unstale":"2024-03-11T21:40:46.060940+0000","last_undegraded":"2024-03-11T21:40:46.060940+0000","last_fullsized":"2024-03-11T21:40:46.060940+0000","mapping_epoch":226,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":227,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:36:06.734921+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:28:37.561826+0000","last_clean_scrub_stamp":"2024-03-11T21:36:06.734921+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":23,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,0],"acting":[7,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":637,"reported_epoch":362,"state":"active+clean","last_fresh":"2024-03-11T21:40:46.104830+0000","last_change":"2024-03-11T21:35:14.636561+0000","last_active":"2024-03-11T21:40:46.104830+0000","last_peered":"2024-03-11T21:40:46.104830+0000","last_clean":"2024-03-11T21:40:46.104830+0000","last_became_active":"2024-03-11T21:25:42.574515+0000","last_became_peered":"2024-03-11T21:25:42.574515+0000","last_unstale":"2024-03-11T21:40:46.104830+0000","last_undegraded":"2024-03-11T21:40:46.104830+0000","last_fullsized":"2024-03-11T21:40:46.104830+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:35:14.636450+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:41.544550+0000","last_clean_scrub_stamp":"2024-03-11T21:35:14.636450+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,5],"acting":[2,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.c","version":"0'0","reported_seq":645,"reported_epoch":362,"state":"active+clean","last_fresh":"2024-03-11T21:40:46.104842+0000","last_change":"2024-03-11T21:36:00.849272+0000","last_active":"2024-03-11T21:40:46.104842+0000","last_peered":"2024-03-11T21:40:46.104842+0000","last_clean":"2024-03-11T21:40:46.104842+0000","last_became_active":"2024-03-11T21:25:38.533240+0000","last_became_peered":"2024-03-11T21:25:38.533240+0000","last_unstale":"2024-03-11T21:40:46.104842+0000","last_undegraded":"2024-03-11T21:40:46.104842+0000","last_fullsized":"2024-03-11T21:40:46.104842+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:36:00.849153+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:29:30.279910+0000","last_clean_scrub_stamp":"2024-03-11T21:36:00.849153+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,0],"acting":[2,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.a","version":"260'44","reported_seq":783,"reported_epoch":362,"state":"active+clean","last_fresh":"2024-03-11T21:40:46.103988+0000","last_change":"2024-03-11T21:40:02.183543+0000","last_active":"2024-03-11T21:40:46.103988+0000","last_peered":"2024-03-11T21:40:46.103988+0000","last_clean":"2024-03-11T21:40:46.103988+0000","last_became_active":"2024-03-11T21:34:41.148863+0000","last_became_peered":"2024-03-11T21:34:41.148863+0000","last_unstale":"2024-03-11T21:40:46.103988+0000","last_undegraded":"2024-03-11T21:40:46.103988+0000","last_fullsized":"2024-03-11T21:40:46.103988+0000","mapping_epoch":226,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":227,"parent":"0.0","parent_split_bits":0,"last_scrub":"260'44","last_scrub_stamp":"2024-03-11T21:40:02.183417+0000","last_deep_scrub":"109'28","last_deep_scrub_stamp":"2024-03-11T21:28:24.016389+0000","last_clean_scrub_stamp":"2024-03-11T21:40:02.183417+0000","objects_scrubbed":9,"log_size":44,"log_dups_size":0,"ondisk_log_size":44,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:41:31.089360+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":56,"num_read_kb":45,"num_write":36,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,7],"acting":[6,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"10.5","version":"0'0","reported_seq":21,"reported_epoch":362,"state":"active+clean","last_fresh":"2024-03-11T21:40:46.104311+0000","last_change":"2024-03-11T21:40:39.541067+0000","last_active":"2024-03-11T21:40:46.104311+0000","last_peered":"2024-03-11T21:40:46.104311+0000","last_clean":"2024-03-11T21:40:46.104311+0000","last_became_active":"2024-03-11T21:40:39.540135+0000","last_became_peered":"2024-03-11T21:40:39.540135+0000","last_unstale":"2024-03-11T21:40:46.104311+0000","last_undegraded":"2024-03-11T21:40:46.104311+0000","last_fullsized":"2024-03-11T21:40:46.104311+0000","mapping_epoch":356,"log_start":"0'0","ondisk_log_start":"0'0","created":356,"last_epoch_clean":357,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:40:38.219602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:40:38.219602+0000","last_clean_scrub_stamp":"2024-03-11T21:40:38.219602+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:41:59.306328+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3],"acting":[2,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":649,"reported_epoch":363,"state":"active+clean","last_fresh":"2024-03-11T21:40:48.411016+0000","last_change":"2024-03-11T21:35:37.687281+0000","last_active":"2024-03-11T21:40:48.411016+0000","last_peered":"2024-03-11T21:40:48.411016+0000","last_clean":"2024-03-11T21:40:48.411016+0000","last_became_active":"2024-03-11T21:34:41.154507+0000","last_became_peered":"2024-03-11T21:34:41.154507+0000","last_unstale":"2024-03-11T21:40:48.411016+0000","last_undegraded":"2024-03-11T21:40:48.411016+0000","last_fullsized":"2024-03-11T21:40:48.411016+0000","mapping_epoch":226,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":227,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:35:37.687187+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:27:11.361913+0000","last_clean_scrub_stamp":"2024-03-11T21:35:37.687187+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:40:49.118345+0000","scrub_duration":39,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":646,"reported_epoch":362,"state":"active+clean","last_fresh":"2024-03-11T21:40:46.103274+0000","last_change":"2024-03-11T21:40:28.015350+0000","last_active":"2024-03-11T21:40:46.103274+0000","last_peered":"2024-03-11T21:40:46.103274+0000","last_clean":"2024-03-11T21:40:46.103274+0000","last_became_active":"2024-03-11T21:36:35.059192+0000","last_became_peered":"2024-03-11T21:36:35.059192+0000","last_unstale":"2024-03-11T21:40:46.103274+0000","last_undegraded":"2024-03-11T21:40:46.103274+0000","last_fullsized":"2024-03-11T21:40:46.103274+0000","mapping_epoch":277,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":278,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:40:28.015231+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:41.544550+0000","last_clean_scrub_stamp":"2024-03-11T21:40:28.015231+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:41:54.304353+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,3],"acting":[2,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.d","version":"0'0","reported_seq":656,"reported_epoch":362,"state":"active+clean","last_fresh":"2024-03-11T21:40:45.412480+0000","last_change":"2024-03-11T21:40:31.873781+0000","last_active":"2024-03-11T21:40:45.412480+0000","last_peered":"2024-03-11T21:40:45.412480+0000","last_clean":"2024-03-11T21:40:45.412480+0000","last_became_active":"2024-03-11T21:36:35.058296+0000","last_became_peered":"2024-03-11T21:36:35.058296+0000","last_unstale":"2024-03-11T21:40:45.412480+0000","last_undegraded":"2024-03-11T21:40:45.412480+0000","last_fullsized":"2024-03-11T21:40:45.412480+0000","mapping_epoch":277,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":278,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:40:31.873695+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:37.512018+0000","last_clean_scrub_stamp":"2024-03-11T21:40:31.873695+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:41:53.890729+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,3],"acting":[1,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.b","version":"260'15","reported_seq":670,"reported_epoch":363,"state":"active+clean","last_fresh":"2024-03-11T21:40:48.411560+0000","last_change":"2024-03-11T21:35:27.760318+0000","last_active":"2024-03-11T21:40:48.411560+0000","last_peered":"2024-03-11T21:40:48.411560+0000","last_clean":"2024-03-11T21:40:48.411560+0000","last_became_active":"2024-03-11T21:25:40.549612+0000","last_became_peered":"2024-03-11T21:25:40.549612+0000","last_unstale":"2024-03-11T21:40:48.411560+0000","last_undegraded":"2024-03-11T21:40:48.411560+0000","last_fullsized":"2024-03-11T21:40:48.411560+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'9","last_scrub_stamp":"2024-03-11T21:35:27.760208+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:39.525554+0000","last_clean_scrub_stamp":"2024-03-11T21:35:27.760208+0000","objects_scrubbed":4,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:40:49.118382+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.6","version":"0'0","reported_seq":21,"reported_epoch":362,"state":"active+clean","last_fresh":"2024-03-11T21:40:46.104813+0000","last_change":"2024-03-11T21:40:39.243542+0000","last_active":"2024-03-11T21:40:46.104813+0000","last_peered":"2024-03-11T21:40:46.104813+0000","last_clean":"2024-03-11T21:40:46.104813+0000","last_became_active":"2024-03-11T21:40:39.243049+0000","last_became_peered":"2024-03-11T21:40:39.243049+0000","last_unstale":"2024-03-11T21:40:46.104813+0000","last_undegraded":"2024-03-11T21:40:46.104813+0000","last_fullsized":"2024-03-11T21:40:46.104813+0000","mapping_epoch":356,"log_start":"0'0","ondisk_log_start":"0'0","created":356,"last_epoch_clean":357,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:40:38.219602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:40:38.219602+0000","last_clean_scrub_stamp":"2024-03-11T21:40:38.219602+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:41:54.991800+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,4],"acting":[5,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":549,"reported_epoch":362,"state":"active+clean","last_fresh":"2024-03-11T21:40:45.410693+0000","last_change":"2024-03-11T21:36:35.057118+0000","last_active":"2024-03-11T21:40:45.410693+0000","last_peered":"2024-03-11T21:40:45.410693+0000","last_clean":"2024-03-11T21:40:45.410693+0000","last_became_active":"2024-03-11T21:36:35.056518+0000","last_became_peered":"2024-03-11T21:36:35.056518+0000","last_unstale":"2024-03-11T21:40:45.410693+0000","last_undegraded":"2024-03-11T21:40:45.410693+0000","last_fullsized":"2024-03-11T21:40:45.410693+0000","mapping_epoch":277,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":278,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:35:24.514584+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:43.559452+0000","last_clean_scrub_stamp":"2024-03-11T21:35:24.514584+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,3],"acting":[4,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.e","version":"0'0","reported_seq":674,"reported_epoch":362,"state":"active+clean","last_fresh":"2024-03-11T21:40:46.105056+0000","last_change":"2024-03-11T21:40:11.026129+0000","last_active":"2024-03-11T21:40:46.105056+0000","last_peered":"2024-03-11T21:40:46.105056+0000","last_clean":"2024-03-11T21:40:46.105056+0000","last_became_active":"2024-03-11T21:36:35.056777+0000","last_became_peered":"2024-03-11T21:36:35.056777+0000","last_unstale":"2024-03-11T21:40:46.105056+0000","last_undegraded":"2024-03-11T21:40:46.105056+0000","last_fullsized":"2024-03-11T21:40:46.105056+0000","mapping_epoch":277,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":278,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:40:11.025986+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:37.512018+0000","last_clean_scrub_stamp":"2024-03-11T21:40:11.025986+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:41:21.568389+0000","scrub_duration":114,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,7],"acting":[2,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.8","version":"260'34","reported_seq":736,"reported_epoch":362,"state":"active+clean","last_fresh":"2024-03-11T21:40:46.103647+0000","last_change":"2024-03-11T21:35:20.304345+0000","last_active":"2024-03-11T21:40:46.103647+0000","last_peered":"2024-03-11T21:40:46.103647+0000","last_clean":"2024-03-11T21:40:46.103647+0000","last_became_active":"2024-03-11T21:34:41.146631+0000","last_became_peered":"2024-03-11T21:34:41.146631+0000","last_unstale":"2024-03-11T21:40:46.103647+0000","last_undegraded":"2024-03-11T21:40:46.103647+0000","last_fullsized":"2024-03-11T21:40:46.103647+0000","mapping_epoch":226,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":227,"parent":"0.0","parent_split_bits":0,"last_scrub":"109'22","last_scrub_stamp":"2024-03-11T21:35:20.304114+0000","last_deep_scrub":"109'22","last_deep_scrub_stamp":"2024-03-11T21:32:07.573197+0000","last_clean_scrub_stamp":"2024-03-11T21:35:20.304114+0000","objects_scrubbed":7,"log_size":34,"log_dups_size":0,"ondisk_log_size":34,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":44,"num_read_kb":35,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.9","version":"229'11","reported_seq":368,"reported_epoch":362,"state":"active+clean","last_fresh":"2024-03-11T21:40:46.061507+0000","last_change":"2024-03-11T21:36:18.726876+0000","last_active":"2024-03-11T21:40:46.061507+0000","last_peered":"2024-03-11T21:40:46.061507+0000","last_clean":"2024-03-11T21:40:46.061507+0000","last_became_active":"2024-03-11T21:34:41.156892+0000","last_became_peered":"2024-03-11T21:34:41.156892+0000","last_unstale":"2024-03-11T21:40:46.061507+0000","last_undegraded":"2024-03-11T21:40:46.061507+0000","last_fullsized":"2024-03-11T21:40:46.061507+0000","mapping_epoch":226,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":227,"parent":"0.0","parent_split_bits":0,"last_scrub":"229'11","last_scrub_stamp":"2024-03-11T21:36:18.726763+0000","last_deep_scrub":"94'9","last_deep_scrub_stamp":"2024-03-11T21:26:53.422552+0000","last_clean_scrub_stamp":"2024-03-11T21:36:18.726763+0000","objects_scrubbed":1,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":10,"num_pg":16,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":48,"acting":48,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":1531,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":17,"num_read_kb":17,"num_write":10,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":1308,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":4593,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":6,"ondisk_log_size":6,"up":96,"acting":96,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":198,"num_read_kb":294,"num_write":179,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":2951840,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":2375680,"data_stored":2361472,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":128,"ondisk_log_size":128,"up":3,"acting":3,"num_store_stats":4},{"poolid":3,"num_pg":3,"stat_sum":{"num_bytes":408,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":902,"num_read_kb":897,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1224,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":8,"ondisk_log_size":8,"up":9,"acting":9,"num_store_stats":7},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":3702,"num_objects":178,"num_object_clones":0,"num_object_copies":534,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":178,"num_whiteouts":0,"num_read":1120,"num_read_kb":877,"num_write":703,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":186,"num_bytes_recovered":2200,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":417792,"data_stored":11106,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":770,"ondisk_log_size":770,"up":96,"acting":96,"num_store_stats":8},{"poolid":5,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":78,"ondisk_log_size":78,"up":96,"acting":96,"num_store_stats":8},{"poolid":6,"num_pg":32,"stat_sum":{"num_bytes":383,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":370,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1149,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":96,"acting":96,"num_store_stats":8}],"osd_stats":[{"osd":7,"up_from":60,"seq":257698037945,"num_pgs":60,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":511592,"kb_used_data":2840,"kb_used_omap":1,"kb_used_meta":508734,"kb_avail":93233560,"statfs":{"total":95995035648,"available":95471165440,"internally_reserved":0,"allocated":2908160,"data_stored":1901152,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":520944074},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":6,"up_from":50,"seq":214748364991,"num_pgs":50,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":413036,"kb_used_data":2840,"kb_used_omap":1,"kb_used_meta":410174,"kb_avail":93332116,"statfs":{"total":95995035648,"available":95572086784,"internally_reserved":0,"allocated":2908160,"data_stored":1901881,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":420018630},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834575065,"num_pgs":54,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":579076,"kb_used_data":2288,"kb_used_omap":1,"kb_used_meta":576766,"kb_avail":93166076,"statfs":{"total":95995035648,"available":95402061824,"internally_reserved":0,"allocated":2342912,"data_stored":1311511,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":590608837},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738590,"num_pgs":60,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":565752,"kb_used_data":2852,"kb_used_omap":1,"kb_used_meta":562878,"kb_avail":93179400,"statfs":{"total":95995035648,"available":95415705600,"internally_reserved":0,"allocated":2920448,"data_stored":1901915,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":576387525},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411539,"num_pgs":46,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":438812,"kb_used_data":2256,"kb_used_omap":1,"kb_used_meta":436542,"kb_avail":93306340,"statfs":{"total":95995035648,"available":95545692160,"internally_reserved":0,"allocated":2310144,"data_stored":1311281,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":447019461},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":277,"seq":1189705941044,"num_pgs":59,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":278920,"kb_used_data":2292,"kb_used_omap":4,"kb_used_meta":276603,"kb_avail":93466232,"statfs":{"total":95995035648,"available":95709421568,"internally_reserved":0,"allocated":2347008,"data_stored":1311955,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4442,"internal_metadata":283242150},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":32,"seq":137438953672,"num_pgs":61,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":408304,"kb_used_data":2276,"kb_used_omap":1,"kb_used_meta":406014,"kb_avail":93336848,"statfs":{"total":95995035648,"available":95576932352,"internally_reserved":0,"allocated":2330624,"data_stored":1311129,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":415758789},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":41,"seq":176093659331,"num_pgs":54,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":552668,"kb_used_data":2256,"kb_used_omap":1,"kb_used_meta":550398,"kb_avail":93192484,"statfs":{"total":95995035648,"available":95429103616,"internally_reserved":0,"allocated":2310144,"data_stored":1310896,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":563608005},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1049,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":46,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":436,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":957,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":574,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":436,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1003,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":92,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":1430,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":77824,"data_stored":2008,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":1018,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":1898,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":1474,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":990,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":1034,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":1254,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":370,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":383,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":370,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-11T21:40:54.450 INFO:tasks.cephadm.ceph_manager.ceph:recovered! 2024-03-11T21:40:54.451 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-03-11T21:40:54.451 INFO:tasks.thrashosds.thrasher:Growing pool unique_pool_3 2024-03-11T21:40:54.451 DEBUG:teuthology.orchestra.run.smithi092:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid 00db0016-dfed-11ee-95c8-87774f69a715 -- ceph pg dump --format=json 2024-03-11T21:40:54.460 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:40:54.495 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:40:54.496 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:40:54.496 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:40:54.597 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:40:54.638 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:40:54.639 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:40:54.639 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:40:54.740 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:40:54.776 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:40:54.777 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:40:54.777 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:40:54.829 INFO:tasks.radosbench.radosbench.0.smithi092.stdout: 2 16 7109 7093 221.635 215.875 0.00491983 0.00450263 2024-03-11T21:40:54.831 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:40:54 smithi097 ceph-mon[33317]: pgmap v716: 148 pgs: 148 active+clean; 582 KiB data, 3.6 GiB used, 712 GiB / 715 GiB avail; 646 B/s rd, 0 op/s 2024-03-11T21:40:54.843 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:40:54 smithi092 ceph-mon[29572]: pgmap v716: 148 pgs: 148 active+clean; 582 KiB data, 3.6 GiB used, 712 GiB / 715 GiB avail; 646 B/s rd, 0 op/s 2024-03-11T21:40:54.843 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:40:54 smithi092 ceph-mon[37563]: pgmap v716: 148 pgs: 148 active+clean; 582 KiB data, 3.6 GiB used, 712 GiB / 715 GiB avail; 646 B/s rd, 0 op/s 2024-03-11T21:40:54.879 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:40:54.928 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:40:54.929 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:40:54.929 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:40:55.030 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:40:55.067 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:40:55.067 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:40:55.068 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:40:55.169 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:40:55.207 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:40:55.208 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:40:55.208 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:40:55.309 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:40:55.346 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:40:55.346 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:40:55.347 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:40:55.448 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:40:55.476 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:40:55.476 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:40:55.477 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:40:55.578 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:40:55.617 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:40:55.617 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:40:55.618 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:40:55.719 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:40:55.753 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:40:55.753 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:40:55.754 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:40:55.829 INFO:tasks.radosbench.radosbench.0.smithi092.stdout: 3 16 10182 10166 211.772 192.062 0.00416238 0.00471428 2024-03-11T21:40:55.854 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:40:55.892 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:40:55.892 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:40:55.893 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:40:55.993 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:40:56.029 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:40:56.030 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:40:56.030 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:40:56.131 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:40:56.170 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:40:56.171 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:40:56.171 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:40:56.272 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:40:56.306 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:40:56.307 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:40:56.307 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:40:56.408 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:40:56.447 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:40:56.447 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:40:56.448 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:40:56.548 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:40:56.588 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:40:56.589 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:40:56.589 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:40:56.690 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:40:56.728 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:40:56.729 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:40:56.729 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:40:56.829 INFO:tasks.radosbench.radosbench.0.smithi092.stdout: 4 16 13259 13243 206.903 192.312 0.00547647 0.00482618 2024-03-11T21:40:56.830 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:40:56.830 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:40:56 smithi097 ceph-mon[33317]: pgmap v717: 148 pgs: 148 active+clean; 37 MiB data, 3.7 GiB used, 712 GiB / 715 GiB avail; 1.2 KiB/s rd, 4.6 MiB/s wr, 74 op/s 2024-03-11T21:40:56.865 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:40:56 smithi092 ceph-mon[29572]: pgmap v717: 148 pgs: 148 active+clean; 37 MiB data, 3.7 GiB used, 712 GiB / 715 GiB avail; 1.2 KiB/s rd, 4.6 MiB/s wr, 74 op/s 2024-03-11T21:40:56.865 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:40:56 smithi092 ceph-mon[37563]: pgmap v717: 148 pgs: 148 active+clean; 37 MiB data, 3.7 GiB used, 712 GiB / 715 GiB avail; 1.2 KiB/s rd, 4.6 MiB/s wr, 74 op/s 2024-03-11T21:40:56.869 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:40:56.869 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:40:56.870 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:40:56.971 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:40:57.007 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:40:57.007 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:40:57.008 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:40:57.109 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:40:57.158 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:40:57.159 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:40:57.159 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:40:57.260 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:40:57.296 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:40:57.297 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:40:57.297 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:40:57.398 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:40:57.470 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:40:57.471 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:40:57.471 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:40:57.572 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:40:57.614 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:40:57.614 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:40:57.615 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:40:57.715 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:40:57.752 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:40:57.753 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:40:57.753 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:40:57.829 INFO:tasks.radosbench.radosbench.0.smithi092.stdout: 5 16 16198 16182 202.256 183.688 0.00430519 0.00493782 2024-03-11T21:40:57.854 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:40:57.892 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:40:57.893 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:40:57.894 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:40:57.994 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:40:58.032 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:40:58.033 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:40:58.033 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:40:58.134 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:40:58.181 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:40:58.182 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:40:58.182 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:40:58.190 INFO:journalctl@ceph.mgr.y.smithi092.stdout:Mar 11 21:40:57 smithi092 ceph-00db0016-dfed-11ee-95c8-87774f69a715-mgr-y[29801]: ::ffff:172.21.15.97 - - [11/Mar/2024:21:40:57] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-11T21:40:58.283 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:40:58.323 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:40:58.324 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:40:58.324 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:40:58.425 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:40:58.464 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:40:58.465 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:40:58.465 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:40:58.566 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:40:58.601 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:40:58.601 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:40:58.602 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:40:58.703 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:40:58.739 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:40:58.739 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:40:58.740 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:40:58.829 INFO:tasks.radosbench.radosbench.0.smithi092.stdout: 6 16 19003 18987 197.763 175.312 0.00420672 0.00504915 2024-03-11T21:40:58.830 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:40:58 smithi097 ceph-mon[33317]: pgmap v718: 148 pgs: 148 active+clean; 151 MiB data, 4.1 GiB used, 711 GiB / 715 GiB avail; 573 B/s rd, 17 MiB/s wr, 270 op/s 2024-03-11T21:40:58.842 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:40:58.864 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:40:58 smithi092 ceph-mon[37563]: pgmap v718: 148 pgs: 148 active+clean; 151 MiB data, 4.1 GiB used, 711 GiB / 715 GiB avail; 573 B/s rd, 17 MiB/s wr, 270 op/s 2024-03-11T21:40:58.864 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:40:58 smithi092 ceph-mon[29572]: pgmap v718: 148 pgs: 148 active+clean; 151 MiB data, 4.1 GiB used, 711 GiB / 715 GiB avail; 573 B/s rd, 17 MiB/s wr, 270 op/s 2024-03-11T21:40:58.877 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:40:58.878 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:40:58.879 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:40:58.979 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:40:59.034 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:40:59.035 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:40:59.035 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:40:59.136 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:40:59.179 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:40:59.180 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:40:59.180 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:40:59.281 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:40:59.318 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:40:59.319 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:40:59.319 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:40:59.420 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:40:59.454 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:40:59.455 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:40:59.455 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:40:59.468 INFO:teuthology.orchestra.run.smithi092.stderr:noscrub is unset 2024-03-11T21:40:59.555 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:40:59.593 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:40:59.594 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:40:59.594 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:40:59.695 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:40:59.731 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:40:59.731 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:40:59.732 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:40:59.830 INFO:tasks.radosbench.radosbench.0.smithi092.stdout: 7 16 21457 21441 191.42 153.375 0.00744503 0.00521809 2024-03-11T21:40:59.830 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:40:59 smithi097 ceph-mon[33317]: from='client.? v1:172.21.15.92:0/1814759268' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-11T21:40:59.832 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:40:59.864 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:40:59 smithi092 ceph-mon[29572]: from='client.? v1:172.21.15.92:0/1814759268' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-11T21:40:59.864 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:40:59 smithi092 ceph-mon[37563]: from='client.? v1:172.21.15.92:0/1814759268' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-11T21:40:59.868 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:40:59.869 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:40:59.870 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:40:59.971 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:41:00.003 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:41:00.004 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:41:00.004 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:41:00.054 INFO:teuthology.orchestra.run.smithi092.stdout:osd.0: {} 2024-03-11T21:41:00.095 INFO:teuthology.orchestra.run.smithi092.stdout:{ 2024-03-11T21:41:00.095 INFO:teuthology.orchestra.run.smithi092.stdout: "size": 20, 2024-03-11T21:41:00.095 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 600, 2024-03-11T21:41:00.095 INFO:teuthology.orchestra.run.smithi092.stdout: "ops": [ 2024-03-11T21:41:00.095 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:41:00.095 INFO:teuthology.orchestra.run.smithi092.stdout: "description": "osd_op(client.15150.0:88135 8.1 8:8d497fa8:::benchmark_data_smithi092_104865_object88134:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e180)", 2024-03-11T21:41:00.095 INFO:teuthology.orchestra.run.smithi092.stdout: "initiated_at": "2024-03-11T21:32:56.040513+0000", 2024-03-11T21:41:00.095 INFO:teuthology.orchestra.run.smithi092.stdout: "age": 484.06294614500001, 2024-03-11T21:41:00.095 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.17163281, 2024-03-11T21:41:00.096 INFO:teuthology.orchestra.run.smithi092.stdout: "type_data": { 2024-03-11T21:41:00.096 INFO:teuthology.orchestra.run.smithi092.stdout: "flag_point": "reached pg", 2024-03-11T21:41:00.096 INFO:teuthology.orchestra.run.smithi092.stdout: "client_info": { 2024-03-11T21:41:00.096 INFO:teuthology.orchestra.run.smithi092.stdout: "client": "client.15150", 2024-03-11T21:41:00.096 INFO:teuthology.orchestra.run.smithi092.stdout: "client_addr": "v1:172.21.15.92:0/436606717", 2024-03-11T21:41:00.096 INFO:teuthology.orchestra.run.smithi092.stdout: "tid": 88135 2024-03-11T21:41:00.096 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:41:00.096 INFO:teuthology.orchestra.run.smithi092.stdout: "events": [ 2024-03-11T21:41:00.096 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:41:00.096 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "initiated", 2024-03-11T21:41:00.096 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:32:56.040513+0000", 2024-03-11T21:41:00.096 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-03-11T21:41:00.096 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:41:00.097 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:41:00.097 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "header_read", 2024-03-11T21:41:00.097 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:32:56.040513+0000", 2024-03-11T21:41:00.097 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-03-11T21:41:00.097 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:41:00.097 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:41:00.097 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "throttled", 2024-03-11T21:41:00.097 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:32:56.040515+0000", 2024-03-11T21:41:00.097 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.767e-06 2024-03-11T21:41:00.097 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:41:00.097 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:41:00.097 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "all_read", 2024-03-11T21:41:00.097 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:32:56.040539+0000", 2024-03-11T21:41:00.097 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 2.4485e-05 2024-03-11T21:41:00.098 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:41:00.098 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:41:00.098 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "dispatched", 2024-03-11T21:41:00.098 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:32:56.040554+0000", 2024-03-11T21:41:00.098 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.5034e-05 2024-03-11T21:41:00.098 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:41:00.098 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:41:00.098 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "queued_for_pg", 2024-03-11T21:41:00.098 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:32:56.040562+0000", 2024-03-11T21:41:00.098 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 8.0220000000000004e-06 2024-03-11T21:41:00.098 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:41:00.098 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:41:00.098 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-03-11T21:41:00.099 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:32:56.040611+0000", 2024-03-11T21:41:00.099 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 4.8692000000000001e-05 2024-03-11T21:41:00.099 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:41:00.099 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:41:00.099 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "started", 2024-03-11T21:41:00.099 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:32:56.040724+0000", 2024-03-11T21:41:00.099 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.000113192 2024-03-11T21:41:00.099 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:41:00.099 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:41:00.099 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "waiting for subops from 0,2", 2024-03-11T21:41:00.099 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:32:56.040878+0000", 2024-03-11T21:41:00.099 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00015365999999999999 2024-03-11T21:41:00.099 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:41:00.099 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:41:00.100 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "op_commit", 2024-03-11T21:41:00.100 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:32:56.061039+0000", 2024-03-11T21:41:00.100 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.020160919999999999 2024-03-11T21:41:00.100 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:41:00.100 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:41:00.100 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-03-11T21:41:00.100 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:32:56.212128+0000", 2024-03-11T21:41:00.100 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.151089433 2024-03-11T21:41:00.100 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:41:00.100 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:41:00.100 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "done", 2024-03-11T21:41:00.100 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:32:56.212146+0000", 2024-03-11T21:41:00.100 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.7604999999999998e-05 2024-03-11T21:41:00.100 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-03-11T21:41:00.100 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-03-11T21:41:00.101 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-03-11T21:41:00.101 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:41:00.101 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:41:00.101 INFO:teuthology.orchestra.run.smithi092.stdout: "description": "osd_op(client.15150.0:88143 8.1 8:88a030b3:::benchmark_data_smithi092_104865_object88142:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e180)", 2024-03-11T21:41:00.101 INFO:teuthology.orchestra.run.smithi092.stdout: "initiated_at": "2024-03-11T21:32:56.054816+0000", 2024-03-11T21:41:00.101 INFO:teuthology.orchestra.run.smithi092.stdout: "age": 484.04864278399998, 2024-03-11T21:41:00.101 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.16188037399999999, 2024-03-11T21:41:00.101 INFO:teuthology.orchestra.run.smithi092.stdout: "type_data": { 2024-03-11T21:41:00.101 INFO:teuthology.orchestra.run.smithi092.stdout: "flag_point": "reached pg", 2024-03-11T21:41:00.101 INFO:teuthology.orchestra.run.smithi092.stdout: "client_info": { 2024-03-11T21:41:00.101 INFO:teuthology.orchestra.run.smithi092.stdout: "client": "client.15150", 2024-03-11T21:41:00.101 INFO:teuthology.orchestra.run.smithi092.stdout: "client_addr": "v1:172.21.15.92:0/436606717", 2024-03-11T21:41:00.101 INFO:teuthology.orchestra.run.smithi092.stdout: "tid": 88143 2024-03-11T21:41:00.101 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:41:00.102 INFO:teuthology.orchestra.run.smithi092.stdout: "events": [ 2024-03-11T21:41:00.102 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:41:00.102 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "initiated", 2024-03-11T21:41:00.102 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:32:56.054816+0000", 2024-03-11T21:41:00.102 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-03-11T21:41:00.102 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:41:00.102 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:41:00.102 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "header_read", 2024-03-11T21:41:00.102 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:32:56.054816+0000", 2024-03-11T21:41:00.102 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-03-11T21:41:00.102 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:41:00.102 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:41:00.102 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "throttled", 2024-03-11T21:41:00.102 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:32:56.054819+0000", 2024-03-11T21:41:00.103 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 2.915e-06 2024-03-11T21:41:00.103 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:41:00.103 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:41:00.103 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "all_read", 2024-03-11T21:41:00.103 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:32:56.054863+0000", 2024-03-11T21:41:00.103 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 4.3383000000000001e-05 2024-03-11T21:41:00.103 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:41:00.103 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:41:00.103 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "dispatched", 2024-03-11T21:41:00.103 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:32:56.054885+0000", 2024-03-11T21:41:00.103 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 2.2507999999999999e-05 2024-03-11T21:41:00.103 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:41:00.103 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:41:00.103 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "queued_for_pg", 2024-03-11T21:41:00.104 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:32:56.054896+0000", 2024-03-11T21:41:00.104 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.1450999999999999e-05 2024-03-11T21:41:00.104 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:41:00.104 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:41:00.104 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-03-11T21:41:00.104 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:32:56.058263+0000", 2024-03-11T21:41:00.104 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.003366399 2024-03-11T21:41:00.104 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:41:00.104 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:41:00.104 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "started", 2024-03-11T21:41:00.104 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:32:56.058391+0000", 2024-03-11T21:41:00.104 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.000128477 2024-03-11T21:41:00.104 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:41:00.104 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:41:00.104 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "waiting for subops from 0,2", 2024-03-11T21:41:00.104 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:32:56.058527+0000", 2024-03-11T21:41:00.105 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00013525700000000001 2024-03-11T21:41:00.105 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:41:00.105 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:41:00.105 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "op_commit", 2024-03-11T21:41:00.105 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:32:56.061984+0000", 2024-03-11T21:41:00.105 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00345729 2024-03-11T21:41:00.105 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:41:00.105 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:41:00.105 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-03-11T21:41:00.105 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:32:56.216682+0000", 2024-03-11T21:41:00.105 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.15469781699999999 2024-03-11T21:41:00.105 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:41:00.105 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:41:00.106 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "done", 2024-03-11T21:41:00.106 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:32:56.216697+0000", 2024-03-11T21:41:00.106 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.4877e-05 2024-03-11T21:41:00.106 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-03-11T21:41:00.106 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-03-11T21:41:00.106 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-03-11T21:41:00.106 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:41:00.106 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:41:00.106 INFO:teuthology.orchestra.run.smithi092.stdout: "description": "osd_op(client.15150.0:88144 8.3 8:c2cccf37:::benchmark_data_smithi092_104865_object88143:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e180)", 2024-03-11T21:41:00.106 INFO:teuthology.orchestra.run.smithi092.stdout: "initiated_at": "2024-03-11T21:32:56.054918+0000", 2024-03-11T21:41:00.106 INFO:teuthology.orchestra.run.smithi092.stdout: "age": 484.04854113800002, 2024-03-11T21:41:00.106 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.16754129500000001, 2024-03-11T21:41:00.106 INFO:teuthology.orchestra.run.smithi092.stdout: "type_data": { 2024-03-11T21:41:00.107 INFO:teuthology.orchestra.run.smithi092.stdout: "flag_point": "reached pg", 2024-03-11T21:41:00.107 INFO:teuthology.orchestra.run.smithi092.stdout: "client_info": { 2024-03-11T21:41:00.107 INFO:teuthology.orchestra.run.smithi092.stdout: "client": "client.15150", 2024-03-11T21:41:00.107 INFO:teuthology.orchestra.run.smithi092.stdout: "client_addr": "v1:172.21.15.92:0/436606717", 2024-03-11T21:41:00.107 INFO:teuthology.orchestra.run.smithi092.stdout: "tid": 88144 2024-03-11T21:41:00.107 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:41:00.107 INFO:teuthology.orchestra.run.smithi092.stdout: "events": [ 2024-03-11T21:41:00.107 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:41:00.107 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "initiated", 2024-03-11T21:41:00.107 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:32:56.054918+0000", 2024-03-11T21:41:00.107 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-03-11T21:41:00.107 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:41:00.107 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:41:00.107 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "header_read", 2024-03-11T21:41:00.107 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:32:56.054918+0000", 2024-03-11T21:41:00.108 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-03-11T21:41:00.108 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:41:00.108 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:41:00.108 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "throttled", 2024-03-11T21:41:00.108 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:32:56.054919+0000", 2024-03-11T21:41:00.108 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 7.4700000000000001e-07 2024-03-11T21:41:00.108 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:41:00.108 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:41:00.108 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "all_read", 2024-03-11T21:41:00.108 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:32:56.054941+0000", 2024-03-11T21:41:00.109 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 2.2838999999999999e-05 2024-03-11T21:41:00.109 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:41:00.109 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:41:00.109 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "dispatched", 2024-03-11T21:41:00.109 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:32:56.054951+0000", 2024-03-11T21:41:00.109 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 9.7499999999999998e-06 2024-03-11T21:41:00.109 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:41:00.109 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:41:00.109 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "queued_for_pg", 2024-03-11T21:41:00.109 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:32:56.054957+0000", 2024-03-11T21:41:00.109 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 5.7729999999999996e-06 2024-03-11T21:41:00.109 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:41:00.109 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:41:00.110 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-03-11T21:41:00.110 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:32:56.055030+0000", 2024-03-11T21:41:00.110 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 7.3035999999999997e-05 2024-03-11T21:41:00.110 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:41:00.110 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:41:00.110 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "started", 2024-03-11T21:41:00.110 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:32:56.055180+0000", 2024-03-11T21:41:00.110 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00014947800000000001 2024-03-11T21:41:00.110 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:41:00.110 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:41:00.110 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "waiting for subops from 5,6", 2024-03-11T21:41:00.111 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:32:56.055328+0000", 2024-03-11T21:41:00.111 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00014811800000000001 2024-03-11T21:41:00.111 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:41:00.111 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:41:00.111 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "op_commit", 2024-03-11T21:41:00.111 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:32:56.060662+0000", 2024-03-11T21:41:00.111 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.005333979 2024-03-11T21:41:00.111 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:41:00.111 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:41:00.111 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "sub_op_commit_rec", 2024-03-11T21:41:00.112 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:32:56.065170+0000", 2024-03-11T21:41:00.112 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.0045079669999999999 2024-03-11T21:41:00.112 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:41:00.112 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:41:00.112 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-03-11T21:41:00.112 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:32:56.222444+0000", 2024-03-11T21:41:00.112 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.157274264 2024-03-11T21:41:00.112 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:41:00.112 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:41:00.112 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "done", 2024-03-11T21:41:00.112 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:32:56.222459+0000", 2024-03-11T21:41:00.112 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.5344000000000001e-05 2024-03-11T21:41:00.112 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-03-11T21:41:00.112 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-03-11T21:41:00.113 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-03-11T21:41:00.113 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:41:00.113 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:41:00.113 INFO:teuthology.orchestra.run.smithi092.stdout: "description": "osd_op(client.15150.0:107411 8.1 8:85752a3c:::benchmark_data_smithi092_104865_object107410:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e192)", 2024-03-11T21:41:00.113 INFO:teuthology.orchestra.run.smithi092.stdout: "initiated_at": "2024-03-11T21:33:17.604654+0000", 2024-03-11T21:41:00.113 INFO:teuthology.orchestra.run.smithi092.stdout: "age": 462.498804667, 2024-03-11T21:41:00.113 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.14808184699999999, 2024-03-11T21:41:00.113 INFO:teuthology.orchestra.run.smithi092.stdout: "type_data": { 2024-03-11T21:41:00.113 INFO:teuthology.orchestra.run.smithi092.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-11T21:41:00.113 INFO:teuthology.orchestra.run.smithi092.stdout: "client_info": { 2024-03-11T21:41:00.113 INFO:teuthology.orchestra.run.smithi092.stdout: "client": "client.15150", 2024-03-11T21:41:00.113 INFO:teuthology.orchestra.run.smithi092.stdout: "client_addr": "v1:172.21.15.92:0/436606717", 2024-03-11T21:41:00.113 INFO:teuthology.orchestra.run.smithi092.stdout: "tid": 107411 2024-03-11T21:41:00.113 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:41:00.113 INFO:teuthology.orchestra.run.smithi092.stdout: "events": [ 2024-03-11T21:41:00.114 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:41:00.114 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "initiated", 2024-03-11T21:41:00.114 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:33:17.604654+0000", 2024-03-11T21:41:00.114 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-03-11T21:41:00.114 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:41:00.114 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:41:00.114 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "header_read", 2024-03-11T21:41:00.114 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:33:17.604654+0000", 2024-03-11T21:41:00.114 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-03-11T21:41:00.114 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:41:00.114 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:41:00.114 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "throttled", 2024-03-11T21:41:00.114 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:33:17.604656+0000", 2024-03-11T21:41:00.115 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.812e-06 2024-03-11T21:41:00.115 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:41:00.115 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:41:00.115 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "all_read", 2024-03-11T21:41:00.115 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:33:17.604681+0000", 2024-03-11T21:41:00.115 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 2.5219999999999999e-05 2024-03-11T21:41:00.115 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:41:00.115 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:41:00.115 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "dispatched", 2024-03-11T21:41:00.115 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:33:17.604693+0000", 2024-03-11T21:41:00.116 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.1678999999999999e-05 2024-03-11T21:41:00.116 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:41:00.116 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:41:00.116 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "queued_for_pg", 2024-03-11T21:41:00.117 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:33:17.604701+0000", 2024-03-11T21:41:00.117 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 7.4479999999999997e-06 2024-03-11T21:41:00.117 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:41:00.117 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:41:00.117 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-03-11T21:41:00.117 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:33:17.604757+0000", 2024-03-11T21:41:00.117 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 5.6030999999999999e-05 2024-03-11T21:41:00.117 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:41:00.117 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:41:00.117 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "started", 2024-03-11T21:41:00.117 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:33:17.604889+0000", 2024-03-11T21:41:00.117 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.000132297 2024-03-11T21:41:00.117 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:41:00.118 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:41:00.118 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "waiting for subops from 0,2", 2024-03-11T21:41:00.118 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:33:17.605200+0000", 2024-03-11T21:41:00.118 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00031093899999999999 2024-03-11T21:41:00.118 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:41:00.118 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:41:00.118 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "op_commit", 2024-03-11T21:41:00.118 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:33:17.606932+0000", 2024-03-11T21:41:00.118 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.001732253 2024-03-11T21:41:00.118 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:41:00.118 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:41:00.118 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "sub_op_commit_rec", 2024-03-11T21:41:00.118 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:33:17.608901+0000", 2024-03-11T21:41:00.119 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.0019693549999999999 2024-03-11T21:41:00.119 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:41:00.119 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:41:00.119 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "sub_op_commit_rec", 2024-03-11T21:41:00.119 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:33:17.752511+0000", 2024-03-11T21:41:00.119 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.14360931699999999 2024-03-11T21:41:00.119 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:41:00.119 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:41:00.119 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "commit_sent", 2024-03-11T21:41:00.119 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:33:17.752688+0000", 2024-03-11T21:41:00.119 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.000177596 2024-03-11T21:41:00.119 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:41:00.119 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:41:00.119 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "done", 2024-03-11T21:41:00.120 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:33:17.752736+0000", 2024-03-11T21:41:00.120 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 4.7899999999999999e-05 2024-03-11T21:41:00.120 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-03-11T21:41:00.120 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-03-11T21:41:00.120 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-03-11T21:41:00.120 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:41:00.120 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:41:00.120 INFO:teuthology.orchestra.run.smithi092.stdout: "description": "osd_op(client.15150.0:107417 8.17 8:ed093fcf:::benchmark_data_smithi092_104865_object107416:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e192)", 2024-03-11T21:41:00.120 INFO:teuthology.orchestra.run.smithi092.stdout: "initiated_at": "2024-03-11T21:33:17.608265+0000", 2024-03-11T21:41:00.120 INFO:teuthology.orchestra.run.smithi092.stdout: "age": 462.49519372700001, 2024-03-11T21:41:00.120 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.14778530400000001, 2024-03-11T21:41:00.120 INFO:teuthology.orchestra.run.smithi092.stdout: "type_data": { 2024-03-11T21:41:00.120 INFO:teuthology.orchestra.run.smithi092.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-11T21:41:00.120 INFO:teuthology.orchestra.run.smithi092.stdout: "client_info": { 2024-03-11T21:41:00.121 INFO:teuthology.orchestra.run.smithi092.stdout: "client": "client.15150", 2024-03-11T21:41:00.121 INFO:teuthology.orchestra.run.smithi092.stdout: "client_addr": "v1:172.21.15.92:0/436606717", 2024-03-11T21:41:00.121 INFO:teuthology.orchestra.run.smithi092.stdout: "tid": 107417 2024-03-11T21:41:00.121 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:41:00.121 INFO:teuthology.orchestra.run.smithi092.stdout: "events": [ 2024-03-11T21:41:00.121 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:41:00.121 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "initiated", 2024-03-11T21:41:00.121 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:33:17.608265+0000", 2024-03-11T21:41:00.121 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-03-11T21:41:00.121 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:41:00.121 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:41:00.121 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "header_read", 2024-03-11T21:41:00.121 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:33:17.608265+0000", 2024-03-11T21:41:00.121 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-03-11T21:41:00.122 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:41:00.122 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:41:00.122 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "throttled", 2024-03-11T21:41:00.122 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:33:17.608267+0000", 2024-03-11T21:41:00.122 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.254e-06 2024-03-11T21:41:00.122 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:41:00.122 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:41:00.122 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "all_read", 2024-03-11T21:41:00.122 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:33:17.608287+0000", 2024-03-11T21:41:00.122 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 2.0678000000000001e-05 2024-03-11T21:41:00.122 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:41:00.122 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:41:00.122 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "dispatched", 2024-03-11T21:41:00.122 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:33:17.608297+0000", 2024-03-11T21:41:00.123 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 9.7580000000000004e-06 2024-03-11T21:41:00.123 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:41:00.123 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:41:00.123 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "queued_for_pg", 2024-03-11T21:41:00.123 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:33:17.608304+0000", 2024-03-11T21:41:00.123 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 7.1439999999999997e-06 2024-03-11T21:41:00.123 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:41:00.123 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:41:00.123 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-03-11T21:41:00.123 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:33:17.608364+0000", 2024-03-11T21:41:00.123 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 5.9524999999999997e-05 2024-03-11T21:41:00.123 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:41:00.123 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:41:00.123 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "started", 2024-03-11T21:41:00.124 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:33:17.608508+0000", 2024-03-11T21:41:00.124 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00014461699999999999 2024-03-11T21:41:00.124 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:41:00.124 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:41:00.124 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:41:00.125 INFO:teuthology.orchestra.run.smithi092.stdout:osd.1: {} 2024-03-11T21:41:00.126 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "waiting for subops from 0,2,3,6", 2024-03-11T21:41:00.126 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:33:17.608969+0000", 2024-03-11T21:41:00.126 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00046111699999999998 2024-03-11T21:41:00.126 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:41:00.126 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:41:00.126 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "op_commit", 2024-03-11T21:41:00.126 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:33:17.610759+0000", 2024-03-11T21:41:00.126 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.001789971 2024-03-11T21:41:00.126 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:41:00.126 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:41:00.127 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "sub_op_commit_rec", 2024-03-11T21:41:00.127 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:33:17.610972+0000", 2024-03-11T21:41:00.127 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.000213103 2024-03-11T21:41:00.127 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:41:00.127 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:41:00.127 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "sub_op_commit_rec", 2024-03-11T21:41:00.127 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:33:17.611916+0000", 2024-03-11T21:41:00.127 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00094380699999999996 2024-03-11T21:41:00.127 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:41:00.127 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:41:00.127 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "sub_op_commit_rec", 2024-03-11T21:41:00.127 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:33:17.612145+0000", 2024-03-11T21:41:00.127 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00022870400000000001 2024-03-11T21:41:00.127 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:41:00.128 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:41:00.128 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "sub_op_commit_rec", 2024-03-11T21:41:00.128 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:33:17.755847+0000", 2024-03-11T21:41:00.128 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.14370169099999999 2024-03-11T21:41:00.128 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:41:00.128 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:41:00.128 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "commit_sent", 2024-03-11T21:41:00.128 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:33:17.755910+0000", 2024-03-11T21:41:00.128 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 6.3010999999999997e-05 2024-03-11T21:41:00.128 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:41:00.128 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:41:00.128 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "done", 2024-03-11T21:41:00.128 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:33:17.756051+0000", 2024-03-11T21:41:00.128 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00014092400000000001 2024-03-11T21:41:00.129 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-03-11T21:41:00.129 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-03-11T21:41:00.129 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-03-11T21:41:00.129 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:41:00.129 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:41:00.129 INFO:teuthology.orchestra.run.smithi092.stdout: "description": "osd_op(client.25282.0:76305 9.4 9:24afab6b:::benchmark_data_smithi092_130291_object76304:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e277)", 2024-03-11T21:41:00.129 INFO:teuthology.orchestra.run.smithi092.stdout: "initiated_at": "2024-03-11T21:36:34.066675+0000", 2024-03-11T21:41:00.129 INFO:teuthology.orchestra.run.smithi092.stdout: "age": 266.03678391099999, 2024-03-11T21:41:00.129 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 2.4656751789999998, 2024-03-11T21:41:00.129 INFO:teuthology.orchestra.run.smithi092.stdout: "type_data": { 2024-03-11T21:41:00.129 INFO:teuthology.orchestra.run.smithi092.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-11T21:41:00.129 INFO:teuthology.orchestra.run.smithi092.stdout: "client_info": { 2024-03-11T21:41:00.129 INFO:teuthology.orchestra.run.smithi092.stdout: "client": "client.25282", 2024-03-11T21:41:00.129 INFO:teuthology.orchestra.run.smithi092.stdout: "client_addr": "v1:172.21.15.92:0/2489519821", 2024-03-11T21:41:00.130 INFO:teuthology.orchestra.run.smithi092.stdout: "tid": 76305 2024-03-11T21:41:00.130 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:41:00.130 INFO:teuthology.orchestra.run.smithi092.stdout: "events": [ 2024-03-11T21:41:00.130 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:41:00.130 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "initiated", 2024-03-11T21:41:00.130 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:36:34.066675+0000", 2024-03-11T21:41:00.130 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-03-11T21:41:00.130 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:41:00.130 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:41:00.130 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "header_read", 2024-03-11T21:41:00.130 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:36:34.066675+0000", 2024-03-11T21:41:00.130 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-03-11T21:41:00.130 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:41:00.130 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:41:00.131 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "throttled", 2024-03-11T21:41:00.131 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:36:34.066677+0000", 2024-03-11T21:41:00.131 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 2.148e-06 2024-03-11T21:41:00.131 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:41:00.131 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:41:00.131 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "all_read", 2024-03-11T21:41:00.131 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:36:34.066702+0000", 2024-03-11T21:41:00.131 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 2.4715999999999998e-05 2024-03-11T21:41:00.131 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:41:00.131 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:41:00.131 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "dispatched", 2024-03-11T21:41:00.131 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:36:34.066716+0000", 2024-03-11T21:41:00.131 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.3918e-05 2024-03-11T21:41:00.131 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:41:00.132 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:41:00.132 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "queued_for_pg", 2024-03-11T21:41:00.132 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:36:34.066725+0000", 2024-03-11T21:41:00.132 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 9.1849999999999993e-06 2024-03-11T21:41:00.132 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:41:00.132 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:41:00.132 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-03-11T21:41:00.132 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:36:34.066790+0000", 2024-03-11T21:41:00.132 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 6.4807999999999993e-05 2024-03-11T21:41:00.132 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:41:00.132 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:41:00.132 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "waiting for peered", 2024-03-11T21:41:00.132 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:36:34.066800+0000", 2024-03-11T21:41:00.132 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.0501e-05 2024-03-11T21:41:00.133 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:41:00.133 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:41:00.133 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-03-11T21:41:00.133 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:36:35.049363+0000", 2024-03-11T21:41:00.133 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.98256274399999999 2024-03-11T21:41:00.133 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:41:00.133 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:41:00.133 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "waiting for peered", 2024-03-11T21:41:00.133 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:36:35.049377+0000", 2024-03-11T21:41:00.133 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.4369000000000001e-05 2024-03-11T21:41:00.133 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:41:00.133 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:41:00.133 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-03-11T21:41:00.134 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:36:36.513910+0000", 2024-03-11T21:41:00.134 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.4645325759999999 2024-03-11T21:41:00.134 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:41:00.134 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:41:00.134 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "started", 2024-03-11T21:41:00.134 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:36:36.514088+0000", 2024-03-11T21:41:00.134 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00017830499999999999 2024-03-11T21:41:00.134 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:41:00.134 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:41:00.134 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "waiting for subops from 3,4", 2024-03-11T21:41:00.134 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:36:36.514279+0000", 2024-03-11T21:41:00.134 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.000191032 2024-03-11T21:41:00.134 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:41:00.134 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:41:00.135 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "op_commit", 2024-03-11T21:41:00.135 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:36:36.520031+0000", 2024-03-11T21:41:00.135 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.0057516420000000004 2024-03-11T21:41:00.135 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:41:00.135 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:41:00.135 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "sub_op_commit_rec", 2024-03-11T21:41:00.135 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:36:36.527515+0000", 2024-03-11T21:41:00.135 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.0074835020000000004 2024-03-11T21:41:00.135 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:41:00.135 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:41:00.135 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "sub_op_commit_rec", 2024-03-11T21:41:00.135 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:36:36.528312+0000", 2024-03-11T21:41:00.135 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00079732100000000003 2024-03-11T21:41:00.135 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:41:00.136 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:41:00.136 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "commit_sent", 2024-03-11T21:41:00.136 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:36:36.529068+0000", 2024-03-11T21:41:00.136 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00075650100000000003 2024-03-11T21:41:00.136 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:41:00.136 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:41:00.136 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "done", 2024-03-11T21:41:00.136 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:36:36.532350+0000", 2024-03-11T21:41:00.136 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.0032819110000000002 2024-03-11T21:41:00.136 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-03-11T21:41:00.136 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-03-11T21:41:00.136 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-03-11T21:41:00.136 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:41:00.136 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:41:00.137 INFO:teuthology.orchestra.run.smithi092.stdout: "description": "osd_op(client.25282.0:76293 9.2 9:470137e4:::benchmark_data_smithi092_130291_object76292:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e278)", 2024-03-11T21:41:00.137 INFO:teuthology.orchestra.run.smithi092.stdout: "initiated_at": "2024-03-11T21:36:35.047541+0000", 2024-03-11T21:41:00.137 INFO:teuthology.orchestra.run.smithi092.stdout: "age": 265.05591835199999, 2024-03-11T21:41:00.137 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.466307545, 2024-03-11T21:41:00.137 INFO:teuthology.orchestra.run.smithi092.stdout: "type_data": { 2024-03-11T21:41:00.137 INFO:teuthology.orchestra.run.smithi092.stdout: "flag_point": "reached pg", 2024-03-11T21:41:00.137 INFO:teuthology.orchestra.run.smithi092.stdout: "client_info": { 2024-03-11T21:41:00.137 INFO:teuthology.orchestra.run.smithi092.stdout: "client": "client.25282", 2024-03-11T21:41:00.137 INFO:teuthology.orchestra.run.smithi092.stdout: "client_addr": "v1:172.21.15.92:0/2489519821", 2024-03-11T21:41:00.137 INFO:teuthology.orchestra.run.smithi092.stdout: "tid": 76293 2024-03-11T21:41:00.137 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:41:00.137 INFO:teuthology.orchestra.run.smithi092.stdout: "events": [ 2024-03-11T21:41:00.137 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:41:00.138 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "initiated", 2024-03-11T21:41:00.138 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:36:35.047541+0000", 2024-03-11T21:41:00.138 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-03-11T21:41:00.138 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:41:00.138 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:41:00.138 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "header_read", 2024-03-11T21:41:00.138 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:36:35.047541+0000", 2024-03-11T21:41:00.138 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-03-11T21:41:00.138 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:41:00.138 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:41:00.138 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "throttled", 2024-03-11T21:41:00.138 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:36:35.047544+0000", 2024-03-11T21:41:00.138 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 3.0000000000000001e-06 2024-03-11T21:41:00.138 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:41:00.139 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:41:00.139 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "all_read", 2024-03-11T21:41:00.139 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:36:35.047590+0000", 2024-03-11T21:41:00.139 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 4.6313000000000002e-05 2024-03-11T21:41:00.139 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:41:00.139 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:41:00.139 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "dispatched", 2024-03-11T21:41:00.139 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:36:35.047605+0000", 2024-03-11T21:41:00.139 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.5004000000000001e-05 2024-03-11T21:41:00.139 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:41:00.139 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:41:00.139 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "queued_for_pg", 2024-03-11T21:41:00.139 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:36:35.047625+0000", 2024-03-11T21:41:00.139 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 2.0290000000000001e-05 2024-03-11T21:41:00.140 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:41:00.140 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:41:00.140 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-03-11T21:41:00.140 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:36:35.052040+0000", 2024-03-11T21:41:00.140 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.0044150500000000002 2024-03-11T21:41:00.140 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:41:00.140 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:41:00.140 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "waiting for peered", 2024-03-11T21:41:00.140 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:36:35.052052+0000", 2024-03-11T21:41:00.140 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.1242e-05 2024-03-11T21:41:00.140 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:41:00.140 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:41:00.140 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-03-11T21:41:00.141 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:36:36.513737+0000", 2024-03-11T21:41:00.141 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.4616853000000001 2024-03-11T21:41:00.141 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:41:00.141 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:41:00.141 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "done", 2024-03-11T21:41:00.141 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:36:36.513848+0000", 2024-03-11T21:41:00.141 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.000111346 2024-03-11T21:41:00.141 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-03-11T21:41:00.141 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-03-11T21:41:00.141 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-03-11T21:41:00.141 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:41:00.141 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:41:00.141 INFO:teuthology.orchestra.run.smithi092.stdout: "description": "osd_op(client.25282.0:76303 9.2 9:4fecfdb0:::benchmark_data_smithi092_130291_object76302:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e278)", 2024-03-11T21:41:00.141 INFO:teuthology.orchestra.run.smithi092.stdout: "initiated_at": "2024-03-11T21:36:35.047641+0000", 2024-03-11T21:41:00.142 INFO:teuthology.orchestra.run.smithi092.stdout: "age": 265.05581845, 2024-03-11T21:41:00.142 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.486106173, 2024-03-11T21:41:00.142 INFO:teuthology.orchestra.run.smithi092.stdout: "type_data": { 2024-03-11T21:41:00.142 INFO:teuthology.orchestra.run.smithi092.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-11T21:41:00.142 INFO:teuthology.orchestra.run.smithi092.stdout: "client_info": { 2024-03-11T21:41:00.142 INFO:teuthology.orchestra.run.smithi092.stdout: "client": "client.25282", 2024-03-11T21:41:00.142 INFO:teuthology.orchestra.run.smithi092.stdout: "client_addr": "v1:172.21.15.92:0/2489519821", 2024-03-11T21:41:00.142 INFO:teuthology.orchestra.run.smithi092.stdout: "tid": 76303 2024-03-11T21:41:00.142 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:41:00.142 INFO:teuthology.orchestra.run.smithi092.stdout: "events": [ 2024-03-11T21:41:00.142 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:41:00.142 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "initiated", 2024-03-11T21:41:00.143 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:36:35.047641+0000", 2024-03-11T21:41:00.143 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-03-11T21:41:00.143 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:41:00.143 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:41:00.143 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "header_read", 2024-03-11T21:41:00.143 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:36:35.047641+0000", 2024-03-11T21:41:00.143 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-03-11T21:41:00.143 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:41:00.143 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:41:00.143 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "throttled", 2024-03-11T21:41:00.143 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:36:35.047642+0000", 2024-03-11T21:41:00.143 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.02e-06 2024-03-11T21:41:00.143 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:41:00.143 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:41:00.144 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "all_read", 2024-03-11T21:41:00.144 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:36:35.047672+0000", 2024-03-11T21:41:00.144 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 3.0846000000000002e-05 2024-03-11T21:41:00.144 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:41:00.144 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:41:00.144 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "dispatched", 2024-03-11T21:41:00.144 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:36:35.047681+0000", 2024-03-11T21:41:00.144 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 8.123e-06 2024-03-11T21:41:00.144 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:41:00.144 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:41:00.144 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "queued_for_pg", 2024-03-11T21:41:00.144 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:36:35.047687+0000", 2024-03-11T21:41:00.144 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 6.2639999999999997e-06 2024-03-11T21:41:00.144 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:41:00.145 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:41:00.145 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-03-11T21:41:00.145 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:36:35.052079+0000", 2024-03-11T21:41:00.145 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.0043926099999999999 2024-03-11T21:41:00.145 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:41:00.145 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:41:00.145 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "waiting for peered", 2024-03-11T21:41:00.145 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:36:35.052087+0000", 2024-03-11T21:41:00.145 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 7.3440000000000003e-06 2024-03-11T21:41:00.145 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:41:00.145 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:41:00.145 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-03-11T21:41:00.146 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:36:36.516340+0000", 2024-03-11T21:41:00.146 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.4642529550000001 2024-03-11T21:41:00.146 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:41:00.146 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:41:00.146 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "started", 2024-03-11T21:41:00.146 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:36:36.516472+0000", 2024-03-11T21:41:00.146 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.000132256 2024-03-11T21:41:00.146 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:41:00.146 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:41:00.146 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "waiting for subops from 2,3", 2024-03-11T21:41:00.146 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:36:36.516588+0000", 2024-03-11T21:41:00.146 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00011624399999999999 2024-03-11T21:41:00.146 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:41:00.147 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:41:00.147 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "op_commit", 2024-03-11T21:41:00.147 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:36:36.523219+0000", 2024-03-11T21:41:00.147 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.0066303819999999998 2024-03-11T21:41:00.147 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:41:00.147 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:41:00.147 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "sub_op_commit_rec", 2024-03-11T21:41:00.147 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:36:36.527823+0000", 2024-03-11T21:41:00.147 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.0046046430000000003 2024-03-11T21:41:00.147 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:41:00.147 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:41:00.147 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "sub_op_commit_rec", 2024-03-11T21:41:00.147 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:36:36.527903+0000", 2024-03-11T21:41:00.147 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 7.9942000000000001e-05 2024-03-11T21:41:00.147 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:41:00.148 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:41:00.148 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "commit_sent", 2024-03-11T21:41:00.148 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:36:36.528983+0000", 2024-03-11T21:41:00.148 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.0010797770000000001 2024-03-11T21:41:00.148 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:41:00.148 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:41:00.148 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "done", 2024-03-11T21:41:00.148 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:36:36.533747+0000", 2024-03-11T21:41:00.148 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.0047637670000000003 2024-03-11T21:41:00.148 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-03-11T21:41:00.148 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-03-11T21:41:00.148 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-03-11T21:41:00.148 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:41:00.148 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:41:00.149 INFO:teuthology.orchestra.run.smithi092.stdout: "description": "osd_op(client.25282.0:76304 9.2 9:40dc0b9f:::benchmark_data_smithi092_130291_object76303:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e278)", 2024-03-11T21:41:00.149 INFO:teuthology.orchestra.run.smithi092.stdout: "initiated_at": "2024-03-11T21:36:35.047696+0000", 2024-03-11T21:41:00.149 INFO:teuthology.orchestra.run.smithi092.stdout: "age": 265.055763367, 2024-03-11T21:41:00.149 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.4932335409999999, 2024-03-11T21:41:00.149 INFO:teuthology.orchestra.run.smithi092.stdout: "type_data": { 2024-03-11T21:41:00.149 INFO:teuthology.orchestra.run.smithi092.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-11T21:41:00.149 INFO:teuthology.orchestra.run.smithi092.stdout: "client_info": { 2024-03-11T21:41:00.149 INFO:teuthology.orchestra.run.smithi092.stdout: "client": "client.25282", 2024-03-11T21:41:00.149 INFO:teuthology.orchestra.run.smithi092.stdout: "client_addr": "v1:172.21.15.92:0/2489519821", 2024-03-11T21:41:00.149 INFO:teuthology.orchestra.run.smithi092.stdout: "tid": 76304 2024-03-11T21:41:00.149 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:41:00.149 INFO:teuthology.orchestra.run.smithi092.stdout: "events": [ 2024-03-11T21:41:00.149 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:41:00.149 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "initiated", 2024-03-11T21:41:00.150 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:36:35.047696+0000", 2024-03-11T21:41:00.150 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-03-11T21:41:00.150 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:41:00.150 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:41:00.150 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "header_read", 2024-03-11T21:41:00.150 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:36:35.047696+0000", 2024-03-11T21:41:00.150 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-03-11T21:41:00.150 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:41:00.150 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:41:00.150 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "throttled", 2024-03-11T21:41:00.150 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:36:35.047696+0000", 2024-03-11T21:41:00.150 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 7.23e-07 2024-03-11T21:41:00.150 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:41:00.150 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:41:00.151 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "all_read", 2024-03-11T21:41:00.151 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:36:35.047723+0000", 2024-03-11T21:41:00.151 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 2.6811000000000001e-05 2024-03-11T21:41:00.151 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:41:00.151 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:41:00.151 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "dispatched", 2024-03-11T21:41:00.151 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:36:35.047731+0000", 2024-03-11T21:41:00.151 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 7.7700000000000001e-06 2024-03-11T21:41:00.151 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:41:00.151 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:41:00.151 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "queued_for_pg", 2024-03-11T21:41:00.151 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:36:35.047737+0000", 2024-03-11T21:41:00.151 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 6.2260000000000002e-06 2024-03-11T21:41:00.152 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:41:00.152 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:41:00.152 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-03-11T21:41:00.152 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:36:35.052114+0000", 2024-03-11T21:41:00.152 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.004376698 2024-03-11T21:41:00.152 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:41:00.152 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:41:00.152 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "waiting for peered", 2024-03-11T21:41:00.152 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:36:35.052121+0000", 2024-03-11T21:41:00.152 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 6.832e-06 2024-03-11T21:41:00.152 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:41:00.152 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:41:00.152 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-03-11T21:41:00.152 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:36:36.519831+0000", 2024-03-11T21:41:00.153 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.467709943 2024-03-11T21:41:00.153 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:41:00.153 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:41:00.153 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "started", 2024-03-11T21:41:00.153 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:36:36.520062+0000", 2024-03-11T21:41:00.153 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00023085900000000001 2024-03-11T21:41:00.153 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:41:00.153 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:41:00.153 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "waiting for subops from 2,3", 2024-03-11T21:41:00.153 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:36:36.520266+0000", 2024-03-11T21:41:00.153 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00020451599999999999 2024-03-11T21:41:00.153 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:41:00.153 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:41:00.153 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "op_commit", 2024-03-11T21:41:00.154 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:36:36.526472+0000", 2024-03-11T21:41:00.154 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.0062056050000000003 2024-03-11T21:41:00.154 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:41:00.154 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:41:00.154 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "sub_op_commit_rec", 2024-03-11T21:41:00.154 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:36:36.534867+0000", 2024-03-11T21:41:00.154 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.0083950359999999998 2024-03-11T21:41:00.154 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:41:00.154 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:41:00.154 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "sub_op_commit_rec", 2024-03-11T21:41:00.154 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:36:36.537251+0000", 2024-03-11T21:41:00.154 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.0023839130000000001 2024-03-11T21:41:00.154 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:41:00.154 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:41:00.155 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "commit_sent", 2024-03-11T21:41:00.155 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:36:36.537314+0000", 2024-03-11T21:41:00.155 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 6.2983000000000003e-05 2024-03-11T21:41:00.155 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:41:00.155 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:41:00.155 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "done", 2024-03-11T21:41:00.155 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:36:36.540929+0000", 2024-03-11T21:41:00.155 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.0036156259999999998 2024-03-11T21:41:00.155 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-03-11T21:41:00.155 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-03-11T21:41:00.155 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-03-11T21:41:00.155 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:41:00.155 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:41:00.155 INFO:teuthology.orchestra.run.smithi092.stdout: "description": "osd_op(client.25282.0:174715 9.4 9:29fe388f:::benchmark_data_smithi092_130291_object58331:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e323)", 2024-03-11T21:41:00.156 INFO:teuthology.orchestra.run.smithi092.stdout: "initiated_at": "2024-03-11T21:38:42.796916+0000", 2024-03-11T21:41:00.156 INFO:teuthology.orchestra.run.smithi092.stdout: "age": 137.30654347399999, 2024-03-11T21:41:00.156 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.20056705699999999, 2024-03-11T21:41:00.156 INFO:teuthology.orchestra.run.smithi092.stdout: "type_data": { 2024-03-11T21:41:00.156 INFO:teuthology.orchestra.run.smithi092.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-11T21:41:00.156 INFO:teuthology.orchestra.run.smithi092.stdout: "client_info": { 2024-03-11T21:41:00.156 INFO:teuthology.orchestra.run.smithi092.stdout: "client": "client.25282", 2024-03-11T21:41:00.156 INFO:teuthology.orchestra.run.smithi092.stdout: "client_addr": "v1:172.21.15.92:0/2489519821", 2024-03-11T21:41:00.156 INFO:teuthology.orchestra.run.smithi092.stdout: "tid": 174715 2024-03-11T21:41:00.156 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:41:00.156 INFO:teuthology.orchestra.run.smithi092.stdout: "events": [ 2024-03-11T21:41:00.156 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:41:00.156 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "initiated", 2024-03-11T21:41:00.156 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:38:42.796916+0000", 2024-03-11T21:41:00.157 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-03-11T21:41:00.157 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:41:00.157 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:41:00.157 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "header_read", 2024-03-11T21:41:00.157 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:38:42.796916+0000", 2024-03-11T21:41:00.157 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-03-11T21:41:00.157 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:41:00.157 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:41:00.157 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "throttled", 2024-03-11T21:41:00.157 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:38:42.796918+0000", 2024-03-11T21:41:00.157 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 2.0379999999999998e-06 2024-03-11T21:41:00.157 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:41:00.157 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:41:00.157 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "all_read", 2024-03-11T21:41:00.158 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:38:42.796925+0000", 2024-03-11T21:41:00.158 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 6.9530000000000002e-06 2024-03-11T21:41:00.158 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:41:00.158 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:41:00.158 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "dispatched", 2024-03-11T21:41:00.158 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:38:42.796938+0000", 2024-03-11T21:41:00.158 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.3485e-05 2024-03-11T21:41:00.158 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:41:00.158 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:41:00.158 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "queued_for_pg", 2024-03-11T21:41:00.158 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:38:42.796947+0000", 2024-03-11T21:41:00.158 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 8.8200000000000003e-06 2024-03-11T21:41:00.158 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:41:00.158 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:41:00.159 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-03-11T21:41:00.159 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:38:42.810477+0000", 2024-03-11T21:41:00.159 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.013530179 2024-03-11T21:41:00.159 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:41:00.159 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:41:00.159 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "waiting for rw locks", 2024-03-11T21:41:00.159 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:38:42.810609+0000", 2024-03-11T21:41:00.159 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00013229100000000001 2024-03-11T21:41:00.159 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:41:00.159 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:41:00.159 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-03-11T21:41:00.159 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:38:42.846030+0000", 2024-03-11T21:41:00.159 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.035421033999999997 2024-03-11T21:41:00.160 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:41:00.160 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:41:00.160 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "started", 2024-03-11T21:41:00.160 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:38:42.846135+0000", 2024-03-11T21:41:00.160 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.000104394 2024-03-11T21:41:00.160 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:41:00.160 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:41:00.160 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "waiting for subops from 3,4", 2024-03-11T21:41:00.160 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:38:42.846284+0000", 2024-03-11T21:41:00.160 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00014913700000000001 2024-03-11T21:41:00.160 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:41:00.160 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:41:00.160 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "sub_op_commit_rec", 2024-03-11T21:41:00.160 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:38:42.859187+0000", 2024-03-11T21:41:00.161 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.012903379 2024-03-11T21:41:00.161 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:41:00.161 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:41:00.161 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "op_commit", 2024-03-11T21:41:00.161 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:38:42.859201+0000", 2024-03-11T21:41:00.161 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.3329999999999999e-05 2024-03-11T21:41:00.161 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:41:00.161 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:41:00.161 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "sub_op_commit_rec", 2024-03-11T21:41:00.161 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:38:42.991126+0000", 2024-03-11T21:41:00.161 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.13192508999999999 2024-03-11T21:41:00.161 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:41:00.161 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:41:00.162 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "commit_sent", 2024-03-11T21:41:00.162 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:38:42.991672+0000", 2024-03-11T21:41:00.162 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00054652900000000003 2024-03-11T21:41:00.162 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:41:00.162 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:41:00.162 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "done", 2024-03-11T21:41:00.162 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:38:42.997483+0000", 2024-03-11T21:41:00.162 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.0058103979999999996 2024-03-11T21:41:00.162 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-03-11T21:41:00.162 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-03-11T21:41:00.162 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-03-11T21:41:00.162 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:41:00.162 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:41:00.163 INFO:teuthology.orchestra.run.smithi092.stdout: "description": "osd_op(client.25282.0:174738 9.4 9:2d417731:::benchmark_data_smithi092_130291_object58354:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e323)", 2024-03-11T21:41:00.163 INFO:teuthology.orchestra.run.smithi092.stdout: "initiated_at": "2024-03-11T21:38:42.826461+0000", 2024-03-11T21:41:00.163 INFO:teuthology.orchestra.run.smithi092.stdout: "age": 137.276997997, 2024-03-11T21:41:00.163 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.158588329, 2024-03-11T21:41:00.163 INFO:teuthology.orchestra.run.smithi092.stdout: "type_data": { 2024-03-11T21:41:00.163 INFO:teuthology.orchestra.run.smithi092.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-11T21:41:00.163 INFO:teuthology.orchestra.run.smithi092.stdout: "client_info": { 2024-03-11T21:41:00.163 INFO:teuthology.orchestra.run.smithi092.stdout: "client": "client.25282", 2024-03-11T21:41:00.163 INFO:teuthology.orchestra.run.smithi092.stdout: "client_addr": "v1:172.21.15.92:0/2489519821", 2024-03-11T21:41:00.163 INFO:teuthology.orchestra.run.smithi092.stdout: "tid": 174738 2024-03-11T21:41:00.163 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:41:00.163 INFO:teuthology.orchestra.run.smithi092.stdout: "events": [ 2024-03-11T21:41:00.163 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:41:00.164 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "initiated", 2024-03-11T21:41:00.164 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:38:42.826461+0000", 2024-03-11T21:41:00.164 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-03-11T21:41:00.164 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:41:00.164 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:41:00.164 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "header_read", 2024-03-11T21:41:00.164 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:38:42.826461+0000", 2024-03-11T21:41:00.164 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-03-11T21:41:00.164 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:41:00.164 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:41:00.164 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "throttled", 2024-03-11T21:41:00.164 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:38:42.826462+0000", 2024-03-11T21:41:00.164 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 6.3499999999999996e-07 2024-03-11T21:41:00.165 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:41:00.165 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:41:00.165 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "all_read", 2024-03-11T21:41:00.165 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:38:42.826465+0000", 2024-03-11T21:41:00.165 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 2.8590000000000001e-06 2024-03-11T21:41:00.165 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:41:00.165 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:41:00.165 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "dispatched", 2024-03-11T21:41:00.165 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:38:42.826470+0000", 2024-03-11T21:41:00.165 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 5.6790000000000002e-06 2024-03-11T21:41:00.165 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:41:00.165 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:41:00.165 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "queued_for_pg", 2024-03-11T21:41:00.165 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:38:42.826475+0000", 2024-03-11T21:41:00.166 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 5.1100000000000002e-06 2024-03-11T21:41:00.166 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:41:00.166 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:41:00.166 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-03-11T21:41:00.166 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:38:42.834833+0000", 2024-03-11T21:41:00.166 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.0083580219999999997 2024-03-11T21:41:00.166 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:41:00.166 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:41:00.166 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "started", 2024-03-11T21:41:00.166 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:38:42.834986+0000", 2024-03-11T21:41:00.166 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00015255499999999999 2024-03-11T21:41:00.166 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:41:00.166 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:41:00.167 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "waiting for subops from 3,4", 2024-03-11T21:41:00.167 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:38:42.835147+0000", 2024-03-11T21:41:00.167 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00016076999999999999 2024-03-11T21:41:00.167 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:41:00.167 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:41:00.167 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "op_commit", 2024-03-11T21:41:00.167 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:38:42.849119+0000", 2024-03-11T21:41:00.167 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.013971957 2024-03-11T21:41:00.167 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:41:00.168 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:41:00.168 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "sub_op_commit_rec", 2024-03-11T21:41:00.168 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:38:42.849163+0000", 2024-03-11T21:41:00.168 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 4.4304999999999997e-05 2024-03-11T21:41:00.168 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:41:00.168 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:41:00.168 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "sub_op_commit_rec", 2024-03-11T21:41:00.168 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:38:42.982532+0000", 2024-03-11T21:41:00.169 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.13336952699999999 2024-03-11T21:41:00.169 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:41:00.169 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:41:00.169 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "commit_sent", 2024-03-11T21:41:00.169 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:38:42.982727+0000", 2024-03-11T21:41:00.169 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00019427100000000001 2024-03-11T21:41:00.169 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:41:00.169 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:41:00.169 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "done", 2024-03-11T21:41:00.169 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:38:42.985049+0000", 2024-03-11T21:41:00.169 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.002322639 2024-03-11T21:41:00.169 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-03-11T21:41:00.169 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-03-11T21:41:00.169 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-03-11T21:41:00.170 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:41:00.170 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:41:00.170 INFO:teuthology.orchestra.run.smithi092.stdout: "description": "osd_op(client.25282.0:174742 9.2 9:41360745:::benchmark_data_smithi092_130291_object58358:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e323)", 2024-03-11T21:41:00.170 INFO:teuthology.orchestra.run.smithi092.stdout: "initiated_at": "2024-03-11T21:38:42.833809+0000", 2024-03-11T21:41:00.170 INFO:teuthology.orchestra.run.smithi092.stdout: "age": 137.26965003999999, 2024-03-11T21:41:00.170 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.151544015, 2024-03-11T21:41:00.170 INFO:teuthology.orchestra.run.smithi092.stdout: "type_data": { 2024-03-11T21:41:00.170 INFO:teuthology.orchestra.run.smithi092.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-11T21:41:00.170 INFO:teuthology.orchestra.run.smithi092.stdout: "client_info": { 2024-03-11T21:41:00.170 INFO:teuthology.orchestra.run.smithi092.stdout: "client": "client.25282", 2024-03-11T21:41:00.170 INFO:teuthology.orchestra.run.smithi092.stdout: "client_addr": "v1:172.21.15.92:0/2489519821", 2024-03-11T21:41:00.170 INFO:teuthology.orchestra.run.smithi092.stdout: "tid": 174742 2024-03-11T21:41:00.171 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:41:00.171 INFO:teuthology.orchestra.run.smithi092.stdout: "events": [ 2024-03-11T21:41:00.171 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:41:00.171 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "initiated", 2024-03-11T21:41:00.171 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:38:42.833809+0000", 2024-03-11T21:41:00.171 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-03-11T21:41:00.171 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:41:00.171 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:41:00.171 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "header_read", 2024-03-11T21:41:00.171 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:38:42.833809+0000", 2024-03-11T21:41:00.171 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-03-11T21:41:00.171 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:41:00.171 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:41:00.171 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "throttled", 2024-03-11T21:41:00.172 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:38:42.833811+0000", 2024-03-11T21:41:00.172 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.9240000000000001e-06 2024-03-11T21:41:00.172 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:41:00.172 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:41:00.172 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "all_read", 2024-03-11T21:41:00.172 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:38:42.833818+0000", 2024-03-11T21:41:00.172 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 6.8809999999999998e-06 2024-03-11T21:41:00.172 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:41:00.172 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:41:00.172 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "dispatched", 2024-03-11T21:41:00.172 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:38:42.833830+0000", 2024-03-11T21:41:00.172 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.2009e-05 2024-03-11T21:41:00.172 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:41:00.172 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:41:00.172 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "queued_for_pg", 2024-03-11T21:41:00.173 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:38:42.833838+0000", 2024-03-11T21:41:00.173 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 8.4290000000000001e-06 2024-03-11T21:41:00.173 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:41:00.173 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:41:00.173 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-03-11T21:41:00.173 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:38:42.838152+0000", 2024-03-11T21:41:00.173 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.0043141890000000004 2024-03-11T21:41:00.173 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:41:00.173 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:41:00.173 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "waiting for rw locks", 2024-03-11T21:41:00.173 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:38:42.838255+0000", 2024-03-11T21:41:00.173 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.000102304 2024-03-11T21:41:00.173 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:41:00.173 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:41:00.174 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-03-11T21:41:00.174 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:38:42.850432+0000", 2024-03-11T21:41:00.174 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.012177273000000001 2024-03-11T21:41:00.174 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:41:00.174 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:41:00.174 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "started", 2024-03-11T21:41:00.174 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:38:42.850523+0000", 2024-03-11T21:41:00.174 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 9.1278000000000003e-05 2024-03-11T21:41:00.174 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:41:00.174 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:41:00.174 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "waiting for subops from 2,3", 2024-03-11T21:41:00.174 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:38:42.850658+0000", 2024-03-11T21:41:00.174 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.000134361 2024-03-11T21:41:00.174 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:41:00.175 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:41:00.175 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "sub_op_commit_rec", 2024-03-11T21:41:00.175 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:38:42.983132+0000", 2024-03-11T21:41:00.175 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.132474329 2024-03-11T21:41:00.175 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:41:00.175 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:41:00.175 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "sub_op_commit_rec", 2024-03-11T21:41:00.175 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:38:42.983190+0000", 2024-03-11T21:41:00.175 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 5.7692999999999998e-05 2024-03-11T21:41:00.175 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:41:00.175 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:41:00.175 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "op_commit", 2024-03-11T21:41:00.175 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:38:42.983203+0000", 2024-03-11T21:41:00.175 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.3841e-05 2024-03-11T21:41:00.175 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:41:00.176 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:41:00.176 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "commit_sent", 2024-03-11T21:41:00.176 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:38:42.983257+0000", 2024-03-11T21:41:00.176 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 5.3962000000000003e-05 2024-03-11T21:41:00.176 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:41:00.176 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:41:00.176 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "done", 2024-03-11T21:41:00.176 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:38:42.985353+0000", 2024-03-11T21:41:00.176 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.0020955420000000002 2024-03-11T21:41:00.176 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-03-11T21:41:00.176 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-03-11T21:41:00.176 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-03-11T21:41:00.176 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:41:00.176 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:41:00.176 INFO:teuthology.orchestra.run.smithi092.stdout: "description": "osd_op(client.25282.0:174767 9.4 9:25543c18:::benchmark_data_smithi092_130291_object58383:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e323)", 2024-03-11T21:41:00.176 INFO:teuthology.orchestra.run.smithi092.stdout: "initiated_at": "2024-03-11T21:38:42.860879+0000", 2024-03-11T21:41:00.177 INFO:teuthology.orchestra.run.smithi092.stdout: "age": 137.24257986500001, 2024-03-11T21:41:00.177 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.175225782, 2024-03-11T21:41:00.177 INFO:teuthology.orchestra.run.smithi092.stdout: "type_data": { 2024-03-11T21:41:00.177 INFO:teuthology.orchestra.run.smithi092.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-11T21:41:00.177 INFO:teuthology.orchestra.run.smithi092.stdout: "client_info": { 2024-03-11T21:41:00.177 INFO:teuthology.orchestra.run.smithi092.stdout: "client": "client.25282", 2024-03-11T21:41:00.177 INFO:teuthology.orchestra.run.smithi092.stdout: "client_addr": "v1:172.21.15.92:0/2489519821", 2024-03-11T21:41:00.177 INFO:teuthology.orchestra.run.smithi092.stdout: "tid": 174767 2024-03-11T21:41:00.177 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:41:00.177 INFO:teuthology.orchestra.run.smithi092.stdout: "events": [ 2024-03-11T21:41:00.177 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:41:00.177 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "initiated", 2024-03-11T21:41:00.177 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:38:42.860879+0000", 2024-03-11T21:41:00.177 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-03-11T21:41:00.177 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:41:00.178 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:41:00.178 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "header_read", 2024-03-11T21:41:00.178 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:38:42.860879+0000", 2024-03-11T21:41:00.178 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-03-11T21:41:00.178 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:41:00.178 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:41:00.178 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "throttled", 2024-03-11T21:41:00.178 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:38:42.860882+0000", 2024-03-11T21:41:00.178 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 2.3599999999999999e-06 2024-03-11T21:41:00.178 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:41:00.178 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:41:00.178 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "all_read", 2024-03-11T21:41:00.178 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:38:42.860887+0000", 2024-03-11T21:41:00.178 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 5.8379999999999998e-06 2024-03-11T21:41:00.179 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:41:00.179 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:41:00.179 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "dispatched", 2024-03-11T21:41:00.179 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:38:42.979888+0000", 2024-03-11T21:41:00.179 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.119001077 2024-03-11T21:41:00.179 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:41:00.179 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:41:00.179 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "queued_for_pg", 2024-03-11T21:41:00.179 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:38:42.979902+0000", 2024-03-11T21:41:00.179 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.3939e-05 2024-03-11T21:41:00.179 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:41:00.179 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:41:00.179 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-03-11T21:41:00.179 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:38:42.979967+0000", 2024-03-11T21:41:00.180 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 6.4906999999999994e-05 2024-03-11T21:41:00.180 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:41:00.180 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:41:00.180 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "waiting for rw locks", 2024-03-11T21:41:00.180 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:38:42.980104+0000", 2024-03-11T21:41:00.180 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00013705999999999999 2024-03-11T21:41:00.180 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:41:00.180 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:41:00.180 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-03-11T21:41:00.180 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:38:43.013851+0000", 2024-03-11T21:41:00.180 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.033747045000000003 2024-03-11T21:41:00.180 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:41:00.180 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:41:00.180 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "started", 2024-03-11T21:41:00.180 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:38:43.013954+0000", 2024-03-11T21:41:00.181 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.000102849 2024-03-11T21:41:00.181 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:41:00.181 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:41:00.181 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "waiting for subops from 3,4", 2024-03-11T21:41:00.181 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:38:43.014114+0000", 2024-03-11T21:41:00.181 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00016013400000000001 2024-03-11T21:41:00.181 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:41:00.181 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:41:00.181 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "op_commit", 2024-03-11T21:41:00.181 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:38:43.020120+0000", 2024-03-11T21:41:00.182 INFO:teuthology.orchestra.run.smithi092.stdout:osd.2: {} 2024-03-11T21:41:00.182 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.0060053179999999999 2024-03-11T21:41:00.182 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:41:00.182 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:41:00.182 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "sub_op_commit_rec", 2024-03-11T21:41:00.182 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:38:43.022357+0000", 2024-03-11T21:41:00.183 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.0022377360000000002 2024-03-11T21:41:00.183 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:41:00.183 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:41:00.183 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "sub_op_commit_rec", 2024-03-11T21:41:00.183 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:38:43.033609+0000", 2024-03-11T21:41:00.183 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.011252028000000001 2024-03-11T21:41:00.183 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:41:00.183 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:41:00.183 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "commit_sent", 2024-03-11T21:41:00.183 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:38:43.033658+0000", 2024-03-11T21:41:00.183 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 4.8276999999999999e-05 2024-03-11T21:41:00.183 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:41:00.183 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:41:00.183 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "done", 2024-03-11T21:41:00.184 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:38:43.036105+0000", 2024-03-11T21:41:00.184 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.0024472140000000001 2024-03-11T21:41:00.184 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-03-11T21:41:00.184 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-03-11T21:41:00.184 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-03-11T21:41:00.184 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:41:00.184 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:41:00.184 INFO:teuthology.orchestra.run.smithi092.stdout: "description": "osd_op(client.25282.0:175028 9.4 9:26b7b4b2:::benchmark_data_smithi092_130291_object58644:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e323)", 2024-03-11T21:41:00.184 INFO:teuthology.orchestra.run.smithi092.stdout: "initiated_at": "2024-03-11T21:38:43.326699+0000", 2024-03-11T21:41:00.184 INFO:teuthology.orchestra.run.smithi092.stdout: "age": 136.776759889, 2024-03-11T21:41:00.184 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.15555291099999999, 2024-03-11T21:41:00.184 INFO:teuthology.orchestra.run.smithi092.stdout: "type_data": { 2024-03-11T21:41:00.184 INFO:teuthology.orchestra.run.smithi092.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-11T21:41:00.184 INFO:teuthology.orchestra.run.smithi092.stdout: "client_info": { 2024-03-11T21:41:00.184 INFO:teuthology.orchestra.run.smithi092.stdout: "client": "client.25282", 2024-03-11T21:41:00.185 INFO:teuthology.orchestra.run.smithi092.stdout: "client_addr": "v1:172.21.15.92:0/2489519821", 2024-03-11T21:41:00.185 INFO:teuthology.orchestra.run.smithi092.stdout: "tid": 175028 2024-03-11T21:41:00.185 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:41:00.185 INFO:teuthology.orchestra.run.smithi092.stdout: "events": [ 2024-03-11T21:41:00.185 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:41:00.185 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "initiated", 2024-03-11T21:41:00.185 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:38:43.326699+0000", 2024-03-11T21:41:00.185 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-03-11T21:41:00.185 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:41:00.185 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:41:00.185 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "header_read", 2024-03-11T21:41:00.185 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:38:43.326699+0000", 2024-03-11T21:41:00.185 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-03-11T21:41:00.186 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:41:00.186 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:41:00.186 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "throttled", 2024-03-11T21:41:00.186 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:38:43.326701+0000", 2024-03-11T21:41:00.186 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 2.3e-06 2024-03-11T21:41:00.186 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:41:00.186 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:41:00.186 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "all_read", 2024-03-11T21:41:00.186 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:38:43.326707+0000", 2024-03-11T21:41:00.186 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 5.6679999999999998e-06 2024-03-11T21:41:00.186 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:41:00.186 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:41:00.186 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "dispatched", 2024-03-11T21:41:00.186 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:38:43.326719+0000", 2024-03-11T21:41:00.187 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.237e-05 2024-03-11T21:41:00.187 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:41:00.187 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:41:00.187 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "queued_for_pg", 2024-03-11T21:41:00.187 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:38:43.326728+0000", 2024-03-11T21:41:00.187 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 8.0569999999999993e-06 2024-03-11T21:41:00.187 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:41:00.187 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:41:00.187 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-03-11T21:41:00.187 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:38:43.328798+0000", 2024-03-11T21:41:00.187 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.002070929 2024-03-11T21:41:00.187 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:41:00.187 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:41:00.187 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "waiting for rw locks", 2024-03-11T21:41:00.188 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:38:43.328938+0000", 2024-03-11T21:41:00.188 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00013944399999999999 2024-03-11T21:41:00.188 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:41:00.188 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:41:00.188 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-03-11T21:41:00.188 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:38:43.455044+0000", 2024-03-11T21:41:00.188 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.12610570500000001 2024-03-11T21:41:00.188 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:41:00.188 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:41:00.188 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "started", 2024-03-11T21:41:00.188 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:38:43.455146+0000", 2024-03-11T21:41:00.188 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00010265200000000001 2024-03-11T21:41:00.188 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:41:00.189 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:41:00.189 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "waiting for subops from 3,4", 2024-03-11T21:41:00.189 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:38:43.455291+0000", 2024-03-11T21:41:00.189 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00014481100000000001 2024-03-11T21:41:00.189 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:41:00.189 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:41:00.189 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "sub_op_commit_rec", 2024-03-11T21:41:00.189 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:38:43.459945+0000", 2024-03-11T21:41:00.190 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.0046541849999999999 2024-03-11T21:41:00.190 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:41:00.190 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:41:00.190 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "op_commit", 2024-03-11T21:41:00.190 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:38:43.459961+0000", 2024-03-11T21:41:00.190 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.5622000000000001e-05 2024-03-11T21:41:00.190 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:41:00.190 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:41:00.190 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "sub_op_commit_rec", 2024-03-11T21:41:00.190 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:38:43.479848+0000", 2024-03-11T21:41:00.190 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.019886640000000001 2024-03-11T21:41:00.190 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:41:00.190 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:41:00.191 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "commit_sent", 2024-03-11T21:41:00.191 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:38:43.479901+0000", 2024-03-11T21:41:00.191 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 5.3137999999999998e-05 2024-03-11T21:41:00.191 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:41:00.191 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:41:00.191 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "done", 2024-03-11T21:41:00.191 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:38:43.482252+0000", 2024-03-11T21:41:00.191 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.0023513900000000001 2024-03-11T21:41:00.191 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-03-11T21:41:00.191 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-03-11T21:41:00.191 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-03-11T21:41:00.191 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:41:00.191 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:41:00.191 INFO:teuthology.orchestra.run.smithi092.stdout: "description": "osd_op(client.25282.0:179812 9.4 9:209c45ae:::benchmark_data_smithi092_130291_object63428:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e325)", 2024-03-11T21:41:00.191 INFO:teuthology.orchestra.run.smithi092.stdout: "initiated_at": "2024-03-11T21:38:49.604815+0000", 2024-03-11T21:41:00.192 INFO:teuthology.orchestra.run.smithi092.stdout: "age": 130.49864355599999, 2024-03-11T21:41:00.192 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.15039382700000001, 2024-03-11T21:41:00.192 INFO:teuthology.orchestra.run.smithi092.stdout: "type_data": { 2024-03-11T21:41:00.192 INFO:teuthology.orchestra.run.smithi092.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-11T21:41:00.192 INFO:teuthology.orchestra.run.smithi092.stdout: "client_info": { 2024-03-11T21:41:00.192 INFO:teuthology.orchestra.run.smithi092.stdout: "client": "client.25282", 2024-03-11T21:41:00.192 INFO:teuthology.orchestra.run.smithi092.stdout: "client_addr": "v1:172.21.15.92:0/2489519821", 2024-03-11T21:41:00.192 INFO:teuthology.orchestra.run.smithi092.stdout: "tid": 179812 2024-03-11T21:41:00.192 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:41:00.192 INFO:teuthology.orchestra.run.smithi092.stdout: "events": [ 2024-03-11T21:41:00.193 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:41:00.193 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "initiated", 2024-03-11T21:41:00.193 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:38:49.604815+0000", 2024-03-11T21:41:00.193 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-03-11T21:41:00.193 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:41:00.193 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:41:00.193 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "header_read", 2024-03-11T21:41:00.193 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:38:49.604815+0000", 2024-03-11T21:41:00.193 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-03-11T21:41:00.193 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:41:00.193 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:41:00.193 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "throttled", 2024-03-11T21:41:00.193 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:38:49.604817+0000", 2024-03-11T21:41:00.194 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.61e-06 2024-03-11T21:41:00.194 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:41:00.194 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:41:00.194 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "all_read", 2024-03-11T21:41:00.194 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:38:49.604821+0000", 2024-03-11T21:41:00.194 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 4.3780000000000003e-06 2024-03-11T21:41:00.194 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:41:00.194 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:41:00.194 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "dispatched", 2024-03-11T21:41:00.194 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:38:49.604829+0000", 2024-03-11T21:41:00.194 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 7.2960000000000001e-06 2024-03-11T21:41:00.194 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:41:00.194 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:41:00.195 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "queued_for_pg", 2024-03-11T21:41:00.195 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:38:49.604834+0000", 2024-03-11T21:41:00.195 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 5.6189999999999999e-06 2024-03-11T21:41:00.195 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:41:00.195 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:41:00.195 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-03-11T21:41:00.195 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:38:49.616852+0000", 2024-03-11T21:41:00.195 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.012017753000000001 2024-03-11T21:41:00.195 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:41:00.195 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:41:00.195 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "waiting for rw locks", 2024-03-11T21:41:00.195 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:38:49.616966+0000", 2024-03-11T21:41:00.195 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.000114268 2024-03-11T21:41:00.195 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:41:00.196 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:41:00.196 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-03-11T21:41:00.196 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:38:49.727165+0000", 2024-03-11T21:41:00.196 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.110198646 2024-03-11T21:41:00.196 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:41:00.196 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:41:00.196 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "started", 2024-03-11T21:41:00.196 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:38:49.727258+0000", 2024-03-11T21:41:00.196 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 9.3012999999999999e-05 2024-03-11T21:41:00.196 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:41:00.196 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:41:00.196 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "waiting for subops from 3,4", 2024-03-11T21:41:00.196 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:38:49.727424+0000", 2024-03-11T21:41:00.196 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.000165864 2024-03-11T21:41:00.196 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:41:00.197 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:41:00.197 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "sub_op_commit_rec", 2024-03-11T21:41:00.197 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:38:49.736962+0000", 2024-03-11T21:41:00.197 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.0095376149999999993 2024-03-11T21:41:00.197 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:41:00.197 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:41:00.197 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "op_commit", 2024-03-11T21:41:00.197 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:38:49.752984+0000", 2024-03-11T21:41:00.197 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.016022014000000001 2024-03-11T21:41:00.197 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:41:00.197 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:41:00.197 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "sub_op_commit_rec", 2024-03-11T21:41:00.197 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:38:49.753089+0000", 2024-03-11T21:41:00.197 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.000105599 2024-03-11T21:41:00.197 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:41:00.198 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:41:00.198 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "commit_sent", 2024-03-11T21:41:00.198 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:38:49.753144+0000", 2024-03-11T21:41:00.198 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 5.4953000000000001e-05 2024-03-11T21:41:00.198 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:41:00.198 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:41:00.198 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "done", 2024-03-11T21:41:00.198 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:38:49.755209+0000", 2024-03-11T21:41:00.198 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.0020651990000000002 2024-03-11T21:41:00.198 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-03-11T21:41:00.198 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-03-11T21:41:00.198 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-03-11T21:41:00.198 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:41:00.198 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:41:00.199 INFO:teuthology.orchestra.run.smithi092.stdout: "description": "osd_op(client.25282.0:185006 9.4 9:2b59c30c:::benchmark_data_smithi092_130291_object68622:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e326)", 2024-03-11T21:41:00.199 INFO:teuthology.orchestra.run.smithi092.stdout: "initiated_at": "2024-03-11T21:38:55.945039+0000", 2024-03-11T21:41:00.199 INFO:teuthology.orchestra.run.smithi092.stdout: "age": 124.158419622, 2024-03-11T21:41:00.199 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.15018426400000001, 2024-03-11T21:41:00.199 INFO:teuthology.orchestra.run.smithi092.stdout: "type_data": { 2024-03-11T21:41:00.199 INFO:teuthology.orchestra.run.smithi092.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-11T21:41:00.199 INFO:teuthology.orchestra.run.smithi092.stdout: "client_info": { 2024-03-11T21:41:00.199 INFO:teuthology.orchestra.run.smithi092.stdout: "client": "client.25282", 2024-03-11T21:41:00.199 INFO:teuthology.orchestra.run.smithi092.stdout: "client_addr": "v1:172.21.15.92:0/2489519821", 2024-03-11T21:41:00.199 INFO:teuthology.orchestra.run.smithi092.stdout: "tid": 185006 2024-03-11T21:41:00.199 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:41:00.199 INFO:teuthology.orchestra.run.smithi092.stdout: "events": [ 2024-03-11T21:41:00.199 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:41:00.200 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "initiated", 2024-03-11T21:41:00.200 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:38:55.945039+0000", 2024-03-11T21:41:00.200 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-03-11T21:41:00.200 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:41:00.200 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:41:00.200 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "header_read", 2024-03-11T21:41:00.200 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:38:55.945039+0000", 2024-03-11T21:41:00.200 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-03-11T21:41:00.200 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:41:00.200 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:41:00.200 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "throttled", 2024-03-11T21:41:00.200 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:38:55.945041+0000", 2024-03-11T21:41:00.200 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.178e-06 2024-03-11T21:41:00.201 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:41:00.201 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:41:00.201 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "all_read", 2024-03-11T21:41:00.201 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:38:55.945049+0000", 2024-03-11T21:41:00.201 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 8.1589999999999993e-06 2024-03-11T21:41:00.201 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:41:00.201 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:41:00.201 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "dispatched", 2024-03-11T21:41:00.201 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:38:55.945059+0000", 2024-03-11T21:41:00.201 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 9.7839999999999998e-06 2024-03-11T21:41:00.201 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:41:00.201 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:41:00.201 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "queued_for_pg", 2024-03-11T21:41:00.201 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:38:55.945065+0000", 2024-03-11T21:41:00.201 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 6.9360000000000002e-06 2024-03-11T21:41:00.202 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:41:00.202 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:41:00.202 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-03-11T21:41:00.202 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:38:55.974357+0000", 2024-03-11T21:41:00.202 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.029291405 2024-03-11T21:41:00.202 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:41:00.202 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:41:00.202 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "waiting for rw locks", 2024-03-11T21:41:00.202 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:38:55.974464+0000", 2024-03-11T21:41:00.202 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.000106813 2024-03-11T21:41:00.202 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:41:00.202 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:41:00.202 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-03-11T21:41:00.202 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:38:56.060614+0000", 2024-03-11T21:41:00.203 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.086150112000000001 2024-03-11T21:41:00.203 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:41:00.203 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:41:00.203 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "started", 2024-03-11T21:41:00.203 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:38:56.060714+0000", 2024-03-11T21:41:00.203 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.000100339 2024-03-11T21:41:00.203 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:41:00.203 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:41:00.203 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "waiting for subops from 3,4", 2024-03-11T21:41:00.203 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:38:56.060864+0000", 2024-03-11T21:41:00.203 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00015015699999999999 2024-03-11T21:41:00.203 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:41:00.203 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:41:00.203 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "op_commit", 2024-03-11T21:41:00.203 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:38:56.065805+0000", 2024-03-11T21:41:00.204 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.0049407449999999999 2024-03-11T21:41:00.204 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:41:00.204 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:41:00.204 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "sub_op_commit_rec", 2024-03-11T21:41:00.204 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:38:56.065847+0000", 2024-03-11T21:41:00.204 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 4.2453000000000002e-05 2024-03-11T21:41:00.204 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:41:00.204 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:41:00.204 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "sub_op_commit_rec", 2024-03-11T21:41:00.204 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:38:56.093108+0000", 2024-03-11T21:41:00.204 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.027260258999999998 2024-03-11T21:41:00.204 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:41:00.204 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:41:00.204 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "commit_sent", 2024-03-11T21:41:00.205 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:38:56.093166+0000", 2024-03-11T21:41:00.205 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 5.8541000000000001e-05 2024-03-11T21:41:00.205 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:41:00.205 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:41:00.205 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "done", 2024-03-11T21:41:00.205 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:38:56.095224+0000", 2024-03-11T21:41:00.205 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.0020573829999999999 2024-03-11T21:41:00.205 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-03-11T21:41:00.205 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-03-11T21:41:00.205 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-03-11T21:41:00.205 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:41:00.205 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:41:00.205 INFO:teuthology.orchestra.run.smithi092.stdout: "description": "osd_op(client.25282.0:206866 9.4 9:2200ac0b:::benchmark_data_smithi092_130291_object90482:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e330)", 2024-03-11T21:41:00.205 INFO:teuthology.orchestra.run.smithi092.stdout: "initiated_at": "2024-03-11T21:39:23.071969+0000", 2024-03-11T21:41:00.205 INFO:teuthology.orchestra.run.smithi092.stdout: "age": 97.031490253000001, 2024-03-11T21:41:00.206 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.17973709900000001, 2024-03-11T21:41:00.206 INFO:teuthology.orchestra.run.smithi092.stdout: "type_data": { 2024-03-11T21:41:00.206 INFO:teuthology.orchestra.run.smithi092.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-11T21:41:00.206 INFO:teuthology.orchestra.run.smithi092.stdout: "client_info": { 2024-03-11T21:41:00.206 INFO:teuthology.orchestra.run.smithi092.stdout: "client": "client.25282", 2024-03-11T21:41:00.206 INFO:teuthology.orchestra.run.smithi092.stdout: "client_addr": "v1:172.21.15.92:0/2489519821", 2024-03-11T21:41:00.206 INFO:teuthology.orchestra.run.smithi092.stdout: "tid": 206866 2024-03-11T21:41:00.206 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:41:00.206 INFO:teuthology.orchestra.run.smithi092.stdout: "events": [ 2024-03-11T21:41:00.206 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:41:00.206 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "initiated", 2024-03-11T21:41:00.206 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:39:23.071969+0000", 2024-03-11T21:41:00.206 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-03-11T21:41:00.206 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:41:00.207 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:41:00.207 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "header_read", 2024-03-11T21:41:00.207 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:39:23.071969+0000", 2024-03-11T21:41:00.207 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-03-11T21:41:00.207 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:41:00.207 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:41:00.207 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "throttled", 2024-03-11T21:41:00.207 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:39:23.071970+0000", 2024-03-11T21:41:00.207 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.3030000000000001e-06 2024-03-11T21:41:00.207 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:41:00.207 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:41:00.207 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "all_read", 2024-03-11T21:41:00.207 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:39:23.071975+0000", 2024-03-11T21:41:00.207 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 5.1789999999999999e-06 2024-03-11T21:41:00.208 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:41:00.208 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:41:00.208 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "dispatched", 2024-03-11T21:41:00.208 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:39:23.071984+0000", 2024-03-11T21:41:00.208 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 8.6889999999999993e-06 2024-03-11T21:41:00.208 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:41:00.208 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:41:00.208 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "queued_for_pg", 2024-03-11T21:41:00.208 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:39:23.071991+0000", 2024-03-11T21:41:00.208 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 6.8480000000000003e-06 2024-03-11T21:41:00.208 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:41:00.208 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:41:00.208 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-03-11T21:41:00.208 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:39:23.078305+0000", 2024-03-11T21:41:00.209 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.0063143540000000003 2024-03-11T21:41:00.209 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:41:00.209 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:41:00.209 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "started", 2024-03-11T21:41:00.209 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:39:23.078432+0000", 2024-03-11T21:41:00.209 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00012656599999999999 2024-03-11T21:41:00.209 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:41:00.209 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:41:00.209 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "waiting for subops from 3,4", 2024-03-11T21:41:00.209 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:39:23.078558+0000", 2024-03-11T21:41:00.209 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00012665800000000001 2024-03-11T21:41:00.209 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:41:00.209 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:41:00.209 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "op_commit", 2024-03-11T21:41:00.209 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:39:23.092916+0000", 2024-03-11T21:41:00.210 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.014357481 2024-03-11T21:41:00.210 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:41:00.210 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:41:00.210 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "sub_op_commit_rec", 2024-03-11T21:41:00.210 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:39:23.092956+0000", 2024-03-11T21:41:00.211 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 4.0176999999999998e-05 2024-03-11T21:41:00.211 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:41:00.211 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:41:00.211 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "sub_op_commit_rec", 2024-03-11T21:41:00.211 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:39:23.250060+0000", 2024-03-11T21:41:00.211 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.15710440000000001 2024-03-11T21:41:00.211 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:41:00.211 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:41:00.211 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "commit_sent", 2024-03-11T21:41:00.211 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:39:23.250122+0000", 2024-03-11T21:41:00.212 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 6.1305999999999999e-05 2024-03-11T21:41:00.212 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:41:00.212 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:41:00.212 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "done", 2024-03-11T21:41:00.212 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:39:23.251706+0000", 2024-03-11T21:41:00.212 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.0015841379999999999 2024-03-11T21:41:00.212 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-03-11T21:41:00.212 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-03-11T21:41:00.212 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-03-11T21:41:00.212 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:41:00.212 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:41:00.212 INFO:teuthology.orchestra.run.smithi092.stdout: "description": "osd_op(client.25282.0:206873 9.4 9:2200c4ef:::benchmark_data_smithi092_130291_object90489:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e330)", 2024-03-11T21:41:00.212 INFO:teuthology.orchestra.run.smithi092.stdout: "initiated_at": "2024-03-11T21:39:23.081132+0000", 2024-03-11T21:41:00.212 INFO:teuthology.orchestra.run.smithi092.stdout: "age": 97.022326978999999, 2024-03-11T21:41:00.212 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.20148022800000001, 2024-03-11T21:41:00.213 INFO:teuthology.orchestra.run.smithi092.stdout: "type_data": { 2024-03-11T21:41:00.213 INFO:teuthology.orchestra.run.smithi092.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-11T21:41:00.213 INFO:teuthology.orchestra.run.smithi092.stdout: "client_info": { 2024-03-11T21:41:00.213 INFO:teuthology.orchestra.run.smithi092.stdout: "client": "client.25282", 2024-03-11T21:41:00.213 INFO:teuthology.orchestra.run.smithi092.stdout: "client_addr": "v1:172.21.15.92:0/2489519821", 2024-03-11T21:41:00.213 INFO:teuthology.orchestra.run.smithi092.stdout: "tid": 206873 2024-03-11T21:41:00.213 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:41:00.213 INFO:teuthology.orchestra.run.smithi092.stdout: "events": [ 2024-03-11T21:41:00.213 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:41:00.213 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "initiated", 2024-03-11T21:41:00.213 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:39:23.081132+0000", 2024-03-11T21:41:00.213 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-03-11T21:41:00.213 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:41:00.214 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:41:00.214 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "header_read", 2024-03-11T21:41:00.214 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:39:23.081132+0000", 2024-03-11T21:41:00.214 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-03-11T21:41:00.214 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:41:00.214 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:41:00.214 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "throttled", 2024-03-11T21:41:00.214 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:39:23.081134+0000", 2024-03-11T21:41:00.214 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.9209999999999999e-06 2024-03-11T21:41:00.214 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:41:00.214 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:41:00.214 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "all_read", 2024-03-11T21:41:00.214 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:39:23.081141+0000", 2024-03-11T21:41:00.215 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 7.0180000000000004e-06 2024-03-11T21:41:00.215 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:41:00.215 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:41:00.215 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "dispatched", 2024-03-11T21:41:00.215 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:39:23.081153+0000", 2024-03-11T21:41:00.215 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.1552000000000001e-05 2024-03-11T21:41:00.215 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:41:00.215 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:41:00.215 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "queued_for_pg", 2024-03-11T21:41:00.215 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:39:23.081161+0000", 2024-03-11T21:41:00.215 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 8.4970000000000001e-06 2024-03-11T21:41:00.215 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:41:00.215 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:41:00.215 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-03-11T21:41:00.216 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:39:23.088967+0000", 2024-03-11T21:41:00.216 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.0078056719999999996 2024-03-11T21:41:00.216 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:41:00.216 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:41:00.216 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "started", 2024-03-11T21:41:00.216 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:39:23.089102+0000", 2024-03-11T21:41:00.216 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.000135105 2024-03-11T21:41:00.216 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:41:00.216 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:41:00.216 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "waiting for subops from 3,4", 2024-03-11T21:41:00.216 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:39:23.089236+0000", 2024-03-11T21:41:00.216 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00013442599999999999 2024-03-11T21:41:00.216 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:41:00.216 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:41:00.217 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "sub_op_commit_rec", 2024-03-11T21:41:00.217 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:39:23.093012+0000", 2024-03-11T21:41:00.217 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.0037753719999999999 2024-03-11T21:41:00.217 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:41:00.217 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:41:00.217 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "op_commit", 2024-03-11T21:41:00.217 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:39:23.098156+0000", 2024-03-11T21:41:00.217 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.0051447849999999998 2024-03-11T21:41:00.217 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:41:00.217 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:41:00.217 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "sub_op_commit_rec", 2024-03-11T21:41:00.217 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:39:23.280946+0000", 2024-03-11T21:41:00.217 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.18278986799999999 2024-03-11T21:41:00.217 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:41:00.217 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:41:00.218 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "commit_sent", 2024-03-11T21:41:00.218 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:39:23.281060+0000", 2024-03-11T21:41:00.218 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00011405599999999999 2024-03-11T21:41:00.218 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:41:00.218 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:41:00.218 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "done", 2024-03-11T21:41:00.218 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:39:23.282612+0000", 2024-03-11T21:41:00.218 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.001551956 2024-03-11T21:41:00.218 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-03-11T21:41:00.218 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-03-11T21:41:00.218 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-03-11T21:41:00.218 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:41:00.218 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:41:00.218 INFO:teuthology.orchestra.run.smithi092.stdout: "description": "osd_op(client.25282.0:206876 9.4 9:26996f31:::benchmark_data_smithi092_130291_object90492:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e330)", 2024-03-11T21:41:00.219 INFO:teuthology.orchestra.run.smithi092.stdout: "initiated_at": "2024-03-11T21:39:23.082023+0000", 2024-03-11T21:41:00.219 INFO:teuthology.orchestra.run.smithi092.stdout: "age": 97.021435671999996, 2024-03-11T21:41:00.219 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.20668336500000001, 2024-03-11T21:41:00.219 INFO:teuthology.orchestra.run.smithi092.stdout: "type_data": { 2024-03-11T21:41:00.219 INFO:teuthology.orchestra.run.smithi092.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-11T21:41:00.219 INFO:teuthology.orchestra.run.smithi092.stdout: "client_info": { 2024-03-11T21:41:00.219 INFO:teuthology.orchestra.run.smithi092.stdout: "client": "client.25282", 2024-03-11T21:41:00.219 INFO:teuthology.orchestra.run.smithi092.stdout: "client_addr": "v1:172.21.15.92:0/2489519821", 2024-03-11T21:41:00.219 INFO:teuthology.orchestra.run.smithi092.stdout: "tid": 206876 2024-03-11T21:41:00.219 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:41:00.219 INFO:teuthology.orchestra.run.smithi092.stdout: "events": [ 2024-03-11T21:41:00.219 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:41:00.219 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "initiated", 2024-03-11T21:41:00.219 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:39:23.082023+0000", 2024-03-11T21:41:00.220 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-03-11T21:41:00.220 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:41:00.220 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:41:00.220 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "header_read", 2024-03-11T21:41:00.220 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:39:23.082023+0000", 2024-03-11T21:41:00.220 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-03-11T21:41:00.220 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:41:00.220 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:41:00.220 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "throttled", 2024-03-11T21:41:00.220 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:39:23.082025+0000", 2024-03-11T21:41:00.220 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.764e-06 2024-03-11T21:41:00.220 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:41:00.220 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:41:00.220 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "all_read", 2024-03-11T21:41:00.220 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:39:23.082031+0000", 2024-03-11T21:41:00.221 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 6.2230000000000004e-06 2024-03-11T21:41:00.221 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:41:00.221 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:41:00.221 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "dispatched", 2024-03-11T21:41:00.221 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:39:23.082042+0000", 2024-03-11T21:41:00.221 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.0774000000000001e-05 2024-03-11T21:41:00.221 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:41:00.221 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:41:00.221 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "queued_for_pg", 2024-03-11T21:41:00.221 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:39:23.082050+0000", 2024-03-11T21:41:00.221 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 8.2300000000000008e-06 2024-03-11T21:41:00.221 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:41:00.221 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:41:00.221 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-03-11T21:41:00.222 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:39:23.093065+0000", 2024-03-11T21:41:00.222 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.011015118000000001 2024-03-11T21:41:00.222 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:41:00.222 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:41:00.222 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "started", 2024-03-11T21:41:00.222 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:39:23.093228+0000", 2024-03-11T21:41:00.222 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00016292800000000001 2024-03-11T21:41:00.222 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:41:00.222 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:41:00.222 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "waiting for subops from 3,4", 2024-03-11T21:41:00.222 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:39:23.093375+0000", 2024-03-11T21:41:00.222 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.000146768 2024-03-11T21:41:00.222 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:41:00.222 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:41:00.223 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "sub_op_commit_rec", 2024-03-11T21:41:00.223 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:39:23.098144+0000", 2024-03-11T21:41:00.223 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.0047686710000000004 2024-03-11T21:41:00.223 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:41:00.223 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:41:00.223 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "op_commit", 2024-03-11T21:41:00.223 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:39:23.098161+0000", 2024-03-11T21:41:00.223 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.6721000000000001e-05 2024-03-11T21:41:00.223 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:41:00.223 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:41:00.223 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "sub_op_commit_rec", 2024-03-11T21:41:00.223 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:39:23.282669+0000", 2024-03-11T21:41:00.223 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.184508846 2024-03-11T21:41:00.223 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:41:00.224 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:41:00.224 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "commit_sent", 2024-03-11T21:41:00.224 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:39:23.286938+0000", 2024-03-11T21:41:00.224 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.0042686690000000001 2024-03-11T21:41:00.224 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:41:00.224 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:41:00.224 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "done", 2024-03-11T21:41:00.224 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:39:23.288707+0000", 2024-03-11T21:41:00.224 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.001768653 2024-03-11T21:41:00.224 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-03-11T21:41:00.224 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-03-11T21:41:00.224 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-03-11T21:41:00.224 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:41:00.224 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:41:00.225 INFO:teuthology.orchestra.run.smithi092.stdout:osd.3: {} 2024-03-11T21:41:00.225 INFO:teuthology.orchestra.run.smithi092.stdout: "description": "osd_op(client.25282.0:206886 9.4 9:27afbb07:::benchmark_data_smithi092_130291_object90502:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e330)", 2024-03-11T21:41:00.225 INFO:teuthology.orchestra.run.smithi092.stdout: "initiated_at": "2024-03-11T21:39:23.092864+0000", 2024-03-11T21:41:00.225 INFO:teuthology.orchestra.run.smithi092.stdout: "age": 97.010595504999998, 2024-03-11T21:41:00.225 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.197616346, 2024-03-11T21:41:00.225 INFO:teuthology.orchestra.run.smithi092.stdout: "type_data": { 2024-03-11T21:41:00.226 INFO:teuthology.orchestra.run.smithi092.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-11T21:41:00.226 INFO:teuthology.orchestra.run.smithi092.stdout: "client_info": { 2024-03-11T21:41:00.226 INFO:teuthology.orchestra.run.smithi092.stdout: "client": "client.25282", 2024-03-11T21:41:00.226 INFO:teuthology.orchestra.run.smithi092.stdout: "client_addr": "v1:172.21.15.92:0/2489519821", 2024-03-11T21:41:00.226 INFO:teuthology.orchestra.run.smithi092.stdout: "tid": 206886 2024-03-11T21:41:00.226 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:41:00.226 INFO:teuthology.orchestra.run.smithi092.stdout: "events": [ 2024-03-11T21:41:00.226 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:41:00.226 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "initiated", 2024-03-11T21:41:00.226 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:39:23.092864+0000", 2024-03-11T21:41:00.226 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-03-11T21:41:00.226 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:41:00.226 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:41:00.226 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "header_read", 2024-03-11T21:41:00.227 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:39:23.092864+0000", 2024-03-11T21:41:00.227 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-03-11T21:41:00.227 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:41:00.227 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:41:00.227 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "throttled", 2024-03-11T21:41:00.227 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:39:23.092865+0000", 2024-03-11T21:41:00.227 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.6670000000000001e-06 2024-03-11T21:41:00.227 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:41:00.227 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:41:00.227 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "all_read", 2024-03-11T21:41:00.227 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:39:23.092871+0000", 2024-03-11T21:41:00.227 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 5.412e-06 2024-03-11T21:41:00.227 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:41:00.227 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:41:00.227 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "dispatched", 2024-03-11T21:41:00.228 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:39:23.092882+0000", 2024-03-11T21:41:00.228 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.1464000000000001e-05 2024-03-11T21:41:00.228 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:41:00.228 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:41:00.228 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "queued_for_pg", 2024-03-11T21:41:00.228 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:39:23.092888+0000", 2024-03-11T21:41:00.228 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 5.9070000000000004e-06 2024-03-11T21:41:00.228 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:41:00.228 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:41:00.228 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-03-11T21:41:00.228 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:39:23.095820+0000", 2024-03-11T21:41:00.228 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.0029321669999999998 2024-03-11T21:41:00.228 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:41:00.228 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:41:00.228 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "started", 2024-03-11T21:41:00.229 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:39:23.095959+0000", 2024-03-11T21:41:00.229 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00013850700000000001 2024-03-11T21:41:00.229 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:41:00.229 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:41:00.229 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "waiting for subops from 3,4", 2024-03-11T21:41:00.229 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:39:23.096107+0000", 2024-03-11T21:41:00.229 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00014845700000000001 2024-03-11T21:41:00.229 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:41:00.229 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:41:00.229 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "sub_op_commit_rec", 2024-03-11T21:41:00.229 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:39:23.098248+0000", 2024-03-11T21:41:00.229 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.002140617 2024-03-11T21:41:00.229 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:41:00.229 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:41:00.229 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "op_commit", 2024-03-11T21:41:00.230 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:39:23.098258+0000", 2024-03-11T21:41:00.230 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.0624e-05 2024-03-11T21:41:00.230 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:41:00.230 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:41:00.230 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "sub_op_commit_rec", 2024-03-11T21:41:00.230 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:39:23.288773+0000", 2024-03-11T21:41:00.230 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.190514506 2024-03-11T21:41:00.230 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:41:00.230 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:41:00.230 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "commit_sent", 2024-03-11T21:41:00.230 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:39:23.289202+0000", 2024-03-11T21:41:00.230 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.000429553 2024-03-11T21:41:00.230 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:41:00.230 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:41:00.231 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "done", 2024-03-11T21:41:00.231 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:39:23.290480+0000", 2024-03-11T21:41:00.231 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.001277465 2024-03-11T21:41:00.231 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-03-11T21:41:00.231 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-03-11T21:41:00.231 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-03-11T21:41:00.231 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-03-11T21:41:00.231 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-03-11T21:41:00.231 INFO:teuthology.orchestra.run.smithi092.stdout:} 2024-03-11T21:41:00.232 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:41:00.232 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:41:00.232 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:41:00.232 INFO:teuthology.orchestra.run.smithi092.stdout:osd.4: {} 2024-03-11T21:41:00.265 INFO:teuthology.orchestra.run.smithi092.stdout:osd.5: {} 2024-03-11T21:41:00.308 INFO:teuthology.orchestra.run.smithi092.stdout:osd.6: {} 2024-03-11T21:41:00.333 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:41:00.347 INFO:teuthology.orchestra.run.smithi092.stdout:osd.7: {} 2024-03-11T21:41:00.354 INFO:teuthology.orchestra.run.smithi092.stderr:osd.0: osd_enable_op_tracker = '' 2024-03-11T21:41:00.354 INFO:teuthology.orchestra.run.smithi092.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-11T21:41:00.354 INFO:teuthology.orchestra.run.smithi092.stderr:osd.2: osd_enable_op_tracker = '' 2024-03-11T21:41:00.354 INFO:teuthology.orchestra.run.smithi092.stderr:osd.3: osd_enable_op_tracker = '' 2024-03-11T21:41:00.354 INFO:teuthology.orchestra.run.smithi092.stderr:osd.4: osd_enable_op_tracker = '' 2024-03-11T21:41:00.354 INFO:teuthology.orchestra.run.smithi092.stderr:osd.5: osd_enable_op_tracker = '' 2024-03-11T21:41:00.355 INFO:teuthology.orchestra.run.smithi092.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-11T21:41:00.355 INFO:teuthology.orchestra.run.smithi092.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-11T21:41:00.373 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:41:00.373 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:41:00.374 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:41:00.474 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:41:00.513 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:41:00.514 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:41:00.515 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:41:00.617 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:41:00.654 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:41:00.656 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:41:00.657 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:41:00.758 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:41:00.804 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:41:00.805 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:41:00.805 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:41:00.829 INFO:tasks.radosbench.radosbench.0.smithi092.stdout: 8 15 23841 23826 186.124 149.062 0.00741833 0.0053667 2024-03-11T21:41:00.830 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:41:00 smithi097 ceph-mon[33317]: pgmap v719: 148 pgs: 148 active+clean; 969 MiB data, 6.3 GiB used, 709 GiB / 715 GiB avail; 1023 B/s rd, 97 MiB/s wr, 1.55k op/s 2024-03-11T21:41:00.830 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:41:00 smithi097 ceph-mon[33317]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-11T21:41:00.830 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:41:00 smithi097 ceph-mon[33317]: from='client.? v1:172.21.15.92:0/1814759268' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-03-11T21:41:00.830 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:41:00 smithi097 ceph-mon[33317]: osdmap e365: 8 total, 8 up, 8 in 2024-03-11T21:41:00.830 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:41:00 smithi097 ceph-mon[33317]: from='client.? v1:172.21.15.92:0/1461653552' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-11T21:41:00.864 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:41:00 smithi092 ceph-mon[29572]: pgmap v719: 148 pgs: 148 active+clean; 969 MiB data, 6.3 GiB used, 709 GiB / 715 GiB avail; 1023 B/s rd, 97 MiB/s wr, 1.55k op/s 2024-03-11T21:41:00.864 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:41:00 smithi092 ceph-mon[29572]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-11T21:41:00.864 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:41:00 smithi092 ceph-mon[29572]: from='client.? v1:172.21.15.92:0/1814759268' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-03-11T21:41:00.864 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:41:00 smithi092 ceph-mon[29572]: osdmap e365: 8 total, 8 up, 8 in 2024-03-11T21:41:00.864 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:41:00 smithi092 ceph-mon[29572]: from='client.? v1:172.21.15.92:0/1461653552' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-11T21:41:00.864 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:41:00 smithi092 ceph-mon[37563]: pgmap v719: 148 pgs: 148 active+clean; 969 MiB data, 6.3 GiB used, 709 GiB / 715 GiB avail; 1023 B/s rd, 97 MiB/s wr, 1.55k op/s 2024-03-11T21:41:00.865 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:41:00 smithi092 ceph-mon[37563]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-11T21:41:00.865 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:41:00 smithi092 ceph-mon[37563]: from='client.? v1:172.21.15.92:0/1814759268' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-03-11T21:41:00.865 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:41:00 smithi092 ceph-mon[37563]: osdmap e365: 8 total, 8 up, 8 in 2024-03-11T21:41:00.865 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:41:00 smithi092 ceph-mon[37563]: from='client.? v1:172.21.15.92:0/1461653552' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-11T21:41:00.906 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:41:00.946 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:41:00.947 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:41:00.947 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:41:01.048 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:41:01.086 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:41:01.086 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:41:01.087 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:41:01.187 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:41:01.225 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:41:01.226 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:41:01.226 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:41:01.327 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:41:01.361 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:41:01.362 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:41:01.362 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:41:01.463 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:41:01.499 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:41:01.499 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:41:01.499 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:41:01.600 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:41:01.638 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:41:01.638 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:41:01.639 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:41:01.739 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:41:01.777 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:41:01.777 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:41:01.778 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:41:01.830 INFO:tasks.radosbench.radosbench.0.smithi092.stdout: 9 16 26311 26295 182.588 154.312 0.00389457 0.00547113 2024-03-11T21:41:01.830 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:41:01 smithi097 ceph-mon[33317]: 5.4 scrub starts 2024-03-11T21:41:01.830 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:41:01 smithi097 ceph-mon[33317]: 5.4 scrub ok 2024-03-11T21:41:01.830 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:41:01 smithi097 ceph-mon[33317]: 6.4 scrub starts 2024-03-11T21:41:01.831 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:41:01 smithi097 ceph-mon[33317]: 6.4 scrub ok 2024-03-11T21:41:01.864 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:41:01 smithi092 ceph-mon[29572]: 5.4 scrub starts 2024-03-11T21:41:01.864 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:41:01 smithi092 ceph-mon[29572]: 5.4 scrub ok 2024-03-11T21:41:01.864 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:41:01 smithi092 ceph-mon[29572]: 6.4 scrub starts 2024-03-11T21:41:01.864 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:41:01 smithi092 ceph-mon[29572]: 6.4 scrub ok 2024-03-11T21:41:01.865 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:41:01 smithi092 ceph-mon[37563]: 5.4 scrub starts 2024-03-11T21:41:01.865 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:41:01 smithi092 ceph-mon[37563]: 5.4 scrub ok 2024-03-11T21:41:01.865 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:41:01 smithi092 ceph-mon[37563]: 6.4 scrub starts 2024-03-11T21:41:01.865 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:41:01 smithi092 ceph-mon[37563]: 6.4 scrub ok 2024-03-11T21:41:01.879 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:41:01.903 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:41:01.904 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:41:01.904 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:41:02.005 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:41:02.045 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:41:02.045 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:41:02.046 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:41:02.147 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:41:02.195 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:41:02.195 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:41:02.196 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:41:02.296 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:41:02.334 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:41:02.334 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:41:02.335 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:41:02.435 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:41:02.472 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:41:02.472 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:41:02.473 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:41:02.573 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:41:02.609 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:41:02.609 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:41:02.610 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:41:02.711 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:41:02.750 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:41:02.751 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:41:02.751 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:41:02.830 INFO:tasks.radosbench.radosbench.0.smithi092.stdout: 10 16 28228 28212 176.309 119.812 0.00979427 0.00565287 2024-03-11T21:41:02.830 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:41:02 smithi097 ceph-mon[33317]: 4.17 scrub starts 2024-03-11T21:41:02.830 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:41:02 smithi097 ceph-mon[33317]: 4.17 scrub ok 2024-03-11T21:41:02.830 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:41:02 smithi097 ceph-mon[33317]: 5.1a scrub starts 2024-03-11T21:41:02.831 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:41:02 smithi097 ceph-mon[33317]: 5.1a scrub ok 2024-03-11T21:41:02.831 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:41:02 smithi097 ceph-mon[33317]: pgmap v721: 148 pgs: 148 active+clean; 969 MiB data, 6.3 GiB used, 709 GiB / 715 GiB avail; 1023 B/s rd, 97 MiB/s wr, 1.55k op/s 2024-03-11T21:41:02.831 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:41:02 smithi097 ceph-mon[33317]: purged_snaps scrub starts 2024-03-11T21:41:02.831 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:41:02 smithi097 ceph-mon[33317]: purged_snaps scrub ok 2024-03-11T21:41:02.831 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:41:02 smithi097 ceph-mon[33317]: 2.a scrub starts 2024-03-11T21:41:02.831 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:41:02 smithi097 ceph-mon[33317]: 2.a scrub ok 2024-03-11T21:41:02.852 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:41:02.864 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:41:02 smithi092 ceph-mon[29572]: 4.17 scrub starts 2024-03-11T21:41:02.864 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:41:02 smithi092 ceph-mon[29572]: 4.17 scrub ok 2024-03-11T21:41:02.864 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:41:02 smithi092 ceph-mon[29572]: 5.1a scrub starts 2024-03-11T21:41:02.864 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:41:02 smithi092 ceph-mon[29572]: 5.1a scrub ok 2024-03-11T21:41:02.864 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:41:02 smithi092 ceph-mon[29572]: pgmap v721: 148 pgs: 148 active+clean; 969 MiB data, 6.3 GiB used, 709 GiB / 715 GiB avail; 1023 B/s rd, 97 MiB/s wr, 1.55k op/s 2024-03-11T21:41:02.864 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:41:02 smithi092 ceph-mon[29572]: purged_snaps scrub starts 2024-03-11T21:41:02.864 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:41:02 smithi092 ceph-mon[29572]: purged_snaps scrub ok 2024-03-11T21:41:02.864 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:41:02 smithi092 ceph-mon[29572]: 2.a scrub starts 2024-03-11T21:41:02.865 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:41:02 smithi092 ceph-mon[29572]: 2.a scrub ok 2024-03-11T21:41:02.865 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:41:02 smithi092 ceph-mon[37563]: 4.17 scrub starts 2024-03-11T21:41:02.865 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:41:02 smithi092 ceph-mon[37563]: 4.17 scrub ok 2024-03-11T21:41:02.865 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:41:02 smithi092 ceph-mon[37563]: 5.1a scrub starts 2024-03-11T21:41:02.865 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:41:02 smithi092 ceph-mon[37563]: 5.1a scrub ok 2024-03-11T21:41:02.865 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:41:02 smithi092 ceph-mon[37563]: pgmap v721: 148 pgs: 148 active+clean; 969 MiB data, 6.3 GiB used, 709 GiB / 715 GiB avail; 1023 B/s rd, 97 MiB/s wr, 1.55k op/s 2024-03-11T21:41:02.865 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:41:02 smithi092 ceph-mon[37563]: purged_snaps scrub starts 2024-03-11T21:41:02.865 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:41:02 smithi092 ceph-mon[37563]: purged_snaps scrub ok 2024-03-11T21:41:02.865 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:41:02 smithi092 ceph-mon[37563]: 2.a scrub starts 2024-03-11T21:41:02.865 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:41:02 smithi092 ceph-mon[37563]: 2.a scrub ok 2024-03-11T21:41:02.879 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:41:02.880 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:41:02.880 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:41:02.981 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:41:03.004 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:41:03.005 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:41:03.005 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:41:03.106 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:41:03.141 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:41:03.143 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:41:03.143 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:41:03.244 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:41:03.279 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:41:03.279 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:41:03.280 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:41:03.330 INFO:journalctl@ceph.iscsi.iscsi.a.smithi097.stdout:Mar 11 21:41:02 smithi097 ceph-00db0016-dfed-11ee-95c8-87774f69a715-iscsi-iscsi-a[60943]: debug there is no tcmu-runner data available 2024-03-11T21:41:03.381 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:41:03.422 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:41:03.422 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:41:03.423 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:41:03.524 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:41:03.551 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:41:03.552 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:41:03.552 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:41:03.653 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:41:03.689 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:41:03.689 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:41:03.690 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:41:03.790 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:41:03.826 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:41:03.827 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:41:03.827 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:41:03.830 INFO:tasks.radosbench.radosbench.0.smithi092.stdout: 11 15 30024 30009 170.49 112.312 0.00627112 0.00586056 2024-03-11T21:41:03.830 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:41:03 smithi097 ceph-mon[33317]: 2.19 scrub starts 2024-03-11T21:41:03.830 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:41:03 smithi097 ceph-mon[33317]: 2.19 scrub ok 2024-03-11T21:41:03.830 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:41:03 smithi097 ceph-mon[33317]: 3.1 scrub starts 2024-03-11T21:41:03.830 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:41:03 smithi097 ceph-mon[33317]: 3.1 scrub ok 2024-03-11T21:41:03.830 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:41:03 smithi097 ceph-mon[33317]: 2.1d scrub starts 2024-03-11T21:41:03.830 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:41:03 smithi097 ceph-mon[33317]: 2.1d scrub ok 2024-03-11T21:41:03.831 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:41:03 smithi097 ceph-mon[33317]: 6.1a scrub starts 2024-03-11T21:41:03.831 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:41:03 smithi097 ceph-mon[33317]: osdmap e366: 8 total, 8 up, 8 in 2024-03-11T21:41:03.831 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:41:03 smithi097 ceph-mon[33317]: 6.1a scrub ok 2024-03-11T21:41:03.831 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:41:03 smithi097 ceph-mon[33317]: from='client.24469 v1:172.21.15.97:0/404208310' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:41:03.864 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:41:03 smithi092 ceph-mon[29572]: 2.19 scrub starts 2024-03-11T21:41:03.864 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:41:03 smithi092 ceph-mon[29572]: 2.19 scrub ok 2024-03-11T21:41:03.864 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:41:03 smithi092 ceph-mon[29572]: 3.1 scrub starts 2024-03-11T21:41:03.865 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:41:03 smithi092 ceph-mon[29572]: 3.1 scrub ok 2024-03-11T21:41:03.865 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:41:03 smithi092 ceph-mon[29572]: 2.1d scrub starts 2024-03-11T21:41:03.865 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:41:03 smithi092 ceph-mon[29572]: 2.1d scrub ok 2024-03-11T21:41:03.865 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:41:03 smithi092 ceph-mon[29572]: 6.1a scrub starts 2024-03-11T21:41:03.865 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:41:03 smithi092 ceph-mon[29572]: osdmap e366: 8 total, 8 up, 8 in 2024-03-11T21:41:03.865 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:41:03 smithi092 ceph-mon[29572]: 6.1a scrub ok 2024-03-11T21:41:03.865 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:41:03 smithi092 ceph-mon[29572]: from='client.24469 v1:172.21.15.97:0/404208310' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:41:03.865 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:41:03 smithi092 ceph-mon[37563]: 2.19 scrub starts 2024-03-11T21:41:03.865 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:41:03 smithi092 ceph-mon[37563]: 2.19 scrub ok 2024-03-11T21:41:03.865 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:41:03 smithi092 ceph-mon[37563]: 3.1 scrub starts 2024-03-11T21:41:03.865 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:41:03 smithi092 ceph-mon[37563]: 3.1 scrub ok 2024-03-11T21:41:03.866 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:41:03 smithi092 ceph-mon[37563]: 2.1d scrub starts 2024-03-11T21:41:03.866 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:41:03 smithi092 ceph-mon[37563]: 2.1d scrub ok 2024-03-11T21:41:03.866 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:41:03 smithi092 ceph-mon[37563]: 6.1a scrub starts 2024-03-11T21:41:03.866 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:41:03 smithi092 ceph-mon[37563]: osdmap e366: 8 total, 8 up, 8 in 2024-03-11T21:41:03.866 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:41:03 smithi092 ceph-mon[37563]: 6.1a scrub ok 2024-03-11T21:41:03.866 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:41:03 smithi092 ceph-mon[37563]: from='client.24469 v1:172.21.15.97:0/404208310' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:41:03.928 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:41:03.967 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:41:03.967 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:41:03.967 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:41:04.068 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:41:04.093 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:41:04.093 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:41:04.093 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:41:04.194 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:41:04.218 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:41:04.218 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:41:04.219 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:41:04.320 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:41:04.354 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:41:04.354 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:41:04.355 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:41:04.456 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:41:04.491 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:41:04.491 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:41:04.491 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:41:04.592 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:41:04.628 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:41:04.628 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:41:04.629 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:41:04.730 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:41:04.765 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:41:04.766 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:41:04.767 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:41:04.830 INFO:tasks.radosbench.radosbench.0.smithi092.stdout: 12 16 31692 31676 164.964 104.188 0.00389767 0.00605604 2024-03-11T21:41:04.864 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:41:04 smithi092 ceph-mon[29572]: pgmap v723: 148 pgs: 148 active+clean; 1.6 GiB data, 7.9 GiB used, 707 GiB / 715 GiB avail; 639 B/s rd, 196 MiB/s wr, 3.13k op/s 2024-03-11T21:41:04.864 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:41:04 smithi092 ceph-mon[37563]: pgmap v723: 148 pgs: 148 active+clean; 1.6 GiB data, 7.9 GiB used, 707 GiB / 715 GiB avail; 639 B/s rd, 196 MiB/s wr, 3.13k op/s 2024-03-11T21:41:04.868 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:41:04.904 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:41:04.905 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:41:04.906 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:41:05.006 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:41:05.045 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:41:05.046 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:41:05.046 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:41:05.080 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:41:04 smithi097 ceph-mon[33317]: pgmap v723: 148 pgs: 148 active+clean; 1.6 GiB data, 7.9 GiB used, 707 GiB / 715 GiB avail; 639 B/s rd, 196 MiB/s wr, 3.13k op/s 2024-03-11T21:41:05.147 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:41:05.183 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:41:05.184 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:41:05.184 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:41:05.285 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:41:05.319 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:41:05.319 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:41:05.319 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:41:05.420 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:41:05.446 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:41:05.447 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:41:05.447 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:41:05.548 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:41:05.589 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:41:05.589 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:41:05.590 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:41:05.691 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:41:05.726 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:41:05.727 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:41:05.727 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:41:05.828 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:41:05.830 INFO:tasks.radosbench.radosbench.0.smithi092.stdout: 13 16 33631 33615 161.596 121.188 0.0036932 0.00618355 2024-03-11T21:41:05.862 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:41:05.862 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:41:05.863 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:41:05.864 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:41:05 smithi092 ceph-mon[29572]: 5.6 scrub starts 2024-03-11T21:41:05.864 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:41:05 smithi092 ceph-mon[29572]: 5.6 scrub ok 2024-03-11T21:41:05.864 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:41:05 smithi092 ceph-mon[29572]: 5.12 scrub starts 2024-03-11T21:41:05.864 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:41:05 smithi092 ceph-mon[29572]: 5.12 scrub ok 2024-03-11T21:41:05.864 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:41:05 smithi092 ceph-mon[29572]: 2.1e scrub starts 2024-03-11T21:41:05.865 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:41:05 smithi092 ceph-mon[29572]: 2.1e scrub ok 2024-03-11T21:41:05.865 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:41:05 smithi092 ceph-mon[29572]: 4.4 scrub starts 2024-03-11T21:41:05.865 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:41:05 smithi092 ceph-mon[29572]: 4.4 scrub ok 2024-03-11T21:41:05.865 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:41:05 smithi092 ceph-mon[37563]: 5.6 scrub starts 2024-03-11T21:41:05.865 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:41:05 smithi092 ceph-mon[37563]: 5.6 scrub ok 2024-03-11T21:41:05.865 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:41:05 smithi092 ceph-mon[37563]: 5.12 scrub starts 2024-03-11T21:41:05.865 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:41:05 smithi092 ceph-mon[37563]: 5.12 scrub ok 2024-03-11T21:41:05.865 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:41:05 smithi092 ceph-mon[37563]: 2.1e scrub starts 2024-03-11T21:41:05.865 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:41:05 smithi092 ceph-mon[37563]: 2.1e scrub ok 2024-03-11T21:41:05.866 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:41:05 smithi092 ceph-mon[37563]: 4.4 scrub starts 2024-03-11T21:41:05.866 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:41:05 smithi092 ceph-mon[37563]: 4.4 scrub ok 2024-03-11T21:41:05.963 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:41:05.988 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:41:05.989 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:41:05.989 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:41:06.080 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:41:05 smithi097 ceph-mon[33317]: 5.6 scrub starts 2024-03-11T21:41:06.080 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:41:05 smithi097 ceph-mon[33317]: 5.6 scrub ok 2024-03-11T21:41:06.080 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:41:05 smithi097 ceph-mon[33317]: 5.12 scrub starts 2024-03-11T21:41:06.080 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:41:05 smithi097 ceph-mon[33317]: 5.12 scrub ok 2024-03-11T21:41:06.080 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:41:05 smithi097 ceph-mon[33317]: 2.1e scrub starts 2024-03-11T21:41:06.081 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:41:05 smithi097 ceph-mon[33317]: 2.1e scrub ok 2024-03-11T21:41:06.081 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:41:05 smithi097 ceph-mon[33317]: 4.4 scrub starts 2024-03-11T21:41:06.081 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:41:05 smithi097 ceph-mon[33317]: 4.4 scrub ok 2024-03-11T21:41:06.090 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:41:06.130 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:41:06.130 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:41:06.131 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:41:06.231 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:41:06.267 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:41:06.268 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:41:06.268 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:41:06.369 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:41:06.405 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:41:06.406 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:41:06.406 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:41:06.507 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:41:06.543 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:41:06.543 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:41:06.544 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:41:06.644 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:41:06.681 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:41:06.682 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:41:06.682 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:41:06.783 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:41:06.818 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:41:06.819 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:41:06.819 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:41:06.830 INFO:tasks.radosbench.radosbench.0.smithi092.stdout: 14 16 35674 35658 159.174 127.688 0.00700026 0.00627704 2024-03-11T21:41:06.864 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:41:06 smithi092 ceph-mon[29572]: pgmap v724: 148 pgs: 148 active+clean; 1.7 GiB data, 8.7 GiB used, 707 GiB / 715 GiB avail; 1.2 KiB/s rd, 198 MiB/s wr, 3.17k op/s 2024-03-11T21:41:06.864 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:41:06 smithi092 ceph-mon[37563]: pgmap v724: 148 pgs: 148 active+clean; 1.7 GiB data, 8.7 GiB used, 707 GiB / 715 GiB avail; 1.2 KiB/s rd, 198 MiB/s wr, 3.17k op/s 2024-03-11T21:41:06.920 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:41:06.957 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:41:06.957 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:41:06.957 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:41:07.058 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:41:07.080 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:41:06 smithi097 ceph-mon[33317]: pgmap v724: 148 pgs: 148 active+clean; 1.7 GiB data, 8.7 GiB used, 707 GiB / 715 GiB avail; 1.2 KiB/s rd, 198 MiB/s wr, 3.17k op/s 2024-03-11T21:41:07.095 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:41:07.096 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:41:07.096 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:41:07.197 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:41:07.233 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:41:07.233 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:41:07.233 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:41:07.334 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:41:07.368 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:41:07.369 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:41:07.369 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:41:07.470 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:41:07.506 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:41:07.507 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:41:07.508 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:41:07.540 INFO:teuthology.orchestra.run.smithi092.stderr:Inferring config /var/lib/ceph/00db0016-dfed-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T21:41:07.608 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:41:07.643 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:41:07.644 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:41:07.644 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:41:07.745 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:41:07.785 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:41:07.785 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:41:07.786 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:41:07.830 INFO:tasks.radosbench.radosbench.0.smithi092.stdout: 15 15 37725 37710 157.111 128.25 0.005256 0.00636016 2024-03-11T21:41:07.864 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:41:07 smithi092 ceph-mon[29572]: 5.d scrub starts 2024-03-11T21:41:07.864 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:41:07 smithi092 ceph-mon[29572]: 4.3 scrub starts 2024-03-11T21:41:07.865 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:41:07 smithi092 ceph-mon[29572]: 5.d scrub ok 2024-03-11T21:41:07.865 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:41:07 smithi092 ceph-mon[29572]: 4.3 scrub ok 2024-03-11T21:41:07.865 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:41:07 smithi092 ceph-mon[29572]: 4.8 scrub starts 2024-03-11T21:41:07.865 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:41:07 smithi092 ceph-mon[29572]: 4.8 scrub ok 2024-03-11T21:41:07.865 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:41:07 smithi092 ceph-mon[29572]: 4.2 scrub starts 2024-03-11T21:41:07.865 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:41:07 smithi092 ceph-mon[29572]: 4.2 scrub ok 2024-03-11T21:41:07.865 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:41:07 smithi092 ceph-mon[29572]: 3.2 scrub starts 2024-03-11T21:41:07.865 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:41:07 smithi092 ceph-mon[29572]: 3.2 scrub ok 2024-03-11T21:41:07.865 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:41:07 smithi092 ceph-mon[37563]: 5.d scrub starts 2024-03-11T21:41:07.865 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:41:07 smithi092 ceph-mon[37563]: 4.3 scrub starts 2024-03-11T21:41:07.865 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:41:07 smithi092 ceph-mon[37563]: 5.d scrub ok 2024-03-11T21:41:07.866 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:41:07 smithi092 ceph-mon[37563]: 4.3 scrub ok 2024-03-11T21:41:07.866 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:41:07 smithi092 ceph-mon[37563]: 4.8 scrub starts 2024-03-11T21:41:07.866 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:41:07 smithi092 ceph-mon[37563]: 4.8 scrub ok 2024-03-11T21:41:07.866 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:41:07 smithi092 ceph-mon[37563]: 4.2 scrub starts 2024-03-11T21:41:07.866 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:41:07 smithi092 ceph-mon[37563]: 4.2 scrub ok 2024-03-11T21:41:07.866 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:41:07 smithi092 ceph-mon[37563]: 3.2 scrub starts 2024-03-11T21:41:07.866 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:41:07 smithi092 ceph-mon[37563]: 3.2 scrub ok 2024-03-11T21:41:07.886 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:41:07.921 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:41:07.922 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:41:07.922 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:41:08.023 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:41:08.061 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:41:08.061 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:41:08.062 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:41:08.080 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:41:07 smithi097 ceph-mon[33317]: 5.d scrub starts 2024-03-11T21:41:08.080 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:41:07 smithi097 ceph-mon[33317]: 4.3 scrub starts 2024-03-11T21:41:08.081 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:41:07 smithi097 ceph-mon[33317]: 5.d scrub ok 2024-03-11T21:41:08.081 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:41:07 smithi097 ceph-mon[33317]: 4.3 scrub ok 2024-03-11T21:41:08.081 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:41:07 smithi097 ceph-mon[33317]: 4.8 scrub starts 2024-03-11T21:41:08.081 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:41:07 smithi097 ceph-mon[33317]: 4.8 scrub ok 2024-03-11T21:41:08.081 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:41:07 smithi097 ceph-mon[33317]: 4.2 scrub starts 2024-03-11T21:41:08.081 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:41:07 smithi097 ceph-mon[33317]: 4.2 scrub ok 2024-03-11T21:41:08.081 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:41:07 smithi097 ceph-mon[33317]: 3.2 scrub starts 2024-03-11T21:41:08.081 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:41:07 smithi097 ceph-mon[33317]: 3.2 scrub ok 2024-03-11T21:41:08.142 DEBUG:teuthology.orchestra.run.smithi092:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid 00db0016-dfed-11ee-95c8-87774f69a715 -- ceph daemon osd.2 dump_ops_in_flight 2024-03-11T21:41:08.163 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:41:08.181 INFO:journalctl@ceph.mgr.y.smithi092.stdout:Mar 11 21:41:07 smithi092 ceph-00db0016-dfed-11ee-95c8-87774f69a715-mgr-y[29801]: ::ffff:172.21.15.97 - - [11/Mar/2024:21:41:07] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-11T21:41:08.222 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:41:08.223 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:41:08.224 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:41:08.324 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:41:08.372 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:41:08.372 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:41:08.373 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:41:08.473 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:41:08.510 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:41:08.510 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:41:08.511 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:41:08.611 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:41:08.654 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:41:08.655 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:41:08.655 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:41:08.756 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:41:08.795 DEBUG:teuthology.orchestra.run.smithi092:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid 00db0016-dfed-11ee-95c8-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-03-11T21:41:08.796 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:41:08 smithi092 ceph-mon[29572]: pgmap v725: 148 pgs: 148 active+clean; 1.8 GiB data, 9.0 GiB used, 706 GiB / 715 GiB avail; 639 B/s rd, 105 MiB/s wr, 1.67k op/s 2024-03-11T21:41:08.796 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:41:08 smithi092 ceph-mon[29572]: 4.1 scrub starts 2024-03-11T21:41:08.796 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:41:08 smithi092 ceph-mon[29572]: 4.1 scrub ok 2024-03-11T21:41:08.796 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:41:08 smithi092 ceph-mon[29572]: from='mgr.14607 v1:172.21.15.92:0/1702717597' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T21:41:08.796 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:41:08 smithi092 ceph-mon[29572]: 4.5 scrub starts 2024-03-11T21:41:08.796 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:41:08 smithi092 ceph-mon[29572]: 4.5 scrub ok 2024-03-11T21:41:08.797 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:41:08 smithi092 ceph-mon[29572]: 6.1f scrub starts 2024-03-11T21:41:08.797 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:41:08 smithi092 ceph-mon[37563]: pgmap v725: 148 pgs: 148 active+clean; 1.8 GiB data, 9.0 GiB used, 706 GiB / 715 GiB avail; 639 B/s rd, 105 MiB/s wr, 1.67k op/s 2024-03-11T21:41:08.797 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:41:08 smithi092 ceph-mon[37563]: 4.1 scrub starts 2024-03-11T21:41:08.797 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:41:08 smithi092 ceph-mon[37563]: 4.1 scrub ok 2024-03-11T21:41:08.797 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:41:08 smithi092 ceph-mon[37563]: from='mgr.14607 v1:172.21.15.92:0/1702717597' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T21:41:08.797 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:41:08 smithi092 ceph-mon[37563]: 4.5 scrub starts 2024-03-11T21:41:08.797 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:41:08 smithi092 ceph-mon[37563]: 4.5 scrub ok 2024-03-11T21:41:08.797 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:41:08 smithi092 ceph-mon[37563]: 6.1f scrub starts 2024-03-11T21:41:08.798 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:41:08.798 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:41:08.799 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:41:08.831 INFO:tasks.radosbench.radosbench.0.smithi092.stdout: 16 16 39521 39505 154.299 112.188 0.00472888 0.00647584 2024-03-11T21:41:08.899 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:41:08.947 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:41:08.947 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:41:08.947 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:41:09.048 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:41:09.080 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:41:08 smithi097 ceph-mon[33317]: pgmap v725: 148 pgs: 148 active+clean; 1.8 GiB data, 9.0 GiB used, 706 GiB / 715 GiB avail; 639 B/s rd, 105 MiB/s wr, 1.67k op/s 2024-03-11T21:41:09.081 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:41:08 smithi097 ceph-mon[33317]: 4.1 scrub starts 2024-03-11T21:41:09.081 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:41:08 smithi097 ceph-mon[33317]: 4.1 scrub ok 2024-03-11T21:41:09.081 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:41:08 smithi097 ceph-mon[33317]: from='mgr.14607 v1:172.21.15.92:0/1702717597' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T21:41:09.081 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:41:08 smithi097 ceph-mon[33317]: 4.5 scrub starts 2024-03-11T21:41:09.081 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:41:08 smithi097 ceph-mon[33317]: 4.5 scrub ok 2024-03-11T21:41:09.081 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:41:08 smithi097 ceph-mon[33317]: 6.1f scrub starts 2024-03-11T21:41:09.092 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:41:09.093 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:41:09.093 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:41:09.194 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:41:09.243 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:41:09.244 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:41:09.244 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:41:09.345 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:41:09.363 DEBUG:teuthology.orchestra.run.smithi092:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid 00db0016-dfed-11ee-95c8-87774f69a715 -- ceph osd unset nodeep-scrub 2024-03-11T21:41:09.390 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:41:09.391 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:41:09.391 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:41:09.492 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:41:09.538 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:41:09.539 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:41:09.539 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:41:09.642 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:41:09.672 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:41:09.672 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:41:09.673 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:41:09.747 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:41:09 smithi092 ceph-mon[29572]: 6.14 scrub starts 2024-03-11T21:41:09.747 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:41:09 smithi092 ceph-mon[29572]: 6.14 scrub ok 2024-03-11T21:41:09.747 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:41:09 smithi092 ceph-mon[29572]: 6.1f scrub ok 2024-03-11T21:41:09.747 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:41:09 smithi092 ceph-mon[37563]: 6.14 scrub starts 2024-03-11T21:41:09.747 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:41:09 smithi092 ceph-mon[37563]: 6.14 scrub ok 2024-03-11T21:41:09.747 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:41:09 smithi092 ceph-mon[37563]: 6.1f scrub ok 2024-03-11T21:41:09.774 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:41:09.805 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:41:09.806 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:41:09.806 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:41:09.831 INFO:tasks.radosbench.radosbench.0.smithi092.stdout: 17 16 41276 41260 151.674 109.688 0.0145201 0.00658555 2024-03-11T21:41:09.836 INFO:teuthology.orchestra.run.smithi092.stderr:Inferring config /var/lib/ceph/00db0016-dfed-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T21:41:09.907 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:41:09.918 INFO:teuthology.orchestra.run.smithi092.stderr:Inferring config /var/lib/ceph/00db0016-dfed-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T21:41:09.934 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:41:09.935 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:41:09.935 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:41:09.938 INFO:teuthology.orchestra.run.smithi092.stderr:Inferring config /var/lib/ceph/00db0016-dfed-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T21:41:10.036 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:41:10.073 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:41:10.074 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:41:10.074 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:41:10.080 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:41:09 smithi097 ceph-mon[33317]: 6.14 scrub starts 2024-03-11T21:41:10.080 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:41:09 smithi097 ceph-mon[33317]: 6.14 scrub ok 2024-03-11T21:41:10.080 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:41:09 smithi097 ceph-mon[33317]: 6.1f scrub ok 2024-03-11T21:41:10.175 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:41:10.217 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:41:10.218 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:41:10.218 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:41:10.319 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:41:10.358 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:41:10.359 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:41:10.359 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:41:10.460 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:41:10.507 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:41:10.507 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:41:10.507 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:41:10.608 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:41:10.645 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:41:10.646 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:41:10.646 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:41:10.747 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:41:10.783 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:41:10.784 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:41:10.785 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:41:10.831 INFO:tasks.radosbench.radosbench.0.smithi092.stdout: 18 16 43138 43122 149.712 116.375 0.0119516 0.0066739 2024-03-11T21:41:10.864 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:41:10 smithi092 ceph-mon[29572]: 5.b scrub starts 2024-03-11T21:41:10.864 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:41:10 smithi092 ceph-mon[29572]: 5.b scrub ok 2024-03-11T21:41:10.864 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:41:10 smithi092 ceph-mon[29572]: 1.0 scrub starts 2024-03-11T21:41:10.864 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:41:10 smithi092 ceph-mon[29572]: 1.0 scrub ok 2024-03-11T21:41:10.865 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:41:10 smithi092 ceph-mon[29572]: pgmap v726: 148 pgs: 148 active+clean; 2.3 GiB data, 10 GiB used, 705 GiB / 715 GiB avail; 1.0 KiB/s rd, 138 MiB/s wr, 2.20k op/s 2024-03-11T21:41:10.865 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:41:10 smithi092 ceph-mon[29572]: 5.1f scrub starts 2024-03-11T21:41:10.865 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:41:10 smithi092 ceph-mon[29572]: 5.1f scrub ok 2024-03-11T21:41:10.865 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:41:10 smithi092 ceph-mon[37563]: 5.b scrub starts 2024-03-11T21:41:10.865 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:41:10 smithi092 ceph-mon[37563]: 5.b scrub ok 2024-03-11T21:41:10.865 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:41:10 smithi092 ceph-mon[37563]: 1.0 scrub starts 2024-03-11T21:41:10.865 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:41:10 smithi092 ceph-mon[37563]: 1.0 scrub ok 2024-03-11T21:41:10.865 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:41:10 smithi092 ceph-mon[37563]: pgmap v726: 148 pgs: 148 active+clean; 2.3 GiB data, 10 GiB used, 705 GiB / 715 GiB avail; 1.0 KiB/s rd, 138 MiB/s wr, 2.20k op/s 2024-03-11T21:41:10.865 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:41:10 smithi092 ceph-mon[37563]: 5.1f scrub starts 2024-03-11T21:41:10.865 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:41:10 smithi092 ceph-mon[37563]: 5.1f scrub ok 2024-03-11T21:41:10.886 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:41:10.929 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:41:10.929 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:41:10.929 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:41:11.030 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:41:11.068 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:41:11.069 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:41:11.069 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:41:11.080 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:41:10 smithi097 ceph-mon[33317]: 5.b scrub starts 2024-03-11T21:41:11.080 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:41:10 smithi097 ceph-mon[33317]: 5.b scrub ok 2024-03-11T21:41:11.080 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:41:10 smithi097 ceph-mon[33317]: 1.0 scrub starts 2024-03-11T21:41:11.080 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:41:10 smithi097 ceph-mon[33317]: 1.0 scrub ok 2024-03-11T21:41:11.080 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:41:10 smithi097 ceph-mon[33317]: pgmap v726: 148 pgs: 148 active+clean; 2.3 GiB data, 10 GiB used, 705 GiB / 715 GiB avail; 1.0 KiB/s rd, 138 MiB/s wr, 2.20k op/s 2024-03-11T21:41:11.081 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:41:10 smithi097 ceph-mon[33317]: 5.1f scrub starts 2024-03-11T21:41:11.081 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:41:10 smithi097 ceph-mon[33317]: 5.1f scrub ok 2024-03-11T21:41:11.169 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:41:11.208 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:41:11.208 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:41:11.209 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:41:11.310 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:41:11.352 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:41:11.353 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:41:11.353 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:41:11.454 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:41:11.497 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:41:11.497 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:41:11.498 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:41:11.599 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:41:11.634 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:41:11.634 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:41:11.635 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:41:11.736 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:41:11.774 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:41:11.775 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:41:11.775 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:41:11.831 INFO:tasks.radosbench.radosbench.0.smithi092.stdout: 19 16 44954 44938 147.806 113.5 0.0095333 0.00676011 2024-03-11T21:41:11.864 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:41:11 smithi092 ceph-mon[29572]: 4.1e scrub starts 2024-03-11T21:41:11.864 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:41:11 smithi092 ceph-mon[29572]: 4.1e scrub ok 2024-03-11T21:41:11.864 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:41:11 smithi092 ceph-mon[29572]: 6.1e scrub starts 2024-03-11T21:41:11.865 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:41:11 smithi092 ceph-mon[29572]: 6.1e scrub ok 2024-03-11T21:41:11.865 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:41:11 smithi092 ceph-mon[29572]: 2.3 scrub starts 2024-03-11T21:41:11.865 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:41:11 smithi092 ceph-mon[29572]: 2.3 scrub ok 2024-03-11T21:41:11.865 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:41:11 smithi092 ceph-mon[37563]: 4.1e scrub starts 2024-03-11T21:41:11.865 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:41:11 smithi092 ceph-mon[37563]: 4.1e scrub ok 2024-03-11T21:41:11.865 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:41:11 smithi092 ceph-mon[37563]: 6.1e scrub starts 2024-03-11T21:41:11.865 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:41:11 smithi092 ceph-mon[37563]: 6.1e scrub ok 2024-03-11T21:41:11.865 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:41:11 smithi092 ceph-mon[37563]: 2.3 scrub starts 2024-03-11T21:41:11.865 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:41:11 smithi092 ceph-mon[37563]: 2.3 scrub ok 2024-03-11T21:41:11.876 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:41:11.913 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:41:11.914 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:41:11.914 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:41:12.015 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:41:12.051 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:41:12.051 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:41:12.052 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:41:12.080 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:41:11 smithi097 ceph-mon[33317]: 4.1e scrub starts 2024-03-11T21:41:12.080 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:41:11 smithi097 ceph-mon[33317]: 4.1e scrub ok 2024-03-11T21:41:12.080 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:41:11 smithi097 ceph-mon[33317]: 6.1e scrub starts 2024-03-11T21:41:12.080 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:41:11 smithi097 ceph-mon[33317]: 6.1e scrub ok 2024-03-11T21:41:12.080 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:41:11 smithi097 ceph-mon[33317]: 2.3 scrub starts 2024-03-11T21:41:12.080 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:41:11 smithi097 ceph-mon[33317]: 2.3 scrub ok 2024-03-11T21:41:12.153 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:41:12.189 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:41:12.190 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:41:12.190 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:41:12.291 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:41:12.329 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:41:12.330 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:41:12.330 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:41:12.431 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:41:12.467 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:41:12.468 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:41:12.469 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:41:12.569 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:41:12.613 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:41:12.614 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:41:12.614 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:41:12.715 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:41:12.749 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:41:12.749 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:41:12.749 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:41:12.831 INFO:tasks.radosbench.radosbench.0.smithi092.stdout:2024-03-11T21:41:12.846034+0000 min lat: 0.00121416 max lat: 0.225124 avg lat: 0.00683894 2024-03-11T21:41:12.831 INFO:tasks.radosbench.radosbench.0.smithi092.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-03-11T21:41:12.832 INFO:tasks.radosbench.radosbench.0.smithi092.stdout: 20 16 46776 46760 146.109 113.875 0.00817803 0.00683894 2024-03-11T21:41:12.850 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:41:12.887 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:41:12.888 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:41:12.888 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:41:12.964 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:41:12 smithi092 ceph-mon[29572]: pgmap v727: 148 pgs: 148 active+clean; 2.3 GiB data, 10 GiB used, 705 GiB / 715 GiB avail; 1023 B/s rd, 136 MiB/s wr, 2.17k op/s 2024-03-11T21:41:12.964 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:41:12 smithi092 ceph-mon[29572]: 4.d scrub starts 2024-03-11T21:41:12.964 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:41:12 smithi092 ceph-mon[29572]: 4.d scrub ok 2024-03-11T21:41:12.965 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:41:12 smithi092 ceph-mon[29572]: 6.5 scrub starts 2024-03-11T21:41:12.965 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:41:12 smithi092 ceph-mon[29572]: 6.5 scrub ok 2024-03-11T21:41:12.965 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:41:12 smithi092 ceph-mon[37563]: pgmap v727: 148 pgs: 148 active+clean; 2.3 GiB data, 10 GiB used, 705 GiB / 715 GiB avail; 1023 B/s rd, 136 MiB/s wr, 2.17k op/s 2024-03-11T21:41:12.965 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:41:12 smithi092 ceph-mon[37563]: 4.d scrub starts 2024-03-11T21:41:12.965 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:41:12 smithi092 ceph-mon[37563]: 4.d scrub ok 2024-03-11T21:41:12.965 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:41:12 smithi092 ceph-mon[37563]: 6.5 scrub starts 2024-03-11T21:41:12.965 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:41:12 smithi092 ceph-mon[37563]: 6.5 scrub ok 2024-03-11T21:41:12.989 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:41:13.029 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:41:13.029 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:41:13.030 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:41:13.080 INFO:journalctl@ceph.iscsi.iscsi.a.smithi097.stdout:Mar 11 21:41:12 smithi097 ceph-00db0016-dfed-11ee-95c8-87774f69a715-iscsi-iscsi-a[60943]: debug there is no tcmu-runner data available 2024-03-11T21:41:13.081 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:41:12 smithi097 ceph-mon[33317]: pgmap v727: 148 pgs: 148 active+clean; 2.3 GiB data, 10 GiB used, 705 GiB / 715 GiB avail; 1023 B/s rd, 136 MiB/s wr, 2.17k op/s 2024-03-11T21:41:13.081 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:41:12 smithi097 ceph-mon[33317]: 4.d scrub starts 2024-03-11T21:41:13.081 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:41:12 smithi097 ceph-mon[33317]: 4.d scrub ok 2024-03-11T21:41:13.081 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:41:12 smithi097 ceph-mon[33317]: 6.5 scrub starts 2024-03-11T21:41:13.081 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:41:12 smithi097 ceph-mon[33317]: 6.5 scrub ok 2024-03-11T21:41:13.130 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:41:13.165 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:41:13.166 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:41:13.166 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:41:13.267 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:41:13.307 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:41:13.308 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:41:13.308 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:41:13.409 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:41:13.443 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:41:13.444 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:41:13.444 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:41:13.545 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:41:13.583 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:41:13.584 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:41:13.584 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:41:13.685 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:41:13.724 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:41:13.725 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:41:13.725 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:41:13.826 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:41:13.832 INFO:tasks.radosbench.radosbench.0.smithi092.stdout: 21 16 48612 48596 144.615 114.75 0.00510085 0.00691016 2024-03-11T21:41:13.864 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:41:13.865 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:41:13.865 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:41:13.966 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:41:14.004 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:41:14.005 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:41:14.005 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:41:14.016 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:41:13 smithi092 ceph-mon[29572]: 6.a scrub starts 2024-03-11T21:41:14.016 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:41:13 smithi092 ceph-mon[29572]: 6.a scrub ok 2024-03-11T21:41:14.016 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:41:13 smithi092 ceph-mon[29572]: from='client.24469 v1:172.21.15.97:0/404208310' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:41:14.017 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:41:13 smithi092 ceph-mon[37563]: 6.a scrub starts 2024-03-11T21:41:14.017 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:41:13 smithi092 ceph-mon[37563]: 6.a scrub ok 2024-03-11T21:41:14.017 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:41:13 smithi092 ceph-mon[37563]: from='client.24469 v1:172.21.15.97:0/404208310' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:41:14.080 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:41:13 smithi097 ceph-mon[33317]: 6.a scrub starts 2024-03-11T21:41:14.080 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:41:13 smithi097 ceph-mon[33317]: 6.a scrub ok 2024-03-11T21:41:14.080 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:41:13 smithi097 ceph-mon[33317]: from='client.24469 v1:172.21.15.97:0/404208310' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:41:14.107 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:41:14.138 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:41:14.139 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:41:14.140 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:41:14.240 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:41:14.282 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:41:14.283 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:41:14.283 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:41:14.384 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:41:14.418 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:41:14.418 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:41:14.419 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:41:14.519 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:41:14.575 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:41:14.575 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:41:14.576 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:41:14.677 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:41:14.709 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:41:14.709 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:41:14.710 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:41:14.811 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:41:14.831 INFO:tasks.radosbench.radosbench.0.smithi092.stdout: 22 16 50213 50197 142.589 100.062 0.0125813 0.00700825 2024-03-11T21:41:14.848 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:41:14 smithi092 ceph-mon[29572]: pgmap v728: 148 pgs: 148 active+clean; 2.7 GiB data, 12 GiB used, 704 GiB / 715 GiB avail; 952 B/s rd, 111 MiB/s wr, 1.77k op/s 2024-03-11T21:41:14.848 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:41:14 smithi092 ceph-mon[29572]: 2.f scrub starts 2024-03-11T21:41:14.848 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:41:14 smithi092 ceph-mon[29572]: 2.f scrub ok 2024-03-11T21:41:14.848 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:41:14 smithi092 ceph-mon[29572]: 6.19 scrub starts 2024-03-11T21:41:14.848 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:41:14 smithi092 ceph-mon[29572]: 6.19 scrub ok 2024-03-11T21:41:14.848 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:41:14 smithi092 ceph-mon[29572]: 2.1c scrub starts 2024-03-11T21:41:14.848 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:41:14 smithi092 ceph-mon[29572]: 2.1c scrub ok 2024-03-11T21:41:14.849 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:41:14 smithi092 ceph-mon[37563]: pgmap v728: 148 pgs: 148 active+clean; 2.7 GiB data, 12 GiB used, 704 GiB / 715 GiB avail; 952 B/s rd, 111 MiB/s wr, 1.77k op/s 2024-03-11T21:41:14.849 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:41:14 smithi092 ceph-mon[37563]: 2.f scrub starts 2024-03-11T21:41:14.849 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:41:14 smithi092 ceph-mon[37563]: 2.f scrub ok 2024-03-11T21:41:14.849 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:41:14 smithi092 ceph-mon[37563]: 6.19 scrub starts 2024-03-11T21:41:14.849 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:41:14 smithi092 ceph-mon[37563]: 6.19 scrub ok 2024-03-11T21:41:14.849 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:41:14 smithi092 ceph-mon[37563]: 2.1c scrub starts 2024-03-11T21:41:14.849 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:41:14 smithi092 ceph-mon[37563]: 2.1c scrub ok 2024-03-11T21:41:14.853 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:41:14.854 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:41:14.854 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:41:14.955 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:41:14.992 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:41:14.992 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:41:14.993 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:41:15.080 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:41:14 smithi097 ceph-mon[33317]: pgmap v728: 148 pgs: 148 active+clean; 2.7 GiB data, 12 GiB used, 704 GiB / 715 GiB avail; 952 B/s rd, 111 MiB/s wr, 1.77k op/s 2024-03-11T21:41:15.080 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:41:14 smithi097 ceph-mon[33317]: 2.f scrub starts 2024-03-11T21:41:15.080 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:41:14 smithi097 ceph-mon[33317]: 2.f scrub ok 2024-03-11T21:41:15.080 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:41:14 smithi097 ceph-mon[33317]: 6.19 scrub starts 2024-03-11T21:41:15.081 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:41:14 smithi097 ceph-mon[33317]: 6.19 scrub ok 2024-03-11T21:41:15.081 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:41:14 smithi097 ceph-mon[33317]: 2.1c scrub starts 2024-03-11T21:41:15.081 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:41:14 smithi097 ceph-mon[33317]: 2.1c scrub ok 2024-03-11T21:41:15.094 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:41:15.132 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:41:15.132 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:41:15.133 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:41:15.212 INFO:teuthology.orchestra.run.smithi092.stdout: 2024-03-11T21:41:15.216 INFO:teuthology.orchestra.run.smithi092.stderr:dumped all 2024-03-11T21:41:15.234 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:41:15.275 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:41:15.275 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:41:15.276 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:41:15.377 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:41:15.406 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:41:15.406 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:41:15.407 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:41:15.509 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:41:15.546 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:41:15.546 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:41:15.547 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:41:15.648 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:41:15.681 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:41:15.682 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:41:15.682 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:41:15.783 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:41:15.824 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:41:15.824 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:41:15.825 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:41:15.831 INFO:tasks.radosbench.radosbench.0.smithi092.stdout: 23 16 51723 51707 140.493 94.375 0.0064239 0.00711095 2024-03-11T21:41:15.864 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:41:15 smithi092 ceph-mon[29572]: 4.e scrub starts 2024-03-11T21:41:15.864 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:41:15 smithi092 ceph-mon[29572]: 4.e scrub ok 2024-03-11T21:41:15.865 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:41:15 smithi092 ceph-mon[29572]: 2.14 scrub starts 2024-03-11T21:41:15.865 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:41:15 smithi092 ceph-mon[29572]: 2.14 scrub ok 2024-03-11T21:41:15.865 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:41:15 smithi092 ceph-mon[29572]: from='client.15759 v1:172.21.15.92:0/2877137765' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-11T21:41:15.865 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:41:15 smithi092 ceph-mon[37563]: 4.e scrub starts 2024-03-11T21:41:15.865 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:41:15 smithi092 ceph-mon[37563]: 4.e scrub ok 2024-03-11T21:41:15.865 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:41:15 smithi092 ceph-mon[37563]: 2.14 scrub starts 2024-03-11T21:41:15.865 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:41:15 smithi092 ceph-mon[37563]: 2.14 scrub ok 2024-03-11T21:41:15.865 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:41:15 smithi092 ceph-mon[37563]: from='client.15759 v1:172.21.15.92:0/2877137765' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-11T21:41:15.926 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:41:15.964 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:41:15.965 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:41:15.965 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:41:16.058 INFO:teuthology.orchestra.run.smithi092.stdout:{"pg_ready":true,"pg_map":{"version":728,"stamp":"2024-03-11T21:41:13.332721+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":2927827368,"num_objects":44864,"num_object_clones":0,"num_object_copies":134592,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":44864,"num_whiteouts":0,"num_read":2257,"num_read_kb":2105,"num_write":45567,"num_write_kb":2861085,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":203,"num_bytes_recovered":2955718,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":45659,"ondisk_log_size":45659,"up":444,"acting":444,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":444,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":749961216,"kb_used":12095356,"kb_used_data":8203324,"kb_used_omap":15,"kb_used_meta":3891952,"kb_avail":737865860,"statfs":{"total":767960285184,"available":755574640640,"internally_reserved":0,"allocated":8400203776,"data_stored":8391922160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15601,"internal_metadata":3985359631},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":1246429184,"num_objects":19019,"num_object_clones":0,"num_object_copies":57057,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19019,"num_whiteouts":0,"num_read":10,"num_read_kb":10,"num_write":19019,"num_write_kb":1217216,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"10.753034"},"pg_stats":[{"pgid":"6.1b","version":"0'0","reported_seq":582,"reported_epoch":366,"state":"active+clean","last_fresh":"2024-03-11T21:41:02.594909+0000","last_change":"2024-03-11T21:36:35.074931+0000","last_active":"2024-03-11T21:41:02.594909+0000","last_peered":"2024-03-11T21:41:02.594909+0000","last_clean":"2024-03-11T21:41:02.594909+0000","last_became_active":"2024-03-11T21:36:35.072636+0000","last_became_peered":"2024-03-11T21:36:35.072636+0000","last_unstale":"2024-03-11T21:41:02.594909+0000","last_undegraded":"2024-03-11T21:41:02.594909+0000","last_fullsized":"2024-03-11T21:41:02.594909+0000","mapping_epoch":277,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":278,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:35:07.021520+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:43.559452+0000","last_clean_scrub_stamp":"2024-03-11T21:35:07.021520+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":647,"reported_epoch":366,"state":"active+clean","last_fresh":"2024-03-11T21:41:02.591480+0000","last_change":"2024-03-11T21:36:00.720089+0000","last_active":"2024-03-11T21:41:02.591480+0000","last_peered":"2024-03-11T21:41:02.591480+0000","last_clean":"2024-03-11T21:41:02.591480+0000","last_became_active":"2024-03-11T21:25:42.578893+0000","last_became_peered":"2024-03-11T21:25:42.578893+0000","last_unstale":"2024-03-11T21:41:02.591480+0000","last_undegraded":"2024-03-11T21:41:02.591480+0000","last_fullsized":"2024-03-11T21:41:02.591480+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:36:00.719956+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:41.544550+0000","last_clean_scrub_stamp":"2024-03-11T21:36:00.719956+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1f","version":"0'0","reported_seq":676,"reported_epoch":366,"state":"active+clean","last_fresh":"2024-03-11T21:41:02.588645+0000","last_change":"2024-03-11T21:36:00.843838+0000","last_active":"2024-03-11T21:41:02.588645+0000","last_peered":"2024-03-11T21:41:02.588645+0000","last_clean":"2024-03-11T21:41:02.588645+0000","last_became_active":"2024-03-11T21:34:41.144945+0000","last_became_peered":"2024-03-11T21:34:41.144945+0000","last_unstale":"2024-03-11T21:41:02.588645+0000","last_undegraded":"2024-03-11T21:41:02.588645+0000","last_fullsized":"2024-03-11T21:41:02.588645+0000","mapping_epoch":226,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":227,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:36:00.843718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:37.512018+0000","last_clean_scrub_stamp":"2024-03-11T21:36:00.843718+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,4],"acting":[0,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.19","version":"260'27","reported_seq":629,"reported_epoch":366,"state":"active+clean","last_fresh":"2024-03-11T21:41:02.592047+0000","last_change":"2024-03-11T21:36:35.089901+0000","last_active":"2024-03-11T21:41:02.592047+0000","last_peered":"2024-03-11T21:41:02.592047+0000","last_clean":"2024-03-11T21:41:02.592047+0000","last_became_active":"2024-03-11T21:36:35.067652+0000","last_became_peered":"2024-03-11T21:36:35.067652+0000","last_unstale":"2024-03-11T21:41:02.592047+0000","last_undegraded":"2024-03-11T21:41:02.592047+0000","last_fullsized":"2024-03-11T21:41:02.592047+0000","mapping_epoch":277,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":278,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'15","last_scrub_stamp":"2024-03-11T21:35:22.099374+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:39.525554+0000","last_clean_scrub_stamp":"2024-03-11T21:35:22.099374+0000","objects_scrubbed":0,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":42,"num_read_kb":33,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.1a","version":"0'0","reported_seq":643,"reported_epoch":366,"state":"active+clean","last_fresh":"2024-03-11T21:41:02.614528+0000","last_change":"2024-03-11T21:41:02.614569+0000","last_active":"2024-03-11T21:41:02.614528+0000","last_peered":"2024-03-11T21:41:02.614528+0000","last_clean":"2024-03-11T21:41:02.614528+0000","last_became_active":"2024-03-11T21:25:44.590923+0000","last_became_peered":"2024-03-11T21:25:44.590923+0000","last_unstale":"2024-03-11T21:41:02.614528+0000","last_undegraded":"2024-03-11T21:41:02.614528+0000","last_fullsized":"2024-03-11T21:41:02.614528+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:41:02.614438+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:43.559452+0000","last_clean_scrub_stamp":"2024-03-11T21:41:02.614438+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:42:20.282283+0000","scrub_duration":38,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,1],"acting":[4,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":659,"reported_epoch":366,"state":"active+clean","last_fresh":"2024-03-11T21:41:02.583540+0000","last_change":"2024-03-11T21:36:15.576138+0000","last_active":"2024-03-11T21:41:02.583540+0000","last_peered":"2024-03-11T21:41:02.583540+0000","last_clean":"2024-03-11T21:41:02.583540+0000","last_became_active":"2024-03-11T21:34:41.154216+0000","last_became_peered":"2024-03-11T21:34:41.154216+0000","last_unstale":"2024-03-11T21:41:02.583540+0000","last_undegraded":"2024-03-11T21:41:02.583540+0000","last_fullsized":"2024-03-11T21:41:02.583540+0000","mapping_epoch":226,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":227,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:36:15.575887+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:41.544550+0000","last_clean_scrub_stamp":"2024-03-11T21:36:15.575887+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,7],"acting":[1,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1e","version":"0'0","reported_seq":604,"reported_epoch":366,"state":"active+clean","last_fresh":"2024-03-11T21:41:04.598119+0000","last_change":"2024-03-11T21:41:04.598202+0000","last_active":"2024-03-11T21:41:04.598119+0000","last_peered":"2024-03-11T21:41:04.598119+0000","last_clean":"2024-03-11T21:41:04.598119+0000","last_became_active":"2024-03-11T21:36:35.061786+0000","last_became_peered":"2024-03-11T21:36:35.061786+0000","last_unstale":"2024-03-11T21:41:04.598119+0000","last_undegraded":"2024-03-11T21:41:04.598119+0000","last_fullsized":"2024-03-11T21:41:04.598119+0000","mapping_epoch":277,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":278,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:41:04.597983+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:37.512018+0000","last_clean_scrub_stamp":"2024-03-11T21:41:04.597983+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:42:06.765681+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.18","version":"260'15","reported_seq":684,"reported_epoch":366,"state":"active+clean","last_fresh":"2024-03-11T21:41:02.591310+0000","last_change":"2024-03-11T21:36:09.662765+0000","last_active":"2024-03-11T21:41:02.591310+0000","last_peered":"2024-03-11T21:41:02.591310+0000","last_clean":"2024-03-11T21:41:02.591310+0000","last_became_active":"2024-03-11T21:25:40.552216+0000","last_became_peered":"2024-03-11T21:25:40.552216+0000","last_unstale":"2024-03-11T21:41:02.591310+0000","last_undegraded":"2024-03-11T21:41:02.591310+0000","last_fullsized":"2024-03-11T21:41:02.591310+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"260'15","last_scrub_stamp":"2024-03-11T21:36:09.662615+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:39.525554+0000","last_clean_scrub_stamp":"2024-03-11T21:36:09.662615+0000","objects_scrubbed":4,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":651,"reported_epoch":366,"state":"active+clean","last_fresh":"2024-03-11T21:41:02.587222+0000","last_change":"2024-03-11T21:36:35.068915+0000","last_active":"2024-03-11T21:41:02.587222+0000","last_peered":"2024-03-11T21:41:02.587222+0000","last_clean":"2024-03-11T21:41:02.587222+0000","last_became_active":"2024-03-11T21:36:35.068453+0000","last_became_peered":"2024-03-11T21:36:35.068453+0000","last_unstale":"2024-03-11T21:41:02.587222+0000","last_undegraded":"2024-03-11T21:41:02.587222+0000","last_fullsized":"2024-03-11T21:41:02.587222+0000","mapping_epoch":277,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":278,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:35:26.190634+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:32:12.654573+0000","last_clean_scrub_stamp":"2024-03-11T21:35:26.190634+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:41:03.775119+0000","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3],"acting":[5,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":307,"reported_epoch":366,"state":"active+clean","last_fresh":"2024-03-11T21:41:02.590117+0000","last_change":"2024-03-11T21:41:01.164718+0000","last_active":"2024-03-11T21:41:02.590117+0000","last_peered":"2024-03-11T21:41:02.590117+0000","last_clean":"2024-03-11T21:41:02.590117+0000","last_became_active":"2024-03-11T21:34:41.145299+0000","last_became_peered":"2024-03-11T21:34:41.145299+0000","last_unstale":"2024-03-11T21:41:02.590117+0000","last_undegraded":"2024-03-11T21:41:02.590117+0000","last_fullsized":"2024-03-11T21:41:02.590117+0000","mapping_epoch":226,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":227,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:41:01.164619+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:41.544550+0000","last_clean_scrub_stamp":"2024-03-11T21:41:01.164619+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:42:08.050312+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.1d","version":"0'0","reported_seq":670,"reported_epoch":366,"state":"active+clean","last_fresh":"2024-03-11T21:41:02.592455+0000","last_change":"2024-03-11T21:41:02.217938+0000","last_active":"2024-03-11T21:41:02.592455+0000","last_peered":"2024-03-11T21:41:02.592455+0000","last_clean":"2024-03-11T21:41:02.592455+0000","last_became_active":"2024-03-11T21:25:38.542628+0000","last_became_peered":"2024-03-11T21:25:38.542628+0000","last_unstale":"2024-03-11T21:41:02.592455+0000","last_undegraded":"2024-03-11T21:41:02.592455+0000","last_fullsized":"2024-03-11T21:41:02.592455+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:41:02.217799+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:37.512018+0000","last_clean_scrub_stamp":"2024-03-11T21:41:02.217799+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:42:05.036261+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,0],"acting":[7,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.1b","version":"260'7","reported_seq":678,"reported_epoch":366,"state":"active+clean","last_fresh":"2024-03-11T21:41:02.591121+0000","last_change":"2024-03-11T21:40:29.597326+0000","last_active":"2024-03-11T21:41:02.591121+0000","last_peered":"2024-03-11T21:41:02.591121+0000","last_clean":"2024-03-11T21:41:02.591121+0000","last_became_active":"2024-03-11T21:36:35.057678+0000","last_became_peered":"2024-03-11T21:36:35.057678+0000","last_unstale":"2024-03-11T21:41:02.591121+0000","last_undegraded":"2024-03-11T21:41:02.591121+0000","last_fullsized":"2024-03-11T21:41:02.591121+0000","mapping_epoch":277,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":278,"parent":"0.0","parent_split_bits":0,"last_scrub":"260'7","last_scrub_stamp":"2024-03-11T21:40:29.597219+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:39.525554+0000","last_clean_scrub_stamp":"2024-03-11T21:40:29.597219+0000","objects_scrubbed":2,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:41:33.394843+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":14,"num_read_kb":10,"num_write":8,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,1],"acting":[4,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":643,"reported_epoch":366,"state":"active+clean","last_fresh":"2024-03-11T21:41:02.588941+0000","last_change":"2024-03-11T21:35:39.691825+0000","last_active":"2024-03-11T21:41:02.588941+0000","last_peered":"2024-03-11T21:41:02.588941+0000","last_clean":"2024-03-11T21:41:02.588941+0000","last_became_active":"2024-03-11T21:25:44.589786+0000","last_became_peered":"2024-03-11T21:25:44.589786+0000","last_unstale":"2024-03-11T21:41:02.588941+0000","last_undegraded":"2024-03-11T21:41:02.588941+0000","last_fullsized":"2024-03-11T21:41:02.588941+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:35:39.691690+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:43.559452+0000","last_clean_scrub_stamp":"2024-03-11T21:35:39.691690+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":18,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,7],"acting":[0,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":666,"reported_epoch":366,"state":"active+clean","last_fresh":"2024-03-11T21:41:02.589121+0000","last_change":"2024-03-11T21:40:27.604550+0000","last_active":"2024-03-11T21:41:02.589121+0000","last_peered":"2024-03-11T21:41:02.589121+0000","last_clean":"2024-03-11T21:41:02.589121+0000","last_became_active":"2024-03-11T21:34:41.147185+0000","last_became_peered":"2024-03-11T21:34:41.147185+0000","last_unstale":"2024-03-11T21:41:02.589121+0000","last_undegraded":"2024-03-11T21:41:02.589121+0000","last_fullsized":"2024-03-11T21:41:02.589121+0000","mapping_epoch":226,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":227,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:40:27.604453+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:32:21.535034+0000","last_clean_scrub_stamp":"2024-03-11T21:40:27.604453+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:41:27.648926+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,7],"acting":[5,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1c","version":"0'0","reported_seq":662,"reported_epoch":366,"state":"active+clean","last_fresh":"2024-03-11T21:41:02.590542+0000","last_change":"2024-03-11T21:35:28.789693+0000","last_active":"2024-03-11T21:41:02.590542+0000","last_peered":"2024-03-11T21:41:02.590542+0000","last_clean":"2024-03-11T21:41:02.590542+0000","last_became_active":"2024-03-11T21:25:38.541364+0000","last_became_peered":"2024-03-11T21:25:38.541364+0000","last_unstale":"2024-03-11T21:41:02.590542+0000","last_undegraded":"2024-03-11T21:41:02.590542+0000","last_fullsized":"2024-03-11T21:41:02.590542+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:35:28.789592+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:28:32.566437+0000","last_clean_scrub_stamp":"2024-03-11T21:35:28.789592+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.1a","version":"350'19","reported_seq":706,"reported_epoch":366,"state":"active+clean","last_fresh":"2024-03-11T21:41:02.593235+0000","last_change":"2024-03-11T21:40:20.484449+0000","last_active":"2024-03-11T21:41:02.593235+0000","last_peered":"2024-03-11T21:41:02.593235+0000","last_clean":"2024-03-11T21:41:02.593235+0000","last_became_active":"2024-03-11T21:36:35.057029+0000","last_became_peered":"2024-03-11T21:36:35.057029+0000","last_unstale":"2024-03-11T21:41:02.593235+0000","last_undegraded":"2024-03-11T21:41:02.593235+0000","last_fullsized":"2024-03-11T21:41:02.593235+0000","mapping_epoch":277,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":278,"parent":"0.0","parent_split_bits":0,"last_scrub":"350'19","last_scrub_stamp":"2024-03-11T21:40:20.484308+0000","last_deep_scrub":"350'19","last_deep_scrub_stamp":"2024-03-11T21:40:20.484308+0000","last_clean_scrub_stamp":"2024-03-11T21:40:20.484308+0000","objects_scrubbed":4,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:41:24.916676+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0],"acting":[4,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1f","version":"0'0","reported_seq":527,"reported_epoch":366,"state":"active+clean","last_fresh":"2024-03-11T21:41:08.652920+0000","last_change":"2024-03-11T21:41:08.652958+0000","last_active":"2024-03-11T21:41:08.652920+0000","last_peered":"2024-03-11T21:41:08.652920+0000","last_clean":"2024-03-11T21:41:08.652920+0000","last_became_active":"2024-03-11T21:36:35.071755+0000","last_became_peered":"2024-03-11T21:36:35.071755+0000","last_unstale":"2024-03-11T21:41:08.652920+0000","last_undegraded":"2024-03-11T21:41:08.652920+0000","last_fullsized":"2024-03-11T21:41:08.652920+0000","mapping_epoch":277,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":278,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:41:08.652836+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:43.559452+0000","last_clean_scrub_stamp":"2024-03-11T21:41:08.652836+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:42:25.863145+0000","scrub_duration":137,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1c","version":"0'0","reported_seq":656,"reported_epoch":366,"state":"active+clean","last_fresh":"2024-03-11T21:41:02.591752+0000","last_change":"2024-03-11T21:40:07.678642+0000","last_active":"2024-03-11T21:41:02.591752+0000","last_peered":"2024-03-11T21:41:02.591752+0000","last_clean":"2024-03-11T21:41:02.591752+0000","last_became_active":"2024-03-11T21:36:35.067127+0000","last_became_peered":"2024-03-11T21:36:35.067127+0000","last_unstale":"2024-03-11T21:41:02.591752+0000","last_undegraded":"2024-03-11T21:41:02.591752+0000","last_fullsized":"2024-03-11T21:41:02.591752+0000","mapping_epoch":277,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":278,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:40:07.678561+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:40:07.678561+0000","last_clean_scrub_stamp":"2024-03-11T21:40:07.678561+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:41:27.033148+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1b","version":"0'0","reported_seq":567,"reported_epoch":366,"state":"active+clean","last_fresh":"2024-03-11T21:41:02.595007+0000","last_change":"2024-03-11T21:36:35.075385+0000","last_active":"2024-03-11T21:41:02.595007+0000","last_peered":"2024-03-11T21:41:02.595007+0000","last_clean":"2024-03-11T21:41:02.595007+0000","last_became_active":"2024-03-11T21:36:35.073959+0000","last_became_peered":"2024-03-11T21:36:35.073959+0000","last_unstale":"2024-03-11T21:41:02.595007+0000","last_undegraded":"2024-03-11T21:41:02.595007+0000","last_fullsized":"2024-03-11T21:41:02.595007+0000","mapping_epoch":277,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":278,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:34:27.998724+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:34:27.998724+0000","last_clean_scrub_stamp":"2024-03-11T21:34:27.998724+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1d","version":"260'18","reported_seq":611,"reported_epoch":366,"state":"active+clean","last_fresh":"2024-03-11T21:41:02.595609+0000","last_change":"2024-03-11T21:36:35.080207+0000","last_active":"2024-03-11T21:41:02.595609+0000","last_peered":"2024-03-11T21:41:02.595609+0000","last_clean":"2024-03-11T21:41:02.595609+0000","last_became_active":"2024-03-11T21:36:35.055880+0000","last_became_peered":"2024-03-11T21:36:35.055880+0000","last_unstale":"2024-03-11T21:41:02.595609+0000","last_undegraded":"2024-03-11T21:41:02.595609+0000","last_fullsized":"2024-03-11T21:41:02.595609+0000","mapping_epoch":277,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":278,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'12","last_scrub_stamp":"2024-03-11T21:35:07.971581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:39.525554+0000","last_clean_scrub_stamp":"2024-03-11T21:35:07.971581+0000","objects_scrubbed":0,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":33,"num_read_kb":24,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.1e","version":"0'0","reported_seq":644,"reported_epoch":366,"state":"active+clean","last_fresh":"2024-03-11T21:41:10.551140+0000","last_change":"2024-03-11T21:41:10.551185+0000","last_active":"2024-03-11T21:41:10.551140+0000","last_peered":"2024-03-11T21:41:10.551140+0000","last_clean":"2024-03-11T21:41:10.551140+0000","last_became_active":"2024-03-11T21:25:45.300355+0000","last_became_peered":"2024-03-11T21:25:45.300355+0000","last_unstale":"2024-03-11T21:41:10.551140+0000","last_undegraded":"2024-03-11T21:41:10.551140+0000","last_fullsized":"2024-03-11T21:41:10.551140+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:41:10.550887+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:43.559452+0000","last_clean_scrub_stamp":"2024-03-11T21:41:10.550887+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:42:31.211347+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5],"acting":[4,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1d","version":"0'0","reported_seq":647,"reported_epoch":366,"state":"active+clean","last_fresh":"2024-03-11T21:41:02.590389+0000","last_change":"2024-03-11T21:36:07.624196+0000","last_active":"2024-03-11T21:41:02.590389+0000","last_peered":"2024-03-11T21:41:02.590389+0000","last_clean":"2024-03-11T21:41:02.590389+0000","last_became_active":"2024-03-11T21:25:42.579407+0000","last_became_peered":"2024-03-11T21:25:42.579407+0000","last_unstale":"2024-03-11T21:41:02.590389+0000","last_undegraded":"2024-03-11T21:41:02.590389+0000","last_fullsized":"2024-03-11T21:41:02.590389+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:36:07.624075+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:30:18.678531+0000","last_clean_scrub_stamp":"2024-03-11T21:36:07.624075+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1a","version":"0'0","reported_seq":683,"reported_epoch":366,"state":"active+clean","last_fresh":"2024-03-11T21:41:02.586600+0000","last_change":"2024-03-11T21:39:53.299021+0000","last_active":"2024-03-11T21:41:02.586600+0000","last_peered":"2024-03-11T21:41:02.586600+0000","last_clean":"2024-03-11T21:41:02.586600+0000","last_became_active":"2024-03-11T21:34:41.145014+0000","last_became_peered":"2024-03-11T21:34:41.145014+0000","last_unstale":"2024-03-11T21:41:02.586600+0000","last_undegraded":"2024-03-11T21:41:02.586600+0000","last_fullsized":"2024-03-11T21:41:02.586600+0000","mapping_epoch":226,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":227,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:39:53.298917+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:37.512018+0000","last_clean_scrub_stamp":"2024-03-11T21:39:53.298917+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:41:21.887727+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.1c","version":"260'27","reported_seq":716,"reported_epoch":366,"state":"active+clean","last_fresh":"2024-03-11T21:41:02.590237+0000","last_change":"2024-03-11T21:40:15.885360+0000","last_active":"2024-03-11T21:41:02.590237+0000","last_peered":"2024-03-11T21:41:02.590237+0000","last_clean":"2024-03-11T21:41:02.590237+0000","last_became_active":"2024-03-11T21:36:35.074579+0000","last_became_peered":"2024-03-11T21:36:35.074579+0000","last_unstale":"2024-03-11T21:41:02.590237+0000","last_undegraded":"2024-03-11T21:41:02.590237+0000","last_fullsized":"2024-03-11T21:41:02.590237+0000","mapping_epoch":277,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":278,"parent":"0.0","parent_split_bits":0,"last_scrub":"260'27","last_scrub_stamp":"2024-03-11T21:40:15.885265+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:39.525554+0000","last_clean_scrub_stamp":"2024-03-11T21:40:15.885265+0000","objects_scrubbed":7,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:41:39.688604+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":44,"num_read_kb":35,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.1d","version":"0'0","reported_seq":635,"reported_epoch":366,"state":"active+clean","last_fresh":"2024-03-11T21:41:02.589951+0000","last_change":"2024-03-11T21:35:21.744646+0000","last_active":"2024-03-11T21:41:02.589951+0000","last_peered":"2024-03-11T21:41:02.589951+0000","last_clean":"2024-03-11T21:41:02.589951+0000","last_became_active":"2024-03-11T21:25:44.588845+0000","last_became_peered":"2024-03-11T21:25:44.588845+0000","last_unstale":"2024-03-11T21:41:02.589951+0000","last_undegraded":"2024-03-11T21:41:02.589951+0000","last_fullsized":"2024-03-11T21:41:02.589951+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:35:21.744223+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:43.559452+0000","last_clean_scrub_stamp":"2024-03-11T21:35:21.744223+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":99,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.1e","version":"0'0","reported_seq":656,"reported_epoch":366,"state":"active+clean","last_fresh":"2024-03-11T21:41:02.591048+0000","last_change":"2024-03-11T21:40:28.163223+0000","last_active":"2024-03-11T21:41:02.591048+0000","last_peered":"2024-03-11T21:41:02.591048+0000","last_clean":"2024-03-11T21:41:02.591048+0000","last_became_active":"2024-03-11T21:25:42.582959+0000","last_became_peered":"2024-03-11T21:25:42.582959+0000","last_unstale":"2024-03-11T21:41:02.591048+0000","last_undegraded":"2024-03-11T21:41:02.591048+0000","last_fullsized":"2024-03-11T21:41:02.591048+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:40:28.163142+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:26:58.335431+0000","last_clean_scrub_stamp":"2024-03-11T21:40:28.163142+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:41:55.681758+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.19","version":"67'2","reported_seq":601,"reported_epoch":366,"state":"active+clean","last_fresh":"2024-03-11T21:41:02.591924+0000","last_change":"2024-03-11T21:41:01.612764+0000","last_active":"2024-03-11T21:41:02.591924+0000","last_peered":"2024-03-11T21:41:02.591924+0000","last_clean":"2024-03-11T21:41:02.591924+0000","last_became_active":"2024-03-11T21:36:35.064249+0000","last_became_peered":"2024-03-11T21:36:35.064249+0000","last_unstale":"2024-03-11T21:41:02.591924+0000","last_undegraded":"2024-03-11T21:41:02.591924+0000","last_fullsized":"2024-03-11T21:41:02.591924+0000","mapping_epoch":277,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":278,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'2","last_scrub_stamp":"2024-03-11T21:41:01.612632+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:37.512018+0000","last_clean_scrub_stamp":"2024-03-11T21:41:01.612632+0000","objects_scrubbed":2,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:42:05.275629+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":957,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,0],"acting":[3,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1f","version":"260'24","reported_seq":730,"reported_epoch":366,"state":"active+clean","last_fresh":"2024-03-11T21:41:02.585313+0000","last_change":"2024-03-11T21:40:25.079945+0000","last_active":"2024-03-11T21:41:02.585313+0000","last_peered":"2024-03-11T21:41:02.585313+0000","last_clean":"2024-03-11T21:41:02.585313+0000","last_became_active":"2024-03-11T21:32:22.798242+0000","last_became_peered":"2024-03-11T21:32:22.798242+0000","last_unstale":"2024-03-11T21:41:02.585313+0000","last_undegraded":"2024-03-11T21:41:02.585313+0000","last_fullsized":"2024-03-11T21:41:02.585313+0000","mapping_epoch":166,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":167,"parent":"0.0","parent_split_bits":0,"last_scrub":"260'24","last_scrub_stamp":"2024-03-11T21:40:25.079808+0000","last_deep_scrub":"124'16","last_deep_scrub_stamp":"2024-03-11T21:35:38.173841+0000","last_clean_scrub_stamp":"2024-03-11T21:40:25.079808+0000","objects_scrubbed":5,"log_size":24,"log_dups_size":0,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:41:47.993849+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":32,"num_read_kb":25,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,0],"acting":[6,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.1c","version":"74'1","reported_seq":303,"reported_epoch":366,"state":"active+clean","last_fresh":"2024-03-11T21:41:02.590490+0000","last_change":"2024-03-11T21:36:11.732043+0000","last_active":"2024-03-11T21:41:02.590490+0000","last_peered":"2024-03-11T21:41:02.590490+0000","last_clean":"2024-03-11T21:41:02.590490+0000","last_became_active":"2024-03-11T21:34:41.144839+0000","last_became_peered":"2024-03-11T21:34:41.144839+0000","last_unstale":"2024-03-11T21:41:02.590490+0000","last_undegraded":"2024-03-11T21:41:02.590490+0000","last_fullsized":"2024-03-11T21:41:02.590490+0000","mapping_epoch":226,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":227,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'1","last_scrub_stamp":"2024-03-11T21:36:11.731914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:43.559452+0000","last_clean_scrub_stamp":"2024-03-11T21:36:11.731914+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":370,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":370,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.1f","version":"229'10","reported_seq":898,"reported_epoch":366,"state":"active+clean","last_fresh":"2024-03-11T21:41:09.868689+0000","last_change":"2024-03-11T21:41:09.868735+0000","last_active":"2024-03-11T21:41:09.868689+0000","last_peered":"2024-03-11T21:41:09.868689+0000","last_clean":"2024-03-11T21:41:09.868689+0000","last_became_active":"2024-03-11T21:34:41.145862+0000","last_became_peered":"2024-03-11T21:34:41.145862+0000","last_unstale":"2024-03-11T21:41:09.868689+0000","last_undegraded":"2024-03-11T21:41:09.868689+0000","last_fullsized":"2024-03-11T21:41:09.868689+0000","mapping_epoch":226,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":227,"parent":"0.0","parent_split_bits":0,"last_scrub":"229'10","last_scrub_stamp":"2024-03-11T21:41:09.868606+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:41.544550+0000","last_clean_scrub_stamp":"2024-03-11T21:41:09.868606+0000","objects_scrubbed":1,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:42:28.936123+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.18","version":"67'1","reported_seq":698,"reported_epoch":366,"state":"active+clean","last_fresh":"2024-03-11T21:41:02.586944+0000","last_change":"2024-03-11T21:36:10.458150+0000","last_active":"2024-03-11T21:41:02.586944+0000","last_peered":"2024-03-11T21:41:02.586944+0000","last_clean":"2024-03-11T21:41:02.586944+0000","last_became_active":"2024-03-11T21:35:23.310241+0000","last_became_peered":"2024-03-11T21:35:23.310241+0000","last_unstale":"2024-03-11T21:41:02.586944+0000","last_undegraded":"2024-03-11T21:41:02.586944+0000","last_fullsized":"2024-03-11T21:41:02.586944+0000","mapping_epoch":248,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":249,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'1","last_scrub_stamp":"2024-03-11T21:36:10.458028+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:37.512018+0000","last_clean_scrub_stamp":"2024-03-11T21:36:10.458028+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:41:03.775395+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":436,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":7,"num_read_kb":7,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":1308,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,2],"acting":[5,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1e","version":"260'20","reported_seq":740,"reported_epoch":366,"state":"active+clean","last_fresh":"2024-03-11T21:41:02.590948+0000","last_change":"2024-03-11T21:36:35.083447+0000","last_active":"2024-03-11T21:41:02.590948+0000","last_peered":"2024-03-11T21:41:02.590948+0000","last_clean":"2024-03-11T21:41:02.590948+0000","last_became_active":"2024-03-11T21:36:35.057843+0000","last_became_peered":"2024-03-11T21:36:35.057843+0000","last_unstale":"2024-03-11T21:41:02.590948+0000","last_undegraded":"2024-03-11T21:41:02.590948+0000","last_fullsized":"2024-03-11T21:41:02.590948+0000","mapping_epoch":277,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":278,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'10","last_scrub_stamp":"2024-03-11T21:35:16.825146+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:39.525554+0000","last_clean_scrub_stamp":"2024-03-11T21:35:16.825146+0000","objects_scrubbed":5,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":44,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":25,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.13","version":"0'0","reported_seq":526,"reported_epoch":366,"state":"active+clean","last_fresh":"2024-03-11T21:41:02.595253+0000","last_change":"2024-03-11T21:36:35.063744+0000","last_active":"2024-03-11T21:41:02.595253+0000","last_peered":"2024-03-11T21:41:02.595253+0000","last_clean":"2024-03-11T21:41:02.595253+0000","last_became_active":"2024-03-11T21:36:35.062618+0000","last_became_peered":"2024-03-11T21:36:35.062618+0000","last_unstale":"2024-03-11T21:41:02.595253+0000","last_undegraded":"2024-03-11T21:41:02.595253+0000","last_fullsized":"2024-03-11T21:41:02.595253+0000","mapping_epoch":277,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":278,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:34:13.822803+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:43.559452+0000","last_clean_scrub_stamp":"2024-03-11T21:34:13.822803+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":307,"reported_epoch":366,"state":"active+clean","last_fresh":"2024-03-11T21:41:02.590248+0000","last_change":"2024-03-11T21:40:29.084178+0000","last_active":"2024-03-11T21:41:02.590248+0000","last_peered":"2024-03-11T21:41:02.590248+0000","last_clean":"2024-03-11T21:41:02.590248+0000","last_became_active":"2024-03-11T21:34:41.142569+0000","last_became_peered":"2024-03-11T21:34:41.142569+0000","last_unstale":"2024-03-11T21:41:02.590248+0000","last_undegraded":"2024-03-11T21:41:02.590248+0000","last_fullsized":"2024-03-11T21:41:02.590248+0000","mapping_epoch":226,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":227,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:40:29.084062+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:40:29.084062+0000","last_clean_scrub_stamp":"2024-03-11T21:40:29.084062+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:41:31.066228+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.17","version":"0'0","reported_seq":671,"reported_epoch":366,"state":"active+clean","last_fresh":"2024-03-11T21:41:02.585119+0000","last_change":"2024-03-11T21:40:18.112951+0000","last_active":"2024-03-11T21:41:02.585119+0000","last_peered":"2024-03-11T21:41:02.585119+0000","last_clean":"2024-03-11T21:41:02.585119+0000","last_became_active":"2024-03-11T21:25:38.540349+0000","last_became_peered":"2024-03-11T21:25:38.540349+0000","last_unstale":"2024-03-11T21:41:02.585119+0000","last_undegraded":"2024-03-11T21:41:02.585119+0000","last_fullsized":"2024-03-11T21:41:02.585119+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:40:18.112829+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:28:25.918101+0000","last_clean_scrub_stamp":"2024-03-11T21:40:18.112829+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:41:20.570429+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,2],"acting":[6,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.11","version":"260'19","reported_seq":652,"reported_epoch":366,"state":"active+clean","last_fresh":"2024-03-11T21:41:02.594040+0000","last_change":"2024-03-11T21:36:35.096720+0000","last_active":"2024-03-11T21:41:02.594040+0000","last_peered":"2024-03-11T21:41:02.594040+0000","last_clean":"2024-03-11T21:41:02.594040+0000","last_became_active":"2024-03-11T21:36:35.073689+0000","last_became_peered":"2024-03-11T21:36:35.073689+0000","last_unstale":"2024-03-11T21:41:02.594040+0000","last_undegraded":"2024-03-11T21:41:02.594040+0000","last_fullsized":"2024-03-11T21:41:02.594040+0000","mapping_epoch":277,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":278,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'11","last_scrub_stamp":"2024-03-11T21:35:16.044397+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:39.525554+0000","last_clean_scrub_stamp":"2024-03-11T21:35:16.044397+0000","objects_scrubbed":5,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":31,"num_read_kb":24,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":299,"reported_epoch":366,"state":"active+clean","last_fresh":"2024-03-11T21:41:02.589959+0000","last_change":"2024-03-11T21:35:32.719869+0000","last_active":"2024-03-11T21:41:02.589959+0000","last_peered":"2024-03-11T21:41:02.589959+0000","last_clean":"2024-03-11T21:41:02.589959+0000","last_became_active":"2024-03-11T21:34:41.137235+0000","last_became_peered":"2024-03-11T21:34:41.137235+0000","last_unstale":"2024-03-11T21:41:02.589959+0000","last_undegraded":"2024-03-11T21:41:02.589959+0000","last_fullsized":"2024-03-11T21:41:02.589959+0000","mapping_epoch":226,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":227,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:35:32.719789+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:43.559452+0000","last_clean_scrub_stamp":"2024-03-11T21:35:32.719789+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,4],"acting":[7,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":683,"reported_epoch":366,"state":"active+clean","last_fresh":"2024-03-11T21:41:02.584482+0000","last_change":"2024-03-11T21:40:03.181212+0000","last_active":"2024-03-11T21:41:02.584482+0000","last_peered":"2024-03-11T21:41:02.584482+0000","last_clean":"2024-03-11T21:41:02.584482+0000","last_became_active":"2024-03-11T21:34:41.156036+0000","last_became_peered":"2024-03-11T21:34:41.156036+0000","last_unstale":"2024-03-11T21:41:02.584482+0000","last_undegraded":"2024-03-11T21:41:02.584482+0000","last_fullsized":"2024-03-11T21:41:02.584482+0000","mapping_epoch":226,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":227,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:40:03.181074+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:38:51.036736+0000","last_clean_scrub_stamp":"2024-03-11T21:40:03.181074+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:41:28.087147+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.16","version":"0'0","reported_seq":662,"reported_epoch":366,"state":"active+clean","last_fresh":"2024-03-11T21:41:02.588359+0000","last_change":"2024-03-11T21:35:39.108623+0000","last_active":"2024-03-11T21:41:02.588359+0000","last_peered":"2024-03-11T21:41:02.588359+0000","last_clean":"2024-03-11T21:41:02.588359+0000","last_became_active":"2024-03-11T21:25:38.539103+0000","last_became_peered":"2024-03-11T21:25:38.539103+0000","last_unstale":"2024-03-11T21:41:02.588359+0000","last_undegraded":"2024-03-11T21:41:02.588359+0000","last_fullsized":"2024-03-11T21:41:02.588359+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:35:39.108546+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:27:11.938909+0000","last_clean_scrub_stamp":"2024-03-11T21:35:39.108546+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:41:03.775268+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,2],"acting":[5,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.10","version":"260'10","reported_seq":534,"reported_epoch":366,"state":"active+clean","last_fresh":"2024-03-11T21:41:02.592041+0000","last_change":"2024-03-11T21:36:35.084268+0000","last_active":"2024-03-11T21:41:02.592041+0000","last_peered":"2024-03-11T21:41:02.592041+0000","last_clean":"2024-03-11T21:41:02.592041+0000","last_became_active":"2024-03-11T21:36:35.067500+0000","last_became_peered":"2024-03-11T21:36:35.067500+0000","last_unstale":"2024-03-11T21:41:02.592041+0000","last_undegraded":"2024-03-11T21:41:02.592041+0000","last_fullsized":"2024-03-11T21:41:02.592041+0000","mapping_epoch":277,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":278,"parent":"0.0","parent_split_bits":0,"last_scrub":"94'6","last_scrub_stamp":"2024-03-11T21:34:00.727416+0000","last_deep_scrub":"94'6","last_deep_scrub_stamp":"2024-03-11T21:28:24.603974+0000","last_clean_scrub_stamp":"2024-03-11T21:34:00.727416+0000","objects_scrubbed":2,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":12,"num_read_kb":10,"num_write":8,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6],"acting":[3,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":579,"reported_epoch":366,"state":"active+clean","last_fresh":"2024-03-11T21:41:02.593875+0000","last_change":"2024-03-11T21:36:35.070468+0000","last_active":"2024-03-11T21:41:02.593875+0000","last_peered":"2024-03-11T21:41:02.593875+0000","last_clean":"2024-03-11T21:41:02.593875+0000","last_became_active":"2024-03-11T21:36:35.070176+0000","last_became_peered":"2024-03-11T21:36:35.070176+0000","last_unstale":"2024-03-11T21:41:02.593875+0000","last_undegraded":"2024-03-11T21:41:02.593875+0000","last_fullsized":"2024-03-11T21:41:02.593875+0000","mapping_epoch":277,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":278,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:35:18.062272+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:43.559452+0000","last_clean_scrub_stamp":"2024-03-11T21:35:18.062272+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":656,"reported_epoch":366,"state":"active+clean","last_fresh":"2024-03-11T21:41:04.276070+0000","last_change":"2024-03-11T21:41:04.276105+0000","last_active":"2024-03-11T21:41:04.276070+0000","last_peered":"2024-03-11T21:41:04.276070+0000","last_clean":"2024-03-11T21:41:04.276070+0000","last_became_active":"2024-03-11T21:36:35.057906+0000","last_became_peered":"2024-03-11T21:36:35.057906+0000","last_unstale":"2024-03-11T21:41:04.276070+0000","last_undegraded":"2024-03-11T21:41:04.276070+0000","last_fullsized":"2024-03-11T21:41:04.276070+0000","mapping_epoch":277,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":278,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:41:04.275820+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:27:21.207303+0000","last_clean_scrub_stamp":"2024-03-11T21:41:04.275820+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:42:25.100823+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.15","version":"0'0","reported_seq":655,"reported_epoch":366,"state":"active+clean","last_fresh":"2024-03-11T21:41:02.590145+0000","last_change":"2024-03-11T21:36:06.579754+0000","last_active":"2024-03-11T21:41:02.590145+0000","last_peered":"2024-03-11T21:41:02.590145+0000","last_clean":"2024-03-11T21:41:02.590145+0000","last_became_active":"2024-03-11T21:25:38.541705+0000","last_became_peered":"2024-03-11T21:25:38.541705+0000","last_unstale":"2024-03-11T21:41:02.590145+0000","last_undegraded":"2024-03-11T21:41:02.590145+0000","last_fullsized":"2024-03-11T21:41:02.590145+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:36:06.579636+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:26:44.860499+0000","last_clean_scrub_stamp":"2024-03-11T21:36:06.579636+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.13","version":"260'24","reported_seq":713,"reported_epoch":366,"state":"active+clean","last_fresh":"2024-03-11T21:41:02.590947+0000","last_change":"2024-03-11T21:40:25.595320+0000","last_active":"2024-03-11T21:41:02.590947+0000","last_peered":"2024-03-11T21:41:02.590947+0000","last_clean":"2024-03-11T21:41:02.590947+0000","last_became_active":"2024-03-11T21:26:11.214879+0000","last_became_peered":"2024-03-11T21:26:11.214879+0000","last_unstale":"2024-03-11T21:41:02.590947+0000","last_undegraded":"2024-03-11T21:41:02.590947+0000","last_fullsized":"2024-03-11T21:41:02.590947+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"260'24","last_scrub_stamp":"2024-03-11T21:40:25.595220+0000","last_deep_scrub":"260'24","last_deep_scrub_stamp":"2024-03-11T21:40:25.595220+0000","last_clean_scrub_stamp":"2024-03-11T21:40:25.595220+0000","objects_scrubbed":5,"log_size":24,"log_dups_size":0,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:41:39.345264+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":32,"num_read_kb":25,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,2],"acting":[4,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.10","version":"0'0","reported_seq":643,"reported_epoch":366,"state":"active+clean","last_fresh":"2024-03-11T21:41:02.588119+0000","last_change":"2024-03-11T21:35:29.712658+0000","last_active":"2024-03-11T21:41:02.588119+0000","last_peered":"2024-03-11T21:41:02.588119+0000","last_clean":"2024-03-11T21:41:02.588119+0000","last_became_active":"2024-03-11T21:25:44.591189+0000","last_became_peered":"2024-03-11T21:25:44.591189+0000","last_unstale":"2024-03-11T21:41:02.588119+0000","last_undegraded":"2024-03-11T21:41:02.588119+0000","last_fullsized":"2024-03-11T21:41:02.588119+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:35:29.712557+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:27:25.417450+0000","last_clean_scrub_stamp":"2024-03-11T21:35:29.712557+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,1],"acting":[0,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":577,"reported_epoch":366,"state":"active+clean","last_fresh":"2024-03-11T21:41:02.595347+0000","last_change":"2024-03-11T21:36:35.062276+0000","last_active":"2024-03-11T21:41:02.595347+0000","last_peered":"2024-03-11T21:41:02.595347+0000","last_clean":"2024-03-11T21:41:02.595347+0000","last_became_active":"2024-03-11T21:36:35.056619+0000","last_became_peered":"2024-03-11T21:36:35.056619+0000","last_unstale":"2024-03-11T21:41:02.595347+0000","last_undegraded":"2024-03-11T21:41:02.595347+0000","last_fullsized":"2024-03-11T21:41:02.595347+0000","mapping_epoch":277,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":278,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:35:13.060487+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:31:15.367898+0000","last_clean_scrub_stamp":"2024-03-11T21:35:13.060487+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.14","version":"0'0","reported_seq":688,"reported_epoch":366,"state":"active+clean","last_fresh":"2024-03-11T21:41:02.584820+0000","last_change":"2024-03-11T21:40:01.212243+0000","last_active":"2024-03-11T21:41:02.584820+0000","last_peered":"2024-03-11T21:41:02.584820+0000","last_clean":"2024-03-11T21:41:02.584820+0000","last_became_active":"2024-03-11T21:36:35.068038+0000","last_became_peered":"2024-03-11T21:36:35.068038+0000","last_unstale":"2024-03-11T21:41:02.584820+0000","last_undegraded":"2024-03-11T21:41:02.584820+0000","last_fullsized":"2024-03-11T21:41:02.584820+0000","mapping_epoch":277,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":278,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:40:01.212145+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:37.512018+0000","last_clean_scrub_stamp":"2024-03-11T21:40:01.212145+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:41:09.997136+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,5],"acting":[6,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.12","version":"260'15","reported_seq":676,"reported_epoch":366,"state":"active+clean","last_fresh":"2024-03-11T21:41:02.612773+0000","last_change":"2024-03-11T21:35:32.590439+0000","last_active":"2024-03-11T21:41:02.612773+0000","last_peered":"2024-03-11T21:41:02.612773+0000","last_clean":"2024-03-11T21:41:02.612773+0000","last_became_active":"2024-03-11T21:25:40.547082+0000","last_became_peered":"2024-03-11T21:25:40.547082+0000","last_unstale":"2024-03-11T21:41:02.612773+0000","last_undegraded":"2024-03-11T21:41:02.612773+0000","last_fullsized":"2024-03-11T21:41:02.612773+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'9","last_scrub_stamp":"2024-03-11T21:35:32.590318+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:39.525554+0000","last_clean_scrub_stamp":"2024-03-11T21:35:32.590318+0000","objects_scrubbed":4,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":643,"reported_epoch":366,"state":"active+clean","last_fresh":"2024-03-11T21:41:02.593711+0000","last_change":"2024-03-11T21:36:04.754168+0000","last_active":"2024-03-11T21:41:02.593711+0000","last_peered":"2024-03-11T21:41:02.593711+0000","last_clean":"2024-03-11T21:41:02.593711+0000","last_became_active":"2024-03-11T21:25:44.589586+0000","last_became_peered":"2024-03-11T21:25:44.589586+0000","last_unstale":"2024-03-11T21:41:02.593711+0000","last_undegraded":"2024-03-11T21:41:02.593711+0000","last_fullsized":"2024-03-11T21:41:02.593711+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:36:04.753979+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:27:09.801138+0000","last_clean_scrub_stamp":"2024-03-11T21:36:04.753979+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":113,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,5],"acting":[4,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.14","version":"280'12","reported_seq":794,"reported_epoch":366,"state":"active+clean","last_fresh":"2024-03-11T21:41:03.780807+0000","last_change":"2024-03-11T21:36:35.074550+0000","last_active":"2024-03-11T21:41:03.780807+0000","last_peered":"2024-03-11T21:41:03.780807+0000","last_clean":"2024-03-11T21:41:03.780807+0000","last_became_active":"2024-03-11T21:36:35.073912+0000","last_became_peered":"2024-03-11T21:36:35.073912+0000","last_unstale":"2024-03-11T21:41:03.780807+0000","last_undegraded":"2024-03-11T21:41:03.780807+0000","last_fullsized":"2024-03-11T21:41:03.780807+0000","mapping_epoch":277,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":278,"parent":"0.0","parent_split_bits":0,"last_scrub":"229'10","last_scrub_stamp":"2024-03-11T21:35:26.182107+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:41.544550+0000","last_clean_scrub_stamp":"2024-03-11T21:35:26.182107+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,2],"acting":[3,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.13","version":"0'0","reported_seq":664,"reported_epoch":366,"state":"active+clean","last_fresh":"2024-03-11T21:41:02.588796+0000","last_change":"2024-03-11T21:40:11.027382+0000","last_active":"2024-03-11T21:41:02.588796+0000","last_peered":"2024-03-11T21:41:02.588796+0000","last_clean":"2024-03-11T21:41:02.588796+0000","last_became_active":"2024-03-11T21:36:35.057132+0000","last_became_peered":"2024-03-11T21:36:35.057132+0000","last_unstale":"2024-03-11T21:41:02.588796+0000","last_undegraded":"2024-03-11T21:41:02.588796+0000","last_fullsized":"2024-03-11T21:41:02.588796+0000","mapping_epoch":277,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":278,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:40:11.027268+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:37.512018+0000","last_clean_scrub_stamp":"2024-03-11T21:40:11.027268+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:41:40.995455+0000","scrub_duration":24,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.15","version":"260'19","reported_seq":729,"reported_epoch":366,"state":"active+clean","last_fresh":"2024-03-11T21:41:02.587390+0000","last_change":"2024-03-11T21:36:35.088986+0000","last_active":"2024-03-11T21:41:02.587390+0000","last_peered":"2024-03-11T21:41:02.587390+0000","last_clean":"2024-03-11T21:41:02.587390+0000","last_became_active":"2024-03-11T21:36:35.057285+0000","last_became_peered":"2024-03-11T21:36:35.057285+0000","last_unstale":"2024-03-11T21:41:02.587390+0000","last_undegraded":"2024-03-11T21:41:02.587390+0000","last_fullsized":"2024-03-11T21:41:02.587390+0000","mapping_epoch":277,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":278,"parent":"0.0","parent_split_bits":0,"last_scrub":"94'13","last_scrub_stamp":"2024-03-11T21:35:29.178543+0000","last_deep_scrub":"94'13","last_deep_scrub_stamp":"2024-03-11T21:29:41.853158+0000","last_clean_scrub_stamp":"2024-03-11T21:35:29.178543+0000","objects_scrubbed":4,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:41:03.775180+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":11,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.16","version":"0'0","reported_seq":672,"reported_epoch":366,"state":"active+clean","last_fresh":"2024-03-11T21:41:02.589475+0000","last_change":"2024-03-11T21:40:30.249810+0000","last_active":"2024-03-11T21:41:02.589475+0000","last_peered":"2024-03-11T21:41:02.589475+0000","last_clean":"2024-03-11T21:41:02.589475+0000","last_became_active":"2024-03-11T21:36:35.057355+0000","last_became_peered":"2024-03-11T21:36:35.057355+0000","last_unstale":"2024-03-11T21:41:02.589475+0000","last_undegraded":"2024-03-11T21:41:02.589475+0000","last_fullsized":"2024-03-11T21:41:02.589475+0000","mapping_epoch":277,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":278,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:40:30.249679+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:40:30.249679+0000","last_clean_scrub_stamp":"2024-03-11T21:40:30.249679+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:41:59.927156+0000","scrub_duration":40,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.15","version":"98'9","reported_seq":842,"reported_epoch":366,"state":"active+clean","last_fresh":"2024-03-11T21:41:03.780705+0000","last_change":"2024-03-11T21:36:03.366581+0000","last_active":"2024-03-11T21:41:03.780705+0000","last_peered":"2024-03-11T21:41:03.780705+0000","last_clean":"2024-03-11T21:41:03.780705+0000","last_became_active":"2024-03-11T21:25:42.572555+0000","last_became_peered":"2024-03-11T21:25:42.572555+0000","last_unstale":"2024-03-11T21:41:03.780705+0000","last_undegraded":"2024-03-11T21:41:03.780705+0000","last_fullsized":"2024-03-11T21:41:03.780705+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"98'9","last_scrub_stamp":"2024-03-11T21:36:03.366450+0000","last_deep_scrub":"98'9","last_deep_scrub_stamp":"2024-03-11T21:31:20.624923+0000","last_clean_scrub_stamp":"2024-03-11T21:36:03.366450+0000","objects_scrubbed":1,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.12","version":"0'0","reported_seq":675,"reported_epoch":366,"state":"active+clean","last_fresh":"2024-03-11T21:41:02.587997+0000","last_change":"2024-03-11T21:36:35.058369+0000","last_active":"2024-03-11T21:41:02.587997+0000","last_peered":"2024-03-11T21:41:02.587997+0000","last_clean":"2024-03-11T21:41:02.587997+0000","last_became_active":"2024-03-11T21:36:35.058033+0000","last_became_peered":"2024-03-11T21:36:35.058033+0000","last_unstale":"2024-03-11T21:41:02.587997+0000","last_undegraded":"2024-03-11T21:41:02.587997+0000","last_fullsized":"2024-03-11T21:41:02.587997+0000","mapping_epoch":277,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":278,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:35:27.158468+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:37.512018+0000","last_clean_scrub_stamp":"2024-03-11T21:35:27.158468+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:41:03.775210+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.14","version":"260'20","reported_seq":660,"reported_epoch":366,"state":"active+clean","last_fresh":"2024-03-11T21:41:02.594449+0000","last_change":"2024-03-11T21:36:35.095382+0000","last_active":"2024-03-11T21:41:02.594449+0000","last_peered":"2024-03-11T21:41:02.594449+0000","last_clean":"2024-03-11T21:41:02.594449+0000","last_became_active":"2024-03-11T21:36:35.069638+0000","last_became_peered":"2024-03-11T21:36:35.069638+0000","last_unstale":"2024-03-11T21:41:02.594449+0000","last_undegraded":"2024-03-11T21:41:02.594449+0000","last_fullsized":"2024-03-11T21:41:02.594449+0000","mapping_epoch":277,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":278,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'10","last_scrub_stamp":"2024-03-11T21:35:28.189296+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:39.525554+0000","last_clean_scrub_stamp":"2024-03-11T21:35:28.189296+0000","objects_scrubbed":0,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":29,"num_read_kb":24,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.15","version":"0'0","reported_seq":299,"reported_epoch":366,"state":"active+clean","last_fresh":"2024-03-11T21:41:02.592454+0000","last_change":"2024-03-11T21:36:14.736240+0000","last_active":"2024-03-11T21:41:02.592454+0000","last_peered":"2024-03-11T21:41:02.592454+0000","last_clean":"2024-03-11T21:41:02.592454+0000","last_became_active":"2024-03-11T21:34:41.142760+0000","last_became_peered":"2024-03-11T21:34:41.142760+0000","last_unstale":"2024-03-11T21:41:02.592454+0000","last_undegraded":"2024-03-11T21:41:02.592454+0000","last_fullsized":"2024-03-11T21:41:02.592454+0000","mapping_epoch":226,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":227,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:36:14.735985+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:43.559452+0000","last_clean_scrub_stamp":"2024-03-11T21:36:14.735985+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":36,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":655,"reported_epoch":366,"state":"active+clean","last_fresh":"2024-03-11T21:41:02.588244+0000","last_change":"2024-03-11T21:40:26.587777+0000","last_active":"2024-03-11T21:41:02.588244+0000","last_peered":"2024-03-11T21:41:02.588244+0000","last_clean":"2024-03-11T21:41:02.588244+0000","last_became_active":"2024-03-11T21:36:35.060342+0000","last_became_peered":"2024-03-11T21:36:35.060342+0000","last_unstale":"2024-03-11T21:41:02.588244+0000","last_undegraded":"2024-03-11T21:41:02.588244+0000","last_fullsized":"2024-03-11T21:41:02.588244+0000","mapping_epoch":277,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":278,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:40:26.587672+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:41.544550+0000","last_clean_scrub_stamp":"2024-03-11T21:40:26.587672+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:41:33.380076+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,1],"acting":[5,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.11","version":"0'0","reported_seq":671,"reported_epoch":366,"state":"active+clean","last_fresh":"2024-03-11T21:41:02.584567+0000","last_change":"2024-03-11T21:40:09.218006+0000","last_active":"2024-03-11T21:41:02.584567+0000","last_peered":"2024-03-11T21:41:02.584567+0000","last_clean":"2024-03-11T21:41:02.584567+0000","last_became_active":"2024-03-11T21:25:38.537415+0000","last_became_peered":"2024-03-11T21:25:38.537415+0000","last_unstale":"2024-03-11T21:41:02.584567+0000","last_undegraded":"2024-03-11T21:41:02.584567+0000","last_fullsized":"2024-03-11T21:41:02.584567+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:40:09.217927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:37.512018+0000","last_clean_scrub_stamp":"2024-03-11T21:40:09.217927+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:41:24.546509+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.17","version":"260'15","reported_seq":613,"reported_epoch":366,"state":"active+clean","last_fresh":"2024-03-11T21:41:02.595498+0000","last_change":"2024-03-11T21:41:00.652483+0000","last_active":"2024-03-11T21:41:02.595498+0000","last_peered":"2024-03-11T21:41:02.595498+0000","last_clean":"2024-03-11T21:41:02.595498+0000","last_became_active":"2024-03-11T21:36:35.071346+0000","last_became_peered":"2024-03-11T21:36:35.071346+0000","last_unstale":"2024-03-11T21:41:02.595498+0000","last_undegraded":"2024-03-11T21:41:02.595498+0000","last_fullsized":"2024-03-11T21:41:02.595498+0000","mapping_epoch":277,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":278,"parent":"0.0","parent_split_bits":0,"last_scrub":"260'15","last_scrub_stamp":"2024-03-11T21:41:00.652358+0000","last_deep_scrub":"97'9","last_deep_scrub_stamp":"2024-03-11T21:27:25.803132+0000","last_clean_scrub_stamp":"2024-03-11T21:41:00.652358+0000","objects_scrubbed":3,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:42:10.592433+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":17,"num_read_kb":14,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,5],"acting":[3,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.14","version":"0'0","reported_seq":661,"reported_epoch":366,"state":"active+clean","last_fresh":"2024-03-11T21:41:08.130072+0000","last_change":"2024-03-11T21:41:08.130109+0000","last_active":"2024-03-11T21:41:08.130072+0000","last_peered":"2024-03-11T21:41:08.130072+0000","last_clean":"2024-03-11T21:41:08.130072+0000","last_became_active":"2024-03-11T21:34:41.136946+0000","last_became_peered":"2024-03-11T21:34:41.136946+0000","last_unstale":"2024-03-11T21:41:08.130072+0000","last_undegraded":"2024-03-11T21:41:08.130072+0000","last_fullsized":"2024-03-11T21:41:08.130072+0000","mapping_epoch":226,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":227,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:41:08.129982+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:43.559452+0000","last_clean_scrub_stamp":"2024-03-11T21:41:08.129982+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:42:13.725836+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,7],"acting":[2,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":559,"reported_epoch":366,"state":"active+clean","last_fresh":"2024-03-11T21:41:02.595289+0000","last_change":"2024-03-11T21:36:35.076782+0000","last_active":"2024-03-11T21:41:02.595289+0000","last_peered":"2024-03-11T21:41:02.595289+0000","last_clean":"2024-03-11T21:41:02.595289+0000","last_became_active":"2024-03-11T21:36:35.076335+0000","last_became_peered":"2024-03-11T21:36:35.076335+0000","last_unstale":"2024-03-11T21:41:02.595289+0000","last_undegraded":"2024-03-11T21:41:02.595289+0000","last_fullsized":"2024-03-11T21:41:02.595289+0000","mapping_epoch":277,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":278,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:34:33.082577+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:41.544550+0000","last_clean_scrub_stamp":"2024-03-11T21:34:33.082577+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":35,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.10","version":"0'0","reported_seq":653,"reported_epoch":366,"state":"active+clean","last_fresh":"2024-03-11T21:41:02.589565+0000","last_change":"2024-03-11T21:35:37.694623+0000","last_active":"2024-03-11T21:41:02.589565+0000","last_peered":"2024-03-11T21:41:02.589565+0000","last_clean":"2024-03-11T21:41:02.589565+0000","last_became_active":"2024-03-11T21:25:38.533138+0000","last_became_peered":"2024-03-11T21:25:38.533138+0000","last_unstale":"2024-03-11T21:41:02.589565+0000","last_undegraded":"2024-03-11T21:41:02.589565+0000","last_fullsized":"2024-03-11T21:41:02.589565+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:35:37.694533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:37.512018+0000","last_clean_scrub_stamp":"2024-03-11T21:35:37.694533+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:41:04.082992+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,0],"acting":[2,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.16","version":"260'15","reported_seq":726,"reported_epoch":366,"state":"active+clean","last_fresh":"2024-03-11T21:41:02.590328+0000","last_change":"2024-03-11T21:36:35.080158+0000","last_active":"2024-03-11T21:41:02.590328+0000","last_peered":"2024-03-11T21:41:02.590328+0000","last_clean":"2024-03-11T21:41:02.590328+0000","last_became_active":"2024-03-11T21:36:35.056862+0000","last_became_peered":"2024-03-11T21:36:35.056862+0000","last_unstale":"2024-03-11T21:41:02.590328+0000","last_undegraded":"2024-03-11T21:41:02.590328+0000","last_fullsized":"2024-03-11T21:41:02.590328+0000","mapping_epoch":277,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":278,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'9","last_scrub_stamp":"2024-03-11T21:35:12.741225+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:39.525554+0000","last_clean_scrub_stamp":"2024-03-11T21:35:12.741225+0000","objects_scrubbed":4,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":11,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.7","version":"366'2812","reported_seq":2842,"reported_epoch":366,"state":"active+clean","last_fresh":"2024-03-11T21:41:12.342233+0000","last_change":"2024-03-11T21:40:39.541210+0000","last_active":"2024-03-11T21:41:12.342233+0000","last_peered":"2024-03-11T21:41:12.342233+0000","last_clean":"2024-03-11T21:41:12.342233+0000","last_became_active":"2024-03-11T21:40:39.540781+0000","last_became_peered":"2024-03-11T21:40:39.540781+0000","last_unstale":"2024-03-11T21:41:12.342233+0000","last_undegraded":"2024-03-11T21:41:12.342233+0000","last_fullsized":"2024-03-11T21:41:12.342233+0000","mapping_epoch":356,"log_start":"0'0","ondisk_log_start":"0'0","created":356,"last_epoch_clean":357,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:40:38.219602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:40:38.219602+0000","last_clean_scrub_stamp":"2024-03-11T21:40:38.219602+0000","objects_scrubbed":0,"log_size":2812,"log_dups_size":0,"ondisk_log_size":2812,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:41:53.429960+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":184287232,"num_objects":2812,"num_object_clones":0,"num_object_copies":8436,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2812,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2812,"num_write_kb":179968,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,5],"acting":[6,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.b","version":"0'0","reported_seq":556,"reported_epoch":366,"state":"active+clean","last_fresh":"2024-03-11T21:41:02.595144+0000","last_change":"2024-03-11T21:40:27.611092+0000","last_active":"2024-03-11T21:41:02.595144+0000","last_peered":"2024-03-11T21:41:02.595144+0000","last_clean":"2024-03-11T21:41:02.595144+0000","last_became_active":"2024-03-11T21:36:35.072147+0000","last_became_peered":"2024-03-11T21:36:35.072147+0000","last_unstale":"2024-03-11T21:41:02.595144+0000","last_undegraded":"2024-03-11T21:41:02.595144+0000","last_fullsized":"2024-03-11T21:41:02.595144+0000","mapping_epoch":277,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":278,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:40:27.610987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:43.559452+0000","last_clean_scrub_stamp":"2024-03-11T21:40:27.610987+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:41:47.989897+0000","scrub_duration":60,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.f","version":"67'2","reported_seq":667,"reported_epoch":366,"state":"active+clean","last_fresh":"2024-03-11T21:41:02.593493+0000","last_change":"2024-03-11T21:35:36.613667+0000","last_active":"2024-03-11T21:41:02.593493+0000","last_peered":"2024-03-11T21:41:02.593493+0000","last_clean":"2024-03-11T21:41:02.593493+0000","last_became_active":"2024-03-11T21:25:38.537711+0000","last_became_peered":"2024-03-11T21:25:38.537711+0000","last_unstale":"2024-03-11T21:41:02.593493+0000","last_undegraded":"2024-03-11T21:41:02.593493+0000","last_fullsized":"2024-03-11T21:41:02.593493+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'2","last_scrub_stamp":"2024-03-11T21:35:36.613516+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:37.512018+0000","last_clean_scrub_stamp":"2024-03-11T21:35:36.613516+0000","objects_scrubbed":2,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":92,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":10,"num_read_kb":10,"num_write":4,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,7],"acting":[4,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.9","version":"260'23","reported_seq":707,"reported_epoch":366,"state":"active+clean","last_fresh":"2024-03-11T21:41:02.591214+0000","last_change":"2024-03-11T21:40:14.543701+0000","last_active":"2024-03-11T21:41:02.591214+0000","last_peered":"2024-03-11T21:41:02.591214+0000","last_clean":"2024-03-11T21:41:02.591214+0000","last_became_active":"2024-03-11T21:36:35.059930+0000","last_became_peered":"2024-03-11T21:36:35.059930+0000","last_unstale":"2024-03-11T21:41:02.591214+0000","last_undegraded":"2024-03-11T21:41:02.591214+0000","last_fullsized":"2024-03-11T21:41:02.591214+0000","mapping_epoch":277,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":278,"parent":"0.0","parent_split_bits":0,"last_scrub":"260'23","last_scrub_stamp":"2024-03-11T21:40:14.543596+0000","last_deep_scrub":"146'17","last_deep_scrub_stamp":"2024-03-11T21:34:26.765075+0000","last_clean_scrub_stamp":"2024-03-11T21:40:14.543596+0000","objects_scrubbed":5,"log_size":23,"log_dups_size":0,"ondisk_log_size":23,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:41:34.525219+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":34,"num_read_kb":25,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,3],"acting":[4,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":646,"reported_epoch":366,"state":"active+clean","last_fresh":"2024-03-11T21:41:02.589978+0000","last_change":"2024-03-11T21:35:16.643421+0000","last_active":"2024-03-11T21:41:02.589978+0000","last_peered":"2024-03-11T21:41:02.589978+0000","last_clean":"2024-03-11T21:41:02.589978+0000","last_became_active":"2024-03-11T21:25:42.573987+0000","last_became_peered":"2024-03-11T21:25:42.573987+0000","last_unstale":"2024-03-11T21:41:02.589978+0000","last_undegraded":"2024-03-11T21:41:02.589978+0000","last_fullsized":"2024-03-11T21:41:02.589978+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:35:16.643311+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:41.544550+0000","last_clean_scrub_stamp":"2024-03-11T21:35:16.643311+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:41:04.082942+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"10.c","version":"366'2799","reported_seq":2827,"reported_epoch":366,"state":"active+clean","last_fresh":"2024-03-11T21:41:12.397050+0000","last_change":"2024-03-11T21:40:39.242008+0000","last_active":"2024-03-11T21:41:12.397050+0000","last_peered":"2024-03-11T21:41:12.397050+0000","last_clean":"2024-03-11T21:41:12.397050+0000","last_became_active":"2024-03-11T21:40:39.241699+0000","last_became_peered":"2024-03-11T21:40:39.241699+0000","last_unstale":"2024-03-11T21:41:12.397050+0000","last_undegraded":"2024-03-11T21:41:12.397050+0000","last_fullsized":"2024-03-11T21:41:12.397050+0000","mapping_epoch":356,"log_start":"0'0","ondisk_log_start":"0'0","created":356,"last_epoch_clean":357,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:40:38.219602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:40:38.219602+0000","last_clean_scrub_stamp":"2024-03-11T21:40:38.219602+0000","objects_scrubbed":0,"log_size":2799,"log_dups_size":0,"ondisk_log_size":2799,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:42:05.061104+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":183435264,"num_objects":2799,"num_object_clones":0,"num_object_copies":8397,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2799,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2799,"num_write_kb":179136,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,0],"acting":[7,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.0","version":"0'0","reported_seq":653,"reported_epoch":366,"state":"active+clean","last_fresh":"2024-03-11T21:41:02.588209+0000","last_change":"2024-03-11T21:40:27.230013+0000","last_active":"2024-03-11T21:41:02.588209+0000","last_peered":"2024-03-11T21:41:02.588209+0000","last_clean":"2024-03-11T21:41:02.588209+0000","last_became_active":"2024-03-11T21:36:35.058054+0000","last_became_peered":"2024-03-11T21:36:35.058054+0000","last_unstale":"2024-03-11T21:41:02.588209+0000","last_undegraded":"2024-03-11T21:41:02.588209+0000","last_fullsized":"2024-03-11T21:41:02.588209+0000","mapping_epoch":277,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":278,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:40:27.229904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:29:39.498784+0000","last_clean_scrub_stamp":"2024-03-11T21:40:27.229904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:41:49.392723+0000","scrub_duration":38,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":668,"reported_epoch":366,"state":"active+clean","last_fresh":"2024-03-11T21:41:02.590631+0000","last_change":"2024-03-11T21:35:33.608063+0000","last_active":"2024-03-11T21:41:02.590631+0000","last_peered":"2024-03-11T21:41:02.590631+0000","last_clean":"2024-03-11T21:41:02.590631+0000","last_became_active":"2024-03-11T21:34:41.150913+0000","last_became_peered":"2024-03-11T21:34:41.150913+0000","last_unstale":"2024-03-11T21:41:02.590631+0000","last_undegraded":"2024-03-11T21:41:02.590631+0000","last_fullsized":"2024-03-11T21:41:02.590631+0000","mapping_epoch":226,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":227,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:35:33.607969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:37.512018+0000","last_clean_scrub_stamp":"2024-03-11T21:35:33.607969+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,7],"acting":[1,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.2","version":"260'25","reported_seq":695,"reported_epoch":366,"state":"active+clean","last_fresh":"2024-03-11T21:41:07.243140+0000","last_change":"2024-03-11T21:41:07.243196+0000","last_active":"2024-03-11T21:41:07.243140+0000","last_peered":"2024-03-11T21:41:07.243140+0000","last_clean":"2024-03-11T21:41:07.243140+0000","last_became_active":"2024-03-11T21:25:40.548833+0000","last_became_peered":"2024-03-11T21:25:40.548833+0000","last_unstale":"2024-03-11T21:41:07.243140+0000","last_undegraded":"2024-03-11T21:41:07.243140+0000","last_fullsized":"2024-03-11T21:41:07.243140+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"260'25","last_scrub_stamp":"2024-03-11T21:41:07.242950+0000","last_deep_scrub":"94'15","last_deep_scrub_stamp":"2024-03-11T21:26:49.993060+0000","last_clean_scrub_stamp":"2024-03-11T21:41:07.242950+0000","objects_scrubbed":5,"log_size":25,"log_dups_size":0,"ondisk_log_size":25,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:42:16.983313+0000","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":25,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.3","version":"74'8","reported_seq":843,"reported_epoch":366,"state":"active+clean","last_fresh":"2024-03-11T21:41:03.796480+0000","last_change":"2024-03-11T21:35:31.695369+0000","last_active":"2024-03-11T21:41:03.796480+0000","last_peered":"2024-03-11T21:41:03.796480+0000","last_clean":"2024-03-11T21:41:03.796480+0000","last_became_active":"2024-03-11T21:25:42.578161+0000","last_became_peered":"2024-03-11T21:25:42.578161+0000","last_unstale":"2024-03-11T21:41:03.796480+0000","last_undegraded":"2024-03-11T21:41:03.796480+0000","last_fullsized":"2024-03-11T21:41:03.796480+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'8","last_scrub_stamp":"2024-03-11T21:35:31.695263+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:41.544550+0000","last_clean_scrub_stamp":"2024-03-11T21:35:31.695263+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,5],"acting":[0,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.a","version":"366'2880","reported_seq":2910,"reported_epoch":366,"state":"active+clean","last_fresh":"2024-03-11T21:41:12.341691+0000","last_change":"2024-03-11T21:40:39.243111+0000","last_active":"2024-03-11T21:41:12.341691+0000","last_peered":"2024-03-11T21:41:12.341691+0000","last_clean":"2024-03-11T21:41:12.341691+0000","last_became_active":"2024-03-11T21:40:39.242809+0000","last_became_peered":"2024-03-11T21:40:39.242809+0000","last_unstale":"2024-03-11T21:41:12.341691+0000","last_undegraded":"2024-03-11T21:41:12.341691+0000","last_fullsized":"2024-03-11T21:41:12.341691+0000","mapping_epoch":356,"log_start":"0'0","ondisk_log_start":"0'0","created":356,"last_epoch_clean":357,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:40:38.219602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:40:38.219602+0000","last_clean_scrub_stamp":"2024-03-11T21:40:38.219602+0000","objects_scrubbed":0,"log_size":2880,"log_dups_size":0,"ondisk_log_size":2880,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:41:54.098464+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":188743680,"num_objects":2880,"num_object_clones":0,"num_object_copies":8640,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2880,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2880,"num_write_kb":184320,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":586,"reported_epoch":366,"state":"active+clean","last_fresh":"2024-03-11T21:41:02.594768+0000","last_change":"2024-03-11T21:36:35.068977+0000","last_active":"2024-03-11T21:41:02.594768+0000","last_peered":"2024-03-11T21:41:02.594768+0000","last_clean":"2024-03-11T21:41:02.594768+0000","last_became_active":"2024-03-11T21:36:35.067288+0000","last_became_peered":"2024-03-11T21:36:35.067288+0000","last_unstale":"2024-03-11T21:41:02.594768+0000","last_undegraded":"2024-03-11T21:41:02.594768+0000","last_fullsized":"2024-03-11T21:41:02.594768+0000","mapping_epoch":277,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":278,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:35:09.985377+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:28:30.563712+0000","last_clean_scrub_stamp":"2024-03-11T21:35:09.985377+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,7],"acting":[3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":663,"reported_epoch":366,"state":"active+clean","last_fresh":"2024-03-11T21:41:02.588155+0000","last_change":"2024-03-11T21:35:40.116979+0000","last_active":"2024-03-11T21:41:02.588155+0000","last_peered":"2024-03-11T21:41:02.588155+0000","last_clean":"2024-03-11T21:41:02.588155+0000","last_became_active":"2024-03-11T21:25:38.538540+0000","last_became_peered":"2024-03-11T21:25:38.538540+0000","last_unstale":"2024-03-11T21:41:02.588155+0000","last_undegraded":"2024-03-11T21:41:02.588155+0000","last_fullsized":"2024-03-11T21:41:02.588155+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:35:40.116877+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:37.512018+0000","last_clean_scrub_stamp":"2024-03-11T21:35:40.116877+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:41:03.775240+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.4","version":"280'42","reported_seq":929,"reported_epoch":366,"state":"active+clean","last_fresh":"2024-03-11T21:41:05.234088+0000","last_change":"2024-03-11T21:41:05.234160+0000","last_active":"2024-03-11T21:41:05.234088+0000","last_peered":"2024-03-11T21:41:05.234088+0000","last_clean":"2024-03-11T21:41:05.234088+0000","last_became_active":"2024-03-11T21:36:35.073430+0000","last_became_peered":"2024-03-11T21:36:35.073430+0000","last_unstale":"2024-03-11T21:41:05.234088+0000","last_undegraded":"2024-03-11T21:41:05.234088+0000","last_fullsized":"2024-03-11T21:41:05.234088+0000","mapping_epoch":277,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":278,"parent":"0.0","parent_split_bits":0,"last_scrub":"280'42","last_scrub_stamp":"2024-03-11T21:41:05.233993+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:39.525554+0000","last_clean_scrub_stamp":"2024-03-11T21:41:05.233993+0000","objects_scrubbed":10,"log_size":42,"log_dups_size":0,"ondisk_log_size":42,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:42:33.447830+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":358,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":66,"num_read_kb":51,"num_write":38,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3],"acting":[1,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":648,"reported_epoch":366,"state":"active+clean","last_fresh":"2024-03-11T21:41:02.595233+0000","last_change":"2024-03-11T21:35:32.651731+0000","last_active":"2024-03-11T21:41:02.595233+0000","last_peered":"2024-03-11T21:41:02.595233+0000","last_clean":"2024-03-11T21:41:02.595233+0000","last_became_active":"2024-03-11T21:25:42.572673+0000","last_became_peered":"2024-03-11T21:25:42.572673+0000","last_unstale":"2024-03-11T21:41:02.595233+0000","last_undegraded":"2024-03-11T21:41:02.595233+0000","last_fullsized":"2024-03-11T21:41:02.595233+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:35:32.651652+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:41.544550+0000","last_clean_scrub_stamp":"2024-03-11T21:35:32.651652+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.9","version":"366'2908","reported_seq":2938,"reported_epoch":366,"state":"active+clean","last_fresh":"2024-03-11T21:41:12.521858+0000","last_change":"2024-03-11T21:40:39.241786+0000","last_active":"2024-03-11T21:41:12.521858+0000","last_peered":"2024-03-11T21:41:12.521858+0000","last_clean":"2024-03-11T21:41:12.521858+0000","last_became_active":"2024-03-11T21:40:39.241438+0000","last_became_peered":"2024-03-11T21:40:39.241438+0000","last_unstale":"2024-03-11T21:41:12.521858+0000","last_undegraded":"2024-03-11T21:41:12.521858+0000","last_fullsized":"2024-03-11T21:41:12.521858+0000","mapping_epoch":356,"log_start":"0'0","ondisk_log_start":"0'0","created":356,"last_epoch_clean":357,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:40:38.219602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:40:38.219602+0000","last_clean_scrub_stamp":"2024-03-11T21:40:38.219602+0000","objects_scrubbed":0,"log_size":2908,"log_dups_size":0,"ondisk_log_size":2908,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:42:00.768735+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":190578688,"num_objects":2908,"num_object_clones":0,"num_object_copies":8724,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2908,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2908,"num_write_kb":186112,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.5","version":"0'0","reported_seq":317,"reported_epoch":366,"state":"active+clean","last_fresh":"2024-03-11T21:41:12.380239+0000","last_change":"2024-03-11T21:41:12.380275+0000","last_active":"2024-03-11T21:41:12.380239+0000","last_peered":"2024-03-11T21:41:12.380239+0000","last_clean":"2024-03-11T21:41:12.380239+0000","last_became_active":"2024-03-11T21:36:35.069358+0000","last_became_peered":"2024-03-11T21:36:35.069358+0000","last_unstale":"2024-03-11T21:41:12.380239+0000","last_undegraded":"2024-03-11T21:41:12.380239+0000","last_fullsized":"2024-03-11T21:41:12.380239+0000","mapping_epoch":277,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":278,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:41:12.380019+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:43.559452+0000","last_clean_scrub_stamp":"2024-03-11T21:41:12.380019+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:42:39.666062+0000","scrub_duration":53,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,3],"acting":[7,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.0","version":"164'6","reported_seq":1589,"reported_epoch":366,"state":"active+clean","last_fresh":"2024-03-11T21:41:07.271569+0000","last_change":"2024-03-11T21:35:38.518243+0000","last_active":"2024-03-11T21:41:07.271569+0000","last_peered":"2024-03-11T21:41:07.271569+0000","last_clean":"2024-03-11T21:41:07.271569+0000","last_became_active":"2024-03-11T21:25:38.536890+0000","last_became_peered":"2024-03-11T21:25:38.536890+0000","last_unstale":"2024-03-11T21:41:07.271569+0000","last_undegraded":"2024-03-11T21:41:07.271569+0000","last_fullsized":"2024-03-11T21:41:07.271569+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"164'6","last_scrub_stamp":"2024-03-11T21:35:38.518097+0000","last_deep_scrub":"164'6","last_deep_scrub_stamp":"2024-03-11T21:32:17.597666+0000","last_clean_scrub_stamp":"2024-03-11T21:35:38.518097+0000","objects_scrubbed":1,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0.000753559,"stat_sum":{"num_bytes":389,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":922,"num_read_kb":917,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,6],"acting":[1,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":671,"reported_epoch":366,"state":"active+clean","last_fresh":"2024-03-11T21:41:02.592446+0000","last_change":"2024-03-11T21:40:09.915304+0000","last_active":"2024-03-11T21:41:02.592446+0000","last_peered":"2024-03-11T21:41:02.592446+0000","last_clean":"2024-03-11T21:41:02.592446+0000","last_became_active":"2024-03-11T21:36:35.056966+0000","last_became_peered":"2024-03-11T21:36:35.056966+0000","last_unstale":"2024-03-11T21:41:02.592446+0000","last_undegraded":"2024-03-11T21:41:02.592446+0000","last_fullsized":"2024-03-11T21:41:02.592446+0000","mapping_epoch":277,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":278,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:40:09.915179+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:40:09.915179+0000","last_clean_scrub_stamp":"2024-03-11T21:40:09.915179+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:41:27.096360+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.7","version":"260'22","reported_seq":693,"reported_epoch":366,"state":"active+clean","last_fresh":"2024-03-11T21:41:02.591194+0000","last_change":"2024-03-11T21:35:37.508679+0000","last_active":"2024-03-11T21:41:02.591194+0000","last_peered":"2024-03-11T21:41:02.591194+0000","last_clean":"2024-03-11T21:41:02.591194+0000","last_became_active":"2024-03-11T21:25:40.541769+0000","last_became_peered":"2024-03-11T21:25:40.541769+0000","last_unstale":"2024-03-11T21:41:02.591194+0000","last_undegraded":"2024-03-11T21:41:02.591194+0000","last_fullsized":"2024-03-11T21:41:02.591194+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"97'18","last_scrub_stamp":"2024-03-11T21:35:37.508567+0000","last_deep_scrub":"97'18","last_deep_scrub_stamp":"2024-03-11T21:27:19.239404+0000","last_clean_scrub_stamp":"2024-03-11T21:35:37.508567+0000","objects_scrubbed":5,"log_size":22,"log_dups_size":0,"ondisk_log_size":22,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":36,"num_read_kb":25,"num_write":20,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":666,"reported_epoch":366,"state":"active+clean","last_fresh":"2024-03-11T21:41:04.111539+0000","last_change":"2024-03-11T21:41:04.111560+0000","last_active":"2024-03-11T21:41:04.111539+0000","last_peered":"2024-03-11T21:41:04.111539+0000","last_clean":"2024-03-11T21:41:04.111539+0000","last_became_active":"2024-03-11T21:34:41.143660+0000","last_became_peered":"2024-03-11T21:34:41.143660+0000","last_unstale":"2024-03-11T21:41:04.111539+0000","last_undegraded":"2024-03-11T21:41:04.111539+0000","last_fullsized":"2024-03-11T21:41:04.111539+0000","mapping_epoch":226,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":227,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:41:04.111481+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:41.544550+0000","last_clean_scrub_stamp":"2024-03-11T21:41:04.111481+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:42:15.140933+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7],"acting":[2,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"10.b","version":"366'2928","reported_seq":2957,"reported_epoch":366,"state":"active+clean","last_fresh":"2024-03-11T21:41:12.338382+0000","last_change":"2024-03-11T21:40:39.241086+0000","last_active":"2024-03-11T21:41:12.338382+0000","last_peered":"2024-03-11T21:41:12.338382+0000","last_clean":"2024-03-11T21:41:12.338382+0000","last_became_active":"2024-03-11T21:40:39.240775+0000","last_became_peered":"2024-03-11T21:40:39.240775+0000","last_unstale":"2024-03-11T21:41:12.338382+0000","last_undegraded":"2024-03-11T21:41:12.338382+0000","last_fullsized":"2024-03-11T21:41:12.338382+0000","mapping_epoch":356,"log_start":"0'0","ondisk_log_start":"0'0","created":356,"last_epoch_clean":357,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:40:38.219602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:40:38.219602+0000","last_clean_scrub_stamp":"2024-03-11T21:40:38.219602+0000","objects_scrubbed":0,"log_size":2928,"log_dups_size":0,"ondisk_log_size":2928,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:41:50.594649+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":191889408,"num_objects":2928,"num_object_clones":0,"num_object_copies":8784,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2928,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2928,"num_write_kb":187392,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,0],"acting":[6,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.7","version":"124'2","reported_seq":654,"reported_epoch":366,"state":"active+clean","last_fresh":"2024-03-11T21:41:02.590113+0000","last_change":"2024-03-11T21:40:31.722243+0000","last_active":"2024-03-11T21:41:02.590113+0000","last_peered":"2024-03-11T21:41:02.590113+0000","last_clean":"2024-03-11T21:41:02.590113+0000","last_became_active":"2024-03-11T21:36:35.058786+0000","last_became_peered":"2024-03-11T21:36:35.058786+0000","last_unstale":"2024-03-11T21:41:02.590113+0000","last_undegraded":"2024-03-11T21:41:02.590113+0000","last_fullsized":"2024-03-11T21:41:02.590113+0000","mapping_epoch":277,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":278,"parent":"0.0","parent_split_bits":0,"last_scrub":"124'2","last_scrub_stamp":"2024-03-11T21:40:31.722100+0000","last_deep_scrub":"124'2","last_deep_scrub_stamp":"2024-03-11T21:29:34.956908+0000","last_clean_scrub_stamp":"2024-03-11T21:40:31.722100+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:41:45.441808+0000","scrub_duration":87,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,4],"acting":[5,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2","version":"71'2","reported_seq":615,"reported_epoch":366,"state":"active+clean","last_fresh":"2024-03-11T21:41:07.517240+0000","last_change":"2024-03-11T21:41:07.517286+0000","last_active":"2024-03-11T21:41:07.517240+0000","last_peered":"2024-03-11T21:41:07.517240+0000","last_clean":"2024-03-11T21:41:07.517240+0000","last_became_active":"2024-03-11T21:36:35.070783+0000","last_became_peered":"2024-03-11T21:36:35.070783+0000","last_unstale":"2024-03-11T21:41:07.517240+0000","last_undegraded":"2024-03-11T21:41:07.517240+0000","last_fullsized":"2024-03-11T21:41:07.517240+0000","mapping_epoch":277,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":278,"parent":"0.0","parent_split_bits":0,"last_scrub":"71'2","last_scrub_stamp":"2024-03-11T21:41:07.517154+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:37.512018+0000","last_clean_scrub_stamp":"2024-03-11T21:41:07.517154+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:42:32.692076+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0.0014206449999999999,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":681,"reported_epoch":366,"state":"active+clean","last_fresh":"2024-03-11T21:41:10.803012+0000","last_change":"2024-03-11T21:41:10.803049+0000","last_active":"2024-03-11T21:41:10.803012+0000","last_peered":"2024-03-11T21:41:10.803012+0000","last_clean":"2024-03-11T21:41:10.803012+0000","last_became_active":"2024-03-11T21:34:41.146672+0000","last_became_peered":"2024-03-11T21:34:41.146672+0000","last_unstale":"2024-03-11T21:41:10.803012+0000","last_undegraded":"2024-03-11T21:41:10.803012+0000","last_fullsized":"2024-03-11T21:41:10.803012+0000","mapping_epoch":226,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":227,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:41:10.802921+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:37.512018+0000","last_clean_scrub_stamp":"2024-03-11T21:41:10.802921+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:42:24.686198+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"87'128","reported_seq":779,"reported_epoch":366,"state":"active+clean","last_fresh":"2024-03-11T21:41:09.328274+0000","last_change":"2024-03-11T21:41:09.328321+0000","last_active":"2024-03-11T21:41:09.328274+0000","last_peered":"2024-03-11T21:41:09.328274+0000","last_clean":"2024-03-11T21:41:09.328274+0000","last_became_active":"2024-03-11T21:34:44.164569+0000","last_became_peered":"2024-03-11T21:34:44.164569+0000","last_unstale":"2024-03-11T21:41:09.328274+0000","last_undegraded":"2024-03-11T21:41:09.328274+0000","last_fullsized":"2024-03-11T21:41:09.328274+0000","mapping_epoch":229,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":230,"parent":"0.0","parent_split_bits":0,"last_scrub":"87'128","last_scrub_stamp":"2024-03-11T21:41:09.328197+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:24.261974+0000","last_clean_scrub_stamp":"2024-03-11T21:41:09.328197+0000","objects_scrubbed":2,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:42:21.367178+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":198,"num_read_kb":294,"num_write":179,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":2951840,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.5","version":"260'28","reported_seq":903,"reported_epoch":366,"state":"active+clean","last_fresh":"2024-03-11T21:41:07.923860+0000","last_change":"2024-03-11T21:41:07.923860+0000","last_active":"2024-03-11T21:41:07.923860+0000","last_peered":"2024-03-11T21:41:07.923860+0000","last_clean":"2024-03-11T21:41:07.923860+0000","last_became_active":"2024-03-11T21:25:40.761457+0000","last_became_peered":"2024-03-11T21:25:40.761457+0000","last_unstale":"2024-03-11T21:41:07.923860+0000","last_undegraded":"2024-03-11T21:41:07.923860+0000","last_fullsized":"2024-03-11T21:41:07.923860+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"260'28","last_scrub_stamp":"2024-03-11T21:41:07.923729+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:39.525554+0000","last_clean_scrub_stamp":"2024-03-11T21:41:07.923729+0000","objects_scrubbed":8,"log_size":28,"log_dups_size":0,"ondisk_log_size":28,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:42:18.026233+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":154,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":43,"num_read_kb":33,"num_write":25,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":308,"reported_epoch":366,"state":"active+clean","last_fresh":"2024-03-11T21:41:02.590455+0000","last_change":"2024-03-11T21:41:00.152627+0000","last_active":"2024-03-11T21:41:02.590455+0000","last_peered":"2024-03-11T21:41:02.590455+0000","last_clean":"2024-03-11T21:41:02.590455+0000","last_became_active":"2024-03-11T21:34:41.139782+0000","last_became_peered":"2024-03-11T21:34:41.139782+0000","last_unstale":"2024-03-11T21:41:02.590455+0000","last_undegraded":"2024-03-11T21:41:02.590455+0000","last_fullsized":"2024-03-11T21:41:02.590455+0000","mapping_epoch":226,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":227,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:41:00.152377+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:32:21.266672+0000","last_clean_scrub_stamp":"2024-03-11T21:41:00.152377+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:42:07.847860+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,5],"acting":[7,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.8","version":"366'2483","reported_seq":2513,"reported_epoch":366,"state":"active+clean","last_fresh":"2024-03-11T21:41:08.802394+0000","last_change":"2024-03-11T21:40:39.237338+0000","last_active":"2024-03-11T21:41:08.802394+0000","last_peered":"2024-03-11T21:41:08.802394+0000","last_clean":"2024-03-11T21:41:08.802394+0000","last_became_active":"2024-03-11T21:40:39.237027+0000","last_became_peered":"2024-03-11T21:40:39.237027+0000","last_unstale":"2024-03-11T21:41:08.802394+0000","last_undegraded":"2024-03-11T21:41:08.802394+0000","last_fullsized":"2024-03-11T21:41:08.802394+0000","mapping_epoch":356,"log_start":"0'0","ondisk_log_start":"0'0","created":356,"last_epoch_clean":357,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:40:38.219602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:40:38.219602+0000","last_clean_scrub_stamp":"2024-03-11T21:40:38.219602+0000","objects_scrubbed":0,"log_size":2483,"log_dups_size":0,"ondisk_log_size":2483,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:42:03.918404+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":162725888,"num_objects":2483,"num_object_clones":0,"num_object_copies":7449,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2483,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2483,"num_write_kb":158912,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,7],"acting":[0,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.4","version":"0'0","reported_seq":653,"reported_epoch":366,"state":"active+clean","last_fresh":"2024-03-11T21:41:02.590836+0000","last_change":"2024-03-11T21:41:01.134631+0000","last_active":"2024-03-11T21:41:02.590836+0000","last_peered":"2024-03-11T21:41:02.590836+0000","last_clean":"2024-03-11T21:41:02.590836+0000","last_became_active":"2024-03-11T21:36:35.072824+0000","last_became_peered":"2024-03-11T21:36:35.072824+0000","last_unstale":"2024-03-11T21:41:02.590836+0000","last_undegraded":"2024-03-11T21:41:02.590836+0000","last_fullsized":"2024-03-11T21:41:02.590836+0000","mapping_epoch":277,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":278,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:41:01.134504+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:43.559452+0000","last_clean_scrub_stamp":"2024-03-11T21:41:01.134504+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:42:10.364426+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":680,"reported_epoch":366,"state":"active+clean","last_fresh":"2024-03-11T21:41:02.588316+0000","last_change":"2024-03-11T21:41:02.116795+0000","last_active":"2024-03-11T21:41:02.588316+0000","last_peered":"2024-03-11T21:41:02.588316+0000","last_clean":"2024-03-11T21:41:02.588316+0000","last_became_active":"2024-03-11T21:36:35.069452+0000","last_became_peered":"2024-03-11T21:36:35.069452+0000","last_unstale":"2024-03-11T21:41:02.588316+0000","last_undegraded":"2024-03-11T21:41:02.588316+0000","last_fullsized":"2024-03-11T21:41:02.588316+0000","mapping_epoch":277,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":278,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:41:02.116682+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:37.512018+0000","last_clean_scrub_stamp":"2024-03-11T21:41:02.116682+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:42:10.327752+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0.00089342499999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":300,"reported_epoch":366,"state":"active+clean","last_fresh":"2024-03-11T21:41:02.589738+0000","last_change":"2024-03-11T21:40:27.117606+0000","last_active":"2024-03-11T21:41:02.589738+0000","last_peered":"2024-03-11T21:41:02.589738+0000","last_clean":"2024-03-11T21:41:02.589738+0000","last_became_active":"2024-03-11T21:34:41.149219+0000","last_became_peered":"2024-03-11T21:34:41.149219+0000","last_unstale":"2024-03-11T21:41:02.589738+0000","last_undegraded":"2024-03-11T21:41:02.589738+0000","last_fullsized":"2024-03-11T21:41:02.589738+0000","mapping_epoch":226,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":227,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:40:27.117515+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:37.512018+0000","last_clean_scrub_stamp":"2024-03-11T21:40:27.117515+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:41:35.677145+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,0],"acting":[7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.6","version":"352'30","reported_seq":722,"reported_epoch":366,"state":"active+clean","last_fresh":"2024-03-11T21:41:02.590192+0000","last_change":"2024-03-11T21:40:31.175417+0000","last_active":"2024-03-11T21:41:02.590192+0000","last_peered":"2024-03-11T21:41:02.590192+0000","last_clean":"2024-03-11T21:41:02.590192+0000","last_became_active":"2024-03-11T21:32:22.796486+0000","last_became_peered":"2024-03-11T21:32:22.796486+0000","last_unstale":"2024-03-11T21:41:02.590192+0000","last_undegraded":"2024-03-11T21:41:02.590192+0000","last_fullsized":"2024-03-11T21:41:02.590192+0000","mapping_epoch":166,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":167,"parent":"0.0","parent_split_bits":0,"last_scrub":"352'30","last_scrub_stamp":"2024-03-11T21:40:31.175323+0000","last_deep_scrub":"352'30","last_deep_scrub_stamp":"2024-03-11T21:40:31.175323+0000","last_clean_scrub_stamp":"2024-03-11T21:40:31.175323+0000","objects_scrubbed":6,"log_size":30,"log_dups_size":0,"ondisk_log_size":30,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:41:40.436187+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":36,"num_read_kb":30,"num_write":24,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":647,"reported_epoch":366,"state":"active+clean","last_fresh":"2024-03-11T21:41:02.590028+0000","last_change":"2024-03-11T21:36:07.451482+0000","last_active":"2024-03-11T21:41:02.590028+0000","last_peered":"2024-03-11T21:41:02.590028+0000","last_clean":"2024-03-11T21:41:02.590028+0000","last_became_active":"2024-03-11T21:25:42.572324+0000","last_became_peered":"2024-03-11T21:25:42.572324+0000","last_unstale":"2024-03-11T21:41:02.590028+0000","last_undegraded":"2024-03-11T21:41:02.590028+0000","last_fullsized":"2024-03-11T21:41:02.590028+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:36:07.451102+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:41.544550+0000","last_clean_scrub_stamp":"2024-03-11T21:36:07.451102+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:41:03.775363+0000","scrub_duration":54,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.d","version":"366'2870","reported_seq":2900,"reported_epoch":366,"state":"active+clean","last_fresh":"2024-03-11T21:41:12.334468+0000","last_change":"2024-03-11T21:40:39.242068+0000","last_active":"2024-03-11T21:41:12.334468+0000","last_peered":"2024-03-11T21:41:12.334468+0000","last_clean":"2024-03-11T21:41:12.334468+0000","last_became_active":"2024-03-11T21:40:39.241769+0000","last_became_peered":"2024-03-11T21:40:39.241769+0000","last_unstale":"2024-03-11T21:41:12.334468+0000","last_undegraded":"2024-03-11T21:41:12.334468+0000","last_fullsized":"2024-03-11T21:41:12.334468+0000","mapping_epoch":356,"log_start":"0'0","ondisk_log_start":"0'0","created":356,"last_epoch_clean":357,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:40:38.219602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:40:38.219602+0000","last_clean_scrub_stamp":"2024-03-11T21:40:38.219602+0000","objects_scrubbed":0,"log_size":2870,"log_dups_size":0,"ondisk_log_size":2870,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:41:57.523479+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":188088320,"num_objects":2870,"num_object_clones":0,"num_object_copies":8610,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2870,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2870,"num_write_kb":183680,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,0],"acting":[6,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":652,"reported_epoch":366,"state":"active+clean","last_fresh":"2024-03-11T21:41:02.583619+0000","last_change":"2024-03-11T21:36:18.529506+0000","last_active":"2024-03-11T21:41:02.583619+0000","last_peered":"2024-03-11T21:41:02.583619+0000","last_clean":"2024-03-11T21:41:02.583619+0000","last_became_active":"2024-03-11T21:25:45.301416+0000","last_became_peered":"2024-03-11T21:25:45.301416+0000","last_unstale":"2024-03-11T21:41:02.583619+0000","last_undegraded":"2024-03-11T21:41:02.583619+0000","last_fullsized":"2024-03-11T21:41:02.583619+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:36:18.529370+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:43.559452+0000","last_clean_scrub_stamp":"2024-03-11T21:36:18.529370+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":54,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":300,"reported_epoch":366,"state":"active+clean","last_fresh":"2024-03-11T21:41:02.592388+0000","last_change":"2024-03-11T21:35:33.740615+0000","last_active":"2024-03-11T21:41:02.592388+0000","last_peered":"2024-03-11T21:41:02.592388+0000","last_clean":"2024-03-11T21:41:02.592388+0000","last_became_active":"2024-03-11T21:34:41.152721+0000","last_became_peered":"2024-03-11T21:34:41.152721+0000","last_unstale":"2024-03-11T21:41:02.592388+0000","last_undegraded":"2024-03-11T21:41:02.592388+0000","last_fullsized":"2024-03-11T21:41:02.592388+0000","mapping_epoch":226,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":227,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:35:33.740538+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:26:57.333669+0000","last_clean_scrub_stamp":"2024-03-11T21:35:33.740538+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.3","version":"260'37","reported_seq":767,"reported_epoch":366,"state":"active+clean","last_fresh":"2024-03-11T21:41:06.123319+0000","last_change":"2024-03-11T21:41:06.123397+0000","last_active":"2024-03-11T21:41:06.123319+0000","last_peered":"2024-03-11T21:41:06.123319+0000","last_clean":"2024-03-11T21:41:06.123319+0000","last_became_active":"2024-03-11T21:34:41.145607+0000","last_became_peered":"2024-03-11T21:34:41.145607+0000","last_unstale":"2024-03-11T21:41:06.123319+0000","last_undegraded":"2024-03-11T21:41:06.123319+0000","last_fullsized":"2024-03-11T21:41:06.123319+0000","mapping_epoch":226,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":227,"parent":"0.0","parent_split_bits":0,"last_scrub":"260'37","last_scrub_stamp":"2024-03-11T21:41:06.123055+0000","last_deep_scrub":"165'27","last_deep_scrub_stamp":"2024-03-11T21:32:21.576732+0000","last_clean_scrub_stamp":"2024-03-11T21:41:06.123055+0000","objects_scrubbed":8,"log_size":37,"log_dups_size":0,"ondisk_log_size":37,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:42:13.040362+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":54,"num_read_kb":40,"num_write":32,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":660,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":672,"reported_epoch":366,"state":"active+clean","last_fresh":"2024-03-11T21:41:02.586652+0000","last_change":"2024-03-11T21:40:15.172829+0000","last_active":"2024-03-11T21:41:02.586652+0000","last_peered":"2024-03-11T21:41:02.586652+0000","last_clean":"2024-03-11T21:41:02.586652+0000","last_became_active":"2024-03-11T21:25:42.579045+0000","last_became_peered":"2024-03-11T21:25:42.579045+0000","last_unstale":"2024-03-11T21:41:02.586652+0000","last_undegraded":"2024-03-11T21:41:02.586652+0000","last_fullsized":"2024-03-11T21:41:02.586652+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:40:15.172720+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:40:15.172720+0000","last_clean_scrub_stamp":"2024-03-11T21:40:15.172720+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:41:19.672684+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"10.e","version":"366'2831","reported_seq":2861,"reported_epoch":366,"state":"active+clean","last_fresh":"2024-03-11T21:41:12.195923+0000","last_change":"2024-03-11T21:40:39.239887+0000","last_active":"2024-03-11T21:41:12.195923+0000","last_peered":"2024-03-11T21:41:12.195923+0000","last_clean":"2024-03-11T21:41:12.195923+0000","last_became_active":"2024-03-11T21:40:39.239590+0000","last_became_peered":"2024-03-11T21:40:39.239590+0000","last_unstale":"2024-03-11T21:41:12.195923+0000","last_undegraded":"2024-03-11T21:41:12.195923+0000","last_fullsized":"2024-03-11T21:41:12.195923+0000","mapping_epoch":356,"log_start":"0'0","ondisk_log_start":"0'0","created":356,"last_epoch_clean":357,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:40:38.219602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:40:38.219602+0000","last_clean_scrub_stamp":"2024-03-11T21:40:38.219602+0000","objects_scrubbed":0,"log_size":2831,"log_dups_size":0,"ondisk_log_size":2831,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:42:05.905824+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":185532416,"num_objects":2831,"num_object_clones":0,"num_object_copies":8493,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2831,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2831,"num_write_kb":181184,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,2],"acting":[4,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":658,"reported_epoch":366,"state":"active+clean","last_fresh":"2024-03-11T21:41:02.593527+0000","last_change":"2024-03-11T21:40:28.582452+0000","last_active":"2024-03-11T21:41:02.593527+0000","last_peered":"2024-03-11T21:41:02.593527+0000","last_clean":"2024-03-11T21:41:02.593527+0000","last_became_active":"2024-03-11T21:32:22.791439+0000","last_became_peered":"2024-03-11T21:32:22.791439+0000","last_unstale":"2024-03-11T21:41:02.593527+0000","last_undegraded":"2024-03-11T21:41:02.593527+0000","last_fullsized":"2024-03-11T21:41:02.593527+0000","mapping_epoch":166,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":167,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:40:28.582339+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:43.559452+0000","last_clean_scrub_stamp":"2024-03-11T21:40:28.582339+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:41:49.291632+0000","scrub_duration":20,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,2],"acting":[4,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.6","version":"67'1","reported_seq":657,"reported_epoch":366,"state":"active+clean","last_fresh":"2024-03-11T21:41:02.591148+0000","last_change":"2024-03-11T21:35:30.581783+0000","last_active":"2024-03-11T21:41:02.591148+0000","last_peered":"2024-03-11T21:41:02.591148+0000","last_clean":"2024-03-11T21:41:02.591148+0000","last_became_active":"2024-03-11T21:25:38.536142+0000","last_became_peered":"2024-03-11T21:25:38.536142+0000","last_unstale":"2024-03-11T21:41:02.591148+0000","last_undegraded":"2024-03-11T21:41:02.591148+0000","last_fullsized":"2024-03-11T21:41:02.591148+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'1","last_scrub_stamp":"2024-03-11T21:35:30.581701+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:37.512018+0000","last_clean_scrub_stamp":"2024-03-11T21:35:30.581701+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,4],"acting":[1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.0","version":"260'38","reported_seq":655,"reported_epoch":366,"state":"active+clean","last_fresh":"2024-03-11T21:41:02.592171+0000","last_change":"2024-03-11T21:36:35.087771+0000","last_active":"2024-03-11T21:41:02.592171+0000","last_peered":"2024-03-11T21:41:02.592171+0000","last_clean":"2024-03-11T21:41:02.592171+0000","last_became_active":"2024-03-11T21:36:35.068071+0000","last_became_peered":"2024-03-11T21:36:35.068071+0000","last_unstale":"2024-03-11T21:41:02.592171+0000","last_undegraded":"2024-03-11T21:41:02.592171+0000","last_fullsized":"2024-03-11T21:41:02.592171+0000","mapping_epoch":277,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":278,"parent":"0.0","parent_split_bits":0,"last_scrub":"93'26","last_scrub_stamp":"2024-03-11T21:34:16.863984+0000","last_deep_scrub":"93'26","last_deep_scrub_stamp":"2024-03-11T21:26:40.680324+0000","last_clean_scrub_stamp":"2024-03-11T21:34:16.863984+0000","objects_scrubbed":8,"log_size":38,"log_dups_size":0,"ondisk_log_size":38,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":22,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":51,"num_read_kb":39,"num_write":32,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":677,"reported_epoch":366,"state":"active+clean","last_fresh":"2024-03-11T21:41:02.591282+0000","last_change":"2024-03-11T21:40:21.518921+0000","last_active":"2024-03-11T21:41:02.591282+0000","last_peered":"2024-03-11T21:41:02.591282+0000","last_clean":"2024-03-11T21:41:02.591282+0000","last_became_active":"2024-03-11T21:36:35.057517+0000","last_became_peered":"2024-03-11T21:36:35.057517+0000","last_unstale":"2024-03-11T21:41:02.591282+0000","last_undegraded":"2024-03-11T21:41:02.591282+0000","last_fullsized":"2024-03-11T21:41:02.591282+0000","mapping_epoch":277,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":278,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:40:21.518840+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:40:21.518840+0000","last_clean_scrub_stamp":"2024-03-11T21:40:21.518840+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:41:26.549696+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,7],"acting":[4,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"10.f","version":"366'2853","reported_seq":2883,"reported_epoch":366,"state":"active+clean","last_fresh":"2024-03-11T21:41:12.326538+0000","last_change":"2024-03-11T21:40:39.541796+0000","last_active":"2024-03-11T21:41:12.326538+0000","last_peered":"2024-03-11T21:41:12.326538+0000","last_clean":"2024-03-11T21:41:12.326538+0000","last_became_active":"2024-03-11T21:40:39.541420+0000","last_became_peered":"2024-03-11T21:40:39.541420+0000","last_unstale":"2024-03-11T21:41:12.326538+0000","last_undegraded":"2024-03-11T21:41:12.326538+0000","last_fullsized":"2024-03-11T21:41:12.326538+0000","mapping_epoch":356,"log_start":"0'0","ondisk_log_start":"0'0","created":356,"last_epoch_clean":357,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:40:38.219602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:40:38.219602+0000","last_clean_scrub_stamp":"2024-03-11T21:40:38.219602+0000","objects_scrubbed":0,"log_size":2853,"log_dups_size":0,"ondisk_log_size":2853,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:42:06.741344+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":186974208,"num_objects":2853,"num_object_clones":0,"num_object_copies":8559,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2853,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2853,"num_write_kb":182592,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,2],"acting":[6,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.3","version":"0'0","reported_seq":300,"reported_epoch":366,"state":"active+clean","last_fresh":"2024-03-11T21:41:02.592069+0000","last_change":"2024-03-11T21:40:25.234980+0000","last_active":"2024-03-11T21:41:02.592069+0000","last_peered":"2024-03-11T21:41:02.592069+0000","last_clean":"2024-03-11T21:41:02.592069+0000","last_became_active":"2024-03-11T21:34:41.142683+0000","last_became_peered":"2024-03-11T21:34:41.142683+0000","last_unstale":"2024-03-11T21:41:02.592069+0000","last_undegraded":"2024-03-11T21:41:02.592069+0000","last_fullsized":"2024-03-11T21:41:02.592069+0000","mapping_epoch":226,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":227,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:40:25.213541+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:29:42.485740+0000","last_clean_scrub_stamp":"2024-03-11T21:40:25.213541+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:41:30.275517+0000","scrub_duration":55,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,2],"acting":[7,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":692,"reported_epoch":366,"state":"active+clean","last_fresh":"2024-03-11T21:41:02.585221+0000","last_change":"2024-03-11T21:40:07.187684+0000","last_active":"2024-03-11T21:41:02.585221+0000","last_peered":"2024-03-11T21:41:02.585221+0000","last_clean":"2024-03-11T21:41:02.585221+0000","last_became_active":"2024-03-11T21:34:41.145628+0000","last_became_peered":"2024-03-11T21:34:41.145628+0000","last_unstale":"2024-03-11T21:41:02.585221+0000","last_undegraded":"2024-03-11T21:41:02.585221+0000","last_fullsized":"2024-03-11T21:41:02.585221+0000","mapping_epoch":226,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":227,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:40:07.187595+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:37.512018+0000","last_clean_scrub_stamp":"2024-03-11T21:40:07.187595+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:41:20.424112+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,2],"acting":[6,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.1","version":"260'28","reported_seq":702,"reported_epoch":366,"state":"active+clean","last_fresh":"2024-03-11T21:41:07.512997+0000","last_change":"2024-03-11T21:41:07.513056+0000","last_active":"2024-03-11T21:41:07.512997+0000","last_peered":"2024-03-11T21:41:07.512997+0000","last_clean":"2024-03-11T21:41:07.512997+0000","last_became_active":"2024-03-11T21:25:40.546219+0000","last_became_peered":"2024-03-11T21:25:40.546219+0000","last_unstale":"2024-03-11T21:41:07.512997+0000","last_undegraded":"2024-03-11T21:41:07.512997+0000","last_fullsized":"2024-03-11T21:41:07.512997+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"260'28","last_scrub_stamp":"2024-03-11T21:41:07.512907+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:39.525554+0000","last_clean_scrub_stamp":"2024-03-11T21:41:07.512907+0000","objects_scrubbed":7,"log_size":28,"log_dups_size":0,"ondisk_log_size":28,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:42:29.651448+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":42,"num_read_kb":35,"num_write":28,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6],"acting":[4,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.0","version":"280'11","reported_seq":769,"reported_epoch":366,"state":"active+clean","last_fresh":"2024-03-11T21:41:03.780791+0000","last_change":"2024-03-11T21:36:35.064085+0000","last_active":"2024-03-11T21:41:03.780791+0000","last_peered":"2024-03-11T21:41:03.780791+0000","last_clean":"2024-03-11T21:41:03.780791+0000","last_became_active":"2024-03-11T21:36:35.055250+0000","last_became_peered":"2024-03-11T21:36:35.055250+0000","last_unstale":"2024-03-11T21:41:03.780791+0000","last_undegraded":"2024-03-11T21:41:03.780791+0000","last_fullsized":"2024-03-11T21:41:03.780791+0000","mapping_epoch":277,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":278,"parent":"0.0","parent_split_bits":0,"last_scrub":"98'9","last_scrub_stamp":"2024-03-11T21:35:21.143578+0000","last_deep_scrub":"98'9","last_deep_scrub_stamp":"2024-03-11T21:27:29.745711+0000","last_clean_scrub_stamp":"2024-03-11T21:35:21.143578+0000","objects_scrubbed":1,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.0","version":"366'2888","reported_seq":2917,"reported_epoch":366,"state":"active+clean","last_fresh":"2024-03-11T21:41:12.338519+0000","last_change":"2024-03-11T21:40:39.242132+0000","last_active":"2024-03-11T21:41:12.338519+0000","last_peered":"2024-03-11T21:41:12.338519+0000","last_clean":"2024-03-11T21:41:12.338519+0000","last_became_active":"2024-03-11T21:40:39.241802+0000","last_became_peered":"2024-03-11T21:40:39.241802+0000","last_unstale":"2024-03-11T21:41:12.338519+0000","last_undegraded":"2024-03-11T21:41:12.338519+0000","last_fullsized":"2024-03-11T21:41:12.338519+0000","mapping_epoch":356,"log_start":"0'0","ondisk_log_start":"0'0","created":356,"last_epoch_clean":357,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:40:38.219602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:40:38.219602+0000","last_clean_scrub_stamp":"2024-03-11T21:40:38.219602+0000","objects_scrubbed":0,"log_size":2888,"log_dups_size":0,"ondisk_log_size":2888,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:41:53.023515+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":189267968,"num_objects":2888,"num_object_clones":0,"num_object_copies":8664,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2888,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2888,"num_write_kb":184832,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,4],"acting":[6,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.c","version":"0'0","reported_seq":584,"reported_epoch":366,"state":"active+clean","last_fresh":"2024-03-11T21:41:02.594598+0000","last_change":"2024-03-11T21:40:30.659272+0000","last_active":"2024-03-11T21:41:02.594598+0000","last_peered":"2024-03-11T21:41:02.594598+0000","last_clean":"2024-03-11T21:41:02.594598+0000","last_became_active":"2024-03-11T21:36:35.073778+0000","last_became_peered":"2024-03-11T21:36:35.073778+0000","last_unstale":"2024-03-11T21:41:02.594598+0000","last_undegraded":"2024-03-11T21:41:02.594598+0000","last_fullsized":"2024-03-11T21:41:02.594598+0000","mapping_epoch":277,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":278,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:40:30.659160+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:43.559452+0000","last_clean_scrub_stamp":"2024-03-11T21:40:30.659160+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:41:38.577899+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.f","version":"0'0","reported_seq":654,"reported_epoch":366,"state":"active+clean","last_fresh":"2024-03-11T21:41:02.589816+0000","last_change":"2024-03-11T21:36:08.439013+0000","last_active":"2024-03-11T21:41:02.589816+0000","last_peered":"2024-03-11T21:41:02.589816+0000","last_clean":"2024-03-11T21:41:02.589816+0000","last_became_active":"2024-03-11T21:25:42.575935+0000","last_became_peered":"2024-03-11T21:25:42.575935+0000","last_unstale":"2024-03-11T21:41:02.589816+0000","last_undegraded":"2024-03-11T21:41:02.589816+0000","last_fullsized":"2024-03-11T21:41:02.589816+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:36:08.438895+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:41.544550+0000","last_clean_scrub_stamp":"2024-03-11T21:36:08.438895+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:41:03.775333+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.8","version":"0'0","reported_seq":300,"reported_epoch":366,"state":"active+clean","last_fresh":"2024-03-11T21:41:02.590377+0000","last_change":"2024-03-11T21:35:34.733595+0000","last_active":"2024-03-11T21:41:02.590377+0000","last_peered":"2024-03-11T21:41:02.590377+0000","last_clean":"2024-03-11T21:41:02.590377+0000","last_became_active":"2024-03-11T21:34:41.144681+0000","last_became_peered":"2024-03-11T21:34:41.144681+0000","last_unstale":"2024-03-11T21:41:02.590377+0000","last_undegraded":"2024-03-11T21:41:02.590377+0000","last_fullsized":"2024-03-11T21:41:02.590377+0000","mapping_epoch":226,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":227,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:35:34.733491+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:37.512018+0000","last_clean_scrub_stamp":"2024-03-11T21:35:34.733491+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,1],"acting":[7,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.e","version":"260'24","reported_seq":687,"reported_epoch":366,"state":"active+clean","last_fresh":"2024-03-11T21:41:02.593391+0000","last_change":"2024-03-11T21:35:34.679906+0000","last_active":"2024-03-11T21:41:02.593391+0000","last_peered":"2024-03-11T21:41:02.593391+0000","last_clean":"2024-03-11T21:41:02.593391+0000","last_became_active":"2024-03-11T21:25:40.552178+0000","last_became_peered":"2024-03-11T21:25:40.552178+0000","last_unstale":"2024-03-11T21:41:02.593391+0000","last_undegraded":"2024-03-11T21:41:02.593391+0000","last_fullsized":"2024-03-11T21:41:02.593391+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"109'16","last_scrub_stamp":"2024-03-11T21:35:34.679817+0000","last_deep_scrub":"109'16","last_deep_scrub_stamp":"2024-03-11T21:28:24.739395+0000","last_clean_scrub_stamp":"2024-03-11T21:35:34.679817+0000","objects_scrubbed":5,"log_size":24,"log_dups_size":0,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":32,"num_read_kb":25,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"10.1","version":"366'2842","reported_seq":2872,"reported_epoch":366,"state":"active+clean","last_fresh":"2024-03-11T21:41:12.226761+0000","last_change":"2024-03-11T21:40:39.236538+0000","last_active":"2024-03-11T21:41:12.226761+0000","last_peered":"2024-03-11T21:41:12.226761+0000","last_clean":"2024-03-11T21:41:12.226761+0000","last_became_active":"2024-03-11T21:40:39.236129+0000","last_became_peered":"2024-03-11T21:40:39.236129+0000","last_unstale":"2024-03-11T21:41:12.226761+0000","last_undegraded":"2024-03-11T21:41:12.226761+0000","last_fullsized":"2024-03-11T21:41:12.226761+0000","mapping_epoch":356,"log_start":"0'0","ondisk_log_start":"0'0","created":356,"last_epoch_clean":357,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:40:38.219602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:40:38.219602+0000","last_clean_scrub_stamp":"2024-03-11T21:40:38.219602+0000","objects_scrubbed":0,"log_size":2842,"log_dups_size":0,"ondisk_log_size":2842,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:41:38.778584+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":186253312,"num_objects":2842,"num_object_clones":0,"num_object_copies":8526,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2842,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2842,"num_write_kb":181888,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,0],"acting":[4,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.d","version":"0'0","reported_seq":642,"reported_epoch":366,"state":"active+clean","last_fresh":"2024-03-11T21:41:02.587678+0000","last_change":"2024-03-11T21:35:37.185210+0000","last_active":"2024-03-11T21:41:02.587678+0000","last_peered":"2024-03-11T21:41:02.587678+0000","last_clean":"2024-03-11T21:41:02.587678+0000","last_became_active":"2024-03-11T21:25:44.590333+0000","last_became_peered":"2024-03-11T21:25:44.590333+0000","last_unstale":"2024-03-11T21:41:02.587678+0000","last_undegraded":"2024-03-11T21:41:02.587678+0000","last_fullsized":"2024-03-11T21:41:02.587678+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:35:37.182850+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:35:37.182850+0000","last_clean_scrub_stamp":"2024-03-11T21:35:37.182850+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:41:03.775148+0000","scrub_duration":40,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.e","version":"74'8","reported_seq":842,"reported_epoch":366,"state":"active+clean","last_fresh":"2024-03-11T21:41:03.780836+0000","last_change":"2024-03-11T21:36:07.734369+0000","last_active":"2024-03-11T21:41:03.780836+0000","last_peered":"2024-03-11T21:41:03.780836+0000","last_clean":"2024-03-11T21:41:03.780836+0000","last_became_active":"2024-03-11T21:25:42.576742+0000","last_became_peered":"2024-03-11T21:25:42.576742+0000","last_unstale":"2024-03-11T21:41:03.780836+0000","last_undegraded":"2024-03-11T21:41:03.780836+0000","last_fullsized":"2024-03-11T21:41:03.780836+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'8","last_scrub_stamp":"2024-03-11T21:36:07.734115+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:41.544550+0000","last_clean_scrub_stamp":"2024-03-11T21:36:07.734115+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,0],"acting":[4,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.9","version":"0'0","reported_seq":677,"reported_epoch":366,"state":"active+clean","last_fresh":"2024-03-11T21:41:02.583446+0000","last_change":"2024-03-11T21:40:29.915912+0000","last_active":"2024-03-11T21:41:02.583446+0000","last_peered":"2024-03-11T21:41:02.583446+0000","last_clean":"2024-03-11T21:41:02.583446+0000","last_became_active":"2024-03-11T21:36:35.065733+0000","last_became_peered":"2024-03-11T21:36:35.065733+0000","last_unstale":"2024-03-11T21:41:02.583446+0000","last_undegraded":"2024-03-11T21:41:02.583446+0000","last_fullsized":"2024-03-11T21:41:02.583446+0000","mapping_epoch":277,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":278,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:40:29.915835+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:26:50.991712+0000","last_clean_scrub_stamp":"2024-03-11T21:40:29.915835+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:41:46.722143+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,3],"acting":[1,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.f","version":"260'27","reported_seq":677,"reported_epoch":366,"state":"active+clean","last_fresh":"2024-03-11T21:41:02.591748+0000","last_change":"2024-03-11T21:40:12.838694+0000","last_active":"2024-03-11T21:41:02.591748+0000","last_peered":"2024-03-11T21:41:02.591748+0000","last_clean":"2024-03-11T21:41:02.591748+0000","last_became_active":"2024-03-11T21:36:35.058272+0000","last_became_peered":"2024-03-11T21:36:35.058272+0000","last_unstale":"2024-03-11T21:41:02.591748+0000","last_undegraded":"2024-03-11T21:41:02.591748+0000","last_fullsized":"2024-03-11T21:41:02.591748+0000","mapping_epoch":277,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":278,"parent":"0.0","parent_split_bits":0,"last_scrub":"260'27","last_scrub_stamp":"2024-03-11T21:40:12.838597+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:39.525554+0000","last_clean_scrub_stamp":"2024-03-11T21:40:12.838597+0000","objects_scrubbed":7,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:41:21.967430+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":44,"num_read_kb":35,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,4],"acting":[2,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"10.2","version":"366'2468","reported_seq":2498,"reported_epoch":366,"state":"active+clean","last_fresh":"2024-03-11T21:41:08.307804+0000","last_change":"2024-03-11T21:40:39.233862+0000","last_active":"2024-03-11T21:41:08.307804+0000","last_peered":"2024-03-11T21:41:08.307804+0000","last_clean":"2024-03-11T21:41:08.307804+0000","last_became_active":"2024-03-11T21:40:39.233499+0000","last_became_peered":"2024-03-11T21:40:39.233499+0000","last_unstale":"2024-03-11T21:41:08.307804+0000","last_undegraded":"2024-03-11T21:41:08.307804+0000","last_fullsized":"2024-03-11T21:41:08.307804+0000","mapping_epoch":356,"log_start":"0'0","ondisk_log_start":"0'0","created":356,"last_epoch_clean":357,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:40:38.219602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:40:38.219602+0000","last_clean_scrub_stamp":"2024-03-11T21:40:38.219602+0000","objects_scrubbed":0,"log_size":2468,"log_dups_size":0,"ondisk_log_size":2468,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:41:44.996023+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":161742848,"num_objects":2468,"num_object_clones":0,"num_object_copies":7404,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2468,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2468,"num_write_kb":157952,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.e","version":"0'0","reported_seq":643,"reported_epoch":366,"state":"active+clean","last_fresh":"2024-03-11T21:41:02.593728+0000","last_change":"2024-03-11T21:36:10.769557+0000","last_active":"2024-03-11T21:41:02.593728+0000","last_peered":"2024-03-11T21:41:02.593728+0000","last_clean":"2024-03-11T21:41:02.593728+0000","last_became_active":"2024-03-11T21:25:44.590198+0000","last_became_peered":"2024-03-11T21:25:44.590198+0000","last_unstale":"2024-03-11T21:41:02.593728+0000","last_undegraded":"2024-03-11T21:41:02.593728+0000","last_fullsized":"2024-03-11T21:41:02.593728+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:36:10.769460+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:43.559452+0000","last_clean_scrub_stamp":"2024-03-11T21:36:10.769460+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":75,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.d","version":"109'9","reported_seq":849,"reported_epoch":366,"state":"active+clean","last_fresh":"2024-03-11T21:41:06.117858+0000","last_change":"2024-03-11T21:41:06.117858+0000","last_active":"2024-03-11T21:41:06.117858+0000","last_peered":"2024-03-11T21:41:06.117858+0000","last_clean":"2024-03-11T21:41:06.117858+0000","last_became_active":"2024-03-11T21:25:42.575583+0000","last_became_peered":"2024-03-11T21:25:42.575583+0000","last_unstale":"2024-03-11T21:41:06.117858+0000","last_undegraded":"2024-03-11T21:41:06.117858+0000","last_fullsized":"2024-03-11T21:41:06.117858+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"109'9","last_scrub_stamp":"2024-03-11T21:41:06.117483+0000","last_deep_scrub":"109'9","last_deep_scrub_stamp":"2024-03-11T21:28:27.082975+0000","last_clean_scrub_stamp":"2024-03-11T21:41:06.117483+0000","objects_scrubbed":1,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:42:19.490537+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,5],"acting":[2,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.a","version":"0'0","reported_seq":685,"reported_epoch":366,"state":"active+clean","last_fresh":"2024-03-11T21:41:02.612663+0000","last_change":"2024-03-11T21:41:02.132633+0000","last_active":"2024-03-11T21:41:02.612663+0000","last_peered":"2024-03-11T21:41:02.612663+0000","last_clean":"2024-03-11T21:41:02.612663+0000","last_became_active":"2024-03-11T21:36:35.066832+0000","last_became_peered":"2024-03-11T21:36:35.066832+0000","last_unstale":"2024-03-11T21:41:02.612663+0000","last_undegraded":"2024-03-11T21:41:02.612663+0000","last_fullsized":"2024-03-11T21:41:02.612663+0000","mapping_epoch":277,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":278,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:41:02.132355+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:37.512018+0000","last_clean_scrub_stamp":"2024-03-11T21:41:02.132355+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:42:27.586284+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,7],"acting":[1,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.c","version":"260'20","reported_seq":703,"reported_epoch":366,"state":"active+clean","last_fresh":"2024-03-11T21:41:02.591601+0000","last_change":"2024-03-11T21:40:02.706928+0000","last_active":"2024-03-11T21:41:02.591601+0000","last_peered":"2024-03-11T21:41:02.591601+0000","last_clean":"2024-03-11T21:41:02.591601+0000","last_became_active":"2024-03-11T21:36:35.067269+0000","last_became_peered":"2024-03-11T21:36:35.067269+0000","last_unstale":"2024-03-11T21:41:02.591601+0000","last_undegraded":"2024-03-11T21:41:02.591601+0000","last_fullsized":"2024-03-11T21:41:02.591601+0000","mapping_epoch":277,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":278,"parent":"0.0","parent_split_bits":0,"last_scrub":"260'20","last_scrub_stamp":"2024-03-11T21:40:02.706832+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:39.525554+0000","last_clean_scrub_stamp":"2024-03-11T21:40:02.706832+0000","objects_scrubbed":5,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:41:04.628603+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":25,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"10.3","version":"366'2869","reported_seq":2899,"reported_epoch":366,"state":"active+clean","last_fresh":"2024-03-11T21:41:12.428479+0000","last_change":"2024-03-11T21:40:39.541091+0000","last_active":"2024-03-11T21:41:12.428479+0000","last_peered":"2024-03-11T21:41:12.428479+0000","last_clean":"2024-03-11T21:41:12.428479+0000","last_became_active":"2024-03-11T21:40:39.540740+0000","last_became_peered":"2024-03-11T21:40:39.540740+0000","last_unstale":"2024-03-11T21:41:12.428479+0000","last_undegraded":"2024-03-11T21:41:12.428479+0000","last_fullsized":"2024-03-11T21:41:12.428479+0000","mapping_epoch":356,"log_start":"0'0","ondisk_log_start":"0'0","created":356,"last_epoch_clean":357,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:40:38.219602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:40:38.219602+0000","last_clean_scrub_stamp":"2024-03-11T21:40:38.219602+0000","objects_scrubbed":0,"log_size":2869,"log_dups_size":0,"ondisk_log_size":2869,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:41:46.923637+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":188022784,"num_objects":2869,"num_object_clones":0,"num_object_copies":8607,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2869,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2869,"num_write_kb":183616,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,3],"acting":[2,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.f","version":"0'0","reported_seq":650,"reported_epoch":366,"state":"active+clean","last_fresh":"2024-03-11T21:41:02.590776+0000","last_change":"2024-03-11T21:36:35.058630+0000","last_active":"2024-03-11T21:41:02.590776+0000","last_peered":"2024-03-11T21:41:02.590776+0000","last_clean":"2024-03-11T21:41:02.590776+0000","last_became_active":"2024-03-11T21:36:35.057800+0000","last_became_peered":"2024-03-11T21:36:35.057800+0000","last_unstale":"2024-03-11T21:41:02.590776+0000","last_undegraded":"2024-03-11T21:41:02.590776+0000","last_fullsized":"2024-03-11T21:41:02.590776+0000","mapping_epoch":277,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":278,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:35:10.608981+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:27:03.010969+0000","last_clean_scrub_stamp":"2024-03-11T21:35:10.608981+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:41:04.082967+0000","scrub_duration":94,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,4],"acting":[2,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":647,"reported_epoch":366,"state":"active+clean","last_fresh":"2024-03-11T21:41:02.590763+0000","last_change":"2024-03-11T21:36:10.608789+0000","last_active":"2024-03-11T21:41:02.590763+0000","last_peered":"2024-03-11T21:41:02.590763+0000","last_clean":"2024-03-11T21:41:02.590763+0000","last_became_active":"2024-03-11T21:25:42.579409+0000","last_became_peered":"2024-03-11T21:25:42.579409+0000","last_unstale":"2024-03-11T21:41:02.590763+0000","last_undegraded":"2024-03-11T21:41:02.590763+0000","last_fullsized":"2024-03-11T21:41:02.590763+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:36:10.608552+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:30:22.750053+0000","last_clean_scrub_stamp":"2024-03-11T21:36:10.608552+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.b","version":"0'0","reported_seq":299,"reported_epoch":366,"state":"active+clean","last_fresh":"2024-03-11T21:41:02.592176+0000","last_change":"2024-03-11T21:36:16.719432+0000","last_active":"2024-03-11T21:41:02.592176+0000","last_peered":"2024-03-11T21:41:02.592176+0000","last_clean":"2024-03-11T21:41:02.592176+0000","last_became_active":"2024-03-11T21:34:41.134987+0000","last_became_peered":"2024-03-11T21:34:41.134987+0000","last_unstale":"2024-03-11T21:41:02.592176+0000","last_undegraded":"2024-03-11T21:41:02.592176+0000","last_fullsized":"2024-03-11T21:41:02.592176+0000","mapping_epoch":226,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":227,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:36:16.719309+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:36:16.719309+0000","last_clean_scrub_stamp":"2024-03-11T21:36:16.719309+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,5],"acting":[7,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.d","version":"260'39","reported_seq":716,"reported_epoch":366,"state":"active+clean","last_fresh":"2024-03-11T21:41:11.544712+0000","last_change":"2024-03-11T21:41:11.544772+0000","last_active":"2024-03-11T21:41:11.544712+0000","last_peered":"2024-03-11T21:41:11.544712+0000","last_clean":"2024-03-11T21:41:11.544712+0000","last_became_active":"2024-03-11T21:25:40.553454+0000","last_became_peered":"2024-03-11T21:25:40.553454+0000","last_unstale":"2024-03-11T21:41:11.544712+0000","last_undegraded":"2024-03-11T21:41:11.544712+0000","last_fullsized":"2024-03-11T21:41:11.544712+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"260'39","last_scrub_stamp":"2024-03-11T21:41:11.544605+0000","last_deep_scrub":"165'25","last_deep_scrub_stamp":"2024-03-11T21:32:21.678328+0000","last_clean_scrub_stamp":"2024-03-11T21:41:11.544605+0000","objects_scrubbed":8,"log_size":39,"log_dups_size":0,"ondisk_log_size":39,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:42:25.446828+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":50,"num_read_kb":40,"num_write":32,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,1],"acting":[4,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"10.4","version":"366'2427","reported_seq":2456,"reported_epoch":366,"state":"active+clean","last_fresh":"2024-03-11T21:41:08.814990+0000","last_change":"2024-03-11T21:40:39.540876+0000","last_active":"2024-03-11T21:41:08.814990+0000","last_peered":"2024-03-11T21:41:08.814990+0000","last_clean":"2024-03-11T21:41:08.814990+0000","last_became_active":"2024-03-11T21:40:39.540563+0000","last_became_peered":"2024-03-11T21:40:39.540563+0000","last_unstale":"2024-03-11T21:41:08.814990+0000","last_undegraded":"2024-03-11T21:41:08.814990+0000","last_fullsized":"2024-03-11T21:41:08.814990+0000","mapping_epoch":356,"log_start":"0'0","ondisk_log_start":"0'0","created":356,"last_epoch_clean":357,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:40:38.219602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:40:38.219602+0000","last_clean_scrub_stamp":"2024-03-11T21:40:38.219602+0000","objects_scrubbed":0,"log_size":2427,"log_dups_size":0,"ondisk_log_size":2427,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:41:39.088274+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":159055872,"num_objects":2427,"num_object_clones":0,"num_object_copies":7281,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2427,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2427,"num_write_kb":155328,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":300,"reported_epoch":366,"state":"active+clean","last_fresh":"2024-03-11T21:41:02.590127+0000","last_change":"2024-03-11T21:36:06.735035+0000","last_active":"2024-03-11T21:41:02.590127+0000","last_peered":"2024-03-11T21:41:02.590127+0000","last_clean":"2024-03-11T21:41:02.590127+0000","last_became_active":"2024-03-11T21:34:41.145012+0000","last_became_peered":"2024-03-11T21:34:41.145012+0000","last_unstale":"2024-03-11T21:41:02.590127+0000","last_undegraded":"2024-03-11T21:41:02.590127+0000","last_fullsized":"2024-03-11T21:41:02.590127+0000","mapping_epoch":226,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":227,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:36:06.734921+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:28:37.561826+0000","last_clean_scrub_stamp":"2024-03-11T21:36:06.734921+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":23,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,0],"acting":[7,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":653,"reported_epoch":366,"state":"active+clean","last_fresh":"2024-03-11T21:41:09.181665+0000","last_change":"2024-03-11T21:41:09.181704+0000","last_active":"2024-03-11T21:41:09.181665+0000","last_peered":"2024-03-11T21:41:09.181665+0000","last_clean":"2024-03-11T21:41:09.181665+0000","last_became_active":"2024-03-11T21:25:42.574515+0000","last_became_peered":"2024-03-11T21:25:42.574515+0000","last_unstale":"2024-03-11T21:41:09.181665+0000","last_undegraded":"2024-03-11T21:41:09.181665+0000","last_fullsized":"2024-03-11T21:41:09.181665+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:41:09.181567+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:41.544550+0000","last_clean_scrub_stamp":"2024-03-11T21:41:09.181567+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:42:23.605325+0000","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,5],"acting":[2,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.c","version":"0'0","reported_seq":653,"reported_epoch":366,"state":"active+clean","last_fresh":"2024-03-11T21:41:02.590151+0000","last_change":"2024-03-11T21:36:00.849272+0000","last_active":"2024-03-11T21:41:02.590151+0000","last_peered":"2024-03-11T21:41:02.590151+0000","last_clean":"2024-03-11T21:41:02.590151+0000","last_became_active":"2024-03-11T21:25:38.533240+0000","last_became_peered":"2024-03-11T21:25:38.533240+0000","last_unstale":"2024-03-11T21:41:02.590151+0000","last_undegraded":"2024-03-11T21:41:02.590151+0000","last_fullsized":"2024-03-11T21:41:02.590151+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:36:00.849153+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:29:30.279910+0000","last_clean_scrub_stamp":"2024-03-11T21:36:00.849153+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:41:04.083023+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,0],"acting":[2,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.a","version":"260'44","reported_seq":792,"reported_epoch":366,"state":"active+clean","last_fresh":"2024-03-11T21:41:02.586482+0000","last_change":"2024-03-11T21:40:02.183543+0000","last_active":"2024-03-11T21:41:02.586482+0000","last_peered":"2024-03-11T21:41:02.586482+0000","last_clean":"2024-03-11T21:41:02.586482+0000","last_became_active":"2024-03-11T21:34:41.148863+0000","last_became_peered":"2024-03-11T21:34:41.148863+0000","last_unstale":"2024-03-11T21:41:02.586482+0000","last_undegraded":"2024-03-11T21:41:02.586482+0000","last_fullsized":"2024-03-11T21:41:02.586482+0000","mapping_epoch":226,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":227,"parent":"0.0","parent_split_bits":0,"last_scrub":"260'44","last_scrub_stamp":"2024-03-11T21:40:02.183417+0000","last_deep_scrub":"109'28","last_deep_scrub_stamp":"2024-03-11T21:28:24.016389+0000","last_clean_scrub_stamp":"2024-03-11T21:40:02.183417+0000","objects_scrubbed":9,"log_size":44,"log_dups_size":0,"ondisk_log_size":44,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:41:31.089360+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":56,"num_read_kb":45,"num_write":36,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,7],"acting":[6,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"10.5","version":"366'2946","reported_seq":2976,"reported_epoch":366,"state":"active+clean","last_fresh":"2024-03-11T21:41:12.440690+0000","last_change":"2024-03-11T21:40:39.541067+0000","last_active":"2024-03-11T21:41:12.440690+0000","last_peered":"2024-03-11T21:41:12.440690+0000","last_clean":"2024-03-11T21:41:12.440690+0000","last_became_active":"2024-03-11T21:40:39.540135+0000","last_became_peered":"2024-03-11T21:40:39.540135+0000","last_unstale":"2024-03-11T21:41:12.440690+0000","last_undegraded":"2024-03-11T21:41:12.440690+0000","last_fullsized":"2024-03-11T21:41:12.440690+0000","mapping_epoch":356,"log_start":"0'0","ondisk_log_start":"0'0","created":356,"last_epoch_clean":357,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:40:38.219602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:40:38.219602+0000","last_clean_scrub_stamp":"2024-03-11T21:40:38.219602+0000","objects_scrubbed":0,"log_size":2946,"log_dups_size":0,"ondisk_log_size":2946,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:41:51.853957+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":193069056,"num_objects":2946,"num_object_clones":0,"num_object_copies":8838,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2946,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2946,"num_write_kb":188544,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3],"acting":[2,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":656,"reported_epoch":366,"state":"active+clean","last_fresh":"2024-03-11T21:41:02.588953+0000","last_change":"2024-03-11T21:35:37.687281+0000","last_active":"2024-03-11T21:41:02.588953+0000","last_peered":"2024-03-11T21:41:02.588953+0000","last_clean":"2024-03-11T21:41:02.588953+0000","last_became_active":"2024-03-11T21:34:41.154507+0000","last_became_peered":"2024-03-11T21:34:41.154507+0000","last_unstale":"2024-03-11T21:41:02.588953+0000","last_undegraded":"2024-03-11T21:41:02.588953+0000","last_fullsized":"2024-03-11T21:41:02.588953+0000","mapping_epoch":226,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":227,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:35:37.687187+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:27:11.361913+0000","last_clean_scrub_stamp":"2024-03-11T21:35:37.687187+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":39,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":655,"reported_epoch":366,"state":"active+clean","last_fresh":"2024-03-11T21:41:02.590498+0000","last_change":"2024-03-11T21:40:28.015350+0000","last_active":"2024-03-11T21:41:02.590498+0000","last_peered":"2024-03-11T21:41:02.590498+0000","last_clean":"2024-03-11T21:41:02.590498+0000","last_became_active":"2024-03-11T21:36:35.059192+0000","last_became_peered":"2024-03-11T21:36:35.059192+0000","last_unstale":"2024-03-11T21:41:02.590498+0000","last_undegraded":"2024-03-11T21:41:02.590498+0000","last_fullsized":"2024-03-11T21:41:02.590498+0000","mapping_epoch":277,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":278,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:40:28.015231+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:41.544550+0000","last_clean_scrub_stamp":"2024-03-11T21:40:28.015231+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:41:54.304353+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,3],"acting":[2,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.d","version":"0'0","reported_seq":664,"reported_epoch":366,"state":"active+clean","last_fresh":"2024-03-11T21:41:02.590268+0000","last_change":"2024-03-11T21:40:31.873781+0000","last_active":"2024-03-11T21:41:02.590268+0000","last_peered":"2024-03-11T21:41:02.590268+0000","last_clean":"2024-03-11T21:41:02.590268+0000","last_became_active":"2024-03-11T21:36:35.058296+0000","last_became_peered":"2024-03-11T21:36:35.058296+0000","last_unstale":"2024-03-11T21:41:02.590268+0000","last_undegraded":"2024-03-11T21:41:02.590268+0000","last_fullsized":"2024-03-11T21:41:02.590268+0000","mapping_epoch":277,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":278,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:40:31.873695+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:37.512018+0000","last_clean_scrub_stamp":"2024-03-11T21:40:31.873695+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:41:53.890729+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,3],"acting":[1,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.b","version":"260'15","reported_seq":676,"reported_epoch":366,"state":"active+clean","last_fresh":"2024-03-11T21:41:02.588904+0000","last_change":"2024-03-11T21:35:27.760318+0000","last_active":"2024-03-11T21:41:02.588904+0000","last_peered":"2024-03-11T21:41:02.588904+0000","last_clean":"2024-03-11T21:41:02.588904+0000","last_became_active":"2024-03-11T21:25:40.549612+0000","last_became_peered":"2024-03-11T21:25:40.549612+0000","last_unstale":"2024-03-11T21:41:02.588904+0000","last_undegraded":"2024-03-11T21:41:02.588904+0000","last_fullsized":"2024-03-11T21:41:02.588904+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'9","last_scrub_stamp":"2024-03-11T21:35:27.760208+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:39.525554+0000","last_clean_scrub_stamp":"2024-03-11T21:35:27.760208+0000","objects_scrubbed":4,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.6","version":"366'2862","reported_seq":2892,"reported_epoch":366,"state":"active+clean","last_fresh":"2024-03-11T21:41:12.547609+0000","last_change":"2024-03-11T21:40:39.243542+0000","last_active":"2024-03-11T21:41:12.547609+0000","last_peered":"2024-03-11T21:41:12.547609+0000","last_clean":"2024-03-11T21:41:12.547609+0000","last_became_active":"2024-03-11T21:40:39.243049+0000","last_became_peered":"2024-03-11T21:40:39.243049+0000","last_unstale":"2024-03-11T21:41:12.547609+0000","last_undegraded":"2024-03-11T21:41:12.547609+0000","last_fullsized":"2024-03-11T21:41:12.547609+0000","mapping_epoch":356,"log_start":"0'0","ondisk_log_start":"0'0","created":356,"last_epoch_clean":357,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:40:38.219602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:40:38.219602+0000","last_clean_scrub_stamp":"2024-03-11T21:40:38.219602+0000","objects_scrubbed":0,"log_size":2862,"log_dups_size":0,"ondisk_log_size":2862,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:41:58.698763+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":187564032,"num_objects":2862,"num_object_clones":0,"num_object_copies":8586,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2862,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2862,"num_write_kb":183168,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,4],"acting":[5,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":557,"reported_epoch":366,"state":"active+clean","last_fresh":"2024-03-11T21:41:02.592899+0000","last_change":"2024-03-11T21:36:35.057118+0000","last_active":"2024-03-11T21:41:02.592899+0000","last_peered":"2024-03-11T21:41:02.592899+0000","last_clean":"2024-03-11T21:41:02.592899+0000","last_became_active":"2024-03-11T21:36:35.056518+0000","last_became_peered":"2024-03-11T21:36:35.056518+0000","last_unstale":"2024-03-11T21:41:02.592899+0000","last_undegraded":"2024-03-11T21:41:02.592899+0000","last_fullsized":"2024-03-11T21:41:02.592899+0000","mapping_epoch":277,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":278,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:35:24.514584+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:43.559452+0000","last_clean_scrub_stamp":"2024-03-11T21:35:24.514584+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,3],"acting":[4,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.e","version":"0'0","reported_seq":682,"reported_epoch":366,"state":"active+clean","last_fresh":"2024-03-11T21:41:02.590574+0000","last_change":"2024-03-11T21:40:11.026129+0000","last_active":"2024-03-11T21:41:02.590574+0000","last_peered":"2024-03-11T21:41:02.590574+0000","last_clean":"2024-03-11T21:41:02.590574+0000","last_became_active":"2024-03-11T21:36:35.056777+0000","last_became_peered":"2024-03-11T21:36:35.056777+0000","last_unstale":"2024-03-11T21:41:02.590574+0000","last_undegraded":"2024-03-11T21:41:02.590574+0000","last_fullsized":"2024-03-11T21:41:02.590574+0000","mapping_epoch":277,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":278,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:40:11.025986+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:37.512018+0000","last_clean_scrub_stamp":"2024-03-11T21:40:11.025986+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:41:21.568389+0000","scrub_duration":114,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,7],"acting":[2,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.8","version":"260'34","reported_seq":753,"reported_epoch":366,"state":"active+clean","last_fresh":"2024-03-11T21:41:06.881502+0000","last_change":"2024-03-11T21:41:06.881563+0000","last_active":"2024-03-11T21:41:06.881502+0000","last_peered":"2024-03-11T21:41:06.881502+0000","last_clean":"2024-03-11T21:41:06.881502+0000","last_became_active":"2024-03-11T21:34:41.146631+0000","last_became_peered":"2024-03-11T21:34:41.146631+0000","last_unstale":"2024-03-11T21:41:06.881502+0000","last_undegraded":"2024-03-11T21:41:06.881502+0000","last_fullsized":"2024-03-11T21:41:06.881502+0000","mapping_epoch":226,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":227,"parent":"0.0","parent_split_bits":0,"last_scrub":"260'34","last_scrub_stamp":"2024-03-11T21:41:06.881414+0000","last_deep_scrub":"109'22","last_deep_scrub_stamp":"2024-03-11T21:32:07.573197+0000","last_clean_scrub_stamp":"2024-03-11T21:41:06.881414+0000","objects_scrubbed":7,"log_size":34,"log_dups_size":0,"ondisk_log_size":34,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:42:16.451438+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":44,"num_read_kb":35,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.9","version":"229'11","reported_seq":382,"reported_epoch":366,"state":"active+clean","last_fresh":"2024-03-11T21:41:08.781817+0000","last_change":"2024-03-11T21:36:18.726876+0000","last_active":"2024-03-11T21:41:08.781817+0000","last_peered":"2024-03-11T21:41:08.781817+0000","last_clean":"2024-03-11T21:41:08.781817+0000","last_became_active":"2024-03-11T21:34:41.156892+0000","last_became_peered":"2024-03-11T21:34:41.156892+0000","last_unstale":"2024-03-11T21:41:08.781817+0000","last_undegraded":"2024-03-11T21:41:08.781817+0000","last_fullsized":"2024-03-11T21:41:08.781817+0000","mapping_epoch":226,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":227,"parent":"0.0","parent_split_bits":0,"last_scrub":"229'11","last_scrub_stamp":"2024-03-11T21:36:18.726763+0000","last_deep_scrub":"94'9","last_deep_scrub_stamp":"2024-03-11T21:26:53.422552+0000","last_clean_scrub_stamp":"2024-03-11T21:36:18.726763+0000","objects_scrubbed":1,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":10,"num_pg":16,"stat_sum":{"num_bytes":2927230976,"num_objects":44666,"num_object_clones":0,"num_object_copies":133998,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":44666,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44666,"num_write_kb":2858624,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8565751808,"data_stored":8565751808,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":44666,"ondisk_log_size":44666,"up":48,"acting":48,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":1531,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":17,"num_read_kb":17,"num_write":10,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":1308,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":4593,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":6,"ondisk_log_size":6,"up":96,"acting":96,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":198,"num_read_kb":294,"num_write":179,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":2951840,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":2375680,"data_stored":2361472,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":128,"ondisk_log_size":128,"up":3,"acting":3,"num_store_stats":4},{"poolid":3,"num_pg":3,"stat_sum":{"num_bytes":408,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":922,"num_read_kb":917,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1224,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":8,"ondisk_log_size":8,"up":9,"acting":9,"num_store_stats":7},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":3702,"num_objects":178,"num_object_clones":0,"num_object_copies":534,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":178,"num_whiteouts":0,"num_read":1120,"num_read_kb":877,"num_write":703,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":186,"num_bytes_recovered":2200,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":417792,"data_stored":11106,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":770,"ondisk_log_size":770,"up":96,"acting":96,"num_store_stats":8},{"poolid":5,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":78,"ondisk_log_size":78,"up":96,"acting":96,"num_store_stats":8},{"poolid":6,"num_pg":32,"stat_sum":{"num_bytes":383,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":370,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1149,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":96,"acting":96,"num_store_stats":8}],"osd_stats":[{"osd":7,"up_from":60,"seq":257698037950,"num_pgs":60,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1238864,"kb_used_data":734280,"kb_used_omap":1,"kb_used_meta":504574,"kb_avail":92506288,"statfs":{"total":95995035648,"available":94726438912,"internally_reserved":0,"allocated":751902720,"data_stored":750874995,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":516684234},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":50,"seq":214748364996,"num_pgs":50,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1690704,"kb_used_data":1262472,"kb_used_omap":1,"kb_used_meta":428222,"kb_avail":92054448,"statfs":{"total":95995035648,"available":94263754752,"internally_reserved":0,"allocated":1292771328,"data_stored":1291744332,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":438499782},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834575069,"num_pgs":54,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1063656,"kb_used_data":472992,"kb_used_omap":1,"kb_used_meta":590654,"kb_avail":92681496,"statfs":{"total":95995035648,"available":94905851904,"internally_reserved":0,"allocated":484343808,"data_stored":483291690,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":604830149},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738594,"num_pgs":60,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2130716,"kb_used_data":1544212,"kb_used_omap":1,"kb_used_meta":586494,"kb_avail":91614436,"statfs":{"total":95995035648,"available":93813182464,"internally_reserved":0,"allocated":1581273088,"data_stored":1580233838,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":600570309},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411544,"num_pgs":46,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1570568,"kb_used_data":1102080,"kb_used_omap":1,"kb_used_meta":468478,"kb_avail":92174584,"statfs":{"total":95995035648,"available":94386774016,"internally_reserved":0,"allocated":1128529920,"data_stored":1127510340,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":479721925},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":277,"seq":1189705941048,"num_pgs":59,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1084588,"kb_used_data":776036,"kb_used_omap":4,"kb_used_meta":308539,"kb_avail":92660564,"statfs":{"total":95995035648,"available":94884417536,"internally_reserved":0,"allocated":794660864,"data_stored":793605094,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4442,"internal_metadata":315944614},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":32,"seq":137438953677,"num_pgs":61,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1864732,"kb_used_data":1414292,"kb_used_omap":1,"kb_used_meta":450430,"kb_avail":91880420,"statfs":{"total":95995035648,"available":94085550080,"internally_reserved":0,"allocated":1448235008,"data_stored":1447194796,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":461240773},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":41,"seq":176093659336,"num_pgs":54,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1451528,"kb_used_data":896960,"kb_used_omap":1,"kb_used_meta":554558,"kb_avail":92293624,"statfs":{"total":95995035648,"available":94508670976,"internally_reserved":0,"allocated":918487040,"data_stored":917467075,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":567867845},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1049,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":46,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":436,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":957,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":574,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":436,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1003,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":92,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":1430,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":77824,"data_stored":2008,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":1018,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":1898,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":1474,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":990,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":1034,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":1254,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":370,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":383,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":370,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":1624768512,"data_stored":1624768512,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":484114432,"data_stored":484114432,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":1138425856,"data_stored":1138425856,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":817954816,"data_stored":817954816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":1489698816,"data_stored":1489698816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":943718400,"data_stored":943718400,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":1315504128,"data_stored":1315504128,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":751566848,"data_stored":751566848,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-11T21:41:16.064 INFO:tasks.cephadm.ceph_manager.ceph:increase pool size by 10 2024-03-11T21:41:16.065 DEBUG:teuthology.orchestra.run.smithi092:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid 00db0016-dfed-11ee-95c8-87774f69a715 -- ceph osd pool set unique_pool_3 pg_num 26 2024-03-11T21:41:16.066 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:41:16.080 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:41:15 smithi097 ceph-mon[33317]: 4.e scrub starts 2024-03-11T21:41:16.080 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:41:15 smithi097 ceph-mon[33317]: 4.e scrub ok 2024-03-11T21:41:16.080 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:41:15 smithi097 ceph-mon[33317]: 2.14 scrub starts 2024-03-11T21:41:16.080 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:41:15 smithi097 ceph-mon[33317]: 2.14 scrub ok 2024-03-11T21:41:16.081 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:41:15 smithi097 ceph-mon[33317]: from='client.15759 v1:172.21.15.92:0/2877137765' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-11T21:41:16.103 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:41:16.104 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:41:16.104 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:41:16.205 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:41:16.256 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:41:16.257 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:41:16.257 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:41:16.358 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:41:16.406 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:41:16.407 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:41:16.407 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:41:16.454 INFO:teuthology.orchestra.run.smithi092.stdout:ERROR: (22) Invalid argument 2024-03-11T21:41:16.454 INFO:teuthology.orchestra.run.smithi092.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-11T21:41:16.509 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:41:16.542 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:41:16.542 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:41:16.543 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:41:16.643 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:41:16.676 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:41:16.677 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:41:16.677 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:41:16.719 INFO:teuthology.orchestra.run.smithi092.stdout:osd.0: {} 2024-03-11T21:41:16.757 INFO:teuthology.orchestra.run.smithi092.stdout:osd.1: {} 2024-03-11T21:41:16.778 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:41:16.795 INFO:teuthology.orchestra.run.smithi092.stdout:osd.2: {} 2024-03-11T21:41:16.814 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:41:16.814 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:41:16.815 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:41:16.832 INFO:tasks.radosbench.radosbench.0.smithi092.stdout: 24 15 53239 53224 138.589 94.8125 0.0167374 0.00721017 2024-03-11T21:41:16.835 INFO:teuthology.orchestra.run.smithi092.stdout:osd.3: {} 2024-03-11T21:41:16.865 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:41:16 smithi092 ceph-mon[29572]: purged_snaps scrub starts 2024-03-11T21:41:16.865 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:41:16 smithi092 ceph-mon[29572]: purged_snaps scrub ok 2024-03-11T21:41:16.865 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:41:16 smithi092 ceph-mon[29572]: 5.8 scrub starts 2024-03-11T21:41:16.865 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:41:16 smithi092 ceph-mon[29572]: 5.8 scrub ok 2024-03-11T21:41:16.865 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:41:16 smithi092 ceph-mon[29572]: pgmap v729: 148 pgs: 148 active+clean; 2.8 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 1.2 KiB/s rd, 108 MiB/s wr, 1.73k op/s 2024-03-11T21:41:16.865 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:41:16 smithi092 ceph-mon[29572]: osdmap e367: 8 total, 8 up, 8 in 2024-03-11T21:41:16.865 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:41:16 smithi092 ceph-mon[29572]: 6.d scrub starts 2024-03-11T21:41:16.865 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:41:16 smithi092 ceph-mon[29572]: 6.d scrub ok 2024-03-11T21:41:16.865 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:41:16 smithi092 ceph-mon[29572]: 6.12 scrub starts 2024-03-11T21:41:16.865 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:41:16 smithi092 ceph-mon[29572]: 6.12 scrub ok 2024-03-11T21:41:16.866 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:41:16 smithi092 ceph-mon[37563]: purged_snaps scrub starts 2024-03-11T21:41:16.866 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:41:16 smithi092 ceph-mon[37563]: purged_snaps scrub ok 2024-03-11T21:41:16.866 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:41:16 smithi092 ceph-mon[37563]: 5.8 scrub starts 2024-03-11T21:41:16.866 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:41:16 smithi092 ceph-mon[37563]: 5.8 scrub ok 2024-03-11T21:41:16.866 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:41:16 smithi092 ceph-mon[37563]: pgmap v729: 148 pgs: 148 active+clean; 2.8 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 1.2 KiB/s rd, 108 MiB/s wr, 1.73k op/s 2024-03-11T21:41:16.866 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:41:16 smithi092 ceph-mon[37563]: osdmap e367: 8 total, 8 up, 8 in 2024-03-11T21:41:16.866 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:41:16 smithi092 ceph-mon[37563]: 6.d scrub starts 2024-03-11T21:41:16.866 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:41:16 smithi092 ceph-mon[37563]: 6.d scrub ok 2024-03-11T21:41:16.866 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:41:16 smithi092 ceph-mon[37563]: 6.12 scrub starts 2024-03-11T21:41:16.866 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:41:16 smithi092 ceph-mon[37563]: 6.12 scrub ok 2024-03-11T21:41:16.876 INFO:teuthology.orchestra.run.smithi092.stdout:osd.4: {} 2024-03-11T21:41:16.916 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:41:16.917 INFO:teuthology.orchestra.run.smithi092.stdout:osd.5: {} 2024-03-11T21:41:16.958 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:41:16.959 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:41:16.959 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:41:16.965 INFO:teuthology.orchestra.run.smithi092.stdout:osd.6: {} 2024-03-11T21:41:17.003 INFO:teuthology.orchestra.run.smithi092.stdout:osd.7: {} 2024-03-11T21:41:17.011 INFO:teuthology.orchestra.run.smithi092.stderr:osd.0: osd_enable_op_tracker = '' 2024-03-11T21:41:17.012 INFO:teuthology.orchestra.run.smithi092.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-11T21:41:17.012 INFO:teuthology.orchestra.run.smithi092.stderr:osd.2: osd_enable_op_tracker = '' 2024-03-11T21:41:17.012 INFO:teuthology.orchestra.run.smithi092.stderr:osd.3: osd_enable_op_tracker = '' 2024-03-11T21:41:17.012 INFO:teuthology.orchestra.run.smithi092.stderr:osd.4: osd_enable_op_tracker = '' 2024-03-11T21:41:17.012 INFO:teuthology.orchestra.run.smithi092.stderr:osd.5: osd_enable_op_tracker = '' 2024-03-11T21:41:17.012 INFO:teuthology.orchestra.run.smithi092.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-11T21:41:17.012 INFO:teuthology.orchestra.run.smithi092.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-11T21:41:17.060 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:41:17.080 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:41:16 smithi097 ceph-mon[33317]: purged_snaps scrub starts 2024-03-11T21:41:17.081 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:41:16 smithi097 ceph-mon[33317]: purged_snaps scrub ok 2024-03-11T21:41:17.081 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:41:16 smithi097 ceph-mon[33317]: 5.8 scrub starts 2024-03-11T21:41:17.081 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:41:16 smithi097 ceph-mon[33317]: 5.8 scrub ok 2024-03-11T21:41:17.081 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:41:16 smithi097 ceph-mon[33317]: pgmap v729: 148 pgs: 148 active+clean; 2.8 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 1.2 KiB/s rd, 108 MiB/s wr, 1.73k op/s 2024-03-11T21:41:17.081 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:41:16 smithi097 ceph-mon[33317]: osdmap e367: 8 total, 8 up, 8 in 2024-03-11T21:41:17.081 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:41:16 smithi097 ceph-mon[33317]: 6.d scrub starts 2024-03-11T21:41:17.081 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:41:16 smithi097 ceph-mon[33317]: 6.d scrub ok 2024-03-11T21:41:17.081 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:41:16 smithi097 ceph-mon[33317]: 6.12 scrub starts 2024-03-11T21:41:17.082 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:41:16 smithi097 ceph-mon[33317]: 6.12 scrub ok 2024-03-11T21:41:17.094 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:41:17.095 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:41:17.095 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:41:17.196 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:41:17.232 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:41:17.233 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:41:17.233 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:41:17.335 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:41:17.376 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:41:17.377 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:41:17.377 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:41:17.478 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:41:17.515 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:41:17.516 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:41:17.516 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:41:17.617 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:41:17.657 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:41:17.658 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:41:17.658 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:41:17.759 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:41:17.801 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:41:17.802 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:41:17.802 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:41:17.832 INFO:tasks.radosbench.radosbench.0.smithi092.stdout: 25 16 54825 54809 137.008 99.0625 0.0076288 0.00729409 2024-03-11T21:41:17.864 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:41:17 smithi092 ceph-mon[29572]: 6.1d scrub starts 2024-03-11T21:41:17.865 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:41:17 smithi092 ceph-mon[29572]: 6.1d scrub ok 2024-03-11T21:41:17.865 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:41:17 smithi092 ceph-mon[29572]: 5.e scrub starts 2024-03-11T21:41:17.865 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:41:17 smithi092 ceph-mon[29572]: 5.e scrub ok 2024-03-11T21:41:17.865 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:41:17 smithi092 ceph-mon[29572]: from='client.? v1:172.21.15.92:0/3525214351' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-11T21:41:17.865 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:41:17 smithi092 ceph-mon[29572]: 4.10 scrub starts 2024-03-11T21:41:17.865 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:41:17 smithi092 ceph-mon[29572]: 4.10 scrub ok 2024-03-11T21:41:17.865 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:41:17 smithi092 ceph-mon[37563]: 6.1d scrub starts 2024-03-11T21:41:17.865 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:41:17 smithi092 ceph-mon[37563]: 6.1d scrub ok 2024-03-11T21:41:17.865 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:41:17 smithi092 ceph-mon[37563]: 5.e scrub starts 2024-03-11T21:41:17.866 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:41:17 smithi092 ceph-mon[37563]: 5.e scrub ok 2024-03-11T21:41:17.866 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:41:17 smithi092 ceph-mon[37563]: from='client.? v1:172.21.15.92:0/3525214351' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-11T21:41:17.866 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:41:17 smithi092 ceph-mon[37563]: 4.10 scrub starts 2024-03-11T21:41:17.866 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:41:17 smithi092 ceph-mon[37563]: 4.10 scrub ok 2024-03-11T21:41:17.903 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:41:17.937 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:41:17.938 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:41:17.938 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:41:18.039 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:41:18.078 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:41:18.079 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:41:18.079 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:41:18.080 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:41:17 smithi097 ceph-mon[33317]: 6.1d scrub starts 2024-03-11T21:41:18.080 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:41:17 smithi097 ceph-mon[33317]: 6.1d scrub ok 2024-03-11T21:41:18.080 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:41:17 smithi097 ceph-mon[33317]: 5.e scrub starts 2024-03-11T21:41:18.080 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:41:17 smithi097 ceph-mon[33317]: 5.e scrub ok 2024-03-11T21:41:18.081 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:41:17 smithi097 ceph-mon[33317]: from='client.? v1:172.21.15.92:0/3525214351' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-11T21:41:18.081 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:41:17 smithi097 ceph-mon[33317]: 4.10 scrub starts 2024-03-11T21:41:18.081 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:41:17 smithi097 ceph-mon[33317]: 4.10 scrub ok 2024-03-11T21:41:18.180 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:41:18.219 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:41:18.220 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:41:18.220 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:41:18.321 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:41:18.356 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:41:18.357 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:41:18.357 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:41:18.364 INFO:journalctl@ceph.mgr.y.smithi092.stdout:Mar 11 21:41:17 smithi092 ceph-00db0016-dfed-11ee-95c8-87774f69a715-mgr-y[29801]: ::ffff:172.21.15.97 - - [11/Mar/2024:21:41:17] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-11T21:41:18.458 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:41:18.493 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:41:18.494 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:41:18.494 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:41:18.595 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:41:18.629 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:41:18.630 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:41:18.630 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:41:18.731 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:41:18.774 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:41:18.774 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:41:18.775 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:41:18.832 INFO:tasks.radosbench.radosbench.0.smithi092.stdout: 26 16 56398 56382 135.519 98.3125 0.00957915 0.007374 2024-03-11T21:41:18.875 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:41:18.921 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:41:18.921 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:41:18.922 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:41:19.022 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:41:19.066 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:41:19.066 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:41:19.066 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:41:19.069 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:41:18 smithi092 ceph-mon[29572]: 6.f scrub starts 2024-03-11T21:41:19.069 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:41:18 smithi092 ceph-mon[29572]: pgmap v731: 148 pgs: 148 active+clean; 2.9 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 1023 B/s rd, 116 MiB/s wr, 1.85k op/s 2024-03-11T21:41:19.069 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:41:18 smithi092 ceph-mon[29572]: 6.f scrub ok 2024-03-11T21:41:19.069 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:41:18 smithi092 ceph-mon[29572]: 4.18 scrub starts 2024-03-11T21:41:19.069 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:41:18 smithi092 ceph-mon[29572]: 4.18 scrub ok 2024-03-11T21:41:19.069 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:41:18 smithi092 ceph-mon[29572]: osdmap e368: 8 total, 8 up, 8 in 2024-03-11T21:41:19.070 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:41:18 smithi092 ceph-mon[37563]: 6.f scrub starts 2024-03-11T21:41:19.070 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:41:18 smithi092 ceph-mon[37563]: pgmap v731: 148 pgs: 148 active+clean; 2.9 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 1023 B/s rd, 116 MiB/s wr, 1.85k op/s 2024-03-11T21:41:19.070 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:41:18 smithi092 ceph-mon[37563]: 6.f scrub ok 2024-03-11T21:41:19.070 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:41:18 smithi092 ceph-mon[37563]: 4.18 scrub starts 2024-03-11T21:41:19.070 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:41:18 smithi092 ceph-mon[37563]: 4.18 scrub ok 2024-03-11T21:41:19.070 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:41:18 smithi092 ceph-mon[37563]: osdmap e368: 8 total, 8 up, 8 in 2024-03-11T21:41:19.080 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:41:18 smithi097 ceph-mon[33317]: 6.f scrub starts 2024-03-11T21:41:19.080 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:41:18 smithi097 ceph-mon[33317]: pgmap v731: 148 pgs: 148 active+clean; 2.9 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 1023 B/s rd, 116 MiB/s wr, 1.85k op/s 2024-03-11T21:41:19.080 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:41:18 smithi097 ceph-mon[33317]: 6.f scrub ok 2024-03-11T21:41:19.080 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:41:18 smithi097 ceph-mon[33317]: 4.18 scrub starts 2024-03-11T21:41:19.080 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:41:18 smithi097 ceph-mon[33317]: 4.18 scrub ok 2024-03-11T21:41:19.081 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:41:18 smithi097 ceph-mon[33317]: osdmap e368: 8 total, 8 up, 8 in 2024-03-11T21:41:19.168 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:41:19.207 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:41:19.208 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:41:19.208 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:41:19.309 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:41:19.352 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:41:19.353 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:41:19.353 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:41:19.450 DEBUG:teuthology.orchestra.run.smithi092:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid 00db0016-dfed-11ee-95c8-87774f69a715 -- ceph daemon osd.2 dump_blocked_ops 2024-03-11T21:41:19.454 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:41:19.463 INFO:teuthology.orchestra.run.smithi092.stderr:Inferring config /var/lib/ceph/00db0016-dfed-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T21:41:19.522 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:41:19.522 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:41:19.523 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:41:19.623 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:41:19.667 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:41:19.667 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:41:19.667 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:41:19.723 INFO:teuthology.orchestra.run.smithi092.stderr:nodeep-scrub is unset 2024-03-11T21:41:19.769 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:41:19.813 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:41:19.814 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:41:19.814 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:41:19.832 INFO:tasks.radosbench.radosbench.0.smithi092.stdout: 27 16 57829 57813 133.811 89.4375 0.0105867 0.00746803 2024-03-11T21:41:19.915 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:41:19.957 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:41:19.958 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:41:19.958 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:41:19.967 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:41:19 smithi092 ceph-mon[29572]: 4.12 scrub starts 2024-03-11T21:41:19.967 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:41:19 smithi092 ceph-mon[29572]: 4.12 scrub ok 2024-03-11T21:41:19.967 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:41:19 smithi092 ceph-mon[29572]: 4.15 scrub starts 2024-03-11T21:41:19.967 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:41:19 smithi092 ceph-mon[29572]: 4.15 scrub ok 2024-03-11T21:41:19.967 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:41:19 smithi092 ceph-mon[29572]: 2.5 scrub starts 2024-03-11T21:41:19.967 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:41:19 smithi092 ceph-mon[29572]: 2.5 scrub ok 2024-03-11T21:41:19.967 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:41:19 smithi092 ceph-mon[29572]: from='client.? v1:172.21.15.92:0/13770655' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-11T21:41:19.967 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:41:19 smithi092 ceph-mon[29572]: from='client.35624 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-11T21:41:19.967 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:41:19 smithi092 ceph-mon[37563]: 4.12 scrub starts 2024-03-11T21:41:19.968 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:41:19 smithi092 ceph-mon[37563]: 4.12 scrub ok 2024-03-11T21:41:19.968 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:41:19 smithi092 ceph-mon[37563]: 4.15 scrub starts 2024-03-11T21:41:19.968 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:41:19 smithi092 ceph-mon[37563]: 4.15 scrub ok 2024-03-11T21:41:19.968 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:41:19 smithi092 ceph-mon[37563]: 2.5 scrub starts 2024-03-11T21:41:19.968 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:41:19 smithi092 ceph-mon[37563]: 2.5 scrub ok 2024-03-11T21:41:19.968 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:41:19 smithi092 ceph-mon[37563]: from='client.? v1:172.21.15.92:0/13770655' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-11T21:41:19.968 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:41:19 smithi092 ceph-mon[37563]: from='client.35624 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-11T21:41:20.059 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:41:20.080 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:41:19 smithi097 ceph-mon[33317]: 4.12 scrub starts 2024-03-11T21:41:20.080 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:41:19 smithi097 ceph-mon[33317]: 4.12 scrub ok 2024-03-11T21:41:20.081 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:41:19 smithi097 ceph-mon[33317]: 4.15 scrub starts 2024-03-11T21:41:20.081 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:41:19 smithi097 ceph-mon[33317]: 4.15 scrub ok 2024-03-11T21:41:20.081 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:41:19 smithi097 ceph-mon[33317]: 2.5 scrub starts 2024-03-11T21:41:20.081 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:41:19 smithi097 ceph-mon[33317]: 2.5 scrub ok 2024-03-11T21:41:20.081 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:41:19 smithi097 ceph-mon[33317]: from='client.? v1:172.21.15.92:0/13770655' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-11T21:41:20.081 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:41:19 smithi097 ceph-mon[33317]: from='client.35624 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-11T21:41:20.095 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:41:20.095 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:41:20.095 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:41:20.196 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:41:20.238 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:41:20.238 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:41:20.238 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:41:20.339 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:41:20.376 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:41:20.376 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:41:20.377 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:41:20.478 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:41:20.512 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:41:20.513 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:41:20.513 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:41:20.614 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:41:20.652 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:41:20.653 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:41:20.653 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:41:20.742 DEBUG:teuthology.orchestra.run.smithi092:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid 00db0016-dfed-11ee-95c8-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-03-11T21:41:20.754 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:41:20.790 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:41:20.791 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:41:20.791 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:41:20.832 INFO:tasks.radosbench.radosbench.0.smithi092.stdout: 28 16 59439 59423 132.626 100.625 0.0100455 0.00753561 2024-03-11T21:41:20.892 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:41:20.925 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:41:20 smithi092 ceph-mon[29572]: pgmap v733: 148 pgs: 148 active+clean; 3.3 GiB data, 14 GiB used, 702 GiB / 715 GiB avail; 1.2 KiB/s rd, 135 MiB/s wr, 2.16k op/s 2024-03-11T21:41:20.925 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:41:20 smithi092 ceph-mon[29572]: 5.18 scrub starts 2024-03-11T21:41:20.925 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:41:20 smithi092 ceph-mon[29572]: 5.18 scrub ok 2024-03-11T21:41:20.925 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:41:20 smithi092 ceph-mon[29572]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-03-11T21:41:20.926 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:41:20 smithi092 ceph-mon[29572]: Cluster is now healthy 2024-03-11T21:41:20.926 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:41:20 smithi092 ceph-mon[29572]: from='client.35624 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-03-11T21:41:20.926 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:41:20 smithi092 ceph-mon[29572]: osdmap e369: 8 total, 8 up, 8 in 2024-03-11T21:41:20.926 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:41:20 smithi092 ceph-mon[37563]: pgmap v733: 148 pgs: 148 active+clean; 3.3 GiB data, 14 GiB used, 702 GiB / 715 GiB avail; 1.2 KiB/s rd, 135 MiB/s wr, 2.16k op/s 2024-03-11T21:41:20.926 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:41:20 smithi092 ceph-mon[37563]: 5.18 scrub starts 2024-03-11T21:41:20.926 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:41:20 smithi092 ceph-mon[37563]: 5.18 scrub ok 2024-03-11T21:41:20.926 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:41:20 smithi092 ceph-mon[37563]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-03-11T21:41:20.926 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:41:20 smithi092 ceph-mon[37563]: Cluster is now healthy 2024-03-11T21:41:20.926 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:41:20 smithi092 ceph-mon[37563]: from='client.35624 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-03-11T21:41:20.927 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:41:20 smithi092 ceph-mon[37563]: osdmap e369: 8 total, 8 up, 8 in 2024-03-11T21:41:20.928 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:41:20.928 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:41:20.929 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:41:21.030 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:41:21.066 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:41:21.067 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:41:21.067 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:41:21.080 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:41:20 smithi097 ceph-mon[33317]: pgmap v733: 148 pgs: 148 active+clean; 3.3 GiB data, 14 GiB used, 702 GiB / 715 GiB avail; 1.2 KiB/s rd, 135 MiB/s wr, 2.16k op/s 2024-03-11T21:41:21.080 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:41:20 smithi097 ceph-mon[33317]: 5.18 scrub starts 2024-03-11T21:41:21.080 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:41:20 smithi097 ceph-mon[33317]: 5.18 scrub ok 2024-03-11T21:41:21.080 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:41:20 smithi097 ceph-mon[33317]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-03-11T21:41:21.081 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:41:20 smithi097 ceph-mon[33317]: Cluster is now healthy 2024-03-11T21:41:21.081 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:41:20 smithi097 ceph-mon[33317]: from='client.35624 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-03-11T21:41:21.081 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:41:20 smithi097 ceph-mon[33317]: osdmap e369: 8 total, 8 up, 8 in 2024-03-11T21:41:21.167 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:41:21.208 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:41:21.210 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:41:21.210 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:41:21.311 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:41:21.347 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:41:21.348 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:41:21.349 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:41:21.449 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:41:21.492 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:41:21.492 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:41:21.493 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:41:21.593 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:41:21.630 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:41:21.630 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:41:21.631 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:41:21.731 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:41:21.756 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:41:21.757 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:41:21.757 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:41:21.832 INFO:tasks.radosbench.radosbench.0.smithi092.stdout: 29 16 60940 60924 131.287 93.8125 0.00912363 0.00761157 2024-03-11T21:41:21.858 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:41:21.864 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:41:21 smithi092 ceph-mon[29572]: 6.e scrub starts 2024-03-11T21:41:21.864 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:41:21 smithi092 ceph-mon[29572]: 6.e scrub ok 2024-03-11T21:41:21.864 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:41:21 smithi092 ceph-mon[37563]: 6.e scrub starts 2024-03-11T21:41:21.865 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:41:21 smithi092 ceph-mon[37563]: 6.e scrub ok 2024-03-11T21:41:21.897 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:41:21.898 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:41:21.899 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:41:21.981 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:41:21 smithi097 ceph-mon[33317]: 6.e scrub starts 2024-03-11T21:41:21.981 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:41:21 smithi097 ceph-mon[33317]: 6.e scrub ok 2024-03-11T21:41:22.000 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:41:22.036 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:41:22.037 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:41:22.037 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:41:22.138 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:41:22.173 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:41:22.174 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:41:22.174 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:41:22.275 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:41:22.308 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:41:22.308 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:41:22.309 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:41:22.410 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:41:22.444 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:41:22.445 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:41:22.445 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:41:22.546 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:41:22.584 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:41:22.584 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:41:22.585 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:41:22.686 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:41:22.708 DEBUG:teuthology.orchestra.run.smithi092:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid 00db0016-dfed-11ee-95c8-87774f69a715 -- ceph osd set noscrub 2024-03-11T21:41:22.722 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:41:22.723 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:41:22.723 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:41:22.824 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:41:22.832 INFO:tasks.radosbench.radosbench.0.smithi092.stdout: 30 16 62443 62427 130.042 93.9375 0.0104751 0.00768417 2024-03-11T21:41:22.869 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:41:22.869 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:41:22.870 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:41:22.970 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:41:23.018 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:41:23.018 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:41:23.018 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:41:23.081 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:41:22 smithi097 ceph-mon[33317]: 2.6 scrub starts 2024-03-11T21:41:23.081 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:41:22 smithi097 ceph-mon[33317]: 2.6 scrub ok 2024-03-11T21:41:23.081 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:41:22 smithi097 ceph-mon[33317]: pgmap v735: 148 pgs: 148 active+clean; 3.3 GiB data, 14 GiB used, 702 GiB / 715 GiB avail; 853 B/s rd, 84 MiB/s wr, 1.35k op/s 2024-03-11T21:41:23.081 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:41:22 smithi097 ceph-mon[33317]: 2.12 scrub starts 2024-03-11T21:41:23.081 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:41:22 smithi097 ceph-mon[33317]: 2.12 scrub ok 2024-03-11T21:41:23.081 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:41:22 smithi097 ceph-mon[33317]: 5.2 deep-scrub starts 2024-03-11T21:41:23.081 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:41:22 smithi097 ceph-mon[33317]: 5.2 deep-scrub ok 2024-03-11T21:41:23.081 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:41:22 smithi097 ceph-mon[33317]: purged_snaps scrub starts 2024-03-11T21:41:23.081 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:41:22 smithi097 ceph-mon[33317]: purged_snaps scrub ok 2024-03-11T21:41:23.082 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:41:22 smithi097 ceph-mon[33317]: 2.8 scrub starts 2024-03-11T21:41:23.082 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:41:22 smithi097 ceph-mon[33317]: 2.8 scrub ok 2024-03-11T21:41:23.082 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:41:22 smithi097 ceph-mon[33317]: from='mgr.14607 ' entity='mgr.y' 2024-03-11T21:41:23.082 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:41:22 smithi097 ceph-mon[33317]: from='mgr.14607 v1:172.21.15.92:0/1702717597' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T21:41:23.082 INFO:journalctl@ceph.iscsi.iscsi.a.smithi097.stdout:Mar 11 21:41:22 smithi097 ceph-00db0016-dfed-11ee-95c8-87774f69a715-iscsi-iscsi-a[60943]: debug there is no tcmu-runner data available 2024-03-11T21:41:23.115 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:41:22 smithi092 ceph-mon[29572]: 2.6 scrub starts 2024-03-11T21:41:23.150 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:41:22 smithi092 ceph-mon[29572]: 2.6 scrub ok 2024-03-11T21:41:23.150 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:41:22 smithi092 ceph-mon[29572]: pgmap v735: 148 pgs: 148 active+clean; 3.3 GiB data, 14 GiB used, 702 GiB / 715 GiB avail; 853 B/s rd, 84 MiB/s wr, 1.35k op/s 2024-03-11T21:41:23.150 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:41:22 smithi092 ceph-mon[29572]: 2.12 scrub starts 2024-03-11T21:41:23.150 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:41:22 smithi092 ceph-mon[29572]: 2.12 scrub ok 2024-03-11T21:41:23.150 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:41:22 smithi092 ceph-mon[29572]: 5.2 deep-scrub starts 2024-03-11T21:41:23.150 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:41:22 smithi092 ceph-mon[29572]: 5.2 deep-scrub ok 2024-03-11T21:41:23.150 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:41:22 smithi092 ceph-mon[29572]: purged_snaps scrub starts 2024-03-11T21:41:23.150 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:41:22 smithi092 ceph-mon[29572]: purged_snaps scrub ok 2024-03-11T21:41:23.151 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:41:22 smithi092 ceph-mon[29572]: 2.8 scrub starts 2024-03-11T21:41:23.151 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:41:22 smithi092 ceph-mon[29572]: 2.8 scrub ok 2024-03-11T21:41:23.151 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:41:22 smithi092 ceph-mon[29572]: from='mgr.14607 ' entity='mgr.y' 2024-03-11T21:41:23.151 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:41:22 smithi092 ceph-mon[29572]: from='mgr.14607 v1:172.21.15.92:0/1702717597' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T21:41:23.151 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:41:22 smithi092 ceph-mon[37563]: 2.6 scrub starts 2024-03-11T21:41:23.151 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:41:22 smithi092 ceph-mon[37563]: 2.6 scrub ok 2024-03-11T21:41:23.151 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:41:22 smithi092 ceph-mon[37563]: pgmap v735: 148 pgs: 148 active+clean; 3.3 GiB data, 14 GiB used, 702 GiB / 715 GiB avail; 853 B/s rd, 84 MiB/s wr, 1.35k op/s 2024-03-11T21:41:23.151 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:41:22 smithi092 ceph-mon[37563]: 2.12 scrub starts 2024-03-11T21:41:23.151 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:41:22 smithi092 ceph-mon[37563]: 2.12 scrub ok 2024-03-11T21:41:23.151 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:41:22 smithi092 ceph-mon[37563]: 5.2 deep-scrub starts 2024-03-11T21:41:23.151 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:41:22 smithi092 ceph-mon[37563]: 5.2 deep-scrub ok 2024-03-11T21:41:23.153 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:41:22 smithi092 ceph-mon[37563]: purged_snaps scrub starts 2024-03-11T21:41:23.153 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:41:22 smithi092 ceph-mon[37563]: purged_snaps scrub ok 2024-03-11T21:41:23.153 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:41:22 smithi092 ceph-mon[37563]: 2.8 scrub starts 2024-03-11T21:41:23.153 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:41:22 smithi092 ceph-mon[37563]: 2.8 scrub ok 2024-03-11T21:41:23.153 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:41:22 smithi092 ceph-mon[37563]: from='mgr.14607 ' entity='mgr.y' 2024-03-11T21:41:23.153 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:41:22 smithi092 ceph-mon[37563]: from='mgr.14607 v1:172.21.15.92:0/1702717597' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T21:41:23.154 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:41:23.189 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:41:23.190 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:41:23.190 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:41:23.291 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:41:23.326 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:41:23.327 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:41:23.327 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:41:23.428 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:41:23.462 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:41:23.463 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:41:23.463 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:41:23.563 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:41:23.599 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:41:23.600 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:41:23.600 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:41:23.689 INFO:teuthology.orchestra.run.smithi092.stderr:Inferring config /var/lib/ceph/00db0016-dfed-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T21:41:23.701 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:41:23.768 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:41:23.769 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:41:23.769 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:41:23.832 INFO:tasks.radosbench.radosbench.0.smithi092.stdout: 31 16 63722 63706 128.425 79.9375 0.00674426 0.00778128 2024-03-11T21:41:23.870 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:41:23.887 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:41:23 smithi092 ceph-mon[29572]: 4.16 deep-scrub starts 2024-03-11T21:41:23.887 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:41:23 smithi092 ceph-mon[29572]: 4.16 deep-scrub ok 2024-03-11T21:41:23.887 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:41:23 smithi092 ceph-mon[29572]: 2.4 scrub starts 2024-03-11T21:41:23.887 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:41:23 smithi092 ceph-mon[29572]: 2.4 scrub ok 2024-03-11T21:41:23.887 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:41:23 smithi092 ceph-mon[29572]: osdmap e370: 8 total, 8 up, 8 in 2024-03-11T21:41:23.887 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:41:23 smithi092 ceph-mon[29572]: 2.7 scrub starts 2024-03-11T21:41:23.887 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:41:23 smithi092 ceph-mon[29572]: 2.7 scrub ok 2024-03-11T21:41:23.887 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:41:23 smithi092 ceph-mon[29572]: from='client.24469 v1:172.21.15.97:0/404208310' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:41:23.887 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:41:23 smithi092 ceph-mon[37563]: 4.16 deep-scrub starts 2024-03-11T21:41:23.888 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:41:23 smithi092 ceph-mon[37563]: 4.16 deep-scrub ok 2024-03-11T21:41:23.888 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:41:23 smithi092 ceph-mon[37563]: 2.4 scrub starts 2024-03-11T21:41:23.888 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:41:23 smithi092 ceph-mon[37563]: 2.4 scrub ok 2024-03-11T21:41:23.888 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:41:23 smithi092 ceph-mon[37563]: osdmap e370: 8 total, 8 up, 8 in 2024-03-11T21:41:23.888 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:41:23 smithi092 ceph-mon[37563]: 2.7 scrub starts 2024-03-11T21:41:23.888 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:41:23 smithi092 ceph-mon[37563]: 2.7 scrub ok 2024-03-11T21:41:23.888 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:41:23 smithi092 ceph-mon[37563]: from='client.24469 v1:172.21.15.97:0/404208310' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:41:23.913 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:41:23.913 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:41:23.913 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:41:23.983 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:41:23 smithi097 ceph-mon[33317]: 4.16 deep-scrub starts 2024-03-11T21:41:23.983 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:41:23 smithi097 ceph-mon[33317]: 4.16 deep-scrub ok 2024-03-11T21:41:23.983 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:41:23 smithi097 ceph-mon[33317]: 2.4 scrub starts 2024-03-11T21:41:23.983 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:41:23 smithi097 ceph-mon[33317]: 2.4 scrub ok 2024-03-11T21:41:23.983 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:41:23 smithi097 ceph-mon[33317]: osdmap e370: 8 total, 8 up, 8 in 2024-03-11T21:41:23.983 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:41:23 smithi097 ceph-mon[33317]: 2.7 scrub starts 2024-03-11T21:41:23.984 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:41:23 smithi097 ceph-mon[33317]: 2.7 scrub ok 2024-03-11T21:41:23.984 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:41:23 smithi097 ceph-mon[33317]: from='client.24469 v1:172.21.15.97:0/404208310' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:41:24.014 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:41:24.059 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:41:24.060 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:41:24.061 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:41:24.161 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:41:24.198 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:41:24.198 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:41:24.198 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:41:24.299 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:41:24.321 INFO:teuthology.orchestra.run.smithi092.stderr:Inferring config /var/lib/ceph/00db0016-dfed-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T21:41:24.336 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:41:24.337 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:41:24.337 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:41:24.438 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:41:24.480 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:41:24.480 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:41:24.481 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:41:24.581 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:41:24.615 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:41:24.616 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:41:24.616 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:41:24.717 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:41:24.748 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:41:24.749 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:41:24.749 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:41:24.833 INFO:tasks.radosbench.radosbench.0.smithi092.stdout: 32 16 65080 65064 127.064 84.875 0.0390164 0.00786515 2024-03-11T21:41:24.850 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:41:24.889 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:41:24.892 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:41:24.892 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:41:24.993 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:41:25.033 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:41:25.034 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:41:25.034 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:41:25.080 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:41:24 smithi097 ceph-mon[33317]: 6.9 scrub starts 2024-03-11T21:41:25.081 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:41:24 smithi097 ceph-mon[33317]: 6.9 scrub ok 2024-03-11T21:41:25.081 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:41:24 smithi097 ceph-mon[33317]: 2.10 scrub starts 2024-03-11T21:41:25.081 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:41:24 smithi097 ceph-mon[33317]: 2.10 scrub ok 2024-03-11T21:41:25.081 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:41:24 smithi097 ceph-mon[33317]: pgmap v737: 148 pgs: 148 active+clean; 3.7 GiB data, 15 GiB used, 701 GiB / 715 GiB avail; 852 B/s rd, 137 MiB/s wr, 2.20k op/s 2024-03-11T21:41:25.081 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:41:24 smithi097 ceph-mon[33317]: 2.1b scrub starts 2024-03-11T21:41:25.081 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:41:24 smithi097 ceph-mon[33317]: 2.1b scrub ok 2024-03-11T21:41:25.115 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:41:24 smithi092 ceph-mon[29572]: 6.9 scrub starts 2024-03-11T21:41:25.115 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:41:24 smithi092 ceph-mon[29572]: 6.9 scrub ok 2024-03-11T21:41:25.115 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:41:24 smithi092 ceph-mon[29572]: 2.10 scrub starts 2024-03-11T21:41:25.115 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:41:24 smithi092 ceph-mon[29572]: 2.10 scrub ok 2024-03-11T21:41:25.115 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:41:24 smithi092 ceph-mon[29572]: pgmap v737: 148 pgs: 148 active+clean; 3.7 GiB data, 15 GiB used, 701 GiB / 715 GiB avail; 852 B/s rd, 137 MiB/s wr, 2.20k op/s 2024-03-11T21:41:25.115 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:41:24 smithi092 ceph-mon[29572]: 2.1b scrub starts 2024-03-11T21:41:25.115 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:41:24 smithi092 ceph-mon[29572]: 2.1b scrub ok 2024-03-11T21:41:25.115 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:41:24 smithi092 ceph-mon[37563]: 6.9 scrub starts 2024-03-11T21:41:25.115 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:41:24 smithi092 ceph-mon[37563]: 6.9 scrub ok 2024-03-11T21:41:25.116 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:41:24 smithi092 ceph-mon[37563]: 2.10 scrub starts 2024-03-11T21:41:25.116 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:41:24 smithi092 ceph-mon[37563]: 2.10 scrub ok 2024-03-11T21:41:25.116 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:41:24 smithi092 ceph-mon[37563]: pgmap v737: 148 pgs: 148 active+clean; 3.7 GiB data, 15 GiB used, 701 GiB / 715 GiB avail; 852 B/s rd, 137 MiB/s wr, 2.20k op/s 2024-03-11T21:41:25.116 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:41:24 smithi092 ceph-mon[37563]: 2.1b scrub starts 2024-03-11T21:41:25.116 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:41:24 smithi092 ceph-mon[37563]: 2.1b scrub ok 2024-03-11T21:41:25.135 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:41:25.170 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:41:25.171 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:41:25.171 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:41:25.279 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:41:25.314 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:41:25.315 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:41:25.315 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:41:25.416 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:41:25.453 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:41:25.453 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:41:25.454 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:41:25.554 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:41:25.590 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:41:25.591 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:41:25.591 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:41:25.692 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:41:25.730 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:41:25.731 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:41:25.731 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:41:25.769 INFO:teuthology.orchestra.run.smithi092.stderr:set pool 10 pg_num to 26 2024-03-11T21:41:25.832 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:41:25.833 INFO:tasks.radosbench.radosbench.0.smithi092.stdout: 33 16 66466 66450 125.839 86.625 0.00749944 0.00794219 2024-03-11T21:41:25.872 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:41:25.873 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:41:25.873 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:41:25.892 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:41:25 smithi092 ceph-mon[29572]: 2.c scrub starts 2024-03-11T21:41:25.892 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:41:25 smithi092 ceph-mon[29572]: 2.c scrub ok 2024-03-11T21:41:25.893 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:41:25 smithi092 ceph-mon[29572]: 2.2 scrub starts 2024-03-11T21:41:25.893 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:41:25 smithi092 ceph-mon[29572]: 2.2 scrub ok 2024-03-11T21:41:25.893 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:41:25 smithi092 ceph-mon[29572]: from='client.? v1:172.21.15.92:0/3571371033' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "unique_pool_3", "var": "pg_num", "val": "26"}]: dispatch 2024-03-11T21:41:25.893 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:41:25 smithi092 ceph-mon[37563]: 2.c scrub starts 2024-03-11T21:41:25.893 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:41:25 smithi092 ceph-mon[37563]: 2.c scrub ok 2024-03-11T21:41:25.893 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:41:25 smithi092 ceph-mon[37563]: 2.2 scrub starts 2024-03-11T21:41:25.893 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:41:25 smithi092 ceph-mon[37563]: 2.2 scrub ok 2024-03-11T21:41:25.894 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:41:25 smithi092 ceph-mon[37563]: from='client.? v1:172.21.15.92:0/3571371033' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "unique_pool_3", "var": "pg_num", "val": "26"}]: dispatch 2024-03-11T21:41:25.974 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:41:26.013 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:41:26.013 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:41:26.013 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:41:26.081 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:41:25 smithi097 ceph-mon[33317]: 2.c scrub starts 2024-03-11T21:41:26.081 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:41:25 smithi097 ceph-mon[33317]: 2.c scrub ok 2024-03-11T21:41:26.081 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:41:25 smithi097 ceph-mon[33317]: 2.2 scrub starts 2024-03-11T21:41:26.081 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:41:25 smithi097 ceph-mon[33317]: 2.2 scrub ok 2024-03-11T21:41:26.082 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:41:25 smithi097 ceph-mon[33317]: from='client.? v1:172.21.15.92:0/3571371033' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "unique_pool_3", "var": "pg_num", "val": "26"}]: dispatch 2024-03-11T21:41:26.114 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:41:26.150 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:41:26.151 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:41:26.151 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:41:26.170 INFO:teuthology.orchestra.run.smithi092.stderr:Inferring config /var/lib/ceph/00db0016-dfed-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T21:41:26.253 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:41:26.286 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:41:26.286 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:41:26.287 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:41:26.387 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:41:26.426 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:41:26.427 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:41:26.427 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:41:26.528 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:41:26.567 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:41:26.568 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:41:26.568 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:41:26.669 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:41:26.705 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:41:26.706 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:41:26.707 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:41:26.807 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:41:26.833 INFO:tasks.radosbench.radosbench.0.smithi092.stdout: 34 16 67889 67873 124.753 88.9375 0.00892556 0.00801068 2024-03-11T21:41:26.844 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:41:26.844 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:41:26.844 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:41:26.945 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:41:26.982 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:41:26.982 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:41:26.982 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:41:27.080 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:41:26 smithi097 ceph-mon[33317]: pgmap v738: 148 pgs: 148 active+clean; 3.8 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 738 B/s rd, 76 MiB/s wr, 1.21k op/s 2024-03-11T21:41:27.081 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:41:26 smithi097 ceph-mon[33317]: 6.17 scrub starts 2024-03-11T21:41:27.081 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:41:26 smithi097 ceph-mon[33317]: 6.17 scrub ok 2024-03-11T21:41:27.081 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:41:26 smithi097 ceph-mon[33317]: from='client.? v1:172.21.15.92:0/3571371033' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_3", "var": "pg_num", "val": "26"}]': finished 2024-03-11T21:41:27.081 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:41:26 smithi097 ceph-mon[33317]: osdmap e371: 8 total, 8 up, 8 in 2024-03-11T21:41:27.081 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:41:26 smithi097 ceph-mon[33317]: 2.16 scrub starts 2024-03-11T21:41:27.081 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:41:26 smithi097 ceph-mon[33317]: 2.16 scrub ok 2024-03-11T21:41:27.081 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:41:26 smithi097 ceph-mon[33317]: 6.8 scrub starts 2024-03-11T21:41:27.081 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:41:26 smithi097 ceph-mon[33317]: 6.8 scrub ok 2024-03-11T21:41:27.081 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:41:26 smithi097 ceph-mon[33317]: 6.13 scrub starts 2024-03-11T21:41:27.084 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:41:27.115 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:41:26 smithi092 ceph-mon[29572]: pgmap v738: 148 pgs: 148 active+clean; 3.8 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 738 B/s rd, 76 MiB/s wr, 1.21k op/s 2024-03-11T21:41:27.115 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:41:26 smithi092 ceph-mon[29572]: 6.17 scrub starts 2024-03-11T21:41:27.115 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:41:26 smithi092 ceph-mon[29572]: 6.17 scrub ok 2024-03-11T21:41:27.115 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:41:26 smithi092 ceph-mon[29572]: from='client.? v1:172.21.15.92:0/3571371033' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_3", "var": "pg_num", "val": "26"}]': finished 2024-03-11T21:41:27.115 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:41:26 smithi092 ceph-mon[29572]: osdmap e371: 8 total, 8 up, 8 in 2024-03-11T21:41:27.115 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:41:26 smithi092 ceph-mon[29572]: 2.16 scrub starts 2024-03-11T21:41:27.115 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:41:26 smithi092 ceph-mon[29572]: 2.16 scrub ok 2024-03-11T21:41:27.115 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:41:26 smithi092 ceph-mon[29572]: 6.8 scrub starts 2024-03-11T21:41:27.115 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:41:26 smithi092 ceph-mon[29572]: 6.8 scrub ok 2024-03-11T21:41:27.115 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:41:26 smithi092 ceph-mon[29572]: 6.13 scrub starts 2024-03-11T21:41:27.116 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:41:26 smithi092 ceph-mon[37563]: pgmap v738: 148 pgs: 148 active+clean; 3.8 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 738 B/s rd, 76 MiB/s wr, 1.21k op/s 2024-03-11T21:41:27.116 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:41:26 smithi092 ceph-mon[37563]: 6.17 scrub starts 2024-03-11T21:41:27.116 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:41:26 smithi092 ceph-mon[37563]: 6.17 scrub ok 2024-03-11T21:41:27.116 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:41:26 smithi092 ceph-mon[37563]: from='client.? v1:172.21.15.92:0/3571371033' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_3", "var": "pg_num", "val": "26"}]': finished 2024-03-11T21:41:27.116 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:41:26 smithi092 ceph-mon[37563]: osdmap e371: 8 total, 8 up, 8 in 2024-03-11T21:41:27.116 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:41:26 smithi092 ceph-mon[37563]: 2.16 scrub starts 2024-03-11T21:41:27.116 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:41:26 smithi092 ceph-mon[37563]: 2.16 scrub ok 2024-03-11T21:41:27.116 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:41:26 smithi092 ceph-mon[37563]: 6.8 scrub starts 2024-03-11T21:41:27.116 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:41:26 smithi092 ceph-mon[37563]: 6.8 scrub ok 2024-03-11T21:41:27.116 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:41:26 smithi092 ceph-mon[37563]: 6.13 scrub starts 2024-03-11T21:41:27.122 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:41:27.122 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:41:27.123 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:41:27.224 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:41:27.264 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:41:27.265 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:41:27.265 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:41:27.366 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:41:27.400 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:41:27.401 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:41:27.401 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:41:27.502 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:41:27.539 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:41:27.540 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:41:27.540 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:41:27.641 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:41:27.680 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:41:27.681 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:41:27.682 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:41:27.782 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:41:27.820 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:41:27.821 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:41:27.822 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:41:27.833 INFO:tasks.radosbench.radosbench.0.smithi092.stdout: 35 16 69298 69282 123.704 88.0625 0.0292778 0.00807228 2024-03-11T21:41:27.922 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:41:28.000 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:41:28.001 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:41:28.001 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:41:28.080 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:41:27 smithi097 ceph-mon[33317]: 6.13 scrub ok 2024-03-11T21:41:28.081 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:41:27 smithi097 ceph-mon[33317]: 2.17 scrub starts 2024-03-11T21:41:28.081 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:41:27 smithi097 ceph-mon[33317]: 2.17 scrub ok 2024-03-11T21:41:28.081 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:41:27 smithi097 ceph-mon[33317]: 4.7 deep-scrub starts 2024-03-11T21:41:28.081 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:41:27 smithi097 ceph-mon[33317]: 4.7 deep-scrub ok 2024-03-11T21:41:28.081 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:41:27 smithi097 ceph-mon[33317]: from='mgr.14607 v1:172.21.15.92:0/1702717597' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "unique_pool_3", "var": "pg_num_actual", "val": "26"}]: dispatch 2024-03-11T21:41:28.081 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:41:27 smithi097 ceph-mon[33317]: from='mgr.14607 ' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "unique_pool_3", "var": "pg_num_actual", "val": "26"}]: dispatch 2024-03-11T21:41:28.101 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:41:28.114 INFO:journalctl@ceph.mgr.y.smithi092.stdout:Mar 11 21:41:27 smithi092 ceph-00db0016-dfed-11ee-95c8-87774f69a715-mgr-y[29801]: ::ffff:172.21.15.97 - - [11/Mar/2024:21:41:27] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-11T21:41:28.115 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:41:27 smithi092 ceph-mon[29572]: 6.13 scrub ok 2024-03-11T21:41:28.115 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:41:27 smithi092 ceph-mon[29572]: 2.17 scrub starts 2024-03-11T21:41:28.115 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:41:27 smithi092 ceph-mon[29572]: 2.17 scrub ok 2024-03-11T21:41:28.115 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:41:27 smithi092 ceph-mon[29572]: 4.7 deep-scrub starts 2024-03-11T21:41:28.115 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:41:27 smithi092 ceph-mon[29572]: 4.7 deep-scrub ok 2024-03-11T21:41:28.115 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:41:27 smithi092 ceph-mon[29572]: from='mgr.14607 v1:172.21.15.92:0/1702717597' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "unique_pool_3", "var": "pg_num_actual", "val": "26"}]: dispatch 2024-03-11T21:41:28.115 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:41:27 smithi092 ceph-mon[29572]: from='mgr.14607 ' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "unique_pool_3", "var": "pg_num_actual", "val": "26"}]: dispatch 2024-03-11T21:41:28.115 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:41:27 smithi092 ceph-mon[37563]: 6.13 scrub ok 2024-03-11T21:41:28.115 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:41:27 smithi092 ceph-mon[37563]: 2.17 scrub starts 2024-03-11T21:41:28.115 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:41:27 smithi092 ceph-mon[37563]: 2.17 scrub ok 2024-03-11T21:41:28.116 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:41:27 smithi092 ceph-mon[37563]: 4.7 deep-scrub starts 2024-03-11T21:41:28.116 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:41:27 smithi092 ceph-mon[37563]: 4.7 deep-scrub ok 2024-03-11T21:41:28.116 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:41:27 smithi092 ceph-mon[37563]: from='mgr.14607 v1:172.21.15.92:0/1702717597' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "unique_pool_3", "var": "pg_num_actual", "val": "26"}]: dispatch 2024-03-11T21:41:28.116 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:41:27 smithi092 ceph-mon[37563]: from='mgr.14607 ' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "unique_pool_3", "var": "pg_num_actual", "val": "26"}]: dispatch 2024-03-11T21:41:28.125 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:41:28.126 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:41:28.126 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:41:28.227 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:41:28.251 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:41:28.252 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:41:28.252 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:41:28.353 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:41:28.379 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:41:28.380 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:41:28.381 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:41:28.481 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:41:28.506 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:41:28.507 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:41:28.507 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:41:28.608 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:41:28.632 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:41:28.633 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:41:28.633 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:41:28.734 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:41:28.757 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:41:28.758 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:41:28.758 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:41:28.833 INFO:tasks.radosbench.radosbench.0.smithi092.stdout: 36 16 69310 69294 120.289 0.75 1.03624 0.00815222 2024-03-11T21:41:28.859 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:41:28.895 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:41:28.896 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:41:28.896 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:41:28.997 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:41:29.034 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:41:29.035 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:41:29.035 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:41:29.080 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:41:28 smithi097 ceph-mon[33317]: pgmap v740: 148 pgs: 148 active+clean; 3.9 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 671 B/s rd, 75 MiB/s wr, 1.20k op/s 2024-03-11T21:41:29.081 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:41:28 smithi097 ceph-mon[33317]: 2.e scrub starts 2024-03-11T21:41:29.081 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:41:28 smithi097 ceph-mon[33317]: 2.e scrub ok 2024-03-11T21:41:29.081 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:41:28 smithi097 ceph-mon[33317]: from='mgr.14607 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_3", "var": "pg_num_actual", "val": "26"}]': finished 2024-03-11T21:41:29.081 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:41:28 smithi097 ceph-mon[33317]: osdmap e372: 8 total, 8 up, 8 in 2024-03-11T21:41:29.081 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:41:28 smithi097 ceph-mon[33317]: 5.9 scrub starts 2024-03-11T21:41:29.081 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:41:28 smithi097 ceph-mon[33317]: 5.9 scrub ok 2024-03-11T21:41:29.115 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:41:28 smithi092 ceph-mon[29572]: pgmap v740: 148 pgs: 148 active+clean; 3.9 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 671 B/s rd, 75 MiB/s wr, 1.20k op/s 2024-03-11T21:41:29.115 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:41:28 smithi092 ceph-mon[29572]: 2.e scrub starts 2024-03-11T21:41:29.115 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:41:28 smithi092 ceph-mon[29572]: 2.e scrub ok 2024-03-11T21:41:29.115 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:41:28 smithi092 ceph-mon[29572]: from='mgr.14607 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_3", "var": "pg_num_actual", "val": "26"}]': finished 2024-03-11T21:41:29.115 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:41:28 smithi092 ceph-mon[29572]: osdmap e372: 8 total, 8 up, 8 in 2024-03-11T21:41:29.115 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:41:28 smithi092 ceph-mon[29572]: 5.9 scrub starts 2024-03-11T21:41:29.115 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:41:28 smithi092 ceph-mon[29572]: 5.9 scrub ok 2024-03-11T21:41:29.115 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:41:28 smithi092 ceph-mon[37563]: pgmap v740: 148 pgs: 148 active+clean; 3.9 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 671 B/s rd, 75 MiB/s wr, 1.20k op/s 2024-03-11T21:41:29.115 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:41:28 smithi092 ceph-mon[37563]: 2.e scrub starts 2024-03-11T21:41:29.116 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:41:28 smithi092 ceph-mon[37563]: 2.e scrub ok 2024-03-11T21:41:29.116 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:41:28 smithi092 ceph-mon[37563]: from='mgr.14607 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_3", "var": "pg_num_actual", "val": "26"}]': finished 2024-03-11T21:41:29.116 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:41:28 smithi092 ceph-mon[37563]: osdmap e372: 8 total, 8 up, 8 in 2024-03-11T21:41:29.116 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:41:28 smithi092 ceph-mon[37563]: 5.9 scrub starts 2024-03-11T21:41:29.116 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:41:28 smithi092 ceph-mon[37563]: 5.9 scrub ok 2024-03-11T21:41:29.136 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:41:29.173 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:41:29.174 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:41:29.174 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:41:29.275 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:41:29.311 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:41:29.312 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:41:29.312 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:41:29.413 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:41:29.436 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:41:29.436 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:41:29.436 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:41:29.537 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:41:29.570 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:41:29.570 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:41:29.571 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:41:29.672 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:41:29.711 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:41:29.713 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:41:29.714 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:41:29.815 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:41:29.834 INFO:tasks.radosbench.radosbench.0.smithi092.stdout: 37 16 70487 70471 119.024 73.5625 0.0257843 0.0083921 2024-03-11T21:41:29.848 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:41:29.848 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:41:29.849 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:41:29.950 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:41:29.979 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:41:29.979 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:41:29.979 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:41:30.080 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:41:30.081 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:41:29 smithi097 ceph-mon[33317]: osdmap e373: 8 total, 8 up, 8 in 2024-03-11T21:41:30.081 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:41:29 smithi097 ceph-mon[33317]: 10.0 scrub starts 2024-03-11T21:41:30.081 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:41:29 smithi097 ceph-mon[33317]: 10.9 deep-scrub starts 2024-03-11T21:41:30.082 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:41:29 smithi097 ceph-mon[33317]: 10.2 scrub starts 2024-03-11T21:41:30.082 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:41:29 smithi097 ceph-mon[33317]: 2.b scrub starts 2024-03-11T21:41:30.082 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:41:29 smithi097 ceph-mon[33317]: 2.b scrub ok 2024-03-11T21:41:30.082 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:41:29 smithi097 ceph-mon[33317]: 10.0 scrub ok 2024-03-11T21:41:30.082 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:41:29 smithi097 ceph-mon[33317]: from='mgr.14607 v1:172.21.15.92:0/1702717597' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "unique_pool_3", "var": "pgp_num_actual", "val": "26"}]: dispatch 2024-03-11T21:41:30.082 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:41:29 smithi097 ceph-mon[33317]: from='mgr.14607 ' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "unique_pool_3", "var": "pgp_num_actual", "val": "26"}]: dispatch 2024-03-11T21:41:30.105 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:41:30.106 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:41:30.106 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:41:30.115 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:41:29 smithi092 ceph-mon[29572]: osdmap e373: 8 total, 8 up, 8 in 2024-03-11T21:41:30.115 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:41:29 smithi092 ceph-mon[29572]: 10.0 scrub starts 2024-03-11T21:41:30.115 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:41:29 smithi092 ceph-mon[29572]: 10.9 deep-scrub starts 2024-03-11T21:41:30.115 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:41:29 smithi092 ceph-mon[29572]: 10.2 scrub starts 2024-03-11T21:41:30.115 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:41:29 smithi092 ceph-mon[29572]: 2.b scrub starts 2024-03-11T21:41:30.115 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:41:29 smithi092 ceph-mon[29572]: 2.b scrub ok 2024-03-11T21:41:30.115 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:41:29 smithi092 ceph-mon[29572]: 10.0 scrub ok 2024-03-11T21:41:30.115 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:41:29 smithi092 ceph-mon[29572]: from='mgr.14607 v1:172.21.15.92:0/1702717597' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "unique_pool_3", "var": "pgp_num_actual", "val": "26"}]: dispatch 2024-03-11T21:41:30.115 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:41:29 smithi092 ceph-mon[29572]: from='mgr.14607 ' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "unique_pool_3", "var": "pgp_num_actual", "val": "26"}]: dispatch 2024-03-11T21:41:30.115 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:41:29 smithi092 ceph-mon[37563]: osdmap e373: 8 total, 8 up, 8 in 2024-03-11T21:41:30.116 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:41:29 smithi092 ceph-mon[37563]: 10.0 scrub starts 2024-03-11T21:41:30.116 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:41:29 smithi092 ceph-mon[37563]: 10.9 deep-scrub starts 2024-03-11T21:41:30.116 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:41:29 smithi092 ceph-mon[37563]: 10.2 scrub starts 2024-03-11T21:41:30.116 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:41:29 smithi092 ceph-mon[37563]: 2.b scrub starts 2024-03-11T21:41:30.116 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:41:29 smithi092 ceph-mon[37563]: 2.b scrub ok 2024-03-11T21:41:30.116 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:41:29 smithi092 ceph-mon[37563]: 10.0 scrub ok 2024-03-11T21:41:30.116 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:41:29 smithi092 ceph-mon[37563]: from='mgr.14607 v1:172.21.15.92:0/1702717597' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "unique_pool_3", "var": "pgp_num_actual", "val": "26"}]: dispatch 2024-03-11T21:41:30.116 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:41:29 smithi092 ceph-mon[37563]: from='mgr.14607 ' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "unique_pool_3", "var": "pgp_num_actual", "val": "26"}]: dispatch 2024-03-11T21:41:30.206 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:41:30.237 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:41:30.238 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:41:30.238 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:41:30.339 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:41:30.372 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:41:30.372 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:41:30.373 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:41:30.474 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:41:30.498 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:41:30.499 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:41:30.499 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:41:30.600 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:41:30.630 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:41:30.631 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:41:30.631 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:41:30.732 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:41:30.757 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:41:30.757 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:41:30.757 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:41:30.834 INFO:tasks.radosbench.radosbench.0.smithi092.stdout: 38 16 70529 70513 115.961 2.625 0.00752967 0.00839491 2024-03-11T21:41:30.858 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:41:30.889 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:41:30.889 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:41:30.890 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:41:30.892 INFO:teuthology.orchestra.run.smithi092.stdout:{ 2024-03-11T21:41:30.892 INFO:teuthology.orchestra.run.smithi092.stdout: "ops": [], 2024-03-11T21:41:30.892 INFO:teuthology.orchestra.run.smithi092.stdout: "complaint_time": 30, 2024-03-11T21:41:30.892 INFO:teuthology.orchestra.run.smithi092.stdout: "num_blocked_ops": 0 2024-03-11T21:41:30.893 INFO:teuthology.orchestra.run.smithi092.stdout:} 2024-03-11T21:41:30.990 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:41:31.017 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:41:31.017 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:41:31.018 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:41:31.081 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:41:30 smithi097 ceph-mon[33317]: 10.8 scrub starts 2024-03-11T21:41:31.081 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:41:30 smithi097 ceph-mon[33317]: pgmap v743: 158 pgs: 2 activating, 1 peering, 8 unknown, 147 active+clean; 4.2 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 1.7 KiB/s rd, 84 MiB/s wr, 1.34k op/s 2024-03-11T21:41:31.081 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:41:30 smithi097 ceph-mon[33317]: 10.5 deep-scrub starts 2024-03-11T21:41:31.081 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:41:30 smithi097 ceph-mon[33317]: 10.1 scrub starts 2024-03-11T21:41:31.081 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:41:30 smithi097 ceph-mon[33317]: from='mgr.14607 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_3", "var": "pgp_num_actual", "val": "26"}]': finished 2024-03-11T21:41:31.081 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:41:30 smithi097 ceph-mon[33317]: 10.10 scrub starts 2024-03-11T21:41:31.081 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:41:30 smithi097 ceph-mon[33317]: osdmap e374: 8 total, 8 up, 8 in 2024-03-11T21:41:31.081 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:41:30 smithi097 ceph-mon[33317]: 10.2 scrub ok 2024-03-11T21:41:31.081 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:41:30 smithi097 ceph-mon[33317]: 10.8 scrub ok 2024-03-11T21:41:31.082 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:41:30 smithi097 ceph-mon[33317]: 10.6 scrub starts 2024-03-11T21:41:31.082 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:41:30 smithi097 ceph-mon[33317]: 3.0 scrub starts 2024-03-11T21:41:31.082 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:41:30 smithi097 ceph-mon[33317]: 3.0 scrub ok 2024-03-11T21:41:31.082 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:41:30 smithi097 ceph-mon[33317]: 10.1 scrub ok 2024-03-11T21:41:31.082 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:41:30 smithi097 ceph-mon[33317]: 4.0 scrub starts 2024-03-11T21:41:31.082 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:41:30 smithi097 ceph-mon[33317]: 4.0 scrub ok 2024-03-11T21:41:31.115 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:41:30 smithi092 ceph-mon[29572]: 10.8 scrub starts 2024-03-11T21:41:31.115 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:41:30 smithi092 ceph-mon[29572]: pgmap v743: 158 pgs: 2 activating, 1 peering, 8 unknown, 147 active+clean; 4.2 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 1.7 KiB/s rd, 84 MiB/s wr, 1.34k op/s 2024-03-11T21:41:31.115 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:41:30 smithi092 ceph-mon[29572]: 10.5 deep-scrub starts 2024-03-11T21:41:31.115 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:41:30 smithi092 ceph-mon[29572]: 10.1 scrub starts 2024-03-11T21:41:31.115 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:41:30 smithi092 ceph-mon[29572]: from='mgr.14607 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_3", "var": "pgp_num_actual", "val": "26"}]': finished 2024-03-11T21:41:31.116 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:41:30 smithi092 ceph-mon[29572]: 10.10 scrub starts 2024-03-11T21:41:31.116 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:41:30 smithi092 ceph-mon[29572]: osdmap e374: 8 total, 8 up, 8 in 2024-03-11T21:41:31.116 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:41:30 smithi092 ceph-mon[29572]: 10.2 scrub ok 2024-03-11T21:41:31.116 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:41:30 smithi092 ceph-mon[29572]: 10.8 scrub ok 2024-03-11T21:41:31.116 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:41:30 smithi092 ceph-mon[29572]: 10.6 scrub starts 2024-03-11T21:41:31.116 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:41:30 smithi092 ceph-mon[29572]: 3.0 scrub starts 2024-03-11T21:41:31.116 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:41:30 smithi092 ceph-mon[29572]: 3.0 scrub ok 2024-03-11T21:41:31.116 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:41:30 smithi092 ceph-mon[29572]: 10.1 scrub ok 2024-03-11T21:41:31.116 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:41:30 smithi092 ceph-mon[29572]: 4.0 scrub starts 2024-03-11T21:41:31.116 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:41:30 smithi092 ceph-mon[29572]: 4.0 scrub ok 2024-03-11T21:41:31.116 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:41:30 smithi092 ceph-mon[37563]: 10.8 scrub starts 2024-03-11T21:41:31.117 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:41:30 smithi092 ceph-mon[37563]: pgmap v743: 158 pgs: 2 activating, 1 peering, 8 unknown, 147 active+clean; 4.2 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 1.7 KiB/s rd, 84 MiB/s wr, 1.34k op/s 2024-03-11T21:41:31.117 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:41:30 smithi092 ceph-mon[37563]: 10.5 deep-scrub starts 2024-03-11T21:41:31.117 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:41:30 smithi092 ceph-mon[37563]: 10.1 scrub starts 2024-03-11T21:41:31.117 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:41:30 smithi092 ceph-mon[37563]: from='mgr.14607 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_3", "var": "pgp_num_actual", "val": "26"}]': finished 2024-03-11T21:41:31.117 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:41:30 smithi092 ceph-mon[37563]: 10.10 scrub starts 2024-03-11T21:41:31.117 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:41:30 smithi092 ceph-mon[37563]: osdmap e374: 8 total, 8 up, 8 in 2024-03-11T21:41:31.117 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:41:30 smithi092 ceph-mon[37563]: 10.2 scrub ok 2024-03-11T21:41:31.117 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:41:30 smithi092 ceph-mon[37563]: 10.8 scrub ok 2024-03-11T21:41:31.117 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:41:30 smithi092 ceph-mon[37563]: 10.6 scrub starts 2024-03-11T21:41:31.117 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:41:30 smithi092 ceph-mon[37563]: 3.0 scrub starts 2024-03-11T21:41:31.117 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:41:30 smithi092 ceph-mon[37563]: 3.0 scrub ok 2024-03-11T21:41:31.117 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:41:30 smithi092 ceph-mon[37563]: 10.1 scrub ok 2024-03-11T21:41:31.117 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:41:30 smithi092 ceph-mon[37563]: 4.0 scrub starts 2024-03-11T21:41:31.117 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:41:30 smithi092 ceph-mon[37563]: 4.0 scrub ok 2024-03-11T21:41:31.119 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:41:31.147 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:41:31.148 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:41:31.148 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:41:31.249 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:41:31.276 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:41:31.277 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:41:31.277 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:41:31.378 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:41:31.407 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:41:31.407 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:41:31.407 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:41:31.508 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:41:31.537 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:41:31.538 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:41:31.538 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:41:31.611 INFO:teuthology.orchestra.run.smithi092.stdout:osd.0: {} 2024-03-11T21:41:31.635 INFO:teuthology.orchestra.run.smithi092.stdout:osd.1: {} 2024-03-11T21:41:31.639 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:41:31.659 INFO:teuthology.orchestra.run.smithi092.stdout:osd.2: {} 2024-03-11T21:41:31.668 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:41:31.669 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:41:31.669 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:41:31.687 INFO:teuthology.orchestra.run.smithi092.stdout:osd.3: {} 2024-03-11T21:41:31.715 INFO:teuthology.orchestra.run.smithi092.stdout:osd.4: {} 2024-03-11T21:41:31.741 INFO:teuthology.orchestra.run.smithi092.stdout:osd.5: {} 2024-03-11T21:41:31.769 INFO:teuthology.orchestra.run.smithi092.stdout:osd.6: {} 2024-03-11T21:41:31.770 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:41:31.798 INFO:teuthology.orchestra.run.smithi092.stdout:osd.7: {} 2024-03-11T21:41:31.800 INFO:teuthology.orchestra.run.smithi092.stderr:osd.0: osd_enable_op_tracker = '' 2024-03-11T21:41:31.801 INFO:teuthology.orchestra.run.smithi092.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-11T21:41:31.801 INFO:teuthology.orchestra.run.smithi092.stderr:osd.2: osd_enable_op_tracker = '' 2024-03-11T21:41:31.801 INFO:teuthology.orchestra.run.smithi092.stderr:osd.3: osd_enable_op_tracker = '' 2024-03-11T21:41:31.801 INFO:teuthology.orchestra.run.smithi092.stderr:osd.4: osd_enable_op_tracker = '' 2024-03-11T21:41:31.801 INFO:teuthology.orchestra.run.smithi092.stderr:osd.5: osd_enable_op_tracker = '' 2024-03-11T21:41:31.801 INFO:teuthology.orchestra.run.smithi092.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-11T21:41:31.801 INFO:teuthology.orchestra.run.smithi092.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-11T21:41:31.802 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:41:31.803 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:41:31.803 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:41:31.834 INFO:tasks.radosbench.radosbench.0.smithi092.stdout: 39 16 70529 70513 112.987 0 - 0.00839491 2024-03-11T21:41:31.904 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:41:31.940 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:41:31.941 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:41:31.941 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:41:32.042 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:41:32.081 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:41:31 smithi097 ceph-mon[33317]: 10.4 scrub starts 2024-03-11T21:41:32.081 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:41:31 smithi097 ceph-mon[33317]: 10.6 scrub ok 2024-03-11T21:41:32.081 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:41:31 smithi097 ceph-mon[33317]: 10.4 scrub ok 2024-03-11T21:41:32.081 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:41:31 smithi097 ceph-mon[33317]: 10.3 scrub starts 2024-03-11T21:41:32.081 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:41:31 smithi097 ceph-mon[33317]: 4.c scrub starts 2024-03-11T21:41:32.081 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:41:31 smithi097 ceph-mon[33317]: 4.c scrub ok 2024-03-11T21:41:32.081 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:41:31 smithi097 ceph-mon[33317]: 10.3 scrub ok 2024-03-11T21:41:32.081 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:41:31 smithi097 ceph-mon[33317]: osdmap e375: 8 total, 8 up, 8 in 2024-03-11T21:41:32.081 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:41:31 smithi097 ceph-mon[33317]: 10.7 scrub starts 2024-03-11T21:41:32.081 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:41:31 smithi097 ceph-mon[33317]: 10.7 scrub ok 2024-03-11T21:41:32.081 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:41:31 smithi097 ceph-mon[33317]: 5.c scrub starts 2024-03-11T21:41:32.081 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:41:31 smithi097 ceph-mon[33317]: 5.c scrub ok 2024-03-11T21:41:32.082 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:41:31 smithi097 ceph-mon[33317]: 6.15 scrub starts 2024-03-11T21:41:32.082 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:41:31 smithi097 ceph-mon[33317]: 6.15 scrub ok 2024-03-11T21:41:32.082 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:41:31 smithi097 ceph-mon[33317]: 5.17 scrub starts 2024-03-11T21:41:32.082 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:41:31 smithi097 ceph-mon[33317]: 5.17 scrub ok 2024-03-11T21:41:32.082 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:41:31 smithi097 ceph-mon[33317]: from='client.? v1:172.21.15.92:0/2181317857' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-11T21:41:32.082 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:41:31 smithi097 ceph-mon[33317]: osdmap e376: 8 total, 8 up, 8 in 2024-03-11T21:41:32.109 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:41:32.110 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:41:32.110 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:41:32.115 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:41:31 smithi092 ceph-mon[29572]: 10.4 scrub starts 2024-03-11T21:41:32.115 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:41:31 smithi092 ceph-mon[29572]: 10.6 scrub ok 2024-03-11T21:41:32.115 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:41:31 smithi092 ceph-mon[29572]: 10.4 scrub ok 2024-03-11T21:41:32.115 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:41:31 smithi092 ceph-mon[29572]: 10.3 scrub starts 2024-03-11T21:41:32.115 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:41:31 smithi092 ceph-mon[29572]: 4.c scrub starts 2024-03-11T21:41:32.115 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:41:31 smithi092 ceph-mon[29572]: 4.c scrub ok 2024-03-11T21:41:32.115 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:41:31 smithi092 ceph-mon[29572]: 10.3 scrub ok 2024-03-11T21:41:32.115 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:41:31 smithi092 ceph-mon[29572]: osdmap e375: 8 total, 8 up, 8 in 2024-03-11T21:41:32.115 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:41:31 smithi092 ceph-mon[29572]: 10.7 scrub starts 2024-03-11T21:41:32.115 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:41:31 smithi092 ceph-mon[29572]: 10.7 scrub ok 2024-03-11T21:41:32.116 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:41:31 smithi092 ceph-mon[29572]: 5.c scrub starts 2024-03-11T21:41:32.116 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:41:31 smithi092 ceph-mon[29572]: 5.c scrub ok 2024-03-11T21:41:32.116 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:41:31 smithi092 ceph-mon[29572]: 6.15 scrub starts 2024-03-11T21:41:32.116 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:41:31 smithi092 ceph-mon[29572]: 6.15 scrub ok 2024-03-11T21:41:32.116 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:41:31 smithi092 ceph-mon[29572]: 5.17 scrub starts 2024-03-11T21:41:32.116 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:41:31 smithi092 ceph-mon[29572]: 5.17 scrub ok 2024-03-11T21:41:32.116 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:41:31 smithi092 ceph-mon[29572]: from='client.? v1:172.21.15.92:0/2181317857' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-11T21:41:32.116 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:41:31 smithi092 ceph-mon[29572]: osdmap e376: 8 total, 8 up, 8 in 2024-03-11T21:41:32.116 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:41:31 smithi092 ceph-mon[37563]: 10.4 scrub starts 2024-03-11T21:41:32.116 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:41:31 smithi092 ceph-mon[37563]: 10.6 scrub ok 2024-03-11T21:41:32.116 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:41:31 smithi092 ceph-mon[37563]: 10.4 scrub ok 2024-03-11T21:41:32.116 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:41:31 smithi092 ceph-mon[37563]: 10.3 scrub starts 2024-03-11T21:41:32.116 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:41:31 smithi092 ceph-mon[37563]: 4.c scrub starts 2024-03-11T21:41:32.117 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:41:31 smithi092 ceph-mon[37563]: 4.c scrub ok 2024-03-11T21:41:32.117 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:41:31 smithi092 ceph-mon[37563]: 10.3 scrub ok 2024-03-11T21:41:32.117 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:41:31 smithi092 ceph-mon[37563]: osdmap e375: 8 total, 8 up, 8 in 2024-03-11T21:41:32.117 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:41:31 smithi092 ceph-mon[37563]: 10.7 scrub starts 2024-03-11T21:41:32.117 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:41:31 smithi092 ceph-mon[37563]: 10.7 scrub ok 2024-03-11T21:41:32.117 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:41:31 smithi092 ceph-mon[37563]: 5.c scrub starts 2024-03-11T21:41:32.117 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:41:31 smithi092 ceph-mon[37563]: 5.c scrub ok 2024-03-11T21:41:32.117 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:41:31 smithi092 ceph-mon[37563]: 6.15 scrub starts 2024-03-11T21:41:32.117 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:41:31 smithi092 ceph-mon[37563]: 6.15 scrub ok 2024-03-11T21:41:32.117 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:41:31 smithi092 ceph-mon[37563]: 5.17 scrub starts 2024-03-11T21:41:32.117 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:41:31 smithi092 ceph-mon[37563]: 5.17 scrub ok 2024-03-11T21:41:32.117 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:41:31 smithi092 ceph-mon[37563]: from='client.? v1:172.21.15.92:0/2181317857' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-11T21:41:32.117 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:41:31 smithi092 ceph-mon[37563]: osdmap e376: 8 total, 8 up, 8 in 2024-03-11T21:41:32.211 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:41:32.255 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5, 6, 7] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 4, 5, 6, 7, 3] 2024-03-11T21:41:32.255 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-03-11T21:41:32.255 INFO:tasks.thrashosds.thrasher:thrash_pg_upmap_items 2024-03-11T21:41:32.256 DEBUG:teuthology.orchestra.run.smithi092:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid 00db0016-dfed-11ee-95c8-87774f69a715 -- ceph osd dump -f json-pretty 2024-03-11T21:41:32.263 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:41:32.264 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:41:32.264 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:41:32.365 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:41:32.423 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:41:32.423 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:41:32.424 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:41:32.524 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:41:32.575 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:41:32.575 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:41:32.576 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:41:32.676 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:41:32.729 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:41:32.730 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:41:32.730 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:41:32.831 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:41:32.834 INFO:tasks.radosbench.radosbench.0.smithi092.stdout:2024-03-11T21:41:32.848763+0000 min lat: 0.00121416 max lat: 2.54835 avg lat: 0.00901793 2024-03-11T21:41:32.834 INFO:tasks.radosbench.radosbench.0.smithi092.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-03-11T21:41:32.834 INFO:tasks.radosbench.radosbench.0.smithi092.stdout: 40 16 70947 70931 110.816 13.0625 0.0303402 0.00901793 2024-03-11T21:41:32.864 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:41:32.865 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:41:32.865 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:41:32.870 INFO:teuthology.orchestra.run.smithi092.stderr:noscrub is set 2024-03-11T21:41:32.890 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:41:32 smithi092 ceph-mon[29572]: 5.3 scrub starts 2024-03-11T21:41:32.890 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:41:32 smithi092 ceph-mon[29572]: 5.3 scrub ok 2024-03-11T21:41:32.890 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:41:32 smithi092 ceph-mon[29572]: pgmap v746: 158 pgs: 2 activating, 1 peering, 8 unknown, 147 active+clean; 4.2 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 1.2 KiB/s rd, 78 MiB/s wr, 1.26k op/s 2024-03-11T21:41:32.890 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:41:32 smithi092 ceph-mon[29572]: 4.f scrub starts 2024-03-11T21:41:32.890 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:41:32 smithi092 ceph-mon[29572]: 4.f scrub ok 2024-03-11T21:41:32.890 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:41:32 smithi092 ceph-mon[29572]: 2.1a scrub starts 2024-03-11T21:41:32.890 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:41:32 smithi092 ceph-mon[29572]: 2.1a scrub ok 2024-03-11T21:41:32.890 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:41:32 smithi092 ceph-mon[29572]: from='client.? v1:172.21.15.92:0/1274068360' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-11T21:41:32.890 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:41:32 smithi092 ceph-mon[29572]: from='client.25642 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-11T21:41:32.891 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:41:32 smithi092 ceph-mon[37563]: 5.3 scrub starts 2024-03-11T21:41:32.891 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:41:32 smithi092 ceph-mon[37563]: 5.3 scrub ok 2024-03-11T21:41:32.891 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:41:32 smithi092 ceph-mon[37563]: pgmap v746: 158 pgs: 2 activating, 1 peering, 8 unknown, 147 active+clean; 4.2 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 1.2 KiB/s rd, 78 MiB/s wr, 1.26k op/s 2024-03-11T21:41:32.891 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:41:32 smithi092 ceph-mon[37563]: 4.f scrub starts 2024-03-11T21:41:32.891 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:41:32 smithi092 ceph-mon[37563]: 4.f scrub ok 2024-03-11T21:41:32.891 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:41:32 smithi092 ceph-mon[37563]: 2.1a scrub starts 2024-03-11T21:41:32.891 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:41:32 smithi092 ceph-mon[37563]: 2.1a scrub ok 2024-03-11T21:41:32.891 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:41:32 smithi092 ceph-mon[37563]: from='client.? v1:172.21.15.92:0/1274068360' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-11T21:41:32.891 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:41:32 smithi092 ceph-mon[37563]: from='client.25642 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-11T21:41:32.966 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:41:33.007 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:41:33.007 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:41:33.008 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:41:33.109 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:41:33.153 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:41:33.154 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:41:33.154 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:41:33.255 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:41:33.288 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:41:33.289 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:41:33.289 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:41:33.330 INFO:journalctl@ceph.iscsi.iscsi.a.smithi097.stdout:Mar 11 21:41:32 smithi097 ceph-00db0016-dfed-11ee-95c8-87774f69a715-iscsi-iscsi-a[60943]: debug there is no tcmu-runner data available 2024-03-11T21:41:33.331 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:41:32 smithi097 ceph-mon[33317]: 5.3 scrub starts 2024-03-11T21:41:33.331 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:41:32 smithi097 ceph-mon[33317]: 5.3 scrub ok 2024-03-11T21:41:33.331 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:41:32 smithi097 ceph-mon[33317]: pgmap v746: 158 pgs: 2 activating, 1 peering, 8 unknown, 147 active+clean; 4.2 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 1.2 KiB/s rd, 78 MiB/s wr, 1.26k op/s 2024-03-11T21:41:33.331 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:41:32 smithi097 ceph-mon[33317]: 4.f scrub starts 2024-03-11T21:41:33.331 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:41:32 smithi097 ceph-mon[33317]: 4.f scrub ok 2024-03-11T21:41:33.331 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:41:32 smithi097 ceph-mon[33317]: 2.1a scrub starts 2024-03-11T21:41:33.331 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:41:32 smithi097 ceph-mon[33317]: 2.1a scrub ok 2024-03-11T21:41:33.331 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:41:32 smithi097 ceph-mon[33317]: from='client.? v1:172.21.15.92:0/1274068360' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-11T21:41:33.332 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:41:32 smithi097 ceph-mon[33317]: from='client.25642 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-11T21:41:33.390 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:41:33.408 DEBUG:teuthology.orchestra.run.smithi092:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid 00db0016-dfed-11ee-95c8-87774f69a715 -- ceph daemon osd.2 dump_historic_ops 2024-03-11T21:41:33.462 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:41:33.463 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:41:33.463 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:41:33.564 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:41:33.620 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:41:33.621 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:41:33.621 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:41:33.722 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:41:33.785 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:41:33.786 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:41:33.786 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:41:33.835 INFO:tasks.radosbench.radosbench.0.smithi092.stdout: 41 16 71777 71761 109.378 51.875 0.026991 0.00913711 2024-03-11T21:41:33.887 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:41:33.920 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:41:33.921 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:41:33.921 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:41:34.022 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:41:34.061 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:41:34.062 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:41:34.062 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:41:34.115 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:41:33 smithi092 ceph-mon[29572]: 10.9 deep-scrub ok 2024-03-11T21:41:34.115 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:41:33 smithi092 ceph-mon[29572]: 10.5 deep-scrub ok 2024-03-11T21:41:34.115 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:41:33 smithi092 ceph-mon[29572]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-03-11T21:41:34.115 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:41:33 smithi092 ceph-mon[29572]: from='client.25642 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-03-11T21:41:34.115 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:41:33 smithi092 ceph-mon[29572]: osdmap e377: 8 total, 8 up, 8 in 2024-03-11T21:41:34.115 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:41:33 smithi092 ceph-mon[29572]: from='client.24469 v1:172.21.15.97:0/404208310' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:41:34.115 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:41:33 smithi092 ceph-mon[29572]: osdmap e378: 8 total, 8 up, 8 in 2024-03-11T21:41:34.115 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:41:33 smithi092 ceph-mon[37563]: 10.9 deep-scrub ok 2024-03-11T21:41:34.116 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:41:33 smithi092 ceph-mon[37563]: 10.5 deep-scrub ok 2024-03-11T21:41:34.116 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:41:33 smithi092 ceph-mon[37563]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-03-11T21:41:34.116 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:41:33 smithi092 ceph-mon[37563]: from='client.25642 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-03-11T21:41:34.116 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:41:33 smithi092 ceph-mon[37563]: osdmap e377: 8 total, 8 up, 8 in 2024-03-11T21:41:34.116 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:41:33 smithi092 ceph-mon[37563]: from='client.24469 v1:172.21.15.97:0/404208310' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:41:34.116 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:41:33 smithi092 ceph-mon[37563]: osdmap e378: 8 total, 8 up, 8 in 2024-03-11T21:41:34.164 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:41:34.213 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:41:34.213 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:41:34.214 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:41:34.314 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:41:34.330 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:41:33 smithi097 ceph-mon[33317]: 10.9 deep-scrub ok 2024-03-11T21:41:34.331 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:41:33 smithi097 ceph-mon[33317]: 10.5 deep-scrub ok 2024-03-11T21:41:34.331 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:41:33 smithi097 ceph-mon[33317]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-03-11T21:41:34.331 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:41:33 smithi097 ceph-mon[33317]: from='client.25642 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-03-11T21:41:34.331 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:41:33 smithi097 ceph-mon[33317]: osdmap e377: 8 total, 8 up, 8 in 2024-03-11T21:41:34.331 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:41:33 smithi097 ceph-mon[33317]: from='client.24469 v1:172.21.15.97:0/404208310' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:41:34.331 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:41:33 smithi097 ceph-mon[33317]: osdmap e378: 8 total, 8 up, 8 in 2024-03-11T21:41:34.350 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:41:34.351 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:41:34.351 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:41:34.452 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:41:34.492 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:41:34.493 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:41:34.493 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:41:34.594 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:41:34.652 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:41:34.652 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:41:34.653 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:41:34.690 DEBUG:teuthology.orchestra.run.smithi092:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid 00db0016-dfed-11ee-95c8-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-03-11T21:41:34.746 INFO:teuthology.orchestra.run.smithi092.stderr:Inferring config /var/lib/ceph/00db0016-dfed-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T21:41:34.753 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:41:34.800 INFO:teuthology.orchestra.run.smithi092.stderr:Inferring config /var/lib/ceph/00db0016-dfed-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T21:41:34.835 INFO:tasks.radosbench.radosbench.0.smithi092.stdout: 42 16 72635 72619 108.051 53.625 0.0476917 0.00924757 2024-03-11T21:41:34.841 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:41:34.842 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:41:34.842 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:41:34.943 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:41:35.002 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:41:35.003 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:41:35.003 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:41:35.104 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:41:35.154 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:41:35.155 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:41:35.155 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:41:35.256 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:41:35.301 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:41:35.301 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:41:35.301 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:41:35.330 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:41:34 smithi097 ceph-mon[33317]: pgmap v749: 158 pgs: 145 active+clean, 5 unknown, 1 peering, 1 activating, 3 active+recovery_wait+undersized+degraded+remapped, 2 active+clean+scrubbing+deep, 1 active+recovering+undersized+remapped; 3.8 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 13318/185013 objects degraded (7.198%); 4509/185013 objects misplaced (2.437%); 96 KiB/s, 1 objects/s recovering 2024-03-11T21:41:35.331 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:41:34 smithi097 ceph-mon[33317]: Health check failed: Degraded data redundancy: 13318/185013 objects degraded (7.198%), 3 pgs degraded (PG_DEGRADED) 2024-03-11T21:41:35.365 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:41:34 smithi092 ceph-mon[37563]: pgmap v749: 158 pgs: 145 active+clean, 5 unknown, 1 peering, 1 activating, 3 active+recovery_wait+undersized+degraded+remapped, 2 active+clean+scrubbing+deep, 1 active+recovering+undersized+remapped; 3.8 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 13318/185013 objects degraded (7.198%); 4509/185013 objects misplaced (2.437%); 96 KiB/s, 1 objects/s recovering 2024-03-11T21:41:35.365 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:41:34 smithi092 ceph-mon[37563]: Health check failed: Degraded data redundancy: 13318/185013 objects degraded (7.198%), 3 pgs degraded (PG_DEGRADED) 2024-03-11T21:41:35.365 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:41:34 smithi092 ceph-mon[29572]: pgmap v749: 158 pgs: 145 active+clean, 5 unknown, 1 peering, 1 activating, 3 active+recovery_wait+undersized+degraded+remapped, 2 active+clean+scrubbing+deep, 1 active+recovering+undersized+remapped; 3.8 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 13318/185013 objects degraded (7.198%); 4509/185013 objects misplaced (2.437%); 96 KiB/s, 1 objects/s recovering 2024-03-11T21:41:35.365 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:41:34 smithi092 ceph-mon[29572]: Health check failed: Degraded data redundancy: 13318/185013 objects degraded (7.198%), 3 pgs degraded (PG_DEGRADED) 2024-03-11T21:41:35.403 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:41:35.437 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:41:35.437 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:41:35.438 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:41:35.538 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:41:35.596 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:41:35.596 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:41:35.597 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:41:35.697 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:41:35.726 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:41:35.727 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:41:35.728 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:41:35.828 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:41:35.835 INFO:tasks.radosbench.radosbench.0.smithi092.stdout: 43 16 73500 73484 106.793 54.0625 0.0254386 0.00935892 2024-03-11T21:41:35.865 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:41:35.866 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:41:35.866 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:41:35.967 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:41:36.012 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:41:36.013 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:41:36.013 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:41:36.114 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:41:36.159 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:41:36.159 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:41:36.160 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:41:36.261 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:41:36.293 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:41:36.294 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:41:36.295 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:41:36.395 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:41:36.428 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:41:36.429 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:41:36.429 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:41:36.530 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:41:36.565 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:41:36.566 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:41:36.566 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:41:36.666 DEBUG:teuthology.orchestra.run.smithi092:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid 00db0016-dfed-11ee-95c8-87774f69a715 -- ceph osd set nodeep-scrub 2024-03-11T21:41:36.666 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:41:36.690 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:41:36.691 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:41:36.691 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:41:36.792 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:41:36.838 INFO:tasks.radosbench.radosbench.0.smithi092.stdout: 44 16 74319 74303 105.529 51.1875 0.0159077 0.00947053 2024-03-11T21:41:36.862 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:41:36.862 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:41:36.862 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:41:36.963 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:41:37.012 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:41:37.013 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:41:37.013 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:41:37.069 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:41:36 smithi092 ceph-mon[29572]: pgmap v751: 158 pgs: 146 active+clean, 4 active+recovery_wait+undersized+degraded+remapped, 2 active+clean+scrubbing+deep, 6 active+recovering+undersized+remapped; 4.3 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s; 17932/213087 objects degraded (8.415%); 24128/213087 objects misplaced (11.323%); 4.8 MiB/s, 76 objects/s recovering 2024-03-11T21:41:37.069 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:41:36 smithi092 ceph-mon[37563]: pgmap v751: 158 pgs: 146 active+clean, 4 active+recovery_wait+undersized+degraded+remapped, 2 active+clean+scrubbing+deep, 6 active+recovering+undersized+remapped; 4.3 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s; 17932/213087 objects degraded (8.415%); 24128/213087 objects misplaced (11.323%); 4.8 MiB/s, 76 objects/s recovering 2024-03-11T21:41:37.114 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:41:37.170 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:41:37.170 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:41:37.171 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:41:37.272 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:41:37.305 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:41:37.306 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:41:37.306 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:41:37.330 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:41:36 smithi097 ceph-mon[33317]: pgmap v751: 158 pgs: 146 active+clean, 4 active+recovery_wait+undersized+degraded+remapped, 2 active+clean+scrubbing+deep, 6 active+recovering+undersized+remapped; 4.3 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s; 17932/213087 objects degraded (8.415%); 24128/213087 objects misplaced (11.323%); 4.8 MiB/s, 76 objects/s recovering 2024-03-11T21:41:37.407 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:41:37.441 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:41:37.442 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:41:37.442 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:41:37.543 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:41:37.596 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:41:37.597 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:41:37.597 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:41:37.698 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:41:37.753 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:41:37.753 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:41:37.753 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:41:37.836 INFO:tasks.radosbench.radosbench.0.smithi092.stdout: 45 16 75039 75023 104.184 45 0.00933505 0.00959026 2024-03-11T21:41:37.855 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:41:37.882 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:41:37.883 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:41:37.883 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:41:37.984 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:41:38.055 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:41:38.056 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:41:38.056 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:41:38.157 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:41:38.223 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:41:38.224 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:41:38.224 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:41:38.325 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:41:38.330 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:41:37 smithi097 ceph-mon[33317]: from='mgr.14607 v1:172.21.15.92:0/1702717597' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T21:41:38.330 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:41:37 smithi097 ceph-mon[33317]: from='mgr.14607 ' entity='mgr.y' 2024-03-11T21:41:38.331 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:41:37 smithi097 ceph-mon[33317]: from='mgr.14607 v1:172.21.15.92:0/1702717597' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T21:41:38.350 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:41:38.350 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:41:38.351 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:41:38.365 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:41:37 smithi092 ceph-mon[37563]: from='mgr.14607 v1:172.21.15.92:0/1702717597' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T21:41:38.365 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:41:37 smithi092 ceph-mon[37563]: from='mgr.14607 ' entity='mgr.y' 2024-03-11T21:41:38.365 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:41:37 smithi092 ceph-mon[37563]: from='mgr.14607 v1:172.21.15.92:0/1702717597' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T21:41:38.365 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:41:37 smithi092 ceph-mon[29572]: from='mgr.14607 v1:172.21.15.92:0/1702717597' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T21:41:38.365 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:41:37 smithi092 ceph-mon[29572]: from='mgr.14607 ' entity='mgr.y' 2024-03-11T21:41:38.365 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:41:37 smithi092 ceph-mon[29572]: from='mgr.14607 v1:172.21.15.92:0/1702717597' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T21:41:38.366 INFO:journalctl@ceph.mgr.y.smithi092.stdout:Mar 11 21:41:37 smithi092 ceph-00db0016-dfed-11ee-95c8-87774f69a715-mgr-y[29801]: ::ffff:172.21.15.97 - - [11/Mar/2024:21:41:37] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-11T21:41:38.451 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:41:38.489 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:41:38.490 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:41:38.491 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:41:38.591 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:41:38.623 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:41:38.624 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:41:38.624 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:41:38.725 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:41:38.759 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:41:38.760 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:41:38.760 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:41:38.836 INFO:tasks.radosbench.radosbench.0.smithi092.stdout: 46 16 75752 75736 102.888 44.5625 0.0212881 0.00971366 2024-03-11T21:41:38.861 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:41:38.900 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:41:38.901 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:41:38.901 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:41:39.002 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:41:39.045 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:41:39.045 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:41:39.046 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:41:39.147 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:41:39.166 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:41:38 smithi097 ceph-mon[33317]: pgmap v752: 158 pgs: 147 active+clean, 4 active+recovery_wait+undersized+degraded+remapped, 1 active+clean+scrubbing+deep, 6 active+recovering+undersized+remapped; 4.4 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 853 B/s rd, 0 op/s; 17932/215082 objects degraded (8.337%); 23522/215082 objects misplaced (10.936%); 9.9 MiB/s, 158 objects/s recovering 2024-03-11T21:41:39.191 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:41:39.191 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:41:39.192 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:41:39.292 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:41:39.321 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:41:39.322 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:41:39.322 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:41:39.365 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:41:38 smithi092 ceph-mon[29572]: pgmap v752: 158 pgs: 147 active+clean, 4 active+recovery_wait+undersized+degraded+remapped, 1 active+clean+scrubbing+deep, 6 active+recovering+undersized+remapped; 4.4 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 853 B/s rd, 0 op/s; 17932/215082 objects degraded (8.337%); 23522/215082 objects misplaced (10.936%); 9.9 MiB/s, 158 objects/s recovering 2024-03-11T21:41:39.365 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:41:38 smithi092 ceph-mon[37563]: pgmap v752: 158 pgs: 147 active+clean, 4 active+recovery_wait+undersized+degraded+remapped, 1 active+clean+scrubbing+deep, 6 active+recovering+undersized+remapped; 4.4 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 853 B/s rd, 0 op/s; 17932/215082 objects degraded (8.337%); 23522/215082 objects misplaced (10.936%); 9.9 MiB/s, 158 objects/s recovering 2024-03-11T21:41:39.422 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:41:39.471 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:41:39.472 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:41:39.472 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:41:39.573 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:41:39.617 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:41:39.618 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:41:39.618 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:41:39.719 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:41:39.744 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:41:39.745 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:41:39.746 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:41:39.836 INFO:tasks.radosbench.radosbench.0.smithi092.stdout: 47 16 76509 76493 101.705 47.3125 0.00815159 0.00982722 2024-03-11T21:41:39.846 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:41:39.886 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:41:39.886 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:41:39.887 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:41:39.988 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:41:40.013 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:41:40.014 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:41:40.014 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:41:40.115 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:41:40.156 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:41:40.157 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:41:40.157 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:41:40.258 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:41:40.282 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:41:40.282 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:41:40.283 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:41:40.330 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:41:39 smithi097 ceph-mon[33317]: Health check update: Degraded data redundancy: 17932/224784 objects degraded (7.977%), 4 pgs degraded (PG_DEGRADED) 2024-03-11T21:41:40.369 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:41:39 smithi092 ceph-mon[29572]: Health check update: Degraded data redundancy: 17932/224784 objects degraded (7.977%), 4 pgs degraded (PG_DEGRADED) 2024-03-11T21:41:40.370 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:41:39 smithi092 ceph-mon[37563]: Health check update: Degraded data redundancy: 17932/224784 objects degraded (7.977%), 4 pgs degraded (PG_DEGRADED) 2024-03-11T21:41:40.384 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:41:40.419 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:41:40.420 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:41:40.420 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:41:40.521 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:41:40.560 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:41:40.560 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:41:40.561 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:41:40.661 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:41:40.709 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:41:40.710 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:41:40.710 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:41:40.811 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:41:40.836 INFO:tasks.radosbench.radosbench.0.smithi092.stdout: 48 16 77267 77251 100.573 47.375 0.0231285 0.009937 2024-03-11T21:41:40.853 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:41:40.854 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:41:40.855 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:41:40.956 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:41:40.999 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:41:41.000 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:41:41.000 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:41:41.101 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:41:41.175 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:41:41.175 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:41:41.175 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:41:41.276 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:41:41.329 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:41:41.330 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:41:41.330 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:41:41.331 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:41:40 smithi097 ceph-mon[33317]: pgmap v753: 158 pgs: 148 active+clean, 4 active+recovery_wait+undersized+degraded+remapped, 6 active+recovering+undersized+remapped; 4.6 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 1.3 KiB/s rd, 1 op/s; 17932/224784 objects degraded (7.977%); 22230/224784 objects misplaced (9.889%); 15 MiB/s, 243 objects/s recovering 2024-03-11T21:41:41.365 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:41:40 smithi092 ceph-mon[29572]: pgmap v753: 158 pgs: 148 active+clean, 4 active+recovery_wait+undersized+degraded+remapped, 6 active+recovering+undersized+remapped; 4.6 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 1.3 KiB/s rd, 1 op/s; 17932/224784 objects degraded (7.977%); 22230/224784 objects misplaced (9.889%); 15 MiB/s, 243 objects/s recovering 2024-03-11T21:41:41.365 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:41:40 smithi092 ceph-mon[37563]: pgmap v753: 158 pgs: 148 active+clean, 4 active+recovery_wait+undersized+degraded+remapped, 6 active+recovering+undersized+remapped; 4.6 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 1.3 KiB/s rd, 1 op/s; 17932/224784 objects degraded (7.977%); 22230/224784 objects misplaced (9.889%); 15 MiB/s, 243 objects/s recovering 2024-03-11T21:41:41.431 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:41:41.474 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:41:41.475 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:41:41.476 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:41:41.576 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:41:41.601 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:41:41.602 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:41:41.602 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:41:41.703 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:41:41.736 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:41:41.737 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:41:41.737 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:41:41.836 INFO:tasks.radosbench.radosbench.0.smithi092.stdout: 49 16 78042 78026 99.5094 48.4375 0.0123884 0.0100439 2024-03-11T21:41:41.838 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:41:41.886 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:41:41.886 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:41:41.887 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:41:41.987 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:41:42.027 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:41:42.027 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:41:42.028 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:41:42.128 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:41:42.161 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:41:42.161 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:41:42.162 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:41:42.263 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:41:42.291 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:41:42.291 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:41:42.292 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:41:42.393 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:41:42.429 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:41:42.429 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:41:42.430 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:41:42.530 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:41:42.566 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:41:42.567 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:41:42.567 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:41:42.668 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:41:42.694 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:41:42.695 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:41:42.695 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:41:42.796 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:41:42.836 INFO:tasks.radosbench.radosbench.0.smithi092.stdout: 50 16 78846 78830 98.5242 50.25 0.0129066 0.0101446 2024-03-11T21:41:42.837 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:41:42.838 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:41:42.838 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:41:42.939 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:41:42.982 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:41:42.983 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:41:42.983 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:41:43.084 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:41:43.115 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:41:42 smithi092 ceph-mon[29572]: pgmap v754: 158 pgs: 148 active+clean, 4 active+recovery_wait+undersized+degraded+remapped, 6 active+recovering+undersized+remapped; 4.6 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 1.2 KiB/s rd, 18 MiB/s wr, 292 op/s; 17932/224784 objects degraded (7.977%); 22230/224784 objects misplaced (9.889%); 13 MiB/s, 214 objects/s recovering 2024-03-11T21:41:43.115 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:41:42 smithi092 ceph-mon[37563]: pgmap v754: 158 pgs: 148 active+clean, 4 active+recovery_wait+undersized+degraded+remapped, 6 active+recovering+undersized+remapped; 4.6 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 1.2 KiB/s rd, 18 MiB/s wr, 292 op/s; 17932/224784 objects degraded (7.977%); 22230/224784 objects misplaced (9.889%); 13 MiB/s, 214 objects/s recovering 2024-03-11T21:41:43.126 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:41:43.127 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:41:43.127 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:41:43.228 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:41:43.263 INFO:journalctl@ceph.iscsi.iscsi.a.smithi097.stdout:Mar 11 21:41:42 smithi097 ceph-00db0016-dfed-11ee-95c8-87774f69a715-iscsi-iscsi-a[60943]: debug there is no tcmu-runner data available 2024-03-11T21:41:43.263 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:41:42 smithi097 ceph-mon[33317]: pgmap v754: 158 pgs: 148 active+clean, 4 active+recovery_wait+undersized+degraded+remapped, 6 active+recovering+undersized+remapped; 4.6 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 1.2 KiB/s rd, 18 MiB/s wr, 292 op/s; 17932/224784 objects degraded (7.977%); 22230/224784 objects misplaced (9.889%); 13 MiB/s, 214 objects/s recovering 2024-03-11T21:41:43.272 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:41:43.273 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:41:43.273 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:41:43.374 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:41:43.409 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:41:43.410 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:41:43.410 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:41:43.511 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:41:43.557 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:41:43.557 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:41:43.558 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:41:43.659 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:41:43.686 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:41:43.686 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:41:43.686 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:41:43.787 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:41:43.828 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:41:43.829 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:41:43.829 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:41:43.836 INFO:tasks.radosbench.radosbench.0.smithi092.stdout: 51 16 79621 79605 97.542 48.4375 0.0359298 0.0102455 2024-03-11T21:41:43.930 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:41:43.971 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:41:43.971 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:41:43.972 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:41:44.073 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:41:44.112 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:41:44.113 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:41:44.113 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:41:44.214 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:41:44.256 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:41:44.256 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:41:44.257 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:41:44.331 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:41:43 smithi097 ceph-mon[33317]: from='client.24469 v1:172.21.15.97:0/404208310' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:41:44.331 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:41:43 smithi097 ceph-mon[33317]: from='mgr.14607 ' entity='mgr.y' 2024-03-11T21:41:44.331 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:41:43 smithi097 ceph-mon[33317]: from='mgr.14607 ' entity='mgr.y' 2024-03-11T21:41:44.358 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:41:44.366 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:41:43 smithi092 ceph-mon[29572]: from='client.24469 v1:172.21.15.97:0/404208310' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:41:44.366 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:41:43 smithi092 ceph-mon[29572]: from='mgr.14607 ' entity='mgr.y' 2024-03-11T21:41:44.366 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:41:43 smithi092 ceph-mon[29572]: from='mgr.14607 ' entity='mgr.y' 2024-03-11T21:41:44.367 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:41:43 smithi092 ceph-mon[37563]: from='client.24469 v1:172.21.15.97:0/404208310' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:41:44.367 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:41:43 smithi092 ceph-mon[37563]: from='mgr.14607 ' entity='mgr.y' 2024-03-11T21:41:44.367 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:41:43 smithi092 ceph-mon[37563]: from='mgr.14607 ' entity='mgr.y' 2024-03-11T21:41:44.404 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:41:44.404 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:41:44.404 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:41:44.505 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:41:44.531 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:41:44.532 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:41:44.532 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:41:44.633 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:41:44.680 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:41:44.681 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:41:44.681 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:41:44.782 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:41:44.814 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:41:44.815 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:41:44.815 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:41:44.836 INFO:tasks.radosbench.radosbench.0.smithi092.stdout: 52 16 80360 80344 96.5544 46.1875 0.00673289 0.0103517 2024-03-11T21:41:44.916 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:41:44.954 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:41:44.954 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:41:44.955 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:41:45.056 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:41:45.094 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:41:45.094 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:41:45.095 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:41:45.195 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:41:45.230 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:41:45.231 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:41:45.231 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:41:45.331 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:41:44 smithi097 ceph-mon[33317]: pgmap v755: 158 pgs: 148 active+clean, 4 active+recovery_wait+undersized+degraded+remapped, 6 active+recovering+undersized+remapped; 4.7 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 1023 B/s rd, 33 MiB/s wr, 522 op/s; 17932/233034 objects degraded (7.695%); 21477/233034 objects misplaced (9.216%); 16 MiB/s, 257 objects/s recovering 2024-03-11T21:41:45.331 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:41:44 smithi097 ceph-mon[33317]: purged_snaps scrub starts 2024-03-11T21:41:45.331 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:41:44 smithi097 ceph-mon[33317]: purged_snaps scrub ok 2024-03-11T21:41:45.332 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:41:45.365 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:41:44 smithi092 ceph-mon[29572]: pgmap v755: 158 pgs: 148 active+clean, 4 active+recovery_wait+undersized+degraded+remapped, 6 active+recovering+undersized+remapped; 4.7 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 1023 B/s rd, 33 MiB/s wr, 522 op/s; 17932/233034 objects degraded (7.695%); 21477/233034 objects misplaced (9.216%); 16 MiB/s, 257 objects/s recovering 2024-03-11T21:41:45.365 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:41:45 smithi092 ceph-mon[29572]: purged_snaps scrub starts 2024-03-11T21:41:45.365 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:41:45 smithi092 ceph-mon[29572]: purged_snaps scrub ok 2024-03-11T21:41:45.365 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:41:44 smithi092 ceph-mon[37563]: pgmap v755: 158 pgs: 148 active+clean, 4 active+recovery_wait+undersized+degraded+remapped, 6 active+recovering+undersized+remapped; 4.7 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 1023 B/s rd, 33 MiB/s wr, 522 op/s; 17932/233034 objects degraded (7.695%); 21477/233034 objects misplaced (9.216%); 16 MiB/s, 257 objects/s recovering 2024-03-11T21:41:45.365 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:41:45 smithi092 ceph-mon[37563]: purged_snaps scrub starts 2024-03-11T21:41:45.365 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:41:45 smithi092 ceph-mon[37563]: purged_snaps scrub ok 2024-03-11T21:41:45.376 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:41:45.377 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:41:45.378 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:41:45.478 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:41:45.515 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:41:45.515 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:41:45.516 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:41:45.616 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:41:45.649 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:41:45.650 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:41:45.650 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:41:45.751 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:41:45.775 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:41:45.776 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:41:45.776 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:41:45.837 INFO:tasks.radosbench.radosbench.0.smithi092.stdout: 53 16 81033 81017 95.5262 42.0625 0.116027 0.0104534 2024-03-11T21:41:45.877 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:41:45.928 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:41:45.929 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:41:45.929 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:41:46.030 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:41:46.055 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:41:46.056 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:41:46.056 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:41:46.157 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:41:46.196 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:41:46.196 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:41:46.196 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:41:46.297 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:41:46.330 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:41:46.331 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:41:46 smithi097 ceph-mon[33317]: osdmap e379: 8 total, 8 up, 8 in 2024-03-11T21:41:46.332 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:41:46.332 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:41:46.365 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:41:46 smithi092 ceph-mon[29572]: osdmap e379: 8 total, 8 up, 8 in 2024-03-11T21:41:46.365 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:41:46 smithi092 ceph-mon[37563]: osdmap e379: 8 total, 8 up, 8 in 2024-03-11T21:41:46.433 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:41:46.461 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:41:46.462 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:41:46.462 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:41:46.563 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:41:46.600 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:41:46.601 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:41:46.601 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:41:46.702 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:41:46.728 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:41:46.729 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:41:46.729 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:41:46.830 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:41:46.837 INFO:tasks.radosbench.radosbench.0.smithi092.stdout: 54 16 81604 81588 94.4178 35.6875 0.00482102 0.0105066 2024-03-11T21:41:46.864 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:41:46.865 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:41:46.865 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:41:46.966 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:41:46.989 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:41:46.990 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:41:46.990 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:41:47.091 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:41:47.128 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:41:47.129 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:41:47.129 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:41:47.230 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:41:47.270 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:41:47.271 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:41:47.272 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:41:47.331 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:41:47 smithi097 ceph-mon[33317]: pgmap v757: 158 pgs: 148 active+clean, 4 active+recovery_wait+undersized+degraded+remapped, 6 active+recovering+undersized+remapped; 4.8 GiB data, 19 GiB used, 697 GiB / 715 GiB avail; 1023 B/s rd, 21 MiB/s wr, 338 op/s; 17932/236229 objects degraded (7.591%); 20193/236229 objects misplaced (8.548%); 19 MiB/s, 310 objects/s recovering 2024-03-11T21:41:47.331 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:41:47 smithi097 ceph-mon[33317]: Health check update: Degraded data redundancy: 17932/236229 objects degraded (7.591%), 4 pgs degraded (PG_DEGRADED) 2024-03-11T21:41:47.331 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:41:47 smithi097 ceph-mon[33317]: osdmap e380: 8 total, 8 up, 8 in 2024-03-11T21:41:47.331 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:41:47 smithi097 ceph-mon[33317]: purged_snaps scrub starts 2024-03-11T21:41:47.331 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:41:47 smithi097 ceph-mon[33317]: purged_snaps scrub ok 2024-03-11T21:41:47.365 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:41:47 smithi092 ceph-mon[29572]: pgmap v757: 158 pgs: 148 active+clean, 4 active+recovery_wait+undersized+degraded+remapped, 6 active+recovering+undersized+remapped; 4.8 GiB data, 19 GiB used, 697 GiB / 715 GiB avail; 1023 B/s rd, 21 MiB/s wr, 338 op/s; 17932/236229 objects degraded (7.591%); 20193/236229 objects misplaced (8.548%); 19 MiB/s, 310 objects/s recovering 2024-03-11T21:41:47.365 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:41:47 smithi092 ceph-mon[29572]: Health check update: Degraded data redundancy: 17932/236229 objects degraded (7.591%), 4 pgs degraded (PG_DEGRADED) 2024-03-11T21:41:47.365 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:41:47 smithi092 ceph-mon[29572]: osdmap e380: 8 total, 8 up, 8 in 2024-03-11T21:41:47.365 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:41:47 smithi092 ceph-mon[29572]: purged_snaps scrub starts 2024-03-11T21:41:47.365 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:41:47 smithi092 ceph-mon[29572]: purged_snaps scrub ok 2024-03-11T21:41:47.366 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:41:47 smithi092 ceph-mon[37563]: pgmap v757: 158 pgs: 148 active+clean, 4 active+recovery_wait+undersized+degraded+remapped, 6 active+recovering+undersized+remapped; 4.8 GiB data, 19 GiB used, 697 GiB / 715 GiB avail; 1023 B/s rd, 21 MiB/s wr, 338 op/s; 17932/236229 objects degraded (7.591%); 20193/236229 objects misplaced (8.548%); 19 MiB/s, 310 objects/s recovering 2024-03-11T21:41:47.366 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:41:47 smithi092 ceph-mon[37563]: Health check update: Degraded data redundancy: 17932/236229 objects degraded (7.591%), 4 pgs degraded (PG_DEGRADED) 2024-03-11T21:41:47.366 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:41:47 smithi092 ceph-mon[37563]: osdmap e380: 8 total, 8 up, 8 in 2024-03-11T21:41:47.366 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:41:47 smithi092 ceph-mon[37563]: purged_snaps scrub starts 2024-03-11T21:41:47.366 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:41:47 smithi092 ceph-mon[37563]: purged_snaps scrub ok 2024-03-11T21:41:47.372 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:41:47.414 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:41:47.414 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:41:47.415 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:41:47.516 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:41:47.557 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:41:47.558 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:41:47.558 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:41:47.659 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:41:47.683 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:41:47.684 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:41:47.684 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:41:47.785 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:41:47.829 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:41:47.830 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:41:47.830 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:41:47.837 INFO:tasks.radosbench.radosbench.0.smithi092.stdout: 55 16 82194 82178 93.3716 36.875 0.0153644 0.0107046 2024-03-11T21:41:47.932 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:41:47.959 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:41:47.960 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:41:47.960 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:41:48.061 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:41:48.085 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:41:48.085 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:41:48.086 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:41:48.186 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:41:48.229 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:41:48.230 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:41:48.230 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:41:48.331 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:41:48 smithi097 ceph-mon[33317]: osdmap e381: 8 total, 8 up, 8 in 2024-03-11T21:41:48.331 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:41:48.365 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:41:48 smithi092 ceph-mon[29572]: osdmap e381: 8 total, 8 up, 8 in 2024-03-11T21:41:48.365 INFO:journalctl@ceph.mgr.y.smithi092.stdout:Mar 11 21:41:47 smithi092 ceph-00db0016-dfed-11ee-95c8-87774f69a715-mgr-y[29801]: ::ffff:172.21.15.97 - - [11/Mar/2024:21:41:47] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-11T21:41:48.366 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:41:48 smithi092 ceph-mon[37563]: osdmap e381: 8 total, 8 up, 8 in 2024-03-11T21:41:48.375 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:41:48.376 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:41:48.376 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:41:48.477 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:41:48.503 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:41:48.504 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:41:48.504 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:41:48.605 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:41:48.648 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:41:48.649 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:41:48.649 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:41:48.750 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:41:48.775 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:41:48.776 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:41:48.776 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:41:48.837 INFO:tasks.radosbench.radosbench.0.smithi092.stdout: 56 16 82921 82905 92.5156 45.4375 0.0991417 0.0107946 2024-03-11T21:41:48.877 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:41:48.914 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:41:48.915 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:41:48.916 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:41:49.016 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:41:49.059 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:41:49.059 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:41:49.059 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:41:49.160 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:41:49.208 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:41:49.208 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:41:49.209 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:41:49.309 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:41:49.330 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:41:49 smithi097 ceph-mon[33317]: pgmap v760: 158 pgs: 1 peering, 148 active+clean, 4 active+recovery_wait+undersized+degraded+remapped, 5 active+recovering+undersized+remapped; 4.8 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 852 B/s rd, 46 MiB/s wr, 741 op/s; 17932/238122 objects degraded (7.531%); 19303/238122 objects misplaced (8.106%); 26 MiB/s, 418 objects/s recovering 2024-03-11T21:41:49.331 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:41:49 smithi097 ceph-mon[33317]: osdmap e382: 8 total, 8 up, 8 in 2024-03-11T21:41:49.353 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:41:49.353 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:41:49.354 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:41:49.365 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:41:49 smithi092 ceph-mon[29572]: pgmap v760: 158 pgs: 1 peering, 148 active+clean, 4 active+recovery_wait+undersized+degraded+remapped, 5 active+recovering+undersized+remapped; 4.8 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 852 B/s rd, 46 MiB/s wr, 741 op/s; 17932/238122 objects degraded (7.531%); 19303/238122 objects misplaced (8.106%); 26 MiB/s, 418 objects/s recovering 2024-03-11T21:41:49.365 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:41:49 smithi092 ceph-mon[29572]: osdmap e382: 8 total, 8 up, 8 in 2024-03-11T21:41:49.365 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:41:49 smithi092 ceph-mon[37563]: pgmap v760: 158 pgs: 1 peering, 148 active+clean, 4 active+recovery_wait+undersized+degraded+remapped, 5 active+recovering+undersized+remapped; 4.8 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 852 B/s rd, 46 MiB/s wr, 741 op/s; 17932/238122 objects degraded (7.531%); 19303/238122 objects misplaced (8.106%); 26 MiB/s, 418 objects/s recovering 2024-03-11T21:41:49.365 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:41:49 smithi092 ceph-mon[37563]: osdmap e382: 8 total, 8 up, 8 in 2024-03-11T21:41:49.455 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:41:49.503 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:41:49.503 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:41:49.504 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:41:49.605 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:41:49.637 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:41:49.638 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:41:49.638 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:41:49.739 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:41:49.771 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:41:49.772 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:41:49.772 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:41:49.837 INFO:tasks.radosbench.radosbench.0.smithi092.stdout: 57 16 83662 83646 91.7049 46.3125 0.00801946 0.0108911 2024-03-11T21:41:49.873 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:41:49.911 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:41:49.911 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:41:49.912 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:41:50.012 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:41:50.037 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:41:50.037 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:41:50.037 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:41:50.138 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:41:50.170 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:41:50.171 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:41:50.171 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:41:50.272 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:41:50.315 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:41:50.315 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:41:50.316 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:41:50.417 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:41:50.447 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:41:50.447 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:41:50.448 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:41:50.549 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:41:50.600 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:41:50.601 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:41:50.602 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:41:50.702 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:41:50.747 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:41:50.747 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:41:50.748 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:41:50.837 INFO:tasks.radosbench.radosbench.0.smithi092.stdout: 58 16 84418 84402 90.9384 47.25 0.00321235 0.010984 2024-03-11T21:41:50.849 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:41:50.873 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:41:50.874 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:41:50.874 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:41:50.975 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:41:51.005 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:41:51.006 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:41:51.006 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:41:51.107 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:41:51.141 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:41:51.142 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:41:51.142 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:41:51.245 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:41:51.288 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:41:51.288 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:41:51.288 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:41:51.331 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:41:51 smithi097 ceph-mon[33317]: pgmap v762: 158 pgs: 1 peering, 148 active+clean, 4 active+recovery_wait+undersized+degraded+remapped, 5 active+recovering+undersized+remapped; 5.0 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 1.2 KiB/s rd, 47 MiB/s wr, 749 op/s; 17932/245841 objects degraded (7.294%); 18504/245841 objects misplaced (7.527%); 21 MiB/s, 328 objects/s recovering 2024-03-11T21:41:51.365 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:41:51 smithi092 ceph-mon[29572]: pgmap v762: 158 pgs: 1 peering, 148 active+clean, 4 active+recovery_wait+undersized+degraded+remapped, 5 active+recovering+undersized+remapped; 5.0 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 1.2 KiB/s rd, 47 MiB/s wr, 749 op/s; 17932/245841 objects degraded (7.294%); 18504/245841 objects misplaced (7.527%); 21 MiB/s, 328 objects/s recovering 2024-03-11T21:41:51.365 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:41:51 smithi092 ceph-mon[37563]: pgmap v762: 158 pgs: 1 peering, 148 active+clean, 4 active+recovery_wait+undersized+degraded+remapped, 5 active+recovering+undersized+remapped; 5.0 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 1.2 KiB/s rd, 47 MiB/s wr, 749 op/s; 17932/245841 objects degraded (7.294%); 18504/245841 objects misplaced (7.527%); 21 MiB/s, 328 objects/s recovering 2024-03-11T21:41:51.389 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:41:51.436 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:41:51.437 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:41:51.437 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:41:51.538 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:41:51.574 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:41:51.575 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:41:51.575 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:41:51.676 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:41:51.713 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:41:51.714 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:41:51.714 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:41:51.815 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:41:51.838 INFO:tasks.radosbench.radosbench.0.smithi092.stdout: 59 16 85129 85113 90.1492 44.4375 0.0182306 0.0110863 2024-03-11T21:41:51.854 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:41:51.855 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:41:51.855 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:41:51.956 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:41:51.993 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:41:51.994 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:41:51.994 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:41:52.095 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:41:52.132 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:41:52.133 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:41:52.133 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:41:52.234 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:41:52.271 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:41:52.271 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:41:52.272 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:41:52.372 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:41:52.418 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:41:52.419 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:41:52.419 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:41:52.494 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:41:52 smithi092 ceph-mon[29572]: Health check update: Degraded data redundancy: 17932/246402 objects degraded (7.278%), 4 pgs degraded (PG_DEGRADED) 2024-03-11T21:41:52.494 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:41:52 smithi092 ceph-mon[37563]: Health check update: Degraded data redundancy: 17932/246402 objects degraded (7.278%), 4 pgs degraded (PG_DEGRADED) 2024-03-11T21:41:52.520 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:41:52.558 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:41:52.559 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:41:52.559 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:41:52.581 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:41:52 smithi097 ceph-mon[33317]: Health check update: Degraded data redundancy: 17932/246402 objects degraded (7.278%), 4 pgs degraded (PG_DEGRADED) 2024-03-11T21:41:52.660 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:41:52.690 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:41:52.691 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:41:52.691 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:41:52.792 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:41:52.823 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:41:52.823 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:41:52.823 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:41:52.838 INFO:tasks.radosbench.radosbench.0.smithi092.stdout:2024-03-11T21:41:52.852279+0000 min lat: 0.00121416 max lat: 2.54835 avg lat: 0.0111643 2024-03-11T21:41:52.838 INFO:tasks.radosbench.radosbench.0.smithi092.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-03-11T21:41:52.838 INFO:tasks.radosbench.radosbench.0.smithi092.stdout: 60 16 85975 85959 89.528 52.875 0.00455451 0.0111643 2024-03-11T21:41:52.924 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:41:52.947 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:41:52.948 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:41:52.948 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:41:53.049 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:41:53.083 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:41:53.083 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:41:53.083 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:41:53.184 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:41:53.210 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:41:53.210 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:41:53.211 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:41:53.311 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:41:53.331 INFO:journalctl@ceph.iscsi.iscsi.a.smithi097.stdout:Mar 11 21:41:52 smithi097 ceph-00db0016-dfed-11ee-95c8-87774f69a715-iscsi-iscsi-a[60943]: debug there is no tcmu-runner data available 2024-03-11T21:41:53.332 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:41:53 smithi097 ceph-mon[33317]: pgmap v763: 158 pgs: 1 peering, 148 active+clean, 4 active+recovery_wait+undersized+degraded+remapped, 5 active+recovering+undersized+remapped; 5.0 GiB data, 20 GiB used, 696 GiB / 715 GiB avail; 853 B/s rd, 35 MiB/s wr, 565 op/s; 17932/246402 objects degraded (7.278%); 17868/246402 objects misplaced (7.252%); 19 MiB/s, 310 objects/s recovering 2024-03-11T21:41:53.332 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:41:53 smithi097 ceph-mon[33317]: from='mgr.14607 v1:172.21.15.92:0/1702717597' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T21:41:53.342 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:41:53.342 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:41:53.343 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:41:53.444 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:41:53.485 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:41:53.486 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:41:53.487 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:41:53.588 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:41:53.611 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:41:53.611 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:41:53.611 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:41:53.615 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:41:53 smithi092 ceph-mon[29572]: pgmap v763: 158 pgs: 1 peering, 148 active+clean, 4 active+recovery_wait+undersized+degraded+remapped, 5 active+recovering+undersized+remapped; 5.0 GiB data, 20 GiB used, 696 GiB / 715 GiB avail; 853 B/s rd, 35 MiB/s wr, 565 op/s; 17932/246402 objects degraded (7.278%); 17868/246402 objects misplaced (7.252%); 19 MiB/s, 310 objects/s recovering 2024-03-11T21:41:53.615 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:41:53 smithi092 ceph-mon[29572]: from='mgr.14607 v1:172.21.15.92:0/1702717597' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T21:41:53.615 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:41:53 smithi092 ceph-mon[37563]: pgmap v763: 158 pgs: 1 peering, 148 active+clean, 4 active+recovery_wait+undersized+degraded+remapped, 5 active+recovering+undersized+remapped; 5.0 GiB data, 20 GiB used, 696 GiB / 715 GiB avail; 853 B/s rd, 35 MiB/s wr, 565 op/s; 17932/246402 objects degraded (7.278%); 17868/246402 objects misplaced (7.252%); 19 MiB/s, 310 objects/s recovering 2024-03-11T21:41:53.615 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:41:53 smithi092 ceph-mon[37563]: from='mgr.14607 v1:172.21.15.92:0/1702717597' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T21:41:53.712 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:41:53.754 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:41:53.755 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:41:53.755 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:41:53.838 INFO:tasks.radosbench.radosbench.0.smithi092.stdout: 61 16 86756 86740 88.8604 48.8125 0.0170759 0.0112477 2024-03-11T21:41:53.856 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:41:53.898 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:41:53.899 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:41:53.900 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:41:54.000 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:41:54.032 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:41:54.033 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:41:54.033 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:41:54.124 INFO:teuthology.orchestra.run.smithi092.stderr:Inferring config /var/lib/ceph/00db0016-dfed-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T21:41:54.135 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:41:54.199 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:41:54.200 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:41:54.200 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:41:54.301 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:41:54.331 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:41:54.332 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:41:54.332 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:41:54.433 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:41:54.472 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:41:54.472 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:41:54.473 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:41:54.574 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:41:54.580 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:41:54 smithi097 ceph-mon[33317]: purged_snaps scrub starts 2024-03-11T21:41:54.581 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:41:54 smithi097 ceph-mon[33317]: purged_snaps scrub ok 2024-03-11T21:41:54.581 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:41:54 smithi097 ceph-mon[33317]: from='client.24469 v1:172.21.15.97:0/404208310' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:41:54.581 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:41:54 smithi097 ceph-mon[33317]: osdmap e383: 8 total, 8 up, 8 in 2024-03-11T21:41:54.581 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:41:54 smithi097 ceph-mon[33317]: osdmap e384: 8 total, 8 up, 8 in 2024-03-11T21:41:54.597 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:41:54.598 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:41:54.598 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:41:54.615 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:41:54 smithi092 ceph-mon[29572]: purged_snaps scrub starts 2024-03-11T21:41:54.615 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:41:54 smithi092 ceph-mon[29572]: purged_snaps scrub ok 2024-03-11T21:41:54.615 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:41:54 smithi092 ceph-mon[29572]: from='client.24469 v1:172.21.15.97:0/404208310' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:41:54.615 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:41:54 smithi092 ceph-mon[29572]: osdmap e383: 8 total, 8 up, 8 in 2024-03-11T21:41:54.615 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:41:54 smithi092 ceph-mon[29572]: osdmap e384: 8 total, 8 up, 8 in 2024-03-11T21:41:54.615 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:41:54 smithi092 ceph-mon[37563]: purged_snaps scrub starts 2024-03-11T21:41:54.615 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:41:54 smithi092 ceph-mon[37563]: purged_snaps scrub ok 2024-03-11T21:41:54.616 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:41:54 smithi092 ceph-mon[37563]: from='client.24469 v1:172.21.15.97:0/404208310' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:41:54.616 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:41:54 smithi092 ceph-mon[37563]: osdmap e383: 8 total, 8 up, 8 in 2024-03-11T21:41:54.616 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:41:54 smithi092 ceph-mon[37563]: osdmap e384: 8 total, 8 up, 8 in 2024-03-11T21:41:54.699 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:41:54.739 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:41:54.740 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:41:54.740 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:41:54.838 INFO:tasks.radosbench.radosbench.0.smithi092.stdout: 62 16 87541 87525 88.2185 49.0625 0.00835576 0.0113308 2024-03-11T21:41:54.841 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:41:54.871 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:41:54.871 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:41:54.872 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:41:54.973 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:41:55.011 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:41:55.012 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:41:55.012 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:41:55.113 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:41:55.161 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:41:55.162 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:41:55.163 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:41:55.263 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:41:55.310 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:41:55.311 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:41:55.311 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:41:55.365 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:41:55 smithi092 ceph-mon[29572]: pgmap v765: 158 pgs: 149 active+clean, 4 active+recovery_wait+undersized+degraded+remapped, 5 active+recovering+undersized+remapped; 5.2 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 815 B/s rd, 54 MiB/s wr, 864 op/s; 17932/254397 objects degraded (7.049%); 17868/254397 objects misplaced (7.024%); 9.7 MiB/s, 155 objects/s recovering 2024-03-11T21:41:55.365 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:41:55 smithi092 ceph-mon[37563]: pgmap v765: 158 pgs: 149 active+clean, 4 active+recovery_wait+undersized+degraded+remapped, 5 active+recovering+undersized+remapped; 5.2 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 815 B/s rd, 54 MiB/s wr, 864 op/s; 17932/254397 objects degraded (7.049%); 17868/254397 objects misplaced (7.024%); 9.7 MiB/s, 155 objects/s recovering 2024-03-11T21:41:55.412 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:41:55.437 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:41:55.438 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:41:55.438 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:41:55.539 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:41:55.581 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:41:55 smithi097 ceph-mon[33317]: pgmap v765: 158 pgs: 149 active+clean, 4 active+recovery_wait+undersized+degraded+remapped, 5 active+recovering+undersized+remapped; 5.2 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 815 B/s rd, 54 MiB/s wr, 864 op/s; 17932/254397 objects degraded (7.049%); 17868/254397 objects misplaced (7.024%); 9.7 MiB/s, 155 objects/s recovering 2024-03-11T21:41:55.583 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:41:55.584 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:41:55.584 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:41:55.685 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:41:55.729 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:41:55.730 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:41:55.730 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:41:55.831 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:41:55.838 INFO:tasks.radosbench.radosbench.0.smithi092.stdout: 63 16 88347 88331 87.6178 50.375 0.0338433 0.011408 2024-03-11T21:41:55.859 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:41:55.860 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:41:55.860 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:41:55.961 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:41:55.964 INFO:teuthology.orchestra.run.smithi092.stderr:Inferring config /var/lib/ceph/00db0016-dfed-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T21:41:56.041 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:41:56.041 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:41:56.041 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:41:56.142 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:41:56.191 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:41:56.191 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:41:56.192 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:41:56.292 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:41:56.324 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:41:56.324 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:41:56.325 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:41:56.425 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:41:56.457 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:41:56.458 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:41:56.458 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:41:56.559 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:41:56.588 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:41:56.588 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:41:56.589 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:41:56.690 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:41:56.714 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:41:56.715 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:41:56.715 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:41:56.816 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:41:56.838 INFO:tasks.radosbench.radosbench.0.smithi092.stdout: 64 15 89170 89155 87.0534 51.5 0.0134055 0.011482 2024-03-11T21:41:56.850 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:41:56.851 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:41:56.851 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:41:56.952 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:41:56.994 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:41:56.995 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:41:56.996 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:41:57.096 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:41:57.125 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:41:57.125 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:41:57.126 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:41:57.227 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:41:57.258 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:41:57.258 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:41:57.259 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:41:57.360 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:41:57.365 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:41:57 smithi092 ceph-mon[29572]: pgmap v767: 158 pgs: 149 active+clean, 4 active+recovery_wait+undersized+degraded+remapped, 5 active+recovering+undersized+remapped; 5.2 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 737 B/s rd, 41 MiB/s wr, 662 op/s; 17932/257775 objects degraded (6.956%); 16370/257775 objects misplaced (6.350%); 18 MiB/s, 291 objects/s recovering 2024-03-11T21:41:57.365 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:41:57 smithi092 ceph-mon[37563]: pgmap v767: 158 pgs: 149 active+clean, 4 active+recovery_wait+undersized+degraded+remapped, 5 active+recovering+undersized+remapped; 5.2 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 737 B/s rd, 41 MiB/s wr, 662 op/s; 17932/257775 objects degraded (6.956%); 16370/257775 objects misplaced (6.350%); 18 MiB/s, 291 objects/s recovering 2024-03-11T21:41:57.383 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:41:57.383 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:41:57.384 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:41:57.485 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:41:57.523 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:41:57.524 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:41:57.524 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:41:57.581 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:41:57 smithi097 ceph-mon[33317]: pgmap v767: 158 pgs: 149 active+clean, 4 active+recovery_wait+undersized+degraded+remapped, 5 active+recovering+undersized+remapped; 5.2 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 737 B/s rd, 41 MiB/s wr, 662 op/s; 17932/257775 objects degraded (6.956%); 16370/257775 objects misplaced (6.350%); 18 MiB/s, 291 objects/s recovering 2024-03-11T21:41:57.625 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:41:57.667 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:41:57.668 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:41:57.668 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:41:57.769 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:41:57.810 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:41:57.811 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:41:57.811 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:41:57.839 INFO:tasks.radosbench.radosbench.0.smithi092.stdout: 65 16 89987 89971 86.4987 51 0.00840944 0.0115544 2024-03-11T21:41:57.913 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:41:57.953 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:41:57.954 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:41:57.954 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:41:58.055 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:41:58.090 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:41:58.090 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:41:58.090 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:41:58.191 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:41:58.233 INFO:journalctl@ceph.mgr.y.smithi092.stdout:Mar 11 21:41:57 smithi092 ceph-00db0016-dfed-11ee-95c8-87774f69a715-mgr-y[29801]: ::ffff:172.21.15.97 - - [11/Mar/2024:21:41:57] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-11T21:41:58.235 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:41:58.235 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:41:58.236 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:41:58.337 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:41:58.357 INFO:teuthology.orchestra.run.smithi092.stdout: 2024-03-11T21:41:58.358 INFO:teuthology.orchestra.run.smithi092.stdout:{ 2024-03-11T21:41:58.358 INFO:teuthology.orchestra.run.smithi092.stdout: "epoch": 384, 2024-03-11T21:41:58.358 INFO:teuthology.orchestra.run.smithi092.stdout: "fsid": "00db0016-dfed-11ee-95c8-87774f69a715", 2024-03-11T21:41:58.358 INFO:teuthology.orchestra.run.smithi092.stdout: "created": "2024-03-11T21:20:10.488059+0000", 2024-03-11T21:41:58.358 INFO:teuthology.orchestra.run.smithi092.stdout: "modified": "2024-03-11T21:41:53.407479+0000", 2024-03-11T21:41:58.358 INFO:teuthology.orchestra.run.smithi092.stdout: "last_up_change": "2024-03-11T21:36:34.026798+0000", 2024-03-11T21:41:58.358 INFO:teuthology.orchestra.run.smithi092.stdout: "last_in_change": "2024-03-11T21:25:08.189557+0000", 2024-03-11T21:41:58.358 INFO:teuthology.orchestra.run.smithi092.stdout: "flags": "noscrub,sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit", 2024-03-11T21:41:58.358 INFO:teuthology.orchestra.run.smithi092.stdout: "flags_num": 5801984, 2024-03-11T21:41:58.358 INFO:teuthology.orchestra.run.smithi092.stdout: "flags_set": [ 2024-03-11T21:41:58.358 INFO:teuthology.orchestra.run.smithi092.stdout: "noscrub", 2024-03-11T21:41:58.359 INFO:teuthology.orchestra.run.smithi092.stdout: "pglog_hardlimit", 2024-03-11T21:41:58.359 INFO:teuthology.orchestra.run.smithi092.stdout: "purged_snapdirs", 2024-03-11T21:41:58.359 INFO:teuthology.orchestra.run.smithi092.stdout: "recovery_deletes", 2024-03-11T21:41:58.359 INFO:teuthology.orchestra.run.smithi092.stdout: "sortbitwise" 2024-03-11T21:41:58.359 INFO:teuthology.orchestra.run.smithi092.stdout: ], 2024-03-11T21:41:58.359 INFO:teuthology.orchestra.run.smithi092.stdout: "crush_version": 19, 2024-03-11T21:41:58.359 INFO:teuthology.orchestra.run.smithi092.stdout: "full_ratio": 0.94999998807907104, 2024-03-11T21:41:58.359 INFO:teuthology.orchestra.run.smithi092.stdout: "backfillfull_ratio": 0.89999997615814209, 2024-03-11T21:41:58.359 INFO:teuthology.orchestra.run.smithi092.stdout: "nearfull_ratio": 0.85000002384185791, 2024-03-11T21:41:58.359 INFO:teuthology.orchestra.run.smithi092.stdout: "cluster_snapshot": "", 2024-03-11T21:41:58.359 INFO:teuthology.orchestra.run.smithi092.stdout: "pool_max": 10, 2024-03-11T21:41:58.359 INFO:teuthology.orchestra.run.smithi092.stdout: "max_osd": 8, 2024-03-11T21:41:58.359 INFO:teuthology.orchestra.run.smithi092.stdout: "require_min_compat_client": "luminous", 2024-03-11T21:41:58.359 INFO:teuthology.orchestra.run.smithi092.stdout: "min_compat_client": "luminous", 2024-03-11T21:41:58.360 INFO:teuthology.orchestra.run.smithi092.stdout: "require_osd_release": "squid", 2024-03-11T21:41:58.360 INFO:teuthology.orchestra.run.smithi092.stdout: "allow_crimson": false, 2024-03-11T21:41:58.360 INFO:teuthology.orchestra.run.smithi092.stdout: "pools": [ 2024-03-11T21:41:58.360 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:41:58.360 INFO:teuthology.orchestra.run.smithi092.stdout: "pool": 1, 2024-03-11T21:41:58.360 INFO:teuthology.orchestra.run.smithi092.stdout: "pool_name": ".mgr", 2024-03-11T21:41:58.360 INFO:teuthology.orchestra.run.smithi092.stdout: "create_time": "2024-03-11T21:23:23.805029+0000", 2024-03-11T21:41:58.360 INFO:teuthology.orchestra.run.smithi092.stdout: "flags": 1, 2024-03-11T21:41:58.360 INFO:teuthology.orchestra.run.smithi092.stdout: "flags_names": "hashpspool", 2024-03-11T21:41:58.360 INFO:teuthology.orchestra.run.smithi092.stdout: "type": 1, 2024-03-11T21:41:58.360 INFO:teuthology.orchestra.run.smithi092.stdout: "size": 3, 2024-03-11T21:41:58.360 INFO:teuthology.orchestra.run.smithi092.stdout: "min_size": 2, 2024-03-11T21:41:58.360 INFO:teuthology.orchestra.run.smithi092.stdout: "crush_rule": 0, 2024-03-11T21:41:58.361 INFO:teuthology.orchestra.run.smithi092.stdout: "peering_crush_bucket_count": 0, 2024-03-11T21:41:58.361 INFO:teuthology.orchestra.run.smithi092.stdout: "peering_crush_bucket_target": 0, 2024-03-11T21:41:58.361 INFO:teuthology.orchestra.run.smithi092.stdout: "peering_crush_bucket_barrier": 0, 2024-03-11T21:41:58.361 INFO:teuthology.orchestra.run.smithi092.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-03-11T21:41:58.361 INFO:teuthology.orchestra.run.smithi092.stdout: "object_hash": 2, 2024-03-11T21:41:58.361 INFO:teuthology.orchestra.run.smithi092.stdout: "pg_autoscale_mode": "off", 2024-03-11T21:41:58.361 INFO:teuthology.orchestra.run.smithi092.stdout: "pg_num": 1, 2024-03-11T21:41:58.361 INFO:teuthology.orchestra.run.smithi092.stdout: "pg_placement_num": 1, 2024-03-11T21:41:58.361 INFO:teuthology.orchestra.run.smithi092.stdout: "pg_placement_num_target": 1, 2024-03-11T21:41:58.361 INFO:teuthology.orchestra.run.smithi092.stdout: "pg_num_target": 1, 2024-03-11T21:41:58.361 INFO:teuthology.orchestra.run.smithi092.stdout: "pg_num_pending": 1, 2024-03-11T21:41:58.361 INFO:teuthology.orchestra.run.smithi092.stdout: "last_pg_merge_meta": { 2024-03-11T21:41:58.361 INFO:teuthology.orchestra.run.smithi092.stdout: "source_pgid": "0.0", 2024-03-11T21:41:58.362 INFO:teuthology.orchestra.run.smithi092.stdout: "ready_epoch": 0, 2024-03-11T21:41:58.362 INFO:teuthology.orchestra.run.smithi092.stdout: "last_epoch_started": 0, 2024-03-11T21:41:58.362 INFO:teuthology.orchestra.run.smithi092.stdout: "last_epoch_clean": 0, 2024-03-11T21:41:58.362 INFO:teuthology.orchestra.run.smithi092.stdout: "source_version": "0'0", 2024-03-11T21:41:58.362 INFO:teuthology.orchestra.run.smithi092.stdout: "target_version": "0'0" 2024-03-11T21:41:58.362 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:41:58.362 INFO:teuthology.orchestra.run.smithi092.stdout: "last_change": "121", 2024-03-11T21:41:58.362 INFO:teuthology.orchestra.run.smithi092.stdout: "last_force_op_resend": "0", 2024-03-11T21:41:58.362 INFO:teuthology.orchestra.run.smithi092.stdout: "last_force_op_resend_prenautilus": "0", 2024-03-11T21:41:58.362 INFO:teuthology.orchestra.run.smithi092.stdout: "last_force_op_resend_preluminous": "0", 2024-03-11T21:41:58.362 INFO:teuthology.orchestra.run.smithi092.stdout: "auid": 0, 2024-03-11T21:41:58.362 INFO:teuthology.orchestra.run.smithi092.stdout: "snap_mode": "selfmanaged", 2024-03-11T21:41:58.362 INFO:teuthology.orchestra.run.smithi092.stdout: "snap_seq": 0, 2024-03-11T21:41:58.363 INFO:teuthology.orchestra.run.smithi092.stdout: "snap_epoch": 0, 2024-03-11T21:41:58.363 INFO:teuthology.orchestra.run.smithi092.stdout: "pool_snaps": [], 2024-03-11T21:41:58.363 INFO:teuthology.orchestra.run.smithi092.stdout: "removed_snaps": "[]", 2024-03-11T21:41:58.363 INFO:teuthology.orchestra.run.smithi092.stdout: "quota_max_bytes": 0, 2024-03-11T21:41:58.363 INFO:teuthology.orchestra.run.smithi092.stdout: "quota_max_objects": 0, 2024-03-11T21:41:58.363 INFO:teuthology.orchestra.run.smithi092.stdout: "tiers": [], 2024-03-11T21:41:58.363 INFO:teuthology.orchestra.run.smithi092.stdout: "tier_of": -1, 2024-03-11T21:41:58.363 INFO:teuthology.orchestra.run.smithi092.stdout: "read_tier": -1, 2024-03-11T21:41:58.363 INFO:teuthology.orchestra.run.smithi092.stdout: "write_tier": -1, 2024-03-11T21:41:58.363 INFO:teuthology.orchestra.run.smithi092.stdout: "cache_mode": "none", 2024-03-11T21:41:58.363 INFO:teuthology.orchestra.run.smithi092.stdout: "target_max_bytes": 0, 2024-03-11T21:41:58.363 INFO:teuthology.orchestra.run.smithi092.stdout: "target_max_objects": 0, 2024-03-11T21:41:58.363 INFO:teuthology.orchestra.run.smithi092.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-03-11T21:41:58.364 INFO:teuthology.orchestra.run.smithi092.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-03-11T21:41:58.364 INFO:teuthology.orchestra.run.smithi092.stdout: "cache_target_full_ratio_micro": 800000, 2024-03-11T21:41:58.364 INFO:teuthology.orchestra.run.smithi092.stdout: "cache_min_flush_age": 0, 2024-03-11T21:41:58.364 INFO:teuthology.orchestra.run.smithi092.stdout: "cache_min_evict_age": 0, 2024-03-11T21:41:58.364 INFO:teuthology.orchestra.run.smithi092.stdout: "erasure_code_profile": "", 2024-03-11T21:41:58.364 INFO:teuthology.orchestra.run.smithi092.stdout: "hit_set_params": { 2024-03-11T21:41:58.364 INFO:teuthology.orchestra.run.smithi092.stdout: "type": "none" 2024-03-11T21:41:58.364 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:41:58.364 INFO:teuthology.orchestra.run.smithi092.stdout: "hit_set_period": 0, 2024-03-11T21:41:58.364 INFO:teuthology.orchestra.run.smithi092.stdout: "hit_set_count": 0, 2024-03-11T21:41:58.364 INFO:teuthology.orchestra.run.smithi092.stdout: "use_gmt_hitset": true, 2024-03-11T21:41:58.364 INFO:teuthology.orchestra.run.smithi092.stdout: "min_read_recency_for_promote": 0, 2024-03-11T21:41:58.364 INFO:teuthology.orchestra.run.smithi092.stdout: "min_write_recency_for_promote": 0, 2024-03-11T21:41:58.364 INFO:teuthology.orchestra.run.smithi092.stdout: "hit_set_grade_decay_rate": 0, 2024-03-11T21:41:58.365 INFO:teuthology.orchestra.run.smithi092.stdout: "hit_set_search_last_n": 0, 2024-03-11T21:41:58.365 INFO:teuthology.orchestra.run.smithi092.stdout: "grade_table": [], 2024-03-11T21:41:58.365 INFO:teuthology.orchestra.run.smithi092.stdout: "stripe_width": 0, 2024-03-11T21:41:58.365 INFO:teuthology.orchestra.run.smithi092.stdout: "expected_num_objects": 0, 2024-03-11T21:41:58.365 INFO:teuthology.orchestra.run.smithi092.stdout: "fast_read": false, 2024-03-11T21:41:58.365 INFO:teuthology.orchestra.run.smithi092.stdout: "options": { 2024-03-11T21:41:58.365 INFO:teuthology.orchestra.run.smithi092.stdout: "pg_num_max": 32, 2024-03-11T21:41:58.365 INFO:teuthology.orchestra.run.smithi092.stdout: "pg_num_min": 1 2024-03-11T21:41:58.365 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:41:58.365 INFO:teuthology.orchestra.run.smithi092.stdout: "application_metadata": { 2024-03-11T21:41:58.365 INFO:teuthology.orchestra.run.smithi092.stdout: "mgr": {} 2024-03-11T21:41:58.365 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:41:58.365 INFO:teuthology.orchestra.run.smithi092.stdout: "read_balance": { 2024-03-11T21:41:58.365 INFO:teuthology.orchestra.run.smithi092.stdout: "score_type": "Fair distribution", 2024-03-11T21:41:58.365 INFO:teuthology.orchestra.run.smithi092.stdout: "score_acting": 7.8899998664855957, 2024-03-11T21:41:58.366 INFO:teuthology.orchestra.run.smithi092.stdout: "score_stable": 7.8899998664855957, 2024-03-11T21:41:58.366 INFO:teuthology.orchestra.run.smithi092.stdout: "optimal_score": 0.37999999523162842, 2024-03-11T21:41:58.366 INFO:teuthology.orchestra.run.smithi092.stdout: "raw_score_acting": 3, 2024-03-11T21:41:58.366 INFO:teuthology.orchestra.run.smithi092.stdout: "raw_score_stable": 3, 2024-03-11T21:41:58.366 INFO:teuthology.orchestra.run.smithi092.stdout: "primary_affinity_weighted": 1, 2024-03-11T21:41:58.366 INFO:teuthology.orchestra.run.smithi092.stdout: "average_primary_affinity": 1, 2024-03-11T21:41:58.366 INFO:teuthology.orchestra.run.smithi092.stdout: "average_primary_affinity_weighted": 1 2024-03-11T21:41:58.366 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-03-11T21:41:58.366 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:41:58.366 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:41:58.366 INFO:teuthology.orchestra.run.smithi092.stdout: "pool": 2, 2024-03-11T21:41:58.366 INFO:teuthology.orchestra.run.smithi092.stdout: "pool_name": ".rgw.root", 2024-03-11T21:41:58.366 INFO:teuthology.orchestra.run.smithi092.stdout: "create_time": "2024-03-11T21:25:37.044504+0000", 2024-03-11T21:41:58.367 INFO:teuthology.orchestra.run.smithi092.stdout: "flags": 1, 2024-03-11T21:41:58.367 INFO:teuthology.orchestra.run.smithi092.stdout: "flags_names": "hashpspool", 2024-03-11T21:41:58.367 INFO:teuthology.orchestra.run.smithi092.stdout: "type": 1, 2024-03-11T21:41:58.367 INFO:teuthology.orchestra.run.smithi092.stdout: "size": 3, 2024-03-11T21:41:58.367 INFO:teuthology.orchestra.run.smithi092.stdout: "min_size": 2, 2024-03-11T21:41:58.367 INFO:teuthology.orchestra.run.smithi092.stdout: "crush_rule": 0, 2024-03-11T21:41:58.367 INFO:teuthology.orchestra.run.smithi092.stdout: "peering_crush_bucket_count": 0, 2024-03-11T21:41:58.367 INFO:teuthology.orchestra.run.smithi092.stdout: "peering_crush_bucket_target": 0, 2024-03-11T21:41:58.367 INFO:teuthology.orchestra.run.smithi092.stdout: "peering_crush_bucket_barrier": 0, 2024-03-11T21:41:58.367 INFO:teuthology.orchestra.run.smithi092.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-03-11T21:41:58.367 INFO:teuthology.orchestra.run.smithi092.stdout: "object_hash": 2, 2024-03-11T21:41:58.367 INFO:teuthology.orchestra.run.smithi092.stdout: "pg_autoscale_mode": "off", 2024-03-11T21:41:58.367 INFO:teuthology.orchestra.run.smithi092.stdout: "pg_num": 32, 2024-03-11T21:41:58.367 INFO:teuthology.orchestra.run.smithi092.stdout: "pg_placement_num": 32, 2024-03-11T21:41:58.368 INFO:teuthology.orchestra.run.smithi092.stdout: "pg_placement_num_target": 32, 2024-03-11T21:41:58.368 INFO:teuthology.orchestra.run.smithi092.stdout: "pg_num_target": 32, 2024-03-11T21:41:58.368 INFO:teuthology.orchestra.run.smithi092.stdout: "pg_num_pending": 32, 2024-03-11T21:41:58.368 INFO:teuthology.orchestra.run.smithi092.stdout: "last_pg_merge_meta": { 2024-03-11T21:41:58.368 INFO:teuthology.orchestra.run.smithi092.stdout: "source_pgid": "0.0", 2024-03-11T21:41:58.368 INFO:teuthology.orchestra.run.smithi092.stdout: "ready_epoch": 0, 2024-03-11T21:41:58.368 INFO:teuthology.orchestra.run.smithi092.stdout: "last_epoch_started": 0, 2024-03-11T21:41:58.368 INFO:teuthology.orchestra.run.smithi092.stdout: "last_epoch_clean": 0, 2024-03-11T21:41:58.368 INFO:teuthology.orchestra.run.smithi092.stdout: "source_version": "0'0", 2024-03-11T21:41:58.368 INFO:teuthology.orchestra.run.smithi092.stdout: "target_version": "0'0" 2024-03-11T21:41:58.368 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:41:58.368 INFO:teuthology.orchestra.run.smithi092.stdout: "last_change": "68", 2024-03-11T21:41:58.368 INFO:teuthology.orchestra.run.smithi092.stdout: "last_force_op_resend": "0", 2024-03-11T21:41:58.368 INFO:teuthology.orchestra.run.smithi092.stdout: "last_force_op_resend_prenautilus": "0", 2024-03-11T21:41:58.369 INFO:teuthology.orchestra.run.smithi092.stdout: "last_force_op_resend_preluminous": "0", 2024-03-11T21:41:58.369 INFO:teuthology.orchestra.run.smithi092.stdout: "auid": 0, 2024-03-11T21:41:58.369 INFO:teuthology.orchestra.run.smithi092.stdout: "snap_mode": "selfmanaged", 2024-03-11T21:41:58.369 INFO:teuthology.orchestra.run.smithi092.stdout: "snap_seq": 0, 2024-03-11T21:41:58.369 INFO:teuthology.orchestra.run.smithi092.stdout: "snap_epoch": 0, 2024-03-11T21:41:58.369 INFO:teuthology.orchestra.run.smithi092.stdout: "pool_snaps": [], 2024-03-11T21:41:58.369 INFO:teuthology.orchestra.run.smithi092.stdout: "removed_snaps": "[]", 2024-03-11T21:41:58.369 INFO:teuthology.orchestra.run.smithi092.stdout: "quota_max_bytes": 0, 2024-03-11T21:41:58.369 INFO:teuthology.orchestra.run.smithi092.stdout: "quota_max_objects": 0, 2024-03-11T21:41:58.369 INFO:teuthology.orchestra.run.smithi092.stdout: "tiers": [], 2024-03-11T21:41:58.369 INFO:teuthology.orchestra.run.smithi092.stdout: "tier_of": -1, 2024-03-11T21:41:58.369 INFO:teuthology.orchestra.run.smithi092.stdout: "read_tier": -1, 2024-03-11T21:41:58.369 INFO:teuthology.orchestra.run.smithi092.stdout: "write_tier": -1, 2024-03-11T21:41:58.369 INFO:teuthology.orchestra.run.smithi092.stdout: "cache_mode": "none", 2024-03-11T21:41:58.369 INFO:teuthology.orchestra.run.smithi092.stdout: "target_max_bytes": 0, 2024-03-11T21:41:58.370 INFO:teuthology.orchestra.run.smithi092.stdout: "target_max_objects": 0, 2024-03-11T21:41:58.370 INFO:teuthology.orchestra.run.smithi092.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-03-11T21:41:58.370 INFO:teuthology.orchestra.run.smithi092.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-03-11T21:41:58.370 INFO:teuthology.orchestra.run.smithi092.stdout: "cache_target_full_ratio_micro": 800000, 2024-03-11T21:41:58.370 INFO:teuthology.orchestra.run.smithi092.stdout: "cache_min_flush_age": 0, 2024-03-11T21:41:58.370 INFO:teuthology.orchestra.run.smithi092.stdout: "cache_min_evict_age": 0, 2024-03-11T21:41:58.370 INFO:teuthology.orchestra.run.smithi092.stdout: "erasure_code_profile": "", 2024-03-11T21:41:58.370 INFO:teuthology.orchestra.run.smithi092.stdout: "hit_set_params": { 2024-03-11T21:41:58.370 INFO:teuthology.orchestra.run.smithi092.stdout: "type": "none" 2024-03-11T21:41:58.370 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:41:58.370 INFO:teuthology.orchestra.run.smithi092.stdout: "hit_set_period": 0, 2024-03-11T21:41:58.370 INFO:teuthology.orchestra.run.smithi092.stdout: "hit_set_count": 0, 2024-03-11T21:41:58.370 INFO:teuthology.orchestra.run.smithi092.stdout: "use_gmt_hitset": true, 2024-03-11T21:41:58.370 INFO:teuthology.orchestra.run.smithi092.stdout: "min_read_recency_for_promote": 0, 2024-03-11T21:41:58.371 INFO:teuthology.orchestra.run.smithi092.stdout: "min_write_recency_for_promote": 0, 2024-03-11T21:41:58.371 INFO:teuthology.orchestra.run.smithi092.stdout: "hit_set_grade_decay_rate": 0, 2024-03-11T21:41:58.371 INFO:teuthology.orchestra.run.smithi092.stdout: "hit_set_search_last_n": 0, 2024-03-11T21:41:58.371 INFO:teuthology.orchestra.run.smithi092.stdout: "grade_table": [], 2024-03-11T21:41:58.371 INFO:teuthology.orchestra.run.smithi092.stdout: "stripe_width": 0, 2024-03-11T21:41:58.371 INFO:teuthology.orchestra.run.smithi092.stdout: "expected_num_objects": 0, 2024-03-11T21:41:58.371 INFO:teuthology.orchestra.run.smithi092.stdout: "fast_read": false, 2024-03-11T21:41:58.371 INFO:teuthology.orchestra.run.smithi092.stdout: "options": {}, 2024-03-11T21:41:58.371 INFO:teuthology.orchestra.run.smithi092.stdout: "application_metadata": { 2024-03-11T21:41:58.371 INFO:teuthology.orchestra.run.smithi092.stdout: "rgw": {} 2024-03-11T21:41:58.372 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:41:58.372 INFO:teuthology.orchestra.run.smithi092.stdout: "read_balance": { 2024-03-11T21:41:58.372 INFO:teuthology.orchestra.run.smithi092.stdout: "score_type": "Fair distribution", 2024-03-11T21:41:58.372 INFO:teuthology.orchestra.run.smithi092.stdout: "score_acting": 1.5, 2024-03-11T21:41:58.372 INFO:teuthology.orchestra.run.smithi092.stdout: "score_stable": 1.5, 2024-03-11T21:41:58.372 INFO:teuthology.orchestra.run.smithi092.stdout: "optimal_score": 1, 2024-03-11T21:41:58.372 INFO:teuthology.orchestra.run.smithi092.stdout: "raw_score_acting": 1.5, 2024-03-11T21:41:58.372 INFO:teuthology.orchestra.run.smithi092.stdout: "raw_score_stable": 1.5, 2024-03-11T21:41:58.372 INFO:teuthology.orchestra.run.smithi092.stdout: "primary_affinity_weighted": 1, 2024-03-11T21:41:58.372 INFO:teuthology.orchestra.run.smithi092.stdout: "average_primary_affinity": 1, 2024-03-11T21:41:58.372 INFO:teuthology.orchestra.run.smithi092.stdout: "average_primary_affinity_weighted": 1 2024-03-11T21:41:58.372 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-03-11T21:41:58.372 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:41:58.372 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:41:58.373 INFO:teuthology.orchestra.run.smithi092.stdout: "pool": 3, 2024-03-11T21:41:58.373 INFO:teuthology.orchestra.run.smithi092.stdout: "pool_name": "datapool", 2024-03-11T21:41:58.373 INFO:teuthology.orchestra.run.smithi092.stdout: "create_time": "2024-03-11T21:25:37.280820+0000", 2024-03-11T21:41:58.373 INFO:teuthology.orchestra.run.smithi092.stdout: "flags": 8193, 2024-03-11T21:41:58.373 INFO:teuthology.orchestra.run.smithi092.stdout: "flags_names": "hashpspool,selfmanaged_snaps", 2024-03-11T21:41:58.373 INFO:teuthology.orchestra.run.smithi092.stdout: "type": 1, 2024-03-11T21:41:58.373 INFO:teuthology.orchestra.run.smithi092.stdout: "size": 3, 2024-03-11T21:41:58.373 INFO:teuthology.orchestra.run.smithi092.stdout: "min_size": 2, 2024-03-11T21:41:58.373 INFO:teuthology.orchestra.run.smithi092.stdout: "crush_rule": 0, 2024-03-11T21:41:58.373 INFO:teuthology.orchestra.run.smithi092.stdout: "peering_crush_bucket_count": 0, 2024-03-11T21:41:58.373 INFO:teuthology.orchestra.run.smithi092.stdout: "peering_crush_bucket_target": 0, 2024-03-11T21:41:58.373 INFO:teuthology.orchestra.run.smithi092.stdout: "peering_crush_bucket_barrier": 0, 2024-03-11T21:41:58.373 INFO:teuthology.orchestra.run.smithi092.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-03-11T21:41:58.373 INFO:teuthology.orchestra.run.smithi092.stdout: "object_hash": 2, 2024-03-11T21:41:58.374 INFO:teuthology.orchestra.run.smithi092.stdout: "pg_autoscale_mode": "off", 2024-03-11T21:41:58.374 INFO:teuthology.orchestra.run.smithi092.stdout: "pg_num": 3, 2024-03-11T21:41:58.374 INFO:teuthology.orchestra.run.smithi092.stdout: "pg_placement_num": 3, 2024-03-11T21:41:58.374 INFO:teuthology.orchestra.run.smithi092.stdout: "pg_placement_num_target": 3, 2024-03-11T21:41:58.374 INFO:teuthology.orchestra.run.smithi092.stdout: "pg_num_target": 3, 2024-03-11T21:41:58.374 INFO:teuthology.orchestra.run.smithi092.stdout: "pg_num_pending": 3, 2024-03-11T21:41:58.374 INFO:teuthology.orchestra.run.smithi092.stdout: "last_pg_merge_meta": { 2024-03-11T21:41:58.374 INFO:teuthology.orchestra.run.smithi092.stdout: "source_pgid": "0.0", 2024-03-11T21:41:58.374 INFO:teuthology.orchestra.run.smithi092.stdout: "ready_epoch": 0, 2024-03-11T21:41:58.374 INFO:teuthology.orchestra.run.smithi092.stdout: "last_epoch_started": 0, 2024-03-11T21:41:58.374 INFO:teuthology.orchestra.run.smithi092.stdout: "last_epoch_clean": 0, 2024-03-11T21:41:58.374 INFO:teuthology.orchestra.run.smithi092.stdout: "source_version": "0'0", 2024-03-11T21:41:58.374 INFO:teuthology.orchestra.run.smithi092.stdout: "target_version": "0'0" 2024-03-11T21:41:58.375 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:41:58.375 INFO:teuthology.orchestra.run.smithi092.stdout: "last_change": "71", 2024-03-11T21:41:58.375 INFO:teuthology.orchestra.run.smithi092.stdout: "last_force_op_resend": "0", 2024-03-11T21:41:58.375 INFO:teuthology.orchestra.run.smithi092.stdout: "last_force_op_resend_prenautilus": "0", 2024-03-11T21:41:58.375 INFO:teuthology.orchestra.run.smithi092.stdout: "last_force_op_resend_preluminous": "0", 2024-03-11T21:41:58.375 INFO:teuthology.orchestra.run.smithi092.stdout: "auid": 0, 2024-03-11T21:41:58.375 INFO:teuthology.orchestra.run.smithi092.stdout: "snap_mode": "selfmanaged", 2024-03-11T21:41:58.375 INFO:teuthology.orchestra.run.smithi092.stdout: "snap_seq": 2, 2024-03-11T21:41:58.375 INFO:teuthology.orchestra.run.smithi092.stdout: "snap_epoch": 71, 2024-03-11T21:41:58.375 INFO:teuthology.orchestra.run.smithi092.stdout: "pool_snaps": [], 2024-03-11T21:41:58.375 INFO:teuthology.orchestra.run.smithi092.stdout: "removed_snaps": "[]", 2024-03-11T21:41:58.375 INFO:teuthology.orchestra.run.smithi092.stdout: "quota_max_bytes": 0, 2024-03-11T21:41:58.375 INFO:teuthology.orchestra.run.smithi092.stdout: "quota_max_objects": 0, 2024-03-11T21:41:58.376 INFO:teuthology.orchestra.run.smithi092.stdout: "tiers": [], 2024-03-11T21:41:58.376 INFO:teuthology.orchestra.run.smithi092.stdout: "tier_of": -1, 2024-03-11T21:41:58.376 INFO:teuthology.orchestra.run.smithi092.stdout: "read_tier": -1, 2024-03-11T21:41:58.376 INFO:teuthology.orchestra.run.smithi092.stdout: "write_tier": -1, 2024-03-11T21:41:58.376 INFO:teuthology.orchestra.run.smithi092.stdout: "cache_mode": "none", 2024-03-11T21:41:58.376 INFO:teuthology.orchestra.run.smithi092.stdout: "target_max_bytes": 0, 2024-03-11T21:41:58.376 INFO:teuthology.orchestra.run.smithi092.stdout: "target_max_objects": 0, 2024-03-11T21:41:58.376 INFO:teuthology.orchestra.run.smithi092.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-03-11T21:41:58.376 INFO:teuthology.orchestra.run.smithi092.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-03-11T21:41:58.376 INFO:teuthology.orchestra.run.smithi092.stdout: "cache_target_full_ratio_micro": 800000, 2024-03-11T21:41:58.376 INFO:teuthology.orchestra.run.smithi092.stdout: "cache_min_flush_age": 0, 2024-03-11T21:41:58.376 INFO:teuthology.orchestra.run.smithi092.stdout: "cache_min_evict_age": 0, 2024-03-11T21:41:58.376 INFO:teuthology.orchestra.run.smithi092.stdout: "erasure_code_profile": "", 2024-03-11T21:41:58.376 INFO:teuthology.orchestra.run.smithi092.stdout: "hit_set_params": { 2024-03-11T21:41:58.376 INFO:teuthology.orchestra.run.smithi092.stdout: "type": "none" 2024-03-11T21:41:58.377 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:41:58.377 INFO:teuthology.orchestra.run.smithi092.stdout: "hit_set_period": 0, 2024-03-11T21:41:58.377 INFO:teuthology.orchestra.run.smithi092.stdout: "hit_set_count": 0, 2024-03-11T21:41:58.377 INFO:teuthology.orchestra.run.smithi092.stdout: "use_gmt_hitset": true, 2024-03-11T21:41:58.377 INFO:teuthology.orchestra.run.smithi092.stdout: "min_read_recency_for_promote": 0, 2024-03-11T21:41:58.377 INFO:teuthology.orchestra.run.smithi092.stdout: "min_write_recency_for_promote": 0, 2024-03-11T21:41:58.377 INFO:teuthology.orchestra.run.smithi092.stdout: "hit_set_grade_decay_rate": 0, 2024-03-11T21:41:58.377 INFO:teuthology.orchestra.run.smithi092.stdout: "hit_set_search_last_n": 0, 2024-03-11T21:41:58.378 INFO:teuthology.orchestra.run.smithi092.stdout: "grade_table": [], 2024-03-11T21:41:58.378 INFO:teuthology.orchestra.run.smithi092.stdout: "stripe_width": 0, 2024-03-11T21:41:58.378 INFO:teuthology.orchestra.run.smithi092.stdout: "expected_num_objects": 0, 2024-03-11T21:41:58.378 INFO:teuthology.orchestra.run.smithi092.stdout: "fast_read": false, 2024-03-11T21:41:58.378 INFO:teuthology.orchestra.run.smithi092.stdout: "options": {}, 2024-03-11T21:41:58.378 INFO:teuthology.orchestra.run.smithi092.stdout: "application_metadata": { 2024-03-11T21:41:58.378 INFO:teuthology.orchestra.run.smithi092.stdout: "rbd": {} 2024-03-11T21:41:58.378 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:41:58.378 INFO:teuthology.orchestra.run.smithi092.stdout: "read_balance": { 2024-03-11T21:41:58.378 INFO:teuthology.orchestra.run.smithi092.stdout: "score_type": "Fair distribution", 2024-03-11T21:41:58.378 INFO:teuthology.orchestra.run.smithi092.stdout: "score_acting": 2.6500000953674316, 2024-03-11T21:41:58.379 INFO:teuthology.orchestra.run.smithi092.stdout: "score_stable": 2.6500000953674316, 2024-03-11T21:41:58.379 INFO:teuthology.orchestra.run.smithi092.stdout: "optimal_score": 0.87999999523162842, 2024-03-11T21:41:58.379 INFO:teuthology.orchestra.run.smithi092.stdout: "raw_score_acting": 2.3299999237060547, 2024-03-11T21:41:58.380 INFO:teuthology.orchestra.run.smithi092.stdout: "raw_score_stable": 2.3299999237060547, 2024-03-11T21:41:58.380 INFO:teuthology.orchestra.run.smithi092.stdout: "primary_affinity_weighted": 1, 2024-03-11T21:41:58.380 INFO:teuthology.orchestra.run.smithi092.stdout: "average_primary_affinity": 1, 2024-03-11T21:41:58.380 INFO:teuthology.orchestra.run.smithi092.stdout: "average_primary_affinity_weighted": 1 2024-03-11T21:41:58.380 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-03-11T21:41:58.380 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:41:58.380 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:41:58.380 INFO:teuthology.orchestra.run.smithi092.stdout: "pool": 4, 2024-03-11T21:41:58.380 INFO:teuthology.orchestra.run.smithi092.stdout: "pool_name": "default.rgw.log", 2024-03-11T21:41:58.380 INFO:teuthology.orchestra.run.smithi092.stdout: "create_time": "2024-03-11T21:25:38.850207+0000", 2024-03-11T21:41:58.380 INFO:teuthology.orchestra.run.smithi092.stdout: "flags": 1, 2024-03-11T21:41:58.380 INFO:teuthology.orchestra.run.smithi092.stdout: "flags_names": "hashpspool", 2024-03-11T21:41:58.380 INFO:teuthology.orchestra.run.smithi092.stdout: "type": 1, 2024-03-11T21:41:58.381 INFO:teuthology.orchestra.run.smithi092.stdout: "size": 3, 2024-03-11T21:41:58.381 INFO:teuthology.orchestra.run.smithi092.stdout: "min_size": 2, 2024-03-11T21:41:58.381 INFO:teuthology.orchestra.run.smithi092.stdout: "crush_rule": 0, 2024-03-11T21:41:58.381 INFO:teuthology.orchestra.run.smithi092.stdout: "peering_crush_bucket_count": 0, 2024-03-11T21:41:58.381 INFO:teuthology.orchestra.run.smithi092.stdout: "peering_crush_bucket_target": 0, 2024-03-11T21:41:58.381 INFO:teuthology.orchestra.run.smithi092.stdout: "peering_crush_bucket_barrier": 0, 2024-03-11T21:41:58.381 INFO:teuthology.orchestra.run.smithi092.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-03-11T21:41:58.381 INFO:teuthology.orchestra.run.smithi092.stdout: "object_hash": 2, 2024-03-11T21:41:58.381 INFO:teuthology.orchestra.run.smithi092.stdout: "pg_autoscale_mode": "off", 2024-03-11T21:41:58.381 INFO:teuthology.orchestra.run.smithi092.stdout: "pg_num": 32, 2024-03-11T21:41:58.381 INFO:teuthology.orchestra.run.smithi092.stdout: "pg_placement_num": 32, 2024-03-11T21:41:58.381 INFO:teuthology.orchestra.run.smithi092.stdout: "pg_placement_num_target": 32, 2024-03-11T21:41:58.381 INFO:teuthology.orchestra.run.smithi092.stdout: "pg_num_target": 32, 2024-03-11T21:41:58.382 INFO:teuthology.orchestra.run.smithi092.stdout: "pg_num_pending": 32, 2024-03-11T21:41:58.382 INFO:teuthology.orchestra.run.smithi092.stdout: "last_pg_merge_meta": { 2024-03-11T21:41:58.382 INFO:teuthology.orchestra.run.smithi092.stdout: "source_pgid": "0.0", 2024-03-11T21:41:58.382 INFO:teuthology.orchestra.run.smithi092.stdout: "ready_epoch": 0, 2024-03-11T21:41:58.382 INFO:teuthology.orchestra.run.smithi092.stdout: "last_epoch_started": 0, 2024-03-11T21:41:58.382 INFO:teuthology.orchestra.run.smithi092.stdout: "last_epoch_clean": 0, 2024-03-11T21:41:58.382 INFO:teuthology.orchestra.run.smithi092.stdout: "source_version": "0'0", 2024-03-11T21:41:58.382 INFO:teuthology.orchestra.run.smithi092.stdout: "target_version": "0'0" 2024-03-11T21:41:58.382 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:41:58.382 INFO:teuthology.orchestra.run.smithi092.stdout: "last_change": "70", 2024-03-11T21:41:58.382 INFO:teuthology.orchestra.run.smithi092.stdout: "last_force_op_resend": "0", 2024-03-11T21:41:58.382 INFO:teuthology.orchestra.run.smithi092.stdout: "last_force_op_resend_prenautilus": "0", 2024-03-11T21:41:58.382 INFO:teuthology.orchestra.run.smithi092.stdout: "last_force_op_resend_preluminous": "0", 2024-03-11T21:41:58.383 INFO:teuthology.orchestra.run.smithi092.stdout: "auid": 0, 2024-03-11T21:41:58.383 INFO:teuthology.orchestra.run.smithi092.stdout: "snap_mode": "selfmanaged", 2024-03-11T21:41:58.383 INFO:teuthology.orchestra.run.smithi092.stdout: "snap_seq": 0, 2024-03-11T21:41:58.383 INFO:teuthology.orchestra.run.smithi092.stdout: "snap_epoch": 0, 2024-03-11T21:41:58.383 INFO:teuthology.orchestra.run.smithi092.stdout: "pool_snaps": [], 2024-03-11T21:41:58.383 INFO:teuthology.orchestra.run.smithi092.stdout: "removed_snaps": "[]", 2024-03-11T21:41:58.383 INFO:teuthology.orchestra.run.smithi092.stdout: "quota_max_bytes": 0, 2024-03-11T21:41:58.383 INFO:teuthology.orchestra.run.smithi092.stdout: "quota_max_objects": 0, 2024-03-11T21:41:58.383 INFO:teuthology.orchestra.run.smithi092.stdout: "tiers": [], 2024-03-11T21:41:58.383 INFO:teuthology.orchestra.run.smithi092.stdout: "tier_of": -1, 2024-03-11T21:41:58.384 INFO:teuthology.orchestra.run.smithi092.stdout: "read_tier": -1, 2024-03-11T21:41:58.384 INFO:teuthology.orchestra.run.smithi092.stdout: "write_tier": -1, 2024-03-11T21:41:58.384 INFO:teuthology.orchestra.run.smithi092.stdout: "cache_mode": "none", 2024-03-11T21:41:58.384 INFO:teuthology.orchestra.run.smithi092.stdout: "target_max_bytes": 0, 2024-03-11T21:41:58.384 INFO:teuthology.orchestra.run.smithi092.stdout: "target_max_objects": 0, 2024-03-11T21:41:58.384 INFO:teuthology.orchestra.run.smithi092.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-03-11T21:41:58.384 INFO:teuthology.orchestra.run.smithi092.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-03-11T21:41:58.384 INFO:teuthology.orchestra.run.smithi092.stdout: "cache_target_full_ratio_micro": 800000, 2024-03-11T21:41:58.384 INFO:teuthology.orchestra.run.smithi092.stdout: "cache_min_flush_age": 0, 2024-03-11T21:41:58.384 INFO:teuthology.orchestra.run.smithi092.stdout: "cache_min_evict_age": 0, 2024-03-11T21:41:58.384 INFO:teuthology.orchestra.run.smithi092.stdout: "erasure_code_profile": "", 2024-03-11T21:41:58.384 INFO:teuthology.orchestra.run.smithi092.stdout: "hit_set_params": { 2024-03-11T21:41:58.384 INFO:teuthology.orchestra.run.smithi092.stdout: "type": "none" 2024-03-11T21:41:58.385 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:41:58.385 INFO:teuthology.orchestra.run.smithi092.stdout: "hit_set_period": 0, 2024-03-11T21:41:58.385 INFO:teuthology.orchestra.run.smithi092.stdout: "hit_set_count": 0, 2024-03-11T21:41:58.385 INFO:teuthology.orchestra.run.smithi092.stdout: "use_gmt_hitset": true, 2024-03-11T21:41:58.385 INFO:teuthology.orchestra.run.smithi092.stdout: "min_read_recency_for_promote": 0, 2024-03-11T21:41:58.385 INFO:teuthology.orchestra.run.smithi092.stdout: "min_write_recency_for_promote": 0, 2024-03-11T21:41:58.385 INFO:teuthology.orchestra.run.smithi092.stdout: "hit_set_grade_decay_rate": 0, 2024-03-11T21:41:58.385 INFO:teuthology.orchestra.run.smithi092.stdout: "hit_set_search_last_n": 0, 2024-03-11T21:41:58.386 INFO:teuthology.orchestra.run.smithi092.stdout: "grade_table": [], 2024-03-11T21:41:58.386 INFO:teuthology.orchestra.run.smithi092.stdout: "stripe_width": 0, 2024-03-11T21:41:58.386 INFO:teuthology.orchestra.run.smithi092.stdout: "expected_num_objects": 0, 2024-03-11T21:41:58.386 INFO:teuthology.orchestra.run.smithi092.stdout: "fast_read": false, 2024-03-11T21:41:58.386 INFO:teuthology.orchestra.run.smithi092.stdout: "options": {}, 2024-03-11T21:41:58.386 INFO:teuthology.orchestra.run.smithi092.stdout: "application_metadata": { 2024-03-11T21:41:58.386 INFO:teuthology.orchestra.run.smithi092.stdout: "rgw": {} 2024-03-11T21:41:58.386 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:41:58.387 INFO:teuthology.orchestra.run.smithi092.stdout: "read_balance": { 2024-03-11T21:41:58.387 INFO:teuthology.orchestra.run.smithi092.stdout: "score_type": "Fair distribution", 2024-03-11T21:41:58.387 INFO:teuthology.orchestra.run.smithi092.stdout: "score_acting": 2.25, 2024-03-11T21:41:58.387 INFO:teuthology.orchestra.run.smithi092.stdout: "score_stable": 2.25, 2024-03-11T21:41:58.387 INFO:teuthology.orchestra.run.smithi092.stdout: "optimal_score": 1, 2024-03-11T21:41:58.387 INFO:teuthology.orchestra.run.smithi092.stdout: "raw_score_acting": 2.25, 2024-03-11T21:41:58.387 INFO:teuthology.orchestra.run.smithi092.stdout: "raw_score_stable": 2.25, 2024-03-11T21:41:58.387 INFO:teuthology.orchestra.run.smithi092.stdout: "primary_affinity_weighted": 1, 2024-03-11T21:41:58.387 INFO:teuthology.orchestra.run.smithi092.stdout: "average_primary_affinity": 1, 2024-03-11T21:41:58.387 INFO:teuthology.orchestra.run.smithi092.stdout: "average_primary_affinity_weighted": 1 2024-03-11T21:41:58.387 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-03-11T21:41:58.387 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:41:58.387 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:41:58.387 INFO:teuthology.orchestra.run.smithi092.stdout: "pool": 5, 2024-03-11T21:41:58.388 INFO:teuthology.orchestra.run.smithi092.stdout: "pool_name": "default.rgw.control", 2024-03-11T21:41:58.388 INFO:teuthology.orchestra.run.smithi092.stdout: "create_time": "2024-03-11T21:25:40.794686+0000", 2024-03-11T21:41:58.388 INFO:teuthology.orchestra.run.smithi092.stdout: "flags": 1, 2024-03-11T21:41:58.388 INFO:teuthology.orchestra.run.smithi092.stdout: "flags_names": "hashpspool", 2024-03-11T21:41:58.388 INFO:teuthology.orchestra.run.smithi092.stdout: "type": 1, 2024-03-11T21:41:58.388 INFO:teuthology.orchestra.run.smithi092.stdout: "size": 3, 2024-03-11T21:41:58.388 INFO:teuthology.orchestra.run.smithi092.stdout: "min_size": 2, 2024-03-11T21:41:58.388 INFO:teuthology.orchestra.run.smithi092.stdout: "crush_rule": 0, 2024-03-11T21:41:58.388 INFO:teuthology.orchestra.run.smithi092.stdout: "peering_crush_bucket_count": 0, 2024-03-11T21:41:58.388 INFO:teuthology.orchestra.run.smithi092.stdout: "peering_crush_bucket_target": 0, 2024-03-11T21:41:58.388 INFO:teuthology.orchestra.run.smithi092.stdout: "peering_crush_bucket_barrier": 0, 2024-03-11T21:41:58.388 INFO:teuthology.orchestra.run.smithi092.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-03-11T21:41:58.388 INFO:teuthology.orchestra.run.smithi092.stdout: "object_hash": 2, 2024-03-11T21:41:58.388 INFO:teuthology.orchestra.run.smithi092.stdout: "pg_autoscale_mode": "off", 2024-03-11T21:41:58.389 INFO:teuthology.orchestra.run.smithi092.stdout: "pg_num": 32, 2024-03-11T21:41:58.389 INFO:teuthology.orchestra.run.smithi092.stdout: "pg_placement_num": 32, 2024-03-11T21:41:58.389 INFO:teuthology.orchestra.run.smithi092.stdout: "pg_placement_num_target": 32, 2024-03-11T21:41:58.389 INFO:teuthology.orchestra.run.smithi092.stdout: "pg_num_target": 32, 2024-03-11T21:41:58.389 INFO:teuthology.orchestra.run.smithi092.stdout: "pg_num_pending": 32, 2024-03-11T21:41:58.389 INFO:teuthology.orchestra.run.smithi092.stdout: "last_pg_merge_meta": { 2024-03-11T21:41:58.389 INFO:teuthology.orchestra.run.smithi092.stdout: "source_pgid": "0.0", 2024-03-11T21:41:58.389 INFO:teuthology.orchestra.run.smithi092.stdout: "ready_epoch": 0, 2024-03-11T21:41:58.389 INFO:teuthology.orchestra.run.smithi092.stdout: "last_epoch_started": 0, 2024-03-11T21:41:58.389 INFO:teuthology.orchestra.run.smithi092.stdout: "last_epoch_clean": 0, 2024-03-11T21:41:58.389 INFO:teuthology.orchestra.run.smithi092.stdout: "source_version": "0'0", 2024-03-11T21:41:58.389 INFO:teuthology.orchestra.run.smithi092.stdout: "target_version": "0'0" 2024-03-11T21:41:58.389 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:41:58.389 INFO:teuthology.orchestra.run.smithi092.stdout: "last_change": "72", 2024-03-11T21:41:58.390 INFO:teuthology.orchestra.run.smithi092.stdout: "last_force_op_resend": "0", 2024-03-11T21:41:58.390 INFO:teuthology.orchestra.run.smithi092.stdout: "last_force_op_resend_prenautilus": "0", 2024-03-11T21:41:58.390 INFO:teuthology.orchestra.run.smithi092.stdout: "last_force_op_resend_preluminous": "0", 2024-03-11T21:41:58.390 INFO:teuthology.orchestra.run.smithi092.stdout: "auid": 0, 2024-03-11T21:41:58.390 INFO:teuthology.orchestra.run.smithi092.stdout: "snap_mode": "selfmanaged", 2024-03-11T21:41:58.390 INFO:teuthology.orchestra.run.smithi092.stdout: "snap_seq": 0, 2024-03-11T21:41:58.390 INFO:teuthology.orchestra.run.smithi092.stdout: "snap_epoch": 0, 2024-03-11T21:41:58.390 INFO:teuthology.orchestra.run.smithi092.stdout: "pool_snaps": [], 2024-03-11T21:41:58.390 INFO:teuthology.orchestra.run.smithi092.stdout: "removed_snaps": "[]", 2024-03-11T21:41:58.390 INFO:teuthology.orchestra.run.smithi092.stdout: "quota_max_bytes": 0, 2024-03-11T21:41:58.390 INFO:teuthology.orchestra.run.smithi092.stdout: "quota_max_objects": 0, 2024-03-11T21:41:58.390 INFO:teuthology.orchestra.run.smithi092.stdout: "tiers": [], 2024-03-11T21:41:58.390 INFO:teuthology.orchestra.run.smithi092.stdout: "tier_of": -1, 2024-03-11T21:41:58.390 INFO:teuthology.orchestra.run.smithi092.stdout: "read_tier": -1, 2024-03-11T21:41:58.391 INFO:teuthology.orchestra.run.smithi092.stdout: "write_tier": -1, 2024-03-11T21:41:58.391 INFO:teuthology.orchestra.run.smithi092.stdout: "cache_mode": "none", 2024-03-11T21:41:58.391 INFO:teuthology.orchestra.run.smithi092.stdout: "target_max_bytes": 0, 2024-03-11T21:41:58.391 INFO:teuthology.orchestra.run.smithi092.stdout: "target_max_objects": 0, 2024-03-11T21:41:58.391 INFO:teuthology.orchestra.run.smithi092.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-03-11T21:41:58.391 INFO:teuthology.orchestra.run.smithi092.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-03-11T21:41:58.391 INFO:teuthology.orchestra.run.smithi092.stdout: "cache_target_full_ratio_micro": 800000, 2024-03-11T21:41:58.391 INFO:teuthology.orchestra.run.smithi092.stdout: "cache_min_flush_age": 0, 2024-03-11T21:41:58.391 INFO:teuthology.orchestra.run.smithi092.stdout: "cache_min_evict_age": 0, 2024-03-11T21:41:58.391 INFO:teuthology.orchestra.run.smithi092.stdout: "erasure_code_profile": "", 2024-03-11T21:41:58.391 INFO:teuthology.orchestra.run.smithi092.stdout: "hit_set_params": { 2024-03-11T21:41:58.391 INFO:teuthology.orchestra.run.smithi092.stdout: "type": "none" 2024-03-11T21:41:58.391 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:41:58.392 INFO:teuthology.orchestra.run.smithi092.stdout: "hit_set_period": 0, 2024-03-11T21:41:58.392 INFO:teuthology.orchestra.run.smithi092.stdout: "hit_set_count": 0, 2024-03-11T21:41:58.392 INFO:teuthology.orchestra.run.smithi092.stdout: "use_gmt_hitset": true, 2024-03-11T21:41:58.392 INFO:teuthology.orchestra.run.smithi092.stdout: "min_read_recency_for_promote": 0, 2024-03-11T21:41:58.392 INFO:teuthology.orchestra.run.smithi092.stdout: "min_write_recency_for_promote": 0, 2024-03-11T21:41:58.392 INFO:teuthology.orchestra.run.smithi092.stdout: "hit_set_grade_decay_rate": 0, 2024-03-11T21:41:58.392 INFO:teuthology.orchestra.run.smithi092.stdout: "hit_set_search_last_n": 0, 2024-03-11T21:41:58.392 INFO:teuthology.orchestra.run.smithi092.stdout: "grade_table": [], 2024-03-11T21:41:58.392 INFO:teuthology.orchestra.run.smithi092.stdout: "stripe_width": 0, 2024-03-11T21:41:58.392 INFO:teuthology.orchestra.run.smithi092.stdout: "expected_num_objects": 0, 2024-03-11T21:41:58.392 INFO:teuthology.orchestra.run.smithi092.stdout: "fast_read": false, 2024-03-11T21:41:58.392 INFO:teuthology.orchestra.run.smithi092.stdout: "options": {}, 2024-03-11T21:41:58.392 INFO:teuthology.orchestra.run.smithi092.stdout: "application_metadata": { 2024-03-11T21:41:58.392 INFO:teuthology.orchestra.run.smithi092.stdout: "rgw": {} 2024-03-11T21:41:58.392 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:41:58.393 INFO:teuthology.orchestra.run.smithi092.stdout: "read_balance": { 2024-03-11T21:41:58.393 INFO:teuthology.orchestra.run.smithi092.stdout: "score_type": "Fair distribution", 2024-03-11T21:41:58.393 INFO:teuthology.orchestra.run.smithi092.stdout: "score_acting": 1.25, 2024-03-11T21:41:58.393 INFO:teuthology.orchestra.run.smithi092.stdout: "score_stable": 1.25, 2024-03-11T21:41:58.393 INFO:teuthology.orchestra.run.smithi092.stdout: "optimal_score": 1, 2024-03-11T21:41:58.393 INFO:teuthology.orchestra.run.smithi092.stdout: "raw_score_acting": 1.25, 2024-03-11T21:41:58.393 INFO:teuthology.orchestra.run.smithi092.stdout: "raw_score_stable": 1.25, 2024-03-11T21:41:58.393 INFO:teuthology.orchestra.run.smithi092.stdout: "primary_affinity_weighted": 1, 2024-03-11T21:41:58.393 INFO:teuthology.orchestra.run.smithi092.stdout: "average_primary_affinity": 1, 2024-03-11T21:41:58.393 INFO:teuthology.orchestra.run.smithi092.stdout: "average_primary_affinity_weighted": 1 2024-03-11T21:41:58.393 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-03-11T21:41:58.393 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:41:58.393 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:41:58.393 INFO:teuthology.orchestra.run.smithi092.stdout: "pool": 6, 2024-03-11T21:41:58.393 INFO:teuthology.orchestra.run.smithi092.stdout: "pool_name": "default.rgw.meta", 2024-03-11T21:41:58.394 INFO:teuthology.orchestra.run.smithi092.stdout: "create_time": "2024-03-11T21:25:42.872077+0000", 2024-03-11T21:41:58.394 INFO:teuthology.orchestra.run.smithi092.stdout: "flags": 1, 2024-03-11T21:41:58.394 INFO:teuthology.orchestra.run.smithi092.stdout: "flags_names": "hashpspool", 2024-03-11T21:41:58.394 INFO:teuthology.orchestra.run.smithi092.stdout: "type": 1, 2024-03-11T21:41:58.394 INFO:teuthology.orchestra.run.smithi092.stdout: "size": 3, 2024-03-11T21:41:58.394 INFO:teuthology.orchestra.run.smithi092.stdout: "min_size": 2, 2024-03-11T21:41:58.394 INFO:teuthology.orchestra.run.smithi092.stdout: "crush_rule": 0, 2024-03-11T21:41:58.394 INFO:teuthology.orchestra.run.smithi092.stdout: "peering_crush_bucket_count": 0, 2024-03-11T21:41:58.394 INFO:teuthology.orchestra.run.smithi092.stdout: "peering_crush_bucket_target": 0, 2024-03-11T21:41:58.394 INFO:teuthology.orchestra.run.smithi092.stdout: "peering_crush_bucket_barrier": 0, 2024-03-11T21:41:58.394 INFO:teuthology.orchestra.run.smithi092.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-03-11T21:41:58.394 INFO:teuthology.orchestra.run.smithi092.stdout: "object_hash": 2, 2024-03-11T21:41:58.394 INFO:teuthology.orchestra.run.smithi092.stdout: "pg_autoscale_mode": "off", 2024-03-11T21:41:58.395 INFO:teuthology.orchestra.run.smithi092.stdout: "pg_num": 32, 2024-03-11T21:41:58.395 INFO:teuthology.orchestra.run.smithi092.stdout: "pg_placement_num": 32, 2024-03-11T21:41:58.395 INFO:teuthology.orchestra.run.smithi092.stdout: "pg_placement_num_target": 32, 2024-03-11T21:41:58.395 INFO:teuthology.orchestra.run.smithi092.stdout: "pg_num_target": 32, 2024-03-11T21:41:58.395 INFO:teuthology.orchestra.run.smithi092.stdout: "pg_num_pending": 32, 2024-03-11T21:41:58.395 INFO:teuthology.orchestra.run.smithi092.stdout: "last_pg_merge_meta": { 2024-03-11T21:41:58.395 INFO:teuthology.orchestra.run.smithi092.stdout: "source_pgid": "0.0", 2024-03-11T21:41:58.395 INFO:teuthology.orchestra.run.smithi092.stdout: "ready_epoch": 0, 2024-03-11T21:41:58.395 INFO:teuthology.orchestra.run.smithi092.stdout: "last_epoch_started": 0, 2024-03-11T21:41:58.395 INFO:teuthology.orchestra.run.smithi092.stdout: "last_epoch_clean": 0, 2024-03-11T21:41:58.395 INFO:teuthology.orchestra.run.smithi092.stdout: "source_version": "0'0", 2024-03-11T21:41:58.395 INFO:teuthology.orchestra.run.smithi092.stdout: "target_version": "0'0" 2024-03-11T21:41:58.395 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:41:58.395 INFO:teuthology.orchestra.run.smithi092.stdout: "last_change": "74", 2024-03-11T21:41:58.396 INFO:teuthology.orchestra.run.smithi092.stdout: "last_force_op_resend": "0", 2024-03-11T21:41:58.396 INFO:teuthology.orchestra.run.smithi092.stdout: "last_force_op_resend_prenautilus": "0", 2024-03-11T21:41:58.396 INFO:teuthology.orchestra.run.smithi092.stdout: "last_force_op_resend_preluminous": "0", 2024-03-11T21:41:58.396 INFO:teuthology.orchestra.run.smithi092.stdout: "auid": 0, 2024-03-11T21:41:58.396 INFO:teuthology.orchestra.run.smithi092.stdout: "snap_mode": "selfmanaged", 2024-03-11T21:41:58.396 INFO:teuthology.orchestra.run.smithi092.stdout: "snap_seq": 0, 2024-03-11T21:41:58.396 INFO:teuthology.orchestra.run.smithi092.stdout: "snap_epoch": 0, 2024-03-11T21:41:58.396 INFO:teuthology.orchestra.run.smithi092.stdout: "pool_snaps": [], 2024-03-11T21:41:58.396 INFO:teuthology.orchestra.run.smithi092.stdout: "removed_snaps": "[]", 2024-03-11T21:41:58.396 INFO:teuthology.orchestra.run.smithi092.stdout: "quota_max_bytes": 0, 2024-03-11T21:41:58.396 INFO:teuthology.orchestra.run.smithi092.stdout: "quota_max_objects": 0, 2024-03-11T21:41:58.396 INFO:teuthology.orchestra.run.smithi092.stdout: "tiers": [], 2024-03-11T21:41:58.396 INFO:teuthology.orchestra.run.smithi092.stdout: "tier_of": -1, 2024-03-11T21:41:58.397 INFO:teuthology.orchestra.run.smithi092.stdout: "read_tier": -1, 2024-03-11T21:41:58.397 INFO:teuthology.orchestra.run.smithi092.stdout: "write_tier": -1, 2024-03-11T21:41:58.397 INFO:teuthology.orchestra.run.smithi092.stdout: "cache_mode": "none", 2024-03-11T21:41:58.397 INFO:teuthology.orchestra.run.smithi092.stdout: "target_max_bytes": 0, 2024-03-11T21:41:58.397 INFO:teuthology.orchestra.run.smithi092.stdout: "target_max_objects": 0, 2024-03-11T21:41:58.397 INFO:teuthology.orchestra.run.smithi092.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-03-11T21:41:58.397 INFO:teuthology.orchestra.run.smithi092.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-03-11T21:41:58.397 INFO:teuthology.orchestra.run.smithi092.stdout: "cache_target_full_ratio_micro": 800000, 2024-03-11T21:41:58.397 INFO:teuthology.orchestra.run.smithi092.stdout: "cache_min_flush_age": 0, 2024-03-11T21:41:58.397 INFO:teuthology.orchestra.run.smithi092.stdout: "cache_min_evict_age": 0, 2024-03-11T21:41:58.397 INFO:teuthology.orchestra.run.smithi092.stdout: "erasure_code_profile": "", 2024-03-11T21:41:58.397 INFO:teuthology.orchestra.run.smithi092.stdout: "hit_set_params": { 2024-03-11T21:41:58.397 INFO:teuthology.orchestra.run.smithi092.stdout: "type": "none" 2024-03-11T21:41:58.397 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:41:58.398 INFO:teuthology.orchestra.run.smithi092.stdout: "hit_set_period": 0, 2024-03-11T21:41:58.398 INFO:teuthology.orchestra.run.smithi092.stdout: "hit_set_count": 0, 2024-03-11T21:41:58.398 INFO:teuthology.orchestra.run.smithi092.stdout: "use_gmt_hitset": true, 2024-03-11T21:41:58.398 INFO:teuthology.orchestra.run.smithi092.stdout: "min_read_recency_for_promote": 0, 2024-03-11T21:41:58.398 INFO:teuthology.orchestra.run.smithi092.stdout: "min_write_recency_for_promote": 0, 2024-03-11T21:41:58.398 INFO:teuthology.orchestra.run.smithi092.stdout: "hit_set_grade_decay_rate": 0, 2024-03-11T21:41:58.398 INFO:teuthology.orchestra.run.smithi092.stdout: "hit_set_search_last_n": 0, 2024-03-11T21:41:58.398 INFO:teuthology.orchestra.run.smithi092.stdout: "grade_table": [], 2024-03-11T21:41:58.398 INFO:teuthology.orchestra.run.smithi092.stdout: "stripe_width": 0, 2024-03-11T21:41:58.398 INFO:teuthology.orchestra.run.smithi092.stdout: "expected_num_objects": 0, 2024-03-11T21:41:58.398 INFO:teuthology.orchestra.run.smithi092.stdout: "fast_read": false, 2024-03-11T21:41:58.399 INFO:teuthology.orchestra.run.smithi092.stdout: "options": { 2024-03-11T21:41:58.399 INFO:teuthology.orchestra.run.smithi092.stdout: "pg_autoscale_bias": 4 2024-03-11T21:41:58.399 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:41:58.399 INFO:teuthology.orchestra.run.smithi092.stdout: "application_metadata": { 2024-03-11T21:41:58.399 INFO:teuthology.orchestra.run.smithi092.stdout: "rgw": {} 2024-03-11T21:41:58.399 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:41:58.399 INFO:teuthology.orchestra.run.smithi092.stdout: "read_balance": { 2024-03-11T21:41:58.399 INFO:teuthology.orchestra.run.smithi092.stdout: "score_type": "Fair distribution", 2024-03-11T21:41:58.399 INFO:teuthology.orchestra.run.smithi092.stdout: "score_acting": 1.75, 2024-03-11T21:41:58.400 INFO:teuthology.orchestra.run.smithi092.stdout: "score_stable": 1.75, 2024-03-11T21:41:58.400 INFO:teuthology.orchestra.run.smithi092.stdout: "optimal_score": 1, 2024-03-11T21:41:58.400 INFO:teuthology.orchestra.run.smithi092.stdout: "raw_score_acting": 1.75, 2024-03-11T21:41:58.400 INFO:teuthology.orchestra.run.smithi092.stdout: "raw_score_stable": 1.75, 2024-03-11T21:41:58.400 INFO:teuthology.orchestra.run.smithi092.stdout: "primary_affinity_weighted": 1, 2024-03-11T21:41:58.400 INFO:teuthology.orchestra.run.smithi092.stdout: "average_primary_affinity": 1, 2024-03-11T21:41:58.400 INFO:teuthology.orchestra.run.smithi092.stdout: "average_primary_affinity_weighted": 1 2024-03-11T21:41:58.400 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-03-11T21:41:58.400 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:41:58.400 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:41:58.400 INFO:teuthology.orchestra.run.smithi092.stdout: "pool": 10, 2024-03-11T21:41:58.400 INFO:teuthology.orchestra.run.smithi092.stdout: "pool_name": "unique_pool_3", 2024-03-11T21:41:58.400 INFO:teuthology.orchestra.run.smithi092.stdout: "create_time": "2024-03-11T21:40:37.970897+0000", 2024-03-11T21:41:58.400 INFO:teuthology.orchestra.run.smithi092.stdout: "flags": 1, 2024-03-11T21:41:58.401 INFO:teuthology.orchestra.run.smithi092.stdout: "flags_names": "hashpspool", 2024-03-11T21:41:58.401 INFO:teuthology.orchestra.run.smithi092.stdout: "type": 1, 2024-03-11T21:41:58.401 INFO:teuthology.orchestra.run.smithi092.stdout: "size": 3, 2024-03-11T21:41:58.401 INFO:teuthology.orchestra.run.smithi092.stdout: "min_size": 2, 2024-03-11T21:41:58.401 INFO:teuthology.orchestra.run.smithi092.stdout: "crush_rule": 0, 2024-03-11T21:41:58.401 INFO:teuthology.orchestra.run.smithi092.stdout: "peering_crush_bucket_count": 0, 2024-03-11T21:41:58.401 INFO:teuthology.orchestra.run.smithi092.stdout: "peering_crush_bucket_target": 0, 2024-03-11T21:41:58.401 INFO:teuthology.orchestra.run.smithi092.stdout: "peering_crush_bucket_barrier": 0, 2024-03-11T21:41:58.401 INFO:teuthology.orchestra.run.smithi092.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-03-11T21:41:58.401 INFO:teuthology.orchestra.run.smithi092.stdout: "object_hash": 2, 2024-03-11T21:41:58.401 INFO:teuthology.orchestra.run.smithi092.stdout: "pg_autoscale_mode": "off", 2024-03-11T21:41:58.401 INFO:teuthology.orchestra.run.smithi092.stdout: "pg_num": 26, 2024-03-11T21:41:58.401 INFO:teuthology.orchestra.run.smithi092.stdout: "pg_placement_num": 26, 2024-03-11T21:41:58.401 INFO:teuthology.orchestra.run.smithi092.stdout: "pg_placement_num_target": 26, 2024-03-11T21:41:58.401 INFO:teuthology.orchestra.run.smithi092.stdout: "pg_num_target": 26, 2024-03-11T21:41:58.402 INFO:teuthology.orchestra.run.smithi092.stdout: "pg_num_pending": 26, 2024-03-11T21:41:58.402 INFO:teuthology.orchestra.run.smithi092.stdout: "last_pg_merge_meta": { 2024-03-11T21:41:58.402 INFO:teuthology.orchestra.run.smithi092.stdout: "source_pgid": "0.0", 2024-03-11T21:41:58.402 INFO:teuthology.orchestra.run.smithi092.stdout: "ready_epoch": 0, 2024-03-11T21:41:58.402 INFO:teuthology.orchestra.run.smithi092.stdout: "last_epoch_started": 0, 2024-03-11T21:41:58.402 INFO:teuthology.orchestra.run.smithi092.stdout: "last_epoch_clean": 0, 2024-03-11T21:41:58.402 INFO:teuthology.orchestra.run.smithi092.stdout: "source_version": "0'0", 2024-03-11T21:41:58.402 INFO:teuthology.orchestra.run.smithi092.stdout: "target_version": "0'0" 2024-03-11T21:41:58.402 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:41:58.402 INFO:teuthology.orchestra.run.smithi092.stdout: "last_change": "374", 2024-03-11T21:41:58.402 INFO:teuthology.orchestra.run.smithi092.stdout: "last_force_op_resend": "0", 2024-03-11T21:41:58.402 INFO:teuthology.orchestra.run.smithi092.stdout: "last_force_op_resend_prenautilus": "0", 2024-03-11T21:41:58.402 INFO:teuthology.orchestra.run.smithi092.stdout: "last_force_op_resend_preluminous": "372", 2024-03-11T21:41:58.403 INFO:teuthology.orchestra.run.smithi092.stdout: "auid": 0, 2024-03-11T21:41:58.403 INFO:teuthology.orchestra.run.smithi092.stdout: "snap_mode": "selfmanaged", 2024-03-11T21:41:58.403 INFO:teuthology.orchestra.run.smithi092.stdout: "snap_seq": 0, 2024-03-11T21:41:58.403 INFO:teuthology.orchestra.run.smithi092.stdout: "snap_epoch": 0, 2024-03-11T21:41:58.403 INFO:teuthology.orchestra.run.smithi092.stdout: "pool_snaps": [], 2024-03-11T21:41:58.403 INFO:teuthology.orchestra.run.smithi092.stdout: "removed_snaps": "[]", 2024-03-11T21:41:58.403 INFO:teuthology.orchestra.run.smithi092.stdout: "quota_max_bytes": 0, 2024-03-11T21:41:58.403 INFO:teuthology.orchestra.run.smithi092.stdout: "quota_max_objects": 0, 2024-03-11T21:41:58.403 INFO:teuthology.orchestra.run.smithi092.stdout: "tiers": [], 2024-03-11T21:41:58.403 INFO:teuthology.orchestra.run.smithi092.stdout: "tier_of": -1, 2024-03-11T21:41:58.403 INFO:teuthology.orchestra.run.smithi092.stdout: "read_tier": -1, 2024-03-11T21:41:58.403 INFO:teuthology.orchestra.run.smithi092.stdout: "write_tier": -1, 2024-03-11T21:41:58.403 INFO:teuthology.orchestra.run.smithi092.stdout: "cache_mode": "none", 2024-03-11T21:41:58.403 INFO:teuthology.orchestra.run.smithi092.stdout: "target_max_bytes": 0, 2024-03-11T21:41:58.404 INFO:teuthology.orchestra.run.smithi092.stdout: "target_max_objects": 0, 2024-03-11T21:41:58.404 INFO:teuthology.orchestra.run.smithi092.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-03-11T21:41:58.404 INFO:teuthology.orchestra.run.smithi092.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-03-11T21:41:58.404 INFO:teuthology.orchestra.run.smithi092.stdout: "cache_target_full_ratio_micro": 800000, 2024-03-11T21:41:58.404 INFO:teuthology.orchestra.run.smithi092.stdout: "cache_min_flush_age": 0, 2024-03-11T21:41:58.404 INFO:teuthology.orchestra.run.smithi092.stdout: "cache_min_evict_age": 0, 2024-03-11T21:41:58.404 INFO:teuthology.orchestra.run.smithi092.stdout: "erasure_code_profile": "", 2024-03-11T21:41:58.404 INFO:teuthology.orchestra.run.smithi092.stdout: "hit_set_params": { 2024-03-11T21:41:58.404 INFO:teuthology.orchestra.run.smithi092.stdout: "type": "none" 2024-03-11T21:41:58.404 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:41:58.404 INFO:teuthology.orchestra.run.smithi092.stdout: "hit_set_period": 0, 2024-03-11T21:41:58.404 INFO:teuthology.orchestra.run.smithi092.stdout: "hit_set_count": 0, 2024-03-11T21:41:58.404 INFO:teuthology.orchestra.run.smithi092.stdout: "use_gmt_hitset": true, 2024-03-11T21:41:58.404 INFO:teuthology.orchestra.run.smithi092.stdout: "min_read_recency_for_promote": 0, 2024-03-11T21:41:58.404 INFO:teuthology.orchestra.run.smithi092.stdout: "min_write_recency_for_promote": 0, 2024-03-11T21:41:58.405 INFO:teuthology.orchestra.run.smithi092.stdout: "hit_set_grade_decay_rate": 0, 2024-03-11T21:41:58.405 INFO:teuthology.orchestra.run.smithi092.stdout: "hit_set_search_last_n": 0, 2024-03-11T21:41:58.405 INFO:teuthology.orchestra.run.smithi092.stdout: "grade_table": [], 2024-03-11T21:41:58.405 INFO:teuthology.orchestra.run.smithi092.stdout: "stripe_width": 0, 2024-03-11T21:41:58.405 INFO:teuthology.orchestra.run.smithi092.stdout: "expected_num_objects": 0, 2024-03-11T21:41:58.405 INFO:teuthology.orchestra.run.smithi092.stdout: "fast_read": false, 2024-03-11T21:41:58.405 INFO:teuthology.orchestra.run.smithi092.stdout: "options": {}, 2024-03-11T21:41:58.405 INFO:teuthology.orchestra.run.smithi092.stdout: "application_metadata": { 2024-03-11T21:41:58.405 INFO:teuthology.orchestra.run.smithi092.stdout: "rados": {} 2024-03-11T21:41:58.405 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:41:58.405 INFO:teuthology.orchestra.run.smithi092.stdout: "read_balance": { 2024-03-11T21:41:58.405 INFO:teuthology.orchestra.run.smithi092.stdout: "score_type": "Fair distribution", 2024-03-11T21:41:58.405 INFO:teuthology.orchestra.run.smithi092.stdout: "score_acting": 1.8500000238418579, 2024-03-11T21:41:58.405 INFO:teuthology.orchestra.run.smithi092.stdout: "score_stable": 2.4600000381469727, 2024-03-11T21:41:58.406 INFO:teuthology.orchestra.run.smithi092.stdout: "optimal_score": 1, 2024-03-11T21:41:58.406 INFO:teuthology.orchestra.run.smithi092.stdout: "raw_score_acting": 1.8500000238418579, 2024-03-11T21:41:58.406 INFO:teuthology.orchestra.run.smithi092.stdout: "raw_score_stable": 2.4600000381469727, 2024-03-11T21:41:58.406 INFO:teuthology.orchestra.run.smithi092.stdout: "primary_affinity_weighted": 1, 2024-03-11T21:41:58.406 INFO:teuthology.orchestra.run.smithi092.stdout: "average_primary_affinity": 1, 2024-03-11T21:41:58.406 INFO:teuthology.orchestra.run.smithi092.stdout: "average_primary_affinity_weighted": 1 2024-03-11T21:41:58.406 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-03-11T21:41:58.406 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-03-11T21:41:58.406 INFO:teuthology.orchestra.run.smithi092.stdout: ], 2024-03-11T21:41:58.406 INFO:teuthology.orchestra.run.smithi092.stdout: "osds": [ 2024-03-11T21:41:58.406 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:41:58.406 INFO:teuthology.orchestra.run.smithi092.stdout: "osd": 0, 2024-03-11T21:41:58.406 INFO:teuthology.orchestra.run.smithi092.stdout: "uuid": "79f15843-f0ec-44aa-931e-4e07a74d3768", 2024-03-11T21:41:58.406 INFO:teuthology.orchestra.run.smithi092.stdout: "up": 1, 2024-03-11T21:41:58.407 INFO:teuthology.orchestra.run.smithi092.stdout: "in": 1, 2024-03-11T21:41:58.407 INFO:teuthology.orchestra.run.smithi092.stdout: "weight": 1, 2024-03-11T21:41:58.407 INFO:teuthology.orchestra.run.smithi092.stdout: "primary_affinity": 1, 2024-03-11T21:41:58.407 INFO:teuthology.orchestra.run.smithi092.stdout: "last_clean_begin": 0, 2024-03-11T21:41:58.407 INFO:teuthology.orchestra.run.smithi092.stdout: "last_clean_end": 0, 2024-03-11T21:41:58.407 INFO:teuthology.orchestra.run.smithi092.stdout: "up_from": 8, 2024-03-11T21:41:58.407 INFO:teuthology.orchestra.run.smithi092.stdout: "up_thru": 375, 2024-03-11T21:41:58.407 INFO:teuthology.orchestra.run.smithi092.stdout: "down_at": 0, 2024-03-11T21:41:58.407 INFO:teuthology.orchestra.run.smithi092.stdout: "lost_at": 0, 2024-03-11T21:41:58.407 INFO:teuthology.orchestra.run.smithi092.stdout: "public_addrs": { 2024-03-11T21:41:58.407 INFO:teuthology.orchestra.run.smithi092.stdout: "addrvec": [ 2024-03-11T21:41:58.407 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:41:58.407 INFO:teuthology.orchestra.run.smithi092.stdout: "type": "v1", 2024-03-11T21:41:58.407 INFO:teuthology.orchestra.run.smithi092.stdout: "addr": "172.21.15.92:6801", 2024-03-11T21:41:58.408 INFO:teuthology.orchestra.run.smithi092.stdout: "nonce": 2844100201 2024-03-11T21:41:58.408 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-03-11T21:41:58.408 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-03-11T21:41:58.408 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:41:58.408 INFO:teuthology.orchestra.run.smithi092.stdout: "cluster_addrs": { 2024-03-11T21:41:58.408 INFO:teuthology.orchestra.run.smithi092.stdout: "addrvec": [ 2024-03-11T21:41:58.408 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:41:58.408 INFO:teuthology.orchestra.run.smithi092.stdout: "type": "v1", 2024-03-11T21:41:58.408 INFO:teuthology.orchestra.run.smithi092.stdout: "addr": "172.21.15.92:6802", 2024-03-11T21:41:58.408 INFO:teuthology.orchestra.run.smithi092.stdout: "nonce": 2844100201 2024-03-11T21:41:58.408 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-03-11T21:41:58.408 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-03-11T21:41:58.408 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:41:58.408 INFO:teuthology.orchestra.run.smithi092.stdout: "heartbeat_back_addrs": { 2024-03-11T21:41:58.408 INFO:teuthology.orchestra.run.smithi092.stdout: "addrvec": [ 2024-03-11T21:41:58.409 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:41:58.409 INFO:teuthology.orchestra.run.smithi092.stdout: "type": "v1", 2024-03-11T21:41:58.409 INFO:teuthology.orchestra.run.smithi092.stdout: "addr": "172.21.15.92:6804", 2024-03-11T21:41:58.409 INFO:teuthology.orchestra.run.smithi092.stdout: "nonce": 2844100201 2024-03-11T21:41:58.409 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-03-11T21:41:58.409 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-03-11T21:41:58.409 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:41:58.409 INFO:teuthology.orchestra.run.smithi092.stdout: "heartbeat_front_addrs": { 2024-03-11T21:41:58.409 INFO:teuthology.orchestra.run.smithi092.stdout: "addrvec": [ 2024-03-11T21:41:58.409 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:41:58.409 INFO:teuthology.orchestra.run.smithi092.stdout: "type": "v1", 2024-03-11T21:41:58.409 INFO:teuthology.orchestra.run.smithi092.stdout: "addr": "172.21.15.92:6803", 2024-03-11T21:41:58.409 INFO:teuthology.orchestra.run.smithi092.stdout: "nonce": 2844100201 2024-03-11T21:41:58.410 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-03-11T21:41:58.410 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-03-11T21:41:58.410 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:41:58.410 INFO:teuthology.orchestra.run.smithi092.stdout: "public_addr": "172.21.15.92:6801/2844100201", 2024-03-11T21:41:58.410 INFO:teuthology.orchestra.run.smithi092.stdout: "cluster_addr": "172.21.15.92:6802/2844100201", 2024-03-11T21:41:58.410 INFO:teuthology.orchestra.run.smithi092.stdout: "heartbeat_back_addr": "172.21.15.92:6804/2844100201", 2024-03-11T21:41:58.410 INFO:teuthology.orchestra.run.smithi092.stdout: "heartbeat_front_addr": "172.21.15.92:6803/2844100201", 2024-03-11T21:41:58.410 INFO:teuthology.orchestra.run.smithi092.stdout: "state": [ 2024-03-11T21:41:58.410 INFO:teuthology.orchestra.run.smithi092.stdout: "exists", 2024-03-11T21:41:58.410 INFO:teuthology.orchestra.run.smithi092.stdout: "up" 2024-03-11T21:41:58.410 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-03-11T21:41:58.410 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:41:58.410 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:41:58.410 INFO:teuthology.orchestra.run.smithi092.stdout: "osd": 1, 2024-03-11T21:41:58.411 INFO:teuthology.orchestra.run.smithi092.stdout: "uuid": "3a5a0e78-1293-4ace-a93f-69156098571d", 2024-03-11T21:41:58.411 INFO:teuthology.orchestra.run.smithi092.stdout: "up": 1, 2024-03-11T21:41:58.411 INFO:teuthology.orchestra.run.smithi092.stdout: "in": 1, 2024-03-11T21:41:58.411 INFO:teuthology.orchestra.run.smithi092.stdout: "weight": 1, 2024-03-11T21:41:58.411 INFO:teuthology.orchestra.run.smithi092.stdout: "primary_affinity": 1, 2024-03-11T21:41:58.411 INFO:teuthology.orchestra.run.smithi092.stdout: "last_clean_begin": 0, 2024-03-11T21:41:58.411 INFO:teuthology.orchestra.run.smithi092.stdout: "last_clean_end": 0, 2024-03-11T21:41:58.411 INFO:teuthology.orchestra.run.smithi092.stdout: "up_from": 13, 2024-03-11T21:41:58.411 INFO:teuthology.orchestra.run.smithi092.stdout: "up_thru": 374, 2024-03-11T21:41:58.411 INFO:teuthology.orchestra.run.smithi092.stdout: "down_at": 0, 2024-03-11T21:41:58.411 INFO:teuthology.orchestra.run.smithi092.stdout: "lost_at": 0, 2024-03-11T21:41:58.411 INFO:teuthology.orchestra.run.smithi092.stdout: "public_addrs": { 2024-03-11T21:41:58.411 INFO:teuthology.orchestra.run.smithi092.stdout: "addrvec": [ 2024-03-11T21:41:58.411 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:41:58.411 INFO:teuthology.orchestra.run.smithi092.stdout: "type": "v1", 2024-03-11T21:41:58.412 INFO:teuthology.orchestra.run.smithi092.stdout: "addr": "172.21.15.92:6805", 2024-03-11T21:41:58.412 INFO:teuthology.orchestra.run.smithi092.stdout: "nonce": 3577871157 2024-03-11T21:41:58.412 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-03-11T21:41:58.412 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-03-11T21:41:58.412 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:41:58.412 INFO:teuthology.orchestra.run.smithi092.stdout: "cluster_addrs": { 2024-03-11T21:41:58.412 INFO:teuthology.orchestra.run.smithi092.stdout: "addrvec": [ 2024-03-11T21:41:58.412 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:41:58.412 INFO:teuthology.orchestra.run.smithi092.stdout: "type": "v1", 2024-03-11T21:41:58.412 INFO:teuthology.orchestra.run.smithi092.stdout: "addr": "172.21.15.92:6806", 2024-03-11T21:41:58.412 INFO:teuthology.orchestra.run.smithi092.stdout: "nonce": 3577871157 2024-03-11T21:41:58.412 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-03-11T21:41:58.412 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-03-11T21:41:58.413 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:41:58.413 INFO:teuthology.orchestra.run.smithi092.stdout: "heartbeat_back_addrs": { 2024-03-11T21:41:58.413 INFO:teuthology.orchestra.run.smithi092.stdout: "addrvec": [ 2024-03-11T21:41:58.413 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:41:58.413 INFO:teuthology.orchestra.run.smithi092.stdout: "type": "v1", 2024-03-11T21:41:58.413 INFO:teuthology.orchestra.run.smithi092.stdout: "addr": "172.21.15.92:6808", 2024-03-11T21:41:58.413 INFO:teuthology.orchestra.run.smithi092.stdout: "nonce": 3577871157 2024-03-11T21:41:58.413 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-03-11T21:41:58.413 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-03-11T21:41:58.413 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:41:58.413 INFO:teuthology.orchestra.run.smithi092.stdout: "heartbeat_front_addrs": { 2024-03-11T21:41:58.413 INFO:teuthology.orchestra.run.smithi092.stdout: "addrvec": [ 2024-03-11T21:41:58.413 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:41:58.413 INFO:teuthology.orchestra.run.smithi092.stdout: "type": "v1", 2024-03-11T21:41:58.414 INFO:teuthology.orchestra.run.smithi092.stdout: "addr": "172.21.15.92:6807", 2024-03-11T21:41:58.414 INFO:teuthology.orchestra.run.smithi092.stdout: "nonce": 3577871157 2024-03-11T21:41:58.414 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-03-11T21:41:58.414 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-03-11T21:41:58.414 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:41:58.414 INFO:teuthology.orchestra.run.smithi092.stdout: "public_addr": "172.21.15.92:6805/3577871157", 2024-03-11T21:41:58.414 INFO:teuthology.orchestra.run.smithi092.stdout: "cluster_addr": "172.21.15.92:6806/3577871157", 2024-03-11T21:41:58.414 INFO:teuthology.orchestra.run.smithi092.stdout: "heartbeat_back_addr": "172.21.15.92:6808/3577871157", 2024-03-11T21:41:58.414 INFO:teuthology.orchestra.run.smithi092.stdout: "heartbeat_front_addr": "172.21.15.92:6807/3577871157", 2024-03-11T21:41:58.414 INFO:teuthology.orchestra.run.smithi092.stdout: "state": [ 2024-03-11T21:41:58.414 INFO:teuthology.orchestra.run.smithi092.stdout: "exists", 2024-03-11T21:41:58.414 INFO:teuthology.orchestra.run.smithi092.stdout: "up" 2024-03-11T21:41:58.414 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-03-11T21:41:58.415 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:41:58.415 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:41:58.415 INFO:teuthology.orchestra.run.smithi092.stdout: "osd": 2, 2024-03-11T21:41:58.415 INFO:teuthology.orchestra.run.smithi092.stdout: "uuid": "f207b743-0df4-40ed-a735-950e272fea60", 2024-03-11T21:41:58.415 INFO:teuthology.orchestra.run.smithi092.stdout: "up": 1, 2024-03-11T21:41:58.415 INFO:teuthology.orchestra.run.smithi092.stdout: "in": 1, 2024-03-11T21:41:58.415 INFO:teuthology.orchestra.run.smithi092.stdout: "weight": 1, 2024-03-11T21:41:58.415 INFO:teuthology.orchestra.run.smithi092.stdout: "primary_affinity": 1, 2024-03-11T21:41:58.415 INFO:teuthology.orchestra.run.smithi092.stdout: "last_clean_begin": 0, 2024-03-11T21:41:58.415 INFO:teuthology.orchestra.run.smithi092.stdout: "last_clean_end": 0, 2024-03-11T21:41:58.415 INFO:teuthology.orchestra.run.smithi092.stdout: "up_from": 18, 2024-03-11T21:41:58.415 INFO:teuthology.orchestra.run.smithi092.stdout: "up_thru": 375, 2024-03-11T21:41:58.415 INFO:teuthology.orchestra.run.smithi092.stdout: "down_at": 0, 2024-03-11T21:41:58.415 INFO:teuthology.orchestra.run.smithi092.stdout: "lost_at": 0, 2024-03-11T21:41:58.416 INFO:teuthology.orchestra.run.smithi092.stdout: "public_addrs": { 2024-03-11T21:41:58.416 INFO:teuthology.orchestra.run.smithi092.stdout: "addrvec": [ 2024-03-11T21:41:58.416 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:41:58.416 INFO:teuthology.orchestra.run.smithi092.stdout: "type": "v1", 2024-03-11T21:41:58.416 INFO:teuthology.orchestra.run.smithi092.stdout: "addr": "172.21.15.92:6809", 2024-03-11T21:41:58.416 INFO:teuthology.orchestra.run.smithi092.stdout: "nonce": 760825152 2024-03-11T21:41:58.416 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-03-11T21:41:58.416 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-03-11T21:41:58.416 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:41:58.416 INFO:teuthology.orchestra.run.smithi092.stdout: "cluster_addrs": { 2024-03-11T21:41:58.417 INFO:teuthology.orchestra.run.smithi092.stdout: "addrvec": [ 2024-03-11T21:41:58.417 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:41:58.417 INFO:teuthology.orchestra.run.smithi092.stdout: "type": "v1", 2024-03-11T21:41:58.417 INFO:teuthology.orchestra.run.smithi092.stdout: "addr": "172.21.15.92:6810", 2024-03-11T21:41:58.417 INFO:teuthology.orchestra.run.smithi092.stdout: "nonce": 760825152 2024-03-11T21:41:58.417 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-03-11T21:41:58.417 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-03-11T21:41:58.417 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:41:58.417 INFO:teuthology.orchestra.run.smithi092.stdout: "heartbeat_back_addrs": { 2024-03-11T21:41:58.417 INFO:teuthology.orchestra.run.smithi092.stdout: "addrvec": [ 2024-03-11T21:41:58.417 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:41:58.417 INFO:teuthology.orchestra.run.smithi092.stdout: "type": "v1", 2024-03-11T21:41:58.417 INFO:teuthology.orchestra.run.smithi092.stdout: "addr": "172.21.15.92:6812", 2024-03-11T21:41:58.417 INFO:teuthology.orchestra.run.smithi092.stdout: "nonce": 760825152 2024-03-11T21:41:58.418 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-03-11T21:41:58.418 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-03-11T21:41:58.418 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:41:58.418 INFO:teuthology.orchestra.run.smithi092.stdout: "heartbeat_front_addrs": { 2024-03-11T21:41:58.418 INFO:teuthology.orchestra.run.smithi092.stdout: "addrvec": [ 2024-03-11T21:41:58.418 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:41:58.418 INFO:teuthology.orchestra.run.smithi092.stdout: "type": "v1", 2024-03-11T21:41:58.418 INFO:teuthology.orchestra.run.smithi092.stdout: "addr": "172.21.15.92:6811", 2024-03-11T21:41:58.418 INFO:teuthology.orchestra.run.smithi092.stdout: "nonce": 760825152 2024-03-11T21:41:58.418 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-03-11T21:41:58.418 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-03-11T21:41:58.418 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:41:58.418 INFO:teuthology.orchestra.run.smithi092.stdout: "public_addr": "172.21.15.92:6809/760825152", 2024-03-11T21:41:58.421 INFO:teuthology.orchestra.run.smithi092.stdout: "cluster_addr": "172.21.15.92:6810/760825152", 2024-03-11T21:41:58.421 INFO:teuthology.orchestra.run.smithi092.stdout: "heartbeat_back_addr": "172.21.15.92:6812/760825152", 2024-03-11T21:41:58.421 INFO:teuthology.orchestra.run.smithi092.stdout: "heartbeat_front_addr": "172.21.15.92:6811/760825152", 2024-03-11T21:41:58.422 INFO:teuthology.orchestra.run.smithi092.stdout: "state": [ 2024-03-11T21:41:58.422 INFO:teuthology.orchestra.run.smithi092.stdout: "exists", 2024-03-11T21:41:58.422 INFO:teuthology.orchestra.run.smithi092.stdout: "up" 2024-03-11T21:41:58.422 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-03-11T21:41:58.422 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:41:58.422 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:41:58.422 INFO:teuthology.orchestra.run.smithi092.stdout: "osd": 3, 2024-03-11T21:41:58.422 INFO:teuthology.orchestra.run.smithi092.stdout: "uuid": "bbb6f69a-8f0f-47e6-937a-062e1902ae13", 2024-03-11T21:41:58.422 INFO:teuthology.orchestra.run.smithi092.stdout: "up": 1, 2024-03-11T21:41:58.422 INFO:teuthology.orchestra.run.smithi092.stdout: "in": 1, 2024-03-11T21:41:58.422 INFO:teuthology.orchestra.run.smithi092.stdout: "weight": 1, 2024-03-11T21:41:58.422 INFO:teuthology.orchestra.run.smithi092.stdout: "primary_affinity": 1, 2024-03-11T21:41:58.422 INFO:teuthology.orchestra.run.smithi092.stdout: "last_clean_begin": 26, 2024-03-11T21:41:58.422 INFO:teuthology.orchestra.run.smithi092.stdout: "last_clean_end": 251, 2024-03-11T21:41:58.423 INFO:teuthology.orchestra.run.smithi092.stdout: "up_from": 277, 2024-03-11T21:41:58.423 INFO:teuthology.orchestra.run.smithi092.stdout: "up_thru": 375, 2024-03-11T21:41:58.423 INFO:teuthology.orchestra.run.smithi092.stdout: "down_at": 252, 2024-03-11T21:41:58.423 INFO:teuthology.orchestra.run.smithi092.stdout: "lost_at": 0, 2024-03-11T21:41:58.423 INFO:teuthology.orchestra.run.smithi092.stdout: "public_addrs": { 2024-03-11T21:41:58.423 INFO:teuthology.orchestra.run.smithi092.stdout: "addrvec": [ 2024-03-11T21:41:58.423 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:41:58.423 INFO:teuthology.orchestra.run.smithi092.stdout: "type": "v1", 2024-03-11T21:41:58.423 INFO:teuthology.orchestra.run.smithi092.stdout: "addr": "172.21.15.92:6813", 2024-03-11T21:41:58.424 INFO:teuthology.orchestra.run.smithi092.stdout: "nonce": 3215041236 2024-03-11T21:41:58.424 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-03-11T21:41:58.424 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-03-11T21:41:58.424 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:41:58.424 INFO:teuthology.orchestra.run.smithi092.stdout: "cluster_addrs": { 2024-03-11T21:41:58.424 INFO:teuthology.orchestra.run.smithi092.stdout: "addrvec": [ 2024-03-11T21:41:58.424 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:41:58.424 INFO:teuthology.orchestra.run.smithi092.stdout: "type": "v1", 2024-03-11T21:41:58.424 INFO:teuthology.orchestra.run.smithi092.stdout: "addr": "172.21.15.92:6814", 2024-03-11T21:41:58.424 INFO:teuthology.orchestra.run.smithi092.stdout: "nonce": 3215041236 2024-03-11T21:41:58.424 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-03-11T21:41:58.425 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-03-11T21:41:58.425 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:41:58.425 INFO:teuthology.orchestra.run.smithi092.stdout: "heartbeat_back_addrs": { 2024-03-11T21:41:58.425 INFO:teuthology.orchestra.run.smithi092.stdout: "addrvec": [ 2024-03-11T21:41:58.425 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:41:58.425 INFO:teuthology.orchestra.run.smithi092.stdout: "type": "v1", 2024-03-11T21:41:58.425 INFO:teuthology.orchestra.run.smithi092.stdout: "addr": "172.21.15.92:6816", 2024-03-11T21:41:58.425 INFO:teuthology.orchestra.run.smithi092.stdout: "nonce": 3215041236 2024-03-11T21:41:58.425 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-03-11T21:41:58.425 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-03-11T21:41:58.425 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:41:58.425 INFO:teuthology.orchestra.run.smithi092.stdout: "heartbeat_front_addrs": { 2024-03-11T21:41:58.425 INFO:teuthology.orchestra.run.smithi092.stdout: "addrvec": [ 2024-03-11T21:41:58.425 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:41:58.426 INFO:teuthology.orchestra.run.smithi092.stdout: "type": "v1", 2024-03-11T21:41:58.426 INFO:teuthology.orchestra.run.smithi092.stdout: "addr": "172.21.15.92:6815", 2024-03-11T21:41:58.426 INFO:teuthology.orchestra.run.smithi092.stdout: "nonce": 3215041236 2024-03-11T21:41:58.426 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-03-11T21:41:58.426 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-03-11T21:41:58.426 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:41:58.426 INFO:teuthology.orchestra.run.smithi092.stdout: "public_addr": "172.21.15.92:6813/3215041236", 2024-03-11T21:41:58.426 INFO:teuthology.orchestra.run.smithi092.stdout: "cluster_addr": "172.21.15.92:6814/3215041236", 2024-03-11T21:41:58.426 INFO:teuthology.orchestra.run.smithi092.stdout: "heartbeat_back_addr": "172.21.15.92:6816/3215041236", 2024-03-11T21:41:58.426 INFO:teuthology.orchestra.run.smithi092.stdout: "heartbeat_front_addr": "172.21.15.92:6815/3215041236", 2024-03-11T21:41:58.426 INFO:teuthology.orchestra.run.smithi092.stdout: "state": [ 2024-03-11T21:41:58.426 INFO:teuthology.orchestra.run.smithi092.stdout: "exists", 2024-03-11T21:41:58.426 INFO:teuthology.orchestra.run.smithi092.stdout: "up" 2024-03-11T21:41:58.426 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-03-11T21:41:58.427 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:41:58.427 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:41:58.427 INFO:teuthology.orchestra.run.smithi092.stdout: "osd": 4, 2024-03-11T21:41:58.427 INFO:teuthology.orchestra.run.smithi092.stdout: "uuid": "00266cbc-d2cf-4dee-8a96-3753ebc95ca8", 2024-03-11T21:41:58.427 INFO:teuthology.orchestra.run.smithi092.stdout: "up": 1, 2024-03-11T21:41:58.427 INFO:teuthology.orchestra.run.smithi092.stdout: "in": 1, 2024-03-11T21:41:58.427 INFO:teuthology.orchestra.run.smithi092.stdout: "weight": 1, 2024-03-11T21:41:58.427 INFO:teuthology.orchestra.run.smithi092.stdout: "primary_affinity": 1, 2024-03-11T21:41:58.427 INFO:teuthology.orchestra.run.smithi092.stdout: "last_clean_begin": 0, 2024-03-11T21:41:58.427 INFO:teuthology.orchestra.run.smithi092.stdout: "last_clean_end": 0, 2024-03-11T21:41:58.427 INFO:teuthology.orchestra.run.smithi092.stdout: "up_from": 32, 2024-03-11T21:41:58.427 INFO:teuthology.orchestra.run.smithi092.stdout: "up_thru": 380, 2024-03-11T21:41:58.427 INFO:teuthology.orchestra.run.smithi092.stdout: "down_at": 0, 2024-03-11T21:41:58.427 INFO:teuthology.orchestra.run.smithi092.stdout: "lost_at": 0, 2024-03-11T21:41:58.428 INFO:teuthology.orchestra.run.smithi092.stdout: "public_addrs": { 2024-03-11T21:41:58.428 INFO:teuthology.orchestra.run.smithi092.stdout: "addrvec": [ 2024-03-11T21:41:58.428 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:41:58.428 INFO:teuthology.orchestra.run.smithi092.stdout: "type": "v1", 2024-03-11T21:41:58.428 INFO:teuthology.orchestra.run.smithi092.stdout: "addr": "172.21.15.97:6800", 2024-03-11T21:41:58.428 INFO:teuthology.orchestra.run.smithi092.stdout: "nonce": 420733620 2024-03-11T21:41:58.428 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-03-11T21:41:58.428 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-03-11T21:41:58.428 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:41:58.428 INFO:teuthology.orchestra.run.smithi092.stdout: "cluster_addrs": { 2024-03-11T21:41:58.428 INFO:teuthology.orchestra.run.smithi092.stdout: "addrvec": [ 2024-03-11T21:41:58.428 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:41:58.428 INFO:teuthology.orchestra.run.smithi092.stdout: "type": "v1", 2024-03-11T21:41:58.428 INFO:teuthology.orchestra.run.smithi092.stdout: "addr": "172.21.15.97:6801", 2024-03-11T21:41:58.429 INFO:teuthology.orchestra.run.smithi092.stdout: "nonce": 420733620 2024-03-11T21:41:58.429 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-03-11T21:41:58.429 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-03-11T21:41:58.429 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:41:58.429 INFO:teuthology.orchestra.run.smithi092.stdout: "heartbeat_back_addrs": { 2024-03-11T21:41:58.429 INFO:teuthology.orchestra.run.smithi092.stdout: "addrvec": [ 2024-03-11T21:41:58.429 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:41:58.429 INFO:teuthology.orchestra.run.smithi092.stdout: "type": "v1", 2024-03-11T21:41:58.429 INFO:teuthology.orchestra.run.smithi092.stdout: "addr": "172.21.15.97:6803", 2024-03-11T21:41:58.429 INFO:teuthology.orchestra.run.smithi092.stdout: "nonce": 420733620 2024-03-11T21:41:58.429 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-03-11T21:41:58.429 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-03-11T21:41:58.429 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:41:58.429 INFO:teuthology.orchestra.run.smithi092.stdout: "heartbeat_front_addrs": { 2024-03-11T21:41:58.430 INFO:teuthology.orchestra.run.smithi092.stdout: "addrvec": [ 2024-03-11T21:41:58.430 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:41:58.430 INFO:teuthology.orchestra.run.smithi092.stdout: "type": "v1", 2024-03-11T21:41:58.430 INFO:teuthology.orchestra.run.smithi092.stdout: "addr": "172.21.15.97:6802", 2024-03-11T21:41:58.430 INFO:teuthology.orchestra.run.smithi092.stdout: "nonce": 420733620 2024-03-11T21:41:58.430 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-03-11T21:41:58.430 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-03-11T21:41:58.431 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:41:58.431 INFO:teuthology.orchestra.run.smithi092.stdout: "public_addr": "172.21.15.97:6800/420733620", 2024-03-11T21:41:58.431 INFO:teuthology.orchestra.run.smithi092.stdout: "cluster_addr": "172.21.15.97:6801/420733620", 2024-03-11T21:41:58.431 INFO:teuthology.orchestra.run.smithi092.stdout: "heartbeat_back_addr": "172.21.15.97:6803/420733620", 2024-03-11T21:41:58.431 INFO:teuthology.orchestra.run.smithi092.stdout: "heartbeat_front_addr": "172.21.15.97:6802/420733620", 2024-03-11T21:41:58.431 INFO:teuthology.orchestra.run.smithi092.stdout: "state": [ 2024-03-11T21:41:58.431 INFO:teuthology.orchestra.run.smithi092.stdout: "exists", 2024-03-11T21:41:58.431 INFO:teuthology.orchestra.run.smithi092.stdout: "up" 2024-03-11T21:41:58.431 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-03-11T21:41:58.431 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:41:58.431 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:41:58.431 INFO:teuthology.orchestra.run.smithi092.stdout: "osd": 5, 2024-03-11T21:41:58.431 INFO:teuthology.orchestra.run.smithi092.stdout: "uuid": "593a6f6b-8874-40ae-8848-d51c5723a5e4", 2024-03-11T21:41:58.431 INFO:teuthology.orchestra.run.smithi092.stdout: "up": 1, 2024-03-11T21:41:58.431 INFO:teuthology.orchestra.run.smithi092.stdout: "in": 1, 2024-03-11T21:41:58.432 INFO:teuthology.orchestra.run.smithi092.stdout: "weight": 1, 2024-03-11T21:41:58.432 INFO:teuthology.orchestra.run.smithi092.stdout: "primary_affinity": 1, 2024-03-11T21:41:58.432 INFO:teuthology.orchestra.run.smithi092.stdout: "last_clean_begin": 0, 2024-03-11T21:41:58.432 INFO:teuthology.orchestra.run.smithi092.stdout: "last_clean_end": 0, 2024-03-11T21:41:58.432 INFO:teuthology.orchestra.run.smithi092.stdout: "up_from": 41, 2024-03-11T21:41:58.432 INFO:teuthology.orchestra.run.smithi092.stdout: "up_thru": 374, 2024-03-11T21:41:58.432 INFO:teuthology.orchestra.run.smithi092.stdout: "down_at": 0, 2024-03-11T21:41:58.432 INFO:teuthology.orchestra.run.smithi092.stdout: "lost_at": 0, 2024-03-11T21:41:58.432 INFO:teuthology.orchestra.run.smithi092.stdout: "public_addrs": { 2024-03-11T21:41:58.432 INFO:teuthology.orchestra.run.smithi092.stdout: "addrvec": [ 2024-03-11T21:41:58.432 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:41:58.432 INFO:teuthology.orchestra.run.smithi092.stdout: "type": "v1", 2024-03-11T21:41:58.432 INFO:teuthology.orchestra.run.smithi092.stdout: "addr": "172.21.15.97:6804", 2024-03-11T21:41:58.432 INFO:teuthology.orchestra.run.smithi092.stdout: "nonce": 312056041 2024-03-11T21:41:58.432 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-03-11T21:41:58.433 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-03-11T21:41:58.433 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:41:58.433 INFO:teuthology.orchestra.run.smithi092.stdout: "cluster_addrs": { 2024-03-11T21:41:58.433 INFO:teuthology.orchestra.run.smithi092.stdout: "addrvec": [ 2024-03-11T21:41:58.433 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:41:58.433 INFO:teuthology.orchestra.run.smithi092.stdout: "type": "v1", 2024-03-11T21:41:58.433 INFO:teuthology.orchestra.run.smithi092.stdout: "addr": "172.21.15.97:6805", 2024-03-11T21:41:58.433 INFO:teuthology.orchestra.run.smithi092.stdout: "nonce": 312056041 2024-03-11T21:41:58.433 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-03-11T21:41:58.433 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-03-11T21:41:58.433 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:41:58.433 INFO:teuthology.orchestra.run.smithi092.stdout: "heartbeat_back_addrs": { 2024-03-11T21:41:58.433 INFO:teuthology.orchestra.run.smithi092.stdout: "addrvec": [ 2024-03-11T21:41:58.433 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:41:58.434 INFO:teuthology.orchestra.run.smithi092.stdout: "type": "v1", 2024-03-11T21:41:58.434 INFO:teuthology.orchestra.run.smithi092.stdout: "addr": "172.21.15.97:6807", 2024-03-11T21:41:58.434 INFO:teuthology.orchestra.run.smithi092.stdout: "nonce": 312056041 2024-03-11T21:41:58.434 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-03-11T21:41:58.434 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-03-11T21:41:58.434 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:41:58.434 INFO:teuthology.orchestra.run.smithi092.stdout: "heartbeat_front_addrs": { 2024-03-11T21:41:58.434 INFO:teuthology.orchestra.run.smithi092.stdout: "addrvec": [ 2024-03-11T21:41:58.434 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:41:58.434 INFO:teuthology.orchestra.run.smithi092.stdout: "type": "v1", 2024-03-11T21:41:58.435 INFO:teuthology.orchestra.run.smithi092.stdout: "addr": "172.21.15.97:6806", 2024-03-11T21:41:58.436 INFO:teuthology.orchestra.run.smithi092.stdout: "nonce": 312056041 2024-03-11T21:41:58.436 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-03-11T21:41:58.436 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-03-11T21:41:58.436 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:41:58.436 INFO:teuthology.orchestra.run.smithi092.stdout: "public_addr": "172.21.15.97:6804/312056041", 2024-03-11T21:41:58.436 INFO:teuthology.orchestra.run.smithi092.stdout: "cluster_addr": "172.21.15.97:6805/312056041", 2024-03-11T21:41:58.436 INFO:teuthology.orchestra.run.smithi092.stdout: "heartbeat_back_addr": "172.21.15.97:6807/312056041", 2024-03-11T21:41:58.436 INFO:teuthology.orchestra.run.smithi092.stdout: "heartbeat_front_addr": "172.21.15.97:6806/312056041", 2024-03-11T21:41:58.436 INFO:teuthology.orchestra.run.smithi092.stdout: "state": [ 2024-03-11T21:41:58.436 INFO:teuthology.orchestra.run.smithi092.stdout: "exists", 2024-03-11T21:41:58.436 INFO:teuthology.orchestra.run.smithi092.stdout: "up" 2024-03-11T21:41:58.437 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-03-11T21:41:58.437 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:41:58.437 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:41:58.437 INFO:teuthology.orchestra.run.smithi092.stdout: "osd": 6, 2024-03-11T21:41:58.437 INFO:teuthology.orchestra.run.smithi092.stdout: "uuid": "91035ecc-6f1a-4cec-bda6-090af9d53eff", 2024-03-11T21:41:58.437 INFO:teuthology.orchestra.run.smithi092.stdout: "up": 1, 2024-03-11T21:41:58.437 INFO:teuthology.orchestra.run.smithi092.stdout: "in": 1, 2024-03-11T21:41:58.437 INFO:teuthology.orchestra.run.smithi092.stdout: "weight": 1, 2024-03-11T21:41:58.437 INFO:teuthology.orchestra.run.smithi092.stdout: "primary_affinity": 1, 2024-03-11T21:41:58.437 INFO:teuthology.orchestra.run.smithi092.stdout: "last_clean_begin": 0, 2024-03-11T21:41:58.437 INFO:teuthology.orchestra.run.smithi092.stdout: "last_clean_end": 0, 2024-03-11T21:41:58.437 INFO:teuthology.orchestra.run.smithi092.stdout: "up_from": 50, 2024-03-11T21:41:58.437 INFO:teuthology.orchestra.run.smithi092.stdout: "up_thru": 374, 2024-03-11T21:41:58.437 INFO:teuthology.orchestra.run.smithi092.stdout: "down_at": 0, 2024-03-11T21:41:58.437 INFO:teuthology.orchestra.run.smithi092.stdout: "lost_at": 0, 2024-03-11T21:41:58.438 INFO:teuthology.orchestra.run.smithi092.stdout: "public_addrs": { 2024-03-11T21:41:58.438 INFO:teuthology.orchestra.run.smithi092.stdout: "addrvec": [ 2024-03-11T21:41:58.438 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:41:58.438 INFO:teuthology.orchestra.run.smithi092.stdout: "type": "v1", 2024-03-11T21:41:58.438 INFO:teuthology.orchestra.run.smithi092.stdout: "addr": "172.21.15.97:6808", 2024-03-11T21:41:58.438 INFO:teuthology.orchestra.run.smithi092.stdout: "nonce": 178994999 2024-03-11T21:41:58.438 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-03-11T21:41:58.438 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-03-11T21:41:58.438 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:41:58.438 INFO:teuthology.orchestra.run.smithi092.stdout: "cluster_addrs": { 2024-03-11T21:41:58.438 INFO:teuthology.orchestra.run.smithi092.stdout: "addrvec": [ 2024-03-11T21:41:58.438 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:41:58.438 INFO:teuthology.orchestra.run.smithi092.stdout: "type": "v1", 2024-03-11T21:41:58.439 INFO:teuthology.orchestra.run.smithi092.stdout: "addr": "172.21.15.97:6809", 2024-03-11T21:41:58.439 INFO:teuthology.orchestra.run.smithi092.stdout: "nonce": 178994999 2024-03-11T21:41:58.439 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-03-11T21:41:58.439 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-03-11T21:41:58.439 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:41:58.439 INFO:teuthology.orchestra.run.smithi092.stdout: "heartbeat_back_addrs": { 2024-03-11T21:41:58.439 INFO:teuthology.orchestra.run.smithi092.stdout: "addrvec": [ 2024-03-11T21:41:58.439 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:41:58.439 INFO:teuthology.orchestra.run.smithi092.stdout: "type": "v1", 2024-03-11T21:41:58.439 INFO:teuthology.orchestra.run.smithi092.stdout: "addr": "172.21.15.97:6811", 2024-03-11T21:41:58.439 INFO:teuthology.orchestra.run.smithi092.stdout: "nonce": 178994999 2024-03-11T21:41:58.439 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-03-11T21:41:58.439 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-03-11T21:41:58.439 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:41:58.439 INFO:teuthology.orchestra.run.smithi092.stdout: "heartbeat_front_addrs": { 2024-03-11T21:41:58.440 INFO:teuthology.orchestra.run.smithi092.stdout: "addrvec": [ 2024-03-11T21:41:58.440 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:41:58.440 INFO:teuthology.orchestra.run.smithi092.stdout: "type": "v1", 2024-03-11T21:41:58.440 INFO:teuthology.orchestra.run.smithi092.stdout: "addr": "172.21.15.97:6810", 2024-03-11T21:41:58.440 INFO:teuthology.orchestra.run.smithi092.stdout: "nonce": 178994999 2024-03-11T21:41:58.440 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-03-11T21:41:58.440 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-03-11T21:41:58.440 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:41:58.440 INFO:teuthology.orchestra.run.smithi092.stdout: "public_addr": "172.21.15.97:6808/178994999", 2024-03-11T21:41:58.440 INFO:teuthology.orchestra.run.smithi092.stdout: "cluster_addr": "172.21.15.97:6809/178994999", 2024-03-11T21:41:58.440 INFO:teuthology.orchestra.run.smithi092.stdout: "heartbeat_back_addr": "172.21.15.97:6811/178994999", 2024-03-11T21:41:58.440 INFO:teuthology.orchestra.run.smithi092.stdout: "heartbeat_front_addr": "172.21.15.97:6810/178994999", 2024-03-11T21:41:58.440 INFO:teuthology.orchestra.run.smithi092.stdout: "state": [ 2024-03-11T21:41:58.440 INFO:teuthology.orchestra.run.smithi092.stdout: "exists", 2024-03-11T21:41:58.440 INFO:teuthology.orchestra.run.smithi092.stdout: "up" 2024-03-11T21:41:58.441 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-03-11T21:41:58.441 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:41:58.441 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:41:58.441 INFO:teuthology.orchestra.run.smithi092.stdout: "osd": 7, 2024-03-11T21:41:58.441 INFO:teuthology.orchestra.run.smithi092.stdout: "uuid": "3eb28c24-664a-4bd6-869d-6bf5db98f52f", 2024-03-11T21:41:58.441 INFO:teuthology.orchestra.run.smithi092.stdout: "up": 1, 2024-03-11T21:41:58.441 INFO:teuthology.orchestra.run.smithi092.stdout: "in": 1, 2024-03-11T21:41:58.441 INFO:teuthology.orchestra.run.smithi092.stdout: "weight": 1, 2024-03-11T21:41:58.441 INFO:teuthology.orchestra.run.smithi092.stdout: "primary_affinity": 1, 2024-03-11T21:41:58.441 INFO:teuthology.orchestra.run.smithi092.stdout: "last_clean_begin": 0, 2024-03-11T21:41:58.441 INFO:teuthology.orchestra.run.smithi092.stdout: "last_clean_end": 0, 2024-03-11T21:41:58.441 INFO:teuthology.orchestra.run.smithi092.stdout: "up_from": 60, 2024-03-11T21:41:58.441 INFO:teuthology.orchestra.run.smithi092.stdout: "up_thru": 356, 2024-03-11T21:41:58.441 INFO:teuthology.orchestra.run.smithi092.stdout: "down_at": 0, 2024-03-11T21:41:58.441 INFO:teuthology.orchestra.run.smithi092.stdout: "lost_at": 0, 2024-03-11T21:41:58.442 INFO:teuthology.orchestra.run.smithi092.stdout: "public_addrs": { 2024-03-11T21:41:58.442 INFO:teuthology.orchestra.run.smithi092.stdout: "addrvec": [ 2024-03-11T21:41:58.442 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:41:58.442 INFO:teuthology.orchestra.run.smithi092.stdout: "type": "v1", 2024-03-11T21:41:58.442 INFO:teuthology.orchestra.run.smithi092.stdout: "addr": "172.21.15.97:6812", 2024-03-11T21:41:58.442 INFO:teuthology.orchestra.run.smithi092.stdout: "nonce": 3241560509 2024-03-11T21:41:58.442 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-03-11T21:41:58.442 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-03-11T21:41:58.442 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:41:58.442 INFO:teuthology.orchestra.run.smithi092.stdout: "cluster_addrs": { 2024-03-11T21:41:58.442 INFO:teuthology.orchestra.run.smithi092.stdout: "addrvec": [ 2024-03-11T21:41:58.442 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:41:58.442 INFO:teuthology.orchestra.run.smithi092.stdout: "type": "v1", 2024-03-11T21:41:58.442 INFO:teuthology.orchestra.run.smithi092.stdout: "addr": "172.21.15.97:6813", 2024-03-11T21:41:58.442 INFO:teuthology.orchestra.run.smithi092.stdout: "nonce": 3241560509 2024-03-11T21:41:58.443 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-03-11T21:41:58.443 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-03-11T21:41:58.443 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:41:58.443 INFO:teuthology.orchestra.run.smithi092.stdout: "heartbeat_back_addrs": { 2024-03-11T21:41:58.443 INFO:teuthology.orchestra.run.smithi092.stdout: "addrvec": [ 2024-03-11T21:41:58.443 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:41:58.443 INFO:teuthology.orchestra.run.smithi092.stdout: "type": "v1", 2024-03-11T21:41:58.443 INFO:teuthology.orchestra.run.smithi092.stdout: "addr": "172.21.15.97:6815", 2024-03-11T21:41:58.443 INFO:teuthology.orchestra.run.smithi092.stdout: "nonce": 3241560509 2024-03-11T21:41:58.443 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-03-11T21:41:58.443 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-03-11T21:41:58.443 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:41:58.443 INFO:teuthology.orchestra.run.smithi092.stdout: "heartbeat_front_addrs": { 2024-03-11T21:41:58.443 INFO:teuthology.orchestra.run.smithi092.stdout: "addrvec": [ 2024-03-11T21:41:58.444 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:41:58.444 INFO:teuthology.orchestra.run.smithi092.stdout: "type": "v1", 2024-03-11T21:41:58.444 INFO:teuthology.orchestra.run.smithi092.stdout: "addr": "172.21.15.97:6814", 2024-03-11T21:41:58.444 INFO:teuthology.orchestra.run.smithi092.stdout: "nonce": 3241560509 2024-03-11T21:41:58.444 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-03-11T21:41:58.444 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-03-11T21:41:58.444 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:41:58.444 INFO:teuthology.orchestra.run.smithi092.stdout: "public_addr": "172.21.15.97:6812/3241560509", 2024-03-11T21:41:58.444 INFO:teuthology.orchestra.run.smithi092.stdout: "cluster_addr": "172.21.15.97:6813/3241560509", 2024-03-11T21:41:58.444 INFO:teuthology.orchestra.run.smithi092.stdout: "heartbeat_back_addr": "172.21.15.97:6815/3241560509", 2024-03-11T21:41:58.444 INFO:teuthology.orchestra.run.smithi092.stdout: "heartbeat_front_addr": "172.21.15.97:6814/3241560509", 2024-03-11T21:41:58.444 INFO:teuthology.orchestra.run.smithi092.stdout: "state": [ 2024-03-11T21:41:58.444 INFO:teuthology.orchestra.run.smithi092.stdout: "exists", 2024-03-11T21:41:58.444 INFO:teuthology.orchestra.run.smithi092.stdout: "up" 2024-03-11T21:41:58.444 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-03-11T21:41:58.445 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-03-11T21:41:58.445 INFO:teuthology.orchestra.run.smithi092.stdout: ], 2024-03-11T21:41:58.445 INFO:teuthology.orchestra.run.smithi092.stdout: "osd_xinfo": [ 2024-03-11T21:41:58.445 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:41:58.445 INFO:teuthology.orchestra.run.smithi092.stdout: "osd": 0, 2024-03-11T21:41:58.445 INFO:teuthology.orchestra.run.smithi092.stdout: "down_stamp": "0.000000", 2024-03-11T21:41:58.445 INFO:teuthology.orchestra.run.smithi092.stdout: "laggy_probability": 0, 2024-03-11T21:41:58.445 INFO:teuthology.orchestra.run.smithi092.stdout: "laggy_interval": 0, 2024-03-11T21:41:58.445 INFO:teuthology.orchestra.run.smithi092.stdout: "features": 4540701547738038271, 2024-03-11T21:41:58.445 INFO:teuthology.orchestra.run.smithi092.stdout: "old_weight": 0, 2024-03-11T21:41:58.445 INFO:teuthology.orchestra.run.smithi092.stdout: "last_purged_snaps_scrub": "2024-03-11T21:40:47.505890+0000", 2024-03-11T21:41:58.445 INFO:teuthology.orchestra.run.smithi092.stdout: "dead_epoch": 0 2024-03-11T21:41:58.445 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:41:58.445 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:41:58.445 INFO:teuthology.orchestra.run.smithi092.stdout: "osd": 1, 2024-03-11T21:41:58.446 INFO:teuthology.orchestra.run.smithi092.stdout: "down_stamp": "0.000000", 2024-03-11T21:41:58.446 INFO:teuthology.orchestra.run.smithi092.stdout: "laggy_probability": 0, 2024-03-11T21:41:58.446 INFO:teuthology.orchestra.run.smithi092.stdout: "laggy_interval": 0, 2024-03-11T21:41:58.446 INFO:teuthology.orchestra.run.smithi092.stdout: "features": 4540701547738038271, 2024-03-11T21:41:58.446 INFO:teuthology.orchestra.run.smithi092.stdout: "old_weight": 0, 2024-03-11T21:41:58.446 INFO:teuthology.orchestra.run.smithi092.stdout: "last_purged_snaps_scrub": "2024-03-11T21:41:21.884755+0000", 2024-03-11T21:41:58.446 INFO:teuthology.orchestra.run.smithi092.stdout: "dead_epoch": 0 2024-03-11T21:41:58.446 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:41:58.446 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:41:58.446 INFO:teuthology.orchestra.run.smithi092.stdout: "osd": 2, 2024-03-11T21:41:58.446 INFO:teuthology.orchestra.run.smithi092.stdout: "down_stamp": "0.000000", 2024-03-11T21:41:58.446 INFO:teuthology.orchestra.run.smithi092.stdout: "laggy_probability": 0, 2024-03-11T21:41:58.446 INFO:teuthology.orchestra.run.smithi092.stdout: "laggy_interval": 0, 2024-03-11T21:41:58.446 INFO:teuthology.orchestra.run.smithi092.stdout: "features": 4540701547738038271, 2024-03-11T21:41:58.447 INFO:teuthology.orchestra.run.smithi092.stdout: "old_weight": 0, 2024-03-11T21:41:58.447 INFO:teuthology.orchestra.run.smithi092.stdout: "last_purged_snaps_scrub": "2024-03-11T21:41:15.172620+0000", 2024-03-11T21:41:58.447 INFO:teuthology.orchestra.run.smithi092.stdout: "dead_epoch": 0 2024-03-11T21:41:58.447 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:41:58.447 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:41:58.447 INFO:teuthology.orchestra.run.smithi092.stdout: "osd": 3, 2024-03-11T21:41:58.447 INFO:teuthology.orchestra.run.smithi092.stdout: "down_stamp": "0.000000", 2024-03-11T21:41:58.447 INFO:teuthology.orchestra.run.smithi092.stdout: "laggy_probability": 0, 2024-03-11T21:41:58.447 INFO:teuthology.orchestra.run.smithi092.stdout: "laggy_interval": 0, 2024-03-11T21:41:58.447 INFO:teuthology.orchestra.run.smithi092.stdout: "features": 4540701547738038271, 2024-03-11T21:41:58.447 INFO:teuthology.orchestra.run.smithi092.stdout: "old_weight": 0, 2024-03-11T21:41:58.447 INFO:teuthology.orchestra.run.smithi092.stdout: "last_purged_snaps_scrub": "2024-03-11T21:41:44.563257+0000", 2024-03-11T21:41:58.447 INFO:teuthology.orchestra.run.smithi092.stdout: "dead_epoch": 251 2024-03-11T21:41:58.447 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:41:58.447 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:41:58.448 INFO:teuthology.orchestra.run.smithi092.stdout: "osd": 4, 2024-03-11T21:41:58.448 INFO:teuthology.orchestra.run.smithi092.stdout: "down_stamp": "0.000000", 2024-03-11T21:41:58.448 INFO:teuthology.orchestra.run.smithi092.stdout: "laggy_probability": 0, 2024-03-11T21:41:58.448 INFO:teuthology.orchestra.run.smithi092.stdout: "laggy_interval": 0, 2024-03-11T21:41:58.448 INFO:teuthology.orchestra.run.smithi092.stdout: "features": 4540701547738038271, 2024-03-11T21:41:58.448 INFO:teuthology.orchestra.run.smithi092.stdout: "old_weight": 0, 2024-03-11T21:41:58.448 INFO:teuthology.orchestra.run.smithi092.stdout: "last_purged_snaps_scrub": "2024-03-11T21:40:43.574292+0000", 2024-03-11T21:41:58.448 INFO:teuthology.orchestra.run.smithi092.stdout: "dead_epoch": 0 2024-03-11T21:41:58.448 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:41:58.448 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:41:58.448 INFO:teuthology.orchestra.run.smithi092.stdout: "osd": 5, 2024-03-11T21:41:58.448 INFO:teuthology.orchestra.run.smithi092.stdout: "down_stamp": "0.000000", 2024-03-11T21:41:58.448 INFO:teuthology.orchestra.run.smithi092.stdout: "laggy_probability": 0, 2024-03-11T21:41:58.448 INFO:teuthology.orchestra.run.smithi092.stdout: "laggy_interval": 0, 2024-03-11T21:41:58.448 INFO:teuthology.orchestra.run.smithi092.stdout: "features": 4540701547738038271, 2024-03-11T21:41:58.449 INFO:teuthology.orchestra.run.smithi092.stdout: "old_weight": 0, 2024-03-11T21:41:58.449 INFO:teuthology.orchestra.run.smithi092.stdout: "last_purged_snaps_scrub": "2024-03-11T21:41:01.995024+0000", 2024-03-11T21:41:58.449 INFO:teuthology.orchestra.run.smithi092.stdout: "dead_epoch": 0 2024-03-11T21:41:58.449 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:41:58.449 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:41:58.449 INFO:teuthology.orchestra.run.smithi092.stdout: "osd": 6, 2024-03-11T21:41:58.449 INFO:teuthology.orchestra.run.smithi092.stdout: "down_stamp": "0.000000", 2024-03-11T21:41:58.449 INFO:teuthology.orchestra.run.smithi092.stdout: "laggy_probability": 0, 2024-03-11T21:41:58.449 INFO:teuthology.orchestra.run.smithi092.stdout: "laggy_interval": 0, 2024-03-11T21:41:58.449 INFO:teuthology.orchestra.run.smithi092.stdout: "features": 4540701547738038271, 2024-03-11T21:41:58.449 INFO:teuthology.orchestra.run.smithi092.stdout: "old_weight": 0, 2024-03-11T21:41:58.449 INFO:teuthology.orchestra.run.smithi092.stdout: "last_purged_snaps_scrub": "2024-03-11T21:41:52.523555+0000", 2024-03-11T21:41:58.449 INFO:teuthology.orchestra.run.smithi092.stdout: "dead_epoch": 0 2024-03-11T21:41:58.450 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:41:58.450 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:41:58.450 INFO:teuthology.orchestra.run.smithi092.stdout: "osd": 7, 2024-03-11T21:41:58.450 INFO:teuthology.orchestra.run.smithi092.stdout: "down_stamp": "0.000000", 2024-03-11T21:41:58.450 INFO:teuthology.orchestra.run.smithi092.stdout: "laggy_probability": 0, 2024-03-11T21:41:58.450 INFO:teuthology.orchestra.run.smithi092.stdout: "laggy_interval": 0, 2024-03-11T21:41:58.450 INFO:teuthology.orchestra.run.smithi092.stdout: "features": 4540701547738038271, 2024-03-11T21:41:58.450 INFO:teuthology.orchestra.run.smithi092.stdout: "old_weight": 0, 2024-03-11T21:41:58.450 INFO:teuthology.orchestra.run.smithi092.stdout: "last_purged_snaps_scrub": "2024-03-11T21:41:46.204948+0000", 2024-03-11T21:41:58.450 INFO:teuthology.orchestra.run.smithi092.stdout: "dead_epoch": 0 2024-03-11T21:41:58.450 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-03-11T21:41:58.450 INFO:teuthology.orchestra.run.smithi092.stdout: ], 2024-03-11T21:41:58.450 INFO:teuthology.orchestra.run.smithi092.stdout: "pg_upmap": [ 2024-03-11T21:41:58.450 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:41:58.450 INFO:teuthology.orchestra.run.smithi092.stdout: "pgid": "6.a", 2024-03-11T21:41:58.451 INFO:teuthology.orchestra.run.smithi092.stdout: "osds": [ 2024-03-11T21:41:58.451 INFO:teuthology.orchestra.run.smithi092.stdout: 4, 2024-03-11T21:41:58.451 INFO:teuthology.orchestra.run.smithi092.stdout: 0, 2024-03-11T21:41:58.451 INFO:teuthology.orchestra.run.smithi092.stdout: 3 2024-03-11T21:41:58.451 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-03-11T21:41:58.451 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-03-11T21:41:58.451 INFO:teuthology.orchestra.run.smithi092.stdout: ], 2024-03-11T21:41:58.451 INFO:teuthology.orchestra.run.smithi092.stdout: "pg_upmap_items": [ 2024-03-11T21:41:58.451 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:41:58.451 INFO:teuthology.orchestra.run.smithi092.stdout: "pgid": "2.18", 2024-03-11T21:41:58.451 INFO:teuthology.orchestra.run.smithi092.stdout: "mappings": [ 2024-03-11T21:41:58.451 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:41:58.451 INFO:teuthology.orchestra.run.smithi092.stdout: "from": 7, 2024-03-11T21:41:58.451 INFO:teuthology.orchestra.run.smithi092.stdout: "to": 2 2024-03-11T21:41:58.451 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-03-11T21:41:58.451 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-03-11T21:41:58.452 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:41:58.452 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:41:58.452 INFO:teuthology.orchestra.run.smithi092.stdout: "pgid": "4.6", 2024-03-11T21:41:58.452 INFO:teuthology.orchestra.run.smithi092.stdout: "mappings": [ 2024-03-11T21:41:58.452 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:41:58.452 INFO:teuthology.orchestra.run.smithi092.stdout: "from": 1, 2024-03-11T21:41:58.452 INFO:teuthology.orchestra.run.smithi092.stdout: "to": 5 2024-03-11T21:41:58.452 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-03-11T21:41:58.452 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-03-11T21:41:58.452 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:41:58.452 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:41:58.452 INFO:teuthology.orchestra.run.smithi092.stdout: "pgid": "4.f", 2024-03-11T21:41:58.452 INFO:teuthology.orchestra.run.smithi092.stdout: "mappings": [ 2024-03-11T21:41:58.452 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:41:58.453 INFO:teuthology.orchestra.run.smithi092.stdout: "from": 1, 2024-03-11T21:41:58.453 INFO:teuthology.orchestra.run.smithi092.stdout: "to": 2 2024-03-11T21:41:58.453 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-03-11T21:41:58.453 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-03-11T21:41:58.453 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:41:58.453 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:41:58.453 INFO:teuthology.orchestra.run.smithi092.stdout: "pgid": "4.13", 2024-03-11T21:41:58.453 INFO:teuthology.orchestra.run.smithi092.stdout: "mappings": [ 2024-03-11T21:41:58.453 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:41:58.453 INFO:teuthology.orchestra.run.smithi092.stdout: "from": 1, 2024-03-11T21:41:58.453 INFO:teuthology.orchestra.run.smithi092.stdout: "to": 2 2024-03-11T21:41:58.453 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-03-11T21:41:58.453 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-03-11T21:41:58.453 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:41:58.453 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:41:58.454 INFO:teuthology.orchestra.run.smithi092.stdout: "pgid": "4.17", 2024-03-11T21:41:58.454 INFO:teuthology.orchestra.run.smithi092.stdout: "mappings": [ 2024-03-11T21:41:58.454 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:41:58.454 INFO:teuthology.orchestra.run.smithi092.stdout: "from": 1, 2024-03-11T21:41:58.454 INFO:teuthology.orchestra.run.smithi092.stdout: "to": 5 2024-03-11T21:41:58.454 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-03-11T21:41:58.454 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-03-11T21:41:58.454 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:41:58.454 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:41:58.454 INFO:teuthology.orchestra.run.smithi092.stdout: "pgid": "4.1f", 2024-03-11T21:41:58.454 INFO:teuthology.orchestra.run.smithi092.stdout: "mappings": [ 2024-03-11T21:41:58.454 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:41:58.454 INFO:teuthology.orchestra.run.smithi092.stdout: "from": 1, 2024-03-11T21:41:58.454 INFO:teuthology.orchestra.run.smithi092.stdout: "to": 0 2024-03-11T21:41:58.455 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-03-11T21:41:58.455 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-03-11T21:41:58.455 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:41:58.455 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:41:58.455 INFO:teuthology.orchestra.run.smithi092.stdout: "pgid": "6.2", 2024-03-11T21:41:58.455 INFO:teuthology.orchestra.run.smithi092.stdout: "mappings": [ 2024-03-11T21:41:58.455 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:41:58.455 INFO:teuthology.orchestra.run.smithi092.stdout: "from": 3, 2024-03-11T21:41:58.455 INFO:teuthology.orchestra.run.smithi092.stdout: "to": 0 2024-03-11T21:41:58.455 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-03-11T21:41:58.456 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-03-11T21:41:58.456 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:41:58.456 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:41:58.456 INFO:teuthology.orchestra.run.smithi092.stdout: "pgid": "6.8", 2024-03-11T21:41:58.456 INFO:teuthology.orchestra.run.smithi092.stdout: "mappings": [ 2024-03-11T21:41:58.456 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:41:58.456 INFO:teuthology.orchestra.run.smithi092.stdout: "from": 3, 2024-03-11T21:41:58.456 INFO:teuthology.orchestra.run.smithi092.stdout: "to": 0 2024-03-11T21:41:58.456 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-03-11T21:41:58.456 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-03-11T21:41:58.456 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-03-11T21:41:58.456 INFO:teuthology.orchestra.run.smithi092.stdout: ], 2024-03-11T21:41:58.456 INFO:teuthology.orchestra.run.smithi092.stdout: "pg_upmap_primaries": [], 2024-03-11T21:41:58.456 INFO:teuthology.orchestra.run.smithi092.stdout: "pg_temp": [ 2024-03-11T21:41:58.456 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:41:58.457 INFO:teuthology.orchestra.run.smithi092.stdout: "pgid": "10.11", 2024-03-11T21:41:58.457 INFO:teuthology.orchestra.run.smithi092.stdout: "osds": [ 2024-03-11T21:41:58.457 INFO:teuthology.orchestra.run.smithi092.stdout: 0, 2024-03-11T21:41:58.457 INFO:teuthology.orchestra.run.smithi092.stdout: 7 2024-03-11T21:41:58.457 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-03-11T21:41:58.457 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:41:58.457 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:41:58.457 INFO:teuthology.orchestra.run.smithi092.stdout: "pgid": "10.12", 2024-03-11T21:41:58.457 INFO:teuthology.orchestra.run.smithi092.stdout: "osds": [ 2024-03-11T21:41:58.457 INFO:teuthology.orchestra.run.smithi092.stdout: 0, 2024-03-11T21:41:58.457 INFO:teuthology.orchestra.run.smithi092.stdout: 2 2024-03-11T21:41:58.457 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-03-11T21:41:58.457 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:41:58.457 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:41:58.457 INFO:teuthology.orchestra.run.smithi092.stdout: "pgid": "10.13", 2024-03-11T21:41:58.458 INFO:teuthology.orchestra.run.smithi092.stdout: "osds": [ 2024-03-11T21:41:58.458 INFO:teuthology.orchestra.run.smithi092.stdout: 2, 2024-03-11T21:41:58.458 INFO:teuthology.orchestra.run.smithi092.stdout: 1 2024-03-11T21:41:58.458 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-03-11T21:41:58.458 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:41:58.458 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:41:58.458 INFO:teuthology.orchestra.run.smithi092.stdout: "pgid": "10.14", 2024-03-11T21:41:58.458 INFO:teuthology.orchestra.run.smithi092.stdout: "osds": [ 2024-03-11T21:41:58.458 INFO:teuthology.orchestra.run.smithi092.stdout: 0, 2024-03-11T21:41:58.458 INFO:teuthology.orchestra.run.smithi092.stdout: 4 2024-03-11T21:41:58.458 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-03-11T21:41:58.458 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:41:58.458 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:41:58.458 INFO:teuthology.orchestra.run.smithi092.stdout: "pgid": "10.15", 2024-03-11T21:41:58.458 INFO:teuthology.orchestra.run.smithi092.stdout: "osds": [ 2024-03-11T21:41:58.459 INFO:teuthology.orchestra.run.smithi092.stdout: 3, 2024-03-11T21:41:58.459 INFO:teuthology.orchestra.run.smithi092.stdout: 1 2024-03-11T21:41:58.459 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-03-11T21:41:58.459 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:41:58.459 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:41:58.459 INFO:teuthology.orchestra.run.smithi092.stdout: "pgid": "10.16", 2024-03-11T21:41:58.459 INFO:teuthology.orchestra.run.smithi092.stdout: "osds": [ 2024-03-11T21:41:58.459 INFO:teuthology.orchestra.run.smithi092.stdout: 0, 2024-03-11T21:41:58.459 INFO:teuthology.orchestra.run.smithi092.stdout: 5 2024-03-11T21:41:58.459 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-03-11T21:41:58.459 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:41:58.459 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:41:58.459 INFO:teuthology.orchestra.run.smithi092.stdout: "pgid": "10.17", 2024-03-11T21:41:58.459 INFO:teuthology.orchestra.run.smithi092.stdout: "osds": [ 2024-03-11T21:41:58.459 INFO:teuthology.orchestra.run.smithi092.stdout: 3, 2024-03-11T21:41:58.460 INFO:teuthology.orchestra.run.smithi092.stdout: 6 2024-03-11T21:41:58.460 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-03-11T21:41:58.460 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:41:58.460 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:41:58.460 INFO:teuthology.orchestra.run.smithi092.stdout: "pgid": "10.18", 2024-03-11T21:41:58.460 INFO:teuthology.orchestra.run.smithi092.stdout: "osds": [ 2024-03-11T21:41:58.460 INFO:teuthology.orchestra.run.smithi092.stdout: 2, 2024-03-11T21:41:58.460 INFO:teuthology.orchestra.run.smithi092.stdout: 1 2024-03-11T21:41:58.460 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-03-11T21:41:58.460 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:41:58.460 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:41:58.460 INFO:teuthology.orchestra.run.smithi092.stdout: "pgid": "10.19", 2024-03-11T21:41:58.460 INFO:teuthology.orchestra.run.smithi092.stdout: "osds": [ 2024-03-11T21:41:58.460 INFO:teuthology.orchestra.run.smithi092.stdout: 2, 2024-03-11T21:41:58.460 INFO:teuthology.orchestra.run.smithi092.stdout: 1 2024-03-11T21:41:58.461 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-03-11T21:41:58.461 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-03-11T21:41:58.461 INFO:teuthology.orchestra.run.smithi092.stdout: ], 2024-03-11T21:41:58.461 INFO:teuthology.orchestra.run.smithi092.stdout: "primary_temp": [], 2024-03-11T21:41:58.461 INFO:teuthology.orchestra.run.smithi092.stdout: "blocklist": { 2024-03-11T21:41:58.461 INFO:teuthology.orchestra.run.smithi092.stdout: "172.21.15.92:0/303695351": "2024-03-12T21:26:20.943013+0000", 2024-03-11T21:41:58.461 INFO:teuthology.orchestra.run.smithi092.stdout: "172.21.15.92:0/437911229": "2024-03-12T21:26:20.943013+0000", 2024-03-11T21:41:58.461 INFO:teuthology.orchestra.run.smithi092.stdout: "172.21.15.92:0/4246054050": "2024-03-12T21:26:20.943013+0000", 2024-03-11T21:41:58.461 INFO:teuthology.orchestra.run.smithi092.stdout: "172.21.15.92:0/2912406656": "2024-03-12T21:26:20.943013+0000", 2024-03-11T21:41:58.461 INFO:teuthology.orchestra.run.smithi092.stdout: "172.21.15.92:0/2706821241": "2024-03-12T21:26:20.943013+0000", 2024-03-11T21:41:58.461 INFO:teuthology.orchestra.run.smithi092.stdout: "172.21.15.92:0/3817549754": "2024-03-12T21:21:09.473916+0000", 2024-03-11T21:41:58.461 INFO:teuthology.orchestra.run.smithi092.stdout: "172.21.15.92:0/2211594569": "2024-03-12T21:21:09.473916+0000", 2024-03-11T21:41:58.461 INFO:teuthology.orchestra.run.smithi092.stdout: "172.21.15.92:0/3569900179": "2024-03-12T21:20:43.677016+0000", 2024-03-11T21:41:58.461 INFO:teuthology.orchestra.run.smithi092.stdout: "172.21.15.92:0/3380359534": "2024-03-12T21:20:43.677016+0000", 2024-03-11T21:41:58.461 INFO:teuthology.orchestra.run.smithi092.stdout: "172.21.15.92:0/2509373130": "2024-03-12T21:20:43.677016+0000", 2024-03-11T21:41:58.462 INFO:teuthology.orchestra.run.smithi092.stdout: "172.21.15.92:6800/2343383852": "2024-03-12T21:26:20.943013+0000", 2024-03-11T21:41:58.462 INFO:teuthology.orchestra.run.smithi092.stdout: "172.21.15.92:6800/1559007462": "2024-03-12T21:20:43.677016+0000", 2024-03-11T21:41:58.462 INFO:teuthology.orchestra.run.smithi092.stdout: "172.21.15.92:6800/1270902819": "2024-03-12T21:21:09.473916+0000", 2024-03-11T21:41:58.462 INFO:teuthology.orchestra.run.smithi092.stdout: "172.21.15.92:0/3562645553": "2024-03-12T21:21:09.473916+0000" 2024-03-11T21:41:58.462 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:41:58.462 INFO:teuthology.orchestra.run.smithi092.stdout: "range_blocklist": {}, 2024-03-11T21:41:58.462 INFO:teuthology.orchestra.run.smithi092.stdout: "erasure_code_profiles": { 2024-03-11T21:41:58.462 INFO:teuthology.orchestra.run.smithi092.stdout: "default": { 2024-03-11T21:41:58.462 INFO:teuthology.orchestra.run.smithi092.stdout: "crush-failure-domain": "osd", 2024-03-11T21:41:58.462 INFO:teuthology.orchestra.run.smithi092.stdout: "k": "2", 2024-03-11T21:41:58.462 INFO:teuthology.orchestra.run.smithi092.stdout: "m": "1", 2024-03-11T21:41:58.462 INFO:teuthology.orchestra.run.smithi092.stdout: "plugin": "jerasure", 2024-03-11T21:41:58.462 INFO:teuthology.orchestra.run.smithi092.stdout: "technique": "reed_sol_van" 2024-03-11T21:41:58.462 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-03-11T21:41:58.462 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:41:58.463 INFO:teuthology.orchestra.run.smithi092.stdout: "removed_snaps_queue": [], 2024-03-11T21:41:58.463 INFO:teuthology.orchestra.run.smithi092.stdout: "new_removed_snaps": [], 2024-03-11T21:41:58.463 INFO:teuthology.orchestra.run.smithi092.stdout: "new_purged_snaps": [], 2024-03-11T21:41:58.463 INFO:teuthology.orchestra.run.smithi092.stdout: "crush_node_flags": {}, 2024-03-11T21:41:58.463 INFO:teuthology.orchestra.run.smithi092.stdout: "device_class_flags": {}, 2024-03-11T21:41:58.463 INFO:teuthology.orchestra.run.smithi092.stdout: "stretch_mode": { 2024-03-11T21:41:58.463 INFO:teuthology.orchestra.run.smithi092.stdout: "stretch_mode_enabled": false, 2024-03-11T21:41:58.463 INFO:teuthology.orchestra.run.smithi092.stdout: "stretch_bucket_count": 0, 2024-03-11T21:41:58.463 INFO:teuthology.orchestra.run.smithi092.stdout: "degraded_stretch_mode": 0, 2024-03-11T21:41:58.463 INFO:teuthology.orchestra.run.smithi092.stdout: "recovering_stretch_mode": 0, 2024-03-11T21:41:58.463 INFO:teuthology.orchestra.run.smithi092.stdout: "stretch_mode_bucket": 0 2024-03-11T21:41:58.463 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-03-11T21:41:58.463 INFO:teuthology.orchestra.run.smithi092.stdout:} 2024-03-11T21:41:58.463 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:41:58.464 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:41:58.464 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:41:58.565 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:41:58.581 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:41:58 smithi097 ceph-mon[33317]: Health check update: Degraded data redundancy: 17932/260031 objects degraded (6.896%), 4 pgs degraded (PG_DEGRADED) 2024-03-11T21:41:58.594 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:41:58.595 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:41:58.595 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:41:58.615 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:41:58 smithi092 ceph-mon[29572]: Health check update: Degraded data redundancy: 17932/260031 objects degraded (6.896%), 4 pgs degraded (PG_DEGRADED) 2024-03-11T21:41:58.615 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:41:58 smithi092 ceph-mon[37563]: Health check update: Degraded data redundancy: 17932/260031 objects degraded (6.896%), 4 pgs degraded (PG_DEGRADED) 2024-03-11T21:41:58.696 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:41:58.720 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:41:58.720 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:41:58.721 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:41:58.821 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:41:58.839 INFO:tasks.radosbench.radosbench.0.smithi092.stdout: 66 16 90614 90598 85.7818 39.1875 0.00963726 0.0116515 2024-03-11T21:41:58.850 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:41:58.851 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:41:58.852 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:41:58.952 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:41:58.998 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:41:58.998 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:41:58.999 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:41:59.100 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:41:59.131 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:41:59.132 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:41:59.132 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:41:59.233 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:41:59.264 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:41:59.265 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:41:59.265 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:41:59.366 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:41:59.395 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:41:59.396 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:41:59.396 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:41:59.497 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:41:59.537 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:41:59.538 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:41:59.538 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:41:59.581 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:41:59 smithi097 ceph-mon[33317]: pgmap v768: 158 pgs: 149 active+clean, 4 active+recovery_wait+undersized+degraded+remapped, 5 active+recovering+undersized+remapped; 5.3 GiB data, 21 GiB used, 695 GiB / 715 GiB avail; 639 B/s rd, 37 MiB/s wr, 591 op/s; 17932/260031 objects degraded (6.896%); 16370/260031 objects misplaced (6.295%); 12 MiB/s, 188 objects/s recovering 2024-03-11T21:41:59.581 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:41:59 smithi097 ceph-mon[33317]: from='client.? v1:172.21.15.92:0/2062645807' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json-pretty"}]: dispatch 2024-03-11T21:41:59.615 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:41:59 smithi092 ceph-mon[29572]: pgmap v768: 158 pgs: 149 active+clean, 4 active+recovery_wait+undersized+degraded+remapped, 5 active+recovering+undersized+remapped; 5.3 GiB data, 21 GiB used, 695 GiB / 715 GiB avail; 639 B/s rd, 37 MiB/s wr, 591 op/s; 17932/260031 objects degraded (6.896%); 16370/260031 objects misplaced (6.295%); 12 MiB/s, 188 objects/s recovering 2024-03-11T21:41:59.615 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:41:59 smithi092 ceph-mon[29572]: from='client.? v1:172.21.15.92:0/2062645807' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json-pretty"}]: dispatch 2024-03-11T21:41:59.615 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:41:59 smithi092 ceph-mon[37563]: pgmap v768: 158 pgs: 149 active+clean, 4 active+recovery_wait+undersized+degraded+remapped, 5 active+recovering+undersized+remapped; 5.3 GiB data, 21 GiB used, 695 GiB / 715 GiB avail; 639 B/s rd, 37 MiB/s wr, 591 op/s; 17932/260031 objects degraded (6.896%); 16370/260031 objects misplaced (6.295%); 12 MiB/s, 188 objects/s recovering 2024-03-11T21:41:59.615 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:41:59 smithi092 ceph-mon[37563]: from='client.? v1:172.21.15.92:0/2062645807' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json-pretty"}]: dispatch 2024-03-11T21:41:59.639 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:41:59.662 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:41:59.663 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:41:59.663 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:41:59.764 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:41:59.797 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:41:59.798 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:41:59.798 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:41:59.839 INFO:tasks.radosbench.radosbench.0.smithi092.stdout: 67 16 91396 91380 85.2309 48.875 0.0243028 0.011727 2024-03-11T21:41:59.899 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:41:59.927 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:41:59.927 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:41:59.928 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:42:00.028 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:42:00.069 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:42:00.071 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:42:00.071 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:42:00.172 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:42:00.207 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:42:00.208 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:42:00.208 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:42:00.309 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:42:00.349 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:42:00.350 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:42:00.351 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:42:00.451 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:42:00.484 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:42:00.484 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:42:00.485 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:42:00.581 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:42:00 smithi097 ceph-mon[33317]: pgmap v769: 158 pgs: 149 active+clean, 4 active+recovery_wait+undersized+degraded+remapped, 5 active+recovering+undersized+remapped; 5.5 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 1.1 KiB/s rd, 59 MiB/s wr, 941 op/s; 17932/268977 objects degraded (6.667%); 15547/268977 objects misplaced (5.780%); 13 MiB/s, 200 objects/s recovering 2024-03-11T21:42:00.585 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:42:00.611 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:42:00.611 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:42:00.612 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:42:00.615 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:42:00 smithi092 ceph-mon[29572]: pgmap v769: 158 pgs: 149 active+clean, 4 active+recovery_wait+undersized+degraded+remapped, 5 active+recovering+undersized+remapped; 5.5 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 1.1 KiB/s rd, 59 MiB/s wr, 941 op/s; 17932/268977 objects degraded (6.667%); 15547/268977 objects misplaced (5.780%); 13 MiB/s, 200 objects/s recovering 2024-03-11T21:42:00.615 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:42:00 smithi092 ceph-mon[37563]: pgmap v769: 158 pgs: 149 active+clean, 4 active+recovery_wait+undersized+degraded+remapped, 5 active+recovering+undersized+remapped; 5.5 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 1.1 KiB/s rd, 59 MiB/s wr, 941 op/s; 17932/268977 objects degraded (6.667%); 15547/268977 objects misplaced (5.780%); 13 MiB/s, 200 objects/s recovering 2024-03-11T21:42:00.712 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:42:00.743 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:42:00.744 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:42:00.745 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:42:00.839 INFO:tasks.radosbench.radosbench.0.smithi092.stdout: 68 16 92271 92255 84.7817 54.6875 0.0164085 0.0117902 2024-03-11T21:42:00.845 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:42:00.896 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:42:00.896 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:42:00.897 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:42:00.998 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:42:01.025 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:42:01.026 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:42:01.026 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:42:01.127 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:42:01.162 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:42:01.162 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:42:01.163 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:42:01.264 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:42:01.331 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:42:01.332 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:42:01.332 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:42:01.433 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:42:01.463 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:42:01.463 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:42:01.464 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:42:01.564 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:42:01.581 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:42:01 smithi097 ceph-mon[33317]: purged_snaps scrub starts 2024-03-11T21:42:01.581 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:42:01 smithi097 ceph-mon[33317]: purged_snaps scrub ok 2024-03-11T21:42:01.581 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:42:01 smithi097 ceph-mon[33317]: osdmap e385: 8 total, 8 up, 8 in 2024-03-11T21:42:01.594 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:42:01.594 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:42:01.595 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:42:01.615 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:42:01 smithi092 ceph-mon[29572]: purged_snaps scrub starts 2024-03-11T21:42:01.615 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:42:01 smithi092 ceph-mon[29572]: purged_snaps scrub ok 2024-03-11T21:42:01.615 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:42:01 smithi092 ceph-mon[29572]: osdmap e385: 8 total, 8 up, 8 in 2024-03-11T21:42:01.616 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:42:01 smithi092 ceph-mon[37563]: purged_snaps scrub starts 2024-03-11T21:42:01.616 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:42:01 smithi092 ceph-mon[37563]: purged_snaps scrub ok 2024-03-11T21:42:01.616 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:42:01 smithi092 ceph-mon[37563]: osdmap e385: 8 total, 8 up, 8 in 2024-03-11T21:42:01.695 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:42:01.734 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:42:01.734 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:42:01.735 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:42:01.836 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:42:01.839 INFO:tasks.radosbench.radosbench.0.smithi092.stdout: 69 16 93124 93108 84.3256 53.3125 0.00739333 0.0118535 2024-03-11T21:42:01.877 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:42:01.878 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:42:01.878 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:42:01.979 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:42:02.008 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:42:02.009 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:42:02.009 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:42:02.110 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:42:02.144 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:42:02.145 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:42:02.145 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:42:02.246 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:42:02.282 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:42:02.283 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:42:02.283 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:42:02.384 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:42:02.429 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:42:02.430 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:42:02.430 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:42:02.531 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:42:02.569 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:42:02.570 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:42:02.570 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:42:02.581 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:42:02 smithi097 ceph-mon[33317]: pgmap v771: 158 pgs: 149 active+clean, 4 active+recovery_wait+undersized+degraded+remapped, 5 active+recovering+undersized+remapped; 5.5 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 1.1 KiB/s rd, 40 MiB/s wr, 638 op/s; 17932/269706 objects degraded (6.649%); 14847/269706 objects misplaced (5.505%); 17 MiB/s, 264 objects/s recovering 2024-03-11T21:42:02.615 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:42:02 smithi092 ceph-mon[29572]: pgmap v771: 158 pgs: 149 active+clean, 4 active+recovery_wait+undersized+degraded+remapped, 5 active+recovering+undersized+remapped; 5.5 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 1.1 KiB/s rd, 40 MiB/s wr, 638 op/s; 17932/269706 objects degraded (6.649%); 14847/269706 objects misplaced (5.505%); 17 MiB/s, 264 objects/s recovering 2024-03-11T21:42:02.615 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:42:02 smithi092 ceph-mon[37563]: pgmap v771: 158 pgs: 149 active+clean, 4 active+recovery_wait+undersized+degraded+remapped, 5 active+recovering+undersized+remapped; 5.5 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 1.1 KiB/s rd, 40 MiB/s wr, 638 op/s; 17932/269706 objects degraded (6.649%); 14847/269706 objects misplaced (5.505%); 17 MiB/s, 264 objects/s recovering 2024-03-11T21:42:02.672 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:42:02.720 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:42:02.721 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:42:02.721 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:42:02.822 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:42:02.839 INFO:tasks.radosbench.radosbench.0.smithi092.stdout: 70 16 94095 94079 83.9878 60.6875 0.0128727 0.0119012 2024-03-11T21:42:02.864 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:42:02.865 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:42:02.865 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:42:02.966 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:42:02.998 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:42:02.998 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:42:02.999 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:42:03.100 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:42:03.132 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:42:03.133 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:42:03.134 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:42:03.234 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:42:03.261 INFO:journalctl@ceph.iscsi.iscsi.a.smithi097.stdout:Mar 11 21:42:02 smithi097 ceph-00db0016-dfed-11ee-95c8-87774f69a715-iscsi-iscsi-a[60943]: debug there is no tcmu-runner data available 2024-03-11T21:42:03.265 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:42:03.265 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:42:03.266 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:42:03.366 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:42:03.407 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:42:03.408 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:42:03.408 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:42:03.503 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:42:03 smithi092 ceph-mon[29572]: from='client.24469 v1:172.21.15.97:0/404208310' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:42:03.503 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:42:03 smithi092 ceph-mon[37563]: from='client.24469 v1:172.21.15.97:0/404208310' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:42:03.509 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:42:03.543 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:42:03.544 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:42:03.544 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:42:03.560 INFO:tasks.thrashosds.thrasher:j is {'epoch': 384, 'fsid': '00db0016-dfed-11ee-95c8-87774f69a715', 'created': '2024-03-11T21:20:10.488059+0000', 'modified': '2024-03-11T21:41:53.407479+0000', 'last_up_change': '2024-03-11T21:36:34.026798+0000', 'last_in_change': '2024-03-11T21:25:08.189557+0000', 'flags': 'noscrub,sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit', 'flags_num': 5801984, 'flags_set': ['noscrub', 'pglog_hardlimit', 'purged_snapdirs', 'recovery_deletes', 'sortbitwise'], 'crush_version': 19, 'full_ratio': 0.949999988079071, 'backfillfull_ratio': 0.8999999761581421, 'nearfull_ratio': 0.8500000238418579, 'cluster_snapshot': '', 'pool_max': 10, 'max_osd': 8, 'require_min_compat_client': 'luminous', 'min_compat_client': 'luminous', 'require_osd_release': 'squid', 'allow_crimson': False, 'pools': [{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-11T21:23:23.805029+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '121', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 7.889999866485596, 'score_stable': 7.889999866485596, 'optimal_score': 0.3799999952316284, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': '.rgw.root', 'create_time': '2024-03-11T21:25:37.044504+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '68', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rgw': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'datapool', 'create_time': '2024-03-11T21:25:37.280820+0000', 'flags': 8193, 'flags_names': 'hashpspool,selfmanaged_snaps', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 3, 'pg_placement_num': 3, 'pg_placement_num_target': 3, 'pg_num_target': 3, 'pg_num_pending': 3, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '71', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 2, 'snap_epoch': 71, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 2.6500000953674316, 'score_stable': 2.6500000953674316, 'optimal_score': 0.8799999952316284, 'raw_score_acting': 2.3299999237060547, 'raw_score_stable': 2.3299999237060547, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'default.rgw.log', 'create_time': '2024-03-11T21:25:38.850207+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '70', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rgw': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 2.25, 'score_stable': 2.25, 'optimal_score': 1, 'raw_score_acting': 2.25, 'raw_score_stable': 2.25, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 5, 'pool_name': 'default.rgw.control', 'create_time': '2024-03-11T21:25:40.794686+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '72', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rgw': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.25, 'score_stable': 1.25, 'optimal_score': 1, 'raw_score_acting': 1.25, 'raw_score_stable': 1.25, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 6, 'pool_name': 'default.rgw.meta', 'create_time': '2024-03-11T21:25:42.872077+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '74', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4}, 'application_metadata': {'rgw': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.75, 'score_stable': 1.75, 'optimal_score': 1, 'raw_score_acting': 1.75, 'raw_score_stable': 1.75, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 10, 'pool_name': 'unique_pool_3', 'create_time': '2024-03-11T21:40:37.970897+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 26, 'pg_placement_num': 26, 'pg_placement_num_target': 26, 'pg_num_target': 26, 'pg_num_pending': 26, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '374', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '372', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rados': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.850000023841858, 'score_stable': 2.4600000381469727, 'optimal_score': 1, 'raw_score_acting': 1.850000023841858, 'raw_score_stable': 2.4600000381469727, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}], 'osds': [{'osd': 0, 'uuid': '79f15843-f0ec-44aa-931e-4e07a74d3768', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 8, 'up_thru': 375, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.92:6801', 'nonce': 2844100201}]}, 'cluster_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.92:6802', 'nonce': 2844100201}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.92:6804', 'nonce': 2844100201}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.92:6803', 'nonce': 2844100201}]}, 'public_addr': '172.21.15.92:6801/2844100201', 'cluster_addr': '172.21.15.92:6802/2844100201', 'heartbeat_back_addr': '172.21.15.92:6804/2844100201', 'heartbeat_front_addr': '172.21.15.92:6803/2844100201', 'state': ['exists', 'up']}, {'osd': 1, 'uuid': '3a5a0e78-1293-4ace-a93f-69156098571d', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 13, 'up_thru': 374, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.92:6805', 'nonce': 3577871157}]}, 'cluster_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.92:6806', 'nonce': 3577871157}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.92:6808', 'nonce': 3577871157}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.92:6807', 'nonce': 3577871157}]}, 'public_addr': '172.21.15.92:6805/3577871157', 'cluster_addr': '172.21.15.92:6806/3577871157', 'heartbeat_back_addr': '172.21.15.92:6808/3577871157', 'heartbeat_front_addr': '172.21.15.92:6807/3577871157', 'state': ['exists', 'up']}, {'osd': 2, 'uuid': 'f207b743-0df4-40ed-a735-950e272fea60', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 18, 'up_thru': 375, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.92:6809', 'nonce': 760825152}]}, 'cluster_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.92:6810', 'nonce': 760825152}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.92:6812', 'nonce': 760825152}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.92:6811', 'nonce': 760825152}]}, 'public_addr': '172.21.15.92:6809/760825152', 'cluster_addr': '172.21.15.92:6810/760825152', 'heartbeat_back_addr': '172.21.15.92:6812/760825152', 'heartbeat_front_addr': '172.21.15.92:6811/760825152', 'state': ['exists', 'up']}, {'osd': 3, 'uuid': 'bbb6f69a-8f0f-47e6-937a-062e1902ae13', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 26, 'last_clean_end': 251, 'up_from': 277, 'up_thru': 375, 'down_at': 252, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.92:6813', 'nonce': 3215041236}]}, 'cluster_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.92:6814', 'nonce': 3215041236}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.92:6816', 'nonce': 3215041236}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.92:6815', 'nonce': 3215041236}]}, 'public_addr': '172.21.15.92:6813/3215041236', 'cluster_addr': '172.21.15.92:6814/3215041236', 'heartbeat_back_addr': '172.21.15.92:6816/3215041236', 'heartbeat_front_addr': '172.21.15.92:6815/3215041236', 'state': ['exists', 'up']}, {'osd': 4, 'uuid': '00266cbc-d2cf-4dee-8a96-3753ebc95ca8', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 32, 'up_thru': 380, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.97:6800', 'nonce': 420733620}]}, 'cluster_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.97:6801', 'nonce': 420733620}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.97:6803', 'nonce': 420733620}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.97:6802', 'nonce': 420733620}]}, 'public_addr': '172.21.15.97:6800/420733620', 'cluster_addr': '172.21.15.97:6801/420733620', 'heartbeat_back_addr': '172.21.15.97:6803/420733620', 'heartbeat_front_addr': '172.21.15.97:6802/420733620', 'state': ['exists', 'up']}, {'osd': 5, 'uuid': '593a6f6b-8874-40ae-8848-d51c5723a5e4', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 41, 'up_thru': 374, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.97:6804', 'nonce': 312056041}]}, 'cluster_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.97:6805', 'nonce': 312056041}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.97:6807', 'nonce': 312056041}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.97:6806', 'nonce': 312056041}]}, 'public_addr': '172.21.15.97:6804/312056041', 'cluster_addr': '172.21.15.97:6805/312056041', 'heartbeat_back_addr': '172.21.15.97:6807/312056041', 'heartbeat_front_addr': '172.21.15.97:6806/312056041', 'state': ['exists', 'up']}, {'osd': 6, 'uuid': '91035ecc-6f1a-4cec-bda6-090af9d53eff', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 50, 'up_thru': 374, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.97:6808', 'nonce': 178994999}]}, 'cluster_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.97:6809', 'nonce': 178994999}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.97:6811', 'nonce': 178994999}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.97:6810', 'nonce': 178994999}]}, 'public_addr': '172.21.15.97:6808/178994999', 'cluster_addr': '172.21.15.97:6809/178994999', 'heartbeat_back_addr': '172.21.15.97:6811/178994999', 'heartbeat_front_addr': '172.21.15.97:6810/178994999', 'state': ['exists', 'up']}, {'osd': 7, 'uuid': '3eb28c24-664a-4bd6-869d-6bf5db98f52f', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 60, 'up_thru': 356, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.97:6812', 'nonce': 3241560509}]}, 'cluster_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.97:6813', 'nonce': 3241560509}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.97:6815', 'nonce': 3241560509}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.97:6814', 'nonce': 3241560509}]}, 'public_addr': '172.21.15.97:6812/3241560509', 'cluster_addr': '172.21.15.97:6813/3241560509', 'heartbeat_back_addr': '172.21.15.97:6815/3241560509', 'heartbeat_front_addr': '172.21.15.97:6814/3241560509', 'state': ['exists', 'up']}], 'osd_xinfo': [{'osd': 0, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-03-11T21:40:47.505890+0000', 'dead_epoch': 0}, {'osd': 1, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-03-11T21:41:21.884755+0000', 'dead_epoch': 0}, {'osd': 2, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-03-11T21:41:15.172620+0000', 'dead_epoch': 0}, {'osd': 3, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-03-11T21:41:44.563257+0000', 'dead_epoch': 251}, {'osd': 4, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-03-11T21:40:43.574292+0000', 'dead_epoch': 0}, {'osd': 5, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-03-11T21:41:01.995024+0000', 'dead_epoch': 0}, {'osd': 6, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-03-11T21:41:52.523555+0000', 'dead_epoch': 0}, {'osd': 7, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-03-11T21:41:46.204948+0000', 'dead_epoch': 0}], 'pg_upmap': [{'pgid': '6.a', 'osds': [4, 0, 3]}], 'pg_upmap_items': [{'pgid': '2.18', 'mappings': [{'from': 7, 'to': 2}]}, {'pgid': '4.6', 'mappings': [{'from': 1, 'to': 5}]}, {'pgid': '4.f', 'mappings': [{'from': 1, 'to': 2}]}, {'pgid': '4.13', 'mappings': [{'from': 1, 'to': 2}]}, {'pgid': '4.17', 'mappings': [{'from': 1, 'to': 5}]}, {'pgid': '4.1f', 'mappings': [{'from': 1, 'to': 0}]}, {'pgid': '6.2', 'mappings': [{'from': 3, 'to': 0}]}, {'pgid': '6.8', 'mappings': [{'from': 3, 'to': 0}]}], 'pg_upmap_primaries': [], 'pg_temp': [{'pgid': '10.11', 'osds': [0, 7]}, {'pgid': '10.12', 'osds': [0, 2]}, {'pgid': '10.13', 'osds': [2, 1]}, {'pgid': '10.14', 'osds': [0, 4]}, {'pgid': '10.15', 'osds': [3, 1]}, {'pgid': '10.16', 'osds': [0, 5]}, {'pgid': '10.17', 'osds': [3, 6]}, {'pgid': '10.18', 'osds': [2, 1]}, {'pgid': '10.19', 'osds': [2, 1]}], 'primary_temp': [], 'blocklist': {'172.21.15.92:0/303695351': '2024-03-12T21:26:20.943013+0000', '172.21.15.92:0/437911229': '2024-03-12T21:26:20.943013+0000', '172.21.15.92:0/4246054050': '2024-03-12T21:26:20.943013+0000', '172.21.15.92:0/2912406656': '2024-03-12T21:26:20.943013+0000', '172.21.15.92:0/2706821241': '2024-03-12T21:26:20.943013+0000', '172.21.15.92:0/3817549754': '2024-03-12T21:21:09.473916+0000', '172.21.15.92:0/2211594569': '2024-03-12T21:21:09.473916+0000', '172.21.15.92:0/3569900179': '2024-03-12T21:20:43.677016+0000', '172.21.15.92:0/3380359534': '2024-03-12T21:20:43.677016+0000', '172.21.15.92:0/2509373130': '2024-03-12T21:20:43.677016+0000', '172.21.15.92:6800/2343383852': '2024-03-12T21:26:20.943013+0000', '172.21.15.92:6800/1559007462': '2024-03-12T21:20:43.677016+0000', '172.21.15.92:6800/1270902819': '2024-03-12T21:21:09.473916+0000', '172.21.15.92:0/3562645553': '2024-03-12T21:21:09.473916+0000'}, 'range_blocklist': {}, 'erasure_code_profiles': {'default': {'crush-failure-domain': 'osd', 'k': '2', 'm': '1', 'plugin': 'jerasure', 'technique': 'reed_sol_van'}}, 'removed_snaps_queue': [], 'new_removed_snaps': [], 'new_purged_snaps': [], 'crush_node_flags': {}, 'device_class_flags': {}, 'stretch_mode': {'stretch_mode_enabled': False, 'stretch_bucket_count': 0, 'degraded_stretch_mode': 0, 'recovering_stretch_mode': 0, 'stretch_mode_bucket': 0}} 2024-03-11T21:42:03.561 DEBUG:teuthology.orchestra.run.smithi092:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid 00db0016-dfed-11ee-95c8-87774f69a715 -- ceph pg dump --format=json 2024-03-11T21:42:03.581 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:42:03 smithi097 ceph-mon[33317]: from='client.24469 v1:172.21.15.97:0/404208310' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:42:03.645 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:42:03.704 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:42:03.704 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:42:03.705 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:42:03.806 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:42:03.835 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:42:03.836 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:42:03.836 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:42:03.839 INFO:tasks.radosbench.radosbench.0.smithi092.stdout: 71 16 94980 94964 83.5839 55.3125 0.0260355 0.0119583 2024-03-11T21:42:03.937 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:42:04.017 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:42:04.018 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:42:04.018 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:42:04.119 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:42:04.178 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:42:04.179 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:42:04.179 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:42:04.280 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:42:04.312 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:42:04.313 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:42:04.314 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:42:04.331 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:42:04 smithi092 ceph-mon[29572]: pgmap v772: 158 pgs: 149 active+clean, 4 active+recovery_wait+undersized+degraded+remapped, 5 active+recovering+undersized+remapped; 5.7 GiB data, 22 GiB used, 694 GiB / 715 GiB avail; 412 B/s rd, 49 MiB/s wr, 790 op/s; 17932/278331 objects degraded (6.443%); 14847/278331 objects misplaced (5.334%); 13 MiB/s, 212 objects/s recovering 2024-03-11T21:42:04.331 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:42:04 smithi092 ceph-mon[29572]: Health check update: Degraded data redundancy: 17932/269706 objects degraded (6.649%), 4 pgs degraded (PG_DEGRADED) 2024-03-11T21:42:04.331 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:42:04 smithi092 ceph-mon[29572]: osdmap e386: 8 total, 8 up, 8 in 2024-03-11T21:42:04.331 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:42:04 smithi092 ceph-mon[29572]: purged_snaps scrub starts 2024-03-11T21:42:04.332 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:42:04 smithi092 ceph-mon[29572]: purged_snaps scrub ok 2024-03-11T21:42:04.332 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:42:04 smithi092 ceph-mon[37563]: pgmap v772: 158 pgs: 149 active+clean, 4 active+recovery_wait+undersized+degraded+remapped, 5 active+recovering+undersized+remapped; 5.7 GiB data, 22 GiB used, 694 GiB / 715 GiB avail; 412 B/s rd, 49 MiB/s wr, 790 op/s; 17932/278331 objects degraded (6.443%); 14847/278331 objects misplaced (5.334%); 13 MiB/s, 212 objects/s recovering 2024-03-11T21:42:04.332 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:42:04 smithi092 ceph-mon[37563]: Health check update: Degraded data redundancy: 17932/269706 objects degraded (6.649%), 4 pgs degraded (PG_DEGRADED) 2024-03-11T21:42:04.332 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:42:04 smithi092 ceph-mon[37563]: osdmap e386: 8 total, 8 up, 8 in 2024-03-11T21:42:04.332 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:42:04 smithi092 ceph-mon[37563]: purged_snaps scrub starts 2024-03-11T21:42:04.332 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:42:04 smithi092 ceph-mon[37563]: purged_snaps scrub ok 2024-03-11T21:42:04.414 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:42:04.443 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:42:04.443 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:42:04.444 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:42:04.545 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:42:04.581 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:42:04 smithi097 ceph-mon[33317]: pgmap v772: 158 pgs: 149 active+clean, 4 active+recovery_wait+undersized+degraded+remapped, 5 active+recovering+undersized+remapped; 5.7 GiB data, 22 GiB used, 694 GiB / 715 GiB avail; 412 B/s rd, 49 MiB/s wr, 790 op/s; 17932/278331 objects degraded (6.443%); 14847/278331 objects misplaced (5.334%); 13 MiB/s, 212 objects/s recovering 2024-03-11T21:42:04.581 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:42:04 smithi097 ceph-mon[33317]: Health check update: Degraded data redundancy: 17932/269706 objects degraded (6.649%), 4 pgs degraded (PG_DEGRADED) 2024-03-11T21:42:04.581 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:42:04 smithi097 ceph-mon[33317]: osdmap e386: 8 total, 8 up, 8 in 2024-03-11T21:42:04.581 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:42:04 smithi097 ceph-mon[33317]: purged_snaps scrub starts 2024-03-11T21:42:04.581 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:42:04 smithi097 ceph-mon[33317]: purged_snaps scrub ok 2024-03-11T21:42:04.591 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:42:04.592 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:42:04.592 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:42:04.694 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:42:04.726 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:42:04.727 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:42:04.728 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:42:04.829 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:42:04.839 INFO:tasks.radosbench.radosbench.0.smithi092.stdout: 72 16 95840 95824 83.1695 53.75 0.0068123 0.0120181 2024-03-11T21:42:04.861 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:42:04.861 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:42:04.862 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:42:04.963 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:42:05.008 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:42:05.009 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:42:05.009 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:42:05.110 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:42:05.139 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:42:05.140 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:42:05.140 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:42:05.241 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:42:05.288 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:42:05.289 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:42:05.289 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:42:05.390 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:42:05.420 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:42:05.421 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:42:05.421 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:42:05.495 INFO:teuthology.orchestra.run.smithi092.stdout:ERROR: (22) Invalid argument 2024-03-11T21:42:05.495 INFO:teuthology.orchestra.run.smithi092.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-11T21:42:05.522 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:42:05.553 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:42:05.553 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:42:05.554 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:42:05.615 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:42:05 smithi092 ceph-mon[29572]: osdmap e387: 8 total, 8 up, 8 in 2024-03-11T21:42:05.615 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:42:05 smithi092 ceph-mon[37563]: osdmap e387: 8 total, 8 up, 8 in 2024-03-11T21:42:05.654 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:42:05.686 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:42:05.687 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:42:05.687 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:42:05.788 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:42:05.829 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:42:05.829 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:42:05.830 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:42:05.831 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:42:05 smithi097 ceph-mon[33317]: osdmap e387: 8 total, 8 up, 8 in 2024-03-11T21:42:05.839 INFO:tasks.radosbench.radosbench.0.smithi092.stdout: 73 16 96759 96743 82.8169 57.4375 0.00645709 0.0120692 2024-03-11T21:42:05.931 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:42:05.963 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:42:05.964 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:42:05.964 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:42:06.065 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:42:06.095 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:42:06.095 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:42:06.096 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:42:06.196 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:42:06.226 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:42:06.226 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:42:06.227 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:42:06.327 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:42:06.358 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:42:06.358 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:42:06.359 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:42:06.459 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:42:06.496 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:42:06.496 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:42:06.497 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:42:06.598 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:42:06.632 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:42:06.632 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:42:06.632 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:42:06.733 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:42:06.765 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:42:06.766 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:42:06.766 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:42:06.831 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:42:06 smithi097 ceph-mon[33317]: pgmap v775: 158 pgs: 149 active+clean, 4 active+recovery_wait+undersized+degraded+remapped, 5 active+recovering+undersized+remapped; 5.7 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 852 B/s rd, 45 MiB/s wr, 719 op/s; 17932/281919 objects degraded (6.361%); 12908/281919 objects misplaced (4.579%); 20 MiB/s, 314 objects/s recovering 2024-03-11T21:42:06.840 INFO:tasks.radosbench.radosbench.0.smithi092.stdout: 74 16 97769 97753 82.5508 63.125 0.0117433 0.0121087 2024-03-11T21:42:06.865 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:42:06 smithi092 ceph-mon[29572]: pgmap v775: 158 pgs: 149 active+clean, 4 active+recovery_wait+undersized+degraded+remapped, 5 active+recovering+undersized+remapped; 5.7 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 852 B/s rd, 45 MiB/s wr, 719 op/s; 17932/281919 objects degraded (6.361%); 12908/281919 objects misplaced (4.579%); 20 MiB/s, 314 objects/s recovering 2024-03-11T21:42:06.865 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:42:06 smithi092 ceph-mon[37563]: pgmap v775: 158 pgs: 149 active+clean, 4 active+recovery_wait+undersized+degraded+remapped, 5 active+recovering+undersized+remapped; 5.7 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 852 B/s rd, 45 MiB/s wr, 719 op/s; 17932/281919 objects degraded (6.361%); 12908/281919 objects misplaced (4.579%); 20 MiB/s, 314 objects/s recovering 2024-03-11T21:42:06.868 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:42:06.903 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:42:06.903 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:42:06.904 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:42:07.004 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:42:07.039 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:42:07.039 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:42:07.039 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:42:07.140 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:42:07.171 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:42:07.172 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:42:07.172 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:42:07.273 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:42:07.307 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:42:07.308 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:42:07.308 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:42:07.409 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:42:07.439 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:42:07.439 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:42:07.440 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:42:07.541 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:42:07.568 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:42:07.568 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:42:07.569 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:42:07.670 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:42:07.708 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:42:07.709 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:42:07.709 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:42:07.810 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:42:07.834 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:42:07.835 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:42:07.835 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:42:07.840 INFO:tasks.radosbench.radosbench.0.smithi092.stdout: 75 16 98496 98480 82.0559 45.4375 0.00321943 0.0121543 2024-03-11T21:42:07.936 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:42:07.964 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:42:07.965 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:42:07.966 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:42:08.066 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:42:08.100 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:42:08.101 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:42:08.101 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:42:08.202 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:42:08.228 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:42:08.229 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:42:08.230 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:42:08.331 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:42:08.356 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:42:08.356 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:42:08.356 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:42:08.365 INFO:journalctl@ceph.mgr.y.smithi092.stdout:Mar 11 21:42:07 smithi092 ceph-00db0016-dfed-11ee-95c8-87774f69a715-mgr-y[29801]: ::ffff:172.21.15.97 - - [11/Mar/2024:21:42:07] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-11T21:42:08.457 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:42:08.495 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:42:08.496 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:42:08.496 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:42:08.597 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:42:08.642 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:42:08.643 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:42:08.643 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:42:08.744 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:42:08.768 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:42:08.769 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:42:08.769 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:42:08.831 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:42:08 smithi097 ceph-mon[33317]: pgmap v776: 158 pgs: 149 active+clean, 4 active+recovery_wait+undersized+degraded+remapped, 5 active+recovering+undersized+remapped; 5.8 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 721 B/s rd, 43 MiB/s wr, 691 op/s; 17932/284421 objects degraded (6.305%); 12908/284421 objects misplaced (4.538%); 12 MiB/s, 193 objects/s recovering 2024-03-11T21:42:08.831 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:42:08 smithi097 ceph-mon[33317]: osdmap e388: 8 total, 8 up, 8 in 2024-03-11T21:42:08.831 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:42:08 smithi097 ceph-mon[33317]: from='mgr.14607 v1:172.21.15.92:0/1702717597' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T21:42:08.831 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:42:08 smithi097 ceph-mon[33317]: Health check update: Degraded data redundancy: 17932/284421 objects degraded (6.305%), 4 pgs degraded (PG_DEGRADED) 2024-03-11T21:42:08.831 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:42:08 smithi097 ceph-mon[33317]: osdmap e389: 8 total, 8 up, 8 in 2024-03-11T21:42:08.840 INFO:tasks.radosbench.radosbench.0.smithi092.stdout: 76 16 98981 98965 81.375 30.3125 0.0145431 0.0122676 2024-03-11T21:42:08.865 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:42:08 smithi092 ceph-mon[29572]: pgmap v776: 158 pgs: 149 active+clean, 4 active+recovery_wait+undersized+degraded+remapped, 5 active+recovering+undersized+remapped; 5.8 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 721 B/s rd, 43 MiB/s wr, 691 op/s; 17932/284421 objects degraded (6.305%); 12908/284421 objects misplaced (4.538%); 12 MiB/s, 193 objects/s recovering 2024-03-11T21:42:08.865 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:42:08 smithi092 ceph-mon[29572]: osdmap e388: 8 total, 8 up, 8 in 2024-03-11T21:42:08.865 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:42:08 smithi092 ceph-mon[29572]: from='mgr.14607 v1:172.21.15.92:0/1702717597' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T21:42:08.865 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:42:08 smithi092 ceph-mon[29572]: Health check update: Degraded data redundancy: 17932/284421 objects degraded (6.305%), 4 pgs degraded (PG_DEGRADED) 2024-03-11T21:42:08.865 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:42:08 smithi092 ceph-mon[29572]: osdmap e389: 8 total, 8 up, 8 in 2024-03-11T21:42:08.866 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:42:08 smithi092 ceph-mon[37563]: pgmap v776: 158 pgs: 149 active+clean, 4 active+recovery_wait+undersized+degraded+remapped, 5 active+recovering+undersized+remapped; 5.8 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 721 B/s rd, 43 MiB/s wr, 691 op/s; 17932/284421 objects degraded (6.305%); 12908/284421 objects misplaced (4.538%); 12 MiB/s, 193 objects/s recovering 2024-03-11T21:42:08.866 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:42:08 smithi092 ceph-mon[37563]: osdmap e388: 8 total, 8 up, 8 in 2024-03-11T21:42:08.866 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:42:08 smithi092 ceph-mon[37563]: from='mgr.14607 v1:172.21.15.92:0/1702717597' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T21:42:08.866 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:42:08 smithi092 ceph-mon[37563]: Health check update: Degraded data redundancy: 17932/284421 objects degraded (6.305%), 4 pgs degraded (PG_DEGRADED) 2024-03-11T21:42:08.866 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:42:08 smithi092 ceph-mon[37563]: osdmap e389: 8 total, 8 up, 8 in 2024-03-11T21:42:08.870 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:42:08.898 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:42:08.899 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:42:08.900 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:42:09.000 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:42:09.028 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:42:09.028 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:42:09.029 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:42:09.130 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:42:09.154 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:42:09.155 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:42:09.156 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:42:09.256 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:42:09.281 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:42:09.282 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:42:09.282 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:42:09.383 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:42:09.408 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:42:09.408 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:42:09.409 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:42:09.510 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:42:09.556 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:42:09.556 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:42:09.557 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:42:09.657 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:42:09.688 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:42:09.689 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:42:09.689 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:42:09.790 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:42:09.832 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:42:09.833 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:42:09.833 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:42:09.840 INFO:tasks.radosbench.radosbench.0.smithi092.stdout: 77 16 99502 99486 80.7411 32.5625 0.012006 0.0123804 2024-03-11T21:42:09.934 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:42:09.959 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:42:09.960 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:42:09.960 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:42:10.061 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:42:10.102 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:42:10.103 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:42:10.103 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:42:10.204 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:42:10.238 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:42:10.238 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:42:10.238 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:42:10.339 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:42:10.368 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:42:10.368 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:42:10.369 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:42:10.470 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:42:10.519 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:42:10.520 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:42:10.520 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:42:10.621 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:42:10.654 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:42:10.655 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:42:10.655 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:42:10.756 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:42:10.787 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:42:10.788 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:42:10.788 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:42:10.831 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:42:10 smithi097 ceph-mon[33317]: pgmap v779: 158 pgs: 149 active+clean, 3 active+recovery_wait+undersized+degraded+remapped, 6 active+recovering+undersized+remapped; 6.0 GiB data, 23 GiB used, 692 GiB / 715 GiB avail; 862 B/s rd, 54 MiB/s wr, 866 op/s; 13318/294129 objects degraded (4.528%); 16521/294129 objects misplaced (5.617%); 20 MiB/s, 313 objects/s recovering 2024-03-11T21:42:10.831 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:42:10 smithi097 ceph-mon[33317]: osdmap e390: 8 total, 8 up, 8 in 2024-03-11T21:42:10.840 INFO:tasks.radosbench.radosbench.0.smithi092.stdout: 78 16 100344 100328 80.3806 52.625 0.00566686 0.0124359 2024-03-11T21:42:10.865 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:42:10 smithi092 ceph-mon[29572]: pgmap v779: 158 pgs: 149 active+clean, 3 active+recovery_wait+undersized+degraded+remapped, 6 active+recovering+undersized+remapped; 6.0 GiB data, 23 GiB used, 692 GiB / 715 GiB avail; 862 B/s rd, 54 MiB/s wr, 866 op/s; 13318/294129 objects degraded (4.528%); 16521/294129 objects misplaced (5.617%); 20 MiB/s, 313 objects/s recovering 2024-03-11T21:42:10.865 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:42:10 smithi092 ceph-mon[29572]: osdmap e390: 8 total, 8 up, 8 in 2024-03-11T21:42:10.866 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:42:10 smithi092 ceph-mon[37563]: pgmap v779: 158 pgs: 149 active+clean, 3 active+recovery_wait+undersized+degraded+remapped, 6 active+recovering+undersized+remapped; 6.0 GiB data, 23 GiB used, 692 GiB / 715 GiB avail; 862 B/s rd, 54 MiB/s wr, 866 op/s; 13318/294129 objects degraded (4.528%); 16521/294129 objects misplaced (5.617%); 20 MiB/s, 313 objects/s recovering 2024-03-11T21:42:10.866 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:42:10 smithi092 ceph-mon[37563]: osdmap e390: 8 total, 8 up, 8 in 2024-03-11T21:42:10.889 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:42:10.921 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:42:10.921 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:42:10.922 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:42:11.023 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:42:11.052 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:42:11.053 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:42:11.053 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:42:11.154 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:42:11.187 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:42:11.187 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:42:11.187 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:42:11.288 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:42:11.317 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:42:11.317 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:42:11.318 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:42:11.419 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:42:11.450 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:42:11.450 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:42:11.450 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:42:11.551 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:42:11.593 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:42:11.593 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:42:11.594 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:42:11.695 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:42:11.722 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:42:11.723 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:42:11.723 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:42:11.824 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:42:11.840 INFO:tasks.radosbench.radosbench.0.smithi092.stdout: 79 16 101199 101183 80.0395 53.4375 0.00908276 0.0124874 2024-03-11T21:42:11.859 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:42:11.859 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:42:11.860 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:42:11.961 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:42:12.005 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:42:12.006 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:42:12.006 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:42:12.107 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:42:12.146 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:42:12.147 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:42:12.147 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:42:12.248 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:42:12.277 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:42:12.278 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:42:12.279 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:42:12.379 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:42:12.431 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:42:12.431 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:42:12.431 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:42:12.532 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:42:12.556 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:42:12.556 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:42:12.557 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:42:12.658 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:42:12.694 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:42:12.695 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:42:12.695 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:42:12.796 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:42:12.830 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:42:12.830 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:42:12.831 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:42:12.831 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:42:12 smithi097 ceph-mon[33317]: pgmap v781: 158 pgs: 149 active+clean, 3 active+recovery_wait+undersized+degraded+remapped, 6 active+recovering+undersized+remapped; 6.0 GiB data, 23 GiB used, 692 GiB / 715 GiB avail; 853 B/s rd, 44 MiB/s wr, 701 op/s; 13318/294534 objects degraded (4.522%); 15835/294534 objects misplaced (5.376%); 8.6 MiB/s, 138 objects/s recovering 2024-03-11T21:42:12.840 INFO:tasks.radosbench.radosbench.0.smithi092.stdout:2024-03-11T21:42:12.854113+0000 min lat: 0.00121416 max lat: 2.54835 avg lat: 0.0125501 2024-03-11T21:42:12.840 INFO:tasks.radosbench.radosbench.0.smithi092.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-03-11T21:42:12.840 INFO:tasks.radosbench.radosbench.0.smithi092.stdout: 80 16 101950 101934 79.6257 46.9375 0.00575903 0.0125501 2024-03-11T21:42:12.865 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:42:12 smithi092 ceph-mon[29572]: pgmap v781: 158 pgs: 149 active+clean, 3 active+recovery_wait+undersized+degraded+remapped, 6 active+recovering+undersized+remapped; 6.0 GiB data, 23 GiB used, 692 GiB / 715 GiB avail; 853 B/s rd, 44 MiB/s wr, 701 op/s; 13318/294534 objects degraded (4.522%); 15835/294534 objects misplaced (5.376%); 8.6 MiB/s, 138 objects/s recovering 2024-03-11T21:42:12.865 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:42:12 smithi092 ceph-mon[37563]: pgmap v781: 158 pgs: 149 active+clean, 3 active+recovery_wait+undersized+degraded+remapped, 6 active+recovering+undersized+remapped; 6.0 GiB data, 23 GiB used, 692 GiB / 715 GiB avail; 853 B/s rd, 44 MiB/s wr, 701 op/s; 13318/294534 objects degraded (4.522%); 15835/294534 objects misplaced (5.376%); 8.6 MiB/s, 138 objects/s recovering 2024-03-11T21:42:12.932 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:42:12.979 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:42:12.979 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:42:12.980 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:42:13.081 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:42:13.122 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:42:13.123 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:42:13.123 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:42:13.224 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:42:13.253 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:42:13.253 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:42:13.254 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:42:13.331 INFO:journalctl@ceph.iscsi.iscsi.a.smithi097.stdout:Mar 11 21:42:12 smithi097 ceph-00db0016-dfed-11ee-95c8-87774f69a715-iscsi-iscsi-a[60943]: debug there is no tcmu-runner data available 2024-03-11T21:42:13.354 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:42:13.385 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:42:13.385 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:42:13.386 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:42:13.487 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:42:13.525 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:42:13.526 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:42:13.526 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:42:13.627 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:42:13.672 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:42:13.673 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:42:13.673 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:42:13.774 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:42:13.806 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:42:13.810 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:42:13.811 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:42:13.831 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:42:13 smithi097 ceph-mon[33317]: from='client.24469 v1:172.21.15.97:0/404208310' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:42:13.831 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:42:13 smithi097 ceph-mon[33317]: Health check update: Degraded data redundancy: 13318/294534 objects degraded (4.522%), 3 pgs degraded (PG_DEGRADED) 2024-03-11T21:42:13.840 INFO:tasks.radosbench.radosbench.0.smithi092.stdout: 81 16 102788 102772 79.2891 52.375 0.00260441 0.0125985 2024-03-11T21:42:13.865 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:42:13 smithi092 ceph-mon[29572]: from='client.24469 v1:172.21.15.97:0/404208310' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:42:13.865 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:42:13 smithi092 ceph-mon[29572]: Health check update: Degraded data redundancy: 13318/294534 objects degraded (4.522%), 3 pgs degraded (PG_DEGRADED) 2024-03-11T21:42:13.865 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:42:13 smithi092 ceph-mon[37563]: from='client.24469 v1:172.21.15.97:0/404208310' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:42:13.865 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:42:13 smithi092 ceph-mon[37563]: Health check update: Degraded data redundancy: 13318/294534 objects degraded (4.522%), 3 pgs degraded (PG_DEGRADED) 2024-03-11T21:42:13.912 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:42:13.949 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:42:13.950 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:42:13.950 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:42:14.051 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:42:14.098 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:42:14.098 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:42:14.099 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:42:14.199 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:42:14.241 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:42:14.241 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:42:14.242 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:42:14.343 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:42:14.381 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:42:14.382 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:42:14.382 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:42:14.482 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:42:14.518 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:42:14.519 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:42:14.519 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:42:14.620 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:42:14.668 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:42:14.668 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:42:14.669 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:42:14.770 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:42:14.806 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:42:14.806 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:42:14.807 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:42:14.833 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:42:14 smithi097 ceph-mon[33317]: pgmap v782: 158 pgs: 151 active+clean, 3 active+recovery_wait+undersized+degraded+remapped, 4 active+recovering+undersized+remapped; 6.2 GiB data, 23 GiB used, 692 GiB / 715 GiB avail; 853 B/s rd, 64 MiB/s wr, 1.03k op/s; 13318/302994 objects degraded (4.395%); 15072/302994 objects misplaced (4.974%); 17 MiB/s, 265 objects/s recovering 2024-03-11T21:42:14.840 INFO:tasks.radosbench.radosbench.0.smithi092.stdout: 82 15 103715 103700 79.0295 58 0.0133494 0.0126484 2024-03-11T21:42:14.865 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:42:14 smithi092 ceph-mon[29572]: pgmap v782: 158 pgs: 151 active+clean, 3 active+recovery_wait+undersized+degraded+remapped, 4 active+recovering+undersized+remapped; 6.2 GiB data, 23 GiB used, 692 GiB / 715 GiB avail; 853 B/s rd, 64 MiB/s wr, 1.03k op/s; 13318/302994 objects degraded (4.395%); 15072/302994 objects misplaced (4.974%); 17 MiB/s, 265 objects/s recovering 2024-03-11T21:42:14.865 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:42:14 smithi092 ceph-mon[37563]: pgmap v782: 158 pgs: 151 active+clean, 3 active+recovery_wait+undersized+degraded+remapped, 4 active+recovering+undersized+remapped; 6.2 GiB data, 23 GiB used, 692 GiB / 715 GiB avail; 853 B/s rd, 64 MiB/s wr, 1.03k op/s; 13318/302994 objects degraded (4.395%); 15072/302994 objects misplaced (4.974%); 17 MiB/s, 265 objects/s recovering 2024-03-11T21:42:14.908 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:42:14.947 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:42:14.947 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:42:14.947 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:42:15.048 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:42:15.082 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:42:15.083 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:42:15.083 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:42:15.184 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:42:15.208 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:42:15.209 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:42:15.209 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:42:15.310 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:42:15.355 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:42:15.356 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:42:15.356 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:42:15.457 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:42:15.501 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:42:15.502 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:42:15.502 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:42:15.603 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:42:15.638 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:42:15.639 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:42:15.639 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:42:15.740 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:42:15.772 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:42:15.773 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:42:15.773 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:42:15.841 INFO:tasks.radosbench.radosbench.0.smithi092.stdout: 83 16 104586 104570 78.7324 54.375 0.00378049 0.0126904 2024-03-11T21:42:15.874 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:42:15.912 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:42:15.912 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:42:15.913 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:42:16.013 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:42:16.046 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:42:16.047 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:42:16.047 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:42:16.148 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:42:16.172 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:42:16.173 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:42:16.173 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:42:16.274 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:42:16.324 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:42:16.325 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:42:16.326 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:42:16.427 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:42:16.459 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:42:16.459 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:42:16.460 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:42:16.560 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:42:16.600 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:42:16.601 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:42:16.601 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:42:16.702 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:42:16.751 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:42:16.752 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:42:16.752 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:42:16.831 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:42:16 smithi097 ceph-mon[33317]: pgmap v783: 158 pgs: 151 active+clean, 3 active+recovery_wait+undersized+degraded+remapped, 4 active+recovering+undersized+remapped; 6.2 GiB data, 24 GiB used, 691 GiB / 715 GiB avail; 1.3 KiB/s rd, 41 MiB/s wr, 663 op/s; 13318/305523 objects degraded (4.359%); 13400/305523 objects misplaced (4.386%); 19 MiB/s, 300 objects/s recovering 2024-03-11T21:42:16.840 INFO:tasks.radosbench.radosbench.0.smithi092.stdout: 84 16 105489 105473 78.4669 56.4375 0.00785371 0.0127377 2024-03-11T21:42:16.853 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:42:16.865 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:42:16 smithi092 ceph-mon[29572]: pgmap v783: 158 pgs: 151 active+clean, 3 active+recovery_wait+undersized+degraded+remapped, 4 active+recovering+undersized+remapped; 6.2 GiB data, 24 GiB used, 691 GiB / 715 GiB avail; 1.3 KiB/s rd, 41 MiB/s wr, 663 op/s; 13318/305523 objects degraded (4.359%); 13400/305523 objects misplaced (4.386%); 19 MiB/s, 300 objects/s recovering 2024-03-11T21:42:16.865 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:42:16 smithi092 ceph-mon[37563]: pgmap v783: 158 pgs: 151 active+clean, 3 active+recovery_wait+undersized+degraded+remapped, 4 active+recovering+undersized+remapped; 6.2 GiB data, 24 GiB used, 691 GiB / 715 GiB avail; 1.3 KiB/s rd, 41 MiB/s wr, 663 op/s; 13318/305523 objects degraded (4.359%); 13400/305523 objects misplaced (4.386%); 19 MiB/s, 300 objects/s recovering 2024-03-11T21:42:16.884 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:42:16.885 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:42:16.885 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:42:16.986 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:42:17.013 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:42:17.013 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:42:17.014 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:42:17.114 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:42:17.148 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:42:17.148 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:42:17.149 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:42:17.249 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:42:17.273 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:42:17.274 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:42:17.274 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:42:17.375 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:42:17.401 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:42:17.401 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:42:17.402 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:42:17.503 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:42:17.540 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:42:17.541 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:42:17.542 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:42:17.642 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:42:17.683 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:42:17.683 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:42:17.684 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:42:17.785 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:42:17.815 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:42:17.816 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:42:17.816 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:42:17.841 INFO:tasks.radosbench.radosbench.0.smithi092.stdout: 85 16 106185 106169 78.0555 43.5 0.0128008 0.0128063 2024-03-11T21:42:17.917 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:42:17.969 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:42:17.970 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:42:17.970 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:42:18.071 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:42:18.103 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:42:18.104 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:42:18.104 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:42:18.205 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:42:18.241 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:42:18.242 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:42:18.242 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:42:18.343 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:42:18.365 INFO:journalctl@ceph.mgr.y.smithi092.stdout:Mar 11 21:42:17 smithi092 ceph-00db0016-dfed-11ee-95c8-87774f69a715-mgr-y[29801]: ::ffff:172.21.15.97 - - [11/Mar/2024:21:42:17] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-11T21:42:18.376 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:42:18.377 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:42:18.377 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:42:18.478 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:42:18.511 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:42:18.511 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:42:18.512 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:42:18.612 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:42:18.656 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:42:18.657 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:42:18.658 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:42:18.758 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:42:18.797 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:42:18.797 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:42:18.798 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:42:18.831 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:42:18 smithi097 ceph-mon[33317]: pgmap v784: 158 pgs: 151 active+clean, 3 active+recovery_wait+undersized+degraded+remapped, 4 active+recovering+undersized+remapped; 6.3 GiB data, 24 GiB used, 691 GiB / 715 GiB avail; 572 B/s rd, 39 MiB/s wr, 616 op/s; 13318/308868 objects degraded (4.312%); 13400/308868 objects misplaced (4.338%); 17 MiB/s, 265 objects/s recovering 2024-03-11T21:42:18.831 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:42:18 smithi097 ceph-mon[33317]: Health check update: Degraded data redundancy: 13318/308868 objects degraded (4.312%), 3 pgs degraded (PG_DEGRADED) 2024-03-11T21:42:18.841 INFO:tasks.radosbench.radosbench.0.smithi092.stdout: 86 16 107134 107118 77.8375 59.3125 0.0277945 0.0128424 2024-03-11T21:42:18.866 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:42:18 smithi092 ceph-mon[29572]: pgmap v784: 158 pgs: 151 active+clean, 3 active+recovery_wait+undersized+degraded+remapped, 4 active+recovering+undersized+remapped; 6.3 GiB data, 24 GiB used, 691 GiB / 715 GiB avail; 572 B/s rd, 39 MiB/s wr, 616 op/s; 13318/308868 objects degraded (4.312%); 13400/308868 objects misplaced (4.338%); 17 MiB/s, 265 objects/s recovering 2024-03-11T21:42:18.866 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:42:18 smithi092 ceph-mon[29572]: Health check update: Degraded data redundancy: 13318/308868 objects degraded (4.312%), 3 pgs degraded (PG_DEGRADED) 2024-03-11T21:42:18.866 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:42:18 smithi092 ceph-mon[37563]: pgmap v784: 158 pgs: 151 active+clean, 3 active+recovery_wait+undersized+degraded+remapped, 4 active+recovering+undersized+remapped; 6.3 GiB data, 24 GiB used, 691 GiB / 715 GiB avail; 572 B/s rd, 39 MiB/s wr, 616 op/s; 13318/308868 objects degraded (4.312%); 13400/308868 objects misplaced (4.338%); 17 MiB/s, 265 objects/s recovering 2024-03-11T21:42:18.866 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:42:18 smithi092 ceph-mon[37563]: Health check update: Degraded data redundancy: 13318/308868 objects degraded (4.312%), 3 pgs degraded (PG_DEGRADED) 2024-03-11T21:42:18.899 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:42:18.943 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:42:18.944 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:42:18.944 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:42:19.045 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:42:19.070 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:42:19.070 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:42:19.071 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:42:19.171 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:42:19.215 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:42:19.215 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:42:19.216 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:42:19.316 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:42:19.350 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:42:19.350 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:42:19.350 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:42:19.451 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:42:19.489 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:42:19.490 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:42:19.490 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:42:19.591 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:42:19.625 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:42:19.625 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:42:19.626 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:42:19.727 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:42:19.751 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:42:19.752 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:42:19.752 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:42:19.841 INFO:tasks.radosbench.radosbench.0.smithi092.stdout: 87 16 107923 107907 77.5096 49.3125 0.110332 0.0128953 2024-03-11T21:42:19.853 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:42:19.886 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:42:19.886 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:42:19.887 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:42:19.987 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:42:20.028 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:42:20.029 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:42:20.030 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:42:20.130 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:42:20.156 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:42:20.156 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:42:20.157 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:42:20.258 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:42:20.289 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:42:20.290 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:42:20.290 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:42:20.391 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:42:20.435 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:42:20 smithi092 ceph-mon[29572]: pgmap v785: 158 pgs: 151 active+clean, 3 active+recovery_wait+undersized+degraded+remapped, 4 active+recovering+undersized+remapped; 6.5 GiB data, 25 GiB used, 691 GiB / 715 GiB avail; 1023 B/s rd, 50 MiB/s wr, 796 op/s; 13318/318012 objects degraded (4.188%); 12288/318012 objects misplaced (3.864%); 15 MiB/s, 240 objects/s recovering 2024-03-11T21:42:20.435 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:42:20 smithi092 ceph-mon[29572]: from='client.? v1:172.21.15.92:0/2836237835' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-11T21:42:20.438 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:42:20.438 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:42:20.439 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:42:20.540 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:42:20.579 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:42:20.580 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:42:20.580 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:42:20.681 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:42:20.709 INFO:teuthology.orchestra.run.smithi092.stdout:osd.0: {} 2024-03-11T21:42:20.709 INFO:teuthology.orchestra.run.smithi092.stdout:osd.1: {} 2024-03-11T21:42:20.709 INFO:teuthology.orchestra.run.smithi092.stdout:osd.2: {} 2024-03-11T21:42:20.709 INFO:teuthology.orchestra.run.smithi092.stdout:osd.3: {} 2024-03-11T21:42:20.709 INFO:teuthology.orchestra.run.smithi092.stdout:osd.4: {} 2024-03-11T21:42:20.709 INFO:teuthology.orchestra.run.smithi092.stdout:osd.5: {} 2024-03-11T21:42:20.709 INFO:teuthology.orchestra.run.smithi092.stdout:osd.6: {} 2024-03-11T21:42:20.709 INFO:teuthology.orchestra.run.smithi092.stdout:osd.7: {} 2024-03-11T21:42:20.710 INFO:teuthology.orchestra.run.smithi092.stderr:osd.0: osd_enable_op_tracker = '' 2024-03-11T21:42:20.710 INFO:teuthology.orchestra.run.smithi092.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-11T21:42:20.710 INFO:teuthology.orchestra.run.smithi092.stderr:osd.2: osd_enable_op_tracker = '' 2024-03-11T21:42:20.710 INFO:teuthology.orchestra.run.smithi092.stderr:osd.3: osd_enable_op_tracker = '' 2024-03-11T21:42:20.710 INFO:teuthology.orchestra.run.smithi092.stderr:osd.4: osd_enable_op_tracker = '' 2024-03-11T21:42:20.710 INFO:teuthology.orchestra.run.smithi092.stderr:osd.5: osd_enable_op_tracker = '' 2024-03-11T21:42:20.710 INFO:teuthology.orchestra.run.smithi092.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-11T21:42:20.710 INFO:teuthology.orchestra.run.smithi092.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-11T21:42:20.710 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:42:20 smithi092 ceph-mon[37563]: pgmap v785: 158 pgs: 151 active+clean, 3 active+recovery_wait+undersized+degraded+remapped, 4 active+recovering+undersized+remapped; 6.5 GiB data, 25 GiB used, 691 GiB / 715 GiB avail; 1023 B/s rd, 50 MiB/s wr, 796 op/s; 13318/318012 objects degraded (4.188%); 12288/318012 objects misplaced (3.864%); 15 MiB/s, 240 objects/s recovering 2024-03-11T21:42:20.710 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:42:20 smithi092 ceph-mon[37563]: from='client.? v1:172.21.15.92:0/2836237835' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-11T21:42:20.711 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:42:20.711 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:42:20.712 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:42:20.812 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:42:20.831 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:42:20 smithi097 ceph-mon[33317]: pgmap v785: 158 pgs: 151 active+clean, 3 active+recovery_wait+undersized+degraded+remapped, 4 active+recovering+undersized+remapped; 6.5 GiB data, 25 GiB used, 691 GiB / 715 GiB avail; 1023 B/s rd, 50 MiB/s wr, 796 op/s; 13318/318012 objects degraded (4.188%); 12288/318012 objects misplaced (3.864%); 15 MiB/s, 240 objects/s recovering 2024-03-11T21:42:20.831 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:42:20 smithi097 ceph-mon[33317]: from='client.? v1:172.21.15.92:0/2836237835' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-11T21:42:20.838 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:42:20.838 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:42:20.839 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:42:20.841 INFO:tasks.radosbench.radosbench.0.smithi092.stdout: 88 15 108732 108717 77.2041 50.625 0.00769687 0.012927 2024-03-11T21:42:20.939 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:42:20.967 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:42:20.967 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:42:20.968 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:42:21.068 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:42:21.094 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:42:21.095 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:42:21.095 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:42:21.196 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:42:21.229 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:42:21.230 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:42:21.230 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:42:21.331 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:42:21.355 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:42:21.355 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:42:21.356 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:42:21.457 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:42:21.480 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:42:21.481 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:42:21.481 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:42:21.582 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:42:21.615 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:42:21.615 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:42:21.616 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:42:21.717 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:42:21.732 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:42:21 smithi097 ceph-mon[33317]: osdmap e391: 8 total, 8 up, 8 in 2024-03-11T21:42:21.747 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:42:21.747 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:42:21.747 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:42:21.841 INFO:tasks.radosbench.radosbench.0.smithi092.stdout: 89 16 109048 109032 76.5578 19.6875 0.137019 0.0130492 2024-03-11T21:42:21.848 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:42:21.865 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:42:21 smithi092 ceph-mon[29572]: osdmap e391: 8 total, 8 up, 8 in 2024-03-11T21:42:21.865 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:42:21 smithi092 ceph-mon[37563]: osdmap e391: 8 total, 8 up, 8 in 2024-03-11T21:42:21.887 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:42:21.888 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:42:21.888 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:42:21.989 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:42:22.026 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:42:22.026 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:42:22.027 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:42:22.128 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:42:22.155 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:42:22.156 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:42:22.156 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:42:22.257 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:42:22.289 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:42:22.290 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:42:22.290 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:42:22.391 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:42:22.421 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:42:22.422 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:42:22.422 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:42:22.523 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:42:22.553 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:42:22.553 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:42:22.554 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:42:22.654 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:42:22.684 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:42:22.684 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:42:22.685 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:42:22.785 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:42:22.817 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:42:22.818 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:42:22.818 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:42:22.831 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:42:22 smithi097 ceph-mon[33317]: pgmap v787: 158 pgs: 151 active+clean, 3 active+recovery_wait+undersized+degraded+remapped, 4 active+recovering+undersized+remapped; 6.5 GiB data, 25 GiB used, 690 GiB / 715 GiB avail; 1023 B/s rd, 50 MiB/s wr, 794 op/s; 13318/318360 objects degraded (4.183%); 11352/318360 objects misplaced (3.566%); 16 MiB/s, 252 objects/s recovering 2024-03-11T21:42:22.831 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:42:22 smithi097 ceph-mon[33317]: osdmap e392: 8 total, 8 up, 8 in 2024-03-11T21:42:22.841 INFO:tasks.radosbench.radosbench.0.smithi092.stdout: 90 1 109973 109972 76.3599 58.75 0.00910968 0.0130911 2024-03-11T21:42:22.865 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:42:22 smithi092 ceph-mon[29572]: pgmap v787: 158 pgs: 151 active+clean, 3 active+recovery_wait+undersized+degraded+remapped, 4 active+recovering+undersized+remapped; 6.5 GiB data, 25 GiB used, 690 GiB / 715 GiB avail; 1023 B/s rd, 50 MiB/s wr, 794 op/s; 13318/318360 objects degraded (4.183%); 11352/318360 objects misplaced (3.566%); 16 MiB/s, 252 objects/s recovering 2024-03-11T21:42:22.865 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:42:22 smithi092 ceph-mon[29572]: osdmap e392: 8 total, 8 up, 8 in 2024-03-11T21:42:22.866 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:42:22 smithi092 ceph-mon[37563]: pgmap v787: 158 pgs: 151 active+clean, 3 active+recovery_wait+undersized+degraded+remapped, 4 active+recovering+undersized+remapped; 6.5 GiB data, 25 GiB used, 690 GiB / 715 GiB avail; 1023 B/s rd, 50 MiB/s wr, 794 op/s; 13318/318360 objects degraded (4.183%); 11352/318360 objects misplaced (3.566%); 16 MiB/s, 252 objects/s recovering 2024-03-11T21:42:22.866 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:42:22 smithi092 ceph-mon[37563]: osdmap e392: 8 total, 8 up, 8 in 2024-03-11T21:42:22.919 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:42:22.944 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:42:22.944 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:42:22.945 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:42:23.045 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:42:23.077 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:42:23.078 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:42:23.078 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:42:23.179 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:42:23.203 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:42:23.204 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:42:23.204 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:42:23.305 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:42:23.329 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:42:23.330 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:42:23.330 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:42:23.331 INFO:journalctl@ceph.iscsi.iscsi.a.smithi097.stdout:Mar 11 21:42:22 smithi097 ceph-00db0016-dfed-11ee-95c8-87774f69a715-iscsi-iscsi-a[60943]: debug there is no tcmu-runner data available 2024-03-11T21:42:23.432 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:42:23.464 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:42:23.465 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:42:23.466 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:42:23.567 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:42:23.591 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:42:23.592 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:42:23.592 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:42:23.618 INFO:teuthology.orchestra.run.smithi092.stderr:Inferring config /var/lib/ceph/00db0016-dfed-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T21:42:23.693 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:42:23.722 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:42:23 smithi092 ceph-mon[37563]: from='mgr.14607 v1:172.21.15.92:0/1702717597' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T21:42:23.722 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:42:23 smithi092 ceph-mon[37563]: from='client.24469 v1:172.21.15.97:0/404208310' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:42:23.722 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:42:23 smithi092 ceph-mon[37563]: Health check update: Degraded data redundancy: 13318/318360 objects degraded (4.183%), 3 pgs degraded (PG_DEGRADED) 2024-03-11T21:42:23.723 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:42:23 smithi092 ceph-mon[29572]: from='mgr.14607 v1:172.21.15.92:0/1702717597' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T21:42:23.723 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:42:23 smithi092 ceph-mon[29572]: from='client.24469 v1:172.21.15.97:0/404208310' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:42:23.723 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:42:23 smithi092 ceph-mon[29572]: Health check update: Degraded data redundancy: 13318/318360 objects degraded (4.183%), 3 pgs degraded (PG_DEGRADED) 2024-03-11T21:42:23.730 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:42:23.731 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:42:23.732 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:42:23.831 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:42:23 smithi097 ceph-mon[33317]: from='mgr.14607 v1:172.21.15.92:0/1702717597' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T21:42:23.832 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:42:23 smithi097 ceph-mon[33317]: from='client.24469 v1:172.21.15.97:0/404208310' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:42:23.832 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:42:23 smithi097 ceph-mon[33317]: Health check update: Degraded data redundancy: 13318/318360 objects degraded (4.183%), 3 pgs degraded (PG_DEGRADED) 2024-03-11T21:42:23.832 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:42:23.841 INFO:tasks.radosbench.radosbench.0.smithi092.stdout:Total time run: 90.0113 2024-03-11T21:42:23.841 INFO:tasks.radosbench.radosbench.0.smithi092.stdout:Total writes made: 109973 2024-03-11T21:42:23.841 INFO:tasks.radosbench.radosbench.0.smithi092.stdout:Write size: 65536 2024-03-11T21:42:23.842 INFO:tasks.radosbench.radosbench.0.smithi092.stdout:Object size: 65536 2024-03-11T21:42:23.842 INFO:tasks.radosbench.radosbench.0.smithi092.stdout:Bandwidth (MB/sec): 76.3606 2024-03-11T21:42:23.842 INFO:tasks.radosbench.radosbench.0.smithi092.stdout:Stddev Bandwidth: 46.6203 2024-03-11T21:42:23.842 INFO:tasks.radosbench.radosbench.0.smithi092.stdout:Max bandwidth (MB/sec): 227.438 2024-03-11T21:42:23.842 INFO:tasks.radosbench.radosbench.0.smithi092.stdout:Min bandwidth (MB/sec): 0 2024-03-11T21:42:23.842 INFO:tasks.radosbench.radosbench.0.smithi092.stdout:Average IOPS: 1221 2024-03-11T21:42:23.842 INFO:tasks.radosbench.radosbench.0.smithi092.stdout:Stddev IOPS: 745.924 2024-03-11T21:42:23.842 INFO:tasks.radosbench.radosbench.0.smithi092.stdout:Max IOPS: 3639 2024-03-11T21:42:23.842 INFO:tasks.radosbench.radosbench.0.smithi092.stdout:Min IOPS: 0 2024-03-11T21:42:23.842 INFO:tasks.radosbench.radosbench.0.smithi092.stdout:Average Latency(s): 0.0130911 2024-03-11T21:42:23.842 INFO:tasks.radosbench.radosbench.0.smithi092.stdout:Stddev Latency(s): 0.0374155 2024-03-11T21:42:23.842 INFO:tasks.radosbench.radosbench.0.smithi092.stdout:Max latency(s): 2.54835 2024-03-11T21:42:23.842 INFO:tasks.radosbench.radosbench.0.smithi092.stdout:Min latency(s): 0.00121416 2024-03-11T21:42:23.846 INFO:tasks.radosbench.radosbench.0.smithi092.stdout:Cleaning up (deleting benchmark objects) 2024-03-11T21:42:23.860 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:42:23.861 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:42:23.861 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:42:23.962 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:42:23.990 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:42:23.991 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:42:23.991 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:42:24.092 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:42:24.143 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:42:24.144 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:42:24.144 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:42:24.163 DEBUG:teuthology.orchestra.run.smithi097:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid 00db0016-dfed-11ee-95c8-87774f69a715 -- ceph daemon osd.4 dump_ops_in_flight 2024-03-11T21:42:24.245 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:42:24.283 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:42:24.283 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:42:24.284 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:42:24.385 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:42:24.432 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:42:24.433 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:42:24.434 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:42:24.534 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:42:24.565 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:42:24.565 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:42:24.566 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:42:24.579 INFO:teuthology.orchestra.run.smithi097.stderr:Inferring config /var/lib/ceph/00db0016-dfed-11ee-95c8-87774f69a715/mon.b/config 2024-03-11T21:42:24.667 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:42:24.706 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:42:24.707 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:42:24.707 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:42:24.721 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:42:24 smithi097 ceph-mon[33317]: pgmap v789: 158 pgs: 151 active+clean, 2 active+recovery_wait+undersized+degraded+remapped, 5 active+recovering+undersized+remapped; 6.6 GiB data, 25 GiB used, 690 GiB / 715 GiB avail; 639 B/s rd, 56 MiB/s wr, 895 op/s; 8894/327015 objects degraded (2.720%); 15558/327015 objects misplaced (4.758%); 8.5 MiB/s, 135 objects/s recovering 2024-03-11T21:42:24.785 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:42:24 smithi092 ceph-mon[29572]: pgmap v789: 158 pgs: 151 active+clean, 2 active+recovery_wait+undersized+degraded+remapped, 5 active+recovering+undersized+remapped; 6.6 GiB data, 25 GiB used, 690 GiB / 715 GiB avail; 639 B/s rd, 56 MiB/s wr, 895 op/s; 8894/327015 objects degraded (2.720%); 15558/327015 objects misplaced (4.758%); 8.5 MiB/s, 135 objects/s recovering 2024-03-11T21:42:24.785 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:42:24 smithi092 ceph-mon[37563]: pgmap v789: 158 pgs: 151 active+clean, 2 active+recovery_wait+undersized+degraded+remapped, 5 active+recovering+undersized+remapped; 6.6 GiB data, 25 GiB used, 690 GiB / 715 GiB avail; 639 B/s rd, 56 MiB/s wr, 895 op/s; 8894/327015 objects degraded (2.720%); 15558/327015 objects misplaced (4.758%); 8.5 MiB/s, 135 objects/s recovering 2024-03-11T21:42:24.809 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:42:24.847 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:42:24.848 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:42:24.848 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:42:24.949 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:42:24.986 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:42:24.986 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:42:24.987 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:42:25.088 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:42:25.112 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:42:25.113 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:42:25.114 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:42:25.215 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:42:25.239 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:42:25.240 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:42:25.240 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:42:25.341 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:42:25.373 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:42:25.374 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:42:25.374 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:42:25.452 DEBUG:teuthology.orchestra.run.smithi092:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid 00db0016-dfed-11ee-95c8-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-03-11T21:42:25.477 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:42:25.498 INFO:teuthology.orchestra.run.smithi092.stderr:nodeep-scrub is set 2024-03-11T21:42:25.504 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:42:25.504 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:42:25.505 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:42:25.606 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:42:25.615 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:42:25 smithi092 ceph-mon[29572]: osdmap e393: 8 total, 8 up, 8 in 2024-03-11T21:42:25.615 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:42:25 smithi092 ceph-mon[29572]: from='client.? v1:172.21.15.92:0/1679702323' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-11T21:42:25.615 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:42:25 smithi092 ceph-mon[37563]: osdmap e393: 8 total, 8 up, 8 in 2024-03-11T21:42:25.615 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:42:25 smithi092 ceph-mon[37563]: from='client.? v1:172.21.15.92:0/1679702323' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-11T21:42:25.661 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:42:25.662 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:42:25.662 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:42:25.763 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:42:25.807 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:42:25 smithi097 ceph-mon[33317]: osdmap e393: 8 total, 8 up, 8 in 2024-03-11T21:42:25.807 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:42:25 smithi097 ceph-mon[33317]: from='client.? v1:172.21.15.92:0/1679702323' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-11T21:42:25.824 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:42:25.825 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:42:25.825 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:42:25.926 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:42:25.984 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:42:25.984 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:42:25.984 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:42:26.085 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:42:26.126 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:42:26.127 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:42:26.127 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:42:26.228 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:42:26.256 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:42:26.257 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:42:26.257 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:42:26.358 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:42:26.392 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:42:26.392 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:42:26.393 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:42:26.493 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:42:26.548 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:42:26.548 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:42:26.548 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:42:26.649 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:42:26.683 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:42:26.684 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:42:26.684 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:42:26.718 INFO:teuthology.orchestra.run.smithi097.stdout:{ 2024-03-11T21:42:26.718 INFO:teuthology.orchestra.run.smithi097.stdout: "ops": [ 2024-03-11T21:42:26.718 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:42:26.719 INFO:teuthology.orchestra.run.smithi097.stdout: "description": "osd_op(client.15744.0:111961 10.e 10:7e0dd75b:::benchmark_data_smithi092_161462_object1985:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e394)", 2024-03-11T21:42:26.719 INFO:teuthology.orchestra.run.smithi097.stdout: "initiated_at": "2024-03-11T21:42:26.722947+0000", 2024-03-11T21:42:26.719 INFO:teuthology.orchestra.run.smithi097.stdout: "age": 0.0069718260000000004, 2024-03-11T21:42:26.719 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.0070033689999999997, 2024-03-11T21:42:26.719 INFO:teuthology.orchestra.run.smithi097.stdout: "type_data": { 2024-03-11T21:42:26.719 INFO:teuthology.orchestra.run.smithi097.stdout: "flag_point": "waiting for sub ops", 2024-03-11T21:42:26.719 INFO:teuthology.orchestra.run.smithi097.stdout: "client_info": { 2024-03-11T21:42:26.719 INFO:teuthology.orchestra.run.smithi097.stdout: "client": "client.15744", 2024-03-11T21:42:26.719 INFO:teuthology.orchestra.run.smithi097.stdout: "client_addr": "v1:172.21.15.92:0/3334000812", 2024-03-11T21:42:26.719 INFO:teuthology.orchestra.run.smithi097.stdout: "tid": 111961 2024-03-11T21:42:26.719 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:42:26.719 INFO:teuthology.orchestra.run.smithi097.stdout: "events": [ 2024-03-11T21:42:26.719 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:42:26.719 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "initiated", 2024-03-11T21:42:26.720 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:42:26.722947+0000", 2024-03-11T21:42:26.720 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-03-11T21:42:26.720 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:42:26.720 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:42:26.720 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "header_read", 2024-03-11T21:42:26.720 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:42:26.722947+0000", 2024-03-11T21:42:26.720 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-03-11T21:42:26.720 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:42:26.720 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:42:26.720 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "throttled", 2024-03-11T21:42:26.720 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:42:26.722949+0000", 2024-03-11T21:42:26.720 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.5889999999999999e-06 2024-03-11T21:42:26.720 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:42:26.720 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:42:26.721 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "all_read", 2024-03-11T21:42:26.721 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:42:26.722953+0000", 2024-03-11T21:42:26.721 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 4.8409999999999999e-06 2024-03-11T21:42:26.721 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:42:26.721 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:42:26.721 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "dispatched", 2024-03-11T21:42:26.721 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:42:26.722965+0000", 2024-03-11T21:42:26.721 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.2111e-05 2024-03-11T21:42:26.721 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:42:26.721 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:42:26.721 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "queued_for_pg", 2024-03-11T21:42:26.721 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:42:26.722976+0000", 2024-03-11T21:42:26.721 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.0892e-05 2024-03-11T21:42:26.721 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:42:26.722 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:42:26.722 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "reached_pg", 2024-03-11T21:42:26.722 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:42:26.723029+0000", 2024-03-11T21:42:26.722 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 5.2346000000000003e-05 2024-03-11T21:42:26.722 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:42:26.722 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:42:26.722 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "started", 2024-03-11T21:42:26.722 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:42:26.723163+0000", 2024-03-11T21:42:26.722 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.000134394 2024-03-11T21:42:26.722 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:42:26.722 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:42:26.722 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "waiting for subops from 2,6", 2024-03-11T21:42:26.722 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:42:26.723252+0000", 2024-03-11T21:42:26.722 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 8.9276000000000004e-05 2024-03-11T21:42:26.723 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:42:26.723 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:42:26.723 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "op_commit", 2024-03-11T21:42:26.723 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:42:26.725283+0000", 2024-03-11T21:42:26.723 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.0020301479999999998 2024-03-11T21:42:26.723 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:42:26.723 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:42:26.723 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "sub_op_commit_rec", 2024-03-11T21:42:26.723 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:42:26.725471+0000", 2024-03-11T21:42:26.723 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.000188011 2024-03-11T21:42:26.723 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-03-11T21:42:26.723 INFO:teuthology.orchestra.run.smithi097.stdout: ] 2024-03-11T21:42:26.723 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-03-11T21:42:26.724 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:42:26.724 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:42:26.724 INFO:teuthology.orchestra.run.smithi097.stdout: "description": "osd_repop(client.15744.0:111967 10.6 e394/372 10:64813341:::benchmark_data_smithi092_161462_object1991:head v 394'5685, mlcod=394'5683)", 2024-03-11T21:42:26.724 INFO:teuthology.orchestra.run.smithi097.stdout: "initiated_at": "2024-03-11T21:42:26.728876+0000", 2024-03-11T21:42:26.724 INFO:teuthology.orchestra.run.smithi097.stdout: "age": 0.0010430280000000001, 2024-03-11T21:42:26.724 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.00118593, 2024-03-11T21:42:26.724 INFO:teuthology.orchestra.run.smithi097.stdout: "type_data": { 2024-03-11T21:42:26.724 INFO:teuthology.orchestra.run.smithi097.stdout: "flag_point": "started", 2024-03-11T21:42:26.724 INFO:teuthology.orchestra.run.smithi097.stdout: "events": [ 2024-03-11T21:42:26.724 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:42:26.724 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "initiated", 2024-03-11T21:42:26.724 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:42:26.728876+0000", 2024-03-11T21:42:26.724 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-03-11T21:42:26.724 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:42:26.725 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:42:26.725 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "header_read", 2024-03-11T21:42:26.725 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:42:26.728876+0000", 2024-03-11T21:42:26.725 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-03-11T21:42:26.725 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:42:26.725 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:42:26.725 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "throttled", 2024-03-11T21:42:26.725 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:42:26.728877+0000", 2024-03-11T21:42:26.725 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.3039999999999999e-06 2024-03-11T21:42:26.725 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:42:26.725 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:42:26.725 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "all_read", 2024-03-11T21:42:26.725 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:42:26.728884+0000", 2024-03-11T21:42:26.725 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 7.4189999999999996e-06 2024-03-11T21:42:26.726 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:42:26.726 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:42:26.726 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "dispatched", 2024-03-11T21:42:26.726 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:42:26.728900+0000", 2024-03-11T21:42:26.726 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.5369000000000001e-05 2024-03-11T21:42:26.726 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:42:26.726 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:42:26.726 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "queued_for_pg", 2024-03-11T21:42:26.726 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:42:26.728907+0000", 2024-03-11T21:42:26.726 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 7.6170000000000001e-06 2024-03-11T21:42:26.726 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:42:26.726 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:42:26.726 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "reached_pg", 2024-03-11T21:42:26.727 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:42:26.728973+0000", 2024-03-11T21:42:26.727 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 6.5741000000000006e-05 2024-03-11T21:42:26.727 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:42:26.727 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:42:26.727 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "started", 2024-03-11T21:42:26.727 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:42:26.728992+0000", 2024-03-11T21:42:26.727 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.8955999999999999e-05 2024-03-11T21:42:26.727 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-03-11T21:42:26.727 INFO:teuthology.orchestra.run.smithi097.stdout: ] 2024-03-11T21:42:26.727 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-03-11T21:42:26.727 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:42:26.727 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:42:26.727 INFO:teuthology.orchestra.run.smithi097.stdout: "description": "osd_op(client.15744.0:111968 10.17 10:e917ec73:::benchmark_data_smithi092_161462_object1992:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e394)", 2024-03-11T21:42:26.727 INFO:teuthology.orchestra.run.smithi097.stdout: "initiated_at": "2024-03-11T21:42:26.729920+0000", 2024-03-11T21:42:26.727 INFO:teuthology.orchestra.run.smithi097.stdout: "age": 4294967295.999999, 2024-03-11T21:42:26.728 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.00021791700000000001, 2024-03-11T21:42:26.728 INFO:teuthology.orchestra.run.smithi097.stdout: "type_data": { 2024-03-11T21:42:26.728 INFO:teuthology.orchestra.run.smithi097.stdout: "flag_point": "started", 2024-03-11T21:42:26.728 INFO:teuthology.orchestra.run.smithi097.stdout: "client_info": { 2024-03-11T21:42:26.728 INFO:teuthology.orchestra.run.smithi097.stdout: "client": "client.15744", 2024-03-11T21:42:26.728 INFO:teuthology.orchestra.run.smithi097.stdout: "client_addr": "v1:172.21.15.92:0/3334000812", 2024-03-11T21:42:26.728 INFO:teuthology.orchestra.run.smithi097.stdout: "tid": 111968 2024-03-11T21:42:26.728 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:42:26.728 INFO:teuthology.orchestra.run.smithi097.stdout: "events": [ 2024-03-11T21:42:26.728 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:42:26.728 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "initiated", 2024-03-11T21:42:26.728 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:42:26.729920+0000", 2024-03-11T21:42:26.728 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-03-11T21:42:26.728 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:42:26.728 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:42:26.729 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "header_read", 2024-03-11T21:42:26.729 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:42:26.729920+0000", 2024-03-11T21:42:26.729 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-03-11T21:42:26.729 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:42:26.729 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:42:26.729 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "throttled", 2024-03-11T21:42:26.729 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:42:26.729922+0000", 2024-03-11T21:42:26.729 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 2.018e-06 2024-03-11T21:42:26.729 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:42:26.729 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:42:26.729 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "all_read", 2024-03-11T21:42:26.729 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:42:26.729928+0000", 2024-03-11T21:42:26.729 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 6.1800000000000001e-06 2024-03-11T21:42:26.729 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:42:26.730 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:42:26.730 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "dispatched", 2024-03-11T21:42:26.730 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:42:26.729943+0000", 2024-03-11T21:42:26.730 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.4853e-05 2024-03-11T21:42:26.730 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:42:26.730 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:42:26.730 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "queued_for_pg", 2024-03-11T21:42:26.730 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:42:26.729951+0000", 2024-03-11T21:42:26.730 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 7.9340000000000006e-06 2024-03-11T21:42:26.730 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:42:26.730 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:42:26.730 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "reached_pg", 2024-03-11T21:42:26.730 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:42:26.730003+0000", 2024-03-11T21:42:26.730 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 5.2062999999999999e-05 2024-03-11T21:42:26.730 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:42:26.731 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:42:26.731 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "started", 2024-03-11T21:42:26.731 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:42:26.730120+0000", 2024-03-11T21:42:26.731 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.000116636 2024-03-11T21:42:26.731 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-03-11T21:42:26.731 INFO:teuthology.orchestra.run.smithi097.stdout: ] 2024-03-11T21:42:26.731 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-03-11T21:42:26.731 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:42:26.731 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:42:26.731 INFO:teuthology.orchestra.run.smithi097.stdout: "description": "osd_op(client.15744.0:111969 10.e 10:7e294f5b:::benchmark_data_smithi092_161462_object1993:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e394)", 2024-03-11T21:42:26.731 INFO:teuthology.orchestra.run.smithi097.stdout: "initiated_at": "2024-03-11T21:42:26.729969+0000", 2024-03-11T21:42:26.731 INFO:teuthology.orchestra.run.smithi097.stdout: "age": 4294967295.9999495, 2024-03-11T21:42:26.731 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.00021330599999999999, 2024-03-11T21:42:26.731 INFO:teuthology.orchestra.run.smithi097.stdout: "type_data": { 2024-03-11T21:42:26.732 INFO:teuthology.orchestra.run.smithi097.stdout: "flag_point": "started", 2024-03-11T21:42:26.732 INFO:teuthology.orchestra.run.smithi097.stdout: "client_info": { 2024-03-11T21:42:26.732 INFO:teuthology.orchestra.run.smithi097.stdout: "client": "client.15744", 2024-03-11T21:42:26.732 INFO:teuthology.orchestra.run.smithi097.stdout: "client_addr": "v1:172.21.15.92:0/3334000812", 2024-03-11T21:42:26.732 INFO:teuthology.orchestra.run.smithi097.stdout: "tid": 111969 2024-03-11T21:42:26.732 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:42:26.732 INFO:teuthology.orchestra.run.smithi097.stdout: "events": [ 2024-03-11T21:42:26.732 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:42:26.732 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "initiated", 2024-03-11T21:42:26.732 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:42:26.729969+0000", 2024-03-11T21:42:26.733 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-03-11T21:42:26.733 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:42:26.733 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:42:26.733 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "header_read", 2024-03-11T21:42:26.733 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:42:26.729969+0000", 2024-03-11T21:42:26.733 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-03-11T21:42:26.733 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:42:26.733 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:42:26.733 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "throttled", 2024-03-11T21:42:26.733 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:42:26.729970+0000", 2024-03-11T21:42:26.733 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 6.8100000000000002e-07 2024-03-11T21:42:26.733 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:42:26.733 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:42:26.733 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "all_read", 2024-03-11T21:42:26.733 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:42:26.729973+0000", 2024-03-11T21:42:26.734 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 3.1250000000000001e-06 2024-03-11T21:42:26.734 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:42:26.734 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:42:26.734 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "dispatched", 2024-03-11T21:42:26.734 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:42:26.729979+0000", 2024-03-11T21:42:26.734 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 5.7629999999999997e-06 2024-03-11T21:42:26.734 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:42:26.734 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:42:26.734 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "queued_for_pg", 2024-03-11T21:42:26.734 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:42:26.729984+0000", 2024-03-11T21:42:26.734 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 5.2859999999999999e-06 2024-03-11T21:42:26.734 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:42:26.734 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:42:26.734 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "reached_pg", 2024-03-11T21:42:26.735 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:42:26.730052+0000", 2024-03-11T21:42:26.735 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 6.8156000000000003e-05 2024-03-11T21:42:26.735 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:42:26.735 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:42:26.735 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "started", 2024-03-11T21:42:26.735 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:42:26.730176+0000", 2024-03-11T21:42:26.735 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.00012351800000000001 2024-03-11T21:42:26.735 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-03-11T21:42:26.735 INFO:teuthology.orchestra.run.smithi097.stdout: ] 2024-03-11T21:42:26.735 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-03-11T21:42:26.735 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-03-11T21:42:26.735 INFO:teuthology.orchestra.run.smithi097.stdout: ], 2024-03-11T21:42:26.735 INFO:teuthology.orchestra.run.smithi097.stdout: "num_ops": 4 2024-03-11T21:42:26.735 INFO:teuthology.orchestra.run.smithi097.stdout:} 2024-03-11T21:42:26.785 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:42:26.816 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:42:26.816 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:42:26.817 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:42:26.831 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:42:26 smithi097 ceph-mon[33317]: pgmap v791: 158 pgs: 151 active+clean, 1 active+undersized+remapped, 1 active+recovery_wait, 2 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+remapped; 6.7 GiB data, 26 GiB used, 690 GiB / 715 GiB avail; 852 B/s rd, 37 MiB/s wr, 603 op/s; 8894/328470 objects degraded (2.708%); 13218/328470 objects misplaced (4.024%); 17 MiB/s, 277 objects/s recovering 2024-03-11T21:42:26.831 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:42:26 smithi097 ceph-mon[33317]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-11T21:42:26.831 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:42:26 smithi097 ceph-mon[33317]: from='client.? v1:172.21.15.92:0/1679702323' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-03-11T21:42:26.832 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:42:26 smithi097 ceph-mon[33317]: osdmap e394: 8 total, 8 up, 8 in 2024-03-11T21:42:26.865 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:42:26 smithi092 ceph-mon[29572]: pgmap v791: 158 pgs: 151 active+clean, 1 active+undersized+remapped, 1 active+recovery_wait, 2 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+remapped; 6.7 GiB data, 26 GiB used, 690 GiB / 715 GiB avail; 852 B/s rd, 37 MiB/s wr, 603 op/s; 8894/328470 objects degraded (2.708%); 13218/328470 objects misplaced (4.024%); 17 MiB/s, 277 objects/s recovering 2024-03-11T21:42:26.865 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:42:26 smithi092 ceph-mon[29572]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-11T21:42:26.865 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:42:26 smithi092 ceph-mon[29572]: from='client.? v1:172.21.15.92:0/1679702323' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-03-11T21:42:26.865 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:42:26 smithi092 ceph-mon[29572]: osdmap e394: 8 total, 8 up, 8 in 2024-03-11T21:42:26.866 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:42:26 smithi092 ceph-mon[37563]: pgmap v791: 158 pgs: 151 active+clean, 1 active+undersized+remapped, 1 active+recovery_wait, 2 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+remapped; 6.7 GiB data, 26 GiB used, 690 GiB / 715 GiB avail; 852 B/s rd, 37 MiB/s wr, 603 op/s; 8894/328470 objects degraded (2.708%); 13218/328470 objects misplaced (4.024%); 17 MiB/s, 277 objects/s recovering 2024-03-11T21:42:26.866 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:42:26 smithi092 ceph-mon[37563]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-11T21:42:26.866 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:42:26 smithi092 ceph-mon[37563]: from='client.? v1:172.21.15.92:0/1679702323' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-03-11T21:42:26.866 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:42:26 smithi092 ceph-mon[37563]: osdmap e394: 8 total, 8 up, 8 in 2024-03-11T21:42:26.917 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:42:26.961 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:42:26.962 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:42:26.962 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:42:27.063 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:42:27.128 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:42:27.129 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:42:27.129 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:42:27.230 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:42:27.299 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:42:27.300 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:42:27.300 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:42:27.401 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:42:27.446 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:42:27.446 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:42:27.447 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:42:27.453 DEBUG:teuthology.orchestra.run.smithi097:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid 00db0016-dfed-11ee-95c8-87774f69a715 -- ceph daemon osd.4 dump_blocked_ops 2024-03-11T21:42:27.547 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:42:27.582 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:42:27.582 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:42:27.583 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:42:27.617 INFO:teuthology.orchestra.run.smithi092.stderr:Inferring config /var/lib/ceph/00db0016-dfed-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T21:42:27.683 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:42:27.715 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:42:27.715 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:42:27.716 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:42:27.817 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:42:27.839 INFO:teuthology.orchestra.run.smithi097.stderr:Inferring config /var/lib/ceph/00db0016-dfed-11ee-95c8-87774f69a715/mon.b/config 2024-03-11T21:42:27.852 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:42:27.852 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:42:27.853 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:42:27.953 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:42:27.981 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:42:27.981 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:42:27.982 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:42:28.083 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:42:28.112 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:42:28.113 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:42:28.113 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:42:28.131 INFO:teuthology.orchestra.run.smithi092.stdout: 2024-03-11T21:42:28.148 INFO:teuthology.orchestra.run.smithi092.stderr:dumped all 2024-03-11T21:42:28.181 INFO:journalctl@ceph.mgr.y.smithi092.stdout:Mar 11 21:42:27 smithi092 ceph-00db0016-dfed-11ee-95c8-87774f69a715-mgr-y[29801]: ::ffff:172.21.15.97 - - [11/Mar/2024:21:42:27] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-11T21:42:28.214 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:42:28.252 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:42:28.253 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:42:28.253 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:42:28.354 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:42:28.388 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:42:28.389 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:42:28.389 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:42:28.490 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:42:28.537 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:42:28.538 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:42:28.538 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:42:28.639 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:42:28.672 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:42:28.672 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:42:28.673 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:42:28.773 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:42:28.801 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:42:28.802 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:42:28.803 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:42:28.831 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:42:28 smithi097 ceph-mon[33317]: purged_snaps scrub starts 2024-03-11T21:42:28.831 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:42:28 smithi097 ceph-mon[33317]: purged_snaps scrub ok 2024-03-11T21:42:28.832 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:42:28 smithi097 ceph-mon[33317]: pgmap v793: 158 pgs: 151 active+clean, 1 active+undersized+remapped, 1 active+recovery_wait, 2 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+remapped; 6.6 GiB data, 25 GiB used, 690 GiB / 715 GiB avail; 853 B/s rd, 38 MiB/s wr, 713 op/s; 8894/327252 objects degraded (2.718%); 13218/327252 objects misplaced (4.039%); 12 MiB/s, 199 objects/s recovering 2024-03-11T21:42:28.832 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:42:28 smithi097 ceph-mon[33317]: osdmap e395: 8 total, 8 up, 8 in 2024-03-11T21:42:28.832 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:42:28 smithi097 ceph-mon[33317]: from='client.25660 v1:172.21.15.92:0/4041593403' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-11T21:42:28.832 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:42:28 smithi097 ceph-mon[33317]: Health check update: Degraded data redundancy: 8894/327252 objects degraded (2.718%), 2 pgs degraded (PG_DEGRADED) 2024-03-11T21:42:28.832 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:42:28 smithi097 ceph-mon[33317]: osdmap e396: 8 total, 8 up, 8 in 2024-03-11T21:42:28.866 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:42:28 smithi092 ceph-mon[29572]: purged_snaps scrub starts 2024-03-11T21:42:28.866 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:42:28 smithi092 ceph-mon[29572]: purged_snaps scrub ok 2024-03-11T21:42:28.866 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:42:28 smithi092 ceph-mon[29572]: pgmap v793: 158 pgs: 151 active+clean, 1 active+undersized+remapped, 1 active+recovery_wait, 2 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+remapped; 6.6 GiB data, 25 GiB used, 690 GiB / 715 GiB avail; 853 B/s rd, 38 MiB/s wr, 713 op/s; 8894/327252 objects degraded (2.718%); 13218/327252 objects misplaced (4.039%); 12 MiB/s, 199 objects/s recovering 2024-03-11T21:42:28.866 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:42:28 smithi092 ceph-mon[29572]: osdmap e395: 8 total, 8 up, 8 in 2024-03-11T21:42:28.866 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:42:28 smithi092 ceph-mon[29572]: from='client.25660 v1:172.21.15.92:0/4041593403' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-11T21:42:28.866 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:42:28 smithi092 ceph-mon[29572]: Health check update: Degraded data redundancy: 8894/327252 objects degraded (2.718%), 2 pgs degraded (PG_DEGRADED) 2024-03-11T21:42:28.866 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:42:28 smithi092 ceph-mon[29572]: osdmap e396: 8 total, 8 up, 8 in 2024-03-11T21:42:28.867 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:42:28 smithi092 ceph-mon[37563]: purged_snaps scrub starts 2024-03-11T21:42:28.867 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:42:28 smithi092 ceph-mon[37563]: purged_snaps scrub ok 2024-03-11T21:42:28.867 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:42:28 smithi092 ceph-mon[37563]: pgmap v793: 158 pgs: 151 active+clean, 1 active+undersized+remapped, 1 active+recovery_wait, 2 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+remapped; 6.6 GiB data, 25 GiB used, 690 GiB / 715 GiB avail; 853 B/s rd, 38 MiB/s wr, 713 op/s; 8894/327252 objects degraded (2.718%); 13218/327252 objects misplaced (4.039%); 12 MiB/s, 199 objects/s recovering 2024-03-11T21:42:28.867 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:42:28 smithi092 ceph-mon[37563]: osdmap e395: 8 total, 8 up, 8 in 2024-03-11T21:42:28.867 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:42:28 smithi092 ceph-mon[37563]: from='client.25660 v1:172.21.15.92:0/4041593403' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-11T21:42:28.867 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:42:28 smithi092 ceph-mon[37563]: Health check update: Degraded data redundancy: 8894/327252 objects degraded (2.718%), 2 pgs degraded (PG_DEGRADED) 2024-03-11T21:42:28.867 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:42:28 smithi092 ceph-mon[37563]: osdmap e396: 8 total, 8 up, 8 in 2024-03-11T21:42:28.904 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:42:28.946 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:42:28.947 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:42:28.947 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:42:29.034 DEBUG:teuthology.orchestra.run.smithi092:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid 00db0016-dfed-11ee-95c8-87774f69a715 -- ceph osd unset noscrub 2024-03-11T21:42:29.048 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:42:29.074 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:42:29.074 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:42:29.075 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:42:29.176 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:42:29.231 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:42:29.231 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:42:29.232 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:42:29.332 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:42:29.361 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:42:29.362 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:42:29.362 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:42:29.463 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:42:29.491 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:42:29.492 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:42:29.492 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:42:29.593 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:42:29.621 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:42:29.622 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:42:29.622 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:42:29.723 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:42:29.787 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:42:29.788 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:42:29.788 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:42:29.889 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:42:29.925 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:42:29.926 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:42:29.926 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:42:30.027 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:42:30.058 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:42:30.058 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:42:30.059 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:42:30.160 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:42:30.195 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:42:30.195 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:42:30.195 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:42:30.245 INFO:teuthology.orchestra.run.smithi097.stdout:{ 2024-03-11T21:42:30.245 INFO:teuthology.orchestra.run.smithi097.stdout: "ops": [], 2024-03-11T21:42:30.246 INFO:teuthology.orchestra.run.smithi097.stdout: "complaint_time": 30, 2024-03-11T21:42:30.246 INFO:teuthology.orchestra.run.smithi097.stdout: "num_blocked_ops": 0 2024-03-11T21:42:30.246 INFO:teuthology.orchestra.run.smithi097.stdout:} 2024-03-11T21:42:30.297 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:42:30.344 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:42:30.345 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:42:30.345 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:42:30.446 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:42:30.479 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:42:30.480 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:42:30.480 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:42:30.581 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:42:30.604 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:42:30.605 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:42:30.605 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:42:30.706 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:42:30.751 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:42:30.752 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:42:30.752 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:42:30.832 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:42:30 smithi097 ceph-mon[33317]: pgmap v796: 158 pgs: 152 active+clean, 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovery_wait, 4 active+recovering+undersized+remapped; 6.5 GiB data, 25 GiB used, 690 GiB / 715 GiB avail; 1.2 KiB/s rd, 5.1 MiB/s wr, 617 op/s; 4354/321858 objects degraded (1.353%); 15593/321858 objects misplaced (4.845%); 12 MiB/s, 198 objects/s recovering 2024-03-11T21:42:30.853 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:42:30.865 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:42:30 smithi092 ceph-mon[29572]: pgmap v796: 158 pgs: 152 active+clean, 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovery_wait, 4 active+recovering+undersized+remapped; 6.5 GiB data, 25 GiB used, 690 GiB / 715 GiB avail; 1.2 KiB/s rd, 5.1 MiB/s wr, 617 op/s; 4354/321858 objects degraded (1.353%); 15593/321858 objects misplaced (4.845%); 12 MiB/s, 198 objects/s recovering 2024-03-11T21:42:30.866 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:42:30 smithi092 ceph-mon[37563]: pgmap v796: 158 pgs: 152 active+clean, 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovery_wait, 4 active+recovering+undersized+remapped; 6.5 GiB data, 25 GiB used, 690 GiB / 715 GiB avail; 1.2 KiB/s rd, 5.1 MiB/s wr, 617 op/s; 4354/321858 objects degraded (1.353%); 15593/321858 objects misplaced (4.845%); 12 MiB/s, 198 objects/s recovering 2024-03-11T21:42:30.895 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:42:30.895 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:42:30.896 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:42:30.991 DEBUG:teuthology.orchestra.run.smithi097:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid 00db0016-dfed-11ee-95c8-87774f69a715 -- ceph daemon osd.4 dump_historic_ops 2024-03-11T21:42:30.996 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:42:31.036 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:42:31.036 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:42:31.037 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:42:31.137 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:42:31.170 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:42:31.171 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:42:31.171 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:42:31.272 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:42:31.303 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:42:31.304 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:42:31.304 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:42:31.334 INFO:teuthology.orchestra.run.smithi097.stderr:Inferring config /var/lib/ceph/00db0016-dfed-11ee-95c8-87774f69a715/mon.b/config 2024-03-11T21:42:31.405 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:42:31.437 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:42:31.438 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:42:31.438 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:42:31.539 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:42:31.576 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:42:31.577 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:42:31.577 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:42:31.678 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:42:31.711 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:42:31.712 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:42:31.713 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:42:31.813 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:42:31.843 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:42:31.844 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:42:31.844 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:42:31.946 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:42:31.973 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:42:31.974 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:42:31.974 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:42:32.075 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:42:32.121 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:42:32.121 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:42:32.122 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:42:32.223 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:42:32.260 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:42:32.261 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:42:32.261 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:42:32.362 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:42:32.390 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:42:32.390 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:42:32.391 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:42:32.492 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:42:32.524 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:42:32.525 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:42:32.525 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:42:32.626 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:42:32.658 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:42:32.658 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:42:32.658 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:42:32.759 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:42:32.796 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:42:32.796 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:42:32.797 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:42:32.815 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:42:32 smithi097 ceph-mon[33317]: pgmap v797: 158 pgs: 152 active+clean, 1 active+recovery_wait, 1 active+recovery_wait+undersized+degraded+remapped, 4 active+recovering+undersized+remapped; 6.5 GiB data, 25 GiB used, 690 GiB / 715 GiB avail; 1023 B/s rd, 4.1 MiB/s wr, 519 op/s; 4354/321513 objects degraded (1.354%); 15593/321513 objects misplaced (4.850%); 10 MiB/s, 160 objects/s recovering 2024-03-11T21:42:32.865 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:42:32 smithi092 ceph-mon[29572]: pgmap v797: 158 pgs: 152 active+clean, 1 active+recovery_wait, 1 active+recovery_wait+undersized+degraded+remapped, 4 active+recovering+undersized+remapped; 6.5 GiB data, 25 GiB used, 690 GiB / 715 GiB avail; 1023 B/s rd, 4.1 MiB/s wr, 519 op/s; 4354/321513 objects degraded (1.354%); 15593/321513 objects misplaced (4.850%); 10 MiB/s, 160 objects/s recovering 2024-03-11T21:42:32.866 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:42:32 smithi092 ceph-mon[37563]: pgmap v797: 158 pgs: 152 active+clean, 1 active+recovery_wait, 1 active+recovery_wait+undersized+degraded+remapped, 4 active+recovering+undersized+remapped; 6.5 GiB data, 25 GiB used, 690 GiB / 715 GiB avail; 1023 B/s rd, 4.1 MiB/s wr, 519 op/s; 4354/321513 objects degraded (1.354%); 15593/321513 objects misplaced (4.850%); 10 MiB/s, 160 objects/s recovering 2024-03-11T21:42:32.898 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:42:32.938 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:42:32.938 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:42:32.939 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:42:33.039 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:42:33.071 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:42:33.071 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:42:33.072 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:42:33.077 INFO:journalctl@ceph.iscsi.iscsi.a.smithi097.stdout:Mar 11 21:42:32 smithi097 ceph-00db0016-dfed-11ee-95c8-87774f69a715-iscsi-iscsi-a[60943]: debug there is no tcmu-runner data available 2024-03-11T21:42:33.173 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:42:33.199 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:42:33.200 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:42:33.200 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:42:33.222 INFO:teuthology.orchestra.run.smithi097.stdout:{ 2024-03-11T21:42:33.223 INFO:teuthology.orchestra.run.smithi097.stdout: "size": 20, 2024-03-11T21:42:33.223 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 600, 2024-03-11T21:42:33.223 INFO:teuthology.orchestra.run.smithi097.stdout: "ops": [ 2024-03-11T21:42:33.223 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:42:33.223 INFO:teuthology.orchestra.run.smithi097.stdout: "description": "pg_backfill(progress 8.18 e 191/191 lb 8:1c11ae06:::benchmark_data_smithi092_104865_object61961:head)", 2024-03-11T21:42:33.223 INFO:teuthology.orchestra.run.smithi097.stdout: "initiated_at": "2024-03-11T21:33:12.941905+0000", 2024-03-11T21:42:33.223 INFO:teuthology.orchestra.run.smithi097.stdout: "age": 560.28976927899998, 2024-03-11T21:42:33.223 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.145791578, 2024-03-11T21:42:33.223 INFO:teuthology.orchestra.run.smithi097.stdout: "type_data": { 2024-03-11T21:42:33.223 INFO:teuthology.orchestra.run.smithi097.stdout: "flag_point": "started", 2024-03-11T21:42:33.224 INFO:teuthology.orchestra.run.smithi097.stdout: "events": [ 2024-03-11T21:42:33.224 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:42:33.224 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "initiated", 2024-03-11T21:42:33.224 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:33:12.941905+0000", 2024-03-11T21:42:33.224 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-03-11T21:42:33.224 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:42:33.224 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:42:33.224 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "header_read", 2024-03-11T21:42:33.224 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:33:12.941905+0000", 2024-03-11T21:42:33.224 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-03-11T21:42:33.224 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:42:33.224 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:42:33.224 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "throttled", 2024-03-11T21:42:33.225 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:33:12.941906+0000", 2024-03-11T21:42:33.225 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 9.8400000000000002e-07 2024-03-11T21:42:33.225 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:42:33.225 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:42:33.225 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "all_read", 2024-03-11T21:42:33.225 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:33:12.941914+0000", 2024-03-11T21:42:33.225 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 8.3690000000000007e-06 2024-03-11T21:42:33.225 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:42:33.225 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:42:33.225 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "dispatched", 2024-03-11T21:42:33.226 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:33:12.941924+0000", 2024-03-11T21:42:33.226 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.043e-05 2024-03-11T21:42:33.226 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:42:33.226 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:42:33.226 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "queued_for_pg", 2024-03-11T21:42:33.226 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:33:12.941933+0000", 2024-03-11T21:42:33.226 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 8.174e-06 2024-03-11T21:42:33.226 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:42:33.226 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:42:33.226 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "reached_pg", 2024-03-11T21:42:33.227 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:33:13.087628+0000", 2024-03-11T21:42:33.227 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.14569512200000001 2024-03-11T21:42:33.227 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:42:33.227 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:42:33.227 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "started", 2024-03-11T21:42:33.227 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:33:13.087649+0000", 2024-03-11T21:42:33.227 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 2.122e-05 2024-03-11T21:42:33.227 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:42:33.227 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:42:33.228 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "done", 2024-03-11T21:42:33.228 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:33:13.087696+0000", 2024-03-11T21:42:33.228 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 4.7278999999999999e-05 2024-03-11T21:42:33.228 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-03-11T21:42:33.228 INFO:teuthology.orchestra.run.smithi097.stdout: ] 2024-03-11T21:42:33.228 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-03-11T21:42:33.228 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:42:33.228 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:42:33.229 INFO:teuthology.orchestra.run.smithi097.stdout: "description": "MOSDPGPush(8.18 191/184 [PushOp(8:1c11c6ca:::benchmark_data_smithi092_104865_object53857:head, version: 170'3474, data_included: [0~65536], data_size: 65536, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(8:1c11c6ca:::benchmark_data_smithi092_104865_object53857:head@170'3474, size: 65536, copy_subset: [(0, 65536)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 65536, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-03-11T21:42:33.229 INFO:teuthology.orchestra.run.smithi097.stdout: "initiated_at": "2024-03-11T21:33:12.941954+0000", 2024-03-11T21:42:33.229 INFO:teuthology.orchestra.run.smithi097.stdout: "age": 560.28971945700005, 2024-03-11T21:42:33.229 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.14781918199999999, 2024-03-11T21:42:33.229 INFO:teuthology.orchestra.run.smithi097.stdout: "type_data": { 2024-03-11T21:42:33.229 INFO:teuthology.orchestra.run.smithi097.stdout: "flag_point": "started", 2024-03-11T21:42:33.229 INFO:teuthology.orchestra.run.smithi097.stdout: "events": [ 2024-03-11T21:42:33.229 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:42:33.229 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "initiated", 2024-03-11T21:42:33.230 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:33:12.941954+0000", 2024-03-11T21:42:33.230 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-03-11T21:42:33.230 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:42:33.230 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:42:33.230 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "header_read", 2024-03-11T21:42:33.230 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:33:12.941954+0000", 2024-03-11T21:42:33.230 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-03-11T21:42:33.230 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:42:33.230 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:42:33.230 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "throttled", 2024-03-11T21:42:33.231 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:33:12.941955+0000", 2024-03-11T21:42:33.231 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 5.2900000000000004e-07 2024-03-11T21:42:33.231 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:42:33.231 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:42:33.231 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "all_read", 2024-03-11T21:42:33.231 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:33:12.941992+0000", 2024-03-11T21:42:33.231 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 3.7035e-05 2024-03-11T21:42:33.231 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:42:33.231 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:42:33.231 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "dispatched", 2024-03-11T21:42:33.232 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:33:13.087940+0000", 2024-03-11T21:42:33.232 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.14594783 2024-03-11T21:42:33.232 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:42:33.232 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:42:33.232 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "queued_for_pg", 2024-03-11T21:42:33.232 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:33:13.087963+0000", 2024-03-11T21:42:33.232 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 2.3234000000000002e-05 2024-03-11T21:42:33.232 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:42:33.232 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:42:33.232 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "reached_pg", 2024-03-11T21:42:33.232 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:33:13.089568+0000", 2024-03-11T21:42:33.232 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.001605399 2024-03-11T21:42:33.232 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:42:33.233 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:42:33.233 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "started", 2024-03-11T21:42:33.233 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:33:13.089583+0000", 2024-03-11T21:42:33.233 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.4151999999999999e-05 2024-03-11T21:42:33.233 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:42:33.233 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:42:33.233 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "done", 2024-03-11T21:42:33.233 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:33:13.089774+0000", 2024-03-11T21:42:33.233 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.000191003 2024-03-11T21:42:33.233 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-03-11T21:42:33.233 INFO:teuthology.orchestra.run.smithi097.stdout: ] 2024-03-11T21:42:33.234 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-03-11T21:42:33.234 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:42:33.234 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:42:33.234 INFO:teuthology.orchestra.run.smithi097.stdout: "description": "osd_repop(client.15150.0:103689 8.10 e191/184 8:09131ca0:::benchmark_data_smithi092_104865_object103688:head v 191'5947, mlcod=191'5945)", 2024-03-11T21:42:33.234 INFO:teuthology.orchestra.run.smithi097.stdout: "initiated_at": "2024-03-11T21:33:12.942004+0000", 2024-03-11T21:42:33.234 INFO:teuthology.orchestra.run.smithi097.stdout: "age": 560.28967004100002, 2024-03-11T21:42:33.234 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.151318802, 2024-03-11T21:42:33.234 INFO:teuthology.orchestra.run.smithi097.stdout: "type_data": { 2024-03-11T21:42:33.234 INFO:teuthology.orchestra.run.smithi097.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-11T21:42:33.234 INFO:teuthology.orchestra.run.smithi097.stdout: "events": [ 2024-03-11T21:42:33.234 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:42:33.234 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "initiated", 2024-03-11T21:42:33.235 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:33:12.942004+0000", 2024-03-11T21:42:33.235 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-03-11T21:42:33.235 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:42:33.235 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:42:33.235 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "header_read", 2024-03-11T21:42:33.235 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:33:12.942004+0000", 2024-03-11T21:42:33.235 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-03-11T21:42:33.235 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:42:33.235 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:42:33.235 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "throttled", 2024-03-11T21:42:33.236 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:33:12.942005+0000", 2024-03-11T21:42:33.236 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.5039999999999999e-06 2024-03-11T21:42:33.236 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:42:33.236 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:42:33.236 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "all_read", 2024-03-11T21:42:33.236 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:33:12.942012+0000", 2024-03-11T21:42:33.236 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 6.4980000000000003e-06 2024-03-11T21:42:33.236 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:42:33.236 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:42:33.236 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "dispatched", 2024-03-11T21:42:33.236 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:33:13.087428+0000", 2024-03-11T21:42:33.236 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.145416192 2024-03-11T21:42:33.237 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:42:33.237 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:42:33.237 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "queued_for_pg", 2024-03-11T21:42:33.237 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:33:13.087443+0000", 2024-03-11T21:42:33.237 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.4555e-05 2024-03-11T21:42:33.237 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:42:33.237 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:42:33.237 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "reached_pg", 2024-03-11T21:42:33.237 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:33:13.087722+0000", 2024-03-11T21:42:33.237 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.00027943500000000002 2024-03-11T21:42:33.238 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:42:33.238 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:42:33.238 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "started", 2024-03-11T21:42:33.238 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:33:13.087740+0000", 2024-03-11T21:42:33.238 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.819e-05 2024-03-11T21:42:33.238 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:42:33.238 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:42:33.238 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "commit_sent", 2024-03-11T21:42:33.238 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:33:13.093288+0000", 2024-03-11T21:42:33.238 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.0055473019999999996 2024-03-11T21:42:33.238 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:42:33.238 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:42:33.238 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "done", 2024-03-11T21:42:33.238 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:33:13.093323+0000", 2024-03-11T21:42:33.239 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 3.5126000000000002e-05 2024-03-11T21:42:33.239 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-03-11T21:42:33.239 INFO:teuthology.orchestra.run.smithi097.stdout: ] 2024-03-11T21:42:33.239 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-03-11T21:42:33.239 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:42:33.239 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:42:33.239 INFO:teuthology.orchestra.run.smithi097.stdout: "description": "osd_repop(client.15150.0:103703 8.5 e191/181 8:a2c135c2:::benchmark_data_smithi092_104865_object103702:head v 191'6015, mlcod=191'6013)", 2024-03-11T21:42:33.239 INFO:teuthology.orchestra.run.smithi097.stdout: "initiated_at": "2024-03-11T21:33:12.961140+0000", 2024-03-11T21:42:33.239 INFO:teuthology.orchestra.run.smithi097.stdout: "age": 560.27053371800002, 2024-03-11T21:42:33.239 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.130235251, 2024-03-11T21:42:33.239 INFO:teuthology.orchestra.run.smithi097.stdout: "type_data": { 2024-03-11T21:42:33.239 INFO:teuthology.orchestra.run.smithi097.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-11T21:42:33.240 INFO:teuthology.orchestra.run.smithi097.stdout: "events": [ 2024-03-11T21:42:33.240 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:42:33.240 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "initiated", 2024-03-11T21:42:33.240 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:33:12.961140+0000", 2024-03-11T21:42:33.240 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-03-11T21:42:33.240 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:42:33.240 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:42:33.240 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "header_read", 2024-03-11T21:42:33.240 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:33:12.961140+0000", 2024-03-11T21:42:33.240 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-03-11T21:42:33.241 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:42:33.241 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:42:33.241 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "throttled", 2024-03-11T21:42:33.241 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:33:12.961142+0000", 2024-03-11T21:42:33.241 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.5969999999999999e-06 2024-03-11T21:42:33.241 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:42:33.241 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:42:33.241 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "all_read", 2024-03-11T21:42:33.241 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:33:12.961163+0000", 2024-03-11T21:42:33.241 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 2.1311999999999999e-05 2024-03-11T21:42:33.241 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:42:33.242 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:42:33.242 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "dispatched", 2024-03-11T21:42:33.242 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:33:13.087461+0000", 2024-03-11T21:42:33.242 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.12629823100000001 2024-03-11T21:42:33.242 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:42:33.242 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:42:33.242 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "queued_for_pg", 2024-03-11T21:42:33.242 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:33:13.087468+0000", 2024-03-11T21:42:33.242 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 6.2729999999999999e-06 2024-03-11T21:42:33.242 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:42:33.242 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:42:33.242 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "reached_pg", 2024-03-11T21:42:33.243 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:33:13.087522+0000", 2024-03-11T21:42:33.243 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 5.4503000000000003e-05 2024-03-11T21:42:33.243 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:42:33.243 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:42:33.243 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "started", 2024-03-11T21:42:33.243 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:33:13.087544+0000", 2024-03-11T21:42:33.243 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 2.1478e-05 2024-03-11T21:42:33.243 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:42:33.243 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:42:33.243 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "commit_sent", 2024-03-11T21:42:33.243 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:33:13.091329+0000", 2024-03-11T21:42:33.243 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.0037852189999999998 2024-03-11T21:42:33.243 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:42:33.244 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:42:33.244 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "done", 2024-03-11T21:42:33.244 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:33:13.091375+0000", 2024-03-11T21:42:33.244 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 4.6638000000000003e-05 2024-03-11T21:42:33.244 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-03-11T21:42:33.244 INFO:teuthology.orchestra.run.smithi097.stdout: ] 2024-03-11T21:42:33.244 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-03-11T21:42:33.244 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:42:33.244 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:42:33.244 INFO:teuthology.orchestra.run.smithi097.stdout: "description": "osd_op(client.15150.0:104833 8.14 8:2c24d294:::benchmark_data_smithi092_104865_object104832:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e191)", 2024-03-11T21:42:33.244 INFO:teuthology.orchestra.run.smithi097.stdout: "initiated_at": "2024-03-11T21:33:14.408980+0000", 2024-03-11T21:42:33.244 INFO:teuthology.orchestra.run.smithi097.stdout: "age": 558.82269428699999, 2024-03-11T21:42:33.244 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.16163967700000001, 2024-03-11T21:42:33.244 INFO:teuthology.orchestra.run.smithi097.stdout: "type_data": { 2024-03-11T21:42:33.245 INFO:teuthology.orchestra.run.smithi097.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-11T21:42:33.245 INFO:teuthology.orchestra.run.smithi097.stdout: "client_info": { 2024-03-11T21:42:33.245 INFO:teuthology.orchestra.run.smithi097.stdout: "client": "client.15150", 2024-03-11T21:42:33.245 INFO:teuthology.orchestra.run.smithi097.stdout: "client_addr": "v1:172.21.15.92:0/436606717", 2024-03-11T21:42:33.245 INFO:teuthology.orchestra.run.smithi097.stdout: "tid": 104833 2024-03-11T21:42:33.245 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:42:33.245 INFO:teuthology.orchestra.run.smithi097.stdout: "events": [ 2024-03-11T21:42:33.245 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:42:33.245 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "initiated", 2024-03-11T21:42:33.245 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:33:14.408980+0000", 2024-03-11T21:42:33.246 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-03-11T21:42:33.246 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:42:33.246 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:42:33.246 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "header_read", 2024-03-11T21:42:33.246 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:33:14.408980+0000", 2024-03-11T21:42:33.246 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-03-11T21:42:33.246 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:42:33.246 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:42:33.246 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "throttled", 2024-03-11T21:42:33.246 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:33:14.408982+0000", 2024-03-11T21:42:33.247 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.897e-06 2024-03-11T21:42:33.247 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:42:33.247 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:42:33.247 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "all_read", 2024-03-11T21:42:33.247 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:33:14.409054+0000", 2024-03-11T21:42:33.247 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 7.2113999999999994e-05 2024-03-11T21:42:33.247 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:42:33.247 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:42:33.247 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "dispatched", 2024-03-11T21:42:33.247 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:33:14.409066+0000", 2024-03-11T21:42:33.247 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.2031999999999999e-05 2024-03-11T21:42:33.247 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:42:33.247 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:42:33.247 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "queued_for_pg", 2024-03-11T21:42:33.248 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:33:14.409078+0000", 2024-03-11T21:42:33.248 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.2717999999999999e-05 2024-03-11T21:42:33.248 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:42:33.248 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:42:33.248 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "reached_pg", 2024-03-11T21:42:33.248 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:33:14.409140+0000", 2024-03-11T21:42:33.248 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 6.1717999999999994e-05 2024-03-11T21:42:33.248 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:42:33.248 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:42:33.248 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "started", 2024-03-11T21:42:33.248 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:33:14.409237+0000", 2024-03-11T21:42:33.248 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 9.7377e-05 2024-03-11T21:42:33.249 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:42:33.249 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:42:33.249 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "waiting for subops from 0,2,5", 2024-03-11T21:42:33.249 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:33:14.409493+0000", 2024-03-11T21:42:33.249 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.00025598299999999999 2024-03-11T21:42:33.249 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:42:33.249 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:42:33.249 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "op_commit", 2024-03-11T21:42:33.249 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:33:14.410376+0000", 2024-03-11T21:42:33.249 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.00088214200000000004 2024-03-11T21:42:33.249 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:42:33.249 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:42:33.249 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "sub_op_commit_rec", 2024-03-11T21:42:33.250 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:33:14.412759+0000", 2024-03-11T21:42:33.250 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.0023838599999999998 2024-03-11T21:42:33.250 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:42:33.250 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:42:33.250 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "sub_op_commit_rec", 2024-03-11T21:42:33.250 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:33:14.413794+0000", 2024-03-11T21:42:33.250 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.001034224 2024-03-11T21:42:33.250 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:42:33.250 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:42:33.250 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "sub_op_commit_rec", 2024-03-11T21:42:33.251 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:33:14.570411+0000", 2024-03-11T21:42:33.251 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.15661767100000001 2024-03-11T21:42:33.251 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:42:33.251 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:42:33.251 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "commit_sent", 2024-03-11T21:42:33.251 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:33:14.570484+0000", 2024-03-11T21:42:33.251 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 7.2185000000000001e-05 2024-03-11T21:42:33.251 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:42:33.252 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:42:33.252 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "done", 2024-03-11T21:42:33.252 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:33:14.570619+0000", 2024-03-11T21:42:33.252 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.00013575600000000001 2024-03-11T21:42:33.252 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-03-11T21:42:33.252 INFO:teuthology.orchestra.run.smithi097.stdout: ] 2024-03-11T21:42:33.252 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-03-11T21:42:33.252 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:42:33.252 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:42:33.252 INFO:teuthology.orchestra.run.smithi097.stdout: "description": "osd_op(client.15150.0:104838 8.2 8:43f9e813:::benchmark_data_smithi092_104865_object104837:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e191)", 2024-03-11T21:42:33.253 INFO:teuthology.orchestra.run.smithi097.stdout: "initiated_at": "2024-03-11T21:33:14.413753+0000", 2024-03-11T21:42:33.253 INFO:teuthology.orchestra.run.smithi097.stdout: "age": 558.81792132400005, 2024-03-11T21:42:33.253 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.16188380799999999, 2024-03-11T21:42:33.253 INFO:teuthology.orchestra.run.smithi097.stdout: "type_data": { 2024-03-11T21:42:33.253 INFO:teuthology.orchestra.run.smithi097.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-11T21:42:33.253 INFO:teuthology.orchestra.run.smithi097.stdout: "client_info": { 2024-03-11T21:42:33.253 INFO:teuthology.orchestra.run.smithi097.stdout: "client": "client.15150", 2024-03-11T21:42:33.253 INFO:teuthology.orchestra.run.smithi097.stdout: "client_addr": "v1:172.21.15.92:0/436606717", 2024-03-11T21:42:33.253 INFO:teuthology.orchestra.run.smithi097.stdout: "tid": 104838 2024-03-11T21:42:33.253 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:42:33.254 INFO:teuthology.orchestra.run.smithi097.stdout: "events": [ 2024-03-11T21:42:33.254 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:42:33.254 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "initiated", 2024-03-11T21:42:33.254 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:33:14.413753+0000", 2024-03-11T21:42:33.254 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-03-11T21:42:33.254 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:42:33.254 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:42:33.254 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "header_read", 2024-03-11T21:42:33.254 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:33:14.413753+0000", 2024-03-11T21:42:33.255 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-03-11T21:42:33.255 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:42:33.255 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:42:33.255 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "throttled", 2024-03-11T21:42:33.255 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:33:14.413754+0000", 2024-03-11T21:42:33.255 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.7689999999999999e-06 2024-03-11T21:42:33.256 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:42:33.256 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:42:33.256 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "all_read", 2024-03-11T21:42:33.256 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:33:14.413811+0000", 2024-03-11T21:42:33.256 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 5.6573999999999997e-05 2024-03-11T21:42:33.256 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:42:33.256 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:42:33.256 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "dispatched", 2024-03-11T21:42:33.257 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:33:14.413818+0000", 2024-03-11T21:42:33.257 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 7.3459999999999996e-06 2024-03-11T21:42:33.257 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:42:33.257 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:42:33.257 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "queued_for_pg", 2024-03-11T21:42:33.257 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:33:14.413823+0000", 2024-03-11T21:42:33.257 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 4.6149999999999999e-06 2024-03-11T21:42:33.259 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:42:33.259 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:42:33.259 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "reached_pg", 2024-03-11T21:42:33.259 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:33:14.413869+0000", 2024-03-11T21:42:33.259 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 4.6105e-05 2024-03-11T21:42:33.260 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:42:33.260 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:42:33.260 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "started", 2024-03-11T21:42:33.260 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:33:14.414004+0000", 2024-03-11T21:42:33.260 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.000135017 2024-03-11T21:42:33.260 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:42:33.260 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:42:33.260 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "waiting for subops from 0,5", 2024-03-11T21:42:33.260 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:33:14.414327+0000", 2024-03-11T21:42:33.261 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.00032254799999999998 2024-03-11T21:42:33.261 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:42:33.261 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:42:33.261 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "op_commit", 2024-03-11T21:42:33.261 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:33:14.415927+0000", 2024-03-11T21:42:33.261 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.0016000610000000001 2024-03-11T21:42:33.261 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:42:33.261 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:42:33.262 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "sub_op_commit_rec", 2024-03-11T21:42:33.262 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:33:14.422347+0000", 2024-03-11T21:42:33.262 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.0064206109999999997 2024-03-11T21:42:33.262 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:42:33.262 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:42:33.262 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "sub_op_commit_rec", 2024-03-11T21:42:33.262 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:33:14.575507+0000", 2024-03-11T21:42:33.263 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.15315997100000001 2024-03-11T21:42:33.263 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:42:33.263 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:42:33.263 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "commit_sent", 2024-03-11T21:42:33.263 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:33:14.575583+0000", 2024-03-11T21:42:33.263 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 7.5488000000000006e-05 2024-03-11T21:42:33.263 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:42:33.263 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:42:33.263 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "done", 2024-03-11T21:42:33.264 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:33:14.575636+0000", 2024-03-11T21:42:33.264 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 5.3702999999999997e-05 2024-03-11T21:42:33.264 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-03-11T21:42:33.264 INFO:teuthology.orchestra.run.smithi097.stdout: ] 2024-03-11T21:42:33.264 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-03-11T21:42:33.264 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:42:33.264 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:42:33.264 INFO:teuthology.orchestra.run.smithi097.stdout: "description": "osd_repop_reply(client.15150.0:104833 8.14 e191/184 ondisk, result = 0)", 2024-03-11T21:42:33.264 INFO:teuthology.orchestra.run.smithi097.stdout: "initiated_at": "2024-03-11T21:33:14.423255+0000", 2024-03-11T21:42:33.265 INFO:teuthology.orchestra.run.smithi097.stdout: "age": 558.80841855400001, 2024-03-11T21:42:33.265 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.14737776999999999, 2024-03-11T21:42:33.265 INFO:teuthology.orchestra.run.smithi097.stdout: "type_data": { 2024-03-11T21:42:33.265 INFO:teuthology.orchestra.run.smithi097.stdout: "flag_point": "started", 2024-03-11T21:42:33.265 INFO:teuthology.orchestra.run.smithi097.stdout: "events": [ 2024-03-11T21:42:33.265 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:42:33.265 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "initiated", 2024-03-11T21:42:33.265 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:33:14.423255+0000", 2024-03-11T21:42:33.266 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-03-11T21:42:33.266 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:42:33.266 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:42:33.266 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "header_read", 2024-03-11T21:42:33.266 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:33:14.423255+0000", 2024-03-11T21:42:33.266 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-03-11T21:42:33.266 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:42:33.266 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:42:33.267 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "throttled", 2024-03-11T21:42:33.267 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:33:14.423256+0000", 2024-03-11T21:42:33.267 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 5.4000000000000002e-07 2024-03-11T21:42:33.267 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:42:33.267 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:42:33.267 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "all_read", 2024-03-11T21:42:33.267 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:33:14.423259+0000", 2024-03-11T21:42:33.267 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 2.7760000000000002e-06 2024-03-11T21:42:33.268 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:42:33.268 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:42:33.268 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "dispatched", 2024-03-11T21:42:33.268 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:33:14.423264+0000", 2024-03-11T21:42:33.268 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 5.4430000000000002e-06 2024-03-11T21:42:33.268 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:42:33.268 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:42:33.268 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "queued_for_pg", 2024-03-11T21:42:33.268 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:33:14.569972+0000", 2024-03-11T21:42:33.268 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.14670749199999999 2024-03-11T21:42:33.268 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:42:33.268 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:42:33.269 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "reached_pg", 2024-03-11T21:42:33.269 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:33:14.570394+0000", 2024-03-11T21:42:33.269 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.00042243599999999999 2024-03-11T21:42:33.269 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:42:33.269 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:42:33.269 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "started", 2024-03-11T21:42:33.269 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:33:14.570404+0000", 2024-03-11T21:42:33.269 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 9.9329999999999996e-06 2024-03-11T21:42:33.269 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:42:33.269 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:42:33.269 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "done", 2024-03-11T21:42:33.270 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:33:14.570633+0000", 2024-03-11T21:42:33.270 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.00022915000000000001 2024-03-11T21:42:33.270 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-03-11T21:42:33.270 INFO:teuthology.orchestra.run.smithi097.stdout: ] 2024-03-11T21:42:33.270 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-03-11T21:42:33.270 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:42:33.270 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:42:33.270 INFO:teuthology.orchestra.run.smithi097.stdout: "description": "osd_op(client.15150.0:105691 8.14 8:2e116b12:::benchmark_data_smithi092_104865_object105690:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e192)", 2024-03-11T21:42:33.270 INFO:teuthology.orchestra.run.smithi097.stdout: "initiated_at": "2024-03-11T21:33:15.570276+0000", 2024-03-11T21:42:33.270 INFO:teuthology.orchestra.run.smithi097.stdout: "age": 557.66139768100004, 2024-03-11T21:42:33.270 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.14029594000000001, 2024-03-11T21:42:33.270 INFO:teuthology.orchestra.run.smithi097.stdout: "type_data": { 2024-03-11T21:42:33.270 INFO:teuthology.orchestra.run.smithi097.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-11T21:42:33.271 INFO:teuthology.orchestra.run.smithi097.stdout: "client_info": { 2024-03-11T21:42:33.271 INFO:teuthology.orchestra.run.smithi097.stdout: "client": "client.15150", 2024-03-11T21:42:33.271 INFO:teuthology.orchestra.run.smithi097.stdout: "client_addr": "v1:172.21.15.92:0/436606717", 2024-03-11T21:42:33.271 INFO:teuthology.orchestra.run.smithi097.stdout: "tid": 105691 2024-03-11T21:42:33.271 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:42:33.271 INFO:teuthology.orchestra.run.smithi097.stdout: "events": [ 2024-03-11T21:42:33.271 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:42:33.271 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "initiated", 2024-03-11T21:42:33.271 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:33:15.570276+0000", 2024-03-11T21:42:33.271 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-03-11T21:42:33.271 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:42:33.271 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:42:33.271 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "header_read", 2024-03-11T21:42:33.271 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:33:15.570276+0000", 2024-03-11T21:42:33.271 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-03-11T21:42:33.272 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:42:33.272 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:42:33.272 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "throttled", 2024-03-11T21:42:33.272 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:33:15.570277+0000", 2024-03-11T21:42:33.272 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.2190000000000001e-06 2024-03-11T21:42:33.272 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:42:33.272 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:42:33.272 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "all_read", 2024-03-11T21:42:33.272 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:33:15.570382+0000", 2024-03-11T21:42:33.273 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.000104387 2024-03-11T21:42:33.273 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:42:33.273 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:42:33.273 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "dispatched", 2024-03-11T21:42:33.273 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:33:15.570390+0000", 2024-03-11T21:42:33.273 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 8.5150000000000006e-06 2024-03-11T21:42:33.273 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:42:33.273 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:42:33.273 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "queued_for_pg", 2024-03-11T21:42:33.273 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:33:15.570397+0000", 2024-03-11T21:42:33.274 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 6.8410000000000002e-06 2024-03-11T21:42:33.274 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:42:33.274 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:42:33.274 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "reached_pg", 2024-03-11T21:42:33.274 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:33:15.570442+0000", 2024-03-11T21:42:33.274 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 4.5188999999999997e-05 2024-03-11T21:42:33.274 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:42:33.274 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:42:33.274 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "started", 2024-03-11T21:42:33.275 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:33:15.570574+0000", 2024-03-11T21:42:33.275 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.00013190200000000001 2024-03-11T21:42:33.275 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:42:33.275 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:42:33.275 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "waiting for subops from 0,2,5", 2024-03-11T21:42:33.275 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:33:15.570902+0000", 2024-03-11T21:42:33.275 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.00032810599999999999 2024-03-11T21:42:33.275 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:42:33.275 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:42:33.275 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "op_commit", 2024-03-11T21:42:33.275 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:33:15.572979+0000", 2024-03-11T21:42:33.275 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.002076297 2024-03-11T21:42:33.275 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:42:33.275 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:42:33.276 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "sub_op_commit_rec", 2024-03-11T21:42:33.276 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:33:15.573189+0000", 2024-03-11T21:42:33.276 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.000210115 2024-03-11T21:42:33.276 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:42:33.276 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:42:33.276 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "sub_op_commit_rec", 2024-03-11T21:42:33.276 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:33:15.710342+0000", 2024-03-11T21:42:33.276 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.13715286700000001 2024-03-11T21:42:33.276 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:42:33.276 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:42:33.276 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "sub_op_commit_rec", 2024-03-11T21:42:33.276 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:33:15.710380+0000", 2024-03-11T21:42:33.276 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 3.8250000000000001e-05 2024-03-11T21:42:33.276 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:42:33.277 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:42:33.277 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "commit_sent", 2024-03-11T21:42:33.277 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:33:15.710443+0000", 2024-03-11T21:42:33.277 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 6.2643000000000003e-05 2024-03-11T21:42:33.277 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:42:33.277 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:42:33.277 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "done", 2024-03-11T21:42:33.277 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:33:15.710572+0000", 2024-03-11T21:42:33.277 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.00012960900000000001 2024-03-11T21:42:33.277 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-03-11T21:42:33.277 INFO:teuthology.orchestra.run.smithi097.stdout: ] 2024-03-11T21:42:33.277 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-03-11T21:42:33.277 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:42:33.277 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:42:33.277 INFO:teuthology.orchestra.run.smithi097.stdout: "description": "osd_repop(client.15150.0:105699 8.19 e192/190 8:9f1f6df3:::benchmark_data_smithi092_104865_object105698:head v 192'6115, mlcod=192'6113)", 2024-03-11T21:42:33.278 INFO:teuthology.orchestra.run.smithi097.stdout: "initiated_at": "2024-03-11T21:33:15.575196+0000", 2024-03-11T21:42:33.278 INFO:teuthology.orchestra.run.smithi097.stdout: "age": 557.65647828800002, 2024-03-11T21:42:33.278 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.13566550599999999, 2024-03-11T21:42:33.278 INFO:teuthology.orchestra.run.smithi097.stdout: "type_data": { 2024-03-11T21:42:33.278 INFO:teuthology.orchestra.run.smithi097.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-11T21:42:33.278 INFO:teuthology.orchestra.run.smithi097.stdout: "events": [ 2024-03-11T21:42:33.278 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:42:33.278 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "initiated", 2024-03-11T21:42:33.279 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:33:15.575196+0000", 2024-03-11T21:42:33.279 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-03-11T21:42:33.279 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:42:33.279 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:42:33.279 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "header_read", 2024-03-11T21:42:33.279 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:33:15.575196+0000", 2024-03-11T21:42:33.279 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-03-11T21:42:33.279 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:42:33.279 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:42:33.279 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "throttled", 2024-03-11T21:42:33.279 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:33:15.575197+0000", 2024-03-11T21:42:33.279 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.062e-06 2024-03-11T21:42:33.279 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:42:33.279 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:42:33.280 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "all_read", 2024-03-11T21:42:33.280 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:33:15.575219+0000", 2024-03-11T21:42:33.280 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 2.2705999999999999e-05 2024-03-11T21:42:33.280 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:42:33.280 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:42:33.280 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "dispatched", 2024-03-11T21:42:33.280 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:33:15.575227+0000", 2024-03-11T21:42:33.280 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 7.2729999999999997e-06 2024-03-11T21:42:33.280 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:42:33.280 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:42:33.280 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "queued_for_pg", 2024-03-11T21:42:33.280 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:33:15.575231+0000", 2024-03-11T21:42:33.280 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 4.7060000000000003e-06 2024-03-11T21:42:33.280 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:42:33.281 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:42:33.281 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "reached_pg", 2024-03-11T21:42:33.281 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:33:15.575284+0000", 2024-03-11T21:42:33.281 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 5.2312000000000003e-05 2024-03-11T21:42:33.281 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:42:33.281 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:42:33.281 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "started", 2024-03-11T21:42:33.281 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:33:15.575296+0000", 2024-03-11T21:42:33.281 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.2199e-05 2024-03-11T21:42:33.281 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:42:33.281 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:42:33.281 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "commit_sent", 2024-03-11T21:42:33.281 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:33:15.576718+0000", 2024-03-11T21:42:33.281 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.0014220299999999999 2024-03-11T21:42:33.281 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:42:33.282 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:42:33.282 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "done", 2024-03-11T21:42:33.282 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:33:15.710861+0000", 2024-03-11T21:42:33.282 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.13414321800000001 2024-03-11T21:42:33.282 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-03-11T21:42:33.282 INFO:teuthology.orchestra.run.smithi097.stdout: ] 2024-03-11T21:42:33.282 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-03-11T21:42:33.282 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:42:33.282 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:42:33.282 INFO:teuthology.orchestra.run.smithi097.stdout: "description": "osd_repop(client.15150.0:105698 8.15 e192/184 8:ae50fbab:::benchmark_data_smithi092_104865_object105697:head v 192'6110, mlcod=192'6108)", 2024-03-11T21:42:33.282 INFO:teuthology.orchestra.run.smithi097.stdout: "initiated_at": "2024-03-11T21:33:15.576540+0000", 2024-03-11T21:42:33.282 INFO:teuthology.orchestra.run.smithi097.stdout: "age": 557.65513355300004, 2024-03-11T21:42:33.283 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.13498511699999999, 2024-03-11T21:42:33.283 INFO:teuthology.orchestra.run.smithi097.stdout: "type_data": { 2024-03-11T21:42:33.283 INFO:teuthology.orchestra.run.smithi097.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-11T21:42:33.283 INFO:teuthology.orchestra.run.smithi097.stdout: "events": [ 2024-03-11T21:42:33.283 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:42:33.283 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "initiated", 2024-03-11T21:42:33.283 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:33:15.576540+0000", 2024-03-11T21:42:33.283 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-03-11T21:42:33.283 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:42:33.283 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:42:33.283 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "header_read", 2024-03-11T21:42:33.283 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:33:15.576540+0000", 2024-03-11T21:42:33.284 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-03-11T21:42:33.284 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:42:33.284 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:42:33.284 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "throttled", 2024-03-11T21:42:33.284 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:33:15.576541+0000", 2024-03-11T21:42:33.284 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 8.9599999999999998e-07 2024-03-11T21:42:33.284 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:42:33.284 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:42:33.284 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "all_read", 2024-03-11T21:42:33.284 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:33:15.576612+0000", 2024-03-11T21:42:33.284 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 7.0414999999999996e-05 2024-03-11T21:42:33.284 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:42:33.284 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:42:33.284 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "dispatched", 2024-03-11T21:42:33.284 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:33:15.576619+0000", 2024-03-11T21:42:33.285 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 7.5900000000000002e-06 2024-03-11T21:42:33.285 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:42:33.285 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:42:33.285 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "queued_for_pg", 2024-03-11T21:42:33.285 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:33:15.576624+0000", 2024-03-11T21:42:33.285 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 5.0490000000000003e-06 2024-03-11T21:42:33.285 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:42:33.285 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:42:33.285 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "reached_pg", 2024-03-11T21:42:33.285 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:33:15.576662+0000", 2024-03-11T21:42:33.285 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 3.8167999999999998e-05 2024-03-11T21:42:33.285 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:42:33.285 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:42:33.285 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "started", 2024-03-11T21:42:33.285 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:33:15.576678+0000", 2024-03-11T21:42:33.286 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.5284000000000001e-05 2024-03-11T21:42:33.286 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:42:33.286 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:42:33.286 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "commit_sent", 2024-03-11T21:42:33.286 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:33:15.711489+0000", 2024-03-11T21:42:33.286 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.13481104399999999 2024-03-11T21:42:33.286 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:42:33.286 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:42:33.286 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "done", 2024-03-11T21:42:33.286 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:33:15.711525+0000", 2024-03-11T21:42:33.286 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 3.6671000000000002e-05 2024-03-11T21:42:33.287 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-03-11T21:42:33.287 INFO:teuthology.orchestra.run.smithi097.stdout: ] 2024-03-11T21:42:33.287 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-03-11T21:42:33.287 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:42:33.287 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:42:33.287 INFO:teuthology.orchestra.run.smithi097.stdout: "description": "osd_repop(client.15150.0:105687 8.e e192/157 8:70cc9ca8:::benchmark_data_smithi092_104865_object105686:head v 192'6655, mlcod=192'6652)", 2024-03-11T21:42:33.287 INFO:teuthology.orchestra.run.smithi097.stdout: "initiated_at": "2024-03-11T21:33:15.576636+0000", 2024-03-11T21:42:33.287 INFO:teuthology.orchestra.run.smithi097.stdout: "age": 557.65503782200005, 2024-03-11T21:42:33.287 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.14071803299999999, 2024-03-11T21:42:33.287 INFO:teuthology.orchestra.run.smithi097.stdout: "type_data": { 2024-03-11T21:42:33.287 INFO:teuthology.orchestra.run.smithi097.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-11T21:42:33.287 INFO:teuthology.orchestra.run.smithi097.stdout: "events": [ 2024-03-11T21:42:33.287 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:42:33.287 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "initiated", 2024-03-11T21:42:33.288 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:33:15.576636+0000", 2024-03-11T21:42:33.288 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-03-11T21:42:33.288 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:42:33.288 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:42:33.288 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "header_read", 2024-03-11T21:42:33.288 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:33:15.576636+0000", 2024-03-11T21:42:33.288 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-03-11T21:42:33.288 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:42:33.288 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:42:33.288 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "throttled", 2024-03-11T21:42:33.288 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:33:15.576637+0000", 2024-03-11T21:42:33.288 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 5.1200000000000003e-07 2024-03-11T21:42:33.288 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:42:33.288 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:42:33.288 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "all_read", 2024-03-11T21:42:33.289 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:33:15.576675+0000", 2024-03-11T21:42:33.289 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 3.8831000000000001e-05 2024-03-11T21:42:33.289 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:42:33.289 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:42:33.289 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "dispatched", 2024-03-11T21:42:33.289 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:33:15.576682+0000", 2024-03-11T21:42:33.289 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 6.1380000000000004e-06 2024-03-11T21:42:33.289 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:42:33.289 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:42:33.289 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "queued_for_pg", 2024-03-11T21:42:33.289 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:33:15.576687+0000", 2024-03-11T21:42:33.289 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 5.3879999999999999e-06 2024-03-11T21:42:33.289 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:42:33.289 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:42:33.290 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "reached_pg", 2024-03-11T21:42:33.290 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:33:15.710970+0000", 2024-03-11T21:42:33.290 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.13428273499999999 2024-03-11T21:42:33.290 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:42:33.290 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:42:33.290 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "started", 2024-03-11T21:42:33.290 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:33:15.710995+0000", 2024-03-11T21:42:33.290 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 2.4825999999999999e-05 2024-03-11T21:42:33.290 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:42:33.290 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:42:33.290 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "commit_sent", 2024-03-11T21:42:33.290 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:33:15.717312+0000", 2024-03-11T21:42:33.290 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.0063174679999999997 2024-03-11T21:42:33.290 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:42:33.291 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:42:33.291 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "done", 2024-03-11T21:42:33.291 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:33:15.717354+0000", 2024-03-11T21:42:33.291 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 4.2135000000000002e-05 2024-03-11T21:42:33.291 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-03-11T21:42:33.291 INFO:teuthology.orchestra.run.smithi097.stdout: ] 2024-03-11T21:42:33.291 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-03-11T21:42:33.291 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:42:33.291 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:42:33.291 INFO:teuthology.orchestra.run.smithi097.stdout: "description": "osd_repop_reply(client.15150.0:105691 8.14 e192/184 ondisk, result = 0)", 2024-03-11T21:42:33.291 INFO:teuthology.orchestra.run.smithi097.stdout: "initiated_at": "2024-03-11T21:33:15.579477+0000", 2024-03-11T21:42:33.291 INFO:teuthology.orchestra.run.smithi097.stdout: "age": 557.65219645499997, 2024-03-11T21:42:33.291 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.130872288, 2024-03-11T21:42:33.291 INFO:teuthology.orchestra.run.smithi097.stdout: "type_data": { 2024-03-11T21:42:33.291 INFO:teuthology.orchestra.run.smithi097.stdout: "flag_point": "started", 2024-03-11T21:42:33.292 INFO:teuthology.orchestra.run.smithi097.stdout: "events": [ 2024-03-11T21:42:33.292 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:42:33.292 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "initiated", 2024-03-11T21:42:33.292 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:33:15.579477+0000", 2024-03-11T21:42:33.292 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-03-11T21:42:33.292 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:42:33.292 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:42:33.292 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "header_read", 2024-03-11T21:42:33.292 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:33:15.579477+0000", 2024-03-11T21:42:33.292 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-03-11T21:42:33.292 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:42:33.292 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:42:33.292 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "throttled", 2024-03-11T21:42:33.293 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:33:15.579479+0000", 2024-03-11T21:42:33.293 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.119e-06 2024-03-11T21:42:33.293 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:42:33.293 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:42:33.293 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "all_read", 2024-03-11T21:42:33.293 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:33:15.579483+0000", 2024-03-11T21:42:33.293 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 3.9280000000000003e-06 2024-03-11T21:42:33.293 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:42:33.293 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:42:33.293 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "dispatched", 2024-03-11T21:42:33.293 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:33:15.709618+0000", 2024-03-11T21:42:33.293 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.13013551700000001 2024-03-11T21:42:33.293 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:42:33.294 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:42:33.294 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "queued_for_pg", 2024-03-11T21:42:33.294 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:33:15.709632+0000", 2024-03-11T21:42:33.294 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.3949e-05 2024-03-11T21:42:33.294 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:42:33.294 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:42:33.294 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "reached_pg", 2024-03-11T21:42:33.294 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:33:15.710327+0000", 2024-03-11T21:42:33.294 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.00069474000000000001 2024-03-11T21:42:33.294 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:42:33.294 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:42:33.294 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "started", 2024-03-11T21:42:33.295 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:33:15.710335+0000", 2024-03-11T21:42:33.295 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 8.3599999999999996e-06 2024-03-11T21:42:33.295 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:42:33.295 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:42:33.295 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "done", 2024-03-11T21:42:33.295 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:33:15.710350+0000", 2024-03-11T21:42:33.295 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.4674999999999999e-05 2024-03-11T21:42:33.295 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-03-11T21:42:33.295 INFO:teuthology.orchestra.run.smithi097.stdout: ] 2024-03-11T21:42:33.295 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-03-11T21:42:33.295 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:42:33.295 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:42:33.295 INFO:teuthology.orchestra.run.smithi097.stdout: "description": "MOSDPGPush(8.18 192/184 [PushOp(8:1ca50ff6:::benchmark_data_smithi092_104865_object101503:head, version: 191'6019, data_included: [0~65536], data_size: 65536, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(8:1ca50ff6:::benchmark_data_smithi092_104865_object101503:head@191'6019, size: 65536, copy_subset: [(0, 65536)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 65536, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-03-11T21:42:33.295 INFO:teuthology.orchestra.run.smithi097.stdout: "initiated_at": "2024-03-11T21:33:15.579714+0000", 2024-03-11T21:42:33.296 INFO:teuthology.orchestra.run.smithi097.stdout: "age": 557.65195954599994, 2024-03-11T21:42:33.296 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.13040016800000001, 2024-03-11T21:42:33.296 INFO:teuthology.orchestra.run.smithi097.stdout: "type_data": { 2024-03-11T21:42:33.296 INFO:teuthology.orchestra.run.smithi097.stdout: "flag_point": "started", 2024-03-11T21:42:33.296 INFO:teuthology.orchestra.run.smithi097.stdout: "events": [ 2024-03-11T21:42:33.296 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:42:33.296 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "initiated", 2024-03-11T21:42:33.296 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:33:15.579714+0000", 2024-03-11T21:42:33.296 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-03-11T21:42:33.297 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:42:33.297 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:42:33.297 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "header_read", 2024-03-11T21:42:33.297 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:33:15.579714+0000", 2024-03-11T21:42:33.297 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-03-11T21:42:33.297 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:42:33.297 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:42:33.297 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "throttled", 2024-03-11T21:42:33.297 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:33:15.579715+0000", 2024-03-11T21:42:33.297 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 8.8800000000000001e-07 2024-03-11T21:42:33.297 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:42:33.297 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:42:33.298 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "all_read", 2024-03-11T21:42:33.298 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:33:15.579829+0000", 2024-03-11T21:42:33.298 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.000113684 2024-03-11T21:42:33.298 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:42:33.298 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:42:33.298 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "dispatched", 2024-03-11T21:42:33.298 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:33:15.709645+0000", 2024-03-11T21:42:33.298 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.129815556 2024-03-11T21:42:33.298 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:42:33.298 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:42:33.298 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "queued_for_pg", 2024-03-11T21:42:33.298 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:33:15.709672+0000", 2024-03-11T21:42:33.298 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 2.7046000000000002e-05 2024-03-11T21:42:33.298 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:42:33.299 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:42:33.299 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "reached_pg", 2024-03-11T21:42:33.299 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:33:15.709893+0000", 2024-03-11T21:42:33.299 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.00022120000000000001 2024-03-11T21:42:33.299 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:42:33.299 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:42:33.299 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "started", 2024-03-11T21:42:33.299 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:33:15.709904+0000", 2024-03-11T21:42:33.299 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.1348e-05 2024-03-11T21:42:33.299 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:42:33.299 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:42:33.299 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "done", 2024-03-11T21:42:33.299 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:33:15.710115+0000", 2024-03-11T21:42:33.299 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.00021044600000000001 2024-03-11T21:42:33.299 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-03-11T21:42:33.300 INFO:teuthology.orchestra.run.smithi097.stdout: ] 2024-03-11T21:42:33.300 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-03-11T21:42:33.300 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:42:33.300 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:42:33.300 INFO:teuthology.orchestra.run.smithi097.stdout: "description": "osd_op(client.15150.0:107413 8.2 8:40ceb2f9:::benchmark_data_smithi092_104865_object107412:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e192)", 2024-03-11T21:42:33.300 INFO:teuthology.orchestra.run.smithi097.stdout: "initiated_at": "2024-03-11T21:33:17.605806+0000", 2024-03-11T21:42:33.300 INFO:teuthology.orchestra.run.smithi097.stdout: "age": 555.62586822499998, 2024-03-11T21:42:33.300 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.144884283, 2024-03-11T21:42:33.300 INFO:teuthology.orchestra.run.smithi097.stdout: "type_data": { 2024-03-11T21:42:33.300 INFO:teuthology.orchestra.run.smithi097.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-11T21:42:33.300 INFO:teuthology.orchestra.run.smithi097.stdout: "client_info": { 2024-03-11T21:42:33.300 INFO:teuthology.orchestra.run.smithi097.stdout: "client": "client.15150", 2024-03-11T21:42:33.300 INFO:teuthology.orchestra.run.smithi097.stdout: "client_addr": "v1:172.21.15.92:0/436606717", 2024-03-11T21:42:33.300 INFO:teuthology.orchestra.run.smithi097.stdout: "tid": 107413 2024-03-11T21:42:33.300 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:42:33.301 INFO:teuthology.orchestra.run.smithi097.stdout: "events": [ 2024-03-11T21:42:33.301 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:42:33.301 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "initiated", 2024-03-11T21:42:33.301 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:33:17.605806+0000", 2024-03-11T21:42:33.301 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-03-11T21:42:33.301 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:42:33.301 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:42:33.301 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "header_read", 2024-03-11T21:42:33.301 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:33:17.605806+0000", 2024-03-11T21:42:33.301 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-03-11T21:42:33.301 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:42:33.301 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:42:33.301 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "throttled", 2024-03-11T21:42:33.301 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:33:17.605807+0000", 2024-03-11T21:42:33.302 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.725e-06 2024-03-11T21:42:33.302 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:42:33.302 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:42:33.302 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "all_read", 2024-03-11T21:42:33.302 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:33:17.605956+0000", 2024-03-11T21:42:33.302 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.00014857600000000001 2024-03-11T21:42:33.302 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:42:33.302 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:42:33.302 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "dispatched", 2024-03-11T21:42:33.302 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:33:17.605965+0000", 2024-03-11T21:42:33.302 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 9.2390000000000008e-06 2024-03-11T21:42:33.302 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:42:33.302 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:42:33.302 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "queued_for_pg", 2024-03-11T21:42:33.302 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:33:17.605972+0000", 2024-03-11T21:42:33.303 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 7.2309999999999999e-06 2024-03-11T21:42:33.303 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:42:33.303 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:42:33.303 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "reached_pg", 2024-03-11T21:42:33.303 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:33:17.606036+0000", 2024-03-11T21:42:33.303 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 6.3769000000000006e-05 2024-03-11T21:42:33.303 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:42:33.303 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:42:33.303 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "started", 2024-03-11T21:42:33.303 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:33:17.606194+0000", 2024-03-11T21:42:33.303 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.00015822000000000001 2024-03-11T21:42:33.303 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:42:33.303 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:42:33.303 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "waiting for subops from 0,5", 2024-03-11T21:42:33.304 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:33:17.606509+0000", 2024-03-11T21:42:33.304 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.00031476399999999998 2024-03-11T21:42:33.304 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:42:33.304 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:42:33.304 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "op_commit", 2024-03-11T21:42:33.304 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:33:17.608696+0000", 2024-03-11T21:42:33.304 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.002186463 2024-03-11T21:42:33.304 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:42:33.304 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:42:33.304 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "sub_op_commit_rec", 2024-03-11T21:42:33.304 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:33:17.609084+0000", 2024-03-11T21:42:33.304 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.00038835699999999998 2024-03-11T21:42:33.304 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:42:33.304 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:42:33.304 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "sub_op_commit_rec", 2024-03-11T21:42:33.305 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:33:17.750621+0000", 2024-03-11T21:42:33.305 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.14153661000000001 2024-03-11T21:42:33.305 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:42:33.305 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:42:33.305 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "commit_sent", 2024-03-11T21:42:33.305 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:33:17.750660+0000", 2024-03-11T21:42:33.305 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 3.9623000000000003e-05 2024-03-11T21:42:33.305 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:42:33.305 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:42:33.305 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "done", 2024-03-11T21:42:33.305 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:33:17.750690+0000", 2024-03-11T21:42:33.305 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 2.9706e-05 2024-03-11T21:42:33.306 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-03-11T21:42:33.306 INFO:teuthology.orchestra.run.smithi097.stdout: ] 2024-03-11T21:42:33.306 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-03-11T21:42:33.306 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:42:33.306 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:42:33.306 INFO:teuthology.orchestra.run.smithi097.stdout: "description": "osd_op(client.15150.0:107415 8.14 8:2aa43716:::benchmark_data_smithi092_104865_object107414:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e192)", 2024-03-11T21:42:33.307 INFO:teuthology.orchestra.run.smithi097.stdout: "initiated_at": "2024-03-11T21:33:17.607663+0000", 2024-03-11T21:42:33.307 INFO:teuthology.orchestra.run.smithi097.stdout: "age": 555.62401120599998, 2024-03-11T21:42:33.307 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.14653977300000001, 2024-03-11T21:42:33.307 INFO:teuthology.orchestra.run.smithi097.stdout: "type_data": { 2024-03-11T21:42:33.307 INFO:teuthology.orchestra.run.smithi097.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-11T21:42:33.307 INFO:teuthology.orchestra.run.smithi097.stdout: "client_info": { 2024-03-11T21:42:33.307 INFO:teuthology.orchestra.run.smithi097.stdout: "client": "client.15150", 2024-03-11T21:42:33.307 INFO:teuthology.orchestra.run.smithi097.stdout: "client_addr": "v1:172.21.15.92:0/436606717", 2024-03-11T21:42:33.307 INFO:teuthology.orchestra.run.smithi097.stdout: "tid": 107415 2024-03-11T21:42:33.307 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:42:33.307 INFO:teuthology.orchestra.run.smithi097.stdout: "events": [ 2024-03-11T21:42:33.307 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:42:33.307 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "initiated", 2024-03-11T21:42:33.307 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:33:17.607663+0000", 2024-03-11T21:42:33.307 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-03-11T21:42:33.308 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:42:33.308 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:42:33.308 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "header_read", 2024-03-11T21:42:33.308 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:33:17.607663+0000", 2024-03-11T21:42:33.308 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-03-11T21:42:33.308 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:42:33.308 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:42:33.308 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "throttled", 2024-03-11T21:42:33.308 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:33:17.607664+0000", 2024-03-11T21:42:33.308 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.4109999999999999e-06 2024-03-11T21:42:33.308 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:42:33.308 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:42:33.308 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "all_read", 2024-03-11T21:42:33.308 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:33:17.607694+0000", 2024-03-11T21:42:33.308 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 2.9876999999999999e-05 2024-03-11T21:42:33.309 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:42:33.309 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:42:33.309 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "dispatched", 2024-03-11T21:42:33.309 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:33:17.607702+0000", 2024-03-11T21:42:33.309 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 8.0509999999999997e-06 2024-03-11T21:42:33.309 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:42:33.309 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:42:33.309 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "queued_for_pg", 2024-03-11T21:42:33.309 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:33:17.607708+0000", 2024-03-11T21:42:33.309 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 6.0719999999999996e-06 2024-03-11T21:42:33.309 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:42:33.309 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:42:33.309 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "reached_pg", 2024-03-11T21:42:33.310 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:33:17.607976+0000", 2024-03-11T21:42:33.310 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.00026800400000000002 2024-03-11T21:42:33.310 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:42:33.310 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:42:33.310 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "started", 2024-03-11T21:42:33.310 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:33:17.608127+0000", 2024-03-11T21:42:33.310 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.00015091700000000001 2024-03-11T21:42:33.310 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:42:33.310 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:42:33.310 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "waiting for subops from 0,2,5", 2024-03-11T21:42:33.310 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:33:17.608443+0000", 2024-03-11T21:42:33.310 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.00031553499999999998 2024-03-11T21:42:33.310 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:42:33.311 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:42:33.311 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "op_commit", 2024-03-11T21:42:33.311 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:33:17.609960+0000", 2024-03-11T21:42:33.311 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.001517755 2024-03-11T21:42:33.311 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:42:33.311 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:42:33.311 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "sub_op_commit_rec", 2024-03-11T21:42:33.311 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:33:17.610552+0000", 2024-03-11T21:42:33.311 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.00059122799999999996 2024-03-11T21:42:33.311 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:42:33.311 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:42:33.311 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "sub_op_commit_rec", 2024-03-11T21:42:33.311 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:33:17.611040+0000", 2024-03-11T21:42:33.311 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.00048814799999999999 2024-03-11T21:42:33.312 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:42:33.312 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:42:33.312 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "sub_op_commit_rec", 2024-03-11T21:42:33.312 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:33:17.754041+0000", 2024-03-11T21:42:33.312 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.14300163799999999 2024-03-11T21:42:33.312 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:42:33.312 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:42:33.312 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "commit_sent", 2024-03-11T21:42:33.312 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:33:17.754084+0000", 2024-03-11T21:42:33.312 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 4.2675000000000001e-05 2024-03-11T21:42:33.312 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:42:33.312 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:42:33.312 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "done", 2024-03-11T21:42:33.312 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:33:17.754202+0000", 2024-03-11T21:42:33.312 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.00011846200000000001 2024-03-11T21:42:33.313 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-03-11T21:42:33.313 INFO:teuthology.orchestra.run.smithi097.stdout: ] 2024-03-11T21:42:33.313 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-03-11T21:42:33.313 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:42:33.313 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:42:33.313 INFO:teuthology.orchestra.run.smithi097.stdout: "description": "osd_repop(client.15744.0:69290 10.6 e371/356 10:64e7935d:::benchmark_data_smithi092_161462_object69289:head v 371'4305, mlcod=371'4303)", 2024-03-11T21:42:33.313 INFO:teuthology.orchestra.run.smithi097.stdout: "initiated_at": "2024-03-11T21:41:27.813843+0000", 2024-03-11T21:42:33.313 INFO:teuthology.orchestra.run.smithi097.stdout: "age": 65.417830617000007, 2024-03-11T21:42:33.313 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.151117429, 2024-03-11T21:42:33.313 INFO:teuthology.orchestra.run.smithi097.stdout: "type_data": { 2024-03-11T21:42:33.313 INFO:teuthology.orchestra.run.smithi097.stdout: "flag_point": "started", 2024-03-11T21:42:33.313 INFO:teuthology.orchestra.run.smithi097.stdout: "events": [ 2024-03-11T21:42:33.313 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:42:33.313 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "initiated", 2024-03-11T21:42:33.314 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:41:27.813843+0000", 2024-03-11T21:42:33.314 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-03-11T21:42:33.314 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:42:33.314 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:42:33.314 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "header_read", 2024-03-11T21:42:33.314 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:41:27.813843+0000", 2024-03-11T21:42:33.314 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-03-11T21:42:33.314 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:42:33.314 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:42:33.314 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "throttled", 2024-03-11T21:42:33.314 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:41:27.813844+0000", 2024-03-11T21:42:33.314 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 4.15e-07 2024-03-11T21:42:33.314 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:42:33.315 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:42:33.315 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "all_read", 2024-03-11T21:42:33.315 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:41:27.813871+0000", 2024-03-11T21:42:33.315 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 2.7078999999999999e-05 2024-03-11T21:42:33.315 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:42:33.315 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:42:33.315 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "dispatched", 2024-03-11T21:42:33.315 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:41:27.813883+0000", 2024-03-11T21:42:33.315 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.1977000000000001e-05 2024-03-11T21:42:33.315 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:42:33.315 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:42:33.315 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "queued_for_pg", 2024-03-11T21:42:33.315 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:41:27.813892+0000", 2024-03-11T21:42:33.316 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 9.0999999999999993e-06 2024-03-11T21:42:33.316 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:42:33.316 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:42:33.316 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "reached_pg", 2024-03-11T21:42:33.316 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:41:27.814328+0000", 2024-03-11T21:42:33.316 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.00043647500000000002 2024-03-11T21:42:33.316 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:42:33.316 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:42:33.316 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "started", 2024-03-11T21:42:33.316 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:41:27.814350+0000", 2024-03-11T21:42:33.316 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 2.1223e-05 2024-03-11T21:42:33.316 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:42:33.316 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:42:33.316 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "done", 2024-03-11T21:42:33.316 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:41:27.964961+0000", 2024-03-11T21:42:33.317 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.15061115999999999 2024-03-11T21:42:33.317 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-03-11T21:42:33.317 INFO:teuthology.orchestra.run.smithi097.stdout: ] 2024-03-11T21:42:33.317 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-03-11T21:42:33.317 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:42:33.317 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:42:33.317 INFO:teuthology.orchestra.run.smithi097.stdout: "description": "osd_op(client.15744.0:69295 10.1 10.ecc5e7a1 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e371)", 2024-03-11T21:42:33.317 INFO:teuthology.orchestra.run.smithi097.stdout: "initiated_at": "2024-03-11T21:41:27.826949+0000", 2024-03-11T21:42:33.317 INFO:teuthology.orchestra.run.smithi097.stdout: "age": 65.404724512000001, 2024-03-11T21:42:33.317 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.14217101400000001, 2024-03-11T21:42:33.317 INFO:teuthology.orchestra.run.smithi097.stdout: "type_data": { 2024-03-11T21:42:33.317 INFO:teuthology.orchestra.run.smithi097.stdout: "flag_point": "reached pg", 2024-03-11T21:42:33.317 INFO:teuthology.orchestra.run.smithi097.stdout: "client_info": { 2024-03-11T21:42:33.317 INFO:teuthology.orchestra.run.smithi097.stdout: "client": "client.15744", 2024-03-11T21:42:33.317 INFO:teuthology.orchestra.run.smithi097.stdout: "client_addr": "v1:172.21.15.92:0/3334000812", 2024-03-11T21:42:33.318 INFO:teuthology.orchestra.run.smithi097.stdout: "tid": 69295 2024-03-11T21:42:33.318 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:42:33.318 INFO:teuthology.orchestra.run.smithi097.stdout: "events": [ 2024-03-11T21:42:33.318 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:42:33.318 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "initiated", 2024-03-11T21:42:33.318 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:41:27.826949+0000", 2024-03-11T21:42:33.318 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-03-11T21:42:33.318 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:42:33.318 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:42:33.318 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "header_read", 2024-03-11T21:42:33.318 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:41:27.826949+0000", 2024-03-11T21:42:33.318 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-03-11T21:42:33.318 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:42:33.318 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:42:33.319 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "throttled", 2024-03-11T21:42:33.319 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:41:27.826952+0000", 2024-03-11T21:42:33.319 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 2.751e-06 2024-03-11T21:42:33.319 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:42:33.319 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:42:33.319 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "all_read", 2024-03-11T21:42:33.319 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:41:27.826991+0000", 2024-03-11T21:42:33.319 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 3.9298000000000002e-05 2024-03-11T21:42:33.319 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:42:33.319 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:42:33.319 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "dispatched", 2024-03-11T21:42:33.320 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:41:27.827007+0000", 2024-03-11T21:42:33.320 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.5141e-05 2024-03-11T21:42:33.320 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:42:33.320 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:42:33.320 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "queued_for_pg", 2024-03-11T21:42:33.320 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:41:27.827016+0000", 2024-03-11T21:42:33.320 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 9.8460000000000003e-06 2024-03-11T21:42:33.320 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:42:33.320 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:42:33.320 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "reached_pg", 2024-03-11T21:42:33.320 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:41:27.969022+0000", 2024-03-11T21:42:33.320 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.14200602800000001 2024-03-11T21:42:33.320 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:42:33.320 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:42:33.321 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "done", 2024-03-11T21:42:33.321 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:41:27.969120+0000", 2024-03-11T21:42:33.321 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 9.7949999999999996e-05 2024-03-11T21:42:33.321 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-03-11T21:42:33.321 INFO:teuthology.orchestra.run.smithi097.stdout: ] 2024-03-11T21:42:33.321 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-03-11T21:42:33.321 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:42:33.321 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:42:33.321 INFO:teuthology.orchestra.run.smithi097.stdout: "description": "osd_op(client.15744.0:69295 10.1 10:85e7a337:::benchmark_data_smithi092_161462_object69294:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e372)", 2024-03-11T21:42:33.321 INFO:teuthology.orchestra.run.smithi097.stdout: "initiated_at": "2024-03-11T21:41:27.827037+0000", 2024-03-11T21:42:33.321 INFO:teuthology.orchestra.run.smithi097.stdout: "age": 65.404637354000002, 2024-03-11T21:42:33.321 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.0170514900000001, 2024-03-11T21:42:33.321 INFO:teuthology.orchestra.run.smithi097.stdout: "type_data": { 2024-03-11T21:42:33.322 INFO:teuthology.orchestra.run.smithi097.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-11T21:42:33.322 INFO:teuthology.orchestra.run.smithi097.stdout: "client_info": { 2024-03-11T21:42:33.322 INFO:teuthology.orchestra.run.smithi097.stdout: "client": "client.15744", 2024-03-11T21:42:33.322 INFO:teuthology.orchestra.run.smithi097.stdout: "client_addr": "v1:172.21.15.92:0/3334000812", 2024-03-11T21:42:33.322 INFO:teuthology.orchestra.run.smithi097.stdout: "tid": 69295 2024-03-11T21:42:33.322 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:42:33.322 INFO:teuthology.orchestra.run.smithi097.stdout: "events": [ 2024-03-11T21:42:33.322 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:42:33.322 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "initiated", 2024-03-11T21:42:33.322 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:41:27.827037+0000", 2024-03-11T21:42:33.322 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-03-11T21:42:33.323 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:42:33.323 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:42:33.323 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "header_read", 2024-03-11T21:42:33.323 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:41:27.827037+0000", 2024-03-11T21:42:33.323 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-03-11T21:42:33.323 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:42:33.323 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:42:33.323 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "throttled", 2024-03-11T21:42:33.323 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:41:27.827037+0000", 2024-03-11T21:42:33.323 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 7.0999999999999998e-07 2024-03-11T21:42:33.323 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:42:33.323 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:42:33.323 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "all_read", 2024-03-11T21:42:33.323 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:41:27.827067+0000", 2024-03-11T21:42:33.324 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 2.9376000000000001e-05 2024-03-11T21:42:33.324 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:42:33.324 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:42:33.324 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "dispatched", 2024-03-11T21:42:33.324 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:41:27.827074+0000", 2024-03-11T21:42:33.324 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 6.8539999999999999e-06 2024-03-11T21:42:33.324 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:42:33.324 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:42:33.324 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "queued_for_pg", 2024-03-11T21:42:33.324 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:41:27.827079+0000", 2024-03-11T21:42:33.324 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 5.6459999999999998e-06 2024-03-11T21:42:33.324 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:42:33.324 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:42:33.324 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "reached_pg", 2024-03-11T21:42:33.325 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:41:27.969273+0000", 2024-03-11T21:42:33.325 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.142194131 2024-03-11T21:42:33.325 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:42:33.325 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:42:33.325 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "waiting for peered", 2024-03-11T21:42:33.325 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:41:27.969284+0000", 2024-03-11T21:42:33.325 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.0491e-05 2024-03-11T21:42:33.325 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:42:33.325 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:42:33.325 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "reached_pg", 2024-03-11T21:42:33.325 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:41:28.830462+0000", 2024-03-11T21:42:33.325 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.861178208 2024-03-11T21:42:33.325 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:42:33.326 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:42:33.326 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "started", 2024-03-11T21:42:33.326 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:41:28.830604+0000", 2024-03-11T21:42:33.327 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.00014221599999999999 2024-03-11T21:42:33.327 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:42:33.327 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:42:33.327 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "waiting for subops from 0,7", 2024-03-11T21:42:33.327 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:41:28.830767+0000", 2024-03-11T21:42:33.327 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.00016259400000000001 2024-03-11T21:42:33.327 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:42:33.327 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:42:33.327 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "op_commit", 2024-03-11T21:42:33.327 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:41:28.836048+0000", 2024-03-11T21:42:33.327 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.0052813030000000002 2024-03-11T21:42:33.327 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:42:33.328 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:42:33.328 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "sub_op_commit_rec", 2024-03-11T21:42:33.328 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:41:28.843753+0000", 2024-03-11T21:42:33.328 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.0077048439999999998 2024-03-11T21:42:33.328 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:42:33.328 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:42:33.328 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "sub_op_commit_rec", 2024-03-11T21:42:33.328 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:41:28.843975+0000", 2024-03-11T21:42:33.328 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.000221788 2024-03-11T21:42:33.328 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:42:33.328 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:42:33.328 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "commit_sent", 2024-03-11T21:42:33.328 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:41:28.844039+0000", 2024-03-11T21:42:33.329 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 6.4274000000000002e-05 2024-03-11T21:42:33.329 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:42:33.329 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:42:33.329 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "done", 2024-03-11T21:42:33.329 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:41:28.844088+0000", 2024-03-11T21:42:33.329 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 4.9054999999999997e-05 2024-03-11T21:42:33.329 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-03-11T21:42:33.329 INFO:teuthology.orchestra.run.smithi097.stdout: ] 2024-03-11T21:42:33.329 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-03-11T21:42:33.329 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:42:33.329 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:42:33.329 INFO:teuthology.orchestra.run.smithi097.stdout: "description": "osd_op(client.15744.0:114336 10.1 10:82704bfd:::benchmark_data_smithi092_161462_object4360:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e396)", 2024-03-11T21:42:33.329 INFO:teuthology.orchestra.run.smithi097.stdout: "initiated_at": "2024-03-11T21:42:29.699303+0000", 2024-03-11T21:42:33.329 INFO:teuthology.orchestra.run.smithi097.stdout: "age": 3.5323708759999999, 2024-03-11T21:42:33.330 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.073560707000000003, 2024-03-11T21:42:33.330 INFO:teuthology.orchestra.run.smithi097.stdout: "type_data": { 2024-03-11T21:42:33.330 INFO:teuthology.orchestra.run.smithi097.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-11T21:42:33.330 INFO:teuthology.orchestra.run.smithi097.stdout: "client_info": { 2024-03-11T21:42:33.330 INFO:teuthology.orchestra.run.smithi097.stdout: "client": "client.15744", 2024-03-11T21:42:33.330 INFO:teuthology.orchestra.run.smithi097.stdout: "client_addr": "v1:172.21.15.92:0/3334000812", 2024-03-11T21:42:33.330 INFO:teuthology.orchestra.run.smithi097.stdout: "tid": 114336 2024-03-11T21:42:33.330 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:42:33.330 INFO:teuthology.orchestra.run.smithi097.stdout: "events": [ 2024-03-11T21:42:33.330 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:42:33.330 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "initiated", 2024-03-11T21:42:33.330 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:42:29.699303+0000", 2024-03-11T21:42:33.330 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-03-11T21:42:33.330 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:42:33.331 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:42:33.331 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "header_read", 2024-03-11T21:42:33.331 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:42:29.699303+0000", 2024-03-11T21:42:33.331 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-03-11T21:42:33.331 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:42:33.331 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:42:33.331 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "throttled", 2024-03-11T21:42:33.331 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:42:29.699304+0000", 2024-03-11T21:42:33.331 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.42e-06 2024-03-11T21:42:33.331 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:42:33.331 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:42:33.331 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "all_read", 2024-03-11T21:42:33.331 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:42:29.699308+0000", 2024-03-11T21:42:33.331 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 3.799e-06 2024-03-11T21:42:33.332 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:42:33.332 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:42:33.332 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "dispatched", 2024-03-11T21:42:33.332 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:42:29.699318+0000", 2024-03-11T21:42:33.332 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 9.7389999999999994e-06 2024-03-11T21:42:33.332 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:42:33.332 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:42:33.332 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "queued_for_pg", 2024-03-11T21:42:33.332 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:42:29.699326+0000", 2024-03-11T21:42:33.332 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 8.1289999999999996e-06 2024-03-11T21:42:33.332 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:42:33.332 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:42:33.332 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "reached_pg", 2024-03-11T21:42:33.332 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:42:29.699381+0000", 2024-03-11T21:42:33.333 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 5.4941000000000001e-05 2024-03-11T21:42:33.333 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:42:33.333 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:42:33.333 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "started", 2024-03-11T21:42:33.333 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:42:29.699504+0000", 2024-03-11T21:42:33.333 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.00012269900000000001 2024-03-11T21:42:33.333 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:42:33.333 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:42:33.333 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "waiting for subops from 0,7", 2024-03-11T21:42:33.333 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:42:29.699820+0000", 2024-03-11T21:42:33.333 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.00031608100000000001 2024-03-11T21:42:33.333 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:42:33.333 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:42:33.333 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "op_commit", 2024-03-11T21:42:33.334 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:42:29.701815+0000", 2024-03-11T21:42:33.334 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.001995433 2024-03-11T21:42:33.334 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:42:33.334 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:42:33.334 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "sub_op_commit_rec", 2024-03-11T21:42:33.334 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:42:29.703284+0000", 2024-03-11T21:42:33.334 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.0014686790000000001 2024-03-11T21:42:33.334 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:42:33.334 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:42:33.334 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "sub_op_commit_rec", 2024-03-11T21:42:33.334 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:42:29.772765+0000", 2024-03-11T21:42:33.335 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:42:33.335 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.069481381999999994 2024-03-11T21:42:33.335 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:42:33.336 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:42:33.336 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "commit_sent", 2024-03-11T21:42:33.336 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:42:29.772817+0000", 2024-03-11T21:42:33.336 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 5.2145999999999998e-05 2024-03-11T21:42:33.336 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:42:33.336 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:42:33.336 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "done", 2024-03-11T21:42:33.336 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:42:29.772864+0000", 2024-03-11T21:42:33.336 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 4.6258999999999999e-05 2024-03-11T21:42:33.336 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-03-11T21:42:33.336 INFO:teuthology.orchestra.run.smithi097.stdout: ] 2024-03-11T21:42:33.336 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-03-11T21:42:33.336 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:42:33.336 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:42:33.336 INFO:teuthology.orchestra.run.smithi097.stdout: "description": "osd_op(client.15744.0:117314 10.e 10:760a43f6:::benchmark_data_smithi092_161462_object7338:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e397)", 2024-03-11T21:42:33.337 INFO:teuthology.orchestra.run.smithi097.stdout: "initiated_at": "2024-03-11T21:42:33.121226+0000", 2024-03-11T21:42:33.337 INFO:teuthology.orchestra.run.smithi097.stdout: "age": 0.11044762800000001, 2024-03-11T21:42:33.337 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.082866044, 2024-03-11T21:42:33.337 INFO:teuthology.orchestra.run.smithi097.stdout: "type_data": { 2024-03-11T21:42:33.337 INFO:teuthology.orchestra.run.smithi097.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-11T21:42:33.337 INFO:teuthology.orchestra.run.smithi097.stdout: "client_info": { 2024-03-11T21:42:33.337 INFO:teuthology.orchestra.run.smithi097.stdout: "client": "client.15744", 2024-03-11T21:42:33.337 INFO:teuthology.orchestra.run.smithi097.stdout: "client_addr": "v1:172.21.15.92:0/3334000812", 2024-03-11T21:42:33.337 INFO:teuthology.orchestra.run.smithi097.stdout: "tid": 117314 2024-03-11T21:42:33.337 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:42:33.337 INFO:teuthology.orchestra.run.smithi097.stdout: "events": [ 2024-03-11T21:42:33.337 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:42:33.337 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "initiated", 2024-03-11T21:42:33.337 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:42:33.121226+0000", 2024-03-11T21:42:33.338 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-03-11T21:42:33.338 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:42:33.338 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:42:33.338 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "header_read", 2024-03-11T21:42:33.338 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:42:33.121226+0000", 2024-03-11T21:42:33.338 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-03-11T21:42:33.338 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:42:33.338 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:42:33.338 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "throttled", 2024-03-11T21:42:33.338 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:42:33.121228+0000", 2024-03-11T21:42:33.338 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.432e-06 2024-03-11T21:42:33.338 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:42:33.338 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:42:33.338 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "all_read", 2024-03-11T21:42:33.338 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:42:33.121232+0000", 2024-03-11T21:42:33.339 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 4.2470000000000002e-06 2024-03-11T21:42:33.339 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:42:33.339 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:42:33.339 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "dispatched", 2024-03-11T21:42:33.339 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:42:33.121241+0000", 2024-03-11T21:42:33.339 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 9.4550000000000001e-06 2024-03-11T21:42:33.339 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:42:33.339 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:42:33.339 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "queued_for_pg", 2024-03-11T21:42:33.339 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:42:33.121247+0000", 2024-03-11T21:42:33.339 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 5.4960000000000004e-06 2024-03-11T21:42:33.339 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:42:33.339 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:42:33.339 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "reached_pg", 2024-03-11T21:42:33.340 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:42:33.121299+0000", 2024-03-11T21:42:33.340 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 5.2346999999999999e-05 2024-03-11T21:42:33.340 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:42:33.340 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:42:33.340 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "started", 2024-03-11T21:42:33.340 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:42:33.121409+0000", 2024-03-11T21:42:33.340 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.000109591 2024-03-11T21:42:33.340 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:42:33.340 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:42:33.340 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "waiting for subops from 2,6", 2024-03-11T21:42:33.340 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:42:33.121521+0000", 2024-03-11T21:42:33.340 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.00011247300000000001 2024-03-11T21:42:33.340 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:42:33.340 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:42:33.340 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "op_commit", 2024-03-11T21:42:33.341 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:42:33.123568+0000", 2024-03-11T21:42:33.341 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.0020465560000000002 2024-03-11T21:42:33.341 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:42:33.341 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:42:33.341 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "sub_op_commit_rec", 2024-03-11T21:42:33.341 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:42:33.124075+0000", 2024-03-11T21:42:33.341 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.00050735199999999998 2024-03-11T21:42:33.341 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:42:33.341 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:42:33.341 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "sub_op_commit_rec", 2024-03-11T21:42:33.341 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:42:33.203952+0000", 2024-03-11T21:42:33.341 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.079876587999999998 2024-03-11T21:42:33.341 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:42:33.341 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:42:33.342 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "commit_sent", 2024-03-11T21:42:33.342 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:42:33.204046+0000", 2024-03-11T21:42:33.342 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 9.3912000000000005e-05 2024-03-11T21:42:33.342 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:42:33.342 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:42:33.342 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "done", 2024-03-11T21:42:33.342 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:42:33.204092+0000", 2024-03-11T21:42:33.342 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 4.6594999999999997e-05 2024-03-11T21:42:33.342 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-03-11T21:42:33.342 INFO:teuthology.orchestra.run.smithi097.stdout: ] 2024-03-11T21:42:33.342 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-03-11T21:42:33.342 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-03-11T21:42:33.342 INFO:teuthology.orchestra.run.smithi097.stdout: ] 2024-03-11T21:42:33.342 INFO:teuthology.orchestra.run.smithi097.stdout:} 2024-03-11T21:42:33.369 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:42:33.369 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:42:33.369 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:42:33.470 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:42:33.501 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:42:33.502 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:42:33.502 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:42:33.603 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:42:33.644 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:42:33.645 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:42:33.645 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:42:33.746 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:42:33.789 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:42:33.789 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:42:33.789 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:42:33.865 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:42:33 smithi092 ceph-mon[29572]: osdmap e397: 8 total, 8 up, 8 in 2024-03-11T21:42:33.866 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:42:33 smithi092 ceph-mon[29572]: from='client.24469 v1:172.21.15.97:0/404208310' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:42:33.866 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:42:33 smithi092 ceph-mon[29572]: Health check update: Degraded data redundancy: 4354/321513 objects degraded (1.354%), 1 pg degraded (PG_DEGRADED) 2024-03-11T21:42:33.866 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:42:33 smithi092 ceph-mon[29572]: osdmap e398: 8 total, 8 up, 8 in 2024-03-11T21:42:33.866 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:42:33 smithi092 ceph-mon[37563]: osdmap e397: 8 total, 8 up, 8 in 2024-03-11T21:42:33.866 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:42:33 smithi092 ceph-mon[37563]: from='client.24469 v1:172.21.15.97:0/404208310' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:42:33.866 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:42:33 smithi092 ceph-mon[37563]: Health check update: Degraded data redundancy: 4354/321513 objects degraded (1.354%), 1 pg degraded (PG_DEGRADED) 2024-03-11T21:42:33.866 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:42:33 smithi092 ceph-mon[37563]: osdmap e398: 8 total, 8 up, 8 in 2024-03-11T21:42:33.893 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:42:33.929 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:42:33.930 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:42:33.931 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:42:34.031 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:42:34.053 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:42:33 smithi097 ceph-mon[33317]: osdmap e397: 8 total, 8 up, 8 in 2024-03-11T21:42:34.054 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:42:33 smithi097 ceph-mon[33317]: from='client.24469 v1:172.21.15.97:0/404208310' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:42:34.054 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:42:33 smithi097 ceph-mon[33317]: Health check update: Degraded data redundancy: 4354/321513 objects degraded (1.354%), 1 pg degraded (PG_DEGRADED) 2024-03-11T21:42:34.054 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:42:33 smithi097 ceph-mon[33317]: osdmap e398: 8 total, 8 up, 8 in 2024-03-11T21:42:34.064 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:42:34.065 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:42:34.066 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:42:34.097 DEBUG:teuthology.orchestra.run.smithi097:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid 00db0016-dfed-11ee-95c8-87774f69a715 -- ceph daemon osd.5 dump_ops_in_flight 2024-03-11T21:42:34.167 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:42:34.208 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:42:34.209 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:42:34.209 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:42:34.310 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:42:34.334 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:42:34.335 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:42:34.335 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:42:34.436 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:42:34.475 INFO:teuthology.orchestra.run.smithi097.stderr:Inferring config /var/lib/ceph/00db0016-dfed-11ee-95c8-87774f69a715/mon.b/config 2024-03-11T21:42:34.480 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:42:34.481 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:42:34.481 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:42:34.582 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:42:34.611 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:42:34.612 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:42:34.612 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:42:34.713 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:42:34.744 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:42:34.745 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:42:34.746 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:42:34.846 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:42:34.865 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:42:34 smithi092 ceph-mon[29572]: pgmap v799: 158 pgs: 152 active+clean, 1 active+recovery_wait, 5 active+recovering+undersized+remapped; 6.3 GiB data, 25 GiB used, 690 GiB / 715 GiB avail; 1023 B/s rd, 2.2 MiB/s wr, 969 op/s; 18907/311094 objects misplaced (6.078%); 15 MiB/s, 235 objects/s recovering 2024-03-11T21:42:34.865 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:42:34 smithi092 ceph-mon[29572]: purged_snaps scrub starts 2024-03-11T21:42:34.865 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:42:34 smithi092 ceph-mon[29572]: purged_snaps scrub ok 2024-03-11T21:42:34.866 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:42:34 smithi092 ceph-mon[37563]: pgmap v799: 158 pgs: 152 active+clean, 1 active+recovery_wait, 5 active+recovering+undersized+remapped; 6.3 GiB data, 25 GiB used, 690 GiB / 715 GiB avail; 1023 B/s rd, 2.2 MiB/s wr, 969 op/s; 18907/311094 objects misplaced (6.078%); 15 MiB/s, 235 objects/s recovering 2024-03-11T21:42:34.866 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:42:34 smithi092 ceph-mon[37563]: purged_snaps scrub starts 2024-03-11T21:42:34.866 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:42:34 smithi092 ceph-mon[37563]: purged_snaps scrub ok 2024-03-11T21:42:34.883 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:42:34.884 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:42:34.884 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:42:34.930 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:42:34 smithi097 ceph-mon[33317]: pgmap v799: 158 pgs: 152 active+clean, 1 active+recovery_wait, 5 active+recovering+undersized+remapped; 6.3 GiB data, 25 GiB used, 690 GiB / 715 GiB avail; 1023 B/s rd, 2.2 MiB/s wr, 969 op/s; 18907/311094 objects misplaced (6.078%); 15 MiB/s, 235 objects/s recovering 2024-03-11T21:42:34.930 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:42:34 smithi097 ceph-mon[33317]: purged_snaps scrub starts 2024-03-11T21:42:34.931 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:42:34 smithi097 ceph-mon[33317]: purged_snaps scrub ok 2024-03-11T21:42:34.985 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:42:35.012 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:42:35.013 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:42:35.013 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:42:35.114 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:42:35.147 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:42:35.148 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:42:35.148 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:42:35.249 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:42:35.280 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:42:35.281 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:42:35.281 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:42:35.382 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:42:35.430 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:42:35.431 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:42:35.431 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:42:35.532 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:42:35.569 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:42:35.570 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:42:35.570 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:42:35.671 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:42:35.704 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:42:35.705 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:42:35.705 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:42:35.734 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:42:35 smithi097 ceph-mon[33317]: osdmap e399: 8 total, 8 up, 8 in 2024-03-11T21:42:35.734 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:42:35 smithi097 ceph-mon[33317]: purged_snaps scrub starts 2024-03-11T21:42:35.734 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:42:35 smithi097 ceph-mon[33317]: purged_snaps scrub ok 2024-03-11T21:42:35.806 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:42:35.830 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:42:35.830 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:42:35.831 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:42:35.865 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:42:35 smithi092 ceph-mon[29572]: osdmap e399: 8 total, 8 up, 8 in 2024-03-11T21:42:35.866 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:42:35 smithi092 ceph-mon[29572]: purged_snaps scrub starts 2024-03-11T21:42:35.866 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:42:35 smithi092 ceph-mon[29572]: purged_snaps scrub ok 2024-03-11T21:42:35.866 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:42:35 smithi092 ceph-mon[37563]: osdmap e399: 8 total, 8 up, 8 in 2024-03-11T21:42:35.866 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:42:35 smithi092 ceph-mon[37563]: purged_snaps scrub starts 2024-03-11T21:42:35.866 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:42:35 smithi092 ceph-mon[37563]: purged_snaps scrub ok 2024-03-11T21:42:35.932 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:42:35.969 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:42:35.969 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:42:35.969 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:42:36.070 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:42:36.117 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:42:36.118 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:42:36.118 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:42:36.219 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:42:36.249 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:42:36.250 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:42:36.250 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:42:36.351 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:42:36.375 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:42:36.376 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:42:36.376 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:42:36.477 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:42:36.506 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:42:36.507 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:42:36.507 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:42:36.608 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:42:36.646 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:42:36.647 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:42:36.648 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:42:36.732 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:42:36 smithi097 ceph-mon[33317]: pgmap v802: 158 pgs: 1 active+recovery_wait, 1 creating+peering, 152 active+clean, 4 active+recovering+undersized+remapped; 6.3 GiB data, 24 GiB used, 691 GiB / 715 GiB avail; 853 B/s rd, 0 B/s wr, 721 op/s; 18204/308886 objects misplaced (5.893%); 7.6 MiB/s, 121 objects/s recovering 2024-03-11T21:42:36.732 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:42:36 smithi097 ceph-mon[33317]: osdmap e400: 8 total, 8 up, 8 in 2024-03-11T21:42:36.748 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:42:36.789 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:42:36.790 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:42:36.790 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:42:36.865 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:42:36 smithi092 ceph-mon[29572]: pgmap v802: 158 pgs: 1 active+recovery_wait, 1 creating+peering, 152 active+clean, 4 active+recovering+undersized+remapped; 6.3 GiB data, 24 GiB used, 691 GiB / 715 GiB avail; 853 B/s rd, 0 B/s wr, 721 op/s; 18204/308886 objects misplaced (5.893%); 7.6 MiB/s, 121 objects/s recovering 2024-03-11T21:42:36.866 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:42:36 smithi092 ceph-mon[29572]: osdmap e400: 8 total, 8 up, 8 in 2024-03-11T21:42:36.866 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:42:36 smithi092 ceph-mon[37563]: pgmap v802: 158 pgs: 1 active+recovery_wait, 1 creating+peering, 152 active+clean, 4 active+recovering+undersized+remapped; 6.3 GiB data, 24 GiB used, 691 GiB / 715 GiB avail; 853 B/s rd, 0 B/s wr, 721 op/s; 18204/308886 objects misplaced (5.893%); 7.6 MiB/s, 121 objects/s recovering 2024-03-11T21:42:36.866 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:42:36 smithi092 ceph-mon[37563]: osdmap e400: 8 total, 8 up, 8 in 2024-03-11T21:42:36.891 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:42:36.930 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:42:36.930 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:42:36.931 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:42:37.031 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:42:37.069 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:42:37.069 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:42:37.070 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:42:37.171 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:42:37.220 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:42:37.221 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:42:37.221 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:42:37.322 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:42:37.348 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:42:37.349 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:42:37.349 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:42:37.419 INFO:teuthology.orchestra.run.smithi097.stdout:{ 2024-03-11T21:42:37.419 INFO:teuthology.orchestra.run.smithi097.stdout: "ops": [ 2024-03-11T21:42:37.419 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:42:37.419 INFO:teuthology.orchestra.run.smithi097.stdout: "description": "osd_op(client.15744.0:120845 10.9 10:9231998c:::benchmark_data_smithi092_161462_object10869:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e400)", 2024-03-11T21:42:37.419 INFO:teuthology.orchestra.run.smithi097.stdout: "initiated_at": "2024-03-11T21:42:37.419261+0000", 2024-03-11T21:42:37.419 INFO:teuthology.orchestra.run.smithi097.stdout: "age": 0.0064240599999999997, 2024-03-11T21:42:37.419 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.0064582370000000004, 2024-03-11T21:42:37.419 INFO:teuthology.orchestra.run.smithi097.stdout: "type_data": { 2024-03-11T21:42:37.419 INFO:teuthology.orchestra.run.smithi097.stdout: "flag_point": "waiting for sub ops", 2024-03-11T21:42:37.419 INFO:teuthology.orchestra.run.smithi097.stdout: "client_info": { 2024-03-11T21:42:37.419 INFO:teuthology.orchestra.run.smithi097.stdout: "client": "client.15744", 2024-03-11T21:42:37.420 INFO:teuthology.orchestra.run.smithi097.stdout: "client_addr": "v1:172.21.15.92:0/3334000812", 2024-03-11T21:42:37.420 INFO:teuthology.orchestra.run.smithi097.stdout: "tid": 120845 2024-03-11T21:42:37.420 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:42:37.420 INFO:teuthology.orchestra.run.smithi097.stdout: "events": [ 2024-03-11T21:42:37.420 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:42:37.420 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "initiated", 2024-03-11T21:42:37.420 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:42:37.419261+0000", 2024-03-11T21:42:37.420 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-03-11T21:42:37.420 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:42:37.420 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:42:37.420 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "header_read", 2024-03-11T21:42:37.420 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:42:37.419261+0000", 2024-03-11T21:42:37.420 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-03-11T21:42:37.420 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:42:37.420 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:42:37.421 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "throttled", 2024-03-11T21:42:37.421 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:42:37.419263+0000", 2024-03-11T21:42:37.421 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 2.0669999999999999e-06 2024-03-11T21:42:37.421 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:42:37.421 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:42:37.421 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "all_read", 2024-03-11T21:42:37.421 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:42:37.419270+0000", 2024-03-11T21:42:37.421 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 6.6309999999999997e-06 2024-03-11T21:42:37.421 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:42:37.421 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:42:37.421 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "dispatched", 2024-03-11T21:42:37.421 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:42:37.419285+0000", 2024-03-11T21:42:37.421 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.5082999999999999e-05 2024-03-11T21:42:37.421 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:42:37.421 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:42:37.422 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "queued_for_pg", 2024-03-11T21:42:37.422 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:42:37.419294+0000", 2024-03-11T21:42:37.422 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 9.5829999999999996e-06 2024-03-11T21:42:37.422 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:42:37.422 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:42:37.422 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "reached_pg", 2024-03-11T21:42:37.422 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:42:37.419368+0000", 2024-03-11T21:42:37.422 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 7.4009000000000002e-05 2024-03-11T21:42:37.422 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:42:37.422 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:42:37.422 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "started", 2024-03-11T21:42:37.422 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:42:37.419516+0000", 2024-03-11T21:42:37.422 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.000148169 2024-03-11T21:42:37.423 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:42:37.423 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:42:37.423 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "waiting for subops from 2,7", 2024-03-11T21:42:37.423 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:42:37.419937+0000", 2024-03-11T21:42:37.423 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.000420553 2024-03-11T21:42:37.423 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:42:37.423 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:42:37.423 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "sub_op_commit_rec", 2024-03-11T21:42:37.423 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:42:37.422967+0000", 2024-03-11T21:42:37.423 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.003029627 2024-03-11T21:42:37.423 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:42:37.423 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:42:37.424 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "op_commit", 2024-03-11T21:42:37.424 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:42:37.422979+0000", 2024-03-11T21:42:37.424 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.2840000000000001e-05 2024-03-11T21:42:37.424 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-03-11T21:42:37.424 INFO:teuthology.orchestra.run.smithi097.stdout: ] 2024-03-11T21:42:37.424 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-03-11T21:42:37.424 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:42:37.424 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:42:37.424 INFO:teuthology.orchestra.run.smithi097.stdout: "description": "osd_repop(client.15744.0:120844 10.16 e400/393 10:6c15bedb:::benchmark_data_smithi092_161462_object10868:head v 400'5883, mlcod=400'5880)", 2024-03-11T21:42:37.424 INFO:teuthology.orchestra.run.smithi097.stdout: "initiated_at": "2024-03-11T21:42:37.424366+0000", 2024-03-11T21:42:37.424 INFO:teuthology.orchestra.run.smithi097.stdout: "age": 0.0013193219999999999, 2024-03-11T21:42:37.424 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.00144863, 2024-03-11T21:42:37.425 INFO:teuthology.orchestra.run.smithi097.stdout: "type_data": { 2024-03-11T21:42:37.425 INFO:teuthology.orchestra.run.smithi097.stdout: "flag_point": "started", 2024-03-11T21:42:37.425 INFO:teuthology.orchestra.run.smithi097.stdout: "events": [ 2024-03-11T21:42:37.425 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:42:37.425 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "initiated", 2024-03-11T21:42:37.425 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:42:37.424366+0000", 2024-03-11T21:42:37.425 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-03-11T21:42:37.425 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:42:37.425 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:42:37.425 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "header_read", 2024-03-11T21:42:37.425 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:42:37.424366+0000", 2024-03-11T21:42:37.425 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-03-11T21:42:37.425 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:42:37.425 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:42:37.425 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "throttled", 2024-03-11T21:42:37.426 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:42:37.424367+0000", 2024-03-11T21:42:37.426 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.4789999999999999e-06 2024-03-11T21:42:37.426 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:42:37.426 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:42:37.426 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "all_read", 2024-03-11T21:42:37.426 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:42:37.424374+0000", 2024-03-11T21:42:37.426 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 7.3239999999999997e-06 2024-03-11T21:42:37.426 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:42:37.426 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:42:37.426 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "dispatched", 2024-03-11T21:42:37.426 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:42:37.424390+0000", 2024-03-11T21:42:37.426 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.525e-05 2024-03-11T21:42:37.426 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:42:37.426 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:42:37.427 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "queued_for_pg", 2024-03-11T21:42:37.427 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:42:37.424397+0000", 2024-03-11T21:42:37.427 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 7.7139999999999993e-06 2024-03-11T21:42:37.427 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:42:37.427 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:42:37.427 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "reached_pg", 2024-03-11T21:42:37.427 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:42:37.424472+0000", 2024-03-11T21:42:37.427 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 7.4881000000000004e-05 2024-03-11T21:42:37.427 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:42:37.427 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:42:37.427 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "started", 2024-03-11T21:42:37.427 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:42:37.424502+0000", 2024-03-11T21:42:37.427 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 2.9799999999999999e-05 2024-03-11T21:42:37.427 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-03-11T21:42:37.427 INFO:teuthology.orchestra.run.smithi097.stdout: ] 2024-03-11T21:42:37.428 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-03-11T21:42:37.428 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-03-11T21:42:37.428 INFO:teuthology.orchestra.run.smithi097.stdout: ], 2024-03-11T21:42:37.428 INFO:teuthology.orchestra.run.smithi097.stdout: "num_ops": 2 2024-03-11T21:42:37.428 INFO:teuthology.orchestra.run.smithi097.stdout:} 2024-03-11T21:42:37.450 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:42:37.485 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:42:37.486 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:42:37.486 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:42:37.587 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:42:37.612 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:42:37.613 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:42:37.613 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:42:37.714 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:42:37.753 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:42:37.753 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:42:37.754 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:42:37.854 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:42:37.880 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:42:37.881 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:42:37.881 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:42:37.982 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:42:38.014 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:42:38.015 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:42:38.015 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:42:38.116 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:42:38.149 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:42:38.150 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:42:38.150 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:42:38.186 INFO:journalctl@ceph.mgr.y.smithi092.stdout:Mar 11 21:42:37 smithi092 ceph-00db0016-dfed-11ee-95c8-87774f69a715-mgr-y[29801]: ::ffff:172.21.15.97 - - [11/Mar/2024:21:42:37] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-11T21:42:38.246 DEBUG:teuthology.orchestra.run.smithi097:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid 00db0016-dfed-11ee-95c8-87774f69a715 -- ceph daemon osd.5 dump_blocked_ops 2024-03-11T21:42:38.251 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:42:38.317 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:42:38.318 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:42:38.318 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:42:38.419 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:42:38.454 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:42:38.455 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:42:38.455 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:42:38.556 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:42:38.593 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:42:38.593 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:42:38.594 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:42:38.601 INFO:teuthology.orchestra.run.smithi097.stderr:Inferring config /var/lib/ceph/00db0016-dfed-11ee-95c8-87774f69a715/mon.b/config 2024-03-11T21:42:38.694 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:42:38.728 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:42:38.729 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:42:38.729 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:42:38.731 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:42:38 smithi097 ceph-mon[33317]: pgmap v804: 158 pgs: 1 active+recovery_wait, 1 creating+peering, 152 active+clean, 4 active+recovering+undersized+remapped; 6.2 GiB data, 24 GiB used, 691 GiB / 715 GiB avail; 1.0 KiB/s rd, 0 B/s wr, 523 op/s; 18204/305811 objects misplaced (5.953%); 9.5 MiB/s, 152 objects/s recovering 2024-03-11T21:42:38.731 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:42:38 smithi097 ceph-mon[33317]: from='mgr.14607 v1:172.21.15.92:0/1702717597' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T21:42:38.731 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:42:38 smithi097 ceph-mon[33317]: Health check update: Degraded data redundancy: 4 pgs undersized (PG_DEGRADED) 2024-03-11T21:42:38.830 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:42:38.865 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:42:38.866 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:42:38.866 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:42:38.967 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:42:39.007 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:42:39.007 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:42:39.008 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:42:39.023 INFO:journalctl@ceph.grafana.a.smithi097.stdout:Mar 11 21:42:38 smithi097 ceph-00db0016-dfed-11ee-95c8-87774f69a715-grafana-a[65126]: logger=sqlstore.transactions t=2024-03-11T21:42:38.882323086Z level=info msg="Database locked, sleeping then retrying" error="database is locked" retry=0 code="database is locked" 2024-03-11T21:42:39.109 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:42:39.117 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:42:38 smithi092 ceph-mon[29572]: pgmap v804: 158 pgs: 1 active+recovery_wait, 1 creating+peering, 152 active+clean, 4 active+recovering+undersized+remapped; 6.2 GiB data, 24 GiB used, 691 GiB / 715 GiB avail; 1.0 KiB/s rd, 0 B/s wr, 523 op/s; 18204/305811 objects misplaced (5.953%); 9.5 MiB/s, 152 objects/s recovering 2024-03-11T21:42:39.117 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:42:38 smithi092 ceph-mon[29572]: from='mgr.14607 v1:172.21.15.92:0/1702717597' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T21:42:39.117 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:42:38 smithi092 ceph-mon[29572]: Health check update: Degraded data redundancy: 4 pgs undersized (PG_DEGRADED) 2024-03-11T21:42:39.117 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:42:38 smithi092 ceph-mon[37563]: pgmap v804: 158 pgs: 1 active+recovery_wait, 1 creating+peering, 152 active+clean, 4 active+recovering+undersized+remapped; 6.2 GiB data, 24 GiB used, 691 GiB / 715 GiB avail; 1.0 KiB/s rd, 0 B/s wr, 523 op/s; 18204/305811 objects misplaced (5.953%); 9.5 MiB/s, 152 objects/s recovering 2024-03-11T21:42:39.117 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:42:38 smithi092 ceph-mon[37563]: from='mgr.14607 v1:172.21.15.92:0/1702717597' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T21:42:39.118 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:42:38 smithi092 ceph-mon[37563]: Health check update: Degraded data redundancy: 4 pgs undersized (PG_DEGRADED) 2024-03-11T21:42:39.150 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:42:39.151 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:42:39.151 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:42:39.252 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:42:39.288 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:42:39.289 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:42:39.289 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:42:39.390 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:42:39.428 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:42:39.429 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:42:39.429 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:42:39.530 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:42:39.561 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:42:39.562 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:42:39.562 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:42:39.663 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:42:39.695 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:42:39.696 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:42:39.696 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:42:39.797 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:42:39.839 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:42:39.840 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:42:39.840 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:42:39.941 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:42:39.972 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:42:39.973 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:42:39.973 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:42:40.074 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:42:40.123 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:42:40.124 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:42:40.124 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:42:40.225 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:42:40.274 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:42:40.275 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:42:40.275 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:42:40.376 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:42:40.405 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:42:40.405 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:42:40.406 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:42:40.507 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:42:40.538 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:42:40.539 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:42:40.539 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:42:40.640 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:42:40.673 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:42:40.673 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:42:40.673 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:42:40.774 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:42:40.805 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:42:40.806 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:42:40.807 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:42:40.831 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:42:40 smithi097 ceph-mon[33317]: pgmap v805: 158 pgs: 1 active+recovery_wait, 153 active+clean, 4 active+recovering+undersized+remapped; 6.1 GiB data, 24 GiB used, 691 GiB / 715 GiB avail; 1.7 KiB/s rd, 0 B/s wr, 720 op/s; 17145/298155 objects misplaced (5.750%); 7.7 MiB/s, 123 objects/s recovering 2024-03-11T21:42:40.908 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:42:40.953 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:42:40.953 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:42:40.953 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:42:40.988 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:42:40 smithi092 ceph-mon[29572]: pgmap v805: 158 pgs: 1 active+recovery_wait, 153 active+clean, 4 active+recovering+undersized+remapped; 6.1 GiB data, 24 GiB used, 691 GiB / 715 GiB avail; 1.7 KiB/s rd, 0 B/s wr, 720 op/s; 17145/298155 objects misplaced (5.750%); 7.7 MiB/s, 123 objects/s recovering 2024-03-11T21:42:40.988 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:42:40 smithi092 ceph-mon[37563]: pgmap v805: 158 pgs: 1 active+recovery_wait, 153 active+clean, 4 active+recovering+undersized+remapped; 6.1 GiB data, 24 GiB used, 691 GiB / 715 GiB avail; 1.7 KiB/s rd, 0 B/s wr, 720 op/s; 17145/298155 objects misplaced (5.750%); 7.7 MiB/s, 123 objects/s recovering 2024-03-11T21:42:41.054 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:42:41.098 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:42:41.098 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:42:41.099 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:42:41.200 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:42:41.226 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:42:41.226 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:42:41.227 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:42:41.325 INFO:teuthology.orchestra.run.smithi097.stdout:{ 2024-03-11T21:42:41.325 INFO:teuthology.orchestra.run.smithi097.stdout: "ops": [], 2024-03-11T21:42:41.325 INFO:teuthology.orchestra.run.smithi097.stdout: "complaint_time": 30, 2024-03-11T21:42:41.325 INFO:teuthology.orchestra.run.smithi097.stdout: "num_blocked_ops": 0 2024-03-11T21:42:41.325 INFO:teuthology.orchestra.run.smithi097.stdout:} 2024-03-11T21:42:41.328 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:42:41.372 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:42:41.373 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:42:41.373 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:42:41.474 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:42:41.507 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:42:41.508 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:42:41.508 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:42:41.609 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:42:41.648 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:42:41.649 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:42:41.649 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:42:41.750 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:42:41.788 INFO:teuthology.orchestra.run.smithi092.stdout:{"pg_ready":true,"pg_map":{"version":794,"stamp":"2024-03-11T21:42:27.530019+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":7080319424,"num_objects":108229,"num_object_clones":0,"num_object_copies":324687,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":8894,"num_objects_misplaced":13218,"num_objects_unfound":0,"num_objects_dirty":108229,"num_whiteouts":0,"num_read":2332,"num_read_kb":2180,"num_write":90450,"num_write_kb":5626398,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13564,"num_bytes_recovered":878582214,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":146808,"ondisk_log_size":146808,"up":474,"acting":469,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":492,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":749961216,"kb_used":26690708,"kb_used_data":22262760,"kb_used_omap":15,"kb_used_meta":4427824,"kb_avail":723270508,"statfs":{"total":767960285184,"available":740629000192,"internally_reserved":0,"allocated":22797066240,"data_stored":22787683684,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15601,"internal_metadata":4534092559},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,4,2,0,0,0,3,0,10,1],"upper_bound":2048},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":138149912,"num_objects":2109,"num_object_clones":0,"num_object_copies":6327,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-4424,"num_objects_misplaced":1866,"num_objects_unfound":0,"num_objects_dirty":2109,"num_whiteouts":0,"num_read":6,"num_read_kb":6,"num_write":5453,"num_write_kb":241921,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1199,"num_bytes_recovered":78577664,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.061202"},"pg_stats":[{"pgid":"10.17","version":"394'5721","reported_seq":10281,"reported_epoch":394,"state":"active+clean","last_fresh":"2024-03-11T21:42:26.971483+0000","last_change":"2024-03-11T21:42:08.480192+0000","last_active":"2024-03-11T21:42:26.971483+0000","last_peered":"2024-03-11T21:42:26.971483+0000","last_clean":"2024-03-11T21:42:26.971483+0000","last_became_active":"2024-03-11T21:42:08.476625+0000","last_became_peered":"2024-03-11T21:42:08.476625+0000","last_unstale":"2024-03-11T21:42:26.971483+0000","last_undegraded":"2024-03-11T21:42:26.971483+0000","last_fullsized":"2024-03-11T21:42:26.971483+0000","mapping_epoch":388,"log_start":"0'0","ondisk_log_start":"0'0","created":372,"last_epoch_clean":389,"parent":"0.0","parent_split_bits":5,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:40:38.219602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:40:38.219602+0000","last_clean_scrub_stamp":"2024-03-11T21:40:38.219602+0000","objects_scrubbed":0,"log_size":5721,"log_dups_size":0,"ondisk_log_size":5721,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":221118464,"num_objects":3374,"num_object_clones":0,"num_object_copies":10122,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3374,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3536,"num_write_kb":221120,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2228,"num_bytes_recovered":146014208,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1b","version":"0'0","reported_seq":635,"reported_epoch":392,"state":"active+clean","last_fresh":"2024-03-11T21:42:21.480903+0000","last_change":"2024-03-11T21:36:35.074931+0000","last_active":"2024-03-11T21:42:21.480903+0000","last_peered":"2024-03-11T21:42:21.480903+0000","last_clean":"2024-03-11T21:42:21.480903+0000","last_became_active":"2024-03-11T21:36:35.072636+0000","last_became_peered":"2024-03-11T21:36:35.072636+0000","last_unstale":"2024-03-11T21:42:21.480903+0000","last_undegraded":"2024-03-11T21:42:21.480903+0000","last_fullsized":"2024-03-11T21:42:21.480903+0000","mapping_epoch":277,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":278,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:35:07.021520+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:43.559452+0000","last_clean_scrub_stamp":"2024-03-11T21:35:07.021520+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":711,"reported_epoch":394,"state":"active+clean","last_fresh":"2024-03-11T21:42:25.536371+0000","last_change":"2024-03-11T21:41:19.522757+0000","last_active":"2024-03-11T21:42:25.536371+0000","last_peered":"2024-03-11T21:42:25.536371+0000","last_clean":"2024-03-11T21:42:25.536371+0000","last_became_active":"2024-03-11T21:25:42.578893+0000","last_became_peered":"2024-03-11T21:25:42.578893+0000","last_unstale":"2024-03-11T21:42:25.536371+0000","last_undegraded":"2024-03-11T21:42:25.536371+0000","last_fullsized":"2024-03-11T21:42:25.536371+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:41:19.522682+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:41.544550+0000","last_clean_scrub_stamp":"2024-03-11T21:41:19.522682+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:42:31.402864+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1f","version":"0'0","reported_seq":729,"reported_epoch":392,"state":"active+clean","last_fresh":"2024-03-11T21:42:21.478852+0000","last_change":"2024-03-11T21:36:00.843838+0000","last_active":"2024-03-11T21:42:21.478852+0000","last_peered":"2024-03-11T21:42:21.478852+0000","last_clean":"2024-03-11T21:42:21.478852+0000","last_became_active":"2024-03-11T21:34:41.144945+0000","last_became_peered":"2024-03-11T21:34:41.144945+0000","last_unstale":"2024-03-11T21:42:21.478852+0000","last_undegraded":"2024-03-11T21:42:21.478852+0000","last_fullsized":"2024-03-11T21:42:21.478852+0000","mapping_epoch":226,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":227,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:36:00.843718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:37.512018+0000","last_clean_scrub_stamp":"2024-03-11T21:36:00.843718+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,4],"acting":[0,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.19","version":"260'27","reported_seq":682,"reported_epoch":392,"state":"active+clean","last_fresh":"2024-03-11T21:42:21.491468+0000","last_change":"2024-03-11T21:36:35.089901+0000","last_active":"2024-03-11T21:42:21.491468+0000","last_peered":"2024-03-11T21:42:21.491468+0000","last_clean":"2024-03-11T21:42:21.491468+0000","last_became_active":"2024-03-11T21:36:35.067652+0000","last_became_peered":"2024-03-11T21:36:35.067652+0000","last_unstale":"2024-03-11T21:42:21.491468+0000","last_undegraded":"2024-03-11T21:42:21.491468+0000","last_fullsized":"2024-03-11T21:42:21.491468+0000","mapping_epoch":277,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":278,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'15","last_scrub_stamp":"2024-03-11T21:35:22.099374+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:39.525554+0000","last_clean_scrub_stamp":"2024-03-11T21:35:22.099374+0000","objects_scrubbed":0,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":42,"num_read_kb":33,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.16","version":"394'5618","reported_seq":12242,"reported_epoch":394,"state":"active+clean","last_fresh":"2024-03-11T21:42:27.360314+0000","last_change":"2024-03-11T21:42:25.519869+0000","last_active":"2024-03-11T21:42:27.360314+0000","last_peered":"2024-03-11T21:42:27.360314+0000","last_clean":"2024-03-11T21:42:27.360314+0000","last_became_active":"2024-03-11T21:42:25.516791+0000","last_became_peered":"2024-03-11T21:42:25.516791+0000","last_unstale":"2024-03-11T21:42:27.360314+0000","last_undegraded":"2024-03-11T21:42:27.360314+0000","last_fullsized":"2024-03-11T21:42:27.360314+0000","mapping_epoch":393,"log_start":"0'0","ondisk_log_start":"0'0","created":372,"last_epoch_clean":394,"parent":"0.0","parent_split_bits":5,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:40:38.219602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:40:38.219602+0000","last_clean_scrub_stamp":"2024-03-11T21:40:38.219602+0000","objects_scrubbed":0,"log_size":5618,"log_dups_size":0,"ondisk_log_size":5618,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:42:30.769436+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":216104960,"num_objects":3298,"num_object_clones":0,"num_object_copies":9894,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3298,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3466,"num_write_kb":216416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2173,"num_bytes_recovered":142409728,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,1],"acting":[6,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.1a","version":"0'0","reported_seq":699,"reported_epoch":394,"state":"active+clean","last_fresh":"2024-03-11T21:42:25.536365+0000","last_change":"2024-03-11T21:41:02.614569+0000","last_active":"2024-03-11T21:42:25.536365+0000","last_peered":"2024-03-11T21:42:25.536365+0000","last_clean":"2024-03-11T21:42:25.536365+0000","last_became_active":"2024-03-11T21:25:44.590923+0000","last_became_peered":"2024-03-11T21:25:44.590923+0000","last_unstale":"2024-03-11T21:42:25.536365+0000","last_undegraded":"2024-03-11T21:42:25.536365+0000","last_fullsized":"2024-03-11T21:42:25.536365+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:41:02.614438+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:43.559452+0000","last_clean_scrub_stamp":"2024-03-11T21:41:02.614438+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":38,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,1],"acting":[4,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":711,"reported_epoch":392,"state":"active+clean","last_fresh":"2024-03-11T21:42:21.473895+0000","last_change":"2024-03-11T21:36:15.576138+0000","last_active":"2024-03-11T21:42:21.473895+0000","last_peered":"2024-03-11T21:42:21.473895+0000","last_clean":"2024-03-11T21:42:21.473895+0000","last_became_active":"2024-03-11T21:34:41.154216+0000","last_became_peered":"2024-03-11T21:34:41.154216+0000","last_unstale":"2024-03-11T21:42:21.473895+0000","last_undegraded":"2024-03-11T21:42:21.473895+0000","last_fullsized":"2024-03-11T21:42:21.473895+0000","mapping_epoch":226,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":227,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:36:15.575887+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:41.544550+0000","last_clean_scrub_stamp":"2024-03-11T21:36:15.575887+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,7],"acting":[1,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1e","version":"0'0","reported_seq":656,"reported_epoch":392,"state":"active+clean","last_fresh":"2024-03-11T21:42:21.486335+0000","last_change":"2024-03-11T21:41:04.598202+0000","last_active":"2024-03-11T21:42:21.486335+0000","last_peered":"2024-03-11T21:42:21.486335+0000","last_clean":"2024-03-11T21:42:21.486335+0000","last_became_active":"2024-03-11T21:36:35.061786+0000","last_became_peered":"2024-03-11T21:36:35.061786+0000","last_unstale":"2024-03-11T21:42:21.486335+0000","last_undegraded":"2024-03-11T21:42:21.486335+0000","last_fullsized":"2024-03-11T21:42:21.486335+0000","mapping_epoch":277,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":278,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:41:04.597983+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:37.512018+0000","last_clean_scrub_stamp":"2024-03-11T21:41:04.597983+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.18","version":"260'15","reported_seq":748,"reported_epoch":394,"state":"active+clean","last_fresh":"2024-03-11T21:42:25.535972+0000","last_change":"2024-03-11T21:41:17.537672+0000","last_active":"2024-03-11T21:42:25.535972+0000","last_peered":"2024-03-11T21:42:25.535972+0000","last_clean":"2024-03-11T21:42:25.535972+0000","last_became_active":"2024-03-11T21:25:40.552216+0000","last_became_peered":"2024-03-11T21:25:40.552216+0000","last_unstale":"2024-03-11T21:42:25.535972+0000","last_undegraded":"2024-03-11T21:42:25.535972+0000","last_fullsized":"2024-03-11T21:42:25.535972+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"260'15","last_scrub_stamp":"2024-03-11T21:41:17.537536+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:39.525554+0000","last_clean_scrub_stamp":"2024-03-11T21:41:17.537536+0000","objects_scrubbed":4,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:42:40.297307+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"10.15","version":"392'5668","reported_seq":12745,"reported_epoch":392,"state":"active+recovery_wait","last_fresh":"2024-03-11T21:42:24.141576+0000","last_change":"2024-03-11T21:42:21.536310+0000","last_active":"2024-03-11T21:42:24.141576+0000","last_peered":"2024-03-11T21:42:24.141576+0000","last_clean":"2024-03-11T21:41:29.797892+0000","last_became_active":"2024-03-11T21:42:21.511969+0000","last_became_peered":"2024-03-11T21:42:21.511969+0000","last_unstale":"2024-03-11T21:42:24.141576+0000","last_undegraded":"2024-03-11T21:42:24.141576+0000","last_fullsized":"2024-03-11T21:42:24.141576+0000","mapping_epoch":391,"log_start":"0'0","ondisk_log_start":"0'0","created":372,"last_epoch_clean":373,"parent":"0.0","parent_split_bits":5,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:40:38.219602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:40:38.219602+0000","last_clean_scrub_stamp":"2024-03-11T21:40:38.219602+0000","objects_scrubbed":0,"log_size":5668,"log_dups_size":0,"ondisk_log_size":5668,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":224395264,"num_objects":3424,"num_object_clones":0,"num_object_copies":10272,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3424,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3440,"num_write_kb":219648,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2327,"num_bytes_recovered":152502272,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,1],"acting":[3,5,1],"avail_no_missing":["3","0","1","2","5"],"object_location_counts":[{"shards":"1,3,5","objects":3424}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":716,"reported_epoch":394,"state":"active+clean","last_fresh":"2024-03-11T21:42:25.502148+0000","last_change":"2024-03-11T21:41:13.893454+0000","last_active":"2024-03-11T21:42:25.502148+0000","last_peered":"2024-03-11T21:42:25.502148+0000","last_clean":"2024-03-11T21:42:25.502148+0000","last_became_active":"2024-03-11T21:36:35.068453+0000","last_became_peered":"2024-03-11T21:36:35.068453+0000","last_unstale":"2024-03-11T21:42:25.502148+0000","last_undegraded":"2024-03-11T21:42:25.502148+0000","last_fullsized":"2024-03-11T21:42:25.502148+0000","mapping_epoch":277,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":278,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:41:13.893233+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:32:12.654573+0000","last_clean_scrub_stamp":"2024-03-11T21:41:13.893233+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":78,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3],"acting":[5,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":364,"reported_epoch":394,"state":"active+clean","last_fresh":"2024-03-11T21:42:25.533038+0000","last_change":"2024-03-11T21:41:01.164718+0000","last_active":"2024-03-11T21:42:25.533038+0000","last_peered":"2024-03-11T21:42:25.533038+0000","last_clean":"2024-03-11T21:42:25.533038+0000","last_became_active":"2024-03-11T21:34:41.145299+0000","last_became_peered":"2024-03-11T21:34:41.145299+0000","last_unstale":"2024-03-11T21:42:25.533038+0000","last_undegraded":"2024-03-11T21:42:25.533038+0000","last_fullsized":"2024-03-11T21:42:25.533038+0000","mapping_epoch":226,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":227,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:41:01.164619+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:41.544550+0000","last_clean_scrub_stamp":"2024-03-11T21:41:01.164619+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:42:28.198410+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.1d","version":"0'0","reported_seq":726,"reported_epoch":394,"state":"active+clean","last_fresh":"2024-03-11T21:42:25.535030+0000","last_change":"2024-03-11T21:41:02.217938+0000","last_active":"2024-03-11T21:42:25.535030+0000","last_peered":"2024-03-11T21:42:25.535030+0000","last_clean":"2024-03-11T21:42:25.535030+0000","last_became_active":"2024-03-11T21:25:38.542628+0000","last_became_peered":"2024-03-11T21:25:38.542628+0000","last_unstale":"2024-03-11T21:42:25.535030+0000","last_undegraded":"2024-03-11T21:42:25.535030+0000","last_fullsized":"2024-03-11T21:42:25.535030+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:41:02.217799+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:37.512018+0000","last_clean_scrub_stamp":"2024-03-11T21:41:02.217799+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:42:28.198378+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,0],"acting":[7,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.1b","version":"260'7","reported_seq":734,"reported_epoch":394,"state":"active+clean","last_fresh":"2024-03-11T21:42:25.537908+0000","last_change":"2024-03-11T21:40:29.597326+0000","last_active":"2024-03-11T21:42:25.537908+0000","last_peered":"2024-03-11T21:42:25.537908+0000","last_clean":"2024-03-11T21:42:25.537908+0000","last_became_active":"2024-03-11T21:36:35.057678+0000","last_became_peered":"2024-03-11T21:36:35.057678+0000","last_unstale":"2024-03-11T21:42:25.537908+0000","last_undegraded":"2024-03-11T21:42:25.537908+0000","last_fullsized":"2024-03-11T21:42:25.537908+0000","mapping_epoch":277,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":278,"parent":"0.0","parent_split_bits":0,"last_scrub":"260'7","last_scrub_stamp":"2024-03-11T21:40:29.597219+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:39.525554+0000","last_clean_scrub_stamp":"2024-03-11T21:40:29.597219+0000","objects_scrubbed":2,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":14,"num_read_kb":10,"num_write":8,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,1],"acting":[4,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"10.14","version":"392'5498","reported_seq":10727,"reported_epoch":392,"state":"active+recovering+undersized+remapped","last_fresh":"2024-03-11T21:42:24.268780+0000","last_change":"2024-03-11T21:41:32.225109+0000","last_active":"2024-03-11T21:42:24.268780+0000","last_peered":"2024-03-11T21:42:24.268780+0000","last_clean":"2024-03-11T21:41:29.817200+0000","last_became_active":"2024-03-11T21:41:32.164233+0000","last_became_peered":"2024-03-11T21:41:32.164233+0000","last_unstale":"2024-03-11T21:42:24.268780+0000","last_undegraded":"2024-03-11T21:42:24.268780+0000","last_fullsized":"2024-03-11T21:41:31.832797+0000","mapping_epoch":375,"log_start":"0'0","ondisk_log_start":"0'0","created":372,"last_epoch_clean":373,"parent":"0.0","parent_split_bits":5,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:40:38.219602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:40:38.219602+0000","last_clean_scrub_stamp":"2024-03-11T21:40:38.219602+0000","objects_scrubbed":0,"log_size":5498,"log_dups_size":0,"ondisk_log_size":5498,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":221151232,"num_objects":3375,"num_object_clones":0,"num_object_copies":10125,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":986,"num_objects_unfound":0,"num_objects_dirty":3375,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3397,"num_write_kb":216672,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1712,"num_bytes_recovered":112197632,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,4],"acting":[0,4],"avail_no_missing":["0","3","4"],"object_location_counts":[{"shards":"0,3,4","objects":493},{"shards":"0,4","objects":2882}],"blocked_by":[],"up_primary":1,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":695,"reported_epoch":392,"state":"active+clean","last_fresh":"2024-03-11T21:42:21.479976+0000","last_change":"2024-03-11T21:35:39.691825+0000","last_active":"2024-03-11T21:42:21.479976+0000","last_peered":"2024-03-11T21:42:21.479976+0000","last_clean":"2024-03-11T21:42:21.479976+0000","last_became_active":"2024-03-11T21:25:44.589786+0000","last_became_peered":"2024-03-11T21:25:44.589786+0000","last_unstale":"2024-03-11T21:42:21.479976+0000","last_undegraded":"2024-03-11T21:42:21.479976+0000","last_fullsized":"2024-03-11T21:42:21.479976+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:35:39.691690+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:43.559452+0000","last_clean_scrub_stamp":"2024-03-11T21:35:39.691690+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":18,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,7],"acting":[0,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":722,"reported_epoch":394,"state":"active+clean","last_fresh":"2024-03-11T21:42:25.502473+0000","last_change":"2024-03-11T21:40:27.604550+0000","last_active":"2024-03-11T21:42:25.502473+0000","last_peered":"2024-03-11T21:42:25.502473+0000","last_clean":"2024-03-11T21:42:25.502473+0000","last_became_active":"2024-03-11T21:34:41.147185+0000","last_became_peered":"2024-03-11T21:34:41.147185+0000","last_unstale":"2024-03-11T21:42:25.502473+0000","last_undegraded":"2024-03-11T21:42:25.502473+0000","last_fullsized":"2024-03-11T21:42:25.502473+0000","mapping_epoch":226,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":227,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:40:27.604453+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:32:21.535034+0000","last_clean_scrub_stamp":"2024-03-11T21:40:27.604453+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,7],"acting":[5,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1c","version":"0'0","reported_seq":726,"reported_epoch":394,"state":"active+clean","last_fresh":"2024-03-11T21:42:25.534274+0000","last_change":"2024-03-11T21:41:14.385957+0000","last_active":"2024-03-11T21:42:25.534274+0000","last_peered":"2024-03-11T21:42:25.534274+0000","last_clean":"2024-03-11T21:42:25.534274+0000","last_became_active":"2024-03-11T21:25:38.541364+0000","last_became_peered":"2024-03-11T21:25:38.541364+0000","last_unstale":"2024-03-11T21:42:25.534274+0000","last_undegraded":"2024-03-11T21:42:25.534274+0000","last_fullsized":"2024-03-11T21:42:25.534274+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:41:14.385682+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:28:32.566437+0000","last_clean_scrub_stamp":"2024-03-11T21:41:14.385682+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":27,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.1a","version":"350'19","reported_seq":762,"reported_epoch":394,"state":"active+clean","last_fresh":"2024-03-11T21:42:25.536450+0000","last_change":"2024-03-11T21:40:20.484449+0000","last_active":"2024-03-11T21:42:25.536450+0000","last_peered":"2024-03-11T21:42:25.536450+0000","last_clean":"2024-03-11T21:42:25.536450+0000","last_became_active":"2024-03-11T21:36:35.057029+0000","last_became_peered":"2024-03-11T21:36:35.057029+0000","last_unstale":"2024-03-11T21:42:25.536450+0000","last_undegraded":"2024-03-11T21:42:25.536450+0000","last_fullsized":"2024-03-11T21:42:25.536450+0000","mapping_epoch":277,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":278,"parent":"0.0","parent_split_bits":0,"last_scrub":"350'19","last_scrub_stamp":"2024-03-11T21:40:20.484308+0000","last_deep_scrub":"350'19","last_deep_scrub_stamp":"2024-03-11T21:40:20.484308+0000","last_clean_scrub_stamp":"2024-03-11T21:40:20.484308+0000","objects_scrubbed":4,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0],"acting":[4,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"10.13","version":"392'5555","reported_seq":5649,"reported_epoch":392,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-03-11T21:42:22.493838+0000","last_change":"2024-03-11T21:41:32.316094+0000","last_active":"2024-03-11T21:42:22.493838+0000","last_peered":"2024-03-11T21:42:22.493838+0000","last_clean":"2024-03-11T21:41:29.694388+0000","last_became_active":"2024-03-11T21:41:32.300093+0000","last_became_peered":"2024-03-11T21:41:32.300093+0000","last_unstale":"2024-03-11T21:42:22.493838+0000","last_undegraded":"2024-03-11T21:41:32.160494+0000","last_fullsized":"2024-03-11T21:41:31.840140+0000","mapping_epoch":375,"log_start":"0'0","ondisk_log_start":"0'0","created":372,"last_epoch_clean":373,"parent":"0.0","parent_split_bits":5,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:40:38.219602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:40:38.219602+0000","last_clean_scrub_stamp":"2024-03-11T21:40:38.219602+0000","objects_scrubbed":0,"log_size":5555,"log_dups_size":0,"ondisk_log_size":5555,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":222560256,"num_objects":3396,"num_object_clones":0,"num_object_copies":10188,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4374,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3396,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3396,"num_write_kb":217344,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":65536,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,2],"acting":[2,1],"avail_no_missing":["2","3","5"],"object_location_counts":[{"shards":"1,2","objects":1209},{"shards":"2,3,5","objects":2187}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.1f","version":"0'0","reported_seq":579,"reported_epoch":392,"state":"active+clean","last_fresh":"2024-03-11T21:42:21.492233+0000","last_change":"2024-03-11T21:41:08.652958+0000","last_active":"2024-03-11T21:42:21.492233+0000","last_peered":"2024-03-11T21:42:21.492233+0000","last_clean":"2024-03-11T21:42:21.492233+0000","last_became_active":"2024-03-11T21:36:35.071755+0000","last_became_peered":"2024-03-11T21:36:35.071755+0000","last_unstale":"2024-03-11T21:42:21.492233+0000","last_undegraded":"2024-03-11T21:42:21.492233+0000","last_fullsized":"2024-03-11T21:42:21.492233+0000","mapping_epoch":277,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":278,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:41:08.652836+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:43.559452+0000","last_clean_scrub_stamp":"2024-03-11T21:41:08.652836+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:42:25.863145+0000","scrub_duration":137,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1c","version":"0'0","reported_seq":712,"reported_epoch":394,"state":"active+clean","last_fresh":"2024-03-11T21:42:25.537425+0000","last_change":"2024-03-11T21:40:07.678642+0000","last_active":"2024-03-11T21:42:25.537425+0000","last_peered":"2024-03-11T21:42:25.537425+0000","last_clean":"2024-03-11T21:42:25.537425+0000","last_became_active":"2024-03-11T21:36:35.067127+0000","last_became_peered":"2024-03-11T21:36:35.067127+0000","last_unstale":"2024-03-11T21:42:25.537425+0000","last_undegraded":"2024-03-11T21:42:25.537425+0000","last_fullsized":"2024-03-11T21:42:25.537425+0000","mapping_epoch":277,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":278,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:40:07.678561+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:40:07.678561+0000","last_clean_scrub_stamp":"2024-03-11T21:40:07.678561+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1b","version":"0'0","reported_seq":628,"reported_epoch":392,"state":"active+clean","last_fresh":"2024-03-11T21:42:21.481076+0000","last_change":"2024-03-11T21:41:23.590257+0000","last_active":"2024-03-11T21:42:21.481076+0000","last_peered":"2024-03-11T21:42:21.481076+0000","last_clean":"2024-03-11T21:42:21.481076+0000","last_became_active":"2024-03-11T21:36:35.073959+0000","last_became_peered":"2024-03-11T21:36:35.073959+0000","last_unstale":"2024-03-11T21:42:21.481076+0000","last_undegraded":"2024-03-11T21:42:21.481076+0000","last_fullsized":"2024-03-11T21:42:21.481076+0000","mapping_epoch":277,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":278,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:41:23.589546+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:34:27.998724+0000","last_clean_scrub_stamp":"2024-03-11T21:41:23.589546+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:42:52.777394+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1d","version":"260'18","reported_seq":664,"reported_epoch":392,"state":"active+clean","last_fresh":"2024-03-11T21:42:21.492580+0000","last_change":"2024-03-11T21:36:35.080207+0000","last_active":"2024-03-11T21:42:21.492580+0000","last_peered":"2024-03-11T21:42:21.492580+0000","last_clean":"2024-03-11T21:42:21.492580+0000","last_became_active":"2024-03-11T21:36:35.055880+0000","last_became_peered":"2024-03-11T21:36:35.055880+0000","last_unstale":"2024-03-11T21:42:21.492580+0000","last_undegraded":"2024-03-11T21:42:21.492580+0000","last_fullsized":"2024-03-11T21:42:21.492580+0000","mapping_epoch":277,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":278,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'12","last_scrub_stamp":"2024-03-11T21:35:07.971581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:39.525554+0000","last_clean_scrub_stamp":"2024-03-11T21:35:07.971581+0000","objects_scrubbed":0,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":33,"num_read_kb":24,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.12","version":"392'5694","reported_seq":7334,"reported_epoch":392,"state":"active+recovering+undersized+remapped","last_fresh":"2024-03-11T21:42:24.266998+0000","last_change":"2024-03-11T21:42:08.298106+0000","last_active":"2024-03-11T21:42:24.266998+0000","last_peered":"2024-03-11T21:42:24.266998+0000","last_clean":"2024-03-11T21:41:29.821793+0000","last_became_active":"2024-03-11T21:41:32.262303+0000","last_became_peered":"2024-03-11T21:41:32.262303+0000","last_unstale":"2024-03-11T21:42:24.266998+0000","last_undegraded":"2024-03-11T21:42:24.266998+0000","last_fullsized":"2024-03-11T21:41:31.832178+0000","mapping_epoch":375,"log_start":"0'0","ondisk_log_start":"0'0","created":372,"last_epoch_clean":373,"parent":"0.0","parent_split_bits":5,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:40:38.219602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:40:38.219602+0000","last_clean_scrub_stamp":"2024-03-11T21:40:38.219602+0000","objects_scrubbed":0,"log_size":5694,"log_dups_size":0,"ondisk_log_size":5694,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":227016704,"num_objects":3464,"num_object_clones":0,"num_object_copies":10392,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":5766,"num_objects_unfound":0,"num_objects_dirty":3464,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3490,"num_write_kb":222528,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":387,"num_bytes_recovered":25362432,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,5],"acting":[0,2],"avail_no_missing":["0","1","4"],"object_location_counts":[{"shards":"0,1,4","objects":1922},{"shards":"0,2","objects":1542}],"blocked_by":[],"up_primary":6,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.1e","version":"0'0","reported_seq":701,"reported_epoch":394,"state":"active+clean","last_fresh":"2024-03-11T21:42:25.538051+0000","last_change":"2024-03-11T21:41:10.551185+0000","last_active":"2024-03-11T21:42:25.538051+0000","last_peered":"2024-03-11T21:42:25.538051+0000","last_clean":"2024-03-11T21:42:25.538051+0000","last_became_active":"2024-03-11T21:25:45.300355+0000","last_became_peered":"2024-03-11T21:25:45.300355+0000","last_unstale":"2024-03-11T21:42:25.538051+0000","last_undegraded":"2024-03-11T21:42:25.538051+0000","last_fullsized":"2024-03-11T21:42:25.538051+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:41:10.550887+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:43.559452+0000","last_clean_scrub_stamp":"2024-03-11T21:41:10.550887+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:42:31.211347+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5],"acting":[4,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1d","version":"0'0","reported_seq":699,"reported_epoch":392,"state":"active+clean","last_fresh":"2024-03-11T21:42:21.477168+0000","last_change":"2024-03-11T21:36:07.624196+0000","last_active":"2024-03-11T21:42:21.477168+0000","last_peered":"2024-03-11T21:42:21.477168+0000","last_clean":"2024-03-11T21:42:21.477168+0000","last_became_active":"2024-03-11T21:25:42.579407+0000","last_became_peered":"2024-03-11T21:25:42.579407+0000","last_unstale":"2024-03-11T21:42:21.477168+0000","last_undegraded":"2024-03-11T21:42:21.477168+0000","last_fullsized":"2024-03-11T21:42:21.477168+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:36:07.624075+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:30:18.678531+0000","last_clean_scrub_stamp":"2024-03-11T21:36:07.624075+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1a","version":"0'0","reported_seq":747,"reported_epoch":394,"state":"active+clean","last_fresh":"2024-03-11T21:42:25.504629+0000","last_change":"2024-03-11T21:41:31.856350+0000","last_active":"2024-03-11T21:42:25.504629+0000","last_peered":"2024-03-11T21:42:25.504629+0000","last_clean":"2024-03-11T21:42:25.504629+0000","last_became_active":"2024-03-11T21:34:41.145014+0000","last_became_peered":"2024-03-11T21:34:41.145014+0000","last_unstale":"2024-03-11T21:42:25.504629+0000","last_undegraded":"2024-03-11T21:42:25.504629+0000","last_fullsized":"2024-03-11T21:42:25.504629+0000","mapping_epoch":226,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":227,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:41:31.856243+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:37.512018+0000","last_clean_scrub_stamp":"2024-03-11T21:41:31.856243+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:42:40.323335+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.1c","version":"260'27","reported_seq":768,"reported_epoch":392,"state":"active+clean","last_fresh":"2024-03-11T21:42:21.487210+0000","last_change":"2024-03-11T21:40:15.885360+0000","last_active":"2024-03-11T21:42:21.487210+0000","last_peered":"2024-03-11T21:42:21.487210+0000","last_clean":"2024-03-11T21:42:21.487210+0000","last_became_active":"2024-03-11T21:36:35.074579+0000","last_became_peered":"2024-03-11T21:36:35.074579+0000","last_unstale":"2024-03-11T21:42:21.487210+0000","last_undegraded":"2024-03-11T21:42:21.487210+0000","last_fullsized":"2024-03-11T21:42:21.487210+0000","mapping_epoch":277,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":278,"parent":"0.0","parent_split_bits":0,"last_scrub":"260'27","last_scrub_stamp":"2024-03-11T21:40:15.885265+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:39.525554+0000","last_clean_scrub_stamp":"2024-03-11T21:40:15.885265+0000","objects_scrubbed":7,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":44,"num_read_kb":35,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"10.11","version":"394'5700","reported_seq":10062,"reported_epoch":394,"state":"active+clean","last_fresh":"2024-03-11T21:42:26.855276+0000","last_change":"2024-03-11T21:42:09.434799+0000","last_active":"2024-03-11T21:42:26.855276+0000","last_peered":"2024-03-11T21:42:26.855276+0000","last_clean":"2024-03-11T21:42:26.855276+0000","last_became_active":"2024-03-11T21:42:09.430469+0000","last_became_peered":"2024-03-11T21:42:09.430469+0000","last_unstale":"2024-03-11T21:42:26.855276+0000","last_undegraded":"2024-03-11T21:42:26.855276+0000","last_fullsized":"2024-03-11T21:42:26.855276+0000","mapping_epoch":389,"log_start":"0'0","ondisk_log_start":"0'0","created":372,"last_epoch_clean":390,"parent":"0.0","parent_split_bits":5,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:40:38.219602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:40:38.219602+0000","last_clean_scrub_stamp":"2024-03-11T21:40:38.219602+0000","objects_scrubbed":0,"log_size":5700,"log_dups_size":0,"ondisk_log_size":5700,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:42:30.020642+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":222068736,"num_objects":3389,"num_object_clones":0,"num_object_copies":10167,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3389,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3513,"num_write_kb":220832,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2170,"num_bytes_recovered":142213120,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,7],"acting":[5,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.1d","version":"0'0","reported_seq":695,"reported_epoch":392,"state":"active+clean","last_fresh":"2024-03-11T21:42:21.476787+0000","last_change":"2024-03-11T21:41:16.533828+0000","last_active":"2024-03-11T21:42:21.476787+0000","last_peered":"2024-03-11T21:42:21.476787+0000","last_clean":"2024-03-11T21:42:21.476787+0000","last_became_active":"2024-03-11T21:25:44.588845+0000","last_became_peered":"2024-03-11T21:25:44.588845+0000","last_unstale":"2024-03-11T21:42:21.476787+0000","last_undegraded":"2024-03-11T21:42:21.476787+0000","last_fullsized":"2024-03-11T21:42:21.476787+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:41:16.533692+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:43.559452+0000","last_clean_scrub_stamp":"2024-03-11T21:41:16.533692+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:42:41.098465+0000","scrub_duration":207,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.1e","version":"0'0","reported_seq":709,"reported_epoch":392,"state":"active+clean","last_fresh":"2024-03-11T21:42:21.477914+0000","last_change":"2024-03-11T21:40:28.163223+0000","last_active":"2024-03-11T21:42:21.477914+0000","last_peered":"2024-03-11T21:42:21.477914+0000","last_clean":"2024-03-11T21:42:21.477914+0000","last_became_active":"2024-03-11T21:25:42.582959+0000","last_became_peered":"2024-03-11T21:25:42.582959+0000","last_unstale":"2024-03-11T21:42:21.477914+0000","last_undegraded":"2024-03-11T21:42:21.477914+0000","last_fullsized":"2024-03-11T21:42:21.477914+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:40:28.163142+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:26:58.335431+0000","last_clean_scrub_stamp":"2024-03-11T21:40:28.163142+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.19","version":"67'2","reported_seq":654,"reported_epoch":392,"state":"active+clean","last_fresh":"2024-03-11T21:42:21.481294+0000","last_change":"2024-03-11T21:41:01.612764+0000","last_active":"2024-03-11T21:42:21.481294+0000","last_peered":"2024-03-11T21:42:21.481294+0000","last_clean":"2024-03-11T21:42:21.481294+0000","last_became_active":"2024-03-11T21:36:35.064249+0000","last_became_peered":"2024-03-11T21:36:35.064249+0000","last_unstale":"2024-03-11T21:42:21.481294+0000","last_undegraded":"2024-03-11T21:42:21.481294+0000","last_fullsized":"2024-03-11T21:42:21.481294+0000","mapping_epoch":277,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":278,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'2","last_scrub_stamp":"2024-03-11T21:41:01.612632+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:37.512018+0000","last_clean_scrub_stamp":"2024-03-11T21:41:01.612632+0000","objects_scrubbed":2,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":957,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,0],"acting":[3,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1f","version":"260'24","reported_seq":787,"reported_epoch":394,"state":"active+clean","last_fresh":"2024-03-11T21:42:25.505571+0000","last_change":"2024-03-11T21:40:25.079945+0000","last_active":"2024-03-11T21:42:25.505571+0000","last_peered":"2024-03-11T21:42:25.505571+0000","last_clean":"2024-03-11T21:42:25.505571+0000","last_became_active":"2024-03-11T21:32:22.798242+0000","last_became_peered":"2024-03-11T21:32:22.798242+0000","last_unstale":"2024-03-11T21:42:25.505571+0000","last_undegraded":"2024-03-11T21:42:25.505571+0000","last_fullsized":"2024-03-11T21:42:25.505571+0000","mapping_epoch":166,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":167,"parent":"0.0","parent_split_bits":0,"last_scrub":"260'24","last_scrub_stamp":"2024-03-11T21:40:25.079808+0000","last_deep_scrub":"124'16","last_deep_scrub_stamp":"2024-03-11T21:35:38.173841+0000","last_clean_scrub_stamp":"2024-03-11T21:40:25.079808+0000","objects_scrubbed":5,"log_size":24,"log_dups_size":0,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":32,"num_read_kb":25,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,0],"acting":[6,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"10.10","version":"394'5651","reported_seq":10263,"reported_epoch":394,"state":"active+clean","last_fresh":"2024-03-11T21:42:26.912512+0000","last_change":"2024-03-11T21:41:47.080938+0000","last_active":"2024-03-11T21:42:26.912512+0000","last_peered":"2024-03-11T21:42:26.912512+0000","last_clean":"2024-03-11T21:42:26.912512+0000","last_became_active":"2024-03-11T21:41:47.077087+0000","last_became_peered":"2024-03-11T21:41:47.077087+0000","last_unstale":"2024-03-11T21:42:26.912512+0000","last_undegraded":"2024-03-11T21:42:26.912512+0000","last_fullsized":"2024-03-11T21:42:26.912512+0000","mapping_epoch":380,"log_start":"0'0","ondisk_log_start":"0'0","created":372,"last_epoch_clean":381,"parent":"0.0","parent_split_bits":5,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:40:38.219602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:40:38.219602+0000","last_clean_scrub_stamp":"2024-03-11T21:40:38.219602+0000","objects_scrubbed":0,"log_size":5651,"log_dups_size":0,"ondisk_log_size":5651,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":219021312,"num_objects":3342,"num_object_clones":0,"num_object_copies":10026,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3342,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3488,"num_write_kb":218560,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2254,"num_bytes_recovered":147718144,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,6],"acting":[4,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1c","version":"74'1","reported_seq":360,"reported_epoch":394,"state":"active+clean","last_fresh":"2024-03-11T21:42:25.533595+0000","last_change":"2024-03-11T21:36:11.732043+0000","last_active":"2024-03-11T21:42:25.533595+0000","last_peered":"2024-03-11T21:42:25.533595+0000","last_clean":"2024-03-11T21:42:25.533595+0000","last_became_active":"2024-03-11T21:34:41.144839+0000","last_became_peered":"2024-03-11T21:34:41.144839+0000","last_unstale":"2024-03-11T21:42:25.533595+0000","last_undegraded":"2024-03-11T21:42:25.533595+0000","last_fullsized":"2024-03-11T21:42:25.533595+0000","mapping_epoch":226,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":227,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'1","last_scrub_stamp":"2024-03-11T21:36:11.731914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:43.559452+0000","last_clean_scrub_stamp":"2024-03-11T21:36:11.731914+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:42:28.198289+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":370,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":370,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.1f","version":"229'10","reported_seq":970,"reported_epoch":394,"state":"active+clean","last_fresh":"2024-03-11T21:42:25.505531+0000","last_change":"2024-03-11T21:41:09.868735+0000","last_active":"2024-03-11T21:42:25.505531+0000","last_peered":"2024-03-11T21:42:25.505531+0000","last_clean":"2024-03-11T21:42:25.505531+0000","last_became_active":"2024-03-11T21:34:41.145862+0000","last_became_peered":"2024-03-11T21:34:41.145862+0000","last_unstale":"2024-03-11T21:42:25.505531+0000","last_undegraded":"2024-03-11T21:42:25.505531+0000","last_fullsized":"2024-03-11T21:42:25.505531+0000","mapping_epoch":226,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":227,"parent":"0.0","parent_split_bits":0,"last_scrub":"229'10","last_scrub_stamp":"2024-03-11T21:41:09.868606+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:41.544550+0000","last_clean_scrub_stamp":"2024-03-11T21:41:09.868606+0000","objects_scrubbed":1,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:42:28.936123+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.18","version":"67'1","reported_seq":754,"reported_epoch":394,"state":"active+clean","last_fresh":"2024-03-11T21:42:25.501933+0000","last_change":"2024-03-11T21:36:10.458150+0000","last_active":"2024-03-11T21:42:25.501933+0000","last_peered":"2024-03-11T21:42:25.501933+0000","last_clean":"2024-03-11T21:42:25.501933+0000","last_became_active":"2024-03-11T21:35:23.310241+0000","last_became_peered":"2024-03-11T21:35:23.310241+0000","last_unstale":"2024-03-11T21:42:25.501933+0000","last_undegraded":"2024-03-11T21:42:25.501933+0000","last_fullsized":"2024-03-11T21:42:25.501933+0000","mapping_epoch":248,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":249,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'1","last_scrub_stamp":"2024-03-11T21:36:10.458028+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:37.512018+0000","last_clean_scrub_stamp":"2024-03-11T21:36:10.458028+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":436,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":7,"num_read_kb":7,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":1308,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,2],"acting":[5,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1e","version":"260'20","reported_seq":801,"reported_epoch":392,"state":"active+clean","last_fresh":"2024-03-11T21:42:21.477817+0000","last_change":"2024-03-11T21:41:10.256209+0000","last_active":"2024-03-11T21:42:21.477817+0000","last_peered":"2024-03-11T21:42:21.477817+0000","last_clean":"2024-03-11T21:42:21.477817+0000","last_became_active":"2024-03-11T21:36:35.057843+0000","last_became_peered":"2024-03-11T21:36:35.057843+0000","last_unstale":"2024-03-11T21:42:21.477817+0000","last_undegraded":"2024-03-11T21:42:21.477817+0000","last_fullsized":"2024-03-11T21:42:21.477817+0000","mapping_epoch":277,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":278,"parent":"0.0","parent_split_bits":0,"last_scrub":"260'20","last_scrub_stamp":"2024-03-11T21:41:10.255971+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:39.525554+0000","last_clean_scrub_stamp":"2024-03-11T21:41:10.255971+0000","objects_scrubbed":5,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":25,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.13","version":"0'0","reported_seq":587,"reported_epoch":392,"state":"active+clean","last_fresh":"2024-03-11T21:42:21.481217+0000","last_change":"2024-03-11T21:41:26.639950+0000","last_active":"2024-03-11T21:42:21.481217+0000","last_peered":"2024-03-11T21:42:21.481217+0000","last_clean":"2024-03-11T21:42:21.481217+0000","last_became_active":"2024-03-11T21:36:35.062618+0000","last_became_peered":"2024-03-11T21:36:35.062618+0000","last_unstale":"2024-03-11T21:42:21.481217+0000","last_undegraded":"2024-03-11T21:42:21.481217+0000","last_fullsized":"2024-03-11T21:42:21.481217+0000","mapping_epoch":277,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":278,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:41:26.639290+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:43.559452+0000","last_clean_scrub_stamp":"2024-03-11T21:41:26.639290+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:42:56.069265+0000","scrub_duration":114,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":364,"reported_epoch":394,"state":"active+clean","last_fresh":"2024-03-11T21:42:25.532877+0000","last_change":"2024-03-11T21:40:29.084178+0000","last_active":"2024-03-11T21:42:25.532877+0000","last_peered":"2024-03-11T21:42:25.532877+0000","last_clean":"2024-03-11T21:42:25.532877+0000","last_became_active":"2024-03-11T21:34:41.142569+0000","last_became_peered":"2024-03-11T21:34:41.142569+0000","last_unstale":"2024-03-11T21:42:25.532877+0000","last_undegraded":"2024-03-11T21:42:25.532877+0000","last_fullsized":"2024-03-11T21:42:25.532877+0000","mapping_epoch":226,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":227,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:40:29.084062+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:40:29.084062+0000","last_clean_scrub_stamp":"2024-03-11T21:40:29.084062+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:42:28.198326+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.17","version":"0'0","reported_seq":735,"reported_epoch":394,"state":"active+clean","last_fresh":"2024-03-11T21:42:25.505430+0000","last_change":"2024-03-11T21:41:26.816265+0000","last_active":"2024-03-11T21:42:25.505430+0000","last_peered":"2024-03-11T21:42:25.505430+0000","last_clean":"2024-03-11T21:42:25.505430+0000","last_became_active":"2024-03-11T21:25:38.540349+0000","last_became_peered":"2024-03-11T21:25:38.540349+0000","last_unstale":"2024-03-11T21:42:25.505430+0000","last_undegraded":"2024-03-11T21:42:25.505430+0000","last_fullsized":"2024-03-11T21:42:25.505430+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:41:26.816135+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:28:25.918101+0000","last_clean_scrub_stamp":"2024-03-11T21:41:26.816135+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:42:48.936860+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,2],"acting":[6,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.11","version":"260'19","reported_seq":705,"reported_epoch":392,"state":"active+clean","last_fresh":"2024-03-11T21:42:21.491877+0000","last_change":"2024-03-11T21:36:35.096720+0000","last_active":"2024-03-11T21:42:21.491877+0000","last_peered":"2024-03-11T21:42:21.491877+0000","last_clean":"2024-03-11T21:42:21.491877+0000","last_became_active":"2024-03-11T21:36:35.073689+0000","last_became_peered":"2024-03-11T21:36:35.073689+0000","last_unstale":"2024-03-11T21:42:21.491877+0000","last_undegraded":"2024-03-11T21:42:21.491877+0000","last_fullsized":"2024-03-11T21:42:21.491877+0000","mapping_epoch":277,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":278,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'11","last_scrub_stamp":"2024-03-11T21:35:16.044397+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:39.525554+0000","last_clean_scrub_stamp":"2024-03-11T21:35:16.044397+0000","objects_scrubbed":5,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":31,"num_read_kb":24,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":364,"reported_epoch":394,"state":"active+clean","last_fresh":"2024-03-11T21:42:25.532880+0000","last_change":"2024-03-11T21:41:16.345639+0000","last_active":"2024-03-11T21:42:25.532880+0000","last_peered":"2024-03-11T21:42:25.532880+0000","last_clean":"2024-03-11T21:42:25.532880+0000","last_became_active":"2024-03-11T21:34:41.137235+0000","last_became_peered":"2024-03-11T21:34:41.137235+0000","last_unstale":"2024-03-11T21:42:25.532880+0000","last_undegraded":"2024-03-11T21:42:25.532880+0000","last_fullsized":"2024-03-11T21:42:25.532880+0000","mapping_epoch":226,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":227,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:41:16.345432+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:43.559452+0000","last_clean_scrub_stamp":"2024-03-11T21:41:16.345432+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:42:45.034048+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,4],"acting":[7,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":740,"reported_epoch":394,"state":"active+clean","last_fresh":"2024-03-11T21:42:25.504131+0000","last_change":"2024-03-11T21:40:03.181212+0000","last_active":"2024-03-11T21:42:25.504131+0000","last_peered":"2024-03-11T21:42:25.504131+0000","last_clean":"2024-03-11T21:42:25.504131+0000","last_became_active":"2024-03-11T21:34:41.156036+0000","last_became_peered":"2024-03-11T21:34:41.156036+0000","last_unstale":"2024-03-11T21:42:25.504131+0000","last_undegraded":"2024-03-11T21:42:25.504131+0000","last_fullsized":"2024-03-11T21:42:25.504131+0000","mapping_epoch":226,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":227,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:40:03.181074+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:38:51.036736+0000","last_clean_scrub_stamp":"2024-03-11T21:40:03.181074+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.16","version":"0'0","reported_seq":726,"reported_epoch":394,"state":"active+clean","last_fresh":"2024-03-11T21:42:25.502687+0000","last_change":"2024-03-11T21:41:25.953645+0000","last_active":"2024-03-11T21:42:25.502687+0000","last_peered":"2024-03-11T21:42:25.502687+0000","last_clean":"2024-03-11T21:42:25.502687+0000","last_became_active":"2024-03-11T21:25:38.539103+0000","last_became_peered":"2024-03-11T21:25:38.539103+0000","last_unstale":"2024-03-11T21:42:25.502687+0000","last_undegraded":"2024-03-11T21:42:25.502687+0000","last_fullsized":"2024-03-11T21:42:25.502687+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:41:25.953529+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:27:11.938909+0000","last_clean_scrub_stamp":"2024-03-11T21:41:25.953529+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:42:41.374895+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,2],"acting":[5,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.10","version":"260'10","reported_seq":595,"reported_epoch":392,"state":"active+clean","last_fresh":"2024-03-11T21:42:21.479487+0000","last_change":"2024-03-11T21:41:17.565434+0000","last_active":"2024-03-11T21:42:21.479487+0000","last_peered":"2024-03-11T21:42:21.479487+0000","last_clean":"2024-03-11T21:42:21.479487+0000","last_became_active":"2024-03-11T21:36:35.067500+0000","last_became_peered":"2024-03-11T21:36:35.067500+0000","last_unstale":"2024-03-11T21:42:21.479487+0000","last_undegraded":"2024-03-11T21:42:21.479487+0000","last_fullsized":"2024-03-11T21:42:21.479487+0000","mapping_epoch":277,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":278,"parent":"0.0","parent_split_bits":0,"last_scrub":"260'10","last_scrub_stamp":"2024-03-11T21:41:17.565310+0000","last_deep_scrub":"94'6","last_deep_scrub_stamp":"2024-03-11T21:28:24.603974+0000","last_clean_scrub_stamp":"2024-03-11T21:41:17.565310+0000","objects_scrubbed":2,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":12,"num_read_kb":10,"num_write":8,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6],"acting":[3,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":632,"reported_epoch":392,"state":"active+clean","last_fresh":"2024-03-11T21:42:21.491720+0000","last_change":"2024-03-11T21:36:35.070468+0000","last_active":"2024-03-11T21:42:21.491720+0000","last_peered":"2024-03-11T21:42:21.491720+0000","last_clean":"2024-03-11T21:42:21.491720+0000","last_became_active":"2024-03-11T21:36:35.070176+0000","last_became_peered":"2024-03-11T21:36:35.070176+0000","last_unstale":"2024-03-11T21:42:21.491720+0000","last_undegraded":"2024-03-11T21:42:21.491720+0000","last_fullsized":"2024-03-11T21:42:21.491720+0000","mapping_epoch":277,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":278,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:35:18.062272+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:43.559452+0000","last_clean_scrub_stamp":"2024-03-11T21:35:18.062272+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":709,"reported_epoch":392,"state":"active+clean","last_fresh":"2024-03-11T21:42:21.474790+0000","last_change":"2024-03-11T21:41:04.276105+0000","last_active":"2024-03-11T21:42:21.474790+0000","last_peered":"2024-03-11T21:42:21.474790+0000","last_clean":"2024-03-11T21:42:21.474790+0000","last_became_active":"2024-03-11T21:36:35.057906+0000","last_became_peered":"2024-03-11T21:36:35.057906+0000","last_unstale":"2024-03-11T21:42:21.474790+0000","last_undegraded":"2024-03-11T21:42:21.474790+0000","last_fullsized":"2024-03-11T21:42:21.474790+0000","mapping_epoch":277,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":278,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:41:04.275820+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:27:21.207303+0000","last_clean_scrub_stamp":"2024-03-11T21:41:04.275820+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:42:25.100823+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.15","version":"0'0","reported_seq":707,"reported_epoch":392,"state":"active+clean","last_fresh":"2024-03-11T21:42:21.476952+0000","last_change":"2024-03-11T21:36:06.579754+0000","last_active":"2024-03-11T21:42:21.476952+0000","last_peered":"2024-03-11T21:42:21.476952+0000","last_clean":"2024-03-11T21:42:21.476952+0000","last_became_active":"2024-03-11T21:25:38.541705+0000","last_became_peered":"2024-03-11T21:25:38.541705+0000","last_unstale":"2024-03-11T21:42:21.476952+0000","last_undegraded":"2024-03-11T21:42:21.476952+0000","last_fullsized":"2024-03-11T21:42:21.476952+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:36:06.579636+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:26:44.860499+0000","last_clean_scrub_stamp":"2024-03-11T21:36:06.579636+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.13","version":"260'24","reported_seq":770,"reported_epoch":394,"state":"active+clean","last_fresh":"2024-03-11T21:42:25.536927+0000","last_change":"2024-03-11T21:40:25.595320+0000","last_active":"2024-03-11T21:42:25.536927+0000","last_peered":"2024-03-11T21:42:25.536927+0000","last_clean":"2024-03-11T21:42:25.536927+0000","last_became_active":"2024-03-11T21:26:11.214879+0000","last_became_peered":"2024-03-11T21:26:11.214879+0000","last_unstale":"2024-03-11T21:42:25.536927+0000","last_undegraded":"2024-03-11T21:42:25.536927+0000","last_fullsized":"2024-03-11T21:42:25.536927+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"260'24","last_scrub_stamp":"2024-03-11T21:40:25.595220+0000","last_deep_scrub":"260'24","last_deep_scrub_stamp":"2024-03-11T21:40:25.595220+0000","last_clean_scrub_stamp":"2024-03-11T21:40:25.595220+0000","objects_scrubbed":5,"log_size":24,"log_dups_size":0,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":32,"num_read_kb":25,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,2],"acting":[4,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.10","version":"0'0","reported_seq":696,"reported_epoch":392,"state":"active+clean","last_fresh":"2024-03-11T21:42:21.474968+0000","last_change":"2024-03-11T21:35:29.712658+0000","last_active":"2024-03-11T21:42:21.474968+0000","last_peered":"2024-03-11T21:42:21.474968+0000","last_clean":"2024-03-11T21:42:21.474968+0000","last_became_active":"2024-03-11T21:25:44.591189+0000","last_became_peered":"2024-03-11T21:25:44.591189+0000","last_unstale":"2024-03-11T21:42:21.474968+0000","last_undegraded":"2024-03-11T21:42:21.474968+0000","last_fullsized":"2024-03-11T21:42:21.474968+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:35:29.712557+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:27:25.417450+0000","last_clean_scrub_stamp":"2024-03-11T21:35:29.712557+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,1],"acting":[0,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":630,"reported_epoch":392,"state":"active+clean","last_fresh":"2024-03-11T21:42:21.481317+0000","last_change":"2024-03-11T21:36:35.062276+0000","last_active":"2024-03-11T21:42:21.481317+0000","last_peered":"2024-03-11T21:42:21.481317+0000","last_clean":"2024-03-11T21:42:21.481317+0000","last_became_active":"2024-03-11T21:36:35.056619+0000","last_became_peered":"2024-03-11T21:36:35.056619+0000","last_unstale":"2024-03-11T21:42:21.481317+0000","last_undegraded":"2024-03-11T21:42:21.481317+0000","last_fullsized":"2024-03-11T21:42:21.481317+0000","mapping_epoch":277,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":278,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:35:13.060487+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:31:15.367898+0000","last_clean_scrub_stamp":"2024-03-11T21:35:13.060487+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.14","version":"0'0","reported_seq":753,"reported_epoch":394,"state":"active+clean","last_fresh":"2024-03-11T21:42:25.504728+0000","last_change":"2024-03-11T21:41:14.822928+0000","last_active":"2024-03-11T21:42:25.504728+0000","last_peered":"2024-03-11T21:42:25.504728+0000","last_clean":"2024-03-11T21:42:25.504728+0000","last_became_active":"2024-03-11T21:36:35.068038+0000","last_became_peered":"2024-03-11T21:36:35.068038+0000","last_unstale":"2024-03-11T21:42:25.504728+0000","last_undegraded":"2024-03-11T21:42:25.504728+0000","last_fullsized":"2024-03-11T21:42:25.504728+0000","mapping_epoch":277,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":278,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:41:14.822805+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:37.512018+0000","last_clean_scrub_stamp":"2024-03-11T21:41:14.822805+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,5],"acting":[6,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.12","version":"260'15","reported_seq":737,"reported_epoch":392,"state":"active+clean","last_fresh":"2024-03-11T21:42:21.474612+0000","last_change":"2024-03-11T21:41:18.319324+0000","last_active":"2024-03-11T21:42:21.474612+0000","last_peered":"2024-03-11T21:42:21.474612+0000","last_clean":"2024-03-11T21:42:21.474612+0000","last_became_active":"2024-03-11T21:25:40.547082+0000","last_became_peered":"2024-03-11T21:25:40.547082+0000","last_unstale":"2024-03-11T21:42:21.474612+0000","last_undegraded":"2024-03-11T21:42:21.474612+0000","last_fullsized":"2024-03-11T21:42:21.474612+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"260'15","last_scrub_stamp":"2024-03-11T21:41:18.319190+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:39.525554+0000","last_clean_scrub_stamp":"2024-03-11T21:41:18.319190+0000","objects_scrubbed":4,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:42:45.299210+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":707,"reported_epoch":394,"state":"active+clean","last_fresh":"2024-03-11T21:42:25.538379+0000","last_change":"2024-03-11T21:41:25.741034+0000","last_active":"2024-03-11T21:42:25.538379+0000","last_peered":"2024-03-11T21:42:25.538379+0000","last_clean":"2024-03-11T21:42:25.538379+0000","last_became_active":"2024-03-11T21:25:44.589586+0000","last_became_peered":"2024-03-11T21:25:44.589586+0000","last_unstale":"2024-03-11T21:42:25.538379+0000","last_undegraded":"2024-03-11T21:42:25.538379+0000","last_fullsized":"2024-03-11T21:42:25.538379+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:41:25.740917+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:27:09.801138+0000","last_clean_scrub_stamp":"2024-03-11T21:41:25.740917+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:42:52.578690+0000","scrub_duration":158,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,5],"acting":[4,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.14","version":"280'12","reported_seq":862,"reported_epoch":392,"state":"active+clean","last_fresh":"2024-03-11T21:42:21.487019+0000","last_change":"2024-03-11T21:36:35.074550+0000","last_active":"2024-03-11T21:42:21.487019+0000","last_peered":"2024-03-11T21:42:21.487019+0000","last_clean":"2024-03-11T21:42:21.487019+0000","last_became_active":"2024-03-11T21:36:35.073912+0000","last_became_peered":"2024-03-11T21:36:35.073912+0000","last_unstale":"2024-03-11T21:42:21.487019+0000","last_undegraded":"2024-03-11T21:42:21.487019+0000","last_fullsized":"2024-03-11T21:42:21.487019+0000","mapping_epoch":277,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":278,"parent":"0.0","parent_split_bits":0,"last_scrub":"229'10","last_scrub_stamp":"2024-03-11T21:35:26.182107+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:41.544550+0000","last_clean_scrub_stamp":"2024-03-11T21:35:26.182107+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,2],"acting":[3,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.13","version":"0'0","reported_seq":717,"reported_epoch":392,"state":"active+clean","last_fresh":"2024-03-11T21:42:21.478237+0000","last_change":"2024-03-11T21:40:11.027382+0000","last_active":"2024-03-11T21:42:21.478237+0000","last_peered":"2024-03-11T21:42:21.478237+0000","last_clean":"2024-03-11T21:42:21.478237+0000","last_became_active":"2024-03-11T21:36:35.057132+0000","last_became_peered":"2024-03-11T21:36:35.057132+0000","last_unstale":"2024-03-11T21:42:21.478237+0000","last_undegraded":"2024-03-11T21:42:21.478237+0000","last_fullsized":"2024-03-11T21:42:21.478237+0000","mapping_epoch":277,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":278,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:40:11.027268+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:37.512018+0000","last_clean_scrub_stamp":"2024-03-11T21:40:11.027268+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":24,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.15","version":"260'19","reported_seq":793,"reported_epoch":394,"state":"active+clean","last_fresh":"2024-03-11T21:42:25.502428+0000","last_change":"2024-03-11T21:41:18.838796+0000","last_active":"2024-03-11T21:42:25.502428+0000","last_peered":"2024-03-11T21:42:25.502428+0000","last_clean":"2024-03-11T21:42:25.502428+0000","last_became_active":"2024-03-11T21:36:35.057285+0000","last_became_peered":"2024-03-11T21:36:35.057285+0000","last_unstale":"2024-03-11T21:42:25.502428+0000","last_undegraded":"2024-03-11T21:42:25.502428+0000","last_fullsized":"2024-03-11T21:42:25.502428+0000","mapping_epoch":277,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":278,"parent":"0.0","parent_split_bits":0,"last_scrub":"260'19","last_scrub_stamp":"2024-03-11T21:41:18.838669+0000","last_deep_scrub":"94'13","last_deep_scrub_stamp":"2024-03-11T21:29:41.853158+0000","last_clean_scrub_stamp":"2024-03-11T21:41:18.838669+0000","objects_scrubbed":4,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:42:37.444464+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":11,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.16","version":"0'0","reported_seq":724,"reported_epoch":392,"state":"active+clean","last_fresh":"2024-03-11T21:42:21.476212+0000","last_change":"2024-03-11T21:40:30.249810+0000","last_active":"2024-03-11T21:42:21.476212+0000","last_peered":"2024-03-11T21:42:21.476212+0000","last_clean":"2024-03-11T21:42:21.476212+0000","last_became_active":"2024-03-11T21:36:35.057355+0000","last_became_peered":"2024-03-11T21:36:35.057355+0000","last_unstale":"2024-03-11T21:42:21.476212+0000","last_undegraded":"2024-03-11T21:42:21.476212+0000","last_fullsized":"2024-03-11T21:42:21.476212+0000","mapping_epoch":277,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":278,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:40:30.249679+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:40:30.249679+0000","last_clean_scrub_stamp":"2024-03-11T21:40:30.249679+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":40,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.15","version":"98'9","reported_seq":913,"reported_epoch":394,"state":"active+clean","last_fresh":"2024-03-11T21:42:25.502595+0000","last_change":"2024-03-11T21:36:03.366581+0000","last_active":"2024-03-11T21:42:25.502595+0000","last_peered":"2024-03-11T21:42:25.502595+0000","last_clean":"2024-03-11T21:42:25.502595+0000","last_became_active":"2024-03-11T21:25:42.572555+0000","last_became_peered":"2024-03-11T21:25:42.572555+0000","last_unstale":"2024-03-11T21:42:25.502595+0000","last_undegraded":"2024-03-11T21:42:25.502595+0000","last_fullsized":"2024-03-11T21:42:25.502595+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"98'9","last_scrub_stamp":"2024-03-11T21:36:03.366450+0000","last_deep_scrub":"98'9","last_deep_scrub_stamp":"2024-03-11T21:31:20.624923+0000","last_clean_scrub_stamp":"2024-03-11T21:36:03.366450+0000","objects_scrubbed":1,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.12","version":"0'0","reported_seq":740,"reported_epoch":394,"state":"active+clean","last_fresh":"2024-03-11T21:42:25.502166+0000","last_change":"2024-03-11T21:41:21.853550+0000","last_active":"2024-03-11T21:42:25.502166+0000","last_peered":"2024-03-11T21:42:25.502166+0000","last_clean":"2024-03-11T21:42:25.502166+0000","last_became_active":"2024-03-11T21:36:35.058033+0000","last_became_peered":"2024-03-11T21:36:35.058033+0000","last_unstale":"2024-03-11T21:42:25.502166+0000","last_undegraded":"2024-03-11T21:42:25.502166+0000","last_fullsized":"2024-03-11T21:42:25.502166+0000","mapping_epoch":277,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":278,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:41:21.853428+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:37.512018+0000","last_clean_scrub_stamp":"2024-03-11T21:41:21.853428+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:42:32.633151+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.14","version":"260'20","reported_seq":713,"reported_epoch":392,"state":"active+clean","last_fresh":"2024-03-11T21:42:21.486664+0000","last_change":"2024-03-11T21:36:35.095382+0000","last_active":"2024-03-11T21:42:21.486664+0000","last_peered":"2024-03-11T21:42:21.486664+0000","last_clean":"2024-03-11T21:42:21.486664+0000","last_became_active":"2024-03-11T21:36:35.069638+0000","last_became_peered":"2024-03-11T21:36:35.069638+0000","last_unstale":"2024-03-11T21:42:21.486664+0000","last_undegraded":"2024-03-11T21:42:21.486664+0000","last_fullsized":"2024-03-11T21:42:21.486664+0000","mapping_epoch":277,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":278,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'10","last_scrub_stamp":"2024-03-11T21:35:28.189296+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:39.525554+0000","last_clean_scrub_stamp":"2024-03-11T21:35:28.189296+0000","objects_scrubbed":0,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":29,"num_read_kb":24,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.19","version":"392'5616","reported_seq":5706,"reported_epoch":392,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-03-11T21:42:22.422286+0000","last_change":"2024-03-11T21:41:32.336298+0000","last_active":"2024-03-11T21:42:22.422286+0000","last_peered":"2024-03-11T21:42:22.422286+0000","last_clean":"2024-03-11T21:41:29.745622+0000","last_became_active":"2024-03-11T21:41:32.289876+0000","last_became_peered":"2024-03-11T21:41:32.289876+0000","last_unstale":"2024-03-11T21:42:22.422286+0000","last_undegraded":"2024-03-11T21:41:32.227784+0000","last_fullsized":"2024-03-11T21:41:31.839405+0000","mapping_epoch":375,"log_start":"0'0","ondisk_log_start":"0'0","created":372,"last_epoch_clean":373,"parent":"0.0","parent_split_bits":5,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:40:38.219602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:40:38.219602+0000","last_clean_scrub_stamp":"2024-03-11T21:40:38.219602+0000","objects_scrubbed":0,"log_size":5616,"log_dups_size":0,"ondisk_log_size":5616,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":226721792,"num_objects":3460,"num_object_clones":0,"num_object_copies":10380,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4520,"num_objects_misplaced":2260,"num_objects_unfound":0,"num_objects_dirty":3460,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3460,"num_write_kb":221408,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,3],"acting":[2,1],"avail_no_missing":["2","5","7"],"object_location_counts":[{"shards":"1,2","objects":1200},{"shards":"2,5,7","objects":2260}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.15","version":"0'0","reported_seq":364,"reported_epoch":394,"state":"active+clean","last_fresh":"2024-03-11T21:42:25.534942+0000","last_change":"2024-03-11T21:41:31.306328+0000","last_active":"2024-03-11T21:42:25.534942+0000","last_peered":"2024-03-11T21:42:25.534942+0000","last_clean":"2024-03-11T21:42:25.534942+0000","last_became_active":"2024-03-11T21:34:41.142760+0000","last_became_peered":"2024-03-11T21:34:41.142760+0000","last_unstale":"2024-03-11T21:42:25.534942+0000","last_undegraded":"2024-03-11T21:42:25.534942+0000","last_fullsized":"2024-03-11T21:42:25.534942+0000","mapping_epoch":226,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":227,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:41:31.306234+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:43.559452+0000","last_clean_scrub_stamp":"2024-03-11T21:41:31.306234+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:42:46.518294+0000","scrub_duration":38,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":711,"reported_epoch":394,"state":"active+clean","last_fresh":"2024-03-11T21:42:25.502361+0000","last_change":"2024-03-11T21:40:26.587777+0000","last_active":"2024-03-11T21:42:25.502361+0000","last_peered":"2024-03-11T21:42:25.502361+0000","last_clean":"2024-03-11T21:42:25.502361+0000","last_became_active":"2024-03-11T21:36:35.060342+0000","last_became_peered":"2024-03-11T21:36:35.060342+0000","last_unstale":"2024-03-11T21:42:25.502361+0000","last_undegraded":"2024-03-11T21:42:25.502361+0000","last_fullsized":"2024-03-11T21:42:25.502361+0000","mapping_epoch":277,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":278,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:40:26.587672+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:41.544550+0000","last_clean_scrub_stamp":"2024-03-11T21:40:26.587672+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,1],"acting":[5,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.11","version":"0'0","reported_seq":728,"reported_epoch":394,"state":"active+clean","last_fresh":"2024-03-11T21:42:25.504185+0000","last_change":"2024-03-11T21:40:09.218006+0000","last_active":"2024-03-11T21:42:25.504185+0000","last_peered":"2024-03-11T21:42:25.504185+0000","last_clean":"2024-03-11T21:42:25.504185+0000","last_became_active":"2024-03-11T21:25:38.537415+0000","last_became_peered":"2024-03-11T21:25:38.537415+0000","last_unstale":"2024-03-11T21:42:25.504185+0000","last_undegraded":"2024-03-11T21:42:25.504185+0000","last_fullsized":"2024-03-11T21:42:25.504185+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:40:09.217927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:37.512018+0000","last_clean_scrub_stamp":"2024-03-11T21:40:09.217927+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.17","version":"260'15","reported_seq":666,"reported_epoch":392,"state":"active+clean","last_fresh":"2024-03-11T21:42:21.492314+0000","last_change":"2024-03-11T21:41:00.652483+0000","last_active":"2024-03-11T21:42:21.492314+0000","last_peered":"2024-03-11T21:42:21.492314+0000","last_clean":"2024-03-11T21:42:21.492314+0000","last_became_active":"2024-03-11T21:36:35.071346+0000","last_became_peered":"2024-03-11T21:36:35.071346+0000","last_unstale":"2024-03-11T21:42:21.492314+0000","last_undegraded":"2024-03-11T21:42:21.492314+0000","last_fullsized":"2024-03-11T21:42:21.492314+0000","mapping_epoch":277,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":278,"parent":"0.0","parent_split_bits":0,"last_scrub":"260'15","last_scrub_stamp":"2024-03-11T21:41:00.652358+0000","last_deep_scrub":"97'9","last_deep_scrub_stamp":"2024-03-11T21:27:25.803132+0000","last_clean_scrub_stamp":"2024-03-11T21:41:00.652358+0000","objects_scrubbed":3,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":17,"num_read_kb":14,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,5],"acting":[3,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.18","version":"392'5642","reported_seq":6059,"reported_epoch":392,"state":"active+recovering+undersized+remapped","last_fresh":"2024-03-11T21:42:22.546857+0000","last_change":"2024-03-11T21:42:19.922849+0000","last_active":"2024-03-11T21:42:22.546857+0000","last_peered":"2024-03-11T21:42:22.546857+0000","last_clean":"2024-03-11T21:41:29.736519+0000","last_became_active":"2024-03-11T21:41:32.292464+0000","last_became_peered":"2024-03-11T21:41:32.292464+0000","last_unstale":"2024-03-11T21:42:22.546857+0000","last_undegraded":"2024-03-11T21:42:22.546857+0000","last_fullsized":"2024-03-11T21:41:31.840680+0000","mapping_epoch":375,"log_start":"0'0","ondisk_log_start":"0'0","created":372,"last_epoch_clean":373,"parent":"0.0","parent_split_bits":5,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:40:38.219602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:40:38.219602+0000","last_clean_scrub_stamp":"2024-03-11T21:40:38.219602+0000","objects_scrubbed":0,"log_size":5642,"log_dups_size":0,"ondisk_log_size":5642,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":227409920,"num_objects":3470,"num_object_clones":0,"num_object_copies":10410,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":4206,"num_objects_unfound":0,"num_objects_dirty":3470,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3470,"num_write_kb":222080,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":109,"num_bytes_recovered":7143424,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1],"avail_no_missing":["2","0","7"],"object_location_counts":[{"shards":"0,2,7","objects":2103},{"shards":"1,2","objects":1367}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.14","version":"0'0","reported_seq":714,"reported_epoch":392,"state":"active+clean","last_fresh":"2024-03-11T21:42:21.486770+0000","last_change":"2024-03-11T21:41:08.130109+0000","last_active":"2024-03-11T21:42:21.486770+0000","last_peered":"2024-03-11T21:42:21.486770+0000","last_clean":"2024-03-11T21:42:21.486770+0000","last_became_active":"2024-03-11T21:34:41.136946+0000","last_became_peered":"2024-03-11T21:34:41.136946+0000","last_unstale":"2024-03-11T21:42:21.486770+0000","last_undegraded":"2024-03-11T21:42:21.486770+0000","last_fullsized":"2024-03-11T21:42:21.486770+0000","mapping_epoch":226,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":227,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:41:08.129982+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:43.559452+0000","last_clean_scrub_stamp":"2024-03-11T21:41:08.129982+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,7],"acting":[2,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":620,"reported_epoch":392,"state":"active+clean","last_fresh":"2024-03-11T21:42:21.481365+0000","last_change":"2024-03-11T21:41:31.530082+0000","last_active":"2024-03-11T21:42:21.481365+0000","last_peered":"2024-03-11T21:42:21.481365+0000","last_clean":"2024-03-11T21:42:21.481365+0000","last_became_active":"2024-03-11T21:36:35.076335+0000","last_became_peered":"2024-03-11T21:36:35.076335+0000","last_unstale":"2024-03-11T21:42:21.481365+0000","last_undegraded":"2024-03-11T21:42:21.481365+0000","last_fullsized":"2024-03-11T21:42:21.481365+0000","mapping_epoch":277,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":278,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:41:31.529958+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:41.544550+0000","last_clean_scrub_stamp":"2024-03-11T21:41:31.529958+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:42:38.422074+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.10","version":"0'0","reported_seq":714,"reported_epoch":392,"state":"active+clean","last_fresh":"2024-03-11T21:42:21.488912+0000","last_change":"2024-03-11T21:41:23.299628+0000","last_active":"2024-03-11T21:42:21.488912+0000","last_peered":"2024-03-11T21:42:21.488912+0000","last_clean":"2024-03-11T21:42:21.488912+0000","last_became_active":"2024-03-11T21:25:38.533138+0000","last_became_peered":"2024-03-11T21:25:38.533138+0000","last_unstale":"2024-03-11T21:42:21.488912+0000","last_undegraded":"2024-03-11T21:42:21.488912+0000","last_fullsized":"2024-03-11T21:42:21.488912+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:41:23.299503+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:37.512018+0000","last_clean_scrub_stamp":"2024-03-11T21:41:23.299503+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:42:26.025672+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,0],"acting":[2,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.16","version":"369'19","reported_seq":790,"reported_epoch":392,"state":"active+clean","last_fresh":"2024-03-11T21:42:21.476308+0000","last_change":"2024-03-11T21:41:22.115878+0000","last_active":"2024-03-11T21:42:21.476308+0000","last_peered":"2024-03-11T21:42:21.476308+0000","last_clean":"2024-03-11T21:42:21.476308+0000","last_became_active":"2024-03-11T21:36:35.056862+0000","last_became_peered":"2024-03-11T21:36:35.056862+0000","last_unstale":"2024-03-11T21:42:21.476308+0000","last_undegraded":"2024-03-11T21:42:21.476308+0000","last_fullsized":"2024-03-11T21:42:21.476308+0000","mapping_epoch":277,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":278,"parent":"0.0","parent_split_bits":0,"last_scrub":"369'19","last_scrub_stamp":"2024-03-11T21:41:22.115634+0000","last_deep_scrub":"369'19","last_deep_scrub_stamp":"2024-03-11T21:41:22.115634+0000","last_clean_scrub_stamp":"2024-03-11T21:41:22.115634+0000","objects_scrubbed":4,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:42:24.859913+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":11,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.7","version":"394'5685","reported_seq":5809,"reported_epoch":394,"state":"active+clean","last_fresh":"2024-03-11T21:42:27.254736+0000","last_change":"2024-03-11T21:41:31.118161+0000","last_active":"2024-03-11T21:42:27.254736+0000","last_peered":"2024-03-11T21:42:27.254736+0000","last_clean":"2024-03-11T21:42:27.254736+0000","last_became_active":"2024-03-11T21:41:28.840566+0000","last_became_peered":"2024-03-11T21:41:28.840566+0000","last_unstale":"2024-03-11T21:42:27.254736+0000","last_undegraded":"2024-03-11T21:42:27.254736+0000","last_fullsized":"2024-03-11T21:42:27.254736+0000","mapping_epoch":356,"log_start":"0'0","ondisk_log_start":"0'0","created":356,"last_epoch_clean":373,"parent":"0.0","parent_split_bits":0,"last_scrub":"374'4410","last_scrub_stamp":"2024-03-11T21:41:31.117966+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:40:38.219602+0000","last_clean_scrub_stamp":"2024-03-11T21:41:31.117966+0000","objects_scrubbed":2221,"log_size":5685,"log_dups_size":0,"ondisk_log_size":5685,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:42:58.305558+0000","scrub_duration":256,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220725248,"num_objects":3368,"num_object_clones":0,"num_object_copies":10104,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3368,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1275,"num_write_kb":77504,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,5],"acting":[6,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.b","version":"0'0","reported_seq":608,"reported_epoch":392,"state":"active+clean","last_fresh":"2024-03-11T21:42:21.481129+0000","last_change":"2024-03-11T21:40:27.611092+0000","last_active":"2024-03-11T21:42:21.481129+0000","last_peered":"2024-03-11T21:42:21.481129+0000","last_clean":"2024-03-11T21:42:21.481129+0000","last_became_active":"2024-03-11T21:36:35.072147+0000","last_became_peered":"2024-03-11T21:36:35.072147+0000","last_unstale":"2024-03-11T21:42:21.481129+0000","last_undegraded":"2024-03-11T21:42:21.481129+0000","last_fullsized":"2024-03-11T21:42:21.481129+0000","mapping_epoch":277,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":278,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:40:27.610987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:43.559452+0000","last_clean_scrub_stamp":"2024-03-11T21:40:27.610987+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":60,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.f","version":"67'2","reported_seq":732,"reported_epoch":394,"state":"active+clean","last_fresh":"2024-03-11T21:42:25.538168+0000","last_change":"2024-03-11T21:41:13.539379+0000","last_active":"2024-03-11T21:42:25.538168+0000","last_peered":"2024-03-11T21:42:25.538168+0000","last_clean":"2024-03-11T21:42:25.538168+0000","last_became_active":"2024-03-11T21:25:38.537711+0000","last_became_peered":"2024-03-11T21:25:38.537711+0000","last_unstale":"2024-03-11T21:42:25.538168+0000","last_undegraded":"2024-03-11T21:42:25.538168+0000","last_fullsized":"2024-03-11T21:42:25.538168+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'2","last_scrub_stamp":"2024-03-11T21:41:13.539240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:37.512018+0000","last_clean_scrub_stamp":"2024-03-11T21:41:13.539240+0000","objects_scrubbed":2,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":92,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":10,"num_read_kb":10,"num_write":4,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,7],"acting":[4,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.9","version":"260'23","reported_seq":763,"reported_epoch":394,"state":"active+clean","last_fresh":"2024-03-11T21:42:25.537409+0000","last_change":"2024-03-11T21:40:14.543701+0000","last_active":"2024-03-11T21:42:25.537409+0000","last_peered":"2024-03-11T21:42:25.537409+0000","last_clean":"2024-03-11T21:42:25.537409+0000","last_became_active":"2024-03-11T21:36:35.059930+0000","last_became_peered":"2024-03-11T21:36:35.059930+0000","last_unstale":"2024-03-11T21:42:25.537409+0000","last_undegraded":"2024-03-11T21:42:25.537409+0000","last_fullsized":"2024-03-11T21:42:25.537409+0000","mapping_epoch":277,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":278,"parent":"0.0","parent_split_bits":0,"last_scrub":"260'23","last_scrub_stamp":"2024-03-11T21:40:14.543596+0000","last_deep_scrub":"146'17","last_deep_scrub_stamp":"2024-03-11T21:34:26.765075+0000","last_clean_scrub_stamp":"2024-03-11T21:40:14.543596+0000","objects_scrubbed":5,"log_size":23,"log_dups_size":0,"ondisk_log_size":23,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":34,"num_read_kb":25,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,3],"acting":[4,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":706,"reported_epoch":392,"state":"active+clean","last_fresh":"2024-03-11T21:42:21.488997+0000","last_change":"2024-03-11T21:41:15.238256+0000","last_active":"2024-03-11T21:42:21.488997+0000","last_peered":"2024-03-11T21:42:21.488997+0000","last_clean":"2024-03-11T21:42:21.488997+0000","last_became_active":"2024-03-11T21:25:42.573987+0000","last_became_peered":"2024-03-11T21:25:42.573987+0000","last_unstale":"2024-03-11T21:42:21.488997+0000","last_undegraded":"2024-03-11T21:42:21.488997+0000","last_fullsized":"2024-03-11T21:42:21.488997+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:41:15.238017+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:41.544550+0000","last_clean_scrub_stamp":"2024-03-11T21:41:15.238017+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:42:35.162641+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"10.c","version":"394'7025","reported_seq":7114,"reported_epoch":394,"state":"active+clean","last_fresh":"2024-03-11T21:42:27.354882+0000","last_change":"2024-03-11T21:40:39.242008+0000","last_active":"2024-03-11T21:42:27.354882+0000","last_peered":"2024-03-11T21:42:27.354882+0000","last_clean":"2024-03-11T21:42:27.354882+0000","last_became_active":"2024-03-11T21:40:39.241699+0000","last_became_peered":"2024-03-11T21:40:39.241699+0000","last_unstale":"2024-03-11T21:42:27.354882+0000","last_undegraded":"2024-03-11T21:42:27.354882+0000","last_fullsized":"2024-03-11T21:42:27.354882+0000","mapping_epoch":356,"log_start":"366'2400","ondisk_log_start":"366'2400","created":356,"last_epoch_clean":357,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:40:38.219602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:40:38.219602+0000","last_clean_scrub_stamp":"2024-03-11T21:40:38.219602+0000","objects_scrubbed":0,"log_size":4625,"log_dups_size":0,"ondisk_log_size":4625,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:42:28.198361+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":440467456,"num_objects":6721,"num_object_clones":0,"num_object_copies":20163,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6721,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7025,"num_write_kb":439872,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,0],"acting":[7,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.0","version":"0'0","reported_seq":706,"reported_epoch":392,"state":"active+clean","last_fresh":"2024-03-11T21:42:21.475094+0000","last_change":"2024-03-11T21:40:27.230013+0000","last_active":"2024-03-11T21:42:21.475094+0000","last_peered":"2024-03-11T21:42:21.475094+0000","last_clean":"2024-03-11T21:42:21.475094+0000","last_became_active":"2024-03-11T21:36:35.058054+0000","last_became_peered":"2024-03-11T21:36:35.058054+0000","last_unstale":"2024-03-11T21:42:21.475094+0000","last_undegraded":"2024-03-11T21:42:21.475094+0000","last_fullsized":"2024-03-11T21:42:21.475094+0000","mapping_epoch":277,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":278,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:40:27.229904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:29:39.498784+0000","last_clean_scrub_stamp":"2024-03-11T21:40:27.229904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":38,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":729,"reported_epoch":392,"state":"active+clean","last_fresh":"2024-03-11T21:42:21.476951+0000","last_change":"2024-03-11T21:41:22.292906+0000","last_active":"2024-03-11T21:42:21.476951+0000","last_peered":"2024-03-11T21:42:21.476951+0000","last_clean":"2024-03-11T21:42:21.476951+0000","last_became_active":"2024-03-11T21:34:41.150913+0000","last_became_peered":"2024-03-11T21:34:41.150913+0000","last_unstale":"2024-03-11T21:42:21.476951+0000","last_undegraded":"2024-03-11T21:42:21.476951+0000","last_fullsized":"2024-03-11T21:42:21.476951+0000","mapping_epoch":226,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":227,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:41:22.292789+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:37.512018+0000","last_clean_scrub_stamp":"2024-03-11T21:41:22.292789+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:42:46.637753+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,7],"acting":[1,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.2","version":"260'25","reported_seq":748,"reported_epoch":392,"state":"active+clean","last_fresh":"2024-03-11T21:42:21.474742+0000","last_change":"2024-03-11T21:41:07.243196+0000","last_active":"2024-03-11T21:42:21.474742+0000","last_peered":"2024-03-11T21:42:21.474742+0000","last_clean":"2024-03-11T21:42:21.474742+0000","last_became_active":"2024-03-11T21:25:40.548833+0000","last_became_peered":"2024-03-11T21:25:40.548833+0000","last_unstale":"2024-03-11T21:42:21.474742+0000","last_undegraded":"2024-03-11T21:42:21.474742+0000","last_fullsized":"2024-03-11T21:42:21.474742+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"260'25","last_scrub_stamp":"2024-03-11T21:41:07.242950+0000","last_deep_scrub":"94'15","last_deep_scrub_stamp":"2024-03-11T21:26:49.993060+0000","last_clean_scrub_stamp":"2024-03-11T21:41:07.242950+0000","objects_scrubbed":5,"log_size":25,"log_dups_size":0,"ondisk_log_size":25,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":25,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.3","version":"74'8","reported_seq":919,"reported_epoch":392,"state":"active+clean","last_fresh":"2024-03-11T21:42:23.794566+0000","last_change":"2024-03-11T21:41:31.189011+0000","last_active":"2024-03-11T21:42:23.794566+0000","last_peered":"2024-03-11T21:42:23.794566+0000","last_clean":"2024-03-11T21:42:23.794566+0000","last_became_active":"2024-03-11T21:25:42.578161+0000","last_became_peered":"2024-03-11T21:25:42.578161+0000","last_unstale":"2024-03-11T21:42:23.794566+0000","last_undegraded":"2024-03-11T21:42:23.794566+0000","last_fullsized":"2024-03-11T21:42:23.794566+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'8","last_scrub_stamp":"2024-03-11T21:41:31.188904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:41.544550+0000","last_clean_scrub_stamp":"2024-03-11T21:41:31.188904+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:42:46.603231+0000","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,5],"acting":[0,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.a","version":"394'6989","reported_seq":7078,"reported_epoch":394,"state":"active+clean","last_fresh":"2024-03-11T21:42:27.343137+0000","last_change":"2024-03-11T21:40:39.243111+0000","last_active":"2024-03-11T21:42:27.343137+0000","last_peered":"2024-03-11T21:42:27.343137+0000","last_clean":"2024-03-11T21:42:27.343137+0000","last_became_active":"2024-03-11T21:40:39.242809+0000","last_became_peered":"2024-03-11T21:40:39.242809+0000","last_unstale":"2024-03-11T21:42:27.343137+0000","last_undegraded":"2024-03-11T21:42:27.343137+0000","last_fullsized":"2024-03-11T21:42:27.343137+0000","mapping_epoch":356,"log_start":"365'1400","ondisk_log_start":"365'1400","created":356,"last_epoch_clean":357,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:40:38.219602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:40:38.219602+0000","last_clean_scrub_stamp":"2024-03-11T21:40:38.219602+0000","objects_scrubbed":0,"log_size":5589,"log_dups_size":0,"ondisk_log_size":5589,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":434307072,"num_objects":6627,"num_object_clones":0,"num_object_copies":19881,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6627,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6989,"num_write_kb":435712,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":638,"reported_epoch":392,"state":"active+clean","last_fresh":"2024-03-11T21:42:21.486243+0000","last_change":"2024-03-11T21:36:35.068977+0000","last_active":"2024-03-11T21:42:21.486243+0000","last_peered":"2024-03-11T21:42:21.486243+0000","last_clean":"2024-03-11T21:42:21.486243+0000","last_became_active":"2024-03-11T21:36:35.067288+0000","last_became_peered":"2024-03-11T21:36:35.067288+0000","last_unstale":"2024-03-11T21:42:21.486243+0000","last_undegraded":"2024-03-11T21:42:21.486243+0000","last_fullsized":"2024-03-11T21:42:21.486243+0000","mapping_epoch":277,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":278,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:35:09.985377+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:28:30.563712+0000","last_clean_scrub_stamp":"2024-03-11T21:35:09.985377+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,7],"acting":[3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":728,"reported_epoch":394,"state":"active+clean","last_fresh":"2024-03-11T21:42:25.502259+0000","last_change":"2024-03-11T21:41:24.921735+0000","last_active":"2024-03-11T21:42:25.502259+0000","last_peered":"2024-03-11T21:42:25.502259+0000","last_clean":"2024-03-11T21:42:25.502259+0000","last_became_active":"2024-03-11T21:25:38.538540+0000","last_became_peered":"2024-03-11T21:25:38.538540+0000","last_unstale":"2024-03-11T21:42:25.502259+0000","last_undegraded":"2024-03-11T21:42:25.502259+0000","last_fullsized":"2024-03-11T21:42:25.502259+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:41:24.921598+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:37.512018+0000","last_clean_scrub_stamp":"2024-03-11T21:41:24.921598+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:42:41.607350+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.4","version":"280'42","reported_seq":997,"reported_epoch":392,"state":"active+clean","last_fresh":"2024-03-11T21:42:21.476893+0000","last_change":"2024-03-11T21:41:05.234160+0000","last_active":"2024-03-11T21:42:21.476893+0000","last_peered":"2024-03-11T21:42:21.476893+0000","last_clean":"2024-03-11T21:42:21.476893+0000","last_became_active":"2024-03-11T21:36:35.073430+0000","last_became_peered":"2024-03-11T21:36:35.073430+0000","last_unstale":"2024-03-11T21:42:21.476893+0000","last_undegraded":"2024-03-11T21:42:21.476893+0000","last_fullsized":"2024-03-11T21:42:21.476893+0000","mapping_epoch":277,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":278,"parent":"0.0","parent_split_bits":0,"last_scrub":"280'42","last_scrub_stamp":"2024-03-11T21:41:05.233993+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:39.525554+0000","last_clean_scrub_stamp":"2024-03-11T21:41:05.233993+0000","objects_scrubbed":10,"log_size":42,"log_dups_size":0,"ondisk_log_size":42,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:42:33.447830+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":358,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":66,"num_read_kb":51,"num_write":38,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3],"acting":[1,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":700,"reported_epoch":392,"state":"active+clean","last_fresh":"2024-03-11T21:42:21.479085+0000","last_change":"2024-03-11T21:35:32.651731+0000","last_active":"2024-03-11T21:42:21.479085+0000","last_peered":"2024-03-11T21:42:21.479085+0000","last_clean":"2024-03-11T21:42:21.479085+0000","last_became_active":"2024-03-11T21:25:42.572673+0000","last_became_peered":"2024-03-11T21:25:42.572673+0000","last_unstale":"2024-03-11T21:42:21.479085+0000","last_undegraded":"2024-03-11T21:42:21.479085+0000","last_fullsized":"2024-03-11T21:42:21.479085+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:35:32.651652+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:41.544550+0000","last_clean_scrub_stamp":"2024-03-11T21:35:32.651652+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.9","version":"394'6428","reported_seq":6615,"reported_epoch":394,"state":"active+clean","last_fresh":"2024-03-11T21:42:26.906606+0000","last_change":"2024-03-11T21:41:32.409763+0000","last_active":"2024-03-11T21:42:26.906606+0000","last_peered":"2024-03-11T21:42:26.906606+0000","last_clean":"2024-03-11T21:42:26.906606+0000","last_became_active":"2024-03-11T21:41:28.824488+0000","last_became_peered":"2024-03-11T21:41:28.824488+0000","last_unstale":"2024-03-11T21:42:26.906606+0000","last_undegraded":"2024-03-11T21:42:26.906606+0000","last_fullsized":"2024-03-11T21:42:26.906606+0000","mapping_epoch":356,"log_start":"0'0","ondisk_log_start":"0'0","created":356,"last_epoch_clean":373,"parent":"0.0","parent_split_bits":0,"last_scrub":"376'5094","last_scrub_stamp":"2024-03-11T21:41:32.409608+0000","last_deep_scrub":"376'5094","last_deep_scrub_stamp":"2024-03-11T21:41:32.409608+0000","last_clean_scrub_stamp":"2024-03-11T21:41:32.409608+0000","objects_scrubbed":2132,"log_size":6428,"log_dups_size":0,"ondisk_log_size":6428,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":4,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:42:32.836673+0000","scrub_duration":3401,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":218038272,"num_objects":3327,"num_object_clones":0,"num_object_copies":9981,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3327,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1337,"num_write_kb":81024,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.5","version":"0'0","reported_seq":373,"reported_epoch":394,"state":"active+clean","last_fresh":"2024-03-11T21:42:25.533435+0000","last_change":"2024-03-11T21:41:12.380275+0000","last_active":"2024-03-11T21:42:25.533435+0000","last_peered":"2024-03-11T21:42:25.533435+0000","last_clean":"2024-03-11T21:42:25.533435+0000","last_became_active":"2024-03-11T21:36:35.069358+0000","last_became_peered":"2024-03-11T21:36:35.069358+0000","last_unstale":"2024-03-11T21:42:25.533435+0000","last_undegraded":"2024-03-11T21:42:25.533435+0000","last_fullsized":"2024-03-11T21:42:25.533435+0000","mapping_epoch":277,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":278,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:41:12.380019+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:43.559452+0000","last_clean_scrub_stamp":"2024-03-11T21:41:12.380019+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:42:39.666062+0000","scrub_duration":53,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,3],"acting":[7,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.0","version":"164'6","reported_seq":1724,"reported_epoch":392,"state":"active+clean","last_fresh":"2024-03-11T21:42:21.474188+0000","last_change":"2024-03-11T21:41:30.197407+0000","last_active":"2024-03-11T21:42:21.474188+0000","last_peered":"2024-03-11T21:42:21.474188+0000","last_clean":"2024-03-11T21:42:21.474188+0000","last_became_active":"2024-03-11T21:25:38.536890+0000","last_became_peered":"2024-03-11T21:25:38.536890+0000","last_unstale":"2024-03-11T21:42:21.474188+0000","last_undegraded":"2024-03-11T21:42:21.474188+0000","last_fullsized":"2024-03-11T21:42:21.474188+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"164'6","last_scrub_stamp":"2024-03-11T21:41:30.197276+0000","last_deep_scrub":"164'6","last_deep_scrub_stamp":"2024-03-11T21:32:17.597666+0000","last_clean_scrub_stamp":"2024-03-11T21:41:30.197276+0000","objects_scrubbed":1,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:42:45.804414+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0.000753559,"stat_sum":{"num_bytes":389,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":996,"num_read_kb":991,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,6],"acting":[1,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":724,"reported_epoch":392,"state":"active+clean","last_fresh":"2024-03-11T21:42:21.484885+0000","last_change":"2024-03-11T21:40:09.915304+0000","last_active":"2024-03-11T21:42:21.484885+0000","last_peered":"2024-03-11T21:42:21.484885+0000","last_clean":"2024-03-11T21:42:21.484885+0000","last_became_active":"2024-03-11T21:36:35.056966+0000","last_became_peered":"2024-03-11T21:36:35.056966+0000","last_unstale":"2024-03-11T21:42:21.484885+0000","last_undegraded":"2024-03-11T21:42:21.484885+0000","last_fullsized":"2024-03-11T21:42:21.484885+0000","mapping_epoch":277,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":278,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:40:09.915179+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:40:09.915179+0000","last_clean_scrub_stamp":"2024-03-11T21:40:09.915179+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.7","version":"260'22","reported_seq":753,"reported_epoch":392,"state":"active+clean","last_fresh":"2024-03-11T21:42:21.477527+0000","last_change":"2024-03-11T21:41:27.232567+0000","last_active":"2024-03-11T21:42:21.477527+0000","last_peered":"2024-03-11T21:42:21.477527+0000","last_clean":"2024-03-11T21:42:21.477527+0000","last_became_active":"2024-03-11T21:25:40.541769+0000","last_became_peered":"2024-03-11T21:25:40.541769+0000","last_unstale":"2024-03-11T21:42:21.477527+0000","last_undegraded":"2024-03-11T21:42:21.477527+0000","last_fullsized":"2024-03-11T21:42:21.477527+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"260'22","last_scrub_stamp":"2024-03-11T21:41:27.232403+0000","last_deep_scrub":"260'22","last_deep_scrub_stamp":"2024-03-11T21:41:27.232403+0000","last_clean_scrub_stamp":"2024-03-11T21:41:27.232403+0000","objects_scrubbed":5,"log_size":22,"log_dups_size":0,"ondisk_log_size":22,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:42:52.057690+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":36,"num_read_kb":25,"num_write":20,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":718,"reported_epoch":392,"state":"active+clean","last_fresh":"2024-03-11T21:42:21.485093+0000","last_change":"2024-03-11T21:41:04.111560+0000","last_active":"2024-03-11T21:42:21.485093+0000","last_peered":"2024-03-11T21:42:21.485093+0000","last_clean":"2024-03-11T21:42:21.485093+0000","last_became_active":"2024-03-11T21:34:41.143660+0000","last_became_peered":"2024-03-11T21:34:41.143660+0000","last_unstale":"2024-03-11T21:42:21.485093+0000","last_undegraded":"2024-03-11T21:42:21.485093+0000","last_fullsized":"2024-03-11T21:42:21.485093+0000","mapping_epoch":226,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":227,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:41:04.111481+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:41.544550+0000","last_clean_scrub_stamp":"2024-03-11T21:41:04.111481+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7],"acting":[2,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"10.b","version":"394'7057","reported_seq":7146,"reported_epoch":394,"state":"active+clean","last_fresh":"2024-03-11T21:42:27.371379+0000","last_change":"2024-03-11T21:40:39.241086+0000","last_active":"2024-03-11T21:42:27.371379+0000","last_peered":"2024-03-11T21:42:27.371379+0000","last_clean":"2024-03-11T21:42:27.371379+0000","last_became_active":"2024-03-11T21:40:39.240775+0000","last_became_peered":"2024-03-11T21:40:39.240775+0000","last_unstale":"2024-03-11T21:42:27.371379+0000","last_undegraded":"2024-03-11T21:42:27.371379+0000","last_fullsized":"2024-03-11T21:42:27.371379+0000","mapping_epoch":356,"log_start":"365'1500","ondisk_log_start":"365'1500","created":356,"last_epoch_clean":357,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:40:38.219602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:40:38.219602+0000","last_clean_scrub_stamp":"2024-03-11T21:40:38.219602+0000","objects_scrubbed":0,"log_size":5557,"log_dups_size":0,"ondisk_log_size":5557,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":440860672,"num_objects":6727,"num_object_clones":0,"num_object_copies":20181,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6727,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7057,"num_write_kb":441088,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,0],"acting":[6,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.7","version":"124'2","reported_seq":710,"reported_epoch":394,"state":"active+clean","last_fresh":"2024-03-11T21:42:25.503184+0000","last_change":"2024-03-11T21:40:31.722243+0000","last_active":"2024-03-11T21:42:25.503184+0000","last_peered":"2024-03-11T21:42:25.503184+0000","last_clean":"2024-03-11T21:42:25.503184+0000","last_became_active":"2024-03-11T21:36:35.058786+0000","last_became_peered":"2024-03-11T21:36:35.058786+0000","last_unstale":"2024-03-11T21:42:25.503184+0000","last_undegraded":"2024-03-11T21:42:25.503184+0000","last_fullsized":"2024-03-11T21:42:25.503184+0000","mapping_epoch":277,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":278,"parent":"0.0","parent_split_bits":0,"last_scrub":"124'2","last_scrub_stamp":"2024-03-11T21:40:31.722100+0000","last_deep_scrub":"124'2","last_deep_scrub_stamp":"2024-03-11T21:29:34.956908+0000","last_clean_scrub_stamp":"2024-03-11T21:40:31.722100+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":87,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,4],"acting":[5,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2","version":"71'2","reported_seq":667,"reported_epoch":392,"state":"active+clean","last_fresh":"2024-03-11T21:42:21.479699+0000","last_change":"2024-03-11T21:41:07.517286+0000","last_active":"2024-03-11T21:42:21.479699+0000","last_peered":"2024-03-11T21:42:21.479699+0000","last_clean":"2024-03-11T21:42:21.479699+0000","last_became_active":"2024-03-11T21:36:35.070783+0000","last_became_peered":"2024-03-11T21:36:35.070783+0000","last_unstale":"2024-03-11T21:42:21.479699+0000","last_undegraded":"2024-03-11T21:42:21.479699+0000","last_fullsized":"2024-03-11T21:42:21.479699+0000","mapping_epoch":277,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":278,"parent":"0.0","parent_split_bits":0,"last_scrub":"71'2","last_scrub_stamp":"2024-03-11T21:41:07.517154+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:37.512018+0000","last_clean_scrub_stamp":"2024-03-11T21:41:07.517154+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:42:32.692076+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0.0014206449999999999,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":738,"reported_epoch":394,"state":"active+clean","last_fresh":"2024-03-11T21:42:25.502561+0000","last_change":"2024-03-11T21:41:10.803049+0000","last_active":"2024-03-11T21:42:25.502561+0000","last_peered":"2024-03-11T21:42:25.502561+0000","last_clean":"2024-03-11T21:42:25.502561+0000","last_became_active":"2024-03-11T21:34:41.146672+0000","last_became_peered":"2024-03-11T21:34:41.146672+0000","last_unstale":"2024-03-11T21:42:25.502561+0000","last_undegraded":"2024-03-11T21:42:25.502561+0000","last_fullsized":"2024-03-11T21:42:25.502561+0000","mapping_epoch":226,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":227,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:41:10.802921+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:37.512018+0000","last_clean_scrub_stamp":"2024-03-11T21:41:10.802921+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"87'128","reported_seq":835,"reported_epoch":394,"state":"active+clean","last_fresh":"2024-03-11T21:42:25.532753+0000","last_change":"2024-03-11T21:41:09.328321+0000","last_active":"2024-03-11T21:42:25.532753+0000","last_peered":"2024-03-11T21:42:25.532753+0000","last_clean":"2024-03-11T21:42:25.532753+0000","last_became_active":"2024-03-11T21:34:44.164569+0000","last_became_peered":"2024-03-11T21:34:44.164569+0000","last_unstale":"2024-03-11T21:42:25.532753+0000","last_undegraded":"2024-03-11T21:42:25.532753+0000","last_fullsized":"2024-03-11T21:42:25.532753+0000","mapping_epoch":229,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":230,"parent":"0.0","parent_split_bits":0,"last_scrub":"87'128","last_scrub_stamp":"2024-03-11T21:41:09.328197+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:24.261974+0000","last_clean_scrub_stamp":"2024-03-11T21:41:09.328197+0000","objects_scrubbed":2,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:42:28.198429+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":198,"num_read_kb":294,"num_write":179,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":2951840,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.5","version":"260'28","reported_seq":975,"reported_epoch":394,"state":"active+clean","last_fresh":"2024-03-11T21:42:25.505014+0000","last_change":"2024-03-11T21:41:07.923860+0000","last_active":"2024-03-11T21:42:25.505014+0000","last_peered":"2024-03-11T21:42:25.505014+0000","last_clean":"2024-03-11T21:42:25.505014+0000","last_became_active":"2024-03-11T21:25:40.761457+0000","last_became_peered":"2024-03-11T21:25:40.761457+0000","last_unstale":"2024-03-11T21:42:25.505014+0000","last_undegraded":"2024-03-11T21:42:25.505014+0000","last_fullsized":"2024-03-11T21:42:25.505014+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"260'28","last_scrub_stamp":"2024-03-11T21:41:07.923729+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:39.525554+0000","last_clean_scrub_stamp":"2024-03-11T21:41:07.923729+0000","objects_scrubbed":8,"log_size":28,"log_dups_size":0,"ondisk_log_size":28,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":154,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":43,"num_read_kb":33,"num_write":25,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":364,"reported_epoch":394,"state":"active+clean","last_fresh":"2024-03-11T21:42:25.533508+0000","last_change":"2024-03-11T21:41:00.152627+0000","last_active":"2024-03-11T21:42:25.533508+0000","last_peered":"2024-03-11T21:42:25.533508+0000","last_clean":"2024-03-11T21:42:25.533508+0000","last_became_active":"2024-03-11T21:34:41.139782+0000","last_became_peered":"2024-03-11T21:34:41.139782+0000","last_unstale":"2024-03-11T21:42:25.533508+0000","last_undegraded":"2024-03-11T21:42:25.533508+0000","last_fullsized":"2024-03-11T21:42:25.533508+0000","mapping_epoch":226,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":227,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:41:00.152377+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:32:21.266672+0000","last_clean_scrub_stamp":"2024-03-11T21:41:00.152377+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:42:28.198395+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,5],"acting":[7,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.8","version":"392'5613","reported_seq":5732,"reported_epoch":392,"state":"active+clean","last_fresh":"2024-03-11T21:42:24.246834+0000","last_change":"2024-03-11T21:41:29.898309+0000","last_active":"2024-03-11T21:42:24.246834+0000","last_peered":"2024-03-11T21:42:24.246834+0000","last_clean":"2024-03-11T21:42:24.246834+0000","last_became_active":"2024-03-11T21:41:28.829878+0000","last_became_peered":"2024-03-11T21:41:28.829878+0000","last_unstale":"2024-03-11T21:42:24.246834+0000","last_undegraded":"2024-03-11T21:42:24.246834+0000","last_fullsized":"2024-03-11T21:42:24.246834+0000","mapping_epoch":356,"log_start":"0'0","ondisk_log_start":"0'0","created":356,"last_epoch_clean":373,"parent":"0.0","parent_split_bits":0,"last_scrub":"374'4392","last_scrub_stamp":"2024-03-11T21:41:29.898148+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:40:38.219602+0000","last_clean_scrub_stamp":"2024-03-11T21:41:29.898148+0000","objects_scrubbed":2198,"log_size":5613,"log_dups_size":0,"ondisk_log_size":5613,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:42:51.493233+0000","scrub_duration":722,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":223870976,"num_objects":3416,"num_object_clones":0,"num_object_copies":10248,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3416,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1240,"num_write_kb":78656,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,7],"acting":[0,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.4","version":"0'0","reported_seq":705,"reported_epoch":392,"state":"active+clean","last_fresh":"2024-03-11T21:42:21.477220+0000","last_change":"2024-03-11T21:41:01.134631+0000","last_active":"2024-03-11T21:42:21.477220+0000","last_peered":"2024-03-11T21:42:21.477220+0000","last_clean":"2024-03-11T21:42:21.477220+0000","last_became_active":"2024-03-11T21:36:35.072824+0000","last_became_peered":"2024-03-11T21:36:35.072824+0000","last_unstale":"2024-03-11T21:42:21.477220+0000","last_undegraded":"2024-03-11T21:42:21.477220+0000","last_fullsized":"2024-03-11T21:42:21.477220+0000","mapping_epoch":277,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":278,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:41:01.134504+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:43.559452+0000","last_clean_scrub_stamp":"2024-03-11T21:41:01.134504+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":733,"reported_epoch":392,"state":"active+clean","last_fresh":"2024-03-11T21:42:21.476023+0000","last_change":"2024-03-11T21:41:02.116795+0000","last_active":"2024-03-11T21:42:21.476023+0000","last_peered":"2024-03-11T21:42:21.476023+0000","last_clean":"2024-03-11T21:42:21.476023+0000","last_became_active":"2024-03-11T21:36:35.069452+0000","last_became_peered":"2024-03-11T21:36:35.069452+0000","last_unstale":"2024-03-11T21:42:21.476023+0000","last_undegraded":"2024-03-11T21:42:21.476023+0000","last_fullsized":"2024-03-11T21:42:21.476023+0000","mapping_epoch":277,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":278,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:41:02.116682+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:37.512018+0000","last_clean_scrub_stamp":"2024-03-11T21:41:02.116682+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0.00089342499999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":356,"reported_epoch":394,"state":"active+clean","last_fresh":"2024-03-11T21:42:25.532624+0000","last_change":"2024-03-11T21:40:27.117606+0000","last_active":"2024-03-11T21:42:25.532624+0000","last_peered":"2024-03-11T21:42:25.532624+0000","last_clean":"2024-03-11T21:42:25.532624+0000","last_became_active":"2024-03-11T21:34:41.149219+0000","last_became_peered":"2024-03-11T21:34:41.149219+0000","last_unstale":"2024-03-11T21:42:25.532624+0000","last_undegraded":"2024-03-11T21:42:25.532624+0000","last_fullsized":"2024-03-11T21:42:25.532624+0000","mapping_epoch":226,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":227,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:40:27.117515+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:37.512018+0000","last_clean_scrub_stamp":"2024-03-11T21:40:27.117515+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:42:28.198341+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,0],"acting":[7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.6","version":"352'30","reported_seq":774,"reported_epoch":392,"state":"active+clean","last_fresh":"2024-03-11T21:42:21.476302+0000","last_change":"2024-03-11T21:40:31.175417+0000","last_active":"2024-03-11T21:42:21.476302+0000","last_peered":"2024-03-11T21:42:21.476302+0000","last_clean":"2024-03-11T21:42:21.476302+0000","last_became_active":"2024-03-11T21:32:22.796486+0000","last_became_peered":"2024-03-11T21:32:22.796486+0000","last_unstale":"2024-03-11T21:42:21.476302+0000","last_undegraded":"2024-03-11T21:42:21.476302+0000","last_fullsized":"2024-03-11T21:42:21.476302+0000","mapping_epoch":166,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":167,"parent":"0.0","parent_split_bits":0,"last_scrub":"352'30","last_scrub_stamp":"2024-03-11T21:40:31.175323+0000","last_deep_scrub":"352'30","last_deep_scrub_stamp":"2024-03-11T21:40:31.175323+0000","last_clean_scrub_stamp":"2024-03-11T21:40:31.175323+0000","objects_scrubbed":6,"log_size":30,"log_dups_size":0,"ondisk_log_size":30,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":36,"num_read_kb":30,"num_write":24,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":703,"reported_epoch":394,"state":"active+clean","last_fresh":"2024-03-11T21:42:25.503090+0000","last_change":"2024-03-11T21:36:07.451482+0000","last_active":"2024-03-11T21:42:25.503090+0000","last_peered":"2024-03-11T21:42:25.503090+0000","last_clean":"2024-03-11T21:42:25.503090+0000","last_became_active":"2024-03-11T21:25:42.572324+0000","last_became_peered":"2024-03-11T21:25:42.572324+0000","last_unstale":"2024-03-11T21:42:25.503090+0000","last_undegraded":"2024-03-11T21:42:25.503090+0000","last_fullsized":"2024-03-11T21:42:25.503090+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:36:07.451102+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:41.544550+0000","last_clean_scrub_stamp":"2024-03-11T21:36:07.451102+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":54,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.d","version":"394'6993","reported_seq":7083,"reported_epoch":394,"state":"active+clean","last_fresh":"2024-03-11T21:42:27.377511+0000","last_change":"2024-03-11T21:40:39.242068+0000","last_active":"2024-03-11T21:42:27.377511+0000","last_peered":"2024-03-11T21:42:27.377511+0000","last_clean":"2024-03-11T21:42:27.377511+0000","last_became_active":"2024-03-11T21:40:39.241769+0000","last_became_peered":"2024-03-11T21:40:39.241769+0000","last_unstale":"2024-03-11T21:42:27.377511+0000","last_undegraded":"2024-03-11T21:42:27.377511+0000","last_fullsized":"2024-03-11T21:42:27.377511+0000","mapping_epoch":356,"log_start":"365'1400","ondisk_log_start":"365'1400","created":356,"last_epoch_clean":357,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:40:38.219602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:40:38.219602+0000","last_clean_scrub_stamp":"2024-03-11T21:40:38.219602+0000","objects_scrubbed":0,"log_size":5593,"log_dups_size":0,"ondisk_log_size":5593,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":437780504,"num_objects":6681,"num_object_clones":0,"num_object_copies":20043,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6681,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":6993,"num_write_kb":437505,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,0],"acting":[6,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":704,"reported_epoch":392,"state":"active+clean","last_fresh":"2024-03-11T21:42:21.473981+0000","last_change":"2024-03-11T21:36:18.529506+0000","last_active":"2024-03-11T21:42:21.473981+0000","last_peered":"2024-03-11T21:42:21.473981+0000","last_clean":"2024-03-11T21:42:21.473981+0000","last_became_active":"2024-03-11T21:25:45.301416+0000","last_became_peered":"2024-03-11T21:25:45.301416+0000","last_unstale":"2024-03-11T21:42:21.473981+0000","last_undegraded":"2024-03-11T21:42:21.473981+0000","last_fullsized":"2024-03-11T21:42:21.473981+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:36:18.529370+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:43.559452+0000","last_clean_scrub_stamp":"2024-03-11T21:36:18.529370+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":54,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":364,"reported_epoch":394,"state":"active+clean","last_fresh":"2024-03-11T21:42:25.534853+0000","last_change":"2024-03-11T21:41:19.303174+0000","last_active":"2024-03-11T21:42:25.534853+0000","last_peered":"2024-03-11T21:42:25.534853+0000","last_clean":"2024-03-11T21:42:25.534853+0000","last_became_active":"2024-03-11T21:34:41.152721+0000","last_became_peered":"2024-03-11T21:34:41.152721+0000","last_unstale":"2024-03-11T21:42:25.534853+0000","last_undegraded":"2024-03-11T21:42:25.534853+0000","last_fullsized":"2024-03-11T21:42:25.534853+0000","mapping_epoch":226,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":227,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:41:19.302991+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:26:57.333669+0000","last_clean_scrub_stamp":"2024-03-11T21:41:19.302991+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:42:31.475833+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.3","version":"260'37","reported_seq":820,"reported_epoch":392,"state":"active+clean","last_fresh":"2024-03-11T21:42:21.479937+0000","last_change":"2024-03-11T21:41:06.123397+0000","last_active":"2024-03-11T21:42:21.479937+0000","last_peered":"2024-03-11T21:42:21.479937+0000","last_clean":"2024-03-11T21:42:21.479937+0000","last_became_active":"2024-03-11T21:34:41.145607+0000","last_became_peered":"2024-03-11T21:34:41.145607+0000","last_unstale":"2024-03-11T21:42:21.479937+0000","last_undegraded":"2024-03-11T21:42:21.479937+0000","last_fullsized":"2024-03-11T21:42:21.479937+0000","mapping_epoch":226,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":227,"parent":"0.0","parent_split_bits":0,"last_scrub":"260'37","last_scrub_stamp":"2024-03-11T21:41:06.123055+0000","last_deep_scrub":"165'27","last_deep_scrub_stamp":"2024-03-11T21:32:21.576732+0000","last_clean_scrub_stamp":"2024-03-11T21:41:06.123055+0000","objects_scrubbed":8,"log_size":37,"log_dups_size":0,"ondisk_log_size":37,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":54,"num_read_kb":40,"num_write":32,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":660,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":737,"reported_epoch":394,"state":"active+clean","last_fresh":"2024-03-11T21:42:25.504704+0000","last_change":"2024-03-11T21:41:21.857353+0000","last_active":"2024-03-11T21:42:25.504704+0000","last_peered":"2024-03-11T21:42:25.504704+0000","last_clean":"2024-03-11T21:42:25.504704+0000","last_became_active":"2024-03-11T21:25:42.579045+0000","last_became_peered":"2024-03-11T21:25:42.579045+0000","last_unstale":"2024-03-11T21:42:25.504704+0000","last_undegraded":"2024-03-11T21:42:25.504704+0000","last_fullsized":"2024-03-11T21:42:25.504704+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:41:21.857205+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:41:21.857205+0000","last_clean_scrub_stamp":"2024-03-11T21:41:21.857205+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:42:41.525189+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"10.e","version":"394'6887","reported_seq":6976,"reported_epoch":394,"state":"active+clean","last_fresh":"2024-03-11T21:42:26.959388+0000","last_change":"2024-03-11T21:40:39.239887+0000","last_active":"2024-03-11T21:42:26.959388+0000","last_peered":"2024-03-11T21:42:26.959388+0000","last_clean":"2024-03-11T21:42:26.959388+0000","last_became_active":"2024-03-11T21:40:39.239590+0000","last_became_peered":"2024-03-11T21:40:39.239590+0000","last_unstale":"2024-03-11T21:42:26.959388+0000","last_undegraded":"2024-03-11T21:42:26.959388+0000","last_fullsized":"2024-03-11T21:42:26.959388+0000","mapping_epoch":356,"log_start":"366'2400","ondisk_log_start":"366'2400","created":356,"last_epoch_clean":357,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:40:38.219602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:40:38.219602+0000","last_clean_scrub_stamp":"2024-03-11T21:40:38.219602+0000","objects_scrubbed":0,"log_size":4487,"log_dups_size":0,"ondisk_log_size":4487,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":432603136,"num_objects":6601,"num_object_clones":0,"num_object_copies":19803,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6601,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6887,"num_write_kb":431616,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,2],"acting":[4,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":715,"reported_epoch":394,"state":"active+clean","last_fresh":"2024-03-11T21:42:25.536824+0000","last_change":"2024-03-11T21:40:28.582452+0000","last_active":"2024-03-11T21:42:25.536824+0000","last_peered":"2024-03-11T21:42:25.536824+0000","last_clean":"2024-03-11T21:42:25.536824+0000","last_became_active":"2024-03-11T21:32:22.791439+0000","last_became_peered":"2024-03-11T21:32:22.791439+0000","last_unstale":"2024-03-11T21:42:25.536824+0000","last_undegraded":"2024-03-11T21:42:25.536824+0000","last_fullsized":"2024-03-11T21:42:25.536824+0000","mapping_epoch":166,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":167,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:40:28.582339+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:43.559452+0000","last_clean_scrub_stamp":"2024-03-11T21:40:28.582339+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":20,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,2],"acting":[4,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.6","version":"67'1","reported_seq":717,"reported_epoch":392,"state":"active+clean","last_fresh":"2024-03-11T21:42:21.477486+0000","last_change":"2024-03-11T21:41:21.288511+0000","last_active":"2024-03-11T21:42:21.477486+0000","last_peered":"2024-03-11T21:42:21.477486+0000","last_clean":"2024-03-11T21:42:21.477486+0000","last_became_active":"2024-03-11T21:25:38.536142+0000","last_became_peered":"2024-03-11T21:25:38.536142+0000","last_unstale":"2024-03-11T21:42:21.477486+0000","last_undegraded":"2024-03-11T21:42:21.477486+0000","last_fullsized":"2024-03-11T21:42:21.477486+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'1","last_scrub_stamp":"2024-03-11T21:41:21.288031+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:37.512018+0000","last_clean_scrub_stamp":"2024-03-11T21:41:21.288031+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:42:32.623482+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,4],"acting":[1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.0","version":"260'38","reported_seq":715,"reported_epoch":392,"state":"active+clean","last_fresh":"2024-03-11T21:42:21.479584+0000","last_change":"2024-03-11T21:41:30.511287+0000","last_active":"2024-03-11T21:42:21.479584+0000","last_peered":"2024-03-11T21:42:21.479584+0000","last_clean":"2024-03-11T21:42:21.479584+0000","last_became_active":"2024-03-11T21:36:35.068071+0000","last_became_peered":"2024-03-11T21:36:35.068071+0000","last_unstale":"2024-03-11T21:42:21.479584+0000","last_undegraded":"2024-03-11T21:42:21.479584+0000","last_fullsized":"2024-03-11T21:42:21.479584+0000","mapping_epoch":277,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":278,"parent":"0.0","parent_split_bits":0,"last_scrub":"260'38","last_scrub_stamp":"2024-03-11T21:41:30.511168+0000","last_deep_scrub":"93'26","last_deep_scrub_stamp":"2024-03-11T21:26:40.680324+0000","last_clean_scrub_stamp":"2024-03-11T21:41:30.511168+0000","objects_scrubbed":8,"log_size":38,"log_dups_size":0,"ondisk_log_size":38,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:42:50.320052+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":51,"num_read_kb":39,"num_write":32,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":734,"reported_epoch":394,"state":"active+clean","last_fresh":"2024-03-11T21:42:25.537514+0000","last_change":"2024-03-11T21:40:21.518921+0000","last_active":"2024-03-11T21:42:25.537514+0000","last_peered":"2024-03-11T21:42:25.537514+0000","last_clean":"2024-03-11T21:42:25.537514+0000","last_became_active":"2024-03-11T21:36:35.057517+0000","last_became_peered":"2024-03-11T21:36:35.057517+0000","last_unstale":"2024-03-11T21:42:25.537514+0000","last_undegraded":"2024-03-11T21:42:25.537514+0000","last_fullsized":"2024-03-11T21:42:25.537514+0000","mapping_epoch":277,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":278,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:40:21.518840+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:40:21.518840+0000","last_clean_scrub_stamp":"2024-03-11T21:40:21.518840+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,7],"acting":[4,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"10.f","version":"394'7167","reported_seq":7256,"reported_epoch":394,"state":"active+clean","last_fresh":"2024-03-11T21:42:27.417547+0000","last_change":"2024-03-11T21:40:39.541796+0000","last_active":"2024-03-11T21:42:27.417547+0000","last_peered":"2024-03-11T21:42:27.417547+0000","last_clean":"2024-03-11T21:42:27.417547+0000","last_became_active":"2024-03-11T21:40:39.541420+0000","last_became_peered":"2024-03-11T21:40:39.541420+0000","last_unstale":"2024-03-11T21:42:27.417547+0000","last_undegraded":"2024-03-11T21:42:27.417547+0000","last_fullsized":"2024-03-11T21:42:27.417547+0000","mapping_epoch":356,"log_start":"365'1600","ondisk_log_start":"365'1600","created":356,"last_epoch_clean":357,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:40:38.219602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:40:38.219602+0000","last_clean_scrub_stamp":"2024-03-11T21:40:38.219602+0000","objects_scrubbed":0,"log_size":5567,"log_dups_size":0,"ondisk_log_size":5567,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":448069632,"num_objects":6837,"num_object_clones":0,"num_object_copies":20511,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6837,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7167,"num_write_kb":448128,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,2],"acting":[6,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.3","version":"0'0","reported_seq":356,"reported_epoch":394,"state":"active+clean","last_fresh":"2024-03-11T21:42:25.534382+0000","last_change":"2024-03-11T21:40:25.234980+0000","last_active":"2024-03-11T21:42:25.534382+0000","last_peered":"2024-03-11T21:42:25.534382+0000","last_clean":"2024-03-11T21:42:25.534382+0000","last_became_active":"2024-03-11T21:34:41.142683+0000","last_became_peered":"2024-03-11T21:34:41.142683+0000","last_unstale":"2024-03-11T21:42:25.534382+0000","last_undegraded":"2024-03-11T21:42:25.534382+0000","last_fullsized":"2024-03-11T21:42:25.534382+0000","mapping_epoch":226,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":227,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:40:25.213541+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:29:42.485740+0000","last_clean_scrub_stamp":"2024-03-11T21:40:25.213541+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:42:28.198308+0000","scrub_duration":55,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,2],"acting":[7,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":756,"reported_epoch":394,"state":"active+clean","last_fresh":"2024-03-11T21:42:25.505480+0000","last_change":"2024-03-11T21:41:22.817490+0000","last_active":"2024-03-11T21:42:25.505480+0000","last_peered":"2024-03-11T21:42:25.505480+0000","last_clean":"2024-03-11T21:42:25.505480+0000","last_became_active":"2024-03-11T21:34:41.145628+0000","last_became_peered":"2024-03-11T21:34:41.145628+0000","last_unstale":"2024-03-11T21:42:25.505480+0000","last_undegraded":"2024-03-11T21:42:25.505480+0000","last_fullsized":"2024-03-11T21:42:25.505480+0000","mapping_epoch":226,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":227,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:41:22.817380+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:37.512018+0000","last_clean_scrub_stamp":"2024-03-11T21:41:22.817380+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:42:48.113549+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,2],"acting":[6,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.1","version":"260'28","reported_seq":759,"reported_epoch":394,"state":"active+clean","last_fresh":"2024-03-11T21:42:25.536646+0000","last_change":"2024-03-11T21:41:07.513056+0000","last_active":"2024-03-11T21:42:25.536646+0000","last_peered":"2024-03-11T21:42:25.536646+0000","last_clean":"2024-03-11T21:42:25.536646+0000","last_became_active":"2024-03-11T21:25:40.546219+0000","last_became_peered":"2024-03-11T21:25:40.546219+0000","last_unstale":"2024-03-11T21:42:25.536646+0000","last_undegraded":"2024-03-11T21:42:25.536646+0000","last_fullsized":"2024-03-11T21:42:25.536646+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"260'28","last_scrub_stamp":"2024-03-11T21:41:07.512907+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:39.525554+0000","last_clean_scrub_stamp":"2024-03-11T21:41:07.512907+0000","objects_scrubbed":7,"log_size":28,"log_dups_size":0,"ondisk_log_size":28,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:42:29.651448+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":42,"num_read_kb":35,"num_write":28,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6],"acting":[4,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.0","version":"280'11","reported_seq":836,"reported_epoch":392,"state":"active+clean","last_fresh":"2024-03-11T21:42:21.479329+0000","last_change":"2024-03-11T21:36:35.064085+0000","last_active":"2024-03-11T21:42:21.479329+0000","last_peered":"2024-03-11T21:42:21.479329+0000","last_clean":"2024-03-11T21:42:21.479329+0000","last_became_active":"2024-03-11T21:36:35.055250+0000","last_became_peered":"2024-03-11T21:36:35.055250+0000","last_unstale":"2024-03-11T21:42:21.479329+0000","last_undegraded":"2024-03-11T21:42:21.479329+0000","last_fullsized":"2024-03-11T21:42:21.479329+0000","mapping_epoch":277,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":278,"parent":"0.0","parent_split_bits":0,"last_scrub":"98'9","last_scrub_stamp":"2024-03-11T21:35:21.143578+0000","last_deep_scrub":"98'9","last_deep_scrub_stamp":"2024-03-11T21:27:29.745711+0000","last_clean_scrub_stamp":"2024-03-11T21:35:21.143578+0000","objects_scrubbed":1,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.0","version":"394'5674","reported_seq":5798,"reported_epoch":394,"state":"active+clean","last_fresh":"2024-03-11T21:42:27.273942+0000","last_change":"2024-03-11T21:41:29.309475+0000","last_active":"2024-03-11T21:42:27.273942+0000","last_peered":"2024-03-11T21:42:27.273942+0000","last_clean":"2024-03-11T21:42:27.273942+0000","last_became_active":"2024-03-11T21:41:28.819914+0000","last_became_peered":"2024-03-11T21:41:28.819914+0000","last_unstale":"2024-03-11T21:42:27.273942+0000","last_undegraded":"2024-03-11T21:42:27.273942+0000","last_fullsized":"2024-03-11T21:42:27.273942+0000","mapping_epoch":356,"log_start":"0'0","ondisk_log_start":"0'0","created":356,"last_epoch_clean":373,"parent":"0.0","parent_split_bits":0,"last_scrub":"373'4340","last_scrub_stamp":"2024-03-11T21:41:29.309268+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:40:38.219602+0000","last_clean_scrub_stamp":"2024-03-11T21:41:29.309268+0000","objects_scrubbed":2123,"log_size":5674,"log_dups_size":0,"ondisk_log_size":5674,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:42:41.812280+0000","scrub_duration":481,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":218759168,"num_objects":3338,"num_object_clones":0,"num_object_copies":10014,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3338,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1341,"num_write_kb":81792,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,4],"acting":[6,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.c","version":"0'0","reported_seq":636,"reported_epoch":392,"state":"active+clean","last_fresh":"2024-03-11T21:42:21.486748+0000","last_change":"2024-03-11T21:40:30.659272+0000","last_active":"2024-03-11T21:42:21.486748+0000","last_peered":"2024-03-11T21:42:21.486748+0000","last_clean":"2024-03-11T21:42:21.486748+0000","last_became_active":"2024-03-11T21:36:35.073778+0000","last_became_peered":"2024-03-11T21:36:35.073778+0000","last_unstale":"2024-03-11T21:42:21.486748+0000","last_undegraded":"2024-03-11T21:42:21.486748+0000","last_fullsized":"2024-03-11T21:42:21.486748+0000","mapping_epoch":277,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":278,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:40:30.659160+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:43.559452+0000","last_clean_scrub_stamp":"2024-03-11T21:40:30.659160+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.f","version":"0'0","reported_seq":711,"reported_epoch":394,"state":"active+clean","last_fresh":"2024-03-11T21:42:25.502637+0000","last_change":"2024-03-11T21:36:08.439013+0000","last_active":"2024-03-11T21:42:25.502637+0000","last_peered":"2024-03-11T21:42:25.502637+0000","last_clean":"2024-03-11T21:42:25.502637+0000","last_became_active":"2024-03-11T21:25:42.575935+0000","last_became_peered":"2024-03-11T21:25:42.575935+0000","last_unstale":"2024-03-11T21:42:25.502637+0000","last_undegraded":"2024-03-11T21:42:25.502637+0000","last_fullsized":"2024-03-11T21:42:25.502637+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:36:08.438895+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:41.544550+0000","last_clean_scrub_stamp":"2024-03-11T21:36:08.438895+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.8","version":"0'0","reported_seq":364,"reported_epoch":394,"state":"active+clean","last_fresh":"2024-03-11T21:42:25.532917+0000","last_change":"2024-03-11T21:41:22.253981+0000","last_active":"2024-03-11T21:42:25.532917+0000","last_peered":"2024-03-11T21:42:25.532917+0000","last_clean":"2024-03-11T21:42:25.532917+0000","last_became_active":"2024-03-11T21:34:41.144681+0000","last_became_peered":"2024-03-11T21:34:41.144681+0000","last_unstale":"2024-03-11T21:42:25.532917+0000","last_undegraded":"2024-03-11T21:42:25.532917+0000","last_fullsized":"2024-03-11T21:42:25.532917+0000","mapping_epoch":226,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":227,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:41:22.253866+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:37.512018+0000","last_clean_scrub_stamp":"2024-03-11T21:41:22.253866+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,1],"acting":[7,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.e","version":"260'24","reported_seq":752,"reported_epoch":394,"state":"active+clean","last_fresh":"2024-03-11T21:42:25.537950+0000","last_change":"2024-03-11T21:41:14.567374+0000","last_active":"2024-03-11T21:42:25.537950+0000","last_peered":"2024-03-11T21:42:25.537950+0000","last_clean":"2024-03-11T21:42:25.537950+0000","last_became_active":"2024-03-11T21:25:40.552178+0000","last_became_peered":"2024-03-11T21:25:40.552178+0000","last_unstale":"2024-03-11T21:42:25.537950+0000","last_undegraded":"2024-03-11T21:42:25.537950+0000","last_fullsized":"2024-03-11T21:42:25.537950+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"260'24","last_scrub_stamp":"2024-03-11T21:41:14.567244+0000","last_deep_scrub":"109'16","last_deep_scrub_stamp":"2024-03-11T21:28:24.739395+0000","last_clean_scrub_stamp":"2024-03-11T21:41:14.567244+0000","objects_scrubbed":5,"log_size":24,"log_dups_size":0,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":32,"num_read_kb":25,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"10.1","version":"394'5715","reported_seq":5841,"reported_epoch":394,"state":"active+clean","last_fresh":"2024-03-11T21:42:26.917670+0000","last_change":"2024-03-11T21:41:30.222240+0000","last_active":"2024-03-11T21:42:26.917670+0000","last_peered":"2024-03-11T21:42:26.917670+0000","last_clean":"2024-03-11T21:42:26.917670+0000","last_became_active":"2024-03-11T21:41:28.822620+0000","last_became_peered":"2024-03-11T21:41:28.822620+0000","last_unstale":"2024-03-11T21:42:26.917670+0000","last_undegraded":"2024-03-11T21:42:26.917670+0000","last_fullsized":"2024-03-11T21:42:26.917670+0000","mapping_epoch":356,"log_start":"0'0","ondisk_log_start":"0'0","created":356,"last_epoch_clean":373,"parent":"0.0","parent_split_bits":0,"last_scrub":"374'4413","last_scrub_stamp":"2024-03-11T21:41:30.221933+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:40:38.219602+0000","last_clean_scrub_stamp":"2024-03-11T21:41:30.221933+0000","objects_scrubbed":2282,"log_size":5715,"log_dups_size":0,"ondisk_log_size":5715,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:42:37.031785+0000","scrub_duration":640,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":226426880,"num_objects":3455,"num_object_clones":0,"num_object_copies":10365,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3455,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1303,"num_write_kb":79232,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,0],"acting":[4,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.d","version":"0'0","reported_seq":707,"reported_epoch":394,"state":"active+clean","last_fresh":"2024-03-11T21:42:25.502768+0000","last_change":"2024-03-11T21:41:15.915172+0000","last_active":"2024-03-11T21:42:25.502768+0000","last_peered":"2024-03-11T21:42:25.502768+0000","last_clean":"2024-03-11T21:42:25.502768+0000","last_became_active":"2024-03-11T21:25:44.590333+0000","last_became_peered":"2024-03-11T21:25:44.590333+0000","last_unstale":"2024-03-11T21:42:25.502768+0000","last_undegraded":"2024-03-11T21:42:25.502768+0000","last_fullsized":"2024-03-11T21:42:25.502768+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:41:15.915041+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:35:37.182850+0000","last_clean_scrub_stamp":"2024-03-11T21:41:15.915041+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:42:36.590722+0000","scrub_duration":117,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.e","version":"74'8","reported_seq":922,"reported_epoch":394,"state":"active+clean","last_fresh":"2024-03-11T21:42:25.538350+0000","last_change":"2024-03-11T21:41:16.574147+0000","last_active":"2024-03-11T21:42:25.538350+0000","last_peered":"2024-03-11T21:42:25.538350+0000","last_clean":"2024-03-11T21:42:25.538350+0000","last_became_active":"2024-03-11T21:25:42.576742+0000","last_became_peered":"2024-03-11T21:25:42.576742+0000","last_unstale":"2024-03-11T21:42:25.538350+0000","last_undegraded":"2024-03-11T21:42:25.538350+0000","last_fullsized":"2024-03-11T21:42:25.538350+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'8","last_scrub_stamp":"2024-03-11T21:41:16.573269+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:41.544550+0000","last_clean_scrub_stamp":"2024-03-11T21:41:16.573269+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:42:34.081922+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,0],"acting":[4,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.9","version":"0'0","reported_seq":729,"reported_epoch":392,"state":"active+clean","last_fresh":"2024-03-11T21:42:21.473796+0000","last_change":"2024-03-11T21:40:29.915912+0000","last_active":"2024-03-11T21:42:21.473796+0000","last_peered":"2024-03-11T21:42:21.473796+0000","last_clean":"2024-03-11T21:42:21.473796+0000","last_became_active":"2024-03-11T21:36:35.065733+0000","last_became_peered":"2024-03-11T21:36:35.065733+0000","last_unstale":"2024-03-11T21:42:21.473796+0000","last_undegraded":"2024-03-11T21:42:21.473796+0000","last_fullsized":"2024-03-11T21:42:21.473796+0000","mapping_epoch":277,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":278,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:40:29.915835+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:26:50.991712+0000","last_clean_scrub_stamp":"2024-03-11T21:40:29.915835+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,3],"acting":[1,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.f","version":"260'27","reported_seq":738,"reported_epoch":392,"state":"active+clean","last_fresh":"2024-03-11T21:42:21.485231+0000","last_change":"2024-03-11T21:41:31.414618+0000","last_active":"2024-03-11T21:42:21.485231+0000","last_peered":"2024-03-11T21:42:21.485231+0000","last_clean":"2024-03-11T21:42:21.485231+0000","last_became_active":"2024-03-11T21:36:35.058272+0000","last_became_peered":"2024-03-11T21:36:35.058272+0000","last_unstale":"2024-03-11T21:42:21.485231+0000","last_undegraded":"2024-03-11T21:42:21.485231+0000","last_fullsized":"2024-03-11T21:42:21.485231+0000","mapping_epoch":277,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":278,"parent":"0.0","parent_split_bits":0,"last_scrub":"260'27","last_scrub_stamp":"2024-03-11T21:41:31.414510+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:39.525554+0000","last_clean_scrub_stamp":"2024-03-11T21:41:31.414510+0000","objects_scrubbed":7,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:42:52.884588+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":44,"num_read_kb":35,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,4],"acting":[2,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"10.2","version":"392'5721","reported_seq":5840,"reported_epoch":392,"state":"active+clean","last_fresh":"2024-03-11T21:42:22.753598+0000","last_change":"2024-03-11T21:41:29.850911+0000","last_active":"2024-03-11T21:42:22.753598+0000","last_peered":"2024-03-11T21:42:22.753598+0000","last_clean":"2024-03-11T21:42:22.753598+0000","last_became_active":"2024-03-11T21:41:28.834067+0000","last_became_peered":"2024-03-11T21:41:28.834067+0000","last_unstale":"2024-03-11T21:42:22.753598+0000","last_undegraded":"2024-03-11T21:42:22.753598+0000","last_fullsized":"2024-03-11T21:42:22.753598+0000","mapping_epoch":356,"log_start":"0'0","ondisk_log_start":"0'0","created":356,"last_epoch_clean":373,"parent":"0.0","parent_split_bits":0,"last_scrub":"373'4444","last_scrub_stamp":"2024-03-11T21:41:29.850283+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:40:38.219602+0000","last_clean_scrub_stamp":"2024-03-11T21:41:29.850283+0000","objects_scrubbed":2169,"log_size":5721,"log_dups_size":0,"ondisk_log_size":5721,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:42:47.523061+0000","scrub_duration":620,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":226295808,"num_objects":3453,"num_object_clones":0,"num_object_copies":10359,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3453,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1284,"num_write_kb":82176,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.e","version":"0'0","reported_seq":708,"reported_epoch":394,"state":"active+clean","last_fresh":"2024-03-11T21:42:25.538442+0000","last_change":"2024-03-11T21:41:20.548370+0000","last_active":"2024-03-11T21:42:25.538442+0000","last_peered":"2024-03-11T21:42:25.538442+0000","last_clean":"2024-03-11T21:42:25.538442+0000","last_became_active":"2024-03-11T21:25:44.590198+0000","last_became_peered":"2024-03-11T21:25:44.590198+0000","last_unstale":"2024-03-11T21:42:25.538442+0000","last_undegraded":"2024-03-11T21:42:25.538442+0000","last_fullsized":"2024-03-11T21:42:25.538442+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:41:20.548259+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:43.559452+0000","last_clean_scrub_stamp":"2024-03-11T21:41:20.548259+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:42:47.105654+0000","scrub_duration":42,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.d","version":"109'9","reported_seq":916,"reported_epoch":392,"state":"active+clean","last_fresh":"2024-03-11T21:42:21.484681+0000","last_change":"2024-03-11T21:41:06.117858+0000","last_active":"2024-03-11T21:42:21.484681+0000","last_peered":"2024-03-11T21:42:21.484681+0000","last_clean":"2024-03-11T21:42:21.484681+0000","last_became_active":"2024-03-11T21:25:42.575583+0000","last_became_peered":"2024-03-11T21:25:42.575583+0000","last_unstale":"2024-03-11T21:42:21.484681+0000","last_undegraded":"2024-03-11T21:42:21.484681+0000","last_fullsized":"2024-03-11T21:42:21.484681+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"109'9","last_scrub_stamp":"2024-03-11T21:41:06.117483+0000","last_deep_scrub":"109'9","last_deep_scrub_stamp":"2024-03-11T21:28:27.082975+0000","last_clean_scrub_stamp":"2024-03-11T21:41:06.117483+0000","objects_scrubbed":1,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,5],"acting":[2,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.a","version":"0'0","reported_seq":737,"reported_epoch":392,"state":"active+clean","last_fresh":"2024-03-11T21:42:21.474530+0000","last_change":"2024-03-11T21:41:02.132633+0000","last_active":"2024-03-11T21:42:21.474530+0000","last_peered":"2024-03-11T21:42:21.474530+0000","last_clean":"2024-03-11T21:42:21.474530+0000","last_became_active":"2024-03-11T21:36:35.066832+0000","last_became_peered":"2024-03-11T21:36:35.066832+0000","last_unstale":"2024-03-11T21:42:21.474530+0000","last_undegraded":"2024-03-11T21:42:21.474530+0000","last_fullsized":"2024-03-11T21:42:21.474530+0000","mapping_epoch":277,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":278,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:41:02.132355+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:37.512018+0000","last_clean_scrub_stamp":"2024-03-11T21:41:02.132355+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:42:27.586284+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,7],"acting":[1,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.c","version":"260'20","reported_seq":768,"reported_epoch":394,"state":"active+clean","last_fresh":"2024-03-11T21:42:25.537482+0000","last_change":"2024-03-11T21:41:30.618505+0000","last_active":"2024-03-11T21:42:25.537482+0000","last_peered":"2024-03-11T21:42:25.537482+0000","last_clean":"2024-03-11T21:42:25.537482+0000","last_became_active":"2024-03-11T21:36:35.067269+0000","last_became_peered":"2024-03-11T21:36:35.067269+0000","last_unstale":"2024-03-11T21:42:25.537482+0000","last_undegraded":"2024-03-11T21:42:25.537482+0000","last_fullsized":"2024-03-11T21:42:25.537482+0000","mapping_epoch":277,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":278,"parent":"0.0","parent_split_bits":0,"last_scrub":"260'20","last_scrub_stamp":"2024-03-11T21:41:30.618393+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:39.525554+0000","last_clean_scrub_stamp":"2024-03-11T21:41:30.618393+0000","objects_scrubbed":5,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:42:49.510883+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":25,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"10.3","version":"392'5530","reported_seq":5653,"reported_epoch":392,"state":"active+clean","last_fresh":"2024-03-11T21:42:22.482255+0000","last_change":"2024-03-11T21:41:30.759270+0000","last_active":"2024-03-11T21:42:22.482255+0000","last_peered":"2024-03-11T21:42:22.482255+0000","last_clean":"2024-03-11T21:42:22.482255+0000","last_became_active":"2024-03-11T21:41:28.840374+0000","last_became_peered":"2024-03-11T21:41:28.840374+0000","last_unstale":"2024-03-11T21:42:22.482255+0000","last_undegraded":"2024-03-11T21:42:22.482255+0000","last_fullsized":"2024-03-11T21:42:22.482255+0000","mapping_epoch":356,"log_start":"0'0","ondisk_log_start":"0'0","created":356,"last_epoch_clean":373,"parent":"0.0","parent_split_bits":0,"last_scrub":"373'4357","last_scrub_stamp":"2024-03-11T21:41:30.759188+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:40:38.219602+0000","last_clean_scrub_stamp":"2024-03-11T21:41:30.759188+0000","objects_scrubbed":2208,"log_size":5530,"log_dups_size":0,"ondisk_log_size":5530,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:42:47.730696+0000","scrub_duration":314,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":221577216,"num_objects":3381,"num_object_clones":0,"num_object_copies":10143,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3381,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1173,"num_write_kb":75072,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,3],"acting":[2,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.f","version":"0'0","reported_seq":711,"reported_epoch":392,"state":"active+clean","last_fresh":"2024-03-11T21:42:21.484949+0000","last_change":"2024-03-11T21:41:17.377926+0000","last_active":"2024-03-11T21:42:21.484949+0000","last_peered":"2024-03-11T21:42:21.484949+0000","last_clean":"2024-03-11T21:42:21.484949+0000","last_became_active":"2024-03-11T21:36:35.057800+0000","last_became_peered":"2024-03-11T21:36:35.057800+0000","last_unstale":"2024-03-11T21:42:21.484949+0000","last_undegraded":"2024-03-11T21:42:21.484949+0000","last_fullsized":"2024-03-11T21:42:21.484949+0000","mapping_epoch":277,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":278,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:41:17.377779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:27:03.010969+0000","last_clean_scrub_stamp":"2024-03-11T21:41:17.377779+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:42:34.848520+0000","scrub_duration":149,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,4],"acting":[2,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":708,"reported_epoch":392,"state":"active+clean","last_fresh":"2024-03-11T21:42:21.477133+0000","last_change":"2024-03-11T21:41:31.221210+0000","last_active":"2024-03-11T21:42:21.477133+0000","last_peered":"2024-03-11T21:42:21.477133+0000","last_clean":"2024-03-11T21:42:21.477133+0000","last_became_active":"2024-03-11T21:25:42.579409+0000","last_became_peered":"2024-03-11T21:25:42.579409+0000","last_unstale":"2024-03-11T21:42:21.477133+0000","last_undegraded":"2024-03-11T21:42:21.477133+0000","last_fullsized":"2024-03-11T21:42:21.477133+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:41:31.221094+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:30:22.750053+0000","last_clean_scrub_stamp":"2024-03-11T21:41:31.221094+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:42:45.147214+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.b","version":"0'0","reported_seq":364,"reported_epoch":394,"state":"active+clean","last_fresh":"2024-03-11T21:42:25.534439+0000","last_change":"2024-03-11T21:41:29.255398+0000","last_active":"2024-03-11T21:42:25.534439+0000","last_peered":"2024-03-11T21:42:25.534439+0000","last_clean":"2024-03-11T21:42:25.534439+0000","last_became_active":"2024-03-11T21:34:41.134987+0000","last_became_peered":"2024-03-11T21:34:41.134987+0000","last_unstale":"2024-03-11T21:42:25.534439+0000","last_undegraded":"2024-03-11T21:42:25.534439+0000","last_fullsized":"2024-03-11T21:42:25.534439+0000","mapping_epoch":226,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":227,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:41:29.255282+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:36:16.719309+0000","last_clean_scrub_stamp":"2024-03-11T21:41:29.255282+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:42:49.733610+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,5],"acting":[7,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.d","version":"260'39","reported_seq":773,"reported_epoch":394,"state":"active+clean","last_fresh":"2024-03-11T21:42:25.537516+0000","last_change":"2024-03-11T21:41:11.544772+0000","last_active":"2024-03-11T21:42:25.537516+0000","last_peered":"2024-03-11T21:42:25.537516+0000","last_clean":"2024-03-11T21:42:25.537516+0000","last_became_active":"2024-03-11T21:25:40.553454+0000","last_became_peered":"2024-03-11T21:25:40.553454+0000","last_unstale":"2024-03-11T21:42:25.537516+0000","last_undegraded":"2024-03-11T21:42:25.537516+0000","last_fullsized":"2024-03-11T21:42:25.537516+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"260'39","last_scrub_stamp":"2024-03-11T21:41:11.544605+0000","last_deep_scrub":"165'25","last_deep_scrub_stamp":"2024-03-11T21:32:21.678328+0000","last_clean_scrub_stamp":"2024-03-11T21:41:11.544605+0000","objects_scrubbed":8,"log_size":39,"log_dups_size":0,"ondisk_log_size":39,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":50,"num_read_kb":40,"num_write":32,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,1],"acting":[4,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"10.4","version":"392'5451","reported_seq":5569,"reported_epoch":392,"state":"active+clean","last_fresh":"2024-03-11T21:42:24.244569+0000","last_change":"2024-03-11T21:41:30.436753+0000","last_active":"2024-03-11T21:42:24.244569+0000","last_peered":"2024-03-11T21:42:24.244569+0000","last_clean":"2024-03-11T21:42:24.244569+0000","last_became_active":"2024-03-11T21:41:28.837095+0000","last_became_peered":"2024-03-11T21:41:28.837095+0000","last_unstale":"2024-03-11T21:42:24.244569+0000","last_undegraded":"2024-03-11T21:42:24.244569+0000","last_fullsized":"2024-03-11T21:42:24.244569+0000","mapping_epoch":356,"log_start":"0'0","ondisk_log_start":"0'0","created":356,"last_epoch_clean":373,"parent":"0.0","parent_split_bits":0,"last_scrub":"374'4241","last_scrub_stamp":"2024-03-11T21:41:30.436548+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:40:38.219602+0000","last_clean_scrub_stamp":"2024-03-11T21:41:30.436548+0000","objects_scrubbed":2083,"log_size":5451,"log_dups_size":0,"ondisk_log_size":5451,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:42:50.828058+0000","scrub_duration":281,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":214499328,"num_objects":3273,"num_object_clones":0,"num_object_copies":9819,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3273,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1210,"num_write_kb":76800,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":364,"reported_epoch":394,"state":"active+clean","last_fresh":"2024-03-11T21:42:25.532775+0000","last_change":"2024-03-11T21:41:26.263600+0000","last_active":"2024-03-11T21:42:25.532775+0000","last_peered":"2024-03-11T21:42:25.532775+0000","last_clean":"2024-03-11T21:42:25.532775+0000","last_became_active":"2024-03-11T21:34:41.145012+0000","last_became_peered":"2024-03-11T21:34:41.145012+0000","last_unstale":"2024-03-11T21:42:25.532775+0000","last_undegraded":"2024-03-11T21:42:25.532775+0000","last_fullsized":"2024-03-11T21:42:25.532775+0000","mapping_epoch":226,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":227,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:41:26.263517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:28:37.561826+0000","last_clean_scrub_stamp":"2024-03-11T21:41:26.263517+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:42:42.924528+0000","scrub_duration":48,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,0],"acting":[7,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":706,"reported_epoch":392,"state":"active+clean","last_fresh":"2024-03-11T21:42:21.486393+0000","last_change":"2024-03-11T21:41:09.181704+0000","last_active":"2024-03-11T21:42:21.486393+0000","last_peered":"2024-03-11T21:42:21.486393+0000","last_clean":"2024-03-11T21:42:21.486393+0000","last_became_active":"2024-03-11T21:25:42.574515+0000","last_became_peered":"2024-03-11T21:25:42.574515+0000","last_unstale":"2024-03-11T21:42:21.486393+0000","last_undegraded":"2024-03-11T21:42:21.486393+0000","last_fullsized":"2024-03-11T21:42:21.486393+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:41:09.181567+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:41.544550+0000","last_clean_scrub_stamp":"2024-03-11T21:41:09.181567+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:42:23.605325+0000","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,5],"acting":[2,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.c","version":"0'0","reported_seq":714,"reported_epoch":392,"state":"active+clean","last_fresh":"2024-03-11T21:42:21.487095+0000","last_change":"2024-03-11T21:41:24.303833+0000","last_active":"2024-03-11T21:42:21.487095+0000","last_peered":"2024-03-11T21:42:21.487095+0000","last_clean":"2024-03-11T21:42:21.487095+0000","last_became_active":"2024-03-11T21:25:38.533240+0000","last_became_peered":"2024-03-11T21:25:38.533240+0000","last_unstale":"2024-03-11T21:42:21.487095+0000","last_undegraded":"2024-03-11T21:42:21.487095+0000","last_fullsized":"2024-03-11T21:42:21.487095+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:41:24.303681+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:29:30.279910+0000","last_clean_scrub_stamp":"2024-03-11T21:41:24.303681+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:42:48.090725+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,0],"acting":[2,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.a","version":"260'44","reported_seq":848,"reported_epoch":394,"state":"active+clean","last_fresh":"2024-03-11T21:42:25.504529+0000","last_change":"2024-03-11T21:40:02.183543+0000","last_active":"2024-03-11T21:42:25.504529+0000","last_peered":"2024-03-11T21:42:25.504529+0000","last_clean":"2024-03-11T21:42:25.504529+0000","last_became_active":"2024-03-11T21:34:41.148863+0000","last_became_peered":"2024-03-11T21:34:41.148863+0000","last_unstale":"2024-03-11T21:42:25.504529+0000","last_undegraded":"2024-03-11T21:42:25.504529+0000","last_fullsized":"2024-03-11T21:42:25.504529+0000","mapping_epoch":226,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":227,"parent":"0.0","parent_split_bits":0,"last_scrub":"260'44","last_scrub_stamp":"2024-03-11T21:40:02.183417+0000","last_deep_scrub":"109'28","last_deep_scrub_stamp":"2024-03-11T21:28:24.016389+0000","last_clean_scrub_stamp":"2024-03-11T21:40:02.183417+0000","objects_scrubbed":9,"log_size":44,"log_dups_size":0,"ondisk_log_size":44,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":56,"num_read_kb":45,"num_write":36,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,7],"acting":[6,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"10.5","version":"392'6513","reported_seq":6701,"reported_epoch":392,"state":"active+clean","last_fresh":"2024-03-11T21:42:22.530253+0000","last_change":"2024-03-11T21:41:32.489861+0000","last_active":"2024-03-11T21:42:22.530253+0000","last_peered":"2024-03-11T21:42:22.530253+0000","last_clean":"2024-03-11T21:42:22.530253+0000","last_became_active":"2024-03-11T21:41:28.844869+0000","last_became_peered":"2024-03-11T21:41:28.844869+0000","last_unstale":"2024-03-11T21:42:22.530253+0000","last_undegraded":"2024-03-11T21:42:22.530253+0000","last_fullsized":"2024-03-11T21:42:22.530253+0000","mapping_epoch":356,"log_start":"0'0","ondisk_log_start":"0'0","created":356,"last_epoch_clean":373,"parent":"0.0","parent_split_bits":0,"last_scrub":"376'5260","last_scrub_stamp":"2024-03-11T21:41:32.489702+0000","last_deep_scrub":"376'5260","last_deep_scrub_stamp":"2024-03-11T21:41:32.489702+0000","last_clean_scrub_stamp":"2024-03-11T21:41:32.489702+0000","objects_scrubbed":2205,"log_size":6513,"log_dups_size":0,"ondisk_log_size":6513,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":4,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:42:49.329953+0000","scrub_duration":3023,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":226820096,"num_objects":3461,"num_object_clones":0,"num_object_copies":10383,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3461,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1256,"num_write_kb":80384,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3],"acting":[2,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":716,"reported_epoch":392,"state":"active+clean","last_fresh":"2024-03-11T21:42:21.477425+0000","last_change":"2024-03-11T21:41:23.288877+0000","last_active":"2024-03-11T21:42:21.477425+0000","last_peered":"2024-03-11T21:42:21.477425+0000","last_clean":"2024-03-11T21:42:21.477425+0000","last_became_active":"2024-03-11T21:34:41.154507+0000","last_became_peered":"2024-03-11T21:34:41.154507+0000","last_unstale":"2024-03-11T21:42:21.477425+0000","last_undegraded":"2024-03-11T21:42:21.477425+0000","last_fullsized":"2024-03-11T21:42:21.477425+0000","mapping_epoch":226,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":227,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:41:23.288651+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:27:11.361913+0000","last_clean_scrub_stamp":"2024-03-11T21:41:23.288651+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:42:25.034214+0000","scrub_duration":136,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":707,"reported_epoch":392,"state":"active+clean","last_fresh":"2024-03-11T21:42:21.484235+0000","last_change":"2024-03-11T21:40:28.015350+0000","last_active":"2024-03-11T21:42:21.484235+0000","last_peered":"2024-03-11T21:42:21.484235+0000","last_clean":"2024-03-11T21:42:21.484235+0000","last_became_active":"2024-03-11T21:36:35.059192+0000","last_became_peered":"2024-03-11T21:36:35.059192+0000","last_unstale":"2024-03-11T21:42:21.484235+0000","last_undegraded":"2024-03-11T21:42:21.484235+0000","last_fullsized":"2024-03-11T21:42:21.484235+0000","mapping_epoch":277,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":278,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:40:28.015231+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:41.544550+0000","last_clean_scrub_stamp":"2024-03-11T21:40:28.015231+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,3],"acting":[2,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.d","version":"0'0","reported_seq":717,"reported_epoch":392,"state":"active+clean","last_fresh":"2024-03-11T21:42:21.477050+0000","last_change":"2024-03-11T21:40:31.873781+0000","last_active":"2024-03-11T21:42:21.477050+0000","last_peered":"2024-03-11T21:42:21.477050+0000","last_clean":"2024-03-11T21:42:21.477050+0000","last_became_active":"2024-03-11T21:36:35.058296+0000","last_became_peered":"2024-03-11T21:36:35.058296+0000","last_unstale":"2024-03-11T21:42:21.477050+0000","last_undegraded":"2024-03-11T21:42:21.477050+0000","last_fullsized":"2024-03-11T21:42:21.477050+0000","mapping_epoch":277,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":278,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:40:31.873695+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:37.512018+0000","last_clean_scrub_stamp":"2024-03-11T21:40:31.873695+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,3],"acting":[1,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.b","version":"260'15","reported_seq":729,"reported_epoch":392,"state":"active+clean","last_fresh":"2024-03-11T21:42:21.478400+0000","last_change":"2024-03-11T21:35:27.760318+0000","last_active":"2024-03-11T21:42:21.478400+0000","last_peered":"2024-03-11T21:42:21.478400+0000","last_clean":"2024-03-11T21:42:21.478400+0000","last_became_active":"2024-03-11T21:25:40.549612+0000","last_became_peered":"2024-03-11T21:25:40.549612+0000","last_unstale":"2024-03-11T21:42:21.478400+0000","last_undegraded":"2024-03-11T21:42:21.478400+0000","last_fullsized":"2024-03-11T21:42:21.478400+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'9","last_scrub_stamp":"2024-03-11T21:35:27.760208+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:39.525554+0000","last_clean_scrub_stamp":"2024-03-11T21:35:27.760208+0000","objects_scrubbed":4,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.6","version":"394'5700","reported_seq":5824,"reported_epoch":394,"state":"active+clean","last_fresh":"2024-03-11T21:42:26.970883+0000","last_change":"2024-03-11T21:41:30.231854+0000","last_active":"2024-03-11T21:42:26.970883+0000","last_peered":"2024-03-11T21:42:26.970883+0000","last_clean":"2024-03-11T21:42:26.970883+0000","last_became_active":"2024-03-11T21:41:28.841669+0000","last_became_peered":"2024-03-11T21:41:28.841669+0000","last_unstale":"2024-03-11T21:42:26.970883+0000","last_undegraded":"2024-03-11T21:42:26.970883+0000","last_fullsized":"2024-03-11T21:42:26.970883+0000","mapping_epoch":356,"log_start":"0'0","ondisk_log_start":"0'0","created":356,"last_epoch_clean":373,"parent":"0.0","parent_split_bits":0,"last_scrub":"374'4344","last_scrub_stamp":"2024-03-11T21:41:30.231749+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:40:38.219602+0000","last_clean_scrub_stamp":"2024-03-11T21:41:30.231749+0000","objects_scrubbed":2205,"log_size":5700,"log_dups_size":0,"ondisk_log_size":5700,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:42:58.466055+0000","scrub_duration":259,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":221052928,"num_objects":3373,"num_object_clones":0,"num_object_copies":10119,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3373,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1356,"num_write_kb":80768,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,4],"acting":[5,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":622,"reported_epoch":394,"state":"active+clean","last_fresh":"2024-03-11T21:42:25.536222+0000","last_change":"2024-03-11T21:41:12.624195+0000","last_active":"2024-03-11T21:42:25.536222+0000","last_peered":"2024-03-11T21:42:25.536222+0000","last_clean":"2024-03-11T21:42:25.536222+0000","last_became_active":"2024-03-11T21:36:35.056518+0000","last_became_peered":"2024-03-11T21:36:35.056518+0000","last_unstale":"2024-03-11T21:42:25.536222+0000","last_undegraded":"2024-03-11T21:42:25.536222+0000","last_fullsized":"2024-03-11T21:42:25.536222+0000","mapping_epoch":277,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":278,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:41:12.624073+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:43.559452+0000","last_clean_scrub_stamp":"2024-03-11T21:41:12.624073+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:42:36.867182+0000","scrub_duration":41,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,3],"acting":[4,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.e","version":"0'0","reported_seq":743,"reported_epoch":392,"state":"active+clean","last_fresh":"2024-03-11T21:42:21.484895+0000","last_change":"2024-03-11T21:41:27.388270+0000","last_active":"2024-03-11T21:42:21.484895+0000","last_peered":"2024-03-11T21:42:21.484895+0000","last_clean":"2024-03-11T21:42:21.484895+0000","last_became_active":"2024-03-11T21:36:35.056777+0000","last_became_peered":"2024-03-11T21:36:35.056777+0000","last_unstale":"2024-03-11T21:42:21.484895+0000","last_undegraded":"2024-03-11T21:42:21.484895+0000","last_fullsized":"2024-03-11T21:42:21.484895+0000","mapping_epoch":277,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":278,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:41:27.387998+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:37.512018+0000","last_clean_scrub_stamp":"2024-03-11T21:41:27.387998+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:42:31.296577+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,7],"acting":[2,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.8","version":"260'34","reported_seq":809,"reported_epoch":394,"state":"active+clean","last_fresh":"2024-03-11T21:42:25.501933+0000","last_change":"2024-03-11T21:41:06.881563+0000","last_active":"2024-03-11T21:42:25.501933+0000","last_peered":"2024-03-11T21:42:25.501933+0000","last_clean":"2024-03-11T21:42:25.501933+0000","last_became_active":"2024-03-11T21:34:41.146631+0000","last_became_peered":"2024-03-11T21:34:41.146631+0000","last_unstale":"2024-03-11T21:42:25.501933+0000","last_undegraded":"2024-03-11T21:42:25.501933+0000","last_fullsized":"2024-03-11T21:42:25.501933+0000","mapping_epoch":226,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":227,"parent":"0.0","parent_split_bits":0,"last_scrub":"260'34","last_scrub_stamp":"2024-03-11T21:41:06.881414+0000","last_deep_scrub":"109'22","last_deep_scrub_stamp":"2024-03-11T21:32:07.573197+0000","last_clean_scrub_stamp":"2024-03-11T21:41:06.881414+0000","objects_scrubbed":7,"log_size":34,"log_dups_size":0,"ondisk_log_size":34,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":44,"num_read_kb":35,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.9","version":"229'11","reported_seq":461,"reported_epoch":394,"state":"active+clean","last_fresh":"2024-03-11T21:42:25.533184+0000","last_change":"2024-03-11T21:41:28.240859+0000","last_active":"2024-03-11T21:42:25.533184+0000","last_peered":"2024-03-11T21:42:25.533184+0000","last_clean":"2024-03-11T21:42:25.533184+0000","last_became_active":"2024-03-11T21:34:41.156892+0000","last_became_peered":"2024-03-11T21:34:41.156892+0000","last_unstale":"2024-03-11T21:42:25.533184+0000","last_undegraded":"2024-03-11T21:42:25.533184+0000","last_fullsized":"2024-03-11T21:42:25.533184+0000","mapping_epoch":226,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":227,"parent":"0.0","parent_split_bits":0,"last_scrub":"229'11","last_scrub_stamp":"2024-03-11T21:41:28.240763+0000","last_deep_scrub":"94'9","last_deep_scrub_stamp":"2024-03-11T21:26:53.422552+0000","last_clean_scrub_stamp":"2024-03-11T21:41:28.240763+0000","objects_scrubbed":1,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:42:55.415720+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":10,"num_pg":26,"stat_sum":{"num_bytes":7079723032,"num_objects":108031,"num_object_clones":0,"num_object_copies":324093,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":8894,"num_objects_misplaced":13218,"num_objects_unfound":0,"num_objects_dirty":108031,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":89549,"num_write_kb":5623937,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13361,"num_bytes_recovered":875626496,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":22761975808,"data_stored":22761963592,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":145811,"ondisk_log_size":145811,"up":78,"acting":73,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":1531,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":17,"num_read_kb":17,"num_write":10,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":1308,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":4593,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":6,"ondisk_log_size":6,"up":96,"acting":96,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":198,"num_read_kb":294,"num_write":179,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":2951840,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":2375680,"data_stored":2361472,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":128,"ondisk_log_size":128,"up":3,"acting":3,"num_store_stats":4},{"poolid":3,"num_pg":3,"stat_sum":{"num_bytes":408,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":996,"num_read_kb":991,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1224,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":8,"ondisk_log_size":8,"up":9,"acting":9,"num_store_stats":7},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":3702,"num_objects":178,"num_object_clones":0,"num_object_copies":534,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":178,"num_whiteouts":0,"num_read":1120,"num_read_kb":877,"num_write":703,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":186,"num_bytes_recovered":2200,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":417792,"data_stored":11106,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":774,"ondisk_log_size":774,"up":96,"acting":96,"num_store_stats":8},{"poolid":5,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":78,"ondisk_log_size":78,"up":96,"acting":96,"num_store_stats":8},{"poolid":6,"num_pg":32,"stat_sum":{"num_bytes":383,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":370,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1149,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":96,"acting":96,"num_store_stats":8}],"osd_stats":[{"osd":7,"up_from":60,"seq":257698037965,"num_pgs":65,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2414760,"kb_used_data":1878232,"kb_used_omap":1,"kb_used_meta":536510,"kb_avail":91330392,"statfs":{"total":95995035648,"available":93522321408,"internally_reserved":0,"allocated":1923309568,"data_stored":1922140630,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":549386698},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1],"upper_bound":8},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":50,"seq":214748365011,"num_pgs":54,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":3818276,"kb_used_data":3335900,"kb_used_omap":1,"kb_used_meta":482366,"kb_avail":89926876,"statfs":{"total":95995035648,"available":92085121024,"internally_reserved":0,"allocated":3415961600,"data_stored":3414789319,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":493943238},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,0,0,0,3,0,10,1],"upper_bound":2048},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834575084,"num_pgs":62,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2613540,"kb_used_data":1946520,"kb_used_omap":1,"kb_used_meta":667006,"kb_avail":91131612,"statfs":{"total":95995035648,"available":93318770688,"internally_reserved":0,"allocated":1993236480,"data_stored":1992053394,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":683014597},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738609,"num_pgs":66,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":4908188,"kb_used_data":4241168,"kb_used_omap":1,"kb_used_meta":667006,"kb_avail":88836964,"statfs":{"total":95995035648,"available":90969051136,"internally_reserved":0,"allocated":4342956032,"data_stored":4341781742,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":683014597},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,3,2],"upper_bound":16},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411558,"num_pgs":52,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":3396168,"kb_used_data":2819384,"kb_used_omap":1,"kb_used_meta":576766,"kb_avail":90348984,"statfs":{"total":95995035648,"available":92517359616,"internally_reserved":0,"allocated":2887049216,"data_stored":2885898668,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":590608837},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":3,"up_from":277,"seq":1189705941063,"num_pgs":65,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2564264,"kb_used_data":2219612,"kb_used_omap":4,"kb_used_meta":344635,"kb_avail":91180888,"statfs":{"total":95995035648,"available":93369229312,"internally_reserved":0,"allocated":2272882688,"data_stored":2271695950,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4442,"internal_metadata":352906918},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":4,"up_from":32,"seq":137438953692,"num_pgs":68,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":4026296,"kb_used_data":3503656,"kb_used_omap":1,"kb_used_meta":522622,"kb_avail":89718856,"statfs":{"total":95995035648,"available":91872108544,"internally_reserved":0,"allocated":3587743744,"data_stored":3586558247,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":535165381},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1],"upper_bound":4},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":41,"seq":176093659351,"num_pgs":60,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2949216,"kb_used_data":2318288,"kb_used_omap":1,"kb_used_meta":630910,"kb_avail":90795936,"statfs":{"total":95995035648,"available":92975038464,"internally_reserved":0,"allocated":2373926912,"data_stored":2372765734,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":646052293},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1049,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":46,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":436,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":957,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":574,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":436,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1003,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":92,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":1430,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":77824,"data_stored":2008,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":1018,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":1898,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":1474,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":990,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":1034,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":1254,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":370,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":383,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":370,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4333178880,"data_stored":4333174808,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":2015428608,"data_stored":2015428608,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":2906128384,"data_stored":2906128384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":2263875584,"data_stored":2263875584,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":3557429248,"data_stored":3557425176,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":2371026944,"data_stored":2371026944,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":3396669440,"data_stored":3396665368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":1918238720,"data_stored":1918238720,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-11T21:42:41.795 INFO:tasks.thrashosds.thrasher:Setting 2.12 to [3, 7, 4, 0, 1, 5] 2024-03-11T21:42:41.795 INFO:tasks.thrashosds.thrasher:cmd ['osd', 'pg-upmap-items', '2.12', '3', '7', '4', '0', '1', '5'] 2024-03-11T21:42:41.796 DEBUG:teuthology.orchestra.run.smithi092:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid 00db0016-dfed-11ee-95c8-87774f69a715 -- ceph osd pg-upmap-items 2.12 3 7 4 0 1 5 2024-03-11T21:42:41.814 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:42:41.814 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:42:41.815 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:42:41.916 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:42:41.950 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:42:41.951 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:42:41.951 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:42:42.052 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:42:42.096 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:42:42.097 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:42:42.097 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:42:42.198 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:42:42.222 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:42:42.223 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:42:42.223 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:42:42.324 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:42:42.353 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:42:42.354 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:42:42.355 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:42:42.456 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:42:42.499 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:42:42.500 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:42:42.500 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:42:42.570 DEBUG:teuthology.orchestra.run.smithi097:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid 00db0016-dfed-11ee-95c8-87774f69a715 -- ceph daemon osd.5 dump_historic_ops 2024-03-11T21:42:42.601 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:42:42.642 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:42:42.643 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:42:42.644 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:42:42.744 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:42:42.778 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:42:42.779 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:42:42.779 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:42:42.831 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:42:42 smithi097 ceph-mon[33317]: pgmap v806: 158 pgs: 1 active+recovery_wait, 153 active+clean, 4 active+recovering+undersized+remapped; 6.0 GiB data, 24 GiB used, 691 GiB / 715 GiB avail; 644 B/s rd, 0 B/s wr, 586 op/s; 16748/296475 objects misplaced (5.649%); 5.8 MiB/s, 92 objects/s recovering 2024-03-11T21:42:42.880 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:42:42.899 INFO:teuthology.orchestra.run.smithi097.stderr:Inferring config /var/lib/ceph/00db0016-dfed-11ee-95c8-87774f69a715/mon.b/config 2024-03-11T21:42:42.903 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:42:42.904 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:42:42.904 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:42:43.005 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:42:43.047 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:42:43.048 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:42:43.048 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:42:43.116 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:42:42 smithi092 ceph-mon[29572]: pgmap v806: 158 pgs: 1 active+recovery_wait, 153 active+clean, 4 active+recovering+undersized+remapped; 6.0 GiB data, 24 GiB used, 691 GiB / 715 GiB avail; 644 B/s rd, 0 B/s wr, 586 op/s; 16748/296475 objects misplaced (5.649%); 5.8 MiB/s, 92 objects/s recovering 2024-03-11T21:42:43.116 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:42:42 smithi092 ceph-mon[37563]: pgmap v806: 158 pgs: 1 active+recovery_wait, 153 active+clean, 4 active+recovering+undersized+remapped; 6.0 GiB data, 24 GiB used, 691 GiB / 715 GiB avail; 644 B/s rd, 0 B/s wr, 586 op/s; 16748/296475 objects misplaced (5.649%); 5.8 MiB/s, 92 objects/s recovering 2024-03-11T21:42:43.150 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:42:43.177 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:42:43.178 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:42:43.178 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:42:43.193 INFO:journalctl@ceph.iscsi.iscsi.a.smithi097.stdout:Mar 11 21:42:42 smithi097 ceph-00db0016-dfed-11ee-95c8-87774f69a715-iscsi-iscsi-a[60943]: debug there is no tcmu-runner data available 2024-03-11T21:42:43.279 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:42:43.318 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:42:43.318 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:42:43.319 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:42:43.420 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:42:43.453 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:42:43.453 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:42:43.454 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:42:43.555 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:42:43.586 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:42:43.587 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:42:43.587 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:42:43.688 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:42:43.726 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:42:43.727 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:42:43.727 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:42:43.828 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:42:43.852 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:42:43.852 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:42:43.852 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:42:43.953 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:42:43.984 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:42:43.985 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:42:43.986 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:42:44.072 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:42:43 smithi097 ceph-mon[33317]: from='client.24469 v1:172.21.15.97:0/404208310' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:42:44.087 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:42:44.116 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:42:43 smithi092 ceph-mon[29572]: from='client.24469 v1:172.21.15.97:0/404208310' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:42:44.116 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:42:43 smithi092 ceph-mon[37563]: from='client.24469 v1:172.21.15.97:0/404208310' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:42:44.130 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:42:44.131 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:42:44.131 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:42:44.232 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:42:44.263 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:42:44.264 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:42:44.264 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:42:44.365 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:42:44.397 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:42:44.397 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:42:44.398 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:42:44.499 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:42:44.536 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:42:44.536 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:42:44.537 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:42:44.638 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:42:44.676 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:42:44.676 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:42:44.677 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:42:44.777 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:42:44.825 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:42:44.826 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:42:44.826 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:42:44.833 INFO:teuthology.orchestra.run.smithi092.stderr:Inferring config /var/lib/ceph/00db0016-dfed-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T21:42:44.882 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:42:44 smithi097 ceph-mon[33317]: pgmap v807: 158 pgs: 1 active+recovery_wait, 153 active+clean, 4 active+recovering+undersized+remapped; 5.8 GiB data, 23 GiB used, 692 GiB / 715 GiB avail; 583 B/s rd, 0 B/s wr, 902 op/s; 15747/285144 objects misplaced (5.522%); 7.3 MiB/s, 117 objects/s recovering 2024-03-11T21:42:44.927 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:42:44.986 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:42:44.987 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:42:44.987 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:42:45.000 INFO:teuthology.orchestra.run.smithi097.stdout:{ 2024-03-11T21:42:45.000 INFO:teuthology.orchestra.run.smithi097.stdout: "size": 20, 2024-03-11T21:42:45.000 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 600, 2024-03-11T21:42:45.001 INFO:teuthology.orchestra.run.smithi097.stdout: "ops": [ 2024-03-11T21:42:45.001 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:42:45.001 INFO:teuthology.orchestra.run.smithi097.stdout: "description": "osd_op(client.15150.0:208574 8.d 8:b4f0a13a:::benchmark_data_smithi092_104865_object88073:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e238)", 2024-03-11T21:42:45.001 INFO:teuthology.orchestra.run.smithi097.stdout: "initiated_at": "2024-03-11T21:34:57.352453+0000", 2024-03-11T21:42:45.001 INFO:teuthology.orchestra.run.smithi097.stdout: "age": 467.47603980500003, 2024-03-11T21:42:45.001 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.0319107139999999, 2024-03-11T21:42:45.001 INFO:teuthology.orchestra.run.smithi097.stdout: "type_data": { 2024-03-11T21:42:45.001 INFO:teuthology.orchestra.run.smithi097.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-11T21:42:45.001 INFO:teuthology.orchestra.run.smithi097.stdout: "client_info": { 2024-03-11T21:42:45.001 INFO:teuthology.orchestra.run.smithi097.stdout: "client": "client.15150", 2024-03-11T21:42:45.001 INFO:teuthology.orchestra.run.smithi097.stdout: "client_addr": "v1:172.21.15.92:0/436606717", 2024-03-11T21:42:45.001 INFO:teuthology.orchestra.run.smithi097.stdout: "tid": 208574 2024-03-11T21:42:45.001 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:42:45.001 INFO:teuthology.orchestra.run.smithi097.stdout: "events": [ 2024-03-11T21:42:45.002 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:42:45.002 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "initiated", 2024-03-11T21:42:45.002 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:34:57.352453+0000", 2024-03-11T21:42:45.002 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-03-11T21:42:45.002 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:42:45.002 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:42:45.002 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "header_read", 2024-03-11T21:42:45.002 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:34:57.352453+0000", 2024-03-11T21:42:45.002 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-03-11T21:42:45.002 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:42:45.002 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:42:45.002 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "throttled", 2024-03-11T21:42:45.002 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:34:57.352455+0000", 2024-03-11T21:42:45.002 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 2.0600000000000002e-06 2024-03-11T21:42:45.002 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:42:45.003 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:42:45.003 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "all_read", 2024-03-11T21:42:45.003 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:34:57.352462+0000", 2024-03-11T21:42:45.003 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 6.229e-06 2024-03-11T21:42:45.003 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:42:45.003 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:42:45.003 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "dispatched", 2024-03-11T21:42:45.003 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:34:57.353033+0000", 2024-03-11T21:42:45.003 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.00057142200000000003 2024-03-11T21:42:45.003 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:42:45.003 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:42:45.003 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "queued_for_pg", 2024-03-11T21:42:45.003 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:34:57.353041+0000", 2024-03-11T21:42:45.003 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 8.4379999999999995e-06 2024-03-11T21:42:45.003 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:42:45.004 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:42:45.004 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "reached_pg", 2024-03-11T21:42:45.004 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:34:57.353352+0000", 2024-03-11T21:42:45.004 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.00031082400000000002 2024-03-11T21:42:45.004 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:42:45.004 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:42:45.004 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "waiting for peered", 2024-03-11T21:42:45.004 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:34:57.353366+0000", 2024-03-11T21:42:45.004 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.4037e-05 2024-03-11T21:42:45.004 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:42:45.004 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:42:45.004 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "reached_pg", 2024-03-11T21:42:45.004 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:34:58.381460+0000", 2024-03-11T21:42:45.004 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.0280934779999999 2024-03-11T21:42:45.005 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:42:45.005 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:42:45.005 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "started", 2024-03-11T21:42:45.005 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:34:58.381562+0000", 2024-03-11T21:42:45.005 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.000102308 2024-03-11T21:42:45.005 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:42:45.005 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:42:45.005 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "waiting for subops from 2,7", 2024-03-11T21:42:45.005 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:34:58.381651+0000", 2024-03-11T21:42:45.005 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 8.9386000000000001e-05 2024-03-11T21:42:45.005 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:42:45.005 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:42:45.005 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "sub_op_commit_rec", 2024-03-11T21:42:45.006 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:34:58.384224+0000", 2024-03-11T21:42:45.006 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.0025727839999999998 2024-03-11T21:42:45.006 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:42:45.006 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:42:45.006 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "sub_op_commit_rec", 2024-03-11T21:42:45.006 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:34:58.384274+0000", 2024-03-11T21:42:45.006 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 4.994e-05 2024-03-11T21:42:45.006 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:42:45.006 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:42:45.006 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "op_commit", 2024-03-11T21:42:45.006 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:34:58.384287+0000", 2024-03-11T21:42:45.006 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.2376e-05 2024-03-11T21:42:45.006 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:42:45.006 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:42:45.007 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "commit_sent", 2024-03-11T21:42:45.007 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:34:58.384318+0000", 2024-03-11T21:42:45.007 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 3.1905e-05 2024-03-11T21:42:45.007 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:42:45.007 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:42:45.007 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "done", 2024-03-11T21:42:45.007 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:34:58.384364+0000", 2024-03-11T21:42:45.007 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 4.5527e-05 2024-03-11T21:42:45.007 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-03-11T21:42:45.007 INFO:teuthology.orchestra.run.smithi097.stdout: ] 2024-03-11T21:42:45.007 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-03-11T21:42:45.007 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:42:45.007 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:42:45.007 INFO:teuthology.orchestra.run.smithi097.stdout: "description": "osd_op(client.15150.0:208578 8.d 8:b8981014:::benchmark_data_smithi092_104865_object88077:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e239)", 2024-03-11T21:42:45.008 INFO:teuthology.orchestra.run.smithi097.stdout: "initiated_at": "2024-03-11T21:34:57.356717+0000", 2024-03-11T21:42:45.008 INFO:teuthology.orchestra.run.smithi097.stdout: "age": 467.471775691, 2024-03-11T21:42:45.008 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.0313246620000001, 2024-03-11T21:42:45.008 INFO:teuthology.orchestra.run.smithi097.stdout: "type_data": { 2024-03-11T21:42:45.008 INFO:teuthology.orchestra.run.smithi097.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-11T21:42:45.008 INFO:teuthology.orchestra.run.smithi097.stdout: "client_info": { 2024-03-11T21:42:45.008 INFO:teuthology.orchestra.run.smithi097.stdout: "client": "client.15150", 2024-03-11T21:42:45.008 INFO:teuthology.orchestra.run.smithi097.stdout: "client_addr": "v1:172.21.15.92:0/436606717", 2024-03-11T21:42:45.008 INFO:teuthology.orchestra.run.smithi097.stdout: "tid": 208578 2024-03-11T21:42:45.008 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:42:45.008 INFO:teuthology.orchestra.run.smithi097.stdout: "events": [ 2024-03-11T21:42:45.008 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:42:45.008 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "initiated", 2024-03-11T21:42:45.009 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:34:57.356717+0000", 2024-03-11T21:42:45.009 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-03-11T21:42:45.009 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:42:45.009 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:42:45.009 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "header_read", 2024-03-11T21:42:45.009 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:34:57.356717+0000", 2024-03-11T21:42:45.009 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-03-11T21:42:45.009 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:42:45.009 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:42:45.009 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "throttled", 2024-03-11T21:42:45.009 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:34:57.356719+0000", 2024-03-11T21:42:45.009 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.866e-06 2024-03-11T21:42:45.009 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:42:45.009 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:42:45.009 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "all_read", 2024-03-11T21:42:45.010 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:34:57.356724+0000", 2024-03-11T21:42:45.010 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 5.1909999999999999e-06 2024-03-11T21:42:45.010 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:42:45.010 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:42:45.010 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "dispatched", 2024-03-11T21:42:45.010 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:34:57.356736+0000", 2024-03-11T21:42:45.010 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.1231e-05 2024-03-11T21:42:45.010 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:42:45.010 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:42:45.010 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "queued_for_pg", 2024-03-11T21:42:45.010 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:34:57.356743+0000", 2024-03-11T21:42:45.010 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 7.5830000000000001e-06 2024-03-11T21:42:45.010 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:42:45.010 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:42:45.011 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "reached_pg", 2024-03-11T21:42:45.011 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:34:57.356802+0000", 2024-03-11T21:42:45.011 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 5.9107999999999997e-05 2024-03-11T21:42:45.011 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:42:45.011 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:42:45.011 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "waiting for peered", 2024-03-11T21:42:45.011 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:34:57.356811+0000", 2024-03-11T21:42:45.011 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 8.8869999999999998e-06 2024-03-11T21:42:45.011 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:42:45.011 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:42:45.011 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "reached_pg", 2024-03-11T21:42:45.011 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:34:58.384395+0000", 2024-03-11T21:42:45.011 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.0275842260000001 2024-03-11T21:42:45.011 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:42:45.011 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:42:45.012 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "started", 2024-03-11T21:42:45.012 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:34:58.384476+0000", 2024-03-11T21:42:45.012 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 8.0187999999999999e-05 2024-03-11T21:42:45.012 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:42:45.012 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:42:45.012 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "waiting for subops from 2,7", 2024-03-11T21:42:45.012 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:34:58.384553+0000", 2024-03-11T21:42:45.012 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 7.6897999999999995e-05 2024-03-11T21:42:45.012 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:42:45.012 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:42:45.012 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "sub_op_commit_rec", 2024-03-11T21:42:45.012 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:34:58.387913+0000", 2024-03-11T21:42:45.012 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.0033606370000000001 2024-03-11T21:42:45.012 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:42:45.012 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:42:45.012 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "sub_op_commit_rec", 2024-03-11T21:42:45.013 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:34:58.387977+0000", 2024-03-11T21:42:45.013 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 6.3377e-05 2024-03-11T21:42:45.013 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:42:45.013 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:42:45.013 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "op_commit", 2024-03-11T21:42:45.013 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:34:58.387987+0000", 2024-03-11T21:42:45.013 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.0173000000000001e-05 2024-03-11T21:42:45.013 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:42:45.013 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:42:45.013 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "commit_sent", 2024-03-11T21:42:45.013 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:34:58.388017+0000", 2024-03-11T21:42:45.013 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 3.0439000000000001e-05 2024-03-11T21:42:45.013 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:42:45.013 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:42:45.013 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "done", 2024-03-11T21:42:45.013 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:34:58.388042+0000", 2024-03-11T21:42:45.014 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 2.4858000000000002e-05 2024-03-11T21:42:45.014 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-03-11T21:42:45.014 INFO:teuthology.orchestra.run.smithi097.stdout: ] 2024-03-11T21:42:45.014 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-03-11T21:42:45.014 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:42:45.014 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:42:45.014 INFO:teuthology.orchestra.run.smithi097.stdout: "description": "osd_op(client.15150.0:208589 8.d 8:b0d80d0e:::benchmark_data_smithi092_104865_object88088:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e239)", 2024-03-11T21:42:45.014 INFO:teuthology.orchestra.run.smithi097.stdout: "initiated_at": "2024-03-11T21:34:57.366673+0000", 2024-03-11T21:42:45.014 INFO:teuthology.orchestra.run.smithi097.stdout: "age": 467.46181962200001, 2024-03-11T21:42:45.014 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.026816476, 2024-03-11T21:42:45.014 INFO:teuthology.orchestra.run.smithi097.stdout: "type_data": { 2024-03-11T21:42:45.014 INFO:teuthology.orchestra.run.smithi097.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-11T21:42:45.015 INFO:teuthology.orchestra.run.smithi097.stdout: "client_info": { 2024-03-11T21:42:45.015 INFO:teuthology.orchestra.run.smithi097.stdout: "client": "client.15150", 2024-03-11T21:42:45.015 INFO:teuthology.orchestra.run.smithi097.stdout: "client_addr": "v1:172.21.15.92:0/436606717", 2024-03-11T21:42:45.015 INFO:teuthology.orchestra.run.smithi097.stdout: "tid": 208589 2024-03-11T21:42:45.015 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:42:45.015 INFO:teuthology.orchestra.run.smithi097.stdout: "events": [ 2024-03-11T21:42:45.015 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:42:45.015 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "initiated", 2024-03-11T21:42:45.015 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:34:57.366673+0000", 2024-03-11T21:42:45.015 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-03-11T21:42:45.015 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:42:45.015 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:42:45.015 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "header_read", 2024-03-11T21:42:45.015 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:34:57.366673+0000", 2024-03-11T21:42:45.016 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-03-11T21:42:45.016 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:42:45.016 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:42:45.016 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "throttled", 2024-03-11T21:42:45.016 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:34:57.366675+0000", 2024-03-11T21:42:45.016 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.567e-06 2024-03-11T21:42:45.016 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:42:45.016 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:42:45.016 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "all_read", 2024-03-11T21:42:45.016 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:34:57.366679+0000", 2024-03-11T21:42:45.016 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 4.4299999999999999e-06 2024-03-11T21:42:45.016 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:42:45.016 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:42:45.016 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "dispatched", 2024-03-11T21:42:45.016 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:34:57.366690+0000", 2024-03-11T21:42:45.016 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.1028e-05 2024-03-11T21:42:45.017 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:42:45.017 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:42:45.017 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "queued_for_pg", 2024-03-11T21:42:45.017 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:34:57.366696+0000", 2024-03-11T21:42:45.017 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 5.8860000000000001e-06 2024-03-11T21:42:45.017 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:42:45.017 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:42:45.017 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "reached_pg", 2024-03-11T21:42:45.017 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:34:57.366752+0000", 2024-03-11T21:42:45.017 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 5.5566999999999998e-05 2024-03-11T21:42:45.017 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:42:45.017 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:42:45.017 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "waiting for peered", 2024-03-11T21:42:45.017 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:34:57.366758+0000", 2024-03-11T21:42:45.017 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 6.2269999999999998e-06 2024-03-11T21:42:45.018 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:42:45.018 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:42:45.018 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "reached_pg", 2024-03-11T21:42:45.018 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:34:58.386291+0000", 2024-03-11T21:42:45.018 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.019533238 2024-03-11T21:42:45.018 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:42:45.018 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:42:45.018 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "started", 2024-03-11T21:42:45.018 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:34:58.386368+0000", 2024-03-11T21:42:45.018 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 7.6126999999999998e-05 2024-03-11T21:42:45.018 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:42:45.018 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:42:45.018 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "waiting for subops from 2,7", 2024-03-11T21:42:45.018 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:34:58.386440+0000", 2024-03-11T21:42:45.019 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 7.2539999999999993e-05 2024-03-11T21:42:45.019 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:42:45.019 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:42:45.019 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "sub_op_commit_rec", 2024-03-11T21:42:45.019 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:34:58.388089+0000", 2024-03-11T21:42:45.019 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.001648722 2024-03-11T21:42:45.019 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:42:45.019 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:42:45.019 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "op_commit", 2024-03-11T21:42:45.019 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:34:58.390342+0000", 2024-03-11T21:42:45.019 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.0022533879999999998 2024-03-11T21:42:45.019 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:42:45.019 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:42:45.020 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "sub_op_commit_rec", 2024-03-11T21:42:45.020 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:34:58.393325+0000", 2024-03-11T21:42:45.020 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.0029828979999999999 2024-03-11T21:42:45.020 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:42:45.020 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:42:45.020 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "commit_sent", 2024-03-11T21:42:45.020 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:34:58.393440+0000", 2024-03-11T21:42:45.020 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.000115418 2024-03-11T21:42:45.020 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:42:45.020 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:42:45.020 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "done", 2024-03-11T21:42:45.020 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:34:58.393490+0000", 2024-03-11T21:42:45.020 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 4.9440000000000001e-05 2024-03-11T21:42:45.020 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-03-11T21:42:45.021 INFO:teuthology.orchestra.run.smithi097.stdout: ] 2024-03-11T21:42:45.021 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-03-11T21:42:45.021 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:42:45.021 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:42:45.021 INFO:teuthology.orchestra.run.smithi097.stdout: "description": "osd_op(client.15150.0:208601 8.d 8:b93ad02e:::benchmark_data_smithi092_104865_object88100:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e239)", 2024-03-11T21:42:45.021 INFO:teuthology.orchestra.run.smithi097.stdout: "initiated_at": "2024-03-11T21:34:57.372084+0000", 2024-03-11T21:42:45.021 INFO:teuthology.orchestra.run.smithi097.stdout: "age": 467.45640874899999, 2024-03-11T21:42:45.021 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.021649343, 2024-03-11T21:42:45.021 INFO:teuthology.orchestra.run.smithi097.stdout: "type_data": { 2024-03-11T21:42:45.021 INFO:teuthology.orchestra.run.smithi097.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-11T21:42:45.021 INFO:teuthology.orchestra.run.smithi097.stdout: "client_info": { 2024-03-11T21:42:45.021 INFO:teuthology.orchestra.run.smithi097.stdout: "client": "client.15150", 2024-03-11T21:42:45.021 INFO:teuthology.orchestra.run.smithi097.stdout: "client_addr": "v1:172.21.15.92:0/436606717", 2024-03-11T21:42:45.021 INFO:teuthology.orchestra.run.smithi097.stdout: "tid": 208601 2024-03-11T21:42:45.022 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:42:45.022 INFO:teuthology.orchestra.run.smithi097.stdout: "events": [ 2024-03-11T21:42:45.022 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:42:45.022 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "initiated", 2024-03-11T21:42:45.022 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:34:57.372084+0000", 2024-03-11T21:42:45.022 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-03-11T21:42:45.022 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:42:45.022 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:42:45.022 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "header_read", 2024-03-11T21:42:45.022 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:34:57.372084+0000", 2024-03-11T21:42:45.022 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-03-11T21:42:45.022 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:42:45.022 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:42:45.022 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "throttled", 2024-03-11T21:42:45.023 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:34:57.372086+0000", 2024-03-11T21:42:45.023 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.5209999999999999e-06 2024-03-11T21:42:45.023 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:42:45.023 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:42:45.023 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "all_read", 2024-03-11T21:42:45.023 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:34:57.372090+0000", 2024-03-11T21:42:45.023 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 4.5909999999999998e-06 2024-03-11T21:42:45.023 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:42:45.023 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:42:45.023 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "dispatched", 2024-03-11T21:42:45.023 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:34:57.372101+0000", 2024-03-11T21:42:45.023 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.0912e-05 2024-03-11T21:42:45.023 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:42:45.023 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:42:45.023 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "queued_for_pg", 2024-03-11T21:42:45.075 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:34:57.372117+0000", 2024-03-11T21:42:45.075 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.5247e-05 2024-03-11T21:42:45.075 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:42:45.075 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:42:45.075 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "reached_pg", 2024-03-11T21:42:45.075 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:34:57.372177+0000", 2024-03-11T21:42:45.075 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 5.9957999999999998e-05 2024-03-11T21:42:45.075 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:42:45.075 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:42:45.075 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "waiting for peered", 2024-03-11T21:42:45.076 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:34:57.372186+0000", 2024-03-11T21:42:45.076 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 9.5750000000000007e-06 2024-03-11T21:42:45.076 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:42:45.076 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:42:45.076 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "reached_pg", 2024-03-11T21:42:45.076 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:34:58.388134+0000", 2024-03-11T21:42:45.076 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.0159478159999999 2024-03-11T21:42:45.076 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:42:45.076 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:42:45.076 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "started", 2024-03-11T21:42:45.076 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:34:58.388202+0000", 2024-03-11T21:42:45.076 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 6.7717000000000002e-05 2024-03-11T21:42:45.076 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:42:45.076 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:42:45.077 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "waiting for subops from 2,7", 2024-03-11T21:42:45.077 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:34:58.388274+0000", 2024-03-11T21:42:45.077 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 7.2342999999999995e-05 2024-03-11T21:42:45.077 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:42:45.077 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:42:45.077 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "sub_op_commit_rec", 2024-03-11T21:42:45.077 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:34:58.393561+0000", 2024-03-11T21:42:45.077 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.0052871180000000004 2024-03-11T21:42:45.077 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:42:45.077 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:42:45.077 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "sub_op_commit_rec", 2024-03-11T21:42:45.077 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:34:58.393612+0000", 2024-03-11T21:42:45.077 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 5.0923e-05 2024-03-11T21:42:45.077 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:42:45.077 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:42:45.078 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "op_commit", 2024-03-11T21:42:45.078 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:34:58.393630+0000", 2024-03-11T21:42:45.078 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.7716000000000001e-05 2024-03-11T21:42:45.078 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:42:45.078 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:42:45.078 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "commit_sent", 2024-03-11T21:42:45.078 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:34:58.393683+0000", 2024-03-11T21:42:45.078 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 5.3656999999999998e-05 2024-03-11T21:42:45.078 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:42:45.078 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:42:45.078 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "done", 2024-03-11T21:42:45.078 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:34:58.393734+0000", 2024-03-11T21:42:45.078 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 5.0249e-05 2024-03-11T21:42:45.079 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-03-11T21:42:45.079 INFO:teuthology.orchestra.run.smithi097.stdout: ] 2024-03-11T21:42:45.079 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-03-11T21:42:45.079 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:42:45.079 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:42:45.079 INFO:teuthology.orchestra.run.smithi097.stdout: "description": "osd_op(client.15150.0:208608 8.d 8:b39a191f:::benchmark_data_smithi092_104865_object88107:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e239)", 2024-03-11T21:42:45.079 INFO:teuthology.orchestra.run.smithi097.stdout: "initiated_at": "2024-03-11T21:34:57.377729+0000", 2024-03-11T21:42:45.079 INFO:teuthology.orchestra.run.smithi097.stdout: "age": 467.45076429, 2024-03-11T21:42:45.079 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.016253541, 2024-03-11T21:42:45.079 INFO:teuthology.orchestra.run.smithi097.stdout: "type_data": { 2024-03-11T21:42:45.079 INFO:teuthology.orchestra.run.smithi097.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-11T21:42:45.079 INFO:teuthology.orchestra.run.smithi097.stdout: "client_info": { 2024-03-11T21:42:45.079 INFO:teuthology.orchestra.run.smithi097.stdout: "client": "client.15150", 2024-03-11T21:42:45.079 INFO:teuthology.orchestra.run.smithi097.stdout: "client_addr": "v1:172.21.15.92:0/436606717", 2024-03-11T21:42:45.079 INFO:teuthology.orchestra.run.smithi097.stdout: "tid": 208608 2024-03-11T21:42:45.080 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:42:45.080 INFO:teuthology.orchestra.run.smithi097.stdout: "events": [ 2024-03-11T21:42:45.080 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:42:45.080 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "initiated", 2024-03-11T21:42:45.080 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:34:57.377729+0000", 2024-03-11T21:42:45.080 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-03-11T21:42:45.080 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:42:45.080 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:42:45.080 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "header_read", 2024-03-11T21:42:45.080 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:34:57.377729+0000", 2024-03-11T21:42:45.080 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-03-11T21:42:45.080 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:42:45.080 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:42:45.080 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "throttled", 2024-03-11T21:42:45.081 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:34:57.377731+0000", 2024-03-11T21:42:45.081 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 2.1059999999999998e-06 2024-03-11T21:42:45.081 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:42:45.081 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:42:45.081 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "all_read", 2024-03-11T21:42:45.081 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:34:57.377739+0000", 2024-03-11T21:42:45.081 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 8.5110000000000003e-06 2024-03-11T21:42:45.081 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:42:45.081 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:42:45.081 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "dispatched", 2024-03-11T21:42:45.081 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:34:57.377752+0000", 2024-03-11T21:42:45.081 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.2523e-05 2024-03-11T21:42:45.081 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:42:45.081 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:42:45.081 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "queued_for_pg", 2024-03-11T21:42:45.082 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:34:57.377760+0000", 2024-03-11T21:42:45.082 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 7.8490000000000006e-06 2024-03-11T21:42:45.082 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:42:45.082 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:42:45.082 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "reached_pg", 2024-03-11T21:42:45.082 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:34:57.377823+0000", 2024-03-11T21:42:45.082 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 6.3337999999999996e-05 2024-03-11T21:42:45.082 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:42:45.082 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:42:45.082 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "waiting for peered", 2024-03-11T21:42:45.082 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:34:57.377833+0000", 2024-03-11T21:42:45.082 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 9.7070000000000004e-06 2024-03-11T21:42:45.082 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:42:45.082 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:42:45.082 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "reached_pg", 2024-03-11T21:42:45.083 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:34:58.390377+0000", 2024-03-11T21:42:45.083 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.0125443629999999 2024-03-11T21:42:45.083 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:42:45.083 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:42:45.083 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "started", 2024-03-11T21:42:45.083 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:34:58.390455+0000", 2024-03-11T21:42:45.083 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 7.7832999999999998e-05 2024-03-11T21:42:45.083 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:42:45.083 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:42:45.083 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "waiting for subops from 2,7", 2024-03-11T21:42:45.083 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:34:58.390541+0000", 2024-03-11T21:42:45.083 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 8.6014999999999996e-05 2024-03-11T21:42:45.083 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:42:45.083 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:42:45.084 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "sub_op_commit_rec", 2024-03-11T21:42:45.084 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:34:58.393817+0000", 2024-03-11T21:42:45.084 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.0032762799999999999 2024-03-11T21:42:45.084 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:42:45.084 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:42:45.084 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "op_commit", 2024-03-11T21:42:45.084 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:34:58.393832+0000", 2024-03-11T21:42:45.084 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.4773999999999999e-05 2024-03-11T21:42:45.084 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:42:45.084 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:42:45.084 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "sub_op_commit_rec", 2024-03-11T21:42:45.084 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:34:58.393890+0000", 2024-03-11T21:42:45.084 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 5.7969e-05 2024-03-11T21:42:45.084 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:42:45.084 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:42:45.085 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "commit_sent", 2024-03-11T21:42:45.085 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:34:58.393941+0000", 2024-03-11T21:42:45.085 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 5.1403999999999998e-05 2024-03-11T21:42:45.085 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:42:45.085 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:42:45.085 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "done", 2024-03-11T21:42:45.085 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:34:58.393982+0000", 2024-03-11T21:42:45.085 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 4.0868999999999998e-05 2024-03-11T21:42:45.085 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-03-11T21:42:45.085 INFO:teuthology.orchestra.run.smithi097.stdout: ] 2024-03-11T21:42:45.085 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-03-11T21:42:45.085 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:42:45.085 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:42:45.085 INFO:teuthology.orchestra.run.smithi097.stdout: "description": "osd_op(client.25282.0:76283 9.f 9:fcbba6a0:::benchmark_data_smithi092_130291_object76282:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e276)", 2024-03-11T21:42:45.085 INFO:teuthology.orchestra.run.smithi097.stdout: "initiated_at": "2024-03-11T21:36:34.033579+0000", 2024-03-11T21:42:45.086 INFO:teuthology.orchestra.run.smithi097.stdout: "age": 370.79491426999999, 2024-03-11T21:42:45.086 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 2.490014623, 2024-03-11T21:42:45.086 INFO:teuthology.orchestra.run.smithi097.stdout: "type_data": { 2024-03-11T21:42:45.086 INFO:teuthology.orchestra.run.smithi097.stdout: "flag_point": "reached pg", 2024-03-11T21:42:45.086 INFO:teuthology.orchestra.run.smithi097.stdout: "client_info": { 2024-03-11T21:42:45.086 INFO:teuthology.orchestra.run.smithi097.stdout: "client": "client.25282", 2024-03-11T21:42:45.086 INFO:teuthology.orchestra.run.smithi097.stdout: "client_addr": "v1:172.21.15.92:0/2489519821", 2024-03-11T21:42:45.086 INFO:teuthology.orchestra.run.smithi097.stdout: "tid": 76283 2024-03-11T21:42:45.086 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:42:45.086 INFO:teuthology.orchestra.run.smithi097.stdout: "events": [ 2024-03-11T21:42:45.086 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:42:45.086 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "initiated", 2024-03-11T21:42:45.086 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:36:34.033579+0000", 2024-03-11T21:42:45.086 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-03-11T21:42:45.087 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:42:45.087 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:42:45.087 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "header_read", 2024-03-11T21:42:45.087 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:36:34.033579+0000", 2024-03-11T21:42:45.087 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-03-11T21:42:45.087 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:42:45.087 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:42:45.087 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "throttled", 2024-03-11T21:42:45.087 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:36:34.033580+0000", 2024-03-11T21:42:45.087 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 7.3200000000000004e-07 2024-03-11T21:42:45.087 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:42:45.087 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:42:45.087 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "all_read", 2024-03-11T21:42:45.087 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:36:34.033612+0000", 2024-03-11T21:42:45.088 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 3.2396e-05 2024-03-11T21:42:45.088 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:42:45.088 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:42:45.088 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "dispatched", 2024-03-11T21:42:45.088 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:36:34.033620+0000", 2024-03-11T21:42:45.088 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 7.7039999999999994e-06 2024-03-11T21:42:45.088 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:42:45.088 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:42:45.088 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "queued_for_pg", 2024-03-11T21:42:45.088 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:36:34.033625+0000", 2024-03-11T21:42:45.088 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 5.5779999999999998e-06 2024-03-11T21:42:45.088 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:42:45.088 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:42:45.088 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "reached_pg", 2024-03-11T21:42:45.088 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:36:34.033753+0000", 2024-03-11T21:42:45.089 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.00012819 2024-03-11T21:42:45.089 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:42:45.089 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:42:45.089 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "started", 2024-03-11T21:42:45.089 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:36:34.034007+0000", 2024-03-11T21:42:45.089 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.00025316200000000003 2024-03-11T21:42:45.089 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:42:45.089 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:42:45.089 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "waiting for subops from 1", 2024-03-11T21:42:45.089 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:36:34.034136+0000", 2024-03-11T21:42:45.089 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.00012985399999999999 2024-03-11T21:42:45.089 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:42:45.089 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:42:45.089 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "op_commit", 2024-03-11T21:42:45.089 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:36:34.046308+0000", 2024-03-11T21:42:45.090 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.012171131 2024-03-11T21:42:45.090 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:42:45.090 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:42:45.090 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "reached_pg", 2024-03-11T21:42:45.090 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:36:34.058375+0000", 2024-03-11T21:42:45.090 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.012067791 2024-03-11T21:42:45.090 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:42:45.090 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:42:45.090 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "waiting for peered", 2024-03-11T21:42:45.090 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:36:34.058387+0000", 2024-03-11T21:42:45.090 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.1623999999999999e-05 2024-03-11T21:42:45.090 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:42:45.090 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:42:45.090 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "reached_pg", 2024-03-11T21:42:45.090 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:36:35.046799+0000", 2024-03-11T21:42:45.091 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.98841167900000004 2024-03-11T21:42:45.091 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:42:45.091 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:42:45.091 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "waiting for peered", 2024-03-11T21:42:45.091 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:36:35.046813+0000", 2024-03-11T21:42:45.091 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.4258e-05 2024-03-11T21:42:45.091 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:42:45.091 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:42:45.091 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "reached_pg", 2024-03-11T21:42:45.091 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:36:36.523493+0000", 2024-03-11T21:42:45.091 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.476680408 2024-03-11T21:42:45.091 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:42:45.091 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:42:45.091 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "done", 2024-03-11T21:42:45.092 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:36:36.523593+0000", 2024-03-11T21:42:45.092 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.000100116 2024-03-11T21:42:45.092 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-03-11T21:42:45.092 INFO:teuthology.orchestra.run.smithi097.stdout: ] 2024-03-11T21:42:45.092 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-03-11T21:42:45.092 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:42:45.092 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:42:45.092 INFO:teuthology.orchestra.run.smithi097.stdout: "description": "osd_op(client.25282.0:76287 9.7 9:eeb358ff:::benchmark_data_smithi092_130291_object76286:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e276)", 2024-03-11T21:42:45.092 INFO:teuthology.orchestra.run.smithi097.stdout: "initiated_at": "2024-03-11T21:36:34.036194+0000", 2024-03-11T21:42:45.092 INFO:teuthology.orchestra.run.smithi097.stdout: "age": 370.79229904200002, 2024-03-11T21:42:45.092 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 2.5035221519999999, 2024-03-11T21:42:45.092 INFO:teuthology.orchestra.run.smithi097.stdout: "type_data": { 2024-03-11T21:42:45.092 INFO:teuthology.orchestra.run.smithi097.stdout: "flag_point": "reached pg", 2024-03-11T21:42:45.092 INFO:teuthology.orchestra.run.smithi097.stdout: "client_info": { 2024-03-11T21:42:45.093 INFO:teuthology.orchestra.run.smithi097.stdout: "client": "client.25282", 2024-03-11T21:42:45.093 INFO:teuthology.orchestra.run.smithi097.stdout: "client_addr": "v1:172.21.15.92:0/2489519821", 2024-03-11T21:42:45.093 INFO:teuthology.orchestra.run.smithi097.stdout: "tid": 76287 2024-03-11T21:42:45.093 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:42:45.093 INFO:teuthology.orchestra.run.smithi097.stdout: "events": [ 2024-03-11T21:42:45.093 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:42:45.093 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "initiated", 2024-03-11T21:42:45.093 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:36:34.036194+0000", 2024-03-11T21:42:45.093 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-03-11T21:42:45.093 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:42:45.093 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:42:45.093 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "header_read", 2024-03-11T21:42:45.093 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:36:34.036194+0000", 2024-03-11T21:42:45.093 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-03-11T21:42:45.093 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:42:45.094 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:42:45.094 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "throttled", 2024-03-11T21:42:45.094 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:36:34.036196+0000", 2024-03-11T21:42:45.094 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 2.317e-06 2024-03-11T21:42:45.094 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:42:45.094 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:42:45.094 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "all_read", 2024-03-11T21:42:45.094 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:36:34.036261+0000", 2024-03-11T21:42:45.094 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 6.4343000000000004e-05 2024-03-11T21:42:45.094 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:42:45.094 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:42:45.094 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "dispatched", 2024-03-11T21:42:45.094 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:36:34.036272+0000", 2024-03-11T21:42:45.094 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.1790000000000001e-05 2024-03-11T21:42:45.095 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:42:45.095 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:42:45.095 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "queued_for_pg", 2024-03-11T21:42:45.095 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:36:34.036281+0000", 2024-03-11T21:42:45.095 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 8.3110000000000006e-06 2024-03-11T21:42:45.095 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:42:45.095 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:42:45.095 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "reached_pg", 2024-03-11T21:42:45.095 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:36:34.037543+0000", 2024-03-11T21:42:45.095 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.001262631 2024-03-11T21:42:45.095 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:42:45.095 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:42:45.095 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "started", 2024-03-11T21:42:45.095 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:36:34.037890+0000", 2024-03-11T21:42:45.095 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.00034664199999999998 2024-03-11T21:42:45.096 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:42:45.096 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:42:45.096 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "waiting for subops from 7", 2024-03-11T21:42:45.096 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:36:34.038099+0000", 2024-03-11T21:42:45.096 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.000208438 2024-03-11T21:42:45.096 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:42:45.096 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:42:45.096 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "reached_pg", 2024-03-11T21:42:45.096 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:36:34.058425+0000", 2024-03-11T21:42:45.096 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.020326765 2024-03-11T21:42:45.096 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:42:45.096 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:42:45.096 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "waiting for peered", 2024-03-11T21:42:45.096 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:36:34.058434+0000", 2024-03-11T21:42:45.097 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 8.9099999999999994e-06 2024-03-11T21:42:45.097 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:42:45.097 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:42:45.097 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "reached_pg", 2024-03-11T21:42:45.097 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:36:35.046759+0000", 2024-03-11T21:42:45.097 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.98832484399999998 2024-03-11T21:42:45.097 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:42:45.097 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:42:45.097 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "waiting for peered", 2024-03-11T21:42:45.097 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:36:35.046795+0000", 2024-03-11T21:42:45.097 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 3.6155999999999997e-05 2024-03-11T21:42:45.097 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:42:45.097 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:42:45.097 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "reached_pg", 2024-03-11T21:42:45.097 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:36:36.523236+0000", 2024-03-11T21:42:45.098 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.476440669 2024-03-11T21:42:45.098 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:42:45.098 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:42:45.098 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "waiting for degraded object", 2024-03-11T21:42:45.098 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:36:36.523912+0000", 2024-03-11T21:42:45.098 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.00067594499999999995 2024-03-11T21:42:45.098 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:42:45.098 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:42:45.098 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "reached_pg", 2024-03-11T21:42:45.098 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:36:36.539590+0000", 2024-03-11T21:42:45.098 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.015678455000000001 2024-03-11T21:42:45.098 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:42:45.098 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:42:45.098 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "done", 2024-03-11T21:42:45.098 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:36:36.539716+0000", 2024-03-11T21:42:45.099 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.00012593599999999999 2024-03-11T21:42:45.099 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-03-11T21:42:45.099 INFO:teuthology.orchestra.run.smithi097.stdout: ] 2024-03-11T21:42:45.099 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-03-11T21:42:45.099 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:42:45.099 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:42:45.099 INFO:teuthology.orchestra.run.smithi097.stdout: "description": "osd_op(client.25282.0:76292 9.f 9:f3aa237a:::benchmark_data_smithi092_130291_object76291:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e276)", 2024-03-11T21:42:45.099 INFO:teuthology.orchestra.run.smithi097.stdout: "initiated_at": "2024-03-11T21:36:34.038523+0000", 2024-03-11T21:42:45.099 INFO:teuthology.orchestra.run.smithi097.stdout: "age": 370.78997051599998, 2024-03-11T21:42:45.099 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 2.5077632140000001, 2024-03-11T21:42:45.099 INFO:teuthology.orchestra.run.smithi097.stdout: "type_data": { 2024-03-11T21:42:45.099 INFO:teuthology.orchestra.run.smithi097.stdout: "flag_point": "reached pg", 2024-03-11T21:42:45.099 INFO:teuthology.orchestra.run.smithi097.stdout: "client_info": { 2024-03-11T21:42:45.099 INFO:teuthology.orchestra.run.smithi097.stdout: "client": "client.25282", 2024-03-11T21:42:45.099 INFO:teuthology.orchestra.run.smithi097.stdout: "client_addr": "v1:172.21.15.92:0/2489519821", 2024-03-11T21:42:45.100 INFO:teuthology.orchestra.run.smithi097.stdout: "tid": 76292 2024-03-11T21:42:45.100 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:42:45.100 INFO:teuthology.orchestra.run.smithi097.stdout: "events": [ 2024-03-11T21:42:45.100 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:42:45.100 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "initiated", 2024-03-11T21:42:45.100 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:36:34.038523+0000", 2024-03-11T21:42:45.100 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-03-11T21:42:45.100 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:42:45.100 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:42:45.100 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "header_read", 2024-03-11T21:42:45.100 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:36:34.038523+0000", 2024-03-11T21:42:45.100 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-03-11T21:42:45.100 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:42:45.100 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:42:45.101 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "throttled", 2024-03-11T21:42:45.101 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:36:34.038524+0000", 2024-03-11T21:42:45.101 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.206e-06 2024-03-11T21:42:45.101 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:42:45.101 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:42:45.101 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "all_read", 2024-03-11T21:42:45.101 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:36:34.038578+0000", 2024-03-11T21:42:45.101 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 5.3983000000000001e-05 2024-03-11T21:42:45.101 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:42:45.101 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:42:45.101 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "dispatched", 2024-03-11T21:42:45.101 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:36:34.038586+0000", 2024-03-11T21:42:45.101 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 8.1610000000000003e-06 2024-03-11T21:42:45.101 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:42:45.101 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:42:45.102 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "queued_for_pg", 2024-03-11T21:42:45.102 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:36:34.038593+0000", 2024-03-11T21:42:45.102 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 6.6220000000000003e-06 2024-03-11T21:42:45.102 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:42:45.102 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:42:45.102 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "reached_pg", 2024-03-11T21:42:45.102 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:36:34.040758+0000", 2024-03-11T21:42:45.102 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.0021650860000000001 2024-03-11T21:42:45.102 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:42:45.102 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:42:45.102 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "started", 2024-03-11T21:42:45.102 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:36:34.040879+0000", 2024-03-11T21:42:45.102 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.000121019 2024-03-11T21:42:45.102 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:42:45.102 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:42:45.103 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "waiting for subops from 1", 2024-03-11T21:42:45.103 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:36:34.041015+0000", 2024-03-11T21:42:45.103 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.00013673799999999999 2024-03-11T21:42:45.103 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:42:45.103 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:42:45.103 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "reached_pg", 2024-03-11T21:42:45.103 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:36:34.058475+0000", 2024-03-11T21:42:45.103 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.017459204999999998 2024-03-11T21:42:45.103 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:42:45.103 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:42:45.103 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "waiting for peered", 2024-03-11T21:42:45.103 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:36:34.058485+0000", 2024-03-11T21:42:45.103 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.0876999999999999e-05 2024-03-11T21:42:45.103 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:42:45.103 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:42:45.104 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "reached_pg", 2024-03-11T21:42:45.104 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:36:35.046859+0000", 2024-03-11T21:42:45.104 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.98837370899999999 2024-03-11T21:42:45.104 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:42:45.104 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:42:45.104 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "waiting for peered", 2024-03-11T21:42:45.104 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:36:35.046870+0000", 2024-03-11T21:42:45.104 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.0985000000000001e-05 2024-03-11T21:42:45.104 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:42:45.104 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:42:45.104 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "reached_pg", 2024-03-11T21:42:45.104 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:36:36.523712+0000", 2024-03-11T21:42:45.104 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.4768413869999999 2024-03-11T21:42:45.104 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:42:45.105 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:42:45.105 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "waiting for degraded object", 2024-03-11T21:42:45.105 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:36:36.524491+0000", 2024-03-11T21:42:45.105 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.00077957099999999995 2024-03-11T21:42:45.105 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:42:45.105 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:42:45.105 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "reached_pg", 2024-03-11T21:42:45.105 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:36:36.546182+0000", 2024-03-11T21:42:45.105 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.021691298000000001 2024-03-11T21:42:45.105 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:42:45.105 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:42:45.105 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "done", 2024-03-11T21:42:45.105 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:36:36.546286+0000", 2024-03-11T21:42:45.105 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.000103367 2024-03-11T21:42:45.105 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-03-11T21:42:45.106 INFO:teuthology.orchestra.run.smithi097.stdout: ] 2024-03-11T21:42:45.106 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-03-11T21:42:45.106 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:42:45.106 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:42:45.106 INFO:teuthology.orchestra.run.smithi097.stdout: "description": "osd_op(client.25282.0:76296 9.f 9:f93c9389:::benchmark_data_smithi092_130291_object76295:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e277)", 2024-03-11T21:42:45.106 INFO:teuthology.orchestra.run.smithi097.stdout: "initiated_at": "2024-03-11T21:36:34.055957+0000", 2024-03-11T21:42:45.106 INFO:teuthology.orchestra.run.smithi097.stdout: "age": 370.77253568499998, 2024-03-11T21:42:45.106 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 2.4942122050000002, 2024-03-11T21:42:45.106 INFO:teuthology.orchestra.run.smithi097.stdout: "type_data": { 2024-03-11T21:42:45.106 INFO:teuthology.orchestra.run.smithi097.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-11T21:42:45.106 INFO:teuthology.orchestra.run.smithi097.stdout: "client_info": { 2024-03-11T21:42:45.106 INFO:teuthology.orchestra.run.smithi097.stdout: "client": "client.25282", 2024-03-11T21:42:45.106 INFO:teuthology.orchestra.run.smithi097.stdout: "client_addr": "v1:172.21.15.92:0/2489519821", 2024-03-11T21:42:45.106 INFO:teuthology.orchestra.run.smithi097.stdout: "tid": 76296 2024-03-11T21:42:45.107 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:42:45.107 INFO:teuthology.orchestra.run.smithi097.stdout: "events": [ 2024-03-11T21:42:45.107 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:42:45.107 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "initiated", 2024-03-11T21:42:45.107 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:36:34.055957+0000", 2024-03-11T21:42:45.107 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-03-11T21:42:45.107 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:42:45.107 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:42:45.107 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "header_read", 2024-03-11T21:42:45.107 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:36:34.055957+0000", 2024-03-11T21:42:45.107 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-03-11T21:42:45.107 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:42:45.107 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:42:45.107 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "throttled", 2024-03-11T21:42:45.107 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:36:34.055960+0000", 2024-03-11T21:42:45.107 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 2.2800000000000002e-06 2024-03-11T21:42:45.107 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:42:45.108 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:42:45.108 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "all_read", 2024-03-11T21:42:45.108 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:36:34.055993+0000", 2024-03-11T21:42:45.108 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 3.2874999999999999e-05 2024-03-11T21:42:45.108 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:42:45.108 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:42:45.108 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "dispatched", 2024-03-11T21:42:45.108 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:36:34.056006+0000", 2024-03-11T21:42:45.108 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.2979999999999999e-05 2024-03-11T21:42:45.108 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:42:45.108 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:42:45.108 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "queued_for_pg", 2024-03-11T21:42:45.108 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:36:34.056015+0000", 2024-03-11T21:42:45.108 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 9.3009999999999995e-06 2024-03-11T21:42:45.108 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:42:45.108 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:42:45.109 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "reached_pg", 2024-03-11T21:42:45.109 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:36:34.058531+0000", 2024-03-11T21:42:45.109 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.0025166609999999999 2024-03-11T21:42:45.109 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:42:45.109 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:42:45.109 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "waiting for peered", 2024-03-11T21:42:45.109 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:36:34.058540+0000", 2024-03-11T21:42:45.109 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 8.9390000000000003e-06 2024-03-11T21:42:45.109 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:42:45.109 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:42:45.109 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "reached_pg", 2024-03-11T21:42:45.109 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:36:35.046903+0000", 2024-03-11T21:42:45.109 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.98836246699999997 2024-03-11T21:42:45.109 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:42:45.110 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:42:45.110 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "waiting for peered", 2024-03-11T21:42:45.110 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:36:35.046911+0000", 2024-03-11T21:42:45.110 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 8.5900000000000008e-06 2024-03-11T21:42:45.110 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:42:45.110 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:42:45.110 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "reached_pg", 2024-03-11T21:42:45.110 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:36:36.524565+0000", 2024-03-11T21:42:45.110 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.4776533839999999 2024-03-11T21:42:45.110 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:42:45.110 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:42:45.110 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "started", 2024-03-11T21:42:45.110 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:36:36.524737+0000", 2024-03-11T21:42:45.110 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.00017203499999999999 2024-03-11T21:42:45.111 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:42:45.111 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:42:45.111 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "waiting for subops from 1,3", 2024-03-11T21:42:45.111 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:36:36.524977+0000", 2024-03-11T21:42:45.111 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.00023978000000000001 2024-03-11T21:42:45.111 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:42:45.111 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:42:45.111 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "op_commit", 2024-03-11T21:42:45.111 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:36:36.532361+0000", 2024-03-11T21:42:45.111 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.0073842630000000003 2024-03-11T21:42:45.111 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:42:45.111 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:42:45.111 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "sub_op_commit_rec", 2024-03-11T21:42:45.111 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:36:36.546393+0000", 2024-03-11T21:42:45.111 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.014032282 2024-03-11T21:42:45.112 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:42:45.112 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:42:45.112 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "sub_op_commit_rec", 2024-03-11T21:42:45.112 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:36:36.546479+0000", 2024-03-11T21:42:45.112 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 8.5855999999999999e-05 2024-03-11T21:42:45.112 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:42:45.112 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:42:45.112 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "commit_sent", 2024-03-11T21:42:45.112 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:36:36.546554+0000", 2024-03-11T21:42:45.112 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 7.4998000000000003e-05 2024-03-11T21:42:45.112 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:42:45.112 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:42:45.112 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "done", 2024-03-11T21:42:45.113 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:36:36.550170+0000", 2024-03-11T21:42:45.113 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.0036155139999999998 2024-03-11T21:42:45.113 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-03-11T21:42:45.113 INFO:teuthology.orchestra.run.smithi097.stdout: ] 2024-03-11T21:42:45.113 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-03-11T21:42:45.113 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:42:45.113 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:42:45.113 INFO:teuthology.orchestra.run.smithi097.stdout: "description": "osd_op(client.25282.0:76298 9.f 9:f515a6f8:::benchmark_data_smithi092_130291_object76297:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e277)", 2024-03-11T21:42:45.113 INFO:teuthology.orchestra.run.smithi097.stdout: "initiated_at": "2024-03-11T21:36:34.059383+0000", 2024-03-11T21:42:45.113 INFO:teuthology.orchestra.run.smithi097.stdout: "age": 370.76911007799998, 2024-03-11T21:42:45.113 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 2.4975940599999999, 2024-03-11T21:42:45.113 INFO:teuthology.orchestra.run.smithi097.stdout: "type_data": { 2024-03-11T21:42:45.113 INFO:teuthology.orchestra.run.smithi097.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-11T21:42:45.113 INFO:teuthology.orchestra.run.smithi097.stdout: "client_info": { 2024-03-11T21:42:45.114 INFO:teuthology.orchestra.run.smithi097.stdout: "client": "client.25282", 2024-03-11T21:42:45.114 INFO:teuthology.orchestra.run.smithi097.stdout: "client_addr": "v1:172.21.15.92:0/2489519821", 2024-03-11T21:42:45.114 INFO:teuthology.orchestra.run.smithi097.stdout: "tid": 76298 2024-03-11T21:42:45.114 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:42:45.114 INFO:teuthology.orchestra.run.smithi097.stdout: "events": [ 2024-03-11T21:42:45.114 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:42:45.114 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "initiated", 2024-03-11T21:42:45.114 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:36:34.059383+0000", 2024-03-11T21:42:45.114 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-03-11T21:42:45.114 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:42:45.114 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:42:45.114 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "header_read", 2024-03-11T21:42:45.114 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:36:34.059383+0000", 2024-03-11T21:42:45.114 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-03-11T21:42:45.114 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:42:45.115 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:42:45.115 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "throttled", 2024-03-11T21:42:45.115 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:36:34.059385+0000", 2024-03-11T21:42:45.115 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.7099999999999999e-06 2024-03-11T21:42:45.115 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:42:45.115 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:42:45.115 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "all_read", 2024-03-11T21:42:45.115 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:36:34.059449+0000", 2024-03-11T21:42:45.115 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 6.4513999999999999e-05 2024-03-11T21:42:45.115 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:42:45.115 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:42:45.115 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "dispatched", 2024-03-11T21:42:45.115 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:36:34.059459+0000", 2024-03-11T21:42:45.115 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.0175e-05 2024-03-11T21:42:45.115 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:42:45.116 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:42:45.116 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "queued_for_pg", 2024-03-11T21:42:45.116 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:36:34.059467+0000", 2024-03-11T21:42:45.116 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 7.2790000000000001e-06 2024-03-11T21:42:45.116 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:42:45.116 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:42:45.116 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "reached_pg", 2024-03-11T21:42:45.116 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:36:34.059525+0000", 2024-03-11T21:42:45.116 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 5.8496999999999999e-05 2024-03-11T21:42:45.116 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:42:45.116 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:42:45.116 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "waiting for peered", 2024-03-11T21:42:45.116 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:36:34.059534+0000", 2024-03-11T21:42:45.116 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 9.1519999999999998e-06 2024-03-11T21:42:45.116 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:42:45.117 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:42:45.117 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "reached_pg", 2024-03-11T21:42:45.117 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:36:35.046940+0000", 2024-03-11T21:42:45.117 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.98740518300000002 2024-03-11T21:42:45.117 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:42:45.117 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:42:45.117 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "waiting for peered", 2024-03-11T21:42:45.117 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:36:35.046947+0000", 2024-03-11T21:42:45.117 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 7.9759999999999995e-06 2024-03-11T21:42:45.117 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:42:45.117 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:42:45.117 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "reached_pg", 2024-03-11T21:42:45.117 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:36:36.528284+0000", 2024-03-11T21:42:45.117 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.481336177 2024-03-11T21:42:45.118 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:42:45.118 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:42:45.118 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "started", 2024-03-11T21:42:45.118 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:36:36.528443+0000", 2024-03-11T21:42:45.118 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.000159209 2024-03-11T21:42:45.118 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:42:45.118 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:42:45.118 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "waiting for subops from 1,3", 2024-03-11T21:42:45.118 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:36:36.528658+0000", 2024-03-11T21:42:45.118 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.00021467900000000001 2024-03-11T21:42:45.118 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:42:45.118 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:42:45.118 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "op_commit", 2024-03-11T21:42:45.118 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:36:36.532370+0000", 2024-03-11T21:42:45.118 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.0037121099999999998 2024-03-11T21:42:45.119 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:42:45.119 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:42:45.119 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "sub_op_commit_rec", 2024-03-11T21:42:45.119 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:36:36.546446+0000", 2024-03-11T21:42:45.119 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.014075966000000001 2024-03-11T21:42:45.119 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:42:45.119 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:42:45.119 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "sub_op_commit_rec", 2024-03-11T21:42:45.119 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:36:36.553366+0000", 2024-03-11T21:42:45.119 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.0069203570000000002 2024-03-11T21:42:45.119 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:42:45.119 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:42:45.119 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "commit_sent", 2024-03-11T21:42:45.119 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:36:36.553421+0000", 2024-03-11T21:42:45.119 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 5.5148000000000001e-05 2024-03-11T21:42:45.120 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:42:45.120 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:42:45.120 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "done", 2024-03-11T21:42:45.120 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:36:36.556977+0000", 2024-03-11T21:42:45.120 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.0035559279999999999 2024-03-11T21:42:45.120 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-03-11T21:42:45.120 INFO:teuthology.orchestra.run.smithi097.stdout: ] 2024-03-11T21:42:45.120 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-03-11T21:42:45.120 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:42:45.120 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:42:45.120 INFO:teuthology.orchestra.run.smithi097.stdout: "description": "osd_op(client.25282.0:76300 9.f 9:f8b776ca:::benchmark_data_smithi092_130291_object76299:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e277)", 2024-03-11T21:42:45.120 INFO:teuthology.orchestra.run.smithi097.stdout: "initiated_at": "2024-03-11T21:36:34.064758+0000", 2024-03-11T21:42:45.120 INFO:teuthology.orchestra.run.smithi097.stdout: "age": 370.76373514400001, 2024-03-11T21:42:45.120 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 2.5087678960000002, 2024-03-11T21:42:45.120 INFO:teuthology.orchestra.run.smithi097.stdout: "type_data": { 2024-03-11T21:42:45.121 INFO:teuthology.orchestra.run.smithi097.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-11T21:42:45.121 INFO:teuthology.orchestra.run.smithi097.stdout: "client_info": { 2024-03-11T21:42:45.121 INFO:teuthology.orchestra.run.smithi097.stdout: "client": "client.25282", 2024-03-11T21:42:45.121 INFO:teuthology.orchestra.run.smithi097.stdout: "client_addr": "v1:172.21.15.92:0/2489519821", 2024-03-11T21:42:45.121 INFO:teuthology.orchestra.run.smithi097.stdout: "tid": 76300 2024-03-11T21:42:45.121 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:42:45.121 INFO:teuthology.orchestra.run.smithi097.stdout: "events": [ 2024-03-11T21:42:45.121 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:42:45.121 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "initiated", 2024-03-11T21:42:45.121 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:36:34.064758+0000", 2024-03-11T21:42:45.121 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-03-11T21:42:45.121 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:42:45.121 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:42:45.121 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "header_read", 2024-03-11T21:42:45.121 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:36:34.064758+0000", 2024-03-11T21:42:45.122 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-03-11T21:42:45.122 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:42:45.122 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:42:45.122 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "throttled", 2024-03-11T21:42:45.122 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:36:34.064759+0000", 2024-03-11T21:42:45.122 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.012e-06 2024-03-11T21:42:45.122 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:42:45.122 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:42:45.122 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "all_read", 2024-03-11T21:42:45.122 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:36:34.064790+0000", 2024-03-11T21:42:45.122 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 3.0901000000000001e-05 2024-03-11T21:42:45.122 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:42:45.122 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:42:45.122 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "dispatched", 2024-03-11T21:42:45.122 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:36:34.064799+0000", 2024-03-11T21:42:45.123 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 9.1190000000000003e-06 2024-03-11T21:42:45.123 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:42:45.123 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:42:45.123 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "queued_for_pg", 2024-03-11T21:42:45.123 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:36:34.064807+0000", 2024-03-11T21:42:45.123 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 8.3610000000000001e-06 2024-03-11T21:42:45.123 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:42:45.123 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:42:45.123 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "reached_pg", 2024-03-11T21:42:45.123 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:36:34.064853+0000", 2024-03-11T21:42:45.123 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 4.5867e-05 2024-03-11T21:42:45.123 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:42:45.123 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:42:45.123 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "waiting for peered", 2024-03-11T21:42:45.123 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:36:34.064862+0000", 2024-03-11T21:42:45.124 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 8.8599999999999999e-06 2024-03-11T21:42:45.124 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:42:45.124 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:42:45.124 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "reached_pg", 2024-03-11T21:42:45.124 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:36:35.046976+0000", 2024-03-11T21:42:45.124 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.98211411100000001 2024-03-11T21:42:45.124 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:42:45.124 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:42:45.124 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "waiting for peered", 2024-03-11T21:42:45.124 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:36:35.046984+0000", 2024-03-11T21:42:45.124 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 7.9449999999999993e-06 2024-03-11T21:42:45.124 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:42:45.124 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:42:45.124 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "reached_pg", 2024-03-11T21:42:45.125 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:36:36.550220+0000", 2024-03-11T21:42:45.125 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.5032355509999999 2024-03-11T21:42:45.125 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:42:45.125 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:42:45.125 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "started", 2024-03-11T21:42:45.125 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:36:36.550382+0000", 2024-03-11T21:42:45.125 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.00016266499999999999 2024-03-11T21:42:45.125 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:42:45.125 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:42:45.125 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "waiting for subops from 1,3", 2024-03-11T21:42:45.125 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:36:36.550548+0000", 2024-03-11T21:42:45.125 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.00016551599999999999 2024-03-11T21:42:45.125 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:42:45.125 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:42:45.125 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "sub_op_commit_rec", 2024-03-11T21:42:45.125 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:36:36.560198+0000", 2024-03-11T21:42:45.126 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.009649899 2024-03-11T21:42:45.126 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:42:45.126 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:42:45.126 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "sub_op_commit_rec", 2024-03-11T21:42:45.126 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:36:36.560301+0000", 2024-03-11T21:42:45.126 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.000103558 2024-03-11T21:42:45.126 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:42:45.126 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:42:45.126 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "op_commit", 2024-03-11T21:42:45.126 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:36:36.569928+0000", 2024-03-11T21:42:45.126 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.0096262550000000002 2024-03-11T21:42:45.126 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:42:45.126 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:42:45.126 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "commit_sent", 2024-03-11T21:42:45.127 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:36:36.569984+0000", 2024-03-11T21:42:45.127 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 5.6273999999999997e-05 2024-03-11T21:42:45.127 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:42:45.127 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:42:45.127 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "done", 2024-03-11T21:42:45.127 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:36:36.573526+0000", 2024-03-11T21:42:45.127 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.0035420019999999998 2024-03-11T21:42:45.127 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-03-11T21:42:45.127 INFO:teuthology.orchestra.run.smithi097.stdout: ] 2024-03-11T21:42:45.127 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-03-11T21:42:45.127 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:42:45.127 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:42:45.127 INFO:teuthology.orchestra.run.smithi097.stdout: "description": "osd_op(client.25282.0:76308 9.f 9:fb5a14a2:::benchmark_data_smithi092_130291_object76307:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e277)", 2024-03-11T21:42:45.127 INFO:teuthology.orchestra.run.smithi097.stdout: "initiated_at": "2024-03-11T21:36:34.074082+0000", 2024-03-11T21:42:45.127 INFO:teuthology.orchestra.run.smithi097.stdout: "age": 370.75441066500002, 2024-03-11T21:42:45.128 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 2.503648359, 2024-03-11T21:42:45.128 INFO:teuthology.orchestra.run.smithi097.stdout: "type_data": { 2024-03-11T21:42:45.128 INFO:teuthology.orchestra.run.smithi097.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-11T21:42:45.128 INFO:teuthology.orchestra.run.smithi097.stdout: "client_info": { 2024-03-11T21:42:45.128 INFO:teuthology.orchestra.run.smithi097.stdout: "client": "client.25282", 2024-03-11T21:42:45.128 INFO:teuthology.orchestra.run.smithi097.stdout: "client_addr": "v1:172.21.15.92:0/2489519821", 2024-03-11T21:42:45.128 INFO:teuthology.orchestra.run.smithi097.stdout: "tid": 76308 2024-03-11T21:42:45.128 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:42:45.128 INFO:teuthology.orchestra.run.smithi097.stdout: "events": [ 2024-03-11T21:42:45.128 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:42:45.128 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "initiated", 2024-03-11T21:42:45.128 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:36:34.074082+0000", 2024-03-11T21:42:45.128 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-03-11T21:42:45.128 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:42:45.128 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:42:45.129 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "header_read", 2024-03-11T21:42:45.129 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:36:34.074082+0000", 2024-03-11T21:42:45.129 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-03-11T21:42:45.129 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:42:45.129 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:42:45.129 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "throttled", 2024-03-11T21:42:45.129 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:36:34.074085+0000", 2024-03-11T21:42:45.129 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 2.323e-06 2024-03-11T21:42:45.129 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:42:45.129 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:42:45.129 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "all_read", 2024-03-11T21:42:45.129 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:36:34.074145+0000", 2024-03-11T21:42:45.129 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 6.0176000000000001e-05 2024-03-11T21:42:45.129 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:42:45.130 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:42:45.130 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "dispatched", 2024-03-11T21:42:45.130 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:36:34.074158+0000", 2024-03-11T21:42:45.130 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.2911000000000001e-05 2024-03-11T21:42:45.130 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:42:45.130 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:42:45.130 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "queued_for_pg", 2024-03-11T21:42:45.130 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:36:34.074164+0000", 2024-03-11T21:42:45.130 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 5.9490000000000001e-06 2024-03-11T21:42:45.130 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:42:45.130 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:42:45.130 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "reached_pg", 2024-03-11T21:42:45.130 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:36:34.074208+0000", 2024-03-11T21:42:45.130 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 4.4642999999999999e-05 2024-03-11T21:42:45.130 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:42:45.131 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:42:45.131 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "waiting for peered", 2024-03-11T21:42:45.131 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:36:34.074215+0000", 2024-03-11T21:42:45.131 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 7.0010000000000004e-06 2024-03-11T21:42:45.131 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:42:45.131 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:42:45.131 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "reached_pg", 2024-03-11T21:42:45.131 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:36:35.047052+0000", 2024-03-11T21:42:45.131 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.97283687399999996 2024-03-11T21:42:45.131 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:42:45.131 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:42:45.131 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "waiting for peered", 2024-03-11T21:42:45.131 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:36:35.047063+0000", 2024-03-11T21:42:45.131 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.1052e-05 2024-03-11T21:42:45.131 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:42:45.132 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:42:45.132 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "reached_pg", 2024-03-11T21:42:45.132 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:36:36.557028+0000", 2024-03-11T21:42:45.132 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.5099643540000001 2024-03-11T21:42:45.132 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:42:45.132 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:42:45.132 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "started", 2024-03-11T21:42:45.132 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:36:36.557189+0000", 2024-03-11T21:42:45.132 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.00016171499999999999 2024-03-11T21:42:45.132 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:42:45.132 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:42:45.132 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "waiting for subops from 1,3", 2024-03-11T21:42:45.132 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:36:36.557367+0000", 2024-03-11T21:42:45.132 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.000177238 2024-03-11T21:42:45.132 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:42:45.133 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:42:45.133 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "sub_op_commit_rec", 2024-03-11T21:42:45.133 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:36:36.567005+0000", 2024-03-11T21:42:45.133 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.0096378829999999999 2024-03-11T21:42:45.133 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:42:45.133 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:42:45.133 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "op_commit", 2024-03-11T21:42:45.133 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:36:36.573545+0000", 2024-03-11T21:42:45.133 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.0065399819999999997 2024-03-11T21:42:45.133 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:42:45.133 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:42:45.133 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "sub_op_commit_rec", 2024-03-11T21:42:45.133 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:36:36.574036+0000", 2024-03-11T21:42:45.133 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.00049146699999999999 2024-03-11T21:42:45.133 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:42:45.134 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:42:45.134 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "commit_sent", 2024-03-11T21:42:45.134 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:36:36.574095+0000", 2024-03-11T21:42:45.134 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 5.8768999999999999e-05 2024-03-11T21:42:45.134 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:42:45.134 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:42:45.134 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "done", 2024-03-11T21:42:45.134 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:36:36.577731+0000", 2024-03-11T21:42:45.134 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.003636022 2024-03-11T21:42:45.134 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-03-11T21:42:45.134 INFO:teuthology.orchestra.run.smithi097.stdout: ] 2024-03-11T21:42:45.134 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-03-11T21:42:45.134 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:42:45.134 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:42:45.135 INFO:teuthology.orchestra.run.smithi097.stdout: "description": "osd_op(client.15744.0:69277 10.9 10:92e95475:::benchmark_data_smithi092_161462_object69276:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e372)", 2024-03-11T21:42:45.135 INFO:teuthology.orchestra.run.smithi097.stdout: "initiated_at": "2024-03-11T21:41:27.829175+0000", 2024-03-11T21:42:45.135 INFO:teuthology.orchestra.run.smithi097.stdout: "age": 76.999318575000004, 2024-03-11T21:42:45.135 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.99767732799999997, 2024-03-11T21:42:45.135 INFO:teuthology.orchestra.run.smithi097.stdout: "type_data": { 2024-03-11T21:42:45.135 INFO:teuthology.orchestra.run.smithi097.stdout: "flag_point": "reached pg", 2024-03-11T21:42:45.135 INFO:teuthology.orchestra.run.smithi097.stdout: "client_info": { 2024-03-11T21:42:45.135 INFO:teuthology.orchestra.run.smithi097.stdout: "client": "client.15744", 2024-03-11T21:42:45.135 INFO:teuthology.orchestra.run.smithi097.stdout: "client_addr": "v1:172.21.15.92:0/3334000812", 2024-03-11T21:42:45.135 INFO:teuthology.orchestra.run.smithi097.stdout: "tid": 69277 2024-03-11T21:42:45.135 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:42:45.135 INFO:teuthology.orchestra.run.smithi097.stdout: "events": [ 2024-03-11T21:42:45.135 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:42:45.135 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "initiated", 2024-03-11T21:42:45.136 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:41:27.829175+0000", 2024-03-11T21:42:45.136 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-03-11T21:42:45.136 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:42:45.136 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:42:45.136 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "header_read", 2024-03-11T21:42:45.136 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:41:27.829175+0000", 2024-03-11T21:42:45.136 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-03-11T21:42:45.136 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:42:45.136 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:42:45.136 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "throttled", 2024-03-11T21:42:45.136 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:41:27.829179+0000", 2024-03-11T21:42:45.136 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 4.3080000000000001e-06 2024-03-11T21:42:45.136 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:42:45.136 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:42:45.137 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "all_read", 2024-03-11T21:42:45.137 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:41:27.829232+0000", 2024-03-11T21:42:45.137 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 5.3551999999999998e-05 2024-03-11T21:42:45.137 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:42:45.137 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:42:45.137 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "dispatched", 2024-03-11T21:42:45.137 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:41:27.829294+0000", 2024-03-11T21:42:45.137 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 6.2012000000000002e-05 2024-03-11T21:42:45.137 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:42:45.137 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:42:45.137 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "queued_for_pg", 2024-03-11T21:42:45.137 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:41:27.829309+0000", 2024-03-11T21:42:45.137 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.4506e-05 2024-03-11T21:42:45.137 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:42:45.137 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:42:45.138 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "reached_pg", 2024-03-11T21:42:45.138 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:41:27.977027+0000", 2024-03-11T21:42:45.138 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.147718087 2024-03-11T21:42:45.138 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:42:45.138 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:42:45.138 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "waiting for peered", 2024-03-11T21:42:45.138 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:41:27.977039+0000", 2024-03-11T21:42:45.138 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.2295e-05 2024-03-11T21:42:45.138 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:42:45.138 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:42:45.138 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "reached_pg", 2024-03-11T21:42:45.138 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:41:28.826770+0000", 2024-03-11T21:42:45.138 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.84973065199999998 2024-03-11T21:42:45.138 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:42:45.138 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:42:45.139 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "done", 2024-03-11T21:42:45.139 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:41:28.826852+0000", 2024-03-11T21:42:45.139 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 8.1916000000000006e-05 2024-03-11T21:42:45.139 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-03-11T21:42:45.139 INFO:teuthology.orchestra.run.smithi097.stdout: ] 2024-03-11T21:42:45.139 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-03-11T21:42:45.139 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:42:45.139 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:42:45.139 INFO:teuthology.orchestra.run.smithi097.stdout: "description": "osd_op(client.15744.0:69287 10.9 10:93b721b6:::benchmark_data_smithi092_161462_object69286:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e372)", 2024-03-11T21:42:45.139 INFO:teuthology.orchestra.run.smithi097.stdout: "initiated_at": "2024-03-11T21:41:27.829334+0000", 2024-03-11T21:42:45.139 INFO:teuthology.orchestra.run.smithi097.stdout: "age": 76.999159573, 2024-03-11T21:42:45.139 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.99759708400000002, 2024-03-11T21:42:45.139 INFO:teuthology.orchestra.run.smithi097.stdout: "type_data": { 2024-03-11T21:42:45.140 INFO:teuthology.orchestra.run.smithi097.stdout: "flag_point": "reached pg", 2024-03-11T21:42:45.140 INFO:teuthology.orchestra.run.smithi097.stdout: "client_info": { 2024-03-11T21:42:45.140 INFO:teuthology.orchestra.run.smithi097.stdout: "client": "client.15744", 2024-03-11T21:42:45.140 INFO:teuthology.orchestra.run.smithi097.stdout: "client_addr": "v1:172.21.15.92:0/3334000812", 2024-03-11T21:42:45.140 INFO:teuthology.orchestra.run.smithi097.stdout: "tid": 69287 2024-03-11T21:42:45.140 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:42:45.140 INFO:teuthology.orchestra.run.smithi097.stdout: "events": [ 2024-03-11T21:42:45.140 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:42:45.140 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "initiated", 2024-03-11T21:42:45.140 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:41:27.829334+0000", 2024-03-11T21:42:45.140 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-03-11T21:42:45.140 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:42:45.140 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:42:45.141 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "header_read", 2024-03-11T21:42:45.141 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:41:27.829334+0000", 2024-03-11T21:42:45.141 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-03-11T21:42:45.141 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:42:45.141 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:42:45.141 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "throttled", 2024-03-11T21:42:45.141 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:41:27.829334+0000", 2024-03-11T21:42:45.141 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 9.4200000000000004e-07 2024-03-11T21:42:45.141 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:42:45.141 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:42:45.141 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "all_read", 2024-03-11T21:42:45.141 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:41:27.829385+0000", 2024-03-11T21:42:45.141 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 5.0713e-05 2024-03-11T21:42:45.141 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:42:45.141 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:42:45.142 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "dispatched", 2024-03-11T21:42:45.142 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:41:27.829393+0000", 2024-03-11T21:42:45.142 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 7.7279999999999995e-06 2024-03-11T21:42:45.142 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:42:45.142 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:42:45.142 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "queued_for_pg", 2024-03-11T21:42:45.142 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:41:27.829400+0000", 2024-03-11T21:42:45.142 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 6.6980000000000001e-06 2024-03-11T21:42:45.142 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:42:45.142 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:42:45.142 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "reached_pg", 2024-03-11T21:42:45.142 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:41:27.977066+0000", 2024-03-11T21:42:45.142 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.147666511 2024-03-11T21:42:45.143 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:42:45.143 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:42:45.143 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "waiting for peered", 2024-03-11T21:42:45.143 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:41:27.977073+0000", 2024-03-11T21:42:45.143 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 7.3100000000000003e-06 2024-03-11T21:42:45.143 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:42:45.143 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:42:45.143 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "reached_pg", 2024-03-11T21:42:45.143 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:41:28.826877+0000", 2024-03-11T21:42:45.143 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.84980390699999997 2024-03-11T21:42:45.143 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:42:45.143 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:42:45.143 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "done", 2024-03-11T21:42:45.143 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:41:28.826931+0000", 2024-03-11T21:42:45.143 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 5.3275000000000001e-05 2024-03-11T21:42:45.144 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-03-11T21:42:45.144 INFO:teuthology.orchestra.run.smithi097.stdout: ] 2024-03-11T21:42:45.144 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-03-11T21:42:45.144 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:42:45.144 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:42:45.144 INFO:teuthology.orchestra.run.smithi097.stdout: "description": "osd_op(client.15744.0:69288 10.9 10:92935038:::benchmark_data_smithi092_161462_object69287:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e372)", 2024-03-11T21:42:45.144 INFO:teuthology.orchestra.run.smithi097.stdout: "initiated_at": "2024-03-11T21:41:27.829408+0000", 2024-03-11T21:42:45.144 INFO:teuthology.orchestra.run.smithi097.stdout: "age": 76.999084956999994, 2024-03-11T21:42:45.144 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.015902442, 2024-03-11T21:42:45.144 INFO:teuthology.orchestra.run.smithi097.stdout: "type_data": { 2024-03-11T21:42:45.144 INFO:teuthology.orchestra.run.smithi097.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-11T21:42:45.144 INFO:teuthology.orchestra.run.smithi097.stdout: "client_info": { 2024-03-11T21:42:45.144 INFO:teuthology.orchestra.run.smithi097.stdout: "client": "client.15744", 2024-03-11T21:42:45.144 INFO:teuthology.orchestra.run.smithi097.stdout: "client_addr": "v1:172.21.15.92:0/3334000812", 2024-03-11T21:42:45.145 INFO:teuthology.orchestra.run.smithi097.stdout: "tid": 69288 2024-03-11T21:42:45.145 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:42:45.145 INFO:teuthology.orchestra.run.smithi097.stdout: "events": [ 2024-03-11T21:42:45.145 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:42:45.145 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "initiated", 2024-03-11T21:42:45.145 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:41:27.829408+0000", 2024-03-11T21:42:45.145 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-03-11T21:42:45.145 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:42:45.145 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:42:45.145 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "header_read", 2024-03-11T21:42:45.145 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:41:27.829408+0000", 2024-03-11T21:42:45.145 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-03-11T21:42:45.145 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:42:45.145 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:42:45.145 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "throttled", 2024-03-11T21:42:45.146 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:41:27.829409+0000", 2024-03-11T21:42:45.146 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 7.8800000000000002e-07 2024-03-11T21:42:45.146 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:42:45.146 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:42:45.146 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "all_read", 2024-03-11T21:42:45.146 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:41:27.829459+0000", 2024-03-11T21:42:45.146 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 4.9675999999999997e-05 2024-03-11T21:42:45.146 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:42:45.146 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:42:45.146 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "dispatched", 2024-03-11T21:42:45.146 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:41:27.829465+0000", 2024-03-11T21:42:45.146 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 6.8750000000000002e-06 2024-03-11T21:42:45.146 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:42:45.146 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:42:45.147 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "queued_for_pg", 2024-03-11T21:42:45.147 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:41:27.829472+0000", 2024-03-11T21:42:45.147 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 6.2489999999999998e-06 2024-03-11T21:42:45.147 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:42:45.147 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:42:45.147 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "reached_pg", 2024-03-11T21:42:45.147 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:41:27.977097+0000", 2024-03-11T21:42:45.147 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.147624901 2024-03-11T21:42:45.147 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:42:45.147 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:42:45.147 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "waiting for peered", 2024-03-11T21:42:45.147 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:41:27.977104+0000", 2024-03-11T21:42:45.147 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 7.165e-06 2024-03-11T21:42:45.147 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:42:45.147 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:42:45.148 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "reached_pg", 2024-03-11T21:42:45.148 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:41:28.826951+0000", 2024-03-11T21:42:45.148 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.849847354 2024-03-11T21:42:45.148 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:42:45.148 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:42:45.148 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "started", 2024-03-11T21:42:45.148 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:41:28.827075+0000", 2024-03-11T21:42:45.148 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.00012410899999999999 2024-03-11T21:42:45.148 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:42:45.148 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:42:45.148 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "waiting for subops from 2,7", 2024-03-11T21:42:45.148 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:41:28.827231+0000", 2024-03-11T21:42:45.148 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.00015579 2024-03-11T21:42:45.148 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:42:45.148 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:42:45.149 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "op_commit", 2024-03-11T21:42:45.149 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:41:28.830775+0000", 2024-03-11T21:42:45.149 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.0035436370000000001 2024-03-11T21:42:45.149 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:42:45.149 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:42:45.149 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "sub_op_commit_rec", 2024-03-11T21:42:45.149 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:41:28.837674+0000", 2024-03-11T21:42:45.149 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.0068991429999999999 2024-03-11T21:42:45.149 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:42:45.149 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:42:45.149 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "sub_op_commit_rec", 2024-03-11T21:42:45.149 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:41:28.845212+0000", 2024-03-11T21:42:45.149 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.0075384919999999999 2024-03-11T21:42:45.149 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:42:45.149 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:42:45.150 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "commit_sent", 2024-03-11T21:42:45.150 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:41:28.845269+0000", 2024-03-11T21:42:45.150 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 5.7074000000000003e-05 2024-03-11T21:42:45.150 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:42:45.150 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:42:45.150 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "done", 2024-03-11T21:42:45.150 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:41:28.845311+0000", 2024-03-11T21:42:45.150 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 4.1189000000000002e-05 2024-03-11T21:42:45.150 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-03-11T21:42:45.150 INFO:teuthology.orchestra.run.smithi097.stdout: ] 2024-03-11T21:42:45.150 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-03-11T21:42:45.150 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:42:45.151 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:42:45.151 INFO:teuthology.orchestra.run.smithi097.stdout: "description": "osd_op(client.15744.0:69290 10.6 10:64e7935d:::benchmark_data_smithi092_161462_object69289:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e372)", 2024-03-11T21:42:45.151 INFO:teuthology.orchestra.run.smithi097.stdout: "initiated_at": "2024-03-11T21:41:27.829479+0000", 2024-03-11T21:42:45.151 INFO:teuthology.orchestra.run.smithi097.stdout: "age": 76.999013664000003, 2024-03-11T21:42:45.151 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.0144195920000001, 2024-03-11T21:42:45.151 INFO:teuthology.orchestra.run.smithi097.stdout: "type_data": { 2024-03-11T21:42:45.151 INFO:teuthology.orchestra.run.smithi097.stdout: "flag_point": "reached pg", 2024-03-11T21:42:45.151 INFO:teuthology.orchestra.run.smithi097.stdout: "client_info": { 2024-03-11T21:42:45.151 INFO:teuthology.orchestra.run.smithi097.stdout: "client": "client.15744", 2024-03-11T21:42:45.151 INFO:teuthology.orchestra.run.smithi097.stdout: "client_addr": "v1:172.21.15.92:0/3334000812", 2024-03-11T21:42:45.151 INFO:teuthology.orchestra.run.smithi097.stdout: "tid": 69290 2024-03-11T21:42:45.151 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:42:45.151 INFO:teuthology.orchestra.run.smithi097.stdout: "events": [ 2024-03-11T21:42:45.151 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:42:45.151 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "initiated", 2024-03-11T21:42:45.151 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:41:27.829479+0000", 2024-03-11T21:42:45.152 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-03-11T21:42:45.152 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:42:45.152 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:42:45.152 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "header_read", 2024-03-11T21:42:45.152 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:41:27.829479+0000", 2024-03-11T21:42:45.152 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-03-11T21:42:45.152 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:42:45.152 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:42:45.152 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "throttled", 2024-03-11T21:42:45.152 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:41:27.829480+0000", 2024-03-11T21:42:45.152 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 6.5499999999999998e-07 2024-03-11T21:42:45.152 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:42:45.152 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:42:45.152 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "all_read", 2024-03-11T21:42:45.153 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:41:27.829510+0000", 2024-03-11T21:42:45.153 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 2.9456999999999999e-05 2024-03-11T21:42:45.153 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:42:45.153 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:42:45.153 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "dispatched", 2024-03-11T21:42:45.153 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:41:27.829516+0000", 2024-03-11T21:42:45.153 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 6.4459999999999998e-06 2024-03-11T21:42:45.153 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:42:45.153 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:42:45.153 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "queued_for_pg", 2024-03-11T21:42:45.153 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:41:27.829522+0000", 2024-03-11T21:42:45.153 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 5.6529999999999999e-06 2024-03-11T21:42:45.153 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:42:45.153 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:42:45.153 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "reached_pg", 2024-03-11T21:42:45.154 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:41:28.000484+0000", 2024-03-11T21:42:45.154 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.170962688 2024-03-11T21:42:45.154 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:42:45.154 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:42:45.154 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "waiting for peered", 2024-03-11T21:42:45.154 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:41:28.000496+0000", 2024-03-11T21:42:45.154 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.2018000000000001e-05 2024-03-11T21:42:45.154 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:42:45.154 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:42:45.154 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "reached_pg", 2024-03-11T21:42:45.154 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:41:28.843770+0000", 2024-03-11T21:42:45.154 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.84327316600000002 2024-03-11T21:42:45.154 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:42:45.154 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:42:45.155 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "done", 2024-03-11T21:42:45.155 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:41:28.843899+0000", 2024-03-11T21:42:45.155 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.00012950900000000001 2024-03-11T21:42:45.155 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-03-11T21:42:45.155 INFO:teuthology.orchestra.run.smithi097.stdout: ] 2024-03-11T21:42:45.155 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-03-11T21:42:45.155 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:42:45.155 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:42:45.155 INFO:teuthology.orchestra.run.smithi097.stdout: "description": "osd_op(client.15744.0:70472 10.11 10.4ff1d3d1 (undecoded) ondisk+retry+write+known_if_redirected+supports_pool_eio e374)", 2024-03-11T21:42:45.155 INFO:teuthology.orchestra.run.smithi097.stdout: "initiated_at": "2024-03-11T21:41:29.844450+0000", 2024-03-11T21:42:45.155 INFO:teuthology.orchestra.run.smithi097.stdout: "age": 74.984043471000007, 2024-03-11T21:42:45.155 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.0282935, 2024-03-11T21:42:45.155 INFO:teuthology.orchestra.run.smithi097.stdout: "type_data": { 2024-03-11T21:42:45.155 INFO:teuthology.orchestra.run.smithi097.stdout: "flag_point": "reached pg", 2024-03-11T21:42:45.155 INFO:teuthology.orchestra.run.smithi097.stdout: "client_info": { 2024-03-11T21:42:45.155 INFO:teuthology.orchestra.run.smithi097.stdout: "client": "client.15744", 2024-03-11T21:42:45.156 INFO:teuthology.orchestra.run.smithi097.stdout: "client_addr": "v1:172.21.15.92:0/3334000812", 2024-03-11T21:42:45.156 INFO:teuthology.orchestra.run.smithi097.stdout: "tid": 70472 2024-03-11T21:42:45.156 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:42:45.156 INFO:teuthology.orchestra.run.smithi097.stdout: "events": [ 2024-03-11T21:42:45.156 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:42:45.156 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "initiated", 2024-03-11T21:42:45.156 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:41:29.844450+0000", 2024-03-11T21:42:45.156 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-03-11T21:42:45.156 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:42:45.156 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:42:45.156 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "header_read", 2024-03-11T21:42:45.156 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:41:29.844450+0000", 2024-03-11T21:42:45.156 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-03-11T21:42:45.156 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:42:45.156 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:42:45.157 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "throttled", 2024-03-11T21:42:45.157 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:41:29.844452+0000", 2024-03-11T21:42:45.157 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 2.0930000000000001e-06 2024-03-11T21:42:45.157 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:42:45.157 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:42:45.157 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "all_read", 2024-03-11T21:42:45.157 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:41:29.844516+0000", 2024-03-11T21:42:45.157 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 6.3874999999999994e-05 2024-03-11T21:42:45.157 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:42:45.157 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:42:45.157 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "dispatched", 2024-03-11T21:42:45.157 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:41:29.844533+0000", 2024-03-11T21:42:45.157 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.7371e-05 2024-03-11T21:42:45.157 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:42:45.157 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:42:45.158 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "queued_for_pg", 2024-03-11T21:42:45.158 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:41:29.844540+0000", 2024-03-11T21:42:45.158 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 7.1339999999999998e-06 2024-03-11T21:42:45.158 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:42:45.158 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:42:45.158 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "reached_pg", 2024-03-11T21:42:45.158 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:41:29.844595+0000", 2024-03-11T21:42:45.158 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 5.5065000000000002e-05 2024-03-11T21:42:45.158 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:42:45.158 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:42:45.158 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "waiting for peered", 2024-03-11T21:42:45.158 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:41:29.844604+0000", 2024-03-11T21:42:45.158 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 8.524e-06 2024-03-11T21:42:45.158 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:42:45.159 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:42:45.159 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "reached_pg", 2024-03-11T21:42:45.159 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:41:30.872734+0000", 2024-03-11T21:42:45.159 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.028130035 2024-03-11T21:42:45.159 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:42:45.159 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:42:45.159 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "done", 2024-03-11T21:42:45.159 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:41:30.872743+0000", 2024-03-11T21:42:45.159 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 9.4029999999999996e-06 2024-03-11T21:42:45.159 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-03-11T21:42:45.159 INFO:teuthology.orchestra.run.smithi097.stdout: ] 2024-03-11T21:42:45.159 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-03-11T21:42:45.159 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:42:45.159 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:42:45.159 INFO:teuthology.orchestra.run.smithi097.stdout: "description": "osd_op(client.15744.0:70484 10.11 10.b04c4bd1 (undecoded) ondisk+retry+write+known_if_redirected+supports_pool_eio e374)", 2024-03-11T21:42:45.160 INFO:teuthology.orchestra.run.smithi097.stdout: "initiated_at": "2024-03-11T21:41:29.844556+0000", 2024-03-11T21:42:45.160 INFO:teuthology.orchestra.run.smithi097.stdout: "age": 74.983937561999994, 2024-03-11T21:42:45.160 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.028216705, 2024-03-11T21:42:45.160 INFO:teuthology.orchestra.run.smithi097.stdout: "type_data": { 2024-03-11T21:42:45.160 INFO:teuthology.orchestra.run.smithi097.stdout: "flag_point": "reached pg", 2024-03-11T21:42:45.160 INFO:teuthology.orchestra.run.smithi097.stdout: "client_info": { 2024-03-11T21:42:45.160 INFO:teuthology.orchestra.run.smithi097.stdout: "client": "client.15744", 2024-03-11T21:42:45.160 INFO:teuthology.orchestra.run.smithi097.stdout: "client_addr": "v1:172.21.15.92:0/3334000812", 2024-03-11T21:42:45.160 INFO:teuthology.orchestra.run.smithi097.stdout: "tid": 70484 2024-03-11T21:42:45.160 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:42:45.160 INFO:teuthology.orchestra.run.smithi097.stdout: "events": [ 2024-03-11T21:42:45.160 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:42:45.160 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "initiated", 2024-03-11T21:42:45.160 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:41:29.844556+0000", 2024-03-11T21:42:45.160 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-03-11T21:42:45.161 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:42:45.161 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:42:45.161 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "header_read", 2024-03-11T21:42:45.161 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:41:29.844556+0000", 2024-03-11T21:42:45.161 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-03-11T21:42:45.161 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:42:45.161 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:42:45.161 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "throttled", 2024-03-11T21:42:45.161 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:41:29.844556+0000", 2024-03-11T21:42:45.161 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 8.0800000000000004e-07 2024-03-11T21:42:45.161 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:42:45.161 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:42:45.161 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "all_read", 2024-03-11T21:42:45.161 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:41:29.844612+0000", 2024-03-11T21:42:45.161 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 5.5507999999999998e-05 2024-03-11T21:42:45.162 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:42:45.162 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:42:45.162 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "dispatched", 2024-03-11T21:42:45.162 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:41:29.844618+0000", 2024-03-11T21:42:45.162 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 6.5699999999999998e-06 2024-03-11T21:42:45.162 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:42:45.162 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:42:45.162 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "queued_for_pg", 2024-03-11T21:42:45.162 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:41:29.844623+0000", 2024-03-11T21:42:45.162 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 4.6140000000000002e-06 2024-03-11T21:42:45.162 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:42:45.162 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:42:45.162 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "reached_pg", 2024-03-11T21:42:45.162 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:41:29.844659+0000", 2024-03-11T21:42:45.163 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 3.6102000000000001e-05 2024-03-11T21:42:45.163 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:42:45.163 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:42:45.163 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "waiting for peered", 2024-03-11T21:42:45.163 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:41:29.844665+0000", 2024-03-11T21:42:45.163 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 5.84e-06 2024-03-11T21:42:45.163 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:42:45.163 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:42:45.163 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "reached_pg", 2024-03-11T21:42:45.163 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:41:30.872766+0000", 2024-03-11T21:42:45.163 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.028101368 2024-03-11T21:42:45.163 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:42:45.163 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:42:45.163 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "done", 2024-03-11T21:42:45.163 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:41:30.872772+0000", 2024-03-11T21:42:45.163 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 5.8950000000000003e-06 2024-03-11T21:42:45.164 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-03-11T21:42:45.164 INFO:teuthology.orchestra.run.smithi097.stdout: ] 2024-03-11T21:42:45.164 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-03-11T21:42:45.164 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:42:45.164 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:42:45.164 INFO:teuthology.orchestra.run.smithi097.stdout: "description": "osd_op(client.15744.0:70492 10.11 10.25e828f1 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e374)", 2024-03-11T21:42:45.164 INFO:teuthology.orchestra.run.smithi097.stdout: "initiated_at": "2024-03-11T21:41:29.854518+0000", 2024-03-11T21:42:45.164 INFO:teuthology.orchestra.run.smithi097.stdout: "age": 74.973974882999997, 2024-03-11T21:42:45.164 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.018270996, 2024-03-11T21:42:45.164 INFO:teuthology.orchestra.run.smithi097.stdout: "type_data": { 2024-03-11T21:42:45.164 INFO:teuthology.orchestra.run.smithi097.stdout: "flag_point": "reached pg", 2024-03-11T21:42:45.164 INFO:teuthology.orchestra.run.smithi097.stdout: "client_info": { 2024-03-11T21:42:45.164 INFO:teuthology.orchestra.run.smithi097.stdout: "client": "client.15744", 2024-03-11T21:42:45.164 INFO:teuthology.orchestra.run.smithi097.stdout: "client_addr": "v1:172.21.15.92:0/3334000812", 2024-03-11T21:42:45.165 INFO:teuthology.orchestra.run.smithi097.stdout: "tid": 70492 2024-03-11T21:42:45.165 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:42:45.165 INFO:teuthology.orchestra.run.smithi097.stdout: "events": [ 2024-03-11T21:42:45.165 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:42:45.165 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "initiated", 2024-03-11T21:42:45.165 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:41:29.854518+0000", 2024-03-11T21:42:45.165 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-03-11T21:42:45.165 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:42:45.165 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:42:45.165 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "header_read", 2024-03-11T21:42:45.165 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:41:29.854518+0000", 2024-03-11T21:42:45.165 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-03-11T21:42:45.165 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:42:45.165 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:42:45.165 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "throttled", 2024-03-11T21:42:45.166 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:41:29.854519+0000", 2024-03-11T21:42:45.166 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 7.5300000000000003e-07 2024-03-11T21:42:45.166 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:42:45.166 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:42:45.166 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "all_read", 2024-03-11T21:42:45.166 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:41:29.854571+0000", 2024-03-11T21:42:45.166 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 5.2125e-05 2024-03-11T21:42:45.166 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:42:45.166 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:42:45.166 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "dispatched", 2024-03-11T21:42:45.166 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:41:29.854578+0000", 2024-03-11T21:42:45.166 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 7.0029999999999997e-06 2024-03-11T21:42:45.166 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:42:45.166 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:42:45.167 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "queued_for_pg", 2024-03-11T21:42:45.167 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:41:29.854584+0000", 2024-03-11T21:42:45.169 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 5.8680000000000004e-06 2024-03-11T21:42:45.169 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:42:45.169 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:42:45.169 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "reached_pg", 2024-03-11T21:42:45.169 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:41:29.865070+0000", 2024-03-11T21:42:45.169 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.010485789000000001 2024-03-11T21:42:45.170 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:42:45.170 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:42:45.170 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "waiting for peered", 2024-03-11T21:42:45.170 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:41:29.865078+0000", 2024-03-11T21:42:45.170 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 7.8490000000000006e-06 2024-03-11T21:42:45.170 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:42:45.170 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:42:45.170 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "reached_pg", 2024-03-11T21:42:45.170 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:41:30.872785+0000", 2024-03-11T21:42:45.170 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.00770771 2024-03-11T21:42:45.170 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:42:45.170 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:42:45.170 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "done", 2024-03-11T21:42:45.170 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:41:30.872789+0000", 2024-03-11T21:42:45.171 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 3.8990000000000003e-06 2024-03-11T21:42:45.171 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-03-11T21:42:45.171 INFO:teuthology.orchestra.run.smithi097.stdout: ] 2024-03-11T21:42:45.171 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-03-11T21:42:45.171 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:42:45.171 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:42:45.171 INFO:teuthology.orchestra.run.smithi097.stdout: "description": "osd_op(client.15744.0:70505 10.11 10.fdec1d51 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e374)", 2024-03-11T21:42:45.171 INFO:teuthology.orchestra.run.smithi097.stdout: "initiated_at": "2024-03-11T21:41:29.866294+0000", 2024-03-11T21:42:45.171 INFO:teuthology.orchestra.run.smithi097.stdout: "age": 74.962199119999994, 2024-03-11T21:42:45.171 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.006510808, 2024-03-11T21:42:45.171 INFO:teuthology.orchestra.run.smithi097.stdout: "type_data": { 2024-03-11T21:42:45.171 INFO:teuthology.orchestra.run.smithi097.stdout: "flag_point": "reached pg", 2024-03-11T21:42:45.171 INFO:teuthology.orchestra.run.smithi097.stdout: "client_info": { 2024-03-11T21:42:45.171 INFO:teuthology.orchestra.run.smithi097.stdout: "client": "client.15744", 2024-03-11T21:42:45.171 INFO:teuthology.orchestra.run.smithi097.stdout: "client_addr": "v1:172.21.15.92:0/3334000812", 2024-03-11T21:42:45.171 INFO:teuthology.orchestra.run.smithi097.stdout: "tid": 70505 2024-03-11T21:42:45.172 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:42:45.172 INFO:teuthology.orchestra.run.smithi097.stdout: "events": [ 2024-03-11T21:42:45.172 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:42:45.172 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "initiated", 2024-03-11T21:42:45.172 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:41:29.866294+0000", 2024-03-11T21:42:45.172 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-03-11T21:42:45.172 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:42:45.172 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:42:45.172 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "header_read", 2024-03-11T21:42:45.172 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:41:29.866294+0000", 2024-03-11T21:42:45.172 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-03-11T21:42:45.172 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:42:45.172 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:42:45.172 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "throttled", 2024-03-11T21:42:45.172 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:41:29.866296+0000", 2024-03-11T21:42:45.172 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 2.3659999999999999e-06 2024-03-11T21:42:45.173 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:42:45.173 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:42:45.173 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "all_read", 2024-03-11T21:42:45.173 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:41:29.866365+0000", 2024-03-11T21:42:45.173 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 6.8263999999999996e-05 2024-03-11T21:42:45.173 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:42:45.173 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:42:45.173 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "dispatched", 2024-03-11T21:42:45.173 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:41:29.866381+0000", 2024-03-11T21:42:45.173 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.6605000000000001e-05 2024-03-11T21:42:45.173 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:42:45.173 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:42:45.173 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "queued_for_pg", 2024-03-11T21:42:45.173 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:41:29.866398+0000", 2024-03-11T21:42:45.173 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.6764000000000001e-05 2024-03-11T21:42:45.174 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:42:45.174 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:42:45.174 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "reached_pg", 2024-03-11T21:42:45.174 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:41:29.866623+0000", 2024-03-11T21:42:45.174 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.00022518899999999999 2024-03-11T21:42:45.174 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:42:45.174 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:42:45.174 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "waiting for peered", 2024-03-11T21:42:45.174 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:41:29.866631+0000", 2024-03-11T21:42:45.174 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 7.4050000000000003e-06 2024-03-11T21:42:45.174 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:42:45.174 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:42:45.174 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "reached_pg", 2024-03-11T21:42:45.174 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:41:30.872801+0000", 2024-03-11T21:42:45.175 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.0061703289999999 2024-03-11T21:42:45.175 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:42:45.175 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:42:45.175 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "done", 2024-03-11T21:42:45.175 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:41:30.872805+0000", 2024-03-11T21:42:45.175 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 3.8859999999999997e-06 2024-03-11T21:42:45.175 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-03-11T21:42:45.175 INFO:teuthology.orchestra.run.smithi097.stdout: ] 2024-03-11T21:42:45.175 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-03-11T21:42:45.175 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-03-11T21:42:45.175 INFO:teuthology.orchestra.run.smithi097.stdout: ] 2024-03-11T21:42:45.175 INFO:teuthology.orchestra.run.smithi097.stdout:} 2024-03-11T21:42:45.176 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:42:45.177 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:42:44 smithi092 ceph-mon[37563]: pgmap v807: 158 pgs: 1 active+recovery_wait, 153 active+clean, 4 active+recovering+undersized+remapped; 5.8 GiB data, 23 GiB used, 692 GiB / 715 GiB avail; 583 B/s rd, 0 B/s wr, 902 op/s; 15747/285144 objects misplaced (5.522%); 7.3 MiB/s, 117 objects/s recovering 2024-03-11T21:42:45.177 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:42:44 smithi092 ceph-mon[29572]: pgmap v807: 158 pgs: 1 active+recovery_wait, 153 active+clean, 4 active+recovering+undersized+remapped; 5.8 GiB data, 23 GiB used, 692 GiB / 715 GiB avail; 583 B/s rd, 0 B/s wr, 902 op/s; 15747/285144 objects misplaced (5.522%); 7.3 MiB/s, 117 objects/s recovering 2024-03-11T21:42:45.206 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:42:45.207 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:42:45.208 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:42:45.308 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:42:45.347 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:42:45.347 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:42:45.347 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:42:45.448 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:42:45.480 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:42:45.480 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:42:45.481 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:42:45.582 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:42:45.624 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:42:45.624 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:42:45.625 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:42:45.725 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:42:45.755 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:42:45.756 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:42:45.756 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:42:45.857 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:42:45.900 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:42:45.900 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:42:45.901 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:42:46.002 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:42:46.038 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:42:46.038 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:42:46.039 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:42:46.140 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:42:46.164 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:42:46.164 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:42:46.165 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:42:46.266 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:42:46.299 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:42:46.300 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:42:46.300 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:42:46.401 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:42:46.434 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:42:46.434 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:42:46.434 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:42:46.535 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:42:46.568 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:42:46.569 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:42:46.569 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:42:46.670 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:42:46.685 INFO:teuthology.orchestra.run.smithi092.stdout:osd.0: {} 2024-03-11T21:42:46.686 INFO:teuthology.orchestra.run.smithi092.stdout:osd.1: {} 2024-03-11T21:42:46.686 INFO:teuthology.orchestra.run.smithi092.stdout:osd.2: {} 2024-03-11T21:42:46.686 INFO:teuthology.orchestra.run.smithi092.stdout:osd.3: {} 2024-03-11T21:42:46.686 INFO:teuthology.orchestra.run.smithi092.stdout:osd.4: {} 2024-03-11T21:42:46.686 INFO:teuthology.orchestra.run.smithi092.stdout:osd.5: {} 2024-03-11T21:42:46.686 INFO:teuthology.orchestra.run.smithi092.stdout:osd.6: {} 2024-03-11T21:42:46.686 INFO:teuthology.orchestra.run.smithi092.stdout:osd.7: {} 2024-03-11T21:42:46.686 INFO:teuthology.orchestra.run.smithi092.stderr:osd.0: osd_enable_op_tracker = '' 2024-03-11T21:42:46.686 INFO:teuthology.orchestra.run.smithi092.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-11T21:42:46.686 INFO:teuthology.orchestra.run.smithi092.stderr:osd.2: osd_enable_op_tracker = '' 2024-03-11T21:42:46.687 INFO:teuthology.orchestra.run.smithi092.stderr:osd.3: osd_enable_op_tracker = '' 2024-03-11T21:42:46.687 INFO:teuthology.orchestra.run.smithi092.stderr:osd.4: osd_enable_op_tracker = '' 2024-03-11T21:42:46.687 INFO:teuthology.orchestra.run.smithi092.stderr:osd.5: osd_enable_op_tracker = '' 2024-03-11T21:42:46.687 INFO:teuthology.orchestra.run.smithi092.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-11T21:42:46.687 INFO:teuthology.orchestra.run.smithi092.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-11T21:42:46.717 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:42:46.717 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:42:46.717 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:42:46.749 DEBUG:teuthology.orchestra.run.smithi097:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid 00db0016-dfed-11ee-95c8-87774f69a715 -- ceph daemon osd.6 dump_ops_in_flight 2024-03-11T21:42:46.818 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:42:46.859 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:42:46.860 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:42:46.860 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:42:46.865 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:42:46 smithi092 ceph-mon[29572]: pgmap v808: 158 pgs: 1 active+recovery_wait, 153 active+clean, 4 active+recovering+undersized+remapped; 5.7 GiB data, 23 GiB used, 692 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 873 op/s; 15360/282717 objects misplaced (5.433%); 7.2 MiB/s, 115 objects/s recovering 2024-03-11T21:42:46.866 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:42:46 smithi092 ceph-mon[29572]: from='client.? v1:172.21.15.92:0/3628012917' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-11T21:42:46.866 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:42:46 smithi092 ceph-mon[37563]: pgmap v808: 158 pgs: 1 active+recovery_wait, 153 active+clean, 4 active+recovering+undersized+remapped; 5.7 GiB data, 23 GiB used, 692 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 873 op/s; 15360/282717 objects misplaced (5.433%); 7.2 MiB/s, 115 objects/s recovering 2024-03-11T21:42:46.866 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:42:46 smithi092 ceph-mon[37563]: from='client.? v1:172.21.15.92:0/3628012917' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-11T21:42:46.960 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:42:47.001 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:42:47.002 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:42:47.002 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:42:47.082 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:42:46 smithi097 ceph-mon[33317]: pgmap v808: 158 pgs: 1 active+recovery_wait, 153 active+clean, 4 active+recovering+undersized+remapped; 5.7 GiB data, 23 GiB used, 692 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 873 op/s; 15360/282717 objects misplaced (5.433%); 7.2 MiB/s, 115 objects/s recovering 2024-03-11T21:42:47.082 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:42:46 smithi097 ceph-mon[33317]: from='client.? v1:172.21.15.92:0/3628012917' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-11T21:42:47.103 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:42:47.111 INFO:teuthology.orchestra.run.smithi097.stderr:Inferring config /var/lib/ceph/00db0016-dfed-11ee-95c8-87774f69a715/mon.b/config 2024-03-11T21:42:47.139 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:42:47.140 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:42:47.140 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:42:47.241 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:42:47.273 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:42:47.273 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:42:47.274 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:42:47.374 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:42:47.402 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:42:47.403 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:42:47.403 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:42:47.504 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:42:47.548 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:42:47.548 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:42:47.549 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:42:47.649 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:42:47.690 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:42:47.690 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:42:47.691 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:42:47.792 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:42:47.824 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:42:47.825 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:42:47.825 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:42:47.926 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:42:47.969 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:42:47.970 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:42:47.970 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:42:48.071 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:42:48.116 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:42:48.116 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:42:48.117 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:42:48.218 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:42:48.249 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:42:48.249 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:42:48.250 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:42:48.250 INFO:journalctl@ceph.mgr.y.smithi092.stdout:Mar 11 21:42:47 smithi092 ceph-00db0016-dfed-11ee-95c8-87774f69a715-mgr-y[29801]: ::ffff:172.21.15.97 - - [11/Mar/2024:21:42:47] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-11T21:42:48.350 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:42:48.388 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:42:48.389 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:42:48.389 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:42:48.490 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:42:48.521 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:42:48.521 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:42:48.522 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:42:48.623 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:42:48.678 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:42:48.678 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:42:48.679 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:42:48.779 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:42:48.804 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:42:48.804 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:42:48.805 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:42:48.903 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:42:48 smithi097 ceph-mon[33317]: pgmap v809: 158 pgs: 1 active+recovery_wait, 153 active+clean, 4 active+recovering+undersized+remapped; 5.7 GiB data, 23 GiB used, 692 GiB / 715 GiB avail; 871 B/s rd, 0 B/s wr, 831 op/s; 15360/279624 objects misplaced (5.493%); 6.2 MiB/s, 98 objects/s recovering 2024-03-11T21:42:48.905 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:42:48.908 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:42:48 smithi092 ceph-mon[29572]: pgmap v809: 158 pgs: 1 active+recovery_wait, 153 active+clean, 4 active+recovering+undersized+remapped; 5.7 GiB data, 23 GiB used, 692 GiB / 715 GiB avail; 871 B/s rd, 0 B/s wr, 831 op/s; 15360/279624 objects misplaced (5.493%); 6.2 MiB/s, 98 objects/s recovering 2024-03-11T21:42:48.908 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:42:48 smithi092 ceph-mon[37563]: pgmap v809: 158 pgs: 1 active+recovery_wait, 153 active+clean, 4 active+recovering+undersized+remapped; 5.7 GiB data, 23 GiB used, 692 GiB / 715 GiB avail; 871 B/s rd, 0 B/s wr, 831 op/s; 15360/279624 objects misplaced (5.493%); 6.2 MiB/s, 98 objects/s recovering 2024-03-11T21:42:48.947 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:42:48.948 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:42:48.948 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:42:49.049 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:42:49.053 INFO:teuthology.orchestra.run.smithi097.stdout:ERROR: (22) Invalid argument 2024-03-11T21:42:49.053 INFO:teuthology.orchestra.run.smithi097.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-11T21:42:49.090 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:42:49.091 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:42:49.091 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:42:49.192 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:42:49.224 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:42:49.225 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:42:49.225 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:42:49.326 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:42:49.373 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:42:49.374 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:42:49.374 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:42:49.475 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:42:49.513 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:42:49.514 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:42:49.514 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:42:49.615 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:42:49.664 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:42:49.664 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:42:49.665 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:42:49.765 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:42:49.769 DEBUG:teuthology.orchestra.run.smithi097:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid 00db0016-dfed-11ee-95c8-87774f69a715 -- ceph daemon osd.6 dump_blocked_ops 2024-03-11T21:42:49.798 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:42:49.799 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:42:49.799 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:42:49.901 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:42:49.940 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:42:49.940 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:42:49.941 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:42:50.041 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:42:50.083 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:42:50.084 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:42:50.084 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:42:50.152 INFO:teuthology.orchestra.run.smithi097.stderr:Inferring config /var/lib/ceph/00db0016-dfed-11ee-95c8-87774f69a715/mon.b/config 2024-03-11T21:42:50.185 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:42:50.210 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:42:50.211 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:42:50.211 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:42:50.312 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:42:50.365 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:42:50.366 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:42:50.366 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:42:50.467 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:42:50.514 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:42:50.515 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:42:50.516 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:42:50.617 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:42:50.661 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:42:50.662 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:42:50.662 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:42:50.763 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:42:50.809 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:42:50.810 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:42:50.810 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:42:50.831 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:42:50 smithi097 ceph-mon[33317]: pgmap v810: 158 pgs: 1 active+recovery_wait, 153 active+clean, 4 active+recovering+undersized+remapped; 5.5 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 951 op/s; 14235/271578 objects misplaced (5.242%); 8.6 MiB/s, 136 objects/s recovering 2024-03-11T21:42:50.911 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:42:50.934 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:42:50.935 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:42:50.935 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:42:51.036 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:42:51.074 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:42:51.074 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:42:51.075 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:42:51.116 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:42:50 smithi092 ceph-mon[29572]: pgmap v810: 158 pgs: 1 active+recovery_wait, 153 active+clean, 4 active+recovering+undersized+remapped; 5.5 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 951 op/s; 14235/271578 objects misplaced (5.242%); 8.6 MiB/s, 136 objects/s recovering 2024-03-11T21:42:51.117 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:42:50 smithi092 ceph-mon[37563]: pgmap v810: 158 pgs: 1 active+recovery_wait, 153 active+clean, 4 active+recovering+undersized+remapped; 5.5 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 951 op/s; 14235/271578 objects misplaced (5.242%); 8.6 MiB/s, 136 objects/s recovering 2024-03-11T21:42:51.175 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:42:51.218 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:42:51.219 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:42:51.219 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:42:51.320 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:42:51.367 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:42:51.368 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:42:51.368 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:42:51.469 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:42:51.505 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:42:51.506 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:42:51.506 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:42:51.607 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:42:51.648 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:42:51.649 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:42:51.649 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:42:51.750 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:42:51.785 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:42:51.786 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:42:51.787 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:42:51.888 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:42:51.914 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:42:51.916 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:42:51.916 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:42:52.017 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:42:52.055 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:42:52.056 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:42:52.056 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:42:52.157 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:42:52.201 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:42:52.201 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:42:52.201 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:42:52.302 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:42:52.334 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:42:52.335 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:42:52.335 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:42:52.436 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:42:52.470 INFO:teuthology.orchestra.run.smithi097.stdout:ERROR: (22) Invalid argument 2024-03-11T21:42:52.470 INFO:teuthology.orchestra.run.smithi097.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-11T21:42:52.476 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:42:52.477 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:42:52.477 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:42:52.578 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:42:52.610 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:42:52.610 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:42:52.610 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:42:52.711 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:42:52.752 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:42:52.753 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:42:52.753 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:42:52.854 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:42:52.891 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:42:52.892 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:42:52.892 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:42:52.954 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:42:52 smithi097 ceph-mon[33317]: pgmap v811: 158 pgs: 1 active+recovery_wait, 153 active+clean, 4 active+recovering+undersized+remapped; 5.5 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 853 B/s rd, 0 B/s wr, 787 op/s; 13830/269811 objects misplaced (5.126%); 6.9 MiB/s, 109 objects/s recovering 2024-03-11T21:42:52.993 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:42:53.021 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:42:53.021 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:42:53.022 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:42:53.116 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:42:52 smithi092 ceph-mon[37563]: pgmap v811: 158 pgs: 1 active+recovery_wait, 153 active+clean, 4 active+recovering+undersized+remapped; 5.5 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 853 B/s rd, 0 B/s wr, 787 op/s; 13830/269811 objects misplaced (5.126%); 6.9 MiB/s, 109 objects/s recovering 2024-03-11T21:42:53.116 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:42:52 smithi092 ceph-mon[29572]: pgmap v811: 158 pgs: 1 active+recovery_wait, 153 active+clean, 4 active+recovering+undersized+remapped; 5.5 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 853 B/s rd, 0 B/s wr, 787 op/s; 13830/269811 objects misplaced (5.126%); 6.9 MiB/s, 109 objects/s recovering 2024-03-11T21:42:53.122 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:42:53.166 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:42:53.166 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:42:53.167 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:42:53.238 INFO:journalctl@ceph.iscsi.iscsi.a.smithi097.stdout:Mar 11 21:42:52 smithi097 ceph-00db0016-dfed-11ee-95c8-87774f69a715-iscsi-iscsi-a[60943]: debug there is no tcmu-runner data available 2024-03-11T21:42:53.239 DEBUG:teuthology.orchestra.run.smithi097:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid 00db0016-dfed-11ee-95c8-87774f69a715 -- ceph daemon osd.6 dump_historic_ops 2024-03-11T21:42:53.268 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:42:53.308 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:42:53.308 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:42:53.308 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:42:53.409 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:42:53.449 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:42:53.450 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:42:53.450 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:42:53.551 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:42:53.586 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:42:53.587 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:42:53.587 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:42:53.625 INFO:teuthology.orchestra.run.smithi097.stderr:Inferring config /var/lib/ceph/00db0016-dfed-11ee-95c8-87774f69a715/mon.b/config 2024-03-11T21:42:53.687 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:42:53.729 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:42:53.730 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:42:53.731 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:42:53.831 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:42:53.865 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:42:53.865 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:42:53.865 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:42:53.950 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:42:53 smithi097 ceph-mon[33317]: from='mgr.14607 v1:172.21.15.92:0/1702717597' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T21:42:53.950 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:42:53 smithi097 ceph-mon[33317]: from='client.24469 v1:172.21.15.97:0/404208310' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:42:53.967 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:42:54.014 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:42:54.015 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:42:54.015 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:42:54.102 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:42:53 smithi092 ceph-mon[29572]: from='mgr.14607 v1:172.21.15.92:0/1702717597' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T21:42:54.103 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:42:53 smithi092 ceph-mon[29572]: from='client.24469 v1:172.21.15.97:0/404208310' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:42:54.104 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:42:53 smithi092 ceph-mon[37563]: from='mgr.14607 v1:172.21.15.92:0/1702717597' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T21:42:54.104 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:42:53 smithi092 ceph-mon[37563]: from='client.24469 v1:172.21.15.97:0/404208310' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:42:54.116 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:42:54.154 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:42:54.154 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:42:54.155 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:42:54.255 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:42:54.288 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:42:54.290 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:42:54.290 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:42:54.391 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:42:54.436 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:42:54.437 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:42:54.438 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:42:54.538 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:42:54.572 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:42:54.573 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:42:54.573 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:42:54.674 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:42:54.701 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:42:54.703 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:42:54.704 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:42:54.804 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:42:54.850 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:42:54.850 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:42:54.851 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:42:54.952 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:42:54.985 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:42:54.986 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:42:54.986 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:42:54.986 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:42:54 smithi097 ceph-mon[33317]: pgmap v812: 158 pgs: 1 active+recovery_wait, 153 active+clean, 4 active+recovering+undersized+remapped; 5.3 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 853 B/s rd, 0 B/s wr, 1.04k op/s; 12747/258921 objects misplaced (4.923%); 8.6 MiB/s, 137 objects/s recovering 2024-03-11T21:42:55.087 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:42:55.116 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:42:54 smithi092 ceph-mon[37563]: pgmap v812: 158 pgs: 1 active+recovery_wait, 153 active+clean, 4 active+recovering+undersized+remapped; 5.3 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 853 B/s rd, 0 B/s wr, 1.04k op/s; 12747/258921 objects misplaced (4.923%); 8.6 MiB/s, 137 objects/s recovering 2024-03-11T21:42:55.116 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:42:54 smithi092 ceph-mon[29572]: pgmap v812: 158 pgs: 1 active+recovery_wait, 153 active+clean, 4 active+recovering+undersized+remapped; 5.3 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 853 B/s rd, 0 B/s wr, 1.04k op/s; 12747/258921 objects misplaced (4.923%); 8.6 MiB/s, 137 objects/s recovering 2024-03-11T21:42:55.136 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:42:55.136 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:42:55.137 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:42:55.238 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:42:55.268 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:42:55.269 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:42:55.269 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:42:55.370 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:42:55.421 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:42:55.421 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:42:55.422 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:42:55.523 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:42:55.563 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:42:55.563 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:42:55.564 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:42:55.664 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:42:55.708 INFO:teuthology.orchestra.run.smithi097.stdout:ERROR: (22) Invalid argument 2024-03-11T21:42:55.708 INFO:teuthology.orchestra.run.smithi097.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-11T21:42:55.712 INFO:teuthology.orchestra.run.smithi092.stderr:Inferring config /var/lib/ceph/00db0016-dfed-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T21:42:55.722 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:42:55.722 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:42:55.723 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:42:55.823 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:42:55.853 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:42:55.854 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:42:55.854 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:42:55.884 DEBUG:teuthology.orchestra.run.smithi092:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid 00db0016-dfed-11ee-95c8-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-03-11T21:42:55.955 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:42:55.986 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:42:55.987 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:42:55.987 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:42:56.088 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:42:56.117 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:42:56.117 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:42:56.118 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:42:56.219 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:42:56.250 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:42:56.250 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:42:56.251 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:42:56.352 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:42:56.445 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:42:56.446 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:42:56.446 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:42:56.547 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:42:56.582 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:42:56.582 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:42:56.582 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:42:56.683 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:42:56.729 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:42:56.730 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:42:56.730 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:42:56.750 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:42:56 smithi097 ceph-mon[33317]: pgmap v813: 158 pgs: 1 active+recovery_wait, 153 active+clean, 4 active+recovering+undersized+remapped; 5.2 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 804 op/s; 12289/256218 objects misplaced (4.796%); 7.1 MiB/s, 114 objects/s recovering 2024-03-11T21:42:56.772 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:42:56 smithi092 ceph-mon[37563]: pgmap v813: 158 pgs: 1 active+recovery_wait, 153 active+clean, 4 active+recovering+undersized+remapped; 5.2 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 804 op/s; 12289/256218 objects misplaced (4.796%); 7.1 MiB/s, 114 objects/s recovering 2024-03-11T21:42:56.773 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:42:56 smithi092 ceph-mon[29572]: pgmap v813: 158 pgs: 1 active+recovery_wait, 153 active+clean, 4 active+recovering+undersized+remapped; 5.2 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 804 op/s; 12289/256218 objects misplaced (4.796%); 7.1 MiB/s, 114 objects/s recovering 2024-03-11T21:42:56.783 DEBUG:teuthology.orchestra.run.smithi097:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid 00db0016-dfed-11ee-95c8-87774f69a715 -- ceph daemon osd.7 dump_ops_in_flight 2024-03-11T21:42:56.831 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:42:56.869 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:42:56.870 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:42:56.870 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:42:56.971 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:42:57.011 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:42:57.012 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:42:57.012 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:42:57.113 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:42:57.139 INFO:teuthology.orchestra.run.smithi097.stderr:Inferring config /var/lib/ceph/00db0016-dfed-11ee-95c8-87774f69a715/mon.b/config 2024-03-11T21:42:57.151 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:42:57.152 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:42:57.153 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:42:57.253 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:42:57.286 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:42:57.287 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:42:57.287 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:42:57.388 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:42:57.419 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:42:57.420 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:42:57.420 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:42:57.521 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:42:57.572 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:42:57.573 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:42:57.574 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:42:57.675 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:42:57.706 INFO:teuthology.orchestra.run.smithi092.stderr:noscrub is unset 2024-03-11T21:42:57.730 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:42:57.731 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:42:57.731 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:42:57.832 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:42:57.876 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:42:57.876 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:42:57.877 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:42:57.977 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:42:58.008 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:42:58.009 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:42:58.009 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:42:58.019 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:42:57 smithi097 ceph-mon[33317]: from='client.? v1:172.21.15.92:0/2426433473' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-11T21:42:58.019 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:42:57 smithi097 ceph-mon[33317]: from='client.25669 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-11T21:42:58.095 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:42:57 smithi092 ceph-mon[29572]: from='client.? v1:172.21.15.92:0/2426433473' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-11T21:42:58.096 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:42:57 smithi092 ceph-mon[29572]: from='client.25669 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-11T21:42:58.096 INFO:journalctl@ceph.mgr.y.smithi092.stdout:Mar 11 21:42:57 smithi092 ceph-00db0016-dfed-11ee-95c8-87774f69a715-mgr-y[29801]: ::ffff:172.21.15.97 - - [11/Mar/2024:21:42:57] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-11T21:42:58.096 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:42:57 smithi092 ceph-mon[37563]: from='client.? v1:172.21.15.92:0/2426433473' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-11T21:42:58.096 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:42:57 smithi092 ceph-mon[37563]: from='client.25669 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-11T21:42:58.111 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:42:58.141 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:42:58.142 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:42:58.142 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:42:58.243 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:42:58.276 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:42:58.276 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:42:58.277 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:42:58.377 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:42:58.428 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:42:58.429 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:42:58.430 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:42:58.530 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:42:58.563 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:42:58.563 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:42:58.564 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:42:58.664 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:42:58.722 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:42:58.723 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:42:58.723 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:42:58.824 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:42:58.862 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:42:58.863 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:42:58.863 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:42:58.963 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:42:58.975 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:42:58 smithi092 ceph-mon[29572]: pgmap v814: 158 pgs: 1 active+recovery_wait, 153 active+clean, 4 active+recovering+undersized+remapped; 5.1 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 852 B/s rd, 0 B/s wr, 835 op/s; 12289/252672 objects misplaced (4.864%); 6.5 MiB/s, 103 objects/s recovering 2024-03-11T21:42:58.975 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:42:58 smithi092 ceph-mon[29572]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-11T21:42:58.975 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:42:58 smithi092 ceph-mon[29572]: from='client.25669 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-03-11T21:42:58.975 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:42:58 smithi092 ceph-mon[29572]: osdmap e401: 8 total, 8 up, 8 in 2024-03-11T21:42:58.975 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:42:58 smithi092 ceph-mon[29572]: from='mgr.14607 ' entity='mgr.y' 2024-03-11T21:42:58.975 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:42:58 smithi092 ceph-mon[29572]: from='mgr.14607 ' entity='mgr.y' 2024-03-11T21:42:58.975 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:42:58 smithi092 ceph-mon[37563]: pgmap v814: 158 pgs: 1 active+recovery_wait, 153 active+clean, 4 active+recovering+undersized+remapped; 5.1 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 852 B/s rd, 0 B/s wr, 835 op/s; 12289/252672 objects misplaced (4.864%); 6.5 MiB/s, 103 objects/s recovering 2024-03-11T21:42:58.975 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:42:58 smithi092 ceph-mon[37563]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-11T21:42:58.975 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:42:58 smithi092 ceph-mon[37563]: from='client.25669 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-03-11T21:42:58.975 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:42:58 smithi092 ceph-mon[37563]: osdmap e401: 8 total, 8 up, 8 in 2024-03-11T21:42:58.976 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:42:58 smithi092 ceph-mon[37563]: from='mgr.14607 ' entity='mgr.y' 2024-03-11T21:42:58.976 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:42:58 smithi092 ceph-mon[37563]: from='mgr.14607 ' entity='mgr.y' 2024-03-11T21:42:59.003 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:42:59.004 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:42:59.004 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:42:59.013 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:42:58 smithi097 ceph-mon[33317]: pgmap v814: 158 pgs: 1 active+recovery_wait, 153 active+clean, 4 active+recovering+undersized+remapped; 5.1 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 852 B/s rd, 0 B/s wr, 835 op/s; 12289/252672 objects misplaced (4.864%); 6.5 MiB/s, 103 objects/s recovering 2024-03-11T21:42:59.013 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:42:58 smithi097 ceph-mon[33317]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-11T21:42:59.013 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:42:58 smithi097 ceph-mon[33317]: from='client.25669 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-03-11T21:42:59.013 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:42:58 smithi097 ceph-mon[33317]: osdmap e401: 8 total, 8 up, 8 in 2024-03-11T21:42:59.013 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:42:58 smithi097 ceph-mon[33317]: from='mgr.14607 ' entity='mgr.y' 2024-03-11T21:42:59.013 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:42:58 smithi097 ceph-mon[33317]: from='mgr.14607 ' entity='mgr.y' 2024-03-11T21:42:59.106 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:42:59.148 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:42:59.149 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:42:59.150 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:42:59.250 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:42:59.286 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:42:59.287 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:42:59.287 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:42:59.388 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:42:59.428 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:42:59.428 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:42:59.429 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:42:59.530 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:42:59.558 INFO:teuthology.orchestra.run.smithi097.stdout:ERROR: (22) Invalid argument 2024-03-11T21:42:59.558 INFO:teuthology.orchestra.run.smithi097.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-11T21:42:59.565 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:42:59.566 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:42:59.566 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:42:59.667 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:42:59.711 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:42:59.711 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:42:59.711 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:42:59.812 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:42:59.862 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:42:59.863 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:42:59.864 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:42:59.964 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:42:59.995 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:42:59.995 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:42:59.995 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:43:00.082 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:42:59 smithi097 ceph-mon[33317]: purged_snaps scrub starts 2024-03-11T21:43:00.082 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:42:59 smithi097 ceph-mon[33317]: purged_snaps scrub ok 2024-03-11T21:43:00.096 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:43:00.116 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:42:59 smithi092 ceph-mon[29572]: purged_snaps scrub starts 2024-03-11T21:43:00.116 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:42:59 smithi092 ceph-mon[29572]: purged_snaps scrub ok 2024-03-11T21:43:00.117 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:42:59 smithi092 ceph-mon[37563]: purged_snaps scrub starts 2024-03-11T21:43:00.117 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:42:59 smithi092 ceph-mon[37563]: purged_snaps scrub ok 2024-03-11T21:43:00.129 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:43:00.129 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:43:00.130 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:43:00.231 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:43:00.264 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:43:00.265 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:43:00.265 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:43:00.358 DEBUG:teuthology.orchestra.run.smithi097:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid 00db0016-dfed-11ee-95c8-87774f69a715 -- ceph daemon osd.7 dump_blocked_ops 2024-03-11T21:43:00.366 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:43:00.445 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:43:00.503 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:43:00.503 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:43:00.604 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:43:00.656 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:43:00.656 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:43:00.656 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:43:00.735 INFO:teuthology.orchestra.run.smithi097.stderr:Inferring config /var/lib/ceph/00db0016-dfed-11ee-95c8-87774f69a715/mon.b/config 2024-03-11T21:43:00.757 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:43:00.798 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:43:00.798 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:43:00.798 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:43:00.899 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:43:00.929 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:43:00.930 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:43:00.931 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:43:01.031 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:43:01.034 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:43:00 smithi097 ceph-mon[33317]: pgmap v816: 158 pgs: 1 active+recovery_wait, 153 active+clean, 4 active+recovering+undersized+remapped; 4.9 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 930 op/s; 11084/243666 objects misplaced (4.549%); 8.0 MiB/s, 128 objects/s recovering 2024-03-11T21:43:01.034 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:43:00 smithi097 ceph-mon[33317]: from='mgr.14607 v1:172.21.15.92:0/1702717597' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T21:43:01.035 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:43:00 smithi097 ceph-mon[33317]: osdmap e402: 8 total, 8 up, 8 in 2024-03-11T21:43:01.035 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:43:00 smithi097 ceph-mon[33317]: from='mgr.14607 v1:172.21.15.92:0/1702717597' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T21:43:01.035 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:43:00 smithi097 ceph-mon[33317]: from='mgr.14607 ' entity='mgr.y' 2024-03-11T21:43:01.055 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:43:01.055 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:43:01.055 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:43:01.116 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:43:00 smithi092 ceph-mon[37563]: pgmap v816: 158 pgs: 1 active+recovery_wait, 153 active+clean, 4 active+recovering+undersized+remapped; 4.9 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 930 op/s; 11084/243666 objects misplaced (4.549%); 8.0 MiB/s, 128 objects/s recovering 2024-03-11T21:43:01.116 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:43:00 smithi092 ceph-mon[37563]: from='mgr.14607 v1:172.21.15.92:0/1702717597' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T21:43:01.116 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:43:00 smithi092 ceph-mon[37563]: osdmap e402: 8 total, 8 up, 8 in 2024-03-11T21:43:01.116 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:43:00 smithi092 ceph-mon[37563]: from='mgr.14607 v1:172.21.15.92:0/1702717597' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T21:43:01.116 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:43:00 smithi092 ceph-mon[37563]: from='mgr.14607 ' entity='mgr.y' 2024-03-11T21:43:01.116 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:43:00 smithi092 ceph-mon[29572]: pgmap v816: 158 pgs: 1 active+recovery_wait, 153 active+clean, 4 active+recovering+undersized+remapped; 4.9 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 930 op/s; 11084/243666 objects misplaced (4.549%); 8.0 MiB/s, 128 objects/s recovering 2024-03-11T21:43:01.117 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:43:00 smithi092 ceph-mon[29572]: from='mgr.14607 v1:172.21.15.92:0/1702717597' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T21:43:01.117 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:43:00 smithi092 ceph-mon[29572]: osdmap e402: 8 total, 8 up, 8 in 2024-03-11T21:43:01.117 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:43:00 smithi092 ceph-mon[29572]: from='mgr.14607 v1:172.21.15.92:0/1702717597' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T21:43:01.117 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:43:00 smithi092 ceph-mon[29572]: from='mgr.14607 ' entity='mgr.y' 2024-03-11T21:43:01.156 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:43:01.190 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:43:01.190 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:43:01.191 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:43:01.292 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:43:01.332 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:43:01.332 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:43:01.333 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:43:01.433 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:43:01.475 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:43:01.475 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:43:01.476 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:43:01.576 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:43:01.619 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:43:01.619 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:43:01.620 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:43:01.721 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:43:01.764 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:43:01.764 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:43:01.765 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:43:01.866 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:43:01.895 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:43:01.895 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:43:01.896 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:43:01.997 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:43:02.017 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:43:01 smithi092 ceph-mon[29572]: purged_snaps scrub starts 2024-03-11T21:43:02.017 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:43:01 smithi092 ceph-mon[29572]: purged_snaps scrub ok 2024-03-11T21:43:02.017 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:43:01 smithi092 ceph-mon[29572]: osdmap e403: 8 total, 8 up, 8 in 2024-03-11T21:43:02.018 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:43:01 smithi092 ceph-mon[37563]: purged_snaps scrub starts 2024-03-11T21:43:02.018 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:43:01 smithi092 ceph-mon[37563]: purged_snaps scrub ok 2024-03-11T21:43:02.018 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:43:01 smithi092 ceph-mon[37563]: osdmap e403: 8 total, 8 up, 8 in 2024-03-11T21:43:02.030 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:43:02.031 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:43:02.031 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:43:02.039 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:43:01 smithi097 ceph-mon[33317]: purged_snaps scrub starts 2024-03-11T21:43:02.039 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:43:01 smithi097 ceph-mon[33317]: purged_snaps scrub ok 2024-03-11T21:43:02.039 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:43:01 smithi097 ceph-mon[33317]: osdmap e403: 8 total, 8 up, 8 in 2024-03-11T21:43:02.132 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:43:02.171 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:43:02.172 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:43:02.172 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:43:02.273 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:43:02.307 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:43:02.308 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:43:02.308 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:43:02.409 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:43:02.447 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:43:02.447 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:43:02.448 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:43:02.549 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:43:02.589 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:43:02.590 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:43:02.591 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:43:02.691 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:43:02.727 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:43:02.727 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:43:02.727 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:43:02.828 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:43:02.867 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:43:02.868 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:43:02.868 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:43:02.969 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:43:02.993 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:43:02.993 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:43:02.993 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:43:03.082 INFO:journalctl@ceph.iscsi.iscsi.a.smithi097.stdout:Mar 11 21:43:02 smithi097 ceph-00db0016-dfed-11ee-95c8-87774f69a715-iscsi-iscsi-a[60943]: debug there is no tcmu-runner data available 2024-03-11T21:43:03.082 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:43:02 smithi097 ceph-mon[33317]: pgmap v819: 158 pgs: 1 active+recovery_wait, 153 active+clean, 4 active+recovering+undersized+remapped; 4.9 GiB data, 21 GiB used, 695 GiB / 715 GiB avail; 852 B/s rd, 0 B/s wr, 795 op/s; 10584/241908 objects misplaced (4.375%); 7.2 MiB/s, 115 objects/s recovering 2024-03-11T21:43:03.095 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:43:03.116 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:43:02 smithi092 ceph-mon[29572]: pgmap v819: 158 pgs: 1 active+recovery_wait, 153 active+clean, 4 active+recovering+undersized+remapped; 4.9 GiB data, 21 GiB used, 695 GiB / 715 GiB avail; 852 B/s rd, 0 B/s wr, 795 op/s; 10584/241908 objects misplaced (4.375%); 7.2 MiB/s, 115 objects/s recovering 2024-03-11T21:43:03.116 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:43:02 smithi092 ceph-mon[37563]: pgmap v819: 158 pgs: 1 active+recovery_wait, 153 active+clean, 4 active+recovering+undersized+remapped; 4.9 GiB data, 21 GiB used, 695 GiB / 715 GiB avail; 852 B/s rd, 0 B/s wr, 795 op/s; 10584/241908 objects misplaced (4.375%); 7.2 MiB/s, 115 objects/s recovering 2024-03-11T21:43:03.127 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:43:03.128 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:43:03.128 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:43:03.229 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:43:03.260 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:43:03.261 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:43:03.261 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:43:03.362 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:43:03.389 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:43:03.390 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:43:03.390 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:43:03.491 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:43:03.530 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:43:03.530 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:43:03.531 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:43:03.631 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:43:03.663 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:43:03.663 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:43:03.663 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:43:03.764 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:43:03.802 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:43:03.802 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:43:03.803 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:43:03.903 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:43:03.947 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:43:03.948 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:43:03.948 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:43:03.968 INFO:teuthology.orchestra.run.smithi092.stderr:Inferring config /var/lib/ceph/00db0016-dfed-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T21:43:04.049 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:43:04.081 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:43:04.081 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:43:04.082 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:43:04.082 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:43:03 smithi097 ceph-mon[33317]: from='client.24469 v1:172.21.15.97:0/404208310' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:43:04.116 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:43:03 smithi092 ceph-mon[29572]: from='client.24469 v1:172.21.15.97:0/404208310' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:43:04.116 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:43:03 smithi092 ceph-mon[37563]: from='client.24469 v1:172.21.15.97:0/404208310' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:43:04.182 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:43:04.206 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:43:04.207 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:43:04.207 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:43:04.308 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:43:04.336 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:43:04.336 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:43:04.337 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:43:04.438 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:43:04.473 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:43:04.474 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:43:04.474 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:43:04.575 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:43:04.609 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:43:04.609 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:43:04.609 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:43:04.710 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:43:04.745 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:43:04.745 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:43:04.746 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:43:04.849 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:43:04.892 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:43:04.893 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:43:04.894 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:43:04.994 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:43:04.996 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:43:04 smithi097 ceph-mon[33317]: pgmap v820: 158 pgs: 1 active+recovery_wait, 153 active+clean, 4 active+recovering+undersized+remapped; 4.7 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 852 B/s rd, 0 B/s wr, 1.21k op/s; 9383/230973 objects misplaced (4.062%); 13 MiB/s, 203 objects/s recovering 2024-03-11T21:43:05.041 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:43:05.041 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:43:05.042 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:43:05.116 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:43:04 smithi092 ceph-mon[29572]: pgmap v820: 158 pgs: 1 active+recovery_wait, 153 active+clean, 4 active+recovering+undersized+remapped; 4.7 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 852 B/s rd, 0 B/s wr, 1.21k op/s; 9383/230973 objects misplaced (4.062%); 13 MiB/s, 203 objects/s recovering 2024-03-11T21:43:05.116 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:43:04 smithi092 ceph-mon[37563]: pgmap v820: 158 pgs: 1 active+recovery_wait, 153 active+clean, 4 active+recovering+undersized+remapped; 4.7 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 852 B/s rd, 0 B/s wr, 1.21k op/s; 9383/230973 objects misplaced (4.062%); 13 MiB/s, 203 objects/s recovering 2024-03-11T21:43:05.142 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:43:05.182 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:43:05.183 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:43:05.184 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:43:05.284 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:43:05.308 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:43:05.309 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:43:05.309 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:43:05.410 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:43:05.452 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:43:05.452 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:43:05.453 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:43:05.554 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:43:05.562 INFO:teuthology.orchestra.run.smithi097.stdout:ERROR: (22) Invalid argument 2024-03-11T21:43:05.562 INFO:teuthology.orchestra.run.smithi097.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-11T21:43:05.581 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:43:05.581 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:43:05.582 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:43:05.683 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:43:05.724 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:43:05.724 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:43:05.725 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:43:05.825 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:43:05.850 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:43:05.851 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:43:05.851 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:43:05.913 DEBUG:teuthology.orchestra.run.smithi092:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid 00db0016-dfed-11ee-95c8-87774f69a715 -- ceph osd unset nodeep-scrub 2024-03-11T21:43:05.951 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:43:06.011 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:43:06.011 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:43:06.012 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:43:06.112 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:43:06.143 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:43:06.144 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:43:06.145 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:43:06.245 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:43:06.278 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:43:06.279 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:43:06.279 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:43:06.380 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:43:06.382 DEBUG:teuthology.orchestra.run.smithi097:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid 00db0016-dfed-11ee-95c8-87774f69a715 -- ceph daemon osd.7 dump_historic_ops 2024-03-11T21:43:06.412 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:43:06.413 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:43:06.413 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:43:06.514 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:43:06.555 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:43:06.555 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:43:06.556 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:43:06.656 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:43:06.689 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:43:06.690 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:43:06.690 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:43:06.784 INFO:teuthology.orchestra.run.smithi097.stderr:Inferring config /var/lib/ceph/00db0016-dfed-11ee-95c8-87774f69a715/mon.b/config 2024-03-11T21:43:06.791 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:43:06.828 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:43:06.828 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:43:06.828 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:43:06.929 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:43:06.958 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:43:06.959 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:43:06.959 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:43:07.060 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:43:07.081 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:43:06 smithi097 ceph-mon[33317]: pgmap v821: 158 pgs: 1 active+recovery_wait, 153 active+clean, 4 active+recovering+undersized+remapped; 4.6 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 1.3 KiB/s rd, 0 B/s wr, 790 op/s; 8910/228585 objects misplaced (3.898%); 11 MiB/s, 180 objects/s recovering 2024-03-11T21:43:07.083 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:43:07.084 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:43:07.084 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:43:07.116 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:43:06 smithi092 ceph-mon[29572]: pgmap v821: 158 pgs: 1 active+recovery_wait, 153 active+clean, 4 active+recovering+undersized+remapped; 4.6 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 1.3 KiB/s rd, 0 B/s wr, 790 op/s; 8910/228585 objects misplaced (3.898%); 11 MiB/s, 180 objects/s recovering 2024-03-11T21:43:07.116 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:43:06 smithi092 ceph-mon[37563]: pgmap v821: 158 pgs: 1 active+recovery_wait, 153 active+clean, 4 active+recovering+undersized+remapped; 4.6 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 1.3 KiB/s rd, 0 B/s wr, 790 op/s; 8910/228585 objects misplaced (3.898%); 11 MiB/s, 180 objects/s recovering 2024-03-11T21:43:07.184 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:43:07.207 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:43:07.208 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:43:07.208 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:43:07.309 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:43:07.351 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:43:07.351 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:43:07.351 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:43:07.452 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:43:07.498 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:43:07.498 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:43:07.498 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:43:07.599 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:43:07.638 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:43:07.639 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:43:07.639 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:43:07.740 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:43:07.765 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:43:07.766 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:43:07.766 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:43:07.867 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:43:07.901 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:43:07.901 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:43:07.902 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:43:08.002 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:43:08.043 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:43:08.044 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:43:08.045 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:43:08.145 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:43:08.181 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:43:08.181 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:43:08.181 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:43:08.282 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:43:08.317 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:43:08.318 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:43:08.319 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:43:08.366 INFO:journalctl@ceph.mgr.y.smithi092.stdout:Mar 11 21:43:07 smithi092 ceph-00db0016-dfed-11ee-95c8-87774f69a715-mgr-y[29801]: ::ffff:172.21.15.97 - - [11/Mar/2024:21:43:07] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-11T21:43:08.420 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:43:08.481 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:43:08.482 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:43:08.482 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:43:08.583 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:43:08.606 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:43:08.607 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:43:08.608 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:43:08.708 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:43:08.753 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:43:08.754 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:43:08.754 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:43:08.855 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:43:08.866 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:43:08 smithi092 ceph-mon[29572]: pgmap v822: 158 pgs: 1 active+recovery_wait, 153 active+clean, 4 active+recovering+undersized+remapped; 4.6 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 639 B/s rd, 0 B/s wr, 756 op/s; 8910/225510 objects misplaced (3.951%); 6.7 MiB/s, 106 objects/s recovering 2024-03-11T21:43:08.866 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:43:08 smithi092 ceph-mon[29572]: from='mgr.14607 v1:172.21.15.92:0/1702717597' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T21:43:08.866 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:43:08 smithi092 ceph-mon[37563]: pgmap v822: 158 pgs: 1 active+recovery_wait, 153 active+clean, 4 active+recovering+undersized+remapped; 4.6 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 639 B/s rd, 0 B/s wr, 756 op/s; 8910/225510 objects misplaced (3.951%); 6.7 MiB/s, 106 objects/s recovering 2024-03-11T21:43:08.866 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:43:08 smithi092 ceph-mon[37563]: from='mgr.14607 v1:172.21.15.92:0/1702717597' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T21:43:08.879 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:43:08.880 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:43:08.880 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:43:08.981 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:43:09.015 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:43:09.015 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:43:09.016 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:43:09.082 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:43:08 smithi097 ceph-mon[33317]: pgmap v822: 158 pgs: 1 active+recovery_wait, 153 active+clean, 4 active+recovering+undersized+remapped; 4.6 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 639 B/s rd, 0 B/s wr, 756 op/s; 8910/225510 objects misplaced (3.951%); 6.7 MiB/s, 106 objects/s recovering 2024-03-11T21:43:09.082 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:43:08 smithi097 ceph-mon[33317]: from='mgr.14607 v1:172.21.15.92:0/1702717597' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T21:43:09.116 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:43:09.160 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:43:09.161 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:43:09.161 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:43:09.262 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:43:09.293 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:43:09.294 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:43:09.294 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:43:09.395 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:43:09.427 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:43:09.427 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:43:09.428 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:43:09.529 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:43:09.567 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:43:09.568 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:43:09.568 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:43:09.669 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:43:09.701 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:43:09.702 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:43:09.702 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:43:09.803 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:43:09.831 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:43:09.832 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:43:09.832 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:43:09.933 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:43:09.966 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:43:09.967 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:43:09.967 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:43:10.068 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:43:10.106 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:43:10.107 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:43:10.107 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:43:10.190 INFO:teuthology.orchestra.run.smithi097.stdout:ERROR: (22) Invalid argument 2024-03-11T21:43:10.191 INFO:teuthology.orchestra.run.smithi097.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-11T21:43:10.208 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:43:10.243 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:43:10.244 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:43:10.244 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:43:10.345 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:43:10.376 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:43:10.377 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:43:10.377 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:43:10.478 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:43:10.504 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:43:10.504 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:43:10.505 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:43:10.606 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:43:10.642 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:43:10.643 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:43:10.643 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:43:10.744 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:43:10.786 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:43:10.787 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:43:10.787 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:43:10.888 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:43:10.930 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:43:10.931 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:43:10 smithi097 ceph-mon[33317]: pgmap v823: 158 pgs: 1 active+recovery_wait, 153 active+clean, 4 active+recovering+undersized+remapped; 4.4 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 1.0 KiB/s rd, 0 B/s wr, 823 op/s; 7584/218118 objects misplaced (3.477%); 8.3 MiB/s, 132 objects/s recovering 2024-03-11T21:43:10.931 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:43:10.932 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:43:10.974 DEBUG:teuthology.orchestra.run.smithi092:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid 00db0016-dfed-11ee-95c8-87774f69a715 -- ceph daemon osd.3 dump_ops_in_flight 2024-03-11T21:43:11.032 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:43:11.062 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:43:11.063 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:43:11.063 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:43:11.116 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:43:10 smithi092 ceph-mon[29572]: pgmap v823: 158 pgs: 1 active+recovery_wait, 153 active+clean, 4 active+recovering+undersized+remapped; 4.4 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 1.0 KiB/s rd, 0 B/s wr, 823 op/s; 7584/218118 objects misplaced (3.477%); 8.3 MiB/s, 132 objects/s recovering 2024-03-11T21:43:11.116 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:43:10 smithi092 ceph-mon[37563]: pgmap v823: 158 pgs: 1 active+recovery_wait, 153 active+clean, 4 active+recovering+undersized+remapped; 4.4 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 1.0 KiB/s rd, 0 B/s wr, 823 op/s; 7584/218118 objects misplaced (3.477%); 8.3 MiB/s, 132 objects/s recovering 2024-03-11T21:43:11.164 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:43:11.233 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:43:11.233 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:43:11.233 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:43:11.334 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:43:11.389 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:43:11.390 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:43:11.390 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:43:11.491 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:43:11.523 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:43:11.523 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:43:11.524 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:43:11.624 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:43:11.654 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:43:11.655 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:43:11.655 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:43:11.756 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:43:11.784 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:43:11.784 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:43:11.785 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:43:11.885 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:43:11.944 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:43:11.945 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:43:11.945 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:43:12.046 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:43:12.075 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:43:12.076 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:43:12.076 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:43:12.177 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:43:12.213 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:43:12.214 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:43:12.214 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:43:12.315 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:43:12.339 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:43:12.340 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:43:12.340 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:43:12.441 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:43:12.484 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:43:12.485 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:43:12.485 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:43:12.586 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:43:12.622 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:43:12.623 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:43:12.623 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:43:12.724 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:43:12.770 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:43:12.770 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:43:12.771 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:43:12.872 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:43:12.904 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:43:12.904 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:43:12.905 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:43:13.006 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:43:13.041 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:43:13.041 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:43:13.042 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:43:13.059 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:43:12 smithi092 ceph-mon[29572]: pgmap v824: 158 pgs: 1 active+recovery_wait, 153 active+clean, 4 active+recovering+undersized+remapped; 4.4 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 963 B/s rd, 0 B/s wr, 797 op/s; 7085/216501 objects misplaced (3.273%); 8.5 MiB/s, 135 objects/s recovering 2024-03-11T21:43:13.060 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:43:12 smithi092 ceph-mon[37563]: pgmap v824: 158 pgs: 1 active+recovery_wait, 153 active+clean, 4 active+recovering+undersized+remapped; 4.4 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 963 B/s rd, 0 B/s wr, 797 op/s; 7085/216501 objects misplaced (3.273%); 8.5 MiB/s, 135 objects/s recovering 2024-03-11T21:43:13.082 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:43:12 smithi097 ceph-mon[33317]: pgmap v824: 158 pgs: 1 active+recovery_wait, 153 active+clean, 4 active+recovering+undersized+remapped; 4.4 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 963 B/s rd, 0 B/s wr, 797 op/s; 7085/216501 objects misplaced (3.273%); 8.5 MiB/s, 135 objects/s recovering 2024-03-11T21:43:13.082 INFO:journalctl@ceph.iscsi.iscsi.a.smithi097.stdout:Mar 11 21:43:12 smithi097 ceph-00db0016-dfed-11ee-95c8-87774f69a715-iscsi-iscsi-a[60943]: debug there is no tcmu-runner data available 2024-03-11T21:43:13.142 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:43:13.187 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:43:13.188 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:43:13.188 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:43:13.289 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:43:13.333 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:43:13.334 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:43:13.334 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:43:13.435 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:43:13.478 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:43:13.479 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:43:13.479 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:43:13.579 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:43:13.610 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:43:13.611 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:43:13.611 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:43:13.712 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:43:14.082 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:43:13 smithi097 ceph-mon[33317]: from='client.24469 v1:172.21.15.97:0/404208310' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:43:14.116 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:43:13 smithi092 ceph-mon[29572]: from='client.24469 v1:172.21.15.97:0/404208310' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:43:14.116 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:43:13 smithi092 ceph-mon[37563]: from='client.24469 v1:172.21.15.97:0/404208310' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:43:14.563 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:43:14.564 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:43:14.565 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:43:14.666 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:43:14.703 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:43:14.703 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:43:14.703 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:43:14.804 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:43:14.845 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:43:14.846 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:43:14.846 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:43:14.948 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:43:14.983 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:43:14.984 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:43:14.984 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:43:15.082 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:43:14 smithi097 ceph-mon[33317]: pgmap v825: 158 pgs: 1 active+recovery_wait, 153 active+clean, 4 active+recovering+undersized+remapped; 4.2 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 852 B/s rd, 0 B/s wr, 1.02k op/s; 5691/205170 objects misplaced (2.774%); 11 MiB/s, 171 objects/s recovering 2024-03-11T21:43:15.086 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:43:15.109 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:43:15.110 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:43:15.110 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:43:15.116 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:43:14 smithi092 ceph-mon[29572]: pgmap v825: 158 pgs: 1 active+recovery_wait, 153 active+clean, 4 active+recovering+undersized+remapped; 4.2 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 852 B/s rd, 0 B/s wr, 1.02k op/s; 5691/205170 objects misplaced (2.774%); 11 MiB/s, 171 objects/s recovering 2024-03-11T21:43:15.116 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:43:14 smithi092 ceph-mon[37563]: pgmap v825: 158 pgs: 1 active+recovery_wait, 153 active+clean, 4 active+recovering+undersized+remapped; 4.2 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 852 B/s rd, 0 B/s wr, 1.02k op/s; 5691/205170 objects misplaced (2.774%); 11 MiB/s, 171 objects/s recovering 2024-03-11T21:43:15.211 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:43:15.258 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:43:15.259 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:43:15.259 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:43:15.360 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:43:15.395 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:43:15.396 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:43:15.396 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:43:15.441 INFO:teuthology.orchestra.run.smithi092.stderr:Inferring config /var/lib/ceph/00db0016-dfed-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T21:43:15.498 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:43:15.528 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:43:15.528 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:43:15.529 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:43:15.629 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:43:15.682 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:43:15.683 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:43:15.684 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:43:15.784 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:43:15.816 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:43:15.816 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:43:15.817 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:43:15.918 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:43:15.947 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:43:15.948 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:43:15.948 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:43:16.049 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:43:16.103 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:43:16.103 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:43:16.104 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:43:16.204 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:43:16.250 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:43:16.250 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:43:16.251 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:43:16.352 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:43:16.381 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:43:16.382 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:43:16.382 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:43:16.483 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:43:16.522 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:43:16.523 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:43:16.524 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:43:16.624 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:43:16.667 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:43:16.667 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:43:16.667 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:43:16.768 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:43:16.802 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:43:16.803 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:43:16.803 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:43:16.904 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:43:16.953 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:43:16.954 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:43:16.955 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:43:17.055 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:43:17.066 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:43:16 smithi092 ceph-mon[29572]: pgmap v826: 158 pgs: 1 active+recovery_wait, 153 active+clean, 4 active+recovering+undersized+remapped; 4.1 GiB data, 19 GiB used, 697 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 797 op/s; 5077/202296 objects misplaced (2.510%); 9.0 MiB/s, 144 objects/s recovering 2024-03-11T21:43:17.066 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:43:16 smithi092 ceph-mon[29572]: purged_snaps scrub starts 2024-03-11T21:43:17.067 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:43:16 smithi092 ceph-mon[29572]: purged_snaps scrub ok 2024-03-11T21:43:17.067 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:43:16 smithi092 ceph-mon[29572]: osdmap e404: 8 total, 8 up, 8 in 2024-03-11T21:43:17.067 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:43:16 smithi092 ceph-mon[29572]: from='client.? v1:172.21.15.92:0/3298144807' entity='client.admin' cmd=[{"prefix": "osd pg-upmap-items", "pgid": "2.12", "id": [3, 7, 4, 0, 1, 5]}]: dispatch 2024-03-11T21:43:17.067 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:43:16 smithi092 ceph-mon[29572]: from='client.35669 ' entity='client.admin' cmd=[{"prefix": "osd pg-upmap-items", "pgid": "2.12", "id": [3, 7, 4, 0, 1, 5]}]: dispatch 2024-03-11T21:43:17.067 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:43:16 smithi092 ceph-mon[37563]: pgmap v826: 158 pgs: 1 active+recovery_wait, 153 active+clean, 4 active+recovering+undersized+remapped; 4.1 GiB data, 19 GiB used, 697 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 797 op/s; 5077/202296 objects misplaced (2.510%); 9.0 MiB/s, 144 objects/s recovering 2024-03-11T21:43:17.067 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:43:16 smithi092 ceph-mon[37563]: purged_snaps scrub starts 2024-03-11T21:43:17.067 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:43:16 smithi092 ceph-mon[37563]: purged_snaps scrub ok 2024-03-11T21:43:17.067 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:43:16 smithi092 ceph-mon[37563]: osdmap e404: 8 total, 8 up, 8 in 2024-03-11T21:43:17.067 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:43:16 smithi092 ceph-mon[37563]: from='client.? v1:172.21.15.92:0/3298144807' entity='client.admin' cmd=[{"prefix": "osd pg-upmap-items", "pgid": "2.12", "id": [3, 7, 4, 0, 1, 5]}]: dispatch 2024-03-11T21:43:17.067 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:43:16 smithi092 ceph-mon[37563]: from='client.35669 ' entity='client.admin' cmd=[{"prefix": "osd pg-upmap-items", "pgid": "2.12", "id": [3, 7, 4, 0, 1, 5]}]: dispatch 2024-03-11T21:43:17.082 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:43:16 smithi097 ceph-mon[33317]: pgmap v826: 158 pgs: 1 active+recovery_wait, 153 active+clean, 4 active+recovering+undersized+remapped; 4.1 GiB data, 19 GiB used, 697 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 797 op/s; 5077/202296 objects misplaced (2.510%); 9.0 MiB/s, 144 objects/s recovering 2024-03-11T21:43:17.082 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:43:16 smithi097 ceph-mon[33317]: purged_snaps scrub starts 2024-03-11T21:43:17.082 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:43:16 smithi097 ceph-mon[33317]: purged_snaps scrub ok 2024-03-11T21:43:17.082 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:43:16 smithi097 ceph-mon[33317]: osdmap e404: 8 total, 8 up, 8 in 2024-03-11T21:43:17.082 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:43:16 smithi097 ceph-mon[33317]: from='client.? v1:172.21.15.92:0/3298144807' entity='client.admin' cmd=[{"prefix": "osd pg-upmap-items", "pgid": "2.12", "id": [3, 7, 4, 0, 1, 5]}]: dispatch 2024-03-11T21:43:17.082 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:43:16 smithi097 ceph-mon[33317]: from='client.35669 ' entity='client.admin' cmd=[{"prefix": "osd pg-upmap-items", "pgid": "2.12", "id": [3, 7, 4, 0, 1, 5]}]: dispatch 2024-03-11T21:43:17.084 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:43:17.085 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:43:17.085 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:43:17.186 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:43:17.214 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:43:17.214 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:43:17.215 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:43:17.316 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:43:17.345 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:43:17.346 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:43:17.346 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:43:17.447 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:43:17.477 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:43:17.477 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:43:17.478 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:43:17.578 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:43:17.606 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:43:17.607 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:43:17.607 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:43:17.708 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:43:17.738 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:43:17.738 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:43:17.739 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:43:17.840 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:43:17.869 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:43:17.869 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:43:17.870 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:43:17.970 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:43:18.116 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:43:17 smithi092 ceph-mon[29572]: from='client.35669 ' entity='client.admin' cmd='[{"prefix": "osd pg-upmap-items", "pgid": "2.12", "id": [3, 7, 4, 0, 1, 5]}]': finished 2024-03-11T21:43:18.116 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:43:17 smithi092 ceph-mon[29572]: osdmap e405: 8 total, 8 up, 8 in 2024-03-11T21:43:18.117 INFO:journalctl@ceph.mgr.y.smithi092.stdout:Mar 11 21:43:17 smithi092 ceph-00db0016-dfed-11ee-95c8-87774f69a715-mgr-y[29801]: ::ffff:172.21.15.97 - - [11/Mar/2024:21:43:17] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-11T21:43:18.117 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:43:17 smithi092 ceph-mon[37563]: from='client.35669 ' entity='client.admin' cmd='[{"prefix": "osd pg-upmap-items", "pgid": "2.12", "id": [3, 7, 4, 0, 1, 5]}]': finished 2024-03-11T21:43:18.117 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:43:17 smithi092 ceph-mon[37563]: osdmap e405: 8 total, 8 up, 8 in 2024-03-11T21:43:18.136 INFO:teuthology.orchestra.run.smithi092.stderr:set 2.12 pg_upmap_items mapping to [3->7,4->0,1->5] 2024-03-11T21:43:18.332 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:43:17 smithi097 ceph-mon[33317]: from='client.35669 ' entity='client.admin' cmd='[{"prefix": "osd pg-upmap-items", "pgid": "2.12", "id": [3, 7, 4, 0, 1, 5]}]': finished 2024-03-11T21:43:18.332 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:43:17 smithi097 ceph-mon[33317]: osdmap e405: 8 total, 8 up, 8 in 2024-03-11T21:43:18.511 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:43:18.512 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:43:18.512 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:43:18.613 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:43:18.650 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:43:18.651 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:43:18.651 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:43:18.752 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:43:18.791 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:43:18.791 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:43:18.793 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:43:18.894 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:43:18.939 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:43:18.939 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:43:18.940 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:43:19.041 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:43:19.083 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:43:19.083 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:43:19.083 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:43:19.116 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:43:18 smithi092 ceph-mon[29572]: pgmap v829: 158 pgs: 1 active+recovery_wait, 153 active+clean, 4 active+recovering+undersized+remapped; 4.0 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 639 B/s rd, 0 B/s wr, 812 op/s; 5077/198633 objects misplaced (2.556%); 7.7 MiB/s, 123 objects/s recovering 2024-03-11T21:43:19.116 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:43:18 smithi092 ceph-mon[29572]: osdmap e406: 8 total, 8 up, 8 in 2024-03-11T21:43:19.116 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:43:18 smithi092 ceph-mon[37563]: pgmap v829: 158 pgs: 1 active+recovery_wait, 153 active+clean, 4 active+recovering+undersized+remapped; 4.0 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 639 B/s rd, 0 B/s wr, 812 op/s; 5077/198633 objects misplaced (2.556%); 7.7 MiB/s, 123 objects/s recovering 2024-03-11T21:43:19.117 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:43:18 smithi092 ceph-mon[37563]: osdmap e406: 8 total, 8 up, 8 in 2024-03-11T21:43:19.184 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:43:19.222 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:43:19.223 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:43:19.223 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:43:19.324 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:43:19.332 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:43:18 smithi097 ceph-mon[33317]: pgmap v829: 158 pgs: 1 active+recovery_wait, 153 active+clean, 4 active+recovering+undersized+remapped; 4.0 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 639 B/s rd, 0 B/s wr, 812 op/s; 5077/198633 objects misplaced (2.556%); 7.7 MiB/s, 123 objects/s recovering 2024-03-11T21:43:19.332 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:43:18 smithi097 ceph-mon[33317]: osdmap e406: 8 total, 8 up, 8 in 2024-03-11T21:43:19.364 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:43:19.365 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:43:19.365 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:43:19.466 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:43:19.496 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:43:19.497 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:43:19.497 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:43:19.598 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:43:19.632 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:43:19.633 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:43:19.633 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:43:19.734 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:43:19.777 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:43:19.777 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:43:19.777 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:43:19.878 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:43:19.922 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:43:19.922 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:43:19.923 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:43:20.024 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:43:20.061 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:43:20.061 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:43:20.062 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:43:20.116 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:43:19 smithi092 ceph-mon[29572]: Health check update: Degraded data redundancy: 3 pgs undersized (PG_DEGRADED) 2024-03-11T21:43:20.116 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:43:19 smithi092 ceph-mon[29572]: osdmap e407: 8 total, 8 up, 8 in 2024-03-11T21:43:20.116 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:43:19 smithi092 ceph-mon[37563]: Health check update: Degraded data redundancy: 3 pgs undersized (PG_DEGRADED) 2024-03-11T21:43:20.117 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:43:19 smithi092 ceph-mon[37563]: osdmap e407: 8 total, 8 up, 8 in 2024-03-11T21:43:20.163 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:43:20.192 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:43:20.192 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:43:20.193 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:43:20.294 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:43:20.324 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:43:20.325 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:43:20.325 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:43:20.332 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:43:19 smithi097 ceph-mon[33317]: Health check update: Degraded data redundancy: 3 pgs undersized (PG_DEGRADED) 2024-03-11T21:43:20.332 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:43:19 smithi097 ceph-mon[33317]: osdmap e407: 8 total, 8 up, 8 in 2024-03-11T21:43:20.426 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:43:20.454 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:43:20.454 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:43:20.455 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:43:20.540 INFO:teuthology.orchestra.run.smithi092.stderr:Inferring config /var/lib/ceph/00db0016-dfed-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T21:43:20.556 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:43:20.591 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:43:20.592 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:43:20.592 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:43:20.693 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:43:20.718 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:43:20.718 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:43:20.719 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:43:20.819 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:43:20.843 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:43:20.844 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:43:20.844 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:43:20.945 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:43:20.983 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:43:20.983 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:43:20.984 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:43:21.084 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:43:21.116 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:43:20 smithi092 ceph-mon[29572]: pgmap v831: 158 pgs: 1 active+recovery_wait, 154 active+clean, 3 active+recovering+undersized+remapped; 3.9 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 1.7 KiB/s rd, 0 B/s wr, 839 op/s; 3207/190083 objects misplaced (1.687%); 11 MiB/s, 168 objects/s recovering 2024-03-11T21:43:21.117 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:43:20 smithi092 ceph-mon[29572]: 4.b scrub starts 2024-03-11T21:43:21.117 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:43:20 smithi092 ceph-mon[29572]: 4.b scrub ok 2024-03-11T21:43:21.117 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:43:20 smithi092 ceph-mon[29572]: 10.16 scrub starts 2024-03-11T21:43:21.117 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:43:20 smithi092 ceph-mon[29572]: 6.1c scrub starts 2024-03-11T21:43:21.117 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:43:20 smithi092 ceph-mon[29572]: 6.1c scrub ok 2024-03-11T21:43:21.117 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:43:20 smithi092 ceph-mon[29572]: osdmap e408: 8 total, 8 up, 8 in 2024-03-11T21:43:21.117 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:43:20 smithi092 ceph-mon[37563]: pgmap v831: 158 pgs: 1 active+recovery_wait, 154 active+clean, 3 active+recovering+undersized+remapped; 3.9 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 1.7 KiB/s rd, 0 B/s wr, 839 op/s; 3207/190083 objects misplaced (1.687%); 11 MiB/s, 168 objects/s recovering 2024-03-11T21:43:21.117 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:43:20 smithi092 ceph-mon[37563]: 4.b scrub starts 2024-03-11T21:43:21.117 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:43:20 smithi092 ceph-mon[37563]: 4.b scrub ok 2024-03-11T21:43:21.117 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:43:20 smithi092 ceph-mon[37563]: 10.16 scrub starts 2024-03-11T21:43:21.117 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:43:20 smithi092 ceph-mon[37563]: 6.1c scrub starts 2024-03-11T21:43:21.118 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:43:20 smithi092 ceph-mon[37563]: 6.1c scrub ok 2024-03-11T21:43:21.118 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:43:20 smithi092 ceph-mon[37563]: osdmap e408: 8 total, 8 up, 8 in 2024-03-11T21:43:21.119 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:43:21.120 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:43:21.120 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:43:21.221 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:43:21.232 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:43:20 smithi097 ceph-mon[33317]: pgmap v831: 158 pgs: 1 active+recovery_wait, 154 active+clean, 3 active+recovering+undersized+remapped; 3.9 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 1.7 KiB/s rd, 0 B/s wr, 839 op/s; 3207/190083 objects misplaced (1.687%); 11 MiB/s, 168 objects/s recovering 2024-03-11T21:43:21.233 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:43:20 smithi097 ceph-mon[33317]: 4.b scrub starts 2024-03-11T21:43:21.233 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:43:20 smithi097 ceph-mon[33317]: 4.b scrub ok 2024-03-11T21:43:21.233 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:43:20 smithi097 ceph-mon[33317]: 10.16 scrub starts 2024-03-11T21:43:21.233 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:43:20 smithi097 ceph-mon[33317]: 6.1c scrub starts 2024-03-11T21:43:21.233 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:43:20 smithi097 ceph-mon[33317]: 6.1c scrub ok 2024-03-11T21:43:21.233 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:43:20 smithi097 ceph-mon[33317]: osdmap e408: 8 total, 8 up, 8 in 2024-03-11T21:43:21.254 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:43:21.255 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:43:21.255 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:43:21.356 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:43:21.383 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:43:21.383 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:43:21.384 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:43:21.485 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:43:21.508 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:43:21.509 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:43:21.509 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:43:21.610 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:43:21.635 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:43:21.636 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:43:21.636 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:43:21.737 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:43:21.763 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:43:21.764 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:43:21.764 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:43:21.865 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:43:21.902 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:43:21.903 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:43:21.903 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:43:22.004 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:43:22.028 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:43:22.028 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:43:22.028 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:43:22.082 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:43:21 smithi097 ceph-mon[33317]: 10.16 scrub ok 2024-03-11T21:43:22.082 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:43:21 smithi097 ceph-mon[33317]: osdmap e409: 8 total, 8 up, 8 in 2024-03-11T21:43:22.129 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:43:22.169 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:43:22.169 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:43:22.170 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:43:22.200 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:43:21 smithi092 ceph-mon[29572]: 10.16 scrub ok 2024-03-11T21:43:22.200 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:43:21 smithi092 ceph-mon[29572]: osdmap e409: 8 total, 8 up, 8 in 2024-03-11T21:43:22.201 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:43:21 smithi092 ceph-mon[37563]: 10.16 scrub ok 2024-03-11T21:43:22.201 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:43:21 smithi092 ceph-mon[37563]: osdmap e409: 8 total, 8 up, 8 in 2024-03-11T21:43:22.273 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:43:22.308 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:43:22.309 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:43:22.309 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:43:22.410 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:43:22.451 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:43:22.452 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:43:22.452 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:43:22.553 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:43:22.595 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:43:22.596 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:43:22.596 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:43:22.697 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:43:22.739 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:43:22.740 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:43:22.741 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:43:22.841 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:43:22.875 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:43:22.876 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:43:22.876 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:43:22.977 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:43:23.020 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:43:23.021 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:43:23.021 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:43:23.116 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:43:22 smithi092 ceph-mon[29572]: pgmap v834: 158 pgs: 155 active+clean, 3 active+recovering+undersized+remapped; 3.8 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 1.1 KiB/s rd, 0 B/s wr, 1.02k op/s; 2109/188340 objects misplaced (1.120%); 16 MiB/s, 257 objects/s recovering 2024-03-11T21:43:23.116 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:43:22 smithi092 ceph-mon[29572]: 6.18 scrub starts 2024-03-11T21:43:23.116 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:43:22 smithi092 ceph-mon[29572]: 6.18 scrub ok 2024-03-11T21:43:23.116 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:43:22 smithi092 ceph-mon[29572]: 10.12 scrub starts 2024-03-11T21:43:23.117 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:43:22 smithi092 ceph-mon[29572]: from='mgr.14607 v1:172.21.15.92:0/1702717597' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T21:43:23.117 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:43:22 smithi092 ceph-mon[37563]: pgmap v834: 158 pgs: 155 active+clean, 3 active+recovering+undersized+remapped; 3.8 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 1.1 KiB/s rd, 0 B/s wr, 1.02k op/s; 2109/188340 objects misplaced (1.120%); 16 MiB/s, 257 objects/s recovering 2024-03-11T21:43:23.117 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:43:22 smithi092 ceph-mon[37563]: 6.18 scrub starts 2024-03-11T21:43:23.117 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:43:22 smithi092 ceph-mon[37563]: 6.18 scrub ok 2024-03-11T21:43:23.117 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:43:22 smithi092 ceph-mon[37563]: 10.12 scrub starts 2024-03-11T21:43:23.117 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:43:22 smithi092 ceph-mon[37563]: from='mgr.14607 v1:172.21.15.92:0/1702717597' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T21:43:23.121 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:43:23.152 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:43:23.152 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:43:23.153 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:43:23.226 INFO:journalctl@ceph.iscsi.iscsi.a.smithi097.stdout:Mar 11 21:43:23 smithi097 ceph-00db0016-dfed-11ee-95c8-87774f69a715-iscsi-iscsi-a[60943]: debug there is no tcmu-runner data available 2024-03-11T21:43:23.226 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:43:22 smithi097 ceph-mon[33317]: pgmap v834: 158 pgs: 155 active+clean, 3 active+recovering+undersized+remapped; 3.8 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 1.1 KiB/s rd, 0 B/s wr, 1.02k op/s; 2109/188340 objects misplaced (1.120%); 16 MiB/s, 257 objects/s recovering 2024-03-11T21:43:23.226 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:43:22 smithi097 ceph-mon[33317]: 6.18 scrub starts 2024-03-11T21:43:23.227 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:43:22 smithi097 ceph-mon[33317]: 6.18 scrub ok 2024-03-11T21:43:23.227 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:43:22 smithi097 ceph-mon[33317]: 10.12 scrub starts 2024-03-11T21:43:23.227 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:43:22 smithi097 ceph-mon[33317]: from='mgr.14607 v1:172.21.15.92:0/1702717597' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T21:43:23.254 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:43:23.302 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:43:23.302 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:43:23.303 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:43:23.393 INFO:teuthology.orchestra.run.smithi092.stdout:osd.0: {} 2024-03-11T21:43:23.404 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:43:23.440 INFO:teuthology.orchestra.run.smithi092.stdout:osd.1: {} 2024-03-11T21:43:23.450 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:43:23.450 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:43:23.451 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:43:23.488 INFO:teuthology.orchestra.run.smithi092.stdout:osd.2: {} 2024-03-11T21:43:23.525 INFO:teuthology.orchestra.run.smithi092.stdout:osd.3: {} 2024-03-11T21:43:23.551 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:43:23.555 INFO:teuthology.orchestra.run.smithi092.stdout:osd.4: {} 2024-03-11T21:43:23.577 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:43:23.577 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:43:23.578 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:43:23.580 INFO:teuthology.orchestra.run.smithi092.stdout:osd.5: {} 2024-03-11T21:43:23.623 INFO:teuthology.orchestra.run.smithi092.stdout:osd.6: {} 2024-03-11T21:43:23.662 INFO:teuthology.orchestra.run.smithi092.stdout:osd.7: {} 2024-03-11T21:43:23.669 INFO:teuthology.orchestra.run.smithi092.stderr:osd.0: osd_enable_op_tracker = '' 2024-03-11T21:43:23.669 INFO:teuthology.orchestra.run.smithi092.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-11T21:43:23.669 INFO:teuthology.orchestra.run.smithi092.stderr:osd.2: osd_enable_op_tracker = '' 2024-03-11T21:43:23.669 INFO:teuthology.orchestra.run.smithi092.stderr:osd.3: osd_enable_op_tracker = '' 2024-03-11T21:43:23.669 INFO:teuthology.orchestra.run.smithi092.stderr:osd.4: osd_enable_op_tracker = '' 2024-03-11T21:43:23.669 INFO:teuthology.orchestra.run.smithi092.stderr:osd.5: osd_enable_op_tracker = '' 2024-03-11T21:43:23.669 INFO:teuthology.orchestra.run.smithi092.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-11T21:43:23.669 INFO:teuthology.orchestra.run.smithi092.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-11T21:43:23.679 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:43:23.709 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:43:23.710 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:43:23.710 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:43:23.811 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:43:23.849 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:43:23.850 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:43:23.850 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:43:23.937 INFO:teuthology.orchestra.run.smithi092.stderr:nodeep-scrub is unset 2024-03-11T21:43:23.951 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:43:23.994 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:43:23.995 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:43:23.995 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:43:24.096 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:43:24.118 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:43:23 smithi092 ceph-mon[29572]: 5.15 scrub starts 2024-03-11T21:43:24.118 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:43:23 smithi092 ceph-mon[29572]: 5.15 scrub ok 2024-03-11T21:43:24.118 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:43:23 smithi092 ceph-mon[29572]: from='client.24469 v1:172.21.15.97:0/404208310' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:43:24.118 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:43:23 smithi092 ceph-mon[29572]: 10.12 scrub ok 2024-03-11T21:43:24.119 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:43:23 smithi092 ceph-mon[29572]: from='client.? v1:172.21.15.92:0/112611584' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-11T21:43:24.119 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:43:23 smithi092 ceph-mon[29572]: from='client.? v1:172.21.15.92:0/2455644991' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-11T21:43:24.119 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:43:23 smithi092 ceph-mon[29572]: from='client.35672 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-11T21:43:24.119 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:43:23 smithi092 ceph-mon[37563]: 5.15 scrub starts 2024-03-11T21:43:24.119 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:43:23 smithi092 ceph-mon[37563]: 5.15 scrub ok 2024-03-11T21:43:24.120 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:43:23 smithi092 ceph-mon[37563]: from='client.24469 v1:172.21.15.97:0/404208310' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:43:24.120 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:43:23 smithi092 ceph-mon[37563]: 10.12 scrub ok 2024-03-11T21:43:24.120 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:43:23 smithi092 ceph-mon[37563]: from='client.? v1:172.21.15.92:0/112611584' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-11T21:43:24.120 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:43:23 smithi092 ceph-mon[37563]: from='client.? v1:172.21.15.92:0/2455644991' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-11T21:43:24.120 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:43:23 smithi092 ceph-mon[37563]: from='client.35672 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-11T21:43:24.132 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:43:24.133 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:43:24.133 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:43:24.234 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:43:24.271 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:43:24.271 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:43:24.272 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:43:24.314 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5, 6, 7] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 4, 5, 6, 7, 3] 2024-03-11T21:43:24.314 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-03-11T21:43:24.315 INFO:tasks.thrashosds.thrasher:Shrinking pool .mgr 2024-03-11T21:43:24.315 INFO:tasks.cephadm.ceph_manager.ceph:contract_pool .mgr by 10 min 8 2024-03-11T21:43:24.315 DEBUG:teuthology.orchestra.run.smithi092:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid 00db0016-dfed-11ee-95c8-87774f69a715 -- ceph pg dump --format=json 2024-03-11T21:43:24.332 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:43:23 smithi097 ceph-mon[33317]: 5.15 scrub starts 2024-03-11T21:43:24.332 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:43:23 smithi097 ceph-mon[33317]: 5.15 scrub ok 2024-03-11T21:43:24.332 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:43:23 smithi097 ceph-mon[33317]: from='client.24469 v1:172.21.15.97:0/404208310' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:43:24.332 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:43:23 smithi097 ceph-mon[33317]: 10.12 scrub ok 2024-03-11T21:43:24.332 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:43:23 smithi097 ceph-mon[33317]: from='client.? v1:172.21.15.92:0/112611584' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-11T21:43:24.332 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:43:23 smithi097 ceph-mon[33317]: from='client.? v1:172.21.15.92:0/2455644991' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-11T21:43:24.332 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:43:23 smithi097 ceph-mon[33317]: from='client.35672 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-11T21:43:24.373 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:43:24.406 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:43:24.407 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:43:24.407 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:43:24.508 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:43:24.551 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:43:24.551 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:43:24.552 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:43:24.652 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:43:24.685 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:43:24.685 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:43:24.685 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:43:24.786 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:43:24.828 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:43:24.828 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:43:24.829 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:43:24.929 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:43:24.968 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:43:24.968 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:43:24.969 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:43:25.069 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:43:25.106 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:43:25.107 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:43:25.107 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:43:25.207 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:43:25.245 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:43:25.246 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:43:25.247 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:43:25.284 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:43:24 smithi092 ceph-mon[29572]: pgmap v836: 158 pgs: 1 active+clean+scrubbing, 155 active+clean, 2 active+recovering+undersized+remapped; 3.6 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 923 B/s rd, 0 op/s; 708/179217 objects misplaced (0.395%) 2024-03-11T21:43:25.285 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:43:24 smithi092 ceph-mon[29572]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-03-11T21:43:25.285 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:43:24 smithi092 ceph-mon[29572]: from='client.35672 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-03-11T21:43:25.285 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:43:24 smithi092 ceph-mon[29572]: osdmap e410: 8 total, 8 up, 8 in 2024-03-11T21:43:25.285 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:43:24 smithi092 ceph-mon[29572]: 5.f deep-scrub starts 2024-03-11T21:43:25.285 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:43:24 smithi092 ceph-mon[29572]: 5.f deep-scrub ok 2024-03-11T21:43:25.285 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:43:24 smithi092 ceph-mon[37563]: pgmap v836: 158 pgs: 1 active+clean+scrubbing, 155 active+clean, 2 active+recovering+undersized+remapped; 3.6 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 923 B/s rd, 0 op/s; 708/179217 objects misplaced (0.395%) 2024-03-11T21:43:25.285 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:43:24 smithi092 ceph-mon[37563]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-03-11T21:43:25.285 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:43:24 smithi092 ceph-mon[37563]: from='client.35672 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-03-11T21:43:25.285 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:43:24 smithi092 ceph-mon[37563]: osdmap e410: 8 total, 8 up, 8 in 2024-03-11T21:43:25.285 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:43:24 smithi092 ceph-mon[37563]: 5.f deep-scrub starts 2024-03-11T21:43:25.285 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:43:24 smithi092 ceph-mon[37563]: 5.f deep-scrub ok 2024-03-11T21:43:25.332 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:43:24 smithi097 ceph-mon[33317]: pgmap v836: 158 pgs: 1 active+clean+scrubbing, 155 active+clean, 2 active+recovering+undersized+remapped; 3.6 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 923 B/s rd, 0 op/s; 708/179217 objects misplaced (0.395%) 2024-03-11T21:43:25.332 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:43:24 smithi097 ceph-mon[33317]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-03-11T21:43:25.332 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:43:24 smithi097 ceph-mon[33317]: from='client.35672 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-03-11T21:43:25.332 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:43:24 smithi097 ceph-mon[33317]: osdmap e410: 8 total, 8 up, 8 in 2024-03-11T21:43:25.332 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:43:24 smithi097 ceph-mon[33317]: 5.f deep-scrub starts 2024-03-11T21:43:25.333 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:43:24 smithi097 ceph-mon[33317]: 5.f deep-scrub ok 2024-03-11T21:43:25.348 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:43:25.383 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:43:25.383 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:43:25.384 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:43:25.484 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:43:25.521 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:43:25.522 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:43:25.522 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:43:25.623 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:43:25.661 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:43:25.662 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:43:25.662 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:43:25.763 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:43:25.800 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:43:25.800 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:43:25.800 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:43:25.901 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:43:25.935 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:43:25.936 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:43:25.937 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:43:26.038 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:43:26.073 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:43:26.073 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:43:26.073 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:43:26.174 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:43:26.210 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:43:26.211 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:43:26.211 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:43:26.229 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:43:25 smithi092 ceph-mon[29572]: 10.11 deep-scrub starts 2024-03-11T21:43:26.229 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:43:25 smithi092 ceph-mon[29572]: 6.10 scrub starts 2024-03-11T21:43:26.229 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:43:25 smithi092 ceph-mon[29572]: 6.10 scrub ok 2024-03-11T21:43:26.229 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:43:25 smithi092 ceph-mon[29572]: 6.3 scrub starts 2024-03-11T21:43:26.230 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:43:25 smithi092 ceph-mon[37563]: 10.11 deep-scrub starts 2024-03-11T21:43:26.230 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:43:25 smithi092 ceph-mon[37563]: 6.10 scrub starts 2024-03-11T21:43:26.230 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:43:25 smithi092 ceph-mon[37563]: 6.10 scrub ok 2024-03-11T21:43:26.230 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:43:25 smithi092 ceph-mon[37563]: 6.3 scrub starts 2024-03-11T21:43:26.313 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:43:26.332 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:43:25 smithi097 ceph-mon[33317]: 10.11 deep-scrub starts 2024-03-11T21:43:26.332 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:43:25 smithi097 ceph-mon[33317]: 6.10 scrub starts 2024-03-11T21:43:26.332 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:43:25 smithi097 ceph-mon[33317]: 6.10 scrub ok 2024-03-11T21:43:26.332 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:43:25 smithi097 ceph-mon[33317]: 6.3 scrub starts 2024-03-11T21:43:26.346 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:43:26.347 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:43:26.347 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:43:26.448 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:43:26.481 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:43:26.482 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:43:26.482 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:43:26.577 INFO:teuthology.orchestra.run.smithi092.stderr:Inferring config /var/lib/ceph/00db0016-dfed-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T21:43:26.582 INFO:teuthology.orchestra.run.smithi092.stdout:{ 2024-03-11T21:43:26.582 INFO:teuthology.orchestra.run.smithi092.stdout: "ops": [], 2024-03-11T21:43:26.583 INFO:teuthology.orchestra.run.smithi092.stdout: "num_ops": 0 2024-03-11T21:43:26.583 INFO:teuthology.orchestra.run.smithi092.stdout:} 2024-03-11T21:43:26.583 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:43:26.612 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:43:26.612 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:43:26.613 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:43:26.714 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:43:26.742 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:43:26.743 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:43:26.743 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:43:26.844 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:43:26.869 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:43:26.870 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:43:26.870 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:43:26.971 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:43:27.007 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:43:27.007 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:43:27.008 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:43:27.108 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:43:27.142 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:43:27.143 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:43:27.144 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:43:27.245 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:43:27.277 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:43:27.277 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:43:27.278 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:43:27.332 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:43:26 smithi097 ceph-mon[33317]: pgmap v838: 158 pgs: 1 active+clean+scrubbing, 156 active+clean, 1 active+recovering+undersized+remapped; 3.6 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 925 B/s rd, 0 op/s; 498/176787 objects misplaced (0.282%) 2024-03-11T21:43:27.332 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:43:26 smithi097 ceph-mon[33317]: 6.3 scrub ok 2024-03-11T21:43:27.333 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:43:26 smithi097 ceph-mon[33317]: 4.1a deep-scrub starts 2024-03-11T21:43:27.333 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:43:26 smithi097 ceph-mon[33317]: 4.1a deep-scrub ok 2024-03-11T21:43:27.333 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:43:26 smithi097 ceph-mon[33317]: Health check update: Degraded data redundancy: 1 pg undersized (PG_DEGRADED) 2024-03-11T21:43:27.333 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:43:26 smithi097 ceph-mon[33317]: osdmap e411: 8 total, 8 up, 8 in 2024-03-11T21:43:27.333 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:43:26 smithi097 ceph-mon[33317]: 5.5 scrub starts 2024-03-11T21:43:27.333 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:43:26 smithi097 ceph-mon[33317]: 5.5 scrub ok 2024-03-11T21:43:27.333 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:43:26 smithi097 ceph-mon[33317]: purged_snaps scrub starts 2024-03-11T21:43:27.333 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:43:26 smithi097 ceph-mon[33317]: purged_snaps scrub ok 2024-03-11T21:43:27.334 DEBUG:teuthology.orchestra.run.smithi092:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid 00db0016-dfed-11ee-95c8-87774f69a715 -- ceph osd set noscrub 2024-03-11T21:43:27.364 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:43:26 smithi092 ceph-mon[29572]: pgmap v838: 158 pgs: 1 active+clean+scrubbing, 156 active+clean, 1 active+recovering+undersized+remapped; 3.6 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 925 B/s rd, 0 op/s; 498/176787 objects misplaced (0.282%) 2024-03-11T21:43:27.365 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:43:26 smithi092 ceph-mon[29572]: 6.3 scrub ok 2024-03-11T21:43:27.365 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:43:26 smithi092 ceph-mon[29572]: 4.1a deep-scrub starts 2024-03-11T21:43:27.365 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:43:26 smithi092 ceph-mon[29572]: 4.1a deep-scrub ok 2024-03-11T21:43:27.365 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:43:26 smithi092 ceph-mon[29572]: Health check update: Degraded data redundancy: 1 pg undersized (PG_DEGRADED) 2024-03-11T21:43:27.365 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:43:26 smithi092 ceph-mon[29572]: osdmap e411: 8 total, 8 up, 8 in 2024-03-11T21:43:27.365 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:43:26 smithi092 ceph-mon[29572]: 5.5 scrub starts 2024-03-11T21:43:27.365 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:43:26 smithi092 ceph-mon[29572]: 5.5 scrub ok 2024-03-11T21:43:27.365 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:43:26 smithi092 ceph-mon[29572]: purged_snaps scrub starts 2024-03-11T21:43:27.365 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:43:26 smithi092 ceph-mon[29572]: purged_snaps scrub ok 2024-03-11T21:43:27.365 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:43:26 smithi092 ceph-mon[37563]: pgmap v838: 158 pgs: 1 active+clean+scrubbing, 156 active+clean, 1 active+recovering+undersized+remapped; 3.6 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 925 B/s rd, 0 op/s; 498/176787 objects misplaced (0.282%) 2024-03-11T21:43:27.365 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:43:26 smithi092 ceph-mon[37563]: 6.3 scrub ok 2024-03-11T21:43:27.366 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:43:26 smithi092 ceph-mon[37563]: 4.1a deep-scrub starts 2024-03-11T21:43:27.366 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:43:26 smithi092 ceph-mon[37563]: 4.1a deep-scrub ok 2024-03-11T21:43:27.366 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:43:26 smithi092 ceph-mon[37563]: Health check update: Degraded data redundancy: 1 pg undersized (PG_DEGRADED) 2024-03-11T21:43:27.366 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:43:26 smithi092 ceph-mon[37563]: osdmap e411: 8 total, 8 up, 8 in 2024-03-11T21:43:27.366 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:43:26 smithi092 ceph-mon[37563]: 5.5 scrub starts 2024-03-11T21:43:27.366 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:43:26 smithi092 ceph-mon[37563]: 5.5 scrub ok 2024-03-11T21:43:27.366 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:43:26 smithi092 ceph-mon[37563]: purged_snaps scrub starts 2024-03-11T21:43:27.366 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:43:26 smithi092 ceph-mon[37563]: purged_snaps scrub ok 2024-03-11T21:43:27.378 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:43:27.427 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:43:27.428 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:43:27.428 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:43:27.454 DEBUG:teuthology.orchestra.run.smithi092:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid 00db0016-dfed-11ee-95c8-87774f69a715 -- ceph daemon osd.3 dump_blocked_ops 2024-03-11T21:43:27.528 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:43:27.568 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:43:27.569 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:43:27.569 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:43:27.670 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:43:27.704 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:43:27.705 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:43:27.705 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:43:27.807 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:43:27.843 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:43:27.844 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:43:27.844 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:43:27.946 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:43:27.971 DEBUG:teuthology.orchestra.run.smithi092:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid 00db0016-dfed-11ee-95c8-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-03-11T21:43:27.979 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:43:27.979 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:43:27.980 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:43:28.081 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:43:28.109 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:43:28.110 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:43:28.111 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:43:28.116 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:43:27 smithi092 ceph-mon[29572]: 5.10 deep-scrub starts 2024-03-11T21:43:28.116 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:43:27 smithi092 ceph-mon[29572]: 5.10 deep-scrub ok 2024-03-11T21:43:28.116 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:43:27 smithi092 ceph-mon[29572]: 10.10 deep-scrub starts 2024-03-11T21:43:28.116 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:43:27 smithi092 ceph-mon[29572]: osdmap e412: 8 total, 8 up, 8 in 2024-03-11T21:43:28.116 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:43:27 smithi092 ceph-mon[29572]: 10.13 scrub starts 2024-03-11T21:43:28.116 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:43:27 smithi092 ceph-mon[29572]: 2.11 scrub starts 2024-03-11T21:43:28.117 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:43:27 smithi092 ceph-mon[29572]: 2.11 scrub ok 2024-03-11T21:43:28.117 INFO:journalctl@ceph.mgr.y.smithi092.stdout:Mar 11 21:43:27 smithi092 ceph-00db0016-dfed-11ee-95c8-87774f69a715-mgr-y[29801]: ::ffff:172.21.15.97 - - [11/Mar/2024:21:43:27] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-11T21:43:28.117 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:43:27 smithi092 ceph-mon[37563]: 5.10 deep-scrub starts 2024-03-11T21:43:28.117 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:43:27 smithi092 ceph-mon[37563]: 5.10 deep-scrub ok 2024-03-11T21:43:28.117 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:43:27 smithi092 ceph-mon[37563]: 10.10 deep-scrub starts 2024-03-11T21:43:28.117 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:43:27 smithi092 ceph-mon[37563]: osdmap e412: 8 total, 8 up, 8 in 2024-03-11T21:43:28.117 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:43:27 smithi092 ceph-mon[37563]: 10.13 scrub starts 2024-03-11T21:43:28.117 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:43:27 smithi092 ceph-mon[37563]: 2.11 scrub starts 2024-03-11T21:43:28.117 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:43:27 smithi092 ceph-mon[37563]: 2.11 scrub ok 2024-03-11T21:43:28.211 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:43:28.250 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:43:28.251 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:43:28.252 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:43:28.332 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:43:27 smithi097 ceph-mon[33317]: 5.10 deep-scrub starts 2024-03-11T21:43:28.332 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:43:27 smithi097 ceph-mon[33317]: 5.10 deep-scrub ok 2024-03-11T21:43:28.332 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:43:27 smithi097 ceph-mon[33317]: 10.10 deep-scrub starts 2024-03-11T21:43:28.332 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:43:27 smithi097 ceph-mon[33317]: osdmap e412: 8 total, 8 up, 8 in 2024-03-11T21:43:28.332 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:43:27 smithi097 ceph-mon[33317]: 10.13 scrub starts 2024-03-11T21:43:28.332 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:43:27 smithi097 ceph-mon[33317]: 2.11 scrub starts 2024-03-11T21:43:28.332 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:43:27 smithi097 ceph-mon[33317]: 2.11 scrub ok 2024-03-11T21:43:28.353 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:43:28.384 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:43:28.385 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:43:28.385 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:43:28.486 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:43:28.523 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:43:28.524 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:43:28.524 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:43:28.625 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:43:28.659 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:43:28.660 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:43:28.660 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:43:28.761 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:43:28.796 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:43:28.796 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:43:28.796 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:43:28.897 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:43:28.934 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:43:28.934 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:43:28.935 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:43:28.998 INFO:teuthology.orchestra.run.smithi092.stderr:Inferring config /var/lib/ceph/00db0016-dfed-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T21:43:29.036 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:43:29.047 INFO:teuthology.orchestra.run.smithi092.stderr:Inferring config /var/lib/ceph/00db0016-dfed-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T21:43:29.058 INFO:teuthology.orchestra.run.smithi092.stderr:Inferring config /var/lib/ceph/00db0016-dfed-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T21:43:29.111 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:43:29.112 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:43:29.112 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:43:29.164 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:43:28 smithi092 ceph-mon[29572]: pgmap v841: 158 pgs: 2 active+clean+scrubbing+deep, 1 active+clean+scrubbing, 154 active+clean, 1 active+recovering+undersized+remapped; 3.5 GiB data, 17 GiB used, 699 GiB / 715 GiB avail; 925 B/s rd, 0 B/s wr, 36 op/s; 498/173007 objects misplaced (0.288%) 2024-03-11T21:43:29.164 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:43:28 smithi092 ceph-mon[29572]: 10.13 scrub ok 2024-03-11T21:43:29.164 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:43:28 smithi092 ceph-mon[29572]: 10.17 scrub starts 2024-03-11T21:43:29.164 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:43:28 smithi092 ceph-mon[29572]: 2.1f scrub starts 2024-03-11T21:43:29.164 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:43:28 smithi092 ceph-mon[29572]: 2.1f scrub ok 2024-03-11T21:43:29.164 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:43:28 smithi092 ceph-mon[29572]: 10.14 scrub starts 2024-03-11T21:43:29.164 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:43:28 smithi092 ceph-mon[29572]: 2.1 deep-scrub starts 2024-03-11T21:43:29.164 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:43:28 smithi092 ceph-mon[29572]: 2.1 deep-scrub ok 2024-03-11T21:43:29.165 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:43:28 smithi092 ceph-mon[29572]: 10.17 scrub ok 2024-03-11T21:43:29.165 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:43:28 smithi092 ceph-mon[37563]: pgmap v841: 158 pgs: 2 active+clean+scrubbing+deep, 1 active+clean+scrubbing, 154 active+clean, 1 active+recovering+undersized+remapped; 3.5 GiB data, 17 GiB used, 699 GiB / 715 GiB avail; 925 B/s rd, 0 B/s wr, 36 op/s; 498/173007 objects misplaced (0.288%) 2024-03-11T21:43:29.165 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:43:28 smithi092 ceph-mon[37563]: 10.13 scrub ok 2024-03-11T21:43:29.165 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:43:28 smithi092 ceph-mon[37563]: 10.17 scrub starts 2024-03-11T21:43:29.165 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:43:28 smithi092 ceph-mon[37563]: 2.1f scrub starts 2024-03-11T21:43:29.165 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:43:28 smithi092 ceph-mon[37563]: 2.1f scrub ok 2024-03-11T21:43:29.165 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:43:28 smithi092 ceph-mon[37563]: 10.14 scrub starts 2024-03-11T21:43:29.165 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:43:28 smithi092 ceph-mon[37563]: 2.1 deep-scrub starts 2024-03-11T21:43:29.165 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:43:28 smithi092 ceph-mon[37563]: 2.1 deep-scrub ok 2024-03-11T21:43:29.165 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:43:28 smithi092 ceph-mon[37563]: 10.17 scrub ok 2024-03-11T21:43:29.213 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:43:29.256 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:43:29.257 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:43:29.257 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:43:29.332 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:43:28 smithi097 ceph-mon[33317]: pgmap v841: 158 pgs: 2 active+clean+scrubbing+deep, 1 active+clean+scrubbing, 154 active+clean, 1 active+recovering+undersized+remapped; 3.5 GiB data, 17 GiB used, 699 GiB / 715 GiB avail; 925 B/s rd, 0 B/s wr, 36 op/s; 498/173007 objects misplaced (0.288%) 2024-03-11T21:43:29.332 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:43:28 smithi097 ceph-mon[33317]: 10.13 scrub ok 2024-03-11T21:43:29.332 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:43:28 smithi097 ceph-mon[33317]: 10.17 scrub starts 2024-03-11T21:43:29.332 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:43:28 smithi097 ceph-mon[33317]: 2.1f scrub starts 2024-03-11T21:43:29.332 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:43:28 smithi097 ceph-mon[33317]: 2.1f scrub ok 2024-03-11T21:43:29.333 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:43:28 smithi097 ceph-mon[33317]: 10.14 scrub starts 2024-03-11T21:43:29.333 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:43:28 smithi097 ceph-mon[33317]: 2.1 deep-scrub starts 2024-03-11T21:43:29.333 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:43:28 smithi097 ceph-mon[33317]: 2.1 deep-scrub ok 2024-03-11T21:43:29.333 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:43:28 smithi097 ceph-mon[33317]: 10.17 scrub ok 2024-03-11T21:43:29.358 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:43:29.397 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:43:29.398 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:43:29.398 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:43:29.499 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:43:29.535 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:43:29.536 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:43:29.536 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:43:29.637 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:43:29.671 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:43:29.672 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:43:29.672 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:43:29.773 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:43:29.814 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:43:29.815 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:43:29.816 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:43:29.916 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:43:29.949 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:43:29.950 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:43:29.951 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:43:29.981 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:43:29 smithi092 ceph-mon[29572]: 10.14 scrub ok 2024-03-11T21:43:29.981 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:43:29 smithi092 ceph-mon[29572]: 10.15 scrub starts 2024-03-11T21:43:29.981 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:43:29 smithi092 ceph-mon[29572]: 4.6 scrub starts 2024-03-11T21:43:29.981 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:43:29 smithi092 ceph-mon[29572]: 4.6 scrub ok 2024-03-11T21:43:29.981 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:43:29 smithi092 ceph-mon[29572]: 10.19 scrub starts 2024-03-11T21:43:29.981 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:43:29 smithi092 ceph-mon[29572]: 5.11 deep-scrub starts 2024-03-11T21:43:29.981 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:43:29 smithi092 ceph-mon[29572]: 5.11 deep-scrub ok 2024-03-11T21:43:29.981 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:43:29 smithi092 ceph-mon[37563]: 10.14 scrub ok 2024-03-11T21:43:29.981 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:43:29 smithi092 ceph-mon[37563]: 10.15 scrub starts 2024-03-11T21:43:29.982 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:43:29 smithi092 ceph-mon[37563]: 4.6 scrub starts 2024-03-11T21:43:29.982 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:43:29 smithi092 ceph-mon[37563]: 4.6 scrub ok 2024-03-11T21:43:29.982 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:43:29 smithi092 ceph-mon[37563]: 10.19 scrub starts 2024-03-11T21:43:29.982 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:43:29 smithi092 ceph-mon[37563]: 5.11 deep-scrub starts 2024-03-11T21:43:29.982 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:43:29 smithi092 ceph-mon[37563]: 5.11 deep-scrub ok 2024-03-11T21:43:30.052 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:43:30.097 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:43:30.097 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:43:30.098 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:43:30.199 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:43:30.244 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:43:30.244 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:43:30.245 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:43:30.332 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:43:29 smithi097 ceph-mon[33317]: 10.14 scrub ok 2024-03-11T21:43:30.332 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:43:29 smithi097 ceph-mon[33317]: 10.15 scrub starts 2024-03-11T21:43:30.332 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:43:29 smithi097 ceph-mon[33317]: 4.6 scrub starts 2024-03-11T21:43:30.332 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:43:29 smithi097 ceph-mon[33317]: 4.6 scrub ok 2024-03-11T21:43:30.332 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:43:29 smithi097 ceph-mon[33317]: 10.19 scrub starts 2024-03-11T21:43:30.333 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:43:29 smithi097 ceph-mon[33317]: 5.11 deep-scrub starts 2024-03-11T21:43:30.333 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:43:29 smithi097 ceph-mon[33317]: 5.11 deep-scrub ok 2024-03-11T21:43:30.345 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:43:30.378 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:43:30.379 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:43:30.379 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:43:30.422 INFO:teuthology.orchestra.run.smithi092.stdout: 2024-03-11T21:43:30.433 INFO:teuthology.orchestra.run.smithi092.stderr:dumped all 2024-03-11T21:43:30.480 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:43:30.511 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:43:30.512 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:43:30.512 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:43:30.613 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:43:30.640 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:43:30.641 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:43:30.641 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:43:30.742 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:43:30.772 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:43:30.773 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:43:30.773 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:43:30.874 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:43:30.910 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:43:30.910 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:43:30.911 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:43:31.012 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:43:31.052 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:43:31.052 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:43:31.053 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:43:31.154 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:43:31.187 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:43:31.187 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:43:31.188 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:43:31.289 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:43:31.320 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:43:30 smithi092 ceph-mon[37563]: 10.15 scrub ok 2024-03-11T21:43:31.320 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:43:30 smithi092 ceph-mon[37563]: pgmap v842: 158 pgs: 1 active+clean+scrubbing, 2 active+clean+scrubbing+deep, 155 active+clean; 3.3 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 1.7 KiB/s rd, 1 op/s 2024-03-11T21:43:31.320 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:43:30 smithi092 ceph-mon[37563]: 10.19 scrub ok 2024-03-11T21:43:31.320 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:43:30 smithi092 ceph-mon[37563]: 4.11 scrub starts 2024-03-11T21:43:31.320 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:43:30 smithi092 ceph-mon[37563]: 4.11 scrub ok 2024-03-11T21:43:31.320 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:43:30 smithi092 ceph-mon[37563]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 1 pg undersized) 2024-03-11T21:43:31.320 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:43:30 smithi092 ceph-mon[37563]: Cluster is now healthy 2024-03-11T21:43:31.321 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:43:30 smithi092 ceph-mon[37563]: 5.1 scrub starts 2024-03-11T21:43:31.321 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:43:30 smithi092 ceph-mon[37563]: 2.13 scrub starts 2024-03-11T21:43:31.321 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:43:30 smithi092 ceph-mon[37563]: 2.13 scrub ok 2024-03-11T21:43:31.321 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:43:30 smithi092 ceph-mon[37563]: 5.1 scrub ok 2024-03-11T21:43:31.321 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:43:30 smithi092 ceph-mon[37563]: 10.18 scrub starts 2024-03-11T21:43:31.321 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:43:30 smithi092 ceph-mon[37563]: 4.a scrub starts 2024-03-11T21:43:31.321 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:43:30 smithi092 ceph-mon[37563]: 4.a scrub ok 2024-03-11T21:43:31.321 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:43:30 smithi092 ceph-mon[37563]: purged_snaps scrub starts 2024-03-11T21:43:31.321 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:43:30 smithi092 ceph-mon[37563]: purged_snaps scrub ok 2024-03-11T21:43:31.321 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:43:30 smithi092 ceph-mon[29572]: 10.15 scrub ok 2024-03-11T21:43:31.322 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:43:30 smithi092 ceph-mon[29572]: pgmap v842: 158 pgs: 1 active+clean+scrubbing, 2 active+clean+scrubbing+deep, 155 active+clean; 3.3 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 1.7 KiB/s rd, 1 op/s 2024-03-11T21:43:31.322 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:43:30 smithi092 ceph-mon[29572]: 10.19 scrub ok 2024-03-11T21:43:31.322 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:43:30 smithi092 ceph-mon[29572]: 4.11 scrub starts 2024-03-11T21:43:31.322 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:43:30 smithi092 ceph-mon[29572]: 4.11 scrub ok 2024-03-11T21:43:31.322 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:43:30 smithi092 ceph-mon[29572]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 1 pg undersized) 2024-03-11T21:43:31.322 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:43:30 smithi092 ceph-mon[29572]: Cluster is now healthy 2024-03-11T21:43:31.322 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:43:30 smithi092 ceph-mon[29572]: 5.1 scrub starts 2024-03-11T21:43:31.322 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:43:30 smithi092 ceph-mon[29572]: 2.13 scrub starts 2024-03-11T21:43:31.322 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:43:30 smithi092 ceph-mon[29572]: 2.13 scrub ok 2024-03-11T21:43:31.322 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:43:30 smithi092 ceph-mon[29572]: 5.1 scrub ok 2024-03-11T21:43:31.322 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:43:30 smithi092 ceph-mon[29572]: 10.18 scrub starts 2024-03-11T21:43:31.322 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:43:30 smithi092 ceph-mon[29572]: 4.a scrub starts 2024-03-11T21:43:31.322 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:43:30 smithi092 ceph-mon[29572]: 4.a scrub ok 2024-03-11T21:43:31.323 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:43:30 smithi092 ceph-mon[29572]: purged_snaps scrub starts 2024-03-11T21:43:31.323 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:43:30 smithi092 ceph-mon[29572]: purged_snaps scrub ok 2024-03-11T21:43:31.332 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:43:30 smithi097 ceph-mon[33317]: 10.15 scrub ok 2024-03-11T21:43:31.332 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:43:30 smithi097 ceph-mon[33317]: pgmap v842: 158 pgs: 1 active+clean+scrubbing, 2 active+clean+scrubbing+deep, 155 active+clean; 3.3 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 1.7 KiB/s rd, 1 op/s 2024-03-11T21:43:31.332 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:43:30 smithi097 ceph-mon[33317]: 10.19 scrub ok 2024-03-11T21:43:31.333 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:43:30 smithi097 ceph-mon[33317]: 4.11 scrub starts 2024-03-11T21:43:31.333 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:43:30 smithi097 ceph-mon[33317]: 4.11 scrub ok 2024-03-11T21:43:31.333 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:43:30 smithi097 ceph-mon[33317]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 1 pg undersized) 2024-03-11T21:43:31.333 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:43:30 smithi097 ceph-mon[33317]: Cluster is now healthy 2024-03-11T21:43:31.333 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:43:30 smithi097 ceph-mon[33317]: 5.1 scrub starts 2024-03-11T21:43:31.333 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:43:30 smithi097 ceph-mon[33317]: 2.13 scrub starts 2024-03-11T21:43:31.333 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:43:30 smithi097 ceph-mon[33317]: 2.13 scrub ok 2024-03-11T21:43:31.333 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:43:30 smithi097 ceph-mon[33317]: 5.1 scrub ok 2024-03-11T21:43:31.333 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:43:30 smithi097 ceph-mon[33317]: 10.18 scrub starts 2024-03-11T21:43:31.333 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:43:30 smithi097 ceph-mon[33317]: 4.a scrub starts 2024-03-11T21:43:31.333 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:43:30 smithi097 ceph-mon[33317]: 4.a scrub ok 2024-03-11T21:43:31.333 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:43:30 smithi097 ceph-mon[33317]: purged_snaps scrub starts 2024-03-11T21:43:31.333 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:43:30 smithi097 ceph-mon[33317]: purged_snaps scrub ok 2024-03-11T21:43:31.338 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:43:31.338 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:43:31.338 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:43:31.439 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:43:31.475 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:43:31.476 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:43:31.477 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:43:31.577 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:43:31.613 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:43:31.613 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:43:31.614 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:43:31.715 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:43:31.747 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:43:31.747 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:43:31.748 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:43:31.848 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:43:31.886 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:43:31.887 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:43:31.887 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:43:31.904 INFO:teuthology.orchestra.run.smithi092.stdout:{"pg_ready":true,"pg_map":{"version":842,"stamp":"2024-03-11T21:43:29.393267+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":3563821504,"num_objects":54571,"num_object_clones":0,"num_object_copies":163713,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":54571,"num_whiteouts":0,"num_read":2397,"num_read_kb":2245,"num_write":129114,"num_write_kb":4972798,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15859,"num_bytes_recovered":1028987334,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":167563,"ondisk_log_size":167563,"up":474,"acting":474,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":487,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":749961216,"kb_used":16977044,"kb_used_data":12153684,"kb_used_omap":15,"kb_used_meta":4822064,"kb_avail":732984172,"statfs":{"total":767960285184,"available":750575792128,"internally_reserved":0,"allocated":12445372416,"data_stored":12441791120,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15601,"internal_metadata":4937794319},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[3,3,6,10,14,6],"upper_bound":64},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-338690048,"num_objects":-5168,"num_object_clones":0,"num_object_copies":-15504,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-708,"num_objects_unfound":0,"num_objects_dirty":-5168,"num_whiteouts":0,"num_read":10,"num_read_kb":10,"num_write":-5015,"num_write_kb":-440512,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":-4226,"num_bytes_recovered":-276955136,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.002099"},"pg_stats":[{"pgid":"10.17","version":"412'7383","reported_seq":11979,"reported_epoch":412,"state":"active+clean","last_fresh":"2024-03-11T21:43:27.061805+0000","last_change":"2024-03-11T21:42:08.480192+0000","last_active":"2024-03-11T21:43:27.061805+0000","last_peered":"2024-03-11T21:43:27.061805+0000","last_clean":"2024-03-11T21:43:27.061805+0000","last_became_active":"2024-03-11T21:42:08.476625+0000","last_became_peered":"2024-03-11T21:42:08.476625+0000","last_unstale":"2024-03-11T21:43:27.061805+0000","last_undegraded":"2024-03-11T21:43:27.061805+0000","last_fullsized":"2024-03-11T21:43:27.061805+0000","mapping_epoch":388,"log_start":"365'1417","ondisk_log_start":"365'1417","created":372,"last_epoch_clean":389,"parent":"0.0","parent_split_bits":5,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:40:38.219602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:40:38.219602+0000","last_clean_scrub_stamp":"2024-03-11T21:40:38.219602+0000","objects_scrubbed":0,"log_size":5966,"log_dups_size":0,"ondisk_log_size":5966,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112197632,"num_objects":1712,"num_object_clones":0,"num_object_copies":5136,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1712,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5198,"num_write_kb":221120,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2228,"num_bytes_recovered":146014208,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1b","version":"0'0","reported_seq":671,"reported_epoch":410,"state":"active+clean","last_fresh":"2024-03-11T21:43:23.939837+0000","last_change":"2024-03-11T21:36:35.074931+0000","last_active":"2024-03-11T21:43:23.939837+0000","last_peered":"2024-03-11T21:43:23.939837+0000","last_clean":"2024-03-11T21:43:23.939837+0000","last_became_active":"2024-03-11T21:36:35.072636+0000","last_became_peered":"2024-03-11T21:36:35.072636+0000","last_unstale":"2024-03-11T21:43:23.939837+0000","last_undegraded":"2024-03-11T21:43:23.939837+0000","last_fullsized":"2024-03-11T21:43:23.939837+0000","mapping_epoch":277,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":278,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:35:07.021520+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:43.559452+0000","last_clean_scrub_stamp":"2024-03-11T21:35:07.021520+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":748,"reported_epoch":412,"state":"active+clean","last_fresh":"2024-03-11T21:43:26.951697+0000","last_change":"2024-03-11T21:41:19.522757+0000","last_active":"2024-03-11T21:43:26.951697+0000","last_peered":"2024-03-11T21:43:26.951697+0000","last_clean":"2024-03-11T21:43:26.951697+0000","last_became_active":"2024-03-11T21:25:42.578893+0000","last_became_peered":"2024-03-11T21:25:42.578893+0000","last_unstale":"2024-03-11T21:43:26.951697+0000","last_undegraded":"2024-03-11T21:43:26.951697+0000","last_fullsized":"2024-03-11T21:43:26.951697+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:41:19.522682+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:41.544550+0000","last_clean_scrub_stamp":"2024-03-11T21:41:19.522682+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1f","version":"0'0","reported_seq":765,"reported_epoch":410,"state":"active+clean","last_fresh":"2024-03-11T21:43:23.932881+0000","last_change":"2024-03-11T21:36:00.843838+0000","last_active":"2024-03-11T21:43:23.932881+0000","last_peered":"2024-03-11T21:43:23.932881+0000","last_clean":"2024-03-11T21:43:23.932881+0000","last_became_active":"2024-03-11T21:34:41.144945+0000","last_became_peered":"2024-03-11T21:34:41.144945+0000","last_unstale":"2024-03-11T21:43:23.932881+0000","last_undegraded":"2024-03-11T21:43:23.932881+0000","last_fullsized":"2024-03-11T21:43:23.932881+0000","mapping_epoch":226,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":227,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:36:00.843718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:37.512018+0000","last_clean_scrub_stamp":"2024-03-11T21:36:00.843718+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,4],"acting":[0,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.19","version":"260'27","reported_seq":718,"reported_epoch":410,"state":"active+clean","last_fresh":"2024-03-11T21:43:23.943230+0000","last_change":"2024-03-11T21:36:35.089901+0000","last_active":"2024-03-11T21:43:23.943230+0000","last_peered":"2024-03-11T21:43:23.943230+0000","last_clean":"2024-03-11T21:43:23.943230+0000","last_became_active":"2024-03-11T21:36:35.067652+0000","last_became_peered":"2024-03-11T21:36:35.067652+0000","last_unstale":"2024-03-11T21:43:23.943230+0000","last_undegraded":"2024-03-11T21:43:23.943230+0000","last_fullsized":"2024-03-11T21:43:23.943230+0000","mapping_epoch":277,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":278,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'15","last_scrub_stamp":"2024-03-11T21:35:22.099374+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:39.525554+0000","last_clean_scrub_stamp":"2024-03-11T21:35:22.099374+0000","objects_scrubbed":0,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":42,"num_read_kb":33,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.16","version":"412'7241","reported_seq":13927,"reported_epoch":412,"state":"active+clean","last_fresh":"2024-03-11T21:43:27.457022+0000","last_change":"2024-03-11T21:43:20.681951+0000","last_active":"2024-03-11T21:43:27.457022+0000","last_peered":"2024-03-11T21:43:27.457022+0000","last_clean":"2024-03-11T21:43:27.457022+0000","last_became_active":"2024-03-11T21:42:25.516791+0000","last_became_peered":"2024-03-11T21:42:25.516791+0000","last_unstale":"2024-03-11T21:43:27.457022+0000","last_undegraded":"2024-03-11T21:43:27.457022+0000","last_fullsized":"2024-03-11T21:43:27.457022+0000","mapping_epoch":393,"log_start":"0'0","ondisk_log_start":"0'0","created":372,"last_epoch_clean":394,"parent":"0.0","parent_split_bits":5,"last_scrub":"407'7053","last_scrub_stamp":"2024-03-11T21:43:20.681823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:40:38.219602+0000","last_clean_scrub_stamp":"2024-03-11T21:43:20.681823+0000","objects_scrubbed":1849,"log_size":7241,"log_dups_size":0,"ondisk_log_size":7241,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:44:29.254313+0000","scrub_duration":218,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":108855296,"num_objects":1661,"num_object_clones":0,"num_object_copies":4983,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1661,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":188,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,1],"acting":[6,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.1a","version":"0'0","reported_seq":736,"reported_epoch":412,"state":"active+clean","last_fresh":"2024-03-11T21:43:26.943989+0000","last_change":"2024-03-11T21:41:02.614569+0000","last_active":"2024-03-11T21:43:26.943989+0000","last_peered":"2024-03-11T21:43:26.943989+0000","last_clean":"2024-03-11T21:43:26.943989+0000","last_became_active":"2024-03-11T21:25:44.590923+0000","last_became_peered":"2024-03-11T21:25:44.590923+0000","last_unstale":"2024-03-11T21:43:26.943989+0000","last_undegraded":"2024-03-11T21:43:26.943989+0000","last_fullsized":"2024-03-11T21:43:26.943989+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:41:02.614438+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:43.559452+0000","last_clean_scrub_stamp":"2024-03-11T21:41:02.614438+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":38,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,1],"acting":[4,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":752,"reported_epoch":412,"state":"active+clean","last_fresh":"2024-03-11T21:43:26.939767+0000","last_change":"2024-03-11T21:36:15.576138+0000","last_active":"2024-03-11T21:43:26.939767+0000","last_peered":"2024-03-11T21:43:26.939767+0000","last_clean":"2024-03-11T21:43:26.939767+0000","last_became_active":"2024-03-11T21:34:41.154216+0000","last_became_peered":"2024-03-11T21:34:41.154216+0000","last_unstale":"2024-03-11T21:43:26.939767+0000","last_undegraded":"2024-03-11T21:43:26.939767+0000","last_fullsized":"2024-03-11T21:43:26.939767+0000","mapping_epoch":226,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":227,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:36:15.575887+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:41.544550+0000","last_clean_scrub_stamp":"2024-03-11T21:36:15.575887+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,7],"acting":[1,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1e","version":"0'0","reported_seq":693,"reported_epoch":410,"state":"active+clean","last_fresh":"2024-03-11T21:43:23.940512+0000","last_change":"2024-03-11T21:41:04.598202+0000","last_active":"2024-03-11T21:43:23.940512+0000","last_peered":"2024-03-11T21:43:23.940512+0000","last_clean":"2024-03-11T21:43:23.940512+0000","last_became_active":"2024-03-11T21:36:35.061786+0000","last_became_peered":"2024-03-11T21:36:35.061786+0000","last_unstale":"2024-03-11T21:43:23.940512+0000","last_undegraded":"2024-03-11T21:43:23.940512+0000","last_fullsized":"2024-03-11T21:43:23.940512+0000","mapping_epoch":277,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":278,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:41:04.597983+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:37.512018+0000","last_clean_scrub_stamp":"2024-03-11T21:41:04.597983+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.18","version":"260'15","reported_seq":785,"reported_epoch":412,"state":"active+clean","last_fresh":"2024-03-11T21:43:26.943556+0000","last_change":"2024-03-11T21:41:17.537672+0000","last_active":"2024-03-11T21:43:26.943556+0000","last_peered":"2024-03-11T21:43:26.943556+0000","last_clean":"2024-03-11T21:43:26.943556+0000","last_became_active":"2024-03-11T21:25:40.552216+0000","last_became_peered":"2024-03-11T21:25:40.552216+0000","last_unstale":"2024-03-11T21:43:26.943556+0000","last_undegraded":"2024-03-11T21:43:26.943556+0000","last_fullsized":"2024-03-11T21:43:26.943556+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"260'15","last_scrub_stamp":"2024-03-11T21:41:17.537536+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:39.525554+0000","last_clean_scrub_stamp":"2024-03-11T21:41:17.537536+0000","objects_scrubbed":4,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"10.15","version":"410'7410","reported_seq":14527,"reported_epoch":410,"state":"active+clean","last_fresh":"2024-03-11T21:43:24.495176+0000","last_change":"2024-03-11T21:43:16.486965+0000","last_active":"2024-03-11T21:43:24.495176+0000","last_peered":"2024-03-11T21:43:24.495176+0000","last_clean":"2024-03-11T21:43:24.495176+0000","last_became_active":"2024-03-11T21:42:21.511969+0000","last_became_peered":"2024-03-11T21:42:21.511969+0000","last_unstale":"2024-03-11T21:43:24.495176+0000","last_undegraded":"2024-03-11T21:43:24.495176+0000","last_fullsized":"2024-03-11T21:43:24.495176+0000","mapping_epoch":391,"log_start":"364'988","ondisk_log_start":"364'988","created":372,"last_epoch_clean":404,"parent":"0.0","parent_split_bits":5,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:40:38.219602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:40:38.219602+0000","last_clean_scrub_stamp":"2024-03-11T21:40:38.219602+0000","objects_scrubbed":0,"log_size":6422,"log_dups_size":0,"ondisk_log_size":6422,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:43:27.708690+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110231552,"num_objects":1682,"num_object_clones":0,"num_object_copies":5046,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1682,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5182,"num_write_kb":219648,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2327,"num_bytes_recovered":152502272,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,1],"acting":[3,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":752,"reported_epoch":412,"state":"active+clean","last_fresh":"2024-03-11T21:43:26.940841+0000","last_change":"2024-03-11T21:41:13.893454+0000","last_active":"2024-03-11T21:43:26.940841+0000","last_peered":"2024-03-11T21:43:26.940841+0000","last_clean":"2024-03-11T21:43:26.940841+0000","last_became_active":"2024-03-11T21:36:35.068453+0000","last_became_peered":"2024-03-11T21:36:35.068453+0000","last_unstale":"2024-03-11T21:43:26.940841+0000","last_undegraded":"2024-03-11T21:43:26.940841+0000","last_fullsized":"2024-03-11T21:43:26.940841+0000","mapping_epoch":277,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":278,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:41:13.893233+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:32:12.654573+0000","last_clean_scrub_stamp":"2024-03-11T21:41:13.893233+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":78,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3],"acting":[5,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":400,"reported_epoch":412,"state":"active+clean","last_fresh":"2024-03-11T21:43:26.945515+0000","last_change":"2024-03-11T21:41:01.164718+0000","last_active":"2024-03-11T21:43:26.945515+0000","last_peered":"2024-03-11T21:43:26.945515+0000","last_clean":"2024-03-11T21:43:26.945515+0000","last_became_active":"2024-03-11T21:34:41.145299+0000","last_became_peered":"2024-03-11T21:34:41.145299+0000","last_unstale":"2024-03-11T21:43:26.945515+0000","last_undegraded":"2024-03-11T21:43:26.945515+0000","last_fullsized":"2024-03-11T21:43:26.945515+0000","mapping_epoch":226,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":227,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:41:01.164619+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:41.544550+0000","last_clean_scrub_stamp":"2024-03-11T21:41:01.164619+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.1d","version":"0'0","reported_seq":763,"reported_epoch":412,"state":"active+clean","last_fresh":"2024-03-11T21:43:26.946590+0000","last_change":"2024-03-11T21:41:02.217938+0000","last_active":"2024-03-11T21:43:26.946590+0000","last_peered":"2024-03-11T21:43:26.946590+0000","last_clean":"2024-03-11T21:43:26.946590+0000","last_became_active":"2024-03-11T21:25:38.542628+0000","last_became_peered":"2024-03-11T21:25:38.542628+0000","last_unstale":"2024-03-11T21:43:26.946590+0000","last_undegraded":"2024-03-11T21:43:26.946590+0000","last_fullsized":"2024-03-11T21:43:26.946590+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:41:02.217799+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:37.512018+0000","last_clean_scrub_stamp":"2024-03-11T21:41:02.217799+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,0],"acting":[7,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.1b","version":"260'7","reported_seq":771,"reported_epoch":412,"state":"active+clean","last_fresh":"2024-03-11T21:43:26.944806+0000","last_change":"2024-03-11T21:40:29.597326+0000","last_active":"2024-03-11T21:43:26.944806+0000","last_peered":"2024-03-11T21:43:26.944806+0000","last_clean":"2024-03-11T21:43:26.944806+0000","last_became_active":"2024-03-11T21:36:35.057678+0000","last_became_peered":"2024-03-11T21:36:35.057678+0000","last_unstale":"2024-03-11T21:43:26.944806+0000","last_undegraded":"2024-03-11T21:43:26.944806+0000","last_fullsized":"2024-03-11T21:43:26.944806+0000","mapping_epoch":277,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":278,"parent":"0.0","parent_split_bits":0,"last_scrub":"260'7","last_scrub_stamp":"2024-03-11T21:40:29.597219+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:39.525554+0000","last_clean_scrub_stamp":"2024-03-11T21:40:29.597219+0000","objects_scrubbed":2,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":14,"num_read_kb":10,"num_write":8,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,1],"acting":[4,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"10.14","version":"412'7250","reported_seq":14026,"reported_epoch":412,"state":"active+clean+scrubbing","last_fresh":"2024-03-11T21:43:28.455525+0000","last_change":"2024-03-11T21:43:28.172486+0000","last_active":"2024-03-11T21:43:28.455525+0000","last_peered":"2024-03-11T21:43:28.455525+0000","last_clean":"2024-03-11T21:43:28.455525+0000","last_became_active":"2024-03-11T21:42:33.464147+0000","last_became_peered":"2024-03-11T21:42:33.464147+0000","last_unstale":"2024-03-11T21:43:28.455525+0000","last_undegraded":"2024-03-11T21:43:28.455525+0000","last_fullsized":"2024-03-11T21:43:28.455525+0000","mapping_epoch":397,"log_start":"364'532","ondisk_log_start":"364'532","created":372,"last_epoch_clean":398,"parent":"0.0","parent_split_bits":5,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:40:38.219602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:40:38.219602+0000","last_clean_scrub_stamp":"2024-03-11T21:40:38.219602+0000","objects_scrubbed":1499,"log_size":6718,"log_dups_size":0,"ondisk_log_size":6718,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"scrubbing for 0s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":106332160,"num_objects":1623,"num_object_clones":0,"num_object_copies":4869,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1623,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5149,"num_write_kb":216672,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2205,"num_bytes_recovered":144506880,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,4],"acting":[1,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":740,"reported_epoch":410,"state":"active+clean","last_fresh":"2024-03-11T21:43:23.933307+0000","last_change":"2024-03-11T21:43:22.157433+0000","last_active":"2024-03-11T21:43:23.933307+0000","last_peered":"2024-03-11T21:43:23.933307+0000","last_clean":"2024-03-11T21:43:23.933307+0000","last_became_active":"2024-03-11T21:25:44.589786+0000","last_became_peered":"2024-03-11T21:25:44.589786+0000","last_unstale":"2024-03-11T21:43:23.933307+0000","last_undegraded":"2024-03-11T21:43:23.933307+0000","last_fullsized":"2024-03-11T21:43:23.933307+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:43:22.156831+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:43.559452+0000","last_clean_scrub_stamp":"2024-03-11T21:43:22.156831+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:44:25.859937+0000","scrub_duration":93,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,7],"acting":[0,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":759,"reported_epoch":412,"state":"active+clean","last_fresh":"2024-03-11T21:43:26.938569+0000","last_change":"2024-03-11T21:40:27.604550+0000","last_active":"2024-03-11T21:43:26.938569+0000","last_peered":"2024-03-11T21:43:26.938569+0000","last_clean":"2024-03-11T21:43:26.938569+0000","last_became_active":"2024-03-11T21:34:41.147185+0000","last_became_peered":"2024-03-11T21:34:41.147185+0000","last_unstale":"2024-03-11T21:43:26.938569+0000","last_undegraded":"2024-03-11T21:43:26.938569+0000","last_fullsized":"2024-03-11T21:43:26.938569+0000","mapping_epoch":226,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":227,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:40:27.604453+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:32:21.535034+0000","last_clean_scrub_stamp":"2024-03-11T21:40:27.604453+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,7],"acting":[5,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1c","version":"0'0","reported_seq":763,"reported_epoch":412,"state":"active+clean","last_fresh":"2024-03-11T21:43:26.946164+0000","last_change":"2024-03-11T21:41:14.385957+0000","last_active":"2024-03-11T21:43:26.946164+0000","last_peered":"2024-03-11T21:43:26.946164+0000","last_clean":"2024-03-11T21:43:26.946164+0000","last_became_active":"2024-03-11T21:25:38.541364+0000","last_became_peered":"2024-03-11T21:25:38.541364+0000","last_unstale":"2024-03-11T21:43:26.946164+0000","last_undegraded":"2024-03-11T21:43:26.946164+0000","last_fullsized":"2024-03-11T21:43:26.946164+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:41:14.385682+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:28:32.566437+0000","last_clean_scrub_stamp":"2024-03-11T21:41:14.385682+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":27,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.1a","version":"350'19","reported_seq":807,"reported_epoch":412,"state":"active+clean","last_fresh":"2024-03-11T21:43:26.944081+0000","last_change":"2024-03-11T21:43:25.898959+0000","last_active":"2024-03-11T21:43:26.944081+0000","last_peered":"2024-03-11T21:43:26.944081+0000","last_clean":"2024-03-11T21:43:26.944081+0000","last_became_active":"2024-03-11T21:36:35.057029+0000","last_became_peered":"2024-03-11T21:36:35.057029+0000","last_unstale":"2024-03-11T21:43:26.944081+0000","last_undegraded":"2024-03-11T21:43:26.944081+0000","last_fullsized":"2024-03-11T21:43:26.944081+0000","mapping_epoch":277,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":278,"parent":"0.0","parent_split_bits":0,"last_scrub":"350'19","last_scrub_stamp":"2024-03-11T21:43:25.898708+0000","last_deep_scrub":"350'19","last_deep_scrub_stamp":"2024-03-11T21:43:25.898708+0000","last_clean_scrub_stamp":"2024-03-11T21:43:25.898708+0000","objects_scrubbed":4,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:44:26.480834+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0],"acting":[4,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"10.13","version":"412'7292","reported_seq":14020,"reported_epoch":412,"state":"active+clean","last_fresh":"2024-03-11T21:43:28.422993+0000","last_change":"2024-03-11T21:43:27.572518+0000","last_active":"2024-03-11T21:43:28.422993+0000","last_peered":"2024-03-11T21:43:28.422993+0000","last_clean":"2024-03-11T21:43:28.422993+0000","last_became_active":"2024-03-11T21:43:20.870191+0000","last_became_peered":"2024-03-11T21:43:20.870191+0000","last_unstale":"2024-03-11T21:43:28.422993+0000","last_undegraded":"2024-03-11T21:43:28.422993+0000","last_fullsized":"2024-03-11T21:43:28.422993+0000","mapping_epoch":407,"log_start":"364'438","ondisk_log_start":"364'438","created":372,"last_epoch_clean":408,"parent":"0.0","parent_split_bits":5,"last_scrub":"412'7266","last_scrub_stamp":"2024-03-11T21:43:27.572401+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:40:38.219602+0000","last_clean_scrub_stamp":"2024-03-11T21:43:27.572401+0000","objects_scrubbed":1700,"log_size":6854,"log_dups_size":0,"ondisk_log_size":6854,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:44:30.154382+0000","scrub_duration":412,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":109379584,"num_objects":1669,"num_object_clones":0,"num_object_copies":5007,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1669,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":31,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,2],"acting":[1,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1f","version":"0'0","reported_seq":616,"reported_epoch":410,"state":"active+clean","last_fresh":"2024-03-11T21:43:23.943107+0000","last_change":"2024-03-11T21:41:08.652958+0000","last_active":"2024-03-11T21:43:23.943107+0000","last_peered":"2024-03-11T21:43:23.943107+0000","last_clean":"2024-03-11T21:43:23.943107+0000","last_became_active":"2024-03-11T21:36:35.071755+0000","last_became_peered":"2024-03-11T21:36:35.071755+0000","last_unstale":"2024-03-11T21:43:23.943107+0000","last_undegraded":"2024-03-11T21:43:23.943107+0000","last_fullsized":"2024-03-11T21:43:23.943107+0000","mapping_epoch":277,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":278,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:41:08.652836+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:43.559452+0000","last_clean_scrub_stamp":"2024-03-11T21:41:08.652836+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":137,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1c","version":"0'0","reported_seq":749,"reported_epoch":412,"state":"active+clean","last_fresh":"2024-03-11T21:43:26.944016+0000","last_change":"2024-03-11T21:40:07.678642+0000","last_active":"2024-03-11T21:43:26.944016+0000","last_peered":"2024-03-11T21:43:26.944016+0000","last_clean":"2024-03-11T21:43:26.944016+0000","last_became_active":"2024-03-11T21:36:35.067127+0000","last_became_peered":"2024-03-11T21:36:35.067127+0000","last_unstale":"2024-03-11T21:43:26.944016+0000","last_undegraded":"2024-03-11T21:43:26.944016+0000","last_fullsized":"2024-03-11T21:43:26.944016+0000","mapping_epoch":277,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":278,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:40:07.678561+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:40:07.678561+0000","last_clean_scrub_stamp":"2024-03-11T21:40:07.678561+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1b","version":"0'0","reported_seq":664,"reported_epoch":410,"state":"active+clean","last_fresh":"2024-03-11T21:43:23.939955+0000","last_change":"2024-03-11T21:41:23.590257+0000","last_active":"2024-03-11T21:43:23.939955+0000","last_peered":"2024-03-11T21:43:23.939955+0000","last_clean":"2024-03-11T21:43:23.939955+0000","last_became_active":"2024-03-11T21:36:35.073959+0000","last_became_peered":"2024-03-11T21:36:35.073959+0000","last_unstale":"2024-03-11T21:43:23.939955+0000","last_undegraded":"2024-03-11T21:43:23.939955+0000","last_fullsized":"2024-03-11T21:43:23.939955+0000","mapping_epoch":277,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":278,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:41:23.589546+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:34:27.998724+0000","last_clean_scrub_stamp":"2024-03-11T21:41:23.589546+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1d","version":"260'18","reported_seq":700,"reported_epoch":410,"state":"active+clean","last_fresh":"2024-03-11T21:43:23.944974+0000","last_change":"2024-03-11T21:36:35.080207+0000","last_active":"2024-03-11T21:43:23.944974+0000","last_peered":"2024-03-11T21:43:23.944974+0000","last_clean":"2024-03-11T21:43:23.944974+0000","last_became_active":"2024-03-11T21:36:35.055880+0000","last_became_peered":"2024-03-11T21:36:35.055880+0000","last_unstale":"2024-03-11T21:43:23.944974+0000","last_undegraded":"2024-03-11T21:43:23.944974+0000","last_fullsized":"2024-03-11T21:43:23.944974+0000","mapping_epoch":277,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":278,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'12","last_scrub_stamp":"2024-03-11T21:35:07.971581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:39.525554+0000","last_clean_scrub_stamp":"2024-03-11T21:35:07.971581+0000","objects_scrubbed":0,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":33,"num_read_kb":24,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.12","version":"412'7482","reported_seq":16885,"reported_epoch":412,"state":"active+clean","last_fresh":"2024-03-11T21:43:27.322355+0000","last_change":"2024-03-11T21:43:23.063128+0000","last_active":"2024-03-11T21:43:27.322355+0000","last_peered":"2024-03-11T21:43:27.322355+0000","last_clean":"2024-03-11T21:43:27.322355+0000","last_became_active":"2024-03-11T21:43:21.868307+0000","last_became_peered":"2024-03-11T21:43:21.868307+0000","last_unstale":"2024-03-11T21:43:27.322355+0000","last_undegraded":"2024-03-11T21:43:27.322355+0000","last_fullsized":"2024-03-11T21:43:27.322355+0000","mapping_epoch":408,"log_start":"364'1214","ondisk_log_start":"364'1214","created":372,"last_epoch_clean":409,"parent":"0.0","parent_split_bits":5,"last_scrub":"409'7349","last_scrub_stamp":"2024-03-11T21:43:23.060318+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:40:38.219602+0000","last_clean_scrub_stamp":"2024-03-11T21:43:23.060318+0000","objects_scrubbed":1883,"log_size":6268,"log_dups_size":0,"ondisk_log_size":6268,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:44:32.875071+0000","scrub_duration":649,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":114032640,"num_objects":1740,"num_object_clones":0,"num_object_copies":5220,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1740,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":143,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,5],"acting":[6,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.1e","version":"0'0","reported_seq":737,"reported_epoch":412,"state":"active+clean","last_fresh":"2024-03-11T21:43:26.944996+0000","last_change":"2024-03-11T21:41:10.551185+0000","last_active":"2024-03-11T21:43:26.944996+0000","last_peered":"2024-03-11T21:43:26.944996+0000","last_clean":"2024-03-11T21:43:26.944996+0000","last_became_active":"2024-03-11T21:25:45.300355+0000","last_became_peered":"2024-03-11T21:25:45.300355+0000","last_unstale":"2024-03-11T21:43:26.944996+0000","last_undegraded":"2024-03-11T21:43:26.944996+0000","last_fullsized":"2024-03-11T21:43:26.944996+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:41:10.550887+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:43.559452+0000","last_clean_scrub_stamp":"2024-03-11T21:41:10.550887+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5],"acting":[4,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1d","version":"0'0","reported_seq":740,"reported_epoch":412,"state":"active+clean","last_fresh":"2024-03-11T21:43:26.941777+0000","last_change":"2024-03-11T21:36:07.624196+0000","last_active":"2024-03-11T21:43:26.941777+0000","last_peered":"2024-03-11T21:43:26.941777+0000","last_clean":"2024-03-11T21:43:26.941777+0000","last_became_active":"2024-03-11T21:25:42.579407+0000","last_became_peered":"2024-03-11T21:25:42.579407+0000","last_unstale":"2024-03-11T21:43:26.941777+0000","last_undegraded":"2024-03-11T21:43:26.941777+0000","last_fullsized":"2024-03-11T21:43:26.941777+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:36:07.624075+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:30:18.678531+0000","last_clean_scrub_stamp":"2024-03-11T21:36:07.624075+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1a","version":"0'0","reported_seq":784,"reported_epoch":412,"state":"active+clean","last_fresh":"2024-03-11T21:43:26.955391+0000","last_change":"2024-03-11T21:41:31.856350+0000","last_active":"2024-03-11T21:43:26.955391+0000","last_peered":"2024-03-11T21:43:26.955391+0000","last_clean":"2024-03-11T21:43:26.955391+0000","last_became_active":"2024-03-11T21:34:41.145014+0000","last_became_peered":"2024-03-11T21:34:41.145014+0000","last_unstale":"2024-03-11T21:43:26.955391+0000","last_undegraded":"2024-03-11T21:43:26.955391+0000","last_fullsized":"2024-03-11T21:43:26.955391+0000","mapping_epoch":226,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":227,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:41:31.856243+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:37.512018+0000","last_clean_scrub_stamp":"2024-03-11T21:41:31.856243+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.1c","version":"260'27","reported_seq":809,"reported_epoch":412,"state":"active+clean","last_fresh":"2024-03-11T21:43:26.965810+0000","last_change":"2024-03-11T21:40:15.885360+0000","last_active":"2024-03-11T21:43:26.965810+0000","last_peered":"2024-03-11T21:43:26.965810+0000","last_clean":"2024-03-11T21:43:26.965810+0000","last_became_active":"2024-03-11T21:36:35.074579+0000","last_became_peered":"2024-03-11T21:36:35.074579+0000","last_unstale":"2024-03-11T21:43:26.965810+0000","last_undegraded":"2024-03-11T21:43:26.965810+0000","last_fullsized":"2024-03-11T21:43:26.965810+0000","mapping_epoch":277,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":278,"parent":"0.0","parent_split_bits":0,"last_scrub":"260'27","last_scrub_stamp":"2024-03-11T21:40:15.885265+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:39.525554+0000","last_clean_scrub_stamp":"2024-03-11T21:40:15.885265+0000","objects_scrubbed":7,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":44,"num_read_kb":35,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"10.11","version":"412'7806","reported_seq":12225,"reported_epoch":412,"state":"active+clean+scrubbing+deep","last_fresh":"2024-03-11T21:43:27.036199+0000","last_change":"2024-03-11T21:43:24.993365+0000","last_active":"2024-03-11T21:43:27.036199+0000","last_peered":"2024-03-11T21:43:27.036199+0000","last_clean":"2024-03-11T21:43:27.036199+0000","last_became_active":"2024-03-11T21:42:09.430469+0000","last_became_peered":"2024-03-11T21:42:09.430469+0000","last_unstale":"2024-03-11T21:43:27.036199+0000","last_undegraded":"2024-03-11T21:43:27.036199+0000","last_fullsized":"2024-03-11T21:43:27.036199+0000","mapping_epoch":389,"log_start":"364'933","ondisk_log_start":"364'933","created":372,"last_epoch_clean":390,"parent":"0.0","parent_split_bits":5,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:40:38.219602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:40:38.219602+0000","last_clean_scrub_stamp":"2024-03-11T21:40:38.219602+0000","objects_scrubbed":486,"log_size":6873,"log_dups_size":0,"ondisk_log_size":6873,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 2s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":115900416,"num_objects":1769,"num_object_clones":0,"num_object_copies":5307,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1769,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5133,"num_write_kb":220832,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2170,"num_bytes_recovered":142213120,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,7],"acting":[5,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.1d","version":"0'0","reported_seq":736,"reported_epoch":412,"state":"active+clean","last_fresh":"2024-03-11T21:43:26.941228+0000","last_change":"2024-03-11T21:41:16.533828+0000","last_active":"2024-03-11T21:43:26.941228+0000","last_peered":"2024-03-11T21:43:26.941228+0000","last_clean":"2024-03-11T21:43:26.941228+0000","last_became_active":"2024-03-11T21:25:44.588845+0000","last_became_peered":"2024-03-11T21:25:44.588845+0000","last_unstale":"2024-03-11T21:43:26.941228+0000","last_undegraded":"2024-03-11T21:43:26.941228+0000","last_fullsized":"2024-03-11T21:43:26.941228+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:41:16.533692+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:43.559452+0000","last_clean_scrub_stamp":"2024-03-11T21:41:16.533692+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":207,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.1e","version":"0'0","reported_seq":745,"reported_epoch":410,"state":"active+clean","last_fresh":"2024-03-11T21:43:23.933660+0000","last_change":"2024-03-11T21:40:28.163223+0000","last_active":"2024-03-11T21:43:23.933660+0000","last_peered":"2024-03-11T21:43:23.933660+0000","last_clean":"2024-03-11T21:43:23.933660+0000","last_became_active":"2024-03-11T21:25:42.582959+0000","last_became_peered":"2024-03-11T21:25:42.582959+0000","last_unstale":"2024-03-11T21:43:23.933660+0000","last_undegraded":"2024-03-11T21:43:23.933660+0000","last_fullsized":"2024-03-11T21:43:23.933660+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:40:28.163142+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:26:58.335431+0000","last_clean_scrub_stamp":"2024-03-11T21:40:28.163142+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.19","version":"67'2","reported_seq":690,"reported_epoch":410,"state":"active+clean","last_fresh":"2024-03-11T21:43:23.942300+0000","last_change":"2024-03-11T21:41:01.612764+0000","last_active":"2024-03-11T21:43:23.942300+0000","last_peered":"2024-03-11T21:43:23.942300+0000","last_clean":"2024-03-11T21:43:23.942300+0000","last_became_active":"2024-03-11T21:36:35.064249+0000","last_became_peered":"2024-03-11T21:36:35.064249+0000","last_unstale":"2024-03-11T21:43:23.942300+0000","last_undegraded":"2024-03-11T21:43:23.942300+0000","last_fullsized":"2024-03-11T21:43:23.942300+0000","mapping_epoch":277,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":278,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'2","last_scrub_stamp":"2024-03-11T21:41:01.612632+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:37.512018+0000","last_clean_scrub_stamp":"2024-03-11T21:41:01.612632+0000","objects_scrubbed":2,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":957,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,0],"acting":[3,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1f","version":"260'24","reported_seq":823,"reported_epoch":412,"state":"active+clean","last_fresh":"2024-03-11T21:43:26.955942+0000","last_change":"2024-03-11T21:40:25.079945+0000","last_active":"2024-03-11T21:43:26.955942+0000","last_peered":"2024-03-11T21:43:26.955942+0000","last_clean":"2024-03-11T21:43:26.955942+0000","last_became_active":"2024-03-11T21:32:22.798242+0000","last_became_peered":"2024-03-11T21:32:22.798242+0000","last_unstale":"2024-03-11T21:43:26.955942+0000","last_undegraded":"2024-03-11T21:43:26.955942+0000","last_fullsized":"2024-03-11T21:43:26.955942+0000","mapping_epoch":166,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":167,"parent":"0.0","parent_split_bits":0,"last_scrub":"260'24","last_scrub_stamp":"2024-03-11T21:40:25.079808+0000","last_deep_scrub":"124'16","last_deep_scrub_stamp":"2024-03-11T21:35:38.173841+0000","last_clean_scrub_stamp":"2024-03-11T21:40:25.079808+0000","objects_scrubbed":5,"log_size":24,"log_dups_size":0,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":32,"num_read_kb":25,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,0],"acting":[6,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"10.10","version":"412'7372","reported_seq":12024,"reported_epoch":412,"state":"active+clean+scrubbing+deep","last_fresh":"2024-03-11T21:43:27.047985+0000","last_change":"2024-03-11T21:43:26.936139+0000","last_active":"2024-03-11T21:43:27.047985+0000","last_peered":"2024-03-11T21:43:27.047985+0000","last_clean":"2024-03-11T21:43:27.047985+0000","last_became_active":"2024-03-11T21:41:47.077087+0000","last_became_peered":"2024-03-11T21:41:47.077087+0000","last_unstale":"2024-03-11T21:43:27.047985+0000","last_undegraded":"2024-03-11T21:43:27.047985+0000","last_fullsized":"2024-03-11T21:43:27.047985+0000","mapping_epoch":380,"log_start":"365'1496","ondisk_log_start":"365'1496","created":372,"last_epoch_clean":381,"parent":"0.0","parent_split_bits":5,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:40:38.219602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:40:38.219602+0000","last_clean_scrub_stamp":"2024-03-11T21:40:38.219602+0000","objects_scrubbed":24,"log_size":5876,"log_dups_size":0,"ondisk_log_size":5876,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 0s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":107806720,"num_objects":1645,"num_object_clones":0,"num_object_copies":4935,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1645,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5185,"num_write_kb":218560,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2254,"num_bytes_recovered":147718144,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,6],"acting":[4,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1c","version":"74'1","reported_seq":404,"reported_epoch":412,"state":"active+clean","last_fresh":"2024-03-11T21:43:26.946093+0000","last_change":"2024-03-11T21:43:20.543062+0000","last_active":"2024-03-11T21:43:26.946093+0000","last_peered":"2024-03-11T21:43:26.946093+0000","last_clean":"2024-03-11T21:43:26.946093+0000","last_became_active":"2024-03-11T21:34:41.144839+0000","last_became_peered":"2024-03-11T21:34:41.144839+0000","last_unstale":"2024-03-11T21:43:26.946093+0000","last_undegraded":"2024-03-11T21:43:26.946093+0000","last_fullsized":"2024-03-11T21:43:26.946093+0000","mapping_epoch":226,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":227,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'1","last_scrub_stamp":"2024-03-11T21:43:20.542918+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:43.559452+0000","last_clean_scrub_stamp":"2024-03-11T21:43:20.542918+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:44:38.671474+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":370,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":370,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.1f","version":"229'10","reported_seq":1018,"reported_epoch":412,"state":"active+clean","last_fresh":"2024-03-11T21:43:26.955852+0000","last_change":"2024-03-11T21:41:09.868735+0000","last_active":"2024-03-11T21:43:26.955852+0000","last_peered":"2024-03-11T21:43:26.955852+0000","last_clean":"2024-03-11T21:43:26.955852+0000","last_became_active":"2024-03-11T21:34:41.145862+0000","last_became_peered":"2024-03-11T21:34:41.145862+0000","last_unstale":"2024-03-11T21:43:26.955852+0000","last_undegraded":"2024-03-11T21:43:26.955852+0000","last_fullsized":"2024-03-11T21:43:26.955852+0000","mapping_epoch":226,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":227,"parent":"0.0","parent_split_bits":0,"last_scrub":"229'10","last_scrub_stamp":"2024-03-11T21:41:09.868606+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:41.544550+0000","last_clean_scrub_stamp":"2024-03-11T21:41:09.868606+0000","objects_scrubbed":1,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.18","version":"67'1","reported_seq":791,"reported_epoch":412,"state":"active+clean","last_fresh":"2024-03-11T21:43:26.938266+0000","last_change":"2024-03-11T21:36:10.458150+0000","last_active":"2024-03-11T21:43:26.938266+0000","last_peered":"2024-03-11T21:43:26.938266+0000","last_clean":"2024-03-11T21:43:26.938266+0000","last_became_active":"2024-03-11T21:35:23.310241+0000","last_became_peered":"2024-03-11T21:35:23.310241+0000","last_unstale":"2024-03-11T21:43:26.938266+0000","last_undegraded":"2024-03-11T21:43:26.938266+0000","last_fullsized":"2024-03-11T21:43:26.938266+0000","mapping_epoch":248,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":249,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'1","last_scrub_stamp":"2024-03-11T21:36:10.458028+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:37.512018+0000","last_clean_scrub_stamp":"2024-03-11T21:36:10.458028+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":436,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":7,"num_read_kb":7,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":1308,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,2],"acting":[5,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1e","version":"260'20","reported_seq":837,"reported_epoch":410,"state":"active+clean","last_fresh":"2024-03-11T21:43:23.933566+0000","last_change":"2024-03-11T21:41:10.256209+0000","last_active":"2024-03-11T21:43:23.933566+0000","last_peered":"2024-03-11T21:43:23.933566+0000","last_clean":"2024-03-11T21:43:23.933566+0000","last_became_active":"2024-03-11T21:36:35.057843+0000","last_became_peered":"2024-03-11T21:36:35.057843+0000","last_unstale":"2024-03-11T21:43:23.933566+0000","last_undegraded":"2024-03-11T21:43:23.933566+0000","last_fullsized":"2024-03-11T21:43:23.933566+0000","mapping_epoch":277,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":278,"parent":"0.0","parent_split_bits":0,"last_scrub":"260'20","last_scrub_stamp":"2024-03-11T21:41:10.255971+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:39.525554+0000","last_clean_scrub_stamp":"2024-03-11T21:41:10.255971+0000","objects_scrubbed":5,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":25,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.13","version":"0'0","reported_seq":623,"reported_epoch":410,"state":"active+clean","last_fresh":"2024-03-11T21:43:23.940216+0000","last_change":"2024-03-11T21:41:26.639950+0000","last_active":"2024-03-11T21:43:23.940216+0000","last_peered":"2024-03-11T21:43:23.940216+0000","last_clean":"2024-03-11T21:43:23.940216+0000","last_became_active":"2024-03-11T21:36:35.062618+0000","last_became_peered":"2024-03-11T21:36:35.062618+0000","last_unstale":"2024-03-11T21:43:23.940216+0000","last_undegraded":"2024-03-11T21:43:23.940216+0000","last_fullsized":"2024-03-11T21:43:23.940216+0000","mapping_epoch":277,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":278,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:41:26.639290+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:43.559452+0000","last_clean_scrub_stamp":"2024-03-11T21:41:26.639290+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":114,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":408,"reported_epoch":412,"state":"active+clean","last_fresh":"2024-03-11T21:43:26.960018+0000","last_change":"2024-03-11T21:43:26.665857+0000","last_active":"2024-03-11T21:43:26.960018+0000","last_peered":"2024-03-11T21:43:26.960018+0000","last_clean":"2024-03-11T21:43:26.960018+0000","last_became_active":"2024-03-11T21:34:41.142569+0000","last_became_peered":"2024-03-11T21:34:41.142569+0000","last_unstale":"2024-03-11T21:43:26.960018+0000","last_undegraded":"2024-03-11T21:43:26.960018+0000","last_fullsized":"2024-03-11T21:43:26.960018+0000","mapping_epoch":226,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":227,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:43:26.665719+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:43:26.665719+0000","last_clean_scrub_stamp":"2024-03-11T21:43:26.665719+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:44:46.668281+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.17","version":"0'0","reported_seq":772,"reported_epoch":412,"state":"active+clean","last_fresh":"2024-03-11T21:43:26.955625+0000","last_change":"2024-03-11T21:41:26.816265+0000","last_active":"2024-03-11T21:43:26.955625+0000","last_peered":"2024-03-11T21:43:26.955625+0000","last_clean":"2024-03-11T21:43:26.955625+0000","last_became_active":"2024-03-11T21:25:38.540349+0000","last_became_peered":"2024-03-11T21:25:38.540349+0000","last_unstale":"2024-03-11T21:43:26.955625+0000","last_undegraded":"2024-03-11T21:43:26.955625+0000","last_fullsized":"2024-03-11T21:43:26.955625+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:41:26.816135+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:28:25.918101+0000","last_clean_scrub_stamp":"2024-03-11T21:41:26.816135+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,2],"acting":[6,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.11","version":"260'19","reported_seq":741,"reported_epoch":410,"state":"active+clean","last_fresh":"2024-03-11T21:43:23.943702+0000","last_change":"2024-03-11T21:36:35.096720+0000","last_active":"2024-03-11T21:43:23.943702+0000","last_peered":"2024-03-11T21:43:23.943702+0000","last_clean":"2024-03-11T21:43:23.943702+0000","last_became_active":"2024-03-11T21:36:35.073689+0000","last_became_peered":"2024-03-11T21:36:35.073689+0000","last_unstale":"2024-03-11T21:43:23.943702+0000","last_undegraded":"2024-03-11T21:43:23.943702+0000","last_fullsized":"2024-03-11T21:43:23.943702+0000","mapping_epoch":277,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":278,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'11","last_scrub_stamp":"2024-03-11T21:35:16.044397+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:39.525554+0000","last_clean_scrub_stamp":"2024-03-11T21:35:16.044397+0000","objects_scrubbed":5,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":31,"num_read_kb":24,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":400,"reported_epoch":412,"state":"active+clean","last_fresh":"2024-03-11T21:43:26.945368+0000","last_change":"2024-03-11T21:41:16.345639+0000","last_active":"2024-03-11T21:43:26.945368+0000","last_peered":"2024-03-11T21:43:26.945368+0000","last_clean":"2024-03-11T21:43:26.945368+0000","last_became_active":"2024-03-11T21:34:41.137235+0000","last_became_peered":"2024-03-11T21:34:41.137235+0000","last_unstale":"2024-03-11T21:43:26.945368+0000","last_undegraded":"2024-03-11T21:43:26.945368+0000","last_fullsized":"2024-03-11T21:43:26.945368+0000","mapping_epoch":226,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":227,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:41:16.345432+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:43.559452+0000","last_clean_scrub_stamp":"2024-03-11T21:41:16.345432+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,4],"acting":[7,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":776,"reported_epoch":412,"state":"active+clean","last_fresh":"2024-03-11T21:43:26.954739+0000","last_change":"2024-03-11T21:40:03.181212+0000","last_active":"2024-03-11T21:43:26.954739+0000","last_peered":"2024-03-11T21:43:26.954739+0000","last_clean":"2024-03-11T21:43:26.954739+0000","last_became_active":"2024-03-11T21:34:41.156036+0000","last_became_peered":"2024-03-11T21:34:41.156036+0000","last_unstale":"2024-03-11T21:43:26.954739+0000","last_undegraded":"2024-03-11T21:43:26.954739+0000","last_fullsized":"2024-03-11T21:43:26.954739+0000","mapping_epoch":226,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":227,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:40:03.181074+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:38:51.036736+0000","last_clean_scrub_stamp":"2024-03-11T21:40:03.181074+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.16","version":"0'0","reported_seq":763,"reported_epoch":412,"state":"active+clean","last_fresh":"2024-03-11T21:43:26.939020+0000","last_change":"2024-03-11T21:41:25.953645+0000","last_active":"2024-03-11T21:43:26.939020+0000","last_peered":"2024-03-11T21:43:26.939020+0000","last_clean":"2024-03-11T21:43:26.939020+0000","last_became_active":"2024-03-11T21:25:38.539103+0000","last_became_peered":"2024-03-11T21:25:38.539103+0000","last_unstale":"2024-03-11T21:43:26.939020+0000","last_undegraded":"2024-03-11T21:43:26.939020+0000","last_fullsized":"2024-03-11T21:43:26.939020+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:41:25.953529+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:27:11.938909+0000","last_clean_scrub_stamp":"2024-03-11T21:41:25.953529+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,2],"acting":[5,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.10","version":"260'10","reported_seq":631,"reported_epoch":410,"state":"active+clean","last_fresh":"2024-03-11T21:43:23.939753+0000","last_change":"2024-03-11T21:41:17.565434+0000","last_active":"2024-03-11T21:43:23.939753+0000","last_peered":"2024-03-11T21:43:23.939753+0000","last_clean":"2024-03-11T21:43:23.939753+0000","last_became_active":"2024-03-11T21:36:35.067500+0000","last_became_peered":"2024-03-11T21:36:35.067500+0000","last_unstale":"2024-03-11T21:43:23.939753+0000","last_undegraded":"2024-03-11T21:43:23.939753+0000","last_fullsized":"2024-03-11T21:43:23.939753+0000","mapping_epoch":277,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":278,"parent":"0.0","parent_split_bits":0,"last_scrub":"260'10","last_scrub_stamp":"2024-03-11T21:41:17.565310+0000","last_deep_scrub":"94'6","last_deep_scrub_stamp":"2024-03-11T21:28:24.603974+0000","last_clean_scrub_stamp":"2024-03-11T21:41:17.565310+0000","objects_scrubbed":2,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":12,"num_read_kb":10,"num_write":8,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6],"acting":[3,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":668,"reported_epoch":410,"state":"active+clean","last_fresh":"2024-03-11T21:43:23.943522+0000","last_change":"2024-03-11T21:36:35.070468+0000","last_active":"2024-03-11T21:43:23.943522+0000","last_peered":"2024-03-11T21:43:23.943522+0000","last_clean":"2024-03-11T21:43:23.943522+0000","last_became_active":"2024-03-11T21:36:35.070176+0000","last_became_peered":"2024-03-11T21:36:35.070176+0000","last_unstale":"2024-03-11T21:43:23.943522+0000","last_undegraded":"2024-03-11T21:43:23.943522+0000","last_fullsized":"2024-03-11T21:43:23.943522+0000","mapping_epoch":277,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":278,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:35:18.062272+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:43.559452+0000","last_clean_scrub_stamp":"2024-03-11T21:35:18.062272+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":749,"reported_epoch":412,"state":"active+clean","last_fresh":"2024-03-11T21:43:26.940600+0000","last_change":"2024-03-11T21:41:04.276105+0000","last_active":"2024-03-11T21:43:26.940600+0000","last_peered":"2024-03-11T21:43:26.940600+0000","last_clean":"2024-03-11T21:43:26.940600+0000","last_became_active":"2024-03-11T21:36:35.057906+0000","last_became_peered":"2024-03-11T21:36:35.057906+0000","last_unstale":"2024-03-11T21:43:26.940600+0000","last_undegraded":"2024-03-11T21:43:26.940600+0000","last_fullsized":"2024-03-11T21:43:26.940600+0000","mapping_epoch":277,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":278,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:41:04.275820+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:27:21.207303+0000","last_clean_scrub_stamp":"2024-03-11T21:41:04.275820+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.15","version":"0'0","reported_seq":748,"reported_epoch":412,"state":"active+clean","last_fresh":"2024-03-11T21:43:26.941688+0000","last_change":"2024-03-11T21:36:06.579754+0000","last_active":"2024-03-11T21:43:26.941688+0000","last_peered":"2024-03-11T21:43:26.941688+0000","last_clean":"2024-03-11T21:43:26.941688+0000","last_became_active":"2024-03-11T21:25:38.541705+0000","last_became_peered":"2024-03-11T21:25:38.541705+0000","last_unstale":"2024-03-11T21:43:26.941688+0000","last_undegraded":"2024-03-11T21:43:26.941688+0000","last_fullsized":"2024-03-11T21:43:26.941688+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:36:06.579636+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:26:44.860499+0000","last_clean_scrub_stamp":"2024-03-11T21:36:06.579636+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.13","version":"260'24","reported_seq":806,"reported_epoch":412,"state":"active+clean","last_fresh":"2024-03-11T21:43:26.944658+0000","last_change":"2024-03-11T21:40:25.595320+0000","last_active":"2024-03-11T21:43:26.944658+0000","last_peered":"2024-03-11T21:43:26.944658+0000","last_clean":"2024-03-11T21:43:26.944658+0000","last_became_active":"2024-03-11T21:26:11.214879+0000","last_became_peered":"2024-03-11T21:26:11.214879+0000","last_unstale":"2024-03-11T21:43:26.944658+0000","last_undegraded":"2024-03-11T21:43:26.944658+0000","last_fullsized":"2024-03-11T21:43:26.944658+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"260'24","last_scrub_stamp":"2024-03-11T21:40:25.595220+0000","last_deep_scrub":"260'24","last_deep_scrub_stamp":"2024-03-11T21:40:25.595220+0000","last_clean_scrub_stamp":"2024-03-11T21:40:25.595220+0000","objects_scrubbed":5,"log_size":24,"log_dups_size":0,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":32,"num_read_kb":25,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,2],"acting":[4,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.10","version":"0'0","reported_seq":732,"reported_epoch":410,"state":"active+clean","last_fresh":"2024-03-11T21:43:23.932475+0000","last_change":"2024-03-11T21:35:29.712658+0000","last_active":"2024-03-11T21:43:23.932475+0000","last_peered":"2024-03-11T21:43:23.932475+0000","last_clean":"2024-03-11T21:43:23.932475+0000","last_became_active":"2024-03-11T21:25:44.591189+0000","last_became_peered":"2024-03-11T21:25:44.591189+0000","last_unstale":"2024-03-11T21:43:23.932475+0000","last_undegraded":"2024-03-11T21:43:23.932475+0000","last_fullsized":"2024-03-11T21:43:23.932475+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:35:29.712557+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:27:25.417450+0000","last_clean_scrub_stamp":"2024-03-11T21:35:29.712557+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,1],"acting":[0,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":666,"reported_epoch":410,"state":"active+clean","last_fresh":"2024-03-11T21:43:23.940309+0000","last_change":"2024-03-11T21:36:35.062276+0000","last_active":"2024-03-11T21:43:23.940309+0000","last_peered":"2024-03-11T21:43:23.940309+0000","last_clean":"2024-03-11T21:43:23.940309+0000","last_became_active":"2024-03-11T21:36:35.056619+0000","last_became_peered":"2024-03-11T21:36:35.056619+0000","last_unstale":"2024-03-11T21:43:23.940309+0000","last_undegraded":"2024-03-11T21:43:23.940309+0000","last_fullsized":"2024-03-11T21:43:23.940309+0000","mapping_epoch":277,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":278,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:35:13.060487+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:31:15.367898+0000","last_clean_scrub_stamp":"2024-03-11T21:35:13.060487+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.14","version":"0'0","reported_seq":789,"reported_epoch":412,"state":"active+clean","last_fresh":"2024-03-11T21:43:26.955223+0000","last_change":"2024-03-11T21:41:14.822928+0000","last_active":"2024-03-11T21:43:26.955223+0000","last_peered":"2024-03-11T21:43:26.955223+0000","last_clean":"2024-03-11T21:43:26.955223+0000","last_became_active":"2024-03-11T21:36:35.068038+0000","last_became_peered":"2024-03-11T21:36:35.068038+0000","last_unstale":"2024-03-11T21:43:26.955223+0000","last_undegraded":"2024-03-11T21:43:26.955223+0000","last_fullsized":"2024-03-11T21:43:26.955223+0000","mapping_epoch":277,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":278,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:41:14.822805+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:37.512018+0000","last_clean_scrub_stamp":"2024-03-11T21:41:14.822805+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,5],"acting":[6,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.12","version":"260'15","reported_seq":777,"reported_epoch":412,"state":"active+clean","last_fresh":"2024-03-11T21:43:26.940392+0000","last_change":"2024-03-11T21:41:18.319324+0000","last_active":"2024-03-11T21:43:26.940392+0000","last_peered":"2024-03-11T21:43:26.940392+0000","last_clean":"2024-03-11T21:43:26.940392+0000","last_became_active":"2024-03-11T21:25:40.547082+0000","last_became_peered":"2024-03-11T21:25:40.547082+0000","last_unstale":"2024-03-11T21:43:26.940392+0000","last_undegraded":"2024-03-11T21:43:26.940392+0000","last_fullsized":"2024-03-11T21:43:26.940392+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"260'15","last_scrub_stamp":"2024-03-11T21:41:18.319190+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:39.525554+0000","last_clean_scrub_stamp":"2024-03-11T21:41:18.319190+0000","objects_scrubbed":4,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":744,"reported_epoch":412,"state":"active+clean","last_fresh":"2024-03-11T21:43:26.945255+0000","last_change":"2024-03-11T21:41:25.741034+0000","last_active":"2024-03-11T21:43:26.945255+0000","last_peered":"2024-03-11T21:43:26.945255+0000","last_clean":"2024-03-11T21:43:26.945255+0000","last_became_active":"2024-03-11T21:25:44.589586+0000","last_became_peered":"2024-03-11T21:25:44.589586+0000","last_unstale":"2024-03-11T21:43:26.945255+0000","last_undegraded":"2024-03-11T21:43:26.945255+0000","last_fullsized":"2024-03-11T21:43:26.945255+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:41:25.740917+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:27:09.801138+0000","last_clean_scrub_stamp":"2024-03-11T21:41:25.740917+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":158,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,5],"acting":[4,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.14","version":"280'12","reported_seq":910,"reported_epoch":410,"state":"active+clean","last_fresh":"2024-03-11T21:43:23.945218+0000","last_change":"2024-03-11T21:36:35.074550+0000","last_active":"2024-03-11T21:43:23.945218+0000","last_peered":"2024-03-11T21:43:23.945218+0000","last_clean":"2024-03-11T21:43:23.945218+0000","last_became_active":"2024-03-11T21:36:35.073912+0000","last_became_peered":"2024-03-11T21:36:35.073912+0000","last_unstale":"2024-03-11T21:43:23.945218+0000","last_undegraded":"2024-03-11T21:43:23.945218+0000","last_fullsized":"2024-03-11T21:43:23.945218+0000","mapping_epoch":277,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":278,"parent":"0.0","parent_split_bits":0,"last_scrub":"229'10","last_scrub_stamp":"2024-03-11T21:35:26.182107+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:41.544550+0000","last_clean_scrub_stamp":"2024-03-11T21:35:26.182107+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,2],"acting":[3,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.13","version":"0'0","reported_seq":753,"reported_epoch":410,"state":"active+clean","last_fresh":"2024-03-11T21:43:23.936302+0000","last_change":"2024-03-11T21:40:11.027382+0000","last_active":"2024-03-11T21:43:23.936302+0000","last_peered":"2024-03-11T21:43:23.936302+0000","last_clean":"2024-03-11T21:43:23.936302+0000","last_became_active":"2024-03-11T21:36:35.057132+0000","last_became_peered":"2024-03-11T21:36:35.057132+0000","last_unstale":"2024-03-11T21:43:23.936302+0000","last_undegraded":"2024-03-11T21:43:23.936302+0000","last_fullsized":"2024-03-11T21:43:23.936302+0000","mapping_epoch":277,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":278,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:40:11.027268+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:37.512018+0000","last_clean_scrub_stamp":"2024-03-11T21:40:11.027268+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":24,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.15","version":"260'19","reported_seq":830,"reported_epoch":412,"state":"active+clean","last_fresh":"2024-03-11T21:43:26.938854+0000","last_change":"2024-03-11T21:41:18.838796+0000","last_active":"2024-03-11T21:43:26.938854+0000","last_peered":"2024-03-11T21:43:26.938854+0000","last_clean":"2024-03-11T21:43:26.938854+0000","last_became_active":"2024-03-11T21:36:35.057285+0000","last_became_peered":"2024-03-11T21:36:35.057285+0000","last_unstale":"2024-03-11T21:43:26.938854+0000","last_undegraded":"2024-03-11T21:43:26.938854+0000","last_fullsized":"2024-03-11T21:43:26.938854+0000","mapping_epoch":277,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":278,"parent":"0.0","parent_split_bits":0,"last_scrub":"260'19","last_scrub_stamp":"2024-03-11T21:41:18.838669+0000","last_deep_scrub":"94'13","last_deep_scrub_stamp":"2024-03-11T21:29:41.853158+0000","last_clean_scrub_stamp":"2024-03-11T21:41:18.838669+0000","objects_scrubbed":4,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":11,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.16","version":"0'0","reported_seq":761,"reported_epoch":410,"state":"active+clean","last_fresh":"2024-03-11T21:43:23.933063+0000","last_change":"2024-03-11T21:40:30.249810+0000","last_active":"2024-03-11T21:43:23.933063+0000","last_peered":"2024-03-11T21:43:23.933063+0000","last_clean":"2024-03-11T21:43:23.933063+0000","last_became_active":"2024-03-11T21:36:35.057355+0000","last_became_peered":"2024-03-11T21:36:35.057355+0000","last_unstale":"2024-03-11T21:43:23.933063+0000","last_undegraded":"2024-03-11T21:43:23.933063+0000","last_fullsized":"2024-03-11T21:43:23.933063+0000","mapping_epoch":277,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":278,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:40:30.249679+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:40:30.249679+0000","last_clean_scrub_stamp":"2024-03-11T21:40:30.249679+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":40,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.15","version":"98'9","reported_seq":970,"reported_epoch":412,"state":"active+clean","last_fresh":"2024-03-11T21:43:26.938961+0000","last_change":"2024-03-11T21:43:22.974608+0000","last_active":"2024-03-11T21:43:26.938961+0000","last_peered":"2024-03-11T21:43:26.938961+0000","last_clean":"2024-03-11T21:43:26.938961+0000","last_became_active":"2024-03-11T21:25:42.572555+0000","last_became_peered":"2024-03-11T21:25:42.572555+0000","last_unstale":"2024-03-11T21:43:26.938961+0000","last_undegraded":"2024-03-11T21:43:26.938961+0000","last_fullsized":"2024-03-11T21:43:26.938961+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"98'9","last_scrub_stamp":"2024-03-11T21:43:22.974462+0000","last_deep_scrub":"98'9","last_deep_scrub_stamp":"2024-03-11T21:31:20.624923+0000","last_clean_scrub_stamp":"2024-03-11T21:43:22.974462+0000","objects_scrubbed":1,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:44:42.702256+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.12","version":"0'0","reported_seq":776,"reported_epoch":412,"state":"active+clean","last_fresh":"2024-03-11T21:43:26.938432+0000","last_change":"2024-03-11T21:41:21.853550+0000","last_active":"2024-03-11T21:43:26.938432+0000","last_peered":"2024-03-11T21:43:26.938432+0000","last_clean":"2024-03-11T21:43:26.938432+0000","last_became_active":"2024-03-11T21:36:35.058033+0000","last_became_peered":"2024-03-11T21:36:35.058033+0000","last_unstale":"2024-03-11T21:43:26.938432+0000","last_undegraded":"2024-03-11T21:43:26.938432+0000","last_fullsized":"2024-03-11T21:43:26.938432+0000","mapping_epoch":277,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":278,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:41:21.853428+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:37.512018+0000","last_clean_scrub_stamp":"2024-03-11T21:41:21.853428+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.14","version":"260'20","reported_seq":749,"reported_epoch":410,"state":"active+clean","last_fresh":"2024-03-11T21:43:23.944862+0000","last_change":"2024-03-11T21:36:35.095382+0000","last_active":"2024-03-11T21:43:23.944862+0000","last_peered":"2024-03-11T21:43:23.944862+0000","last_clean":"2024-03-11T21:43:23.944862+0000","last_became_active":"2024-03-11T21:36:35.069638+0000","last_became_peered":"2024-03-11T21:36:35.069638+0000","last_unstale":"2024-03-11T21:43:23.944862+0000","last_undegraded":"2024-03-11T21:43:23.944862+0000","last_fullsized":"2024-03-11T21:43:23.944862+0000","mapping_epoch":277,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":278,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'10","last_scrub_stamp":"2024-03-11T21:35:28.189296+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:39.525554+0000","last_clean_scrub_stamp":"2024-03-11T21:35:28.189296+0000","objects_scrubbed":0,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":29,"num_read_kb":24,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.19","version":"412'7459","reported_seq":16637,"reported_epoch":412,"state":"active+clean","last_fresh":"2024-03-11T21:43:28.407479+0000","last_change":"2024-03-11T21:43:26.959581+0000","last_active":"2024-03-11T21:43:28.407479+0000","last_peered":"2024-03-11T21:43:28.407479+0000","last_clean":"2024-03-11T21:43:28.407479+0000","last_became_active":"2024-03-11T21:43:26.945126+0000","last_became_peered":"2024-03-11T21:43:26.945126+0000","last_unstale":"2024-03-11T21:43:28.407479+0000","last_undegraded":"2024-03-11T21:43:28.407479+0000","last_fullsized":"2024-03-11T21:43:28.407479+0000","mapping_epoch":411,"log_start":"364'902","ondisk_log_start":"364'902","created":372,"last_epoch_clean":412,"parent":"0.0","parent_split_bits":5,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:40:38.219602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:40:38.219602+0000","last_clean_scrub_stamp":"2024-03-11T21:40:38.219602+0000","objects_scrubbed":0,"log_size":6557,"log_dups_size":0,"ondisk_log_size":6557,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":107380736,"num_objects":1639,"num_object_clones":0,"num_object_copies":4917,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1639,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5303,"num_write_kb":222112,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2260,"num_bytes_recovered":148111360,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,3],"acting":[1,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.15","version":"0'0","reported_seq":400,"reported_epoch":412,"state":"active+clean","last_fresh":"2024-03-11T21:43:26.946373+0000","last_change":"2024-03-11T21:41:31.306328+0000","last_active":"2024-03-11T21:43:26.946373+0000","last_peered":"2024-03-11T21:43:26.946373+0000","last_clean":"2024-03-11T21:43:26.946373+0000","last_became_active":"2024-03-11T21:34:41.142760+0000","last_became_peered":"2024-03-11T21:34:41.142760+0000","last_unstale":"2024-03-11T21:43:26.946373+0000","last_undegraded":"2024-03-11T21:43:26.946373+0000","last_fullsized":"2024-03-11T21:43:26.946373+0000","mapping_epoch":226,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":227,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:41:31.306234+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:43.559452+0000","last_clean_scrub_stamp":"2024-03-11T21:41:31.306234+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":38,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":748,"reported_epoch":412,"state":"active+clean","last_fresh":"2024-03-11T21:43:26.938987+0000","last_change":"2024-03-11T21:40:26.587777+0000","last_active":"2024-03-11T21:43:26.938987+0000","last_peered":"2024-03-11T21:43:26.938987+0000","last_clean":"2024-03-11T21:43:26.938987+0000","last_became_active":"2024-03-11T21:36:35.060342+0000","last_became_peered":"2024-03-11T21:36:35.060342+0000","last_unstale":"2024-03-11T21:43:26.938987+0000","last_undegraded":"2024-03-11T21:43:26.938987+0000","last_fullsized":"2024-03-11T21:43:26.938987+0000","mapping_epoch":277,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":278,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:40:26.587672+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:41.544550+0000","last_clean_scrub_stamp":"2024-03-11T21:40:26.587672+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,1],"acting":[5,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.11","version":"0'0","reported_seq":772,"reported_epoch":412,"state":"active+clean","last_fresh":"2024-03-11T21:43:27.464017+0000","last_change":"2024-03-11T21:43:27.464052+0000","last_active":"2024-03-11T21:43:27.464017+0000","last_peered":"2024-03-11T21:43:27.464017+0000","last_clean":"2024-03-11T21:43:27.464017+0000","last_became_active":"2024-03-11T21:25:38.537415+0000","last_became_peered":"2024-03-11T21:25:38.537415+0000","last_unstale":"2024-03-11T21:43:27.464017+0000","last_undegraded":"2024-03-11T21:43:27.464017+0000","last_fullsized":"2024-03-11T21:43:27.464017+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:43:27.463941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:37.512018+0000","last_clean_scrub_stamp":"2024-03-11T21:43:27.463941+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:44:38.380709+0000","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.17","version":"260'15","reported_seq":702,"reported_epoch":410,"state":"active+clean","last_fresh":"2024-03-11T21:43:23.943285+0000","last_change":"2024-03-11T21:41:00.652483+0000","last_active":"2024-03-11T21:43:23.943285+0000","last_peered":"2024-03-11T21:43:23.943285+0000","last_clean":"2024-03-11T21:43:23.943285+0000","last_became_active":"2024-03-11T21:36:35.071346+0000","last_became_peered":"2024-03-11T21:36:35.071346+0000","last_unstale":"2024-03-11T21:43:23.943285+0000","last_undegraded":"2024-03-11T21:43:23.943285+0000","last_fullsized":"2024-03-11T21:43:23.943285+0000","mapping_epoch":277,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":278,"parent":"0.0","parent_split_bits":0,"last_scrub":"260'15","last_scrub_stamp":"2024-03-11T21:41:00.652358+0000","last_deep_scrub":"97'9","last_deep_scrub_stamp":"2024-03-11T21:27:25.803132+0000","last_clean_scrub_stamp":"2024-03-11T21:41:00.652358+0000","objects_scrubbed":3,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":17,"num_read_kb":14,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,5],"acting":[3,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.18","version":"412'7394","reported_seq":14166,"reported_epoch":412,"state":"active+clean","last_fresh":"2024-03-11T21:43:28.157235+0000","last_change":"2024-03-11T21:43:17.871944+0000","last_active":"2024-03-11T21:43:28.157235+0000","last_peered":"2024-03-11T21:43:28.157235+0000","last_clean":"2024-03-11T21:43:28.157235+0000","last_became_active":"2024-03-11T21:43:17.856511+0000","last_became_peered":"2024-03-11T21:43:17.856511+0000","last_unstale":"2024-03-11T21:43:28.157235+0000","last_undegraded":"2024-03-11T21:43:28.157235+0000","last_fullsized":"2024-03-11T21:43:28.157235+0000","mapping_epoch":405,"log_start":"0'0","ondisk_log_start":"0'0","created":372,"last_epoch_clean":406,"parent":"0.0","parent_split_bits":5,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:40:38.219602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:40:38.219602+0000","last_clean_scrub_stamp":"2024-03-11T21:40:38.219602+0000","objects_scrubbed":0,"log_size":7394,"log_dups_size":0,"ondisk_log_size":7394,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:43:29.258781+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":114294784,"num_objects":1744,"num_object_clones":0,"num_object_copies":5232,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1744,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5222,"num_write_kb":222912,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2212,"num_bytes_recovered":144965632,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.14","version":"0'0","reported_seq":754,"reported_epoch":412,"state":"active+clean","last_fresh":"2024-03-11T21:43:26.965477+0000","last_change":"2024-03-11T21:41:08.130109+0000","last_active":"2024-03-11T21:43:26.965477+0000","last_peered":"2024-03-11T21:43:26.965477+0000","last_clean":"2024-03-11T21:43:26.965477+0000","last_became_active":"2024-03-11T21:34:41.136946+0000","last_became_peered":"2024-03-11T21:34:41.136946+0000","last_unstale":"2024-03-11T21:43:26.965477+0000","last_undegraded":"2024-03-11T21:43:26.965477+0000","last_fullsized":"2024-03-11T21:43:26.965477+0000","mapping_epoch":226,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":227,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:41:08.129982+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:43.559452+0000","last_clean_scrub_stamp":"2024-03-11T21:41:08.129982+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,7],"acting":[2,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":656,"reported_epoch":410,"state":"active+clean","last_fresh":"2024-03-11T21:43:23.942848+0000","last_change":"2024-03-11T21:41:31.530082+0000","last_active":"2024-03-11T21:43:23.942848+0000","last_peered":"2024-03-11T21:43:23.942848+0000","last_clean":"2024-03-11T21:43:23.942848+0000","last_became_active":"2024-03-11T21:36:35.076335+0000","last_became_peered":"2024-03-11T21:36:35.076335+0000","last_unstale":"2024-03-11T21:43:23.942848+0000","last_undegraded":"2024-03-11T21:43:23.942848+0000","last_fullsized":"2024-03-11T21:43:23.942848+0000","mapping_epoch":277,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":278,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:41:31.529958+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:41.544550+0000","last_clean_scrub_stamp":"2024-03-11T21:41:31.529958+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.10","version":"0'0","reported_seq":754,"reported_epoch":412,"state":"active+clean","last_fresh":"2024-03-11T21:43:26.965295+0000","last_change":"2024-03-11T21:41:23.299628+0000","last_active":"2024-03-11T21:43:26.965295+0000","last_peered":"2024-03-11T21:43:26.965295+0000","last_clean":"2024-03-11T21:43:26.965295+0000","last_became_active":"2024-03-11T21:25:38.533138+0000","last_became_peered":"2024-03-11T21:25:38.533138+0000","last_unstale":"2024-03-11T21:43:26.965295+0000","last_undegraded":"2024-03-11T21:43:26.965295+0000","last_fullsized":"2024-03-11T21:43:26.965295+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:41:23.299503+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:37.512018+0000","last_clean_scrub_stamp":"2024-03-11T21:41:23.299503+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,0],"acting":[2,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.16","version":"369'19","reported_seq":827,"reported_epoch":410,"state":"active+clean","last_fresh":"2024-03-11T21:43:23.933246+0000","last_change":"2024-03-11T21:41:22.115878+0000","last_active":"2024-03-11T21:43:23.933246+0000","last_peered":"2024-03-11T21:43:23.933246+0000","last_clean":"2024-03-11T21:43:23.933246+0000","last_became_active":"2024-03-11T21:36:35.056862+0000","last_became_peered":"2024-03-11T21:36:35.056862+0000","last_unstale":"2024-03-11T21:43:23.933246+0000","last_undegraded":"2024-03-11T21:43:23.933246+0000","last_fullsized":"2024-03-11T21:43:23.933246+0000","mapping_epoch":277,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":278,"parent":"0.0","parent_split_bits":0,"last_scrub":"369'19","last_scrub_stamp":"2024-03-11T21:41:22.115634+0000","last_deep_scrub":"369'19","last_deep_scrub_stamp":"2024-03-11T21:41:22.115634+0000","last_clean_scrub_stamp":"2024-03-11T21:41:22.115634+0000","objects_scrubbed":4,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":11,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.7","version":"412'7352","reported_seq":7513,"reported_epoch":412,"state":"active+clean","last_fresh":"2024-03-11T21:43:27.400254+0000","last_change":"2024-03-11T21:41:31.118161+0000","last_active":"2024-03-11T21:43:27.400254+0000","last_peered":"2024-03-11T21:43:27.400254+0000","last_clean":"2024-03-11T21:43:27.400254+0000","last_became_active":"2024-03-11T21:41:28.840566+0000","last_became_peered":"2024-03-11T21:41:28.840566+0000","last_unstale":"2024-03-11T21:43:27.400254+0000","last_undegraded":"2024-03-11T21:43:27.400254+0000","last_fullsized":"2024-03-11T21:43:27.400254+0000","mapping_epoch":356,"log_start":"0'0","ondisk_log_start":"0'0","created":356,"last_epoch_clean":373,"parent":"0.0","parent_split_bits":0,"last_scrub":"374'4410","last_scrub_stamp":"2024-03-11T21:41:31.117966+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:40:38.219602+0000","last_clean_scrub_stamp":"2024-03-11T21:41:31.117966+0000","objects_scrubbed":2221,"log_size":7352,"log_dups_size":0,"ondisk_log_size":7352,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":256,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":111476736,"num_objects":1701,"num_object_clones":0,"num_object_copies":5103,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1701,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2942,"num_write_kb":77504,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,5],"acting":[6,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.b","version":"0'0","reported_seq":645,"reported_epoch":410,"state":"active+clean","last_fresh":"2024-03-11T21:43:23.940061+0000","last_change":"2024-03-11T21:40:27.611092+0000","last_active":"2024-03-11T21:43:23.940061+0000","last_peered":"2024-03-11T21:43:23.940061+0000","last_clean":"2024-03-11T21:43:23.940061+0000","last_became_active":"2024-03-11T21:36:35.072147+0000","last_became_peered":"2024-03-11T21:36:35.072147+0000","last_unstale":"2024-03-11T21:43:23.940061+0000","last_undegraded":"2024-03-11T21:43:23.940061+0000","last_fullsized":"2024-03-11T21:43:23.940061+0000","mapping_epoch":277,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":278,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:40:27.610987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:43.559452+0000","last_clean_scrub_stamp":"2024-03-11T21:40:27.610987+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":60,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.f","version":"67'2","reported_seq":768,"reported_epoch":412,"state":"active+clean","last_fresh":"2024-03-11T21:43:26.945084+0000","last_change":"2024-03-11T21:41:13.539379+0000","last_active":"2024-03-11T21:43:26.945084+0000","last_peered":"2024-03-11T21:43:26.945084+0000","last_clean":"2024-03-11T21:43:26.945084+0000","last_became_active":"2024-03-11T21:25:38.537711+0000","last_became_peered":"2024-03-11T21:25:38.537711+0000","last_unstale":"2024-03-11T21:43:26.945084+0000","last_undegraded":"2024-03-11T21:43:26.945084+0000","last_fullsized":"2024-03-11T21:43:26.945084+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'2","last_scrub_stamp":"2024-03-11T21:41:13.539240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:37.512018+0000","last_clean_scrub_stamp":"2024-03-11T21:41:13.539240+0000","objects_scrubbed":2,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":92,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":10,"num_read_kb":10,"num_write":4,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,7],"acting":[4,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.9","version":"260'23","reported_seq":800,"reported_epoch":412,"state":"active+clean","last_fresh":"2024-03-11T21:43:26.944417+0000","last_change":"2024-03-11T21:40:14.543701+0000","last_active":"2024-03-11T21:43:26.944417+0000","last_peered":"2024-03-11T21:43:26.944417+0000","last_clean":"2024-03-11T21:43:26.944417+0000","last_became_active":"2024-03-11T21:36:35.059930+0000","last_became_peered":"2024-03-11T21:36:35.059930+0000","last_unstale":"2024-03-11T21:43:26.944417+0000","last_undegraded":"2024-03-11T21:43:26.944417+0000","last_fullsized":"2024-03-11T21:43:26.944417+0000","mapping_epoch":277,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":278,"parent":"0.0","parent_split_bits":0,"last_scrub":"260'23","last_scrub_stamp":"2024-03-11T21:40:14.543596+0000","last_deep_scrub":"146'17","last_deep_scrub_stamp":"2024-03-11T21:34:26.765075+0000","last_clean_scrub_stamp":"2024-03-11T21:40:14.543596+0000","objects_scrubbed":5,"log_size":23,"log_dups_size":0,"ondisk_log_size":23,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":34,"num_read_kb":25,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,3],"acting":[4,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":747,"reported_epoch":412,"state":"active+clean","last_fresh":"2024-03-11T21:43:26.965526+0000","last_change":"2024-03-11T21:41:15.238256+0000","last_active":"2024-03-11T21:43:26.965526+0000","last_peered":"2024-03-11T21:43:26.965526+0000","last_clean":"2024-03-11T21:43:26.965526+0000","last_became_active":"2024-03-11T21:25:42.573987+0000","last_became_peered":"2024-03-11T21:25:42.573987+0000","last_unstale":"2024-03-11T21:43:26.965526+0000","last_undegraded":"2024-03-11T21:43:26.965526+0000","last_fullsized":"2024-03-11T21:43:26.965526+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:41:15.238017+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:41.544550+0000","last_clean_scrub_stamp":"2024-03-11T21:41:15.238017+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"10.c","version":"412'10300","reported_seq":10426,"reported_epoch":412,"state":"active+clean","last_fresh":"2024-03-11T21:43:27.439667+0000","last_change":"2024-03-11T21:40:39.242008+0000","last_active":"2024-03-11T21:43:27.439667+0000","last_peered":"2024-03-11T21:43:27.439667+0000","last_clean":"2024-03-11T21:43:27.439667+0000","last_became_active":"2024-03-11T21:40:39.241699+0000","last_became_peered":"2024-03-11T21:40:39.241699+0000","last_unstale":"2024-03-11T21:43:27.439667+0000","last_undegraded":"2024-03-11T21:43:27.439667+0000","last_fullsized":"2024-03-11T21:43:27.439667+0000","mapping_epoch":356,"log_start":"384'5600","ondisk_log_start":"384'5600","created":356,"last_epoch_clean":357,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:40:38.219602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:40:38.219602+0000","last_clean_scrub_stamp":"2024-03-11T21:40:38.219602+0000","objects_scrubbed":0,"log_size":4700,"log_dups_size":0,"ondisk_log_size":4700,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":225837056,"num_objects":3446,"num_object_clones":0,"num_object_copies":10338,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3446,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10300,"num_write_kb":439872,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,0],"acting":[7,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.0","version":"0'0","reported_seq":742,"reported_epoch":410,"state":"active+clean","last_fresh":"2024-03-11T21:43:23.932701+0000","last_change":"2024-03-11T21:40:27.230013+0000","last_active":"2024-03-11T21:43:23.932701+0000","last_peered":"2024-03-11T21:43:23.932701+0000","last_clean":"2024-03-11T21:43:23.932701+0000","last_became_active":"2024-03-11T21:36:35.058054+0000","last_became_peered":"2024-03-11T21:36:35.058054+0000","last_unstale":"2024-03-11T21:43:23.932701+0000","last_undegraded":"2024-03-11T21:43:23.932701+0000","last_fullsized":"2024-03-11T21:43:23.932701+0000","mapping_epoch":277,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":278,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:40:27.229904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:29:39.498784+0000","last_clean_scrub_stamp":"2024-03-11T21:40:27.229904+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":38,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":769,"reported_epoch":412,"state":"active+clean","last_fresh":"2024-03-11T21:43:26.941310+0000","last_change":"2024-03-11T21:41:22.292906+0000","last_active":"2024-03-11T21:43:26.941310+0000","last_peered":"2024-03-11T21:43:26.941310+0000","last_clean":"2024-03-11T21:43:26.941310+0000","last_became_active":"2024-03-11T21:34:41.150913+0000","last_became_peered":"2024-03-11T21:34:41.150913+0000","last_unstale":"2024-03-11T21:43:26.941310+0000","last_undegraded":"2024-03-11T21:43:26.941310+0000","last_fullsized":"2024-03-11T21:43:26.941310+0000","mapping_epoch":226,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":227,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:41:22.292789+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:37.512018+0000","last_clean_scrub_stamp":"2024-03-11T21:41:22.292789+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,7],"acting":[1,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.2","version":"260'25","reported_seq":788,"reported_epoch":412,"state":"active+clean","last_fresh":"2024-03-11T21:43:26.940494+0000","last_change":"2024-03-11T21:41:07.243196+0000","last_active":"2024-03-11T21:43:26.940494+0000","last_peered":"2024-03-11T21:43:26.940494+0000","last_clean":"2024-03-11T21:43:26.940494+0000","last_became_active":"2024-03-11T21:25:40.548833+0000","last_became_peered":"2024-03-11T21:25:40.548833+0000","last_unstale":"2024-03-11T21:43:26.940494+0000","last_undegraded":"2024-03-11T21:43:26.940494+0000","last_fullsized":"2024-03-11T21:43:26.940494+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"260'25","last_scrub_stamp":"2024-03-11T21:41:07.242950+0000","last_deep_scrub":"94'15","last_deep_scrub_stamp":"2024-03-11T21:26:49.993060+0000","last_clean_scrub_stamp":"2024-03-11T21:41:07.242950+0000","objects_scrubbed":5,"log_size":25,"log_dups_size":0,"ondisk_log_size":25,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":25,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.3","version":"74'8","reported_seq":967,"reported_epoch":410,"state":"active+clean","last_fresh":"2024-03-11T21:43:23.936483+0000","last_change":"2024-03-11T21:41:31.189011+0000","last_active":"2024-03-11T21:43:23.936483+0000","last_peered":"2024-03-11T21:43:23.936483+0000","last_clean":"2024-03-11T21:43:23.936483+0000","last_became_active":"2024-03-11T21:25:42.578161+0000","last_became_peered":"2024-03-11T21:25:42.578161+0000","last_unstale":"2024-03-11T21:43:23.936483+0000","last_undegraded":"2024-03-11T21:43:23.936483+0000","last_fullsized":"2024-03-11T21:43:23.936483+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'8","last_scrub_stamp":"2024-03-11T21:41:31.188904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:41.544550+0000","last_clean_scrub_stamp":"2024-03-11T21:41:31.188904+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,5],"acting":[0,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.a","version":"412'10289","reported_seq":10414,"reported_epoch":412,"state":"active+clean","last_fresh":"2024-03-11T21:43:27.446140+0000","last_change":"2024-03-11T21:40:39.243111+0000","last_active":"2024-03-11T21:43:27.446140+0000","last_peered":"2024-03-11T21:43:27.446140+0000","last_clean":"2024-03-11T21:43:27.446140+0000","last_became_active":"2024-03-11T21:40:39.242809+0000","last_became_peered":"2024-03-11T21:40:39.242809+0000","last_unstale":"2024-03-11T21:43:27.446140+0000","last_undegraded":"2024-03-11T21:43:27.446140+0000","last_fullsized":"2024-03-11T21:43:27.446140+0000","mapping_epoch":356,"log_start":"378'4700","ondisk_log_start":"378'4700","created":356,"last_epoch_clean":357,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:40:38.219602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:40:38.219602+0000","last_clean_scrub_stamp":"2024-03-11T21:40:38.219602+0000","objects_scrubbed":0,"log_size":5589,"log_dups_size":0,"ondisk_log_size":5589,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":218038272,"num_objects":3327,"num_object_clones":0,"num_object_copies":9981,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3327,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10289,"num_write_kb":435712,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":675,"reported_epoch":410,"state":"active+clean","last_fresh":"2024-03-11T21:43:23.940418+0000","last_change":"2024-03-11T21:36:35.068977+0000","last_active":"2024-03-11T21:43:23.940418+0000","last_peered":"2024-03-11T21:43:23.940418+0000","last_clean":"2024-03-11T21:43:23.940418+0000","last_became_active":"2024-03-11T21:36:35.067288+0000","last_became_peered":"2024-03-11T21:36:35.067288+0000","last_unstale":"2024-03-11T21:43:23.940418+0000","last_undegraded":"2024-03-11T21:43:23.940418+0000","last_fullsized":"2024-03-11T21:43:23.940418+0000","mapping_epoch":277,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":278,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:35:09.985377+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:28:30.563712+0000","last_clean_scrub_stamp":"2024-03-11T21:35:09.985377+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,7],"acting":[3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":764,"reported_epoch":412,"state":"active+clean","last_fresh":"2024-03-11T21:43:26.938524+0000","last_change":"2024-03-11T21:41:24.921735+0000","last_active":"2024-03-11T21:43:26.938524+0000","last_peered":"2024-03-11T21:43:26.938524+0000","last_clean":"2024-03-11T21:43:26.938524+0000","last_became_active":"2024-03-11T21:25:38.538540+0000","last_became_peered":"2024-03-11T21:25:38.538540+0000","last_unstale":"2024-03-11T21:43:26.938524+0000","last_undegraded":"2024-03-11T21:43:26.938524+0000","last_fullsized":"2024-03-11T21:43:26.938524+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:41:24.921598+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:37.512018+0000","last_clean_scrub_stamp":"2024-03-11T21:41:24.921598+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.4","version":"280'42","reported_seq":1050,"reported_epoch":412,"state":"active+clean","last_fresh":"2024-03-11T21:43:26.941229+0000","last_change":"2024-03-11T21:41:05.234160+0000","last_active":"2024-03-11T21:43:26.941229+0000","last_peered":"2024-03-11T21:43:26.941229+0000","last_clean":"2024-03-11T21:43:26.941229+0000","last_became_active":"2024-03-11T21:36:35.073430+0000","last_became_peered":"2024-03-11T21:36:35.073430+0000","last_unstale":"2024-03-11T21:43:26.941229+0000","last_undegraded":"2024-03-11T21:43:26.941229+0000","last_fullsized":"2024-03-11T21:43:26.941229+0000","mapping_epoch":277,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":278,"parent":"0.0","parent_split_bits":0,"last_scrub":"280'42","last_scrub_stamp":"2024-03-11T21:41:05.233993+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:39.525554+0000","last_clean_scrub_stamp":"2024-03-11T21:41:05.233993+0000","objects_scrubbed":10,"log_size":42,"log_dups_size":0,"ondisk_log_size":42,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":358,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":66,"num_read_kb":51,"num_write":38,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3],"acting":[1,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":737,"reported_epoch":410,"state":"active+clean","last_fresh":"2024-03-11T21:43:23.938207+0000","last_change":"2024-03-11T21:35:32.651731+0000","last_active":"2024-03-11T21:43:23.938207+0000","last_peered":"2024-03-11T21:43:23.938207+0000","last_clean":"2024-03-11T21:43:23.938207+0000","last_became_active":"2024-03-11T21:25:42.572673+0000","last_became_peered":"2024-03-11T21:25:42.572673+0000","last_unstale":"2024-03-11T21:43:23.938207+0000","last_undegraded":"2024-03-11T21:43:23.938207+0000","last_fullsized":"2024-03-11T21:43:23.938207+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:35:32.651652+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:41.544550+0000","last_clean_scrub_stamp":"2024-03-11T21:35:32.651652+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.9","version":"412'8053","reported_seq":8276,"reported_epoch":412,"state":"active+clean","last_fresh":"2024-03-11T21:43:26.940783+0000","last_change":"2024-03-11T21:41:32.409763+0000","last_active":"2024-03-11T21:43:26.940783+0000","last_peered":"2024-03-11T21:43:26.940783+0000","last_clean":"2024-03-11T21:43:26.940783+0000","last_became_active":"2024-03-11T21:41:28.824488+0000","last_became_peered":"2024-03-11T21:41:28.824488+0000","last_unstale":"2024-03-11T21:43:26.940783+0000","last_undegraded":"2024-03-11T21:43:26.940783+0000","last_fullsized":"2024-03-11T21:43:26.940783+0000","mapping_epoch":356,"log_start":"365'1488","ondisk_log_start":"365'1488","created":356,"last_epoch_clean":373,"parent":"0.0","parent_split_bits":0,"last_scrub":"376'5094","last_scrub_stamp":"2024-03-11T21:41:32.409608+0000","last_deep_scrub":"376'5094","last_deep_scrub_stamp":"2024-03-11T21:41:32.409608+0000","last_clean_scrub_stamp":"2024-03-11T21:41:32.409608+0000","objects_scrubbed":2132,"log_size":6565,"log_dups_size":0,"ondisk_log_size":6565,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":4,"scrub_schedule":"queued for scrub","scrub_duration":3401,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":111542272,"num_objects":1702,"num_object_clones":0,"num_object_copies":5106,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1702,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2962,"num_write_kb":81024,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.5","version":"0'0","reported_seq":410,"reported_epoch":412,"state":"active+clean","last_fresh":"2024-03-11T21:43:26.946172+0000","last_change":"2024-03-11T21:41:12.380275+0000","last_active":"2024-03-11T21:43:26.946172+0000","last_peered":"2024-03-11T21:43:26.946172+0000","last_clean":"2024-03-11T21:43:26.946172+0000","last_became_active":"2024-03-11T21:36:35.069358+0000","last_became_peered":"2024-03-11T21:36:35.069358+0000","last_unstale":"2024-03-11T21:43:26.946172+0000","last_undegraded":"2024-03-11T21:43:26.946172+0000","last_fullsized":"2024-03-11T21:43:26.946172+0000","mapping_epoch":277,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":278,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:41:12.380019+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:43.559452+0000","last_clean_scrub_stamp":"2024-03-11T21:41:12.380019+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":53,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,3],"acting":[7,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.0","version":"164'6","reported_seq":1829,"reported_epoch":412,"state":"active+clean","last_fresh":"2024-03-11T21:43:26.939664+0000","last_change":"2024-03-11T21:41:30.197407+0000","last_active":"2024-03-11T21:43:26.939664+0000","last_peered":"2024-03-11T21:43:26.939664+0000","last_clean":"2024-03-11T21:43:26.939664+0000","last_became_active":"2024-03-11T21:25:38.536890+0000","last_became_peered":"2024-03-11T21:25:38.536890+0000","last_unstale":"2024-03-11T21:43:26.939664+0000","last_undegraded":"2024-03-11T21:43:26.939664+0000","last_fullsized":"2024-03-11T21:43:26.939664+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"164'6","last_scrub_stamp":"2024-03-11T21:41:30.197276+0000","last_deep_scrub":"164'6","last_deep_scrub_stamp":"2024-03-11T21:32:17.597666+0000","last_clean_scrub_stamp":"2024-03-11T21:41:30.197276+0000","objects_scrubbed":1,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0.000753559,"stat_sum":{"num_bytes":389,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1061,"num_read_kb":1056,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,6],"acting":[1,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":772,"reported_epoch":412,"state":"active+clean","last_fresh":"2024-03-11T21:43:28.183447+0000","last_change":"2024-03-11T21:43:28.183519+0000","last_active":"2024-03-11T21:43:28.183447+0000","last_peered":"2024-03-11T21:43:28.183447+0000","last_clean":"2024-03-11T21:43:28.183447+0000","last_became_active":"2024-03-11T21:36:35.056966+0000","last_became_peered":"2024-03-11T21:36:35.056966+0000","last_unstale":"2024-03-11T21:43:28.183447+0000","last_undegraded":"2024-03-11T21:43:28.183447+0000","last_fullsized":"2024-03-11T21:43:28.183447+0000","mapping_epoch":277,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":278,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:43:28.183315+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:43:28.183315+0000","last_clean_scrub_stamp":"2024-03-11T21:43:28.183315+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:44:29.807489+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.7","version":"260'22","reported_seq":794,"reported_epoch":412,"state":"active+clean","last_fresh":"2024-03-11T21:43:26.941553+0000","last_change":"2024-03-11T21:41:27.232567+0000","last_active":"2024-03-11T21:43:26.941553+0000","last_peered":"2024-03-11T21:43:26.941553+0000","last_clean":"2024-03-11T21:43:26.941553+0000","last_became_active":"2024-03-11T21:25:40.541769+0000","last_became_peered":"2024-03-11T21:25:40.541769+0000","last_unstale":"2024-03-11T21:43:26.941553+0000","last_undegraded":"2024-03-11T21:43:26.941553+0000","last_fullsized":"2024-03-11T21:43:26.941553+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"260'22","last_scrub_stamp":"2024-03-11T21:41:27.232403+0000","last_deep_scrub":"260'22","last_deep_scrub_stamp":"2024-03-11T21:41:27.232403+0000","last_clean_scrub_stamp":"2024-03-11T21:41:27.232403+0000","objects_scrubbed":5,"log_size":22,"log_dups_size":0,"ondisk_log_size":22,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":36,"num_read_kb":25,"num_write":20,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":759,"reported_epoch":412,"state":"active+clean","last_fresh":"2024-03-11T21:43:26.965951+0000","last_change":"2024-03-11T21:41:04.111560+0000","last_active":"2024-03-11T21:43:26.965951+0000","last_peered":"2024-03-11T21:43:26.965951+0000","last_clean":"2024-03-11T21:43:26.965951+0000","last_became_active":"2024-03-11T21:34:41.143660+0000","last_became_peered":"2024-03-11T21:34:41.143660+0000","last_unstale":"2024-03-11T21:43:26.965951+0000","last_undegraded":"2024-03-11T21:43:26.965951+0000","last_fullsized":"2024-03-11T21:43:26.965951+0000","mapping_epoch":226,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":227,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:41:04.111481+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:41.544550+0000","last_clean_scrub_stamp":"2024-03-11T21:41:04.111481+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7],"acting":[2,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"10.b","version":"412'10425","reported_seq":10551,"reported_epoch":412,"state":"active+clean","last_fresh":"2024-03-11T21:43:27.470217+0000","last_change":"2024-03-11T21:40:39.241086+0000","last_active":"2024-03-11T21:43:27.470217+0000","last_peered":"2024-03-11T21:43:27.470217+0000","last_clean":"2024-03-11T21:43:27.470217+0000","last_became_active":"2024-03-11T21:40:39.240775+0000","last_became_peered":"2024-03-11T21:40:39.240775+0000","last_unstale":"2024-03-11T21:43:27.470217+0000","last_undegraded":"2024-03-11T21:43:27.470217+0000","last_fullsized":"2024-03-11T21:43:27.470217+0000","mapping_epoch":356,"log_start":"378'4800","ondisk_log_start":"378'4800","created":356,"last_epoch_clean":357,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:40:38.219602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:40:38.219602+0000","last_clean_scrub_stamp":"2024-03-11T21:40:38.219602+0000","objects_scrubbed":0,"log_size":5625,"log_dups_size":0,"ondisk_log_size":5625,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220135424,"num_objects":3359,"num_object_clones":0,"num_object_copies":10077,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3359,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10425,"num_write_kb":441088,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,0],"acting":[6,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.7","version":"124'2","reported_seq":747,"reported_epoch":412,"state":"active+clean","last_fresh":"2024-03-11T21:43:26.939469+0000","last_change":"2024-03-11T21:40:31.722243+0000","last_active":"2024-03-11T21:43:26.939469+0000","last_peered":"2024-03-11T21:43:26.939469+0000","last_clean":"2024-03-11T21:43:26.939469+0000","last_became_active":"2024-03-11T21:36:35.058786+0000","last_became_peered":"2024-03-11T21:36:35.058786+0000","last_unstale":"2024-03-11T21:43:26.939469+0000","last_undegraded":"2024-03-11T21:43:26.939469+0000","last_fullsized":"2024-03-11T21:43:26.939469+0000","mapping_epoch":277,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":278,"parent":"0.0","parent_split_bits":0,"last_scrub":"124'2","last_scrub_stamp":"2024-03-11T21:40:31.722100+0000","last_deep_scrub":"124'2","last_deep_scrub_stamp":"2024-03-11T21:29:34.956908+0000","last_clean_scrub_stamp":"2024-03-11T21:40:31.722100+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":87,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,4],"acting":[5,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2","version":"71'2","reported_seq":704,"reported_epoch":410,"state":"active+clean","last_fresh":"2024-03-11T21:43:23.939878+0000","last_change":"2024-03-11T21:41:07.517286+0000","last_active":"2024-03-11T21:43:23.939878+0000","last_peered":"2024-03-11T21:43:23.939878+0000","last_clean":"2024-03-11T21:43:23.939878+0000","last_became_active":"2024-03-11T21:36:35.070783+0000","last_became_peered":"2024-03-11T21:36:35.070783+0000","last_unstale":"2024-03-11T21:43:23.939878+0000","last_undegraded":"2024-03-11T21:43:23.939878+0000","last_fullsized":"2024-03-11T21:43:23.939878+0000","mapping_epoch":277,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":278,"parent":"0.0","parent_split_bits":0,"last_scrub":"71'2","last_scrub_stamp":"2024-03-11T21:41:07.517154+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:37.512018+0000","last_clean_scrub_stamp":"2024-03-11T21:41:07.517154+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0.0014206449999999999,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":774,"reported_epoch":412,"state":"active+clean","last_fresh":"2024-03-11T21:43:26.938626+0000","last_change":"2024-03-11T21:41:10.803049+0000","last_active":"2024-03-11T21:43:26.938626+0000","last_peered":"2024-03-11T21:43:26.938626+0000","last_clean":"2024-03-11T21:43:26.938626+0000","last_became_active":"2024-03-11T21:34:41.146672+0000","last_became_peered":"2024-03-11T21:34:41.146672+0000","last_unstale":"2024-03-11T21:43:26.938626+0000","last_undegraded":"2024-03-11T21:43:26.938626+0000","last_fullsized":"2024-03-11T21:43:26.938626+0000","mapping_epoch":226,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":227,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:41:10.802921+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:37.512018+0000","last_clean_scrub_stamp":"2024-03-11T21:41:10.802921+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"87'128","reported_seq":872,"reported_epoch":412,"state":"active+clean","last_fresh":"2024-03-11T21:43:26.959702+0000","last_change":"2024-03-11T21:41:09.328321+0000","last_active":"2024-03-11T21:43:26.959702+0000","last_peered":"2024-03-11T21:43:26.959702+0000","last_clean":"2024-03-11T21:43:26.959702+0000","last_became_active":"2024-03-11T21:34:44.164569+0000","last_became_peered":"2024-03-11T21:34:44.164569+0000","last_unstale":"2024-03-11T21:43:26.959702+0000","last_undegraded":"2024-03-11T21:43:26.959702+0000","last_fullsized":"2024-03-11T21:43:26.959702+0000","mapping_epoch":229,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":230,"parent":"0.0","parent_split_bits":0,"last_scrub":"87'128","last_scrub_stamp":"2024-03-11T21:41:09.328197+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:24.261974+0000","last_clean_scrub_stamp":"2024-03-11T21:41:09.328197+0000","objects_scrubbed":2,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":198,"num_read_kb":294,"num_write":179,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":2951840,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.5","version":"260'28","reported_seq":1023,"reported_epoch":412,"state":"active+clean","last_fresh":"2024-03-11T21:43:26.955400+0000","last_change":"2024-03-11T21:41:07.923860+0000","last_active":"2024-03-11T21:43:26.955400+0000","last_peered":"2024-03-11T21:43:26.955400+0000","last_clean":"2024-03-11T21:43:26.955400+0000","last_became_active":"2024-03-11T21:25:40.761457+0000","last_became_peered":"2024-03-11T21:25:40.761457+0000","last_unstale":"2024-03-11T21:43:26.955400+0000","last_undegraded":"2024-03-11T21:43:26.955400+0000","last_fullsized":"2024-03-11T21:43:26.955400+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"260'28","last_scrub_stamp":"2024-03-11T21:41:07.923729+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:39.525554+0000","last_clean_scrub_stamp":"2024-03-11T21:41:07.923729+0000","objects_scrubbed":8,"log_size":28,"log_dups_size":0,"ondisk_log_size":28,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":154,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":43,"num_read_kb":33,"num_write":25,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":401,"reported_epoch":412,"state":"active+clean","last_fresh":"2024-03-11T21:43:26.946046+0000","last_change":"2024-03-11T21:41:00.152627+0000","last_active":"2024-03-11T21:43:26.946046+0000","last_peered":"2024-03-11T21:43:26.946046+0000","last_clean":"2024-03-11T21:43:26.946046+0000","last_became_active":"2024-03-11T21:34:41.139782+0000","last_became_peered":"2024-03-11T21:34:41.139782+0000","last_unstale":"2024-03-11T21:43:26.946046+0000","last_undegraded":"2024-03-11T21:43:26.946046+0000","last_fullsized":"2024-03-11T21:43:26.946046+0000","mapping_epoch":226,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":227,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:41:00.152377+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:32:21.266672+0000","last_clean_scrub_stamp":"2024-03-11T21:41:00.152377+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,5],"acting":[7,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.8","version":"410'7292","reported_seq":7448,"reported_epoch":410,"state":"active+clean","last_fresh":"2024-03-11T21:43:24.380524+0000","last_change":"2024-03-11T21:41:29.898309+0000","last_active":"2024-03-11T21:43:24.380524+0000","last_peered":"2024-03-11T21:43:24.380524+0000","last_clean":"2024-03-11T21:43:24.380524+0000","last_became_active":"2024-03-11T21:41:28.829878+0000","last_became_peered":"2024-03-11T21:41:28.829878+0000","last_unstale":"2024-03-11T21:43:24.380524+0000","last_undegraded":"2024-03-11T21:43:24.380524+0000","last_fullsized":"2024-03-11T21:43:24.380524+0000","mapping_epoch":356,"log_start":"364'1153","ondisk_log_start":"364'1153","created":356,"last_epoch_clean":373,"parent":"0.0","parent_split_bits":0,"last_scrub":"374'4392","last_scrub_stamp":"2024-03-11T21:41:29.898148+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:40:38.219602+0000","last_clean_scrub_stamp":"2024-03-11T21:41:29.898148+0000","objects_scrubbed":2198,"log_size":6139,"log_dups_size":0,"ondisk_log_size":6139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":722,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":113836032,"num_objects":1737,"num_object_clones":0,"num_object_copies":5211,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1737,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2919,"num_write_kb":78656,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,7],"acting":[0,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.4","version":"0'0","reported_seq":746,"reported_epoch":412,"state":"active+clean","last_fresh":"2024-03-11T21:43:26.941492+0000","last_change":"2024-03-11T21:41:01.134631+0000","last_active":"2024-03-11T21:43:26.941492+0000","last_peered":"2024-03-11T21:43:26.941492+0000","last_clean":"2024-03-11T21:43:26.941492+0000","last_became_active":"2024-03-11T21:36:35.072824+0000","last_became_peered":"2024-03-11T21:36:35.072824+0000","last_unstale":"2024-03-11T21:43:26.941492+0000","last_undegraded":"2024-03-11T21:43:26.941492+0000","last_fullsized":"2024-03-11T21:43:26.941492+0000","mapping_epoch":277,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":278,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:41:01.134504+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:43.559452+0000","last_clean_scrub_stamp":"2024-03-11T21:41:01.134504+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":769,"reported_epoch":410,"state":"active+clean","last_fresh":"2024-03-11T21:43:23.932906+0000","last_change":"2024-03-11T21:41:02.116795+0000","last_active":"2024-03-11T21:43:23.932906+0000","last_peered":"2024-03-11T21:43:23.932906+0000","last_clean":"2024-03-11T21:43:23.932906+0000","last_became_active":"2024-03-11T21:36:35.069452+0000","last_became_peered":"2024-03-11T21:36:35.069452+0000","last_unstale":"2024-03-11T21:43:23.932906+0000","last_undegraded":"2024-03-11T21:43:23.932906+0000","last_fullsized":"2024-03-11T21:43:23.932906+0000","mapping_epoch":277,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":278,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:41:02.116682+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:37.512018+0000","last_clean_scrub_stamp":"2024-03-11T21:41:02.116682+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0.00089342499999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":393,"reported_epoch":412,"state":"active+clean","last_fresh":"2024-03-11T21:43:26.959517+0000","last_change":"2024-03-11T21:40:27.117606+0000","last_active":"2024-03-11T21:43:26.959517+0000","last_peered":"2024-03-11T21:43:26.959517+0000","last_clean":"2024-03-11T21:43:26.959517+0000","last_became_active":"2024-03-11T21:34:41.149219+0000","last_became_peered":"2024-03-11T21:34:41.149219+0000","last_unstale":"2024-03-11T21:43:26.959517+0000","last_undegraded":"2024-03-11T21:43:26.959517+0000","last_fullsized":"2024-03-11T21:43:26.959517+0000","mapping_epoch":226,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":227,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:40:27.117515+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:37.512018+0000","last_clean_scrub_stamp":"2024-03-11T21:40:27.117515+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,0],"acting":[7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.6","version":"352'30","reported_seq":811,"reported_epoch":410,"state":"active+clean","last_fresh":"2024-03-11T21:43:23.933153+0000","last_change":"2024-03-11T21:40:31.175417+0000","last_active":"2024-03-11T21:43:23.933153+0000","last_peered":"2024-03-11T21:43:23.933153+0000","last_clean":"2024-03-11T21:43:23.933153+0000","last_became_active":"2024-03-11T21:32:22.796486+0000","last_became_peered":"2024-03-11T21:32:22.796486+0000","last_unstale":"2024-03-11T21:43:23.933153+0000","last_undegraded":"2024-03-11T21:43:23.933153+0000","last_fullsized":"2024-03-11T21:43:23.933153+0000","mapping_epoch":166,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":167,"parent":"0.0","parent_split_bits":0,"last_scrub":"352'30","last_scrub_stamp":"2024-03-11T21:40:31.175323+0000","last_deep_scrub":"352'30","last_deep_scrub_stamp":"2024-03-11T21:40:31.175323+0000","last_clean_scrub_stamp":"2024-03-11T21:40:31.175323+0000","objects_scrubbed":6,"log_size":30,"log_dups_size":0,"ondisk_log_size":30,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":36,"num_read_kb":30,"num_write":24,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":740,"reported_epoch":412,"state":"active+clean","last_fresh":"2024-03-11T21:43:26.939424+0000","last_change":"2024-03-11T21:36:07.451482+0000","last_active":"2024-03-11T21:43:26.939424+0000","last_peered":"2024-03-11T21:43:26.939424+0000","last_clean":"2024-03-11T21:43:26.939424+0000","last_became_active":"2024-03-11T21:25:42.572324+0000","last_became_peered":"2024-03-11T21:25:42.572324+0000","last_unstale":"2024-03-11T21:43:26.939424+0000","last_undegraded":"2024-03-11T21:43:26.939424+0000","last_fullsized":"2024-03-11T21:43:26.939424+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:36:07.451102+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:41.544550+0000","last_clean_scrub_stamp":"2024-03-11T21:36:07.451102+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":54,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.d","version":"412'10301","reported_seq":10428,"reported_epoch":412,"state":"active+clean","last_fresh":"2024-03-11T21:43:27.452335+0000","last_change":"2024-03-11T21:40:39.242068+0000","last_active":"2024-03-11T21:43:27.452335+0000","last_peered":"2024-03-11T21:43:27.452335+0000","last_clean":"2024-03-11T21:43:27.452335+0000","last_became_active":"2024-03-11T21:40:39.241769+0000","last_became_peered":"2024-03-11T21:40:39.241769+0000","last_unstale":"2024-03-11T21:43:27.452335+0000","last_undegraded":"2024-03-11T21:43:27.452335+0000","last_fullsized":"2024-03-11T21:43:27.452335+0000","mapping_epoch":356,"log_start":"378'4700","ondisk_log_start":"378'4700","created":356,"last_epoch_clean":357,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:40:38.219602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:40:38.219602+0000","last_clean_scrub_stamp":"2024-03-11T21:40:38.219602+0000","objects_scrubbed":0,"log_size":5601,"log_dups_size":0,"ondisk_log_size":5601,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220987416,"num_objects":3373,"num_object_clones":0,"num_object_copies":10119,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3373,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":10301,"num_write_kb":437505,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,0],"acting":[6,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":745,"reported_epoch":412,"state":"active+clean","last_fresh":"2024-03-11T21:43:26.939788+0000","last_change":"2024-03-11T21:36:18.529506+0000","last_active":"2024-03-11T21:43:26.939788+0000","last_peered":"2024-03-11T21:43:26.939788+0000","last_clean":"2024-03-11T21:43:26.939788+0000","last_became_active":"2024-03-11T21:25:45.301416+0000","last_became_peered":"2024-03-11T21:25:45.301416+0000","last_unstale":"2024-03-11T21:43:26.939788+0000","last_undegraded":"2024-03-11T21:43:26.939788+0000","last_fullsized":"2024-03-11T21:43:26.939788+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:36:18.529370+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:43.559452+0000","last_clean_scrub_stamp":"2024-03-11T21:36:18.529370+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":54,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":401,"reported_epoch":412,"state":"active+clean","last_fresh":"2024-03-11T21:43:26.946284+0000","last_change":"2024-03-11T21:41:19.303174+0000","last_active":"2024-03-11T21:43:26.946284+0000","last_peered":"2024-03-11T21:43:26.946284+0000","last_clean":"2024-03-11T21:43:26.946284+0000","last_became_active":"2024-03-11T21:34:41.152721+0000","last_became_peered":"2024-03-11T21:34:41.152721+0000","last_unstale":"2024-03-11T21:43:26.946284+0000","last_undegraded":"2024-03-11T21:43:26.946284+0000","last_fullsized":"2024-03-11T21:43:26.946284+0000","mapping_epoch":226,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":227,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:41:19.302991+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:26:57.333669+0000","last_clean_scrub_stamp":"2024-03-11T21:41:19.302991+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.3","version":"260'37","reported_seq":856,"reported_epoch":410,"state":"active+clean","last_fresh":"2024-03-11T21:43:23.936200+0000","last_change":"2024-03-11T21:41:06.123397+0000","last_active":"2024-03-11T21:43:23.936200+0000","last_peered":"2024-03-11T21:43:23.936200+0000","last_clean":"2024-03-11T21:43:23.936200+0000","last_became_active":"2024-03-11T21:34:41.145607+0000","last_became_peered":"2024-03-11T21:34:41.145607+0000","last_unstale":"2024-03-11T21:43:23.936200+0000","last_undegraded":"2024-03-11T21:43:23.936200+0000","last_fullsized":"2024-03-11T21:43:23.936200+0000","mapping_epoch":226,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":227,"parent":"0.0","parent_split_bits":0,"last_scrub":"260'37","last_scrub_stamp":"2024-03-11T21:41:06.123055+0000","last_deep_scrub":"165'27","last_deep_scrub_stamp":"2024-03-11T21:32:21.576732+0000","last_clean_scrub_stamp":"2024-03-11T21:41:06.123055+0000","objects_scrubbed":8,"log_size":37,"log_dups_size":0,"ondisk_log_size":37,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":54,"num_read_kb":40,"num_write":32,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":660,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":773,"reported_epoch":412,"state":"active+clean","last_fresh":"2024-03-11T21:43:26.955434+0000","last_change":"2024-03-11T21:41:21.857353+0000","last_active":"2024-03-11T21:43:26.955434+0000","last_peered":"2024-03-11T21:43:26.955434+0000","last_clean":"2024-03-11T21:43:26.955434+0000","last_became_active":"2024-03-11T21:25:42.579045+0000","last_became_peered":"2024-03-11T21:25:42.579045+0000","last_unstale":"2024-03-11T21:43:26.955434+0000","last_undegraded":"2024-03-11T21:43:26.955434+0000","last_fullsized":"2024-03-11T21:43:26.955434+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:41:21.857205+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:41:21.857205+0000","last_clean_scrub_stamp":"2024-03-11T21:41:21.857205+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"10.e","version":"412'10151","reported_seq":10277,"reported_epoch":412,"state":"active+clean","last_fresh":"2024-03-11T21:43:27.061758+0000","last_change":"2024-03-11T21:40:39.239887+0000","last_active":"2024-03-11T21:43:27.061758+0000","last_peered":"2024-03-11T21:43:27.061758+0000","last_clean":"2024-03-11T21:43:27.061758+0000","last_became_active":"2024-03-11T21:40:39.239590+0000","last_became_peered":"2024-03-11T21:40:39.239590+0000","last_unstale":"2024-03-11T21:43:27.061758+0000","last_undegraded":"2024-03-11T21:43:27.061758+0000","last_fullsized":"2024-03-11T21:43:27.061758+0000","mapping_epoch":356,"log_start":"384'5600","ondisk_log_start":"384'5600","created":356,"last_epoch_clean":357,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:40:38.219602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:40:38.219602+0000","last_clean_scrub_stamp":"2024-03-11T21:40:38.219602+0000","objects_scrubbed":0,"log_size":4551,"log_dups_size":0,"ondisk_log_size":4551,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":218693632,"num_objects":3337,"num_object_clones":0,"num_object_copies":10011,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3337,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10151,"num_write_kb":431616,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,2],"acting":[4,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":751,"reported_epoch":412,"state":"active+clean","last_fresh":"2024-03-11T21:43:26.944494+0000","last_change":"2024-03-11T21:40:28.582452+0000","last_active":"2024-03-11T21:43:26.944494+0000","last_peered":"2024-03-11T21:43:26.944494+0000","last_clean":"2024-03-11T21:43:26.944494+0000","last_became_active":"2024-03-11T21:32:22.791439+0000","last_became_peered":"2024-03-11T21:32:22.791439+0000","last_unstale":"2024-03-11T21:43:26.944494+0000","last_undegraded":"2024-03-11T21:43:26.944494+0000","last_fullsized":"2024-03-11T21:43:26.944494+0000","mapping_epoch":166,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":167,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:40:28.582339+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:43.559452+0000","last_clean_scrub_stamp":"2024-03-11T21:40:28.582339+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":20,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,2],"acting":[4,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.6","version":"67'1","reported_seq":758,"reported_epoch":412,"state":"active+clean","last_fresh":"2024-03-11T21:43:26.941584+0000","last_change":"2024-03-11T21:41:21.288511+0000","last_active":"2024-03-11T21:43:26.941584+0000","last_peered":"2024-03-11T21:43:26.941584+0000","last_clean":"2024-03-11T21:43:26.941584+0000","last_became_active":"2024-03-11T21:25:38.536142+0000","last_became_peered":"2024-03-11T21:25:38.536142+0000","last_unstale":"2024-03-11T21:43:26.941584+0000","last_undegraded":"2024-03-11T21:43:26.941584+0000","last_fullsized":"2024-03-11T21:43:26.941584+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'1","last_scrub_stamp":"2024-03-11T21:41:21.288031+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:37.512018+0000","last_clean_scrub_stamp":"2024-03-11T21:41:21.288031+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,4],"acting":[1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.0","version":"260'38","reported_seq":752,"reported_epoch":410,"state":"active+clean","last_fresh":"2024-03-11T21:43:23.939839+0000","last_change":"2024-03-11T21:41:30.511287+0000","last_active":"2024-03-11T21:43:23.939839+0000","last_peered":"2024-03-11T21:43:23.939839+0000","last_clean":"2024-03-11T21:43:23.939839+0000","last_became_active":"2024-03-11T21:36:35.068071+0000","last_became_peered":"2024-03-11T21:36:35.068071+0000","last_unstale":"2024-03-11T21:43:23.939839+0000","last_undegraded":"2024-03-11T21:43:23.939839+0000","last_fullsized":"2024-03-11T21:43:23.939839+0000","mapping_epoch":277,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":278,"parent":"0.0","parent_split_bits":0,"last_scrub":"260'38","last_scrub_stamp":"2024-03-11T21:41:30.511168+0000","last_deep_scrub":"93'26","last_deep_scrub_stamp":"2024-03-11T21:26:40.680324+0000","last_clean_scrub_stamp":"2024-03-11T21:41:30.511168+0000","objects_scrubbed":8,"log_size":38,"log_dups_size":0,"ondisk_log_size":38,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":51,"num_read_kb":39,"num_write":32,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":770,"reported_epoch":412,"state":"active+clean","last_fresh":"2024-03-11T21:43:26.944516+0000","last_change":"2024-03-11T21:40:21.518921+0000","last_active":"2024-03-11T21:43:26.944516+0000","last_peered":"2024-03-11T21:43:26.944516+0000","last_clean":"2024-03-11T21:43:26.944516+0000","last_became_active":"2024-03-11T21:36:35.057517+0000","last_became_peered":"2024-03-11T21:36:35.057517+0000","last_unstale":"2024-03-11T21:43:26.944516+0000","last_undegraded":"2024-03-11T21:43:26.944516+0000","last_fullsized":"2024-03-11T21:43:26.944516+0000","mapping_epoch":277,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":278,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:40:21.518840+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:40:21.518840+0000","last_clean_scrub_stamp":"2024-03-11T21:40:21.518840+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,7],"acting":[4,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"10.f","version":"412'10479","reported_seq":10605,"reported_epoch":412,"state":"active+clean","last_fresh":"2024-03-11T21:43:27.438955+0000","last_change":"2024-03-11T21:40:39.541796+0000","last_active":"2024-03-11T21:43:27.438955+0000","last_peered":"2024-03-11T21:43:27.438955+0000","last_clean":"2024-03-11T21:43:27.438955+0000","last_became_active":"2024-03-11T21:40:39.541420+0000","last_became_peered":"2024-03-11T21:40:39.541420+0000","last_unstale":"2024-03-11T21:43:27.438955+0000","last_undegraded":"2024-03-11T21:43:27.438955+0000","last_fullsized":"2024-03-11T21:43:27.438955+0000","mapping_epoch":356,"log_start":"378'4900","ondisk_log_start":"378'4900","created":356,"last_epoch_clean":357,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:40:38.219602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:40:38.219602+0000","last_clean_scrub_stamp":"2024-03-11T21:40:38.219602+0000","objects_scrubbed":0,"log_size":5579,"log_dups_size":0,"ondisk_log_size":5579,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":231014400,"num_objects":3525,"num_object_clones":0,"num_object_copies":10575,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3525,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10479,"num_write_kb":448128,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,2],"acting":[6,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.3","version":"0'0","reported_seq":401,"reported_epoch":412,"state":"active+clean","last_fresh":"2024-03-11T21:43:26.945667+0000","last_change":"2024-03-11T21:43:25.746794+0000","last_active":"2024-03-11T21:43:26.945667+0000","last_peered":"2024-03-11T21:43:26.945667+0000","last_clean":"2024-03-11T21:43:26.945667+0000","last_became_active":"2024-03-11T21:34:41.142683+0000","last_became_peered":"2024-03-11T21:34:41.142683+0000","last_unstale":"2024-03-11T21:43:26.945667+0000","last_undegraded":"2024-03-11T21:43:26.945667+0000","last_fullsized":"2024-03-11T21:43:26.945667+0000","mapping_epoch":226,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":227,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:43:25.746680+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:29:42.485740+0000","last_clean_scrub_stamp":"2024-03-11T21:43:25.746680+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:44:53.921851+0000","scrub_duration":116,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,2],"acting":[7,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":793,"reported_epoch":412,"state":"active+clean","last_fresh":"2024-03-11T21:43:26.955760+0000","last_change":"2024-03-11T21:41:22.817490+0000","last_active":"2024-03-11T21:43:26.955760+0000","last_peered":"2024-03-11T21:43:26.955760+0000","last_clean":"2024-03-11T21:43:26.955760+0000","last_became_active":"2024-03-11T21:34:41.145628+0000","last_became_peered":"2024-03-11T21:34:41.145628+0000","last_unstale":"2024-03-11T21:43:26.955760+0000","last_undegraded":"2024-03-11T21:43:26.955760+0000","last_fullsized":"2024-03-11T21:43:26.955760+0000","mapping_epoch":226,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":227,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:41:22.817380+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:37.512018+0000","last_clean_scrub_stamp":"2024-03-11T21:41:22.817380+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,2],"acting":[6,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.1","version":"260'28","reported_seq":795,"reported_epoch":412,"state":"active+clean","last_fresh":"2024-03-11T21:43:26.944171+0000","last_change":"2024-03-11T21:41:07.513056+0000","last_active":"2024-03-11T21:43:26.944171+0000","last_peered":"2024-03-11T21:43:26.944171+0000","last_clean":"2024-03-11T21:43:26.944171+0000","last_became_active":"2024-03-11T21:25:40.546219+0000","last_became_peered":"2024-03-11T21:25:40.546219+0000","last_unstale":"2024-03-11T21:43:26.944171+0000","last_undegraded":"2024-03-11T21:43:26.944171+0000","last_fullsized":"2024-03-11T21:43:26.944171+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"260'28","last_scrub_stamp":"2024-03-11T21:41:07.512907+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:39.525554+0000","last_clean_scrub_stamp":"2024-03-11T21:41:07.512907+0000","objects_scrubbed":7,"log_size":28,"log_dups_size":0,"ondisk_log_size":28,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":42,"num_read_kb":35,"num_write":28,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6],"acting":[4,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.0","version":"280'11","reported_seq":885,"reported_epoch":410,"state":"active+clean","last_fresh":"2024-03-11T21:43:23.939572+0000","last_change":"2024-03-11T21:36:35.064085+0000","last_active":"2024-03-11T21:43:23.939572+0000","last_peered":"2024-03-11T21:43:23.939572+0000","last_clean":"2024-03-11T21:43:23.939572+0000","last_became_active":"2024-03-11T21:36:35.055250+0000","last_became_peered":"2024-03-11T21:36:35.055250+0000","last_unstale":"2024-03-11T21:43:23.939572+0000","last_undegraded":"2024-03-11T21:43:23.939572+0000","last_fullsized":"2024-03-11T21:43:23.939572+0000","mapping_epoch":277,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":278,"parent":"0.0","parent_split_bits":0,"last_scrub":"98'9","last_scrub_stamp":"2024-03-11T21:35:21.143578+0000","last_deep_scrub":"98'9","last_deep_scrub_stamp":"2024-03-11T21:27:29.745711+0000","last_clean_scrub_stamp":"2024-03-11T21:35:21.143578+0000","objects_scrubbed":1,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.0","version":"412'7297","reported_seq":7458,"reported_epoch":412,"state":"active+clean","last_fresh":"2024-03-11T21:43:27.284210+0000","last_change":"2024-03-11T21:41:29.309475+0000","last_active":"2024-03-11T21:43:27.284210+0000","last_peered":"2024-03-11T21:43:27.284210+0000","last_clean":"2024-03-11T21:43:27.284210+0000","last_became_active":"2024-03-11T21:41:28.819914+0000","last_became_peered":"2024-03-11T21:41:28.819914+0000","last_unstale":"2024-03-11T21:43:27.284210+0000","last_undegraded":"2024-03-11T21:43:27.284210+0000","last_fullsized":"2024-03-11T21:43:27.284210+0000","mapping_epoch":356,"log_start":"0'0","ondisk_log_start":"0'0","created":356,"last_epoch_clean":373,"parent":"0.0","parent_split_bits":0,"last_scrub":"373'4340","last_scrub_stamp":"2024-03-11T21:41:29.309268+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:40:38.219602+0000","last_clean_scrub_stamp":"2024-03-11T21:41:29.309268+0000","objects_scrubbed":2123,"log_size":7297,"log_dups_size":0,"ondisk_log_size":7297,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":481,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112394240,"num_objects":1715,"num_object_clones":0,"num_object_copies":5145,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1715,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2964,"num_write_kb":81792,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,4],"acting":[6,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.c","version":"0'0","reported_seq":673,"reported_epoch":410,"state":"active+clean","last_fresh":"2024-03-11T21:43:23.944961+0000","last_change":"2024-03-11T21:40:30.659272+0000","last_active":"2024-03-11T21:43:23.944961+0000","last_peered":"2024-03-11T21:43:23.944961+0000","last_clean":"2024-03-11T21:43:23.944961+0000","last_became_active":"2024-03-11T21:36:35.073778+0000","last_became_peered":"2024-03-11T21:36:35.073778+0000","last_unstale":"2024-03-11T21:43:23.944961+0000","last_undegraded":"2024-03-11T21:43:23.944961+0000","last_fullsized":"2024-03-11T21:43:23.944961+0000","mapping_epoch":277,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":278,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:40:30.659160+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:43.559452+0000","last_clean_scrub_stamp":"2024-03-11T21:40:30.659160+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.f","version":"0'0","reported_seq":755,"reported_epoch":412,"state":"active+clean","last_fresh":"2024-03-11T21:43:26.939335+0000","last_change":"2024-03-11T21:43:24.002447+0000","last_active":"2024-03-11T21:43:26.939335+0000","last_peered":"2024-03-11T21:43:26.939335+0000","last_clean":"2024-03-11T21:43:26.939335+0000","last_became_active":"2024-03-11T21:25:42.575935+0000","last_became_peered":"2024-03-11T21:25:42.575935+0000","last_unstale":"2024-03-11T21:43:26.939335+0000","last_undegraded":"2024-03-11T21:43:26.939335+0000","last_fullsized":"2024-03-11T21:43:26.939335+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:43:24.002346+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:43:24.002346+0000","last_clean_scrub_stamp":"2024-03-11T21:43:24.002346+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:44:26.043970+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.8","version":"0'0","reported_seq":401,"reported_epoch":412,"state":"active+clean","last_fresh":"2024-03-11T21:43:26.960053+0000","last_change":"2024-03-11T21:41:22.253981+0000","last_active":"2024-03-11T21:43:26.960053+0000","last_peered":"2024-03-11T21:43:26.960053+0000","last_clean":"2024-03-11T21:43:26.960053+0000","last_became_active":"2024-03-11T21:34:41.144681+0000","last_became_peered":"2024-03-11T21:34:41.144681+0000","last_unstale":"2024-03-11T21:43:26.960053+0000","last_undegraded":"2024-03-11T21:43:26.960053+0000","last_fullsized":"2024-03-11T21:43:26.960053+0000","mapping_epoch":226,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":227,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:41:22.253866+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:37.512018+0000","last_clean_scrub_stamp":"2024-03-11T21:41:22.253866+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,1],"acting":[7,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.e","version":"260'24","reported_seq":788,"reported_epoch":412,"state":"active+clean","last_fresh":"2024-03-11T21:43:26.944898+0000","last_change":"2024-03-11T21:41:14.567374+0000","last_active":"2024-03-11T21:43:26.944898+0000","last_peered":"2024-03-11T21:43:26.944898+0000","last_clean":"2024-03-11T21:43:26.944898+0000","last_became_active":"2024-03-11T21:25:40.552178+0000","last_became_peered":"2024-03-11T21:25:40.552178+0000","last_unstale":"2024-03-11T21:43:26.944898+0000","last_undegraded":"2024-03-11T21:43:26.944898+0000","last_fullsized":"2024-03-11T21:43:26.944898+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"260'24","last_scrub_stamp":"2024-03-11T21:41:14.567244+0000","last_deep_scrub":"109'16","last_deep_scrub_stamp":"2024-03-11T21:28:24.739395+0000","last_clean_scrub_stamp":"2024-03-11T21:41:14.567244+0000","objects_scrubbed":5,"log_size":24,"log_dups_size":0,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":32,"num_read_kb":25,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"10.1","version":"412'7439","reported_seq":7602,"reported_epoch":412,"state":"active+clean","last_fresh":"2024-03-11T21:43:27.030808+0000","last_change":"2024-03-11T21:41:30.222240+0000","last_active":"2024-03-11T21:43:27.030808+0000","last_peered":"2024-03-11T21:43:27.030808+0000","last_clean":"2024-03-11T21:43:27.030808+0000","last_became_active":"2024-03-11T21:41:28.822620+0000","last_became_peered":"2024-03-11T21:41:28.822620+0000","last_unstale":"2024-03-11T21:43:27.030808+0000","last_undegraded":"2024-03-11T21:43:27.030808+0000","last_fullsized":"2024-03-11T21:43:27.030808+0000","mapping_epoch":356,"log_start":"365'1648","ondisk_log_start":"365'1648","created":356,"last_epoch_clean":373,"parent":"0.0","parent_split_bits":0,"last_scrub":"374'4413","last_scrub_stamp":"2024-03-11T21:41:30.221933+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:40:38.219602+0000","last_clean_scrub_stamp":"2024-03-11T21:41:30.221933+0000","objects_scrubbed":2282,"log_size":5791,"log_dups_size":0,"ondisk_log_size":5791,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":640,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":113442816,"num_objects":1731,"num_object_clones":0,"num_object_copies":5193,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1731,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3027,"num_write_kb":79232,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,0],"acting":[4,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.d","version":"0'0","reported_seq":743,"reported_epoch":412,"state":"active+clean","last_fresh":"2024-03-11T21:43:26.939075+0000","last_change":"2024-03-11T21:41:15.915172+0000","last_active":"2024-03-11T21:43:26.939075+0000","last_peered":"2024-03-11T21:43:26.939075+0000","last_clean":"2024-03-11T21:43:26.939075+0000","last_became_active":"2024-03-11T21:25:44.590333+0000","last_became_peered":"2024-03-11T21:25:44.590333+0000","last_unstale":"2024-03-11T21:43:26.939075+0000","last_undegraded":"2024-03-11T21:43:26.939075+0000","last_fullsized":"2024-03-11T21:43:26.939075+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:41:15.915041+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:35:37.182850+0000","last_clean_scrub_stamp":"2024-03-11T21:41:15.915041+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":117,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.e","version":"74'8","reported_seq":970,"reported_epoch":412,"state":"active+clean","last_fresh":"2024-03-11T21:43:26.945582+0000","last_change":"2024-03-11T21:41:16.574147+0000","last_active":"2024-03-11T21:43:26.945582+0000","last_peered":"2024-03-11T21:43:26.945582+0000","last_clean":"2024-03-11T21:43:26.945582+0000","last_became_active":"2024-03-11T21:25:42.576742+0000","last_became_peered":"2024-03-11T21:25:42.576742+0000","last_unstale":"2024-03-11T21:43:26.945582+0000","last_undegraded":"2024-03-11T21:43:26.945582+0000","last_fullsized":"2024-03-11T21:43:26.945582+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'8","last_scrub_stamp":"2024-03-11T21:41:16.573269+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:41.544550+0000","last_clean_scrub_stamp":"2024-03-11T21:41:16.573269+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,0],"acting":[4,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.9","version":"0'0","reported_seq":770,"reported_epoch":412,"state":"active+clean","last_fresh":"2024-03-11T21:43:26.939705+0000","last_change":"2024-03-11T21:40:29.915912+0000","last_active":"2024-03-11T21:43:26.939705+0000","last_peered":"2024-03-11T21:43:26.939705+0000","last_clean":"2024-03-11T21:43:26.939705+0000","last_became_active":"2024-03-11T21:36:35.065733+0000","last_became_peered":"2024-03-11T21:36:35.065733+0000","last_unstale":"2024-03-11T21:43:26.939705+0000","last_undegraded":"2024-03-11T21:43:26.939705+0000","last_fullsized":"2024-03-11T21:43:26.939705+0000","mapping_epoch":277,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":278,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:40:29.915835+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:26:50.991712+0000","last_clean_scrub_stamp":"2024-03-11T21:40:29.915835+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,3],"acting":[1,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.f","version":"260'27","reported_seq":778,"reported_epoch":412,"state":"active+clean","last_fresh":"2024-03-11T21:43:26.967289+0000","last_change":"2024-03-11T21:41:31.414618+0000","last_active":"2024-03-11T21:43:26.967289+0000","last_peered":"2024-03-11T21:43:26.967289+0000","last_clean":"2024-03-11T21:43:26.967289+0000","last_became_active":"2024-03-11T21:36:35.058272+0000","last_became_peered":"2024-03-11T21:36:35.058272+0000","last_unstale":"2024-03-11T21:43:26.967289+0000","last_undegraded":"2024-03-11T21:43:26.967289+0000","last_fullsized":"2024-03-11T21:43:26.967289+0000","mapping_epoch":277,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":278,"parent":"0.0","parent_split_bits":0,"last_scrub":"260'27","last_scrub_stamp":"2024-03-11T21:41:31.414510+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:39.525554+0000","last_clean_scrub_stamp":"2024-03-11T21:41:31.414510+0000","objects_scrubbed":7,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":44,"num_read_kb":35,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,4],"acting":[2,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"10.2","version":"412'7462","reported_seq":7622,"reported_epoch":412,"state":"active+clean","last_fresh":"2024-03-11T21:43:28.410541+0000","last_change":"2024-03-11T21:41:29.850911+0000","last_active":"2024-03-11T21:43:28.410541+0000","last_peered":"2024-03-11T21:43:28.410541+0000","last_clean":"2024-03-11T21:43:28.410541+0000","last_became_active":"2024-03-11T21:41:28.834067+0000","last_became_peered":"2024-03-11T21:41:28.834067+0000","last_unstale":"2024-03-11T21:43:28.410541+0000","last_undegraded":"2024-03-11T21:43:28.410541+0000","last_fullsized":"2024-03-11T21:43:28.410541+0000","mapping_epoch":356,"log_start":"364'558","ondisk_log_start":"364'558","created":356,"last_epoch_clean":373,"parent":"0.0","parent_split_bits":0,"last_scrub":"373'4444","last_scrub_stamp":"2024-03-11T21:41:29.850283+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:40:38.219602+0000","last_clean_scrub_stamp":"2024-03-11T21:41:29.850283+0000","objects_scrubbed":2169,"log_size":6904,"log_dups_size":0,"ondisk_log_size":6904,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":620,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112197632,"num_objects":1712,"num_object_clones":0,"num_object_copies":5136,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1712,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3025,"num_write_kb":82176,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.e","version":"0'0","reported_seq":744,"reported_epoch":412,"state":"active+clean","last_fresh":"2024-03-11T21:43:26.945720+0000","last_change":"2024-03-11T21:41:20.548370+0000","last_active":"2024-03-11T21:43:26.945720+0000","last_peered":"2024-03-11T21:43:26.945720+0000","last_clean":"2024-03-11T21:43:26.945720+0000","last_became_active":"2024-03-11T21:25:44.590198+0000","last_became_peered":"2024-03-11T21:25:44.590198+0000","last_unstale":"2024-03-11T21:43:26.945720+0000","last_undegraded":"2024-03-11T21:43:26.945720+0000","last_fullsized":"2024-03-11T21:43:26.945720+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:41:20.548259+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:43.559452+0000","last_clean_scrub_stamp":"2024-03-11T21:41:20.548259+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":42,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.d","version":"109'9","reported_seq":969,"reported_epoch":412,"state":"active+clean","last_fresh":"2024-03-11T21:43:26.965848+0000","last_change":"2024-03-11T21:41:06.117858+0000","last_active":"2024-03-11T21:43:26.965848+0000","last_peered":"2024-03-11T21:43:26.965848+0000","last_clean":"2024-03-11T21:43:26.965848+0000","last_became_active":"2024-03-11T21:25:42.575583+0000","last_became_peered":"2024-03-11T21:25:42.575583+0000","last_unstale":"2024-03-11T21:43:26.965848+0000","last_undegraded":"2024-03-11T21:43:26.965848+0000","last_fullsized":"2024-03-11T21:43:26.965848+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"109'9","last_scrub_stamp":"2024-03-11T21:41:06.117483+0000","last_deep_scrub":"109'9","last_deep_scrub_stamp":"2024-03-11T21:28:27.082975+0000","last_clean_scrub_stamp":"2024-03-11T21:41:06.117483+0000","objects_scrubbed":1,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,5],"acting":[2,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.a","version":"0'0","reported_seq":778,"reported_epoch":412,"state":"active+clean","last_fresh":"2024-03-11T21:43:26.940307+0000","last_change":"2024-03-11T21:41:02.132633+0000","last_active":"2024-03-11T21:43:26.940307+0000","last_peered":"2024-03-11T21:43:26.940307+0000","last_clean":"2024-03-11T21:43:26.940307+0000","last_became_active":"2024-03-11T21:36:35.066832+0000","last_became_peered":"2024-03-11T21:36:35.066832+0000","last_unstale":"2024-03-11T21:43:26.940307+0000","last_undegraded":"2024-03-11T21:43:26.940307+0000","last_fullsized":"2024-03-11T21:43:26.940307+0000","mapping_epoch":277,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":278,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:41:02.132355+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:37.512018+0000","last_clean_scrub_stamp":"2024-03-11T21:41:02.132355+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,7],"acting":[1,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.c","version":"260'20","reported_seq":804,"reported_epoch":412,"state":"active+clean","last_fresh":"2024-03-11T21:43:26.944122+0000","last_change":"2024-03-11T21:41:30.618505+0000","last_active":"2024-03-11T21:43:26.944122+0000","last_peered":"2024-03-11T21:43:26.944122+0000","last_clean":"2024-03-11T21:43:26.944122+0000","last_became_active":"2024-03-11T21:36:35.067269+0000","last_became_peered":"2024-03-11T21:36:35.067269+0000","last_unstale":"2024-03-11T21:43:26.944122+0000","last_undegraded":"2024-03-11T21:43:26.944122+0000","last_fullsized":"2024-03-11T21:43:26.944122+0000","mapping_epoch":277,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":278,"parent":"0.0","parent_split_bits":0,"last_scrub":"260'20","last_scrub_stamp":"2024-03-11T21:41:30.618393+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:39.525554+0000","last_clean_scrub_stamp":"2024-03-11T21:41:30.618393+0000","objects_scrubbed":5,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":25,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"10.3","version":"412'7307","reported_seq":7471,"reported_epoch":412,"state":"active+clean","last_fresh":"2024-03-11T21:43:28.169537+0000","last_change":"2024-03-11T21:41:30.759270+0000","last_active":"2024-03-11T21:43:28.169537+0000","last_peered":"2024-03-11T21:43:28.169537+0000","last_clean":"2024-03-11T21:43:28.169537+0000","last_became_active":"2024-03-11T21:41:28.840374+0000","last_became_peered":"2024-03-11T21:41:28.840374+0000","last_unstale":"2024-03-11T21:43:28.169537+0000","last_undegraded":"2024-03-11T21:43:28.169537+0000","last_fullsized":"2024-03-11T21:43:28.169537+0000","mapping_epoch":356,"log_start":"0'0","ondisk_log_start":"0'0","created":356,"last_epoch_clean":373,"parent":"0.0","parent_split_bits":0,"last_scrub":"373'4357","last_scrub_stamp":"2024-03-11T21:41:30.759188+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:40:38.219602+0000","last_clean_scrub_stamp":"2024-03-11T21:41:30.759188+0000","objects_scrubbed":2208,"log_size":7307,"log_dups_size":0,"ondisk_log_size":7307,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":314,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":106168320,"num_objects":1620,"num_object_clones":0,"num_object_copies":4860,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1620,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2950,"num_write_kb":75584,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,3],"acting":[2,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.f","version":"0'0","reported_seq":751,"reported_epoch":412,"state":"active+clean","last_fresh":"2024-03-11T21:43:26.967188+0000","last_change":"2024-03-11T21:41:17.377926+0000","last_active":"2024-03-11T21:43:26.967188+0000","last_peered":"2024-03-11T21:43:26.967188+0000","last_clean":"2024-03-11T21:43:26.967188+0000","last_became_active":"2024-03-11T21:36:35.057800+0000","last_became_peered":"2024-03-11T21:36:35.057800+0000","last_unstale":"2024-03-11T21:43:26.967188+0000","last_undegraded":"2024-03-11T21:43:26.967188+0000","last_fullsized":"2024-03-11T21:43:26.967188+0000","mapping_epoch":277,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":278,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:41:17.377779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:27:03.010969+0000","last_clean_scrub_stamp":"2024-03-11T21:41:17.377779+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":149,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,4],"acting":[2,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":748,"reported_epoch":412,"state":"active+clean","last_fresh":"2024-03-11T21:43:26.941406+0000","last_change":"2024-03-11T21:41:31.221210+0000","last_active":"2024-03-11T21:43:26.941406+0000","last_peered":"2024-03-11T21:43:26.941406+0000","last_clean":"2024-03-11T21:43:26.941406+0000","last_became_active":"2024-03-11T21:25:42.579409+0000","last_became_peered":"2024-03-11T21:25:42.579409+0000","last_unstale":"2024-03-11T21:43:26.941406+0000","last_undegraded":"2024-03-11T21:43:26.941406+0000","last_fullsized":"2024-03-11T21:43:26.941406+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:41:31.221094+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:30:22.750053+0000","last_clean_scrub_stamp":"2024-03-11T21:41:31.221094+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.b","version":"0'0","reported_seq":400,"reported_epoch":412,"state":"active+clean","last_fresh":"2024-03-11T21:43:26.945758+0000","last_change":"2024-03-11T21:41:29.255398+0000","last_active":"2024-03-11T21:43:26.945758+0000","last_peered":"2024-03-11T21:43:26.945758+0000","last_clean":"2024-03-11T21:43:26.945758+0000","last_became_active":"2024-03-11T21:34:41.134987+0000","last_became_peered":"2024-03-11T21:34:41.134987+0000","last_unstale":"2024-03-11T21:43:26.945758+0000","last_undegraded":"2024-03-11T21:43:26.945758+0000","last_fullsized":"2024-03-11T21:43:26.945758+0000","mapping_epoch":226,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":227,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:41:29.255282+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:36:16.719309+0000","last_clean_scrub_stamp":"2024-03-11T21:41:29.255282+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,5],"acting":[7,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.d","version":"260'39","reported_seq":809,"reported_epoch":412,"state":"active+clean","last_fresh":"2024-03-11T21:43:26.944320+0000","last_change":"2024-03-11T21:41:11.544772+0000","last_active":"2024-03-11T21:43:26.944320+0000","last_peered":"2024-03-11T21:43:26.944320+0000","last_clean":"2024-03-11T21:43:26.944320+0000","last_became_active":"2024-03-11T21:25:40.553454+0000","last_became_peered":"2024-03-11T21:25:40.553454+0000","last_unstale":"2024-03-11T21:43:26.944320+0000","last_undegraded":"2024-03-11T21:43:26.944320+0000","last_fullsized":"2024-03-11T21:43:26.944320+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"260'39","last_scrub_stamp":"2024-03-11T21:41:11.544605+0000","last_deep_scrub":"165'25","last_deep_scrub_stamp":"2024-03-11T21:32:21.678328+0000","last_clean_scrub_stamp":"2024-03-11T21:41:11.544605+0000","objects_scrubbed":8,"log_size":39,"log_dups_size":0,"ondisk_log_size":39,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":50,"num_read_kb":40,"num_write":32,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,1],"acting":[4,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"10.4","version":"410'7004","reported_seq":7159,"reported_epoch":410,"state":"active+clean","last_fresh":"2024-03-11T21:43:24.507018+0000","last_change":"2024-03-11T21:41:30.436753+0000","last_active":"2024-03-11T21:43:24.507018+0000","last_peered":"2024-03-11T21:43:24.507018+0000","last_clean":"2024-03-11T21:43:24.507018+0000","last_became_active":"2024-03-11T21:41:28.837095+0000","last_became_peered":"2024-03-11T21:41:28.837095+0000","last_unstale":"2024-03-11T21:43:24.507018+0000","last_undegraded":"2024-03-11T21:43:24.507018+0000","last_fullsized":"2024-03-11T21:43:24.507018+0000","mapping_epoch":356,"log_start":"364'628","ondisk_log_start":"364'628","created":356,"last_epoch_clean":373,"parent":"0.0","parent_split_bits":0,"last_scrub":"374'4241","last_scrub_stamp":"2024-03-11T21:41:30.436548+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:40:38.219602+0000","last_clean_scrub_stamp":"2024-03-11T21:41:30.436548+0000","objects_scrubbed":2083,"log_size":6376,"log_dups_size":0,"ondisk_log_size":6376,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":281,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112721920,"num_objects":1720,"num_object_clones":0,"num_object_copies":5160,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1720,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2763,"num_write_kb":76800,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":401,"reported_epoch":412,"state":"active+clean","last_fresh":"2024-03-11T21:43:26.959830+0000","last_change":"2024-03-11T21:41:26.263600+0000","last_active":"2024-03-11T21:43:26.959830+0000","last_peered":"2024-03-11T21:43:26.959830+0000","last_clean":"2024-03-11T21:43:26.959830+0000","last_became_active":"2024-03-11T21:34:41.145012+0000","last_became_peered":"2024-03-11T21:34:41.145012+0000","last_unstale":"2024-03-11T21:43:26.959830+0000","last_undegraded":"2024-03-11T21:43:26.959830+0000","last_fullsized":"2024-03-11T21:43:26.959830+0000","mapping_epoch":226,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":227,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:41:26.263517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:28:37.561826+0000","last_clean_scrub_stamp":"2024-03-11T21:41:26.263517+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":48,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,0],"acting":[7,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":746,"reported_epoch":412,"state":"active+clean","last_fresh":"2024-03-11T21:43:26.966385+0000","last_change":"2024-03-11T21:41:09.181704+0000","last_active":"2024-03-11T21:43:26.966385+0000","last_peered":"2024-03-11T21:43:26.966385+0000","last_clean":"2024-03-11T21:43:26.966385+0000","last_became_active":"2024-03-11T21:25:42.574515+0000","last_became_peered":"2024-03-11T21:25:42.574515+0000","last_unstale":"2024-03-11T21:43:26.966385+0000","last_undegraded":"2024-03-11T21:43:26.966385+0000","last_fullsized":"2024-03-11T21:43:26.966385+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:41:09.181567+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:41.544550+0000","last_clean_scrub_stamp":"2024-03-11T21:41:09.181567+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,5],"acting":[2,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.c","version":"0'0","reported_seq":754,"reported_epoch":412,"state":"active+clean","last_fresh":"2024-03-11T21:43:26.965768+0000","last_change":"2024-03-11T21:41:24.303833+0000","last_active":"2024-03-11T21:43:26.965768+0000","last_peered":"2024-03-11T21:43:26.965768+0000","last_clean":"2024-03-11T21:43:26.965768+0000","last_became_active":"2024-03-11T21:25:38.533240+0000","last_became_peered":"2024-03-11T21:25:38.533240+0000","last_unstale":"2024-03-11T21:43:26.965768+0000","last_undegraded":"2024-03-11T21:43:26.965768+0000","last_fullsized":"2024-03-11T21:43:26.965768+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:41:24.303681+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:29:30.279910+0000","last_clean_scrub_stamp":"2024-03-11T21:41:24.303681+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,0],"acting":[2,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.a","version":"260'44","reported_seq":885,"reported_epoch":412,"state":"active+clean","last_fresh":"2024-03-11T21:43:26.955048+0000","last_change":"2024-03-11T21:40:02.183543+0000","last_active":"2024-03-11T21:43:26.955048+0000","last_peered":"2024-03-11T21:43:26.955048+0000","last_clean":"2024-03-11T21:43:26.955048+0000","last_became_active":"2024-03-11T21:34:41.148863+0000","last_became_peered":"2024-03-11T21:34:41.148863+0000","last_unstale":"2024-03-11T21:43:26.955048+0000","last_undegraded":"2024-03-11T21:43:26.955048+0000","last_fullsized":"2024-03-11T21:43:26.955048+0000","mapping_epoch":226,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":227,"parent":"0.0","parent_split_bits":0,"last_scrub":"260'44","last_scrub_stamp":"2024-03-11T21:40:02.183417+0000","last_deep_scrub":"109'28","last_deep_scrub_stamp":"2024-03-11T21:28:24.016389+0000","last_clean_scrub_stamp":"2024-03-11T21:40:02.183417+0000","objects_scrubbed":9,"log_size":44,"log_dups_size":0,"ondisk_log_size":44,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":56,"num_read_kb":45,"num_write":36,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,7],"acting":[6,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"10.5","version":"412'8276","reported_seq":8504,"reported_epoch":412,"state":"active+clean","last_fresh":"2024-03-11T21:43:28.181607+0000","last_change":"2024-03-11T21:41:32.489861+0000","last_active":"2024-03-11T21:43:28.181607+0000","last_peered":"2024-03-11T21:43:28.181607+0000","last_clean":"2024-03-11T21:43:28.181607+0000","last_became_active":"2024-03-11T21:41:28.844869+0000","last_became_peered":"2024-03-11T21:41:28.844869+0000","last_unstale":"2024-03-11T21:43:28.181607+0000","last_undegraded":"2024-03-11T21:43:28.181607+0000","last_fullsized":"2024-03-11T21:43:28.181607+0000","mapping_epoch":356,"log_start":"364'404","ondisk_log_start":"364'404","created":356,"last_epoch_clean":373,"parent":"0.0","parent_split_bits":0,"last_scrub":"376'5260","last_scrub_stamp":"2024-03-11T21:41:32.489702+0000","last_deep_scrub":"376'5260","last_deep_scrub_stamp":"2024-03-11T21:41:32.489702+0000","last_clean_scrub_stamp":"2024-03-11T21:41:32.489702+0000","objects_scrubbed":2205,"log_size":7872,"log_dups_size":0,"ondisk_log_size":7872,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":4,"scrub_schedule":"queued for scrub","scrub_duration":3023,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112590848,"num_objects":1718,"num_object_clones":0,"num_object_copies":5154,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1718,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3019,"num_write_kb":81024,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3],"acting":[2,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":753,"reported_epoch":410,"state":"active+clean","last_fresh":"2024-03-11T21:43:23.934492+0000","last_change":"2024-03-11T21:41:23.288877+0000","last_active":"2024-03-11T21:43:23.934492+0000","last_peered":"2024-03-11T21:43:23.934492+0000","last_clean":"2024-03-11T21:43:23.934492+0000","last_became_active":"2024-03-11T21:34:41.154507+0000","last_became_peered":"2024-03-11T21:34:41.154507+0000","last_unstale":"2024-03-11T21:43:23.934492+0000","last_undegraded":"2024-03-11T21:43:23.934492+0000","last_fullsized":"2024-03-11T21:43:23.934492+0000","mapping_epoch":226,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":227,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:41:23.288651+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:27:11.361913+0000","last_clean_scrub_stamp":"2024-03-11T21:41:23.288651+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":136,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":748,"reported_epoch":412,"state":"active+clean","last_fresh":"2024-03-11T21:43:26.966092+0000","last_change":"2024-03-11T21:40:28.015350+0000","last_active":"2024-03-11T21:43:26.966092+0000","last_peered":"2024-03-11T21:43:26.966092+0000","last_clean":"2024-03-11T21:43:26.966092+0000","last_became_active":"2024-03-11T21:36:35.059192+0000","last_became_peered":"2024-03-11T21:36:35.059192+0000","last_unstale":"2024-03-11T21:43:26.966092+0000","last_undegraded":"2024-03-11T21:43:26.966092+0000","last_fullsized":"2024-03-11T21:43:26.966092+0000","mapping_epoch":277,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":278,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:40:28.015231+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:41.544550+0000","last_clean_scrub_stamp":"2024-03-11T21:40:28.015231+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,3],"acting":[2,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.d","version":"0'0","reported_seq":757,"reported_epoch":412,"state":"active+clean","last_fresh":"2024-03-11T21:43:26.941425+0000","last_change":"2024-03-11T21:40:31.873781+0000","last_active":"2024-03-11T21:43:26.941425+0000","last_peered":"2024-03-11T21:43:26.941425+0000","last_clean":"2024-03-11T21:43:26.941425+0000","last_became_active":"2024-03-11T21:36:35.058296+0000","last_became_peered":"2024-03-11T21:36:35.058296+0000","last_unstale":"2024-03-11T21:43:26.941425+0000","last_undegraded":"2024-03-11T21:43:26.941425+0000","last_fullsized":"2024-03-11T21:43:26.941425+0000","mapping_epoch":277,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":278,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:40:31.873695+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:37.512018+0000","last_clean_scrub_stamp":"2024-03-11T21:40:31.873695+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,3],"acting":[1,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.b","version":"260'15","reported_seq":773,"reported_epoch":410,"state":"active+clean","last_fresh":"2024-03-11T21:43:23.936387+0000","last_change":"2024-03-11T21:43:20.043522+0000","last_active":"2024-03-11T21:43:23.936387+0000","last_peered":"2024-03-11T21:43:23.936387+0000","last_clean":"2024-03-11T21:43:23.936387+0000","last_became_active":"2024-03-11T21:25:40.549612+0000","last_became_peered":"2024-03-11T21:25:40.549612+0000","last_unstale":"2024-03-11T21:43:23.936387+0000","last_undegraded":"2024-03-11T21:43:23.936387+0000","last_fullsized":"2024-03-11T21:43:23.936387+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"260'15","last_scrub_stamp":"2024-03-11T21:43:20.043394+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:39.525554+0000","last_clean_scrub_stamp":"2024-03-11T21:43:20.043394+0000","objects_scrubbed":4,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:44:35.981062+0000","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.6","version":"412'7307","reported_seq":7467,"reported_epoch":412,"state":"active+clean","last_fresh":"2024-03-11T21:43:27.015558+0000","last_change":"2024-03-11T21:41:30.231854+0000","last_active":"2024-03-11T21:43:27.015558+0000","last_peered":"2024-03-11T21:43:27.015558+0000","last_clean":"2024-03-11T21:43:27.015558+0000","last_became_active":"2024-03-11T21:41:28.841669+0000","last_became_peered":"2024-03-11T21:41:28.841669+0000","last_unstale":"2024-03-11T21:43:27.015558+0000","last_undegraded":"2024-03-11T21:43:27.015558+0000","last_fullsized":"2024-03-11T21:43:27.015558+0000","mapping_epoch":356,"log_start":"364'158","ondisk_log_start":"364'158","created":356,"last_epoch_clean":373,"parent":"0.0","parent_split_bits":0,"last_scrub":"374'4344","last_scrub_stamp":"2024-03-11T21:41:30.231749+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:40:38.219602+0000","last_clean_scrub_stamp":"2024-03-11T21:41:30.231749+0000","objects_scrubbed":2205,"log_size":7149,"log_dups_size":0,"ondisk_log_size":7149,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":259,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":115736576,"num_objects":1766,"num_object_clones":0,"num_object_copies":5298,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1766,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2963,"num_write_kb":80768,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,4],"acting":[5,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":658,"reported_epoch":412,"state":"active+clean","last_fresh":"2024-03-11T21:43:26.943829+0000","last_change":"2024-03-11T21:41:12.624195+0000","last_active":"2024-03-11T21:43:26.943829+0000","last_peered":"2024-03-11T21:43:26.943829+0000","last_clean":"2024-03-11T21:43:26.943829+0000","last_became_active":"2024-03-11T21:36:35.056518+0000","last_became_peered":"2024-03-11T21:36:35.056518+0000","last_unstale":"2024-03-11T21:43:26.943829+0000","last_undegraded":"2024-03-11T21:43:26.943829+0000","last_fullsized":"2024-03-11T21:43:26.943829+0000","mapping_epoch":277,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":278,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:41:12.624073+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:43.559452+0000","last_clean_scrub_stamp":"2024-03-11T21:41:12.624073+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":41,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,3],"acting":[4,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.e","version":"0'0","reported_seq":783,"reported_epoch":412,"state":"active+clean","last_fresh":"2024-03-11T21:43:26.965698+0000","last_change":"2024-03-11T21:41:27.388270+0000","last_active":"2024-03-11T21:43:26.965698+0000","last_peered":"2024-03-11T21:43:26.965698+0000","last_clean":"2024-03-11T21:43:26.965698+0000","last_became_active":"2024-03-11T21:36:35.056777+0000","last_became_peered":"2024-03-11T21:36:35.056777+0000","last_unstale":"2024-03-11T21:43:26.965698+0000","last_undegraded":"2024-03-11T21:43:26.965698+0000","last_fullsized":"2024-03-11T21:43:26.965698+0000","mapping_epoch":277,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":278,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:41:27.387998+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:37.512018+0000","last_clean_scrub_stamp":"2024-03-11T21:41:27.387998+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,7],"acting":[2,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.8","version":"260'34","reported_seq":846,"reported_epoch":412,"state":"active+clean","last_fresh":"2024-03-11T21:43:26.938262+0000","last_change":"2024-03-11T21:41:06.881563+0000","last_active":"2024-03-11T21:43:26.938262+0000","last_peered":"2024-03-11T21:43:26.938262+0000","last_clean":"2024-03-11T21:43:26.938262+0000","last_became_active":"2024-03-11T21:34:41.146631+0000","last_became_peered":"2024-03-11T21:34:41.146631+0000","last_unstale":"2024-03-11T21:43:26.938262+0000","last_undegraded":"2024-03-11T21:43:26.938262+0000","last_fullsized":"2024-03-11T21:43:26.938262+0000","mapping_epoch":226,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":227,"parent":"0.0","parent_split_bits":0,"last_scrub":"260'34","last_scrub_stamp":"2024-03-11T21:41:06.881414+0000","last_deep_scrub":"109'22","last_deep_scrub_stamp":"2024-03-11T21:32:07.573197+0000","last_clean_scrub_stamp":"2024-03-11T21:41:06.881414+0000","objects_scrubbed":7,"log_size":34,"log_dups_size":0,"ondisk_log_size":34,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":44,"num_read_kb":35,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.9","version":"229'11","reported_seq":510,"reported_epoch":412,"state":"active+clean","last_fresh":"2024-03-11T21:43:26.946466+0000","last_change":"2024-03-11T21:41:28.240859+0000","last_active":"2024-03-11T21:43:26.946466+0000","last_peered":"2024-03-11T21:43:26.946466+0000","last_clean":"2024-03-11T21:43:26.946466+0000","last_became_active":"2024-03-11T21:34:41.156892+0000","last_became_peered":"2024-03-11T21:34:41.156892+0000","last_unstale":"2024-03-11T21:43:26.946466+0000","last_undegraded":"2024-03-11T21:43:26.946466+0000","last_fullsized":"2024-03-11T21:43:26.946466+0000","mapping_epoch":226,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":227,"parent":"0.0","parent_split_bits":0,"last_scrub":"229'11","last_scrub_stamp":"2024-03-11T21:41:28.240763+0000","last_deep_scrub":"94'9","last_deep_scrub_stamp":"2024-03-11T21:26:53.422552+0000","last_clean_scrub_stamp":"2024-03-11T21:41:28.240763+0000","objects_scrubbed":1,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":10,"num_pg":26,"stat_sum":{"num_bytes":3563225112,"num_objects":54373,"num_object_clones":0,"num_object_copies":163119,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":54373,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":128213,"num_write_kb":4970337,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15656,"num_bytes_recovered":1026031616,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12349747200,"data_stored":12349734984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":166566,"ondisk_log_size":166566,"up":78,"acting":78,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":1531,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":17,"num_read_kb":17,"num_write":10,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":1308,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":4593,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":6,"ondisk_log_size":6,"up":96,"acting":96,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":198,"num_read_kb":294,"num_write":179,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":2951840,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":2375680,"data_stored":2361472,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":128,"ondisk_log_size":128,"up":3,"acting":3,"num_store_stats":4},{"poolid":3,"num_pg":3,"stat_sum":{"num_bytes":408,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1061,"num_read_kb":1056,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1224,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":8,"ondisk_log_size":8,"up":9,"acting":9,"num_store_stats":7},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":3702,"num_objects":178,"num_object_clones":0,"num_object_copies":534,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":178,"num_whiteouts":0,"num_read":1120,"num_read_kb":877,"num_write":703,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":186,"num_bytes_recovered":2200,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":417792,"data_stored":11106,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":774,"ondisk_log_size":774,"up":96,"acting":96,"num_store_stats":8},{"poolid":5,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":78,"ondisk_log_size":78,"up":96,"acting":96,"num_store_stats":8},{"poolid":6,"num_pg":32,"stat_sum":{"num_bytes":383,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":370,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1149,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":96,"acting":96,"num_store_stats":8}],"osd_stats":[{"osd":7,"up_from":60,"seq":257698037977,"num_pgs":65,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1728744,"kb_used_data":1147624,"kb_used_omap":1,"kb_used_meta":580926,"kb_avail":92016408,"statfs":{"total":95995035648,"available":94224801792,"internally_reserved":0,"allocated":1175166976,"data_stored":1174749275,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":594868682},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,1,2],"upper_bound":16},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":50,"seq":214748365023,"num_pgs":54,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2317572,"kb_used_data":1736456,"kb_used_omap":1,"kb_used_meta":580926,"kb_avail":91427580,"statfs":{"total":95995035648,"available":93621841920,"internally_reserved":0,"allocated":1778130944,"data_stored":1777725460,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":594868678},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,2,2,2,3,2],"upper_bound":64},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834575097,"num_pgs":61,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1907044,"kb_used_data":1212252,"kb_used_omap":1,"kb_used_meta":694782,"kb_avail":91838108,"statfs":{"total":95995035648,"available":94042222592,"internally_reserved":0,"allocated":1241346048,"data_stored":1240919514,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":711457221},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738621,"num_pgs":66,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":3003232,"kb_used_data":2313824,"kb_used_omap":1,"kb_used_meta":689214,"kb_avail":90741920,"statfs":{"total":95995035648,"available":92919726080,"internally_reserved":0,"allocated":2369355776,"data_stored":2368850891,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":705755589},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411571,"num_pgs":52,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2147768,"kb_used_data":1530620,"kb_used_omap":1,"kb_used_meta":617022,"kb_avail":91597384,"statfs":{"total":95995035648,"available":93795721216,"internally_reserved":0,"allocated":1567354880,"data_stored":1566960884,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":631830981},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,0,2,5,4],"upper_bound":64},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":3,"up_from":277,"seq":1189705941075,"num_pgs":63,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1550804,"kb_used_data":1161556,"kb_used_omap":4,"kb_used_meta":389051,"kb_avail":92194348,"statfs":{"total":95995035648,"available":94407012352,"internally_reserved":0,"allocated":1189433344,"data_stored":1188844849,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4442,"internal_metadata":398388902},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,1,3,1],"upper_bound":32},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":4,"up_from":32,"seq":137438953704,"num_pgs":67,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2403896,"kb_used_data":1804728,"kb_used_omap":1,"kb_used_meta":598974,"kb_avail":91341256,"statfs":{"total":95995035648,"available":93533446144,"internally_reserved":0,"allocated":1848041472,"data_stored":1847607212,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":613349829},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":5,"up_from":41,"seq":176093659363,"num_pgs":59,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1917984,"kb_used_data":1246624,"kb_used_omap":1,"kb_used_meta":671166,"kb_avail":91827168,"statfs":{"total":95995035648,"available":94031020032,"internally_reserved":0,"allocated":1276542976,"data_stored":1276133035,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":687274437},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,2,1,5],"upper_bound":32},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1049,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":46,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":436,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":957,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":574,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":436,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1003,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":92,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":1430,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":77824,"data_stored":2008,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":1018,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":1898,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":1474,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":990,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":1034,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":1254,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":370,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":383,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":370,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":2348683264,"data_stored":2348679192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":1236008960,"data_stored":1236008960,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":1565655040,"data_stored":1565655040,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":1170997248,"data_stored":1170997248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":1842679808,"data_stored":1842675736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":1248395264,"data_stored":1248395264,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":1775898624,"data_stored":1775894552,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":1161428992,"data_stored":1161428992,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-11T21:43:31.947 INFO:tasks.cephadm.ceph_manager.ceph:would drop below min_pgs, proj -9, currently 1 2024-03-11T21:43:31.989 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:43:32.028 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:43:32.029 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:43:32.030 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:43:32.130 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:43:32.173 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:43:32.173 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:43:32.174 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:43:32.274 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:43:32.307 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:43:32.308 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:43:32.308 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:43:32.332 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:43:31 smithi097 ceph-mon[33317]: from='client.25687 v1:172.21.15.92:0/1229426964' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-11T21:43:32.332 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:43:31 smithi097 ceph-mon[33317]: 10.18 scrub ok 2024-03-11T21:43:32.332 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:43:31 smithi097 ceph-mon[33317]: 2.0 deep-scrub starts 2024-03-11T21:43:32.332 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:43:31 smithi097 ceph-mon[33317]: 2.0 deep-scrub ok 2024-03-11T21:43:32.332 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:43:31 smithi097 ceph-mon[33317]: osdmap e413: 8 total, 8 up, 8 in 2024-03-11T21:43:32.332 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:43:31 smithi097 ceph-mon[33317]: 4.1c scrub starts 2024-03-11T21:43:32.332 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:43:31 smithi097 ceph-mon[33317]: 4.1c scrub ok 2024-03-11T21:43:32.366 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:43:31 smithi092 ceph-mon[29572]: from='client.25687 v1:172.21.15.92:0/1229426964' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-11T21:43:32.366 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:43:31 smithi092 ceph-mon[29572]: 10.18 scrub ok 2024-03-11T21:43:32.367 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:43:31 smithi092 ceph-mon[29572]: 2.0 deep-scrub starts 2024-03-11T21:43:32.367 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:43:31 smithi092 ceph-mon[29572]: 2.0 deep-scrub ok 2024-03-11T21:43:32.367 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:43:31 smithi092 ceph-mon[29572]: osdmap e413: 8 total, 8 up, 8 in 2024-03-11T21:43:32.367 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:43:31 smithi092 ceph-mon[29572]: 4.1c scrub starts 2024-03-11T21:43:32.367 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:43:31 smithi092 ceph-mon[29572]: 4.1c scrub ok 2024-03-11T21:43:32.367 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:43:31 smithi092 ceph-mon[37563]: from='client.25687 v1:172.21.15.92:0/1229426964' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-11T21:43:32.367 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:43:31 smithi092 ceph-mon[37563]: 10.18 scrub ok 2024-03-11T21:43:32.367 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:43:31 smithi092 ceph-mon[37563]: 2.0 deep-scrub starts 2024-03-11T21:43:32.367 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:43:31 smithi092 ceph-mon[37563]: 2.0 deep-scrub ok 2024-03-11T21:43:32.367 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:43:31 smithi092 ceph-mon[37563]: osdmap e413: 8 total, 8 up, 8 in 2024-03-11T21:43:32.367 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:43:31 smithi092 ceph-mon[37563]: 4.1c scrub starts 2024-03-11T21:43:32.367 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:43:31 smithi092 ceph-mon[37563]: 4.1c scrub ok 2024-03-11T21:43:32.409 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:43:32.446 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:43:32.447 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:43:32.447 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:43:32.548 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:43:32.585 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:43:32.585 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:43:32.586 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:43:32.686 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:43:32.722 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:43:32.723 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:43:32.723 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:43:32.824 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:43:32.860 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:43:32.860 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:43:32.861 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:43:32.961 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:43:32.992 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:43:32.993 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:43:32.993 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:43:33.094 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:43:33.136 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:43:33.137 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:43:33.137 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:43:33.237 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:43:33.271 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:43:33.272 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:43:33.272 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:43:33.328 INFO:journalctl@ceph.iscsi.iscsi.a.smithi097.stdout:Mar 11 21:43:33 smithi097 ceph-00db0016-dfed-11ee-95c8-87774f69a715-iscsi-iscsi-a[60943]: debug there is no tcmu-runner data available 2024-03-11T21:43:33.329 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:43:32 smithi097 ceph-mon[33317]: pgmap v844: 158 pgs: 1 active+clean+scrubbing, 2 active+clean+scrubbing+deep, 155 active+clean; 3.3 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-11T21:43:33.329 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:43:32 smithi097 ceph-mon[33317]: 5.7 scrub starts 2024-03-11T21:43:33.329 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:43:32 smithi097 ceph-mon[33317]: 5.7 scrub ok 2024-03-11T21:43:33.329 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:43:32 smithi097 ceph-mon[33317]: 5.1c deep-scrub starts 2024-03-11T21:43:33.329 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:43:32 smithi097 ceph-mon[33317]: 5.1c deep-scrub ok 2024-03-11T21:43:33.329 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:43:32 smithi097 ceph-mon[33317]: 5.a deep-scrub starts 2024-03-11T21:43:33.329 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:43:32 smithi097 ceph-mon[33317]: 5.a deep-scrub ok 2024-03-11T21:43:33.366 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:43:32 smithi092 ceph-mon[29572]: pgmap v844: 158 pgs: 1 active+clean+scrubbing, 2 active+clean+scrubbing+deep, 155 active+clean; 3.3 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-11T21:43:33.367 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:43:32 smithi092 ceph-mon[29572]: 5.7 scrub starts 2024-03-11T21:43:33.367 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:43:32 smithi092 ceph-mon[29572]: 5.7 scrub ok 2024-03-11T21:43:33.367 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:43:32 smithi092 ceph-mon[29572]: 5.1c deep-scrub starts 2024-03-11T21:43:33.367 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:43:32 smithi092 ceph-mon[29572]: 5.1c deep-scrub ok 2024-03-11T21:43:33.367 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:43:32 smithi092 ceph-mon[29572]: 5.a deep-scrub starts 2024-03-11T21:43:33.367 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:43:32 smithi092 ceph-mon[29572]: 5.a deep-scrub ok 2024-03-11T21:43:33.367 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:43:32 smithi092 ceph-mon[37563]: pgmap v844: 158 pgs: 1 active+clean+scrubbing, 2 active+clean+scrubbing+deep, 155 active+clean; 3.3 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-11T21:43:33.367 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:43:32 smithi092 ceph-mon[37563]: 5.7 scrub starts 2024-03-11T21:43:33.367 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:43:32 smithi092 ceph-mon[37563]: 5.7 scrub ok 2024-03-11T21:43:33.367 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:43:32 smithi092 ceph-mon[37563]: 5.1c deep-scrub starts 2024-03-11T21:43:33.367 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:43:32 smithi092 ceph-mon[37563]: 5.1c deep-scrub ok 2024-03-11T21:43:33.368 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:43:32 smithi092 ceph-mon[37563]: 5.a deep-scrub starts 2024-03-11T21:43:33.368 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:43:32 smithi092 ceph-mon[37563]: 5.a deep-scrub ok 2024-03-11T21:43:33.373 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:43:33.407 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:43:33.408 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:43:33.408 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:43:33.509 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:43:33.540 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:43:33.541 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:43:33.541 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:43:33.642 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:43:33.679 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:43:33.679 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:43:33.680 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:43:33.781 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:43:33.823 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:43:33.824 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:43:33.824 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:43:33.925 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:43:33.959 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:43:33.960 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:43:33.960 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:43:34.061 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:43:34.105 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:43:34.106 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:43:34.106 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:43:34.116 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:43:34 smithi092 ceph-mon[29572]: 6.11 scrub starts 2024-03-11T21:43:34.116 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:43:34 smithi092 ceph-mon[29572]: 6.11 scrub ok 2024-03-11T21:43:34.116 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:43:34 smithi092 ceph-mon[29572]: from='client.24469 v1:172.21.15.97:0/404208310' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:43:34.116 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:43:34 smithi092 ceph-mon[29572]: 6.0 scrub starts 2024-03-11T21:43:34.116 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:43:34 smithi092 ceph-mon[29572]: 2.15 scrub starts 2024-03-11T21:43:34.116 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:43:34 smithi092 ceph-mon[29572]: 2.15 scrub ok 2024-03-11T21:43:34.116 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:43:34 smithi092 ceph-mon[29572]: 6.0 scrub ok 2024-03-11T21:43:34.117 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:43:34 smithi092 ceph-mon[29572]: 10.d scrub starts 2024-03-11T21:43:34.117 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:43:34 smithi092 ceph-mon[37563]: 6.11 scrub starts 2024-03-11T21:43:34.117 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:43:34 smithi092 ceph-mon[37563]: 6.11 scrub ok 2024-03-11T21:43:34.117 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:43:34 smithi092 ceph-mon[37563]: from='client.24469 v1:172.21.15.97:0/404208310' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:43:34.117 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:43:34 smithi092 ceph-mon[37563]: 6.0 scrub starts 2024-03-11T21:43:34.117 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:43:34 smithi092 ceph-mon[37563]: 2.15 scrub starts 2024-03-11T21:43:34.117 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:43:34 smithi092 ceph-mon[37563]: 2.15 scrub ok 2024-03-11T21:43:34.117 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:43:34 smithi092 ceph-mon[37563]: 6.0 scrub ok 2024-03-11T21:43:34.117 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:43:34 smithi092 ceph-mon[37563]: 10.d scrub starts 2024-03-11T21:43:34.207 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:43:34.243 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:43:34.244 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:43:34.244 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:43:34.332 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:43:34 smithi097 ceph-mon[33317]: 6.11 scrub starts 2024-03-11T21:43:34.332 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:43:34 smithi097 ceph-mon[33317]: 6.11 scrub ok 2024-03-11T21:43:34.332 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:43:34 smithi097 ceph-mon[33317]: from='client.24469 v1:172.21.15.97:0/404208310' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:43:34.332 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:43:34 smithi097 ceph-mon[33317]: 6.0 scrub starts 2024-03-11T21:43:34.333 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:43:34 smithi097 ceph-mon[33317]: 2.15 scrub starts 2024-03-11T21:43:34.333 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:43:34 smithi097 ceph-mon[33317]: 2.15 scrub ok 2024-03-11T21:43:34.333 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:43:34 smithi097 ceph-mon[33317]: 6.0 scrub ok 2024-03-11T21:43:34.333 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:43:34 smithi097 ceph-mon[33317]: 10.d scrub starts 2024-03-11T21:43:34.344 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:43:34.372 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:43:34.372 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:43:34.373 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:43:34.474 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:43:34.506 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:43:34.507 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:43:34.507 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:43:34.608 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:43:34.644 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:43:34.645 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:43:34.645 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:43:34.746 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:43:34.775 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:43:34.775 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:43:34.776 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:43:34.876 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:43:34.916 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:43:34.917 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:43:34.917 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:43:35.018 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:43:35.057 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:43:35.057 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:43:35.057 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:43:35.116 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:43:35 smithi092 ceph-mon[29572]: pgmap v845: 158 pgs: 1 active+clean+scrubbing, 2 active+clean+scrubbing+deep, 155 active+clean; 3.1 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 686 B/s rd, 0 op/s 2024-03-11T21:43:35.117 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:43:35 smithi092 ceph-mon[29572]: 2.18 scrub starts 2024-03-11T21:43:35.117 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:43:35 smithi092 ceph-mon[29572]: 2.18 scrub ok 2024-03-11T21:43:35.117 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:43:35 smithi092 ceph-mon[29572]: 4.19 scrub starts 2024-03-11T21:43:35.117 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:43:35 smithi092 ceph-mon[29572]: 4.19 scrub ok 2024-03-11T21:43:35.117 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:43:35 smithi092 ceph-mon[29572]: 10.11 deep-scrub ok 2024-03-11T21:43:35.117 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:43:35 smithi092 ceph-mon[29572]: 10.d scrub ok 2024-03-11T21:43:35.117 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:43:35 smithi092 ceph-mon[37563]: pgmap v845: 158 pgs: 1 active+clean+scrubbing, 2 active+clean+scrubbing+deep, 155 active+clean; 3.1 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 686 B/s rd, 0 op/s 2024-03-11T21:43:35.117 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:43:35 smithi092 ceph-mon[37563]: 2.18 scrub starts 2024-03-11T21:43:35.117 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:43:35 smithi092 ceph-mon[37563]: 2.18 scrub ok 2024-03-11T21:43:35.117 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:43:35 smithi092 ceph-mon[37563]: 4.19 scrub starts 2024-03-11T21:43:35.117 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:43:35 smithi092 ceph-mon[37563]: 4.19 scrub ok 2024-03-11T21:43:35.117 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:43:35 smithi092 ceph-mon[37563]: 10.11 deep-scrub ok 2024-03-11T21:43:35.118 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:43:35 smithi092 ceph-mon[37563]: 10.d scrub ok 2024-03-11T21:43:35.158 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:43:35.201 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:43:35.202 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:43:35.202 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:43:35.303 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:43:35.332 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:43:35 smithi097 ceph-mon[33317]: pgmap v845: 158 pgs: 1 active+clean+scrubbing, 2 active+clean+scrubbing+deep, 155 active+clean; 3.1 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 686 B/s rd, 0 op/s 2024-03-11T21:43:35.332 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:43:35 smithi097 ceph-mon[33317]: 2.18 scrub starts 2024-03-11T21:43:35.332 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:43:35 smithi097 ceph-mon[33317]: 2.18 scrub ok 2024-03-11T21:43:35.333 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:43:35 smithi097 ceph-mon[33317]: 4.19 scrub starts 2024-03-11T21:43:35.333 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:43:35 smithi097 ceph-mon[33317]: 4.19 scrub ok 2024-03-11T21:43:35.333 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:43:35 smithi097 ceph-mon[33317]: 10.11 deep-scrub ok 2024-03-11T21:43:35.333 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:43:35 smithi097 ceph-mon[33317]: 10.d scrub ok 2024-03-11T21:43:35.348 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:43:35.349 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:43:35.349 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:43:35.368 INFO:teuthology.orchestra.run.smithi092.stdout:{ 2024-03-11T21:43:35.368 INFO:teuthology.orchestra.run.smithi092.stdout: "ops": [], 2024-03-11T21:43:35.368 INFO:teuthology.orchestra.run.smithi092.stdout: "complaint_time": 30, 2024-03-11T21:43:35.369 INFO:teuthology.orchestra.run.smithi092.stdout: "num_blocked_ops": 0 2024-03-11T21:43:35.369 INFO:teuthology.orchestra.run.smithi092.stdout:} 2024-03-11T21:43:35.382 INFO:teuthology.orchestra.run.smithi092.stdout:osd.0: {} 2024-03-11T21:43:35.417 INFO:teuthology.orchestra.run.smithi092.stdout:osd.1: {} 2024-03-11T21:43:35.450 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:43:35.458 INFO:teuthology.orchestra.run.smithi092.stdout:osd.2: {} 2024-03-11T21:43:35.491 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:43:35.492 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:43:35.493 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:43:35.497 INFO:teuthology.orchestra.run.smithi092.stdout:osd.3: {} 2024-03-11T21:43:35.534 INFO:teuthology.orchestra.run.smithi092.stdout:osd.4: {} 2024-03-11T21:43:35.574 INFO:teuthology.orchestra.run.smithi092.stdout:osd.5: {} 2024-03-11T21:43:35.594 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:43:35.610 INFO:teuthology.orchestra.run.smithi092.stdout:osd.6: {} 2024-03-11T21:43:35.630 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:43:35.630 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:43:35.631 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:43:35.650 INFO:teuthology.orchestra.run.smithi092.stdout:osd.7: {} 2024-03-11T21:43:35.653 INFO:teuthology.orchestra.run.smithi092.stderr:osd.0: osd_enable_op_tracker = '' 2024-03-11T21:43:35.653 INFO:teuthology.orchestra.run.smithi092.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-11T21:43:35.653 INFO:teuthology.orchestra.run.smithi092.stderr:osd.2: osd_enable_op_tracker = '' 2024-03-11T21:43:35.653 INFO:teuthology.orchestra.run.smithi092.stderr:osd.3: osd_enable_op_tracker = '' 2024-03-11T21:43:35.653 INFO:teuthology.orchestra.run.smithi092.stderr:osd.4: osd_enable_op_tracker = '' 2024-03-11T21:43:35.653 INFO:teuthology.orchestra.run.smithi092.stderr:osd.5: osd_enable_op_tracker = '' 2024-03-11T21:43:35.654 INFO:teuthology.orchestra.run.smithi092.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-11T21:43:35.654 INFO:teuthology.orchestra.run.smithi092.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-11T21:43:35.731 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:43:35.768 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:43:35.769 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:43:35.769 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:43:35.870 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:43:35.900 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:43:35.901 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:43:35.901 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:43:36.002 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:43:36.042 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:43:36.043 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:43:36.043 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:43:36.144 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:43:36.179 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:43:36.180 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:43:36.180 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:43:36.281 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:43:36.320 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:43:36.320 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:43:36.321 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:43:36.332 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:43:36 smithi097 ceph-mon[33317]: 10.10 deep-scrub ok 2024-03-11T21:43:36.332 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:43:36 smithi097 ceph-mon[33317]: from='client.? v1:172.21.15.92:0/1832379646' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-11T21:43:36.366 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:43:36 smithi092 ceph-mon[29572]: 10.10 deep-scrub ok 2024-03-11T21:43:36.366 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:43:36 smithi092 ceph-mon[29572]: from='client.? v1:172.21.15.92:0/1832379646' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-11T21:43:36.367 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:43:36 smithi092 ceph-mon[37563]: 10.10 deep-scrub ok 2024-03-11T21:43:36.367 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:43:36 smithi092 ceph-mon[37563]: from='client.? v1:172.21.15.92:0/1832379646' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-11T21:43:36.422 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:43:36.459 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:43:36.460 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:43:36.460 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:43:36.561 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:43:36.605 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:43:36.605 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:43:36.606 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:43:36.706 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:43:36.746 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:43:36.746 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:43:36.746 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:43:36.847 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:43:36.887 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:43:36.887 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:43:36.887 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:43:36.948 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5, 6, 7] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 4, 5, 6, 7, 3] 2024-03-11T21:43:36.948 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-03-11T21:43:36.949 INFO:tasks.thrashosds.thrasher:Shrinking pool unique_pool_3 2024-03-11T21:43:36.949 INFO:tasks.cephadm.ceph_manager.ceph:contract_pool unique_pool_3 by 10 min 8 2024-03-11T21:43:36.949 DEBUG:teuthology.orchestra.run.smithi092:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid 00db0016-dfed-11ee-95c8-87774f69a715 -- ceph pg dump --format=json 2024-03-11T21:43:36.988 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:43:37.023 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:43:37.024 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:43:37.024 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:43:37.024 INFO:teuthology.orchestra.run.smithi092.stderr:noscrub is set 2024-03-11T21:43:37.125 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:43:37.158 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:43:37.159 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:43:37.159 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:43:37.260 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:43:37.297 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:43:37.297 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:43:37.298 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:43:37.332 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:43:37 smithi097 ceph-mon[33317]: pgmap v846: 158 pgs: 2 active+clean+scrubbing+deep, 156 active+clean; 3.0 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T21:43:37.332 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:43:37 smithi097 ceph-mon[33317]: 5.1b deep-scrub starts 2024-03-11T21:43:37.332 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:43:37 smithi097 ceph-mon[33317]: 5.1b deep-scrub ok 2024-03-11T21:43:37.332 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:43:37 smithi097 ceph-mon[33317]: 5.1d scrub starts 2024-03-11T21:43:37.333 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:43:37 smithi097 ceph-mon[33317]: 5.1d scrub ok 2024-03-11T21:43:37.333 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:43:37 smithi097 ceph-mon[33317]: purged_snaps scrub starts 2024-03-11T21:43:37.333 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:43:37 smithi097 ceph-mon[33317]: purged_snaps scrub ok 2024-03-11T21:43:37.333 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:43:37 smithi097 ceph-mon[33317]: 4.1b scrub starts 2024-03-11T21:43:37.333 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:43:37 smithi097 ceph-mon[33317]: 4.1b scrub ok 2024-03-11T21:43:37.333 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:43:37 smithi097 ceph-mon[33317]: 10.f scrub starts 2024-03-11T21:43:37.333 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:43:37 smithi097 ceph-mon[33317]: from='client.? v1:172.21.15.92:0/1117227176' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-11T21:43:37.366 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:43:37 smithi092 ceph-mon[29572]: pgmap v846: 158 pgs: 2 active+clean+scrubbing+deep, 156 active+clean; 3.0 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T21:43:37.366 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:43:37 smithi092 ceph-mon[29572]: 5.1b deep-scrub starts 2024-03-11T21:43:37.366 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:43:37 smithi092 ceph-mon[29572]: 5.1b deep-scrub ok 2024-03-11T21:43:37.367 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:43:37 smithi092 ceph-mon[29572]: 5.1d scrub starts 2024-03-11T21:43:37.367 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:43:37 smithi092 ceph-mon[29572]: 5.1d scrub ok 2024-03-11T21:43:37.367 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:43:37 smithi092 ceph-mon[29572]: purged_snaps scrub starts 2024-03-11T21:43:37.367 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:43:37 smithi092 ceph-mon[29572]: purged_snaps scrub ok 2024-03-11T21:43:37.367 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:43:37 smithi092 ceph-mon[29572]: 4.1b scrub starts 2024-03-11T21:43:37.367 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:43:37 smithi092 ceph-mon[29572]: 4.1b scrub ok 2024-03-11T21:43:37.367 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:43:37 smithi092 ceph-mon[29572]: 10.f scrub starts 2024-03-11T21:43:37.367 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:43:37 smithi092 ceph-mon[29572]: from='client.? v1:172.21.15.92:0/1117227176' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-11T21:43:37.367 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:43:37 smithi092 ceph-mon[37563]: pgmap v846: 158 pgs: 2 active+clean+scrubbing+deep, 156 active+clean; 3.0 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T21:43:37.367 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:43:37 smithi092 ceph-mon[37563]: 5.1b deep-scrub starts 2024-03-11T21:43:37.367 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:43:37 smithi092 ceph-mon[37563]: 5.1b deep-scrub ok 2024-03-11T21:43:37.367 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:43:37 smithi092 ceph-mon[37563]: 5.1d scrub starts 2024-03-11T21:43:37.368 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:43:37 smithi092 ceph-mon[37563]: 5.1d scrub ok 2024-03-11T21:43:37.368 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:43:37 smithi092 ceph-mon[37563]: purged_snaps scrub starts 2024-03-11T21:43:37.368 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:43:37 smithi092 ceph-mon[37563]: purged_snaps scrub ok 2024-03-11T21:43:37.368 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:43:37 smithi092 ceph-mon[37563]: 4.1b scrub starts 2024-03-11T21:43:37.368 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:43:37 smithi092 ceph-mon[37563]: 4.1b scrub ok 2024-03-11T21:43:37.368 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:43:37 smithi092 ceph-mon[37563]: 10.f scrub starts 2024-03-11T21:43:37.368 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:43:37 smithi092 ceph-mon[37563]: from='client.? v1:172.21.15.92:0/1117227176' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-11T21:43:37.399 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:43:37.441 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:43:37.441 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:43:37.442 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:43:37.543 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:43:37.578 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:43:37.579 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:43:37.579 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:43:37.680 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:43:37.715 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:43:37.715 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:43:37.715 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:43:37.816 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:43:37.850 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:43:37.850 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:43:37.851 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:43:37.951 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:43:37.984 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:43:37.985 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:43:37.985 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:43:38.086 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:43:38.119 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:43:38.119 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:43:38.120 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:43:38.221 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:43:38.267 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:43:38.268 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:43:38.268 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:43:38.332 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:43:38 smithi097 ceph-mon[33317]: 5.16 deep-scrub starts 2024-03-11T21:43:38.332 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:43:38 smithi097 ceph-mon[33317]: 5.16 deep-scrub ok 2024-03-11T21:43:38.333 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:43:38 smithi097 ceph-mon[33317]: 10.c deep-scrub starts 2024-03-11T21:43:38.333 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:43:38 smithi097 ceph-mon[33317]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-03-11T21:43:38.333 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:43:38 smithi097 ceph-mon[33317]: from='client.? v1:172.21.15.92:0/1117227176' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-03-11T21:43:38.333 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:43:38 smithi097 ceph-mon[33317]: osdmap e414: 8 total, 8 up, 8 in 2024-03-11T21:43:38.333 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:43:38 smithi097 ceph-mon[33317]: from='mgr.14607 ' entity='mgr.y' 2024-03-11T21:43:38.333 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:43:38 smithi097 ceph-mon[33317]: from='mgr.14607 v1:172.21.15.92:0/1702717597' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T21:43:38.366 INFO:journalctl@ceph.mgr.y.smithi092.stdout:Mar 11 21:43:37 smithi092 ceph-00db0016-dfed-11ee-95c8-87774f69a715-mgr-y[29801]: ::ffff:172.21.15.97 - - [11/Mar/2024:21:43:37] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-11T21:43:38.367 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:43:38 smithi092 ceph-mon[29572]: 5.16 deep-scrub starts 2024-03-11T21:43:38.367 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:43:38 smithi092 ceph-mon[29572]: 5.16 deep-scrub ok 2024-03-11T21:43:38.367 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:43:38 smithi092 ceph-mon[29572]: 10.c deep-scrub starts 2024-03-11T21:43:38.367 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:43:38 smithi092 ceph-mon[29572]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-03-11T21:43:38.367 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:43:38 smithi092 ceph-mon[29572]: from='client.? v1:172.21.15.92:0/1117227176' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-03-11T21:43:38.367 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:43:38 smithi092 ceph-mon[29572]: osdmap e414: 8 total, 8 up, 8 in 2024-03-11T21:43:38.367 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:43:38 smithi092 ceph-mon[29572]: from='mgr.14607 ' entity='mgr.y' 2024-03-11T21:43:38.367 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:43:38 smithi092 ceph-mon[29572]: from='mgr.14607 v1:172.21.15.92:0/1702717597' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T21:43:38.367 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:43:38 smithi092 ceph-mon[37563]: 5.16 deep-scrub starts 2024-03-11T21:43:38.367 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:43:38 smithi092 ceph-mon[37563]: 5.16 deep-scrub ok 2024-03-11T21:43:38.367 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:43:38 smithi092 ceph-mon[37563]: 10.c deep-scrub starts 2024-03-11T21:43:38.368 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:43:38 smithi092 ceph-mon[37563]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-03-11T21:43:38.368 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:43:38 smithi092 ceph-mon[37563]: from='client.? v1:172.21.15.92:0/1117227176' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-03-11T21:43:38.368 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:43:38 smithi092 ceph-mon[37563]: osdmap e414: 8 total, 8 up, 8 in 2024-03-11T21:43:38.368 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:43:38 smithi092 ceph-mon[37563]: from='mgr.14607 ' entity='mgr.y' 2024-03-11T21:43:38.368 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:43:38 smithi092 ceph-mon[37563]: from='mgr.14607 v1:172.21.15.92:0/1702717597' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T21:43:38.369 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:43:38.393 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:43:38.393 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:43:38.393 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:43:38.494 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:43:38.531 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:43:38.532 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:43:38.532 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:43:38.633 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:43:38.668 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:43:38.668 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:43:38.669 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:43:38.770 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:43:38.811 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:43:38.811 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:43:38.812 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:43:38.913 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:43:38.941 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:43:38.942 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:43:38.942 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:43:39.043 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:43:39.077 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:43:39.078 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:43:39.078 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:43:39.179 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:43:39.216 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:43:39.217 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:43:39.217 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:43:39.318 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:43:39.332 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:43:39 smithi097 ceph-mon[33317]: pgmap v848: 158 pgs: 158 active+clean; 3.0 GiB data, 15 GiB used, 701 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-03-11T21:43:39.360 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:43:39.361 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:43:39.361 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:43:39.366 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:43:39 smithi092 ceph-mon[29572]: pgmap v848: 158 pgs: 158 active+clean; 3.0 GiB data, 15 GiB used, 701 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-03-11T21:43:39.366 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:43:39 smithi092 ceph-mon[37563]: pgmap v848: 158 pgs: 158 active+clean; 3.0 GiB data, 15 GiB used, 701 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-03-11T21:43:39.462 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:43:39.498 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:43:39.498 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:43:39.499 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:43:39.599 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:43:39.637 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:43:39.638 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:43:39.638 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:43:39.739 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:43:39.768 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:43:39.769 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:43:39.769 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:43:39.870 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:43:39.907 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:43:39.908 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:43:39.908 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:43:40.009 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:43:40.044 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:43:40.045 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:43:40.045 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:43:40.146 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:43:40.181 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:43:40.182 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:43:40.183 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:43:40.283 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:43:40.323 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:43:40.324 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:43:40.325 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:43:40.423 DEBUG:teuthology.orchestra.run.smithi092:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid 00db0016-dfed-11ee-95c8-87774f69a715 -- ceph daemon osd.3 dump_historic_ops 2024-03-11T21:43:40.426 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:43:40.463 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:43:40.464 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:43:40.464 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:43:40.565 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:43:40.599 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:43:40.599 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:43:40.600 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:43:40.700 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:43:40.734 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:43:40.735 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:43:40.735 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:43:40.836 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:43:40.882 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:43:40.883 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:43:40.883 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:43:40.984 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:43:41.022 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:43:41.022 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:43:41.023 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:43:41.116 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:43:41 smithi092 ceph-mon[29572]: pgmap v849: 158 pgs: 1 active+clean+scrubbing+deep, 157 active+clean; 2.8 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T21:43:41.116 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:43:41 smithi092 ceph-mon[37563]: pgmap v849: 158 pgs: 1 active+clean+scrubbing+deep, 157 active+clean; 2.8 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T21:43:41.124 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:43:41.162 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:43:41.163 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:43:41.164 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:43:41.265 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:43:41.302 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:43:41.302 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:43:41.302 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:43:41.334 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:43:41 smithi097 ceph-mon[33317]: pgmap v849: 158 pgs: 1 active+clean+scrubbing+deep, 157 active+clean; 2.8 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T21:43:41.404 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:43:41.437 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:43:41.437 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:43:41.438 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:43:41.539 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:43:41.571 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:43:41.571 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:43:41.572 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:43:41.673 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:43:41.709 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:43:41.709 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:43:41.710 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:43:41.804 INFO:teuthology.orchestra.run.smithi092.stderr:Inferring config /var/lib/ceph/00db0016-dfed-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T21:43:41.810 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:43:41.843 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:43:41.843 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:43:41.844 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:43:41.880 INFO:teuthology.orchestra.run.smithi092.stderr:Inferring config /var/lib/ceph/00db0016-dfed-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T21:43:41.945 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:43:41.976 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:43:41.977 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:43:41.978 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:43:42.079 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:43:42.115 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:43:42.116 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:43:42.116 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:43:42.217 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:43:42.257 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:43:42.257 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:43:42.258 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:43:42.358 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:43:42.395 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:43:42.396 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:43:42.396 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:43:42.497 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:43:42.532 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:43:42.533 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:43:42.533 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:43:42.634 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:43:42.672 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:43:42.672 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:43:42.673 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:43:42.774 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:43:42.813 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:43:42.814 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:43:42.814 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:43:42.915 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:43:42.949 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:43:42.949 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:43:42.950 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:43:43.050 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:43:43.091 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:43:43.091 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:43:43.092 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:43:43.193 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:43:43.230 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:43:43.231 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:43:43.231 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:43:43.332 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:43:43.333 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:43:43 smithi097 ceph-mon[33317]: pgmap v850: 158 pgs: 1 active+clean+scrubbing+deep, 157 active+clean; 2.7 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-11T21:43:43.333 INFO:journalctl@ceph.iscsi.iscsi.a.smithi097.stdout:Mar 11 21:43:43 smithi097 ceph-00db0016-dfed-11ee-95c8-87774f69a715-iscsi-iscsi-a[60943]: debug there is no tcmu-runner data available 2024-03-11T21:43:43.334 DEBUG:teuthology.orchestra.run.smithi092:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid 00db0016-dfed-11ee-95c8-87774f69a715 -- ceph osd set nodeep-scrub 2024-03-11T21:43:43.366 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:43:43 smithi092 ceph-mon[29572]: pgmap v850: 158 pgs: 1 active+clean+scrubbing+deep, 157 active+clean; 2.7 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-11T21:43:43.366 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:43:43 smithi092 ceph-mon[37563]: pgmap v850: 158 pgs: 1 active+clean+scrubbing+deep, 157 active+clean; 2.7 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-11T21:43:43.378 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:43:43.381 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:43:43.381 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:43:43.483 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:43:43.530 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:43:43.531 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:43:43.531 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:43:43.632 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:43:43.673 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:43:43.673 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:43:43.674 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:43:43.775 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:43:43.779 DEBUG:teuthology.orchestra.run.smithi092:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid 00db0016-dfed-11ee-95c8-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-03-11T21:43:43.823 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:43:43.824 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:43:43.824 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:43:43.925 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:43:43.968 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:43:43.969 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:43:43.969 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:43:44.070 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:43:44.103 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:43:44.104 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:43:44.104 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:43:44.205 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:43:44.257 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:43:44.258 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:43:44.258 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:43:44.332 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:43:44 smithi097 ceph-mon[33317]: from='client.24469 v1:172.21.15.97:0/404208310' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:43:44.332 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:43:44 smithi097 ceph-mon[33317]: purged_snaps scrub starts 2024-03-11T21:43:44.332 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:43:44 smithi097 ceph-mon[33317]: purged_snaps scrub ok 2024-03-11T21:43:44.359 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:43:44.366 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:43:44 smithi092 ceph-mon[29572]: from='client.24469 v1:172.21.15.97:0/404208310' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:43:44.366 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:43:44 smithi092 ceph-mon[29572]: purged_snaps scrub starts 2024-03-11T21:43:44.366 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:43:44 smithi092 ceph-mon[29572]: purged_snaps scrub ok 2024-03-11T21:43:44.367 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:43:44 smithi092 ceph-mon[37563]: from='client.24469 v1:172.21.15.97:0/404208310' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:43:44.367 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:43:44 smithi092 ceph-mon[37563]: purged_snaps scrub starts 2024-03-11T21:43:44.367 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:43:44 smithi092 ceph-mon[37563]: purged_snaps scrub ok 2024-03-11T21:43:44.395 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:43:44.395 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:43:44.396 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:43:44.497 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:43:44.536 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:43:44.536 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:43:44.536 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:43:44.637 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:43:44.668 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:43:44.668 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:43:44.669 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:43:44.769 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:43:44.801 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:43:44.802 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:43:44.802 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:43:44.861 INFO:teuthology.orchestra.run.smithi092.stderr:Inferring config /var/lib/ceph/00db0016-dfed-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T21:43:44.903 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:43:44.961 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:43:44.962 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:43:44.962 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:43:45.063 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:43:45.099 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:43:45.100 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:43:45.100 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:43:45.179 INFO:teuthology.orchestra.run.smithi092.stderr:Inferring config /var/lib/ceph/00db0016-dfed-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T21:43:45.201 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:43:45.236 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:43:45.237 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:43:45.237 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:43:45.317 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:43:45 smithi092 ceph-mon[37563]: pgmap v851: 158 pgs: 1 active+clean+scrubbing+deep, 157 active+clean; 2.5 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-11T21:43:45.317 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:43:45 smithi092 ceph-mon[37563]: osdmap e415: 8 total, 8 up, 8 in 2024-03-11T21:43:45.317 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:43:45 smithi092 ceph-mon[29572]: pgmap v851: 158 pgs: 1 active+clean+scrubbing+deep, 157 active+clean; 2.5 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-11T21:43:45.318 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:43:45 smithi092 ceph-mon[29572]: osdmap e415: 8 total, 8 up, 8 in 2024-03-11T21:43:45.332 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:43:45 smithi097 ceph-mon[33317]: pgmap v851: 158 pgs: 1 active+clean+scrubbing+deep, 157 active+clean; 2.5 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-11T21:43:45.332 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:43:45 smithi097 ceph-mon[33317]: osdmap e415: 8 total, 8 up, 8 in 2024-03-11T21:43:45.338 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:43:45.374 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:43:45.374 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:43:45.374 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:43:45.475 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:43:45.512 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:43:45.512 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:43:45.513 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:43:45.614 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:43:45.653 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:43:45.654 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:43:45.654 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:43:45.755 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:43:45.780 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:43:45.780 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:43:45.780 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:43:45.881 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:43:45.919 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:43:45.919 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:43:45.920 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:43:46.021 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:43:46.051 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:43:46.051 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:43:46.052 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:43:46.152 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:43:46.185 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:43:46.186 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:43:46.186 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:43:46.287 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:43:46.331 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:43:46.331 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:43:46.332 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:43:46.433 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:43:46.470 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:43:46.540 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:43:46.540 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:43:46.641 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:43:46.679 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:43:46.679 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:43:46.680 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:43:46.780 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:43:46.814 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:43:46.815 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:43:46.815 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:43:46.916 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:43:46.957 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:43:46.957 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:43:46.957 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:43:47.058 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:43:47.094 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:43:47.095 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:43:47.096 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:43:47.176 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:43:47 smithi092 ceph-mon[29572]: pgmap v853: 158 pgs: 1 active+clean+scrubbing+deep, 157 active+clean; 2.4 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 21 op/s 2024-03-11T21:43:47.177 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:43:47 smithi092 ceph-mon[37563]: pgmap v853: 158 pgs: 1 active+clean+scrubbing+deep, 157 active+clean; 2.4 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 21 op/s 2024-03-11T21:43:47.196 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:43:47.231 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:43:47.231 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:43:47.232 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:43:47.332 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:43:47.333 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:43:47 smithi097 ceph-mon[33317]: pgmap v853: 158 pgs: 1 active+clean+scrubbing+deep, 157 active+clean; 2.4 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 21 op/s 2024-03-11T21:43:47.365 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:43:47.366 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:43:47.366 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:43:47.473 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:43:47.474 INFO:teuthology.orchestra.run.smithi092.stdout: 2024-03-11T21:43:47.478 INFO:teuthology.orchestra.run.smithi092.stderr:dumped all 2024-03-11T21:43:47.553 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:43:47.553 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:43:47.554 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:43:47.655 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:43:47.705 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:43:47.706 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:43:47.706 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:43:47.807 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:43:47.842 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:43:47.843 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:43:47.843 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:43:47.944 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:43:47.980 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:43:47.981 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:43:47.981 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:43:48.082 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:43:48.122 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:43:48.122 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:43:48.123 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:43:48.224 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:43:48.260 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:43:48.262 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:43:48.262 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:43:48.269 INFO:journalctl@ceph.mgr.y.smithi092.stdout:Mar 11 21:43:47 smithi092 ceph-00db0016-dfed-11ee-95c8-87774f69a715-mgr-y[29801]: ::ffff:172.21.15.97 - - [11/Mar/2024:21:43:47] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-11T21:43:48.364 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:43:48.404 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:43:48.405 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:43:48.405 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:43:48.506 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:43:48.542 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:43:48.543 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:43:48.544 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:43:48.644 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:43:48.680 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:43:48.681 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:43:48.681 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:43:48.782 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:43:48.818 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:43:48.819 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:43:48.819 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:43:48.920 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:43:48.954 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:43:48.954 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:43:48.955 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:43:49.056 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:43:49.087 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:43:49.088 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:43:49.088 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:43:49.189 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:43:49.231 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:43:49.232 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:43:49.232 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:43:49.333 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:43:49.366 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:43:49 smithi092 ceph-mon[29572]: pgmap v854: 158 pgs: 1 active+clean+scrubbing+deep, 157 active+clean; 2.3 GiB data, 13 GiB used, 703 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 1.08k op/s 2024-03-11T21:43:49.367 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:43:49 smithi092 ceph-mon[29572]: from='client.35699 v1:172.21.15.92:0/2323425843' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-11T21:43:49.367 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:43:49 smithi092 ceph-mon[37563]: pgmap v854: 158 pgs: 1 active+clean+scrubbing+deep, 157 active+clean; 2.3 GiB data, 13 GiB used, 703 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 1.08k op/s 2024-03-11T21:43:49.367 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:43:49 smithi092 ceph-mon[37563]: from='client.35699 v1:172.21.15.92:0/2323425843' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-11T21:43:49.367 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:43:49.368 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:43:49.368 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:43:49.469 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:43:49.501 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:43:49.502 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:43:49.502 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:43:49.582 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:43:49 smithi097 ceph-mon[33317]: pgmap v854: 158 pgs: 1 active+clean+scrubbing+deep, 157 active+clean; 2.3 GiB data, 13 GiB used, 703 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 1.08k op/s 2024-03-11T21:43:49.582 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:43:49 smithi097 ceph-mon[33317]: from='client.35699 v1:172.21.15.92:0/2323425843' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-11T21:43:49.603 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:43:49.639 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:43:49.640 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:43:49.640 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:43:49.741 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:43:49.778 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:43:49.778 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:43:49.778 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:43:49.879 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:43:49.916 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:43:49.917 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:43:49.917 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:43:50.018 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:43:50.053 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:43:50.053 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:43:50.054 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:43:50.154 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:43:50.180 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:43:50.180 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:43:50.181 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:43:50.282 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:43:50.321 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:43:50.322 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:43:50.322 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:43:50.423 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:43:50.461 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:43:50.462 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:43:50.462 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:43:50.563 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:43:50.596 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:43:50.597 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:43:50.597 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:43:50.698 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:43:50.731 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:43:50.731 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:43:50.732 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:43:50.833 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:43:50.857 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:43:50.857 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:43:50.858 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:43:50.958 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:43:50.992 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:43:50.992 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:43:50.993 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:43:51.093 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:43:51.130 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:43:51.130 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:43:51.130 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:43:51.231 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:43:51.268 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:43:51.268 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:43:51.269 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:43:51.366 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:43:51 smithi092 ceph-mon[29572]: pgmap v855: 158 pgs: 1 active+clean+scrubbing+deep, 157 active+clean; 2.1 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 1.10k op/s 2024-03-11T21:43:51.366 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:43:51 smithi092 ceph-mon[29572]: 10.c deep-scrub ok 2024-03-11T21:43:51.367 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:43:51 smithi092 ceph-mon[37563]: pgmap v855: 158 pgs: 1 active+clean+scrubbing+deep, 157 active+clean; 2.1 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 1.10k op/s 2024-03-11T21:43:51.367 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:43:51 smithi092 ceph-mon[37563]: 10.c deep-scrub ok 2024-03-11T21:43:51.369 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:43:51.384 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:43:51 smithi097 ceph-mon[33317]: pgmap v855: 158 pgs: 1 active+clean+scrubbing+deep, 157 active+clean; 2.1 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 1.10k op/s 2024-03-11T21:43:51.384 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:43:51 smithi097 ceph-mon[33317]: 10.c deep-scrub ok 2024-03-11T21:43:51.400 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:43:51.401 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:43:51.401 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:43:51.502 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:43:51.536 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:43:51.536 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:43:51.537 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:43:51.543 INFO:teuthology.orchestra.run.smithi092.stdout:ERROR: (22) Invalid argument 2024-03-11T21:43:51.543 INFO:teuthology.orchestra.run.smithi092.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-11T21:43:51.637 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:43:51.670 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:43:51.671 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:43:51.672 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:43:51.772 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:43:51.812 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:43:51.813 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:43:51.813 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:43:51.914 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:43:51.953 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:43:51.953 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:43:51.954 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:43:52.055 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:43:52.095 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:43:52.096 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:43:52.096 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:43:52.197 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:43:52.251 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:43:52.252 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:43:52.252 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:43:52.255 INFO:teuthology.orchestra.run.smithi092.stdout:{"pg_ready":true,"pg_map":{"version":854,"stamp":"2024-03-11T21:43:47.400908+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":2472352192,"num_objects":37915,"num_object_clones":0,"num_object_copies":113745,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":37915,"num_whiteouts":0,"num_read":2412,"num_read_kb":2260,"num_write":109167,"num_write_kb":3430942,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":203,"num_bytes_recovered":2955718,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":161157,"ondisk_log_size":161157,"up":474,"acting":474,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":485,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":749961216,"kb_used":13303616,"kb_used_data":8297856,"kb_used_omap":15,"kb_used_meta":5005360,"kb_avail":736657600,"statfs":{"total":767960285184,"available":754337382400,"internally_reserved":0,"allocated":8497004544,"data_stored":8494135738,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15601,"internal_metadata":5125489423},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-706412544,"num_objects":-10779,"num_object_clones":0,"num_object_copies":-32337,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-10779,"num_whiteouts":0,"num_read":10,"num_read_kb":10,"num_write":10779,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"10.004788"},"pg_stats":[{"pgid":"10.17","version":"415'8029","reported_seq":12657,"reported_epoch":415,"state":"active+clean","last_fresh":"2024-03-11T21:43:47.062255+0000","last_change":"2024-03-11T21:43:28.442602+0000","last_active":"2024-03-11T21:43:47.062255+0000","last_peered":"2024-03-11T21:43:47.062255+0000","last_clean":"2024-03-11T21:43:47.062255+0000","last_became_active":"2024-03-11T21:42:08.476625+0000","last_became_peered":"2024-03-11T21:42:08.476625+0000","last_unstale":"2024-03-11T21:43:47.062255+0000","last_undegraded":"2024-03-11T21:43:47.062255+0000","last_fullsized":"2024-03-11T21:43:47.062255+0000","mapping_epoch":388,"log_start":"366'2550","ondisk_log_start":"366'2550","created":372,"last_epoch_clean":389,"parent":"0.0","parent_split_bits":5,"last_scrub":"412'7415","last_scrub_stamp":"2024-03-11T21:43:28.442317+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:40:38.219602+0000","last_clean_scrub_stamp":"2024-03-11T21:43:28.442317+0000","objects_scrubbed":1685,"log_size":5479,"log_dups_size":0,"ondisk_log_size":5479,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:44:36.779597+0000","scrub_duration":462,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70123520,"num_objects":1070,"num_object_clones":0,"num_object_copies":3210,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1070,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":615,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1b","version":"0'0","reported_seq":681,"reported_epoch":415,"state":"active+clean","last_fresh":"2024-03-11T21:43:44.088991+0000","last_change":"2024-03-11T21:36:35.074931+0000","last_active":"2024-03-11T21:43:44.088991+0000","last_peered":"2024-03-11T21:43:44.088991+0000","last_clean":"2024-03-11T21:43:44.088991+0000","last_became_active":"2024-03-11T21:36:35.072636+0000","last_became_peered":"2024-03-11T21:36:35.072636+0000","last_unstale":"2024-03-11T21:43:44.088991+0000","last_undegraded":"2024-03-11T21:43:44.088991+0000","last_fullsized":"2024-03-11T21:43:44.088991+0000","mapping_epoch":277,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":278,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:35:07.021520+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:43.559452+0000","last_clean_scrub_stamp":"2024-03-11T21:35:07.021520+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":754,"reported_epoch":415,"state":"active+clean","last_fresh":"2024-03-11T21:43:44.075367+0000","last_change":"2024-03-11T21:41:19.522757+0000","last_active":"2024-03-11T21:43:44.075367+0000","last_peered":"2024-03-11T21:43:44.075367+0000","last_clean":"2024-03-11T21:43:44.075367+0000","last_became_active":"2024-03-11T21:25:42.578893+0000","last_became_peered":"2024-03-11T21:25:42.578893+0000","last_unstale":"2024-03-11T21:43:44.075367+0000","last_undegraded":"2024-03-11T21:43:44.075367+0000","last_fullsized":"2024-03-11T21:43:44.075367+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:41:19.522682+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:41.544550+0000","last_clean_scrub_stamp":"2024-03-11T21:41:19.522682+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:43:48.166510+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1f","version":"0'0","reported_seq":783,"reported_epoch":415,"state":"active+clean","last_fresh":"2024-03-11T21:43:44.088284+0000","last_change":"2024-03-11T21:43:28.129064+0000","last_active":"2024-03-11T21:43:44.088284+0000","last_peered":"2024-03-11T21:43:44.088284+0000","last_clean":"2024-03-11T21:43:44.088284+0000","last_became_active":"2024-03-11T21:34:41.144945+0000","last_became_peered":"2024-03-11T21:34:41.144945+0000","last_unstale":"2024-03-11T21:43:44.088284+0000","last_undegraded":"2024-03-11T21:43:44.088284+0000","last_fullsized":"2024-03-11T21:43:44.088284+0000","mapping_epoch":226,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":227,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:43:28.128287+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:37.512018+0000","last_clean_scrub_stamp":"2024-03-11T21:43:28.128287+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:44:56.476032+0000","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,4],"acting":[0,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.19","version":"260'27","reported_seq":736,"reported_epoch":415,"state":"active+clean","last_fresh":"2024-03-11T21:43:44.082732+0000","last_change":"2024-03-11T21:43:33.673024+0000","last_active":"2024-03-11T21:43:44.082732+0000","last_peered":"2024-03-11T21:43:44.082732+0000","last_clean":"2024-03-11T21:43:44.082732+0000","last_became_active":"2024-03-11T21:36:35.067652+0000","last_became_peered":"2024-03-11T21:36:35.067652+0000","last_unstale":"2024-03-11T21:43:44.082732+0000","last_undegraded":"2024-03-11T21:43:44.082732+0000","last_fullsized":"2024-03-11T21:43:44.082732+0000","mapping_epoch":277,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":278,"parent":"0.0","parent_split_bits":0,"last_scrub":"260'27","last_scrub_stamp":"2024-03-11T21:43:33.672876+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:39.525554+0000","last_clean_scrub_stamp":"2024-03-11T21:43:33.672876+0000","objects_scrubbed":7,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:44:39.350698+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":42,"num_read_kb":33,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.16","version":"414'7707","reported_seq":14397,"reported_epoch":414,"state":"active+clean","last_fresh":"2024-03-11T21:43:42.416348+0000","last_change":"2024-03-11T21:43:20.681951+0000","last_active":"2024-03-11T21:43:42.416348+0000","last_peered":"2024-03-11T21:43:42.416348+0000","last_clean":"2024-03-11T21:43:42.416348+0000","last_became_active":"2024-03-11T21:42:25.516791+0000","last_became_peered":"2024-03-11T21:42:25.516791+0000","last_unstale":"2024-03-11T21:43:42.416348+0000","last_undegraded":"2024-03-11T21:43:42.416348+0000","last_fullsized":"2024-03-11T21:43:42.416348+0000","mapping_epoch":393,"log_start":"0'0","ondisk_log_start":"0'0","created":372,"last_epoch_clean":394,"parent":"0.0","parent_split_bits":5,"last_scrub":"407'7053","last_scrub_stamp":"2024-03-11T21:43:20.681823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:40:38.219602+0000","last_clean_scrub_stamp":"2024-03-11T21:43:20.681823+0000","objects_scrubbed":1849,"log_size":7707,"log_dups_size":0,"ondisk_log_size":7707,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:44:29.254313+0000","scrub_duration":218,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":78315520,"num_objects":1195,"num_object_clones":0,"num_object_copies":3585,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1195,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":654,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,1],"acting":[6,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.1a","version":"0'0","reported_seq":742,"reported_epoch":415,"state":"active+clean","last_fresh":"2024-03-11T21:43:44.075441+0000","last_change":"2024-03-11T21:41:02.614569+0000","last_active":"2024-03-11T21:43:44.075441+0000","last_peered":"2024-03-11T21:43:44.075441+0000","last_clean":"2024-03-11T21:43:44.075441+0000","last_became_active":"2024-03-11T21:25:44.590923+0000","last_became_peered":"2024-03-11T21:25:44.590923+0000","last_unstale":"2024-03-11T21:43:44.075441+0000","last_undegraded":"2024-03-11T21:43:44.075441+0000","last_fullsized":"2024-03-11T21:43:44.075441+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:41:02.614438+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:43.559452+0000","last_clean_scrub_stamp":"2024-03-11T21:41:02.614438+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:43:48.166338+0000","scrub_duration":38,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,1],"acting":[4,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":756,"reported_epoch":414,"state":"active+clean","last_fresh":"2024-03-11T21:43:37.047901+0000","last_change":"2024-03-11T21:36:15.576138+0000","last_active":"2024-03-11T21:43:37.047901+0000","last_peered":"2024-03-11T21:43:37.047901+0000","last_clean":"2024-03-11T21:43:37.047901+0000","last_became_active":"2024-03-11T21:34:41.154216+0000","last_became_peered":"2024-03-11T21:34:41.154216+0000","last_unstale":"2024-03-11T21:43:37.047901+0000","last_undegraded":"2024-03-11T21:43:37.047901+0000","last_fullsized":"2024-03-11T21:43:37.047901+0000","mapping_epoch":226,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":227,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:36:15.575887+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:41.544550+0000","last_clean_scrub_stamp":"2024-03-11T21:36:15.575887+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,7],"acting":[1,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1e","version":"0'0","reported_seq":703,"reported_epoch":415,"state":"active+clean","last_fresh":"2024-03-11T21:43:44.083186+0000","last_change":"2024-03-11T21:41:04.598202+0000","last_active":"2024-03-11T21:43:44.083186+0000","last_peered":"2024-03-11T21:43:44.083186+0000","last_clean":"2024-03-11T21:43:44.083186+0000","last_became_active":"2024-03-11T21:36:35.061786+0000","last_became_peered":"2024-03-11T21:36:35.061786+0000","last_unstale":"2024-03-11T21:43:44.083186+0000","last_undegraded":"2024-03-11T21:43:44.083186+0000","last_fullsized":"2024-03-11T21:43:44.083186+0000","mapping_epoch":277,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":278,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:41:04.597983+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:37.512018+0000","last_clean_scrub_stamp":"2024-03-11T21:41:04.597983+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.18","version":"260'15","reported_seq":791,"reported_epoch":415,"state":"active+clean","last_fresh":"2024-03-11T21:43:44.074964+0000","last_change":"2024-03-11T21:41:17.537672+0000","last_active":"2024-03-11T21:43:44.074964+0000","last_peered":"2024-03-11T21:43:44.074964+0000","last_clean":"2024-03-11T21:43:44.074964+0000","last_became_active":"2024-03-11T21:25:40.552216+0000","last_became_peered":"2024-03-11T21:25:40.552216+0000","last_unstale":"2024-03-11T21:43:44.074964+0000","last_undegraded":"2024-03-11T21:43:44.074964+0000","last_fullsized":"2024-03-11T21:43:44.074964+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"260'15","last_scrub_stamp":"2024-03-11T21:41:17.537536+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:39.525554+0000","last_clean_scrub_stamp":"2024-03-11T21:41:17.537536+0000","objects_scrubbed":4,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:43:48.166846+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"10.15","version":"415'8067","reported_seq":15218,"reported_epoch":415,"state":"active+clean","last_fresh":"2024-03-11T21:43:44.508111+0000","last_change":"2024-03-11T21:43:29.057818+0000","last_active":"2024-03-11T21:43:44.508111+0000","last_peered":"2024-03-11T21:43:44.508111+0000","last_clean":"2024-03-11T21:43:44.508111+0000","last_became_active":"2024-03-11T21:42:21.511969+0000","last_became_peered":"2024-03-11T21:42:21.511969+0000","last_unstale":"2024-03-11T21:43:44.508111+0000","last_undegraded":"2024-03-11T21:43:44.508111+0000","last_fullsized":"2024-03-11T21:43:44.508111+0000","mapping_epoch":391,"log_start":"366'2249","ondisk_log_start":"366'2249","created":372,"last_epoch_clean":404,"parent":"0.0","parent_split_bits":5,"last_scrub":"412'7550","last_scrub_stamp":"2024-03-11T21:43:29.057514+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:40:38.219602+0000","last_clean_scrub_stamp":"2024-03-11T21:43:29.057514+0000","objects_scrubbed":1610,"log_size":5818,"log_dups_size":0,"ondisk_log_size":5818,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:44:48.927453+0000","scrub_duration":413,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":71172096,"num_objects":1086,"num_object_clones":0,"num_object_copies":3258,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1086,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":524,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,1],"acting":[3,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":758,"reported_epoch":415,"state":"active+clean","last_fresh":"2024-03-11T21:43:44.084396+0000","last_change":"2024-03-11T21:41:13.893454+0000","last_active":"2024-03-11T21:43:44.084396+0000","last_peered":"2024-03-11T21:43:44.084396+0000","last_clean":"2024-03-11T21:43:44.084396+0000","last_became_active":"2024-03-11T21:36:35.068453+0000","last_became_peered":"2024-03-11T21:36:35.068453+0000","last_unstale":"2024-03-11T21:43:44.084396+0000","last_undegraded":"2024-03-11T21:43:44.084396+0000","last_fullsized":"2024-03-11T21:43:44.084396+0000","mapping_epoch":277,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":278,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:41:13.893233+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:32:12.654573+0000","last_clean_scrub_stamp":"2024-03-11T21:41:13.893233+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:43:45.574613+0000","scrub_duration":78,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3],"acting":[5,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":404,"reported_epoch":414,"state":"active+clean","last_fresh":"2024-03-11T21:43:37.035858+0000","last_change":"2024-03-11T21:41:01.164718+0000","last_active":"2024-03-11T21:43:37.035858+0000","last_peered":"2024-03-11T21:43:37.035858+0000","last_clean":"2024-03-11T21:43:37.035858+0000","last_became_active":"2024-03-11T21:34:41.145299+0000","last_became_peered":"2024-03-11T21:34:41.145299+0000","last_unstale":"2024-03-11T21:43:37.035858+0000","last_undegraded":"2024-03-11T21:43:37.035858+0000","last_fullsized":"2024-03-11T21:43:37.035858+0000","mapping_epoch":226,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":227,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:41:01.164619+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:41.544550+0000","last_clean_scrub_stamp":"2024-03-11T21:41:01.164619+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.1d","version":"0'0","reported_seq":767,"reported_epoch":414,"state":"active+clean","last_fresh":"2024-03-11T21:43:37.034947+0000","last_change":"2024-03-11T21:41:02.217938+0000","last_active":"2024-03-11T21:43:37.034947+0000","last_peered":"2024-03-11T21:43:37.034947+0000","last_clean":"2024-03-11T21:43:37.034947+0000","last_became_active":"2024-03-11T21:25:38.542628+0000","last_became_peered":"2024-03-11T21:25:38.542628+0000","last_unstale":"2024-03-11T21:43:37.034947+0000","last_undegraded":"2024-03-11T21:43:37.034947+0000","last_fullsized":"2024-03-11T21:43:37.034947+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:41:02.217799+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:37.512018+0000","last_clean_scrub_stamp":"2024-03-11T21:41:02.217799+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,0],"acting":[7,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.1b","version":"260'7","reported_seq":785,"reported_epoch":415,"state":"active+clean","last_fresh":"2024-03-11T21:43:44.075436+0000","last_change":"2024-03-11T21:43:36.255756+0000","last_active":"2024-03-11T21:43:44.075436+0000","last_peered":"2024-03-11T21:43:44.075436+0000","last_clean":"2024-03-11T21:43:44.075436+0000","last_became_active":"2024-03-11T21:36:35.057678+0000","last_became_peered":"2024-03-11T21:36:35.057678+0000","last_unstale":"2024-03-11T21:43:44.075436+0000","last_undegraded":"2024-03-11T21:43:44.075436+0000","last_fullsized":"2024-03-11T21:43:44.075436+0000","mapping_epoch":277,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":278,"parent":"0.0","parent_split_bits":0,"last_scrub":"260'7","last_scrub_stamp":"2024-03-11T21:43:36.255599+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:39.525554+0000","last_clean_scrub_stamp":"2024-03-11T21:43:36.255599+0000","objects_scrubbed":2,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:44:41.777765+0000","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":14,"num_read_kb":10,"num_write":8,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,1],"acting":[4,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"10.14","version":"414'7738","reported_seq":14524,"reported_epoch":414,"state":"active+clean","last_fresh":"2024-03-11T21:43:43.439018+0000","last_change":"2024-03-11T21:43:28.545835+0000","last_active":"2024-03-11T21:43:43.439018+0000","last_peered":"2024-03-11T21:43:43.439018+0000","last_clean":"2024-03-11T21:43:43.439018+0000","last_became_active":"2024-03-11T21:42:33.464147+0000","last_became_peered":"2024-03-11T21:42:33.464147+0000","last_unstale":"2024-03-11T21:43:43.439018+0000","last_undegraded":"2024-03-11T21:43:43.439018+0000","last_fullsized":"2024-03-11T21:43:43.439018+0000","mapping_epoch":397,"log_start":"365'1477","ondisk_log_start":"365'1477","created":372,"last_epoch_clean":398,"parent":"0.0","parent_split_bits":5,"last_scrub":"412'7251","last_scrub_stamp":"2024-03-11T21:43:28.545495+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:40:38.219602+0000","last_clean_scrub_stamp":"2024-03-11T21:43:28.545495+0000","objects_scrubbed":1680,"log_size":6261,"log_dups_size":0,"ondisk_log_size":6261,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:44:47.964644+0000","scrub_duration":372,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":78053376,"num_objects":1191,"num_object_clones":0,"num_object_copies":3573,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1191,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":489,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,4],"acting":[1,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":750,"reported_epoch":415,"state":"active+clean","last_fresh":"2024-03-11T21:43:44.084365+0000","last_change":"2024-03-11T21:43:22.157433+0000","last_active":"2024-03-11T21:43:44.084365+0000","last_peered":"2024-03-11T21:43:44.084365+0000","last_clean":"2024-03-11T21:43:44.084365+0000","last_became_active":"2024-03-11T21:25:44.589786+0000","last_became_peered":"2024-03-11T21:25:44.589786+0000","last_unstale":"2024-03-11T21:43:44.084365+0000","last_undegraded":"2024-03-11T21:43:44.084365+0000","last_fullsized":"2024-03-11T21:43:44.084365+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:43:22.156831+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:43.559452+0000","last_clean_scrub_stamp":"2024-03-11T21:43:22.156831+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:44:25.859937+0000","scrub_duration":93,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,7],"acting":[0,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":773,"reported_epoch":415,"state":"active+clean","last_fresh":"2024-03-11T21:43:44.084488+0000","last_change":"2024-03-11T21:43:35.536286+0000","last_active":"2024-03-11T21:43:44.084488+0000","last_peered":"2024-03-11T21:43:44.084488+0000","last_clean":"2024-03-11T21:43:44.084488+0000","last_became_active":"2024-03-11T21:34:41.147185+0000","last_became_peered":"2024-03-11T21:34:41.147185+0000","last_unstale":"2024-03-11T21:43:44.084488+0000","last_undegraded":"2024-03-11T21:43:44.084488+0000","last_fullsized":"2024-03-11T21:43:44.084488+0000","mapping_epoch":226,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":227,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:43:35.536164+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:43:35.536164+0000","last_clean_scrub_stamp":"2024-03-11T21:43:35.536164+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:45:01.908207+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,7],"acting":[5,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1c","version":"0'0","reported_seq":767,"reported_epoch":414,"state":"active+clean","last_fresh":"2024-03-11T21:43:37.039949+0000","last_change":"2024-03-11T21:41:14.385957+0000","last_active":"2024-03-11T21:43:37.039949+0000","last_peered":"2024-03-11T21:43:37.039949+0000","last_clean":"2024-03-11T21:43:37.039949+0000","last_became_active":"2024-03-11T21:25:38.541364+0000","last_became_peered":"2024-03-11T21:25:38.541364+0000","last_unstale":"2024-03-11T21:43:37.039949+0000","last_undegraded":"2024-03-11T21:43:37.039949+0000","last_fullsized":"2024-03-11T21:43:37.039949+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:41:14.385682+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:28:32.566437+0000","last_clean_scrub_stamp":"2024-03-11T21:41:14.385682+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":27,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.1a","version":"350'19","reported_seq":813,"reported_epoch":415,"state":"active+clean","last_fresh":"2024-03-11T21:43:44.075553+0000","last_change":"2024-03-11T21:43:25.898959+0000","last_active":"2024-03-11T21:43:44.075553+0000","last_peered":"2024-03-11T21:43:44.075553+0000","last_clean":"2024-03-11T21:43:44.075553+0000","last_became_active":"2024-03-11T21:36:35.057029+0000","last_became_peered":"2024-03-11T21:36:35.057029+0000","last_unstale":"2024-03-11T21:43:44.075553+0000","last_undegraded":"2024-03-11T21:43:44.075553+0000","last_fullsized":"2024-03-11T21:43:44.075553+0000","mapping_epoch":277,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":278,"parent":"0.0","parent_split_bits":0,"last_scrub":"350'19","last_scrub_stamp":"2024-03-11T21:43:25.898708+0000","last_deep_scrub":"350'19","last_deep_scrub_stamp":"2024-03-11T21:43:25.898708+0000","last_clean_scrub_stamp":"2024-03-11T21:43:25.898708+0000","objects_scrubbed":4,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:44:26.480834+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0],"acting":[4,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"10.13","version":"414'7785","reported_seq":14517,"reported_epoch":414,"state":"active+clean","last_fresh":"2024-03-11T21:43:43.409545+0000","last_change":"2024-03-11T21:43:27.572518+0000","last_active":"2024-03-11T21:43:43.409545+0000","last_peered":"2024-03-11T21:43:43.409545+0000","last_clean":"2024-03-11T21:43:43.409545+0000","last_became_active":"2024-03-11T21:43:20.870191+0000","last_became_peered":"2024-03-11T21:43:20.870191+0000","last_unstale":"2024-03-11T21:43:43.409545+0000","last_undegraded":"2024-03-11T21:43:43.409545+0000","last_fullsized":"2024-03-11T21:43:43.409545+0000","mapping_epoch":407,"log_start":"365'1409","ondisk_log_start":"365'1409","created":372,"last_epoch_clean":408,"parent":"0.0","parent_split_bits":5,"last_scrub":"412'7266","last_scrub_stamp":"2024-03-11T21:43:27.572401+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:40:38.219602+0000","last_clean_scrub_stamp":"2024-03-11T21:43:27.572401+0000","objects_scrubbed":1700,"log_size":6376,"log_dups_size":0,"ondisk_log_size":6376,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:44:30.154382+0000","scrub_duration":412,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":77070336,"num_objects":1176,"num_object_clones":0,"num_object_copies":3528,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1176,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":524,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,2],"acting":[1,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1f","version":"0'0","reported_seq":626,"reported_epoch":415,"state":"active+clean","last_fresh":"2024-03-11T21:43:44.083884+0000","last_change":"2024-03-11T21:41:08.652958+0000","last_active":"2024-03-11T21:43:44.083884+0000","last_peered":"2024-03-11T21:43:44.083884+0000","last_clean":"2024-03-11T21:43:44.083884+0000","last_became_active":"2024-03-11T21:36:35.071755+0000","last_became_peered":"2024-03-11T21:36:35.071755+0000","last_unstale":"2024-03-11T21:43:44.083884+0000","last_undegraded":"2024-03-11T21:43:44.083884+0000","last_fullsized":"2024-03-11T21:43:44.083884+0000","mapping_epoch":277,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":278,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:41:08.652836+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:43.559452+0000","last_clean_scrub_stamp":"2024-03-11T21:41:08.652836+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":137,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1c","version":"0'0","reported_seq":763,"reported_epoch":415,"state":"active+clean","last_fresh":"2024-03-11T21:43:44.076821+0000","last_change":"2024-03-11T21:43:32.021607+0000","last_active":"2024-03-11T21:43:44.076821+0000","last_peered":"2024-03-11T21:43:44.076821+0000","last_clean":"2024-03-11T21:43:44.076821+0000","last_became_active":"2024-03-11T21:36:35.067127+0000","last_became_peered":"2024-03-11T21:36:35.067127+0000","last_unstale":"2024-03-11T21:43:44.076821+0000","last_undegraded":"2024-03-11T21:43:44.076821+0000","last_fullsized":"2024-03-11T21:43:44.076821+0000","mapping_epoch":277,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":278,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:43:32.021344+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:43:32.021344+0000","last_clean_scrub_stamp":"2024-03-11T21:43:32.021344+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:44:42.796100+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1b","version":"0'0","reported_seq":674,"reported_epoch":415,"state":"active+clean","last_fresh":"2024-03-11T21:43:44.089096+0000","last_change":"2024-03-11T21:41:23.590257+0000","last_active":"2024-03-11T21:43:44.089096+0000","last_peered":"2024-03-11T21:43:44.089096+0000","last_clean":"2024-03-11T21:43:44.089096+0000","last_became_active":"2024-03-11T21:36:35.073959+0000","last_became_peered":"2024-03-11T21:36:35.073959+0000","last_unstale":"2024-03-11T21:43:44.089096+0000","last_undegraded":"2024-03-11T21:43:44.089096+0000","last_fullsized":"2024-03-11T21:43:44.089096+0000","mapping_epoch":277,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":278,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:41:23.589546+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:34:27.998724+0000","last_clean_scrub_stamp":"2024-03-11T21:41:23.589546+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1d","version":"260'18","reported_seq":710,"reported_epoch":415,"state":"active+clean","last_fresh":"2024-03-11T21:43:44.084939+0000","last_change":"2024-03-11T21:36:35.080207+0000","last_active":"2024-03-11T21:43:44.084939+0000","last_peered":"2024-03-11T21:43:44.084939+0000","last_clean":"2024-03-11T21:43:44.084939+0000","last_became_active":"2024-03-11T21:36:35.055880+0000","last_became_peered":"2024-03-11T21:36:35.055880+0000","last_unstale":"2024-03-11T21:43:44.084939+0000","last_undegraded":"2024-03-11T21:43:44.084939+0000","last_fullsized":"2024-03-11T21:43:44.084939+0000","mapping_epoch":277,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":278,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'12","last_scrub_stamp":"2024-03-11T21:35:07.971581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:39.525554+0000","last_clean_scrub_stamp":"2024-03-11T21:35:07.971581+0000","objects_scrubbed":0,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":33,"num_read_kb":24,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.12","version":"414'7992","reported_seq":17399,"reported_epoch":414,"state":"active+clean","last_fresh":"2024-03-11T21:43:42.497693+0000","last_change":"2024-03-11T21:43:23.063128+0000","last_active":"2024-03-11T21:43:42.497693+0000","last_peered":"2024-03-11T21:43:42.497693+0000","last_clean":"2024-03-11T21:43:42.497693+0000","last_became_active":"2024-03-11T21:43:21.868307+0000","last_became_peered":"2024-03-11T21:43:21.868307+0000","last_unstale":"2024-03-11T21:43:42.497693+0000","last_undegraded":"2024-03-11T21:43:42.497693+0000","last_fullsized":"2024-03-11T21:43:42.497693+0000","mapping_epoch":408,"log_start":"364'1214","ondisk_log_start":"364'1214","created":372,"last_epoch_clean":409,"parent":"0.0","parent_split_bits":5,"last_scrub":"409'7349","last_scrub_stamp":"2024-03-11T21:43:23.060318+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:40:38.219602+0000","last_clean_scrub_stamp":"2024-03-11T21:43:23.060318+0000","objects_scrubbed":1883,"log_size":6778,"log_dups_size":0,"ondisk_log_size":6778,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:44:32.875071+0000","scrub_duration":649,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":80609280,"num_objects":1230,"num_object_clones":0,"num_object_copies":3690,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1230,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":653,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,5],"acting":[6,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.1e","version":"0'0","reported_seq":743,"reported_epoch":415,"state":"active+clean","last_fresh":"2024-03-11T21:43:44.075750+0000","last_change":"2024-03-11T21:41:10.551185+0000","last_active":"2024-03-11T21:43:44.075750+0000","last_peered":"2024-03-11T21:43:44.075750+0000","last_clean":"2024-03-11T21:43:44.075750+0000","last_became_active":"2024-03-11T21:25:45.300355+0000","last_became_peered":"2024-03-11T21:25:45.300355+0000","last_unstale":"2024-03-11T21:43:44.075750+0000","last_undegraded":"2024-03-11T21:43:44.075750+0000","last_fullsized":"2024-03-11T21:43:44.075750+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:41:10.550887+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:43.559452+0000","last_clean_scrub_stamp":"2024-03-11T21:41:10.550887+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:43:48.166483+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5],"acting":[4,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1d","version":"0'0","reported_seq":752,"reported_epoch":414,"state":"active+clean","last_fresh":"2024-03-11T21:43:37.045255+0000","last_change":"2024-03-11T21:43:36.053527+0000","last_active":"2024-03-11T21:43:37.045255+0000","last_peered":"2024-03-11T21:43:37.045255+0000","last_clean":"2024-03-11T21:43:37.045255+0000","last_became_active":"2024-03-11T21:25:42.579407+0000","last_became_peered":"2024-03-11T21:25:42.579407+0000","last_unstale":"2024-03-11T21:43:37.045255+0000","last_undegraded":"2024-03-11T21:43:37.045255+0000","last_fullsized":"2024-03-11T21:43:37.045255+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:43:36.053418+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:30:18.678531+0000","last_clean_scrub_stamp":"2024-03-11T21:43:36.053418+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:45:02.948381+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1a","version":"0'0","reported_seq":788,"reported_epoch":414,"state":"active+clean","last_fresh":"2024-03-11T21:43:37.038099+0000","last_change":"2024-03-11T21:41:31.856350+0000","last_active":"2024-03-11T21:43:37.038099+0000","last_peered":"2024-03-11T21:43:37.038099+0000","last_clean":"2024-03-11T21:43:37.038099+0000","last_became_active":"2024-03-11T21:34:41.145014+0000","last_became_peered":"2024-03-11T21:34:41.145014+0000","last_unstale":"2024-03-11T21:43:37.038099+0000","last_undegraded":"2024-03-11T21:43:37.038099+0000","last_fullsized":"2024-03-11T21:43:37.038099+0000","mapping_epoch":226,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":227,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:41:31.856243+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:37.512018+0000","last_clean_scrub_stamp":"2024-03-11T21:41:31.856243+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.1c","version":"260'27","reported_seq":821,"reported_epoch":414,"state":"active+clean","last_fresh":"2024-03-11T21:43:37.030991+0000","last_change":"2024-03-11T21:43:31.103413+0000","last_active":"2024-03-11T21:43:37.030991+0000","last_peered":"2024-03-11T21:43:37.030991+0000","last_clean":"2024-03-11T21:43:37.030991+0000","last_became_active":"2024-03-11T21:36:35.074579+0000","last_became_peered":"2024-03-11T21:36:35.074579+0000","last_unstale":"2024-03-11T21:43:37.030991+0000","last_undegraded":"2024-03-11T21:43:37.030991+0000","last_fullsized":"2024-03-11T21:43:37.030991+0000","mapping_epoch":277,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":278,"parent":"0.0","parent_split_bits":0,"last_scrub":"260'27","last_scrub_stamp":"2024-03-11T21:43:31.103274+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:39.525554+0000","last_clean_scrub_stamp":"2024-03-11T21:43:31.103274+0000","objects_scrubbed":7,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:44:32.059741+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":44,"num_read_kb":35,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"10.11","version":"415'9609","reported_seq":14088,"reported_epoch":415,"state":"active+clean","last_fresh":"2024-03-11T21:43:47.074504+0000","last_change":"2024-03-11T21:43:34.236856+0000","last_active":"2024-03-11T21:43:47.074504+0000","last_peered":"2024-03-11T21:43:47.074504+0000","last_clean":"2024-03-11T21:43:47.074504+0000","last_became_active":"2024-03-11T21:42:09.430469+0000","last_became_peered":"2024-03-11T21:42:09.430469+0000","last_unstale":"2024-03-11T21:43:47.074504+0000","last_undegraded":"2024-03-11T21:43:47.074504+0000","last_fullsized":"2024-03-11T21:43:47.074504+0000","mapping_epoch":389,"log_start":"373'4386","ondisk_log_start":"373'4386","created":372,"last_epoch_clean":390,"parent":"0.0","parent_split_bits":5,"last_scrub":"413'9163","last_scrub_stamp":"2024-03-11T21:43:34.236691+0000","last_deep_scrub":"413'9163","last_deep_scrub_stamp":"2024-03-11T21:43:34.236691+0000","last_clean_scrub_stamp":"2024-03-11T21:43:34.236691+0000","objects_scrubbed":1646,"log_size":5223,"log_dups_size":0,"ondisk_log_size":5223,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":10,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:44:34.624498+0000","scrub_duration":9242,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":69992448,"num_objects":1068,"num_object_clones":0,"num_object_copies":3204,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1068,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":578,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,7],"acting":[5,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.1d","version":"0'0","reported_seq":740,"reported_epoch":414,"state":"active+clean","last_fresh":"2024-03-11T21:43:37.044782+0000","last_change":"2024-03-11T21:41:16.533828+0000","last_active":"2024-03-11T21:43:37.044782+0000","last_peered":"2024-03-11T21:43:37.044782+0000","last_clean":"2024-03-11T21:43:37.044782+0000","last_became_active":"2024-03-11T21:25:44.588845+0000","last_became_peered":"2024-03-11T21:25:44.588845+0000","last_unstale":"2024-03-11T21:43:37.044782+0000","last_undegraded":"2024-03-11T21:43:37.044782+0000","last_fullsized":"2024-03-11T21:43:37.044782+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:41:16.533692+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:43.559452+0000","last_clean_scrub_stamp":"2024-03-11T21:41:16.533692+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":207,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.1e","version":"0'0","reported_seq":755,"reported_epoch":415,"state":"active+clean","last_fresh":"2024-03-11T21:43:44.088421+0000","last_change":"2024-03-11T21:40:28.163223+0000","last_active":"2024-03-11T21:43:44.088421+0000","last_peered":"2024-03-11T21:43:44.088421+0000","last_clean":"2024-03-11T21:43:44.088421+0000","last_became_active":"2024-03-11T21:25:42.582959+0000","last_became_peered":"2024-03-11T21:25:42.582959+0000","last_unstale":"2024-03-11T21:43:44.088421+0000","last_undegraded":"2024-03-11T21:43:44.088421+0000","last_fullsized":"2024-03-11T21:43:44.088421+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:40:28.163142+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:26:58.335431+0000","last_clean_scrub_stamp":"2024-03-11T21:40:28.163142+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:43:49.041258+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.19","version":"67'2","reported_seq":700,"reported_epoch":415,"state":"active+clean","last_fresh":"2024-03-11T21:43:44.082627+0000","last_change":"2024-03-11T21:41:01.612764+0000","last_active":"2024-03-11T21:43:44.082627+0000","last_peered":"2024-03-11T21:43:44.082627+0000","last_clean":"2024-03-11T21:43:44.082627+0000","last_became_active":"2024-03-11T21:36:35.064249+0000","last_became_peered":"2024-03-11T21:36:35.064249+0000","last_unstale":"2024-03-11T21:43:44.082627+0000","last_undegraded":"2024-03-11T21:43:44.082627+0000","last_fullsized":"2024-03-11T21:43:44.082627+0000","mapping_epoch":277,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":278,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'2","last_scrub_stamp":"2024-03-11T21:41:01.612632+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:37.512018+0000","last_clean_scrub_stamp":"2024-03-11T21:41:01.612632+0000","objects_scrubbed":2,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":957,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,0],"acting":[3,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1f","version":"260'24","reported_seq":827,"reported_epoch":414,"state":"active+clean","last_fresh":"2024-03-11T21:43:37.042588+0000","last_change":"2024-03-11T21:40:25.079945+0000","last_active":"2024-03-11T21:43:37.042588+0000","last_peered":"2024-03-11T21:43:37.042588+0000","last_clean":"2024-03-11T21:43:37.042588+0000","last_became_active":"2024-03-11T21:32:22.798242+0000","last_became_peered":"2024-03-11T21:32:22.798242+0000","last_unstale":"2024-03-11T21:43:37.042588+0000","last_undegraded":"2024-03-11T21:43:37.042588+0000","last_fullsized":"2024-03-11T21:43:37.042588+0000","mapping_epoch":166,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":167,"parent":"0.0","parent_split_bits":0,"last_scrub":"260'24","last_scrub_stamp":"2024-03-11T21:40:25.079808+0000","last_deep_scrub":"124'16","last_deep_scrub_stamp":"2024-03-11T21:35:38.173841+0000","last_clean_scrub_stamp":"2024-03-11T21:40:25.079808+0000","objects_scrubbed":5,"log_size":24,"log_dups_size":0,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":32,"num_read_kb":25,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,0],"acting":[6,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"10.10","version":"415'9593","reported_seq":14322,"reported_epoch":415,"state":"active+clean","last_fresh":"2024-03-11T21:43:47.140701+0000","last_change":"2024-03-11T21:43:35.291190+0000","last_active":"2024-03-11T21:43:47.140701+0000","last_peered":"2024-03-11T21:43:47.140701+0000","last_clean":"2024-03-11T21:43:47.140701+0000","last_became_active":"2024-03-11T21:41:47.077087+0000","last_became_peered":"2024-03-11T21:41:47.077087+0000","last_unstale":"2024-03-11T21:43:47.140701+0000","last_undegraded":"2024-03-11T21:43:47.140701+0000","last_fullsized":"2024-03-11T21:43:47.140701+0000","mapping_epoch":380,"log_start":"385'5040","ondisk_log_start":"385'5040","created":372,"last_epoch_clean":381,"parent":"0.0","parent_split_bits":5,"last_scrub":"413'9161","last_scrub_stamp":"2024-03-11T21:43:35.290968+0000","last_deep_scrub":"413'9161","last_deep_scrub_stamp":"2024-03-11T21:43:35.290968+0000","last_clean_scrub_stamp":"2024-03-11T21:43:35.290968+0000","objects_scrubbed":1584,"log_size":4553,"log_dups_size":0,"ondisk_log_size":4553,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":9,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:44:42.163948+0000","scrub_duration":8353,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":67567616,"num_objects":1031,"num_object_clones":0,"num_object_copies":3093,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1031,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":553,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,6],"acting":[4,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1c","version":"74'1","reported_seq":408,"reported_epoch":414,"state":"active+clean","last_fresh":"2024-03-11T21:43:37.039850+0000","last_change":"2024-03-11T21:43:20.543062+0000","last_active":"2024-03-11T21:43:37.039850+0000","last_peered":"2024-03-11T21:43:37.039850+0000","last_clean":"2024-03-11T21:43:37.039850+0000","last_became_active":"2024-03-11T21:34:41.144839+0000","last_became_peered":"2024-03-11T21:34:41.144839+0000","last_unstale":"2024-03-11T21:43:37.039850+0000","last_undegraded":"2024-03-11T21:43:37.039850+0000","last_fullsized":"2024-03-11T21:43:37.039850+0000","mapping_epoch":226,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":227,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'1","last_scrub_stamp":"2024-03-11T21:43:20.542918+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:43.559452+0000","last_clean_scrub_stamp":"2024-03-11T21:43:20.542918+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:44:38.671474+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":370,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":370,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.1f","version":"229'10","reported_seq":1025,"reported_epoch":414,"state":"active+clean","last_fresh":"2024-03-11T21:43:37.042542+0000","last_change":"2024-03-11T21:41:09.868735+0000","last_active":"2024-03-11T21:43:37.042542+0000","last_peered":"2024-03-11T21:43:37.042542+0000","last_clean":"2024-03-11T21:43:37.042542+0000","last_became_active":"2024-03-11T21:34:41.145862+0000","last_became_peered":"2024-03-11T21:34:41.145862+0000","last_unstale":"2024-03-11T21:43:37.042542+0000","last_undegraded":"2024-03-11T21:43:37.042542+0000","last_fullsized":"2024-03-11T21:43:37.042542+0000","mapping_epoch":226,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":227,"parent":"0.0","parent_split_bits":0,"last_scrub":"229'10","last_scrub_stamp":"2024-03-11T21:41:09.868606+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:41.544550+0000","last_clean_scrub_stamp":"2024-03-11T21:41:09.868606+0000","objects_scrubbed":1,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:43:42.400352+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.18","version":"67'1","reported_seq":805,"reported_epoch":415,"state":"active+clean","last_fresh":"2024-03-11T21:43:44.084137+0000","last_change":"2024-03-11T21:43:33.566815+0000","last_active":"2024-03-11T21:43:44.084137+0000","last_peered":"2024-03-11T21:43:44.084137+0000","last_clean":"2024-03-11T21:43:44.084137+0000","last_became_active":"2024-03-11T21:35:23.310241+0000","last_became_peered":"2024-03-11T21:35:23.310241+0000","last_unstale":"2024-03-11T21:43:44.084137+0000","last_undegraded":"2024-03-11T21:43:44.084137+0000","last_fullsized":"2024-03-11T21:43:44.084137+0000","mapping_epoch":248,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":249,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'1","last_scrub_stamp":"2024-03-11T21:43:33.566666+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:37.512018+0000","last_clean_scrub_stamp":"2024-03-11T21:43:33.566666+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:44:58.189435+0000","scrub_duration":24,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":436,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":7,"num_read_kb":7,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":1308,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,2],"acting":[5,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1e","version":"260'20","reported_seq":847,"reported_epoch":415,"state":"active+clean","last_fresh":"2024-03-11T21:43:44.088383+0000","last_change":"2024-03-11T21:41:10.256209+0000","last_active":"2024-03-11T21:43:44.088383+0000","last_peered":"2024-03-11T21:43:44.088383+0000","last_clean":"2024-03-11T21:43:44.088383+0000","last_became_active":"2024-03-11T21:36:35.057843+0000","last_became_peered":"2024-03-11T21:36:35.057843+0000","last_unstale":"2024-03-11T21:43:44.088383+0000","last_undegraded":"2024-03-11T21:43:44.088383+0000","last_fullsized":"2024-03-11T21:43:44.088383+0000","mapping_epoch":277,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":278,"parent":"0.0","parent_split_bits":0,"last_scrub":"260'20","last_scrub_stamp":"2024-03-11T21:41:10.255971+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:39.525554+0000","last_clean_scrub_stamp":"2024-03-11T21:41:10.255971+0000","objects_scrubbed":5,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:43:49.041429+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":25,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.13","version":"0'0","reported_seq":634,"reported_epoch":415,"state":"active+clean","last_fresh":"2024-03-11T21:43:44.089308+0000","last_change":"2024-03-11T21:41:26.639950+0000","last_active":"2024-03-11T21:43:44.089308+0000","last_peered":"2024-03-11T21:43:44.089308+0000","last_clean":"2024-03-11T21:43:44.089308+0000","last_became_active":"2024-03-11T21:36:35.062618+0000","last_became_peered":"2024-03-11T21:36:35.062618+0000","last_unstale":"2024-03-11T21:43:44.089308+0000","last_undegraded":"2024-03-11T21:43:44.089308+0000","last_fullsized":"2024-03-11T21:43:44.089308+0000","mapping_epoch":277,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":278,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:41:26.639290+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:43.559452+0000","last_clean_scrub_stamp":"2024-03-11T21:41:26.639290+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":114,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":412,"reported_epoch":414,"state":"active+clean","last_fresh":"2024-03-11T21:43:37.036193+0000","last_change":"2024-03-11T21:43:26.665857+0000","last_active":"2024-03-11T21:43:37.036193+0000","last_peered":"2024-03-11T21:43:37.036193+0000","last_clean":"2024-03-11T21:43:37.036193+0000","last_became_active":"2024-03-11T21:34:41.142569+0000","last_became_peered":"2024-03-11T21:34:41.142569+0000","last_unstale":"2024-03-11T21:43:37.036193+0000","last_undegraded":"2024-03-11T21:43:37.036193+0000","last_fullsized":"2024-03-11T21:43:37.036193+0000","mapping_epoch":226,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":227,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:43:26.665719+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:43:26.665719+0000","last_clean_scrub_stamp":"2024-03-11T21:43:26.665719+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:44:46.668281+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.17","version":"0'0","reported_seq":776,"reported_epoch":414,"state":"active+clean","last_fresh":"2024-03-11T21:43:37.038811+0000","last_change":"2024-03-11T21:41:26.816265+0000","last_active":"2024-03-11T21:43:37.038811+0000","last_peered":"2024-03-11T21:43:37.038811+0000","last_clean":"2024-03-11T21:43:37.038811+0000","last_became_active":"2024-03-11T21:25:38.540349+0000","last_became_peered":"2024-03-11T21:25:38.540349+0000","last_unstale":"2024-03-11T21:43:37.038811+0000","last_undegraded":"2024-03-11T21:43:37.038811+0000","last_fullsized":"2024-03-11T21:43:37.038811+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:41:26.816135+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:28:25.918101+0000","last_clean_scrub_stamp":"2024-03-11T21:41:26.816135+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,2],"acting":[6,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.11","version":"260'19","reported_seq":760,"reported_epoch":415,"state":"active+clean","last_fresh":"2024-03-11T21:43:44.083595+0000","last_change":"2024-03-11T21:43:29.688029+0000","last_active":"2024-03-11T21:43:44.083595+0000","last_peered":"2024-03-11T21:43:44.083595+0000","last_clean":"2024-03-11T21:43:44.083595+0000","last_became_active":"2024-03-11T21:36:35.073689+0000","last_became_peered":"2024-03-11T21:36:35.073689+0000","last_unstale":"2024-03-11T21:43:44.083595+0000","last_undegraded":"2024-03-11T21:43:44.083595+0000","last_fullsized":"2024-03-11T21:43:44.083595+0000","mapping_epoch":277,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":278,"parent":"0.0","parent_split_bits":0,"last_scrub":"260'19","last_scrub_stamp":"2024-03-11T21:43:29.687897+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:39.525554+0000","last_clean_scrub_stamp":"2024-03-11T21:43:29.687897+0000","objects_scrubbed":5,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:44:31.965084+0000","scrub_duration":31,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":31,"num_read_kb":24,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":404,"reported_epoch":414,"state":"active+clean","last_fresh":"2024-03-11T21:43:37.033885+0000","last_change":"2024-03-11T21:41:16.345639+0000","last_active":"2024-03-11T21:43:37.033885+0000","last_peered":"2024-03-11T21:43:37.033885+0000","last_clean":"2024-03-11T21:43:37.033885+0000","last_became_active":"2024-03-11T21:34:41.137235+0000","last_became_peered":"2024-03-11T21:34:41.137235+0000","last_unstale":"2024-03-11T21:43:37.033885+0000","last_undegraded":"2024-03-11T21:43:37.033885+0000","last_fullsized":"2024-03-11T21:43:37.033885+0000","mapping_epoch":226,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":227,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:41:16.345432+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:43.559452+0000","last_clean_scrub_stamp":"2024-03-11T21:41:16.345432+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,4],"acting":[7,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":788,"reported_epoch":414,"state":"active+clean","last_fresh":"2024-03-11T21:43:37.035964+0000","last_change":"2024-03-11T21:43:29.425546+0000","last_active":"2024-03-11T21:43:37.035964+0000","last_peered":"2024-03-11T21:43:37.035964+0000","last_clean":"2024-03-11T21:43:37.035964+0000","last_became_active":"2024-03-11T21:34:41.156036+0000","last_became_peered":"2024-03-11T21:34:41.156036+0000","last_unstale":"2024-03-11T21:43:37.035964+0000","last_undegraded":"2024-03-11T21:43:37.035964+0000","last_fullsized":"2024-03-11T21:43:37.035964+0000","mapping_epoch":226,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":227,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:43:29.425414+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:43:29.425414+0000","last_clean_scrub_stamp":"2024-03-11T21:43:29.425414+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:44:36.703967+0000","scrub_duration":31,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.16","version":"0'0","reported_seq":769,"reported_epoch":415,"state":"active+clean","last_fresh":"2024-03-11T21:43:44.085460+0000","last_change":"2024-03-11T21:41:25.953645+0000","last_active":"2024-03-11T21:43:44.085460+0000","last_peered":"2024-03-11T21:43:44.085460+0000","last_clean":"2024-03-11T21:43:44.085460+0000","last_became_active":"2024-03-11T21:25:38.539103+0000","last_became_peered":"2024-03-11T21:25:38.539103+0000","last_unstale":"2024-03-11T21:43:44.085460+0000","last_undegraded":"2024-03-11T21:43:44.085460+0000","last_fullsized":"2024-03-11T21:43:44.085460+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:41:25.953529+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:27:11.938909+0000","last_clean_scrub_stamp":"2024-03-11T21:41:25.953529+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:43:45.574802+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,2],"acting":[5,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.10","version":"260'10","reported_seq":642,"reported_epoch":415,"state":"active+clean","last_fresh":"2024-03-11T21:43:44.086687+0000","last_change":"2024-03-11T21:41:17.565434+0000","last_active":"2024-03-11T21:43:44.086687+0000","last_peered":"2024-03-11T21:43:44.086687+0000","last_clean":"2024-03-11T21:43:44.086687+0000","last_became_active":"2024-03-11T21:36:35.067500+0000","last_became_peered":"2024-03-11T21:36:35.067500+0000","last_unstale":"2024-03-11T21:43:44.086687+0000","last_undegraded":"2024-03-11T21:43:44.086687+0000","last_fullsized":"2024-03-11T21:43:44.086687+0000","mapping_epoch":277,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":278,"parent":"0.0","parent_split_bits":0,"last_scrub":"260'10","last_scrub_stamp":"2024-03-11T21:41:17.565310+0000","last_deep_scrub":"94'6","last_deep_scrub_stamp":"2024-03-11T21:28:24.603974+0000","last_clean_scrub_stamp":"2024-03-11T21:41:17.565310+0000","objects_scrubbed":2,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":12,"num_read_kb":10,"num_write":8,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6],"acting":[3,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":687,"reported_epoch":415,"state":"active+clean","last_fresh":"2024-03-11T21:43:44.083782+0000","last_change":"2024-03-11T21:43:32.679847+0000","last_active":"2024-03-11T21:43:44.083782+0000","last_peered":"2024-03-11T21:43:44.083782+0000","last_clean":"2024-03-11T21:43:44.083782+0000","last_became_active":"2024-03-11T21:36:35.070176+0000","last_became_peered":"2024-03-11T21:36:35.070176+0000","last_unstale":"2024-03-11T21:43:44.083782+0000","last_undegraded":"2024-03-11T21:43:44.083782+0000","last_fullsized":"2024-03-11T21:43:44.083782+0000","mapping_epoch":277,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":278,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:43:32.679729+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:43.559452+0000","last_clean_scrub_stamp":"2024-03-11T21:43:32.679729+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:44:49.731766+0000","scrub_duration":54,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":753,"reported_epoch":414,"state":"active+clean","last_fresh":"2024-03-11T21:43:37.049665+0000","last_change":"2024-03-11T21:41:04.276105+0000","last_active":"2024-03-11T21:43:37.049665+0000","last_peered":"2024-03-11T21:43:37.049665+0000","last_clean":"2024-03-11T21:43:37.049665+0000","last_became_active":"2024-03-11T21:36:35.057906+0000","last_became_peered":"2024-03-11T21:36:35.057906+0000","last_unstale":"2024-03-11T21:43:37.049665+0000","last_undegraded":"2024-03-11T21:43:37.049665+0000","last_fullsized":"2024-03-11T21:43:37.049665+0000","mapping_epoch":277,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":278,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:41:04.275820+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:27:21.207303+0000","last_clean_scrub_stamp":"2024-03-11T21:41:04.275820+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.15","version":"0'0","reported_seq":760,"reported_epoch":414,"state":"active+clean","last_fresh":"2024-03-11T21:43:37.044940+0000","last_change":"2024-03-11T21:43:33.102461+0000","last_active":"2024-03-11T21:43:37.044940+0000","last_peered":"2024-03-11T21:43:37.044940+0000","last_clean":"2024-03-11T21:43:37.044940+0000","last_became_active":"2024-03-11T21:25:38.541705+0000","last_became_peered":"2024-03-11T21:25:38.541705+0000","last_unstale":"2024-03-11T21:43:37.044940+0000","last_undegraded":"2024-03-11T21:43:37.044940+0000","last_fullsized":"2024-03-11T21:43:37.044940+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:43:33.102157+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:26:44.860499+0000","last_clean_scrub_stamp":"2024-03-11T21:43:33.102157+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:44:41.421719+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.13","version":"260'24","reported_seq":812,"reported_epoch":415,"state":"active+clean","last_fresh":"2024-03-11T21:43:44.075329+0000","last_change":"2024-03-11T21:40:25.595320+0000","last_active":"2024-03-11T21:43:44.075329+0000","last_peered":"2024-03-11T21:43:44.075329+0000","last_clean":"2024-03-11T21:43:44.075329+0000","last_became_active":"2024-03-11T21:26:11.214879+0000","last_became_peered":"2024-03-11T21:26:11.214879+0000","last_unstale":"2024-03-11T21:43:44.075329+0000","last_undegraded":"2024-03-11T21:43:44.075329+0000","last_fullsized":"2024-03-11T21:43:44.075329+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"260'24","last_scrub_stamp":"2024-03-11T21:40:25.595220+0000","last_deep_scrub":"260'24","last_deep_scrub_stamp":"2024-03-11T21:40:25.595220+0000","last_clean_scrub_stamp":"2024-03-11T21:40:25.595220+0000","objects_scrubbed":5,"log_size":24,"log_dups_size":0,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:43:48.166258+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":32,"num_read_kb":25,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,2],"acting":[4,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.10","version":"0'0","reported_seq":750,"reported_epoch":415,"state":"active+clean","last_fresh":"2024-03-11T21:43:44.079593+0000","last_change":"2024-03-11T21:43:25.155539+0000","last_active":"2024-03-11T21:43:44.079593+0000","last_peered":"2024-03-11T21:43:44.079593+0000","last_clean":"2024-03-11T21:43:44.079593+0000","last_became_active":"2024-03-11T21:25:44.591189+0000","last_became_peered":"2024-03-11T21:25:44.591189+0000","last_unstale":"2024-03-11T21:43:44.079593+0000","last_undegraded":"2024-03-11T21:43:44.079593+0000","last_fullsized":"2024-03-11T21:43:44.079593+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:43:25.155431+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:27:25.417450+0000","last_clean_scrub_stamp":"2024-03-11T21:43:25.155431+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:44:35.127291+0000","scrub_duration":93,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,1],"acting":[0,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":677,"reported_epoch":415,"state":"active+clean","last_fresh":"2024-03-11T21:43:44.089325+0000","last_change":"2024-03-11T21:36:35.062276+0000","last_active":"2024-03-11T21:43:44.089325+0000","last_peered":"2024-03-11T21:43:44.089325+0000","last_clean":"2024-03-11T21:43:44.089325+0000","last_became_active":"2024-03-11T21:36:35.056619+0000","last_became_peered":"2024-03-11T21:36:35.056619+0000","last_unstale":"2024-03-11T21:43:44.089325+0000","last_undegraded":"2024-03-11T21:43:44.089325+0000","last_fullsized":"2024-03-11T21:43:44.089325+0000","mapping_epoch":277,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":278,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:35:13.060487+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:31:15.367898+0000","last_clean_scrub_stamp":"2024-03-11T21:35:13.060487+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.14","version":"0'0","reported_seq":793,"reported_epoch":414,"state":"active+clean","last_fresh":"2024-03-11T21:43:37.036247+0000","last_change":"2024-03-11T21:41:14.822928+0000","last_active":"2024-03-11T21:43:37.036247+0000","last_peered":"2024-03-11T21:43:37.036247+0000","last_clean":"2024-03-11T21:43:37.036247+0000","last_became_active":"2024-03-11T21:36:35.068038+0000","last_became_peered":"2024-03-11T21:36:35.068038+0000","last_unstale":"2024-03-11T21:43:37.036247+0000","last_undegraded":"2024-03-11T21:43:37.036247+0000","last_fullsized":"2024-03-11T21:43:37.036247+0000","mapping_epoch":277,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":278,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:41:14.822805+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:37.512018+0000","last_clean_scrub_stamp":"2024-03-11T21:41:14.822805+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,5],"acting":[6,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.12","version":"260'15","reported_seq":781,"reported_epoch":414,"state":"active+clean","last_fresh":"2024-03-11T21:43:37.049492+0000","last_change":"2024-03-11T21:41:18.319324+0000","last_active":"2024-03-11T21:43:37.049492+0000","last_peered":"2024-03-11T21:43:37.049492+0000","last_clean":"2024-03-11T21:43:37.049492+0000","last_became_active":"2024-03-11T21:25:40.547082+0000","last_became_peered":"2024-03-11T21:25:40.547082+0000","last_unstale":"2024-03-11T21:43:37.049492+0000","last_undegraded":"2024-03-11T21:43:37.049492+0000","last_fullsized":"2024-03-11T21:43:37.049492+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"260'15","last_scrub_stamp":"2024-03-11T21:41:18.319190+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:39.525554+0000","last_clean_scrub_stamp":"2024-03-11T21:41:18.319190+0000","objects_scrubbed":4,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":750,"reported_epoch":415,"state":"active+clean","last_fresh":"2024-03-11T21:43:44.075885+0000","last_change":"2024-03-11T21:41:25.741034+0000","last_active":"2024-03-11T21:43:44.075885+0000","last_peered":"2024-03-11T21:43:44.075885+0000","last_clean":"2024-03-11T21:43:44.075885+0000","last_became_active":"2024-03-11T21:25:44.589586+0000","last_became_peered":"2024-03-11T21:25:44.589586+0000","last_unstale":"2024-03-11T21:43:44.075885+0000","last_undegraded":"2024-03-11T21:43:44.075885+0000","last_fullsized":"2024-03-11T21:43:44.075885+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:41:25.740917+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:27:09.801138+0000","last_clean_scrub_stamp":"2024-03-11T21:41:25.740917+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:43:48.166924+0000","scrub_duration":158,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,5],"acting":[4,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.14","version":"280'12","reported_seq":925,"reported_epoch":415,"state":"active+clean","last_fresh":"2024-03-11T21:43:44.083564+0000","last_change":"2024-03-11T21:36:35.074550+0000","last_active":"2024-03-11T21:43:44.083564+0000","last_peered":"2024-03-11T21:43:44.083564+0000","last_clean":"2024-03-11T21:43:44.083564+0000","last_became_active":"2024-03-11T21:36:35.073912+0000","last_became_peered":"2024-03-11T21:36:35.073912+0000","last_unstale":"2024-03-11T21:43:44.083564+0000","last_undegraded":"2024-03-11T21:43:44.083564+0000","last_fullsized":"2024-03-11T21:43:44.083564+0000","mapping_epoch":277,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":278,"parent":"0.0","parent_split_bits":0,"last_scrub":"229'10","last_scrub_stamp":"2024-03-11T21:35:26.182107+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:41.544550+0000","last_clean_scrub_stamp":"2024-03-11T21:35:26.182107+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,2],"acting":[3,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.13","version":"0'0","reported_seq":771,"reported_epoch":415,"state":"active+clean","last_fresh":"2024-03-11T21:43:44.084155+0000","last_change":"2024-03-11T21:43:30.076304+0000","last_active":"2024-03-11T21:43:44.084155+0000","last_peered":"2024-03-11T21:43:44.084155+0000","last_clean":"2024-03-11T21:43:44.084155+0000","last_became_active":"2024-03-11T21:36:35.057132+0000","last_became_peered":"2024-03-11T21:36:35.057132+0000","last_unstale":"2024-03-11T21:43:44.084155+0000","last_undegraded":"2024-03-11T21:43:44.084155+0000","last_fullsized":"2024-03-11T21:43:44.084155+0000","mapping_epoch":277,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":278,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:43:30.076191+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:37.512018+0000","last_clean_scrub_stamp":"2024-03-11T21:43:30.076191+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:44:34.615583+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.15","version":"260'19","reported_seq":836,"reported_epoch":415,"state":"active+clean","last_fresh":"2024-03-11T21:43:44.084927+0000","last_change":"2024-03-11T21:41:18.838796+0000","last_active":"2024-03-11T21:43:44.084927+0000","last_peered":"2024-03-11T21:43:44.084927+0000","last_clean":"2024-03-11T21:43:44.084927+0000","last_became_active":"2024-03-11T21:36:35.057285+0000","last_became_peered":"2024-03-11T21:36:35.057285+0000","last_unstale":"2024-03-11T21:43:44.084927+0000","last_undegraded":"2024-03-11T21:43:44.084927+0000","last_fullsized":"2024-03-11T21:43:44.084927+0000","mapping_epoch":277,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":278,"parent":"0.0","parent_split_bits":0,"last_scrub":"260'19","last_scrub_stamp":"2024-03-11T21:41:18.838669+0000","last_deep_scrub":"94'13","last_deep_scrub_stamp":"2024-03-11T21:29:41.853158+0000","last_clean_scrub_stamp":"2024-03-11T21:41:18.838669+0000","objects_scrubbed":4,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:43:45.574776+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":11,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.16","version":"0'0","reported_seq":771,"reported_epoch":415,"state":"active+clean","last_fresh":"2024-03-11T21:43:44.088137+0000","last_change":"2024-03-11T21:40:30.249810+0000","last_active":"2024-03-11T21:43:44.088137+0000","last_peered":"2024-03-11T21:43:44.088137+0000","last_clean":"2024-03-11T21:43:44.088137+0000","last_became_active":"2024-03-11T21:36:35.057355+0000","last_became_peered":"2024-03-11T21:36:35.057355+0000","last_unstale":"2024-03-11T21:43:44.088137+0000","last_undegraded":"2024-03-11T21:43:44.088137+0000","last_fullsized":"2024-03-11T21:43:44.088137+0000","mapping_epoch":277,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":278,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:40:30.249679+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:40:30.249679+0000","last_clean_scrub_stamp":"2024-03-11T21:40:30.249679+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:43:49.041395+0000","scrub_duration":40,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.15","version":"98'9","reported_seq":980,"reported_epoch":415,"state":"active+clean","last_fresh":"2024-03-11T21:43:44.085065+0000","last_change":"2024-03-11T21:43:22.974608+0000","last_active":"2024-03-11T21:43:44.085065+0000","last_peered":"2024-03-11T21:43:44.085065+0000","last_clean":"2024-03-11T21:43:44.085065+0000","last_became_active":"2024-03-11T21:25:42.572555+0000","last_became_peered":"2024-03-11T21:25:42.572555+0000","last_unstale":"2024-03-11T21:43:44.085065+0000","last_undegraded":"2024-03-11T21:43:44.085065+0000","last_fullsized":"2024-03-11T21:43:44.085065+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"98'9","last_scrub_stamp":"2024-03-11T21:43:22.974462+0000","last_deep_scrub":"98'9","last_deep_scrub_stamp":"2024-03-11T21:31:20.624923+0000","last_clean_scrub_stamp":"2024-03-11T21:43:22.974462+0000","objects_scrubbed":1,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:44:42.702256+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.12","version":"0'0","reported_seq":782,"reported_epoch":415,"state":"active+clean","last_fresh":"2024-03-11T21:43:44.084742+0000","last_change":"2024-03-11T21:41:21.853550+0000","last_active":"2024-03-11T21:43:44.084742+0000","last_peered":"2024-03-11T21:43:44.084742+0000","last_clean":"2024-03-11T21:43:44.084742+0000","last_became_active":"2024-03-11T21:36:35.058033+0000","last_became_peered":"2024-03-11T21:36:35.058033+0000","last_unstale":"2024-03-11T21:43:44.084742+0000","last_undegraded":"2024-03-11T21:43:44.084742+0000","last_fullsized":"2024-03-11T21:43:44.084742+0000","mapping_epoch":277,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":278,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:41:21.853428+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:37.512018+0000","last_clean_scrub_stamp":"2024-03-11T21:41:21.853428+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:43:45.574669+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.14","version":"260'20","reported_seq":760,"reported_epoch":415,"state":"active+clean","last_fresh":"2024-03-11T21:43:44.083296+0000","last_change":"2024-03-11T21:36:35.095382+0000","last_active":"2024-03-11T21:43:44.083296+0000","last_peered":"2024-03-11T21:43:44.083296+0000","last_clean":"2024-03-11T21:43:44.083296+0000","last_became_active":"2024-03-11T21:36:35.069638+0000","last_became_peered":"2024-03-11T21:36:35.069638+0000","last_unstale":"2024-03-11T21:43:44.083296+0000","last_undegraded":"2024-03-11T21:43:44.083296+0000","last_fullsized":"2024-03-11T21:43:44.083296+0000","mapping_epoch":277,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":278,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'10","last_scrub_stamp":"2024-03-11T21:35:28.189296+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:39.525554+0000","last_clean_scrub_stamp":"2024-03-11T21:35:28.189296+0000","objects_scrubbed":0,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":29,"num_read_kb":24,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.19","version":"414'7933","reported_seq":17140,"reported_epoch":414,"state":"active+clean","last_fresh":"2024-03-11T21:43:43.483863+0000","last_change":"2024-03-11T21:43:29.486253+0000","last_active":"2024-03-11T21:43:43.483863+0000","last_peered":"2024-03-11T21:43:43.483863+0000","last_clean":"2024-03-11T21:43:43.483863+0000","last_became_active":"2024-03-11T21:43:26.945126+0000","last_became_peered":"2024-03-11T21:43:26.945126+0000","last_unstale":"2024-03-11T21:43:43.483863+0000","last_undegraded":"2024-03-11T21:43:43.483863+0000","last_fullsized":"2024-03-11T21:43:43.483863+0000","mapping_epoch":411,"log_start":"366'1851","ondisk_log_start":"366'1851","created":372,"last_epoch_clean":412,"parent":"0.0","parent_split_bits":5,"last_scrub":"412'7483","last_scrub_stamp":"2024-03-11T21:43:29.485816+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:40:38.219602+0000","last_clean_scrub_stamp":"2024-03-11T21:43:29.485816+0000","objects_scrubbed":1689,"log_size":6082,"log_dups_size":0,"ondisk_log_size":6082,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:44:52.086430+0000","scrub_duration":354,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":80674816,"num_objects":1231,"num_object_clones":0,"num_object_copies":3693,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1231,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":458,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,3],"acting":[1,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.15","version":"0'0","reported_seq":404,"reported_epoch":414,"state":"active+clean","last_fresh":"2024-03-11T21:43:37.034867+0000","last_change":"2024-03-11T21:41:31.306328+0000","last_active":"2024-03-11T21:43:37.034867+0000","last_peered":"2024-03-11T21:43:37.034867+0000","last_clean":"2024-03-11T21:43:37.034867+0000","last_became_active":"2024-03-11T21:34:41.142760+0000","last_became_peered":"2024-03-11T21:34:41.142760+0000","last_unstale":"2024-03-11T21:43:37.034867+0000","last_undegraded":"2024-03-11T21:43:37.034867+0000","last_fullsized":"2024-03-11T21:43:37.034867+0000","mapping_epoch":226,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":227,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:41:31.306234+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:43.559452+0000","last_clean_scrub_stamp":"2024-03-11T21:41:31.306234+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":38,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":762,"reported_epoch":415,"state":"active+clean","last_fresh":"2024-03-11T21:43:44.085471+0000","last_change":"2024-03-11T21:43:36.585771+0000","last_active":"2024-03-11T21:43:44.085471+0000","last_peered":"2024-03-11T21:43:44.085471+0000","last_clean":"2024-03-11T21:43:44.085471+0000","last_became_active":"2024-03-11T21:36:35.060342+0000","last_became_peered":"2024-03-11T21:36:35.060342+0000","last_unstale":"2024-03-11T21:43:44.085471+0000","last_undegraded":"2024-03-11T21:43:44.085471+0000","last_fullsized":"2024-03-11T21:43:44.085471+0000","mapping_epoch":277,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":278,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:43:36.585632+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:43:36.585632+0000","last_clean_scrub_stamp":"2024-03-11T21:43:36.585632+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:45:00.152625+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,1],"acting":[5,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.11","version":"0'0","reported_seq":776,"reported_epoch":414,"state":"active+clean","last_fresh":"2024-03-11T21:43:37.036049+0000","last_change":"2024-03-11T21:43:27.464052+0000","last_active":"2024-03-11T21:43:37.036049+0000","last_peered":"2024-03-11T21:43:37.036049+0000","last_clean":"2024-03-11T21:43:37.036049+0000","last_became_active":"2024-03-11T21:25:38.537415+0000","last_became_peered":"2024-03-11T21:25:38.537415+0000","last_unstale":"2024-03-11T21:43:37.036049+0000","last_undegraded":"2024-03-11T21:43:37.036049+0000","last_fullsized":"2024-03-11T21:43:37.036049+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:43:27.463941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:37.512018+0000","last_clean_scrub_stamp":"2024-03-11T21:43:27.463941+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:44:38.380709+0000","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.17","version":"260'15","reported_seq":712,"reported_epoch":415,"state":"active+clean","last_fresh":"2024-03-11T21:43:44.083960+0000","last_change":"2024-03-11T21:41:00.652483+0000","last_active":"2024-03-11T21:43:44.083960+0000","last_peered":"2024-03-11T21:43:44.083960+0000","last_clean":"2024-03-11T21:43:44.083960+0000","last_became_active":"2024-03-11T21:36:35.071346+0000","last_became_peered":"2024-03-11T21:36:35.071346+0000","last_unstale":"2024-03-11T21:43:44.083960+0000","last_undegraded":"2024-03-11T21:43:44.083960+0000","last_fullsized":"2024-03-11T21:43:44.083960+0000","mapping_epoch":277,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":278,"parent":"0.0","parent_split_bits":0,"last_scrub":"260'15","last_scrub_stamp":"2024-03-11T21:41:00.652358+0000","last_deep_scrub":"97'9","last_deep_scrub_stamp":"2024-03-11T21:27:25.803132+0000","last_clean_scrub_stamp":"2024-03-11T21:41:00.652358+0000","objects_scrubbed":3,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":17,"num_read_kb":14,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,5],"acting":[3,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.18","version":"414'7890","reported_seq":14690,"reported_epoch":414,"state":"active+clean","last_fresh":"2024-03-11T21:43:43.205620+0000","last_change":"2024-03-11T21:43:30.494128+0000","last_active":"2024-03-11T21:43:43.205620+0000","last_peered":"2024-03-11T21:43:43.205620+0000","last_clean":"2024-03-11T21:43:43.205620+0000","last_became_active":"2024-03-11T21:43:17.856511+0000","last_became_peered":"2024-03-11T21:43:17.856511+0000","last_unstale":"2024-03-11T21:43:43.205620+0000","last_undegraded":"2024-03-11T21:43:43.205620+0000","last_fullsized":"2024-03-11T21:43:43.205620+0000","mapping_epoch":405,"log_start":"0'0","ondisk_log_start":"0'0","created":372,"last_epoch_clean":406,"parent":"0.0","parent_split_bits":5,"last_scrub":"412'7454","last_scrub_stamp":"2024-03-11T21:43:30.493611+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:40:38.219602+0000","last_clean_scrub_stamp":"2024-03-11T21:43:30.493611+0000","objects_scrubbed":1689,"log_size":7890,"log_dups_size":0,"ondisk_log_size":7890,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:44:35.457355+0000","scrub_duration":354,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":81985536,"num_objects":1251,"num_object_clones":0,"num_object_copies":3753,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1251,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":438,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.14","version":"0'0","reported_seq":758,"reported_epoch":414,"state":"active+clean","last_fresh":"2024-03-11T21:43:37.030605+0000","last_change":"2024-03-11T21:41:08.130109+0000","last_active":"2024-03-11T21:43:37.030605+0000","last_peered":"2024-03-11T21:43:37.030605+0000","last_clean":"2024-03-11T21:43:37.030605+0000","last_became_active":"2024-03-11T21:34:41.136946+0000","last_became_peered":"2024-03-11T21:34:41.136946+0000","last_unstale":"2024-03-11T21:43:37.030605+0000","last_undegraded":"2024-03-11T21:43:37.030605+0000","last_fullsized":"2024-03-11T21:43:37.030605+0000","mapping_epoch":226,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":227,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:41:08.129982+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:43.559452+0000","last_clean_scrub_stamp":"2024-03-11T21:41:08.129982+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,7],"acting":[2,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":666,"reported_epoch":415,"state":"active+clean","last_fresh":"2024-03-11T21:43:44.083771+0000","last_change":"2024-03-11T21:41:31.530082+0000","last_active":"2024-03-11T21:43:44.083771+0000","last_peered":"2024-03-11T21:43:44.083771+0000","last_clean":"2024-03-11T21:43:44.083771+0000","last_became_active":"2024-03-11T21:36:35.076335+0000","last_became_peered":"2024-03-11T21:36:35.076335+0000","last_unstale":"2024-03-11T21:43:44.083771+0000","last_undegraded":"2024-03-11T21:43:44.083771+0000","last_fullsized":"2024-03-11T21:43:44.083771+0000","mapping_epoch":277,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":278,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:41:31.529958+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:41.544550+0000","last_clean_scrub_stamp":"2024-03-11T21:41:31.529958+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.10","version":"0'0","reported_seq":758,"reported_epoch":414,"state":"active+clean","last_fresh":"2024-03-11T21:43:37.030123+0000","last_change":"2024-03-11T21:41:23.299628+0000","last_active":"2024-03-11T21:43:37.030123+0000","last_peered":"2024-03-11T21:43:37.030123+0000","last_clean":"2024-03-11T21:43:37.030123+0000","last_became_active":"2024-03-11T21:25:38.533138+0000","last_became_peered":"2024-03-11T21:25:38.533138+0000","last_unstale":"2024-03-11T21:43:37.030123+0000","last_undegraded":"2024-03-11T21:43:37.030123+0000","last_fullsized":"2024-03-11T21:43:37.030123+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:41:23.299503+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:37.512018+0000","last_clean_scrub_stamp":"2024-03-11T21:41:23.299503+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,0],"acting":[2,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.16","version":"369'19","reported_seq":837,"reported_epoch":415,"state":"active+clean","last_fresh":"2024-03-11T21:43:44.088253+0000","last_change":"2024-03-11T21:41:22.115878+0000","last_active":"2024-03-11T21:43:44.088253+0000","last_peered":"2024-03-11T21:43:44.088253+0000","last_clean":"2024-03-11T21:43:44.088253+0000","last_became_active":"2024-03-11T21:36:35.056862+0000","last_became_peered":"2024-03-11T21:36:35.056862+0000","last_unstale":"2024-03-11T21:43:44.088253+0000","last_undegraded":"2024-03-11T21:43:44.088253+0000","last_fullsized":"2024-03-11T21:43:44.088253+0000","mapping_epoch":277,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":278,"parent":"0.0","parent_split_bits":0,"last_scrub":"369'19","last_scrub_stamp":"2024-03-11T21:41:22.115634+0000","last_deep_scrub":"369'19","last_deep_scrub_stamp":"2024-03-11T21:41:22.115634+0000","last_clean_scrub_stamp":"2024-03-11T21:41:22.115634+0000","objects_scrubbed":4,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:43:49.041521+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":11,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.7","version":"414'7842","reported_seq":8007,"reported_epoch":414,"state":"active+clean","last_fresh":"2024-03-11T21:43:42.485266+0000","last_change":"2024-03-11T21:41:31.118161+0000","last_active":"2024-03-11T21:43:42.485266+0000","last_peered":"2024-03-11T21:43:42.485266+0000","last_clean":"2024-03-11T21:43:42.485266+0000","last_became_active":"2024-03-11T21:41:28.840566+0000","last_became_peered":"2024-03-11T21:41:28.840566+0000","last_unstale":"2024-03-11T21:43:42.485266+0000","last_undegraded":"2024-03-11T21:43:42.485266+0000","last_fullsized":"2024-03-11T21:43:42.485266+0000","mapping_epoch":356,"log_start":"364'221","ondisk_log_start":"364'221","created":356,"last_epoch_clean":373,"parent":"0.0","parent_split_bits":0,"last_scrub":"374'4410","last_scrub_stamp":"2024-03-11T21:41:31.117966+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:40:38.219602+0000","last_clean_scrub_stamp":"2024-03-11T21:41:31.117966+0000","objects_scrubbed":2221,"log_size":7621,"log_dups_size":0,"ondisk_log_size":7621,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":256,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":79364096,"num_objects":1211,"num_object_clones":0,"num_object_copies":3633,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1211,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3432,"num_write_kb":77504,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,5],"acting":[6,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.b","version":"0'0","reported_seq":655,"reported_epoch":415,"state":"active+clean","last_fresh":"2024-03-11T21:43:44.089175+0000","last_change":"2024-03-11T21:40:27.611092+0000","last_active":"2024-03-11T21:43:44.089175+0000","last_peered":"2024-03-11T21:43:44.089175+0000","last_clean":"2024-03-11T21:43:44.089175+0000","last_became_active":"2024-03-11T21:36:35.072147+0000","last_became_peered":"2024-03-11T21:36:35.072147+0000","last_unstale":"2024-03-11T21:43:44.089175+0000","last_undegraded":"2024-03-11T21:43:44.089175+0000","last_fullsized":"2024-03-11T21:43:44.089175+0000","mapping_epoch":277,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":278,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:40:27.610987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:43.559452+0000","last_clean_scrub_stamp":"2024-03-11T21:40:27.610987+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":60,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.f","version":"67'2","reported_seq":775,"reported_epoch":415,"state":"active+clean","last_fresh":"2024-03-11T21:43:44.075820+0000","last_change":"2024-03-11T21:41:13.539379+0000","last_active":"2024-03-11T21:43:44.075820+0000","last_peered":"2024-03-11T21:43:44.075820+0000","last_clean":"2024-03-11T21:43:44.075820+0000","last_became_active":"2024-03-11T21:25:38.537711+0000","last_became_peered":"2024-03-11T21:25:38.537711+0000","last_unstale":"2024-03-11T21:43:44.075820+0000","last_undegraded":"2024-03-11T21:43:44.075820+0000","last_fullsized":"2024-03-11T21:43:44.075820+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'2","last_scrub_stamp":"2024-03-11T21:41:13.539240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:37.512018+0000","last_clean_scrub_stamp":"2024-03-11T21:41:13.539240+0000","objects_scrubbed":2,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:43:48.166367+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":92,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":10,"num_read_kb":10,"num_write":4,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,7],"acting":[4,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.9","version":"260'23","reported_seq":806,"reported_epoch":415,"state":"active+clean","last_fresh":"2024-03-11T21:43:44.075338+0000","last_change":"2024-03-11T21:40:14.543701+0000","last_active":"2024-03-11T21:43:44.075338+0000","last_peered":"2024-03-11T21:43:44.075338+0000","last_clean":"2024-03-11T21:43:44.075338+0000","last_became_active":"2024-03-11T21:36:35.059930+0000","last_became_peered":"2024-03-11T21:36:35.059930+0000","last_unstale":"2024-03-11T21:43:44.075338+0000","last_undegraded":"2024-03-11T21:43:44.075338+0000","last_fullsized":"2024-03-11T21:43:44.075338+0000","mapping_epoch":277,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":278,"parent":"0.0","parent_split_bits":0,"last_scrub":"260'23","last_scrub_stamp":"2024-03-11T21:40:14.543596+0000","last_deep_scrub":"146'17","last_deep_scrub_stamp":"2024-03-11T21:34:26.765075+0000","last_clean_scrub_stamp":"2024-03-11T21:40:14.543596+0000","objects_scrubbed":5,"log_size":23,"log_dups_size":0,"ondisk_log_size":23,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:43:48.166226+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":34,"num_read_kb":25,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,3],"acting":[4,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":751,"reported_epoch":414,"state":"active+clean","last_fresh":"2024-03-11T21:43:37.030206+0000","last_change":"2024-03-11T21:41:15.238256+0000","last_active":"2024-03-11T21:43:37.030206+0000","last_peered":"2024-03-11T21:43:37.030206+0000","last_clean":"2024-03-11T21:43:37.030206+0000","last_became_active":"2024-03-11T21:25:42.573987+0000","last_became_peered":"2024-03-11T21:25:42.573987+0000","last_unstale":"2024-03-11T21:43:37.030206+0000","last_undegraded":"2024-03-11T21:43:37.030206+0000","last_fullsized":"2024-03-11T21:43:37.030206+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:41:15.238017+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:41.544550+0000","last_clean_scrub_stamp":"2024-03-11T21:41:15.238017+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"10.c","version":"414'12325","reported_seq":12478,"reported_epoch":414,"state":"active+clean+scrubbing+deep","last_fresh":"2024-03-11T21:43:42.492203+0000","last_change":"2024-03-11T21:43:36.766776+0000","last_active":"2024-03-11T21:43:42.492203+0000","last_peered":"2024-03-11T21:43:42.492203+0000","last_clean":"2024-03-11T21:43:42.492203+0000","last_became_active":"2024-03-11T21:40:39.241699+0000","last_became_peered":"2024-03-11T21:40:39.241699+0000","last_unstale":"2024-03-11T21:43:42.492203+0000","last_undegraded":"2024-03-11T21:43:42.492203+0000","last_fullsized":"2024-03-11T21:43:42.492203+0000","mapping_epoch":356,"log_start":"400'7701","ondisk_log_start":"400'7701","created":356,"last_epoch_clean":357,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:40:38.219602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:40:38.219602+0000","last_clean_scrub_stamp":"2024-03-11T21:40:38.219602+0000","objects_scrubbed":1073,"log_size":4624,"log_dups_size":0,"ondisk_log_size":4624,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 5s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":163446784,"num_objects":2494,"num_object_clones":0,"num_object_copies":7482,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2494,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11252,"num_write_kb":439872,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,0],"acting":[7,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.0","version":"0'0","reported_seq":760,"reported_epoch":415,"state":"active+clean","last_fresh":"2024-03-11T21:43:44.079617+0000","last_change":"2024-03-11T21:43:33.135522+0000","last_active":"2024-03-11T21:43:44.079617+0000","last_peered":"2024-03-11T21:43:44.079617+0000","last_clean":"2024-03-11T21:43:44.079617+0000","last_became_active":"2024-03-11T21:36:35.058054+0000","last_became_peered":"2024-03-11T21:36:35.058054+0000","last_unstale":"2024-03-11T21:43:44.079617+0000","last_undegraded":"2024-03-11T21:43:44.079617+0000","last_fullsized":"2024-03-11T21:43:44.079617+0000","mapping_epoch":277,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":278,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:43:33.135442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:29:39.498784+0000","last_clean_scrub_stamp":"2024-03-11T21:43:33.135442+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:44:41.342771+0000","scrub_duration":73,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":773,"reported_epoch":414,"state":"active+clean","last_fresh":"2024-03-11T21:43:37.048373+0000","last_change":"2024-03-11T21:41:22.292906+0000","last_active":"2024-03-11T21:43:37.048373+0000","last_peered":"2024-03-11T21:43:37.048373+0000","last_clean":"2024-03-11T21:43:37.048373+0000","last_became_active":"2024-03-11T21:34:41.150913+0000","last_became_peered":"2024-03-11T21:34:41.150913+0000","last_unstale":"2024-03-11T21:43:37.048373+0000","last_undegraded":"2024-03-11T21:43:37.048373+0000","last_fullsized":"2024-03-11T21:43:37.048373+0000","mapping_epoch":226,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":227,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:41:22.292789+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:37.512018+0000","last_clean_scrub_stamp":"2024-03-11T21:41:22.292789+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,7],"acting":[1,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.2","version":"260'25","reported_seq":792,"reported_epoch":414,"state":"active+clean","last_fresh":"2024-03-11T21:43:37.049577+0000","last_change":"2024-03-11T21:41:07.243196+0000","last_active":"2024-03-11T21:43:37.049577+0000","last_peered":"2024-03-11T21:43:37.049577+0000","last_clean":"2024-03-11T21:43:37.049577+0000","last_became_active":"2024-03-11T21:25:40.548833+0000","last_became_peered":"2024-03-11T21:25:40.548833+0000","last_unstale":"2024-03-11T21:43:37.049577+0000","last_undegraded":"2024-03-11T21:43:37.049577+0000","last_fullsized":"2024-03-11T21:43:37.049577+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"260'25","last_scrub_stamp":"2024-03-11T21:41:07.242950+0000","last_deep_scrub":"94'15","last_deep_scrub_stamp":"2024-03-11T21:26:49.993060+0000","last_clean_scrub_stamp":"2024-03-11T21:41:07.242950+0000","objects_scrubbed":5,"log_size":25,"log_dups_size":0,"ondisk_log_size":25,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":25,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.3","version":"74'8","reported_seq":981,"reported_epoch":415,"state":"active+clean","last_fresh":"2024-03-11T21:43:44.084554+0000","last_change":"2024-03-11T21:41:31.189011+0000","last_active":"2024-03-11T21:43:44.084554+0000","last_peered":"2024-03-11T21:43:44.084554+0000","last_clean":"2024-03-11T21:43:44.084554+0000","last_became_active":"2024-03-11T21:25:42.578161+0000","last_became_peered":"2024-03-11T21:25:42.578161+0000","last_unstale":"2024-03-11T21:43:44.084554+0000","last_undegraded":"2024-03-11T21:43:44.084554+0000","last_fullsized":"2024-03-11T21:43:44.084554+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'8","last_scrub_stamp":"2024-03-11T21:41:31.188904+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:41.544550+0000","last_clean_scrub_stamp":"2024-03-11T21:41:31.188904+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:43:49.041581+0000","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,5],"acting":[0,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.a","version":"414'11199","reported_seq":11329,"reported_epoch":414,"state":"active+clean","last_fresh":"2024-03-11T21:43:42.469564+0000","last_change":"2024-03-11T21:40:39.243111+0000","last_active":"2024-03-11T21:43:42.469564+0000","last_peered":"2024-03-11T21:43:42.469564+0000","last_clean":"2024-03-11T21:43:42.469564+0000","last_became_active":"2024-03-11T21:40:39.242809+0000","last_became_peered":"2024-03-11T21:40:39.242809+0000","last_unstale":"2024-03-11T21:43:42.469564+0000","last_undegraded":"2024-03-11T21:43:42.469564+0000","last_fullsized":"2024-03-11T21:43:42.469564+0000","mapping_epoch":356,"log_start":"384'5600","ondisk_log_start":"384'5600","created":356,"last_epoch_clean":357,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:40:38.219602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:40:38.219602+0000","last_clean_scrub_stamp":"2024-03-11T21:40:38.219602+0000","objects_scrubbed":0,"log_size":5599,"log_dups_size":0,"ondisk_log_size":5599,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":158400512,"num_objects":2417,"num_object_clones":0,"num_object_copies":7251,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2417,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11199,"num_write_kb":435712,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":685,"reported_epoch":415,"state":"active+clean","last_fresh":"2024-03-11T21:43:44.083098+0000","last_change":"2024-03-11T21:36:35.068977+0000","last_active":"2024-03-11T21:43:44.083098+0000","last_peered":"2024-03-11T21:43:44.083098+0000","last_clean":"2024-03-11T21:43:44.083098+0000","last_became_active":"2024-03-11T21:36:35.067288+0000","last_became_peered":"2024-03-11T21:36:35.067288+0000","last_unstale":"2024-03-11T21:43:44.083098+0000","last_undegraded":"2024-03-11T21:43:44.083098+0000","last_fullsized":"2024-03-11T21:43:44.083098+0000","mapping_epoch":277,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":278,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:35:09.985377+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:28:30.563712+0000","last_clean_scrub_stamp":"2024-03-11T21:35:09.985377+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,7],"acting":[3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":770,"reported_epoch":415,"state":"active+clean","last_fresh":"2024-03-11T21:43:44.085290+0000","last_change":"2024-03-11T21:41:24.921735+0000","last_active":"2024-03-11T21:43:44.085290+0000","last_peered":"2024-03-11T21:43:44.085290+0000","last_clean":"2024-03-11T21:43:44.085290+0000","last_became_active":"2024-03-11T21:25:38.538540+0000","last_became_peered":"2024-03-11T21:25:38.538540+0000","last_unstale":"2024-03-11T21:43:44.085290+0000","last_undegraded":"2024-03-11T21:43:44.085290+0000","last_fullsized":"2024-03-11T21:43:44.085290+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:41:24.921598+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:37.512018+0000","last_clean_scrub_stamp":"2024-03-11T21:41:24.921598+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:43:45.574828+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.4","version":"280'42","reported_seq":1057,"reported_epoch":414,"state":"active+clean","last_fresh":"2024-03-11T21:43:37.048320+0000","last_change":"2024-03-11T21:41:05.234160+0000","last_active":"2024-03-11T21:43:37.048320+0000","last_peered":"2024-03-11T21:43:37.048320+0000","last_clean":"2024-03-11T21:43:37.048320+0000","last_became_active":"2024-03-11T21:36:35.073430+0000","last_became_peered":"2024-03-11T21:36:35.073430+0000","last_unstale":"2024-03-11T21:43:37.048320+0000","last_undegraded":"2024-03-11T21:43:37.048320+0000","last_fullsized":"2024-03-11T21:43:37.048320+0000","mapping_epoch":277,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":278,"parent":"0.0","parent_split_bits":0,"last_scrub":"280'42","last_scrub_stamp":"2024-03-11T21:41:05.233993+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:39.525554+0000","last_clean_scrub_stamp":"2024-03-11T21:41:05.233993+0000","objects_scrubbed":10,"log_size":42,"log_dups_size":0,"ondisk_log_size":42,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":358,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":66,"num_read_kb":51,"num_write":38,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3],"acting":[1,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":755,"reported_epoch":415,"state":"active+clean","last_fresh":"2024-03-11T21:43:44.084821+0000","last_change":"2024-03-11T21:43:26.112273+0000","last_active":"2024-03-11T21:43:44.084821+0000","last_peered":"2024-03-11T21:43:44.084821+0000","last_clean":"2024-03-11T21:43:44.084821+0000","last_became_active":"2024-03-11T21:25:42.572673+0000","last_became_peered":"2024-03-11T21:25:42.572673+0000","last_unstale":"2024-03-11T21:43:44.084821+0000","last_undegraded":"2024-03-11T21:43:44.084821+0000","last_fullsized":"2024-03-11T21:43:44.084821+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:43:26.111942+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:41.544550+0000","last_clean_scrub_stamp":"2024-03-11T21:43:26.111942+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:44:35.864591+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.9","version":"415'8661","reported_seq":8891,"reported_epoch":415,"state":"active+clean","last_fresh":"2024-03-11T21:43:47.107050+0000","last_change":"2024-03-11T21:41:32.409763+0000","last_active":"2024-03-11T21:43:47.107050+0000","last_peered":"2024-03-11T21:43:47.107050+0000","last_clean":"2024-03-11T21:43:47.107050+0000","last_became_active":"2024-03-11T21:41:28.824488+0000","last_became_peered":"2024-03-11T21:41:28.824488+0000","last_unstale":"2024-03-11T21:43:47.107050+0000","last_undegraded":"2024-03-11T21:43:47.107050+0000","last_fullsized":"2024-03-11T21:43:47.107050+0000","mapping_epoch":356,"log_start":"366'2575","ondisk_log_start":"366'2575","created":356,"last_epoch_clean":373,"parent":"0.0","parent_split_bits":0,"last_scrub":"376'5094","last_scrub_stamp":"2024-03-11T21:41:32.409608+0000","last_deep_scrub":"376'5094","last_deep_scrub_stamp":"2024-03-11T21:41:32.409608+0000","last_clean_scrub_stamp":"2024-03-11T21:41:32.409608+0000","objects_scrubbed":2132,"log_size":6086,"log_dups_size":0,"ondisk_log_size":6086,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":4,"scrub_schedule":"queued for scrub","scrub_duration":3401,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":71696384,"num_objects":1094,"num_object_clones":0,"num_object_copies":3282,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1094,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3570,"num_write_kb":81024,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.5","version":"0'0","reported_seq":414,"reported_epoch":414,"state":"active+clean","last_fresh":"2024-03-11T21:43:37.034699+0000","last_change":"2024-03-11T21:41:12.380275+0000","last_active":"2024-03-11T21:43:37.034699+0000","last_peered":"2024-03-11T21:43:37.034699+0000","last_clean":"2024-03-11T21:43:37.034699+0000","last_became_active":"2024-03-11T21:36:35.069358+0000","last_became_peered":"2024-03-11T21:36:35.069358+0000","last_unstale":"2024-03-11T21:43:37.034699+0000","last_undegraded":"2024-03-11T21:43:37.034699+0000","last_fullsized":"2024-03-11T21:43:37.034699+0000","mapping_epoch":277,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":278,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:41:12.380019+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:43.559452+0000","last_clean_scrub_stamp":"2024-03-11T21:41:12.380019+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":53,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,3],"acting":[7,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.0","version":"164'6","reported_seq":1849,"reported_epoch":414,"state":"active+clean","last_fresh":"2024-03-11T21:43:42.736484+0000","last_change":"2024-03-11T21:41:30.197407+0000","last_active":"2024-03-11T21:43:42.736484+0000","last_peered":"2024-03-11T21:43:42.736484+0000","last_clean":"2024-03-11T21:43:42.736484+0000","last_became_active":"2024-03-11T21:25:38.536890+0000","last_became_peered":"2024-03-11T21:25:38.536890+0000","last_unstale":"2024-03-11T21:43:42.736484+0000","last_undegraded":"2024-03-11T21:43:42.736484+0000","last_fullsized":"2024-03-11T21:43:42.736484+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"164'6","last_scrub_stamp":"2024-03-11T21:41:30.197276+0000","last_deep_scrub":"164'6","last_deep_scrub_stamp":"2024-03-11T21:32:17.597666+0000","last_clean_scrub_stamp":"2024-03-11T21:41:30.197276+0000","objects_scrubbed":1,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:43:44.069637+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0.000753559,"stat_sum":{"num_bytes":389,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1076,"num_read_kb":1071,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,6],"acting":[1,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":776,"reported_epoch":414,"state":"active+clean","last_fresh":"2024-03-11T21:43:37.030932+0000","last_change":"2024-03-11T21:43:28.183519+0000","last_active":"2024-03-11T21:43:37.030932+0000","last_peered":"2024-03-11T21:43:37.030932+0000","last_clean":"2024-03-11T21:43:37.030932+0000","last_became_active":"2024-03-11T21:36:35.056966+0000","last_became_peered":"2024-03-11T21:36:35.056966+0000","last_unstale":"2024-03-11T21:43:37.030932+0000","last_undegraded":"2024-03-11T21:43:37.030932+0000","last_fullsized":"2024-03-11T21:43:37.030932+0000","mapping_epoch":277,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":278,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:43:28.183315+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:43:28.183315+0000","last_clean_scrub_stamp":"2024-03-11T21:43:28.183315+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:44:29.807489+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.7","version":"260'22","reported_seq":798,"reported_epoch":414,"state":"active+clean","last_fresh":"2024-03-11T21:43:37.044634+0000","last_change":"2024-03-11T21:41:27.232567+0000","last_active":"2024-03-11T21:43:37.044634+0000","last_peered":"2024-03-11T21:43:37.044634+0000","last_clean":"2024-03-11T21:43:37.044634+0000","last_became_active":"2024-03-11T21:25:40.541769+0000","last_became_peered":"2024-03-11T21:25:40.541769+0000","last_unstale":"2024-03-11T21:43:37.044634+0000","last_undegraded":"2024-03-11T21:43:37.044634+0000","last_fullsized":"2024-03-11T21:43:37.044634+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"260'22","last_scrub_stamp":"2024-03-11T21:41:27.232403+0000","last_deep_scrub":"260'22","last_deep_scrub_stamp":"2024-03-11T21:41:27.232403+0000","last_clean_scrub_stamp":"2024-03-11T21:41:27.232403+0000","objects_scrubbed":5,"log_size":22,"log_dups_size":0,"ondisk_log_size":22,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":36,"num_read_kb":25,"num_write":20,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":763,"reported_epoch":414,"state":"active+clean","last_fresh":"2024-03-11T21:43:37.032064+0000","last_change":"2024-03-11T21:41:04.111560+0000","last_active":"2024-03-11T21:43:37.032064+0000","last_peered":"2024-03-11T21:43:37.032064+0000","last_clean":"2024-03-11T21:43:37.032064+0000","last_became_active":"2024-03-11T21:34:41.143660+0000","last_became_peered":"2024-03-11T21:34:41.143660+0000","last_unstale":"2024-03-11T21:43:37.032064+0000","last_undegraded":"2024-03-11T21:43:37.032064+0000","last_fullsized":"2024-03-11T21:43:37.032064+0000","mapping_epoch":226,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":227,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:41:04.111481+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:41.544550+0000","last_clean_scrub_stamp":"2024-03-11T21:41:04.111481+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7],"acting":[2,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"10.b","version":"414'11326","reported_seq":11456,"reported_epoch":414,"state":"active+clean","last_fresh":"2024-03-11T21:43:42.460481+0000","last_change":"2024-03-11T21:40:39.241086+0000","last_active":"2024-03-11T21:43:42.460481+0000","last_peered":"2024-03-11T21:43:42.460481+0000","last_clean":"2024-03-11T21:43:42.460481+0000","last_became_active":"2024-03-11T21:40:39.240775+0000","last_became_peered":"2024-03-11T21:40:39.240775+0000","last_unstale":"2024-03-11T21:43:42.460481+0000","last_undegraded":"2024-03-11T21:43:42.460481+0000","last_fullsized":"2024-03-11T21:43:42.460481+0000","mapping_epoch":356,"log_start":"384'5700","ondisk_log_start":"384'5700","created":356,"last_epoch_clean":357,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:40:38.219602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:40:38.219602+0000","last_clean_scrub_stamp":"2024-03-11T21:40:38.219602+0000","objects_scrubbed":0,"log_size":5626,"log_dups_size":0,"ondisk_log_size":5626,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":161087488,"num_objects":2458,"num_object_clones":0,"num_object_copies":7374,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2458,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11326,"num_write_kb":441088,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,0],"acting":[6,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.7","version":"124'2","reported_seq":753,"reported_epoch":415,"state":"active+clean","last_fresh":"2024-03-11T21:43:44.085158+0000","last_change":"2024-03-11T21:40:31.722243+0000","last_active":"2024-03-11T21:43:44.085158+0000","last_peered":"2024-03-11T21:43:44.085158+0000","last_clean":"2024-03-11T21:43:44.085158+0000","last_became_active":"2024-03-11T21:36:35.058786+0000","last_became_peered":"2024-03-11T21:36:35.058786+0000","last_unstale":"2024-03-11T21:43:44.085158+0000","last_undegraded":"2024-03-11T21:43:44.085158+0000","last_fullsized":"2024-03-11T21:43:44.085158+0000","mapping_epoch":277,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":278,"parent":"0.0","parent_split_bits":0,"last_scrub":"124'2","last_scrub_stamp":"2024-03-11T21:40:31.722100+0000","last_deep_scrub":"124'2","last_deep_scrub_stamp":"2024-03-11T21:29:34.956908+0000","last_clean_scrub_stamp":"2024-03-11T21:40:31.722100+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:43:45.574554+0000","scrub_duration":87,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,4],"acting":[5,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2","version":"71'2","reported_seq":714,"reported_epoch":415,"state":"active+clean","last_fresh":"2024-03-11T21:43:44.082918+0000","last_change":"2024-03-11T21:41:07.517286+0000","last_active":"2024-03-11T21:43:44.082918+0000","last_peered":"2024-03-11T21:43:44.082918+0000","last_clean":"2024-03-11T21:43:44.082918+0000","last_became_active":"2024-03-11T21:36:35.070783+0000","last_became_peered":"2024-03-11T21:36:35.070783+0000","last_unstale":"2024-03-11T21:43:44.082918+0000","last_undegraded":"2024-03-11T21:43:44.082918+0000","last_fullsized":"2024-03-11T21:43:44.082918+0000","mapping_epoch":277,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":278,"parent":"0.0","parent_split_bits":0,"last_scrub":"71'2","last_scrub_stamp":"2024-03-11T21:41:07.517154+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:37.512018+0000","last_clean_scrub_stamp":"2024-03-11T21:41:07.517154+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0.0014206449999999999,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":780,"reported_epoch":415,"state":"active+clean","last_fresh":"2024-03-11T21:43:44.084578+0000","last_change":"2024-03-11T21:41:10.803049+0000","last_active":"2024-03-11T21:43:44.084578+0000","last_peered":"2024-03-11T21:43:44.084578+0000","last_clean":"2024-03-11T21:43:44.084578+0000","last_became_active":"2024-03-11T21:34:41.146672+0000","last_became_peered":"2024-03-11T21:34:41.146672+0000","last_unstale":"2024-03-11T21:43:44.084578+0000","last_undegraded":"2024-03-11T21:43:44.084578+0000","last_fullsized":"2024-03-11T21:43:44.084578+0000","mapping_epoch":226,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":227,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:41:10.802921+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:37.512018+0000","last_clean_scrub_stamp":"2024-03-11T21:41:10.802921+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:43:45.574640+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"87'128","reported_seq":876,"reported_epoch":414,"state":"active+clean","last_fresh":"2024-03-11T21:43:37.035905+0000","last_change":"2024-03-11T21:41:09.328321+0000","last_active":"2024-03-11T21:43:37.035905+0000","last_peered":"2024-03-11T21:43:37.035905+0000","last_clean":"2024-03-11T21:43:37.035905+0000","last_became_active":"2024-03-11T21:34:44.164569+0000","last_became_peered":"2024-03-11T21:34:44.164569+0000","last_unstale":"2024-03-11T21:43:37.035905+0000","last_undegraded":"2024-03-11T21:43:37.035905+0000","last_fullsized":"2024-03-11T21:43:37.035905+0000","mapping_epoch":229,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":230,"parent":"0.0","parent_split_bits":0,"last_scrub":"87'128","last_scrub_stamp":"2024-03-11T21:41:09.328197+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:24.261974+0000","last_clean_scrub_stamp":"2024-03-11T21:41:09.328197+0000","objects_scrubbed":2,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":198,"num_read_kb":294,"num_write":179,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":2951840,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.5","version":"260'28","reported_seq":1030,"reported_epoch":414,"state":"active+clean","last_fresh":"2024-03-11T21:43:37.036196+0000","last_change":"2024-03-11T21:41:07.923860+0000","last_active":"2024-03-11T21:43:37.036196+0000","last_peered":"2024-03-11T21:43:37.036196+0000","last_clean":"2024-03-11T21:43:37.036196+0000","last_became_active":"2024-03-11T21:25:40.761457+0000","last_became_peered":"2024-03-11T21:25:40.761457+0000","last_unstale":"2024-03-11T21:43:37.036196+0000","last_undegraded":"2024-03-11T21:43:37.036196+0000","last_fullsized":"2024-03-11T21:43:37.036196+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"260'28","last_scrub_stamp":"2024-03-11T21:41:07.923729+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:39.525554+0000","last_clean_scrub_stamp":"2024-03-11T21:41:07.923729+0000","objects_scrubbed":8,"log_size":28,"log_dups_size":0,"ondisk_log_size":28,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:43:42.400285+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":154,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":43,"num_read_kb":33,"num_write":25,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":405,"reported_epoch":414,"state":"active+clean","last_fresh":"2024-03-11T21:43:37.039754+0000","last_change":"2024-03-11T21:41:00.152627+0000","last_active":"2024-03-11T21:43:37.039754+0000","last_peered":"2024-03-11T21:43:37.039754+0000","last_clean":"2024-03-11T21:43:37.039754+0000","last_became_active":"2024-03-11T21:34:41.139782+0000","last_became_peered":"2024-03-11T21:34:41.139782+0000","last_unstale":"2024-03-11T21:43:37.039754+0000","last_undegraded":"2024-03-11T21:43:37.039754+0000","last_fullsized":"2024-03-11T21:43:37.039754+0000","mapping_epoch":226,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":227,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:41:00.152377+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:32:21.266672+0000","last_clean_scrub_stamp":"2024-03-11T21:41:00.152377+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,5],"acting":[7,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.8","version":"415'7885","reported_seq":8051,"reported_epoch":415,"state":"active+clean","last_fresh":"2024-03-11T21:43:44.566991+0000","last_change":"2024-03-11T21:41:29.898309+0000","last_active":"2024-03-11T21:43:44.566991+0000","last_peered":"2024-03-11T21:43:44.566991+0000","last_clean":"2024-03-11T21:43:44.566991+0000","last_became_active":"2024-03-11T21:41:28.829878+0000","last_became_peered":"2024-03-11T21:41:28.829878+0000","last_unstale":"2024-03-11T21:43:44.566991+0000","last_undegraded":"2024-03-11T21:43:44.566991+0000","last_fullsized":"2024-03-11T21:43:44.566991+0000","mapping_epoch":356,"log_start":"366'2309","ondisk_log_start":"366'2309","created":356,"last_epoch_clean":373,"parent":"0.0","parent_split_bits":0,"last_scrub":"374'4392","last_scrub_stamp":"2024-03-11T21:41:29.898148+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:40:38.219602+0000","last_clean_scrub_stamp":"2024-03-11T21:41:29.898148+0000","objects_scrubbed":2198,"log_size":5576,"log_dups_size":0,"ondisk_log_size":5576,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:43:49.041646+0000","scrub_duration":722,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":74973184,"num_objects":1144,"num_object_clones":0,"num_object_copies":3432,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1144,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3512,"num_write_kb":78656,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,7],"acting":[0,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.4","version":"0'0","reported_seq":750,"reported_epoch":414,"state":"active+clean","last_fresh":"2024-03-11T21:43:37.048615+0000","last_change":"2024-03-11T21:41:01.134631+0000","last_active":"2024-03-11T21:43:37.048615+0000","last_peered":"2024-03-11T21:43:37.048615+0000","last_clean":"2024-03-11T21:43:37.048615+0000","last_became_active":"2024-03-11T21:36:35.072824+0000","last_became_peered":"2024-03-11T21:36:35.072824+0000","last_unstale":"2024-03-11T21:43:37.048615+0000","last_undegraded":"2024-03-11T21:43:37.048615+0000","last_fullsized":"2024-03-11T21:43:37.048615+0000","mapping_epoch":277,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":278,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:41:01.134504+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:43.559452+0000","last_clean_scrub_stamp":"2024-03-11T21:41:01.134504+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":779,"reported_epoch":415,"state":"active+clean","last_fresh":"2024-03-11T21:43:44.087415+0000","last_change":"2024-03-11T21:41:02.116795+0000","last_active":"2024-03-11T21:43:44.087415+0000","last_peered":"2024-03-11T21:43:44.087415+0000","last_clean":"2024-03-11T21:43:44.087415+0000","last_became_active":"2024-03-11T21:36:35.069452+0000","last_became_peered":"2024-03-11T21:36:35.069452+0000","last_unstale":"2024-03-11T21:43:44.087415+0000","last_undegraded":"2024-03-11T21:43:44.087415+0000","last_fullsized":"2024-03-11T21:43:44.087415+0000","mapping_epoch":277,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":278,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:41:02.116682+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:37.512018+0000","last_clean_scrub_stamp":"2024-03-11T21:41:02.116682+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:43:49.041457+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0.00089342499999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":405,"reported_epoch":414,"state":"active+clean","last_fresh":"2024-03-11T21:43:37.035726+0000","last_change":"2024-03-11T21:43:30.734142+0000","last_active":"2024-03-11T21:43:37.035726+0000","last_peered":"2024-03-11T21:43:37.035726+0000","last_clean":"2024-03-11T21:43:37.035726+0000","last_became_active":"2024-03-11T21:34:41.149219+0000","last_became_peered":"2024-03-11T21:34:41.149219+0000","last_unstale":"2024-03-11T21:43:37.035726+0000","last_undegraded":"2024-03-11T21:43:37.035726+0000","last_fullsized":"2024-03-11T21:43:37.035726+0000","mapping_epoch":226,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":227,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:43:30.734024+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:43:30.734024+0000","last_clean_scrub_stamp":"2024-03-11T21:43:30.734024+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:44:38.048220+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,0],"acting":[7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.6","version":"352'30","reported_seq":829,"reported_epoch":415,"state":"active+clean","last_fresh":"2024-03-11T21:43:44.088092+0000","last_change":"2024-03-11T21:43:29.087996+0000","last_active":"2024-03-11T21:43:44.088092+0000","last_peered":"2024-03-11T21:43:44.088092+0000","last_clean":"2024-03-11T21:43:44.088092+0000","last_became_active":"2024-03-11T21:32:22.796486+0000","last_became_peered":"2024-03-11T21:32:22.796486+0000","last_unstale":"2024-03-11T21:43:44.088092+0000","last_undegraded":"2024-03-11T21:43:44.088092+0000","last_fullsized":"2024-03-11T21:43:44.088092+0000","mapping_epoch":166,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":167,"parent":"0.0","parent_split_bits":0,"last_scrub":"352'30","last_scrub_stamp":"2024-03-11T21:43:29.087842+0000","last_deep_scrub":"352'30","last_deep_scrub_stamp":"2024-03-11T21:40:31.175323+0000","last_clean_scrub_stamp":"2024-03-11T21:43:29.087842+0000","objects_scrubbed":6,"log_size":30,"log_dups_size":0,"ondisk_log_size":30,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:44:50.495270+0000","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":36,"num_read_kb":30,"num_write":24,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":754,"reported_epoch":415,"state":"active+clean","last_fresh":"2024-03-11T21:43:44.085151+0000","last_change":"2024-03-11T21:43:31.468598+0000","last_active":"2024-03-11T21:43:44.085151+0000","last_peered":"2024-03-11T21:43:44.085151+0000","last_clean":"2024-03-11T21:43:44.085151+0000","last_became_active":"2024-03-11T21:25:42.572324+0000","last_became_peered":"2024-03-11T21:25:42.572324+0000","last_unstale":"2024-03-11T21:43:44.085151+0000","last_undegraded":"2024-03-11T21:43:44.085151+0000","last_fullsized":"2024-03-11T21:43:44.085151+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:43:31.468494+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:41.544550+0000","last_clean_scrub_stamp":"2024-03-11T21:43:31.468494+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:44:35.121710+0000","scrub_duration":32,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.d","version":"414'11168","reported_seq":11339,"reported_epoch":414,"state":"active+clean","last_fresh":"2024-03-11T21:43:42.489307+0000","last_change":"2024-03-11T21:43:34.312919+0000","last_active":"2024-03-11T21:43:42.489307+0000","last_peered":"2024-03-11T21:43:42.489307+0000","last_clean":"2024-03-11T21:43:42.489307+0000","last_became_active":"2024-03-11T21:40:39.241769+0000","last_became_peered":"2024-03-11T21:40:39.241769+0000","last_unstale":"2024-03-11T21:43:42.489307+0000","last_undegraded":"2024-03-11T21:43:42.489307+0000","last_fullsized":"2024-03-11T21:43:42.489307+0000","mapping_epoch":356,"log_start":"384'5600","ondisk_log_start":"384'5600","created":356,"last_epoch_clean":357,"parent":"0.0","parent_split_bits":0,"last_scrub":"413'10646","last_scrub_stamp":"2024-03-11T21:43:34.312786+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:40:38.219602+0000","last_clean_scrub_stamp":"2024-03-11T21:43:34.312786+0000","objects_scrubbed":3052,"log_size":5568,"log_dups_size":0,"ondisk_log_size":5568,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:44:44.600532+0000","scrub_duration":971,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":164167704,"num_objects":2506,"num_object_clones":0,"num_object_copies":7518,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2506,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":11168,"num_write_kb":437505,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,0],"acting":[6,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":749,"reported_epoch":414,"state":"active+clean","last_fresh":"2024-03-11T21:43:37.047994+0000","last_change":"2024-03-11T21:36:18.529506+0000","last_active":"2024-03-11T21:43:37.047994+0000","last_peered":"2024-03-11T21:43:37.047994+0000","last_clean":"2024-03-11T21:43:37.047994+0000","last_became_active":"2024-03-11T21:25:45.301416+0000","last_became_peered":"2024-03-11T21:25:45.301416+0000","last_unstale":"2024-03-11T21:43:37.047994+0000","last_undegraded":"2024-03-11T21:43:37.047994+0000","last_fullsized":"2024-03-11T21:43:37.047994+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:36:18.529370+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:43.559452+0000","last_clean_scrub_stamp":"2024-03-11T21:36:18.529370+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":54,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":405,"reported_epoch":414,"state":"active+clean","last_fresh":"2024-03-11T21:43:37.034790+0000","last_change":"2024-03-11T21:41:19.303174+0000","last_active":"2024-03-11T21:43:37.034790+0000","last_peered":"2024-03-11T21:43:37.034790+0000","last_clean":"2024-03-11T21:43:37.034790+0000","last_became_active":"2024-03-11T21:34:41.152721+0000","last_became_peered":"2024-03-11T21:34:41.152721+0000","last_unstale":"2024-03-11T21:43:37.034790+0000","last_undegraded":"2024-03-11T21:43:37.034790+0000","last_fullsized":"2024-03-11T21:43:37.034790+0000","mapping_epoch":226,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":227,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:41:19.302991+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:26:57.333669+0000","last_clean_scrub_stamp":"2024-03-11T21:41:19.302991+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.3","version":"260'37","reported_seq":866,"reported_epoch":415,"state":"active+clean","last_fresh":"2024-03-11T21:43:44.084427+0000","last_change":"2024-03-11T21:41:06.123397+0000","last_active":"2024-03-11T21:43:44.084427+0000","last_peered":"2024-03-11T21:43:44.084427+0000","last_clean":"2024-03-11T21:43:44.084427+0000","last_became_active":"2024-03-11T21:34:41.145607+0000","last_became_peered":"2024-03-11T21:34:41.145607+0000","last_unstale":"2024-03-11T21:43:44.084427+0000","last_undegraded":"2024-03-11T21:43:44.084427+0000","last_fullsized":"2024-03-11T21:43:44.084427+0000","mapping_epoch":226,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":227,"parent":"0.0","parent_split_bits":0,"last_scrub":"260'37","last_scrub_stamp":"2024-03-11T21:41:06.123055+0000","last_deep_scrub":"165'27","last_deep_scrub_stamp":"2024-03-11T21:32:21.576732+0000","last_clean_scrub_stamp":"2024-03-11T21:41:06.123055+0000","objects_scrubbed":8,"log_size":37,"log_dups_size":0,"ondisk_log_size":37,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:43:49.041490+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":54,"num_read_kb":40,"num_write":32,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":660,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":777,"reported_epoch":414,"state":"active+clean","last_fresh":"2024-03-11T21:43:37.038206+0000","last_change":"2024-03-11T21:41:21.857353+0000","last_active":"2024-03-11T21:43:37.038206+0000","last_peered":"2024-03-11T21:43:37.038206+0000","last_clean":"2024-03-11T21:43:37.038206+0000","last_became_active":"2024-03-11T21:25:42.579045+0000","last_became_peered":"2024-03-11T21:25:42.579045+0000","last_unstale":"2024-03-11T21:43:37.038206+0000","last_undegraded":"2024-03-11T21:43:37.038206+0000","last_fullsized":"2024-03-11T21:43:37.038206+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:41:21.857205+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:41:21.857205+0000","last_clean_scrub_stamp":"2024-03-11T21:41:21.857205+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"10.e","version":"415'11422","reported_seq":11554,"reported_epoch":415,"state":"active+clean","last_fresh":"2024-03-11T21:43:47.154503+0000","last_change":"2024-03-11T21:40:39.239887+0000","last_active":"2024-03-11T21:43:47.154503+0000","last_peered":"2024-03-11T21:43:47.154503+0000","last_clean":"2024-03-11T21:43:47.154503+0000","last_became_active":"2024-03-11T21:40:39.239590+0000","last_became_peered":"2024-03-11T21:40:39.239590+0000","last_unstale":"2024-03-11T21:43:47.154503+0000","last_undegraded":"2024-03-11T21:43:47.154503+0000","last_fullsized":"2024-03-11T21:43:47.154503+0000","mapping_epoch":356,"log_start":"393'6800","ondisk_log_start":"393'6800","created":356,"last_epoch_clean":357,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:40:38.219602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:40:38.219602+0000","last_clean_scrub_stamp":"2024-03-11T21:40:38.219602+0000","objects_scrubbed":0,"log_size":4622,"log_dups_size":0,"ondisk_log_size":4622,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:43:48.166287+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":135397376,"num_objects":2066,"num_object_clones":0,"num_object_copies":6198,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2066,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11422,"num_write_kb":431616,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,2],"acting":[4,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":757,"reported_epoch":415,"state":"active+clean","last_fresh":"2024-03-11T21:43:44.075916+0000","last_change":"2024-03-11T21:40:28.582452+0000","last_active":"2024-03-11T21:43:44.075916+0000","last_peered":"2024-03-11T21:43:44.075916+0000","last_clean":"2024-03-11T21:43:44.075916+0000","last_became_active":"2024-03-11T21:32:22.791439+0000","last_became_peered":"2024-03-11T21:32:22.791439+0000","last_unstale":"2024-03-11T21:43:44.075916+0000","last_undegraded":"2024-03-11T21:43:44.075916+0000","last_fullsized":"2024-03-11T21:43:44.075916+0000","mapping_epoch":166,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":167,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:40:28.582339+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:43.559452+0000","last_clean_scrub_stamp":"2024-03-11T21:40:28.582339+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:43:48.166313+0000","scrub_duration":20,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,2],"acting":[4,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.6","version":"67'1","reported_seq":762,"reported_epoch":414,"state":"active+clean","last_fresh":"2024-03-11T21:43:37.046237+0000","last_change":"2024-03-11T21:41:21.288511+0000","last_active":"2024-03-11T21:43:37.046237+0000","last_peered":"2024-03-11T21:43:37.046237+0000","last_clean":"2024-03-11T21:43:37.046237+0000","last_became_active":"2024-03-11T21:25:38.536142+0000","last_became_peered":"2024-03-11T21:25:38.536142+0000","last_unstale":"2024-03-11T21:43:37.046237+0000","last_undegraded":"2024-03-11T21:43:37.046237+0000","last_fullsized":"2024-03-11T21:43:37.046237+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'1","last_scrub_stamp":"2024-03-11T21:41:21.288031+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:37.512018+0000","last_clean_scrub_stamp":"2024-03-11T21:41:21.288031+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,4],"acting":[1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.0","version":"260'38","reported_seq":762,"reported_epoch":415,"state":"active+clean","last_fresh":"2024-03-11T21:43:44.082790+0000","last_change":"2024-03-11T21:41:30.511287+0000","last_active":"2024-03-11T21:43:44.082790+0000","last_peered":"2024-03-11T21:43:44.082790+0000","last_clean":"2024-03-11T21:43:44.082790+0000","last_became_active":"2024-03-11T21:36:35.068071+0000","last_became_peered":"2024-03-11T21:36:35.068071+0000","last_unstale":"2024-03-11T21:43:44.082790+0000","last_undegraded":"2024-03-11T21:43:44.082790+0000","last_fullsized":"2024-03-11T21:43:44.082790+0000","mapping_epoch":277,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":278,"parent":"0.0","parent_split_bits":0,"last_scrub":"260'38","last_scrub_stamp":"2024-03-11T21:41:30.511168+0000","last_deep_scrub":"93'26","last_deep_scrub_stamp":"2024-03-11T21:26:40.680324+0000","last_clean_scrub_stamp":"2024-03-11T21:41:30.511168+0000","objects_scrubbed":8,"log_size":38,"log_dups_size":0,"ondisk_log_size":38,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":51,"num_read_kb":39,"num_write":32,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":784,"reported_epoch":415,"state":"active+clean","last_fresh":"2024-03-11T21:43:44.075393+0000","last_change":"2024-03-11T21:43:30.090841+0000","last_active":"2024-03-11T21:43:44.075393+0000","last_peered":"2024-03-11T21:43:44.075393+0000","last_clean":"2024-03-11T21:43:44.075393+0000","last_became_active":"2024-03-11T21:36:35.057517+0000","last_became_peered":"2024-03-11T21:36:35.057517+0000","last_unstale":"2024-03-11T21:43:44.075393+0000","last_undegraded":"2024-03-11T21:43:44.075393+0000","last_fullsized":"2024-03-11T21:43:44.075393+0000","mapping_epoch":277,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":278,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:43:30.090729+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:40:21.518840+0000","last_clean_scrub_stamp":"2024-03-11T21:43:30.090729+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:44:36.823179+0000","scrub_duration":59,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,7],"acting":[4,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"10.f","version":"414'11412","reported_seq":11561,"reported_epoch":414,"state":"active+clean","last_fresh":"2024-03-11T21:43:42.475352+0000","last_change":"2024-03-11T21:43:37.054305+0000","last_active":"2024-03-11T21:43:42.475352+0000","last_peered":"2024-03-11T21:43:42.475352+0000","last_clean":"2024-03-11T21:43:42.475352+0000","last_became_active":"2024-03-11T21:40:39.541420+0000","last_became_peered":"2024-03-11T21:40:39.541420+0000","last_unstale":"2024-03-11T21:43:42.475352+0000","last_undegraded":"2024-03-11T21:43:42.475352+0000","last_fullsized":"2024-03-11T21:43:42.475352+0000","mapping_epoch":356,"log_start":"385'5800","ondisk_log_start":"385'5800","created":356,"last_epoch_clean":357,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:40:38.219602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:40:38.219602+0000","last_clean_scrub_stamp":"2024-03-11T21:40:38.219602+0000","objects_scrubbed":1549,"log_size":5612,"log_dups_size":0,"ondisk_log_size":5612,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:43:47.374000+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":169869312,"num_objects":2592,"num_object_clones":0,"num_object_copies":7776,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2592,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11412,"num_write_kb":448128,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,2],"acting":[6,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.3","version":"0'0","reported_seq":405,"reported_epoch":414,"state":"active+clean","last_fresh":"2024-03-11T21:43:37.036212+0000","last_change":"2024-03-11T21:43:25.746794+0000","last_active":"2024-03-11T21:43:37.036212+0000","last_peered":"2024-03-11T21:43:37.036212+0000","last_clean":"2024-03-11T21:43:37.036212+0000","last_became_active":"2024-03-11T21:34:41.142683+0000","last_became_peered":"2024-03-11T21:34:41.142683+0000","last_unstale":"2024-03-11T21:43:37.036212+0000","last_undegraded":"2024-03-11T21:43:37.036212+0000","last_fullsized":"2024-03-11T21:43:37.036212+0000","mapping_epoch":226,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":227,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:43:25.746680+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:29:42.485740+0000","last_clean_scrub_stamp":"2024-03-11T21:43:25.746680+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:44:53.921851+0000","scrub_duration":116,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,2],"acting":[7,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":797,"reported_epoch":414,"state":"active+clean","last_fresh":"2024-03-11T21:43:37.042409+0000","last_change":"2024-03-11T21:41:22.817490+0000","last_active":"2024-03-11T21:43:37.042409+0000","last_peered":"2024-03-11T21:43:37.042409+0000","last_clean":"2024-03-11T21:43:37.042409+0000","last_became_active":"2024-03-11T21:34:41.145628+0000","last_became_peered":"2024-03-11T21:34:41.145628+0000","last_unstale":"2024-03-11T21:43:37.042409+0000","last_undegraded":"2024-03-11T21:43:37.042409+0000","last_fullsized":"2024-03-11T21:43:37.042409+0000","mapping_epoch":226,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":227,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:41:22.817380+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:37.512018+0000","last_clean_scrub_stamp":"2024-03-11T21:41:22.817380+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,2],"acting":[6,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.1","version":"260'28","reported_seq":801,"reported_epoch":415,"state":"active+clean","last_fresh":"2024-03-11T21:43:44.075196+0000","last_change":"2024-03-11T21:41:07.513056+0000","last_active":"2024-03-11T21:43:44.075196+0000","last_peered":"2024-03-11T21:43:44.075196+0000","last_clean":"2024-03-11T21:43:44.075196+0000","last_became_active":"2024-03-11T21:25:40.546219+0000","last_became_peered":"2024-03-11T21:25:40.546219+0000","last_unstale":"2024-03-11T21:43:44.075196+0000","last_undegraded":"2024-03-11T21:43:44.075196+0000","last_fullsized":"2024-03-11T21:43:44.075196+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"260'28","last_scrub_stamp":"2024-03-11T21:41:07.512907+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:39.525554+0000","last_clean_scrub_stamp":"2024-03-11T21:41:07.512907+0000","objects_scrubbed":7,"log_size":28,"log_dups_size":0,"ondisk_log_size":28,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:43:48.166456+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":42,"num_read_kb":35,"num_write":28,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6],"acting":[4,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.0","version":"280'11","reported_seq":899,"reported_epoch":415,"state":"active+clean","last_fresh":"2024-03-11T21:43:44.082591+0000","last_change":"2024-03-11T21:36:35.064085+0000","last_active":"2024-03-11T21:43:44.082591+0000","last_peered":"2024-03-11T21:43:44.082591+0000","last_clean":"2024-03-11T21:43:44.082591+0000","last_became_active":"2024-03-11T21:36:35.055250+0000","last_became_peered":"2024-03-11T21:36:35.055250+0000","last_unstale":"2024-03-11T21:43:44.082591+0000","last_undegraded":"2024-03-11T21:43:44.082591+0000","last_fullsized":"2024-03-11T21:43:44.082591+0000","mapping_epoch":277,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":278,"parent":"0.0","parent_split_bits":0,"last_scrub":"98'9","last_scrub_stamp":"2024-03-11T21:35:21.143578+0000","last_deep_scrub":"98'9","last_deep_scrub_stamp":"2024-03-11T21:27:29.745711+0000","last_clean_scrub_stamp":"2024-03-11T21:35:21.143578+0000","objects_scrubbed":1,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.0","version":"414'7762","reported_seq":7927,"reported_epoch":414,"state":"active+clean","last_fresh":"2024-03-11T21:43:42.420182+0000","last_change":"2024-03-11T21:41:29.309475+0000","last_active":"2024-03-11T21:43:42.420182+0000","last_peered":"2024-03-11T21:43:42.420182+0000","last_clean":"2024-03-11T21:43:42.420182+0000","last_became_active":"2024-03-11T21:41:28.819914+0000","last_became_peered":"2024-03-11T21:41:28.819914+0000","last_unstale":"2024-03-11T21:43:42.420182+0000","last_undegraded":"2024-03-11T21:43:42.420182+0000","last_fullsized":"2024-03-11T21:43:42.420182+0000","mapping_epoch":356,"log_start":"0'0","ondisk_log_start":"0'0","created":356,"last_epoch_clean":373,"parent":"0.0","parent_split_bits":0,"last_scrub":"373'4340","last_scrub_stamp":"2024-03-11T21:41:29.309268+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:40:38.219602+0000","last_clean_scrub_stamp":"2024-03-11T21:41:29.309268+0000","objects_scrubbed":2123,"log_size":7762,"log_dups_size":0,"ondisk_log_size":7762,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":481,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":81920000,"num_objects":1250,"num_object_clones":0,"num_object_copies":3750,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1250,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3429,"num_write_kb":81792,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,4],"acting":[6,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.c","version":"0'0","reported_seq":683,"reported_epoch":415,"state":"active+clean","last_fresh":"2024-03-11T21:43:44.083353+0000","last_change":"2024-03-11T21:40:30.659272+0000","last_active":"2024-03-11T21:43:44.083353+0000","last_peered":"2024-03-11T21:43:44.083353+0000","last_clean":"2024-03-11T21:43:44.083353+0000","last_became_active":"2024-03-11T21:36:35.073778+0000","last_became_peered":"2024-03-11T21:36:35.073778+0000","last_unstale":"2024-03-11T21:43:44.083353+0000","last_undegraded":"2024-03-11T21:43:44.083353+0000","last_fullsized":"2024-03-11T21:43:44.083353+0000","mapping_epoch":277,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":278,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:40:30.659160+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:43.559452+0000","last_clean_scrub_stamp":"2024-03-11T21:40:30.659160+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.f","version":"0'0","reported_seq":761,"reported_epoch":415,"state":"active+clean","last_fresh":"2024-03-11T21:43:44.084884+0000","last_change":"2024-03-11T21:43:24.002447+0000","last_active":"2024-03-11T21:43:44.084884+0000","last_peered":"2024-03-11T21:43:44.084884+0000","last_clean":"2024-03-11T21:43:44.084884+0000","last_became_active":"2024-03-11T21:25:42.575935+0000","last_became_peered":"2024-03-11T21:25:42.575935+0000","last_unstale":"2024-03-11T21:43:44.084884+0000","last_undegraded":"2024-03-11T21:43:44.084884+0000","last_fullsized":"2024-03-11T21:43:44.084884+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:43:24.002346+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:43:24.002346+0000","last_clean_scrub_stamp":"2024-03-11T21:43:24.002346+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:44:26.043970+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.8","version":"0'0","reported_seq":405,"reported_epoch":414,"state":"active+clean","last_fresh":"2024-03-11T21:43:37.036282+0000","last_change":"2024-03-11T21:41:22.253981+0000","last_active":"2024-03-11T21:43:37.036282+0000","last_peered":"2024-03-11T21:43:37.036282+0000","last_clean":"2024-03-11T21:43:37.036282+0000","last_became_active":"2024-03-11T21:34:41.144681+0000","last_became_peered":"2024-03-11T21:34:41.144681+0000","last_unstale":"2024-03-11T21:43:37.036282+0000","last_undegraded":"2024-03-11T21:43:37.036282+0000","last_fullsized":"2024-03-11T21:43:37.036282+0000","mapping_epoch":226,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":227,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:41:22.253866+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:37.512018+0000","last_clean_scrub_stamp":"2024-03-11T21:41:22.253866+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,1],"acting":[7,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.e","version":"260'24","reported_seq":795,"reported_epoch":415,"state":"active+clean","last_fresh":"2024-03-11T21:43:44.075626+0000","last_change":"2024-03-11T21:41:14.567374+0000","last_active":"2024-03-11T21:43:44.075626+0000","last_peered":"2024-03-11T21:43:44.075626+0000","last_clean":"2024-03-11T21:43:44.075626+0000","last_became_active":"2024-03-11T21:25:40.552178+0000","last_became_peered":"2024-03-11T21:25:40.552178+0000","last_unstale":"2024-03-11T21:43:44.075626+0000","last_undegraded":"2024-03-11T21:43:44.075626+0000","last_fullsized":"2024-03-11T21:43:44.075626+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"260'24","last_scrub_stamp":"2024-03-11T21:41:14.567244+0000","last_deep_scrub":"109'16","last_deep_scrub_stamp":"2024-03-11T21:28:24.739395+0000","last_clean_scrub_stamp":"2024-03-11T21:41:14.567244+0000","objects_scrubbed":5,"log_size":24,"log_dups_size":0,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:43:48.166397+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":32,"num_read_kb":25,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"10.1","version":"415'8086","reported_seq":8255,"reported_epoch":415,"state":"active+clean","last_fresh":"2024-03-11T21:43:47.093587+0000","last_change":"2024-03-11T21:41:30.222240+0000","last_active":"2024-03-11T21:43:47.093587+0000","last_peered":"2024-03-11T21:43:47.093587+0000","last_clean":"2024-03-11T21:43:47.093587+0000","last_became_active":"2024-03-11T21:41:28.822620+0000","last_became_peered":"2024-03-11T21:41:28.822620+0000","last_unstale":"2024-03-11T21:43:47.093587+0000","last_undegraded":"2024-03-11T21:43:47.093587+0000","last_fullsized":"2024-03-11T21:43:47.093587+0000","mapping_epoch":356,"log_start":"366'2739","ondisk_log_start":"366'2739","created":356,"last_epoch_clean":373,"parent":"0.0","parent_split_bits":0,"last_scrub":"374'4413","last_scrub_stamp":"2024-03-11T21:41:30.221933+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:40:38.219602+0000","last_clean_scrub_stamp":"2024-03-11T21:41:30.221933+0000","objects_scrubbed":2282,"log_size":5347,"log_dups_size":0,"ondisk_log_size":5347,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:43:48.166815+0000","scrub_duration":640,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":71041024,"num_objects":1084,"num_object_clones":0,"num_object_copies":3252,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1084,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3674,"num_write_kb":79232,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,0],"acting":[4,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.d","version":"0'0","reported_seq":750,"reported_epoch":415,"state":"active+clean","last_fresh":"2024-03-11T21:43:44.085249+0000","last_change":"2024-03-11T21:41:15.915172+0000","last_active":"2024-03-11T21:43:44.085249+0000","last_peered":"2024-03-11T21:43:44.085249+0000","last_clean":"2024-03-11T21:43:44.085249+0000","last_became_active":"2024-03-11T21:25:44.590333+0000","last_became_peered":"2024-03-11T21:25:44.590333+0000","last_unstale":"2024-03-11T21:43:44.085249+0000","last_undegraded":"2024-03-11T21:43:44.085249+0000","last_fullsized":"2024-03-11T21:43:44.085249+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:41:15.915041+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:35:37.182850+0000","last_clean_scrub_stamp":"2024-03-11T21:41:15.915041+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:43:45.574741+0000","scrub_duration":117,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.e","version":"74'8","reported_seq":981,"reported_epoch":415,"state":"active+clean","last_fresh":"2024-03-11T21:43:44.076129+0000","last_change":"2024-03-11T21:41:16.574147+0000","last_active":"2024-03-11T21:43:44.076129+0000","last_peered":"2024-03-11T21:43:44.076129+0000","last_clean":"2024-03-11T21:43:44.076129+0000","last_became_active":"2024-03-11T21:25:42.576742+0000","last_became_peered":"2024-03-11T21:25:42.576742+0000","last_unstale":"2024-03-11T21:43:44.076129+0000","last_undegraded":"2024-03-11T21:43:44.076129+0000","last_fullsized":"2024-03-11T21:43:44.076129+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'8","last_scrub_stamp":"2024-03-11T21:41:16.573269+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:41.544550+0000","last_clean_scrub_stamp":"2024-03-11T21:41:16.573269+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:43:48.166747+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,0],"acting":[4,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.9","version":"0'0","reported_seq":774,"reported_epoch":414,"state":"active+clean","last_fresh":"2024-03-11T21:43:37.047807+0000","last_change":"2024-03-11T21:40:29.915912+0000","last_active":"2024-03-11T21:43:37.047807+0000","last_peered":"2024-03-11T21:43:37.047807+0000","last_clean":"2024-03-11T21:43:37.047807+0000","last_became_active":"2024-03-11T21:36:35.065733+0000","last_became_peered":"2024-03-11T21:36:35.065733+0000","last_unstale":"2024-03-11T21:43:37.047807+0000","last_undegraded":"2024-03-11T21:43:37.047807+0000","last_fullsized":"2024-03-11T21:43:37.047807+0000","mapping_epoch":277,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":278,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:40:29.915835+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:26:50.991712+0000","last_clean_scrub_stamp":"2024-03-11T21:40:29.915835+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,3],"acting":[1,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.f","version":"260'27","reported_seq":782,"reported_epoch":414,"state":"active+clean","last_fresh":"2024-03-11T21:43:37.031561+0000","last_change":"2024-03-11T21:41:31.414618+0000","last_active":"2024-03-11T21:43:37.031561+0000","last_peered":"2024-03-11T21:43:37.031561+0000","last_clean":"2024-03-11T21:43:37.031561+0000","last_became_active":"2024-03-11T21:36:35.058272+0000","last_became_peered":"2024-03-11T21:36:35.058272+0000","last_unstale":"2024-03-11T21:43:37.031561+0000","last_undegraded":"2024-03-11T21:43:37.031561+0000","last_fullsized":"2024-03-11T21:43:37.031561+0000","mapping_epoch":277,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":278,"parent":"0.0","parent_split_bits":0,"last_scrub":"260'27","last_scrub_stamp":"2024-03-11T21:41:31.414510+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:39.525554+0000","last_clean_scrub_stamp":"2024-03-11T21:41:31.414510+0000","objects_scrubbed":7,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":44,"num_read_kb":35,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,4],"acting":[2,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"10.2","version":"414'7935","reported_seq":8099,"reported_epoch":414,"state":"active+clean","last_fresh":"2024-03-11T21:43:43.449324+0000","last_change":"2024-03-11T21:41:29.850911+0000","last_active":"2024-03-11T21:43:43.449324+0000","last_peered":"2024-03-11T21:43:43.449324+0000","last_clean":"2024-03-11T21:43:43.449324+0000","last_became_active":"2024-03-11T21:41:28.834067+0000","last_became_peered":"2024-03-11T21:41:28.834067+0000","last_unstale":"2024-03-11T21:43:43.449324+0000","last_undegraded":"2024-03-11T21:43:43.449324+0000","last_fullsized":"2024-03-11T21:43:43.449324+0000","mapping_epoch":356,"log_start":"365'1520","ondisk_log_start":"365'1520","created":356,"last_epoch_clean":373,"parent":"0.0","parent_split_bits":0,"last_scrub":"373'4444","last_scrub_stamp":"2024-03-11T21:41:29.850283+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:40:38.219602+0000","last_clean_scrub_stamp":"2024-03-11T21:41:29.850283+0000","objects_scrubbed":2169,"log_size":6415,"log_dups_size":0,"ondisk_log_size":6415,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:43:44.069696+0000","scrub_duration":620,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":81199104,"num_objects":1239,"num_object_clones":0,"num_object_copies":3717,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1239,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3498,"num_write_kb":82176,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.e","version":"0'0","reported_seq":751,"reported_epoch":415,"state":"active+clean","last_fresh":"2024-03-11T21:43:44.076230+0000","last_change":"2024-03-11T21:41:20.548370+0000","last_active":"2024-03-11T21:43:44.076230+0000","last_peered":"2024-03-11T21:43:44.076230+0000","last_clean":"2024-03-11T21:43:44.076230+0000","last_became_active":"2024-03-11T21:25:44.590198+0000","last_became_peered":"2024-03-11T21:25:44.590198+0000","last_unstale":"2024-03-11T21:43:44.076230+0000","last_undegraded":"2024-03-11T21:43:44.076230+0000","last_fullsized":"2024-03-11T21:43:44.076230+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:41:20.548259+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:43.559452+0000","last_clean_scrub_stamp":"2024-03-11T21:41:20.548259+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:43:48.166871+0000","scrub_duration":42,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.d","version":"109'9","reported_seq":977,"reported_epoch":414,"state":"active+clean","last_fresh":"2024-03-11T21:43:38.807802+0000","last_change":"2024-03-11T21:41:06.117858+0000","last_active":"2024-03-11T21:43:38.807802+0000","last_peered":"2024-03-11T21:43:38.807802+0000","last_clean":"2024-03-11T21:43:38.807802+0000","last_became_active":"2024-03-11T21:25:42.575583+0000","last_became_peered":"2024-03-11T21:25:42.575583+0000","last_unstale":"2024-03-11T21:43:38.807802+0000","last_undegraded":"2024-03-11T21:43:38.807802+0000","last_fullsized":"2024-03-11T21:43:38.807802+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"109'9","last_scrub_stamp":"2024-03-11T21:41:06.117483+0000","last_deep_scrub":"109'9","last_deep_scrub_stamp":"2024-03-11T21:28:27.082975+0000","last_clean_scrub_stamp":"2024-03-11T21:41:06.117483+0000","objects_scrubbed":1,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:43:45.240557+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,5],"acting":[2,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.a","version":"0'0","reported_seq":782,"reported_epoch":414,"state":"active+clean","last_fresh":"2024-03-11T21:43:37.049415+0000","last_change":"2024-03-11T21:41:02.132633+0000","last_active":"2024-03-11T21:43:37.049415+0000","last_peered":"2024-03-11T21:43:37.049415+0000","last_clean":"2024-03-11T21:43:37.049415+0000","last_became_active":"2024-03-11T21:36:35.066832+0000","last_became_peered":"2024-03-11T21:36:35.066832+0000","last_unstale":"2024-03-11T21:43:37.049415+0000","last_undegraded":"2024-03-11T21:43:37.049415+0000","last_fullsized":"2024-03-11T21:43:37.049415+0000","mapping_epoch":277,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":278,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:41:02.132355+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:37.512018+0000","last_clean_scrub_stamp":"2024-03-11T21:41:02.132355+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,7],"acting":[1,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.c","version":"260'20","reported_seq":811,"reported_epoch":415,"state":"active+clean","last_fresh":"2024-03-11T21:43:44.076901+0000","last_change":"2024-03-11T21:41:30.618505+0000","last_active":"2024-03-11T21:43:44.076901+0000","last_peered":"2024-03-11T21:43:44.076901+0000","last_clean":"2024-03-11T21:43:44.076901+0000","last_became_active":"2024-03-11T21:36:35.067269+0000","last_became_peered":"2024-03-11T21:36:35.067269+0000","last_unstale":"2024-03-11T21:43:44.076901+0000","last_undegraded":"2024-03-11T21:43:44.076901+0000","last_fullsized":"2024-03-11T21:43:44.076901+0000","mapping_epoch":277,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":278,"parent":"0.0","parent_split_bits":0,"last_scrub":"260'20","last_scrub_stamp":"2024-03-11T21:41:30.618393+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:39.525554+0000","last_clean_scrub_stamp":"2024-03-11T21:41:30.618393+0000","objects_scrubbed":5,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:43:48.166900+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":25,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"10.3","version":"414'7770","reported_seq":7938,"reported_epoch":414,"state":"active+clean","last_fresh":"2024-03-11T21:43:43.124478+0000","last_change":"2024-03-11T21:41:30.759270+0000","last_active":"2024-03-11T21:43:43.124478+0000","last_peered":"2024-03-11T21:43:43.124478+0000","last_clean":"2024-03-11T21:43:43.124478+0000","last_became_active":"2024-03-11T21:41:28.840374+0000","last_became_peered":"2024-03-11T21:41:28.840374+0000","last_unstale":"2024-03-11T21:43:43.124478+0000","last_undegraded":"2024-03-11T21:43:43.124478+0000","last_fullsized":"2024-03-11T21:43:43.124478+0000","mapping_epoch":356,"log_start":"0'0","ondisk_log_start":"0'0","created":356,"last_epoch_clean":373,"parent":"0.0","parent_split_bits":0,"last_scrub":"373'4357","last_scrub_stamp":"2024-03-11T21:41:30.759188+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:40:38.219602+0000","last_clean_scrub_stamp":"2024-03-11T21:41:30.759188+0000","objects_scrubbed":2208,"log_size":7770,"log_dups_size":0,"ondisk_log_size":7770,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:43:45.240730+0000","scrub_duration":314,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":75825152,"num_objects":1157,"num_object_clones":0,"num_object_copies":3471,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1157,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3413,"num_write_kb":75584,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,3],"acting":[2,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.f","version":"0'0","reported_seq":755,"reported_epoch":414,"state":"active+clean","last_fresh":"2024-03-11T21:43:37.031317+0000","last_change":"2024-03-11T21:41:17.377926+0000","last_active":"2024-03-11T21:43:37.031317+0000","last_peered":"2024-03-11T21:43:37.031317+0000","last_clean":"2024-03-11T21:43:37.031317+0000","last_became_active":"2024-03-11T21:36:35.057800+0000","last_became_peered":"2024-03-11T21:36:35.057800+0000","last_unstale":"2024-03-11T21:43:37.031317+0000","last_undegraded":"2024-03-11T21:43:37.031317+0000","last_fullsized":"2024-03-11T21:43:37.031317+0000","mapping_epoch":277,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":278,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:41:17.377779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:27:03.010969+0000","last_clean_scrub_stamp":"2024-03-11T21:41:17.377779+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":149,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,4],"acting":[2,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":753,"reported_epoch":414,"state":"active+clean","last_fresh":"2024-03-11T21:43:37.048563+0000","last_change":"2024-03-11T21:41:31.221210+0000","last_active":"2024-03-11T21:43:37.048563+0000","last_peered":"2024-03-11T21:43:37.048563+0000","last_clean":"2024-03-11T21:43:37.048563+0000","last_became_active":"2024-03-11T21:25:42.579409+0000","last_became_peered":"2024-03-11T21:25:42.579409+0000","last_unstale":"2024-03-11T21:43:37.048563+0000","last_undegraded":"2024-03-11T21:43:37.048563+0000","last_fullsized":"2024-03-11T21:43:37.048563+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:41:31.221094+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:30:22.750053+0000","last_clean_scrub_stamp":"2024-03-11T21:41:31.221094+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:43:44.069577+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.b","version":"0'0","reported_seq":405,"reported_epoch":414,"state":"active+clean","last_fresh":"2024-03-11T21:43:37.036339+0000","last_change":"2024-03-11T21:41:29.255398+0000","last_active":"2024-03-11T21:43:37.036339+0000","last_peered":"2024-03-11T21:43:37.036339+0000","last_clean":"2024-03-11T21:43:37.036339+0000","last_became_active":"2024-03-11T21:34:41.134987+0000","last_became_peered":"2024-03-11T21:34:41.134987+0000","last_unstale":"2024-03-11T21:43:37.036339+0000","last_undegraded":"2024-03-11T21:43:37.036339+0000","last_fullsized":"2024-03-11T21:43:37.036339+0000","mapping_epoch":226,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":227,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:41:29.255282+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:36:16.719309+0000","last_clean_scrub_stamp":"2024-03-11T21:41:29.255282+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,5],"acting":[7,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.d","version":"260'39","reported_seq":816,"reported_epoch":415,"state":"active+clean","last_fresh":"2024-03-11T21:43:44.075797+0000","last_change":"2024-03-11T21:41:11.544772+0000","last_active":"2024-03-11T21:43:44.075797+0000","last_peered":"2024-03-11T21:43:44.075797+0000","last_clean":"2024-03-11T21:43:44.075797+0000","last_became_active":"2024-03-11T21:25:40.553454+0000","last_became_peered":"2024-03-11T21:25:40.553454+0000","last_unstale":"2024-03-11T21:43:44.075797+0000","last_undegraded":"2024-03-11T21:43:44.075797+0000","last_fullsized":"2024-03-11T21:43:44.075797+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"260'39","last_scrub_stamp":"2024-03-11T21:41:11.544605+0000","last_deep_scrub":"165'25","last_deep_scrub_stamp":"2024-03-11T21:32:21.678328+0000","last_clean_scrub_stamp":"2024-03-11T21:41:11.544605+0000","objects_scrubbed":8,"log_size":39,"log_dups_size":0,"ondisk_log_size":39,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:43:48.166425+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":50,"num_read_kb":40,"num_write":32,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,1],"acting":[4,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"10.4","version":"415'7586","reported_seq":7751,"reported_epoch":415,"state":"active+clean","last_fresh":"2024-03-11T21:43:44.548813+0000","last_change":"2024-03-11T21:41:30.436753+0000","last_active":"2024-03-11T21:43:44.548813+0000","last_peered":"2024-03-11T21:43:44.548813+0000","last_clean":"2024-03-11T21:43:44.548813+0000","last_became_active":"2024-03-11T21:41:28.837095+0000","last_became_peered":"2024-03-11T21:41:28.837095+0000","last_unstale":"2024-03-11T21:43:44.548813+0000","last_undegraded":"2024-03-11T21:43:44.548813+0000","last_fullsized":"2024-03-11T21:43:44.548813+0000","mapping_epoch":356,"log_start":"366'1845","ondisk_log_start":"366'1845","created":356,"last_epoch_clean":373,"parent":"0.0","parent_split_bits":0,"last_scrub":"374'4241","last_scrub_stamp":"2024-03-11T21:41:30.436548+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:40:38.219602+0000","last_clean_scrub_stamp":"2024-03-11T21:41:30.436548+0000","objects_scrubbed":2083,"log_size":5741,"log_dups_size":0,"ondisk_log_size":5741,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:43:49.041614+0000","scrub_duration":281,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":74579968,"num_objects":1138,"num_object_clones":0,"num_object_copies":3414,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1138,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3345,"num_write_kb":76800,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":405,"reported_epoch":414,"state":"active+clean","last_fresh":"2024-03-11T21:43:37.036015+0000","last_change":"2024-03-11T21:41:26.263600+0000","last_active":"2024-03-11T21:43:37.036015+0000","last_peered":"2024-03-11T21:43:37.036015+0000","last_clean":"2024-03-11T21:43:37.036015+0000","last_became_active":"2024-03-11T21:34:41.145012+0000","last_became_peered":"2024-03-11T21:34:41.145012+0000","last_unstale":"2024-03-11T21:43:37.036015+0000","last_undegraded":"2024-03-11T21:43:37.036015+0000","last_fullsized":"2024-03-11T21:43:37.036015+0000","mapping_epoch":226,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":227,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:41:26.263517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:28:37.561826+0000","last_clean_scrub_stamp":"2024-03-11T21:41:26.263517+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":48,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,0],"acting":[7,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":751,"reported_epoch":414,"state":"active+clean","last_fresh":"2024-03-11T21:43:37.030953+0000","last_change":"2024-03-11T21:41:09.181704+0000","last_active":"2024-03-11T21:43:37.030953+0000","last_peered":"2024-03-11T21:43:37.030953+0000","last_clean":"2024-03-11T21:43:37.030953+0000","last_became_active":"2024-03-11T21:25:42.574515+0000","last_became_peered":"2024-03-11T21:25:42.574515+0000","last_unstale":"2024-03-11T21:43:37.030953+0000","last_undegraded":"2024-03-11T21:43:37.030953+0000","last_fullsized":"2024-03-11T21:43:37.030953+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:41:09.181567+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:41.544550+0000","last_clean_scrub_stamp":"2024-03-11T21:41:09.181567+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:43:45.240584+0000","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,5],"acting":[2,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.c","version":"0'0","reported_seq":759,"reported_epoch":414,"state":"active+clean","last_fresh":"2024-03-11T21:43:37.030944+0000","last_change":"2024-03-11T21:41:24.303833+0000","last_active":"2024-03-11T21:43:37.030944+0000","last_peered":"2024-03-11T21:43:37.030944+0000","last_clean":"2024-03-11T21:43:37.030944+0000","last_became_active":"2024-03-11T21:25:38.533240+0000","last_became_peered":"2024-03-11T21:25:38.533240+0000","last_unstale":"2024-03-11T21:43:37.030944+0000","last_undegraded":"2024-03-11T21:43:37.030944+0000","last_fullsized":"2024-03-11T21:43:37.030944+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:41:24.303681+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:29:30.279910+0000","last_clean_scrub_stamp":"2024-03-11T21:41:24.303681+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,0],"acting":[2,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.a","version":"260'44","reported_seq":897,"reported_epoch":414,"state":"active+clean","last_fresh":"2024-03-11T21:43:37.037178+0000","last_change":"2024-03-11T21:43:30.364236+0000","last_active":"2024-03-11T21:43:37.037178+0000","last_peered":"2024-03-11T21:43:37.037178+0000","last_clean":"2024-03-11T21:43:37.037178+0000","last_became_active":"2024-03-11T21:34:41.148863+0000","last_became_peered":"2024-03-11T21:34:41.148863+0000","last_unstale":"2024-03-11T21:43:37.037178+0000","last_undegraded":"2024-03-11T21:43:37.037178+0000","last_fullsized":"2024-03-11T21:43:37.037178+0000","mapping_epoch":226,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":227,"parent":"0.0","parent_split_bits":0,"last_scrub":"260'44","last_scrub_stamp":"2024-03-11T21:43:30.359516+0000","last_deep_scrub":"109'28","last_deep_scrub_stamp":"2024-03-11T21:28:24.016389+0000","last_clean_scrub_stamp":"2024-03-11T21:43:30.359516+0000","objects_scrubbed":9,"log_size":44,"log_dups_size":0,"ondisk_log_size":44,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:44:30.404654+0000","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":56,"num_read_kb":45,"num_write":36,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,7],"acting":[6,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"10.5","version":"414'8754","reported_seq":8987,"reported_epoch":414,"state":"active+clean","last_fresh":"2024-03-11T21:43:43.222243+0000","last_change":"2024-03-11T21:41:32.489861+0000","last_active":"2024-03-11T21:43:43.222243+0000","last_peered":"2024-03-11T21:43:43.222243+0000","last_clean":"2024-03-11T21:43:43.222243+0000","last_became_active":"2024-03-11T21:41:28.844869+0000","last_became_peered":"2024-03-11T21:41:28.844869+0000","last_unstale":"2024-03-11T21:43:43.222243+0000","last_undegraded":"2024-03-11T21:43:43.222243+0000","last_fullsized":"2024-03-11T21:43:43.222243+0000","mapping_epoch":356,"log_start":"365'1395","ondisk_log_start":"365'1395","created":356,"last_epoch_clean":373,"parent":"0.0","parent_split_bits":0,"last_scrub":"376'5260","last_scrub_stamp":"2024-03-11T21:41:32.489702+0000","last_deep_scrub":"376'5260","last_deep_scrub_stamp":"2024-03-11T21:41:32.489702+0000","last_clean_scrub_stamp":"2024-03-11T21:41:32.489702+0000","objects_scrubbed":2205,"log_size":7359,"log_dups_size":0,"ondisk_log_size":7359,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":4,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:43:45.240792+0000","scrub_duration":3023,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":81264640,"num_objects":1240,"num_object_clones":0,"num_object_copies":3720,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1240,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3497,"num_write_kb":81024,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,3],"acting":[2,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":763,"reported_epoch":415,"state":"active+clean","last_fresh":"2024-03-11T21:43:44.087639+0000","last_change":"2024-03-11T21:41:23.288877+0000","last_active":"2024-03-11T21:43:44.087639+0000","last_peered":"2024-03-11T21:43:44.087639+0000","last_clean":"2024-03-11T21:43:44.087639+0000","last_became_active":"2024-03-11T21:34:41.154507+0000","last_became_peered":"2024-03-11T21:34:41.154507+0000","last_unstale":"2024-03-11T21:43:44.087639+0000","last_undegraded":"2024-03-11T21:43:44.087639+0000","last_fullsized":"2024-03-11T21:43:44.087639+0000","mapping_epoch":226,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":227,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:41:23.288651+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:27:11.361913+0000","last_clean_scrub_stamp":"2024-03-11T21:41:23.288651+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:43:49.041549+0000","scrub_duration":136,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":760,"reported_epoch":414,"state":"active+clean","last_fresh":"2024-03-11T21:43:37.030298+0000","last_change":"2024-03-11T21:43:32.115667+0000","last_active":"2024-03-11T21:43:37.030298+0000","last_peered":"2024-03-11T21:43:37.030298+0000","last_clean":"2024-03-11T21:43:37.030298+0000","last_became_active":"2024-03-11T21:36:35.059192+0000","last_became_peered":"2024-03-11T21:36:35.059192+0000","last_unstale":"2024-03-11T21:43:37.030298+0000","last_undegraded":"2024-03-11T21:43:37.030298+0000","last_fullsized":"2024-03-11T21:43:37.030298+0000","mapping_epoch":277,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":278,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:43:32.115556+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:43:32.115556+0000","last_clean_scrub_stamp":"2024-03-11T21:43:32.115556+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:44:42.203117+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,3],"acting":[2,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.d","version":"0'0","reported_seq":762,"reported_epoch":414,"state":"active+clean","last_fresh":"2024-03-11T21:43:37.045140+0000","last_change":"2024-03-11T21:40:31.873781+0000","last_active":"2024-03-11T21:43:37.045140+0000","last_peered":"2024-03-11T21:43:37.045140+0000","last_clean":"2024-03-11T21:43:37.045140+0000","last_became_active":"2024-03-11T21:36:35.058296+0000","last_became_peered":"2024-03-11T21:36:35.058296+0000","last_unstale":"2024-03-11T21:43:37.045140+0000","last_undegraded":"2024-03-11T21:43:37.045140+0000","last_fullsized":"2024-03-11T21:43:37.045140+0000","mapping_epoch":277,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":278,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:40:31.873695+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:37.512018+0000","last_clean_scrub_stamp":"2024-03-11T21:40:31.873695+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,3],"acting":[1,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.b","version":"260'15","reported_seq":784,"reported_epoch":415,"state":"active+clean","last_fresh":"2024-03-11T21:43:44.084317+0000","last_change":"2024-03-11T21:43:20.043522+0000","last_active":"2024-03-11T21:43:44.084317+0000","last_peered":"2024-03-11T21:43:44.084317+0000","last_clean":"2024-03-11T21:43:44.084317+0000","last_became_active":"2024-03-11T21:25:40.549612+0000","last_became_peered":"2024-03-11T21:25:40.549612+0000","last_unstale":"2024-03-11T21:43:44.084317+0000","last_undegraded":"2024-03-11T21:43:44.084317+0000","last_fullsized":"2024-03-11T21:43:44.084317+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"260'15","last_scrub_stamp":"2024-03-11T21:43:20.043394+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:39.525554+0000","last_clean_scrub_stamp":"2024-03-11T21:43:20.043394+0000","objects_scrubbed":4,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:44:35.981062+0000","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.6","version":"415'7975","reported_seq":8142,"reported_epoch":415,"state":"active+clean","last_fresh":"2024-03-11T21:43:47.067873+0000","last_change":"2024-03-11T21:41:30.231854+0000","last_active":"2024-03-11T21:43:47.067873+0000","last_peered":"2024-03-11T21:43:47.067873+0000","last_clean":"2024-03-11T21:43:47.067873+0000","last_became_active":"2024-03-11T21:41:28.841669+0000","last_became_peered":"2024-03-11T21:41:28.841669+0000","last_unstale":"2024-03-11T21:43:47.067873+0000","last_undegraded":"2024-03-11T21:43:47.067873+0000","last_fullsized":"2024-03-11T21:43:47.067873+0000","mapping_epoch":356,"log_start":"365'1310","ondisk_log_start":"365'1310","created":356,"last_epoch_clean":373,"parent":"0.0","parent_split_bits":0,"last_scrub":"374'4344","last_scrub_stamp":"2024-03-11T21:41:30.231749+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:40:38.219602+0000","last_clean_scrub_stamp":"2024-03-11T21:41:30.231749+0000","objects_scrubbed":2205,"log_size":6665,"log_dups_size":0,"ondisk_log_size":6665,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":259,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":71958528,"num_objects":1098,"num_object_clones":0,"num_object_copies":3294,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1098,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3631,"num_write_kb":80768,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,4],"acting":[5,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":664,"reported_epoch":415,"state":"active+clean","last_fresh":"2024-03-11T21:43:44.075201+0000","last_change":"2024-03-11T21:41:12.624195+0000","last_active":"2024-03-11T21:43:44.075201+0000","last_peered":"2024-03-11T21:43:44.075201+0000","last_clean":"2024-03-11T21:43:44.075201+0000","last_became_active":"2024-03-11T21:36:35.056518+0000","last_became_peered":"2024-03-11T21:36:35.056518+0000","last_unstale":"2024-03-11T21:43:44.075201+0000","last_undegraded":"2024-03-11T21:43:44.075201+0000","last_fullsized":"2024-03-11T21:43:44.075201+0000","mapping_epoch":277,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":278,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:41:12.624073+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:43.559452+0000","last_clean_scrub_stamp":"2024-03-11T21:41:12.624073+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:43:48.166782+0000","scrub_duration":41,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,3],"acting":[4,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.e","version":"0'0","reported_seq":788,"reported_epoch":414,"state":"active+clean","last_fresh":"2024-03-11T21:43:37.031283+0000","last_change":"2024-03-11T21:41:27.388270+0000","last_active":"2024-03-11T21:43:37.031283+0000","last_peered":"2024-03-11T21:43:37.031283+0000","last_clean":"2024-03-11T21:43:37.031283+0000","last_became_active":"2024-03-11T21:36:35.056777+0000","last_became_peered":"2024-03-11T21:36:35.056777+0000","last_unstale":"2024-03-11T21:43:37.031283+0000","last_undegraded":"2024-03-11T21:43:37.031283+0000","last_fullsized":"2024-03-11T21:43:37.031283+0000","mapping_epoch":277,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":278,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:41:27.387998+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:37.512018+0000","last_clean_scrub_stamp":"2024-03-11T21:41:27.387998+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:43:45.240641+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,7],"acting":[2,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.8","version":"260'34","reported_seq":852,"reported_epoch":415,"state":"active+clean","last_fresh":"2024-03-11T21:43:44.084130+0000","last_change":"2024-03-11T21:41:06.881563+0000","last_active":"2024-03-11T21:43:44.084130+0000","last_peered":"2024-03-11T21:43:44.084130+0000","last_clean":"2024-03-11T21:43:44.084130+0000","last_became_active":"2024-03-11T21:34:41.146631+0000","last_became_peered":"2024-03-11T21:34:41.146631+0000","last_unstale":"2024-03-11T21:43:44.084130+0000","last_undegraded":"2024-03-11T21:43:44.084130+0000","last_fullsized":"2024-03-11T21:43:44.084130+0000","mapping_epoch":226,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":227,"parent":"0.0","parent_split_bits":0,"last_scrub":"260'34","last_scrub_stamp":"2024-03-11T21:41:06.881414+0000","last_deep_scrub":"109'22","last_deep_scrub_stamp":"2024-03-11T21:32:07.573197+0000","last_clean_scrub_stamp":"2024-03-11T21:41:06.881414+0000","objects_scrubbed":7,"log_size":34,"log_dups_size":0,"ondisk_log_size":34,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:43:45.574587+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":44,"num_read_kb":35,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.9","version":"229'11","reported_seq":517,"reported_epoch":414,"state":"active+clean","last_fresh":"2024-03-11T21:43:37.036799+0000","last_change":"2024-03-11T21:41:28.240859+0000","last_active":"2024-03-11T21:43:37.036799+0000","last_peered":"2024-03-11T21:43:37.036799+0000","last_clean":"2024-03-11T21:43:37.036799+0000","last_became_active":"2024-03-11T21:34:41.156892+0000","last_became_peered":"2024-03-11T21:34:41.156892+0000","last_unstale":"2024-03-11T21:43:37.036799+0000","last_undegraded":"2024-03-11T21:43:37.036799+0000","last_fullsized":"2024-03-11T21:43:37.036799+0000","mapping_epoch":226,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":227,"parent":"0.0","parent_split_bits":0,"last_scrub":"229'11","last_scrub_stamp":"2024-03-11T21:41:28.240763+0000","last_deep_scrub":"94'9","last_deep_scrub_stamp":"2024-03-11T21:26:53.422552+0000","last_clean_scrub_stamp":"2024-03-11T21:41:28.240763+0000","objects_scrubbed":1,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":10,"num_pg":26,"stat_sum":{"num_bytes":2471755800,"num_objects":37717,"num_object_clones":0,"num_object_copies":113151,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":37717,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":108266,"num_write_kb":3428481,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8337960960,"data_stored":8337948744,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":160160,"ondisk_log_size":160160,"up":78,"acting":78,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":1531,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":17,"num_read_kb":17,"num_write":10,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":1308,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":4593,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":6,"ondisk_log_size":6,"up":96,"acting":96,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":198,"num_read_kb":294,"num_write":179,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":2951840,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":2375680,"data_stored":2361472,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":128,"ondisk_log_size":128,"up":3,"acting":3,"num_store_stats":4},{"poolid":3,"num_pg":3,"stat_sum":{"num_bytes":408,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1076,"num_read_kb":1071,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1224,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":8,"ondisk_log_size":8,"up":9,"acting":9,"num_store_stats":7},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":3702,"num_objects":178,"num_object_clones":0,"num_object_copies":534,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":178,"num_whiteouts":0,"num_read":1120,"num_read_kb":877,"num_write":703,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":186,"num_bytes_recovered":2200,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":417792,"data_stored":11106,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":774,"ondisk_log_size":774,"up":96,"acting":96,"num_store_stats":8},{"poolid":5,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":78,"ondisk_log_size":78,"up":96,"acting":96,"num_store_stats":8},{"poolid":6,"num_pg":32,"stat_sum":{"num_bytes":383,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":370,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1149,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":96,"acting":96,"num_store_stats":8}],"osd_stats":[{"osd":7,"up_from":60,"seq":257698037980,"num_pgs":65,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1504724,"kb_used_data":855756,"kb_used_omap":1,"kb_used_meta":648958,"kb_avail":92240428,"statfs":{"total":95995035648,"available":94454198272,"internally_reserved":0,"allocated":876294144,"data_stored":875945849,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":664533450},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":50,"seq":214748365026,"num_pgs":54,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1865368,"kb_used_data":1266384,"kb_used_omap":1,"kb_used_meta":598974,"kb_avail":91879784,"statfs":{"total":95995035648,"available":94084898816,"internally_reserved":0,"allocated":1296777216,"data_stored":1296425578,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":613349830},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834575100,"num_pgs":61,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1612268,"kb_used_data":881380,"kb_used_omap":1,"kb_used_meta":730878,"kb_avail":92132884,"statfs":{"total":95995035648,"available":94344073216,"internally_reserved":0,"allocated":902533120,"data_stored":902160432,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":748419525},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738625,"num_pgs":66,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2238564,"kb_used_data":1535324,"kb_used_omap":1,"kb_used_meta":703102,"kb_avail":91506588,"statfs":{"total":95995035648,"available":93702746112,"internally_reserved":0,"allocated":1572171776,"data_stored":1571807884,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":719976901},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411574,"num_pgs":52,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1742092,"kb_used_data":1137540,"kb_used_omap":1,"kb_used_meta":604542,"kb_avail":92003060,"statfs":{"total":95995035648,"available":94211133440,"internally_reserved":0,"allocated":1164840960,"data_stored":1164500810,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":619051461},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":277,"seq":1189705941079,"num_pgs":63,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1182896,"kb_used_data":766056,"kb_used_omap":4,"kb_used_meta":416827,"kb_avail":92562256,"statfs":{"total":95995035648,"available":94783750144,"internally_reserved":0,"allocated":784441344,"data_stored":784065004,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4442,"internal_metadata":426831526},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":32,"seq":137438953708,"num_pgs":66,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1793904,"kb_used_data":1162984,"kb_used_omap":1,"kb_used_meta":630910,"kb_avail":91951248,"statfs":{"total":95995035648,"available":94158077952,"internally_reserved":0,"allocated":1190895616,"data_stored":1190525555,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":646052293},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":41,"seq":176093659367,"num_pgs":58,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1363800,"kb_used_data":692432,"kb_used_omap":1,"kb_used_meta":671166,"kb_avail":92381352,"statfs":{"total":95995035648,"available":94598504448,"internally_reserved":0,"allocated":709050368,"data_stored":708704626,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":687274437},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1049,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":46,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":436,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":957,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":574,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":436,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1003,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":92,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":1430,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":77824,"data_stored":2008,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":1018,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":1898,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":1474,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":990,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":1034,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":1254,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":370,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":383,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":370,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":1550979072,"data_stored":1550975000,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":892141568,"data_stored":892141568,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":1133445120,"data_stored":1133445120,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":766771200,"data_stored":766771200,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":1153568768,"data_stored":1153564696,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":693043200,"data_stored":693043200,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":1290145792,"data_stored":1290141720,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":857866240,"data_stored":857866240,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-11T21:43:52.262 INFO:tasks.cephadm.ceph_manager.ceph:decrease pool size by 10 2024-03-11T21:43:52.262 DEBUG:teuthology.orchestra.run.smithi092:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid 00db0016-dfed-11ee-95c8-87774f69a715 -- ceph osd pool set unique_pool_3 pg_num 16 2024-03-11T21:43:52.354 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:43:52.393 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:43:52.394 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:43:52.394 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:43:52.495 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:43:52.526 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:43:52.527 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:43:52.527 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:43:52.628 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:43:52.671 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:43:52.672 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:43:52.672 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:43:52.773 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:43:52.813 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:43:52.814 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:43:52.814 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:43:52.915 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:43:52.948 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:43:52.949 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:43:52.949 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:43:53.050 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:43:53.087 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:43:53.087 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:43:53.088 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:43:53.189 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:43:53.228 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:43:53.230 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:43:53.230 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:43:53.331 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:43:53.332 INFO:journalctl@ceph.iscsi.iscsi.a.smithi097.stdout:Mar 11 21:43:53 smithi097 ceph-00db0016-dfed-11ee-95c8-87774f69a715-iscsi-iscsi-a[60943]: debug there is no tcmu-runner data available 2024-03-11T21:43:53.332 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:43:53 smithi097 ceph-mon[33317]: pgmap v856: 158 pgs: 1 active+clean+scrubbing+deep, 157 active+clean; 2.0 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 1.10k op/s 2024-03-11T21:43:53.332 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:43:53 smithi097 ceph-mon[33317]: purged_snaps scrub starts 2024-03-11T21:43:53.333 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:43:53 smithi097 ceph-mon[33317]: purged_snaps scrub ok 2024-03-11T21:43:53.333 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:43:53 smithi097 ceph-mon[33317]: osdmap e416: 8 total, 8 up, 8 in 2024-03-11T21:43:53.333 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:43:53 smithi097 ceph-mon[33317]: from='mgr.14607 v1:172.21.15.92:0/1702717597' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T21:43:53.367 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:43:53 smithi092 ceph-mon[29572]: pgmap v856: 158 pgs: 1 active+clean+scrubbing+deep, 157 active+clean; 2.0 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 1.10k op/s 2024-03-11T21:43:53.367 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:43:53 smithi092 ceph-mon[29572]: purged_snaps scrub starts 2024-03-11T21:43:53.368 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:43:53 smithi092 ceph-mon[29572]: purged_snaps scrub ok 2024-03-11T21:43:53.368 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:43:53 smithi092 ceph-mon[29572]: osdmap e416: 8 total, 8 up, 8 in 2024-03-11T21:43:53.368 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:43:53 smithi092 ceph-mon[29572]: from='mgr.14607 v1:172.21.15.92:0/1702717597' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T21:43:53.368 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:43:53 smithi092 ceph-mon[37563]: pgmap v856: 158 pgs: 1 active+clean+scrubbing+deep, 157 active+clean; 2.0 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 1023 B/s rd, 0 B/s wr, 1.10k op/s 2024-03-11T21:43:53.368 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:43:53 smithi092 ceph-mon[37563]: purged_snaps scrub starts 2024-03-11T21:43:53.368 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:43:53 smithi092 ceph-mon[37563]: purged_snaps scrub ok 2024-03-11T21:43:53.368 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:43:53 smithi092 ceph-mon[37563]: osdmap e416: 8 total, 8 up, 8 in 2024-03-11T21:43:53.368 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:43:53 smithi092 ceph-mon[37563]: from='mgr.14607 v1:172.21.15.92:0/1702717597' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T21:43:53.389 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:43:53.390 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:43:53.390 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:43:53.491 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:43:53.532 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:43:53.533 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:43:53.533 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:43:53.634 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:43:53.664 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:43:53.665 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:43:53.665 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:43:53.766 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:43:53.797 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:43:53.798 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:43:53.798 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:43:53.899 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:43:53.940 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:43:53.941 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:43:53.941 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:43:53.942 INFO:teuthology.orchestra.run.smithi092.stdout:osd.0: {} 2024-03-11T21:43:53.989 INFO:teuthology.orchestra.run.smithi092.stdout:osd.1: {} 2024-03-11T21:43:54.033 INFO:teuthology.orchestra.run.smithi092.stdout:osd.2: {} 2024-03-11T21:43:54.042 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:43:54.073 INFO:teuthology.orchestra.run.smithi092.stdout:osd.3: {} 2024-03-11T21:43:54.074 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:43:54.075 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:43:54.076 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:43:54.111 INFO:teuthology.orchestra.run.smithi092.stdout:osd.4: {} 2024-03-11T21:43:54.130 INFO:teuthology.orchestra.run.smithi092.stderr:nodeep-scrub is set 2024-03-11T21:43:54.152 INFO:teuthology.orchestra.run.smithi092.stdout:osd.5: {} 2024-03-11T21:43:54.177 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:43:54.195 INFO:teuthology.orchestra.run.smithi092.stdout:osd.6: {} 2024-03-11T21:43:54.227 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:43:54.227 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:43:54.228 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:43:54.233 INFO:teuthology.orchestra.run.smithi092.stdout:osd.7: {} 2024-03-11T21:43:54.246 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:43:54 smithi092 ceph-mon[29572]: from='client.24469 v1:172.21.15.97:0/404208310' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:43:54.246 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:43:54 smithi092 ceph-mon[29572]: from='client.? v1:172.21.15.92:0/826751241' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-11T21:43:54.246 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:43:54 smithi092 ceph-mon[29572]: from='client.? v1:172.21.15.92:0/355452700' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-11T21:43:54.246 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:43:54 smithi092 ceph-mon[37563]: from='client.24469 v1:172.21.15.97:0/404208310' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:43:54.246 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:43:54 smithi092 ceph-mon[37563]: from='client.? v1:172.21.15.92:0/826751241' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-11T21:43:54.246 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:43:54 smithi092 ceph-mon[37563]: from='client.? v1:172.21.15.92:0/355452700' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-11T21:43:54.246 INFO:teuthology.orchestra.run.smithi092.stderr:osd.0: osd_enable_op_tracker = '' 2024-03-11T21:43:54.247 INFO:teuthology.orchestra.run.smithi092.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-11T21:43:54.247 INFO:teuthology.orchestra.run.smithi092.stderr:osd.2: osd_enable_op_tracker = '' 2024-03-11T21:43:54.247 INFO:teuthology.orchestra.run.smithi092.stderr:osd.3: osd_enable_op_tracker = '' 2024-03-11T21:43:54.247 INFO:teuthology.orchestra.run.smithi092.stderr:osd.4: osd_enable_op_tracker = '' 2024-03-11T21:43:54.247 INFO:teuthology.orchestra.run.smithi092.stderr:osd.5: osd_enable_op_tracker = '' 2024-03-11T21:43:54.247 INFO:teuthology.orchestra.run.smithi092.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-11T21:43:54.247 INFO:teuthology.orchestra.run.smithi092.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-11T21:43:54.329 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:43:54.369 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:43:54.370 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:43:54.370 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:43:54.471 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:43:54.515 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:43:54.515 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:43:54.516 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:43:54.582 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:43:54 smithi097 ceph-mon[33317]: from='client.24469 v1:172.21.15.97:0/404208310' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:43:54.582 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:43:54 smithi097 ceph-mon[33317]: from='client.? v1:172.21.15.92:0/826751241' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-11T21:43:54.583 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:43:54 smithi097 ceph-mon[33317]: from='client.? v1:172.21.15.92:0/355452700' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-11T21:43:54.616 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:43:54.658 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:43:54.659 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:43:54.659 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:43:54.760 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:43:54.796 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:43:54.796 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:43:54.796 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:43:54.897 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:43:54.931 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:43:54.932 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:43:54.932 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:43:55.033 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:43:55.079 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:43:55.080 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:43:55.080 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:43:55.181 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:43:55.215 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:43:55.216 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:43:55.216 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:43:55.317 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:43:55.352 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:43:55.352 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:43:55.353 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:43:55.453 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:43:55.487 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:43:55.488 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:43:55.488 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:43:55.582 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:43:55 smithi097 ceph-mon[33317]: pgmap v858: 158 pgs: 158 active+clean; 1.8 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 548 B/s rd, 0 B/s wr, 1.13k op/s 2024-03-11T21:43:55.583 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:43:55 smithi097 ceph-mon[33317]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-11T21:43:55.583 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:43:55 smithi097 ceph-mon[33317]: from='client.? v1:172.21.15.92:0/826751241' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-03-11T21:43:55.583 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:43:55 smithi097 ceph-mon[33317]: osdmap e417: 8 total, 8 up, 8 in 2024-03-11T21:43:55.589 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:43:55.617 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:43:55 smithi092 ceph-mon[29572]: pgmap v858: 158 pgs: 158 active+clean; 1.8 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 548 B/s rd, 0 B/s wr, 1.13k op/s 2024-03-11T21:43:55.617 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:43:55 smithi092 ceph-mon[29572]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-11T21:43:55.617 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:43:55 smithi092 ceph-mon[29572]: from='client.? v1:172.21.15.92:0/826751241' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-03-11T21:43:55.617 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:43:55 smithi092 ceph-mon[29572]: osdmap e417: 8 total, 8 up, 8 in 2024-03-11T21:43:55.617 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:43:55 smithi092 ceph-mon[37563]: pgmap v858: 158 pgs: 158 active+clean; 1.8 GiB data, 11 GiB used, 704 GiB / 715 GiB avail; 548 B/s rd, 0 B/s wr, 1.13k op/s 2024-03-11T21:43:55.617 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:43:55 smithi092 ceph-mon[37563]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-11T21:43:55.617 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:43:55 smithi092 ceph-mon[37563]: from='client.? v1:172.21.15.92:0/826751241' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-03-11T21:43:55.618 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:43:55 smithi092 ceph-mon[37563]: osdmap e417: 8 total, 8 up, 8 in 2024-03-11T21:43:55.620 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:43:55.621 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:43:55.621 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:43:55.722 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:43:55.768 DEBUG:teuthology.orchestra.run.smithi092:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid 00db0016-dfed-11ee-95c8-87774f69a715 -- ceph daemon osd.0 dump_ops_in_flight 2024-03-11T21:43:55.771 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:43:55.772 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:43:55.772 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:43:55.804 INFO:teuthology.orchestra.run.smithi092.stderr:Inferring config /var/lib/ceph/00db0016-dfed-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T21:43:55.873 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:43:55.949 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:43:55.950 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:43:55.950 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:43:56.051 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:43:56.100 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:43:56.100 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:43:56.101 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:43:56.202 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:43:56.233 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:43:56.234 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:43:56.234 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:43:56.336 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:43:56.382 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:43:56.383 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:43:56.383 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:43:56.484 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:43:56.518 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:43:56.519 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:43:56.519 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:43:56.621 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:43:56.652 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:43:56.653 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:43:56.653 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:43:56.754 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:43:56.789 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:43:56.790 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:43:56.790 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:43:56.891 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:43:56.929 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:43:56.930 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:43:56.930 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:43:57.031 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:43:57.066 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:43:57.066 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:43:57.067 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:43:57.167 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:43:57.199 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:43:57.199 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:43:57.200 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:43:57.277 DEBUG:teuthology.orchestra.run.smithi092:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid 00db0016-dfed-11ee-95c8-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-03-11T21:43:57.300 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:43:57.361 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:43:57.362 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:43:57.362 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:43:57.460 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:43:57 smithi092 ceph-mon[29572]: pgmap v860: 158 pgs: 158 active+clean; 1.7 GiB data, 11 GiB used, 705 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.23k op/s 2024-03-11T21:43:57.460 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:43:57 smithi092 ceph-mon[37563]: pgmap v860: 158 pgs: 158 active+clean; 1.7 GiB data, 11 GiB used, 705 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.23k op/s 2024-03-11T21:43:57.463 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:43:57.506 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:43:57.506 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:43:57.506 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:43:57.582 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:43:57 smithi097 ceph-mon[33317]: pgmap v860: 158 pgs: 158 active+clean; 1.7 GiB data, 11 GiB used, 705 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.23k op/s 2024-03-11T21:43:57.607 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:43:57.655 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:43:57.655 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:43:57.656 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:43:57.757 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:43:57.790 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:43:57.791 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:43:57.792 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:43:57.893 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:43:57.929 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:43:57.929 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:43:57.930 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:43:57.961 INFO:teuthology.orchestra.run.smithi092.stderr:Inferring config /var/lib/ceph/00db0016-dfed-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T21:43:58.030 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:43:58.076 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:43:58.077 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:43:58.077 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:43:58.131 INFO:journalctl@ceph.mgr.y.smithi092.stdout:Mar 11 21:43:57 smithi092 ceph-00db0016-dfed-11ee-95c8-87774f69a715-mgr-y[29801]: ::ffff:172.21.15.97 - - [11/Mar/2024:21:43:57] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-11T21:43:58.179 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:43:58.214 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:43:58.214 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:43:58.215 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:43:58.302 DEBUG:teuthology.orchestra.run.smithi092:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid 00db0016-dfed-11ee-95c8-87774f69a715 -- ceph osd unset noscrub 2024-03-11T21:43:58.316 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:43:58.399 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:43:58.400 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:43:58.400 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:43:58.501 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:43:58.512 INFO:teuthology.orchestra.run.smithi092.stderr:Inferring config /var/lib/ceph/00db0016-dfed-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T21:43:58.546 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:43:58.547 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:43:58.547 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:43:58.648 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:43:58.698 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:43:58.698 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:43:58.699 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:43:58.799 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:43:58.845 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:43:58.845 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:43:58.845 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:43:58.946 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:43:58.993 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:43:58.994 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:43:58.995 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:43:59.095 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:43:59.131 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:43:59.132 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:43:59.133 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:43:59.233 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:43:59.268 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:43:59.268 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:43:59.268 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:43:59.367 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:43:59 smithi092 ceph-mon[29572]: pgmap v861: 158 pgs: 158 active+clean; 1.6 GiB data, 10 GiB used, 705 GiB / 715 GiB avail; 639 B/s rd, 0 B/s wr, 963 op/s 2024-03-11T21:43:59.367 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:43:59 smithi092 ceph-mon[29572]: from='client.? v1:172.21.15.92:0/1633808540' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "unique_pool_3", "var": "pg_num", "val": "16"}]: dispatch 2024-03-11T21:43:59.367 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:43:59 smithi092 ceph-mon[29572]: from='client.25720 ' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "unique_pool_3", "var": "pg_num", "val": "16"}]: dispatch 2024-03-11T21:43:59.367 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:43:59 smithi092 ceph-mon[37563]: pgmap v861: 158 pgs: 158 active+clean; 1.6 GiB data, 10 GiB used, 705 GiB / 715 GiB avail; 639 B/s rd, 0 B/s wr, 963 op/s 2024-03-11T21:43:59.367 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:43:59 smithi092 ceph-mon[37563]: from='client.? v1:172.21.15.92:0/1633808540' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "unique_pool_3", "var": "pg_num", "val": "16"}]: dispatch 2024-03-11T21:43:59.367 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:43:59 smithi092 ceph-mon[37563]: from='client.25720 ' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "unique_pool_3", "var": "pg_num", "val": "16"}]: dispatch 2024-03-11T21:43:59.369 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:43:59.402 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:43:59.402 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:43:59.403 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:43:59.479 INFO:teuthology.orchestra.run.smithi092.stderr:set pool 10 pg_num to 16 2024-03-11T21:43:59.504 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:43:59.541 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:43:59.542 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:43:59.543 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:43:59.583 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:43:59 smithi097 ceph-mon[33317]: pgmap v861: 158 pgs: 158 active+clean; 1.6 GiB data, 10 GiB used, 705 GiB / 715 GiB avail; 639 B/s rd, 0 B/s wr, 963 op/s 2024-03-11T21:43:59.583 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:43:59 smithi097 ceph-mon[33317]: from='client.? v1:172.21.15.92:0/1633808540' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "unique_pool_3", "var": "pg_num", "val": "16"}]: dispatch 2024-03-11T21:43:59.583 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:43:59 smithi097 ceph-mon[33317]: from='client.25720 ' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "unique_pool_3", "var": "pg_num", "val": "16"}]: dispatch 2024-03-11T21:43:59.643 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:43:59.682 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:43:59.683 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:43:59.683 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:43:59.784 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:43:59.819 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:43:59.820 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:43:59.821 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:43:59.921 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:43:59.959 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:43:59.959 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:43:59.960 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:44:00.061 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:44:00.096 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:44:00.096 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:44:00.096 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:44:00.197 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:44:00.235 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:44:00.236 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:44:00.236 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:44:00.337 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:44:00.377 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:44:00.378 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:44:00.378 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:44:00.479 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:44:00.518 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:44:00.519 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:44:00.520 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:44:00.617 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:44:00 smithi092 ceph-mon[29572]: pgmap v862: 158 pgs: 158 active+clean; 1.4 GiB data, 9.8 GiB used, 705 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.42k op/s 2024-03-11T21:44:00.617 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:44:00 smithi092 ceph-mon[29572]: from='client.25720 ' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_3", "var": "pg_num", "val": "16"}]': finished 2024-03-11T21:44:00.617 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:44:00 smithi092 ceph-mon[29572]: osdmap e418: 8 total, 8 up, 8 in 2024-03-11T21:44:00.617 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:44:00 smithi092 ceph-mon[29572]: from='mgr.14607 v1:172.21.15.92:0/1702717597' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T21:44:00.617 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:44:00 smithi092 ceph-mon[37563]: pgmap v862: 158 pgs: 158 active+clean; 1.4 GiB data, 9.8 GiB used, 705 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.42k op/s 2024-03-11T21:44:00.617 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:44:00 smithi092 ceph-mon[37563]: from='client.25720 ' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_3", "var": "pg_num", "val": "16"}]': finished 2024-03-11T21:44:00.617 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:44:00 smithi092 ceph-mon[37563]: osdmap e418: 8 total, 8 up, 8 in 2024-03-11T21:44:00.617 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:44:00 smithi092 ceph-mon[37563]: from='mgr.14607 v1:172.21.15.92:0/1702717597' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T21:44:00.621 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:44:00.664 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:44:00.664 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:44:00.665 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:44:00.765 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:44:00.810 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:44:00.811 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:44:00.811 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:44:00.832 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:44:00 smithi097 ceph-mon[33317]: pgmap v862: 158 pgs: 158 active+clean; 1.4 GiB data, 9.8 GiB used, 705 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.42k op/s 2024-03-11T21:44:00.832 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:44:00 smithi097 ceph-mon[33317]: from='client.25720 ' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_3", "var": "pg_num", "val": "16"}]': finished 2024-03-11T21:44:00.832 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:44:00 smithi097 ceph-mon[33317]: osdmap e418: 8 total, 8 up, 8 in 2024-03-11T21:44:00.832 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:44:00 smithi097 ceph-mon[33317]: from='mgr.14607 v1:172.21.15.92:0/1702717597' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T21:44:00.912 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:44:00.944 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:44:00.945 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:44:00.946 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:44:01.046 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:44:01.095 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:44:01.096 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:44:01.096 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:44:01.197 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:44:01.238 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:44:01.238 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:44:01.238 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:44:01.339 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:44:01.378 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:44:01.378 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:44:01.378 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:44:01.479 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:44:01.525 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:44:01.526 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:44:01.526 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:44:01.627 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:44:01.671 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:44:01.672 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:44:01.672 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:44:01.773 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:44:01.812 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:44:01.813 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:44:01.813 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:44:01.914 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:44:01.956 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:44:01.957 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:44:01.957 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:44:02.058 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:44:02.119 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:44:02.120 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:44:02.120 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:44:02.221 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:44:02.255 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:44:02 smithi092 ceph-mon[29572]: from='mgr.14607 ' entity='mgr.y' 2024-03-11T21:44:02.255 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:44:02 smithi092 ceph-mon[29572]: from='mgr.14607 ' entity='mgr.y' 2024-03-11T21:44:02.255 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:44:02 smithi092 ceph-mon[29572]: from='mgr.14607 ' entity='mgr.y' 2024-03-11T21:44:02.255 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:44:02 smithi092 ceph-mon[29572]: from='mgr.14607 ' entity='mgr.y' 2024-03-11T21:44:02.256 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:44:02 smithi092 ceph-mon[29572]: from='mgr.14607 v1:172.21.15.92:0/1702717597' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "unique_pool_3", "var": "pgp_num_actual", "val": "16"}]: dispatch 2024-03-11T21:44:02.256 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:44:02 smithi092 ceph-mon[29572]: from='mgr.14607 ' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "unique_pool_3", "var": "pgp_num_actual", "val": "16"}]: dispatch 2024-03-11T21:44:02.256 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:44:02 smithi092 ceph-mon[37563]: from='mgr.14607 ' entity='mgr.y' 2024-03-11T21:44:02.256 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:44:02 smithi092 ceph-mon[37563]: from='mgr.14607 ' entity='mgr.y' 2024-03-11T21:44:02.256 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:44:02 smithi092 ceph-mon[37563]: from='mgr.14607 ' entity='mgr.y' 2024-03-11T21:44:02.256 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:44:02 smithi092 ceph-mon[37563]: from='mgr.14607 ' entity='mgr.y' 2024-03-11T21:44:02.256 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:44:02 smithi092 ceph-mon[37563]: from='mgr.14607 v1:172.21.15.92:0/1702717597' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "unique_pool_3", "var": "pgp_num_actual", "val": "16"}]: dispatch 2024-03-11T21:44:02.256 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:44:02 smithi092 ceph-mon[37563]: from='mgr.14607 ' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "unique_pool_3", "var": "pgp_num_actual", "val": "16"}]: dispatch 2024-03-11T21:44:02.259 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:44:02.260 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:44:02.260 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:44:02.270 INFO:teuthology.orchestra.run.smithi092.stdout:osd.0: {} 2024-03-11T21:44:02.326 INFO:teuthology.orchestra.run.smithi092.stdout:osd.1: {} 2024-03-11T21:44:02.332 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:44:02 smithi097 ceph-mon[33317]: from='mgr.14607 ' entity='mgr.y' 2024-03-11T21:44:02.332 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:44:02 smithi097 ceph-mon[33317]: from='mgr.14607 ' entity='mgr.y' 2024-03-11T21:44:02.332 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:44:02 smithi097 ceph-mon[33317]: from='mgr.14607 ' entity='mgr.y' 2024-03-11T21:44:02.333 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:44:02 smithi097 ceph-mon[33317]: from='mgr.14607 ' entity='mgr.y' 2024-03-11T21:44:02.333 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:44:02 smithi097 ceph-mon[33317]: from='mgr.14607 v1:172.21.15.92:0/1702717597' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "unique_pool_3", "var": "pgp_num_actual", "val": "16"}]: dispatch 2024-03-11T21:44:02.333 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:44:02 smithi097 ceph-mon[33317]: from='mgr.14607 ' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "unique_pool_3", "var": "pgp_num_actual", "val": "16"}]: dispatch 2024-03-11T21:44:02.346 INFO:teuthology.orchestra.run.smithi092.stdout:osd.2: {} 2024-03-11T21:44:02.361 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:44:02.368 INFO:teuthology.orchestra.run.smithi092.stdout:osd.3: {} 2024-03-11T21:44:02.385 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:44:02.385 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:44:02.386 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:44:02.389 INFO:teuthology.orchestra.run.smithi092.stdout:osd.4: {} 2024-03-11T21:44:02.408 INFO:teuthology.orchestra.run.smithi092.stdout:osd.5: {} 2024-03-11T21:44:02.429 INFO:teuthology.orchestra.run.smithi092.stdout:osd.6: {} 2024-03-11T21:44:02.448 INFO:teuthology.orchestra.run.smithi092.stdout:osd.7: {} 2024-03-11T21:44:02.449 INFO:teuthology.orchestra.run.smithi092.stderr:osd.0: osd_enable_op_tracker = '' 2024-03-11T21:44:02.449 INFO:teuthology.orchestra.run.smithi092.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-11T21:44:02.449 INFO:teuthology.orchestra.run.smithi092.stderr:osd.2: osd_enable_op_tracker = '' 2024-03-11T21:44:02.449 INFO:teuthology.orchestra.run.smithi092.stderr:osd.3: osd_enable_op_tracker = '' 2024-03-11T21:44:02.450 INFO:teuthology.orchestra.run.smithi092.stderr:osd.4: osd_enable_op_tracker = '' 2024-03-11T21:44:02.450 INFO:teuthology.orchestra.run.smithi092.stderr:osd.5: osd_enable_op_tracker = '' 2024-03-11T21:44:02.450 INFO:teuthology.orchestra.run.smithi092.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-11T21:44:02.450 INFO:teuthology.orchestra.run.smithi092.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-11T21:44:02.486 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:44:02.511 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:44:02.512 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:44:02.512 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:44:02.613 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:44:02.637 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:44:02.637 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:44:02.638 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:44:02.739 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:44:02.763 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:44:02.764 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:44:02.764 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:44:02.865 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:44:02.889 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:44:02.890 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:44:02.890 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:44:02.991 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:44:03.015 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:44:03.015 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:44:03.016 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:44:03.116 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:44:03.120 INFO:teuthology.orchestra.run.smithi092.stdout:ERROR: (22) Invalid argument 2024-03-11T21:44:03.120 INFO:teuthology.orchestra.run.smithi092.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-11T21:44:03.142 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:44:03.142 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:44:03.143 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:44:03.243 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:44:03.271 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:44:03.271 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:44:03.271 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:44:03.332 INFO:journalctl@ceph.iscsi.iscsi.a.smithi097.stdout:Mar 11 21:44:03 smithi097 ceph-00db0016-dfed-11ee-95c8-87774f69a715-iscsi-iscsi-a[60943]: debug there is no tcmu-runner data available 2024-03-11T21:44:03.333 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:44:03 smithi097 ceph-mon[33317]: pgmap v864: 158 pgs: 158 active+clean; 1.3 GiB data, 9.5 GiB used, 706 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 939 op/s 2024-03-11T21:44:03.333 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:44:03 smithi097 ceph-mon[33317]: from='client.? v1:172.21.15.92:0/521494760' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-11T21:44:03.333 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:44:03 smithi097 ceph-mon[33317]: from='mgr.14607 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_3", "var": "pgp_num_actual", "val": "16"}]': finished 2024-03-11T21:44:03.333 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:44:03 smithi097 ceph-mon[33317]: osdmap e419: 8 total, 8 up, 8 in 2024-03-11T21:44:03.333 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:44:03 smithi097 ceph-mon[33317]: from='mgr.14607 v1:172.21.15.92:0/1702717597' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T21:44:03.333 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:44:03 smithi097 ceph-mon[33317]: from='mgr.14607 v1:172.21.15.92:0/1702717597' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T21:44:03.333 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:44:03 smithi097 ceph-mon[33317]: from='mgr.14607 ' entity='mgr.y' 2024-03-11T21:44:03.366 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:44:03 smithi092 ceph-mon[29572]: pgmap v864: 158 pgs: 158 active+clean; 1.3 GiB data, 9.5 GiB used, 706 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 939 op/s 2024-03-11T21:44:03.367 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:44:03 smithi092 ceph-mon[29572]: from='client.? v1:172.21.15.92:0/521494760' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-11T21:44:03.367 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:44:03 smithi092 ceph-mon[29572]: from='mgr.14607 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_3", "var": "pgp_num_actual", "val": "16"}]': finished 2024-03-11T21:44:03.367 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:44:03 smithi092 ceph-mon[29572]: osdmap e419: 8 total, 8 up, 8 in 2024-03-11T21:44:03.367 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:44:03 smithi092 ceph-mon[29572]: from='mgr.14607 v1:172.21.15.92:0/1702717597' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T21:44:03.367 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:44:03 smithi092 ceph-mon[29572]: from='mgr.14607 v1:172.21.15.92:0/1702717597' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T21:44:03.367 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:44:03 smithi092 ceph-mon[29572]: from='mgr.14607 ' entity='mgr.y' 2024-03-11T21:44:03.367 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:44:03 smithi092 ceph-mon[37563]: pgmap v864: 158 pgs: 158 active+clean; 1.3 GiB data, 9.5 GiB used, 706 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 939 op/s 2024-03-11T21:44:03.367 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:44:03 smithi092 ceph-mon[37563]: from='client.? v1:172.21.15.92:0/521494760' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-11T21:44:03.367 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:44:03 smithi092 ceph-mon[37563]: from='mgr.14607 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_3", "var": "pgp_num_actual", "val": "16"}]': finished 2024-03-11T21:44:03.367 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:44:03 smithi092 ceph-mon[37563]: osdmap e419: 8 total, 8 up, 8 in 2024-03-11T21:44:03.368 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:44:03 smithi092 ceph-mon[37563]: from='mgr.14607 v1:172.21.15.92:0/1702717597' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T21:44:03.368 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:44:03 smithi092 ceph-mon[37563]: from='mgr.14607 v1:172.21.15.92:0/1702717597' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T21:44:03.368 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:44:03 smithi092 ceph-mon[37563]: from='mgr.14607 ' entity='mgr.y' 2024-03-11T21:44:03.372 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:44:03.397 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:44:03.398 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:44:03.398 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:44:03.499 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:44:03.523 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:44:03.524 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:44:03.524 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:44:03.625 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:44:03.649 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:44:03.649 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:44:03.650 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:44:03.751 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:44:03.774 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:44:03.775 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:44:03.775 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:44:03.872 INFO:teuthology.orchestra.run.smithi092.stderr:Inferring config /var/lib/ceph/00db0016-dfed-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T21:44:03.876 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:44:03.899 DEBUG:teuthology.orchestra.run.smithi092:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid 00db0016-dfed-11ee-95c8-87774f69a715 -- ceph daemon osd.0 dump_blocked_ops 2024-03-11T21:44:03.903 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:44:03.904 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:44:03.904 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:44:04.005 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:44:04.030 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:44:04.030 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:44:04.031 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:44:04.132 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:44:04.156 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:44:04.157 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:44:04.157 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:44:04.258 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:44:04.288 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:44:04.289 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:44:04.289 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:44:04.390 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:44:04.458 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:44:04.459 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:44:04.459 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:44:04.560 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:44:04.582 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:44:04 smithi097 ceph-mon[33317]: from='client.24469 v1:172.21.15.97:0/404208310' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:44:04.582 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:44:04 smithi097 ceph-mon[33317]: osdmap e420: 8 total, 8 up, 8 in 2024-03-11T21:44:04.599 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:44:04.600 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:44:04.600 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:44:04.616 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:44:04 smithi092 ceph-mon[29572]: from='client.24469 v1:172.21.15.97:0/404208310' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:44:04.616 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:44:04 smithi092 ceph-mon[29572]: osdmap e420: 8 total, 8 up, 8 in 2024-03-11T21:44:04.617 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:44:04 smithi092 ceph-mon[37563]: from='client.24469 v1:172.21.15.97:0/404208310' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:44:04.617 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:44:04 smithi092 ceph-mon[37563]: osdmap e420: 8 total, 8 up, 8 in 2024-03-11T21:44:04.701 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:44:04.742 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:44:04.742 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:44:04.743 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:44:04.844 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:44:04.879 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:44:04.880 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:44:04.880 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:44:04.981 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:44:05.018 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:44:05.019 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:44:05.020 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:44:05.120 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:44:05.172 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:44:05.172 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:44:05.173 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:44:05.273 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:44:05.322 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:44:05.323 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:44:05.323 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:44:05.424 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:44:05.458 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:44:05.459 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:44:05.459 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:44:05.530 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:44:05 smithi092 ceph-mon[29572]: pgmap v867: 158 pgs: 4 unknown, 154 active+clean; 1.2 GiB data, 8.8 GiB used, 706 GiB / 715 GiB avail; 853 B/s rd, 24 MiB/s wr, 1.68k op/s 2024-03-11T21:44:05.530 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:44:05 smithi092 ceph-mon[29572]: 10.11 starting backfill to osd.4 from (0'0,0'0] MAX to 418'10171 2024-03-11T21:44:05.531 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:44:05 smithi092 ceph-mon[29572]: 10.18 starting backfill to osd.0 from (0'0,373'4381] MIN to 418'8604 2024-03-11T21:44:05.531 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:44:05 smithi092 ceph-mon[29572]: osdmap e421: 8 total, 8 up, 8 in 2024-03-11T21:44:05.531 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:44:05 smithi092 ceph-mon[29572]: 10.10 starting backfill to osd.1 from (0'0,0'0] MAX to 418'10124 2024-03-11T21:44:05.531 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:44:05 smithi092 ceph-mon[29572]: 10.14 starting backfill to osd.0 from (0'0,396'5729] MIN to 418'8413 2024-03-11T21:44:05.531 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:44:05 smithi092 ceph-mon[29572]: 10.17 starting backfill to osd.5 from (0'0,0'0] MAX to 418'8565 2024-03-11T21:44:05.531 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:44:05 smithi092 ceph-mon[29572]: 10.12 starting backfill to osd.0 from (364'1214,407'7296] MIN to 418'8725 2024-03-11T21:44:05.531 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:44:05 smithi092 ceph-mon[29572]: 10.16 starting backfill to osd.0 from (0'0,392'5557] MIN to 418'8404 2024-03-11T21:44:05.531 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:44:05 smithi092 ceph-mon[29572]: 10.13 starting backfill to osd.3 from (0'0,0'0] MAX to 418'8459 2024-03-11T21:44:05.531 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:44:05 smithi092 ceph-mon[29572]: 10.15 starting backfill to osd.0 from (0'0,373'4493] MIN to 418'8681 2024-03-11T21:44:05.531 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:44:05 smithi092 ceph-mon[29572]: 10.19 starting backfill to osd.2 from (0'0,410'7395] MIN to 418'8650 2024-03-11T21:44:05.531 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:44:05 smithi092 ceph-mon[29572]: 10.16 starting backfill to osd.4 from (0'0,0'0] MAX to 418'8404 2024-03-11T21:44:05.531 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:44:05 smithi092 ceph-mon[29572]: 10.13 starting backfill to osd.5 from (0'0,0'0] MAX to 418'8459 2024-03-11T21:44:05.531 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:44:05 smithi092 ceph-mon[29572]: 10.12 starting backfill to osd.1 from (0'0,373'4449] MIN to 418'8725 2024-03-11T21:44:05.531 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:44:05 smithi092 ceph-mon[29572]: 10.15 starting backfill to osd.2 from (0'0,373'4493] MIN to 418'8681 2024-03-11T21:44:05.532 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:44:05 smithi092 ceph-mon[29572]: 10.12 starting backfill to osd.4 from (0'0,373'4449] MIN to 418'8725 2024-03-11T21:44:05.532 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:44:05 smithi092 ceph-mon[29572]: 10.14 starting backfill to osd.3 from (0'0,0'0] MAX to 418'8413 2024-03-11T21:44:05.532 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:44:05 smithi092 ceph-mon[29572]: 10.19 starting backfill to osd.5 from (0'0,0'0] MAX to 418'8650 2024-03-11T21:44:05.532 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:44:05 smithi092 ceph-mon[29572]: 10.18 starting backfill to osd.7 from (0'0,373'4381] MIN to 418'8604 2024-03-11T21:44:05.532 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:44:05 smithi092 ceph-mon[37563]: pgmap v867: 158 pgs: 4 unknown, 154 active+clean; 1.2 GiB data, 8.8 GiB used, 706 GiB / 715 GiB avail; 853 B/s rd, 24 MiB/s wr, 1.68k op/s 2024-03-11T21:44:05.532 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:44:05 smithi092 ceph-mon[37563]: 10.11 starting backfill to osd.4 from (0'0,0'0] MAX to 418'10171 2024-03-11T21:44:05.532 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:44:05 smithi092 ceph-mon[37563]: 10.18 starting backfill to osd.0 from (0'0,373'4381] MIN to 418'8604 2024-03-11T21:44:05.532 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:44:05 smithi092 ceph-mon[37563]: osdmap e421: 8 total, 8 up, 8 in 2024-03-11T21:44:05.532 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:44:05 smithi092 ceph-mon[37563]: 10.10 starting backfill to osd.1 from (0'0,0'0] MAX to 418'10124 2024-03-11T21:44:05.532 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:44:05 smithi092 ceph-mon[37563]: 10.14 starting backfill to osd.0 from (0'0,396'5729] MIN to 418'8413 2024-03-11T21:44:05.533 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:44:05 smithi092 ceph-mon[37563]: 10.17 starting backfill to osd.5 from (0'0,0'0] MAX to 418'8565 2024-03-11T21:44:05.533 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:44:05 smithi092 ceph-mon[37563]: 10.12 starting backfill to osd.0 from (364'1214,407'7296] MIN to 418'8725 2024-03-11T21:44:05.533 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:44:05 smithi092 ceph-mon[37563]: 10.16 starting backfill to osd.0 from (0'0,392'5557] MIN to 418'8404 2024-03-11T21:44:05.533 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:44:05 smithi092 ceph-mon[37563]: 10.13 starting backfill to osd.3 from (0'0,0'0] MAX to 418'8459 2024-03-11T21:44:05.533 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:44:05 smithi092 ceph-mon[37563]: 10.15 starting backfill to osd.0 from (0'0,373'4493] MIN to 418'8681 2024-03-11T21:44:05.533 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:44:05 smithi092 ceph-mon[37563]: 10.19 starting backfill to osd.2 from (0'0,410'7395] MIN to 418'8650 2024-03-11T21:44:05.533 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:44:05 smithi092 ceph-mon[37563]: 10.16 starting backfill to osd.4 from (0'0,0'0] MAX to 418'8404 2024-03-11T21:44:05.533 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:44:05 smithi092 ceph-mon[37563]: 10.13 starting backfill to osd.5 from (0'0,0'0] MAX to 418'8459 2024-03-11T21:44:05.533 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:44:05 smithi092 ceph-mon[37563]: 10.12 starting backfill to osd.1 from (0'0,373'4449] MIN to 418'8725 2024-03-11T21:44:05.533 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:44:05 smithi092 ceph-mon[37563]: 10.15 starting backfill to osd.2 from (0'0,373'4493] MIN to 418'8681 2024-03-11T21:44:05.533 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:44:05 smithi092 ceph-mon[37563]: 10.12 starting backfill to osd.4 from (0'0,373'4449] MIN to 418'8725 2024-03-11T21:44:05.533 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:44:05 smithi092 ceph-mon[37563]: 10.14 starting backfill to osd.3 from (0'0,0'0] MAX to 418'8413 2024-03-11T21:44:05.533 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:44:05 smithi092 ceph-mon[37563]: 10.19 starting backfill to osd.5 from (0'0,0'0] MAX to 418'8650 2024-03-11T21:44:05.533 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:44:05 smithi092 ceph-mon[37563]: 10.18 starting backfill to osd.7 from (0'0,373'4381] MIN to 418'8604 2024-03-11T21:44:05.560 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:44:05.583 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:44:05 smithi097 ceph-mon[33317]: pgmap v867: 158 pgs: 4 unknown, 154 active+clean; 1.2 GiB data, 8.8 GiB used, 706 GiB / 715 GiB avail; 853 B/s rd, 24 MiB/s wr, 1.68k op/s 2024-03-11T21:44:05.583 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:44:05 smithi097 ceph-mon[33317]: 10.11 starting backfill to osd.4 from (0'0,0'0] MAX to 418'10171 2024-03-11T21:44:05.583 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:44:05 smithi097 ceph-mon[33317]: 10.18 starting backfill to osd.0 from (0'0,373'4381] MIN to 418'8604 2024-03-11T21:44:05.583 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:44:05 smithi097 ceph-mon[33317]: osdmap e421: 8 total, 8 up, 8 in 2024-03-11T21:44:05.583 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:44:05 smithi097 ceph-mon[33317]: 10.10 starting backfill to osd.1 from (0'0,0'0] MAX to 418'10124 2024-03-11T21:44:05.583 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:44:05 smithi097 ceph-mon[33317]: 10.14 starting backfill to osd.0 from (0'0,396'5729] MIN to 418'8413 2024-03-11T21:44:05.583 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:44:05 smithi097 ceph-mon[33317]: 10.17 starting backfill to osd.5 from (0'0,0'0] MAX to 418'8565 2024-03-11T21:44:05.583 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:44:05 smithi097 ceph-mon[33317]: 10.12 starting backfill to osd.0 from (364'1214,407'7296] MIN to 418'8725 2024-03-11T21:44:05.583 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:44:05 smithi097 ceph-mon[33317]: 10.16 starting backfill to osd.0 from (0'0,392'5557] MIN to 418'8404 2024-03-11T21:44:05.583 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:44:05 smithi097 ceph-mon[33317]: 10.13 starting backfill to osd.3 from (0'0,0'0] MAX to 418'8459 2024-03-11T21:44:05.584 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:44:05 smithi097 ceph-mon[33317]: 10.15 starting backfill to osd.0 from (0'0,373'4493] MIN to 418'8681 2024-03-11T21:44:05.584 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:44:05 smithi097 ceph-mon[33317]: 10.19 starting backfill to osd.2 from (0'0,410'7395] MIN to 418'8650 2024-03-11T21:44:05.584 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:44:05 smithi097 ceph-mon[33317]: 10.16 starting backfill to osd.4 from (0'0,0'0] MAX to 418'8404 2024-03-11T21:44:05.584 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:44:05 smithi097 ceph-mon[33317]: 10.13 starting backfill to osd.5 from (0'0,0'0] MAX to 418'8459 2024-03-11T21:44:05.584 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:44:05 smithi097 ceph-mon[33317]: 10.12 starting backfill to osd.1 from (0'0,373'4449] MIN to 418'8725 2024-03-11T21:44:05.584 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:44:05 smithi097 ceph-mon[33317]: 10.15 starting backfill to osd.2 from (0'0,373'4493] MIN to 418'8681 2024-03-11T21:44:05.584 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:44:05 smithi097 ceph-mon[33317]: 10.12 starting backfill to osd.4 from (0'0,373'4449] MIN to 418'8725 2024-03-11T21:44:05.584 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:44:05 smithi097 ceph-mon[33317]: 10.14 starting backfill to osd.3 from (0'0,0'0] MAX to 418'8413 2024-03-11T21:44:05.584 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:44:05 smithi097 ceph-mon[33317]: 10.19 starting backfill to osd.5 from (0'0,0'0] MAX to 418'8650 2024-03-11T21:44:05.584 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:44:05 smithi097 ceph-mon[33317]: 10.18 starting backfill to osd.7 from (0'0,373'4381] MIN to 418'8604 2024-03-11T21:44:05.607 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:44:05.608 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:44:05.608 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:44:05.709 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:44:05.750 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:44:05.751 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:44:05.751 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:44:05.852 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:44:05.897 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:44:05.897 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:44:05.898 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:44:05.998 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:44:06.032 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:44:06.033 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:44:06.033 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:44:06.092 INFO:teuthology.orchestra.run.smithi092.stderr:Inferring config /var/lib/ceph/00db0016-dfed-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T21:44:06.134 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:44:06.192 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:44:06.193 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:44:06.194 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:44:06.294 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:44:06.326 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:44:06.327 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:44:06.327 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:44:06.428 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:44:06.476 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:44:06.477 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:44:06.477 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:44:06.578 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:44:06.582 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:44:06 smithi097 ceph-mon[33317]: 10.19 starting backfill to osd.7 from (0'0,373'4350] MIN to 418'8650 2024-03-11T21:44:06.582 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:44:06 smithi097 ceph-mon[33317]: purged_snaps scrub starts 2024-03-11T21:44:06.582 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:44:06 smithi097 ceph-mon[33317]: purged_snaps scrub ok 2024-03-11T21:44:06.582 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:44:06 smithi097 ceph-mon[33317]: purged_snaps scrub starts 2024-03-11T21:44:06.583 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:44:06 smithi097 ceph-mon[33317]: purged_snaps scrub ok 2024-03-11T21:44:06.612 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:44:06.613 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:44:06.613 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:44:06.616 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:44:06 smithi092 ceph-mon[29572]: 10.19 starting backfill to osd.7 from (0'0,373'4350] MIN to 418'8650 2024-03-11T21:44:06.616 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:44:06 smithi092 ceph-mon[29572]: purged_snaps scrub starts 2024-03-11T21:44:06.617 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:44:06 smithi092 ceph-mon[29572]: purged_snaps scrub ok 2024-03-11T21:44:06.617 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:44:06 smithi092 ceph-mon[29572]: purged_snaps scrub starts 2024-03-11T21:44:06.617 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:44:06 smithi092 ceph-mon[29572]: purged_snaps scrub ok 2024-03-11T21:44:06.617 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:44:06 smithi092 ceph-mon[37563]: 10.19 starting backfill to osd.7 from (0'0,373'4350] MIN to 418'8650 2024-03-11T21:44:06.617 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:44:06 smithi092 ceph-mon[37563]: purged_snaps scrub starts 2024-03-11T21:44:06.617 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:44:06 smithi092 ceph-mon[37563]: purged_snaps scrub ok 2024-03-11T21:44:06.617 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:44:06 smithi092 ceph-mon[37563]: purged_snaps scrub starts 2024-03-11T21:44:06.617 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:44:06 smithi092 ceph-mon[37563]: purged_snaps scrub ok 2024-03-11T21:44:06.714 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:44:06.750 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:44:06.751 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:44:06.751 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:44:06.852 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:44:06.884 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:44:06.885 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:44:06.885 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:44:06.986 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:44:07.045 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:44:07.045 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:44:07.046 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:44:07.147 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:44:07.188 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:44:07.189 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:44:07.189 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:44:07.290 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:44:07.337 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:44:07.338 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:44:07.339 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:44:07.439 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:44:07.475 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:44:07.475 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:44:07.476 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:44:07.576 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:44:07.582 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:44:07 smithi097 ceph-mon[33317]: pgmap v869: 158 pgs: 1 active+remapped+backfilling, 4 remapped+peering, 3 unknown, 150 active+clean; 1.0 GiB data, 8.4 GiB used, 707 GiB / 715 GiB avail; 863 B/s rd, 0 B/s wr, 940 op/s; 501/50547 objects misplaced (0.991%); 43 KiB/s, 0 objects/s recovering 2024-03-11T21:44:07.582 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:44:07 smithi097 ceph-mon[33317]: osdmap e422: 8 total, 8 up, 8 in 2024-03-11T21:44:07.586 DEBUG:teuthology.orchestra.run.smithi092:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid 00db0016-dfed-11ee-95c8-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-03-11T21:44:07.606 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:44:07.607 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:44:07.607 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:44:07.616 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:44:07 smithi092 ceph-mon[29572]: pgmap v869: 158 pgs: 1 active+remapped+backfilling, 4 remapped+peering, 3 unknown, 150 active+clean; 1.0 GiB data, 8.4 GiB used, 707 GiB / 715 GiB avail; 863 B/s rd, 0 B/s wr, 940 op/s; 501/50547 objects misplaced (0.991%); 43 KiB/s, 0 objects/s recovering 2024-03-11T21:44:07.617 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:44:07 smithi092 ceph-mon[29572]: osdmap e422: 8 total, 8 up, 8 in 2024-03-11T21:44:07.617 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:44:07 smithi092 ceph-mon[37563]: pgmap v869: 158 pgs: 1 active+remapped+backfilling, 4 remapped+peering, 3 unknown, 150 active+clean; 1.0 GiB data, 8.4 GiB used, 707 GiB / 715 GiB avail; 863 B/s rd, 0 B/s wr, 940 op/s; 501/50547 objects misplaced (0.991%); 43 KiB/s, 0 objects/s recovering 2024-03-11T21:44:07.617 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:44:07 smithi092 ceph-mon[37563]: osdmap e422: 8 total, 8 up, 8 in 2024-03-11T21:44:07.708 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:44:07.745 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:44:07.745 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:44:07.746 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:44:07.847 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:44:07.875 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:44:07.876 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:44:07.877 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:44:07.937 INFO:journalctl@ceph.mgr.y.smithi092.stdout:Mar 11 21:44:07 smithi092 ceph-00db0016-dfed-11ee-95c8-87774f69a715-mgr-y[29801]: ::ffff:172.21.15.97 - - [11/Mar/2024:21:44:07] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-11T21:44:07.977 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:44:08.006 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:44:08.007 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:44:08.007 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:44:08.108 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:44:08.134 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:44:08.135 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:44:08.136 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:44:08.236 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:44:08.260 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:44:08.261 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:44:08.261 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:44:08.362 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:44:08.400 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:44:08.400 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:44:08.400 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:44:08.501 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:44:08.530 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:44:08.530 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:44:08.531 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:44:08.582 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:44:08 smithi097 ceph-mon[33317]: osdmap e423: 8 total, 8 up, 8 in 2024-03-11T21:44:08.582 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:44:08 smithi097 ceph-mon[33317]: pgmap v872: 158 pgs: 2 active+remapped+backfilling, 4 remapped+peering, 3 unknown, 149 active+clean; 1002 MiB data, 8.4 GiB used, 707 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 64 op/s; 903/48681 objects misplaced (1.855%); 3.8 MiB/s, 61 objects/s recovering 2024-03-11T21:44:08.583 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:44:08 smithi097 ceph-mon[33317]: from='mgr.14607 v1:172.21.15.92:0/1702717597' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T21:44:08.616 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:44:08 smithi092 ceph-mon[29572]: osdmap e423: 8 total, 8 up, 8 in 2024-03-11T21:44:08.617 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:44:08 smithi092 ceph-mon[29572]: pgmap v872: 158 pgs: 2 active+remapped+backfilling, 4 remapped+peering, 3 unknown, 149 active+clean; 1002 MiB data, 8.4 GiB used, 707 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 64 op/s; 903/48681 objects misplaced (1.855%); 3.8 MiB/s, 61 objects/s recovering 2024-03-11T21:44:08.617 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:44:08 smithi092 ceph-mon[29572]: from='mgr.14607 v1:172.21.15.92:0/1702717597' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T21:44:08.617 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:44:08 smithi092 ceph-mon[37563]: osdmap e423: 8 total, 8 up, 8 in 2024-03-11T21:44:08.617 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:44:08 smithi092 ceph-mon[37563]: pgmap v872: 158 pgs: 2 active+remapped+backfilling, 4 remapped+peering, 3 unknown, 149 active+clean; 1002 MiB data, 8.4 GiB used, 707 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 64 op/s; 903/48681 objects misplaced (1.855%); 3.8 MiB/s, 61 objects/s recovering 2024-03-11T21:44:08.617 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:44:08 smithi092 ceph-mon[37563]: from='mgr.14607 v1:172.21.15.92:0/1702717597' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T21:44:08.632 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:44:08.660 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:44:08.661 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:44:08.661 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:44:08.762 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:44:08.789 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:44:08.790 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:44:08.790 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:44:08.891 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:44:08.915 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:44:08.915 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:44:08.915 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:44:09.016 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:44:09.041 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:44:09.041 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:44:09.041 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:44:09.142 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:44:09.167 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:44:09.168 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:44:09.168 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:44:09.269 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:44:09.294 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:44:09.294 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:44:09.295 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:44:09.395 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:44:09.432 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:44:09.432 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:44:09.432 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:44:09.533 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:44:09.566 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:44:09.567 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:44:09.567 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:44:09.582 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:44:09 smithi097 ceph-mon[33317]: osdmap e424: 8 total, 8 up, 8 in 2024-03-11T21:44:09.616 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:44:09 smithi092 ceph-mon[29572]: osdmap e424: 8 total, 8 up, 8 in 2024-03-11T21:44:09.617 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:44:09 smithi092 ceph-mon[37563]: osdmap e424: 8 total, 8 up, 8 in 2024-03-11T21:44:09.668 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:44:09.700 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:44:09.701 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:44:09.701 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:44:09.803 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:44:09.827 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:44:09.828 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:44:09.828 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:44:09.929 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:44:09.952 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:44:09.953 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:44:09.953 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:44:10.054 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:44:10.078 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:44:10.079 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:44:10.080 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:44:10.180 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:44:10.205 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:44:10.205 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:44:10.205 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:44:10.306 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:44:10.315 INFO:teuthology.orchestra.run.smithi092.stderr:noscrub is unset 2024-03-11T21:44:10.339 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:44:10.340 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:44:10.341 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:44:10.441 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:44:10.473 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:44:10.474 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:44:10.474 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:44:10.575 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:44:10.582 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:44:10 smithi097 ceph-mon[33317]: osdmap e425: 8 total, 8 up, 8 in 2024-03-11T21:44:10.583 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:44:10 smithi097 ceph-mon[33317]: pgmap v875: 158 pgs: 1 peering, 1 active+clean+remapped, 3 active+remapped+backfill_toofull, 5 active+remapped+backfilling, 148 active+clean; 866 MiB data, 8.2 GiB used, 707 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 699 op/s; 4779/42159 objects misplaced (11.336%); 32 MiB/s, 507 objects/s recovering 2024-03-11T21:44:10.583 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:44:10 smithi097 ceph-mon[33317]: from='client.? v1:172.21.15.92:0/591105363' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-11T21:44:10.583 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:44:10 smithi097 ceph-mon[33317]: from='client.35723 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-11T21:44:10.583 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:44:10 smithi097 ceph-mon[33317]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-11T21:44:10.583 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:44:10 smithi097 ceph-mon[33317]: from='client.35723 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-03-11T21:44:10.583 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:44:10 smithi097 ceph-mon[33317]: osdmap e426: 8 total, 8 up, 8 in 2024-03-11T21:44:10.601 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:44:10.602 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:44:10.602 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:44:10.617 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:44:10 smithi092 ceph-mon[29572]: osdmap e425: 8 total, 8 up, 8 in 2024-03-11T21:44:10.617 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:44:10 smithi092 ceph-mon[29572]: pgmap v875: 158 pgs: 1 peering, 1 active+clean+remapped, 3 active+remapped+backfill_toofull, 5 active+remapped+backfilling, 148 active+clean; 866 MiB data, 8.2 GiB used, 707 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 699 op/s; 4779/42159 objects misplaced (11.336%); 32 MiB/s, 507 objects/s recovering 2024-03-11T21:44:10.617 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:44:10 smithi092 ceph-mon[29572]: from='client.? v1:172.21.15.92:0/591105363' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-11T21:44:10.617 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:44:10 smithi092 ceph-mon[29572]: from='client.35723 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-11T21:44:10.617 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:44:10 smithi092 ceph-mon[29572]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-11T21:44:10.617 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:44:10 smithi092 ceph-mon[29572]: from='client.35723 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-03-11T21:44:10.617 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:44:10 smithi092 ceph-mon[29572]: osdmap e426: 8 total, 8 up, 8 in 2024-03-11T21:44:10.618 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:44:10 smithi092 ceph-mon[37563]: osdmap e425: 8 total, 8 up, 8 in 2024-03-11T21:44:10.618 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:44:10 smithi092 ceph-mon[37563]: pgmap v875: 158 pgs: 1 peering, 1 active+clean+remapped, 3 active+remapped+backfill_toofull, 5 active+remapped+backfilling, 148 active+clean; 866 MiB data, 8.2 GiB used, 707 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 699 op/s; 4779/42159 objects misplaced (11.336%); 32 MiB/s, 507 objects/s recovering 2024-03-11T21:44:10.618 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:44:10 smithi092 ceph-mon[37563]: from='client.? v1:172.21.15.92:0/591105363' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-11T21:44:10.618 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:44:10 smithi092 ceph-mon[37563]: from='client.35723 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-11T21:44:10.618 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:44:10 smithi092 ceph-mon[37563]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-11T21:44:10.618 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:44:10 smithi092 ceph-mon[37563]: from='client.35723 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-03-11T21:44:10.618 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:44:10 smithi092 ceph-mon[37563]: osdmap e426: 8 total, 8 up, 8 in 2024-03-11T21:44:10.703 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:44:10.738 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:44:10.738 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:44:10.739 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:44:10.840 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:44:10.865 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:44:10.866 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:44:10.866 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:44:10.967 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:44:10.991 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:44:10.992 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:44:10.992 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:44:11.067 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5, 6, 7] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 4, 5, 6, 7, 3] 2024-03-11T21:44:11.067 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-03-11T21:44:11.068 INFO:tasks.thrashosds.thrasher:Removing osd 3, in_osds are: [0, 1, 2, 3, 4, 5, 6, 7] 2024-03-11T21:44:11.068 DEBUG:teuthology.orchestra.run.smithi092:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid 00db0016-dfed-11ee-95c8-87774f69a715 -- ceph osd out 3 2024-03-11T21:44:11.094 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:44:11.117 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:44:11.118 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:44:11.118 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:44:11.219 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:44:11.243 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:44:11.243 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:44:11.244 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:44:11.345 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:44:11.367 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:44:11 smithi092 ceph-mon[37563]: Health check failed: Low space hindering backfill (add storage if this doesn't resolve itself): 3 pgs backfill_toofull (PG_BACKFILL_FULL) 2024-03-11T21:44:11.367 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:44:11 smithi092 ceph-mon[37563]: 6.7 scrub starts 2024-03-11T21:44:11.367 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:44:11 smithi092 ceph-mon[37563]: 6.7 scrub ok 2024-03-11T21:44:11.367 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:44:11 smithi092 ceph-mon[37563]: osdmap e427: 8 total, 8 up, 8 in 2024-03-11T21:44:11.367 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:44:11 smithi092 ceph-mon[29572]: Health check failed: Low space hindering backfill (add storage if this doesn't resolve itself): 3 pgs backfill_toofull (PG_BACKFILL_FULL) 2024-03-11T21:44:11.367 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:44:11 smithi092 ceph-mon[29572]: 6.7 scrub starts 2024-03-11T21:44:11.367 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:44:11 smithi092 ceph-mon[29572]: 6.7 scrub ok 2024-03-11T21:44:11.368 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:44:11 smithi092 ceph-mon[29572]: osdmap e427: 8 total, 8 up, 8 in 2024-03-11T21:44:11.397 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:44:11.397 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:44:11.397 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:44:11.498 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:44:11.532 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:44:11.533 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:44:11.533 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:44:11.582 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:44:11 smithi097 ceph-mon[33317]: Health check failed: Low space hindering backfill (add storage if this doesn't resolve itself): 3 pgs backfill_toofull (PG_BACKFILL_FULL) 2024-03-11T21:44:11.583 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:44:11 smithi097 ceph-mon[33317]: 6.7 scrub starts 2024-03-11T21:44:11.583 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:44:11 smithi097 ceph-mon[33317]: 6.7 scrub ok 2024-03-11T21:44:11.583 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:44:11 smithi097 ceph-mon[33317]: osdmap e427: 8 total, 8 up, 8 in 2024-03-11T21:44:11.634 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:44:11.673 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:44:11.674 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:44:11.674 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:44:11.775 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:44:11.816 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:44:11.817 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:44:11.817 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:44:11.918 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:44:11.948 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:44:11.949 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:44:11.949 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:44:12.050 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:44:12.088 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:44:12.088 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:44:12.088 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:44:12.189 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:44:12.220 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:44:12.221 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:44:12.221 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:44:12.322 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:44:12.381 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:44:12.382 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:44:12.382 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:44:12.483 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:44:12.517 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:44:12.518 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:44:12.518 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:44:12.617 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:44:12 smithi092 ceph-mon[29572]: 6.14 scrub starts 2024-03-11T21:44:12.617 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:44:12 smithi092 ceph-mon[29572]: 6.14 scrub ok 2024-03-11T21:44:12.617 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:44:12 smithi092 ceph-mon[29572]: 4.9 scrub starts 2024-03-11T21:44:12.617 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:44:12 smithi092 ceph-mon[29572]: 4.9 scrub ok 2024-03-11T21:44:12.617 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:44:12 smithi092 ceph-mon[29572]: pgmap v878: 158 pgs: 1 peering, 1 active+clean+remapped, 3 active+remapped+backfill_toofull, 5 active+remapped+backfilling, 148 active+clean; 855 MiB data, 8.1 GiB used, 707 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 590 op/s; 4779/41613 objects misplaced (11.484%); 28 MiB/s, 445 objects/s recovering 2024-03-11T21:44:12.618 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:44:12 smithi092 ceph-mon[29572]: 10.a scrub starts 2024-03-11T21:44:12.618 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:44:12 smithi092 ceph-mon[37563]: 6.14 scrub starts 2024-03-11T21:44:12.618 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:44:12 smithi092 ceph-mon[37563]: 6.14 scrub ok 2024-03-11T21:44:12.618 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:44:12 smithi092 ceph-mon[37563]: 4.9 scrub starts 2024-03-11T21:44:12.618 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:44:12 smithi092 ceph-mon[37563]: 4.9 scrub ok 2024-03-11T21:44:12.618 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:44:12 smithi092 ceph-mon[37563]: pgmap v878: 158 pgs: 1 peering, 1 active+clean+remapped, 3 active+remapped+backfill_toofull, 5 active+remapped+backfilling, 148 active+clean; 855 MiB data, 8.1 GiB used, 707 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 590 op/s; 4779/41613 objects misplaced (11.484%); 28 MiB/s, 445 objects/s recovering 2024-03-11T21:44:12.618 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:44:12 smithi092 ceph-mon[37563]: 10.a scrub starts 2024-03-11T21:44:12.619 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:44:12.659 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:44:12.659 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:44:12.660 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:44:12.761 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:44:12.794 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:44:12.795 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:44:12.795 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:44:12.832 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:44:12 smithi097 ceph-mon[33317]: 6.14 scrub starts 2024-03-11T21:44:12.833 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:44:12 smithi097 ceph-mon[33317]: 6.14 scrub ok 2024-03-11T21:44:12.833 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:44:12 smithi097 ceph-mon[33317]: 4.9 scrub starts 2024-03-11T21:44:12.833 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:44:12 smithi097 ceph-mon[33317]: 4.9 scrub ok 2024-03-11T21:44:12.833 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:44:12 smithi097 ceph-mon[33317]: pgmap v878: 158 pgs: 1 peering, 1 active+clean+remapped, 3 active+remapped+backfill_toofull, 5 active+remapped+backfilling, 148 active+clean; 855 MiB data, 8.1 GiB used, 707 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 590 op/s; 4779/41613 objects misplaced (11.484%); 28 MiB/s, 445 objects/s recovering 2024-03-11T21:44:12.833 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:44:12 smithi097 ceph-mon[33317]: 10.a scrub starts 2024-03-11T21:44:12.897 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:44:12.930 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:44:12.931 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:44:12.931 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:44:13.032 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:44:13.074 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:44:13.074 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:44:13.077 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:44:13.178 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:44:13.211 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:44:13.212 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:44:13.212 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:44:13.313 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:44:13.332 INFO:journalctl@ceph.iscsi.iscsi.a.smithi097.stdout:Mar 11 21:44:13 smithi097 ceph-00db0016-dfed-11ee-95c8-87774f69a715-iscsi-iscsi-a[60943]: debug there is no tcmu-runner data available 2024-03-11T21:44:13.357 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:44:13.357 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:44:13.358 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:44:13.459 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:44:13.493 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:44:13.494 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:44:13.495 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:44:13.595 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:44:13.616 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:44:13 smithi092 ceph-mon[29572]: 10.a scrub ok 2024-03-11T21:44:13.616 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:44:13 smithi092 ceph-mon[29572]: from='client.24469 v1:172.21.15.97:0/404208310' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:44:13.617 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:44:13 smithi092 ceph-mon[37563]: 10.a scrub ok 2024-03-11T21:44:13.617 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:44:13 smithi092 ceph-mon[37563]: from='client.24469 v1:172.21.15.97:0/404208310' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:44:13.632 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:44:13.632 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:44:13.632 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:44:13.733 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:44:13.769 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:44:13.772 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:44:13.772 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:44:13.832 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:44:13 smithi097 ceph-mon[33317]: 10.a scrub ok 2024-03-11T21:44:13.833 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:44:13 smithi097 ceph-mon[33317]: from='client.24469 v1:172.21.15.97:0/404208310' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:44:13.873 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:44:13.913 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:44:13.914 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:44:13.914 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:44:14.015 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:44:14.059 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:44:14.060 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:44:14.061 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:44:14.162 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:44:14.193 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:44:14.194 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:44:14.194 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:44:14.295 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:44:14.337 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:44:14.337 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:44:14.337 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:44:14.397 DEBUG:teuthology.orchestra.run.smithi092:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid 00db0016-dfed-11ee-95c8-87774f69a715 -- ceph osd unset nodeep-scrub 2024-03-11T21:44:14.438 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:44:14.481 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:44:14.481 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:44:14.482 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:44:14.582 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:44:14.591 INFO:teuthology.orchestra.run.smithi092.stderr:Inferring config /var/lib/ceph/00db0016-dfed-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T21:44:14.615 INFO:teuthology.orchestra.run.smithi092.stderr:Inferring config /var/lib/ceph/00db0016-dfed-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T21:44:14.617 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:44:14.619 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:44:14.619 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:44:14.620 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:44:14 smithi092 ceph-mon[29572]: 4.13 scrub starts 2024-03-11T21:44:14.620 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:44:14 smithi092 ceph-mon[29572]: 5.0 scrub starts 2024-03-11T21:44:14.620 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:44:14 smithi092 ceph-mon[29572]: 5.0 scrub ok 2024-03-11T21:44:14.620 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:44:14 smithi092 ceph-mon[29572]: 4.13 scrub ok 2024-03-11T21:44:14.621 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:44:14 smithi092 ceph-mon[29572]: 5.6 scrub starts 2024-03-11T21:44:14.621 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:44:14 smithi092 ceph-mon[29572]: 5.6 scrub ok 2024-03-11T21:44:14.621 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:44:14 smithi092 ceph-mon[29572]: 2.1d scrub starts 2024-03-11T21:44:14.621 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:44:14 smithi092 ceph-mon[29572]: pgmap v879: 158 pgs: 3 active+remapped+backfill_toofull, 155 active+clean; 705 MiB data, 7.7 GiB used, 708 GiB / 715 GiB avail; 1004 B/s rd, 0 B/s wr, 596 op/s; 2721/34404 objects misplaced (7.909%); 20 MiB/s, 320 objects/s recovering 2024-03-11T21:44:14.621 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:44:14 smithi092 ceph-mon[29572]: from='mgr.14607 v1:172.21.15.92:0/1702717597' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "unique_pool_3", "var": "pg_num_actual", "val": "25"}]: dispatch 2024-03-11T21:44:14.621 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:44:14 smithi092 ceph-mon[29572]: 2.1d scrub ok 2024-03-11T21:44:14.622 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:44:14 smithi092 ceph-mon[29572]: from='mgr.14607 ' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "unique_pool_3", "var": "pg_num_actual", "val": "25"}]: dispatch 2024-03-11T21:44:14.622 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:44:14 smithi092 ceph-mon[37563]: 4.13 scrub starts 2024-03-11T21:44:14.622 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:44:14 smithi092 ceph-mon[37563]: 5.0 scrub starts 2024-03-11T21:44:14.622 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:44:14 smithi092 ceph-mon[37563]: 5.0 scrub ok 2024-03-11T21:44:14.622 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:44:14 smithi092 ceph-mon[37563]: 4.13 scrub ok 2024-03-11T21:44:14.623 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:44:14 smithi092 ceph-mon[37563]: 5.6 scrub starts 2024-03-11T21:44:14.623 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:44:14 smithi092 ceph-mon[37563]: 5.6 scrub ok 2024-03-11T21:44:14.623 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:44:14 smithi092 ceph-mon[37563]: 2.1d scrub starts 2024-03-11T21:44:14.623 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:44:14 smithi092 ceph-mon[37563]: pgmap v879: 158 pgs: 3 active+remapped+backfill_toofull, 155 active+clean; 705 MiB data, 7.7 GiB used, 708 GiB / 715 GiB avail; 1004 B/s rd, 0 B/s wr, 596 op/s; 2721/34404 objects misplaced (7.909%); 20 MiB/s, 320 objects/s recovering 2024-03-11T21:44:14.623 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:44:14 smithi092 ceph-mon[37563]: from='mgr.14607 v1:172.21.15.92:0/1702717597' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "unique_pool_3", "var": "pg_num_actual", "val": "25"}]: dispatch 2024-03-11T21:44:14.624 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:44:14 smithi092 ceph-mon[37563]: 2.1d scrub ok 2024-03-11T21:44:14.624 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:44:14 smithi092 ceph-mon[37563]: from='mgr.14607 ' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "unique_pool_3", "var": "pg_num_actual", "val": "25"}]: dispatch 2024-03-11T21:44:14.720 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:44:14.744 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:44:14.745 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:44:14.745 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:44:14.833 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:44:14 smithi097 ceph-mon[33317]: 4.13 scrub starts 2024-03-11T21:44:14.833 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:44:14 smithi097 ceph-mon[33317]: 5.0 scrub starts 2024-03-11T21:44:14.833 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:44:14 smithi097 ceph-mon[33317]: 5.0 scrub ok 2024-03-11T21:44:14.833 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:44:14 smithi097 ceph-mon[33317]: 4.13 scrub ok 2024-03-11T21:44:14.833 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:44:14 smithi097 ceph-mon[33317]: 5.6 scrub starts 2024-03-11T21:44:14.833 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:44:14 smithi097 ceph-mon[33317]: 5.6 scrub ok 2024-03-11T21:44:14.833 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:44:14 smithi097 ceph-mon[33317]: 2.1d scrub starts 2024-03-11T21:44:14.833 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:44:14 smithi097 ceph-mon[33317]: pgmap v879: 158 pgs: 3 active+remapped+backfill_toofull, 155 active+clean; 705 MiB data, 7.7 GiB used, 708 GiB / 715 GiB avail; 1004 B/s rd, 0 B/s wr, 596 op/s; 2721/34404 objects misplaced (7.909%); 20 MiB/s, 320 objects/s recovering 2024-03-11T21:44:14.833 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:44:14 smithi097 ceph-mon[33317]: from='mgr.14607 v1:172.21.15.92:0/1702717597' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "unique_pool_3", "var": "pg_num_actual", "val": "25"}]: dispatch 2024-03-11T21:44:14.833 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:44:14 smithi097 ceph-mon[33317]: 2.1d scrub ok 2024-03-11T21:44:14.833 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:44:14 smithi097 ceph-mon[33317]: from='mgr.14607 ' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "unique_pool_3", "var": "pg_num_actual", "val": "25"}]: dispatch 2024-03-11T21:44:14.846 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:44:14.870 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:44:14.870 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:44:14.870 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:44:14.971 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:44:14.996 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:44:14.996 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:44:14.997 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:44:15.097 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:44:15.122 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:44:15.122 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:44:15.123 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:44:15.224 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:44:15.247 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:44:15.248 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:44:15.248 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:44:15.349 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:44:15.355 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:44:15 smithi092 ceph-mon[29572]: 6.2 scrub starts 2024-03-11T21:44:15.379 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:44:15.379 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:44:15.379 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:44:15.480 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:44:15.505 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:44:15.505 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:44:15.505 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:44:15.605 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:44:15 smithi092 ceph-mon[37563]: 6.2 scrub starts 2024-03-11T21:44:15.606 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:44:15 smithi092 ceph-mon[37563]: 6.2 scrub ok 2024-03-11T21:44:15.606 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:44:15 smithi092 ceph-mon[37563]: from='mgr.14607 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_3", "var": "pg_num_actual", "val": "25"}]': finished 2024-03-11T21:44:15.606 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:44:15 smithi092 ceph-mon[37563]: osdmap e428: 8 total, 8 up, 8 in 2024-03-11T21:44:15.606 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:44:15 smithi092 ceph-mon[29572]: 6.2 scrub ok 2024-03-11T21:44:15.606 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:44:15 smithi092 ceph-mon[29572]: from='mgr.14607 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_3", "var": "pg_num_actual", "val": "25"}]': finished 2024-03-11T21:44:15.606 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:44:15 smithi092 ceph-mon[29572]: osdmap e428: 8 total, 8 up, 8 in 2024-03-11T21:44:15.607 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:44:15.631 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:44:15.631 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:44:15.632 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:44:15.732 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:44:15.757 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:44:15.757 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:44:15.758 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:44:15.833 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:44:15 smithi097 ceph-mon[33317]: 6.2 scrub starts 2024-03-11T21:44:15.833 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:44:15 smithi097 ceph-mon[33317]: 6.2 scrub ok 2024-03-11T21:44:15.833 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:44:15 smithi097 ceph-mon[33317]: from='mgr.14607 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_3", "var": "pg_num_actual", "val": "25"}]': finished 2024-03-11T21:44:15.833 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:44:15 smithi097 ceph-mon[33317]: osdmap e428: 8 total, 8 up, 8 in 2024-03-11T21:44:15.859 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:44:15.885 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:44:15.885 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:44:15.885 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:44:15.986 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:44:16.012 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:44:16.013 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:44:16.013 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:44:16.063 INFO:teuthology.orchestra.run.smithi092.stdout:ERROR: (22) Invalid argument 2024-03-11T21:44:16.063 INFO:teuthology.orchestra.run.smithi092.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-11T21:44:16.113 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:44:16.138 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:44:16.138 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:44:16.138 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:44:16.239 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:44:16.264 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:44:16.264 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:44:16.265 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:44:16.365 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:44:16.391 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:44:16.392 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:44:16.392 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:44:16.493 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:44:16.519 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:44:16.519 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:44:16.520 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:44:16.621 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:44:16.645 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:44:16.645 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:44:16.646 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:44:16.726 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:44:16 smithi092 ceph-mon[29572]: osdmap e429: 8 total, 8 up, 8 in 2024-03-11T21:44:16.726 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:44:16 smithi092 ceph-mon[29572]: pgmap v882: 158 pgs: 3 active+remapped+backfill_toofull, 155 active+clean; 664 MiB data, 7.4 GiB used, 708 GiB / 715 GiB avail; 1005 B/s rd, 0 B/s wr, 600 op/s; 2215/32445 objects misplaced (6.827%); 13 MiB/s, 209 objects/s recovering 2024-03-11T21:44:16.726 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:44:16 smithi092 ceph-mon[29572]: 4.1d scrub starts 2024-03-11T21:44:16.726 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:44:16 smithi092 ceph-mon[29572]: 4.1d scrub ok 2024-03-11T21:44:16.727 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:44:16 smithi092 ceph-mon[37563]: osdmap e429: 8 total, 8 up, 8 in 2024-03-11T21:44:16.727 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:44:16 smithi092 ceph-mon[37563]: pgmap v882: 158 pgs: 3 active+remapped+backfill_toofull, 155 active+clean; 664 MiB data, 7.4 GiB used, 708 GiB / 715 GiB avail; 1005 B/s rd, 0 B/s wr, 600 op/s; 2215/32445 objects misplaced (6.827%); 13 MiB/s, 209 objects/s recovering 2024-03-11T21:44:16.727 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:44:16 smithi092 ceph-mon[37563]: 4.1d scrub starts 2024-03-11T21:44:16.727 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:44:16 smithi092 ceph-mon[37563]: 4.1d scrub ok 2024-03-11T21:44:16.746 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:44:16.751 DEBUG:teuthology.orchestra.run.smithi092:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid 00db0016-dfed-11ee-95c8-87774f69a715 -- ceph daemon osd.0 dump_historic_ops 2024-03-11T21:44:16.771 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:44:16.771 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:44:16.771 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:44:16.832 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:44:16 smithi097 ceph-mon[33317]: osdmap e429: 8 total, 8 up, 8 in 2024-03-11T21:44:16.833 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:44:16 smithi097 ceph-mon[33317]: pgmap v882: 158 pgs: 3 active+remapped+backfill_toofull, 155 active+clean; 664 MiB data, 7.4 GiB used, 708 GiB / 715 GiB avail; 1005 B/s rd, 0 B/s wr, 600 op/s; 2215/32445 objects misplaced (6.827%); 13 MiB/s, 209 objects/s recovering 2024-03-11T21:44:16.833 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:44:16 smithi097 ceph-mon[33317]: 4.1d scrub starts 2024-03-11T21:44:16.833 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:44:16 smithi097 ceph-mon[33317]: 4.1d scrub ok 2024-03-11T21:44:16.872 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:44:16.898 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:44:16.898 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:44:16.899 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:44:16.999 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:44:17.024 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:44:17.024 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:44:17.025 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:44:17.126 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:44:17.151 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:44:17.152 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:44:17.152 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:44:17.253 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:44:17.278 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:44:17.279 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:44:17.280 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:44:17.381 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:44:17.420 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:44:17.421 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:44:17.422 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:44:17.522 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:44:17.550 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:44:17 smithi092 ceph-mon[29572]: 6.1 scrub starts 2024-03-11T21:44:17.551 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:44:17 smithi092 ceph-mon[29572]: 6.1 scrub ok 2024-03-11T21:44:17.551 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:44:17 smithi092 ceph-mon[29572]: 10.e scrub starts 2024-03-11T21:44:17.551 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:44:17 smithi092 ceph-mon[29572]: 10.e scrub ok 2024-03-11T21:44:17.551 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:44:17 smithi092 ceph-mon[29572]: osdmap e430: 8 total, 8 up, 8 in 2024-03-11T21:44:17.551 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:44:17 smithi092 ceph-mon[29572]: 10.b scrub starts 2024-03-11T21:44:17.552 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:44:17 smithi092 ceph-mon[29572]: 4.8 scrub starts 2024-03-11T21:44:17.552 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:44:17 smithi092 ceph-mon[29572]: 4.8 scrub ok 2024-03-11T21:44:17.552 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:44:17 smithi092 ceph-mon[37563]: 6.1 scrub starts 2024-03-11T21:44:17.552 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:44:17 smithi092 ceph-mon[37563]: 6.1 scrub ok 2024-03-11T21:44:17.552 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:44:17 smithi092 ceph-mon[37563]: 10.e scrub starts 2024-03-11T21:44:17.553 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:44:17 smithi092 ceph-mon[37563]: 10.e scrub ok 2024-03-11T21:44:17.553 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:44:17 smithi092 ceph-mon[37563]: osdmap e430: 8 total, 8 up, 8 in 2024-03-11T21:44:17.553 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:44:17 smithi092 ceph-mon[37563]: 10.b scrub starts 2024-03-11T21:44:17.553 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:44:17 smithi092 ceph-mon[37563]: 4.8 scrub starts 2024-03-11T21:44:17.553 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:44:17 smithi092 ceph-mon[37563]: 4.8 scrub ok 2024-03-11T21:44:17.555 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:44:17.555 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:44:17.555 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:44:17.656 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:44:17.708 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:44:17.709 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:44:17.709 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:44:17.810 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:44:17.833 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:44:17 smithi097 ceph-mon[33317]: 6.1 scrub starts 2024-03-11T21:44:17.833 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:44:17 smithi097 ceph-mon[33317]: 6.1 scrub ok 2024-03-11T21:44:17.833 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:44:17 smithi097 ceph-mon[33317]: 10.e scrub starts 2024-03-11T21:44:17.834 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:44:17 smithi097 ceph-mon[33317]: 10.e scrub ok 2024-03-11T21:44:17.834 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:44:17 smithi097 ceph-mon[33317]: osdmap e430: 8 total, 8 up, 8 in 2024-03-11T21:44:17.834 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:44:17 smithi097 ceph-mon[33317]: 10.b scrub starts 2024-03-11T21:44:17.834 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:44:17 smithi097 ceph-mon[33317]: 4.8 scrub starts 2024-03-11T21:44:17.834 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:44:17 smithi097 ceph-mon[33317]: 4.8 scrub ok 2024-03-11T21:44:17.850 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:44:17.851 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:44:17.851 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:44:17.952 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:44:17.986 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:44:17.987 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:44:17.987 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:44:18.088 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:44:18.119 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:44:18.119 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:44:18.120 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:44:18.221 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:44:18.264 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:44:18.265 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:44:18.265 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:44:18.360 INFO:journalctl@ceph.mgr.y.smithi092.stdout:Mar 11 21:44:17 smithi092 ceph-00db0016-dfed-11ee-95c8-87774f69a715-mgr-y[29801]: ::ffff:172.21.15.97 - - [11/Mar/2024:21:44:17] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-11T21:44:18.366 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:44:18.392 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:44:18.392 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:44:18.393 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:44:18.494 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:44:18.526 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:44:18.526 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:44:18.527 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:44:18.617 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:44:18 smithi092 ceph-mon[29572]: 10.b scrub ok 2024-03-11T21:44:18.617 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:44:18 smithi092 ceph-mon[29572]: 5.19 scrub starts 2024-03-11T21:44:18.617 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:44:18 smithi092 ceph-mon[29572]: 5.19 scrub ok 2024-03-11T21:44:18.617 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:44:18 smithi092 ceph-mon[29572]: 6.1a scrub starts 2024-03-11T21:44:18.617 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:44:18 smithi092 ceph-mon[29572]: 6.1a scrub ok 2024-03-11T21:44:18.618 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:44:18 smithi092 ceph-mon[29572]: osdmap e431: 8 total, 8 up, 8 in 2024-03-11T21:44:18.618 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:44:18 smithi092 ceph-mon[29572]: pgmap v885: 157 pgs: 1 peering, 3 active+remapped+backfill_toofull, 153 active+clean; 619 MiB data, 7.2 GiB used, 708 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 345 op/s; 2215/30276 objects misplaced (7.316%) 2024-03-11T21:44:18.618 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:44:18 smithi092 ceph-mon[29572]: 5.4 scrub starts 2024-03-11T21:44:18.618 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:44:18 smithi092 ceph-mon[29572]: 5.4 scrub ok 2024-03-11T21:44:18.618 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:44:18 smithi092 ceph-mon[29572]: 6.19 scrub starts 2024-03-11T21:44:18.618 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:44:18 smithi092 ceph-mon[29572]: 6.19 scrub ok 2024-03-11T21:44:18.618 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:44:18 smithi092 ceph-mon[37563]: 10.b scrub ok 2024-03-11T21:44:18.618 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:44:18 smithi092 ceph-mon[37563]: 5.19 scrub starts 2024-03-11T21:44:18.618 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:44:18 smithi092 ceph-mon[37563]: 5.19 scrub ok 2024-03-11T21:44:18.618 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:44:18 smithi092 ceph-mon[37563]: 6.1a scrub starts 2024-03-11T21:44:18.618 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:44:18 smithi092 ceph-mon[37563]: 6.1a scrub ok 2024-03-11T21:44:18.618 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:44:18 smithi092 ceph-mon[37563]: osdmap e431: 8 total, 8 up, 8 in 2024-03-11T21:44:18.619 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:44:18 smithi092 ceph-mon[37563]: pgmap v885: 157 pgs: 1 peering, 3 active+remapped+backfill_toofull, 153 active+clean; 619 MiB data, 7.2 GiB used, 708 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 345 op/s; 2215/30276 objects misplaced (7.316%) 2024-03-11T21:44:18.619 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:44:18 smithi092 ceph-mon[37563]: 5.4 scrub starts 2024-03-11T21:44:18.619 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:44:18 smithi092 ceph-mon[37563]: 5.4 scrub ok 2024-03-11T21:44:18.619 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:44:18 smithi092 ceph-mon[37563]: 6.19 scrub starts 2024-03-11T21:44:18.619 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:44:18 smithi092 ceph-mon[37563]: 6.19 scrub ok 2024-03-11T21:44:18.627 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:44:18.666 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:44:18.667 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:44:18.667 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:44:18.768 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:44:18.799 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:44:18.799 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:44:18.800 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:44:18.833 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:44:18 smithi097 ceph-mon[33317]: 10.b scrub ok 2024-03-11T21:44:18.833 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:44:18 smithi097 ceph-mon[33317]: 5.19 scrub starts 2024-03-11T21:44:18.833 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:44:18 smithi097 ceph-mon[33317]: 5.19 scrub ok 2024-03-11T21:44:18.833 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:44:18 smithi097 ceph-mon[33317]: 6.1a scrub starts 2024-03-11T21:44:18.833 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:44:18 smithi097 ceph-mon[33317]: 6.1a scrub ok 2024-03-11T21:44:18.833 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:44:18 smithi097 ceph-mon[33317]: osdmap e431: 8 total, 8 up, 8 in 2024-03-11T21:44:18.833 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:44:18 smithi097 ceph-mon[33317]: pgmap v885: 157 pgs: 1 peering, 3 active+remapped+backfill_toofull, 153 active+clean; 619 MiB data, 7.2 GiB used, 708 GiB / 715 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 345 op/s; 2215/30276 objects misplaced (7.316%) 2024-03-11T21:44:18.833 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:44:18 smithi097 ceph-mon[33317]: 5.4 scrub starts 2024-03-11T21:44:18.833 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:44:18 smithi097 ceph-mon[33317]: 5.4 scrub ok 2024-03-11T21:44:18.833 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:44:18 smithi097 ceph-mon[33317]: 6.19 scrub starts 2024-03-11T21:44:18.834 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:44:18 smithi097 ceph-mon[33317]: 6.19 scrub ok 2024-03-11T21:44:18.900 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:44:18.938 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:44:18.938 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:44:18.939 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:44:19.039 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:44:19.088 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:44:19.089 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:44:19.089 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:44:19.190 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:44:19.228 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:44:19.229 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:44:19.229 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:44:19.330 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:44:19.378 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:44:19.378 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:44:19.379 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:44:19.479 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:44:19.532 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:44:19.533 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:44:19.533 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:44:19.617 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:44:19 smithi092 ceph-mon[29572]: 5.d scrub starts 2024-03-11T21:44:19.617 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:44:19 smithi092 ceph-mon[29572]: 5.d scrub ok 2024-03-11T21:44:19.617 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:44:19 smithi092 ceph-mon[29572]: 2.f scrub starts 2024-03-11T21:44:19.617 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:44:19 smithi092 ceph-mon[29572]: 2.f scrub ok 2024-03-11T21:44:19.618 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:44:19 smithi092 ceph-mon[37563]: 5.d scrub starts 2024-03-11T21:44:19.618 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:44:19 smithi092 ceph-mon[37563]: 5.d scrub ok 2024-03-11T21:44:19.618 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:44:19 smithi092 ceph-mon[37563]: 2.f scrub starts 2024-03-11T21:44:19.618 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:44:19 smithi092 ceph-mon[37563]: 2.f scrub ok 2024-03-11T21:44:19.634 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:44:19.662 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:44:19.663 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:44:19.663 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:44:19.704 INFO:teuthology.orchestra.run.smithi092.stderr:Inferring config /var/lib/ceph/00db0016-dfed-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T21:44:19.728 INFO:teuthology.orchestra.run.smithi092.stdout:osd.0: {} 2024-03-11T21:44:19.764 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:44:19.777 INFO:teuthology.orchestra.run.smithi092.stdout:osd.1: {} 2024-03-11T21:44:19.792 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:44:19.792 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:44:19.793 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:44:19.797 INFO:teuthology.orchestra.run.smithi092.stderr:Inferring config /var/lib/ceph/00db0016-dfed-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T21:44:19.832 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:44:19 smithi097 ceph-mon[33317]: 5.d scrub starts 2024-03-11T21:44:19.833 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:44:19 smithi097 ceph-mon[33317]: 5.d scrub ok 2024-03-11T21:44:19.833 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:44:19 smithi097 ceph-mon[33317]: 2.f scrub starts 2024-03-11T21:44:19.833 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:44:19 smithi097 ceph-mon[33317]: 2.f scrub ok 2024-03-11T21:44:19.839 INFO:teuthology.orchestra.run.smithi092.stdout:osd.2: {} 2024-03-11T21:44:19.882 INFO:teuthology.orchestra.run.smithi092.stdout:osd.3: {} 2024-03-11T21:44:19.882 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:44:19 smithi092 ceph-00db0016-dfed-11ee-95c8-87774f69a715-mon-a[29568]: 2024-03-11T21:44:19.881+0000 7f87a4389700 -1 mon.a@0(leader).osd e431 definitely_dead 0 2024-03-11T21:44:19.893 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:44:19.917 INFO:teuthology.orchestra.run.smithi092.stdout:osd.4: {} 2024-03-11T21:44:19.930 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:44:19.930 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:44:19.931 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:44:19.951 INFO:teuthology.orchestra.run.smithi092.stdout:osd.5: {} 2024-03-11T21:44:19.994 INFO:teuthology.orchestra.run.smithi092.stdout:osd.6: {} 2024-03-11T21:44:20.031 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:44:20.032 INFO:teuthology.orchestra.run.smithi092.stdout:osd.7: {} 2024-03-11T21:44:20.037 INFO:teuthology.orchestra.run.smithi092.stderr:osd.0: osd_enable_op_tracker = '' 2024-03-11T21:44:20.037 INFO:teuthology.orchestra.run.smithi092.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-11T21:44:20.037 INFO:teuthology.orchestra.run.smithi092.stderr:osd.2: osd_enable_op_tracker = '' 2024-03-11T21:44:20.037 INFO:teuthology.orchestra.run.smithi092.stderr:osd.3: osd_enable_op_tracker = '' 2024-03-11T21:44:20.037 INFO:teuthology.orchestra.run.smithi092.stderr:osd.4: osd_enable_op_tracker = '' 2024-03-11T21:44:20.037 INFO:teuthology.orchestra.run.smithi092.stderr:osd.5: osd_enable_op_tracker = '' 2024-03-11T21:44:20.037 INFO:teuthology.orchestra.run.smithi092.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-11T21:44:20.038 INFO:teuthology.orchestra.run.smithi092.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-11T21:44:20.074 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:44:20.075 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:44:20.076 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:44:20.177 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:44:20.210 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:44:20.211 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:44:20.211 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:44:20.312 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:44:20.351 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:44:20.351 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:44:20.352 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:44:20.446 INFO:teuthology.orchestra.run.smithi092.stderr:marked out osd.3. 2024-03-11T21:44:20.453 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:44:20.480 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:44:20.480 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:44:20.481 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:44:20.581 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:44:20.606 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:44:20.606 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:44:20.606 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:44:20.707 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:44:20.731 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:44:20.732 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:44:20.732 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:44:20.832 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:44:20.833 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:44:20 smithi097 ceph-mon[33317]: 4.e scrub starts 2024-03-11T21:44:20.833 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:44:20 smithi097 ceph-mon[33317]: 4.e scrub ok 2024-03-11T21:44:20.834 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:44:20 smithi097 ceph-mon[33317]: 4.1f scrub starts 2024-03-11T21:44:20.834 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:44:20 smithi097 ceph-mon[33317]: 4.1f scrub ok 2024-03-11T21:44:20.834 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:44:20 smithi097 ceph-mon[33317]: pgmap v886: 157 pgs: 1 peering, 3 active+remapped+backfill_toofull, 153 active+clean; 504 MiB data, 6.9 GiB used, 708 GiB / 715 GiB avail; 1014 B/s rd, 0 B/s wr, 567 op/s; 1653/24741 objects misplaced (6.681%) 2024-03-11T21:44:20.834 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:44:20 smithi097 ceph-mon[33317]: 5.1a scrub starts 2024-03-11T21:44:20.834 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:44:20 smithi097 ceph-mon[33317]: 5.1a scrub ok 2024-03-11T21:44:20.834 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:44:20 smithi097 ceph-mon[33317]: from='client.? v1:172.21.15.92:0/2513675881' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-11T21:44:20.834 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:44:20 smithi097 ceph-mon[33317]: from='client.? v1:172.21.15.92:0/2761461617' entity='client.admin' cmd=[{"prefix": "osd out", "ids": ["3"]}]: dispatch 2024-03-11T21:44:20.834 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:44:20 smithi097 ceph-mon[33317]: from='client.35729 ' entity='client.admin' cmd=[{"prefix": "osd out", "ids": ["3"]}]: dispatch 2024-03-11T21:44:20.834 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:44:20 smithi097 ceph-mon[33317]: Client client.admin marked osd.3 out, while it was still marked up 2024-03-11T21:44:20.834 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:44:20 smithi097 ceph-mon[33317]: 5.13 scrub starts 2024-03-11T21:44:20.834 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:44:20 smithi097 ceph-mon[33317]: 5.13 scrub ok 2024-03-11T21:44:20.858 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:44:20.859 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:44:20.859 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:44:20.867 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:44:20 smithi092 ceph-mon[37563]: 4.e scrub starts 2024-03-11T21:44:20.867 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:44:20 smithi092 ceph-mon[37563]: 4.e scrub ok 2024-03-11T21:44:20.867 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:44:20 smithi092 ceph-mon[37563]: 4.1f scrub starts 2024-03-11T21:44:20.867 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:44:20 smithi092 ceph-mon[37563]: 4.1f scrub ok 2024-03-11T21:44:20.867 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:44:20 smithi092 ceph-mon[37563]: pgmap v886: 157 pgs: 1 peering, 3 active+remapped+backfill_toofull, 153 active+clean; 504 MiB data, 6.9 GiB used, 708 GiB / 715 GiB avail; 1014 B/s rd, 0 B/s wr, 567 op/s; 1653/24741 objects misplaced (6.681%) 2024-03-11T21:44:20.867 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:44:20 smithi092 ceph-mon[37563]: 5.1a scrub starts 2024-03-11T21:44:20.867 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:44:20 smithi092 ceph-mon[37563]: 5.1a scrub ok 2024-03-11T21:44:20.867 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:44:20 smithi092 ceph-mon[37563]: from='client.? v1:172.21.15.92:0/2513675881' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-11T21:44:20.867 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:44:20 smithi092 ceph-mon[37563]: from='client.? v1:172.21.15.92:0/2761461617' entity='client.admin' cmd=[{"prefix": "osd out", "ids": ["3"]}]: dispatch 2024-03-11T21:44:20.867 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:44:20 smithi092 ceph-mon[37563]: from='client.35729 ' entity='client.admin' cmd=[{"prefix": "osd out", "ids": ["3"]}]: dispatch 2024-03-11T21:44:20.867 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:44:20 smithi092 ceph-mon[37563]: Client client.admin marked osd.3 out, while it was still marked up 2024-03-11T21:44:20.868 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:44:20 smithi092 ceph-mon[37563]: 5.13 scrub starts 2024-03-11T21:44:20.868 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:44:20 smithi092 ceph-mon[37563]: 5.13 scrub ok 2024-03-11T21:44:20.868 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:44:20 smithi092 ceph-mon[29572]: 4.e scrub starts 2024-03-11T21:44:20.868 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:44:20 smithi092 ceph-mon[29572]: 4.e scrub ok 2024-03-11T21:44:20.868 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:44:20 smithi092 ceph-mon[29572]: 4.1f scrub starts 2024-03-11T21:44:20.868 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:44:20 smithi092 ceph-mon[29572]: 4.1f scrub ok 2024-03-11T21:44:20.868 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:44:20 smithi092 ceph-mon[29572]: pgmap v886: 157 pgs: 1 peering, 3 active+remapped+backfill_toofull, 153 active+clean; 504 MiB data, 6.9 GiB used, 708 GiB / 715 GiB avail; 1014 B/s rd, 0 B/s wr, 567 op/s; 1653/24741 objects misplaced (6.681%) 2024-03-11T21:44:20.868 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:44:20 smithi092 ceph-mon[29572]: 5.1a scrub starts 2024-03-11T21:44:20.868 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:44:20 smithi092 ceph-mon[29572]: 5.1a scrub ok 2024-03-11T21:44:20.868 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:44:20 smithi092 ceph-mon[29572]: from='client.? v1:172.21.15.92:0/2513675881' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-11T21:44:20.868 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:44:20 smithi092 ceph-mon[29572]: from='client.? v1:172.21.15.92:0/2761461617' entity='client.admin' cmd=[{"prefix": "osd out", "ids": ["3"]}]: dispatch 2024-03-11T21:44:20.869 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:44:20 smithi092 ceph-mon[29572]: from='client.35729 ' entity='client.admin' cmd=[{"prefix": "osd out", "ids": ["3"]}]: dispatch 2024-03-11T21:44:20.869 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:44:20 smithi092 ceph-mon[29572]: Client client.admin marked osd.3 out, while it was still marked up 2024-03-11T21:44:20.869 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:44:20 smithi092 ceph-mon[29572]: 5.13 scrub starts 2024-03-11T21:44:20.869 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:44:20 smithi092 ceph-mon[29572]: 5.13 scrub ok 2024-03-11T21:44:20.959 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:44:20.984 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:44:20.984 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:44:20.984 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:44:21.085 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:44:21.109 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:44:21.109 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:44:21.109 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:44:21.210 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:44:21.236 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:44:21.237 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:44:21.237 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:44:21.338 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:44:21.363 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:44:21.363 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:44:21.363 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:44:21.464 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:44:21.490 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:44:21.491 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:44:21.492 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:44:21.592 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:44:21.617 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:44:21.617 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:44:21.617 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:44:21.618 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:44:21 smithi092 ceph-mon[29572]: 5.b scrub starts 2024-03-11T21:44:21.618 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:44:21 smithi092 ceph-mon[29572]: 5.b scrub ok 2024-03-11T21:44:21.618 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:44:21 smithi092 ceph-mon[29572]: 2.9 scrub starts 2024-03-11T21:44:21.618 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:44:21 smithi092 ceph-mon[29572]: 2.9 scrub ok 2024-03-11T21:44:21.618 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:44:21 smithi092 ceph-mon[29572]: from='client.35729 ' entity='client.admin' cmd='[{"prefix": "osd out", "ids": ["3"]}]': finished 2024-03-11T21:44:21.618 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:44:21 smithi092 ceph-mon[29572]: osdmap e432: 8 total, 8 up, 7 in 2024-03-11T21:44:21.618 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:44:21 smithi092 ceph-mon[29572]: 10.f scrub starts 2024-03-11T21:44:21.618 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:44:21 smithi092 ceph-mon[29572]: 1.0 scrub starts 2024-03-11T21:44:21.618 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:44:21 smithi092 ceph-mon[29572]: 1.0 scrub ok 2024-03-11T21:44:21.619 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:44:21 smithi092 ceph-mon[29572]: 2.3 scrub starts 2024-03-11T21:44:21.619 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:44:21 smithi092 ceph-mon[29572]: 2.3 scrub ok 2024-03-11T21:44:21.619 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:44:21 smithi092 ceph-mon[37563]: 5.b scrub starts 2024-03-11T21:44:21.619 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:44:21 smithi092 ceph-mon[37563]: 5.b scrub ok 2024-03-11T21:44:21.619 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:44:21 smithi092 ceph-mon[37563]: 2.9 scrub starts 2024-03-11T21:44:21.619 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:44:21 smithi092 ceph-mon[37563]: 2.9 scrub ok 2024-03-11T21:44:21.619 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:44:21 smithi092 ceph-mon[37563]: from='client.35729 ' entity='client.admin' cmd='[{"prefix": "osd out", "ids": ["3"]}]': finished 2024-03-11T21:44:21.619 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:44:21 smithi092 ceph-mon[37563]: osdmap e432: 8 total, 8 up, 7 in 2024-03-11T21:44:21.619 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:44:21 smithi092 ceph-mon[37563]: 10.f scrub starts 2024-03-11T21:44:21.619 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:44:21 smithi092 ceph-mon[37563]: 1.0 scrub starts 2024-03-11T21:44:21.619 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:44:21 smithi092 ceph-mon[37563]: 1.0 scrub ok 2024-03-11T21:44:21.619 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:44:21 smithi092 ceph-mon[37563]: 2.3 scrub starts 2024-03-11T21:44:21.619 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:44:21 smithi092 ceph-mon[37563]: 2.3 scrub ok 2024-03-11T21:44:21.718 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:44:21.733 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:44:21 smithi097 ceph-mon[33317]: 5.b scrub starts 2024-03-11T21:44:21.734 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:44:21 smithi097 ceph-mon[33317]: 5.b scrub ok 2024-03-11T21:44:21.734 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:44:21 smithi097 ceph-mon[33317]: 2.9 scrub starts 2024-03-11T21:44:21.734 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:44:21 smithi097 ceph-mon[33317]: 2.9 scrub ok 2024-03-11T21:44:21.734 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:44:21 smithi097 ceph-mon[33317]: from='client.35729 ' entity='client.admin' cmd='[{"prefix": "osd out", "ids": ["3"]}]': finished 2024-03-11T21:44:21.734 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:44:21 smithi097 ceph-mon[33317]: osdmap e432: 8 total, 8 up, 7 in 2024-03-11T21:44:21.734 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:44:21 smithi097 ceph-mon[33317]: 10.f scrub starts 2024-03-11T21:44:21.734 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:44:21 smithi097 ceph-mon[33317]: 1.0 scrub starts 2024-03-11T21:44:21.734 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:44:21 smithi097 ceph-mon[33317]: 1.0 scrub ok 2024-03-11T21:44:21.734 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:44:21 smithi097 ceph-mon[33317]: 2.3 scrub starts 2024-03-11T21:44:21.734 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:44:21 smithi097 ceph-mon[33317]: 2.3 scrub ok 2024-03-11T21:44:21.742 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:44:21.742 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:44:21.743 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:44:21.844 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:44:21.867 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:44:21.867 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:44:21.868 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:44:21.969 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:44:21.992 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:44:21.993 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:44:21.994 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:44:22.095 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:44:22.120 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:44:22.120 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:44:22.120 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:44:22.221 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:44:22.245 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:44:22.245 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:44:22.246 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:44:22.346 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:44:22.372 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:44:22.373 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:44:22.373 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:44:22.473 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:44:22.502 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:44:22.503 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:44:22.503 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:44:22.604 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:44:22.645 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:44:22.646 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:44:22.646 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:44:22.747 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:44:22.755 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:44:22 smithi092 ceph-mon[29572]: 4.2 scrub starts 2024-03-11T21:44:22.756 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:44:22 smithi092 ceph-mon[29572]: 4.2 scrub ok 2024-03-11T21:44:22.756 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:44:22 smithi092 ceph-mon[29572]: purged_snaps scrub starts 2024-03-11T21:44:22.756 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:44:22 smithi092 ceph-mon[29572]: purged_snaps scrub ok 2024-03-11T21:44:22.756 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:44:22 smithi092 ceph-mon[29572]: pgmap v888: 157 pgs: 1 peering, 3 active+remapped+backfill_toofull, 153 active+clean; 494 MiB data, 6.3 GiB used, 620 GiB / 626 GiB avail; 853 B/s rd, 0 B/s wr, 454 op/s; 1653/24273 objects misplaced (6.810%) 2024-03-11T21:44:22.756 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:44:22 smithi092 ceph-mon[29572]: 10.14 continuing backfill to osd.0 from (369'3693,431'8776] MIN to 431'8776 2024-03-11T21:44:22.756 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:44:22 smithi092 ceph-mon[29572]: osdmap e433: 8 total, 8 up, 7 in 2024-03-11T21:44:22.756 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:44:22 smithi092 ceph-mon[29572]: 2.10 scrub starts 2024-03-11T21:44:22.756 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:44:22 smithi092 ceph-mon[29572]: 2.10 scrub ok 2024-03-11T21:44:22.756 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:44:22 smithi092 ceph-mon[37563]: 4.2 scrub starts 2024-03-11T21:44:22.756 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:44:22 smithi092 ceph-mon[37563]: 4.2 scrub ok 2024-03-11T21:44:22.756 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:44:22 smithi092 ceph-mon[37563]: purged_snaps scrub starts 2024-03-11T21:44:22.757 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:44:22 smithi092 ceph-mon[37563]: purged_snaps scrub ok 2024-03-11T21:44:22.757 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:44:22 smithi092 ceph-mon[37563]: pgmap v888: 157 pgs: 1 peering, 3 active+remapped+backfill_toofull, 153 active+clean; 494 MiB data, 6.3 GiB used, 620 GiB / 626 GiB avail; 853 B/s rd, 0 B/s wr, 454 op/s; 1653/24273 objects misplaced (6.810%) 2024-03-11T21:44:22.757 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:44:22 smithi092 ceph-mon[37563]: 10.14 continuing backfill to osd.0 from (369'3693,431'8776] MIN to 431'8776 2024-03-11T21:44:22.757 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:44:22 smithi092 ceph-mon[37563]: osdmap e433: 8 total, 8 up, 7 in 2024-03-11T21:44:22.757 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:44:22 smithi092 ceph-mon[37563]: 2.10 scrub starts 2024-03-11T21:44:22.757 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:44:22 smithi092 ceph-mon[37563]: 2.10 scrub ok 2024-03-11T21:44:22.783 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:44:22.783 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:44:22.783 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:44:22.833 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:44:22 smithi097 ceph-mon[33317]: 4.2 scrub starts 2024-03-11T21:44:22.833 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:44:22 smithi097 ceph-mon[33317]: 4.2 scrub ok 2024-03-11T21:44:22.833 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:44:22 smithi097 ceph-mon[33317]: purged_snaps scrub starts 2024-03-11T21:44:22.833 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:44:22 smithi097 ceph-mon[33317]: purged_snaps scrub ok 2024-03-11T21:44:22.833 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:44:22 smithi097 ceph-mon[33317]: pgmap v888: 157 pgs: 1 peering, 3 active+remapped+backfill_toofull, 153 active+clean; 494 MiB data, 6.3 GiB used, 620 GiB / 626 GiB avail; 853 B/s rd, 0 B/s wr, 454 op/s; 1653/24273 objects misplaced (6.810%) 2024-03-11T21:44:22.833 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:44:22 smithi097 ceph-mon[33317]: 10.14 continuing backfill to osd.0 from (369'3693,431'8776] MIN to 431'8776 2024-03-11T21:44:22.833 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:44:22 smithi097 ceph-mon[33317]: osdmap e433: 8 total, 8 up, 7 in 2024-03-11T21:44:22.833 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:44:22 smithi097 ceph-mon[33317]: 2.10 scrub starts 2024-03-11T21:44:22.833 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:44:22 smithi097 ceph-mon[33317]: 2.10 scrub ok 2024-03-11T21:44:22.884 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:44:22.917 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:44:22.918 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:44:22.918 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:44:23.019 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:44:23.054 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:44:23.055 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:44:23.055 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:44:23.156 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:44:23.198 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:44:23.199 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:44:23.200 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:44:23.300 DEBUG:teuthology.orchestra.run.smithi092:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid 00db0016-dfed-11ee-95c8-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-03-11T21:44:23.301 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:44:23.333 INFO:journalctl@ceph.iscsi.iscsi.a.smithi097.stdout:Mar 11 21:44:23 smithi097 ceph-00db0016-dfed-11ee-95c8-87774f69a715-iscsi-iscsi-a[60943]: debug there is no tcmu-runner data available 2024-03-11T21:44:23.334 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:44:23.335 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:44:23.335 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:44:23.436 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:44:23.463 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:44:23.463 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:44:23.464 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:44:23.565 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:44:23.615 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:44:23.616 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:44:23.617 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:44:23.617 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:44:23 smithi092 ceph-mon[37563]: 4.14 scrub starts 2024-03-11T21:44:23.617 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:44:23 smithi092 ceph-mon[37563]: 4.14 scrub ok 2024-03-11T21:44:23.617 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:44:23 smithi092 ceph-mon[37563]: 10.4 starting backfill to osd.1 from (0'0,0'0] MAX to 431'8579 2024-03-11T21:44:23.617 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:44:23 smithi092 ceph-mon[37563]: 10.17 starting backfill to osd.2 from (0'0,0'0] MAX to 431'8944 2024-03-11T21:44:23.617 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:44:23 smithi092 ceph-mon[37563]: osdmap e434: 8 total, 8 up, 7 in 2024-03-11T21:44:23.617 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:44:23 smithi092 ceph-mon[37563]: 10.3 starting backfill to osd.4 from (0'0,0'0] MAX to 431'8768 2024-03-11T21:44:23.617 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:44:23 smithi092 ceph-mon[37563]: 10.15 starting backfill to osd.7 from (0'0,0'0] MAX to 431'9009 2024-03-11T21:44:23.617 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:44:23 smithi092 ceph-mon[37563]: 10.7 starting backfill to osd.2 from (0'0,0'0] MAX to 431'8903 2024-03-11T21:44:23.617 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:44:23 smithi092 ceph-mon[37563]: 10.f starting backfill to osd.4 from (0'0,0'0] MAX to 431'13655 2024-03-11T21:44:23.618 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:44:23 smithi092 ceph-mon[37563]: 10.13 starting backfill to osd.4 from (0'0,0'0] MAX to 431'8812 2024-03-11T21:44:23.618 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:44:23 smithi092 ceph-mon[37563]: 10.5 starting backfill to osd.7 from (0'0,0'0] MAX to 431'9834 2024-03-11T21:44:23.618 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:44:23 smithi092 ceph-mon[37563]: from='mgr.14607 ' entity='mgr.y' 2024-03-11T21:44:23.618 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:44:23 smithi092 ceph-mon[37563]: from='mgr.14607 v1:172.21.15.92:0/1702717597' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T21:44:23.618 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:44:23 smithi092 ceph-mon[37563]: from='client.24469 v1:172.21.15.97:0/404208310' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:44:23.618 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:44:23 smithi092 ceph-mon[29572]: 4.14 scrub starts 2024-03-11T21:44:23.618 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:44:23 smithi092 ceph-mon[29572]: 4.14 scrub ok 2024-03-11T21:44:23.618 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:44:23 smithi092 ceph-mon[29572]: 10.4 starting backfill to osd.1 from (0'0,0'0] MAX to 431'8579 2024-03-11T21:44:23.618 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:44:23 smithi092 ceph-mon[29572]: 10.17 starting backfill to osd.2 from (0'0,0'0] MAX to 431'8944 2024-03-11T21:44:23.618 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:44:23 smithi092 ceph-mon[29572]: osdmap e434: 8 total, 8 up, 7 in 2024-03-11T21:44:23.618 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:44:23 smithi092 ceph-mon[29572]: 10.3 starting backfill to osd.4 from (0'0,0'0] MAX to 431'8768 2024-03-11T21:44:23.619 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:44:23 smithi092 ceph-mon[29572]: 10.15 starting backfill to osd.7 from (0'0,0'0] MAX to 431'9009 2024-03-11T21:44:23.619 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:44:23 smithi092 ceph-mon[29572]: 10.7 starting backfill to osd.2 from (0'0,0'0] MAX to 431'8903 2024-03-11T21:44:23.619 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:44:23 smithi092 ceph-mon[29572]: 10.f starting backfill to osd.4 from (0'0,0'0] MAX to 431'13655 2024-03-11T21:44:23.619 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:44:23 smithi092 ceph-mon[29572]: 10.13 starting backfill to osd.4 from (0'0,0'0] MAX to 431'8812 2024-03-11T21:44:23.619 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:44:23 smithi092 ceph-mon[29572]: 10.5 starting backfill to osd.7 from (0'0,0'0] MAX to 431'9834 2024-03-11T21:44:23.619 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:44:23 smithi092 ceph-mon[29572]: from='mgr.14607 ' entity='mgr.y' 2024-03-11T21:44:23.619 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:44:23 smithi092 ceph-mon[29572]: from='mgr.14607 v1:172.21.15.92:0/1702717597' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T21:44:23.619 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:44:23 smithi092 ceph-mon[29572]: from='client.24469 v1:172.21.15.97:0/404208310' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:44:23.718 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:44:23.758 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:44:23.759 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:44:23.759 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:44:23.833 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:44:23 smithi097 ceph-mon[33317]: 4.14 scrub starts 2024-03-11T21:44:23.833 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:44:23 smithi097 ceph-mon[33317]: 4.14 scrub ok 2024-03-11T21:44:23.833 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:44:23 smithi097 ceph-mon[33317]: 10.4 starting backfill to osd.1 from (0'0,0'0] MAX to 431'8579 2024-03-11T21:44:23.833 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:44:23 smithi097 ceph-mon[33317]: 10.17 starting backfill to osd.2 from (0'0,0'0] MAX to 431'8944 2024-03-11T21:44:23.833 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:44:23 smithi097 ceph-mon[33317]: osdmap e434: 8 total, 8 up, 7 in 2024-03-11T21:44:23.834 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:44:23 smithi097 ceph-mon[33317]: 10.3 starting backfill to osd.4 from (0'0,0'0] MAX to 431'8768 2024-03-11T21:44:23.834 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:44:23 smithi097 ceph-mon[33317]: 10.15 starting backfill to osd.7 from (0'0,0'0] MAX to 431'9009 2024-03-11T21:44:23.834 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:44:23 smithi097 ceph-mon[33317]: 10.7 starting backfill to osd.2 from (0'0,0'0] MAX to 431'8903 2024-03-11T21:44:23.834 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:44:23 smithi097 ceph-mon[33317]: 10.f starting backfill to osd.4 from (0'0,0'0] MAX to 431'13655 2024-03-11T21:44:23.834 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:44:23 smithi097 ceph-mon[33317]: 10.13 starting backfill to osd.4 from (0'0,0'0] MAX to 431'8812 2024-03-11T21:44:23.834 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:44:23 smithi097 ceph-mon[33317]: 10.5 starting backfill to osd.7 from (0'0,0'0] MAX to 431'9834 2024-03-11T21:44:23.834 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:44:23 smithi097 ceph-mon[33317]: from='mgr.14607 ' entity='mgr.y' 2024-03-11T21:44:23.834 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:44:23 smithi097 ceph-mon[33317]: from='mgr.14607 v1:172.21.15.92:0/1702717597' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T21:44:23.834 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:44:23 smithi097 ceph-mon[33317]: from='client.24469 v1:172.21.15.97:0/404208310' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:44:23.860 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:44:23.895 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:44:23.896 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:44:23.897 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:44:23.997 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:44:24.037 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:44:24.038 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:44:24.038 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:44:24.139 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:44:24.212 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:44:24.214 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:44:24.214 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:44:24.315 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:44:24.348 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:44:24.348 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:44:24.348 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:44:24.449 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:44:24.484 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:44:24.485 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:44:24.485 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:44:24.586 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:44:24.610 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:44:24.611 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:44:24.612 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:44:24.617 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:44:24 smithi092 ceph-mon[29572]: pgmap v891: 157 pgs: 2 active+remapped+backfilling, 3 activating+remapped, 5 active+remapped+backfill_toofull, 147 active+clean; 324 MiB data, 5.9 GiB used, 620 GiB / 626 GiB avail; 1.5 KiB/s rd, 0 B/s wr, 788 op/s; 2389/16098 objects misplaced (14.840%); 1.7 MiB/s, 34 objects/s recovering 2024-03-11T21:44:24.617 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:44:24 smithi092 ceph-mon[29572]: Health check update: Low space hindering backfill (add storage if this doesn't resolve itself): 5 pgs backfill_toofull (PG_BACKFILL_FULL) 2024-03-11T21:44:24.617 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:44:24 smithi092 ceph-mon[29572]: 2.d scrub starts 2024-03-11T21:44:24.618 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:44:24 smithi092 ceph-mon[29572]: 2.d scrub ok 2024-03-11T21:44:24.618 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:44:24 smithi092 ceph-mon[29572]: 6.c scrub starts 2024-03-11T21:44:24.618 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:44:24 smithi092 ceph-mon[29572]: 6.c scrub ok 2024-03-11T21:44:24.618 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:44:24 smithi092 ceph-mon[29572]: osdmap e435: 8 total, 8 up, 7 in 2024-03-11T21:44:24.618 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:44:24 smithi092 ceph-mon[37563]: pgmap v891: 157 pgs: 2 active+remapped+backfilling, 3 activating+remapped, 5 active+remapped+backfill_toofull, 147 active+clean; 324 MiB data, 5.9 GiB used, 620 GiB / 626 GiB avail; 1.5 KiB/s rd, 0 B/s wr, 788 op/s; 2389/16098 objects misplaced (14.840%); 1.7 MiB/s, 34 objects/s recovering 2024-03-11T21:44:24.619 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:44:24 smithi092 ceph-mon[37563]: Health check update: Low space hindering backfill (add storage if this doesn't resolve itself): 5 pgs backfill_toofull (PG_BACKFILL_FULL) 2024-03-11T21:44:24.619 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:44:24 smithi092 ceph-mon[37563]: 2.d scrub starts 2024-03-11T21:44:24.619 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:44:24 smithi092 ceph-mon[37563]: 2.d scrub ok 2024-03-11T21:44:24.619 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:44:24 smithi092 ceph-mon[37563]: 6.c scrub starts 2024-03-11T21:44:24.619 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:44:24 smithi092 ceph-mon[37563]: 6.c scrub ok 2024-03-11T21:44:24.619 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:44:24 smithi092 ceph-mon[37563]: osdmap e435: 8 total, 8 up, 7 in 2024-03-11T21:44:24.713 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:44:24.736 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:44:24.737 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:44:24.738 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:44:24.833 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:44:24 smithi097 ceph-mon[33317]: pgmap v891: 157 pgs: 2 active+remapped+backfilling, 3 activating+remapped, 5 active+remapped+backfill_toofull, 147 active+clean; 324 MiB data, 5.9 GiB used, 620 GiB / 626 GiB avail; 1.5 KiB/s rd, 0 B/s wr, 788 op/s; 2389/16098 objects misplaced (14.840%); 1.7 MiB/s, 34 objects/s recovering 2024-03-11T21:44:24.833 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:44:24 smithi097 ceph-mon[33317]: Health check update: Low space hindering backfill (add storage if this doesn't resolve itself): 5 pgs backfill_toofull (PG_BACKFILL_FULL) 2024-03-11T21:44:24.833 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:44:24 smithi097 ceph-mon[33317]: 2.d scrub starts 2024-03-11T21:44:24.833 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:44:24 smithi097 ceph-mon[33317]: 2.d scrub ok 2024-03-11T21:44:24.833 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:44:24 smithi097 ceph-mon[33317]: 6.c scrub starts 2024-03-11T21:44:24.833 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:44:24 smithi097 ceph-mon[33317]: 6.c scrub ok 2024-03-11T21:44:24.834 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:44:24 smithi097 ceph-mon[33317]: osdmap e435: 8 total, 8 up, 7 in 2024-03-11T21:44:24.838 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:44:24.863 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:44:24.864 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:44:24.864 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:44:24.895 INFO:teuthology.orchestra.run.smithi092.stderr:Inferring config /var/lib/ceph/00db0016-dfed-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T21:44:24.965 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:44:24.990 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:44:24.990 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:44:24.990 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:44:25.091 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:44:25.114 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:44:25.115 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:44:25.116 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:44:25.217 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:44:25.240 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:44:25.241 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:44:25.241 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:44:25.342 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:44:25.367 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:44:25.367 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:44:25.368 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:44:25.440 INFO:teuthology.orchestra.run.smithi092.stderr:nodeep-scrub is unset 2024-03-11T21:44:25.469 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:44:25.506 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:44:25.507 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:44:25.507 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:44:25.608 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:44:25.646 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:44:25.646 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:44:25.647 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:44:25.748 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:44:25.783 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:44:25.784 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:44:25.784 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:44:25.787 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:44:25 smithi092 ceph-mon[29572]: from='client.? v1:172.21.15.92:0/946596143' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-11T21:44:25.787 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:44:25 smithi092 ceph-mon[29572]: from='client.35738 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-11T21:44:25.787 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:44:25 smithi092 ceph-mon[29572]: 10.6 scrub starts 2024-03-11T21:44:25.787 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:44:25 smithi092 ceph-mon[29572]: 10.6 scrub ok 2024-03-11T21:44:25.787 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:44:25 smithi092 ceph-mon[29572]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-03-11T21:44:25.788 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:44:25 smithi092 ceph-mon[29572]: from='client.35738 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-03-11T21:44:25.788 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:44:25 smithi092 ceph-mon[29572]: osdmap e436: 8 total, 8 up, 7 in 2024-03-11T21:44:25.788 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:44:25 smithi092 ceph-mon[37563]: from='client.? v1:172.21.15.92:0/946596143' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-11T21:44:25.788 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:44:25 smithi092 ceph-mon[37563]: from='client.35738 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-11T21:44:25.788 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:44:25 smithi092 ceph-mon[37563]: 10.6 scrub starts 2024-03-11T21:44:25.788 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:44:25 smithi092 ceph-mon[37563]: 10.6 scrub ok 2024-03-11T21:44:25.788 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:44:25 smithi092 ceph-mon[37563]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-03-11T21:44:25.788 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:44:25 smithi092 ceph-mon[37563]: from='client.35738 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-03-11T21:44:25.788 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:44:25 smithi092 ceph-mon[37563]: osdmap e436: 8 total, 8 up, 7 in 2024-03-11T21:44:25.833 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:44:25 smithi097 ceph-mon[33317]: from='client.? v1:172.21.15.92:0/946596143' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-11T21:44:25.833 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:44:25 smithi097 ceph-mon[33317]: from='client.35738 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-11T21:44:25.833 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:44:25 smithi097 ceph-mon[33317]: 10.6 scrub starts 2024-03-11T21:44:25.833 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:44:25 smithi097 ceph-mon[33317]: 10.6 scrub ok 2024-03-11T21:44:25.833 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:44:25 smithi097 ceph-mon[33317]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-03-11T21:44:25.833 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:44:25 smithi097 ceph-mon[33317]: from='client.35738 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-03-11T21:44:25.833 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:44:25 smithi097 ceph-mon[33317]: osdmap e436: 8 total, 8 up, 7 in 2024-03-11T21:44:25.886 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:44:25.923 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:44:25.923 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:44:25.924 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:44:26.025 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:44:26.060 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:44:26.061 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:44:26.061 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:44:26.162 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:44:26.196 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:44:26.197 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:44:26.198 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:44:26.299 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:44:26.307 INFO:teuthology.orchestra.run.smithi092.stdout:{ 2024-03-11T21:44:26.307 INFO:teuthology.orchestra.run.smithi092.stdout: "size": 20, 2024-03-11T21:44:26.307 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 600, 2024-03-11T21:44:26.307 INFO:teuthology.orchestra.run.smithi092.stdout: "ops": [ 2024-03-11T21:44:26.307 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:44:26.307 INFO:teuthology.orchestra.run.smithi092.stdout: "description": "osd_op(client.15150.0:207121 8.c 8:3b538f8b:::benchmark_data_smithi092_104865_object86620:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e237)", 2024-03-11T21:44:26.307 INFO:teuthology.orchestra.run.smithi092.stdout: "initiated_at": "2024-03-11T21:34:55.373083+0000", 2024-03-11T21:44:26.307 INFO:teuthology.orchestra.run.smithi092.stdout: "age": 570.93975501, 2024-03-11T21:44:26.307 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.98710652499999996, 2024-03-11T21:44:26.307 INFO:teuthology.orchestra.run.smithi092.stdout: "type_data": { 2024-03-11T21:44:26.307 INFO:teuthology.orchestra.run.smithi092.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-11T21:44:26.308 INFO:teuthology.orchestra.run.smithi092.stdout: "client_info": { 2024-03-11T21:44:26.308 INFO:teuthology.orchestra.run.smithi092.stdout: "client": "client.15150", 2024-03-11T21:44:26.308 INFO:teuthology.orchestra.run.smithi092.stdout: "client_addr": "v1:172.21.15.92:0/436606717", 2024-03-11T21:44:26.308 INFO:teuthology.orchestra.run.smithi092.stdout: "tid": 207121 2024-03-11T21:44:26.308 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:44:26.308 INFO:teuthology.orchestra.run.smithi092.stdout: "events": [ 2024-03-11T21:44:26.308 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:44:26.308 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "initiated", 2024-03-11T21:44:26.308 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:34:55.373083+0000", 2024-03-11T21:44:26.308 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-03-11T21:44:26.308 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:44:26.308 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:44:26.308 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "header_read", 2024-03-11T21:44:26.308 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:34:55.373083+0000", 2024-03-11T21:44:26.309 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-03-11T21:44:26.309 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:44:26.309 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:44:26.309 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "throttled", 2024-03-11T21:44:26.309 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:34:55.373085+0000", 2024-03-11T21:44:26.309 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.206e-06 2024-03-11T21:44:26.309 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:44:26.309 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:44:26.309 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "all_read", 2024-03-11T21:44:26.310 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:34:55.373090+0000", 2024-03-11T21:44:26.310 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 5.6760000000000004e-06 2024-03-11T21:44:26.310 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:44:26.310 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:44:26.310 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "dispatched", 2024-03-11T21:44:26.310 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:34:55.373107+0000", 2024-03-11T21:44:26.310 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.6362999999999999e-05 2024-03-11T21:44:26.310 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:44:26.311 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:44:26.311 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "queued_for_pg", 2024-03-11T21:44:26.311 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:34:55.373115+0000", 2024-03-11T21:44:26.311 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 8.2660000000000001e-06 2024-03-11T21:44:26.311 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:44:26.311 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:44:26.311 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-03-11T21:44:26.311 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:34:55.373173+0000", 2024-03-11T21:44:26.312 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 5.7991000000000001e-05 2024-03-11T21:44:26.312 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:44:26.312 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:44:26.312 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "waiting for peered", 2024-03-11T21:44:26.312 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:34:55.373182+0000", 2024-03-11T21:44:26.312 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 9.0049999999999993e-06 2024-03-11T21:44:26.312 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:44:26.312 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:44:26.313 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-03-11T21:44:26.313 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:34:56.354869+0000", 2024-03-11T21:44:26.313 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.98168716199999995 2024-03-11T21:44:26.313 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:44:26.313 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:44:26.313 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "started", 2024-03-11T21:44:26.313 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:34:56.355005+0000", 2024-03-11T21:44:26.313 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00013610799999999999 2024-03-11T21:44:26.313 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:44:26.314 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:44:26.314 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "waiting for subops from 3,7", 2024-03-11T21:44:26.314 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:34:56.355119+0000", 2024-03-11T21:44:26.314 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00011411 2024-03-11T21:44:26.314 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:44:26.314 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:44:26.314 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "sub_op_commit_rec", 2024-03-11T21:44:26.314 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:34:56.357333+0000", 2024-03-11T21:44:26.315 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.0022135430000000001 2024-03-11T21:44:26.315 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:44:26.315 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:44:26.315 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "op_commit", 2024-03-11T21:44:26.315 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:34:56.358863+0000", 2024-03-11T21:44:26.315 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.001530399 2024-03-11T21:44:26.315 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:44:26.315 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:44:26.315 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "sub_op_commit_rec", 2024-03-11T21:44:26.316 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:34:56.360128+0000", 2024-03-11T21:44:26.316 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.001264409 2024-03-11T21:44:26.316 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:44:26.316 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:44:26.316 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "commit_sent", 2024-03-11T21:44:26.316 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:34:56.360162+0000", 2024-03-11T21:44:26.316 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 3.4585999999999997e-05 2024-03-11T21:44:26.316 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:44:26.316 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:44:26.317 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "done", 2024-03-11T21:44:26.317 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:34:56.360190+0000", 2024-03-11T21:44:26.317 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 2.7701000000000001e-05 2024-03-11T21:44:26.317 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-03-11T21:44:26.317 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-03-11T21:44:26.317 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-03-11T21:44:26.317 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:44:26.317 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:44:26.317 INFO:teuthology.orchestra.run.smithi092.stdout: "description": "osd_op(client.15150.0:207122 8.c 8:3cecc2cc:::benchmark_data_smithi092_104865_object86621:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e237)", 2024-03-11T21:44:26.318 INFO:teuthology.orchestra.run.smithi092.stdout: "initiated_at": "2024-03-11T21:34:55.374081+0000", 2024-03-11T21:44:26.318 INFO:teuthology.orchestra.run.smithi092.stdout: "age": 570.938757603, 2024-03-11T21:44:26.318 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.98636186100000001, 2024-03-11T21:44:26.318 INFO:teuthology.orchestra.run.smithi092.stdout: "type_data": { 2024-03-11T21:44:26.318 INFO:teuthology.orchestra.run.smithi092.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-11T21:44:26.318 INFO:teuthology.orchestra.run.smithi092.stdout: "client_info": { 2024-03-11T21:44:26.318 INFO:teuthology.orchestra.run.smithi092.stdout: "client": "client.15150", 2024-03-11T21:44:26.318 INFO:teuthology.orchestra.run.smithi092.stdout: "client_addr": "v1:172.21.15.92:0/436606717", 2024-03-11T21:44:26.318 INFO:teuthology.orchestra.run.smithi092.stdout: "tid": 207122 2024-03-11T21:44:26.319 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:44:26.319 INFO:teuthology.orchestra.run.smithi092.stdout: "events": [ 2024-03-11T21:44:26.319 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:44:26.319 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "initiated", 2024-03-11T21:44:26.319 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:34:55.374081+0000", 2024-03-11T21:44:26.319 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-03-11T21:44:26.319 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:44:26.319 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:44:26.320 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "header_read", 2024-03-11T21:44:26.320 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:34:55.374081+0000", 2024-03-11T21:44:26.320 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-03-11T21:44:26.320 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:44:26.320 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:44:26.320 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "throttled", 2024-03-11T21:44:26.320 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:34:55.374082+0000", 2024-03-11T21:44:26.320 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.6020000000000001e-06 2024-03-11T21:44:26.320 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:44:26.321 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:44:26.321 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "all_read", 2024-03-11T21:44:26.321 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:34:55.374088+0000", 2024-03-11T21:44:26.321 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 5.4360000000000001e-06 2024-03-11T21:44:26.321 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:44:26.321 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:44:26.321 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "dispatched", 2024-03-11T21:44:26.321 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:34:55.374102+0000", 2024-03-11T21:44:26.321 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.4222e-05 2024-03-11T21:44:26.322 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:44:26.322 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:44:26.322 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "queued_for_pg", 2024-03-11T21:44:26.322 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:34:55.374110+0000", 2024-03-11T21:44:26.322 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 8.1200000000000002e-06 2024-03-11T21:44:26.322 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:44:26.322 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:44:26.322 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-03-11T21:44:26.322 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:34:55.374167+0000", 2024-03-11T21:44:26.323 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 5.6768999999999998e-05 2024-03-11T21:44:26.323 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:44:26.323 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:44:26.323 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "waiting for peered", 2024-03-11T21:44:26.323 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:34:55.374178+0000", 2024-03-11T21:44:26.323 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.0730000000000001e-05 2024-03-11T21:44:26.323 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:44:26.323 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:44:26.324 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-03-11T21:44:26.324 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:34:56.357391+0000", 2024-03-11T21:44:26.324 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.98321307000000002 2024-03-11T21:44:26.324 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:44:26.324 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:44:26.324 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "started", 2024-03-11T21:44:26.324 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:34:56.357504+0000", 2024-03-11T21:44:26.325 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.000113727 2024-03-11T21:44:26.325 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:44:26.325 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:44:26.325 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "waiting for subops from 3,7", 2024-03-11T21:44:26.325 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:34:56.357573+0000", 2024-03-11T21:44:26.325 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 6.8965000000000001e-05 2024-03-11T21:44:26.325 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:44:26.325 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:44:26.325 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "sub_op_commit_rec", 2024-03-11T21:44:26.325 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:34:56.360278+0000", 2024-03-11T21:44:26.326 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.0027041729999999998 2024-03-11T21:44:26.326 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:44:26.326 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:44:26.326 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "op_commit", 2024-03-11T21:44:26.326 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:34:56.360289+0000", 2024-03-11T21:44:26.326 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.1141000000000001e-05 2024-03-11T21:44:26.326 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:44:26.326 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:44:26.326 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "sub_op_commit_rec", 2024-03-11T21:44:26.327 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:34:56.360324+0000", 2024-03-11T21:44:26.327 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 3.5330999999999998e-05 2024-03-11T21:44:26.327 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:44:26.327 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:44:26.327 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "commit_sent", 2024-03-11T21:44:26.327 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:34:56.360404+0000", 2024-03-11T21:44:26.328 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 8.0290999999999994e-05 2024-03-11T21:44:26.328 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:44:26.328 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:44:26.328 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "done", 2024-03-11T21:44:26.328 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:34:56.360443+0000", 2024-03-11T21:44:26.328 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 3.8284000000000001e-05 2024-03-11T21:44:26.328 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-03-11T21:44:26.329 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-03-11T21:44:26.329 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-03-11T21:44:26.329 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:44:26.329 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:44:26.329 INFO:teuthology.orchestra.run.smithi092.stdout: "description": "osd_op(client.15150.0:207130 8.c 8:3e52addc:::benchmark_data_smithi092_104865_object86629:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e237)", 2024-03-11T21:44:26.329 INFO:teuthology.orchestra.run.smithi092.stdout: "initiated_at": "2024-03-11T21:34:55.382721+0000", 2024-03-11T21:44:26.329 INFO:teuthology.orchestra.run.smithi092.stdout: "age": 570.93011707200003, 2024-03-11T21:44:26.329 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.97792294199999996, 2024-03-11T21:44:26.329 INFO:teuthology.orchestra.run.smithi092.stdout: "type_data": { 2024-03-11T21:44:26.329 INFO:teuthology.orchestra.run.smithi092.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-11T21:44:26.329 INFO:teuthology.orchestra.run.smithi092.stdout: "client_info": { 2024-03-11T21:44:26.329 INFO:teuthology.orchestra.run.smithi092.stdout: "client": "client.15150", 2024-03-11T21:44:26.329 INFO:teuthology.orchestra.run.smithi092.stdout: "client_addr": "v1:172.21.15.92:0/436606717", 2024-03-11T21:44:26.330 INFO:teuthology.orchestra.run.smithi092.stdout: "tid": 207130 2024-03-11T21:44:26.330 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:44:26.330 INFO:teuthology.orchestra.run.smithi092.stdout: "events": [ 2024-03-11T21:44:26.330 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:44:26.330 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "initiated", 2024-03-11T21:44:26.330 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:34:55.382721+0000", 2024-03-11T21:44:26.330 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-03-11T21:44:26.330 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:44:26.330 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:44:26.330 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "header_read", 2024-03-11T21:44:26.330 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:34:55.382721+0000", 2024-03-11T21:44:26.331 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-03-11T21:44:26.331 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:44:26.331 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:44:26.331 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "throttled", 2024-03-11T21:44:26.331 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:34:55.382723+0000", 2024-03-11T21:44:26.331 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 2.1160000000000002e-06 2024-03-11T21:44:26.331 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:44:26.331 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:44:26.331 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "all_read", 2024-03-11T21:44:26.331 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:34:55.382729+0000", 2024-03-11T21:44:26.331 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 5.9240000000000004e-06 2024-03-11T21:44:26.331 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:44:26.331 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:44:26.332 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "dispatched", 2024-03-11T21:44:26.332 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:34:55.382742+0000", 2024-03-11T21:44:26.332 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.3161e-05 2024-03-11T21:44:26.332 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:44:26.332 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:44:26.332 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "queued_for_pg", 2024-03-11T21:44:26.332 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:34:55.382749+0000", 2024-03-11T21:44:26.332 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 6.9689999999999997e-06 2024-03-11T21:44:26.332 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:44:26.332 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:44:26.332 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-03-11T21:44:26.332 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:34:55.382797+0000", 2024-03-11T21:44:26.332 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 4.7277999999999997e-05 2024-03-11T21:44:26.332 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:44:26.332 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:44:26.333 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "waiting for peered", 2024-03-11T21:44:26.333 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:34:55.382804+0000", 2024-03-11T21:44:26.333 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 7.4800000000000004e-06 2024-03-11T21:44:26.333 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:44:26.333 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:44:26.333 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-03-11T21:44:26.333 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:34:56.358897+0000", 2024-03-11T21:44:26.333 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.97609253399999996 2024-03-11T21:44:26.333 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:44:26.333 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:44:26.333 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "started", 2024-03-11T21:44:26.333 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:34:56.358972+0000", 2024-03-11T21:44:26.333 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 7.4806000000000002e-05 2024-03-11T21:44:26.333 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:44:26.333 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:44:26.334 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "waiting for subops from 3,7", 2024-03-11T21:44:26.334 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:34:56.359055+0000", 2024-03-11T21:44:26.334 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 8.3790999999999998e-05 2024-03-11T21:44:26.334 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:44:26.334 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:44:26.334 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "op_commit", 2024-03-11T21:44:26.334 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:34:56.360293+0000", 2024-03-11T21:44:26.334 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.0012380970000000001 2024-03-11T21:44:26.334 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:44:26.334 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:44:26.334 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "sub_op_commit_rec", 2024-03-11T21:44:26.334 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:34:56.360502+0000", 2024-03-11T21:44:26.334 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.000208858 2024-03-11T21:44:26.335 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:44:26.335 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:44:26.335 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "sub_op_commit_rec", 2024-03-11T21:44:26.335 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:34:56.360563+0000", 2024-03-11T21:44:26.335 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 6.1173000000000005e-05 2024-03-11T21:44:26.335 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:44:26.335 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:44:26.335 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "commit_sent", 2024-03-11T21:44:26.335 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:34:56.360608+0000", 2024-03-11T21:44:26.335 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 4.4193999999999997e-05 2024-03-11T21:44:26.335 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:44:26.335 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:44:26.335 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "done", 2024-03-11T21:44:26.335 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:34:56.360644+0000", 2024-03-11T21:44:26.335 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 3.6560999999999998e-05 2024-03-11T21:44:26.336 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-03-11T21:44:26.336 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-03-11T21:44:26.336 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-03-11T21:44:26.336 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:44:26.336 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:44:26.336 INFO:teuthology.orchestra.run.smithi092.stdout: "description": "osd_op(client.15150.0:207239 8.c 8:3f0835ea:::benchmark_data_smithi092_104865_object86738:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e237)", 2024-03-11T21:44:26.336 INFO:teuthology.orchestra.run.smithi092.stdout: "initiated_at": "2024-03-11T21:34:55.465013+0000", 2024-03-11T21:44:26.336 INFO:teuthology.orchestra.run.smithi092.stdout: "age": 570.84782509299998, 2024-03-11T21:44:26.336 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.908045836, 2024-03-11T21:44:26.336 INFO:teuthology.orchestra.run.smithi092.stdout: "type_data": { 2024-03-11T21:44:26.336 INFO:teuthology.orchestra.run.smithi092.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-11T21:44:26.337 INFO:teuthology.orchestra.run.smithi092.stdout: "client_info": { 2024-03-11T21:44:26.337 INFO:teuthology.orchestra.run.smithi092.stdout: "client": "client.15150", 2024-03-11T21:44:26.337 INFO:teuthology.orchestra.run.smithi092.stdout: "client_addr": "v1:172.21.15.92:0/436606717", 2024-03-11T21:44:26.337 INFO:teuthology.orchestra.run.smithi092.stdout: "tid": 207239 2024-03-11T21:44:26.337 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:44:26.337 INFO:teuthology.orchestra.run.smithi092.stdout: "events": [ 2024-03-11T21:44:26.337 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:44:26.338 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "initiated", 2024-03-11T21:44:26.338 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:34:55.465013+0000", 2024-03-11T21:44:26.338 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-03-11T21:44:26.338 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:44:26.338 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:44:26.338 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "header_read", 2024-03-11T21:44:26.338 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:34:55.465013+0000", 2024-03-11T21:44:26.338 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-03-11T21:44:26.338 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:44:26.338 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:44:26.339 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "throttled", 2024-03-11T21:44:26.339 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:34:55.465016+0000", 2024-03-11T21:44:26.339 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 2.615e-06 2024-03-11T21:44:26.339 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:44:26.339 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:44:26.339 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "all_read", 2024-03-11T21:44:26.339 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:34:55.465021+0000", 2024-03-11T21:44:26.339 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 5.2889999999999997e-06 2024-03-11T21:44:26.340 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:44:26.340 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:44:26.340 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "dispatched", 2024-03-11T21:44:26.340 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:34:55.465036+0000", 2024-03-11T21:44:26.340 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.5023e-05 2024-03-11T21:44:26.340 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:44:26.340 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:44:26.340 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "queued_for_pg", 2024-03-11T21:44:26.340 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:34:55.465044+0000", 2024-03-11T21:44:26.340 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 7.9999999999999996e-06 2024-03-11T21:44:26.340 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:44:26.340 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:44:26.341 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-03-11T21:44:26.341 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:34:55.465100+0000", 2024-03-11T21:44:26.341 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 5.5337999999999998e-05 2024-03-11T21:44:26.341 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:44:26.341 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:44:26.341 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "waiting for peered", 2024-03-11T21:44:26.341 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:34:55.465108+0000", 2024-03-11T21:44:26.341 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 8.5259999999999993e-06 2024-03-11T21:44:26.341 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:44:26.341 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:44:26.341 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-03-11T21:44:26.341 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:34:56.360688+0000", 2024-03-11T21:44:26.341 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.89557976299999997 2024-03-11T21:44:26.341 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:44:26.341 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:44:26.342 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "started", 2024-03-11T21:44:26.342 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:34:56.360781+0000", 2024-03-11T21:44:26.342 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 9.3274000000000002e-05 2024-03-11T21:44:26.342 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:44:26.342 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:44:26.342 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "waiting for subops from 3,7", 2024-03-11T21:44:26.342 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:34:56.360842+0000", 2024-03-11T21:44:26.342 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 6.1334000000000006e-05 2024-03-11T21:44:26.342 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:44:26.342 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:44:26.342 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "sub_op_commit_rec", 2024-03-11T21:44:26.342 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:34:56.367917+0000", 2024-03-11T21:44:26.342 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.0070748219999999997 2024-03-11T21:44:26.342 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:44:26.343 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:44:26.343 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "sub_op_commit_rec", 2024-03-11T21:44:26.343 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:34:56.367996+0000", 2024-03-11T21:44:26.343 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 7.8614000000000003e-05 2024-03-11T21:44:26.343 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:44:26.343 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:44:26.343 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "op_commit", 2024-03-11T21:44:26.343 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:34:56.372955+0000", 2024-03-11T21:44:26.343 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.0049590809999999997 2024-03-11T21:44:26.343 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:44:26.343 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:44:26.343 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "commit_sent", 2024-03-11T21:44:26.343 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:34:56.373012+0000", 2024-03-11T21:44:26.344 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 5.7241000000000003e-05 2024-03-11T21:44:26.344 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:44:26.345 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:44:26.345 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "done", 2024-03-11T21:44:26.345 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:34:56.373059+0000", 2024-03-11T21:44:26.345 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 4.6916000000000003e-05 2024-03-11T21:44:26.345 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-03-11T21:44:26.345 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-03-11T21:44:26.345 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-03-11T21:44:26.345 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:44:26.345 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:44:26.345 INFO:teuthology.orchestra.run.smithi092.stdout: "description": "osd_op(client.15150.0:207246 8.c 8:36f7cf62:::benchmark_data_smithi092_104865_object86745:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e237)", 2024-03-11T21:44:26.345 INFO:teuthology.orchestra.run.smithi092.stdout: "initiated_at": "2024-03-11T21:34:55.472104+0000", 2024-03-11T21:44:26.345 INFO:teuthology.orchestra.run.smithi092.stdout: "age": 570.84073437999996, 2024-03-11T21:44:26.345 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.90107643000000004, 2024-03-11T21:44:26.345 INFO:teuthology.orchestra.run.smithi092.stdout: "type_data": { 2024-03-11T21:44:26.345 INFO:teuthology.orchestra.run.smithi092.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-11T21:44:26.346 INFO:teuthology.orchestra.run.smithi092.stdout: "client_info": { 2024-03-11T21:44:26.346 INFO:teuthology.orchestra.run.smithi092.stdout: "client": "client.15150", 2024-03-11T21:44:26.346 INFO:teuthology.orchestra.run.smithi092.stdout: "client_addr": "v1:172.21.15.92:0/436606717", 2024-03-11T21:44:26.346 INFO:teuthology.orchestra.run.smithi092.stdout: "tid": 207246 2024-03-11T21:44:26.346 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:44:26.346 INFO:teuthology.orchestra.run.smithi092.stdout: "events": [ 2024-03-11T21:44:26.346 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:44:26.346 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "initiated", 2024-03-11T21:44:26.346 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:34:55.472104+0000", 2024-03-11T21:44:26.346 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-03-11T21:44:26.346 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:44:26.346 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:44:26.346 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "header_read", 2024-03-11T21:44:26.347 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:34:55.472104+0000", 2024-03-11T21:44:26.347 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-03-11T21:44:26.347 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:44:26.347 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:44:26.347 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "throttled", 2024-03-11T21:44:26.347 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:34:55.472106+0000", 2024-03-11T21:44:26.347 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.9130000000000001e-06 2024-03-11T21:44:26.347 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:44:26.347 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:44:26.347 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "all_read", 2024-03-11T21:44:26.347 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:34:55.472110+0000", 2024-03-11T21:44:26.347 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 4.3769999999999998e-06 2024-03-11T21:44:26.347 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:44:26.347 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:44:26.347 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "dispatched", 2024-03-11T21:44:26.348 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:34:55.472120+0000", 2024-03-11T21:44:26.348 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.0010000000000001e-05 2024-03-11T21:44:26.348 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:44:26.348 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:44:26.348 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "queued_for_pg", 2024-03-11T21:44:26.348 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:34:55.472127+0000", 2024-03-11T21:44:26.348 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 6.8029999999999999e-06 2024-03-11T21:44:26.348 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:44:26.348 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:44:26.348 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-03-11T21:44:26.348 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:34:55.472219+0000", 2024-03-11T21:44:26.348 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 9.1914000000000001e-05 2024-03-11T21:44:26.348 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:44:26.349 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:44:26.349 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "waiting for peered", 2024-03-11T21:44:26.349 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:34:55.472250+0000", 2024-03-11T21:44:26.349 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 3.0528000000000003e-05 2024-03-11T21:44:26.349 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:44:26.349 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:44:26.349 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-03-11T21:44:26.349 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:34:56.362790+0000", 2024-03-11T21:44:26.349 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.89054004200000003 2024-03-11T21:44:26.349 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:44:26.349 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:44:26.349 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "started", 2024-03-11T21:44:26.349 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:34:56.362911+0000", 2024-03-11T21:44:26.349 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00012171999999999999 2024-03-11T21:44:26.350 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:44:26.350 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:44:26.350 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "waiting for subops from 3,7", 2024-03-11T21:44:26.350 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:34:56.363042+0000", 2024-03-11T21:44:26.350 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00013084699999999999 2024-03-11T21:44:26.350 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:44:26.350 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:44:26.350 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "sub_op_commit_rec", 2024-03-11T21:44:26.350 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:34:56.367956+0000", 2024-03-11T21:44:26.350 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.0049135979999999999 2024-03-11T21:44:26.350 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:44:26.350 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:44:26.350 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "sub_op_commit_rec", 2024-03-11T21:44:26.350 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:34:56.373103+0000", 2024-03-11T21:44:26.351 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.0051477320000000003 2024-03-11T21:44:26.351 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:44:26.351 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:44:26.351 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "op_commit", 2024-03-11T21:44:26.351 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:34:56.373117+0000", 2024-03-11T21:44:26.351 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.3678e-05 2024-03-11T21:44:26.351 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:44:26.351 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:44:26.351 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "commit_sent", 2024-03-11T21:44:26.351 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:34:56.373153+0000", 2024-03-11T21:44:26.351 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 3.5952999999999999e-05 2024-03-11T21:44:26.351 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:44:26.351 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:44:26.351 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "done", 2024-03-11T21:44:26.351 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:34:56.373180+0000", 2024-03-11T21:44:26.352 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 2.7314999999999999e-05 2024-03-11T21:44:26.352 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-03-11T21:44:26.352 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-03-11T21:44:26.352 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-03-11T21:44:26.352 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:44:26.352 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:44:26.352 INFO:teuthology.orchestra.run.smithi092.stdout: "description": "osd_op(client.15150.0:207247 8.c 8:3b696c26:::benchmark_data_smithi092_104865_object86746:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e237)", 2024-03-11T21:44:26.352 INFO:teuthology.orchestra.run.smithi092.stdout: "initiated_at": "2024-03-11T21:34:55.472328+0000", 2024-03-11T21:44:26.352 INFO:teuthology.orchestra.run.smithi092.stdout: "age": 570.84051081699999, 2024-03-11T21:44:26.352 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.90103873400000001, 2024-03-11T21:44:26.352 INFO:teuthology.orchestra.run.smithi092.stdout: "type_data": { 2024-03-11T21:44:26.352 INFO:teuthology.orchestra.run.smithi092.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-11T21:44:26.352 INFO:teuthology.orchestra.run.smithi092.stdout: "client_info": { 2024-03-11T21:44:26.352 INFO:teuthology.orchestra.run.smithi092.stdout: "client": "client.15150", 2024-03-11T21:44:26.352 INFO:teuthology.orchestra.run.smithi092.stdout: "client_addr": "v1:172.21.15.92:0/436606717", 2024-03-11T21:44:26.353 INFO:teuthology.orchestra.run.smithi092.stdout: "tid": 207247 2024-03-11T21:44:26.353 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:44:26.353 INFO:teuthology.orchestra.run.smithi092.stdout: "events": [ 2024-03-11T21:44:26.353 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:44:26.353 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "initiated", 2024-03-11T21:44:26.353 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:34:55.472328+0000", 2024-03-11T21:44:26.353 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-03-11T21:44:26.353 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:44:26.353 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:44:26.353 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "header_read", 2024-03-11T21:44:26.353 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:34:55.472328+0000", 2024-03-11T21:44:26.353 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-03-11T21:44:26.353 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:44:26.353 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:44:26.354 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "throttled", 2024-03-11T21:44:26.354 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:34:55.472329+0000", 2024-03-11T21:44:26.354 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.0020000000000001e-06 2024-03-11T21:44:26.354 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:44:26.354 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:44:26.354 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "all_read", 2024-03-11T21:44:26.354 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:34:55.472333+0000", 2024-03-11T21:44:26.354 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 4.0690000000000003e-06 2024-03-11T21:44:26.355 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:44:26.355 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:44:26.355 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "dispatched", 2024-03-11T21:44:26.355 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:34:55.472341+0000", 2024-03-11T21:44:26.355 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 8.3629999999999994e-06 2024-03-11T21:44:26.355 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:44:26.355 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:44:26.355 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "queued_for_pg", 2024-03-11T21:44:26.355 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:34:55.472347+0000", 2024-03-11T21:44:26.356 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 6.455e-06 2024-03-11T21:44:26.356 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:44:26.356 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:44:26.356 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-03-11T21:44:26.356 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:34:55.472409+0000", 2024-03-11T21:44:26.356 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 6.1939000000000004e-05 2024-03-11T21:44:26.356 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:44:26.356 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:44:26.356 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "waiting for peered", 2024-03-11T21:44:26.356 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:34:55.472418+0000", 2024-03-11T21:44:26.356 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 8.5299999999999996e-06 2024-03-11T21:44:26.356 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:44:26.356 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:44:26.356 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-03-11T21:44:26.356 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:34:56.365407+0000", 2024-03-11T21:44:26.357 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.89298929900000001 2024-03-11T21:44:26.357 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:44:26.357 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:44:26.357 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "started", 2024-03-11T21:44:26.357 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:34:56.365528+0000", 2024-03-11T21:44:26.357 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.000121049 2024-03-11T21:44:26.357 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:44:26.357 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:44:26.357 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "waiting for subops from 3,7", 2024-03-11T21:44:26.357 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:34:56.365636+0000", 2024-03-11T21:44:26.357 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.000107936 2024-03-11T21:44:26.357 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:44:26.357 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:44:26.357 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "sub_op_commit_rec", 2024-03-11T21:44:26.357 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:34:56.368039+0000", 2024-03-11T21:44:26.358 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.0024027969999999999 2024-03-11T21:44:26.358 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:44:26.358 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:44:26.358 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "op_commit", 2024-03-11T21:44:26.358 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:34:56.373189+0000", 2024-03-11T21:44:26.358 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.0051496520000000002 2024-03-11T21:44:26.358 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:44:26.358 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:44:26.358 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "sub_op_commit_rec", 2024-03-11T21:44:26.358 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:34:56.373254+0000", 2024-03-11T21:44:26.358 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 6.4906999999999994e-05 2024-03-11T21:44:26.358 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:44:26.358 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:44:26.359 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "commit_sent", 2024-03-11T21:44:26.359 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:34:56.373312+0000", 2024-03-11T21:44:26.359 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 5.8545000000000003e-05 2024-03-11T21:44:26.359 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:44:26.359 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:44:26.359 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "done", 2024-03-11T21:44:26.359 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:34:56.373366+0000", 2024-03-11T21:44:26.359 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 5.4190999999999997e-05 2024-03-11T21:44:26.359 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-03-11T21:44:26.359 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-03-11T21:44:26.359 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-03-11T21:44:26.359 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:44:26.359 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:44:26.360 INFO:teuthology.orchestra.run.smithi092.stdout: "description": "osd_op(client.15150.0:207254 8.c 8:33b1c360:::benchmark_data_smithi092_104865_object86753:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e237)", 2024-03-11T21:44:26.360 INFO:teuthology.orchestra.run.smithi092.stdout: "initiated_at": "2024-03-11T21:34:55.476305+0000", 2024-03-11T21:44:26.360 INFO:teuthology.orchestra.run.smithi092.stdout: "age": 570.83653355399997, 2024-03-11T21:44:26.360 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.897760224, 2024-03-11T21:44:26.360 INFO:teuthology.orchestra.run.smithi092.stdout: "type_data": { 2024-03-11T21:44:26.360 INFO:teuthology.orchestra.run.smithi092.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-11T21:44:26.360 INFO:teuthology.orchestra.run.smithi092.stdout: "client_info": { 2024-03-11T21:44:26.360 INFO:teuthology.orchestra.run.smithi092.stdout: "client": "client.15150", 2024-03-11T21:44:26.360 INFO:teuthology.orchestra.run.smithi092.stdout: "client_addr": "v1:172.21.15.92:0/436606717", 2024-03-11T21:44:26.360 INFO:teuthology.orchestra.run.smithi092.stdout: "tid": 207254 2024-03-11T21:44:26.360 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:44:26.360 INFO:teuthology.orchestra.run.smithi092.stdout: "events": [ 2024-03-11T21:44:26.360 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:44:26.361 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "initiated", 2024-03-11T21:44:26.361 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:34:55.476305+0000", 2024-03-11T21:44:26.361 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-03-11T21:44:26.361 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:44:26.361 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:44:26.361 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "header_read", 2024-03-11T21:44:26.361 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:34:55.476305+0000", 2024-03-11T21:44:26.361 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-03-11T21:44:26.361 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:44:26.361 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:44:26.361 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "throttled", 2024-03-11T21:44:26.362 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:34:55.476306+0000", 2024-03-11T21:44:26.362 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.491e-06 2024-03-11T21:44:26.362 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:44:26.362 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:44:26.362 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "all_read", 2024-03-11T21:44:26.362 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:34:55.476311+0000", 2024-03-11T21:44:26.362 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 4.6120000000000001e-06 2024-03-11T21:44:26.362 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:44:26.362 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:44:26.362 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "dispatched", 2024-03-11T21:44:26.362 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:34:55.476321+0000", 2024-03-11T21:44:26.362 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.0526999999999999e-05 2024-03-11T21:44:26.362 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:44:26.362 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:44:26.363 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "queued_for_pg", 2024-03-11T21:44:26.363 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:34:55.476329+0000", 2024-03-11T21:44:26.363 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 7.0899999999999999e-06 2024-03-11T21:44:26.363 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:44:26.363 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:44:26.363 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-03-11T21:44:26.363 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:34:55.476379+0000", 2024-03-11T21:44:26.363 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 5.0643999999999998e-05 2024-03-11T21:44:26.363 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:44:26.363 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:44:26.363 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "waiting for peered", 2024-03-11T21:44:26.363 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:34:55.476389+0000", 2024-03-11T21:44:26.363 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 9.465e-06 2024-03-11T21:44:26.364 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:44:26.364 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:44:26.364 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-03-11T21:44:26.364 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:34:56.368072+0000", 2024-03-11T21:44:26.364 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.89168363299999998 2024-03-11T21:44:26.364 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:44:26.364 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:44:26.364 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "started", 2024-03-11T21:44:26.364 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:34:56.368172+0000", 2024-03-11T21:44:26.364 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 9.9549000000000005e-05 2024-03-11T21:44:26.364 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:44:26.364 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:44:26.364 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "waiting for subops from 3,7", 2024-03-11T21:44:26.364 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:34:56.368536+0000", 2024-03-11T21:44:26.365 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00036459799999999999 2024-03-11T21:44:26.365 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:44:26.365 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:44:26.365 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "op_commit", 2024-03-11T21:44:26.365 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:34:56.373195+0000", 2024-03-11T21:44:26.365 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.0046583249999999996 2024-03-11T21:44:26.365 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:44:26.365 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:44:26.365 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "sub_op_commit_rec", 2024-03-11T21:44:26.365 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:34:56.373416+0000", 2024-03-11T21:44:26.365 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.000220955 2024-03-11T21:44:26.366 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:44:26.366 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:44:26.366 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "sub_op_commit_rec", 2024-03-11T21:44:26.366 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:34:56.373948+0000", 2024-03-11T21:44:26.366 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00053270499999999996 2024-03-11T21:44:26.366 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:44:26.366 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:44:26.366 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "commit_sent", 2024-03-11T21:44:26.366 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:34:56.374022+0000", 2024-03-11T21:44:26.366 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 7.3209999999999999e-05 2024-03-11T21:44:26.366 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:44:26.366 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:44:26.366 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "done", 2024-03-11T21:44:26.367 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:34:56.374065+0000", 2024-03-11T21:44:26.367 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 4.3420000000000001e-05 2024-03-11T21:44:26.367 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-03-11T21:44:26.367 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-03-11T21:44:26.367 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-03-11T21:44:26.367 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:44:26.368 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:44:26.368 INFO:teuthology.orchestra.run.smithi092.stdout: "description": "osd_op(client.15150.0:207261 8.c 8:3bacfc79:::benchmark_data_smithi092_104865_object86760:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e237)", 2024-03-11T21:44:26.368 INFO:teuthology.orchestra.run.smithi092.stdout: "initiated_at": "2024-03-11T21:34:55.481650+0000", 2024-03-11T21:44:26.368 INFO:teuthology.orchestra.run.smithi092.stdout: "age": 570.83118881400003, 2024-03-11T21:44:26.368 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.90050556400000004, 2024-03-11T21:44:26.368 INFO:teuthology.orchestra.run.smithi092.stdout: "type_data": { 2024-03-11T21:44:26.368 INFO:teuthology.orchestra.run.smithi092.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-11T21:44:26.369 INFO:teuthology.orchestra.run.smithi092.stdout: "client_info": { 2024-03-11T21:44:26.369 INFO:teuthology.orchestra.run.smithi092.stdout: "client": "client.15150", 2024-03-11T21:44:26.369 INFO:teuthology.orchestra.run.smithi092.stdout: "client_addr": "v1:172.21.15.92:0/436606717", 2024-03-11T21:44:26.369 INFO:teuthology.orchestra.run.smithi092.stdout: "tid": 207261 2024-03-11T21:44:26.369 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:44:26.369 INFO:teuthology.orchestra.run.smithi092.stdout: "events": [ 2024-03-11T21:44:26.370 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:44:26.370 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "initiated", 2024-03-11T21:44:26.370 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:34:55.481650+0000", 2024-03-11T21:44:26.370 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-03-11T21:44:26.370 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:44:26.370 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:44:26.370 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "header_read", 2024-03-11T21:44:26.371 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:34:55.481650+0000", 2024-03-11T21:44:26.371 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-03-11T21:44:26.371 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:44:26.371 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:44:26.371 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "throttled", 2024-03-11T21:44:26.371 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:34:55.481652+0000", 2024-03-11T21:44:26.372 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 2.069e-06 2024-03-11T21:44:26.372 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:44:26.372 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:44:26.372 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "all_read", 2024-03-11T21:44:26.372 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:34:55.481659+0000", 2024-03-11T21:44:26.372 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 6.9480000000000002e-06 2024-03-11T21:44:26.372 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:44:26.373 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:44:26.373 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "dispatched", 2024-03-11T21:44:26.373 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:34:55.481675+0000", 2024-03-11T21:44:26.373 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.6773e-05 2024-03-11T21:44:26.373 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:44:26.373 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:44:26.373 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "queued_for_pg", 2024-03-11T21:44:26.374 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:34:55.481685+0000", 2024-03-11T21:44:26.374 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 9.9839999999999996e-06 2024-03-11T21:44:26.374 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:44:26.374 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:44:26.374 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-03-11T21:44:26.374 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:34:55.481744+0000", 2024-03-11T21:44:26.375 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 5.8618e-05 2024-03-11T21:44:26.375 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:44:26.375 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:44:26.375 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "waiting for peered", 2024-03-11T21:44:26.375 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:34:55.481752+0000", 2024-03-11T21:44:26.375 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 8.0309999999999998e-06 2024-03-11T21:44:26.375 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:44:26.376 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:44:26.376 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-03-11T21:44:26.376 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:34:56.370719+0000", 2024-03-11T21:44:26.376 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.88896741000000001 2024-03-11T21:44:26.376 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:44:26.376 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:44:26.377 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "started", 2024-03-11T21:44:26.377 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:34:56.370828+0000", 2024-03-11T21:44:26.377 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00010908600000000001 2024-03-11T21:44:26.377 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:44:26.377 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:44:26.377 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "waiting for subops from 3,7", 2024-03-11T21:44:26.378 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:34:56.370943+0000", 2024-03-11T21:44:26.378 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.000114456 2024-03-11T21:44:26.378 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:44:26.378 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:44:26.378 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "op_commit", 2024-03-11T21:44:26.378 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:34:56.374079+0000", 2024-03-11T21:44:26.378 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.0031359790000000001 2024-03-11T21:44:26.379 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:44:26.379 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:44:26.379 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "sub_op_commit_rec", 2024-03-11T21:44:26.379 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:34:56.374111+0000", 2024-03-11T21:44:26.379 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 3.2277999999999998e-05 2024-03-11T21:44:26.379 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:44:26.380 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:44:26.380 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "sub_op_commit_rec", 2024-03-11T21:44:26.380 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:34:56.382056+0000", 2024-03-11T21:44:26.380 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.0079444860000000006 2024-03-11T21:44:26.380 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:44:26.380 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:44:26.381 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "commit_sent", 2024-03-11T21:44:26.381 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:34:56.382114+0000", 2024-03-11T21:44:26.381 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 5.8307999999999998e-05 2024-03-11T21:44:26.381 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:44:26.381 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:44:26.381 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "done", 2024-03-11T21:44:26.382 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:34:56.382155+0000", 2024-03-11T21:44:26.382 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 4.1137999999999998e-05 2024-03-11T21:44:26.382 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-03-11T21:44:26.382 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-03-11T21:44:26.385 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-03-11T21:44:26.385 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:44:26.385 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:44:26.386 INFO:teuthology.orchestra.run.smithi092.stdout: "description": "osd_op(client.15150.0:207274 8.c 8:35340fad:::benchmark_data_smithi092_104865_object86773:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e237)", 2024-03-11T21:44:26.386 INFO:teuthology.orchestra.run.smithi092.stdout: "initiated_at": "2024-03-11T21:34:55.489338+0000", 2024-03-11T21:44:26.386 INFO:teuthology.orchestra.run.smithi092.stdout: "age": 570.82350034700005, 2024-03-11T21:44:26.386 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.89598779799999995, 2024-03-11T21:44:26.386 INFO:teuthology.orchestra.run.smithi092.stdout: "type_data": { 2024-03-11T21:44:26.386 INFO:teuthology.orchestra.run.smithi092.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-11T21:44:26.387 INFO:teuthology.orchestra.run.smithi092.stdout: "client_info": { 2024-03-11T21:44:26.387 INFO:teuthology.orchestra.run.smithi092.stdout: "client": "client.15150", 2024-03-11T21:44:26.387 INFO:teuthology.orchestra.run.smithi092.stdout: "client_addr": "v1:172.21.15.92:0/436606717", 2024-03-11T21:44:26.387 INFO:teuthology.orchestra.run.smithi092.stdout: "tid": 207274 2024-03-11T21:44:26.387 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:44:26.388 INFO:teuthology.orchestra.run.smithi092.stdout: "events": [ 2024-03-11T21:44:26.388 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:44:26.388 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "initiated", 2024-03-11T21:44:26.388 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:34:55.489338+0000", 2024-03-11T21:44:26.388 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-03-11T21:44:26.388 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:44:26.389 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:44:26.389 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "header_read", 2024-03-11T21:44:26.389 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:34:55.489338+0000", 2024-03-11T21:44:26.389 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-03-11T21:44:26.389 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:44:26.390 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:44:26.390 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "throttled", 2024-03-11T21:44:26.390 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:34:55.489339+0000", 2024-03-11T21:44:26.390 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.2559999999999999e-06 2024-03-11T21:44:26.390 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:44:26.390 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:44:26.391 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "all_read", 2024-03-11T21:44:26.391 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:34:55.489344+0000", 2024-03-11T21:44:26.391 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 4.7049999999999998e-06 2024-03-11T21:44:26.391 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:44:26.391 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:44:26.391 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "dispatched", 2024-03-11T21:44:26.391 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:34:55.489353+0000", 2024-03-11T21:44:26.391 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 9.1419999999999999e-06 2024-03-11T21:44:26.391 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:44:26.392 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:44:26.392 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "queued_for_pg", 2024-03-11T21:44:26.392 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:34:55.489386+0000", 2024-03-11T21:44:26.392 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 3.2929999999999998e-05 2024-03-11T21:44:26.392 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:44:26.392 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:44:26.392 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-03-11T21:44:26.392 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:34:55.489435+0000", 2024-03-11T21:44:26.392 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 4.8609000000000002e-05 2024-03-11T21:44:26.392 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:44:26.393 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:44:26.393 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "waiting for peered", 2024-03-11T21:44:26.393 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:34:55.489443+0000", 2024-03-11T21:44:26.393 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 8.3580000000000003e-06 2024-03-11T21:44:26.393 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:44:26.393 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:44:26.393 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-03-11T21:44:26.393 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:34:56.374137+0000", 2024-03-11T21:44:26.393 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.884694325 2024-03-11T21:44:26.393 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:44:26.393 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:44:26.393 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "started", 2024-03-11T21:44:26.393 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:34:56.374240+0000", 2024-03-11T21:44:26.393 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.000102196 2024-03-11T21:44:26.394 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:44:26.394 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:44:26.394 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "waiting for subops from 3,7", 2024-03-11T21:44:26.394 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:34:56.374354+0000", 2024-03-11T21:44:26.394 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.000114065 2024-03-11T21:44:26.394 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:44:26.394 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:44:26.394 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "sub_op_commit_rec", 2024-03-11T21:44:26.394 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:34:56.382018+0000", 2024-03-11T21:44:26.394 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.0076644570000000004 2024-03-11T21:44:26.394 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:44:26.394 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:44:26.394 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "op_commit", 2024-03-11T21:44:26.394 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:34:56.385190+0000", 2024-03-11T21:44:26.395 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.0031721290000000001 2024-03-11T21:44:26.395 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:44:26.395 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:44:26.395 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "sub_op_commit_rec", 2024-03-11T21:44:26.395 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:34:56.385231+0000", 2024-03-11T21:44:26.395 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 4.0587000000000003e-05 2024-03-11T21:44:26.395 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:44:26.395 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:44:26.395 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "commit_sent", 2024-03-11T21:44:26.395 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:34:56.385282+0000", 2024-03-11T21:44:26.395 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 5.0899000000000002e-05 2024-03-11T21:44:26.395 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:44:26.395 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:44:26.395 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "done", 2024-03-11T21:44:26.395 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:34:56.385326+0000", 2024-03-11T21:44:26.396 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 4.4140000000000001e-05 2024-03-11T21:44:26.396 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-03-11T21:44:26.396 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-03-11T21:44:26.396 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-03-11T21:44:26.396 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:44:26.396 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:44:26.396 INFO:teuthology.orchestra.run.smithi092.stdout: "description": "osd_op(client.15150.0:207275 8.c 8:3a3c7648:::benchmark_data_smithi092_104865_object86774:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e237)", 2024-03-11T21:44:26.396 INFO:teuthology.orchestra.run.smithi092.stdout: "initiated_at": "2024-03-11T21:34:55.489720+0000", 2024-03-11T21:44:26.396 INFO:teuthology.orchestra.run.smithi092.stdout: "age": 570.82311794300006, 2024-03-11T21:44:26.396 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.89580858900000004, 2024-03-11T21:44:26.396 INFO:teuthology.orchestra.run.smithi092.stdout: "type_data": { 2024-03-11T21:44:26.396 INFO:teuthology.orchestra.run.smithi092.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-11T21:44:26.396 INFO:teuthology.orchestra.run.smithi092.stdout: "client_info": { 2024-03-11T21:44:26.396 INFO:teuthology.orchestra.run.smithi092.stdout: "client": "client.15150", 2024-03-11T21:44:26.396 INFO:teuthology.orchestra.run.smithi092.stdout: "client_addr": "v1:172.21.15.92:0/436606717", 2024-03-11T21:44:26.397 INFO:teuthology.orchestra.run.smithi092.stdout: "tid": 207275 2024-03-11T21:44:26.397 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:44:26.397 INFO:teuthology.orchestra.run.smithi092.stdout: "events": [ 2024-03-11T21:44:26.397 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:44:26.397 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "initiated", 2024-03-11T21:44:26.397 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:34:55.489720+0000", 2024-03-11T21:44:26.397 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-03-11T21:44:26.397 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:44:26.397 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:44:26.397 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "header_read", 2024-03-11T21:44:26.397 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:34:55.489720+0000", 2024-03-11T21:44:26.397 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-03-11T21:44:26.397 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:44:26.397 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:44:26.397 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "throttled", 2024-03-11T21:44:26.398 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:34:55.489722+0000", 2024-03-11T21:44:26.398 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.1370000000000001e-06 2024-03-11T21:44:26.398 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:44:26.398 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:44:26.398 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "all_read", 2024-03-11T21:44:26.398 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:34:55.489726+0000", 2024-03-11T21:44:26.398 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 4.6129999999999997e-06 2024-03-11T21:44:26.398 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:44:26.398 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:44:26.398 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "dispatched", 2024-03-11T21:44:26.398 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:34:55.489735+0000", 2024-03-11T21:44:26.398 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 9.3160000000000003e-06 2024-03-11T21:44:26.398 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:44:26.398 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:44:26.399 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "queued_for_pg", 2024-03-11T21:44:26.399 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:34:55.489742+0000", 2024-03-11T21:44:26.399 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 6.9720000000000003e-06 2024-03-11T21:44:26.399 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:44:26.399 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:44:26.399 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-03-11T21:44:26.399 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:34:55.489795+0000", 2024-03-11T21:44:26.399 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 5.2796000000000001e-05 2024-03-11T21:44:26.399 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:44:26.399 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:44:26.399 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "waiting for peered", 2024-03-11T21:44:26.399 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:34:55.489802+0000", 2024-03-11T21:44:26.399 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 7.2230000000000001e-06 2024-03-11T21:44:26.399 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:44:26.399 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:44:26.400 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-03-11T21:44:26.400 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:34:56.376733+0000", 2024-03-11T21:44:26.400 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.88693052100000003 2024-03-11T21:44:26.400 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:44:26.400 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:44:26.400 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "started", 2024-03-11T21:44:26.400 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:34:56.376845+0000", 2024-03-11T21:44:26.400 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.000111606 2024-03-11T21:44:26.400 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:44:26.400 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:44:26.400 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "waiting for subops from 3,7", 2024-03-11T21:44:26.400 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:34:56.376959+0000", 2024-03-11T21:44:26.400 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.000114636 2024-03-11T21:44:26.400 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:44:26.401 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:44:26.401 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "sub_op_commit_rec", 2024-03-11T21:44:26.401 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:34:56.382200+0000", 2024-03-11T21:44:26.401 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.0052405400000000001 2024-03-11T21:44:26.401 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:44:26.401 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:44:26.401 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "op_commit", 2024-03-11T21:44:26.401 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:34:56.385339+0000", 2024-03-11T21:44:26.401 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.0031393789999999999 2024-03-11T21:44:26.401 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:44:26.401 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:44:26.402 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:44:26.402 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:44:26.403 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:44:26.403 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "sub_op_commit_rec", 2024-03-11T21:44:26.404 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:34:56.385434+0000", 2024-03-11T21:44:26.404 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 9.4402999999999997e-05 2024-03-11T21:44:26.404 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:44:26.404 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:44:26.404 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "commit_sent", 2024-03-11T21:44:26.404 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:34:56.385491+0000", 2024-03-11T21:44:26.404 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 5.7373000000000001e-05 2024-03-11T21:44:26.404 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:44:26.404 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:44:26.404 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "done", 2024-03-11T21:44:26.404 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:34:56.385529+0000", 2024-03-11T21:44:26.404 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 3.8074000000000002e-05 2024-03-11T21:44:26.404 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-03-11T21:44:26.404 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-03-11T21:44:26.405 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-03-11T21:44:26.405 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:44:26.405 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:44:26.405 INFO:teuthology.orchestra.run.smithi092.stdout: "description": "osd_op(client.25282.0:76299 9.8 9:17331e66:::benchmark_data_smithi092_130291_object76298:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e278)", 2024-03-11T21:44:26.405 INFO:teuthology.orchestra.run.smithi092.stdout: "initiated_at": "2024-03-11T21:36:35.048550+0000", 2024-03-11T21:44:26.405 INFO:teuthology.orchestra.run.smithi092.stdout: "age": 471.26428874499999, 2024-03-11T21:44:26.405 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.4852829970000001, 2024-03-11T21:44:26.405 INFO:teuthology.orchestra.run.smithi092.stdout: "type_data": { 2024-03-11T21:44:26.405 INFO:teuthology.orchestra.run.smithi092.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-11T21:44:26.405 INFO:teuthology.orchestra.run.smithi092.stdout: "client_info": { 2024-03-11T21:44:26.405 INFO:teuthology.orchestra.run.smithi092.stdout: "client": "client.25282", 2024-03-11T21:44:26.405 INFO:teuthology.orchestra.run.smithi092.stdout: "client_addr": "v1:172.21.15.92:0/2489519821", 2024-03-11T21:44:26.405 INFO:teuthology.orchestra.run.smithi092.stdout: "tid": 76299 2024-03-11T21:44:26.406 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:44:26.406 INFO:teuthology.orchestra.run.smithi092.stdout: "events": [ 2024-03-11T21:44:26.406 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:44:26.406 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "initiated", 2024-03-11T21:44:26.406 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:36:35.048550+0000", 2024-03-11T21:44:26.406 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-03-11T21:44:26.406 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:44:26.406 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:44:26.406 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "header_read", 2024-03-11T21:44:26.406 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:36:35.048550+0000", 2024-03-11T21:44:26.406 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-03-11T21:44:26.406 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:44:26.406 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:44:26.406 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "throttled", 2024-03-11T21:44:26.406 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:36:35.048552+0000", 2024-03-11T21:44:26.407 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 2.0710000000000002e-06 2024-03-11T21:44:26.407 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:44:26.407 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:44:26.407 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "all_read", 2024-03-11T21:44:26.407 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:36:35.048584+0000", 2024-03-11T21:44:26.407 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 3.2676999999999999e-05 2024-03-11T21:44:26.407 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:44:26.407 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:44:26.407 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "dispatched", 2024-03-11T21:44:26.407 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:36:35.048597+0000", 2024-03-11T21:44:26.407 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.2845e-05 2024-03-11T21:44:26.407 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:44:26.407 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:44:26.407 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "queued_for_pg", 2024-03-11T21:44:26.407 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:36:35.048608+0000", 2024-03-11T21:44:26.408 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.1005000000000001e-05 2024-03-11T21:44:26.408 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:44:26.408 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:44:26.408 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-03-11T21:44:26.408 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:36:35.049475+0000", 2024-03-11T21:44:26.408 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00086655599999999999 2024-03-11T21:44:26.408 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:44:26.408 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:44:26.408 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "waiting for peered", 2024-03-11T21:44:26.408 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:36:35.049486+0000", 2024-03-11T21:44:26.408 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.0842000000000001e-05 2024-03-11T21:44:26.408 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:44:26.408 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:44:26.408 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-03-11T21:44:26.409 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:36:36.516700+0000", 2024-03-11T21:44:26.409 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.467214099 2024-03-11T21:44:26.409 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:44:26.409 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:44:26.409 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "started", 2024-03-11T21:44:26.409 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:36:36.516847+0000", 2024-03-11T21:44:26.409 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00014753699999999999 2024-03-11T21:44:26.409 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:44:26.409 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:44:26.409 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "waiting for subops from 3,5", 2024-03-11T21:44:26.409 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:36:36.517015+0000", 2024-03-11T21:44:26.409 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00016770700000000001 2024-03-11T21:44:26.409 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:44:26.409 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:44:26.409 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "sub_op_commit_rec", 2024-03-11T21:44:26.410 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:36:36.527882+0000", 2024-03-11T21:44:26.410 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.010866607 2024-03-11T21:44:26.410 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:44:26.410 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:44:26.410 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "sub_op_commit_rec", 2024-03-11T21:44:26.410 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:36:36.527930+0000", 2024-03-11T21:44:26.410 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 4.8575000000000002e-05 2024-03-11T21:44:26.410 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:44:26.410 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:44:26.410 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "op_commit", 2024-03-11T21:44:26.410 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:36:36.530142+0000", 2024-03-11T21:44:26.410 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.002212206 2024-03-11T21:44:26.410 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:44:26.410 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:44:26.410 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "commit_sent", 2024-03-11T21:44:26.411 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:36:36.530209+0000", 2024-03-11T21:44:26.411 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 6.6594999999999995e-05 2024-03-11T21:44:26.411 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:44:26.411 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:44:26.411 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "done", 2024-03-11T21:44:26.411 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:36:36.533833+0000", 2024-03-11T21:44:26.411 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.0036236749999999998 2024-03-11T21:44:26.411 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-03-11T21:44:26.411 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-03-11T21:44:26.411 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-03-11T21:44:26.411 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:44:26.411 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:44:26.411 INFO:teuthology.orchestra.run.smithi092.stdout: "description": "osd_op(client.25282.0:76301 9.8 9:1db40ce0:::benchmark_data_smithi092_130291_object76300:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e278)", 2024-03-11T21:44:26.411 INFO:teuthology.orchestra.run.smithi092.stdout: "initiated_at": "2024-03-11T21:36:35.048631+0000", 2024-03-11T21:44:26.411 INFO:teuthology.orchestra.run.smithi092.stdout: "age": 471.26420776700002, 2024-03-11T21:44:26.411 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.495054493, 2024-03-11T21:44:26.412 INFO:teuthology.orchestra.run.smithi092.stdout: "type_data": { 2024-03-11T21:44:26.412 INFO:teuthology.orchestra.run.smithi092.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-11T21:44:26.412 INFO:teuthology.orchestra.run.smithi092.stdout: "client_info": { 2024-03-11T21:44:26.412 INFO:teuthology.orchestra.run.smithi092.stdout: "client": "client.25282", 2024-03-11T21:44:26.412 INFO:teuthology.orchestra.run.smithi092.stdout: "client_addr": "v1:172.21.15.92:0/2489519821", 2024-03-11T21:44:26.412 INFO:teuthology.orchestra.run.smithi092.stdout: "tid": 76301 2024-03-11T21:44:26.412 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:44:26.412 INFO:teuthology.orchestra.run.smithi092.stdout: "events": [ 2024-03-11T21:44:26.412 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:44:26.412 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "initiated", 2024-03-11T21:44:26.412 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:36:35.048631+0000", 2024-03-11T21:44:26.412 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-03-11T21:44:26.412 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:44:26.412 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:44:26.412 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "header_read", 2024-03-11T21:44:26.413 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:36:35.048631+0000", 2024-03-11T21:44:26.413 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-03-11T21:44:26.413 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:44:26.413 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:44:26.413 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "throttled", 2024-03-11T21:44:26.413 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:36:35.048632+0000", 2024-03-11T21:44:26.413 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.1170000000000001e-06 2024-03-11T21:44:26.413 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:44:26.413 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:44:26.413 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "all_read", 2024-03-11T21:44:26.413 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:36:35.048660+0000", 2024-03-11T21:44:26.413 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 2.8243e-05 2024-03-11T21:44:26.413 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:44:26.413 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:44:26.413 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "dispatched", 2024-03-11T21:44:26.413 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:36:35.048669+0000", 2024-03-11T21:44:26.414 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 8.7579999999999998e-06 2024-03-11T21:44:26.414 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:44:26.414 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:44:26.414 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "queued_for_pg", 2024-03-11T21:44:26.414 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:36:35.048697+0000", 2024-03-11T21:44:26.414 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 2.8138999999999999e-05 2024-03-11T21:44:26.414 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:44:26.414 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:44:26.414 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-03-11T21:44:26.414 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:36:35.049509+0000", 2024-03-11T21:44:26.414 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00081172500000000001 2024-03-11T21:44:26.414 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:44:26.414 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:44:26.414 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "waiting for peered", 2024-03-11T21:44:26.414 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:36:35.049515+0000", 2024-03-11T21:44:26.415 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 5.9909999999999999e-06 2024-03-11T21:44:26.415 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:44:26.415 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:44:26.415 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-03-11T21:44:26.415 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:36:36.523893+0000", 2024-03-11T21:44:26.415 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.4743779799999999 2024-03-11T21:44:26.415 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:44:26.415 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:44:26.415 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "started", 2024-03-11T21:44:26.415 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:36:36.524187+0000", 2024-03-11T21:44:26.415 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00029435900000000001 2024-03-11T21:44:26.415 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:44:26.415 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:44:26.415 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "waiting for subops from 3,5", 2024-03-11T21:44:26.415 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:36:36.524376+0000", 2024-03-11T21:44:26.416 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00018891599999999999 2024-03-11T21:44:26.416 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:44:26.416 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:44:26.416 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "sub_op_commit_rec", 2024-03-11T21:44:26.416 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:36:36.530124+0000", 2024-03-11T21:44:26.416 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.0057478800000000004 2024-03-11T21:44:26.416 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:44:26.416 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:44:26.416 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "op_commit", 2024-03-11T21:44:26.416 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:36:36.533846+0000", 2024-03-11T21:44:26.416 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.0037218709999999999 2024-03-11T21:44:26.416 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:44:26.416 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:44:26.416 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "sub_op_commit_rec", 2024-03-11T21:44:26.416 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:36:36.538924+0000", 2024-03-11T21:44:26.417 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.0050780479999999999 2024-03-11T21:44:26.417 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:44:26.417 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:44:26.417 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "commit_sent", 2024-03-11T21:44:26.417 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:36:36.539008+0000", 2024-03-11T21:44:26.417 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 8.4528000000000001e-05 2024-03-11T21:44:26.417 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:44:26.417 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:44:26.417 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "done", 2024-03-11T21:44:26.417 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:36:36.543685+0000", 2024-03-11T21:44:26.417 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.0046769380000000003 2024-03-11T21:44:26.417 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-03-11T21:44:26.417 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-03-11T21:44:26.417 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-03-11T21:44:26.418 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:44:26.418 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:44:26.418 INFO:teuthology.orchestra.run.smithi092.stdout: "description": "osd_op(client.15744.0:69283 10.18 10:1a981db8:::benchmark_data_smithi092_161462_object69282:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e372)", 2024-03-11T21:44:26.418 INFO:teuthology.orchestra.run.smithi092.stdout: "initiated_at": "2024-03-11T21:41:27.826441+0000", 2024-03-11T21:44:26.418 INFO:teuthology.orchestra.run.smithi092.stdout: "age": 178.48639740499999, 2024-03-11T21:44:26.418 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.0249358310000001, 2024-03-11T21:44:26.418 INFO:teuthology.orchestra.run.smithi092.stdout: "type_data": { 2024-03-11T21:44:26.418 INFO:teuthology.orchestra.run.smithi092.stdout: "flag_point": "reached pg", 2024-03-11T21:44:26.418 INFO:teuthology.orchestra.run.smithi092.stdout: "client_info": { 2024-03-11T21:44:26.418 INFO:teuthology.orchestra.run.smithi092.stdout: "client": "client.15744", 2024-03-11T21:44:26.418 INFO:teuthology.orchestra.run.smithi092.stdout: "client_addr": "v1:172.21.15.92:0/3334000812", 2024-03-11T21:44:26.418 INFO:teuthology.orchestra.run.smithi092.stdout: "tid": 69283 2024-03-11T21:44:26.418 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:44:26.419 INFO:teuthology.orchestra.run.smithi092.stdout: "events": [ 2024-03-11T21:44:26.419 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:44:26.419 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "initiated", 2024-03-11T21:44:26.419 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:41:27.826441+0000", 2024-03-11T21:44:26.419 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-03-11T21:44:26.419 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:44:26.419 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:44:26.419 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "header_read", 2024-03-11T21:44:26.419 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:41:27.826441+0000", 2024-03-11T21:44:26.419 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-03-11T21:44:26.419 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:44:26.419 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:44:26.419 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "throttled", 2024-03-11T21:44:26.419 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:41:27.826444+0000", 2024-03-11T21:44:26.419 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 2.8310000000000001e-06 2024-03-11T21:44:26.420 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:44:26.420 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:44:26.420 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "all_read", 2024-03-11T21:44:26.420 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:41:27.826475+0000", 2024-03-11T21:44:26.420 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 3.1402000000000002e-05 2024-03-11T21:44:26.420 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:44:26.420 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:44:26.420 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "dispatched", 2024-03-11T21:44:26.420 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:41:27.826519+0000", 2024-03-11T21:44:26.420 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 4.4168000000000001e-05 2024-03-11T21:44:26.420 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:44:26.420 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:44:26.420 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "queued_for_pg", 2024-03-11T21:44:26.420 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:41:27.826601+0000", 2024-03-11T21:44:26.421 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 8.1167000000000004e-05 2024-03-11T21:44:26.421 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:44:26.421 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:44:26.421 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-03-11T21:44:26.421 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:41:28.014594+0000", 2024-03-11T21:44:26.421 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.18799348299999999 2024-03-11T21:44:26.421 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:44:26.421 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:44:26.421 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "waiting for peered", 2024-03-11T21:44:26.421 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:41:28.014602+0000", 2024-03-11T21:44:26.421 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 8.4770000000000003e-06 2024-03-11T21:44:26.421 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:44:26.421 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:44:26.421 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-03-11T21:44:26.421 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:41:28.851313+0000", 2024-03-11T21:44:26.422 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.83671013999999999 2024-03-11T21:44:26.422 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:44:26.422 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:44:26.422 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "done", 2024-03-11T21:44:26.422 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:41:28.851377+0000", 2024-03-11T21:44:26.422 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 6.4163000000000002e-05 2024-03-11T21:44:26.422 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-03-11T21:44:26.422 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-03-11T21:44:26.422 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-03-11T21:44:26.422 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:44:26.422 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:44:26.422 INFO:teuthology.orchestra.run.smithi092.stdout: "description": "osd_op(client.15744.0:69285 10.14 10:299b39d9:::benchmark_data_smithi092_161462_object69284:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e372)", 2024-03-11T21:44:26.422 INFO:teuthology.orchestra.run.smithi092.stdout: "initiated_at": "2024-03-11T21:41:27.826624+0000", 2024-03-11T21:44:26.422 INFO:teuthology.orchestra.run.smithi092.stdout: "age": 178.48621473399999, 2024-03-11T21:44:26.423 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.028890876, 2024-03-11T21:44:26.423 INFO:teuthology.orchestra.run.smithi092.stdout: "type_data": { 2024-03-11T21:44:26.423 INFO:teuthology.orchestra.run.smithi092.stdout: "flag_point": "reached pg", 2024-03-11T21:44:26.423 INFO:teuthology.orchestra.run.smithi092.stdout: "client_info": { 2024-03-11T21:44:26.423 INFO:teuthology.orchestra.run.smithi092.stdout: "client": "client.15744", 2024-03-11T21:44:26.423 INFO:teuthology.orchestra.run.smithi092.stdout: "client_addr": "v1:172.21.15.92:0/3334000812", 2024-03-11T21:44:26.423 INFO:teuthology.orchestra.run.smithi092.stdout: "tid": 69285 2024-03-11T21:44:26.423 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:44:26.423 INFO:teuthology.orchestra.run.smithi092.stdout: "events": [ 2024-03-11T21:44:26.423 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:44:26.423 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "initiated", 2024-03-11T21:44:26.423 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:41:27.826624+0000", 2024-03-11T21:44:26.423 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-03-11T21:44:26.423 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:44:26.423 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:44:26.424 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "header_read", 2024-03-11T21:44:26.424 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:41:27.826624+0000", 2024-03-11T21:44:26.424 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-03-11T21:44:26.424 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:44:26.424 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:44:26.424 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "throttled", 2024-03-11T21:44:26.424 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:41:27.826625+0000", 2024-03-11T21:44:26.424 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.04e-06 2024-03-11T21:44:26.424 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:44:26.424 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:44:26.424 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "all_read", 2024-03-11T21:44:26.424 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:41:27.826649+0000", 2024-03-11T21:44:26.424 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 2.4522999999999999e-05 2024-03-11T21:44:26.424 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:44:26.424 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:44:26.425 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "dispatched", 2024-03-11T21:44:26.425 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:41:27.826658+0000", 2024-03-11T21:44:26.425 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 8.7010000000000002e-06 2024-03-11T21:44:26.425 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:44:26.425 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:44:26.425 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "queued_for_pg", 2024-03-11T21:44:26.425 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:41:27.826665+0000", 2024-03-11T21:44:26.425 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 7.4619999999999999e-06 2024-03-11T21:44:26.425 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:44:26.425 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:44:26.425 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-03-11T21:44:26.425 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:41:28.023946+0000", 2024-03-11T21:44:26.425 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.19728109699999999 2024-03-11T21:44:26.425 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:44:26.425 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:44:26.426 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "waiting for peered", 2024-03-11T21:44:26.426 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:41:28.023955+0000", 2024-03-11T21:44:26.426 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 8.1890000000000007e-06 2024-03-11T21:44:26.426 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:44:26.426 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:44:26.426 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-03-11T21:44:26.426 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:41:28.855438+0000", 2024-03-11T21:44:26.426 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.831483105 2024-03-11T21:44:26.426 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:44:26.426 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:44:26.426 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "done", 2024-03-11T21:44:26.426 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:41:28.855514+0000", 2024-03-11T21:44:26.426 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 7.6759000000000001e-05 2024-03-11T21:44:26.426 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-03-11T21:44:26.426 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-03-11T21:44:26.427 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-03-11T21:44:26.427 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:44:26.427 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:44:26.427 INFO:teuthology.orchestra.run.smithi092.stdout: "description": "osd_op(client.15744.0:69286 10.18 10:192779a6:::benchmark_data_smithi092_161462_object69285:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e372)", 2024-03-11T21:44:26.427 INFO:teuthology.orchestra.run.smithi092.stdout: "initiated_at": "2024-03-11T21:41:27.826676+0000", 2024-03-11T21:44:26.427 INFO:teuthology.orchestra.run.smithi092.stdout: "age": 178.48616221099999, 2024-03-11T21:44:26.427 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.02529297, 2024-03-11T21:44:26.427 INFO:teuthology.orchestra.run.smithi092.stdout: "type_data": { 2024-03-11T21:44:26.427 INFO:teuthology.orchestra.run.smithi092.stdout: "flag_point": "reached pg", 2024-03-11T21:44:26.427 INFO:teuthology.orchestra.run.smithi092.stdout: "client_info": { 2024-03-11T21:44:26.427 INFO:teuthology.orchestra.run.smithi092.stdout: "client": "client.15744", 2024-03-11T21:44:26.427 INFO:teuthology.orchestra.run.smithi092.stdout: "client_addr": "v1:172.21.15.92:0/3334000812", 2024-03-11T21:44:26.427 INFO:teuthology.orchestra.run.smithi092.stdout: "tid": 69286 2024-03-11T21:44:26.427 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:44:26.427 INFO:teuthology.orchestra.run.smithi092.stdout: "events": [ 2024-03-11T21:44:26.428 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:44:26.428 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "initiated", 2024-03-11T21:44:26.428 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:41:27.826676+0000", 2024-03-11T21:44:26.428 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-03-11T21:44:26.428 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:44:26.428 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:44:26.428 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "header_read", 2024-03-11T21:44:26.428 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:41:27.826676+0000", 2024-03-11T21:44:26.428 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-03-11T21:44:26.428 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:44:26.428 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:44:26.428 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "throttled", 2024-03-11T21:44:26.428 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:41:27.826677+0000", 2024-03-11T21:44:26.428 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 8.4300000000000002e-07 2024-03-11T21:44:26.428 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:44:26.429 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:44:26.429 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "all_read", 2024-03-11T21:44:26.429 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:41:27.826700+0000", 2024-03-11T21:44:26.429 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 2.2886000000000001e-05 2024-03-11T21:44:26.429 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:44:26.429 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:44:26.429 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "dispatched", 2024-03-11T21:44:26.429 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:41:27.826709+0000", 2024-03-11T21:44:26.429 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 9.4520000000000003e-06 2024-03-11T21:44:26.429 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:44:26.429 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:44:26.429 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "queued_for_pg", 2024-03-11T21:44:26.429 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:41:27.826716+0000", 2024-03-11T21:44:26.429 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 6.3080000000000004e-06 2024-03-11T21:44:26.429 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:44:26.430 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:44:26.430 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-03-11T21:44:26.430 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:41:28.014627+0000", 2024-03-11T21:44:26.430 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.18791107400000001 2024-03-11T21:44:26.430 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:44:26.430 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:44:26.430 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "waiting for peered", 2024-03-11T21:44:26.430 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:41:28.014633+0000", 2024-03-11T21:44:26.430 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 6.2110000000000003e-06 2024-03-11T21:44:26.430 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:44:26.430 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:44:26.430 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-03-11T21:44:26.430 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:41:28.851906+0000", 2024-03-11T21:44:26.430 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.83727308899999997 2024-03-11T21:44:26.431 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:44:26.431 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:44:26.431 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "done", 2024-03-11T21:44:26.431 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:41:28.851969+0000", 2024-03-11T21:44:26.431 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 6.3107000000000004e-05 2024-03-11T21:44:26.431 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-03-11T21:44:26.431 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-03-11T21:44:26.431 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-03-11T21:44:26.431 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:44:26.431 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:44:26.431 INFO:teuthology.orchestra.run.smithi092.stdout: "description": "osd_op(client.15744.0:217052 10.4 10:272c84ab:::benchmark_data_smithi092_161462_object107076:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e435)", 2024-03-11T21:44:26.431 INFO:teuthology.orchestra.run.smithi092.stdout: "initiated_at": "2024-03-11T21:44:24.465063+0000", 2024-03-11T21:44:26.431 INFO:teuthology.orchestra.run.smithi092.stdout: "age": 1.8477756009999999, 2024-03-11T21:44:26.431 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.026855337, 2024-03-11T21:44:26.431 INFO:teuthology.orchestra.run.smithi092.stdout: "type_data": { 2024-03-11T21:44:26.432 INFO:teuthology.orchestra.run.smithi092.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-11T21:44:26.432 INFO:teuthology.orchestra.run.smithi092.stdout: "client_info": { 2024-03-11T21:44:26.432 INFO:teuthology.orchestra.run.smithi092.stdout: "client": "client.15744", 2024-03-11T21:44:26.432 INFO:teuthology.orchestra.run.smithi092.stdout: "client_addr": "v1:172.21.15.92:0/3334000812", 2024-03-11T21:44:26.432 INFO:teuthology.orchestra.run.smithi092.stdout: "tid": 217052 2024-03-11T21:44:26.432 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:44:26.432 INFO:teuthology.orchestra.run.smithi092.stdout: "events": [ 2024-03-11T21:44:26.432 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:44:26.432 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "initiated", 2024-03-11T21:44:26.432 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:44:24.465063+0000", 2024-03-11T21:44:26.432 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-03-11T21:44:26.432 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:44:26.432 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:44:26.432 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "header_read", 2024-03-11T21:44:26.432 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:44:24.465063+0000", 2024-03-11T21:44:26.433 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-03-11T21:44:26.433 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:44:26.433 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:44:26.433 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "throttled", 2024-03-11T21:44:26.433 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:44:24.465065+0000", 2024-03-11T21:44:26.433 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 2.2350000000000002e-06 2024-03-11T21:44:26.433 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:44:26.433 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:44:26.433 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "all_read", 2024-03-11T21:44:26.433 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:44:24.465072+0000", 2024-03-11T21:44:26.433 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 6.9369999999999998e-06 2024-03-11T21:44:26.433 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:44:26.433 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:44:26.433 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "dispatched", 2024-03-11T21:44:26.433 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:44:24.465085+0000", 2024-03-11T21:44:26.434 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.3312999999999999e-05 2024-03-11T21:44:26.434 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:44:26.434 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:44:26.434 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "queued_for_pg", 2024-03-11T21:44:26.434 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:44:24.465094+0000", 2024-03-11T21:44:26.434 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 8.8570000000000001e-06 2024-03-11T21:44:26.434 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:44:26.434 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:44:26.434 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-03-11T21:44:26.434 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:44:24.465155+0000", 2024-03-11T21:44:26.434 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 6.1311999999999998e-05 2024-03-11T21:44:26.434 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:44:26.434 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:44:26.434 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "waiting for peered", 2024-03-11T21:44:26.434 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:44:24.465165+0000", 2024-03-11T21:44:26.435 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 9.4180000000000003e-06 2024-03-11T21:44:26.435 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:44:26.435 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:44:26.435 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-03-11T21:44:26.435 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:44:25.476597+0000", 2024-03-11T21:44:26.435 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.0114318630000001 2024-03-11T21:44:26.435 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:44:26.435 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:44:26.435 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "started", 2024-03-11T21:44:26.435 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:44:25.476717+0000", 2024-03-11T21:44:26.435 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00012019900000000001 2024-03-11T21:44:26.435 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:44:26.435 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:44:26.435 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "waiting for subops from 1,4", 2024-03-11T21:44:26.435 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:44:25.476812+0000", 2024-03-11T21:44:26.436 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 9.5292999999999997e-05 2024-03-11T21:44:26.436 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:44:26.436 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:44:26.436 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "sub_op_commit_rec", 2024-03-11T21:44:26.436 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:44:25.480067+0000", 2024-03-11T21:44:26.436 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.0032546609999999998 2024-03-11T21:44:26.436 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:44:26.436 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:44:26.436 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "sub_op_commit_rec", 2024-03-11T21:44:26.436 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:44:25.480109+0000", 2024-03-11T21:44:26.436 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 4.2574999999999998e-05 2024-03-11T21:44:26.436 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:44:26.436 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:44:26.436 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "op_commit", 2024-03-11T21:44:26.436 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:44:25.491770+0000", 2024-03-11T21:44:26.437 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.011660129 2024-03-11T21:44:26.437 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:44:26.437 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:44:26.437 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "commit_sent", 2024-03-11T21:44:26.437 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:44:25.491868+0000", 2024-03-11T21:44:26.437 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 9.8858999999999997e-05 2024-03-11T21:44:26.437 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:44:26.437 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:44:26.437 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "done", 2024-03-11T21:44:26.437 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:44:25.491918+0000", 2024-03-11T21:44:26.437 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 4.9685999999999998e-05 2024-03-11T21:44:26.437 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-03-11T21:44:26.437 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-03-11T21:44:26.437 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-03-11T21:44:26.437 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:44:26.437 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:44:26.438 INFO:teuthology.orchestra.run.smithi092.stdout: "description": "osd_op(client.15744.0:217054 10.4 10:22858cc6:::benchmark_data_smithi092_161462_object107078:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e435)", 2024-03-11T21:44:26.438 INFO:teuthology.orchestra.run.smithi092.stdout: "initiated_at": "2024-03-11T21:44:24.469395+0000", 2024-03-11T21:44:26.438 INFO:teuthology.orchestra.run.smithi092.stdout: "age": 1.843443699, 2024-03-11T21:44:26.438 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.023835211, 2024-03-11T21:44:26.438 INFO:teuthology.orchestra.run.smithi092.stdout: "type_data": { 2024-03-11T21:44:26.438 INFO:teuthology.orchestra.run.smithi092.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-11T21:44:26.438 INFO:teuthology.orchestra.run.smithi092.stdout: "client_info": { 2024-03-11T21:44:26.438 INFO:teuthology.orchestra.run.smithi092.stdout: "client": "client.15744", 2024-03-11T21:44:26.438 INFO:teuthology.orchestra.run.smithi092.stdout: "client_addr": "v1:172.21.15.92:0/3334000812", 2024-03-11T21:44:26.438 INFO:teuthology.orchestra.run.smithi092.stdout: "tid": 217054 2024-03-11T21:44:26.438 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:44:26.438 INFO:teuthology.orchestra.run.smithi092.stdout: "events": [ 2024-03-11T21:44:26.438 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:44:26.438 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "initiated", 2024-03-11T21:44:26.439 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:44:24.469395+0000", 2024-03-11T21:44:26.439 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-03-11T21:44:26.439 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:44:26.439 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:44:26.439 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "header_read", 2024-03-11T21:44:26.439 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:44:24.469395+0000", 2024-03-11T21:44:26.439 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-03-11T21:44:26.439 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:44:26.439 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:44:26.439 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "throttled", 2024-03-11T21:44:26.439 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:44:24.469397+0000", 2024-03-11T21:44:26.439 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 2.322e-06 2024-03-11T21:44:26.439 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:44:26.439 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:44:26.439 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "all_read", 2024-03-11T21:44:26.439 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:44:24.469405+0000", 2024-03-11T21:44:26.440 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 7.9039999999999992e-06 2024-03-11T21:44:26.440 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:44:26.440 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:44:26.440 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "dispatched", 2024-03-11T21:44:26.440 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:44:24.469420+0000", 2024-03-11T21:44:26.440 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.4929999999999999e-05 2024-03-11T21:44:26.440 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:44:26.440 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:44:26.440 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "queued_for_pg", 2024-03-11T21:44:26.440 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:44:24.469430+0000", 2024-03-11T21:44:26.440 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.0205e-05 2024-03-11T21:44:26.440 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:44:26.440 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:44:26.440 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-03-11T21:44:26.440 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:44:24.469588+0000", 2024-03-11T21:44:26.441 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00015770399999999999 2024-03-11T21:44:26.441 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:44:26.441 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:44:26.441 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "waiting for peered", 2024-03-11T21:44:26.441 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:44:24.469598+0000", 2024-03-11T21:44:26.441 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 9.8009999999999998e-06 2024-03-11T21:44:26.441 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:44:26.441 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:44:26.441 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-03-11T21:44:26.441 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:44:25.480143+0000", 2024-03-11T21:44:26.441 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.0105453440000001 2024-03-11T21:44:26.441 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:44:26.441 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:44:26.441 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "started", 2024-03-11T21:44:26.441 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:44:25.480248+0000", 2024-03-11T21:44:26.442 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00010474199999999999 2024-03-11T21:44:26.442 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:44:26.442 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:44:26.442 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "waiting for subops from 1,4", 2024-03-11T21:44:26.442 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:44:25.480369+0000", 2024-03-11T21:44:26.442 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00012146099999999999 2024-03-11T21:44:26.442 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:44:26.442 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:44:26.442 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "sub_op_commit_rec", 2024-03-11T21:44:26.442 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:44:25.489205+0000", 2024-03-11T21:44:26.442 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.0088363000000000001 2024-03-11T21:44:26.442 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:44:26.442 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:44:26.442 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "sub_op_commit_rec", 2024-03-11T21:44:26.442 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:44:25.489287+0000", 2024-03-11T21:44:26.443 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 8.1477999999999995e-05 2024-03-11T21:44:26.443 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:44:26.443 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:44:26.443 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "op_commit", 2024-03-11T21:44:26.443 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:44:25.493106+0000", 2024-03-11T21:44:26.443 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.0038188050000000002 2024-03-11T21:44:26.443 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:44:26.443 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:44:26.443 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "commit_sent", 2024-03-11T21:44:26.443 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:44:25.493191+0000", 2024-03-11T21:44:26.443 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 8.5185999999999994e-05 2024-03-11T21:44:26.443 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:44:26.443 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:44:26.443 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "done", 2024-03-11T21:44:26.443 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:44:25.493230+0000", 2024-03-11T21:44:26.444 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 3.9029000000000002e-05 2024-03-11T21:44:26.444 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-03-11T21:44:26.444 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-03-11T21:44:26.444 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-03-11T21:44:26.444 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:44:26.444 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:44:26.444 INFO:teuthology.orchestra.run.smithi092.stdout: "description": "osd_op(client.15744.0:217055 10.4 10:21c7e5a7:::benchmark_data_smithi092_161462_object107079:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e435)", 2024-03-11T21:44:26.444 INFO:teuthology.orchestra.run.smithi092.stdout: "initiated_at": "2024-03-11T21:44:24.469454+0000", 2024-03-11T21:44:26.444 INFO:teuthology.orchestra.run.smithi092.stdout: "age": 1.8433847110000001, 2024-03-11T21:44:26.444 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.023932952, 2024-03-11T21:44:26.444 INFO:teuthology.orchestra.run.smithi092.stdout: "type_data": { 2024-03-11T21:44:26.444 INFO:teuthology.orchestra.run.smithi092.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-11T21:44:26.444 INFO:teuthology.orchestra.run.smithi092.stdout: "client_info": { 2024-03-11T21:44:26.444 INFO:teuthology.orchestra.run.smithi092.stdout: "client": "client.15744", 2024-03-11T21:44:26.444 INFO:teuthology.orchestra.run.smithi092.stdout: "client_addr": "v1:172.21.15.92:0/3334000812", 2024-03-11T21:44:26.445 INFO:teuthology.orchestra.run.smithi092.stdout: "tid": 217055 2024-03-11T21:44:26.445 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:44:26.445 INFO:teuthology.orchestra.run.smithi092.stdout: "events": [ 2024-03-11T21:44:26.445 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:44:26.445 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "initiated", 2024-03-11T21:44:26.445 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:44:24.469454+0000", 2024-03-11T21:44:26.445 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-03-11T21:44:26.445 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:44:26.445 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:44:26.445 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "header_read", 2024-03-11T21:44:26.445 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:44:24.469454+0000", 2024-03-11T21:44:26.445 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-03-11T21:44:26.445 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:44:26.445 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:44:26.445 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "throttled", 2024-03-11T21:44:26.446 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:44:24.469454+0000", 2024-03-11T21:44:26.446 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 7.0500000000000003e-07 2024-03-11T21:44:26.446 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:44:26.446 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:44:26.446 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "all_read", 2024-03-11T21:44:26.446 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:44:24.469458+0000", 2024-03-11T21:44:26.446 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 3.3790000000000002e-06 2024-03-11T21:44:26.446 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:44:26.446 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:44:26.446 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "dispatched", 2024-03-11T21:44:26.446 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:44:24.469464+0000", 2024-03-11T21:44:26.446 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 6.3099999999999997e-06 2024-03-11T21:44:26.446 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:44:26.446 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:44:26.446 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "queued_for_pg", 2024-03-11T21:44:26.447 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:44:24.469471+0000", 2024-03-11T21:44:26.447 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 6.5479999999999998e-06 2024-03-11T21:44:26.447 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:44:26.447 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:44:26.447 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-03-11T21:44:26.447 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:44:24.469620+0000", 2024-03-11T21:44:26.447 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00014907399999999999 2024-03-11T21:44:26.447 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:44:26.447 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:44:26.447 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "waiting for peered", 2024-03-11T21:44:26.447 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:44:24.469625+0000", 2024-03-11T21:44:26.447 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 5.5840000000000002e-06 2024-03-11T21:44:26.447 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:44:26.447 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:44:26.447 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-03-11T21:44:26.448 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:44:25.483217+0000", 2024-03-11T21:44:26.448 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.0135916730000001 2024-03-11T21:44:26.448 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:44:26.448 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:44:26.448 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "started", 2024-03-11T21:44:26.448 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:44:25.483325+0000", 2024-03-11T21:44:26.448 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00010842 2024-03-11T21:44:26.448 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:44:26.448 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:44:26.448 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "waiting for subops from 1,4", 2024-03-11T21:44:26.448 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:44:25.483433+0000", 2024-03-11T21:44:26.448 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00010748500000000001 2024-03-11T21:44:26.448 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:44:26.448 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:44:26.448 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "sub_op_commit_rec", 2024-03-11T21:44:26.449 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:44:25.489250+0000", 2024-03-11T21:44:26.449 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.0058175060000000001 2024-03-11T21:44:26.449 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:44:26.449 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:44:26.449 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "op_commit", 2024-03-11T21:44:26.449 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:44:25.493237+0000", 2024-03-11T21:44:26.449 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.003986379 2024-03-11T21:44:26.449 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:44:26.449 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:44:26.449 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "sub_op_commit_rec", 2024-03-11T21:44:26.449 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:44:25.493275+0000", 2024-03-11T21:44:26.449 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 3.7846999999999999e-05 2024-03-11T21:44:26.449 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:44:26.449 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:44:26.449 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "commit_sent", 2024-03-11T21:44:26.450 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:44:25.493350+0000", 2024-03-11T21:44:26.450 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 7.5718999999999998e-05 2024-03-11T21:44:26.450 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:44:26.450 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:44:26.450 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "done", 2024-03-11T21:44:26.450 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:44:25.493387+0000", 2024-03-11T21:44:26.450 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 3.6322999999999998e-05 2024-03-11T21:44:26.450 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-03-11T21:44:26.450 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-03-11T21:44:26.450 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-03-11T21:44:26.450 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:44:26.450 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:44:26.450 INFO:teuthology.orchestra.run.smithi092.stdout: "description": "osd_op(client.15744.0:217071 10.4 10:22553656:::benchmark_data_smithi092_161462_object107095:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e435)", 2024-03-11T21:44:26.450 INFO:teuthology.orchestra.run.smithi092.stdout: "initiated_at": "2024-03-11T21:44:24.480901+0000", 2024-03-11T21:44:26.451 INFO:teuthology.orchestra.run.smithi092.stdout: "age": 1.831937414, 2024-03-11T21:44:26.451 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.013271341, 2024-03-11T21:44:26.451 INFO:teuthology.orchestra.run.smithi092.stdout: "type_data": { 2024-03-11T21:44:26.451 INFO:teuthology.orchestra.run.smithi092.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-11T21:44:26.451 INFO:teuthology.orchestra.run.smithi092.stdout: "client_info": { 2024-03-11T21:44:26.451 INFO:teuthology.orchestra.run.smithi092.stdout: "client": "client.15744", 2024-03-11T21:44:26.451 INFO:teuthology.orchestra.run.smithi092.stdout: "client_addr": "v1:172.21.15.92:0/3334000812", 2024-03-11T21:44:26.451 INFO:teuthology.orchestra.run.smithi092.stdout: "tid": 217071 2024-03-11T21:44:26.451 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:44:26.451 INFO:teuthology.orchestra.run.smithi092.stdout: "events": [ 2024-03-11T21:44:26.451 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:44:26.451 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "initiated", 2024-03-11T21:44:26.451 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:44:24.480901+0000", 2024-03-11T21:44:26.451 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-03-11T21:44:26.451 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:44:26.452 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:44:26.452 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "header_read", 2024-03-11T21:44:26.452 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:44:24.480901+0000", 2024-03-11T21:44:26.452 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-03-11T21:44:26.452 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:44:26.452 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:44:26.452 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "throttled", 2024-03-11T21:44:26.452 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:44:24.480907+0000", 2024-03-11T21:44:26.452 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 5.7849999999999997e-06 2024-03-11T21:44:26.452 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:44:26.452 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:44:26.452 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "all_read", 2024-03-11T21:44:26.452 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:44:24.480912+0000", 2024-03-11T21:44:26.452 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 5.6740000000000002e-06 2024-03-11T21:44:26.452 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:44:26.453 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:44:26.453 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "dispatched", 2024-03-11T21:44:26.453 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:44:24.480923+0000", 2024-03-11T21:44:26.453 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.1097000000000001e-05 2024-03-11T21:44:26.453 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:44:26.453 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:44:26.453 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "queued_for_pg", 2024-03-11T21:44:26.453 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:44:24.480931+0000", 2024-03-11T21:44:26.453 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 7.593e-06 2024-03-11T21:44:26.453 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:44:26.453 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:44:26.453 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-03-11T21:44:26.453 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:44:24.481059+0000", 2024-03-11T21:44:26.453 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00012835200000000001 2024-03-11T21:44:26.453 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:44:26.454 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:44:26.454 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "waiting for peered", 2024-03-11T21:44:26.454 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:44:24.481077+0000", 2024-03-11T21:44:26.454 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.7578999999999999e-05 2024-03-11T21:44:26.454 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:44:26.454 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:44:26.454 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-03-11T21:44:26.454 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:44:25.486306+0000", 2024-03-11T21:44:26.454 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.0052291170000001 2024-03-11T21:44:26.454 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:44:26.454 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:44:26.454 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "started", 2024-03-11T21:44:26.454 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:44:25.486410+0000", 2024-03-11T21:44:26.454 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.000103539 2024-03-11T21:44:26.455 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:44:26.455 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:44:26.455 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "waiting for subops from 1,4", 2024-03-11T21:44:26.455 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:44:25.486518+0000", 2024-03-11T21:44:26.455 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00010817199999999999 2024-03-11T21:44:26.455 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:44:26.455 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:44:26.455 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "op_commit", 2024-03-11T21:44:26.455 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:44:25.493399+0000", 2024-03-11T21:44:26.455 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.0068813529999999998 2024-03-11T21:44:26.455 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:44:26.455 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:44:26.455 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "sub_op_commit_rec", 2024-03-11T21:44:26.455 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:44:25.493831+0000", 2024-03-11T21:44:26.455 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00043177899999999997 2024-03-11T21:44:26.455 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:44:26.456 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:44:26.456 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "sub_op_commit_rec", 2024-03-11T21:44:26.456 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:44:25.494033+0000", 2024-03-11T21:44:26.456 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00020214199999999999 2024-03-11T21:44:26.456 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:44:26.456 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:44:26.456 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "commit_sent", 2024-03-11T21:44:26.456 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:44:25.494128+0000", 2024-03-11T21:44:26.456 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 9.4732e-05 2024-03-11T21:44:26.456 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:44:26.456 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:44:26.456 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "done", 2024-03-11T21:44:26.456 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:44:25.494172+0000", 2024-03-11T21:44:26.456 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 4.4427e-05 2024-03-11T21:44:26.456 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-03-11T21:44:26.457 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-03-11T21:44:26.457 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-03-11T21:44:26.457 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:44:26.457 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:44:26.457 INFO:teuthology.orchestra.run.smithi092.stdout: "description": "osd_op(client.15744.0:217114 10.4 10:21d6bf86:::benchmark_data_smithi092_161462_object107138:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e435)", 2024-03-11T21:44:26.457 INFO:teuthology.orchestra.run.smithi092.stdout: "initiated_at": "2024-03-11T21:44:24.528462+0000", 2024-03-11T21:44:26.457 INFO:teuthology.orchestra.run.smithi092.stdout: "age": 1.7843758950000002, 2024-03-11T21:44:26.457 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.96773134100000002, 2024-03-11T21:44:26.457 INFO:teuthology.orchestra.run.smithi092.stdout: "type_data": { 2024-03-11T21:44:26.457 INFO:teuthology.orchestra.run.smithi092.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-11T21:44:26.457 INFO:teuthology.orchestra.run.smithi092.stdout: "client_info": { 2024-03-11T21:44:26.457 INFO:teuthology.orchestra.run.smithi092.stdout: "client": "client.15744", 2024-03-11T21:44:26.457 INFO:teuthology.orchestra.run.smithi092.stdout: "client_addr": "v1:172.21.15.92:0/3334000812", 2024-03-11T21:44:26.457 INFO:teuthology.orchestra.run.smithi092.stdout: "tid": 217114 2024-03-11T21:44:26.457 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:44:26.458 INFO:teuthology.orchestra.run.smithi092.stdout: "events": [ 2024-03-11T21:44:26.458 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:44:26.458 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "initiated", 2024-03-11T21:44:26.458 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:44:24.528462+0000", 2024-03-11T21:44:26.458 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-03-11T21:44:26.458 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:44:26.458 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:44:26.458 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "header_read", 2024-03-11T21:44:26.458 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:44:24.528462+0000", 2024-03-11T21:44:26.458 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-03-11T21:44:26.458 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:44:26.458 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:44:26.458 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "throttled", 2024-03-11T21:44:26.458 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:44:24.528465+0000", 2024-03-11T21:44:26.458 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 2.1009999999999999e-06 2024-03-11T21:44:26.459 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:44:26.459 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:44:26.459 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "all_read", 2024-03-11T21:44:26.459 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:44:24.528471+0000", 2024-03-11T21:44:26.459 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 6.3589999999999996e-06 2024-03-11T21:44:26.459 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:44:26.459 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:44:26.459 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "dispatched", 2024-03-11T21:44:26.459 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:44:24.528484+0000", 2024-03-11T21:44:26.459 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.3301e-05 2024-03-11T21:44:26.459 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:44:26.459 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:44:26.459 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "queued_for_pg", 2024-03-11T21:44:26.459 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:44:24.528492+0000", 2024-03-11T21:44:26.459 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 7.824e-06 2024-03-11T21:44:26.460 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:44:26.460 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:44:26.460 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-03-11T21:44:26.460 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:44:24.528544+0000", 2024-03-11T21:44:26.460 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 5.2269999999999999e-05 2024-03-11T21:44:26.460 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:44:26.460 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:44:26.460 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "waiting for peered", 2024-03-11T21:44:26.460 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:44:24.528572+0000", 2024-03-11T21:44:26.460 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 2.76e-05 2024-03-11T21:44:26.460 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:44:26.460 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:44:26.460 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-03-11T21:44:26.460 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:44:25.489304+0000", 2024-03-11T21:44:26.460 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.96073213999999996 2024-03-11T21:44:26.461 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:44:26.461 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:44:26.461 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "started", 2024-03-11T21:44:26.461 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:44:25.489382+0000", 2024-03-11T21:44:26.461 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 7.8200000000000003e-05 2024-03-11T21:44:26.461 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:44:26.461 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:44:26.461 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "waiting for subops from 1,4", 2024-03-11T21:44:26.461 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:44:25.489464+0000", 2024-03-11T21:44:26.461 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 8.1495000000000005e-05 2024-03-11T21:44:26.461 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:44:26.461 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:44:26.461 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "op_commit", 2024-03-11T21:44:26.461 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:44:25.493404+0000", 2024-03-11T21:44:26.461 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.0039406049999999998 2024-03-11T21:44:26.462 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:44:26.462 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:44:26.462 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "sub_op_commit_rec", 2024-03-11T21:44:26.462 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:44:25.493988+0000", 2024-03-11T21:44:26.462 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00058407000000000001 2024-03-11T21:44:26.462 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:44:26.462 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:44:26.462 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "sub_op_commit_rec", 2024-03-11T21:44:26.462 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:44:25.496053+0000", 2024-03-11T21:44:26.462 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.0020642880000000001 2024-03-11T21:44:26.462 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:44:26.462 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:44:26.462 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "commit_sent", 2024-03-11T21:44:26.462 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:44:25.496152+0000", 2024-03-11T21:44:26.462 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 9.9684000000000003e-05 2024-03-11T21:44:26.463 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:44:26.463 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:44:26.463 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "done", 2024-03-11T21:44:26.463 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:44:25.496194+0000", 2024-03-11T21:44:26.463 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 4.1403999999999999e-05 2024-03-11T21:44:26.463 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-03-11T21:44:26.463 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-03-11T21:44:26.463 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-03-11T21:44:26.463 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-03-11T21:44:26.463 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-03-11T21:44:26.463 INFO:teuthology.orchestra.run.smithi092.stdout:} 2024-03-11T21:44:26.504 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:44:26.548 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:44:26.549 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:44:26.549 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:44:26.617 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:44:26 smithi092 ceph-mon[29572]: pgmap v893: 157 pgs: 1 peering, 2 active+remapped+backfilling, 3 activating+remapped, 5 active+remapped+backfill_toofull, 146 active+clean; 291 MiB data, 5.7 GiB used, 620 GiB / 626 GiB avail; 1.8 KiB/s rd, 0 B/s wr, 654 op/s; 1852/14520 objects misplaced (12.755%); 3.7 MiB/s, 72 objects/s recovering 2024-03-11T21:44:26.617 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:44:26 smithi092 ceph-mon[29572]: 5.8 scrub starts 2024-03-11T21:44:26.617 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:44:26 smithi092 ceph-mon[29572]: 5.8 scrub ok 2024-03-11T21:44:26.618 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:44:26 smithi092 ceph-mon[29572]: 4.5 scrub starts 2024-03-11T21:44:26.618 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:44:26 smithi092 ceph-mon[29572]: 4.5 scrub ok 2024-03-11T21:44:26.618 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:44:26 smithi092 ceph-mon[37563]: pgmap v893: 157 pgs: 1 peering, 2 active+remapped+backfilling, 3 activating+remapped, 5 active+remapped+backfill_toofull, 146 active+clean; 291 MiB data, 5.7 GiB used, 620 GiB / 626 GiB avail; 1.8 KiB/s rd, 0 B/s wr, 654 op/s; 1852/14520 objects misplaced (12.755%); 3.7 MiB/s, 72 objects/s recovering 2024-03-11T21:44:26.618 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:44:26 smithi092 ceph-mon[37563]: 5.8 scrub starts 2024-03-11T21:44:26.619 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:44:26 smithi092 ceph-mon[37563]: 5.8 scrub ok 2024-03-11T21:44:26.619 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:44:26 smithi092 ceph-mon[37563]: 4.5 scrub starts 2024-03-11T21:44:26.619 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:44:26 smithi092 ceph-mon[37563]: 4.5 scrub ok 2024-03-11T21:44:26.650 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:44:26.687 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:44:26.689 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:44:26.689 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:44:26.790 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:44:26.827 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:44:26.828 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:44:26.828 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:44:26.832 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:44:26 smithi097 ceph-mon[33317]: pgmap v893: 157 pgs: 1 peering, 2 active+remapped+backfilling, 3 activating+remapped, 5 active+remapped+backfill_toofull, 146 active+clean; 291 MiB data, 5.7 GiB used, 620 GiB / 626 GiB avail; 1.8 KiB/s rd, 0 B/s wr, 654 op/s; 1852/14520 objects misplaced (12.755%); 3.7 MiB/s, 72 objects/s recovering 2024-03-11T21:44:26.833 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:44:26 smithi097 ceph-mon[33317]: 5.8 scrub starts 2024-03-11T21:44:26.833 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:44:26 smithi097 ceph-mon[33317]: 5.8 scrub ok 2024-03-11T21:44:26.833 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:44:26 smithi097 ceph-mon[33317]: 4.5 scrub starts 2024-03-11T21:44:26.833 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:44:26 smithi097 ceph-mon[33317]: 4.5 scrub ok 2024-03-11T21:44:26.929 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:44:26.967 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:44:26.968 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:44:26.968 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:44:27.069 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:44:27.103 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:44:27.103 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:44:27.103 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:44:27.204 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:44:27.240 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:44:27.241 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:44:27.241 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:44:27.342 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:44:27.379 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:44:27.380 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:44:27.380 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:44:27.437 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 4, 5, 6, 7] out_osds: [3] dead_osds: [] live_osds: [0, 1, 2, 4, 5, 6, 7, 3] 2024-03-11T21:44:27.437 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-03-11T21:44:27.437 INFO:tasks.thrashosds.thrasher:Removing osd 4, in_osds are: [0, 1, 2, 4, 5, 6, 7] 2024-03-11T21:44:27.437 DEBUG:teuthology.orchestra.run.smithi092:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid 00db0016-dfed-11ee-95c8-87774f69a715 -- ceph osd out 4 2024-03-11T21:44:27.482 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:44:27.519 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:44:27.519 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:44:27.519 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:44:27.526 INFO:tasks.radosbench.radosbench.0.smithi092.stdout:Removed 109973 objects 2024-03-11T21:44:27.527 INFO:tasks.radosbench.radosbench.0.smithi092.stdout:Clean up completed and total clean up time :123.679 2024-03-11T21:44:27.620 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:44:27.644 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:44:27.644 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:44:27.645 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:44:27.746 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:44:27.770 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:44:27.771 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:44:27.771 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:44:27.834 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:44:27 smithi097 ceph-mon[33317]: 6.6 scrub starts 2024-03-11T21:44:27.834 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:44:27 smithi097 ceph-mon[33317]: 6.6 scrub ok 2024-03-11T21:44:27.834 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:44:27 smithi097 ceph-mon[33317]: 6.4 scrub starts 2024-03-11T21:44:27.834 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:44:27 smithi097 ceph-mon[33317]: 6.4 scrub ok 2024-03-11T21:44:27.867 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:44:27 smithi092 ceph-mon[29572]: 6.6 scrub starts 2024-03-11T21:44:27.867 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:44:27 smithi092 ceph-mon[29572]: 6.6 scrub ok 2024-03-11T21:44:27.867 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:44:27 smithi092 ceph-mon[29572]: 6.4 scrub starts 2024-03-11T21:44:27.867 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:44:27 smithi092 ceph-mon[29572]: 6.4 scrub ok 2024-03-11T21:44:27.867 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:44:27 smithi092 ceph-mon[37563]: 6.6 scrub starts 2024-03-11T21:44:27.867 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:44:27 smithi092 ceph-mon[37563]: 6.6 scrub ok 2024-03-11T21:44:27.867 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:44:27 smithi092 ceph-mon[37563]: 6.4 scrub starts 2024-03-11T21:44:27.867 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:44:27 smithi092 ceph-mon[37563]: 6.4 scrub ok 2024-03-11T21:44:27.872 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:44:27.896 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:44:27.897 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:44:27.897 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:44:27.998 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:44:28.023 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:44:28.024 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:44:28.025 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:44:28.126 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:44:28.150 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:44:28.150 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:44:28.150 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:44:28.251 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:44:28.275 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:44:28.276 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:44:28.276 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:44:28.352 INFO:journalctl@ceph.mgr.y.smithi092.stdout:Mar 11 21:44:27 smithi092 ceph-00db0016-dfed-11ee-95c8-87774f69a715-mgr-y[29801]: ::ffff:172.21.15.97 - - [11/Mar/2024:21:44:27] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-11T21:44:28.377 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:44:28.401 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:44:28.402 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:44:28.402 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:44:28.503 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:44:28.527 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:44:28.528 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:44:28.528 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:44:28.617 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:44:28 smithi092 ceph-mon[29572]: 6.16 scrub starts 2024-03-11T21:44:28.617 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:44:28 smithi092 ceph-mon[29572]: 6.16 scrub ok 2024-03-11T21:44:28.617 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:44:28 smithi092 ceph-mon[29572]: pgmap v895: 157 pgs: 1 peering, 2 active+remapped+backfilling, 3 activating+remapped, 5 active+remapped+backfill_toofull, 146 active+clean; 220 MiB data, 5.5 GiB used, 620 GiB / 626 GiB avail; 1.5 KiB/s rd, 0 B/s wr, 734 op/s; 1852/11118 objects misplaced (16.658%); 3.1 MiB/s, 60 objects/s recovering 2024-03-11T21:44:28.617 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:44:28 smithi092 ceph-mon[29572]: 3.2 scrub starts 2024-03-11T21:44:28.617 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:44:28 smithi092 ceph-mon[29572]: 3.2 scrub ok 2024-03-11T21:44:28.617 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:44:28 smithi092 ceph-mon[29572]: 2.a deep-scrub starts 2024-03-11T21:44:28.617 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:44:28 smithi092 ceph-mon[29572]: 2.a deep-scrub ok 2024-03-11T21:44:28.618 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:44:28 smithi092 ceph-mon[37563]: 6.16 scrub starts 2024-03-11T21:44:28.618 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:44:28 smithi092 ceph-mon[37563]: 6.16 scrub ok 2024-03-11T21:44:28.618 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:44:28 smithi092 ceph-mon[37563]: pgmap v895: 157 pgs: 1 peering, 2 active+remapped+backfilling, 3 activating+remapped, 5 active+remapped+backfill_toofull, 146 active+clean; 220 MiB data, 5.5 GiB used, 620 GiB / 626 GiB avail; 1.5 KiB/s rd, 0 B/s wr, 734 op/s; 1852/11118 objects misplaced (16.658%); 3.1 MiB/s, 60 objects/s recovering 2024-03-11T21:44:28.618 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:44:28 smithi092 ceph-mon[37563]: 3.2 scrub starts 2024-03-11T21:44:28.618 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:44:28 smithi092 ceph-mon[37563]: 3.2 scrub ok 2024-03-11T21:44:28.618 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:44:28 smithi092 ceph-mon[37563]: 2.a deep-scrub starts 2024-03-11T21:44:28.618 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:44:28 smithi092 ceph-mon[37563]: 2.a deep-scrub ok 2024-03-11T21:44:28.629 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:44:28.653 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:44:28.653 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:44:28.654 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:44:28.755 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:44:28.780 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:44:28.780 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:44:28.780 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:44:28.833 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:44:28 smithi097 ceph-mon[33317]: 6.16 scrub starts 2024-03-11T21:44:28.833 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:44:28 smithi097 ceph-mon[33317]: 6.16 scrub ok 2024-03-11T21:44:28.833 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:44:28 smithi097 ceph-mon[33317]: pgmap v895: 157 pgs: 1 peering, 2 active+remapped+backfilling, 3 activating+remapped, 5 active+remapped+backfill_toofull, 146 active+clean; 220 MiB data, 5.5 GiB used, 620 GiB / 626 GiB avail; 1.5 KiB/s rd, 0 B/s wr, 734 op/s; 1852/11118 objects misplaced (16.658%); 3.1 MiB/s, 60 objects/s recovering 2024-03-11T21:44:28.833 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:44:28 smithi097 ceph-mon[33317]: 3.2 scrub starts 2024-03-11T21:44:28.833 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:44:28 smithi097 ceph-mon[33317]: 3.2 scrub ok 2024-03-11T21:44:28.833 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:44:28 smithi097 ceph-mon[33317]: 2.a deep-scrub starts 2024-03-11T21:44:28.833 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:44:28 smithi097 ceph-mon[33317]: 2.a deep-scrub ok 2024-03-11T21:44:28.881 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:44:28.904 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:44:28.905 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:44:28.905 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:44:29.006 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:44:29.031 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:44:29.032 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:44:29.032 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:44:29.133 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:44:29.156 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:44:29.157 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:44:29.157 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:44:29.258 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:44:29.282 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:44:29.282 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:44:29.283 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:44:29.383 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:44:29.406 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:44:29.407 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:44:29.407 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:44:29.508 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:44:29.531 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:44:29.532 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:44:29.532 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:44:29.633 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:44:29.656 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:44:29.657 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:44:29.657 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:44:29.758 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:44:29.782 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:44:29.782 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:44:29.782 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:44:29.833 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:44:29 smithi097 ceph-mon[33317]: 5.1e deep-scrub starts 2024-03-11T21:44:29.833 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:44:29 smithi097 ceph-mon[33317]: 5.1e deep-scrub ok 2024-03-11T21:44:29.833 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:44:29 smithi097 ceph-mon[33317]: 2.e scrub starts 2024-03-11T21:44:29.833 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:44:29 smithi097 ceph-mon[33317]: 2.e scrub ok 2024-03-11T21:44:29.867 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:44:29 smithi092 ceph-mon[29572]: 5.1e deep-scrub starts 2024-03-11T21:44:29.867 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:44:29 smithi092 ceph-mon[29572]: 5.1e deep-scrub ok 2024-03-11T21:44:29.867 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:44:29 smithi092 ceph-mon[29572]: 2.e scrub starts 2024-03-11T21:44:29.867 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:44:29 smithi092 ceph-mon[29572]: 2.e scrub ok 2024-03-11T21:44:29.867 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:44:29 smithi092 ceph-mon[37563]: 5.1e deep-scrub starts 2024-03-11T21:44:29.867 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:44:29 smithi092 ceph-mon[37563]: 5.1e deep-scrub ok 2024-03-11T21:44:29.867 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:44:29 smithi092 ceph-mon[37563]: 2.e scrub starts 2024-03-11T21:44:29.867 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:44:29 smithi092 ceph-mon[37563]: 2.e scrub ok 2024-03-11T21:44:29.884 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:44:29.909 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:44:29.910 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:44:29.910 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:44:30.011 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:44:30.041 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:44:30.041 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:44:30.041 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:44:30.115 INFO:teuthology.orchestra.run.smithi092.stderr:Inferring config /var/lib/ceph/00db0016-dfed-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T21:44:30.135 DEBUG:teuthology.orchestra.run.smithi092:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid 00db0016-dfed-11ee-95c8-87774f69a715 -- ceph daemon osd.1 dump_ops_in_flight 2024-03-11T21:44:30.142 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:44:30.202 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:44:30.202 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:44:30.202 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:44:30.303 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:44:30.327 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:44:30.327 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:44:30.327 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:44:30.379 DEBUG:teuthology.orchestra.run.smithi092:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid 00db0016-dfed-11ee-95c8-87774f69a715 -- ceph osd set noscrub 2024-03-11T21:44:30.428 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:44:30.453 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:44:30.453 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:44:30.454 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:44:30.555 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:44:30.578 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:44:30.579 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:44:30.580 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:44:30.680 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:44:30.705 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:44:30.706 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:44:30.706 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:44:30.807 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:44:30.832 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:44:30.832 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:44:30.833 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:44:30.833 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:44:30 smithi097 ceph-mon[33317]: pgmap v896: 157 pgs: 1 peering, 7 active+remapped+backfill_toofull, 149 active+clean; 19 MiB data, 5.1 GiB used, 621 GiB / 626 GiB avail; 2.0 KiB/s rd, 0 B/s wr, 966 op/s; 4/1476 objects misplaced (0.271%); 4.6 MiB/s, 80 objects/s recovering 2024-03-11T21:44:30.833 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:44:30 smithi097 ceph-mon[33317]: Health check update: Low space hindering backfill (add storage if this doesn't resolve itself): 7 pgs backfill_toofull (PG_BACKFILL_FULL) 2024-03-11T21:44:30.833 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:44:30 smithi097 ceph-mon[33317]: 4.d scrub starts 2024-03-11T21:44:30.833 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:44:30 smithi097 ceph-mon[33317]: 4.d scrub ok 2024-03-11T21:44:30.833 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:44:30 smithi097 ceph-mon[33317]: 2.19 scrub starts 2024-03-11T21:44:30.834 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:44:30 smithi097 ceph-mon[33317]: 2.19 scrub ok 2024-03-11T21:44:30.867 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:44:30 smithi092 ceph-mon[37563]: pgmap v896: 157 pgs: 1 peering, 7 active+remapped+backfill_toofull, 149 active+clean; 19 MiB data, 5.1 GiB used, 621 GiB / 626 GiB avail; 2.0 KiB/s rd, 0 B/s wr, 966 op/s; 4/1476 objects misplaced (0.271%); 4.6 MiB/s, 80 objects/s recovering 2024-03-11T21:44:30.867 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:44:30 smithi092 ceph-mon[37563]: Health check update: Low space hindering backfill (add storage if this doesn't resolve itself): 7 pgs backfill_toofull (PG_BACKFILL_FULL) 2024-03-11T21:44:30.867 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:44:30 smithi092 ceph-mon[37563]: 4.d scrub starts 2024-03-11T21:44:30.867 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:44:30 smithi092 ceph-mon[37563]: 4.d scrub ok 2024-03-11T21:44:30.867 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:44:30 smithi092 ceph-mon[37563]: 2.19 scrub starts 2024-03-11T21:44:30.867 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:44:30 smithi092 ceph-mon[37563]: 2.19 scrub ok 2024-03-11T21:44:30.867 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:44:30 smithi092 ceph-mon[29572]: pgmap v896: 157 pgs: 1 peering, 7 active+remapped+backfill_toofull, 149 active+clean; 19 MiB data, 5.1 GiB used, 621 GiB / 626 GiB avail; 2.0 KiB/s rd, 0 B/s wr, 966 op/s; 4/1476 objects misplaced (0.271%); 4.6 MiB/s, 80 objects/s recovering 2024-03-11T21:44:30.868 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:44:30 smithi092 ceph-mon[29572]: Health check update: Low space hindering backfill (add storage if this doesn't resolve itself): 7 pgs backfill_toofull (PG_BACKFILL_FULL) 2024-03-11T21:44:30.868 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:44:30 smithi092 ceph-mon[29572]: 4.d scrub starts 2024-03-11T21:44:30.868 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:44:30 smithi092 ceph-mon[29572]: 4.d scrub ok 2024-03-11T21:44:30.868 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:44:30 smithi092 ceph-mon[29572]: 2.19 scrub starts 2024-03-11T21:44:30.868 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:44:30 smithi092 ceph-mon[29572]: 2.19 scrub ok 2024-03-11T21:44:30.933 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:44:30.958 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:44:30.959 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:44:30.959 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:44:31.060 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:44:31.084 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:44:31.084 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:44:31.084 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:44:31.185 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:44:31.210 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:44:31.211 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:44:31.211 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:44:31.257 INFO:teuthology.orchestra.run.smithi092.stdout:osd.0: {} 2024-03-11T21:44:31.258 INFO:teuthology.orchestra.run.smithi092.stdout:osd.1: {} 2024-03-11T21:44:31.280 INFO:teuthology.orchestra.run.smithi092.stdout:osd.2: {} 2024-03-11T21:44:31.302 INFO:teuthology.orchestra.run.smithi092.stdout:osd.3: {} 2024-03-11T21:44:31.312 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:44:31.321 INFO:teuthology.orchestra.run.smithi092.stdout:osd.4: {} 2024-03-11T21:44:31.337 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:44:31.338 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:44:31.338 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:44:31.340 INFO:teuthology.orchestra.run.smithi092.stdout:osd.5: {} 2024-03-11T21:44:31.360 INFO:teuthology.orchestra.run.smithi092.stdout:osd.6: {} 2024-03-11T21:44:31.381 INFO:teuthology.orchestra.run.smithi092.stdout:osd.7: {} 2024-03-11T21:44:31.382 INFO:teuthology.orchestra.run.smithi092.stderr:osd.0: osd_enable_op_tracker = '' 2024-03-11T21:44:31.382 INFO:teuthology.orchestra.run.smithi092.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-11T21:44:31.382 INFO:teuthology.orchestra.run.smithi092.stderr:osd.2: osd_enable_op_tracker = '' 2024-03-11T21:44:31.382 INFO:teuthology.orchestra.run.smithi092.stderr:osd.3: osd_enable_op_tracker = '' 2024-03-11T21:44:31.382 INFO:teuthology.orchestra.run.smithi092.stderr:osd.4: osd_enable_op_tracker = '' 2024-03-11T21:44:31.383 INFO:teuthology.orchestra.run.smithi092.stderr:osd.5: osd_enable_op_tracker = '' 2024-03-11T21:44:31.383 INFO:teuthology.orchestra.run.smithi092.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-11T21:44:31.383 INFO:teuthology.orchestra.run.smithi092.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-11T21:44:31.439 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:44:31.463 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:44:31.463 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:44:31.463 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:44:31.564 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:44:31.589 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:44:31.589 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:44:31.590 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:44:31.690 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:44:31.714 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:44:31.717 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:44:31.718 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:44:31.818 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:44:31.833 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:44:31 smithi097 ceph-mon[33317]: 6.1b scrub starts 2024-03-11T21:44:31.833 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:44:31 smithi097 ceph-mon[33317]: 6.1b scrub ok 2024-03-11T21:44:31.833 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:44:31 smithi097 ceph-mon[33317]: from='client.? v1:172.21.15.92:0/3469848350' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-11T21:44:31.833 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:44:31 smithi097 ceph-mon[33317]: 6.1f deep-scrub starts 2024-03-11T21:44:31.833 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:44:31 smithi097 ceph-mon[33317]: 6.1f deep-scrub ok 2024-03-11T21:44:31.844 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:44:31.844 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:44:31.844 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:44:31.867 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:44:31 smithi092 ceph-mon[29572]: 6.1b scrub starts 2024-03-11T21:44:31.867 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:44:31 smithi092 ceph-mon[29572]: 6.1b scrub ok 2024-03-11T21:44:31.867 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:44:31 smithi092 ceph-mon[29572]: from='client.? v1:172.21.15.92:0/3469848350' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-11T21:44:31.867 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:44:31 smithi092 ceph-mon[29572]: 6.1f deep-scrub starts 2024-03-11T21:44:31.867 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:44:31 smithi092 ceph-mon[29572]: 6.1f deep-scrub ok 2024-03-11T21:44:31.868 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:44:31 smithi092 ceph-mon[37563]: 6.1b scrub starts 2024-03-11T21:44:31.868 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:44:31 smithi092 ceph-mon[37563]: 6.1b scrub ok 2024-03-11T21:44:31.868 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:44:31 smithi092 ceph-mon[37563]: from='client.? v1:172.21.15.92:0/3469848350' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-11T21:44:31.868 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:44:31 smithi092 ceph-mon[37563]: 6.1f deep-scrub starts 2024-03-11T21:44:31.868 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:44:31 smithi092 ceph-mon[37563]: 6.1f deep-scrub ok 2024-03-11T21:44:31.945 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:44:31.969 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:44:31.970 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:44:31.971 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:44:32.071 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:44:32.096 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:44:32.096 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:44:32.097 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:44:32.191 INFO:tasks.cephadm.ceph_manager.ceph:removing pool_name unique_pool_3 2024-03-11T21:44:32.192 DEBUG:teuthology.orchestra.run.smithi092:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid 00db0016-dfed-11ee-95c8-87774f69a715 -- ceph osd pool rm unique_pool_3 unique_pool_3 --yes-i-really-really-mean-it 2024-03-11T21:44:32.197 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:44:32.221 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:44:32.222 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:44:32.222 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:44:32.323 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:44:32.347 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:44:32.347 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:44:32.348 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:44:32.448 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:44:32.473 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:44:32.474 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:44:32.474 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:44:32.575 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:44:32.599 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:44:32.599 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:44:32.600 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:44:32.644 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:44:32 smithi092 ceph-mon[37563]: pgmap v897: 157 pgs: 7 active+remapped+backfill_toofull, 150 active+clean; 8.8 MiB data, 5.0 GiB used, 621 GiB / 626 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 630 op/s; 4/990 objects misplaced (0.404%); 2.8 MiB/s, 47 objects/s recovering 2024-03-11T21:44:32.644 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:44:32 smithi092 ceph-mon[37563]: 4.10 deep-scrub starts 2024-03-11T21:44:32.644 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:44:32 smithi092 ceph-mon[37563]: 4.10 deep-scrub ok 2024-03-11T21:44:32.644 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:44:32 smithi092 ceph-mon[37563]: 5.1f scrub starts 2024-03-11T21:44:32.644 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:44:32 smithi092 ceph-mon[37563]: 5.1f scrub ok 2024-03-11T21:44:32.645 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:44:32 smithi092 ceph-mon[29572]: pgmap v897: 157 pgs: 7 active+remapped+backfill_toofull, 150 active+clean; 8.8 MiB data, 5.0 GiB used, 621 GiB / 626 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 630 op/s; 4/990 objects misplaced (0.404%); 2.8 MiB/s, 47 objects/s recovering 2024-03-11T21:44:32.645 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:44:32 smithi092 ceph-mon[29572]: 4.10 deep-scrub starts 2024-03-11T21:44:32.645 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:44:32 smithi092 ceph-mon[29572]: 4.10 deep-scrub ok 2024-03-11T21:44:32.645 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:44:32 smithi092 ceph-mon[29572]: 5.1f scrub starts 2024-03-11T21:44:32.645 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:44:32 smithi092 ceph-mon[29572]: 5.1f scrub ok 2024-03-11T21:44:32.700 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:44:32.724 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:44:32.725 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:44:32.726 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:44:32.826 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:44:32.832 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:44:32 smithi097 ceph-mon[33317]: pgmap v897: 157 pgs: 7 active+remapped+backfill_toofull, 150 active+clean; 8.8 MiB data, 5.0 GiB used, 621 GiB / 626 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 630 op/s; 4/990 objects misplaced (0.404%); 2.8 MiB/s, 47 objects/s recovering 2024-03-11T21:44:32.833 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:44:32 smithi097 ceph-mon[33317]: 4.10 deep-scrub starts 2024-03-11T21:44:32.833 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:44:32 smithi097 ceph-mon[33317]: 4.10 deep-scrub ok 2024-03-11T21:44:32.833 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:44:32 smithi097 ceph-mon[33317]: 5.1f scrub starts 2024-03-11T21:44:32.833 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:44:32 smithi097 ceph-mon[33317]: 5.1f scrub ok 2024-03-11T21:44:32.850 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:44:32.851 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:44:32.851 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:44:32.953 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:44:32.977 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:44:32.977 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:44:32.977 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:44:32.988 INFO:teuthology.orchestra.run.smithi092.stderr:Inferring config /var/lib/ceph/00db0016-dfed-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T21:44:32.988 INFO:teuthology.orchestra.run.smithi092.stderr:Inferring config /var/lib/ceph/00db0016-dfed-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T21:44:33.078 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:44:33.102 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:44:33.102 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:44:33.102 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:44:33.203 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:44:33.226 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:44:33.227 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:44:33.227 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:44:33.328 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:44:33.332 INFO:journalctl@ceph.iscsi.iscsi.a.smithi097.stdout:Mar 11 21:44:33 smithi097 ceph-00db0016-dfed-11ee-95c8-87774f69a715-iscsi-iscsi-a[60943]: debug there is no tcmu-runner data available 2024-03-11T21:44:33.354 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:44:33.355 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:44:33.355 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:44:33.456 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:44:33.481 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:44:33.482 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:44:33.482 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:44:33.583 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:44:33.608 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:44:33.608 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:44:33.609 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:44:33.701 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:44:33 smithi092 ceph-mon[29572]: 5.14 scrub starts 2024-03-11T21:44:33.701 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:44:33 smithi092 ceph-mon[29572]: 5.14 scrub ok 2024-03-11T21:44:33.701 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:44:33 smithi092 ceph-mon[29572]: 6.f scrub starts 2024-03-11T21:44:33.701 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:44:33 smithi092 ceph-mon[29572]: 6.f scrub ok 2024-03-11T21:44:33.701 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:44:33 smithi092 ceph-mon[29572]: 6.d scrub starts 2024-03-11T21:44:33.701 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:44:33 smithi092 ceph-mon[29572]: 6.d scrub ok 2024-03-11T21:44:33.701 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:44:33 smithi092 ceph-mon[29572]: from='client.24469 v1:172.21.15.97:0/404208310' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:44:33.702 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:44:33 smithi092 ceph-mon[37563]: 5.14 scrub starts 2024-03-11T21:44:33.702 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:44:33 smithi092 ceph-mon[37563]: 5.14 scrub ok 2024-03-11T21:44:33.702 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:44:33 smithi092 ceph-mon[37563]: 6.f scrub starts 2024-03-11T21:44:33.702 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:44:33 smithi092 ceph-mon[37563]: 6.f scrub ok 2024-03-11T21:44:33.702 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:44:33 smithi092 ceph-mon[37563]: 6.d scrub starts 2024-03-11T21:44:33.702 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:44:33 smithi092 ceph-mon[37563]: 6.d scrub ok 2024-03-11T21:44:33.702 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:44:33 smithi092 ceph-mon[37563]: from='client.24469 v1:172.21.15.97:0/404208310' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:44:33.710 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:44:33.734 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:44:33.735 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:44:33.735 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:44:33.833 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:44:33 smithi097 ceph-mon[33317]: 5.14 scrub starts 2024-03-11T21:44:33.833 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:44:33 smithi097 ceph-mon[33317]: 5.14 scrub ok 2024-03-11T21:44:33.834 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:44:33 smithi097 ceph-mon[33317]: 6.f scrub starts 2024-03-11T21:44:33.834 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:44:33 smithi097 ceph-mon[33317]: 6.f scrub ok 2024-03-11T21:44:33.834 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:44:33 smithi097 ceph-mon[33317]: 6.d scrub starts 2024-03-11T21:44:33.834 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:44:33 smithi097 ceph-mon[33317]: 6.d scrub ok 2024-03-11T21:44:33.834 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:44:33 smithi097 ceph-mon[33317]: from='client.24469 v1:172.21.15.97:0/404208310' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:44:33.836 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:44:33.860 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:44:33.861 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:44:33.861 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:44:33.962 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:44:33.987 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:44:33.988 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:44:33.988 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:44:34.003 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:44:33 smithi092 ceph-00db0016-dfed-11ee-95c8-87774f69a715-mon-a[29568]: 2024-03-11T21:44:33.789+0000 7f87a4389700 -1 mon.a@0(leader).osd e436 definitely_dead 0 2024-03-11T21:44:34.089 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:44:34.113 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:44:34.113 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:44:34.114 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:44:34.215 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:44:34.239 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:44:34.239 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:44:34.240 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:44:34.341 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:44:34.365 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:44:34.366 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:44:34.366 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:44:34.467 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:44:34.490 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:44:34.491 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:44:34.492 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:44:34.591 INFO:teuthology.orchestra.run.smithi092.stderr:marked out osd.4. 2024-03-11T21:44:34.593 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:44:34.622 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:44:34.623 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:44:34.623 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:44:34.724 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:44:34.748 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:44:34.748 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:44:34.749 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:44:34.833 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:44:34 smithi097 ceph-mon[33317]: pgmap v898: 157 pgs: 7 active+remapped+backfill_toofull, 150 active+clean; 582 KiB data, 4.9 GiB used, 621 GiB / 626 GiB avail; 571 B/s rd, 0 B/s wr, 540 op/s; 2.5 MiB/s, 40 objects/s recovering 2024-03-11T21:44:34.833 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:44:34 smithi097 ceph-mon[33317]: from='client.? v1:172.21.15.92:0/2037949787' entity='client.admin' cmd=[{"prefix": "osd out", "ids": ["4"]}]: dispatch 2024-03-11T21:44:34.833 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:44:34 smithi097 ceph-mon[33317]: from='client.35750 ' entity='client.admin' cmd=[{"prefix": "osd out", "ids": ["4"]}]: dispatch 2024-03-11T21:44:34.833 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:44:34 smithi097 ceph-mon[33317]: Client client.admin marked osd.4 out, while it was still marked up 2024-03-11T21:44:34.833 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:44:34 smithi097 ceph-mon[33317]: 2.14 scrub starts 2024-03-11T21:44:34.833 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:44:34 smithi097 ceph-mon[33317]: 2.14 scrub ok 2024-03-11T21:44:34.849 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:44:34.867 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:44:34 smithi092 ceph-mon[29572]: pgmap v898: 157 pgs: 7 active+remapped+backfill_toofull, 150 active+clean; 582 KiB data, 4.9 GiB used, 621 GiB / 626 GiB avail; 571 B/s rd, 0 B/s wr, 540 op/s; 2.5 MiB/s, 40 objects/s recovering 2024-03-11T21:44:34.867 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:44:34 smithi092 ceph-mon[29572]: from='client.? v1:172.21.15.92:0/2037949787' entity='client.admin' cmd=[{"prefix": "osd out", "ids": ["4"]}]: dispatch 2024-03-11T21:44:34.867 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:44:34 smithi092 ceph-mon[29572]: from='client.35750 ' entity='client.admin' cmd=[{"prefix": "osd out", "ids": ["4"]}]: dispatch 2024-03-11T21:44:34.867 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:44:34 smithi092 ceph-mon[29572]: Client client.admin marked osd.4 out, while it was still marked up 2024-03-11T21:44:34.867 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:44:34 smithi092 ceph-mon[29572]: 2.14 scrub starts 2024-03-11T21:44:34.867 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:44:34 smithi092 ceph-mon[29572]: 2.14 scrub ok 2024-03-11T21:44:34.868 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:44:34 smithi092 ceph-mon[37563]: pgmap v898: 157 pgs: 7 active+remapped+backfill_toofull, 150 active+clean; 582 KiB data, 4.9 GiB used, 621 GiB / 626 GiB avail; 571 B/s rd, 0 B/s wr, 540 op/s; 2.5 MiB/s, 40 objects/s recovering 2024-03-11T21:44:34.868 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:44:34 smithi092 ceph-mon[37563]: from='client.? v1:172.21.15.92:0/2037949787' entity='client.admin' cmd=[{"prefix": "osd out", "ids": ["4"]}]: dispatch 2024-03-11T21:44:34.868 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:44:34 smithi092 ceph-mon[37563]: from='client.35750 ' entity='client.admin' cmd=[{"prefix": "osd out", "ids": ["4"]}]: dispatch 2024-03-11T21:44:34.868 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:44:34 smithi092 ceph-mon[37563]: Client client.admin marked osd.4 out, while it was still marked up 2024-03-11T21:44:34.868 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:44:34 smithi092 ceph-mon[37563]: 2.14 scrub starts 2024-03-11T21:44:34.868 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:44:34 smithi092 ceph-mon[37563]: 2.14 scrub ok 2024-03-11T21:44:34.874 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:44:34.875 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:44:34.876 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:44:34.947 DEBUG:teuthology.orchestra.run.smithi092:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid 00db0016-dfed-11ee-95c8-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-03-11T21:44:34.976 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:44:35.016 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:44:35.017 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:44:35.017 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:44:35.118 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:44:35.142 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:44:35.143 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:44:35.143 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:44:35.244 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:44:35.268 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:44:35.269 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:44:35.269 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:44:35.370 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:44:35.394 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:44:35.395 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:44:35.395 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:44:35.496 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:44:35.520 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:44:35.521 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:44:35.521 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:44:35.622 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:44:35.661 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:44:35.662 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:44:35.662 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:44:35.763 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:44:35.787 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:44:35.788 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:44:35.788 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:44:35.833 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:44:35 smithi097 ceph-mon[33317]: 4.3 scrub starts 2024-03-11T21:44:35.833 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:44:35 smithi097 ceph-mon[33317]: 4.3 scrub ok 2024-03-11T21:44:35.833 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:44:35 smithi097 ceph-mon[33317]: from='client.35750 ' entity='client.admin' cmd='[{"prefix": "osd out", "ids": ["4"]}]': finished 2024-03-11T21:44:35.833 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:44:35 smithi097 ceph-mon[33317]: osdmap e437: 8 total, 8 up, 6 in 2024-03-11T21:44:35.833 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:44:35 smithi097 ceph-mon[33317]: 5.2 scrub starts 2024-03-11T21:44:35.833 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:44:35 smithi097 ceph-mon[33317]: 5.2 scrub ok 2024-03-11T21:44:35.867 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:44:35 smithi092 ceph-mon[37563]: 4.3 scrub starts 2024-03-11T21:44:35.867 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:44:35 smithi092 ceph-mon[37563]: 4.3 scrub ok 2024-03-11T21:44:35.867 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:44:35 smithi092 ceph-mon[37563]: from='client.35750 ' entity='client.admin' cmd='[{"prefix": "osd out", "ids": ["4"]}]': finished 2024-03-11T21:44:35.867 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:44:35 smithi092 ceph-mon[37563]: osdmap e437: 8 total, 8 up, 6 in 2024-03-11T21:44:35.867 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:44:35 smithi092 ceph-mon[37563]: 5.2 scrub starts 2024-03-11T21:44:35.867 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:44:35 smithi092 ceph-mon[37563]: 5.2 scrub ok 2024-03-11T21:44:35.867 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:44:35 smithi092 ceph-mon[29572]: 4.3 scrub starts 2024-03-11T21:44:35.868 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:44:35 smithi092 ceph-mon[29572]: 4.3 scrub ok 2024-03-11T21:44:35.868 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:44:35 smithi092 ceph-mon[29572]: from='client.35750 ' entity='client.admin' cmd='[{"prefix": "osd out", "ids": ["4"]}]': finished 2024-03-11T21:44:35.868 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:44:35 smithi092 ceph-mon[29572]: osdmap e437: 8 total, 8 up, 6 in 2024-03-11T21:44:35.868 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:44:35 smithi092 ceph-mon[29572]: 5.2 scrub starts 2024-03-11T21:44:35.868 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:44:35 smithi092 ceph-mon[29572]: 5.2 scrub ok 2024-03-11T21:44:35.874 INFO:teuthology.orchestra.run.smithi092.stderr:Inferring config /var/lib/ceph/00db0016-dfed-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T21:44:35.889 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:44:35.913 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:44:35.914 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:44:35.914 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:44:36.015 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:44:36.039 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:44:36.040 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:44:36.040 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:44:36.141 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:44:36.165 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:44:36.165 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:44:36.166 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:44:36.266 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:44:36.290 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:44:36.291 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:44:36.291 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:44:36.392 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:44:36.415 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:44:36.416 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:44:36.416 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:44:36.517 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:44:36.541 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:44:36.542 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:44:36.542 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:44:36.643 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:44:36.704 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:44:36.705 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:44:36.705 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:44:36.806 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:44:36.830 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:44:36.831 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:44:36.831 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:44:36.833 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:44:36 smithi097 ceph-mon[33317]: pgmap v900: 157 pgs: 9 peering, 7 active+remapped+backfill_toofull, 141 active+clean; 582 KiB data, 4.2 GiB used, 532 GiB / 536 GiB avail; 0 B/s wr, 465 op/s; 1.4 MiB/s, 22 objects/s recovering 2024-03-11T21:44:36.833 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:44:36 smithi097 ceph-mon[33317]: 6.b deep-scrub starts 2024-03-11T21:44:36.833 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:44:36 smithi097 ceph-mon[33317]: Health check failed: Reduced data availability: 2 pgs peering (PG_AVAILABILITY) 2024-03-11T21:44:36.833 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:44:36 smithi097 ceph-mon[33317]: osdmap e438: 8 total, 8 up, 6 in 2024-03-11T21:44:36.833 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:44:36 smithi097 ceph-mon[33317]: 10.f starting backfill to osd.1 from (0'0,0'0] MAX to 436'14004 2024-03-11T21:44:36.833 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:44:36 smithi097 ceph-mon[33317]: 10.3 starting backfill to osd.0 from (0'0,0'0] MAX to 436'8927 2024-03-11T21:44:36.833 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:44:36 smithi097 ceph-mon[33317]: 6.b deep-scrub ok 2024-03-11T21:44:36.833 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:44:36 smithi097 ceph-mon[33317]: 5.e scrub starts 2024-03-11T21:44:36.833 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:44:36 smithi097 ceph-mon[33317]: 5.e scrub ok 2024-03-11T21:44:36.867 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:44:36 smithi092 ceph-mon[29572]: pgmap v900: 157 pgs: 9 peering, 7 active+remapped+backfill_toofull, 141 active+clean; 582 KiB data, 4.2 GiB used, 532 GiB / 536 GiB avail; 0 B/s wr, 465 op/s; 1.4 MiB/s, 22 objects/s recovering 2024-03-11T21:44:36.867 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:44:36 smithi092 ceph-mon[29572]: 6.b deep-scrub starts 2024-03-11T21:44:36.867 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:44:36 smithi092 ceph-mon[29572]: Health check failed: Reduced data availability: 2 pgs peering (PG_AVAILABILITY) 2024-03-11T21:44:36.867 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:44:36 smithi092 ceph-mon[29572]: osdmap e438: 8 total, 8 up, 6 in 2024-03-11T21:44:36.867 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:44:36 smithi092 ceph-mon[29572]: 10.f starting backfill to osd.1 from (0'0,0'0] MAX to 436'14004 2024-03-11T21:44:36.867 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:44:36 smithi092 ceph-mon[29572]: 10.3 starting backfill to osd.0 from (0'0,0'0] MAX to 436'8927 2024-03-11T21:44:36.867 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:44:36 smithi092 ceph-mon[29572]: 6.b deep-scrub ok 2024-03-11T21:44:36.868 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:44:36 smithi092 ceph-mon[29572]: 5.e scrub starts 2024-03-11T21:44:36.868 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:44:36 smithi092 ceph-mon[29572]: 5.e scrub ok 2024-03-11T21:44:36.868 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:44:36 smithi092 ceph-mon[37563]: pgmap v900: 157 pgs: 9 peering, 7 active+remapped+backfill_toofull, 141 active+clean; 582 KiB data, 4.2 GiB used, 532 GiB / 536 GiB avail; 0 B/s wr, 465 op/s; 1.4 MiB/s, 22 objects/s recovering 2024-03-11T21:44:36.868 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:44:36 smithi092 ceph-mon[37563]: 6.b deep-scrub starts 2024-03-11T21:44:36.868 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:44:36 smithi092 ceph-mon[37563]: Health check failed: Reduced data availability: 2 pgs peering (PG_AVAILABILITY) 2024-03-11T21:44:36.868 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:44:36 smithi092 ceph-mon[37563]: osdmap e438: 8 total, 8 up, 6 in 2024-03-11T21:44:36.868 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:44:36 smithi092 ceph-mon[37563]: 10.f starting backfill to osd.1 from (0'0,0'0] MAX to 436'14004 2024-03-11T21:44:36.868 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:44:36 smithi092 ceph-mon[37563]: 10.3 starting backfill to osd.0 from (0'0,0'0] MAX to 436'8927 2024-03-11T21:44:36.868 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:44:36 smithi092 ceph-mon[37563]: 6.b deep-scrub ok 2024-03-11T21:44:36.868 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:44:36 smithi092 ceph-mon[37563]: 5.e scrub starts 2024-03-11T21:44:36.868 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:44:36 smithi092 ceph-mon[37563]: 5.e scrub ok 2024-03-11T21:44:36.933 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:44:36.956 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:44:36.957 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:44:36.958 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:44:37.058 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:44:37.081 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:44:37.082 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:44:37.083 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:44:37.183 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:44:37.207 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:44:37.208 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:44:37.208 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:44:37.309 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:44:37.332 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:44:37.333 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:44:37.333 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:44:37.434 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:44:37.460 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:44:37.461 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:44:37.461 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:44:37.562 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:44:37.586 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:44:37.588 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:44:37 smithi092 ceph-mon[37563]: osdmap e439: 8 total, 8 up, 6 in 2024-03-11T21:44:37.588 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:44:37 smithi092 ceph-mon[37563]: 10.2 starting backfill to osd.6 from (0'0,0'0] MAX to 436'9174 2024-03-11T21:44:37.588 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:44:37 smithi092 ceph-mon[37563]: 10.10 starting backfill to osd.7 from (0'0,0'0] MAX to 436'10624 2024-03-11T21:44:37.588 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:44:37 smithi092 ceph-mon[37563]: 10.d starting backfill to osd.7 from (0'0,0'0] MAX to 436'13674 2024-03-11T21:44:37.588 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:44:37 smithi092 ceph-mon[37563]: 10.4 starting backfill to osd.2 from (0'0,0'0] MAX to 436'8724 2024-03-11T21:44:37.588 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:44:37 smithi092 ceph-mon[37563]: 10.12 continuing backfill to osd.0 from (378'4614,436'9222] MIN to 436'9222 2024-03-11T21:44:37.588 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:44:37 smithi092 ceph-mon[37563]: 10.12 continuing backfill to osd.1 from (378'4614,436'9222] MIN to 436'9222 2024-03-11T21:44:37.589 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:44:37 smithi092 ceph-mon[37563]: 10.c starting backfill to osd.6 from (0'0,0'0] MAX to 436'16116 2024-03-11T21:44:37.589 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:44:37 smithi092 ceph-mon[37563]: 10.14 continuing backfill to osd.0 from (381'4591,436'8929] MIN to 436'8929 2024-03-11T21:44:37.589 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:44:37 smithi092 ceph-mon[37563]: 10.11 starting backfill to osd.1 from (0'0,0'0] MAX to 436'10677 2024-03-11T21:44:37.589 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:44:37 smithi092 ceph-mon[37563]: 10.1 starting backfill to osd.1 from (0'0,0'0] MAX to 436'9170 2024-03-11T21:44:37.589 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:44:37 smithi092 ceph-mon[37563]: 10.16 starting backfill to osd.1 from (0'0,0'0] MAX to 436'8902 2024-03-11T21:44:37.589 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:44:37 smithi092 ceph-mon[37563]: 10.e starting backfill to osd.5 from (0'0,0'0] MAX to 436'13488 2024-03-11T21:44:37.589 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:44:37 smithi092 ceph-mon[37563]: 10.13 starting backfill to osd.0 from (0'0,0'0] MAX to 436'8961 2024-03-11T21:44:37.589 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:44:37 smithi092 ceph-mon[37563]: 10.6 starting backfill to osd.1 from (0'0,0'0] MAX to 436'9073 2024-03-11T21:44:37.589 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:44:37 smithi092 ceph-mon[37563]: 10.0 starting backfill to osd.7 from (0'0,0'0] MAX to 436'9012 2024-03-11T21:44:37.589 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:44:37 smithi092 ceph-mon[37563]: 2.1e scrub starts 2024-03-11T21:44:37.589 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:44:37 smithi092 ceph-mon[37563]: 2.1e scrub ok 2024-03-11T21:44:37.590 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:44:37.590 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:44:37.691 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:44:37.765 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:44:37.765 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:44:37.765 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:44:37.869 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:44:37.870 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:44:37 smithi092 ceph-mon[29572]: osdmap e439: 8 total, 8 up, 6 in 2024-03-11T21:44:37.870 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:44:37 smithi092 ceph-mon[29572]: 10.2 starting backfill to osd.6 from (0'0,0'0] MAX to 436'9174 2024-03-11T21:44:37.871 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:44:37 smithi092 ceph-mon[29572]: 10.10 starting backfill to osd.7 from (0'0,0'0] MAX to 436'10624 2024-03-11T21:44:37.871 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:44:37 smithi092 ceph-mon[29572]: 10.d starting backfill to osd.7 from (0'0,0'0] MAX to 436'13674 2024-03-11T21:44:37.871 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:44:37 smithi092 ceph-mon[29572]: 10.4 starting backfill to osd.2 from (0'0,0'0] MAX to 436'8724 2024-03-11T21:44:37.871 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:44:37 smithi092 ceph-mon[29572]: 10.12 continuing backfill to osd.0 from (378'4614,436'9222] MIN to 436'9222 2024-03-11T21:44:37.871 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:44:37 smithi092 ceph-mon[29572]: 10.12 continuing backfill to osd.1 from (378'4614,436'9222] MIN to 436'9222 2024-03-11T21:44:37.871 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:44:37 smithi092 ceph-mon[29572]: 10.c starting backfill to osd.6 from (0'0,0'0] MAX to 436'16116 2024-03-11T21:44:37.871 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:44:37 smithi092 ceph-mon[29572]: 10.14 continuing backfill to osd.0 from (381'4591,436'8929] MIN to 436'8929 2024-03-11T21:44:37.871 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:44:37 smithi092 ceph-mon[29572]: 10.11 starting backfill to osd.1 from (0'0,0'0] MAX to 436'10677 2024-03-11T21:44:37.871 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:44:37 smithi092 ceph-mon[29572]: 10.1 starting backfill to osd.1 from (0'0,0'0] MAX to 436'9170 2024-03-11T21:44:37.871 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:44:37 smithi092 ceph-mon[29572]: 10.16 starting backfill to osd.1 from (0'0,0'0] MAX to 436'8902 2024-03-11T21:44:37.871 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:44:37 smithi092 ceph-mon[29572]: 10.e starting backfill to osd.5 from (0'0,0'0] MAX to 436'13488 2024-03-11T21:44:37.871 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:44:37 smithi092 ceph-mon[29572]: 10.13 starting backfill to osd.0 from (0'0,0'0] MAX to 436'8961 2024-03-11T21:44:37.871 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:44:37 smithi092 ceph-mon[29572]: 10.6 starting backfill to osd.1 from (0'0,0'0] MAX to 436'9073 2024-03-11T21:44:37.872 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:44:37 smithi092 ceph-mon[29572]: 10.0 starting backfill to osd.7 from (0'0,0'0] MAX to 436'9012 2024-03-11T21:44:37.872 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:44:37 smithi092 ceph-mon[29572]: 2.1e scrub starts 2024-03-11T21:44:37.872 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:44:37 smithi092 ceph-mon[29572]: 2.1e scrub ok 2024-03-11T21:44:37.895 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:44:37.896 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:44:37.896 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:44:37.997 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:44:38.020 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:44:38.021 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:44:38.022 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:44:38.083 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:44:37 smithi097 ceph-mon[33317]: osdmap e439: 8 total, 8 up, 6 in 2024-03-11T21:44:38.083 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:44:37 smithi097 ceph-mon[33317]: 10.2 starting backfill to osd.6 from (0'0,0'0] MAX to 436'9174 2024-03-11T21:44:38.083 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:44:37 smithi097 ceph-mon[33317]: 10.10 starting backfill to osd.7 from (0'0,0'0] MAX to 436'10624 2024-03-11T21:44:38.083 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:44:37 smithi097 ceph-mon[33317]: 10.d starting backfill to osd.7 from (0'0,0'0] MAX to 436'13674 2024-03-11T21:44:38.083 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:44:37 smithi097 ceph-mon[33317]: 10.4 starting backfill to osd.2 from (0'0,0'0] MAX to 436'8724 2024-03-11T21:44:38.083 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:44:37 smithi097 ceph-mon[33317]: 10.12 continuing backfill to osd.0 from (378'4614,436'9222] MIN to 436'9222 2024-03-11T21:44:38.084 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:44:37 smithi097 ceph-mon[33317]: 10.12 continuing backfill to osd.1 from (378'4614,436'9222] MIN to 436'9222 2024-03-11T21:44:38.084 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:44:37 smithi097 ceph-mon[33317]: 10.c starting backfill to osd.6 from (0'0,0'0] MAX to 436'16116 2024-03-11T21:44:38.084 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:44:37 smithi097 ceph-mon[33317]: 10.14 continuing backfill to osd.0 from (381'4591,436'8929] MIN to 436'8929 2024-03-11T21:44:38.084 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:44:37 smithi097 ceph-mon[33317]: 10.11 starting backfill to osd.1 from (0'0,0'0] MAX to 436'10677 2024-03-11T21:44:38.084 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:44:37 smithi097 ceph-mon[33317]: 10.1 starting backfill to osd.1 from (0'0,0'0] MAX to 436'9170 2024-03-11T21:44:38.084 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:44:37 smithi097 ceph-mon[33317]: 10.16 starting backfill to osd.1 from (0'0,0'0] MAX to 436'8902 2024-03-11T21:44:38.084 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:44:37 smithi097 ceph-mon[33317]: 10.e starting backfill to osd.5 from (0'0,0'0] MAX to 436'13488 2024-03-11T21:44:38.084 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:44:37 smithi097 ceph-mon[33317]: 10.13 starting backfill to osd.0 from (0'0,0'0] MAX to 436'8961 2024-03-11T21:44:38.084 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:44:37 smithi097 ceph-mon[33317]: 10.6 starting backfill to osd.1 from (0'0,0'0] MAX to 436'9073 2024-03-11T21:44:38.084 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:44:37 smithi097 ceph-mon[33317]: 10.0 starting backfill to osd.7 from (0'0,0'0] MAX to 436'9012 2024-03-11T21:44:38.084 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:44:37 smithi097 ceph-mon[33317]: 2.1e scrub starts 2024-03-11T21:44:38.084 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:44:37 smithi097 ceph-mon[33317]: 2.1e scrub ok 2024-03-11T21:44:38.123 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:44:38.146 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:44:38.147 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:44:38.147 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:44:38.162 INFO:journalctl@ceph.mgr.y.smithi092.stdout:Mar 11 21:44:37 smithi092 ceph-00db0016-dfed-11ee-95c8-87774f69a715-mgr-y[29801]: ::ffff:172.21.15.97 - - [11/Mar/2024:21:44:37] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-11T21:44:38.248 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:44:38.274 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:44:38.274 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:44:38.277 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:44:38.378 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:44:38.403 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:44:38.403 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:44:38.403 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:44:38.504 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:44:38.536 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:44:38.536 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:44:38.537 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:44:38.637 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:44:38.661 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:44:38.661 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:44:38.662 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:44:38.762 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:44:38.786 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:44:38.787 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:44:38.787 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:44:38.850 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:44:38 smithi092 ceph-mon[29572]: pgmap v903: 157 pgs: 2 active+clean+remapped, 9 peering, 7 active+remapped+backfill_toofull, 139 active+clean; 582 KiB data, 4.2 GiB used, 532 GiB / 536 GiB avail; 0 B/s wr, 21 op/s; 59 B/s, 1 objects/s recovering 2024-03-11T21:44:38.850 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:44:38 smithi092 ceph-mon[29572]: osdmap e440: 8 total, 8 up, 6 in 2024-03-11T21:44:38.850 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:44:38 smithi092 ceph-mon[29572]: from='mgr.14607 ' entity='mgr.y' 2024-03-11T21:44:38.851 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:44:38 smithi092 ceph-mon[29572]: from='mgr.14607 v1:172.21.15.92:0/1702717597' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T21:44:38.851 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:44:38 smithi092 ceph-mon[29572]: osdmap e441: 8 total, 8 up, 6 in 2024-03-11T21:44:38.851 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:44:38 smithi092 ceph-mon[37563]: pgmap v903: 157 pgs: 2 active+clean+remapped, 9 peering, 7 active+remapped+backfill_toofull, 139 active+clean; 582 KiB data, 4.2 GiB used, 532 GiB / 536 GiB avail; 0 B/s wr, 21 op/s; 59 B/s, 1 objects/s recovering 2024-03-11T21:44:38.851 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:44:38 smithi092 ceph-mon[37563]: osdmap e440: 8 total, 8 up, 6 in 2024-03-11T21:44:38.851 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:44:38 smithi092 ceph-mon[37563]: from='mgr.14607 ' entity='mgr.y' 2024-03-11T21:44:38.851 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:44:38 smithi092 ceph-mon[37563]: from='mgr.14607 v1:172.21.15.92:0/1702717597' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T21:44:38.851 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:44:38 smithi092 ceph-mon[37563]: osdmap e441: 8 total, 8 up, 6 in 2024-03-11T21:44:38.888 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:44:38.912 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:44:38.913 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:44:38.913 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:44:39.014 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:44:39.038 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:44:39.039 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:44:39.039 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:44:39.083 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:44:38 smithi097 ceph-mon[33317]: pgmap v903: 157 pgs: 2 active+clean+remapped, 9 peering, 7 active+remapped+backfill_toofull, 139 active+clean; 582 KiB data, 4.2 GiB used, 532 GiB / 536 GiB avail; 0 B/s wr, 21 op/s; 59 B/s, 1 objects/s recovering 2024-03-11T21:44:39.083 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:44:38 smithi097 ceph-mon[33317]: osdmap e440: 8 total, 8 up, 6 in 2024-03-11T21:44:39.083 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:44:38 smithi097 ceph-mon[33317]: from='mgr.14607 ' entity='mgr.y' 2024-03-11T21:44:39.083 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:44:38 smithi097 ceph-mon[33317]: from='mgr.14607 v1:172.21.15.92:0/1702717597' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T21:44:39.083 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:44:38 smithi097 ceph-mon[33317]: osdmap e441: 8 total, 8 up, 6 in 2024-03-11T21:44:39.140 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:44:39.164 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:44:39.164 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:44:39.165 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:44:39.266 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:44:39.290 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:44:39.291 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:44:39.291 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:44:39.392 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:44:39.416 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:44:39.417 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:44:39.417 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:44:39.518 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:44:39.542 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:44:39.543 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:44:39.543 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:44:39.644 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:44:39.669 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:44:39.669 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:44:39.670 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:44:39.736 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:44:39 smithi092 ceph-mon[29572]: 6.1e scrub starts 2024-03-11T21:44:39.736 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:44:39 smithi092 ceph-mon[29572]: 6.1e scrub ok 2024-03-11T21:44:39.736 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:44:39 smithi092 ceph-mon[37563]: 6.1e scrub starts 2024-03-11T21:44:39.736 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:44:39 smithi092 ceph-mon[37563]: 6.1e scrub ok 2024-03-11T21:44:39.770 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:44:39.794 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:44:39.795 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:44:39.795 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:44:39.896 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:44:39.921 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:44:39.922 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:44:39.922 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:44:39.944 INFO:teuthology.orchestra.run.smithi092.stdout:ERROR: (22) Invalid argument 2024-03-11T21:44:39.944 INFO:teuthology.orchestra.run.smithi092.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-11T21:44:40.023 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:44:40.047 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:44:40.048 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:44:40.048 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:44:40.083 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:44:39 smithi097 ceph-mon[33317]: 6.1e scrub starts 2024-03-11T21:44:40.083 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:44:39 smithi097 ceph-mon[33317]: 6.1e scrub ok 2024-03-11T21:44:40.143 INFO:teuthology.orchestra.run.smithi092.stderr:Inferring config /var/lib/ceph/00db0016-dfed-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T21:44:40.149 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:44:40.173 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:44:40.174 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:44:40.174 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:44:40.275 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:44:40.300 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:44:40.300 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:44:40.300 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:44:40.401 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:44:40.425 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:44:40.426 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:44:40.426 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:44:40.527 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:44:40.551 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:44:40.552 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:44:40.552 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:44:40.615 INFO:teuthology.orchestra.run.smithi092.stderr:pool 'unique_pool_3' removed 2024-03-11T21:44:40.623 INFO:teuthology.orchestra.run.smithi092.stderr:noscrub is set 2024-03-11T21:44:40.652 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:44:40.676 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:44:40.676 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:44:40.677 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:44:40.702 DEBUG:teuthology.orchestra.run.smithi092:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid 00db0016-dfed-11ee-95c8-87774f69a715 -- ceph daemon osd.1 dump_blocked_ops 2024-03-11T21:44:40.777 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:44:40.801 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:44:40.801 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:44:40.802 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:44:40.867 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:44:40 smithi092 ceph-mon[37563]: pgmap v906: 157 pgs: 5 active+clean+remapped, 10 peering, 11 active+remapped+backfill_toofull, 131 active+clean; 582 KiB data, 4.1 GiB used, 532 GiB / 536 GiB avail; 359 B/s, 13 objects/s recovering 2024-03-11T21:44:40.867 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:44:40 smithi092 ceph-mon[37563]: 6.17 scrub starts 2024-03-11T21:44:40.867 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:44:40 smithi092 ceph-mon[37563]: 4.17 scrub starts 2024-03-11T21:44:40.868 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:44:40 smithi092 ceph-mon[37563]: 4.17 scrub ok 2024-03-11T21:44:40.868 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:44:40 smithi092 ceph-mon[37563]: 6.17 scrub ok 2024-03-11T21:44:40.868 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:44:40 smithi092 ceph-mon[37563]: Health check update: Low space hindering backfill (add storage if this doesn't resolve itself): 11 pgs backfill_toofull (PG_BACKFILL_FULL) 2024-03-11T21:44:40.868 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:44:40 smithi092 ceph-mon[37563]: from='client.? v1:172.21.15.92:0/585342648' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "unique_pool_3", "pool2": "unique_pool_3", "yes_i_really_really_mean_it": true}]: dispatch 2024-03-11T21:44:40.868 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:44:40 smithi092 ceph-mon[37563]: from='client.? v1:172.21.15.92:0/1260236674' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-11T21:44:40.868 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:44:40 smithi092 ceph-mon[37563]: 6.9 scrub starts 2024-03-11T21:44:40.868 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:44:40 smithi092 ceph-mon[37563]: 6.9 scrub ok 2024-03-11T21:44:40.868 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:44:40 smithi092 ceph-mon[29572]: pgmap v906: 157 pgs: 5 active+clean+remapped, 10 peering, 11 active+remapped+backfill_toofull, 131 active+clean; 582 KiB data, 4.1 GiB used, 532 GiB / 536 GiB avail; 359 B/s, 13 objects/s recovering 2024-03-11T21:44:40.868 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:44:40 smithi092 ceph-mon[29572]: 6.17 scrub starts 2024-03-11T21:44:40.868 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:44:40 smithi092 ceph-mon[29572]: 4.17 scrub starts 2024-03-11T21:44:40.868 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:44:40 smithi092 ceph-mon[29572]: 4.17 scrub ok 2024-03-11T21:44:40.869 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:44:40 smithi092 ceph-mon[29572]: 6.17 scrub ok 2024-03-11T21:44:40.869 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:44:40 smithi092 ceph-mon[29572]: Health check update: Low space hindering backfill (add storage if this doesn't resolve itself): 11 pgs backfill_toofull (PG_BACKFILL_FULL) 2024-03-11T21:44:40.869 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:44:40 smithi092 ceph-mon[29572]: from='client.? v1:172.21.15.92:0/585342648' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "unique_pool_3", "pool2": "unique_pool_3", "yes_i_really_really_mean_it": true}]: dispatch 2024-03-11T21:44:40.869 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:44:40 smithi092 ceph-mon[29572]: from='client.? v1:172.21.15.92:0/1260236674' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-11T21:44:40.869 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:44:40 smithi092 ceph-mon[29572]: 6.9 scrub starts 2024-03-11T21:44:40.869 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:44:40 smithi092 ceph-mon[29572]: 6.9 scrub ok 2024-03-11T21:44:40.902 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:44:40.932 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:44:40.932 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:44:40.933 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:44:41.033 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:44:41.061 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:44:41.061 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:44:41.062 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:44:41.083 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:44:40 smithi097 ceph-mon[33317]: pgmap v906: 157 pgs: 5 active+clean+remapped, 10 peering, 11 active+remapped+backfill_toofull, 131 active+clean; 582 KiB data, 4.1 GiB used, 532 GiB / 536 GiB avail; 359 B/s, 13 objects/s recovering 2024-03-11T21:44:41.083 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:44:40 smithi097 ceph-mon[33317]: 6.17 scrub starts 2024-03-11T21:44:41.083 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:44:40 smithi097 ceph-mon[33317]: 4.17 scrub starts 2024-03-11T21:44:41.083 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:44:40 smithi097 ceph-mon[33317]: 4.17 scrub ok 2024-03-11T21:44:41.083 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:44:40 smithi097 ceph-mon[33317]: 6.17 scrub ok 2024-03-11T21:44:41.083 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:44:40 smithi097 ceph-mon[33317]: Health check update: Low space hindering backfill (add storage if this doesn't resolve itself): 11 pgs backfill_toofull (PG_BACKFILL_FULL) 2024-03-11T21:44:41.083 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:44:40 smithi097 ceph-mon[33317]: from='client.? v1:172.21.15.92:0/585342648' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "unique_pool_3", "pool2": "unique_pool_3", "yes_i_really_really_mean_it": true}]: dispatch 2024-03-11T21:44:41.083 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:44:40 smithi097 ceph-mon[33317]: from='client.? v1:172.21.15.92:0/1260236674' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-11T21:44:41.083 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:44:40 smithi097 ceph-mon[33317]: 6.9 scrub starts 2024-03-11T21:44:41.084 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:44:40 smithi097 ceph-mon[33317]: 6.9 scrub ok 2024-03-11T21:44:41.162 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:44:41.187 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:44:41.188 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:44:41.188 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:44:41.289 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:44:41.313 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:44:41.314 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:44:41.314 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:44:41.415 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:44:41.441 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:44:41.442 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:44:41.442 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:44:41.543 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:44:41.568 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:44:41.568 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:44:41.568 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:44:41.669 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:44:41.695 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:44:41.696 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:44:41.696 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:44:41.731 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:44:41 smithi092 ceph-mon[37563]: 6.12 scrub starts 2024-03-11T21:44:41.731 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:44:41 smithi092 ceph-mon[37563]: 6.12 scrub ok 2024-03-11T21:44:41.732 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:44:41 smithi092 ceph-mon[37563]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-03-11T21:44:41.732 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:44:41 smithi092 ceph-mon[37563]: from='client.? v1:172.21.15.92:0/585342648' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "unique_pool_3", "pool2": "unique_pool_3", "yes_i_really_really_mean_it": true}]': finished 2024-03-11T21:44:41.732 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:44:41 smithi092 ceph-mon[37563]: from='client.? v1:172.21.15.92:0/1260236674' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-03-11T21:44:41.732 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:44:41 smithi092 ceph-mon[37563]: osdmap e442: 8 total, 8 up, 6 in 2024-03-11T21:44:41.732 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:44:41 smithi092 ceph-mon[37563]: 4.4 scrub starts 2024-03-11T21:44:41.732 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:44:41 smithi092 ceph-mon[29572]: 6.12 scrub starts 2024-03-11T21:44:41.732 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:44:41 smithi092 ceph-mon[29572]: 6.12 scrub ok 2024-03-11T21:44:41.732 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:44:41 smithi092 ceph-mon[29572]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-03-11T21:44:41.732 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:44:41 smithi092 ceph-mon[29572]: from='client.? v1:172.21.15.92:0/585342648' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "unique_pool_3", "pool2": "unique_pool_3", "yes_i_really_really_mean_it": true}]': finished 2024-03-11T21:44:41.732 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:44:41 smithi092 ceph-mon[29572]: from='client.? v1:172.21.15.92:0/1260236674' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-03-11T21:44:41.732 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:44:41 smithi092 ceph-mon[29572]: osdmap e442: 8 total, 8 up, 6 in 2024-03-11T21:44:41.733 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:44:41 smithi092 ceph-mon[29572]: 4.4 scrub starts 2024-03-11T21:44:41.758 INFO:teuthology.task.full_sequential:In full_sequential, running task radosbench... 2024-03-11T21:44:41.759 INFO:tasks.radosbench:Beginning radosbench... 2024-03-11T21:44:41.759 INFO:tasks.cephadm.ceph_manager.ceph:creating pool_name unique_pool_4 2024-03-11T21:44:41.760 DEBUG:teuthology.orchestra.run.smithi092:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid 00db0016-dfed-11ee-95c8-87774f69a715 -- ceph osd pool create unique_pool_4 16 2024-03-11T21:44:41.797 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:44:41.822 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:44:41.823 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:44:41.823 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:44:41.924 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:44:41.949 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:44:41.950 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:44:41.950 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:44:41.996 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 5, 6, 7] out_osds: [3, 4] dead_osds: [] live_osds: [0, 1, 2, 4, 5, 6, 7, 3] 2024-03-11T21:44:41.996 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-03-11T21:44:41.996 INFO:tasks.thrashosds.thrasher:reset_purged_snaps_last 2024-03-11T21:44:41.996 DEBUG:teuthology.orchestra.run.smithi092:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid 00db0016-dfed-11ee-95c8-87774f69a715 -- ceph tell osd.0 reset_purged_snaps_last 2024-03-11T21:44:42.051 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:44:42.075 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:44:42.076 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:44:42.076 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:44:42.083 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:44:41 smithi097 ceph-mon[33317]: 6.12 scrub starts 2024-03-11T21:44:42.083 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:44:41 smithi097 ceph-mon[33317]: 6.12 scrub ok 2024-03-11T21:44:42.083 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:44:41 smithi097 ceph-mon[33317]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-03-11T21:44:42.083 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:44:41 smithi097 ceph-mon[33317]: from='client.? v1:172.21.15.92:0/585342648' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "unique_pool_3", "pool2": "unique_pool_3", "yes_i_really_really_mean_it": true}]': finished 2024-03-11T21:44:42.083 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:44:41 smithi097 ceph-mon[33317]: from='client.? v1:172.21.15.92:0/1260236674' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-03-11T21:44:42.084 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:44:41 smithi097 ceph-mon[33317]: osdmap e442: 8 total, 8 up, 6 in 2024-03-11T21:44:42.084 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:44:41 smithi097 ceph-mon[33317]: 4.4 scrub starts 2024-03-11T21:44:42.177 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:44:42.205 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:44:42.206 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:44:42.206 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:44:42.223 INFO:teuthology.orchestra.run.smithi092.stderr:Inferring config /var/lib/ceph/00db0016-dfed-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T21:44:42.308 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:44:42.333 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:44:42.333 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:44:42.334 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:44:42.339 INFO:teuthology.orchestra.run.smithi092.stderr:Inferring config /var/lib/ceph/00db0016-dfed-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T21:44:42.434 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:44:42.458 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:44:42.459 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:44:42.459 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:44:42.560 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:44:42.587 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:44:42.587 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:44:42.588 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:44:42.679 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:44:42 smithi092 ceph-mon[29572]: pgmap v908: 132 pgs: 132 active+clean; 582 KiB data, 4.1 GiB used, 532 GiB / 536 GiB avail 2024-03-11T21:44:42.680 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:44:42 smithi092 ceph-mon[29572]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 2 pgs peering) 2024-03-11T21:44:42.680 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:44:42 smithi092 ceph-mon[29572]: Health check cleared: PG_BACKFILL_FULL (was: Low space hindering backfill (add storage if this doesn't resolve itself): 11 pgs backfill_toofull) 2024-03-11T21:44:42.680 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:44:42 smithi092 ceph-mon[37563]: pgmap v908: 132 pgs: 132 active+clean; 582 KiB data, 4.1 GiB used, 532 GiB / 536 GiB avail 2024-03-11T21:44:42.680 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:44:42 smithi092 ceph-mon[37563]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 2 pgs peering) 2024-03-11T21:44:42.680 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:44:42 smithi092 ceph-mon[37563]: Health check cleared: PG_BACKFILL_FULL (was: Low space hindering backfill (add storage if this doesn't resolve itself): 11 pgs backfill_toofull) 2024-03-11T21:44:42.688 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:44:42.712 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:44:42.713 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:44:42.713 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:44:42.814 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:44:42.838 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:44:42.838 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:44:42.839 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:44:42.940 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:44:42.964 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:44:42.964 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:44:42.965 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:44:43.066 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:44:43.076 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:44:42 smithi097 ceph-mon[33317]: pgmap v908: 132 pgs: 132 active+clean; 582 KiB data, 4.1 GiB used, 532 GiB / 536 GiB avail 2024-03-11T21:44:43.076 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:44:42 smithi097 ceph-mon[33317]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 2 pgs peering) 2024-03-11T21:44:43.076 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:44:42 smithi097 ceph-mon[33317]: Health check cleared: PG_BACKFILL_FULL (was: Low space hindering backfill (add storage if this doesn't resolve itself): 11 pgs backfill_toofull) 2024-03-11T21:44:43.090 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:44:43.091 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:44:43.091 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:44:43.192 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:44:43.215 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:44:43.216 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:44:43.216 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:44:43.317 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:44:43.333 INFO:journalctl@ceph.iscsi.iscsi.a.smithi097.stdout:Mar 11 21:44:43 smithi097 ceph-00db0016-dfed-11ee-95c8-87774f69a715-iscsi-iscsi-a[60943]: debug there is no tcmu-runner data available 2024-03-11T21:44:43.345 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:44:43.346 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:44:43.346 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:44:43.447 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:44:43.475 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:44:43.476 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:44:43.476 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:44:43.577 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:44:43.602 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:44:43.603 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:44:43.603 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:44:43.617 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:44:43 smithi092 ceph-mon[29572]: from='client.24469 v1:172.21.15.97:0/404208310' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:44:43.617 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:44:43 smithi092 ceph-mon[29572]: osdmap e443: 8 total, 8 up, 6 in 2024-03-11T21:44:43.617 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:44:43 smithi092 ceph-mon[37563]: from='client.24469 v1:172.21.15.97:0/404208310' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:44:43.617 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:44:43 smithi092 ceph-mon[37563]: osdmap e443: 8 total, 8 up, 6 in 2024-03-11T21:44:43.704 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:44:43.728 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:44:43.729 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:44:43.729 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:44:43.830 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:44:43.858 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:44:43.859 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:44:43.859 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:44:43.960 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:44:43.988 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:44:43.989 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:44:43.989 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:44:44.083 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:44:43 smithi097 ceph-mon[33317]: from='client.24469 v1:172.21.15.97:0/404208310' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:44:44.083 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:44:43 smithi097 ceph-mon[33317]: osdmap e443: 8 total, 8 up, 6 in 2024-03-11T21:44:44.090 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:44:44.117 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:44:44.118 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:44:44.119 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:44:44.163 DEBUG:teuthology.orchestra.run.smithi092:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid 00db0016-dfed-11ee-95c8-87774f69a715 -- ceph osd set nodeep-scrub 2024-03-11T21:44:44.219 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:44:44.246 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:44:44.247 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:44:44.247 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:44:44.348 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:44:44.371 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:44:44.372 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:44:44.372 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:44:44.473 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:44:44.501 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:44:44.501 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:44:44.502 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:44:44.546 INFO:teuthology.orchestra.run.smithi092.stderr:Inferring config /var/lib/ceph/00db0016-dfed-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T21:44:44.602 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:44:44.630 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:44:44.631 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:44:44.631 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:44:44.732 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:44:44.747 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:44:44 smithi092 ceph-mon[29572]: pgmap v909: 132 pgs: 132 active+clean; 582 KiB data, 4.1 GiB used, 532 GiB / 536 GiB avail 2024-03-11T21:44:44.747 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:44:44 smithi092 ceph-mon[29572]: purged_snaps scrub starts 2024-03-11T21:44:44.747 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:44:44 smithi092 ceph-mon[29572]: purged_snaps scrub ok 2024-03-11T21:44:44.747 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:44:44 smithi092 ceph-mon[37563]: pgmap v909: 132 pgs: 132 active+clean; 582 KiB data, 4.1 GiB used, 532 GiB / 536 GiB avail 2024-03-11T21:44:44.748 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:44:44 smithi092 ceph-mon[37563]: purged_snaps scrub starts 2024-03-11T21:44:44.748 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:44:44 smithi092 ceph-mon[37563]: purged_snaps scrub ok 2024-03-11T21:44:44.759 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:44:44.760 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:44:44.760 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:44:44.861 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:44:44.890 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:44:44.890 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:44:44.891 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:44:44.991 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:44:45.019 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:44:45.020 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:44:45.020 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:44:45.083 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:44:44 smithi097 ceph-mon[33317]: pgmap v909: 132 pgs: 132 active+clean; 582 KiB data, 4.1 GiB used, 532 GiB / 536 GiB avail 2024-03-11T21:44:45.083 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:44:44 smithi097 ceph-mon[33317]: purged_snaps scrub starts 2024-03-11T21:44:45.083 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:44:44 smithi097 ceph-mon[33317]: purged_snaps scrub ok 2024-03-11T21:44:45.122 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:44:45.150 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:44:45.150 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:44:45.151 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:44:45.169 INFO:teuthology.orchestra.run.smithi092.stdout:osd.0: {} 2024-03-11T21:44:45.187 INFO:teuthology.orchestra.run.smithi092.stdout:osd.1: {} 2024-03-11T21:44:45.207 INFO:teuthology.orchestra.run.smithi092.stdout:osd.2: {} 2024-03-11T21:44:45.226 INFO:teuthology.orchestra.run.smithi092.stdout:osd.3: {} 2024-03-11T21:44:45.246 INFO:teuthology.orchestra.run.smithi092.stdout:osd.4: {} 2024-03-11T21:44:45.252 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:44:45.268 INFO:teuthology.orchestra.run.smithi092.stdout:osd.5: {} 2024-03-11T21:44:45.276 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:44:45.277 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:44:45.277 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:44:45.289 INFO:teuthology.orchestra.run.smithi092.stdout:osd.6: {} 2024-03-11T21:44:45.309 INFO:teuthology.orchestra.run.smithi092.stdout:osd.7: {} 2024-03-11T21:44:45.310 INFO:teuthology.orchestra.run.smithi092.stderr:osd.0: osd_enable_op_tracker = '' 2024-03-11T21:44:45.310 INFO:teuthology.orchestra.run.smithi092.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-11T21:44:45.310 INFO:teuthology.orchestra.run.smithi092.stderr:osd.2: osd_enable_op_tracker = '' 2024-03-11T21:44:45.310 INFO:teuthology.orchestra.run.smithi092.stderr:osd.3: osd_enable_op_tracker = '' 2024-03-11T21:44:45.311 INFO:teuthology.orchestra.run.smithi092.stderr:osd.4: osd_enable_op_tracker = '' 2024-03-11T21:44:45.311 INFO:teuthology.orchestra.run.smithi092.stderr:osd.5: osd_enable_op_tracker = '' 2024-03-11T21:44:45.311 INFO:teuthology.orchestra.run.smithi092.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-11T21:44:45.311 INFO:teuthology.orchestra.run.smithi092.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-11T21:44:45.378 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:44:45.407 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:44:45.407 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:44:45.408 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:44:45.508 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:44:45.533 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:44:45.533 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:44:45.533 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:44:45.634 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:44:45.661 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:44:45.662 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:44:45.662 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:44:45.763 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:44:45.787 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:44:45.788 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:44:45.789 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:44:45.867 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:44:45 smithi092 ceph-mon[29572]: osdmap e444: 8 total, 8 up, 6 in 2024-03-11T21:44:45.867 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:44:45 smithi092 ceph-mon[29572]: from='client.? v1:172.21.15.92:0/358721103' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-11T21:44:45.867 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:44:45 smithi092 ceph-mon[37563]: osdmap e444: 8 total, 8 up, 6 in 2024-03-11T21:44:45.867 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:44:45 smithi092 ceph-mon[37563]: from='client.? v1:172.21.15.92:0/358721103' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-11T21:44:45.890 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:44:45.917 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:44:45.918 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:44:45.918 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:44:46.019 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:44:46.044 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:44:46.044 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:44:46.044 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:44:46.083 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:44:45 smithi097 ceph-mon[33317]: osdmap e444: 8 total, 8 up, 6 in 2024-03-11T21:44:46.083 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:44:45 smithi097 ceph-mon[33317]: from='client.? v1:172.21.15.92:0/358721103' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-11T21:44:46.145 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:44:46.172 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:44:46.173 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:44:46.173 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:44:46.274 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:44:46.301 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:44:46.302 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:44:46.302 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:44:46.403 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:44:46.428 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:44:46.428 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:44:46.429 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:44:46.529 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:44:46.553 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:44:46.553 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:44:46.554 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:44:46.654 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:44:46.683 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:44:46.683 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:44:46.684 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:44:46.784 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:44:46.808 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:44:46.809 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:44:46.809 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:44:46.867 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:44:46 smithi092 ceph-mon[29572]: pgmap v912: 132 pgs: 132 active+clean; 582 KiB data, 4.1 GiB used, 532 GiB / 536 GiB avail; 682 B/s rd, 0 op/s 2024-03-11T21:44:46.867 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:44:46 smithi092 ceph-mon[37563]: pgmap v912: 132 pgs: 132 active+clean; 582 KiB data, 4.1 GiB used, 532 GiB / 536 GiB avail; 682 B/s rd, 0 op/s 2024-03-11T21:44:46.910 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:44:46.933 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:44:46.934 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:44:46.935 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:44:47.035 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:44:47.059 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:44:47.059 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:44:47.060 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:44:47.083 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:44:46 smithi097 ceph-mon[33317]: pgmap v912: 132 pgs: 132 active+clean; 582 KiB data, 4.1 GiB used, 532 GiB / 536 GiB avail; 682 B/s rd, 0 op/s 2024-03-11T21:44:47.160 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:44:47.185 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:44:47.185 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:44:47.186 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:44:47.287 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:44:47.311 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:44:47.312 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:44:47.312 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:44:47.413 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:44:47.442 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:44:47.442 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:44:47.443 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:44:47.544 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:44:47.568 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:44:47.568 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:44:47.569 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:44:47.670 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:44:47.694 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:44:47.695 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:44:47.695 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:44:47.796 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:44:47.819 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:44:47.820 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:44:47.820 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:44:47.922 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:44:47.945 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:44:47.946 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:44:47.946 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:44:48.047 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:44:48.075 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:44:48.076 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:44:48.076 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:44:48.177 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:44:48.192 INFO:journalctl@ceph.mgr.y.smithi092.stdout:Mar 11 21:44:47 smithi092 ceph-00db0016-dfed-11ee-95c8-87774f69a715-mgr-y[29801]: ::ffff:172.21.15.97 - - [11/Mar/2024:21:44:47] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-11T21:44:48.201 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:44:48.202 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:44:48.203 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:44:48.304 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:44:48.330 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:44:48.330 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:44:48.330 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:44:48.431 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:44:48.455 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:44:48.456 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:44:48.456 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:44:48.557 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:44:48.581 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:44:48.581 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:44:48.582 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:44:48.683 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:44:48.707 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:44:48.708 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:44:48.708 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:44:48.809 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:44:48.815 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:44:48 smithi092 ceph-mon[29572]: pgmap v913: 132 pgs: 132 active+clean; 582 KiB data, 4.1 GiB used, 532 GiB / 536 GiB avail; 751 B/s rd, 0 op/s 2024-03-11T21:44:48.816 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:44:48 smithi092 ceph-mon[29572]: purged_snaps scrub starts 2024-03-11T21:44:48.816 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:44:48 smithi092 ceph-mon[29572]: purged_snaps scrub ok 2024-03-11T21:44:48.816 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:44:48 smithi092 ceph-mon[37563]: pgmap v913: 132 pgs: 132 active+clean; 582 KiB data, 4.1 GiB used, 532 GiB / 536 GiB avail; 751 B/s rd, 0 op/s 2024-03-11T21:44:48.816 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:44:48 smithi092 ceph-mon[37563]: purged_snaps scrub starts 2024-03-11T21:44:48.816 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:44:48 smithi092 ceph-mon[37563]: purged_snaps scrub ok 2024-03-11T21:44:48.833 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:44:48.834 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:44:48.834 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:44:48.935 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:44:48.960 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:44:48.960 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:44:48.961 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:44:49.053 INFO:teuthology.orchestra.run.smithi092.stderr:Inferring config /var/lib/ceph/00db0016-dfed-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T21:44:49.062 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:44:49.065 INFO:teuthology.orchestra.run.smithi092.stdout:{ 2024-03-11T21:44:49.065 INFO:teuthology.orchestra.run.smithi092.stdout: "ops": [], 2024-03-11T21:44:49.065 INFO:teuthology.orchestra.run.smithi092.stdout: "complaint_time": 30, 2024-03-11T21:44:49.065 INFO:teuthology.orchestra.run.smithi092.stdout: "num_blocked_ops": 0 2024-03-11T21:44:49.065 INFO:teuthology.orchestra.run.smithi092.stdout:} 2024-03-11T21:44:49.083 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:44:48 smithi097 ceph-mon[33317]: pgmap v913: 132 pgs: 132 active+clean; 582 KiB data, 4.1 GiB used, 532 GiB / 536 GiB avail; 751 B/s rd, 0 op/s 2024-03-11T21:44:49.083 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:44:48 smithi097 ceph-mon[33317]: purged_snaps scrub starts 2024-03-11T21:44:49.083 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:44:48 smithi097 ceph-mon[33317]: purged_snaps scrub ok 2024-03-11T21:44:49.086 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:44:49.086 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:44:49.087 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:44:49.188 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:44:49.212 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:44:49.212 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:44:49.212 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:44:49.313 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:44:49.336 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:44:49.337 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:44:49.337 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:44:49.438 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:44:49.462 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:44:49.462 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:44:49.463 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:44:49.564 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:44:49.587 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:44:49.588 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:44:49.588 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:44:49.605 DEBUG:teuthology.orchestra.run.smithi092:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid 00db0016-dfed-11ee-95c8-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-03-11T21:44:49.689 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:44:49.713 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:44:49.714 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:44:49.715 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:44:49.815 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:44:49.839 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:44:49.840 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:44:49.840 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:44:49.867 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:44:49 smithi092 ceph-mon[29572]: osdmap e445: 8 total, 8 up, 6 in 2024-03-11T21:44:49.867 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:44:49 smithi092 ceph-mon[37563]: osdmap e445: 8 total, 8 up, 6 in 2024-03-11T21:44:49.941 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:44:49.965 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:44:49.965 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:44:49.966 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:44:50.066 DEBUG:teuthology.orchestra.run.smithi092:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid 00db0016-dfed-11ee-95c8-87774f69a715 -- ceph daemon osd.1 dump_historic_ops 2024-03-11T21:44:50.066 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:44:50.083 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:44:49 smithi097 ceph-mon[33317]: osdmap e445: 8 total, 8 up, 6 in 2024-03-11T21:44:50.091 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:44:50.092 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:44:50.092 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:44:50.193 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:44:50.219 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:44:50.219 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:44:50.220 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:44:50.320 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:44:50.344 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:44:50.345 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:44:50.346 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:44:50.446 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:44:50.470 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:44:50.471 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:44:50.471 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:44:50.572 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:44:50.594 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:44:50.595 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:44:50.595 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:44:50.696 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:44:50.724 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:44:50.725 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:44:50.725 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:44:50.826 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:44:50.854 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:44:50.854 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:44:50.855 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:44:50.956 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:44:50.984 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:44:50.984 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:44:50.985 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:44:51.053 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:44:50 smithi092 ceph-mon[29572]: pgmap v915: 132 pgs: 132 active+clean; 582 KiB data, 4.1 GiB used, 532 GiB / 536 GiB avail; 1.7 KiB/s rd, 1 op/s 2024-03-11T21:44:51.054 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:44:50 smithi092 ceph-mon[37563]: pgmap v915: 132 pgs: 132 active+clean; 582 KiB data, 4.1 GiB used, 532 GiB / 536 GiB avail; 1.7 KiB/s rd, 1 op/s 2024-03-11T21:44:51.083 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:44:50 smithi097 ceph-mon[33317]: pgmap v915: 132 pgs: 132 active+clean; 582 KiB data, 4.1 GiB used, 532 GiB / 536 GiB avail; 1.7 KiB/s rd, 1 op/s 2024-03-11T21:44:51.086 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:44:51.111 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:44:51.111 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:44:51.112 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:44:51.212 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:44:51.241 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:44:51.241 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:44:51.242 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:44:51.343 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:44:51.367 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:44:51.368 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:44:51.368 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:44:51.469 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:44:51.493 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:44:51.493 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:44:51.494 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:44:51.594 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:44:51.618 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:44:51.619 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:44:51.619 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:44:51.720 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:44:51.744 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:44:51.745 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:44:51.745 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:44:51.846 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:44:51.870 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:44:51.871 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:44:51.871 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:44:51.972 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:44:52.001 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:44:52.002 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:44:52.002 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:44:52.103 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:44:52.131 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:44:52.131 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:44:52.132 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:44:52.232 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:44:52.260 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:44:52.260 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:44:52.261 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:44:52.361 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:44:52.388 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:44:52.389 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:44:52.389 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:44:52.490 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:44:52.514 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:44:52.515 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:44:52.515 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:44:52.616 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:44:52.641 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:44:52.642 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:44:52.642 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:44:52.647 INFO:teuthology.orchestra.run.smithi092.stderr:pool 'unique_pool_4' created 2024-03-11T21:44:52.710 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:44:52 smithi092 ceph-mon[29572]: pgmap v916: 132 pgs: 132 active+clean; 582 KiB data, 4.1 GiB used, 532 GiB / 536 GiB avail; 1.3 KiB/s rd, 1 op/s 2024-03-11T21:44:52.710 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:44:52 smithi092 ceph-mon[29572]: purged_snaps scrub starts 2024-03-11T21:44:52.710 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:44:52 smithi092 ceph-mon[29572]: purged_snaps scrub ok 2024-03-11T21:44:52.710 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:44:52 smithi092 ceph-mon[29572]: from='client.? v1:172.21.15.92:0/4255345830' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "unique_pool_4", "pg_num": 16}]: dispatch 2024-03-11T21:44:52.711 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:44:52 smithi092 ceph-mon[29572]: from='client.35780 ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "unique_pool_4", "pg_num": 16}]: dispatch 2024-03-11T21:44:52.711 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:44:52 smithi092 ceph-mon[37563]: pgmap v916: 132 pgs: 132 active+clean; 582 KiB data, 4.1 GiB used, 532 GiB / 536 GiB avail; 1.3 KiB/s rd, 1 op/s 2024-03-11T21:44:52.711 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:44:52 smithi092 ceph-mon[37563]: purged_snaps scrub starts 2024-03-11T21:44:52.711 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:44:52 smithi092 ceph-mon[37563]: purged_snaps scrub ok 2024-03-11T21:44:52.711 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:44:52 smithi092 ceph-mon[37563]: from='client.? v1:172.21.15.92:0/4255345830' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "unique_pool_4", "pg_num": 16}]: dispatch 2024-03-11T21:44:52.711 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:44:52 smithi092 ceph-mon[37563]: from='client.35780 ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "unique_pool_4", "pg_num": 16}]: dispatch 2024-03-11T21:44:52.744 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:44:52.767 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:44:52.768 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:44:52.768 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:44:52.869 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:44:52.897 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:44:52.898 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:44:52.898 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:44:52.924 INFO:teuthology.orchestra.run.smithi092.stderr:Inferring config /var/lib/ceph/00db0016-dfed-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T21:44:52.999 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:44:53.022 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:44:53.023 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:44:53.023 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:44:53.077 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:44:52 smithi097 ceph-mon[33317]: pgmap v916: 132 pgs: 132 active+clean; 582 KiB data, 4.1 GiB used, 532 GiB / 536 GiB avail; 1.3 KiB/s rd, 1 op/s 2024-03-11T21:44:53.078 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:44:52 smithi097 ceph-mon[33317]: purged_snaps scrub starts 2024-03-11T21:44:53.078 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:44:52 smithi097 ceph-mon[33317]: purged_snaps scrub ok 2024-03-11T21:44:53.078 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:44:52 smithi097 ceph-mon[33317]: from='client.? v1:172.21.15.92:0/4255345830' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "unique_pool_4", "pg_num": 16}]: dispatch 2024-03-11T21:44:53.078 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:44:52 smithi097 ceph-mon[33317]: from='client.35780 ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "unique_pool_4", "pg_num": 16}]: dispatch 2024-03-11T21:44:53.124 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:44:53.147 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:44:53.148 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:44:53.148 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:44:53.249 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:44:53.277 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:44:53.278 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:44:53.278 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:44:53.333 INFO:journalctl@ceph.iscsi.iscsi.a.smithi097.stdout:Mar 11 21:44:53 smithi097 ceph-00db0016-dfed-11ee-95c8-87774f69a715-iscsi-iscsi-a[60943]: debug there is no tcmu-runner data available 2024-03-11T21:44:53.380 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:44:53.404 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:44:53.405 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:44:53.405 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:44:53.506 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:44:53.533 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:44:53.533 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:44:53.534 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:44:53.556 DEBUG:teuthology.orchestra.run.smithi092:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid 00db0016-dfed-11ee-95c8-87774f69a715 -- ceph osd pool application enable unique_pool_4 rados --yes-i-really-mean-it || true 2024-03-11T21:44:53.634 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:44:53.652 INFO:teuthology.orchestra.run.smithi092.stderr:nodeep-scrub is set 2024-03-11T21:44:53.667 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:44:53.667 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:44:53.668 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:44:53.768 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:44:53.794 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:44:53 smithi092 ceph-mon[29572]: from='client.35780 ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "unique_pool_4", "pg_num": 16}]': finished 2024-03-11T21:44:53.794 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:44:53 smithi092 ceph-mon[29572]: osdmap e446: 8 total, 8 up, 6 in 2024-03-11T21:44:53.795 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:44:53 smithi092 ceph-mon[29572]: from='client.? v1:172.21.15.92:0/3589948859' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-11T21:44:53.795 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:44:53 smithi092 ceph-mon[29572]: from='client.35786 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-11T21:44:53.795 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:44:53 smithi092 ceph-mon[29572]: from='mgr.14607 ' entity='mgr.y' 2024-03-11T21:44:53.795 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:44:53 smithi092 ceph-mon[29572]: from='mgr.14607 v1:172.21.15.92:0/1702717597' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T21:44:53.795 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:44:53 smithi092 ceph-mon[29572]: from='client.24469 v1:172.21.15.97:0/404208310' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:44:53.795 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:44:53 smithi092 ceph-mon[29572]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-11T21:44:53.795 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:44:53 smithi092 ceph-mon[29572]: from='client.35786 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-03-11T21:44:53.795 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:44:53 smithi092 ceph-mon[29572]: osdmap e447: 8 total, 8 up, 6 in 2024-03-11T21:44:53.795 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:44:53 smithi092 ceph-mon[37563]: from='client.35780 ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "unique_pool_4", "pg_num": 16}]': finished 2024-03-11T21:44:53.795 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:44:53 smithi092 ceph-mon[37563]: osdmap e446: 8 total, 8 up, 6 in 2024-03-11T21:44:53.795 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:44:53 smithi092 ceph-mon[37563]: from='client.? v1:172.21.15.92:0/3589948859' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-11T21:44:53.796 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:44:53 smithi092 ceph-mon[37563]: from='client.35786 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-11T21:44:53.796 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:44:53 smithi092 ceph-mon[37563]: from='mgr.14607 ' entity='mgr.y' 2024-03-11T21:44:53.796 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:44:53 smithi092 ceph-mon[37563]: from='mgr.14607 v1:172.21.15.92:0/1702717597' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T21:44:53.796 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:44:53 smithi092 ceph-mon[37563]: from='client.24469 v1:172.21.15.97:0/404208310' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:44:53.796 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:44:53 smithi092 ceph-mon[37563]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-11T21:44:53.796 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:44:53 smithi092 ceph-mon[37563]: from='client.35786 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-03-11T21:44:53.796 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:44:53 smithi092 ceph-mon[37563]: osdmap e447: 8 total, 8 up, 6 in 2024-03-11T21:44:53.797 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:44:53.797 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:44:53.798 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:44:53.898 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:44:53.922 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:44:53.923 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:44:53.923 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:44:54.024 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:44:54.052 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:44:54.052 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:44:54.053 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:44:54.083 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:44:53 smithi097 ceph-mon[33317]: from='client.35780 ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "unique_pool_4", "pg_num": 16}]': finished 2024-03-11T21:44:54.083 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:44:53 smithi097 ceph-mon[33317]: osdmap e446: 8 total, 8 up, 6 in 2024-03-11T21:44:54.083 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:44:53 smithi097 ceph-mon[33317]: from='client.? v1:172.21.15.92:0/3589948859' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-11T21:44:54.083 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:44:53 smithi097 ceph-mon[33317]: from='client.35786 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-11T21:44:54.083 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:44:53 smithi097 ceph-mon[33317]: from='mgr.14607 ' entity='mgr.y' 2024-03-11T21:44:54.083 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:44:53 smithi097 ceph-mon[33317]: from='mgr.14607 v1:172.21.15.92:0/1702717597' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T21:44:54.083 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:44:53 smithi097 ceph-mon[33317]: from='client.24469 v1:172.21.15.97:0/404208310' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:44:54.083 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:44:53 smithi097 ceph-mon[33317]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-11T21:44:54.084 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:44:53 smithi097 ceph-mon[33317]: from='client.35786 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-03-11T21:44:54.084 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:44:53 smithi097 ceph-mon[33317]: osdmap e447: 8 total, 8 up, 6 in 2024-03-11T21:44:54.153 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:44:54.181 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:44:54.182 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:44:54.182 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:44:54.283 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:44:54.309 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:44:54.310 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:44:54.310 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:44:54.411 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:44:54.435 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:44:54.436 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:44:54.436 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:44:54.537 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:44:54.560 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:44:54.561 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:44:54.562 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:44:54.663 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:44:54.687 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:44:54.688 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:44:54.688 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:44:54.789 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:44:54.815 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:44:54 smithi092 ceph-mon[37563]: pgmap v918: 148 pgs: 2 creating+peering, 14 unknown, 132 active+clean; 582 KiB data, 4.1 GiB used, 532 GiB / 536 GiB avail; 639 B/s rd, 0 op/s 2024-03-11T21:44:54.815 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:44:54 smithi092 ceph-mon[37563]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-11T21:44:54.815 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:44:54 smithi092 ceph-mon[29572]: pgmap v918: 148 pgs: 2 creating+peering, 14 unknown, 132 active+clean; 582 KiB data, 4.1 GiB used, 532 GiB / 536 GiB avail; 639 B/s rd, 0 op/s 2024-03-11T21:44:54.815 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:44:54 smithi092 ceph-mon[29572]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-11T21:44:54.817 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:44:54.818 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:44:54.818 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:44:54.844 DEBUG:teuthology.orchestra.run.smithi092:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid 00db0016-dfed-11ee-95c8-87774f69a715 -- ceph tell osd.1 reset_purged_snaps_last 2024-03-11T21:44:54.919 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:44:54.946 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:44:54.947 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:44:54.947 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:44:55.048 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:44:55.076 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:44:55.076 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:44:55.076 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:44:55.083 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:44:54 smithi097 ceph-mon[33317]: pgmap v918: 148 pgs: 2 creating+peering, 14 unknown, 132 active+clean; 582 KiB data, 4.1 GiB used, 532 GiB / 536 GiB avail; 639 B/s rd, 0 op/s 2024-03-11T21:44:55.083 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:44:54 smithi097 ceph-mon[33317]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-11T21:44:55.178 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:44:55.205 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:44:55.206 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:44:55.207 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:44:55.307 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:44:55.331 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:44:55.332 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:44:55.332 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:44:55.361 INFO:teuthology.orchestra.run.smithi092.stderr:Inferring config /var/lib/ceph/00db0016-dfed-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T21:44:55.433 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:44:55.462 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:44:55.463 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:44:55.463 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:44:55.564 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:44:55.591 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:44:55.592 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:44:55.592 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:44:55.693 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:44:55.720 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:44:55.721 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:44:55.721 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:44:55.822 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:44:55.849 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:44:55.850 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:44:55.850 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:44:55.906 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:44:55 smithi092 ceph-mon[29572]: osdmap e448: 8 total, 8 up, 6 in 2024-03-11T21:44:55.906 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:44:55 smithi092 ceph-mon[37563]: osdmap e448: 8 total, 8 up, 6 in 2024-03-11T21:44:55.951 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:44:55.975 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:44:55.975 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:44:55.976 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:44:56.077 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:44:56.083 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:44:55 smithi097 ceph-mon[33317]: osdmap e448: 8 total, 8 up, 6 in 2024-03-11T21:44:56.101 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:44:56.101 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:44:56.102 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:44:56.203 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:44:56.230 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:44:56.231 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:44:56.231 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:44:56.332 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:44:56.360 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:44:56.360 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:44:56.361 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:44:56.462 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:44:56.489 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:44:56.490 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:44:56.490 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:44:56.591 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:44:56.616 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:44:56.616 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:44:56.616 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:44:56.625 INFO:teuthology.orchestra.run.smithi092.stdout:osd.0: {} 2024-03-11T21:44:56.625 INFO:teuthology.orchestra.run.smithi092.stdout:osd.1: {} 2024-03-11T21:44:56.626 INFO:teuthology.orchestra.run.smithi092.stdout:osd.2: {} 2024-03-11T21:44:56.626 INFO:teuthology.orchestra.run.smithi092.stdout:osd.3: {} 2024-03-11T21:44:56.626 INFO:teuthology.orchestra.run.smithi092.stdout:osd.4: {} 2024-03-11T21:44:56.626 INFO:teuthology.orchestra.run.smithi092.stdout:osd.5: {} 2024-03-11T21:44:56.626 INFO:teuthology.orchestra.run.smithi092.stdout:osd.6: {} 2024-03-11T21:44:56.631 INFO:teuthology.orchestra.run.smithi092.stderr:Inferring config /var/lib/ceph/00db0016-dfed-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T21:44:56.650 INFO:teuthology.orchestra.run.smithi092.stdout:osd.7: {} 2024-03-11T21:44:56.652 INFO:teuthology.orchestra.run.smithi092.stderr:osd.0: osd_enable_op_tracker = '' 2024-03-11T21:44:56.652 INFO:teuthology.orchestra.run.smithi092.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-11T21:44:56.652 INFO:teuthology.orchestra.run.smithi092.stderr:osd.2: osd_enable_op_tracker = '' 2024-03-11T21:44:56.652 INFO:teuthology.orchestra.run.smithi092.stderr:osd.3: osd_enable_op_tracker = '' 2024-03-11T21:44:56.652 INFO:teuthology.orchestra.run.smithi092.stderr:osd.4: osd_enable_op_tracker = '' 2024-03-11T21:44:56.652 INFO:teuthology.orchestra.run.smithi092.stderr:osd.5: osd_enable_op_tracker = '' 2024-03-11T21:44:56.652 INFO:teuthology.orchestra.run.smithi092.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-11T21:44:56.652 INFO:teuthology.orchestra.run.smithi092.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-11T21:44:56.718 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:44:56.742 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:44:56.743 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:44:56.743 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:44:56.764 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:44:56 smithi092 ceph-mon[29572]: pgmap v921: 148 pgs: 6 creating+peering, 9 unknown, 133 active+clean; 582 KiB data, 4.0 GiB used, 532 GiB / 536 GiB avail; 853 B/s rd, 0 op/s 2024-03-11T21:44:56.764 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:44:56 smithi092 ceph-mon[29572]: from='client.? v1:172.21.15.92:0/3361749045' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-11T21:44:56.764 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:44:56 smithi092 ceph-mon[37563]: pgmap v921: 148 pgs: 6 creating+peering, 9 unknown, 133 active+clean; 582 KiB data, 4.0 GiB used, 532 GiB / 536 GiB avail; 853 B/s rd, 0 op/s 2024-03-11T21:44:56.764 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:44:56 smithi092 ceph-mon[37563]: from='client.? v1:172.21.15.92:0/3361749045' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-11T21:44:56.844 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:44:56.872 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:44:56.873 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:44:56.873 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:44:56.974 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:44:57.002 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:44:57.003 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:44:57.003 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:44:57.083 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:44:56 smithi097 ceph-mon[33317]: pgmap v921: 148 pgs: 6 creating+peering, 9 unknown, 133 active+clean; 582 KiB data, 4.0 GiB used, 532 GiB / 536 GiB avail; 853 B/s rd, 0 op/s 2024-03-11T21:44:57.083 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:44:56 smithi097 ceph-mon[33317]: from='client.? v1:172.21.15.92:0/3361749045' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-11T21:44:57.101 INFO:teuthology.orchestra.run.smithi092.stderr:Inferring config /var/lib/ceph/00db0016-dfed-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T21:44:57.104 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:44:57.132 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:44:57.133 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:44:57.133 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:44:57.234 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:44:57.257 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:44:57.258 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:44:57.258 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:44:57.359 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:44:57.383 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:44:57.383 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:44:57.384 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:44:57.484 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:44:57.512 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:44:57.513 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:44:57.513 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:44:57.614 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:44:57.639 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:44:57.640 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:44:57.640 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:44:57.741 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:44:57.746 DEBUG:teuthology.orchestra.run.smithi092:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid 00db0016-dfed-11ee-95c8-87774f69a715 -- ceph osd unset noscrub 2024-03-11T21:44:57.769 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:44:57.770 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:44:57.771 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:44:57.872 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:44:57.897 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:44:57.898 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:44:57.898 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:44:57.999 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:44:58.024 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:44:58.024 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:44:58.025 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:44:58.125 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:44:58.154 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:44:58.155 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:44:58.155 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:44:58.256 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:44:58.281 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:44:58.281 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:44:58.282 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:44:58.286 INFO:journalctl@ceph.mgr.y.smithi092.stdout:Mar 11 21:44:57 smithi092 ceph-00db0016-dfed-11ee-95c8-87774f69a715-mgr-y[29801]: ::ffff:172.21.15.97 - - [11/Mar/2024:21:44:57] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-11T21:44:58.383 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:44:58.411 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:44:58.412 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:44:58.412 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:44:58.513 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:44:58.541 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:44:58.542 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:44:58.542 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:44:58.643 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:44:58.670 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:44:58.670 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:44:58.671 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:44:58.771 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:44:58.796 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:44:58.797 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:44:58.797 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:44:58.898 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:44:58.923 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:44:58.923 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:44:58.923 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:44:59.024 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:44:59.053 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:44:59.053 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:44:59.054 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:44:59.083 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:44:58 smithi097 ceph-mon[33317]: pgmap v922: 148 pgs: 6 creating+peering, 8 unknown, 134 active+clean; 582 KiB data, 4.0 GiB used, 532 GiB / 536 GiB avail; 853 B/s rd, 0 op/s 2024-03-11T21:44:59.091 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:44:58 smithi092 ceph-mon[29572]: pgmap v922: 148 pgs: 6 creating+peering, 8 unknown, 134 active+clean; 582 KiB data, 4.0 GiB used, 532 GiB / 536 GiB avail; 853 B/s rd, 0 op/s 2024-03-11T21:44:59.091 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:44:58 smithi092 ceph-mon[37563]: pgmap v922: 148 pgs: 6 creating+peering, 8 unknown, 134 active+clean; 582 KiB data, 4.0 GiB used, 532 GiB / 536 GiB avail; 853 B/s rd, 0 op/s 2024-03-11T21:44:59.154 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:44:59.178 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:44:59.179 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:44:59.179 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:44:59.280 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:44:59.304 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:44:59.304 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:44:59.305 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:44:59.405 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:44:59.429 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:44:59.429 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:44:59.430 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:44:59.531 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:44:59.559 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:44:59.559 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:44:59.560 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:44:59.661 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:44:59.684 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:44:59.685 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:44:59.685 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:44:59.786 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:44:59.814 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:44:59.815 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:44:59.815 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:44:59.916 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:44:59.944 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:44:59.945 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:44:59.945 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:45:00.046 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:45:00.071 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:45:00.072 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:45:00.072 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:45:00.173 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:45:00.196 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:45:00.197 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:45:00.197 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:45:00.298 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:45:00.326 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:45:00.327 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:45:00.327 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:45:00.428 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:45:00.451 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:45:00.452 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:45:00.452 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:45:00.553 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:45:00.577 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:45:00.577 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:45:00.578 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:45:00.679 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:45:00.702 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:45:00.703 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:45:00.703 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:45:00.804 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:45:00.828 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:45:00.828 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:45:00.829 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:45:00.912 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:45:00 smithi092 ceph-mon[29572]: pgmap v923: 148 pgs: 148 active+clean; 582 KiB data, 4.0 GiB used, 532 GiB / 536 GiB avail; 1.5 KiB/s rd, 1 op/s 2024-03-11T21:45:00.913 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:45:00 smithi092 ceph-mon[29572]: purged_snaps scrub starts 2024-03-11T21:45:00.913 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:45:00 smithi092 ceph-mon[29572]: purged_snaps scrub ok 2024-03-11T21:45:00.913 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:45:00 smithi092 ceph-mon[37563]: pgmap v923: 148 pgs: 148 active+clean; 582 KiB data, 4.0 GiB used, 532 GiB / 536 GiB avail; 1.5 KiB/s rd, 1 op/s 2024-03-11T21:45:00.913 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:45:00 smithi092 ceph-mon[37563]: purged_snaps scrub starts 2024-03-11T21:45:00.913 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:45:00 smithi092 ceph-mon[37563]: purged_snaps scrub ok 2024-03-11T21:45:00.930 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:45:00.953 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:45:00.954 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:45:00.954 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:45:00.967 INFO:teuthology.orchestra.run.smithi092.stdout:ERROR: (22) Invalid argument 2024-03-11T21:45:00.967 INFO:teuthology.orchestra.run.smithi092.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-11T21:45:01.055 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:45:01.080 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:45:01.081 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:45:01.081 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:45:01.083 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:45:00 smithi097 ceph-mon[33317]: pgmap v923: 148 pgs: 148 active+clean; 582 KiB data, 4.0 GiB used, 532 GiB / 536 GiB avail; 1.5 KiB/s rd, 1 op/s 2024-03-11T21:45:01.083 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:45:00 smithi097 ceph-mon[33317]: purged_snaps scrub starts 2024-03-11T21:45:01.083 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:45:00 smithi097 ceph-mon[33317]: purged_snaps scrub ok 2024-03-11T21:45:01.182 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:45:01.210 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:45:01.211 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:45:01.211 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:45:01.312 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:45:01.339 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:45:01.340 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:45:01.340 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:45:01.441 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:45:01.466 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:45:01.467 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:45:01.467 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:45:01.555 DEBUG:teuthology.orchestra.run.smithi092:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid 00db0016-dfed-11ee-95c8-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-03-11T21:45:01.568 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:45:01.598 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:45:01.599 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:45:01.599 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:45:01.700 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:45:01.727 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:45:01.727 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:45:01.728 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:45:01.828 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:45:01.855 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:45:01.855 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:45:01.856 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:45:01.867 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:45:01 smithi092 ceph-mon[29572]: osdmap e449: 8 total, 8 up, 6 in 2024-03-11T21:45:01.867 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:45:01 smithi092 ceph-mon[37563]: osdmap e449: 8 total, 8 up, 6 in 2024-03-11T21:45:01.957 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:45:01.984 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:45:01.984 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:45:01.985 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:45:02.083 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:45:01 smithi097 ceph-mon[33317]: osdmap e449: 8 total, 8 up, 6 in 2024-03-11T21:45:02.085 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:45:02.109 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:45:02.110 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:45:02.110 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:45:02.211 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:45:02.239 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:45:02.240 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:45:02.240 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:45:02.341 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:45:02.369 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:45:02.370 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:45:02.370 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:45:02.471 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:45:02.497 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:45:02.497 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:45:02.498 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:45:02.598 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:45:02.612 DEBUG:teuthology.orchestra.run.smithi092:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid 00db0016-dfed-11ee-95c8-87774f69a715 -- ceph daemon osd.2 dump_ops_in_flight 2024-03-11T21:45:02.622 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:45:02.622 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:45:02.623 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:45:02.724 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:45:02.747 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:45:02.748 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:45:02.749 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:45:02.850 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:45:02.874 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:45:02.875 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:45:02.875 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:45:02.975 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:45:02.999 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:45:03.000 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:45:03.000 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:45:03.001 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:45:02 smithi092 ceph-mon[29572]: pgmap v925: 148 pgs: 148 active+clean; 582 KiB data, 4.0 GiB used, 532 GiB / 536 GiB avail; 658 B/s rd, 0 op/s 2024-03-11T21:45:03.001 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:45:02 smithi092 ceph-mon[29572]: from='mgr.14607 v1:172.21.15.92:0/1702717597' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T21:45:03.001 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:45:02 smithi092 ceph-mon[37563]: pgmap v925: 148 pgs: 148 active+clean; 582 KiB data, 4.0 GiB used, 532 GiB / 536 GiB avail; 658 B/s rd, 0 op/s 2024-03-11T21:45:03.001 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:45:02 smithi092 ceph-mon[37563]: from='mgr.14607 v1:172.21.15.92:0/1702717597' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T21:45:03.083 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:45:02 smithi097 ceph-mon[33317]: pgmap v925: 148 pgs: 148 active+clean; 582 KiB data, 4.0 GiB used, 532 GiB / 536 GiB avail; 658 B/s rd, 0 op/s 2024-03-11T21:45:03.083 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:45:02 smithi097 ceph-mon[33317]: from='mgr.14607 v1:172.21.15.92:0/1702717597' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T21:45:03.101 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:45:03.128 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:45:03.129 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:45:03.129 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:45:03.230 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:45:03.255 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:45:03.255 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:45:03.256 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:45:03.356 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:45:03.380 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:45:03.381 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:45:03.381 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:45:03.482 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:45:03.507 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:45:03.507 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:45:03.507 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:45:03.551 INFO:teuthology.orchestra.run.smithi092.stderr:Inferring config /var/lib/ceph/00db0016-dfed-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T21:45:03.583 INFO:journalctl@ceph.iscsi.iscsi.a.smithi097.stdout:Mar 11 21:45:03 smithi097 ceph-00db0016-dfed-11ee-95c8-87774f69a715-iscsi-iscsi-a[60943]: debug there is no tcmu-runner data available 2024-03-11T21:45:03.608 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:45:03.632 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:45:03.633 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:45:03.633 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:45:03.686 INFO:teuthology.orchestra.run.smithi092.stderr:enabled application 'rados' on pool 'unique_pool_4' 2024-03-11T21:45:03.734 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:45:03.759 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:45:03.760 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:45:03.760 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:45:03.861 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:45:03.887 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:45:03.887 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:45:03.888 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:45:03.988 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:45:04.012 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:45:04.013 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:45:04.013 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:45:04.083 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:45:03 smithi097 ceph-mon[33317]: from='client.? v1:172.21.15.92:0/377583514' entity='client.admin' cmd=[{"prefix": "osd pool application enable", "pool": "unique_pool_4", "app": "rados", "yes_i_really_mean_it": true}]: dispatch 2024-03-11T21:45:04.083 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:45:03 smithi097 ceph-mon[33317]: from='client.24469 v1:172.21.15.97:0/404208310' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:45:04.084 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:45:03 smithi097 ceph-mon[33317]: from='mgr.14607 v1:172.21.15.92:0/1702717597' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T21:45:04.084 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:45:03 smithi097 ceph-mon[33317]: from='mgr.14607 v1:172.21.15.92:0/1702717597' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T21:45:04.084 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:45:03 smithi097 ceph-mon[33317]: from='mgr.14607 ' entity='mgr.y' 2024-03-11T21:45:04.114 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:45:04.118 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:45:03 smithi092 ceph-mon[37563]: from='client.? v1:172.21.15.92:0/377583514' entity='client.admin' cmd=[{"prefix": "osd pool application enable", "pool": "unique_pool_4", "app": "rados", "yes_i_really_mean_it": true}]: dispatch 2024-03-11T21:45:04.118 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:45:03 smithi092 ceph-mon[37563]: from='client.24469 v1:172.21.15.97:0/404208310' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:45:04.118 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:45:03 smithi092 ceph-mon[37563]: from='mgr.14607 v1:172.21.15.92:0/1702717597' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T21:45:04.118 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:45:03 smithi092 ceph-mon[37563]: from='mgr.14607 v1:172.21.15.92:0/1702717597' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T21:45:04.118 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:45:03 smithi092 ceph-mon[37563]: from='mgr.14607 ' entity='mgr.y' 2024-03-11T21:45:04.118 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:45:03 smithi092 ceph-mon[29572]: from='client.? v1:172.21.15.92:0/377583514' entity='client.admin' cmd=[{"prefix": "osd pool application enable", "pool": "unique_pool_4", "app": "rados", "yes_i_really_mean_it": true}]: dispatch 2024-03-11T21:45:04.119 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:45:03 smithi092 ceph-mon[29572]: from='client.24469 v1:172.21.15.97:0/404208310' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:45:04.119 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:45:03 smithi092 ceph-mon[29572]: from='mgr.14607 v1:172.21.15.92:0/1702717597' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T21:45:04.119 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:45:03 smithi092 ceph-mon[29572]: from='mgr.14607 v1:172.21.15.92:0/1702717597' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T21:45:04.119 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:45:03 smithi092 ceph-mon[29572]: from='mgr.14607 ' entity='mgr.y' 2024-03-11T21:45:04.137 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:45:04.138 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:45:04.138 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:45:04.239 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:45:04.262 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:45:04.263 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:45:04.263 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:45:04.364 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:45:04.392 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:45:04.393 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:45:04.393 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:45:04.494 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:45:04.522 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:45:04.522 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:45:04.523 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:45:04.608 INFO:teuthology.orchestra.run.smithi092.stderr:Inferring config /var/lib/ceph/00db0016-dfed-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T21:45:04.624 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:45:04.651 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:45:04.652 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:45:04.652 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:45:04.753 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:45:04.755 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:45:04 smithi092 ceph-mon[29572]: pgmap v926: 148 pgs: 148 active+clean; 582 KiB data, 4.0 GiB used, 532 GiB / 536 GiB avail; 584 B/s rd, 0 op/s 2024-03-11T21:45:04.755 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:45:04 smithi092 ceph-mon[29572]: from='client.? v1:172.21.15.92:0/377583514' entity='client.admin' cmd='[{"prefix": "osd pool application enable", "pool": "unique_pool_4", "app": "rados", "yes_i_really_mean_it": true}]': finished 2024-03-11T21:45:04.755 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:45:04 smithi092 ceph-mon[29572]: osdmap e450: 8 total, 8 up, 6 in 2024-03-11T21:45:04.755 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:45:04 smithi092 ceph-mon[37563]: pgmap v926: 148 pgs: 148 active+clean; 582 KiB data, 4.0 GiB used, 532 GiB / 536 GiB avail; 584 B/s rd, 0 op/s 2024-03-11T21:45:04.755 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:45:04 smithi092 ceph-mon[37563]: from='client.? v1:172.21.15.92:0/377583514' entity='client.admin' cmd='[{"prefix": "osd pool application enable", "pool": "unique_pool_4", "app": "rados", "yes_i_really_mean_it": true}]': finished 2024-03-11T21:45:04.755 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:45:04 smithi092 ceph-mon[37563]: osdmap e450: 8 total, 8 up, 6 in 2024-03-11T21:45:04.778 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:45:04.778 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:45:04.779 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:45:04.879 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:45:04.908 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:45:04.908 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:45:04.909 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:45:05.009 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:45:05.033 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:45:05.034 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:45:05.034 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:45:05.083 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:45:04 smithi097 ceph-mon[33317]: pgmap v926: 148 pgs: 148 active+clean; 582 KiB data, 4.0 GiB used, 532 GiB / 536 GiB avail; 584 B/s rd, 0 op/s 2024-03-11T21:45:05.083 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:45:04 smithi097 ceph-mon[33317]: from='client.? v1:172.21.15.92:0/377583514' entity='client.admin' cmd='[{"prefix": "osd pool application enable", "pool": "unique_pool_4", "app": "rados", "yes_i_really_mean_it": true}]': finished 2024-03-11T21:45:05.083 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:45:04 smithi097 ceph-mon[33317]: osdmap e450: 8 total, 8 up, 6 in 2024-03-11T21:45:05.135 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:45:05.158 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:45:05.159 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:45:05.159 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:45:05.260 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:45:05.288 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:45:05.289 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:45:05.290 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:45:05.381 INFO:teuthology.orchestra.run.smithi092.stderr:Inferring config /var/lib/ceph/00db0016-dfed-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T21:45:05.390 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:45:05.418 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:45:05.419 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:45:05.419 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:45:05.520 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:45:05.544 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:45:05.545 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:45:05.545 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:45:05.646 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:45:05.674 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:45:05.675 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:45:05.676 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:45:05.776 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:45:05.805 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:45:05.805 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:45:05.806 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:45:05.907 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:45:05.935 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:45:05.936 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:45:05.936 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:45:06.037 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:45:06.047 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:45:05 smithi092 ceph-mon[29572]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-03-11T21:45:06.047 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:45:05 smithi092 ceph-mon[37563]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-03-11T21:45:06.063 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:45:06.063 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:45:06.064 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:45:06.083 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:45:05 smithi097 ceph-mon[33317]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-03-11T21:45:06.164 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:45:06.188 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:45:06.189 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:45:06.190 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:45:06.290 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:45:06.318 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:45:06.319 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:45:06.319 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:45:06.420 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:45:06.448 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:45:06.449 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:45:06.449 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:45:06.499 DEBUG:teuthology.orchestra.run.smithi092:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid 00db0016-dfed-11ee-95c8-87774f69a715 -- ceph tell osd.2 reset_purged_snaps_last 2024-03-11T21:45:06.550 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:45:06.577 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:45:06.578 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:45:06.578 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:45:06.679 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:45:06.706 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:45:06.707 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:45:06.707 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:45:06.808 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:45:06.835 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:45:06.835 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:45:06.836 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:45:06.937 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:45:06.961 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:45:06.962 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:45:06.962 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:45:06.994 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:45:06 smithi092 ceph-mon[29572]: pgmap v928: 148 pgs: 148 active+clean; 582 KiB data, 4.0 GiB used, 532 GiB / 536 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T21:45:06.995 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:45:06 smithi092 ceph-mon[37563]: pgmap v928: 148 pgs: 148 active+clean; 582 KiB data, 4.0 GiB used, 532 GiB / 536 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T21:45:07.063 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:45:07.075 DEBUG:teuthology.orchestra.run.smithi092:> /bin/sh -c 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados --no-log-to-stderr --name client.0 -b 65536 --object-size 65536 -p unique_pool_4 bench 90 write' 2024-03-11T21:45:07.077 INFO:tasks.radosbench:joining radosbench (timing out after 3000s) 2024-03-11T21:45:07.078 INFO:teuthology.orchestra.run:waiting for 3000 2024-03-11T21:45:07.083 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:45:06 smithi097 ceph-mon[33317]: pgmap v928: 148 pgs: 148 active+clean; 582 KiB data, 4.0 GiB used, 532 GiB / 536 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T21:45:07.090 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:45:07.090 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:45:07.091 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:45:07.192 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:45:07.215 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:45:07.216 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:45:07.216 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:45:07.253 INFO:tasks.radosbench.radosbench.0.smithi092.stdout:hints = 1 2024-03-11T21:45:07.253 INFO:tasks.radosbench.radosbench.0.smithi092.stdout:Maintaining 16 concurrent writes of 65536 bytes to objects of size 65536 for up to 90 seconds or 0 objects 2024-03-11T21:45:07.253 INFO:tasks.radosbench.radosbench.0.smithi092.stdout:Object prefix: benchmark_data_smithi092_186520 2024-03-11T21:45:07.253 INFO:tasks.radosbench.radosbench.0.smithi092.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-03-11T21:45:07.253 INFO:tasks.radosbench.radosbench.0.smithi092.stdout: 0 0 0 0 0 0 - 0 2024-03-11T21:45:07.317 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:45:07.355 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:45:07.356 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:45:07.357 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:45:07.458 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:45:07.498 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:45:07.499 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:45:07.499 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:45:07.600 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:45:07.642 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:45:07.642 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:45:07.643 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:45:07.743 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:45:07.781 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:45:07.782 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:45:07.782 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:45:07.883 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:45:07.922 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:45:07.923 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:45:07.923 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:45:08.024 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:45:08.057 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:45:08.058 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:45:08.058 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:45:08.079 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:45:07 smithi092 ceph-mon[29572]: purged_snaps scrub starts 2024-03-11T21:45:08.079 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:45:07 smithi092 ceph-mon[29572]: purged_snaps scrub ok 2024-03-11T21:45:08.079 INFO:journalctl@ceph.mgr.y.smithi092.stdout:Mar 11 21:45:07 smithi092 ceph-00db0016-dfed-11ee-95c8-87774f69a715-mgr-y[29801]: ::ffff:172.21.15.97 - - [11/Mar/2024:21:45:07] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-11T21:45:08.079 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:45:07 smithi092 ceph-mon[37563]: purged_snaps scrub starts 2024-03-11T21:45:08.079 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:45:07 smithi092 ceph-mon[37563]: purged_snaps scrub ok 2024-03-11T21:45:08.083 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:45:07 smithi097 ceph-mon[33317]: purged_snaps scrub starts 2024-03-11T21:45:08.083 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:45:07 smithi097 ceph-mon[33317]: purged_snaps scrub ok 2024-03-11T21:45:08.159 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:45:08.195 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:45:08.196 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:45:08.196 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:45:08.253 INFO:tasks.radosbench.radosbench.0.smithi092.stdout: 1 16 3723 3707 231.641 231.688 0.00957178 0.00430106 2024-03-11T21:45:08.297 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:45:08.343 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:45:08.344 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:45:08.344 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:45:08.417 INFO:teuthology.orchestra.run.smithi092.stdout:ERROR: (22) Invalid argument 2024-03-11T21:45:08.448 INFO:teuthology.orchestra.run.smithi092.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-11T21:45:08.449 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:45:08.491 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:45:08.491 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:45:08.491 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:45:08.592 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:45:08.631 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:45:08.631 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:45:08.632 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:45:08.732 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:45:08.763 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:45:08.764 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:45:08.764 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:45:08.865 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:45:08.899 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:45:08.900 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:45:08.900 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:45:09.001 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:45:09.040 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:45:09.040 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:45:09.041 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:45:09.083 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:45:08 smithi097 ceph-mon[33317]: pgmap v929: 148 pgs: 148 active+clean; 582 KiB data, 4.0 GiB used, 532 GiB / 536 GiB avail; 639 B/s rd, 0 op/s 2024-03-11T21:45:09.083 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:45:08 smithi097 ceph-mon[33317]: osdmap e451: 8 total, 8 up, 6 in 2024-03-11T21:45:09.083 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:45:08 smithi097 ceph-mon[33317]: from='mgr.14607 v1:172.21.15.92:0/1702717597' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T21:45:09.117 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:45:08 smithi092 ceph-mon[29572]: pgmap v929: 148 pgs: 148 active+clean; 582 KiB data, 4.0 GiB used, 532 GiB / 536 GiB avail; 639 B/s rd, 0 op/s 2024-03-11T21:45:09.117 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:45:08 smithi092 ceph-mon[29572]: osdmap e451: 8 total, 8 up, 6 in 2024-03-11T21:45:09.117 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:45:08 smithi092 ceph-mon[29572]: from='mgr.14607 v1:172.21.15.92:0/1702717597' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T21:45:09.118 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:45:08 smithi092 ceph-mon[37563]: pgmap v929: 148 pgs: 148 active+clean; 582 KiB data, 4.0 GiB used, 532 GiB / 536 GiB avail; 639 B/s rd, 0 op/s 2024-03-11T21:45:09.118 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:45:08 smithi092 ceph-mon[37563]: osdmap e451: 8 total, 8 up, 6 in 2024-03-11T21:45:09.118 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:45:08 smithi092 ceph-mon[37563]: from='mgr.14607 v1:172.21.15.92:0/1702717597' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T21:45:09.141 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:45:09.184 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:45:09.185 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:45:09.185 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:45:09.253 INFO:tasks.radosbench.radosbench.0.smithi092.stdout: 2 15 6961 6946 217.031 202.438 0.00271224 0.00459926 2024-03-11T21:45:09.286 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:45:09.326 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:45:09.326 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:45:09.327 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:45:09.427 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:45:09.467 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:45:09.467 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:45:09.468 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:45:09.568 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:45:09.609 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:45:09.610 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:45:09.610 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:45:09.661 DEBUG:teuthology.orchestra.run.smithi092:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid 00db0016-dfed-11ee-95c8-87774f69a715 -- ceph daemon osd.2 dump_blocked_ops 2024-03-11T21:45:09.711 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:45:09.745 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:45:09.745 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:45:09.746 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:45:09.846 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:45:09.879 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:45:09.879 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:45:09.880 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:45:09.981 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:45:10.013 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:45:10.014 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:45:10.014 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:45:10.115 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:45:10.148 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:45:10.149 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:45:10.149 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:45:10.250 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:45:10.253 INFO:tasks.radosbench.radosbench.0.smithi092.stdout: 3 16 10050 10034 208.988 193 0.00663587 0.0047739 2024-03-11T21:45:10.297 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:45:10.298 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:45:10.298 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:45:10.399 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:45:10.430 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:45:10.431 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:45:10.431 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:45:10.532 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:45:10.577 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:45:10.578 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:45:10.578 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:45:10.604 INFO:teuthology.orchestra.run.smithi092.stderr:Inferring config /var/lib/ceph/00db0016-dfed-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T21:45:10.617 INFO:teuthology.orchestra.run.smithi092.stderr:Inferring config /var/lib/ceph/00db0016-dfed-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T21:45:10.649 INFO:teuthology.orchestra.run.smithi092.stdout:osd.0: {} 2024-03-11T21:45:10.679 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:45:10.694 INFO:teuthology.orchestra.run.smithi092.stdout:osd.1: {} 2024-03-11T21:45:10.728 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:45:10.728 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:45:10.729 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:45:10.737 INFO:teuthology.orchestra.run.smithi092.stdout:osd.2: {} 2024-03-11T21:45:10.771 INFO:teuthology.orchestra.run.smithi092.stderr:noscrub is unset 2024-03-11T21:45:10.776 INFO:teuthology.orchestra.run.smithi092.stdout:osd.3: {} 2024-03-11T21:45:10.785 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:45:10 smithi092 ceph-mon[29572]: pgmap v931: 148 pgs: 148 active+clean; 139 MiB data, 4.2 GiB used, 532 GiB / 536 GiB avail; 1.2 KiB/s rd, 17 MiB/s wr, 278 op/s 2024-03-11T21:45:10.785 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:45:10 smithi092 ceph-mon[29572]: from='client.? v1:172.21.15.92:0/518868200' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-11T21:45:10.785 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:45:10 smithi092 ceph-mon[29572]: from='client.25825 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-11T21:45:10.785 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:45:10 smithi092 ceph-mon[29572]: from='client.? v1:172.21.15.92:0/1209680638' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-11T21:45:10.785 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:45:10 smithi092 ceph-mon[37563]: pgmap v931: 148 pgs: 148 active+clean; 139 MiB data, 4.2 GiB used, 532 GiB / 536 GiB avail; 1.2 KiB/s rd, 17 MiB/s wr, 278 op/s 2024-03-11T21:45:10.785 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:45:10 smithi092 ceph-mon[37563]: from='client.? v1:172.21.15.92:0/518868200' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-11T21:45:10.786 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:45:10 smithi092 ceph-mon[37563]: from='client.25825 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-11T21:45:10.786 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:45:10 smithi092 ceph-mon[37563]: from='client.? v1:172.21.15.92:0/1209680638' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-11T21:45:10.818 INFO:teuthology.orchestra.run.smithi092.stdout:osd.4: {} 2024-03-11T21:45:10.829 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:45:10.862 INFO:teuthology.orchestra.run.smithi092.stdout:osd.5: {} 2024-03-11T21:45:10.865 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:45:10.865 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:45:10.865 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:45:10.898 INFO:teuthology.orchestra.run.smithi092.stdout:osd.6: {} 2024-03-11T21:45:10.935 INFO:teuthology.orchestra.run.smithi092.stdout:osd.7: {} 2024-03-11T21:45:10.939 INFO:teuthology.orchestra.run.smithi092.stderr:osd.0: osd_enable_op_tracker = '' 2024-03-11T21:45:10.939 INFO:teuthology.orchestra.run.smithi092.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-11T21:45:10.939 INFO:teuthology.orchestra.run.smithi092.stderr:osd.2: osd_enable_op_tracker = '' 2024-03-11T21:45:10.939 INFO:teuthology.orchestra.run.smithi092.stderr:osd.3: osd_enable_op_tracker = '' 2024-03-11T21:45:10.939 INFO:teuthology.orchestra.run.smithi092.stderr:osd.4: osd_enable_op_tracker = '' 2024-03-11T21:45:10.939 INFO:teuthology.orchestra.run.smithi092.stderr:osd.5: osd_enable_op_tracker = '' 2024-03-11T21:45:10.939 INFO:teuthology.orchestra.run.smithi092.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-11T21:45:10.939 INFO:teuthology.orchestra.run.smithi092.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-11T21:45:10.967 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:45:11.000 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:45:11.000 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:45:11.001 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:45:11.083 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:45:10 smithi097 ceph-mon[33317]: pgmap v931: 148 pgs: 148 active+clean; 139 MiB data, 4.2 GiB used, 532 GiB / 536 GiB avail; 1.2 KiB/s rd, 17 MiB/s wr, 278 op/s 2024-03-11T21:45:11.083 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:45:10 smithi097 ceph-mon[33317]: from='client.? v1:172.21.15.92:0/518868200' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-11T21:45:11.083 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:45:10 smithi097 ceph-mon[33317]: from='client.25825 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-11T21:45:11.083 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:45:10 smithi097 ceph-mon[33317]: from='client.? v1:172.21.15.92:0/1209680638' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-11T21:45:11.102 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:45:11.143 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:45:11.144 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:45:11.144 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:45:11.245 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:45:11.254 INFO:tasks.radosbench.radosbench.0.smithi092.stdout: 4 16 12642 12626 197.239 162 0.00711255 0.00506411 2024-03-11T21:45:11.282 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:45:11.283 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:45:11.283 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:45:11.384 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:45:11.423 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:45:11.423 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:45:11.424 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:45:11.525 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:45:11.559 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:45:11.560 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:45:11.561 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:45:11.662 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:45:11.704 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:45:11.705 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:45:11.706 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:45:11.807 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:45:11.845 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:45:11.846 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:45:11.847 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:45:11.948 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:45:11.982 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:45:11.983 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:45:11.983 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:45:12.083 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:45:11 smithi097 ceph-mon[33317]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-11T21:45:12.083 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:45:11 smithi097 ceph-mon[33317]: from='client.25825 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-03-11T21:45:12.083 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:45:11 smithi097 ceph-mon[33317]: osdmap e452: 8 total, 8 up, 6 in 2024-03-11T21:45:12.084 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:45:12.117 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:45:11 smithi092 ceph-mon[29572]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-11T21:45:12.117 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:45:11 smithi092 ceph-mon[29572]: from='client.25825 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-03-11T21:45:12.117 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:45:11 smithi092 ceph-mon[29572]: osdmap e452: 8 total, 8 up, 6 in 2024-03-11T21:45:12.118 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:45:11 smithi092 ceph-mon[37563]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-11T21:45:12.118 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:45:11 smithi092 ceph-mon[37563]: from='client.25825 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-03-11T21:45:12.118 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:45:11 smithi092 ceph-mon[37563]: osdmap e452: 8 total, 8 up, 6 in 2024-03-11T21:45:12.119 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:45:12.120 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:45:12.120 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:45:12.221 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:45:12.254 INFO:tasks.radosbench.radosbench.0.smithi092.stdout: 5 16 15696 15680 195.963 190.875 0.00818274 0.00509697 2024-03-11T21:45:12.258 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:45:12.258 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:45:12.259 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:45:12.359 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:45:12.395 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:45:12.396 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:45:12.396 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:45:12.497 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:45:12.535 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:45:12.536 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:45:12.537 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:45:12.637 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:45:12.677 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:45:12.679 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:45:12.679 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:45:12.780 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:45:12.818 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:45:12.819 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:45:12.820 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:45:12.920 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:45:12.957 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:45:12.957 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:45:12.958 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:45:13.059 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:45:13.077 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:45:12 smithi092 ceph-mon[29572]: pgmap v933: 148 pgs: 148 active+clean; 171 MiB data, 4.4 GiB used, 532 GiB / 536 GiB avail; 661 B/s rd, 22 MiB/s wr, 352 op/s 2024-03-11T21:45:13.077 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:45:12 smithi092 ceph-mon[29572]: 3.1 scrub starts 2024-03-11T21:45:13.077 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:45:12 smithi092 ceph-mon[29572]: 3.1 scrub ok 2024-03-11T21:45:13.077 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:45:12 smithi092 ceph-mon[37563]: pgmap v933: 148 pgs: 148 active+clean; 171 MiB data, 4.4 GiB used, 532 GiB / 536 GiB avail; 661 B/s rd, 22 MiB/s wr, 352 op/s 2024-03-11T21:45:13.077 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:45:12 smithi092 ceph-mon[37563]: 3.1 scrub starts 2024-03-11T21:45:13.077 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:45:12 smithi092 ceph-mon[37563]: 3.1 scrub ok 2024-03-11T21:45:13.083 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:45:12 smithi097 ceph-mon[33317]: pgmap v933: 148 pgs: 148 active+clean; 171 MiB data, 4.4 GiB used, 532 GiB / 536 GiB avail; 661 B/s rd, 22 MiB/s wr, 352 op/s 2024-03-11T21:45:13.083 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:45:12 smithi097 ceph-mon[33317]: 3.1 scrub starts 2024-03-11T21:45:13.083 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:45:12 smithi097 ceph-mon[33317]: 3.1 scrub ok 2024-03-11T21:45:13.101 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:45:13.101 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:45:13.102 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:45:13.203 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:45:13.240 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:45:13.241 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:45:13.241 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:45:13.254 INFO:tasks.radosbench.radosbench.0.smithi092.stdout: 6 16 18564 18548 193.175 179.25 0.00375083 0.00516989 2024-03-11T21:45:13.343 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:45:13.379 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:45:13.379 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:45:13.380 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:45:13.481 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:45:13.518 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:45:13.519 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:45:13.519 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:45:13.583 INFO:journalctl@ceph.iscsi.iscsi.a.smithi097.stdout:Mar 11 21:45:13 smithi097 ceph-00db0016-dfed-11ee-95c8-87774f69a715-iscsi-iscsi-a[60943]: debug there is no tcmu-runner data available 2024-03-11T21:45:13.606 DEBUG:teuthology.orchestra.run.smithi092:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid 00db0016-dfed-11ee-95c8-87774f69a715 -- ceph osd unset nodeep-scrub 2024-03-11T21:45:13.620 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:45:13.703 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:45:13.704 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:45:13.704 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:45:13.805 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:45:13.839 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:45:13.840 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:45:13.840 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:45:13.941 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:45:13.975 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:45:13.976 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:45:13.977 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:45:14.078 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:45:14.083 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:45:13 smithi097 ceph-mon[33317]: 4.4 scrub starts 2024-03-11T21:45:14.083 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:45:13 smithi097 ceph-mon[33317]: 4.4 scrub ok 2024-03-11T21:45:14.083 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:45:13 smithi097 ceph-mon[33317]: from='client.24469 v1:172.21.15.97:0/404208310' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:45:14.112 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:45:14.113 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:45:14.113 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:45:14.117 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:45:13 smithi092 ceph-mon[29572]: 4.4 scrub starts 2024-03-11T21:45:14.117 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:45:13 smithi092 ceph-mon[29572]: 4.4 scrub ok 2024-03-11T21:45:14.117 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:45:13 smithi092 ceph-mon[29572]: from='client.24469 v1:172.21.15.97:0/404208310' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:45:14.118 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:45:13 smithi092 ceph-mon[37563]: 4.4 scrub starts 2024-03-11T21:45:14.118 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:45:13 smithi092 ceph-mon[37563]: 4.4 scrub ok 2024-03-11T21:45:14.118 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:45:13 smithi092 ceph-mon[37563]: from='client.24469 v1:172.21.15.97:0/404208310' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:45:14.214 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:45:14.247 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:45:14.248 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:45:14.248 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:45:14.254 INFO:tasks.radosbench.radosbench.0.smithi092.stdout: 7 16 21183 21167 188.961 163.688 0.0029641 0.00528561 2024-03-11T21:45:14.350 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:45:14.384 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:45:14.384 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:45:14.385 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:45:14.486 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:45:14.526 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:45:14.527 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:45:14.527 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:45:14.628 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:45:14.672 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:45:14.672 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:45:14.673 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:45:14.774 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:45:14.812 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:45:14.812 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:45:14.813 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:45:14.914 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:45:14.914 DEBUG:teuthology.orchestra.run.smithi092:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid 00db0016-dfed-11ee-95c8-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-03-11T21:45:14.948 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:45:14.949 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:45:14.949 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:45:15.050 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:45:15.083 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:45:14 smithi097 ceph-mon[33317]: pgmap v934: 148 pgs: 148 active+clean; 833 MiB data, 6.0 GiB used, 530 GiB / 536 GiB avail; 639 B/s rd, 104 MiB/s wr, 1.66k op/s 2024-03-11T21:45:15.083 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:45:14 smithi097 ceph-mon[33317]: 5.12 scrub starts 2024-03-11T21:45:15.083 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:45:14 smithi097 ceph-mon[33317]: 5.12 scrub ok 2024-03-11T21:45:15.093 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:45:15.095 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:45:15.095 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:45:15.117 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:45:14 smithi092 ceph-mon[29572]: pgmap v934: 148 pgs: 148 active+clean; 833 MiB data, 6.0 GiB used, 530 GiB / 536 GiB avail; 639 B/s rd, 104 MiB/s wr, 1.66k op/s 2024-03-11T21:45:15.117 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:45:14 smithi092 ceph-mon[29572]: 5.12 scrub starts 2024-03-11T21:45:15.117 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:45:14 smithi092 ceph-mon[29572]: 5.12 scrub ok 2024-03-11T21:45:15.118 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:45:14 smithi092 ceph-mon[37563]: pgmap v934: 148 pgs: 148 active+clean; 833 MiB data, 6.0 GiB used, 530 GiB / 536 GiB avail; 639 B/s rd, 104 MiB/s wr, 1.66k op/s 2024-03-11T21:45:15.118 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:45:14 smithi092 ceph-mon[37563]: 5.12 scrub starts 2024-03-11T21:45:15.118 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:45:14 smithi092 ceph-mon[37563]: 5.12 scrub ok 2024-03-11T21:45:15.197 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:45:15.244 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:45:15.245 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:45:15.245 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:45:15.255 INFO:tasks.radosbench.radosbench.0.smithi092.stdout: 8 16 23592 23576 184.16 150.562 0.00761255 0.00542546 2024-03-11T21:45:15.347 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:45:15.395 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:45:15.395 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:45:15.396 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:45:15.444 INFO:teuthology.orchestra.run.smithi092.stderr:Inferring config /var/lib/ceph/00db0016-dfed-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T21:45:15.497 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:45:15.540 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:45:15.541 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:45:15.541 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:45:15.642 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:45:15.684 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:45:15.685 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:45:15.685 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:45:15.786 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:45:15.820 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:45:15.821 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:45:15.821 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:45:15.922 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:45:15.960 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:45:15.960 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:45:15.961 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:45:15.967 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:45:15 smithi092 ceph-mon[29572]: 4.1e scrub starts 2024-03-11T21:45:15.967 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:45:15 smithi092 ceph-mon[29572]: 4.1e scrub ok 2024-03-11T21:45:15.967 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:45:15 smithi092 ceph-mon[37563]: 4.1e scrub starts 2024-03-11T21:45:15.967 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:45:15 smithi092 ceph-mon[37563]: 4.1e scrub ok 2024-03-11T21:45:16.062 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:45:16.083 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:45:15 smithi097 ceph-mon[33317]: 4.1e scrub starts 2024-03-11T21:45:16.083 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:45:15 smithi097 ceph-mon[33317]: 4.1e scrub ok 2024-03-11T21:45:16.097 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:45:16.097 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:45:16.098 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:45:16.198 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:45:16.233 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:45:16.234 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:45:16.234 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:45:16.254 INFO:tasks.radosbench.radosbench.0.smithi092.stdout: 9 16 25692 25676 178.28 131.25 0.014927 0.00560326 2024-03-11T21:45:16.335 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:45:16.369 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:45:16.370 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:45:16.370 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:45:16.471 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:45:16.505 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:45:16.505 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:45:16.506 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:45:16.606 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:45:16.647 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:45:16.648 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:45:16.648 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:45:16.749 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:45:16.784 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:45:16.784 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:45:16.785 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:45:16.867 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:45:16 smithi092 ceph-mon[29572]: pgmap v935: 148 pgs: 148 active+clean; 1.1 GiB data, 7.1 GiB used, 529 GiB / 536 GiB avail; 1.2 KiB/s rd, 139 MiB/s wr, 2.23k op/s 2024-03-11T21:45:16.867 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:45:16 smithi092 ceph-mon[29572]: 2.c scrub starts 2024-03-11T21:45:16.867 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:45:16 smithi092 ceph-mon[29572]: 2.c scrub ok 2024-03-11T21:45:16.868 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:45:16 smithi092 ceph-mon[37563]: pgmap v935: 148 pgs: 148 active+clean; 1.1 GiB data, 7.1 GiB used, 529 GiB / 536 GiB avail; 1.2 KiB/s rd, 139 MiB/s wr, 2.23k op/s 2024-03-11T21:45:16.868 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:45:16 smithi092 ceph-mon[37563]: 2.c scrub starts 2024-03-11T21:45:16.868 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:45:16 smithi092 ceph-mon[37563]: 2.c scrub ok 2024-03-11T21:45:16.885 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:45:16.924 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:45:16.924 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:45:16.925 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:45:17.026 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:45:17.064 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:45:17.064 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:45:17.065 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:45:17.083 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:45:16 smithi097 ceph-mon[33317]: pgmap v935: 148 pgs: 148 active+clean; 1.1 GiB data, 7.1 GiB used, 529 GiB / 536 GiB avail; 1.2 KiB/s rd, 139 MiB/s wr, 2.23k op/s 2024-03-11T21:45:17.083 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:45:16 smithi097 ceph-mon[33317]: 2.c scrub starts 2024-03-11T21:45:17.083 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:45:16 smithi097 ceph-mon[33317]: 2.c scrub ok 2024-03-11T21:45:17.166 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:45:17.189 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:45:17.190 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:45:17.190 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:45:17.254 INFO:tasks.radosbench.radosbench.0.smithi092.stdout: 10 15 27737 27722 173.239 127.875 0.00998648 0.00576675 2024-03-11T21:45:17.291 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:45:17.328 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:45:17.329 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:45:17.330 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:45:17.430 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:45:17.468 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:45:17.469 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:45:17.469 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:45:17.567 DEBUG:teuthology.orchestra.run.smithi092:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid 00db0016-dfed-11ee-95c8-87774f69a715 -- ceph tell osd.5 reset_purged_snaps_last 2024-03-11T21:45:17.570 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:45:17.667 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:45:17.668 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:45:17.668 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:45:17.769 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:45:17.801 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:45:17.802 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:45:17.803 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:45:17.869 INFO:teuthology.orchestra.run.smithi092.stdout:{ 2024-03-11T21:45:17.869 INFO:teuthology.orchestra.run.smithi092.stdout: "ops": [], 2024-03-11T21:45:17.869 INFO:teuthology.orchestra.run.smithi092.stdout: "complaint_time": 30, 2024-03-11T21:45:17.869 INFO:teuthology.orchestra.run.smithi092.stdout: "num_blocked_ops": 0 2024-03-11T21:45:17.870 INFO:teuthology.orchestra.run.smithi092.stdout:} 2024-03-11T21:45:17.903 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:45:17.936 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:45:17.937 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:45:17.937 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:45:18.038 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:45:18.080 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:45:18.081 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:45:18.081 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:45:18.083 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:45:17 smithi097 ceph-mon[33317]: 5.18 scrub starts 2024-03-11T21:45:18.083 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:45:17 smithi097 ceph-mon[33317]: 5.18 scrub ok 2024-03-11T21:45:18.083 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:45:17 smithi097 ceph-mon[33317]: 2.6 scrub starts 2024-03-11T21:45:18.083 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:45:17 smithi097 ceph-mon[33317]: 2.6 scrub ok 2024-03-11T21:45:18.083 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:45:17 smithi097 ceph-mon[33317]: 6.13 scrub starts 2024-03-11T21:45:18.083 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:45:17 smithi097 ceph-mon[33317]: 6.13 scrub ok 2024-03-11T21:45:18.117 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:45:17 smithi092 ceph-mon[29572]: 5.18 scrub starts 2024-03-11T21:45:18.118 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:45:17 smithi092 ceph-mon[29572]: 5.18 scrub ok 2024-03-11T21:45:18.118 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:45:17 smithi092 ceph-mon[29572]: 2.6 scrub starts 2024-03-11T21:45:18.118 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:45:17 smithi092 ceph-mon[29572]: 2.6 scrub ok 2024-03-11T21:45:18.118 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:45:17 smithi092 ceph-mon[29572]: 6.13 scrub starts 2024-03-11T21:45:18.118 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:45:17 smithi092 ceph-mon[29572]: 6.13 scrub ok 2024-03-11T21:45:18.118 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:45:17 smithi092 ceph-mon[37563]: 5.18 scrub starts 2024-03-11T21:45:18.118 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:45:17 smithi092 ceph-mon[37563]: 5.18 scrub ok 2024-03-11T21:45:18.118 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:45:17 smithi092 ceph-mon[37563]: 2.6 scrub starts 2024-03-11T21:45:18.118 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:45:17 smithi092 ceph-mon[37563]: 2.6 scrub ok 2024-03-11T21:45:18.119 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:45:17 smithi092 ceph-mon[37563]: 6.13 scrub starts 2024-03-11T21:45:18.119 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:45:17 smithi092 ceph-mon[37563]: 6.13 scrub ok 2024-03-11T21:45:18.119 INFO:journalctl@ceph.mgr.y.smithi092.stdout:Mar 11 21:45:17 smithi092 ceph-00db0016-dfed-11ee-95c8-87774f69a715-mgr-y[29801]: ::ffff:172.21.15.97 - - [11/Mar/2024:21:45:17] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-11T21:45:18.182 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:45:18.220 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:45:18.221 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:45:18.221 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:45:18.254 INFO:tasks.radosbench.radosbench.0.smithi092.stdout: 11 15 29841 29826 169.443 131.5 0.0185806 0.0058955 2024-03-11T21:45:18.322 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:45:18.345 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:45:18.345 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:45:18.346 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:45:18.446 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:45:18.476 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:45:18.477 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:45:18.478 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:45:18.579 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:45:18.615 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:45:18.616 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:45:18.616 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:45:18.717 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:45:18.741 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:45:18.741 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:45:18.742 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:45:18.842 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:45:18.882 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:45:18.883 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:45:18.884 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:45:18.984 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:45:19.012 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:45:19.012 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:45:19.012 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:45:19.083 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:45:18 smithi097 ceph-mon[33317]: pgmap v936: 148 pgs: 148 active+clean; 1.1 GiB data, 7.4 GiB used, 529 GiB / 536 GiB avail; 1.0 KiB/s rd, 116 MiB/s wr, 1.86k op/s 2024-03-11T21:45:19.083 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:45:18 smithi097 ceph-mon[33317]: 4.f scrub starts 2024-03-11T21:45:19.084 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:45:18 smithi097 ceph-mon[33317]: 4.f scrub ok 2024-03-11T21:45:19.084 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:45:18 smithi097 ceph-mon[33317]: 6.e scrub starts 2024-03-11T21:45:19.084 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:45:18 smithi097 ceph-mon[33317]: 6.e scrub ok 2024-03-11T21:45:19.084 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:45:18 smithi097 ceph-mon[33317]: 5.3 scrub starts 2024-03-11T21:45:19.084 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:45:18 smithi097 ceph-mon[33317]: 5.3 scrub ok 2024-03-11T21:45:19.114 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:45:19.117 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:45:18 smithi092 ceph-mon[29572]: pgmap v936: 148 pgs: 148 active+clean; 1.1 GiB data, 7.4 GiB used, 529 GiB / 536 GiB avail; 1.0 KiB/s rd, 116 MiB/s wr, 1.86k op/s 2024-03-11T21:45:19.117 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:45:18 smithi092 ceph-mon[29572]: 4.f scrub starts 2024-03-11T21:45:19.117 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:45:18 smithi092 ceph-mon[29572]: 4.f scrub ok 2024-03-11T21:45:19.117 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:45:18 smithi092 ceph-mon[29572]: 6.e scrub starts 2024-03-11T21:45:19.117 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:45:18 smithi092 ceph-mon[29572]: 6.e scrub ok 2024-03-11T21:45:19.117 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:45:18 smithi092 ceph-mon[29572]: 5.3 scrub starts 2024-03-11T21:45:19.118 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:45:18 smithi092 ceph-mon[29572]: 5.3 scrub ok 2024-03-11T21:45:19.118 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:45:18 smithi092 ceph-mon[37563]: pgmap v936: 148 pgs: 148 active+clean; 1.1 GiB data, 7.4 GiB used, 529 GiB / 536 GiB avail; 1.0 KiB/s rd, 116 MiB/s wr, 1.86k op/s 2024-03-11T21:45:19.118 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:45:18 smithi092 ceph-mon[37563]: 4.f scrub starts 2024-03-11T21:45:19.118 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:45:18 smithi092 ceph-mon[37563]: 4.f scrub ok 2024-03-11T21:45:19.118 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:45:18 smithi092 ceph-mon[37563]: 6.e scrub starts 2024-03-11T21:45:19.118 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:45:18 smithi092 ceph-mon[37563]: 6.e scrub ok 2024-03-11T21:45:19.118 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:45:18 smithi092 ceph-mon[37563]: 5.3 scrub starts 2024-03-11T21:45:19.118 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:45:18 smithi092 ceph-mon[37563]: 5.3 scrub ok 2024-03-11T21:45:19.148 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:45:19.149 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:45:19.150 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:45:19.251 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:45:19.254 INFO:tasks.radosbench.radosbench.0.smithi092.stdout: 12 16 31507 31491 163.994 104.062 0.00565653 0.00609275 2024-03-11T21:45:19.289 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:45:19.290 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:45:19.290 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:45:19.393 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:45:19.431 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:45:19.432 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:45:19.432 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:45:19.533 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:45:19.568 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:45:19.569 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:45:19.570 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:45:19.671 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:45:19.709 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:45:19.710 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:45:19.710 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:45:19.811 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:45:19.854 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:45:19.855 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:45:19.855 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:45:19.956 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:45:19.997 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:45:19.997 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:45:19.998 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:45:20.083 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:45:19 smithi097 ceph-mon[33317]: 4.c scrub starts 2024-03-11T21:45:20.083 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:45:19 smithi097 ceph-mon[33317]: 4.c scrub ok 2024-03-11T21:45:20.083 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:45:19 smithi097 ceph-mon[33317]: 5.c scrub starts 2024-03-11T21:45:20.084 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:45:19 smithi097 ceph-mon[33317]: 5.c scrub ok 2024-03-11T21:45:20.098 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:45:20.117 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:45:19 smithi092 ceph-mon[29572]: 4.c scrub starts 2024-03-11T21:45:20.117 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:45:19 smithi092 ceph-mon[29572]: 4.c scrub ok 2024-03-11T21:45:20.118 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:45:19 smithi092 ceph-mon[29572]: 5.c scrub starts 2024-03-11T21:45:20.118 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:45:19 smithi092 ceph-mon[29572]: 5.c scrub ok 2024-03-11T21:45:20.118 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:45:19 smithi092 ceph-mon[37563]: 4.c scrub starts 2024-03-11T21:45:20.118 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:45:19 smithi092 ceph-mon[37563]: 4.c scrub ok 2024-03-11T21:45:20.118 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:45:19 smithi092 ceph-mon[37563]: 5.c scrub starts 2024-03-11T21:45:20.118 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:45:19 smithi092 ceph-mon[37563]: 5.c scrub ok 2024-03-11T21:45:20.136 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:45:20.137 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:45:20.138 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:45:20.238 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:45:20.254 INFO:tasks.radosbench.radosbench.0.smithi092.stdout: 13 16 33512 33496 161.018 125.312 0.0106225 0.00620541 2024-03-11T21:45:20.270 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:45:20.270 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:45:20.271 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:45:20.372 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:45:20.408 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:45:20.408 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:45:20.409 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:45:20.510 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:45:20.533 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:45:20.533 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:45:20.534 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:45:20.634 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:45:20.668 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:45:20.669 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:45:20.670 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:45:20.770 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:45:20.795 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:45:20.795 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:45:20.796 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:45:20.896 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:45:20.933 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:45:20.934 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:45:20.934 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:45:21.035 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:45:21.064 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:45:21.065 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:45:21.065 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:45:21.083 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:45:20 smithi097 ceph-mon[33317]: pgmap v937: 148 pgs: 148 active+clean; 1.7 GiB data, 9.1 GiB used, 527 GiB / 536 GiB avail; 1023 B/s rd, 165 MiB/s wr, 2.64k op/s 2024-03-11T21:45:21.083 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:45:20 smithi097 ceph-mon[33317]: 4.19 scrub starts 2024-03-11T21:45:21.083 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:45:20 smithi097 ceph-mon[33317]: 4.19 scrub ok 2024-03-11T21:45:21.083 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:45:20 smithi097 ceph-mon[33317]: 4.1 scrub starts 2024-03-11T21:45:21.083 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:45:20 smithi097 ceph-mon[33317]: 4.1 scrub ok 2024-03-11T21:45:21.117 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:45:20 smithi092 ceph-mon[29572]: pgmap v937: 148 pgs: 148 active+clean; 1.7 GiB data, 9.1 GiB used, 527 GiB / 536 GiB avail; 1023 B/s rd, 165 MiB/s wr, 2.64k op/s 2024-03-11T21:45:21.117 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:45:20 smithi092 ceph-mon[29572]: 4.19 scrub starts 2024-03-11T21:45:21.118 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:45:20 smithi092 ceph-mon[29572]: 4.19 scrub ok 2024-03-11T21:45:21.118 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:45:20 smithi092 ceph-mon[29572]: 4.1 scrub starts 2024-03-11T21:45:21.118 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:45:20 smithi092 ceph-mon[29572]: 4.1 scrub ok 2024-03-11T21:45:21.118 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:45:20 smithi092 ceph-mon[37563]: pgmap v937: 148 pgs: 148 active+clean; 1.7 GiB data, 9.1 GiB used, 527 GiB / 536 GiB avail; 1023 B/s rd, 165 MiB/s wr, 2.64k op/s 2024-03-11T21:45:21.118 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:45:20 smithi092 ceph-mon[37563]: 4.19 scrub starts 2024-03-11T21:45:21.118 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:45:20 smithi092 ceph-mon[37563]: 4.19 scrub ok 2024-03-11T21:45:21.118 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:45:20 smithi092 ceph-mon[37563]: 4.1 scrub starts 2024-03-11T21:45:21.118 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:45:20 smithi092 ceph-mon[37563]: 4.1 scrub ok 2024-03-11T21:45:21.167 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:45:21.190 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:45:21.190 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:45:21.191 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:45:21.255 INFO:tasks.radosbench.radosbench.0.smithi092.stdout: 14 16 34867 34851 155.565 84.6875 0.00236235 0.00632805 2024-03-11T21:45:21.292 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:45:21.332 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:45:21.333 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:45:21.333 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:45:21.434 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:45:21.467 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:45:21.468 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:45:21.468 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:45:21.569 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:45:21.583 INFO:teuthology.orchestra.run.smithi092.stderr:Inferring config /var/lib/ceph/00db0016-dfed-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T21:45:21.602 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:45:21.603 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:45:21.603 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:45:21.704 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:45:21.733 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:45:21.734 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:45:21.735 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:45:21.835 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:45:21.875 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:45:21.875 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:45:21.876 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:45:21.976 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:45:22.012 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:45:22.013 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:45:22.013 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:45:22.114 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:45:22.117 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:45:21 smithi092 ceph-mon[29572]: 5.a scrub starts 2024-03-11T21:45:22.117 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:45:21 smithi092 ceph-mon[29572]: 5.a scrub ok 2024-03-11T21:45:22.117 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:45:21 smithi092 ceph-mon[29572]: 2.5 scrub starts 2024-03-11T21:45:22.117 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:45:21 smithi092 ceph-mon[29572]: 2.5 scrub ok 2024-03-11T21:45:22.118 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:45:21 smithi092 ceph-mon[37563]: 5.a scrub starts 2024-03-11T21:45:22.118 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:45:21 smithi092 ceph-mon[37563]: 5.a scrub ok 2024-03-11T21:45:22.118 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:45:21 smithi092 ceph-mon[37563]: 2.5 scrub starts 2024-03-11T21:45:22.118 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:45:21 smithi092 ceph-mon[37563]: 2.5 scrub ok 2024-03-11T21:45:22.141 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:45:22.141 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:45:22.142 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:45:22.243 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:45:22.257 INFO:tasks.radosbench.radosbench.0.smithi092.stdout: 15 16 36791 36775 153.192 120.25 0.00665615 0.00652245 2024-03-11T21:45:22.282 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:45:22.283 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:45:22.283 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:45:22.333 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:45:21 smithi097 ceph-mon[33317]: 5.a scrub starts 2024-03-11T21:45:22.333 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:45:21 smithi097 ceph-mon[33317]: 5.a scrub ok 2024-03-11T21:45:22.334 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:45:21 smithi097 ceph-mon[33317]: 2.5 scrub starts 2024-03-11T21:45:22.334 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:45:21 smithi097 ceph-mon[33317]: 2.5 scrub ok 2024-03-11T21:45:22.384 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:45:22.417 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:45:22.418 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:45:22.419 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:45:22.519 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:45:22.551 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:45:22.552 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:45:22.552 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:45:22.653 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:45:22.684 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:45:22.685 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:45:22.685 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:45:22.786 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:45:22.818 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:45:22.819 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:45:22.819 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:45:22.920 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:45:22.944 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:45:22.944 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:45:22.945 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:45:23.045 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:45:23.083 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:45:23.084 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:45:23.084 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:45:23.104 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:45:22 smithi097 ceph-mon[33317]: pgmap v938: 148 pgs: 148 active+clean; 1.8 GiB data, 9.4 GiB used, 527 GiB / 536 GiB avail; 960 B/s rd, 156 MiB/s wr, 2.49k op/s 2024-03-11T21:45:23.104 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:45:22 smithi097 ceph-mon[33317]: 2.1 scrub starts 2024-03-11T21:45:23.104 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:45:22 smithi097 ceph-mon[33317]: 2.1 scrub ok 2024-03-11T21:45:23.104 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:45:22 smithi097 ceph-mon[33317]: 4.15 scrub starts 2024-03-11T21:45:23.104 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:45:22 smithi097 ceph-mon[33317]: 4.15 scrub ok 2024-03-11T21:45:23.117 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:45:22 smithi092 ceph-mon[29572]: pgmap v938: 148 pgs: 148 active+clean; 1.8 GiB data, 9.4 GiB used, 527 GiB / 536 GiB avail; 960 B/s rd, 156 MiB/s wr, 2.49k op/s 2024-03-11T21:45:23.117 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:45:22 smithi092 ceph-mon[29572]: 2.1 scrub starts 2024-03-11T21:45:23.117 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:45:22 smithi092 ceph-mon[29572]: 2.1 scrub ok 2024-03-11T21:45:23.117 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:45:22 smithi092 ceph-mon[29572]: 4.15 scrub starts 2024-03-11T21:45:23.117 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:45:22 smithi092 ceph-mon[29572]: 4.15 scrub ok 2024-03-11T21:45:23.118 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:45:22 smithi092 ceph-mon[37563]: pgmap v938: 148 pgs: 148 active+clean; 1.8 GiB data, 9.4 GiB used, 527 GiB / 536 GiB avail; 960 B/s rd, 156 MiB/s wr, 2.49k op/s 2024-03-11T21:45:23.118 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:45:22 smithi092 ceph-mon[37563]: 2.1 scrub starts 2024-03-11T21:45:23.118 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:45:22 smithi092 ceph-mon[37563]: 2.1 scrub ok 2024-03-11T21:45:23.118 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:45:22 smithi092 ceph-mon[37563]: 4.15 scrub starts 2024-03-11T21:45:23.118 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:45:22 smithi092 ceph-mon[37563]: 4.15 scrub ok 2024-03-11T21:45:23.186 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:45:23.225 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:45:23.225 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:45:23.226 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:45:23.257 INFO:tasks.radosbench.radosbench.0.smithi092.stdout: 16 15 38573 38558 150.582 111.438 0.0108192 0.00663656 2024-03-11T21:45:23.326 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:45:23.356 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:45:23.357 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:45:23.357 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:45:23.458 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:45:23.494 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:45:23.494 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:45:23.495 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:45:23.583 INFO:journalctl@ceph.iscsi.iscsi.a.smithi097.stdout:Mar 11 21:45:23 smithi097 ceph-00db0016-dfed-11ee-95c8-87774f69a715-iscsi-iscsi-a[60943]: debug there is no tcmu-runner data available 2024-03-11T21:45:23.595 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:45:23.635 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:45:23.635 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:45:23.636 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:45:23.736 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:45:23.765 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:45:23.766 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:45:23.767 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:45:23.867 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:45:23.907 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:45:23.907 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:45:23.908 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:45:24.009 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:45:24.042 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:45:24.042 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:45:24.043 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:45:24.117 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:45:23 smithi092 ceph-mon[29572]: purged_snaps scrub starts 2024-03-11T21:45:24.118 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:45:23 smithi092 ceph-mon[29572]: purged_snaps scrub ok 2024-03-11T21:45:24.118 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:45:23 smithi092 ceph-mon[29572]: osdmap e453: 8 total, 8 up, 6 in 2024-03-11T21:45:24.118 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:45:23 smithi092 ceph-mon[29572]: 6.1d scrub starts 2024-03-11T21:45:24.118 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:45:23 smithi092 ceph-mon[29572]: 6.1d scrub ok 2024-03-11T21:45:24.118 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:45:23 smithi092 ceph-mon[29572]: from='mgr.14607 v1:172.21.15.92:0/1702717597' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T21:45:24.118 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:45:23 smithi092 ceph-mon[29572]: 2.12 scrub starts 2024-03-11T21:45:24.118 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:45:23 smithi092 ceph-mon[29572]: 2.12 scrub ok 2024-03-11T21:45:24.118 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:45:23 smithi092 ceph-mon[29572]: from='client.24469 v1:172.21.15.97:0/404208310' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:45:24.118 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:45:23 smithi092 ceph-mon[29572]: purged_snaps scrub starts 2024-03-11T21:45:24.118 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:45:23 smithi092 ceph-mon[29572]: purged_snaps scrub ok 2024-03-11T21:45:24.119 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:45:23 smithi092 ceph-mon[37563]: purged_snaps scrub starts 2024-03-11T21:45:24.119 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:45:23 smithi092 ceph-mon[37563]: purged_snaps scrub ok 2024-03-11T21:45:24.119 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:45:23 smithi092 ceph-mon[37563]: osdmap e453: 8 total, 8 up, 6 in 2024-03-11T21:45:24.119 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:45:23 smithi092 ceph-mon[37563]: 6.1d scrub starts 2024-03-11T21:45:24.119 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:45:23 smithi092 ceph-mon[37563]: 6.1d scrub ok 2024-03-11T21:45:24.119 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:45:23 smithi092 ceph-mon[37563]: from='mgr.14607 v1:172.21.15.92:0/1702717597' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T21:45:24.119 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:45:23 smithi092 ceph-mon[37563]: 2.12 scrub starts 2024-03-11T21:45:24.119 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:45:23 smithi092 ceph-mon[37563]: 2.12 scrub ok 2024-03-11T21:45:24.119 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:45:23 smithi092 ceph-mon[37563]: from='client.24469 v1:172.21.15.97:0/404208310' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:45:24.119 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:45:23 smithi092 ceph-mon[37563]: purged_snaps scrub starts 2024-03-11T21:45:24.119 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:45:23 smithi092 ceph-mon[37563]: purged_snaps scrub ok 2024-03-11T21:45:24.144 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:45:24.184 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:45:24.185 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:45:24.186 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:45:24.257 INFO:tasks.radosbench.radosbench.0.smithi092.stdout: 17 16 40212 40196 147.746 102.375 0.00851022 0.0067635 2024-03-11T21:45:24.287 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:45:24.324 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:45:24.325 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:45:24.326 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:45:24.333 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:45:23 smithi097 ceph-mon[33317]: purged_snaps scrub starts 2024-03-11T21:45:24.333 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:45:23 smithi097 ceph-mon[33317]: purged_snaps scrub ok 2024-03-11T21:45:24.333 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:45:23 smithi097 ceph-mon[33317]: osdmap e453: 8 total, 8 up, 6 in 2024-03-11T21:45:24.333 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:45:23 smithi097 ceph-mon[33317]: 6.1d scrub starts 2024-03-11T21:45:24.334 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:45:23 smithi097 ceph-mon[33317]: 6.1d scrub ok 2024-03-11T21:45:24.334 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:45:23 smithi097 ceph-mon[33317]: from='mgr.14607 v1:172.21.15.92:0/1702717597' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T21:45:24.334 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:45:23 smithi097 ceph-mon[33317]: 2.12 scrub starts 2024-03-11T21:45:24.334 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:45:23 smithi097 ceph-mon[33317]: 2.12 scrub ok 2024-03-11T21:45:24.334 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:45:23 smithi097 ceph-mon[33317]: from='client.24469 v1:172.21.15.97:0/404208310' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:45:24.334 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:45:23 smithi097 ceph-mon[33317]: purged_snaps scrub starts 2024-03-11T21:45:24.334 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:45:23 smithi097 ceph-mon[33317]: purged_snaps scrub ok 2024-03-11T21:45:24.427 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:45:24.458 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:45:24.459 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:45:24.459 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:45:24.560 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:45:24.589 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:45:24.590 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:45:24.590 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:45:24.691 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:45:24.721 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:45:24.722 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:45:24.722 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:45:24.823 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:45:24.861 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:45:24.861 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:45:24.862 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:45:24.963 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:45:24.987 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:45:24.988 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:45:24.988 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:45:25.089 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:45:25.125 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:45:25.126 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:45:25.126 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:45:25.227 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:45:25.257 INFO:tasks.radosbench.radosbench.0.smithi092.stdout: 18 16 41883 41867 145.337 104.438 0.00678053 0.00687405 2024-03-11T21:45:25.264 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:45:25.265 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:45:25.265 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:45:25.333 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:45:24 smithi097 ceph-mon[33317]: pgmap v940: 148 pgs: 148 active+clean; 2.2 GiB data, 10 GiB used, 526 GiB / 536 GiB avail; 1023 B/s rd, 138 MiB/s wr, 2.21k op/s 2024-03-11T21:45:25.333 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:45:24 smithi097 ceph-mon[33317]: osdmap e454: 8 total, 8 up, 6 in 2024-03-11T21:45:25.333 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:45:24 smithi097 ceph-mon[33317]: 4.12 scrub starts 2024-03-11T21:45:25.334 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:45:24 smithi097 ceph-mon[33317]: 4.12 scrub ok 2024-03-11T21:45:25.334 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:45:24 smithi097 ceph-mon[33317]: 2.16 scrub starts 2024-03-11T21:45:25.334 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:45:24 smithi097 ceph-mon[33317]: 2.16 scrub ok 2024-03-11T21:45:25.334 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:45:24 smithi097 ceph-mon[33317]: 2.1c scrub starts 2024-03-11T21:45:25.334 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:45:24 smithi097 ceph-mon[33317]: 2.1c scrub ok 2024-03-11T21:45:25.365 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:45:25.368 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:45:24 smithi092 ceph-mon[29572]: pgmap v940: 148 pgs: 148 active+clean; 2.2 GiB data, 10 GiB used, 526 GiB / 536 GiB avail; 1023 B/s rd, 138 MiB/s wr, 2.21k op/s 2024-03-11T21:45:25.368 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:45:24 smithi092 ceph-mon[29572]: osdmap e454: 8 total, 8 up, 6 in 2024-03-11T21:45:25.368 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:45:24 smithi092 ceph-mon[29572]: 4.12 scrub starts 2024-03-11T21:45:25.368 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:45:24 smithi092 ceph-mon[29572]: 4.12 scrub ok 2024-03-11T21:45:25.368 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:45:24 smithi092 ceph-mon[29572]: 2.16 scrub starts 2024-03-11T21:45:25.368 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:45:24 smithi092 ceph-mon[29572]: 2.16 scrub ok 2024-03-11T21:45:25.368 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:45:24 smithi092 ceph-mon[29572]: 2.1c scrub starts 2024-03-11T21:45:25.368 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:45:24 smithi092 ceph-mon[29572]: 2.1c scrub ok 2024-03-11T21:45:25.369 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:45:24 smithi092 ceph-mon[37563]: pgmap v940: 148 pgs: 148 active+clean; 2.2 GiB data, 10 GiB used, 526 GiB / 536 GiB avail; 1023 B/s rd, 138 MiB/s wr, 2.21k op/s 2024-03-11T21:45:25.369 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:45:24 smithi092 ceph-mon[37563]: osdmap e454: 8 total, 8 up, 6 in 2024-03-11T21:45:25.369 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:45:24 smithi092 ceph-mon[37563]: 4.12 scrub starts 2024-03-11T21:45:25.369 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:45:24 smithi092 ceph-mon[37563]: 4.12 scrub ok 2024-03-11T21:45:25.369 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:45:24 smithi092 ceph-mon[37563]: 2.16 scrub starts 2024-03-11T21:45:25.369 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:45:24 smithi092 ceph-mon[37563]: 2.16 scrub ok 2024-03-11T21:45:25.369 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:45:24 smithi092 ceph-mon[37563]: 2.1c scrub starts 2024-03-11T21:45:25.369 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:45:24 smithi092 ceph-mon[37563]: 2.1c scrub ok 2024-03-11T21:45:25.402 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:45:25.402 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:45:25.403 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:45:25.483 DEBUG:teuthology.orchestra.run.smithi092:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid 00db0016-dfed-11ee-95c8-87774f69a715 -- ceph daemon osd.2 dump_historic_ops 2024-03-11T21:45:25.503 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:45:25.535 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:45:25.535 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:45:25.535 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:45:25.636 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:45:25.670 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:45:25.671 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:45:25.671 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:45:25.772 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:45:25.807 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:45:25.808 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:45:25.808 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:45:25.909 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:45:25.946 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:45:25.947 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:45:25.947 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:45:26.048 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:45:26.073 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:45:26.074 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:45:26.075 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:45:26.175 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:45:26.209 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:45:26.210 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:45:26.210 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:45:26.258 INFO:tasks.radosbench.radosbench.0.smithi092.stdout: 19 16 43540 43524 143.138 103.562 0.006556 0.00698123 2024-03-11T21:45:26.311 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:45:26.315 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:45:25 smithi092 ceph-mon[29572]: 4.18 scrub starts 2024-03-11T21:45:26.315 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:45:25 smithi092 ceph-mon[29572]: 4.18 scrub ok 2024-03-11T21:45:26.315 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:45:25 smithi092 ceph-mon[29572]: 4.16 scrub starts 2024-03-11T21:45:26.315 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:45:25 smithi092 ceph-mon[29572]: 4.16 scrub ok 2024-03-11T21:45:26.315 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:45:25 smithi092 ceph-mon[29572]: 2.8 scrub starts 2024-03-11T21:45:26.315 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:45:25 smithi092 ceph-mon[29572]: 2.8 scrub ok 2024-03-11T21:45:26.315 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:45:25 smithi092 ceph-mon[37563]: 4.18 scrub starts 2024-03-11T21:45:26.315 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:45:25 smithi092 ceph-mon[37563]: 4.18 scrub ok 2024-03-11T21:45:26.316 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:45:25 smithi092 ceph-mon[37563]: 4.16 scrub starts 2024-03-11T21:45:26.316 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:45:25 smithi092 ceph-mon[37563]: 4.16 scrub ok 2024-03-11T21:45:26.316 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:45:25 smithi092 ceph-mon[37563]: 2.8 scrub starts 2024-03-11T21:45:26.316 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:45:25 smithi092 ceph-mon[37563]: 2.8 scrub ok 2024-03-11T21:45:26.333 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:45:25 smithi097 ceph-mon[33317]: 4.18 scrub starts 2024-03-11T21:45:26.333 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:45:25 smithi097 ceph-mon[33317]: 4.18 scrub ok 2024-03-11T21:45:26.333 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:45:25 smithi097 ceph-mon[33317]: 4.16 scrub starts 2024-03-11T21:45:26.333 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:45:25 smithi097 ceph-mon[33317]: 4.16 scrub ok 2024-03-11T21:45:26.333 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:45:25 smithi097 ceph-mon[33317]: 2.8 scrub starts 2024-03-11T21:45:26.334 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:45:25 smithi097 ceph-mon[33317]: 2.8 scrub ok 2024-03-11T21:45:26.344 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:45:26.345 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:45:26.345 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:45:26.446 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:45:26.485 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:45:26.486 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:45:26.486 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:45:26.587 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:45:26.612 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:45:26.613 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:45:26.613 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:45:26.714 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:45:26.754 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:45:26.755 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:45:26.755 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:45:26.856 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:45:26.888 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:45:26.889 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:45:26.890 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:45:26.990 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:45:27.031 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:45:27.032 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:45:27.032 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:45:27.133 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:45:27.163 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:45:27.163 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:45:27.164 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:45:27.258 INFO:tasks.radosbench.radosbench.0.smithi092.stdout:2024-03-11T21:45:27.269260+0000 min lat: 0.00128848 max lat: 0.229714 avg lat: 0.00705933 2024-03-11T21:45:27.258 INFO:tasks.radosbench.radosbench.0.smithi092.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-03-11T21:45:27.258 INFO:tasks.radosbench.radosbench.0.smithi092.stdout: 20 16 45321 45305 141.547 111.312 0.00438632 0.00705933 2024-03-11T21:45:27.264 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:45:27.288 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:45:27.289 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:45:27.289 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:45:27.333 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:45:26 smithi097 ceph-mon[33317]: pgmap v942: 148 pgs: 148 active+clean; 2.3 GiB data, 11 GiB used, 525 GiB / 536 GiB avail; 1.2 KiB/s rd, 153 MiB/s wr, 2.45k op/s 2024-03-11T21:45:27.333 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:45:26 smithi097 ceph-mon[33317]: 4.1c scrub starts 2024-03-11T21:45:27.333 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:45:26 smithi097 ceph-mon[33317]: 4.1c scrub ok 2024-03-11T21:45:27.333 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:45:26 smithi097 ceph-mon[33317]: 3.0 scrub starts 2024-03-11T21:45:27.333 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:45:26 smithi097 ceph-mon[33317]: 3.0 scrub ok 2024-03-11T21:45:27.334 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:45:26 smithi097 ceph-mon[33317]: 6.5 scrub starts 2024-03-11T21:45:27.334 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:45:26 smithi097 ceph-mon[33317]: 6.5 scrub ok 2024-03-11T21:45:27.367 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:45:26 smithi092 ceph-mon[29572]: pgmap v942: 148 pgs: 148 active+clean; 2.3 GiB data, 11 GiB used, 525 GiB / 536 GiB avail; 1.2 KiB/s rd, 153 MiB/s wr, 2.45k op/s 2024-03-11T21:45:27.367 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:45:26 smithi092 ceph-mon[29572]: 4.1c scrub starts 2024-03-11T21:45:27.367 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:45:26 smithi092 ceph-mon[29572]: 4.1c scrub ok 2024-03-11T21:45:27.368 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:45:26 smithi092 ceph-mon[29572]: 3.0 scrub starts 2024-03-11T21:45:27.368 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:45:26 smithi092 ceph-mon[29572]: 3.0 scrub ok 2024-03-11T21:45:27.368 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:45:26 smithi092 ceph-mon[29572]: 6.5 scrub starts 2024-03-11T21:45:27.368 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:45:26 smithi092 ceph-mon[29572]: 6.5 scrub ok 2024-03-11T21:45:27.368 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:45:26 smithi092 ceph-mon[37563]: pgmap v942: 148 pgs: 148 active+clean; 2.3 GiB data, 11 GiB used, 525 GiB / 536 GiB avail; 1.2 KiB/s rd, 153 MiB/s wr, 2.45k op/s 2024-03-11T21:45:27.368 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:45:26 smithi092 ceph-mon[37563]: 4.1c scrub starts 2024-03-11T21:45:27.368 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:45:26 smithi092 ceph-mon[37563]: 4.1c scrub ok 2024-03-11T21:45:27.368 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:45:26 smithi092 ceph-mon[37563]: 3.0 scrub starts 2024-03-11T21:45:27.368 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:45:26 smithi092 ceph-mon[37563]: 3.0 scrub ok 2024-03-11T21:45:27.368 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:45:26 smithi092 ceph-mon[37563]: 6.5 scrub starts 2024-03-11T21:45:27.368 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:45:26 smithi092 ceph-mon[37563]: 6.5 scrub ok 2024-03-11T21:45:27.391 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:45:27.426 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:45:27.426 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:45:27.427 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:45:27.527 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:45:27.568 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:45:27.569 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:45:27.569 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:45:27.670 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:45:27.701 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:45:27.701 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:45:27.702 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:45:27.802 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:45:27.837 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:45:27.837 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:45:27.838 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:45:27.909 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:45:27 smithi092 ceph-mon[37563]: 5.1c scrub starts 2024-03-11T21:45:27.909 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:45:27 smithi092 ceph-mon[37563]: 5.1c scrub ok 2024-03-11T21:45:27.909 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:45:27 smithi092 ceph-mon[37563]: 5.9 scrub starts 2024-03-11T21:45:27.909 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:45:27 smithi092 ceph-mon[37563]: 5.9 scrub ok 2024-03-11T21:45:27.910 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:45:27 smithi092 ceph-mon[29572]: 5.1c scrub starts 2024-03-11T21:45:27.910 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:45:27 smithi092 ceph-mon[29572]: 5.1c scrub ok 2024-03-11T21:45:27.910 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:45:27 smithi092 ceph-mon[29572]: 5.9 scrub starts 2024-03-11T21:45:27.910 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:45:27 smithi092 ceph-mon[29572]: 5.9 scrub ok 2024-03-11T21:45:27.939 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:45:27.971 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:45:27.971 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:45:27.972 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:45:28.073 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:45:28.126 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:45:28.127 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:45:28.127 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:45:28.228 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:45:28.258 INFO:tasks.radosbench.radosbench.0.smithi092.stdout: 21 16 46968 46952 139.704 102.938 0.00601041 0.00715328 2024-03-11T21:45:28.269 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:45:28.269 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:45:28.270 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:45:28.287 INFO:journalctl@ceph.mgr.y.smithi092.stdout:Mar 11 21:45:27 smithi092 ceph-00db0016-dfed-11ee-95c8-87774f69a715-mgr-y[29801]: ::ffff:172.21.15.97 - - [11/Mar/2024:21:45:27] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-11T21:45:28.333 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:45:27 smithi097 ceph-mon[33317]: 5.1c scrub starts 2024-03-11T21:45:28.333 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:45:27 smithi097 ceph-mon[33317]: 5.1c scrub ok 2024-03-11T21:45:28.333 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:45:27 smithi097 ceph-mon[33317]: 5.9 scrub starts 2024-03-11T21:45:28.334 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:45:27 smithi097 ceph-mon[33317]: 5.9 scrub ok 2024-03-11T21:45:28.371 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:45:28.415 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:45:28.416 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:45:28.416 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:45:28.517 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:45:28.548 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:45:28.548 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:45:28.549 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:45:28.650 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:45:28.685 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:45:28.686 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:45:28.687 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:45:28.788 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:45:28.818 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:45:28.818 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:45:28.819 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:45:28.919 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:45:28.972 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:45:28.973 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:45:28.973 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:45:29.074 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:45:29.111 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:45:29.112 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:45:29.112 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:45:29.213 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:45:29.251 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:45:29.252 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:45:29.253 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:45:29.258 INFO:tasks.radosbench.radosbench.0.smithi092.stdout: 22 16 48524 48508 137.774 97.25 0.00817211 0.00725229 2024-03-11T21:45:29.334 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:45:28 smithi097 ceph-mon[33317]: pgmap v943: 148 pgs: 148 active+clean; 2.4 GiB data, 11 GiB used, 525 GiB / 536 GiB avail; 639 B/s rd, 78 MiB/s wr, 1.25k op/s 2024-03-11T21:45:29.335 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:45:28 smithi097 ceph-mon[33317]: 2.10 scrub starts 2024-03-11T21:45:29.335 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:45:28 smithi097 ceph-mon[33317]: 2.10 scrub ok 2024-03-11T21:45:29.335 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:45:28 smithi097 ceph-mon[33317]: 2.4 scrub starts 2024-03-11T21:45:29.335 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:45:28 smithi097 ceph-mon[33317]: 2.4 scrub ok 2024-03-11T21:45:29.335 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:45:28 smithi097 ceph-mon[33317]: 2.2 scrub starts 2024-03-11T21:45:29.335 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:45:28 smithi097 ceph-mon[33317]: 2.2 scrub ok 2024-03-11T21:45:29.353 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:45:29.367 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:45:28 smithi092 ceph-mon[29572]: pgmap v943: 148 pgs: 148 active+clean; 2.4 GiB data, 11 GiB used, 525 GiB / 536 GiB avail; 639 B/s rd, 78 MiB/s wr, 1.25k op/s 2024-03-11T21:45:29.367 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:45:28 smithi092 ceph-mon[29572]: 2.10 scrub starts 2024-03-11T21:45:29.367 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:45:28 smithi092 ceph-mon[29572]: 2.10 scrub ok 2024-03-11T21:45:29.368 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:45:28 smithi092 ceph-mon[29572]: 2.4 scrub starts 2024-03-11T21:45:29.368 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:45:28 smithi092 ceph-mon[29572]: 2.4 scrub ok 2024-03-11T21:45:29.368 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:45:28 smithi092 ceph-mon[29572]: 2.2 scrub starts 2024-03-11T21:45:29.368 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:45:28 smithi092 ceph-mon[29572]: 2.2 scrub ok 2024-03-11T21:45:29.368 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:45:28 smithi092 ceph-mon[37563]: pgmap v943: 148 pgs: 148 active+clean; 2.4 GiB data, 11 GiB used, 525 GiB / 536 GiB avail; 639 B/s rd, 78 MiB/s wr, 1.25k op/s 2024-03-11T21:45:29.368 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:45:28 smithi092 ceph-mon[37563]: 2.10 scrub starts 2024-03-11T21:45:29.368 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:45:28 smithi092 ceph-mon[37563]: 2.10 scrub ok 2024-03-11T21:45:29.368 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:45:28 smithi092 ceph-mon[37563]: 2.4 scrub starts 2024-03-11T21:45:29.368 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:45:28 smithi092 ceph-mon[37563]: 2.4 scrub ok 2024-03-11T21:45:29.368 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:45:28 smithi092 ceph-mon[37563]: 2.2 scrub starts 2024-03-11T21:45:29.368 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:45:28 smithi092 ceph-mon[37563]: 2.2 scrub ok 2024-03-11T21:45:29.387 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:45:29.388 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:45:29.388 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:45:29.489 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:45:29.520 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:45:29.520 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:45:29.521 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:45:29.581 INFO:teuthology.orchestra.run.smithi092.stderr:Inferring config /var/lib/ceph/00db0016-dfed-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T21:45:29.621 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:45:29.652 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:45:29.652 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:45:29.653 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:45:29.658 INFO:teuthology.orchestra.run.smithi092.stderr:Inferring config /var/lib/ceph/00db0016-dfed-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T21:45:29.754 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:45:29.784 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:45:29.785 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:45:29.785 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:45:29.886 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:45:29.922 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:45:29.922 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:45:29.923 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:45:29.926 INFO:teuthology.orchestra.run.smithi092.stderr:nodeep-scrub is unset 2024-03-11T21:45:30.023 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:45:30.057 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:45:29 smithi092 ceph-mon[37563]: from='client.? v1:172.21.15.92:0/3495763053' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-11T21:45:30.057 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:45:29 smithi092 ceph-mon[37563]: 2.1b scrub starts 2024-03-11T21:45:30.057 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:45:29 smithi092 ceph-mon[37563]: 2.1b scrub ok 2024-03-11T21:45:30.058 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:45:29 smithi092 ceph-mon[29572]: from='client.? v1:172.21.15.92:0/3495763053' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-11T21:45:30.059 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:45:29 smithi092 ceph-mon[29572]: 2.1b scrub starts 2024-03-11T21:45:30.059 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:45:29 smithi092 ceph-mon[29572]: 2.1b scrub ok 2024-03-11T21:45:30.060 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:45:30.061 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:45:30.061 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:45:30.162 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:45:30.197 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:45:30.198 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:45:30.198 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:45:30.258 INFO:tasks.radosbench.radosbench.0.smithi092.stdout: 23 16 50182 50166 136.289 103.625 0.00751747 0.00733222 2024-03-11T21:45:30.299 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:45:30.333 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:45:29 smithi097 ceph-mon[33317]: from='client.? v1:172.21.15.92:0/3495763053' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-11T21:45:30.333 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:45:29 smithi097 ceph-mon[33317]: 2.1b scrub starts 2024-03-11T21:45:30.333 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:45:29 smithi097 ceph-mon[33317]: 2.1b scrub ok 2024-03-11T21:45:30.334 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:45:30.335 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:45:30.335 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:45:30.436 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:45:30.467 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:45:30.468 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:45:30.468 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:45:30.569 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:45:30.601 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:45:30.602 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:45:30.602 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:45:30.703 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:45:30.734 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:45:30.735 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:45:30.735 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:45:30.836 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:45:30.866 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:45:30.867 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:45:30.867 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:45:30.968 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:45:31.002 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:45:31.002 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:45:31.002 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:45:31.015 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:45:30 smithi092 ceph-mon[29572]: pgmap v944: 148 pgs: 148 active+clean; 2.8 GiB data, 12 GiB used, 524 GiB / 536 GiB avail; 1.2 KiB/s rd, 131 MiB/s wr, 2.09k op/s 2024-03-11T21:45:31.015 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:45:30 smithi092 ceph-mon[29572]: 2.7 scrub starts 2024-03-11T21:45:31.015 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:45:30 smithi092 ceph-mon[29572]: 2.7 scrub ok 2024-03-11T21:45:31.015 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:45:30 smithi092 ceph-mon[29572]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-03-11T21:45:31.015 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:45:30 smithi092 ceph-mon[29572]: Cluster is now healthy 2024-03-11T21:45:31.015 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:45:30 smithi092 ceph-mon[29572]: from='client.? v1:172.21.15.92:0/3495763053' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-03-11T21:45:31.015 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:45:30 smithi092 ceph-mon[29572]: osdmap e455: 8 total, 8 up, 6 in 2024-03-11T21:45:31.015 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:45:30 smithi092 ceph-mon[29572]: 2.b deep-scrub starts 2024-03-11T21:45:31.015 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:45:30 smithi092 ceph-mon[29572]: 2.b deep-scrub ok 2024-03-11T21:45:31.015 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:45:30 smithi092 ceph-mon[29572]: 4.b scrub starts 2024-03-11T21:45:31.016 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:45:30 smithi092 ceph-mon[29572]: 4.b scrub ok 2024-03-11T21:45:31.016 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:45:30 smithi092 ceph-mon[37563]: pgmap v944: 148 pgs: 148 active+clean; 2.8 GiB data, 12 GiB used, 524 GiB / 536 GiB avail; 1.2 KiB/s rd, 131 MiB/s wr, 2.09k op/s 2024-03-11T21:45:31.016 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:45:30 smithi092 ceph-mon[37563]: 2.7 scrub starts 2024-03-11T21:45:31.016 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:45:30 smithi092 ceph-mon[37563]: 2.7 scrub ok 2024-03-11T21:45:31.016 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:45:30 smithi092 ceph-mon[37563]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-03-11T21:45:31.016 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:45:30 smithi092 ceph-mon[37563]: Cluster is now healthy 2024-03-11T21:45:31.016 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:45:30 smithi092 ceph-mon[37563]: from='client.? v1:172.21.15.92:0/3495763053' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-03-11T21:45:31.016 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:45:30 smithi092 ceph-mon[37563]: osdmap e455: 8 total, 8 up, 6 in 2024-03-11T21:45:31.016 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:45:30 smithi092 ceph-mon[37563]: 2.b deep-scrub starts 2024-03-11T21:45:31.016 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:45:30 smithi092 ceph-mon[37563]: 2.b deep-scrub ok 2024-03-11T21:45:31.016 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:45:30 smithi092 ceph-mon[37563]: 4.b scrub starts 2024-03-11T21:45:31.016 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:45:30 smithi092 ceph-mon[37563]: 4.b scrub ok 2024-03-11T21:45:31.103 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:45:31.136 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:45:31.137 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:45:31.137 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:45:31.238 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:45:31.263 INFO:tasks.radosbench.radosbench.0.smithi092.stdout: 24 16 51889 51873 135.055 106.688 0.00814157 0.00739575 2024-03-11T21:45:31.283 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:45:31.283 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:45:31.283 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:45:31.333 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:45:30 smithi097 ceph-mon[33317]: pgmap v944: 148 pgs: 148 active+clean; 2.8 GiB data, 12 GiB used, 524 GiB / 536 GiB avail; 1.2 KiB/s rd, 131 MiB/s wr, 2.09k op/s 2024-03-11T21:45:31.333 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:45:30 smithi097 ceph-mon[33317]: 2.7 scrub starts 2024-03-11T21:45:31.333 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:45:30 smithi097 ceph-mon[33317]: 2.7 scrub ok 2024-03-11T21:45:31.334 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:45:30 smithi097 ceph-mon[33317]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-03-11T21:45:31.334 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:45:30 smithi097 ceph-mon[33317]: Cluster is now healthy 2024-03-11T21:45:31.334 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:45:30 smithi097 ceph-mon[33317]: from='client.? v1:172.21.15.92:0/3495763053' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-03-11T21:45:31.334 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:45:30 smithi097 ceph-mon[33317]: osdmap e455: 8 total, 8 up, 6 in 2024-03-11T21:45:31.334 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:45:30 smithi097 ceph-mon[33317]: 2.b deep-scrub starts 2024-03-11T21:45:31.334 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:45:30 smithi097 ceph-mon[33317]: 2.b deep-scrub ok 2024-03-11T21:45:31.334 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:45:30 smithi097 ceph-mon[33317]: 4.b scrub starts 2024-03-11T21:45:31.334 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:45:30 smithi097 ceph-mon[33317]: 4.b scrub ok 2024-03-11T21:45:31.384 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:45:31.415 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:45:31.416 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:45:31.416 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:45:31.517 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:45:31.554 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:45:31.555 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:45:31.555 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:45:31.656 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:45:31.698 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:45:31.698 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:45:31.699 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:45:31.800 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:45:31.834 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:45:31.835 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:45:31.835 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:45:31.936 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:45:31.972 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:45:31.973 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:45:31.974 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:45:32.075 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:45:32.110 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:45:32.110 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:45:32.111 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:45:32.117 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:45:31 smithi092 ceph-mon[29572]: 2.17 deep-scrub starts 2024-03-11T21:45:32.118 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:45:31 smithi092 ceph-mon[29572]: 2.17 deep-scrub ok 2024-03-11T21:45:32.118 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:45:31 smithi092 ceph-mon[37563]: 2.17 deep-scrub starts 2024-03-11T21:45:32.118 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:45:31 smithi092 ceph-mon[37563]: 2.17 deep-scrub ok 2024-03-11T21:45:32.211 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:45:32.251 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:45:32.251 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:45:32.252 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:45:32.259 INFO:tasks.radosbench.radosbench.0.smithi092.stdout: 25 16 53577 53561 133.873 105.5 0.0120527 0.00746503 2024-03-11T21:45:32.333 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:45:31 smithi097 ceph-mon[33317]: 2.17 deep-scrub starts 2024-03-11T21:45:32.333 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:45:31 smithi097 ceph-mon[33317]: 2.17 deep-scrub ok 2024-03-11T21:45:32.353 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:45:32.393 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:45:32.393 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:45:32.394 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:45:32.494 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:45:32.545 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:45:32.546 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:45:32.546 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:45:32.647 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:45:32.690 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:45:32.690 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:45:32.691 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:45:32.792 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:45:32.833 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:45:32.833 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:45:32.834 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:45:32.934 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:45:32.952 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:45:32 smithi092 ceph-mon[37563]: pgmap v946: 148 pgs: 148 active+clean; 2.8 GiB data, 13 GiB used, 524 GiB / 536 GiB avail; 1.2 KiB/s rd, 86 MiB/s wr, 1.38k op/s 2024-03-11T21:45:32.952 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:45:32 smithi092 ceph-mon[37563]: 2.1a scrub starts 2024-03-11T21:45:32.952 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:45:32 smithi092 ceph-mon[37563]: 2.1a scrub ok 2024-03-11T21:45:32.952 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:45:32 smithi092 ceph-mon[37563]: 6.18 scrub starts 2024-03-11T21:45:32.952 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:45:32 smithi092 ceph-mon[37563]: 6.18 scrub ok 2024-03-11T21:45:32.953 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:45:32 smithi092 ceph-mon[37563]: from='client.? v1:172.21.15.92:0/709554261' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-11T21:45:32.956 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:45:32 smithi092 ceph-mon[29572]: pgmap v946: 148 pgs: 148 active+clean; 2.8 GiB data, 13 GiB used, 524 GiB / 536 GiB avail; 1.2 KiB/s rd, 86 MiB/s wr, 1.38k op/s 2024-03-11T21:45:32.956 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:45:32 smithi092 ceph-mon[29572]: 2.1a scrub starts 2024-03-11T21:45:32.956 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:45:32 smithi092 ceph-mon[29572]: 2.1a scrub ok 2024-03-11T21:45:32.956 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:45:32 smithi092 ceph-mon[29572]: 6.18 scrub starts 2024-03-11T21:45:32.957 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:45:32 smithi092 ceph-mon[29572]: 6.18 scrub ok 2024-03-11T21:45:32.957 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:45:32 smithi092 ceph-mon[29572]: from='client.? v1:172.21.15.92:0/709554261' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-11T21:45:32.977 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:45:32.978 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:45:32.978 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:45:33.073 INFO:teuthology.orchestra.run.smithi092.stdout:osd.0: {} 2024-03-11T21:45:33.074 INFO:teuthology.orchestra.run.smithi092.stdout:osd.1: {} 2024-03-11T21:45:33.074 INFO:teuthology.orchestra.run.smithi092.stdout:osd.2: {} 2024-03-11T21:45:33.074 INFO:teuthology.orchestra.run.smithi092.stdout:osd.3: {} 2024-03-11T21:45:33.074 INFO:teuthology.orchestra.run.smithi092.stdout:osd.4: {} 2024-03-11T21:45:33.074 INFO:teuthology.orchestra.run.smithi092.stdout:osd.5: {} 2024-03-11T21:45:33.074 INFO:teuthology.orchestra.run.smithi092.stdout:osd.6: {} 2024-03-11T21:45:33.074 INFO:teuthology.orchestra.run.smithi092.stdout:osd.7: {} 2024-03-11T21:45:33.074 INFO:teuthology.orchestra.run.smithi092.stderr:osd.0: osd_enable_op_tracker = '' 2024-03-11T21:45:33.074 INFO:teuthology.orchestra.run.smithi092.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-11T21:45:33.074 INFO:teuthology.orchestra.run.smithi092.stderr:osd.2: osd_enable_op_tracker = '' 2024-03-11T21:45:33.075 INFO:teuthology.orchestra.run.smithi092.stderr:osd.3: osd_enable_op_tracker = '' 2024-03-11T21:45:33.075 INFO:teuthology.orchestra.run.smithi092.stderr:osd.4: osd_enable_op_tracker = '' 2024-03-11T21:45:33.075 INFO:teuthology.orchestra.run.smithi092.stderr:osd.5: osd_enable_op_tracker = '' 2024-03-11T21:45:33.075 INFO:teuthology.orchestra.run.smithi092.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-11T21:45:33.075 INFO:teuthology.orchestra.run.smithi092.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-11T21:45:33.079 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:45:33.112 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:45:33.112 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:45:33.113 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:45:33.213 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:45:33.252 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:45:33.253 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:45:33.253 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:45:33.259 INFO:tasks.radosbench.radosbench.0.smithi092.stdout: 26 16 55129 55113 132.455 97 0.0048976 0.00754467 2024-03-11T21:45:33.333 INFO:journalctl@ceph.iscsi.iscsi.a.smithi097.stdout:Mar 11 21:45:33 smithi097 ceph-00db0016-dfed-11ee-95c8-87774f69a715-iscsi-iscsi-a[60943]: debug there is no tcmu-runner data available 2024-03-11T21:45:33.334 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:45:32 smithi097 ceph-mon[33317]: pgmap v946: 148 pgs: 148 active+clean; 2.8 GiB data, 13 GiB used, 524 GiB / 536 GiB avail; 1.2 KiB/s rd, 86 MiB/s wr, 1.38k op/s 2024-03-11T21:45:33.334 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:45:32 smithi097 ceph-mon[33317]: 2.1a scrub starts 2024-03-11T21:45:33.334 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:45:32 smithi097 ceph-mon[33317]: 2.1a scrub ok 2024-03-11T21:45:33.334 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:45:32 smithi097 ceph-mon[33317]: 6.18 scrub starts 2024-03-11T21:45:33.334 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:45:32 smithi097 ceph-mon[33317]: 6.18 scrub ok 2024-03-11T21:45:33.334 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:45:32 smithi097 ceph-mon[33317]: from='client.? v1:172.21.15.92:0/709554261' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-11T21:45:33.354 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:45:33.392 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:45:33.392 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:45:33.393 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:45:33.493 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:45:33.527 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:45:33.528 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:45:33.528 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:45:33.629 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:45:33.663 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:45:33.663 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:45:33.664 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:45:33.765 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:45:33.800 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:45:33.800 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:45:33.801 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:45:33.901 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:45:33.944 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:45:33.945 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:45:33.945 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:45:34.046 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:45:34.062 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:45:33 smithi092 ceph-mon[29572]: 4.a deep-scrub starts 2024-03-11T21:45:34.062 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:45:33 smithi092 ceph-mon[29572]: 4.a deep-scrub ok 2024-03-11T21:45:34.062 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:45:33 smithi092 ceph-mon[29572]: from='client.24469 v1:172.21.15.97:0/404208310' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:45:34.063 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:45:33 smithi092 ceph-mon[37563]: 4.a deep-scrub starts 2024-03-11T21:45:34.063 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:45:33 smithi092 ceph-mon[37563]: 4.a deep-scrub ok 2024-03-11T21:45:34.063 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:45:33 smithi092 ceph-mon[37563]: from='client.24469 v1:172.21.15.97:0/404208310' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:45:34.080 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:45:34.081 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:45:34.081 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:45:34.182 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:45:34.225 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:45:34.226 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:45:34.226 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:45:34.259 INFO:tasks.radosbench.radosbench.0.smithi092.stdout: 27 16 56707 56691 131.201 98.625 0.00487166 0.00761681 2024-03-11T21:45:34.327 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:45:34.333 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:45:33 smithi097 ceph-mon[33317]: 4.a deep-scrub starts 2024-03-11T21:45:34.333 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:45:33 smithi097 ceph-mon[33317]: 4.a deep-scrub ok 2024-03-11T21:45:34.333 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:45:33 smithi097 ceph-mon[33317]: from='client.24469 v1:172.21.15.97:0/404208310' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:45:34.365 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:45:34.365 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:45:34.366 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:45:34.467 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:45:34.499 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:45:34.500 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:45:34.500 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:45:34.601 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:45:34.644 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:45:34.645 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:45:34.645 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:45:34.746 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:45:34.754 INFO:teuthology.orchestra.run.smithi092.stdout:ERROR: (22) Invalid argument 2024-03-11T21:45:34.754 INFO:teuthology.orchestra.run.smithi092.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-11T21:45:34.785 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:45:34.786 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:45:34.786 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:45:34.887 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:45:34.918 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:45:34.918 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:45:34.919 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:45:34.962 DEBUG:teuthology.orchestra.run.smithi092:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid 00db0016-dfed-11ee-95c8-87774f69a715 -- ceph osd set noscrub 2024-03-11T21:45:35.019 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:45:35.049 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:45:35.049 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:45:35.050 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:45:35.150 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:45:35.189 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:45:35.190 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:45:35.190 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:45:35.259 INFO:tasks.radosbench.radosbench.0.smithi092.stdout: 28 16 58159 58143 129.757 90.75 0.0190804 0.00770173 2024-03-11T21:45:35.291 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:45:35.295 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:45:34 smithi092 ceph-mon[37563]: pgmap v947: 148 pgs: 148 active+clean; 3.2 GiB data, 14 GiB used, 523 GiB / 536 GiB avail; 535 B/s rd, 96 MiB/s wr, 1.53k op/s 2024-03-11T21:45:35.295 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:45:34 smithi092 ceph-mon[37563]: 5.8 scrub starts 2024-03-11T21:45:35.295 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:45:34 smithi092 ceph-mon[37563]: 5.8 scrub ok 2024-03-11T21:45:35.295 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:45:34 smithi092 ceph-mon[37563]: 4.7 scrub starts 2024-03-11T21:45:35.295 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:45:34 smithi092 ceph-mon[37563]: 4.7 scrub ok 2024-03-11T21:45:35.295 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:45:34 smithi092 ceph-mon[37563]: 6.a scrub starts 2024-03-11T21:45:35.295 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:45:34 smithi092 ceph-mon[37563]: 6.a scrub ok 2024-03-11T21:45:35.296 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:45:34 smithi092 ceph-mon[37563]: 2.1f scrub starts 2024-03-11T21:45:35.296 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:45:34 smithi092 ceph-mon[37563]: 2.1f scrub ok 2024-03-11T21:45:35.296 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:45:34 smithi092 ceph-mon[29572]: pgmap v947: 148 pgs: 148 active+clean; 3.2 GiB data, 14 GiB used, 523 GiB / 536 GiB avail; 535 B/s rd, 96 MiB/s wr, 1.53k op/s 2024-03-11T21:45:35.296 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:45:34 smithi092 ceph-mon[29572]: 5.8 scrub starts 2024-03-11T21:45:35.296 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:45:34 smithi092 ceph-mon[29572]: 5.8 scrub ok 2024-03-11T21:45:35.296 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:45:34 smithi092 ceph-mon[29572]: 4.7 scrub starts 2024-03-11T21:45:35.296 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:45:34 smithi092 ceph-mon[29572]: 4.7 scrub ok 2024-03-11T21:45:35.296 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:45:34 smithi092 ceph-mon[29572]: 6.a scrub starts 2024-03-11T21:45:35.296 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:45:34 smithi092 ceph-mon[29572]: 6.a scrub ok 2024-03-11T21:45:35.297 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:45:34 smithi092 ceph-mon[29572]: 2.1f scrub starts 2024-03-11T21:45:35.297 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:45:34 smithi092 ceph-mon[29572]: 2.1f scrub ok 2024-03-11T21:45:35.322 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:45:35.322 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:45:35.323 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:45:35.333 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:45:34 smithi097 ceph-mon[33317]: pgmap v947: 148 pgs: 148 active+clean; 3.2 GiB data, 14 GiB used, 523 GiB / 536 GiB avail; 535 B/s rd, 96 MiB/s wr, 1.53k op/s 2024-03-11T21:45:35.333 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:45:34 smithi097 ceph-mon[33317]: 5.8 scrub starts 2024-03-11T21:45:35.333 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:45:34 smithi097 ceph-mon[33317]: 5.8 scrub ok 2024-03-11T21:45:35.333 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:45:34 smithi097 ceph-mon[33317]: 4.7 scrub starts 2024-03-11T21:45:35.333 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:45:34 smithi097 ceph-mon[33317]: 4.7 scrub ok 2024-03-11T21:45:35.334 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:45:34 smithi097 ceph-mon[33317]: 6.a scrub starts 2024-03-11T21:45:35.334 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:45:34 smithi097 ceph-mon[33317]: 6.a scrub ok 2024-03-11T21:45:35.334 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:45:34 smithi097 ceph-mon[33317]: 2.1f scrub starts 2024-03-11T21:45:35.334 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:45:34 smithi097 ceph-mon[33317]: 2.1f scrub ok 2024-03-11T21:45:35.425 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:45:35.458 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:45:35.459 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:45:35.459 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:45:35.560 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:45:35.615 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:45:35.615 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:45:35.616 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:45:35.717 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:45:35.749 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:45:35.750 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:45:35.750 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:45:35.851 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:45:35.889 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:45:35.890 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:45:35.890 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:45:35.991 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:45:36.028 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:45:36.029 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:45:36.029 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:45:36.102 DEBUG:teuthology.orchestra.run.smithi092:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid 00db0016-dfed-11ee-95c8-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-03-11T21:45:36.130 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:45:36.162 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:45:36.162 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:45:36.163 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:45:36.251 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:45:35 smithi092 ceph-mon[29572]: 6.10 scrub starts 2024-03-11T21:45:36.251 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:45:35 smithi092 ceph-mon[29572]: 6.10 scrub ok 2024-03-11T21:45:36.251 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:45:35 smithi092 ceph-mon[37563]: 6.10 scrub starts 2024-03-11T21:45:36.251 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:45:35 smithi092 ceph-mon[37563]: 6.10 scrub ok 2024-03-11T21:45:36.259 INFO:tasks.radosbench.radosbench.0.smithi092.stdout: 29 16 59648 59632 128.491 93.0625 0.00754909 0.00777059 2024-03-11T21:45:36.264 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:45:36.305 DEBUG:teuthology.orchestra.run.smithi097:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid 00db0016-dfed-11ee-95c8-87774f69a715 -- ceph daemon osd.4 dump_ops_in_flight 2024-03-11T21:45:36.319 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:45:36.319 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:45:36.320 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:45:36.333 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:45:35 smithi097 ceph-mon[33317]: 6.10 scrub starts 2024-03-11T21:45:36.333 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:45:35 smithi097 ceph-mon[33317]: 6.10 scrub ok 2024-03-11T21:45:36.421 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:45:36.466 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:45:36.467 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:45:36.468 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:45:36.568 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:45:36.596 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:45:36.598 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:45:36.599 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:45:36.684 INFO:teuthology.orchestra.run.smithi097.stderr:Inferring config /var/lib/ceph/00db0016-dfed-11ee-95c8-87774f69a715/mon.b/config 2024-03-11T21:45:36.700 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:45:36.741 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:45:36.742 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:45:36.742 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:45:36.842 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:45:36.868 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:45:36.869 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:45:36.869 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:45:36.970 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:45:37.001 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:45:37.002 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:45:37.003 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:45:37.084 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:45:36 smithi097 ceph-mon[33317]: pgmap v948: 148 pgs: 148 active+clean; 3.3 GiB data, 14 GiB used, 522 GiB / 536 GiB avail; 1023 B/s rd, 104 MiB/s wr, 1.66k op/s 2024-03-11T21:45:37.084 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:45:36 smithi097 ceph-mon[33317]: 6.2 scrub starts 2024-03-11T21:45:37.084 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:45:36 smithi097 ceph-mon[33317]: 6.2 scrub ok 2024-03-11T21:45:37.084 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:45:36 smithi097 ceph-mon[33317]: 6.11 scrub starts 2024-03-11T21:45:37.084 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:45:36 smithi097 ceph-mon[33317]: 6.11 scrub ok 2024-03-11T21:45:37.103 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:45:37.138 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:45:37.138 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:45:37.138 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:45:37.239 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:45:37.259 INFO:tasks.radosbench.radosbench.0.smithi092.stdout: 30 16 61044 61028 127.116 87.25 0.00819947 0.00786228 2024-03-11T21:45:37.275 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:45:37.276 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:45:37.276 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:45:37.367 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:45:36 smithi092 ceph-mon[29572]: pgmap v948: 148 pgs: 148 active+clean; 3.3 GiB data, 14 GiB used, 522 GiB / 536 GiB avail; 1023 B/s rd, 104 MiB/s wr, 1.66k op/s 2024-03-11T21:45:37.368 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:45:36 smithi092 ceph-mon[29572]: 6.2 scrub starts 2024-03-11T21:45:37.368 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:45:36 smithi092 ceph-mon[29572]: 6.2 scrub ok 2024-03-11T21:45:37.368 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:45:36 smithi092 ceph-mon[29572]: 6.11 scrub starts 2024-03-11T21:45:37.368 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:45:36 smithi092 ceph-mon[29572]: 6.11 scrub ok 2024-03-11T21:45:37.368 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:45:36 smithi092 ceph-mon[37563]: pgmap v948: 148 pgs: 148 active+clean; 3.3 GiB data, 14 GiB used, 522 GiB / 536 GiB avail; 1023 B/s rd, 104 MiB/s wr, 1.66k op/s 2024-03-11T21:45:37.368 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:45:36 smithi092 ceph-mon[37563]: 6.2 scrub starts 2024-03-11T21:45:37.368 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:45:36 smithi092 ceph-mon[37563]: 6.2 scrub ok 2024-03-11T21:45:37.368 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:45:36 smithi092 ceph-mon[37563]: 6.11 scrub starts 2024-03-11T21:45:37.368 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:45:36 smithi092 ceph-mon[37563]: 6.11 scrub ok 2024-03-11T21:45:37.377 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:45:37.409 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:45:37.410 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:45:37.410 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:45:37.511 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:45:37.544 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:45:37.545 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:45:37.545 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:45:37.646 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:45:37.682 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:45:37.682 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:45:37.682 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:45:37.783 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:45:37.820 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:45:37.821 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:45:37.821 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:45:37.922 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:45:37.957 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:45:37.957 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:45:37.958 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:45:37.989 DEBUG:teuthology.orchestra.run.smithi092:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid 00db0016-dfed-11ee-95c8-87774f69a715 -- ceph tell osd.6 reset_purged_snaps_last 2024-03-11T21:45:37.991 INFO:teuthology.orchestra.run.smithi092.stderr:Inferring config /var/lib/ceph/00db0016-dfed-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T21:45:38.059 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:45:38.111 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:45:38.111 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:45:38.111 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:45:38.207 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:45:37 smithi092 ceph-mon[29572]: 5.17 deep-scrub starts 2024-03-11T21:45:38.207 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:45:37 smithi092 ceph-mon[29572]: 5.17 deep-scrub ok 2024-03-11T21:45:38.207 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:45:37 smithi092 ceph-mon[29572]: 5.5 scrub starts 2024-03-11T21:45:38.208 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:45:37 smithi092 ceph-mon[29572]: 5.5 scrub ok 2024-03-11T21:45:38.208 INFO:journalctl@ceph.mgr.y.smithi092.stdout:Mar 11 21:45:37 smithi092 ceph-00db0016-dfed-11ee-95c8-87774f69a715-mgr-y[29801]: ::ffff:172.21.15.97 - - [11/Mar/2024:21:45:37] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-11T21:45:38.208 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:45:37 smithi092 ceph-mon[37563]: 5.17 deep-scrub starts 2024-03-11T21:45:38.208 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:45:37 smithi092 ceph-mon[37563]: 5.17 deep-scrub ok 2024-03-11T21:45:38.208 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:45:37 smithi092 ceph-mon[37563]: 5.5 scrub starts 2024-03-11T21:45:38.208 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:45:37 smithi092 ceph-mon[37563]: 5.5 scrub ok 2024-03-11T21:45:38.212 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:45:38.239 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:45:37 smithi097 ceph-mon[33317]: 5.17 deep-scrub starts 2024-03-11T21:45:38.239 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:45:37 smithi097 ceph-mon[33317]: 5.17 deep-scrub ok 2024-03-11T21:45:38.239 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:45:37 smithi097 ceph-mon[33317]: 5.5 scrub starts 2024-03-11T21:45:38.239 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:45:37 smithi097 ceph-mon[33317]: 5.5 scrub ok 2024-03-11T21:45:38.259 INFO:tasks.radosbench.radosbench.0.smithi092.stdout: 31 16 62509 62493 125.968 91.5625 0.0194005 0.00793301 2024-03-11T21:45:38.267 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:45:38.268 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:45:38.268 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:45:38.369 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:45:38.405 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:45:38.405 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:45:38.406 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:45:38.507 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:45:38.552 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:45:38.553 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:45:38.553 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:45:38.653 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:45:38.689 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:45:38.689 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:45:38.690 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:45:38.791 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:45:38.823 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:45:38.824 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:45:38.824 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:45:38.876 INFO:teuthology.orchestra.run.smithi092.stderr:Inferring config /var/lib/ceph/00db0016-dfed-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T21:45:38.925 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:45:38.968 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:45:38.969 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:45:38.969 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:45:39.070 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:45:39.109 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:45:39.110 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:45:39.110 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:45:39.211 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:45:39.245 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:45:39.246 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:45:39.246 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:45:39.259 INFO:tasks.radosbench.radosbench.0.smithi092.stdout: 32 16 63986 63970 124.916 92.3125 0.00889098 0.00800069 2024-03-11T21:45:39.333 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:45:39 smithi097 ceph-mon[33317]: pgmap v949: 148 pgs: 148 active+clean; 3.4 GiB data, 14 GiB used, 522 GiB / 536 GiB avail; 1023 B/s rd, 104 MiB/s wr, 1.66k op/s 2024-03-11T21:45:39.334 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:45:39 smithi097 ceph-mon[33317]: from='mgr.14607 ' entity='mgr.y' 2024-03-11T21:45:39.334 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:45:39 smithi097 ceph-mon[33317]: from='mgr.14607 v1:172.21.15.92:0/1702717597' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T21:45:39.347 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:45:39.367 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:45:39 smithi092 ceph-mon[29572]: pgmap v949: 148 pgs: 148 active+clean; 3.4 GiB data, 14 GiB used, 522 GiB / 536 GiB avail; 1023 B/s rd, 104 MiB/s wr, 1.66k op/s 2024-03-11T21:45:39.367 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:45:39 smithi092 ceph-mon[29572]: from='mgr.14607 ' entity='mgr.y' 2024-03-11T21:45:39.367 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:45:39 smithi092 ceph-mon[29572]: from='mgr.14607 v1:172.21.15.92:0/1702717597' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T21:45:39.368 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:45:39 smithi092 ceph-mon[37563]: pgmap v949: 148 pgs: 148 active+clean; 3.4 GiB data, 14 GiB used, 522 GiB / 536 GiB avail; 1023 B/s rd, 104 MiB/s wr, 1.66k op/s 2024-03-11T21:45:39.368 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:45:39 smithi092 ceph-mon[37563]: from='mgr.14607 ' entity='mgr.y' 2024-03-11T21:45:39.368 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:45:39 smithi092 ceph-mon[37563]: from='mgr.14607 v1:172.21.15.92:0/1702717597' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T21:45:39.379 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:45:39.380 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:45:39.380 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:45:39.481 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:45:39.516 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:45:39.517 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:45:39.518 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:45:39.619 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:45:39.658 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:45:39.659 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:45:39.659 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:45:39.760 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:45:39.808 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:45:39.809 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:45:39.809 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:45:39.910 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:45:39.945 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:45:39.945 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:45:39.946 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:45:40.047 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:45:40.092 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:45:40.093 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:45:40.093 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:45:40.194 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:45:40.232 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:45:40.233 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:45:40.233 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:45:40.260 INFO:tasks.radosbench.radosbench.0.smithi092.stdout: 33 16 65453 65437 123.909 91.6875 0.00681981 0.00806561 2024-03-11T21:45:40.264 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:45:40 smithi092 ceph-mon[29572]: 4.0 deep-scrub starts 2024-03-11T21:45:40.264 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:45:40 smithi092 ceph-mon[29572]: 4.0 deep-scrub ok 2024-03-11T21:45:40.264 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:45:40 smithi092 ceph-mon[37563]: 4.0 deep-scrub starts 2024-03-11T21:45:40.264 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:45:40 smithi092 ceph-mon[37563]: 4.0 deep-scrub ok 2024-03-11T21:45:40.334 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:45:40 smithi097 ceph-mon[33317]: 4.0 deep-scrub starts 2024-03-11T21:45:40.334 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:45:40 smithi097 ceph-mon[33317]: 4.0 deep-scrub ok 2024-03-11T21:45:40.334 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:45:40.370 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:45:40.371 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:45:40.371 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:45:40.472 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:45:40.499 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:45:40.500 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:45:40.500 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:45:40.601 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:45:40.643 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:45:40.644 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:45:40.644 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:45:40.745 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:45:40.785 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:45:40.786 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:45:40.786 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:45:40.887 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:45:40.918 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:45:40.918 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:45:40.919 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:45:40.967 INFO:teuthology.orchestra.run.smithi092.stderr:Inferring config /var/lib/ceph/00db0016-dfed-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T21:45:41.020 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:45:41.044 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:45:41.044 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:45:41.044 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:45:41.103 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:45:41 smithi092 ceph-mon[29572]: pgmap v950: 148 pgs: 148 active+clean; 3.8 GiB data, 15 GiB used, 521 GiB / 536 GiB avail; 1023 B/s rd, 98 MiB/s wr, 1.57k op/s 2024-03-11T21:45:41.104 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:45:41 smithi092 ceph-mon[29572]: 5.6 scrub starts 2024-03-11T21:45:41.104 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:45:41 smithi092 ceph-mon[29572]: 5.6 scrub ok 2024-03-11T21:45:41.104 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:45:41 smithi092 ceph-mon[29572]: 5.11 scrub starts 2024-03-11T21:45:41.104 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:45:41 smithi092 ceph-mon[29572]: 5.11 scrub ok 2024-03-11T21:45:41.104 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:45:41 smithi092 ceph-mon[29572]: 5.7 scrub starts 2024-03-11T21:45:41.104 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:45:41 smithi092 ceph-mon[29572]: 5.7 scrub ok 2024-03-11T21:45:41.104 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:45:41 smithi092 ceph-mon[37563]: pgmap v950: 148 pgs: 148 active+clean; 3.8 GiB data, 15 GiB used, 521 GiB / 536 GiB avail; 1023 B/s rd, 98 MiB/s wr, 1.57k op/s 2024-03-11T21:45:41.104 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:45:41 smithi092 ceph-mon[37563]: 5.6 scrub starts 2024-03-11T21:45:41.104 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:45:41 smithi092 ceph-mon[37563]: 5.6 scrub ok 2024-03-11T21:45:41.104 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:45:41 smithi092 ceph-mon[37563]: 5.11 scrub starts 2024-03-11T21:45:41.104 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:45:41 smithi092 ceph-mon[37563]: 5.11 scrub ok 2024-03-11T21:45:41.104 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:45:41 smithi092 ceph-mon[37563]: 5.7 scrub starts 2024-03-11T21:45:41.104 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:45:41 smithi092 ceph-mon[37563]: 5.7 scrub ok 2024-03-11T21:45:41.145 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:45:41.178 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:45:41.179 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:45:41.180 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:45:41.260 INFO:tasks.radosbench.radosbench.0.smithi092.stdout: 34 16 66558 66542 122.296 69.0625 0.0131 0.00817232 2024-03-11T21:45:41.280 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:45:41.318 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:45:41.319 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:45:41.319 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:45:41.333 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:45:41 smithi097 ceph-mon[33317]: pgmap v950: 148 pgs: 148 active+clean; 3.8 GiB data, 15 GiB used, 521 GiB / 536 GiB avail; 1023 B/s rd, 98 MiB/s wr, 1.57k op/s 2024-03-11T21:45:41.333 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:45:41 smithi097 ceph-mon[33317]: 5.6 scrub starts 2024-03-11T21:45:41.334 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:45:41 smithi097 ceph-mon[33317]: 5.6 scrub ok 2024-03-11T21:45:41.334 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:45:41 smithi097 ceph-mon[33317]: 5.11 scrub starts 2024-03-11T21:45:41.334 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:45:41 smithi097 ceph-mon[33317]: 5.11 scrub ok 2024-03-11T21:45:41.334 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:45:41 smithi097 ceph-mon[33317]: 5.7 scrub starts 2024-03-11T21:45:41.334 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:45:41 smithi097 ceph-mon[33317]: 5.7 scrub ok 2024-03-11T21:45:41.420 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:45:41.459 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:45:41.459 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:45:41.460 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:45:41.560 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:45:41.599 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:45:41.600 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:45:41.600 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:45:41.701 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:45:41.730 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:45:41.731 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:45:41.731 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:45:41.832 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:45:41.872 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:45:41.873 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:45:41.873 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:45:41.974 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:45:42.011 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:45:42.011 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:45:42.012 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:45:42.074 INFO:teuthology.orchestra.run.smithi092.stderr:noscrub is set 2024-03-11T21:45:42.112 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:45:42.150 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:45:42.151 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:45:42.152 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:45:42.243 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:45:42 smithi097 ceph-mon[33317]: 2.18 scrub starts 2024-03-11T21:45:42.243 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:45:42 smithi097 ceph-mon[33317]: 2.18 scrub ok 2024-03-11T21:45:42.243 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:45:42 smithi097 ceph-mon[33317]: from='client.? v1:172.21.15.92:0/2470231161' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-11T21:45:42.243 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:45:42 smithi097 ceph-mon[33317]: from='client.35849 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-11T21:45:42.253 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:45:42.260 INFO:tasks.radosbench.radosbench.0.smithi092.stdout: 35 16 67864 67848 121.134 81.625 0.0123685 0.00825047 2024-03-11T21:45:42.294 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:45:42.294 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:45:42.294 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:45:42.368 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:45:42 smithi092 ceph-mon[29572]: 2.18 scrub starts 2024-03-11T21:45:42.368 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:45:42 smithi092 ceph-mon[29572]: 2.18 scrub ok 2024-03-11T21:45:42.368 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:45:42 smithi092 ceph-mon[29572]: from='client.? v1:172.21.15.92:0/2470231161' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-11T21:45:42.368 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:45:42 smithi092 ceph-mon[29572]: from='client.35849 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-11T21:45:42.368 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:45:42 smithi092 ceph-mon[37563]: 2.18 scrub starts 2024-03-11T21:45:42.368 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:45:42 smithi092 ceph-mon[37563]: 2.18 scrub ok 2024-03-11T21:45:42.368 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:45:42 smithi092 ceph-mon[37563]: from='client.? v1:172.21.15.92:0/2470231161' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-11T21:45:42.368 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:45:42 smithi092 ceph-mon[37563]: from='client.35849 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-11T21:45:42.396 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:45:42.435 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:45:42.435 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:45:42.435 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:45:42.536 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:45:42.571 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:45:42.572 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:45:42.572 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:45:42.673 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:45:42.713 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:45:42.714 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:45:42.714 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:45:42.815 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:45:42.845 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:45:42.846 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:45:42.846 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:45:42.947 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:45:42.986 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:45:42.986 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:45:42.987 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:45:43.087 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:45:43.123 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:45:43.123 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:45:43.124 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:45:43.225 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:45:43.252 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:45:43.253 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:45:43.253 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:45:43.260 INFO:tasks.radosbench.radosbench.0.smithi092.stdout: 36 16 69248 69232 120.172 86.5 0.00363597 0.0083168 2024-03-11T21:45:43.271 INFO:teuthology.orchestra.run.smithi097.stdout:ERROR: (22) Invalid argument 2024-03-11T21:45:43.271 INFO:teuthology.orchestra.run.smithi097.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-11T21:45:43.271 INFO:journalctl@ceph.iscsi.iscsi.a.smithi097.stdout:Mar 11 21:45:43 smithi097 ceph-00db0016-dfed-11ee-95c8-87774f69a715-iscsi-iscsi-a[60943]: debug there is no tcmu-runner data available 2024-03-11T21:45:43.272 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:45:43 smithi097 ceph-mon[33317]: pgmap v951: 148 pgs: 148 active+clean; 3.8 GiB data, 16 GiB used, 521 GiB / 536 GiB avail; 889 B/s rd, 85 MiB/s wr, 1.36k op/s 2024-03-11T21:45:43.272 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:45:43 smithi097 ceph-mon[33317]: 6.14 scrub starts 2024-03-11T21:45:43.272 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:45:43 smithi097 ceph-mon[33317]: 6.14 scrub ok 2024-03-11T21:45:43.272 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:45:43 smithi097 ceph-mon[33317]: 2.15 scrub starts 2024-03-11T21:45:43.272 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:45:43 smithi097 ceph-mon[33317]: 2.15 scrub ok 2024-03-11T21:45:43.272 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:45:43 smithi097 ceph-mon[33317]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-03-11T21:45:43.272 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:45:43 smithi097 ceph-mon[33317]: from='client.35849 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-03-11T21:45:43.272 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:45:43 smithi097 ceph-mon[33317]: 5.15 scrub starts 2024-03-11T21:45:43.272 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:45:43 smithi097 ceph-mon[33317]: osdmap e456: 8 total, 8 up, 6 in 2024-03-11T21:45:43.348 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:45:43 smithi092 ceph-mon[29572]: pgmap v951: 148 pgs: 148 active+clean; 3.8 GiB data, 16 GiB used, 521 GiB / 536 GiB avail; 889 B/s rd, 85 MiB/s wr, 1.36k op/s 2024-03-11T21:45:43.348 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:45:43 smithi092 ceph-mon[29572]: 6.14 scrub starts 2024-03-11T21:45:43.348 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:45:43 smithi092 ceph-mon[29572]: 6.14 scrub ok 2024-03-11T21:45:43.348 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:45:43 smithi092 ceph-mon[29572]: 2.15 scrub starts 2024-03-11T21:45:43.348 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:45:43 smithi092 ceph-mon[29572]: 2.15 scrub ok 2024-03-11T21:45:43.348 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:45:43 smithi092 ceph-mon[29572]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-03-11T21:45:43.348 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:45:43 smithi092 ceph-mon[29572]: from='client.35849 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-03-11T21:45:43.348 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:45:43 smithi092 ceph-mon[29572]: 5.15 scrub starts 2024-03-11T21:45:43.348 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:45:43 smithi092 ceph-mon[29572]: osdmap e456: 8 total, 8 up, 6 in 2024-03-11T21:45:43.349 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:45:43 smithi092 ceph-mon[37563]: pgmap v951: 148 pgs: 148 active+clean; 3.8 GiB data, 16 GiB used, 521 GiB / 536 GiB avail; 889 B/s rd, 85 MiB/s wr, 1.36k op/s 2024-03-11T21:45:43.349 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:45:43 smithi092 ceph-mon[37563]: 6.14 scrub starts 2024-03-11T21:45:43.349 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:45:43 smithi092 ceph-mon[37563]: 6.14 scrub ok 2024-03-11T21:45:43.349 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:45:43 smithi092 ceph-mon[37563]: 2.15 scrub starts 2024-03-11T21:45:43.349 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:45:43 smithi092 ceph-mon[37563]: 2.15 scrub ok 2024-03-11T21:45:43.349 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:45:43 smithi092 ceph-mon[37563]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-03-11T21:45:43.349 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:45:43 smithi092 ceph-mon[37563]: from='client.35849 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-03-11T21:45:43.349 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:45:43 smithi092 ceph-mon[37563]: 5.15 scrub starts 2024-03-11T21:45:43.349 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:45:43 smithi092 ceph-mon[37563]: osdmap e456: 8 total, 8 up, 6 in 2024-03-11T21:45:43.355 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:45:43.392 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:45:43.393 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:45:43.393 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:45:43.494 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:45:43.533 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:45:43.534 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:45:43.534 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:45:43.635 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:45:43.666 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:45:43.666 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:45:43.666 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:45:43.767 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:45:43.800 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:45:43.801 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:45:43.801 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:45:43.902 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:45:43.913 INFO:teuthology.orchestra.run.smithi092.stdout:osd.0: {} 2024-03-11T21:45:43.943 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:45:43.943 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:45:43.944 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:45:43.962 INFO:teuthology.orchestra.run.smithi092.stdout:osd.1: {} 2024-03-11T21:45:43.997 INFO:teuthology.orchestra.run.smithi092.stdout:osd.2: {} 2024-03-11T21:45:44.033 INFO:teuthology.orchestra.run.smithi092.stdout:osd.3: {} 2024-03-11T21:45:44.045 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:45:44.063 INFO:teuthology.orchestra.run.smithi092.stdout:osd.4: {} 2024-03-11T21:45:44.068 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:45:44.069 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:45:44.069 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:45:44.083 INFO:teuthology.orchestra.run.smithi092.stdout:osd.5: {} 2024-03-11T21:45:44.169 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:45:44.175 INFO:teuthology.orchestra.run.smithi092.stdout:osd.6: {} 2024-03-11T21:45:44.214 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:45:44 smithi092 ceph-mon[29572]: from='client.24469 v1:172.21.15.97:0/404208310' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:45:44.214 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:45:44 smithi092 ceph-mon[29572]: from='client.? v1:172.21.15.92:0/883382779' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-11T21:45:44.215 INFO:teuthology.orchestra.run.smithi092.stdout:osd.7: {} 2024-03-11T21:45:44.215 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:45:44 smithi092 ceph-mon[37563]: from='client.24469 v1:172.21.15.97:0/404208310' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:45:44.215 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:45:44 smithi092 ceph-mon[37563]: from='client.? v1:172.21.15.92:0/883382779' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-11T21:45:44.216 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:45:44.216 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:45:44.216 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:45:44.217 INFO:teuthology.orchestra.run.smithi092.stderr:osd.0: osd_enable_op_tracker = '' 2024-03-11T21:45:44.217 INFO:teuthology.orchestra.run.smithi092.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-11T21:45:44.217 INFO:teuthology.orchestra.run.smithi092.stderr:osd.2: osd_enable_op_tracker = '' 2024-03-11T21:45:44.218 INFO:teuthology.orchestra.run.smithi092.stderr:osd.3: osd_enable_op_tracker = '' 2024-03-11T21:45:44.218 INFO:teuthology.orchestra.run.smithi092.stderr:osd.4: osd_enable_op_tracker = '' 2024-03-11T21:45:44.218 INFO:teuthology.orchestra.run.smithi092.stderr:osd.5: osd_enable_op_tracker = '' 2024-03-11T21:45:44.218 INFO:teuthology.orchestra.run.smithi092.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-11T21:45:44.218 INFO:teuthology.orchestra.run.smithi092.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-11T21:45:44.260 INFO:tasks.radosbench.radosbench.0.smithi092.stdout: 37 16 70357 70341 118.797 69.3125 0.0233718 0.00840348 2024-03-11T21:45:44.318 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:45:44.335 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:45:44 smithi097 ceph-mon[33317]: from='client.24469 v1:172.21.15.97:0/404208310' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:45:44.335 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:45:44 smithi097 ceph-mon[33317]: from='client.? v1:172.21.15.92:0/883382779' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-11T21:45:44.358 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:45:44.358 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:45:44.359 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:45:44.460 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:45:44.492 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:45:44.493 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:45:44.494 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:45:44.594 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:45:44.630 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:45:44.631 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:45:44.631 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:45:44.732 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:45:44.767 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:45:44.768 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:45:44.768 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:45:44.869 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:45:44.893 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:45:44.893 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:45:44.893 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:45:44.994 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:45:45.028 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:45:45.029 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:45:45.030 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:45:45.130 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:45:45.169 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:45:45.170 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:45:45.171 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:45:45.260 INFO:tasks.radosbench.radosbench.0.smithi092.stdout: 38 16 71664 71648 117.82 81.6875 0.0089773 0.00848302 2024-03-11T21:45:45.271 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:45:45.306 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:45:45.307 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:45:45.307 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:45:45.313 DEBUG:teuthology.orchestra.run.smithi092:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid 00db0016-dfed-11ee-95c8-87774f69a715 -- ceph osd set nodeep-scrub 2024-03-11T21:45:45.352 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:45:45 smithi092 ceph-mon[29572]: pgmap v953: 148 pgs: 1 active+clean+scrubbing, 147 active+clean; 4.1 GiB data, 16 GiB used, 520 GiB / 536 GiB avail; 1023 B/s rd, 91 MiB/s wr, 1.46k op/s 2024-03-11T21:45:45.352 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:45:45 smithi092 ceph-mon[37563]: pgmap v953: 148 pgs: 1 active+clean+scrubbing, 147 active+clean; 4.1 GiB data, 16 GiB used, 520 GiB / 536 GiB avail; 1023 B/s rd, 91 MiB/s wr, 1.46k op/s 2024-03-11T21:45:45.408 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:45:45.439 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:45:45.440 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:45:45.440 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:45:45.541 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:45:45.584 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:45:45 smithi097 ceph-mon[33317]: pgmap v953: 148 pgs: 1 active+clean+scrubbing, 147 active+clean; 4.1 GiB data, 16 GiB used, 520 GiB / 536 GiB avail; 1023 B/s rd, 91 MiB/s wr, 1.46k op/s 2024-03-11T21:45:45.585 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:45:45.585 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:45:45.585 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:45:45.686 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:45:45.725 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:45:45.726 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:45:45.727 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:45:45.732 DEBUG:teuthology.orchestra.run.smithi097:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid 00db0016-dfed-11ee-95c8-87774f69a715 -- ceph daemon osd.4 dump_blocked_ops 2024-03-11T21:45:45.827 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:45:45.865 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:45:45.865 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:45:45.866 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:45:45.967 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:45:45.995 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:45:45.996 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:45:45.996 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:45:46.097 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:45:46.128 INFO:teuthology.orchestra.run.smithi097.stderr:Inferring config /var/lib/ceph/00db0016-dfed-11ee-95c8-87774f69a715/mon.b/config 2024-03-11T21:45:46.132 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:45:46.132 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:45:46.133 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:45:46.234 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:45:46.260 INFO:tasks.radosbench.radosbench.0.smithi092.stdout: 39 16 72949 72933 116.859 80.3125 0.0174176 0.00855205 2024-03-11T21:45:46.268 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:45:46.268 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:45:46.269 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:45:46.370 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:45:46.411 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:45:46.411 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:45:46.412 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:45:46.513 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:45:46.547 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:45:46.547 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:45:46.548 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:45:46.649 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:45:46.700 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:45:46.701 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:45:46.701 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:45:46.802 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:45:46.847 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:45:46.848 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:45:46.848 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:45:46.949 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:45:46.993 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:45:46.994 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:45:46.994 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:45:47.095 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:45:47.128 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:45:47.129 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:45:47.129 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:45:47.189 INFO:teuthology.orchestra.run.smithi092.stderr:Inferring config /var/lib/ceph/00db0016-dfed-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T21:45:47.230 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:45:47.260 INFO:tasks.radosbench.radosbench.0.smithi092.stdout:2024-03-11T21:45:47.272037+0000 min lat: 0.00128848 max lat: 0.230006 avg lat: 0.00861548 2024-03-11T21:45:47.261 INFO:tasks.radosbench.radosbench.0.smithi092.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-03-11T21:45:47.261 INFO:tasks.radosbench.radosbench.0.smithi092.stdout: 40 16 74269 74253 115.999 82.5 0.0208156 0.00861548 2024-03-11T21:45:47.262 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:45:47.262 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:45:47.263 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:45:47.334 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:45:47 smithi097 ceph-mon[33317]: pgmap v954: 148 pgs: 1 active+clean+scrubbing, 147 active+clean; 4.2 GiB data, 17 GiB used, 520 GiB / 536 GiB avail; 1023 B/s rd, 88 MiB/s wr, 1.41k op/s 2024-03-11T21:45:47.364 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:45:47.368 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:45:47 smithi092 ceph-mon[29572]: pgmap v954: 148 pgs: 1 active+clean+scrubbing, 147 active+clean; 4.2 GiB data, 17 GiB used, 520 GiB / 536 GiB avail; 1023 B/s rd, 88 MiB/s wr, 1.41k op/s 2024-03-11T21:45:47.368 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:45:47 smithi092 ceph-mon[37563]: pgmap v954: 148 pgs: 1 active+clean+scrubbing, 147 active+clean; 4.2 GiB data, 17 GiB used, 520 GiB / 536 GiB avail; 1023 B/s rd, 88 MiB/s wr, 1.41k op/s 2024-03-11T21:45:47.399 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:45:47.399 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:45:47.400 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:45:47.500 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:45:47.535 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:45:47.536 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:45:47.536 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:45:47.637 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:45:47.677 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:45:47.678 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:45:47.678 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:45:47.779 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:45:47.819 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:45:47.819 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:45:47.820 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:45:47.921 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:45:47.962 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:45:47.963 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:45:47.963 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:45:47.992 INFO:journalctl@ceph.mgr.y.smithi092.stdout:Mar 11 21:45:47 smithi092 ceph-00db0016-dfed-11ee-95c8-87774f69a715-mgr-y[29801]: ::ffff:172.21.15.97 - - [11/Mar/2024:21:45:47] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-11T21:45:48.051 DEBUG:teuthology.orchestra.run.smithi092:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid 00db0016-dfed-11ee-95c8-87774f69a715 -- ceph tell osd.7 reset_purged_snaps_last 2024-03-11T21:45:48.064 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:45:48.133 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:45:48.134 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:45:48.135 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:45:48.236 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:45:48.261 INFO:tasks.radosbench.radosbench.0.smithi092.stdout: 41 16 75494 75478 115.037 76.5625 0.0165491 0.00868708 2024-03-11T21:45:48.270 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:45:48.271 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:45:48.272 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:45:48.372 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:45:48.407 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:45:48.407 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:45:48.408 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:45:48.475 INFO:teuthology.orchestra.run.smithi097.stdout:{ 2024-03-11T21:45:48.475 INFO:teuthology.orchestra.run.smithi097.stdout: "ops": [], 2024-03-11T21:45:48.475 INFO:teuthology.orchestra.run.smithi097.stdout: "complaint_time": 30, 2024-03-11T21:45:48.475 INFO:teuthology.orchestra.run.smithi097.stdout: "num_blocked_ops": 0 2024-03-11T21:45:48.475 INFO:teuthology.orchestra.run.smithi097.stdout:} 2024-03-11T21:45:48.508 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:45:48.552 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:45:48.553 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:45:48.553 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:45:48.564 DEBUG:teuthology.orchestra.run.smithi092:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid 00db0016-dfed-11ee-95c8-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-03-11T21:45:48.654 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:45:48.706 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:45:48.706 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:45:48.706 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:45:48.807 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:45:48.861 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:45:48.861 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:45:48.862 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:45:48.963 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:45:48.998 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:45:48.999 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:45:48.999 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:45:49.100 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:45:49.137 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:45:49.137 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:45:49.138 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:45:49.228 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:45:49 smithi097 ceph-mon[33317]: pgmap v955: 148 pgs: 148 active+clean; 4.2 GiB data, 17 GiB used, 519 GiB / 536 GiB avail; 4.3 KiB/s rd, 87 MiB/s wr, 1.41k op/s 2024-03-11T21:45:49.239 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:45:49.261 INFO:tasks.radosbench.radosbench.0.smithi092.stdout: 42 15 76634 76619 113.996 71.3125 0.00478084 0.00876738 2024-03-11T21:45:49.273 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:45:49.274 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:45:49.274 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:45:49.279 DEBUG:teuthology.orchestra.run.smithi097:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid 00db0016-dfed-11ee-95c8-87774f69a715 -- ceph daemon osd.4 dump_historic_ops 2024-03-11T21:45:49.367 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:45:49 smithi092 ceph-mon[29572]: pgmap v955: 148 pgs: 148 active+clean; 4.2 GiB data, 17 GiB used, 519 GiB / 536 GiB avail; 4.3 KiB/s rd, 87 MiB/s wr, 1.41k op/s 2024-03-11T21:45:49.368 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:45:49 smithi092 ceph-mon[37563]: pgmap v955: 148 pgs: 148 active+clean; 4.2 GiB data, 17 GiB used, 519 GiB / 536 GiB avail; 4.3 KiB/s rd, 87 MiB/s wr, 1.41k op/s 2024-03-11T21:45:49.374 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:45:49.415 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:45:49.415 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:45:49.416 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:45:49.516 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:45:49.566 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:45:49.566 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:45:49.567 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:45:49.668 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:45:49.709 INFO:teuthology.orchestra.run.smithi097.stderr:Inferring config /var/lib/ceph/00db0016-dfed-11ee-95c8-87774f69a715/mon.b/config 2024-03-11T21:45:49.719 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:45:49.720 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:45:49.720 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:45:49.821 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:45:49.865 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:45:49.867 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:45:49.868 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:45:49.969 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:45:50.003 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:45:50.003 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:45:50.004 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:45:50.104 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:45:50.139 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:45:50.139 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:45:50.140 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:45:50.240 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:45:50.261 INFO:tasks.radosbench.radosbench.0.smithi092.stdout: 43 16 77963 77947 113.275 83 0.0158721 0.00882331 2024-03-11T21:45:50.274 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:45:50.275 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:45:50.275 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:45:50.333 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:45:50 smithi097 ceph-mon[33317]: purged_snaps scrub starts 2024-03-11T21:45:50.334 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:45:50 smithi097 ceph-mon[33317]: purged_snaps scrub ok 2024-03-11T21:45:50.334 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:45:50 smithi097 ceph-mon[33317]: osdmap e457: 8 total, 8 up, 6 in 2024-03-11T21:45:50.367 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:45:50 smithi092 ceph-mon[29572]: purged_snaps scrub starts 2024-03-11T21:45:50.368 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:45:50 smithi092 ceph-mon[29572]: purged_snaps scrub ok 2024-03-11T21:45:50.368 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:45:50 smithi092 ceph-mon[29572]: osdmap e457: 8 total, 8 up, 6 in 2024-03-11T21:45:50.368 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:45:50 smithi092 ceph-mon[37563]: purged_snaps scrub starts 2024-03-11T21:45:50.368 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:45:50 smithi092 ceph-mon[37563]: purged_snaps scrub ok 2024-03-11T21:45:50.368 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:45:50 smithi092 ceph-mon[37563]: osdmap e457: 8 total, 8 up, 6 in 2024-03-11T21:45:50.376 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:45:50.406 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:45:50.406 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:45:50.407 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:45:50.507 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:45:50.534 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:45:50.534 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:45:50.535 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:45:50.636 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:45:50.671 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:45:50.672 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:45:50.672 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:45:50.773 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:45:50.810 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:45:50.811 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:45:50.811 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:45:50.912 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:45:50.955 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:45:50.956 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:45:50.956 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:45:51.057 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:45:51.096 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:45:51.097 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:45:51.098 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:45:51.198 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:45:51.234 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:45:51.235 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:45:51.235 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:45:51.261 INFO:tasks.radosbench.radosbench.0.smithi092.stdout: 44 16 79190 79174 112.444 76.6875 0.00510493 0.0088886 2024-03-11T21:45:51.335 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:45:51.368 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:45:51 smithi092 ceph-mon[29572]: pgmap v956: 148 pgs: 148 active+clean; 4.6 GiB data, 18 GiB used, 519 GiB / 536 GiB avail; 31 KiB/s rd, 82 MiB/s wr, 1.36k op/s 2024-03-11T21:45:51.368 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:45:51.368 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:45:51 smithi092 ceph-mon[37563]: pgmap v956: 148 pgs: 148 active+clean; 4.6 GiB data, 18 GiB used, 519 GiB / 536 GiB avail; 31 KiB/s rd, 82 MiB/s wr, 1.36k op/s 2024-03-11T21:45:51.369 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:45:51.369 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:45:51.461 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:45:51 smithi097 ceph-mon[33317]: pgmap v956: 148 pgs: 148 active+clean; 4.6 GiB data, 18 GiB used, 519 GiB / 536 GiB avail; 31 KiB/s rd, 82 MiB/s wr, 1.36k op/s 2024-03-11T21:45:51.470 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:45:51.508 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:45:51.509 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:45:51.509 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:45:51.610 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:45:51.646 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:45:51.646 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:45:51.646 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:45:51.747 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:45:51.786 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:45:51.787 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:45:51.787 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:45:51.888 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:45:51.927 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:45:51.928 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:45:51.928 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:45:52.029 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:45:52.068 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:45:52.069 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:45:52.069 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:45:52.170 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:45:52.209 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:45:52.210 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:45:52.211 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:45:52.261 INFO:tasks.radosbench.radosbench.0.smithi092.stdout: 45 16 80516 80500 111.787 82.875 0.00391334 0.0089368 2024-03-11T21:45:52.311 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:45:52.351 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:45:52.351 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:45:52.352 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:45:52.452 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:45:52.490 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:45:52.490 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:45:52.491 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:45:52.555 INFO:teuthology.orchestra.run.smithi097.stdout:{ 2024-03-11T21:45:52.555 INFO:teuthology.orchestra.run.smithi097.stdout: "size": 20, 2024-03-11T21:45:52.555 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 600, 2024-03-11T21:45:52.555 INFO:teuthology.orchestra.run.smithi097.stdout: "ops": [ 2024-03-11T21:45:52.555 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:45:52.556 INFO:teuthology.orchestra.run.smithi097.stdout: "description": "osd_repop(client.15744.0:69290 10.6 e371/356 10:64e7935d:::benchmark_data_smithi092_161462_object69289:head v 371'4305, mlcod=371'4303)", 2024-03-11T21:45:52.556 INFO:teuthology.orchestra.run.smithi097.stdout: "initiated_at": "2024-03-11T21:41:27.813843+0000", 2024-03-11T21:45:52.556 INFO:teuthology.orchestra.run.smithi097.stdout: "age": 264.74910339500002, 2024-03-11T21:45:52.556 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.151117429, 2024-03-11T21:45:52.556 INFO:teuthology.orchestra.run.smithi097.stdout: "type_data": { 2024-03-11T21:45:52.556 INFO:teuthology.orchestra.run.smithi097.stdout: "flag_point": "started", 2024-03-11T21:45:52.556 INFO:teuthology.orchestra.run.smithi097.stdout: "events": [ 2024-03-11T21:45:52.556 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:45:52.556 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "initiated", 2024-03-11T21:45:52.556 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:41:27.813843+0000", 2024-03-11T21:45:52.556 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-03-11T21:45:52.556 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:45:52.556 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:45:52.557 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "header_read", 2024-03-11T21:45:52.557 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:41:27.813843+0000", 2024-03-11T21:45:52.557 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-03-11T21:45:52.557 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:45:52.557 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:45:52.557 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "throttled", 2024-03-11T21:45:52.557 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:41:27.813844+0000", 2024-03-11T21:45:52.557 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 4.15e-07 2024-03-11T21:45:52.557 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:45:52.557 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:45:52.557 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "all_read", 2024-03-11T21:45:52.557 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:41:27.813871+0000", 2024-03-11T21:45:52.557 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 2.7078999999999999e-05 2024-03-11T21:45:52.557 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:45:52.558 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:45:52.558 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "dispatched", 2024-03-11T21:45:52.558 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:41:27.813883+0000", 2024-03-11T21:45:52.558 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.1977000000000001e-05 2024-03-11T21:45:52.558 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:45:52.558 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:45:52.558 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "queued_for_pg", 2024-03-11T21:45:52.558 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:41:27.813892+0000", 2024-03-11T21:45:52.558 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 9.0999999999999993e-06 2024-03-11T21:45:52.558 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:45:52.558 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:45:52.558 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "reached_pg", 2024-03-11T21:45:52.558 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:41:27.814328+0000", 2024-03-11T21:45:52.559 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.00043647500000000002 2024-03-11T21:45:52.559 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:45:52.559 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:45:52.559 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "started", 2024-03-11T21:45:52.559 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:41:27.814350+0000", 2024-03-11T21:45:52.559 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 2.1223e-05 2024-03-11T21:45:52.559 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:45:52.559 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:45:52.559 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "done", 2024-03-11T21:45:52.559 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:41:27.964961+0000", 2024-03-11T21:45:52.559 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.15061115999999999 2024-03-11T21:45:52.559 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-03-11T21:45:52.559 INFO:teuthology.orchestra.run.smithi097.stdout: ] 2024-03-11T21:45:52.559 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-03-11T21:45:52.560 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:45:52.560 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:45:52.560 INFO:teuthology.orchestra.run.smithi097.stdout: "description": "osd_op(client.15744.0:69295 10.1 10.ecc5e7a1 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e371)", 2024-03-11T21:45:52.560 INFO:teuthology.orchestra.run.smithi097.stdout: "initiated_at": "2024-03-11T21:41:27.826949+0000", 2024-03-11T21:45:52.560 INFO:teuthology.orchestra.run.smithi097.stdout: "age": 264.73599729, 2024-03-11T21:45:52.560 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.14217101400000001, 2024-03-11T21:45:52.560 INFO:teuthology.orchestra.run.smithi097.stdout: "type_data": { 2024-03-11T21:45:52.560 INFO:teuthology.orchestra.run.smithi097.stdout: "flag_point": "reached pg", 2024-03-11T21:45:52.560 INFO:teuthology.orchestra.run.smithi097.stdout: "client_info": { 2024-03-11T21:45:52.560 INFO:teuthology.orchestra.run.smithi097.stdout: "client": "client.15744", 2024-03-11T21:45:52.560 INFO:teuthology.orchestra.run.smithi097.stdout: "client_addr": "v1:172.21.15.92:0/3334000812", 2024-03-11T21:45:52.560 INFO:teuthology.orchestra.run.smithi097.stdout: "tid": 69295 2024-03-11T21:45:52.560 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:45:52.561 INFO:teuthology.orchestra.run.smithi097.stdout: "events": [ 2024-03-11T21:45:52.561 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:45:52.561 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "initiated", 2024-03-11T21:45:52.561 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:41:27.826949+0000", 2024-03-11T21:45:52.561 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-03-11T21:45:52.561 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:45:52.561 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:45:52.561 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "header_read", 2024-03-11T21:45:52.561 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:41:27.826949+0000", 2024-03-11T21:45:52.561 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-03-11T21:45:52.561 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:45:52.561 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:45:52.561 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "throttled", 2024-03-11T21:45:52.561 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:41:27.826952+0000", 2024-03-11T21:45:52.562 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 2.751e-06 2024-03-11T21:45:52.562 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:45:52.562 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:45:52.562 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "all_read", 2024-03-11T21:45:52.562 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:41:27.826991+0000", 2024-03-11T21:45:52.562 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 3.9298000000000002e-05 2024-03-11T21:45:52.562 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:45:52.562 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:45:52.562 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "dispatched", 2024-03-11T21:45:52.562 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:41:27.827007+0000", 2024-03-11T21:45:52.562 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.5141e-05 2024-03-11T21:45:52.562 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:45:52.562 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:45:52.562 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "queued_for_pg", 2024-03-11T21:45:52.562 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:41:27.827016+0000", 2024-03-11T21:45:52.563 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 9.8460000000000003e-06 2024-03-11T21:45:52.563 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:45:52.563 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:45:52.563 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "reached_pg", 2024-03-11T21:45:52.563 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:41:27.969022+0000", 2024-03-11T21:45:52.563 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.14200602800000001 2024-03-11T21:45:52.563 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:45:52.563 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:45:52.563 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "done", 2024-03-11T21:45:52.563 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:41:27.969120+0000", 2024-03-11T21:45:52.563 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 9.7949999999999996e-05 2024-03-11T21:45:52.563 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-03-11T21:45:52.563 INFO:teuthology.orchestra.run.smithi097.stdout: ] 2024-03-11T21:45:52.563 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-03-11T21:45:52.563 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:45:52.564 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:45:52.564 INFO:teuthology.orchestra.run.smithi097.stdout: "description": "osd_op(client.15744.0:69295 10.1 10:85e7a337:::benchmark_data_smithi092_161462_object69294:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e372)", 2024-03-11T21:45:52.564 INFO:teuthology.orchestra.run.smithi097.stdout: "initiated_at": "2024-03-11T21:41:27.827037+0000", 2024-03-11T21:45:52.564 INFO:teuthology.orchestra.run.smithi097.stdout: "age": 264.73591013200001, 2024-03-11T21:45:52.564 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.0170514900000001, 2024-03-11T21:45:52.564 INFO:teuthology.orchestra.run.smithi097.stdout: "type_data": { 2024-03-11T21:45:52.564 INFO:teuthology.orchestra.run.smithi097.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-11T21:45:52.564 INFO:teuthology.orchestra.run.smithi097.stdout: "client_info": { 2024-03-11T21:45:52.564 INFO:teuthology.orchestra.run.smithi097.stdout: "client": "client.15744", 2024-03-11T21:45:52.564 INFO:teuthology.orchestra.run.smithi097.stdout: "client_addr": "v1:172.21.15.92:0/3334000812", 2024-03-11T21:45:52.564 INFO:teuthology.orchestra.run.smithi097.stdout: "tid": 69295 2024-03-11T21:45:52.564 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:45:52.564 INFO:teuthology.orchestra.run.smithi097.stdout: "events": [ 2024-03-11T21:45:52.564 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:45:52.564 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "initiated", 2024-03-11T21:45:52.564 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:41:27.827037+0000", 2024-03-11T21:45:52.565 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-03-11T21:45:52.565 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:45:52.565 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:45:52.565 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "header_read", 2024-03-11T21:45:52.565 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:41:27.827037+0000", 2024-03-11T21:45:52.565 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-03-11T21:45:52.565 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:45:52.565 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:45:52.565 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "throttled", 2024-03-11T21:45:52.565 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:41:27.827037+0000", 2024-03-11T21:45:52.565 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 7.0999999999999998e-07 2024-03-11T21:45:52.565 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:45:52.565 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:45:52.565 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "all_read", 2024-03-11T21:45:52.565 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:41:27.827067+0000", 2024-03-11T21:45:52.566 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 2.9376000000000001e-05 2024-03-11T21:45:52.566 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:45:52.566 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:45:52.566 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "dispatched", 2024-03-11T21:45:52.566 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:41:27.827074+0000", 2024-03-11T21:45:52.566 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 6.8539999999999999e-06 2024-03-11T21:45:52.566 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:45:52.566 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:45:52.566 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "queued_for_pg", 2024-03-11T21:45:52.566 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:41:27.827079+0000", 2024-03-11T21:45:52.566 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 5.6459999999999998e-06 2024-03-11T21:45:52.566 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:45:52.566 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:45:52.567 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "reached_pg", 2024-03-11T21:45:52.567 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:41:27.969273+0000", 2024-03-11T21:45:52.567 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.142194131 2024-03-11T21:45:52.567 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:45:52.567 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:45:52.567 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "waiting for peered", 2024-03-11T21:45:52.567 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:41:27.969284+0000", 2024-03-11T21:45:52.567 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.0491e-05 2024-03-11T21:45:52.567 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:45:52.567 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:45:52.567 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "reached_pg", 2024-03-11T21:45:52.567 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:41:28.830462+0000", 2024-03-11T21:45:52.567 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.861178208 2024-03-11T21:45:52.567 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:45:52.568 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:45:52.568 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "started", 2024-03-11T21:45:52.568 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:41:28.830604+0000", 2024-03-11T21:45:52.568 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.00014221599999999999 2024-03-11T21:45:52.568 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:45:52.568 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:45:52.568 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "waiting for subops from 0,7", 2024-03-11T21:45:52.568 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:41:28.830767+0000", 2024-03-11T21:45:52.568 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.00016259400000000001 2024-03-11T21:45:52.568 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:45:52.568 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:45:52.568 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "op_commit", 2024-03-11T21:45:52.568 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:41:28.836048+0000", 2024-03-11T21:45:52.568 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.0052813030000000002 2024-03-11T21:45:52.569 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:45:52.569 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:45:52.569 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "sub_op_commit_rec", 2024-03-11T21:45:52.569 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:41:28.843753+0000", 2024-03-11T21:45:52.569 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.0077048439999999998 2024-03-11T21:45:52.569 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:45:52.569 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:45:52.569 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "sub_op_commit_rec", 2024-03-11T21:45:52.569 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:41:28.843975+0000", 2024-03-11T21:45:52.569 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.000221788 2024-03-11T21:45:52.569 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:45:52.569 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:45:52.569 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "commit_sent", 2024-03-11T21:45:52.569 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:41:28.844039+0000", 2024-03-11T21:45:52.569 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 6.4274000000000002e-05 2024-03-11T21:45:52.570 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:45:52.570 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:45:52.570 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "done", 2024-03-11T21:45:52.570 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:41:28.844088+0000", 2024-03-11T21:45:52.570 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 4.9054999999999997e-05 2024-03-11T21:45:52.570 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-03-11T21:45:52.570 INFO:teuthology.orchestra.run.smithi097.stdout: ] 2024-03-11T21:45:52.570 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-03-11T21:45:52.570 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:45:52.570 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:45:52.570 INFO:teuthology.orchestra.run.smithi097.stdout: "description": "osd_op(client.15744.0:120195 10.1 10:873498a8:::benchmark_data_smithi092_161462_object10219:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e400)", 2024-03-11T21:45:52.570 INFO:teuthology.orchestra.run.smithi097.stdout: "initiated_at": "2024-03-11T21:42:36.641796+0000", 2024-03-11T21:45:52.570 INFO:teuthology.orchestra.run.smithi097.stdout: "age": 195.921150453, 2024-03-11T21:45:52.570 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.12297045199999999, 2024-03-11T21:45:52.571 INFO:teuthology.orchestra.run.smithi097.stdout: "type_data": { 2024-03-11T21:45:52.571 INFO:teuthology.orchestra.run.smithi097.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-11T21:45:52.571 INFO:teuthology.orchestra.run.smithi097.stdout: "client_info": { 2024-03-11T21:45:52.571 INFO:teuthology.orchestra.run.smithi097.stdout: "client": "client.15744", 2024-03-11T21:45:52.571 INFO:teuthology.orchestra.run.smithi097.stdout: "client_addr": "v1:172.21.15.92:0/3334000812", 2024-03-11T21:45:52.571 INFO:teuthology.orchestra.run.smithi097.stdout: "tid": 120195 2024-03-11T21:45:52.571 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:45:52.571 INFO:teuthology.orchestra.run.smithi097.stdout: "events": [ 2024-03-11T21:45:52.571 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:45:52.571 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "initiated", 2024-03-11T21:45:52.571 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:42:36.641796+0000", 2024-03-11T21:45:52.571 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-03-11T21:45:52.571 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:45:52.571 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:45:52.571 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "header_read", 2024-03-11T21:45:52.572 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:42:36.641796+0000", 2024-03-11T21:45:52.572 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-03-11T21:45:52.572 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:45:52.572 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:45:52.572 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "throttled", 2024-03-11T21:45:52.572 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:42:36.641798+0000", 2024-03-11T21:45:52.572 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.9589999999999998e-06 2024-03-11T21:45:52.572 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:45:52.572 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:45:52.572 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "all_read", 2024-03-11T21:45:52.572 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:42:36.641805+0000", 2024-03-11T21:45:52.572 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 7.0090000000000001e-06 2024-03-11T21:45:52.572 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:45:52.572 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:45:52.572 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "dispatched", 2024-03-11T21:45:52.573 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:42:36.641818+0000", 2024-03-11T21:45:52.573 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.3067e-05 2024-03-11T21:45:52.573 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:45:52.573 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:45:52.573 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "queued_for_pg", 2024-03-11T21:45:52.573 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:42:36.641827+0000", 2024-03-11T21:45:52.573 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 8.8859999999999993e-06 2024-03-11T21:45:52.573 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:45:52.573 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:45:52.573 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "reached_pg", 2024-03-11T21:45:52.573 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:42:36.641892+0000", 2024-03-11T21:45:52.573 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 6.4879e-05 2024-03-11T21:45:52.573 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:45:52.573 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:45:52.573 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "started", 2024-03-11T21:45:52.574 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:42:36.642022+0000", 2024-03-11T21:45:52.574 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.00013038899999999999 2024-03-11T21:45:52.574 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:45:52.574 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:45:52.574 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "waiting for subops from 0,7", 2024-03-11T21:45:52.574 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:42:36.642363+0000", 2024-03-11T21:45:52.574 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.00034059699999999999 2024-03-11T21:45:52.574 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:45:52.574 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:45:52.574 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "op_commit", 2024-03-11T21:45:52.574 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:42:36.645218+0000", 2024-03-11T21:45:52.574 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.0028546399999999999 2024-03-11T21:45:52.574 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:45:52.574 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:45:52.575 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "sub_op_commit_rec", 2024-03-11T21:45:52.575 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:42:36.648232+0000", 2024-03-11T21:45:52.575 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.003014255 2024-03-11T21:45:52.575 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:45:52.575 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:45:52.575 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "sub_op_commit_rec", 2024-03-11T21:45:52.575 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:42:36.764601+0000", 2024-03-11T21:45:52.575 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.116369163 2024-03-11T21:45:52.575 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:45:52.575 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:45:52.575 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "commit_sent", 2024-03-11T21:45:52.575 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:42:36.764707+0000", 2024-03-11T21:45:52.575 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.000105682 2024-03-11T21:45:52.575 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:45:52.576 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:45:52.576 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "done", 2024-03-11T21:45:52.576 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:42:36.764767+0000", 2024-03-11T21:45:52.576 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 5.9926000000000002e-05 2024-03-11T21:45:52.576 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-03-11T21:45:52.576 INFO:teuthology.orchestra.run.smithi097.stdout: ] 2024-03-11T21:45:52.576 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-03-11T21:45:52.576 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:45:52.576 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:45:52.576 INFO:teuthology.orchestra.run.smithi097.stdout: "description": "osd_op(client.15744.0:166095 10.10 10:0dd81a90:::benchmark_data_smithi092_161462_object56119:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e412)", 2024-03-11T21:45:52.576 INFO:teuthology.orchestra.run.smithi097.stdout: "initiated_at": "2024-03-11T21:43:28.028828+0000", 2024-03-11T21:45:52.576 INFO:teuthology.orchestra.run.smithi097.stdout: "age": 144.53411901999999, 2024-03-11T21:45:52.576 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.116539198, 2024-03-11T21:45:52.576 INFO:teuthology.orchestra.run.smithi097.stdout: "type_data": { 2024-03-11T21:45:52.576 INFO:teuthology.orchestra.run.smithi097.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-11T21:45:52.577 INFO:teuthology.orchestra.run.smithi097.stdout: "client_info": { 2024-03-11T21:45:52.577 INFO:teuthology.orchestra.run.smithi097.stdout: "client": "client.15744", 2024-03-11T21:45:52.577 INFO:teuthology.orchestra.run.smithi097.stdout: "client_addr": "v1:172.21.15.92:0/3334000812", 2024-03-11T21:45:52.577 INFO:teuthology.orchestra.run.smithi097.stdout: "tid": 166095 2024-03-11T21:45:52.577 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:45:52.577 INFO:teuthology.orchestra.run.smithi097.stdout: "events": [ 2024-03-11T21:45:52.577 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:45:52.577 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "initiated", 2024-03-11T21:45:52.577 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:43:28.028828+0000", 2024-03-11T21:45:52.577 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-03-11T21:45:52.577 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:45:52.577 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:45:52.577 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "header_read", 2024-03-11T21:45:52.577 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:43:28.028828+0000", 2024-03-11T21:45:52.578 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-03-11T21:45:52.578 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:45:52.578 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:45:52.578 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "throttled", 2024-03-11T21:45:52.578 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:43:28.028829+0000", 2024-03-11T21:45:52.578 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.5659999999999999e-06 2024-03-11T21:45:52.578 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:45:52.578 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:45:52.578 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "all_read", 2024-03-11T21:45:52.578 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:43:28.028835+0000", 2024-03-11T21:45:52.578 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 5.7400000000000001e-06 2024-03-11T21:45:52.578 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:45:52.578 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:45:52.578 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "dispatched", 2024-03-11T21:45:52.578 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:43:28.028845+0000", 2024-03-11T21:45:52.579 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.0200999999999999e-05 2024-03-11T21:45:52.579 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:45:52.579 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:45:52.579 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "queued_for_pg", 2024-03-11T21:45:52.579 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:43:28.028853+0000", 2024-03-11T21:45:52.579 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 7.5619999999999998e-06 2024-03-11T21:45:52.579 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:45:52.579 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:45:52.579 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "reached_pg", 2024-03-11T21:45:52.579 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:43:28.105350+0000", 2024-03-11T21:45:52.579 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.076497239999999994 2024-03-11T21:45:52.579 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:45:52.579 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:45:52.579 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "started", 2024-03-11T21:45:52.579 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:43:28.105465+0000", 2024-03-11T21:45:52.580 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.000115223 2024-03-11T21:45:52.580 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:45:52.580 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:45:52.580 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "waiting for subops from 6,7", 2024-03-11T21:45:52.580 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:43:28.105823+0000", 2024-03-11T21:45:52.580 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.00035800900000000001 2024-03-11T21:45:52.580 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:45:52.580 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:45:52.580 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "op_commit", 2024-03-11T21:45:52.580 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:43:28.107963+0000", 2024-03-11T21:45:52.580 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.002140229 2024-03-11T21:45:52.580 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:45:52.580 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:45:52.581 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "sub_op_commit_rec", 2024-03-11T21:45:52.581 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:43:28.115957+0000", 2024-03-11T21:45:52.581 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.0079932709999999997 2024-03-11T21:45:52.581 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:45:52.581 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:45:52.581 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "sub_op_commit_rec", 2024-03-11T21:45:52.581 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:43:28.145210+0000", 2024-03-11T21:45:52.581 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.029253126000000001 2024-03-11T21:45:52.581 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:45:52.581 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:45:52.581 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "commit_sent", 2024-03-11T21:45:52.581 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:43:28.145299+0000", 2024-03-11T21:45:52.581 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 8.9135000000000006e-05 2024-03-11T21:45:52.581 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:45:52.582 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:45:52.582 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "done", 2024-03-11T21:45:52.582 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:43:28.145367+0000", 2024-03-11T21:45:52.582 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 6.7896000000000002e-05 2024-03-11T21:45:52.582 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-03-11T21:45:52.582 INFO:teuthology.orchestra.run.smithi097.stdout: ] 2024-03-11T21:45:52.582 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-03-11T21:45:52.582 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:45:52.582 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:45:52.583 INFO:teuthology.orchestra.run.smithi097.stdout: "description": "osd_op(client.15744.0:166470 10.10 10:0a859604:::benchmark_data_smithi092_161462_object56494:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e412)", 2024-03-11T21:45:52.583 INFO:teuthology.orchestra.run.smithi097.stdout: "initiated_at": "2024-03-11T21:43:28.545582+0000", 2024-03-11T21:45:52.583 INFO:teuthology.orchestra.run.smithi097.stdout: "age": 144.017364456, 2024-03-11T21:45:52.583 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.117852763, 2024-03-11T21:45:52.583 INFO:teuthology.orchestra.run.smithi097.stdout: "type_data": { 2024-03-11T21:45:52.583 INFO:teuthology.orchestra.run.smithi097.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-11T21:45:52.584 INFO:teuthology.orchestra.run.smithi097.stdout: "client_info": { 2024-03-11T21:45:52.584 INFO:teuthology.orchestra.run.smithi097.stdout: "client": "client.15744", 2024-03-11T21:45:52.584 INFO:teuthology.orchestra.run.smithi097.stdout: "client_addr": "v1:172.21.15.92:0/3334000812", 2024-03-11T21:45:52.584 INFO:teuthology.orchestra.run.smithi097.stdout: "tid": 166470 2024-03-11T21:45:52.584 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:45:52.584 INFO:teuthology.orchestra.run.smithi097.stdout: "events": [ 2024-03-11T21:45:52.584 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:45:52.584 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "initiated", 2024-03-11T21:45:52.584 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:43:28.545582+0000", 2024-03-11T21:45:52.584 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-03-11T21:45:52.584 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:45:52.584 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:45:52.584 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "header_read", 2024-03-11T21:45:52.584 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:43:28.545582+0000", 2024-03-11T21:45:52.585 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-03-11T21:45:52.585 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:45:52.585 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:45:52.585 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "throttled", 2024-03-11T21:45:52.585 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:43:28.545584+0000", 2024-03-11T21:45:52.585 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.7099999999999999e-06 2024-03-11T21:45:52.585 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:45:52.585 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:45:52.585 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "all_read", 2024-03-11T21:45:52.585 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:43:28.545589+0000", 2024-03-11T21:45:52.585 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 5.2830000000000001e-06 2024-03-11T21:45:52.585 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:45:52.585 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:45:52.585 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "dispatched", 2024-03-11T21:45:52.586 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:43:28.545599+0000", 2024-03-11T21:45:52.586 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 9.9969999999999993e-06 2024-03-11T21:45:52.586 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:45:52.586 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:45:52.586 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "queued_for_pg", 2024-03-11T21:45:52.586 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:43:28.545606+0000", 2024-03-11T21:45:52.586 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 7.1790000000000002e-06 2024-03-11T21:45:52.586 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:45:52.586 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:45:52.586 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "reached_pg", 2024-03-11T21:45:52.586 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:43:28.611373+0000", 2024-03-11T21:45:52.586 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.065766609000000004 2024-03-11T21:45:52.586 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:45:52.586 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:45:52.587 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "started", 2024-03-11T21:45:52.587 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:43:28.611479+0000", 2024-03-11T21:45:52.587 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.000106086 2024-03-11T21:45:52.587 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:45:52.587 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:45:52.587 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "waiting for subops from 6,7", 2024-03-11T21:45:52.587 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:43:28.611683+0000", 2024-03-11T21:45:52.587 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.000203857 2024-03-11T21:45:52.587 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:45:52.587 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:45:52.587 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "op_commit", 2024-03-11T21:45:52.587 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:43:28.624789+0000", 2024-03-11T21:45:52.587 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.013105818999999999 2024-03-11T21:45:52.587 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:45:52.588 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:45:52.588 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "sub_op_commit_rec", 2024-03-11T21:45:52.588 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:43:28.628029+0000", 2024-03-11T21:45:52.588 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.0032404140000000001 2024-03-11T21:45:52.588 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:45:52.588 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:45:52.588 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "sub_op_commit_rec", 2024-03-11T21:45:52.588 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:43:28.663284+0000", 2024-03-11T21:45:52.588 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.035255226000000001 2024-03-11T21:45:52.588 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:45:52.588 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:45:52.588 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "commit_sent", 2024-03-11T21:45:52.588 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:43:28.663351+0000", 2024-03-11T21:45:52.589 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 6.7024000000000001e-05 2024-03-11T21:45:52.589 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:45:52.589 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:45:52.589 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "done", 2024-03-11T21:45:52.589 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:43:28.663435+0000", 2024-03-11T21:45:52.589 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 8.3559000000000004e-05 2024-03-11T21:45:52.589 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-03-11T21:45:52.589 INFO:teuthology.orchestra.run.smithi097.stdout: ] 2024-03-11T21:45:52.589 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-03-11T21:45:52.589 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:45:52.589 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:45:52.589 INFO:teuthology.orchestra.run.smithi097.stdout: "description": "osd_op(client.15744.0:166477 10.10 10:0fe276f5:::benchmark_data_smithi092_161462_object56501:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e412)", 2024-03-11T21:45:52.589 INFO:teuthology.orchestra.run.smithi097.stdout: "initiated_at": "2024-03-11T21:43:28.546600+0000", 2024-03-11T21:45:52.589 INFO:teuthology.orchestra.run.smithi097.stdout: "age": 144.016346865, 2024-03-11T21:45:52.589 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.11697613900000001, 2024-03-11T21:45:52.590 INFO:teuthology.orchestra.run.smithi097.stdout: "type_data": { 2024-03-11T21:45:52.590 INFO:teuthology.orchestra.run.smithi097.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-11T21:45:52.590 INFO:teuthology.orchestra.run.smithi097.stdout: "client_info": { 2024-03-11T21:45:52.590 INFO:teuthology.orchestra.run.smithi097.stdout: "client": "client.15744", 2024-03-11T21:45:52.590 INFO:teuthology.orchestra.run.smithi097.stdout: "client_addr": "v1:172.21.15.92:0/3334000812", 2024-03-11T21:45:52.590 INFO:teuthology.orchestra.run.smithi097.stdout: "tid": 166477 2024-03-11T21:45:52.590 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:45:52.590 INFO:teuthology.orchestra.run.smithi097.stdout: "events": [ 2024-03-11T21:45:52.590 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:45:52.590 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "initiated", 2024-03-11T21:45:52.590 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:43:28.546600+0000", 2024-03-11T21:45:52.590 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-03-11T21:45:52.590 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:45:52.591 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:45:52.591 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "header_read", 2024-03-11T21:45:52.591 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:43:28.546600+0000", 2024-03-11T21:45:52.591 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-03-11T21:45:52.591 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:45:52.591 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:45:52.591 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "throttled", 2024-03-11T21:45:52.591 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:43:28.546601+0000", 2024-03-11T21:45:52.591 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.429e-06 2024-03-11T21:45:52.591 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:45:52.591 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:45:52.591 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "all_read", 2024-03-11T21:45:52.591 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:43:28.546611+0000", 2024-03-11T21:45:52.591 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 9.5060000000000001e-06 2024-03-11T21:45:52.592 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:45:52.592 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:45:52.592 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "dispatched", 2024-03-11T21:45:52.592 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:43:28.546621+0000", 2024-03-11T21:45:52.592 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.0669000000000001e-05 2024-03-11T21:45:52.592 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:45:52.592 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:45:52.592 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "queued_for_pg", 2024-03-11T21:45:52.592 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:43:28.546629+0000", 2024-03-11T21:45:52.592 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 7.6059999999999997e-06 2024-03-11T21:45:52.592 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:45:52.592 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:45:52.592 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "reached_pg", 2024-03-11T21:45:52.592 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:43:28.617805+0000", 2024-03-11T21:45:52.593 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.071175955999999999 2024-03-11T21:45:52.593 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:45:52.593 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:45:52.593 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "started", 2024-03-11T21:45:52.593 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:43:28.617927+0000", 2024-03-11T21:45:52.593 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.00012186200000000001 2024-03-11T21:45:52.593 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:45:52.593 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:45:52.593 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "waiting for subops from 6,7", 2024-03-11T21:45:52.593 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:43:28.618403+0000", 2024-03-11T21:45:52.593 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.000475802 2024-03-11T21:45:52.593 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:45:52.593 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:45:52.593 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "op_commit", 2024-03-11T21:45:52.593 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:43:28.624902+0000", 2024-03-11T21:45:52.594 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.0064997500000000003 2024-03-11T21:45:52.594 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:45:52.594 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:45:52.594 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "sub_op_commit_rec", 2024-03-11T21:45:52.594 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:43:28.628066+0000", 2024-03-11T21:45:52.594 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.003163796 2024-03-11T21:45:52.594 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:45:52.594 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:45:52.594 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "sub_op_commit_rec", 2024-03-11T21:45:52.594 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:43:28.663476+0000", 2024-03-11T21:45:52.594 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.035410204000000001 2024-03-11T21:45:52.594 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:45:52.594 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:45:52.594 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "commit_sent", 2024-03-11T21:45:52.595 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:43:28.663531+0000", 2024-03-11T21:45:52.595 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 5.4675000000000001e-05 2024-03-11T21:45:52.595 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:45:52.595 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:45:52.595 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "done", 2024-03-11T21:45:52.595 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:43:28.663576+0000", 2024-03-11T21:45:52.595 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 4.4883999999999999e-05 2024-03-11T21:45:52.595 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-03-11T21:45:52.595 INFO:teuthology.orchestra.run.smithi097.stdout: ] 2024-03-11T21:45:52.595 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-03-11T21:45:52.595 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:45:52.595 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:45:52.595 INFO:teuthology.orchestra.run.smithi097.stdout: "description": "osd_op(client.15744.0:167508 10.10 10:0ab0c736:::benchmark_data_smithi092_161462_object57532:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e412)", 2024-03-11T21:45:52.595 INFO:teuthology.orchestra.run.smithi097.stdout: "initiated_at": "2024-03-11T21:43:29.740748+0000", 2024-03-11T21:45:52.595 INFO:teuthology.orchestra.run.smithi097.stdout: "age": 142.82219869799999, 2024-03-11T21:45:52.596 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.114395159, 2024-03-11T21:45:52.596 INFO:teuthology.orchestra.run.smithi097.stdout: "type_data": { 2024-03-11T21:45:52.596 INFO:teuthology.orchestra.run.smithi097.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-11T21:45:52.596 INFO:teuthology.orchestra.run.smithi097.stdout: "client_info": { 2024-03-11T21:45:52.596 INFO:teuthology.orchestra.run.smithi097.stdout: "client": "client.15744", 2024-03-11T21:45:52.596 INFO:teuthology.orchestra.run.smithi097.stdout: "client_addr": "v1:172.21.15.92:0/3334000812", 2024-03-11T21:45:52.596 INFO:teuthology.orchestra.run.smithi097.stdout: "tid": 167508 2024-03-11T21:45:52.596 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:45:52.596 INFO:teuthology.orchestra.run.smithi097.stdout: "events": [ 2024-03-11T21:45:52.596 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:45:52.596 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "initiated", 2024-03-11T21:45:52.596 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:43:29.740748+0000", 2024-03-11T21:45:52.596 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-03-11T21:45:52.596 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:45:52.597 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:45:52.597 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "header_read", 2024-03-11T21:45:52.597 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:43:29.740748+0000", 2024-03-11T21:45:52.597 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-03-11T21:45:52.597 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:45:52.597 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:45:52.597 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "throttled", 2024-03-11T21:45:52.597 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:43:29.740749+0000", 2024-03-11T21:45:52.597 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.001e-06 2024-03-11T21:45:52.597 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:45:52.597 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:45:52.597 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "all_read", 2024-03-11T21:45:52.597 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:43:29.740753+0000", 2024-03-11T21:45:52.597 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 4.1409999999999998e-06 2024-03-11T21:45:52.598 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:45:52.598 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:45:52.598 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "dispatched", 2024-03-11T21:45:52.598 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:43:29.740770+0000", 2024-03-11T21:45:52.598 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.6799e-05 2024-03-11T21:45:52.598 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:45:52.598 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:45:52.598 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "queued_for_pg", 2024-03-11T21:45:52.598 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:43:29.740778+0000", 2024-03-11T21:45:52.598 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 7.7640000000000005e-06 2024-03-11T21:45:52.598 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:45:52.598 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:45:52.598 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "reached_pg", 2024-03-11T21:45:52.598 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:43:29.815270+0000", 2024-03-11T21:45:52.599 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.074492244999999999 2024-03-11T21:45:52.599 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:45:52.599 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:45:52.599 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "started", 2024-03-11T21:45:52.599 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:43:29.815380+0000", 2024-03-11T21:45:52.599 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.000109696 2024-03-11T21:45:52.599 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:45:52.599 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:45:52.599 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "waiting for subops from 6,7", 2024-03-11T21:45:52.599 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:43:29.815735+0000", 2024-03-11T21:45:52.599 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.00035573499999999998 2024-03-11T21:45:52.599 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:45:52.599 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:45:52.599 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "op_commit", 2024-03-11T21:45:52.599 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:43:29.817863+0000", 2024-03-11T21:45:52.600 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.0021276369999999999 2024-03-11T21:45:52.600 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:45:52.600 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:45:52.600 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "sub_op_commit_rec", 2024-03-11T21:45:52.600 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:43:29.827837+0000", 2024-03-11T21:45:52.600 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.0099740179999999994 2024-03-11T21:45:52.600 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:45:52.600 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:45:52.600 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "sub_op_commit_rec", 2024-03-11T21:45:52.600 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:43:29.854796+0000", 2024-03-11T21:45:52.600 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.026959278 2024-03-11T21:45:52.600 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:45:52.600 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:45:52.600 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "commit_sent", 2024-03-11T21:45:52.600 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:43:29.854982+0000", 2024-03-11T21:45:52.601 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.00018553800000000001 2024-03-11T21:45:52.601 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:45:52.601 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:45:52.601 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "done", 2024-03-11T21:45:52.601 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:43:29.855143+0000", 2024-03-11T21:45:52.601 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.00016130699999999999 2024-03-11T21:45:52.601 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-03-11T21:45:52.601 INFO:teuthology.orchestra.run.smithi097.stdout: ] 2024-03-11T21:45:52.601 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-03-11T21:45:52.601 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:45:52.601 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:45:52.601 INFO:teuthology.orchestra.run.smithi097.stdout: "description": "osd_op(client.15744.0:167609 10.10 10:0a38f69c:::benchmark_data_smithi092_161462_object57633:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e412)", 2024-03-11T21:45:52.601 INFO:teuthology.orchestra.run.smithi097.stdout: "initiated_at": "2024-03-11T21:43:29.856636+0000", 2024-03-11T21:45:52.601 INFO:teuthology.orchestra.run.smithi097.stdout: "age": 142.70631047399999, 2024-03-11T21:45:52.602 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.112643593, 2024-03-11T21:45:52.602 INFO:teuthology.orchestra.run.smithi097.stdout: "type_data": { 2024-03-11T21:45:52.602 INFO:teuthology.orchestra.run.smithi097.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-11T21:45:52.602 INFO:teuthology.orchestra.run.smithi097.stdout: "client_info": { 2024-03-11T21:45:52.602 INFO:teuthology.orchestra.run.smithi097.stdout: "client": "client.15744", 2024-03-11T21:45:52.602 INFO:teuthology.orchestra.run.smithi097.stdout: "client_addr": "v1:172.21.15.92:0/3334000812", 2024-03-11T21:45:52.602 INFO:teuthology.orchestra.run.smithi097.stdout: "tid": 167609 2024-03-11T21:45:52.602 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:45:52.602 INFO:teuthology.orchestra.run.smithi097.stdout: "events": [ 2024-03-11T21:45:52.602 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:45:52.602 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "initiated", 2024-03-11T21:45:52.602 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:43:29.856636+0000", 2024-03-11T21:45:52.602 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-03-11T21:45:52.602 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:45:52.603 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:45:52.603 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "header_read", 2024-03-11T21:45:52.603 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:43:29.856636+0000", 2024-03-11T21:45:52.603 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-03-11T21:45:52.603 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:45:52.603 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:45:52.603 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "throttled", 2024-03-11T21:45:52.603 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:43:29.856637+0000", 2024-03-11T21:45:52.603 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.26e-06 2024-03-11T21:45:52.603 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:45:52.603 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:45:52.603 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "all_read", 2024-03-11T21:45:52.603 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:43:29.856642+0000", 2024-03-11T21:45:52.603 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 4.6110000000000004e-06 2024-03-11T21:45:52.604 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:45:52.604 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:45:52.604 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "dispatched", 2024-03-11T21:45:52.604 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:43:29.856657+0000", 2024-03-11T21:45:52.604 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.5282999999999999e-05 2024-03-11T21:45:52.604 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:45:52.604 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:45:52.604 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "queued_for_pg", 2024-03-11T21:45:52.604 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:43:29.856664+0000", 2024-03-11T21:45:52.604 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 6.9240000000000001e-06 2024-03-11T21:45:52.604 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:45:52.604 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:45:52.604 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "reached_pg", 2024-03-11T21:45:52.604 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:43:29.863965+0000", 2024-03-11T21:45:52.605 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.0073006900000000003 2024-03-11T21:45:52.605 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:45:52.605 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:45:52.605 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "started", 2024-03-11T21:45:52.605 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:43:29.864807+0000", 2024-03-11T21:45:52.605 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.00084223100000000003 2024-03-11T21:45:52.605 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:45:52.605 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:45:52.605 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "waiting for subops from 6,7", 2024-03-11T21:45:52.605 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:43:29.865470+0000", 2024-03-11T21:45:52.605 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.00066290499999999998 2024-03-11T21:45:52.605 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:45:52.605 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:45:52.605 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "sub_op_commit_rec", 2024-03-11T21:45:52.605 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:43:29.871990+0000", 2024-03-11T21:45:52.606 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.0065194529999999997 2024-03-11T21:45:52.606 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:45:52.606 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:45:52.606 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "op_commit", 2024-03-11T21:45:52.606 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:43:29.900561+0000", 2024-03-11T21:45:52.606 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.028571602000000001 2024-03-11T21:45:52.606 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:45:52.606 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:45:52.606 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "sub_op_commit_rec", 2024-03-11T21:45:52.606 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:43:29.969130+0000", 2024-03-11T21:45:52.606 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.068568585000000001 2024-03-11T21:45:52.606 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:45:52.606 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:45:52.606 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "commit_sent", 2024-03-11T21:45:52.606 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:43:29.969217+0000", 2024-03-11T21:45:52.607 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 8.7251999999999997e-05 2024-03-11T21:45:52.607 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:45:52.607 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:45:52.607 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "done", 2024-03-11T21:45:52.607 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:43:29.969280+0000", 2024-03-11T21:45:52.607 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 6.2797000000000002e-05 2024-03-11T21:45:52.607 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-03-11T21:45:52.607 INFO:teuthology.orchestra.run.smithi097.stdout: ] 2024-03-11T21:45:52.607 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-03-11T21:45:52.607 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:45:52.607 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:45:52.607 INFO:teuthology.orchestra.run.smithi097.stdout: "description": "osd_op(client.15744.0:168249 10.10 10:08f9f7a3:::benchmark_data_smithi092_161462_object58273:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e412)", 2024-03-11T21:45:52.607 INFO:teuthology.orchestra.run.smithi097.stdout: "initiated_at": "2024-03-11T21:43:30.578200+0000", 2024-03-11T21:45:52.607 INFO:teuthology.orchestra.run.smithi097.stdout: "age": 141.98474633800001, 2024-03-11T21:45:52.607 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.116707932, 2024-03-11T21:45:52.608 INFO:teuthology.orchestra.run.smithi097.stdout: "type_data": { 2024-03-11T21:45:52.608 INFO:teuthology.orchestra.run.smithi097.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-11T21:45:52.608 INFO:teuthology.orchestra.run.smithi097.stdout: "client_info": { 2024-03-11T21:45:52.608 INFO:teuthology.orchestra.run.smithi097.stdout: "client": "client.15744", 2024-03-11T21:45:52.608 INFO:teuthology.orchestra.run.smithi097.stdout: "client_addr": "v1:172.21.15.92:0/3334000812", 2024-03-11T21:45:52.608 INFO:teuthology.orchestra.run.smithi097.stdout: "tid": 168249 2024-03-11T21:45:52.608 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:45:52.608 INFO:teuthology.orchestra.run.smithi097.stdout: "events": [ 2024-03-11T21:45:52.608 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:45:52.608 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "initiated", 2024-03-11T21:45:52.608 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:43:30.578200+0000", 2024-03-11T21:45:52.608 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-03-11T21:45:52.608 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:45:52.608 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:45:52.609 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "header_read", 2024-03-11T21:45:52.609 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:43:30.578200+0000", 2024-03-11T21:45:52.609 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-03-11T21:45:52.609 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:45:52.609 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:45:52.609 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "throttled", 2024-03-11T21:45:52.609 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:43:30.578202+0000", 2024-03-11T21:45:52.609 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.7209999999999999e-06 2024-03-11T21:45:52.609 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:45:52.609 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:45:52.609 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "all_read", 2024-03-11T21:45:52.609 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:43:30.578206+0000", 2024-03-11T21:45:52.609 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 4.3900000000000003e-06 2024-03-11T21:45:52.609 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:45:52.609 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:45:52.610 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "dispatched", 2024-03-11T21:45:52.610 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:43:30.578218+0000", 2024-03-11T21:45:52.610 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.1303e-05 2024-03-11T21:45:52.610 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:45:52.610 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:45:52.610 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "queued_for_pg", 2024-03-11T21:45:52.610 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:43:30.578225+0000", 2024-03-11T21:45:52.610 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 7.4699999999999996e-06 2024-03-11T21:45:52.610 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:45:52.610 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:45:52.610 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "reached_pg", 2024-03-11T21:45:52.610 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:43:30.659659+0000", 2024-03-11T21:45:52.610 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.081433503000000004 2024-03-11T21:45:52.610 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:45:52.611 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:45:52.611 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "started", 2024-03-11T21:45:52.611 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:43:30.659782+0000", 2024-03-11T21:45:52.611 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.00012370099999999999 2024-03-11T21:45:52.611 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:45:52.611 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:45:52.611 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "waiting for subops from 6,7", 2024-03-11T21:45:52.611 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:43:30.660169+0000", 2024-03-11T21:45:52.611 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.000386268 2024-03-11T21:45:52.611 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:45:52.611 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:45:52.611 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "sub_op_commit_rec", 2024-03-11T21:45:52.611 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:43:30.668159+0000", 2024-03-11T21:45:52.611 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.0079898429999999999 2024-03-11T21:45:52.612 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:45:52.612 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:45:52.612 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "sub_op_commit_rec", 2024-03-11T21:45:52.612 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:43:30.671302+0000", 2024-03-11T21:45:52.612 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.0031431929999999999 2024-03-11T21:45:52.612 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:45:52.612 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:45:52.612 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "op_commit", 2024-03-11T21:45:52.612 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:43:30.694776+0000", 2024-03-11T21:45:52.612 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.023474343000000002 2024-03-11T21:45:52.612 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:45:52.612 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:45:52.612 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "commit_sent", 2024-03-11T21:45:52.612 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:43:30.694835+0000", 2024-03-11T21:45:52.612 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 5.8502000000000003e-05 2024-03-11T21:45:52.613 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:45:52.613 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:45:52.613 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "done", 2024-03-11T21:45:52.613 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:43:30.694908+0000", 2024-03-11T21:45:52.613 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 7.3695000000000005e-05 2024-03-11T21:45:52.613 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-03-11T21:45:52.613 INFO:teuthology.orchestra.run.smithi097.stdout: ] 2024-03-11T21:45:52.613 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-03-11T21:45:52.613 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:45:52.613 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:45:52.613 INFO:teuthology.orchestra.run.smithi097.stdout: "description": "osd_op(client.15744.0:168455 10.10 10:0ac7b8d6:::benchmark_data_smithi092_161462_object58479:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e412)", 2024-03-11T21:45:52.613 INFO:teuthology.orchestra.run.smithi097.stdout: "initiated_at": "2024-03-11T21:43:30.814472+0000", 2024-03-11T21:45:52.613 INFO:teuthology.orchestra.run.smithi097.stdout: "age": 141.74847488, 2024-03-11T21:45:52.613 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.16906712900000001, 2024-03-11T21:45:52.614 INFO:teuthology.orchestra.run.smithi097.stdout: "type_data": { 2024-03-11T21:45:52.614 INFO:teuthology.orchestra.run.smithi097.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-11T21:45:52.614 INFO:teuthology.orchestra.run.smithi097.stdout: "client_info": { 2024-03-11T21:45:52.614 INFO:teuthology.orchestra.run.smithi097.stdout: "client": "client.15744", 2024-03-11T21:45:52.614 INFO:teuthology.orchestra.run.smithi097.stdout: "client_addr": "v1:172.21.15.92:0/3334000812", 2024-03-11T21:45:52.614 INFO:teuthology.orchestra.run.smithi097.stdout: "tid": 168455 2024-03-11T21:45:52.614 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:45:52.614 INFO:teuthology.orchestra.run.smithi097.stdout: "events": [ 2024-03-11T21:45:52.614 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:45:52.614 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "initiated", 2024-03-11T21:45:52.614 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:43:30.814472+0000", 2024-03-11T21:45:52.614 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-03-11T21:45:52.614 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:45:52.614 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:45:52.615 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "header_read", 2024-03-11T21:45:52.615 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:43:30.814472+0000", 2024-03-11T21:45:52.615 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-03-11T21:45:52.615 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:45:52.615 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:45:52.615 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "throttled", 2024-03-11T21:45:52.615 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:43:30.814474+0000", 2024-03-11T21:45:52.615 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.9180000000000001e-06 2024-03-11T21:45:52.615 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:45:52.615 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:45:52.615 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "all_read", 2024-03-11T21:45:52.615 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:43:30.814479+0000", 2024-03-11T21:45:52.615 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 5.2650000000000004e-06 2024-03-11T21:45:52.615 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:45:52.615 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:45:52.615 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "dispatched", 2024-03-11T21:45:52.616 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:43:30.814492+0000", 2024-03-11T21:45:52.616 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.2996999999999999e-05 2024-03-11T21:45:52.616 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:45:52.616 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:45:52.616 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "queued_for_pg", 2024-03-11T21:45:52.616 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:43:30.814500+0000", 2024-03-11T21:45:52.616 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 8.1419999999999993e-06 2024-03-11T21:45:52.616 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:45:52.616 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:45:52.616 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "reached_pg", 2024-03-11T21:45:52.616 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:43:30.872217+0000", 2024-03-11T21:45:52.616 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.057716942 2024-03-11T21:45:52.616 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:45:52.616 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:45:52.617 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "started", 2024-03-11T21:45:52.617 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:43:30.872335+0000", 2024-03-11T21:45:52.617 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.000117736 2024-03-11T21:45:52.617 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:45:52.617 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:45:52.617 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "waiting for subops from 6,7", 2024-03-11T21:45:52.617 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:43:30.872712+0000", 2024-03-11T21:45:52.617 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.00037696799999999999 2024-03-11T21:45:52.617 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:45:52.617 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:45:52.617 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "op_commit", 2024-03-11T21:45:52.617 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:43:30.888330+0000", 2024-03-11T21:45:52.617 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.015618019 2024-03-11T21:45:52.617 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:45:52.618 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:45:52.618 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "sub_op_commit_rec", 2024-03-11T21:45:52.618 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:43:30.888366+0000", 2024-03-11T21:45:52.618 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 3.6591000000000003e-05 2024-03-11T21:45:52.618 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:45:52.618 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:45:52.618 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "sub_op_commit_rec", 2024-03-11T21:45:52.618 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:43:30.983417+0000", 2024-03-11T21:45:52.618 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.095050468999999999 2024-03-11T21:45:52.618 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:45:52.618 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:45:52.618 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "commit_sent", 2024-03-11T21:45:52.618 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:43:30.983472+0000", 2024-03-11T21:45:52.618 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 5.5461999999999999e-05 2024-03-11T21:45:52.619 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:45:52.619 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:45:52.619 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "done", 2024-03-11T21:45:52.619 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:43:30.983539+0000", 2024-03-11T21:45:52.619 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 6.6619999999999996e-05 2024-03-11T21:45:52.619 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-03-11T21:45:52.619 INFO:teuthology.orchestra.run.smithi097.stdout: ] 2024-03-11T21:45:52.619 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-03-11T21:45:52.619 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:45:52.619 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:45:52.619 INFO:teuthology.orchestra.run.smithi097.stdout: "description": "osd_op(client.15744.0:168457 10.10 10:08bf6365:::benchmark_data_smithi092_161462_object58481:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e412)", 2024-03-11T21:45:52.619 INFO:teuthology.orchestra.run.smithi097.stdout: "initiated_at": "2024-03-11T21:43:30.817874+0000", 2024-03-11T21:45:52.619 INFO:teuthology.orchestra.run.smithi097.stdout: "age": 141.745072978, 2024-03-11T21:45:52.619 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.165812404, 2024-03-11T21:45:52.620 INFO:teuthology.orchestra.run.smithi097.stdout: "type_data": { 2024-03-11T21:45:52.620 INFO:teuthology.orchestra.run.smithi097.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-11T21:45:52.620 INFO:teuthology.orchestra.run.smithi097.stdout: "client_info": { 2024-03-11T21:45:52.620 INFO:teuthology.orchestra.run.smithi097.stdout: "client": "client.15744", 2024-03-11T21:45:52.620 INFO:teuthology.orchestra.run.smithi097.stdout: "client_addr": "v1:172.21.15.92:0/3334000812", 2024-03-11T21:45:52.620 INFO:teuthology.orchestra.run.smithi097.stdout: "tid": 168457 2024-03-11T21:45:52.620 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:45:52.620 INFO:teuthology.orchestra.run.smithi097.stdout: "events": [ 2024-03-11T21:45:52.620 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:45:52.620 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "initiated", 2024-03-11T21:45:52.620 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:43:30.817874+0000", 2024-03-11T21:45:52.620 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-03-11T21:45:52.620 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:45:52.620 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:45:52.621 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "header_read", 2024-03-11T21:45:52.621 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:43:30.817874+0000", 2024-03-11T21:45:52.621 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-03-11T21:45:52.621 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:45:52.621 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:45:52.621 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "throttled", 2024-03-11T21:45:52.621 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:43:30.817875+0000", 2024-03-11T21:45:52.621 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.674e-06 2024-03-11T21:45:52.621 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:45:52.621 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:45:52.621 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "all_read", 2024-03-11T21:45:52.621 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:43:30.817881+0000", 2024-03-11T21:45:52.621 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 5.3750000000000002e-06 2024-03-11T21:45:52.621 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:45:52.621 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:45:52.622 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "dispatched", 2024-03-11T21:45:52.622 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:43:30.817894+0000", 2024-03-11T21:45:52.622 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.3239e-05 2024-03-11T21:45:52.622 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:45:52.622 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:45:52.622 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "queued_for_pg", 2024-03-11T21:45:52.622 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:43:30.817902+0000", 2024-03-11T21:45:52.622 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 7.6559999999999992e-06 2024-03-11T21:45:52.622 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:45:52.622 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:45:52.622 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "reached_pg", 2024-03-11T21:45:52.622 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:43:30.878126+0000", 2024-03-11T21:45:52.622 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.060223994000000003 2024-03-11T21:45:52.623 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:45:52.623 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:45:52.623 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "started", 2024-03-11T21:45:52.623 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:43:30.878232+0000", 2024-03-11T21:45:52.623 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.000106672 2024-03-11T21:45:52.623 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:45:52.623 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:45:52.623 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:45:52.624 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "waiting for subops from 6,7", 2024-03-11T21:45:52.624 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:43:30.878581+0000", 2024-03-11T21:45:52.624 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.000349002 2024-03-11T21:45:52.624 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:45:52.625 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:45:52.625 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "op_commit", 2024-03-11T21:45:52.625 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:43:30.888378+0000", 2024-03-11T21:45:52.625 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.0097962039999999993 2024-03-11T21:45:52.625 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:45:52.625 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:45:52.625 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "sub_op_commit_rec", 2024-03-11T21:45:52.625 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:43:30.890979+0000", 2024-03-11T21:45:52.625 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.0026017200000000001 2024-03-11T21:45:52.625 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:45:52.625 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:45:52.625 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "sub_op_commit_rec", 2024-03-11T21:45:52.625 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:43:30.983583+0000", 2024-03-11T21:45:52.625 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.092604231999999995 2024-03-11T21:45:52.626 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:45:52.626 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:45:52.626 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "commit_sent", 2024-03-11T21:45:52.626 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:43:30.983631+0000", 2024-03-11T21:45:52.626 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 4.7299999999999998e-05 2024-03-11T21:45:52.626 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:45:52.626 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:45:52.626 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "done", 2024-03-11T21:45:52.626 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:43:30.983686+0000", 2024-03-11T21:45:52.626 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 5.5336e-05 2024-03-11T21:45:52.626 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-03-11T21:45:52.626 INFO:teuthology.orchestra.run.smithi097.stdout: ] 2024-03-11T21:45:52.626 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-03-11T21:45:52.626 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:45:52.627 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:45:52.627 INFO:teuthology.orchestra.run.smithi097.stdout: "description": "osd_op(client.15744.0:168466 10.10 10:0e4995f9:::benchmark_data_smithi092_161462_object58490:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e412)", 2024-03-11T21:45:52.627 INFO:teuthology.orchestra.run.smithi097.stdout: "initiated_at": "2024-03-11T21:43:30.833613+0000", 2024-03-11T21:45:52.627 INFO:teuthology.orchestra.run.smithi097.stdout: "age": 141.729334064, 2024-03-11T21:45:52.627 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.15028402399999999, 2024-03-11T21:45:52.627 INFO:teuthology.orchestra.run.smithi097.stdout: "type_data": { 2024-03-11T21:45:52.627 INFO:teuthology.orchestra.run.smithi097.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-11T21:45:52.627 INFO:teuthology.orchestra.run.smithi097.stdout: "client_info": { 2024-03-11T21:45:52.627 INFO:teuthology.orchestra.run.smithi097.stdout: "client": "client.15744", 2024-03-11T21:45:52.627 INFO:teuthology.orchestra.run.smithi097.stdout: "client_addr": "v1:172.21.15.92:0/3334000812", 2024-03-11T21:45:52.627 INFO:teuthology.orchestra.run.smithi097.stdout: "tid": 168466 2024-03-11T21:45:52.627 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:45:52.627 INFO:teuthology.orchestra.run.smithi097.stdout: "events": [ 2024-03-11T21:45:52.628 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:45:52.628 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "initiated", 2024-03-11T21:45:52.628 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:43:30.833613+0000", 2024-03-11T21:45:52.628 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-03-11T21:45:52.628 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:45:52.628 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:45:52.628 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "header_read", 2024-03-11T21:45:52.628 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:43:30.833613+0000", 2024-03-11T21:45:52.628 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-03-11T21:45:52.628 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:45:52.628 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:45:52.628 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "throttled", 2024-03-11T21:45:52.628 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:43:30.833615+0000", 2024-03-11T21:45:52.628 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.9250000000000002e-06 2024-03-11T21:45:52.629 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:45:52.629 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:45:52.629 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "all_read", 2024-03-11T21:45:52.629 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:43:30.833624+0000", 2024-03-11T21:45:52.629 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 8.9949999999999994e-06 2024-03-11T21:45:52.629 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:45:52.629 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:45:52.629 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "dispatched", 2024-03-11T21:45:52.629 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:43:30.833637+0000", 2024-03-11T21:45:52.629 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.3185e-05 2024-03-11T21:45:52.629 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:45:52.629 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:45:52.629 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "queued_for_pg", 2024-03-11T21:45:52.629 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:43:30.833645+0000", 2024-03-11T21:45:52.630 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 7.9699999999999999e-06 2024-03-11T21:45:52.630 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:45:52.630 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:45:52.630 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "reached_pg", 2024-03-11T21:45:52.630 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:43:30.880617+0000", 2024-03-11T21:45:52.630 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.046971952999999997 2024-03-11T21:45:52.630 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:45:52.630 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:45:52.630 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "started", 2024-03-11T21:45:52.630 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:43:30.880726+0000", 2024-03-11T21:45:52.630 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.000109648 2024-03-11T21:45:52.630 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:45:52.630 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:45:52.630 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "waiting for subops from 6,7", 2024-03-11T21:45:52.630 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:43:30.880993+0000", 2024-03-11T21:45:52.631 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.00026681799999999999 2024-03-11T21:45:52.631 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:45:52.631 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:45:52.631 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "op_commit", 2024-03-11T21:45:52.631 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:43:30.888383+0000", 2024-03-11T21:45:52.631 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.0073894379999999999 2024-03-11T21:45:52.631 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:45:52.631 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:45:52.631 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "sub_op_commit_rec", 2024-03-11T21:45:52.631 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:43:30.891958+0000", 2024-03-11T21:45:52.631 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.003575801 2024-03-11T21:45:52.631 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:45:52.631 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:45:52.631 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "sub_op_commit_rec", 2024-03-11T21:45:52.632 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:43:30.983800+0000", 2024-03-11T21:45:52.632 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.091841171999999999 2024-03-11T21:45:52.632 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:45:52.632 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:45:52.632 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "commit_sent", 2024-03-11T21:45:52.632 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:43:30.983846+0000", 2024-03-11T21:45:52.632 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 4.6025000000000001e-05 2024-03-11T21:45:52.632 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:45:52.632 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:45:52.632 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "done", 2024-03-11T21:45:52.632 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:43:30.983897+0000", 2024-03-11T21:45:52.632 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 5.1094000000000002e-05 2024-03-11T21:45:52.632 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-03-11T21:45:52.632 INFO:teuthology.orchestra.run.smithi097.stdout: ] 2024-03-11T21:45:52.632 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-03-11T21:45:52.633 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:45:52.633 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:45:52.633 INFO:teuthology.orchestra.run.smithi097.stdout: "description": "osd_op(client.15744.0:168485 10.10 10:0e091f3c:::benchmark_data_smithi092_161462_object58509:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e412)", 2024-03-11T21:45:52.633 INFO:teuthology.orchestra.run.smithi097.stdout: "initiated_at": "2024-03-11T21:43:30.854429+0000", 2024-03-11T21:45:52.633 INFO:teuthology.orchestra.run.smithi097.stdout: "age": 141.708517857, 2024-03-11T21:45:52.633 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.12984559300000001, 2024-03-11T21:45:52.633 INFO:teuthology.orchestra.run.smithi097.stdout: "type_data": { 2024-03-11T21:45:52.633 INFO:teuthology.orchestra.run.smithi097.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-11T21:45:52.633 INFO:teuthology.orchestra.run.smithi097.stdout: "client_info": { 2024-03-11T21:45:52.633 INFO:teuthology.orchestra.run.smithi097.stdout: "client": "client.15744", 2024-03-11T21:45:52.633 INFO:teuthology.orchestra.run.smithi097.stdout: "client_addr": "v1:172.21.15.92:0/3334000812", 2024-03-11T21:45:52.633 INFO:teuthology.orchestra.run.smithi097.stdout: "tid": 168485 2024-03-11T21:45:52.633 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:45:52.633 INFO:teuthology.orchestra.run.smithi097.stdout: "events": [ 2024-03-11T21:45:52.633 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:45:52.634 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "initiated", 2024-03-11T21:45:52.634 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:43:30.854429+0000", 2024-03-11T21:45:52.634 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-03-11T21:45:52.634 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:45:52.634 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:45:52.634 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "header_read", 2024-03-11T21:45:52.634 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:43:30.854429+0000", 2024-03-11T21:45:52.634 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-03-11T21:45:52.634 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:45:52.634 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:45:52.634 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "throttled", 2024-03-11T21:45:52.634 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:43:30.854430+0000", 2024-03-11T21:45:52.634 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.395e-06 2024-03-11T21:45:52.634 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:45:52.634 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:45:52.635 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "all_read", 2024-03-11T21:45:52.635 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:43:30.854435+0000", 2024-03-11T21:45:52.635 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 4.9409999999999998e-06 2024-03-11T21:45:52.635 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:45:52.635 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:45:52.635 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "dispatched", 2024-03-11T21:45:52.635 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:43:30.854445+0000", 2024-03-11T21:45:52.635 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.0045e-05 2024-03-11T21:45:52.635 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:45:52.635 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:45:52.635 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "queued_for_pg", 2024-03-11T21:45:52.635 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:43:30.854453+0000", 2024-03-11T21:45:52.635 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 7.6190000000000002e-06 2024-03-11T21:45:52.635 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:45:52.636 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:45:52.636 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "reached_pg", 2024-03-11T21:45:52.636 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:43:30.883145+0000", 2024-03-11T21:45:52.636 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.028691952 2024-03-11T21:45:52.636 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:45:52.636 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:45:52.636 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "started", 2024-03-11T21:45:52.636 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:43:30.883253+0000", 2024-03-11T21:45:52.636 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.00010772099999999999 2024-03-11T21:45:52.636 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:45:52.636 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:45:52.636 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "waiting for subops from 6,7", 2024-03-11T21:45:52.636 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:43:30.883637+0000", 2024-03-11T21:45:52.636 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.00038442000000000001 2024-03-11T21:45:52.636 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:45:52.637 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:45:52.637 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "op_commit", 2024-03-11T21:45:52.637 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:43:30.888388+0000", 2024-03-11T21:45:52.637 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.0047506570000000001 2024-03-11T21:45:52.637 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:45:52.637 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:45:52.637 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "sub_op_commit_rec", 2024-03-11T21:45:52.637 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:43:30.891993+0000", 2024-03-11T21:45:52.637 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.0036058040000000002 2024-03-11T21:45:52.637 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:45:52.637 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:45:52.637 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "sub_op_commit_rec", 2024-03-11T21:45:52.637 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:43:30.984183+0000", 2024-03-11T21:45:52.637 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.092189560000000004 2024-03-11T21:45:52.638 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:45:52.638 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:45:52.638 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "commit_sent", 2024-03-11T21:45:52.638 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:43:30.984227+0000", 2024-03-11T21:45:52.638 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 4.3968000000000003e-05 2024-03-11T21:45:52.638 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:45:52.638 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:45:52.638 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "done", 2024-03-11T21:45:52.638 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:43:30.984274+0000", 2024-03-11T21:45:52.638 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 4.7511e-05 2024-03-11T21:45:52.638 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-03-11T21:45:52.638 INFO:teuthology.orchestra.run.smithi097.stdout: ] 2024-03-11T21:45:52.638 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-03-11T21:45:52.638 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:45:52.638 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:45:52.639 INFO:teuthology.orchestra.run.smithi097.stdout: "description": "osd_op(client.15744.0:168775 10.10 10:0f0b9868:::benchmark_data_smithi092_161462_object58799:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e413)", 2024-03-11T21:45:52.639 INFO:teuthology.orchestra.run.smithi097.stdout: "initiated_at": "2024-03-11T21:43:31.190070+0000", 2024-03-11T21:45:52.639 INFO:teuthology.orchestra.run.smithi097.stdout: "age": 141.37287700799999, 2024-03-11T21:45:52.639 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.127788502, 2024-03-11T21:45:52.639 INFO:teuthology.orchestra.run.smithi097.stdout: "type_data": { 2024-03-11T21:45:52.639 INFO:teuthology.orchestra.run.smithi097.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-11T21:45:52.639 INFO:teuthology.orchestra.run.smithi097.stdout: "client_info": { 2024-03-11T21:45:52.639 INFO:teuthology.orchestra.run.smithi097.stdout: "client": "client.15744", 2024-03-11T21:45:52.639 INFO:teuthology.orchestra.run.smithi097.stdout: "client_addr": "v1:172.21.15.92:0/3334000812", 2024-03-11T21:45:52.639 INFO:teuthology.orchestra.run.smithi097.stdout: "tid": 168775 2024-03-11T21:45:52.639 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:45:52.639 INFO:teuthology.orchestra.run.smithi097.stdout: "events": [ 2024-03-11T21:45:52.639 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:45:52.639 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "initiated", 2024-03-11T21:45:52.639 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:43:31.190070+0000", 2024-03-11T21:45:52.640 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-03-11T21:45:52.640 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:45:52.640 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:45:52.640 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "header_read", 2024-03-11T21:45:52.640 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:43:31.190070+0000", 2024-03-11T21:45:52.640 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-03-11T21:45:52.640 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:45:52.640 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:45:52.640 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "throttled", 2024-03-11T21:45:52.640 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:43:31.190071+0000", 2024-03-11T21:45:52.640 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.7659999999999999e-06 2024-03-11T21:45:52.640 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:45:52.640 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:45:52.640 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "all_read", 2024-03-11T21:45:52.640 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:43:31.190076+0000", 2024-03-11T21:45:52.640 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 4.4769999999999997e-06 2024-03-11T21:45:52.641 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:45:52.641 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:45:52.641 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "dispatched", 2024-03-11T21:45:52.641 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:43:31.190086+0000", 2024-03-11T21:45:52.641 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.0241000000000001e-05 2024-03-11T21:45:52.641 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:45:52.641 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:45:52.641 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "queued_for_pg", 2024-03-11T21:45:52.641 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:43:31.190095+0000", 2024-03-11T21:45:52.641 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 8.5189999999999992e-06 2024-03-11T21:45:52.641 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:45:52.641 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:45:52.641 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "reached_pg", 2024-03-11T21:45:52.641 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:43:31.266873+0000", 2024-03-11T21:45:52.641 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.076778585999999996 2024-03-11T21:45:52.642 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:45:52.642 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:45:52.642 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "started", 2024-03-11T21:45:52.642 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:43:31.267022+0000", 2024-03-11T21:45:52.642 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.000148858 2024-03-11T21:45:52.642 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:45:52.642 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:45:52.642 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "waiting for subops from 6,7", 2024-03-11T21:45:52.642 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:43:31.267292+0000", 2024-03-11T21:45:52.642 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.00027015900000000001 2024-03-11T21:45:52.642 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:45:52.642 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:45:52.642 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "op_commit", 2024-03-11T21:45:52.642 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:43:31.282627+0000", 2024-03-11T21:45:52.642 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.015334831 2024-03-11T21:45:52.643 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:45:52.643 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:45:52.643 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "sub_op_commit_rec", 2024-03-11T21:45:52.643 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:43:31.286267+0000", 2024-03-11T21:45:52.643 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.003639626 2024-03-11T21:45:52.643 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:45:52.643 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:45:52.643 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "sub_op_commit_rec", 2024-03-11T21:45:52.643 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:43:31.317752+0000", 2024-03-11T21:45:52.643 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.031485338000000002 2024-03-11T21:45:52.643 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:45:52.643 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:45:52.643 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "commit_sent", 2024-03-11T21:45:52.643 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:43:31.317800+0000", 2024-03-11T21:45:52.643 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 4.7783e-05 2024-03-11T21:45:52.643 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:45:52.644 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:45:52.644 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "done", 2024-03-11T21:45:52.644 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:43:31.317858+0000", 2024-03-11T21:45:52.644 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 5.8318e-05 2024-03-11T21:45:52.644 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-03-11T21:45:52.644 INFO:teuthology.orchestra.run.smithi097.stdout: ] 2024-03-11T21:45:52.644 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-03-11T21:45:52.644 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:45:52.644 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:45:52.644 INFO:teuthology.orchestra.run.smithi097.stdout: "description": "osd_op(client.15744.0:170193 10.10 10:0de9636d:::benchmark_data_smithi092_161462_object60217:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e413)", 2024-03-11T21:45:52.644 INFO:teuthology.orchestra.run.smithi097.stdout: "initiated_at": "2024-03-11T21:43:32.812912+0000", 2024-03-11T21:45:52.644 INFO:teuthology.orchestra.run.smithi097.stdout: "age": 139.75003486, 2024-03-11T21:45:52.644 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.11707511399999999, 2024-03-11T21:45:52.644 INFO:teuthology.orchestra.run.smithi097.stdout: "type_data": { 2024-03-11T21:45:52.644 INFO:teuthology.orchestra.run.smithi097.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-11T21:45:52.644 INFO:teuthology.orchestra.run.smithi097.stdout: "client_info": { 2024-03-11T21:45:52.645 INFO:teuthology.orchestra.run.smithi097.stdout: "client": "client.15744", 2024-03-11T21:45:52.645 INFO:teuthology.orchestra.run.smithi097.stdout: "client_addr": "v1:172.21.15.92:0/3334000812", 2024-03-11T21:45:52.645 INFO:teuthology.orchestra.run.smithi097.stdout: "tid": 170193 2024-03-11T21:45:52.645 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:45:52.645 INFO:teuthology.orchestra.run.smithi097.stdout: "events": [ 2024-03-11T21:45:52.645 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:45:52.645 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "initiated", 2024-03-11T21:45:52.645 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:43:32.812912+0000", 2024-03-11T21:45:52.645 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-03-11T21:45:52.645 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:45:52.645 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:45:52.645 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "header_read", 2024-03-11T21:45:52.645 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:43:32.812912+0000", 2024-03-11T21:45:52.645 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-03-11T21:45:52.645 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:45:52.645 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:45:52.646 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "throttled", 2024-03-11T21:45:52.646 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:43:32.812914+0000", 2024-03-11T21:45:52.646 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.7740000000000001e-06 2024-03-11T21:45:52.646 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:45:52.646 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:45:52.646 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "all_read", 2024-03-11T21:45:52.646 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:43:32.812919+0000", 2024-03-11T21:45:52.646 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 5.0019999999999997e-06 2024-03-11T21:45:52.646 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:45:52.646 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:45:52.646 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "dispatched", 2024-03-11T21:45:52.646 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:43:32.812928+0000", 2024-03-11T21:45:52.646 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 8.9050000000000003e-06 2024-03-11T21:45:52.646 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:45:52.647 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:45:52.647 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "queued_for_pg", 2024-03-11T21:45:52.647 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:43:32.812935+0000", 2024-03-11T21:45:52.647 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 7.092e-06 2024-03-11T21:45:52.647 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:45:52.647 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:45:52.647 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "reached_pg", 2024-03-11T21:45:52.647 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:43:32.828122+0000", 2024-03-11T21:45:52.647 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.015187256 2024-03-11T21:45:52.647 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:45:52.647 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:45:52.647 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "started", 2024-03-11T21:45:52.647 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:43:32.828238+0000", 2024-03-11T21:45:52.647 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.000115807 2024-03-11T21:45:52.648 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:45:52.648 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:45:52.648 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "waiting for subops from 6,7", 2024-03-11T21:45:52.648 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:43:32.828632+0000", 2024-03-11T21:45:52.648 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.00039436499999999998 2024-03-11T21:45:52.648 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:45:52.648 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:45:52.648 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "op_commit", 2024-03-11T21:45:52.648 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:43:32.832176+0000", 2024-03-11T21:45:52.648 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.0035439740000000001 2024-03-11T21:45:52.648 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:45:52.648 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:45:52.648 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "sub_op_commit_rec", 2024-03-11T21:45:52.648 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:43:32.929465+0000", 2024-03-11T21:45:52.648 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.097288702000000005 2024-03-11T21:45:52.649 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:45:52.649 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:45:52.649 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "sub_op_commit_rec", 2024-03-11T21:45:52.649 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:43:32.929633+0000", 2024-03-11T21:45:52.649 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.00016832300000000001 2024-03-11T21:45:52.649 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:45:52.649 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:45:52.649 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "commit_sent", 2024-03-11T21:45:52.649 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:43:32.929832+0000", 2024-03-11T21:45:52.649 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.00019918099999999999 2024-03-11T21:45:52.649 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:45:52.649 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:45:52.649 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "done", 2024-03-11T21:45:52.649 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:43:32.929987+0000", 2024-03-11T21:45:52.649 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.000154733 2024-03-11T21:45:52.650 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-03-11T21:45:52.650 INFO:teuthology.orchestra.run.smithi097.stdout: ] 2024-03-11T21:45:52.650 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-03-11T21:45:52.650 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:45:52.650 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:45:52.650 INFO:teuthology.orchestra.run.smithi097.stdout: "description": "osd_op(client.15744.0:170432 10.10 10:09914d52:::benchmark_data_smithi092_161462_object60456:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e413)", 2024-03-11T21:45:52.650 INFO:teuthology.orchestra.run.smithi097.stdout: "initiated_at": "2024-03-11T21:43:33.063255+0000", 2024-03-11T21:45:52.650 INFO:teuthology.orchestra.run.smithi097.stdout: "age": 139.49969189199999, 2024-03-11T21:45:52.650 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.110706422, 2024-03-11T21:45:52.650 INFO:teuthology.orchestra.run.smithi097.stdout: "type_data": { 2024-03-11T21:45:52.650 INFO:teuthology.orchestra.run.smithi097.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-11T21:45:52.650 INFO:teuthology.orchestra.run.smithi097.stdout: "client_info": { 2024-03-11T21:45:52.650 INFO:teuthology.orchestra.run.smithi097.stdout: "client": "client.15744", 2024-03-11T21:45:52.650 INFO:teuthology.orchestra.run.smithi097.stdout: "client_addr": "v1:172.21.15.92:0/3334000812", 2024-03-11T21:45:52.651 INFO:teuthology.orchestra.run.smithi097.stdout: "tid": 170432 2024-03-11T21:45:52.651 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:45:52.651 INFO:teuthology.orchestra.run.smithi097.stdout: "events": [ 2024-03-11T21:45:52.651 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:45:52.651 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "initiated", 2024-03-11T21:45:52.651 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:43:33.063255+0000", 2024-03-11T21:45:52.651 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-03-11T21:45:52.651 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:45:52.651 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:45:52.651 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "header_read", 2024-03-11T21:45:52.651 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:43:33.063255+0000", 2024-03-11T21:45:52.651 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-03-11T21:45:52.651 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:45:52.651 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:45:52.652 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "throttled", 2024-03-11T21:45:52.652 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:43:33.063257+0000", 2024-03-11T21:45:52.652 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.7260000000000001e-06 2024-03-11T21:45:52.652 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:45:52.652 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:45:52.652 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "all_read", 2024-03-11T21:45:52.652 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:43:33.063261+0000", 2024-03-11T21:45:52.652 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 4.6600000000000003e-06 2024-03-11T21:45:52.652 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:45:52.652 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:45:52.652 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "dispatched", 2024-03-11T21:45:52.652 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:43:33.063269+0000", 2024-03-11T21:45:52.652 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 7.9880000000000004e-06 2024-03-11T21:45:52.652 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:45:52.653 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:45:52.653 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "queued_for_pg", 2024-03-11T21:45:52.653 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:43:33.063276+0000", 2024-03-11T21:45:52.653 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 6.4579999999999998e-06 2024-03-11T21:45:52.653 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:45:52.653 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:45:52.653 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "reached_pg", 2024-03-11T21:45:52.653 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:43:33.064043+0000", 2024-03-11T21:45:52.653 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.00076738100000000003 2024-03-11T21:45:52.653 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:45:52.653 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:45:52.653 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "started", 2024-03-11T21:45:52.653 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:43:33.064417+0000", 2024-03-11T21:45:52.653 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.00037403499999999999 2024-03-11T21:45:52.653 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:45:52.653 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:45:52.653 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "waiting for subops from 6,7", 2024-03-11T21:45:52.654 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:43:33.065074+0000", 2024-03-11T21:45:52.654 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.00065659799999999999 2024-03-11T21:45:52.654 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:45:52.654 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:45:52.654 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "sub_op_commit_rec", 2024-03-11T21:45:52.654 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:43:33.083017+0000", 2024-03-11T21:45:52.654 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.017943187999999999 2024-03-11T21:45:52.654 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:45:52.654 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:45:52.654 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "op_commit", 2024-03-11T21:45:52.654 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:43:33.086802+0000", 2024-03-11T21:45:52.654 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.003784843 2024-03-11T21:45:52.654 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:45:52.654 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:45:52.654 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "sub_op_commit_rec", 2024-03-11T21:45:52.655 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:43:33.173859+0000", 2024-03-11T21:45:52.655 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.087057128999999997 2024-03-11T21:45:52.655 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:45:52.655 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:45:52.655 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "commit_sent", 2024-03-11T21:45:52.655 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:43:33.173905+0000", 2024-03-11T21:45:52.655 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 4.6523000000000002e-05 2024-03-11T21:45:52.655 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:45:52.655 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:45:52.655 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "done", 2024-03-11T21:45:52.655 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:43:33.173961+0000", 2024-03-11T21:45:52.655 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 5.5893000000000002e-05 2024-03-11T21:45:52.655 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-03-11T21:45:52.655 INFO:teuthology.orchestra.run.smithi097.stdout: ] 2024-03-11T21:45:52.656 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-03-11T21:45:52.656 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:45:52.656 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:45:52.656 INFO:teuthology.orchestra.run.smithi097.stdout: "description": "osd_op(client.15744.0:170597 10.10 10:0c0c80f7:::benchmark_data_smithi092_161462_object60621:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e413)", 2024-03-11T21:45:52.656 INFO:teuthology.orchestra.run.smithi097.stdout: "initiated_at": "2024-03-11T21:43:33.250794+0000", 2024-03-11T21:45:52.656 INFO:teuthology.orchestra.run.smithi097.stdout: "age": 139.31215226500001, 2024-03-11T21:45:52.656 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.11162490799999999, 2024-03-11T21:45:52.656 INFO:teuthology.orchestra.run.smithi097.stdout: "type_data": { 2024-03-11T21:45:52.656 INFO:teuthology.orchestra.run.smithi097.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-11T21:45:52.656 INFO:teuthology.orchestra.run.smithi097.stdout: "client_info": { 2024-03-11T21:45:52.656 INFO:teuthology.orchestra.run.smithi097.stdout: "client": "client.15744", 2024-03-11T21:45:52.656 INFO:teuthology.orchestra.run.smithi097.stdout: "client_addr": "v1:172.21.15.92:0/3334000812", 2024-03-11T21:45:52.656 INFO:teuthology.orchestra.run.smithi097.stdout: "tid": 170597 2024-03-11T21:45:52.656 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:45:52.657 INFO:teuthology.orchestra.run.smithi097.stdout: "events": [ 2024-03-11T21:45:52.657 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:45:52.657 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "initiated", 2024-03-11T21:45:52.657 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:43:33.250794+0000", 2024-03-11T21:45:52.657 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-03-11T21:45:52.657 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:45:52.657 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:45:52.657 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "header_read", 2024-03-11T21:45:52.657 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:43:33.250794+0000", 2024-03-11T21:45:52.657 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-03-11T21:45:52.657 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:45:52.657 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:45:52.657 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "throttled", 2024-03-11T21:45:52.657 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:43:33.250796+0000", 2024-03-11T21:45:52.658 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.7460000000000001e-06 2024-03-11T21:45:52.658 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:45:52.658 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:45:52.658 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "all_read", 2024-03-11T21:45:52.658 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:43:33.250801+0000", 2024-03-11T21:45:52.658 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 4.9289999999999997e-06 2024-03-11T21:45:52.658 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:45:52.658 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:45:52.658 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "dispatched", 2024-03-11T21:45:52.658 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:43:33.250814+0000", 2024-03-11T21:45:52.658 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.2812e-05 2024-03-11T21:45:52.658 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:45:52.658 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:45:52.658 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "queued_for_pg", 2024-03-11T21:45:52.659 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:43:33.250822+0000", 2024-03-11T21:45:52.659 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 7.7030000000000006e-06 2024-03-11T21:45:52.659 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:45:52.659 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:45:52.659 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "reached_pg", 2024-03-11T21:45:52.659 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:43:33.333823+0000", 2024-03-11T21:45:52.659 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.083001349000000002 2024-03-11T21:45:52.659 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:45:52.659 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:45:52.659 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "started", 2024-03-11T21:45:52.659 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:43:33.333965+0000", 2024-03-11T21:45:52.659 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.00014242399999999999 2024-03-11T21:45:52.659 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:45:52.659 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:45:52.659 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "waiting for subops from 6,7", 2024-03-11T21:45:52.660 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:43:33.334349+0000", 2024-03-11T21:45:52.660 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.00038314700000000002 2024-03-11T21:45:52.660 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:45:52.660 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:45:52.660 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "sub_op_commit_rec", 2024-03-11T21:45:52.660 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:43:33.340134+0000", 2024-03-11T21:45:52.660 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.00578504 2024-03-11T21:45:52.660 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:45:52.660 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:45:52.660 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "op_commit", 2024-03-11T21:45:52.660 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:43:33.344266+0000", 2024-03-11T21:45:52.660 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.0041321250000000004 2024-03-11T21:45:52.660 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:45:52.660 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:45:52.660 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "sub_op_commit_rec", 2024-03-11T21:45:52.661 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:43:33.362285+0000", 2024-03-11T21:45:52.661 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.018019215000000002 2024-03-11T21:45:52.661 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:45:52.661 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:45:52.661 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "commit_sent", 2024-03-11T21:45:52.661 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:43:33.362349+0000", 2024-03-11T21:45:52.661 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 6.4364999999999998e-05 2024-03-11T21:45:52.661 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:45:52.661 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:45:52.661 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "done", 2024-03-11T21:45:52.661 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:43:33.362419+0000", 2024-03-11T21:45:52.661 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 7.0053000000000002e-05 2024-03-11T21:45:52.661 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-03-11T21:45:52.661 INFO:teuthology.orchestra.run.smithi097.stdout: ] 2024-03-11T21:45:52.661 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-03-11T21:45:52.662 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:45:52.662 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:45:52.662 INFO:teuthology.orchestra.run.smithi097.stdout: "description": "osd_op(client.15744.0:171737 10.10 10:098cc278:::benchmark_data_smithi092_161462_object61761:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e413)", 2024-03-11T21:45:52.662 INFO:teuthology.orchestra.run.smithi097.stdout: "initiated_at": "2024-03-11T21:43:34.576591+0000", 2024-03-11T21:45:52.662 INFO:teuthology.orchestra.run.smithi097.stdout: "age": 137.98635532, 2024-03-11T21:45:52.662 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.121563038, 2024-03-11T21:45:52.662 INFO:teuthology.orchestra.run.smithi097.stdout: "type_data": { 2024-03-11T21:45:52.662 INFO:teuthology.orchestra.run.smithi097.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-11T21:45:52.662 INFO:teuthology.orchestra.run.smithi097.stdout: "client_info": { 2024-03-11T21:45:52.662 INFO:teuthology.orchestra.run.smithi097.stdout: "client": "client.15744", 2024-03-11T21:45:52.662 INFO:teuthology.orchestra.run.smithi097.stdout: "client_addr": "v1:172.21.15.92:0/3334000812", 2024-03-11T21:45:52.662 INFO:teuthology.orchestra.run.smithi097.stdout: "tid": 171737 2024-03-11T21:45:52.662 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:45:52.662 INFO:teuthology.orchestra.run.smithi097.stdout: "events": [ 2024-03-11T21:45:52.662 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:45:52.662 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "initiated", 2024-03-11T21:45:52.663 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:43:34.576591+0000", 2024-03-11T21:45:52.663 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-03-11T21:45:52.663 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:45:52.663 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:45:52.663 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "header_read", 2024-03-11T21:45:52.663 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:43:34.576591+0000", 2024-03-11T21:45:52.663 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-03-11T21:45:52.663 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:45:52.663 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:45:52.663 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "throttled", 2024-03-11T21:45:52.663 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:43:34.576593+0000", 2024-03-11T21:45:52.663 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.658e-06 2024-03-11T21:45:52.663 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:45:52.663 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:45:52.663 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "all_read", 2024-03-11T21:45:52.664 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:43:34.576598+0000", 2024-03-11T21:45:52.664 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 4.7859999999999996e-06 2024-03-11T21:45:52.664 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:45:52.664 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:45:52.664 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "dispatched", 2024-03-11T21:45:52.664 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:43:34.576607+0000", 2024-03-11T21:45:52.664 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 9.0850000000000003e-06 2024-03-11T21:45:52.664 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:45:52.664 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:45:52.664 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "queued_for_pg", 2024-03-11T21:45:52.664 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:43:34.576614+0000", 2024-03-11T21:45:52.664 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 7.0740000000000004e-06 2024-03-11T21:45:52.664 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:45:52.664 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:45:52.664 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "reached_pg", 2024-03-11T21:45:52.664 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:43:34.596536+0000", 2024-03-11T21:45:52.665 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.019921642999999999 2024-03-11T21:45:52.665 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:45:52.665 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:45:52.665 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "started", 2024-03-11T21:45:52.665 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:43:34.596612+0000", 2024-03-11T21:45:52.665 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 7.6410999999999997e-05 2024-03-11T21:45:52.665 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:45:52.665 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:45:52.665 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "waiting for subops from 6,7", 2024-03-11T21:45:52.665 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:43:34.596687+0000", 2024-03-11T21:45:52.665 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 7.4896999999999998e-05 2024-03-11T21:45:52.665 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:45:52.665 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:45:52.665 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "op_commit", 2024-03-11T21:45:52.666 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:43:34.602042+0000", 2024-03-11T21:45:52.666 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.0053553250000000002 2024-03-11T21:45:52.666 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:45:52.666 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:45:52.666 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "sub_op_commit_rec", 2024-03-11T21:45:52.666 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:43:34.616639+0000", 2024-03-11T21:45:52.666 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.014596444 2024-03-11T21:45:52.666 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:45:52.666 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:45:52.666 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "sub_op_commit_rec", 2024-03-11T21:45:52.666 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:43:34.698024+0000", 2024-03-11T21:45:52.666 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.081385682000000001 2024-03-11T21:45:52.666 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:45:52.666 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:45:52.667 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "commit_sent", 2024-03-11T21:45:52.667 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:43:34.698081+0000", 2024-03-11T21:45:52.667 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 5.6987999999999997e-05 2024-03-11T21:45:52.667 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:45:52.667 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:45:52.667 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "done", 2024-03-11T21:45:52.667 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:43:34.698154+0000", 2024-03-11T21:45:52.667 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 7.3045000000000003e-05 2024-03-11T21:45:52.667 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-03-11T21:45:52.667 INFO:teuthology.orchestra.run.smithi097.stdout: ] 2024-03-11T21:45:52.667 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-03-11T21:45:52.667 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:45:52.667 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:45:52.667 INFO:teuthology.orchestra.run.smithi097.stdout: "description": "osd_op(client.15744.0:171740 10.10 10:08235f9d:::benchmark_data_smithi092_161462_object61764:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e413)", 2024-03-11T21:45:52.668 INFO:teuthology.orchestra.run.smithi097.stdout: "initiated_at": "2024-03-11T21:43:34.579569+0000", 2024-03-11T21:45:52.668 INFO:teuthology.orchestra.run.smithi097.stdout: "age": 137.98337797600001, 2024-03-11T21:45:52.668 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.118767341, 2024-03-11T21:45:52.668 INFO:teuthology.orchestra.run.smithi097.stdout: "type_data": { 2024-03-11T21:45:52.668 INFO:teuthology.orchestra.run.smithi097.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-11T21:45:52.668 INFO:teuthology.orchestra.run.smithi097.stdout: "client_info": { 2024-03-11T21:45:52.668 INFO:teuthology.orchestra.run.smithi097.stdout: "client": "client.15744", 2024-03-11T21:45:52.668 INFO:teuthology.orchestra.run.smithi097.stdout: "client_addr": "v1:172.21.15.92:0/3334000812", 2024-03-11T21:45:52.668 INFO:teuthology.orchestra.run.smithi097.stdout: "tid": 171740 2024-03-11T21:45:52.668 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:45:52.668 INFO:teuthology.orchestra.run.smithi097.stdout: "events": [ 2024-03-11T21:45:52.668 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:45:52.668 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "initiated", 2024-03-11T21:45:52.668 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:43:34.579569+0000", 2024-03-11T21:45:52.669 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-03-11T21:45:52.669 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:45:52.669 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:45:52.669 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "header_read", 2024-03-11T21:45:52.669 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:43:34.579569+0000", 2024-03-11T21:45:52.669 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-03-11T21:45:52.669 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:45:52.669 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:45:52.669 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "throttled", 2024-03-11T21:45:52.669 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:43:34.579570+0000", 2024-03-11T21:45:52.669 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.7370000000000001e-06 2024-03-11T21:45:52.669 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:45:52.669 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:45:52.669 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "all_read", 2024-03-11T21:45:52.669 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:43:34.579576+0000", 2024-03-11T21:45:52.670 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 6.032e-06 2024-03-11T21:45:52.670 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:45:52.670 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:45:52.670 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "dispatched", 2024-03-11T21:45:52.670 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:43:34.579593+0000", 2024-03-11T21:45:52.670 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.6038000000000001e-05 2024-03-11T21:45:52.670 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:45:52.670 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:45:52.670 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "queued_for_pg", 2024-03-11T21:45:52.670 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:43:34.579600+0000", 2024-03-11T21:45:52.670 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 7.8399999999999995e-06 2024-03-11T21:45:52.670 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:45:52.670 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:45:52.670 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "reached_pg", 2024-03-11T21:45:52.670 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:43:34.598974+0000", 2024-03-11T21:45:52.671 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.019373221999999999 2024-03-11T21:45:52.671 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:45:52.671 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:45:52.671 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "started", 2024-03-11T21:45:52.671 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:43:34.599078+0000", 2024-03-11T21:45:52.671 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.000104237 2024-03-11T21:45:52.671 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:45:52.671 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:45:52.671 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "waiting for subops from 6,7", 2024-03-11T21:45:52.671 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:43:34.599217+0000", 2024-03-11T21:45:52.671 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.00013922500000000001 2024-03-11T21:45:52.671 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:45:52.671 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:45:52.671 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "op_commit", 2024-03-11T21:45:52.671 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:43:34.602507+0000", 2024-03-11T21:45:52.672 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.0032898139999999998 2024-03-11T21:45:52.672 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:45:52.672 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:45:52.672 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "sub_op_commit_rec", 2024-03-11T21:45:52.672 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:43:34.697913+0000", 2024-03-11T21:45:52.672 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.095406216000000002 2024-03-11T21:45:52.672 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:45:52.672 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:45:52.672 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "sub_op_commit_rec", 2024-03-11T21:45:52.672 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:43:34.698238+0000", 2024-03-11T21:45:52.672 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.00032525300000000001 2024-03-11T21:45:52.672 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:45:52.672 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:45:52.672 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "commit_sent", 2024-03-11T21:45:52.673 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:43:34.698290+0000", 2024-03-11T21:45:52.673 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 5.1465000000000003e-05 2024-03-11T21:45:52.673 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:45:52.673 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:45:52.673 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "done", 2024-03-11T21:45:52.673 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:43:34.698336+0000", 2024-03-11T21:45:52.673 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 4.6261999999999999e-05 2024-03-11T21:45:52.673 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-03-11T21:45:52.673 INFO:teuthology.orchestra.run.smithi097.stdout: ] 2024-03-11T21:45:52.673 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-03-11T21:45:52.673 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-03-11T21:45:52.673 INFO:teuthology.orchestra.run.smithi097.stdout: ] 2024-03-11T21:45:52.673 INFO:teuthology.orchestra.run.smithi097.stdout:} 2024-03-11T21:45:52.718 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:45:52.720 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:45:52.720 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:45:52.821 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:45:52.853 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:45:52.854 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:45:52.854 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:45:52.955 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:45:53.008 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:45:53.009 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:45:53.009 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:45:53.110 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:45:53.149 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:45:53.150 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:45:53.150 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:45:53.251 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:45:53.257 INFO:teuthology.orchestra.run.smithi092.stderr:Inferring config /var/lib/ceph/00db0016-dfed-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T21:45:53.257 INFO:teuthology.orchestra.run.smithi092.stderr:Inferring config /var/lib/ceph/00db0016-dfed-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T21:45:53.263 INFO:tasks.radosbench.radosbench.0.smithi092.stdout: 46 16 81712 81696 110.975 74.75 0.00635611 0.00900368 2024-03-11T21:45:53.276 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:45:53.277 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:45:53.277 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:45:53.333 INFO:journalctl@ceph.iscsi.iscsi.a.smithi097.stdout:Mar 11 21:45:53 smithi097 ceph-00db0016-dfed-11ee-95c8-87774f69a715-iscsi-iscsi-a[60943]: debug there is no tcmu-runner data available 2024-03-11T21:45:53.334 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:45:53 smithi097 ceph-mon[33317]: pgmap v958: 148 pgs: 148 active+clean; 4.6 GiB data, 18 GiB used, 518 GiB / 536 GiB avail; 41 KiB/s rd, 87 MiB/s wr, 1.46k op/s 2024-03-11T21:45:53.334 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:45:53 smithi097 ceph-mon[33317]: from='mgr.14607 ' entity='mgr.y' 2024-03-11T21:45:53.334 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:45:53 smithi097 ceph-mon[33317]: from='mgr.14607 v1:172.21.15.92:0/1702717597' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T21:45:53.368 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:45:53 smithi092 ceph-mon[37563]: pgmap v958: 148 pgs: 148 active+clean; 4.6 GiB data, 18 GiB used, 518 GiB / 536 GiB avail; 41 KiB/s rd, 87 MiB/s wr, 1.46k op/s 2024-03-11T21:45:53.368 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:45:53 smithi092 ceph-mon[37563]: from='mgr.14607 ' entity='mgr.y' 2024-03-11T21:45:53.368 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:45:53 smithi092 ceph-mon[37563]: from='mgr.14607 v1:172.21.15.92:0/1702717597' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T21:45:53.368 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:45:53 smithi092 ceph-mon[29572]: pgmap v958: 148 pgs: 148 active+clean; 4.6 GiB data, 18 GiB used, 518 GiB / 536 GiB avail; 41 KiB/s rd, 87 MiB/s wr, 1.46k op/s 2024-03-11T21:45:53.368 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:45:53 smithi092 ceph-mon[29572]: from='mgr.14607 ' entity='mgr.y' 2024-03-11T21:45:53.368 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:45:53 smithi092 ceph-mon[29572]: from='mgr.14607 v1:172.21.15.92:0/1702717597' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T21:45:53.379 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:45:53.411 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:45:53.411 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:45:53.412 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:45:53.512 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:45:53.561 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:45:53.562 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:45:53.562 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:45:53.663 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:45:53.695 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:45:53.695 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:45:53.695 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:45:53.719 DEBUG:teuthology.orchestra.run.smithi097:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid 00db0016-dfed-11ee-95c8-87774f69a715 -- ceph daemon osd.5 dump_ops_in_flight 2024-03-11T21:45:53.796 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:45:53.832 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:45:53.833 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:45:53.833 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:45:53.934 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:45:53.975 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:45:53.976 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:45:53.976 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:45:54.078 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:45:54.118 INFO:teuthology.orchestra.run.smithi097.stderr:Inferring config /var/lib/ceph/00db0016-dfed-11ee-95c8-87774f69a715/mon.b/config 2024-03-11T21:45:54.124 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:45:54.125 INFO:teuthology.orchestra.run.smithi092.stderr:nodeep-scrub is set 2024-03-11T21:45:54.125 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:45:54.126 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:45:54.227 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:45:54.263 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:45:54.264 INFO:tasks.radosbench.radosbench.0.smithi092.stdout: 47 16 83008 82992 110.338 81 0.00798799 0.00905872 2024-03-11T21:45:54.264 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:45:54.264 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:45:54.365 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:45:54.367 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:45:54 smithi092 ceph-mon[29572]: from='client.24469 v1:172.21.15.97:0/404208310' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:45:54.368 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:45:54 smithi092 ceph-mon[29572]: from='client.? v1:172.21.15.92:0/2130662431' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-11T21:45:54.368 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:45:54 smithi092 ceph-mon[37563]: from='client.24469 v1:172.21.15.97:0/404208310' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:45:54.368 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:45:54 smithi092 ceph-mon[37563]: from='client.? v1:172.21.15.92:0/2130662431' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-11T21:45:54.400 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:45:54.401 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:45:54.401 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:45:54.415 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:45:54 smithi097 ceph-mon[33317]: from='client.24469 v1:172.21.15.97:0/404208310' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:45:54.415 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:45:54 smithi097 ceph-mon[33317]: from='client.? v1:172.21.15.92:0/2130662431' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-11T21:45:54.502 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:45:54.534 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:45:54.535 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:45:54.535 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:45:54.636 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:45:54.670 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:45:54.670 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:45:54.671 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:45:54.771 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:45:54.809 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:45:54.809 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:45:54.810 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:45:54.911 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:45:54.948 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:45:54.949 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:45:54.949 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:45:55.050 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:45:55.090 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:45:55.090 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:45:55.091 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:45:55.192 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:45:55.239 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:45:55.239 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:45:55.240 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:45:55.264 INFO:tasks.radosbench.radosbench.0.smithi092.stdout: 48 16 84465 84449 109.936 91.0625 0.0117324 0.00909144 2024-03-11T21:45:55.319 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:45:55 smithi092 ceph-mon[29572]: pgmap v959: 148 pgs: 148 active+clean; 4.9 GiB data, 19 GiB used, 518 GiB / 536 GiB avail; 44 KiB/s rd, 79 MiB/s wr, 1.34k op/s 2024-03-11T21:45:55.319 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:45:55 smithi092 ceph-mon[29572]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-11T21:45:55.319 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:45:55 smithi092 ceph-mon[29572]: from='client.? v1:172.21.15.92:0/2130662431' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-03-11T21:45:55.320 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:45:55 smithi092 ceph-mon[29572]: osdmap e458: 8 total, 8 up, 6 in 2024-03-11T21:45:55.320 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:45:55 smithi092 ceph-mon[37563]: pgmap v959: 148 pgs: 148 active+clean; 4.9 GiB data, 19 GiB used, 518 GiB / 536 GiB avail; 44 KiB/s rd, 79 MiB/s wr, 1.34k op/s 2024-03-11T21:45:55.320 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:45:55 smithi092 ceph-mon[37563]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-11T21:45:55.320 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:45:55 smithi092 ceph-mon[37563]: from='client.? v1:172.21.15.92:0/2130662431' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-03-11T21:45:55.320 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:45:55 smithi092 ceph-mon[37563]: osdmap e458: 8 total, 8 up, 6 in 2024-03-11T21:45:55.340 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:45:55.391 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:45:55.392 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:45:55.392 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:45:55.493 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:45:55.525 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:45:55.526 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:45:55.526 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:45:55.535 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:45:55 smithi097 ceph-mon[33317]: pgmap v959: 148 pgs: 148 active+clean; 4.9 GiB data, 19 GiB used, 518 GiB / 536 GiB avail; 44 KiB/s rd, 79 MiB/s wr, 1.34k op/s 2024-03-11T21:45:55.535 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:45:55 smithi097 ceph-mon[33317]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-11T21:45:55.535 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:45:55 smithi097 ceph-mon[33317]: from='client.? v1:172.21.15.92:0/2130662431' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-03-11T21:45:55.535 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:45:55 smithi097 ceph-mon[33317]: osdmap e458: 8 total, 8 up, 6 in 2024-03-11T21:45:55.627 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:45:55.667 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:45:55.667 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:45:55.668 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:45:55.768 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:45:55.797 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:45:55.798 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:45:55.798 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:45:55.899 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:45:55.940 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:45:55.941 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:45:55.942 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:45:56.042 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:45:56.068 INFO:teuthology.orchestra.run.smithi097.stdout:{ 2024-03-11T21:45:56.068 INFO:teuthology.orchestra.run.smithi097.stdout: "ops": [], 2024-03-11T21:45:56.068 INFO:teuthology.orchestra.run.smithi097.stdout: "num_ops": 0 2024-03-11T21:45:56.068 INFO:teuthology.orchestra.run.smithi097.stdout:} 2024-03-11T21:45:56.079 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:45:56.080 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:45:56.080 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:45:56.181 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:45:56.214 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:45:56.214 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:45:56.215 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:45:56.264 INFO:tasks.radosbench.radosbench.0.smithi092.stdout: 49 16 85879 85863 109.496 88.375 0.0167277 0.00912775 2024-03-11T21:45:56.315 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:45:56.352 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:45:56.352 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:45:56.352 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:45:56.453 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:45:56.492 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:45:56.492 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:45:56.493 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:45:56.594 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:45:56.644 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:45:56.644 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:45:56.645 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:45:56.746 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:45:56.788 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:45:56.789 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:45:56.789 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:45:56.890 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:45:56.930 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:45:56.930 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:45:56.931 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:45:57.031 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:45:57.054 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:45:57.055 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:45:57.056 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:45:57.077 DEBUG:teuthology.orchestra.run.smithi092:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid 00db0016-dfed-11ee-95c8-87774f69a715 -- ceph osd unset noscrub 2024-03-11T21:45:57.157 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:45:57.198 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:45:57.198 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:45:57.198 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:45:57.265 INFO:tasks.radosbench.radosbench.0.smithi092.stdout: 50 16 87190 87174 108.944 81.9375 0.00612498 0.00917423 2024-03-11T21:45:57.272 DEBUG:teuthology.orchestra.run.smithi097:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid 00db0016-dfed-11ee-95c8-87774f69a715 -- ceph daemon osd.5 dump_blocked_ops 2024-03-11T21:45:57.300 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:45:57.329 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:45:57.330 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:45:57.330 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:45:57.368 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:45:57 smithi092 ceph-mon[37563]: pgmap v961: 148 pgs: 148 active+clean; 5.0 GiB data, 19 GiB used, 517 GiB / 536 GiB avail; 51 KiB/s rd, 96 MiB/s wr, 1.61k op/s 2024-03-11T21:45:57.368 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:45:57 smithi092 ceph-mon[29572]: pgmap v961: 148 pgs: 148 active+clean; 5.0 GiB data, 19 GiB used, 517 GiB / 536 GiB avail; 51 KiB/s rd, 96 MiB/s wr, 1.61k op/s 2024-03-11T21:45:57.431 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:45:57.473 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:45:57.474 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:45:57.474 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:45:57.575 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:45:57.586 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:45:57 smithi097 ceph-mon[33317]: pgmap v961: 148 pgs: 148 active+clean; 5.0 GiB data, 19 GiB used, 517 GiB / 536 GiB avail; 51 KiB/s rd, 96 MiB/s wr, 1.61k op/s 2024-03-11T21:45:57.625 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:45:57.625 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:45:57.626 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:45:57.639 INFO:teuthology.orchestra.run.smithi097.stderr:Inferring config /var/lib/ceph/00db0016-dfed-11ee-95c8-87774f69a715/mon.b/config 2024-03-11T21:45:57.727 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:45:57.763 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:45:57.764 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:45:57.764 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:45:57.865 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:45:57.893 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:45:57.894 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:45:57.894 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:45:57.995 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:45:58.029 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:45:58.030 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:45:58.030 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:45:58.102 INFO:teuthology.orchestra.run.smithi092.stdout:osd.0: {} 2024-03-11T21:45:58.130 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:45:58.143 INFO:teuthology.orchestra.run.smithi092.stdout:osd.1: {} 2024-03-11T21:45:58.144 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:45:58 smithi092 ceph-mon[29572]: from='client.? v1:172.21.15.92:0/3902750287' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-11T21:45:58.144 INFO:journalctl@ceph.mgr.y.smithi092.stdout:Mar 11 21:45:57 smithi092 ceph-00db0016-dfed-11ee-95c8-87774f69a715-mgr-y[29801]: ::ffff:172.21.15.97 - - [11/Mar/2024:21:45:57] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-11T21:45:58.144 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:45:58 smithi092 ceph-mon[37563]: from='client.? v1:172.21.15.92:0/3902750287' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-11T21:45:58.163 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:45:58.164 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:45:58.165 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:45:58.182 INFO:teuthology.orchestra.run.smithi092.stdout:osd.2: {} 2024-03-11T21:45:58.217 INFO:teuthology.orchestra.run.smithi092.stdout:osd.3: {} 2024-03-11T21:45:58.253 INFO:teuthology.orchestra.run.smithi092.stdout:osd.4: {} 2024-03-11T21:45:58.264 INFO:tasks.radosbench.radosbench.0.smithi092.stdout: 51 15 88454 88439 108.358 79.0625 0.0110659 0.00922448 2024-03-11T21:45:58.266 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:45:58.286 INFO:teuthology.orchestra.run.smithi092.stdout:osd.5: {} 2024-03-11T21:45:58.303 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:45:58.304 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:45:58.305 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:45:58.323 INFO:teuthology.orchestra.run.smithi092.stdout:osd.6: {} 2024-03-11T21:45:58.333 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:45:58 smithi097 ceph-mon[33317]: from='client.? v1:172.21.15.92:0/3902750287' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-11T21:45:58.363 INFO:teuthology.orchestra.run.smithi092.stdout:osd.7: {} 2024-03-11T21:45:58.365 INFO:teuthology.orchestra.run.smithi092.stderr:osd.0: osd_enable_op_tracker = '' 2024-03-11T21:45:58.365 INFO:teuthology.orchestra.run.smithi092.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-11T21:45:58.365 INFO:teuthology.orchestra.run.smithi092.stderr:osd.2: osd_enable_op_tracker = '' 2024-03-11T21:45:58.366 INFO:teuthology.orchestra.run.smithi092.stderr:osd.3: osd_enable_op_tracker = '' 2024-03-11T21:45:58.366 INFO:teuthology.orchestra.run.smithi092.stderr:osd.4: osd_enable_op_tracker = '' 2024-03-11T21:45:58.366 INFO:teuthology.orchestra.run.smithi092.stderr:osd.5: osd_enable_op_tracker = '' 2024-03-11T21:45:58.366 INFO:teuthology.orchestra.run.smithi092.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-11T21:45:58.366 INFO:teuthology.orchestra.run.smithi092.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-11T21:45:58.405 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:45:58.439 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:45:58.440 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:45:58.440 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:45:58.541 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:45:58.571 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:45:58.572 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:45:58.572 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:45:58.673 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:45:58.709 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:45:58.709 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:45:58.709 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:45:58.810 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:45:58.851 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:45:58.852 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:45:58.852 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:45:58.953 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:45:58.992 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:45:58.993 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:45:58.993 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:45:59.095 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:45:59.132 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:45:59.133 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:45:59.134 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:45:59.155 INFO:teuthology.orchestra.run.smithi092.stderr:Inferring config /var/lib/ceph/00db0016-dfed-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T21:45:59.234 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:45:59.264 INFO:tasks.radosbench.radosbench.0.smithi092.stdout: 52 16 89892 89876 108.001 89.8125 0.00888003 0.00925467 2024-03-11T21:45:59.269 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:45:59.270 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:45:59.270 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:45:59.368 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:45:59 smithi092 ceph-mon[29572]: pgmap v962: 148 pgs: 148 active+clean; 5.0 GiB data, 19 GiB used, 517 GiB / 536 GiB avail; 17 KiB/s rd, 55 MiB/s wr, 911 op/s 2024-03-11T21:45:59.368 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:45:59 smithi092 ceph-mon[37563]: pgmap v962: 148 pgs: 148 active+clean; 5.0 GiB data, 19 GiB used, 517 GiB / 536 GiB avail; 17 KiB/s rd, 55 MiB/s wr, 911 op/s 2024-03-11T21:45:59.371 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:45:59.406 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:45:59.407 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:45:59.408 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:45:59.440 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:45:59 smithi097 ceph-mon[33317]: pgmap v962: 148 pgs: 148 active+clean; 5.0 GiB data, 19 GiB used, 517 GiB / 536 GiB avail; 17 KiB/s rd, 55 MiB/s wr, 911 op/s 2024-03-11T21:45:59.509 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:45:59.546 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:45:59.547 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:45:59.547 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:45:59.643 INFO:teuthology.orchestra.run.smithi097.stdout:ERROR: (22) Invalid argument 2024-03-11T21:45:59.662 INFO:teuthology.orchestra.run.smithi097.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-11T21:45:59.662 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:45:59.690 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:45:59.691 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:45:59.691 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:45:59.792 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:45:59.829 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:45:59.830 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:45:59.830 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:45:59.931 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:45:59.970 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:45:59.971 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:45:59.972 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:46:00.073 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:46:00.096 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:46:00.097 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:46:00.097 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:46:00.198 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:46:00.225 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:46:00.226 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:46:00.226 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:46:00.264 INFO:tasks.radosbench.radosbench.0.smithi092.stdout: 53 16 91049 91033 107.328 72.3125 0.0068955 0.00931319 2024-03-11T21:46:00.327 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:46:00.364 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:46:00.365 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:46:00.365 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:46:00.407 DEBUG:teuthology.orchestra.run.smithi097:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid 00db0016-dfed-11ee-95c8-87774f69a715 -- ceph daemon osd.5 dump_historic_ops 2024-03-11T21:46:00.466 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:46:00.520 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:46:00.521 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:46:00.522 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:46:00.623 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:46:00.664 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:46:00.665 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:46:00.665 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:46:00.766 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:46:00.801 INFO:teuthology.orchestra.run.smithi097.stderr:Inferring config /var/lib/ceph/00db0016-dfed-11ee-95c8-87774f69a715/mon.b/config 2024-03-11T21:46:00.807 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:46:00.808 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:46:00.808 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:46:00.909 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:46:00.939 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:46:00.940 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:46:00.940 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:46:01.041 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:46:01.079 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:46:01.080 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:46:01.080 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:46:01.181 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:46:01.216 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:46:01.217 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:46:01.217 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:46:01.249 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:46:01 smithi097 ceph-mon[33317]: pgmap v963: 148 pgs: 148 active+clean; 5.4 GiB data, 20 GiB used, 516 GiB / 536 GiB avail; 14 KiB/s rd, 82 MiB/s wr, 1.33k op/s 2024-03-11T21:46:01.264 INFO:tasks.radosbench.radosbench.0.smithi092.stdout: 54 16 92507 92491 107.028 91.125 0.00666273 0.00933895 2024-03-11T21:46:01.318 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:46:01.353 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:46:01.355 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:46:01.355 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:46:01.368 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:46:01 smithi092 ceph-mon[29572]: pgmap v963: 148 pgs: 148 active+clean; 5.4 GiB data, 20 GiB used, 516 GiB / 536 GiB avail; 14 KiB/s rd, 82 MiB/s wr, 1.33k op/s 2024-03-11T21:46:01.368 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:46:01 smithi092 ceph-mon[37563]: pgmap v963: 148 pgs: 148 active+clean; 5.4 GiB data, 20 GiB used, 516 GiB / 536 GiB avail; 14 KiB/s rd, 82 MiB/s wr, 1.33k op/s 2024-03-11T21:46:01.456 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:46:01.490 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:46:01.491 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:46:01.491 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:46:01.592 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:46:01.627 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:46:01.627 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:46:01.627 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:46:01.729 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:46:01.759 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:46:01.759 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:46:01.760 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:46:01.861 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:46:01.897 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:46:01.898 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:46:01.898 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:46:01.999 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:46:02.029 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:46:02.030 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:46:02.030 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:46:02.131 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:46:02.172 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:46:02.173 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:46:02.173 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:46:02.265 INFO:tasks.radosbench.radosbench.0.smithi092.stdout: 55 16 93946 93930 106.717 89.9375 0.00742347 0.0093659 2024-03-11T21:46:02.274 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:46:02.321 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:46:02.322 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:46:02.323 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:46:02.424 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:46:02.459 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:46:02.460 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:46:02.460 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:46:02.561 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:46:02.590 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:46:02.591 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:46:02.591 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:46:02.692 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:46:02.721 DEBUG:teuthology.orchestra.run.smithi092:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid 00db0016-dfed-11ee-95c8-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-03-11T21:46:02.726 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:46:02.727 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:46:02.727 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:46:02.828 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:46:02.892 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:46:02.892 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:46:02.893 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:46:02.994 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:46:03.024 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:46:03.025 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:46:03.025 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:46:03.126 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:46:03.157 INFO:teuthology.orchestra.run.smithi092.stderr:Inferring config /var/lib/ceph/00db0016-dfed-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T21:46:03.188 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:46:03.189 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:46:03.189 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:46:03.266 INFO:tasks.radosbench.radosbench.0.smithi092.stdout: 56 16 95081 95065 106.076 70.9375 0.013758 0.00942264 2024-03-11T21:46:03.289 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:46:03.323 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:46:03.324 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:46:03.324 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:46:03.403 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:46:03 smithi097 ceph-mon[33317]: pgmap v964: 148 pgs: 148 active+clean; 5.4 GiB data, 21 GiB used, 516 GiB / 536 GiB avail; 6.4 KiB/s rd, 81 MiB/s wr, 1.31k op/s 2024-03-11T21:46:03.404 INFO:journalctl@ceph.iscsi.iscsi.a.smithi097.stdout:Mar 11 21:46:03 smithi097 ceph-00db0016-dfed-11ee-95c8-87774f69a715-iscsi-iscsi-a[60943]: debug there is no tcmu-runner data available 2024-03-11T21:46:03.425 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:46:03.465 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:46:03.466 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:46:03.466 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:46:03.495 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:46:03 smithi092 ceph-mon[29572]: pgmap v964: 148 pgs: 148 active+clean; 5.4 GiB data, 21 GiB used, 516 GiB / 536 GiB avail; 6.4 KiB/s rd, 81 MiB/s wr, 1.31k op/s 2024-03-11T21:46:03.495 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:46:03 smithi092 ceph-mon[37563]: pgmap v964: 148 pgs: 148 active+clean; 5.4 GiB data, 21 GiB used, 516 GiB / 536 GiB avail; 6.4 KiB/s rd, 81 MiB/s wr, 1.31k op/s 2024-03-11T21:46:03.567 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:46:03.601 INFO:teuthology.orchestra.run.smithi097.stdout:ERROR: (22) Invalid argument 2024-03-11T21:46:03.601 INFO:teuthology.orchestra.run.smithi097.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-11T21:46:03.609 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:46:03.610 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:46:03.610 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:46:03.711 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:46:03.744 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:46:03.745 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:46:03.745 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:46:03.846 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:46:03.887 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:46:03.888 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:46:03.888 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:46:03.989 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:46:04.023 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:46:04.024 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:46:04.024 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:46:04.125 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:46:04.132 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 5, 6, 7] out_osds: [3, 4] dead_osds: [] live_osds: [0, 1, 2, 4, 5, 6, 7, 3] 2024-03-11T21:46:04.132 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-03-11T21:46:04.133 INFO:tasks.thrashosds.thrasher:thrash_pg_upmap_items 2024-03-11T21:46:04.133 DEBUG:teuthology.orchestra.run.smithi092:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid 00db0016-dfed-11ee-95c8-87774f69a715 -- ceph osd dump -f json-pretty 2024-03-11T21:46:04.167 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:46:04.168 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:46:04.168 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:46:04.169 INFO:teuthology.orchestra.run.smithi092.stderr:noscrub is unset 2024-03-11T21:46:04.267 INFO:tasks.radosbench.radosbench.0.smithi092.stdout: 57 16 96201 96185 105.443 70 0.0254152 0.00947856 2024-03-11T21:46:04.269 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:46:04.314 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:46:04.315 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:46:04.317 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:46:04.355 DEBUG:teuthology.orchestra.run.smithi097:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid 00db0016-dfed-11ee-95c8-87774f69a715 -- ceph daemon osd.6 dump_ops_in_flight 2024-03-11T21:46:04.417 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:46:04.464 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:46:04.465 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:46:04.465 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:46:04.566 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:46:04.584 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:46:04 smithi097 ceph-mon[33317]: from='client.24469 v1:172.21.15.97:0/404208310' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:46:04.585 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:46:04 smithi097 ceph-mon[33317]: from='mgr.14607 v1:172.21.15.92:0/1702717597' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T21:46:04.585 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:46:04 smithi097 ceph-mon[33317]: from='client.? v1:172.21.15.92:0/4285874803' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-11T21:46:04.617 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:46:04.618 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:46:04 smithi092 ceph-mon[29572]: from='client.24469 v1:172.21.15.97:0/404208310' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:46:04.618 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:46:04 smithi092 ceph-mon[29572]: from='mgr.14607 v1:172.21.15.92:0/1702717597' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T21:46:04.618 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:46:04 smithi092 ceph-mon[29572]: from='client.? v1:172.21.15.92:0/4285874803' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-11T21:46:04.618 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:46:04 smithi092 ceph-mon[37563]: from='client.24469 v1:172.21.15.97:0/404208310' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:46:04.618 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:46:04 smithi092 ceph-mon[37563]: from='mgr.14607 v1:172.21.15.92:0/1702717597' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T21:46:04.619 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:46:04 smithi092 ceph-mon[37563]: from='client.? v1:172.21.15.92:0/4285874803' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-11T21:46:04.619 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:46:04.627 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:46:04.727 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:46:04.729 INFO:teuthology.orchestra.run.smithi097.stderr:Inferring config /var/lib/ceph/00db0016-dfed-11ee-95c8-87774f69a715/mon.b/config 2024-03-11T21:46:04.765 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:46:04.766 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:46:04.766 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:46:04.867 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:46:04.909 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:46:04.909 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:46:04.910 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:46:05.010 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:46:05.041 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:46:05.042 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:46:05.043 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:46:05.144 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:46:05.180 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:46:05.180 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:46:05.180 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:46:05.266 INFO:tasks.radosbench.radosbench.0.smithi092.stdout: 58 16 97523 97507 105.049 82.625 0.0104221 0.00951493 2024-03-11T21:46:05.281 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:46:05.319 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:46:05.319 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:46:05.319 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:46:05.368 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:46:05 smithi092 ceph-mon[29572]: pgmap v965: 148 pgs: 148 active+clean; 5.7 GiB data, 21 GiB used, 515 GiB / 536 GiB avail; 1023 B/s rd, 83 MiB/s wr, 1.34k op/s 2024-03-11T21:46:05.368 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:46:05 smithi092 ceph-mon[29572]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-11T21:46:05.368 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:46:05 smithi092 ceph-mon[29572]: from='client.? v1:172.21.15.92:0/4285874803' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-03-11T21:46:05.368 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:46:05 smithi092 ceph-mon[29572]: osdmap e459: 8 total, 8 up, 6 in 2024-03-11T21:46:05.368 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:46:05 smithi092 ceph-mon[29572]: from='mgr.14607 v1:172.21.15.92:0/1702717597' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T21:46:05.368 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:46:05 smithi092 ceph-mon[29572]: from='mgr.14607 v1:172.21.15.92:0/1702717597' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T21:46:05.368 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:46:05 smithi092 ceph-mon[29572]: from='mgr.14607 ' entity='mgr.y' 2024-03-11T21:46:05.368 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:46:05 smithi092 ceph-mon[37563]: pgmap v965: 148 pgs: 148 active+clean; 5.7 GiB data, 21 GiB used, 515 GiB / 536 GiB avail; 1023 B/s rd, 83 MiB/s wr, 1.34k op/s 2024-03-11T21:46:05.369 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:46:05 smithi092 ceph-mon[37563]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-11T21:46:05.369 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:46:05 smithi092 ceph-mon[37563]: from='client.? v1:172.21.15.92:0/4285874803' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-03-11T21:46:05.369 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:46:05 smithi092 ceph-mon[37563]: osdmap e459: 8 total, 8 up, 6 in 2024-03-11T21:46:05.369 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:46:05 smithi092 ceph-mon[37563]: from='mgr.14607 v1:172.21.15.92:0/1702717597' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T21:46:05.369 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:46:05 smithi092 ceph-mon[37563]: from='mgr.14607 v1:172.21.15.92:0/1702717597' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T21:46:05.369 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:46:05 smithi092 ceph-mon[37563]: from='mgr.14607 ' entity='mgr.y' 2024-03-11T21:46:05.420 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:46:05.471 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:46:05.471 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:46:05.471 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:46:05.534 INFO:teuthology.orchestra.run.smithi092.stderr:Inferring config /var/lib/ceph/00db0016-dfed-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T21:46:05.572 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:46:05.584 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:46:05 smithi097 ceph-mon[33317]: pgmap v965: 148 pgs: 148 active+clean; 5.7 GiB data, 21 GiB used, 515 GiB / 536 GiB avail; 1023 B/s rd, 83 MiB/s wr, 1.34k op/s 2024-03-11T21:46:05.584 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:46:05 smithi097 ceph-mon[33317]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-11T21:46:05.584 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:46:05 smithi097 ceph-mon[33317]: from='client.? v1:172.21.15.92:0/4285874803' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-03-11T21:46:05.584 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:46:05 smithi097 ceph-mon[33317]: osdmap e459: 8 total, 8 up, 6 in 2024-03-11T21:46:05.584 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:46:05 smithi097 ceph-mon[33317]: from='mgr.14607 v1:172.21.15.92:0/1702717597' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T21:46:05.584 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:46:05 smithi097 ceph-mon[33317]: from='mgr.14607 v1:172.21.15.92:0/1702717597' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T21:46:05.584 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:46:05 smithi097 ceph-mon[33317]: from='mgr.14607 ' entity='mgr.y' 2024-03-11T21:46:05.606 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:46:05.609 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:46:05.609 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:46:05.710 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:46:05.744 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:46:05.745 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:46:05.745 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:46:05.846 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:46:05.871 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:46:05.871 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:46:05.871 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:46:05.972 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:46:05.996 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:46:05.996 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:46:05.997 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:46:06.097 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:46:06.144 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:46:06.145 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:46:06.145 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:46:06.246 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:46:06.266 INFO:tasks.radosbench.radosbench.0.smithi092.stdout: 59 16 98598 98582 104.407 67.1875 0.00753943 0.0095727 2024-03-11T21:46:06.274 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:46:06 smithi092 ceph-mon[29572]: 6.8 scrub starts 2024-03-11T21:46:06.274 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:46:06 smithi092 ceph-mon[29572]: 6.8 scrub ok 2024-03-11T21:46:06.275 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:46:06 smithi092 ceph-mon[29572]: 2.13 scrub starts 2024-03-11T21:46:06.275 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:46:06 smithi092 ceph-mon[29572]: 2.13 scrub ok 2024-03-11T21:46:06.275 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:46:06 smithi092 ceph-mon[37563]: 6.8 scrub starts 2024-03-11T21:46:06.275 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:46:06 smithi092 ceph-mon[37563]: 6.8 scrub ok 2024-03-11T21:46:06.275 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:46:06 smithi092 ceph-mon[37563]: 2.13 scrub starts 2024-03-11T21:46:06.275 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:46:06 smithi092 ceph-mon[37563]: 2.13 scrub ok 2024-03-11T21:46:06.290 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:46:06.290 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:46:06.290 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:46:06.358 INFO:teuthology.orchestra.run.smithi092.stdout:osd.0: {} 2024-03-11T21:46:06.374 INFO:teuthology.orchestra.run.smithi092.stdout:osd.1: {} 2024-03-11T21:46:06.392 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:46:06.411 INFO:teuthology.orchestra.run.smithi092.stdout:osd.2: {} 2024-03-11T21:46:06.425 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:46:06.426 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:46:06.426 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:46:06.452 INFO:teuthology.orchestra.run.smithi092.stdout:osd.3: {} 2024-03-11T21:46:06.491 INFO:teuthology.orchestra.run.smithi092.stdout:osd.4: {} 2024-03-11T21:46:06.511 INFO:teuthology.orchestra.run.smithi092.stdout:osd.5: {} 2024-03-11T21:46:06.527 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:46:06.563 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:46:06.564 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:46:06.564 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:46:06.566 INFO:teuthology.orchestra.run.smithi092.stdout:osd.6: {} 2024-03-11T21:46:06.584 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:46:06 smithi097 ceph-mon[33317]: 6.8 scrub starts 2024-03-11T21:46:06.584 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:46:06 smithi097 ceph-mon[33317]: 6.8 scrub ok 2024-03-11T21:46:06.584 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:46:06 smithi097 ceph-mon[33317]: 2.13 scrub starts 2024-03-11T21:46:06.584 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:46:06 smithi097 ceph-mon[33317]: 2.13 scrub ok 2024-03-11T21:46:06.605 INFO:teuthology.orchestra.run.smithi092.stdout:osd.7: {} 2024-03-11T21:46:06.617 INFO:teuthology.orchestra.run.smithi092.stderr:osd.0: osd_enable_op_tracker = '' 2024-03-11T21:46:06.617 INFO:teuthology.orchestra.run.smithi092.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-11T21:46:06.617 INFO:teuthology.orchestra.run.smithi092.stderr:osd.2: osd_enable_op_tracker = '' 2024-03-11T21:46:06.617 INFO:teuthology.orchestra.run.smithi092.stderr:osd.3: osd_enable_op_tracker = '' 2024-03-11T21:46:06.617 INFO:teuthology.orchestra.run.smithi092.stderr:osd.4: osd_enable_op_tracker = '' 2024-03-11T21:46:06.617 INFO:teuthology.orchestra.run.smithi092.stderr:osd.5: osd_enable_op_tracker = '' 2024-03-11T21:46:06.617 INFO:teuthology.orchestra.run.smithi092.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-11T21:46:06.617 INFO:teuthology.orchestra.run.smithi092.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-11T21:46:06.665 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:46:06.703 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:46:06.704 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:46:06.704 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:46:06.805 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:46:06.835 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:46:06.835 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:46:06.835 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:46:06.936 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:46:06.969 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:46:06.970 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:46:06.971 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:46:07.071 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:46:07.106 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:46:07.107 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:46:07.107 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:46:07.208 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:46:07.254 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:46:07.254 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:46:07.254 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:46:07.266 INFO:tasks.radosbench.radosbench.0.smithi092.stdout:2024-03-11T21:46:07.277820+0000 min lat: 0.00128848 max lat: 0.231127 avg lat: 0.00961726 2024-03-11T21:46:07.266 INFO:tasks.radosbench.radosbench.0.smithi092.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-03-11T21:46:07.267 INFO:tasks.radosbench.radosbench.0.smithi092.stdout: 60 16 99814 99798 103.934 76 0.00950327 0.00961726 2024-03-11T21:46:07.355 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:46:07.379 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:46:07.381 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:46:07.381 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:46:07.482 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:46:07.487 DEBUG:teuthology.orchestra.run.smithi092:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid 00db0016-dfed-11ee-95c8-87774f69a715 -- ceph osd unset nodeep-scrub 2024-03-11T21:46:07.519 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:46:07.520 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:46:07.520 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:46:07.550 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:46:07 smithi092 ceph-mon[29572]: pgmap v967: 148 pgs: 148 active+clean; 5.8 GiB data, 22 GiB used, 515 GiB / 536 GiB avail; 1023 B/s rd, 83 MiB/s wr, 1.34k op/s 2024-03-11T21:46:07.550 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:46:07 smithi092 ceph-mon[29572]: purged_snaps scrub starts 2024-03-11T21:46:07.550 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:46:07 smithi092 ceph-mon[29572]: purged_snaps scrub ok 2024-03-11T21:46:07.550 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:46:07 smithi092 ceph-mon[29572]: 5.1d scrub starts 2024-03-11T21:46:07.550 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:46:07 smithi092 ceph-mon[29572]: 5.1d scrub ok 2024-03-11T21:46:07.550 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:46:07 smithi092 ceph-mon[29572]: osdmap e460: 8 total, 8 up, 6 in 2024-03-11T21:46:07.550 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:46:07 smithi092 ceph-mon[29572]: from='client.? v1:172.21.15.92:0/4080470626' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-11T21:46:07.550 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:46:07 smithi092 ceph-mon[29572]: 4.1a scrub starts 2024-03-11T21:46:07.550 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:46:07 smithi092 ceph-mon[29572]: 4.1a scrub ok 2024-03-11T21:46:07.551 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:46:07 smithi092 ceph-mon[37563]: pgmap v967: 148 pgs: 148 active+clean; 5.8 GiB data, 22 GiB used, 515 GiB / 536 GiB avail; 1023 B/s rd, 83 MiB/s wr, 1.34k op/s 2024-03-11T21:46:07.551 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:46:07 smithi092 ceph-mon[37563]: purged_snaps scrub starts 2024-03-11T21:46:07.551 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:46:07 smithi092 ceph-mon[37563]: purged_snaps scrub ok 2024-03-11T21:46:07.551 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:46:07 smithi092 ceph-mon[37563]: 5.1d scrub starts 2024-03-11T21:46:07.551 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:46:07 smithi092 ceph-mon[37563]: 5.1d scrub ok 2024-03-11T21:46:07.551 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:46:07 smithi092 ceph-mon[37563]: osdmap e460: 8 total, 8 up, 6 in 2024-03-11T21:46:07.551 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:46:07 smithi092 ceph-mon[37563]: from='client.? v1:172.21.15.92:0/4080470626' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-11T21:46:07.551 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:46:07 smithi092 ceph-mon[37563]: 4.1a scrub starts 2024-03-11T21:46:07.551 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:46:07 smithi092 ceph-mon[37563]: 4.1a scrub ok 2024-03-11T21:46:07.584 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:46:07 smithi097 ceph-mon[33317]: pgmap v967: 148 pgs: 148 active+clean; 5.8 GiB data, 22 GiB used, 515 GiB / 536 GiB avail; 1023 B/s rd, 83 MiB/s wr, 1.34k op/s 2024-03-11T21:46:07.584 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:46:07 smithi097 ceph-mon[33317]: purged_snaps scrub starts 2024-03-11T21:46:07.584 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:46:07 smithi097 ceph-mon[33317]: purged_snaps scrub ok 2024-03-11T21:46:07.584 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:46:07 smithi097 ceph-mon[33317]: 5.1d scrub starts 2024-03-11T21:46:07.584 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:46:07 smithi097 ceph-mon[33317]: 5.1d scrub ok 2024-03-11T21:46:07.584 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:46:07 smithi097 ceph-mon[33317]: osdmap e460: 8 total, 8 up, 6 in 2024-03-11T21:46:07.584 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:46:07 smithi097 ceph-mon[33317]: from='client.? v1:172.21.15.92:0/4080470626' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-11T21:46:07.584 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:46:07 smithi097 ceph-mon[33317]: 4.1a scrub starts 2024-03-11T21:46:07.584 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:46:07 smithi097 ceph-mon[33317]: 4.1a scrub ok 2024-03-11T21:46:07.621 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:46:07.644 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:46:07.645 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:46:07.645 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:46:07.746 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:46:07.794 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:46:07.794 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:46:07.795 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:46:07.896 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:46:07.938 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:46:07.940 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:46:07.940 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:46:08.041 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:46:08.065 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:46:08.066 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:46:08.067 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:46:08.167 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:46:08.200 INFO:journalctl@ceph.mgr.y.smithi092.stdout:Mar 11 21:46:07 smithi092 ceph-00db0016-dfed-11ee-95c8-87774f69a715-mgr-y[29801]: ::ffff:172.21.15.97 - - [11/Mar/2024:21:46:07] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-11T21:46:08.200 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:46:08 smithi092 ceph-mon[29572]: 2.11 scrub starts 2024-03-11T21:46:08.203 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:46:08.204 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:46:08.204 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:46:08.267 INFO:tasks.radosbench.radosbench.0.smithi092.stdout: 61 16 100890 100874 103.332 67.25 0.00764537 0.00967277 2024-03-11T21:46:08.304 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:46:08.334 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:46:08 smithi097 ceph-mon[33317]: 2.11 scrub starts 2024-03-11T21:46:08.334 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:46:08 smithi097 ceph-mon[33317]: 2.11 scrub ok 2024-03-11T21:46:08.334 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:46:08 smithi097 ceph-mon[33317]: 6.15 scrub starts 2024-03-11T21:46:08.334 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:46:08 smithi097 ceph-mon[33317]: 6.15 scrub ok 2024-03-11T21:46:08.334 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:46:08 smithi097 ceph-mon[33317]: from='mgr.14607 v1:172.21.15.92:0/1702717597' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T21:46:08.344 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:46:08.345 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:46:08.345 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:46:08.446 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:46:08.481 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:46:08.482 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:46:08.482 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:46:08.583 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:46:08.614 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:46:08.615 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:46:08.615 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:46:08.618 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:46:08 smithi092 ceph-mon[29572]: 2.11 scrub ok 2024-03-11T21:46:08.618 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:46:08 smithi092 ceph-mon[29572]: 6.15 scrub starts 2024-03-11T21:46:08.618 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:46:08 smithi092 ceph-mon[29572]: 6.15 scrub ok 2024-03-11T21:46:08.618 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:46:08 smithi092 ceph-mon[29572]: from='mgr.14607 v1:172.21.15.92:0/1702717597' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T21:46:08.618 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:46:08 smithi092 ceph-mon[37563]: 2.11 scrub starts 2024-03-11T21:46:08.618 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:46:08 smithi092 ceph-mon[37563]: 2.11 scrub ok 2024-03-11T21:46:08.618 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:46:08 smithi092 ceph-mon[37563]: 6.15 scrub starts 2024-03-11T21:46:08.618 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:46:08 smithi092 ceph-mon[37563]: 6.15 scrub ok 2024-03-11T21:46:08.619 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:46:08 smithi092 ceph-mon[37563]: from='mgr.14607 v1:172.21.15.92:0/1702717597' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T21:46:08.716 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:46:08.749 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:46:08.750 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:46:08.751 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:46:08.851 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:46:08.889 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:46:08.889 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:46:08.889 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:46:08.990 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:46:09.034 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:46:09.035 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:46:09.035 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:46:09.036 INFO:teuthology.orchestra.run.smithi097.stdout:{ 2024-03-11T21:46:09.037 INFO:teuthology.orchestra.run.smithi097.stdout: "ops": [ 2024-03-11T21:46:09.037 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:46:09.037 INFO:teuthology.orchestra.run.smithi097.stdout: "description": "osd_op(client.35813.0:100894 11.3 11:c6b36454:::benchmark_data_smithi092_186520_object100893:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536 in=65536b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e460)", 2024-03-11T21:46:09.037 INFO:teuthology.orchestra.run.smithi097.stdout: "initiated_at": "2024-03-11T21:46:08.280382+0000", 2024-03-11T21:46:09.037 INFO:teuthology.orchestra.run.smithi097.stdout: "age": 0.0035710830000000001, 2024-03-11T21:46:09.037 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.0036047010000000001, 2024-03-11T21:46:09.037 INFO:teuthology.orchestra.run.smithi097.stdout: "type_data": { 2024-03-11T21:46:09.037 INFO:teuthology.orchestra.run.smithi097.stdout: "flag_point": "waiting for sub ops", 2024-03-11T21:46:09.037 INFO:teuthology.orchestra.run.smithi097.stdout: "client_info": { 2024-03-11T21:46:09.037 INFO:teuthology.orchestra.run.smithi097.stdout: "client": "client.35813", 2024-03-11T21:46:09.037 INFO:teuthology.orchestra.run.smithi097.stdout: "client_addr": "v1:172.21.15.92:0/4293352030", 2024-03-11T21:46:09.037 INFO:teuthology.orchestra.run.smithi097.stdout: "tid": 100894 2024-03-11T21:46:09.037 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:46:09.038 INFO:teuthology.orchestra.run.smithi097.stdout: "events": [ 2024-03-11T21:46:09.038 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:46:09.038 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "initiated", 2024-03-11T21:46:09.038 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:46:08.280382+0000", 2024-03-11T21:46:09.038 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-03-11T21:46:09.038 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:46:09.038 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:46:09.039 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "header_read", 2024-03-11T21:46:09.039 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:46:08.280382+0000", 2024-03-11T21:46:09.039 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-03-11T21:46:09.039 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:46:09.039 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:46:09.039 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "throttled", 2024-03-11T21:46:09.039 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:46:08.280384+0000", 2024-03-11T21:46:09.039 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.688e-06 2024-03-11T21:46:09.039 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:46:09.040 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:46:09.040 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "all_read", 2024-03-11T21:46:09.040 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:46:08.280427+0000", 2024-03-11T21:46:09.040 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 4.3588999999999999e-05 2024-03-11T21:46:09.040 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:46:09.040 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:46:09.040 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "dispatched", 2024-03-11T21:46:09.040 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:46:08.280441+0000", 2024-03-11T21:46:09.040 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.3541e-05 2024-03-11T21:46:09.040 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:46:09.040 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:46:09.040 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "queued_for_pg", 2024-03-11T21:46:09.040 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:46:08.280448+0000", 2024-03-11T21:46:09.040 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 7.3390000000000004e-06 2024-03-11T21:46:09.040 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:46:09.041 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:46:09.041 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "reached_pg", 2024-03-11T21:46:09.041 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:46:08.280504+0000", 2024-03-11T21:46:09.041 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 5.5844000000000003e-05 2024-03-11T21:46:09.041 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:46:09.041 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:46:09.041 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "started", 2024-03-11T21:46:09.041 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:46:08.280642+0000", 2024-03-11T21:46:09.041 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.00013735 2024-03-11T21:46:09.041 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:46:09.041 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:46:09.041 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "waiting for subops from 1,7", 2024-03-11T21:46:09.041 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:46:08.280791+0000", 2024-03-11T21:46:09.041 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.00014922700000000001 2024-03-11T21:46:09.041 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:46:09.042 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:46:09.042 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "op_commit", 2024-03-11T21:46:09.042 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:46:08.283484+0000", 2024-03-11T21:46:09.042 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.0026932309999999999 2024-03-11T21:46:09.042 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-03-11T21:46:09.042 INFO:teuthology.orchestra.run.smithi097.stdout: ] 2024-03-11T21:46:09.042 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-03-11T21:46:09.042 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:46:09.042 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:46:09.042 INFO:teuthology.orchestra.run.smithi097.stdout: "description": "osd_op(client.35813.0:100898 11.1 11:8b6f1e5a:::benchmark_data_smithi092_186520_object100897:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536 in=65536b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e460)", 2024-03-11T21:46:09.042 INFO:teuthology.orchestra.run.smithi097.stdout: "initiated_at": "2024-03-11T21:46:08.281077+0000", 2024-03-11T21:46:09.042 INFO:teuthology.orchestra.run.smithi097.stdout: "age": 0.0028764680000000001, 2024-03-11T21:46:09.042 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.0030085310000000001, 2024-03-11T21:46:09.043 INFO:teuthology.orchestra.run.smithi097.stdout: "type_data": { 2024-03-11T21:46:09.043 INFO:teuthology.orchestra.run.smithi097.stdout: "flag_point": "waiting for sub ops", 2024-03-11T21:46:09.043 INFO:teuthology.orchestra.run.smithi097.stdout: "client_info": { 2024-03-11T21:46:09.043 INFO:teuthology.orchestra.run.smithi097.stdout: "client": "client.35813", 2024-03-11T21:46:09.043 INFO:teuthology.orchestra.run.smithi097.stdout: "client_addr": "v1:172.21.15.92:0/4293352030", 2024-03-11T21:46:09.043 INFO:teuthology.orchestra.run.smithi097.stdout: "tid": 100898 2024-03-11T21:46:09.043 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:46:09.043 INFO:teuthology.orchestra.run.smithi097.stdout: "events": [ 2024-03-11T21:46:09.043 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:46:09.043 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "initiated", 2024-03-11T21:46:09.043 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:46:08.281077+0000", 2024-03-11T21:46:09.043 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-03-11T21:46:09.043 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:46:09.043 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:46:09.044 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "header_read", 2024-03-11T21:46:09.044 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:46:08.281077+0000", 2024-03-11T21:46:09.044 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-03-11T21:46:09.044 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:46:09.044 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:46:09.044 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "throttled", 2024-03-11T21:46:09.044 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:46:08.281078+0000", 2024-03-11T21:46:09.044 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.173e-06 2024-03-11T21:46:09.044 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:46:09.044 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:46:09.044 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "all_read", 2024-03-11T21:46:09.044 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:46:08.281137+0000", 2024-03-11T21:46:09.044 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 5.8758999999999998e-05 2024-03-11T21:46:09.044 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:46:09.044 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:46:09.045 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "dispatched", 2024-03-11T21:46:09.045 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:46:08.281151+0000", 2024-03-11T21:46:09.045 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.4070000000000001e-05 2024-03-11T21:46:09.045 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:46:09.045 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:46:09.045 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "queued_for_pg", 2024-03-11T21:46:09.045 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:46:08.281158+0000", 2024-03-11T21:46:09.045 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 7.515e-06 2024-03-11T21:46:09.045 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:46:09.045 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:46:09.045 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "reached_pg", 2024-03-11T21:46:09.045 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:46:08.281219+0000", 2024-03-11T21:46:09.045 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 6.0208999999999999e-05 2024-03-11T21:46:09.045 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:46:09.045 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:46:09.046 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "started", 2024-03-11T21:46:09.046 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:46:08.281368+0000", 2024-03-11T21:46:09.046 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.00014987299999999999 2024-03-11T21:46:09.046 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:46:09.046 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:46:09.046 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "waiting for subops from 0,2", 2024-03-11T21:46:09.046 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:46:08.281536+0000", 2024-03-11T21:46:09.046 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.00016796500000000001 2024-03-11T21:46:09.046 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-03-11T21:46:09.046 INFO:teuthology.orchestra.run.smithi097.stdout: ] 2024-03-11T21:46:09.046 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-03-11T21:46:09.046 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-03-11T21:46:09.046 INFO:teuthology.orchestra.run.smithi097.stdout: ], 2024-03-11T21:46:09.046 INFO:teuthology.orchestra.run.smithi097.stdout: "num_ops": 2 2024-03-11T21:46:09.046 INFO:teuthology.orchestra.run.smithi097.stdout:} 2024-03-11T21:46:09.136 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:46:09.168 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:46:09.169 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:46:09.169 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:46:09.267 INFO:tasks.radosbench.radosbench.0.smithi092.stdout: 62 16 102210 102194 102.996 82.5 0.0115349 0.00970458 2024-03-11T21:46:09.270 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:46:09.302 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:46:09.302 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:46:09.302 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:46:09.333 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:46:09 smithi097 ceph-mon[33317]: pgmap v969: 148 pgs: 148 active+clean; 5.8 GiB data, 22 GiB used, 515 GiB / 536 GiB avail; 639 B/s rd, 58 MiB/s wr, 931 op/s 2024-03-11T21:46:09.334 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:46:09 smithi097 ceph-mon[33317]: 4.e scrub starts 2024-03-11T21:46:09.334 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:46:09 smithi097 ceph-mon[33317]: 4.e scrub ok 2024-03-11T21:46:09.334 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:46:09 smithi097 ceph-mon[33317]: osdmap e461: 8 total, 8 up, 6 in 2024-03-11T21:46:09.404 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:46:09.427 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:46:09.428 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:46:09.428 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:46:09.529 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:46:09.570 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:46:09.571 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:46:09.572 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:46:09.618 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:46:09 smithi092 ceph-mon[29572]: pgmap v969: 148 pgs: 148 active+clean; 5.8 GiB data, 22 GiB used, 515 GiB / 536 GiB avail; 639 B/s rd, 58 MiB/s wr, 931 op/s 2024-03-11T21:46:09.618 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:46:09 smithi092 ceph-mon[29572]: 4.e scrub starts 2024-03-11T21:46:09.618 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:46:09 smithi092 ceph-mon[29572]: 4.e scrub ok 2024-03-11T21:46:09.618 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:46:09 smithi092 ceph-mon[29572]: osdmap e461: 8 total, 8 up, 6 in 2024-03-11T21:46:09.618 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:46:09 smithi092 ceph-mon[37563]: pgmap v969: 148 pgs: 148 active+clean; 5.8 GiB data, 22 GiB used, 515 GiB / 536 GiB avail; 639 B/s rd, 58 MiB/s wr, 931 op/s 2024-03-11T21:46:09.618 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:46:09 smithi092 ceph-mon[37563]: 4.e scrub starts 2024-03-11T21:46:09.618 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:46:09 smithi092 ceph-mon[37563]: 4.e scrub ok 2024-03-11T21:46:09.618 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:46:09 smithi092 ceph-mon[37563]: osdmap e461: 8 total, 8 up, 6 in 2024-03-11T21:46:09.673 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:46:09.712 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:46:09.713 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:46:09.713 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:46:09.814 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:46:09.885 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:46:09.885 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:46:09.886 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:46:09.986 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:46:10.020 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:46:10.021 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:46:10.021 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:46:10.122 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:46:10.161 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:46:10.162 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:46:10.163 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:46:10.258 DEBUG:teuthology.orchestra.run.smithi097:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid 00db0016-dfed-11ee-95c8-87774f69a715 -- ceph daemon osd.6 dump_blocked_ops 2024-03-11T21:46:10.263 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:46:10.267 INFO:tasks.radosbench.radosbench.0.smithi092.stdout: 63 16 103421 103405 102.563 75.6875 0.0151379 0.00974582 2024-03-11T21:46:10.313 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:46:10.314 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:46:10.314 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:46:10.328 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:46:10 smithi092 ceph-mon[29572]: 4.1b scrub starts 2024-03-11T21:46:10.328 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:46:10 smithi092 ceph-mon[29572]: 4.1b scrub ok 2024-03-11T21:46:10.328 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:46:10 smithi092 ceph-mon[29572]: 5.15 scrub starts 2024-03-11T21:46:10.328 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:46:10 smithi092 ceph-mon[29572]: 5.15 scrub ok 2024-03-11T21:46:10.329 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:46:10 smithi092 ceph-mon[29572]: purged_snaps scrub starts 2024-03-11T21:46:10.329 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:46:10 smithi092 ceph-mon[29572]: purged_snaps scrub ok 2024-03-11T21:46:10.329 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:46:10 smithi092 ceph-mon[29572]: 5.1 scrub starts 2024-03-11T21:46:10.329 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:46:10 smithi092 ceph-mon[29572]: 5.1 scrub ok 2024-03-11T21:46:10.329 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:46:10 smithi092 ceph-mon[29572]: osdmap e462: 8 total, 8 up, 6 in 2024-03-11T21:46:10.329 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:46:10 smithi092 ceph-mon[37563]: 4.1b scrub starts 2024-03-11T21:46:10.329 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:46:10 smithi092 ceph-mon[37563]: 4.1b scrub ok 2024-03-11T21:46:10.329 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:46:10 smithi092 ceph-mon[37563]: 5.15 scrub starts 2024-03-11T21:46:10.329 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:46:10 smithi092 ceph-mon[37563]: 5.15 scrub ok 2024-03-11T21:46:10.329 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:46:10 smithi092 ceph-mon[37563]: purged_snaps scrub starts 2024-03-11T21:46:10.329 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:46:10 smithi092 ceph-mon[37563]: purged_snaps scrub ok 2024-03-11T21:46:10.329 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:46:10 smithi092 ceph-mon[37563]: 5.1 scrub starts 2024-03-11T21:46:10.329 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:46:10 smithi092 ceph-mon[37563]: 5.1 scrub ok 2024-03-11T21:46:10.329 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:46:10 smithi092 ceph-mon[37563]: osdmap e462: 8 total, 8 up, 6 in 2024-03-11T21:46:10.415 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:46:10.451 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:46:10.451 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:46:10.452 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:46:10.553 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:46:10.585 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:46:10 smithi097 ceph-mon[33317]: 4.1b scrub starts 2024-03-11T21:46:10.585 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:46:10 smithi097 ceph-mon[33317]: 4.1b scrub ok 2024-03-11T21:46:10.585 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:46:10 smithi097 ceph-mon[33317]: 5.15 scrub starts 2024-03-11T21:46:10.585 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:46:10 smithi097 ceph-mon[33317]: 5.15 scrub ok 2024-03-11T21:46:10.586 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:46:10 smithi097 ceph-mon[33317]: purged_snaps scrub starts 2024-03-11T21:46:10.586 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:46:10 smithi097 ceph-mon[33317]: purged_snaps scrub ok 2024-03-11T21:46:10.586 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:46:10 smithi097 ceph-mon[33317]: 5.1 scrub starts 2024-03-11T21:46:10.586 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:46:10 smithi097 ceph-mon[33317]: 5.1 scrub ok 2024-03-11T21:46:10.586 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:46:10 smithi097 ceph-mon[33317]: osdmap e462: 8 total, 8 up, 6 in 2024-03-11T21:46:10.594 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:46:10.594 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:46:10.595 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:46:10.619 INFO:teuthology.orchestra.run.smithi097.stderr:Inferring config /var/lib/ceph/00db0016-dfed-11ee-95c8-87774f69a715/mon.b/config 2024-03-11T21:46:10.696 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:46:10.752 INFO:teuthology.orchestra.run.smithi092.stderr:Inferring config /var/lib/ceph/00db0016-dfed-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T21:46:10.753 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:46:10.754 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:46:10.754 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:46:10.855 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:46:10.890 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:46:10.891 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:46:10.891 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:46:10.992 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:46:11.035 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:46:11.035 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:46:11.036 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:46:11.093 INFO:teuthology.orchestra.run.smithi092.stdout: 2024-03-11T21:46:11.094 INFO:teuthology.orchestra.run.smithi092.stdout:{ 2024-03-11T21:46:11.094 INFO:teuthology.orchestra.run.smithi092.stdout: "epoch": 462, 2024-03-11T21:46:11.094 INFO:teuthology.orchestra.run.smithi092.stdout: "fsid": "00db0016-dfed-11ee-95c8-87774f69a715", 2024-03-11T21:46:11.094 INFO:teuthology.orchestra.run.smithi092.stdout: "created": "2024-03-11T21:20:10.488059+0000", 2024-03-11T21:46:11.094 INFO:teuthology.orchestra.run.smithi092.stdout: "modified": "2024-03-11T21:46:09.563692+0000", 2024-03-11T21:46:11.094 INFO:teuthology.orchestra.run.smithi092.stdout: "last_up_change": "2024-03-11T21:36:34.026798+0000", 2024-03-11T21:46:11.094 INFO:teuthology.orchestra.run.smithi092.stdout: "last_in_change": "2024-03-11T21:44:34.572566+0000", 2024-03-11T21:46:11.094 INFO:teuthology.orchestra.run.smithi092.stdout: "flags": "nodeep-scrub,sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit", 2024-03-11T21:46:11.094 INFO:teuthology.orchestra.run.smithi092.stdout: "flags_num": 5804032, 2024-03-11T21:46:11.094 INFO:teuthology.orchestra.run.smithi092.stdout: "flags_set": [ 2024-03-11T21:46:11.094 INFO:teuthology.orchestra.run.smithi092.stdout: "nodeep-scrub", 2024-03-11T21:46:11.094 INFO:teuthology.orchestra.run.smithi092.stdout: "pglog_hardlimit", 2024-03-11T21:46:11.095 INFO:teuthology.orchestra.run.smithi092.stdout: "purged_snapdirs", 2024-03-11T21:46:11.095 INFO:teuthology.orchestra.run.smithi092.stdout: "recovery_deletes", 2024-03-11T21:46:11.095 INFO:teuthology.orchestra.run.smithi092.stdout: "sortbitwise" 2024-03-11T21:46:11.095 INFO:teuthology.orchestra.run.smithi092.stdout: ], 2024-03-11T21:46:11.095 INFO:teuthology.orchestra.run.smithi092.stdout: "crush_version": 19, 2024-03-11T21:46:11.095 INFO:teuthology.orchestra.run.smithi092.stdout: "full_ratio": 0.94999998807907104, 2024-03-11T21:46:11.095 INFO:teuthology.orchestra.run.smithi092.stdout: "backfillfull_ratio": 0.89999997615814209, 2024-03-11T21:46:11.095 INFO:teuthology.orchestra.run.smithi092.stdout: "nearfull_ratio": 0.85000002384185791, 2024-03-11T21:46:11.095 INFO:teuthology.orchestra.run.smithi092.stdout: "cluster_snapshot": "", 2024-03-11T21:46:11.095 INFO:teuthology.orchestra.run.smithi092.stdout: "pool_max": 11, 2024-03-11T21:46:11.095 INFO:teuthology.orchestra.run.smithi092.stdout: "max_osd": 8, 2024-03-11T21:46:11.095 INFO:teuthology.orchestra.run.smithi092.stdout: "require_min_compat_client": "luminous", 2024-03-11T21:46:11.095 INFO:teuthology.orchestra.run.smithi092.stdout: "min_compat_client": "luminous", 2024-03-11T21:46:11.095 INFO:teuthology.orchestra.run.smithi092.stdout: "require_osd_release": "squid", 2024-03-11T21:46:11.095 INFO:teuthology.orchestra.run.smithi092.stdout: "allow_crimson": false, 2024-03-11T21:46:11.096 INFO:teuthology.orchestra.run.smithi092.stdout: "pools": [ 2024-03-11T21:46:11.096 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:46:11.096 INFO:teuthology.orchestra.run.smithi092.stdout: "pool": 1, 2024-03-11T21:46:11.096 INFO:teuthology.orchestra.run.smithi092.stdout: "pool_name": ".mgr", 2024-03-11T21:46:11.096 INFO:teuthology.orchestra.run.smithi092.stdout: "create_time": "2024-03-11T21:23:23.805029+0000", 2024-03-11T21:46:11.096 INFO:teuthology.orchestra.run.smithi092.stdout: "flags": 1, 2024-03-11T21:46:11.096 INFO:teuthology.orchestra.run.smithi092.stdout: "flags_names": "hashpspool", 2024-03-11T21:46:11.096 INFO:teuthology.orchestra.run.smithi092.stdout: "type": 1, 2024-03-11T21:46:11.096 INFO:teuthology.orchestra.run.smithi092.stdout: "size": 3, 2024-03-11T21:46:11.097 INFO:teuthology.orchestra.run.smithi092.stdout: "min_size": 2, 2024-03-11T21:46:11.097 INFO:teuthology.orchestra.run.smithi092.stdout: "crush_rule": 0, 2024-03-11T21:46:11.097 INFO:teuthology.orchestra.run.smithi092.stdout: "peering_crush_bucket_count": 0, 2024-03-11T21:46:11.097 INFO:teuthology.orchestra.run.smithi092.stdout: "peering_crush_bucket_target": 0, 2024-03-11T21:46:11.097 INFO:teuthology.orchestra.run.smithi092.stdout: "peering_crush_bucket_barrier": 0, 2024-03-11T21:46:11.097 INFO:teuthology.orchestra.run.smithi092.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-03-11T21:46:11.097 INFO:teuthology.orchestra.run.smithi092.stdout: "object_hash": 2, 2024-03-11T21:46:11.097 INFO:teuthology.orchestra.run.smithi092.stdout: "pg_autoscale_mode": "off", 2024-03-11T21:46:11.097 INFO:teuthology.orchestra.run.smithi092.stdout: "pg_num": 1, 2024-03-11T21:46:11.097 INFO:teuthology.orchestra.run.smithi092.stdout: "pg_placement_num": 1, 2024-03-11T21:46:11.097 INFO:teuthology.orchestra.run.smithi092.stdout: "pg_placement_num_target": 1, 2024-03-11T21:46:11.097 INFO:teuthology.orchestra.run.smithi092.stdout: "pg_num_target": 1, 2024-03-11T21:46:11.097 INFO:teuthology.orchestra.run.smithi092.stdout: "pg_num_pending": 1, 2024-03-11T21:46:11.097 INFO:teuthology.orchestra.run.smithi092.stdout: "last_pg_merge_meta": { 2024-03-11T21:46:11.098 INFO:teuthology.orchestra.run.smithi092.stdout: "source_pgid": "0.0", 2024-03-11T21:46:11.098 INFO:teuthology.orchestra.run.smithi092.stdout: "ready_epoch": 0, 2024-03-11T21:46:11.098 INFO:teuthology.orchestra.run.smithi092.stdout: "last_epoch_started": 0, 2024-03-11T21:46:11.098 INFO:teuthology.orchestra.run.smithi092.stdout: "last_epoch_clean": 0, 2024-03-11T21:46:11.098 INFO:teuthology.orchestra.run.smithi092.stdout: "source_version": "0'0", 2024-03-11T21:46:11.098 INFO:teuthology.orchestra.run.smithi092.stdout: "target_version": "0'0" 2024-03-11T21:46:11.098 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:46:11.098 INFO:teuthology.orchestra.run.smithi092.stdout: "last_change": "121", 2024-03-11T21:46:11.098 INFO:teuthology.orchestra.run.smithi092.stdout: "last_force_op_resend": "0", 2024-03-11T21:46:11.098 INFO:teuthology.orchestra.run.smithi092.stdout: "last_force_op_resend_prenautilus": "0", 2024-03-11T21:46:11.098 INFO:teuthology.orchestra.run.smithi092.stdout: "last_force_op_resend_preluminous": "0", 2024-03-11T21:46:11.098 INFO:teuthology.orchestra.run.smithi092.stdout: "auid": 0, 2024-03-11T21:46:11.098 INFO:teuthology.orchestra.run.smithi092.stdout: "snap_mode": "selfmanaged", 2024-03-11T21:46:11.098 INFO:teuthology.orchestra.run.smithi092.stdout: "snap_seq": 0, 2024-03-11T21:46:11.099 INFO:teuthology.orchestra.run.smithi092.stdout: "snap_epoch": 0, 2024-03-11T21:46:11.099 INFO:teuthology.orchestra.run.smithi092.stdout: "pool_snaps": [], 2024-03-11T21:46:11.099 INFO:teuthology.orchestra.run.smithi092.stdout: "removed_snaps": "[]", 2024-03-11T21:46:11.099 INFO:teuthology.orchestra.run.smithi092.stdout: "quota_max_bytes": 0, 2024-03-11T21:46:11.099 INFO:teuthology.orchestra.run.smithi092.stdout: "quota_max_objects": 0, 2024-03-11T21:46:11.099 INFO:teuthology.orchestra.run.smithi092.stdout: "tiers": [], 2024-03-11T21:46:11.099 INFO:teuthology.orchestra.run.smithi092.stdout: "tier_of": -1, 2024-03-11T21:46:11.099 INFO:teuthology.orchestra.run.smithi092.stdout: "read_tier": -1, 2024-03-11T21:46:11.099 INFO:teuthology.orchestra.run.smithi092.stdout: "write_tier": -1, 2024-03-11T21:46:11.099 INFO:teuthology.orchestra.run.smithi092.stdout: "cache_mode": "none", 2024-03-11T21:46:11.099 INFO:teuthology.orchestra.run.smithi092.stdout: "target_max_bytes": 0, 2024-03-11T21:46:11.099 INFO:teuthology.orchestra.run.smithi092.stdout: "target_max_objects": 0, 2024-03-11T21:46:11.099 INFO:teuthology.orchestra.run.smithi092.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-03-11T21:46:11.099 INFO:teuthology.orchestra.run.smithi092.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-03-11T21:46:11.099 INFO:teuthology.orchestra.run.smithi092.stdout: "cache_target_full_ratio_micro": 800000, 2024-03-11T21:46:11.100 INFO:teuthology.orchestra.run.smithi092.stdout: "cache_min_flush_age": 0, 2024-03-11T21:46:11.100 INFO:teuthology.orchestra.run.smithi092.stdout: "cache_min_evict_age": 0, 2024-03-11T21:46:11.100 INFO:teuthology.orchestra.run.smithi092.stdout: "erasure_code_profile": "", 2024-03-11T21:46:11.100 INFO:teuthology.orchestra.run.smithi092.stdout: "hit_set_params": { 2024-03-11T21:46:11.100 INFO:teuthology.orchestra.run.smithi092.stdout: "type": "none" 2024-03-11T21:46:11.100 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:46:11.100 INFO:teuthology.orchestra.run.smithi092.stdout: "hit_set_period": 0, 2024-03-11T21:46:11.100 INFO:teuthology.orchestra.run.smithi092.stdout: "hit_set_count": 0, 2024-03-11T21:46:11.100 INFO:teuthology.orchestra.run.smithi092.stdout: "use_gmt_hitset": true, 2024-03-11T21:46:11.100 INFO:teuthology.orchestra.run.smithi092.stdout: "min_read_recency_for_promote": 0, 2024-03-11T21:46:11.100 INFO:teuthology.orchestra.run.smithi092.stdout: "min_write_recency_for_promote": 0, 2024-03-11T21:46:11.100 INFO:teuthology.orchestra.run.smithi092.stdout: "hit_set_grade_decay_rate": 0, 2024-03-11T21:46:11.100 INFO:teuthology.orchestra.run.smithi092.stdout: "hit_set_search_last_n": 0, 2024-03-11T21:46:11.100 INFO:teuthology.orchestra.run.smithi092.stdout: "grade_table": [], 2024-03-11T21:46:11.100 INFO:teuthology.orchestra.run.smithi092.stdout: "stripe_width": 0, 2024-03-11T21:46:11.101 INFO:teuthology.orchestra.run.smithi092.stdout: "expected_num_objects": 0, 2024-03-11T21:46:11.101 INFO:teuthology.orchestra.run.smithi092.stdout: "fast_read": false, 2024-03-11T21:46:11.101 INFO:teuthology.orchestra.run.smithi092.stdout: "options": { 2024-03-11T21:46:11.101 INFO:teuthology.orchestra.run.smithi092.stdout: "pg_num_max": 32, 2024-03-11T21:46:11.101 INFO:teuthology.orchestra.run.smithi092.stdout: "pg_num_min": 1 2024-03-11T21:46:11.101 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:46:11.101 INFO:teuthology.orchestra.run.smithi092.stdout: "application_metadata": { 2024-03-11T21:46:11.101 INFO:teuthology.orchestra.run.smithi092.stdout: "mgr": {} 2024-03-11T21:46:11.101 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:46:11.101 INFO:teuthology.orchestra.run.smithi092.stdout: "read_balance": { 2024-03-11T21:46:11.101 INFO:teuthology.orchestra.run.smithi092.stdout: "score_type": "Fair distribution", 2024-03-11T21:46:11.101 INFO:teuthology.orchestra.run.smithi092.stdout: "score_acting": 7.8899998664855957, 2024-03-11T21:46:11.101 INFO:teuthology.orchestra.run.smithi092.stdout: "score_stable": 7.8899998664855957, 2024-03-11T21:46:11.101 INFO:teuthology.orchestra.run.smithi092.stdout: "optimal_score": 0.37999999523162842, 2024-03-11T21:46:11.101 INFO:teuthology.orchestra.run.smithi092.stdout: "raw_score_acting": 3, 2024-03-11T21:46:11.102 INFO:teuthology.orchestra.run.smithi092.stdout: "raw_score_stable": 3, 2024-03-11T21:46:11.102 INFO:teuthology.orchestra.run.smithi092.stdout: "primary_affinity_weighted": 1, 2024-03-11T21:46:11.102 INFO:teuthology.orchestra.run.smithi092.stdout: "average_primary_affinity": 1, 2024-03-11T21:46:11.102 INFO:teuthology.orchestra.run.smithi092.stdout: "average_primary_affinity_weighted": 1 2024-03-11T21:46:11.102 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-03-11T21:46:11.102 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:46:11.102 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:46:11.102 INFO:teuthology.orchestra.run.smithi092.stdout: "pool": 2, 2024-03-11T21:46:11.102 INFO:teuthology.orchestra.run.smithi092.stdout: "pool_name": ".rgw.root", 2024-03-11T21:46:11.102 INFO:teuthology.orchestra.run.smithi092.stdout: "create_time": "2024-03-11T21:25:37.044504+0000", 2024-03-11T21:46:11.102 INFO:teuthology.orchestra.run.smithi092.stdout: "flags": 1, 2024-03-11T21:46:11.102 INFO:teuthology.orchestra.run.smithi092.stdout: "flags_names": "hashpspool", 2024-03-11T21:46:11.102 INFO:teuthology.orchestra.run.smithi092.stdout: "type": 1, 2024-03-11T21:46:11.103 INFO:teuthology.orchestra.run.smithi092.stdout: "size": 3, 2024-03-11T21:46:11.103 INFO:teuthology.orchestra.run.smithi092.stdout: "min_size": 2, 2024-03-11T21:46:11.103 INFO:teuthology.orchestra.run.smithi092.stdout: "crush_rule": 0, 2024-03-11T21:46:11.103 INFO:teuthology.orchestra.run.smithi092.stdout: "peering_crush_bucket_count": 0, 2024-03-11T21:46:11.103 INFO:teuthology.orchestra.run.smithi092.stdout: "peering_crush_bucket_target": 0, 2024-03-11T21:46:11.103 INFO:teuthology.orchestra.run.smithi092.stdout: "peering_crush_bucket_barrier": 0, 2024-03-11T21:46:11.103 INFO:teuthology.orchestra.run.smithi092.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-03-11T21:46:11.103 INFO:teuthology.orchestra.run.smithi092.stdout: "object_hash": 2, 2024-03-11T21:46:11.103 INFO:teuthology.orchestra.run.smithi092.stdout: "pg_autoscale_mode": "off", 2024-03-11T21:46:11.103 INFO:teuthology.orchestra.run.smithi092.stdout: "pg_num": 32, 2024-03-11T21:46:11.103 INFO:teuthology.orchestra.run.smithi092.stdout: "pg_placement_num": 32, 2024-03-11T21:46:11.103 INFO:teuthology.orchestra.run.smithi092.stdout: "pg_placement_num_target": 32, 2024-03-11T21:46:11.103 INFO:teuthology.orchestra.run.smithi092.stdout: "pg_num_target": 32, 2024-03-11T21:46:11.103 INFO:teuthology.orchestra.run.smithi092.stdout: "pg_num_pending": 32, 2024-03-11T21:46:11.103 INFO:teuthology.orchestra.run.smithi092.stdout: "last_pg_merge_meta": { 2024-03-11T21:46:11.103 INFO:teuthology.orchestra.run.smithi092.stdout: "source_pgid": "0.0", 2024-03-11T21:46:11.104 INFO:teuthology.orchestra.run.smithi092.stdout: "ready_epoch": 0, 2024-03-11T21:46:11.104 INFO:teuthology.orchestra.run.smithi092.stdout: "last_epoch_started": 0, 2024-03-11T21:46:11.104 INFO:teuthology.orchestra.run.smithi092.stdout: "last_epoch_clean": 0, 2024-03-11T21:46:11.104 INFO:teuthology.orchestra.run.smithi092.stdout: "source_version": "0'0", 2024-03-11T21:46:11.104 INFO:teuthology.orchestra.run.smithi092.stdout: "target_version": "0'0" 2024-03-11T21:46:11.104 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:46:11.104 INFO:teuthology.orchestra.run.smithi092.stdout: "last_change": "68", 2024-03-11T21:46:11.104 INFO:teuthology.orchestra.run.smithi092.stdout: "last_force_op_resend": "0", 2024-03-11T21:46:11.104 INFO:teuthology.orchestra.run.smithi092.stdout: "last_force_op_resend_prenautilus": "0", 2024-03-11T21:46:11.104 INFO:teuthology.orchestra.run.smithi092.stdout: "last_force_op_resend_preluminous": "0", 2024-03-11T21:46:11.104 INFO:teuthology.orchestra.run.smithi092.stdout: "auid": 0, 2024-03-11T21:46:11.104 INFO:teuthology.orchestra.run.smithi092.stdout: "snap_mode": "selfmanaged", 2024-03-11T21:46:11.104 INFO:teuthology.orchestra.run.smithi092.stdout: "snap_seq": 0, 2024-03-11T21:46:11.104 INFO:teuthology.orchestra.run.smithi092.stdout: "snap_epoch": 0, 2024-03-11T21:46:11.104 INFO:teuthology.orchestra.run.smithi092.stdout: "pool_snaps": [], 2024-03-11T21:46:11.104 INFO:teuthology.orchestra.run.smithi092.stdout: "removed_snaps": "[]", 2024-03-11T21:46:11.105 INFO:teuthology.orchestra.run.smithi092.stdout: "quota_max_bytes": 0, 2024-03-11T21:46:11.105 INFO:teuthology.orchestra.run.smithi092.stdout: "quota_max_objects": 0, 2024-03-11T21:46:11.105 INFO:teuthology.orchestra.run.smithi092.stdout: "tiers": [], 2024-03-11T21:46:11.105 INFO:teuthology.orchestra.run.smithi092.stdout: "tier_of": -1, 2024-03-11T21:46:11.105 INFO:teuthology.orchestra.run.smithi092.stdout: "read_tier": -1, 2024-03-11T21:46:11.105 INFO:teuthology.orchestra.run.smithi092.stdout: "write_tier": -1, 2024-03-11T21:46:11.105 INFO:teuthology.orchestra.run.smithi092.stdout: "cache_mode": "none", 2024-03-11T21:46:11.105 INFO:teuthology.orchestra.run.smithi092.stdout: "target_max_bytes": 0, 2024-03-11T21:46:11.105 INFO:teuthology.orchestra.run.smithi092.stdout: "target_max_objects": 0, 2024-03-11T21:46:11.105 INFO:teuthology.orchestra.run.smithi092.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-03-11T21:46:11.105 INFO:teuthology.orchestra.run.smithi092.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-03-11T21:46:11.105 INFO:teuthology.orchestra.run.smithi092.stdout: "cache_target_full_ratio_micro": 800000, 2024-03-11T21:46:11.105 INFO:teuthology.orchestra.run.smithi092.stdout: "cache_min_flush_age": 0, 2024-03-11T21:46:11.105 INFO:teuthology.orchestra.run.smithi092.stdout: "cache_min_evict_age": 0, 2024-03-11T21:46:11.105 INFO:teuthology.orchestra.run.smithi092.stdout: "erasure_code_profile": "", 2024-03-11T21:46:11.105 INFO:teuthology.orchestra.run.smithi092.stdout: "hit_set_params": { 2024-03-11T21:46:11.106 INFO:teuthology.orchestra.run.smithi092.stdout: "type": "none" 2024-03-11T21:46:11.106 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:46:11.106 INFO:teuthology.orchestra.run.smithi092.stdout: "hit_set_period": 0, 2024-03-11T21:46:11.106 INFO:teuthology.orchestra.run.smithi092.stdout: "hit_set_count": 0, 2024-03-11T21:46:11.106 INFO:teuthology.orchestra.run.smithi092.stdout: "use_gmt_hitset": true, 2024-03-11T21:46:11.106 INFO:teuthology.orchestra.run.smithi092.stdout: "min_read_recency_for_promote": 0, 2024-03-11T21:46:11.106 INFO:teuthology.orchestra.run.smithi092.stdout: "min_write_recency_for_promote": 0, 2024-03-11T21:46:11.106 INFO:teuthology.orchestra.run.smithi092.stdout: "hit_set_grade_decay_rate": 0, 2024-03-11T21:46:11.106 INFO:teuthology.orchestra.run.smithi092.stdout: "hit_set_search_last_n": 0, 2024-03-11T21:46:11.106 INFO:teuthology.orchestra.run.smithi092.stdout: "grade_table": [], 2024-03-11T21:46:11.106 INFO:teuthology.orchestra.run.smithi092.stdout: "stripe_width": 0, 2024-03-11T21:46:11.106 INFO:teuthology.orchestra.run.smithi092.stdout: "expected_num_objects": 0, 2024-03-11T21:46:11.106 INFO:teuthology.orchestra.run.smithi092.stdout: "fast_read": false, 2024-03-11T21:46:11.106 INFO:teuthology.orchestra.run.smithi092.stdout: "options": {}, 2024-03-11T21:46:11.107 INFO:teuthology.orchestra.run.smithi092.stdout: "application_metadata": { 2024-03-11T21:46:11.107 INFO:teuthology.orchestra.run.smithi092.stdout: "rgw": {} 2024-03-11T21:46:11.107 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:46:11.107 INFO:teuthology.orchestra.run.smithi092.stdout: "read_balance": { 2024-03-11T21:46:11.107 INFO:teuthology.orchestra.run.smithi092.stdout: "score_type": "Fair distribution", 2024-03-11T21:46:11.107 INFO:teuthology.orchestra.run.smithi092.stdout: "score_acting": 1.75, 2024-03-11T21:46:11.107 INFO:teuthology.orchestra.run.smithi092.stdout: "score_stable": 1.75, 2024-03-11T21:46:11.107 INFO:teuthology.orchestra.run.smithi092.stdout: "optimal_score": 0.75, 2024-03-11T21:46:11.107 INFO:teuthology.orchestra.run.smithi092.stdout: "raw_score_acting": 1.309999942779541, 2024-03-11T21:46:11.107 INFO:teuthology.orchestra.run.smithi092.stdout: "raw_score_stable": 1.309999942779541, 2024-03-11T21:46:11.107 INFO:teuthology.orchestra.run.smithi092.stdout: "primary_affinity_weighted": 1, 2024-03-11T21:46:11.107 INFO:teuthology.orchestra.run.smithi092.stdout: "average_primary_affinity": 1, 2024-03-11T21:46:11.107 INFO:teuthology.orchestra.run.smithi092.stdout: "average_primary_affinity_weighted": 1 2024-03-11T21:46:11.107 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-03-11T21:46:11.107 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:46:11.108 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:46:11.108 INFO:teuthology.orchestra.run.smithi092.stdout: "pool": 3, 2024-03-11T21:46:11.108 INFO:teuthology.orchestra.run.smithi092.stdout: "pool_name": "datapool", 2024-03-11T21:46:11.108 INFO:teuthology.orchestra.run.smithi092.stdout: "create_time": "2024-03-11T21:25:37.280820+0000", 2024-03-11T21:46:11.108 INFO:teuthology.orchestra.run.smithi092.stdout: "flags": 8193, 2024-03-11T21:46:11.108 INFO:teuthology.orchestra.run.smithi092.stdout: "flags_names": "hashpspool,selfmanaged_snaps", 2024-03-11T21:46:11.108 INFO:teuthology.orchestra.run.smithi092.stdout: "type": 1, 2024-03-11T21:46:11.108 INFO:teuthology.orchestra.run.smithi092.stdout: "size": 3, 2024-03-11T21:46:11.108 INFO:teuthology.orchestra.run.smithi092.stdout: "min_size": 2, 2024-03-11T21:46:11.108 INFO:teuthology.orchestra.run.smithi092.stdout: "crush_rule": 0, 2024-03-11T21:46:11.108 INFO:teuthology.orchestra.run.smithi092.stdout: "peering_crush_bucket_count": 0, 2024-03-11T21:46:11.108 INFO:teuthology.orchestra.run.smithi092.stdout: "peering_crush_bucket_target": 0, 2024-03-11T21:46:11.108 INFO:teuthology.orchestra.run.smithi092.stdout: "peering_crush_bucket_barrier": 0, 2024-03-11T21:46:11.108 INFO:teuthology.orchestra.run.smithi092.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-03-11T21:46:11.108 INFO:teuthology.orchestra.run.smithi092.stdout: "object_hash": 2, 2024-03-11T21:46:11.109 INFO:teuthology.orchestra.run.smithi092.stdout: "pg_autoscale_mode": "off", 2024-03-11T21:46:11.109 INFO:teuthology.orchestra.run.smithi092.stdout: "pg_num": 3, 2024-03-11T21:46:11.109 INFO:teuthology.orchestra.run.smithi092.stdout: "pg_placement_num": 3, 2024-03-11T21:46:11.109 INFO:teuthology.orchestra.run.smithi092.stdout: "pg_placement_num_target": 3, 2024-03-11T21:46:11.109 INFO:teuthology.orchestra.run.smithi092.stdout: "pg_num_target": 3, 2024-03-11T21:46:11.109 INFO:teuthology.orchestra.run.smithi092.stdout: "pg_num_pending": 3, 2024-03-11T21:46:11.109 INFO:teuthology.orchestra.run.smithi092.stdout: "last_pg_merge_meta": { 2024-03-11T21:46:11.109 INFO:teuthology.orchestra.run.smithi092.stdout: "source_pgid": "0.0", 2024-03-11T21:46:11.109 INFO:teuthology.orchestra.run.smithi092.stdout: "ready_epoch": 0, 2024-03-11T21:46:11.109 INFO:teuthology.orchestra.run.smithi092.stdout: "last_epoch_started": 0, 2024-03-11T21:46:11.109 INFO:teuthology.orchestra.run.smithi092.stdout: "last_epoch_clean": 0, 2024-03-11T21:46:11.109 INFO:teuthology.orchestra.run.smithi092.stdout: "source_version": "0'0", 2024-03-11T21:46:11.109 INFO:teuthology.orchestra.run.smithi092.stdout: "target_version": "0'0" 2024-03-11T21:46:11.109 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:46:11.109 INFO:teuthology.orchestra.run.smithi092.stdout: "last_change": "71", 2024-03-11T21:46:11.109 INFO:teuthology.orchestra.run.smithi092.stdout: "last_force_op_resend": "0", 2024-03-11T21:46:11.110 INFO:teuthology.orchestra.run.smithi092.stdout: "last_force_op_resend_prenautilus": "0", 2024-03-11T21:46:11.110 INFO:teuthology.orchestra.run.smithi092.stdout: "last_force_op_resend_preluminous": "0", 2024-03-11T21:46:11.110 INFO:teuthology.orchestra.run.smithi092.stdout: "auid": 0, 2024-03-11T21:46:11.110 INFO:teuthology.orchestra.run.smithi092.stdout: "snap_mode": "selfmanaged", 2024-03-11T21:46:11.110 INFO:teuthology.orchestra.run.smithi092.stdout: "snap_seq": 2, 2024-03-11T21:46:11.110 INFO:teuthology.orchestra.run.smithi092.stdout: "snap_epoch": 71, 2024-03-11T21:46:11.110 INFO:teuthology.orchestra.run.smithi092.stdout: "pool_snaps": [], 2024-03-11T21:46:11.110 INFO:teuthology.orchestra.run.smithi092.stdout: "removed_snaps": "[]", 2024-03-11T21:46:11.110 INFO:teuthology.orchestra.run.smithi092.stdout: "quota_max_bytes": 0, 2024-03-11T21:46:11.110 INFO:teuthology.orchestra.run.smithi092.stdout: "quota_max_objects": 0, 2024-03-11T21:46:11.110 INFO:teuthology.orchestra.run.smithi092.stdout: "tiers": [], 2024-03-11T21:46:11.110 INFO:teuthology.orchestra.run.smithi092.stdout: "tier_of": -1, 2024-03-11T21:46:11.110 INFO:teuthology.orchestra.run.smithi092.stdout: "read_tier": -1, 2024-03-11T21:46:11.110 INFO:teuthology.orchestra.run.smithi092.stdout: "write_tier": -1, 2024-03-11T21:46:11.111 INFO:teuthology.orchestra.run.smithi092.stdout: "cache_mode": "none", 2024-03-11T21:46:11.111 INFO:teuthology.orchestra.run.smithi092.stdout: "target_max_bytes": 0, 2024-03-11T21:46:11.111 INFO:teuthology.orchestra.run.smithi092.stdout: "target_max_objects": 0, 2024-03-11T21:46:11.111 INFO:teuthology.orchestra.run.smithi092.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-03-11T21:46:11.111 INFO:teuthology.orchestra.run.smithi092.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-03-11T21:46:11.111 INFO:teuthology.orchestra.run.smithi092.stdout: "cache_target_full_ratio_micro": 800000, 2024-03-11T21:46:11.111 INFO:teuthology.orchestra.run.smithi092.stdout: "cache_min_flush_age": 0, 2024-03-11T21:46:11.111 INFO:teuthology.orchestra.run.smithi092.stdout: "cache_min_evict_age": 0, 2024-03-11T21:46:11.111 INFO:teuthology.orchestra.run.smithi092.stdout: "erasure_code_profile": "", 2024-03-11T21:46:11.111 INFO:teuthology.orchestra.run.smithi092.stdout: "hit_set_params": { 2024-03-11T21:46:11.111 INFO:teuthology.orchestra.run.smithi092.stdout: "type": "none" 2024-03-11T21:46:11.111 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:46:11.111 INFO:teuthology.orchestra.run.smithi092.stdout: "hit_set_period": 0, 2024-03-11T21:46:11.111 INFO:teuthology.orchestra.run.smithi092.stdout: "hit_set_count": 0, 2024-03-11T21:46:11.111 INFO:teuthology.orchestra.run.smithi092.stdout: "use_gmt_hitset": true, 2024-03-11T21:46:11.111 INFO:teuthology.orchestra.run.smithi092.stdout: "min_read_recency_for_promote": 0, 2024-03-11T21:46:11.112 INFO:teuthology.orchestra.run.smithi092.stdout: "min_write_recency_for_promote": 0, 2024-03-11T21:46:11.112 INFO:teuthology.orchestra.run.smithi092.stdout: "hit_set_grade_decay_rate": 0, 2024-03-11T21:46:11.112 INFO:teuthology.orchestra.run.smithi092.stdout: "hit_set_search_last_n": 0, 2024-03-11T21:46:11.112 INFO:teuthology.orchestra.run.smithi092.stdout: "grade_table": [], 2024-03-11T21:46:11.112 INFO:teuthology.orchestra.run.smithi092.stdout: "stripe_width": 0, 2024-03-11T21:46:11.112 INFO:teuthology.orchestra.run.smithi092.stdout: "expected_num_objects": 0, 2024-03-11T21:46:11.112 INFO:teuthology.orchestra.run.smithi092.stdout: "fast_read": false, 2024-03-11T21:46:11.112 INFO:teuthology.orchestra.run.smithi092.stdout: "options": {}, 2024-03-11T21:46:11.112 INFO:teuthology.orchestra.run.smithi092.stdout: "application_metadata": { 2024-03-11T21:46:11.112 INFO:teuthology.orchestra.run.smithi092.stdout: "rbd": {} 2024-03-11T21:46:11.112 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:46:11.112 INFO:teuthology.orchestra.run.smithi092.stdout: "read_balance": { 2024-03-11T21:46:11.112 INFO:teuthology.orchestra.run.smithi092.stdout: "score_type": "Fair distribution", 2024-03-11T21:46:11.112 INFO:teuthology.orchestra.run.smithi092.stdout: "score_acting": 2.6700000762939453, 2024-03-11T21:46:11.112 INFO:teuthology.orchestra.run.smithi092.stdout: "score_stable": 2.6700000762939453, 2024-03-11T21:46:11.113 INFO:teuthology.orchestra.run.smithi092.stdout: "optimal_score": 0.75, 2024-03-11T21:46:11.113 INFO:teuthology.orchestra.run.smithi092.stdout: "raw_score_acting": 2, 2024-03-11T21:46:11.113 INFO:teuthology.orchestra.run.smithi092.stdout: "raw_score_stable": 2, 2024-03-11T21:46:11.113 INFO:teuthology.orchestra.run.smithi092.stdout: "primary_affinity_weighted": 1, 2024-03-11T21:46:11.113 INFO:teuthology.orchestra.run.smithi092.stdout: "average_primary_affinity": 1, 2024-03-11T21:46:11.113 INFO:teuthology.orchestra.run.smithi092.stdout: "average_primary_affinity_weighted": 1 2024-03-11T21:46:11.113 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-03-11T21:46:11.113 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:46:11.113 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:46:11.113 INFO:teuthology.orchestra.run.smithi092.stdout: "pool": 4, 2024-03-11T21:46:11.113 INFO:teuthology.orchestra.run.smithi092.stdout: "pool_name": "default.rgw.log", 2024-03-11T21:46:11.113 INFO:teuthology.orchestra.run.smithi092.stdout: "create_time": "2024-03-11T21:25:38.850207+0000", 2024-03-11T21:46:11.113 INFO:teuthology.orchestra.run.smithi092.stdout: "flags": 1, 2024-03-11T21:46:11.113 INFO:teuthology.orchestra.run.smithi092.stdout: "flags_names": "hashpspool", 2024-03-11T21:46:11.113 INFO:teuthology.orchestra.run.smithi092.stdout: "type": 1, 2024-03-11T21:46:11.114 INFO:teuthology.orchestra.run.smithi092.stdout: "size": 3, 2024-03-11T21:46:11.114 INFO:teuthology.orchestra.run.smithi092.stdout: "min_size": 2, 2024-03-11T21:46:11.114 INFO:teuthology.orchestra.run.smithi092.stdout: "crush_rule": 0, 2024-03-11T21:46:11.114 INFO:teuthology.orchestra.run.smithi092.stdout: "peering_crush_bucket_count": 0, 2024-03-11T21:46:11.114 INFO:teuthology.orchestra.run.smithi092.stdout: "peering_crush_bucket_target": 0, 2024-03-11T21:46:11.114 INFO:teuthology.orchestra.run.smithi092.stdout: "peering_crush_bucket_barrier": 0, 2024-03-11T21:46:11.114 INFO:teuthology.orchestra.run.smithi092.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-03-11T21:46:11.114 INFO:teuthology.orchestra.run.smithi092.stdout: "object_hash": 2, 2024-03-11T21:46:11.114 INFO:teuthology.orchestra.run.smithi092.stdout: "pg_autoscale_mode": "off", 2024-03-11T21:46:11.114 INFO:teuthology.orchestra.run.smithi092.stdout: "pg_num": 32, 2024-03-11T21:46:11.114 INFO:teuthology.orchestra.run.smithi092.stdout: "pg_placement_num": 32, 2024-03-11T21:46:11.114 INFO:teuthology.orchestra.run.smithi092.stdout: "pg_placement_num_target": 32, 2024-03-11T21:46:11.114 INFO:teuthology.orchestra.run.smithi092.stdout: "pg_num_target": 32, 2024-03-11T21:46:11.114 INFO:teuthology.orchestra.run.smithi092.stdout: "pg_num_pending": 32, 2024-03-11T21:46:11.114 INFO:teuthology.orchestra.run.smithi092.stdout: "last_pg_merge_meta": { 2024-03-11T21:46:11.114 INFO:teuthology.orchestra.run.smithi092.stdout: "source_pgid": "0.0", 2024-03-11T21:46:11.115 INFO:teuthology.orchestra.run.smithi092.stdout: "ready_epoch": 0, 2024-03-11T21:46:11.115 INFO:teuthology.orchestra.run.smithi092.stdout: "last_epoch_started": 0, 2024-03-11T21:46:11.115 INFO:teuthology.orchestra.run.smithi092.stdout: "last_epoch_clean": 0, 2024-03-11T21:46:11.115 INFO:teuthology.orchestra.run.smithi092.stdout: "source_version": "0'0", 2024-03-11T21:46:11.115 INFO:teuthology.orchestra.run.smithi092.stdout: "target_version": "0'0" 2024-03-11T21:46:11.115 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:46:11.115 INFO:teuthology.orchestra.run.smithi092.stdout: "last_change": "70", 2024-03-11T21:46:11.115 INFO:teuthology.orchestra.run.smithi092.stdout: "last_force_op_resend": "0", 2024-03-11T21:46:11.115 INFO:teuthology.orchestra.run.smithi092.stdout: "last_force_op_resend_prenautilus": "0", 2024-03-11T21:46:11.115 INFO:teuthology.orchestra.run.smithi092.stdout: "last_force_op_resend_preluminous": "0", 2024-03-11T21:46:11.115 INFO:teuthology.orchestra.run.smithi092.stdout: "auid": 0, 2024-03-11T21:46:11.115 INFO:teuthology.orchestra.run.smithi092.stdout: "snap_mode": "selfmanaged", 2024-03-11T21:46:11.115 INFO:teuthology.orchestra.run.smithi092.stdout: "snap_seq": 0, 2024-03-11T21:46:11.115 INFO:teuthology.orchestra.run.smithi092.stdout: "snap_epoch": 0, 2024-03-11T21:46:11.115 INFO:teuthology.orchestra.run.smithi092.stdout: "pool_snaps": [], 2024-03-11T21:46:11.116 INFO:teuthology.orchestra.run.smithi092.stdout: "removed_snaps": "[]", 2024-03-11T21:46:11.116 INFO:teuthology.orchestra.run.smithi092.stdout: "quota_max_bytes": 0, 2024-03-11T21:46:11.116 INFO:teuthology.orchestra.run.smithi092.stdout: "quota_max_objects": 0, 2024-03-11T21:46:11.116 INFO:teuthology.orchestra.run.smithi092.stdout: "tiers": [], 2024-03-11T21:46:11.116 INFO:teuthology.orchestra.run.smithi092.stdout: "tier_of": -1, 2024-03-11T21:46:11.116 INFO:teuthology.orchestra.run.smithi092.stdout: "read_tier": -1, 2024-03-11T21:46:11.116 INFO:teuthology.orchestra.run.smithi092.stdout: "write_tier": -1, 2024-03-11T21:46:11.116 INFO:teuthology.orchestra.run.smithi092.stdout: "cache_mode": "none", 2024-03-11T21:46:11.116 INFO:teuthology.orchestra.run.smithi092.stdout: "target_max_bytes": 0, 2024-03-11T21:46:11.116 INFO:teuthology.orchestra.run.smithi092.stdout: "target_max_objects": 0, 2024-03-11T21:46:11.116 INFO:teuthology.orchestra.run.smithi092.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-03-11T21:46:11.116 INFO:teuthology.orchestra.run.smithi092.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-03-11T21:46:11.116 INFO:teuthology.orchestra.run.smithi092.stdout: "cache_target_full_ratio_micro": 800000, 2024-03-11T21:46:11.116 INFO:teuthology.orchestra.run.smithi092.stdout: "cache_min_flush_age": 0, 2024-03-11T21:46:11.116 INFO:teuthology.orchestra.run.smithi092.stdout: "cache_min_evict_age": 0, 2024-03-11T21:46:11.117 INFO:teuthology.orchestra.run.smithi092.stdout: "erasure_code_profile": "", 2024-03-11T21:46:11.117 INFO:teuthology.orchestra.run.smithi092.stdout: "hit_set_params": { 2024-03-11T21:46:11.117 INFO:teuthology.orchestra.run.smithi092.stdout: "type": "none" 2024-03-11T21:46:11.117 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:46:11.117 INFO:teuthology.orchestra.run.smithi092.stdout: "hit_set_period": 0, 2024-03-11T21:46:11.117 INFO:teuthology.orchestra.run.smithi092.stdout: "hit_set_count": 0, 2024-03-11T21:46:11.117 INFO:teuthology.orchestra.run.smithi092.stdout: "use_gmt_hitset": true, 2024-03-11T21:46:11.117 INFO:teuthology.orchestra.run.smithi092.stdout: "min_read_recency_for_promote": 0, 2024-03-11T21:46:11.117 INFO:teuthology.orchestra.run.smithi092.stdout: "min_write_recency_for_promote": 0, 2024-03-11T21:46:11.117 INFO:teuthology.orchestra.run.smithi092.stdout: "hit_set_grade_decay_rate": 0, 2024-03-11T21:46:11.117 INFO:teuthology.orchestra.run.smithi092.stdout: "hit_set_search_last_n": 0, 2024-03-11T21:46:11.117 INFO:teuthology.orchestra.run.smithi092.stdout: "grade_table": [], 2024-03-11T21:46:11.117 INFO:teuthology.orchestra.run.smithi092.stdout: "stripe_width": 0, 2024-03-11T21:46:11.117 INFO:teuthology.orchestra.run.smithi092.stdout: "expected_num_objects": 0, 2024-03-11T21:46:11.118 INFO:teuthology.orchestra.run.smithi092.stdout: "fast_read": false, 2024-03-11T21:46:11.118 INFO:teuthology.orchestra.run.smithi092.stdout: "options": {}, 2024-03-11T21:46:11.118 INFO:teuthology.orchestra.run.smithi092.stdout: "application_metadata": { 2024-03-11T21:46:11.118 INFO:teuthology.orchestra.run.smithi092.stdout: "rgw": {} 2024-03-11T21:46:11.118 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:46:11.118 INFO:teuthology.orchestra.run.smithi092.stdout: "read_balance": { 2024-03-11T21:46:11.118 INFO:teuthology.orchestra.run.smithi092.stdout: "score_type": "Fair distribution", 2024-03-11T21:46:11.118 INFO:teuthology.orchestra.run.smithi092.stdout: "score_acting": 2.25, 2024-03-11T21:46:11.118 INFO:teuthology.orchestra.run.smithi092.stdout: "score_stable": 2.25, 2024-03-11T21:46:11.118 INFO:teuthology.orchestra.run.smithi092.stdout: "optimal_score": 0.75, 2024-03-11T21:46:11.118 INFO:teuthology.orchestra.run.smithi092.stdout: "raw_score_acting": 1.690000057220459, 2024-03-11T21:46:11.118 INFO:teuthology.orchestra.run.smithi092.stdout: "raw_score_stable": 1.690000057220459, 2024-03-11T21:46:11.118 INFO:teuthology.orchestra.run.smithi092.stdout: "primary_affinity_weighted": 1, 2024-03-11T21:46:11.118 INFO:teuthology.orchestra.run.smithi092.stdout: "average_primary_affinity": 1, 2024-03-11T21:46:11.119 INFO:teuthology.orchestra.run.smithi092.stdout: "average_primary_affinity_weighted": 1 2024-03-11T21:46:11.119 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-03-11T21:46:11.119 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:46:11.119 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:46:11.119 INFO:teuthology.orchestra.run.smithi092.stdout: "pool": 5, 2024-03-11T21:46:11.119 INFO:teuthology.orchestra.run.smithi092.stdout: "pool_name": "default.rgw.control", 2024-03-11T21:46:11.119 INFO:teuthology.orchestra.run.smithi092.stdout: "create_time": "2024-03-11T21:25:40.794686+0000", 2024-03-11T21:46:11.119 INFO:teuthology.orchestra.run.smithi092.stdout: "flags": 1, 2024-03-11T21:46:11.119 INFO:teuthology.orchestra.run.smithi092.stdout: "flags_names": "hashpspool", 2024-03-11T21:46:11.119 INFO:teuthology.orchestra.run.smithi092.stdout: "type": 1, 2024-03-11T21:46:11.119 INFO:teuthology.orchestra.run.smithi092.stdout: "size": 3, 2024-03-11T21:46:11.119 INFO:teuthology.orchestra.run.smithi092.stdout: "min_size": 2, 2024-03-11T21:46:11.119 INFO:teuthology.orchestra.run.smithi092.stdout: "crush_rule": 0, 2024-03-11T21:46:11.119 INFO:teuthology.orchestra.run.smithi092.stdout: "peering_crush_bucket_count": 0, 2024-03-11T21:46:11.119 INFO:teuthology.orchestra.run.smithi092.stdout: "peering_crush_bucket_target": 0, 2024-03-11T21:46:11.119 INFO:teuthology.orchestra.run.smithi092.stdout: "peering_crush_bucket_barrier": 0, 2024-03-11T21:46:11.120 INFO:teuthology.orchestra.run.smithi092.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-03-11T21:46:11.120 INFO:teuthology.orchestra.run.smithi092.stdout: "object_hash": 2, 2024-03-11T21:46:11.120 INFO:teuthology.orchestra.run.smithi092.stdout: "pg_autoscale_mode": "off", 2024-03-11T21:46:11.120 INFO:teuthology.orchestra.run.smithi092.stdout: "pg_num": 32, 2024-03-11T21:46:11.120 INFO:teuthology.orchestra.run.smithi092.stdout: "pg_placement_num": 32, 2024-03-11T21:46:11.120 INFO:teuthology.orchestra.run.smithi092.stdout: "pg_placement_num_target": 32, 2024-03-11T21:46:11.120 INFO:teuthology.orchestra.run.smithi092.stdout: "pg_num_target": 32, 2024-03-11T21:46:11.120 INFO:teuthology.orchestra.run.smithi092.stdout: "pg_num_pending": 32, 2024-03-11T21:46:11.120 INFO:teuthology.orchestra.run.smithi092.stdout: "last_pg_merge_meta": { 2024-03-11T21:46:11.120 INFO:teuthology.orchestra.run.smithi092.stdout: "source_pgid": "0.0", 2024-03-11T21:46:11.120 INFO:teuthology.orchestra.run.smithi092.stdout: "ready_epoch": 0, 2024-03-11T21:46:11.120 INFO:teuthology.orchestra.run.smithi092.stdout: "last_epoch_started": 0, 2024-03-11T21:46:11.120 INFO:teuthology.orchestra.run.smithi092.stdout: "last_epoch_clean": 0, 2024-03-11T21:46:11.120 INFO:teuthology.orchestra.run.smithi092.stdout: "source_version": "0'0", 2024-03-11T21:46:11.120 INFO:teuthology.orchestra.run.smithi092.stdout: "target_version": "0'0" 2024-03-11T21:46:11.120 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:46:11.121 INFO:teuthology.orchestra.run.smithi092.stdout: "last_change": "72", 2024-03-11T21:46:11.121 INFO:teuthology.orchestra.run.smithi092.stdout: "last_force_op_resend": "0", 2024-03-11T21:46:11.121 INFO:teuthology.orchestra.run.smithi092.stdout: "last_force_op_resend_prenautilus": "0", 2024-03-11T21:46:11.121 INFO:teuthology.orchestra.run.smithi092.stdout: "last_force_op_resend_preluminous": "0", 2024-03-11T21:46:11.121 INFO:teuthology.orchestra.run.smithi092.stdout: "auid": 0, 2024-03-11T21:46:11.121 INFO:teuthology.orchestra.run.smithi092.stdout: "snap_mode": "selfmanaged", 2024-03-11T21:46:11.121 INFO:teuthology.orchestra.run.smithi092.stdout: "snap_seq": 0, 2024-03-11T21:46:11.121 INFO:teuthology.orchestra.run.smithi092.stdout: "snap_epoch": 0, 2024-03-11T21:46:11.121 INFO:teuthology.orchestra.run.smithi092.stdout: "pool_snaps": [], 2024-03-11T21:46:11.121 INFO:teuthology.orchestra.run.smithi092.stdout: "removed_snaps": "[]", 2024-03-11T21:46:11.121 INFO:teuthology.orchestra.run.smithi092.stdout: "quota_max_bytes": 0, 2024-03-11T21:46:11.121 INFO:teuthology.orchestra.run.smithi092.stdout: "quota_max_objects": 0, 2024-03-11T21:46:11.121 INFO:teuthology.orchestra.run.smithi092.stdout: "tiers": [], 2024-03-11T21:46:11.121 INFO:teuthology.orchestra.run.smithi092.stdout: "tier_of": -1, 2024-03-11T21:46:11.121 INFO:teuthology.orchestra.run.smithi092.stdout: "read_tier": -1, 2024-03-11T21:46:11.121 INFO:teuthology.orchestra.run.smithi092.stdout: "write_tier": -1, 2024-03-11T21:46:11.122 INFO:teuthology.orchestra.run.smithi092.stdout: "cache_mode": "none", 2024-03-11T21:46:11.122 INFO:teuthology.orchestra.run.smithi092.stdout: "target_max_bytes": 0, 2024-03-11T21:46:11.122 INFO:teuthology.orchestra.run.smithi092.stdout: "target_max_objects": 0, 2024-03-11T21:46:11.122 INFO:teuthology.orchestra.run.smithi092.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-03-11T21:46:11.122 INFO:teuthology.orchestra.run.smithi092.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-03-11T21:46:11.122 INFO:teuthology.orchestra.run.smithi092.stdout: "cache_target_full_ratio_micro": 800000, 2024-03-11T21:46:11.122 INFO:teuthology.orchestra.run.smithi092.stdout: "cache_min_flush_age": 0, 2024-03-11T21:46:11.122 INFO:teuthology.orchestra.run.smithi092.stdout: "cache_min_evict_age": 0, 2024-03-11T21:46:11.122 INFO:teuthology.orchestra.run.smithi092.stdout: "erasure_code_profile": "", 2024-03-11T21:46:11.122 INFO:teuthology.orchestra.run.smithi092.stdout: "hit_set_params": { 2024-03-11T21:46:11.122 INFO:teuthology.orchestra.run.smithi092.stdout: "type": "none" 2024-03-11T21:46:11.122 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:46:11.122 INFO:teuthology.orchestra.run.smithi092.stdout: "hit_set_period": 0, 2024-03-11T21:46:11.122 INFO:teuthology.orchestra.run.smithi092.stdout: "hit_set_count": 0, 2024-03-11T21:46:11.122 INFO:teuthology.orchestra.run.smithi092.stdout: "use_gmt_hitset": true, 2024-03-11T21:46:11.122 INFO:teuthology.orchestra.run.smithi092.stdout: "min_read_recency_for_promote": 0, 2024-03-11T21:46:11.123 INFO:teuthology.orchestra.run.smithi092.stdout: "min_write_recency_for_promote": 0, 2024-03-11T21:46:11.123 INFO:teuthology.orchestra.run.smithi092.stdout: "hit_set_grade_decay_rate": 0, 2024-03-11T21:46:11.123 INFO:teuthology.orchestra.run.smithi092.stdout: "hit_set_search_last_n": 0, 2024-03-11T21:46:11.123 INFO:teuthology.orchestra.run.smithi092.stdout: "grade_table": [], 2024-03-11T21:46:11.123 INFO:teuthology.orchestra.run.smithi092.stdout: "stripe_width": 0, 2024-03-11T21:46:11.123 INFO:teuthology.orchestra.run.smithi092.stdout: "expected_num_objects": 0, 2024-03-11T21:46:11.123 INFO:teuthology.orchestra.run.smithi092.stdout: "fast_read": false, 2024-03-11T21:46:11.123 INFO:teuthology.orchestra.run.smithi092.stdout: "options": {}, 2024-03-11T21:46:11.123 INFO:teuthology.orchestra.run.smithi092.stdout: "application_metadata": { 2024-03-11T21:46:11.123 INFO:teuthology.orchestra.run.smithi092.stdout: "rgw": {} 2024-03-11T21:46:11.123 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:46:11.123 INFO:teuthology.orchestra.run.smithi092.stdout: "read_balance": { 2024-03-11T21:46:11.123 INFO:teuthology.orchestra.run.smithi092.stdout: "score_type": "Fair distribution", 2024-03-11T21:46:11.123 INFO:teuthology.orchestra.run.smithi092.stdout: "score_acting": 1.5099999904632568, 2024-03-11T21:46:11.123 INFO:teuthology.orchestra.run.smithi092.stdout: "score_stable": 1.5099999904632568, 2024-03-11T21:46:11.124 INFO:teuthology.orchestra.run.smithi092.stdout: "optimal_score": 0.75, 2024-03-11T21:46:11.124 INFO:teuthology.orchestra.run.smithi092.stdout: "raw_score_acting": 1.1299999952316284, 2024-03-11T21:46:11.124 INFO:teuthology.orchestra.run.smithi092.stdout: "raw_score_stable": 1.1299999952316284, 2024-03-11T21:46:11.124 INFO:teuthology.orchestra.run.smithi092.stdout: "primary_affinity_weighted": 1, 2024-03-11T21:46:11.124 INFO:teuthology.orchestra.run.smithi092.stdout: "average_primary_affinity": 1, 2024-03-11T21:46:11.124 INFO:teuthology.orchestra.run.smithi092.stdout: "average_primary_affinity_weighted": 1 2024-03-11T21:46:11.124 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-03-11T21:46:11.124 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:46:11.124 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:46:11.124 INFO:teuthology.orchestra.run.smithi092.stdout: "pool": 6, 2024-03-11T21:46:11.124 INFO:teuthology.orchestra.run.smithi092.stdout: "pool_name": "default.rgw.meta", 2024-03-11T21:46:11.125 INFO:teuthology.orchestra.run.smithi092.stdout: "create_time": "2024-03-11T21:25:42.872077+0000", 2024-03-11T21:46:11.125 INFO:teuthology.orchestra.run.smithi092.stdout: "flags": 1, 2024-03-11T21:46:11.125 INFO:teuthology.orchestra.run.smithi092.stdout: "flags_names": "hashpspool", 2024-03-11T21:46:11.125 INFO:teuthology.orchestra.run.smithi092.stdout: "type": 1, 2024-03-11T21:46:11.125 INFO:teuthology.orchestra.run.smithi092.stdout: "size": 3, 2024-03-11T21:46:11.125 INFO:teuthology.orchestra.run.smithi092.stdout: "min_size": 2, 2024-03-11T21:46:11.125 INFO:teuthology.orchestra.run.smithi092.stdout: "crush_rule": 0, 2024-03-11T21:46:11.125 INFO:teuthology.orchestra.run.smithi092.stdout: "peering_crush_bucket_count": 0, 2024-03-11T21:46:11.125 INFO:teuthology.orchestra.run.smithi092.stdout: "peering_crush_bucket_target": 0, 2024-03-11T21:46:11.125 INFO:teuthology.orchestra.run.smithi092.stdout: "peering_crush_bucket_barrier": 0, 2024-03-11T21:46:11.125 INFO:teuthology.orchestra.run.smithi092.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-03-11T21:46:11.125 INFO:teuthology.orchestra.run.smithi092.stdout: "object_hash": 2, 2024-03-11T21:46:11.126 INFO:teuthology.orchestra.run.smithi092.stdout: "pg_autoscale_mode": "off", 2024-03-11T21:46:11.126 INFO:teuthology.orchestra.run.smithi092.stdout: "pg_num": 32, 2024-03-11T21:46:11.126 INFO:teuthology.orchestra.run.smithi092.stdout: "pg_placement_num": 32, 2024-03-11T21:46:11.126 INFO:teuthology.orchestra.run.smithi092.stdout: "pg_placement_num_target": 32, 2024-03-11T21:46:11.126 INFO:teuthology.orchestra.run.smithi092.stdout: "pg_num_target": 32, 2024-03-11T21:46:11.126 INFO:teuthology.orchestra.run.smithi092.stdout: "pg_num_pending": 32, 2024-03-11T21:46:11.126 INFO:teuthology.orchestra.run.smithi092.stdout: "last_pg_merge_meta": { 2024-03-11T21:46:11.126 INFO:teuthology.orchestra.run.smithi092.stdout: "source_pgid": "0.0", 2024-03-11T21:46:11.126 INFO:teuthology.orchestra.run.smithi092.stdout: "ready_epoch": 0, 2024-03-11T21:46:11.126 INFO:teuthology.orchestra.run.smithi092.stdout: "last_epoch_started": 0, 2024-03-11T21:46:11.126 INFO:teuthology.orchestra.run.smithi092.stdout: "last_epoch_clean": 0, 2024-03-11T21:46:11.126 INFO:teuthology.orchestra.run.smithi092.stdout: "source_version": "0'0", 2024-03-11T21:46:11.126 INFO:teuthology.orchestra.run.smithi092.stdout: "target_version": "0'0" 2024-03-11T21:46:11.126 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:46:11.127 INFO:teuthology.orchestra.run.smithi092.stdout: "last_change": "74", 2024-03-11T21:46:11.127 INFO:teuthology.orchestra.run.smithi092.stdout: "last_force_op_resend": "0", 2024-03-11T21:46:11.127 INFO:teuthology.orchestra.run.smithi092.stdout: "last_force_op_resend_prenautilus": "0", 2024-03-11T21:46:11.127 INFO:teuthology.orchestra.run.smithi092.stdout: "last_force_op_resend_preluminous": "0", 2024-03-11T21:46:11.127 INFO:teuthology.orchestra.run.smithi092.stdout: "auid": 0, 2024-03-11T21:46:11.127 INFO:teuthology.orchestra.run.smithi092.stdout: "snap_mode": "selfmanaged", 2024-03-11T21:46:11.127 INFO:teuthology.orchestra.run.smithi092.stdout: "snap_seq": 0, 2024-03-11T21:46:11.127 INFO:teuthology.orchestra.run.smithi092.stdout: "snap_epoch": 0, 2024-03-11T21:46:11.127 INFO:teuthology.orchestra.run.smithi092.stdout: "pool_snaps": [], 2024-03-11T21:46:11.127 INFO:teuthology.orchestra.run.smithi092.stdout: "removed_snaps": "[]", 2024-03-11T21:46:11.127 INFO:teuthology.orchestra.run.smithi092.stdout: "quota_max_bytes": 0, 2024-03-11T21:46:11.127 INFO:teuthology.orchestra.run.smithi092.stdout: "quota_max_objects": 0, 2024-03-11T21:46:11.127 INFO:teuthology.orchestra.run.smithi092.stdout: "tiers": [], 2024-03-11T21:46:11.128 INFO:teuthology.orchestra.run.smithi092.stdout: "tier_of": -1, 2024-03-11T21:46:11.128 INFO:teuthology.orchestra.run.smithi092.stdout: "read_tier": -1, 2024-03-11T21:46:11.128 INFO:teuthology.orchestra.run.smithi092.stdout: "write_tier": -1, 2024-03-11T21:46:11.128 INFO:teuthology.orchestra.run.smithi092.stdout: "cache_mode": "none", 2024-03-11T21:46:11.128 INFO:teuthology.orchestra.run.smithi092.stdout: "target_max_bytes": 0, 2024-03-11T21:46:11.128 INFO:teuthology.orchestra.run.smithi092.stdout: "target_max_objects": 0, 2024-03-11T21:46:11.128 INFO:teuthology.orchestra.run.smithi092.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-03-11T21:46:11.128 INFO:teuthology.orchestra.run.smithi092.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-03-11T21:46:11.128 INFO:teuthology.orchestra.run.smithi092.stdout: "cache_target_full_ratio_micro": 800000, 2024-03-11T21:46:11.128 INFO:teuthology.orchestra.run.smithi092.stdout: "cache_min_flush_age": 0, 2024-03-11T21:46:11.128 INFO:teuthology.orchestra.run.smithi092.stdout: "cache_min_evict_age": 0, 2024-03-11T21:46:11.128 INFO:teuthology.orchestra.run.smithi092.stdout: "erasure_code_profile": "", 2024-03-11T21:46:11.128 INFO:teuthology.orchestra.run.smithi092.stdout: "hit_set_params": { 2024-03-11T21:46:11.128 INFO:teuthology.orchestra.run.smithi092.stdout: "type": "none" 2024-03-11T21:46:11.128 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:46:11.129 INFO:teuthology.orchestra.run.smithi092.stdout: "hit_set_period": 0, 2024-03-11T21:46:11.129 INFO:teuthology.orchestra.run.smithi092.stdout: "hit_set_count": 0, 2024-03-11T21:46:11.129 INFO:teuthology.orchestra.run.smithi092.stdout: "use_gmt_hitset": true, 2024-03-11T21:46:11.129 INFO:teuthology.orchestra.run.smithi092.stdout: "min_read_recency_for_promote": 0, 2024-03-11T21:46:11.129 INFO:teuthology.orchestra.run.smithi092.stdout: "min_write_recency_for_promote": 0, 2024-03-11T21:46:11.129 INFO:teuthology.orchestra.run.smithi092.stdout: "hit_set_grade_decay_rate": 0, 2024-03-11T21:46:11.129 INFO:teuthology.orchestra.run.smithi092.stdout: "hit_set_search_last_n": 0, 2024-03-11T21:46:11.129 INFO:teuthology.orchestra.run.smithi092.stdout: "grade_table": [], 2024-03-11T21:46:11.129 INFO:teuthology.orchestra.run.smithi092.stdout: "stripe_width": 0, 2024-03-11T21:46:11.129 INFO:teuthology.orchestra.run.smithi092.stdout: "expected_num_objects": 0, 2024-03-11T21:46:11.129 INFO:teuthology.orchestra.run.smithi092.stdout: "fast_read": false, 2024-03-11T21:46:11.129 INFO:teuthology.orchestra.run.smithi092.stdout: "options": { 2024-03-11T21:46:11.129 INFO:teuthology.orchestra.run.smithi092.stdout: "pg_autoscale_bias": 4 2024-03-11T21:46:11.129 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:46:11.129 INFO:teuthology.orchestra.run.smithi092.stdout: "application_metadata": { 2024-03-11T21:46:11.129 INFO:teuthology.orchestra.run.smithi092.stdout: "rgw": {} 2024-03-11T21:46:11.130 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:46:11.130 INFO:teuthology.orchestra.run.smithi092.stdout: "read_balance": { 2024-03-11T21:46:11.130 INFO:teuthology.orchestra.run.smithi092.stdout: "score_type": "Fair distribution", 2024-03-11T21:46:11.130 INFO:teuthology.orchestra.run.smithi092.stdout: "score_acting": 2.25, 2024-03-11T21:46:11.130 INFO:teuthology.orchestra.run.smithi092.stdout: "score_stable": 2.25, 2024-03-11T21:46:11.130 INFO:teuthology.orchestra.run.smithi092.stdout: "optimal_score": 0.75, 2024-03-11T21:46:11.130 INFO:teuthology.orchestra.run.smithi092.stdout: "raw_score_acting": 1.690000057220459, 2024-03-11T21:46:11.130 INFO:teuthology.orchestra.run.smithi092.stdout: "raw_score_stable": 1.690000057220459, 2024-03-11T21:46:11.130 INFO:teuthology.orchestra.run.smithi092.stdout: "primary_affinity_weighted": 1, 2024-03-11T21:46:11.130 INFO:teuthology.orchestra.run.smithi092.stdout: "average_primary_affinity": 1, 2024-03-11T21:46:11.130 INFO:teuthology.orchestra.run.smithi092.stdout: "average_primary_affinity_weighted": 1 2024-03-11T21:46:11.130 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-03-11T21:46:11.130 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:46:11.130 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:46:11.130 INFO:teuthology.orchestra.run.smithi092.stdout: "pool": 11, 2024-03-11T21:46:11.131 INFO:teuthology.orchestra.run.smithi092.stdout: "pool_name": "unique_pool_4", 2024-03-11T21:46:11.131 INFO:teuthology.orchestra.run.smithi092.stdout: "create_time": "2024-03-11T21:44:52.522762+0000", 2024-03-11T21:46:11.131 INFO:teuthology.orchestra.run.smithi092.stdout: "flags": 1, 2024-03-11T21:46:11.131 INFO:teuthology.orchestra.run.smithi092.stdout: "flags_names": "hashpspool", 2024-03-11T21:46:11.131 INFO:teuthology.orchestra.run.smithi092.stdout: "type": 1, 2024-03-11T21:46:11.131 INFO:teuthology.orchestra.run.smithi092.stdout: "size": 3, 2024-03-11T21:46:11.131 INFO:teuthology.orchestra.run.smithi092.stdout: "min_size": 2, 2024-03-11T21:46:11.131 INFO:teuthology.orchestra.run.smithi092.stdout: "crush_rule": 0, 2024-03-11T21:46:11.131 INFO:teuthology.orchestra.run.smithi092.stdout: "peering_crush_bucket_count": 0, 2024-03-11T21:46:11.131 INFO:teuthology.orchestra.run.smithi092.stdout: "peering_crush_bucket_target": 0, 2024-03-11T21:46:11.131 INFO:teuthology.orchestra.run.smithi092.stdout: "peering_crush_bucket_barrier": 0, 2024-03-11T21:46:11.131 INFO:teuthology.orchestra.run.smithi092.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-03-11T21:46:11.131 INFO:teuthology.orchestra.run.smithi092.stdout: "object_hash": 2, 2024-03-11T21:46:11.131 INFO:teuthology.orchestra.run.smithi092.stdout: "pg_autoscale_mode": "off", 2024-03-11T21:46:11.132 INFO:teuthology.orchestra.run.smithi092.stdout: "pg_num": 16, 2024-03-11T21:46:11.132 INFO:teuthology.orchestra.run.smithi092.stdout: "pg_placement_num": 16, 2024-03-11T21:46:11.132 INFO:teuthology.orchestra.run.smithi092.stdout: "pg_placement_num_target": 16, 2024-03-11T21:46:11.132 INFO:teuthology.orchestra.run.smithi092.stdout: "pg_num_target": 16, 2024-03-11T21:46:11.132 INFO:teuthology.orchestra.run.smithi092.stdout: "pg_num_pending": 16, 2024-03-11T21:46:11.132 INFO:teuthology.orchestra.run.smithi092.stdout: "last_pg_merge_meta": { 2024-03-11T21:46:11.132 INFO:teuthology.orchestra.run.smithi092.stdout: "source_pgid": "0.0", 2024-03-11T21:46:11.132 INFO:teuthology.orchestra.run.smithi092.stdout: "ready_epoch": 0, 2024-03-11T21:46:11.132 INFO:teuthology.orchestra.run.smithi092.stdout: "last_epoch_started": 0, 2024-03-11T21:46:11.132 INFO:teuthology.orchestra.run.smithi092.stdout: "last_epoch_clean": 0, 2024-03-11T21:46:11.132 INFO:teuthology.orchestra.run.smithi092.stdout: "source_version": "0'0", 2024-03-11T21:46:11.132 INFO:teuthology.orchestra.run.smithi092.stdout: "target_version": "0'0" 2024-03-11T21:46:11.132 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:46:11.132 INFO:teuthology.orchestra.run.smithi092.stdout: "last_change": "450", 2024-03-11T21:46:11.132 INFO:teuthology.orchestra.run.smithi092.stdout: "last_force_op_resend": "0", 2024-03-11T21:46:11.132 INFO:teuthology.orchestra.run.smithi092.stdout: "last_force_op_resend_prenautilus": "0", 2024-03-11T21:46:11.133 INFO:teuthology.orchestra.run.smithi092.stdout: "last_force_op_resend_preluminous": "0", 2024-03-11T21:46:11.133 INFO:teuthology.orchestra.run.smithi092.stdout: "auid": 0, 2024-03-11T21:46:11.133 INFO:teuthology.orchestra.run.smithi092.stdout: "snap_mode": "selfmanaged", 2024-03-11T21:46:11.133 INFO:teuthology.orchestra.run.smithi092.stdout: "snap_seq": 0, 2024-03-11T21:46:11.133 INFO:teuthology.orchestra.run.smithi092.stdout: "snap_epoch": 0, 2024-03-11T21:46:11.133 INFO:teuthology.orchestra.run.smithi092.stdout: "pool_snaps": [], 2024-03-11T21:46:11.133 INFO:teuthology.orchestra.run.smithi092.stdout: "removed_snaps": "[]", 2024-03-11T21:46:11.133 INFO:teuthology.orchestra.run.smithi092.stdout: "quota_max_bytes": 0, 2024-03-11T21:46:11.133 INFO:teuthology.orchestra.run.smithi092.stdout: "quota_max_objects": 0, 2024-03-11T21:46:11.133 INFO:teuthology.orchestra.run.smithi092.stdout: "tiers": [], 2024-03-11T21:46:11.133 INFO:teuthology.orchestra.run.smithi092.stdout: "tier_of": -1, 2024-03-11T21:46:11.133 INFO:teuthology.orchestra.run.smithi092.stdout: "read_tier": -1, 2024-03-11T21:46:11.133 INFO:teuthology.orchestra.run.smithi092.stdout: "write_tier": -1, 2024-03-11T21:46:11.133 INFO:teuthology.orchestra.run.smithi092.stdout: "cache_mode": "none", 2024-03-11T21:46:11.134 INFO:teuthology.orchestra.run.smithi092.stdout: "target_max_bytes": 0, 2024-03-11T21:46:11.134 INFO:teuthology.orchestra.run.smithi092.stdout: "target_max_objects": 0, 2024-03-11T21:46:11.134 INFO:teuthology.orchestra.run.smithi092.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-03-11T21:46:11.134 INFO:teuthology.orchestra.run.smithi092.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-03-11T21:46:11.134 INFO:teuthology.orchestra.run.smithi092.stdout: "cache_target_full_ratio_micro": 800000, 2024-03-11T21:46:11.134 INFO:teuthology.orchestra.run.smithi092.stdout: "cache_min_flush_age": 0, 2024-03-11T21:46:11.134 INFO:teuthology.orchestra.run.smithi092.stdout: "cache_min_evict_age": 0, 2024-03-11T21:46:11.134 INFO:teuthology.orchestra.run.smithi092.stdout: "erasure_code_profile": "", 2024-03-11T21:46:11.134 INFO:teuthology.orchestra.run.smithi092.stdout: "hit_set_params": { 2024-03-11T21:46:11.134 INFO:teuthology.orchestra.run.smithi092.stdout: "type": "none" 2024-03-11T21:46:11.134 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:46:11.134 INFO:teuthology.orchestra.run.smithi092.stdout: "hit_set_period": 0, 2024-03-11T21:46:11.134 INFO:teuthology.orchestra.run.smithi092.stdout: "hit_set_count": 0, 2024-03-11T21:46:11.134 INFO:teuthology.orchestra.run.smithi092.stdout: "use_gmt_hitset": true, 2024-03-11T21:46:11.134 INFO:teuthology.orchestra.run.smithi092.stdout: "min_read_recency_for_promote": 0, 2024-03-11T21:46:11.135 INFO:teuthology.orchestra.run.smithi092.stdout: "min_write_recency_for_promote": 0, 2024-03-11T21:46:11.135 INFO:teuthology.orchestra.run.smithi092.stdout: "hit_set_grade_decay_rate": 0, 2024-03-11T21:46:11.135 INFO:teuthology.orchestra.run.smithi092.stdout: "hit_set_search_last_n": 0, 2024-03-11T21:46:11.135 INFO:teuthology.orchestra.run.smithi092.stdout: "grade_table": [], 2024-03-11T21:46:11.135 INFO:teuthology.orchestra.run.smithi092.stdout: "stripe_width": 0, 2024-03-11T21:46:11.135 INFO:teuthology.orchestra.run.smithi092.stdout: "expected_num_objects": 0, 2024-03-11T21:46:11.135 INFO:teuthology.orchestra.run.smithi092.stdout: "fast_read": false, 2024-03-11T21:46:11.135 INFO:teuthology.orchestra.run.smithi092.stdout: "options": {}, 2024-03-11T21:46:11.135 INFO:teuthology.orchestra.run.smithi092.stdout: "application_metadata": { 2024-03-11T21:46:11.135 INFO:teuthology.orchestra.run.smithi092.stdout: "rados": {} 2024-03-11T21:46:11.135 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:46:11.135 INFO:teuthology.orchestra.run.smithi092.stdout: "read_balance": { 2024-03-11T21:46:11.135 INFO:teuthology.orchestra.run.smithi092.stdout: "score_type": "Fair distribution", 2024-03-11T21:46:11.135 INFO:teuthology.orchestra.run.smithi092.stdout: "score_acting": 2.5099999904632568, 2024-03-11T21:46:11.135 INFO:teuthology.orchestra.run.smithi092.stdout: "score_stable": 2.5099999904632568, 2024-03-11T21:46:11.136 INFO:teuthology.orchestra.run.smithi092.stdout: "optimal_score": 0.75, 2024-03-11T21:46:11.136 INFO:teuthology.orchestra.run.smithi092.stdout: "raw_score_acting": 1.8799999952316284, 2024-03-11T21:46:11.136 INFO:teuthology.orchestra.run.smithi092.stdout: "raw_score_stable": 1.8799999952316284, 2024-03-11T21:46:11.136 INFO:teuthology.orchestra.run.smithi092.stdout: "primary_affinity_weighted": 1, 2024-03-11T21:46:11.136 INFO:teuthology.orchestra.run.smithi092.stdout: "average_primary_affinity": 1, 2024-03-11T21:46:11.136 INFO:teuthology.orchestra.run.smithi092.stdout: "average_primary_affinity_weighted": 1 2024-03-11T21:46:11.136 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-03-11T21:46:11.136 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-03-11T21:46:11.136 INFO:teuthology.orchestra.run.smithi092.stdout: ], 2024-03-11T21:46:11.136 INFO:teuthology.orchestra.run.smithi092.stdout: "osds": [ 2024-03-11T21:46:11.136 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:46:11.136 INFO:teuthology.orchestra.run.smithi092.stdout: "osd": 0, 2024-03-11T21:46:11.136 INFO:teuthology.orchestra.run.smithi092.stdout: "uuid": "79f15843-f0ec-44aa-931e-4e07a74d3768", 2024-03-11T21:46:11.136 INFO:teuthology.orchestra.run.smithi092.stdout: "up": 1, 2024-03-11T21:46:11.136 INFO:teuthology.orchestra.run.smithi092.stdout: "in": 1, 2024-03-11T21:46:11.137 INFO:teuthology.orchestra.run.smithi092.stdout: "weight": 1, 2024-03-11T21:46:11.137 INFO:teuthology.orchestra.run.smithi092.stdout: "primary_affinity": 1, 2024-03-11T21:46:11.137 INFO:teuthology.orchestra.run.smithi092.stdout: "last_clean_begin": 0, 2024-03-11T21:46:11.137 INFO:teuthology.orchestra.run.smithi092.stdout: "last_clean_end": 0, 2024-03-11T21:46:11.137 INFO:teuthology.orchestra.run.smithi092.stdout: "up_from": 8, 2024-03-11T21:46:11.137 INFO:teuthology.orchestra.run.smithi092.stdout: "up_thru": 446, 2024-03-11T21:46:11.137 INFO:teuthology.orchestra.run.smithi092.stdout: "down_at": 0, 2024-03-11T21:46:11.137 INFO:teuthology.orchestra.run.smithi092.stdout: "lost_at": 0, 2024-03-11T21:46:11.137 INFO:teuthology.orchestra.run.smithi092.stdout: "public_addrs": { 2024-03-11T21:46:11.137 INFO:teuthology.orchestra.run.smithi092.stdout: "addrvec": [ 2024-03-11T21:46:11.137 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:46:11.137 INFO:teuthology.orchestra.run.smithi092.stdout: "type": "v1", 2024-03-11T21:46:11.137 INFO:teuthology.orchestra.run.smithi092.stdout: "addr": "172.21.15.92:6801", 2024-03-11T21:46:11.137 INFO:teuthology.orchestra.run.smithi092.stdout: "nonce": 2844100201 2024-03-11T21:46:11.138 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-03-11T21:46:11.138 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-03-11T21:46:11.138 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:46:11.138 INFO:teuthology.orchestra.run.smithi092.stdout: "cluster_addrs": { 2024-03-11T21:46:11.138 INFO:teuthology.orchestra.run.smithi092.stdout: "addrvec": [ 2024-03-11T21:46:11.138 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:46:11.138 INFO:teuthology.orchestra.run.smithi092.stdout: "type": "v1", 2024-03-11T21:46:11.138 INFO:teuthology.orchestra.run.smithi092.stdout: "addr": "172.21.15.92:6802", 2024-03-11T21:46:11.138 INFO:teuthology.orchestra.run.smithi092.stdout: "nonce": 2844100201 2024-03-11T21:46:11.138 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-03-11T21:46:11.138 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-03-11T21:46:11.138 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:46:11.138 INFO:teuthology.orchestra.run.smithi092.stdout: "heartbeat_back_addrs": { 2024-03-11T21:46:11.138 INFO:teuthology.orchestra.run.smithi092.stdout: "addrvec": [ 2024-03-11T21:46:11.139 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:46:11.139 INFO:teuthology.orchestra.run.smithi092.stdout: "type": "v1", 2024-03-11T21:46:11.139 INFO:teuthology.orchestra.run.smithi092.stdout: "addr": "172.21.15.92:6804", 2024-03-11T21:46:11.139 INFO:teuthology.orchestra.run.smithi092.stdout: "nonce": 2844100201 2024-03-11T21:46:11.139 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-03-11T21:46:11.139 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-03-11T21:46:11.139 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:46:11.139 INFO:teuthology.orchestra.run.smithi092.stdout: "heartbeat_front_addrs": { 2024-03-11T21:46:11.139 INFO:teuthology.orchestra.run.smithi092.stdout: "addrvec": [ 2024-03-11T21:46:11.139 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:46:11.139 INFO:teuthology.orchestra.run.smithi092.stdout: "type": "v1", 2024-03-11T21:46:11.139 INFO:teuthology.orchestra.run.smithi092.stdout: "addr": "172.21.15.92:6803", 2024-03-11T21:46:11.139 INFO:teuthology.orchestra.run.smithi092.stdout: "nonce": 2844100201 2024-03-11T21:46:11.139 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-03-11T21:46:11.139 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-03-11T21:46:11.139 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:46:11.140 INFO:teuthology.orchestra.run.smithi092.stdout: "public_addr": "172.21.15.92:6801/2844100201", 2024-03-11T21:46:11.140 INFO:teuthology.orchestra.run.smithi092.stdout: "cluster_addr": "172.21.15.92:6802/2844100201", 2024-03-11T21:46:11.140 INFO:teuthology.orchestra.run.smithi092.stdout: "heartbeat_back_addr": "172.21.15.92:6804/2844100201", 2024-03-11T21:46:11.140 INFO:teuthology.orchestra.run.smithi092.stdout: "heartbeat_front_addr": "172.21.15.92:6803/2844100201", 2024-03-11T21:46:11.140 INFO:teuthology.orchestra.run.smithi092.stdout: "state": [ 2024-03-11T21:46:11.140 INFO:teuthology.orchestra.run.smithi092.stdout: "exists", 2024-03-11T21:46:11.140 INFO:teuthology.orchestra.run.smithi092.stdout: "up" 2024-03-11T21:46:11.140 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-03-11T21:46:11.140 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:46:11.140 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:46:11.140 INFO:teuthology.orchestra.run.smithi092.stdout: "osd": 1, 2024-03-11T21:46:11.140 INFO:teuthology.orchestra.run.smithi092.stdout: "uuid": "3a5a0e78-1293-4ace-a93f-69156098571d", 2024-03-11T21:46:11.140 INFO:teuthology.orchestra.run.smithi092.stdout: "up": 1, 2024-03-11T21:46:11.140 INFO:teuthology.orchestra.run.smithi092.stdout: "in": 1, 2024-03-11T21:46:11.140 INFO:teuthology.orchestra.run.smithi092.stdout: "weight": 1, 2024-03-11T21:46:11.140 INFO:teuthology.orchestra.run.smithi092.stdout: "primary_affinity": 1, 2024-03-11T21:46:11.141 INFO:teuthology.orchestra.run.smithi092.stdout: "last_clean_begin": 0, 2024-03-11T21:46:11.141 INFO:teuthology.orchestra.run.smithi092.stdout: "last_clean_end": 0, 2024-03-11T21:46:11.141 INFO:teuthology.orchestra.run.smithi092.stdout: "up_from": 13, 2024-03-11T21:46:11.141 INFO:teuthology.orchestra.run.smithi092.stdout: "up_thru": 446, 2024-03-11T21:46:11.141 INFO:teuthology.orchestra.run.smithi092.stdout: "down_at": 0, 2024-03-11T21:46:11.141 INFO:teuthology.orchestra.run.smithi092.stdout: "lost_at": 0, 2024-03-11T21:46:11.141 INFO:teuthology.orchestra.run.smithi092.stdout: "public_addrs": { 2024-03-11T21:46:11.141 INFO:teuthology.orchestra.run.smithi092.stdout: "addrvec": [ 2024-03-11T21:46:11.141 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:46:11.141 INFO:teuthology.orchestra.run.smithi092.stdout: "type": "v1", 2024-03-11T21:46:11.141 INFO:teuthology.orchestra.run.smithi092.stdout: "addr": "172.21.15.92:6805", 2024-03-11T21:46:11.141 INFO:teuthology.orchestra.run.smithi092.stdout: "nonce": 3577871157 2024-03-11T21:46:11.141 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-03-11T21:46:11.141 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-03-11T21:46:11.141 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:46:11.142 INFO:teuthology.orchestra.run.smithi092.stdout: "cluster_addrs": { 2024-03-11T21:46:11.142 INFO:teuthology.orchestra.run.smithi092.stdout: "addrvec": [ 2024-03-11T21:46:11.142 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:46:11.142 INFO:teuthology.orchestra.run.smithi092.stdout: "type": "v1", 2024-03-11T21:46:11.142 INFO:teuthology.orchestra.run.smithi092.stdout: "addr": "172.21.15.92:6806", 2024-03-11T21:46:11.142 INFO:teuthology.orchestra.run.smithi092.stdout: "nonce": 3577871157 2024-03-11T21:46:11.142 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-03-11T21:46:11.142 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-03-11T21:46:11.142 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:46:11.142 INFO:teuthology.orchestra.run.smithi092.stdout: "heartbeat_back_addrs": { 2024-03-11T21:46:11.142 INFO:teuthology.orchestra.run.smithi092.stdout: "addrvec": [ 2024-03-11T21:46:11.142 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:46:11.142 INFO:teuthology.orchestra.run.smithi092.stdout: "type": "v1", 2024-03-11T21:46:11.142 INFO:teuthology.orchestra.run.smithi092.stdout: "addr": "172.21.15.92:6808", 2024-03-11T21:46:11.142 INFO:teuthology.orchestra.run.smithi092.stdout: "nonce": 3577871157 2024-03-11T21:46:11.142 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-03-11T21:46:11.143 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-03-11T21:46:11.143 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:46:11.143 INFO:teuthology.orchestra.run.smithi092.stdout: "heartbeat_front_addrs": { 2024-03-11T21:46:11.143 INFO:teuthology.orchestra.run.smithi092.stdout: "addrvec": [ 2024-03-11T21:46:11.143 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:46:11.143 INFO:teuthology.orchestra.run.smithi092.stdout: "type": "v1", 2024-03-11T21:46:11.143 INFO:teuthology.orchestra.run.smithi092.stdout: "addr": "172.21.15.92:6807", 2024-03-11T21:46:11.143 INFO:teuthology.orchestra.run.smithi092.stdout: "nonce": 3577871157 2024-03-11T21:46:11.143 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-03-11T21:46:11.143 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-03-11T21:46:11.143 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:46:11.143 INFO:teuthology.orchestra.run.smithi092.stdout: "public_addr": "172.21.15.92:6805/3577871157", 2024-03-11T21:46:11.143 INFO:teuthology.orchestra.run.smithi092.stdout: "cluster_addr": "172.21.15.92:6806/3577871157", 2024-03-11T21:46:11.143 INFO:teuthology.orchestra.run.smithi092.stdout: "heartbeat_back_addr": "172.21.15.92:6808/3577871157", 2024-03-11T21:46:11.143 INFO:teuthology.orchestra.run.smithi092.stdout: "heartbeat_front_addr": "172.21.15.92:6807/3577871157", 2024-03-11T21:46:11.144 INFO:teuthology.orchestra.run.smithi092.stdout: "state": [ 2024-03-11T21:46:11.144 INFO:teuthology.orchestra.run.smithi092.stdout: "exists", 2024-03-11T21:46:11.144 INFO:teuthology.orchestra.run.smithi092.stdout: "up" 2024-03-11T21:46:11.144 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-03-11T21:46:11.144 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:46:11.144 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:46:11.144 INFO:teuthology.orchestra.run.smithi092.stdout: "osd": 2, 2024-03-11T21:46:11.144 INFO:teuthology.orchestra.run.smithi092.stdout: "uuid": "f207b743-0df4-40ed-a735-950e272fea60", 2024-03-11T21:46:11.144 INFO:teuthology.orchestra.run.smithi092.stdout: "up": 1, 2024-03-11T21:46:11.144 INFO:teuthology.orchestra.run.smithi092.stdout: "in": 1, 2024-03-11T21:46:11.144 INFO:teuthology.orchestra.run.smithi092.stdout: "weight": 1, 2024-03-11T21:46:11.144 INFO:teuthology.orchestra.run.smithi092.stdout: "primary_affinity": 1, 2024-03-11T21:46:11.144 INFO:teuthology.orchestra.run.smithi092.stdout: "last_clean_begin": 0, 2024-03-11T21:46:11.144 INFO:teuthology.orchestra.run.smithi092.stdout: "last_clean_end": 0, 2024-03-11T21:46:11.144 INFO:teuthology.orchestra.run.smithi092.stdout: "up_from": 18, 2024-03-11T21:46:11.145 INFO:teuthology.orchestra.run.smithi092.stdout: "up_thru": 446, 2024-03-11T21:46:11.145 INFO:teuthology.orchestra.run.smithi092.stdout: "down_at": 0, 2024-03-11T21:46:11.145 INFO:teuthology.orchestra.run.smithi092.stdout: "lost_at": 0, 2024-03-11T21:46:11.145 INFO:teuthology.orchestra.run.smithi092.stdout: "public_addrs": { 2024-03-11T21:46:11.145 INFO:teuthology.orchestra.run.smithi092.stdout: "addrvec": [ 2024-03-11T21:46:11.145 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:46:11.145 INFO:teuthology.orchestra.run.smithi092.stdout: "type": "v1", 2024-03-11T21:46:11.145 INFO:teuthology.orchestra.run.smithi092.stdout: "addr": "172.21.15.92:6809", 2024-03-11T21:46:11.145 INFO:teuthology.orchestra.run.smithi092.stdout: "nonce": 760825152 2024-03-11T21:46:11.145 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-03-11T21:46:11.145 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-03-11T21:46:11.145 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:46:11.145 INFO:teuthology.orchestra.run.smithi092.stdout: "cluster_addrs": { 2024-03-11T21:46:11.145 INFO:teuthology.orchestra.run.smithi092.stdout: "addrvec": [ 2024-03-11T21:46:11.145 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:46:11.145 INFO:teuthology.orchestra.run.smithi092.stdout: "type": "v1", 2024-03-11T21:46:11.146 INFO:teuthology.orchestra.run.smithi092.stdout: "addr": "172.21.15.92:6810", 2024-03-11T21:46:11.146 INFO:teuthology.orchestra.run.smithi092.stdout: "nonce": 760825152 2024-03-11T21:46:11.146 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-03-11T21:46:11.146 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-03-11T21:46:11.146 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:46:11.146 INFO:teuthology.orchestra.run.smithi092.stdout: "heartbeat_back_addrs": { 2024-03-11T21:46:11.146 INFO:teuthology.orchestra.run.smithi092.stdout: "addrvec": [ 2024-03-11T21:46:11.146 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:46:11.146 INFO:teuthology.orchestra.run.smithi092.stdout: "type": "v1", 2024-03-11T21:46:11.146 INFO:teuthology.orchestra.run.smithi092.stdout: "addr": "172.21.15.92:6812", 2024-03-11T21:46:11.146 INFO:teuthology.orchestra.run.smithi092.stdout: "nonce": 760825152 2024-03-11T21:46:11.146 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-03-11T21:46:11.146 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-03-11T21:46:11.146 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:46:11.147 INFO:teuthology.orchestra.run.smithi092.stdout: "heartbeat_front_addrs": { 2024-03-11T21:46:11.147 INFO:teuthology.orchestra.run.smithi092.stdout: "addrvec": [ 2024-03-11T21:46:11.147 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:46:11.147 INFO:teuthology.orchestra.run.smithi092.stdout: "type": "v1", 2024-03-11T21:46:11.147 INFO:teuthology.orchestra.run.smithi092.stdout: "addr": "172.21.15.92:6811", 2024-03-11T21:46:11.147 INFO:teuthology.orchestra.run.smithi092.stdout: "nonce": 760825152 2024-03-11T21:46:11.147 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-03-11T21:46:11.147 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-03-11T21:46:11.147 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:46:11.147 INFO:teuthology.orchestra.run.smithi092.stdout: "public_addr": "172.21.15.92:6809/760825152", 2024-03-11T21:46:11.147 INFO:teuthology.orchestra.run.smithi092.stdout: "cluster_addr": "172.21.15.92:6810/760825152", 2024-03-11T21:46:11.147 INFO:teuthology.orchestra.run.smithi092.stdout: "heartbeat_back_addr": "172.21.15.92:6812/760825152", 2024-03-11T21:46:11.147 INFO:teuthology.orchestra.run.smithi092.stdout: "heartbeat_front_addr": "172.21.15.92:6811/760825152", 2024-03-11T21:46:11.147 INFO:teuthology.orchestra.run.smithi092.stdout: "state": [ 2024-03-11T21:46:11.148 INFO:teuthology.orchestra.run.smithi092.stdout: "exists", 2024-03-11T21:46:11.148 INFO:teuthology.orchestra.run.smithi092.stdout: "up" 2024-03-11T21:46:11.148 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-03-11T21:46:11.148 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:46:11.148 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:46:11.148 INFO:teuthology.orchestra.run.smithi092.stdout: "osd": 3, 2024-03-11T21:46:11.148 INFO:teuthology.orchestra.run.smithi092.stdout: "uuid": "bbb6f69a-8f0f-47e6-937a-062e1902ae13", 2024-03-11T21:46:11.148 INFO:teuthology.orchestra.run.smithi092.stdout: "up": 1, 2024-03-11T21:46:11.148 INFO:teuthology.orchestra.run.smithi092.stdout: "in": 0, 2024-03-11T21:46:11.148 INFO:teuthology.orchestra.run.smithi092.stdout: "weight": 0, 2024-03-11T21:46:11.148 INFO:teuthology.orchestra.run.smithi092.stdout: "primary_affinity": 1, 2024-03-11T21:46:11.148 INFO:teuthology.orchestra.run.smithi092.stdout: "last_clean_begin": 26, 2024-03-11T21:46:11.148 INFO:teuthology.orchestra.run.smithi092.stdout: "last_clean_end": 251, 2024-03-11T21:46:11.148 INFO:teuthology.orchestra.run.smithi092.stdout: "up_from": 277, 2024-03-11T21:46:11.149 INFO:teuthology.orchestra.run.smithi092.stdout: "up_thru": 391, 2024-03-11T21:46:11.149 INFO:teuthology.orchestra.run.smithi092.stdout: "down_at": 252, 2024-03-11T21:46:11.149 INFO:teuthology.orchestra.run.smithi092.stdout: "lost_at": 0, 2024-03-11T21:46:11.149 INFO:teuthology.orchestra.run.smithi092.stdout: "public_addrs": { 2024-03-11T21:46:11.149 INFO:teuthology.orchestra.run.smithi092.stdout: "addrvec": [ 2024-03-11T21:46:11.149 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:46:11.149 INFO:teuthology.orchestra.run.smithi092.stdout: "type": "v1", 2024-03-11T21:46:11.149 INFO:teuthology.orchestra.run.smithi092.stdout: "addr": "172.21.15.92:6813", 2024-03-11T21:46:11.149 INFO:teuthology.orchestra.run.smithi092.stdout: "nonce": 3215041236 2024-03-11T21:46:11.149 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-03-11T21:46:11.149 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-03-11T21:46:11.149 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:46:11.149 INFO:teuthology.orchestra.run.smithi092.stdout: "cluster_addrs": { 2024-03-11T21:46:11.149 INFO:teuthology.orchestra.run.smithi092.stdout: "addrvec": [ 2024-03-11T21:46:11.149 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:46:11.149 INFO:teuthology.orchestra.run.smithi092.stdout: "type": "v1", 2024-03-11T21:46:11.150 INFO:teuthology.orchestra.run.smithi092.stdout: "addr": "172.21.15.92:6814", 2024-03-11T21:46:11.150 INFO:teuthology.orchestra.run.smithi092.stdout: "nonce": 3215041236 2024-03-11T21:46:11.150 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-03-11T21:46:11.150 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-03-11T21:46:11.150 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:46:11.150 INFO:teuthology.orchestra.run.smithi092.stdout: "heartbeat_back_addrs": { 2024-03-11T21:46:11.150 INFO:teuthology.orchestra.run.smithi092.stdout: "addrvec": [ 2024-03-11T21:46:11.150 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:46:11.150 INFO:teuthology.orchestra.run.smithi092.stdout: "type": "v1", 2024-03-11T21:46:11.150 INFO:teuthology.orchestra.run.smithi092.stdout: "addr": "172.21.15.92:6816", 2024-03-11T21:46:11.150 INFO:teuthology.orchestra.run.smithi092.stdout: "nonce": 3215041236 2024-03-11T21:46:11.150 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-03-11T21:46:11.150 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-03-11T21:46:11.150 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:46:11.151 INFO:teuthology.orchestra.run.smithi092.stdout: "heartbeat_front_addrs": { 2024-03-11T21:46:11.151 INFO:teuthology.orchestra.run.smithi092.stdout: "addrvec": [ 2024-03-11T21:46:11.151 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:46:11.151 INFO:teuthology.orchestra.run.smithi092.stdout: "type": "v1", 2024-03-11T21:46:11.151 INFO:teuthology.orchestra.run.smithi092.stdout: "addr": "172.21.15.92:6815", 2024-03-11T21:46:11.151 INFO:teuthology.orchestra.run.smithi092.stdout: "nonce": 3215041236 2024-03-11T21:46:11.151 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-03-11T21:46:11.151 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-03-11T21:46:11.151 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:46:11.151 INFO:teuthology.orchestra.run.smithi092.stdout: "public_addr": "172.21.15.92:6813/3215041236", 2024-03-11T21:46:11.151 INFO:teuthology.orchestra.run.smithi092.stdout: "cluster_addr": "172.21.15.92:6814/3215041236", 2024-03-11T21:46:11.151 INFO:teuthology.orchestra.run.smithi092.stdout: "heartbeat_back_addr": "172.21.15.92:6816/3215041236", 2024-03-11T21:46:11.151 INFO:teuthology.orchestra.run.smithi092.stdout: "heartbeat_front_addr": "172.21.15.92:6815/3215041236", 2024-03-11T21:46:11.151 INFO:teuthology.orchestra.run.smithi092.stdout: "state": [ 2024-03-11T21:46:11.151 INFO:teuthology.orchestra.run.smithi092.stdout: "exists", 2024-03-11T21:46:11.152 INFO:teuthology.orchestra.run.smithi092.stdout: "up" 2024-03-11T21:46:11.152 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-03-11T21:46:11.152 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:46:11.152 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:46:11.152 INFO:teuthology.orchestra.run.smithi092.stdout: "osd": 4, 2024-03-11T21:46:11.152 INFO:teuthology.orchestra.run.smithi092.stdout: "uuid": "00266cbc-d2cf-4dee-8a96-3753ebc95ca8", 2024-03-11T21:46:11.152 INFO:teuthology.orchestra.run.smithi092.stdout: "up": 1, 2024-03-11T21:46:11.152 INFO:teuthology.orchestra.run.smithi092.stdout: "in": 0, 2024-03-11T21:46:11.152 INFO:teuthology.orchestra.run.smithi092.stdout: "weight": 0, 2024-03-11T21:46:11.152 INFO:teuthology.orchestra.run.smithi092.stdout: "primary_affinity": 1, 2024-03-11T21:46:11.152 INFO:teuthology.orchestra.run.smithi092.stdout: "last_clean_begin": 0, 2024-03-11T21:46:11.152 INFO:teuthology.orchestra.run.smithi092.stdout: "last_clean_end": 0, 2024-03-11T21:46:11.152 INFO:teuthology.orchestra.run.smithi092.stdout: "up_from": 32, 2024-03-11T21:46:11.152 INFO:teuthology.orchestra.run.smithi092.stdout: "up_thru": 432, 2024-03-11T21:46:11.152 INFO:teuthology.orchestra.run.smithi092.stdout: "down_at": 0, 2024-03-11T21:46:11.153 INFO:teuthology.orchestra.run.smithi092.stdout: "lost_at": 0, 2024-03-11T21:46:11.153 INFO:teuthology.orchestra.run.smithi092.stdout: "public_addrs": { 2024-03-11T21:46:11.153 INFO:teuthology.orchestra.run.smithi092.stdout: "addrvec": [ 2024-03-11T21:46:11.153 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:46:11.153 INFO:teuthology.orchestra.run.smithi092.stdout: "type": "v1", 2024-03-11T21:46:11.153 INFO:teuthology.orchestra.run.smithi092.stdout: "addr": "172.21.15.97:6800", 2024-03-11T21:46:11.153 INFO:teuthology.orchestra.run.smithi092.stdout: "nonce": 420733620 2024-03-11T21:46:11.153 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-03-11T21:46:11.153 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-03-11T21:46:11.153 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:46:11.153 INFO:teuthology.orchestra.run.smithi092.stdout: "cluster_addrs": { 2024-03-11T21:46:11.153 INFO:teuthology.orchestra.run.smithi092.stdout: "addrvec": [ 2024-03-11T21:46:11.153 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:46:11.153 INFO:teuthology.orchestra.run.smithi092.stdout: "type": "v1", 2024-03-11T21:46:11.153 INFO:teuthology.orchestra.run.smithi092.stdout: "addr": "172.21.15.97:6801", 2024-03-11T21:46:11.154 INFO:teuthology.orchestra.run.smithi092.stdout: "nonce": 420733620 2024-03-11T21:46:11.154 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-03-11T21:46:11.154 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-03-11T21:46:11.154 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:46:11.154 INFO:teuthology.orchestra.run.smithi092.stdout: "heartbeat_back_addrs": { 2024-03-11T21:46:11.154 INFO:teuthology.orchestra.run.smithi092.stdout: "addrvec": [ 2024-03-11T21:46:11.154 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:46:11.154 INFO:teuthology.orchestra.run.smithi092.stdout: "type": "v1", 2024-03-11T21:46:11.154 INFO:teuthology.orchestra.run.smithi092.stdout: "addr": "172.21.15.97:6803", 2024-03-11T21:46:11.154 INFO:teuthology.orchestra.run.smithi092.stdout: "nonce": 420733620 2024-03-11T21:46:11.154 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-03-11T21:46:11.154 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-03-11T21:46:11.154 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:46:11.154 INFO:teuthology.orchestra.run.smithi092.stdout: "heartbeat_front_addrs": { 2024-03-11T21:46:11.155 INFO:teuthology.orchestra.run.smithi092.stdout: "addrvec": [ 2024-03-11T21:46:11.155 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:46:11.155 INFO:teuthology.orchestra.run.smithi092.stdout: "type": "v1", 2024-03-11T21:46:11.155 INFO:teuthology.orchestra.run.smithi092.stdout: "addr": "172.21.15.97:6802", 2024-03-11T21:46:11.155 INFO:teuthology.orchestra.run.smithi092.stdout: "nonce": 420733620 2024-03-11T21:46:11.155 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-03-11T21:46:11.155 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-03-11T21:46:11.155 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:46:11.155 INFO:teuthology.orchestra.run.smithi092.stdout: "public_addr": "172.21.15.97:6800/420733620", 2024-03-11T21:46:11.155 INFO:teuthology.orchestra.run.smithi092.stdout: "cluster_addr": "172.21.15.97:6801/420733620", 2024-03-11T21:46:11.155 INFO:teuthology.orchestra.run.smithi092.stdout: "heartbeat_back_addr": "172.21.15.97:6803/420733620", 2024-03-11T21:46:11.155 INFO:teuthology.orchestra.run.smithi092.stdout: "heartbeat_front_addr": "172.21.15.97:6802/420733620", 2024-03-11T21:46:11.155 INFO:teuthology.orchestra.run.smithi092.stdout: "state": [ 2024-03-11T21:46:11.155 INFO:teuthology.orchestra.run.smithi092.stdout: "exists", 2024-03-11T21:46:11.155 INFO:teuthology.orchestra.run.smithi092.stdout: "up" 2024-03-11T21:46:11.156 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-03-11T21:46:11.156 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:46:11.156 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:46:11.156 INFO:teuthology.orchestra.run.smithi092.stdout: "osd": 5, 2024-03-11T21:46:11.156 INFO:teuthology.orchestra.run.smithi092.stdout: "uuid": "593a6f6b-8874-40ae-8848-d51c5723a5e4", 2024-03-11T21:46:11.156 INFO:teuthology.orchestra.run.smithi092.stdout: "up": 1, 2024-03-11T21:46:11.156 INFO:teuthology.orchestra.run.smithi092.stdout: "in": 1, 2024-03-11T21:46:11.156 INFO:teuthology.orchestra.run.smithi092.stdout: "weight": 1, 2024-03-11T21:46:11.156 INFO:teuthology.orchestra.run.smithi092.stdout: "primary_affinity": 1, 2024-03-11T21:46:11.156 INFO:teuthology.orchestra.run.smithi092.stdout: "last_clean_begin": 0, 2024-03-11T21:46:11.156 INFO:teuthology.orchestra.run.smithi092.stdout: "last_clean_end": 0, 2024-03-11T21:46:11.156 INFO:teuthology.orchestra.run.smithi092.stdout: "up_from": 41, 2024-03-11T21:46:11.156 INFO:teuthology.orchestra.run.smithi092.stdout: "up_thru": 446, 2024-03-11T21:46:11.156 INFO:teuthology.orchestra.run.smithi092.stdout: "down_at": 0, 2024-03-11T21:46:11.156 INFO:teuthology.orchestra.run.smithi092.stdout: "lost_at": 0, 2024-03-11T21:46:11.157 INFO:teuthology.orchestra.run.smithi092.stdout: "public_addrs": { 2024-03-11T21:46:11.157 INFO:teuthology.orchestra.run.smithi092.stdout: "addrvec": [ 2024-03-11T21:46:11.157 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:46:11.157 INFO:teuthology.orchestra.run.smithi092.stdout: "type": "v1", 2024-03-11T21:46:11.157 INFO:teuthology.orchestra.run.smithi092.stdout: "addr": "172.21.15.97:6804", 2024-03-11T21:46:11.157 INFO:teuthology.orchestra.run.smithi092.stdout: "nonce": 312056041 2024-03-11T21:46:11.157 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-03-11T21:46:11.157 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-03-11T21:46:11.157 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:46:11.157 INFO:teuthology.orchestra.run.smithi092.stdout: "cluster_addrs": { 2024-03-11T21:46:11.157 INFO:teuthology.orchestra.run.smithi092.stdout: "addrvec": [ 2024-03-11T21:46:11.157 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:46:11.157 INFO:teuthology.orchestra.run.smithi092.stdout: "type": "v1", 2024-03-11T21:46:11.157 INFO:teuthology.orchestra.run.smithi092.stdout: "addr": "172.21.15.97:6805", 2024-03-11T21:46:11.158 INFO:teuthology.orchestra.run.smithi092.stdout: "nonce": 312056041 2024-03-11T21:46:11.158 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-03-11T21:46:11.158 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-03-11T21:46:11.158 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:46:11.158 INFO:teuthology.orchestra.run.smithi092.stdout: "heartbeat_back_addrs": { 2024-03-11T21:46:11.158 INFO:teuthology.orchestra.run.smithi092.stdout: "addrvec": [ 2024-03-11T21:46:11.158 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:46:11.158 INFO:teuthology.orchestra.run.smithi092.stdout: "type": "v1", 2024-03-11T21:46:11.158 INFO:teuthology.orchestra.run.smithi092.stdout: "addr": "172.21.15.97:6807", 2024-03-11T21:46:11.158 INFO:teuthology.orchestra.run.smithi092.stdout: "nonce": 312056041 2024-03-11T21:46:11.158 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-03-11T21:46:11.158 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-03-11T21:46:11.158 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:46:11.159 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:46:11.159 INFO:teuthology.orchestra.run.smithi092.stdout: "heartbeat_front_addrs": { 2024-03-11T21:46:11.159 INFO:teuthology.orchestra.run.smithi092.stdout: "addrvec": [ 2024-03-11T21:46:11.160 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:46:11.160 INFO:teuthology.orchestra.run.smithi092.stdout: "type": "v1", 2024-03-11T21:46:11.160 INFO:teuthology.orchestra.run.smithi092.stdout: "addr": "172.21.15.97:6806", 2024-03-11T21:46:11.160 INFO:teuthology.orchestra.run.smithi092.stdout: "nonce": 312056041 2024-03-11T21:46:11.160 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-03-11T21:46:11.160 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-03-11T21:46:11.160 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:46:11.160 INFO:teuthology.orchestra.run.smithi092.stdout: "public_addr": "172.21.15.97:6804/312056041", 2024-03-11T21:46:11.160 INFO:teuthology.orchestra.run.smithi092.stdout: "cluster_addr": "172.21.15.97:6805/312056041", 2024-03-11T21:46:11.160 INFO:teuthology.orchestra.run.smithi092.stdout: "heartbeat_back_addr": "172.21.15.97:6807/312056041", 2024-03-11T21:46:11.160 INFO:teuthology.orchestra.run.smithi092.stdout: "heartbeat_front_addr": "172.21.15.97:6806/312056041", 2024-03-11T21:46:11.160 INFO:teuthology.orchestra.run.smithi092.stdout: "state": [ 2024-03-11T21:46:11.160 INFO:teuthology.orchestra.run.smithi092.stdout: "exists", 2024-03-11T21:46:11.160 INFO:teuthology.orchestra.run.smithi092.stdout: "up" 2024-03-11T21:46:11.160 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-03-11T21:46:11.161 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:46:11.161 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:46:11.161 INFO:teuthology.orchestra.run.smithi092.stdout: "osd": 6, 2024-03-11T21:46:11.161 INFO:teuthology.orchestra.run.smithi092.stdout: "uuid": "91035ecc-6f1a-4cec-bda6-090af9d53eff", 2024-03-11T21:46:11.161 INFO:teuthology.orchestra.run.smithi092.stdout: "up": 1, 2024-03-11T21:46:11.161 INFO:teuthology.orchestra.run.smithi092.stdout: "in": 1, 2024-03-11T21:46:11.161 INFO:teuthology.orchestra.run.smithi092.stdout: "weight": 1, 2024-03-11T21:46:11.161 INFO:teuthology.orchestra.run.smithi092.stdout: "primary_affinity": 1, 2024-03-11T21:46:11.161 INFO:teuthology.orchestra.run.smithi092.stdout: "last_clean_begin": 0, 2024-03-11T21:46:11.161 INFO:teuthology.orchestra.run.smithi092.stdout: "last_clean_end": 0, 2024-03-11T21:46:11.161 INFO:teuthology.orchestra.run.smithi092.stdout: "up_from": 50, 2024-03-11T21:46:11.161 INFO:teuthology.orchestra.run.smithi092.stdout: "up_thru": 446, 2024-03-11T21:46:11.161 INFO:teuthology.orchestra.run.smithi092.stdout: "down_at": 0, 2024-03-11T21:46:11.161 INFO:teuthology.orchestra.run.smithi092.stdout: "lost_at": 0, 2024-03-11T21:46:11.161 INFO:teuthology.orchestra.run.smithi092.stdout: "public_addrs": { 2024-03-11T21:46:11.162 INFO:teuthology.orchestra.run.smithi092.stdout: "addrvec": [ 2024-03-11T21:46:11.162 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:46:11.162 INFO:teuthology.orchestra.run.smithi092.stdout: "type": "v1", 2024-03-11T21:46:11.162 INFO:teuthology.orchestra.run.smithi092.stdout: "addr": "172.21.15.97:6808", 2024-03-11T21:46:11.162 INFO:teuthology.orchestra.run.smithi092.stdout: "nonce": 178994999 2024-03-11T21:46:11.162 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-03-11T21:46:11.162 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-03-11T21:46:11.162 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:46:11.162 INFO:teuthology.orchestra.run.smithi092.stdout: "cluster_addrs": { 2024-03-11T21:46:11.162 INFO:teuthology.orchestra.run.smithi092.stdout: "addrvec": [ 2024-03-11T21:46:11.162 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:46:11.162 INFO:teuthology.orchestra.run.smithi092.stdout: "type": "v1", 2024-03-11T21:46:11.163 INFO:teuthology.orchestra.run.smithi092.stdout: "addr": "172.21.15.97:6809", 2024-03-11T21:46:11.163 INFO:teuthology.orchestra.run.smithi092.stdout: "nonce": 178994999 2024-03-11T21:46:11.163 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-03-11T21:46:11.163 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-03-11T21:46:11.163 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:46:11.163 INFO:teuthology.orchestra.run.smithi092.stdout: "heartbeat_back_addrs": { 2024-03-11T21:46:11.163 INFO:teuthology.orchestra.run.smithi092.stdout: "addrvec": [ 2024-03-11T21:46:11.163 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:46:11.163 INFO:teuthology.orchestra.run.smithi092.stdout: "type": "v1", 2024-03-11T21:46:11.163 INFO:teuthology.orchestra.run.smithi092.stdout: "addr": "172.21.15.97:6811", 2024-03-11T21:46:11.163 INFO:teuthology.orchestra.run.smithi092.stdout: "nonce": 178994999 2024-03-11T21:46:11.163 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-03-11T21:46:11.163 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-03-11T21:46:11.164 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:46:11.164 INFO:teuthology.orchestra.run.smithi092.stdout: "heartbeat_front_addrs": { 2024-03-11T21:46:11.164 INFO:teuthology.orchestra.run.smithi092.stdout: "addrvec": [ 2024-03-11T21:46:11.164 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:46:11.164 INFO:teuthology.orchestra.run.smithi092.stdout: "type": "v1", 2024-03-11T21:46:11.164 INFO:teuthology.orchestra.run.smithi092.stdout: "addr": "172.21.15.97:6810", 2024-03-11T21:46:11.164 INFO:teuthology.orchestra.run.smithi092.stdout: "nonce": 178994999 2024-03-11T21:46:11.164 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-03-11T21:46:11.164 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-03-11T21:46:11.164 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:46:11.164 INFO:teuthology.orchestra.run.smithi092.stdout: "public_addr": "172.21.15.97:6808/178994999", 2024-03-11T21:46:11.164 INFO:teuthology.orchestra.run.smithi092.stdout: "cluster_addr": "172.21.15.97:6809/178994999", 2024-03-11T21:46:11.164 INFO:teuthology.orchestra.run.smithi092.stdout: "heartbeat_back_addr": "172.21.15.97:6811/178994999", 2024-03-11T21:46:11.164 INFO:teuthology.orchestra.run.smithi092.stdout: "heartbeat_front_addr": "172.21.15.97:6810/178994999", 2024-03-11T21:46:11.164 INFO:teuthology.orchestra.run.smithi092.stdout: "state": [ 2024-03-11T21:46:11.165 INFO:teuthology.orchestra.run.smithi092.stdout: "exists", 2024-03-11T21:46:11.165 INFO:teuthology.orchestra.run.smithi092.stdout: "up" 2024-03-11T21:46:11.165 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-03-11T21:46:11.165 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:46:11.165 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:46:11.165 INFO:teuthology.orchestra.run.smithi092.stdout: "osd": 7, 2024-03-11T21:46:11.165 INFO:teuthology.orchestra.run.smithi092.stdout: "uuid": "3eb28c24-664a-4bd6-869d-6bf5db98f52f", 2024-03-11T21:46:11.165 INFO:teuthology.orchestra.run.smithi092.stdout: "up": 1, 2024-03-11T21:46:11.165 INFO:teuthology.orchestra.run.smithi092.stdout: "in": 1, 2024-03-11T21:46:11.165 INFO:teuthology.orchestra.run.smithi092.stdout: "weight": 1, 2024-03-11T21:46:11.165 INFO:teuthology.orchestra.run.smithi092.stdout: "primary_affinity": 1, 2024-03-11T21:46:11.165 INFO:teuthology.orchestra.run.smithi092.stdout: "last_clean_begin": 0, 2024-03-11T21:46:11.165 INFO:teuthology.orchestra.run.smithi092.stdout: "last_clean_end": 0, 2024-03-11T21:46:11.165 INFO:teuthology.orchestra.run.smithi092.stdout: "up_from": 60, 2024-03-11T21:46:11.165 INFO:teuthology.orchestra.run.smithi092.stdout: "up_thru": 446, 2024-03-11T21:46:11.166 INFO:teuthology.orchestra.run.smithi092.stdout: "down_at": 0, 2024-03-11T21:46:11.166 INFO:teuthology.orchestra.run.smithi092.stdout: "lost_at": 0, 2024-03-11T21:46:11.166 INFO:teuthology.orchestra.run.smithi092.stdout: "public_addrs": { 2024-03-11T21:46:11.166 INFO:teuthology.orchestra.run.smithi092.stdout: "addrvec": [ 2024-03-11T21:46:11.166 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:46:11.166 INFO:teuthology.orchestra.run.smithi092.stdout: "type": "v1", 2024-03-11T21:46:11.166 INFO:teuthology.orchestra.run.smithi092.stdout: "addr": "172.21.15.97:6812", 2024-03-11T21:46:11.166 INFO:teuthology.orchestra.run.smithi092.stdout: "nonce": 3241560509 2024-03-11T21:46:11.166 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-03-11T21:46:11.166 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-03-11T21:46:11.166 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:46:11.166 INFO:teuthology.orchestra.run.smithi092.stdout: "cluster_addrs": { 2024-03-11T21:46:11.166 INFO:teuthology.orchestra.run.smithi092.stdout: "addrvec": [ 2024-03-11T21:46:11.166 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:46:11.167 INFO:teuthology.orchestra.run.smithi092.stdout: "type": "v1", 2024-03-11T21:46:11.167 INFO:teuthology.orchestra.run.smithi092.stdout: "addr": "172.21.15.97:6813", 2024-03-11T21:46:11.167 INFO:teuthology.orchestra.run.smithi092.stdout: "nonce": 3241560509 2024-03-11T21:46:11.167 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-03-11T21:46:11.167 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-03-11T21:46:11.167 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:46:11.167 INFO:teuthology.orchestra.run.smithi092.stdout: "heartbeat_back_addrs": { 2024-03-11T21:46:11.167 INFO:teuthology.orchestra.run.smithi092.stdout: "addrvec": [ 2024-03-11T21:46:11.167 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:46:11.167 INFO:teuthology.orchestra.run.smithi092.stdout: "type": "v1", 2024-03-11T21:46:11.167 INFO:teuthology.orchestra.run.smithi092.stdout: "addr": "172.21.15.97:6815", 2024-03-11T21:46:11.167 INFO:teuthology.orchestra.run.smithi092.stdout: "nonce": 3241560509 2024-03-11T21:46:11.167 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-03-11T21:46:11.167 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-03-11T21:46:11.167 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:46:11.168 INFO:teuthology.orchestra.run.smithi092.stdout: "heartbeat_front_addrs": { 2024-03-11T21:46:11.168 INFO:teuthology.orchestra.run.smithi092.stdout: "addrvec": [ 2024-03-11T21:46:11.168 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:46:11.168 INFO:teuthology.orchestra.run.smithi092.stdout: "type": "v1", 2024-03-11T21:46:11.168 INFO:teuthology.orchestra.run.smithi092.stdout: "addr": "172.21.15.97:6814", 2024-03-11T21:46:11.168 INFO:teuthology.orchestra.run.smithi092.stdout: "nonce": 3241560509 2024-03-11T21:46:11.168 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-03-11T21:46:11.168 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-03-11T21:46:11.168 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:46:11.168 INFO:teuthology.orchestra.run.smithi092.stdout: "public_addr": "172.21.15.97:6812/3241560509", 2024-03-11T21:46:11.168 INFO:teuthology.orchestra.run.smithi092.stdout: "cluster_addr": "172.21.15.97:6813/3241560509", 2024-03-11T21:46:11.168 INFO:teuthology.orchestra.run.smithi092.stdout: "heartbeat_back_addr": "172.21.15.97:6815/3241560509", 2024-03-11T21:46:11.168 INFO:teuthology.orchestra.run.smithi092.stdout: "heartbeat_front_addr": "172.21.15.97:6814/3241560509", 2024-03-11T21:46:11.168 INFO:teuthology.orchestra.run.smithi092.stdout: "state": [ 2024-03-11T21:46:11.168 INFO:teuthology.orchestra.run.smithi092.stdout: "exists", 2024-03-11T21:46:11.169 INFO:teuthology.orchestra.run.smithi092.stdout: "up" 2024-03-11T21:46:11.169 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-03-11T21:46:11.169 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-03-11T21:46:11.169 INFO:teuthology.orchestra.run.smithi092.stdout: ], 2024-03-11T21:46:11.169 INFO:teuthology.orchestra.run.smithi092.stdout: "osd_xinfo": [ 2024-03-11T21:46:11.169 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:46:11.169 INFO:teuthology.orchestra.run.smithi092.stdout: "osd": 0, 2024-03-11T21:46:11.169 INFO:teuthology.orchestra.run.smithi092.stdout: "down_stamp": "0.000000", 2024-03-11T21:46:11.169 INFO:teuthology.orchestra.run.smithi092.stdout: "laggy_probability": 0, 2024-03-11T21:46:11.169 INFO:teuthology.orchestra.run.smithi092.stdout: "laggy_interval": 0, 2024-03-11T21:46:11.169 INFO:teuthology.orchestra.run.smithi092.stdout: "features": 4540701547738038271, 2024-03-11T21:46:11.169 INFO:teuthology.orchestra.run.smithi092.stdout: "old_weight": 0, 2024-03-11T21:46:11.169 INFO:teuthology.orchestra.run.smithi092.stdout: "last_purged_snaps_scrub": "2024-03-11T21:46:09.095379+0000", 2024-03-11T21:46:11.169 INFO:teuthology.orchestra.run.smithi092.stdout: "dead_epoch": 0 2024-03-11T21:46:11.169 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:46:11.170 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:46:11.170 INFO:teuthology.orchestra.run.smithi092.stdout: "osd": 1, 2024-03-11T21:46:11.170 INFO:teuthology.orchestra.run.smithi092.stdout: "down_stamp": "0.000000", 2024-03-11T21:46:11.170 INFO:teuthology.orchestra.run.smithi092.stdout: "laggy_probability": 0, 2024-03-11T21:46:11.170 INFO:teuthology.orchestra.run.smithi092.stdout: "laggy_interval": 0, 2024-03-11T21:46:11.170 INFO:teuthology.orchestra.run.smithi092.stdout: "features": 4540701547738038271, 2024-03-11T21:46:11.170 INFO:teuthology.orchestra.run.smithi092.stdout: "old_weight": 0, 2024-03-11T21:46:11.170 INFO:teuthology.orchestra.run.smithi092.stdout: "last_purged_snaps_scrub": "2024-03-11T21:44:52.116769+0000", 2024-03-11T21:46:11.170 INFO:teuthology.orchestra.run.smithi092.stdout: "dead_epoch": 0 2024-03-11T21:46:11.170 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:46:11.170 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:46:11.170 INFO:teuthology.orchestra.run.smithi092.stdout: "osd": 2, 2024-03-11T21:46:11.170 INFO:teuthology.orchestra.run.smithi092.stdout: "down_stamp": "0.000000", 2024-03-11T21:46:11.170 INFO:teuthology.orchestra.run.smithi092.stdout: "laggy_probability": 0, 2024-03-11T21:46:11.170 INFO:teuthology.orchestra.run.smithi092.stdout: "laggy_interval": 0, 2024-03-11T21:46:11.171 INFO:teuthology.orchestra.run.smithi092.stdout: "features": 4540701547738038271, 2024-03-11T21:46:11.171 INFO:teuthology.orchestra.run.smithi092.stdout: "old_weight": 0, 2024-03-11T21:46:11.171 INFO:teuthology.orchestra.run.smithi092.stdout: "last_purged_snaps_scrub": "2024-03-11T21:44:59.813525+0000", 2024-03-11T21:46:11.171 INFO:teuthology.orchestra.run.smithi092.stdout: "dead_epoch": 0 2024-03-11T21:46:11.171 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:46:11.171 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:46:11.171 INFO:teuthology.orchestra.run.smithi092.stdout: "osd": 3, 2024-03-11T21:46:11.171 INFO:teuthology.orchestra.run.smithi092.stdout: "down_stamp": "0.000000", 2024-03-11T21:46:11.171 INFO:teuthology.orchestra.run.smithi092.stdout: "laggy_probability": 0, 2024-03-11T21:46:11.171 INFO:teuthology.orchestra.run.smithi092.stdout: "laggy_interval": 0, 2024-03-11T21:46:11.171 INFO:teuthology.orchestra.run.smithi092.stdout: "features": 4540701547738038271, 2024-03-11T21:46:11.171 INFO:teuthology.orchestra.run.smithi092.stdout: "old_weight": 65536, 2024-03-11T21:46:11.171 INFO:teuthology.orchestra.run.smithi092.stdout: "last_purged_snaps_scrub": "2024-03-11T21:45:22.817858+0000", 2024-03-11T21:46:11.171 INFO:teuthology.orchestra.run.smithi092.stdout: "dead_epoch": 251 2024-03-11T21:46:11.171 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:46:11.171 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:46:11.172 INFO:teuthology.orchestra.run.smithi092.stdout: "osd": 4, 2024-03-11T21:46:11.172 INFO:teuthology.orchestra.run.smithi092.stdout: "down_stamp": "0.000000", 2024-03-11T21:46:11.172 INFO:teuthology.orchestra.run.smithi092.stdout: "laggy_probability": 0, 2024-03-11T21:46:11.172 INFO:teuthology.orchestra.run.smithi092.stdout: "laggy_interval": 0, 2024-03-11T21:46:11.172 INFO:teuthology.orchestra.run.smithi092.stdout: "features": 4540701547738038271, 2024-03-11T21:46:11.172 INFO:teuthology.orchestra.run.smithi092.stdout: "old_weight": 65536, 2024-03-11T21:46:11.172 INFO:teuthology.orchestra.run.smithi092.stdout: "last_purged_snaps_scrub": "2024-03-11T21:46:05.723463+0000", 2024-03-11T21:46:11.172 INFO:teuthology.orchestra.run.smithi092.stdout: "dead_epoch": 0 2024-03-11T21:46:11.172 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:46:11.172 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:46:11.172 INFO:teuthology.orchestra.run.smithi092.stdout: "osd": 5, 2024-03-11T21:46:11.172 INFO:teuthology.orchestra.run.smithi092.stdout: "down_stamp": "0.000000", 2024-03-11T21:46:11.172 INFO:teuthology.orchestra.run.smithi092.stdout: "laggy_probability": 0, 2024-03-11T21:46:11.172 INFO:teuthology.orchestra.run.smithi092.stdout: "laggy_interval": 0, 2024-03-11T21:46:11.172 INFO:teuthology.orchestra.run.smithi092.stdout: "features": 4540701547738038271, 2024-03-11T21:46:11.173 INFO:teuthology.orchestra.run.smithi092.stdout: "old_weight": 0, 2024-03-11T21:46:11.173 INFO:teuthology.orchestra.run.smithi092.stdout: "last_purged_snaps_scrub": "2024-03-11T21:45:07.242823+0000", 2024-03-11T21:46:11.173 INFO:teuthology.orchestra.run.smithi092.stdout: "dead_epoch": 0 2024-03-11T21:46:11.173 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:46:11.173 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:46:11.173 INFO:teuthology.orchestra.run.smithi092.stdout: "osd": 6, 2024-03-11T21:46:11.173 INFO:teuthology.orchestra.run.smithi092.stdout: "down_stamp": "0.000000", 2024-03-11T21:46:11.173 INFO:teuthology.orchestra.run.smithi092.stdout: "laggy_probability": 0, 2024-03-11T21:46:11.173 INFO:teuthology.orchestra.run.smithi092.stdout: "laggy_interval": 0, 2024-03-11T21:46:11.173 INFO:teuthology.orchestra.run.smithi092.stdout: "features": 4540701547738038271, 2024-03-11T21:46:11.173 INFO:teuthology.orchestra.run.smithi092.stdout: "old_weight": 0, 2024-03-11T21:46:11.173 INFO:teuthology.orchestra.run.smithi092.stdout: "last_purged_snaps_scrub": "2024-03-11T21:45:48.690289+0000", 2024-03-11T21:46:11.173 INFO:teuthology.orchestra.run.smithi092.stdout: "dead_epoch": 0 2024-03-11T21:46:11.173 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:46:11.173 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:46:11.174 INFO:teuthology.orchestra.run.smithi092.stdout: "osd": 7, 2024-03-11T21:46:11.174 INFO:teuthology.orchestra.run.smithi092.stdout: "down_stamp": "0.000000", 2024-03-11T21:46:11.174 INFO:teuthology.orchestra.run.smithi092.stdout: "laggy_probability": 0, 2024-03-11T21:46:11.174 INFO:teuthology.orchestra.run.smithi092.stdout: "laggy_interval": 0, 2024-03-11T21:46:11.174 INFO:teuthology.orchestra.run.smithi092.stdout: "features": 4540701547738038271, 2024-03-11T21:46:11.174 INFO:teuthology.orchestra.run.smithi092.stdout: "old_weight": 0, 2024-03-11T21:46:11.174 INFO:teuthology.orchestra.run.smithi092.stdout: "last_purged_snaps_scrub": "2024-03-11T21:45:23.323038+0000", 2024-03-11T21:46:11.174 INFO:teuthology.orchestra.run.smithi092.stdout: "dead_epoch": 0 2024-03-11T21:46:11.174 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-03-11T21:46:11.174 INFO:teuthology.orchestra.run.smithi092.stdout: ], 2024-03-11T21:46:11.174 INFO:teuthology.orchestra.run.smithi092.stdout: "pg_upmap": [ 2024-03-11T21:46:11.174 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:46:11.174 INFO:teuthology.orchestra.run.smithi092.stdout: "pgid": "6.a", 2024-03-11T21:46:11.174 INFO:teuthology.orchestra.run.smithi092.stdout: "osds": [ 2024-03-11T21:46:11.174 INFO:teuthology.orchestra.run.smithi092.stdout: 4, 2024-03-11T21:46:11.175 INFO:teuthology.orchestra.run.smithi092.stdout: 0, 2024-03-11T21:46:11.175 INFO:teuthology.orchestra.run.smithi092.stdout: 3 2024-03-11T21:46:11.175 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-03-11T21:46:11.175 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-03-11T21:46:11.175 INFO:teuthology.orchestra.run.smithi092.stdout: ], 2024-03-11T21:46:11.175 INFO:teuthology.orchestra.run.smithi092.stdout: "pg_upmap_items": [ 2024-03-11T21:46:11.175 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:46:11.175 INFO:teuthology.orchestra.run.smithi092.stdout: "pgid": "2.18", 2024-03-11T21:46:11.175 INFO:teuthology.orchestra.run.smithi092.stdout: "mappings": [ 2024-03-11T21:46:11.175 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:46:11.175 INFO:teuthology.orchestra.run.smithi092.stdout: "from": 7, 2024-03-11T21:46:11.175 INFO:teuthology.orchestra.run.smithi092.stdout: "to": 2 2024-03-11T21:46:11.175 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-03-11T21:46:11.175 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-03-11T21:46:11.175 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:46:11.175 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:46:11.176 INFO:teuthology.orchestra.run.smithi092.stdout: "pgid": "4.6", 2024-03-11T21:46:11.176 INFO:teuthology.orchestra.run.smithi092.stdout: "mappings": [ 2024-03-11T21:46:11.176 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:46:11.176 INFO:teuthology.orchestra.run.smithi092.stdout: "from": 1, 2024-03-11T21:46:11.176 INFO:teuthology.orchestra.run.smithi092.stdout: "to": 5 2024-03-11T21:46:11.176 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-03-11T21:46:11.176 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-03-11T21:46:11.176 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:46:11.176 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:46:11.176 INFO:teuthology.orchestra.run.smithi092.stdout: "pgid": "4.f", 2024-03-11T21:46:11.176 INFO:teuthology.orchestra.run.smithi092.stdout: "mappings": [ 2024-03-11T21:46:11.176 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:46:11.176 INFO:teuthology.orchestra.run.smithi092.stdout: "from": 1, 2024-03-11T21:46:11.176 INFO:teuthology.orchestra.run.smithi092.stdout: "to": 2 2024-03-11T21:46:11.176 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-03-11T21:46:11.177 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-03-11T21:46:11.177 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:46:11.177 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:46:11.177 INFO:teuthology.orchestra.run.smithi092.stdout: "pgid": "4.13", 2024-03-11T21:46:11.177 INFO:teuthology.orchestra.run.smithi092.stdout: "mappings": [ 2024-03-11T21:46:11.177 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:46:11.177 INFO:teuthology.orchestra.run.smithi092.stdout: "from": 1, 2024-03-11T21:46:11.177 INFO:teuthology.orchestra.run.smithi092.stdout: "to": 2 2024-03-11T21:46:11.177 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-03-11T21:46:11.177 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-03-11T21:46:11.177 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:46:11.177 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:46:11.177 INFO:teuthology.orchestra.run.smithi092.stdout: "pgid": "4.17", 2024-03-11T21:46:11.177 INFO:teuthology.orchestra.run.smithi092.stdout: "mappings": [ 2024-03-11T21:46:11.178 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:46:11.178 INFO:teuthology.orchestra.run.smithi092.stdout: "from": 1, 2024-03-11T21:46:11.178 INFO:teuthology.orchestra.run.smithi092.stdout: "to": 5 2024-03-11T21:46:11.178 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-03-11T21:46:11.178 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-03-11T21:46:11.178 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:46:11.178 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:46:11.178 INFO:teuthology.orchestra.run.smithi092.stdout: "pgid": "4.1f", 2024-03-11T21:46:11.178 INFO:teuthology.orchestra.run.smithi092.stdout: "mappings": [ 2024-03-11T21:46:11.178 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:46:11.178 INFO:teuthology.orchestra.run.smithi092.stdout: "from": 1, 2024-03-11T21:46:11.178 INFO:teuthology.orchestra.run.smithi092.stdout: "to": 0 2024-03-11T21:46:11.178 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-03-11T21:46:11.179 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-03-11T21:46:11.179 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-03-11T21:46:11.179 INFO:teuthology.orchestra.run.smithi092.stdout: ], 2024-03-11T21:46:11.179 INFO:teuthology.orchestra.run.smithi092.stdout: "pg_upmap_primaries": [], 2024-03-11T21:46:11.179 INFO:teuthology.orchestra.run.smithi092.stdout: "pg_temp": [], 2024-03-11T21:46:11.179 INFO:teuthology.orchestra.run.smithi092.stdout: "primary_temp": [], 2024-03-11T21:46:11.179 INFO:teuthology.orchestra.run.smithi092.stdout: "blocklist": { 2024-03-11T21:46:11.179 INFO:teuthology.orchestra.run.smithi092.stdout: "172.21.15.92:0/303695351": "2024-03-12T21:26:20.943013+0000", 2024-03-11T21:46:11.179 INFO:teuthology.orchestra.run.smithi092.stdout: "172.21.15.92:0/437911229": "2024-03-12T21:26:20.943013+0000", 2024-03-11T21:46:11.179 INFO:teuthology.orchestra.run.smithi092.stdout: "172.21.15.92:0/4246054050": "2024-03-12T21:26:20.943013+0000", 2024-03-11T21:46:11.179 INFO:teuthology.orchestra.run.smithi092.stdout: "172.21.15.92:0/2912406656": "2024-03-12T21:26:20.943013+0000", 2024-03-11T21:46:11.179 INFO:teuthology.orchestra.run.smithi092.stdout: "172.21.15.92:0/2706821241": "2024-03-12T21:26:20.943013+0000", 2024-03-11T21:46:11.179 INFO:teuthology.orchestra.run.smithi092.stdout: "172.21.15.92:0/3817549754": "2024-03-12T21:21:09.473916+0000", 2024-03-11T21:46:11.179 INFO:teuthology.orchestra.run.smithi092.stdout: "172.21.15.92:0/2211594569": "2024-03-12T21:21:09.473916+0000", 2024-03-11T21:46:11.179 INFO:teuthology.orchestra.run.smithi092.stdout: "172.21.15.92:0/3569900179": "2024-03-12T21:20:43.677016+0000", 2024-03-11T21:46:11.179 INFO:teuthology.orchestra.run.smithi092.stdout: "172.21.15.92:0/3380359534": "2024-03-12T21:20:43.677016+0000", 2024-03-11T21:46:11.180 INFO:teuthology.orchestra.run.smithi092.stdout: "172.21.15.92:0/2509373130": "2024-03-12T21:20:43.677016+0000", 2024-03-11T21:46:11.180 INFO:teuthology.orchestra.run.smithi092.stdout: "172.21.15.92:6800/2343383852": "2024-03-12T21:26:20.943013+0000", 2024-03-11T21:46:11.180 INFO:teuthology.orchestra.run.smithi092.stdout: "172.21.15.92:6800/1559007462": "2024-03-12T21:20:43.677016+0000", 2024-03-11T21:46:11.180 INFO:teuthology.orchestra.run.smithi092.stdout: "172.21.15.92:6800/1270902819": "2024-03-12T21:21:09.473916+0000", 2024-03-11T21:46:11.180 INFO:teuthology.orchestra.run.smithi092.stdout: "172.21.15.92:0/3562645553": "2024-03-12T21:21:09.473916+0000" 2024-03-11T21:46:11.180 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:46:11.180 INFO:teuthology.orchestra.run.smithi092.stdout: "range_blocklist": {}, 2024-03-11T21:46:11.180 INFO:teuthology.orchestra.run.smithi092.stdout: "erasure_code_profiles": { 2024-03-11T21:46:11.180 INFO:teuthology.orchestra.run.smithi092.stdout: "default": { 2024-03-11T21:46:11.180 INFO:teuthology.orchestra.run.smithi092.stdout: "crush-failure-domain": "osd", 2024-03-11T21:46:11.180 INFO:teuthology.orchestra.run.smithi092.stdout: "k": "2", 2024-03-11T21:46:11.180 INFO:teuthology.orchestra.run.smithi092.stdout: "m": "1", 2024-03-11T21:46:11.180 INFO:teuthology.orchestra.run.smithi092.stdout: "plugin": "jerasure", 2024-03-11T21:46:11.180 INFO:teuthology.orchestra.run.smithi092.stdout: "technique": "reed_sol_van" 2024-03-11T21:46:11.180 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-03-11T21:46:11.180 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:46:11.181 INFO:teuthology.orchestra.run.smithi092.stdout: "removed_snaps_queue": [], 2024-03-11T21:46:11.181 INFO:teuthology.orchestra.run.smithi092.stdout: "new_removed_snaps": [], 2024-03-11T21:46:11.181 INFO:teuthology.orchestra.run.smithi092.stdout: "new_purged_snaps": [], 2024-03-11T21:46:11.181 INFO:teuthology.orchestra.run.smithi092.stdout: "crush_node_flags": {}, 2024-03-11T21:46:11.181 INFO:teuthology.orchestra.run.smithi092.stdout: "device_class_flags": {}, 2024-03-11T21:46:11.181 INFO:teuthology.orchestra.run.smithi092.stdout: "stretch_mode": { 2024-03-11T21:46:11.181 INFO:teuthology.orchestra.run.smithi092.stdout: "stretch_mode_enabled": false, 2024-03-11T21:46:11.181 INFO:teuthology.orchestra.run.smithi092.stdout: "stretch_bucket_count": 0, 2024-03-11T21:46:11.181 INFO:teuthology.orchestra.run.smithi092.stdout: "degraded_stretch_mode": 0, 2024-03-11T21:46:11.181 INFO:teuthology.orchestra.run.smithi092.stdout: "recovering_stretch_mode": 0, 2024-03-11T21:46:11.181 INFO:teuthology.orchestra.run.smithi092.stdout: "stretch_mode_bucket": 0 2024-03-11T21:46:11.181 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-03-11T21:46:11.181 INFO:teuthology.orchestra.run.smithi092.stdout:} 2024-03-11T21:46:11.219 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:46:11.220 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:46:11.220 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:46:11.267 INFO:tasks.radosbench.radosbench.0.smithi092.stdout: 64 15 104629 104614 102.141 75.5625 0.00426673 0.00978562 2024-03-11T21:46:11.321 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:46:11.334 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:46:11 smithi097 ceph-mon[33317]: pgmap v971: 148 pgs: 148 active+clean; 6.1 GiB data, 23 GiB used, 514 GiB / 536 GiB avail; 1.7 KiB/s rd, 78 MiB/s wr, 1.25k op/s 2024-03-11T21:46:11.334 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:46:11 smithi097 ceph-mon[33317]: 5.f scrub starts 2024-03-11T21:46:11.334 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:46:11 smithi097 ceph-mon[33317]: 5.f scrub ok 2024-03-11T21:46:11.334 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:46:11 smithi097 ceph-mon[33317]: 4.6 scrub starts 2024-03-11T21:46:11.334 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:46:11 smithi097 ceph-mon[33317]: 4.6 scrub ok 2024-03-11T21:46:11.334 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:46:11 smithi097 ceph-mon[33317]: from='client.? v1:172.21.15.92:0/1994186373' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json-pretty"}]: dispatch 2024-03-11T21:46:11.358 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:46:11.359 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:46:11.360 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:46:11.461 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:46:11.503 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:46:11.503 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:46:11.503 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:46:11.604 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:46:11.618 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:46:11 smithi092 ceph-mon[29572]: pgmap v971: 148 pgs: 148 active+clean; 6.1 GiB data, 23 GiB used, 514 GiB / 536 GiB avail; 1.7 KiB/s rd, 78 MiB/s wr, 1.25k op/s 2024-03-11T21:46:11.618 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:46:11 smithi092 ceph-mon[29572]: 5.f scrub starts 2024-03-11T21:46:11.618 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:46:11 smithi092 ceph-mon[29572]: 5.f scrub ok 2024-03-11T21:46:11.618 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:46:11 smithi092 ceph-mon[29572]: 4.6 scrub starts 2024-03-11T21:46:11.618 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:46:11 smithi092 ceph-mon[29572]: 4.6 scrub ok 2024-03-11T21:46:11.618 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:46:11 smithi092 ceph-mon[29572]: from='client.? v1:172.21.15.92:0/1994186373' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json-pretty"}]: dispatch 2024-03-11T21:46:11.618 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:46:11 smithi092 ceph-mon[37563]: pgmap v971: 148 pgs: 148 active+clean; 6.1 GiB data, 23 GiB used, 514 GiB / 536 GiB avail; 1.7 KiB/s rd, 78 MiB/s wr, 1.25k op/s 2024-03-11T21:46:11.618 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:46:11 smithi092 ceph-mon[37563]: 5.f scrub starts 2024-03-11T21:46:11.618 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:46:11 smithi092 ceph-mon[37563]: 5.f scrub ok 2024-03-11T21:46:11.619 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:46:11 smithi092 ceph-mon[37563]: 4.6 scrub starts 2024-03-11T21:46:11.619 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:46:11 smithi092 ceph-mon[37563]: 4.6 scrub ok 2024-03-11T21:46:11.619 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:46:11 smithi092 ceph-mon[37563]: from='client.? v1:172.21.15.92:0/1994186373' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json-pretty"}]: dispatch 2024-03-11T21:46:11.633 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:46:11.634 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:46:11.634 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:46:11.735 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:46:11.772 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:46:11.773 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:46:11.774 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:46:11.874 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:46:11.911 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:46:11.911 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:46:11.912 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:46:12.012 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:46:12.019 DEBUG:teuthology.orchestra.run.smithi092:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid 00db0016-dfed-11ee-95c8-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-03-11T21:46:12.043 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:46:12.043 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:46:12.043 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:46:12.144 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:46:12.197 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:46:12.198 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:46:12.198 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:46:12.267 INFO:tasks.radosbench.radosbench.0.smithi092.stdout: 65 16 105968 105952 101.856 83.625 0.0133419 0.00981331 2024-03-11T21:46:12.301 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:46:12.335 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:46:12.336 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:46:12.336 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:46:12.368 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:46:12 smithi092 ceph-mon[29572]: 4.1d scrub starts 2024-03-11T21:46:12.368 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:46:12 smithi092 ceph-mon[29572]: 4.1d scrub ok 2024-03-11T21:46:12.368 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:46:12 smithi092 ceph-mon[37563]: 4.1d scrub starts 2024-03-11T21:46:12.368 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:46:12 smithi092 ceph-mon[37563]: 4.1d scrub ok 2024-03-11T21:46:12.437 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:46:12.464 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:46:12.465 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:46:12.465 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:46:12.489 INFO:tasks.thrashosds.thrasher:j is {'epoch': 462, 'fsid': '00db0016-dfed-11ee-95c8-87774f69a715', 'created': '2024-03-11T21:20:10.488059+0000', 'modified': '2024-03-11T21:46:09.563692+0000', 'last_up_change': '2024-03-11T21:36:34.026798+0000', 'last_in_change': '2024-03-11T21:44:34.572566+0000', 'flags': 'nodeep-scrub,sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit', 'flags_num': 5804032, 'flags_set': ['nodeep-scrub', 'pglog_hardlimit', 'purged_snapdirs', 'recovery_deletes', 'sortbitwise'], 'crush_version': 19, 'full_ratio': 0.949999988079071, 'backfillfull_ratio': 0.8999999761581421, 'nearfull_ratio': 0.8500000238418579, 'cluster_snapshot': '', 'pool_max': 11, 'max_osd': 8, 'require_min_compat_client': 'luminous', 'min_compat_client': 'luminous', 'require_osd_release': 'squid', 'allow_crimson': False, 'pools': [{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-11T21:23:23.805029+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '121', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 7.889999866485596, 'score_stable': 7.889999866485596, 'optimal_score': 0.3799999952316284, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': '.rgw.root', 'create_time': '2024-03-11T21:25:37.044504+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '68', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rgw': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.75, 'score_stable': 1.75, 'optimal_score': 0.75, 'raw_score_acting': 1.309999942779541, 'raw_score_stable': 1.309999942779541, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'datapool', 'create_time': '2024-03-11T21:25:37.280820+0000', 'flags': 8193, 'flags_names': 'hashpspool,selfmanaged_snaps', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 3, 'pg_placement_num': 3, 'pg_placement_num_target': 3, 'pg_num_target': 3, 'pg_num_pending': 3, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '71', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 2, 'snap_epoch': 71, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 2.6700000762939453, 'score_stable': 2.6700000762939453, 'optimal_score': 0.75, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'default.rgw.log', 'create_time': '2024-03-11T21:25:38.850207+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '70', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rgw': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 2.25, 'score_stable': 2.25, 'optimal_score': 0.75, 'raw_score_acting': 1.690000057220459, 'raw_score_stable': 1.690000057220459, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 5, 'pool_name': 'default.rgw.control', 'create_time': '2024-03-11T21:25:40.794686+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '72', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rgw': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5099999904632568, 'score_stable': 1.5099999904632568, 'optimal_score': 0.75, 'raw_score_acting': 1.1299999952316284, 'raw_score_stable': 1.1299999952316284, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 6, 'pool_name': 'default.rgw.meta', 'create_time': '2024-03-11T21:25:42.872077+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '74', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4}, 'application_metadata': {'rgw': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 2.25, 'score_stable': 2.25, 'optimal_score': 0.75, 'raw_score_acting': 1.690000057220459, 'raw_score_stable': 1.690000057220459, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 11, 'pool_name': 'unique_pool_4', 'create_time': '2024-03-11T21:44:52.522762+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 16, 'pg_placement_num': 16, 'pg_placement_num_target': 16, 'pg_num_target': 16, 'pg_num_pending': 16, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '450', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rados': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 2.509999990463257, 'score_stable': 2.509999990463257, 'optimal_score': 0.75, 'raw_score_acting': 1.8799999952316284, 'raw_score_stable': 1.8799999952316284, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}], 'osds': [{'osd': 0, 'uuid': '79f15843-f0ec-44aa-931e-4e07a74d3768', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 8, 'up_thru': 446, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.92:6801', 'nonce': 2844100201}]}, 'cluster_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.92:6802', 'nonce': 2844100201}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.92:6804', 'nonce': 2844100201}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.92:6803', 'nonce': 2844100201}]}, 'public_addr': '172.21.15.92:6801/2844100201', 'cluster_addr': '172.21.15.92:6802/2844100201', 'heartbeat_back_addr': '172.21.15.92:6804/2844100201', 'heartbeat_front_addr': '172.21.15.92:6803/2844100201', 'state': ['exists', 'up']}, {'osd': 1, 'uuid': '3a5a0e78-1293-4ace-a93f-69156098571d', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 13, 'up_thru': 446, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.92:6805', 'nonce': 3577871157}]}, 'cluster_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.92:6806', 'nonce': 3577871157}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.92:6808', 'nonce': 3577871157}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.92:6807', 'nonce': 3577871157}]}, 'public_addr': '172.21.15.92:6805/3577871157', 'cluster_addr': '172.21.15.92:6806/3577871157', 'heartbeat_back_addr': '172.21.15.92:6808/3577871157', 'heartbeat_front_addr': '172.21.15.92:6807/3577871157', 'state': ['exists', 'up']}, {'osd': 2, 'uuid': 'f207b743-0df4-40ed-a735-950e272fea60', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 18, 'up_thru': 446, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.92:6809', 'nonce': 760825152}]}, 'cluster_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.92:6810', 'nonce': 760825152}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.92:6812', 'nonce': 760825152}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.92:6811', 'nonce': 760825152}]}, 'public_addr': '172.21.15.92:6809/760825152', 'cluster_addr': '172.21.15.92:6810/760825152', 'heartbeat_back_addr': '172.21.15.92:6812/760825152', 'heartbeat_front_addr': '172.21.15.92:6811/760825152', 'state': ['exists', 'up']}, {'osd': 3, 'uuid': 'bbb6f69a-8f0f-47e6-937a-062e1902ae13', 'up': 1, 'in': 0, 'weight': 0, 'primary_affinity': 1, 'last_clean_begin': 26, 'last_clean_end': 251, 'up_from': 277, 'up_thru': 391, 'down_at': 252, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.92:6813', 'nonce': 3215041236}]}, 'cluster_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.92:6814', 'nonce': 3215041236}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.92:6816', 'nonce': 3215041236}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.92:6815', 'nonce': 3215041236}]}, 'public_addr': '172.21.15.92:6813/3215041236', 'cluster_addr': '172.21.15.92:6814/3215041236', 'heartbeat_back_addr': '172.21.15.92:6816/3215041236', 'heartbeat_front_addr': '172.21.15.92:6815/3215041236', 'state': ['exists', 'up']}, {'osd': 4, 'uuid': '00266cbc-d2cf-4dee-8a96-3753ebc95ca8', 'up': 1, 'in': 0, 'weight': 0, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 32, 'up_thru': 432, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.97:6800', 'nonce': 420733620}]}, 'cluster_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.97:6801', 'nonce': 420733620}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.97:6803', 'nonce': 420733620}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.97:6802', 'nonce': 420733620}]}, 'public_addr': '172.21.15.97:6800/420733620', 'cluster_addr': '172.21.15.97:6801/420733620', 'heartbeat_back_addr': '172.21.15.97:6803/420733620', 'heartbeat_front_addr': '172.21.15.97:6802/420733620', 'state': ['exists', 'up']}, {'osd': 5, 'uuid': '593a6f6b-8874-40ae-8848-d51c5723a5e4', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 41, 'up_thru': 446, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.97:6804', 'nonce': 312056041}]}, 'cluster_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.97:6805', 'nonce': 312056041}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.97:6807', 'nonce': 312056041}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.97:6806', 'nonce': 312056041}]}, 'public_addr': '172.21.15.97:6804/312056041', 'cluster_addr': '172.21.15.97:6805/312056041', 'heartbeat_back_addr': '172.21.15.97:6807/312056041', 'heartbeat_front_addr': '172.21.15.97:6806/312056041', 'state': ['exists', 'up']}, {'osd': 6, 'uuid': '91035ecc-6f1a-4cec-bda6-090af9d53eff', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 50, 'up_thru': 446, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.97:6808', 'nonce': 178994999}]}, 'cluster_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.97:6809', 'nonce': 178994999}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.97:6811', 'nonce': 178994999}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.97:6810', 'nonce': 178994999}]}, 'public_addr': '172.21.15.97:6808/178994999', 'cluster_addr': '172.21.15.97:6809/178994999', 'heartbeat_back_addr': '172.21.15.97:6811/178994999', 'heartbeat_front_addr': '172.21.15.97:6810/178994999', 'state': ['exists', 'up']}, {'osd': 7, 'uuid': '3eb28c24-664a-4bd6-869d-6bf5db98f52f', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 60, 'up_thru': 446, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.97:6812', 'nonce': 3241560509}]}, 'cluster_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.97:6813', 'nonce': 3241560509}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.97:6815', 'nonce': 3241560509}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.97:6814', 'nonce': 3241560509}]}, 'public_addr': '172.21.15.97:6812/3241560509', 'cluster_addr': '172.21.15.97:6813/3241560509', 'heartbeat_back_addr': '172.21.15.97:6815/3241560509', 'heartbeat_front_addr': '172.21.15.97:6814/3241560509', 'state': ['exists', 'up']}], 'osd_xinfo': [{'osd': 0, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-03-11T21:46:09.095379+0000', 'dead_epoch': 0}, {'osd': 1, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-03-11T21:44:52.116769+0000', 'dead_epoch': 0}, {'osd': 2, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-03-11T21:44:59.813525+0000', 'dead_epoch': 0}, {'osd': 3, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 65536, 'last_purged_snaps_scrub': '2024-03-11T21:45:22.817858+0000', 'dead_epoch': 251}, {'osd': 4, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 65536, 'last_purged_snaps_scrub': '2024-03-11T21:46:05.723463+0000', 'dead_epoch': 0}, {'osd': 5, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-03-11T21:45:07.242823+0000', 'dead_epoch': 0}, {'osd': 6, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-03-11T21:45:48.690289+0000', 'dead_epoch': 0}, {'osd': 7, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-03-11T21:45:23.323038+0000', 'dead_epoch': 0}], 'pg_upmap': [{'pgid': '6.a', 'osds': [4, 0, 3]}], 'pg_upmap_items': [{'pgid': '2.18', 'mappings': [{'from': 7, 'to': 2}]}, {'pgid': '4.6', 'mappings': [{'from': 1, 'to': 5}]}, {'pgid': '4.f', 'mappings': [{'from': 1, 'to': 2}]}, {'pgid': '4.13', 'mappings': [{'from': 1, 'to': 2}]}, {'pgid': '4.17', 'mappings': [{'from': 1, 'to': 5}]}, {'pgid': '4.1f', 'mappings': [{'from': 1, 'to': 0}]}], 'pg_upmap_primaries': [], 'pg_temp': [], 'primary_temp': [], 'blocklist': {'172.21.15.92:0/303695351': '2024-03-12T21:26:20.943013+0000', '172.21.15.92:0/437911229': '2024-03-12T21:26:20.943013+0000', '172.21.15.92:0/4246054050': '2024-03-12T21:26:20.943013+0000', '172.21.15.92:0/2912406656': '2024-03-12T21:26:20.943013+0000', '172.21.15.92:0/2706821241': '2024-03-12T21:26:20.943013+0000', '172.21.15.92:0/3817549754': '2024-03-12T21:21:09.473916+0000', '172.21.15.92:0/2211594569': '2024-03-12T21:21:09.473916+0000', '172.21.15.92:0/3569900179': '2024-03-12T21:20:43.677016+0000', '172.21.15.92:0/3380359534': '2024-03-12T21:20:43.677016+0000', '172.21.15.92:0/2509373130': '2024-03-12T21:20:43.677016+0000', '172.21.15.92:6800/2343383852': '2024-03-12T21:26:20.943013+0000', '172.21.15.92:6800/1559007462': '2024-03-12T21:20:43.677016+0000', '172.21.15.92:6800/1270902819': '2024-03-12T21:21:09.473916+0000', '172.21.15.92:0/3562645553': '2024-03-12T21:21:09.473916+0000'}, 'range_blocklist': {}, 'erasure_code_profiles': {'default': {'crush-failure-domain': 'osd', 'k': '2', 'm': '1', 'plugin': 'jerasure', 'technique': 'reed_sol_van'}}, 'removed_snaps_queue': [], 'new_removed_snaps': [], 'new_purged_snaps': [], 'crush_node_flags': {}, 'device_class_flags': {}, 'stretch_mode': {'stretch_mode_enabled': False, 'stretch_bucket_count': 0, 'degraded_stretch_mode': 0, 'recovering_stretch_mode': 0, 'stretch_mode_bucket': 0}} 2024-03-11T21:46:12.489 DEBUG:teuthology.orchestra.run.smithi092:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid 00db0016-dfed-11ee-95c8-87774f69a715 -- ceph pg dump --format=json 2024-03-11T21:46:12.566 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:46:12.583 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:46:12 smithi097 ceph-mon[33317]: 4.1d scrub starts 2024-03-11T21:46:12.584 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:46:12 smithi097 ceph-mon[33317]: 4.1d scrub ok 2024-03-11T21:46:12.617 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:46:12.618 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:46:12.618 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:46:12.719 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:46:12.753 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:46:12.753 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:46:12.753 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:46:12.854 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:46:12.888 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:46:12.888 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:46:12.889 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:46:12.989 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:46:13.022 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:46:13.022 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:46:13.023 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:46:13.123 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:46:13.171 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:46:13.174 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:46:13.174 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:46:13.267 INFO:tasks.radosbench.radosbench.0.smithi092.stdout: 66 16 107313 107297 101.586 84.0625 0.0141549 0.00983947 2024-03-11T21:46:13.275 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:46:13.317 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:46:13.317 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:46:13.318 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:46:13.419 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:46:13.458 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:46:13 smithi092 ceph-mon[29572]: pgmap v973: 148 pgs: 148 active+clean; 6.1 GiB data, 23 GiB used, 514 GiB / 536 GiB avail; 852 B/s rd, 61 MiB/s wr, 972 op/s 2024-03-11T21:46:13.458 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:46:13 smithi092 ceph-mon[29572]: 5.16 scrub starts 2024-03-11T21:46:13.458 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:46:13 smithi092 ceph-mon[29572]: 5.16 scrub ok 2024-03-11T21:46:13.459 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:46:13 smithi092 ceph-mon[37563]: pgmap v973: 148 pgs: 148 active+clean; 6.1 GiB data, 23 GiB used, 514 GiB / 536 GiB avail; 852 B/s rd, 61 MiB/s wr, 972 op/s 2024-03-11T21:46:13.459 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:46:13 smithi092 ceph-mon[37563]: 5.16 scrub starts 2024-03-11T21:46:13.460 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:46:13 smithi092 ceph-mon[37563]: 5.16 scrub ok 2024-03-11T21:46:13.461 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:46:13.461 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:46:13.462 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:46:13.491 INFO:teuthology.orchestra.run.smithi092.stderr:Inferring config /var/lib/ceph/00db0016-dfed-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T21:46:13.552 INFO:teuthology.orchestra.run.smithi092.stderr:Inferring config /var/lib/ceph/00db0016-dfed-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T21:46:13.562 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:46:13.583 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:46:13 smithi097 ceph-mon[33317]: pgmap v973: 148 pgs: 148 active+clean; 6.1 GiB data, 23 GiB used, 514 GiB / 536 GiB avail; 852 B/s rd, 61 MiB/s wr, 972 op/s 2024-03-11T21:46:13.584 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:46:13 smithi097 ceph-mon[33317]: 5.16 scrub starts 2024-03-11T21:46:13.584 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:46:13 smithi097 ceph-mon[33317]: 5.16 scrub ok 2024-03-11T21:46:13.584 INFO:journalctl@ceph.iscsi.iscsi.a.smithi097.stdout:Mar 11 21:46:13 smithi097 ceph-00db0016-dfed-11ee-95c8-87774f69a715-iscsi-iscsi-a[60943]: debug there is no tcmu-runner data available 2024-03-11T21:46:13.589 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:46:13.590 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:46:13.590 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:46:13.691 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:46:13.724 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:46:13.725 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:46:13.725 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:46:13.826 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:46:13.860 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:46:13.861 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:46:13.861 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:46:13.962 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:46:13.997 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:46:13.997 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:46:13.998 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:46:14.098 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:46:14.132 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:46:14.133 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:46:14.133 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:46:14.234 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:46:14.266 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:46:14.267 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:46:14.267 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:46:14.267 INFO:tasks.radosbench.radosbench.0.smithi092.stdout: 67 16 108637 108621 101.305 82.75 0.00387182 0.00986329 2024-03-11T21:46:14.368 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:46:14.408 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:46:14.409 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:46:14.409 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:46:14.510 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:46:14.543 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:46:14.543 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:46:14.544 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:46:14.565 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:46:14 smithi097 ceph-mon[33317]: 2.d scrub starts 2024-03-11T21:46:14.565 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:46:14 smithi097 ceph-mon[33317]: 2.d scrub ok 2024-03-11T21:46:14.565 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:46:14 smithi097 ceph-mon[33317]: from='client.24469 v1:172.21.15.97:0/404208310' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:46:14.565 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:46:14 smithi097 ceph-mon[33317]: 6.0 scrub starts 2024-03-11T21:46:14.565 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:46:14 smithi097 ceph-mon[33317]: 6.0 scrub ok 2024-03-11T21:46:14.565 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:46:14 smithi097 ceph-mon[33317]: osdmap e463: 8 total, 8 up, 6 in 2024-03-11T21:46:14.618 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:46:14 smithi092 ceph-mon[29572]: 2.d scrub starts 2024-03-11T21:46:14.618 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:46:14 smithi092 ceph-mon[29572]: 2.d scrub ok 2024-03-11T21:46:14.618 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:46:14 smithi092 ceph-mon[29572]: from='client.24469 v1:172.21.15.97:0/404208310' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:46:14.618 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:46:14 smithi092 ceph-mon[29572]: 6.0 scrub starts 2024-03-11T21:46:14.618 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:46:14 smithi092 ceph-mon[29572]: 6.0 scrub ok 2024-03-11T21:46:14.618 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:46:14 smithi092 ceph-mon[29572]: osdmap e463: 8 total, 8 up, 6 in 2024-03-11T21:46:14.618 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:46:14 smithi092 ceph-mon[37563]: 2.d scrub starts 2024-03-11T21:46:14.619 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:46:14 smithi092 ceph-mon[37563]: 2.d scrub ok 2024-03-11T21:46:14.619 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:46:14 smithi092 ceph-mon[37563]: from='client.24469 v1:172.21.15.97:0/404208310' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:46:14.619 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:46:14 smithi092 ceph-mon[37563]: 6.0 scrub starts 2024-03-11T21:46:14.619 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:46:14 smithi092 ceph-mon[37563]: 6.0 scrub ok 2024-03-11T21:46:14.619 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:46:14 smithi092 ceph-mon[37563]: osdmap e463: 8 total, 8 up, 6 in 2024-03-11T21:46:14.644 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:46:14.677 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:46:14.678 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:46:14.679 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:46:14.780 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:46:14.817 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:46:14.818 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:46:14.818 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:46:14.899 INFO:teuthology.orchestra.run.smithi097.stdout:{ 2024-03-11T21:46:14.899 INFO:teuthology.orchestra.run.smithi097.stdout: "ops": [], 2024-03-11T21:46:14.899 INFO:teuthology.orchestra.run.smithi097.stdout: "complaint_time": 30, 2024-03-11T21:46:14.899 INFO:teuthology.orchestra.run.smithi097.stdout: "num_blocked_ops": 0 2024-03-11T21:46:14.899 INFO:teuthology.orchestra.run.smithi097.stdout:} 2024-03-11T21:46:14.919 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:46:14.958 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:46:14.959 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:46:14.959 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:46:15.060 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:46:15.096 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:46:15.097 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:46:15.097 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:46:15.198 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:46:15.231 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:46:15.231 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:46:15.232 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:46:15.267 INFO:tasks.radosbench.radosbench.0.smithi092.stdout: 68 16 109941 109925 101.014 81.5 0.00646369 0.00989527 2024-03-11T21:46:15.333 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:46:15.356 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:46:15.356 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:46:15.357 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:46:15.458 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:46:15.493 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:46:15.494 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:46:15.494 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:46:15.584 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:46:15 smithi097 ceph-mon[33317]: pgmap v974: 148 pgs: 148 active+clean; 6.4 GiB data, 24 GiB used, 513 GiB / 536 GiB avail; 705 B/s rd, 87 MiB/s wr, 1.39k op/s 2024-03-11T21:46:15.584 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:46:15 smithi097 ceph-mon[33317]: 5.b scrub starts 2024-03-11T21:46:15.584 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:46:15 smithi097 ceph-mon[33317]: 5.b scrub ok 2024-03-11T21:46:15.584 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:46:15 smithi097 ceph-mon[33317]: 4.13 scrub starts 2024-03-11T21:46:15.584 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:46:15 smithi097 ceph-mon[33317]: 4.13 scrub ok 2024-03-11T21:46:15.584 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:46:15 smithi097 ceph-mon[33317]: 5.1b scrub starts 2024-03-11T21:46:15.584 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:46:15 smithi097 ceph-mon[33317]: 5.1b scrub ok 2024-03-11T21:46:15.596 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:46:15.618 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:46:15 smithi092 ceph-mon[29572]: pgmap v974: 148 pgs: 148 active+clean; 6.4 GiB data, 24 GiB used, 513 GiB / 536 GiB avail; 705 B/s rd, 87 MiB/s wr, 1.39k op/s 2024-03-11T21:46:15.618 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:46:15 smithi092 ceph-mon[29572]: 5.b scrub starts 2024-03-11T21:46:15.618 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:46:15 smithi092 ceph-mon[29572]: 5.b scrub ok 2024-03-11T21:46:15.618 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:46:15 smithi092 ceph-mon[29572]: 4.13 scrub starts 2024-03-11T21:46:15.618 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:46:15 smithi092 ceph-mon[29572]: 4.13 scrub ok 2024-03-11T21:46:15.618 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:46:15 smithi092 ceph-mon[29572]: 5.1b scrub starts 2024-03-11T21:46:15.619 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:46:15 smithi092 ceph-mon[29572]: 5.1b scrub ok 2024-03-11T21:46:15.619 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:46:15 smithi092 ceph-mon[37563]: pgmap v974: 148 pgs: 148 active+clean; 6.4 GiB data, 24 GiB used, 513 GiB / 536 GiB avail; 705 B/s rd, 87 MiB/s wr, 1.39k op/s 2024-03-11T21:46:15.619 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:46:15 smithi092 ceph-mon[37563]: 5.b scrub starts 2024-03-11T21:46:15.619 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:46:15 smithi092 ceph-mon[37563]: 5.b scrub ok 2024-03-11T21:46:15.619 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:46:15 smithi092 ceph-mon[37563]: 4.13 scrub starts 2024-03-11T21:46:15.619 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:46:15 smithi092 ceph-mon[37563]: 4.13 scrub ok 2024-03-11T21:46:15.619 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:46:15 smithi092 ceph-mon[37563]: 5.1b scrub starts 2024-03-11T21:46:15.619 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:46:15 smithi092 ceph-mon[37563]: 5.1b scrub ok 2024-03-11T21:46:15.626 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:46:15.627 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:46:15.627 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:46:15.728 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:46:15.769 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:46:15.770 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:46:15.770 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:46:15.845 DEBUG:teuthology.orchestra.run.smithi097:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid 00db0016-dfed-11ee-95c8-87774f69a715 -- ceph daemon osd.6 dump_historic_ops 2024-03-11T21:46:15.870 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:46:15.937 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:46:15.938 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:46:15.938 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:46:16.039 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:46:16.077 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:46:16.078 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:46:16.078 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:46:16.179 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:46:16.230 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:46:16.231 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:46:16.231 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:46:16.242 INFO:teuthology.orchestra.run.smithi097.stderr:Inferring config /var/lib/ceph/00db0016-dfed-11ee-95c8-87774f69a715/mon.b/config 2024-03-11T21:46:16.267 INFO:tasks.radosbench.radosbench.0.smithi092.stdout: 69 16 111171 111155 100.664 76.875 0.00550227 0.00992711 2024-03-11T21:46:16.332 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:46:16.367 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:46:16.368 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:46:16.369 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:46:16.470 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:46:16.498 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:46:16.498 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:46:16.499 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:46:16.584 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:46:16 smithi097 ceph-mon[33317]: 6.f scrub starts 2024-03-11T21:46:16.584 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:46:16 smithi097 ceph-mon[33317]: 6.f scrub ok 2024-03-11T21:46:16.584 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:46:16 smithi097 ceph-mon[33317]: 5.1e scrub starts 2024-03-11T21:46:16.584 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:46:16 smithi097 ceph-mon[33317]: 5.1e scrub ok 2024-03-11T21:46:16.599 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:46:16.618 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:46:16 smithi092 ceph-mon[29572]: 6.f scrub starts 2024-03-11T21:46:16.618 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:46:16 smithi092 ceph-mon[29572]: 6.f scrub ok 2024-03-11T21:46:16.618 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:46:16 smithi092 ceph-mon[29572]: 5.1e scrub starts 2024-03-11T21:46:16.618 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:46:16 smithi092 ceph-mon[29572]: 5.1e scrub ok 2024-03-11T21:46:16.618 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:46:16 smithi092 ceph-mon[37563]: 6.f scrub starts 2024-03-11T21:46:16.618 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:46:16 smithi092 ceph-mon[37563]: 6.f scrub ok 2024-03-11T21:46:16.618 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:46:16 smithi092 ceph-mon[37563]: 5.1e scrub starts 2024-03-11T21:46:16.618 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:46:16 smithi092 ceph-mon[37563]: 5.1e scrub ok 2024-03-11T21:46:16.643 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:46:16.643 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:46:16.643 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:46:16.744 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:46:16.776 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:46:16.778 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:46:16.778 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:46:16.879 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:46:16.912 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:46:16.913 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:46:16.913 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:46:17.014 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:46:17.052 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:46:17.052 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:46:17.053 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:46:17.154 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:46:17.180 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:46:17.181 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:46:17.181 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:46:17.268 INFO:tasks.radosbench.radosbench.0.smithi092.stdout: 70 16 112344 112328 100.273 73.3125 0.0130311 0.00996388 2024-03-11T21:46:17.282 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:46:17.324 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:46:17.324 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:46:17.324 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:46:17.425 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:46:17.466 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:46:17.467 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:46:17.467 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:46:17.568 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:46:17.584 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:46:17 smithi097 ceph-mon[33317]: pgmap v976: 148 pgs: 148 active+clean; 6.5 GiB data, 24 GiB used, 512 GiB / 536 GiB avail; 1.4 KiB/s rd, 74 MiB/s wr, 1.19k op/s 2024-03-11T21:46:17.584 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:46:17 smithi097 ceph-mon[33317]: purged_snaps scrub starts 2024-03-11T21:46:17.584 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:46:17 smithi097 ceph-mon[33317]: purged_snaps scrub ok 2024-03-11T21:46:17.584 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:46:17 smithi097 ceph-mon[33317]: 6.1a scrub starts 2024-03-11T21:46:17.584 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:46:17 smithi097 ceph-mon[33317]: 6.1a scrub ok 2024-03-11T21:46:17.584 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:46:17 smithi097 ceph-mon[33317]: osdmap e464: 8 total, 8 up, 6 in 2024-03-11T21:46:17.604 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:46:17.605 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:46:17.605 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:46:17.618 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:46:17 smithi092 ceph-mon[29572]: pgmap v976: 148 pgs: 148 active+clean; 6.5 GiB data, 24 GiB used, 512 GiB / 536 GiB avail; 1.4 KiB/s rd, 74 MiB/s wr, 1.19k op/s 2024-03-11T21:46:17.618 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:46:17 smithi092 ceph-mon[29572]: purged_snaps scrub starts 2024-03-11T21:46:17.618 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:46:17 smithi092 ceph-mon[29572]: purged_snaps scrub ok 2024-03-11T21:46:17.618 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:46:17 smithi092 ceph-mon[29572]: 6.1a scrub starts 2024-03-11T21:46:17.618 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:46:17 smithi092 ceph-mon[29572]: 6.1a scrub ok 2024-03-11T21:46:17.618 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:46:17 smithi092 ceph-mon[29572]: osdmap e464: 8 total, 8 up, 6 in 2024-03-11T21:46:17.618 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:46:17 smithi092 ceph-mon[37563]: pgmap v976: 148 pgs: 148 active+clean; 6.5 GiB data, 24 GiB used, 512 GiB / 536 GiB avail; 1.4 KiB/s rd, 74 MiB/s wr, 1.19k op/s 2024-03-11T21:46:17.618 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:46:17 smithi092 ceph-mon[37563]: purged_snaps scrub starts 2024-03-11T21:46:17.618 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:46:17 smithi092 ceph-mon[37563]: purged_snaps scrub ok 2024-03-11T21:46:17.619 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:46:17 smithi092 ceph-mon[37563]: 6.1a scrub starts 2024-03-11T21:46:17.619 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:46:17 smithi092 ceph-mon[37563]: 6.1a scrub ok 2024-03-11T21:46:17.619 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:46:17 smithi092 ceph-mon[37563]: osdmap e464: 8 total, 8 up, 6 in 2024-03-11T21:46:17.706 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:46:17.739 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:46:17.740 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:46:17.741 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:46:17.842 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:46:17.901 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:46:17.901 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:46:17.902 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:46:18.002 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:46:18.038 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:46:18.039 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:46:18.039 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:46:18.140 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:46:18.181 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:46:18.182 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:46:18.182 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:46:18.268 INFO:tasks.radosbench.radosbench.0.smithi092.stdout: 71 16 113640 113624 100.001 81 0.0194324 0.00999449 2024-03-11T21:46:18.271 INFO:journalctl@ceph.mgr.y.smithi092.stdout:Mar 11 21:46:17 smithi092 ceph-00db0016-dfed-11ee-95c8-87774f69a715-mgr-y[29801]: ::ffff:172.21.15.97 - - [11/Mar/2024:21:46:17] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-11T21:46:18.283 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:46:18.323 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:46:18 smithi097 ceph-mon[33317]: 4.1f scrub starts 2024-03-11T21:46:18.324 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:46:18 smithi097 ceph-mon[33317]: 4.1f scrub ok 2024-03-11T21:46:18.324 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:46:18.325 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:46:18.325 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:46:18.426 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:46:18.465 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:46:18.465 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:46:18.465 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:46:18.539 INFO:teuthology.orchestra.run.smithi097.stdout:{ 2024-03-11T21:46:18.539 INFO:teuthology.orchestra.run.smithi097.stdout: "size": 20, 2024-03-11T21:46:18.539 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 600, 2024-03-11T21:46:18.539 INFO:teuthology.orchestra.run.smithi097.stdout: "ops": [ 2024-03-11T21:46:18.539 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:46:18.539 INFO:teuthology.orchestra.run.smithi097.stdout: "description": "osd_op(client.25282.0:76286 9.1 9:80462a16:::benchmark_data_smithi092_130291_object76285:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e278)", 2024-03-11T21:46:18.539 INFO:teuthology.orchestra.run.smithi097.stdout: "initiated_at": "2024-03-11T21:36:35.047665+0000", 2024-03-11T21:46:18.539 INFO:teuthology.orchestra.run.smithi097.stdout: "age": 583.49777045999997, 2024-03-11T21:46:18.540 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.4732240720000001, 2024-03-11T21:46:18.540 INFO:teuthology.orchestra.run.smithi097.stdout: "type_data": { 2024-03-11T21:46:18.540 INFO:teuthology.orchestra.run.smithi097.stdout: "flag_point": "reached pg", 2024-03-11T21:46:18.540 INFO:teuthology.orchestra.run.smithi097.stdout: "client_info": { 2024-03-11T21:46:18.540 INFO:teuthology.orchestra.run.smithi097.stdout: "client": "client.25282", 2024-03-11T21:46:18.540 INFO:teuthology.orchestra.run.smithi097.stdout: "client_addr": "v1:172.21.15.92:0/2489519821", 2024-03-11T21:46:18.540 INFO:teuthology.orchestra.run.smithi097.stdout: "tid": 76286 2024-03-11T21:46:18.540 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:46:18.540 INFO:teuthology.orchestra.run.smithi097.stdout: "events": [ 2024-03-11T21:46:18.540 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:46:18.541 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "initiated", 2024-03-11T21:46:18.541 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:36:35.047665+0000", 2024-03-11T21:46:18.541 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-03-11T21:46:18.541 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:46:18.541 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:46:18.541 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "header_read", 2024-03-11T21:46:18.541 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:36:35.047665+0000", 2024-03-11T21:46:18.541 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-03-11T21:46:18.541 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:46:18.542 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:46:18.542 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "throttled", 2024-03-11T21:46:18.542 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:36:35.047667+0000", 2024-03-11T21:46:18.542 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 2.638e-06 2024-03-11T21:46:18.542 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:46:18.542 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:46:18.542 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "all_read", 2024-03-11T21:46:18.542 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:36:35.047806+0000", 2024-03-11T21:46:18.542 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.000139025 2024-03-11T21:46:18.542 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:46:18.542 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:46:18.542 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "dispatched", 2024-03-11T21:46:18.542 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:36:35.047824+0000", 2024-03-11T21:46:18.543 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.7374e-05 2024-03-11T21:46:18.543 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:46:18.543 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:46:18.543 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "queued_for_pg", 2024-03-11T21:46:18.543 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:36:35.047836+0000", 2024-03-11T21:46:18.543 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.1960999999999999e-05 2024-03-11T21:46:18.543 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:46:18.543 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:46:18.543 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "reached_pg", 2024-03-11T21:46:18.543 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:36:35.048011+0000", 2024-03-11T21:46:18.543 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.000174835 2024-03-11T21:46:18.543 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:46:18.543 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:46:18.543 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "waiting for peered", 2024-03-11T21:46:18.544 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:36:35.048021+0000", 2024-03-11T21:46:18.544 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.082e-05 2024-03-11T21:46:18.544 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:46:18.544 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:46:18.544 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "reached_pg", 2024-03-11T21:46:18.544 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:36:36.520717+0000", 2024-03-11T21:46:18.544 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.4726953599999999 2024-03-11T21:46:18.544 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:46:18.544 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:46:18.544 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "done", 2024-03-11T21:46:18.544 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:36:36.520889+0000", 2024-03-11T21:46:18.544 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.00017205899999999999 2024-03-11T21:46:18.544 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-03-11T21:46:18.544 INFO:teuthology.orchestra.run.smithi097.stdout: ] 2024-03-11T21:46:18.545 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-03-11T21:46:18.545 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:46:18.545 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:46:18.545 INFO:teuthology.orchestra.run.smithi097.stdout: "description": "osd_op(client.25282.0:76290 9.1 9:890a0ee5:::benchmark_data_smithi092_130291_object76289:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e278)", 2024-03-11T21:46:18.545 INFO:teuthology.orchestra.run.smithi097.stdout: "initiated_at": "2024-03-11T21:36:35.047859+0000", 2024-03-11T21:46:18.545 INFO:teuthology.orchestra.run.smithi097.stdout: "age": 583.49757621200001, 2024-03-11T21:46:18.545 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.497780396, 2024-03-11T21:46:18.545 INFO:teuthology.orchestra.run.smithi097.stdout: "type_data": { 2024-03-11T21:46:18.545 INFO:teuthology.orchestra.run.smithi097.stdout: "flag_point": "reached pg", 2024-03-11T21:46:18.545 INFO:teuthology.orchestra.run.smithi097.stdout: "client_info": { 2024-03-11T21:46:18.545 INFO:teuthology.orchestra.run.smithi097.stdout: "client": "client.25282", 2024-03-11T21:46:18.545 INFO:teuthology.orchestra.run.smithi097.stdout: "client_addr": "v1:172.21.15.92:0/2489519821", 2024-03-11T21:46:18.545 INFO:teuthology.orchestra.run.smithi097.stdout: "tid": 76290 2024-03-11T21:46:18.546 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:46:18.546 INFO:teuthology.orchestra.run.smithi097.stdout: "events": [ 2024-03-11T21:46:18.546 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:46:18.546 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "initiated", 2024-03-11T21:46:18.546 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:36:35.047859+0000", 2024-03-11T21:46:18.546 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-03-11T21:46:18.546 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:46:18.546 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:46:18.546 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "header_read", 2024-03-11T21:46:18.546 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:36:35.047859+0000", 2024-03-11T21:46:18.546 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-03-11T21:46:18.546 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:46:18.546 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:46:18.546 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "throttled", 2024-03-11T21:46:18.547 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:36:35.047860+0000", 2024-03-11T21:46:18.547 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.114e-06 2024-03-11T21:46:18.547 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:46:18.547 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:46:18.547 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "all_read", 2024-03-11T21:46:18.547 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:36:35.047905+0000", 2024-03-11T21:46:18.547 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 4.5263999999999999e-05 2024-03-11T21:46:18.547 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:46:18.547 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:46:18.547 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "dispatched", 2024-03-11T21:46:18.547 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:36:35.047914+0000", 2024-03-11T21:46:18.547 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 9.1009999999999998e-06 2024-03-11T21:46:18.547 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:46:18.547 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:46:18.548 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "queued_for_pg", 2024-03-11T21:46:18.548 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:36:35.047923+0000", 2024-03-11T21:46:18.548 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 8.3440000000000001e-06 2024-03-11T21:46:18.548 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:46:18.548 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:46:18.548 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "reached_pg", 2024-03-11T21:46:18.548 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:36:35.048076+0000", 2024-03-11T21:46:18.548 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.00015320000000000001 2024-03-11T21:46:18.548 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:46:18.548 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:46:18.548 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "waiting for peered", 2024-03-11T21:46:18.548 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:36:35.048085+0000", 2024-03-11T21:46:18.548 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 9.5089999999999999e-06 2024-03-11T21:46:18.548 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:46:18.548 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:46:18.549 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "reached_pg", 2024-03-11T21:46:18.549 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:36:36.524763+0000", 2024-03-11T21:46:18.549 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.4766779940000001 2024-03-11T21:46:18.549 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:46:18.549 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:46:18.549 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "waiting for degraded object", 2024-03-11T21:46:18.549 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:36:36.525351+0000", 2024-03-11T21:46:18.549 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.00058758699999999998 2024-03-11T21:46:18.549 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:46:18.549 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:46:18.549 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "reached_pg", 2024-03-11T21:46:18.549 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:36:36.545558+0000", 2024-03-11T21:46:18.549 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.020206884000000001 2024-03-11T21:46:18.549 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:46:18.549 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:46:18.550 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "done", 2024-03-11T21:46:18.550 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:36:36.545639+0000", 2024-03-11T21:46:18.550 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 8.1398999999999998e-05 2024-03-11T21:46:18.550 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-03-11T21:46:18.550 INFO:teuthology.orchestra.run.smithi097.stdout: ] 2024-03-11T21:46:18.550 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-03-11T21:46:18.550 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:46:18.550 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:46:18.550 INFO:teuthology.orchestra.run.smithi097.stdout: "description": "osd_op(client.15744.0:69291 10.17 10:eb3f93ab:::benchmark_data_smithi092_161462_object69290:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e372)", 2024-03-11T21:46:18.550 INFO:teuthology.orchestra.run.smithi097.stdout: "initiated_at": "2024-03-11T21:41:27.826508+0000", 2024-03-11T21:46:18.550 INFO:teuthology.orchestra.run.smithi097.stdout: "age": 290.71892699099999, 2024-03-11T21:46:18.550 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.022082605, 2024-03-11T21:46:18.550 INFO:teuthology.orchestra.run.smithi097.stdout: "type_data": { 2024-03-11T21:46:18.551 INFO:teuthology.orchestra.run.smithi097.stdout: "flag_point": "reached pg", 2024-03-11T21:46:18.551 INFO:teuthology.orchestra.run.smithi097.stdout: "client_info": { 2024-03-11T21:46:18.551 INFO:teuthology.orchestra.run.smithi097.stdout: "client": "client.15744", 2024-03-11T21:46:18.551 INFO:teuthology.orchestra.run.smithi097.stdout: "client_addr": "v1:172.21.15.92:0/3334000812", 2024-03-11T21:46:18.551 INFO:teuthology.orchestra.run.smithi097.stdout: "tid": 69291 2024-03-11T21:46:18.551 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:46:18.551 INFO:teuthology.orchestra.run.smithi097.stdout: "events": [ 2024-03-11T21:46:18.551 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:46:18.551 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "initiated", 2024-03-11T21:46:18.551 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:41:27.826508+0000", 2024-03-11T21:46:18.551 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-03-11T21:46:18.551 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:46:18.551 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:46:18.551 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "header_read", 2024-03-11T21:46:18.552 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:41:27.826508+0000", 2024-03-11T21:46:18.552 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-03-11T21:46:18.552 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:46:18.552 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:46:18.552 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "throttled", 2024-03-11T21:46:18.552 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:41:27.826510+0000", 2024-03-11T21:46:18.552 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 2.1799999999999999e-06 2024-03-11T21:46:18.552 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:46:18.552 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:46:18.552 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "all_read", 2024-03-11T21:46:18.552 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:41:27.826614+0000", 2024-03-11T21:46:18.552 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.00010393199999999999 2024-03-11T21:46:18.552 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:46:18.553 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:46:18.553 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "dispatched", 2024-03-11T21:46:18.553 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:41:27.826625+0000", 2024-03-11T21:46:18.553 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.0386e-05 2024-03-11T21:46:18.553 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:46:18.553 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:46:18.553 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "queued_for_pg", 2024-03-11T21:46:18.553 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:41:27.826633+0000", 2024-03-11T21:46:18.553 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 8.1280000000000008e-06 2024-03-11T21:46:18.554 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:46:18.554 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:46:18.554 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "reached_pg", 2024-03-11T21:46:18.554 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:41:27.952571+0000", 2024-03-11T21:46:18.554 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.125938363 2024-03-11T21:46:18.554 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:46:18.554 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:46:18.554 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "waiting for peered", 2024-03-11T21:46:18.554 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:41:27.952582+0000", 2024-03-11T21:46:18.554 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.0701e-05 2024-03-11T21:46:18.555 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:46:18.555 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:46:18.555 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "reached_pg", 2024-03-11T21:46:18.555 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:41:28.848505+0000", 2024-03-11T21:46:18.555 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.89592317700000002 2024-03-11T21:46:18.555 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:46:18.555 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:46:18.555 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "done", 2024-03-11T21:46:18.555 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:41:28.848591+0000", 2024-03-11T21:46:18.555 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 8.5737999999999998e-05 2024-03-11T21:46:18.555 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-03-11T21:46:18.555 INFO:teuthology.orchestra.run.smithi097.stdout: ] 2024-03-11T21:46:18.555 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-03-11T21:46:18.555 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:46:18.556 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:46:18.556 INFO:teuthology.orchestra.run.smithi097.stdout: "description": "osd_op(client.15744.0:70476 10.12 10.8a949fd2 (undecoded) ondisk+retry+write+known_if_redirected+supports_pool_eio e374)", 2024-03-11T21:46:18.556 INFO:teuthology.orchestra.run.smithi097.stdout: "initiated_at": "2024-03-11T21:41:29.844672+0000", 2024-03-11T21:46:18.556 INFO:teuthology.orchestra.run.smithi097.stdout: "age": 288.70076278800002, 2024-03-11T21:46:18.556 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.013244195, 2024-03-11T21:46:18.556 INFO:teuthology.orchestra.run.smithi097.stdout: "type_data": { 2024-03-11T21:46:18.556 INFO:teuthology.orchestra.run.smithi097.stdout: "flag_point": "reached pg", 2024-03-11T21:46:18.556 INFO:teuthology.orchestra.run.smithi097.stdout: "client_info": { 2024-03-11T21:46:18.556 INFO:teuthology.orchestra.run.smithi097.stdout: "client": "client.15744", 2024-03-11T21:46:18.556 INFO:teuthology.orchestra.run.smithi097.stdout: "client_addr": "v1:172.21.15.92:0/3334000812", 2024-03-11T21:46:18.556 INFO:teuthology.orchestra.run.smithi097.stdout: "tid": 70476 2024-03-11T21:46:18.556 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:46:18.556 INFO:teuthology.orchestra.run.smithi097.stdout: "events": [ 2024-03-11T21:46:18.557 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:46:18.557 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "initiated", 2024-03-11T21:46:18.557 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:41:29.844672+0000", 2024-03-11T21:46:18.557 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-03-11T21:46:18.557 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:46:18.557 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:46:18.557 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "header_read", 2024-03-11T21:46:18.557 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:41:29.844672+0000", 2024-03-11T21:46:18.557 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-03-11T21:46:18.557 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:46:18.557 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:46:18.557 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "throttled", 2024-03-11T21:46:18.557 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:41:29.844674+0000", 2024-03-11T21:46:18.557 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 2.0590000000000001e-06 2024-03-11T21:46:18.557 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:46:18.558 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:46:18.558 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "all_read", 2024-03-11T21:46:18.558 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:41:29.844707+0000", 2024-03-11T21:46:18.558 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 3.2085000000000002e-05 2024-03-11T21:46:18.558 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:46:18.558 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:46:18.558 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "dispatched", 2024-03-11T21:46:18.558 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:41:29.844723+0000", 2024-03-11T21:46:18.558 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.6390999999999999e-05 2024-03-11T21:46:18.558 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:46:18.558 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:46:18.558 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "queued_for_pg", 2024-03-11T21:46:18.558 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:41:29.844732+0000", 2024-03-11T21:46:18.558 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 8.7630000000000006e-06 2024-03-11T21:46:18.559 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:46:18.559 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:46:18.559 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "reached_pg", 2024-03-11T21:46:18.559 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:41:29.844792+0000", 2024-03-11T21:46:18.559 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 6.0325000000000003e-05 2024-03-11T21:46:18.559 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:46:18.559 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:46:18.559 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "waiting for peered", 2024-03-11T21:46:18.559 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:41:29.844800+0000", 2024-03-11T21:46:18.559 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 8.2190000000000004e-06 2024-03-11T21:46:18.559 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:46:18.559 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:46:18.559 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "reached_pg", 2024-03-11T21:46:18.559 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:41:30.857903+0000", 2024-03-11T21:46:18.560 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.0131028959999999 2024-03-11T21:46:18.560 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:46:18.560 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:46:18.560 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "done", 2024-03-11T21:46:18.560 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:41:30.857917+0000", 2024-03-11T21:46:18.560 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.3457e-05 2024-03-11T21:46:18.560 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-03-11T21:46:18.560 INFO:teuthology.orchestra.run.smithi097.stdout: ] 2024-03-11T21:46:18.560 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-03-11T21:46:18.560 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:46:18.560 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:46:18.560 INFO:teuthology.orchestra.run.smithi097.stdout: "description": "osd_op(client.15744.0:70481 10.12 10.4b903cf2 (undecoded) ondisk+retry+write+known_if_redirected+supports_pool_eio e374)", 2024-03-11T21:46:18.560 INFO:teuthology.orchestra.run.smithi097.stdout: "initiated_at": "2024-03-11T21:41:29.844814+0000", 2024-03-11T21:46:18.560 INFO:teuthology.orchestra.run.smithi097.stdout: "age": 288.70062148, 2024-03-11T21:46:18.560 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.013178921, 2024-03-11T21:46:18.561 INFO:teuthology.orchestra.run.smithi097.stdout: "type_data": { 2024-03-11T21:46:18.561 INFO:teuthology.orchestra.run.smithi097.stdout: "flag_point": "reached pg", 2024-03-11T21:46:18.561 INFO:teuthology.orchestra.run.smithi097.stdout: "client_info": { 2024-03-11T21:46:18.561 INFO:teuthology.orchestra.run.smithi097.stdout: "client": "client.15744", 2024-03-11T21:46:18.561 INFO:teuthology.orchestra.run.smithi097.stdout: "client_addr": "v1:172.21.15.92:0/3334000812", 2024-03-11T21:46:18.561 INFO:teuthology.orchestra.run.smithi097.stdout: "tid": 70481 2024-03-11T21:46:18.561 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:46:18.561 INFO:teuthology.orchestra.run.smithi097.stdout: "events": [ 2024-03-11T21:46:18.561 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:46:18.561 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "initiated", 2024-03-11T21:46:18.561 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:41:29.844814+0000", 2024-03-11T21:46:18.561 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-03-11T21:46:18.561 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:46:18.561 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:46:18.561 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "header_read", 2024-03-11T21:46:18.561 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:41:29.844814+0000", 2024-03-11T21:46:18.562 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-03-11T21:46:18.562 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:46:18.562 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:46:18.562 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "throttled", 2024-03-11T21:46:18.562 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:41:29.844815+0000", 2024-03-11T21:46:18.562 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 9.3600000000000002e-07 2024-03-11T21:46:18.562 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:46:18.562 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:46:18.562 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "all_read", 2024-03-11T21:46:18.562 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:41:29.844844+0000", 2024-03-11T21:46:18.562 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 2.9799999999999999e-05 2024-03-11T21:46:18.562 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:46:18.562 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:46:18.562 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "dispatched", 2024-03-11T21:46:18.563 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:41:29.844852+0000", 2024-03-11T21:46:18.563 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 7.8590000000000005e-06 2024-03-11T21:46:18.563 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:46:18.563 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:46:18.563 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "queued_for_pg", 2024-03-11T21:46:18.563 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:41:29.844859+0000", 2024-03-11T21:46:18.563 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 6.6540000000000002e-06 2024-03-11T21:46:18.563 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:46:18.563 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:46:18.563 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "reached_pg", 2024-03-11T21:46:18.563 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:41:29.844888+0000", 2024-03-11T21:46:18.563 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 2.8994999999999999e-05 2024-03-11T21:46:18.563 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:46:18.563 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:46:18.564 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "waiting for peered", 2024-03-11T21:46:18.564 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:41:29.844892+0000", 2024-03-11T21:46:18.564 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 3.6239999999999999e-06 2024-03-11T21:46:18.564 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:46:18.564 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:46:18.564 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "reached_pg", 2024-03-11T21:46:18.564 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:41:30.857985+0000", 2024-03-11T21:46:18.564 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.013093762 2024-03-11T21:46:18.564 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:46:18.564 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:46:18.564 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "done", 2024-03-11T21:46:18.564 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:41:30.857993+0000", 2024-03-11T21:46:18.564 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 7.2910000000000002e-06 2024-03-11T21:46:18.564 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-03-11T21:46:18.564 INFO:teuthology.orchestra.run.smithi097.stdout: ] 2024-03-11T21:46:18.565 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-03-11T21:46:18.565 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:46:18.565 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:46:18.565 INFO:teuthology.orchestra.run.smithi097.stdout: "description": "osd_op(client.15744.0:70510 10.12 10.8c0d3692 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e374)", 2024-03-11T21:46:18.565 INFO:teuthology.orchestra.run.smithi097.stdout: "initiated_at": "2024-03-11T21:41:29.870022+0000", 2024-03-11T21:46:18.565 INFO:teuthology.orchestra.run.smithi097.stdout: "age": 288.67541272199998, 2024-03-11T21:46:18.565 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.98800167299999997, 2024-03-11T21:46:18.565 INFO:teuthology.orchestra.run.smithi097.stdout: "type_data": { 2024-03-11T21:46:18.565 INFO:teuthology.orchestra.run.smithi097.stdout: "flag_point": "reached pg", 2024-03-11T21:46:18.565 INFO:teuthology.orchestra.run.smithi097.stdout: "client_info": { 2024-03-11T21:46:18.565 INFO:teuthology.orchestra.run.smithi097.stdout: "client": "client.15744", 2024-03-11T21:46:18.565 INFO:teuthology.orchestra.run.smithi097.stdout: "client_addr": "v1:172.21.15.92:0/3334000812", 2024-03-11T21:46:18.565 INFO:teuthology.orchestra.run.smithi097.stdout: "tid": 70510 2024-03-11T21:46:18.565 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:46:18.566 INFO:teuthology.orchestra.run.smithi097.stdout: "events": [ 2024-03-11T21:46:18.566 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:46:18.566 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "initiated", 2024-03-11T21:46:18.566 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:41:29.870022+0000", 2024-03-11T21:46:18.566 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-03-11T21:46:18.566 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:46:18.566 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:46:18.566 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "header_read", 2024-03-11T21:46:18.566 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:41:29.870022+0000", 2024-03-11T21:46:18.566 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-03-11T21:46:18.566 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:46:18.566 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:46:18.566 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "throttled", 2024-03-11T21:46:18.566 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:41:29.870024+0000", 2024-03-11T21:46:18.567 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.1400000000000001e-06 2024-03-11T21:46:18.567 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:46:18.567 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:46:18.567 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "all_read", 2024-03-11T21:46:18.567 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:41:29.870084+0000", 2024-03-11T21:46:18.567 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 6.0297999999999998e-05 2024-03-11T21:46:18.567 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:46:18.567 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:46:18.567 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "dispatched", 2024-03-11T21:46:18.567 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:41:29.870093+0000", 2024-03-11T21:46:18.567 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 8.7579999999999998e-06 2024-03-11T21:46:18.567 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:46:18.567 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:46:18.567 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "queued_for_pg", 2024-03-11T21:46:18.568 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:41:29.870100+0000", 2024-03-11T21:46:18.568 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 7.0570000000000004e-06 2024-03-11T21:46:18.568 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:46:18.568 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:46:18.568 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "reached_pg", 2024-03-11T21:46:18.568 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:41:29.870165+0000", 2024-03-11T21:46:18.568 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 6.5172000000000006e-05 2024-03-11T21:46:18.568 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:46:18.568 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:46:18.568 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "waiting for peered", 2024-03-11T21:46:18.568 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:41:29.870172+0000", 2024-03-11T21:46:18.568 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 7.3139999999999998e-06 2024-03-11T21:46:18.568 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:46:18.568 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:46:18.569 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "reached_pg", 2024-03-11T21:46:18.569 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:41:30.858016+0000", 2024-03-11T21:46:18.569 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.98784347900000002 2024-03-11T21:46:18.569 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:46:18.569 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:46:18.569 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "done", 2024-03-11T21:46:18.569 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:41:30.858024+0000", 2024-03-11T21:46:18.569 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 8.4549999999999995e-06 2024-03-11T21:46:18.569 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-03-11T21:46:18.569 INFO:teuthology.orchestra.run.smithi097.stdout: ] 2024-03-11T21:46:18.569 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-03-11T21:46:18.569 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:46:18.569 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:46:18.569 INFO:teuthology.orchestra.run.smithi097.stdout: "description": "osd_op(client.15744.0:70517 10.12 10.2dc54812 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e374)", 2024-03-11T21:46:18.569 INFO:teuthology.orchestra.run.smithi097.stdout: "initiated_at": "2024-03-11T21:41:29.882950+0000", 2024-03-11T21:46:18.570 INFO:teuthology.orchestra.run.smithi097.stdout: "age": 288.66248549699998, 2024-03-11T21:46:18.570 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.97510392800000001, 2024-03-11T21:46:18.570 INFO:teuthology.orchestra.run.smithi097.stdout: "type_data": { 2024-03-11T21:46:18.570 INFO:teuthology.orchestra.run.smithi097.stdout: "flag_point": "reached pg", 2024-03-11T21:46:18.570 INFO:teuthology.orchestra.run.smithi097.stdout: "client_info": { 2024-03-11T21:46:18.570 INFO:teuthology.orchestra.run.smithi097.stdout: "client": "client.15744", 2024-03-11T21:46:18.570 INFO:teuthology.orchestra.run.smithi097.stdout: "client_addr": "v1:172.21.15.92:0/3334000812", 2024-03-11T21:46:18.570 INFO:teuthology.orchestra.run.smithi097.stdout: "tid": 70517 2024-03-11T21:46:18.570 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:46:18.570 INFO:teuthology.orchestra.run.smithi097.stdout: "events": [ 2024-03-11T21:46:18.570 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:46:18.570 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "initiated", 2024-03-11T21:46:18.570 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:41:29.882950+0000", 2024-03-11T21:46:18.570 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-03-11T21:46:18.571 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:46:18.571 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:46:18.571 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "header_read", 2024-03-11T21:46:18.571 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:41:29.882950+0000", 2024-03-11T21:46:18.571 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-03-11T21:46:18.571 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:46:18.571 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:46:18.571 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "throttled", 2024-03-11T21:46:18.571 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:41:29.882956+0000", 2024-03-11T21:46:18.571 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 6.0569999999999998e-06 2024-03-11T21:46:18.571 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:46:18.571 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:46:18.571 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "all_read", 2024-03-11T21:46:18.571 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:41:29.883003+0000", 2024-03-11T21:46:18.572 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 4.7611999999999997e-05 2024-03-11T21:46:18.572 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:46:18.572 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:46:18.572 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "dispatched", 2024-03-11T21:46:18.572 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:41:29.883023+0000", 2024-03-11T21:46:18.572 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.9514e-05 2024-03-11T21:46:18.572 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:46:18.572 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:46:18.572 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "queued_for_pg", 2024-03-11T21:46:18.572 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:41:29.883034+0000", 2024-03-11T21:46:18.572 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.0859000000000001e-05 2024-03-11T21:46:18.572 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:46:18.572 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:46:18.572 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "reached_pg", 2024-03-11T21:46:18.572 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:41:29.883099+0000", 2024-03-11T21:46:18.573 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 6.5210999999999996e-05 2024-03-11T21:46:18.573 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:46:18.573 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:46:18.573 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "waiting for peered", 2024-03-11T21:46:18.573 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:41:29.883119+0000", 2024-03-11T21:46:18.573 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.9930000000000001e-05 2024-03-11T21:46:18.573 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:46:18.573 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:46:18.573 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "reached_pg", 2024-03-11T21:46:18.573 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:41:30.858046+0000", 2024-03-11T21:46:18.573 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.97492744600000003 2024-03-11T21:46:18.573 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:46:18.573 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:46:18.573 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "done", 2024-03-11T21:46:18.573 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:41:30.858054+0000", 2024-03-11T21:46:18.574 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 7.2989999999999999e-06 2024-03-11T21:46:18.574 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-03-11T21:46:18.574 INFO:teuthology.orchestra.run.smithi097.stdout: ] 2024-03-11T21:46:18.574 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-03-11T21:46:18.574 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:46:18.574 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:46:18.574 INFO:teuthology.orchestra.run.smithi097.stdout: "description": "osd_op(client.15744.0:110532 10.16 10:6aef0a33:::benchmark_data_smithi092_161462_object556:head [delete] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e393)", 2024-03-11T21:46:18.574 INFO:teuthology.orchestra.run.smithi097.stdout: "initiated_at": "2024-03-11T21:42:24.508416+0000", 2024-03-11T21:46:18.574 INFO:teuthology.orchestra.run.smithi097.stdout: "age": 234.037019063, 2024-03-11T21:46:18.574 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.011266966, 2024-03-11T21:46:18.574 INFO:teuthology.orchestra.run.smithi097.stdout: "type_data": { 2024-03-11T21:46:18.574 INFO:teuthology.orchestra.run.smithi097.stdout: "flag_point": "reached pg", 2024-03-11T21:46:18.574 INFO:teuthology.orchestra.run.smithi097.stdout: "client_info": { 2024-03-11T21:46:18.574 INFO:teuthology.orchestra.run.smithi097.stdout: "client": "client.15744", 2024-03-11T21:46:18.575 INFO:teuthology.orchestra.run.smithi097.stdout: "client_addr": "v1:172.21.15.92:0/3334000812", 2024-03-11T21:46:18.575 INFO:teuthology.orchestra.run.smithi097.stdout: "tid": 110532 2024-03-11T21:46:18.575 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:46:18.575 INFO:teuthology.orchestra.run.smithi097.stdout: "events": [ 2024-03-11T21:46:18.575 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:46:18.575 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "initiated", 2024-03-11T21:46:18.575 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:42:24.508416+0000", 2024-03-11T21:46:18.575 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-03-11T21:46:18.575 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:46:18.575 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:46:18.575 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "header_read", 2024-03-11T21:46:18.575 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:42:24.508416+0000", 2024-03-11T21:46:18.575 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-03-11T21:46:18.575 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:46:18.575 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:46:18.576 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "throttled", 2024-03-11T21:46:18.576 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:42:24.508418+0000", 2024-03-11T21:46:18.576 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.6500000000000001e-06 2024-03-11T21:46:18.576 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:46:18.576 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:46:18.576 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "all_read", 2024-03-11T21:46:18.576 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:42:24.508423+0000", 2024-03-11T21:46:18.576 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 5.045e-06 2024-03-11T21:46:18.576 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:46:18.576 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:46:18.576 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "dispatched", 2024-03-11T21:46:18.576 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:42:24.508432+0000", 2024-03-11T21:46:18.576 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 8.8100000000000004e-06 2024-03-11T21:46:18.576 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:46:18.576 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:46:18.577 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "queued_for_pg", 2024-03-11T21:46:18.577 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:42:24.508437+0000", 2024-03-11T21:46:18.577 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 5.648e-06 2024-03-11T21:46:18.577 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:46:18.577 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:46:18.577 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "reached_pg", 2024-03-11T21:46:18.577 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:42:24.508484+0000", 2024-03-11T21:46:18.577 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 4.6956999999999998e-05 2024-03-11T21:46:18.577 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:46:18.577 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:46:18.577 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "waiting for peered", 2024-03-11T21:46:18.577 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:42:24.508493+0000", 2024-03-11T21:46:18.577 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 8.7819999999999999e-06 2024-03-11T21:46:18.577 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:46:18.578 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:46:18.578 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "reached_pg", 2024-03-11T21:46:18.578 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:42:25.519624+0000", 2024-03-11T21:46:18.578 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.0111309209999999 2024-03-11T21:46:18.578 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:46:18.578 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:46:18.578 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "done", 2024-03-11T21:46:18.578 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:42:25.519683+0000", 2024-03-11T21:46:18.578 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 5.9153000000000001e-05 2024-03-11T21:46:18.578 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-03-11T21:46:18.578 INFO:teuthology.orchestra.run.smithi097.stdout: ] 2024-03-11T21:46:18.578 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-03-11T21:46:18.578 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:46:18.578 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:46:18.578 INFO:teuthology.orchestra.run.smithi097.stdout: "description": "osd_op(client.15744.0:110564 10.16 10:6dc90f41:::benchmark_data_smithi092_161462_object588:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e393)", 2024-03-11T21:46:18.579 INFO:teuthology.orchestra.run.smithi097.stdout: "initiated_at": "2024-03-11T21:42:24.555481+0000", 2024-03-11T21:46:18.579 INFO:teuthology.orchestra.run.smithi097.stdout: "age": 233.98995408900001, 2024-03-11T21:46:18.579 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.97476978800000003, 2024-03-11T21:46:18.579 INFO:teuthology.orchestra.run.smithi097.stdout: "type_data": { 2024-03-11T21:46:18.579 INFO:teuthology.orchestra.run.smithi097.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-11T21:46:18.579 INFO:teuthology.orchestra.run.smithi097.stdout: "client_info": { 2024-03-11T21:46:18.579 INFO:teuthology.orchestra.run.smithi097.stdout: "client": "client.15744", 2024-03-11T21:46:18.579 INFO:teuthology.orchestra.run.smithi097.stdout: "client_addr": "v1:172.21.15.92:0/3334000812", 2024-03-11T21:46:18.579 INFO:teuthology.orchestra.run.smithi097.stdout: "tid": 110564 2024-03-11T21:46:18.579 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:46:18.579 INFO:teuthology.orchestra.run.smithi097.stdout: "events": [ 2024-03-11T21:46:18.579 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:46:18.579 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "initiated", 2024-03-11T21:46:18.579 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:42:24.555481+0000", 2024-03-11T21:46:18.579 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-03-11T21:46:18.580 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:46:18.580 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:46:18.580 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "header_read", 2024-03-11T21:46:18.580 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:42:24.555481+0000", 2024-03-11T21:46:18.580 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-03-11T21:46:18.580 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:46:18.580 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:46:18.580 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "throttled", 2024-03-11T21:46:18.580 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:42:24.555483+0000", 2024-03-11T21:46:18.580 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 2.4360000000000001e-06 2024-03-11T21:46:18.580 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:46:18.580 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:46:18.580 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "all_read", 2024-03-11T21:46:18.580 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:42:24.555491+0000", 2024-03-11T21:46:18.580 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 7.0720000000000002e-06 2024-03-11T21:46:18.581 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:46:18.581 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:46:18.581 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "dispatched", 2024-03-11T21:46:18.581 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:46:18.582 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:42:24.555504+0000", 2024-03-11T21:46:18.582 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.3489000000000001e-05 2024-03-11T21:46:18.583 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:46:18.583 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:46:18.583 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "queued_for_pg", 2024-03-11T21:46:18.583 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:42:24.555512+0000", 2024-03-11T21:46:18.583 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 7.8949999999999998e-06 2024-03-11T21:46:18.583 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:46:18.583 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:46:18.583 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "reached_pg", 2024-03-11T21:46:18.583 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:42:24.555582+0000", 2024-03-11T21:46:18.583 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 6.9824000000000001e-05 2024-03-11T21:46:18.583 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:46:18.583 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:46:18.583 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "waiting for peered", 2024-03-11T21:46:18.583 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:42:24.555594+0000", 2024-03-11T21:46:18.584 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.2238e-05 2024-03-11T21:46:18.584 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:46:18.584 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:46:18.584 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "reached_pg", 2024-03-11T21:46:18.584 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:42:25.520028+0000", 2024-03-11T21:46:18.584 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.964434232 2024-03-11T21:46:18.584 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:46:18.584 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:46:18.584 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "started", 2024-03-11T21:46:18.584 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:42:25.520131+0000", 2024-03-11T21:46:18.584 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.00010289399999999999 2024-03-11T21:46:18.584 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:46:18.584 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:46:18.584 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "waiting for subops from 1,5", 2024-03-11T21:46:18.584 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:42:25.520219+0000", 2024-03-11T21:46:18.585 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 8.8162000000000001e-05 2024-03-11T21:46:18.585 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:46:18.585 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:46:18.585 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "sub_op_commit_rec", 2024-03-11T21:46:18.585 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:42:25.525201+0000", 2024-03-11T21:46:18.585 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.0049820029999999996 2024-03-11T21:46:18.585 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:46:18.585 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:46:18.585 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "op_commit", 2024-03-11T21:46:18.585 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:42:25.525211+0000", 2024-03-11T21:46:18.585 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 9.2779999999999999e-06 2024-03-11T21:46:18.585 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:46:18.585 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:46:18.585 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "sub_op_commit_rec", 2024-03-11T21:46:18.586 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:42:25.530153+0000", 2024-03-11T21:46:18.586 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.0049424559999999996 2024-03-11T21:46:18.586 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:46:18.586 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:46:18.586 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "commit_sent", 2024-03-11T21:46:18.586 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:42:25.530206+0000", 2024-03-11T21:46:18.586 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 5.3285000000000002e-05 2024-03-11T21:46:18.586 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:46:18.586 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:46:18.586 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "done", 2024-03-11T21:46:18.586 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:42:25.530251+0000", 2024-03-11T21:46:18.586 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 4.4524000000000003e-05 2024-03-11T21:46:18.586 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-03-11T21:46:18.586 INFO:teuthology.orchestra.run.smithi097.stdout: ] 2024-03-11T21:46:18.586 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-03-11T21:46:18.587 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:46:18.587 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:46:18.587 INFO:teuthology.orchestra.run.smithi097.stdout: "description": "osd_op(client.15744.0:214899 10.17 10:ed6d6cd1:::benchmark_data_smithi092_161462_object104923:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e431)", 2024-03-11T21:46:18.587 INFO:teuthology.orchestra.run.smithi097.stdout: "initiated_at": "2024-03-11T21:44:20.436254+0000", 2024-03-11T21:46:18.587 INFO:teuthology.orchestra.run.smithi097.stdout: "age": 118.109180966, 2024-03-11T21:46:18.587 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 2.175189123, 2024-03-11T21:46:18.587 INFO:teuthology.orchestra.run.smithi097.stdout: "type_data": { 2024-03-11T21:46:18.587 INFO:teuthology.orchestra.run.smithi097.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-11T21:46:18.587 INFO:teuthology.orchestra.run.smithi097.stdout: "client_info": { 2024-03-11T21:46:18.587 INFO:teuthology.orchestra.run.smithi097.stdout: "client": "client.15744", 2024-03-11T21:46:18.587 INFO:teuthology.orchestra.run.smithi097.stdout: "client_addr": "v1:172.21.15.92:0/3334000812", 2024-03-11T21:46:18.587 INFO:teuthology.orchestra.run.smithi097.stdout: "tid": 214899 2024-03-11T21:46:18.587 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:46:18.587 INFO:teuthology.orchestra.run.smithi097.stdout: "events": [ 2024-03-11T21:46:18.588 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:46:18.588 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "initiated", 2024-03-11T21:46:18.588 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:44:20.436254+0000", 2024-03-11T21:46:18.588 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-03-11T21:46:18.588 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:46:18.588 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:46:18.588 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "header_read", 2024-03-11T21:46:18.588 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:44:20.436254+0000", 2024-03-11T21:46:18.588 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-03-11T21:46:18.588 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:46:18.588 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:46:18.588 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "throttled", 2024-03-11T21:46:18.588 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:44:20.436256+0000", 2024-03-11T21:46:18.588 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.8199999999999999e-06 2024-03-11T21:46:18.588 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:46:18.589 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:46:18.589 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "all_read", 2024-03-11T21:46:18.589 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:44:20.436261+0000", 2024-03-11T21:46:18.589 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 5.0699999999999997e-06 2024-03-11T21:46:18.589 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:46:18.589 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:46:18.589 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "dispatched", 2024-03-11T21:46:18.589 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:44:20.436271+0000", 2024-03-11T21:46:18.589 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.0003000000000001e-05 2024-03-11T21:46:18.589 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:46:18.589 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:46:18.589 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "queued_for_pg", 2024-03-11T21:46:18.589 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:44:20.436279+0000", 2024-03-11T21:46:18.589 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 7.7279999999999995e-06 2024-03-11T21:46:18.589 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:46:18.590 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:46:18.590 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "reached_pg", 2024-03-11T21:46:18.590 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:44:20.438162+0000", 2024-03-11T21:46:18.590 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.0018829999999999999 2024-03-11T21:46:18.590 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:46:18.590 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:46:18.590 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "waiting for peered", 2024-03-11T21:46:18.590 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:44:20.438174+0000", 2024-03-11T21:46:18.590 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.2132e-05 2024-03-11T21:46:18.590 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:46:18.590 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:46:18.590 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "reached_pg", 2024-03-11T21:46:18.590 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:44:21.432513+0000", 2024-03-11T21:46:18.590 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.99433888599999998 2024-03-11T21:46:18.591 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:46:18.591 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:46:18.591 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "waiting for peered", 2024-03-11T21:46:18.591 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:44:21.432523+0000", 2024-03-11T21:46:18.591 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 9.8949999999999993e-06 2024-03-11T21:46:18.591 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:46:18.591 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:46:18.591 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "reached_pg", 2024-03-11T21:46:18.591 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:44:22.595683+0000", 2024-03-11T21:46:18.591 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.163159931 2024-03-11T21:46:18.591 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:46:18.591 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:46:18.591 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "started", 2024-03-11T21:46:18.591 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:44:22.595829+0000", 2024-03-11T21:46:18.591 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.00014639899999999999 2024-03-11T21:46:18.592 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:46:18.592 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:46:18.592 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "waiting for subops from 2,3,5", 2024-03-11T21:46:18.592 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:44:22.596265+0000", 2024-03-11T21:46:18.592 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.00043560300000000001 2024-03-11T21:46:18.592 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:46:18.592 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:46:18.592 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "sub_op_commit_rec", 2024-03-11T21:46:18.592 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:44:22.605366+0000", 2024-03-11T21:46:18.592 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.0091015169999999999 2024-03-11T21:46:18.592 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:46:18.592 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:46:18.592 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "op_commit", 2024-03-11T21:46:18.592 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:44:22.605558+0000", 2024-03-11T21:46:18.593 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.00019154900000000001 2024-03-11T21:46:18.593 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:46:18.593 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:46:18.593 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "sub_op_commit_rec", 2024-03-11T21:46:18.593 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:44:22.611191+0000", 2024-03-11T21:46:18.593 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.0056330740000000001 2024-03-11T21:46:18.593 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:46:18.593 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:46:18.593 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "sub_op_commit_rec", 2024-03-11T21:46:18.593 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:44:22.611251+0000", 2024-03-11T21:46:18.593 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 5.9935000000000001e-05 2024-03-11T21:46:18.593 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:46:18.593 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:46:18.593 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "commit_sent", 2024-03-11T21:46:18.593 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:44:22.611309+0000", 2024-03-11T21:46:18.594 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 5.8301000000000003e-05 2024-03-11T21:46:18.594 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:46:18.594 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:46:18.594 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "done", 2024-03-11T21:46:18.594 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:44:22.611443+0000", 2024-03-11T21:46:18.594 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.00013428 2024-03-11T21:46:18.594 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-03-11T21:46:18.594 INFO:teuthology.orchestra.run.smithi097.stdout: ] 2024-03-11T21:46:18.594 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-03-11T21:46:18.594 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:46:18.594 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:46:18.594 INFO:teuthology.orchestra.run.smithi097.stdout: "description": "osd_op(client.15744.0:214911 10.17 10:ee14355d:::benchmark_data_smithi092_161462_object104935:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e432)", 2024-03-11T21:46:18.594 INFO:teuthology.orchestra.run.smithi097.stdout: "initiated_at": "2024-03-11T21:44:20.472141+0000", 2024-03-11T21:46:18.594 INFO:teuthology.orchestra.run.smithi097.stdout: "age": 118.073293682, 2024-03-11T21:46:18.595 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 2.1396579419999999, 2024-03-11T21:46:18.595 INFO:teuthology.orchestra.run.smithi097.stdout: "type_data": { 2024-03-11T21:46:18.595 INFO:teuthology.orchestra.run.smithi097.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-11T21:46:18.595 INFO:teuthology.orchestra.run.smithi097.stdout: "client_info": { 2024-03-11T21:46:18.595 INFO:teuthology.orchestra.run.smithi097.stdout: "client": "client.15744", 2024-03-11T21:46:18.595 INFO:teuthology.orchestra.run.smithi097.stdout: "client_addr": "v1:172.21.15.92:0/3334000812", 2024-03-11T21:46:18.595 INFO:teuthology.orchestra.run.smithi097.stdout: "tid": 214911 2024-03-11T21:46:18.595 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:46:18.595 INFO:teuthology.orchestra.run.smithi097.stdout: "events": [ 2024-03-11T21:46:18.595 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:46:18.595 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "initiated", 2024-03-11T21:46:18.595 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:44:20.472141+0000", 2024-03-11T21:46:18.595 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-03-11T21:46:18.595 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:46:18.596 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:46:18.596 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "header_read", 2024-03-11T21:46:18.596 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:44:20.472141+0000", 2024-03-11T21:46:18.596 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-03-11T21:46:18.596 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:46:18.596 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:46:18.596 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "throttled", 2024-03-11T21:46:18.596 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:44:20.472143+0000", 2024-03-11T21:46:18.596 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.8220000000000001e-06 2024-03-11T21:46:18.596 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:46:18.596 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:46:18.596 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "all_read", 2024-03-11T21:46:18.596 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:44:20.472147+0000", 2024-03-11T21:46:18.596 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 4.1849999999999997e-06 2024-03-11T21:46:18.597 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:46:18.597 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:46:18.597 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "dispatched", 2024-03-11T21:46:18.597 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:44:20.472159+0000", 2024-03-11T21:46:18.597 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.148e-05 2024-03-11T21:46:18.597 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:46:18.597 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:46:18.597 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "queued_for_pg", 2024-03-11T21:46:18.597 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:44:20.472165+0000", 2024-03-11T21:46:18.597 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 6.1489999999999999e-06 2024-03-11T21:46:18.597 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:46:18.597 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:46:18.597 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "reached_pg", 2024-03-11T21:46:18.597 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:44:20.472233+0000", 2024-03-11T21:46:18.597 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 6.8001000000000002e-05 2024-03-11T21:46:18.598 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:46:18.598 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:46:18.598 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "waiting for peered", 2024-03-11T21:46:18.598 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:44:20.472246+0000", 2024-03-11T21:46:18.598 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.2438e-05 2024-03-11T21:46:18.598 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:46:18.598 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:46:18.598 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "reached_pg", 2024-03-11T21:46:18.598 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:44:21.432557+0000", 2024-03-11T21:46:18.598 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.96031168600000005 2024-03-11T21:46:18.598 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:46:18.598 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:46:18.598 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "waiting for peered", 2024-03-11T21:46:18.598 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:44:21.432568+0000", 2024-03-11T21:46:18.599 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.0380999999999999e-05 2024-03-11T21:46:18.599 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:46:18.599 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:46:18.599 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "reached_pg", 2024-03-11T21:46:18.599 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:44:22.602674+0000", 2024-03-11T21:46:18.599 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.170106815 2024-03-11T21:46:18.599 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:46:18.599 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:46:18.599 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "started", 2024-03-11T21:46:18.599 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:44:22.602829+0000", 2024-03-11T21:46:18.599 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.00015442200000000001 2024-03-11T21:46:18.599 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:46:18.599 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:46:18.599 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "waiting for subops from 2,3,5", 2024-03-11T21:46:18.600 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:44:22.602982+0000", 2024-03-11T21:46:18.600 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.000153313 2024-03-11T21:46:18.600 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:46:18.600 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:46:18.600 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "sub_op_commit_rec", 2024-03-11T21:46:18.600 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:44:22.605591+0000", 2024-03-11T21:46:18.600 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.0026091450000000002 2024-03-11T21:46:18.600 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:46:18.600 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:46:18.600 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "op_commit", 2024-03-11T21:46:18.600 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:44:22.605603+0000", 2024-03-11T21:46:18.600 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.173e-05 2024-03-11T21:46:18.600 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:46:18.600 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:46:18.601 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "sub_op_commit_rec", 2024-03-11T21:46:18.601 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:44:22.611570+0000", 2024-03-11T21:46:18.601 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.0059670499999999998 2024-03-11T21:46:18.601 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:46:18.601 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:46:18.601 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "sub_op_commit_rec", 2024-03-11T21:46:18.601 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:44:22.611624+0000", 2024-03-11T21:46:18.601 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 5.3640999999999997e-05 2024-03-11T21:46:18.601 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:46:18.601 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:46:18.601 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "commit_sent", 2024-03-11T21:46:18.601 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:44:22.611678+0000", 2024-03-11T21:46:18.601 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 5.4713000000000003e-05 2024-03-11T21:46:18.601 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:46:18.601 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:46:18.602 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "done", 2024-03-11T21:46:18.602 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:44:22.611799+0000", 2024-03-11T21:46:18.602 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.000120971 2024-03-11T21:46:18.602 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-03-11T21:46:18.602 INFO:teuthology.orchestra.run.smithi097.stdout: ] 2024-03-11T21:46:18.602 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-03-11T21:46:18.602 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:46:18.602 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:46:18.602 INFO:teuthology.orchestra.run.smithi097.stdout: "description": "osd_op(client.15744.0:214914 10.17 10:e9987e1e:::benchmark_data_smithi092_161462_object104938:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e432)", 2024-03-11T21:46:18.602 INFO:teuthology.orchestra.run.smithi097.stdout: "initiated_at": "2024-03-11T21:44:20.480821+0000", 2024-03-11T21:46:18.602 INFO:teuthology.orchestra.run.smithi097.stdout: "age": 118.064614345, 2024-03-11T21:46:18.602 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 2.1312782029999999, 2024-03-11T21:46:18.602 INFO:teuthology.orchestra.run.smithi097.stdout: "type_data": { 2024-03-11T21:46:18.602 INFO:teuthology.orchestra.run.smithi097.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-11T21:46:18.603 INFO:teuthology.orchestra.run.smithi097.stdout: "client_info": { 2024-03-11T21:46:18.603 INFO:teuthology.orchestra.run.smithi097.stdout: "client": "client.15744", 2024-03-11T21:46:18.603 INFO:teuthology.orchestra.run.smithi097.stdout: "client_addr": "v1:172.21.15.92:0/3334000812", 2024-03-11T21:46:18.603 INFO:teuthology.orchestra.run.smithi097.stdout: "tid": 214914 2024-03-11T21:46:18.603 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:46:18.603 INFO:teuthology.orchestra.run.smithi097.stdout: "events": [ 2024-03-11T21:46:18.603 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:46:18.603 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "initiated", 2024-03-11T21:46:18.603 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:44:20.480821+0000", 2024-03-11T21:46:18.603 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-03-11T21:46:18.603 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:46:18.603 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:46:18.603 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "header_read", 2024-03-11T21:46:18.603 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:44:20.480821+0000", 2024-03-11T21:46:18.604 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-03-11T21:46:18.604 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:46:18.604 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:46:18.604 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "throttled", 2024-03-11T21:46:18.604 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:44:20.480823+0000", 2024-03-11T21:46:18.604 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.871e-06 2024-03-11T21:46:18.604 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:46:18.604 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:46:18.604 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "all_read", 2024-03-11T21:46:18.604 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:44:20.480828+0000", 2024-03-11T21:46:18.604 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 5.3580000000000002e-06 2024-03-11T21:46:18.604 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:46:18.604 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:46:18.604 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "dispatched", 2024-03-11T21:46:18.604 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:44:20.480840+0000", 2024-03-11T21:46:18.605 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.2170999999999999e-05 2024-03-11T21:46:18.605 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:46:18.605 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:46:18.605 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "queued_for_pg", 2024-03-11T21:46:18.605 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:44:20.480849+0000", 2024-03-11T21:46:18.605 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 8.4800000000000001e-06 2024-03-11T21:46:18.605 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:46:18.605 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:46:18.605 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "reached_pg", 2024-03-11T21:46:18.605 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:44:20.480889+0000", 2024-03-11T21:46:18.605 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 4.0513999999999999e-05 2024-03-11T21:46:18.605 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:46:18.605 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:46:18.605 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "waiting for peered", 2024-03-11T21:46:18.605 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:44:20.480897+0000", 2024-03-11T21:46:18.606 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 7.8229999999999995e-06 2024-03-11T21:46:18.606 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:46:18.606 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:46:18.606 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "reached_pg", 2024-03-11T21:46:18.606 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:44:21.432594+0000", 2024-03-11T21:46:18.606 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.95169700800000001 2024-03-11T21:46:18.606 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:46:18.606 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:46:18.606 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "waiting for peered", 2024-03-11T21:46:18.606 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:44:21.432603+0000", 2024-03-11T21:46:18.606 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 8.8070000000000006e-06 2024-03-11T21:46:18.606 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:46:18.606 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:46:18.606 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "reached_pg", 2024-03-11T21:46:18.607 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:44:22.605639+0000", 2024-03-11T21:46:18.607 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.1730364369999999 2024-03-11T21:46:18.607 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:46:18.607 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:46:18.607 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "started", 2024-03-11T21:46:18.607 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:44:22.605767+0000", 2024-03-11T21:46:18.607 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.000127391 2024-03-11T21:46:18.607 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:46:18.607 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:46:18.607 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "waiting for subops from 2,3,5", 2024-03-11T21:46:18.607 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:44:22.605931+0000", 2024-03-11T21:46:18.607 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.000164167 2024-03-11T21:46:18.607 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:46:18.607 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:46:18.607 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "op_commit", 2024-03-11T21:46:18.608 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:44:22.611458+0000", 2024-03-11T21:46:18.608 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.0055274230000000001 2024-03-11T21:46:18.608 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:46:18.608 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:46:18.608 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "sub_op_commit_rec", 2024-03-11T21:46:18.608 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:44:22.611517+0000", 2024-03-11T21:46:18.608 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 5.8532000000000001e-05 2024-03-11T21:46:18.608 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:46:18.608 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:46:18.608 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "sub_op_commit_rec", 2024-03-11T21:46:18.608 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:44:22.611879+0000", 2024-03-11T21:46:18.608 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.00036212199999999999 2024-03-11T21:46:18.608 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:46:18.608 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:46:18.608 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "sub_op_commit_rec", 2024-03-11T21:46:18.609 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:44:22.611927+0000", 2024-03-11T21:46:18.609 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 4.8219e-05 2024-03-11T21:46:18.609 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:46:18.609 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:46:18.609 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "commit_sent", 2024-03-11T21:46:18.609 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:44:22.611981+0000", 2024-03-11T21:46:18.609 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 5.3461000000000002e-05 2024-03-11T21:46:18.609 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:46:18.609 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:46:18.609 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "done", 2024-03-11T21:46:18.609 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:44:22.612099+0000", 2024-03-11T21:46:18.609 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.00011841900000000001 2024-03-11T21:46:18.609 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-03-11T21:46:18.609 INFO:teuthology.orchestra.run.smithi097.stdout: ] 2024-03-11T21:46:18.610 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-03-11T21:46:18.610 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:46:18.610 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:46:18.610 INFO:teuthology.orchestra.run.smithi097.stdout: "description": "osd_op(client.15744.0:214915 10.f 10:f1996a13:::benchmark_data_smithi092_161462_object104939:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e432)", 2024-03-11T21:46:18.610 INFO:teuthology.orchestra.run.smithi097.stdout: "initiated_at": "2024-03-11T21:44:20.480979+0000", 2024-03-11T21:46:18.610 INFO:teuthology.orchestra.run.smithi097.stdout: "age": 118.064455942, 2024-03-11T21:46:18.610 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 2.1245642980000001, 2024-03-11T21:46:18.610 INFO:teuthology.orchestra.run.smithi097.stdout: "type_data": { 2024-03-11T21:46:18.610 INFO:teuthology.orchestra.run.smithi097.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-11T21:46:18.610 INFO:teuthology.orchestra.run.smithi097.stdout: "client_info": { 2024-03-11T21:46:18.610 INFO:teuthology.orchestra.run.smithi097.stdout: "client": "client.15744", 2024-03-11T21:46:18.610 INFO:teuthology.orchestra.run.smithi097.stdout: "client_addr": "v1:172.21.15.92:0/3334000812", 2024-03-11T21:46:18.610 INFO:teuthology.orchestra.run.smithi097.stdout: "tid": 214915 2024-03-11T21:46:18.610 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:46:18.611 INFO:teuthology.orchestra.run.smithi097.stdout: "events": [ 2024-03-11T21:46:18.611 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:46:18.611 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "initiated", 2024-03-11T21:46:18.611 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:44:20.480979+0000", 2024-03-11T21:46:18.611 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-03-11T21:46:18.611 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:46:18.611 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:46:18.611 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "header_read", 2024-03-11T21:46:18.611 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:44:20.480979+0000", 2024-03-11T21:46:18.611 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-03-11T21:46:18.611 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:46:18.611 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:46:18.611 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "throttled", 2024-03-11T21:46:18.611 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:44:20.480980+0000", 2024-03-11T21:46:18.611 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.1349999999999999e-06 2024-03-11T21:46:18.612 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:46:18.612 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:46:18.612 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "all_read", 2024-03-11T21:46:18.612 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:44:20.480985+0000", 2024-03-11T21:46:18.612 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 4.2050000000000004e-06 2024-03-11T21:46:18.612 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:46:18.612 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:46:18.612 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "dispatched", 2024-03-11T21:46:18.612 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:44:20.480995+0000", 2024-03-11T21:46:18.612 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.0129000000000001e-05 2024-03-11T21:46:18.612 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:46:18.612 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:46:18.612 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "queued_for_pg", 2024-03-11T21:46:18.613 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:44:20.481003+0000", 2024-03-11T21:46:18.613 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 8.3809999999999999e-06 2024-03-11T21:46:18.613 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:46:18.613 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:46:18.613 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "reached_pg", 2024-03-11T21:46:18.613 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:44:20.481042+0000", 2024-03-11T21:46:18.613 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 3.8992000000000002e-05 2024-03-11T21:46:18.613 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:46:18.613 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:46:18.613 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "waiting for peered", 2024-03-11T21:46:18.613 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:44:20.481050+0000", 2024-03-11T21:46:18.613 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 7.4800000000000004e-06 2024-03-11T21:46:18.613 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:46:18.613 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:46:18.614 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "reached_pg", 2024-03-11T21:46:18.614 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:44:21.432468+0000", 2024-03-11T21:46:18.614 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.95141802399999997 2024-03-11T21:46:18.614 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:46:18.614 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:46:18.614 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "waiting for peered", 2024-03-11T21:46:18.614 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:44:21.432481+0000", 2024-03-11T21:46:18.614 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.393e-05 2024-03-11T21:46:18.614 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:46:18.614 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:46:18.614 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "reached_pg", 2024-03-11T21:46:18.614 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:44:22.582869+0000", 2024-03-11T21:46:18.614 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.1503876740000001 2024-03-11T21:46:18.614 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:46:18.615 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:46:18.615 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "started", 2024-03-11T21:46:18.615 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:44:22.583001+0000", 2024-03-11T21:46:18.615 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.00013150099999999999 2024-03-11T21:46:18.615 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:46:18.615 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:46:18.615 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "waiting for subops from 2,3,4", 2024-03-11T21:46:18.615 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:44:22.583407+0000", 2024-03-11T21:46:18.615 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.00040668799999999999 2024-03-11T21:46:18.615 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:46:18.615 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:46:18.615 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "sub_op_commit_rec", 2024-03-11T21:46:18.615 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:44:22.595606+0000", 2024-03-11T21:46:18.615 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.0121986 2024-03-11T21:46:18.615 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:46:18.616 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:46:18.616 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "op_commit", 2024-03-11T21:46:18.616 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:44:22.595663+0000", 2024-03-11T21:46:18.616 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 5.7444000000000001e-05 2024-03-11T21:46:18.616 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:46:18.616 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:46:18.616 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "sub_op_commit_rec", 2024-03-11T21:46:18.616 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:44:22.602662+0000", 2024-03-11T21:46:18.616 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.0069990119999999998 2024-03-11T21:46:18.616 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:46:18.616 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:46:18.616 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "sub_op_commit_rec", 2024-03-11T21:46:18.616 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:44:22.605365+0000", 2024-03-11T21:46:18.616 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.0027022470000000001 2024-03-11T21:46:18.617 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:46:18.617 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:46:18.617 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "commit_sent", 2024-03-11T21:46:18.617 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:44:22.605419+0000", 2024-03-11T21:46:18.617 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 5.452e-05 2024-03-11T21:46:18.617 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:46:18.617 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:46:18.617 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "done", 2024-03-11T21:46:18.617 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:44:22.605544+0000", 2024-03-11T21:46:18.617 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.00012433600000000001 2024-03-11T21:46:18.617 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-03-11T21:46:18.617 INFO:teuthology.orchestra.run.smithi097.stdout: ] 2024-03-11T21:46:18.617 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-03-11T21:46:18.617 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:46:18.618 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:46:18.618 INFO:teuthology.orchestra.run.smithi097.stdout: "description": "osd_op(client.15744.0:214916 10.17 10:ec630630:::benchmark_data_smithi092_161462_object104940:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e432)", 2024-03-11T21:46:18.618 INFO:teuthology.orchestra.run.smithi097.stdout: "initiated_at": "2024-03-11T21:44:20.481252+0000", 2024-03-11T21:46:18.618 INFO:teuthology.orchestra.run.smithi097.stdout: "age": 118.064183515, 2024-03-11T21:46:18.618 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 2.1345388320000001, 2024-03-11T21:46:18.618 INFO:teuthology.orchestra.run.smithi097.stdout: "type_data": { 2024-03-11T21:46:18.618 INFO:teuthology.orchestra.run.smithi097.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-11T21:46:18.618 INFO:teuthology.orchestra.run.smithi097.stdout: "client_info": { 2024-03-11T21:46:18.618 INFO:teuthology.orchestra.run.smithi097.stdout: "client": "client.15744", 2024-03-11T21:46:18.618 INFO:teuthology.orchestra.run.smithi097.stdout: "client_addr": "v1:172.21.15.92:0/3334000812", 2024-03-11T21:46:18.618 INFO:teuthology.orchestra.run.smithi097.stdout: "tid": 214916 2024-03-11T21:46:18.618 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:46:18.618 INFO:teuthology.orchestra.run.smithi097.stdout: "events": [ 2024-03-11T21:46:18.619 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:46:18.619 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "initiated", 2024-03-11T21:46:18.619 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:44:20.481252+0000", 2024-03-11T21:46:18.619 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-03-11T21:46:18.619 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:46:18.619 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:46:18.619 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "header_read", 2024-03-11T21:46:18.619 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:44:20.481252+0000", 2024-03-11T21:46:18.619 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-03-11T21:46:18.619 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:46:18.619 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:46:18.619 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "throttled", 2024-03-11T21:46:18.619 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:44:20.481253+0000", 2024-03-11T21:46:18.619 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.1000000000000001e-06 2024-03-11T21:46:18.619 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:46:18.620 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:46:18.620 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "all_read", 2024-03-11T21:46:18.620 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:44:20.481257+0000", 2024-03-11T21:46:18.620 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 4.2590000000000002e-06 2024-03-11T21:46:18.620 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:46:18.620 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:46:18.620 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "dispatched", 2024-03-11T21:46:18.620 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:44:20.481267+0000", 2024-03-11T21:46:18.620 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.0465000000000001e-05 2024-03-11T21:46:18.620 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:46:18.620 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:46:18.620 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "queued_for_pg", 2024-03-11T21:46:18.620 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:44:20.481275+0000", 2024-03-11T21:46:18.620 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 7.289e-06 2024-03-11T21:46:18.621 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:46:18.621 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:46:18.621 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "reached_pg", 2024-03-11T21:46:18.621 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:44:20.481310+0000", 2024-03-11T21:46:18.621 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 3.4955e-05 2024-03-11T21:46:18.621 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:46:18.621 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:46:18.621 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "waiting for peered", 2024-03-11T21:46:18.621 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:44:20.481316+0000", 2024-03-11T21:46:18.621 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 6.0430000000000004e-06 2024-03-11T21:46:18.621 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:46:18.621 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:46:18.621 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "reached_pg", 2024-03-11T21:46:18.621 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:44:21.432627+0000", 2024-03-11T21:46:18.621 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.95131141299999999 2024-03-11T21:46:18.622 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:46:18.622 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:46:18.622 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "waiting for peered", 2024-03-11T21:46:18.622 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:44:21.432636+0000", 2024-03-11T21:46:18.622 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 8.8170000000000005e-06 2024-03-11T21:46:18.624 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:46:18.624 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:46:18.624 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "reached_pg", 2024-03-11T21:46:18.624 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:44:22.608545+0000", 2024-03-11T21:46:18.624 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.175909144 2024-03-11T21:46:18.624 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:46:18.624 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:46:18.624 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "started", 2024-03-11T21:46:18.625 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:44:22.608666+0000", 2024-03-11T21:46:18.625 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.000121222 2024-03-11T21:46:18.625 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:46:18.625 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:46:18.625 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "waiting for subops from 2,3,5", 2024-03-11T21:46:18.625 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:44:22.608804+0000", 2024-03-11T21:46:18.625 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.000137365 2024-03-11T21:46:18.625 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:46:18.625 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:46:18.625 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "op_commit", 2024-03-11T21:46:18.625 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:44:22.611587+0000", 2024-03-11T21:46:18.625 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.002783593 2024-03-11T21:46:18.625 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:46:18.625 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:46:18.625 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "sub_op_commit_rec", 2024-03-11T21:46:18.626 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:44:22.611841+0000", 2024-03-11T21:46:18.626 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.00025328699999999999 2024-03-11T21:46:18.626 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:46:18.626 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:46:18.626 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "sub_op_commit_rec", 2024-03-11T21:46:18.626 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:44:22.614866+0000", 2024-03-11T21:46:18.626 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.003025823 2024-03-11T21:46:18.626 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:46:18.626 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:46:18.626 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "sub_op_commit_rec", 2024-03-11T21:46:18.626 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:44:22.614911+0000", 2024-03-11T21:46:18.626 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 4.4569e-05 2024-03-11T21:46:18.626 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:46:18.626 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:46:18.627 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "commit_sent", 2024-03-11T21:46:18.627 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:44:22.614973+0000", 2024-03-11T21:46:18.627 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 6.2244999999999998e-05 2024-03-11T21:46:18.627 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:46:18.627 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:46:18.627 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "done", 2024-03-11T21:46:18.627 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:44:22.615790+0000", 2024-03-11T21:46:18.627 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.00081724300000000005 2024-03-11T21:46:18.627 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-03-11T21:46:18.627 INFO:teuthology.orchestra.run.smithi097.stdout: ] 2024-03-11T21:46:18.627 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-03-11T21:46:18.627 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:46:18.627 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:46:18.628 INFO:teuthology.orchestra.run.smithi097.stdout: "description": "osd_op(client.15744.0:217010 10.17 10:ed2db962:::benchmark_data_smithi092_161462_object107034:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e434)", 2024-03-11T21:46:18.628 INFO:teuthology.orchestra.run.smithi097.stdout: "initiated_at": "2024-03-11T21:44:24.424632+0000", 2024-03-11T21:46:18.628 INFO:teuthology.orchestra.run.smithi097.stdout: "age": 114.120802916, 2024-03-11T21:46:18.628 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.0482623639999999, 2024-03-11T21:46:18.628 INFO:teuthology.orchestra.run.smithi097.stdout: "type_data": { 2024-03-11T21:46:18.628 INFO:teuthology.orchestra.run.smithi097.stdout: "flag_point": "reached pg", 2024-03-11T21:46:18.628 INFO:teuthology.orchestra.run.smithi097.stdout: "client_info": { 2024-03-11T21:46:18.628 INFO:teuthology.orchestra.run.smithi097.stdout: "client": "client.15744", 2024-03-11T21:46:18.628 INFO:teuthology.orchestra.run.smithi097.stdout: "client_addr": "v1:172.21.15.92:0/3334000812", 2024-03-11T21:46:18.628 INFO:teuthology.orchestra.run.smithi097.stdout: "tid": 217010 2024-03-11T21:46:18.628 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:46:18.628 INFO:teuthology.orchestra.run.smithi097.stdout: "events": [ 2024-03-11T21:46:18.628 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:46:18.628 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "initiated", 2024-03-11T21:46:18.628 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:44:24.424632+0000", 2024-03-11T21:46:18.629 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-03-11T21:46:18.629 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:46:18.629 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:46:18.629 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "header_read", 2024-03-11T21:46:18.629 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:44:24.424632+0000", 2024-03-11T21:46:18.629 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-03-11T21:46:18.629 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:46:18.629 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:46:18.629 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "throttled", 2024-03-11T21:46:18.629 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:44:24.424634+0000", 2024-03-11T21:46:18.629 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.66e-06 2024-03-11T21:46:18.629 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:46:18.629 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:46:18.629 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "all_read", 2024-03-11T21:46:18.630 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:44:24.424639+0000", 2024-03-11T21:46:18.630 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 5.0889999999999999e-06 2024-03-11T21:46:18.630 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:46:18.630 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:46:18.630 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "dispatched", 2024-03-11T21:46:18.630 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:44:24.424651+0000", 2024-03-11T21:46:18.630 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.1799e-05 2024-03-11T21:46:18.630 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:46:18.630 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:46:18.630 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "queued_for_pg", 2024-03-11T21:46:18.630 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:44:24.424658+0000", 2024-03-11T21:46:18.630 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 7.6359999999999994e-06 2024-03-11T21:46:18.630 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:46:18.630 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:46:18.630 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "reached_pg", 2024-03-11T21:46:18.631 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:44:24.424735+0000", 2024-03-11T21:46:18.631 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 7.7013000000000003e-05 2024-03-11T21:46:18.631 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:46:18.631 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:46:18.631 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "started", 2024-03-11T21:46:18.631 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:44:24.424803+0000", 2024-03-11T21:46:18.631 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 6.7836999999999994e-05 2024-03-11T21:46:18.631 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:46:18.631 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:46:18.631 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "waiting for subops from 2,3,5", 2024-03-11T21:46:18.631 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:44:24.424906+0000", 2024-03-11T21:46:18.631 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.00010268999999999999 2024-03-11T21:46:18.631 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:46:18.631 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:46:18.631 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "sub_op_commit_rec", 2024-03-11T21:46:18.632 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:44:24.432605+0000", 2024-03-11T21:46:18.632 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.0076993189999999996 2024-03-11T21:46:18.632 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:46:18.632 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:46:18.632 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "op_commit", 2024-03-11T21:46:18.632 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:44:24.432879+0000", 2024-03-11T21:46:18.632 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.00027418899999999999 2024-03-11T21:46:18.632 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:46:18.632 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:46:18.632 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "sub_op_commit_rec", 2024-03-11T21:46:18.632 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:44:24.441374+0000", 2024-03-11T21:46:18.632 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.0084943150000000005 2024-03-11T21:46:18.632 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:46:18.632 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:46:18.633 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "reached_pg", 2024-03-11T21:46:18.633 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:44:24.452526+0000", 2024-03-11T21:46:18.633 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.011152485 2024-03-11T21:46:18.633 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:46:18.633 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:46:18.633 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "waiting for peered", 2024-03-11T21:46:18.633 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:44:24.452538+0000", 2024-03-11T21:46:18.633 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.1905999999999999e-05 2024-03-11T21:46:18.633 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:46:18.633 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:46:18.633 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "reached_pg", 2024-03-11T21:46:18.633 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:44:25.472810+0000", 2024-03-11T21:46:18.633 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.0202715959999999 2024-03-11T21:46:18.633 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:46:18.633 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:46:18.634 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "done", 2024-03-11T21:46:18.634 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:44:25.472895+0000", 2024-03-11T21:46:18.634 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 8.4829999999999999e-05 2024-03-11T21:46:18.634 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-03-11T21:46:18.634 INFO:teuthology.orchestra.run.smithi097.stdout: ] 2024-03-11T21:46:18.634 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-03-11T21:46:18.634 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:46:18.634 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:46:18.634 INFO:teuthology.orchestra.run.smithi097.stdout: "description": "osd_op(client.15744.0:217014 10.17 10:efb597aa:::benchmark_data_smithi092_161462_object107038:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e434)", 2024-03-11T21:46:18.634 INFO:teuthology.orchestra.run.smithi097.stdout: "initiated_at": "2024-03-11T21:44:24.426189+0000", 2024-03-11T21:46:18.634 INFO:teuthology.orchestra.run.smithi097.stdout: "age": 114.119245892, 2024-03-11T21:46:18.634 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.0472537319999999, 2024-03-11T21:46:18.634 INFO:teuthology.orchestra.run.smithi097.stdout: "type_data": { 2024-03-11T21:46:18.634 INFO:teuthology.orchestra.run.smithi097.stdout: "flag_point": "reached pg", 2024-03-11T21:46:18.635 INFO:teuthology.orchestra.run.smithi097.stdout: "client_info": { 2024-03-11T21:46:18.635 INFO:teuthology.orchestra.run.smithi097.stdout: "client": "client.15744", 2024-03-11T21:46:18.635 INFO:teuthology.orchestra.run.smithi097.stdout: "client_addr": "v1:172.21.15.92:0/3334000812", 2024-03-11T21:46:18.635 INFO:teuthology.orchestra.run.smithi097.stdout: "tid": 217014 2024-03-11T21:46:18.635 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:46:18.635 INFO:teuthology.orchestra.run.smithi097.stdout: "events": [ 2024-03-11T21:46:18.635 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:46:18.635 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "initiated", 2024-03-11T21:46:18.635 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:44:24.426189+0000", 2024-03-11T21:46:18.635 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-03-11T21:46:18.635 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:46:18.635 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:46:18.635 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "header_read", 2024-03-11T21:46:18.635 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:44:24.426189+0000", 2024-03-11T21:46:18.635 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-03-11T21:46:18.636 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:46:18.636 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:46:18.636 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "throttled", 2024-03-11T21:46:18.636 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:44:24.426190+0000", 2024-03-11T21:46:18.636 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 9.8899999999999998e-07 2024-03-11T21:46:18.636 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:46:18.636 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:46:18.636 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "all_read", 2024-03-11T21:46:18.636 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:44:24.426194+0000", 2024-03-11T21:46:18.636 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 3.3689999999999998e-06 2024-03-11T21:46:18.636 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:46:18.636 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:46:18.636 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "dispatched", 2024-03-11T21:46:18.636 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:44:24.426200+0000", 2024-03-11T21:46:18.636 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 6.8539999999999999e-06 2024-03-11T21:46:18.637 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:46:18.637 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:46:18.637 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "queued_for_pg", 2024-03-11T21:46:18.637 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:44:24.426206+0000", 2024-03-11T21:46:18.637 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 5.9150000000000001e-06 2024-03-11T21:46:18.637 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:46:18.637 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:46:18.637 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "reached_pg", 2024-03-11T21:46:18.637 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:44:24.430013+0000", 2024-03-11T21:46:18.637 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.0038069359999999999 2024-03-11T21:46:18.637 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:46:18.637 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:46:18.637 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "started", 2024-03-11T21:46:18.637 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:44:24.430124+0000", 2024-03-11T21:46:18.638 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.000111045 2024-03-11T21:46:18.638 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:46:18.638 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:46:18.638 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "waiting for subops from 2,3,5", 2024-03-11T21:46:18.638 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:44:24.430248+0000", 2024-03-11T21:46:18.638 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.000123771 2024-03-11T21:46:18.638 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:46:18.638 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:46:18.638 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "sub_op_commit_rec", 2024-03-11T21:46:18.638 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:44:24.432654+0000", 2024-03-11T21:46:18.638 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.0024057829999999999 2024-03-11T21:46:18.638 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:46:18.638 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:46:18.638 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "op_commit", 2024-03-11T21:46:18.639 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:44:24.435213+0000", 2024-03-11T21:46:18.639 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.0025588239999999999 2024-03-11T21:46:18.639 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:46:18.639 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:46:18.639 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "sub_op_commit_rec", 2024-03-11T21:46:18.639 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:44:24.441459+0000", 2024-03-11T21:46:18.639 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.0062460429999999997 2024-03-11T21:46:18.639 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:46:18.639 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:46:18.639 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "reached_pg", 2024-03-11T21:46:18.639 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:44:24.452570+0000", 2024-03-11T21:46:18.639 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.011110764 2024-03-11T21:46:18.639 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:46:18.639 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:46:18.640 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "waiting for peered", 2024-03-11T21:46:18.640 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:44:24.452582+0000", 2024-03-11T21:46:18.640 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.2588e-05 2024-03-11T21:46:18.640 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:46:18.640 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:46:18.640 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "reached_pg", 2024-03-11T21:46:18.640 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:44:25.473391+0000", 2024-03-11T21:46:18.640 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.020808691 2024-03-11T21:46:18.640 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:46:18.640 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:46:18.640 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "done", 2024-03-11T21:46:18.640 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:44:25.473443+0000", 2024-03-11T21:46:18.640 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 5.2160000000000002e-05 2024-03-11T21:46:18.640 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-03-11T21:46:18.641 INFO:teuthology.orchestra.run.smithi097.stdout: ] 2024-03-11T21:46:18.641 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-03-11T21:46:18.641 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:46:18.641 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:46:18.641 INFO:teuthology.orchestra.run.smithi097.stdout: "description": "osd_op(client.15744.0:217043 10.17 10:e8bb19d6:::benchmark_data_smithi092_161462_object107067:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e435)", 2024-03-11T21:46:18.641 INFO:teuthology.orchestra.run.smithi097.stdout: "initiated_at": "2024-03-11T21:44:24.458969+0000", 2024-03-11T21:46:18.641 INFO:teuthology.orchestra.run.smithi097.stdout: "age": 114.08646624799999, 2024-03-11T21:46:18.641 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.0208789760000001, 2024-03-11T21:46:18.641 INFO:teuthology.orchestra.run.smithi097.stdout: "type_data": { 2024-03-11T21:46:18.641 INFO:teuthology.orchestra.run.smithi097.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-11T21:46:18.641 INFO:teuthology.orchestra.run.smithi097.stdout: "client_info": { 2024-03-11T21:46:18.641 INFO:teuthology.orchestra.run.smithi097.stdout: "client": "client.15744", 2024-03-11T21:46:18.641 INFO:teuthology.orchestra.run.smithi097.stdout: "client_addr": "v1:172.21.15.92:0/3334000812", 2024-03-11T21:46:18.641 INFO:teuthology.orchestra.run.smithi097.stdout: "tid": 217043 2024-03-11T21:46:18.641 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:46:18.642 INFO:teuthology.orchestra.run.smithi097.stdout: "events": [ 2024-03-11T21:46:18.642 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:46:18.642 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "initiated", 2024-03-11T21:46:18.642 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:44:24.458969+0000", 2024-03-11T21:46:18.642 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-03-11T21:46:18.642 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:46:18.642 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:46:18.642 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "header_read", 2024-03-11T21:46:18.642 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:44:24.458969+0000", 2024-03-11T21:46:18.642 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-03-11T21:46:18.642 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:46:18.642 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:46:18.642 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "throttled", 2024-03-11T21:46:18.642 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:44:24.458970+0000", 2024-03-11T21:46:18.643 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.4589999999999999e-06 2024-03-11T21:46:18.643 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:46:18.643 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:46:18.643 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "all_read", 2024-03-11T21:46:18.643 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:44:24.458975+0000", 2024-03-11T21:46:18.643 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 4.8590000000000004e-06 2024-03-11T21:46:18.643 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:46:18.643 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:46:18.643 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "dispatched", 2024-03-11T21:46:18.643 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:44:24.458986+0000", 2024-03-11T21:46:18.643 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.0626e-05 2024-03-11T21:46:18.643 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:46:18.643 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:46:18.643 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "queued_for_pg", 2024-03-11T21:46:18.643 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:44:24.458994+0000", 2024-03-11T21:46:18.644 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 7.8660000000000006e-06 2024-03-11T21:46:18.644 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:46:18.644 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:46:18.644 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "reached_pg", 2024-03-11T21:46:18.644 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:44:24.459205+0000", 2024-03-11T21:46:18.644 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.00021135899999999999 2024-03-11T21:46:18.644 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:46:18.644 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:46:18.644 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "waiting for peered", 2024-03-11T21:46:18.644 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:44:24.459213+0000", 2024-03-11T21:46:18.644 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 8.1519999999999992e-06 2024-03-11T21:46:18.644 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:46:18.644 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:46:18.644 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "reached_pg", 2024-03-11T21:46:18.645 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:44:25.473478+0000", 2024-03-11T21:46:18.645 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.014264906 2024-03-11T21:46:18.645 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:46:18.645 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:46:18.645 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "started", 2024-03-11T21:46:18.645 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:44:25.473561+0000", 2024-03-11T21:46:18.645 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 8.2705000000000002e-05 2024-03-11T21:46:18.645 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:46:18.645 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:46:18.645 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "waiting for subops from 2,5", 2024-03-11T21:46:18.645 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:44:25.473634+0000", 2024-03-11T21:46:18.645 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 7.2727999999999999e-05 2024-03-11T21:46:18.645 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:46:18.645 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:46:18.645 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "sub_op_commit_rec", 2024-03-11T21:46:18.646 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:44:25.477382+0000", 2024-03-11T21:46:18.646 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.0037484390000000001 2024-03-11T21:46:18.646 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:46:18.646 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:46:18.646 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "op_commit", 2024-03-11T21:46:18.646 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:44:25.477527+0000", 2024-03-11T21:46:18.646 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.00014497300000000001 2024-03-11T21:46:18.646 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:46:18.646 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:46:18.646 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "sub_op_commit_rec", 2024-03-11T21:46:18.646 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:44:25.479764+0000", 2024-03-11T21:46:18.646 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.0022373660000000002 2024-03-11T21:46:18.646 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:46:18.646 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:46:18.646 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "commit_sent", 2024-03-11T21:46:18.647 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:44:25.479801+0000", 2024-03-11T21:46:18.647 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 3.6600000000000002e-05 2024-03-11T21:46:18.647 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:46:18.647 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:46:18.647 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "done", 2024-03-11T21:46:18.647 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:44:25.479848+0000", 2024-03-11T21:46:18.647 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 4.6937999999999997e-05 2024-03-11T21:46:18.647 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-03-11T21:46:18.647 INFO:teuthology.orchestra.run.smithi097.stdout: ] 2024-03-11T21:46:18.647 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-03-11T21:46:18.647 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:46:18.647 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:46:18.647 INFO:teuthology.orchestra.run.smithi097.stdout: "description": "osd_op(client.15744.0:217128 10.17 10:edd91f94:::benchmark_data_smithi092_161462_object107152:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e435)", 2024-03-11T21:46:18.647 INFO:teuthology.orchestra.run.smithi097.stdout: "initiated_at": "2024-03-11T21:44:24.548360+0000", 2024-03-11T21:46:18.648 INFO:teuthology.orchestra.run.smithi097.stdout: "age": 113.99707553099999, 2024-03-11T21:46:18.648 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.93895680699999995, 2024-03-11T21:46:18.648 INFO:teuthology.orchestra.run.smithi097.stdout: "type_data": { 2024-03-11T21:46:18.648 INFO:teuthology.orchestra.run.smithi097.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-11T21:46:18.648 INFO:teuthology.orchestra.run.smithi097.stdout: "client_info": { 2024-03-11T21:46:18.648 INFO:teuthology.orchestra.run.smithi097.stdout: "client": "client.15744", 2024-03-11T21:46:18.648 INFO:teuthology.orchestra.run.smithi097.stdout: "client_addr": "v1:172.21.15.92:0/3334000812", 2024-03-11T21:46:18.648 INFO:teuthology.orchestra.run.smithi097.stdout: "tid": 217128 2024-03-11T21:46:18.648 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:46:18.648 INFO:teuthology.orchestra.run.smithi097.stdout: "events": [ 2024-03-11T21:46:18.648 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:46:18.648 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "initiated", 2024-03-11T21:46:18.648 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:44:24.548360+0000", 2024-03-11T21:46:18.648 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-03-11T21:46:18.648 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:46:18.649 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:46:18.649 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "header_read", 2024-03-11T21:46:18.649 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:44:24.548360+0000", 2024-03-11T21:46:18.649 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-03-11T21:46:18.649 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:46:18.649 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:46:18.649 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "throttled", 2024-03-11T21:46:18.649 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:44:24.548361+0000", 2024-03-11T21:46:18.649 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.454e-06 2024-03-11T21:46:18.649 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:46:18.649 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:46:18.649 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "all_read", 2024-03-11T21:46:18.649 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:44:24.548366+0000", 2024-03-11T21:46:18.649 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 4.9520000000000002e-06 2024-03-11T21:46:18.650 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:46:18.650 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:46:18.650 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "dispatched", 2024-03-11T21:46:18.650 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:44:24.548380+0000", 2024-03-11T21:46:18.650 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.3643e-05 2024-03-11T21:46:18.650 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:46:18.650 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:46:18.650 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "queued_for_pg", 2024-03-11T21:46:18.650 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:44:24.548388+0000", 2024-03-11T21:46:18.650 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 8.7530000000000007e-06 2024-03-11T21:46:18.650 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:46:18.650 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:46:18.650 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "reached_pg", 2024-03-11T21:46:18.651 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:44:24.548431+0000", 2024-03-11T21:46:18.651 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 4.2936999999999999e-05 2024-03-11T21:46:18.651 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:46:18.651 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:46:18.651 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "waiting for peered", 2024-03-11T21:46:18.651 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:44:24.548437+0000", 2024-03-11T21:46:18.651 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 5.6389999999999997e-06 2024-03-11T21:46:18.651 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:46:18.651 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:46:18.651 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "reached_pg", 2024-03-11T21:46:18.651 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:44:25.475472+0000", 2024-03-11T21:46:18.651 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.92703524400000004 2024-03-11T21:46:18.651 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:46:18.651 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:46:18.651 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "started", 2024-03-11T21:46:18.652 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:44:25.475549+0000", 2024-03-11T21:46:18.652 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 7.6334999999999993e-05 2024-03-11T21:46:18.652 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:46:18.652 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:46:18.652 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "waiting for subops from 2,5", 2024-03-11T21:46:18.652 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:44:25.475612+0000", 2024-03-11T21:46:18.652 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 6.3127999999999996e-05 2024-03-11T21:46:18.652 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:46:18.652 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:46:18.652 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "sub_op_commit_rec", 2024-03-11T21:46:18.652 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:44:25.477425+0000", 2024-03-11T21:46:18.652 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.0018130830000000001 2024-03-11T21:46:18.652 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:46:18.652 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:46:18.653 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "op_commit", 2024-03-11T21:46:18.653 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:44:25.477535+0000", 2024-03-11T21:46:18.653 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.00011052600000000001 2024-03-11T21:46:18.653 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:46:18.653 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:46:18.653 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "sub_op_commit_rec", 2024-03-11T21:46:18.653 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:44:25.487232+0000", 2024-03-11T21:46:18.653 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.0096962590000000005 2024-03-11T21:46:18.653 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:46:18.653 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:46:18.653 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "commit_sent", 2024-03-11T21:46:18.653 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:44:25.487267+0000", 2024-03-11T21:46:18.653 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 3.5373999999999997e-05 2024-03-11T21:46:18.653 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:46:18.653 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:46:18.654 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "done", 2024-03-11T21:46:18.654 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:44:25.487316+0000", 2024-03-11T21:46:18.654 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 4.9480000000000001e-05 2024-03-11T21:46:18.654 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-03-11T21:46:18.654 INFO:teuthology.orchestra.run.smithi097.stdout: ] 2024-03-11T21:46:18.654 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-03-11T21:46:18.654 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:46:18.654 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:46:18.654 INFO:teuthology.orchestra.run.smithi097.stdout: "description": "osd_op(client.35813.0:34853 11.1 11:802551d2:::benchmark_data_smithi092_186520_object34852:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e452)", 2024-03-11T21:46:18.654 INFO:teuthology.orchestra.run.smithi097.stdout: "initiated_at": "2024-03-11T21:45:21.057909+0000", 2024-03-11T21:46:18.654 INFO:teuthology.orchestra.run.smithi097.stdout: "age": 57.487526526000003, 2024-03-11T21:46:18.654 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.21724974899999999, 2024-03-11T21:46:18.654 INFO:teuthology.orchestra.run.smithi097.stdout: "type_data": { 2024-03-11T21:46:18.654 INFO:teuthology.orchestra.run.smithi097.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-11T21:46:18.655 INFO:teuthology.orchestra.run.smithi097.stdout: "client_info": { 2024-03-11T21:46:18.655 INFO:teuthology.orchestra.run.smithi097.stdout: "client": "client.35813", 2024-03-11T21:46:18.655 INFO:teuthology.orchestra.run.smithi097.stdout: "client_addr": "v1:172.21.15.92:0/4293352030", 2024-03-11T21:46:18.655 INFO:teuthology.orchestra.run.smithi097.stdout: "tid": 34853 2024-03-11T21:46:18.655 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:46:18.655 INFO:teuthology.orchestra.run.smithi097.stdout: "events": [ 2024-03-11T21:46:18.655 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:46:18.655 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "initiated", 2024-03-11T21:46:18.655 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:45:21.057909+0000", 2024-03-11T21:46:18.655 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-03-11T21:46:18.655 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:46:18.655 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:46:18.655 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "header_read", 2024-03-11T21:46:18.655 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:45:21.057909+0000", 2024-03-11T21:46:18.655 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-03-11T21:46:18.656 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:46:18.656 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:46:18.656 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "throttled", 2024-03-11T21:46:18.656 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:45:21.057911+0000", 2024-03-11T21:46:18.656 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.945e-06 2024-03-11T21:46:18.656 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:46:18.656 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:46:18.656 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "all_read", 2024-03-11T21:46:18.656 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:45:21.057949+0000", 2024-03-11T21:46:18.656 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 3.7991000000000003e-05 2024-03-11T21:46:18.656 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:46:18.656 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:46:18.656 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "dispatched", 2024-03-11T21:46:18.656 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:45:21.057959+0000", 2024-03-11T21:46:18.657 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.0441e-05 2024-03-11T21:46:18.657 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:46:18.657 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:46:18.657 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "queued_for_pg", 2024-03-11T21:46:18.657 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:45:21.057966+0000", 2024-03-11T21:46:18.657 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 7.3669999999999999e-06 2024-03-11T21:46:18.657 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:46:18.657 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:46:18.657 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "reached_pg", 2024-03-11T21:46:18.657 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:45:21.058064+0000", 2024-03-11T21:46:18.657 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 9.7529999999999996e-05 2024-03-11T21:46:18.657 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:46:18.657 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:46:18.657 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "started", 2024-03-11T21:46:18.657 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:45:21.058248+0000", 2024-03-11T21:46:18.658 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.000183745 2024-03-11T21:46:18.658 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:46:18.658 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:46:18.658 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "waiting for subops from 0,2", 2024-03-11T21:46:18.658 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:45:21.058521+0000", 2024-03-11T21:46:18.658 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.00027350900000000001 2024-03-11T21:46:18.658 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:46:18.658 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:46:18.658 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "sub_op_commit_rec", 2024-03-11T21:46:18.658 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:45:21.061444+0000", 2024-03-11T21:46:18.658 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.0029226819999999998 2024-03-11T21:46:18.658 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:46:18.658 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:46:18.658 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "op_commit", 2024-03-11T21:46:18.658 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:45:21.061462+0000", 2024-03-11T21:46:18.659 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.7799999999999999e-05 2024-03-11T21:46:18.659 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:46:18.659 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:46:18.659 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "sub_op_commit_rec", 2024-03-11T21:46:18.659 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:45:21.275079+0000", 2024-03-11T21:46:18.659 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.21361718599999999 2024-03-11T21:46:18.659 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:46:18.659 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:46:18.659 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "commit_sent", 2024-03-11T21:46:18.659 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:45:21.275125+0000", 2024-03-11T21:46:18.659 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 4.6538000000000001e-05 2024-03-11T21:46:18.659 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:46:18.659 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:46:18.659 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "done", 2024-03-11T21:46:18.660 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:45:21.275158+0000", 2024-03-11T21:46:18.660 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 3.3015000000000002e-05 2024-03-11T21:46:18.660 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-03-11T21:46:18.660 INFO:teuthology.orchestra.run.smithi097.stdout: ] 2024-03-11T21:46:18.660 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-03-11T21:46:18.660 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:46:18.660 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:46:18.660 INFO:teuthology.orchestra.run.smithi097.stdout: "description": "osd_op(client.35813.0:34854 11.1 11:8a25fd00:::benchmark_data_smithi092_186520_object34853:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e452)", 2024-03-11T21:46:18.660 INFO:teuthology.orchestra.run.smithi097.stdout: "initiated_at": "2024-03-11T21:45:21.058030+0000", 2024-03-11T21:46:18.660 INFO:teuthology.orchestra.run.smithi097.stdout: "age": 57.487404920000003, 2024-03-11T21:46:18.660 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.21721339200000001, 2024-03-11T21:46:18.660 INFO:teuthology.orchestra.run.smithi097.stdout: "type_data": { 2024-03-11T21:46:18.660 INFO:teuthology.orchestra.run.smithi097.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-11T21:46:18.660 INFO:teuthology.orchestra.run.smithi097.stdout: "client_info": { 2024-03-11T21:46:18.660 INFO:teuthology.orchestra.run.smithi097.stdout: "client": "client.35813", 2024-03-11T21:46:18.661 INFO:teuthology.orchestra.run.smithi097.stdout: "client_addr": "v1:172.21.15.92:0/4293352030", 2024-03-11T21:46:18.661 INFO:teuthology.orchestra.run.smithi097.stdout: "tid": 34854 2024-03-11T21:46:18.661 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:46:18.661 INFO:teuthology.orchestra.run.smithi097.stdout: "events": [ 2024-03-11T21:46:18.661 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:46:18.661 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "initiated", 2024-03-11T21:46:18.661 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:45:21.058030+0000", 2024-03-11T21:46:18.661 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-03-11T21:46:18.661 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:46:18.661 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:46:18.661 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "header_read", 2024-03-11T21:46:18.661 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:45:21.058030+0000", 2024-03-11T21:46:18.661 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-03-11T21:46:18.661 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:46:18.661 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:46:18.662 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "throttled", 2024-03-11T21:46:18.662 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:45:21.058031+0000", 2024-03-11T21:46:18.662 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 7.1800000000000005e-07 2024-03-11T21:46:18.662 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:46:18.662 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:46:18.662 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "all_read", 2024-03-11T21:46:18.662 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:45:21.058099+0000", 2024-03-11T21:46:18.662 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 6.8262000000000005e-05 2024-03-11T21:46:18.662 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:46:18.662 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:46:18.662 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "dispatched", 2024-03-11T21:46:18.662 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:45:21.058113+0000", 2024-03-11T21:46:18.662 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.3317e-05 2024-03-11T21:46:18.662 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:46:18.662 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:46:18.663 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "queued_for_pg", 2024-03-11T21:46:18.663 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:45:21.058120+0000", 2024-03-11T21:46:18.663 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 7.4379999999999998e-06 2024-03-11T21:46:18.663 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:46:18.663 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:46:18.663 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "reached_pg", 2024-03-11T21:46:18.663 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:45:21.059950+0000", 2024-03-11T21:46:18.663 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.001829841 2024-03-11T21:46:18.663 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:46:18.663 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:46:18.663 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "started", 2024-03-11T21:46:18.663 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:45:21.060118+0000", 2024-03-11T21:46:18.663 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.00016834099999999999 2024-03-11T21:46:18.663 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:46:18.664 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:46:18.664 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "waiting for subops from 0,2", 2024-03-11T21:46:18.664 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:45:21.060268+0000", 2024-03-11T21:46:18.664 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.000150328 2024-03-11T21:46:18.664 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:46:18.666 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:46:18.666 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "op_commit", 2024-03-11T21:46:18.666 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:45:21.061619+0000", 2024-03-11T21:46:18.666 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.0013500890000000001 2024-03-11T21:46:18.666 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:46:18.666 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:46:18.666 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "sub_op_commit_rec", 2024-03-11T21:46:18.666 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:45:21.061953+0000", 2024-03-11T21:46:18.666 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.00033418599999999999 2024-03-11T21:46:18.666 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:46:18.666 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:46:18.666 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "sub_op_commit_rec", 2024-03-11T21:46:18.666 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:45:21.275187+0000", 2024-03-11T21:46:18.667 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.21323399800000001 2024-03-11T21:46:18.667 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:46:18.667 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:46:18.667 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "commit_sent", 2024-03-11T21:46:18.667 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:45:21.275221+0000", 2024-03-11T21:46:18.667 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 3.4153999999999999e-05 2024-03-11T21:46:18.667 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:46:18.667 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:46:18.667 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "done", 2024-03-11T21:46:18.667 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:45:21.275244+0000", 2024-03-11T21:46:18.667 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 2.2719999999999999e-05 2024-03-11T21:46:18.667 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-03-11T21:46:18.667 INFO:teuthology.orchestra.run.smithi097.stdout: ] 2024-03-11T21:46:18.667 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-03-11T21:46:18.667 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-03-11T21:46:18.668 INFO:teuthology.orchestra.run.smithi097.stdout: ] 2024-03-11T21:46:18.668 INFO:teuthology.orchestra.run.smithi097.stdout:} 2024-03-11T21:46:18.668 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:46:18.669 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:46:18.669 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:46:18.669 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:46:18 smithi092 ceph-mon[29572]: 4.1f scrub starts 2024-03-11T21:46:18.669 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:46:18 smithi092 ceph-mon[29572]: 4.1f scrub ok 2024-03-11T21:46:18.669 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:46:18 smithi092 ceph-mon[37563]: 4.1f scrub starts 2024-03-11T21:46:18.669 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:46:18 smithi092 ceph-mon[37563]: 4.1f scrub ok 2024-03-11T21:46:18.770 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:46:18.811 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:46:18.812 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:46:18.812 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:46:18.913 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:46:18.944 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:46:18.945 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:46:18.945 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:46:19.046 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:46:19.097 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:46:19.097 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:46:19.098 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:46:19.198 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:46:19.232 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:46:19.232 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:46:19.233 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:46:19.268 INFO:tasks.radosbench.radosbench.0.smithi092.stdout: 72 16 115003 114987 99.7956 85.1875 0.00507969 0.0100162 2024-03-11T21:46:19.333 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:46:19.357 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:46:19.358 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:46:19.358 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:46:19.459 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:46:19.499 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:46:19.499 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:46:19.500 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:46:19.584 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:46:19 smithi097 ceph-mon[33317]: pgmap v978: 148 pgs: 148 active+clean; 6.6 GiB data, 24 GiB used, 512 GiB / 536 GiB avail; 649 B/s rd, 56 MiB/s wr, 901 op/s 2024-03-11T21:46:19.584 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:46:19 smithi097 ceph-mon[33317]: 2.3 scrub starts 2024-03-11T21:46:19.584 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:46:19 smithi097 ceph-mon[33317]: 2.3 scrub ok 2024-03-11T21:46:19.584 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:46:19 smithi097 ceph-mon[33317]: 6.1 scrub starts 2024-03-11T21:46:19.584 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:46:19 smithi097 ceph-mon[33317]: 6.1 scrub ok 2024-03-11T21:46:19.584 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:46:19 smithi097 ceph-mon[33317]: 6.16 scrub starts 2024-03-11T21:46:19.584 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:46:19 smithi097 ceph-mon[33317]: 4.11 scrub starts 2024-03-11T21:46:19.584 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:46:19 smithi097 ceph-mon[33317]: 4.11 scrub ok 2024-03-11T21:46:19.585 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:46:19 smithi097 ceph-mon[33317]: 6.16 scrub ok 2024-03-11T21:46:19.585 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:46:19 smithi097 ceph-mon[33317]: osdmap e465: 8 total, 8 up, 6 in 2024-03-11T21:46:19.601 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:46:19.616 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:46:19 smithi092 ceph-mon[37563]: pgmap v978: 148 pgs: 148 active+clean; 6.6 GiB data, 24 GiB used, 512 GiB / 536 GiB avail; 649 B/s rd, 56 MiB/s wr, 901 op/s 2024-03-11T21:46:19.616 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:46:19 smithi092 ceph-mon[37563]: 2.3 scrub starts 2024-03-11T21:46:19.616 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:46:19 smithi092 ceph-mon[37563]: 2.3 scrub ok 2024-03-11T21:46:19.616 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:46:19 smithi092 ceph-mon[37563]: 6.1 scrub starts 2024-03-11T21:46:19.616 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:46:19 smithi092 ceph-mon[37563]: 6.1 scrub ok 2024-03-11T21:46:19.616 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:46:19 smithi092 ceph-mon[37563]: 6.16 scrub starts 2024-03-11T21:46:19.616 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:46:19 smithi092 ceph-mon[37563]: 4.11 scrub starts 2024-03-11T21:46:19.616 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:46:19 smithi092 ceph-mon[37563]: 4.11 scrub ok 2024-03-11T21:46:19.616 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:46:19 smithi092 ceph-mon[37563]: 6.16 scrub ok 2024-03-11T21:46:19.616 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:46:19 smithi092 ceph-mon[37563]: osdmap e465: 8 total, 8 up, 6 in 2024-03-11T21:46:19.617 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:46:19 smithi092 ceph-mon[29572]: pgmap v978: 148 pgs: 148 active+clean; 6.6 GiB data, 24 GiB used, 512 GiB / 536 GiB avail; 649 B/s rd, 56 MiB/s wr, 901 op/s 2024-03-11T21:46:19.617 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:46:19 smithi092 ceph-mon[29572]: 2.3 scrub starts 2024-03-11T21:46:19.617 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:46:19 smithi092 ceph-mon[29572]: 2.3 scrub ok 2024-03-11T21:46:19.617 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:46:19 smithi092 ceph-mon[29572]: 6.1 scrub starts 2024-03-11T21:46:19.617 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:46:19 smithi092 ceph-mon[29572]: 6.1 scrub ok 2024-03-11T21:46:19.617 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:46:19 smithi092 ceph-mon[29572]: 6.16 scrub starts 2024-03-11T21:46:19.617 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:46:19 smithi092 ceph-mon[29572]: 4.11 scrub starts 2024-03-11T21:46:19.617 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:46:19 smithi092 ceph-mon[29572]: 4.11 scrub ok 2024-03-11T21:46:19.617 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:46:19 smithi092 ceph-mon[29572]: 6.16 scrub ok 2024-03-11T21:46:19.617 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:46:19 smithi092 ceph-mon[29572]: osdmap e465: 8 total, 8 up, 6 in 2024-03-11T21:46:19.653 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:46:19.654 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:46:19.654 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:46:19.755 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:46:19.797 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:46:19.797 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:46:19.798 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:46:19.899 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:46:19.929 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:46:19.929 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:46:19.930 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:46:20.031 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:46:20.061 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:46:20.061 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:46:20.061 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:46:20.162 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:46:20.191 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:46:20.191 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:46:20.191 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:46:20.268 INFO:tasks.radosbench.radosbench.0.smithi092.stdout: 73 16 116186 116170 99.4413 73.9375 0.0093381 0.0100513 2024-03-11T21:46:20.292 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:46:20.307 INFO:teuthology.orchestra.run.smithi092.stderr:nodeep-scrub is unset 2024-03-11T21:46:20.321 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:46:20.322 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:46:20.323 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:46:20.424 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:46:20.453 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:46:20.454 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:46:20.454 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:46:20.474 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:46:20 smithi092 ceph-mon[29572]: 4.8 scrub starts 2024-03-11T21:46:20.474 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:46:20 smithi092 ceph-mon[29572]: 4.8 scrub ok 2024-03-11T21:46:20.474 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:46:20 smithi092 ceph-mon[29572]: 2.0 scrub starts 2024-03-11T21:46:20.474 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:46:20 smithi092 ceph-mon[29572]: 2.0 scrub ok 2024-03-11T21:46:20.474 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:46:20 smithi092 ceph-mon[29572]: from='client.? v1:172.21.15.92:0/1100128880' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-11T21:46:20.474 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:46:20 smithi092 ceph-mon[37563]: 4.8 scrub starts 2024-03-11T21:46:20.474 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:46:20 smithi092 ceph-mon[37563]: 4.8 scrub ok 2024-03-11T21:46:20.474 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:46:20 smithi092 ceph-mon[37563]: 2.0 scrub starts 2024-03-11T21:46:20.474 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:46:20 smithi092 ceph-mon[37563]: 2.0 scrub ok 2024-03-11T21:46:20.475 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:46:20 smithi092 ceph-mon[37563]: from='client.? v1:172.21.15.92:0/1100128880' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-11T21:46:20.555 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:46:20.584 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:46:20 smithi097 ceph-mon[33317]: 4.8 scrub starts 2024-03-11T21:46:20.584 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:46:20 smithi097 ceph-mon[33317]: 4.8 scrub ok 2024-03-11T21:46:20.584 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:46:20 smithi097 ceph-mon[33317]: 2.0 scrub starts 2024-03-11T21:46:20.584 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:46:20 smithi097 ceph-mon[33317]: 2.0 scrub ok 2024-03-11T21:46:20.584 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:46:20 smithi097 ceph-mon[33317]: from='client.? v1:172.21.15.92:0/1100128880' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-11T21:46:20.588 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:46:20.589 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:46:20.589 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:46:20.690 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:46:20.734 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:46:20.734 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:46:20.734 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:46:20.763 INFO:teuthology.orchestra.run.smithi092.stdout: 2024-03-11T21:46:20.768 INFO:teuthology.orchestra.run.smithi092.stderr:dumped all 2024-03-11T21:46:20.835 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:46:20.880 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:46:20.881 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:46:20.881 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:46:20.974 DEBUG:teuthology.orchestra.run.smithi097:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid 00db0016-dfed-11ee-95c8-87774f69a715 -- ceph daemon osd.7 dump_ops_in_flight 2024-03-11T21:46:20.981 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:46:21.020 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:46:21.021 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:46:21.021 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:46:21.123 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:46:21.158 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:46:21.158 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:46:21.159 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:46:21.259 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:46:21.268 INFO:tasks.radosbench.radosbench.0.smithi092.stdout: 74 16 117270 117254 99.0131 67.75 0.00700757 0.0100951 2024-03-11T21:46:21.293 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:46:21 smithi092 ceph-mon[29572]: pgmap v980: 148 pgs: 148 active+clean; 6.9 GiB data, 25 GiB used, 511 GiB / 536 GiB avail; 1.7 KiB/s rd, 84 MiB/s wr, 1.34k op/s 2024-03-11T21:46:21.294 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:46:21 smithi092 ceph-mon[37563]: pgmap v980: 148 pgs: 148 active+clean; 6.9 GiB data, 25 GiB used, 511 GiB / 536 GiB avail; 1.7 KiB/s rd, 84 MiB/s wr, 1.34k op/s 2024-03-11T21:46:21.294 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:46:21 smithi092 ceph-mon[37563]: 5.d scrub starts 2024-03-11T21:46:21.294 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:46:21 smithi092 ceph-mon[37563]: 5.d scrub ok 2024-03-11T21:46:21.294 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:46:21 smithi092 ceph-mon[37563]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-03-11T21:46:21.294 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:46:21 smithi092 ceph-mon[37563]: Cluster is now healthy 2024-03-11T21:46:21.294 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:46:21 smithi092 ceph-mon[37563]: from='client.? v1:172.21.15.92:0/1100128880' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-03-11T21:46:21.294 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:46:21 smithi092 ceph-mon[37563]: osdmap e466: 8 total, 8 up, 6 in 2024-03-11T21:46:21.294 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:46:21 smithi092 ceph-mon[37563]: 2.f scrub starts 2024-03-11T21:46:21.294 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:46:21 smithi092 ceph-mon[37563]: 2.f scrub ok 2024-03-11T21:46:21.298 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:46:21.299 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:46:21.300 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:46:21.351 INFO:teuthology.orchestra.run.smithi097.stderr:Inferring config /var/lib/ceph/00db0016-dfed-11ee-95c8-87774f69a715/mon.b/config 2024-03-11T21:46:21.400 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:46:21.439 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:46:21.440 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:46:21.440 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:46:21.541 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:46:21.573 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:46:21.574 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:46:21.575 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:46:21.584 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:46:21 smithi097 ceph-mon[33317]: pgmap v980: 148 pgs: 148 active+clean; 6.9 GiB data, 25 GiB used, 511 GiB / 536 GiB avail; 1.7 KiB/s rd, 84 MiB/s wr, 1.34k op/s 2024-03-11T21:46:21.584 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:46:21 smithi097 ceph-mon[33317]: 5.d scrub starts 2024-03-11T21:46:21.584 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:46:21 smithi097 ceph-mon[33317]: 5.d scrub ok 2024-03-11T21:46:21.584 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:46:21 smithi097 ceph-mon[33317]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-03-11T21:46:21.585 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:46:21 smithi097 ceph-mon[33317]: Cluster is now healthy 2024-03-11T21:46:21.585 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:46:21 smithi097 ceph-mon[33317]: from='client.? v1:172.21.15.92:0/1100128880' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-03-11T21:46:21.585 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:46:21 smithi097 ceph-mon[33317]: osdmap e466: 8 total, 8 up, 6 in 2024-03-11T21:46:21.585 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:46:21 smithi097 ceph-mon[33317]: 2.f scrub starts 2024-03-11T21:46:21.585 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:46:21 smithi097 ceph-mon[33317]: 2.f scrub ok 2024-03-11T21:46:21.618 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:46:21 smithi092 ceph-mon[29572]: 5.d scrub starts 2024-03-11T21:46:21.618 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:46:21 smithi092 ceph-mon[29572]: 5.d scrub ok 2024-03-11T21:46:21.618 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:46:21 smithi092 ceph-mon[29572]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-03-11T21:46:21.618 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:46:21 smithi092 ceph-mon[29572]: Cluster is now healthy 2024-03-11T21:46:21.618 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:46:21 smithi092 ceph-mon[29572]: from='client.? v1:172.21.15.92:0/1100128880' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-03-11T21:46:21.618 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:46:21 smithi092 ceph-mon[29572]: osdmap e466: 8 total, 8 up, 6 in 2024-03-11T21:46:21.618 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:46:21 smithi092 ceph-mon[29572]: 2.f scrub starts 2024-03-11T21:46:21.619 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:46:21 smithi092 ceph-mon[29572]: 2.f scrub ok 2024-03-11T21:46:21.676 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:46:21.715 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:46:21.715 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:46:21.716 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:46:21.816 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:46:21.859 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:46:21.860 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:46:21.860 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:46:21.958 INFO:teuthology.orchestra.run.smithi092.stdout:{"pg_ready":true,"pg_map":{"version":981,"stamp":"2024-03-11T21:46:20.298437+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":7433558440,"num_objects":113616,"num_object_clones":0,"num_object_copies":340848,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":113616,"num_whiteouts":0,"num_read":2981,"num_read_kb":2829,"num_write":114605,"num_write_kb":7261213,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":330,"num_bytes_recovered":2960245,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":66909,"ondisk_log_size":66909,"up":444,"acting":444,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":444,"num_osds":6,"num_per_pool_osds":6,"num_per_pool_omap_osds":6,"kb":562470912,"kb_used":26441260,"kb_used_data":21697776,"kb_used_omap":9,"kb_used_meta":4743414,"kb_avail":536029652,"statfs":{"total":575970213888,"available":548894363648,"internally_reserved":0,"allocated":22218522624,"data_stored":22215679490,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":9564,"internal_metadata":4857256612},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[7,4,5,4,2,1],"upper_bound":64},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":553713664,"num_objects":8449,"num_object_clones":0,"num_object_copies":25347,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8449,"num_whiteouts":0,"num_read":10,"num_read_kb":10,"num_write":8449,"num_write_kb":540736,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.743919"},"pg_stats":[{"pgid":"6.1b","version":"0'0","reported_seq":484,"reported_epoch":464,"state":"active+clean","last_fresh":"2024-03-11T21:46:16.299368+0000","last_change":"2024-03-11T21:44:30.546716+0000","last_active":"2024-03-11T21:46:16.299368+0000","last_peered":"2024-03-11T21:46:16.299368+0000","last_clean":"2024-03-11T21:46:16.299368+0000","last_became_active":"2024-03-11T21:44:21.459293+0000","last_became_peered":"2024-03-11T21:44:21.459293+0000","last_unstale":"2024-03-11T21:46:16.299368+0000","last_undegraded":"2024-03-11T21:46:16.299368+0000","last_fullsized":"2024-03-11T21:46:16.299368+0000","mapping_epoch":432,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":433,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:44:30.546596+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:43.559452+0000","last_clean_scrub_stamp":"2024-03-11T21:44:30.546596+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":108,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,1],"acting":[7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":72,"reported_epoch":464,"state":"active+clean","last_fresh":"2024-03-11T21:46:16.288941+0000","last_change":"2024-03-11T21:45:16.535519+0000","last_active":"2024-03-11T21:46:16.288941+0000","last_peered":"2024-03-11T21:46:16.288941+0000","last_clean":"2024-03-11T21:46:16.288941+0000","last_became_active":"2024-03-11T21:44:35.624802+0000","last_became_peered":"2024-03-11T21:44:35.624802+0000","last_unstale":"2024-03-11T21:46:16.288941+0000","last_undegraded":"2024-03-11T21:46:16.288941+0000","last_fullsized":"2024-03-11T21:46:16.288941+0000","mapping_epoch":437,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":438,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:45:16.535402+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:41.544550+0000","last_clean_scrub_stamp":"2024-03-11T21:45:16.535402+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:46:31.223437+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,2],"acting":[6,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.1f","version":"0'0","reported_seq":898,"reported_epoch":465,"state":"active+clean","last_fresh":"2024-03-11T21:46:18.568254+0000","last_change":"2024-03-11T21:45:34.145990+0000","last_active":"2024-03-11T21:46:18.568254+0000","last_peered":"2024-03-11T21:46:18.568254+0000","last_clean":"2024-03-11T21:46:18.568254+0000","last_became_active":"2024-03-11T21:44:35.644794+0000","last_became_peered":"2024-03-11T21:44:35.644794+0000","last_unstale":"2024-03-11T21:46:18.568254+0000","last_undegraded":"2024-03-11T21:46:18.568254+0000","last_fullsized":"2024-03-11T21:46:18.568254+0000","mapping_epoch":437,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":438,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:45:34.145740+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:37.512018+0000","last_clean_scrub_stamp":"2024-03-11T21:45:34.145740+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:46:55.729798+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,6],"acting":[0,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.19","version":"456'39","reported_seq":232,"reported_epoch":464,"state":"active+clean","last_fresh":"2024-03-11T21:46:16.292236+0000","last_change":"2024-03-11T21:45:19.593264+0000","last_active":"2024-03-11T21:46:16.292236+0000","last_peered":"2024-03-11T21:46:16.292236+0000","last_clean":"2024-03-11T21:46:16.292236+0000","last_became_active":"2024-03-11T21:44:21.456600+0000","last_became_peered":"2024-03-11T21:44:21.456600+0000","last_unstale":"2024-03-11T21:46:16.292236+0000","last_undegraded":"2024-03-11T21:46:16.292236+0000","last_fullsized":"2024-03-11T21:46:16.292236+0000","mapping_epoch":432,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":433,"parent":"0.0","parent_split_bits":0,"last_scrub":"260'27","last_scrub_stamp":"2024-03-11T21:45:19.592959+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:39.525554+0000","last_clean_scrub_stamp":"2024-03-11T21:45:19.592959+0000","objects_scrubbed":7,"log_size":39,"log_dups_size":0,"ondisk_log_size":39,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:46:44.944685+0000","scrub_duration":22,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":61,"num_read_kb":52,"num_write":40,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,5],"acting":[2,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.1a","version":"0'0","reported_seq":72,"reported_epoch":464,"state":"active+clean","last_fresh":"2024-03-11T21:46:16.299719+0000","last_change":"2024-03-11T21:46:15.911628+0000","last_active":"2024-03-11T21:46:16.299719+0000","last_peered":"2024-03-11T21:46:16.299719+0000","last_clean":"2024-03-11T21:46:16.299719+0000","last_became_active":"2024-03-11T21:44:35.632682+0000","last_became_peered":"2024-03-11T21:44:35.632682+0000","last_unstale":"2024-03-11T21:46:16.299719+0000","last_undegraded":"2024-03-11T21:46:16.299719+0000","last_fullsized":"2024-03-11T21:46:16.299719+0000","mapping_epoch":437,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":438,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:46:15.911512+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:43.559452+0000","last_clean_scrub_stamp":"2024-03-11T21:46:15.911512+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:47:43.114084+0000","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,7],"acting":[5,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":867,"reported_epoch":465,"state":"active+clean","last_fresh":"2024-03-11T21:46:18.566250+0000","last_change":"2024-03-11T21:44:16.914576+0000","last_active":"2024-03-11T21:46:18.566250+0000","last_peered":"2024-03-11T21:46:18.566250+0000","last_clean":"2024-03-11T21:46:18.566250+0000","last_became_active":"2024-03-11T21:34:41.154216+0000","last_became_peered":"2024-03-11T21:34:41.154216+0000","last_unstale":"2024-03-11T21:46:18.566250+0000","last_undegraded":"2024-03-11T21:46:18.566250+0000","last_fullsized":"2024-03-11T21:46:18.566250+0000","mapping_epoch":226,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":227,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:44:16.914492+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:41.544550+0000","last_clean_scrub_stamp":"2024-03-11T21:44:16.914492+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,7],"acting":[1,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1e","version":"0'0","reported_seq":493,"reported_epoch":465,"state":"active+clean","last_fresh":"2024-03-11T21:46:18.568094+0000","last_change":"2024-03-11T21:44:37.214793+0000","last_active":"2024-03-11T21:46:18.568094+0000","last_peered":"2024-03-11T21:46:18.568094+0000","last_clean":"2024-03-11T21:46:18.568094+0000","last_became_active":"2024-03-11T21:44:21.464072+0000","last_became_peered":"2024-03-11T21:44:21.464072+0000","last_unstale":"2024-03-11T21:46:18.568094+0000","last_undegraded":"2024-03-11T21:46:18.568094+0000","last_fullsized":"2024-03-11T21:46:18.568094+0000","mapping_epoch":432,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":433,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:44:37.214702+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:37.512018+0000","last_clean_scrub_stamp":"2024-03-11T21:44:37.214702+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.18","version":"456'21","reported_seq":551,"reported_epoch":464,"state":"active+clean","last_fresh":"2024-03-11T21:46:16.288914+0000","last_change":"2024-03-11T21:45:24.708559+0000","last_active":"2024-03-11T21:46:16.288914+0000","last_peered":"2024-03-11T21:46:16.288914+0000","last_clean":"2024-03-11T21:46:16.288914+0000","last_became_active":"2024-03-11T21:44:35.621442+0000","last_became_peered":"2024-03-11T21:44:35.621442+0000","last_unstale":"2024-03-11T21:46:16.288914+0000","last_undegraded":"2024-03-11T21:46:16.288914+0000","last_fullsized":"2024-03-11T21:46:16.288914+0000","mapping_epoch":437,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":438,"parent":"0.0","parent_split_bits":0,"last_scrub":"260'15","last_scrub_stamp":"2024-03-11T21:45:24.708228+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:39.525554+0000","last_clean_scrub_stamp":"2024-03-11T21:45:24.708228+0000","objects_scrubbed":4,"log_size":21,"log_dups_size":0,"ondisk_log_size":21,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:46:43.001984+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":34,"num_read_kb":28,"num_write":22,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,2],"acting":[6,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":871,"reported_epoch":464,"state":"active+clean","last_fresh":"2024-03-11T21:46:16.299473+0000","last_change":"2024-03-11T21:44:21.453278+0000","last_active":"2024-03-11T21:46:16.299473+0000","last_peered":"2024-03-11T21:46:16.299473+0000","last_clean":"2024-03-11T21:46:16.299473+0000","last_became_active":"2024-03-11T21:44:21.452958+0000","last_became_peered":"2024-03-11T21:44:21.452958+0000","last_unstale":"2024-03-11T21:46:16.299473+0000","last_undegraded":"2024-03-11T21:46:16.299473+0000","last_fullsized":"2024-03-11T21:46:16.299473+0000","mapping_epoch":432,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":433,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:44:17.913837+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:32:12.654573+0000","last_clean_scrub_stamp":"2024-03-11T21:44:17.913837+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":83,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,2],"acting":[5,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":519,"reported_epoch":464,"state":"active+clean","last_fresh":"2024-03-11T21:46:16.300427+0000","last_change":"2024-03-11T21:44:35.626808+0000","last_active":"2024-03-11T21:46:16.300427+0000","last_peered":"2024-03-11T21:46:16.300427+0000","last_clean":"2024-03-11T21:46:16.300427+0000","last_became_active":"2024-03-11T21:44:35.626456+0000","last_became_peered":"2024-03-11T21:44:35.626456+0000","last_unstale":"2024-03-11T21:46:16.300427+0000","last_undegraded":"2024-03-11T21:46:16.300427+0000","last_fullsized":"2024-03-11T21:46:16.300427+0000","mapping_epoch":437,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":438,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:44:19.481984+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:41.544550+0000","last_clean_scrub_stamp":"2024-03-11T21:44:19.481984+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,0],"acting":[7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.1d","version":"0'0","reported_seq":876,"reported_epoch":464,"state":"active+clean","last_fresh":"2024-03-11T21:46:16.301582+0000","last_change":"2024-03-11T21:44:13.414522+0000","last_active":"2024-03-11T21:46:16.301582+0000","last_peered":"2024-03-11T21:46:16.301582+0000","last_clean":"2024-03-11T21:46:16.301582+0000","last_became_active":"2024-03-11T21:25:38.542628+0000","last_became_peered":"2024-03-11T21:25:38.542628+0000","last_unstale":"2024-03-11T21:46:16.301582+0000","last_undegraded":"2024-03-11T21:46:16.301582+0000","last_fullsized":"2024-03-11T21:46:16.301582+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:44:13.414407+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:37.512018+0000","last_clean_scrub_stamp":"2024-03-11T21:44:13.414407+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,0],"acting":[7,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.1b","version":"456'9","reported_seq":526,"reported_epoch":465,"state":"active+clean","last_fresh":"2024-03-11T21:46:18.565868+0000","last_change":"2024-03-11T21:46:08.932984+0000","last_active":"2024-03-11T21:46:18.565868+0000","last_peered":"2024-03-11T21:46:18.565868+0000","last_clean":"2024-03-11T21:46:18.565868+0000","last_became_active":"2024-03-11T21:44:35.639372+0000","last_became_peered":"2024-03-11T21:44:35.639372+0000","last_unstale":"2024-03-11T21:46:18.565868+0000","last_undegraded":"2024-03-11T21:46:18.565868+0000","last_fullsized":"2024-03-11T21:46:18.565868+0000","mapping_epoch":437,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":438,"parent":"0.0","parent_split_bits":0,"last_scrub":"456'9","last_scrub_stamp":"2024-03-11T21:46:08.932675+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:39.525554+0000","last_clean_scrub_stamp":"2024-03-11T21:46:08.932675+0000","objects_scrubbed":2,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:47:27.868481+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":16,"num_read_kb":12,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,7],"acting":[1,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":859,"reported_epoch":465,"state":"active+clean","last_fresh":"2024-03-11T21:46:18.564877+0000","last_change":"2024-03-11T21:45:32.208394+0000","last_active":"2024-03-11T21:46:18.564877+0000","last_peered":"2024-03-11T21:46:18.564877+0000","last_clean":"2024-03-11T21:46:18.564877+0000","last_became_active":"2024-03-11T21:25:44.589786+0000","last_became_peered":"2024-03-11T21:25:44.589786+0000","last_unstale":"2024-03-11T21:46:18.564877+0000","last_undegraded":"2024-03-11T21:46:18.564877+0000","last_fullsized":"2024-03-11T21:46:18.564877+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:45:32.208161+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:43.559452+0000","last_clean_scrub_stamp":"2024-03-11T21:45:32.208161+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:46:57.686755+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,7],"acting":[0,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":880,"reported_epoch":464,"state":"active+clean","last_fresh":"2024-03-11T21:46:16.300027+0000","last_change":"2024-03-11T21:46:13.862522+0000","last_active":"2024-03-11T21:46:16.300027+0000","last_peered":"2024-03-11T21:46:16.300027+0000","last_clean":"2024-03-11T21:46:16.300027+0000","last_became_active":"2024-03-11T21:34:41.147185+0000","last_became_peered":"2024-03-11T21:34:41.147185+0000","last_unstale":"2024-03-11T21:46:16.300027+0000","last_undegraded":"2024-03-11T21:46:16.300027+0000","last_fullsized":"2024-03-11T21:46:16.300027+0000","mapping_epoch":226,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":227,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:46:13.862444+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:43:35.536164+0000","last_clean_scrub_stamp":"2024-03-11T21:46:13.862444+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:47:29.947411+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,7],"acting":[5,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1c","version":"0'0","reported_seq":876,"reported_epoch":464,"state":"active+clean","last_fresh":"2024-03-11T21:46:16.301498+0000","last_change":"2024-03-11T21:45:24.180930+0000","last_active":"2024-03-11T21:46:16.301498+0000","last_peered":"2024-03-11T21:46:16.301498+0000","last_clean":"2024-03-11T21:46:16.301498+0000","last_became_active":"2024-03-11T21:25:38.541364+0000","last_became_peered":"2024-03-11T21:25:38.541364+0000","last_unstale":"2024-03-11T21:46:16.301498+0000","last_undegraded":"2024-03-11T21:46:16.301498+0000","last_fullsized":"2024-03-11T21:46:16.301498+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:45:24.180727+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:28:32.566437+0000","last_clean_scrub_stamp":"2024-03-11T21:45:24.180727+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:46:27.894278+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.1a","version":"456'25","reported_seq":757,"reported_epoch":465,"state":"active+clean","last_fresh":"2024-03-11T21:46:18.567781+0000","last_change":"2024-03-11T21:46:06.451173+0000","last_active":"2024-03-11T21:46:18.567781+0000","last_peered":"2024-03-11T21:46:18.567781+0000","last_clean":"2024-03-11T21:46:18.567781+0000","last_became_active":"2024-03-11T21:44:35.631905+0000","last_became_peered":"2024-03-11T21:44:35.631905+0000","last_unstale":"2024-03-11T21:46:18.567781+0000","last_undegraded":"2024-03-11T21:46:18.567781+0000","last_fullsized":"2024-03-11T21:46:18.567781+0000","mapping_epoch":437,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":438,"parent":"0.0","parent_split_bits":0,"last_scrub":"456'25","last_scrub_stamp":"2024-03-11T21:46:06.451051+0000","last_deep_scrub":"350'19","last_deep_scrub_stamp":"2024-03-11T21:43:25.898708+0000","last_clean_scrub_stamp":"2024-03-11T21:46:06.451051+0000","objects_scrubbed":4,"log_size":25,"log_dups_size":0,"ondisk_log_size":25,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:47:07.468484+0000","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":35,"num_read_kb":29,"num_write":22,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,6],"acting":[0,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.1f","version":"0'0","reported_seq":414,"reported_epoch":464,"state":"active+clean","last_fresh":"2024-03-11T21:46:16.290080+0000","last_change":"2024-03-11T21:44:31.472515+0000","last_active":"2024-03-11T21:46:16.290080+0000","last_peered":"2024-03-11T21:46:16.290080+0000","last_clean":"2024-03-11T21:46:16.290080+0000","last_became_active":"2024-03-11T21:44:21.447850+0000","last_became_peered":"2024-03-11T21:44:21.447850+0000","last_unstale":"2024-03-11T21:46:16.290080+0000","last_undegraded":"2024-03-11T21:46:16.290080+0000","last_fullsized":"2024-03-11T21:46:16.290080+0000","mapping_epoch":432,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":433,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:44:31.472397+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:44:31.472397+0000","last_clean_scrub_stamp":"2024-03-11T21:44:31.472397+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":108,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,1],"acting":[6,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.1c","version":"0'0","reported_seq":868,"reported_epoch":464,"state":"active+clean","last_fresh":"2024-03-11T21:46:16.292233+0000","last_change":"2024-03-11T21:45:26.645211+0000","last_active":"2024-03-11T21:46:16.292233+0000","last_peered":"2024-03-11T21:46:16.292233+0000","last_clean":"2024-03-11T21:46:16.292233+0000","last_became_active":"2024-03-11T21:44:35.624752+0000","last_became_peered":"2024-03-11T21:44:35.624752+0000","last_unstale":"2024-03-11T21:46:16.292233+0000","last_undegraded":"2024-03-11T21:46:16.292233+0000","last_fullsized":"2024-03-11T21:46:16.292233+0000","mapping_epoch":437,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":438,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:45:26.644785+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:43:32.021344+0000","last_clean_scrub_stamp":"2024-03-11T21:45:26.644785+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:46:37.379668+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,1],"acting":[2,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1b","version":"0'0","reported_seq":510,"reported_epoch":464,"state":"active+clean","last_fresh":"2024-03-11T21:46:16.299530+0000","last_change":"2024-03-11T21:45:29.145281+0000","last_active":"2024-03-11T21:46:16.299530+0000","last_peered":"2024-03-11T21:46:16.299530+0000","last_clean":"2024-03-11T21:46:16.299530+0000","last_became_active":"2024-03-11T21:44:35.638681+0000","last_became_peered":"2024-03-11T21:44:35.638681+0000","last_unstale":"2024-03-11T21:46:16.299530+0000","last_undegraded":"2024-03-11T21:46:16.299530+0000","last_fullsized":"2024-03-11T21:46:16.299530+0000","mapping_epoch":437,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":438,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:45:29.145166+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:34:27.998724+0000","last_clean_scrub_stamp":"2024-03-11T21:45:29.145166+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:46:32.581606+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,5],"acting":[7,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.1d","version":"456'24","reported_seq":228,"reported_epoch":465,"state":"active+clean","last_fresh":"2024-03-11T21:46:18.565572+0000","last_change":"2024-03-11T21:46:10.906686+0000","last_active":"2024-03-11T21:46:18.565572+0000","last_peered":"2024-03-11T21:46:18.565572+0000","last_clean":"2024-03-11T21:46:18.565572+0000","last_became_active":"2024-03-11T21:44:35.622640+0000","last_became_peered":"2024-03-11T21:44:35.622640+0000","last_unstale":"2024-03-11T21:46:18.565572+0000","last_undegraded":"2024-03-11T21:46:18.565572+0000","last_fullsized":"2024-03-11T21:46:18.565572+0000","mapping_epoch":437,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":438,"parent":"0.0","parent_split_bits":0,"last_scrub":"456'24","last_scrub_stamp":"2024-03-11T21:46:10.906594+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:39.525554+0000","last_clean_scrub_stamp":"2024-03-11T21:46:10.906594+0000","objects_scrubbed":5,"log_size":24,"log_dups_size":0,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:47:32.187205+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":42,"num_read_kb":33,"num_write":26,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":440,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,5],"acting":[1,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1e","version":"0'0","reported_seq":72,"reported_epoch":464,"state":"active+clean","last_fresh":"2024-03-11T21:46:16.289730+0000","last_change":"2024-03-11T21:44:39.482534+0000","last_active":"2024-03-11T21:46:16.289730+0000","last_peered":"2024-03-11T21:46:16.289730+0000","last_clean":"2024-03-11T21:46:16.289730+0000","last_became_active":"2024-03-11T21:44:35.613979+0000","last_became_peered":"2024-03-11T21:44:35.613979+0000","last_unstale":"2024-03-11T21:46:16.289730+0000","last_undegraded":"2024-03-11T21:46:16.289730+0000","last_fullsized":"2024-03-11T21:46:16.289730+0000","mapping_epoch":437,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":438,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:44:39.482452+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:43.559452+0000","last_clean_scrub_stamp":"2024-03-11T21:44:39.482452+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":28,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,0],"acting":[6,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.1d","version":"0'0","reported_seq":869,"reported_epoch":465,"state":"active+clean","last_fresh":"2024-03-11T21:46:18.567173+0000","last_change":"2024-03-11T21:46:05.965579+0000","last_active":"2024-03-11T21:46:18.567173+0000","last_peered":"2024-03-11T21:46:18.567173+0000","last_clean":"2024-03-11T21:46:18.567173+0000","last_became_active":"2024-03-11T21:44:35.621565+0000","last_became_peered":"2024-03-11T21:44:35.621565+0000","last_unstale":"2024-03-11T21:46:18.567173+0000","last_undegraded":"2024-03-11T21:46:18.567173+0000","last_fullsized":"2024-03-11T21:46:18.567173+0000","mapping_epoch":437,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":438,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:46:05.965504+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:30:18.678531+0000","last_clean_scrub_stamp":"2024-03-11T21:46:05.965504+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:47:33.832096+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,5],"acting":[1,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1a","version":"0'0","reported_seq":903,"reported_epoch":464,"state":"active+clean","last_fresh":"2024-03-11T21:46:16.289715+0000","last_change":"2024-03-11T21:45:31.690813+0000","last_active":"2024-03-11T21:46:16.289715+0000","last_peered":"2024-03-11T21:46:16.289715+0000","last_clean":"2024-03-11T21:46:16.289715+0000","last_became_active":"2024-03-11T21:44:35.630761+0000","last_became_peered":"2024-03-11T21:44:35.630761+0000","last_unstale":"2024-03-11T21:46:16.289715+0000","last_undegraded":"2024-03-11T21:46:16.289715+0000","last_fullsized":"2024-03-11T21:46:16.289715+0000","mapping_epoch":437,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":438,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:45:31.690656+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:37.512018+0000","last_clean_scrub_stamp":"2024-03-11T21:45:31.690656+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:46:35.291100+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,2],"acting":[6,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.1c","version":"456'39","reported_seq":970,"reported_epoch":464,"state":"active+clean","last_fresh":"2024-03-11T21:46:16.292530+0000","last_change":"2024-03-11T21:45:25.649612+0000","last_active":"2024-03-11T21:46:16.292530+0000","last_peered":"2024-03-11T21:46:16.292530+0000","last_clean":"2024-03-11T21:46:16.292530+0000","last_became_active":"2024-03-11T21:44:21.465719+0000","last_became_peered":"2024-03-11T21:44:21.465719+0000","last_unstale":"2024-03-11T21:46:16.292530+0000","last_undegraded":"2024-03-11T21:46:16.292530+0000","last_fullsized":"2024-03-11T21:46:16.292530+0000","mapping_epoch":432,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":433,"parent":"0.0","parent_split_bits":0,"last_scrub":"260'27","last_scrub_stamp":"2024-03-11T21:45:25.649486+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:39.525554+0000","last_clean_scrub_stamp":"2024-03-11T21:45:25.649486+0000","objects_scrubbed":7,"log_size":39,"log_dups_size":0,"ondisk_log_size":39,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:46:45.351344+0000","scrub_duration":36,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":62,"num_read_kb":53,"num_write":40,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,0],"acting":[2,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.1d","version":"0'0","reported_seq":857,"reported_epoch":465,"state":"active+clean","last_fresh":"2024-03-11T21:46:18.565070+0000","last_change":"2024-03-11T21:45:23.007631+0000","last_active":"2024-03-11T21:46:18.565070+0000","last_peered":"2024-03-11T21:46:18.565070+0000","last_clean":"2024-03-11T21:46:18.565070+0000","last_became_active":"2024-03-11T21:44:35.622545+0000","last_became_peered":"2024-03-11T21:44:35.622545+0000","last_unstale":"2024-03-11T21:46:18.565070+0000","last_undegraded":"2024-03-11T21:46:18.565070+0000","last_fullsized":"2024-03-11T21:46:18.565070+0000","mapping_epoch":437,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":438,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:45:23.007509+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:43.559452+0000","last_clean_scrub_stamp":"2024-03-11T21:45:23.007509+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:46:36.165568+0000","scrub_duration":106,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,2],"acting":[1,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.1e","version":"0'0","reported_seq":873,"reported_epoch":465,"state":"active+clean","last_fresh":"2024-03-11T21:46:18.568268+0000","last_change":"2024-03-11T21:46:15.490825+0000","last_active":"2024-03-11T21:46:18.568268+0000","last_peered":"2024-03-11T21:46:18.568268+0000","last_clean":"2024-03-11T21:46:18.568268+0000","last_became_active":"2024-03-11T21:25:42.582959+0000","last_became_peered":"2024-03-11T21:25:42.582959+0000","last_unstale":"2024-03-11T21:46:18.568268+0000","last_undegraded":"2024-03-11T21:46:18.568268+0000","last_fullsized":"2024-03-11T21:46:18.568268+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:46:15.490533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:44:28.222102+0000","last_clean_scrub_stamp":"2024-03-11T21:46:15.490533+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:47:40.547908+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.19","version":"67'2","reported_seq":152,"reported_epoch":464,"state":"active+clean","last_fresh":"2024-03-11T21:46:16.288785+0000","last_change":"2024-03-11T21:44:30.400531+0000","last_active":"2024-03-11T21:46:16.288785+0000","last_peered":"2024-03-11T21:46:16.288785+0000","last_clean":"2024-03-11T21:46:16.288785+0000","last_became_active":"2024-03-11T21:44:21.454547+0000","last_became_peered":"2024-03-11T21:44:21.454547+0000","last_unstale":"2024-03-11T21:46:16.288785+0000","last_undegraded":"2024-03-11T21:46:16.288785+0000","last_fullsized":"2024-03-11T21:46:16.288785+0000","mapping_epoch":432,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":433,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'2","last_scrub_stamp":"2024-03-11T21:44:30.400408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:37.512018+0000","last_clean_scrub_stamp":"2024-03-11T21:44:30.400408+0000","objects_scrubbed":2,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":957,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":957,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,2],"acting":[6,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.1f","version":"456'32","reported_seq":956,"reported_epoch":464,"state":"active+clean","last_fresh":"2024-03-11T21:46:16.737348+0000","last_change":"2024-03-11T21:46:16.737415+0000","last_active":"2024-03-11T21:46:16.737348+0000","last_peered":"2024-03-11T21:46:16.737348+0000","last_clean":"2024-03-11T21:46:16.737348+0000","last_became_active":"2024-03-11T21:32:22.798242+0000","last_became_peered":"2024-03-11T21:32:22.798242+0000","last_unstale":"2024-03-11T21:46:16.737348+0000","last_undegraded":"2024-03-11T21:46:16.737348+0000","last_fullsized":"2024-03-11T21:46:16.737348+0000","mapping_epoch":166,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":167,"parent":"0.0","parent_split_bits":0,"last_scrub":"456'32","last_scrub_stamp":"2024-03-11T21:46:16.737252+0000","last_deep_scrub":"124'16","last_deep_scrub_stamp":"2024-03-11T21:35:38.173841+0000","last_clean_scrub_stamp":"2024-03-11T21:46:16.737252+0000","objects_scrubbed":5,"log_size":32,"log_dups_size":0,"ondisk_log_size":32,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:47:17.517883+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":44,"num_read_kb":37,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,0],"acting":[6,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.1c","version":"74'1","reported_seq":509,"reported_epoch":464,"state":"active+clean","last_fresh":"2024-03-11T21:46:16.301416+0000","last_change":"2024-03-11T21:43:20.543062+0000","last_active":"2024-03-11T21:46:16.301416+0000","last_peered":"2024-03-11T21:46:16.301416+0000","last_clean":"2024-03-11T21:46:16.301416+0000","last_became_active":"2024-03-11T21:34:41.144839+0000","last_became_peered":"2024-03-11T21:34:41.144839+0000","last_unstale":"2024-03-11T21:46:16.301416+0000","last_undegraded":"2024-03-11T21:46:16.301416+0000","last_fullsized":"2024-03-11T21:46:16.301416+0000","mapping_epoch":226,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":227,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'1","last_scrub_stamp":"2024-03-11T21:43:20.542918+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:43.559452+0000","last_clean_scrub_stamp":"2024-03-11T21:43:20.542918+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":370,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":370,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.1f","version":"438'11","reported_seq":1176,"reported_epoch":464,"state":"active+clean","last_fresh":"2024-03-11T21:46:16.289942+0000","last_change":"2024-03-11T21:44:35.663917+0000","last_active":"2024-03-11T21:46:16.289942+0000","last_peered":"2024-03-11T21:46:16.289942+0000","last_clean":"2024-03-11T21:46:16.289942+0000","last_became_active":"2024-03-11T21:44:35.647069+0000","last_became_peered":"2024-03-11T21:44:35.647069+0000","last_unstale":"2024-03-11T21:46:16.289942+0000","last_undegraded":"2024-03-11T21:46:16.289942+0000","last_fullsized":"2024-03-11T21:46:16.289942+0000","mapping_epoch":437,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":438,"parent":"0.0","parent_split_bits":0,"last_scrub":"229'10","last_scrub_stamp":"2024-03-11T21:44:32.375857+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:41.544550+0000","last_clean_scrub_stamp":"2024-03-11T21:44:32.375857+0000","objects_scrubbed":1,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,2],"acting":[6,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.18","version":"67'1","reported_seq":922,"reported_epoch":464,"state":"active+clean","last_fresh":"2024-03-11T21:46:16.299222+0000","last_change":"2024-03-11T21:45:41.138454+0000","last_active":"2024-03-11T21:46:16.299222+0000","last_peered":"2024-03-11T21:46:16.299222+0000","last_clean":"2024-03-11T21:46:16.299222+0000","last_became_active":"2024-03-11T21:44:35.610875+0000","last_became_peered":"2024-03-11T21:44:35.610875+0000","last_unstale":"2024-03-11T21:46:16.299222+0000","last_undegraded":"2024-03-11T21:46:16.299222+0000","last_fullsized":"2024-03-11T21:46:16.299222+0000","mapping_epoch":437,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":438,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'1","last_scrub_stamp":"2024-03-11T21:45:41.138228+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:37.512018+0000","last_clean_scrub_stamp":"2024-03-11T21:45:41.138228+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:47:04.898414+0000","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":436,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":7,"num_read_kb":7,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1744,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,6],"acting":[5,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1e","version":"456'30","reported_seq":990,"reported_epoch":465,"state":"active+clean","last_fresh":"2024-03-11T21:46:18.568183+0000","last_change":"2024-03-11T21:45:15.025938+0000","last_active":"2024-03-11T21:46:18.568183+0000","last_peered":"2024-03-11T21:46:18.568183+0000","last_clean":"2024-03-11T21:46:18.568183+0000","last_became_active":"2024-03-11T21:44:21.467126+0000","last_became_peered":"2024-03-11T21:44:21.467126+0000","last_unstale":"2024-03-11T21:46:18.568183+0000","last_undegraded":"2024-03-11T21:46:18.568183+0000","last_fullsized":"2024-03-11T21:46:18.568183+0000","mapping_epoch":432,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":433,"parent":"0.0","parent_split_bits":0,"last_scrub":"260'20","last_scrub_stamp":"2024-03-11T21:45:15.025805+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:39.525554+0000","last_clean_scrub_stamp":"2024-03-11T21:45:15.025805+0000","objects_scrubbed":5,"log_size":30,"log_dups_size":0,"ondisk_log_size":30,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":45,"num_read_kb":40,"num_write":30,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,1],"acting":[0,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.13","version":"0'0","reported_seq":424,"reported_epoch":465,"state":"active+clean","last_fresh":"2024-03-11T21:46:18.566641+0000","last_change":"2024-03-11T21:45:17.252669+0000","last_active":"2024-03-11T21:46:18.566641+0000","last_peered":"2024-03-11T21:46:18.566641+0000","last_clean":"2024-03-11T21:46:18.566641+0000","last_became_active":"2024-03-11T21:44:21.465436+0000","last_became_peered":"2024-03-11T21:44:21.465436+0000","last_unstale":"2024-03-11T21:46:18.566641+0000","last_undegraded":"2024-03-11T21:46:18.566641+0000","last_fullsized":"2024-03-11T21:46:18.566641+0000","mapping_epoch":432,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":433,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:45:17.252520+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:43.559452+0000","last_clean_scrub_stamp":"2024-03-11T21:45:17.252520+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:46:44.935941+0000","scrub_duration":220,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,2],"acting":[0,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":519,"reported_epoch":464,"state":"active+clean","last_fresh":"2024-03-11T21:46:16.299743+0000","last_change":"2024-03-11T21:44:35.628159+0000","last_active":"2024-03-11T21:46:16.299743+0000","last_peered":"2024-03-11T21:46:16.299743+0000","last_clean":"2024-03-11T21:46:16.299743+0000","last_became_active":"2024-03-11T21:44:35.627685+0000","last_became_peered":"2024-03-11T21:44:35.627685+0000","last_unstale":"2024-03-11T21:46:16.299743+0000","last_undegraded":"2024-03-11T21:46:16.299743+0000","last_fullsized":"2024-03-11T21:46:16.299743+0000","mapping_epoch":437,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":438,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:43:26.665719+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:43:26.665719+0000","last_clean_scrub_stamp":"2024-03-11T21:43:26.665719+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,0],"acting":[7,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.17","version":"0'0","reported_seq":885,"reported_epoch":464,"state":"active+clean","last_fresh":"2024-03-11T21:46:16.289845+0000","last_change":"2024-03-11T21:45:30.741004+0000","last_active":"2024-03-11T21:46:16.289845+0000","last_peered":"2024-03-11T21:46:16.289845+0000","last_clean":"2024-03-11T21:46:16.289845+0000","last_became_active":"2024-03-11T21:25:38.540349+0000","last_became_peered":"2024-03-11T21:25:38.540349+0000","last_unstale":"2024-03-11T21:46:16.289845+0000","last_undegraded":"2024-03-11T21:46:16.289845+0000","last_fullsized":"2024-03-11T21:46:16.289845+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:45:30.740865+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:45:30.740865+0000","last_clean_scrub_stamp":"2024-03-11T21:45:30.740865+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:46:50.398792+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,2],"acting":[6,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.11","version":"456'27","reported_seq":551,"reported_epoch":464,"state":"active+clean","last_fresh":"2024-03-11T21:46:16.300401+0000","last_change":"2024-03-11T21:44:21.473463+0000","last_active":"2024-03-11T21:46:16.300401+0000","last_peered":"2024-03-11T21:46:16.300401+0000","last_clean":"2024-03-11T21:46:16.300401+0000","last_became_active":"2024-03-11T21:44:21.456218+0000","last_became_peered":"2024-03-11T21:44:21.456218+0000","last_unstale":"2024-03-11T21:46:16.300401+0000","last_undegraded":"2024-03-11T21:46:16.300401+0000","last_fullsized":"2024-03-11T21:46:16.300401+0000","mapping_epoch":432,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":433,"parent":"0.0","parent_split_bits":0,"last_scrub":"260'19","last_scrub_stamp":"2024-03-11T21:43:29.687897+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:39.525554+0000","last_clean_scrub_stamp":"2024-03-11T21:43:29.687897+0000","objects_scrubbed":5,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":44,"num_read_kb":37,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,5],"acting":[7,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":519,"reported_epoch":464,"state":"active+clean","last_fresh":"2024-03-11T21:46:16.300281+0000","last_change":"2024-03-11T21:44:40.464549+0000","last_active":"2024-03-11T21:46:16.300281+0000","last_peered":"2024-03-11T21:46:16.300281+0000","last_clean":"2024-03-11T21:46:16.300281+0000","last_became_active":"2024-03-11T21:44:35.634181+0000","last_became_peered":"2024-03-11T21:44:35.634181+0000","last_unstale":"2024-03-11T21:46:16.300281+0000","last_undegraded":"2024-03-11T21:46:16.300281+0000","last_fullsized":"2024-03-11T21:46:16.300281+0000","mapping_epoch":437,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":438,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:44:40.464467+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:43.559452+0000","last_clean_scrub_stamp":"2024-03-11T21:44:40.464467+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,1],"acting":[7,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":903,"reported_epoch":464,"state":"active+clean","last_fresh":"2024-03-11T21:46:16.288856+0000","last_change":"2024-03-11T21:45:39.613711+0000","last_active":"2024-03-11T21:46:16.288856+0000","last_peered":"2024-03-11T21:46:16.288856+0000","last_clean":"2024-03-11T21:46:16.288856+0000","last_became_active":"2024-03-11T21:44:35.630991+0000","last_became_peered":"2024-03-11T21:44:35.630991+0000","last_unstale":"2024-03-11T21:46:16.288856+0000","last_undegraded":"2024-03-11T21:46:16.288856+0000","last_fullsized":"2024-03-11T21:46:16.288856+0000","mapping_epoch":437,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":438,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:45:39.613600+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:43:29.425414+0000","last_clean_scrub_stamp":"2024-03-11T21:45:39.613600+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:46:43.821480+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,0],"acting":[6,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.16","version":"0'0","reported_seq":876,"reported_epoch":464,"state":"active+clean","last_fresh":"2024-03-11T21:46:16.303312+0000","last_change":"2024-03-11T21:45:24.125090+0000","last_active":"2024-03-11T21:46:16.303312+0000","last_peered":"2024-03-11T21:46:16.303312+0000","last_clean":"2024-03-11T21:46:16.303312+0000","last_became_active":"2024-03-11T21:25:38.539103+0000","last_became_peered":"2024-03-11T21:25:38.539103+0000","last_unstale":"2024-03-11T21:46:16.303312+0000","last_undegraded":"2024-03-11T21:46:16.303312+0000","last_fullsized":"2024-03-11T21:46:16.303312+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:45:24.124964+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:27:11.938909+0000","last_clean_scrub_stamp":"2024-03-11T21:45:24.124964+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:46:24.482335+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,2],"acting":[5,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.10","version":"456'14","reported_seq":238,"reported_epoch":465,"state":"active+clean","last_fresh":"2024-03-11T21:46:18.566046+0000","last_change":"2024-03-11T21:44:35.664761+0000","last_active":"2024-03-11T21:46:18.566046+0000","last_peered":"2024-03-11T21:46:18.566046+0000","last_clean":"2024-03-11T21:46:18.566046+0000","last_became_active":"2024-03-11T21:44:35.620127+0000","last_became_peered":"2024-03-11T21:44:35.620127+0000","last_unstale":"2024-03-11T21:46:18.566046+0000","last_undegraded":"2024-03-11T21:46:18.566046+0000","last_fullsized":"2024-03-11T21:46:18.566046+0000","mapping_epoch":437,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":438,"parent":"0.0","parent_split_bits":0,"last_scrub":"260'10","last_scrub_stamp":"2024-03-11T21:44:32.059058+0000","last_deep_scrub":"260'10","last_deep_scrub_stamp":"2024-03-11T21:44:32.059058+0000","last_clean_scrub_stamp":"2024-03-11T21:44:32.059058+0000","objects_scrubbed":2,"log_size":14,"log_dups_size":0,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":18,"num_read_kb":16,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,5],"acting":[1,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":477,"reported_epoch":465,"state":"active+clean","last_fresh":"2024-03-11T21:46:18.569979+0000","last_change":"2024-03-11T21:45:36.298747+0000","last_active":"2024-03-11T21:46:18.569979+0000","last_peered":"2024-03-11T21:46:18.569979+0000","last_clean":"2024-03-11T21:46:18.569979+0000","last_became_active":"2024-03-11T21:44:21.454437+0000","last_became_peered":"2024-03-11T21:44:21.454437+0000","last_unstale":"2024-03-11T21:46:18.569979+0000","last_undegraded":"2024-03-11T21:46:18.569979+0000","last_fullsized":"2024-03-11T21:46:18.569979+0000","mapping_epoch":432,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":433,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:45:36.298635+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:43.559452+0000","last_clean_scrub_stamp":"2024-03-11T21:45:36.298635+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:47:04.798656+0000","scrub_duration":110,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":870,"reported_epoch":465,"state":"active+clean","last_fresh":"2024-03-11T21:46:18.565490+0000","last_change":"2024-03-11T21:45:13.788851+0000","last_active":"2024-03-11T21:46:18.565490+0000","last_peered":"2024-03-11T21:46:18.565490+0000","last_clean":"2024-03-11T21:46:18.565490+0000","last_became_active":"2024-03-11T21:44:21.464563+0000","last_became_peered":"2024-03-11T21:44:21.464563+0000","last_unstale":"2024-03-11T21:46:18.565490+0000","last_undegraded":"2024-03-11T21:46:18.565490+0000","last_fullsized":"2024-03-11T21:46:18.565490+0000","mapping_epoch":432,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":433,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:45:13.788601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:27:21.207303+0000","last_clean_scrub_stamp":"2024-03-11T21:45:13.788601+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,7],"acting":[1,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.15","version":"0'0","reported_seq":871,"reported_epoch":465,"state":"active+clean","last_fresh":"2024-03-11T21:46:18.565326+0000","last_change":"2024-03-11T21:45:41.898361+0000","last_active":"2024-03-11T21:46:18.565326+0000","last_peered":"2024-03-11T21:46:18.565326+0000","last_clean":"2024-03-11T21:46:18.565326+0000","last_became_active":"2024-03-11T21:25:38.541705+0000","last_became_peered":"2024-03-11T21:25:38.541705+0000","last_unstale":"2024-03-11T21:46:18.565326+0000","last_undegraded":"2024-03-11T21:46:18.565326+0000","last_fullsized":"2024-03-11T21:46:18.565326+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:45:41.898237+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:26:44.860499+0000","last_clean_scrub_stamp":"2024-03-11T21:45:41.898237+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:46:44.236840+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.13","version":"456'32","reported_seq":585,"reported_epoch":464,"state":"active+clean","last_fresh":"2024-03-11T21:46:16.289308+0000","last_change":"2024-03-11T21:46:13.717766+0000","last_active":"2024-03-11T21:46:16.289308+0000","last_peered":"2024-03-11T21:46:16.289308+0000","last_clean":"2024-03-11T21:46:16.289308+0000","last_became_active":"2024-03-11T21:44:35.639494+0000","last_became_peered":"2024-03-11T21:44:35.639494+0000","last_unstale":"2024-03-11T21:46:16.289308+0000","last_undegraded":"2024-03-11T21:46:16.289308+0000","last_fullsized":"2024-03-11T21:46:16.289308+0000","mapping_epoch":437,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":438,"parent":"0.0","parent_split_bits":0,"last_scrub":"456'32","last_scrub_stamp":"2024-03-11T21:46:13.717488+0000","last_deep_scrub":"260'24","last_deep_scrub_stamp":"2024-03-11T21:40:25.595220+0000","last_clean_scrub_stamp":"2024-03-11T21:46:13.717488+0000","objects_scrubbed":5,"log_size":32,"log_dups_size":0,"ondisk_log_size":32,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:47:31.211551+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":43,"num_read_kb":36,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,5],"acting":[6,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.10","version":"0'0","reported_seq":859,"reported_epoch":465,"state":"active+clean","last_fresh":"2024-03-11T21:46:18.564377+0000","last_change":"2024-03-11T21:45:35.180073+0000","last_active":"2024-03-11T21:46:18.564377+0000","last_peered":"2024-03-11T21:46:18.564377+0000","last_clean":"2024-03-11T21:46:18.564377+0000","last_became_active":"2024-03-11T21:25:44.591189+0000","last_became_peered":"2024-03-11T21:25:44.591189+0000","last_unstale":"2024-03-11T21:46:18.564377+0000","last_undegraded":"2024-03-11T21:46:18.564377+0000","last_fullsized":"2024-03-11T21:46:18.564377+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:45:35.179951+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:27:25.417450+0000","last_clean_scrub_stamp":"2024-03-11T21:45:35.179951+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:46:43.624707+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,1],"acting":[0,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":467,"reported_epoch":465,"state":"active+clean","last_fresh":"2024-03-11T21:46:18.566748+0000","last_change":"2024-03-11T21:44:21.465764+0000","last_active":"2024-03-11T21:46:18.566748+0000","last_peered":"2024-03-11T21:46:18.566748+0000","last_clean":"2024-03-11T21:46:18.566748+0000","last_became_active":"2024-03-11T21:44:21.464968+0000","last_became_peered":"2024-03-11T21:44:21.464968+0000","last_unstale":"2024-03-11T21:46:18.566748+0000","last_undegraded":"2024-03-11T21:46:18.566748+0000","last_fullsized":"2024-03-11T21:46:18.566748+0000","mapping_epoch":432,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":433,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:44:19.987656+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:31:15.367898+0000","last_clean_scrub_stamp":"2024-03-11T21:44:19.987656+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,5],"acting":[0,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.14","version":"0'0","reported_seq":908,"reported_epoch":464,"state":"active+clean","last_fresh":"2024-03-11T21:46:16.289026+0000","last_change":"2024-03-11T21:44:34.456355+0000","last_active":"2024-03-11T21:46:16.289026+0000","last_peered":"2024-03-11T21:46:16.289026+0000","last_clean":"2024-03-11T21:46:16.289026+0000","last_became_active":"2024-03-11T21:44:21.449166+0000","last_became_peered":"2024-03-11T21:44:21.449166+0000","last_unstale":"2024-03-11T21:46:16.289026+0000","last_undegraded":"2024-03-11T21:46:16.289026+0000","last_fullsized":"2024-03-11T21:46:16.289026+0000","mapping_epoch":432,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":433,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:44:34.456276+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:37.512018+0000","last_clean_scrub_stamp":"2024-03-11T21:44:34.456276+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,7],"acting":[6,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.12","version":"456'21","reported_seq":901,"reported_epoch":465,"state":"active+clean","last_fresh":"2024-03-11T21:46:18.565284+0000","last_change":"2024-03-11T21:45:23.938336+0000","last_active":"2024-03-11T21:46:18.565284+0000","last_peered":"2024-03-11T21:46:18.565284+0000","last_clean":"2024-03-11T21:46:18.565284+0000","last_became_active":"2024-03-11T21:25:40.547082+0000","last_became_peered":"2024-03-11T21:25:40.547082+0000","last_unstale":"2024-03-11T21:46:18.565284+0000","last_undegraded":"2024-03-11T21:46:18.565284+0000","last_fullsized":"2024-03-11T21:46:18.565284+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"260'15","last_scrub_stamp":"2024-03-11T21:45:23.938207+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:39.525554+0000","last_clean_scrub_stamp":"2024-03-11T21:45:23.938207+0000","objects_scrubbed":4,"log_size":21,"log_dups_size":0,"ondisk_log_size":21,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:46:40.936707+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":35,"num_read_kb":29,"num_write":22,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":72,"reported_epoch":464,"state":"active+clean","last_fresh":"2024-03-11T21:46:16.293301+0000","last_change":"2024-03-11T21:44:39.546979+0000","last_active":"2024-03-11T21:46:16.293301+0000","last_peered":"2024-03-11T21:46:16.293301+0000","last_clean":"2024-03-11T21:46:16.293301+0000","last_became_active":"2024-03-11T21:44:35.613735+0000","last_became_peered":"2024-03-11T21:44:35.613735+0000","last_unstale":"2024-03-11T21:46:16.293301+0000","last_undegraded":"2024-03-11T21:46:16.293301+0000","last_fullsized":"2024-03-11T21:46:16.293301+0000","mapping_epoch":437,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":438,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:44:39.546881+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:27:09.801138+0000","last_clean_scrub_stamp":"2024-03-11T21:44:39.546881+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":112,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,0],"acting":[2,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.14","version":"434'13","reported_seq":650,"reported_epoch":464,"state":"active+clean","last_fresh":"2024-03-11T21:46:16.300351+0000","last_change":"2024-03-11T21:44:32.424885+0000","last_active":"2024-03-11T21:46:16.300351+0000","last_peered":"2024-03-11T21:46:16.300351+0000","last_clean":"2024-03-11T21:46:16.300351+0000","last_became_active":"2024-03-11T21:44:21.465719+0000","last_became_peered":"2024-03-11T21:44:21.465719+0000","last_unstale":"2024-03-11T21:46:16.300351+0000","last_undegraded":"2024-03-11T21:46:16.300351+0000","last_fullsized":"2024-03-11T21:46:16.300351+0000","mapping_epoch":432,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":433,"parent":"0.0","parent_split_bits":0,"last_scrub":"434'13","last_scrub_stamp":"2024-03-11T21:44:32.424808+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:41.544550+0000","last_clean_scrub_stamp":"2024-03-11T21:44:32.424808+0000","objects_scrubbed":1,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,1],"acting":[7,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.13","version":"0'0","reported_seq":892,"reported_epoch":465,"state":"active+clean","last_fresh":"2024-03-11T21:46:18.567197+0000","last_change":"2024-03-11T21:46:05.444752+0000","last_active":"2024-03-11T21:46:18.567197+0000","last_peered":"2024-03-11T21:46:18.567197+0000","last_clean":"2024-03-11T21:46:18.567197+0000","last_became_active":"2024-03-11T21:44:35.625399+0000","last_became_peered":"2024-03-11T21:44:35.625399+0000","last_unstale":"2024-03-11T21:46:18.567197+0000","last_undegraded":"2024-03-11T21:46:18.567197+0000","last_fullsized":"2024-03-11T21:46:18.567197+0000","mapping_epoch":437,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":438,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:46:05.444630+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:37.512018+0000","last_clean_scrub_stamp":"2024-03-11T21:46:05.444630+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:47:18.463855+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,5],"acting":[0,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.15","version":"456'25","reported_seq":968,"reported_epoch":464,"state":"active+clean","last_fresh":"2024-03-11T21:46:16.303272+0000","last_change":"2024-03-11T21:45:22.105065+0000","last_active":"2024-03-11T21:46:16.303272+0000","last_peered":"2024-03-11T21:46:16.303272+0000","last_clean":"2024-03-11T21:46:16.303272+0000","last_became_active":"2024-03-11T21:44:21.451878+0000","last_became_peered":"2024-03-11T21:44:21.451878+0000","last_unstale":"2024-03-11T21:46:16.303272+0000","last_undegraded":"2024-03-11T21:46:16.303272+0000","last_fullsized":"2024-03-11T21:46:16.303272+0000","mapping_epoch":432,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":433,"parent":"0.0","parent_split_bits":0,"last_scrub":"260'19","last_scrub_stamp":"2024-03-11T21:45:22.104952+0000","last_deep_scrub":"94'13","last_deep_scrub_stamp":"2024-03-11T21:29:41.853158+0000","last_clean_scrub_stamp":"2024-03-11T21:45:22.104952+0000","objects_scrubbed":4,"log_size":25,"log_dups_size":0,"ondisk_log_size":25,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:46:34.695431+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":35,"num_read_kb":29,"num_write":22,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.16","version":"0'0","reported_seq":900,"reported_epoch":465,"state":"active+clean","last_fresh":"2024-03-11T21:46:18.567634+0000","last_change":"2024-03-11T21:46:18.525995+0000","last_active":"2024-03-11T21:46:18.567634+0000","last_peered":"2024-03-11T21:46:18.567634+0000","last_clean":"2024-03-11T21:46:18.567634+0000","last_became_active":"2024-03-11T21:44:35.626259+0000","last_became_peered":"2024-03-11T21:44:35.626259+0000","last_unstale":"2024-03-11T21:46:18.567634+0000","last_undegraded":"2024-03-11T21:46:18.567634+0000","last_fullsized":"2024-03-11T21:46:18.567634+0000","mapping_epoch":437,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":438,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:46:18.525893+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:40:30.249679+0000","last_clean_scrub_stamp":"2024-03-11T21:46:18.525893+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:47:33.632097+0000","scrub_duration":105,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,5],"acting":[0,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.15","version":"98'9","reported_seq":1120,"reported_epoch":464,"state":"active+clean","last_fresh":"2024-03-11T21:46:16.303418+0000","last_change":"2024-03-11T21:46:08.945719+0000","last_active":"2024-03-11T21:46:16.303418+0000","last_peered":"2024-03-11T21:46:16.303418+0000","last_clean":"2024-03-11T21:46:16.303418+0000","last_became_active":"2024-03-11T21:25:42.572555+0000","last_became_peered":"2024-03-11T21:25:42.572555+0000","last_unstale":"2024-03-11T21:46:16.303418+0000","last_undegraded":"2024-03-11T21:46:16.303418+0000","last_fullsized":"2024-03-11T21:46:16.303418+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"98'9","last_scrub_stamp":"2024-03-11T21:46:08.945626+0000","last_deep_scrub":"98'9","last_deep_scrub_stamp":"2024-03-11T21:31:20.624923+0000","last_clean_scrub_stamp":"2024-03-11T21:46:08.945626+0000","objects_scrubbed":1,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:47:10.318739+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.12","version":"0'0","reported_seq":895,"reported_epoch":464,"state":"active+clean","last_fresh":"2024-03-11T21:46:16.299817+0000","last_change":"2024-03-11T21:45:23.087680+0000","last_active":"2024-03-11T21:46:16.299817+0000","last_peered":"2024-03-11T21:46:16.299817+0000","last_clean":"2024-03-11T21:46:16.299817+0000","last_became_active":"2024-03-11T21:44:21.454917+0000","last_became_peered":"2024-03-11T21:44:21.454917+0000","last_unstale":"2024-03-11T21:46:16.299817+0000","last_undegraded":"2024-03-11T21:46:16.299817+0000","last_fullsized":"2024-03-11T21:46:16.299817+0000","mapping_epoch":432,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":433,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:45:23.087568+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:37.512018+0000","last_clean_scrub_stamp":"2024-03-11T21:45:23.087568+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:46:28.586614+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,0],"acting":[5,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.14","version":"456'30","reported_seq":214,"reported_epoch":465,"state":"active+clean","last_fresh":"2024-03-11T21:46:18.566095+0000","last_change":"2024-03-11T21:44:22.015799+0000","last_active":"2024-03-11T21:46:18.566095+0000","last_peered":"2024-03-11T21:46:18.566095+0000","last_clean":"2024-03-11T21:46:18.566095+0000","last_became_active":"2024-03-11T21:44:21.457852+0000","last_became_peered":"2024-03-11T21:44:21.457852+0000","last_unstale":"2024-03-11T21:46:18.566095+0000","last_undegraded":"2024-03-11T21:46:18.566095+0000","last_fullsized":"2024-03-11T21:46:18.566095+0000","mapping_epoch":432,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":433,"parent":"0.0","parent_split_bits":0,"last_scrub":"260'20","last_scrub_stamp":"2024-03-11T21:44:22.015704+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:39.525554+0000","last_clean_scrub_stamp":"2024-03-11T21:44:22.015704+0000","objects_scrubbed":5,"log_size":30,"log_dups_size":0,"ondisk_log_size":30,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":44,"num_read_kb":39,"num_write":30,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,2],"acting":[1,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.15","version":"0'0","reported_seq":519,"reported_epoch":464,"state":"active+clean","last_fresh":"2024-03-11T21:46:16.301384+0000","last_change":"2024-03-11T21:46:07.457961+0000","last_active":"2024-03-11T21:46:16.301384+0000","last_peered":"2024-03-11T21:46:16.301384+0000","last_clean":"2024-03-11T21:46:16.301384+0000","last_became_active":"2024-03-11T21:44:35.639992+0000","last_became_peered":"2024-03-11T21:44:35.639992+0000","last_unstale":"2024-03-11T21:46:16.301384+0000","last_undegraded":"2024-03-11T21:46:16.301384+0000","last_fullsized":"2024-03-11T21:46:16.301384+0000","mapping_epoch":437,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":438,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:46:07.457807+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:43.559452+0000","last_clean_scrub_stamp":"2024-03-11T21:46:07.457807+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:47:29.190575+0000","scrub_duration":30,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,5],"acting":[7,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":875,"reported_epoch":464,"state":"active+clean","last_fresh":"2024-03-11T21:46:16.303218+0000","last_change":"2024-03-11T21:46:11.892376+0000","last_active":"2024-03-11T21:46:16.303218+0000","last_peered":"2024-03-11T21:46:16.303218+0000","last_clean":"2024-03-11T21:46:16.303218+0000","last_became_active":"2024-03-11T21:44:21.453303+0000","last_became_peered":"2024-03-11T21:44:21.453303+0000","last_unstale":"2024-03-11T21:46:16.303218+0000","last_undegraded":"2024-03-11T21:46:16.303218+0000","last_fullsized":"2024-03-11T21:46:16.303218+0000","mapping_epoch":432,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":433,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:46:11.892240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:43:36.585632+0000","last_clean_scrub_stamp":"2024-03-11T21:46:11.892240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:47:26.504334+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.11","version":"0'0","reported_seq":891,"reported_epoch":464,"state":"active+clean","last_fresh":"2024-03-11T21:46:16.288917+0000","last_change":"2024-03-11T21:46:06.706885+0000","last_active":"2024-03-11T21:46:16.288917+0000","last_peered":"2024-03-11T21:46:16.288917+0000","last_clean":"2024-03-11T21:46:16.288917+0000","last_became_active":"2024-03-11T21:44:35.620193+0000","last_became_peered":"2024-03-11T21:44:35.620193+0000","last_unstale":"2024-03-11T21:46:16.288917+0000","last_undegraded":"2024-03-11T21:46:16.288917+0000","last_fullsized":"2024-03-11T21:46:16.288917+0000","mapping_epoch":437,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":438,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:46:06.705444+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:37.512018+0000","last_clean_scrub_stamp":"2024-03-11T21:46:06.705444+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:47:22.561157+0000","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,5],"acting":[6,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.17","version":"456'21","reported_seq":552,"reported_epoch":464,"state":"active+clean","last_fresh":"2024-03-11T21:46:16.302136+0000","last_change":"2024-03-11T21:44:39.501411+0000","last_active":"2024-03-11T21:46:16.302136+0000","last_peered":"2024-03-11T21:46:16.302136+0000","last_clean":"2024-03-11T21:46:16.302136+0000","last_became_active":"2024-03-11T21:44:21.465960+0000","last_became_peered":"2024-03-11T21:44:21.465960+0000","last_unstale":"2024-03-11T21:46:16.302136+0000","last_undegraded":"2024-03-11T21:46:16.302136+0000","last_fullsized":"2024-03-11T21:46:16.302136+0000","mapping_epoch":432,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":433,"parent":"0.0","parent_split_bits":0,"last_scrub":"260'15","last_scrub_stamp":"2024-03-11T21:44:39.501274+0000","last_deep_scrub":"97'9","last_deep_scrub_stamp":"2024-03-11T21:27:25.803132+0000","last_clean_scrub_stamp":"2024-03-11T21:44:39.501274+0000","objects_scrubbed":3,"log_size":21,"log_dups_size":0,"ondisk_log_size":21,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":27,"num_read_kb":24,"num_write":18,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.14","version":"0'0","reported_seq":881,"reported_epoch":464,"state":"active+clean","last_fresh":"2024-03-11T21:46:16.292082+0000","last_change":"2024-03-11T21:45:41.530744+0000","last_active":"2024-03-11T21:46:16.292082+0000","last_peered":"2024-03-11T21:46:16.292082+0000","last_clean":"2024-03-11T21:46:16.292082+0000","last_became_active":"2024-03-11T21:44:35.626384+0000","last_became_peered":"2024-03-11T21:44:35.626384+0000","last_unstale":"2024-03-11T21:46:16.292082+0000","last_undegraded":"2024-03-11T21:46:16.292082+0000","last_fullsized":"2024-03-11T21:46:16.292082+0000","mapping_epoch":437,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":438,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:45:41.530627+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:43.559452+0000","last_clean_scrub_stamp":"2024-03-11T21:45:41.530627+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:47:07.186472+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,0],"acting":[2,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":498,"reported_epoch":465,"state":"active+clean","last_fresh":"2024-03-11T21:46:18.566947+0000","last_change":"2024-03-11T21:45:36.877859+0000","last_active":"2024-03-11T21:46:18.566947+0000","last_peered":"2024-03-11T21:46:18.566947+0000","last_clean":"2024-03-11T21:46:18.566947+0000","last_became_active":"2024-03-11T21:44:21.458758+0000","last_became_peered":"2024-03-11T21:44:21.458758+0000","last_unstale":"2024-03-11T21:46:18.566947+0000","last_undegraded":"2024-03-11T21:46:18.566947+0000","last_fullsized":"2024-03-11T21:46:18.566947+0000","mapping_epoch":432,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":433,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:45:36.877741+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:45:36.877741+0000","last_clean_scrub_stamp":"2024-03-11T21:45:36.877741+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:46:55.423743+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,0],"acting":[1,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.10","version":"0'0","reported_seq":875,"reported_epoch":464,"state":"active+clean","last_fresh":"2024-03-11T21:46:16.291700+0000","last_change":"2024-03-11T21:45:27.664632+0000","last_active":"2024-03-11T21:46:16.291700+0000","last_peered":"2024-03-11T21:46:16.291700+0000","last_clean":"2024-03-11T21:46:16.291700+0000","last_became_active":"2024-03-11T21:25:38.533138+0000","last_became_peered":"2024-03-11T21:25:38.533138+0000","last_unstale":"2024-03-11T21:46:16.291700+0000","last_undegraded":"2024-03-11T21:46:16.291700+0000","last_fullsized":"2024-03-11T21:46:16.291700+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:45:27.664509+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:37.512018+0000","last_clean_scrub_stamp":"2024-03-11T21:45:27.664509+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:46:35.729292+0000","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,0],"acting":[2,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.16","version":"456'25","reported_seq":971,"reported_epoch":465,"state":"active+clean","last_fresh":"2024-03-11T21:46:18.567998+0000","last_change":"2024-03-11T21:45:25.093159+0000","last_active":"2024-03-11T21:46:18.567998+0000","last_peered":"2024-03-11T21:46:18.567998+0000","last_clean":"2024-03-11T21:46:18.567998+0000","last_became_active":"2024-03-11T21:44:21.463330+0000","last_became_peered":"2024-03-11T21:44:21.463330+0000","last_unstale":"2024-03-11T21:46:18.567998+0000","last_undegraded":"2024-03-11T21:46:18.567998+0000","last_fullsized":"2024-03-11T21:46:18.567998+0000","mapping_epoch":432,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":433,"parent":"0.0","parent_split_bits":0,"last_scrub":"369'19","last_scrub_stamp":"2024-03-11T21:45:25.093077+0000","last_deep_scrub":"369'19","last_deep_scrub_stamp":"2024-03-11T21:41:22.115634+0000","last_clean_scrub_stamp":"2024-03-11T21:45:25.093077+0000","objects_scrubbed":4,"log_size":25,"log_dups_size":0,"ondisk_log_size":25,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:46:45.901184+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":35,"num_read_kb":29,"num_write":22,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.6","version":"465'7266","reported_seq":7314,"reported_epoch":465,"state":"active+clean","last_fresh":"2024-03-11T21:46:19.661588+0000","last_change":"2024-03-11T21:44:53.660589+0000","last_active":"2024-03-11T21:46:19.661588+0000","last_peered":"2024-03-11T21:46:19.661588+0000","last_clean":"2024-03-11T21:46:19.661588+0000","last_became_active":"2024-03-11T21:44:53.660330+0000","last_became_peered":"2024-03-11T21:44:53.660330+0000","last_unstale":"2024-03-11T21:46:19.661588+0000","last_undegraded":"2024-03-11T21:46:19.661588+0000","last_fullsized":"2024-03-11T21:46:19.661588+0000","mapping_epoch":446,"log_start":"454'3100","ondisk_log_start":"454'3100","created":446,"last_epoch_clean":447,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:44:52.644443+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:44:52.644443+0000","last_clean_scrub_stamp":"2024-03-11T21:44:52.644443+0000","objects_scrubbed":0,"log_size":4166,"log_dups_size":0,"ondisk_log_size":4166,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":476184576,"num_objects":7266,"num_object_clones":0,"num_object_copies":21798,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7266,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7266,"num_write_kb":465024,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.b","version":"0'0","reported_seq":484,"reported_epoch":464,"state":"active+clean","last_fresh":"2024-03-11T21:46:16.299628+0000","last_change":"2024-03-11T21:44:35.622604+0000","last_active":"2024-03-11T21:46:16.299628+0000","last_peered":"2024-03-11T21:46:16.299628+0000","last_clean":"2024-03-11T21:46:16.299628+0000","last_became_active":"2024-03-11T21:44:21.449857+0000","last_became_peered":"2024-03-11T21:44:21.449857+0000","last_unstale":"2024-03-11T21:46:16.299628+0000","last_undegraded":"2024-03-11T21:46:16.299628+0000","last_fullsized":"2024-03-11T21:46:16.299628+0000","mapping_epoch":432,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":433,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:44:35.622488+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:44:35.622488+0000","last_clean_scrub_stamp":"2024-03-11T21:44:35.622488+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":116,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,6],"acting":[7,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.f","version":"67'2","reported_seq":81,"reported_epoch":465,"state":"active+clean","last_fresh":"2024-03-11T21:46:18.568092+0000","last_change":"2024-03-11T21:44:35.650921+0000","last_active":"2024-03-11T21:46:18.568092+0000","last_peered":"2024-03-11T21:46:18.568092+0000","last_clean":"2024-03-11T21:46:18.568092+0000","last_became_active":"2024-03-11T21:44:35.627781+0000","last_became_peered":"2024-03-11T21:44:35.627781+0000","last_unstale":"2024-03-11T21:46:18.568092+0000","last_undegraded":"2024-03-11T21:46:18.568092+0000","last_fullsized":"2024-03-11T21:46:18.568092+0000","mapping_epoch":437,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":438,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'2","last_scrub_stamp":"2024-03-11T21:44:18.071546+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:37.512018+0000","last_clean_scrub_stamp":"2024-03-11T21:44:18.071546+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":92,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":92,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,5],"acting":[0,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.9","version":"456'29","reported_seq":554,"reported_epoch":465,"state":"active+clean","last_fresh":"2024-03-11T21:46:18.566054+0000","last_change":"2024-03-11T21:44:35.656384+0000","last_active":"2024-03-11T21:46:18.566054+0000","last_peered":"2024-03-11T21:46:18.566054+0000","last_clean":"2024-03-11T21:46:18.566054+0000","last_became_active":"2024-03-11T21:44:35.618968+0000","last_became_peered":"2024-03-11T21:44:35.618968+0000","last_unstale":"2024-03-11T21:46:18.566054+0000","last_undegraded":"2024-03-11T21:46:18.566054+0000","last_fullsized":"2024-03-11T21:46:18.566054+0000","mapping_epoch":437,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":438,"parent":"0.0","parent_split_bits":0,"last_scrub":"260'23","last_scrub_stamp":"2024-03-11T21:44:11.026427+0000","last_deep_scrub":"146'17","last_deep_scrub_stamp":"2024-03-11T21:34:26.765075+0000","last_clean_scrub_stamp":"2024-03-11T21:44:11.026427+0000","objects_scrubbed":5,"log_size":29,"log_dups_size":0,"ondisk_log_size":29,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":42,"num_read_kb":33,"num_write":26,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,0],"acting":[1,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":868,"reported_epoch":464,"state":"active+clean","last_fresh":"2024-03-11T21:46:16.291814+0000","last_change":"2024-03-11T21:45:33.761870+0000","last_active":"2024-03-11T21:46:16.291814+0000","last_peered":"2024-03-11T21:46:16.291814+0000","last_clean":"2024-03-11T21:46:16.291814+0000","last_became_active":"2024-03-11T21:25:42.573987+0000","last_became_peered":"2024-03-11T21:25:42.573987+0000","last_unstale":"2024-03-11T21:46:16.291814+0000","last_undegraded":"2024-03-11T21:46:16.291814+0000","last_fullsized":"2024-03-11T21:46:16.291814+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:45:33.761758+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:41.544550+0000","last_clean_scrub_stamp":"2024-03-11T21:45:33.761758+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:46:48.713635+0000","scrub_duration":17,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"11.d","version":"464'7242","reported_seq":7288,"reported_epoch":464,"state":"active+clean","last_fresh":"2024-03-11T21:46:18.359039+0000","last_change":"2024-03-11T21:44:53.663253+0000","last_active":"2024-03-11T21:46:18.359039+0000","last_peered":"2024-03-11T21:46:18.359039+0000","last_clean":"2024-03-11T21:46:18.359039+0000","last_became_active":"2024-03-11T21:44:53.662697+0000","last_became_peered":"2024-03-11T21:44:53.662697+0000","last_unstale":"2024-03-11T21:46:18.359039+0000","last_undegraded":"2024-03-11T21:46:18.359039+0000","last_fullsized":"2024-03-11T21:46:18.359039+0000","mapping_epoch":446,"log_start":"454'2800","ondisk_log_start":"454'2800","created":446,"last_epoch_clean":447,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:44:52.644443+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:44:52.644443+0000","last_clean_scrub_stamp":"2024-03-11T21:44:52.644443+0000","objects_scrubbed":0,"log_size":4442,"log_dups_size":0,"ondisk_log_size":4442,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":474611712,"num_objects":7242,"num_object_clones":0,"num_object_copies":21726,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7242,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7242,"num_write_kb":463488,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6,1],"acting":[2,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.0","version":"0'0","reported_seq":876,"reported_epoch":465,"state":"active+clean","last_fresh":"2024-03-11T21:46:18.564471+0000","last_change":"2024-03-11T21:46:13.470811+0000","last_active":"2024-03-11T21:46:18.564471+0000","last_peered":"2024-03-11T21:46:18.564471+0000","last_clean":"2024-03-11T21:46:18.564471+0000","last_became_active":"2024-03-11T21:44:21.464145+0000","last_became_peered":"2024-03-11T21:44:21.464145+0000","last_unstale":"2024-03-11T21:46:18.564471+0000","last_undegraded":"2024-03-11T21:46:18.564471+0000","last_fullsized":"2024-03-11T21:46:18.564471+0000","mapping_epoch":432,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":433,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:46:13.467417+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:29:39.498784+0000","last_clean_scrub_stamp":"2024-03-11T21:46:13.467417+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:47:26.511912+0000","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,5],"acting":[0,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":884,"reported_epoch":465,"state":"active+clean","last_fresh":"2024-03-11T21:46:18.565990+0000","last_change":"2024-03-11T21:45:27.951614+0000","last_active":"2024-03-11T21:46:18.565990+0000","last_peered":"2024-03-11T21:46:18.565990+0000","last_clean":"2024-03-11T21:46:18.565990+0000","last_became_active":"2024-03-11T21:34:41.150913+0000","last_became_peered":"2024-03-11T21:34:41.150913+0000","last_unstale":"2024-03-11T21:46:18.565990+0000","last_undegraded":"2024-03-11T21:46:18.565990+0000","last_fullsized":"2024-03-11T21:46:18.565990+0000","mapping_epoch":226,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":227,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:45:27.951397+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:37.512018+0000","last_clean_scrub_stamp":"2024-03-11T21:45:27.951397+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:46:54.727090+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,7],"acting":[1,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.2","version":"456'35","reported_seq":936,"reported_epoch":465,"state":"active+clean","last_fresh":"2024-03-11T21:46:18.565392+0000","last_change":"2024-03-11T21:44:35.651100+0000","last_active":"2024-03-11T21:46:18.565392+0000","last_peered":"2024-03-11T21:46:18.565392+0000","last_clean":"2024-03-11T21:46:18.565392+0000","last_became_active":"2024-03-11T21:44:35.618152+0000","last_became_peered":"2024-03-11T21:44:35.618152+0000","last_unstale":"2024-03-11T21:46:18.565392+0000","last_undegraded":"2024-03-11T21:46:18.565392+0000","last_fullsized":"2024-03-11T21:46:18.565392+0000","mapping_epoch":437,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":438,"parent":"0.0","parent_split_bits":0,"last_scrub":"260'25","last_scrub_stamp":"2024-03-11T21:44:20.990568+0000","last_deep_scrub":"94'15","last_deep_scrub_stamp":"2024-03-11T21:26:49.993060+0000","last_clean_scrub_stamp":"2024-03-11T21:44:20.990568+0000","objects_scrubbed":5,"log_size":35,"log_dups_size":0,"ondisk_log_size":35,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":45,"num_read_kb":40,"num_write":30,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.3","version":"74'8","reported_seq":1121,"reported_epoch":465,"state":"active+clean","last_fresh":"2024-03-11T21:46:18.567362+0000","last_change":"2024-03-11T21:45:18.050866+0000","last_active":"2024-03-11T21:46:18.567362+0000","last_peered":"2024-03-11T21:46:18.567362+0000","last_clean":"2024-03-11T21:46:18.567362+0000","last_became_active":"2024-03-11T21:25:42.578161+0000","last_became_peered":"2024-03-11T21:25:42.578161+0000","last_unstale":"2024-03-11T21:46:18.567362+0000","last_undegraded":"2024-03-11T21:46:18.567362+0000","last_fullsized":"2024-03-11T21:46:18.567362+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'8","last_scrub_stamp":"2024-03-11T21:45:18.050740+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:41.544550+0000","last_clean_scrub_stamp":"2024-03-11T21:45:18.050740+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:46:31.375018+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,5],"acting":[0,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.b","version":"465'7083","reported_seq":7131,"reported_epoch":465,"state":"active+clean","last_fresh":"2024-03-11T21:46:18.731500+0000","last_change":"2024-03-11T21:44:53.661688+0000","last_active":"2024-03-11T21:46:18.731500+0000","last_peered":"2024-03-11T21:46:18.731500+0000","last_clean":"2024-03-11T21:46:18.731500+0000","last_became_active":"2024-03-11T21:44:53.661396+0000","last_became_peered":"2024-03-11T21:44:53.661396+0000","last_unstale":"2024-03-11T21:46:18.731500+0000","last_undegraded":"2024-03-11T21:46:18.731500+0000","last_fullsized":"2024-03-11T21:46:18.731500+0000","mapping_epoch":446,"log_start":"455'3100","ondisk_log_start":"455'3100","created":446,"last_epoch_clean":447,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:44:52.644443+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:44:52.644443+0000","last_clean_scrub_stamp":"2024-03-11T21:44:52.644443+0000","objects_scrubbed":0,"log_size":3983,"log_dups_size":0,"ondisk_log_size":3983,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":464191488,"num_objects":7083,"num_object_clones":0,"num_object_copies":21249,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7083,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7083,"num_write_kb":453312,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,5],"acting":[1,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":469,"reported_epoch":464,"state":"active+clean","last_fresh":"2024-03-11T21:46:16.299480+0000","last_change":"2024-03-11T21:44:35.638977+0000","last_active":"2024-03-11T21:46:16.299480+0000","last_peered":"2024-03-11T21:46:16.299480+0000","last_clean":"2024-03-11T21:46:16.299480+0000","last_became_active":"2024-03-11T21:44:35.638587+0000","last_became_peered":"2024-03-11T21:44:35.638587+0000","last_unstale":"2024-03-11T21:46:16.299480+0000","last_undegraded":"2024-03-11T21:46:16.299480+0000","last_fullsized":"2024-03-11T21:46:16.299480+0000","mapping_epoch":437,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":438,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:44:27.024978+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:28:30.563712+0000","last_clean_scrub_stamp":"2024-03-11T21:44:27.024978+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,5],"acting":[7,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":877,"reported_epoch":464,"state":"active+clean","last_fresh":"2024-03-11T21:46:16.299899+0000","last_change":"2024-03-11T21:45:28.088759+0000","last_active":"2024-03-11T21:46:16.299899+0000","last_peered":"2024-03-11T21:46:16.299899+0000","last_clean":"2024-03-11T21:46:16.299899+0000","last_became_active":"2024-03-11T21:25:38.538540+0000","last_became_peered":"2024-03-11T21:25:38.538540+0000","last_unstale":"2024-03-11T21:46:16.299899+0000","last_undegraded":"2024-03-11T21:46:16.299899+0000","last_fullsized":"2024-03-11T21:46:16.299899+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:45:28.088517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:37.512018+0000","last_clean_scrub_stamp":"2024-03-11T21:45:28.088517+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:46:37.290364+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.4","version":"456'55","reported_seq":1248,"reported_epoch":465,"state":"active+clean","last_fresh":"2024-03-11T21:46:18.565892+0000","last_change":"2024-03-11T21:45:12.775564+0000","last_active":"2024-03-11T21:46:18.565892+0000","last_peered":"2024-03-11T21:46:18.565892+0000","last_clean":"2024-03-11T21:46:18.565892+0000","last_became_active":"2024-03-11T21:44:21.456395+0000","last_became_peered":"2024-03-11T21:44:21.456395+0000","last_unstale":"2024-03-11T21:46:18.565892+0000","last_undegraded":"2024-03-11T21:46:18.565892+0000","last_fullsized":"2024-03-11T21:46:18.565892+0000","mapping_epoch":432,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":433,"parent":"0.0","parent_split_bits":0,"last_scrub":"434'43","last_scrub_stamp":"2024-03-11T21:45:12.775393+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:39.525554+0000","last_clean_scrub_stamp":"2024-03-11T21:45:12.775393+0000","objects_scrubbed":10,"log_size":55,"log_dups_size":0,"ondisk_log_size":55,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:46:29.427790+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":358,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":84,"num_read_kb":69,"num_write":50,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":358,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,6],"acting":[1,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":870,"reported_epoch":465,"state":"active+clean","last_fresh":"2024-03-11T21:46:18.567675+0000","last_change":"2024-03-11T21:45:37.189812+0000","last_active":"2024-03-11T21:46:18.567675+0000","last_peered":"2024-03-11T21:46:18.567675+0000","last_clean":"2024-03-11T21:46:18.567675+0000","last_became_active":"2024-03-11T21:44:35.620226+0000","last_became_peered":"2024-03-11T21:44:35.620226+0000","last_unstale":"2024-03-11T21:46:18.567675+0000","last_undegraded":"2024-03-11T21:46:18.567675+0000","last_fullsized":"2024-03-11T21:46:18.567675+0000","mapping_epoch":437,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":438,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:45:37.189703+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:41.544550+0000","last_clean_scrub_stamp":"2024-03-11T21:45:37.189703+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:46:50.162183+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,6],"acting":[0,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.8","version":"464'7060","reported_seq":7106,"reported_epoch":464,"state":"active+clean","last_fresh":"2024-03-11T21:46:17.595409+0000","last_change":"2024-03-11T21:44:53.663138+0000","last_active":"2024-03-11T21:46:17.595409+0000","last_peered":"2024-03-11T21:46:17.595409+0000","last_clean":"2024-03-11T21:46:17.595409+0000","last_became_active":"2024-03-11T21:44:53.662592+0000","last_became_peered":"2024-03-11T21:44:53.662592+0000","last_unstale":"2024-03-11T21:46:17.595409+0000","last_undegraded":"2024-03-11T21:46:17.595409+0000","last_fullsized":"2024-03-11T21:46:17.595409+0000","mapping_epoch":446,"log_start":"454'2900","ondisk_log_start":"454'2900","created":446,"last_epoch_clean":447,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:44:52.644443+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:44:52.644443+0000","last_clean_scrub_stamp":"2024-03-11T21:44:52.644443+0000","objects_scrubbed":0,"log_size":4160,"log_dups_size":0,"ondisk_log_size":4160,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":462684160,"num_objects":7060,"num_object_clones":0,"num_object_copies":21180,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7060,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7060,"num_write_kb":451840,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,0],"acting":[6,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.5","version":"0'0","reported_seq":529,"reported_epoch":464,"state":"active+clean","last_fresh":"2024-03-11T21:46:16.301210+0000","last_change":"2024-03-11T21:45:26.194327+0000","last_active":"2024-03-11T21:46:16.301210+0000","last_peered":"2024-03-11T21:46:16.301210+0000","last_clean":"2024-03-11T21:46:16.301210+0000","last_became_active":"2024-03-11T21:44:21.454156+0000","last_became_peered":"2024-03-11T21:44:21.454156+0000","last_unstale":"2024-03-11T21:46:16.301210+0000","last_undegraded":"2024-03-11T21:46:16.301210+0000","last_fullsized":"2024-03-11T21:46:16.301210+0000","mapping_epoch":432,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":433,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:45:26.194208+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:43.559452+0000","last_clean_scrub_stamp":"2024-03-11T21:45:26.194208+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:46:31.560244+0000","scrub_duration":43,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,2],"acting":[7,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.0","version":"164'6","reported_seq":2115,"reported_epoch":465,"state":"active+clean","last_fresh":"2024-03-11T21:46:18.565934+0000","last_change":"2024-03-11T21:45:25.994813+0000","last_active":"2024-03-11T21:46:18.565934+0000","last_peered":"2024-03-11T21:46:18.565934+0000","last_clean":"2024-03-11T21:46:18.565934+0000","last_became_active":"2024-03-11T21:25:38.536890+0000","last_became_peered":"2024-03-11T21:25:38.536890+0000","last_unstale":"2024-03-11T21:46:18.565934+0000","last_undegraded":"2024-03-11T21:46:18.565934+0000","last_fullsized":"2024-03-11T21:46:18.565934+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"164'6","last_scrub_stamp":"2024-03-11T21:45:25.994672+0000","last_deep_scrub":"164'6","last_deep_scrub_stamp":"2024-03-11T21:32:17.597666+0000","last_clean_scrub_stamp":"2024-03-11T21:45:25.994672+0000","objects_scrubbed":1,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:46:29.823739+0000","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0.000753559,"stat_sum":{"num_bytes":389,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1231,"num_read_kb":1226,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,6],"acting":[1,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":891,"reported_epoch":464,"state":"active+clean","last_fresh":"2024-03-11T21:46:16.292380+0000","last_change":"2024-03-11T21:45:21.620901+0000","last_active":"2024-03-11T21:46:16.292380+0000","last_peered":"2024-03-11T21:46:16.292380+0000","last_clean":"2024-03-11T21:46:16.292380+0000","last_became_active":"2024-03-11T21:44:21.462115+0000","last_became_peered":"2024-03-11T21:44:21.462115+0000","last_unstale":"2024-03-11T21:46:16.292380+0000","last_undegraded":"2024-03-11T21:46:16.292380+0000","last_fullsized":"2024-03-11T21:46:16.292380+0000","mapping_epoch":432,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":433,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:45:21.620627+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:43:28.183315+0000","last_clean_scrub_stamp":"2024-03-11T21:45:21.620627+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:46:25.994451+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.7","version":"456'26","reported_seq":915,"reported_epoch":465,"state":"active+clean","last_fresh":"2024-03-11T21:46:18.566734+0000","last_change":"2024-03-11T21:45:33.942582+0000","last_active":"2024-03-11T21:46:18.566734+0000","last_peered":"2024-03-11T21:46:18.566734+0000","last_clean":"2024-03-11T21:46:18.566734+0000","last_became_active":"2024-03-11T21:25:40.541769+0000","last_became_peered":"2024-03-11T21:25:40.541769+0000","last_unstale":"2024-03-11T21:46:18.566734+0000","last_undegraded":"2024-03-11T21:46:18.566734+0000","last_fullsized":"2024-03-11T21:46:18.566734+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"260'22","last_scrub_stamp":"2024-03-11T21:45:33.942306+0000","last_deep_scrub":"260'22","last_deep_scrub_stamp":"2024-03-11T21:41:27.232403+0000","last_clean_scrub_stamp":"2024-03-11T21:45:33.942306+0000","objects_scrubbed":5,"log_size":26,"log_dups_size":0,"ondisk_log_size":26,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:46:35.510775+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":42,"num_read_kb":31,"num_write":24,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":880,"reported_epoch":464,"state":"active+clean","last_fresh":"2024-03-11T21:46:16.292775+0000","last_change":"2024-03-11T21:45:39.556570+0000","last_active":"2024-03-11T21:46:16.292775+0000","last_peered":"2024-03-11T21:46:16.292775+0000","last_clean":"2024-03-11T21:46:16.292775+0000","last_became_active":"2024-03-11T21:34:41.143660+0000","last_became_peered":"2024-03-11T21:34:41.143660+0000","last_unstale":"2024-03-11T21:46:16.292775+0000","last_undegraded":"2024-03-11T21:46:16.292775+0000","last_fullsized":"2024-03-11T21:46:16.292775+0000","mapping_epoch":226,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":227,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:45:39.556342+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:41.544550+0000","last_clean_scrub_stamp":"2024-03-11T21:45:39.556342+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:46:45.041859+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7],"acting":[2,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"11.a","version":"464'7237","reported_seq":7283,"reported_epoch":464,"state":"active+clean","last_fresh":"2024-03-11T21:46:17.622454+0000","last_change":"2024-03-11T21:44:53.663350+0000","last_active":"2024-03-11T21:46:17.622454+0000","last_peered":"2024-03-11T21:46:17.622454+0000","last_clean":"2024-03-11T21:46:17.622454+0000","last_became_active":"2024-03-11T21:44:53.663015+0000","last_became_peered":"2024-03-11T21:44:53.663015+0000","last_unstale":"2024-03-11T21:46:17.622454+0000","last_undegraded":"2024-03-11T21:46:17.622454+0000","last_fullsized":"2024-03-11T21:46:17.622454+0000","mapping_epoch":446,"log_start":"454'3100","ondisk_log_start":"454'3100","created":446,"last_epoch_clean":447,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:44:52.644443+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:44:52.644443+0000","last_clean_scrub_stamp":"2024-03-11T21:44:52.644443+0000","objects_scrubbed":0,"log_size":4137,"log_dups_size":0,"ondisk_log_size":4137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":474284032,"num_objects":7237,"num_object_clones":0,"num_object_copies":21711,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7237,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7237,"num_write_kb":463168,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,1],"acting":[6,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.7","version":"124'2","reported_seq":880,"reported_epoch":464,"state":"active+clean","last_fresh":"2024-03-11T21:46:16.305241+0000","last_change":"2024-03-11T21:44:35.716577+0000","last_active":"2024-03-11T21:46:16.305241+0000","last_peered":"2024-03-11T21:46:16.305241+0000","last_clean":"2024-03-11T21:46:16.305241+0000","last_became_active":"2024-03-11T21:44:35.623378+0000","last_became_peered":"2024-03-11T21:44:35.623378+0000","last_unstale":"2024-03-11T21:46:16.305241+0000","last_undegraded":"2024-03-11T21:46:16.305241+0000","last_fullsized":"2024-03-11T21:46:16.305241+0000","mapping_epoch":437,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":438,"parent":"0.0","parent_split_bits":0,"last_scrub":"124'2","last_scrub_stamp":"2024-03-11T21:44:10.780008+0000","last_deep_scrub":"124'2","last_deep_scrub_stamp":"2024-03-11T21:29:34.956908+0000","last_clean_scrub_stamp":"2024-03-11T21:44:10.780008+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":111,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":26,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,7],"acting":[5,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2","version":"71'2","reported_seq":176,"reported_epoch":464,"state":"active+clean","last_fresh":"2024-03-11T21:46:16.301884+0000","last_change":"2024-03-11T21:44:35.635712+0000","last_active":"2024-03-11T21:46:16.301884+0000","last_peered":"2024-03-11T21:46:16.301884+0000","last_clean":"2024-03-11T21:46:16.301884+0000","last_became_active":"2024-03-11T21:44:35.622571+0000","last_became_peered":"2024-03-11T21:44:35.622571+0000","last_unstale":"2024-03-11T21:46:16.301884+0000","last_undegraded":"2024-03-11T21:46:16.301884+0000","last_fullsized":"2024-03-11T21:46:16.301884+0000","mapping_epoch":437,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":438,"parent":"0.0","parent_split_bits":0,"last_scrub":"71'2","last_scrub_stamp":"2024-03-11T21:44:27.869844+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:37.512018+0000","last_clean_scrub_stamp":"2024-03-11T21:44:27.869844+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.0014206449999999999,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":38,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,1],"acting":[5,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":887,"reported_epoch":464,"state":"active+clean","last_fresh":"2024-03-11T21:46:16.300131+0000","last_change":"2024-03-11T21:44:20.889721+0000","last_active":"2024-03-11T21:46:16.300131+0000","last_peered":"2024-03-11T21:46:16.300131+0000","last_clean":"2024-03-11T21:46:16.300131+0000","last_became_active":"2024-03-11T21:34:41.146672+0000","last_became_peered":"2024-03-11T21:34:41.146672+0000","last_unstale":"2024-03-11T21:46:16.300131+0000","last_undegraded":"2024-03-11T21:46:16.300131+0000","last_fullsized":"2024-03-11T21:46:16.300131+0000","mapping_epoch":226,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":227,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:44:20.889645+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:37.512018+0000","last_clean_scrub_stamp":"2024-03-11T21:44:20.889645+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"87'128","reported_seq":985,"reported_epoch":464,"state":"active+clean","last_fresh":"2024-03-11T21:46:16.299929+0000","last_change":"2024-03-11T21:44:20.468089+0000","last_active":"2024-03-11T21:46:16.299929+0000","last_peered":"2024-03-11T21:46:16.299929+0000","last_clean":"2024-03-11T21:46:16.299929+0000","last_became_active":"2024-03-11T21:34:44.164569+0000","last_became_peered":"2024-03-11T21:34:44.164569+0000","last_unstale":"2024-03-11T21:46:16.299929+0000","last_undegraded":"2024-03-11T21:46:16.299929+0000","last_fullsized":"2024-03-11T21:46:16.299929+0000","mapping_epoch":229,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":230,"parent":"0.0","parent_split_bits":0,"last_scrub":"87'128","last_scrub_stamp":"2024-03-11T21:44:20.468006+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:24.261974+0000","last_clean_scrub_stamp":"2024-03-11T21:44:20.468006+0000","objects_scrubbed":2,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":198,"num_read_kb":294,"num_write":179,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":2951840,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.5","version":"456'41","reported_seq":1213,"reported_epoch":464,"state":"active+clean","last_fresh":"2024-03-11T21:46:16.289434+0000","last_change":"2024-03-11T21:44:35.660956+0000","last_active":"2024-03-11T21:46:16.289434+0000","last_peered":"2024-03-11T21:46:16.289434+0000","last_clean":"2024-03-11T21:46:16.289434+0000","last_became_active":"2024-03-11T21:44:35.634664+0000","last_became_peered":"2024-03-11T21:44:35.634664+0000","last_unstale":"2024-03-11T21:46:16.289434+0000","last_undegraded":"2024-03-11T21:46:16.289434+0000","last_fullsized":"2024-03-11T21:46:16.289434+0000","mapping_epoch":437,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":438,"parent":"0.0","parent_split_bits":0,"last_scrub":"260'28","last_scrub_stamp":"2024-03-11T21:44:26.464384+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:39.525554+0000","last_clean_scrub_stamp":"2024-03-11T21:44:26.464384+0000","objects_scrubbed":8,"log_size":41,"log_dups_size":0,"ondisk_log_size":41,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":154,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":61,"num_read_kb":51,"num_write":37,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":154,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,0],"acting":[6,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":514,"reported_epoch":464,"state":"active+clean","last_fresh":"2024-03-11T21:46:16.301328+0000","last_change":"2024-03-11T21:44:17.426296+0000","last_active":"2024-03-11T21:46:16.301328+0000","last_peered":"2024-03-11T21:46:16.301328+0000","last_clean":"2024-03-11T21:46:16.301328+0000","last_became_active":"2024-03-11T21:34:41.139782+0000","last_became_peered":"2024-03-11T21:34:41.139782+0000","last_unstale":"2024-03-11T21:46:16.301328+0000","last_undegraded":"2024-03-11T21:46:16.301328+0000","last_fullsized":"2024-03-11T21:46:16.301328+0000","mapping_epoch":226,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":227,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:44:17.426179+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:32:21.266672+0000","last_clean_scrub_stamp":"2024-03-11T21:44:17.426179+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,5],"acting":[7,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"11.9","version":"464'6943","reported_seq":6989,"reported_epoch":464,"state":"active+clean","last_fresh":"2024-03-11T21:46:17.627981+0000","last_change":"2024-03-11T21:44:53.662919+0000","last_active":"2024-03-11T21:46:17.627981+0000","last_peered":"2024-03-11T21:46:17.627981+0000","last_clean":"2024-03-11T21:46:17.627981+0000","last_became_active":"2024-03-11T21:44:53.662344+0000","last_became_peered":"2024-03-11T21:44:53.662344+0000","last_unstale":"2024-03-11T21:46:17.627981+0000","last_undegraded":"2024-03-11T21:46:17.627981+0000","last_fullsized":"2024-03-11T21:46:17.627981+0000","mapping_epoch":446,"log_start":"454'2800","ondisk_log_start":"454'2800","created":446,"last_epoch_clean":447,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:44:52.644443+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:44:52.644443+0000","last_clean_scrub_stamp":"2024-03-11T21:44:52.644443+0000","objects_scrubbed":0,"log_size":4143,"log_dups_size":0,"ondisk_log_size":4143,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:46:19.173115+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":455016448,"num_objects":6943,"num_object_clones":0,"num_object_copies":20829,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6943,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6943,"num_write_kb":444352,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,1],"acting":[6,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.4","version":"0'0","reported_seq":867,"reported_epoch":465,"state":"active+clean","last_fresh":"2024-03-11T21:46:18.566268+0000","last_change":"2024-03-11T21:44:27.041389+0000","last_active":"2024-03-11T21:46:18.566268+0000","last_peered":"2024-03-11T21:46:18.566268+0000","last_clean":"2024-03-11T21:46:18.566268+0000","last_became_active":"2024-03-11T21:44:21.444747+0000","last_became_peered":"2024-03-11T21:44:21.444747+0000","last_unstale":"2024-03-11T21:46:18.566268+0000","last_undegraded":"2024-03-11T21:46:18.566268+0000","last_fullsized":"2024-03-11T21:46:18.566268+0000","mapping_epoch":432,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":433,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:44:27.041077+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:43.559452+0000","last_clean_scrub_stamp":"2024-03-11T21:44:27.041077+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,6],"acting":[1,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":900,"reported_epoch":465,"state":"active+clean","last_fresh":"2024-03-11T21:46:18.569897+0000","last_change":"2024-03-11T21:45:12.113933+0000","last_active":"2024-03-11T21:46:18.569897+0000","last_peered":"2024-03-11T21:46:18.569897+0000","last_clean":"2024-03-11T21:46:18.569897+0000","last_became_active":"2024-03-11T21:44:35.624693+0000","last_became_peered":"2024-03-11T21:44:35.624693+0000","last_unstale":"2024-03-11T21:46:18.569897+0000","last_undegraded":"2024-03-11T21:46:18.569897+0000","last_fullsized":"2024-03-11T21:46:18.569897+0000","mapping_epoch":437,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":438,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:45:12.113799+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:37.512018+0000","last_clean_scrub_stamp":"2024-03-11T21:45:12.113799+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0.00089342499999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":506,"reported_epoch":464,"state":"active+clean","last_fresh":"2024-03-11T21:46:16.299407+0000","last_change":"2024-03-11T21:43:30.734142+0000","last_active":"2024-03-11T21:46:16.299407+0000","last_peered":"2024-03-11T21:46:16.299407+0000","last_clean":"2024-03-11T21:46:16.299407+0000","last_became_active":"2024-03-11T21:34:41.149219+0000","last_became_peered":"2024-03-11T21:34:41.149219+0000","last_unstale":"2024-03-11T21:46:16.299407+0000","last_undegraded":"2024-03-11T21:46:16.299407+0000","last_fullsized":"2024-03-11T21:46:16.299407+0000","mapping_epoch":226,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":227,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:43:30.734024+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:43:30.734024+0000","last_clean_scrub_stamp":"2024-03-11T21:43:30.734024+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,0],"acting":[7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.6","version":"456'42","reported_seq":956,"reported_epoch":465,"state":"active+clean","last_fresh":"2024-03-11T21:46:18.567903+0000","last_change":"2024-03-11T21:46:10.509329+0000","last_active":"2024-03-11T21:46:18.567903+0000","last_peered":"2024-03-11T21:46:18.567903+0000","last_clean":"2024-03-11T21:46:18.567903+0000","last_became_active":"2024-03-11T21:32:22.796486+0000","last_became_peered":"2024-03-11T21:32:22.796486+0000","last_unstale":"2024-03-11T21:46:18.567903+0000","last_undegraded":"2024-03-11T21:46:18.567903+0000","last_fullsized":"2024-03-11T21:46:18.567903+0000","mapping_epoch":166,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":167,"parent":"0.0","parent_split_bits":0,"last_scrub":"456'42","last_scrub_stamp":"2024-03-11T21:46:10.509062+0000","last_deep_scrub":"352'30","last_deep_scrub_stamp":"2024-03-11T21:40:31.175323+0000","last_clean_scrub_stamp":"2024-03-11T21:46:10.509062+0000","objects_scrubbed":6,"log_size":42,"log_dups_size":0,"ondisk_log_size":42,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:47:18.612508+0000","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":54,"num_read_kb":48,"num_write":36,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":861,"reported_epoch":464,"state":"active+clean","last_fresh":"2024-03-11T21:46:16.305158+0000","last_change":"2024-03-11T21:45:40.086098+0000","last_active":"2024-03-11T21:46:16.305158+0000","last_peered":"2024-03-11T21:46:16.305158+0000","last_clean":"2024-03-11T21:46:16.305158+0000","last_became_active":"2024-03-11T21:25:42.572324+0000","last_became_peered":"2024-03-11T21:25:42.572324+0000","last_unstale":"2024-03-11T21:46:16.305158+0000","last_undegraded":"2024-03-11T21:46:16.305158+0000","last_fullsized":"2024-03-11T21:46:16.305158+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:45:40.085999+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:41.544550+0000","last_clean_scrub_stamp":"2024-03-11T21:45:40.085999+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:47:02.252873+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"11.c","version":"465'7060","reported_seq":7108,"reported_epoch":465,"state":"active+clean","last_fresh":"2024-03-11T21:46:18.707646+0000","last_change":"2024-03-11T21:44:53.666779+0000","last_active":"2024-03-11T21:46:18.707646+0000","last_peered":"2024-03-11T21:46:18.707646+0000","last_clean":"2024-03-11T21:46:18.707646+0000","last_became_active":"2024-03-11T21:44:53.666491+0000","last_became_peered":"2024-03-11T21:44:53.666491+0000","last_unstale":"2024-03-11T21:46:18.707646+0000","last_undegraded":"2024-03-11T21:46:18.707646+0000","last_fullsized":"2024-03-11T21:46:18.707646+0000","mapping_epoch":446,"log_start":"455'3100","ondisk_log_start":"455'3100","created":446,"last_epoch_clean":447,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:44:52.644443+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:44:52.644443+0000","last_clean_scrub_stamp":"2024-03-11T21:44:52.644443+0000","objects_scrubbed":0,"log_size":3960,"log_dups_size":0,"ondisk_log_size":3960,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":462684160,"num_objects":7060,"num_object_clones":0,"num_object_copies":21180,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7060,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7060,"num_write_kb":451840,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,7],"acting":[1,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":868,"reported_epoch":465,"state":"active+clean","last_fresh":"2024-03-11T21:46:18.566339+0000","last_change":"2024-03-11T21:46:18.006161+0000","last_active":"2024-03-11T21:46:18.566339+0000","last_peered":"2024-03-11T21:46:18.566339+0000","last_clean":"2024-03-11T21:46:18.566339+0000","last_became_active":"2024-03-11T21:25:45.301416+0000","last_became_peered":"2024-03-11T21:25:45.301416+0000","last_unstale":"2024-03-11T21:46:18.566339+0000","last_undegraded":"2024-03-11T21:46:18.566339+0000","last_fullsized":"2024-03-11T21:46:18.566339+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:46:18.006081+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:43.559452+0000","last_clean_scrub_stamp":"2024-03-11T21:46:18.006081+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:47:41.400276+0000","scrub_duration":29,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":520,"reported_epoch":464,"state":"active+clean","last_fresh":"2024-03-11T21:46:16.301297+0000","last_change":"2024-03-11T21:45:21.273974+0000","last_active":"2024-03-11T21:46:16.301297+0000","last_peered":"2024-03-11T21:46:16.301297+0000","last_clean":"2024-03-11T21:46:16.301297+0000","last_became_active":"2024-03-11T21:44:35.624442+0000","last_became_peered":"2024-03-11T21:44:35.624442+0000","last_unstale":"2024-03-11T21:46:16.301297+0000","last_undegraded":"2024-03-11T21:46:16.301297+0000","last_fullsized":"2024-03-11T21:46:16.301297+0000","mapping_epoch":437,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":438,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:45:21.273887+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:26:57.333669+0000","last_clean_scrub_stamp":"2024-03-11T21:45:21.273887+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:46:22.100538+0000","scrub_duration":69,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,1],"acting":[7,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.3","version":"456'47","reported_seq":990,"reported_epoch":465,"state":"active+clean","last_fresh":"2024-03-11T21:46:18.567034+0000","last_change":"2024-03-11T21:44:34.140064+0000","last_active":"2024-03-11T21:46:18.567034+0000","last_peered":"2024-03-11T21:46:18.567034+0000","last_clean":"2024-03-11T21:46:18.567034+0000","last_became_active":"2024-03-11T21:34:41.145607+0000","last_became_peered":"2024-03-11T21:34:41.145607+0000","last_unstale":"2024-03-11T21:46:18.567034+0000","last_undegraded":"2024-03-11T21:46:18.567034+0000","last_fullsized":"2024-03-11T21:46:18.567034+0000","mapping_epoch":226,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":227,"parent":"0.0","parent_split_bits":0,"last_scrub":"260'37","last_scrub_stamp":"2024-03-11T21:44:34.139954+0000","last_deep_scrub":"165'27","last_deep_scrub_stamp":"2024-03-11T21:32:21.576732+0000","last_clean_scrub_stamp":"2024-03-11T21:44:34.139954+0000","objects_scrubbed":8,"log_size":47,"log_dups_size":0,"ondisk_log_size":47,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":69,"num_read_kb":55,"num_write":42,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":660,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":886,"reported_epoch":464,"state":"active+clean","last_fresh":"2024-03-11T21:46:16.289791+0000","last_change":"2024-03-11T21:44:35.462132+0000","last_active":"2024-03-11T21:46:16.289791+0000","last_peered":"2024-03-11T21:46:16.289791+0000","last_clean":"2024-03-11T21:46:16.289791+0000","last_became_active":"2024-03-11T21:25:42.579045+0000","last_became_peered":"2024-03-11T21:25:42.579045+0000","last_unstale":"2024-03-11T21:46:16.289791+0000","last_undegraded":"2024-03-11T21:46:16.289791+0000","last_fullsized":"2024-03-11T21:46:16.289791+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:44:35.462043+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:41:21.857205+0000","last_clean_scrub_stamp":"2024-03-11T21:44:35.462043+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.f","version":"464'7013","reported_seq":7058,"reported_epoch":464,"state":"active+clean","last_fresh":"2024-03-11T21:46:17.586218+0000","last_change":"2024-03-11T21:44:53.666917+0000","last_active":"2024-03-11T21:46:17.586218+0000","last_peered":"2024-03-11T21:46:17.586218+0000","last_clean":"2024-03-11T21:46:17.586218+0000","last_became_active":"2024-03-11T21:44:53.664970+0000","last_became_peered":"2024-03-11T21:44:53.664970+0000","last_unstale":"2024-03-11T21:46:17.586218+0000","last_undegraded":"2024-03-11T21:46:17.586218+0000","last_fullsized":"2024-03-11T21:46:17.586218+0000","mapping_epoch":446,"log_start":"454'3000","ondisk_log_start":"454'3000","created":446,"last_epoch_clean":447,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:44:52.644443+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:44:52.644443+0000","last_clean_scrub_stamp":"2024-03-11T21:44:52.644443+0000","objects_scrubbed":0,"log_size":4013,"log_dups_size":0,"ondisk_log_size":4013,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":459603968,"num_objects":7013,"num_object_clones":0,"num_object_copies":21039,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7013,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7013,"num_write_kb":448832,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,2],"acting":[7,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":870,"reported_epoch":464,"state":"active+clean","last_fresh":"2024-03-11T21:46:16.291942+0000","last_change":"2024-03-11T21:45:35.682015+0000","last_active":"2024-03-11T21:46:16.291942+0000","last_peered":"2024-03-11T21:46:16.291942+0000","last_clean":"2024-03-11T21:46:16.291942+0000","last_became_active":"2024-03-11T21:44:35.628373+0000","last_became_peered":"2024-03-11T21:44:35.628373+0000","last_unstale":"2024-03-11T21:46:16.291942+0000","last_undegraded":"2024-03-11T21:46:16.291942+0000","last_fullsized":"2024-03-11T21:46:16.291942+0000","mapping_epoch":437,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":438,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:45:35.681881+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:43.559452+0000","last_clean_scrub_stamp":"2024-03-11T21:45:35.681881+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:46:37.229386+0000","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6,7],"acting":[2,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.6","version":"67'1","reported_seq":883,"reported_epoch":465,"state":"active+clean","last_fresh":"2024-03-11T21:46:18.573522+0000","last_change":"2024-03-11T21:45:16.765246+0000","last_active":"2024-03-11T21:46:18.573522+0000","last_peered":"2024-03-11T21:46:18.573522+0000","last_clean":"2024-03-11T21:46:18.573522+0000","last_became_active":"2024-03-11T21:44:35.618830+0000","last_became_peered":"2024-03-11T21:44:35.618830+0000","last_unstale":"2024-03-11T21:46:18.573522+0000","last_undegraded":"2024-03-11T21:46:18.573522+0000","last_fullsized":"2024-03-11T21:46:18.573522+0000","mapping_epoch":437,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":438,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'1","last_scrub_stamp":"2024-03-11T21:45:16.765125+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:37.512018+0000","last_clean_scrub_stamp":"2024-03-11T21:45:16.765125+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:46:25.529029+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":46,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,0],"acting":[1,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.0","version":"456'50","reported_seq":613,"reported_epoch":464,"state":"active+clean","last_fresh":"2024-03-11T21:46:16.299652+0000","last_change":"2024-03-11T21:45:39.242650+0000","last_active":"2024-03-11T21:46:16.299652+0000","last_peered":"2024-03-11T21:46:16.299652+0000","last_clean":"2024-03-11T21:46:16.299652+0000","last_became_active":"2024-03-11T21:44:21.449580+0000","last_became_peered":"2024-03-11T21:44:21.449580+0000","last_unstale":"2024-03-11T21:46:16.299652+0000","last_undegraded":"2024-03-11T21:46:16.299652+0000","last_fullsized":"2024-03-11T21:46:16.299652+0000","mapping_epoch":432,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":433,"parent":"0.0","parent_split_bits":0,"last_scrub":"260'38","last_scrub_stamp":"2024-03-11T21:45:39.241846+0000","last_deep_scrub":"260'38","last_deep_scrub_stamp":"2024-03-11T21:45:39.241846+0000","last_clean_scrub_stamp":"2024-03-11T21:45:39.241846+0000","objects_scrubbed":8,"log_size":50,"log_dups_size":0,"ondisk_log_size":50,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:47:07.836051+0000","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":70,"num_read_kb":58,"num_write":44,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":440,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,1],"acting":[7,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":889,"reported_epoch":464,"state":"active+clean","last_fresh":"2024-03-11T21:46:16.300513+0000","last_change":"2024-03-11T21:46:09.504481+0000","last_active":"2024-03-11T21:46:16.300513+0000","last_peered":"2024-03-11T21:46:16.300513+0000","last_clean":"2024-03-11T21:46:16.300513+0000","last_became_active":"2024-03-11T21:44:35.625859+0000","last_became_peered":"2024-03-11T21:44:35.625859+0000","last_unstale":"2024-03-11T21:46:16.300513+0000","last_undegraded":"2024-03-11T21:46:16.300513+0000","last_fullsized":"2024-03-11T21:46:16.300513+0000","mapping_epoch":437,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":438,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:46:09.504070+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:40:21.518840+0000","last_clean_scrub_stamp":"2024-03-11T21:46:09.504070+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:47:37.629730+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,5],"acting":[7,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"11.e","version":"465'7091","reported_seq":7139,"reported_epoch":465,"state":"active+clean","last_fresh":"2024-03-11T21:46:18.692248+0000","last_change":"2024-03-11T21:44:53.665261+0000","last_active":"2024-03-11T21:46:18.692248+0000","last_peered":"2024-03-11T21:46:18.692248+0000","last_clean":"2024-03-11T21:46:18.692248+0000","last_became_active":"2024-03-11T21:44:53.664957+0000","last_became_peered":"2024-03-11T21:44:53.664957+0000","last_unstale":"2024-03-11T21:46:18.692248+0000","last_undegraded":"2024-03-11T21:46:18.692248+0000","last_fullsized":"2024-03-11T21:46:18.692248+0000","mapping_epoch":446,"log_start":"455'3100","ondisk_log_start":"455'3100","created":446,"last_epoch_clean":447,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:44:52.644443+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:44:52.644443+0000","last_clean_scrub_stamp":"2024-03-11T21:44:52.644443+0000","objects_scrubbed":0,"log_size":3991,"log_dups_size":0,"ondisk_log_size":3991,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":464715776,"num_objects":7091,"num_object_clones":0,"num_object_copies":21273,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7091,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7091,"num_write_kb":453824,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,0],"acting":[1,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.3","version":"0'0","reported_seq":506,"reported_epoch":464,"state":"active+clean","last_fresh":"2024-03-11T21:46:16.299822+0000","last_change":"2024-03-11T21:43:25.746794+0000","last_active":"2024-03-11T21:46:16.299822+0000","last_peered":"2024-03-11T21:46:16.299822+0000","last_clean":"2024-03-11T21:46:16.299822+0000","last_became_active":"2024-03-11T21:34:41.142683+0000","last_became_peered":"2024-03-11T21:34:41.142683+0000","last_unstale":"2024-03-11T21:46:16.299822+0000","last_undegraded":"2024-03-11T21:46:16.299822+0000","last_fullsized":"2024-03-11T21:46:16.299822+0000","mapping_epoch":226,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":227,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:43:25.746680+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:29:42.485740+0000","last_clean_scrub_stamp":"2024-03-11T21:43:25.746680+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":116,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,2],"acting":[7,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":906,"reported_epoch":464,"state":"active+clean","last_fresh":"2024-03-11T21:46:16.289843+0000","last_change":"2024-03-11T21:45:29.722211+0000","last_active":"2024-03-11T21:46:16.289843+0000","last_peered":"2024-03-11T21:46:16.289843+0000","last_clean":"2024-03-11T21:46:16.289843+0000","last_became_active":"2024-03-11T21:34:41.145628+0000","last_became_peered":"2024-03-11T21:34:41.145628+0000","last_unstale":"2024-03-11T21:46:16.289843+0000","last_undegraded":"2024-03-11T21:46:16.289843+0000","last_fullsized":"2024-03-11T21:46:16.289843+0000","mapping_epoch":226,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":227,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:45:29.722096+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:37.512018+0000","last_clean_scrub_stamp":"2024-03-11T21:45:29.722096+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:46:30.582162+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,2],"acting":[6,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.1","version":"456'42","reported_seq":586,"reported_epoch":464,"state":"active+clean","last_fresh":"2024-03-11T21:46:16.299370+0000","last_change":"2024-03-11T21:45:20.088691+0000","last_active":"2024-03-11T21:46:16.299370+0000","last_peered":"2024-03-11T21:46:16.299370+0000","last_clean":"2024-03-11T21:46:16.299370+0000","last_became_active":"2024-03-11T21:44:35.603817+0000","last_became_peered":"2024-03-11T21:44:35.603817+0000","last_unstale":"2024-03-11T21:46:16.299370+0000","last_undegraded":"2024-03-11T21:46:16.299370+0000","last_fullsized":"2024-03-11T21:46:16.299370+0000","mapping_epoch":437,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":438,"parent":"0.0","parent_split_bits":0,"last_scrub":"260'28","last_scrub_stamp":"2024-03-11T21:45:20.088592+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:39.525554+0000","last_clean_scrub_stamp":"2024-03-11T21:45:20.088592+0000","objects_scrubbed":7,"log_size":42,"log_dups_size":0,"ondisk_log_size":42,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:46:29.310183+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":62,"num_read_kb":55,"num_write":42,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,0],"acting":[5,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.0","version":"438'13","reported_seq":281,"reported_epoch":465,"state":"active+clean","last_fresh":"2024-03-11T21:46:18.565661+0000","last_change":"2024-03-11T21:44:35.654883+0000","last_active":"2024-03-11T21:46:18.565661+0000","last_peered":"2024-03-11T21:46:18.565661+0000","last_clean":"2024-03-11T21:46:18.565661+0000","last_became_active":"2024-03-11T21:44:35.621524+0000","last_became_peered":"2024-03-11T21:44:35.621524+0000","last_unstale":"2024-03-11T21:46:18.565661+0000","last_undegraded":"2024-03-11T21:46:18.565661+0000","last_fullsized":"2024-03-11T21:46:18.565661+0000","mapping_epoch":437,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":438,"parent":"0.0","parent_split_bits":0,"last_scrub":"280'11","last_scrub_stamp":"2024-03-11T21:44:13.023265+0000","last_deep_scrub":"98'9","last_deep_scrub_stamp":"2024-03-11T21:27:29.745711+0000","last_clean_scrub_stamp":"2024-03-11T21:44:13.023265+0000","objects_scrubbed":1,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.1","version":"464'7049","reported_seq":7095,"reported_epoch":464,"state":"active+clean","last_fresh":"2024-03-11T21:46:17.627912+0000","last_change":"2024-03-11T21:44:53.663619+0000","last_active":"2024-03-11T21:46:17.627912+0000","last_peered":"2024-03-11T21:46:17.627912+0000","last_clean":"2024-03-11T21:46:17.627912+0000","last_became_active":"2024-03-11T21:44:53.663333+0000","last_became_peered":"2024-03-11T21:44:53.663333+0000","last_unstale":"2024-03-11T21:46:17.627912+0000","last_undegraded":"2024-03-11T21:46:17.627912+0000","last_fullsized":"2024-03-11T21:46:17.627912+0000","mapping_epoch":446,"log_start":"454'2900","ondisk_log_start":"454'2900","created":446,"last_epoch_clean":447,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:44:52.644443+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:44:52.644443+0000","last_clean_scrub_stamp":"2024-03-11T21:44:52.644443+0000","objects_scrubbed":0,"log_size":4149,"log_dups_size":0,"ondisk_log_size":4149,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":461963264,"num_objects":7049,"num_object_clones":0,"num_object_copies":21147,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7049,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7049,"num_write_kb":451136,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,2],"acting":[6,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.c","version":"0'0","reported_seq":471,"reported_epoch":464,"state":"active+clean","last_fresh":"2024-03-11T21:46:16.289128+0000","last_change":"2024-03-11T21:44:24.425172+0000","last_active":"2024-03-11T21:46:16.289128+0000","last_peered":"2024-03-11T21:46:16.289128+0000","last_clean":"2024-03-11T21:46:16.289128+0000","last_became_active":"2024-03-11T21:44:21.443774+0000","last_became_peered":"2024-03-11T21:44:21.443774+0000","last_unstale":"2024-03-11T21:46:16.289128+0000","last_undegraded":"2024-03-11T21:46:16.289128+0000","last_fullsized":"2024-03-11T21:46:16.289128+0000","mapping_epoch":432,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":433,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:44:24.425056+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:43.559452+0000","last_clean_scrub_stamp":"2024-03-11T21:44:24.425056+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,7],"acting":[6,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.f","version":"0'0","reported_seq":875,"reported_epoch":464,"state":"active+clean","last_fresh":"2024-03-11T21:46:16.304960+0000","last_change":"2024-03-11T21:46:09.935746+0000","last_active":"2024-03-11T21:46:16.304960+0000","last_peered":"2024-03-11T21:46:16.304960+0000","last_clean":"2024-03-11T21:46:16.304960+0000","last_became_active":"2024-03-11T21:44:35.645274+0000","last_became_peered":"2024-03-11T21:44:35.645274+0000","last_unstale":"2024-03-11T21:46:16.304960+0000","last_undegraded":"2024-03-11T21:46:16.304960+0000","last_fullsized":"2024-03-11T21:46:16.304960+0000","mapping_epoch":437,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":438,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:46:09.935636+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:43:24.002346+0000","last_clean_scrub_stamp":"2024-03-11T21:46:09.935636+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:47:15.326139+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,2],"acting":[5,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.8","version":"0'0","reported_seq":514,"reported_epoch":464,"state":"active+clean","last_fresh":"2024-03-11T21:46:16.299818+0000","last_change":"2024-03-11T21:45:25.159896+0000","last_active":"2024-03-11T21:46:16.299818+0000","last_peered":"2024-03-11T21:46:16.299818+0000","last_clean":"2024-03-11T21:46:16.299818+0000","last_became_active":"2024-03-11T21:34:41.144681+0000","last_became_peered":"2024-03-11T21:34:41.144681+0000","last_unstale":"2024-03-11T21:46:16.299818+0000","last_undegraded":"2024-03-11T21:46:16.299818+0000","last_fullsized":"2024-03-11T21:46:16.299818+0000","mapping_epoch":226,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":227,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:45:25.159772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:37.512018+0000","last_clean_scrub_stamp":"2024-03-11T21:45:25.159772+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:46:51.655751+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,1],"acting":[7,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.e","version":"456'32","reported_seq":567,"reported_epoch":464,"state":"active+clean","last_fresh":"2024-03-11T21:46:16.289681+0000","last_change":"2024-03-11T21:46:07.696677+0000","last_active":"2024-03-11T21:46:16.289681+0000","last_peered":"2024-03-11T21:46:16.289681+0000","last_clean":"2024-03-11T21:46:16.289681+0000","last_became_active":"2024-03-11T21:44:35.616490+0000","last_became_peered":"2024-03-11T21:44:35.616490+0000","last_unstale":"2024-03-11T21:46:16.289681+0000","last_undegraded":"2024-03-11T21:46:16.289681+0000","last_fullsized":"2024-03-11T21:46:16.289681+0000","mapping_epoch":437,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":438,"parent":"0.0","parent_split_bits":0,"last_scrub":"456'32","last_scrub_stamp":"2024-03-11T21:46:07.696550+0000","last_deep_scrub":"109'16","last_deep_scrub_stamp":"2024-03-11T21:28:24.739395+0000","last_clean_scrub_stamp":"2024-03-11T21:46:07.696550+0000","objects_scrubbed":5,"log_size":32,"log_dups_size":0,"ondisk_log_size":32,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:47:22.872084+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":43,"num_read_kb":36,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,5],"acting":[6,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.0","version":"465'7176","reported_seq":7224,"reported_epoch":465,"state":"active+clean","last_fresh":"2024-03-11T21:46:18.709009+0000","last_change":"2024-03-11T21:44:53.667315+0000","last_active":"2024-03-11T21:46:18.709009+0000","last_peered":"2024-03-11T21:46:18.709009+0000","last_clean":"2024-03-11T21:46:18.709009+0000","last_became_active":"2024-03-11T21:44:53.666983+0000","last_became_peered":"2024-03-11T21:44:53.666983+0000","last_unstale":"2024-03-11T21:46:18.709009+0000","last_undegraded":"2024-03-11T21:46:18.709009+0000","last_fullsized":"2024-03-11T21:46:18.709009+0000","mapping_epoch":446,"log_start":"455'3200","ondisk_log_start":"455'3200","created":446,"last_epoch_clean":447,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:44:52.644443+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:44:52.644443+0000","last_clean_scrub_stamp":"2024-03-11T21:44:52.644443+0000","objects_scrubbed":0,"log_size":3976,"log_dups_size":0,"ondisk_log_size":3976,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":470286336,"num_objects":7176,"num_object_clones":0,"num_object_copies":21528,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7176,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7176,"num_write_kb":459264,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,6],"acting":[1,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.d","version":"0'0","reported_seq":857,"reported_epoch":464,"state":"active+clean","last_fresh":"2024-03-11T21:46:16.303626+0000","last_change":"2024-03-11T21:44:33.036306+0000","last_active":"2024-03-11T21:46:16.303626+0000","last_peered":"2024-03-11T21:46:16.303626+0000","last_clean":"2024-03-11T21:46:16.303626+0000","last_became_active":"2024-03-11T21:25:44.590333+0000","last_became_peered":"2024-03-11T21:25:44.590333+0000","last_unstale":"2024-03-11T21:46:16.303626+0000","last_undegraded":"2024-03-11T21:46:16.303626+0000","last_fullsized":"2024-03-11T21:46:16.303626+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:44:33.036191+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:35:37.182850+0000","last_clean_scrub_stamp":"2024-03-11T21:44:33.036191+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":72,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.e","version":"438'9","reported_seq":122,"reported_epoch":464,"state":"active+clean","last_fresh":"2024-03-11T21:46:16.303558+0000","last_change":"2024-03-11T21:44:35.992983+0000","last_active":"2024-03-11T21:46:16.303558+0000","last_peered":"2024-03-11T21:46:16.303558+0000","last_clean":"2024-03-11T21:46:16.303558+0000","last_became_active":"2024-03-11T21:44:35.619861+0000","last_became_peered":"2024-03-11T21:44:35.619861+0000","last_unstale":"2024-03-11T21:46:16.303558+0000","last_undegraded":"2024-03-11T21:46:16.303558+0000","last_fullsized":"2024-03-11T21:46:16.303558+0000","mapping_epoch":437,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":438,"parent":"0.0","parent_split_bits":0,"last_scrub":"438'9","last_scrub_stamp":"2024-03-11T21:44:35.992899+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:41.544550+0000","last_clean_scrub_stamp":"2024-03-11T21:44:35.992899+0000","objects_scrubbed":1,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,2],"acting":[5,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.9","version":"0'0","reported_seq":891,"reported_epoch":465,"state":"active+clean","last_fresh":"2024-03-11T21:46:18.566161+0000","last_change":"2024-03-11T21:44:21.454706+0000","last_active":"2024-03-11T21:46:18.566161+0000","last_peered":"2024-03-11T21:46:18.566161+0000","last_clean":"2024-03-11T21:46:18.566161+0000","last_became_active":"2024-03-11T21:44:21.454167+0000","last_became_peered":"2024-03-11T21:44:21.454167+0000","last_unstale":"2024-03-11T21:46:18.566161+0000","last_undegraded":"2024-03-11T21:46:18.566161+0000","last_fullsized":"2024-03-11T21:46:18.566161+0000","mapping_epoch":432,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":433,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:44:20.028954+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:26:50.991712+0000","last_clean_scrub_stamp":"2024-03-11T21:44:20.028954+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,2],"acting":[1,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.f","version":"456'39","reported_seq":953,"reported_epoch":464,"state":"active+clean","last_fresh":"2024-03-11T21:46:16.293382+0000","last_change":"2024-03-11T21:45:17.566777+0000","last_active":"2024-03-11T21:46:16.293382+0000","last_peered":"2024-03-11T21:46:16.293382+0000","last_clean":"2024-03-11T21:46:16.293382+0000","last_became_active":"2024-03-11T21:44:35.630044+0000","last_became_peered":"2024-03-11T21:44:35.630044+0000","last_unstale":"2024-03-11T21:46:16.293382+0000","last_undegraded":"2024-03-11T21:46:16.293382+0000","last_fullsized":"2024-03-11T21:46:16.293382+0000","mapping_epoch":437,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":438,"parent":"0.0","parent_split_bits":0,"last_scrub":"260'27","last_scrub_stamp":"2024-03-11T21:45:17.566447+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:39.525554+0000","last_clean_scrub_stamp":"2024-03-11T21:45:17.566447+0000","objects_scrubbed":7,"log_size":39,"log_dups_size":0,"ondisk_log_size":39,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:46:22.135842+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":62,"num_read_kb":53,"num_write":40,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":27,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,5],"acting":[2,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"11.3","version":"464'7161","reported_seq":7205,"reported_epoch":464,"state":"active+clean","last_fresh":"2024-03-11T21:46:17.629494+0000","last_change":"2024-03-11T21:44:53.666633+0000","last_active":"2024-03-11T21:46:17.629494+0000","last_peered":"2024-03-11T21:46:17.629494+0000","last_clean":"2024-03-11T21:46:17.629494+0000","last_became_active":"2024-03-11T21:44:53.666323+0000","last_became_peered":"2024-03-11T21:44:53.666323+0000","last_unstale":"2024-03-11T21:46:17.629494+0000","last_undegraded":"2024-03-11T21:46:17.629494+0000","last_fullsized":"2024-03-11T21:46:17.629494+0000","mapping_epoch":446,"log_start":"454'3000","ondisk_log_start":"454'3000","created":446,"last_epoch_clean":447,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:44:52.644443+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:44:52.644443+0000","last_clean_scrub_stamp":"2024-03-11T21:44:52.644443+0000","objects_scrubbed":0,"log_size":4161,"log_dups_size":0,"ondisk_log_size":4161,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":469303296,"num_objects":7161,"num_object_clones":0,"num_object_copies":21483,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7161,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7161,"num_write_kb":458304,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,1],"acting":[6,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.e","version":"0'0","reported_seq":74,"reported_epoch":465,"state":"active+clean","last_fresh":"2024-03-11T21:46:18.573799+0000","last_change":"2024-03-11T21:45:17.769295+0000","last_active":"2024-03-11T21:46:18.573799+0000","last_peered":"2024-03-11T21:46:18.573799+0000","last_clean":"2024-03-11T21:46:18.573799+0000","last_became_active":"2024-03-11T21:44:35.627790+0000","last_became_peered":"2024-03-11T21:44:35.627790+0000","last_unstale":"2024-03-11T21:46:18.573799+0000","last_undegraded":"2024-03-11T21:46:18.573799+0000","last_fullsized":"2024-03-11T21:46:18.573799+0000","mapping_epoch":437,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":438,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:45:17.769045+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:43.559452+0000","last_clean_scrub_stamp":"2024-03-11T21:45:17.769045+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:46:29.536944+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,5],"acting":[1,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.d","version":"109'9","reported_seq":1117,"reported_epoch":464,"state":"active+clean","last_fresh":"2024-03-11T21:46:16.292948+0000","last_change":"2024-03-11T21:44:17.431824+0000","last_active":"2024-03-11T21:46:16.292948+0000","last_peered":"2024-03-11T21:46:16.292948+0000","last_clean":"2024-03-11T21:46:16.292948+0000","last_became_active":"2024-03-11T21:25:42.575583+0000","last_became_peered":"2024-03-11T21:25:42.575583+0000","last_unstale":"2024-03-11T21:46:16.292948+0000","last_undegraded":"2024-03-11T21:46:16.292948+0000","last_fullsized":"2024-03-11T21:46:16.292948+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"109'9","last_scrub_stamp":"2024-03-11T21:44:17.431687+0000","last_deep_scrub":"109'9","last_deep_scrub_stamp":"2024-03-11T21:28:27.082975+0000","last_clean_scrub_stamp":"2024-03-11T21:44:17.431687+0000","objects_scrubbed":1,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,5],"acting":[2,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.a","version":"0'0","reported_seq":899,"reported_epoch":465,"state":"active+clean","last_fresh":"2024-03-11T21:46:18.565186+0000","last_change":"2024-03-11T21:44:27.996603+0000","last_active":"2024-03-11T21:46:18.565186+0000","last_peered":"2024-03-11T21:46:18.565186+0000","last_clean":"2024-03-11T21:46:18.565186+0000","last_became_active":"2024-03-11T21:44:21.462886+0000","last_became_peered":"2024-03-11T21:44:21.462886+0000","last_unstale":"2024-03-11T21:46:18.565186+0000","last_undegraded":"2024-03-11T21:46:18.565186+0000","last_fullsized":"2024-03-11T21:46:18.565186+0000","mapping_epoch":432,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":433,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:44:27.996522+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:44:27.996522+0000","last_clean_scrub_stamp":"2024-03-11T21:44:27.996522+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,5],"acting":[1,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.c","version":"456'30","reported_seq":559,"reported_epoch":464,"state":"active+clean","last_fresh":"2024-03-11T21:46:16.289003+0000","last_change":"2024-03-11T21:45:18.628865+0000","last_active":"2024-03-11T21:46:16.289003+0000","last_peered":"2024-03-11T21:46:16.289003+0000","last_clean":"2024-03-11T21:46:16.289003+0000","last_became_active":"2024-03-11T21:44:35.630913+0000","last_became_peered":"2024-03-11T21:44:35.630913+0000","last_unstale":"2024-03-11T21:46:16.289003+0000","last_undegraded":"2024-03-11T21:46:16.289003+0000","last_fullsized":"2024-03-11T21:46:16.289003+0000","mapping_epoch":437,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":438,"parent":"0.0","parent_split_bits":0,"last_scrub":"260'20","last_scrub_stamp":"2024-03-11T21:45:18.628728+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:39.525554+0000","last_clean_scrub_stamp":"2024-03-11T21:45:18.628728+0000","objects_scrubbed":5,"log_size":30,"log_dups_size":0,"ondisk_log_size":30,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:46:40.024780+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":44,"num_read_kb":39,"num_write":30,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,7],"acting":[6,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.2","version":"464'6925","reported_seq":6971,"reported_epoch":464,"state":"active+clean","last_fresh":"2024-03-11T21:46:18.346261+0000","last_change":"2024-03-11T21:44:53.665467+0000","last_active":"2024-03-11T21:46:18.346261+0000","last_peered":"2024-03-11T21:46:18.346261+0000","last_clean":"2024-03-11T21:46:18.346261+0000","last_became_active":"2024-03-11T21:44:53.665142+0000","last_became_peered":"2024-03-11T21:44:53.665142+0000","last_unstale":"2024-03-11T21:46:18.346261+0000","last_undegraded":"2024-03-11T21:46:18.346261+0000","last_fullsized":"2024-03-11T21:46:18.346261+0000","mapping_epoch":446,"log_start":"454'2500","ondisk_log_start":"454'2500","created":446,"last_epoch_clean":447,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:44:52.644443+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:44:52.644443+0000","last_clean_scrub_stamp":"2024-03-11T21:44:52.644443+0000","objects_scrubbed":0,"log_size":4425,"log_dups_size":0,"ondisk_log_size":4425,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":453836800,"num_objects":6925,"num_object_clones":0,"num_object_copies":20775,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6925,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6925,"num_write_kb":443200,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,7],"acting":[2,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.f","version":"0'0","reported_seq":885,"reported_epoch":464,"state":"active+clean","last_fresh":"2024-03-11T21:46:16.293215+0000","last_change":"2024-03-11T21:46:14.767906+0000","last_active":"2024-03-11T21:46:16.293215+0000","last_peered":"2024-03-11T21:46:16.293215+0000","last_clean":"2024-03-11T21:46:16.293215+0000","last_became_active":"2024-03-11T21:44:35.645282+0000","last_became_peered":"2024-03-11T21:44:35.645282+0000","last_unstale":"2024-03-11T21:46:16.293215+0000","last_undegraded":"2024-03-11T21:46:16.293215+0000","last_fullsized":"2024-03-11T21:46:16.293215+0000","mapping_epoch":437,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":438,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:46:14.767787+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:27:03.010969+0000","last_clean_scrub_stamp":"2024-03-11T21:46:14.767787+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:47:37.501602+0000","scrub_duration":217,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6,1],"acting":[2,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":870,"reported_epoch":465,"state":"active+clean","last_fresh":"2024-03-11T21:46:18.566191+0000","last_change":"2024-03-11T21:45:18.824784+0000","last_active":"2024-03-11T21:46:18.566191+0000","last_peered":"2024-03-11T21:46:18.566191+0000","last_clean":"2024-03-11T21:46:18.566191+0000","last_became_active":"2024-03-11T21:44:35.619857+0000","last_became_peered":"2024-03-11T21:44:35.619857+0000","last_unstale":"2024-03-11T21:46:18.566191+0000","last_undegraded":"2024-03-11T21:46:18.566191+0000","last_fullsized":"2024-03-11T21:46:18.566191+0000","mapping_epoch":437,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":438,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:45:18.824654+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:30:22.750053+0000","last_clean_scrub_stamp":"2024-03-11T21:45:18.824654+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:46:34.946396+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,6],"acting":[1,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.b","version":"0'0","reported_seq":520,"reported_epoch":464,"state":"active+clean","last_fresh":"2024-03-11T21:46:16.299922+0000","last_change":"2024-03-11T21:45:30.102084+0000","last_active":"2024-03-11T21:46:16.299922+0000","last_peered":"2024-03-11T21:46:16.299922+0000","last_clean":"2024-03-11T21:46:16.299922+0000","last_became_active":"2024-03-11T21:44:35.639028+0000","last_became_peered":"2024-03-11T21:44:35.639028+0000","last_unstale":"2024-03-11T21:46:16.299922+0000","last_undegraded":"2024-03-11T21:46:16.299922+0000","last_fullsized":"2024-03-11T21:46:16.299922+0000","mapping_epoch":437,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":438,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:45:30.101973+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:45:30.101973+0000","last_clean_scrub_stamp":"2024-03-11T21:45:30.101973+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:46:42.132246+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,6],"acting":[7,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.d","version":"456'53","reported_seq":595,"reported_epoch":464,"state":"active+clean","last_fresh":"2024-03-11T21:46:16.293108+0000","last_change":"2024-03-11T21:44:35.671270+0000","last_active":"2024-03-11T21:46:16.293108+0000","last_peered":"2024-03-11T21:46:16.293108+0000","last_clean":"2024-03-11T21:46:16.293108+0000","last_became_active":"2024-03-11T21:44:35.634778+0000","last_became_peered":"2024-03-11T21:44:35.634778+0000","last_unstale":"2024-03-11T21:46:16.293108+0000","last_undegraded":"2024-03-11T21:46:16.293108+0000","last_fullsized":"2024-03-11T21:46:16.293108+0000","mapping_epoch":437,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":438,"parent":"0.0","parent_split_bits":0,"last_scrub":"260'39","last_scrub_stamp":"2024-03-11T21:44:30.048617+0000","last_deep_scrub":"165'25","last_deep_scrub_stamp":"2024-03-11T21:32:21.678328+0000","last_clean_scrub_stamp":"2024-03-11T21:44:30.048617+0000","objects_scrubbed":8,"log_size":53,"log_dups_size":0,"ondisk_log_size":53,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":70,"num_read_kb":60,"num_write":46,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,6],"acting":[2,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"11.5","version":"464'7082","reported_seq":7128,"reported_epoch":464,"state":"active+clean","last_fresh":"2024-03-11T21:46:17.558139+0000","last_change":"2024-03-11T21:44:53.668922+0000","last_active":"2024-03-11T21:46:17.558139+0000","last_peered":"2024-03-11T21:46:17.558139+0000","last_clean":"2024-03-11T21:46:17.558139+0000","last_became_active":"2024-03-11T21:44:53.668610+0000","last_became_peered":"2024-03-11T21:44:53.668610+0000","last_unstale":"2024-03-11T21:46:17.558139+0000","last_undegraded":"2024-03-11T21:46:17.558139+0000","last_fullsized":"2024-03-11T21:46:17.558139+0000","mapping_epoch":446,"log_start":"454'3100","ondisk_log_start":"454'3100","created":446,"last_epoch_clean":447,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:44:52.644443+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:44:52.644443+0000","last_clean_scrub_stamp":"2024-03-11T21:44:52.644443+0000","objects_scrubbed":0,"log_size":3982,"log_dups_size":0,"ondisk_log_size":3982,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":464125952,"num_objects":7082,"num_object_clones":0,"num_object_copies":21246,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7082,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7082,"num_write_kb":453248,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,6],"acting":[7,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":520,"reported_epoch":464,"state":"active+clean","last_fresh":"2024-03-11T21:46:16.299534+0000","last_change":"2024-03-11T21:46:05.407295+0000","last_active":"2024-03-11T21:46:16.299534+0000","last_peered":"2024-03-11T21:46:16.299534+0000","last_clean":"2024-03-11T21:46:16.299534+0000","last_became_active":"2024-03-11T21:44:21.461352+0000","last_became_peered":"2024-03-11T21:44:21.461352+0000","last_unstale":"2024-03-11T21:46:16.299534+0000","last_undegraded":"2024-03-11T21:46:16.299534+0000","last_fullsized":"2024-03-11T21:46:16.299534+0000","mapping_epoch":432,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":433,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:46:05.407166+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:28:37.561826+0000","last_clean_scrub_stamp":"2024-03-11T21:46:05.407166+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:47:12.927482+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,6],"acting":[7,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":868,"reported_epoch":464,"state":"active+clean","last_fresh":"2024-03-11T21:46:16.291978+0000","last_change":"2024-03-11T21:46:13.580411+0000","last_active":"2024-03-11T21:46:16.291978+0000","last_peered":"2024-03-11T21:46:16.291978+0000","last_clean":"2024-03-11T21:46:16.291978+0000","last_became_active":"2024-03-11T21:25:42.574515+0000","last_became_peered":"2024-03-11T21:25:42.574515+0000","last_unstale":"2024-03-11T21:46:16.291978+0000","last_undegraded":"2024-03-11T21:46:16.291978+0000","last_fullsized":"2024-03-11T21:46:16.291978+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:46:13.580254+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:41.544550+0000","last_clean_scrub_stamp":"2024-03-11T21:46:13.580254+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:47:23.008391+0000","scrub_duration":20,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,5],"acting":[2,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.c","version":"0'0","reported_seq":868,"reported_epoch":464,"state":"active+clean","last_fresh":"2024-03-11T21:46:16.292375+0000","last_change":"2024-03-11T21:45:15.556816+0000","last_active":"2024-03-11T21:46:16.292375+0000","last_peered":"2024-03-11T21:46:16.292375+0000","last_clean":"2024-03-11T21:46:16.292375+0000","last_became_active":"2024-03-11T21:25:38.533240+0000","last_became_peered":"2024-03-11T21:25:38.533240+0000","last_unstale":"2024-03-11T21:46:16.292375+0000","last_undegraded":"2024-03-11T21:46:16.292375+0000","last_fullsized":"2024-03-11T21:46:16.292375+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:45:15.556610+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:29:30.279910+0000","last_clean_scrub_stamp":"2024-03-11T21:45:15.556610+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:46:44.313224+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,0],"acting":[2,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.a","version":"456'60","reported_seq":1030,"reported_epoch":464,"state":"active+clean","last_fresh":"2024-03-11T21:46:16.289721+0000","last_change":"2024-03-11T21:45:32.673815+0000","last_active":"2024-03-11T21:46:16.289721+0000","last_peered":"2024-03-11T21:46:16.289721+0000","last_clean":"2024-03-11T21:46:16.289721+0000","last_became_active":"2024-03-11T21:34:41.148863+0000","last_became_peered":"2024-03-11T21:34:41.148863+0000","last_unstale":"2024-03-11T21:46:16.289721+0000","last_undegraded":"2024-03-11T21:46:16.289721+0000","last_fullsized":"2024-03-11T21:46:16.289721+0000","mapping_epoch":226,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":227,"parent":"0.0","parent_split_bits":0,"last_scrub":"260'44","last_scrub_stamp":"2024-03-11T21:45:32.673622+0000","last_deep_scrub":"260'44","last_deep_scrub_stamp":"2024-03-11T21:45:32.673622+0000","last_clean_scrub_stamp":"2024-03-11T21:45:32.673622+0000","objects_scrubbed":9,"log_size":60,"log_dups_size":0,"ondisk_log_size":60,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:46:52.167478+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":80,"num_read_kb":69,"num_write":52,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,7],"acting":[6,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.4","version":"464'7068","reported_seq":7114,"reported_epoch":464,"state":"active+clean","last_fresh":"2024-03-11T21:46:17.586258+0000","last_change":"2024-03-11T21:44:53.667742+0000","last_active":"2024-03-11T21:46:17.586258+0000","last_peered":"2024-03-11T21:46:17.586258+0000","last_clean":"2024-03-11T21:46:17.586258+0000","last_became_active":"2024-03-11T21:44:53.667386+0000","last_became_peered":"2024-03-11T21:44:53.667386+0000","last_unstale":"2024-03-11T21:46:17.586258+0000","last_undegraded":"2024-03-11T21:46:17.586258+0000","last_fullsized":"2024-03-11T21:46:17.586258+0000","mapping_epoch":446,"log_start":"454'3100","ondisk_log_start":"454'3100","created":446,"last_epoch_clean":447,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:44:52.644443+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:44:52.644443+0000","last_clean_scrub_stamp":"2024-03-11T21:44:52.644443+0000","objects_scrubbed":0,"log_size":3968,"log_dups_size":0,"ondisk_log_size":3968,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":463208448,"num_objects":7068,"num_object_clones":0,"num_object_copies":21204,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7068,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7068,"num_write_kb":452352,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,2],"acting":[7,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":872,"reported_epoch":465,"state":"active+clean","last_fresh":"2024-03-11T21:46:18.570337+0000","last_change":"2024-03-11T21:44:40.345505+0000","last_active":"2024-03-11T21:46:18.570337+0000","last_peered":"2024-03-11T21:46:18.570337+0000","last_clean":"2024-03-11T21:46:18.570337+0000","last_became_active":"2024-03-11T21:34:41.154507+0000","last_became_peered":"2024-03-11T21:34:41.154507+0000","last_unstale":"2024-03-11T21:46:18.570337+0000","last_undegraded":"2024-03-11T21:46:18.570337+0000","last_fullsized":"2024-03-11T21:46:18.570337+0000","mapping_epoch":226,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":227,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:44:40.345386+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:27:11.361913+0000","last_clean_scrub_stamp":"2024-03-11T21:44:40.345386+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":82,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":881,"reported_epoch":464,"state":"active+clean","last_fresh":"2024-03-11T21:46:16.291816+0000","last_change":"2024-03-11T21:45:20.568882+0000","last_active":"2024-03-11T21:46:16.291816+0000","last_peered":"2024-03-11T21:46:16.291816+0000","last_clean":"2024-03-11T21:46:16.291816+0000","last_became_active":"2024-03-11T21:44:35.631810+0000","last_became_peered":"2024-03-11T21:44:35.631810+0000","last_unstale":"2024-03-11T21:46:16.291816+0000","last_undegraded":"2024-03-11T21:46:16.291816+0000","last_fullsized":"2024-03-11T21:46:16.291816+0000","mapping_epoch":437,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":438,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:45:20.567110+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:43:32.115556+0000","last_clean_scrub_stamp":"2024-03-11T21:45:20.567110+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:46:28.021865+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.d","version":"0'0","reported_seq":893,"reported_epoch":465,"state":"active+clean","last_fresh":"2024-03-11T21:46:18.565481+0000","last_change":"2024-03-11T21:46:12.969419+0000","last_active":"2024-03-11T21:46:18.565481+0000","last_peered":"2024-03-11T21:46:18.565481+0000","last_clean":"2024-03-11T21:46:18.565481+0000","last_became_active":"2024-03-11T21:44:35.621273+0000","last_became_peered":"2024-03-11T21:44:35.621273+0000","last_unstale":"2024-03-11T21:46:18.565481+0000","last_undegraded":"2024-03-11T21:46:18.565481+0000","last_fullsized":"2024-03-11T21:46:18.565481+0000","mapping_epoch":437,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":438,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:46:12.969312+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:37.512018+0000","last_clean_scrub_stamp":"2024-03-11T21:46:12.969312+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:47:24.129060+0000","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,6],"acting":[1,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.b","version":"456'21","reported_seq":918,"reported_epoch":465,"state":"active+clean","last_fresh":"2024-03-11T21:46:18.567225+0000","last_change":"2024-03-11T21:45:30.172863+0000","last_active":"2024-03-11T21:46:18.567225+0000","last_peered":"2024-03-11T21:46:18.567225+0000","last_clean":"2024-03-11T21:46:18.567225+0000","last_became_active":"2024-03-11T21:44:35.619596+0000","last_became_peered":"2024-03-11T21:44:35.619596+0000","last_unstale":"2024-03-11T21:46:18.567225+0000","last_undegraded":"2024-03-11T21:46:18.567225+0000","last_fullsized":"2024-03-11T21:46:18.567225+0000","mapping_epoch":437,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":438,"parent":"0.0","parent_split_bits":0,"last_scrub":"260'15","last_scrub_stamp":"2024-03-11T21:45:30.172740+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:39.525554+0000","last_clean_scrub_stamp":"2024-03-11T21:45:30.172740+0000","objects_scrubbed":4,"log_size":21,"log_dups_size":0,"ondisk_log_size":21,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:46:52.798217+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":35,"num_read_kb":29,"num_write":22,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,5],"acting":[0,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.7","version":"464'6962","reported_seq":7008,"reported_epoch":464,"state":"active+clean","last_fresh":"2024-03-11T21:46:17.130624+0000","last_change":"2024-03-11T21:44:53.664917+0000","last_active":"2024-03-11T21:46:17.130624+0000","last_peered":"2024-03-11T21:46:17.130624+0000","last_clean":"2024-03-11T21:46:17.130624+0000","last_became_active":"2024-03-11T21:44:53.664598+0000","last_became_peered":"2024-03-11T21:44:53.664598+0000","last_unstale":"2024-03-11T21:46:17.130624+0000","last_undegraded":"2024-03-11T21:46:17.130624+0000","last_fullsized":"2024-03-11T21:46:17.130624+0000","mapping_epoch":446,"log_start":"454'3000","ondisk_log_start":"454'3000","created":446,"last_epoch_clean":447,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:44:52.644443+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:44:52.644443+0000","last_clean_scrub_stamp":"2024-03-11T21:44:52.644443+0000","objects_scrubbed":0,"log_size":3962,"log_dups_size":0,"ondisk_log_size":3962,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":456261632,"num_objects":6962,"num_object_clones":0,"num_object_copies":20886,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6962,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6962,"num_write_kb":445568,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":81,"reported_epoch":464,"state":"active+clean","last_fresh":"2024-03-11T21:46:16.299629+0000","last_change":"2024-03-11T21:45:34.114431+0000","last_active":"2024-03-11T21:46:16.299629+0000","last_peered":"2024-03-11T21:46:16.299629+0000","last_clean":"2024-03-11T21:46:16.299629+0000","last_became_active":"2024-03-11T21:44:21.449026+0000","last_became_peered":"2024-03-11T21:44:21.449026+0000","last_unstale":"2024-03-11T21:46:16.299629+0000","last_undegraded":"2024-03-11T21:46:16.299629+0000","last_fullsized":"2024-03-11T21:46:16.299629+0000","mapping_epoch":432,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":433,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:45:34.114276+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:43.559452+0000","last_clean_scrub_stamp":"2024-03-11T21:45:34.114276+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:46:38.750825+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,0],"acting":[5,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.e","version":"0'0","reported_seq":903,"reported_epoch":464,"state":"active+clean","last_fresh":"2024-03-11T21:46:16.292583+0000","last_change":"2024-03-11T21:44:28.438230+0000","last_active":"2024-03-11T21:46:16.292583+0000","last_peered":"2024-03-11T21:46:16.292583+0000","last_clean":"2024-03-11T21:46:16.292583+0000","last_became_active":"2024-03-11T21:44:21.457534+0000","last_became_peered":"2024-03-11T21:44:21.457534+0000","last_unstale":"2024-03-11T21:46:16.292583+0000","last_undegraded":"2024-03-11T21:46:16.292583+0000","last_fullsized":"2024-03-11T21:46:16.292583+0000","mapping_epoch":432,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":433,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:44:28.438139+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:37.512018+0000","last_clean_scrub_stamp":"2024-03-11T21:44:28.438139+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,6],"acting":[2,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.8","version":"456'46","reported_seq":977,"reported_epoch":464,"state":"active+clean","last_fresh":"2024-03-11T21:46:16.299091+0000","last_change":"2024-03-11T21:44:16.790769+0000","last_active":"2024-03-11T21:46:16.299091+0000","last_peered":"2024-03-11T21:46:16.299091+0000","last_clean":"2024-03-11T21:46:16.299091+0000","last_became_active":"2024-03-11T21:34:41.146631+0000","last_became_peered":"2024-03-11T21:34:41.146631+0000","last_unstale":"2024-03-11T21:46:16.299091+0000","last_undegraded":"2024-03-11T21:46:16.299091+0000","last_fullsized":"2024-03-11T21:46:16.299091+0000","mapping_epoch":226,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":227,"parent":"0.0","parent_split_bits":0,"last_scrub":"260'34","last_scrub_stamp":"2024-03-11T21:44:16.790655+0000","last_deep_scrub":"109'22","last_deep_scrub_stamp":"2024-03-11T21:32:07.573197+0000","last_clean_scrub_stamp":"2024-03-11T21:44:16.790655+0000","objects_scrubbed":7,"log_size":46,"log_dups_size":0,"ondisk_log_size":46,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":62,"num_read_kb":53,"num_write":40,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.9","version":"438'12","reported_seq":668,"reported_epoch":464,"state":"active+clean","last_fresh":"2024-03-11T21:46:16.302203+0000","last_change":"2024-03-11T21:45:27.165614+0000","last_active":"2024-03-11T21:46:16.302203+0000","last_peered":"2024-03-11T21:46:16.302203+0000","last_clean":"2024-03-11T21:46:16.302203+0000","last_became_active":"2024-03-11T21:44:35.638711+0000","last_became_peered":"2024-03-11T21:44:35.638711+0000","last_unstale":"2024-03-11T21:46:16.302203+0000","last_undegraded":"2024-03-11T21:46:16.302203+0000","last_fullsized":"2024-03-11T21:46:16.302203+0000","mapping_epoch":437,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":438,"parent":"0.0","parent_split_bits":0,"last_scrub":"438'12","last_scrub_stamp":"2024-03-11T21:45:27.165533+0000","last_deep_scrub":"94'9","last_deep_scrub_stamp":"2024-03-11T21:26:53.422552+0000","last_clean_scrub_stamp":"2024-03-11T21:45:27.165533+0000","objects_scrubbed":1,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:46:45.714196+0000","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,1],"acting":[7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":11,"num_pg":16,"stat_sum":{"num_bytes":7432962048,"num_objects":113418,"num_object_clones":0,"num_object_copies":340254,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":113418,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":113418,"num_write_kb":7258752,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":22347382784,"data_stored":22347382784,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":65618,"ondisk_log_size":65618,"up":48,"acting":48,"num_store_stats":6},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":1531,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":7,"num_read_kb":7,"num_write":10,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":2839,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":98304,"data_stored":6124,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":6,"ondisk_log_size":6,"up":96,"acting":96,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":198,"num_read_kb":294,"num_write":179,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":2951840,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":2375680,"data_stored":2361472,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":128,"ondisk_log_size":128,"up":3,"acting":3,"num_store_stats":4},{"poolid":3,"num_pg":3,"stat_sum":{"num_bytes":408,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1231,"num_read_kb":1226,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":38,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":1262,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":8,"ondisk_log_size":8,"up":9,"acting":9,"num_store_stats":8},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":3702,"num_objects":178,"num_object_clones":0,"num_object_copies":534,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":178,"num_whiteouts":0,"num_read":1545,"num_read_kb":1302,"num_write":989,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":296,"num_bytes_recovered":5132,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":544768,"data_stored":14478,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1062,"ondisk_log_size":1062,"up":96,"acting":96,"num_store_stats":8},{"poolid":5,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":84,"ondisk_log_size":84,"up":96,"acting":96,"num_store_stats":8},{"poolid":6,"num_pg":32,"stat_sum":{"num_bytes":383,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":396,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":1175,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":96,"acting":96,"num_store_stats":8}],"osd_stats":[{"osd":7,"up_from":60,"seq":257698038011,"num_pgs":76,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":4327036,"kb_used_data":3596148,"kb_used_omap":1,"kb_used_meta":730878,"kb_avail":89418116,"statfs":{"total":95995035648,"available":91564150784,"internally_reserved":0,"allocated":3682455552,"data_stored":3682000228,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":748419530},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,2],"upper_bound":4},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":50,"seq":214748365057,"num_pgs":73,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":4758372,"kb_used_data":4051096,"kb_used_omap":1,"kb_used_meta":707262,"kb_avail":88986780,"statfs":{"total":95995035648,"available":91122462720,"internally_reserved":0,"allocated":4148322304,"data_stored":4147832176,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":724236742},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,1,2,1],"upper_bound":16},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834575131,"num_pgs":77,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":5914136,"kb_used_data":4994448,"kb_used_omap":1,"kb_used_meta":919678,"kb_avail":87831016,"statfs":{"total":95995035648,"available":89938960384,"internally_reserved":0,"allocated":5114314752,"data_stored":5113830958,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":941750725},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[4,1,3,2,1],"upper_bound":32},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738656,"num_pgs":73,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":4539232,"kb_used_data":3659796,"kb_used_omap":1,"kb_used_meta":879422,"kb_avail":89205920,"statfs":{"total":95995035648,"available":91346862080,"internally_reserved":0,"allocated":3747631104,"data_stored":3747151016,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":900528581},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,0,0,1,1],"upper_bound":32},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411605,"num_pgs":68,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":3847940,"kb_used_data":3148984,"kb_used_omap":1,"kb_used_meta":698942,"kb_avail":89897212,"statfs":{"total":95995035648,"available":92054745088,"internally_reserved":0,"allocated":3224559616,"data_stored":3224101313,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":715717061},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":0,"seq":1189705941110,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":0,"seq":137438953738,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":41,"seq":176093659397,"num_pgs":77,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":3054544,"kb_used_data":2247304,"kb_used_omap":1,"kb_used_meta":807230,"kb_avail":90690608,"statfs":{"total":95995035648,"available":92867182592,"internally_reserved":0,"allocated":2301239296,"data_stored":2300763799,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":826603973},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,0,1],"upper_bound":64},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":1095,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":46,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1393,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":957,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":574,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":528,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1439,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":92,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":69632,"data_stored":1914,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":86016,"data_stored":2228,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":1458,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":1898,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":1474,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":69632,"data_stored":1870,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":81920,"data_stored":2162,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":1474,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":370,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":383,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":383,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":3758096384,"data_stored":3758096384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":5154930688,"data_stored":5154930688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":3253862400,"data_stored":3253862400,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":2307194880,"data_stored":2307194880,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4186963968,"data_stored":4186963968,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":3686334464,"data_stored":3686334464,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-11T21:46:21.966 INFO:tasks.thrashosds.thrasher:Setting 5.a to [7, 1, 4, 6, 2, 3] 2024-03-11T21:46:21.966 INFO:tasks.thrashosds.thrasher:cmd ['osd', 'pg-upmap-items', '5.a', '7', '1', '4', '6', '2', '3'] 2024-03-11T21:46:21.966 DEBUG:teuthology.orchestra.run.smithi092:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid 00db0016-dfed-11ee-95c8-87774f69a715 -- ceph osd pg-upmap-items 5.a 7 1 4 6 2 3 2024-03-11T21:46:21.967 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:46:22.002 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:46:22.002 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:46:22.003 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:46:22.103 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:46:22.143 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:46:22.144 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:46:22.144 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:46:22.245 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:46:22.268 INFO:tasks.radosbench.radosbench.0.smithi092.stdout: 75 16 118644 118628 98.8378 85.875 0.00887753 0.010113 2024-03-11T21:46:22.283 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:46:22.283 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:46:22.284 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:46:22.385 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:46:22.427 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:46:22.428 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:46:22.428 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:46:22.529 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:46:22.565 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:46:22.566 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:46:22.566 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:46:22.584 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:46:22 smithi097 ceph-mon[33317]: from='client.25900 v1:172.21.15.92:0/3790467460' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-11T21:46:22.584 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:46:22 smithi097 ceph-mon[33317]: purged_snaps scrub starts 2024-03-11T21:46:22.584 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:46:22 smithi097 ceph-mon[33317]: purged_snaps scrub ok 2024-03-11T21:46:22.618 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:46:22 smithi092 ceph-mon[29572]: from='client.25900 v1:172.21.15.92:0/3790467460' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-11T21:46:22.618 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:46:22 smithi092 ceph-mon[29572]: purged_snaps scrub starts 2024-03-11T21:46:22.618 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:46:22 smithi092 ceph-mon[29572]: purged_snaps scrub ok 2024-03-11T21:46:22.618 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:46:22 smithi092 ceph-mon[37563]: from='client.25900 v1:172.21.15.92:0/3790467460' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-11T21:46:22.618 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:46:22 smithi092 ceph-mon[37563]: purged_snaps scrub starts 2024-03-11T21:46:22.618 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:46:22 smithi092 ceph-mon[37563]: purged_snaps scrub ok 2024-03-11T21:46:22.667 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:46:22.705 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:46:22.706 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:46:22.706 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:46:22.807 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:46:22.853 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:46:22.853 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:46:22.853 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:46:22.954 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:46:22.996 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:46:22.996 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:46:22.997 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:46:23.096 DEBUG:teuthology.orchestra.run.smithi092:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid 00db0016-dfed-11ee-95c8-87774f69a715 -- ceph osd set noscrub 2024-03-11T21:46:23.098 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:46:23.142 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:46:23.142 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:46:23.143 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:46:23.244 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:46:23.275 INFO:tasks.radosbench.radosbench.0.smithi092.stdout: 76 16 119950 119934 98.6026 81.625 0.0232306 0.0101365 2024-03-11T21:46:23.303 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:46:23.303 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:46:23.303 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:46:23.313 INFO:teuthology.orchestra.run.smithi097.stdout:{ 2024-03-11T21:46:23.313 INFO:teuthology.orchestra.run.smithi097.stdout: "ops": [ 2024-03-11T21:46:23.313 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:46:23.313 INFO:teuthology.orchestra.run.smithi097.stdout: "description": "osd_op(client.35813.0:119968 11.f 11:f2d82bf9:::benchmark_data_smithi092_186520_object119967:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536 in=65536b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e467)", 2024-03-11T21:46:23.313 INFO:teuthology.orchestra.run.smithi097.stdout: "initiated_at": "2024-03-11T21:46:23.307503+0000", 2024-03-11T21:46:23.314 INFO:teuthology.orchestra.run.smithi097.stdout: "age": 0.014415317, 2024-03-11T21:46:23.314 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.014449205999999999, 2024-03-11T21:46:23.314 INFO:teuthology.orchestra.run.smithi097.stdout: "type_data": { 2024-03-11T21:46:23.314 INFO:teuthology.orchestra.run.smithi097.stdout: "flag_point": "waiting for sub ops", 2024-03-11T21:46:23.314 INFO:teuthology.orchestra.run.smithi097.stdout: "client_info": { 2024-03-11T21:46:23.314 INFO:teuthology.orchestra.run.smithi097.stdout: "client": "client.35813", 2024-03-11T21:46:23.314 INFO:teuthology.orchestra.run.smithi097.stdout: "client_addr": "v1:172.21.15.92:0/4293352030", 2024-03-11T21:46:23.314 INFO:teuthology.orchestra.run.smithi097.stdout: "tid": 119968 2024-03-11T21:46:23.314 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:46:23.314 INFO:teuthology.orchestra.run.smithi097.stdout: "events": [ 2024-03-11T21:46:23.314 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:46:23.314 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "initiated", 2024-03-11T21:46:23.315 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:46:23.307503+0000", 2024-03-11T21:46:23.315 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-03-11T21:46:23.315 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:46:23.315 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:46:23.315 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "header_read", 2024-03-11T21:46:23.315 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:46:23.307503+0000", 2024-03-11T21:46:23.315 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-03-11T21:46:23.315 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:46:23.315 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:46:23.315 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "throttled", 2024-03-11T21:46:23.315 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:46:23.307505+0000", 2024-03-11T21:46:23.315 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.1620000000000001e-06 2024-03-11T21:46:23.315 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:46:23.316 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:46:23.316 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "all_read", 2024-03-11T21:46:23.316 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:46:23.317178+0000", 2024-03-11T21:46:23.316 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.0096734239999999999 2024-03-11T21:46:23.316 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:46:23.316 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:46:23.316 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "dispatched", 2024-03-11T21:46:23.317 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:46:23.317190+0000", 2024-03-11T21:46:23.317 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.1497e-05 2024-03-11T21:46:23.317 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:46:23.317 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:46:23.317 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "queued_for_pg", 2024-03-11T21:46:23.317 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:46:23.317199+0000", 2024-03-11T21:46:23.317 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 9.8719999999999997e-06 2024-03-11T21:46:23.317 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:46:23.317 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:46:23.317 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "reached_pg", 2024-03-11T21:46:23.317 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:46:23.317255+0000", 2024-03-11T21:46:23.317 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 5.6030999999999999e-05 2024-03-11T21:46:23.318 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:46:23.318 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:46:23.318 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "started", 2024-03-11T21:46:23.318 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:46:23.317375+0000", 2024-03-11T21:46:23.318 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.000119653 2024-03-11T21:46:23.318 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:46:23.318 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:46:23.318 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "waiting for subops from 1,2", 2024-03-11T21:46:23.318 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:46:23.317517+0000", 2024-03-11T21:46:23.318 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.00014201700000000001 2024-03-11T21:46:23.318 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-03-11T21:46:23.318 INFO:teuthology.orchestra.run.smithi097.stdout: ] 2024-03-11T21:46:23.319 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-03-11T21:46:23.319 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:46:23.319 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:46:23.319 INFO:teuthology.orchestra.run.smithi097.stdout: "description": "osd_repop(client.35813.0:119958 11.3 e467/446 11:c0aa7128:::benchmark_data_smithi092_186520_object119957:head v 467'7576, mlcod=467'7574)", 2024-03-11T21:46:23.319 INFO:teuthology.orchestra.run.smithi097.stdout: "initiated_at": "2024-03-11T21:46:23.320532+0000", 2024-03-11T21:46:23.319 INFO:teuthology.orchestra.run.smithi097.stdout: "age": 0.0013867720000000001, 2024-03-11T21:46:23.319 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.0014993750000000001, 2024-03-11T21:46:23.319 INFO:teuthology.orchestra.run.smithi097.stdout: "type_data": { 2024-03-11T21:46:23.319 INFO:teuthology.orchestra.run.smithi097.stdout: "flag_point": "started", 2024-03-11T21:46:23.319 INFO:teuthology.orchestra.run.smithi097.stdout: "events": [ 2024-03-11T21:46:23.319 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:46:23.319 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "initiated", 2024-03-11T21:46:23.319 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:46:23.320532+0000", 2024-03-11T21:46:23.320 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-03-11T21:46:23.320 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:46:23.320 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:46:23.320 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "header_read", 2024-03-11T21:46:23.320 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:46:23.320532+0000", 2024-03-11T21:46:23.320 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-03-11T21:46:23.320 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:46:23.320 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:46:23.320 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "throttled", 2024-03-11T21:46:23.320 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:46:23.320534+0000", 2024-03-11T21:46:23.320 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.556e-06 2024-03-11T21:46:23.320 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:46:23.320 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:46:23.321 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "all_read", 2024-03-11T21:46:23.321 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:46:23.320563+0000", 2024-03-11T21:46:23.321 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 2.9048e-05 2024-03-11T21:46:23.321 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:46:23.321 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:46:23.321 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "dispatched", 2024-03-11T21:46:23.321 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:46:23.320575+0000", 2024-03-11T21:46:23.321 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.1989e-05 2024-03-11T21:46:23.321 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:46:23.321 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:46:23.321 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "queued_for_pg", 2024-03-11T21:46:23.321 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:46:23.320583+0000", 2024-03-11T21:46:23.322 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 8.2910000000000007e-06 2024-03-11T21:46:23.322 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:46:23.322 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:46:23.322 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "reached_pg", 2024-03-11T21:46:23.322 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:46:23.320639+0000", 2024-03-11T21:46:23.322 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 5.6042999999999999e-05 2024-03-11T21:46:23.322 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:46:23.322 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:46:23.322 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "started", 2024-03-11T21:46:23.322 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:46:23.320664+0000", 2024-03-11T21:46:23.322 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 2.4567e-05 2024-03-11T21:46:23.322 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-03-11T21:46:23.323 INFO:teuthology.orchestra.run.smithi097.stdout: ] 2024-03-11T21:46:23.323 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-03-11T21:46:23.323 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:46:23.323 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:46:23.323 INFO:teuthology.orchestra.run.smithi097.stdout: "description": "osd_repop_reply(client.35813.0:119968 11.f e467/446)", 2024-03-11T21:46:23.323 INFO:teuthology.orchestra.run.smithi097.stdout: "initiated_at": "2024-03-11T21:46:23.320638+0000", 2024-03-11T21:46:23.323 INFO:teuthology.orchestra.run.smithi097.stdout: "age": 0.0012805970000000001, 2024-03-11T21:46:23.323 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.0014471239999999999, 2024-03-11T21:46:23.323 INFO:teuthology.orchestra.run.smithi097.stdout: "type_data": { 2024-03-11T21:46:23.323 INFO:teuthology.orchestra.run.smithi097.stdout: "flag_point": "queued for pg", 2024-03-11T21:46:23.323 INFO:teuthology.orchestra.run.smithi097.stdout: "events": [ 2024-03-11T21:46:23.323 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:46:23.323 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "initiated", 2024-03-11T21:46:23.324 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:46:23.320638+0000", 2024-03-11T21:46:23.324 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-03-11T21:46:23.324 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:46:23.324 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:46:23.324 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "header_read", 2024-03-11T21:46:23.324 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:46:23.320638+0000", 2024-03-11T21:46:23.324 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-03-11T21:46:23.324 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:46:23.324 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:46:23.324 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "throttled", 2024-03-11T21:46:23.324 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:46:23.320639+0000", 2024-03-11T21:46:23.324 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 9.7699999999999992e-07 2024-03-11T21:46:23.324 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:46:23.325 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:46:23.325 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "all_read", 2024-03-11T21:46:23.325 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:46:23.320644+0000", 2024-03-11T21:46:23.325 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 4.5709999999999999e-06 2024-03-11T21:46:23.325 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:46:23.325 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:46:23.325 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "dispatched", 2024-03-11T21:46:23.325 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:46:23.320652+0000", 2024-03-11T21:46:23.325 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 7.9449999999999993e-06 2024-03-11T21:46:23.325 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:46:23.325 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:46:23.325 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "queued_for_pg", 2024-03-11T21:46:23.325 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:46:23.320657+0000", 2024-03-11T21:46:23.326 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 5.677e-06 2024-03-11T21:46:23.326 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-03-11T21:46:23.326 INFO:teuthology.orchestra.run.smithi097.stdout: ] 2024-03-11T21:46:23.326 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-03-11T21:46:23.326 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:46:23.326 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:46:23.326 INFO:teuthology.orchestra.run.smithi097.stdout: "description": "osd_repop_reply(client.35813.0:119968 11.f e467/446)", 2024-03-11T21:46:23.326 INFO:teuthology.orchestra.run.smithi097.stdout: "initiated_at": "2024-03-11T21:46:23.321758+0000", 2024-03-11T21:46:23.326 INFO:teuthology.orchestra.run.smithi097.stdout: "age": 0.00016036199999999999, 2024-03-11T21:46:23.326 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.00037550300000000001, 2024-03-11T21:46:23.326 INFO:teuthology.orchestra.run.smithi097.stdout: "type_data": { 2024-03-11T21:46:23.326 INFO:teuthology.orchestra.run.smithi097.stdout: "flag_point": "queued for pg", 2024-03-11T21:46:23.326 INFO:teuthology.orchestra.run.smithi097.stdout: "events": [ 2024-03-11T21:46:23.327 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:46:23.327 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "initiated", 2024-03-11T21:46:23.327 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:46:23.321758+0000", 2024-03-11T21:46:23.327 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-03-11T21:46:23.327 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:46:23.327 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:46:23.327 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "header_read", 2024-03-11T21:46:23.327 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:46:23.321758+0000", 2024-03-11T21:46:23.327 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-03-11T21:46:23.327 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:46:23.327 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:46:23.327 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "throttled", 2024-03-11T21:46:23.328 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:46:23.321760+0000", 2024-03-11T21:46:23.328 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.057e-06 2024-03-11T21:46:23.328 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:46:23.328 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:46:23.328 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "all_read", 2024-03-11T21:46:23.328 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:46:23.321764+0000", 2024-03-11T21:46:23.328 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 4.1829999999999996e-06 2024-03-11T21:46:23.328 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:46:23.328 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:46:23.328 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "dispatched", 2024-03-11T21:46:23.328 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:46:23.321776+0000", 2024-03-11T21:46:23.328 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.2345e-05 2024-03-11T21:46:23.328 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:46:23.329 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:46:23.329 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "queued_for_pg", 2024-03-11T21:46:23.329 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:46:23.321783+0000", 2024-03-11T21:46:23.329 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 6.5529999999999998e-06 2024-03-11T21:46:23.329 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-03-11T21:46:23.329 INFO:teuthology.orchestra.run.smithi097.stdout: ] 2024-03-11T21:46:23.329 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-03-11T21:46:23.329 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-03-11T21:46:23.329 INFO:teuthology.orchestra.run.smithi097.stdout: ], 2024-03-11T21:46:23.329 INFO:teuthology.orchestra.run.smithi097.stdout: "num_ops": 4 2024-03-11T21:46:23.329 INFO:teuthology.orchestra.run.smithi097.stdout:} 2024-03-11T21:46:23.400 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:46:23 smithi092 ceph-mon[29572]: pgmap v982: 148 pgs: 148 active+clean; 6.9 GiB data, 25 GiB used, 511 GiB / 536 GiB avail; 853 B/s rd, 67 MiB/s wr, 1.07k op/s 2024-03-11T21:46:23.400 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:46:23 smithi092 ceph-mon[29572]: osdmap e467: 8 total, 8 up, 6 in 2024-03-11T21:46:23.400 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:46:23 smithi092 ceph-mon[29572]: 6.1c deep-scrub starts 2024-03-11T21:46:23.400 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:46:23 smithi092 ceph-mon[29572]: 6.1c deep-scrub ok 2024-03-11T21:46:23.400 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:46:23 smithi092 ceph-mon[29572]: from='mgr.14607 ' entity='mgr.y' 2024-03-11T21:46:23.400 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:46:23 smithi092 ceph-mon[29572]: from='mgr.14607 v1:172.21.15.92:0/1702717597' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T21:46:23.400 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:46:23 smithi092 ceph-mon[37563]: pgmap v982: 148 pgs: 148 active+clean; 6.9 GiB data, 25 GiB used, 511 GiB / 536 GiB avail; 853 B/s rd, 67 MiB/s wr, 1.07k op/s 2024-03-11T21:46:23.400 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:46:23 smithi092 ceph-mon[37563]: osdmap e467: 8 total, 8 up, 6 in 2024-03-11T21:46:23.400 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:46:23 smithi092 ceph-mon[37563]: 6.1c deep-scrub starts 2024-03-11T21:46:23.400 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:46:23 smithi092 ceph-mon[37563]: 6.1c deep-scrub ok 2024-03-11T21:46:23.401 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:46:23 smithi092 ceph-mon[37563]: from='mgr.14607 ' entity='mgr.y' 2024-03-11T21:46:23.401 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:46:23 smithi092 ceph-mon[37563]: from='mgr.14607 v1:172.21.15.92:0/1702717597' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T21:46:23.404 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:46:23.438 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:46:23.439 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:46:23.439 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:46:23.540 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:46:23.575 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:46:23.576 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:46:23.576 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:46:23.584 INFO:journalctl@ceph.iscsi.iscsi.a.smithi097.stdout:Mar 11 21:46:23 smithi097 ceph-00db0016-dfed-11ee-95c8-87774f69a715-iscsi-iscsi-a[60943]: debug there is no tcmu-runner data available 2024-03-11T21:46:23.584 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:46:23 smithi097 ceph-mon[33317]: pgmap v982: 148 pgs: 148 active+clean; 6.9 GiB data, 25 GiB used, 511 GiB / 536 GiB avail; 853 B/s rd, 67 MiB/s wr, 1.07k op/s 2024-03-11T21:46:23.585 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:46:23 smithi097 ceph-mon[33317]: osdmap e467: 8 total, 8 up, 6 in 2024-03-11T21:46:23.585 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:46:23 smithi097 ceph-mon[33317]: 6.1c deep-scrub starts 2024-03-11T21:46:23.585 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:46:23 smithi097 ceph-mon[33317]: 6.1c deep-scrub ok 2024-03-11T21:46:23.585 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:46:23 smithi097 ceph-mon[33317]: from='mgr.14607 ' entity='mgr.y' 2024-03-11T21:46:23.585 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:46:23 smithi097 ceph-mon[33317]: from='mgr.14607 v1:172.21.15.92:0/1702717597' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T21:46:23.605 INFO:teuthology.orchestra.run.smithi092.stderr:Inferring config /var/lib/ceph/00db0016-dfed-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T21:46:23.614 INFO:teuthology.orchestra.run.smithi092.stderr:Inferring config /var/lib/ceph/00db0016-dfed-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T21:46:23.653 INFO:teuthology.orchestra.run.smithi092.stdout:osd.0: {} 2024-03-11T21:46:23.678 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:46:23.700 INFO:teuthology.orchestra.run.smithi092.stdout:osd.1: {} 2024-03-11T21:46:23.731 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:46:23.731 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:46:23.731 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:46:23.751 INFO:teuthology.orchestra.run.smithi092.stdout:osd.2: {} 2024-03-11T21:46:23.787 INFO:teuthology.orchestra.run.smithi092.stdout:osd.3: {} 2024-03-11T21:46:23.831 INFO:teuthology.orchestra.run.smithi092.stdout:osd.4: {} 2024-03-11T21:46:23.833 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:46:23.866 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:46:23.867 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:46:23.867 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:46:23.873 INFO:teuthology.orchestra.run.smithi092.stdout:osd.5: {} 2024-03-11T21:46:23.914 INFO:teuthology.orchestra.run.smithi092.stdout:osd.6: {} 2024-03-11T21:46:23.957 INFO:teuthology.orchestra.run.smithi092.stdout:osd.7: {} 2024-03-11T21:46:23.962 INFO:teuthology.orchestra.run.smithi092.stderr:osd.0: osd_enable_op_tracker = '' 2024-03-11T21:46:23.963 INFO:teuthology.orchestra.run.smithi092.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-11T21:46:23.963 INFO:teuthology.orchestra.run.smithi092.stderr:osd.2: osd_enable_op_tracker = '' 2024-03-11T21:46:23.963 INFO:teuthology.orchestra.run.smithi092.stderr:osd.3: osd_enable_op_tracker = '' 2024-03-11T21:46:23.963 INFO:teuthology.orchestra.run.smithi092.stderr:osd.4: osd_enable_op_tracker = '' 2024-03-11T21:46:23.963 INFO:teuthology.orchestra.run.smithi092.stderr:osd.5: osd_enable_op_tracker = '' 2024-03-11T21:46:23.963 INFO:teuthology.orchestra.run.smithi092.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-11T21:46:23.963 INFO:teuthology.orchestra.run.smithi092.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-11T21:46:23.968 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:46:24.011 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:46:24.013 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:46:24.013 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:46:24.114 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:46:24.121 DEBUG:teuthology.orchestra.run.smithi097:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid 00db0016-dfed-11ee-95c8-87774f69a715 -- ceph daemon osd.7 dump_blocked_ops 2024-03-11T21:46:24.154 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:46:24.155 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:46:24.155 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:46:24.256 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:46:24.275 INFO:tasks.radosbench.radosbench.0.smithi092.stdout: 77 16 121172 121156 98.3139 76.375 0.0115933 0.0101669 2024-03-11T21:46:24.294 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:46:24.294 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:46:24.295 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:46:24.334 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:46:24 smithi097 ceph-mon[33317]: from='client.24469 v1:172.21.15.97:0/404208310' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:46:24.334 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:46:24 smithi097 ceph-mon[33317]: from='client.? v1:172.21.15.92:0/1731338064' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-11T21:46:24.395 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:46:24.449 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:46:24.449 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:46:24.450 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:46:24.522 INFO:teuthology.orchestra.run.smithi097.stderr:Inferring config /var/lib/ceph/00db0016-dfed-11ee-95c8-87774f69a715/mon.b/config 2024-03-11T21:46:24.551 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:46:24.588 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:46:24.588 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:46:24.589 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:46:24.618 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:46:24 smithi092 ceph-mon[29572]: from='client.24469 v1:172.21.15.97:0/404208310' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:46:24.618 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:46:24 smithi092 ceph-mon[29572]: from='client.? v1:172.21.15.92:0/1731338064' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-11T21:46:24.618 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:46:24 smithi092 ceph-mon[37563]: from='client.24469 v1:172.21.15.97:0/404208310' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:46:24.619 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:46:24 smithi092 ceph-mon[37563]: from='client.? v1:172.21.15.92:0/1731338064' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-11T21:46:24.689 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:46:24.729 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:46:24.730 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:46:24.731 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:46:24.832 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:46:24.872 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:46:24.872 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:46:24.873 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:46:24.973 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:46:25.015 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:46:25.015 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:46:25.015 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:46:25.116 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:46:25.153 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:46:25.154 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:46:25.154 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:46:25.255 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:46:25.276 INFO:tasks.radosbench.radosbench.0.smithi092.stdout: 78 16 122620 122604 98.2129 90.5 0.00527249 0.0101776 2024-03-11T21:46:25.294 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:46:25.295 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:46:25.295 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:46:25.396 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:46:25.428 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:46:25.428 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:46:25.429 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:46:25.530 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:46:25.573 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:46:25.574 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:46:25.575 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:46:25.584 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:46:25 smithi097 ceph-mon[33317]: pgmap v984: 148 pgs: 148 active+clean; 7.2 GiB data, 26 GiB used, 510 GiB / 536 GiB avail; 853 B/s rd, 107 MiB/s wr, 1.72k op/s 2024-03-11T21:46:25.584 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:46:25 smithi097 ceph-mon[33317]: 5.13 scrub starts 2024-03-11T21:46:25.584 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:46:25 smithi097 ceph-mon[33317]: 5.13 scrub ok 2024-03-11T21:46:25.584 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:46:25 smithi097 ceph-mon[33317]: 5.10 scrub starts 2024-03-11T21:46:25.585 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:46:25 smithi097 ceph-mon[33317]: 5.10 scrub ok 2024-03-11T21:46:25.618 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:46:25 smithi092 ceph-mon[29572]: pgmap v984: 148 pgs: 148 active+clean; 7.2 GiB data, 26 GiB used, 510 GiB / 536 GiB avail; 853 B/s rd, 107 MiB/s wr, 1.72k op/s 2024-03-11T21:46:25.618 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:46:25 smithi092 ceph-mon[29572]: 5.13 scrub starts 2024-03-11T21:46:25.618 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:46:25 smithi092 ceph-mon[29572]: 5.13 scrub ok 2024-03-11T21:46:25.618 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:46:25 smithi092 ceph-mon[29572]: 5.10 scrub starts 2024-03-11T21:46:25.619 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:46:25 smithi092 ceph-mon[29572]: 5.10 scrub ok 2024-03-11T21:46:25.619 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:46:25 smithi092 ceph-mon[37563]: pgmap v984: 148 pgs: 148 active+clean; 7.2 GiB data, 26 GiB used, 510 GiB / 536 GiB avail; 853 B/s rd, 107 MiB/s wr, 1.72k op/s 2024-03-11T21:46:25.619 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:46:25 smithi092 ceph-mon[37563]: 5.13 scrub starts 2024-03-11T21:46:25.619 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:46:25 smithi092 ceph-mon[37563]: 5.13 scrub ok 2024-03-11T21:46:25.619 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:46:25 smithi092 ceph-mon[37563]: 5.10 scrub starts 2024-03-11T21:46:25.619 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:46:25 smithi092 ceph-mon[37563]: 5.10 scrub ok 2024-03-11T21:46:25.675 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:46:25.717 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:46:25.718 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:46:25.719 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:46:25.820 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:46:25.862 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:46:25.862 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:46:25.863 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:46:25.963 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:46:25.996 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:46:25.996 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:46:25.997 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:46:26.098 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:46:26.131 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:46:26.132 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:46:26.132 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:46:26.233 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:46:26.262 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:46:26.263 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:46:26.263 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:46:26.276 INFO:tasks.radosbench.radosbench.0.smithi092.stdout: 79 16 123998 123982 98.0598 86.125 0.0462306 0.010193 2024-03-11T21:46:26.364 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:46:26.395 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:46:26.396 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:46:26.396 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:46:26.497 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:46:26.532 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:46:26.533 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:46:26.533 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:46:26.584 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:46:26 smithi097 ceph-mon[33317]: 4.d scrub starts 2024-03-11T21:46:26.584 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:46:26 smithi097 ceph-mon[33317]: 4.d scrub ok 2024-03-11T21:46:26.584 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:46:26 smithi097 ceph-mon[33317]: 2.a scrub starts 2024-03-11T21:46:26.584 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:46:26 smithi097 ceph-mon[33317]: 2.a scrub ok 2024-03-11T21:46:26.585 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:46:26 smithi097 ceph-mon[33317]: 6.3 deep-scrub starts 2024-03-11T21:46:26.585 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:46:26 smithi097 ceph-mon[33317]: 6.3 deep-scrub ok 2024-03-11T21:46:26.618 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:46:26 smithi092 ceph-mon[29572]: 4.d scrub starts 2024-03-11T21:46:26.618 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:46:26 smithi092 ceph-mon[29572]: 4.d scrub ok 2024-03-11T21:46:26.618 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:46:26 smithi092 ceph-mon[29572]: 2.a scrub starts 2024-03-11T21:46:26.618 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:46:26 smithi092 ceph-mon[29572]: 2.a scrub ok 2024-03-11T21:46:26.618 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:46:26 smithi092 ceph-mon[29572]: 6.3 deep-scrub starts 2024-03-11T21:46:26.618 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:46:26 smithi092 ceph-mon[29572]: 6.3 deep-scrub ok 2024-03-11T21:46:26.618 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:46:26 smithi092 ceph-mon[37563]: 4.d scrub starts 2024-03-11T21:46:26.618 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:46:26 smithi092 ceph-mon[37563]: 4.d scrub ok 2024-03-11T21:46:26.618 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:46:26 smithi092 ceph-mon[37563]: 2.a scrub starts 2024-03-11T21:46:26.619 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:46:26 smithi092 ceph-mon[37563]: 2.a scrub ok 2024-03-11T21:46:26.619 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:46:26 smithi092 ceph-mon[37563]: 6.3 deep-scrub starts 2024-03-11T21:46:26.619 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:46:26 smithi092 ceph-mon[37563]: 6.3 deep-scrub ok 2024-03-11T21:46:26.634 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:46:26.661 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:46:26.661 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:46:26.662 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:46:26.762 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:46:26.793 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:46:26.794 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:46:26.795 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:46:26.895 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:46:26.936 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:46:26.937 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:46:26.937 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:46:26.962 INFO:teuthology.orchestra.run.smithi097.stdout:ERROR: (22) Invalid argument 2024-03-11T21:46:26.962 INFO:teuthology.orchestra.run.smithi097.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-11T21:46:27.037 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:46:27.065 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:46:27.065 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:46:27.065 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:46:27.166 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:46:27.203 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:46:27.204 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:46:27.204 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:46:27.276 INFO:tasks.radosbench.radosbench.0.smithi092.stdout:2024-03-11T21:46:27.286807+0000 min lat: 0.00128848 max lat: 0.231127 avg lat: 0.0102218 2024-03-11T21:46:27.276 INFO:tasks.radosbench.radosbench.0.smithi092.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-03-11T21:46:27.276 INFO:tasks.radosbench.radosbench.0.smithi092.stdout: 80 16 125224 125208 97.7919 76.625 0.00688746 0.0102218 2024-03-11T21:46:27.305 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:46:27.359 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:46:27.359 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:46:27.360 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:46:27.439 DEBUG:teuthology.orchestra.run.smithi092:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid 00db0016-dfed-11ee-95c8-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-03-11T21:46:27.460 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:46:27.493 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:46:27.494 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:46:27.494 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:46:27.579 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:46:27 smithi092 ceph-mon[29572]: pgmap v985: 148 pgs: 148 active+clean; 7.3 GiB data, 26 GiB used, 510 GiB / 536 GiB avail; 1.4 KiB/s rd, 75 MiB/s wr, 1.20k op/s 2024-03-11T21:46:27.579 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:46:27 smithi092 ceph-mon[29572]: 6.7 scrub starts 2024-03-11T21:46:27.579 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:46:27 smithi092 ceph-mon[29572]: 6.7 scrub ok 2024-03-11T21:46:27.579 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:46:27 smithi092 ceph-mon[37563]: pgmap v985: 148 pgs: 148 active+clean; 7.3 GiB data, 26 GiB used, 510 GiB / 536 GiB avail; 1.4 KiB/s rd, 75 MiB/s wr, 1.20k op/s 2024-03-11T21:46:27.579 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:46:27 smithi092 ceph-mon[37563]: 6.7 scrub starts 2024-03-11T21:46:27.579 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:46:27 smithi092 ceph-mon[37563]: 6.7 scrub ok 2024-03-11T21:46:27.584 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:46:27 smithi097 ceph-mon[33317]: pgmap v985: 148 pgs: 148 active+clean; 7.3 GiB data, 26 GiB used, 510 GiB / 536 GiB avail; 1.4 KiB/s rd, 75 MiB/s wr, 1.20k op/s 2024-03-11T21:46:27.584 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:46:27 smithi097 ceph-mon[33317]: 6.7 scrub starts 2024-03-11T21:46:27.584 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:46:27 smithi097 ceph-mon[33317]: 6.7 scrub ok 2024-03-11T21:46:27.595 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:46:27.625 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:46:27.626 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:46:27.626 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:46:27.727 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:46:27.782 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:46:27.783 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:46:27.783 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:46:27.884 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:46:27.942 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:46:27.943 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:46:27.943 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:46:27.963 DEBUG:teuthology.orchestra.run.smithi097:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid 00db0016-dfed-11ee-95c8-87774f69a715 -- ceph daemon osd.7 dump_historic_ops 2024-03-11T21:46:28.044 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:46:28.078 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:46:28.079 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:46:28.079 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:46:28.179 INFO:journalctl@ceph.mgr.y.smithi092.stdout:Mar 11 21:46:27 smithi092 ceph-00db0016-dfed-11ee-95c8-87774f69a715-mgr-y[29801]: ::ffff:172.21.15.97 - - [11/Mar/2024:21:46:27] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-11T21:46:28.180 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:46:28.222 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:46:28.223 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:46:28.223 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:46:28.276 INFO:tasks.radosbench.radosbench.0.smithi092.stdout: 81 16 126452 126436 97.532 76.75 0.00994616 0.0102486 2024-03-11T21:46:28.323 INFO:teuthology.orchestra.run.smithi097.stderr:Inferring config /var/lib/ceph/00db0016-dfed-11ee-95c8-87774f69a715/mon.b/config 2024-03-11T21:46:28.324 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:46:28.354 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:46:28.355 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:46:28.355 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:46:28.388 INFO:teuthology.orchestra.run.smithi092.stderr:Inferring config /var/lib/ceph/00db0016-dfed-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T21:46:28.456 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:46:28.519 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:46:28.520 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:46:28.520 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:46:28.526 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:46:28 smithi092 ceph-mon[29572]: 4.3 scrub starts 2024-03-11T21:46:28.526 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:46:28 smithi092 ceph-mon[29572]: 4.3 scrub ok 2024-03-11T21:46:28.526 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:46:28 smithi092 ceph-mon[29572]: 5.4 scrub starts 2024-03-11T21:46:28.526 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:46:28 smithi092 ceph-mon[29572]: 5.4 scrub ok 2024-03-11T21:46:28.527 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:46:28 smithi092 ceph-mon[37563]: 4.3 scrub starts 2024-03-11T21:46:28.527 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:46:28 smithi092 ceph-mon[37563]: 4.3 scrub ok 2024-03-11T21:46:28.527 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:46:28 smithi092 ceph-mon[37563]: 5.4 scrub starts 2024-03-11T21:46:28.527 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:46:28 smithi092 ceph-mon[37563]: 5.4 scrub ok 2024-03-11T21:46:28.621 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:46:28.632 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:46:28 smithi097 ceph-mon[33317]: 4.3 scrub starts 2024-03-11T21:46:28.633 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:46:28 smithi097 ceph-mon[33317]: 4.3 scrub ok 2024-03-11T21:46:28.633 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:46:28 smithi097 ceph-mon[33317]: 5.4 scrub starts 2024-03-11T21:46:28.633 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:46:28 smithi097 ceph-mon[33317]: 5.4 scrub ok 2024-03-11T21:46:28.658 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:46:28.659 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:46:28.659 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:46:28.760 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:46:28.810 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:46:28.810 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:46:28.810 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:46:28.911 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:46:28.944 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:46:28.945 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:46:28.945 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:46:29.046 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:46:29.078 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:46:29.078 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:46:29.079 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:46:29.180 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:46:29.213 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:46:29.213 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:46:29.214 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:46:29.276 INFO:tasks.radosbench.radosbench.0.smithi092.stdout: 82 16 127771 127755 97.3479 82.4375 0.00992467 0.0102685 2024-03-11T21:46:29.314 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:46:29.351 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:46:29.351 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:46:29.352 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:46:29.364 INFO:teuthology.orchestra.run.smithi092.stderr:set 5.a pg_upmap_items mapping to [7->1,4->6,2->3] 2024-03-11T21:46:29.368 INFO:teuthology.orchestra.run.smithi092.stderr:noscrub is set 2024-03-11T21:46:29.452 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:46:29.489 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:46:29.489 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:46:29.490 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:46:29.590 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:46:29.618 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:46:29 smithi092 ceph-mon[29572]: pgmap v986: 148 pgs: 148 active+clean; 7.3 GiB data, 27 GiB used, 510 GiB / 536 GiB avail; 639 B/s rd, 55 MiB/s wr, 883 op/s 2024-03-11T21:46:29.619 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:46:29 smithi092 ceph-mon[29572]: 4.9 deep-scrub starts 2024-03-11T21:46:29.619 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:46:29 smithi092 ceph-mon[29572]: 4.9 deep-scrub ok 2024-03-11T21:46:29.619 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:46:29 smithi092 ceph-mon[29572]: from='client.? v1:172.21.15.92:0/4275739545' entity='client.admin' cmd=[{"prefix": "osd pg-upmap-items", "pgid": "5.a", "id": [7, 1, 4, 6, 2, 3]}]: dispatch 2024-03-11T21:46:29.619 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:46:29 smithi092 ceph-mon[29572]: 2.1d scrub starts 2024-03-11T21:46:29.619 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:46:29 smithi092 ceph-mon[29572]: 2.1d scrub ok 2024-03-11T21:46:29.619 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:46:29 smithi092 ceph-mon[29572]: from='client.? v1:172.21.15.92:0/842349208' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-11T21:46:29.619 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:46:29 smithi092 ceph-mon[29572]: from='client.25918 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-11T21:46:29.619 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:46:29 smithi092 ceph-mon[37563]: pgmap v986: 148 pgs: 148 active+clean; 7.3 GiB data, 27 GiB used, 510 GiB / 536 GiB avail; 639 B/s rd, 55 MiB/s wr, 883 op/s 2024-03-11T21:46:29.619 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:46:29 smithi092 ceph-mon[37563]: 4.9 deep-scrub starts 2024-03-11T21:46:29.619 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:46:29 smithi092 ceph-mon[37563]: 4.9 deep-scrub ok 2024-03-11T21:46:29.620 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:46:29 smithi092 ceph-mon[37563]: from='client.? v1:172.21.15.92:0/4275739545' entity='client.admin' cmd=[{"prefix": "osd pg-upmap-items", "pgid": "5.a", "id": [7, 1, 4, 6, 2, 3]}]: dispatch 2024-03-11T21:46:29.620 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:46:29 smithi092 ceph-mon[37563]: 2.1d scrub starts 2024-03-11T21:46:29.620 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:46:29 smithi092 ceph-mon[37563]: 2.1d scrub ok 2024-03-11T21:46:29.620 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:46:29 smithi092 ceph-mon[37563]: from='client.? v1:172.21.15.92:0/842349208' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-11T21:46:29.620 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:46:29 smithi092 ceph-mon[37563]: from='client.25918 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-11T21:46:29.631 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:46:29.631 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:46:29.632 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:46:29.733 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:46:29.774 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:46:29.774 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:46:29.775 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:46:29.834 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:46:29 smithi097 ceph-mon[33317]: pgmap v986: 148 pgs: 148 active+clean; 7.3 GiB data, 27 GiB used, 510 GiB / 536 GiB avail; 639 B/s rd, 55 MiB/s wr, 883 op/s 2024-03-11T21:46:29.834 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:46:29 smithi097 ceph-mon[33317]: 4.9 deep-scrub starts 2024-03-11T21:46:29.834 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:46:29 smithi097 ceph-mon[33317]: 4.9 deep-scrub ok 2024-03-11T21:46:29.834 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:46:29 smithi097 ceph-mon[33317]: from='client.? v1:172.21.15.92:0/4275739545' entity='client.admin' cmd=[{"prefix": "osd pg-upmap-items", "pgid": "5.a", "id": [7, 1, 4, 6, 2, 3]}]: dispatch 2024-03-11T21:46:29.835 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:46:29 smithi097 ceph-mon[33317]: 2.1d scrub starts 2024-03-11T21:46:29.835 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:46:29 smithi097 ceph-mon[33317]: 2.1d scrub ok 2024-03-11T21:46:29.835 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:46:29 smithi097 ceph-mon[33317]: from='client.? v1:172.21.15.92:0/842349208' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-11T21:46:29.835 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:46:29 smithi097 ceph-mon[33317]: from='client.25918 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-11T21:46:29.875 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:46:29.910 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:46:29.911 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:46:29.911 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:46:30.012 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:46:30.053 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:46:30.053 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:46:30.053 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:46:30.154 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:46:30.203 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:46:30.203 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:46:30.203 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:46:30.276 INFO:tasks.radosbench.radosbench.0.smithi092.stdout: 83 16 129165 129149 97.2247 87.125 0.00630929 0.0102814 2024-03-11T21:46:30.305 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:46:30.342 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:46:30.388 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:46:30.388 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:46:30.467 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:46:30 smithi097 ceph-mon[33317]: 5.19 scrub starts 2024-03-11T21:46:30.467 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:46:30 smithi097 ceph-mon[33317]: 5.19 scrub ok 2024-03-11T21:46:30.467 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:46:30 smithi097 ceph-mon[33317]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-03-11T21:46:30.467 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:46:30 smithi097 ceph-mon[33317]: from='client.? v1:172.21.15.92:0/4275739545' entity='client.admin' cmd='[{"prefix": "osd pg-upmap-items", "pgid": "5.a", "id": [7, 1, 4, 6, 2, 3]}]': finished 2024-03-11T21:46:30.467 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:46:30 smithi097 ceph-mon[33317]: from='client.25918 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-03-11T21:46:30.467 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:46:30 smithi097 ceph-mon[33317]: osdmap e468: 8 total, 8 up, 6 in 2024-03-11T21:46:30.467 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:46:30 smithi097 ceph-mon[33317]: pgmap v988: 148 pgs: 148 active+clean; 7.7 GiB data, 27 GiB used, 509 GiB / 536 GiB avail; 1.2 KiB/s rd, 97 MiB/s wr, 1.55k op/s 2024-03-11T21:46:30.489 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:46:30.527 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:46:30.527 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:46:30.528 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:46:30.629 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:46:30.672 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:46:30.673 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:46:30.673 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:46:30.706 INFO:teuthology.orchestra.run.smithi097.stdout:ERROR: (22) Invalid argument 2024-03-11T21:46:30.707 INFO:teuthology.orchestra.run.smithi097.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-11T21:46:30.774 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:46:30.814 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:46:30.815 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:46:30.815 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:46:30.868 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:46:30 smithi092 ceph-mon[29572]: 5.19 scrub starts 2024-03-11T21:46:30.868 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:46:30 smithi092 ceph-mon[29572]: 5.19 scrub ok 2024-03-11T21:46:30.868 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:46:30 smithi092 ceph-mon[29572]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-03-11T21:46:30.869 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:46:30 smithi092 ceph-mon[29572]: from='client.? v1:172.21.15.92:0/4275739545' entity='client.admin' cmd='[{"prefix": "osd pg-upmap-items", "pgid": "5.a", "id": [7, 1, 4, 6, 2, 3]}]': finished 2024-03-11T21:46:30.869 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:46:30 smithi092 ceph-mon[29572]: from='client.25918 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-03-11T21:46:30.869 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:46:30 smithi092 ceph-mon[29572]: osdmap e468: 8 total, 8 up, 6 in 2024-03-11T21:46:30.869 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:46:30 smithi092 ceph-mon[29572]: pgmap v988: 148 pgs: 148 active+clean; 7.7 GiB data, 27 GiB used, 509 GiB / 536 GiB avail; 1.2 KiB/s rd, 97 MiB/s wr, 1.55k op/s 2024-03-11T21:46:30.869 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:46:30 smithi092 ceph-mon[37563]: 5.19 scrub starts 2024-03-11T21:46:30.869 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:46:30 smithi092 ceph-mon[37563]: 5.19 scrub ok 2024-03-11T21:46:30.869 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:46:30 smithi092 ceph-mon[37563]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-03-11T21:46:30.869 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:46:30 smithi092 ceph-mon[37563]: from='client.? v1:172.21.15.92:0/4275739545' entity='client.admin' cmd='[{"prefix": "osd pg-upmap-items", "pgid": "5.a", "id": [7, 1, 4, 6, 2, 3]}]': finished 2024-03-11T21:46:30.869 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:46:30 smithi092 ceph-mon[37563]: from='client.25918 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-03-11T21:46:30.869 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:46:30 smithi092 ceph-mon[37563]: osdmap e468: 8 total, 8 up, 6 in 2024-03-11T21:46:30.870 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:46:30 smithi092 ceph-mon[37563]: pgmap v988: 148 pgs: 148 active+clean; 7.7 GiB data, 27 GiB used, 509 GiB / 536 GiB avail; 1.2 KiB/s rd, 97 MiB/s wr, 1.55k op/s 2024-03-11T21:46:30.915 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:46:30.941 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:46:30.942 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:46:30.942 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:46:31.043 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:46:31.076 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:46:31.077 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:46:31.077 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:46:31.178 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:46:31.218 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:46:31.219 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:46:31.219 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:46:31.276 INFO:tasks.radosbench.radosbench.0.smithi092.stdout: 84 15 130503 130488 97.0635 83.6875 0.0082853 0.0102985 2024-03-11T21:46:31.320 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:46:31.352 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:46:31.352 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:46:31.353 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:46:31.393 DEBUG:teuthology.orchestra.run.smithi092:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid 00db0016-dfed-11ee-95c8-87774f69a715 -- ceph daemon osd.3 dump_ops_in_flight 2024-03-11T21:46:31.454 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:46:31.496 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:46:31.497 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:46:31.497 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:46:31.598 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:46:31.658 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:46:31.659 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:46:31.659 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:46:31.760 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:46:31.798 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:46:31.799 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:46:31.799 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:46:31.900 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:46:31.933 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:46:31.934 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:46:31.934 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:46:32.035 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:46:32.071 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:46:32.071 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:46:32.071 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:46:32.172 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:46:32.204 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:46:32.204 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:46:32.205 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:46:32.276 INFO:tasks.radosbench.radosbench.0.smithi092.stdout: 85 16 131868 131852 96.9244 85.25 0.0148257 0.0103131 2024-03-11T21:46:32.306 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:46:32.336 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:46:32.337 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:46:32.337 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:46:32.438 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:46:32.473 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:46:32.474 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:46:32.474 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:46:32.575 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:46:32.606 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:46:32.607 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:46:32.608 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:46:32.618 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:46:32 smithi092 ceph-mon[29572]: pgmap v989: 148 pgs: 148 active+clean; 7.7 GiB data, 28 GiB used, 509 GiB / 536 GiB avail; 1.1 KiB/s rd, 85 MiB/s wr, 1.36k op/s 2024-03-11T21:46:32.618 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:46:32 smithi092 ceph-mon[37563]: pgmap v989: 148 pgs: 148 active+clean; 7.7 GiB data, 28 GiB used, 509 GiB / 536 GiB avail; 1.1 KiB/s rd, 85 MiB/s wr, 1.36k op/s 2024-03-11T21:46:32.709 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:46:32.742 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:46:32.742 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:46:32.743 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:46:32.834 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:46:32 smithi097 ceph-mon[33317]: pgmap v989: 148 pgs: 148 active+clean; 7.7 GiB data, 28 GiB used, 509 GiB / 536 GiB avail; 1.1 KiB/s rd, 85 MiB/s wr, 1.36k op/s 2024-03-11T21:46:32.843 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:46:32.882 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:46:32.883 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:46:32.883 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:46:32.984 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:46:33.016 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:46:33.017 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:46:33.017 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:46:33.118 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:46:33.161 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:46:33.162 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:46:33.163 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:46:33.264 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:46:33.277 INFO:tasks.radosbench.radosbench.0.smithi092.stdout: 86 15 133208 133193 96.7708 83.8125 0.00705619 0.0103295 2024-03-11T21:46:33.301 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:46:33.302 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:46:33.302 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:46:33.403 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:46:33.447 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:46:33.448 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:46:33.448 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:46:33.455 INFO:journalctl@ceph.iscsi.iscsi.a.smithi097.stdout:Mar 11 21:46:33 smithi097 ceph-00db0016-dfed-11ee-95c8-87774f69a715-iscsi-iscsi-a[60943]: debug there is no tcmu-runner data available 2024-03-11T21:46:33.549 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:46:33.581 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:46:33.582 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:46:33.582 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:46:33.683 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:46:33.713 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:46:33.714 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:46:33.714 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:46:33.770 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:46:33 smithi092 ceph-mon[29572]: from='client.24469 v1:172.21.15.97:0/404208310' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:46:33.771 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:46:33 smithi092 ceph-mon[37563]: from='client.24469 v1:172.21.15.97:0/404208310' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:46:33.815 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:46:33.834 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:46:33 smithi097 ceph-mon[33317]: from='client.24469 v1:172.21.15.97:0/404208310' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:46:33.857 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:46:33.858 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:46:33.858 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:46:33.959 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:46:34.007 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:46:34.008 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:46:34.008 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:46:34.009 INFO:teuthology.orchestra.run.smithi092.stdout:osd.0: {} 2024-03-11T21:46:34.060 INFO:teuthology.orchestra.run.smithi092.stdout:osd.1: {} 2024-03-11T21:46:34.102 INFO:teuthology.orchestra.run.smithi092.stderr:Inferring config /var/lib/ceph/00db0016-dfed-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T21:46:34.103 INFO:teuthology.orchestra.run.smithi092.stdout:osd.2: {} 2024-03-11T21:46:34.109 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:46:34.142 INFO:teuthology.orchestra.run.smithi092.stdout:osd.3: {} 2024-03-11T21:46:34.162 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:46:34.163 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:46:34.163 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:46:34.188 INFO:teuthology.orchestra.run.smithi092.stdout:osd.4: {} 2024-03-11T21:46:34.230 INFO:teuthology.orchestra.run.smithi092.stdout:osd.5: {} 2024-03-11T21:46:34.263 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:46:34.266 INFO:teuthology.orchestra.run.smithi092.stdout:osd.6: {} 2024-03-11T21:46:34.278 INFO:tasks.radosbench.radosbench.0.smithi092.stdout: 87 16 134456 134440 96.5543 77.9375 0.0129492 0.0103525 2024-03-11T21:46:34.308 INFO:teuthology.orchestra.run.smithi092.stdout:osd.7: {} 2024-03-11T21:46:34.316 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:46:34.316 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:46:34.317 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:46:34.317 INFO:teuthology.orchestra.run.smithi092.stderr:osd.0: osd_enable_op_tracker = '' 2024-03-11T21:46:34.317 INFO:teuthology.orchestra.run.smithi092.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-11T21:46:34.317 INFO:teuthology.orchestra.run.smithi092.stderr:osd.2: osd_enable_op_tracker = '' 2024-03-11T21:46:34.317 INFO:teuthology.orchestra.run.smithi092.stderr:osd.3: osd_enable_op_tracker = '' 2024-03-11T21:46:34.317 INFO:teuthology.orchestra.run.smithi092.stderr:osd.4: osd_enable_op_tracker = '' 2024-03-11T21:46:34.317 INFO:teuthology.orchestra.run.smithi092.stderr:osd.5: osd_enable_op_tracker = '' 2024-03-11T21:46:34.317 INFO:teuthology.orchestra.run.smithi092.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-11T21:46:34.317 INFO:teuthology.orchestra.run.smithi092.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-11T21:46:34.417 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:46:34.451 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:46:34.452 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:46:34.452 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:46:34.553 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:46:34.599 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:46:34.600 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:46:34.600 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:46:34.618 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:46:34 smithi092 ceph-mon[29572]: pgmap v990: 148 pgs: 148 active+clean; 8.0 GiB data, 28 GiB used, 508 GiB / 536 GiB avail; 1023 B/s rd, 82 MiB/s wr, 1.30k op/s 2024-03-11T21:46:34.618 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:46:34 smithi092 ceph-mon[29572]: purged_snaps scrub starts 2024-03-11T21:46:34.618 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:46:34 smithi092 ceph-mon[29572]: purged_snaps scrub ok 2024-03-11T21:46:34.618 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:46:34 smithi092 ceph-mon[29572]: from='client.? v1:172.21.15.92:0/2374435498' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-11T21:46:34.619 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:46:34 smithi092 ceph-mon[37563]: pgmap v990: 148 pgs: 148 active+clean; 8.0 GiB data, 28 GiB used, 508 GiB / 536 GiB avail; 1023 B/s rd, 82 MiB/s wr, 1.30k op/s 2024-03-11T21:46:34.619 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:46:34 smithi092 ceph-mon[37563]: purged_snaps scrub starts 2024-03-11T21:46:34.619 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:46:34 smithi092 ceph-mon[37563]: purged_snaps scrub ok 2024-03-11T21:46:34.619 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:46:34 smithi092 ceph-mon[37563]: from='client.? v1:172.21.15.92:0/2374435498' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-11T21:46:34.702 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:46:34.736 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:46:34.737 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:46:34.737 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:46:34.834 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:46:34 smithi097 ceph-mon[33317]: pgmap v990: 148 pgs: 148 active+clean; 8.0 GiB data, 28 GiB used, 508 GiB / 536 GiB avail; 1023 B/s rd, 82 MiB/s wr, 1.30k op/s 2024-03-11T21:46:34.834 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:46:34 smithi097 ceph-mon[33317]: purged_snaps scrub starts 2024-03-11T21:46:34.834 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:46:34 smithi097 ceph-mon[33317]: purged_snaps scrub ok 2024-03-11T21:46:34.835 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:46:34 smithi097 ceph-mon[33317]: from='client.? v1:172.21.15.92:0/2374435498' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-11T21:46:34.838 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:46:34.881 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:46:34.882 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:46:34.882 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:46:34.983 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:46:35.026 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:46:35.027 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:46:35.027 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:46:35.128 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:46:35.164 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:46:35.165 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:46:35.165 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:46:35.266 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:46:35.279 INFO:tasks.radosbench.radosbench.0.smithi092.stdout: 88 16 135732 135716 96.3618 79.75 0.00815721 0.0103732 2024-03-11T21:46:35.308 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:46:35.309 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:46:35.309 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:46:35.410 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:46:35.450 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:46:35.451 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:46:35.451 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:46:35.552 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:46:35.591 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:46:35.591 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:46:35.592 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:46:35.645 DEBUG:teuthology.orchestra.run.smithi092:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid 00db0016-dfed-11ee-95c8-87774f69a715 -- ceph osd set nodeep-scrub 2024-03-11T21:46:35.693 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:46:35.738 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:46:35.739 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:46:35.739 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:46:35.834 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:46:35 smithi097 ceph-mon[33317]: osdmap e469: 8 total, 8 up, 6 in 2024-03-11T21:46:35.840 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:46:35.868 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:46:35 smithi092 ceph-mon[29572]: osdmap e469: 8 total, 8 up, 6 in 2024-03-11T21:46:35.868 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:46:35 smithi092 ceph-mon[37563]: osdmap e469: 8 total, 8 up, 6 in 2024-03-11T21:46:35.895 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:46:35.896 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:46:35.896 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:46:35.997 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:46:36.033 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:46:36.034 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:46:36.034 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:46:36.135 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:46:36.169 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:46:36.169 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:46:36.170 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:46:36.271 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:46:36.279 INFO:tasks.radosbench.radosbench.0.smithi092.stdout: 89 16 137074 137058 96.2213 83.875 0.00583663 0.0103881 2024-03-11T21:46:36.305 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:46:36.306 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:46:36.306 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:46:36.407 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:46:36.435 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:46:36.436 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:46:36.436 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:46:36.537 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:46:36.564 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:46:36 smithi092 ceph-mon[29572]: pgmap v992: 148 pgs: 148 active+clean; 8.1 GiB data, 29 GiB used, 508 GiB / 536 GiB avail; 1.2 KiB/s rd, 100 MiB/s wr, 1.60k op/s 2024-03-11T21:46:36.564 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:46:36 smithi092 ceph-mon[37563]: pgmap v992: 148 pgs: 148 active+clean; 8.1 GiB data, 29 GiB used, 508 GiB / 536 GiB avail; 1.2 KiB/s rd, 100 MiB/s wr, 1.60k op/s 2024-03-11T21:46:36.588 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:46:36.589 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:46:36.590 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:46:36.690 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:46:36.721 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:46:36.722 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:46:36.722 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:46:36.823 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:46:36.834 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:46:36 smithi097 ceph-mon[33317]: pgmap v992: 148 pgs: 148 active+clean; 8.1 GiB data, 29 GiB used, 508 GiB / 536 GiB avail; 1.2 KiB/s rd, 100 MiB/s wr, 1.60k op/s 2024-03-11T21:46:36.864 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:46:36.865 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:46:36.865 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:46:36.966 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:46:36.997 INFO:teuthology.orchestra.run.smithi092.stderr:Inferring config /var/lib/ceph/00db0016-dfed-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T21:46:37.013 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:46:37.014 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:46:37.014 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:46:37.079 INFO:teuthology.orchestra.run.smithi092.stdout:{ 2024-03-11T21:46:37.079 INFO:teuthology.orchestra.run.smithi092.stdout: "ops": [], 2024-03-11T21:46:37.079 INFO:teuthology.orchestra.run.smithi092.stdout: "num_ops": 0 2024-03-11T21:46:37.080 INFO:teuthology.orchestra.run.smithi092.stdout:} 2024-03-11T21:46:37.102 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 5, 6, 7] out_osds: [3, 4] dead_osds: [] live_osds: [0, 1, 2, 4, 5, 6, 7, 3] 2024-03-11T21:46:37.102 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-03-11T21:46:37.102 INFO:tasks.thrashosds.thrasher:Removing osd 5, in_osds are: [0, 1, 2, 5, 6, 7] 2024-03-11T21:46:37.103 DEBUG:teuthology.orchestra.run.smithi092:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid 00db0016-dfed-11ee-95c8-87774f69a715 -- ceph osd out 5 2024-03-11T21:46:37.115 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:46:37.143 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:46:37.143 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:46:37.144 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:46:37.162 DEBUG:teuthology.orchestra.run.smithi092:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid 00db0016-dfed-11ee-95c8-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-03-11T21:46:37.244 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:46:37.271 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:46:37.271 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:46:37.272 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:46:37.279 INFO:tasks.radosbench.radosbench.0.smithi092.stdout:Total time run: 90.0075 2024-03-11T21:46:37.279 INFO:tasks.radosbench.radosbench.0.smithi092.stdout:Total writes made: 138345 2024-03-11T21:46:37.279 INFO:tasks.radosbench.radosbench.0.smithi092.stdout:Write size: 65536 2024-03-11T21:46:37.279 INFO:tasks.radosbench.radosbench.0.smithi092.stdout:Object size: 65536 2024-03-11T21:46:37.279 INFO:tasks.radosbench.radosbench.0.smithi092.stdout:Bandwidth (MB/sec): 96.0649 2024-03-11T21:46:37.279 INFO:tasks.radosbench.radosbench.0.smithi092.stdout:Stddev Bandwidth: 31.9754 2024-03-11T21:46:37.279 INFO:tasks.radosbench.radosbench.0.smithi092.stdout:Max bandwidth (MB/sec): 231.688 2024-03-11T21:46:37.279 INFO:tasks.radosbench.radosbench.0.smithi092.stdout:Min bandwidth (MB/sec): 67.1875 2024-03-11T21:46:37.280 INFO:tasks.radosbench.radosbench.0.smithi092.stdout:Average IOPS: 1537 2024-03-11T21:46:37.280 INFO:tasks.radosbench.radosbench.0.smithi092.stdout:Stddev IOPS: 511.607 2024-03-11T21:46:37.280 INFO:tasks.radosbench.radosbench.0.smithi092.stdout:Max IOPS: 3707 2024-03-11T21:46:37.280 INFO:tasks.radosbench.radosbench.0.smithi092.stdout:Min IOPS: 1075 2024-03-11T21:46:37.280 INFO:tasks.radosbench.radosbench.0.smithi092.stdout:Average Latency(s): 0.0104059 2024-03-11T21:46:37.280 INFO:tasks.radosbench.radosbench.0.smithi092.stdout:Stddev Latency(s): 0.00858281 2024-03-11T21:46:37.280 INFO:tasks.radosbench.radosbench.0.smithi092.stdout:Max latency(s): 0.231127 2024-03-11T21:46:37.280 INFO:tasks.radosbench.radosbench.0.smithi092.stdout:Min latency(s): 0.00128848 2024-03-11T21:46:37.285 INFO:tasks.radosbench.radosbench.0.smithi092.stdout:Cleaning up (deleting benchmark objects) 2024-03-11T21:46:37.373 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:46:37.401 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:46:37.401 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:46:37.402 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:46:37.502 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:46:37.555 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:46:37.556 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:46:37.556 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:46:37.657 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:46:37.691 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:46:37.692 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:46:37.692 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:46:37.793 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:46:37.833 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:46:37.834 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:46:37.834 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:46:37.935 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:46:37.966 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:46:37.967 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:46:37.967 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:46:38.068 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:46:38.111 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:46:38.111 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:46:38.112 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:46:38.118 INFO:journalctl@ceph.mgr.y.smithi092.stdout:Mar 11 21:46:37 smithi092 ceph-00db0016-dfed-11ee-95c8-87774f69a715-mgr-y[29801]: ::ffff:172.21.15.97 - - [11/Mar/2024:21:46:37] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-11T21:46:38.213 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:46:38.254 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:46:38.255 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:46:38.255 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:46:38.356 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:46:38.387 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:46:38.388 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:46:38.388 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:46:38.489 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:46:38.528 DEBUG:teuthology.orchestra.run.smithi092:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid 00db0016-dfed-11ee-95c8-87774f69a715 -- ceph daemon osd.3 dump_blocked_ops 2024-03-11T21:46:38.530 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:46:38.530 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:46:38.530 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:46:38.631 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:46:38.680 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:46:38.681 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:46:38.681 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:46:38.782 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:46:38.813 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:46:38.814 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:46:38.814 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:46:38.915 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:46:38.962 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:46:38.962 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:46:38.963 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:46:39.063 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:46:39.096 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:46:39.097 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:46:39.097 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:46:39.198 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:46:39.234 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:46:39.235 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:46:39.235 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:46:39.335 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:46:39.369 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:46:39.370 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:46:39.371 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:46:39.471 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:46:39.504 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:46:39.504 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:46:39.505 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:46:39.508 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:46:39 smithi092 ceph-mon[29572]: pgmap v993: 148 pgs: 148 active+clean; 8.1 GiB data, 29 GiB used, 507 GiB / 536 GiB avail; 632 B/s rd, 58 MiB/s wr, 933 op/s 2024-03-11T21:46:39.508 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:46:39 smithi092 ceph-mon[29572]: from='mgr.14607 ' entity='mgr.y' 2024-03-11T21:46:39.508 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:46:39 smithi092 ceph-mon[29572]: from='mgr.14607 v1:172.21.15.92:0/1702717597' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T21:46:39.509 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:46:39 smithi092 ceph-mon[37563]: pgmap v993: 148 pgs: 148 active+clean; 8.1 GiB data, 29 GiB used, 507 GiB / 536 GiB avail; 632 B/s rd, 58 MiB/s wr, 933 op/s 2024-03-11T21:46:39.509 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:46:39 smithi092 ceph-mon[37563]: from='mgr.14607 ' entity='mgr.y' 2024-03-11T21:46:39.509 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:46:39 smithi092 ceph-mon[37563]: from='mgr.14607 v1:172.21.15.92:0/1702717597' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T21:46:39.584 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:46:39 smithi097 ceph-mon[33317]: pgmap v993: 148 pgs: 148 active+clean; 8.1 GiB data, 29 GiB used, 507 GiB / 536 GiB avail; 632 B/s rd, 58 MiB/s wr, 933 op/s 2024-03-11T21:46:39.584 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:46:39 smithi097 ceph-mon[33317]: from='mgr.14607 ' entity='mgr.y' 2024-03-11T21:46:39.584 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:46:39 smithi097 ceph-mon[33317]: from='mgr.14607 v1:172.21.15.92:0/1702717597' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T21:46:39.606 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:46:39.642 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:46:39.643 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:46:39.643 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:46:39.744 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:46:39.767 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:46:39.768 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:46:39.768 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:46:39.869 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:46:39.900 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:46:39.901 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:46:39.901 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:46:40.002 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:46:40.027 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:46:40.028 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:46:40.028 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:46:40.129 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:46:40.162 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:46:40.162 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:46:40.162 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:46:40.263 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:46:40.302 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:46:40.302 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:46:40.303 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:46:40.403 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:46:40.442 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:46:40.443 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:46:40.443 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:46:40.544 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:46:40.582 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:46:40.583 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:46:40.583 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:46:40.684 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:46:40.716 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:46:40.716 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:46:40.717 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:46:40.818 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:46:40.857 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:46:40.858 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:46:40.858 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:46:40.959 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:46:40.991 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:46:40.991 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:46:40.992 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:46:41.092 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:46:41.137 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:46:41.138 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:46:41.138 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:46:41.239 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:46:41.262 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:46:41.263 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:46:41.263 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:46:41.364 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:46:41.399 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:46:41.400 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:46:41.400 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:46:41.501 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:46:41.538 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:46:41.539 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:46:41.540 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:46:41.584 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:46:41 smithi097 ceph-mon[33317]: pgmap v994: 148 pgs: 148 active+clean; 8.4 GiB data, 30 GiB used, 507 GiB / 536 GiB avail; 1.1 KiB/s rd, 77 MiB/s wr, 1.33k op/s 2024-03-11T21:46:41.618 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:46:41 smithi092 ceph-mon[29572]: pgmap v994: 148 pgs: 148 active+clean; 8.4 GiB data, 30 GiB used, 507 GiB / 536 GiB avail; 1.1 KiB/s rd, 77 MiB/s wr, 1.33k op/s 2024-03-11T21:46:41.618 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:46:41 smithi092 ceph-mon[37563]: pgmap v994: 148 pgs: 148 active+clean; 8.4 GiB data, 30 GiB used, 507 GiB / 536 GiB avail; 1.1 KiB/s rd, 77 MiB/s wr, 1.33k op/s 2024-03-11T21:46:41.641 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:46:41.674 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:46:41.675 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:46:41.675 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:46:41.776 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:46:41.816 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:46:41.817 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:46:41.817 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:46:41.917 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:46:41.934 INFO:teuthology.orchestra.run.smithi092.stderr:Inferring config /var/lib/ceph/00db0016-dfed-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T21:46:41.949 INFO:teuthology.orchestra.run.smithi092.stderr:Inferring config /var/lib/ceph/00db0016-dfed-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T21:46:41.966 INFO:teuthology.orchestra.run.smithi092.stderr:Inferring config /var/lib/ceph/00db0016-dfed-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T21:46:41.984 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:46:41.985 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:46:41.985 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:46:42.086 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:46:42.119 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:46:42.120 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:46:42.120 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:46:42.221 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:46:42.259 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:46:42.260 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:46:42.260 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:46:42.361 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:46:42.393 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:46:42.394 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:46:42.394 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:46:42.495 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:46:42.528 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:46:42.529 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:46:42.529 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:46:42.630 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:46:42.685 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:46:42.686 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:46:42.686 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:46:42.787 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:46:42.827 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:46:42.828 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:46:42.828 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:46:42.929 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:46:42.960 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:46:42.961 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:46:42.961 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:46:43.062 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:46:43.102 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:46:43.103 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:46:43.103 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:46:43.204 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:46:43.246 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:46:43.247 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:46:43.247 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:46:43.348 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:46:43.382 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:46:43.383 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:46:43.383 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:46:43.478 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:46:43 smithi092 ceph-mon[29572]: pgmap v995: 148 pgs: 148 active+clean; 8.4 GiB data, 30 GiB used, 507 GiB / 536 GiB avail; 1.1 KiB/s rd, 76 MiB/s wr, 1.33k op/s 2024-03-11T21:46:43.478 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:46:43 smithi092 ceph-mon[29572]: purged_snaps scrub starts 2024-03-11T21:46:43.478 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:46:43 smithi092 ceph-mon[29572]: purged_snaps scrub ok 2024-03-11T21:46:43.478 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:46:43 smithi092 ceph-mon[29572]: osdmap e470: 8 total, 8 up, 6 in 2024-03-11T21:46:43.478 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:46:43 smithi092 ceph-mon[37563]: pgmap v995: 148 pgs: 148 active+clean; 8.4 GiB data, 30 GiB used, 507 GiB / 536 GiB avail; 1.1 KiB/s rd, 76 MiB/s wr, 1.33k op/s 2024-03-11T21:46:43.478 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:46:43 smithi092 ceph-mon[37563]: purged_snaps scrub starts 2024-03-11T21:46:43.479 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:46:43 smithi092 ceph-mon[37563]: purged_snaps scrub ok 2024-03-11T21:46:43.479 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:46:43 smithi092 ceph-mon[37563]: osdmap e470: 8 total, 8 up, 6 in 2024-03-11T21:46:43.484 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:46:43.518 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:46:43.518 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:46:43.519 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:46:43.584 INFO:journalctl@ceph.iscsi.iscsi.a.smithi097.stdout:Mar 11 21:46:43 smithi097 ceph-00db0016-dfed-11ee-95c8-87774f69a715-iscsi-iscsi-a[60943]: debug there is no tcmu-runner data available 2024-03-11T21:46:43.585 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:46:43 smithi097 ceph-mon[33317]: pgmap v995: 148 pgs: 148 active+clean; 8.4 GiB data, 30 GiB used, 507 GiB / 536 GiB avail; 1.1 KiB/s rd, 76 MiB/s wr, 1.33k op/s 2024-03-11T21:46:43.585 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:46:43 smithi097 ceph-mon[33317]: purged_snaps scrub starts 2024-03-11T21:46:43.585 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:46:43 smithi097 ceph-mon[33317]: purged_snaps scrub ok 2024-03-11T21:46:43.585 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:46:43 smithi097 ceph-mon[33317]: osdmap e470: 8 total, 8 up, 6 in 2024-03-11T21:46:43.619 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:46:43.653 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:46:43.654 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:46:43.655 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:46:43.755 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:46:43.797 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:46:43.798 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:46:43.798 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:46:43.899 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:46:43.928 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:46:43.929 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:46:43.929 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:46:44.030 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:46:44.069 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:46:44.070 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:46:44.070 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:46:44.171 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:46:44.200 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:46:44.201 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:46:44.201 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:46:44.211 INFO:teuthology.orchestra.run.smithi092.stderr:nodeep-scrub is set 2024-03-11T21:46:44.302 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:46:44.340 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:46:44.340 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:46:44.341 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:46:44.441 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:46:44.468 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:46:44.469 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:46:44.469 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:46:44.556 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:46:44 smithi092 ceph-mon[29572]: from='client.24469 v1:172.21.15.97:0/404208310' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:46:44.557 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:46:44 smithi092 ceph-mon[29572]: from='client.? v1:172.21.15.92:0/3093177244' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-11T21:46:44.557 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:46:44 smithi092 ceph-mon[29572]: from='client.35924 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-11T21:46:44.557 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:46:44 smithi092 ceph-mon[37563]: from='client.24469 v1:172.21.15.97:0/404208310' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:46:44.557 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:46:44 smithi092 ceph-mon[37563]: from='client.? v1:172.21.15.92:0/3093177244' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-11T21:46:44.557 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:46:44 smithi092 ceph-mon[37563]: from='client.35924 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-11T21:46:44.569 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:46:44.584 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:46:44 smithi097 ceph-mon[33317]: from='client.24469 v1:172.21.15.97:0/404208310' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:46:44.584 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:46:44 smithi097 ceph-mon[33317]: from='client.? v1:172.21.15.92:0/3093177244' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-11T21:46:44.584 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:46:44 smithi097 ceph-mon[33317]: from='client.35924 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-11T21:46:44.608 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:46:44.609 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:46:44.609 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:46:44.710 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:46:44.749 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:46:44.750 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:46:44.750 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:46:44.851 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:46:44.886 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:46:44.887 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:46:44.887 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:46:44.988 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:46:45.021 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:46:45.022 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:46:45.022 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:46:45.123 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:46:45.155 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:46:45.156 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:46:45.156 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:46:45.257 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:46:45.291 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:46:45.292 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:46:45.292 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:46:45.368 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:46:45 smithi092 ceph-mon[29572]: pgmap v997: 148 pgs: 148 active+clean; 8.1 GiB data, 29 GiB used, 507 GiB / 536 GiB avail; 687 B/s rd, 41 MiB/s wr, 1.34k op/s 2024-03-11T21:46:45.368 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:46:45 smithi092 ceph-mon[29572]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-11T21:46:45.368 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:46:45 smithi092 ceph-mon[29572]: from='client.35924 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-03-11T21:46:45.368 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:46:45 smithi092 ceph-mon[29572]: osdmap e471: 8 total, 8 up, 6 in 2024-03-11T21:46:45.369 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:46:45 smithi092 ceph-mon[37563]: pgmap v997: 148 pgs: 148 active+clean; 8.1 GiB data, 29 GiB used, 507 GiB / 536 GiB avail; 687 B/s rd, 41 MiB/s wr, 1.34k op/s 2024-03-11T21:46:45.369 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:46:45 smithi092 ceph-mon[37563]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-11T21:46:45.369 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:46:45 smithi092 ceph-mon[37563]: from='client.35924 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-03-11T21:46:45.369 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:46:45 smithi092 ceph-mon[37563]: osdmap e471: 8 total, 8 up, 6 in 2024-03-11T21:46:45.392 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:46:45.426 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:46:45.427 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:46:45.428 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:46:45.529 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:46:45.562 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:46:45.563 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:46:45.563 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:46:45.584 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:46:45 smithi097 ceph-mon[33317]: pgmap v997: 148 pgs: 148 active+clean; 8.1 GiB data, 29 GiB used, 507 GiB / 536 GiB avail; 687 B/s rd, 41 MiB/s wr, 1.34k op/s 2024-03-11T21:46:45.584 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:46:45 smithi097 ceph-mon[33317]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-11T21:46:45.584 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:46:45 smithi097 ceph-mon[33317]: from='client.35924 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-03-11T21:46:45.584 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:46:45 smithi097 ceph-mon[33317]: osdmap e471: 8 total, 8 up, 6 in 2024-03-11T21:46:45.664 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:46:45.694 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:46:45.695 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:46:45.695 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:46:45.796 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:46:45.830 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:46:45.831 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:46:45.831 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:46:45.932 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:46:45.972 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:46:45.972 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:46:45.973 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:46:46.073 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:46:46.113 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:46:46.114 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:46:46.115 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:46:46.215 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:46:46.248 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:46:46.249 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:46:46.249 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:46:46.350 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:46:46.369 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:46:46 smithi092 ceph-mon[29572]: purged_snaps scrub starts 2024-03-11T21:46:46.369 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:46:46 smithi092 ceph-mon[29572]: purged_snaps scrub ok 2024-03-11T21:46:46.369 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:46:46 smithi092 ceph-mon[37563]: purged_snaps scrub starts 2024-03-11T21:46:46.369 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:46:46 smithi092 ceph-mon[37563]: purged_snaps scrub ok 2024-03-11T21:46:46.390 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:46:46.390 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:46:46.391 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:46:46.491 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:46:46.530 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:46:46.530 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:46:46.531 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:46:46.584 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:46:46 smithi097 ceph-mon[33317]: purged_snaps scrub starts 2024-03-11T21:46:46.584 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:46:46 smithi097 ceph-mon[33317]: purged_snaps scrub ok 2024-03-11T21:46:46.632 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:46:46.670 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:46:46.670 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:46:46.671 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:46:46.771 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:46:46.818 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:46:46.819 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:46:46.819 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:46:46.920 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:46:46.954 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:46:46.955 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:46:46.955 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:46:47.056 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:46:47.088 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:46:47.088 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:46:47.089 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:46:47.189 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:46:47.221 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:46:47.222 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:46:47.222 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:46:47.323 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:46:47.366 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:46:47.366 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:46:47.367 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:46:47.467 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:46:47.479 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:46:47 smithi092 ceph-mon[29572]: pgmap v999: 148 pgs: 148 active+clean; 7.9 GiB data, 29 GiB used, 508 GiB / 536 GiB avail; 1.4 KiB/s rd, 39 MiB/s wr, 1.66k op/s 2024-03-11T21:46:47.479 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:46:47 smithi092 ceph-mon[29572]: osdmap e472: 8 total, 8 up, 6 in 2024-03-11T21:46:47.479 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:46:47 smithi092 ceph-mon[37563]: pgmap v999: 148 pgs: 148 active+clean; 7.9 GiB data, 29 GiB used, 508 GiB / 536 GiB avail; 1.4 KiB/s rd, 39 MiB/s wr, 1.66k op/s 2024-03-11T21:46:47.479 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:46:47 smithi092 ceph-mon[37563]: osdmap e472: 8 total, 8 up, 6 in 2024-03-11T21:46:47.513 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:46:47.513 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:46:47.514 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:46:47.584 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:46:47 smithi097 ceph-mon[33317]: pgmap v999: 148 pgs: 148 active+clean; 7.9 GiB data, 29 GiB used, 508 GiB / 536 GiB avail; 1.4 KiB/s rd, 39 MiB/s wr, 1.66k op/s 2024-03-11T21:46:47.584 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:46:47 smithi097 ceph-mon[33317]: osdmap e472: 8 total, 8 up, 6 in 2024-03-11T21:46:47.615 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:46:47.648 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:46:47.649 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:46:47.649 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:46:47.750 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:46:47.790 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:46:47.790 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:46:47.791 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:46:47.891 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:46:47.927 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:46:47.927 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:46:47.928 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:46:48.028 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:46:48.079 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:46:48.079 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:46:48.079 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:46:48.180 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:46:48.213 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:46:48.214 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:46:48.214 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:46:48.217 INFO:journalctl@ceph.mgr.y.smithi092.stdout:Mar 11 21:46:47 smithi092 ceph-00db0016-dfed-11ee-95c8-87774f69a715-mgr-y[29801]: ::ffff:172.21.15.97 - - [11/Mar/2024:21:46:47] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-11T21:46:48.315 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:46:48.349 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:46:48.350 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:46:48.350 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:46:48.451 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:46:48.490 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:46:48.491 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:46:48.491 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:46:48.491 DEBUG:teuthology.orchestra.run.smithi092:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid 00db0016-dfed-11ee-95c8-87774f69a715 -- ceph osd unset noscrub 2024-03-11T21:46:48.591 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:46:48.632 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:46:48.633 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:46:48.633 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:46:48.734 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:46:48.771 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:46:48.771 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:46:48.772 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:46:48.872 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:46:48.927 INFO:teuthology.orchestra.run.smithi092.stdout:osd.0: {} 2024-03-11T21:46:48.929 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:46:48.930 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:46:48.931 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:46:48.965 INFO:teuthology.orchestra.run.smithi092.stdout:osd.1: {} 2024-03-11T21:46:49.004 INFO:teuthology.orchestra.run.smithi092.stdout:osd.2: {} 2024-03-11T21:46:49.032 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:46:49.042 INFO:teuthology.orchestra.run.smithi092.stdout:osd.3: {} 2024-03-11T21:46:49.064 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:46:49.064 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:46:49.065 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:46:49.086 INFO:teuthology.orchestra.run.smithi092.stdout:osd.4: {} 2024-03-11T21:46:49.125 INFO:teuthology.orchestra.run.smithi092.stdout:osd.5: {} 2024-03-11T21:46:49.165 INFO:teuthology.orchestra.run.smithi092.stdout:osd.6: {} 2024-03-11T21:46:49.166 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:46:49.195 INFO:teuthology.orchestra.run.smithi092.stdout:ERROR: (22) Invalid argument 2024-03-11T21:46:49.195 INFO:teuthology.orchestra.run.smithi092.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-11T21:46:49.203 INFO:teuthology.orchestra.run.smithi092.stdout:osd.7: {} 2024-03-11T21:46:49.211 INFO:teuthology.orchestra.run.smithi092.stderr:osd.0: osd_enable_op_tracker = '' 2024-03-11T21:46:49.211 INFO:teuthology.orchestra.run.smithi092.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-11T21:46:49.211 INFO:teuthology.orchestra.run.smithi092.stderr:osd.2: osd_enable_op_tracker = '' 2024-03-11T21:46:49.211 INFO:teuthology.orchestra.run.smithi092.stderr:osd.3: osd_enable_op_tracker = '' 2024-03-11T21:46:49.211 INFO:teuthology.orchestra.run.smithi092.stderr:osd.4: osd_enable_op_tracker = '' 2024-03-11T21:46:49.211 INFO:teuthology.orchestra.run.smithi092.stderr:osd.5: osd_enable_op_tracker = '' 2024-03-11T21:46:49.211 INFO:teuthology.orchestra.run.smithi092.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-11T21:46:49.212 INFO:teuthology.orchestra.run.smithi092.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-11T21:46:49.222 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:46:49.223 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:46:49.223 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:46:49.324 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:46:49.355 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:46:49 smithi092 ceph-mon[29572]: pgmap v1001: 148 pgs: 148 active+clean; 7.9 GiB data, 28 GiB used, 508 GiB / 536 GiB avail; 853 B/s rd, 21 KiB/s wr, 1.26k op/s 2024-03-11T21:46:49.355 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:46:49 smithi092 ceph-mon[29572]: from='client.? v1:172.21.15.92:0/1687724413' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-11T21:46:49.355 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:46:49 smithi092 ceph-mon[37563]: pgmap v1001: 148 pgs: 148 active+clean; 7.9 GiB data, 28 GiB used, 508 GiB / 536 GiB avail; 853 B/s rd, 21 KiB/s wr, 1.26k op/s 2024-03-11T21:46:49.355 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:46:49 smithi092 ceph-mon[37563]: from='client.? v1:172.21.15.92:0/1687724413' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-11T21:46:49.367 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:46:49.367 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:46:49.368 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:46:49.468 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:46:49.502 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:46:49.502 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:46:49.503 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:46:49.584 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:46:49 smithi097 ceph-mon[33317]: pgmap v1001: 148 pgs: 148 active+clean; 7.9 GiB data, 28 GiB used, 508 GiB / 536 GiB avail; 853 B/s rd, 21 KiB/s wr, 1.26k op/s 2024-03-11T21:46:49.584 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:46:49 smithi097 ceph-mon[33317]: from='client.? v1:172.21.15.92:0/1687724413' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-11T21:46:49.604 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:46:49.641 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:46:49.642 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:46:49.642 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:46:49.743 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:46:49.781 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:46:49.781 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:46:49.782 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:46:49.883 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:46:49.913 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:46:49.913 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:46:49.914 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:46:50.014 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:46:50.055 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:46:50.055 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:46:50.056 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:46:50.157 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:46:50.187 DEBUG:teuthology.orchestra.run.smithi092:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid 00db0016-dfed-11ee-95c8-87774f69a715 -- ceph daemon osd.3 dump_historic_ops 2024-03-11T21:46:50.205 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:46:50.205 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:46:50.206 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:46:50.307 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:46:50.341 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:46:50.341 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:46:50.342 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:46:50.442 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:46:50.474 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:46:50.475 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:46:50.475 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:46:50.576 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:46:50.620 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:46:50.620 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:46:50.621 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:46:50.721 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:46:50.749 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:46:50.749 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:46:50.750 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:46:50.851 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:46:50.906 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:46:50.906 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:46:50.907 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:46:51.007 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:46:51.041 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:46:51.042 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:46:51.043 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:46:51.144 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:46:51.182 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:46:51.182 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:46:51.183 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:46:51.279 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:46:51 smithi092 ceph-mon[29572]: pgmap v1002: 148 pgs: 148 active+clean; 7.5 GiB data, 27 GiB used, 509 GiB / 536 GiB avail; 1.4 KiB/s rd, 18 KiB/s wr, 1.95k op/s 2024-03-11T21:46:51.280 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:46:51 smithi092 ceph-mon[37563]: pgmap v1002: 148 pgs: 148 active+clean; 7.5 GiB data, 27 GiB used, 509 GiB / 536 GiB avail; 1.4 KiB/s rd, 18 KiB/s wr, 1.95k op/s 2024-03-11T21:46:51.284 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:46:51.317 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:46:51.318 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:46:51.319 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:46:51.419 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:46:51.441 INFO:teuthology.orchestra.run.smithi092.stderr:Inferring config /var/lib/ceph/00db0016-dfed-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T21:46:51.502 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:46:51.503 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:46:51.503 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:46:51.584 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:46:51 smithi097 ceph-mon[33317]: pgmap v1002: 148 pgs: 148 active+clean; 7.5 GiB data, 27 GiB used, 509 GiB / 536 GiB avail; 1.4 KiB/s rd, 18 KiB/s wr, 1.95k op/s 2024-03-11T21:46:51.590 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:46:51 smithi092 ceph-00db0016-dfed-11ee-95c8-87774f69a715-mon-a[29568]: 2024-03-11T21:46:51.287+0000 7f87a4389700 -1 mon.a@0(leader).osd e472 definitely_dead 0 2024-03-11T21:46:51.604 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:46:51.651 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:46:51.652 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:46:51.652 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:46:51.653 INFO:teuthology.orchestra.run.smithi092.stderr:Inferring config /var/lib/ceph/00db0016-dfed-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T21:46:51.753 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:46:51.805 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:46:51.806 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:46:51.806 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:46:51.907 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:46:51.937 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:46:51.938 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:46:51.938 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:46:52.039 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:46:52.083 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:46:52.084 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:46:52.084 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:46:52.185 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:46:52.214 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:46:52.214 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:46:52.214 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:46:52.251 INFO:teuthology.orchestra.run.smithi092.stderr:marked out osd.5. 2024-03-11T21:46:52.315 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:46:52.338 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:46:52.339 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:46:52.339 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:46:52.440 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:46:52.466 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:46:52.466 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:46:52.466 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:46:52.567 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:46:52.584 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:46:52 smithi097 ceph-mon[33317]: from='client.? v1:172.21.15.92:0/3650630151' entity='client.admin' cmd=[{"prefix": "osd out", "ids": ["5"]}]: dispatch 2024-03-11T21:46:52.584 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:46:52 smithi097 ceph-mon[33317]: from='client.35936 ' entity='client.admin' cmd=[{"prefix": "osd out", "ids": ["5"]}]: dispatch 2024-03-11T21:46:52.584 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:46:52 smithi097 ceph-mon[33317]: Client client.admin marked osd.5 out, while it was still marked up 2024-03-11T21:46:52.591 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:46:52.592 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:46:52.592 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:46:52.618 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:46:52 smithi092 ceph-mon[29572]: from='client.? v1:172.21.15.92:0/3650630151' entity='client.admin' cmd=[{"prefix": "osd out", "ids": ["5"]}]: dispatch 2024-03-11T21:46:52.618 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:46:52 smithi092 ceph-mon[29572]: from='client.35936 ' entity='client.admin' cmd=[{"prefix": "osd out", "ids": ["5"]}]: dispatch 2024-03-11T21:46:52.618 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:46:52 smithi092 ceph-mon[29572]: Client client.admin marked osd.5 out, while it was still marked up 2024-03-11T21:46:52.618 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:46:52 smithi092 ceph-mon[37563]: from='client.? v1:172.21.15.92:0/3650630151' entity='client.admin' cmd=[{"prefix": "osd out", "ids": ["5"]}]: dispatch 2024-03-11T21:46:52.618 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:46:52 smithi092 ceph-mon[37563]: from='client.35936 ' entity='client.admin' cmd=[{"prefix": "osd out", "ids": ["5"]}]: dispatch 2024-03-11T21:46:52.618 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:46:52 smithi092 ceph-mon[37563]: Client client.admin marked osd.5 out, while it was still marked up 2024-03-11T21:46:52.693 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:46:52.718 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:46:52.719 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:46:52.719 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:46:52.820 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:46:52.844 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:46:52.844 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:46:52.845 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:46:52.946 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:46:52.972 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:46:52.972 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:46:52.972 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:46:53.073 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:46:53.098 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:46:53.099 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:46:53.099 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:46:53.200 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:46:53.226 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:46:53.227 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:46:53.227 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:46:53.328 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:46:53.352 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:46:53.352 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:46:53.353 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:46:53.438 DEBUG:teuthology.orchestra.run.smithi092:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid 00db0016-dfed-11ee-95c8-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-03-11T21:46:53.454 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:46:53.477 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:46:53.478 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:46:53.478 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:46:53.484 INFO:journalctl@ceph.iscsi.iscsi.a.smithi097.stdout:Mar 11 21:46:53 smithi097 ceph-00db0016-dfed-11ee-95c8-87774f69a715-iscsi-iscsi-a[60943]: debug there is no tcmu-runner data available 2024-03-11T21:46:53.485 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:46:53 smithi097 ceph-mon[33317]: pgmap v1003: 148 pgs: 148 active+clean; 7.5 GiB data, 27 GiB used, 509 GiB / 536 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.22k op/s 2024-03-11T21:46:53.485 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:46:53 smithi097 ceph-mon[33317]: from='client.35936 ' entity='client.admin' cmd='[{"prefix": "osd out", "ids": ["5"]}]': finished 2024-03-11T21:46:53.485 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:46:53 smithi097 ceph-mon[33317]: osdmap e473: 8 total, 8 up, 5 in 2024-03-11T21:46:53.485 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:46:53 smithi097 ceph-mon[33317]: from='mgr.14607 v1:172.21.15.92:0/1702717597' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T21:46:53.579 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:46:53.603 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:46:53.604 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:46:53.604 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:46:53.618 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:46:53 smithi092 ceph-mon[29572]: pgmap v1003: 148 pgs: 148 active+clean; 7.5 GiB data, 27 GiB used, 509 GiB / 536 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.22k op/s 2024-03-11T21:46:53.618 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:46:53 smithi092 ceph-mon[29572]: from='client.35936 ' entity='client.admin' cmd='[{"prefix": "osd out", "ids": ["5"]}]': finished 2024-03-11T21:46:53.618 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:46:53 smithi092 ceph-mon[29572]: osdmap e473: 8 total, 8 up, 5 in 2024-03-11T21:46:53.618 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:46:53 smithi092 ceph-mon[29572]: from='mgr.14607 v1:172.21.15.92:0/1702717597' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T21:46:53.619 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:46:53 smithi092 ceph-mon[37563]: pgmap v1003: 148 pgs: 148 active+clean; 7.5 GiB data, 27 GiB used, 509 GiB / 536 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.22k op/s 2024-03-11T21:46:53.619 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:46:53 smithi092 ceph-mon[37563]: from='client.35936 ' entity='client.admin' cmd='[{"prefix": "osd out", "ids": ["5"]}]': finished 2024-03-11T21:46:53.619 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:46:53 smithi092 ceph-mon[37563]: osdmap e473: 8 total, 8 up, 5 in 2024-03-11T21:46:53.619 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:46:53 smithi092 ceph-mon[37563]: from='mgr.14607 v1:172.21.15.92:0/1702717597' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T21:46:53.705 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:46:53.729 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:46:53.730 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:46:53.730 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:46:53.831 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:46:53.855 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:46:53.856 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:46:53.856 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:46:53.957 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:46:53.980 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:46:53.981 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:46:53.981 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:46:54.082 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:46:54.107 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:46:54.108 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:46:54.108 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:46:54.209 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:46:54.234 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:46:54.235 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:46:54.235 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:46:54.336 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:46:54.363 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:46:54.364 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:46:54.364 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:46:54.465 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:46:54.502 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:46:54.503 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:46:54.503 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:46:54.520 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:46:54 smithi092 ceph-mon[29572]: from='client.24469 v1:172.21.15.97:0/404208310' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:46:54.520 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:46:54 smithi092 ceph-mon[29572]: osdmap e474: 8 total, 8 up, 5 in 2024-03-11T21:46:54.520 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:46:54 smithi092 ceph-mon[37563]: from='client.24469 v1:172.21.15.97:0/404208310' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:46:54.520 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:46:54 smithi092 ceph-mon[37563]: osdmap e474: 8 total, 8 up, 5 in 2024-03-11T21:46:54.584 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:46:54 smithi097 ceph-mon[33317]: from='client.24469 v1:172.21.15.97:0/404208310' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:46:54.584 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:46:54 smithi097 ceph-mon[33317]: osdmap e474: 8 total, 8 up, 5 in 2024-03-11T21:46:54.604 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:46:54.644 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:46:54.645 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:46:54.645 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:46:54.746 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:46:54.787 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:46:54.787 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:46:54.788 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:46:54.888 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:46:54.924 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:46:54.925 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:46:54.925 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:46:55.026 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:46:55.067 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:46:55.068 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:46:55.068 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:46:55.169 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:46:55.205 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:46:55.205 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:46:55.206 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:46:55.293 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:46:55 smithi092 ceph-mon[29572]: pgmap v1006: 148 pgs: 31 peering, 117 active+clean; 7.2 GiB data, 23 GiB used, 424 GiB / 447 GiB avail; 281 B/s rd, 0 B/s wr, 1.63k op/s; 0 B/s, 0 objects/s recovering 2024-03-11T21:46:55.293 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:46:55 smithi092 ceph-mon[29572]: Health check failed: Reduced data availability: 4 pgs inactive, 8 pgs peering (PG_AVAILABILITY) 2024-03-11T21:46:55.293 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:46:55 smithi092 ceph-mon[29572]: 11.8 starting backfill to osd.2 from (0'0,0'0] MAX to 472'10031 2024-03-11T21:46:55.294 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:46:55 smithi092 ceph-mon[29572]: 11.a starting backfill to osd.2 from (0'0,0'0] MAX to 472'10139 2024-03-11T21:46:55.294 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:46:55 smithi092 ceph-mon[29572]: osdmap e475: 8 total, 8 up, 5 in 2024-03-11T21:46:55.294 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:46:55 smithi092 ceph-mon[29572]: 11.b starting backfill to osd.6 from (0'0,0'0] MAX to 472'9983 2024-03-11T21:46:55.294 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:46:55 smithi092 ceph-mon[29572]: 11.7 starting backfill to osd.7 from (0'0,0'0] MAX to 472'9889 2024-03-11T21:46:55.294 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:46:55 smithi092 ceph-mon[29572]: 11.5 starting backfill to osd.0 from (0'0,0'0] MAX to 472'10055 2024-03-11T21:46:55.295 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:46:55 smithi092 ceph-mon[37563]: pgmap v1006: 148 pgs: 31 peering, 117 active+clean; 7.2 GiB data, 23 GiB used, 424 GiB / 447 GiB avail; 281 B/s rd, 0 B/s wr, 1.63k op/s; 0 B/s, 0 objects/s recovering 2024-03-11T21:46:55.295 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:46:55 smithi092 ceph-mon[37563]: Health check failed: Reduced data availability: 4 pgs inactive, 8 pgs peering (PG_AVAILABILITY) 2024-03-11T21:46:55.295 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:46:55 smithi092 ceph-mon[37563]: 11.8 starting backfill to osd.2 from (0'0,0'0] MAX to 472'10031 2024-03-11T21:46:55.295 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:46:55 smithi092 ceph-mon[37563]: 11.a starting backfill to osd.2 from (0'0,0'0] MAX to 472'10139 2024-03-11T21:46:55.295 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:46:55 smithi092 ceph-mon[37563]: osdmap e475: 8 total, 8 up, 5 in 2024-03-11T21:46:55.295 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:46:55 smithi092 ceph-mon[37563]: 11.b starting backfill to osd.6 from (0'0,0'0] MAX to 472'9983 2024-03-11T21:46:55.295 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:46:55 smithi092 ceph-mon[37563]: 11.7 starting backfill to osd.7 from (0'0,0'0] MAX to 472'9889 2024-03-11T21:46:55.295 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:46:55 smithi092 ceph-mon[37563]: 11.5 starting backfill to osd.0 from (0'0,0'0] MAX to 472'10055 2024-03-11T21:46:55.307 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:46:55.360 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:46:55.361 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:46:55.362 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:46:55.462 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:46:55.498 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:46:55.498 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:46:55.499 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:46:55.585 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:46:55 smithi097 ceph-mon[33317]: pgmap v1006: 148 pgs: 31 peering, 117 active+clean; 7.2 GiB data, 23 GiB used, 424 GiB / 447 GiB avail; 281 B/s rd, 0 B/s wr, 1.63k op/s; 0 B/s, 0 objects/s recovering 2024-03-11T21:46:55.585 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:46:55 smithi097 ceph-mon[33317]: Health check failed: Reduced data availability: 4 pgs inactive, 8 pgs peering (PG_AVAILABILITY) 2024-03-11T21:46:55.585 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:46:55 smithi097 ceph-mon[33317]: 11.8 starting backfill to osd.2 from (0'0,0'0] MAX to 472'10031 2024-03-11T21:46:55.585 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:46:55 smithi097 ceph-mon[33317]: 11.a starting backfill to osd.2 from (0'0,0'0] MAX to 472'10139 2024-03-11T21:46:55.585 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:46:55 smithi097 ceph-mon[33317]: osdmap e475: 8 total, 8 up, 5 in 2024-03-11T21:46:55.585 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:46:55 smithi097 ceph-mon[33317]: 11.b starting backfill to osd.6 from (0'0,0'0] MAX to 472'9983 2024-03-11T21:46:55.585 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:46:55 smithi097 ceph-mon[33317]: 11.7 starting backfill to osd.7 from (0'0,0'0] MAX to 472'9889 2024-03-11T21:46:55.585 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:46:55 smithi097 ceph-mon[33317]: 11.5 starting backfill to osd.0 from (0'0,0'0] MAX to 472'10055 2024-03-11T21:46:55.600 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:46:55.637 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:46:55.638 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:46:55.639 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:46:55.739 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:46:55.777 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:46:55.777 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:46:55.778 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:46:55.879 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:46:55.910 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:46:55.910 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:46:55.911 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:46:56.012 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:46:56.048 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:46:56.049 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:46:56.049 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:46:56.150 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:46:56.190 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:46:56.191 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:46:56.191 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:46:56.292 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:46:56.324 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:46:56.325 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:46:56.325 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:46:56.426 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:46:56.460 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:46:56.460 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:46:56.461 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:46:56.562 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:46:56.589 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:46:56.589 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:46:56.590 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:46:56.690 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:46:56.730 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:46:56.730 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:46:56.731 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:46:56.831 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:46:56.887 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:46:56.888 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:46:56.888 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:46:56.989 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:46:57.024 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:46:57.025 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:46:57.025 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:46:57.126 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:46:57.164 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:46:57.164 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:46:57.165 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:46:57.266 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:46:57.303 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:46:57.303 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:46:57.304 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:46:57.405 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:46:57.450 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:46:57.451 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:46:57.451 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:46:57.552 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:46:57.565 INFO:teuthology.orchestra.run.smithi092.stderr:Inferring config /var/lib/ceph/00db0016-dfed-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T21:46:57.585 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:46:57 smithi097 ceph-mon[33317]: pgmap v1008: 148 pgs: 2 remapped+peering, 31 peering, 115 active+clean; 7.1 GiB data, 23 GiB used, 424 GiB / 447 GiB avail; 341 B/s rd, 0 B/s wr, 1.02k op/s; 135 B/s, 8 objects/s recovering 2024-03-11T21:46:57.585 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:46:57 smithi097 ceph-mon[33317]: purged_snaps scrub starts 2024-03-11T21:46:57.585 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:46:57 smithi097 ceph-mon[33317]: purged_snaps scrub ok 2024-03-11T21:46:57.585 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:46:57 smithi097 ceph-mon[33317]: osdmap e476: 8 total, 8 up, 5 in 2024-03-11T21:46:57.603 INFO:teuthology.orchestra.run.smithi092.stdout:ERROR: (22) Invalid argument 2024-03-11T21:46:57.604 INFO:teuthology.orchestra.run.smithi092.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-11T21:46:57.604 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:46:57 smithi092 ceph-mon[29572]: pgmap v1008: 148 pgs: 2 remapped+peering, 31 peering, 115 active+clean; 7.1 GiB data, 23 GiB used, 424 GiB / 447 GiB avail; 341 B/s rd, 0 B/s wr, 1.02k op/s; 135 B/s, 8 objects/s recovering 2024-03-11T21:46:57.604 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:46:57 smithi092 ceph-mon[29572]: purged_snaps scrub starts 2024-03-11T21:46:57.604 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:46:57 smithi092 ceph-mon[29572]: purged_snaps scrub ok 2024-03-11T21:46:57.604 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:46:57 smithi092 ceph-mon[29572]: osdmap e476: 8 total, 8 up, 5 in 2024-03-11T21:46:57.604 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:46:57 smithi092 ceph-mon[37563]: pgmap v1008: 148 pgs: 2 remapped+peering, 31 peering, 115 active+clean; 7.1 GiB data, 23 GiB used, 424 GiB / 447 GiB avail; 341 B/s rd, 0 B/s wr, 1.02k op/s; 135 B/s, 8 objects/s recovering 2024-03-11T21:46:57.604 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:46:57 smithi092 ceph-mon[37563]: purged_snaps scrub starts 2024-03-11T21:46:57.604 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:46:57 smithi092 ceph-mon[37563]: purged_snaps scrub ok 2024-03-11T21:46:57.605 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:46:57 smithi092 ceph-mon[37563]: osdmap e476: 8 total, 8 up, 5 in 2024-03-11T21:46:57.605 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:46:57.606 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:46:57.606 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:46:57.707 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:46:57.743 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:46:57.743 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:46:57.744 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:46:57.844 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:46:57.876 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:46:57.876 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:46:57.877 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:46:57.977 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:46:58.015 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:46:58.016 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:46:58.016 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:46:58.117 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:46:58.118 INFO:journalctl@ceph.mgr.y.smithi092.stdout:Mar 11 21:46:57 smithi092 ceph-00db0016-dfed-11ee-95c8-87774f69a715-mgr-y[29801]: ::ffff:172.21.15.97 - - [11/Mar/2024:21:46:57] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-11T21:46:58.150 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:46:58.151 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:46:58.151 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:46:58.252 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:46:58.292 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:46:58.293 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:46:58.293 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:46:58.394 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:46:58.433 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:46:58.434 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:46:58.434 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:46:58.535 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:46:58.573 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:46:58.573 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:46:58.574 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:46:58.631 DEBUG:teuthology.orchestra.run.smithi092:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid 00db0016-dfed-11ee-95c8-87774f69a715 -- ceph daemon osd.0 dump_ops_in_flight 2024-03-11T21:46:58.674 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:46:58.717 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:46:58.718 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:46:58.718 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:46:58.819 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:46:58.875 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:46:58.875 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:46:58.876 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:46:58.977 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:46:59.008 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:46:59.008 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:46:59.009 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:46:59.109 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:46:59.153 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:46:59.154 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:46:59.154 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:46:59.255 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:46:59.295 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:46:59.296 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:46:59.296 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:46:59.397 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:46:59.438 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:46:59.438 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:46:59.439 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:46:59.539 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:46:59.572 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:46:59.573 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:46:59.573 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:46:59.584 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:46:59 smithi097 ceph-mon[33317]: pgmap v1010: 148 pgs: 2 remapped+peering, 31 peering, 115 active+clean; 7.1 GiB data, 23 GiB used, 424 GiB / 447 GiB avail; 391 B/s rd, 0 B/s wr, 1.01k op/s; 155 B/s, 9 objects/s recovering 2024-03-11T21:46:59.618 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:46:59 smithi092 ceph-mon[29572]: pgmap v1010: 148 pgs: 2 remapped+peering, 31 peering, 115 active+clean; 7.1 GiB data, 23 GiB used, 424 GiB / 447 GiB avail; 391 B/s rd, 0 B/s wr, 1.01k op/s; 155 B/s, 9 objects/s recovering 2024-03-11T21:46:59.618 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:46:59 smithi092 ceph-mon[37563]: pgmap v1010: 148 pgs: 2 remapped+peering, 31 peering, 115 active+clean; 7.1 GiB data, 23 GiB used, 424 GiB / 447 GiB avail; 391 B/s rd, 0 B/s wr, 1.01k op/s; 155 B/s, 9 objects/s recovering 2024-03-11T21:46:59.675 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:46:59.711 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:46:59.712 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:46:59.712 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:46:59.813 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:46:59.848 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:46:59.848 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:46:59.849 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:46:59.949 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:46:59.988 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:46:59.989 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:46:59.989 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:47:00.090 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:47:00.091 INFO:teuthology.orchestra.run.smithi092.stderr:Inferring config /var/lib/ceph/00db0016-dfed-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T21:47:00.121 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:47:00.122 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:47:00.122 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:47:00.223 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:47:00.257 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:47:00.258 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:47:00.258 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:47:00.284 INFO:teuthology.orchestra.run.smithi092.stderr:noscrub is unset 2024-03-11T21:47:00.347 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 6, 7] out_osds: [3, 4, 5] dead_osds: [] live_osds: [0, 1, 2, 4, 5, 6, 7, 3] 2024-03-11T21:47:00.348 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-03-11T21:47:00.348 INFO:tasks.thrashosds.thrasher:primary_affinity 2024-03-11T21:47:00.348 INFO:tasks.thrashosds.thrasher:Setting osd 1 primary_affinity to 0.556418 2024-03-11T21:47:00.348 DEBUG:teuthology.orchestra.run.smithi092:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid 00db0016-dfed-11ee-95c8-87774f69a715 -- ceph osd primary-affinity 1 0.5564177019329419 2024-03-11T21:47:00.359 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:47:00.437 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:47:00.437 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:47:00.438 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:47:00.538 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:47:00.584 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:47:00 smithi097 ceph-mon[33317]: from='client.? v1:172.21.15.92:0/34336446' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-11T21:47:00.589 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:47:00.590 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:47:00.590 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:47:00.618 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:47:00 smithi092 ceph-mon[29572]: from='client.? v1:172.21.15.92:0/34336446' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-11T21:47:00.619 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:47:00 smithi092 ceph-mon[37563]: from='client.? v1:172.21.15.92:0/34336446' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-11T21:47:00.691 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:47:00.726 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:47:00.727 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:47:00.727 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:47:00.828 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:47:00.863 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:47:00.863 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:47:00.863 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:47:00.964 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:47:01.010 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:47:01.011 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:47:01.011 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:47:01.112 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:47:01.151 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:47:01.151 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:47:01.152 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:47:01.252 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:47:01.286 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:47:01.286 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:47:01.287 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:47:01.387 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:47:01.427 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:47:01.428 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:47:01.428 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:47:01.529 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:47:01.561 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:47:01.562 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:47:01.562 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:47:01.583 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:47:01 smithi092 ceph-mon[29572]: pgmap v1011: 148 pgs: 3 active+remapped+backfill_toofull, 2 active+remapped+backfilling, 143 active+clean; 6.8 GiB data, 23 GiB used, 424 GiB / 447 GiB avail; 1.6 KiB/s rd, 0 B/s wr, 1.16k op/s; 34247/335631 objects misplaced (10.204%); 8.6 MiB/s, 152 objects/s recovering 2024-03-11T21:47:01.583 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:47:01 smithi092 ceph-mon[29572]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-11T21:47:01.583 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:47:01 smithi092 ceph-mon[29572]: Health check failed: Low space hindering backfill (add storage if this doesn't resolve itself): 3 pgs backfill_toofull (PG_BACKFILL_FULL) 2024-03-11T21:47:01.583 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:47:01 smithi092 ceph-mon[29572]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 4 pgs inactive, 8 pgs peering) 2024-03-11T21:47:01.583 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:47:01 smithi092 ceph-mon[29572]: from='client.? v1:172.21.15.92:0/34336446' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-03-11T21:47:01.583 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:47:01 smithi092 ceph-mon[29572]: osdmap e477: 8 total, 8 up, 5 in 2024-03-11T21:47:01.583 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:47:01 smithi092 ceph-mon[37563]: pgmap v1011: 148 pgs: 3 active+remapped+backfill_toofull, 2 active+remapped+backfilling, 143 active+clean; 6.8 GiB data, 23 GiB used, 424 GiB / 447 GiB avail; 1.6 KiB/s rd, 0 B/s wr, 1.16k op/s; 34247/335631 objects misplaced (10.204%); 8.6 MiB/s, 152 objects/s recovering 2024-03-11T21:47:01.584 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:47:01 smithi092 ceph-mon[37563]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-11T21:47:01.584 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:47:01 smithi092 ceph-mon[37563]: Health check failed: Low space hindering backfill (add storage if this doesn't resolve itself): 3 pgs backfill_toofull (PG_BACKFILL_FULL) 2024-03-11T21:47:01.584 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:47:01 smithi092 ceph-mon[37563]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 4 pgs inactive, 8 pgs peering) 2024-03-11T21:47:01.584 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:47:01 smithi092 ceph-mon[37563]: from='client.? v1:172.21.15.92:0/34336446' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-03-11T21:47:01.584 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:47:01 smithi092 ceph-mon[37563]: osdmap e477: 8 total, 8 up, 5 in 2024-03-11T21:47:01.584 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:47:01 smithi097 ceph-mon[33317]: pgmap v1011: 148 pgs: 3 active+remapped+backfill_toofull, 2 active+remapped+backfilling, 143 active+clean; 6.8 GiB data, 23 GiB used, 424 GiB / 447 GiB avail; 1.6 KiB/s rd, 0 B/s wr, 1.16k op/s; 34247/335631 objects misplaced (10.204%); 8.6 MiB/s, 152 objects/s recovering 2024-03-11T21:47:01.584 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:47:01 smithi097 ceph-mon[33317]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-11T21:47:01.584 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:47:01 smithi097 ceph-mon[33317]: Health check failed: Low space hindering backfill (add storage if this doesn't resolve itself): 3 pgs backfill_toofull (PG_BACKFILL_FULL) 2024-03-11T21:47:01.584 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:47:01 smithi097 ceph-mon[33317]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 4 pgs inactive, 8 pgs peering) 2024-03-11T21:47:01.584 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:47:01 smithi097 ceph-mon[33317]: from='client.? v1:172.21.15.92:0/34336446' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-03-11T21:47:01.584 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:47:01 smithi097 ceph-mon[33317]: osdmap e477: 8 total, 8 up, 5 in 2024-03-11T21:47:01.663 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:47:01.703 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:47:01.704 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:47:01.705 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:47:01.805 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:47:01.818 INFO:teuthology.orchestra.run.smithi092.stdout:osd.0: {} 2024-03-11T21:47:01.852 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:47:01.853 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:47:01.853 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:47:01.868 INFO:teuthology.orchestra.run.smithi092.stdout:osd.1: {} 2024-03-11T21:47:01.911 INFO:teuthology.orchestra.run.smithi092.stdout:osd.2: {} 2024-03-11T21:47:01.953 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:47:01.955 INFO:teuthology.orchestra.run.smithi092.stdout:osd.3: {} 2024-03-11T21:47:01.984 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:47:01.985 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:47:01.985 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:47:01.994 INFO:teuthology.orchestra.run.smithi092.stdout:osd.4: {} 2024-03-11T21:47:02.035 INFO:teuthology.orchestra.run.smithi092.stdout:osd.5: {} 2024-03-11T21:47:02.072 INFO:teuthology.orchestra.run.smithi092.stdout:osd.6: {} 2024-03-11T21:47:02.086 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:47:02.115 INFO:teuthology.orchestra.run.smithi092.stdout:osd.7: {} 2024-03-11T21:47:02.119 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:47:02.120 INFO:teuthology.orchestra.run.smithi092.stderr:osd.0: osd_enable_op_tracker = '' 2024-03-11T21:47:02.120 INFO:teuthology.orchestra.run.smithi092.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-11T21:47:02.120 INFO:teuthology.orchestra.run.smithi092.stderr:osd.2: osd_enable_op_tracker = '' 2024-03-11T21:47:02.120 INFO:teuthology.orchestra.run.smithi092.stderr:osd.3: osd_enable_op_tracker = '' 2024-03-11T21:47:02.120 INFO:teuthology.orchestra.run.smithi092.stderr:osd.4: osd_enable_op_tracker = '' 2024-03-11T21:47:02.120 INFO:teuthology.orchestra.run.smithi092.stderr:osd.5: osd_enable_op_tracker = '' 2024-03-11T21:47:02.120 INFO:teuthology.orchestra.run.smithi092.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-11T21:47:02.121 INFO:teuthology.orchestra.run.smithi092.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-11T21:47:02.121 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:47:02.121 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:47:02.222 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:47:02.262 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:47:02.263 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:47:02.263 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:47:02.364 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:47:02.404 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:47:02.405 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:47:02.405 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:47:02.506 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:47:02.541 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:47:02.541 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:47:02.542 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:47:02.584 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:47:02 smithi097 ceph-mon[33317]: from='client.? v1:172.21.15.92:0/1826391830' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-11T21:47:02.618 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:47:02 smithi092 ceph-mon[29572]: from='client.? v1:172.21.15.92:0/1826391830' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-11T21:47:02.619 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:47:02 smithi092 ceph-mon[37563]: from='client.? v1:172.21.15.92:0/1826391830' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-11T21:47:02.643 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:47:02.680 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:47:02.681 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:47:02.681 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:47:02.782 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:47:02.814 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:47:02.814 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:47:02.815 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:47:02.915 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:47:02.941 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:47:02.942 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:47:02.942 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:47:03.043 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:47:03.078 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:47:03.079 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:47:03.079 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:47:03.180 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:47:03.219 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:47:03.220 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:47:03.220 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:47:03.321 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:47:03.354 DEBUG:teuthology.orchestra.run.smithi092:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid 00db0016-dfed-11ee-95c8-87774f69a715 -- ceph osd unset nodeep-scrub 2024-03-11T21:47:03.357 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:47:03.357 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:47:03.358 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:47:03.459 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:47:03.488 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:47:03.489 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:47:03.489 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:47:03.584 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:47:03 smithi097 ceph-mon[33317]: pgmap v1013: 148 pgs: 3 active+remapped+backfill_toofull, 2 active+remapped+backfilling, 143 active+clean; 6.8 GiB data, 22 GiB used, 425 GiB / 447 GiB avail; 708 B/s rd, 0 B/s wr, 807 op/s; 34247/334290 objects misplaced (10.245%); 7.4 MiB/s, 127 objects/s recovering 2024-03-11T21:47:03.584 INFO:journalctl@ceph.iscsi.iscsi.a.smithi097.stdout:Mar 11 21:47:03 smithi097 ceph-00db0016-dfed-11ee-95c8-87774f69a715-iscsi-iscsi-a[60943]: debug there is no tcmu-runner data available 2024-03-11T21:47:03.590 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:47:03.618 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:47:03 smithi092 ceph-mon[29572]: pgmap v1013: 148 pgs: 3 active+remapped+backfill_toofull, 2 active+remapped+backfilling, 143 active+clean; 6.8 GiB data, 22 GiB used, 425 GiB / 447 GiB avail; 708 B/s rd, 0 B/s wr, 807 op/s; 34247/334290 objects misplaced (10.245%); 7.4 MiB/s, 127 objects/s recovering 2024-03-11T21:47:03.619 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:47:03 smithi092 ceph-mon[37563]: pgmap v1013: 148 pgs: 3 active+remapped+backfill_toofull, 2 active+remapped+backfilling, 143 active+clean; 6.8 GiB data, 22 GiB used, 425 GiB / 447 GiB avail; 708 B/s rd, 0 B/s wr, 807 op/s; 34247/334290 objects misplaced (10.245%); 7.4 MiB/s, 127 objects/s recovering 2024-03-11T21:47:03.634 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:47:03.635 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:47:03.635 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:47:03.736 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:47:03.779 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:47:03.780 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:47:03.780 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:47:03.881 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:47:03.918 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:47:03.919 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:47:03.919 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:47:04.020 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:47:04.055 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:47:04.055 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:47:04.056 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:47:04.157 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:47:04.200 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:47:04.200 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:47:04.201 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:47:04.302 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:47:04.335 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:47:04.336 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:47:04.336 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:47:04.368 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:47:04 smithi092 ceph-mon[29572]: from='client.24469 v1:172.21.15.97:0/404208310' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:47:04.369 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:47:04 smithi092 ceph-mon[37563]: from='client.24469 v1:172.21.15.97:0/404208310' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:47:04.437 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:47:04.474 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:47:04.475 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:47:04.475 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:47:04.576 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:47:04.584 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:47:04 smithi097 ceph-mon[33317]: from='client.24469 v1:172.21.15.97:0/404208310' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:47:04.613 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:47:04.614 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:47:04.614 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:47:04.715 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:47:04.758 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:47:04.759 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:47:04.759 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:47:04.860 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:47:04.901 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:47:04.902 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:47:04.902 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:47:05.003 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:47:05.045 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:47:05.045 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:47:05.046 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:47:05.146 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:47:05.180 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:47:05.181 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:47:05.181 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:47:05.282 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:47:05.318 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:47:05.319 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:47:05.319 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:47:05.420 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:47:05.457 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:47:05.458 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:47:05.458 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:47:05.540 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:47:05 smithi097 ceph-mon[33317]: pgmap v1014: 148 pgs: 3 active+remapped+backfill_toofull, 2 active+remapped+backfilling, 143 active+clean; 6.5 GiB data, 22 GiB used, 425 GiB / 447 GiB avail; 639 B/s rd, 0 B/s wr, 1.27k op/s; 31633/320544 objects misplaced (9.869%); 17 MiB/s, 277 objects/s recovering 2024-03-11T21:47:05.540 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:47:05 smithi097 ceph-mon[33317]: from='mgr.14607 v1:172.21.15.92:0/1702717597' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T21:47:05.561 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:47:05.598 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:47:05.599 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:47:05.599 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:47:05.618 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:47:05 smithi092 ceph-mon[29572]: pgmap v1014: 148 pgs: 3 active+remapped+backfill_toofull, 2 active+remapped+backfilling, 143 active+clean; 6.5 GiB data, 22 GiB used, 425 GiB / 447 GiB avail; 639 B/s rd, 0 B/s wr, 1.27k op/s; 31633/320544 objects misplaced (9.869%); 17 MiB/s, 277 objects/s recovering 2024-03-11T21:47:05.618 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:47:05 smithi092 ceph-mon[29572]: from='mgr.14607 v1:172.21.15.92:0/1702717597' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T21:47:05.618 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:47:05 smithi092 ceph-mon[37563]: pgmap v1014: 148 pgs: 3 active+remapped+backfill_toofull, 2 active+remapped+backfilling, 143 active+clean; 6.5 GiB data, 22 GiB used, 425 GiB / 447 GiB avail; 639 B/s rd, 0 B/s wr, 1.27k op/s; 31633/320544 objects misplaced (9.869%); 17 MiB/s, 277 objects/s recovering 2024-03-11T21:47:05.619 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:47:05 smithi092 ceph-mon[37563]: from='mgr.14607 v1:172.21.15.92:0/1702717597' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T21:47:05.700 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:47:05.740 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:47:05.740 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:47:05.741 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:47:05.842 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:47:05.864 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:47:05.865 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:47:05.866 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:47:05.966 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:47:06.003 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:47:06.004 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:47:06.004 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:47:06.105 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:47:06.142 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:47:06.143 DEBUG:teuthology.orchestra.run.smithi092:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid 00db0016-dfed-11ee-95c8-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-03-11T21:47:06.143 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:47:06.144 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:47:06.245 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:47:06.279 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:47:06.279 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:47:06.280 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:47:06.381 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:47:06.426 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:47:06.426 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:47:06.427 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:47:06.527 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:47:06.554 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:47:06.555 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:47:06.555 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:47:06.656 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:47:06.695 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:47:06.696 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:47:06.696 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:47:06.796 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:47:06.838 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:47:06.839 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:47:06.839 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:47:06.940 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:47:06.972 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:47:06.973 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:47:06.973 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:47:07.074 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:47:07.109 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:47:07.109 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:47:07.110 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:47:07.211 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:47:07.247 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:47:07.248 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:47:07.248 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:47:07.349 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:47:07.381 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:47:07.382 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:47:07.382 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:47:07.483 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:47:07.523 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:47:07.523 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:47:07.524 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:47:07.563 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:47:07 smithi092 ceph-mon[29572]: pgmap v1015: 148 pgs: 3 active+remapped+backfill_toofull, 2 active+remapped+backfilling, 143 active+clean; 6.3 GiB data, 21 GiB used, 426 GiB / 447 GiB avail; 1.1 KiB/s rd, 0 B/s wr, 1.39k op/s; 30763/312669 objects misplaced (9.839%); 15 MiB/s, 241 objects/s recovering 2024-03-11T21:47:07.563 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:47:07 smithi092 ceph-mon[29572]: 6.19 scrub starts 2024-03-11T21:47:07.563 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:47:07 smithi092 ceph-mon[29572]: 6.19 scrub ok 2024-03-11T21:47:07.563 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:47:07 smithi092 ceph-mon[29572]: from='mgr.14607 v1:172.21.15.92:0/1702717597' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T21:47:07.563 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:47:07 smithi092 ceph-mon[29572]: from='mgr.14607 v1:172.21.15.92:0/1702717597' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T21:47:07.563 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:47:07 smithi092 ceph-mon[29572]: from='mgr.14607 ' entity='mgr.y' 2024-03-11T21:47:07.563 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:47:07 smithi092 ceph-mon[37563]: pgmap v1015: 148 pgs: 3 active+remapped+backfill_toofull, 2 active+remapped+backfilling, 143 active+clean; 6.3 GiB data, 21 GiB used, 426 GiB / 447 GiB avail; 1.1 KiB/s rd, 0 B/s wr, 1.39k op/s; 30763/312669 objects misplaced (9.839%); 15 MiB/s, 241 objects/s recovering 2024-03-11T21:47:07.564 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:47:07 smithi092 ceph-mon[37563]: 6.19 scrub starts 2024-03-11T21:47:07.564 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:47:07 smithi092 ceph-mon[37563]: 6.19 scrub ok 2024-03-11T21:47:07.564 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:47:07 smithi092 ceph-mon[37563]: from='mgr.14607 v1:172.21.15.92:0/1702717597' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T21:47:07.564 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:47:07 smithi092 ceph-mon[37563]: from='mgr.14607 v1:172.21.15.92:0/1702717597' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T21:47:07.564 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:47:07 smithi092 ceph-mon[37563]: from='mgr.14607 ' entity='mgr.y' 2024-03-11T21:47:07.584 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:47:07 smithi097 ceph-mon[33317]: pgmap v1015: 148 pgs: 3 active+remapped+backfill_toofull, 2 active+remapped+backfilling, 143 active+clean; 6.3 GiB data, 21 GiB used, 426 GiB / 447 GiB avail; 1.1 KiB/s rd, 0 B/s wr, 1.39k op/s; 30763/312669 objects misplaced (9.839%); 15 MiB/s, 241 objects/s recovering 2024-03-11T21:47:07.584 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:47:07 smithi097 ceph-mon[33317]: 6.19 scrub starts 2024-03-11T21:47:07.584 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:47:07 smithi097 ceph-mon[33317]: 6.19 scrub ok 2024-03-11T21:47:07.584 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:47:07 smithi097 ceph-mon[33317]: from='mgr.14607 v1:172.21.15.92:0/1702717597' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T21:47:07.584 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:47:07 smithi097 ceph-mon[33317]: from='mgr.14607 v1:172.21.15.92:0/1702717597' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T21:47:07.584 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:47:07 smithi097 ceph-mon[33317]: from='mgr.14607 ' entity='mgr.y' 2024-03-11T21:47:07.625 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:47:07.636 INFO:teuthology.orchestra.run.smithi092.stderr:Inferring config /var/lib/ceph/00db0016-dfed-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T21:47:07.655 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:47:07.656 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:47:07.656 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:47:07.757 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:47:07.801 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:47:07.802 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:47:07.802 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:47:07.804 INFO:teuthology.orchestra.run.smithi092.stderr:Inferring config /var/lib/ceph/00db0016-dfed-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T21:47:07.903 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:47:07.939 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:47:07.940 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:47:07.940 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:47:08.040 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:47:08.081 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:47:08.082 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:47:08.082 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:47:08.183 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:47:08.213 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:47:08.214 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:47:08.214 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:47:08.303 INFO:journalctl@ceph.mgr.y.smithi092.stdout:Mar 11 21:47:07 smithi092 ceph-00db0016-dfed-11ee-95c8-87774f69a715-mgr-y[29801]: ::ffff:172.21.15.97 - - [11/Mar/2024:21:47:07] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-11T21:47:08.316 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:47:08.353 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:47:08.354 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:47:08.355 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:47:08.456 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:47:08.488 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:47:08.490 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:47:08.491 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:47:08.592 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:47:08.632 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:47:08.632 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:47:08.633 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:47:08.733 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:47:08.775 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:47:08.776 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:47:08.777 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:47:08.866 INFO:teuthology.orchestra.run.smithi092.stdout:{ 2024-03-11T21:47:08.866 INFO:teuthology.orchestra.run.smithi092.stdout: "ops": [ 2024-03-11T21:47:08.866 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:47:08.866 INFO:teuthology.orchestra.run.smithi092.stdout: "description": "MOSDPGPush(11.5 477/474 [PushOp(11:a5008b72:::benchmark_data_smithi092_186520_object46763:head, version: 454'2954, data_included: [0~65536], data_size: 65536, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(11:a5008b72:::benchmark_data_smithi092_186520_object46763:head@454'2954, size: 65536, copy_subset: [(0, 65536)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 65536, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-03-11T21:47:08.867 INFO:teuthology.orchestra.run.smithi092.stdout: "initiated_at": "2024-03-11T21:47:08.871783+0000", 2024-03-11T21:47:08.867 INFO:teuthology.orchestra.run.smithi092.stdout: "age": 0.0020368560000000001, 2024-03-11T21:47:08.867 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.0020916590000000001, 2024-03-11T21:47:08.867 INFO:teuthology.orchestra.run.smithi092.stdout: "type_data": { 2024-03-11T21:47:08.867 INFO:teuthology.orchestra.run.smithi092.stdout: "flag_point": "queued for pg", 2024-03-11T21:47:08.867 INFO:teuthology.orchestra.run.smithi092.stdout: "events": [ 2024-03-11T21:47:08.867 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:47:08.867 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "initiated", 2024-03-11T21:47:08.867 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:47:08.871783+0000", 2024-03-11T21:47:08.867 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-03-11T21:47:08.867 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:47:08.867 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:47:08.867 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "header_read", 2024-03-11T21:47:08.868 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:47:08.871783+0000", 2024-03-11T21:47:08.868 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-03-11T21:47:08.868 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:47:08.868 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:47:08.868 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "throttled", 2024-03-11T21:47:08.868 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:47:08.871784+0000", 2024-03-11T21:47:08.868 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 5.1699999999999998e-07 2024-03-11T21:47:08.868 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:47:08.868 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:47:08.868 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "all_read", 2024-03-11T21:47:08.868 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:47:08.871819+0000", 2024-03-11T21:47:08.868 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 3.5185999999999998e-05 2024-03-11T21:47:08.869 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:47:08.869 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:47:08.869 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "dispatched", 2024-03-11T21:47:08.869 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:47:08.871837+0000", 2024-03-11T21:47:08.869 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.7702000000000001e-05 2024-03-11T21:47:08.869 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:47:08.869 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:47:08.869 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "queued_for_pg", 2024-03-11T21:47:08.869 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:47:08.871852+0000", 2024-03-11T21:47:08.869 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.5413e-05 2024-03-11T21:47:08.869 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-03-11T21:47:08.869 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-03-11T21:47:08.869 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-03-11T21:47:08.870 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:47:08.870 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:47:08.870 INFO:teuthology.orchestra.run.smithi092.stdout: "description": "osd_repop(client.35813.0:180249 11.8 e477/474 11:1b65eb78:::benchmark_data_smithi092_186520_object41901:head v 477'11345, mlcod=477'11343)", 2024-03-11T21:47:08.870 INFO:teuthology.orchestra.run.smithi092.stdout: "initiated_at": "2024-03-11T21:47:08.871854+0000", 2024-03-11T21:47:08.870 INFO:teuthology.orchestra.run.smithi092.stdout: "age": 0.001966588, 2024-03-11T21:47:08.870 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.0020950489999999999, 2024-03-11T21:47:08.870 INFO:teuthology.orchestra.run.smithi092.stdout: "type_data": { 2024-03-11T21:47:08.870 INFO:teuthology.orchestra.run.smithi092.stdout: "flag_point": "started", 2024-03-11T21:47:08.870 INFO:teuthology.orchestra.run.smithi092.stdout: "events": [ 2024-03-11T21:47:08.870 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:47:08.870 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "initiated", 2024-03-11T21:47:08.870 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:47:08.871854+0000", 2024-03-11T21:47:08.870 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-03-11T21:47:08.870 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:47:08.871 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:47:08.871 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "header_read", 2024-03-11T21:47:08.871 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:47:08.871854+0000", 2024-03-11T21:47:08.871 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-03-11T21:47:08.871 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:47:08.871 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:47:08.871 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "throttled", 2024-03-11T21:47:08.871 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:47:08.871855+0000", 2024-03-11T21:47:08.871 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.341e-06 2024-03-11T21:47:08.871 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:47:08.871 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:47:08.871 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "all_read", 2024-03-11T21:47:08.871 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:47:08.871865+0000", 2024-03-11T21:47:08.872 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 9.8050000000000001e-06 2024-03-11T21:47:08.872 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:47:08.872 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:47:08.872 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "dispatched", 2024-03-11T21:47:08.872 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:47:08.871874+0000", 2024-03-11T21:47:08.872 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 9.1269999999999992e-06 2024-03-11T21:47:08.872 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:47:08.872 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:47:08.872 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "queued_for_pg", 2024-03-11T21:47:08.872 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:47:08.871881+0000", 2024-03-11T21:47:08.872 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 6.5919999999999997e-06 2024-03-11T21:47:08.872 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:47:08.872 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:47:08.872 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-03-11T21:47:08.872 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:47:08.871934+0000", 2024-03-11T21:47:08.873 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 5.3675999999999999e-05 2024-03-11T21:47:08.873 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:47:08.873 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:47:08.873 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "started", 2024-03-11T21:47:08.873 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:47:08.871956+0000", 2024-03-11T21:47:08.873 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 2.2036e-05 2024-03-11T21:47:08.873 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-03-11T21:47:08.873 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-03-11T21:47:08.873 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-03-11T21:47:08.873 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:47:08.873 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:47:08.873 INFO:teuthology.orchestra.run.smithi092.stdout: "description": "osd_repop(client.35813.0:180248 11.2 e477/446 11:4b4bcc9d:::benchmark_data_smithi092_186520_object41900:head v 477'11030, mlcod=477'11028)", 2024-03-11T21:47:08.873 INFO:teuthology.orchestra.run.smithi092.stdout: "initiated_at": "2024-03-11T21:47:08.871882+0000", 2024-03-11T21:47:08.873 INFO:teuthology.orchestra.run.smithi092.stdout: "age": 0.0019382119999999999, 2024-03-11T21:47:08.874 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.002143275, 2024-03-11T21:47:08.874 INFO:teuthology.orchestra.run.smithi092.stdout: "type_data": { 2024-03-11T21:47:08.874 INFO:teuthology.orchestra.run.smithi092.stdout: "flag_point": "started", 2024-03-11T21:47:08.874 INFO:teuthology.orchestra.run.smithi092.stdout: "events": [ 2024-03-11T21:47:08.874 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:47:08.874 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "initiated", 2024-03-11T21:47:08.874 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:47:08.871882+0000", 2024-03-11T21:47:08.874 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-03-11T21:47:08.874 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:47:08.874 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:47:08.874 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "header_read", 2024-03-11T21:47:08.874 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:47:08.871882+0000", 2024-03-11T21:47:08.874 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-03-11T21:47:08.875 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:47:08.875 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:47:08.875 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "throttled", 2024-03-11T21:47:08.875 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:47:08.871883+0000", 2024-03-11T21:47:08.875 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.026e-06 2024-03-11T21:47:08.875 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:47:08.875 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:47:08.875 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "all_read", 2024-03-11T21:47:08.875 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:47:08.871889+0000", 2024-03-11T21:47:08.875 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 5.8669999999999999e-06 2024-03-11T21:47:08.875 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:47:08.875 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:47:08.875 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "dispatched", 2024-03-11T21:47:08.875 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:47:08.871896+0000", 2024-03-11T21:47:08.875 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 7.3329999999999999e-06 2024-03-11T21:47:08.876 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:47:08.876 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:47:08.876 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "queued_for_pg", 2024-03-11T21:47:08.876 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:47:08.871902+0000", 2024-03-11T21:47:08.876 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 5.3070000000000002e-06 2024-03-11T21:47:08.876 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:47:08.876 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:47:08.876 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-03-11T21:47:08.876 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:47:08.871952+0000", 2024-03-11T21:47:08.876 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 5.0073e-05 2024-03-11T21:47:08.876 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:47:08.876 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:47:08.876 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "started", 2024-03-11T21:47:08.876 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:47:08.871971+0000", 2024-03-11T21:47:08.876 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.8876e-05 2024-03-11T21:47:08.877 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-03-11T21:47:08.877 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-03-11T21:47:08.877 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-03-11T21:47:08.877 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:47:08.877 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:47:08.877 INFO:teuthology.orchestra.run.smithi092.stdout: "description": "osd_repop(client.35813.0:180250 11.1 e477/446)", 2024-03-11T21:47:08.877 INFO:teuthology.orchestra.run.smithi092.stdout: "initiated_at": "2024-03-11T21:47:08.873839+0000", 2024-03-11T21:47:08.877 INFO:teuthology.orchestra.run.smithi092.stdout: "age": 4294967295.9999814, 2024-03-11T21:47:08.877 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00024868200000000002, 2024-03-11T21:47:08.877 INFO:teuthology.orchestra.run.smithi092.stdout: "type_data": { 2024-03-11T21:47:08.877 INFO:teuthology.orchestra.run.smithi092.stdout: "flag_point": "queued for pg", 2024-03-11T21:47:08.877 INFO:teuthology.orchestra.run.smithi092.stdout: "events": [ 2024-03-11T21:47:08.877 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:47:08.877 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "initiated", 2024-03-11T21:47:08.877 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:47:08.873839+0000", 2024-03-11T21:47:08.878 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-03-11T21:47:08.878 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:47:08.878 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:47:08.878 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "header_read", 2024-03-11T21:47:08.878 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:47:08.873839+0000", 2024-03-11T21:47:08.878 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-03-11T21:47:08.878 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:47:08.878 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:47:08.878 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "throttled", 2024-03-11T21:47:08.878 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:47:08.873840+0000", 2024-03-11T21:47:08.878 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.4389999999999999e-06 2024-03-11T21:47:08.878 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:47:08.878 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:47:08.878 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "all_read", 2024-03-11T21:47:08.879 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:47:08.873851+0000", 2024-03-11T21:47:08.879 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.0628000000000001e-05 2024-03-11T21:47:08.879 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:47:08.879 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:47:08.879 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "dispatched", 2024-03-11T21:47:08.879 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:47:08.873860+0000", 2024-03-11T21:47:08.879 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 9.2099999999999999e-06 2024-03-11T21:47:08.879 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:47:08.879 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:47:08.879 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "queued_for_pg", 2024-03-11T21:47:08.879 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:47:08.873867+0000", 2024-03-11T21:47:08.879 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 7.0299999999999996e-06 2024-03-11T21:47:08.879 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-03-11T21:47:08.880 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:47:08.881 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-03-11T21:47:08.881 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-03-11T21:47:08.881 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:47:08.881 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:47:08.881 INFO:teuthology.orchestra.run.smithi092.stdout: "description": "osd_repop(client.35813.0:180237 11.1 e477/446 11:833a6843:::benchmark_data_smithi092_186520_object41889:head v 477'11293, mlcod=477'11291)", 2024-03-11T21:47:08.881 INFO:teuthology.orchestra.run.smithi092.stdout: "initiated_at": "2024-03-11T21:47:08.863713+0000", 2024-03-11T21:47:08.881 INFO:teuthology.orchestra.run.smithi092.stdout: "age": 0.010107583, 2024-03-11T21:47:08.881 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.010442665, 2024-03-11T21:47:08.881 INFO:teuthology.orchestra.run.smithi092.stdout: "type_data": { 2024-03-11T21:47:08.881 INFO:teuthology.orchestra.run.smithi092.stdout: "flag_point": "started", 2024-03-11T21:47:08.881 INFO:teuthology.orchestra.run.smithi092.stdout: "events": [ 2024-03-11T21:47:08.881 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:47:08.882 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "initiated", 2024-03-11T21:47:08.882 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:47:08.863713+0000", 2024-03-11T21:47:08.882 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-03-11T21:47:08.882 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:47:08.882 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:47:08.882 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "header_read", 2024-03-11T21:47:08.882 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:47:08.863713+0000", 2024-03-11T21:47:08.882 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-03-11T21:47:08.882 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:47:08.882 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:47:08.882 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "throttled", 2024-03-11T21:47:08.882 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:47:08.863714+0000", 2024-03-11T21:47:08.882 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.5099999999999999e-06 2024-03-11T21:47:08.883 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:47:08.883 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:47:08.883 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "all_read", 2024-03-11T21:47:08.883 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:47:08.863724+0000", 2024-03-11T21:47:08.883 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 9.3170000000000008e-06 2024-03-11T21:47:08.883 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:47:08.883 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:47:08.883 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "dispatched", 2024-03-11T21:47:08.883 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:47:08.863750+0000", 2024-03-11T21:47:08.883 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 2.6310000000000001e-05 2024-03-11T21:47:08.883 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:47:08.883 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:47:08.883 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "queued_for_pg", 2024-03-11T21:47:08.883 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:47:08.863758+0000", 2024-03-11T21:47:08.884 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 8.3580000000000003e-06 2024-03-11T21:47:08.884 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:47:08.884 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:47:08.884 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-03-11T21:47:08.884 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:47:08.864358+0000", 2024-03-11T21:47:08.884 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00059933999999999996 2024-03-11T21:47:08.884 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:47:08.884 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:47:08.884 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "started", 2024-03-11T21:47:08.884 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:47:08.864375+0000", 2024-03-11T21:47:08.884 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.7455000000000001e-05 2024-03-11T21:47:08.884 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-03-11T21:47:08.884 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-03-11T21:47:08.884 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-03-11T21:47:08.884 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:47:08.884 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:47:08.884 INFO:teuthology.orchestra.run.smithi092.stdout: "description": "osd_repop(client.35813.0:180240 11.1 e477/446 11:8dd78935:::benchmark_data_smithi092_186520_object41892:head v 477'11294, mlcod=477'11291)", 2024-03-11T21:47:08.885 INFO:teuthology.orchestra.run.smithi092.stdout: "initiated_at": "2024-03-11T21:47:08.865815+0000", 2024-03-11T21:47:08.885 INFO:teuthology.orchestra.run.smithi092.stdout: "age": 0.0080055730000000002, 2024-03-11T21:47:08.885 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.0084101339999999997, 2024-03-11T21:47:08.885 INFO:teuthology.orchestra.run.smithi092.stdout: "type_data": { 2024-03-11T21:47:08.885 INFO:teuthology.orchestra.run.smithi092.stdout: "flag_point": "started", 2024-03-11T21:47:08.885 INFO:teuthology.orchestra.run.smithi092.stdout: "events": [ 2024-03-11T21:47:08.885 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:47:08.885 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "initiated", 2024-03-11T21:47:08.885 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:47:08.865815+0000", 2024-03-11T21:47:08.885 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-03-11T21:47:08.885 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:47:08.885 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:47:08.885 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "header_read", 2024-03-11T21:47:08.885 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:47:08.865815+0000", 2024-03-11T21:47:08.885 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-03-11T21:47:08.886 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:47:08.886 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:47:08.886 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "throttled", 2024-03-11T21:47:08.886 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:47:08.865816+0000", 2024-03-11T21:47:08.886 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.37e-06 2024-03-11T21:47:08.886 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:47:08.886 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:47:08.886 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "all_read", 2024-03-11T21:47:08.886 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:47:08.865824+0000", 2024-03-11T21:47:08.886 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 7.7479999999999993e-06 2024-03-11T21:47:08.886 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:47:08.886 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:47:08.887 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "dispatched", 2024-03-11T21:47:08.887 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:47:08.865835+0000", 2024-03-11T21:47:08.887 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.079e-05 2024-03-11T21:47:08.887 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:47:08.887 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:47:08.887 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "queued_for_pg", 2024-03-11T21:47:08.887 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:47:08.865842+0000", 2024-03-11T21:47:08.887 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 7.3640000000000001e-06 2024-03-11T21:47:08.887 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:47:08.887 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:47:08.887 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-03-11T21:47:08.888 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:47:08.866480+0000", 2024-03-11T21:47:08.888 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.000638399 2024-03-11T21:47:08.888 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:47:08.888 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:47:08.888 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "started", 2024-03-11T21:47:08.888 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:47:08.869116+0000", 2024-03-11T21:47:08.888 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.0026359539999999998 2024-03-11T21:47:08.888 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-03-11T21:47:08.888 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-03-11T21:47:08.888 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-03-11T21:47:08.888 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:47:08.888 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:47:08.889 INFO:teuthology.orchestra.run.smithi092.stdout: "description": "osd_repop(client.35813.0:180217 11.e e477/446 11:7c62ad85:::benchmark_data_smithi092_186520_object41869:head v 477'11086, mlcod=477'11082)", 2024-03-11T21:47:08.889 INFO:teuthology.orchestra.run.smithi092.stdout: "initiated_at": "2024-03-11T21:47:08.866440+0000", 2024-03-11T21:47:08.889 INFO:teuthology.orchestra.run.smithi092.stdout: "age": 0.0073806990000000001, 2024-03-11T21:47:08.889 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.0078440149999999993, 2024-03-11T21:47:08.889 INFO:teuthology.orchestra.run.smithi092.stdout: "type_data": { 2024-03-11T21:47:08.889 INFO:teuthology.orchestra.run.smithi092.stdout: "flag_point": "started", 2024-03-11T21:47:08.889 INFO:teuthology.orchestra.run.smithi092.stdout: "events": [ 2024-03-11T21:47:08.889 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:47:08.889 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "initiated", 2024-03-11T21:47:08.889 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:47:08.866440+0000", 2024-03-11T21:47:08.889 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-03-11T21:47:08.889 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:47:08.889 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:47:08.890 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "header_read", 2024-03-11T21:47:08.890 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:47:08.866440+0000", 2024-03-11T21:47:08.890 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-03-11T21:47:08.890 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:47:08.890 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:47:08.890 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "throttled", 2024-03-11T21:47:08.890 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:47:08.866440+0000", 2024-03-11T21:47:08.890 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 4.0900000000000002e-07 2024-03-11T21:47:08.890 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:47:08.890 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:47:08.890 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "all_read", 2024-03-11T21:47:08.890 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:47:08.866445+0000", 2024-03-11T21:47:08.890 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 5.0520000000000001e-06 2024-03-11T21:47:08.891 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:47:08.891 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:47:08.891 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "dispatched", 2024-03-11T21:47:08.891 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:47:08.866450+0000", 2024-03-11T21:47:08.891 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 5.2700000000000004e-06 2024-03-11T21:47:08.891 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:47:08.891 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:47:08.891 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "queued_for_pg", 2024-03-11T21:47:08.891 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:47:08.866455+0000", 2024-03-11T21:47:08.891 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 4.4880000000000001e-06 2024-03-11T21:47:08.891 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:47:08.891 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:47:08.891 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-03-11T21:47:08.891 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:47:08.867208+0000", 2024-03-11T21:47:08.891 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00075359799999999996 2024-03-11T21:47:08.892 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:47:08.892 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:47:08.892 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "started", 2024-03-11T21:47:08.892 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:47:08.867226+0000", 2024-03-11T21:47:08.892 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.7722000000000001e-05 2024-03-11T21:47:08.892 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-03-11T21:47:08.892 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-03-11T21:47:08.892 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-03-11T21:47:08.892 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:47:08.892 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:47:08.892 INFO:teuthology.orchestra.run.smithi092.stdout: "description": "osd_repop(client.35813.0:180224 11.e e477/446 11:70c7ff84:::benchmark_data_smithi092_186520_object41876:head v 477'11087, mlcod=477'11082)", 2024-03-11T21:47:08.892 INFO:teuthology.orchestra.run.smithi092.stdout: "initiated_at": "2024-03-11T21:47:08.866496+0000", 2024-03-11T21:47:08.892 INFO:teuthology.orchestra.run.smithi092.stdout: "age": 0.0073244299999999998, 2024-03-11T21:47:08.892 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.0078478279999999994, 2024-03-11T21:47:08.892 INFO:teuthology.orchestra.run.smithi092.stdout: "type_data": { 2024-03-11T21:47:08.893 INFO:teuthology.orchestra.run.smithi092.stdout: "flag_point": "started", 2024-03-11T21:47:08.893 INFO:teuthology.orchestra.run.smithi092.stdout: "events": [ 2024-03-11T21:47:08.893 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:47:08.893 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "initiated", 2024-03-11T21:47:08.893 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:47:08.866496+0000", 2024-03-11T21:47:08.893 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-03-11T21:47:08.893 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:47:08.893 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:47:08.893 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "header_read", 2024-03-11T21:47:08.893 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:47:08.866496+0000", 2024-03-11T21:47:08.893 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-03-11T21:47:08.893 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:47:08.893 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:47:08.893 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "throttled", 2024-03-11T21:47:08.894 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:47:08.866496+0000", 2024-03-11T21:47:08.894 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 3.6800000000000001e-07 2024-03-11T21:47:08.894 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:47:08.894 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:47:08.894 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "all_read", 2024-03-11T21:47:08.894 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:47:08.866504+0000", 2024-03-11T21:47:08.894 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 8.0320000000000003e-06 2024-03-11T21:47:08.894 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:47:08.894 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:47:08.894 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "dispatched", 2024-03-11T21:47:08.894 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:47:08.866509+0000", 2024-03-11T21:47:08.894 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 5.0030000000000002e-06 2024-03-11T21:47:08.894 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:47:08.894 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:47:08.894 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "queued_for_pg", 2024-03-11T21:47:08.895 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:47:08.866513+0000", 2024-03-11T21:47:08.895 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 4.1740000000000002e-06 2024-03-11T21:47:08.895 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:47:08.895 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:47:08.895 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-03-11T21:47:08.895 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:47:08.870657+0000", 2024-03-11T21:47:08.895 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.0041440280000000001 2024-03-11T21:47:08.895 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:47:08.895 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:47:08.895 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "started", 2024-03-11T21:47:08.895 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:47:08.870679+0000", 2024-03-11T21:47:08.895 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 2.1310000000000001e-05 2024-03-11T21:47:08.895 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-03-11T21:47:08.896 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-03-11T21:47:08.896 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-03-11T21:47:08.896 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:47:08.896 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:47:08.896 INFO:teuthology.orchestra.run.smithi092.stdout: "description": "osd_repop(client.35813.0:180242 11.9 e477/446 11:9b6f2a76:::benchmark_data_smithi092_186520_object41894:head v 477'11112, mlcod=477'11110)", 2024-03-11T21:47:08.896 INFO:teuthology.orchestra.run.smithi092.stdout: "initiated_at": "2024-03-11T21:47:08.867274+0000", 2024-03-11T21:47:08.896 INFO:teuthology.orchestra.run.smithi092.stdout: "age": 0.006546283, 2024-03-11T21:47:08.896 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.0071339719999999997, 2024-03-11T21:47:08.896 INFO:teuthology.orchestra.run.smithi092.stdout: "type_data": { 2024-03-11T21:47:08.896 INFO:teuthology.orchestra.run.smithi092.stdout: "flag_point": "started", 2024-03-11T21:47:08.896 INFO:teuthology.orchestra.run.smithi092.stdout: "events": [ 2024-03-11T21:47:08.896 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:47:08.897 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "initiated", 2024-03-11T21:47:08.897 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:47:08.867274+0000", 2024-03-11T21:47:08.897 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-03-11T21:47:08.897 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:47:08.897 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:47:08.897 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "header_read", 2024-03-11T21:47:08.897 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:47:08.867274+0000", 2024-03-11T21:47:08.897 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-03-11T21:47:08.897 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:47:08.897 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:47:08.897 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "throttled", 2024-03-11T21:47:08.897 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:47:08.867275+0000", 2024-03-11T21:47:08.897 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.1289999999999999e-06 2024-03-11T21:47:08.898 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:47:08.898 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:47:08.898 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "all_read", 2024-03-11T21:47:08.898 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:47:08.867284+0000", 2024-03-11T21:47:08.898 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 8.8540000000000003e-06 2024-03-11T21:47:08.898 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:47:08.898 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:47:08.898 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "dispatched", 2024-03-11T21:47:08.898 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:47:08.867291+0000", 2024-03-11T21:47:08.898 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 7.2479999999999999e-06 2024-03-11T21:47:08.898 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:47:08.898 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:47:08.898 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "queued_for_pg", 2024-03-11T21:47:08.898 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:47:08.867297+0000", 2024-03-11T21:47:08.899 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 5.8449999999999999e-06 2024-03-11T21:47:08.899 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:47:08.899 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:47:08.899 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-03-11T21:47:08.899 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:47:08.867407+0000", 2024-03-11T21:47:08.899 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.000109473 2024-03-11T21:47:08.899 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:47:08.899 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:47:08.899 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "started", 2024-03-11T21:47:08.899 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:47:08.867423+0000", 2024-03-11T21:47:08.899 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.6169000000000001e-05 2024-03-11T21:47:08.899 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-03-11T21:47:08.899 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-03-11T21:47:08.900 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-03-11T21:47:08.900 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:47:08.900 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:47:08.900 INFO:teuthology.orchestra.run.smithi092.stdout: "description": "osd_repop(client.35813.0:180244 11.1 e477/446 11:852da5c7:::benchmark_data_smithi092_186520_object41896:head v 477'11295, mlcod=477'11291)", 2024-03-11T21:47:08.900 INFO:teuthology.orchestra.run.smithi092.stdout: "initiated_at": "2024-03-11T21:47:08.867696+0000", 2024-03-11T21:47:08.900 INFO:teuthology.orchestra.run.smithi092.stdout: "age": 0.0061241730000000001, 2024-03-11T21:47:08.900 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.0067701209999999996, 2024-03-11T21:47:08.900 INFO:teuthology.orchestra.run.smithi092.stdout: "type_data": { 2024-03-11T21:47:08.900 INFO:teuthology.orchestra.run.smithi092.stdout: "flag_point": "started", 2024-03-11T21:47:08.900 INFO:teuthology.orchestra.run.smithi092.stdout: "events": [ 2024-03-11T21:47:08.900 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:47:08.900 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "initiated", 2024-03-11T21:47:08.900 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:47:08.867696+0000", 2024-03-11T21:47:08.900 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-03-11T21:47:08.901 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:47:08.901 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:47:08.901 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "header_read", 2024-03-11T21:47:08.901 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:47:08.867696+0000", 2024-03-11T21:47:08.901 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-03-11T21:47:08.901 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:47:08.901 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:47:08.901 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "throttled", 2024-03-11T21:47:08.901 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:47:08.867697+0000", 2024-03-11T21:47:08.901 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 9.1200000000000001e-07 2024-03-11T21:47:08.901 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:47:08.901 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:47:08.901 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "all_read", 2024-03-11T21:47:08.901 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:47:08.867705+0000", 2024-03-11T21:47:08.902 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 7.5290000000000003e-06 2024-03-11T21:47:08.902 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:47:08.902 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:47:08.902 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "dispatched", 2024-03-11T21:47:08.902 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:47:08.867712+0000", 2024-03-11T21:47:08.902 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 7.2599999999999999e-06 2024-03-11T21:47:08.902 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:47:08.902 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:47:08.902 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "queued_for_pg", 2024-03-11T21:47:08.902 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:47:08.867717+0000", 2024-03-11T21:47:08.902 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 5.6289999999999998e-06 2024-03-11T21:47:08.902 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:47:08.902 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:47:08.903 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-03-11T21:47:08.903 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:47:08.874104+0000", 2024-03-11T21:47:08.903 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.0063870159999999997 2024-03-11T21:47:08.903 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:47:08.903 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:47:08.903 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "started", 2024-03-11T21:47:08.903 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:47:08.874126+0000", 2024-03-11T21:47:08.903 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 2.1109000000000001e-05 2024-03-11T21:47:08.903 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-03-11T21:47:08.903 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-03-11T21:47:08.903 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-03-11T21:47:08.903 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:47:08.903 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:47:08.903 INFO:teuthology.orchestra.run.smithi092.stdout: "description": "osd_repop(client.35813.0:180243 11.5 e477/474)", 2024-03-11T21:47:08.904 INFO:teuthology.orchestra.run.smithi092.stdout: "initiated_at": "2024-03-11T21:47:08.870008+0000", 2024-03-11T21:47:08.904 INFO:teuthology.orchestra.run.smithi092.stdout: "age": 0.003812466, 2024-03-11T21:47:08.904 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.0045124229999999998, 2024-03-11T21:47:08.904 INFO:teuthology.orchestra.run.smithi092.stdout: "type_data": { 2024-03-11T21:47:08.904 INFO:teuthology.orchestra.run.smithi092.stdout: "flag_point": "reached pg", 2024-03-11T21:47:08.904 INFO:teuthology.orchestra.run.smithi092.stdout: "events": [ 2024-03-11T21:47:08.904 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:47:08.904 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "initiated", 2024-03-11T21:47:08.904 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:47:08.870008+0000", 2024-03-11T21:47:08.904 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-03-11T21:47:08.904 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:47:08.904 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:47:08.904 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "header_read", 2024-03-11T21:47:08.905 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:47:08.870008+0000", 2024-03-11T21:47:08.905 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-03-11T21:47:08.905 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:47:08.905 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:47:08.905 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "throttled", 2024-03-11T21:47:08.905 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:47:08.870009+0000", 2024-03-11T21:47:08.905 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.4699999999999999e-06 2024-03-11T21:47:08.905 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:47:08.905 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:47:08.905 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "all_read", 2024-03-11T21:47:08.905 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:47:08.870017+0000", 2024-03-11T21:47:08.906 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 7.7379999999999994e-06 2024-03-11T21:47:08.906 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:47:08.906 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:47:08.906 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "dispatched", 2024-03-11T21:47:08.906 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:47:08.870030+0000", 2024-03-11T21:47:08.906 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.3182e-05 2024-03-11T21:47:08.906 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:47:08.906 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:47:08.906 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "queued_for_pg", 2024-03-11T21:47:08.906 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:47:08.870038+0000", 2024-03-11T21:47:08.906 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 8.0879999999999995e-06 2024-03-11T21:47:08.906 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:47:08.906 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:47:08.907 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-03-11T21:47:08.907 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:47:08.871053+0000", 2024-03-11T21:47:08.907 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.001015029 2024-03-11T21:47:08.907 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-03-11T21:47:08.907 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-03-11T21:47:08.907 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-03-11T21:47:08.907 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:47:08.907 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:47:08.907 INFO:teuthology.orchestra.run.smithi092.stdout: "description": "osd_repop(client.35813.0:180233 11.e e477/446)", 2024-03-11T21:47:08.907 INFO:teuthology.orchestra.run.smithi092.stdout: "initiated_at": "2024-03-11T21:47:08.870454+0000", 2024-03-11T21:47:08.907 INFO:teuthology.orchestra.run.smithi092.stdout: "age": 0.0033660270000000002, 2024-03-11T21:47:08.907 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.0041150320000000002, 2024-03-11T21:47:08.907 INFO:teuthology.orchestra.run.smithi092.stdout: "type_data": { 2024-03-11T21:47:08.907 INFO:teuthology.orchestra.run.smithi092.stdout: "flag_point": "queued for pg", 2024-03-11T21:47:08.908 INFO:teuthology.orchestra.run.smithi092.stdout: "events": [ 2024-03-11T21:47:08.908 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:47:08.908 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "initiated", 2024-03-11T21:47:08.908 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:47:08.870454+0000", 2024-03-11T21:47:08.908 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-03-11T21:47:08.908 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:47:08.908 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:47:08.908 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "header_read", 2024-03-11T21:47:08.908 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:47:08.870454+0000", 2024-03-11T21:47:08.908 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-03-11T21:47:08.908 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:47:08.908 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:47:08.908 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "throttled", 2024-03-11T21:47:08.909 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:47:08.870455+0000", 2024-03-11T21:47:08.909 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.06e-06 2024-03-11T21:47:08.909 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:47:08.909 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:47:08.909 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "all_read", 2024-03-11T21:47:08.909 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:47:08.870463+0000", 2024-03-11T21:47:08.909 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 7.379e-06 2024-03-11T21:47:08.909 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:47:08.909 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:47:08.909 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "dispatched", 2024-03-11T21:47:08.909 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:47:08.870472+0000", 2024-03-11T21:47:08.909 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 9.4099999999999997e-06 2024-03-11T21:47:08.909 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:47:08.910 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:47:08.910 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "queued_for_pg", 2024-03-11T21:47:08.910 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:47:08.870479+0000", 2024-03-11T21:47:08.910 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 6.4239999999999998e-06 2024-03-11T21:47:08.910 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-03-11T21:47:08.910 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-03-11T21:47:08.910 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-03-11T21:47:08.910 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:47:08.910 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:47:08.910 INFO:teuthology.orchestra.run.smithi092.stdout: "description": "osd_repop(client.35813.0:180246 11.7 e477/474 11:edfebd2b:::benchmark_data_smithi092_186520_object41898:head v 477'11153, mlcod=477'11151)", 2024-03-11T21:47:08.910 INFO:teuthology.orchestra.run.smithi092.stdout: "initiated_at": "2024-03-11T21:47:08.871233+0000", 2024-03-11T21:47:08.910 INFO:teuthology.orchestra.run.smithi092.stdout: "age": 0.002587167, 2024-03-11T21:47:08.910 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.0033828869999999998, 2024-03-11T21:47:08.911 INFO:teuthology.orchestra.run.smithi092.stdout: "type_data": { 2024-03-11T21:47:08.911 INFO:teuthology.orchestra.run.smithi092.stdout: "flag_point": "started", 2024-03-11T21:47:08.911 INFO:teuthology.orchestra.run.smithi092.stdout: "events": [ 2024-03-11T21:47:08.911 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:47:08.911 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "initiated", 2024-03-11T21:47:08.911 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:47:08.871233+0000", 2024-03-11T21:47:08.911 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-03-11T21:47:08.911 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:47:08.911 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:47:08.911 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "header_read", 2024-03-11T21:47:08.911 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:47:08.871233+0000", 2024-03-11T21:47:08.911 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-03-11T21:47:08.911 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:47:08.911 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:47:08.912 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "throttled", 2024-03-11T21:47:08.912 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:47:08.871234+0000", 2024-03-11T21:47:08.912 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.2869999999999999e-06 2024-03-11T21:47:08.912 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:47:08.912 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:47:08.912 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "all_read", 2024-03-11T21:47:08.912 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:47:08.871241+0000", 2024-03-11T21:47:08.912 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 6.7519999999999999e-06 2024-03-11T21:47:08.912 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:47:08.912 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:47:08.912 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "dispatched", 2024-03-11T21:47:08.912 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:47:08.871251+0000", 2024-03-11T21:47:08.912 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 9.9879999999999999e-06 2024-03-11T21:47:08.912 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:47:08.912 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:47:08.913 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "queued_for_pg", 2024-03-11T21:47:08.913 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:47:08.871258+0000", 2024-03-11T21:47:08.913 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 7.2590000000000003e-06 2024-03-11T21:47:08.913 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:47:08.913 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:47:08.913 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-03-11T21:47:08.913 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:47:08.871316+0000", 2024-03-11T21:47:08.913 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 5.7929000000000001e-05 2024-03-11T21:47:08.913 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:47:08.913 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:47:08.913 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "started", 2024-03-11T21:47:08.913 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:47:08.871340+0000", 2024-03-11T21:47:08.913 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 2.4082000000000001e-05 2024-03-11T21:47:08.913 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-03-11T21:47:08.913 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-03-11T21:47:08.913 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-03-11T21:47:08.914 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:47:08.914 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:47:08.914 INFO:teuthology.orchestra.run.smithi092.stdout: "description": "osd_repop(client.35813.0:180245 11.b e477/474 11:dde88ad2:::benchmark_data_smithi092_186520_object41897:head v 477'11196, mlcod=477'11194)", 2024-03-11T21:47:08.914 INFO:teuthology.orchestra.run.smithi092.stdout: "initiated_at": "2024-03-11T21:47:08.871410+0000", 2024-03-11T21:47:08.914 INFO:teuthology.orchestra.run.smithi092.stdout: "age": 0.002410662, 2024-03-11T21:47:08.914 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.0032632490000000002, 2024-03-11T21:47:08.914 INFO:teuthology.orchestra.run.smithi092.stdout: "type_data": { 2024-03-11T21:47:08.914 INFO:teuthology.orchestra.run.smithi092.stdout: "flag_point": "started", 2024-03-11T21:47:08.914 INFO:teuthology.orchestra.run.smithi092.stdout: "events": [ 2024-03-11T21:47:08.914 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:47:08.914 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "initiated", 2024-03-11T21:47:08.914 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:47:08.871410+0000", 2024-03-11T21:47:08.914 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-03-11T21:47:08.915 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:47:08.915 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:47:08.915 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "header_read", 2024-03-11T21:47:08.915 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:47:08.871410+0000", 2024-03-11T21:47:08.915 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-03-11T21:47:08.915 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:47:08.915 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:47:08.915 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "throttled", 2024-03-11T21:47:08.915 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:47:08.871411+0000", 2024-03-11T21:47:08.915 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.088e-06 2024-03-11T21:47:08.915 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:47:08.915 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:47:08.915 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "all_read", 2024-03-11T21:47:08.915 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:47:08.871417+0000", 2024-03-11T21:47:08.915 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 6.1990000000000003e-06 2024-03-11T21:47:08.915 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:47:08.916 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:47:08.916 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "dispatched", 2024-03-11T21:47:08.916 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:47:08.871426+0000", 2024-03-11T21:47:08.916 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 8.9700000000000005e-06 2024-03-11T21:47:08.916 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:47:08.916 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:47:08.916 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "queued_for_pg", 2024-03-11T21:47:08.916 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:47:08.871433+0000", 2024-03-11T21:47:08.916 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 6.63e-06 2024-03-11T21:47:08.916 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:47:08.916 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:47:08.916 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-03-11T21:47:08.916 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:47:08.873933+0000", 2024-03-11T21:47:08.916 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.0025009749999999999 2024-03-11T21:47:08.916 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:47:08.916 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:47:08.917 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "started", 2024-03-11T21:47:08.917 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:47:08.873957+0000", 2024-03-11T21:47:08.917 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 2.3686e-05 2024-03-11T21:47:08.917 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-03-11T21:47:08.917 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-03-11T21:47:08.917 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-03-11T21:47:08.917 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:47:08.917 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:47:08.917 INFO:teuthology.orchestra.run.smithi092.stdout: "description": "pg_backfill(progress 11.5 e 477/477 lb 11:a4ff7de8:::benchmark_data_smithi092_186520_object107844:head)", 2024-03-11T21:47:08.917 INFO:teuthology.orchestra.run.smithi092.stdout: "initiated_at": "2024-03-11T21:47:08.871743+0000", 2024-03-11T21:47:08.917 INFO:teuthology.orchestra.run.smithi092.stdout: "age": 0.002076858, 2024-03-11T21:47:08.917 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.002985709, 2024-03-11T21:47:08.917 INFO:teuthology.orchestra.run.smithi092.stdout: "type_data": { 2024-03-11T21:47:08.917 INFO:teuthology.orchestra.run.smithi092.stdout: "flag_point": "queued for pg", 2024-03-11T21:47:08.918 INFO:teuthology.orchestra.run.smithi092.stdout: "events": [ 2024-03-11T21:47:08.918 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:47:08.918 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "initiated", 2024-03-11T21:47:08.918 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:47:08.871743+0000", 2024-03-11T21:47:08.918 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-03-11T21:47:08.918 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:47:08.918 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:47:08.918 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "header_read", 2024-03-11T21:47:08.918 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:47:08.871743+0000", 2024-03-11T21:47:08.918 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-03-11T21:47:08.918 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:47:08.918 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:47:08.918 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "throttled", 2024-03-11T21:47:08.918 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:47:08.871745+0000", 2024-03-11T21:47:08.918 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.1340000000000001e-06 2024-03-11T21:47:08.919 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:47:08.919 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:47:08.919 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "all_read", 2024-03-11T21:47:08.919 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:47:08.871753+0000", 2024-03-11T21:47:08.919 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 8.0809999999999994e-06 2024-03-11T21:47:08.919 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:47:08.919 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:47:08.919 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "dispatched", 2024-03-11T21:47:08.919 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:47:08.871763+0000", 2024-03-11T21:47:08.919 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.0003000000000001e-05 2024-03-11T21:47:08.919 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:47:08.919 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:47:08.919 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "queued_for_pg", 2024-03-11T21:47:08.919 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:47:08.871770+0000", 2024-03-11T21:47:08.919 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 7.5739999999999998e-06 2024-03-11T21:47:08.919 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-03-11T21:47:08.919 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-03-11T21:47:08.920 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-03-11T21:47:08.920 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-03-11T21:47:08.920 INFO:teuthology.orchestra.run.smithi092.stdout: ], 2024-03-11T21:47:08.920 INFO:teuthology.orchestra.run.smithi092.stdout: "num_ops": 15 2024-03-11T21:47:08.920 INFO:teuthology.orchestra.run.smithi092.stdout:} 2024-03-11T21:47:08.958 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:47:08.959 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:47:08.959 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:47:09.059 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:47:09.096 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:47:09.097 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:47:09.098 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:47:09.199 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:47:09.230 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:47:09.230 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:47:09.231 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:47:09.332 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:47:09.367 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:47:09.368 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:47:09.368 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:47:09.376 INFO:teuthology.orchestra.run.smithi092.stderr:Inferring config /var/lib/ceph/00db0016-dfed-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T21:47:09.469 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:47:09.506 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:47:09.506 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:47:09.506 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:47:09.565 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:47:09 smithi092 ceph-mon[29572]: pgmap v1016: 148 pgs: 3 active+remapped+backfill_toofull, 2 active+remapped+backfilling, 143 active+clean; 6.3 GiB data, 21 GiB used, 426 GiB / 447 GiB avail; 1023 B/s rd, 0 B/s wr, 1.28k op/s; 30763/312669 objects misplaced (9.839%); 14 MiB/s, 222 objects/s recovering 2024-03-11T21:47:09.565 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:47:09 smithi092 ceph-mon[29572]: from='mgr.14607 ' entity='mgr.y' 2024-03-11T21:47:09.566 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:47:09 smithi092 ceph-mon[29572]: from='mgr.14607 v1:172.21.15.92:0/1702717597' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T21:47:09.566 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:47:09 smithi092 ceph-mon[37563]: pgmap v1016: 148 pgs: 3 active+remapped+backfill_toofull, 2 active+remapped+backfilling, 143 active+clean; 6.3 GiB data, 21 GiB used, 426 GiB / 447 GiB avail; 1023 B/s rd, 0 B/s wr, 1.28k op/s; 30763/312669 objects misplaced (9.839%); 14 MiB/s, 222 objects/s recovering 2024-03-11T21:47:09.566 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:47:09 smithi092 ceph-mon[37563]: from='mgr.14607 ' entity='mgr.y' 2024-03-11T21:47:09.566 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:47:09 smithi092 ceph-mon[37563]: from='mgr.14607 v1:172.21.15.92:0/1702717597' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T21:47:09.584 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:47:09 smithi097 ceph-mon[33317]: pgmap v1016: 148 pgs: 3 active+remapped+backfill_toofull, 2 active+remapped+backfilling, 143 active+clean; 6.3 GiB data, 21 GiB used, 426 GiB / 447 GiB avail; 1023 B/s rd, 0 B/s wr, 1.28k op/s; 30763/312669 objects misplaced (9.839%); 14 MiB/s, 222 objects/s recovering 2024-03-11T21:47:09.584 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:47:09 smithi097 ceph-mon[33317]: from='mgr.14607 ' entity='mgr.y' 2024-03-11T21:47:09.584 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:47:09 smithi097 ceph-mon[33317]: from='mgr.14607 v1:172.21.15.92:0/1702717597' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T21:47:09.608 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:47:09.646 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:47:09.647 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:47:09.647 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:47:09.748 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:47:09.778 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:47:09.779 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:47:09.779 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:47:09.880 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:47:09.912 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:47:09.913 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:47:09.913 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:47:10.014 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:47:10.044 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:47:10.047 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:47:10.047 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:47:10.148 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:47:10.179 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:47:10.180 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:47:10.180 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:47:10.281 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:47:10.320 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:47:10.320 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:47:10.321 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:47:10.421 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:47:10.463 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:47:10.464 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:47:10.464 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:47:10.565 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:47:10.598 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:47:10.599 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:47:10.599 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:47:10.700 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:47:10.737 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:47:10.738 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:47:10.738 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:47:10.803 DEBUG:teuthology.orchestra.run.smithi092:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid 00db0016-dfed-11ee-95c8-87774f69a715 -- ceph daemon osd.0 dump_blocked_ops 2024-03-11T21:47:10.839 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:47:10.871 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:47:10.871 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:47:10.872 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:47:10.972 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:47:11.002 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:47:11.003 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:47:11.003 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:47:11.104 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:47:11.152 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:47:11.152 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:47:11.153 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:47:11.253 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:47:11.292 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:47:11.293 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:47:11.293 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:47:11.394 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:47:11.426 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:47:11.427 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:47:11.427 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:47:11.528 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:47:11.565 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:47:11.566 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:47:11.566 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:47:11.584 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:47:11 smithi097 ceph-mon[33317]: pgmap v1017: 148 pgs: 3 active+remapped+backfill_toofull, 2 active+remapped+backfilling, 143 active+clean; 6.0 GiB data, 20 GiB used, 427 GiB / 447 GiB avail; 1023 B/s rd, 0 B/s wr, 1.40k op/s; 27482/293640 objects misplaced (9.359%); 17 MiB/s, 265 objects/s recovering 2024-03-11T21:47:11.615 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:47:11 smithi092 ceph-mon[29572]: pgmap v1017: 148 pgs: 3 active+remapped+backfill_toofull, 2 active+remapped+backfilling, 143 active+clean; 6.0 GiB data, 20 GiB used, 427 GiB / 447 GiB avail; 1023 B/s rd, 0 B/s wr, 1.40k op/s; 27482/293640 objects misplaced (9.359%); 17 MiB/s, 265 objects/s recovering 2024-03-11T21:47:11.615 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:47:11 smithi092 ceph-mon[37563]: pgmap v1017: 148 pgs: 3 active+remapped+backfill_toofull, 2 active+remapped+backfilling, 143 active+clean; 6.0 GiB data, 20 GiB used, 427 GiB / 447 GiB avail; 1023 B/s rd, 0 B/s wr, 1.40k op/s; 27482/293640 objects misplaced (9.359%); 17 MiB/s, 265 objects/s recovering 2024-03-11T21:47:11.667 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:47:11.700 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:47:11.701 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:47:11.701 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:47:11.802 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:47:11.840 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:47:11.841 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:47:11.842 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:47:11.942 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:47:11.976 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:47:11.977 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:47:11.977 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:47:12.078 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:47:12.123 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:47:12.124 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:47:12.124 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:47:12.225 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:47:12.266 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:47:12.268 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:47:12.269 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:47:12.368 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:47:12 smithi092 ceph-mon[29572]: 6.4 scrub starts 2024-03-11T21:47:12.368 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:47:12 smithi092 ceph-mon[29572]: 6.4 scrub ok 2024-03-11T21:47:12.369 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:47:12 smithi092 ceph-mon[37563]: 6.4 scrub starts 2024-03-11T21:47:12.369 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:47:12 smithi092 ceph-mon[37563]: 6.4 scrub ok 2024-03-11T21:47:12.369 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:47:12.408 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:47:12.408 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:47:12.408 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:47:12.509 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:47:12.546 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:47:12.547 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:47:12.548 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:47:12.584 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:47:12 smithi097 ceph-mon[33317]: 6.4 scrub starts 2024-03-11T21:47:12.584 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:47:12 smithi097 ceph-mon[33317]: 6.4 scrub ok 2024-03-11T21:47:12.649 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:47:12.691 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:47:12.691 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:47:12.692 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:47:12.793 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:47:12.837 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:47:12.837 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:47:12.838 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:47:12.939 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:47:12.976 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:47:12.976 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:47:12.977 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:47:13.078 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:47:13.133 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:47:13.134 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:47:13.134 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:47:13.235 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:47:13.280 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:47:13.280 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:47:13.281 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:47:13.381 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:47:13.426 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:47:13.427 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:47:13.427 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:47:13.528 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:47:13.559 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:47:13.559 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:47:13.560 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:47:13.584 INFO:journalctl@ceph.iscsi.iscsi.a.smithi097.stdout:Mar 11 21:47:13 smithi097 ceph-00db0016-dfed-11ee-95c8-87774f69a715-iscsi-iscsi-a[60943]: debug there is no tcmu-runner data available 2024-03-11T21:47:13.584 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:47:13 smithi097 ceph-mon[33317]: pgmap v1018: 148 pgs: 3 active+remapped+backfill_toofull, 2 active+remapped+backfilling, 143 active+clean; 5.9 GiB data, 20 GiB used, 427 GiB / 447 GiB avail; 914 B/s rd, 0 B/s wr, 1.25k op/s; 27482/292251 objects misplaced (9.404%); 15 MiB/s, 237 objects/s recovering 2024-03-11T21:47:13.585 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:47:13 smithi097 ceph-mon[33317]: 4.2 scrub starts 2024-03-11T21:47:13.585 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:47:13 smithi097 ceph-mon[33317]: 4.2 scrub ok 2024-03-11T21:47:13.618 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:47:13 smithi092 ceph-mon[29572]: pgmap v1018: 148 pgs: 3 active+remapped+backfill_toofull, 2 active+remapped+backfilling, 143 active+clean; 5.9 GiB data, 20 GiB used, 427 GiB / 447 GiB avail; 914 B/s rd, 0 B/s wr, 1.25k op/s; 27482/292251 objects misplaced (9.404%); 15 MiB/s, 237 objects/s recovering 2024-03-11T21:47:13.618 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:47:13 smithi092 ceph-mon[29572]: 4.2 scrub starts 2024-03-11T21:47:13.619 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:47:13 smithi092 ceph-mon[29572]: 4.2 scrub ok 2024-03-11T21:47:13.619 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:47:13 smithi092 ceph-mon[37563]: pgmap v1018: 148 pgs: 3 active+remapped+backfill_toofull, 2 active+remapped+backfilling, 143 active+clean; 5.9 GiB data, 20 GiB used, 427 GiB / 447 GiB avail; 914 B/s rd, 0 B/s wr, 1.25k op/s; 27482/292251 objects misplaced (9.404%); 15 MiB/s, 237 objects/s recovering 2024-03-11T21:47:13.619 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:47:13 smithi092 ceph-mon[37563]: 4.2 scrub starts 2024-03-11T21:47:13.619 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:47:13 smithi092 ceph-mon[37563]: 4.2 scrub ok 2024-03-11T21:47:13.660 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:47:13.705 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:47:13.705 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:47:13.705 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:47:13.806 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:47:13.845 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:47:13.846 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:47:13.846 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:47:13.947 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:47:13.986 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:47:13.987 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:47:13.987 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:47:14.088 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:47:14.120 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:47:14.121 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:47:14.121 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:47:14.222 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:47:14.253 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:47:14.253 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:47:14.254 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:47:14.355 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:47:14.368 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:47:14 smithi092 ceph-mon[29572]: 5.0 scrub starts 2024-03-11T21:47:14.368 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:47:14 smithi092 ceph-mon[29572]: 5.0 scrub ok 2024-03-11T21:47:14.368 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:47:14 smithi092 ceph-mon[29572]: from='client.24469 v1:172.21.15.97:0/404208310' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:47:14.369 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:47:14 smithi092 ceph-mon[37563]: 5.0 scrub starts 2024-03-11T21:47:14.369 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:47:14 smithi092 ceph-mon[37563]: 5.0 scrub ok 2024-03-11T21:47:14.369 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:47:14 smithi092 ceph-mon[37563]: from='client.24469 v1:172.21.15.97:0/404208310' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:47:14.390 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:47:14.391 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:47:14.392 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:47:14.492 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:47:14.533 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:47:14.534 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:47:14.535 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:47:14.584 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:47:14 smithi097 ceph-mon[33317]: 5.0 scrub starts 2024-03-11T21:47:14.584 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:47:14 smithi097 ceph-mon[33317]: 5.0 scrub ok 2024-03-11T21:47:14.584 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:47:14 smithi097 ceph-mon[33317]: from='client.24469 v1:172.21.15.97:0/404208310' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:47:14.635 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:47:14.676 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:47:14.677 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:47:14.677 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:47:14.778 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:47:14.813 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:47:14.814 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:47:14.814 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:47:14.915 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:47:14.957 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:47:14.958 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:47:14.958 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:47:15.059 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:47:15.083 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:47:15.083 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:47:15.084 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:47:15.184 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:47:15.217 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:47:15.218 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:47:15.218 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:47:15.319 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:47:15.351 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:47:15.352 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:47:15.353 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:47:15.453 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:47:15.489 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:47:15.490 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:47:15.490 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:47:15.584 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:47:15 smithi097 ceph-mon[33317]: pgmap v1019: 148 pgs: 3 active+remapped+backfill_toofull, 2 active+remapped+backfilling, 143 active+clean; 5.7 GiB data, 20 GiB used, 427 GiB / 447 GiB avail; 852 B/s rd, 0 B/s wr, 1.53k op/s; 25187/279162 objects misplaced (9.022%); 21 MiB/s, 329 objects/s recovering 2024-03-11T21:47:15.591 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:47:15.618 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:47:15 smithi092 ceph-mon[29572]: pgmap v1019: 148 pgs: 3 active+remapped+backfill_toofull, 2 active+remapped+backfilling, 143 active+clean; 5.7 GiB data, 20 GiB used, 427 GiB / 447 GiB avail; 852 B/s rd, 0 B/s wr, 1.53k op/s; 25187/279162 objects misplaced (9.022%); 21 MiB/s, 329 objects/s recovering 2024-03-11T21:47:15.619 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:47:15 smithi092 ceph-mon[37563]: pgmap v1019: 148 pgs: 3 active+remapped+backfill_toofull, 2 active+remapped+backfilling, 143 active+clean; 5.7 GiB data, 20 GiB used, 427 GiB / 447 GiB avail; 852 B/s rd, 0 B/s wr, 1.53k op/s; 25187/279162 objects misplaced (9.022%); 21 MiB/s, 329 objects/s recovering 2024-03-11T21:47:15.631 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:47:15.632 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:47:15.632 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:47:15.733 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:47:15.772 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:47:15.772 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:47:15.772 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:47:15.873 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:47:15.897 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:47:15.898 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:47:15.898 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:47:15.999 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:47:16.024 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:47:16.025 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:47:16.025 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:47:16.126 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:47:16.152 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:47:16.152 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:47:16.153 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:47:16.253 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:47:16.292 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:47:16.293 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:47:16.294 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:47:16.394 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:47:16.495 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:47:16.495 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:47:16.496 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:47:16.596 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:47:16.632 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:47:16.633 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:47:16.633 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:47:16.734 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:47:16.766 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:47:16.767 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:47:16.768 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:47:16.868 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:47:16.900 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:47:16.901 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:47:16.902 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:47:17.002 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:47:17.034 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:47:17.035 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:47:17.035 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:47:17.136 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:47:17.180 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:47:17.181 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:47:17.181 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:47:17.282 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:47:17.314 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:47:17.314 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:47:17.315 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:47:17.416 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:47:17.446 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:47:17.447 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:47:17.448 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:47:17.548 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:47:17.583 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:47:17.583 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:47:17.583 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:47:17.584 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:47:17 smithi097 ceph-mon[33317]: pgmap v1020: 148 pgs: 3 active+remapped+backfill_toofull, 2 active+remapped+backfilling, 143 active+clean; 5.5 GiB data, 19 GiB used, 428 GiB / 447 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.36k op/s; 24266/271431 objects misplaced (8.940%); 14 MiB/s, 219 objects/s recovering 2024-03-11T21:47:17.586 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:47:17 smithi092 ceph-mon[29572]: pgmap v1020: 148 pgs: 3 active+remapped+backfill_toofull, 2 active+remapped+backfilling, 143 active+clean; 5.5 GiB data, 19 GiB used, 428 GiB / 447 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.36k op/s; 24266/271431 objects misplaced (8.940%); 14 MiB/s, 219 objects/s recovering 2024-03-11T21:47:17.586 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:47:17 smithi092 ceph-mon[37563]: pgmap v1020: 148 pgs: 3 active+remapped+backfill_toofull, 2 active+remapped+backfilling, 143 active+clean; 5.5 GiB data, 19 GiB used, 428 GiB / 447 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.36k op/s; 24266/271431 objects misplaced (8.940%); 14 MiB/s, 219 objects/s recovering 2024-03-11T21:47:17.684 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:47:17.731 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:47:17.732 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:47:17.732 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:47:17.833 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:47:17.868 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:47:17.869 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:47:17.869 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:47:17.970 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:47:18.012 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:47:18.013 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:47:18.013 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:47:18.114 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:47:18.119 INFO:journalctl@ceph.mgr.y.smithi092.stdout:Mar 11 21:47:17 smithi092 ceph-00db0016-dfed-11ee-95c8-87774f69a715-mgr-y[29801]: ::ffff:172.21.15.97 - - [11/Mar/2024:21:47:17] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-11T21:47:18.119 INFO:teuthology.orchestra.run.smithi092.stderr:Inferring config /var/lib/ceph/00db0016-dfed-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T21:47:18.144 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:47:18.144 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:47:18.145 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:47:18.245 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:47:18.298 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:47:18.298 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:47:18.298 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:47:18.399 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:47:18.439 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:47:18 smithi092 ceph-mon[29572]: 4.14 scrub starts 2024-03-11T21:47:18.439 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:47:18 smithi092 ceph-mon[29572]: 4.14 scrub ok 2024-03-11T21:47:18.439 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:47:18 smithi092 ceph-mon[37563]: 4.14 scrub starts 2024-03-11T21:47:18.439 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:47:18 smithi092 ceph-mon[37563]: 4.14 scrub ok 2024-03-11T21:47:18.458 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:47:18.458 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:47:18.458 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:47:18.559 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:47:18.584 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:47:18 smithi097 ceph-mon[33317]: 4.14 scrub starts 2024-03-11T21:47:18.584 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:47:18 smithi097 ceph-mon[33317]: 4.14 scrub ok 2024-03-11T21:47:18.590 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:47:18.591 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:47:18.591 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:47:18.692 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:47:18.723 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:47:18.724 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:47:18.724 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:47:18.825 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:47:18.856 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:47:18.856 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:47:18.857 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:47:18.958 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:47:18.992 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:47:18.992 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:47:18.992 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:47:19.093 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:47:19.137 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:47:19.138 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:47:19.138 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:47:19.239 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:47:19.281 INFO:teuthology.orchestra.run.smithi092.stderr:set osd.1 primary-affinity to 0.556418 (8364652) 2024-03-11T21:47:19.281 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:47:19.283 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:47:19.283 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:47:19.300 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:47:19 smithi092 ceph-mon[29572]: pgmap v1021: 148 pgs: 3 active+remapped+backfill_toofull, 2 active+remapped+backfilling, 143 active+clean; 5.5 GiB data, 19 GiB used, 428 GiB / 447 GiB avail; 853 B/s rd, 0 B/s wr, 1.15k op/s; 24266/271431 objects misplaced (8.940%); 14 MiB/s, 219 objects/s recovering 2024-03-11T21:47:19.301 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:47:19 smithi092 ceph-mon[29572]: from='client.? v1:172.21.15.92:0/1008209856' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 1, "weight": 0.5564177019329419}]: dispatch 2024-03-11T21:47:19.301 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:47:19 smithi092 ceph-mon[29572]: from='client.35954 ' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 1, "weight": 0.5564177019329419}]: dispatch 2024-03-11T21:47:19.301 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:47:19 smithi092 ceph-mon[37563]: pgmap v1021: 148 pgs: 3 active+remapped+backfill_toofull, 2 active+remapped+backfilling, 143 active+clean; 5.5 GiB data, 19 GiB used, 428 GiB / 447 GiB avail; 853 B/s rd, 0 B/s wr, 1.15k op/s; 24266/271431 objects misplaced (8.940%); 14 MiB/s, 219 objects/s recovering 2024-03-11T21:47:19.301 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:47:19 smithi092 ceph-mon[37563]: from='client.? v1:172.21.15.92:0/1008209856' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 1, "weight": 0.5564177019329419}]: dispatch 2024-03-11T21:47:19.301 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:47:19 smithi092 ceph-mon[37563]: from='client.35954 ' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 1, "weight": 0.5564177019329419}]: dispatch 2024-03-11T21:47:19.385 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:47:19.413 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:47:19.415 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:47:19.415 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:47:19.516 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:47:19.541 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:47:19.542 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:47:19.542 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:47:19.584 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:47:19 smithi097 ceph-mon[33317]: pgmap v1021: 148 pgs: 3 active+remapped+backfill_toofull, 2 active+remapped+backfilling, 143 active+clean; 5.5 GiB data, 19 GiB used, 428 GiB / 447 GiB avail; 853 B/s rd, 0 B/s wr, 1.15k op/s; 24266/271431 objects misplaced (8.940%); 14 MiB/s, 219 objects/s recovering 2024-03-11T21:47:19.584 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:47:19 smithi097 ceph-mon[33317]: from='client.? v1:172.21.15.92:0/1008209856' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 1, "weight": 0.5564177019329419}]: dispatch 2024-03-11T21:47:19.584 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:47:19 smithi097 ceph-mon[33317]: from='client.35954 ' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 1, "weight": 0.5564177019329419}]: dispatch 2024-03-11T21:47:19.644 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:47:19.669 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:47:19.670 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:47:19.670 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:47:19.771 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:47:19.797 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:47:19.798 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:47:19.798 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:47:19.899 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:47:19.927 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:47:19.927 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:47:19.927 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:47:20.028 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:47:20.053 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:47:20.054 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:47:20.054 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:47:20.155 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:47:20.180 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:47:20.181 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:47:20.181 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:47:20.270 INFO:teuthology.orchestra.run.smithi092.stderr:nodeep-scrub is unset 2024-03-11T21:47:20.282 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:47:20.313 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:47:20.314 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:47:20.315 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:47:20.415 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:47:20.446 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:47:20.446 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:47:20.447 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:47:20.547 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:47:20.555 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:47:20 smithi092 ceph-mon[29572]: 4.10 scrub starts 2024-03-11T21:47:20.555 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:47:20 smithi092 ceph-mon[29572]: 4.10 scrub ok 2024-03-11T21:47:20.555 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:47:20 smithi092 ceph-mon[29572]: from='client.35954 ' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 1, "weight": 0.5564177019329419}]': finished 2024-03-11T21:47:20.555 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:47:20 smithi092 ceph-mon[29572]: osdmap e478: 8 total, 8 up, 5 in 2024-03-11T21:47:20.555 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:47:20 smithi092 ceph-mon[29572]: from='client.? v1:172.21.15.92:0/1135771785' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-11T21:47:20.555 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:47:20 smithi092 ceph-mon[29572]: from='client.35960 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-11T21:47:20.555 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:47:20 smithi092 ceph-mon[29572]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-03-11T21:47:20.555 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:47:20 smithi092 ceph-mon[29572]: from='client.35960 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-03-11T21:47:20.555 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:47:20 smithi092 ceph-mon[29572]: osdmap e479: 8 total, 8 up, 5 in 2024-03-11T21:47:20.556 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:47:20 smithi092 ceph-mon[37563]: 4.10 scrub starts 2024-03-11T21:47:20.556 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:47:20 smithi092 ceph-mon[37563]: 4.10 scrub ok 2024-03-11T21:47:20.556 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:47:20 smithi092 ceph-mon[37563]: from='client.35954 ' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 1, "weight": 0.5564177019329419}]': finished 2024-03-11T21:47:20.556 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:47:20 smithi092 ceph-mon[37563]: osdmap e478: 8 total, 8 up, 5 in 2024-03-11T21:47:20.556 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:47:20 smithi092 ceph-mon[37563]: from='client.? v1:172.21.15.92:0/1135771785' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-11T21:47:20.556 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:47:20 smithi092 ceph-mon[37563]: from='client.35960 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-11T21:47:20.556 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:47:20 smithi092 ceph-mon[37563]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-03-11T21:47:20.556 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:47:20 smithi092 ceph-mon[37563]: from='client.35960 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-03-11T21:47:20.556 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:47:20 smithi092 ceph-mon[37563]: osdmap e479: 8 total, 8 up, 5 in 2024-03-11T21:47:20.573 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:47:20.574 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:47:20.574 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:47:20.584 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:47:20 smithi097 ceph-mon[33317]: 4.10 scrub starts 2024-03-11T21:47:20.584 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:47:20 smithi097 ceph-mon[33317]: 4.10 scrub ok 2024-03-11T21:47:20.584 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:47:20 smithi097 ceph-mon[33317]: from='client.35954 ' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 1, "weight": 0.5564177019329419}]': finished 2024-03-11T21:47:20.584 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:47:20 smithi097 ceph-mon[33317]: osdmap e478: 8 total, 8 up, 5 in 2024-03-11T21:47:20.584 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:47:20 smithi097 ceph-mon[33317]: from='client.? v1:172.21.15.92:0/1135771785' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-11T21:47:20.584 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:47:20 smithi097 ceph-mon[33317]: from='client.35960 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-11T21:47:20.585 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:47:20 smithi097 ceph-mon[33317]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-03-11T21:47:20.585 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:47:20 smithi097 ceph-mon[33317]: from='client.35960 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-03-11T21:47:20.585 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:47:20 smithi097 ceph-mon[33317]: osdmap e479: 8 total, 8 up, 5 in 2024-03-11T21:47:20.675 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:47:20.702 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:47:20.702 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:47:20.703 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:47:20.804 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:47:20.830 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:47:20.831 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:47:20.831 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:47:20.932 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:47:20.957 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:47:20.958 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:47:20.958 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:47:21.059 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:47:21.085 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:47:21.085 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:47:21.086 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:47:21.187 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:47:21.212 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:47:21.213 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:47:21.213 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:47:21.314 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:47:21.355 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:47:21.356 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:47:21.356 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:47:21.457 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:47:21.489 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:47:21.490 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:47:21.490 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:47:21.584 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:47:21 smithi097 ceph-mon[33317]: pgmap v1023: 148 pgs: 3 active+remapped+backfill_toofull, 2 active+remapped+backfilling, 143 active+clean; 5.1 GiB data, 18 GiB used, 429 GiB / 447 GiB avail; 1023 B/s rd, 0 B/s wr, 1.35k op/s; 20935/253263 objects misplaced (8.266%); 18 MiB/s, 294 objects/s recovering 2024-03-11T21:47:21.584 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:47:21 smithi097 ceph-mon[33317]: osdmap e480: 8 total, 8 up, 5 in 2024-03-11T21:47:21.591 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:47:21.618 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:47:21 smithi092 ceph-mon[29572]: pgmap v1023: 148 pgs: 3 active+remapped+backfill_toofull, 2 active+remapped+backfilling, 143 active+clean; 5.1 GiB data, 18 GiB used, 429 GiB / 447 GiB avail; 1023 B/s rd, 0 B/s wr, 1.35k op/s; 20935/253263 objects misplaced (8.266%); 18 MiB/s, 294 objects/s recovering 2024-03-11T21:47:21.618 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:47:21 smithi092 ceph-mon[29572]: osdmap e480: 8 total, 8 up, 5 in 2024-03-11T21:47:21.619 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:47:21 smithi092 ceph-mon[37563]: pgmap v1023: 148 pgs: 3 active+remapped+backfill_toofull, 2 active+remapped+backfilling, 143 active+clean; 5.1 GiB data, 18 GiB used, 429 GiB / 447 GiB avail; 1023 B/s rd, 0 B/s wr, 1.35k op/s; 20935/253263 objects misplaced (8.266%); 18 MiB/s, 294 objects/s recovering 2024-03-11T21:47:21.619 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:47:21 smithi092 ceph-mon[37563]: osdmap e480: 8 total, 8 up, 5 in 2024-03-11T21:47:21.624 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:47:21.625 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:47:21.625 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:47:21.726 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:47:21.766 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:47:21.767 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:47:21.768 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:47:21.868 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:47:21.901 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:47:21.902 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:47:21.902 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:47:22.003 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:47:22.035 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:47:22.035 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:47:22.036 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:47:22.136 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:47:22.167 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:47:22.168 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:47:22.168 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:47:22.269 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:47:22.305 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:47:22.305 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:47:22.305 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:47:22.406 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:47:22.437 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:47:22.438 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:47:22.438 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:47:22.539 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:47:22.572 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:47:22 smithi092 ceph-mon[29572]: 11.b continuing backfill to osd.6 from (469'8991,477'12095] MIN to 477'12095 2024-03-11T21:47:22.573 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:47:22 smithi092 ceph-mon[37563]: 11.b continuing backfill to osd.6 from (469'8991,477'12095] MIN to 477'12095 2024-03-11T21:47:22.577 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:47:22.578 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:47:22.578 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:47:22.584 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:47:22 smithi097 ceph-mon[33317]: 11.b continuing backfill to osd.6 from (469'8991,477'12095] MIN to 477'12095 2024-03-11T21:47:22.679 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:47:22.723 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:47:22.725 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:47:22.725 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:47:22.826 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:47:22.862 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:47:22.863 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:47:22.863 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:47:22.964 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:47:23.000 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:47:23.000 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:47:23.001 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:47:23.082 INFO:teuthology.orchestra.run.smithi092.stdout:{ 2024-03-11T21:47:23.082 INFO:teuthology.orchestra.run.smithi092.stdout: "ops": [], 2024-03-11T21:47:23.083 INFO:teuthology.orchestra.run.smithi092.stdout: "complaint_time": 30, 2024-03-11T21:47:23.083 INFO:teuthology.orchestra.run.smithi092.stdout: "num_blocked_ops": 0 2024-03-11T21:47:23.083 INFO:teuthology.orchestra.run.smithi092.stdout:} 2024-03-11T21:47:23.101 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:47:23.151 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:47:23.152 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:47:23.152 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:47:23.253 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:47:23.299 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:47:23.300 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:47:23.301 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:47:23.401 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:47:23.440 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:47:23.441 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:47:23.441 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:47:23.485 INFO:journalctl@ceph.iscsi.iscsi.a.smithi097.stdout:Mar 11 21:47:23 smithi097 ceph-00db0016-dfed-11ee-95c8-87774f69a715-iscsi-iscsi-a[60943]: debug there is no tcmu-runner data available 2024-03-11T21:47:23.486 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:47:23 smithi097 ceph-mon[33317]: pgmap v1026: 148 pgs: 4 peering, 3 active+remapped+backfill_toofull, 2 active+remapped+backfilling, 139 active+clean; 5.1 GiB data, 18 GiB used, 429 GiB / 447 GiB avail; 853 B/s rd, 0 B/s wr, 1.08k op/s; 20935/252072 objects misplaced (8.305%); 17 MiB/s, 273 objects/s recovering 2024-03-11T21:47:23.486 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:47:23 smithi097 ceph-mon[33317]: 2.9 scrub starts 2024-03-11T21:47:23.486 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:47:23 smithi097 ceph-mon[33317]: 2.9 scrub ok 2024-03-11T21:47:23.486 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:47:23 smithi097 ceph-mon[33317]: Health check failed: Reduced data availability: 3 pgs inactive, 4 pgs peering (PG_AVAILABILITY) 2024-03-11T21:47:23.486 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:47:23 smithi097 ceph-mon[33317]: from='mgr.14607 ' entity='mgr.y' 2024-03-11T21:47:23.486 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:47:23 smithi097 ceph-mon[33317]: from='mgr.14607 v1:172.21.15.92:0/1702717597' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T21:47:23.542 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:47:23.581 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:47:23.582 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:47:23.582 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:47:23.618 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:47:23 smithi092 ceph-mon[29572]: pgmap v1026: 148 pgs: 4 peering, 3 active+remapped+backfill_toofull, 2 active+remapped+backfilling, 139 active+clean; 5.1 GiB data, 18 GiB used, 429 GiB / 447 GiB avail; 853 B/s rd, 0 B/s wr, 1.08k op/s; 20935/252072 objects misplaced (8.305%); 17 MiB/s, 273 objects/s recovering 2024-03-11T21:47:23.619 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:47:23 smithi092 ceph-mon[29572]: 2.9 scrub starts 2024-03-11T21:47:23.619 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:47:23 smithi092 ceph-mon[29572]: 2.9 scrub ok 2024-03-11T21:47:23.619 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:47:23 smithi092 ceph-mon[29572]: Health check failed: Reduced data availability: 3 pgs inactive, 4 pgs peering (PG_AVAILABILITY) 2024-03-11T21:47:23.619 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:47:23 smithi092 ceph-mon[29572]: from='mgr.14607 ' entity='mgr.y' 2024-03-11T21:47:23.619 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:47:23 smithi092 ceph-mon[29572]: from='mgr.14607 v1:172.21.15.92:0/1702717597' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T21:47:23.619 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:47:23 smithi092 ceph-mon[37563]: pgmap v1026: 148 pgs: 4 peering, 3 active+remapped+backfill_toofull, 2 active+remapped+backfilling, 139 active+clean; 5.1 GiB data, 18 GiB used, 429 GiB / 447 GiB avail; 853 B/s rd, 0 B/s wr, 1.08k op/s; 20935/252072 objects misplaced (8.305%); 17 MiB/s, 273 objects/s recovering 2024-03-11T21:47:23.619 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:47:23 smithi092 ceph-mon[37563]: 2.9 scrub starts 2024-03-11T21:47:23.619 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:47:23 smithi092 ceph-mon[37563]: 2.9 scrub ok 2024-03-11T21:47:23.619 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:47:23 smithi092 ceph-mon[37563]: Health check failed: Reduced data availability: 3 pgs inactive, 4 pgs peering (PG_AVAILABILITY) 2024-03-11T21:47:23.620 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:47:23 smithi092 ceph-mon[37563]: from='mgr.14607 ' entity='mgr.y' 2024-03-11T21:47:23.620 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:47:23 smithi092 ceph-mon[37563]: from='mgr.14607 v1:172.21.15.92:0/1702717597' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T21:47:23.683 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:47:23.728 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:47:23.729 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:47:23.729 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:47:23.830 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:47:23.860 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:47:23.860 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:47:23.861 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:47:23.883 INFO:teuthology.orchestra.run.smithi092.stdout:osd.0: {} 2024-03-11T21:47:23.929 INFO:teuthology.orchestra.run.smithi092.stdout:osd.1: {} 2024-03-11T21:47:23.961 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:47:23.970 INFO:teuthology.orchestra.run.smithi092.stdout:osd.2: {} 2024-03-11T21:47:24.006 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:47:24.007 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:47:24.008 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:47:24.009 INFO:teuthology.orchestra.run.smithi092.stdout:osd.3: {} 2024-03-11T21:47:24.047 INFO:teuthology.orchestra.run.smithi092.stdout:osd.4: {} 2024-03-11T21:47:24.086 INFO:teuthology.orchestra.run.smithi092.stdout:osd.5: {} 2024-03-11T21:47:24.109 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:47:24.129 INFO:teuthology.orchestra.run.smithi092.stdout:osd.6: {} 2024-03-11T21:47:24.149 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:47:24.150 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:47:24.150 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:47:24.171 INFO:teuthology.orchestra.run.smithi092.stdout:osd.7: {} 2024-03-11T21:47:24.184 INFO:teuthology.orchestra.run.smithi092.stderr:osd.0: osd_enable_op_tracker = '' 2024-03-11T21:47:24.184 INFO:teuthology.orchestra.run.smithi092.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-11T21:47:24.184 INFO:teuthology.orchestra.run.smithi092.stderr:osd.2: osd_enable_op_tracker = '' 2024-03-11T21:47:24.185 INFO:teuthology.orchestra.run.smithi092.stderr:osd.3: osd_enable_op_tracker = '' 2024-03-11T21:47:24.185 INFO:teuthology.orchestra.run.smithi092.stderr:osd.4: osd_enable_op_tracker = '' 2024-03-11T21:47:24.185 INFO:teuthology.orchestra.run.smithi092.stderr:osd.5: osd_enable_op_tracker = '' 2024-03-11T21:47:24.185 INFO:teuthology.orchestra.run.smithi092.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-11T21:47:24.185 INFO:teuthology.orchestra.run.smithi092.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-11T21:47:24.252 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:47:24.289 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:47:24.290 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:47:24.290 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:47:24.365 DEBUG:teuthology.orchestra.run.smithi092:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid 00db0016-dfed-11ee-95c8-87774f69a715 -- ceph daemon osd.0 dump_historic_ops 2024-03-11T21:47:24.391 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:47:24.425 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:47:24.425 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:47:24.425 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:47:24.444 DEBUG:teuthology.orchestra.run.smithi092:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid 00db0016-dfed-11ee-95c8-87774f69a715 -- ceph osd set noscrub 2024-03-11T21:47:24.498 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:47:24 smithi092 ceph-mon[29572]: 11.c scrub starts 2024-03-11T21:47:24.498 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:47:24 smithi092 ceph-mon[29572]: from='client.24469 v1:172.21.15.97:0/404208310' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:47:24.498 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:47:24 smithi092 ceph-mon[29572]: from='client.? v1:172.21.15.92:0/3377394720' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-11T21:47:24.499 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:47:24 smithi092 ceph-mon[37563]: 11.c scrub starts 2024-03-11T21:47:24.499 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:47:24 smithi092 ceph-mon[37563]: from='client.24469 v1:172.21.15.97:0/404208310' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:47:24.499 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:47:24 smithi092 ceph-mon[37563]: from='client.? v1:172.21.15.92:0/3377394720' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-11T21:47:24.527 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:47:24.558 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:47:24.559 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:47:24.559 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:47:24.584 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:47:24 smithi097 ceph-mon[33317]: 11.c scrub starts 2024-03-11T21:47:24.585 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:47:24 smithi097 ceph-mon[33317]: from='client.24469 v1:172.21.15.97:0/404208310' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:47:24.585 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:47:24 smithi097 ceph-mon[33317]: from='client.? v1:172.21.15.92:0/3377394720' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-11T21:47:24.660 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:47:24.706 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:47:24.707 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:47:24.707 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:47:24.808 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:47:24.852 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:47:24.852 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:47:24.853 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:47:24.954 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:47:24.987 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:47:24.987 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:47:24.988 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:47:25.025 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 6, 7] out_osds: [3, 4, 5] dead_osds: [] live_osds: [0, 1, 2, 4, 5, 6, 7, 3] 2024-03-11T21:47:25.026 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-03-11T21:47:25.026 INFO:tasks.thrashosds.thrasher:fixing pg num pool unique_pool_4 2024-03-11T21:47:25.026 DEBUG:teuthology.orchestra.run.smithi092:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid 00db0016-dfed-11ee-95c8-87774f69a715 -- ceph pg dump --format=json 2024-03-11T21:47:25.088 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:47:25.112 INFO:teuthology.orchestra.run.smithi092.stderr:Inferring config /var/lib/ceph/00db0016-dfed-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T21:47:25.119 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:47:25.120 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:47:25.120 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:47:25.166 INFO:teuthology.orchestra.run.smithi092.stderr:Inferring config /var/lib/ceph/00db0016-dfed-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T21:47:25.221 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:47:25.256 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:47:25.256 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:47:25.257 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:47:25.357 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:47:25.390 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:47:25.390 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:47:25.391 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:47:25.492 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:47:25.526 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:47:25.527 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:47:25.527 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:47:25.539 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:47:25 smithi092 ceph-mon[29572]: pgmap v1027: 148 pgs: 4 peering, 3 active+remapped+backfill_toofull, 2 active+remapped+backfilling, 139 active+clean; 4.9 GiB data, 18 GiB used, 429 GiB / 447 GiB avail; 0 B/s wr, 1.55k op/s; 17368/243423 objects misplaced (7.135%); 49 MiB/s, 789 objects/s recovering 2024-03-11T21:47:25.539 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:47:25 smithi092 ceph-mon[29572]: purged_snaps scrub starts 2024-03-11T21:47:25.539 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:47:25 smithi092 ceph-mon[29572]: purged_snaps scrub ok 2024-03-11T21:47:25.539 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:47:25 smithi092 ceph-mon[29572]: 11.c scrub ok 2024-03-11T21:47:25.540 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:47:25 smithi092 ceph-mon[37563]: pgmap v1027: 148 pgs: 4 peering, 3 active+remapped+backfill_toofull, 2 active+remapped+backfilling, 139 active+clean; 4.9 GiB data, 18 GiB used, 429 GiB / 447 GiB avail; 0 B/s wr, 1.55k op/s; 17368/243423 objects misplaced (7.135%); 49 MiB/s, 789 objects/s recovering 2024-03-11T21:47:25.540 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:47:25 smithi092 ceph-mon[37563]: purged_snaps scrub starts 2024-03-11T21:47:25.540 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:47:25 smithi092 ceph-mon[37563]: purged_snaps scrub ok 2024-03-11T21:47:25.540 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:47:25 smithi092 ceph-mon[37563]: 11.c scrub ok 2024-03-11T21:47:25.584 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:47:25 smithi097 ceph-mon[33317]: pgmap v1027: 148 pgs: 4 peering, 3 active+remapped+backfill_toofull, 2 active+remapped+backfilling, 139 active+clean; 4.9 GiB data, 18 GiB used, 429 GiB / 447 GiB avail; 0 B/s wr, 1.55k op/s; 17368/243423 objects misplaced (7.135%); 49 MiB/s, 789 objects/s recovering 2024-03-11T21:47:25.585 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:47:25 smithi097 ceph-mon[33317]: purged_snaps scrub starts 2024-03-11T21:47:25.585 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:47:25 smithi097 ceph-mon[33317]: purged_snaps scrub ok 2024-03-11T21:47:25.585 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:47:25 smithi097 ceph-mon[33317]: 11.c scrub ok 2024-03-11T21:47:25.628 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:47:25.664 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:47:25.665 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:47:25.665 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:47:25.766 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:47:25.804 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:47:25.805 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:47:25.805 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:47:25.905 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:47:25.941 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:47:25.942 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:47:25.942 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:47:26.043 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:47:26.079 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:47:26.079 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:47:26.080 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:47:26.181 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:47:26.217 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:47:26.218 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:47:26.218 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:47:26.319 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:47:26.354 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:47:26.355 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:47:26.355 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:47:26.456 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:47:26.497 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:47:26.497 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:47:26.498 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:47:26.598 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:47:26.618 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:47:26 smithi092 ceph-mon[29572]: osdmap e481: 8 total, 8 up, 5 in 2024-03-11T21:47:26.619 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:47:26 smithi092 ceph-mon[37563]: osdmap e481: 8 total, 8 up, 5 in 2024-03-11T21:47:26.632 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:47:26.633 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:47:26.633 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:47:26.734 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:47:26.768 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:47:26.768 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:47:26.769 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:47:26.834 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:47:26 smithi097 ceph-mon[33317]: osdmap e481: 8 total, 8 up, 5 in 2024-03-11T21:47:26.870 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:47:26.908 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:47:26.909 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:47:26.910 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:47:26.999 DEBUG:teuthology.orchestra.run.smithi092:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid 00db0016-dfed-11ee-95c8-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-03-11T21:47:27.010 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:47:27.044 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:47:27.045 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:47:27.046 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:47:27.146 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:47:27.179 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:47:27.180 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:47:27.180 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:47:27.281 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:47:27.324 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:47:27.324 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:47:27.325 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:47:27.426 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:47:27.459 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:47:27.460 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:47:27.461 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:47:27.562 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:47:27.584 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:47:27 smithi097 ceph-mon[33317]: pgmap v1029: 148 pgs: 1 active+clean+scrubbing, 2 active+remapped+backfill_toofull, 3 active+remapped+backfilling, 142 active+clean; 4.9 GiB data, 18 GiB used, 429 GiB / 447 GiB avail; 0 B/s wr, 782 op/s; 16356/239166 objects misplaced (6.839%); 37 MiB/s, 585 objects/s recovering 2024-03-11T21:47:27.584 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:47:27 smithi097 ceph-mon[33317]: Health check update: Low space hindering backfill (add storage if this doesn't resolve itself): 2 pgs backfill_toofull (PG_BACKFILL_FULL) 2024-03-11T21:47:27.584 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:47:27 smithi097 ceph-mon[33317]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 3 pgs inactive, 4 pgs peering) 2024-03-11T21:47:27.616 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:47:27.617 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:47:27.618 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:47:27.618 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:47:27 smithi092 ceph-mon[37563]: pgmap v1029: 148 pgs: 1 active+clean+scrubbing, 2 active+remapped+backfill_toofull, 3 active+remapped+backfilling, 142 active+clean; 4.9 GiB data, 18 GiB used, 429 GiB / 447 GiB avail; 0 B/s wr, 782 op/s; 16356/239166 objects misplaced (6.839%); 37 MiB/s, 585 objects/s recovering 2024-03-11T21:47:27.618 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:47:27 smithi092 ceph-mon[37563]: Health check update: Low space hindering backfill (add storage if this doesn't resolve itself): 2 pgs backfill_toofull (PG_BACKFILL_FULL) 2024-03-11T21:47:27.618 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:47:27 smithi092 ceph-mon[37563]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 3 pgs inactive, 4 pgs peering) 2024-03-11T21:47:27.619 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:47:27 smithi092 ceph-mon[29572]: pgmap v1029: 148 pgs: 1 active+clean+scrubbing, 2 active+remapped+backfill_toofull, 3 active+remapped+backfilling, 142 active+clean; 4.9 GiB data, 18 GiB used, 429 GiB / 447 GiB avail; 0 B/s wr, 782 op/s; 16356/239166 objects misplaced (6.839%); 37 MiB/s, 585 objects/s recovering 2024-03-11T21:47:27.619 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:47:27 smithi092 ceph-mon[29572]: Health check update: Low space hindering backfill (add storage if this doesn't resolve itself): 2 pgs backfill_toofull (PG_BACKFILL_FULL) 2024-03-11T21:47:27.619 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:47:27 smithi092 ceph-mon[29572]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 3 pgs inactive, 4 pgs peering) 2024-03-11T21:47:27.685 INFO:teuthology.orchestra.run.smithi092.stderr:Inferring config /var/lib/ceph/00db0016-dfed-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T21:47:27.719 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:47:27.756 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:47:27.757 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:47:27.757 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:47:27.858 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:47:27.895 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:47:27.897 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:47:27.898 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:47:27.998 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:47:28.021 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:47:28.022 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:47:28.022 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:47:28.119 INFO:journalctl@ceph.mgr.y.smithi092.stdout:Mar 11 21:47:27 smithi092 ceph-00db0016-dfed-11ee-95c8-87774f69a715-mgr-y[29801]: ::ffff:172.21.15.97 - - [11/Mar/2024:21:47:27] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-11T21:47:28.123 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:47:28.158 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:47:28.158 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:47:28.159 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:47:28.259 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:47:28.299 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:47:28.299 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:47:28.299 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:47:28.400 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:47:28.426 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:47:28.426 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:47:28.426 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:47:28.527 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:47:28.563 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:47:28.564 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:47:28.564 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:47:28.618 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:47:28 smithi092 ceph-mon[29572]: pgmap v1030: 148 pgs: 1 active+clean+scrubbing, 2 active+remapped+backfill_toofull, 3 active+remapped+backfilling, 142 active+clean; 4.9 GiB data, 18 GiB used, 429 GiB / 447 GiB avail; 0 B/s wr, 595 op/s; 16356/239166 objects misplaced (6.839%); 30 MiB/s, 487 objects/s recovering 2024-03-11T21:47:28.619 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:47:28 smithi092 ceph-mon[37563]: pgmap v1030: 148 pgs: 1 active+clean+scrubbing, 2 active+remapped+backfill_toofull, 3 active+remapped+backfilling, 142 active+clean; 4.9 GiB data, 18 GiB used, 429 GiB / 447 GiB avail; 0 B/s wr, 595 op/s; 16356/239166 objects misplaced (6.839%); 30 MiB/s, 487 objects/s recovering 2024-03-11T21:47:28.665 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:47:28.692 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:47:28.693 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:47:28.693 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:47:28.794 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:47:28.822 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:47:28.822 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:47:28.823 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:47:28.834 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:47:28 smithi097 ceph-mon[33317]: pgmap v1030: 148 pgs: 1 active+clean+scrubbing, 2 active+remapped+backfill_toofull, 3 active+remapped+backfilling, 142 active+clean; 4.9 GiB data, 18 GiB used, 429 GiB / 447 GiB avail; 0 B/s wr, 595 op/s; 16356/239166 objects misplaced (6.839%); 30 MiB/s, 487 objects/s recovering 2024-03-11T21:47:28.924 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:47:28.951 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:47:28.951 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:47:28.952 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:47:29.053 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:47:29.082 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:47:29.082 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:47:29.083 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:47:29.184 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:47:29.210 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:47:29.211 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:47:29.211 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:47:29.312 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:47:29.340 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:47:29.341 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:47:29.341 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:47:29.442 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:47:29.476 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:47:29.477 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:47:29.477 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:47:29.578 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:47:29.616 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:47:29.617 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:47:29.617 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:47:29.618 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:47:29 smithi092 ceph-mon[29572]: osdmap e482: 8 total, 8 up, 5 in 2024-03-11T21:47:29.618 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:47:29 smithi092 ceph-mon[37563]: osdmap e482: 8 total, 8 up, 5 in 2024-03-11T21:47:29.718 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:47:29.758 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:47:29.759 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:47:29.759 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:47:29.834 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:47:29 smithi097 ceph-mon[33317]: osdmap e482: 8 total, 8 up, 5 in 2024-03-11T21:47:29.860 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:47:29.892 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:47:29.893 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:47:29.894 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:47:29.994 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:47:30.028 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:47:30.028 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:47:30.028 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:47:30.129 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:47:30.166 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:47:30.167 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:47:30.168 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:47:30.268 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:47:30.300 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:47:30.301 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:47:30.301 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:47:30.402 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:47:30.433 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:47:30.434 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:47:30.434 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:47:30.535 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:47:30.561 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:47:30.561 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:47:30.562 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:47:30.618 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:47:30 smithi092 ceph-mon[29572]: osdmap e483: 8 total, 8 up, 5 in 2024-03-11T21:47:30.618 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:47:30 smithi092 ceph-mon[29572]: pgmap v1033: 148 pgs: 1 active+clean+remapped, 1 active+remapped+backfill_toofull, 3 active+remapped+backfilling, 143 active+clean; 4.5 GiB data, 17 GiB used, 430 GiB / 447 GiB avail; 853 B/s rd, 0 B/s wr, 1.15k op/s; 13644/222816 objects misplaced (6.123%); 24 MiB/s, 378 objects/s recovering 2024-03-11T21:47:30.619 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:47:30 smithi092 ceph-mon[37563]: osdmap e483: 8 total, 8 up, 5 in 2024-03-11T21:47:30.619 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:47:30 smithi092 ceph-mon[37563]: pgmap v1033: 148 pgs: 1 active+clean+remapped, 1 active+remapped+backfill_toofull, 3 active+remapped+backfilling, 143 active+clean; 4.5 GiB data, 17 GiB used, 430 GiB / 447 GiB avail; 853 B/s rd, 0 B/s wr, 1.15k op/s; 13644/222816 objects misplaced (6.123%); 24 MiB/s, 378 objects/s recovering 2024-03-11T21:47:30.662 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:47:30.688 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:47:30.689 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:47:30.689 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:47:30.790 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:47:30.815 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:47:30.816 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:47:30.817 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:47:30.834 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:47:30 smithi097 ceph-mon[33317]: osdmap e483: 8 total, 8 up, 5 in 2024-03-11T21:47:30.834 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:47:30 smithi097 ceph-mon[33317]: pgmap v1033: 148 pgs: 1 active+clean+remapped, 1 active+remapped+backfill_toofull, 3 active+remapped+backfilling, 143 active+clean; 4.5 GiB data, 17 GiB used, 430 GiB / 447 GiB avail; 853 B/s rd, 0 B/s wr, 1.15k op/s; 13644/222816 objects misplaced (6.123%); 24 MiB/s, 378 objects/s recovering 2024-03-11T21:47:30.918 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:47:30.944 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:47:30.945 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:47:30.945 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:47:31.046 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:47:31.071 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:47:31.072 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:47:31.072 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:47:31.173 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:47:31.199 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:47:31.199 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:47:31.200 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:47:31.300 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:47:31.327 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:47:31.327 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:47:31.328 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:47:31.429 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:47:31.452 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:47:31.452 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:47:31.453 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:47:31.554 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:47:31.590 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:47:31.590 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:47:31.591 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:47:31.691 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:47:31.733 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:47:31.734 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:47:31.734 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:47:31.834 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:47:31 smithi097 ceph-mon[33317]: osdmap e484: 8 total, 8 up, 5 in 2024-03-11T21:47:31.851 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:47:31.868 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:47:31 smithi092 ceph-mon[29572]: osdmap e484: 8 total, 8 up, 5 in 2024-03-11T21:47:31.869 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:47:31 smithi092 ceph-mon[37563]: osdmap e484: 8 total, 8 up, 5 in 2024-03-11T21:47:31.883 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:47:31.884 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:47:31.885 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:47:31.985 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:47:32.027 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:47:32.028 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:47:32.028 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:47:32.129 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:47:32.161 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:47:32.162 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:47:32.162 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:47:32.263 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:47:32.292 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:47:32.293 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:47:32.294 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:47:32.394 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:47:32.451 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:47:32.452 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:47:32.452 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:47:32.512 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:47:32 smithi092 ceph-mon[29572]: osdmap e485: 8 total, 8 up, 5 in 2024-03-11T21:47:32.512 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:47:32 smithi092 ceph-mon[29572]: purged_snaps scrub starts 2024-03-11T21:47:32.512 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:47:32 smithi092 ceph-mon[29572]: purged_snaps scrub ok 2024-03-11T21:47:32.512 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:47:32 smithi092 ceph-mon[29572]: pgmap v1036: 148 pgs: 1 active+clean+remapped, 1 active+remapped+backfill_toofull, 3 active+remapped+backfilling, 143 active+clean; 4.5 GiB data, 17 GiB used, 430 GiB / 447 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.53k op/s; 12498/220854 objects misplaced (5.659%); 43 MiB/s, 680 objects/s recovering 2024-03-11T21:47:32.513 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:47:32 smithi092 ceph-mon[37563]: osdmap e485: 8 total, 8 up, 5 in 2024-03-11T21:47:32.513 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:47:32 smithi092 ceph-mon[37563]: purged_snaps scrub starts 2024-03-11T21:47:32.513 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:47:32 smithi092 ceph-mon[37563]: purged_snaps scrub ok 2024-03-11T21:47:32.513 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:47:32 smithi092 ceph-mon[37563]: pgmap v1036: 148 pgs: 1 active+clean+remapped, 1 active+remapped+backfill_toofull, 3 active+remapped+backfilling, 143 active+clean; 4.5 GiB data, 17 GiB used, 430 GiB / 447 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.53k op/s; 12498/220854 objects misplaced (5.659%); 43 MiB/s, 680 objects/s recovering 2024-03-11T21:47:32.553 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:47:32.604 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:47:32.605 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:47:32.605 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:47:32.706 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:47:32.749 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:47:32.749 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:47:32.750 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:47:32.834 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:47:32 smithi097 ceph-mon[33317]: osdmap e485: 8 total, 8 up, 5 in 2024-03-11T21:47:32.835 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:47:32 smithi097 ceph-mon[33317]: purged_snaps scrub starts 2024-03-11T21:47:32.835 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:47:32 smithi097 ceph-mon[33317]: purged_snaps scrub ok 2024-03-11T21:47:32.835 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:47:32 smithi097 ceph-mon[33317]: pgmap v1036: 148 pgs: 1 active+clean+remapped, 1 active+remapped+backfill_toofull, 3 active+remapped+backfilling, 143 active+clean; 4.5 GiB data, 17 GiB used, 430 GiB / 447 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.53k op/s; 12498/220854 objects misplaced (5.659%); 43 MiB/s, 680 objects/s recovering 2024-03-11T21:47:32.850 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:47:32.893 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:47:32.894 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:47:32.894 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:47:32.976 INFO:teuthology.orchestra.run.smithi092.stderr:Inferring config /var/lib/ceph/00db0016-dfed-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T21:47:32.995 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:47:33.023 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:47:33.024 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:47:33.024 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:47:33.125 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:47:33.155 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:47:33.156 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:47:33.156 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:47:33.257 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:47:33.286 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:47:33.286 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:47:33.287 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:47:33.388 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:47:33.432 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:47:33.433 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:47:33.433 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:47:33.439 INFO:teuthology.orchestra.run.smithi092.stderr:noscrub is set 2024-03-11T21:47:33.534 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:47:33.574 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:47:33.575 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:47:33.576 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:47:33.584 INFO:journalctl@ceph.iscsi.iscsi.a.smithi097.stdout:Mar 11 21:47:33 smithi097 ceph-00db0016-dfed-11ee-95c8-87774f69a715-iscsi-iscsi-a[60943]: debug there is no tcmu-runner data available 2024-03-11T21:47:33.585 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:47:33 smithi097 ceph-mon[33317]: osdmap e486: 8 total, 8 up, 5 in 2024-03-11T21:47:33.585 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:47:33 smithi097 ceph-mon[33317]: from='client.24469 v1:172.21.15.97:0/404208310' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:47:33.585 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:47:33 smithi097 ceph-mon[33317]: from='client.? v1:172.21.15.92:0/2190702552' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-11T21:47:33.585 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:47:33 smithi097 ceph-mon[33317]: from='client.25966 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-11T21:47:33.677 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:47:33.711 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:47:33.712 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:47:33.712 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:47:33.813 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:47:33.845 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:47:33.845 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:47:33.846 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:47:33.869 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:47:33 smithi092 ceph-mon[29572]: osdmap e486: 8 total, 8 up, 5 in 2024-03-11T21:47:33.869 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:47:33 smithi092 ceph-mon[29572]: from='client.24469 v1:172.21.15.97:0/404208310' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:47:33.869 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:47:33 smithi092 ceph-mon[29572]: from='client.? v1:172.21.15.92:0/2190702552' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-11T21:47:33.869 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:47:33 smithi092 ceph-mon[29572]: from='client.25966 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-11T21:47:33.870 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:47:33 smithi092 ceph-mon[37563]: osdmap e486: 8 total, 8 up, 5 in 2024-03-11T21:47:33.870 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:47:33 smithi092 ceph-mon[37563]: from='client.24469 v1:172.21.15.97:0/404208310' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:47:33.870 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:47:33 smithi092 ceph-mon[37563]: from='client.? v1:172.21.15.92:0/2190702552' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-11T21:47:33.870 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:47:33 smithi092 ceph-mon[37563]: from='client.25966 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-11T21:47:33.947 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:47:33.979 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:47:33.980 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:47:33.980 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:47:34.081 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:47:34.112 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:47:34.112 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:47:34.113 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:47:34.211 INFO:teuthology.orchestra.run.smithi092.stdout:ERROR: (22) Invalid argument 2024-03-11T21:47:34.211 INFO:teuthology.orchestra.run.smithi092.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-11T21:47:34.213 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:47:34.249 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:47:34.250 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:47:34.250 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:47:34.351 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:47:34.390 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:47:34.391 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:47:34.392 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:47:34.492 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:47:34.532 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:47:34.533 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:47:34.534 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:47:34.618 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:47:34 smithi092 ceph-mon[29572]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-03-11T21:47:34.619 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:47:34 smithi092 ceph-mon[29572]: from='client.25966 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-03-11T21:47:34.619 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:47:34 smithi092 ceph-mon[29572]: osdmap e487: 8 total, 8 up, 5 in 2024-03-11T21:47:34.619 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:47:34 smithi092 ceph-mon[29572]: pgmap v1039: 148 pgs: 1 active+remapped+backfill_toofull, 2 active+remapped+backfilling, 145 active+clean; 4.3 GiB data, 17 GiB used, 430 GiB / 447 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 862 op/s; 11692/212478 objects misplaced (5.503%); 22 MiB/s, 355 objects/s recovering 2024-03-11T21:47:34.619 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:47:34 smithi092 ceph-mon[29572]: Health check update: Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull (PG_BACKFILL_FULL) 2024-03-11T21:47:34.619 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:47:34 smithi092 ceph-mon[37563]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-03-11T21:47:34.619 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:47:34 smithi092 ceph-mon[37563]: from='client.25966 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-03-11T21:47:34.619 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:47:34 smithi092 ceph-mon[37563]: osdmap e487: 8 total, 8 up, 5 in 2024-03-11T21:47:34.619 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:47:34 smithi092 ceph-mon[37563]: pgmap v1039: 148 pgs: 1 active+remapped+backfill_toofull, 2 active+remapped+backfilling, 145 active+clean; 4.3 GiB data, 17 GiB used, 430 GiB / 447 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 862 op/s; 11692/212478 objects misplaced (5.503%); 22 MiB/s, 355 objects/s recovering 2024-03-11T21:47:34.619 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:47:34 smithi092 ceph-mon[37563]: Health check update: Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull (PG_BACKFILL_FULL) 2024-03-11T21:47:34.635 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:47:34.668 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:47:34.669 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:47:34.670 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:47:34.770 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:47:34.800 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:47:34.801 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:47:34.801 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:47:34.835 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:47:34 smithi097 ceph-mon[33317]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-03-11T21:47:34.835 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:47:34 smithi097 ceph-mon[33317]: from='client.25966 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-03-11T21:47:34.835 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:47:34 smithi097 ceph-mon[33317]: osdmap e487: 8 total, 8 up, 5 in 2024-03-11T21:47:34.835 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:47:34 smithi097 ceph-mon[33317]: pgmap v1039: 148 pgs: 1 active+remapped+backfill_toofull, 2 active+remapped+backfilling, 145 active+clean; 4.3 GiB data, 17 GiB used, 430 GiB / 447 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 862 op/s; 11692/212478 objects misplaced (5.503%); 22 MiB/s, 355 objects/s recovering 2024-03-11T21:47:34.835 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:47:34 smithi097 ceph-mon[33317]: Health check update: Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull (PG_BACKFILL_FULL) 2024-03-11T21:47:34.902 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:47:34.936 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:47:34.936 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:47:34.936 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:47:35.037 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:47:35.077 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:47:35.078 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:47:35.078 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:47:35.179 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:47:35.223 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:47:35.224 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:47:35.224 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:47:35.325 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:47:35.362 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:47:35.362 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:47:35.363 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:47:35.464 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:47:35.496 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:47:35.497 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:47:35.497 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:47:35.598 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:47:35.638 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:47:35.639 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:47:35.639 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:47:35.740 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:47:35.772 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:47:35.773 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:47:35.773 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:47:35.874 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:47:35.906 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:47:35.907 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:47:35.907 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:47:36.008 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:47:36.041 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:47:36.042 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:47:36.042 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:47:36.143 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:47:36.181 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:47:36.181 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:47:36.182 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:47:36.283 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:47:36.314 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:47:36.315 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:47:36.315 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:47:36.416 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:47:36.452 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:47:36.453 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:47:36.453 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:47:36.554 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:47:36.588 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:47:36.588 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:47:36.589 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:47:36.690 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:47:36.729 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:47:36.729 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:47:36.730 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:47:36.830 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:47:36.834 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:47:36 smithi097 ceph-mon[33317]: purged_snaps scrub starts 2024-03-11T21:47:36.834 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:47:36 smithi097 ceph-mon[33317]: purged_snaps scrub ok 2024-03-11T21:47:36.834 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:47:36 smithi097 ceph-mon[33317]: osdmap e488: 8 total, 8 up, 5 in 2024-03-11T21:47:36.834 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:47:36 smithi097 ceph-mon[33317]: pgmap v1041: 148 pgs: 1 active+remapped+backfill_toofull, 2 active+remapped+backfilling, 145 active+clean; 4.2 GiB data, 16 GiB used, 431 GiB / 447 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.04k op/s; 9061/208080 objects misplaced (4.355%); 43 MiB/s, 689 objects/s recovering 2024-03-11T21:47:36.861 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:47:36.862 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:47:36.862 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:47:36.869 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:47:36 smithi092 ceph-mon[29572]: purged_snaps scrub starts 2024-03-11T21:47:36.869 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:47:36 smithi092 ceph-mon[29572]: purged_snaps scrub ok 2024-03-11T21:47:36.869 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:47:36 smithi092 ceph-mon[29572]: osdmap e488: 8 total, 8 up, 5 in 2024-03-11T21:47:36.869 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:47:36 smithi092 ceph-mon[29572]: pgmap v1041: 148 pgs: 1 active+remapped+backfill_toofull, 2 active+remapped+backfilling, 145 active+clean; 4.2 GiB data, 16 GiB used, 431 GiB / 447 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.04k op/s; 9061/208080 objects misplaced (4.355%); 43 MiB/s, 689 objects/s recovering 2024-03-11T21:47:36.869 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:47:36 smithi092 ceph-mon[37563]: purged_snaps scrub starts 2024-03-11T21:47:36.869 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:47:36 smithi092 ceph-mon[37563]: purged_snaps scrub ok 2024-03-11T21:47:36.869 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:47:36 smithi092 ceph-mon[37563]: osdmap e488: 8 total, 8 up, 5 in 2024-03-11T21:47:36.869 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:47:36 smithi092 ceph-mon[37563]: pgmap v1041: 148 pgs: 1 active+remapped+backfill_toofull, 2 active+remapped+backfilling, 145 active+clean; 4.2 GiB data, 16 GiB used, 431 GiB / 447 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.04k op/s; 9061/208080 objects misplaced (4.355%); 43 MiB/s, 689 objects/s recovering 2024-03-11T21:47:36.963 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:47:36.997 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:47:36.998 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:47:36.998 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:47:37.099 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:47:37.140 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:47:37.141 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:47:37.141 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:47:37.242 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:47:37.274 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:47:37.274 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:47:37.275 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:47:37.376 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:47:37.414 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:47:37.414 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:47:37.415 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:47:37.515 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:47:37.558 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:47:37.559 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:47:37.559 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:47:37.660 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:47:37.699 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:47:37.700 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:47:37.700 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:47:37.801 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:47:37.824 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:47:37.825 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:47:37.825 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:47:37.926 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:47:37.967 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:47:37.968 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:47:37.968 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:47:38.069 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:47:38.109 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:47:38.109 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:47:38.109 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:47:38.210 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:47:38.244 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:47:38.245 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:47:38.245 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:47:38.346 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:47:38.368 INFO:journalctl@ceph.mgr.y.smithi092.stdout:Mar 11 21:47:37 smithi092 ceph-00db0016-dfed-11ee-95c8-87774f69a715-mgr-y[29801]: ::ffff:172.21.15.97 - - [11/Mar/2024:21:47:37] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-11T21:47:38.383 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:47:38.383 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:47:38.384 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:47:38.485 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:47:38.524 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:47:38.525 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:47:38.525 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:47:38.626 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:47:38.668 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:47:38.668 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:47:38.669 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:47:38.770 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:47:38.796 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:47:38.797 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:47:38.797 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:47:38.898 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:47:38.949 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:47:38.949 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:47:38.949 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:47:39.050 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:47:39.088 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:47:39.089 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:47:39.089 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:47:39.190 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:47:39.223 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:47:39.223 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:47:39.224 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:47:39.325 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:47:39.364 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:47:39.365 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:47:39.365 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:47:39.466 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:47:39.479 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:47:39 smithi092 ceph-mon[37563]: pgmap v1042: 148 pgs: 1 active+remapped+backfill_toofull, 2 active+remapped+backfilling, 145 active+clean; 4.2 GiB data, 16 GiB used, 431 GiB / 447 GiB avail; 853 B/s rd, 0 B/s wr, 710 op/s; 9061/208080 objects misplaced (4.355%); 29 MiB/s, 469 objects/s recovering 2024-03-11T21:47:39.479 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:47:39 smithi092 ceph-mon[37563]: from='mgr.14607 ' entity='mgr.y' 2024-03-11T21:47:39.479 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:47:39 smithi092 ceph-mon[37563]: from='mgr.14607 v1:172.21.15.92:0/1702717597' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T21:47:39.480 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:47:39 smithi092 ceph-mon[29572]: pgmap v1042: 148 pgs: 1 active+remapped+backfill_toofull, 2 active+remapped+backfilling, 145 active+clean; 4.2 GiB data, 16 GiB used, 431 GiB / 447 GiB avail; 853 B/s rd, 0 B/s wr, 710 op/s; 9061/208080 objects misplaced (4.355%); 29 MiB/s, 469 objects/s recovering 2024-03-11T21:47:39.480 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:47:39 smithi092 ceph-mon[29572]: from='mgr.14607 ' entity='mgr.y' 2024-03-11T21:47:39.480 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:47:39 smithi092 ceph-mon[29572]: from='mgr.14607 v1:172.21.15.92:0/1702717597' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T21:47:39.506 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:47:39.507 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:47:39.507 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:47:39.533 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:47:39 smithi097 ceph-mon[33317]: pgmap v1042: 148 pgs: 1 active+remapped+backfill_toofull, 2 active+remapped+backfilling, 145 active+clean; 4.2 GiB data, 16 GiB used, 431 GiB / 447 GiB avail; 853 B/s rd, 0 B/s wr, 710 op/s; 9061/208080 objects misplaced (4.355%); 29 MiB/s, 469 objects/s recovering 2024-03-11T21:47:39.533 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:47:39 smithi097 ceph-mon[33317]: from='mgr.14607 ' entity='mgr.y' 2024-03-11T21:47:39.533 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:47:39 smithi097 ceph-mon[33317]: from='mgr.14607 v1:172.21.15.92:0/1702717597' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T21:47:39.608 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:47:39.645 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:47:39.646 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:47:39.646 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:47:39.747 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:47:39.782 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:47:39.782 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:47:39.782 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:47:39.834 INFO:journalctl@ceph.grafana.a.smithi097.stdout:Mar 11 21:47:39 smithi097 ceph-00db0016-dfed-11ee-95c8-87774f69a715-grafana-a[65126]: logger=cleanup t=2024-03-11T21:47:39.5432996Z level=info msg="Completed cleanup jobs" duration=19.60408ms 2024-03-11T21:47:39.883 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:47:39.927 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:47:39.928 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:47:39.928 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:47:40.029 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:47:40.057 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:47:40.058 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:47:40.058 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:47:40.159 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:47:40.196 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:47:40.197 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:47:40.197 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:47:40.298 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:47:40.337 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:47:40.337 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:47:40.338 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:47:40.438 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:47:40.483 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:47:40.484 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:47:40.484 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:47:40.585 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:47:40.623 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:47:40.624 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:47:40.624 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:47:40.725 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:47:40.758 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:47:40.758 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:47:40.759 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:47:40.859 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:47:40.910 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:47:40.911 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:47:40.911 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:47:41.012 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:47:41.044 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:47:41.045 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:47:41.045 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:47:41.146 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:47:41.191 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:47:41.192 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:47:41.192 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:47:41.293 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:47:41.325 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:47:41.326 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:47:41.326 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:47:41.427 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:47:41.454 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:47:41 smithi092 ceph-mon[29572]: pgmap v1043: 148 pgs: 1 active+remapped+backfill_toofull, 2 active+remapped+backfilling, 145 active+clean; 3.8 GiB data, 15 GiB used, 432 GiB / 447 GiB avail; 1.4 KiB/s rd, 0 B/s wr, 1.47k op/s; 7754/189711 objects misplaced (4.087%); 31 MiB/s, 488 objects/s recovering 2024-03-11T21:47:41.455 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:47:41 smithi092 ceph-mon[37563]: pgmap v1043: 148 pgs: 1 active+remapped+backfill_toofull, 2 active+remapped+backfilling, 145 active+clean; 3.8 GiB data, 15 GiB used, 432 GiB / 447 GiB avail; 1.4 KiB/s rd, 0 B/s wr, 1.47k op/s; 7754/189711 objects misplaced (4.087%); 31 MiB/s, 488 objects/s recovering 2024-03-11T21:47:41.455 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:47:41.456 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:47:41.456 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:47:41.557 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:47:41.584 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:47:41 smithi097 ceph-mon[33317]: pgmap v1043: 148 pgs: 1 active+remapped+backfill_toofull, 2 active+remapped+backfilling, 145 active+clean; 3.8 GiB data, 15 GiB used, 432 GiB / 447 GiB avail; 1.4 KiB/s rd, 0 B/s wr, 1.47k op/s; 7754/189711 objects misplaced (4.087%); 31 MiB/s, 488 objects/s recovering 2024-03-11T21:47:41.586 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:47:41.587 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:47:41.587 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:47:41.688 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:47:41.729 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:47:41.730 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:47:41.730 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:47:41.831 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:47:41.862 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:47:41.863 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:47:41.863 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:47:41.963 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:47:41.981 DEBUG:teuthology.orchestra.run.smithi092:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid 00db0016-dfed-11ee-95c8-87774f69a715 -- ceph daemon osd.1 dump_ops_in_flight 2024-03-11T21:47:42.001 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:47:42.002 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:47:42.002 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:47:42.103 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:47:42.143 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:47:42.144 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:47:42.144 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:47:42.245 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:47:42.288 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:47:42.288 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:47:42.288 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:47:42.389 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:47:42.390 DEBUG:teuthology.orchestra.run.smithi092:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid 00db0016-dfed-11ee-95c8-87774f69a715 -- ceph osd set nodeep-scrub 2024-03-11T21:47:42.419 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:47:42.420 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:47:42.420 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:47:42.521 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:47:42.550 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:47:42.551 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:47:42.551 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:47:42.652 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:47:42.716 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:47:42.716 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:47:42.717 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:47:42.818 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:47:42.844 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:47:42.845 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:47:42.845 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:47:42.873 INFO:teuthology.orchestra.run.smithi092.stderr:Inferring config /var/lib/ceph/00db0016-dfed-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T21:47:42.899 INFO:teuthology.orchestra.run.smithi092.stdout:osd.0: {} 2024-03-11T21:47:42.946 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:47:42.947 INFO:teuthology.orchestra.run.smithi092.stdout:osd.1: {} 2024-03-11T21:47:42.955 INFO:teuthology.orchestra.run.smithi092.stderr:Inferring config /var/lib/ceph/00db0016-dfed-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T21:47:42.989 INFO:teuthology.orchestra.run.smithi092.stdout:osd.2: {} 2024-03-11T21:47:42.994 INFO:teuthology.orchestra.run.smithi092.stdout: 2024-03-11T21:47:43.008 INFO:teuthology.orchestra.run.smithi092.stderr:dumped all 2024-03-11T21:47:43.011 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:47:43.011 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:47:43.011 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:47:43.043 INFO:teuthology.orchestra.run.smithi092.stdout:osd.3: {} 2024-03-11T21:47:43.086 INFO:teuthology.orchestra.run.smithi092.stdout:osd.4: {} 2024-03-11T21:47:43.112 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:47:43.126 INFO:teuthology.orchestra.run.smithi092.stdout:osd.5: {} 2024-03-11T21:47:43.143 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:47:43.144 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:47:43.144 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:47:43.167 INFO:teuthology.orchestra.run.smithi092.stdout:osd.6: {} 2024-03-11T21:47:43.207 INFO:teuthology.orchestra.run.smithi092.stdout:osd.7: {} 2024-03-11T21:47:43.211 INFO:teuthology.orchestra.run.smithi092.stderr:osd.0: osd_enable_op_tracker = '' 2024-03-11T21:47:43.212 INFO:teuthology.orchestra.run.smithi092.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-11T21:47:43.212 INFO:teuthology.orchestra.run.smithi092.stderr:osd.2: osd_enable_op_tracker = '' 2024-03-11T21:47:43.212 INFO:teuthology.orchestra.run.smithi092.stderr:osd.3: osd_enable_op_tracker = '' 2024-03-11T21:47:43.212 INFO:teuthology.orchestra.run.smithi092.stderr:osd.4: osd_enable_op_tracker = '' 2024-03-11T21:47:43.212 INFO:teuthology.orchestra.run.smithi092.stderr:osd.5: osd_enable_op_tracker = '' 2024-03-11T21:47:43.212 INFO:teuthology.orchestra.run.smithi092.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-11T21:47:43.212 INFO:teuthology.orchestra.run.smithi092.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-11T21:47:43.245 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:47:43.276 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:47:43 smithi092 ceph-mon[29572]: pgmap v1044: 148 pgs: 1 active+remapped+backfill_toofull, 2 active+remapped+backfilling, 145 active+clean; 3.8 GiB data, 15 GiB used, 432 GiB / 447 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.11k op/s; 6867/187122 objects misplaced (3.670%); 28 MiB/s, 448 objects/s recovering 2024-03-11T21:47:43.276 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:47:43 smithi092 ceph-mon[29572]: from='client.? v1:172.21.15.92:0/1477107764' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-11T21:47:43.277 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:47:43 smithi092 ceph-mon[37563]: pgmap v1044: 148 pgs: 1 active+remapped+backfill_toofull, 2 active+remapped+backfilling, 145 active+clean; 3.8 GiB data, 15 GiB used, 432 GiB / 447 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.11k op/s; 6867/187122 objects misplaced (3.670%); 28 MiB/s, 448 objects/s recovering 2024-03-11T21:47:43.277 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:47:43 smithi092 ceph-mon[37563]: from='client.? v1:172.21.15.92:0/1477107764' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-11T21:47:43.278 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:47:43.278 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:47:43.279 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:47:43.380 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:47:43.419 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:47:43.419 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:47:43.420 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:47:43.520 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:47:43.564 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:47:43.565 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:47:43.565 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:47:43.584 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:47:43 smithi097 ceph-mon[33317]: pgmap v1044: 148 pgs: 1 active+remapped+backfill_toofull, 2 active+remapped+backfilling, 145 active+clean; 3.8 GiB data, 15 GiB used, 432 GiB / 447 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.11k op/s; 6867/187122 objects misplaced (3.670%); 28 MiB/s, 448 objects/s recovering 2024-03-11T21:47:43.584 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:47:43 smithi097 ceph-mon[33317]: from='client.? v1:172.21.15.92:0/1477107764' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-11T21:47:43.585 INFO:journalctl@ceph.iscsi.iscsi.a.smithi097.stdout:Mar 11 21:47:43 smithi097 ceph-00db0016-dfed-11ee-95c8-87774f69a715-iscsi-iscsi-a[60943]: debug there is no tcmu-runner data available 2024-03-11T21:47:43.666 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:47:43.699 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:47:43.700 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:47:43.700 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:47:43.801 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:47:43.845 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:47:43.848 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:47:43.848 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:47:43.949 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:47:43.987 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:47:43.988 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:47:43.989 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:47:44.089 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:47:44.125 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:47:44.125 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:47:44.126 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:47:44.227 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:47:44.260 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:47:44.261 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:47:44.261 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:47:44.362 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:47:44.403 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:47:44.404 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:47:44.404 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:47:44.505 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:47:44.545 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:47:44.546 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:47:44.546 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:47:44.584 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:47:44 smithi097 ceph-mon[33317]: from='client.16122 v1:172.21.15.92:0/1989721483' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-11T21:47:44.584 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:47:44 smithi097 ceph-mon[33317]: from='client.24469 v1:172.21.15.97:0/404208310' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:47:44.618 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:47:44 smithi092 ceph-mon[29572]: from='client.16122 v1:172.21.15.92:0/1989721483' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-11T21:47:44.619 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:47:44 smithi092 ceph-mon[29572]: from='client.24469 v1:172.21.15.97:0/404208310' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:47:44.619 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:47:44 smithi092 ceph-mon[37563]: from='client.16122 v1:172.21.15.92:0/1989721483' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-11T21:47:44.619 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:47:44 smithi092 ceph-mon[37563]: from='client.24469 v1:172.21.15.97:0/404208310' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:47:44.647 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:47:44.681 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:47:44.681 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:47:44.682 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:47:44.782 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:47:44.819 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:47:44.819 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:47:44.820 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:47:44.920 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:47:44.951 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:47:44.951 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:47:44.952 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:47:45.053 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:47:45.076 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:47:45.077 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:47:45.077 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:47:45.178 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:47:45.211 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:47:45.212 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:47:45.212 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:47:45.313 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:47:45.347 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:47:45.347 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:47:45.347 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:47:45.448 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:47:45.488 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:47:45.489 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:47:45.489 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:47:45.585 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:47:45 smithi097 ceph-mon[33317]: pgmap v1045: 148 pgs: 1 active+remapped+backfill_toofull, 2 active+remapped+backfilling, 145 active+clean; 3.5 GiB data, 15 GiB used, 432 GiB / 447 GiB avail; 1023 B/s rd, 0 B/s wr, 1.31k op/s; 6446/173235 objects misplaced (3.721%); 23 MiB/s, 360 objects/s recovering 2024-03-11T21:47:45.585 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:47:45 smithi097 ceph-mon[33317]: purged_snaps scrub starts 2024-03-11T21:47:45.585 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:47:45 smithi097 ceph-mon[33317]: purged_snaps scrub ok 2024-03-11T21:47:45.585 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:47:45 smithi097 ceph-mon[33317]: purged_snaps scrub starts 2024-03-11T21:47:45.585 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:47:45 smithi097 ceph-mon[33317]: purged_snaps scrub ok 2024-03-11T21:47:45.585 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:47:45 smithi097 ceph-mon[33317]: osdmap e489: 8 total, 8 up, 5 in 2024-03-11T21:47:45.589 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:47:45.618 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:47:45 smithi092 ceph-mon[29572]: pgmap v1045: 148 pgs: 1 active+remapped+backfill_toofull, 2 active+remapped+backfilling, 145 active+clean; 3.5 GiB data, 15 GiB used, 432 GiB / 447 GiB avail; 1023 B/s rd, 0 B/s wr, 1.31k op/s; 6446/173235 objects misplaced (3.721%); 23 MiB/s, 360 objects/s recovering 2024-03-11T21:47:45.619 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:47:45 smithi092 ceph-mon[29572]: purged_snaps scrub starts 2024-03-11T21:47:45.619 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:47:45 smithi092 ceph-mon[29572]: purged_snaps scrub ok 2024-03-11T21:47:45.619 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:47:45 smithi092 ceph-mon[29572]: purged_snaps scrub starts 2024-03-11T21:47:45.619 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:47:45 smithi092 ceph-mon[29572]: purged_snaps scrub ok 2024-03-11T21:47:45.619 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:47:45 smithi092 ceph-mon[29572]: osdmap e489: 8 total, 8 up, 5 in 2024-03-11T21:47:45.619 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:47:45 smithi092 ceph-mon[37563]: pgmap v1045: 148 pgs: 1 active+remapped+backfill_toofull, 2 active+remapped+backfilling, 145 active+clean; 3.5 GiB data, 15 GiB used, 432 GiB / 447 GiB avail; 1023 B/s rd, 0 B/s wr, 1.31k op/s; 6446/173235 objects misplaced (3.721%); 23 MiB/s, 360 objects/s recovering 2024-03-11T21:47:45.619 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:47:45 smithi092 ceph-mon[37563]: purged_snaps scrub starts 2024-03-11T21:47:45.619 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:47:45 smithi092 ceph-mon[37563]: purged_snaps scrub ok 2024-03-11T21:47:45.619 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:47:45 smithi092 ceph-mon[37563]: purged_snaps scrub starts 2024-03-11T21:47:45.619 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:47:45 smithi092 ceph-mon[37563]: purged_snaps scrub ok 2024-03-11T21:47:45.619 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:47:45 smithi092 ceph-mon[37563]: osdmap e489: 8 total, 8 up, 5 in 2024-03-11T21:47:45.629 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:47:45.629 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:47:45.630 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:47:45.731 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:47:45.770 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:47:45.771 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:47:45.771 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:47:45.872 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:47:45.920 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:47:45.920 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:47:45.920 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:47:46.021 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:47:46.053 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:47:46.054 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:47:46.054 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:47:46.155 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:47:46.187 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:47:46.188 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:47:46.188 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:47:46.289 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:47:46.331 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:47:46.331 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:47:46.332 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:47:46.433 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:47:46.472 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:47:46.474 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:47:46.474 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:47:46.575 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:47:46.606 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:47:46.607 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:47:46.607 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:47:46.708 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:47:46.753 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:47:46.754 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:47:46.754 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:47:46.855 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:47:46.888 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:47:46.889 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:47:46.889 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:47:46.990 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:47:47.025 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:47:47.026 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:47:47.026 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:47:47.127 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:47:47.196 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:47:47.197 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:47:47.197 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:47:47.298 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:47:47.333 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:47:47.333 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:47:47.333 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:47:47.434 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:47:47.471 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:47:47.472 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:47:47.472 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:47:47.554 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:47:47 smithi092 ceph-mon[29572]: pgmap v1047: 148 pgs: 1 active+remapped+backfill_toofull, 2 active+remapped+backfilling, 145 active+clean; 3.4 GiB data, 14 GiB used, 433 GiB / 447 GiB avail; 1023 B/s rd, 0 B/s wr, 1.38k op/s; 4903/166845 objects misplaced (2.939%); 17 MiB/s, 264 objects/s recovering 2024-03-11T21:47:47.555 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:47:47 smithi092 ceph-mon[37563]: pgmap v1047: 148 pgs: 1 active+remapped+backfill_toofull, 2 active+remapped+backfilling, 145 active+clean; 3.4 GiB data, 14 GiB used, 433 GiB / 447 GiB avail; 1023 B/s rd, 0 B/s wr, 1.38k op/s; 4903/166845 objects misplaced (2.939%); 17 MiB/s, 264 objects/s recovering 2024-03-11T21:47:47.573 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:47:47.584 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:47:47 smithi097 ceph-mon[33317]: pgmap v1047: 148 pgs: 1 active+remapped+backfill_toofull, 2 active+remapped+backfilling, 145 active+clean; 3.4 GiB data, 14 GiB used, 433 GiB / 447 GiB avail; 1023 B/s rd, 0 B/s wr, 1.38k op/s; 4903/166845 objects misplaced (2.939%); 17 MiB/s, 264 objects/s recovering 2024-03-11T21:47:47.603 INFO:teuthology.orchestra.run.smithi092.stdout:{"pg_ready":true,"pg_map":{"version":1044,"stamp":"2024-03-11T21:47:41.486876+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":4075297216,"num_objects":62374,"num_object_clones":0,"num_object_copies":187122,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":6867,"num_objects_unfound":0,"num_objects_dirty":62374,"num_whiteouts":0,"num_read":2150,"num_read_kb":1998,"num_write":215703,"num_write_kb":8856542,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":17279,"num_bytes_recovered":1107767997,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":55905,"ondisk_log_size":55905,"up":444,"acting":444,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":444,"num_osds":5,"num_per_pool_osds":5,"num_per_pool_omap_osds":5,"kb":468725760,"kb_used":15909940,"kb_used_data":11490384,"kb_used_omap":7,"kb_used_meta":4419320,"kb_avail":452815820,"statfs":{"total":479975178240,"available":463683399680,"internally_reserved":0,"allocated":11766153216,"data_stored":11764149894,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7969,"internal_metadata":4525383903},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-587661312,"num_objects":-8967,"num_object_clones":0,"num_object_copies":-26901,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-4825,"num_objects_unfound":0,"num_objects_dirty":-8967,"num_whiteouts":0,"num_read":10,"num_read_kb":10,"num_write":8967,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3610,"num_bytes_recovered":236584960,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.057375"},"pg_stats":[{"pgid":"6.1b","version":"0'0","reported_seq":533,"reported_epoch":488,"state":"active+clean","last_fresh":"2024-03-11T21:47:35.478409+0000","last_change":"2024-03-11T21:44:30.546716+0000","last_active":"2024-03-11T21:47:35.478409+0000","last_peered":"2024-03-11T21:47:35.478409+0000","last_clean":"2024-03-11T21:47:35.478409+0000","last_became_active":"2024-03-11T21:44:21.459293+0000","last_became_peered":"2024-03-11T21:44:21.459293+0000","last_unstale":"2024-03-11T21:47:35.478409+0000","last_undegraded":"2024-03-11T21:47:35.478409+0000","last_fullsized":"2024-03-11T21:47:35.478409+0000","mapping_epoch":432,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":433,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:44:30.546596+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:43.559452+0000","last_clean_scrub_stamp":"2024-03-11T21:44:30.546596+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":108,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,1],"acting":[7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":121,"reported_epoch":488,"state":"active+clean","last_fresh":"2024-03-11T21:47:35.473711+0000","last_change":"2024-03-11T21:45:16.535519+0000","last_active":"2024-03-11T21:47:35.473711+0000","last_peered":"2024-03-11T21:47:35.473711+0000","last_clean":"2024-03-11T21:47:35.473711+0000","last_became_active":"2024-03-11T21:44:35.624802+0000","last_became_peered":"2024-03-11T21:44:35.624802+0000","last_unstale":"2024-03-11T21:47:35.473711+0000","last_undegraded":"2024-03-11T21:47:35.473711+0000","last_fullsized":"2024-03-11T21:47:35.473711+0000","mapping_epoch":437,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":438,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:45:16.535402+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:41.544550+0000","last_clean_scrub_stamp":"2024-03-11T21:45:16.535402+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,2],"acting":[6,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.1f","version":"0'0","reported_seq":945,"reported_epoch":488,"state":"active+clean","last_fresh":"2024-03-11T21:47:35.470570+0000","last_change":"2024-03-11T21:45:34.145990+0000","last_active":"2024-03-11T21:47:35.470570+0000","last_peered":"2024-03-11T21:47:35.470570+0000","last_clean":"2024-03-11T21:47:35.470570+0000","last_became_active":"2024-03-11T21:44:35.644794+0000","last_became_peered":"2024-03-11T21:44:35.644794+0000","last_unstale":"2024-03-11T21:47:35.470570+0000","last_undegraded":"2024-03-11T21:47:35.470570+0000","last_fullsized":"2024-03-11T21:47:35.470570+0000","mapping_epoch":437,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":438,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:45:34.145740+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:37.512018+0000","last_clean_scrub_stamp":"2024-03-11T21:45:34.145740+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,6],"acting":[0,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.19","version":"456'39","reported_seq":303,"reported_epoch":488,"state":"active+clean","last_fresh":"2024-03-11T21:47:35.479483+0000","last_change":"2024-03-11T21:46:53.309331+0000","last_active":"2024-03-11T21:47:35.479483+0000","last_peered":"2024-03-11T21:47:35.479483+0000","last_clean":"2024-03-11T21:47:35.479483+0000","last_became_active":"2024-03-11T21:46:53.275832+0000","last_became_peered":"2024-03-11T21:46:53.275832+0000","last_unstale":"2024-03-11T21:47:35.479483+0000","last_undegraded":"2024-03-11T21:47:35.479483+0000","last_fullsized":"2024-03-11T21:47:35.479483+0000","mapping_epoch":473,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":474,"parent":"0.0","parent_split_bits":0,"last_scrub":"260'27","last_scrub_stamp":"2024-03-11T21:45:19.592959+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:39.525554+0000","last_clean_scrub_stamp":"2024-03-11T21:45:19.592959+0000","objects_scrubbed":7,"log_size":39,"log_dups_size":0,"ondisk_log_size":39,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:47:38.777981+0000","scrub_duration":22,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":61,"num_read_kb":52,"num_write":40,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.1a","version":"0'0","reported_seq":34,"reported_epoch":488,"state":"active+clean","last_fresh":"2024-03-11T21:47:35.477482+0000","last_change":"2024-03-11T21:47:20.274985+0000","last_active":"2024-03-11T21:47:35.477482+0000","last_peered":"2024-03-11T21:47:35.477482+0000","last_clean":"2024-03-11T21:47:35.477482+0000","last_became_active":"2024-03-11T21:47:20.274656+0000","last_became_peered":"2024-03-11T21:47:20.274656+0000","last_unstale":"2024-03-11T21:47:35.477482+0000","last_undegraded":"2024-03-11T21:47:35.477482+0000","last_fullsized":"2024-03-11T21:47:35.477482+0000","mapping_epoch":478,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":479,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:46:15.911512+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:43.559452+0000","last_clean_scrub_stamp":"2024-03-11T21:46:15.911512+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,6],"acting":[7,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":920,"reported_epoch":488,"state":"active+clean","last_fresh":"2024-03-11T21:47:35.477574+0000","last_change":"2024-03-11T21:47:20.278290+0000","last_active":"2024-03-11T21:47:35.477574+0000","last_peered":"2024-03-11T21:47:35.477574+0000","last_clean":"2024-03-11T21:47:35.477574+0000","last_became_active":"2024-03-11T21:47:20.277953+0000","last_became_peered":"2024-03-11T21:47:20.277953+0000","last_unstale":"2024-03-11T21:47:35.477574+0000","last_undegraded":"2024-03-11T21:47:35.477574+0000","last_fullsized":"2024-03-11T21:47:35.477574+0000","mapping_epoch":478,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":479,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:46:29.023645+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:41.544550+0000","last_clean_scrub_stamp":"2024-03-11T21:46:29.023645+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:47:57.515895+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,2],"acting":[7,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.1e","version":"0'0","reported_seq":546,"reported_epoch":488,"state":"active+clean","last_fresh":"2024-03-11T21:47:35.472924+0000","last_change":"2024-03-11T21:46:53.259936+0000","last_active":"2024-03-11T21:47:35.472924+0000","last_peered":"2024-03-11T21:47:35.472924+0000","last_clean":"2024-03-11T21:47:35.472924+0000","last_became_active":"2024-03-11T21:46:53.259601+0000","last_became_peered":"2024-03-11T21:46:53.259601+0000","last_unstale":"2024-03-11T21:47:35.472924+0000","last_undegraded":"2024-03-11T21:47:35.472924+0000","last_fullsized":"2024-03-11T21:47:35.472924+0000","mapping_epoch":473,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":474,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:44:37.214702+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:37.512018+0000","last_clean_scrub_stamp":"2024-03-11T21:44:37.214702+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,7],"acting":[0,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.18","version":"456'21","reported_seq":600,"reported_epoch":488,"state":"active+clean","last_fresh":"2024-03-11T21:47:35.473882+0000","last_change":"2024-03-11T21:45:24.708559+0000","last_active":"2024-03-11T21:47:35.473882+0000","last_peered":"2024-03-11T21:47:35.473882+0000","last_clean":"2024-03-11T21:47:35.473882+0000","last_became_active":"2024-03-11T21:44:35.621442+0000","last_became_peered":"2024-03-11T21:44:35.621442+0000","last_unstale":"2024-03-11T21:47:35.473882+0000","last_undegraded":"2024-03-11T21:47:35.473882+0000","last_fullsized":"2024-03-11T21:47:35.473882+0000","mapping_epoch":437,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":438,"parent":"0.0","parent_split_bits":0,"last_scrub":"260'15","last_scrub_stamp":"2024-03-11T21:45:24.708228+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:39.525554+0000","last_clean_scrub_stamp":"2024-03-11T21:45:24.708228+0000","objects_scrubbed":4,"log_size":21,"log_dups_size":0,"ondisk_log_size":21,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":34,"num_read_kb":28,"num_write":22,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,2],"acting":[6,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":847,"reported_epoch":488,"state":"active+clean","last_fresh":"2024-03-11T21:47:35.479567+0000","last_change":"2024-03-11T21:47:05.896514+0000","last_active":"2024-03-11T21:47:35.479567+0000","last_peered":"2024-03-11T21:47:35.479567+0000","last_clean":"2024-03-11T21:47:35.479567+0000","last_became_active":"2024-03-11T21:46:53.273211+0000","last_became_peered":"2024-03-11T21:46:53.273211+0000","last_unstale":"2024-03-11T21:47:35.479567+0000","last_undegraded":"2024-03-11T21:47:35.479567+0000","last_fullsized":"2024-03-11T21:47:35.479567+0000","mapping_epoch":473,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":474,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:47:05.896361+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:32:12.654573+0000","last_clean_scrub_stamp":"2024-03-11T21:47:05.896361+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:48:30.974239+0000","scrub_duration":118,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":568,"reported_epoch":488,"state":"active+clean","last_fresh":"2024-03-11T21:47:35.477761+0000","last_change":"2024-03-11T21:44:35.626808+0000","last_active":"2024-03-11T21:47:35.477761+0000","last_peered":"2024-03-11T21:47:35.477761+0000","last_clean":"2024-03-11T21:47:35.477761+0000","last_became_active":"2024-03-11T21:44:35.626456+0000","last_became_peered":"2024-03-11T21:44:35.626456+0000","last_unstale":"2024-03-11T21:47:35.477761+0000","last_undegraded":"2024-03-11T21:47:35.477761+0000","last_fullsized":"2024-03-11T21:47:35.477761+0000","mapping_epoch":437,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":438,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:44:19.481984+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:41.544550+0000","last_clean_scrub_stamp":"2024-03-11T21:44:19.481984+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,0],"acting":[7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.1d","version":"0'0","reported_seq":932,"reported_epoch":488,"state":"active+clean","last_fresh":"2024-03-11T21:47:35.479921+0000","last_change":"2024-03-11T21:46:28.543345+0000","last_active":"2024-03-11T21:47:35.479921+0000","last_peered":"2024-03-11T21:47:35.479921+0000","last_clean":"2024-03-11T21:47:35.479921+0000","last_became_active":"2024-03-11T21:25:38.542628+0000","last_became_peered":"2024-03-11T21:25:38.542628+0000","last_unstale":"2024-03-11T21:47:35.479921+0000","last_undegraded":"2024-03-11T21:47:35.479921+0000","last_fullsized":"2024-03-11T21:47:35.479921+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:46:28.543232+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:37.512018+0000","last_clean_scrub_stamp":"2024-03-11T21:46:28.543232+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:47:46.672920+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,0],"acting":[7,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.1b","version":"456'9","reported_seq":573,"reported_epoch":488,"state":"active+clean","last_fresh":"2024-03-11T21:47:35.480903+0000","last_change":"2024-03-11T21:46:08.932984+0000","last_active":"2024-03-11T21:47:35.480903+0000","last_peered":"2024-03-11T21:47:35.480903+0000","last_clean":"2024-03-11T21:47:35.480903+0000","last_became_active":"2024-03-11T21:44:35.639372+0000","last_became_peered":"2024-03-11T21:44:35.639372+0000","last_unstale":"2024-03-11T21:47:35.480903+0000","last_undegraded":"2024-03-11T21:47:35.480903+0000","last_fullsized":"2024-03-11T21:47:35.480903+0000","mapping_epoch":437,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":438,"parent":"0.0","parent_split_bits":0,"last_scrub":"456'9","last_scrub_stamp":"2024-03-11T21:46:08.932675+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:39.525554+0000","last_clean_scrub_stamp":"2024-03-11T21:46:08.932675+0000","objects_scrubbed":2,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":16,"num_read_kb":12,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,7],"acting":[1,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":906,"reported_epoch":488,"state":"active+clean","last_fresh":"2024-03-11T21:47:35.474602+0000","last_change":"2024-03-11T21:45:32.208394+0000","last_active":"2024-03-11T21:47:35.474602+0000","last_peered":"2024-03-11T21:47:35.474602+0000","last_clean":"2024-03-11T21:47:35.474602+0000","last_became_active":"2024-03-11T21:25:44.589786+0000","last_became_peered":"2024-03-11T21:25:44.589786+0000","last_unstale":"2024-03-11T21:47:35.474602+0000","last_undegraded":"2024-03-11T21:47:35.474602+0000","last_fullsized":"2024-03-11T21:47:35.474602+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:45:32.208161+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:43.559452+0000","last_clean_scrub_stamp":"2024-03-11T21:45:32.208161+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,7],"acting":[0,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":400,"reported_epoch":488,"state":"active+clean","last_fresh":"2024-03-11T21:47:35.476868+0000","last_change":"2024-03-11T21:46:53.261100+0000","last_active":"2024-03-11T21:47:35.476868+0000","last_peered":"2024-03-11T21:47:35.476868+0000","last_clean":"2024-03-11T21:47:35.476868+0000","last_became_active":"2024-03-11T21:46:53.260095+0000","last_became_peered":"2024-03-11T21:46:53.260095+0000","last_unstale":"2024-03-11T21:47:35.476868+0000","last_undegraded":"2024-03-11T21:47:35.476868+0000","last_fullsized":"2024-03-11T21:47:35.476868+0000","mapping_epoch":473,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":474,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:46:13.862444+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:43:35.536164+0000","last_clean_scrub_stamp":"2024-03-11T21:46:13.862444+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,6],"acting":[0,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1c","version":"0'0","reported_seq":930,"reported_epoch":488,"state":"active+clean","last_fresh":"2024-03-11T21:47:35.479103+0000","last_change":"2024-03-11T21:46:53.275943+0000","last_active":"2024-03-11T21:47:35.479103+0000","last_peered":"2024-03-11T21:47:35.479103+0000","last_clean":"2024-03-11T21:47:35.479103+0000","last_became_active":"2024-03-11T21:46:53.275174+0000","last_became_peered":"2024-03-11T21:46:53.275174+0000","last_unstale":"2024-03-11T21:47:35.479103+0000","last_undegraded":"2024-03-11T21:47:35.479103+0000","last_fullsized":"2024-03-11T21:47:35.479103+0000","mapping_epoch":473,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":474,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:45:24.180727+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:28:32.566437+0000","last_clean_scrub_stamp":"2024-03-11T21:45:24.180727+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,6],"acting":[7,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.1a","version":"456'25","reported_seq":804,"reported_epoch":488,"state":"active+clean","last_fresh":"2024-03-11T21:47:35.470033+0000","last_change":"2024-03-11T21:46:06.451173+0000","last_active":"2024-03-11T21:47:35.470033+0000","last_peered":"2024-03-11T21:47:35.470033+0000","last_clean":"2024-03-11T21:47:35.470033+0000","last_became_active":"2024-03-11T21:44:35.631905+0000","last_became_peered":"2024-03-11T21:44:35.631905+0000","last_unstale":"2024-03-11T21:47:35.470033+0000","last_undegraded":"2024-03-11T21:47:35.470033+0000","last_fullsized":"2024-03-11T21:47:35.470033+0000","mapping_epoch":437,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":438,"parent":"0.0","parent_split_bits":0,"last_scrub":"456'25","last_scrub_stamp":"2024-03-11T21:46:06.451051+0000","last_deep_scrub":"350'19","last_deep_scrub_stamp":"2024-03-11T21:43:25.898708+0000","last_clean_scrub_stamp":"2024-03-11T21:46:06.451051+0000","objects_scrubbed":4,"log_size":25,"log_dups_size":0,"ondisk_log_size":25,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":35,"num_read_kb":29,"num_write":22,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,6],"acting":[0,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.1f","version":"0'0","reported_seq":469,"reported_epoch":488,"state":"active+clean","last_fresh":"2024-03-11T21:47:35.475078+0000","last_change":"2024-03-11T21:46:53.266767+0000","last_active":"2024-03-11T21:47:35.475078+0000","last_peered":"2024-03-11T21:47:35.475078+0000","last_clean":"2024-03-11T21:47:35.475078+0000","last_became_active":"2024-03-11T21:46:53.265941+0000","last_became_peered":"2024-03-11T21:46:53.265941+0000","last_unstale":"2024-03-11T21:47:35.475078+0000","last_undegraded":"2024-03-11T21:47:35.475078+0000","last_fullsized":"2024-03-11T21:47:35.475078+0000","mapping_epoch":473,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":474,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:44:31.472397+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:44:31.472397+0000","last_clean_scrub_stamp":"2024-03-11T21:44:31.472397+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":108,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,2],"acting":[6,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.1c","version":"0'0","reported_seq":917,"reported_epoch":488,"state":"active+clean","last_fresh":"2024-03-11T21:47:35.478663+0000","last_change":"2024-03-11T21:45:26.645211+0000","last_active":"2024-03-11T21:47:35.478663+0000","last_peered":"2024-03-11T21:47:35.478663+0000","last_clean":"2024-03-11T21:47:35.478663+0000","last_became_active":"2024-03-11T21:44:35.624752+0000","last_became_peered":"2024-03-11T21:44:35.624752+0000","last_unstale":"2024-03-11T21:47:35.478663+0000","last_undegraded":"2024-03-11T21:47:35.478663+0000","last_fullsized":"2024-03-11T21:47:35.478663+0000","mapping_epoch":437,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":438,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:45:26.644785+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:43:32.021344+0000","last_clean_scrub_stamp":"2024-03-11T21:45:26.644785+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:47:38.777951+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,1],"acting":[2,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1b","version":"0'0","reported_seq":565,"reported_epoch":488,"state":"active+clean","last_fresh":"2024-03-11T21:47:35.478942+0000","last_change":"2024-03-11T21:46:53.270436+0000","last_active":"2024-03-11T21:47:35.478942+0000","last_peered":"2024-03-11T21:47:35.478942+0000","last_clean":"2024-03-11T21:47:35.478942+0000","last_became_active":"2024-03-11T21:46:53.270094+0000","last_became_peered":"2024-03-11T21:46:53.270094+0000","last_unstale":"2024-03-11T21:47:35.478942+0000","last_undegraded":"2024-03-11T21:47:35.478942+0000","last_fullsized":"2024-03-11T21:47:35.478942+0000","mapping_epoch":473,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":474,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:45:29.145166+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:34:27.998724+0000","last_clean_scrub_stamp":"2024-03-11T21:45:29.145166+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,0],"acting":[7,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.1d","version":"456'24","reported_seq":293,"reported_epoch":488,"state":"active+clean","last_fresh":"2024-03-11T21:47:35.489905+0000","last_change":"2024-03-11T21:46:53.294177+0000","last_active":"2024-03-11T21:47:35.489905+0000","last_peered":"2024-03-11T21:47:35.489905+0000","last_clean":"2024-03-11T21:47:35.489905+0000","last_became_active":"2024-03-11T21:46:53.273491+0000","last_became_peered":"2024-03-11T21:46:53.273491+0000","last_unstale":"2024-03-11T21:47:35.489905+0000","last_undegraded":"2024-03-11T21:47:35.489905+0000","last_fullsized":"2024-03-11T21:47:35.489905+0000","mapping_epoch":473,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":474,"parent":"0.0","parent_split_bits":0,"last_scrub":"456'24","last_scrub_stamp":"2024-03-11T21:46:10.906594+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:39.525554+0000","last_clean_scrub_stamp":"2024-03-11T21:46:10.906594+0000","objects_scrubbed":5,"log_size":24,"log_dups_size":0,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":42,"num_read_kb":33,"num_write":26,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":660,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1e","version":"0'0","reported_seq":127,"reported_epoch":488,"state":"active+clean","last_fresh":"2024-03-11T21:47:35.474273+0000","last_change":"2024-03-11T21:46:53.255530+0000","last_active":"2024-03-11T21:47:35.474273+0000","last_peered":"2024-03-11T21:47:35.474273+0000","last_clean":"2024-03-11T21:47:35.474273+0000","last_became_active":"2024-03-11T21:46:53.255164+0000","last_became_peered":"2024-03-11T21:46:53.255164+0000","last_unstale":"2024-03-11T21:47:35.474273+0000","last_undegraded":"2024-03-11T21:47:35.474273+0000","last_fullsized":"2024-03-11T21:47:35.474273+0000","mapping_epoch":473,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":474,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:44:39.482452+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:43.559452+0000","last_clean_scrub_stamp":"2024-03-11T21:44:39.482452+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":28,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,7],"acting":[6,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.1d","version":"0'0","reported_seq":921,"reported_epoch":488,"state":"active+clean","last_fresh":"2024-03-11T21:47:35.490131+0000","last_change":"2024-03-11T21:46:53.262783+0000","last_active":"2024-03-11T21:47:35.490131+0000","last_peered":"2024-03-11T21:47:35.490131+0000","last_clean":"2024-03-11T21:47:35.490131+0000","last_became_active":"2024-03-11T21:46:53.260759+0000","last_became_peered":"2024-03-11T21:46:53.260759+0000","last_unstale":"2024-03-11T21:47:35.490131+0000","last_undegraded":"2024-03-11T21:47:35.490131+0000","last_fullsized":"2024-03-11T21:47:35.490131+0000","mapping_epoch":473,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":474,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:46:05.965504+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:30:18.678531+0000","last_clean_scrub_stamp":"2024-03-11T21:46:05.965504+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,6],"acting":[1,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1a","version":"0'0","reported_seq":951,"reported_epoch":488,"state":"active+clean","last_fresh":"2024-03-11T21:47:35.473350+0000","last_change":"2024-03-11T21:45:31.690813+0000","last_active":"2024-03-11T21:47:35.473350+0000","last_peered":"2024-03-11T21:47:35.473350+0000","last_clean":"2024-03-11T21:47:35.473350+0000","last_became_active":"2024-03-11T21:44:35.630761+0000","last_became_peered":"2024-03-11T21:44:35.630761+0000","last_unstale":"2024-03-11T21:47:35.473350+0000","last_undegraded":"2024-03-11T21:47:35.473350+0000","last_fullsized":"2024-03-11T21:47:35.473350+0000","mapping_epoch":437,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":438,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:45:31.690656+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:37.512018+0000","last_clean_scrub_stamp":"2024-03-11T21:45:31.690656+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,2],"acting":[6,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.1c","version":"456'39","reported_seq":1018,"reported_epoch":488,"state":"active+clean","last_fresh":"2024-03-11T21:47:35.478960+0000","last_change":"2024-03-11T21:45:25.649612+0000","last_active":"2024-03-11T21:47:35.478960+0000","last_peered":"2024-03-11T21:47:35.478960+0000","last_clean":"2024-03-11T21:47:35.478960+0000","last_became_active":"2024-03-11T21:44:21.465719+0000","last_became_peered":"2024-03-11T21:44:21.465719+0000","last_unstale":"2024-03-11T21:47:35.478960+0000","last_undegraded":"2024-03-11T21:47:35.478960+0000","last_fullsized":"2024-03-11T21:47:35.478960+0000","mapping_epoch":432,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":433,"parent":"0.0","parent_split_bits":0,"last_scrub":"260'27","last_scrub_stamp":"2024-03-11T21:45:25.649486+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:39.525554+0000","last_clean_scrub_stamp":"2024-03-11T21:45:25.649486+0000","objects_scrubbed":7,"log_size":39,"log_dups_size":0,"ondisk_log_size":39,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:47:38.778024+0000","scrub_duration":36,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":62,"num_read_kb":53,"num_write":40,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,0],"acting":[2,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.1d","version":"0'0","reported_seq":902,"reported_epoch":488,"state":"active+clean","last_fresh":"2024-03-11T21:47:35.485227+0000","last_change":"2024-03-11T21:47:20.279983+0000","last_active":"2024-03-11T21:47:35.485227+0000","last_peered":"2024-03-11T21:47:35.485227+0000","last_clean":"2024-03-11T21:47:35.485227+0000","last_became_active":"2024-03-11T21:47:20.277652+0000","last_became_peered":"2024-03-11T21:47:20.277652+0000","last_unstale":"2024-03-11T21:47:35.485227+0000","last_undegraded":"2024-03-11T21:47:35.485227+0000","last_fullsized":"2024-03-11T21:47:35.485227+0000","mapping_epoch":478,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":479,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:45:23.007509+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:43.559452+0000","last_clean_scrub_stamp":"2024-03-11T21:45:23.007509+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:47:38.777784+0000","scrub_duration":106,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,6],"acting":[2,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.1e","version":"0'0","reported_seq":919,"reported_epoch":488,"state":"active+clean","last_fresh":"2024-03-11T21:47:35.473202+0000","last_change":"2024-03-11T21:46:15.490825+0000","last_active":"2024-03-11T21:47:35.473202+0000","last_peered":"2024-03-11T21:47:35.473202+0000","last_clean":"2024-03-11T21:47:35.473202+0000","last_became_active":"2024-03-11T21:25:42.582959+0000","last_became_peered":"2024-03-11T21:25:42.582959+0000","last_unstale":"2024-03-11T21:47:35.473202+0000","last_undegraded":"2024-03-11T21:47:35.473202+0000","last_fullsized":"2024-03-11T21:47:35.473202+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:46:15.490533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:44:28.222102+0000","last_clean_scrub_stamp":"2024-03-11T21:46:15.490533+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:47:40.547908+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.19","version":"67'2","reported_seq":201,"reported_epoch":488,"state":"active+clean","last_fresh":"2024-03-11T21:47:35.473365+0000","last_change":"2024-03-11T21:44:30.400531+0000","last_active":"2024-03-11T21:47:35.473365+0000","last_peered":"2024-03-11T21:47:35.473365+0000","last_clean":"2024-03-11T21:47:35.473365+0000","last_became_active":"2024-03-11T21:44:21.454547+0000","last_became_peered":"2024-03-11T21:44:21.454547+0000","last_unstale":"2024-03-11T21:47:35.473365+0000","last_undegraded":"2024-03-11T21:47:35.473365+0000","last_fullsized":"2024-03-11T21:47:35.473365+0000","mapping_epoch":432,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":433,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'2","last_scrub_stamp":"2024-03-11T21:44:30.400408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:37.512018+0000","last_clean_scrub_stamp":"2024-03-11T21:44:30.400408+0000","objects_scrubbed":2,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":957,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":957,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,2],"acting":[6,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.1f","version":"456'32","reported_seq":1023,"reported_epoch":488,"state":"active+clean","last_fresh":"2024-03-11T21:47:35.474876+0000","last_change":"2024-03-11T21:46:53.301032+0000","last_active":"2024-03-11T21:47:35.474876+0000","last_peered":"2024-03-11T21:47:35.474876+0000","last_clean":"2024-03-11T21:47:35.474876+0000","last_became_active":"2024-03-11T21:46:53.264621+0000","last_became_peered":"2024-03-11T21:46:53.264621+0000","last_unstale":"2024-03-11T21:47:35.474876+0000","last_undegraded":"2024-03-11T21:47:35.474876+0000","last_fullsized":"2024-03-11T21:47:35.474876+0000","mapping_epoch":473,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":474,"parent":"0.0","parent_split_bits":0,"last_scrub":"456'32","last_scrub_stamp":"2024-03-11T21:46:16.737252+0000","last_deep_scrub":"124'16","last_deep_scrub_stamp":"2024-03-11T21:35:38.173841+0000","last_clean_scrub_stamp":"2024-03-11T21:46:16.737252+0000","objects_scrubbed":5,"log_size":32,"log_dups_size":0,"ondisk_log_size":32,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":44,"num_read_kb":37,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,0],"acting":[6,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.1c","version":"467'2","reported_seq":577,"reported_epoch":488,"state":"active+clean","last_fresh":"2024-03-11T21:47:35.479000+0000","last_change":"2024-03-11T21:46:53.285542+0000","last_active":"2024-03-11T21:47:35.479000+0000","last_peered":"2024-03-11T21:47:35.479000+0000","last_clean":"2024-03-11T21:47:35.479000+0000","last_became_active":"2024-03-11T21:46:53.262630+0000","last_became_peered":"2024-03-11T21:46:53.262630+0000","last_unstale":"2024-03-11T21:47:35.479000+0000","last_undegraded":"2024-03-11T21:47:35.479000+0000","last_fullsized":"2024-03-11T21:47:35.479000+0000","mapping_epoch":473,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":474,"parent":"0.0","parent_split_bits":0,"last_scrub":"467'2","last_scrub_stamp":"2024-03-11T21:46:22.517102+0000","last_deep_scrub":"467'2","last_deep_scrub_stamp":"2024-03-11T21:46:22.517102+0000","last_clean_scrub_stamp":"2024-03-11T21:46:22.517102+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:47:50.265905+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":370,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":740,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,6],"acting":[7,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.1f","version":"438'11","reported_seq":1241,"reported_epoch":488,"state":"active+clean","last_fresh":"2024-03-11T21:47:35.474769+0000","last_change":"2024-03-11T21:44:35.663917+0000","last_active":"2024-03-11T21:47:35.474769+0000","last_peered":"2024-03-11T21:47:35.474769+0000","last_clean":"2024-03-11T21:47:35.474769+0000","last_became_active":"2024-03-11T21:44:35.647069+0000","last_became_peered":"2024-03-11T21:44:35.647069+0000","last_unstale":"2024-03-11T21:47:35.474769+0000","last_undegraded":"2024-03-11T21:47:35.474769+0000","last_fullsized":"2024-03-11T21:47:35.474769+0000","mapping_epoch":437,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":438,"parent":"0.0","parent_split_bits":0,"last_scrub":"229'10","last_scrub_stamp":"2024-03-11T21:44:32.375857+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:41.544550+0000","last_clean_scrub_stamp":"2024-03-11T21:44:32.375857+0000","objects_scrubbed":1,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,2],"acting":[6,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.18","version":"67'1","reported_seq":485,"reported_epoch":488,"state":"active+clean","last_fresh":"2024-03-11T21:47:35.480080+0000","last_change":"2024-03-11T21:46:53.287167+0000","last_active":"2024-03-11T21:47:35.480080+0000","last_peered":"2024-03-11T21:47:35.480080+0000","last_clean":"2024-03-11T21:47:35.480080+0000","last_became_active":"2024-03-11T21:46:53.275264+0000","last_became_peered":"2024-03-11T21:46:53.275264+0000","last_unstale":"2024-03-11T21:47:35.480080+0000","last_undegraded":"2024-03-11T21:47:35.480080+0000","last_fullsized":"2024-03-11T21:47:35.480080+0000","mapping_epoch":473,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":474,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'1","last_scrub_stamp":"2024-03-11T21:45:41.138228+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:37.512018+0000","last_clean_scrub_stamp":"2024-03-11T21:45:41.138228+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:47:38.778149+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":436,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":7,"num_read_kb":7,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":1308,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6,1],"acting":[2,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.1e","version":"456'30","reported_seq":1036,"reported_epoch":488,"state":"active+clean","last_fresh":"2024-03-11T21:47:35.473115+0000","last_change":"2024-03-11T21:45:15.025938+0000","last_active":"2024-03-11T21:47:35.473115+0000","last_peered":"2024-03-11T21:47:35.473115+0000","last_clean":"2024-03-11T21:47:35.473115+0000","last_became_active":"2024-03-11T21:44:21.467126+0000","last_became_peered":"2024-03-11T21:44:21.467126+0000","last_unstale":"2024-03-11T21:47:35.473115+0000","last_undegraded":"2024-03-11T21:47:35.473115+0000","last_fullsized":"2024-03-11T21:47:35.473115+0000","mapping_epoch":432,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":433,"parent":"0.0","parent_split_bits":0,"last_scrub":"260'20","last_scrub_stamp":"2024-03-11T21:45:15.025805+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:39.525554+0000","last_clean_scrub_stamp":"2024-03-11T21:45:15.025805+0000","objects_scrubbed":5,"log_size":30,"log_dups_size":0,"ondisk_log_size":30,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":45,"num_read_kb":40,"num_write":30,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,1],"acting":[0,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.13","version":"0'0","reported_seq":471,"reported_epoch":488,"state":"active+clean","last_fresh":"2024-03-11T21:47:35.476648+0000","last_change":"2024-03-11T21:45:17.252669+0000","last_active":"2024-03-11T21:47:35.476648+0000","last_peered":"2024-03-11T21:47:35.476648+0000","last_clean":"2024-03-11T21:47:35.476648+0000","last_became_active":"2024-03-11T21:44:21.465436+0000","last_became_peered":"2024-03-11T21:44:21.465436+0000","last_unstale":"2024-03-11T21:47:35.476648+0000","last_undegraded":"2024-03-11T21:47:35.476648+0000","last_fullsized":"2024-03-11T21:47:35.476648+0000","mapping_epoch":432,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":433,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:45:17.252520+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:43.559452+0000","last_clean_scrub_stamp":"2024-03-11T21:45:17.252520+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":220,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,2],"acting":[0,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":576,"reported_epoch":488,"state":"active+clean","last_fresh":"2024-03-11T21:47:35.476799+0000","last_change":"2024-03-11T21:46:24.563883+0000","last_active":"2024-03-11T21:47:35.476799+0000","last_peered":"2024-03-11T21:47:35.476799+0000","last_clean":"2024-03-11T21:47:35.476799+0000","last_became_active":"2024-03-11T21:44:35.627685+0000","last_became_peered":"2024-03-11T21:44:35.627685+0000","last_unstale":"2024-03-11T21:47:35.476799+0000","last_undegraded":"2024-03-11T21:47:35.476799+0000","last_fullsized":"2024-03-11T21:47:35.476799+0000","mapping_epoch":437,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":438,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:46:24.563752+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:43:26.665719+0000","last_clean_scrub_stamp":"2024-03-11T21:46:24.563752+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,0],"acting":[7,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.17","version":"0'0","reported_seq":940,"reported_epoch":488,"state":"active+clean","last_fresh":"2024-03-11T21:47:35.474425+0000","last_change":"2024-03-11T21:46:53.264128+0000","last_active":"2024-03-11T21:47:35.474425+0000","last_peered":"2024-03-11T21:47:35.474425+0000","last_clean":"2024-03-11T21:47:35.474425+0000","last_became_active":"2024-03-11T21:46:53.263213+0000","last_became_peered":"2024-03-11T21:46:53.263213+0000","last_unstale":"2024-03-11T21:47:35.474425+0000","last_undegraded":"2024-03-11T21:47:35.474425+0000","last_fullsized":"2024-03-11T21:47:35.474425+0000","mapping_epoch":473,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":474,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:45:30.740865+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:45:30.740865+0000","last_clean_scrub_stamp":"2024-03-11T21:45:30.740865+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,1],"acting":[6,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.11","version":"456'27","reported_seq":626,"reported_epoch":488,"state":"active+clean","last_fresh":"2024-03-11T21:47:35.477776+0000","last_change":"2024-03-11T21:46:53.287683+0000","last_active":"2024-03-11T21:47:35.477776+0000","last_peered":"2024-03-11T21:47:35.477776+0000","last_clean":"2024-03-11T21:47:35.477776+0000","last_became_active":"2024-03-11T21:46:53.265875+0000","last_became_peered":"2024-03-11T21:46:53.265875+0000","last_unstale":"2024-03-11T21:47:35.477776+0000","last_undegraded":"2024-03-11T21:47:35.477776+0000","last_fullsized":"2024-03-11T21:47:35.477776+0000","mapping_epoch":473,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":474,"parent":"0.0","parent_split_bits":0,"last_scrub":"456'27","last_scrub_stamp":"2024-03-11T21:46:18.485911+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:39.525554+0000","last_clean_scrub_stamp":"2024-03-11T21:46:18.485911+0000","objects_scrubbed":5,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:47:44.260791+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":44,"num_read_kb":37,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,0],"acting":[7,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":568,"reported_epoch":488,"state":"active+clean","last_fresh":"2024-03-11T21:47:35.477670+0000","last_change":"2024-03-11T21:44:40.464549+0000","last_active":"2024-03-11T21:47:35.477670+0000","last_peered":"2024-03-11T21:47:35.477670+0000","last_clean":"2024-03-11T21:47:35.477670+0000","last_became_active":"2024-03-11T21:44:35.634181+0000","last_became_peered":"2024-03-11T21:44:35.634181+0000","last_unstale":"2024-03-11T21:47:35.477670+0000","last_undegraded":"2024-03-11T21:47:35.477670+0000","last_fullsized":"2024-03-11T21:47:35.477670+0000","mapping_epoch":437,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":438,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:44:40.464467+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:43.559452+0000","last_clean_scrub_stamp":"2024-03-11T21:44:40.464467+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,1],"acting":[7,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":952,"reported_epoch":488,"state":"active+clean","last_fresh":"2024-03-11T21:47:35.473503+0000","last_change":"2024-03-11T21:45:39.613711+0000","last_active":"2024-03-11T21:47:35.473503+0000","last_peered":"2024-03-11T21:47:35.473503+0000","last_clean":"2024-03-11T21:47:35.473503+0000","last_became_active":"2024-03-11T21:44:35.630991+0000","last_became_peered":"2024-03-11T21:44:35.630991+0000","last_unstale":"2024-03-11T21:47:35.473503+0000","last_undegraded":"2024-03-11T21:47:35.473503+0000","last_fullsized":"2024-03-11T21:47:35.473503+0000","mapping_epoch":437,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":438,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:45:39.613600+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:43:29.425414+0000","last_clean_scrub_stamp":"2024-03-11T21:45:39.613600+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,0],"acting":[6,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.16","version":"0'0","reported_seq":40,"reported_epoch":488,"state":"active+clean","last_fresh":"2024-03-11T21:47:35.474229+0000","last_change":"2024-03-11T21:46:53.264965+0000","last_active":"2024-03-11T21:47:35.474229+0000","last_peered":"2024-03-11T21:47:35.474229+0000","last_clean":"2024-03-11T21:47:35.474229+0000","last_became_active":"2024-03-11T21:46:53.264119+0000","last_became_peered":"2024-03-11T21:46:53.264119+0000","last_unstale":"2024-03-11T21:47:35.474229+0000","last_undegraded":"2024-03-11T21:47:35.474229+0000","last_fullsized":"2024-03-11T21:47:35.474229+0000","mapping_epoch":473,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":474,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:45:24.124964+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:27:11.938909+0000","last_clean_scrub_stamp":"2024-03-11T21:45:24.124964+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,1],"acting":[6,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.10","version":"456'14","reported_seq":248,"reported_epoch":488,"state":"active+clean","last_fresh":"2024-03-11T21:47:35.474332+0000","last_change":"2024-03-11T21:47:20.279333+0000","last_active":"2024-03-11T21:47:35.474332+0000","last_peered":"2024-03-11T21:47:35.474332+0000","last_clean":"2024-03-11T21:47:35.474332+0000","last_became_active":"2024-03-11T21:47:20.278961+0000","last_became_peered":"2024-03-11T21:47:20.278961+0000","last_unstale":"2024-03-11T21:47:35.474332+0000","last_undegraded":"2024-03-11T21:47:35.474332+0000","last_fullsized":"2024-03-11T21:47:35.474332+0000","mapping_epoch":478,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":479,"parent":"0.0","parent_split_bits":0,"last_scrub":"456'14","last_scrub_stamp":"2024-03-11T21:47:18.601648+0000","last_deep_scrub":"260'10","last_deep_scrub_stamp":"2024-03-11T21:44:32.059058+0000","last_clean_scrub_stamp":"2024-03-11T21:47:18.601648+0000","objects_scrubbed":2,"log_size":14,"log_dups_size":0,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:48:32.895813+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":17,"num_read_kb":15,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,2],"acting":[6,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":530,"reported_epoch":488,"state":"active+clean","last_fresh":"2024-03-11T21:47:35.469691+0000","last_change":"2024-03-11T21:46:53.268726+0000","last_active":"2024-03-11T21:47:35.469691+0000","last_peered":"2024-03-11T21:47:35.469691+0000","last_clean":"2024-03-11T21:47:35.469691+0000","last_became_active":"2024-03-11T21:46:53.268378+0000","last_became_peered":"2024-03-11T21:46:53.268378+0000","last_unstale":"2024-03-11T21:47:35.469691+0000","last_undegraded":"2024-03-11T21:47:35.469691+0000","last_fullsized":"2024-03-11T21:47:35.469691+0000","mapping_epoch":473,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":474,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:45:36.298635+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:43.559452+0000","last_clean_scrub_stamp":"2024-03-11T21:45:36.298635+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":110,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,6],"acting":[0,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":923,"reported_epoch":488,"state":"active+clean","last_fresh":"2024-03-11T21:47:35.478456+0000","last_change":"2024-03-11T21:46:53.265737+0000","last_active":"2024-03-11T21:47:35.478456+0000","last_peered":"2024-03-11T21:47:35.478456+0000","last_clean":"2024-03-11T21:47:35.478456+0000","last_became_active":"2024-03-11T21:46:53.264999+0000","last_became_peered":"2024-03-11T21:46:53.264999+0000","last_unstale":"2024-03-11T21:47:35.478456+0000","last_undegraded":"2024-03-11T21:47:35.478456+0000","last_fullsized":"2024-03-11T21:47:35.478456+0000","mapping_epoch":473,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":474,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:45:13.788601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:27:21.207303+0000","last_clean_scrub_stamp":"2024-03-11T21:45:13.788601+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,6],"acting":[1,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.15","version":"0'0","reported_seq":924,"reported_epoch":488,"state":"active+clean","last_fresh":"2024-03-11T21:47:35.489639+0000","last_change":"2024-03-11T21:46:53.261948+0000","last_active":"2024-03-11T21:47:35.489639+0000","last_peered":"2024-03-11T21:47:35.489639+0000","last_clean":"2024-03-11T21:47:35.489639+0000","last_became_active":"2024-03-11T21:46:53.260232+0000","last_became_peered":"2024-03-11T21:46:53.260232+0000","last_unstale":"2024-03-11T21:47:35.489639+0000","last_undegraded":"2024-03-11T21:47:35.489639+0000","last_fullsized":"2024-03-11T21:47:35.489639+0000","mapping_epoch":473,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":474,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:45:41.898237+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:26:44.860499+0000","last_clean_scrub_stamp":"2024-03-11T21:45:41.898237+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,6],"acting":[1,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.13","version":"456'32","reported_seq":652,"reported_epoch":488,"state":"active+clean","last_fresh":"2024-03-11T21:47:35.475681+0000","last_change":"2024-03-11T21:46:53.309085+0000","last_active":"2024-03-11T21:47:35.475681+0000","last_peered":"2024-03-11T21:47:35.475681+0000","last_clean":"2024-03-11T21:47:35.475681+0000","last_became_active":"2024-03-11T21:46:53.273931+0000","last_became_peered":"2024-03-11T21:46:53.273931+0000","last_unstale":"2024-03-11T21:47:35.475681+0000","last_undegraded":"2024-03-11T21:47:35.475681+0000","last_fullsized":"2024-03-11T21:47:35.475681+0000","mapping_epoch":473,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":474,"parent":"0.0","parent_split_bits":0,"last_scrub":"456'32","last_scrub_stamp":"2024-03-11T21:46:13.717488+0000","last_deep_scrub":"260'24","last_deep_scrub_stamp":"2024-03-11T21:40:25.595220+0000","last_clean_scrub_stamp":"2024-03-11T21:46:13.717488+0000","objects_scrubbed":5,"log_size":32,"log_dups_size":0,"ondisk_log_size":32,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":43,"num_read_kb":36,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,2],"acting":[6,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.10","version":"0'0","reported_seq":912,"reported_epoch":488,"state":"active+clean","last_fresh":"2024-03-11T21:47:35.473805+0000","last_change":"2024-03-11T21:46:53.264915+0000","last_active":"2024-03-11T21:47:35.473805+0000","last_peered":"2024-03-11T21:47:35.473805+0000","last_clean":"2024-03-11T21:47:35.473805+0000","last_became_active":"2024-03-11T21:46:53.264161+0000","last_became_peered":"2024-03-11T21:46:53.264161+0000","last_unstale":"2024-03-11T21:47:35.473805+0000","last_undegraded":"2024-03-11T21:47:35.473805+0000","last_fullsized":"2024-03-11T21:47:35.473805+0000","mapping_epoch":473,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":474,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:45:35.179951+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:27:25.417450+0000","last_clean_scrub_stamp":"2024-03-11T21:45:35.179951+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,6],"acting":[0,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":528,"reported_epoch":488,"state":"active+clean","last_fresh":"2024-03-11T21:47:35.476740+0000","last_change":"2024-03-11T21:46:53.275057+0000","last_active":"2024-03-11T21:47:35.476740+0000","last_peered":"2024-03-11T21:47:35.476740+0000","last_clean":"2024-03-11T21:47:35.476740+0000","last_became_active":"2024-03-11T21:46:53.274118+0000","last_became_peered":"2024-03-11T21:46:53.274118+0000","last_unstale":"2024-03-11T21:47:35.476740+0000","last_undegraded":"2024-03-11T21:47:35.476740+0000","last_fullsized":"2024-03-11T21:47:35.476740+0000","mapping_epoch":473,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":474,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:46:24.352079+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:31:15.367898+0000","last_clean_scrub_stamp":"2024-03-11T21:46:24.352079+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,7],"acting":[0,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.14","version":"0'0","reported_seq":963,"reported_epoch":488,"state":"active+clean","last_fresh":"2024-03-11T21:47:35.473700+0000","last_change":"2024-03-11T21:46:53.266115+0000","last_active":"2024-03-11T21:47:35.473700+0000","last_peered":"2024-03-11T21:47:35.473700+0000","last_clean":"2024-03-11T21:47:35.473700+0000","last_became_active":"2024-03-11T21:46:53.264162+0000","last_became_peered":"2024-03-11T21:46:53.264162+0000","last_unstale":"2024-03-11T21:47:35.473700+0000","last_undegraded":"2024-03-11T21:47:35.473700+0000","last_fullsized":"2024-03-11T21:47:35.473700+0000","mapping_epoch":473,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":474,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:44:34.456276+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:37.512018+0000","last_clean_scrub_stamp":"2024-03-11T21:44:34.456276+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,2],"acting":[6,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.12","version":"456'21","reported_seq":948,"reported_epoch":488,"state":"active+clean","last_fresh":"2024-03-11T21:47:35.478278+0000","last_change":"2024-03-11T21:45:23.938336+0000","last_active":"2024-03-11T21:47:35.478278+0000","last_peered":"2024-03-11T21:47:35.478278+0000","last_clean":"2024-03-11T21:47:35.478278+0000","last_became_active":"2024-03-11T21:25:40.547082+0000","last_became_peered":"2024-03-11T21:25:40.547082+0000","last_unstale":"2024-03-11T21:47:35.478278+0000","last_undegraded":"2024-03-11T21:47:35.478278+0000","last_fullsized":"2024-03-11T21:47:35.478278+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"260'15","last_scrub_stamp":"2024-03-11T21:45:23.938207+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:39.525554+0000","last_clean_scrub_stamp":"2024-03-11T21:45:23.938207+0000","objects_scrubbed":4,"log_size":21,"log_dups_size":0,"ondisk_log_size":21,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":35,"num_read_kb":29,"num_write":22,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":127,"reported_epoch":488,"state":"active+clean","last_fresh":"2024-03-11T21:47:35.479394+0000","last_change":"2024-03-11T21:46:53.270649+0000","last_active":"2024-03-11T21:47:35.479394+0000","last_peered":"2024-03-11T21:47:35.479394+0000","last_clean":"2024-03-11T21:47:35.479394+0000","last_became_active":"2024-03-11T21:46:53.269901+0000","last_became_peered":"2024-03-11T21:46:53.269901+0000","last_unstale":"2024-03-11T21:47:35.479394+0000","last_undegraded":"2024-03-11T21:47:35.479394+0000","last_fullsized":"2024-03-11T21:47:35.479394+0000","mapping_epoch":473,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":474,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:44:39.546881+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:27:09.801138+0000","last_clean_scrub_stamp":"2024-03-11T21:44:39.546881+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:47:38.775636+0000","scrub_duration":112,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,7],"acting":[2,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.14","version":"434'13","reported_seq":715,"reported_epoch":488,"state":"active+clean","last_fresh":"2024-03-11T21:47:35.478429+0000","last_change":"2024-03-11T21:44:32.424885+0000","last_active":"2024-03-11T21:47:35.478429+0000","last_peered":"2024-03-11T21:47:35.478429+0000","last_clean":"2024-03-11T21:47:35.478429+0000","last_became_active":"2024-03-11T21:44:21.465719+0000","last_became_peered":"2024-03-11T21:44:21.465719+0000","last_unstale":"2024-03-11T21:47:35.478429+0000","last_undegraded":"2024-03-11T21:47:35.478429+0000","last_fullsized":"2024-03-11T21:47:35.478429+0000","mapping_epoch":432,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":433,"parent":"0.0","parent_split_bits":0,"last_scrub":"434'13","last_scrub_stamp":"2024-03-11T21:44:32.424808+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:41.544550+0000","last_clean_scrub_stamp":"2024-03-11T21:44:32.424808+0000","objects_scrubbed":1,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,1],"acting":[7,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.13","version":"0'0","reported_seq":945,"reported_epoch":488,"state":"active+clean","last_fresh":"2024-03-11T21:47:35.477836+0000","last_change":"2024-03-11T21:46:53.260824+0000","last_active":"2024-03-11T21:47:35.477836+0000","last_peered":"2024-03-11T21:47:35.477836+0000","last_clean":"2024-03-11T21:47:35.477836+0000","last_became_active":"2024-03-11T21:46:53.259657+0000","last_became_peered":"2024-03-11T21:46:53.259657+0000","last_unstale":"2024-03-11T21:47:35.477836+0000","last_undegraded":"2024-03-11T21:47:35.477836+0000","last_fullsized":"2024-03-11T21:47:35.477836+0000","mapping_epoch":473,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":474,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:46:05.444630+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:37.512018+0000","last_clean_scrub_stamp":"2024-03-11T21:46:05.444630+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.15","version":"456'25","reported_seq":1000,"reported_epoch":488,"state":"active+clean","last_fresh":"2024-03-11T21:47:35.479288+0000","last_change":"2024-03-11T21:46:53.284572+0000","last_active":"2024-03-11T21:47:35.479288+0000","last_peered":"2024-03-11T21:47:35.479288+0000","last_clean":"2024-03-11T21:47:35.479288+0000","last_became_active":"2024-03-11T21:46:53.263727+0000","last_became_peered":"2024-03-11T21:46:53.263727+0000","last_unstale":"2024-03-11T21:47:35.479288+0000","last_undegraded":"2024-03-11T21:47:35.479288+0000","last_fullsized":"2024-03-11T21:47:35.479288+0000","mapping_epoch":473,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":474,"parent":"0.0","parent_split_bits":0,"last_scrub":"260'19","last_scrub_stamp":"2024-03-11T21:45:22.104952+0000","last_deep_scrub":"94'13","last_deep_scrub_stamp":"2024-03-11T21:29:41.853158+0000","last_clean_scrub_stamp":"2024-03-11T21:45:22.104952+0000","objects_scrubbed":4,"log_size":25,"log_dups_size":0,"ondisk_log_size":25,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":34,"num_read_kb":28,"num_write":22,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19,"num_bytes_recovered":440,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,0],"acting":[7,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.16","version":"0'0","reported_seq":953,"reported_epoch":488,"state":"active+clean","last_fresh":"2024-03-11T21:47:35.470524+0000","last_change":"2024-03-11T21:46:53.264810+0000","last_active":"2024-03-11T21:47:35.470524+0000","last_peered":"2024-03-11T21:47:35.470524+0000","last_clean":"2024-03-11T21:47:35.470524+0000","last_became_active":"2024-03-11T21:46:53.264288+0000","last_became_peered":"2024-03-11T21:46:53.264288+0000","last_unstale":"2024-03-11T21:47:35.470524+0000","last_undegraded":"2024-03-11T21:47:35.470524+0000","last_fullsized":"2024-03-11T21:47:35.470524+0000","mapping_epoch":473,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":474,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:46:18.525893+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:40:30.249679+0000","last_clean_scrub_stamp":"2024-03-11T21:46:18.525893+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":105,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,1],"acting":[0,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.15","version":"479'11","reported_seq":135,"reported_epoch":488,"state":"active+clean","last_fresh":"2024-03-11T21:47:35.472920+0000","last_change":"2024-03-11T21:47:20.276023+0000","last_active":"2024-03-11T21:47:35.472920+0000","last_peered":"2024-03-11T21:47:35.472920+0000","last_clean":"2024-03-11T21:47:35.472920+0000","last_became_active":"2024-03-11T21:47:20.275673+0000","last_became_peered":"2024-03-11T21:47:20.275673+0000","last_unstale":"2024-03-11T21:47:35.472920+0000","last_undegraded":"2024-03-11T21:47:35.472920+0000","last_fullsized":"2024-03-11T21:47:35.472920+0000","mapping_epoch":478,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":479,"parent":"0.0","parent_split_bits":0,"last_scrub":"98'9","last_scrub_stamp":"2024-03-11T21:46:08.945626+0000","last_deep_scrub":"98'9","last_deep_scrub_stamp":"2024-03-11T21:31:20.624923+0000","last_clean_scrub_stamp":"2024-03-11T21:46:08.945626+0000","objects_scrubbed":1,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,7],"acting":[0,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.12","version":"0'0","reported_seq":855,"reported_epoch":488,"state":"active+clean","last_fresh":"2024-03-11T21:47:35.477653+0000","last_change":"2024-03-11T21:46:53.268426+0000","last_active":"2024-03-11T21:47:35.477653+0000","last_peered":"2024-03-11T21:47:35.477653+0000","last_clean":"2024-03-11T21:47:35.477653+0000","last_became_active":"2024-03-11T21:46:53.263636+0000","last_became_peered":"2024-03-11T21:46:53.263636+0000","last_unstale":"2024-03-11T21:47:35.477653+0000","last_undegraded":"2024-03-11T21:47:35.477653+0000","last_fullsized":"2024-03-11T21:47:35.477653+0000","mapping_epoch":473,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":474,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:45:23.087568+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:37.512018+0000","last_clean_scrub_stamp":"2024-03-11T21:45:23.087568+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.14","version":"456'30","reported_seq":224,"reported_epoch":488,"state":"active+clean","last_fresh":"2024-03-11T21:47:35.478529+0000","last_change":"2024-03-11T21:47:20.278549+0000","last_active":"2024-03-11T21:47:35.478529+0000","last_peered":"2024-03-11T21:47:35.478529+0000","last_clean":"2024-03-11T21:47:35.478529+0000","last_became_active":"2024-03-11T21:47:20.278192+0000","last_became_peered":"2024-03-11T21:47:20.278192+0000","last_unstale":"2024-03-11T21:47:35.478529+0000","last_undegraded":"2024-03-11T21:47:35.478529+0000","last_fullsized":"2024-03-11T21:47:35.478529+0000","mapping_epoch":478,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":479,"parent":"0.0","parent_split_bits":0,"last_scrub":"456'30","last_scrub_stamp":"2024-03-11T21:47:17.566980+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:39.525554+0000","last_clean_scrub_stamp":"2024-03-11T21:47:17.566980+0000","objects_scrubbed":5,"log_size":30,"log_dups_size":0,"ondisk_log_size":30,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:48:29.675162+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":43,"num_read_kb":38,"num_write":30,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,2],"acting":[7,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.15","version":"0'0","reported_seq":574,"reported_epoch":488,"state":"active+clean","last_fresh":"2024-03-11T21:47:35.479747+0000","last_change":"2024-03-11T21:46:53.265030+0000","last_active":"2024-03-11T21:47:35.479747+0000","last_peered":"2024-03-11T21:47:35.479747+0000","last_clean":"2024-03-11T21:47:35.479747+0000","last_became_active":"2024-03-11T21:46:53.264524+0000","last_became_peered":"2024-03-11T21:46:53.264524+0000","last_unstale":"2024-03-11T21:47:35.479747+0000","last_undegraded":"2024-03-11T21:47:35.479747+0000","last_fullsized":"2024-03-11T21:47:35.479747+0000","mapping_epoch":473,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":474,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:46:07.457807+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:43.559452+0000","last_clean_scrub_stamp":"2024-03-11T21:46:07.457807+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":30,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,0],"acting":[7,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":827,"reported_epoch":488,"state":"active+clean","last_fresh":"2024-03-11T21:47:35.474005+0000","last_change":"2024-03-11T21:47:20.274338+0000","last_active":"2024-03-11T21:47:35.474005+0000","last_peered":"2024-03-11T21:47:35.474005+0000","last_clean":"2024-03-11T21:47:35.474005+0000","last_became_active":"2024-03-11T21:47:20.274009+0000","last_became_peered":"2024-03-11T21:47:20.274009+0000","last_unstale":"2024-03-11T21:47:35.474005+0000","last_undegraded":"2024-03-11T21:47:35.474005+0000","last_fullsized":"2024-03-11T21:47:35.474005+0000","mapping_epoch":478,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":479,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:46:11.892240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:43:36.585632+0000","last_clean_scrub_stamp":"2024-03-11T21:46:11.892240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,7],"acting":[6,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.11","version":"0'0","reported_seq":946,"reported_epoch":488,"state":"active+clean","last_fresh":"2024-03-11T21:47:35.473673+0000","last_change":"2024-03-11T21:46:53.270053+0000","last_active":"2024-03-11T21:47:35.473673+0000","last_peered":"2024-03-11T21:47:35.473673+0000","last_clean":"2024-03-11T21:47:35.473673+0000","last_became_active":"2024-03-11T21:46:53.269290+0000","last_became_peered":"2024-03-11T21:46:53.269290+0000","last_unstale":"2024-03-11T21:47:35.473673+0000","last_undegraded":"2024-03-11T21:47:35.473673+0000","last_fullsized":"2024-03-11T21:47:35.473673+0000","mapping_epoch":473,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":474,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:46:06.705444+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:37.512018+0000","last_clean_scrub_stamp":"2024-03-11T21:46:06.705444+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,0],"acting":[6,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.17","version":"456'21","reported_seq":615,"reported_epoch":488,"state":"active+clean","last_fresh":"2024-03-11T21:47:35.478428+0000","last_change":"2024-03-11T21:46:53.303009+0000","last_active":"2024-03-11T21:47:35.478428+0000","last_peered":"2024-03-11T21:47:35.478428+0000","last_clean":"2024-03-11T21:47:35.478428+0000","last_became_active":"2024-03-11T21:46:53.270498+0000","last_became_peered":"2024-03-11T21:46:53.270498+0000","last_unstale":"2024-03-11T21:47:35.478428+0000","last_undegraded":"2024-03-11T21:47:35.478428+0000","last_fullsized":"2024-03-11T21:47:35.478428+0000","mapping_epoch":473,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":474,"parent":"0.0","parent_split_bits":0,"last_scrub":"260'15","last_scrub_stamp":"2024-03-11T21:44:39.501274+0000","last_deep_scrub":"97'9","last_deep_scrub_stamp":"2024-03-11T21:27:25.803132+0000","last_clean_scrub_stamp":"2024-03-11T21:44:39.501274+0000","objects_scrubbed":3,"log_size":21,"log_dups_size":0,"ondisk_log_size":21,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":27,"num_read_kb":24,"num_write":18,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,2],"acting":[7,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.14","version":"0'0","reported_seq":930,"reported_epoch":488,"state":"active+clean","last_fresh":"2024-03-11T21:47:35.478479+0000","last_change":"2024-03-11T21:45:41.530744+0000","last_active":"2024-03-11T21:47:35.478479+0000","last_peered":"2024-03-11T21:47:35.478479+0000","last_clean":"2024-03-11T21:47:35.478479+0000","last_became_active":"2024-03-11T21:44:35.626384+0000","last_became_peered":"2024-03-11T21:44:35.626384+0000","last_unstale":"2024-03-11T21:47:35.478479+0000","last_undegraded":"2024-03-11T21:47:35.478479+0000","last_fullsized":"2024-03-11T21:47:35.478479+0000","mapping_epoch":437,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":438,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:45:41.530627+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:43.559452+0000","last_clean_scrub_stamp":"2024-03-11T21:45:41.530627+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:47:38.778177+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,0],"acting":[2,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":545,"reported_epoch":488,"state":"active+clean","last_fresh":"2024-03-11T21:47:35.482110+0000","last_change":"2024-03-11T21:45:36.877859+0000","last_active":"2024-03-11T21:47:35.482110+0000","last_peered":"2024-03-11T21:47:35.482110+0000","last_clean":"2024-03-11T21:47:35.482110+0000","last_became_active":"2024-03-11T21:44:21.458758+0000","last_became_peered":"2024-03-11T21:44:21.458758+0000","last_unstale":"2024-03-11T21:47:35.482110+0000","last_undegraded":"2024-03-11T21:47:35.482110+0000","last_fullsized":"2024-03-11T21:47:35.482110+0000","mapping_epoch":432,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":433,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:45:36.877741+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:45:36.877741+0000","last_clean_scrub_stamp":"2024-03-11T21:45:36.877741+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,0],"acting":[1,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.10","version":"0'0","reported_seq":924,"reported_epoch":488,"state":"active+clean","last_fresh":"2024-03-11T21:47:35.479814+0000","last_change":"2024-03-11T21:45:27.664632+0000","last_active":"2024-03-11T21:47:35.479814+0000","last_peered":"2024-03-11T21:47:35.479814+0000","last_clean":"2024-03-11T21:47:35.479814+0000","last_became_active":"2024-03-11T21:25:38.533138+0000","last_became_peered":"2024-03-11T21:25:38.533138+0000","last_unstale":"2024-03-11T21:47:35.479814+0000","last_undegraded":"2024-03-11T21:47:35.479814+0000","last_fullsized":"2024-03-11T21:47:35.479814+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:45:27.664509+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:37.512018+0000","last_clean_scrub_stamp":"2024-03-11T21:45:27.664509+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:47:38.777921+0000","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,0],"acting":[2,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.16","version":"456'25","reported_seq":1018,"reported_epoch":488,"state":"active+clean","last_fresh":"2024-03-11T21:47:35.470653+0000","last_change":"2024-03-11T21:45:25.093159+0000","last_active":"2024-03-11T21:47:35.470653+0000","last_peered":"2024-03-11T21:47:35.470653+0000","last_clean":"2024-03-11T21:47:35.470653+0000","last_became_active":"2024-03-11T21:44:21.463330+0000","last_became_peered":"2024-03-11T21:44:21.463330+0000","last_unstale":"2024-03-11T21:47:35.470653+0000","last_undegraded":"2024-03-11T21:47:35.470653+0000","last_fullsized":"2024-03-11T21:47:35.470653+0000","mapping_epoch":432,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":433,"parent":"0.0","parent_split_bits":0,"last_scrub":"369'19","last_scrub_stamp":"2024-03-11T21:45:25.093077+0000","last_deep_scrub":"369'19","last_deep_scrub_stamp":"2024-03-11T21:41:22.115634+0000","last_clean_scrub_stamp":"2024-03-11T21:45:25.093077+0000","objects_scrubbed":4,"log_size":25,"log_dups_size":0,"ondisk_log_size":25,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":35,"num_read_kb":29,"num_write":22,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.6","version":"488'13653","reported_seq":13747,"reported_epoch":488,"state":"active+clean","last_fresh":"2024-03-11T21:47:39.767125+0000","last_change":"2024-03-11T21:44:53.660589+0000","last_active":"2024-03-11T21:47:39.767125+0000","last_peered":"2024-03-11T21:47:39.767125+0000","last_clean":"2024-03-11T21:47:39.767125+0000","last_became_active":"2024-03-11T21:44:53.660330+0000","last_became_peered":"2024-03-11T21:44:53.660330+0000","last_unstale":"2024-03-11T21:47:39.767125+0000","last_undegraded":"2024-03-11T21:47:39.767125+0000","last_fullsized":"2024-03-11T21:47:39.767125+0000","mapping_epoch":446,"log_start":"472'9963","ondisk_log_start":"472'9963","created":446,"last_epoch_clean":447,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:44:52.644443+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:44:52.644443+0000","last_clean_scrub_stamp":"2024-03-11T21:44:52.644443+0000","objects_scrubbed":0,"log_size":3690,"log_dups_size":0,"ondisk_log_size":3690,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":250281984,"num_objects":3819,"num_object_clones":0,"num_object_copies":11457,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3819,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13653,"num_write_kb":559104,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.b","version":"0'0","reported_seq":533,"reported_epoch":488,"state":"active+clean","last_fresh":"2024-03-11T21:47:35.479095+0000","last_change":"2024-03-11T21:44:35.622604+0000","last_active":"2024-03-11T21:47:35.479095+0000","last_peered":"2024-03-11T21:47:35.479095+0000","last_clean":"2024-03-11T21:47:35.479095+0000","last_became_active":"2024-03-11T21:44:21.449857+0000","last_became_peered":"2024-03-11T21:44:21.449857+0000","last_unstale":"2024-03-11T21:47:35.479095+0000","last_undegraded":"2024-03-11T21:47:35.479095+0000","last_fullsized":"2024-03-11T21:47:35.479095+0000","mapping_epoch":432,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":433,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:44:35.622488+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:44:35.622488+0000","last_clean_scrub_stamp":"2024-03-11T21:44:35.622488+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":116,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,6],"acting":[7,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.f","version":"67'2","reported_seq":148,"reported_epoch":488,"state":"active+clean","last_fresh":"2024-03-11T21:47:35.470339+0000","last_change":"2024-03-11T21:46:53.283047+0000","last_active":"2024-03-11T21:47:35.470339+0000","last_peered":"2024-03-11T21:47:35.470339+0000","last_clean":"2024-03-11T21:47:35.470339+0000","last_became_active":"2024-03-11T21:46:53.267365+0000","last_became_peered":"2024-03-11T21:46:53.267365+0000","last_unstale":"2024-03-11T21:47:35.470339+0000","last_undegraded":"2024-03-11T21:47:35.470339+0000","last_fullsized":"2024-03-11T21:47:35.470339+0000","mapping_epoch":473,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":474,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'2","last_scrub_stamp":"2024-03-11T21:46:20.404697+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:37.512018+0000","last_clean_scrub_stamp":"2024-03-11T21:46:20.404697+0000","objects_scrubbed":2,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:47:46.711310+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":92,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":184,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,6],"acting":[0,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.9","version":"456'29","reported_seq":609,"reported_epoch":488,"state":"active+clean","last_fresh":"2024-03-11T21:47:35.478452+0000","last_change":"2024-03-11T21:46:28.006582+0000","last_active":"2024-03-11T21:47:35.478452+0000","last_peered":"2024-03-11T21:47:35.478452+0000","last_clean":"2024-03-11T21:47:35.478452+0000","last_became_active":"2024-03-11T21:44:35.618968+0000","last_became_peered":"2024-03-11T21:44:35.618968+0000","last_unstale":"2024-03-11T21:47:35.478452+0000","last_undegraded":"2024-03-11T21:47:35.478452+0000","last_fullsized":"2024-03-11T21:47:35.478452+0000","mapping_epoch":437,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":438,"parent":"0.0","parent_split_bits":0,"last_scrub":"456'29","last_scrub_stamp":"2024-03-11T21:46:28.006437+0000","last_deep_scrub":"456'29","last_deep_scrub_stamp":"2024-03-11T21:46:28.006437+0000","last_clean_scrub_stamp":"2024-03-11T21:46:28.006437+0000","objects_scrubbed":5,"log_size":29,"log_dups_size":0,"ondisk_log_size":29,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:47:42.308493+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":42,"num_read_kb":33,"num_write":26,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,0],"acting":[1,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":917,"reported_epoch":488,"state":"active+clean","last_fresh":"2024-03-11T21:47:35.479889+0000","last_change":"2024-03-11T21:45:33.761870+0000","last_active":"2024-03-11T21:47:35.479889+0000","last_peered":"2024-03-11T21:47:35.479889+0000","last_clean":"2024-03-11T21:47:35.479889+0000","last_became_active":"2024-03-11T21:25:42.573987+0000","last_became_peered":"2024-03-11T21:25:42.573987+0000","last_unstale":"2024-03-11T21:47:35.479889+0000","last_undegraded":"2024-03-11T21:47:35.479889+0000","last_fullsized":"2024-03-11T21:47:35.479889+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:45:33.761758+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:41.544550+0000","last_clean_scrub_stamp":"2024-03-11T21:45:33.761758+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:47:38.778115+0000","scrub_duration":17,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"11.d","version":"488'13693","reported_seq":13789,"reported_epoch":488,"state":"active+clean","last_fresh":"2024-03-11T21:47:38.428234+0000","last_change":"2024-03-11T21:44:53.663253+0000","last_active":"2024-03-11T21:47:38.428234+0000","last_peered":"2024-03-11T21:47:38.428234+0000","last_clean":"2024-03-11T21:47:38.428234+0000","last_became_active":"2024-03-11T21:44:53.662697+0000","last_became_peered":"2024-03-11T21:44:53.662697+0000","last_unstale":"2024-03-11T21:47:38.428234+0000","last_undegraded":"2024-03-11T21:47:38.428234+0000","last_fullsized":"2024-03-11T21:47:38.428234+0000","mapping_epoch":446,"log_start":"472'10076","ondisk_log_start":"472'10076","created":446,"last_epoch_clean":447,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:44:52.644443+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:44:52.644443+0000","last_clean_scrub_stamp":"2024-03-11T21:44:52.644443+0000","objects_scrubbed":0,"log_size":3617,"log_dups_size":0,"ondisk_log_size":3617,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:47:38.777725+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":254672920,"num_objects":3887,"num_object_clones":0,"num_object_copies":11661,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3887,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":13693,"num_write_kb":562497,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6,1],"acting":[2,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.0","version":"0'0","reported_seq":928,"reported_epoch":488,"state":"active+clean","last_fresh":"2024-03-11T21:47:35.474085+0000","last_change":"2024-03-11T21:46:53.265197+0000","last_active":"2024-03-11T21:47:35.474085+0000","last_peered":"2024-03-11T21:47:35.474085+0000","last_clean":"2024-03-11T21:47:35.474085+0000","last_became_active":"2024-03-11T21:46:53.264595+0000","last_became_peered":"2024-03-11T21:46:53.264595+0000","last_unstale":"2024-03-11T21:47:35.474085+0000","last_undegraded":"2024-03-11T21:47:35.474085+0000","last_fullsized":"2024-03-11T21:47:35.474085+0000","mapping_epoch":473,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":474,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:46:13.467417+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:29:39.498784+0000","last_clean_scrub_stamp":"2024-03-11T21:46:13.467417+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":400,"reported_epoch":488,"state":"active+clean","last_fresh":"2024-03-11T21:47:35.476210+0000","last_change":"2024-03-11T21:47:20.274531+0000","last_active":"2024-03-11T21:47:35.476210+0000","last_peered":"2024-03-11T21:47:35.476210+0000","last_clean":"2024-03-11T21:47:35.476210+0000","last_became_active":"2024-03-11T21:47:20.274227+0000","last_became_peered":"2024-03-11T21:47:20.274227+0000","last_unstale":"2024-03-11T21:47:35.476210+0000","last_undegraded":"2024-03-11T21:47:35.476210+0000","last_fullsized":"2024-03-11T21:47:35.476210+0000","mapping_epoch":478,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":479,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:45:27.951397+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:37.512018+0000","last_clean_scrub_stamp":"2024-03-11T21:45:27.951397+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,7],"acting":[0,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.2","version":"456'35","reported_seq":1009,"reported_epoch":488,"state":"active+clean","last_fresh":"2024-03-11T21:47:35.478367+0000","last_change":"2024-03-11T21:47:11.669121+0000","last_active":"2024-03-11T21:47:35.478367+0000","last_peered":"2024-03-11T21:47:35.478367+0000","last_clean":"2024-03-11T21:47:35.478367+0000","last_became_active":"2024-03-11T21:46:53.270401+0000","last_became_peered":"2024-03-11T21:46:53.270401+0000","last_unstale":"2024-03-11T21:47:35.478367+0000","last_undegraded":"2024-03-11T21:47:35.478367+0000","last_fullsized":"2024-03-11T21:47:35.478367+0000","mapping_epoch":473,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":474,"parent":"0.0","parent_split_bits":0,"last_scrub":"456'35","last_scrub_stamp":"2024-03-11T21:47:11.668986+0000","last_deep_scrub":"94'15","last_deep_scrub_stamp":"2024-03-11T21:26:49.993060+0000","last_clean_scrub_stamp":"2024-03-11T21:47:11.668986+0000","objects_scrubbed":5,"log_size":35,"log_dups_size":0,"ondisk_log_size":35,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:48:26.344707+0000","scrub_duration":57,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":45,"num_read_kb":40,"num_write":30,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.3","version":"474'9","reported_seq":1194,"reported_epoch":488,"state":"active+clean","last_fresh":"2024-03-11T21:47:35.478050+0000","last_change":"2024-03-11T21:46:53.302727+0000","last_active":"2024-03-11T21:47:35.478050+0000","last_peered":"2024-03-11T21:47:35.478050+0000","last_clean":"2024-03-11T21:47:35.478050+0000","last_became_active":"2024-03-11T21:46:53.274685+0000","last_became_peered":"2024-03-11T21:46:53.274685+0000","last_unstale":"2024-03-11T21:47:35.478050+0000","last_undegraded":"2024-03-11T21:47:35.478050+0000","last_fullsized":"2024-03-11T21:47:35.478050+0000","mapping_epoch":473,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":474,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'8","last_scrub_stamp":"2024-03-11T21:45:18.050740+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:41.544550+0000","last_clean_scrub_stamp":"2024-03-11T21:45:18.050740+0000","objects_scrubbed":1,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,1],"acting":[0,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.b","version":"488'13438","reported_seq":19763,"reported_epoch":488,"state":"active+remapped+backfilling","last_fresh":"2024-03-11T21:47:39.761368+0000","last_change":"2024-03-11T21:47:21.331974+0000","last_active":"2024-03-11T21:47:39.761368+0000","last_peered":"2024-03-11T21:47:39.761368+0000","last_clean":"2024-03-11T21:46:52.232266+0000","last_became_active":"2024-03-11T21:47:21.272910+0000","last_became_peered":"2024-03-11T21:47:21.272910+0000","last_unstale":"2024-03-11T21:47:39.761368+0000","last_undegraded":"2024-03-11T21:47:39.761368+0000","last_fullsized":"2024-03-11T21:47:39.761368+0000","mapping_epoch":479,"log_start":"472'9791","ondisk_log_start":"472'9791","created":446,"last_epoch_clean":447,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:44:52.644443+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:44:52.644443+0000","last_clean_scrub_stamp":"2024-03-11T21:44:52.644443+0000","objects_scrubbed":0,"log_size":3647,"log_dups_size":0,"ondisk_log_size":3647,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":252837888,"num_objects":3858,"num_object_clones":0,"num_object_copies":11574,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1192,"num_objects_unfound":0,"num_objects_dirty":3858,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13438,"num_write_kb":553472,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3109,"num_bytes_recovered":203685888,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,6],"acting":[0,1,5],"avail_no_missing":["0","1","5"],"object_location_counts":[{"shards":"0,1,5","objects":3858}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":524,"reported_epoch":488,"state":"active+clean","last_fresh":"2024-03-11T21:47:35.479093+0000","last_change":"2024-03-11T21:46:53.269895+0000","last_active":"2024-03-11T21:47:35.479093+0000","last_peered":"2024-03-11T21:47:35.479093+0000","last_clean":"2024-03-11T21:47:35.479093+0000","last_became_active":"2024-03-11T21:46:53.269288+0000","last_became_peered":"2024-03-11T21:46:53.269288+0000","last_unstale":"2024-03-11T21:47:35.479093+0000","last_undegraded":"2024-03-11T21:47:35.479093+0000","last_fullsized":"2024-03-11T21:47:35.479093+0000","mapping_epoch":473,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":474,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:44:27.024978+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:28:30.563712+0000","last_clean_scrub_stamp":"2024-03-11T21:44:27.024978+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,1],"acting":[7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":40,"reported_epoch":488,"state":"active+clean","last_fresh":"2024-03-11T21:47:35.478037+0000","last_change":"2024-03-11T21:46:53.265421+0000","last_active":"2024-03-11T21:47:35.478037+0000","last_peered":"2024-03-11T21:47:35.478037+0000","last_clean":"2024-03-11T21:47:35.478037+0000","last_became_active":"2024-03-11T21:46:53.264169+0000","last_became_peered":"2024-03-11T21:46:53.264169+0000","last_unstale":"2024-03-11T21:47:35.478037+0000","last_undegraded":"2024-03-11T21:47:35.478037+0000","last_fullsized":"2024-03-11T21:47:35.478037+0000","mapping_epoch":473,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":474,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:45:28.088517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:37.512018+0000","last_clean_scrub_stamp":"2024-03-11T21:45:28.088517+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,0],"acting":[1,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.4","version":"479'56","reported_seq":1256,"reported_epoch":488,"state":"active+clean","last_fresh":"2024-03-11T21:47:35.479120+0000","last_change":"2024-03-11T21:47:20.279434+0000","last_active":"2024-03-11T21:47:35.479120+0000","last_peered":"2024-03-11T21:47:35.479120+0000","last_clean":"2024-03-11T21:47:35.479120+0000","last_became_active":"2024-03-11T21:47:20.277794+0000","last_became_peered":"2024-03-11T21:47:20.277794+0000","last_unstale":"2024-03-11T21:47:35.479120+0000","last_undegraded":"2024-03-11T21:47:35.479120+0000","last_fullsized":"2024-03-11T21:47:35.479120+0000","mapping_epoch":478,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":479,"parent":"0.0","parent_split_bits":0,"last_scrub":"434'43","last_scrub_stamp":"2024-03-11T21:45:12.775393+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:39.525554+0000","last_clean_scrub_stamp":"2024-03-11T21:45:12.775393+0000","objects_scrubbed":10,"log_size":56,"log_dups_size":0,"ondisk_log_size":56,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:47:38.777818+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":358,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":83,"num_read_kb":68,"num_write":50,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":358,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,6],"acting":[2,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":917,"reported_epoch":488,"state":"active+clean","last_fresh":"2024-03-11T21:47:35.473261+0000","last_change":"2024-03-11T21:45:37.189812+0000","last_active":"2024-03-11T21:47:35.473261+0000","last_peered":"2024-03-11T21:47:35.473261+0000","last_clean":"2024-03-11T21:47:35.473261+0000","last_became_active":"2024-03-11T21:44:35.620226+0000","last_became_peered":"2024-03-11T21:44:35.620226+0000","last_unstale":"2024-03-11T21:47:35.473261+0000","last_undegraded":"2024-03-11T21:47:35.473261+0000","last_fullsized":"2024-03-11T21:47:35.473261+0000","mapping_epoch":437,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":438,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:45:37.189703+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:41.544550+0000","last_clean_scrub_stamp":"2024-03-11T21:45:37.189703+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,6],"acting":[0,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.8","version":"488'13451","reported_seq":13557,"reported_epoch":488,"state":"active+remapped+backfill_toofull","last_fresh":"2024-03-11T21:47:37.674097+0000","last_change":"2024-03-11T21:47:24.415860+0000","last_active":"2024-03-11T21:47:37.674097+0000","last_peered":"2024-03-11T21:47:37.674097+0000","last_clean":"2024-03-11T21:46:52.225137+0000","last_became_active":"2024-03-11T21:46:54.388322+0000","last_became_peered":"2024-03-11T21:46:54.388322+0000","last_unstale":"2024-03-11T21:47:37.674097+0000","last_undegraded":"2024-03-11T21:47:37.674097+0000","last_fullsized":"2024-03-11T21:47:37.674097+0000","mapping_epoch":474,"log_start":"475'10061","ondisk_log_start":"475'10061","created":446,"last_epoch_clean":447,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:44:52.644443+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:44:52.644443+0000","last_clean_scrub_stamp":"2024-03-11T21:44:52.644443+0000","objects_scrubbed":0,"log_size":3390,"log_dups_size":0,"ondisk_log_size":3390,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":258539520,"num_objects":3945,"num_object_clones":0,"num_object_copies":11835,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":3945,"num_objects_unfound":0,"num_objects_dirty":3945,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13451,"num_write_kb":556672,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,2],"acting":[6,0,5],"avail_no_missing":["6","0","5"],"object_location_counts":[{"shards":"0,5,6","objects":3945}],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.5","version":"0'0","reported_seq":578,"reported_epoch":488,"state":"active+clean","last_fresh":"2024-03-11T21:47:35.479400+0000","last_change":"2024-03-11T21:45:26.194327+0000","last_active":"2024-03-11T21:47:35.479400+0000","last_peered":"2024-03-11T21:47:35.479400+0000","last_clean":"2024-03-11T21:47:35.479400+0000","last_became_active":"2024-03-11T21:44:21.454156+0000","last_became_peered":"2024-03-11T21:44:21.454156+0000","last_unstale":"2024-03-11T21:47:35.479400+0000","last_undegraded":"2024-03-11T21:47:35.479400+0000","last_fullsized":"2024-03-11T21:47:35.479400+0000","mapping_epoch":432,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":433,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:45:26.194208+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:43.559452+0000","last_clean_scrub_stamp":"2024-03-11T21:45:26.194208+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":43,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,2],"acting":[7,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.0","version":"164'6","reported_seq":687,"reported_epoch":488,"state":"active+clean","last_fresh":"2024-03-11T21:47:37.321153+0000","last_change":"2024-03-11T21:47:20.280196+0000","last_active":"2024-03-11T21:47:37.321153+0000","last_peered":"2024-03-11T21:47:37.321153+0000","last_clean":"2024-03-11T21:47:37.321153+0000","last_became_active":"2024-03-11T21:47:20.277766+0000","last_became_peered":"2024-03-11T21:47:20.277766+0000","last_unstale":"2024-03-11T21:47:37.321153+0000","last_undegraded":"2024-03-11T21:47:37.321153+0000","last_fullsized":"2024-03-11T21:47:37.321153+0000","mapping_epoch":478,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":479,"parent":"0.0","parent_split_bits":0,"last_scrub":"164'6","last_scrub_stamp":"2024-03-11T21:45:25.994672+0000","last_deep_scrub":"164'6","last_deep_scrub_stamp":"2024-03-11T21:32:17.597666+0000","last_clean_scrub_stamp":"2024-03-11T21:45:25.994672+0000","objects_scrubbed":1,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:47:38.778058+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0.000753559,"stat_sum":{"num_bytes":389,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":405,"num_read_kb":400,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,6],"acting":[2,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":940,"reported_epoch":488,"state":"active+clean","last_fresh":"2024-03-11T21:47:35.479612+0000","last_change":"2024-03-11T21:45:21.620901+0000","last_active":"2024-03-11T21:47:35.479612+0000","last_peered":"2024-03-11T21:47:35.479612+0000","last_clean":"2024-03-11T21:47:35.479612+0000","last_became_active":"2024-03-11T21:44:21.462115+0000","last_became_peered":"2024-03-11T21:44:21.462115+0000","last_unstale":"2024-03-11T21:47:35.479612+0000","last_undegraded":"2024-03-11T21:47:35.479612+0000","last_fullsized":"2024-03-11T21:47:35.479612+0000","mapping_epoch":432,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":433,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:45:21.620627+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:43:28.183315+0000","last_clean_scrub_stamp":"2024-03-11T21:45:21.620627+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:47:38.777848+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.7","version":"456'26","reported_seq":980,"reported_epoch":488,"state":"active+clean","last_fresh":"2024-03-11T21:47:35.481768+0000","last_change":"2024-03-11T21:46:53.307937+0000","last_active":"2024-03-11T21:47:35.481768+0000","last_peered":"2024-03-11T21:47:35.481768+0000","last_clean":"2024-03-11T21:47:35.481768+0000","last_became_active":"2024-03-11T21:46:53.274008+0000","last_became_peered":"2024-03-11T21:46:53.274008+0000","last_unstale":"2024-03-11T21:47:35.481768+0000","last_undegraded":"2024-03-11T21:47:35.481768+0000","last_fullsized":"2024-03-11T21:47:35.481768+0000","mapping_epoch":473,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":474,"parent":"0.0","parent_split_bits":0,"last_scrub":"260'22","last_scrub_stamp":"2024-03-11T21:45:33.942306+0000","last_deep_scrub":"260'22","last_deep_scrub_stamp":"2024-03-11T21:41:27.232403+0000","last_clean_scrub_stamp":"2024-03-11T21:45:33.942306+0000","objects_scrubbed":5,"log_size":26,"log_dups_size":0,"ondisk_log_size":26,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":42,"num_read_kb":31,"num_write":24,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,7],"acting":[1,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":935,"reported_epoch":488,"state":"active+clean","last_fresh":"2024-03-11T21:47:35.480314+0000","last_change":"2024-03-11T21:46:53.277478+0000","last_active":"2024-03-11T21:47:35.480314+0000","last_peered":"2024-03-11T21:47:35.480314+0000","last_clean":"2024-03-11T21:47:35.480314+0000","last_became_active":"2024-03-11T21:46:53.276947+0000","last_became_peered":"2024-03-11T21:46:53.276947+0000","last_unstale":"2024-03-11T21:47:35.480314+0000","last_undegraded":"2024-03-11T21:47:35.480314+0000","last_fullsized":"2024-03-11T21:47:35.480314+0000","mapping_epoch":473,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":474,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:45:39.556342+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:41.544550+0000","last_clean_scrub_stamp":"2024-03-11T21:45:39.556342+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:47:38.778087+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,1],"acting":[2,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"11.a","version":"488'13627","reported_seq":24960,"reported_epoch":488,"state":"active+clean","last_fresh":"2024-03-11T21:47:37.669004+0000","last_change":"2024-03-11T21:47:31.421385+0000","last_active":"2024-03-11T21:47:37.669004+0000","last_peered":"2024-03-11T21:47:37.669004+0000","last_clean":"2024-03-11T21:47:37.669004+0000","last_became_active":"2024-03-11T21:47:31.404871+0000","last_became_peered":"2024-03-11T21:47:31.404871+0000","last_unstale":"2024-03-11T21:47:37.669004+0000","last_undegraded":"2024-03-11T21:47:37.669004+0000","last_fullsized":"2024-03-11T21:47:37.669004+0000","mapping_epoch":484,"log_start":"475'10169","ondisk_log_start":"475'10169","created":446,"last_epoch_clean":485,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:44:52.644443+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:44:52.644443+0000","last_clean_scrub_stamp":"2024-03-11T21:44:52.644443+0000","objects_scrubbed":0,"log_size":3458,"log_dups_size":0,"ondisk_log_size":3458,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":260243456,"num_objects":3971,"num_object_clones":0,"num_object_copies":11913,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3971,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13627,"num_write_kb":563136,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5613,"num_bytes_recovered":367853568,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,2],"acting":[6,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.7","version":"124'2","reported_seq":838,"reported_epoch":488,"state":"active+clean","last_fresh":"2024-03-11T21:47:35.482118+0000","last_change":"2024-03-11T21:46:53.301059+0000","last_active":"2024-03-11T21:47:35.482118+0000","last_peered":"2024-03-11T21:47:35.482118+0000","last_clean":"2024-03-11T21:47:35.482118+0000","last_became_active":"2024-03-11T21:46:53.279257+0000","last_became_peered":"2024-03-11T21:46:53.279257+0000","last_unstale":"2024-03-11T21:47:35.482118+0000","last_undegraded":"2024-03-11T21:47:35.482118+0000","last_fullsized":"2024-03-11T21:47:35.482118+0000","mapping_epoch":473,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":474,"parent":"0.0","parent_split_bits":0,"last_scrub":"124'2","last_scrub_stamp":"2024-03-11T21:46:27.025988+0000","last_deep_scrub":"124'2","last_deep_scrub_stamp":"2024-03-11T21:29:34.956908+0000","last_clean_scrub_stamp":"2024-03-11T21:46:27.025988+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:47:49.666925+0000","scrub_duration":111,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":13,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,6],"acting":[1,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2","version":"71'2","reported_seq":70,"reported_epoch":488,"state":"active+clean","last_fresh":"2024-03-11T21:47:35.473526+0000","last_change":"2024-03-11T21:46:53.279974+0000","last_active":"2024-03-11T21:47:35.473526+0000","last_peered":"2024-03-11T21:47:35.473526+0000","last_clean":"2024-03-11T21:47:35.473526+0000","last_became_active":"2024-03-11T21:46:53.271756+0000","last_became_peered":"2024-03-11T21:46:53.271756+0000","last_unstale":"2024-03-11T21:47:35.473526+0000","last_undegraded":"2024-03-11T21:47:35.473526+0000","last_fullsized":"2024-03-11T21:47:35.473526+0000","mapping_epoch":473,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":474,"parent":"0.0","parent_split_bits":0,"last_scrub":"71'2","last_scrub_stamp":"2024-03-11T21:44:27.869844+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:37.512018+0000","last_clean_scrub_stamp":"2024-03-11T21:44:27.869844+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014206449999999999,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,7],"acting":[6,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":415,"reported_epoch":488,"state":"active+clean","last_fresh":"2024-03-11T21:47:35.477612+0000","last_change":"2024-03-11T21:46:53.258756+0000","last_active":"2024-03-11T21:47:35.477612+0000","last_peered":"2024-03-11T21:47:35.477612+0000","last_clean":"2024-03-11T21:47:35.477612+0000","last_became_active":"2024-03-11T21:46:53.257137+0000","last_became_peered":"2024-03-11T21:46:53.257137+0000","last_unstale":"2024-03-11T21:47:35.477612+0000","last_undegraded":"2024-03-11T21:47:35.477612+0000","last_fullsized":"2024-03-11T21:47:35.477612+0000","mapping_epoch":473,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":474,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:46:17.861505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:37.512018+0000","last_clean_scrub_stamp":"2024-03-11T21:46:17.861505+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:47:37.620367+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,6],"acting":[2,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"87'128","reported_seq":1034,"reported_epoch":488,"state":"active+clean","last_fresh":"2024-03-11T21:47:35.476610+0000","last_change":"2024-03-11T21:44:20.468089+0000","last_active":"2024-03-11T21:47:35.476610+0000","last_peered":"2024-03-11T21:47:35.476610+0000","last_clean":"2024-03-11T21:47:35.476610+0000","last_became_active":"2024-03-11T21:34:44.164569+0000","last_became_peered":"2024-03-11T21:34:44.164569+0000","last_unstale":"2024-03-11T21:47:35.476610+0000","last_undegraded":"2024-03-11T21:47:35.476610+0000","last_fullsized":"2024-03-11T21:47:35.476610+0000","mapping_epoch":229,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":230,"parent":"0.0","parent_split_bits":0,"last_scrub":"87'128","last_scrub_stamp":"2024-03-11T21:44:20.468006+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:24.261974+0000","last_clean_scrub_stamp":"2024-03-11T21:44:20.468006+0000","objects_scrubbed":2,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":198,"num_read_kb":294,"num_write":179,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":2951840,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.5","version":"456'41","reported_seq":1278,"reported_epoch":488,"state":"active+clean","last_fresh":"2024-03-11T21:47:35.474906+0000","last_change":"2024-03-11T21:44:35.660956+0000","last_active":"2024-03-11T21:47:35.474906+0000","last_peered":"2024-03-11T21:47:35.474906+0000","last_clean":"2024-03-11T21:47:35.474906+0000","last_became_active":"2024-03-11T21:44:35.634664+0000","last_became_peered":"2024-03-11T21:44:35.634664+0000","last_unstale":"2024-03-11T21:47:35.474906+0000","last_undegraded":"2024-03-11T21:47:35.474906+0000","last_fullsized":"2024-03-11T21:47:35.474906+0000","mapping_epoch":437,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":438,"parent":"0.0","parent_split_bits":0,"last_scrub":"260'28","last_scrub_stamp":"2024-03-11T21:44:26.464384+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:39.525554+0000","last_clean_scrub_stamp":"2024-03-11T21:44:26.464384+0000","objects_scrubbed":8,"log_size":41,"log_dups_size":0,"ondisk_log_size":41,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":154,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":61,"num_read_kb":51,"num_write":37,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":154,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,0],"acting":[6,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":577,"reported_epoch":488,"state":"active+clean","last_fresh":"2024-03-11T21:47:35.478907+0000","last_change":"2024-03-11T21:46:53.276498+0000","last_active":"2024-03-11T21:47:35.478907+0000","last_peered":"2024-03-11T21:47:35.478907+0000","last_clean":"2024-03-11T21:47:35.478907+0000","last_became_active":"2024-03-11T21:46:53.276162+0000","last_became_peered":"2024-03-11T21:46:53.276162+0000","last_unstale":"2024-03-11T21:47:35.478907+0000","last_undegraded":"2024-03-11T21:47:35.478907+0000","last_fullsized":"2024-03-11T21:47:35.478907+0000","mapping_epoch":473,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":474,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:46:27.586495+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:32:21.266672+0000","last_clean_scrub_stamp":"2024-03-11T21:46:27.586495+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,0],"acting":[7,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"11.9","version":"488'13166","reported_seq":13261,"reported_epoch":488,"state":"active+clean","last_fresh":"2024-03-11T21:47:37.672004+0000","last_change":"2024-03-11T21:44:53.662919+0000","last_active":"2024-03-11T21:47:37.672004+0000","last_peered":"2024-03-11T21:47:37.672004+0000","last_clean":"2024-03-11T21:47:37.672004+0000","last_became_active":"2024-03-11T21:44:53.662344+0000","last_became_peered":"2024-03-11T21:44:53.662344+0000","last_unstale":"2024-03-11T21:47:37.672004+0000","last_undegraded":"2024-03-11T21:47:37.672004+0000","last_fullsized":"2024-03-11T21:47:37.672004+0000","mapping_epoch":446,"log_start":"472'9724","ondisk_log_start":"472'9724","created":446,"last_epoch_clean":447,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:44:52.644443+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:44:52.644443+0000","last_clean_scrub_stamp":"2024-03-11T21:44:52.644443+0000","objects_scrubbed":0,"log_size":3442,"log_dups_size":0,"ondisk_log_size":3442,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":255459328,"num_objects":3898,"num_object_clones":0,"num_object_copies":11694,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3898,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13166,"num_write_kb":546048,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,1],"acting":[6,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.4","version":"0'0","reported_seq":928,"reported_epoch":488,"state":"active+clean","last_fresh":"2024-03-11T21:47:35.480033+0000","last_change":"2024-03-11T21:47:10.662513+0000","last_active":"2024-03-11T21:47:35.480033+0000","last_peered":"2024-03-11T21:47:35.480033+0000","last_clean":"2024-03-11T21:47:35.480033+0000","last_became_active":"2024-03-11T21:46:53.273797+0000","last_became_peered":"2024-03-11T21:46:53.273797+0000","last_unstale":"2024-03-11T21:47:35.480033+0000","last_undegraded":"2024-03-11T21:47:35.480033+0000","last_fullsized":"2024-03-11T21:47:35.480033+0000","mapping_epoch":473,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":474,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:47:10.662269+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:43.559452+0000","last_clean_scrub_stamp":"2024-03-11T21:47:10.662269+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:48:21.267901+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":953,"reported_epoch":488,"state":"active+clean","last_fresh":"2024-03-11T21:47:35.469596+0000","last_change":"2024-03-11T21:46:53.255662+0000","last_active":"2024-03-11T21:47:35.469596+0000","last_peered":"2024-03-11T21:47:35.469596+0000","last_clean":"2024-03-11T21:47:35.469596+0000","last_became_active":"2024-03-11T21:46:53.254297+0000","last_became_peered":"2024-03-11T21:46:53.254297+0000","last_unstale":"2024-03-11T21:47:35.469596+0000","last_undegraded":"2024-03-11T21:47:35.469596+0000","last_fullsized":"2024-03-11T21:47:35.469596+0000","mapping_epoch":473,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":474,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:45:12.113799+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:37.512018+0000","last_clean_scrub_stamp":"2024-03-11T21:45:12.113799+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0.00089342499999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,1],"acting":[0,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":563,"reported_epoch":488,"state":"active+clean","last_fresh":"2024-03-11T21:47:35.476526+0000","last_change":"2024-03-11T21:46:19.530489+0000","last_active":"2024-03-11T21:47:35.476526+0000","last_peered":"2024-03-11T21:47:35.476526+0000","last_clean":"2024-03-11T21:47:35.476526+0000","last_became_active":"2024-03-11T21:34:41.149219+0000","last_became_peered":"2024-03-11T21:34:41.149219+0000","last_unstale":"2024-03-11T21:47:35.476526+0000","last_undegraded":"2024-03-11T21:47:35.476526+0000","last_fullsized":"2024-03-11T21:47:35.476526+0000","mapping_epoch":226,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":227,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:46:19.530414+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:43:30.734024+0000","last_clean_scrub_stamp":"2024-03-11T21:46:19.530414+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,0],"acting":[7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.6","version":"456'42","reported_seq":1023,"reported_epoch":488,"state":"active+clean","last_fresh":"2024-03-11T21:47:35.470612+0000","last_change":"2024-03-11T21:46:53.286336+0000","last_active":"2024-03-11T21:47:35.470612+0000","last_peered":"2024-03-11T21:47:35.470612+0000","last_clean":"2024-03-11T21:47:35.470612+0000","last_became_active":"2024-03-11T21:46:53.263790+0000","last_became_peered":"2024-03-11T21:46:53.263790+0000","last_unstale":"2024-03-11T21:47:35.470612+0000","last_undegraded":"2024-03-11T21:47:35.470612+0000","last_fullsized":"2024-03-11T21:47:35.470612+0000","mapping_epoch":473,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":474,"parent":"0.0","parent_split_bits":0,"last_scrub":"456'42","last_scrub_stamp":"2024-03-11T21:46:10.509062+0000","last_deep_scrub":"352'30","last_deep_scrub_stamp":"2024-03-11T21:40:31.175323+0000","last_clean_scrub_stamp":"2024-03-11T21:46:10.509062+0000","objects_scrubbed":6,"log_size":42,"log_dups_size":0,"ondisk_log_size":42,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":54,"num_read_kb":48,"num_write":36,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":40,"reported_epoch":488,"state":"active+clean","last_fresh":"2024-03-11T21:47:35.482008+0000","last_change":"2024-03-11T21:46:53.279613+0000","last_active":"2024-03-11T21:47:35.482008+0000","last_peered":"2024-03-11T21:47:35.482008+0000","last_clean":"2024-03-11T21:47:35.482008+0000","last_became_active":"2024-03-11T21:46:53.278310+0000","last_became_peered":"2024-03-11T21:46:53.278310+0000","last_unstale":"2024-03-11T21:47:35.482008+0000","last_undegraded":"2024-03-11T21:47:35.482008+0000","last_fullsized":"2024-03-11T21:47:35.482008+0000","mapping_epoch":473,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":474,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:45:40.085999+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:41.544550+0000","last_clean_scrub_stamp":"2024-03-11T21:45:40.085999+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,7],"acting":[1,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.c","version":"488'13282","reported_seq":13435,"reported_epoch":488,"state":"active+clean","last_fresh":"2024-03-11T21:47:38.882330+0000","last_change":"2024-03-11T21:47:24.484939+0000","last_active":"2024-03-11T21:47:38.882330+0000","last_peered":"2024-03-11T21:47:38.882330+0000","last_clean":"2024-03-11T21:47:38.882330+0000","last_became_active":"2024-03-11T21:44:53.666491+0000","last_became_peered":"2024-03-11T21:44:53.666491+0000","last_unstale":"2024-03-11T21:47:38.882330+0000","last_undegraded":"2024-03-11T21:47:38.882330+0000","last_fullsized":"2024-03-11T21:47:38.882330+0000","mapping_epoch":446,"log_start":"476'10187","ondisk_log_start":"476'10187","created":446,"last_epoch_clean":447,"parent":"0.0","parent_split_bits":0,"last_scrub":"480'12228","last_scrub_stamp":"2024-03-11T21:47:24.484512+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:44:52.644443+0000","last_clean_scrub_stamp":"2024-03-11T21:47:24.484512+0000","objects_scrubbed":4892,"log_size":3095,"log_dups_size":0,"ondisk_log_size":3095,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:48:34.228707+0000","scrub_duration":1840,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":246808576,"num_objects":3766,"num_object_clones":0,"num_object_copies":11298,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3766,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13282,"num_write_kb":545536,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,7],"acting":[1,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":915,"reported_epoch":488,"state":"active+clean","last_fresh":"2024-03-11T21:47:35.478588+0000","last_change":"2024-03-11T21:46:18.006161+0000","last_active":"2024-03-11T21:47:35.478588+0000","last_peered":"2024-03-11T21:47:35.478588+0000","last_clean":"2024-03-11T21:47:35.478588+0000","last_became_active":"2024-03-11T21:25:45.301416+0000","last_became_peered":"2024-03-11T21:25:45.301416+0000","last_unstale":"2024-03-11T21:47:35.478588+0000","last_undegraded":"2024-03-11T21:47:35.478588+0000","last_fullsized":"2024-03-11T21:47:35.478588+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:46:18.006081+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:43.559452+0000","last_clean_scrub_stamp":"2024-03-11T21:46:18.006081+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:47:41.400276+0000","scrub_duration":29,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":569,"reported_epoch":488,"state":"active+clean","last_fresh":"2024-03-11T21:47:35.479667+0000","last_change":"2024-03-11T21:45:21.273974+0000","last_active":"2024-03-11T21:47:35.479667+0000","last_peered":"2024-03-11T21:47:35.479667+0000","last_clean":"2024-03-11T21:47:35.479667+0000","last_became_active":"2024-03-11T21:44:35.624442+0000","last_became_peered":"2024-03-11T21:44:35.624442+0000","last_unstale":"2024-03-11T21:47:35.479667+0000","last_undegraded":"2024-03-11T21:47:35.479667+0000","last_fullsized":"2024-03-11T21:47:35.479667+0000","mapping_epoch":437,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":438,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:45:21.273887+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:26:57.333669+0000","last_clean_scrub_stamp":"2024-03-11T21:45:21.273887+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":69,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,1],"acting":[7,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.3","version":"456'47","reported_seq":1069,"reported_epoch":488,"state":"active+clean","last_fresh":"2024-03-11T21:47:35.476948+0000","last_change":"2024-03-11T21:46:53.297787+0000","last_active":"2024-03-11T21:47:35.476948+0000","last_peered":"2024-03-11T21:47:35.476948+0000","last_clean":"2024-03-11T21:47:35.476948+0000","last_became_active":"2024-03-11T21:46:53.274626+0000","last_became_peered":"2024-03-11T21:46:53.274626+0000","last_unstale":"2024-03-11T21:47:35.476948+0000","last_undegraded":"2024-03-11T21:47:35.476948+0000","last_fullsized":"2024-03-11T21:47:35.476948+0000","mapping_epoch":473,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":474,"parent":"0.0","parent_split_bits":0,"last_scrub":"456'47","last_scrub_stamp":"2024-03-11T21:46:27.467220+0000","last_deep_scrub":"165'27","last_deep_scrub_stamp":"2024-03-11T21:32:21.576732+0000","last_clean_scrub_stamp":"2024-03-11T21:46:27.467220+0000","objects_scrubbed":8,"log_size":47,"log_dups_size":0,"ondisk_log_size":47,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:47:39.990219+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":69,"num_read_kb":55,"num_write":42,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":24,"num_bytes_recovered":990,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,1],"acting":[0,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":941,"reported_epoch":488,"state":"active+clean","last_fresh":"2024-03-11T21:47:35.473417+0000","last_change":"2024-03-11T21:46:53.259845+0000","last_active":"2024-03-11T21:47:35.473417+0000","last_peered":"2024-03-11T21:47:35.473417+0000","last_clean":"2024-03-11T21:47:35.473417+0000","last_became_active":"2024-03-11T21:46:53.259300+0000","last_became_peered":"2024-03-11T21:46:53.259300+0000","last_unstale":"2024-03-11T21:47:35.473417+0000","last_undegraded":"2024-03-11T21:47:35.473417+0000","last_fullsized":"2024-03-11T21:47:35.473417+0000","mapping_epoch":473,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":474,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:44:35.462043+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:41:21.857205+0000","last_clean_scrub_stamp":"2024-03-11T21:44:35.462043+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,2],"acting":[6,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.f","version":"488'13368","reported_seq":13463,"reported_epoch":488,"state":"active+clean","last_fresh":"2024-03-11T21:47:37.599351+0000","last_change":"2024-03-11T21:44:53.666917+0000","last_active":"2024-03-11T21:47:37.599351+0000","last_peered":"2024-03-11T21:47:37.599351+0000","last_clean":"2024-03-11T21:47:37.599351+0000","last_became_active":"2024-03-11T21:44:53.664970+0000","last_became_peered":"2024-03-11T21:44:53.664970+0000","last_unstale":"2024-03-11T21:47:37.599351+0000","last_undegraded":"2024-03-11T21:47:37.599351+0000","last_fullsized":"2024-03-11T21:47:37.599351+0000","mapping_epoch":446,"log_start":"475'10021","ondisk_log_start":"475'10021","created":446,"last_epoch_clean":447,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:44:52.644443+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:44:52.644443+0000","last_clean_scrub_stamp":"2024-03-11T21:44:52.644443+0000","objects_scrubbed":0,"log_size":3347,"log_dups_size":0,"ondisk_log_size":3347,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":260177920,"num_objects":3970,"num_object_clones":0,"num_object_copies":11910,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3970,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13368,"num_write_kb":554816,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,2],"acting":[7,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":919,"reported_epoch":488,"state":"active+clean","last_fresh":"2024-03-11T21:47:35.484845+0000","last_change":"2024-03-11T21:45:35.682015+0000","last_active":"2024-03-11T21:47:35.484845+0000","last_peered":"2024-03-11T21:47:35.484845+0000","last_clean":"2024-03-11T21:47:35.484845+0000","last_became_active":"2024-03-11T21:44:35.628373+0000","last_became_peered":"2024-03-11T21:44:35.628373+0000","last_unstale":"2024-03-11T21:47:35.484845+0000","last_undegraded":"2024-03-11T21:47:35.484845+0000","last_fullsized":"2024-03-11T21:47:35.484845+0000","mapping_epoch":437,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":438,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:45:35.681881+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:43.559452+0000","last_clean_scrub_stamp":"2024-03-11T21:45:35.681881+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:47:38.777937+0000","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6,7],"acting":[2,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.6","version":"67'1","reported_seq":930,"reported_epoch":488,"state":"active+clean","last_fresh":"2024-03-11T21:47:35.480561+0000","last_change":"2024-03-11T21:45:16.765246+0000","last_active":"2024-03-11T21:47:35.480561+0000","last_peered":"2024-03-11T21:47:35.480561+0000","last_clean":"2024-03-11T21:47:35.480561+0000","last_became_active":"2024-03-11T21:44:35.618830+0000","last_became_peered":"2024-03-11T21:44:35.618830+0000","last_unstale":"2024-03-11T21:47:35.480561+0000","last_undegraded":"2024-03-11T21:47:35.480561+0000","last_fullsized":"2024-03-11T21:47:35.480561+0000","mapping_epoch":437,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":438,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'1","last_scrub_stamp":"2024-03-11T21:45:16.765125+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:37.512018+0000","last_clean_scrub_stamp":"2024-03-11T21:45:16.765125+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":46,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,0],"acting":[1,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.0","version":"456'50","reported_seq":662,"reported_epoch":488,"state":"active+clean","last_fresh":"2024-03-11T21:47:35.476802+0000","last_change":"2024-03-11T21:45:39.242650+0000","last_active":"2024-03-11T21:47:35.476802+0000","last_peered":"2024-03-11T21:47:35.476802+0000","last_clean":"2024-03-11T21:47:35.476802+0000","last_became_active":"2024-03-11T21:44:21.449580+0000","last_became_peered":"2024-03-11T21:44:21.449580+0000","last_unstale":"2024-03-11T21:47:35.476802+0000","last_undegraded":"2024-03-11T21:47:35.476802+0000","last_fullsized":"2024-03-11T21:47:35.476802+0000","mapping_epoch":432,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":433,"parent":"0.0","parent_split_bits":0,"last_scrub":"260'38","last_scrub_stamp":"2024-03-11T21:45:39.241846+0000","last_deep_scrub":"260'38","last_deep_scrub_stamp":"2024-03-11T21:45:39.241846+0000","last_clean_scrub_stamp":"2024-03-11T21:45:39.241846+0000","objects_scrubbed":8,"log_size":50,"log_dups_size":0,"ondisk_log_size":50,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":70,"num_read_kb":58,"num_write":44,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":440,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,1],"acting":[7,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":944,"reported_epoch":488,"state":"active+clean","last_fresh":"2024-03-11T21:47:35.477867+0000","last_change":"2024-03-11T21:46:53.261179+0000","last_active":"2024-03-11T21:47:35.477867+0000","last_peered":"2024-03-11T21:47:35.477867+0000","last_clean":"2024-03-11T21:47:35.477867+0000","last_became_active":"2024-03-11T21:46:53.258681+0000","last_became_peered":"2024-03-11T21:46:53.258681+0000","last_unstale":"2024-03-11T21:47:35.477867+0000","last_undegraded":"2024-03-11T21:47:35.477867+0000","last_fullsized":"2024-03-11T21:47:35.477867+0000","mapping_epoch":473,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":474,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:46:09.504070+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:40:21.518840+0000","last_clean_scrub_stamp":"2024-03-11T21:46:09.504070+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,2],"acting":[7,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"11.e","version":"488'13220","reported_seq":13315,"reported_epoch":488,"state":"active+clean","last_fresh":"2024-03-11T21:47:38.889352+0000","last_change":"2024-03-11T21:44:53.665261+0000","last_active":"2024-03-11T21:47:38.889352+0000","last_peered":"2024-03-11T21:47:38.889352+0000","last_clean":"2024-03-11T21:47:38.889352+0000","last_became_active":"2024-03-11T21:44:53.664957+0000","last_became_peered":"2024-03-11T21:44:53.664957+0000","last_unstale":"2024-03-11T21:47:38.889352+0000","last_undegraded":"2024-03-11T21:47:38.889352+0000","last_fullsized":"2024-03-11T21:47:38.889352+0000","mapping_epoch":446,"log_start":"476'10031","ondisk_log_start":"476'10031","created":446,"last_epoch_clean":447,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:44:52.644443+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:44:52.644443+0000","last_clean_scrub_stamp":"2024-03-11T21:44:52.644443+0000","objects_scrubbed":0,"log_size":3189,"log_dups_size":0,"ondisk_log_size":3189,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":251920384,"num_objects":3844,"num_object_clones":0,"num_object_copies":11532,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3844,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13220,"num_write_kb":546048,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,0],"acting":[1,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.3","version":"0'0","reported_seq":563,"reported_epoch":488,"state":"active+clean","last_fresh":"2024-03-11T21:47:35.479315+0000","last_change":"2024-03-11T21:46:25.645839+0000","last_active":"2024-03-11T21:47:35.479315+0000","last_peered":"2024-03-11T21:47:35.479315+0000","last_clean":"2024-03-11T21:47:35.479315+0000","last_became_active":"2024-03-11T21:34:41.142683+0000","last_became_peered":"2024-03-11T21:34:41.142683+0000","last_unstale":"2024-03-11T21:47:35.479315+0000","last_undegraded":"2024-03-11T21:47:35.479315+0000","last_fullsized":"2024-03-11T21:47:35.479315+0000","mapping_epoch":226,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":227,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:46:25.645722+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:46:25.645722+0000","last_clean_scrub_stamp":"2024-03-11T21:46:25.645722+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:47:50.261770+0000","scrub_duration":44,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,2],"acting":[7,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":955,"reported_epoch":488,"state":"active+clean","last_fresh":"2024-03-11T21:47:35.474516+0000","last_change":"2024-03-11T21:45:29.722211+0000","last_active":"2024-03-11T21:47:35.474516+0000","last_peered":"2024-03-11T21:47:35.474516+0000","last_clean":"2024-03-11T21:47:35.474516+0000","last_became_active":"2024-03-11T21:34:41.145628+0000","last_became_peered":"2024-03-11T21:34:41.145628+0000","last_unstale":"2024-03-11T21:47:35.474516+0000","last_undegraded":"2024-03-11T21:47:35.474516+0000","last_fullsized":"2024-03-11T21:47:35.474516+0000","mapping_epoch":226,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":227,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:45:29.722096+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:37.512018+0000","last_clean_scrub_stamp":"2024-03-11T21:45:29.722096+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,2],"acting":[6,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.1","version":"456'42","reported_seq":624,"reported_epoch":488,"state":"active+clean","last_fresh":"2024-03-11T21:47:35.473763+0000","last_change":"2024-03-11T21:46:53.305466+0000","last_active":"2024-03-11T21:47:35.473763+0000","last_peered":"2024-03-11T21:47:35.473763+0000","last_clean":"2024-03-11T21:47:35.473763+0000","last_became_active":"2024-03-11T21:46:53.272533+0000","last_became_peered":"2024-03-11T21:46:53.272533+0000","last_unstale":"2024-03-11T21:47:35.473763+0000","last_undegraded":"2024-03-11T21:47:35.473763+0000","last_fullsized":"2024-03-11T21:47:35.473763+0000","mapping_epoch":473,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":474,"parent":"0.0","parent_split_bits":0,"last_scrub":"260'28","last_scrub_stamp":"2024-03-11T21:45:20.088592+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:39.525554+0000","last_clean_scrub_stamp":"2024-03-11T21:45:20.088592+0000","objects_scrubbed":7,"log_size":42,"log_dups_size":0,"ondisk_log_size":42,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":61,"num_read_kb":54,"num_write":42,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,7],"acting":[6,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.0","version":"438'13","reported_seq":353,"reported_epoch":488,"state":"active+clean","last_fresh":"2024-03-11T21:47:35.477624+0000","last_change":"2024-03-11T21:47:12.660954+0000","last_active":"2024-03-11T21:47:35.477624+0000","last_peered":"2024-03-11T21:47:35.477624+0000","last_clean":"2024-03-11T21:47:35.477624+0000","last_became_active":"2024-03-11T21:44:35.621524+0000","last_became_peered":"2024-03-11T21:44:35.621524+0000","last_unstale":"2024-03-11T21:47:35.477624+0000","last_undegraded":"2024-03-11T21:47:35.477624+0000","last_fullsized":"2024-03-11T21:47:35.477624+0000","mapping_epoch":437,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":438,"parent":"0.0","parent_split_bits":0,"last_scrub":"438'13","last_scrub_stamp":"2024-03-11T21:47:12.660495+0000","last_deep_scrub":"98'9","last_deep_scrub_stamp":"2024-03-11T21:27:29.745711+0000","last_clean_scrub_stamp":"2024-03-11T21:47:12.660495+0000","objects_scrubbed":1,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:48:37.368079+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.1","version":"488'13384","reported_seq":13478,"reported_epoch":488,"state":"active+clean","last_fresh":"2024-03-11T21:47:37.671994+0000","last_change":"2024-03-11T21:44:53.663619+0000","last_active":"2024-03-11T21:47:37.671994+0000","last_peered":"2024-03-11T21:47:37.671994+0000","last_clean":"2024-03-11T21:47:37.671994+0000","last_became_active":"2024-03-11T21:44:53.663333+0000","last_became_peered":"2024-03-11T21:44:53.663333+0000","last_unstale":"2024-03-11T21:47:37.671994+0000","last_undegraded":"2024-03-11T21:47:37.671994+0000","last_fullsized":"2024-03-11T21:47:37.671994+0000","mapping_epoch":446,"log_start":"472'9948","ondisk_log_start":"472'9948","created":446,"last_epoch_clean":447,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:44:52.644443+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:44:52.644443+0000","last_clean_scrub_stamp":"2024-03-11T21:44:52.644443+0000","objects_scrubbed":0,"log_size":3436,"log_dups_size":0,"ondisk_log_size":3436,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":256507904,"num_objects":3914,"num_object_clones":0,"num_object_copies":11742,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3914,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13384,"num_write_kb":553536,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,2],"acting":[6,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.c","version":"0'0","reported_seq":526,"reported_epoch":488,"state":"active+clean","last_fresh":"2024-03-11T21:47:35.473795+0000","last_change":"2024-03-11T21:46:53.265642+0000","last_active":"2024-03-11T21:47:35.473795+0000","last_peered":"2024-03-11T21:47:35.473795+0000","last_clean":"2024-03-11T21:47:35.473795+0000","last_became_active":"2024-03-11T21:46:53.265318+0000","last_became_peered":"2024-03-11T21:46:53.265318+0000","last_unstale":"2024-03-11T21:47:35.473795+0000","last_undegraded":"2024-03-11T21:47:35.473795+0000","last_fullsized":"2024-03-11T21:47:35.473795+0000","mapping_epoch":473,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":474,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:44:24.425056+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:43.559452+0000","last_clean_scrub_stamp":"2024-03-11T21:44:24.425056+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,1],"acting":[6,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.f","version":"0'0","reported_seq":845,"reported_epoch":488,"state":"active+clean","last_fresh":"2024-03-11T21:47:35.474982+0000","last_change":"2024-03-11T21:46:53.265010+0000","last_active":"2024-03-11T21:47:35.474982+0000","last_peered":"2024-03-11T21:47:35.474982+0000","last_clean":"2024-03-11T21:47:35.474982+0000","last_became_active":"2024-03-11T21:46:53.264609+0000","last_became_peered":"2024-03-11T21:46:53.264609+0000","last_unstale":"2024-03-11T21:47:35.474982+0000","last_undegraded":"2024-03-11T21:47:35.474982+0000","last_fullsized":"2024-03-11T21:47:35.474982+0000","mapping_epoch":473,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":474,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:46:09.935636+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:43:24.002346+0000","last_clean_scrub_stamp":"2024-03-11T21:46:09.935636+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,1],"acting":[6,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.8","version":"0'0","reported_seq":562,"reported_epoch":488,"state":"active+clean","last_fresh":"2024-03-11T21:47:35.478790+0000","last_change":"2024-03-11T21:45:25.159896+0000","last_active":"2024-03-11T21:47:35.478790+0000","last_peered":"2024-03-11T21:47:35.478790+0000","last_clean":"2024-03-11T21:47:35.478790+0000","last_became_active":"2024-03-11T21:34:41.144681+0000","last_became_peered":"2024-03-11T21:34:41.144681+0000","last_unstale":"2024-03-11T21:47:35.478790+0000","last_undegraded":"2024-03-11T21:47:35.478790+0000","last_fullsized":"2024-03-11T21:47:35.478790+0000","mapping_epoch":226,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":227,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:45:25.159772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:37.512018+0000","last_clean_scrub_stamp":"2024-03-11T21:45:25.159772+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,1],"acting":[7,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.e","version":"456'32","reported_seq":634,"reported_epoch":488,"state":"active+clean","last_fresh":"2024-03-11T21:47:35.474166+0000","last_change":"2024-03-11T21:46:53.289428+0000","last_active":"2024-03-11T21:47:35.474166+0000","last_peered":"2024-03-11T21:47:35.474166+0000","last_clean":"2024-03-11T21:47:35.474166+0000","last_became_active":"2024-03-11T21:46:53.264561+0000","last_became_peered":"2024-03-11T21:46:53.264561+0000","last_unstale":"2024-03-11T21:47:35.474166+0000","last_undegraded":"2024-03-11T21:47:35.474166+0000","last_fullsized":"2024-03-11T21:47:35.474166+0000","mapping_epoch":473,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":474,"parent":"0.0","parent_split_bits":0,"last_scrub":"456'32","last_scrub_stamp":"2024-03-11T21:46:07.696550+0000","last_deep_scrub":"109'16","last_deep_scrub_stamp":"2024-03-11T21:28:24.739395+0000","last_clean_scrub_stamp":"2024-03-11T21:46:07.696550+0000","objects_scrubbed":5,"log_size":32,"log_dups_size":0,"ondisk_log_size":32,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":43,"num_read_kb":36,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,7],"acting":[6,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.0","version":"488'13395","reported_seq":13495,"reported_epoch":488,"state":"active+clean","last_fresh":"2024-03-11T21:47:37.606138+0000","last_change":"2024-03-11T21:47:20.293953+0000","last_active":"2024-03-11T21:47:37.606138+0000","last_peered":"2024-03-11T21:47:37.606138+0000","last_clean":"2024-03-11T21:47:37.606138+0000","last_became_active":"2024-03-11T21:47:20.274081+0000","last_became_peered":"2024-03-11T21:47:20.274081+0000","last_unstale":"2024-03-11T21:47:37.606138+0000","last_undegraded":"2024-03-11T21:47:37.606138+0000","last_fullsized":"2024-03-11T21:47:37.606138+0000","mapping_epoch":478,"log_start":"475'10002","ondisk_log_start":"475'10002","created":446,"last_epoch_clean":479,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:44:52.644443+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:44:52.644443+0000","last_clean_scrub_stamp":"2024-03-11T21:44:52.644443+0000","objects_scrubbed":0,"log_size":3393,"log_dups_size":0,"ondisk_log_size":3393,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":254345216,"num_objects":3881,"num_object_clones":0,"num_object_copies":11643,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3881,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13395,"num_write_kb":552832,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,6],"acting":[7,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.d","version":"0'0","reported_seq":32,"reported_epoch":488,"state":"active+clean","last_fresh":"2024-03-11T21:47:35.470545+0000","last_change":"2024-03-11T21:47:20.278751+0000","last_active":"2024-03-11T21:47:35.470545+0000","last_peered":"2024-03-11T21:47:35.470545+0000","last_clean":"2024-03-11T21:47:35.470545+0000","last_became_active":"2024-03-11T21:47:20.278406+0000","last_became_peered":"2024-03-11T21:47:20.278406+0000","last_unstale":"2024-03-11T21:47:35.470545+0000","last_undegraded":"2024-03-11T21:47:35.470545+0000","last_fullsized":"2024-03-11T21:47:35.470545+0000","mapping_epoch":478,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":479,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:44:33.036191+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:35:37.182850+0000","last_clean_scrub_stamp":"2024-03-11T21:44:33.036191+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.e","version":"474'10","reported_seq":79,"reported_epoch":488,"state":"active+clean","last_fresh":"2024-03-11T21:47:35.473062+0000","last_change":"2024-03-11T21:46:53.281249+0000","last_active":"2024-03-11T21:47:35.473062+0000","last_peered":"2024-03-11T21:47:35.473062+0000","last_clean":"2024-03-11T21:47:35.473062+0000","last_became_active":"2024-03-11T21:46:53.265891+0000","last_became_peered":"2024-03-11T21:46:53.265891+0000","last_unstale":"2024-03-11T21:47:35.473062+0000","last_undegraded":"2024-03-11T21:47:35.473062+0000","last_fullsized":"2024-03-11T21:47:35.473062+0000","mapping_epoch":473,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":474,"parent":"0.0","parent_split_bits":0,"last_scrub":"438'9","last_scrub_stamp":"2024-03-11T21:44:35.992899+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:41.544550+0000","last_clean_scrub_stamp":"2024-03-11T21:44:35.992899+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,7],"acting":[0,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.9","version":"0'0","reported_seq":946,"reported_epoch":488,"state":"active+clean","last_fresh":"2024-03-11T21:47:35.478524+0000","last_change":"2024-03-11T21:47:21.614548+0000","last_active":"2024-03-11T21:47:35.478524+0000","last_peered":"2024-03-11T21:47:35.478524+0000","last_clean":"2024-03-11T21:47:35.478524+0000","last_became_active":"2024-03-11T21:44:21.454167+0000","last_became_peered":"2024-03-11T21:44:21.454167+0000","last_unstale":"2024-03-11T21:47:35.478524+0000","last_undegraded":"2024-03-11T21:47:35.478524+0000","last_fullsized":"2024-03-11T21:47:35.478524+0000","mapping_epoch":432,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":433,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:47:21.614435+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:26:50.991712+0000","last_clean_scrub_stamp":"2024-03-11T21:47:21.614435+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:48:23.595722+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,2],"acting":[1,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.f","version":"456'39","reported_seq":55,"reported_epoch":488,"state":"active+clean","last_fresh":"2024-03-11T21:47:35.481217+0000","last_change":"2024-03-11T21:46:53.632020+0000","last_active":"2024-03-11T21:47:35.481217+0000","last_peered":"2024-03-11T21:47:35.481217+0000","last_clean":"2024-03-11T21:47:35.481217+0000","last_became_active":"2024-03-11T21:46:53.278498+0000","last_became_peered":"2024-03-11T21:46:53.278498+0000","last_unstale":"2024-03-11T21:47:35.481217+0000","last_undegraded":"2024-03-11T21:47:35.481217+0000","last_fullsized":"2024-03-11T21:47:35.481217+0000","mapping_epoch":473,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":474,"parent":"0.0","parent_split_bits":0,"last_scrub":"260'27","last_scrub_stamp":"2024-03-11T21:45:17.566447+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:39.525554+0000","last_clean_scrub_stamp":"2024-03-11T21:45:17.566447+0000","objects_scrubbed":7,"log_size":39,"log_dups_size":0,"ondisk_log_size":39,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":62,"num_read_kb":53,"num_write":40,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":34,"num_bytes_recovered":440,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,2],"acting":[1,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.3","version":"488'13528","reported_seq":13623,"reported_epoch":488,"state":"active+clean","last_fresh":"2024-03-11T21:47:37.671102+0000","last_change":"2024-03-11T21:44:53.666633+0000","last_active":"2024-03-11T21:47:37.671102+0000","last_peered":"2024-03-11T21:47:37.671102+0000","last_clean":"2024-03-11T21:47:37.671102+0000","last_became_active":"2024-03-11T21:44:53.666323+0000","last_became_peered":"2024-03-11T21:44:53.666323+0000","last_unstale":"2024-03-11T21:47:37.671102+0000","last_undegraded":"2024-03-11T21:47:37.671102+0000","last_fullsized":"2024-03-11T21:47:37.671102+0000","mapping_epoch":446,"log_start":"475'10103","ondisk_log_start":"475'10103","created":446,"last_epoch_clean":447,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:44:52.644443+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:44:52.644443+0000","last_clean_scrub_stamp":"2024-03-11T21:44:52.644443+0000","objects_scrubbed":0,"log_size":3425,"log_dups_size":0,"ondisk_log_size":3425,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":252837888,"num_objects":3858,"num_object_clones":0,"num_object_copies":11574,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3858,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13528,"num_write_kb":556352,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,1],"acting":[6,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.e","version":"0'0","reported_seq":127,"reported_epoch":488,"state":"active+clean","last_fresh":"2024-03-11T21:47:35.480871+0000","last_change":"2024-03-11T21:46:53.279665+0000","last_active":"2024-03-11T21:47:35.480871+0000","last_peered":"2024-03-11T21:47:35.480871+0000","last_clean":"2024-03-11T21:47:35.480871+0000","last_became_active":"2024-03-11T21:46:53.279138+0000","last_became_peered":"2024-03-11T21:46:53.279138+0000","last_unstale":"2024-03-11T21:47:35.480871+0000","last_undegraded":"2024-03-11T21:47:35.480871+0000","last_fullsized":"2024-03-11T21:47:35.480871+0000","mapping_epoch":473,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":474,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:45:17.769045+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:43.559452+0000","last_clean_scrub_stamp":"2024-03-11T21:45:17.769045+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,7],"acting":[1,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.d","version":"474'10","reported_seq":1199,"reported_epoch":488,"state":"active+clean","last_fresh":"2024-03-11T21:47:35.485760+0000","last_change":"2024-03-11T21:46:53.285991+0000","last_active":"2024-03-11T21:47:35.485760+0000","last_peered":"2024-03-11T21:47:35.485760+0000","last_clean":"2024-03-11T21:47:35.485760+0000","last_became_active":"2024-03-11T21:46:53.270121+0000","last_became_peered":"2024-03-11T21:46:53.270121+0000","last_unstale":"2024-03-11T21:47:35.485760+0000","last_undegraded":"2024-03-11T21:47:35.485760+0000","last_fullsized":"2024-03-11T21:47:35.485760+0000","mapping_epoch":473,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":474,"parent":"0.0","parent_split_bits":0,"last_scrub":"109'9","last_scrub_stamp":"2024-03-11T21:46:19.616235+0000","last_deep_scrub":"109'9","last_deep_scrub_stamp":"2024-03-11T21:28:27.082975+0000","last_clean_scrub_stamp":"2024-03-11T21:46:19.616235+0000","objects_scrubbed":1,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:47:40.985821+0000","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,0],"acting":[2,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.a","version":"0'0","reported_seq":952,"reported_epoch":488,"state":"active+clean","last_fresh":"2024-03-11T21:47:35.477585+0000","last_change":"2024-03-11T21:47:20.278478+0000","last_active":"2024-03-11T21:47:35.477585+0000","last_peered":"2024-03-11T21:47:35.477585+0000","last_clean":"2024-03-11T21:47:35.477585+0000","last_became_active":"2024-03-11T21:47:20.278149+0000","last_became_peered":"2024-03-11T21:47:20.278149+0000","last_unstale":"2024-03-11T21:47:35.477585+0000","last_undegraded":"2024-03-11T21:47:35.477585+0000","last_fullsized":"2024-03-11T21:47:35.477585+0000","mapping_epoch":478,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":479,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:46:24.954274+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:44:27.996522+0000","last_clean_scrub_stamp":"2024-03-11T21:46:24.954274+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:47:42.383540+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,2],"acting":[7,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.c","version":"456'30","reported_seq":608,"reported_epoch":488,"state":"active+clean","last_fresh":"2024-03-11T21:47:35.473593+0000","last_change":"2024-03-11T21:45:18.628865+0000","last_active":"2024-03-11T21:47:35.473593+0000","last_peered":"2024-03-11T21:47:35.473593+0000","last_clean":"2024-03-11T21:47:35.473593+0000","last_became_active":"2024-03-11T21:44:35.630913+0000","last_became_peered":"2024-03-11T21:44:35.630913+0000","last_unstale":"2024-03-11T21:47:35.473593+0000","last_undegraded":"2024-03-11T21:47:35.473593+0000","last_fullsized":"2024-03-11T21:47:35.473593+0000","mapping_epoch":437,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":438,"parent":"0.0","parent_split_bits":0,"last_scrub":"260'20","last_scrub_stamp":"2024-03-11T21:45:18.628728+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:39.525554+0000","last_clean_scrub_stamp":"2024-03-11T21:45:18.628728+0000","objects_scrubbed":5,"log_size":30,"log_dups_size":0,"ondisk_log_size":30,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":44,"num_read_kb":39,"num_write":30,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,7],"acting":[6,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.2","version":"488'13167","reported_seq":13262,"reported_epoch":488,"state":"active+clean","last_fresh":"2024-03-11T21:47:38.449551+0000","last_change":"2024-03-11T21:44:53.665467+0000","last_active":"2024-03-11T21:47:38.449551+0000","last_peered":"2024-03-11T21:47:38.449551+0000","last_clean":"2024-03-11T21:47:38.449551+0000","last_became_active":"2024-03-11T21:44:53.665142+0000","last_became_peered":"2024-03-11T21:44:53.665142+0000","last_unstale":"2024-03-11T21:47:38.449551+0000","last_undegraded":"2024-03-11T21:47:38.449551+0000","last_fullsized":"2024-03-11T21:47:38.449551+0000","mapping_epoch":446,"log_start":"472'9547","ondisk_log_start":"472'9547","created":446,"last_epoch_clean":447,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:44:52.644443+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:44:52.644443+0000","last_clean_scrub_stamp":"2024-03-11T21:44:52.644443+0000","objects_scrubbed":0,"log_size":3620,"log_dups_size":0,"ondisk_log_size":3620,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:47:38.777685+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":249757696,"num_objects":3811,"num_object_clones":0,"num_object_copies":11433,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3811,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13167,"num_write_kb":543296,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,7],"acting":[2,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.f","version":"0'0","reported_seq":933,"reported_epoch":488,"state":"active+clean","last_fresh":"2024-03-11T21:47:35.479304+0000","last_change":"2024-03-11T21:46:14.767906+0000","last_active":"2024-03-11T21:47:35.479304+0000","last_peered":"2024-03-11T21:47:35.479304+0000","last_clean":"2024-03-11T21:47:35.479304+0000","last_became_active":"2024-03-11T21:44:35.645282+0000","last_became_peered":"2024-03-11T21:44:35.645282+0000","last_unstale":"2024-03-11T21:47:35.479304+0000","last_undegraded":"2024-03-11T21:47:35.479304+0000","last_fullsized":"2024-03-11T21:47:35.479304+0000","mapping_epoch":437,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":438,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:46:14.767787+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:27:03.010969+0000","last_clean_scrub_stamp":"2024-03-11T21:46:14.767787+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:47:37.501602+0000","scrub_duration":217,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6,1],"acting":[2,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":828,"reported_epoch":488,"state":"active+clean","last_fresh":"2024-03-11T21:47:35.476463+0000","last_change":"2024-03-11T21:47:20.275198+0000","last_active":"2024-03-11T21:47:35.476463+0000","last_peered":"2024-03-11T21:47:35.476463+0000","last_clean":"2024-03-11T21:47:35.476463+0000","last_became_active":"2024-03-11T21:47:20.274869+0000","last_became_peered":"2024-03-11T21:47:20.274869+0000","last_unstale":"2024-03-11T21:47:35.476463+0000","last_undegraded":"2024-03-11T21:47:35.476463+0000","last_fullsized":"2024-03-11T21:47:35.476463+0000","mapping_epoch":478,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":479,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:45:18.824654+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:30:22.750053+0000","last_clean_scrub_stamp":"2024-03-11T21:45:18.824654+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,6],"acting":[0,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.b","version":"0'0","reported_seq":574,"reported_epoch":488,"state":"active+clean","last_fresh":"2024-03-11T21:47:35.479359+0000","last_change":"2024-03-11T21:46:53.259948+0000","last_active":"2024-03-11T21:47:35.479359+0000","last_peered":"2024-03-11T21:47:35.479359+0000","last_clean":"2024-03-11T21:47:35.479359+0000","last_became_active":"2024-03-11T21:46:53.259530+0000","last_became_peered":"2024-03-11T21:46:53.259530+0000","last_unstale":"2024-03-11T21:47:35.479359+0000","last_undegraded":"2024-03-11T21:47:35.479359+0000","last_fullsized":"2024-03-11T21:47:35.479359+0000","mapping_epoch":473,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":474,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:45:30.101973+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:45:30.101973+0000","last_clean_scrub_stamp":"2024-03-11T21:45:30.101973+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,2],"acting":[7,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.d","version":"456'53","reported_seq":652,"reported_epoch":488,"state":"active+clean","last_fresh":"2024-03-11T21:47:35.486096+0000","last_change":"2024-03-11T21:46:24.739804+0000","last_active":"2024-03-11T21:47:35.486096+0000","last_peered":"2024-03-11T21:47:35.486096+0000","last_clean":"2024-03-11T21:47:35.486096+0000","last_became_active":"2024-03-11T21:44:35.634778+0000","last_became_peered":"2024-03-11T21:44:35.634778+0000","last_unstale":"2024-03-11T21:47:35.486096+0000","last_undegraded":"2024-03-11T21:47:35.486096+0000","last_fullsized":"2024-03-11T21:47:35.486096+0000","mapping_epoch":437,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":438,"parent":"0.0","parent_split_bits":0,"last_scrub":"456'53","last_scrub_stamp":"2024-03-11T21:46:24.739434+0000","last_deep_scrub":"165'25","last_deep_scrub_stamp":"2024-03-11T21:32:21.678328+0000","last_clean_scrub_stamp":"2024-03-11T21:46:24.739434+0000","objects_scrubbed":8,"log_size":53,"log_dups_size":0,"ondisk_log_size":53,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:47:50.165880+0000","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":70,"num_read_kb":60,"num_write":46,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,6],"acting":[2,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"11.5","version":"488'13389","reported_seq":24993,"reported_epoch":488,"state":"active+clean","last_fresh":"2024-03-11T21:47:37.577737+0000","last_change":"2024-03-11T21:47:29.398053+0000","last_active":"2024-03-11T21:47:37.577737+0000","last_peered":"2024-03-11T21:47:37.577737+0000","last_clean":"2024-03-11T21:47:37.577737+0000","last_became_active":"2024-03-11T21:47:29.378875+0000","last_became_peered":"2024-03-11T21:47:29.378875+0000","last_unstale":"2024-03-11T21:47:37.577737+0000","last_undegraded":"2024-03-11T21:47:37.577737+0000","last_fullsized":"2024-03-11T21:47:37.577737+0000","mapping_epoch":482,"log_start":"475'10059","ondisk_log_start":"475'10059","created":446,"last_epoch_clean":483,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:44:52.644443+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:44:52.644443+0000","last_clean_scrub_stamp":"2024-03-11T21:44:52.644443+0000","objects_scrubbed":0,"log_size":3330,"log_dups_size":0,"ondisk_log_size":3330,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":262995968,"num_objects":4013,"num_object_clones":0,"num_object_copies":12039,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4013,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13389,"num_write_kb":556864,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5747,"num_bytes_recovered":376635392,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,0],"acting":[7,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":568,"reported_epoch":488,"state":"active+clean","last_fresh":"2024-03-11T21:47:35.476703+0000","last_change":"2024-03-11T21:46:05.407295+0000","last_active":"2024-03-11T21:47:35.476703+0000","last_peered":"2024-03-11T21:47:35.476703+0000","last_clean":"2024-03-11T21:47:35.476703+0000","last_became_active":"2024-03-11T21:44:21.461352+0000","last_became_peered":"2024-03-11T21:44:21.461352+0000","last_unstale":"2024-03-11T21:47:35.476703+0000","last_undegraded":"2024-03-11T21:47:35.476703+0000","last_fullsized":"2024-03-11T21:47:35.476703+0000","mapping_epoch":432,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":433,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:46:05.407166+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:28:37.561826+0000","last_clean_scrub_stamp":"2024-03-11T21:46:05.407166+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,6],"acting":[7,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":922,"reported_epoch":488,"state":"active+clean","last_fresh":"2024-03-11T21:47:35.477514+0000","last_change":"2024-03-11T21:46:53.264861+0000","last_active":"2024-03-11T21:47:35.477514+0000","last_peered":"2024-03-11T21:47:35.477514+0000","last_clean":"2024-03-11T21:47:35.477514+0000","last_became_active":"2024-03-11T21:46:53.264440+0000","last_became_peered":"2024-03-11T21:46:53.264440+0000","last_unstale":"2024-03-11T21:47:35.477514+0000","last_undegraded":"2024-03-11T21:47:35.477514+0000","last_fullsized":"2024-03-11T21:47:35.477514+0000","mapping_epoch":473,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":474,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:46:13.580254+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:41.544550+0000","last_clean_scrub_stamp":"2024-03-11T21:46:13.580254+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:47:38.778203+0000","scrub_duration":20,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,7],"acting":[2,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.c","version":"0'0","reported_seq":922,"reported_epoch":488,"state":"active+clean","last_fresh":"2024-03-11T21:47:35.478808+0000","last_change":"2024-03-11T21:46:53.277590+0000","last_active":"2024-03-11T21:47:35.478808+0000","last_peered":"2024-03-11T21:47:35.478808+0000","last_clean":"2024-03-11T21:47:35.478808+0000","last_became_active":"2024-03-11T21:46:53.276597+0000","last_became_peered":"2024-03-11T21:46:53.276597+0000","last_unstale":"2024-03-11T21:47:35.478808+0000","last_undegraded":"2024-03-11T21:47:35.478808+0000","last_fullsized":"2024-03-11T21:47:35.478808+0000","mapping_epoch":473,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":474,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:45:15.556610+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:29:30.279910+0000","last_clean_scrub_stamp":"2024-03-11T21:45:15.556610+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:47:38.777755+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.a","version":"456'60","reported_seq":1079,"reported_epoch":488,"state":"active+clean","last_fresh":"2024-03-11T21:47:35.473305+0000","last_change":"2024-03-11T21:45:32.673815+0000","last_active":"2024-03-11T21:47:35.473305+0000","last_peered":"2024-03-11T21:47:35.473305+0000","last_clean":"2024-03-11T21:47:35.473305+0000","last_became_active":"2024-03-11T21:34:41.148863+0000","last_became_peered":"2024-03-11T21:34:41.148863+0000","last_unstale":"2024-03-11T21:47:35.473305+0000","last_undegraded":"2024-03-11T21:47:35.473305+0000","last_fullsized":"2024-03-11T21:47:35.473305+0000","mapping_epoch":226,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":227,"parent":"0.0","parent_split_bits":0,"last_scrub":"260'44","last_scrub_stamp":"2024-03-11T21:45:32.673622+0000","last_deep_scrub":"260'44","last_deep_scrub_stamp":"2024-03-11T21:45:32.673622+0000","last_clean_scrub_stamp":"2024-03-11T21:45:32.673622+0000","objects_scrubbed":9,"log_size":60,"log_dups_size":0,"ondisk_log_size":60,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":80,"num_read_kb":69,"num_write":52,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,7],"acting":[6,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.4","version":"488'13387","reported_seq":13482,"reported_epoch":488,"state":"active+clean","last_fresh":"2024-03-11T21:47:37.600237+0000","last_change":"2024-03-11T21:44:53.667742+0000","last_active":"2024-03-11T21:47:37.600237+0000","last_peered":"2024-03-11T21:47:37.600237+0000","last_clean":"2024-03-11T21:47:37.600237+0000","last_became_active":"2024-03-11T21:44:53.667386+0000","last_became_peered":"2024-03-11T21:44:53.667386+0000","last_unstale":"2024-03-11T21:47:37.600237+0000","last_undegraded":"2024-03-11T21:47:37.600237+0000","last_fullsized":"2024-03-11T21:47:37.600237+0000","mapping_epoch":446,"log_start":"475'10030","ondisk_log_start":"475'10030","created":446,"last_epoch_clean":447,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:44:52.644443+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:44:52.644443+0000","last_clean_scrub_stamp":"2024-03-11T21:44:52.644443+0000","objects_scrubbed":0,"log_size":3357,"log_dups_size":0,"ondisk_log_size":3357,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":257622016,"num_objects":3931,"num_object_clones":0,"num_object_copies":11793,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3931,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13387,"num_write_kb":554176,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,2],"acting":[7,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":919,"reported_epoch":488,"state":"active+clean","last_fresh":"2024-03-11T21:47:35.470057+0000","last_change":"2024-03-11T21:44:40.345505+0000","last_active":"2024-03-11T21:47:35.470057+0000","last_peered":"2024-03-11T21:47:35.470057+0000","last_clean":"2024-03-11T21:47:35.470057+0000","last_became_active":"2024-03-11T21:34:41.154507+0000","last_became_peered":"2024-03-11T21:34:41.154507+0000","last_unstale":"2024-03-11T21:47:35.470057+0000","last_undegraded":"2024-03-11T21:47:35.470057+0000","last_fullsized":"2024-03-11T21:47:35.470057+0000","mapping_epoch":226,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":227,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:44:40.345386+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:27:11.361913+0000","last_clean_scrub_stamp":"2024-03-11T21:44:40.345386+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":82,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":930,"reported_epoch":488,"state":"active+clean","last_fresh":"2024-03-11T21:47:35.477614+0000","last_change":"2024-03-11T21:45:20.568882+0000","last_active":"2024-03-11T21:47:35.477614+0000","last_peered":"2024-03-11T21:47:35.477614+0000","last_clean":"2024-03-11T21:47:35.477614+0000","last_became_active":"2024-03-11T21:44:35.631810+0000","last_became_peered":"2024-03-11T21:44:35.631810+0000","last_unstale":"2024-03-11T21:47:35.477614+0000","last_undegraded":"2024-03-11T21:47:35.477614+0000","last_fullsized":"2024-03-11T21:47:35.477614+0000","mapping_epoch":437,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":438,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:45:20.567110+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:43:32.115556+0000","last_clean_scrub_stamp":"2024-03-11T21:45:20.567110+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:47:38.777897+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.d","version":"0'0","reported_seq":939,"reported_epoch":488,"state":"active+clean","last_fresh":"2024-03-11T21:47:35.489811+0000","last_change":"2024-03-11T21:46:12.969419+0000","last_active":"2024-03-11T21:47:35.489811+0000","last_peered":"2024-03-11T21:47:35.489811+0000","last_clean":"2024-03-11T21:47:35.489811+0000","last_became_active":"2024-03-11T21:44:35.621273+0000","last_became_peered":"2024-03-11T21:44:35.621273+0000","last_unstale":"2024-03-11T21:47:35.489811+0000","last_undegraded":"2024-03-11T21:47:35.489811+0000","last_fullsized":"2024-03-11T21:47:35.489811+0000","mapping_epoch":437,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":438,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:46:12.969312+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:37.512018+0000","last_clean_scrub_stamp":"2024-03-11T21:46:12.969312+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,6],"acting":[1,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.b","version":"456'21","reported_seq":980,"reported_epoch":488,"state":"active+clean","last_fresh":"2024-03-11T21:47:35.477936+0000","last_change":"2024-03-11T21:46:53.293795+0000","last_active":"2024-03-11T21:47:35.477936+0000","last_peered":"2024-03-11T21:47:35.477936+0000","last_clean":"2024-03-11T21:47:35.477936+0000","last_became_active":"2024-03-11T21:46:53.274132+0000","last_became_peered":"2024-03-11T21:46:53.274132+0000","last_unstale":"2024-03-11T21:47:35.477936+0000","last_undegraded":"2024-03-11T21:47:35.477936+0000","last_fullsized":"2024-03-11T21:47:35.477936+0000","mapping_epoch":473,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":474,"parent":"0.0","parent_split_bits":0,"last_scrub":"260'15","last_scrub_stamp":"2024-03-11T21:45:30.172740+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:39.525554+0000","last_clean_scrub_stamp":"2024-03-11T21:45:30.172740+0000","objects_scrubbed":4,"log_size":21,"log_dups_size":0,"ondisk_log_size":21,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":35,"num_read_kb":29,"num_write":22,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,6],"acting":[0,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.7","version":"488'13368","reported_seq":18251,"reported_epoch":488,"state":"active+remapped+backfilling","last_fresh":"2024-03-11T21:47:38.899063+0000","last_change":"2024-03-11T21:47:24.342277+0000","last_active":"2024-03-11T21:47:38.899063+0000","last_peered":"2024-03-11T21:47:38.899063+0000","last_clean":"2024-03-11T21:46:52.218494+0000","last_became_active":"2024-03-11T21:46:54.314366+0000","last_became_peered":"2024-03-11T21:46:54.314366+0000","last_unstale":"2024-03-11T21:47:38.899063+0000","last_undegraded":"2024-03-11T21:47:38.899063+0000","last_fullsized":"2024-03-11T21:47:38.899063+0000","mapping_epoch":474,"log_start":"476'10197","ondisk_log_start":"476'10197","created":446,"last_epoch_clean":447,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:44:52.644443+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:44:52.644443+0000","last_clean_scrub_stamp":"2024-03-11T21:44:52.644443+0000","objects_scrubbed":0,"log_size":3171,"log_dups_size":0,"ondisk_log_size":3171,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":249692160,"num_objects":3810,"num_object_clones":0,"num_object_copies":11430,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1730,"num_objects_unfound":0,"num_objects_dirty":3810,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13368,"num_write_kb":549696,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2390,"num_bytes_recovered":156631040,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,7],"acting":[1,0,5],"avail_no_missing":["1","0","5"],"object_location_counts":[{"shards":"0,1,5","objects":3810}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":41,"reported_epoch":488,"state":"active+clean","last_fresh":"2024-03-11T21:47:35.473243+0000","last_change":"2024-03-11T21:46:53.259253+0000","last_active":"2024-03-11T21:47:35.473243+0000","last_peered":"2024-03-11T21:47:35.473243+0000","last_clean":"2024-03-11T21:47:35.473243+0000","last_became_active":"2024-03-11T21:46:53.258659+0000","last_became_peered":"2024-03-11T21:46:53.258659+0000","last_unstale":"2024-03-11T21:47:35.473243+0000","last_undegraded":"2024-03-11T21:47:35.473243+0000","last_fullsized":"2024-03-11T21:47:35.473243+0000","mapping_epoch":473,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":474,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:45:34.114276+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:43.559452+0000","last_clean_scrub_stamp":"2024-03-11T21:45:34.114276+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,2],"acting":[6,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.e","version":"0'0","reported_seq":951,"reported_epoch":488,"state":"active+clean","last_fresh":"2024-03-11T21:47:35.477630+0000","last_change":"2024-03-11T21:44:28.438230+0000","last_active":"2024-03-11T21:47:35.477630+0000","last_peered":"2024-03-11T21:47:35.477630+0000","last_clean":"2024-03-11T21:47:35.477630+0000","last_became_active":"2024-03-11T21:44:21.457534+0000","last_became_peered":"2024-03-11T21:44:21.457534+0000","last_unstale":"2024-03-11T21:47:35.477630+0000","last_undegraded":"2024-03-11T21:47:35.477630+0000","last_fullsized":"2024-03-11T21:47:35.477630+0000","mapping_epoch":432,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":433,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:44:28.438139+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:37.512018+0000","last_clean_scrub_stamp":"2024-03-11T21:44:28.438139+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:47:38.775664+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,6],"acting":[2,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.8","version":"456'46","reported_seq":1015,"reported_epoch":488,"state":"active+clean","last_fresh":"2024-03-11T21:47:35.476429+0000","last_change":"2024-03-11T21:46:53.300361+0000","last_active":"2024-03-11T21:47:35.476429+0000","last_peered":"2024-03-11T21:47:35.476429+0000","last_clean":"2024-03-11T21:47:35.476429+0000","last_became_active":"2024-03-11T21:46:53.267405+0000","last_became_peered":"2024-03-11T21:46:53.267405+0000","last_unstale":"2024-03-11T21:47:35.476429+0000","last_undegraded":"2024-03-11T21:47:35.476429+0000","last_fullsized":"2024-03-11T21:47:35.476429+0000","mapping_epoch":473,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":474,"parent":"0.0","parent_split_bits":0,"last_scrub":"456'46","last_scrub_stamp":"2024-03-11T21:46:18.893747+0000","last_deep_scrub":"109'22","last_deep_scrub_stamp":"2024-03-11T21:32:07.573197+0000","last_clean_scrub_stamp":"2024-03-11T21:46:18.893747+0000","objects_scrubbed":7,"log_size":46,"log_dups_size":0,"ondisk_log_size":46,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":61,"num_read_kb":52,"num_write":40,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":21,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,1],"acting":[7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.9","version":"438'12","reported_seq":732,"reported_epoch":488,"state":"active+clean","last_fresh":"2024-03-11T21:47:35.477955+0000","last_change":"2024-03-11T21:45:27.165614+0000","last_active":"2024-03-11T21:47:35.477955+0000","last_peered":"2024-03-11T21:47:35.477955+0000","last_clean":"2024-03-11T21:47:35.477955+0000","last_became_active":"2024-03-11T21:44:35.638711+0000","last_became_peered":"2024-03-11T21:44:35.638711+0000","last_unstale":"2024-03-11T21:47:35.477955+0000","last_undegraded":"2024-03-11T21:47:35.477955+0000","last_fullsized":"2024-03-11T21:47:35.477955+0000","mapping_epoch":437,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":438,"parent":"0.0","parent_split_bits":0,"last_scrub":"438'12","last_scrub_stamp":"2024-03-11T21:45:27.165533+0000","last_deep_scrub":"94'9","last_deep_scrub_stamp":"2024-03-11T21:26:53.422552+0000","last_clean_scrub_stamp":"2024-03-11T21:45:27.165533+0000","objects_scrubbed":1,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,1],"acting":[7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":11,"num_pg":16,"stat_sum":{"num_bytes":4074700824,"num_objects":62176,"num_object_clones":0,"num_object_copies":186528,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":6867,"num_objects_unfound":0,"num_objects_dirty":62176,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":214516,"num_write_kb":8854081,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16859,"num_bytes_recovered":1104805888,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12461748224,"data_stored":12461736008,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":54607,"ondisk_log_size":54607,"up":48,"acting":48,"num_store_stats":6},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":1531,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":7,"num_read_kb":7,"num_write":10,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":2495,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":110592,"data_stored":6652,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":6,"ondisk_log_size":6,"up":96,"acting":96,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":198,"num_read_kb":294,"num_write":179,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":2951840,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":2375680,"data_stored":2361472,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":128,"ondisk_log_size":128,"up":3,"acting":3,"num_store_stats":4},{"poolid":3,"num_pg":3,"stat_sum":{"num_bytes":408,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":405,"num_read_kb":400,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":1281,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":8,"ondisk_log_size":8,"up":9,"acting":9,"num_store_stats":8},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":3702,"num_objects":178,"num_object_clones":0,"num_object_copies":534,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":178,"num_whiteouts":0,"num_read":1539,"num_read_kb":1296,"num_write":989,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":384,"num_bytes_recovered":7002,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":614400,"data_stored":16348,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1063,"ondisk_log_size":1063,"up":96,"acting":96,"num_store_stats":8},{"poolid":5,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":89,"ondisk_log_size":89,"up":96,"acting":96,"num_store_stats":8},{"poolid":6,"num_pg":32,"stat_sum":{"num_bytes":383,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":753,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":1558,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":4,"ondisk_log_size":4,"up":96,"acting":96,"num_store_stats":8}],"osd_stats":[{"osd":7,"up_from":60,"seq":257698038027,"num_pgs":91,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2977192,"kb_used_data":2173988,"kb_used_omap":1,"kb_used_meta":803070,"kb_avail":90767960,"statfs":{"total":95995035648,"available":92946391040,"internally_reserved":0,"allocated":2226163712,"data_stored":2225780886,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":822344138},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":50,"seq":214748365073,"num_pgs":89,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":3275796,"kb_used_data":2424140,"kb_used_omap":1,"kb_used_meta":851646,"kb_avail":90469356,"statfs":{"total":95995035648,"available":92640620544,"internally_reserved":0,"allocated":2482319360,"data_stored":2481897784,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":872085958},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834575147,"num_pgs":97,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":3680540,"kb_used_data":2696916,"kb_used_omap":1,"kb_used_meta":983550,"kb_avail":90064612,"statfs":{"total":95995035648,"available":92226162688,"internally_reserved":0,"allocated":2761641984,"data_stored":2761211089,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":1007155653},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738672,"num_pgs":83,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":3173428,"kb_used_data":2181548,"kb_used_omap":1,"kb_used_meta":991870,"kb_avail":90571724,"statfs":{"total":95995035648,"available":92745445376,"internally_reserved":0,"allocated":2233905152,"data_stored":2233514955,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":1015675333},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411621,"num_pgs":84,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2802984,"kb_used_data":2013792,"kb_used_omap":1,"kb_used_meta":789182,"kb_avail":90942168,"statfs":{"total":95995035648,"available":93124780032,"internally_reserved":0,"allocated":2062123008,"data_stored":2061745180,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":808122821},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":3,"up_from":0,"seq":1189705941126,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":0,"seq":137438953754,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":0,"seq":176093659413,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":1095,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":482,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1393,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":957,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":574,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":528,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1531,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":92,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":77824,"data_stored":2134,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":118784,"data_stored":3108,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":1678,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":1898,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":1474,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":69632,"data_stored":1870,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":86016,"data_stored":2272,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":69632,"data_stored":1914,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":370,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":383,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":383,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":383,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":2222915584,"data_stored":2222915584,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":2759790592,"data_stored":2759786520,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":2027425792,"data_stored":2027421720,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":787283968,"data_stored":787283968,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":2478313472,"data_stored":2478309400,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":2186018816,"data_stored":2186018816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-11T21:47:47.610 DEBUG:teuthology.orchestra.run.smithi092:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid 00db0016-dfed-11ee-95c8-87774f69a715 -- ceph osd pool set unique_pool_4 pgp_num 16 2024-03-11T21:47:47.614 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:47:47.615 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:47:47.616 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:47:47.716 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:47:47.769 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:47:47.769 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:47:47.769 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:47:47.870 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:47:47.899 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:47:47.899 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:47:47.899 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:47:48.000 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:47:48.041 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:47:48.041 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:47:48.042 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:47:48.142 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:47:48.176 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:47:48.177 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:47:48.177 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:47:48.278 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:47:48.311 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:47:48.311 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:47:48.312 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:47:48.368 INFO:journalctl@ceph.mgr.y.smithi092.stdout:Mar 11 21:47:47 smithi092 ceph-00db0016-dfed-11ee-95c8-87774f69a715-mgr-y[29801]: ::ffff:172.21.15.97 - - [11/Mar/2024:21:47:47] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-11T21:47:48.412 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:47:48.441 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:47:48.442 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:47:48.443 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:47:48.543 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:47:48.571 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:47:48.572 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:47:48.572 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:47:48.673 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:47:48.697 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:47:48.697 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:47:48.698 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:47:48.798 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:47:48.823 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:47:48.824 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:47:48.824 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:47:48.922 DEBUG:teuthology.orchestra.run.smithi092:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid 00db0016-dfed-11ee-95c8-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-03-11T21:47:48.925 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:47:48.948 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:47:48.949 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:47:48.949 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:47:49.050 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:47:49.074 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:47:49.074 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:47:49.075 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:47:49.175 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:47:49.199 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:47:49.199 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:47:49.200 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:47:49.301 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:47:49.325 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:47:49.326 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:47:49.327 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:47:49.427 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:47:49.460 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:47:49.460 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:47:49.461 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:47:49.561 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:47:49.585 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:47:49 smithi097 ceph-mon[33317]: pgmap v1048: 148 pgs: 1 active+remapped+backfill_toofull, 2 active+remapped+backfilling, 145 active+clean; 3.4 GiB data, 14 GiB used, 433 GiB / 447 GiB avail; 1023 B/s rd, 0 B/s wr, 1.38k op/s; 4903/166845 objects misplaced (2.939%); 17 MiB/s, 264 objects/s recovering 2024-03-11T21:47:49.585 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:47:49 smithi097 ceph-mon[33317]: osdmap e490: 8 total, 8 up, 5 in 2024-03-11T21:47:49.585 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:47:49 smithi097 ceph-mon[33317]: purged_snaps scrub starts 2024-03-11T21:47:49.585 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:47:49 smithi097 ceph-mon[33317]: purged_snaps scrub ok 2024-03-11T21:47:49.586 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:47:49.586 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:47:49.586 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:47:49.619 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:47:49 smithi092 ceph-mon[29572]: pgmap v1048: 148 pgs: 1 active+remapped+backfill_toofull, 2 active+remapped+backfilling, 145 active+clean; 3.4 GiB data, 14 GiB used, 433 GiB / 447 GiB avail; 1023 B/s rd, 0 B/s wr, 1.38k op/s; 4903/166845 objects misplaced (2.939%); 17 MiB/s, 264 objects/s recovering 2024-03-11T21:47:49.619 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:47:49 smithi092 ceph-mon[29572]: osdmap e490: 8 total, 8 up, 5 in 2024-03-11T21:47:49.619 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:47:49 smithi092 ceph-mon[29572]: purged_snaps scrub starts 2024-03-11T21:47:49.619 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:47:49 smithi092 ceph-mon[29572]: purged_snaps scrub ok 2024-03-11T21:47:49.619 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:47:49 smithi092 ceph-mon[37563]: pgmap v1048: 148 pgs: 1 active+remapped+backfill_toofull, 2 active+remapped+backfilling, 145 active+clean; 3.4 GiB data, 14 GiB used, 433 GiB / 447 GiB avail; 1023 B/s rd, 0 B/s wr, 1.38k op/s; 4903/166845 objects misplaced (2.939%); 17 MiB/s, 264 objects/s recovering 2024-03-11T21:47:49.619 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:47:49 smithi092 ceph-mon[37563]: osdmap e490: 8 total, 8 up, 5 in 2024-03-11T21:47:49.619 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:47:49 smithi092 ceph-mon[37563]: purged_snaps scrub starts 2024-03-11T21:47:49.619 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:47:49 smithi092 ceph-mon[37563]: purged_snaps scrub ok 2024-03-11T21:47:49.687 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:47:49.711 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:47:49.712 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:47:49.712 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:47:49.813 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:47:49.837 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:47:49.838 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:47:49.838 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:47:49.939 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:47:49.962 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:47:49.963 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:47:49.963 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:47:50.064 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:47:50.087 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:47:50.088 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:47:50.088 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:47:50.189 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:47:50.217 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:47:50.217 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:47:50.218 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:47:50.319 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:47:50.347 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:47:50.347 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:47:50.348 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:47:50.449 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:47:50.491 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:47:50.492 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:47:50.492 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:47:50.584 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:47:50 smithi097 ceph-mon[33317]: osdmap e491: 8 total, 8 up, 5 in 2024-03-11T21:47:50.594 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:47:50.618 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:47:50 smithi092 ceph-mon[29572]: osdmap e491: 8 total, 8 up, 5 in 2024-03-11T21:47:50.619 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:47:50 smithi092 ceph-mon[37563]: osdmap e491: 8 total, 8 up, 5 in 2024-03-11T21:47:50.620 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:47:50.621 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:47:50.621 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:47:50.722 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:47:50.754 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:47:50.755 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:47:50.755 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:47:50.856 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:47:50.880 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:47:50.881 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:47:50.882 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:47:50.983 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:47:51.015 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:47:51.016 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:47:51.017 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:47:51.117 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:47:51.144 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:47:51.145 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:47:51.145 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:47:51.246 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:47:51.285 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:47:51.286 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:47:51.286 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:47:51.387 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:47:51.426 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:47:51.427 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:47:51.430 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:47:51.531 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:47:51.571 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:47:51.572 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:47:51.572 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:47:51.584 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:47:51 smithi097 ceph-mon[33317]: pgmap v1051: 148 pgs: 1 active+remapped+backfill_toofull, 2 active+remapped+backfilling, 145 active+clean; 3.0 GiB data, 13 GiB used, 434 GiB / 447 GiB avail; 853 B/s rd, 0 B/s wr, 1.36k op/s; 3566/148701 objects misplaced (2.398%); 22 MiB/s, 352 objects/s recovering 2024-03-11T21:47:51.585 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:47:51 smithi097 ceph-mon[33317]: osdmap e492: 8 total, 8 up, 5 in 2024-03-11T21:47:51.618 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:47:51 smithi092 ceph-mon[29572]: pgmap v1051: 148 pgs: 1 active+remapped+backfill_toofull, 2 active+remapped+backfilling, 145 active+clean; 3.0 GiB data, 13 GiB used, 434 GiB / 447 GiB avail; 853 B/s rd, 0 B/s wr, 1.36k op/s; 3566/148701 objects misplaced (2.398%); 22 MiB/s, 352 objects/s recovering 2024-03-11T21:47:51.619 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:47:51 smithi092 ceph-mon[29572]: osdmap e492: 8 total, 8 up, 5 in 2024-03-11T21:47:51.619 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:47:51 smithi092 ceph-mon[37563]: pgmap v1051: 148 pgs: 1 active+remapped+backfill_toofull, 2 active+remapped+backfilling, 145 active+clean; 3.0 GiB data, 13 GiB used, 434 GiB / 447 GiB avail; 853 B/s rd, 0 B/s wr, 1.36k op/s; 3566/148701 objects misplaced (2.398%); 22 MiB/s, 352 objects/s recovering 2024-03-11T21:47:51.619 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:47:51 smithi092 ceph-mon[37563]: osdmap e492: 8 total, 8 up, 5 in 2024-03-11T21:47:51.673 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:47:51.711 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:47:51.711 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:47:51.712 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:47:51.813 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:47:51.852 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:47:51.853 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:47:51.853 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:47:51.954 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:47:51.986 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:47:51.987 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:47:51.987 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:47:52.088 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:47:52.121 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:47:52.122 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:47:52.122 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:47:52.223 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:47:52.255 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:47:52.255 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:47:52.255 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:47:52.356 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:47:52.395 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:47:52.395 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:47:52.396 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:47:52.496 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:47:52.524 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:47:52.525 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:47:52.525 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:47:52.618 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:47:52 smithi092 ceph-mon[29572]: purged_snaps scrub starts 2024-03-11T21:47:52.619 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:47:52 smithi092 ceph-mon[29572]: purged_snaps scrub ok 2024-03-11T21:47:52.619 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:47:52 smithi092 ceph-mon[29572]: osdmap e493: 8 total, 8 up, 5 in 2024-03-11T21:47:52.619 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:47:52 smithi092 ceph-mon[29572]: pgmap v1054: 148 pgs: 1 active+remapped+backfill_toofull, 1 active+remapped+backfilling, 146 active+clean; 3.0 GiB data, 13 GiB used, 434 GiB / 447 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.69k op/s; 3152/146535 objects misplaced (2.151%); 19 MiB/s, 301 objects/s recovering 2024-03-11T21:47:52.619 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:47:52 smithi092 ceph-mon[37563]: purged_snaps scrub starts 2024-03-11T21:47:52.619 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:47:52 smithi092 ceph-mon[37563]: purged_snaps scrub ok 2024-03-11T21:47:52.619 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:47:52 smithi092 ceph-mon[37563]: osdmap e493: 8 total, 8 up, 5 in 2024-03-11T21:47:52.619 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:47:52 smithi092 ceph-mon[37563]: pgmap v1054: 148 pgs: 1 active+remapped+backfill_toofull, 1 active+remapped+backfilling, 146 active+clean; 3.0 GiB data, 13 GiB used, 434 GiB / 447 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.69k op/s; 3152/146535 objects misplaced (2.151%); 19 MiB/s, 301 objects/s recovering 2024-03-11T21:47:52.626 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:47:52.658 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:47:52.659 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:47:52.659 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:47:52.760 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:47:52.789 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:47:52.789 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:47:52.790 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:47:52.834 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:47:52 smithi097 ceph-mon[33317]: purged_snaps scrub starts 2024-03-11T21:47:52.835 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:47:52 smithi097 ceph-mon[33317]: purged_snaps scrub ok 2024-03-11T21:47:52.835 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:47:52 smithi097 ceph-mon[33317]: osdmap e493: 8 total, 8 up, 5 in 2024-03-11T21:47:52.835 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:47:52 smithi097 ceph-mon[33317]: pgmap v1054: 148 pgs: 1 active+remapped+backfill_toofull, 1 active+remapped+backfilling, 146 active+clean; 3.0 GiB data, 13 GiB used, 434 GiB / 447 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.69k op/s; 3152/146535 objects misplaced (2.151%); 19 MiB/s, 301 objects/s recovering 2024-03-11T21:47:52.891 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:47:52.920 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:47:52.921 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:47:52.921 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:47:53.022 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:47:53.061 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:47:53.062 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:47:53.062 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:47:53.163 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:47:53.193 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:47:53.194 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:47:53.195 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:47:53.295 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:47:53.323 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:47:53.324 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:47:53.324 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:47:53.425 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:47:53.490 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:47:53.490 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:47:53.491 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:47:53.584 INFO:journalctl@ceph.iscsi.iscsi.a.smithi097.stdout:Mar 11 21:47:53 smithi097 ceph-00db0016-dfed-11ee-95c8-87774f69a715-iscsi-iscsi-a[60943]: debug there is no tcmu-runner data available 2024-03-11T21:47:53.585 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:47:53 smithi097 ceph-mon[33317]: from='client.24469 v1:172.21.15.97:0/404208310' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:47:53.585 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:47:53 smithi097 ceph-mon[33317]: from='mgr.14607 v1:172.21.15.92:0/1702717597' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T21:47:53.591 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:47:53.617 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:47:53.617 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:47:53.618 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:47:53.719 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:47:53.755 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:47:53 smithi092 ceph-mon[29572]: from='client.24469 v1:172.21.15.97:0/404208310' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:47:53.755 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:47:53 smithi092 ceph-mon[29572]: from='mgr.14607 v1:172.21.15.92:0/1702717597' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T21:47:53.755 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:47:53 smithi092 ceph-mon[37563]: from='client.24469 v1:172.21.15.97:0/404208310' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:47:53.755 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:47:53 smithi092 ceph-mon[37563]: from='mgr.14607 v1:172.21.15.92:0/1702717597' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T21:47:53.770 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:47:53.771 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:47:53.771 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:47:53.872 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:47:53.906 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:47:53.906 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:47:53.907 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:47:54.008 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:47:54.038 INFO:teuthology.orchestra.run.smithi092.stderr:Inferring config /var/lib/ceph/00db0016-dfed-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T21:47:54.066 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:47:54.066 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:47:54.067 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:47:54.107 INFO:teuthology.orchestra.run.smithi092.stderr:Inferring config /var/lib/ceph/00db0016-dfed-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T21:47:54.123 INFO:teuthology.orchestra.run.smithi092.stdout:{ 2024-03-11T21:47:54.123 INFO:teuthology.orchestra.run.smithi092.stdout: "ops": [ 2024-03-11T21:47:54.123 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:47:54.124 INFO:teuthology.orchestra.run.smithi092.stdout: "description": "osd_op(client.35813.0:234179 11.7 11:e4c4fd6f:::benchmark_data_smithi092_186520_object95831:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e493)", 2024-03-11T21:47:54.124 INFO:teuthology.orchestra.run.smithi092.stdout: "initiated_at": "2024-03-11T21:47:54.118385+0000", 2024-03-11T21:47:54.124 INFO:teuthology.orchestra.run.smithi092.stdout: "age": 0.010684648, 2024-03-11T21:47:54.124 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.010720136999999999, 2024-03-11T21:47:54.124 INFO:teuthology.orchestra.run.smithi092.stdout: "type_data": { 2024-03-11T21:47:54.124 INFO:teuthology.orchestra.run.smithi092.stdout: "flag_point": "waiting for sub ops", 2024-03-11T21:47:54.124 INFO:teuthology.orchestra.run.smithi092.stdout: "client_info": { 2024-03-11T21:47:54.124 INFO:teuthology.orchestra.run.smithi092.stdout: "client": "client.35813", 2024-03-11T21:47:54.124 INFO:teuthology.orchestra.run.smithi092.stdout: "client_addr": "v1:172.21.15.92:0/4293352030", 2024-03-11T21:47:54.124 INFO:teuthology.orchestra.run.smithi092.stdout: "tid": 234179 2024-03-11T21:47:54.124 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:47:54.124 INFO:teuthology.orchestra.run.smithi092.stdout: "events": [ 2024-03-11T21:47:54.124 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:47:54.124 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "initiated", 2024-03-11T21:47:54.125 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:47:54.118385+0000", 2024-03-11T21:47:54.125 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-03-11T21:47:54.125 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:47:54.125 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:47:54.125 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "header_read", 2024-03-11T21:47:54.125 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:47:54.118385+0000", 2024-03-11T21:47:54.125 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-03-11T21:47:54.125 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:47:54.125 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:47:54.125 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "throttled", 2024-03-11T21:47:54.125 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:47:54.118387+0000", 2024-03-11T21:47:54.125 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.798e-06 2024-03-11T21:47:54.125 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:47:54.126 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:47:54.126 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "all_read", 2024-03-11T21:47:54.126 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:47:54.118393+0000", 2024-03-11T21:47:54.126 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 6.3389999999999998e-06 2024-03-11T21:47:54.126 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:47:54.126 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:47:54.126 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "dispatched", 2024-03-11T21:47:54.126 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:47:54.118404+0000", 2024-03-11T21:47:54.126 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.0939e-05 2024-03-11T21:47:54.126 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:47:54.126 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:47:54.126 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "queued_for_pg", 2024-03-11T21:47:54.126 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:47:54.118412+0000", 2024-03-11T21:47:54.127 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 7.8520000000000004e-06 2024-03-11T21:47:54.127 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:47:54.127 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:47:54.127 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-03-11T21:47:54.127 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:47:54.118460+0000", 2024-03-11T21:47:54.127 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 4.7423000000000003e-05 2024-03-11T21:47:54.127 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:47:54.127 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:47:54.127 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "started", 2024-03-11T21:47:54.127 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:47:54.118589+0000", 2024-03-11T21:47:54.127 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00012940099999999999 2024-03-11T21:47:54.127 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:47:54.127 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:47:54.127 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "waiting for subops from 0,7", 2024-03-11T21:47:54.127 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:47:54.123712+0000", 2024-03-11T21:47:54.128 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.0051233629999999997 2024-03-11T21:47:54.128 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-03-11T21:47:54.128 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-03-11T21:47:54.128 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-03-11T21:47:54.128 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:47:54.128 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:47:54.128 INFO:teuthology.orchestra.run.smithi092.stdout: "description": "osd_repop(client.35813.0:234187 11.9 e493/446 11:913a7e57:::benchmark_data_smithi092_186520_object95839:head v 493'14442, mlcod=493'14440)", 2024-03-11T21:47:54.128 INFO:teuthology.orchestra.run.smithi092.stdout: "initiated_at": "2024-03-11T21:47:54.119302+0000", 2024-03-11T21:47:54.128 INFO:teuthology.orchestra.run.smithi092.stdout: "age": 0.0097675939999999992, 2024-03-11T21:47:54.128 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.0098916709999999995, 2024-03-11T21:47:54.128 INFO:teuthology.orchestra.run.smithi092.stdout: "type_data": { 2024-03-11T21:47:54.128 INFO:teuthology.orchestra.run.smithi092.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-11T21:47:54.128 INFO:teuthology.orchestra.run.smithi092.stdout: "events": [ 2024-03-11T21:47:54.129 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:47:54.129 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "initiated", 2024-03-11T21:47:54.129 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:47:54.119302+0000", 2024-03-11T21:47:54.129 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-03-11T21:47:54.129 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:47:54.129 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:47:54.129 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "header_read", 2024-03-11T21:47:54.129 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:47:54.119302+0000", 2024-03-11T21:47:54.129 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-03-11T21:47:54.129 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:47:54.129 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:47:54.129 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "throttled", 2024-03-11T21:47:54.129 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:47:54.119303+0000", 2024-03-11T21:47:54.129 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 5.5899999999999996e-07 2024-03-11T21:47:54.130 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:47:54.130 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:47:54.130 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "all_read", 2024-03-11T21:47:54.130 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:47:54.119308+0000", 2024-03-11T21:47:54.130 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 4.9490000000000004e-06 2024-03-11T21:47:54.130 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:47:54.130 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:47:54.130 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "dispatched", 2024-03-11T21:47:54.130 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:47:54.119314+0000", 2024-03-11T21:47:54.130 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 6.2369999999999997e-06 2024-03-11T21:47:54.130 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:47:54.130 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:47:54.130 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "queued_for_pg", 2024-03-11T21:47:54.130 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:47:54.119320+0000", 2024-03-11T21:47:54.131 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 5.9320000000000001e-06 2024-03-11T21:47:54.131 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:47:54.131 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:47:54.131 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-03-11T21:47:54.131 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:47:54.119393+0000", 2024-03-11T21:47:54.131 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 7.3415000000000001e-05 2024-03-11T21:47:54.131 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:47:54.131 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:47:54.131 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "started", 2024-03-11T21:47:54.131 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:47:54.119412+0000", 2024-03-11T21:47:54.131 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.8856e-05 2024-03-11T21:47:54.131 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:47:54.131 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:47:54.131 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "commit_sent", 2024-03-11T21:47:54.132 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:47:54.129132+0000", 2024-03-11T21:47:54.132 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.0097195119999999996 2024-03-11T21:47:54.132 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:47:54.132 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:47:54.132 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "done", 2024-03-11T21:47:54.132 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:47:54.129197+0000", 2024-03-11T21:47:54.132 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 6.4967000000000003e-05 2024-03-11T21:47:54.132 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-03-11T21:47:54.132 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-03-11T21:47:54.132 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-03-11T21:47:54.132 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:47:54.132 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:47:54.132 INFO:teuthology.orchestra.run.smithi092.stdout: "description": "osd_repop(client.35813.0:234190 11.d e493/446 11:bb905a73:::benchmark_data_smithi092_186520_object95842:head v 493'14909, mlcod=493'14906)", 2024-03-11T21:47:54.132 INFO:teuthology.orchestra.run.smithi092.stdout: "initiated_at": "2024-03-11T21:47:54.126240+0000", 2024-03-11T21:47:54.132 INFO:teuthology.orchestra.run.smithi092.stdout: "age": 0.0028297740000000002, 2024-03-11T21:47:54.133 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.0030214539999999998, 2024-03-11T21:47:54.133 INFO:teuthology.orchestra.run.smithi092.stdout: "type_data": { 2024-03-11T21:47:54.133 INFO:teuthology.orchestra.run.smithi092.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-11T21:47:54.133 INFO:teuthology.orchestra.run.smithi092.stdout: "events": [ 2024-03-11T21:47:54.133 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:47:54.133 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "initiated", 2024-03-11T21:47:54.133 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:47:54.126240+0000", 2024-03-11T21:47:54.133 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-03-11T21:47:54.133 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:47:54.133 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:47:54.133 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "header_read", 2024-03-11T21:47:54.133 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:47:54.126240+0000", 2024-03-11T21:47:54.133 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-03-11T21:47:54.133 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:47:54.134 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:47:54.134 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "throttled", 2024-03-11T21:47:54.134 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:47:54.126242+0000", 2024-03-11T21:47:54.134 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.773e-06 2024-03-11T21:47:54.134 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:47:54.134 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:47:54.134 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "all_read", 2024-03-11T21:47:54.134 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:47:54.126251+0000", 2024-03-11T21:47:54.134 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 9.6569999999999992e-06 2024-03-11T21:47:54.134 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:47:54.134 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:47:54.134 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "dispatched", 2024-03-11T21:47:54.134 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:47:54.126269+0000", 2024-03-11T21:47:54.134 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.7282000000000002e-05 2024-03-11T21:47:54.135 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:47:54.135 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:47:54.135 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "queued_for_pg", 2024-03-11T21:47:54.135 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:47:54.126278+0000", 2024-03-11T21:47:54.135 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 9.1349999999999998e-06 2024-03-11T21:47:54.135 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:47:54.135 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:47:54.135 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-03-11T21:47:54.135 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:47:54.126357+0000", 2024-03-11T21:47:54.135 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 7.8972000000000002e-05 2024-03-11T21:47:54.135 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:47:54.135 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:47:54.135 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "started", 2024-03-11T21:47:54.135 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:47:54.126380+0000", 2024-03-11T21:47:54.135 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 2.2796e-05 2024-03-11T21:47:54.136 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:47:54.136 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:47:54.136 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "commit_sent", 2024-03-11T21:47:54.136 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:47:54.129239+0000", 2024-03-11T21:47:54.136 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.002859707 2024-03-11T21:47:54.136 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:47:54.136 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:47:54.136 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "done", 2024-03-11T21:47:54.136 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:47:54.129262+0000", 2024-03-11T21:47:54.136 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 2.2132000000000001e-05 2024-03-11T21:47:54.136 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-03-11T21:47:54.136 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-03-11T21:47:54.136 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-03-11T21:47:54.136 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:47:54.137 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:47:54.137 INFO:teuthology.orchestra.run.smithi092.stdout: "description": "osd_repop(client.35813.0:234185 11.6 e493/446 11:6064f255:::benchmark_data_smithi092_186520_object95837:head v 493'14741, mlcod=493'14739)", 2024-03-11T21:47:54.137 INFO:teuthology.orchestra.run.smithi092.stdout: "initiated_at": "2024-03-11T21:47:54.128412+0000", 2024-03-11T21:47:54.137 INFO:teuthology.orchestra.run.smithi092.stdout: "age": 0.00065807599999999995, 2024-03-11T21:47:54.137 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00095224800000000005, 2024-03-11T21:47:54.137 INFO:teuthology.orchestra.run.smithi092.stdout: "type_data": { 2024-03-11T21:47:54.137 INFO:teuthology.orchestra.run.smithi092.stdout: "flag_point": "started", 2024-03-11T21:47:54.137 INFO:teuthology.orchestra.run.smithi092.stdout: "events": [ 2024-03-11T21:47:54.137 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:47:54.137 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "initiated", 2024-03-11T21:47:54.137 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:47:54.128412+0000", 2024-03-11T21:47:54.137 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-03-11T21:47:54.137 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:47:54.138 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:47:54.138 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "header_read", 2024-03-11T21:47:54.138 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:47:54.128412+0000", 2024-03-11T21:47:54.138 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-03-11T21:47:54.138 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:47:54.138 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:47:54.138 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "throttled", 2024-03-11T21:47:54.138 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:47:54.128414+0000", 2024-03-11T21:47:54.138 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.9609999999999999e-06 2024-03-11T21:47:54.138 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:47:54.138 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:47:54.138 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "all_read", 2024-03-11T21:47:54.138 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:47:54.128427+0000", 2024-03-11T21:47:54.138 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.3154e-05 2024-03-11T21:47:54.139 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:47:54.139 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:47:54.139 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "dispatched", 2024-03-11T21:47:54.139 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:47:54.128444+0000", 2024-03-11T21:47:54.139 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.6827999999999999e-05 2024-03-11T21:47:54.139 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:47:54.139 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:47:54.139 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "queued_for_pg", 2024-03-11T21:47:54.139 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:47:54.128453+0000", 2024-03-11T21:47:54.139 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 8.9579999999999996e-06 2024-03-11T21:47:54.139 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:47:54.139 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:47:54.139 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-03-11T21:47:54.139 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:47:54.128627+0000", 2024-03-11T21:47:54.140 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00017431900000000001 2024-03-11T21:47:54.140 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:47:54.140 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:47:54.140 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "started", 2024-03-11T21:47:54.140 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:47:54.128648+0000", 2024-03-11T21:47:54.140 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 2.0737000000000002e-05 2024-03-11T21:47:54.140 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-03-11T21:47:54.140 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-03-11T21:47:54.140 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-03-11T21:47:54.140 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:47:54.140 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:47:54.140 INFO:teuthology.orchestra.run.smithi092.stdout: "description": "osd_repop(client.35813.0:234191 11.b e493/490 11:dc79714e:::benchmark_data_smithi092_186520_object95843:head v 493'14600, mlcod=493'14598)", 2024-03-11T21:47:54.140 INFO:teuthology.orchestra.run.smithi092.stdout: "initiated_at": "2024-03-11T21:47:54.128479+0000", 2024-03-11T21:47:54.140 INFO:teuthology.orchestra.run.smithi092.stdout: "age": 0.00059058600000000004, 2024-03-11T21:47:54.140 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00094630599999999995, 2024-03-11T21:47:54.141 INFO:teuthology.orchestra.run.smithi092.stdout: "type_data": { 2024-03-11T21:47:54.141 INFO:teuthology.orchestra.run.smithi092.stdout: "flag_point": "started", 2024-03-11T21:47:54.141 INFO:teuthology.orchestra.run.smithi092.stdout: "events": [ 2024-03-11T21:47:54.141 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:47:54.141 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "initiated", 2024-03-11T21:47:54.141 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:47:54.128479+0000", 2024-03-11T21:47:54.141 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-03-11T21:47:54.141 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:47:54.141 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:47:54.141 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "header_read", 2024-03-11T21:47:54.141 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:47:54.128479+0000", 2024-03-11T21:47:54.141 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-03-11T21:47:54.141 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:47:54.141 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:47:54.142 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "throttled", 2024-03-11T21:47:54.142 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:47:54.128480+0000", 2024-03-11T21:47:54.142 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 6.8199999999999999e-07 2024-03-11T21:47:54.142 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:47:54.142 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:47:54.142 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "all_read", 2024-03-11T21:47:54.142 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:47:54.128485+0000", 2024-03-11T21:47:54.142 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 4.5920000000000002e-06 2024-03-11T21:47:54.142 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:47:54.142 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:47:54.142 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "dispatched", 2024-03-11T21:47:54.142 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:47:54.128490+0000", 2024-03-11T21:47:54.142 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 5.66e-06 2024-03-11T21:47:54.142 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:47:54.143 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:47:54.143 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "queued_for_pg", 2024-03-11T21:47:54.143 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:47:54.128495+0000", 2024-03-11T21:47:54.143 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 5.0939999999999998e-06 2024-03-11T21:47:54.143 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:47:54.143 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:47:54.143 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-03-11T21:47:54.143 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:47:54.129308+0000", 2024-03-11T21:47:54.143 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00081271299999999996 2024-03-11T21:47:54.143 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:47:54.143 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:47:54.143 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "started", 2024-03-11T21:47:54.143 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:47:54.129327+0000", 2024-03-11T21:47:54.143 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.8711000000000001e-05 2024-03-11T21:47:54.143 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-03-11T21:47:54.143 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-03-11T21:47:54.144 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-03-11T21:47:54.144 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:47:54.144 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:47:54.144 INFO:teuthology.orchestra.run.smithi092.stdout: "description": "osd_repop_reply(client.35813.0:234179 11.7 e493/491)", 2024-03-11T21:47:54.144 INFO:teuthology.orchestra.run.smithi092.stdout: "initiated_at": "2024-03-11T21:47:54.128508+0000", 2024-03-11T21:47:54.144 INFO:teuthology.orchestra.run.smithi092.stdout: "age": 0.00056147200000000003, 2024-03-11T21:47:54.144 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00097295300000000003, 2024-03-11T21:47:54.144 INFO:teuthology.orchestra.run.smithi092.stdout: "type_data": { 2024-03-11T21:47:54.144 INFO:teuthology.orchestra.run.smithi092.stdout: "flag_point": "queued for pg", 2024-03-11T21:47:54.144 INFO:teuthology.orchestra.run.smithi092.stdout: "events": [ 2024-03-11T21:47:54.144 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:47:54.144 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "initiated", 2024-03-11T21:47:54.144 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:47:54.128508+0000", 2024-03-11T21:47:54.144 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-03-11T21:47:54.145 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:47:54.145 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:47:54.145 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "header_read", 2024-03-11T21:47:54.145 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:47:54.128508+0000", 2024-03-11T21:47:54.145 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-03-11T21:47:54.145 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:47:54.145 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:47:54.145 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "throttled", 2024-03-11T21:47:54.145 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:47:54.128509+0000", 2024-03-11T21:47:54.145 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 4.2500000000000001e-07 2024-03-11T21:47:54.145 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:47:54.145 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:47:54.145 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "all_read", 2024-03-11T21:47:54.145 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:47:54.128512+0000", 2024-03-11T21:47:54.145 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 2.729e-06 2024-03-11T21:47:54.146 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:47:54.146 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:47:54.146 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "dispatched", 2024-03-11T21:47:54.146 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:47:54.128517+0000", 2024-03-11T21:47:54.146 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 5.2750000000000003e-06 2024-03-11T21:47:54.146 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:47:54.146 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:47:54.146 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "queued_for_pg", 2024-03-11T21:47:54.146 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:47:54.128521+0000", 2024-03-11T21:47:54.146 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 4.5950000000000001e-06 2024-03-11T21:47:54.146 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-03-11T21:47:54.146 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-03-11T21:47:54.146 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-03-11T21:47:54.146 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:47:54.147 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:47:54.147 INFO:teuthology.orchestra.run.smithi092.stdout: "description": "osd_repop_reply(client.35813.0:234179 11.7 e493/491)", 2024-03-11T21:47:54.147 INFO:teuthology.orchestra.run.smithi092.stdout: "initiated_at": "2024-03-11T21:47:54.128543+0000", 2024-03-11T21:47:54.147 INFO:teuthology.orchestra.run.smithi092.stdout: "age": 0.00052666900000000001, 2024-03-11T21:47:54.147 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00098240400000000009, 2024-03-11T21:47:54.147 INFO:teuthology.orchestra.run.smithi092.stdout: "type_data": { 2024-03-11T21:47:54.147 INFO:teuthology.orchestra.run.smithi092.stdout: "flag_point": "queued for pg", 2024-03-11T21:47:54.147 INFO:teuthology.orchestra.run.smithi092.stdout: "events": [ 2024-03-11T21:47:54.147 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:47:54.147 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "initiated", 2024-03-11T21:47:54.147 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:47:54.128543+0000", 2024-03-11T21:47:54.147 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-03-11T21:47:54.147 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:47:54.147 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:47:54.148 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "header_read", 2024-03-11T21:47:54.148 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:47:54.128543+0000", 2024-03-11T21:47:54.148 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-03-11T21:47:54.148 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:47:54.148 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:47:54.148 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "throttled", 2024-03-11T21:47:54.148 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:47:54.128544+0000", 2024-03-11T21:47:54.148 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 5.9100000000000004e-07 2024-03-11T21:47:54.148 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:47:54.148 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:47:54.148 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "all_read", 2024-03-11T21:47:54.148 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:47:54.128547+0000", 2024-03-11T21:47:54.148 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 3.27e-06 2024-03-11T21:47:54.148 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:47:54.148 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:47:54.149 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "dispatched", 2024-03-11T21:47:54.149 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:47:54.128553+0000", 2024-03-11T21:47:54.149 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 6.3319999999999997e-06 2024-03-11T21:47:54.149 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:47:54.149 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:47:54.149 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "queued_for_pg", 2024-03-11T21:47:54.149 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:47:54.128558+0000", 2024-03-11T21:47:54.149 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 4.9289999999999997e-06 2024-03-11T21:47:54.149 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-03-11T21:47:54.149 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-03-11T21:47:54.149 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-03-11T21:47:54.149 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:47:54.149 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:47:54.149 INFO:teuthology.orchestra.run.smithi092.stdout: "description": "osd_op(client.35813.0:234194 11.7 11.ae9953c7 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e493)", 2024-03-11T21:47:54.150 INFO:teuthology.orchestra.run.smithi092.stdout: "initiated_at": "2024-03-11T21:47:54.128846+0000", 2024-03-11T21:47:54.150 INFO:teuthology.orchestra.run.smithi092.stdout: "age": 0.000224148, 2024-03-11T21:47:54.150 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00072755200000000002, 2024-03-11T21:47:54.150 INFO:teuthology.orchestra.run.smithi092.stdout: "type_data": { 2024-03-11T21:47:54.150 INFO:teuthology.orchestra.run.smithi092.stdout: "flag_point": "queued for pg", 2024-03-11T21:47:54.150 INFO:teuthology.orchestra.run.smithi092.stdout: "client_info": { 2024-03-11T21:47:54.150 INFO:teuthology.orchestra.run.smithi092.stdout: "client": "client.35813", 2024-03-11T21:47:54.150 INFO:teuthology.orchestra.run.smithi092.stdout: "client_addr": "v1:172.21.15.92:0/4293352030", 2024-03-11T21:47:54.150 INFO:teuthology.orchestra.run.smithi092.stdout: "tid": 234194 2024-03-11T21:47:54.150 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:47:54.150 INFO:teuthology.orchestra.run.smithi092.stdout: "events": [ 2024-03-11T21:47:54.150 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:47:54.151 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "initiated", 2024-03-11T21:47:54.151 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:47:54.128846+0000", 2024-03-11T21:47:54.151 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-03-11T21:47:54.151 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:47:54.151 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:47:54.151 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "header_read", 2024-03-11T21:47:54.151 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:47:54.128846+0000", 2024-03-11T21:47:54.151 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-03-11T21:47:54.151 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:47:54.151 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:47:54.151 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "throttled", 2024-03-11T21:47:54.151 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:47:54.128851+0000", 2024-03-11T21:47:54.151 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 4.9660000000000004e-06 2024-03-11T21:47:54.151 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:47:54.151 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:47:54.152 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "all_read", 2024-03-11T21:47:54.152 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:47:54.128858+0000", 2024-03-11T21:47:54.152 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 7.0140000000000001e-06 2024-03-11T21:47:54.152 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:47:54.152 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:47:54.152 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "dispatched", 2024-03-11T21:47:54.152 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:47:54.128870+0000", 2024-03-11T21:47:54.152 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.255e-05 2024-03-11T21:47:54.152 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:47:54.152 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:47:54.152 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "queued_for_pg", 2024-03-11T21:47:54.152 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:47:54.128878+0000", 2024-03-11T21:47:54.152 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 7.7919999999999993e-06 2024-03-11T21:47:54.152 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-03-11T21:47:54.153 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-03-11T21:47:54.153 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-03-11T21:47:54.153 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:47:54.153 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:47:54.153 INFO:teuthology.orchestra.run.smithi092.stdout: "description": "osd_op(client.35813.0:234197 11.e 11.f207c9ee (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e493)", 2024-03-11T21:47:54.153 INFO:teuthology.orchestra.run.smithi092.stdout: "initiated_at": "2024-03-11T21:47:54.128897+0000", 2024-03-11T21:47:54.153 INFO:teuthology.orchestra.run.smithi092.stdout: "age": 0.000172559, 2024-03-11T21:47:54.153 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00073270099999999999, 2024-03-11T21:47:54.153 INFO:teuthology.orchestra.run.smithi092.stdout: "type_data": { 2024-03-11T21:47:54.153 INFO:teuthology.orchestra.run.smithi092.stdout: "flag_point": "queued for pg", 2024-03-11T21:47:54.153 INFO:teuthology.orchestra.run.smithi092.stdout: "client_info": { 2024-03-11T21:47:54.153 INFO:teuthology.orchestra.run.smithi092.stdout: "client": "client.35813", 2024-03-11T21:47:54.153 INFO:teuthology.orchestra.run.smithi092.stdout: "client_addr": "v1:172.21.15.92:0/4293352030", 2024-03-11T21:47:54.153 INFO:teuthology.orchestra.run.smithi092.stdout: "tid": 234197 2024-03-11T21:47:54.153 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:47:54.154 INFO:teuthology.orchestra.run.smithi092.stdout: "events": [ 2024-03-11T21:47:54.154 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:47:54.154 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "initiated", 2024-03-11T21:47:54.154 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:47:54.128897+0000", 2024-03-11T21:47:54.154 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-03-11T21:47:54.154 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:47:54.154 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:47:54.154 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "header_read", 2024-03-11T21:47:54.154 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:47:54.128897+0000", 2024-03-11T21:47:54.154 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-03-11T21:47:54.154 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:47:54.154 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:47:54.154 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "throttled", 2024-03-11T21:47:54.154 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:47:54.128898+0000", 2024-03-11T21:47:54.155 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 6.9400000000000005e-07 2024-03-11T21:47:54.155 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:47:54.155 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:47:54.155 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "all_read", 2024-03-11T21:47:54.155 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:47:54.128901+0000", 2024-03-11T21:47:54.155 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 2.999e-06 2024-03-11T21:47:54.155 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:47:54.155 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:47:54.155 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "dispatched", 2024-03-11T21:47:54.155 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:47:54.128907+0000", 2024-03-11T21:47:54.155 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 5.536e-06 2024-03-11T21:47:54.155 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:47:54.155 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:47:54.155 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "queued_for_pg", 2024-03-11T21:47:54.156 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:47:54.128912+0000", 2024-03-11T21:47:54.156 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 5.3410000000000002e-06 2024-03-11T21:47:54.156 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-03-11T21:47:54.156 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-03-11T21:47:54.156 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-03-11T21:47:54.156 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:47:54.156 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:47:54.156 INFO:teuthology.orchestra.run.smithi092.stdout: "description": "osd_repop(client.35813.0:234193 11.0 e493/478 11:066c7bf9:::benchmark_data_smithi092_186520_object95845:head v 493'14696, mlcod=493'14694)", 2024-03-11T21:47:54.156 INFO:teuthology.orchestra.run.smithi092.stdout: "initiated_at": "2024-03-11T21:47:54.129114+0000", 2024-03-11T21:47:54.156 INFO:teuthology.orchestra.run.smithi092.stdout: "age": 4294967295.9999561, 2024-03-11T21:47:54.156 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00057261900000000001, 2024-03-11T21:47:54.156 INFO:teuthology.orchestra.run.smithi092.stdout: "type_data": { 2024-03-11T21:47:54.156 INFO:teuthology.orchestra.run.smithi092.stdout: "flag_point": "started", 2024-03-11T21:47:54.156 INFO:teuthology.orchestra.run.smithi092.stdout: "events": [ 2024-03-11T21:47:54.157 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:47:54.157 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "initiated", 2024-03-11T21:47:54.157 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:47:54.129114+0000", 2024-03-11T21:47:54.157 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-03-11T21:47:54.157 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:47:54.157 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:47:54.157 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "header_read", 2024-03-11T21:47:54.157 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:47:54.129114+0000", 2024-03-11T21:47:54.157 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-03-11T21:47:54.157 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:47:54.157 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:47:54.157 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "throttled", 2024-03-11T21:47:54.157 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:47:54.129115+0000", 2024-03-11T21:47:54.157 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.1829999999999999e-06 2024-03-11T21:47:54.157 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:47:54.158 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:47:54.158 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "all_read", 2024-03-11T21:47:54.158 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:47:54.129121+0000", 2024-03-11T21:47:54.158 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 6.3990000000000001e-06 2024-03-11T21:47:54.158 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:47:54.158 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:47:54.158 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "dispatched", 2024-03-11T21:47:54.158 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:47:54.129131+0000", 2024-03-11T21:47:54.158 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 9.2010000000000005e-06 2024-03-11T21:47:54.158 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:47:54.158 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:47:54.158 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "queued_for_pg", 2024-03-11T21:47:54.158 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:47:54.129137+0000", 2024-03-11T21:47:54.159 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 6.7909999999999999e-06 2024-03-11T21:47:54.159 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:47:54.159 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:47:54.159 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-03-11T21:47:54.159 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:47:54.129204+0000", 2024-03-11T21:47:54.159 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 6.6507999999999994e-05 2024-03-11T21:47:54.159 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:47:54.159 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:47:54.159 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "started", 2024-03-11T21:47:54.159 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:47:54.129223+0000", 2024-03-11T21:47:54.159 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.9555000000000002e-05 2024-03-11T21:47:54.159 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-03-11T21:47:54.159 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-03-11T21:47:54.159 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-03-11T21:47:54.159 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-03-11T21:47:54.160 INFO:teuthology.orchestra.run.smithi092.stdout: ], 2024-03-11T21:47:54.160 INFO:teuthology.orchestra.run.smithi092.stdout: "num_ops": 10 2024-03-11T21:47:54.160 INFO:teuthology.orchestra.run.smithi092.stdout:} 2024-03-11T21:47:54.167 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:47:54.198 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:47:54.198 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:47:54.198 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:47:54.299 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:47:54.342 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:47:54.342 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:47:54.343 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:47:54.443 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:47:54.474 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:47:54.475 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:47:54.475 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:47:54.576 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:47:54.599 INFO:teuthology.orchestra.run.smithi092.stderr:nodeep-scrub is set 2024-03-11T21:47:54.606 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:47:54.607 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:47:54.608 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:47:54.708 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:47:54.740 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:47:54.741 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:47:54.741 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:47:54.834 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:47:54 smithi097 ceph-mon[33317]: pgmap v1055: 148 pgs: 1 active+remapped+backfill_toofull, 1 active+remapped+backfilling, 146 active+clean; 2.8 GiB data, 13 GiB used, 434 GiB / 447 GiB avail; 1.9 KiB/s rd, 0 B/s wr, 1.17k op/s; 2876/136626 objects misplaced (2.105%); 15 MiB/s, 233 objects/s recovering 2024-03-11T21:47:54.835 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:47:54 smithi097 ceph-mon[33317]: from='client.? v1:172.21.15.92:0/3877485214' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-11T21:47:54.842 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:47:54.868 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:47:54 smithi092 ceph-mon[29572]: pgmap v1055: 148 pgs: 1 active+remapped+backfill_toofull, 1 active+remapped+backfilling, 146 active+clean; 2.8 GiB data, 13 GiB used, 434 GiB / 447 GiB avail; 1.9 KiB/s rd, 0 B/s wr, 1.17k op/s; 2876/136626 objects misplaced (2.105%); 15 MiB/s, 233 objects/s recovering 2024-03-11T21:47:54.869 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:47:54 smithi092 ceph-mon[29572]: from='client.? v1:172.21.15.92:0/3877485214' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-11T21:47:54.869 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:47:54 smithi092 ceph-mon[37563]: pgmap v1055: 148 pgs: 1 active+remapped+backfill_toofull, 1 active+remapped+backfilling, 146 active+clean; 2.8 GiB data, 13 GiB used, 434 GiB / 447 GiB avail; 1.9 KiB/s rd, 0 B/s wr, 1.17k op/s; 2876/136626 objects misplaced (2.105%); 15 MiB/s, 233 objects/s recovering 2024-03-11T21:47:54.869 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:47:54 smithi092 ceph-mon[37563]: from='client.? v1:172.21.15.92:0/3877485214' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-03-11T21:47:54.873 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:47:54.874 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:47:54.874 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:47:54.975 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:47:55.007 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:47:55.007 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:47:55.008 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:47:55.109 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:47:55.155 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:47:55.156 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:47:55.156 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:47:55.257 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:47:55.289 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:47:55.289 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:47:55.290 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:47:55.390 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:47:55.421 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:47:55.422 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:47:55.422 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:47:55.523 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:47:55.560 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:47:55.561 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:47:55.561 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:47:55.619 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:47:55 smithi092 ceph-mon[29572]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-11T21:47:55.619 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:47:55 smithi092 ceph-mon[29572]: from='client.? v1:172.21.15.92:0/3877485214' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-03-11T21:47:55.619 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:47:55 smithi092 ceph-mon[29572]: osdmap e494: 8 total, 8 up, 5 in 2024-03-11T21:47:55.619 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:47:55 smithi092 ceph-mon[37563]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-11T21:47:55.619 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:47:55 smithi092 ceph-mon[37563]: from='client.? v1:172.21.15.92:0/3877485214' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-03-11T21:47:55.619 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:47:55 smithi092 ceph-mon[37563]: osdmap e494: 8 total, 8 up, 5 in 2024-03-11T21:47:55.662 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:47:55.704 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:47:55.705 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:47:55.705 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:47:55.806 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:47:55.834 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:47:55 smithi097 ceph-mon[33317]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-11T21:47:55.834 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:47:55 smithi097 ceph-mon[33317]: from='client.? v1:172.21.15.92:0/3877485214' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-03-11T21:47:55.835 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:47:55 smithi097 ceph-mon[33317]: osdmap e494: 8 total, 8 up, 5 in 2024-03-11T21:47:55.843 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:47:55.844 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:47:55.845 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:47:55.945 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:47:55.984 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:47:55.984 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:47:55.985 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:47:56.086 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:47:56.125 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:47:56.126 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:47:56.127 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:47:56.228 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:47:56.257 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:47:56.258 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:47:56.258 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:47:56.359 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:47:56.412 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:47:56.412 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:47:56.413 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:47:56.514 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:47:56.554 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:47:56.555 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:47:56.556 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:47:56.656 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:47:56.693 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:47:56.693 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:47:56.693 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:47:56.794 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:47:56.835 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:47:56.835 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:47:56.836 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:47:56.868 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:47:56 smithi092 ceph-mon[29572]: pgmap v1057: 148 pgs: 1 active+remapped+backfill_toofull, 147 active+clean; 2.7 GiB data, 12 GiB used, 435 GiB / 447 GiB avail; 853 B/s rd, 0 B/s wr, 965 op/s; 2788/131325 objects misplaced (2.123%); 5.0 MiB/s, 80 objects/s recovering 2024-03-11T21:47:56.869 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:47:56 smithi092 ceph-mon[37563]: pgmap v1057: 148 pgs: 1 active+remapped+backfill_toofull, 147 active+clean; 2.7 GiB data, 12 GiB used, 435 GiB / 447 GiB avail; 853 B/s rd, 0 B/s wr, 965 op/s; 2788/131325 objects misplaced (2.123%); 5.0 MiB/s, 80 objects/s recovering 2024-03-11T21:47:56.936 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:47:56.967 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:47:56.967 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:47:56.967 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:47:57.068 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:47:57.084 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:47:56 smithi097 ceph-mon[33317]: pgmap v1057: 148 pgs: 1 active+remapped+backfill_toofull, 147 active+clean; 2.7 GiB data, 12 GiB used, 435 GiB / 447 GiB avail; 853 B/s rd, 0 B/s wr, 965 op/s; 2788/131325 objects misplaced (2.123%); 5.0 MiB/s, 80 objects/s recovering 2024-03-11T21:47:57.116 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:47:57.117 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:47:57.117 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:47:57.218 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:47:57.256 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:47:57.256 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:47:57.257 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:47:57.358 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:47:57.388 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:47:57.389 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:47:57.389 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:47:57.490 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:47:57.528 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:47:57.528 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:47:57.529 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:47:57.630 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:47:57.661 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:47:57.661 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:47:57.662 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:47:57.762 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:47:57.805 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:47:57.806 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:47:57.807 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:47:57.907 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:47:57.956 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:47:57.957 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:47:57.957 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:47:57.964 INFO:teuthology.orchestra.run.smithi092.stdout:osd.0: {} 2024-03-11T21:47:58.013 INFO:teuthology.orchestra.run.smithi092.stdout:osd.1: {} 2024-03-11T21:47:58.013 INFO:journalctl@ceph.mgr.y.smithi092.stdout:Mar 11 21:47:57 smithi092 ceph-00db0016-dfed-11ee-95c8-87774f69a715-mgr-y[29801]: ::ffff:172.21.15.97 - - [11/Mar/2024:21:47:57] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-11T21:47:58.058 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:47:58.059 INFO:teuthology.orchestra.run.smithi092.stdout:osd.2: {} 2024-03-11T21:47:58.090 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:47:58.091 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:47:58.091 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:47:58.097 INFO:teuthology.orchestra.run.smithi092.stdout:osd.3: {} 2024-03-11T21:47:58.138 INFO:teuthology.orchestra.run.smithi092.stdout:osd.4: {} 2024-03-11T21:47:58.174 INFO:teuthology.orchestra.run.smithi092.stdout:osd.5: {} 2024-03-11T21:47:58.192 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:47:58.214 INFO:teuthology.orchestra.run.smithi092.stdout:osd.6: {} 2024-03-11T21:47:58.223 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:47:58.223 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:47:58.223 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:47:58.254 INFO:teuthology.orchestra.run.smithi092.stdout:osd.7: {} 2024-03-11T21:47:58.259 INFO:teuthology.orchestra.run.smithi092.stderr:osd.0: osd_enable_op_tracker = '' 2024-03-11T21:47:58.259 INFO:teuthology.orchestra.run.smithi092.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-11T21:47:58.259 INFO:teuthology.orchestra.run.smithi092.stderr:osd.2: osd_enable_op_tracker = '' 2024-03-11T21:47:58.259 INFO:teuthology.orchestra.run.smithi092.stderr:osd.3: osd_enable_op_tracker = '' 2024-03-11T21:47:58.259 INFO:teuthology.orchestra.run.smithi092.stderr:osd.4: osd_enable_op_tracker = '' 2024-03-11T21:47:58.259 INFO:teuthology.orchestra.run.smithi092.stderr:osd.5: osd_enable_op_tracker = '' 2024-03-11T21:47:58.259 INFO:teuthology.orchestra.run.smithi092.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-11T21:47:58.259 INFO:teuthology.orchestra.run.smithi092.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-11T21:47:58.324 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:47:58.347 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:47:58.348 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:47:58.348 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:47:58.449 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:47:58.476 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:47:58.477 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:47:58.477 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:47:58.482 DEBUG:teuthology.orchestra.run.smithi092:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid 00db0016-dfed-11ee-95c8-87774f69a715 -- ceph daemon osd.1 dump_blocked_ops 2024-03-11T21:47:58.578 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:47:58.608 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:47:58.609 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:47:58.609 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:47:58.710 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:47:58.741 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:47:58.742 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:47:58.743 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:47:58.843 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:47:58.869 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:47:58 smithi092 ceph-mon[29572]: pgmap v1058: 148 pgs: 1 active+remapped+backfill_toofull, 147 active+clean; 2.7 GiB data, 12 GiB used, 435 GiB / 447 GiB avail; 715 B/s rd, 0 B/s wr, 708 op/s; 2788/131325 objects misplaced (2.123%); 778 KiB/s, 12 objects/s recovering 2024-03-11T21:47:58.869 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:47:58 smithi092 ceph-mon[29572]: from='client.? v1:172.21.15.92:0/1484142344' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-11T21:47:58.869 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:47:58 smithi092 ceph-mon[37563]: pgmap v1058: 148 pgs: 1 active+remapped+backfill_toofull, 147 active+clean; 2.7 GiB data, 12 GiB used, 435 GiB / 447 GiB avail; 715 B/s rd, 0 B/s wr, 708 op/s; 2788/131325 objects misplaced (2.123%); 778 KiB/s, 12 objects/s recovering 2024-03-11T21:47:58.869 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:47:58 smithi092 ceph-mon[37563]: from='client.? v1:172.21.15.92:0/1484142344' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-11T21:47:58.893 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:47:58.893 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:47:58.894 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:47:58.995 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:47:59.022 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:47:59.023 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:47:59.024 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:47:59.085 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:47:58 smithi097 ceph-mon[33317]: pgmap v1058: 148 pgs: 1 active+remapped+backfill_toofull, 147 active+clean; 2.7 GiB data, 12 GiB used, 435 GiB / 447 GiB avail; 715 B/s rd, 0 B/s wr, 708 op/s; 2788/131325 objects misplaced (2.123%); 778 KiB/s, 12 objects/s recovering 2024-03-11T21:47:59.085 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:47:58 smithi097 ceph-mon[33317]: from='client.? v1:172.21.15.92:0/1484142344' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-11T21:47:59.125 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:47:59.128 DEBUG:teuthology.orchestra.run.smithi092:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid 00db0016-dfed-11ee-95c8-87774f69a715 -- ceph osd unset noscrub 2024-03-11T21:47:59.151 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:47:59.152 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:47:59.152 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:47:59.253 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:47:59.308 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:47:59.309 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:47:59.309 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:47:59.410 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:47:59.435 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:47:59.436 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:47:59.436 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:47:59.537 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:47:59.564 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:47:59.565 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:47:59.565 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:47:59.666 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:47:59.698 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:47:59.699 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:47:59.699 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:47:59.800 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:47:59.838 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:47:59.838 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:47:59.839 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:47:59.869 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:47:59 smithi092 ceph-mon[29572]: purged_snaps scrub starts 2024-03-11T21:47:59.869 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:47:59 smithi092 ceph-mon[29572]: purged_snaps scrub ok 2024-03-11T21:47:59.869 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:47:59 smithi092 ceph-mon[29572]: Health check cleared: PG_BACKFILL_FULL (was: Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull) 2024-03-11T21:47:59.869 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:47:59 smithi092 ceph-mon[29572]: osdmap e495: 8 total, 8 up, 5 in 2024-03-11T21:47:59.869 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:47:59 smithi092 ceph-mon[37563]: purged_snaps scrub starts 2024-03-11T21:47:59.869 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:47:59 smithi092 ceph-mon[37563]: purged_snaps scrub ok 2024-03-11T21:47:59.870 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:47:59 smithi092 ceph-mon[37563]: Health check cleared: PG_BACKFILL_FULL (was: Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull) 2024-03-11T21:47:59.870 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:47:59 smithi092 ceph-mon[37563]: osdmap e495: 8 total, 8 up, 5 in 2024-03-11T21:47:59.939 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:47:59.984 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:47:59.984 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:47:59.985 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:48:00.064 INFO:teuthology.orchestra.run.smithi092.stderr:Inferring config /var/lib/ceph/00db0016-dfed-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T21:48:00.085 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:47:59 smithi097 ceph-mon[33317]: purged_snaps scrub starts 2024-03-11T21:48:00.085 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:47:59 smithi097 ceph-mon[33317]: purged_snaps scrub ok 2024-03-11T21:48:00.085 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:47:59 smithi097 ceph-mon[33317]: Health check cleared: PG_BACKFILL_FULL (was: Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull) 2024-03-11T21:48:00.085 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:47:59 smithi097 ceph-mon[33317]: osdmap e495: 8 total, 8 up, 5 in 2024-03-11T21:48:00.085 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:48:00.117 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:48:00.118 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:48:00.119 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:48:00.219 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:48:00.249 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:48:00.250 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:48:00.250 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:48:00.351 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:48:00.389 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:48:00.390 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:48:00.390 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:48:00.491 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:48:00.524 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:48:00.524 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:48:00.525 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:48:00.625 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:48:00.661 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:48:00.662 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:48:00.662 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:48:00.763 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:48:00.792 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:48:00.793 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:48:00.793 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:48:00.815 INFO:teuthology.orchestra.run.smithi092.stderr:Inferring config /var/lib/ceph/00db0016-dfed-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T21:48:00.869 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:48:00 smithi092 ceph-mon[29572]: pgmap v1059: 148 pgs: 1 active+remapped+backfilling, 147 active+clean; 2.3 GiB data, 11 GiB used, 436 GiB / 447 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.39k op/s; 1873/112752 objects misplaced (1.661%); 4.5 MiB/s, 72 objects/s recovering 2024-03-11T21:48:00.869 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:48:00 smithi092 ceph-mon[37563]: pgmap v1059: 148 pgs: 1 active+remapped+backfilling, 147 active+clean; 2.3 GiB data, 11 GiB used, 436 GiB / 447 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.39k op/s; 1873/112752 objects misplaced (1.661%); 4.5 MiB/s, 72 objects/s recovering 2024-03-11T21:48:00.894 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:48:00.922 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:48:00.922 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:48:00.923 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:48:01.023 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:48:01.054 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:48:01.055 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:48:01.055 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:48:01.084 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:48:00 smithi097 ceph-mon[33317]: pgmap v1059: 148 pgs: 1 active+remapped+backfilling, 147 active+clean; 2.3 GiB data, 11 GiB used, 436 GiB / 447 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.39k op/s; 1873/112752 objects misplaced (1.661%); 4.5 MiB/s, 72 objects/s recovering 2024-03-11T21:48:01.156 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:48:01.186 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:48:01.187 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:48:01.188 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:48:01.288 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:48:01.321 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:48:01.322 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:48:01.322 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:48:01.423 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:48:01.455 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:48:01.455 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:48:01.456 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:48:01.557 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:48:01.587 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:48:01.588 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:48:01.588 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:48:01.679 INFO:teuthology.orchestra.run.smithi092.stderr:set pool 11 pgp_num to 16 2024-03-11T21:48:01.690 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:48:01.720 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:48:01.720 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:48:01.721 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:48:01.821 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:48:01.852 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:48:01.853 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:48:01.853 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:48:01.954 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:48:01.986 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:48:01.987 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:48:01.987 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:48:02.016 DEBUG:teuthology.orchestra.run.smithi092:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid 00db0016-dfed-11ee-95c8-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-03-11T21:48:02.058 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:48:01 smithi092 ceph-mon[29572]: from='client.? v1:172.21.15.92:0/2502341930' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "unique_pool_4", "var": "pgp_num", "val": "16"}]: dispatch 2024-03-11T21:48:02.059 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:48:01 smithi092 ceph-mon[29572]: from='client.35993 ' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "unique_pool_4", "var": "pgp_num", "val": "16"}]: dispatch 2024-03-11T21:48:02.059 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:48:01 smithi092 ceph-mon[37563]: from='client.? v1:172.21.15.92:0/2502341930' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "unique_pool_4", "var": "pgp_num", "val": "16"}]: dispatch 2024-03-11T21:48:02.059 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:48:01 smithi092 ceph-mon[37563]: from='client.35993 ' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "unique_pool_4", "var": "pgp_num", "val": "16"}]: dispatch 2024-03-11T21:48:02.084 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:48:01 smithi097 ceph-mon[33317]: from='client.? v1:172.21.15.92:0/2502341930' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "unique_pool_4", "var": "pgp_num", "val": "16"}]: dispatch 2024-03-11T21:48:02.085 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:48:01 smithi097 ceph-mon[33317]: from='client.35993 ' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "unique_pool_4", "var": "pgp_num", "val": "16"}]: dispatch 2024-03-11T21:48:02.088 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:48:02.132 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:48:02.132 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:48:02.132 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:48:02.233 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:48:02.263 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:48:02.264 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:48:02.264 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:48:02.365 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:48:02.421 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:48:02.421 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:48:02.422 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:48:02.522 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:48:02.566 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:48:02.567 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:48:02.567 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:48:02.668 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:48:02.700 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:48:02.700 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:48:02.701 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:48:02.802 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:48:02.840 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:48:02.841 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:48:02.841 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:48:02.923 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:48:02 smithi092 ceph-mon[29572]: pgmap v1061: 148 pgs: 1 active+remapped+backfilling, 147 active+clean; 2.2 GiB data, 11 GiB used, 436 GiB / 447 GiB avail; 638 B/s rd, 0 B/s wr, 1.10k op/s; 1873/110103 objects misplaced (1.701%); 4.6 MiB/s, 73 objects/s recovering 2024-03-11T21:48:02.924 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:48:02 smithi092 ceph-mon[29572]: from='client.35993 ' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_4", "var": "pgp_num", "val": "16"}]': finished 2024-03-11T21:48:02.924 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:48:02 smithi092 ceph-mon[29572]: osdmap e496: 8 total, 8 up, 5 in 2024-03-11T21:48:02.924 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:48:02 smithi092 ceph-mon[37563]: pgmap v1061: 148 pgs: 1 active+remapped+backfilling, 147 active+clean; 2.2 GiB data, 11 GiB used, 436 GiB / 447 GiB avail; 638 B/s rd, 0 B/s wr, 1.10k op/s; 1873/110103 objects misplaced (1.701%); 4.6 MiB/s, 73 objects/s recovering 2024-03-11T21:48:02.924 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:48:02 smithi092 ceph-mon[37563]: from='client.35993 ' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_4", "var": "pgp_num", "val": "16"}]': finished 2024-03-11T21:48:02.924 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:48:02 smithi092 ceph-mon[37563]: osdmap e496: 8 total, 8 up, 5 in 2024-03-11T21:48:02.942 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:48:02.972 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:48:02.973 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:48:02.973 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:48:03.074 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:48:03.084 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:48:02 smithi097 ceph-mon[33317]: pgmap v1061: 148 pgs: 1 active+remapped+backfilling, 147 active+clean; 2.2 GiB data, 11 GiB used, 436 GiB / 447 GiB avail; 638 B/s rd, 0 B/s wr, 1.10k op/s; 1873/110103 objects misplaced (1.701%); 4.6 MiB/s, 73 objects/s recovering 2024-03-11T21:48:03.084 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:48:02 smithi097 ceph-mon[33317]: from='client.35993 ' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_4", "var": "pgp_num", "val": "16"}]': finished 2024-03-11T21:48:03.085 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:48:02 smithi097 ceph-mon[33317]: osdmap e496: 8 total, 8 up, 5 in 2024-03-11T21:48:03.103 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:48:03.104 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:48:03.104 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:48:03.205 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:48:03.251 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:48:03.251 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:48:03.251 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:48:03.303 INFO:teuthology.orchestra.run.smithi092.stdout:ERROR: (22) Invalid argument 2024-03-11T21:48:03.303 INFO:teuthology.orchestra.run.smithi092.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-11T21:48:03.353 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:48:03.383 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:48:03.384 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:48:03.384 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:48:03.485 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:48:03.514 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:48:03.514 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:48:03.514 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:48:03.584 INFO:journalctl@ceph.iscsi.iscsi.a.smithi097.stdout:Mar 11 21:48:03 smithi097 ceph-00db0016-dfed-11ee-95c8-87774f69a715-iscsi-iscsi-a[60943]: debug there is no tcmu-runner data available 2024-03-11T21:48:03.615 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:48:03.654 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:48:03.654 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:48:03.655 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:48:03.756 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:48:03.793 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:48:03.793 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:48:03.794 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:48:03.895 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:48:03.926 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:48:03.927 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:48:03.927 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:48:03.945 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:48:03 smithi092 ceph-mon[29572]: from='client.24469 v1:172.21.15.97:0/404208310' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:48:03.946 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:48:03 smithi092 ceph-mon[37563]: from='client.24469 v1:172.21.15.97:0/404208310' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:48:04.028 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:48:04.068 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:48:04.069 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:48:04.069 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:48:04.084 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:48:03 smithi097 ceph-mon[33317]: from='client.24469 v1:172.21.15.97:0/404208310' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:48:04.170 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:48:04.217 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:48:04.218 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:48:04.218 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:48:04.319 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:48:04.349 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:48:04.350 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:48:04.350 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:48:04.451 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:48:04.483 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:48:04.484 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:48:04.484 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:48:04.585 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:48:04.622 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:48:04.623 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:48:04.623 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:48:04.717 DEBUG:teuthology.orchestra.run.smithi092:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid 00db0016-dfed-11ee-95c8-87774f69a715 -- ceph daemon osd.1 dump_historic_ops 2024-03-11T21:48:04.723 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:48:04.755 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:48:04.755 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:48:04.756 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:48:04.857 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:48:04.912 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:48:04.913 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:48:04.913 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:48:05.014 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:48:05.068 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:48:05.070 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:48:05.071 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:48:05.085 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:48:04 smithi097 ceph-mon[33317]: pgmap v1063: 148 pgs: 1 active+remapped+backfilling, 147 active+clean; 2.0 GiB data, 10 GiB used, 437 GiB / 447 GiB avail; 638 B/s rd, 0 B/s wr, 1.37k op/s; 862/98325 objects misplaced (0.877%); 9.7 MiB/s, 154 objects/s recovering 2024-03-11T21:48:05.119 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:48:04 smithi092 ceph-mon[29572]: pgmap v1063: 148 pgs: 1 active+remapped+backfilling, 147 active+clean; 2.0 GiB data, 10 GiB used, 437 GiB / 447 GiB avail; 638 B/s rd, 0 B/s wr, 1.37k op/s; 862/98325 objects misplaced (0.877%); 9.7 MiB/s, 154 objects/s recovering 2024-03-11T21:48:05.119 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:48:04 smithi092 ceph-mon[37563]: pgmap v1063: 148 pgs: 1 active+remapped+backfilling, 147 active+clean; 2.0 GiB data, 10 GiB used, 437 GiB / 447 GiB avail; 638 B/s rd, 0 B/s wr, 1.37k op/s; 862/98325 objects misplaced (0.877%); 9.7 MiB/s, 154 objects/s recovering 2024-03-11T21:48:05.171 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:48:05.214 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:48:05.215 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:48:05.215 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:48:05.316 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:48:05.355 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:48:05.356 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:48:05.356 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:48:05.457 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:48:05.487 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:48:05.487 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:48:05.488 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:48:05.572 INFO:teuthology.orchestra.run.smithi092.stderr:Inferring config /var/lib/ceph/00db0016-dfed-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T21:48:05.590 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:48:05.594 INFO:teuthology.orchestra.run.smithi092.stderr:Inferring config /var/lib/ceph/00db0016-dfed-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T21:48:05.655 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:48:05.656 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:48:05.656 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:48:05.757 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:48:05.798 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:48:05.799 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:48:05.799 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:48:05.900 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:48:05.944 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:48:05.945 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:48:05.945 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:48:06.046 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:48:06.075 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:48:06.076 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:48:06.076 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:48:06.177 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:48:06.223 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:48:06.224 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:48:06.225 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:48:06.325 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:48:06.362 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:48:06.363 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:48:06.363 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:48:06.464 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:48:06.493 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:48:06.494 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:48:06.494 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:48:06.595 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:48:06.632 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:48:06.633 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:48:06.633 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:48:06.734 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:48:06.763 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:48:06.763 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:48:06.763 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:48:06.864 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:48:06.901 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:48:06.902 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:48:06.902 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:48:06.984 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:48:06 smithi092 ceph-mon[29572]: pgmap v1064: 148 pgs: 1 active+remapped+backfilling, 147 active+clean; 1.8 GiB data, 9.7 GiB used, 437 GiB / 447 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.77k op/s; 862/88878 objects misplaced (0.970%); 9.7 MiB/s, 154 objects/s recovering 2024-03-11T21:48:06.984 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:48:06 smithi092 ceph-mon[37563]: pgmap v1064: 148 pgs: 1 active+remapped+backfilling, 147 active+clean; 1.8 GiB data, 9.7 GiB used, 437 GiB / 447 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.77k op/s; 862/88878 objects misplaced (0.970%); 9.7 MiB/s, 154 objects/s recovering 2024-03-11T21:48:07.003 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:48:07.032 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:48:07.032 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:48:07.033 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:48:07.085 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:48:06 smithi097 ceph-mon[33317]: pgmap v1064: 148 pgs: 1 active+remapped+backfilling, 147 active+clean; 1.8 GiB data, 9.7 GiB used, 437 GiB / 447 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.77k op/s; 862/88878 objects misplaced (0.970%); 9.7 MiB/s, 154 objects/s recovering 2024-03-11T21:48:07.133 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:48:07.161 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:48:07.162 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:48:07.162 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:48:07.263 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:48:07.307 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:48:07.308 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:48:07.308 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:48:07.409 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:48:07.441 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:48:07.442 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:48:07.442 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:48:07.543 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:48:07.572 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:48:07.573 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:48:07.573 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:48:07.658 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 6, 7] out_osds: [3, 4, 5] dead_osds: [] live_osds: [0, 1, 2, 4, 5, 6, 7, 3] 2024-03-11T21:48:07.663 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-03-11T21:48:07.663 INFO:tasks.thrashosds.thrasher:Adding osd 5 2024-03-11T21:48:07.663 DEBUG:teuthology.orchestra.run.smithi092:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid 00db0016-dfed-11ee-95c8-87774f69a715 -- ceph osd in 5 2024-03-11T21:48:07.674 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:48:07.703 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:48:07.703 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:48:07.704 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:48:07.720 INFO:teuthology.orchestra.run.smithi092.stderr:noscrub is unset 2024-03-11T21:48:07.804 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:48:07.837 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:48:07.837 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:48:07.838 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:48:07.939 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:48:07.966 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:48:07.966 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:48:07.967 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:48:08.026 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:48:07 smithi092 ceph-mon[29572]: from='mgr.14607 v1:172.21.15.92:0/1702717597' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T21:48:08.026 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:48:07 smithi092 ceph-mon[29572]: from='client.? v1:172.21.15.92:0/2960117935' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-11T21:48:08.026 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:48:07 smithi092 ceph-mon[29572]: from='client.25990 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-11T21:48:08.026 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:48:07 smithi092 ceph-mon[37563]: from='mgr.14607 v1:172.21.15.92:0/1702717597' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T21:48:08.026 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:48:07 smithi092 ceph-mon[37563]: from='client.? v1:172.21.15.92:0/2960117935' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-11T21:48:08.026 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:48:07 smithi092 ceph-mon[37563]: from='client.25990 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-11T21:48:08.027 INFO:journalctl@ceph.mgr.y.smithi092.stdout:Mar 11 21:48:07 smithi092 ceph-00db0016-dfed-11ee-95c8-87774f69a715-mgr-y[29801]: ::ffff:172.21.15.97 - - [11/Mar/2024:21:48:07] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-11T21:48:08.068 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:48:08.085 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:48:07 smithi097 ceph-mon[33317]: from='mgr.14607 v1:172.21.15.92:0/1702717597' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T21:48:08.085 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:48:07 smithi097 ceph-mon[33317]: from='client.? v1:172.21.15.92:0/2960117935' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-11T21:48:08.085 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:48:07 smithi097 ceph-mon[33317]: from='client.25990 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-11T21:48:08.094 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:48:08.094 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:48:08.094 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:48:08.195 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:48:08.219 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:48:08.219 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:48:08.220 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:48:08.320 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:48:08.344 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:48:08.345 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:48:08.345 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:48:08.446 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:48:08.470 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:48:08.470 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:48:08.471 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:48:08.571 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:48:08.597 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:48:08.598 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:48:08.598 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:48:08.699 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:48:08.726 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:48:08.727 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:48:08.727 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:48:08.828 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:48:08.869 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:48:08.870 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:48:08.871 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:48:08.971 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:48:08 smithi092 ceph-mon[29572]: pgmap v1065: 148 pgs: 1 active+remapped+backfilling, 147 active+clean; 1.8 GiB data, 9.7 GiB used, 437 GiB / 447 GiB avail; 639 B/s rd, 0 B/s wr, 994 op/s; 862/88878 objects misplaced (0.970%); 5.7 MiB/s, 91 objects/s recovering 2024-03-11T21:48:08.971 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:48:08 smithi092 ceph-mon[29572]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-11T21:48:08.971 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:48:08 smithi092 ceph-mon[29572]: from='client.25990 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-03-11T21:48:08.971 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:48:08 smithi092 ceph-mon[29572]: osdmap e497: 8 total, 8 up, 5 in 2024-03-11T21:48:08.972 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:48:08 smithi092 ceph-mon[29572]: from='mgr.14607 v1:172.21.15.92:0/1702717597' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T21:48:08.972 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:48:08 smithi092 ceph-mon[29572]: from='mgr.14607 v1:172.21.15.92:0/1702717597' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T21:48:08.972 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:48:08 smithi092 ceph-mon[29572]: from='mgr.14607 ' entity='mgr.y' 2024-03-11T21:48:08.972 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:48:08 smithi092 ceph-mon[29572]: from='mgr.14607 ' entity='mgr.y' 2024-03-11T21:48:08.972 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:48:08 smithi092 ceph-mon[29572]: from='mgr.14607 v1:172.21.15.92:0/1702717597' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T21:48:08.972 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:48:08 smithi092 ceph-mon[37563]: pgmap v1065: 148 pgs: 1 active+remapped+backfilling, 147 active+clean; 1.8 GiB data, 9.7 GiB used, 437 GiB / 447 GiB avail; 639 B/s rd, 0 B/s wr, 994 op/s; 862/88878 objects misplaced (0.970%); 5.7 MiB/s, 91 objects/s recovering 2024-03-11T21:48:08.972 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:48:08 smithi092 ceph-mon[37563]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-11T21:48:08.972 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:48:08 smithi092 ceph-mon[37563]: from='client.25990 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-03-11T21:48:08.972 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:48:08 smithi092 ceph-mon[37563]: osdmap e497: 8 total, 8 up, 5 in 2024-03-11T21:48:08.972 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:48:08 smithi092 ceph-mon[37563]: from='mgr.14607 v1:172.21.15.92:0/1702717597' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T21:48:08.972 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:48:08 smithi092 ceph-mon[37563]: from='mgr.14607 v1:172.21.15.92:0/1702717597' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T21:48:08.973 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:48:08 smithi092 ceph-mon[37563]: from='mgr.14607 ' entity='mgr.y' 2024-03-11T21:48:08.973 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:48:08 smithi092 ceph-mon[37563]: from='mgr.14607 ' entity='mgr.y' 2024-03-11T21:48:08.973 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:48:08 smithi092 ceph-mon[37563]: from='mgr.14607 v1:172.21.15.92:0/1702717597' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T21:48:08.973 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:48:09.001 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:48:09.002 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:48:09.002 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:48:09.085 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:48:08 smithi097 ceph-mon[33317]: pgmap v1065: 148 pgs: 1 active+remapped+backfilling, 147 active+clean; 1.8 GiB data, 9.7 GiB used, 437 GiB / 447 GiB avail; 639 B/s rd, 0 B/s wr, 994 op/s; 862/88878 objects misplaced (0.970%); 5.7 MiB/s, 91 objects/s recovering 2024-03-11T21:48:09.085 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:48:08 smithi097 ceph-mon[33317]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-03-11T21:48:09.085 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:48:08 smithi097 ceph-mon[33317]: from='client.25990 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-03-11T21:48:09.085 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:48:08 smithi097 ceph-mon[33317]: osdmap e497: 8 total, 8 up, 5 in 2024-03-11T21:48:09.085 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:48:08 smithi097 ceph-mon[33317]: from='mgr.14607 v1:172.21.15.92:0/1702717597' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T21:48:09.085 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:48:08 smithi097 ceph-mon[33317]: from='mgr.14607 v1:172.21.15.92:0/1702717597' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T21:48:09.085 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:48:08 smithi097 ceph-mon[33317]: from='mgr.14607 ' entity='mgr.y' 2024-03-11T21:48:09.085 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:48:08 smithi097 ceph-mon[33317]: from='mgr.14607 ' entity='mgr.y' 2024-03-11T21:48:09.085 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:48:08 smithi097 ceph-mon[33317]: from='mgr.14607 v1:172.21.15.92:0/1702717597' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T21:48:09.104 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:48:09.157 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:48:09.157 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:48:09.158 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:48:09.259 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:48:09.308 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:48:09.308 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:48:09.309 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:48:09.410 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:48:09.455 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:48:09.455 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:48:09.456 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:48:09.557 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:48:09.602 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:48:09.603 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:48:09.604 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:48:09.704 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:48:09.744 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:48:09.745 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:48:09.745 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:48:09.846 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:48:09.889 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:48:09.890 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:48:09.890 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:48:09.952 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:48:09 smithi092 ceph-mon[29572]: osdmap e498: 8 total, 8 up, 5 in 2024-03-11T21:48:09.952 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:48:09 smithi092 ceph-mon[37563]: osdmap e498: 8 total, 8 up, 5 in 2024-03-11T21:48:09.992 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:48:10.027 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:48:10.028 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:48:10.028 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:48:10.085 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:48:09 smithi097 ceph-mon[33317]: osdmap e498: 8 total, 8 up, 5 in 2024-03-11T21:48:10.129 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:48:10.168 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:48:10.169 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:48:10.169 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:48:10.270 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:48:10.300 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:48:10.301 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:48:10.301 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:48:10.351 INFO:teuthology.orchestra.run.smithi092.stdout:osd.0: {} 2024-03-11T21:48:10.390 INFO:teuthology.orchestra.run.smithi092.stdout:osd.1: {} 2024-03-11T21:48:10.402 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:48:10.430 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:48:10.431 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:48:10.431 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:48:10.434 INFO:teuthology.orchestra.run.smithi092.stdout:osd.2: {} 2024-03-11T21:48:10.471 INFO:teuthology.orchestra.run.smithi092.stdout:osd.3: {} 2024-03-11T21:48:10.516 INFO:teuthology.orchestra.run.smithi092.stdout:osd.4: {} 2024-03-11T21:48:10.532 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:48:10.556 INFO:teuthology.orchestra.run.smithi092.stdout:osd.5: {} 2024-03-11T21:48:10.577 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:48:10.578 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:48:10.578 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:48:10.595 INFO:teuthology.orchestra.run.smithi092.stdout:osd.6: {} 2024-03-11T21:48:10.639 INFO:teuthology.orchestra.run.smithi092.stdout:osd.7: {} 2024-03-11T21:48:10.645 INFO:teuthology.orchestra.run.smithi092.stderr:osd.0: osd_enable_op_tracker = '' 2024-03-11T21:48:10.645 INFO:teuthology.orchestra.run.smithi092.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-11T21:48:10.645 INFO:teuthology.orchestra.run.smithi092.stderr:osd.2: osd_enable_op_tracker = '' 2024-03-11T21:48:10.645 INFO:teuthology.orchestra.run.smithi092.stderr:osd.3: osd_enable_op_tracker = '' 2024-03-11T21:48:10.646 INFO:teuthology.orchestra.run.smithi092.stderr:osd.4: osd_enable_op_tracker = '' 2024-03-11T21:48:10.646 INFO:teuthology.orchestra.run.smithi092.stderr:osd.5: osd_enable_op_tracker = '' 2024-03-11T21:48:10.646 INFO:teuthology.orchestra.run.smithi092.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-11T21:48:10.646 INFO:teuthology.orchestra.run.smithi092.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-11T21:48:10.679 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:48:10.722 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:48:10.723 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:48:10.723 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:48:10.824 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:48:10.857 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:48:10 smithi092 ceph-mon[37563]: pgmap v1068: 148 pgs: 1 active+clean+remapped, 147 active+clean; 1.4 GiB data, 8.8 GiB used, 438 GiB / 447 GiB avail; 1.3 KiB/s rd, 0 B/s wr, 1.66k op/s; 12 MiB/s, 190 objects/s recovering 2024-03-11T21:48:10.858 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:48:10 smithi092 ceph-mon[37563]: from='client.? v1:172.21.15.92:0/4049080273' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-11T21:48:10.858 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:48:10 smithi092 ceph-mon[37563]: 2.14 scrub starts 2024-03-11T21:48:10.858 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:48:10 smithi092 ceph-mon[37563]: 2.14 scrub ok 2024-03-11T21:48:10.861 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:48:10 smithi092 ceph-mon[29572]: pgmap v1068: 148 pgs: 1 active+clean+remapped, 147 active+clean; 1.4 GiB data, 8.8 GiB used, 438 GiB / 447 GiB avail; 1.3 KiB/s rd, 0 B/s wr, 1.66k op/s; 12 MiB/s, 190 objects/s recovering 2024-03-11T21:48:10.861 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:48:10 smithi092 ceph-mon[29572]: from='client.? v1:172.21.15.92:0/4049080273' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-11T21:48:10.861 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:48:10 smithi092 ceph-mon[29572]: 2.14 scrub starts 2024-03-11T21:48:10.861 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:48:10 smithi092 ceph-mon[29572]: 2.14 scrub ok 2024-03-11T21:48:10.872 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:48:10.873 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:48:10.873 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:48:10.974 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:48:11.002 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:48:11.003 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:48:11.003 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:48:11.085 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:48:10 smithi097 ceph-mon[33317]: pgmap v1068: 148 pgs: 1 active+clean+remapped, 147 active+clean; 1.4 GiB data, 8.8 GiB used, 438 GiB / 447 GiB avail; 1.3 KiB/s rd, 0 B/s wr, 1.66k op/s; 12 MiB/s, 190 objects/s recovering 2024-03-11T21:48:11.085 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:48:10 smithi097 ceph-mon[33317]: from='client.? v1:172.21.15.92:0/4049080273' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-11T21:48:11.085 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:48:10 smithi097 ceph-mon[33317]: 2.14 scrub starts 2024-03-11T21:48:11.085 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:48:10 smithi097 ceph-mon[33317]: 2.14 scrub ok 2024-03-11T21:48:11.104 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:48:11.134 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:48:11.134 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:48:11.135 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:48:11.236 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:48:11.275 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:48:11.276 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:48:11.276 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:48:11.377 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:48:11.405 INFO:teuthology.orchestra.run.smithi092.stdout:{ 2024-03-11T21:48:11.405 INFO:teuthology.orchestra.run.smithi092.stdout: "size": 20, 2024-03-11T21:48:11.405 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 600, 2024-03-11T21:48:11.405 INFO:teuthology.orchestra.run.smithi092.stdout: "ops": [ 2024-03-11T21:48:11.405 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:48:11.405 INFO:teuthology.orchestra.run.smithi092.stdout: "description": "osd_op(client.15744.0:69284 10.12 10:4c65b9e9:::benchmark_data_smithi092_161462_object69283:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e372)", 2024-03-11T21:48:11.406 INFO:teuthology.orchestra.run.smithi092.stdout: "initiated_at": "2024-03-11T21:41:27.826717+0000", 2024-03-11T21:48:11.406 INFO:teuthology.orchestra.run.smithi092.stdout: "age": 403.03609375399998, 2024-03-11T21:48:11.406 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.0300101580000001, 2024-03-11T21:48:11.406 INFO:teuthology.orchestra.run.smithi092.stdout: "type_data": { 2024-03-11T21:48:11.406 INFO:teuthology.orchestra.run.smithi092.stdout: "flag_point": "reached pg", 2024-03-11T21:48:11.406 INFO:teuthology.orchestra.run.smithi092.stdout: "client_info": { 2024-03-11T21:48:11.406 INFO:teuthology.orchestra.run.smithi092.stdout: "client": "client.15744", 2024-03-11T21:48:11.406 INFO:teuthology.orchestra.run.smithi092.stdout: "client_addr": "v1:172.21.15.92:0/3334000812", 2024-03-11T21:48:11.406 INFO:teuthology.orchestra.run.smithi092.stdout: "tid": 69284 2024-03-11T21:48:11.406 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:48:11.406 INFO:teuthology.orchestra.run.smithi092.stdout: "events": [ 2024-03-11T21:48:11.406 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:48:11.406 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "initiated", 2024-03-11T21:48:11.406 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:41:27.826717+0000", 2024-03-11T21:48:11.407 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-03-11T21:48:11.407 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:48:11.407 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:48:11.407 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "header_read", 2024-03-11T21:48:11.407 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:41:27.826717+0000", 2024-03-11T21:48:11.407 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-03-11T21:48:11.407 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:48:11.407 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:48:11.407 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "throttled", 2024-03-11T21:48:11.407 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:41:27.826721+0000", 2024-03-11T21:48:11.407 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 3.1719999999999999e-06 2024-03-11T21:48:11.407 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:48:11.407 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:48:11.407 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "all_read", 2024-03-11T21:48:11.408 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:41:27.826748+0000", 2024-03-11T21:48:11.408 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 2.7790999999999999e-05 2024-03-11T21:48:11.408 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:48:11.408 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:48:11.408 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "dispatched", 2024-03-11T21:48:11.408 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:41:27.826770+0000", 2024-03-11T21:48:11.408 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 2.1878999999999998e-05 2024-03-11T21:48:11.408 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:48:11.408 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:48:11.408 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "queued_for_pg", 2024-03-11T21:48:11.408 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:41:27.826781+0000", 2024-03-11T21:48:11.408 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.1008e-05 2024-03-11T21:48:11.408 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:48:11.408 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:48:11.409 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-03-11T21:48:11.409 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:41:27.894144+0000", 2024-03-11T21:48:11.409 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.067362834999999996 2024-03-11T21:48:11.409 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:48:11.409 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:48:11.409 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "waiting for peered", 2024-03-11T21:48:11.409 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:41:27.894168+0000", 2024-03-11T21:48:11.409 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 2.4121000000000002e-05 2024-03-11T21:48:11.409 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:48:11.409 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:48:11.409 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-03-11T21:48:11.409 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:41:28.856497+0000", 2024-03-11T21:48:11.409 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.96232841999999996 2024-03-11T21:48:11.409 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:48:11.410 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:48:11.410 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "done", 2024-03-11T21:48:11.410 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:41:28.856728+0000", 2024-03-11T21:48:11.410 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00023093199999999999 2024-03-11T21:48:11.410 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-03-11T21:48:11.410 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-03-11T21:48:11.410 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-03-11T21:48:11.410 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:48:11.410 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:48:11.410 INFO:teuthology.orchestra.run.smithi092.stdout: "description": "osd_op(client.15744.0:214880 10.14 10:2d1b2e86:::benchmark_data_smithi092_161462_object104904:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e431)", 2024-03-11T21:48:11.410 INFO:teuthology.orchestra.run.smithi092.stdout: "initiated_at": "2024-03-11T21:44:20.414357+0000", 2024-03-11T21:48:11.410 INFO:teuthology.orchestra.run.smithi092.stdout: "age": 230.448453863, 2024-03-11T21:48:11.410 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.081828625, 2024-03-11T21:48:11.410 INFO:teuthology.orchestra.run.smithi092.stdout: "type_data": { 2024-03-11T21:48:11.411 INFO:teuthology.orchestra.run.smithi092.stdout: "flag_point": "reached pg", 2024-03-11T21:48:11.411 INFO:teuthology.orchestra.run.smithi092.stdout: "client_info": { 2024-03-11T21:48:11.411 INFO:teuthology.orchestra.run.smithi092.stdout: "client": "client.15744", 2024-03-11T21:48:11.411 INFO:teuthology.orchestra.run.smithi092.stdout: "client_addr": "v1:172.21.15.92:0/3334000812", 2024-03-11T21:48:11.411 INFO:teuthology.orchestra.run.smithi092.stdout: "tid": 214880 2024-03-11T21:48:11.411 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:48:11.411 INFO:teuthology.orchestra.run.smithi092.stdout: "events": [ 2024-03-11T21:48:11.411 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:48:11.411 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "initiated", 2024-03-11T21:48:11.411 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:44:20.414357+0000", 2024-03-11T21:48:11.411 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-03-11T21:48:11.411 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:48:11.411 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:48:11.411 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "header_read", 2024-03-11T21:48:11.412 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:44:20.414357+0000", 2024-03-11T21:48:11.412 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-03-11T21:48:11.412 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:48:11.412 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:48:11.412 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "throttled", 2024-03-11T21:48:11.412 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:44:20.414358+0000", 2024-03-11T21:48:11.412 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 8.8999999999999995e-07 2024-03-11T21:48:11.412 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:48:11.412 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:48:11.412 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "all_read", 2024-03-11T21:48:11.412 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:44:20.414362+0000", 2024-03-11T21:48:11.412 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 4.109e-06 2024-03-11T21:48:11.412 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:48:11.412 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:48:11.412 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "dispatched", 2024-03-11T21:48:11.413 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:44:20.414369+0000", 2024-03-11T21:48:11.413 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 7.0190000000000001e-06 2024-03-11T21:48:11.413 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:48:11.413 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:48:11.413 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "queued_for_pg", 2024-03-11T21:48:11.413 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:44:20.414375+0000", 2024-03-11T21:48:11.413 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 5.9259999999999997e-06 2024-03-11T21:48:11.413 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:48:11.413 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:48:11.413 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-03-11T21:48:11.413 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:44:20.417665+0000", 2024-03-11T21:48:11.413 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.003289737 2024-03-11T21:48:11.413 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:48:11.413 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:48:11.413 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "started", 2024-03-11T21:48:11.414 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:44:20.417828+0000", 2024-03-11T21:48:11.414 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.000162967 2024-03-11T21:48:11.414 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:48:11.414 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:48:11.414 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "waiting for subops from 0,2,3,4", 2024-03-11T21:48:11.414 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:44:20.418344+0000", 2024-03-11T21:48:11.414 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00051615700000000005 2024-03-11T21:48:11.414 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:48:11.414 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:48:11.414 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "sub_op_commit_rec", 2024-03-11T21:48:11.414 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:44:20.425640+0000", 2024-03-11T21:48:11.414 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.0072955340000000002 2024-03-11T21:48:11.414 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:48:11.414 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:48:11.415 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "op_commit", 2024-03-11T21:48:11.415 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:44:20.425670+0000", 2024-03-11T21:48:11.415 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 3.0434e-05 2024-03-11T21:48:11.415 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:48:11.415 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:48:11.415 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "sub_op_commit_rec", 2024-03-11T21:48:11.415 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:44:20.438262+0000", 2024-03-11T21:48:11.415 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.012591532000000001 2024-03-11T21:48:11.415 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:48:11.415 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:48:11.415 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "sub_op_commit_rec", 2024-03-11T21:48:11.415 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:44:20.440361+0000", 2024-03-11T21:48:11.415 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.0020995779999999999 2024-03-11T21:48:11.415 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:48:11.415 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:48:11.416 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-03-11T21:48:11.416 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:44:20.449645+0000", 2024-03-11T21:48:11.416 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.0092834979999999994 2024-03-11T21:48:11.416 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:48:11.416 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:48:11.416 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "waiting for peered", 2024-03-11T21:48:11.416 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:44:20.449659+0000", 2024-03-11T21:48:11.416 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.4195999999999999e-05 2024-03-11T21:48:11.416 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:48:11.416 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:48:11.416 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-03-11T21:48:11.416 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:44:21.496050+0000", 2024-03-11T21:48:11.416 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.046390777 2024-03-11T21:48:11.416 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:48:11.417 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:48:11.417 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "done", 2024-03-11T21:48:11.417 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:44:21.496186+0000", 2024-03-11T21:48:11.417 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00013627100000000001 2024-03-11T21:48:11.417 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-03-11T21:48:11.417 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-03-11T21:48:11.417 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-03-11T21:48:11.417 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:48:11.417 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:48:11.417 INFO:teuthology.orchestra.run.smithi092.stdout: "description": "osd_op(client.15744.0:214882 10.14 10:28cde69f:::benchmark_data_smithi092_161462_object104906:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e431)", 2024-03-11T21:48:11.417 INFO:teuthology.orchestra.run.smithi092.stdout: "initiated_at": "2024-03-11T21:44:20.414382+0000", 2024-03-11T21:48:11.417 INFO:teuthology.orchestra.run.smithi092.stdout: "age": 230.44842877599999, 2024-03-11T21:48:11.417 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.082668924, 2024-03-11T21:48:11.417 INFO:teuthology.orchestra.run.smithi092.stdout: "type_data": { 2024-03-11T21:48:11.417 INFO:teuthology.orchestra.run.smithi092.stdout: "flag_point": "reached pg", 2024-03-11T21:48:11.418 INFO:teuthology.orchestra.run.smithi092.stdout: "client_info": { 2024-03-11T21:48:11.418 INFO:teuthology.orchestra.run.smithi092.stdout: "client": "client.15744", 2024-03-11T21:48:11.418 INFO:teuthology.orchestra.run.smithi092.stdout: "client_addr": "v1:172.21.15.92:0/3334000812", 2024-03-11T21:48:11.418 INFO:teuthology.orchestra.run.smithi092.stdout: "tid": 214882 2024-03-11T21:48:11.418 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:48:11.418 INFO:teuthology.orchestra.run.smithi092.stdout: "events": [ 2024-03-11T21:48:11.418 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:48:11.418 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "initiated", 2024-03-11T21:48:11.418 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:44:20.414382+0000", 2024-03-11T21:48:11.418 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-03-11T21:48:11.418 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:48:11.418 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:48:11.418 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "header_read", 2024-03-11T21:48:11.418 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:44:20.414382+0000", 2024-03-11T21:48:11.419 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-03-11T21:48:11.419 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:48:11.419 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:48:11.419 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "throttled", 2024-03-11T21:48:11.419 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:44:20.414383+0000", 2024-03-11T21:48:11.419 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 6.9100000000000003e-07 2024-03-11T21:48:11.419 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:48:11.419 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:48:11.419 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "all_read", 2024-03-11T21:48:11.419 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:44:20.414386+0000", 2024-03-11T21:48:11.419 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 2.9390000000000002e-06 2024-03-11T21:48:11.419 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:48:11.419 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:48:11.419 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "dispatched", 2024-03-11T21:48:11.419 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:44:20.414391+0000", 2024-03-11T21:48:11.420 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 5.3539999999999999e-06 2024-03-11T21:48:11.420 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:48:11.420 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:48:11.420 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "queued_for_pg", 2024-03-11T21:48:11.420 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:44:20.414397+0000", 2024-03-11T21:48:11.420 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 5.5419999999999996e-06 2024-03-11T21:48:11.420 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:48:11.420 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:48:11.420 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-03-11T21:48:11.420 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:44:20.421980+0000", 2024-03-11T21:48:11.420 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.0075829920000000002 2024-03-11T21:48:11.420 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:48:11.420 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:48:11.420 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "started", 2024-03-11T21:48:11.421 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:44:20.422170+0000", 2024-03-11T21:48:11.421 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00019011700000000001 2024-03-11T21:48:11.421 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:48:11.421 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:48:11.421 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "waiting for subops from 0,2,3,4", 2024-03-11T21:48:11.421 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:44:20.422680+0000", 2024-03-11T21:48:11.421 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00051030499999999996 2024-03-11T21:48:11.421 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:48:11.421 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:48:11.421 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "op_commit", 2024-03-11T21:48:11.421 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:44:20.429224+0000", 2024-03-11T21:48:11.421 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.0065437439999999998 2024-03-11T21:48:11.421 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:48:11.421 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:48:11.421 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "sub_op_commit_rec", 2024-03-11T21:48:11.422 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:44:20.438310+0000", 2024-03-11T21:48:11.422 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.0090864729999999994 2024-03-11T21:48:11.422 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:48:11.422 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:48:11.422 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "sub_op_commit_rec", 2024-03-11T21:48:11.422 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:44:20.440408+0000", 2024-03-11T21:48:11.422 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.0020975619999999999 2024-03-11T21:48:11.422 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:48:11.422 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:48:11.422 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-03-11T21:48:11.422 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:44:20.450868+0000", 2024-03-11T21:48:11.422 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.010459619 2024-03-11T21:48:11.422 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:48:11.423 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:48:11.423 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "waiting for peered", 2024-03-11T21:48:11.423 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:44:20.450878+0000", 2024-03-11T21:48:11.423 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.0087e-05 2024-03-11T21:48:11.423 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:48:11.423 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:48:11.423 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-03-11T21:48:11.423 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:44:21.496965+0000", 2024-03-11T21:48:11.423 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.0460868759999999 2024-03-11T21:48:11.423 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:48:11.423 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:48:11.423 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "done", 2024-03-11T21:48:11.423 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:44:21.497051+0000", 2024-03-11T21:48:11.423 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 8.6623000000000001e-05 2024-03-11T21:48:11.423 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-03-11T21:48:11.424 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-03-11T21:48:11.424 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-03-11T21:48:11.424 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:48:11.424 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:48:11.424 INFO:teuthology.orchestra.run.smithi092.stdout: "description": "osd_op(client.15744.0:214913 10.14 10:2be24572:::benchmark_data_smithi092_161462_object104937:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e432)", 2024-03-11T21:48:11.424 INFO:teuthology.orchestra.run.smithi092.stdout: "initiated_at": "2024-03-11T21:44:20.480969+0000", 2024-03-11T21:48:11.424 INFO:teuthology.orchestra.run.smithi092.stdout: "age": 230.38184253, 2024-03-11T21:48:11.424 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.066973433, 2024-03-11T21:48:11.424 INFO:teuthology.orchestra.run.smithi092.stdout: "type_data": { 2024-03-11T21:48:11.424 INFO:teuthology.orchestra.run.smithi092.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-11T21:48:11.424 INFO:teuthology.orchestra.run.smithi092.stdout: "client_info": { 2024-03-11T21:48:11.424 INFO:teuthology.orchestra.run.smithi092.stdout: "client": "client.15744", 2024-03-11T21:48:11.424 INFO:teuthology.orchestra.run.smithi092.stdout: "client_addr": "v1:172.21.15.92:0/3334000812", 2024-03-11T21:48:11.424 INFO:teuthology.orchestra.run.smithi092.stdout: "tid": 214913 2024-03-11T21:48:11.424 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:48:11.425 INFO:teuthology.orchestra.run.smithi092.stdout: "events": [ 2024-03-11T21:48:11.425 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:48:11.425 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "initiated", 2024-03-11T21:48:11.425 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:44:20.480969+0000", 2024-03-11T21:48:11.425 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-03-11T21:48:11.425 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:48:11.425 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:48:11.425 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "header_read", 2024-03-11T21:48:11.425 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:44:20.480969+0000", 2024-03-11T21:48:11.425 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-03-11T21:48:11.425 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:48:11.425 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:48:11.425 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "throttled", 2024-03-11T21:48:11.425 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:44:20.480971+0000", 2024-03-11T21:48:11.426 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 2.2929999999999999e-06 2024-03-11T21:48:11.426 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:48:11.426 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:48:11.426 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "all_read", 2024-03-11T21:48:11.426 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:44:20.480977+0000", 2024-03-11T21:48:11.426 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 6.3199999999999996e-06 2024-03-11T21:48:11.426 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:48:11.426 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:48:11.426 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "dispatched", 2024-03-11T21:48:11.426 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:44:20.480994+0000", 2024-03-11T21:48:11.426 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.6901999999999998e-05 2024-03-11T21:48:11.426 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:48:11.426 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:48:11.426 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "queued_for_pg", 2024-03-11T21:48:11.426 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:44:20.481011+0000", 2024-03-11T21:48:11.427 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.6657999999999999e-05 2024-03-11T21:48:11.427 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:48:11.427 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:48:11.427 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-03-11T21:48:11.427 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:44:20.481072+0000", 2024-03-11T21:48:11.427 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 6.0825999999999997e-05 2024-03-11T21:48:11.427 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:48:11.427 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:48:11.427 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "waiting for peered", 2024-03-11T21:48:11.427 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:44:20.481082+0000", 2024-03-11T21:48:11.427 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.0583e-05 2024-03-11T21:48:11.427 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:48:11.427 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:48:11.427 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-03-11T21:48:11.427 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:44:21.497314+0000", 2024-03-11T21:48:11.428 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.016232276 2024-03-11T21:48:11.428 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:48:11.428 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:48:11.428 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "started", 2024-03-11T21:48:11.428 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:44:21.497424+0000", 2024-03-11T21:48:11.428 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.000109999 2024-03-11T21:48:11.428 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:48:11.428 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:48:11.428 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "waiting for subops from 0,2,4", 2024-03-11T21:48:11.428 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:44:21.497816+0000", 2024-03-11T21:48:11.428 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00039165700000000001 2024-03-11T21:48:11.428 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:48:11.428 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:48:11.428 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "sub_op_commit_rec", 2024-03-11T21:48:11.429 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:44:21.506963+0000", 2024-03-11T21:48:11.429 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.0091465720000000004 2024-03-11T21:48:11.429 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:48:11.429 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:48:11.429 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "sub_op_commit_rec", 2024-03-11T21:48:11.429 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:44:21.507291+0000", 2024-03-11T21:48:11.429 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00032803500000000001 2024-03-11T21:48:11.429 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:48:11.429 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:48:11.429 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "op_commit", 2024-03-11T21:48:11.429 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:44:21.507905+0000", 2024-03-11T21:48:11.429 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00061426799999999995 2024-03-11T21:48:11.429 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:48:11.429 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:48:11.429 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "sub_op_commit_rec", 2024-03-11T21:48:11.429 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:44:21.547793+0000", 2024-03-11T21:48:11.430 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.039888267999999998 2024-03-11T21:48:11.430 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:48:11.430 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:48:11.430 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "commit_sent", 2024-03-11T21:48:11.430 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:44:21.547844+0000", 2024-03-11T21:48:11.430 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 5.0958000000000002e-05 2024-03-11T21:48:11.430 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:48:11.430 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:48:11.430 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "done", 2024-03-11T21:48:11.430 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:44:21.547942+0000", 2024-03-11T21:48:11.430 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 9.7818000000000004e-05 2024-03-11T21:48:11.430 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-03-11T21:48:11.430 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-03-11T21:48:11.430 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-03-11T21:48:11.431 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:48:11.431 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:48:11.431 INFO:teuthology.orchestra.run.smithi092.stdout: "description": "osd_op(client.35813.0:194234 11.b 11:d98d3370:::benchmark_data_smithi092_186520_object55886:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e477)", 2024-03-11T21:48:11.431 INFO:teuthology.orchestra.run.smithi092.stdout: "initiated_at": "2024-03-11T21:47:19.277211+0000", 2024-03-11T21:48:11.431 INFO:teuthology.orchestra.run.smithi092.stdout: "age": 51.585599831000003, 2024-03-11T21:48:11.431 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.99220599200000004, 2024-03-11T21:48:11.431 INFO:teuthology.orchestra.run.smithi092.stdout: "type_data": { 2024-03-11T21:48:11.431 INFO:teuthology.orchestra.run.smithi092.stdout: "flag_point": "reached pg", 2024-03-11T21:48:11.431 INFO:teuthology.orchestra.run.smithi092.stdout: "client_info": { 2024-03-11T21:48:11.431 INFO:teuthology.orchestra.run.smithi092.stdout: "client": "client.35813", 2024-03-11T21:48:11.431 INFO:teuthology.orchestra.run.smithi092.stdout: "client_addr": "v1:172.21.15.92:0/4293352030", 2024-03-11T21:48:11.431 INFO:teuthology.orchestra.run.smithi092.stdout: "tid": 194234 2024-03-11T21:48:11.431 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:48:11.431 INFO:teuthology.orchestra.run.smithi092.stdout: "events": [ 2024-03-11T21:48:11.431 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:48:11.432 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "initiated", 2024-03-11T21:48:11.432 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:47:19.277211+0000", 2024-03-11T21:48:11.432 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-03-11T21:48:11.432 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:48:11.432 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:48:11.432 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "header_read", 2024-03-11T21:48:11.432 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:47:19.277211+0000", 2024-03-11T21:48:11.432 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-03-11T21:48:11.432 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:48:11.432 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:48:11.432 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "throttled", 2024-03-11T21:48:11.432 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:47:19.277213+0000", 2024-03-11T21:48:11.432 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 2.04e-06 2024-03-11T21:48:11.432 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:48:11.433 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:48:11.433 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "all_read", 2024-03-11T21:48:11.433 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:47:19.277220+0000", 2024-03-11T21:48:11.433 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 6.798e-06 2024-03-11T21:48:11.433 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:48:11.433 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:48:11.433 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "dispatched", 2024-03-11T21:48:11.433 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:47:19.277232+0000", 2024-03-11T21:48:11.433 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.1990999999999999e-05 2024-03-11T21:48:11.433 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:48:11.433 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:48:11.433 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "queued_for_pg", 2024-03-11T21:48:11.433 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:47:19.277241+0000", 2024-03-11T21:48:11.433 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 8.7360000000000007e-06 2024-03-11T21:48:11.433 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:48:11.434 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:48:11.434 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-03-11T21:48:11.434 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:47:19.277438+0000", 2024-03-11T21:48:11.434 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.000196956 2024-03-11T21:48:11.434 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:48:11.434 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:48:11.434 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "started", 2024-03-11T21:48:11.434 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:47:19.277577+0000", 2024-03-11T21:48:11.434 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.000139035 2024-03-11T21:48:11.434 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:48:11.434 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:48:11.434 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "waiting for subops from 0,5,6", 2024-03-11T21:48:11.434 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:47:19.277719+0000", 2024-03-11T21:48:11.434 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.000142266 2024-03-11T21:48:11.435 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:48:11.435 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:48:11.435 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "op_commit", 2024-03-11T21:48:11.435 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:47:19.280081+0000", 2024-03-11T21:48:11.435 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.0023619660000000001 2024-03-11T21:48:11.435 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:48:11.435 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:48:11.435 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "sub_op_commit_rec", 2024-03-11T21:48:11.435 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:47:19.284547+0000", 2024-03-11T21:48:11.435 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.0044655830000000004 2024-03-11T21:48:11.435 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:48:11.435 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:48:11.435 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-03-11T21:48:11.435 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:47:19.306080+0000", 2024-03-11T21:48:11.435 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.021533364999999999 2024-03-11T21:48:11.436 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:48:11.436 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:48:11.436 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "waiting for peered", 2024-03-11T21:48:11.436 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:47:19.306178+0000", 2024-03-11T21:48:11.436 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 9.8183000000000005e-05 2024-03-11T21:48:11.436 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:48:11.436 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:48:11.436 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-03-11T21:48:11.436 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:47:20.269401+0000", 2024-03-11T21:48:11.436 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.96322235 2024-03-11T21:48:11.436 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:48:11.436 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:48:11.436 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "done", 2024-03-11T21:48:11.436 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:47:20.269417+0000", 2024-03-11T21:48:11.436 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.6722999999999999e-05 2024-03-11T21:48:11.437 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-03-11T21:48:11.437 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-03-11T21:48:11.437 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-03-11T21:48:11.437 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:48:11.437 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:48:11.437 INFO:teuthology.orchestra.run.smithi092.stdout: "description": "osd_op(client.35813.0:228721 11.7 11:ea432297:::benchmark_data_smithi092_186520_object90373:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e491)", 2024-03-11T21:48:11.437 INFO:teuthology.orchestra.run.smithi092.stdout: "initiated_at": "2024-03-11T21:47:49.372829+0000", 2024-03-11T21:48:11.437 INFO:teuthology.orchestra.run.smithi092.stdout: "age": 21.489982440999999, 2024-03-11T21:48:11.437 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.0552764379999999, 2024-03-11T21:48:11.437 INFO:teuthology.orchestra.run.smithi092.stdout: "type_data": { 2024-03-11T21:48:11.437 INFO:teuthology.orchestra.run.smithi092.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-11T21:48:11.437 INFO:teuthology.orchestra.run.smithi092.stdout: "client_info": { 2024-03-11T21:48:11.437 INFO:teuthology.orchestra.run.smithi092.stdout: "client": "client.35813", 2024-03-11T21:48:11.437 INFO:teuthology.orchestra.run.smithi092.stdout: "client_addr": "v1:172.21.15.92:0/4293352030", 2024-03-11T21:48:11.437 INFO:teuthology.orchestra.run.smithi092.stdout: "tid": 228721 2024-03-11T21:48:11.437 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:48:11.438 INFO:teuthology.orchestra.run.smithi092.stdout: "events": [ 2024-03-11T21:48:11.438 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:48:11.438 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "initiated", 2024-03-11T21:48:11.438 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:47:49.372829+0000", 2024-03-11T21:48:11.438 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-03-11T21:48:11.438 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:48:11.438 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:48:11.438 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "header_read", 2024-03-11T21:48:11.438 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:47:49.372829+0000", 2024-03-11T21:48:11.438 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-03-11T21:48:11.438 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:48:11.438 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:48:11.438 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "throttled", 2024-03-11T21:48:11.438 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:47:49.372830+0000", 2024-03-11T21:48:11.438 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.269e-06 2024-03-11T21:48:11.439 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:48:11.439 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:48:11.439 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "all_read", 2024-03-11T21:48:11.439 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:47:49.372835+0000", 2024-03-11T21:48:11.439 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 4.5889999999999996e-06 2024-03-11T21:48:11.439 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:48:11.439 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:48:11.439 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "dispatched", 2024-03-11T21:48:11.439 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:47:49.372848+0000", 2024-03-11T21:48:11.439 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.3359e-05 2024-03-11T21:48:11.439 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:48:11.439 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:48:11.439 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "queued_for_pg", 2024-03-11T21:48:11.439 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:47:49.372855+0000", 2024-03-11T21:48:11.439 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 7.5939999999999996e-06 2024-03-11T21:48:11.440 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:48:11.440 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:48:11.440 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-03-11T21:48:11.440 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:47:49.373040+0000", 2024-03-11T21:48:11.440 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00018488399999999999 2024-03-11T21:48:11.440 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:48:11.440 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:48:11.440 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "waiting for peered", 2024-03-11T21:48:11.440 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:47:49.373049+0000", 2024-03-11T21:48:11.440 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 9.0999999999999993e-06 2024-03-11T21:48:11.440 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:48:11.440 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:48:11.440 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-03-11T21:48:11.440 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:47:50.424513+0000", 2024-03-11T21:48:11.440 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.0514637650000001 2024-03-11T21:48:11.440 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:48:11.441 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:48:11.441 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "started", 2024-03-11T21:48:11.441 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:47:50.424676+0000", 2024-03-11T21:48:11.441 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00016310700000000001 2024-03-11T21:48:11.441 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:48:11.441 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:48:11.441 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "waiting for subops from 0,7", 2024-03-11T21:48:11.441 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:47:50.424828+0000", 2024-03-11T21:48:11.441 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.000151846 2024-03-11T21:48:11.441 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:48:11.441 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:48:11.441 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "op_commit", 2024-03-11T21:48:11.441 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:47:50.426529+0000", 2024-03-11T21:48:11.441 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.001701115 2024-03-11T21:48:11.441 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:48:11.442 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:48:11.442 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "sub_op_commit_rec", 2024-03-11T21:48:11.442 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:47:50.427977+0000", 2024-03-11T21:48:11.442 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.001447371 2024-03-11T21:48:11.442 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:48:11.442 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:48:11.442 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "sub_op_commit_rec", 2024-03-11T21:48:11.442 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:47:50.428043+0000", 2024-03-11T21:48:11.442 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 6.6495000000000006e-05 2024-03-11T21:48:11.442 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:48:11.442 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:48:11.442 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "commit_sent", 2024-03-11T21:48:11.442 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:47:50.428078+0000", 2024-03-11T21:48:11.442 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 3.4728999999999999e-05 2024-03-11T21:48:11.442 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:48:11.443 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:48:11.443 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "done", 2024-03-11T21:48:11.443 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:47:50.428105+0000", 2024-03-11T21:48:11.443 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 2.7214999999999999e-05 2024-03-11T21:48:11.443 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-03-11T21:48:11.443 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-03-11T21:48:11.443 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-03-11T21:48:11.443 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:48:11.443 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:48:11.443 INFO:teuthology.orchestra.run.smithi092.stdout: "description": "osd_op(client.35813.0:228726 11.7 11:ed7f199b:::benchmark_data_smithi092_186520_object90378:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e491)", 2024-03-11T21:48:11.443 INFO:teuthology.orchestra.run.smithi092.stdout: "initiated_at": "2024-03-11T21:47:49.376369+0000", 2024-03-11T21:48:11.443 INFO:teuthology.orchestra.run.smithi092.stdout: "age": 21.486441849999999, 2024-03-11T21:48:11.443 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.055356467, 2024-03-11T21:48:11.443 INFO:teuthology.orchestra.run.smithi092.stdout: "type_data": { 2024-03-11T21:48:11.443 INFO:teuthology.orchestra.run.smithi092.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-11T21:48:11.443 INFO:teuthology.orchestra.run.smithi092.stdout: "client_info": { 2024-03-11T21:48:11.444 INFO:teuthology.orchestra.run.smithi092.stdout: "client": "client.35813", 2024-03-11T21:48:11.444 INFO:teuthology.orchestra.run.smithi092.stdout: "client_addr": "v1:172.21.15.92:0/4293352030", 2024-03-11T21:48:11.444 INFO:teuthology.orchestra.run.smithi092.stdout: "tid": 228726 2024-03-11T21:48:11.444 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:48:11.444 INFO:teuthology.orchestra.run.smithi092.stdout: "events": [ 2024-03-11T21:48:11.444 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:48:11.444 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "initiated", 2024-03-11T21:48:11.444 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:47:49.376369+0000", 2024-03-11T21:48:11.444 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-03-11T21:48:11.444 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:48:11.444 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:48:11.444 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "header_read", 2024-03-11T21:48:11.444 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:47:49.376369+0000", 2024-03-11T21:48:11.444 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-03-11T21:48:11.444 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:48:11.445 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:48:11.445 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "throttled", 2024-03-11T21:48:11.445 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:47:49.376371+0000", 2024-03-11T21:48:11.445 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.8250000000000001e-06 2024-03-11T21:48:11.445 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:48:11.445 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:48:11.445 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "all_read", 2024-03-11T21:48:11.445 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:47:49.376377+0000", 2024-03-11T21:48:11.445 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 5.4280000000000004e-06 2024-03-11T21:48:11.445 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:48:11.445 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:48:11.445 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "dispatched", 2024-03-11T21:48:11.445 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:47:49.376387+0000", 2024-03-11T21:48:11.445 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.0203e-05 2024-03-11T21:48:11.446 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:48:11.446 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:48:11.446 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "queued_for_pg", 2024-03-11T21:48:11.446 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:47:49.376394+0000", 2024-03-11T21:48:11.446 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 7.7200000000000006e-06 2024-03-11T21:48:11.446 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:48:11.446 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:48:11.446 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-03-11T21:48:11.446 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:47:49.376446+0000", 2024-03-11T21:48:11.446 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 5.1706000000000003e-05 2024-03-11T21:48:11.446 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:48:11.446 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:48:11.446 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "waiting for peered", 2024-03-11T21:48:11.446 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:47:49.376456+0000", 2024-03-11T21:48:11.447 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 9.5030000000000003e-06 2024-03-11T21:48:11.447 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:48:11.447 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:48:11.447 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-03-11T21:48:11.447 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:47:50.426935+0000", 2024-03-11T21:48:11.447 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.050479194 2024-03-11T21:48:11.447 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:48:11.447 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:48:11.447 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "started", 2024-03-11T21:48:11.447 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:47:50.427066+0000", 2024-03-11T21:48:11.447 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.000130677 2024-03-11T21:48:11.447 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:48:11.447 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:48:11.447 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "waiting for subops from 0,7", 2024-03-11T21:48:11.447 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:47:50.427150+0000", 2024-03-11T21:48:11.447 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 8.4723999999999997e-05 2024-03-11T21:48:11.448 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:48:11.448 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:48:11.448 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "op_commit", 2024-03-11T21:48:11.448 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:47:50.431452+0000", 2024-03-11T21:48:11.448 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.0043014960000000001 2024-03-11T21:48:11.448 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:48:11.448 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:48:11.448 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "sub_op_commit_rec", 2024-03-11T21:48:11.448 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:47:50.431500+0000", 2024-03-11T21:48:11.448 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 4.8427999999999998e-05 2024-03-11T21:48:11.448 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:48:11.448 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:48:11.448 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "sub_op_commit_rec", 2024-03-11T21:48:11.448 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:47:50.431618+0000", 2024-03-11T21:48:11.448 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.000117514 2024-03-11T21:48:11.449 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:48:11.449 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:48:11.449 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "commit_sent", 2024-03-11T21:48:11.449 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:47:50.431678+0000", 2024-03-11T21:48:11.449 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 6.0452000000000003e-05 2024-03-11T21:48:11.449 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:48:11.449 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:48:11.449 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "done", 2024-03-11T21:48:11.449 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:47:50.431726+0000", 2024-03-11T21:48:11.449 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 4.7596999999999998e-05 2024-03-11T21:48:11.449 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-03-11T21:48:11.449 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-03-11T21:48:11.449 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-03-11T21:48:11.449 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:48:11.449 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:48:11.449 INFO:teuthology.orchestra.run.smithi092.stdout: "description": "osd_op(client.35813.0:228727 11.7 11:ef11062a:::benchmark_data_smithi092_186520_object90379:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e491)", 2024-03-11T21:48:11.449 INFO:teuthology.orchestra.run.smithi092.stdout: "initiated_at": "2024-03-11T21:47:49.376422+0000", 2024-03-11T21:48:11.450 INFO:teuthology.orchestra.run.smithi092.stdout: "age": 21.486388765000001, 2024-03-11T21:48:11.450 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.055531352, 2024-03-11T21:48:11.450 INFO:teuthology.orchestra.run.smithi092.stdout: "type_data": { 2024-03-11T21:48:11.450 INFO:teuthology.orchestra.run.smithi092.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-11T21:48:11.450 INFO:teuthology.orchestra.run.smithi092.stdout: "client_info": { 2024-03-11T21:48:11.450 INFO:teuthology.orchestra.run.smithi092.stdout: "client": "client.35813", 2024-03-11T21:48:11.450 INFO:teuthology.orchestra.run.smithi092.stdout: "client_addr": "v1:172.21.15.92:0/4293352030", 2024-03-11T21:48:11.450 INFO:teuthology.orchestra.run.smithi092.stdout: "tid": 228727 2024-03-11T21:48:11.450 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:48:11.450 INFO:teuthology.orchestra.run.smithi092.stdout: "events": [ 2024-03-11T21:48:11.450 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:48:11.450 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "initiated", 2024-03-11T21:48:11.450 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:47:49.376422+0000", 2024-03-11T21:48:11.450 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-03-11T21:48:11.451 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:48:11.451 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:48:11.451 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "header_read", 2024-03-11T21:48:11.451 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:47:49.376422+0000", 2024-03-11T21:48:11.451 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-03-11T21:48:11.451 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:48:11.451 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:48:11.451 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "throttled", 2024-03-11T21:48:11.451 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:47:49.376423+0000", 2024-03-11T21:48:11.451 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 9.3500000000000005e-07 2024-03-11T21:48:11.451 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:48:11.451 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:48:11.451 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "all_read", 2024-03-11T21:48:11.451 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:47:49.376427+0000", 2024-03-11T21:48:11.451 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 3.7299999999999999e-06 2024-03-11T21:48:11.452 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:48:11.452 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:48:11.452 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "dispatched", 2024-03-11T21:48:11.452 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:47:49.376434+0000", 2024-03-11T21:48:11.452 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 7.238e-06 2024-03-11T21:48:11.452 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:48:11.452 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:48:11.452 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "queued_for_pg", 2024-03-11T21:48:11.452 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:47:49.376441+0000", 2024-03-11T21:48:11.452 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 6.5649999999999998e-06 2024-03-11T21:48:11.452 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:48:11.452 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:48:11.452 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-03-11T21:48:11.452 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:47:49.376492+0000", 2024-03-11T21:48:11.452 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 5.0958999999999998e-05 2024-03-11T21:48:11.452 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:48:11.452 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:48:11.453 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "waiting for peered", 2024-03-11T21:48:11.453 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:47:49.376500+0000", 2024-03-11T21:48:11.453 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 8.4460000000000001e-06 2024-03-11T21:48:11.453 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:48:11.453 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:48:11.453 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-03-11T21:48:11.453 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:47:50.428147+0000", 2024-03-11T21:48:11.453 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.051646544 2024-03-11T21:48:11.453 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:48:11.453 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:48:11.453 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "started", 2024-03-11T21:48:11.453 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:47:50.428253+0000", 2024-03-11T21:48:11.453 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00010620000000000001 2024-03-11T21:48:11.453 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:48:11.453 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:48:11.454 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "waiting for subops from 0,7", 2024-03-11T21:48:11.454 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:47:50.428378+0000", 2024-03-11T21:48:11.454 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00012508899999999999 2024-03-11T21:48:11.454 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:48:11.454 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:48:11.454 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "sub_op_commit_rec", 2024-03-11T21:48:11.454 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:47:50.431553+0000", 2024-03-11T21:48:11.454 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.0031750889999999999 2024-03-11T21:48:11.454 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:48:11.454 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:48:11.454 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "op_commit", 2024-03-11T21:48:11.454 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:47:50.431565+0000", 2024-03-11T21:48:11.454 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.2231e-05 2024-03-11T21:48:11.454 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:48:11.455 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:48:11.455 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "sub_op_commit_rec", 2024-03-11T21:48:11.455 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:47:50.431857+0000", 2024-03-11T21:48:11.455 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.000291563 2024-03-11T21:48:11.455 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:48:11.455 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:48:11.455 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "commit_sent", 2024-03-11T21:48:11.455 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:47:50.431911+0000", 2024-03-11T21:48:11.455 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 5.4153999999999997e-05 2024-03-11T21:48:11.455 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:48:11.455 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:48:11.455 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "done", 2024-03-11T21:48:11.455 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:47:50.431954+0000", 2024-03-11T21:48:11.455 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 4.2608999999999998e-05 2024-03-11T21:48:11.455 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-03-11T21:48:11.456 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-03-11T21:48:11.456 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-03-11T21:48:11.456 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:48:11.456 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:48:11.456 INFO:teuthology.orchestra.run.smithi092.stdout: "description": "osd_op(client.35813.0:228728 11.7 11:ef86c274:::benchmark_data_smithi092_186520_object90380:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e491)", 2024-03-11T21:48:11.456 INFO:teuthology.orchestra.run.smithi092.stdout: "initiated_at": "2024-03-11T21:47:49.376811+0000", 2024-03-11T21:48:11.456 INFO:teuthology.orchestra.run.smithi092.stdout: "age": 21.485999969000002, 2024-03-11T21:48:11.456 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.0568614190000001, 2024-03-11T21:48:11.456 INFO:teuthology.orchestra.run.smithi092.stdout: "type_data": { 2024-03-11T21:48:11.456 INFO:teuthology.orchestra.run.smithi092.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-11T21:48:11.456 INFO:teuthology.orchestra.run.smithi092.stdout: "client_info": { 2024-03-11T21:48:11.456 INFO:teuthology.orchestra.run.smithi092.stdout: "client": "client.35813", 2024-03-11T21:48:11.456 INFO:teuthology.orchestra.run.smithi092.stdout: "client_addr": "v1:172.21.15.92:0/4293352030", 2024-03-11T21:48:11.456 INFO:teuthology.orchestra.run.smithi092.stdout: "tid": 228728 2024-03-11T21:48:11.456 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:48:11.457 INFO:teuthology.orchestra.run.smithi092.stdout: "events": [ 2024-03-11T21:48:11.457 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:48:11.457 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "initiated", 2024-03-11T21:48:11.457 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:47:49.376811+0000", 2024-03-11T21:48:11.457 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-03-11T21:48:11.457 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:48:11.457 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:48:11.457 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "header_read", 2024-03-11T21:48:11.457 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:47:49.376811+0000", 2024-03-11T21:48:11.457 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-03-11T21:48:11.457 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:48:11.457 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:48:11.457 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "throttled", 2024-03-11T21:48:11.457 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:47:49.376812+0000", 2024-03-11T21:48:11.458 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.1620000000000001e-06 2024-03-11T21:48:11.458 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:48:11.458 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:48:11.458 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "all_read", 2024-03-11T21:48:11.458 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:47:49.376817+0000", 2024-03-11T21:48:11.458 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 4.8010000000000003e-06 2024-03-11T21:48:11.458 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:48:11.458 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:48:11.458 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "dispatched", 2024-03-11T21:48:11.458 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:47:49.376830+0000", 2024-03-11T21:48:11.458 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.2714999999999999e-05 2024-03-11T21:48:11.458 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:48:11.458 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:48:11.458 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "queued_for_pg", 2024-03-11T21:48:11.458 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:47:49.376837+0000", 2024-03-11T21:48:11.459 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 7.6310000000000003e-06 2024-03-11T21:48:11.459 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:48:11.459 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:48:11.459 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-03-11T21:48:11.459 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:47:49.376947+0000", 2024-03-11T21:48:11.459 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.000109366 2024-03-11T21:48:11.459 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:48:11.459 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:48:11.459 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "waiting for peered", 2024-03-11T21:48:11.459 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:47:49.376965+0000", 2024-03-11T21:48:11.459 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.8451999999999999e-05 2024-03-11T21:48:11.459 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:48:11.459 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:48:11.459 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-03-11T21:48:11.459 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:47:50.429881+0000", 2024-03-11T21:48:11.459 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.052915415 2024-03-11T21:48:11.460 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:48:11.460 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:48:11.460 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "started", 2024-03-11T21:48:11.460 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:47:50.429993+0000", 2024-03-11T21:48:11.460 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00011218900000000001 2024-03-11T21:48:11.460 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:48:11.460 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:48:11.460 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "waiting for subops from 0,7", 2024-03-11T21:48:11.460 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:47:50.430130+0000", 2024-03-11T21:48:11.460 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.000137491 2024-03-11T21:48:11.460 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:48:11.460 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:48:11.460 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "sub_op_commit_rec", 2024-03-11T21:48:11.460 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:47:50.431787+0000", 2024-03-11T21:48:11.460 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.001656355 2024-03-11T21:48:11.461 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:48:11.461 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:48:11.461 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "op_commit", 2024-03-11T21:48:11.461 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:47:50.433522+0000", 2024-03-11T21:48:11.461 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.001735353 2024-03-11T21:48:11.461 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:48:11.461 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:48:11.461 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "sub_op_commit_rec", 2024-03-11T21:48:11.461 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:47:50.433579+0000", 2024-03-11T21:48:11.461 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 5.6496000000000003e-05 2024-03-11T21:48:11.461 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:48:11.461 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:48:11.461 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "commit_sent", 2024-03-11T21:48:11.461 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:47:50.433631+0000", 2024-03-11T21:48:11.462 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 5.2880999999999997e-05 2024-03-11T21:48:11.462 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:48:11.462 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:48:11.462 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "done", 2024-03-11T21:48:11.462 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:47:50.433673+0000", 2024-03-11T21:48:11.462 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 4.1112000000000002e-05 2024-03-11T21:48:11.462 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-03-11T21:48:11.462 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-03-11T21:48:11.462 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-03-11T21:48:11.462 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:48:11.462 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:48:11.462 INFO:teuthology.orchestra.run.smithi092.stdout: "description": "osd_op(client.35813.0:228733 11.7 11:eb783042:::benchmark_data_smithi092_186520_object90385:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e491)", 2024-03-11T21:48:11.462 INFO:teuthology.orchestra.run.smithi092.stdout: "initiated_at": "2024-03-11T21:47:49.379929+0000", 2024-03-11T21:48:11.462 INFO:teuthology.orchestra.run.smithi092.stdout: "age": 21.482881655, 2024-03-11T21:48:11.463 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.0563819649999999, 2024-03-11T21:48:11.463 INFO:teuthology.orchestra.run.smithi092.stdout: "type_data": { 2024-03-11T21:48:11.463 INFO:teuthology.orchestra.run.smithi092.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-11T21:48:11.463 INFO:teuthology.orchestra.run.smithi092.stdout: "client_info": { 2024-03-11T21:48:11.463 INFO:teuthology.orchestra.run.smithi092.stdout: "client": "client.35813", 2024-03-11T21:48:11.463 INFO:teuthology.orchestra.run.smithi092.stdout: "client_addr": "v1:172.21.15.92:0/4293352030", 2024-03-11T21:48:11.463 INFO:teuthology.orchestra.run.smithi092.stdout: "tid": 228733 2024-03-11T21:48:11.463 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:48:11.463 INFO:teuthology.orchestra.run.smithi092.stdout: "events": [ 2024-03-11T21:48:11.463 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:48:11.463 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "initiated", 2024-03-11T21:48:11.463 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:47:49.379929+0000", 2024-03-11T21:48:11.463 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-03-11T21:48:11.463 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:48:11.463 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:48:11.463 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "header_read", 2024-03-11T21:48:11.464 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:47:49.379929+0000", 2024-03-11T21:48:11.464 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-03-11T21:48:11.464 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:48:11.464 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:48:11.464 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "throttled", 2024-03-11T21:48:11.464 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:47:49.379931+0000", 2024-03-11T21:48:11.464 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.3740000000000001e-06 2024-03-11T21:48:11.464 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:48:11.464 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:48:11.464 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "all_read", 2024-03-11T21:48:11.464 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:47:49.379936+0000", 2024-03-11T21:48:11.464 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 4.8509999999999998e-06 2024-03-11T21:48:11.464 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:48:11.464 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:48:11.465 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "dispatched", 2024-03-11T21:48:11.465 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:47:49.379945+0000", 2024-03-11T21:48:11.465 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 9.3149999999999998e-06 2024-03-11T21:48:11.465 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:48:11.465 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:48:11.465 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "queued_for_pg", 2024-03-11T21:48:11.465 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:47:49.379952+0000", 2024-03-11T21:48:11.465 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 7.4229999999999999e-06 2024-03-11T21:48:11.465 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:48:11.465 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:48:11.465 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-03-11T21:48:11.465 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:47:49.379996+0000", 2024-03-11T21:48:11.465 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 4.3714000000000002e-05 2024-03-11T21:48:11.465 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:48:11.465 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:48:11.466 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "waiting for peered", 2024-03-11T21:48:11.466 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:47:49.380012+0000", 2024-03-11T21:48:11.466 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.5894999999999999e-05 2024-03-11T21:48:11.466 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:48:11.466 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:48:11.466 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-03-11T21:48:11.466 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:47:50.431993+0000", 2024-03-11T21:48:11.466 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.0519814649999999 2024-03-11T21:48:11.466 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:48:11.466 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:48:11.466 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "started", 2024-03-11T21:48:11.466 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:47:50.432108+0000", 2024-03-11T21:48:11.466 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00011497600000000001 2024-03-11T21:48:11.466 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:48:11.466 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:48:11.467 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "waiting for subops from 0,7", 2024-03-11T21:48:11.467 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:47:50.432225+0000", 2024-03-11T21:48:11.467 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.000116364 2024-03-11T21:48:11.467 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:48:11.467 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:48:11.467 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "sub_op_commit_rec", 2024-03-11T21:48:11.467 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:47:50.436164+0000", 2024-03-11T21:48:11.467 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.0039390750000000002 2024-03-11T21:48:11.467 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:48:11.467 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:48:11.467 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "op_commit", 2024-03-11T21:48:11.469 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:47:50.436174+0000", 2024-03-11T21:48:11.469 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 9.876e-06 2024-03-11T21:48:11.469 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:48:11.469 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:48:11.469 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "sub_op_commit_rec", 2024-03-11T21:48:11.469 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:47:50.436211+0000", 2024-03-11T21:48:11.469 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 3.7645000000000003e-05 2024-03-11T21:48:11.469 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:48:11.469 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:48:11.470 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "commit_sent", 2024-03-11T21:48:11.470 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:47:50.436266+0000", 2024-03-11T21:48:11.470 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 5.4963000000000002e-05 2024-03-11T21:48:11.470 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:48:11.470 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:48:11.470 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "done", 2024-03-11T21:48:11.470 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:47:50.436311+0000", 2024-03-11T21:48:11.470 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 4.5028999999999999e-05 2024-03-11T21:48:11.470 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-03-11T21:48:11.470 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-03-11T21:48:11.470 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-03-11T21:48:11.470 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:48:11.470 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:48:11.471 INFO:teuthology.orchestra.run.smithi092.stdout: "description": "osd_op(client.35813.0:228743 11.7 11:eb01badf:::benchmark_data_smithi092_186520_object90395:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e491)", 2024-03-11T21:48:11.471 INFO:teuthology.orchestra.run.smithi092.stdout: "initiated_at": "2024-03-11T21:47:49.386884+0000", 2024-03-11T21:48:11.471 INFO:teuthology.orchestra.run.smithi092.stdout: "age": 21.475926724000001, 2024-03-11T21:48:11.471 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.0538973970000001, 2024-03-11T21:48:11.471 INFO:teuthology.orchestra.run.smithi092.stdout: "type_data": { 2024-03-11T21:48:11.471 INFO:teuthology.orchestra.run.smithi092.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-11T21:48:11.471 INFO:teuthology.orchestra.run.smithi092.stdout: "client_info": { 2024-03-11T21:48:11.471 INFO:teuthology.orchestra.run.smithi092.stdout: "client": "client.35813", 2024-03-11T21:48:11.471 INFO:teuthology.orchestra.run.smithi092.stdout: "client_addr": "v1:172.21.15.92:0/4293352030", 2024-03-11T21:48:11.471 INFO:teuthology.orchestra.run.smithi092.stdout: "tid": 228743 2024-03-11T21:48:11.471 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:48:11.471 INFO:teuthology.orchestra.run.smithi092.stdout: "events": [ 2024-03-11T21:48:11.471 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:48:11.471 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "initiated", 2024-03-11T21:48:11.472 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:47:49.386884+0000", 2024-03-11T21:48:11.472 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-03-11T21:48:11.472 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:48:11.472 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:48:11.472 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "header_read", 2024-03-11T21:48:11.472 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:47:49.386884+0000", 2024-03-11T21:48:11.472 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-03-11T21:48:11.472 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:48:11.472 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:48:11.472 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "throttled", 2024-03-11T21:48:11.472 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:47:49.386886+0000", 2024-03-11T21:48:11.472 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 2.0159999999999998e-06 2024-03-11T21:48:11.472 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:48:11.472 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:48:11.472 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "all_read", 2024-03-11T21:48:11.473 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:47:49.386902+0000", 2024-03-11T21:48:11.473 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.5588999999999999e-05 2024-03-11T21:48:11.473 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:48:11.473 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:48:11.473 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "dispatched", 2024-03-11T21:48:11.473 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:47:49.386914+0000", 2024-03-11T21:48:11.473 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.2296999999999999e-05 2024-03-11T21:48:11.473 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:48:11.473 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:48:11.473 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "queued_for_pg", 2024-03-11T21:48:11.473 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:47:49.386921+0000", 2024-03-11T21:48:11.473 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 7.0299999999999996e-06 2024-03-11T21:48:11.473 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:48:11.473 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:48:11.473 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-03-11T21:48:11.474 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:47:49.386967+0000", 2024-03-11T21:48:11.474 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 4.6156000000000003e-05 2024-03-11T21:48:11.474 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:48:11.474 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:48:11.474 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "waiting for peered", 2024-03-11T21:48:11.474 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:47:49.386977+0000", 2024-03-11T21:48:11.474 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 9.3740000000000004e-06 2024-03-11T21:48:11.474 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:48:11.474 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:48:11.474 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-03-11T21:48:11.474 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:47:50.433728+0000", 2024-03-11T21:48:11.474 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.0467508139999999 2024-03-11T21:48:11.474 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:48:11.474 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:48:11.474 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "started", 2024-03-11T21:48:11.475 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:47:50.433846+0000", 2024-03-11T21:48:11.475 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.000117985 2024-03-11T21:48:11.475 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:48:11.475 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:48:11.475 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "waiting for subops from 0,7", 2024-03-11T21:48:11.475 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:47:50.433970+0000", 2024-03-11T21:48:11.475 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00012455800000000001 2024-03-11T21:48:11.475 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:48:11.475 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:48:11.475 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "op_commit", 2024-03-11T21:48:11.475 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:47:50.438060+0000", 2024-03-11T21:48:11.475 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.0040894729999999997 2024-03-11T21:48:11.475 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:48:11.475 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:48:11.475 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "sub_op_commit_rec", 2024-03-11T21:48:11.476 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:47:50.440566+0000", 2024-03-11T21:48:11.476 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.0025063160000000002 2024-03-11T21:48:11.476 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:48:11.476 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:48:11.476 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "sub_op_commit_rec", 2024-03-11T21:48:11.476 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:47:50.440703+0000", 2024-03-11T21:48:11.476 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00013663600000000001 2024-03-11T21:48:11.476 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:48:11.476 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:48:11.476 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "commit_sent", 2024-03-11T21:48:11.476 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:47:50.440748+0000", 2024-03-11T21:48:11.476 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 4.5374000000000003e-05 2024-03-11T21:48:11.476 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:48:11.476 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:48:11.476 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "done", 2024-03-11T21:48:11.476 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:47:50.440782+0000", 2024-03-11T21:48:11.476 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 3.3779000000000003e-05 2024-03-11T21:48:11.477 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-03-11T21:48:11.477 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-03-11T21:48:11.477 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-03-11T21:48:11.477 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:48:11.477 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:48:11.477 INFO:teuthology.orchestra.run.smithi092.stdout: "description": "osd_op(client.35813.0:228752 11.7 11:e2f48072:::benchmark_data_smithi092_186520_object90404:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e491)", 2024-03-11T21:48:11.477 INFO:teuthology.orchestra.run.smithi092.stdout: "initiated_at": "2024-03-11T21:47:49.394700+0000", 2024-03-11T21:48:11.477 INFO:teuthology.orchestra.run.smithi092.stdout: "age": 21.468110802999998, 2024-03-11T21:48:11.477 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.046269466, 2024-03-11T21:48:11.477 INFO:teuthology.orchestra.run.smithi092.stdout: "type_data": { 2024-03-11T21:48:11.477 INFO:teuthology.orchestra.run.smithi092.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-11T21:48:11.477 INFO:teuthology.orchestra.run.smithi092.stdout: "client_info": { 2024-03-11T21:48:11.477 INFO:teuthology.orchestra.run.smithi092.stdout: "client": "client.35813", 2024-03-11T21:48:11.477 INFO:teuthology.orchestra.run.smithi092.stdout: "client_addr": "v1:172.21.15.92:0/4293352030", 2024-03-11T21:48:11.477 INFO:teuthology.orchestra.run.smithi092.stdout: "tid": 228752 2024-03-11T21:48:11.477 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:48:11.478 INFO:teuthology.orchestra.run.smithi092.stdout: "events": [ 2024-03-11T21:48:11.478 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:48:11.478 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "initiated", 2024-03-11T21:48:11.478 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:47:49.394700+0000", 2024-03-11T21:48:11.478 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-03-11T21:48:11.478 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:48:11.478 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:48:11.478 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "header_read", 2024-03-11T21:48:11.478 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:47:49.394700+0000", 2024-03-11T21:48:11.478 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-03-11T21:48:11.478 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:48:11.478 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:48:11.478 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "throttled", 2024-03-11T21:48:11.478 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:47:49.394702+0000", 2024-03-11T21:48:11.478 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.2309999999999999e-06 2024-03-11T21:48:11.478 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:48:11.479 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:48:11.479 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "all_read", 2024-03-11T21:48:11.479 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:47:49.394706+0000", 2024-03-11T21:48:11.479 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 4.8080000000000004e-06 2024-03-11T21:48:11.479 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:48:11.479 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:48:11.479 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "dispatched", 2024-03-11T21:48:11.479 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:47:49.394716+0000", 2024-03-11T21:48:11.479 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 9.9490000000000008e-06 2024-03-11T21:48:11.479 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:48:11.479 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:48:11.479 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "queued_for_pg", 2024-03-11T21:48:11.479 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:47:49.394724+0000", 2024-03-11T21:48:11.479 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 7.3189999999999997e-06 2024-03-11T21:48:11.479 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:48:11.479 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:48:11.479 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-03-11T21:48:11.480 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:47:49.394775+0000", 2024-03-11T21:48:11.480 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 5.1863999999999997e-05 2024-03-11T21:48:11.480 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:48:11.480 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:48:11.480 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "waiting for peered", 2024-03-11T21:48:11.480 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:47:49.394783+0000", 2024-03-11T21:48:11.480 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 8.0150000000000003e-06 2024-03-11T21:48:11.480 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:48:11.480 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:48:11.480 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-03-11T21:48:11.480 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:47:50.435045+0000", 2024-03-11T21:48:11.480 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.040261863 2024-03-11T21:48:11.480 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:48:11.480 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:48:11.481 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "started", 2024-03-11T21:48:11.481 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:47:50.435124+0000", 2024-03-11T21:48:11.481 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 7.8517999999999996e-05 2024-03-11T21:48:11.481 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:48:11.481 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:48:11.481 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "waiting for subops from 0,7", 2024-03-11T21:48:11.481 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:47:50.435204+0000", 2024-03-11T21:48:11.481 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 7.9969999999999995e-05 2024-03-11T21:48:11.481 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:48:11.481 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:48:11.481 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "op_commit", 2024-03-11T21:48:11.481 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:47:50.440509+0000", 2024-03-11T21:48:11.481 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.0053046949999999999 2024-03-11T21:48:11.481 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:48:11.481 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:48:11.481 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "sub_op_commit_rec", 2024-03-11T21:48:11.481 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:47:50.440830+0000", 2024-03-11T21:48:11.482 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.000321362 2024-03-11T21:48:11.482 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:48:11.482 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:48:11.482 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "sub_op_commit_rec", 2024-03-11T21:48:11.482 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:47:50.440880+0000", 2024-03-11T21:48:11.482 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 4.9753999999999998e-05 2024-03-11T21:48:11.482 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:48:11.482 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:48:11.482 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "commit_sent", 2024-03-11T21:48:11.482 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:47:50.440931+0000", 2024-03-11T21:48:11.482 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 5.1304999999999998e-05 2024-03-11T21:48:11.482 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:48:11.482 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:48:11.482 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "done", 2024-03-11T21:48:11.482 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:47:50.440970+0000", 2024-03-11T21:48:11.483 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 3.8813000000000003e-05 2024-03-11T21:48:11.483 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-03-11T21:48:11.483 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-03-11T21:48:11.483 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-03-11T21:48:11.483 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:48:11.483 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:48:11.483 INFO:teuthology.orchestra.run.smithi092.stdout: "description": "osd_op(client.35813.0:228766 11.7 11:e8c1f47f:::benchmark_data_smithi092_186520_object90418:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e491)", 2024-03-11T21:48:11.483 INFO:teuthology.orchestra.run.smithi092.stdout: "initiated_at": "2024-03-11T21:47:49.404895+0000", 2024-03-11T21:48:11.483 INFO:teuthology.orchestra.run.smithi092.stdout: "age": 21.457916379, 2024-03-11T21:48:11.483 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.036265019, 2024-03-11T21:48:11.483 INFO:teuthology.orchestra.run.smithi092.stdout: "type_data": { 2024-03-11T21:48:11.483 INFO:teuthology.orchestra.run.smithi092.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-11T21:48:11.484 INFO:teuthology.orchestra.run.smithi092.stdout: "client_info": { 2024-03-11T21:48:11.484 INFO:teuthology.orchestra.run.smithi092.stdout: "client": "client.35813", 2024-03-11T21:48:11.484 INFO:teuthology.orchestra.run.smithi092.stdout: "client_addr": "v1:172.21.15.92:0/4293352030", 2024-03-11T21:48:11.484 INFO:teuthology.orchestra.run.smithi092.stdout: "tid": 228766 2024-03-11T21:48:11.484 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:48:11.484 INFO:teuthology.orchestra.run.smithi092.stdout: "events": [ 2024-03-11T21:48:11.484 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:48:11.484 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "initiated", 2024-03-11T21:48:11.484 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:47:49.404895+0000", 2024-03-11T21:48:11.484 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-03-11T21:48:11.484 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:48:11.484 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:48:11.484 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "header_read", 2024-03-11T21:48:11.484 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:47:49.404895+0000", 2024-03-11T21:48:11.485 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-03-11T21:48:11.485 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:48:11.485 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:48:11.485 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "throttled", 2024-03-11T21:48:11.485 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:47:49.404897+0000", 2024-03-11T21:48:11.485 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.787e-06 2024-03-11T21:48:11.485 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:48:11.485 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:48:11.485 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "all_read", 2024-03-11T21:48:11.485 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:47:49.404902+0000", 2024-03-11T21:48:11.485 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 5.8710000000000002e-06 2024-03-11T21:48:11.485 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:48:11.485 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:48:11.485 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "dispatched", 2024-03-11T21:48:11.485 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:47:49.404914+0000", 2024-03-11T21:48:11.486 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.1854e-05 2024-03-11T21:48:11.486 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:48:11.486 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:48:11.486 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "queued_for_pg", 2024-03-11T21:48:11.486 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:47:49.404921+0000", 2024-03-11T21:48:11.486 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 7.1049999999999997e-06 2024-03-11T21:48:11.486 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:48:11.486 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:48:11.486 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-03-11T21:48:11.486 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:47:49.404986+0000", 2024-03-11T21:48:11.486 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 6.5000999999999996e-05 2024-03-11T21:48:11.486 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:48:11.486 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:48:11.486 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "waiting for peered", 2024-03-11T21:48:11.487 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:47:49.404994+0000", 2024-03-11T21:48:11.487 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 8.0579999999999998e-06 2024-03-11T21:48:11.487 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:48:11.487 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:48:11.487 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-03-11T21:48:11.487 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:47:50.436368+0000", 2024-03-11T21:48:11.487 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.031373434 2024-03-11T21:48:11.487 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:48:11.487 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:48:11.487 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "started", 2024-03-11T21:48:11.487 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:47:50.436466+0000", 2024-03-11T21:48:11.487 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 9.7991000000000004e-05 2024-03-11T21:48:11.487 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:48:11.488 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:48:11.488 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "waiting for subops from 0,7", 2024-03-11T21:48:11.488 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:47:50.436583+0000", 2024-03-11T21:48:11.488 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.000116801 2024-03-11T21:48:11.488 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:48:11.488 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:48:11.488 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "op_commit", 2024-03-11T21:48:11.488 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:47:50.440657+0000", 2024-03-11T21:48:11.488 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.0040745149999999999 2024-03-11T21:48:11.488 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:48:11.488 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:48:11.488 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "sub_op_commit_rec", 2024-03-11T21:48:11.488 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:47:50.441033+0000", 2024-03-11T21:48:11.488 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00037570300000000002 2024-03-11T21:48:11.489 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:48:11.489 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:48:11.489 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "sub_op_commit_rec", 2024-03-11T21:48:11.489 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:47:50.441073+0000", 2024-03-11T21:48:11.489 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 3.9780000000000002e-05 2024-03-11T21:48:11.489 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:48:11.489 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:48:11.489 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "commit_sent", 2024-03-11T21:48:11.489 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:47:50.441120+0000", 2024-03-11T21:48:11.489 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 4.7681e-05 2024-03-11T21:48:11.489 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:48:11.489 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:48:11.489 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "done", 2024-03-11T21:48:11.489 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:47:50.441160+0000", 2024-03-11T21:48:11.490 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 3.9437999999999997e-05 2024-03-11T21:48:11.490 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-03-11T21:48:11.490 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-03-11T21:48:11.490 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-03-11T21:48:11.490 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:48:11.490 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:48:11.490 INFO:teuthology.orchestra.run.smithi092.stdout: "description": "osd_op(client.35813.0:228789 11.7 11:e5ee4000:::benchmark_data_smithi092_186520_object90441:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e491)", 2024-03-11T21:48:11.490 INFO:teuthology.orchestra.run.smithi092.stdout: "initiated_at": "2024-03-11T21:47:49.420244+0000", 2024-03-11T21:48:11.490 INFO:teuthology.orchestra.run.smithi092.stdout: "age": 21.442567569000001, 2024-03-11T21:48:11.490 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.0260051349999999, 2024-03-11T21:48:11.490 INFO:teuthology.orchestra.run.smithi092.stdout: "type_data": { 2024-03-11T21:48:11.490 INFO:teuthology.orchestra.run.smithi092.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-11T21:48:11.490 INFO:teuthology.orchestra.run.smithi092.stdout: "client_info": { 2024-03-11T21:48:11.490 INFO:teuthology.orchestra.run.smithi092.stdout: "client": "client.35813", 2024-03-11T21:48:11.491 INFO:teuthology.orchestra.run.smithi092.stdout: "client_addr": "v1:172.21.15.92:0/4293352030", 2024-03-11T21:48:11.491 INFO:teuthology.orchestra.run.smithi092.stdout: "tid": 228789 2024-03-11T21:48:11.491 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:48:11.491 INFO:teuthology.orchestra.run.smithi092.stdout: "events": [ 2024-03-11T21:48:11.491 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:48:11.491 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "initiated", 2024-03-11T21:48:11.491 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:47:49.420244+0000", 2024-03-11T21:48:11.491 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-03-11T21:48:11.491 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:48:11.491 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:48:11.491 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "header_read", 2024-03-11T21:48:11.491 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:47:49.420244+0000", 2024-03-11T21:48:11.491 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-03-11T21:48:11.491 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:48:11.491 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:48:11.491 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "throttled", 2024-03-11T21:48:11.492 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:47:49.420245+0000", 2024-03-11T21:48:11.492 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.9549999999999999e-06 2024-03-11T21:48:11.492 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:48:11.492 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:48:11.492 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "all_read", 2024-03-11T21:48:11.492 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:47:49.420251+0000", 2024-03-11T21:48:11.492 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 5.959e-06 2024-03-11T21:48:11.492 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:48:11.492 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:48:11.492 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "dispatched", 2024-03-11T21:48:11.492 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:47:49.420263+0000", 2024-03-11T21:48:11.492 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.1319e-05 2024-03-11T21:48:11.492 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:48:11.492 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:48:11.492 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "queued_for_pg", 2024-03-11T21:48:11.493 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:47:49.420271+0000", 2024-03-11T21:48:11.493 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 7.8390000000000007e-06 2024-03-11T21:48:11.493 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:48:11.493 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:48:11.493 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-03-11T21:48:11.493 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:47:49.420335+0000", 2024-03-11T21:48:11.493 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 6.4234999999999997e-05 2024-03-11T21:48:11.493 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:48:11.493 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:48:11.493 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "waiting for peered", 2024-03-11T21:48:11.493 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:47:49.420346+0000", 2024-03-11T21:48:11.493 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.0902e-05 2024-03-11T21:48:11.493 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:48:11.493 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:48:11.494 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-03-11T21:48:11.494 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:47:50.438090+0000", 2024-03-11T21:48:11.494 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.0177440470000001 2024-03-11T21:48:11.494 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:48:11.494 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:48:11.494 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "started", 2024-03-11T21:48:11.494 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:47:50.438219+0000", 2024-03-11T21:48:11.494 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00012955200000000001 2024-03-11T21:48:11.494 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:48:11.494 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:48:11.494 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "waiting for subops from 0,7", 2024-03-11T21:48:11.494 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:47:50.438330+0000", 2024-03-11T21:48:11.494 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.000110907 2024-03-11T21:48:11.495 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:48:11.495 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:48:11.495 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "op_commit", 2024-03-11T21:48:11.495 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:47:50.440985+0000", 2024-03-11T21:48:11.495 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.0026546989999999999 2024-03-11T21:48:11.495 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:48:11.495 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:48:11.495 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "sub_op_commit_rec", 2024-03-11T21:48:11.495 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:47:50.441221+0000", 2024-03-11T21:48:11.495 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00023635299999999999 2024-03-11T21:48:11.495 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:48:11.495 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:48:11.495 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "sub_op_commit_rec", 2024-03-11T21:48:11.495 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:47:50.446137+0000", 2024-03-11T21:48:11.495 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.0049156110000000003 2024-03-11T21:48:11.496 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:48:11.496 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:48:11.496 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "commit_sent", 2024-03-11T21:48:11.496 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:47:50.446195+0000", 2024-03-11T21:48:11.496 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 5.8536000000000003e-05 2024-03-11T21:48:11.496 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:48:11.496 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:48:11.496 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "done", 2024-03-11T21:48:11.496 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:47:50.446249+0000", 2024-03-11T21:48:11.496 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 5.3220999999999997e-05 2024-03-11T21:48:11.496 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-03-11T21:48:11.496 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-03-11T21:48:11.496 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-03-11T21:48:11.496 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:48:11.496 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:48:11.496 INFO:teuthology.orchestra.run.smithi092.stdout: "description": "osd_op(client.35813.0:228803 11.7 11:e7b14f07:::benchmark_data_smithi092_186520_object90455:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e491)", 2024-03-11T21:48:11.497 INFO:teuthology.orchestra.run.smithi092.stdout: "initiated_at": "2024-03-11T21:47:49.430080+0000", 2024-03-11T21:48:11.497 INFO:teuthology.orchestra.run.smithi092.stdout: "age": 21.432730689, 2024-03-11T21:48:11.497 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.01636751, 2024-03-11T21:48:11.497 INFO:teuthology.orchestra.run.smithi092.stdout: "type_data": { 2024-03-11T21:48:11.497 INFO:teuthology.orchestra.run.smithi092.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-11T21:48:11.497 INFO:teuthology.orchestra.run.smithi092.stdout: "client_info": { 2024-03-11T21:48:11.497 INFO:teuthology.orchestra.run.smithi092.stdout: "client": "client.35813", 2024-03-11T21:48:11.497 INFO:teuthology.orchestra.run.smithi092.stdout: "client_addr": "v1:172.21.15.92:0/4293352030", 2024-03-11T21:48:11.497 INFO:teuthology.orchestra.run.smithi092.stdout: "tid": 228803 2024-03-11T21:48:11.497 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:48:11.497 INFO:teuthology.orchestra.run.smithi092.stdout: "events": [ 2024-03-11T21:48:11.497 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:48:11.497 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "initiated", 2024-03-11T21:48:11.497 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:47:49.430080+0000", 2024-03-11T21:48:11.498 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-03-11T21:48:11.498 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:48:11.498 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:48:11.498 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "header_read", 2024-03-11T21:48:11.498 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:47:49.430080+0000", 2024-03-11T21:48:11.498 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-03-11T21:48:11.498 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:48:11.498 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:48:11.498 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "throttled", 2024-03-11T21:48:11.498 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:47:49.430082+0000", 2024-03-11T21:48:11.498 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.9740000000000001e-06 2024-03-11T21:48:11.498 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:48:11.498 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:48:11.498 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "all_read", 2024-03-11T21:48:11.499 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:47:49.430088+0000", 2024-03-11T21:48:11.499 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 5.9220000000000002e-06 2024-03-11T21:48:11.499 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:48:11.499 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:48:11.499 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "dispatched", 2024-03-11T21:48:11.499 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:47:49.430100+0000", 2024-03-11T21:48:11.499 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.2088e-05 2024-03-11T21:48:11.499 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:48:11.499 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:48:11.499 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "queued_for_pg", 2024-03-11T21:48:11.499 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:47:49.430109+0000", 2024-03-11T21:48:11.499 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 8.1620000000000008e-06 2024-03-11T21:48:11.499 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:48:11.499 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:48:11.499 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-03-11T21:48:11.500 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:47:49.430256+0000", 2024-03-11T21:48:11.500 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00014763600000000001 2024-03-11T21:48:11.500 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:48:11.500 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:48:11.500 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "waiting for peered", 2024-03-11T21:48:11.500 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:47:49.430266+0000", 2024-03-11T21:48:11.500 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 9.5359999999999998e-06 2024-03-11T21:48:11.500 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:48:11.500 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:48:11.500 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-03-11T21:48:11.500 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:47:50.441283+0000", 2024-03-11T21:48:11.500 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.0110177600000001 2024-03-11T21:48:11.500 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:48:11.500 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:48:11.500 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "started", 2024-03-11T21:48:11.500 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:47:50.441413+0000", 2024-03-11T21:48:11.501 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00012959199999999999 2024-03-11T21:48:11.501 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:48:11.501 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:48:11.501 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "waiting for subops from 0,7", 2024-03-11T21:48:11.501 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:47:50.441539+0000", 2024-03-11T21:48:11.501 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00012592699999999999 2024-03-11T21:48:11.501 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:48:11.501 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:48:11.501 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "sub_op_commit_rec", 2024-03-11T21:48:11.501 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:47:50.446063+0000", 2024-03-11T21:48:11.501 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.0045237680000000001 2024-03-11T21:48:11.501 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:48:11.501 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:48:11.501 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "op_commit", 2024-03-11T21:48:11.501 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:47:50.446078+0000", 2024-03-11T21:48:11.501 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.5682999999999999e-05 2024-03-11T21:48:11.501 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:48:11.502 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:48:11.502 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "sub_op_commit_rec", 2024-03-11T21:48:11.502 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:47:50.446316+0000", 2024-03-11T21:48:11.502 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00023751999999999999 2024-03-11T21:48:11.502 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:48:11.502 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:48:11.502 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "commit_sent", 2024-03-11T21:48:11.502 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:47:50.446406+0000", 2024-03-11T21:48:11.502 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 9.0283000000000003e-05 2024-03-11T21:48:11.502 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:48:11.502 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:48:11.502 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "done", 2024-03-11T21:48:11.502 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:47:50.446448+0000", 2024-03-11T21:48:11.503 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 4.1659000000000002e-05 2024-03-11T21:48:11.503 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-03-11T21:48:11.503 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-03-11T21:48:11.503 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-03-11T21:48:11.503 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:48:11.503 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:48:11.503 INFO:teuthology.orchestra.run.smithi092.stdout: "description": "osd_op(client.35813.0:228808 11.7 11:e4dd5910:::benchmark_data_smithi092_186520_object90460:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e491)", 2024-03-11T21:48:11.503 INFO:teuthology.orchestra.run.smithi092.stdout: "initiated_at": "2024-03-11T21:47:49.431531+0000", 2024-03-11T21:48:11.503 INFO:teuthology.orchestra.run.smithi092.stdout: "age": 21.431280398999998, 2024-03-11T21:48:11.503 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.0218557349999999, 2024-03-11T21:48:11.503 INFO:teuthology.orchestra.run.smithi092.stdout: "type_data": { 2024-03-11T21:48:11.503 INFO:teuthology.orchestra.run.smithi092.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-11T21:48:11.503 INFO:teuthology.orchestra.run.smithi092.stdout: "client_info": { 2024-03-11T21:48:11.503 INFO:teuthology.orchestra.run.smithi092.stdout: "client": "client.35813", 2024-03-11T21:48:11.503 INFO:teuthology.orchestra.run.smithi092.stdout: "client_addr": "v1:172.21.15.92:0/4293352030", 2024-03-11T21:48:11.503 INFO:teuthology.orchestra.run.smithi092.stdout: "tid": 228808 2024-03-11T21:48:11.503 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:48:11.504 INFO:teuthology.orchestra.run.smithi092.stdout: "events": [ 2024-03-11T21:48:11.504 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:48:11.504 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "initiated", 2024-03-11T21:48:11.504 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:47:49.431531+0000", 2024-03-11T21:48:11.504 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-03-11T21:48:11.504 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:48:11.504 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:48:11.504 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "header_read", 2024-03-11T21:48:11.504 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:47:49.431531+0000", 2024-03-11T21:48:11.504 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-03-11T21:48:11.504 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:48:11.504 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:48:11.504 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "throttled", 2024-03-11T21:48:11.504 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:47:49.431532+0000", 2024-03-11T21:48:11.504 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.0780000000000001e-06 2024-03-11T21:48:11.505 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:48:11.505 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:48:11.505 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "all_read", 2024-03-11T21:48:11.505 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:47:49.431536+0000", 2024-03-11T21:48:11.505 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 3.7189999999999999e-06 2024-03-11T21:48:11.505 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:48:11.505 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:48:11.505 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "dispatched", 2024-03-11T21:48:11.505 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:47:49.431543+0000", 2024-03-11T21:48:11.505 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 7.6590000000000007e-06 2024-03-11T21:48:11.505 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:48:11.505 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:48:11.505 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "queued_for_pg", 2024-03-11T21:48:11.505 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:47:49.431550+0000", 2024-03-11T21:48:11.505 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 6.669e-06 2024-03-11T21:48:11.506 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:48:11.506 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:48:11.506 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-03-11T21:48:11.506 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:47:49.431627+0000", 2024-03-11T21:48:11.506 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 7.7418999999999999e-05 2024-03-11T21:48:11.506 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:48:11.506 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:48:11.506 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "waiting for peered", 2024-03-11T21:48:11.506 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:47:49.431635+0000", 2024-03-11T21:48:11.506 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 7.8730000000000007e-06 2024-03-11T21:48:11.506 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:48:11.506 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:48:11.506 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-03-11T21:48:11.506 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:47:50.443213+0000", 2024-03-11T21:48:11.506 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.0115781500000001 2024-03-11T21:48:11.507 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:48:11.507 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:48:11.507 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "started", 2024-03-11T21:48:11.507 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:47:50.443314+0000", 2024-03-11T21:48:11.507 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00010101499999999999 2024-03-11T21:48:11.507 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:48:11.507 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:48:11.507 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "waiting for subops from 0,7", 2024-03-11T21:48:11.507 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:47:50.443425+0000", 2024-03-11T21:48:11.507 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.000110226 2024-03-11T21:48:11.507 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:48:11.507 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:48:11.507 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "op_commit", 2024-03-11T21:48:11.507 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:47:50.450051+0000", 2024-03-11T21:48:11.507 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.0066266490000000001 2024-03-11T21:48:11.507 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:48:11.507 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:48:11.508 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "sub_op_commit_rec", 2024-03-11T21:48:11.508 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:47:50.452773+0000", 2024-03-11T21:48:11.508 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.0027216699999999998 2024-03-11T21:48:11.508 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:48:11.508 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:48:11.508 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "sub_op_commit_rec", 2024-03-11T21:48:11.508 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:47:50.453208+0000", 2024-03-11T21:48:11.508 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00043504900000000002 2024-03-11T21:48:11.508 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:48:11.508 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:48:11.508 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "commit_sent", 2024-03-11T21:48:11.508 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:47:50.453334+0000", 2024-03-11T21:48:11.508 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.000125685 2024-03-11T21:48:11.508 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:48:11.508 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:48:11.509 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "done", 2024-03-11T21:48:11.509 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:47:50.453386+0000", 2024-03-11T21:48:11.509 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 5.2874000000000002e-05 2024-03-11T21:48:11.509 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-03-11T21:48:11.509 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-03-11T21:48:11.509 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-03-11T21:48:11.509 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:48:11.509 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:48:11.509 INFO:teuthology.orchestra.run.smithi092.stdout: "description": "osd_op(client.35813.0:228824 11.7 11:ef557c68:::benchmark_data_smithi092_186520_object90476:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e491)", 2024-03-11T21:48:11.509 INFO:teuthology.orchestra.run.smithi092.stdout: "initiated_at": "2024-03-11T21:47:49.443927+0000", 2024-03-11T21:48:11.509 INFO:teuthology.orchestra.run.smithi092.stdout: "age": 21.418884155000001, 2024-03-11T21:48:11.509 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.0118628270000001, 2024-03-11T21:48:11.509 INFO:teuthology.orchestra.run.smithi092.stdout: "type_data": { 2024-03-11T21:48:11.509 INFO:teuthology.orchestra.run.smithi092.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-11T21:48:11.509 INFO:teuthology.orchestra.run.smithi092.stdout: "client_info": { 2024-03-11T21:48:11.510 INFO:teuthology.orchestra.run.smithi092.stdout: "client": "client.35813", 2024-03-11T21:48:11.510 INFO:teuthology.orchestra.run.smithi092.stdout: "client_addr": "v1:172.21.15.92:0/4293352030", 2024-03-11T21:48:11.510 INFO:teuthology.orchestra.run.smithi092.stdout: "tid": 228824 2024-03-11T21:48:11.510 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:48:11.510 INFO:teuthology.orchestra.run.smithi092.stdout: "events": [ 2024-03-11T21:48:11.510 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:48:11.510 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "initiated", 2024-03-11T21:48:11.510 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:47:49.443927+0000", 2024-03-11T21:48:11.510 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-03-11T21:48:11.510 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:48:11.510 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:48:11.510 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "header_read", 2024-03-11T21:48:11.510 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:47:49.443927+0000", 2024-03-11T21:48:11.510 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-03-11T21:48:11.510 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:48:11.511 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:48:11.511 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "throttled", 2024-03-11T21:48:11.511 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:47:49.443928+0000", 2024-03-11T21:48:11.511 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.3790000000000001e-06 2024-03-11T21:48:11.511 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:48:11.511 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:48:11.511 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "all_read", 2024-03-11T21:48:11.511 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:47:49.443933+0000", 2024-03-11T21:48:11.511 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 4.3640000000000001e-06 2024-03-11T21:48:11.511 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:48:11.511 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:48:11.511 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "dispatched", 2024-03-11T21:48:11.511 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:47:49.443951+0000", 2024-03-11T21:48:11.511 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.8502999999999999e-05 2024-03-11T21:48:11.511 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:48:11.512 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:48:11.512 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "queued_for_pg", 2024-03-11T21:48:11.512 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:47:49.443960+0000", 2024-03-11T21:48:11.512 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 8.6209999999999993e-06 2024-03-11T21:48:11.512 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:48:11.512 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:48:11.512 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-03-11T21:48:11.512 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:47:49.444027+0000", 2024-03-11T21:48:11.512 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 6.7520000000000004e-05 2024-03-11T21:48:11.512 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:48:11.512 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:48:11.512 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "waiting for peered", 2024-03-11T21:48:11.512 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:47:49.444036+0000", 2024-03-11T21:48:11.512 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 8.6929999999999996e-06 2024-03-11T21:48:11.512 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:48:11.513 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:48:11.513 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-03-11T21:48:11.513 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:47:50.444764+0000", 2024-03-11T21:48:11.513 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.0007276890000001 2024-03-11T21:48:11.513 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:48:11.513 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:48:11.513 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "started", 2024-03-11T21:48:11.513 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:47:50.444884+0000", 2024-03-11T21:48:11.513 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.000120565 2024-03-11T21:48:11.513 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:48:11.513 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:48:11.513 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "waiting for subops from 0,7", 2024-03-11T21:48:11.513 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:47:50.444984+0000", 2024-03-11T21:48:11.513 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.000100065 2024-03-11T21:48:11.513 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:48:11.514 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:48:11.514 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "op_commit", 2024-03-11T21:48:11.514 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:47:50.452732+0000", 2024-03-11T21:48:11.514 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.0077480969999999998 2024-03-11T21:48:11.514 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:48:11.514 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:48:11.514 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "sub_op_commit_rec", 2024-03-11T21:48:11.514 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:47:50.453472+0000", 2024-03-11T21:48:11.514 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00073925599999999998 2024-03-11T21:48:11.514 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:48:11.514 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:48:11.514 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "sub_op_commit_rec", 2024-03-11T21:48:11.514 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:47:50.453566+0000", 2024-03-11T21:48:11.514 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 9.3857000000000006e-05 2024-03-11T21:48:11.514 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:48:11.515 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:48:11.515 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "commit_sent", 2024-03-11T21:48:11.515 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:47:50.453802+0000", 2024-03-11T21:48:11.515 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.000236164 2024-03-11T21:48:11.515 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:48:11.515 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:48:11.515 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "done", 2024-03-11T21:48:11.515 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:47:50.455790+0000", 2024-03-11T21:48:11.515 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.0019880539999999999 2024-03-11T21:48:11.515 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-03-11T21:48:11.515 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-03-11T21:48:11.515 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-03-11T21:48:11.515 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:48:11.515 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:48:11.515 INFO:teuthology.orchestra.run.smithi092.stdout: "description": "osd_op(client.35813.0:228843 11.7 11:ed6fc1b7:::benchmark_data_smithi092_186520_object90495:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e491)", 2024-03-11T21:48:11.515 INFO:teuthology.orchestra.run.smithi092.stdout: "initiated_at": "2024-03-11T21:47:49.463347+0000", 2024-03-11T21:48:11.516 INFO:teuthology.orchestra.run.smithi092.stdout: "age": 21.399464566999999, 2024-03-11T21:48:11.516 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.99278157199999995, 2024-03-11T21:48:11.516 INFO:teuthology.orchestra.run.smithi092.stdout: "type_data": { 2024-03-11T21:48:11.516 INFO:teuthology.orchestra.run.smithi092.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-11T21:48:11.516 INFO:teuthology.orchestra.run.smithi092.stdout: "client_info": { 2024-03-11T21:48:11.516 INFO:teuthology.orchestra.run.smithi092.stdout: "client": "client.35813", 2024-03-11T21:48:11.516 INFO:teuthology.orchestra.run.smithi092.stdout: "client_addr": "v1:172.21.15.92:0/4293352030", 2024-03-11T21:48:11.516 INFO:teuthology.orchestra.run.smithi092.stdout: "tid": 228843 2024-03-11T21:48:11.516 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:48:11.516 INFO:teuthology.orchestra.run.smithi092.stdout: "events": [ 2024-03-11T21:48:11.516 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:48:11.516 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "initiated", 2024-03-11T21:48:11.516 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:47:49.463347+0000", 2024-03-11T21:48:11.516 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-03-11T21:48:11.517 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:48:11.517 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:48:11.517 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "header_read", 2024-03-11T21:48:11.517 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:47:49.463347+0000", 2024-03-11T21:48:11.517 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-03-11T21:48:11.517 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:48:11.517 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:48:11.517 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "throttled", 2024-03-11T21:48:11.517 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:47:49.463348+0000", 2024-03-11T21:48:11.517 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.314e-06 2024-03-11T21:48:11.517 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:48:11.517 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:48:11.517 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "all_read", 2024-03-11T21:48:11.517 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:47:49.463352+0000", 2024-03-11T21:48:11.517 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 3.7450000000000001e-06 2024-03-11T21:48:11.518 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:48:11.518 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:48:11.518 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "dispatched", 2024-03-11T21:48:11.518 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:47:49.463357+0000", 2024-03-11T21:48:11.518 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 5.3700000000000003e-06 2024-03-11T21:48:11.518 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:48:11.518 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:48:11.518 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "queued_for_pg", 2024-03-11T21:48:11.518 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:47:49.463364+0000", 2024-03-11T21:48:11.518 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 6.934e-06 2024-03-11T21:48:11.518 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:48:11.518 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:48:11.518 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-03-11T21:48:11.518 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:47:49.463417+0000", 2024-03-11T21:48:11.519 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 5.2969000000000001e-05 2024-03-11T21:48:11.519 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:48:11.519 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:48:11.519 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "waiting for peered", 2024-03-11T21:48:11.519 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:47:49.463426+0000", 2024-03-11T21:48:11.519 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 9.0869999999999996e-06 2024-03-11T21:48:11.519 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:48:11.519 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:48:11.519 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-03-11T21:48:11.519 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:47:50.446505+0000", 2024-03-11T21:48:11.519 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.98307900299999995 2024-03-11T21:48:11.519 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:48:11.519 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:48:11.519 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "started", 2024-03-11T21:48:11.519 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:47:50.446618+0000", 2024-03-11T21:48:11.519 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.000113348 2024-03-11T21:48:11.520 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:48:11.520 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:48:11.520 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "waiting for subops from 0,7", 2024-03-11T21:48:11.520 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:47:50.446733+0000", 2024-03-11T21:48:11.520 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.000114325 2024-03-11T21:48:11.520 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:48:11.520 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:48:11.520 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "op_commit", 2024-03-11T21:48:11.520 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:47:50.452785+0000", 2024-03-11T21:48:11.520 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.0060527840000000003 2024-03-11T21:48:11.520 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:48:11.520 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:48:11.520 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "sub_op_commit_rec", 2024-03-11T21:48:11.520 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:47:50.455842+0000", 2024-03-11T21:48:11.521 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.0030561500000000001 2024-03-11T21:48:11.521 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:48:11.521 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:48:11.521 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "sub_op_commit_rec", 2024-03-11T21:48:11.521 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:47:50.455896+0000", 2024-03-11T21:48:11.521 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 5.4242e-05 2024-03-11T21:48:11.521 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:48:11.521 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:48:11.521 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "commit_sent", 2024-03-11T21:48:11.521 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:47:50.456081+0000", 2024-03-11T21:48:11.521 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.000184852 2024-03-11T21:48:11.521 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:48:11.521 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:48:11.521 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "done", 2024-03-11T21:48:11.521 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:47:50.456128+0000", 2024-03-11T21:48:11.522 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 4.7448999999999999e-05 2024-03-11T21:48:11.522 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-03-11T21:48:11.522 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-03-11T21:48:11.522 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-03-11T21:48:11.522 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:48:11.522 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:48:11.522 INFO:teuthology.orchestra.run.smithi092.stdout: "description": "osd_op(client.35813.0:228845 11.7 11:e8c7442c:::benchmark_data_smithi092_186520_object90497:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e491)", 2024-03-11T21:48:11.522 INFO:teuthology.orchestra.run.smithi092.stdout: "initiated_at": "2024-03-11T21:47:49.464117+0000", 2024-03-11T21:48:11.522 INFO:teuthology.orchestra.run.smithi092.stdout: "age": 21.398693843, 2024-03-11T21:48:11.522 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.99334717800000005, 2024-03-11T21:48:11.522 INFO:teuthology.orchestra.run.smithi092.stdout: "type_data": { 2024-03-11T21:48:11.522 INFO:teuthology.orchestra.run.smithi092.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-11T21:48:11.522 INFO:teuthology.orchestra.run.smithi092.stdout: "client_info": { 2024-03-11T21:48:11.522 INFO:teuthology.orchestra.run.smithi092.stdout: "client": "client.35813", 2024-03-11T21:48:11.523 INFO:teuthology.orchestra.run.smithi092.stdout: "client_addr": "v1:172.21.15.92:0/4293352030", 2024-03-11T21:48:11.523 INFO:teuthology.orchestra.run.smithi092.stdout: "tid": 228845 2024-03-11T21:48:11.523 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:48:11.523 INFO:teuthology.orchestra.run.smithi092.stdout: "events": [ 2024-03-11T21:48:11.523 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:48:11.523 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "initiated", 2024-03-11T21:48:11.523 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:47:49.464117+0000", 2024-03-11T21:48:11.523 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-03-11T21:48:11.523 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:48:11.523 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:48:11.523 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "header_read", 2024-03-11T21:48:11.523 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:47:49.464117+0000", 2024-03-11T21:48:11.523 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-03-11T21:48:11.523 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:48:11.524 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:48:11.524 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "throttled", 2024-03-11T21:48:11.524 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:47:49.464119+0000", 2024-03-11T21:48:11.524 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.525e-06 2024-03-11T21:48:11.524 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:48:11.524 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:48:11.524 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "all_read", 2024-03-11T21:48:11.524 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:47:49.464124+0000", 2024-03-11T21:48:11.524 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 5.5199999999999997e-06 2024-03-11T21:48:11.524 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:48:11.524 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:48:11.524 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "dispatched", 2024-03-11T21:48:11.524 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:47:49.464138+0000", 2024-03-11T21:48:11.524 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.3302000000000001e-05 2024-03-11T21:48:11.524 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:48:11.524 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:48:11.525 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "queued_for_pg", 2024-03-11T21:48:11.525 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:47:49.464146+0000", 2024-03-11T21:48:11.525 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 8.4030000000000007e-06 2024-03-11T21:48:11.525 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:48:11.525 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:48:11.525 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-03-11T21:48:11.525 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:47:49.464198+0000", 2024-03-11T21:48:11.525 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 5.1901999999999999e-05 2024-03-11T21:48:11.525 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:48:11.525 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:48:11.525 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "waiting for peered", 2024-03-11T21:48:11.525 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:47:49.464207+0000", 2024-03-11T21:48:11.525 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 9.0240000000000003e-06 2024-03-11T21:48:11.525 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:48:11.525 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:48:11.526 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-03-11T21:48:11.526 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:47:50.450086+0000", 2024-03-11T21:48:11.526 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.98587937599999997 2024-03-11T21:48:11.526 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:48:11.526 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:48:11.526 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "started", 2024-03-11T21:48:11.526 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:47:50.450205+0000", 2024-03-11T21:48:11.526 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.000118226 2024-03-11T21:48:11.526 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:48:11.526 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:48:11.526 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "waiting for subops from 0,7", 2024-03-11T21:48:11.526 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:47:50.450358+0000", 2024-03-11T21:48:11.526 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00015342700000000001 2024-03-11T21:48:11.526 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:48:11.527 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:48:11.527 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "sub_op_commit_rec", 2024-03-11T21:48:11.527 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:47:50.456180+0000", 2024-03-11T21:48:11.527 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.0058215350000000001 2024-03-11T21:48:11.527 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:48:11.527 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:48:11.527 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "sub_op_commit_rec", 2024-03-11T21:48:11.527 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:47:50.457359+0000", 2024-03-11T21:48:11.527 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.0011791530000000001 2024-03-11T21:48:11.527 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:48:11.527 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:48:11.527 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "op_commit", 2024-03-11T21:48:11.527 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:47:50.457373+0000", 2024-03-11T21:48:11.527 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.4063000000000001e-05 2024-03-11T21:48:11.528 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:48:11.528 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:48:11.528 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "commit_sent", 2024-03-11T21:48:11.528 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:47:50.457422+0000", 2024-03-11T21:48:11.528 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 4.9577999999999999e-05 2024-03-11T21:48:11.528 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:48:11.528 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:48:11.528 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "done", 2024-03-11T21:48:11.528 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:47:50.457464+0000", 2024-03-11T21:48:11.528 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 4.2144000000000002e-05 2024-03-11T21:48:11.528 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-03-11T21:48:11.528 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-03-11T21:48:11.528 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-03-11T21:48:11.528 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:48:11.528 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:48:11.529 INFO:teuthology.orchestra.run.smithi092.stdout: "description": "osd_op(client.35813.0:228853 11.7 11:e3123654:::benchmark_data_smithi092_186520_object90505:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e491)", 2024-03-11T21:48:11.529 INFO:teuthology.orchestra.run.smithi092.stdout: "initiated_at": "2024-03-11T21:47:49.474849+0000", 2024-03-11T21:48:11.529 INFO:teuthology.orchestra.run.smithi092.stdout: "age": 21.387961695000001, 2024-03-11T21:48:11.529 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.99407246100000002, 2024-03-11T21:48:11.529 INFO:teuthology.orchestra.run.smithi092.stdout: "type_data": { 2024-03-11T21:48:11.529 INFO:teuthology.orchestra.run.smithi092.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-11T21:48:11.529 INFO:teuthology.orchestra.run.smithi092.stdout: "client_info": { 2024-03-11T21:48:11.529 INFO:teuthology.orchestra.run.smithi092.stdout: "client": "client.35813", 2024-03-11T21:48:11.529 INFO:teuthology.orchestra.run.smithi092.stdout: "client_addr": "v1:172.21.15.92:0/4293352030", 2024-03-11T21:48:11.529 INFO:teuthology.orchestra.run.smithi092.stdout: "tid": 228853 2024-03-11T21:48:11.529 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:48:11.529 INFO:teuthology.orchestra.run.smithi092.stdout: "events": [ 2024-03-11T21:48:11.529 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:48:11.530 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "initiated", 2024-03-11T21:48:11.530 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:47:49.474849+0000", 2024-03-11T21:48:11.530 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-03-11T21:48:11.530 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:48:11.530 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:48:11.530 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "header_read", 2024-03-11T21:48:11.530 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:47:49.474849+0000", 2024-03-11T21:48:11.530 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-03-11T21:48:11.530 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:48:11.530 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:48:11.530 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "throttled", 2024-03-11T21:48:11.530 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:47:49.474851+0000", 2024-03-11T21:48:11.530 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.9649999999999998e-06 2024-03-11T21:48:11.530 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:48:11.531 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:48:11.531 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "all_read", 2024-03-11T21:48:11.531 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:47:49.474857+0000", 2024-03-11T21:48:11.531 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 5.9649999999999996e-06 2024-03-11T21:48:11.531 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:48:11.531 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:48:11.531 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "dispatched", 2024-03-11T21:48:11.531 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:47:49.474871+0000", 2024-03-11T21:48:11.531 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.3249e-05 2024-03-11T21:48:11.531 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:48:11.531 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:48:11.531 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "queued_for_pg", 2024-03-11T21:48:11.531 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:47:49.474879+0000", 2024-03-11T21:48:11.531 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 8.2320000000000001e-06 2024-03-11T21:48:11.531 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:48:11.532 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:48:11.532 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-03-11T21:48:11.532 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:47:49.474945+0000", 2024-03-11T21:48:11.532 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 6.6594999999999995e-05 2024-03-11T21:48:11.532 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:48:11.532 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:48:11.532 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "waiting for peered", 2024-03-11T21:48:11.532 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:47:49.474952+0000", 2024-03-11T21:48:11.532 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 6.5010000000000001e-06 2024-03-11T21:48:11.532 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:48:11.532 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:48:11.532 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-03-11T21:48:11.532 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:47:50.458385+0000", 2024-03-11T21:48:11.532 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.98343275399999996 2024-03-11T21:48:11.532 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:48:11.532 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:48:11.533 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "started", 2024-03-11T21:48:11.533 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:47:50.458498+0000", 2024-03-11T21:48:11.533 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.000113633 2024-03-11T21:48:11.533 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:48:11.533 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:48:11.533 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "waiting for subops from 0,7", 2024-03-11T21:48:11.533 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:47:50.458602+0000", 2024-03-11T21:48:11.533 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.000103211 2024-03-11T21:48:11.533 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:48:11.533 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:48:11.533 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "sub_op_commit_rec", 2024-03-11T21:48:11.533 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:47:50.466595+0000", 2024-03-11T21:48:11.533 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.007993343 2024-03-11T21:48:11.533 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:48:11.534 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:48:11.534 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "sub_op_commit_rec", 2024-03-11T21:48:11.534 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:47:50.466639+0000", 2024-03-11T21:48:11.534 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 4.4403999999999997e-05 2024-03-11T21:48:11.534 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:48:11.534 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:48:11.534 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "op_commit", 2024-03-11T21:48:11.534 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:47:50.468685+0000", 2024-03-11T21:48:11.534 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.002045874 2024-03-11T21:48:11.534 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:48:11.534 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:48:11.534 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "commit_sent", 2024-03-11T21:48:11.534 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:47:50.468873+0000", 2024-03-11T21:48:11.534 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00018790500000000001 2024-03-11T21:48:11.535 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:48:11.535 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:48:11.535 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "done", 2024-03-11T21:48:11.535 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:47:50.468922+0000", 2024-03-11T21:48:11.535 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 4.8829999999999998e-05 2024-03-11T21:48:11.535 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-03-11T21:48:11.535 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-03-11T21:48:11.535 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-03-11T21:48:11.535 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-03-11T21:48:11.535 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-03-11T21:48:11.535 INFO:teuthology.orchestra.run.smithi092.stdout:} 2024-03-11T21:48:11.536 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:48:11.536 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:48:11.537 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:48:11.637 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:48:11.677 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:48:11.678 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:48:11.678 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:48:11.721 DEBUG:teuthology.orchestra.run.smithi092:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid 00db0016-dfed-11ee-95c8-87774f69a715 -- ceph osd unset nodeep-scrub 2024-03-11T21:48:11.779 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:48:11.812 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:48:11.813 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:48:11.813 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:48:11.914 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:48:11.958 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:48:11.959 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:48:11.959 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:48:12.060 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:48:12.085 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:48:11 smithi097 ceph-mon[33317]: 6.17 scrub starts 2024-03-11T21:48:12.085 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:48:11 smithi097 ceph-mon[33317]: 6.17 scrub ok 2024-03-11T21:48:12.086 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:48:11 smithi097 ceph-mon[33317]: 4.5 scrub starts 2024-03-11T21:48:12.086 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:48:11 smithi097 ceph-mon[33317]: 4.5 scrub ok 2024-03-11T21:48:12.086 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:48:11 smithi097 ceph-mon[33317]: 5.14 scrub starts 2024-03-11T21:48:12.086 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:48:11 smithi097 ceph-mon[33317]: 5.14 scrub ok 2024-03-11T21:48:12.110 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:48:12.111 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:48:12.111 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:48:12.119 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:48:11 smithi092 ceph-mon[29572]: 6.17 scrub starts 2024-03-11T21:48:12.119 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:48:11 smithi092 ceph-mon[29572]: 6.17 scrub ok 2024-03-11T21:48:12.119 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:48:11 smithi092 ceph-mon[29572]: 4.5 scrub starts 2024-03-11T21:48:12.120 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:48:11 smithi092 ceph-mon[29572]: 4.5 scrub ok 2024-03-11T21:48:12.120 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:48:11 smithi092 ceph-mon[29572]: 5.14 scrub starts 2024-03-11T21:48:12.120 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:48:11 smithi092 ceph-mon[29572]: 5.14 scrub ok 2024-03-11T21:48:12.120 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:48:11 smithi092 ceph-mon[37563]: 6.17 scrub starts 2024-03-11T21:48:12.120 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:48:11 smithi092 ceph-mon[37563]: 6.17 scrub ok 2024-03-11T21:48:12.120 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:48:11 smithi092 ceph-mon[37563]: 4.5 scrub starts 2024-03-11T21:48:12.120 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:48:11 smithi092 ceph-mon[37563]: 4.5 scrub ok 2024-03-11T21:48:12.120 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:48:11 smithi092 ceph-mon[37563]: 5.14 scrub starts 2024-03-11T21:48:12.120 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:48:11 smithi092 ceph-mon[37563]: 5.14 scrub ok 2024-03-11T21:48:12.212 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:48:12.242 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:48:12.243 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:48:12.243 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:48:12.344 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:48:12.381 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:48:12.381 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:48:12.382 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:48:12.482 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:48:12.518 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:48:12.519 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:48:12.519 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:48:12.620 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:48:12.647 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:48:12.648 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:48:12.648 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:48:12.749 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:48:12.791 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:48:12.792 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:48:12.792 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:48:12.893 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:48:12.924 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:48:12.924 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:48:12.925 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:48:13.025 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:48:13.063 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:48:13.064 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:48:13.064 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:48:13.085 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:48:12 smithi097 ceph-mon[33317]: pgmap v1069: 148 pgs: 1 active+clean+remapped, 147 active+clean; 1.4 GiB data, 8.6 GiB used, 438 GiB / 447 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.24k op/s; 5.9 MiB/s, 94 objects/s recovering 2024-03-11T21:48:13.085 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:48:12 smithi097 ceph-mon[33317]: 11.2 scrub starts 2024-03-11T21:48:13.085 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:48:12 smithi097 ceph-mon[33317]: 6.d scrub starts 2024-03-11T21:48:13.085 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:48:12 smithi097 ceph-mon[33317]: 11.e scrub starts 2024-03-11T21:48:13.085 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:48:12 smithi097 ceph-mon[33317]: 6.d scrub ok 2024-03-11T21:48:13.085 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:48:12 smithi097 ceph-mon[33317]: 11.2 scrub ok 2024-03-11T21:48:13.085 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:48:12 smithi097 ceph-mon[33317]: 11.e scrub ok 2024-03-11T21:48:13.085 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:48:12 smithi097 ceph-mon[33317]: 2.19 scrub starts 2024-03-11T21:48:13.085 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:48:12 smithi097 ceph-mon[33317]: 2.19 scrub ok 2024-03-11T21:48:13.085 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:48:12 smithi097 ceph-mon[33317]: 2.e scrub starts 2024-03-11T21:48:13.085 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:48:12 smithi097 ceph-mon[33317]: 2.e scrub ok 2024-03-11T21:48:13.119 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:48:12 smithi092 ceph-mon[29572]: pgmap v1069: 148 pgs: 1 active+clean+remapped, 147 active+clean; 1.4 GiB data, 8.6 GiB used, 438 GiB / 447 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.24k op/s; 5.9 MiB/s, 94 objects/s recovering 2024-03-11T21:48:13.120 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:48:12 smithi092 ceph-mon[29572]: 11.2 scrub starts 2024-03-11T21:48:13.120 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:48:12 smithi092 ceph-mon[29572]: 6.d scrub starts 2024-03-11T21:48:13.120 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:48:12 smithi092 ceph-mon[29572]: 11.e scrub starts 2024-03-11T21:48:13.120 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:48:12 smithi092 ceph-mon[29572]: 6.d scrub ok 2024-03-11T21:48:13.120 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:48:12 smithi092 ceph-mon[29572]: 11.2 scrub ok 2024-03-11T21:48:13.120 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:48:12 smithi092 ceph-mon[29572]: 11.e scrub ok 2024-03-11T21:48:13.120 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:48:12 smithi092 ceph-mon[29572]: 2.19 scrub starts 2024-03-11T21:48:13.120 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:48:12 smithi092 ceph-mon[29572]: 2.19 scrub ok 2024-03-11T21:48:13.120 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:48:12 smithi092 ceph-mon[29572]: 2.e scrub starts 2024-03-11T21:48:13.120 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:48:12 smithi092 ceph-mon[29572]: 2.e scrub ok 2024-03-11T21:48:13.121 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:48:12 smithi092 ceph-mon[37563]: pgmap v1069: 148 pgs: 1 active+clean+remapped, 147 active+clean; 1.4 GiB data, 8.6 GiB used, 438 GiB / 447 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.24k op/s; 5.9 MiB/s, 94 objects/s recovering 2024-03-11T21:48:13.121 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:48:12 smithi092 ceph-mon[37563]: 11.2 scrub starts 2024-03-11T21:48:13.121 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:48:12 smithi092 ceph-mon[37563]: 6.d scrub starts 2024-03-11T21:48:13.121 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:48:12 smithi092 ceph-mon[37563]: 11.e scrub starts 2024-03-11T21:48:13.121 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:48:12 smithi092 ceph-mon[37563]: 6.d scrub ok 2024-03-11T21:48:13.121 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:48:12 smithi092 ceph-mon[37563]: 11.2 scrub ok 2024-03-11T21:48:13.121 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:48:12 smithi092 ceph-mon[37563]: 11.e scrub ok 2024-03-11T21:48:13.121 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:48:12 smithi092 ceph-mon[37563]: 2.19 scrub starts 2024-03-11T21:48:13.121 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:48:12 smithi092 ceph-mon[37563]: 2.19 scrub ok 2024-03-11T21:48:13.121 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:48:12 smithi092 ceph-mon[37563]: 2.e scrub starts 2024-03-11T21:48:13.121 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:48:12 smithi092 ceph-mon[37563]: 2.e scrub ok 2024-03-11T21:48:13.166 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:48:13.209 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:48:13.210 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:48:13.210 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:48:13.311 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:48:13.341 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:48:13.342 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:48:13.342 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:48:13.443 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:48:13.487 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:48:13.487 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:48:13.488 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:48:13.551 INFO:teuthology.orchestra.run.smithi092.stderr:Inferring config /var/lib/ceph/00db0016-dfed-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T21:48:13.585 INFO:journalctl@ceph.iscsi.iscsi.a.smithi097.stdout:Mar 11 21:48:13 smithi097 ceph-00db0016-dfed-11ee-95c8-87774f69a715-iscsi-iscsi-a[60943]: debug there is no tcmu-runner data available 2024-03-11T21:48:13.589 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:48:13.620 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:48:13.621 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:48:13.621 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:48:13.722 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:48:13.752 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:48:13.753 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:48:13.753 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:48:13.854 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:48:13.869 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:48:13 smithi092 ceph-mon[29572]: 11.7 scrub starts 2024-03-11T21:48:13.869 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:48:13 smithi092 ceph-mon[29572]: from='client.24469 v1:172.21.15.97:0/404208310' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:48:13.869 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:48:13 smithi092 ceph-mon[29572]: 11.7 scrub ok 2024-03-11T21:48:13.869 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:48:13 smithi092 ceph-mon[37563]: 11.7 scrub starts 2024-03-11T21:48:13.869 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:48:13 smithi092 ceph-mon[37563]: from='client.24469 v1:172.21.15.97:0/404208310' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:48:13.869 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:48:13 smithi092 ceph-mon[37563]: 11.7 scrub ok 2024-03-11T21:48:13.883 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:48:13.884 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:48:13.884 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:48:13.985 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:48:14.010 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:48:14.010 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:48:14.011 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:48:14.085 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:48:13 smithi097 ceph-mon[33317]: 11.7 scrub starts 2024-03-11T21:48:14.085 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:48:13 smithi097 ceph-mon[33317]: from='client.24469 v1:172.21.15.97:0/404208310' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:48:14.085 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:48:13 smithi097 ceph-mon[33317]: 11.7 scrub ok 2024-03-11T21:48:14.111 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:48:14.142 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:48:14.142 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:48:14.143 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:48:14.244 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:48:14.282 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:48:14.283 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:48:14.283 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:48:14.384 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:48:14.413 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:48:14.414 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:48:14.415 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:48:14.515 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:48:14.545 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:48:14.546 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:48:14.546 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:48:14.647 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:48:14.677 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:48:14.678 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:48:14.678 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:48:14.779 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:48:14.813 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:48:14.813 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:48:14.814 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:48:14.914 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:48:14.943 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:48:14.944 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:48:14.944 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:48:15.045 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:48:15.076 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:48:15.077 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:48:15.077 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:48:15.084 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:48:14 smithi097 ceph-mon[33317]: pgmap v1070: 148 pgs: 148 active+clean; 1.2 GiB data, 8.1 GiB used, 439 GiB / 447 GiB avail; 639 B/s rd, 0 B/s wr, 1.26k op/s; 5.9 MiB/s, 94 objects/s recovering 2024-03-11T21:48:15.119 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:48:14 smithi092 ceph-mon[29572]: pgmap v1070: 148 pgs: 148 active+clean; 1.2 GiB data, 8.1 GiB used, 439 GiB / 447 GiB avail; 639 B/s rd, 0 B/s wr, 1.26k op/s; 5.9 MiB/s, 94 objects/s recovering 2024-03-11T21:48:15.119 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:48:14 smithi092 ceph-mon[37563]: pgmap v1070: 148 pgs: 148 active+clean; 1.2 GiB data, 8.1 GiB used, 439 GiB / 447 GiB avail; 639 B/s rd, 0 B/s wr, 1.26k op/s; 5.9 MiB/s, 94 objects/s recovering 2024-03-11T21:48:15.178 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:48:15.219 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:48:15.220 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:48:15.220 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:48:15.321 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:48:15.354 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:48:15.355 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:48:15.355 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:48:15.456 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:48:15.486 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:48:15.487 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:48:15.487 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:48:15.588 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:48:15.616 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:48:15.617 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:48:15.617 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:48:15.718 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:48:15.747 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:48:15.747 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:48:15.747 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:48:15.848 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:48:15.879 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:48:15.880 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:48:15.880 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:48:15.981 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:48:16.011 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:48:16.012 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:48:16.012 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:48:16.085 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:48:15 smithi097 ceph-mon[33317]: 5.e scrub starts 2024-03-11T21:48:16.085 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:48:15 smithi097 ceph-mon[33317]: 5.e scrub ok 2024-03-11T21:48:16.085 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:48:15 smithi097 ceph-mon[33317]: 5.1f scrub starts 2024-03-11T21:48:16.085 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:48:15 smithi097 ceph-mon[33317]: 5.1f scrub ok 2024-03-11T21:48:16.085 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:48:15 smithi097 ceph-mon[33317]: 1.0 scrub starts 2024-03-11T21:48:16.085 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:48:15 smithi097 ceph-mon[33317]: 1.0 scrub ok 2024-03-11T21:48:16.113 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:48:16.119 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:48:15 smithi092 ceph-mon[29572]: 5.e scrub starts 2024-03-11T21:48:16.119 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:48:15 smithi092 ceph-mon[29572]: 5.e scrub ok 2024-03-11T21:48:16.119 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:48:15 smithi092 ceph-mon[29572]: 5.1f scrub starts 2024-03-11T21:48:16.119 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:48:15 smithi092 ceph-mon[29572]: 5.1f scrub ok 2024-03-11T21:48:16.119 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:48:15 smithi092 ceph-mon[29572]: 1.0 scrub starts 2024-03-11T21:48:16.119 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:48:15 smithi092 ceph-mon[29572]: 1.0 scrub ok 2024-03-11T21:48:16.119 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:48:15 smithi092 ceph-mon[37563]: 5.e scrub starts 2024-03-11T21:48:16.119 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:48:15 smithi092 ceph-mon[37563]: 5.e scrub ok 2024-03-11T21:48:16.119 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:48:15 smithi092 ceph-mon[37563]: 5.1f scrub starts 2024-03-11T21:48:16.119 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:48:15 smithi092 ceph-mon[37563]: 5.1f scrub ok 2024-03-11T21:48:16.119 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:48:15 smithi092 ceph-mon[37563]: 1.0 scrub starts 2024-03-11T21:48:16.120 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:48:15 smithi092 ceph-mon[37563]: 1.0 scrub ok 2024-03-11T21:48:16.146 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:48:16.147 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:48:16.147 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:48:16.248 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:48:16.276 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:48:16.277 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:48:16.277 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:48:16.378 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:48:16.411 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:48:16.412 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:48:16.412 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:48:16.513 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:48:16.551 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:48:16.551 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:48:16.551 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:48:16.652 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:48:16.684 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:48:16.685 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:48:16.686 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:48:16.786 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:48:16.824 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:48:16.825 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:48:16.825 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:48:16.926 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:48:16.967 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:48:16.968 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:48:16.969 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:48:17.069 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:48:17.085 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:48:16 smithi097 ceph-mon[33317]: pgmap v1071: 148 pgs: 148 active+clean; 1018 MiB data, 7.4 GiB used, 440 GiB / 447 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.64k op/s; 5.9 MiB/s, 94 objects/s recovering 2024-03-11T21:48:17.085 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:48:16 smithi097 ceph-mon[33317]: 6.9 scrub starts 2024-03-11T21:48:17.085 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:48:16 smithi097 ceph-mon[33317]: 6.9 scrub ok 2024-03-11T21:48:17.100 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:48:17.100 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:48:17.101 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:48:17.119 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:48:16 smithi092 ceph-mon[37563]: pgmap v1071: 148 pgs: 148 active+clean; 1018 MiB data, 7.4 GiB used, 440 GiB / 447 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.64k op/s; 5.9 MiB/s, 94 objects/s recovering 2024-03-11T21:48:17.119 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:48:16 smithi092 ceph-mon[37563]: 6.9 scrub starts 2024-03-11T21:48:17.119 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:48:16 smithi092 ceph-mon[37563]: 6.9 scrub ok 2024-03-11T21:48:17.120 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:48:16 smithi092 ceph-mon[29572]: pgmap v1071: 148 pgs: 148 active+clean; 1018 MiB data, 7.4 GiB used, 440 GiB / 447 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.64k op/s; 5.9 MiB/s, 94 objects/s recovering 2024-03-11T21:48:17.120 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:48:16 smithi092 ceph-mon[29572]: 6.9 scrub starts 2024-03-11T21:48:17.120 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:48:16 smithi092 ceph-mon[29572]: 6.9 scrub ok 2024-03-11T21:48:17.201 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:48:17.225 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:48:17.226 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:48:17.226 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:48:17.327 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:48:17.358 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:48:17.359 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:48:17.359 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:48:17.460 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:48:17.499 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:48:17.500 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:48:17.500 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:48:17.601 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:48:17.640 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:48:17.641 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:48:17.641 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:48:17.742 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:48:17.772 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:48:17.773 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:48:17.773 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:48:17.874 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:48:17.918 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:48:17.918 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:48:17.919 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:48:18.021 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:48:18.036 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:48:17 smithi092 ceph-mon[29572]: 4.17 scrub starts 2024-03-11T21:48:18.036 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:48:17 smithi092 ceph-mon[29572]: 4.17 scrub ok 2024-03-11T21:48:18.036 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:48:17 smithi092 ceph-mon[37563]: 4.17 scrub starts 2024-03-11T21:48:18.036 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:48:17 smithi092 ceph-mon[37563]: 4.17 scrub ok 2024-03-11T21:48:18.037 INFO:journalctl@ceph.mgr.y.smithi092.stdout:Mar 11 21:48:17 smithi092 ceph-00db0016-dfed-11ee-95c8-87774f69a715-mgr-y[29801]: ::ffff:172.21.15.97 - - [11/Mar/2024:21:48:17] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-11T21:48:18.051 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:48:18.052 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:48:18.053 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:48:18.085 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:48:17 smithi097 ceph-mon[33317]: 4.17 scrub starts 2024-03-11T21:48:18.085 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:48:17 smithi097 ceph-mon[33317]: 4.17 scrub ok 2024-03-11T21:48:18.154 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:48:18.192 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:48:18.193 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:48:18.193 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:48:18.294 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:48:18.335 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:48:18.336 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:48:18.336 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:48:18.437 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:48:18.475 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:48:18.475 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:48:18.476 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:48:18.577 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:48:18.606 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:48:18.607 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:48:18.607 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:48:18.708 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:48:18.738 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:48:18.738 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:48:18.739 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:48:18.839 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:48:18.879 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:48:18.879 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:48:18.879 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:48:18.980 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:48:19.021 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:48:19.021 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:48:19.022 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:48:19.085 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:48:18 smithi097 ceph-mon[33317]: pgmap v1072: 148 pgs: 148 active+clean; 1018 MiB data, 7.4 GiB used, 440 GiB / 447 GiB avail; 1.0 KiB/s rd, 0 B/s wr, 1.17k op/s; 4.8 MiB/s, 77 objects/s recovering 2024-03-11T21:48:19.085 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:48:18 smithi097 ceph-mon[33317]: 11.d scrub starts 2024-03-11T21:48:19.119 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:48:18 smithi092 ceph-mon[29572]: pgmap v1072: 148 pgs: 148 active+clean; 1018 MiB data, 7.4 GiB used, 440 GiB / 447 GiB avail; 1.0 KiB/s rd, 0 B/s wr, 1.17k op/s; 4.8 MiB/s, 77 objects/s recovering 2024-03-11T21:48:19.119 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:48:18 smithi092 ceph-mon[29572]: 11.d scrub starts 2024-03-11T21:48:19.119 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:48:18 smithi092 ceph-mon[37563]: pgmap v1072: 148 pgs: 148 active+clean; 1018 MiB data, 7.4 GiB used, 440 GiB / 447 GiB avail; 1.0 KiB/s rd, 0 B/s wr, 1.17k op/s; 4.8 MiB/s, 77 objects/s recovering 2024-03-11T21:48:19.119 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:48:18 smithi092 ceph-mon[37563]: 11.d scrub starts 2024-03-11T21:48:19.123 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:48:19.155 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:48:19.155 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:48:19.155 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:48:19.256 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:48:19.284 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:48:19.285 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:48:19.285 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:48:19.386 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:48:19.416 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:48:19.417 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:48:19.417 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:48:19.518 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:48:19.545 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:48:19.546 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:48:19.546 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:48:19.647 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:48:19.672 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:48:19.673 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:48:19.673 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:48:19.774 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:48:19.804 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:48:19.804 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:48:19.805 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:48:19.906 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:48:19.937 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:48:19.937 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:48:19.938 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:48:20.039 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:48:20.078 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:48:20.079 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:48:20.079 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:48:20.084 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:48:19 smithi097 ceph-mon[33317]: 11.d scrub ok 2024-03-11T21:48:20.086 DEBUG:teuthology.orchestra.run.smithi092:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid 00db0016-dfed-11ee-95c8-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-03-11T21:48:20.119 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:48:19 smithi092 ceph-mon[29572]: 11.d scrub ok 2024-03-11T21:48:20.119 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:48:19 smithi092 ceph-mon[37563]: 11.d scrub ok 2024-03-11T21:48:20.180 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:48:20.208 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:48:20.209 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:48:20.209 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:48:20.214 DEBUG:teuthology.orchestra.run.smithi092:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid 00db0016-dfed-11ee-95c8-87774f69a715 -- ceph daemon osd.2 dump_ops_in_flight 2024-03-11T21:48:20.310 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:48:20.337 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:48:20.338 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:48:20.338 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:48:20.439 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:48:20.465 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:48:20.466 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:48:20.466 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:48:20.567 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:48:20.622 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:48:20.622 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:48:20.623 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:48:20.724 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:48:20.757 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:48:20.758 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:48:20.758 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:48:20.859 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:48:20.890 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:48:20.890 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:48:20.891 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:48:20.958 INFO:teuthology.orchestra.run.smithi092.stderr:Inferring config /var/lib/ceph/00db0016-dfed-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T21:48:20.991 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:48:21.039 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:48:21.040 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:48:21.041 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:48:21.085 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:48:20 smithi097 ceph-mon[33317]: pgmap v1073: 148 pgs: 148 active+clean; 611 MiB data, 6.3 GiB used, 441 GiB / 447 GiB avail; 948 B/s rd, 0 B/s wr, 1.38k op/s; 0 B/s, 0 objects/s recovering 2024-03-11T21:48:21.085 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:48:20 smithi097 ceph-mon[33317]: 2.6 scrub starts 2024-03-11T21:48:21.085 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:48:20 smithi097 ceph-mon[33317]: 2.6 scrub ok 2024-03-11T21:48:21.085 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:48:20 smithi097 ceph-mon[33317]: 6.c scrub starts 2024-03-11T21:48:21.085 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:48:20 smithi097 ceph-mon[33317]: 6.c scrub ok 2024-03-11T21:48:21.085 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:48:20 smithi097 ceph-mon[33317]: 5.12 scrub starts 2024-03-11T21:48:21.120 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:48:20 smithi092 ceph-mon[29572]: pgmap v1073: 148 pgs: 148 active+clean; 611 MiB data, 6.3 GiB used, 441 GiB / 447 GiB avail; 948 B/s rd, 0 B/s wr, 1.38k op/s; 0 B/s, 0 objects/s recovering 2024-03-11T21:48:21.120 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:48:20 smithi092 ceph-mon[29572]: 2.6 scrub starts 2024-03-11T21:48:21.120 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:48:20 smithi092 ceph-mon[29572]: 2.6 scrub ok 2024-03-11T21:48:21.120 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:48:20 smithi092 ceph-mon[29572]: 6.c scrub starts 2024-03-11T21:48:21.120 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:48:20 smithi092 ceph-mon[29572]: 6.c scrub ok 2024-03-11T21:48:21.120 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:48:20 smithi092 ceph-mon[29572]: 5.12 scrub starts 2024-03-11T21:48:21.121 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:48:20 smithi092 ceph-mon[37563]: pgmap v1073: 148 pgs: 148 active+clean; 611 MiB data, 6.3 GiB used, 441 GiB / 447 GiB avail; 948 B/s rd, 0 B/s wr, 1.38k op/s; 0 B/s, 0 objects/s recovering 2024-03-11T21:48:21.121 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:48:20 smithi092 ceph-mon[37563]: 2.6 scrub starts 2024-03-11T21:48:21.121 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:48:20 smithi092 ceph-mon[37563]: 2.6 scrub ok 2024-03-11T21:48:21.121 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:48:20 smithi092 ceph-mon[37563]: 6.c scrub starts 2024-03-11T21:48:21.121 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:48:20 smithi092 ceph-mon[37563]: 6.c scrub ok 2024-03-11T21:48:21.121 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:48:20 smithi092 ceph-mon[37563]: 5.12 scrub starts 2024-03-11T21:48:21.141 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:48:21.172 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:48:21.173 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:48:21.173 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:48:21.274 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:48:21.317 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:48:21.318 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:48:21.318 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:48:21.419 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:48:21.440 INFO:teuthology.orchestra.run.smithi092.stderr:Inferring config /var/lib/ceph/00db0016-dfed-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T21:48:21.445 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:48:21.446 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:48:21.446 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:48:21.547 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:48:21.597 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:48:21.598 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:48:21.598 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:48:21.701 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:48:21.731 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:48:21.731 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:48:21.732 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:48:21.832 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:48:21.835 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:48:21 smithi092 ceph-mon[29572]: 5.12 scrub ok 2024-03-11T21:48:21.835 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:48:21 smithi092 ceph-mon[29572]: 6.b scrub starts 2024-03-11T21:48:21.835 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:48:21 smithi092 ceph-mon[29572]: 6.b scrub ok 2024-03-11T21:48:21.836 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:48:21 smithi092 ceph-mon[37563]: 5.12 scrub ok 2024-03-11T21:48:21.836 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:48:21 smithi092 ceph-mon[37563]: 6.b scrub starts 2024-03-11T21:48:21.836 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:48:21 smithi092 ceph-mon[37563]: 6.b scrub ok 2024-03-11T21:48:21.872 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:48:21.873 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:48:21.873 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:48:21.974 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:48:22.005 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:48:22.006 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:48:22.006 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:48:22.085 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:48:21 smithi097 ceph-mon[33317]: 5.12 scrub ok 2024-03-11T21:48:22.085 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:48:21 smithi097 ceph-mon[33317]: 6.b scrub starts 2024-03-11T21:48:22.085 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:48:21 smithi097 ceph-mon[33317]: 6.b scrub ok 2024-03-11T21:48:22.107 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:48:22.168 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:48:22.168 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:48:22.169 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:48:22.269 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:48:22.306 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:48:22.307 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:48:22.307 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:48:22.408 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:48:22.445 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:48:22.445 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:48:22.446 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:48:22.546 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:48:22.588 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:48:22.588 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:48:22.589 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:48:22.689 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:48:22.730 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:48:22.730 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:48:22.731 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:48:22.834 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:48:22.869 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:48:22 smithi092 ceph-mon[29572]: pgmap v1074: 148 pgs: 148 active+clean; 553 MiB data, 6.1 GiB used, 441 GiB / 447 GiB avail; 853 B/s rd, 0 B/s wr, 1.22k op/s; 0 B/s, 0 objects/s recovering 2024-03-11T21:48:22.869 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:48:22 smithi092 ceph-mon[29572]: 2.c scrub starts 2024-03-11T21:48:22.869 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:48:22 smithi092 ceph-mon[29572]: 2.c scrub ok 2024-03-11T21:48:22.869 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:48:22 smithi092 ceph-mon[29572]: 2.1e scrub starts 2024-03-11T21:48:22.870 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:48:22 smithi092 ceph-mon[29572]: 2.1e scrub ok 2024-03-11T21:48:22.870 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:48:22 smithi092 ceph-mon[29572]: 5.1a scrub starts 2024-03-11T21:48:22.870 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:48:22 smithi092 ceph-mon[29572]: 5.1a scrub ok 2024-03-11T21:48:22.870 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:48:22 smithi092 ceph-mon[37563]: pgmap v1074: 148 pgs: 148 active+clean; 553 MiB data, 6.1 GiB used, 441 GiB / 447 GiB avail; 853 B/s rd, 0 B/s wr, 1.22k op/s; 0 B/s, 0 objects/s recovering 2024-03-11T21:48:22.870 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:48:22 smithi092 ceph-mon[37563]: 2.c scrub starts 2024-03-11T21:48:22.870 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:48:22 smithi092 ceph-mon[37563]: 2.c scrub ok 2024-03-11T21:48:22.870 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:48:22 smithi092 ceph-mon[37563]: 2.1e scrub starts 2024-03-11T21:48:22.870 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:48:22 smithi092 ceph-mon[37563]: 2.1e scrub ok 2024-03-11T21:48:22.870 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:48:22 smithi092 ceph-mon[37563]: 5.1a scrub starts 2024-03-11T21:48:22.870 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:48:22 smithi092 ceph-mon[37563]: 5.1a scrub ok 2024-03-11T21:48:22.877 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:48:22.877 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:48:22.878 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:48:22.900 INFO:teuthology.orchestra.run.smithi092.stderr:Inferring config /var/lib/ceph/00db0016-dfed-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T21:48:22.979 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:48:23.010 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:48:23.010 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:48:23.011 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:48:23.085 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:48:22 smithi097 ceph-mon[33317]: pgmap v1074: 148 pgs: 148 active+clean; 553 MiB data, 6.1 GiB used, 441 GiB / 447 GiB avail; 853 B/s rd, 0 B/s wr, 1.22k op/s; 0 B/s, 0 objects/s recovering 2024-03-11T21:48:23.085 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:48:22 smithi097 ceph-mon[33317]: 2.c scrub starts 2024-03-11T21:48:23.085 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:48:22 smithi097 ceph-mon[33317]: 2.c scrub ok 2024-03-11T21:48:23.085 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:48:22 smithi097 ceph-mon[33317]: 2.1e scrub starts 2024-03-11T21:48:23.085 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:48:22 smithi097 ceph-mon[33317]: 2.1e scrub ok 2024-03-11T21:48:23.085 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:48:22 smithi097 ceph-mon[33317]: 5.1a scrub starts 2024-03-11T21:48:23.085 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:48:22 smithi097 ceph-mon[33317]: 5.1a scrub ok 2024-03-11T21:48:23.112 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:48:23.143 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:48:23.143 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:48:23.143 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:48:23.244 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:48:23.273 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:48:23.274 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:48:23.274 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:48:23.375 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:48:23.406 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:48:23.407 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:48:23.407 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:48:23.508 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:48:23.534 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:48:23.535 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:48:23.536 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:48:23.585 INFO:journalctl@ceph.iscsi.iscsi.a.smithi097.stdout:Mar 11 21:48:23 smithi097 ceph-00db0016-dfed-11ee-95c8-87774f69a715-iscsi-iscsi-a[60943]: debug there is no tcmu-runner data available 2024-03-11T21:48:23.636 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:48:23.687 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:48:23.687 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:48:23.688 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:48:23.789 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:48:23.830 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:48:23.830 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:48:23.830 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:48:23.931 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:48:23.962 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:48:23.963 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:48:23.963 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:48:24.064 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:48:24.086 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:48:23 smithi097 ceph-mon[33317]: from='mgr.14607 v1:172.21.15.92:0/1702717597' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.8", "id": [1, 0]}]: dispatch 2024-03-11T21:48:24.086 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:48:23 smithi097 ceph-mon[33317]: from='mgr.14607 v1:172.21.15.92:0/1702717597' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.b", "id": [1, 2]}]: dispatch 2024-03-11T21:48:24.086 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:48:23 smithi097 ceph-mon[33317]: from='mgr.14607 v1:172.21.15.92:0/1702717597' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "5.d", "id": [7, 6]}]: dispatch 2024-03-11T21:48:24.086 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:48:23 smithi097 ceph-mon[33317]: from='mgr.14607 v1:172.21.15.92:0/1702717597' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "6.1f", "id": [6, 0]}]: dispatch 2024-03-11T21:48:24.086 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:48:23 smithi097 ceph-mon[33317]: from='mgr.14607 ' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.8", "id": [1, 0]}]: dispatch 2024-03-11T21:48:24.086 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:48:23 smithi097 ceph-mon[33317]: from='mgr.14607 ' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.b", "id": [1, 2]}]: dispatch 2024-03-11T21:48:24.086 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:48:23 smithi097 ceph-mon[33317]: from='mgr.14607 ' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "5.d", "id": [7, 6]}]: dispatch 2024-03-11T21:48:24.086 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:48:23 smithi097 ceph-mon[33317]: from='mgr.14607 ' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "6.1f", "id": [6, 0]}]: dispatch 2024-03-11T21:48:24.086 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:48:23 smithi097 ceph-mon[33317]: from='client.24469 v1:172.21.15.97:0/404208310' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:48:24.086 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:48:23 smithi097 ceph-mon[33317]: from='mgr.14607 v1:172.21.15.92:0/1702717597' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T21:48:24.087 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:48:23 smithi097 ceph-mon[33317]: 6.1e scrub starts 2024-03-11T21:48:24.087 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:48:23 smithi097 ceph-mon[33317]: 6.1e scrub ok 2024-03-11T21:48:24.091 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:48:23 smithi092 ceph-mon[29572]: from='mgr.14607 v1:172.21.15.92:0/1702717597' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.8", "id": [1, 0]}]: dispatch 2024-03-11T21:48:24.091 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:48:23 smithi092 ceph-mon[29572]: from='mgr.14607 v1:172.21.15.92:0/1702717597' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.b", "id": [1, 2]}]: dispatch 2024-03-11T21:48:24.091 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:48:23 smithi092 ceph-mon[29572]: from='mgr.14607 v1:172.21.15.92:0/1702717597' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "5.d", "id": [7, 6]}]: dispatch 2024-03-11T21:48:24.091 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:48:23 smithi092 ceph-mon[29572]: from='mgr.14607 v1:172.21.15.92:0/1702717597' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "6.1f", "id": [6, 0]}]: dispatch 2024-03-11T21:48:24.091 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:48:23 smithi092 ceph-mon[29572]: from='mgr.14607 ' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.8", "id": [1, 0]}]: dispatch 2024-03-11T21:48:24.092 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:48:23 smithi092 ceph-mon[29572]: from='mgr.14607 ' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.b", "id": [1, 2]}]: dispatch 2024-03-11T21:48:24.092 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:48:23 smithi092 ceph-mon[29572]: from='mgr.14607 ' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "5.d", "id": [7, 6]}]: dispatch 2024-03-11T21:48:24.092 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:48:23 smithi092 ceph-mon[29572]: from='mgr.14607 ' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "6.1f", "id": [6, 0]}]: dispatch 2024-03-11T21:48:24.092 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:48:23 smithi092 ceph-mon[29572]: from='client.24469 v1:172.21.15.97:0/404208310' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:48:24.092 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:48:23 smithi092 ceph-mon[29572]: from='mgr.14607 v1:172.21.15.92:0/1702717597' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T21:48:24.092 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:48:23 smithi092 ceph-mon[29572]: 6.1e scrub starts 2024-03-11T21:48:24.092 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:48:23 smithi092 ceph-mon[29572]: 6.1e scrub ok 2024-03-11T21:48:24.092 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:48:23 smithi092 ceph-mon[37563]: from='mgr.14607 v1:172.21.15.92:0/1702717597' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.8", "id": [1, 0]}]: dispatch 2024-03-11T21:48:24.092 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:48:23 smithi092 ceph-mon[37563]: from='mgr.14607 v1:172.21.15.92:0/1702717597' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.b", "id": [1, 2]}]: dispatch 2024-03-11T21:48:24.092 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:48:23 smithi092 ceph-mon[37563]: from='mgr.14607 v1:172.21.15.92:0/1702717597' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "5.d", "id": [7, 6]}]: dispatch 2024-03-11T21:48:24.092 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:48:23 smithi092 ceph-mon[37563]: from='mgr.14607 v1:172.21.15.92:0/1702717597' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "6.1f", "id": [6, 0]}]: dispatch 2024-03-11T21:48:24.093 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:48:23 smithi092 ceph-mon[37563]: from='mgr.14607 ' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.8", "id": [1, 0]}]: dispatch 2024-03-11T21:48:24.093 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:48:23 smithi092 ceph-mon[37563]: from='mgr.14607 ' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.b", "id": [1, 2]}]: dispatch 2024-03-11T21:48:24.093 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:48:23 smithi092 ceph-mon[37563]: from='mgr.14607 ' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "5.d", "id": [7, 6]}]: dispatch 2024-03-11T21:48:24.093 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:48:23 smithi092 ceph-mon[37563]: from='mgr.14607 ' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "6.1f", "id": [6, 0]}]: dispatch 2024-03-11T21:48:24.093 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:48:23 smithi092 ceph-mon[37563]: from='client.24469 v1:172.21.15.97:0/404208310' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:48:24.093 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:48:23 smithi092 ceph-mon[37563]: from='mgr.14607 v1:172.21.15.92:0/1702717597' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T21:48:24.093 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:48:23 smithi092 ceph-mon[37563]: 6.1e scrub starts 2024-03-11T21:48:24.093 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:48:23 smithi092 ceph-mon[37563]: 6.1e scrub ok 2024-03-11T21:48:24.096 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:48:24.097 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:48:24.097 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:48:24.198 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:48:24.231 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:48:24.231 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:48:24.232 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:48:24.332 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:48:24.361 INFO:tasks.radosbench.radosbench.0.smithi092.stdout:Removed 138345 objects 2024-03-11T21:48:24.361 INFO:tasks.radosbench.radosbench.0.smithi092.stdout:Clean up completed and total clean up time :107.075 2024-03-11T21:48:24.362 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:48:24.363 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:48:24.363 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:48:24.464 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:48:24.487 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:48:24.488 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:48:24.488 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:48:24.589 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:48:24.615 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:48:24.615 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:48:24.615 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:48:24.716 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:48:24.740 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:48:24.741 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:48:24.741 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:48:24.842 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:48:24.867 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:48:24.868 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:48:24.868 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:48:24.969 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:48:24.994 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:48:24.995 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:48:24.995 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:48:25.085 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:48:24 smithi097 ceph-mon[33317]: pgmap v1075: 148 pgs: 148 active+clean; 304 MiB data, 5.6 GiB used, 441 GiB / 447 GiB avail; 852 B/s rd, 0 B/s wr, 1.49k op/s; 0 B/s, 0 objects/s recovering 2024-03-11T21:48:25.085 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:48:24 smithi097 ceph-mon[33317]: 6.1d scrub starts 2024-03-11T21:48:25.085 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:48:24 smithi097 ceph-mon[33317]: 6.1d scrub ok 2024-03-11T21:48:25.085 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:48:24 smithi097 ceph-mon[33317]: from='mgr.14607 ' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.8", "id": [1, 0]}]': finished 2024-03-11T21:48:25.085 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:48:24 smithi097 ceph-mon[33317]: from='mgr.14607 ' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.b", "id": [1, 2]}]': finished 2024-03-11T21:48:25.085 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:48:24 smithi097 ceph-mon[33317]: from='mgr.14607 ' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "5.d", "id": [7, 6]}]': finished 2024-03-11T21:48:25.085 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:48:24 smithi097 ceph-mon[33317]: from='mgr.14607 ' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "6.1f", "id": [6, 0]}]': finished 2024-03-11T21:48:25.086 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:48:24 smithi097 ceph-mon[33317]: osdmap e499: 8 total, 8 up, 5 in 2024-03-11T21:48:25.097 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:48:25.119 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:48:24 smithi092 ceph-mon[29572]: pgmap v1075: 148 pgs: 148 active+clean; 304 MiB data, 5.6 GiB used, 441 GiB / 447 GiB avail; 852 B/s rd, 0 B/s wr, 1.49k op/s; 0 B/s, 0 objects/s recovering 2024-03-11T21:48:25.119 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:48:24 smithi092 ceph-mon[29572]: 6.1d scrub starts 2024-03-11T21:48:25.119 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:48:24 smithi092 ceph-mon[29572]: 6.1d scrub ok 2024-03-11T21:48:25.119 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:48:24 smithi092 ceph-mon[29572]: from='mgr.14607 ' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.8", "id": [1, 0]}]': finished 2024-03-11T21:48:25.119 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:48:24 smithi092 ceph-mon[29572]: from='mgr.14607 ' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.b", "id": [1, 2]}]': finished 2024-03-11T21:48:25.119 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:48:24 smithi092 ceph-mon[29572]: from='mgr.14607 ' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "5.d", "id": [7, 6]}]': finished 2024-03-11T21:48:25.119 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:48:24 smithi092 ceph-mon[29572]: from='mgr.14607 ' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "6.1f", "id": [6, 0]}]': finished 2024-03-11T21:48:25.119 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:48:24 smithi092 ceph-mon[29572]: osdmap e499: 8 total, 8 up, 5 in 2024-03-11T21:48:25.120 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:48:24 smithi092 ceph-mon[37563]: pgmap v1075: 148 pgs: 148 active+clean; 304 MiB data, 5.6 GiB used, 441 GiB / 447 GiB avail; 852 B/s rd, 0 B/s wr, 1.49k op/s; 0 B/s, 0 objects/s recovering 2024-03-11T21:48:25.120 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:48:24 smithi092 ceph-mon[37563]: 6.1d scrub starts 2024-03-11T21:48:25.120 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:48:24 smithi092 ceph-mon[37563]: 6.1d scrub ok 2024-03-11T21:48:25.120 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:48:24 smithi092 ceph-mon[37563]: from='mgr.14607 ' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.8", "id": [1, 0]}]': finished 2024-03-11T21:48:25.120 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:48:24 smithi092 ceph-mon[37563]: from='mgr.14607 ' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.b", "id": [1, 2]}]': finished 2024-03-11T21:48:25.120 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:48:24 smithi092 ceph-mon[37563]: from='mgr.14607 ' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "5.d", "id": [7, 6]}]': finished 2024-03-11T21:48:25.120 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:48:24 smithi092 ceph-mon[37563]: from='mgr.14607 ' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "6.1f", "id": [6, 0]}]': finished 2024-03-11T21:48:25.120 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:48:24 smithi092 ceph-mon[37563]: osdmap e499: 8 total, 8 up, 5 in 2024-03-11T21:48:25.121 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:48:25.121 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:48:25.122 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:48:25.222 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:48:25.247 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:48:25.248 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:48:25.248 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:48:25.349 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:48:25.373 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:48:25.374 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:48:25.374 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:48:25.475 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:48:25.501 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:48:25.501 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:48:25.501 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:48:25.602 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:48:25.628 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:48:25.628 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:48:25.628 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:48:25.729 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:48:25.755 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:48:25.756 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:48:25.756 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:48:25.857 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:48:25.883 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:48:25.884 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:48:25.884 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:48:25.986 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:48:26.010 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:48:26.010 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:48:26.011 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:48:26.046 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:48:25 smithi092 ceph-mon[29572]: 6.e scrub starts 2024-03-11T21:48:26.047 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:48:25 smithi092 ceph-mon[29572]: 6.e scrub ok 2024-03-11T21:48:26.047 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:48:25 smithi092 ceph-mon[29572]: osdmap e500: 8 total, 8 up, 5 in 2024-03-11T21:48:26.047 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:48:25 smithi092 ceph-mon[29572]: 6.1f scrub starts 2024-03-11T21:48:26.047 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:48:25 smithi092 ceph-mon[29572]: 6.1f scrub ok 2024-03-11T21:48:26.047 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:48:25 smithi092 ceph-mon[37563]: 6.e scrub starts 2024-03-11T21:48:26.047 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:48:25 smithi092 ceph-mon[37563]: 6.e scrub ok 2024-03-11T21:48:26.048 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:48:25 smithi092 ceph-mon[37563]: osdmap e500: 8 total, 8 up, 5 in 2024-03-11T21:48:26.048 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:48:25 smithi092 ceph-mon[37563]: 6.1f scrub starts 2024-03-11T21:48:26.048 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:48:25 smithi092 ceph-mon[37563]: 6.1f scrub ok 2024-03-11T21:48:26.085 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:48:25 smithi097 ceph-mon[33317]: 6.e scrub starts 2024-03-11T21:48:26.085 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:48:25 smithi097 ceph-mon[33317]: 6.e scrub ok 2024-03-11T21:48:26.085 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:48:25 smithi097 ceph-mon[33317]: osdmap e500: 8 total, 8 up, 5 in 2024-03-11T21:48:26.085 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:48:25 smithi097 ceph-mon[33317]: 6.1f scrub starts 2024-03-11T21:48:26.085 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:48:25 smithi097 ceph-mon[33317]: 6.1f scrub ok 2024-03-11T21:48:26.112 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:48:26.137 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:48:26.137 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:48:26.138 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:48:26.239 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:48:26.263 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:48:26.263 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:48:26.264 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:48:26.364 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:48:26.389 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:48:26.389 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:48:26.390 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:48:26.490 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:48:26.514 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:48:26.515 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:48:26.515 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:48:26.616 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:48:26.640 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:48:26.641 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:48:26.641 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:48:26.742 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:48:26.766 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:48:26.767 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:48:26.767 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:48:26.798 INFO:teuthology.orchestra.run.smithi092.stdout:osd.0: {} 2024-03-11T21:48:26.817 INFO:teuthology.orchestra.run.smithi092.stdout:osd.1: {} 2024-03-11T21:48:26.832 INFO:teuthology.orchestra.run.smithi092.stderr:nodeep-scrub is unset 2024-03-11T21:48:26.843 INFO:teuthology.orchestra.run.smithi092.stdout:osd.2: {} 2024-03-11T21:48:26.865 INFO:teuthology.orchestra.run.smithi092.stdout:osd.3: {} 2024-03-11T21:48:26.868 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:48:26.883 INFO:teuthology.orchestra.run.smithi092.stdout:osd.4: {} 2024-03-11T21:48:26.891 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:48:26.892 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:48:26.892 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:48:26.903 INFO:teuthology.orchestra.run.smithi092.stdout:osd.5: {} 2024-03-11T21:48:26.925 INFO:teuthology.orchestra.run.smithi092.stdout:osd.6: {} 2024-03-11T21:48:26.945 INFO:teuthology.orchestra.run.smithi092.stdout:osd.7: {} 2024-03-11T21:48:26.946 INFO:teuthology.orchestra.run.smithi092.stderr:osd.0: osd_enable_op_tracker = '' 2024-03-11T21:48:26.946 INFO:teuthology.orchestra.run.smithi092.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-11T21:48:26.946 INFO:teuthology.orchestra.run.smithi092.stderr:osd.2: osd_enable_op_tracker = '' 2024-03-11T21:48:26.946 INFO:teuthology.orchestra.run.smithi092.stderr:osd.3: osd_enable_op_tracker = '' 2024-03-11T21:48:26.946 INFO:teuthology.orchestra.run.smithi092.stderr:osd.4: osd_enable_op_tracker = '' 2024-03-11T21:48:26.946 INFO:teuthology.orchestra.run.smithi092.stderr:osd.5: osd_enable_op_tracker = '' 2024-03-11T21:48:26.946 INFO:teuthology.orchestra.run.smithi092.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-11T21:48:26.946 INFO:teuthology.orchestra.run.smithi092.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-11T21:48:26.993 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:48:27.019 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:48:27.020 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:48:27.020 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:48:27.044 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:48:26 smithi092 ceph-mon[29572]: pgmap v1078: 148 pgs: 1 active+recovering, 147 active+clean; 109 MiB data, 4.8 GiB used, 442 GiB / 447 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.82k op/s; 2/5808 objects misplaced (0.034%); 13 B/s, 0 objects/s recovering 2024-03-11T21:48:27.045 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:48:26 smithi092 ceph-mon[29572]: 4.4 scrub starts 2024-03-11T21:48:27.045 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:48:26 smithi092 ceph-mon[29572]: 4.4 scrub ok 2024-03-11T21:48:27.045 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:48:26 smithi092 ceph-mon[29572]: from='client.? v1:172.21.15.92:0/1718155668' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-11T21:48:27.045 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:48:26 smithi092 ceph-mon[29572]: 11.a scrub starts 2024-03-11T21:48:27.045 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:48:26 smithi092 ceph-mon[29572]: 11.a scrub ok 2024-03-11T21:48:27.045 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:48:26 smithi092 ceph-mon[29572]: from='client.? v1:172.21.15.92:0/2958455657' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-11T21:48:27.045 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:48:26 smithi092 ceph-mon[37563]: pgmap v1078: 148 pgs: 1 active+recovering, 147 active+clean; 109 MiB data, 4.8 GiB used, 442 GiB / 447 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.82k op/s; 2/5808 objects misplaced (0.034%); 13 B/s, 0 objects/s recovering 2024-03-11T21:48:27.045 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:48:26 smithi092 ceph-mon[37563]: 4.4 scrub starts 2024-03-11T21:48:27.045 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:48:26 smithi092 ceph-mon[37563]: 4.4 scrub ok 2024-03-11T21:48:27.045 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:48:26 smithi092 ceph-mon[37563]: from='client.? v1:172.21.15.92:0/1718155668' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-11T21:48:27.045 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:48:26 smithi092 ceph-mon[37563]: 11.a scrub starts 2024-03-11T21:48:27.046 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:48:26 smithi092 ceph-mon[37563]: 11.a scrub ok 2024-03-11T21:48:27.046 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:48:26 smithi092 ceph-mon[37563]: from='client.? v1:172.21.15.92:0/2958455657' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-11T21:48:27.085 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:48:26 smithi097 ceph-mon[33317]: pgmap v1078: 148 pgs: 1 active+recovering, 147 active+clean; 109 MiB data, 4.8 GiB used, 442 GiB / 447 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.82k op/s; 2/5808 objects misplaced (0.034%); 13 B/s, 0 objects/s recovering 2024-03-11T21:48:27.085 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:48:26 smithi097 ceph-mon[33317]: 4.4 scrub starts 2024-03-11T21:48:27.085 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:48:26 smithi097 ceph-mon[33317]: 4.4 scrub ok 2024-03-11T21:48:27.085 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:48:26 smithi097 ceph-mon[33317]: from='client.? v1:172.21.15.92:0/1718155668' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-11T21:48:27.085 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:48:26 smithi097 ceph-mon[33317]: 11.a scrub starts 2024-03-11T21:48:27.085 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:48:26 smithi097 ceph-mon[33317]: 11.a scrub ok 2024-03-11T21:48:27.085 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:48:26 smithi097 ceph-mon[33317]: from='client.? v1:172.21.15.92:0/2958455657' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-11T21:48:27.121 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:48:27.144 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:48:27.145 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:48:27.145 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:48:27.246 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:48:27.274 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:48:27.275 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:48:27.275 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:48:27.376 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:48:27.403 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:48:27.404 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:48:27.404 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:48:27.505 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:48:27.531 INFO:teuthology.orchestra.run.smithi092.stdout:ERROR: (22) Invalid argument 2024-03-11T21:48:27.531 INFO:teuthology.orchestra.run.smithi092.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-11T21:48:27.532 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:48:27.533 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:48:27.533 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:48:27.634 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:48:27.660 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:48:27.661 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:48:27.661 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:48:27.762 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:48:27.787 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:48:27.787 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:48:27.788 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:48:27.822 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:48:27 smithi092 ceph-00db0016-dfed-11ee-95c8-87774f69a715-mon-a[29568]: 2024-03-11T21:48:27.553+0000 7f87a4389700 -1 mon.a@0(leader).osd e501 definitely_dead 0 2024-03-11T21:48:27.848 INFO:teuthology.orchestra.run.smithi092.stderr:marked in osd.5. 2024-03-11T21:48:27.889 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:48:27.914 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:48:27.915 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:48:27.915 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:48:28.016 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:48:28.040 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:48:28.041 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:48:28.041 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:48:28.085 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:48:27 smithi097 ceph-mon[33317]: 2.1 scrub starts 2024-03-11T21:48:28.085 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:48:27 smithi097 ceph-mon[33317]: 2.1 scrub ok 2024-03-11T21:48:28.085 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:48:27 smithi097 ceph-mon[33317]: 2.2 scrub starts 2024-03-11T21:48:28.086 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:48:27 smithi097 ceph-mon[33317]: 2.2 scrub ok 2024-03-11T21:48:28.086 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:48:27 smithi097 ceph-mon[33317]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-03-11T21:48:28.086 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:48:27 smithi097 ceph-mon[33317]: Cluster is now healthy 2024-03-11T21:48:28.086 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:48:27 smithi097 ceph-mon[33317]: from='client.? v1:172.21.15.92:0/1718155668' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-03-11T21:48:28.086 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:48:27 smithi097 ceph-mon[33317]: osdmap e501: 8 total, 8 up, 5 in 2024-03-11T21:48:28.086 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:48:27 smithi097 ceph-mon[33317]: 11.8 scrub starts 2024-03-11T21:48:28.086 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:48:27 smithi097 ceph-mon[33317]: 11.8 scrub ok 2024-03-11T21:48:28.086 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:48:27 smithi097 ceph-mon[33317]: from='client.? v1:172.21.15.92:0/1093815816' entity='client.admin' cmd=[{"prefix": "osd in", "ids": ["5"]}]: dispatch 2024-03-11T21:48:28.119 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:48:27 smithi092 ceph-mon[29572]: 2.1 scrub starts 2024-03-11T21:48:28.119 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:48:27 smithi092 ceph-mon[29572]: 2.1 scrub ok 2024-03-11T21:48:28.119 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:48:27 smithi092 ceph-mon[29572]: 2.2 scrub starts 2024-03-11T21:48:28.119 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:48:27 smithi092 ceph-mon[29572]: 2.2 scrub ok 2024-03-11T21:48:28.119 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:48:27 smithi092 ceph-mon[29572]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-03-11T21:48:28.119 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:48:27 smithi092 ceph-mon[29572]: Cluster is now healthy 2024-03-11T21:48:28.119 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:48:27 smithi092 ceph-mon[29572]: from='client.? v1:172.21.15.92:0/1718155668' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-03-11T21:48:28.120 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:48:27 smithi092 ceph-mon[29572]: osdmap e501: 8 total, 8 up, 5 in 2024-03-11T21:48:28.120 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:48:27 smithi092 ceph-mon[29572]: 11.8 scrub starts 2024-03-11T21:48:28.120 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:48:27 smithi092 ceph-mon[29572]: 11.8 scrub ok 2024-03-11T21:48:28.120 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:48:27 smithi092 ceph-mon[29572]: from='client.? v1:172.21.15.92:0/1093815816' entity='client.admin' cmd=[{"prefix": "osd in", "ids": ["5"]}]: dispatch 2024-03-11T21:48:28.120 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:48:27 smithi092 ceph-mon[37563]: 2.1 scrub starts 2024-03-11T21:48:28.120 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:48:27 smithi092 ceph-mon[37563]: 2.1 scrub ok 2024-03-11T21:48:28.120 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:48:27 smithi092 ceph-mon[37563]: 2.2 scrub starts 2024-03-11T21:48:28.120 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:48:27 smithi092 ceph-mon[37563]: 2.2 scrub ok 2024-03-11T21:48:28.120 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:48:27 smithi092 ceph-mon[37563]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-03-11T21:48:28.121 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:48:27 smithi092 ceph-mon[37563]: Cluster is now healthy 2024-03-11T21:48:28.121 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:48:27 smithi092 ceph-mon[37563]: from='client.? v1:172.21.15.92:0/1718155668' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-03-11T21:48:28.121 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:48:27 smithi092 ceph-mon[37563]: osdmap e501: 8 total, 8 up, 5 in 2024-03-11T21:48:28.121 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:48:27 smithi092 ceph-mon[37563]: 11.8 scrub starts 2024-03-11T21:48:28.121 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:48:27 smithi092 ceph-mon[37563]: 11.8 scrub ok 2024-03-11T21:48:28.121 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:48:27 smithi092 ceph-mon[37563]: from='client.? v1:172.21.15.92:0/1093815816' entity='client.admin' cmd=[{"prefix": "osd in", "ids": ["5"]}]: dispatch 2024-03-11T21:48:28.121 INFO:journalctl@ceph.mgr.y.smithi092.stdout:Mar 11 21:48:27 smithi092 ceph-00db0016-dfed-11ee-95c8-87774f69a715-mgr-y[29801]: ::ffff:172.21.15.97 - - [11/Mar/2024:21:48:27] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-11T21:48:28.142 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:48:28.170 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:48:28.171 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:48:28.171 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:48:28.272 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:48:28.300 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:48:28.300 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:48:28.301 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:48:28.401 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:48:28.419 INFO:tasks.cephadm.ceph_manager.ceph:removing pool_name unique_pool_4 2024-03-11T21:48:28.419 DEBUG:teuthology.orchestra.run.smithi092:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid 00db0016-dfed-11ee-95c8-87774f69a715 -- ceph osd pool rm unique_pool_4 unique_pool_4 --yes-i-really-really-mean-it 2024-03-11T21:48:28.429 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:48:28.429 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:48:28.429 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:48:28.530 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:48:28.555 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:48:28.555 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:48:28.556 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:48:28.657 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:48:28.683 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:48:28.684 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:48:28.684 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:48:28.785 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:48:28.808 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:48:28.808 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:48:28.809 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:48:28.909 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:48:28.932 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:48:28.934 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:48:28.934 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:48:29.035 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:48:29.058 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:48:29.059 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:48:29.060 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:48:29.108 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:48:28 smithi092 ceph-mon[29572]: pgmap v1080: 148 pgs: 1 active+recovering, 147 active+clean; 109 MiB data, 4.8 GiB used, 442 GiB / 447 GiB avail; 852 B/s rd, 0 B/s wr, 1.18k op/s; 2/5808 objects misplaced (0.034%); 18 B/s, 0 objects/s recovering 2024-03-11T21:48:29.108 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:48:28 smithi092 ceph-mon[29572]: 4.12 scrub starts 2024-03-11T21:48:29.108 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:48:28 smithi092 ceph-mon[29572]: 4.12 scrub ok 2024-03-11T21:48:29.108 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:48:28 smithi092 ceph-mon[29572]: from='client.? v1:172.21.15.92:0/1093815816' entity='client.admin' cmd='[{"prefix": "osd in", "ids": ["5"]}]': finished 2024-03-11T21:48:29.108 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:48:28 smithi092 ceph-mon[29572]: osdmap e502: 8 total, 8 up, 6 in 2024-03-11T21:48:29.108 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:48:28 smithi092 ceph-mon[29572]: 11.9 scrub starts 2024-03-11T21:48:29.108 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:48:28 smithi092 ceph-mon[29572]: 11.9 scrub ok 2024-03-11T21:48:29.108 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:48:28 smithi092 ceph-mon[29572]: osdmap e503: 8 total, 8 up, 6 in 2024-03-11T21:48:29.108 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:48:28 smithi092 ceph-mon[37563]: pgmap v1080: 148 pgs: 1 active+recovering, 147 active+clean; 109 MiB data, 4.8 GiB used, 442 GiB / 447 GiB avail; 852 B/s rd, 0 B/s wr, 1.18k op/s; 2/5808 objects misplaced (0.034%); 18 B/s, 0 objects/s recovering 2024-03-11T21:48:29.109 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:48:28 smithi092 ceph-mon[37563]: 4.12 scrub starts 2024-03-11T21:48:29.109 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:48:28 smithi092 ceph-mon[37563]: 4.12 scrub ok 2024-03-11T21:48:29.109 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:48:28 smithi092 ceph-mon[37563]: from='client.? v1:172.21.15.92:0/1093815816' entity='client.admin' cmd='[{"prefix": "osd in", "ids": ["5"]}]': finished 2024-03-11T21:48:29.109 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:48:28 smithi092 ceph-mon[37563]: osdmap e502: 8 total, 8 up, 6 in 2024-03-11T21:48:29.109 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:48:28 smithi092 ceph-mon[37563]: 11.9 scrub starts 2024-03-11T21:48:29.109 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:48:28 smithi092 ceph-mon[37563]: 11.9 scrub ok 2024-03-11T21:48:29.109 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:48:28 smithi092 ceph-mon[37563]: osdmap e503: 8 total, 8 up, 6 in 2024-03-11T21:48:29.135 DEBUG:teuthology.orchestra.run.smithi092:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid 00db0016-dfed-11ee-95c8-87774f69a715 -- ceph daemon osd.2 dump_blocked_ops 2024-03-11T21:48:29.160 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:48:29.201 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:48:29.202 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:48:29.202 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:48:29.303 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:48:29.327 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:48:29.328 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:48:29.328 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:48:29.335 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:48:28 smithi097 ceph-mon[33317]: pgmap v1080: 148 pgs: 1 active+recovering, 147 active+clean; 109 MiB data, 4.8 GiB used, 442 GiB / 447 GiB avail; 852 B/s rd, 0 B/s wr, 1.18k op/s; 2/5808 objects misplaced (0.034%); 18 B/s, 0 objects/s recovering 2024-03-11T21:48:29.335 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:48:28 smithi097 ceph-mon[33317]: 4.12 scrub starts 2024-03-11T21:48:29.335 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:48:28 smithi097 ceph-mon[33317]: 4.12 scrub ok 2024-03-11T21:48:29.335 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:48:28 smithi097 ceph-mon[33317]: from='client.? v1:172.21.15.92:0/1093815816' entity='client.admin' cmd='[{"prefix": "osd in", "ids": ["5"]}]': finished 2024-03-11T21:48:29.335 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:48:28 smithi097 ceph-mon[33317]: osdmap e502: 8 total, 8 up, 6 in 2024-03-11T21:48:29.335 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:48:28 smithi097 ceph-mon[33317]: 11.9 scrub starts 2024-03-11T21:48:29.335 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:48:28 smithi097 ceph-mon[33317]: 11.9 scrub ok 2024-03-11T21:48:29.335 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:48:28 smithi097 ceph-mon[33317]: osdmap e503: 8 total, 8 up, 6 in 2024-03-11T21:48:29.429 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:48:29.456 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:48:29.456 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:48:29.456 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:48:29.557 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:48:29.581 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:48:29.581 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:48:29.582 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:48:29.683 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:48:29.708 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:48:29.709 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:48:29.709 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:48:29.774 INFO:tasks.thrashosds.thrasher:Added osd 5 2024-03-11T21:48:29.810 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:48:29.824 INFO:teuthology.orchestra.run.smithi092.stderr:Inferring config /var/lib/ceph/00db0016-dfed-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T21:48:29.836 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:48:29.837 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:48:29.837 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:48:29.938 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:48:29.961 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:48:29.962 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:48:29.962 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:48:30.063 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:48:30.087 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:48:30.088 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:48:30.088 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:48:30.119 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:48:29 smithi092 ceph-mon[29572]: 3.2 scrub starts 2024-03-11T21:48:30.119 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:48:29 smithi092 ceph-mon[29572]: 3.2 scrub ok 2024-03-11T21:48:30.119 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:48:29 smithi092 ceph-mon[29572]: 5.2 scrub starts 2024-03-11T21:48:30.119 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:48:29 smithi092 ceph-mon[29572]: 5.2 scrub ok 2024-03-11T21:48:30.119 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:48:29 smithi092 ceph-mon[29572]: osdmap e504: 8 total, 8 up, 6 in 2024-03-11T21:48:30.119 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:48:29 smithi092 ceph-mon[29572]: 11.8 starting backfill to osd.5 from (0'0,0'0] MAX to 499'17396 2024-03-11T21:48:30.119 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:48:29 smithi092 ceph-mon[29572]: 11.b starting backfill to osd.5 from (0'0,0'0] MAX to 499'17296 2024-03-11T21:48:30.120 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:48:29 smithi092 ceph-mon[29572]: 11.5 starting backfill to osd.5 from (0'0,0'0] MAX to 499'17402 2024-03-11T21:48:30.120 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:48:29 smithi092 ceph-mon[29572]: 11.7 starting backfill to osd.5 from (0'0,0'0] MAX to 499'17178 2024-03-11T21:48:30.120 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:48:29 smithi092 ceph-mon[29572]: 11.a starting backfill to osd.5 from (0'0,0'0] MAX to 499'17598 2024-03-11T21:48:30.120 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:48:29 smithi092 ceph-mon[37563]: 3.2 scrub starts 2024-03-11T21:48:30.120 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:48:29 smithi092 ceph-mon[37563]: 3.2 scrub ok 2024-03-11T21:48:30.120 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:48:29 smithi092 ceph-mon[37563]: 5.2 scrub starts 2024-03-11T21:48:30.121 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:48:29 smithi092 ceph-mon[37563]: 5.2 scrub ok 2024-03-11T21:48:30.121 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:48:29 smithi092 ceph-mon[37563]: osdmap e504: 8 total, 8 up, 6 in 2024-03-11T21:48:30.121 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:48:29 smithi092 ceph-mon[37563]: 11.8 starting backfill to osd.5 from (0'0,0'0] MAX to 499'17396 2024-03-11T21:48:30.121 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:48:29 smithi092 ceph-mon[37563]: 11.b starting backfill to osd.5 from (0'0,0'0] MAX to 499'17296 2024-03-11T21:48:30.121 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:48:29 smithi092 ceph-mon[37563]: 11.5 starting backfill to osd.5 from (0'0,0'0] MAX to 499'17402 2024-03-11T21:48:30.121 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:48:29 smithi092 ceph-mon[37563]: 11.7 starting backfill to osd.5 from (0'0,0'0] MAX to 499'17178 2024-03-11T21:48:30.121 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:48:29 smithi092 ceph-mon[37563]: 11.a starting backfill to osd.5 from (0'0,0'0] MAX to 499'17598 2024-03-11T21:48:30.189 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:48:30.216 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:48:30.216 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:48:30.217 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:48:30.317 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:48:30.335 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:48:29 smithi097 ceph-mon[33317]: 3.2 scrub starts 2024-03-11T21:48:30.335 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:48:29 smithi097 ceph-mon[33317]: 3.2 scrub ok 2024-03-11T21:48:30.335 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:48:29 smithi097 ceph-mon[33317]: 5.2 scrub starts 2024-03-11T21:48:30.335 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:48:29 smithi097 ceph-mon[33317]: 5.2 scrub ok 2024-03-11T21:48:30.335 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:48:29 smithi097 ceph-mon[33317]: osdmap e504: 8 total, 8 up, 6 in 2024-03-11T21:48:30.335 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:48:29 smithi097 ceph-mon[33317]: 11.8 starting backfill to osd.5 from (0'0,0'0] MAX to 499'17396 2024-03-11T21:48:30.335 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:48:29 smithi097 ceph-mon[33317]: 11.b starting backfill to osd.5 from (0'0,0'0] MAX to 499'17296 2024-03-11T21:48:30.335 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:48:29 smithi097 ceph-mon[33317]: 11.5 starting backfill to osd.5 from (0'0,0'0] MAX to 499'17402 2024-03-11T21:48:30.336 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:48:29 smithi097 ceph-mon[33317]: 11.7 starting backfill to osd.5 from (0'0,0'0] MAX to 499'17178 2024-03-11T21:48:30.336 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:48:29 smithi097 ceph-mon[33317]: 11.a starting backfill to osd.5 from (0'0,0'0] MAX to 499'17598 2024-03-11T21:48:30.340 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:48:30.341 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:48:30.341 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:48:30.442 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:48:30.470 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:48:30.471 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:48:30.471 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:48:30.572 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:48:30.574 DEBUG:teuthology.orchestra.run.smithi092:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid 00db0016-dfed-11ee-95c8-87774f69a715 -- ceph osd set noscrub 2024-03-11T21:48:30.596 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:48:30.596 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:48:30.597 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:48:30.697 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:48:30.726 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:48:30.727 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:48:30.727 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:48:30.828 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:48:30.860 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:48:30.860 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:48:30.861 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:48:30.962 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:48:30.986 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:48:30.987 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:48:30.987 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:48:31.085 INFO:teuthology.orchestra.run.smithi092.stderr:Inferring config /var/lib/ceph/00db0016-dfed-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T21:48:31.088 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:48:31.115 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:48:31.116 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:48:31.116 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:48:31.119 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:48:30 smithi092 ceph-mon[29572]: pgmap v1083: 148 pgs: 6 peering, 1 active+recovering, 141 active+clean; 2.9 MiB data, 4.4 GiB used, 443 GiB / 447 GiB avail; 1.1 KiB/s rd, 0 B/s wr, 552 op/s; 2/708 objects misplaced (0.282%); 164 B/s, 5 objects/s recovering 2024-03-11T21:48:31.119 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:48:30 smithi092 ceph-mon[29572]: 4.f scrub starts 2024-03-11T21:48:31.119 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:48:30 smithi092 ceph-mon[29572]: 4.f scrub ok 2024-03-11T21:48:31.119 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:48:30 smithi092 ceph-mon[37563]: pgmap v1083: 148 pgs: 6 peering, 1 active+recovering, 141 active+clean; 2.9 MiB data, 4.4 GiB used, 443 GiB / 447 GiB avail; 1.1 KiB/s rd, 0 B/s wr, 552 op/s; 2/708 objects misplaced (0.282%); 164 B/s, 5 objects/s recovering 2024-03-11T21:48:31.119 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:48:30 smithi092 ceph-mon[37563]: 4.f scrub starts 2024-03-11T21:48:31.119 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:48:30 smithi092 ceph-mon[37563]: 4.f scrub ok 2024-03-11T21:48:31.218 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:48:31.242 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:48:31.243 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:48:31.243 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:48:31.335 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:48:30 smithi097 ceph-mon[33317]: pgmap v1083: 148 pgs: 6 peering, 1 active+recovering, 141 active+clean; 2.9 MiB data, 4.4 GiB used, 443 GiB / 447 GiB avail; 1.1 KiB/s rd, 0 B/s wr, 552 op/s; 2/708 objects misplaced (0.282%); 164 B/s, 5 objects/s recovering 2024-03-11T21:48:31.335 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:48:30 smithi097 ceph-mon[33317]: 4.f scrub starts 2024-03-11T21:48:31.335 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:48:30 smithi097 ceph-mon[33317]: 4.f scrub ok 2024-03-11T21:48:31.344 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:48:31.367 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:48:31.368 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:48:31.368 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:48:31.469 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:48:31.497 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:48:31.497 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:48:31.498 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:48:31.599 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:48:31.626 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:48:31.627 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:48:31.628 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:48:31.728 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:48:31.756 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:48:31.757 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:48:31.757 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:48:31.858 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:48:31.883 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:48:31.884 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:48:31.884 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:48:31.985 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:48:32.012 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:48:32.012 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:48:32.013 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:48:32.095 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:48:31 smithi092 ceph-mon[29572]: osdmap e505: 8 total, 8 up, 6 in 2024-03-11T21:48:32.096 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:48:31 smithi092 ceph-mon[29572]: 11.6 scrub starts 2024-03-11T21:48:32.096 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:48:31 smithi092 ceph-mon[29572]: 11.6 scrub ok 2024-03-11T21:48:32.096 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:48:31 smithi092 ceph-mon[29572]: 5.a scrub starts 2024-03-11T21:48:32.096 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:48:31 smithi092 ceph-mon[29572]: 5.a scrub ok 2024-03-11T21:48:32.096 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:48:31 smithi092 ceph-mon[29572]: 4.7 scrub starts 2024-03-11T21:48:32.096 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:48:31 smithi092 ceph-mon[29572]: 4.7 scrub ok 2024-03-11T21:48:32.096 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:48:31 smithi092 ceph-mon[29572]: osdmap e506: 8 total, 8 up, 6 in 2024-03-11T21:48:32.096 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:48:31 smithi092 ceph-mon[37563]: osdmap e505: 8 total, 8 up, 6 in 2024-03-11T21:48:32.096 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:48:31 smithi092 ceph-mon[37563]: 11.6 scrub starts 2024-03-11T21:48:32.096 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:48:31 smithi092 ceph-mon[37563]: 11.6 scrub ok 2024-03-11T21:48:32.097 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:48:31 smithi092 ceph-mon[37563]: 5.a scrub starts 2024-03-11T21:48:32.097 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:48:31 smithi092 ceph-mon[37563]: 5.a scrub ok 2024-03-11T21:48:32.097 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:48:31 smithi092 ceph-mon[37563]: 4.7 scrub starts 2024-03-11T21:48:32.097 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:48:31 smithi092 ceph-mon[37563]: 4.7 scrub ok 2024-03-11T21:48:32.097 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:48:31 smithi092 ceph-mon[37563]: osdmap e506: 8 total, 8 up, 6 in 2024-03-11T21:48:32.113 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:48:32.138 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:48:32.139 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:48:32.139 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:48:32.240 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:48:32.264 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:48:32.265 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:48:32.265 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:48:32.325 DEBUG:teuthology.orchestra.run.smithi092:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid 00db0016-dfed-11ee-95c8-87774f69a715 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-03-11T21:48:32.335 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:48:31 smithi097 ceph-mon[33317]: osdmap e505: 8 total, 8 up, 6 in 2024-03-11T21:48:32.335 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:48:31 smithi097 ceph-mon[33317]: 11.6 scrub starts 2024-03-11T21:48:32.335 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:48:31 smithi097 ceph-mon[33317]: 11.6 scrub ok 2024-03-11T21:48:32.335 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:48:31 smithi097 ceph-mon[33317]: 5.a scrub starts 2024-03-11T21:48:32.335 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:48:31 smithi097 ceph-mon[33317]: 5.a scrub ok 2024-03-11T21:48:32.335 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:48:31 smithi097 ceph-mon[33317]: 4.7 scrub starts 2024-03-11T21:48:32.335 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:48:31 smithi097 ceph-mon[33317]: 4.7 scrub ok 2024-03-11T21:48:32.335 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:48:31 smithi097 ceph-mon[33317]: osdmap e506: 8 total, 8 up, 6 in 2024-03-11T21:48:32.367 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:48:32.392 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:48:32.393 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:48:32.393 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:48:32.494 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:48:32.518 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:48:32.519 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:48:32.519 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:48:32.620 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:48:32.644 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:48:32.645 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:48:32.646 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:48:32.746 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0 2024-03-11T21:48:32.771 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:48:32.771 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:48:32.772 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.0' 2024-03-11T21:48:32.872 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:48:32.896 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:48:32.897 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:48:32.897 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:48:32.953 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:48:32 smithi092 ceph-mon[29572]: pgmap v1086: 148 pgs: 2 activating+remapped, 6 peering, 140 active+clean; 582 KiB data, 4.4 GiB used, 443 GiB / 447 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 435 op/s; 297 B/s, 9 objects/s recovering 2024-03-11T21:48:32.954 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:48:32 smithi092 ceph-mon[29572]: 4.15 scrub starts 2024-03-11T21:48:32.954 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:48:32 smithi092 ceph-mon[29572]: 4.15 scrub ok 2024-03-11T21:48:32.954 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:48:32 smithi092 ceph-mon[29572]: 6.1b scrub starts 2024-03-11T21:48:32.954 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:48:32 smithi092 ceph-mon[29572]: 2.10 scrub starts 2024-03-11T21:48:32.954 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:48:32 smithi092 ceph-mon[29572]: 2.10 scrub ok 2024-03-11T21:48:32.954 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:48:32 smithi092 ceph-mon[29572]: 6.1b scrub ok 2024-03-11T21:48:32.954 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:48:32 smithi092 ceph-mon[37563]: pgmap v1086: 148 pgs: 2 activating+remapped, 6 peering, 140 active+clean; 582 KiB data, 4.4 GiB used, 443 GiB / 447 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 435 op/s; 297 B/s, 9 objects/s recovering 2024-03-11T21:48:32.954 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:48:32 smithi092 ceph-mon[37563]: 4.15 scrub starts 2024-03-11T21:48:32.954 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:48:32 smithi092 ceph-mon[37563]: 4.15 scrub ok 2024-03-11T21:48:32.954 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:48:32 smithi092 ceph-mon[37563]: 6.1b scrub starts 2024-03-11T21:48:32.955 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:48:32 smithi092 ceph-mon[37563]: 2.10 scrub starts 2024-03-11T21:48:32.955 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:48:32 smithi092 ceph-mon[37563]: 2.10 scrub ok 2024-03-11T21:48:32.955 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:48:32 smithi092 ceph-mon[37563]: 6.1b scrub ok 2024-03-11T21:48:32.998 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:48:33.023 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:48:33.023 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:48:33.024 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:48:33.125 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:48:33.149 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:48:33.149 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:48:33.149 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:48:33.250 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:48:33.273 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:48:33.274 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:48:33.275 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:48:33.310 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:48:32 smithi097 ceph-mon[33317]: pgmap v1086: 148 pgs: 2 activating+remapped, 6 peering, 140 active+clean; 582 KiB data, 4.4 GiB used, 443 GiB / 447 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 435 op/s; 297 B/s, 9 objects/s recovering 2024-03-11T21:48:33.310 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:48:32 smithi097 ceph-mon[33317]: 4.15 scrub starts 2024-03-11T21:48:33.310 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:48:32 smithi097 ceph-mon[33317]: 4.15 scrub ok 2024-03-11T21:48:33.310 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:48:32 smithi097 ceph-mon[33317]: 6.1b scrub starts 2024-03-11T21:48:33.310 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:48:32 smithi097 ceph-mon[33317]: 2.10 scrub starts 2024-03-11T21:48:33.310 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:48:32 smithi097 ceph-mon[33317]: 2.10 scrub ok 2024-03-11T21:48:33.310 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:48:32 smithi097 ceph-mon[33317]: 6.1b scrub ok 2024-03-11T21:48:33.319 INFO:teuthology.orchestra.run.smithi092.stdout:ERROR: (22) Invalid argument 2024-03-11T21:48:33.319 INFO:teuthology.orchestra.run.smithi092.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-03-11T21:48:33.375 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:48:33.400 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:48:33.400 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:48:33.401 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:48:33.502 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:48:33.527 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:48:33.527 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:48:33.528 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:48:33.585 INFO:journalctl@ceph.iscsi.iscsi.a.smithi097.stdout:Mar 11 21:48:33 smithi097 ceph-00db0016-dfed-11ee-95c8-87774f69a715-iscsi-iscsi-a[60943]: debug there is no tcmu-runner data available 2024-03-11T21:48:33.628 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:48:33.652 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:48:33.652 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:48:33.653 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:48:33.753 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:48:33.776 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:48:33.777 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:48:33.778 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:48:33.878 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:48:33.904 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:48:33.904 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:48:33.904 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:48:34.005 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:48:34.029 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:48:34.029 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:48:34.030 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:48:34.119 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:48:33 smithi092 ceph-mon[29572]: 2.16 scrub starts 2024-03-11T21:48:34.119 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:48:33 smithi092 ceph-mon[29572]: 2.16 scrub ok 2024-03-11T21:48:34.119 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:48:33 smithi092 ceph-mon[29572]: from='client.24469 v1:172.21.15.97:0/404208310' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:48:34.119 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:48:33 smithi092 ceph-mon[29572]: 6.6 scrub starts 2024-03-11T21:48:34.119 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:48:33 smithi092 ceph-mon[29572]: 6.6 scrub ok 2024-03-11T21:48:34.120 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:48:33 smithi092 ceph-mon[37563]: 2.16 scrub starts 2024-03-11T21:48:34.120 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:48:33 smithi092 ceph-mon[37563]: 2.16 scrub ok 2024-03-11T21:48:34.120 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:48:33 smithi092 ceph-mon[37563]: from='client.24469 v1:172.21.15.97:0/404208310' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:48:34.120 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:48:33 smithi092 ceph-mon[37563]: 6.6 scrub starts 2024-03-11T21:48:34.120 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:48:33 smithi092 ceph-mon[37563]: 6.6 scrub ok 2024-03-11T21:48:34.131 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:48:34.156 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:48:34.157 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:48:34.157 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:48:34.258 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:48:34.278 DEBUG:teuthology.orchestra.run.smithi092:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid 00db0016-dfed-11ee-95c8-87774f69a715 -- ceph daemon osd.2 dump_historic_ops 2024-03-11T21:48:34.283 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:48:34.284 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:48:34.285 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:48:34.302 INFO:teuthology.orchestra.run.smithi092.stderr:Inferring config /var/lib/ceph/00db0016-dfed-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T21:48:34.335 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:48:33 smithi097 ceph-mon[33317]: 2.16 scrub starts 2024-03-11T21:48:34.335 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:48:33 smithi097 ceph-mon[33317]: 2.16 scrub ok 2024-03-11T21:48:34.335 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:48:33 smithi097 ceph-mon[33317]: from='client.24469 v1:172.21.15.97:0/404208310' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:48:34.335 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:48:33 smithi097 ceph-mon[33317]: 6.6 scrub starts 2024-03-11T21:48:34.335 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:48:33 smithi097 ceph-mon[33317]: 6.6 scrub ok 2024-03-11T21:48:34.386 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:48:34.410 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:48:34.411 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:48:34.411 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:48:34.512 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:48:34.538 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:48:34.539 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:48:34.539 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:48:34.640 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:48:34.664 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:48:34.664 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:48:34.665 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:48:34.765 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:48:34.774 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 6, 7, 5] out_osds: [3, 4] dead_osds: [] live_osds: [0, 1, 2, 4, 5, 6, 7, 3] 2024-03-11T21:48:34.774 DEBUG:teuthology.orchestra.run.smithi092:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid 00db0016-dfed-11ee-95c8-87774f69a715 -- ceph osd reweight 0 1 2024-03-11T21:48:34.789 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:48:34.790 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:48:34.790 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:48:34.827 INFO:teuthology.orchestra.run.smithi092.stderr:Inferring config /var/lib/ceph/00db0016-dfed-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T21:48:34.874 INFO:teuthology.orchestra.run.smithi092.stderr:Inferring config /var/lib/ceph/00db0016-dfed-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T21:48:34.891 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:48:34.918 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:48:34.919 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:48:34.919 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:48:35.020 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:48:35.043 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:48:35.044 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:48:35.044 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:48:35.103 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:48:34 smithi092 ceph-mon[29572]: pgmap v1088: 148 pgs: 1 active+clean+scrubbing, 1 activating+remapped, 6 peering, 140 active+clean; 582 KiB data, 5.2 GiB used, 531 GiB / 536 GiB avail; 0 B/s wr, 22 op/s; 390 B/s, 11 objects/s recovering 2024-03-11T21:48:35.104 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:48:34 smithi092 ceph-mon[29572]: 2.12 deep-scrub starts 2024-03-11T21:48:35.104 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:48:34 smithi092 ceph-mon[29572]: 2.12 deep-scrub ok 2024-03-11T21:48:35.104 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:48:34 smithi092 ceph-mon[29572]: 6.2 scrub starts 2024-03-11T21:48:35.104 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:48:34 smithi092 ceph-mon[29572]: 6.2 scrub ok 2024-03-11T21:48:35.104 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:48:34 smithi092 ceph-mon[29572]: 11.0 scrub starts 2024-03-11T21:48:35.104 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:48:34 smithi092 ceph-mon[29572]: 11.0 scrub ok 2024-03-11T21:48:35.104 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:48:34 smithi092 ceph-mon[37563]: pgmap v1088: 148 pgs: 1 active+clean+scrubbing, 1 activating+remapped, 6 peering, 140 active+clean; 582 KiB data, 5.2 GiB used, 531 GiB / 536 GiB avail; 0 B/s wr, 22 op/s; 390 B/s, 11 objects/s recovering 2024-03-11T21:48:35.104 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:48:34 smithi092 ceph-mon[37563]: 2.12 deep-scrub starts 2024-03-11T21:48:35.105 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:48:34 smithi092 ceph-mon[37563]: 2.12 deep-scrub ok 2024-03-11T21:48:35.105 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:48:34 smithi092 ceph-mon[37563]: 6.2 scrub starts 2024-03-11T21:48:35.105 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:48:34 smithi092 ceph-mon[37563]: 6.2 scrub ok 2024-03-11T21:48:35.105 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:48:34 smithi092 ceph-mon[37563]: 11.0 scrub starts 2024-03-11T21:48:35.105 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:48:34 smithi092 ceph-mon[37563]: 11.0 scrub ok 2024-03-11T21:48:35.145 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:48:35.169 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:48:35.170 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:48:35.170 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:48:35.271 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:48:35.298 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:48:35.299 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:48:35.299 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:48:35.335 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:48:34 smithi097 ceph-mon[33317]: pgmap v1088: 148 pgs: 1 active+clean+scrubbing, 1 activating+remapped, 6 peering, 140 active+clean; 582 KiB data, 5.2 GiB used, 531 GiB / 536 GiB avail; 0 B/s wr, 22 op/s; 390 B/s, 11 objects/s recovering 2024-03-11T21:48:35.335 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:48:34 smithi097 ceph-mon[33317]: 2.12 deep-scrub starts 2024-03-11T21:48:35.335 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:48:34 smithi097 ceph-mon[33317]: 2.12 deep-scrub ok 2024-03-11T21:48:35.335 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:48:34 smithi097 ceph-mon[33317]: 6.2 scrub starts 2024-03-11T21:48:35.335 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:48:34 smithi097 ceph-mon[33317]: 6.2 scrub ok 2024-03-11T21:48:35.335 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:48:34 smithi097 ceph-mon[33317]: 11.0 scrub starts 2024-03-11T21:48:35.335 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:48:34 smithi097 ceph-mon[33317]: 11.0 scrub ok 2024-03-11T21:48:35.400 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:48:35.425 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:48:35.425 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:48:35.425 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:48:35.526 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:48:35.554 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:48:35.554 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:48:35.555 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:48:35.656 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:48:35.681 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:48:35.681 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:48:35.681 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:48:35.782 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:48:35.806 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:48:35.806 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:48:35.807 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:48:35.888 INFO:teuthology.orchestra.run.smithi092.stderr:pool 'unique_pool_4' removed 2024-03-11T21:48:35.908 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:48:35.966 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:48:35.967 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:48:35.967 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:48:36.068 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:48:36.094 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:48:36.095 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:48:36.095 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:48:36.196 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:48:36.220 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:48:36.221 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:48:36.221 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:48:36.322 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:48:36.335 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:48:35 smithi097 ceph-mon[33317]: from='client.? v1:172.21.15.92:0/796260578' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "unique_pool_4", "pool2": "unique_pool_4", "yes_i_really_really_mean_it": true}]: dispatch 2024-03-11T21:48:36.346 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:48:36.346 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:48:36.347 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:48:36.366 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:48:35 smithi092 ceph-mon[29572]: from='client.? v1:172.21.15.92:0/796260578' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "unique_pool_4", "pool2": "unique_pool_4", "yes_i_really_really_mean_it": true}]: dispatch 2024-03-11T21:48:36.366 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:48:35 smithi092 ceph-mon[37563]: from='client.? v1:172.21.15.92:0/796260578' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "unique_pool_4", "pool2": "unique_pool_4", "yes_i_really_really_mean_it": true}]: dispatch 2024-03-11T21:48:36.449 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:48:36.473 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:48:36.474 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:48:36.474 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:48:36.575 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7 2024-03-11T21:48:36.598 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:48:36.599 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:48:36.599 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.7' 2024-03-11T21:48:36.700 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:48:36.725 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:48:36.725 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:48:36.725 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:48:36.826 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:48:36.850 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:48:36.851 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:48:36.851 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:48:36.952 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6 2024-03-11T21:48:36.976 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:48:36.976 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:48:36.976 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.6' 2024-03-11T21:48:37.077 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:48:37.101 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:48:37.102 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:48:37.102 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:48:37.119 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:48:36 smithi092 ceph-mon[29572]: pgmap v1089: 148 pgs: 1 active+clean+scrubbing, 147 active+clean; 582 KiB data, 5.2 GiB used, 531 GiB / 536 GiB avail; 853 B/s rd, 0 B/s wr, 7 op/s; 247 B/s, 8 objects/s recovering 2024-03-11T21:48:37.119 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:48:36 smithi092 ceph-mon[29572]: 5.17 deep-scrub starts 2024-03-11T21:48:37.119 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:48:36 smithi092 ceph-mon[29572]: 5.17 deep-scrub ok 2024-03-11T21:48:37.119 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:48:36 smithi092 ceph-mon[29572]: from='client.? v1:172.21.15.92:0/796260578' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "unique_pool_4", "pool2": "unique_pool_4", "yes_i_really_really_mean_it": true}]': finished 2024-03-11T21:48:37.119 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:48:36 smithi092 ceph-mon[29572]: osdmap e507: 8 total, 8 up, 6 in 2024-03-11T21:48:37.119 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:48:36 smithi092 ceph-mon[29572]: 4.1 deep-scrub starts 2024-03-11T21:48:37.119 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:48:36 smithi092 ceph-mon[29572]: 4.1 deep-scrub ok 2024-03-11T21:48:37.119 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:48:36 smithi092 ceph-mon[29572]: 6.12 scrub starts 2024-03-11T21:48:37.120 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:48:36 smithi092 ceph-mon[29572]: 6.12 scrub ok 2024-03-11T21:48:37.120 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:48:36 smithi092 ceph-mon[29572]: 5.1c scrub starts 2024-03-11T21:48:37.120 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:48:36 smithi092 ceph-mon[37563]: pgmap v1089: 148 pgs: 1 active+clean+scrubbing, 147 active+clean; 582 KiB data, 5.2 GiB used, 531 GiB / 536 GiB avail; 853 B/s rd, 0 B/s wr, 7 op/s; 247 B/s, 8 objects/s recovering 2024-03-11T21:48:37.120 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:48:36 smithi092 ceph-mon[37563]: 5.17 deep-scrub starts 2024-03-11T21:48:37.120 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:48:36 smithi092 ceph-mon[37563]: 5.17 deep-scrub ok 2024-03-11T21:48:37.120 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:48:36 smithi092 ceph-mon[37563]: from='client.? v1:172.21.15.92:0/796260578' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "unique_pool_4", "pool2": "unique_pool_4", "yes_i_really_really_mean_it": true}]': finished 2024-03-11T21:48:37.120 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:48:36 smithi092 ceph-mon[37563]: osdmap e507: 8 total, 8 up, 6 in 2024-03-11T21:48:37.120 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:48:36 smithi092 ceph-mon[37563]: 4.1 deep-scrub starts 2024-03-11T21:48:37.120 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:48:36 smithi092 ceph-mon[37563]: 4.1 deep-scrub ok 2024-03-11T21:48:37.120 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:48:36 smithi092 ceph-mon[37563]: 6.12 scrub starts 2024-03-11T21:48:37.120 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:48:36 smithi092 ceph-mon[37563]: 6.12 scrub ok 2024-03-11T21:48:37.120 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:48:36 smithi092 ceph-mon[37563]: 5.1c scrub starts 2024-03-11T21:48:37.203 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5 2024-03-11T21:48:37.226 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:48:37.227 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:48:37.227 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.5' 2024-03-11T21:48:37.328 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:48:37.335 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:48:36 smithi097 ceph-mon[33317]: pgmap v1089: 148 pgs: 1 active+clean+scrubbing, 147 active+clean; 582 KiB data, 5.2 GiB used, 531 GiB / 536 GiB avail; 853 B/s rd, 0 B/s wr, 7 op/s; 247 B/s, 8 objects/s recovering 2024-03-11T21:48:37.335 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:48:36 smithi097 ceph-mon[33317]: 5.17 deep-scrub starts 2024-03-11T21:48:37.335 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:48:36 smithi097 ceph-mon[33317]: 5.17 deep-scrub ok 2024-03-11T21:48:37.335 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:48:36 smithi097 ceph-mon[33317]: from='client.? v1:172.21.15.92:0/796260578' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "unique_pool_4", "pool2": "unique_pool_4", "yes_i_really_really_mean_it": true}]': finished 2024-03-11T21:48:37.335 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:48:36 smithi097 ceph-mon[33317]: osdmap e507: 8 total, 8 up, 6 in 2024-03-11T21:48:37.335 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:48:36 smithi097 ceph-mon[33317]: 4.1 deep-scrub starts 2024-03-11T21:48:37.335 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:48:36 smithi097 ceph-mon[33317]: 4.1 deep-scrub ok 2024-03-11T21:48:37.335 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:48:36 smithi097 ceph-mon[33317]: 6.12 scrub starts 2024-03-11T21:48:37.335 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:48:36 smithi097 ceph-mon[33317]: 6.12 scrub ok 2024-03-11T21:48:37.335 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:48:36 smithi097 ceph-mon[33317]: 5.1c scrub starts 2024-03-11T21:48:37.351 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:48:37.352 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:48:37.352 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:48:37.453 DEBUG:teuthology.orchestra.run.smithi097:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4 2024-03-11T21:48:37.476 INFO:teuthology.orchestra.run.smithi097.stderr:sudo: docker: command not found 2024-03-11T21:48:37.477 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:48:37.477 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi097 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.4' 2024-03-11T21:48:37.578 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3 2024-03-11T21:48:37.602 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:48:37.602 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:48:37.603 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.3' 2024-03-11T21:48:37.703 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1 2024-03-11T21:48:37.727 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:48:37.728 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:48:37.728 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.1' 2024-03-11T21:48:37.829 DEBUG:teuthology.orchestra.run.smithi092:> sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2 2024-03-11T21:48:37.852 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: docker: command not found 2024-03-11T21:48:37.853 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T21:48:37.853 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi092 with status 1: 'sudo docker kill -s 1 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd.2' 2024-03-11T21:48:37.892 DEBUG:teuthology.run_tasks:Unwinding manager thrashosds 2024-03-11T21:48:37.904 INFO:tasks.thrashosds:joining thrashosds 2024-03-11T21:48:38.119 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:48:37 smithi092 ceph-mon[29572]: 5.1c scrub ok 2024-03-11T21:48:38.119 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:48:37 smithi092 ceph-mon[29572]: 2.15 scrub starts 2024-03-11T21:48:38.119 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:48:37 smithi092 ceph-mon[29572]: 2.15 scrub ok 2024-03-11T21:48:38.120 INFO:journalctl@ceph.mgr.y.smithi092.stdout:Mar 11 21:48:37 smithi092 ceph-00db0016-dfed-11ee-95c8-87774f69a715-mgr-y[29801]: ::ffff:172.21.15.97 - - [11/Mar/2024:21:48:37] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-11T21:48:38.120 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:48:37 smithi092 ceph-mon[37563]: 5.1c scrub ok 2024-03-11T21:48:38.120 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:48:37 smithi092 ceph-mon[37563]: 2.15 scrub starts 2024-03-11T21:48:38.120 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:48:37 smithi092 ceph-mon[37563]: 2.15 scrub ok 2024-03-11T21:48:38.336 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:48:37 smithi097 ceph-mon[33317]: 5.1c scrub ok 2024-03-11T21:48:38.349 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:48:37 smithi097 ceph-mon[33317]: 2.15 scrub starts 2024-03-11T21:48:38.349 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:48:37 smithi097 ceph-mon[33317]: 2.15 scrub ok 2024-03-11T21:48:38.788 INFO:teuthology.orchestra.run.smithi092.stderr:Inferring config /var/lib/ceph/00db0016-dfed-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T21:48:39.012 INFO:teuthology.orchestra.run.smithi092.stdout:osd.0: {} 2024-03-11T21:48:39.031 INFO:teuthology.orchestra.run.smithi092.stdout:osd.1: {} 2024-03-11T21:48:39.050 INFO:teuthology.orchestra.run.smithi092.stdout:osd.2: {} 2024-03-11T21:48:39.069 INFO:teuthology.orchestra.run.smithi092.stdout:osd.3: {} 2024-03-11T21:48:39.089 INFO:teuthology.orchestra.run.smithi092.stdout:osd.4: {} 2024-03-11T21:48:39.108 INFO:teuthology.orchestra.run.smithi092.stdout:osd.5: {} 2024-03-11T21:48:39.128 INFO:teuthology.orchestra.run.smithi092.stdout:osd.6: {} 2024-03-11T21:48:39.149 INFO:teuthology.orchestra.run.smithi092.stdout:osd.7: {} 2024-03-11T21:48:39.149 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:48:38 smithi092 ceph-mon[29572]: pgmap v1091: 132 pgs: 1 active+clean+scrubbing, 131 active+clean; 582 KiB data, 5.2 GiB used, 531 GiB / 536 GiB avail; 614 B/s rd, 0 op/s 2024-03-11T21:48:39.149 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:48:38 smithi092 ceph-mon[29572]: 3.1 scrub starts 2024-03-11T21:48:39.150 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:48:38 smithi092 ceph-mon[29572]: 3.1 scrub ok 2024-03-11T21:48:39.150 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:48:38 smithi092 ceph-mon[29572]: 2.18 scrub starts 2024-03-11T21:48:39.150 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:48:38 smithi092 ceph-mon[29572]: 2.18 scrub ok 2024-03-11T21:48:39.150 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:48:38 smithi092 ceph-mon[29572]: from='mgr.14607 ' entity='mgr.y' 2024-03-11T21:48:39.150 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:48:38 smithi092 ceph-mon[29572]: from='mgr.14607 v1:172.21.15.92:0/1702717597' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T21:48:39.150 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:48:38 smithi092 ceph-mon[37563]: pgmap v1091: 132 pgs: 1 active+clean+scrubbing, 131 active+clean; 582 KiB data, 5.2 GiB used, 531 GiB / 536 GiB avail; 614 B/s rd, 0 op/s 2024-03-11T21:48:39.150 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:48:38 smithi092 ceph-mon[37563]: 3.1 scrub starts 2024-03-11T21:48:39.150 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:48:38 smithi092 ceph-mon[37563]: 3.1 scrub ok 2024-03-11T21:48:39.151 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:48:38 smithi092 ceph-mon[37563]: 2.18 scrub starts 2024-03-11T21:48:39.151 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:48:38 smithi092 ceph-mon[37563]: 2.18 scrub ok 2024-03-11T21:48:39.151 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:48:38 smithi092 ceph-mon[37563]: from='mgr.14607 ' entity='mgr.y' 2024-03-11T21:48:39.151 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:48:38 smithi092 ceph-mon[37563]: from='mgr.14607 v1:172.21.15.92:0/1702717597' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T21:48:39.151 INFO:teuthology.orchestra.run.smithi092.stderr:osd.0: osd_enable_op_tracker = '' 2024-03-11T21:48:39.151 INFO:teuthology.orchestra.run.smithi092.stderr:osd.1: osd_enable_op_tracker = '' 2024-03-11T21:48:39.151 INFO:teuthology.orchestra.run.smithi092.stderr:osd.2: osd_enable_op_tracker = '' 2024-03-11T21:48:39.151 INFO:teuthology.orchestra.run.smithi092.stderr:osd.3: osd_enable_op_tracker = '' 2024-03-11T21:48:39.151 INFO:teuthology.orchestra.run.smithi092.stderr:osd.4: osd_enable_op_tracker = '' 2024-03-11T21:48:39.151 INFO:teuthology.orchestra.run.smithi092.stderr:osd.5: osd_enable_op_tracker = '' 2024-03-11T21:48:39.152 INFO:teuthology.orchestra.run.smithi092.stderr:osd.6: osd_enable_op_tracker = '' 2024-03-11T21:48:39.152 INFO:teuthology.orchestra.run.smithi092.stderr:osd.7: osd_enable_op_tracker = '' 2024-03-11T21:48:39.335 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:48:38 smithi097 ceph-mon[33317]: pgmap v1091: 132 pgs: 1 active+clean+scrubbing, 131 active+clean; 582 KiB data, 5.2 GiB used, 531 GiB / 536 GiB avail; 614 B/s rd, 0 op/s 2024-03-11T21:48:39.335 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:48:38 smithi097 ceph-mon[33317]: 3.1 scrub starts 2024-03-11T21:48:39.335 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:48:38 smithi097 ceph-mon[33317]: 3.1 scrub ok 2024-03-11T21:48:39.336 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:48:38 smithi097 ceph-mon[33317]: 2.18 scrub starts 2024-03-11T21:48:39.336 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:48:38 smithi097 ceph-mon[33317]: 2.18 scrub ok 2024-03-11T21:48:39.336 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:48:38 smithi097 ceph-mon[33317]: from='mgr.14607 ' entity='mgr.y' 2024-03-11T21:48:39.336 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:48:38 smithi097 ceph-mon[33317]: from='mgr.14607 v1:172.21.15.92:0/1702717597' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T21:48:40.283 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:48:39 smithi092 ceph-mon[29572]: from='client.? v1:172.21.15.92:0/1765708173' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-11T21:48:40.283 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:48:39 smithi092 ceph-mon[29572]: 2.7 scrub starts 2024-03-11T21:48:40.283 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:48:39 smithi092 ceph-mon[29572]: 2.7 scrub ok 2024-03-11T21:48:40.283 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:48:39 smithi092 ceph-mon[37563]: from='client.? v1:172.21.15.92:0/1765708173' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-11T21:48:40.283 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:48:39 smithi092 ceph-mon[37563]: 2.7 scrub starts 2024-03-11T21:48:40.283 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:48:39 smithi092 ceph-mon[37563]: 2.7 scrub ok 2024-03-11T21:48:40.335 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:48:39 smithi097 ceph-mon[33317]: from='client.? v1:172.21.15.92:0/1765708173' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-03-11T21:48:40.335 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:48:39 smithi097 ceph-mon[33317]: 2.7 scrub starts 2024-03-11T21:48:40.335 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:48:39 smithi097 ceph-mon[33317]: 2.7 scrub ok 2024-03-11T21:48:41.210 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:48:40 smithi092 ceph-mon[29572]: pgmap v1092: 132 pgs: 132 active+clean; 582 KiB data, 5.3 GiB used, 531 GiB / 536 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-03-11T21:48:41.210 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:48:40 smithi092 ceph-mon[29572]: 6.a scrub starts 2024-03-11T21:48:41.210 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:48:40 smithi092 ceph-mon[29572]: 6.a scrub ok 2024-03-11T21:48:41.210 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:48:40 smithi092 ceph-mon[29572]: 4.1e scrub starts 2024-03-11T21:48:41.210 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:48:40 smithi092 ceph-mon[29572]: 4.1e scrub ok 2024-03-11T21:48:41.210 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:48:40 smithi092 ceph-mon[29572]: 5.18 scrub starts 2024-03-11T21:48:41.210 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:48:40 smithi092 ceph-mon[29572]: 5.18 scrub ok 2024-03-11T21:48:41.210 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:48:40 smithi092 ceph-mon[37563]: pgmap v1092: 132 pgs: 132 active+clean; 582 KiB data, 5.3 GiB used, 531 GiB / 536 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-03-11T21:48:41.211 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:48:40 smithi092 ceph-mon[37563]: 6.a scrub starts 2024-03-11T21:48:41.211 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:48:40 smithi092 ceph-mon[37563]: 6.a scrub ok 2024-03-11T21:48:41.211 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:48:40 smithi092 ceph-mon[37563]: 4.1e scrub starts 2024-03-11T21:48:41.211 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:48:40 smithi092 ceph-mon[37563]: 4.1e scrub ok 2024-03-11T21:48:41.211 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:48:40 smithi092 ceph-mon[37563]: 5.18 scrub starts 2024-03-11T21:48:41.211 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:48:40 smithi092 ceph-mon[37563]: 5.18 scrub ok 2024-03-11T21:48:41.335 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:48:40 smithi097 ceph-mon[33317]: pgmap v1092: 132 pgs: 132 active+clean; 582 KiB data, 5.3 GiB used, 531 GiB / 536 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-03-11T21:48:41.335 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:48:40 smithi097 ceph-mon[33317]: 6.a scrub starts 2024-03-11T21:48:41.335 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:48:40 smithi097 ceph-mon[33317]: 6.a scrub ok 2024-03-11T21:48:41.335 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:48:40 smithi097 ceph-mon[33317]: 4.1e scrub starts 2024-03-11T21:48:41.335 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:48:40 smithi097 ceph-mon[33317]: 4.1e scrub ok 2024-03-11T21:48:41.336 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:48:40 smithi097 ceph-mon[33317]: 5.18 scrub starts 2024-03-11T21:48:41.336 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:48:40 smithi097 ceph-mon[33317]: 5.18 scrub ok 2024-03-11T21:48:41.906 INFO:teuthology.orchestra.run.smithi092.stderr:noscrub is set 2024-03-11T21:48:42.326 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:48:41 smithi092 ceph-mon[29572]: 4.1d scrub starts 2024-03-11T21:48:42.326 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:48:41 smithi092 ceph-mon[29572]: 4.1d scrub ok 2024-03-11T21:48:42.326 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:48:41 smithi092 ceph-mon[29572]: 5.7 scrub starts 2024-03-11T21:48:42.326 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:48:41 smithi092 ceph-mon[29572]: 5.7 scrub ok 2024-03-11T21:48:42.326 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:48:41 smithi092 ceph-mon[29572]: from='client.? v1:172.21.15.92:0/1688281807' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-11T21:48:42.326 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:48:41 smithi092 ceph-mon[37563]: 4.1d scrub starts 2024-03-11T21:48:42.327 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:48:41 smithi092 ceph-mon[37563]: 4.1d scrub ok 2024-03-11T21:48:42.327 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:48:41 smithi092 ceph-mon[37563]: 5.7 scrub starts 2024-03-11T21:48:42.327 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:48:41 smithi092 ceph-mon[37563]: 5.7 scrub ok 2024-03-11T21:48:42.327 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:48:41 smithi092 ceph-mon[37563]: from='client.? v1:172.21.15.92:0/1688281807' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-11T21:48:42.335 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:48:41 smithi097 ceph-mon[33317]: 4.1d scrub starts 2024-03-11T21:48:42.335 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:48:41 smithi097 ceph-mon[33317]: 4.1d scrub ok 2024-03-11T21:48:42.335 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:48:41 smithi097 ceph-mon[33317]: 5.7 scrub starts 2024-03-11T21:48:42.335 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:48:41 smithi097 ceph-mon[33317]: 5.7 scrub ok 2024-03-11T21:48:42.335 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:48:41 smithi097 ceph-mon[33317]: from='client.? v1:172.21.15.92:0/1688281807' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-03-11T21:48:42.966 INFO:teuthology.orchestra.run.smithi092.stdout:{ 2024-03-11T21:48:42.966 INFO:teuthology.orchestra.run.smithi092.stdout: "size": 20, 2024-03-11T21:48:42.966 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 600, 2024-03-11T21:48:42.966 INFO:teuthology.orchestra.run.smithi092.stdout: "ops": [ 2024-03-11T21:48:42.966 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:48:42.966 INFO:teuthology.orchestra.run.smithi092.stdout: "description": "osd_op(client.15744.0:69289 10.3 10:c0416669:::benchmark_data_smithi092_161462_object69288:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e372)", 2024-03-11T21:48:42.966 INFO:teuthology.orchestra.run.smithi092.stdout: "initiated_at": "2024-03-11T21:41:27.826535+0000", 2024-03-11T21:48:42.966 INFO:teuthology.orchestra.run.smithi092.stdout: "age": 435.145897827, 2024-03-11T21:48:42.966 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.0328515250000001, 2024-03-11T21:48:42.966 INFO:teuthology.orchestra.run.smithi092.stdout: "type_data": { 2024-03-11T21:48:42.966 INFO:teuthology.orchestra.run.smithi092.stdout: "flag_point": "reached pg", 2024-03-11T21:48:42.967 INFO:teuthology.orchestra.run.smithi092.stdout: "client_info": { 2024-03-11T21:48:42.967 INFO:teuthology.orchestra.run.smithi092.stdout: "client": "client.15744", 2024-03-11T21:48:42.967 INFO:teuthology.orchestra.run.smithi092.stdout: "client_addr": "v1:172.21.15.92:0/3334000812", 2024-03-11T21:48:42.967 INFO:teuthology.orchestra.run.smithi092.stdout: "tid": 69289 2024-03-11T21:48:42.967 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:48:42.967 INFO:teuthology.orchestra.run.smithi092.stdout: "events": [ 2024-03-11T21:48:42.967 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:48:42.967 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "initiated", 2024-03-11T21:48:42.967 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:41:27.826535+0000", 2024-03-11T21:48:42.967 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-03-11T21:48:42.967 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:48:42.967 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:48:42.967 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "header_read", 2024-03-11T21:48:42.968 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:41:27.826535+0000", 2024-03-11T21:48:42.968 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-03-11T21:48:42.968 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:48:42.968 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:48:42.968 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "throttled", 2024-03-11T21:48:42.968 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:41:27.826537+0000", 2024-03-11T21:48:42.968 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 2.4360000000000001e-06 2024-03-11T21:48:42.968 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:48:42.968 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:48:42.968 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "all_read", 2024-03-11T21:48:42.968 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:41:27.826566+0000", 2024-03-11T21:48:42.968 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 2.8459e-05 2024-03-11T21:48:42.968 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:48:42.969 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:48:42.969 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "dispatched", 2024-03-11T21:48:42.969 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:41:27.826580+0000", 2024-03-11T21:48:42.969 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.4318000000000001e-05 2024-03-11T21:48:42.969 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:48:42.969 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:48:42.969 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "queued_for_pg", 2024-03-11T21:48:42.969 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:41:27.826590+0000", 2024-03-11T21:48:42.969 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 9.431e-06 2024-03-11T21:48:42.969 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:48:42.969 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:48:42.969 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-03-11T21:48:42.969 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:41:27.826699+0000", 2024-03-11T21:48:42.969 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00010901899999999999 2024-03-11T21:48:42.969 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:48:42.970 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:48:42.970 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "op must wait for map", 2024-03-11T21:48:42.970 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:41:27.826705+0000", 2024-03-11T21:48:42.970 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 6.4749999999999998e-06 2024-03-11T21:48:42.970 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:48:42.970 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:48:42.970 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-03-11T21:48:42.970 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:41:27.974402+0000", 2024-03-11T21:48:42.970 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.14769684599999999 2024-03-11T21:48:42.970 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:48:42.970 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:48:42.970 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "waiting for peered", 2024-03-11T21:48:42.971 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:41:27.974410+0000", 2024-03-11T21:48:42.971 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 8.5150000000000006e-06 2024-03-11T21:48:42.971 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:48:42.971 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:48:42.971 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-03-11T21:48:42.971 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:41:28.842478+0000", 2024-03-11T21:48:42.971 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.86806724300000004 2024-03-11T21:48:42.971 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:48:42.971 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:48:42.971 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "waiting for degraded object", 2024-03-11T21:48:42.971 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:41:28.843638+0000", 2024-03-11T21:48:42.971 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.001160349 2024-03-11T21:48:42.971 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:48:42.971 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:48:42.972 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-03-11T21:48:42.972 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:41:28.859317+0000", 2024-03-11T21:48:42.972 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.015678791000000001 2024-03-11T21:48:42.972 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:48:42.972 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:48:42.972 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "done", 2024-03-11T21:48:42.972 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:41:28.859386+0000", 2024-03-11T21:48:42.972 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 6.9642999999999997e-05 2024-03-11T21:48:42.972 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-03-11T21:48:42.972 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-03-11T21:48:42.972 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-03-11T21:48:42.972 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:48:42.972 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:48:42.972 INFO:teuthology.orchestra.run.smithi092.stdout: "description": "osd_op(client.15744.0:69293 10.13 10:ce43b4dd:::benchmark_data_smithi092_161462_object69292:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e372)", 2024-03-11T21:48:42.973 INFO:teuthology.orchestra.run.smithi092.stdout: "initiated_at": "2024-03-11T21:41:27.826615+0000", 2024-03-11T21:48:42.973 INFO:teuthology.orchestra.run.smithi092.stdout: "age": 435.14581791400002, 2024-03-11T21:48:42.973 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.0419076169999999, 2024-03-11T21:48:42.973 INFO:teuthology.orchestra.run.smithi092.stdout: "type_data": { 2024-03-11T21:48:42.973 INFO:teuthology.orchestra.run.smithi092.stdout: "flag_point": "reached pg", 2024-03-11T21:48:42.973 INFO:teuthology.orchestra.run.smithi092.stdout: "client_info": { 2024-03-11T21:48:42.973 INFO:teuthology.orchestra.run.smithi092.stdout: "client": "client.15744", 2024-03-11T21:48:42.973 INFO:teuthology.orchestra.run.smithi092.stdout: "client_addr": "v1:172.21.15.92:0/3334000812", 2024-03-11T21:48:42.973 INFO:teuthology.orchestra.run.smithi092.stdout: "tid": 69293 2024-03-11T21:48:42.973 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:48:42.973 INFO:teuthology.orchestra.run.smithi092.stdout: "events": [ 2024-03-11T21:48:42.973 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:48:42.973 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "initiated", 2024-03-11T21:48:42.973 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:41:27.826615+0000", 2024-03-11T21:48:42.973 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-03-11T21:48:42.974 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:48:42.974 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:48:42.974 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "header_read", 2024-03-11T21:48:42.974 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:41:27.826615+0000", 2024-03-11T21:48:42.974 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-03-11T21:48:42.974 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:48:42.974 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:48:42.974 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "throttled", 2024-03-11T21:48:42.974 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:41:27.826616+0000", 2024-03-11T21:48:42.974 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 7.8299999999999996e-07 2024-03-11T21:48:42.974 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:48:42.974 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:48:42.974 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "all_read", 2024-03-11T21:48:42.975 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:41:27.826638+0000", 2024-03-11T21:48:42.975 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 2.2036e-05 2024-03-11T21:48:42.975 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:48:42.975 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:48:42.975 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "dispatched", 2024-03-11T21:48:42.975 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:41:27.826644+0000", 2024-03-11T21:48:42.975 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 6.319e-06 2024-03-11T21:48:42.975 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:48:42.975 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:48:42.975 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "queued_for_pg", 2024-03-11T21:48:42.975 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:41:27.826649+0000", 2024-03-11T21:48:42.975 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 5.2989999999999996e-06 2024-03-11T21:48:42.975 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:48:42.975 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:48:42.975 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-03-11T21:48:42.976 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:41:27.974312+0000", 2024-03-11T21:48:42.976 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.147663085 2024-03-11T21:48:42.976 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:48:42.976 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:48:42.976 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "waiting for peered", 2024-03-11T21:48:42.976 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:41:27.974321+0000", 2024-03-11T21:48:42.976 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 9.0119999999999994e-06 2024-03-11T21:48:42.976 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:48:42.976 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:48:42.976 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-03-11T21:48:42.976 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:41:28.854885+0000", 2024-03-11T21:48:42.976 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.88056314899999999 2024-03-11T21:48:42.976 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:48:42.976 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:48:42.977 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "waiting for degraded object", 2024-03-11T21:48:42.977 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:41:28.859080+0000", 2024-03-11T21:48:42.977 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.0041959090000000003 2024-03-11T21:48:42.977 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:48:42.977 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:48:42.977 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-03-11T21:48:42.977 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:41:28.868404+0000", 2024-03-11T21:48:42.977 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.0093230940000000005 2024-03-11T21:48:42.977 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:48:42.977 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:48:42.977 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "done", 2024-03-11T21:48:42.977 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:41:28.868522+0000", 2024-03-11T21:48:42.977 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.000118931 2024-03-11T21:48:42.977 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-03-11T21:48:42.977 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-03-11T21:48:42.978 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-03-11T21:48:42.978 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:48:42.978 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:48:42.978 INFO:teuthology.orchestra.run.smithi092.stdout: "description": "osd_op(client.15744.0:69298 10.15 10:ac2bc3c5:::benchmark_data_smithi092_161462_object69297:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e372)", 2024-03-11T21:48:42.978 INFO:teuthology.orchestra.run.smithi092.stdout: "initiated_at": "2024-03-11T21:41:27.830985+0000", 2024-03-11T21:48:42.978 INFO:teuthology.orchestra.run.smithi092.stdout: "age": 435.14144773100003, 2024-03-11T21:48:42.978 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.0303282, 2024-03-11T21:48:42.978 INFO:teuthology.orchestra.run.smithi092.stdout: "type_data": { 2024-03-11T21:48:42.978 INFO:teuthology.orchestra.run.smithi092.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-11T21:48:42.978 INFO:teuthology.orchestra.run.smithi092.stdout: "client_info": { 2024-03-11T21:48:42.978 INFO:teuthology.orchestra.run.smithi092.stdout: "client": "client.15744", 2024-03-11T21:48:42.978 INFO:teuthology.orchestra.run.smithi092.stdout: "client_addr": "v1:172.21.15.92:0/3334000812", 2024-03-11T21:48:42.978 INFO:teuthology.orchestra.run.smithi092.stdout: "tid": 69298 2024-03-11T21:48:42.979 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:48:42.979 INFO:teuthology.orchestra.run.smithi092.stdout: "events": [ 2024-03-11T21:48:42.979 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:48:42.979 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "initiated", 2024-03-11T21:48:42.979 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:41:27.830985+0000", 2024-03-11T21:48:42.979 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-03-11T21:48:42.979 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:48:42.979 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:48:42.979 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "header_read", 2024-03-11T21:48:42.979 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:41:27.830985+0000", 2024-03-11T21:48:42.979 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-03-11T21:48:42.979 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:48:42.979 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:48:42.979 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "throttled", 2024-03-11T21:48:42.980 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:41:27.830990+0000", 2024-03-11T21:48:42.980 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 4.5800000000000002e-06 2024-03-11T21:48:42.980 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:48:42.980 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:48:42.980 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "all_read", 2024-03-11T21:48:42.980 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:41:27.831036+0000", 2024-03-11T21:48:42.980 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 4.5918000000000003e-05 2024-03-11T21:48:42.980 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:48:42.980 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:48:42.980 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "dispatched", 2024-03-11T21:48:42.980 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:41:27.831069+0000", 2024-03-11T21:48:42.980 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 3.2999e-05 2024-03-11T21:48:42.980 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:48:42.980 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:48:42.980 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "queued_for_pg", 2024-03-11T21:48:42.980 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:41:27.831084+0000", 2024-03-11T21:48:42.981 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.5580999999999999e-05 2024-03-11T21:48:42.981 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:48:42.981 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:48:42.981 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-03-11T21:48:42.981 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:41:27.985517+0000", 2024-03-11T21:48:42.981 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.15443337700000001 2024-03-11T21:48:42.981 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:48:42.981 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:48:42.981 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "waiting for peered", 2024-03-11T21:48:42.981 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:41:27.985528+0000", 2024-03-11T21:48:42.981 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.0657e-05 2024-03-11T21:48:42.981 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:48:42.981 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:48:42.981 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-03-11T21:48:42.981 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:41:28.855828+0000", 2024-03-11T21:48:42.982 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.87029995999999998 2024-03-11T21:48:42.982 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:48:42.982 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:48:42.982 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "started", 2024-03-11T21:48:42.982 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:41:28.855954+0000", 2024-03-11T21:48:42.982 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.000126205 2024-03-11T21:48:42.982 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:48:42.982 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:48:42.982 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "waiting for subops from 0,3", 2024-03-11T21:48:42.982 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:41:28.856140+0000", 2024-03-11T21:48:42.982 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00018534399999999999 2024-03-11T21:48:42.982 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:48:42.982 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:48:42.982 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "sub_op_commit_rec", 2024-03-11T21:48:42.983 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:41:28.861136+0000", 2024-03-11T21:48:42.983 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.0049965729999999998 2024-03-11T21:48:42.983 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:48:42.983 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:48:42.983 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "op_commit", 2024-03-11T21:48:42.983 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:41:28.861148+0000", 2024-03-11T21:48:42.983 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.1793e-05 2024-03-11T21:48:42.983 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:48:42.983 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:48:42.983 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "sub_op_commit_rec", 2024-03-11T21:48:42.983 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:41:28.861222+0000", 2024-03-11T21:48:42.983 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 7.4363999999999995e-05 2024-03-11T21:48:42.983 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:48:42.983 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:48:42.983 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "commit_sent", 2024-03-11T21:48:42.984 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:41:28.861280+0000", 2024-03-11T21:48:42.984 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 5.7598e-05 2024-03-11T21:48:42.984 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:48:42.984 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:48:42.984 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "done", 2024-03-11T21:48:42.984 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:41:28.861313+0000", 2024-03-11T21:48:42.984 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 3.3250999999999997e-05 2024-03-11T21:48:42.984 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-03-11T21:48:42.984 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-03-11T21:48:42.984 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-03-11T21:48:42.984 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:48:42.984 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:48:42.984 INFO:teuthology.orchestra.run.smithi092.stdout: "description": "osd_op(client.15744.0:69299 10.15 10:a8a48bf0:::benchmark_data_smithi092_161462_object69298:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e372)", 2024-03-11T21:48:42.984 INFO:teuthology.orchestra.run.smithi092.stdout: "initiated_at": "2024-03-11T21:41:27.861590+0000", 2024-03-11T21:48:42.984 INFO:teuthology.orchestra.run.smithi092.stdout: "age": 435.11084299800001, 2024-03-11T21:48:42.985 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.0039606750000001, 2024-03-11T21:48:42.985 INFO:teuthology.orchestra.run.smithi092.stdout: "type_data": { 2024-03-11T21:48:42.985 INFO:teuthology.orchestra.run.smithi092.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-11T21:48:42.985 INFO:teuthology.orchestra.run.smithi092.stdout: "client_info": { 2024-03-11T21:48:42.985 INFO:teuthology.orchestra.run.smithi092.stdout: "client": "client.15744", 2024-03-11T21:48:42.985 INFO:teuthology.orchestra.run.smithi092.stdout: "client_addr": "v1:172.21.15.92:0/3334000812", 2024-03-11T21:48:42.985 INFO:teuthology.orchestra.run.smithi092.stdout: "tid": 69299 2024-03-11T21:48:42.985 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:48:42.985 INFO:teuthology.orchestra.run.smithi092.stdout: "events": [ 2024-03-11T21:48:42.985 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:48:42.985 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "initiated", 2024-03-11T21:48:42.985 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:41:27.861590+0000", 2024-03-11T21:48:42.985 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-03-11T21:48:42.985 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:48:42.985 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:48:42.986 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "header_read", 2024-03-11T21:48:42.986 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:41:27.861590+0000", 2024-03-11T21:48:42.986 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-03-11T21:48:42.986 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:48:42.986 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:48:42.986 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "throttled", 2024-03-11T21:48:42.986 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:41:27.861594+0000", 2024-03-11T21:48:42.986 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 4.3449999999999999e-06 2024-03-11T21:48:42.986 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:48:42.986 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:48:42.989 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "all_read", 2024-03-11T21:48:42.989 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:41:27.861638+0000", 2024-03-11T21:48:42.989 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 4.3823999999999999e-05 2024-03-11T21:48:42.989 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:48:42.989 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:48:42.990 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "dispatched", 2024-03-11T21:48:42.990 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:41:27.861664+0000", 2024-03-11T21:48:42.990 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 2.5834999999999999e-05 2024-03-11T21:48:42.990 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:48:42.990 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:48:42.990 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "queued_for_pg", 2024-03-11T21:48:42.990 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:41:27.861679+0000", 2024-03-11T21:48:42.990 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.5226e-05 2024-03-11T21:48:42.990 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:48:42.990 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:48:42.990 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-03-11T21:48:42.990 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:41:27.985549+0000", 2024-03-11T21:48:42.990 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.12387039699999999 2024-03-11T21:48:42.990 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:48:42.991 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:48:42.991 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "waiting for peered", 2024-03-11T21:48:42.991 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:41:27.985555+0000", 2024-03-11T21:48:42.991 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 5.609e-06 2024-03-11T21:48:42.991 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:48:42.991 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:48:42.991 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-03-11T21:48:42.991 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:41:28.858071+0000", 2024-03-11T21:48:42.991 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.872516336 2024-03-11T21:48:42.991 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:48:42.991 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:48:42.991 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "started", 2024-03-11T21:48:42.991 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:41:28.858205+0000", 2024-03-11T21:48:42.991 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00013365799999999999 2024-03-11T21:48:42.991 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:48:42.992 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:48:42.992 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "waiting for subops from 0,3", 2024-03-11T21:48:42.992 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:41:28.858344+0000", 2024-03-11T21:48:42.992 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.000139288 2024-03-11T21:48:42.992 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:48:42.992 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:48:42.992 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "op_commit", 2024-03-11T21:48:42.992 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:41:28.861191+0000", 2024-03-11T21:48:42.992 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.0028463249999999998 2024-03-11T21:48:42.992 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:48:42.992 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:48:42.992 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "sub_op_commit_rec", 2024-03-11T21:48:42.992 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:41:28.861351+0000", 2024-03-11T21:48:42.992 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00016006000000000001 2024-03-11T21:48:42.992 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:48:42.993 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:48:42.993 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "sub_op_commit_rec", 2024-03-11T21:48:42.993 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:41:28.865409+0000", 2024-03-11T21:48:42.993 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.004058842 2024-03-11T21:48:42.993 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:48:42.993 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:48:42.993 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "commit_sent", 2024-03-11T21:48:42.993 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:41:28.865480+0000", 2024-03-11T21:48:42.993 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 7.0081999999999997e-05 2024-03-11T21:48:42.993 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:48:42.993 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:48:42.993 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "done", 2024-03-11T21:48:42.993 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:41:28.865550+0000", 2024-03-11T21:48:42.993 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 7.0847999999999997e-05 2024-03-11T21:48:42.993 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-03-11T21:48:42.994 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-03-11T21:48:42.994 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-03-11T21:48:42.994 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:48:42.994 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:48:42.994 INFO:teuthology.orchestra.run.smithi092.stdout: "description": "osd_op(client.15744.0:214892 10.13 10:ca8ceba7:::benchmark_data_smithi092_161462_object104916:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e431)", 2024-03-11T21:48:42.994 INFO:teuthology.orchestra.run.smithi092.stdout: "initiated_at": "2024-03-11T21:44:20.426255+0000", 2024-03-11T21:48:42.994 INFO:teuthology.orchestra.run.smithi092.stdout: "age": 262.54617740100002, 2024-03-11T21:48:42.994 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 2.158539749, 2024-03-11T21:48:42.994 INFO:teuthology.orchestra.run.smithi092.stdout: "type_data": { 2024-03-11T21:48:42.994 INFO:teuthology.orchestra.run.smithi092.stdout: "flag_point": "reached pg", 2024-03-11T21:48:42.994 INFO:teuthology.orchestra.run.smithi092.stdout: "client_info": { 2024-03-11T21:48:42.994 INFO:teuthology.orchestra.run.smithi092.stdout: "client": "client.15744", 2024-03-11T21:48:42.994 INFO:teuthology.orchestra.run.smithi092.stdout: "client_addr": "v1:172.21.15.92:0/3334000812", 2024-03-11T21:48:42.995 INFO:teuthology.orchestra.run.smithi092.stdout: "tid": 214892 2024-03-11T21:48:42.995 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:48:42.995 INFO:teuthology.orchestra.run.smithi092.stdout: "events": [ 2024-03-11T21:48:42.995 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:48:42.995 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "initiated", 2024-03-11T21:48:42.995 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:44:20.426255+0000", 2024-03-11T21:48:42.995 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-03-11T21:48:42.995 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:48:42.995 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:48:42.996 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "header_read", 2024-03-11T21:48:42.996 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:44:20.426255+0000", 2024-03-11T21:48:42.996 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-03-11T21:48:42.996 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:48:42.996 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:48:42.996 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "throttled", 2024-03-11T21:48:42.996 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:44:20.426257+0000", 2024-03-11T21:48:42.997 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.872e-06 2024-03-11T21:48:42.997 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:48:42.997 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:48:42.997 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "all_read", 2024-03-11T21:48:42.997 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:44:20.426263+0000", 2024-03-11T21:48:42.997 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 5.7640000000000002e-06 2024-03-11T21:48:42.997 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:48:42.997 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:48:42.997 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "dispatched", 2024-03-11T21:48:42.997 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:44:20.426273+0000", 2024-03-11T21:48:42.997 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 9.5789999999999993e-06 2024-03-11T21:48:42.997 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:48:42.997 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:48:42.997 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "queued_for_pg", 2024-03-11T21:48:42.998 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:44:20.426280+0000", 2024-03-11T21:48:42.998 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 7.8129999999999996e-06 2024-03-11T21:48:42.998 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:48:42.998 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:48:42.998 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-03-11T21:48:42.998 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:44:20.426467+0000", 2024-03-11T21:48:42.998 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00018612899999999999 2024-03-11T21:48:42.999 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:48:42.999 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:48:42.999 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "started", 2024-03-11T21:48:42.999 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:44:20.426595+0000", 2024-03-11T21:48:42.999 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.000128045 2024-03-11T21:48:42.999 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:48:42.999 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:48:42.999 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "waiting for subops from 3,5", 2024-03-11T21:48:43.000 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:44:20.427130+0000", 2024-03-11T21:48:43.000 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00053500599999999998 2024-03-11T21:48:43.000 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:48:43.000 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:48:43.000 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "op_commit", 2024-03-11T21:48:43.000 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:44:20.434836+0000", 2024-03-11T21:48:43.001 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.0077061029999999997 2024-03-11T21:48:43.001 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:48:43.001 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:48:43.001 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-03-11T21:48:43.001 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:44:20.483586+0000", 2024-03-11T21:48:43.001 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.048750244999999998 2024-03-11T21:48:43.001 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:48:43.002 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:48:43.002 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "waiting for peered", 2024-03-11T21:48:43.002 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:44:20.483597+0000", 2024-03-11T21:48:43.002 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.0981999999999999e-05 2024-03-11T21:48:43.002 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:48:43.002 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:48:43.002 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-03-11T21:48:43.002 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:44:21.445458+0000", 2024-03-11T21:48:43.003 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.96186076399999998 2024-03-11T21:48:43.003 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:48:43.003 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:48:43.003 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "waiting for peered", 2024-03-11T21:48:43.003 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:44:21.445470+0000", 2024-03-11T21:48:43.003 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.2125e-05 2024-03-11T21:48:43.003 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:48:43.004 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:48:43.004 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-03-11T21:48:43.004 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:44:22.584706+0000", 2024-03-11T21:48:43.004 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.139236014 2024-03-11T21:48:43.004 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:48:43.004 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:48:43.004 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "done", 2024-03-11T21:48:43.004 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:44:22.584795+0000", 2024-03-11T21:48:43.004 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 8.9308000000000006e-05 2024-03-11T21:48:43.005 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-03-11T21:48:43.005 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-03-11T21:48:43.005 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-03-11T21:48:43.005 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:48:43.005 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:48:43.005 INFO:teuthology.orchestra.run.smithi092.stdout: "description": "osd_op(client.15744.0:214895 10.13 10:cde75749:::benchmark_data_smithi092_161462_object104919:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e431)", 2024-03-11T21:48:43.006 INFO:teuthology.orchestra.run.smithi092.stdout: "initiated_at": "2024-03-11T21:44:20.436109+0000", 2024-03-11T21:48:43.006 INFO:teuthology.orchestra.run.smithi092.stdout: "age": 262.53632349499998, 2024-03-11T21:48:43.006 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 2.1941060210000001, 2024-03-11T21:48:43.006 INFO:teuthology.orchestra.run.smithi092.stdout: "type_data": { 2024-03-11T21:48:43.006 INFO:teuthology.orchestra.run.smithi092.stdout: "flag_point": "reached pg", 2024-03-11T21:48:43.006 INFO:teuthology.orchestra.run.smithi092.stdout: "client_info": { 2024-03-11T21:48:43.006 INFO:teuthology.orchestra.run.smithi092.stdout: "client": "client.15744", 2024-03-11T21:48:43.007 INFO:teuthology.orchestra.run.smithi092.stdout: "client_addr": "v1:172.21.15.92:0/3334000812", 2024-03-11T21:48:43.007 INFO:teuthology.orchestra.run.smithi092.stdout: "tid": 214895 2024-03-11T21:48:43.007 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:48:43.007 INFO:teuthology.orchestra.run.smithi092.stdout: "events": [ 2024-03-11T21:48:43.007 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:48:43.007 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "initiated", 2024-03-11T21:48:43.008 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:44:20.436109+0000", 2024-03-11T21:48:43.008 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-03-11T21:48:43.008 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:48:43.008 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:48:43.008 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "header_read", 2024-03-11T21:48:43.008 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:44:20.436109+0000", 2024-03-11T21:48:43.008 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-03-11T21:48:43.008 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:48:43.008 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:48:43.009 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "throttled", 2024-03-11T21:48:43.009 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:44:20.436112+0000", 2024-03-11T21:48:43.009 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 2.8200000000000001e-06 2024-03-11T21:48:43.009 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:48:43.009 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:48:43.009 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "all_read", 2024-03-11T21:48:43.009 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:44:20.436120+0000", 2024-03-11T21:48:43.009 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 7.8059999999999995e-06 2024-03-11T21:48:43.009 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:48:43.009 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:48:43.009 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "dispatched", 2024-03-11T21:48:43.009 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:44:20.436139+0000", 2024-03-11T21:48:43.010 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.9215000000000002e-05 2024-03-11T21:48:43.010 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:48:43.010 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:48:43.010 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "queued_for_pg", 2024-03-11T21:48:43.010 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:44:20.436151+0000", 2024-03-11T21:48:43.010 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.1447000000000001e-05 2024-03-11T21:48:43.010 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:48:43.010 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:48:43.010 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-03-11T21:48:43.010 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:44:20.436231+0000", 2024-03-11T21:48:43.011 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 8.0595999999999999e-05 2024-03-11T21:48:43.011 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:48:43.011 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:48:43.011 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "started", 2024-03-11T21:48:43.011 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:44:20.436385+0000", 2024-03-11T21:48:43.011 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00015394900000000001 2024-03-11T21:48:43.011 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:48:43.011 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:48:43.011 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "waiting for subops from 3,5", 2024-03-11T21:48:43.011 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:44:20.436892+0000", 2024-03-11T21:48:43.011 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00050714800000000002 2024-03-11T21:48:43.011 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:48:43.011 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:48:43.011 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-03-11T21:48:43.011 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:44:20.483627+0000", 2024-03-11T21:48:43.012 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.046734477000000003 2024-03-11T21:48:43.012 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:48:43.012 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:48:43.012 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "waiting for peered", 2024-03-11T21:48:43.012 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:44:20.483636+0000", 2024-03-11T21:48:43.012 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 8.8529999999999998e-06 2024-03-11T21:48:43.012 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:48:43.012 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:48:43.012 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-03-11T21:48:43.012 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:44:21.445503+0000", 2024-03-11T21:48:43.012 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.96186729000000004 2024-03-11T21:48:43.012 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:48:43.012 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:48:43.012 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "waiting for peered", 2024-03-11T21:48:43.012 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:44:21.445512+0000", 2024-03-11T21:48:43.013 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 9.0450000000000006e-06 2024-03-11T21:48:43.013 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:48:43.013 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:48:43.013 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-03-11T21:48:43.013 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:44:22.585104+0000", 2024-03-11T21:48:43.013 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.139591609 2024-03-11T21:48:43.013 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:48:43.013 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:48:43.013 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "waiting for missing object", 2024-03-11T21:48:43.013 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:44:22.585210+0000", 2024-03-11T21:48:43.013 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.000106548 2024-03-11T21:48:43.013 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:48:43.013 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:48:43.013 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-03-11T21:48:43.013 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:44:22.630115+0000", 2024-03-11T21:48:43.014 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.044904804999999999 2024-03-11T21:48:43.014 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:48:43.014 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:48:43.014 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "done", 2024-03-11T21:48:43.014 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:44:22.630215+0000", 2024-03-11T21:48:43.014 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.000100413 2024-03-11T21:48:43.014 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-03-11T21:48:43.014 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-03-11T21:48:43.014 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-03-11T21:48:43.014 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:48:43.014 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:48:43.014 INFO:teuthology.orchestra.run.smithi092.stdout: "description": "osd_op(client.15744.0:214897 10.13 10:cf3a0af0:::benchmark_data_smithi092_161462_object104921:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e431)", 2024-03-11T21:48:43.015 INFO:teuthology.orchestra.run.smithi092.stdout: "initiated_at": "2024-03-11T21:44:20.436171+0000", 2024-03-11T21:48:43.015 INFO:teuthology.orchestra.run.smithi092.stdout: "age": 262.53626164899998, 2024-03-11T21:48:43.015 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 2.1953438419999998, 2024-03-11T21:48:43.015 INFO:teuthology.orchestra.run.smithi092.stdout: "type_data": { 2024-03-11T21:48:43.015 INFO:teuthology.orchestra.run.smithi092.stdout: "flag_point": "reached pg", 2024-03-11T21:48:43.015 INFO:teuthology.orchestra.run.smithi092.stdout: "client_info": { 2024-03-11T21:48:43.015 INFO:teuthology.orchestra.run.smithi092.stdout: "client": "client.15744", 2024-03-11T21:48:43.015 INFO:teuthology.orchestra.run.smithi092.stdout: "client_addr": "v1:172.21.15.92:0/3334000812", 2024-03-11T21:48:43.015 INFO:teuthology.orchestra.run.smithi092.stdout: "tid": 214897 2024-03-11T21:48:43.015 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:48:43.015 INFO:teuthology.orchestra.run.smithi092.stdout: "events": [ 2024-03-11T21:48:43.015 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:48:43.015 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "initiated", 2024-03-11T21:48:43.015 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:44:20.436171+0000", 2024-03-11T21:48:43.015 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-03-11T21:48:43.016 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:48:43.016 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:48:43.016 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "header_read", 2024-03-11T21:48:43.016 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:44:20.436171+0000", 2024-03-11T21:48:43.016 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-03-11T21:48:43.016 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:48:43.016 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:48:43.016 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "throttled", 2024-03-11T21:48:43.016 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:44:20.436172+0000", 2024-03-11T21:48:43.016 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 7.4099999999999998e-07 2024-03-11T21:48:43.016 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:48:43.016 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:48:43.016 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "all_read", 2024-03-11T21:48:43.016 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:44:20.436176+0000", 2024-03-11T21:48:43.017 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 3.7390000000000001e-06 2024-03-11T21:48:43.017 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:48:43.017 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:48:43.017 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "dispatched", 2024-03-11T21:48:43.017 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:44:20.436182+0000", 2024-03-11T21:48:43.017 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 6.0739999999999998e-06 2024-03-11T21:48:43.017 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:48:43.017 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:48:43.017 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "queued_for_pg", 2024-03-11T21:48:43.017 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:44:20.436187+0000", 2024-03-11T21:48:43.017 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 5.1170000000000003e-06 2024-03-11T21:48:43.018 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:48:43.018 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:48:43.018 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-03-11T21:48:43.018 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:44:20.450361+0000", 2024-03-11T21:48:43.018 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.014174679000000001 2024-03-11T21:48:43.018 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:48:43.018 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:48:43.018 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "started", 2024-03-11T21:48:43.019 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:44:20.450497+0000", 2024-03-11T21:48:43.019 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.000135313 2024-03-11T21:48:43.019 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:48:43.019 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:48:43.019 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "waiting for subops from 3,5", 2024-03-11T21:48:43.019 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:44:20.450943+0000", 2024-03-11T21:48:43.019 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00044621899999999998 2024-03-11T21:48:43.019 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:48:43.019 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:48:43.020 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-03-11T21:48:43.020 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:44:20.483673+0000", 2024-03-11T21:48:43.020 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.032730453 2024-03-11T21:48:43.020 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:48:43.020 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:48:43.020 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "waiting for peered", 2024-03-11T21:48:43.020 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:44:20.483682+0000", 2024-03-11T21:48:43.020 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 8.9500000000000007e-06 2024-03-11T21:48:43.021 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:48:43.021 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:48:43.021 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-03-11T21:48:43.021 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:44:21.445542+0000", 2024-03-11T21:48:43.021 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.961859401 2024-03-11T21:48:43.021 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:48:43.021 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:48:43.021 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "waiting for peered", 2024-03-11T21:48:43.021 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:44:21.445551+0000", 2024-03-11T21:48:43.021 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 9.1269999999999992e-06 2024-03-11T21:48:43.022 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:48:43.022 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:48:43.022 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-03-11T21:48:43.022 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:44:22.585301+0000", 2024-03-11T21:48:43.022 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.1397497940000001 2024-03-11T21:48:43.022 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:48:43.022 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:48:43.023 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "waiting for missing object", 2024-03-11T21:48:43.023 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:44:22.585465+0000", 2024-03-11T21:48:43.023 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00016380400000000001 2024-03-11T21:48:43.023 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:48:43.023 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:48:43.023 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-03-11T21:48:43.023 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:44:22.631442+0000", 2024-03-11T21:48:43.023 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.045977463000000003 2024-03-11T21:48:43.023 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:48:43.024 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:48:43.024 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "done", 2024-03-11T21:48:43.024 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:44:22.631515+0000", 2024-03-11T21:48:43.024 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 7.2967999999999997e-05 2024-03-11T21:48:43.024 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-03-11T21:48:43.024 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-03-11T21:48:43.024 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-03-11T21:48:43.024 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:48:43.024 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:48:43.025 INFO:teuthology.orchestra.run.smithi092.stdout: "description": "osd_op(client.15744.0:214903 10.3 10:c0f2058f:::benchmark_data_smithi092_161462_object104927:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e432)", 2024-03-11T21:48:43.025 INFO:teuthology.orchestra.run.smithi092.stdout: "initiated_at": "2024-03-11T21:44:20.444963+0000", 2024-03-11T21:48:43.025 INFO:teuthology.orchestra.run.smithi092.stdout: "age": 262.52746982500003, 2024-03-11T21:48:43.025 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 2.197128975, 2024-03-11T21:48:43.025 INFO:teuthology.orchestra.run.smithi092.stdout: "type_data": { 2024-03-11T21:48:43.025 INFO:teuthology.orchestra.run.smithi092.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-11T21:48:43.025 INFO:teuthology.orchestra.run.smithi092.stdout: "client_info": { 2024-03-11T21:48:43.025 INFO:teuthology.orchestra.run.smithi092.stdout: "client": "client.15744", 2024-03-11T21:48:43.025 INFO:teuthology.orchestra.run.smithi092.stdout: "client_addr": "v1:172.21.15.92:0/3334000812", 2024-03-11T21:48:43.026 INFO:teuthology.orchestra.run.smithi092.stdout: "tid": 214903 2024-03-11T21:48:43.026 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:48:43.026 INFO:teuthology.orchestra.run.smithi092.stdout: "events": [ 2024-03-11T21:48:43.026 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:48:43.026 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "initiated", 2024-03-11T21:48:43.026 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:44:20.444963+0000", 2024-03-11T21:48:43.026 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-03-11T21:48:43.026 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:48:43.027 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:48:43.027 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "header_read", 2024-03-11T21:48:43.027 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:44:20.444963+0000", 2024-03-11T21:48:43.027 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-03-11T21:48:43.027 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:48:43.027 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:48:43.027 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "throttled", 2024-03-11T21:48:43.027 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:44:20.444965+0000", 2024-03-11T21:48:43.028 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.9209999999999999e-06 2024-03-11T21:48:43.028 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:48:43.028 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:48:43.028 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "all_read", 2024-03-11T21:48:43.028 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:44:20.444970+0000", 2024-03-11T21:48:43.028 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 4.7870000000000001e-06 2024-03-11T21:48:43.028 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:48:43.028 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:48:43.028 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "dispatched", 2024-03-11T21:48:43.029 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:44:20.449188+0000", 2024-03-11T21:48:43.029 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.0042179890000000001 2024-03-11T21:48:43.029 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:48:43.029 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:48:43.029 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "queued_for_pg", 2024-03-11T21:48:43.029 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:44:20.449197+0000", 2024-03-11T21:48:43.029 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 9.7249999999999992e-06 2024-03-11T21:48:43.029 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:48:43.029 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:48:43.030 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-03-11T21:48:43.030 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:44:20.483991+0000", 2024-03-11T21:48:43.030 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.034793337000000001 2024-03-11T21:48:43.030 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:48:43.030 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:48:43.030 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "waiting for peered", 2024-03-11T21:48:43.030 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:44:20.483999+0000", 2024-03-11T21:48:43.030 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 8.4549999999999995e-06 2024-03-11T21:48:43.031 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:48:43.031 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:48:43.031 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-03-11T21:48:43.031 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:44:21.456053+0000", 2024-03-11T21:48:43.031 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.97205382200000001 2024-03-11T21:48:43.031 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:48:43.031 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:48:43.031 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "waiting for peered", 2024-03-11T21:48:43.031 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:44:21.456066+0000", 2024-03-11T21:48:43.032 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.2867999999999999e-05 2024-03-11T21:48:43.032 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:48:43.032 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:48:43.032 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-03-11T21:48:43.032 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:44:22.612054+0000", 2024-03-11T21:48:43.032 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.155988016 2024-03-11T21:48:43.032 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:48:43.032 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:48:43.032 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "started", 2024-03-11T21:48:43.032 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:44:22.612213+0000", 2024-03-11T21:48:43.032 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.000158844 2024-03-11T21:48:43.032 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:48:43.033 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:48:43.033 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "waiting for subops from 3,4,5", 2024-03-11T21:48:43.033 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:44:22.612773+0000", 2024-03-11T21:48:43.033 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00056077299999999998 2024-03-11T21:48:43.033 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:48:43.033 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:48:43.033 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "sub_op_commit_rec", 2024-03-11T21:48:43.033 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:44:22.629734+0000", 2024-03-11T21:48:43.033 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.016960452000000001 2024-03-11T21:48:43.033 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:48:43.033 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:48:43.033 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "sub_op_commit_rec", 2024-03-11T21:48:43.033 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:44:22.630686+0000", 2024-03-11T21:48:43.033 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00095239999999999995 2024-03-11T21:48:43.033 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:48:43.033 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:48:43.033 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "sub_op_commit_rec", 2024-03-11T21:48:43.034 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:44:22.630726+0000", 2024-03-11T21:48:43.034 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 4.0173000000000003e-05 2024-03-11T21:48:43.034 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:48:43.034 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:48:43.034 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "op_commit", 2024-03-11T21:48:43.034 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:44:22.641922+0000", 2024-03-11T21:48:43.034 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.011195247 2024-03-11T21:48:43.034 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:48:43.034 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:48:43.034 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "commit_sent", 2024-03-11T21:48:43.034 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:44:22.641973+0000", 2024-03-11T21:48:43.034 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 5.1409999999999997e-05 2024-03-11T21:48:43.034 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:48:43.035 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:48:43.035 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "done", 2024-03-11T21:48:43.035 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:44:22.642092+0000", 2024-03-11T21:48:43.035 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.000118756 2024-03-11T21:48:43.035 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-03-11T21:48:43.035 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-03-11T21:48:43.035 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-03-11T21:48:43.035 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:48:43.035 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:48:43.035 INFO:teuthology.orchestra.run.smithi092.stdout: "description": "osd_op(client.15744.0:214919 10.13 10:caf22172:::benchmark_data_smithi092_161462_object104943:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e432)", 2024-03-11T21:48:43.035 INFO:teuthology.orchestra.run.smithi092.stdout: "initiated_at": "2024-03-11T21:44:20.482157+0000", 2024-03-11T21:48:43.035 INFO:teuthology.orchestra.run.smithi092.stdout: "age": 262.49027598499998, 2024-03-11T21:48:43.035 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 2.1484783520000001, 2024-03-11T21:48:43.035 INFO:teuthology.orchestra.run.smithi092.stdout: "type_data": { 2024-03-11T21:48:43.035 INFO:teuthology.orchestra.run.smithi092.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-11T21:48:43.036 INFO:teuthology.orchestra.run.smithi092.stdout: "client_info": { 2024-03-11T21:48:43.036 INFO:teuthology.orchestra.run.smithi092.stdout: "client": "client.15744", 2024-03-11T21:48:43.036 INFO:teuthology.orchestra.run.smithi092.stdout: "client_addr": "v1:172.21.15.92:0/3334000812", 2024-03-11T21:48:43.036 INFO:teuthology.orchestra.run.smithi092.stdout: "tid": 214919 2024-03-11T21:48:43.036 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:48:43.036 INFO:teuthology.orchestra.run.smithi092.stdout: "events": [ 2024-03-11T21:48:43.036 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:48:43.036 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "initiated", 2024-03-11T21:48:43.036 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:44:20.482157+0000", 2024-03-11T21:48:43.036 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-03-11T21:48:43.036 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:48:43.036 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:48:43.036 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "header_read", 2024-03-11T21:48:43.036 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:44:20.482157+0000", 2024-03-11T21:48:43.036 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-03-11T21:48:43.037 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:48:43.037 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:48:43.037 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "throttled", 2024-03-11T21:48:43.037 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:44:20.482159+0000", 2024-03-11T21:48:43.037 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.7439999999999999e-06 2024-03-11T21:48:43.037 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:48:43.037 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:48:43.037 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "all_read", 2024-03-11T21:48:43.037 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:44:20.482164+0000", 2024-03-11T21:48:43.037 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 5.13e-06 2024-03-11T21:48:43.037 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:48:43.038 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:48:43.038 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "dispatched", 2024-03-11T21:48:43.038 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:44:20.482177+0000", 2024-03-11T21:48:43.038 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.3298999999999999e-05 2024-03-11T21:48:43.038 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:48:43.038 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:48:43.038 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "queued_for_pg", 2024-03-11T21:48:43.038 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:44:20.482184+0000", 2024-03-11T21:48:43.038 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 7.2740000000000002e-06 2024-03-11T21:48:43.038 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:48:43.038 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:48:43.038 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-03-11T21:48:43.038 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:44:20.483954+0000", 2024-03-11T21:48:43.038 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.0017693279999999999 2024-03-11T21:48:43.039 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:48:43.039 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:48:43.039 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "waiting for peered", 2024-03-11T21:48:43.039 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:44:20.483963+0000", 2024-03-11T21:48:43.039 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 9.5470000000000002e-06 2024-03-11T21:48:43.039 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:48:43.039 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:48:43.039 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-03-11T21:48:43.039 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:44:21.445579+0000", 2024-03-11T21:48:43.039 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.96161574000000005 2024-03-11T21:48:43.039 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:48:43.039 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:48:43.039 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "waiting for peered", 2024-03-11T21:48:43.039 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:44:21.445587+0000", 2024-03-11T21:48:43.039 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 8.038e-06 2024-03-11T21:48:43.040 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:48:43.040 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:48:43.040 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-03-11T21:48:43.040 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:44:22.585542+0000", 2024-03-11T21:48:43.040 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.139955595 2024-03-11T21:48:43.040 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:48:43.040 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:48:43.040 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "started", 2024-03-11T21:48:43.040 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:44:22.585659+0000", 2024-03-11T21:48:43.040 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00011671 2024-03-11T21:48:43.040 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:48:43.040 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:48:43.040 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "waiting for subops from 3,4,5", 2024-03-11T21:48:43.040 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:44:22.586115+0000", 2024-03-11T21:48:43.041 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00045580700000000002 2024-03-11T21:48:43.041 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:48:43.041 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:48:43.041 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "op_commit", 2024-03-11T21:48:43.041 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:44:22.629374+0000", 2024-03-11T21:48:43.041 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.043258932 2024-03-11T21:48:43.041 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:48:43.041 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:48:43.041 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "sub_op_commit_rec", 2024-03-11T21:48:43.041 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:44:22.629525+0000", 2024-03-11T21:48:43.041 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00015102 2024-03-11T21:48:43.042 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:48:43.042 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:48:43.042 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "sub_op_commit_rec", 2024-03-11T21:48:43.042 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:44:22.630254+0000", 2024-03-11T21:48:43.042 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00072886900000000004 2024-03-11T21:48:43.042 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:48:43.042 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:48:43.042 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "sub_op_commit_rec", 2024-03-11T21:48:43.042 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:44:22.630388+0000", 2024-03-11T21:48:43.042 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00013433799999999999 2024-03-11T21:48:43.042 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:48:43.042 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:48:43.042 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "commit_sent", 2024-03-11T21:48:43.042 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:44:22.630489+0000", 2024-03-11T21:48:43.043 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00010095 2024-03-11T21:48:43.043 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:48:43.043 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:48:43.043 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "done", 2024-03-11T21:48:43.043 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:44:22.630635+0000", 2024-03-11T21:48:43.043 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.000146031 2024-03-11T21:48:43.043 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-03-11T21:48:43.043 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-03-11T21:48:43.043 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-03-11T21:48:43.043 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:48:43.043 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:48:43.043 INFO:teuthology.orchestra.run.smithi092.stdout: "description": "osd_op(client.15744.0:214920 10.3 10:c2918869:::benchmark_data_smithi092_161462_object104944:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e432)", 2024-03-11T21:48:43.043 INFO:teuthology.orchestra.run.smithi092.stdout: "initiated_at": "2024-03-11T21:44:20.485476+0000", 2024-03-11T21:48:43.043 INFO:teuthology.orchestra.run.smithi092.stdout: "age": 262.48695690400001, 2024-03-11T21:48:43.043 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 2.1567741360000001, 2024-03-11T21:48:43.044 INFO:teuthology.orchestra.run.smithi092.stdout: "type_data": { 2024-03-11T21:48:43.044 INFO:teuthology.orchestra.run.smithi092.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-11T21:48:43.044 INFO:teuthology.orchestra.run.smithi092.stdout: "client_info": { 2024-03-11T21:48:43.044 INFO:teuthology.orchestra.run.smithi092.stdout: "client": "client.15744", 2024-03-11T21:48:43.044 INFO:teuthology.orchestra.run.smithi092.stdout: "client_addr": "v1:172.21.15.92:0/3334000812", 2024-03-11T21:48:43.044 INFO:teuthology.orchestra.run.smithi092.stdout: "tid": 214920 2024-03-11T21:48:43.044 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:48:43.044 INFO:teuthology.orchestra.run.smithi092.stdout: "events": [ 2024-03-11T21:48:43.044 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:48:43.044 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "initiated", 2024-03-11T21:48:43.044 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:44:20.485476+0000", 2024-03-11T21:48:43.044 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-03-11T21:48:43.044 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:48:43.044 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:48:43.044 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "header_read", 2024-03-11T21:48:43.044 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:44:20.485476+0000", 2024-03-11T21:48:43.045 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-03-11T21:48:43.045 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:48:43.045 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:48:43.045 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "throttled", 2024-03-11T21:48:43.045 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:44:20.485478+0000", 2024-03-11T21:48:43.045 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.8980000000000001e-06 2024-03-11T21:48:43.045 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:48:43.045 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:48:43.045 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "all_read", 2024-03-11T21:48:43.045 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:44:20.485483+0000", 2024-03-11T21:48:43.046 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 5.5230000000000003e-06 2024-03-11T21:48:43.046 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:48:43.046 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:48:43.046 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "dispatched", 2024-03-11T21:48:43.046 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:44:20.485493+0000", 2024-03-11T21:48:43.046 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 9.798e-06 2024-03-11T21:48:43.046 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:48:43.046 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:48:43.046 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "queued_for_pg", 2024-03-11T21:48:43.046 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:44:20.485501+0000", 2024-03-11T21:48:43.046 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 7.8399999999999995e-06 2024-03-11T21:48:43.046 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:48:43.047 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:48:43.047 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-03-11T21:48:43.047 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:44:20.485572+0000", 2024-03-11T21:48:43.047 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 7.0976000000000006e-05 2024-03-11T21:48:43.047 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:48:43.047 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:48:43.047 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "waiting for peered", 2024-03-11T21:48:43.047 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:44:20.485583+0000", 2024-03-11T21:48:43.047 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.0791999999999999e-05 2024-03-11T21:48:43.047 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:48:43.047 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:48:43.047 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-03-11T21:48:43.048 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:44:21.456094+0000", 2024-03-11T21:48:43.048 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.97051157499999996 2024-03-11T21:48:43.048 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:48:43.048 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:48:43.048 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "waiting for peered", 2024-03-11T21:48:43.048 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:44:21.456103+0000", 2024-03-11T21:48:43.048 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 8.8340000000000005e-06 2024-03-11T21:48:43.048 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:48:43.048 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:48:43.048 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-03-11T21:48:43.048 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:44:22.630758+0000", 2024-03-11T21:48:43.048 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.1746546609999999 2024-03-11T21:48:43.048 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:48:43.048 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:48:43.048 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "started", 2024-03-11T21:48:43.049 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:44:22.631546+0000", 2024-03-11T21:48:43.049 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00078850899999999995 2024-03-11T21:48:43.049 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:48:43.049 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:48:43.049 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "waiting for subops from 3,4,5", 2024-03-11T21:48:43.049 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:44:22.631690+0000", 2024-03-11T21:48:43.049 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00014423499999999999 2024-03-11T21:48:43.049 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:48:43.049 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:48:43.049 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "sub_op_commit_rec", 2024-03-11T21:48:43.049 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:44:22.641727+0000", 2024-03-11T21:48:43.049 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.010036399 2024-03-11T21:48:43.049 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:48:43.049 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:48:43.050 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "sub_op_commit_rec", 2024-03-11T21:48:43.050 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:44:22.641773+0000", 2024-03-11T21:48:43.050 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 4.5899999999999998e-05 2024-03-11T21:48:43.050 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:48:43.050 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:48:43.050 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "sub_op_commit_rec", 2024-03-11T21:48:43.050 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:44:22.641907+0000", 2024-03-11T21:48:43.050 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.000133864 2024-03-11T21:48:43.050 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:48:43.050 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:48:43.050 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "op_commit", 2024-03-11T21:48:43.050 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:44:22.642100+0000", 2024-03-11T21:48:43.050 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00019317800000000001 2024-03-11T21:48:43.051 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:48:43.051 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:48:43.051 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "commit_sent", 2024-03-11T21:48:43.051 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:44:22.642145+0000", 2024-03-11T21:48:43.051 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 4.5590999999999998e-05 2024-03-11T21:48:43.051 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:48:43.051 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:48:43.051 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "done", 2024-03-11T21:48:43.051 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:44:22.642250+0000", 2024-03-11T21:48:43.051 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00010456299999999999 2024-03-11T21:48:43.051 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-03-11T21:48:43.052 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-03-11T21:48:43.052 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-03-11T21:48:43.052 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:48:43.052 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:48:43.052 INFO:teuthology.orchestra.run.smithi092.stdout: "description": "osd_op(client.15744.0:214922 10.5 10:a7611898:::benchmark_data_smithi092_161462_object104946:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e432)", 2024-03-11T21:48:43.052 INFO:teuthology.orchestra.run.smithi092.stdout: "initiated_at": "2024-03-11T21:44:20.487723+0000", 2024-03-11T21:48:43.052 INFO:teuthology.orchestra.run.smithi092.stdout: "age": 262.48470961800001, 2024-03-11T21:48:43.052 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 2.2020016259999999, 2024-03-11T21:48:43.052 INFO:teuthology.orchestra.run.smithi092.stdout: "type_data": { 2024-03-11T21:48:43.052 INFO:teuthology.orchestra.run.smithi092.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-11T21:48:43.052 INFO:teuthology.orchestra.run.smithi092.stdout: "client_info": { 2024-03-11T21:48:43.052 INFO:teuthology.orchestra.run.smithi092.stdout: "client": "client.15744", 2024-03-11T21:48:43.052 INFO:teuthology.orchestra.run.smithi092.stdout: "client_addr": "v1:172.21.15.92:0/3334000812", 2024-03-11T21:48:43.052 INFO:teuthology.orchestra.run.smithi092.stdout: "tid": 214922 2024-03-11T21:48:43.052 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:48:43.053 INFO:teuthology.orchestra.run.smithi092.stdout: "events": [ 2024-03-11T21:48:43.053 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:48:43.053 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "initiated", 2024-03-11T21:48:43.053 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:44:20.487723+0000", 2024-03-11T21:48:43.053 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-03-11T21:48:43.053 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:48:43.053 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:48:43.053 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "header_read", 2024-03-11T21:48:43.053 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:44:20.487723+0000", 2024-03-11T21:48:43.053 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-03-11T21:48:43.053 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:48:43.053 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:48:43.053 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "throttled", 2024-03-11T21:48:43.053 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:44:20.487725+0000", 2024-03-11T21:48:43.054 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.801e-06 2024-03-11T21:48:43.054 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:48:43.054 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:48:43.054 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "all_read", 2024-03-11T21:48:43.054 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:44:20.487731+0000", 2024-03-11T21:48:43.054 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 5.8409999999999996e-06 2024-03-11T21:48:43.054 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:48:43.054 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:48:43.054 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "dispatched", 2024-03-11T21:48:43.054 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:44:20.487743+0000", 2024-03-11T21:48:43.054 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.1932e-05 2024-03-11T21:48:43.054 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:48:43.054 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:48:43.055 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "queued_for_pg", 2024-03-11T21:48:43.055 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:44:20.487752+0000", 2024-03-11T21:48:43.055 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 9.0410000000000003e-06 2024-03-11T21:48:43.055 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:48:43.055 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:48:43.055 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-03-11T21:48:43.055 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:44:20.487811+0000", 2024-03-11T21:48:43.055 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 5.9454999999999999e-05 2024-03-11T21:48:43.055 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:48:43.055 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:48:43.055 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "waiting for peered", 2024-03-11T21:48:43.055 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:44:20.487820+0000", 2024-03-11T21:48:43.055 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 8.4020000000000002e-06 2024-03-11T21:48:43.055 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:48:43.056 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:48:43.056 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-03-11T21:48:43.056 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:44:21.444732+0000", 2024-03-11T21:48:43.056 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.95691247400000001 2024-03-11T21:48:43.056 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:48:43.056 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:48:43.056 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "waiting for peered", 2024-03-11T21:48:43.056 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:44:21.444741+0000", 2024-03-11T21:48:43.056 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 8.9330000000000007e-06 2024-03-11T21:48:43.056 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:48:43.056 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:48:43.056 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-03-11T21:48:43.056 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:44:22.659031+0000", 2024-03-11T21:48:43.056 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.2142904510000001 2024-03-11T21:48:43.057 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:48:43.057 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:48:43.057 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "started", 2024-03-11T21:48:43.057 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:44:22.659152+0000", 2024-03-11T21:48:43.057 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.000120087 2024-03-11T21:48:43.057 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:48:43.057 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:48:43.057 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "waiting for subops from 0,3,7", 2024-03-11T21:48:43.057 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:44:22.659605+0000", 2024-03-11T21:48:43.057 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00045319199999999999 2024-03-11T21:48:43.057 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:48:43.057 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:48:43.057 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "sub_op_commit_rec", 2024-03-11T21:48:43.057 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:44:22.672560+0000", 2024-03-11T21:48:43.057 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.012955642999999999 2024-03-11T21:48:43.058 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:48:43.058 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:48:43.058 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "sub_op_commit_rec", 2024-03-11T21:48:43.058 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:44:22.672596+0000", 2024-03-11T21:48:43.058 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 3.5586000000000001e-05 2024-03-11T21:48:43.058 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:48:43.058 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:48:43.058 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "sub_op_commit_rec", 2024-03-11T21:48:43.058 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:44:22.686428+0000", 2024-03-11T21:48:43.058 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.013831784999999999 2024-03-11T21:48:43.058 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:48:43.058 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:48:43.058 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "op_commit", 2024-03-11T21:48:43.058 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:44:22.689522+0000", 2024-03-11T21:48:43.059 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.003094739 2024-03-11T21:48:43.059 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:48:43.059 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:48:43.059 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "commit_sent", 2024-03-11T21:48:43.059 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:44:22.689583+0000", 2024-03-11T21:48:43.059 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 6.0188e-05 2024-03-11T21:48:43.059 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:48:43.059 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:48:43.059 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "done", 2024-03-11T21:48:43.059 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:44:22.689725+0000", 2024-03-11T21:48:43.059 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00014207600000000001 2024-03-11T21:48:43.059 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-03-11T21:48:43.059 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-03-11T21:48:43.059 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-03-11T21:48:43.060 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:48:43.060 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:48:43.060 INFO:teuthology.orchestra.run.smithi092.stdout: "description": "osd_op(client.15744.0:214927 10.15 10:aa435d62:::benchmark_data_smithi092_161462_object104951:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e432)", 2024-03-11T21:48:43.060 INFO:teuthology.orchestra.run.smithi092.stdout: "initiated_at": "2024-03-11T21:44:20.504817+0000", 2024-03-11T21:48:43.060 INFO:teuthology.orchestra.run.smithi092.stdout: "age": 262.46761587999998, 2024-03-11T21:48:43.060 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 2.1675725749999999, 2024-03-11T21:48:43.060 INFO:teuthology.orchestra.run.smithi092.stdout: "type_data": { 2024-03-11T21:48:43.060 INFO:teuthology.orchestra.run.smithi092.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-11T21:48:43.060 INFO:teuthology.orchestra.run.smithi092.stdout: "client_info": { 2024-03-11T21:48:43.060 INFO:teuthology.orchestra.run.smithi092.stdout: "client": "client.15744", 2024-03-11T21:48:43.060 INFO:teuthology.orchestra.run.smithi092.stdout: "client_addr": "v1:172.21.15.92:0/3334000812", 2024-03-11T21:48:43.060 INFO:teuthology.orchestra.run.smithi092.stdout: "tid": 214927 2024-03-11T21:48:43.060 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:48:43.060 INFO:teuthology.orchestra.run.smithi092.stdout: "events": [ 2024-03-11T21:48:43.060 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:48:43.060 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "initiated", 2024-03-11T21:48:43.060 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:44:20.504817+0000", 2024-03-11T21:48:43.061 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-03-11T21:48:43.061 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:48:43.061 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:48:43.061 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "header_read", 2024-03-11T21:48:43.061 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:44:20.504817+0000", 2024-03-11T21:48:43.061 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-03-11T21:48:43.061 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:48:43.061 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:48:43.061 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "throttled", 2024-03-11T21:48:43.061 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:44:20.504819+0000", 2024-03-11T21:48:43.061 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 2.311e-06 2024-03-11T21:48:43.061 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:48:43.061 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:48:43.062 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "all_read", 2024-03-11T21:48:43.062 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:44:20.504827+0000", 2024-03-11T21:48:43.062 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 8.0490000000000004e-06 2024-03-11T21:48:43.062 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:48:43.062 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:48:43.062 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "dispatched", 2024-03-11T21:48:43.062 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:44:20.504841+0000", 2024-03-11T21:48:43.062 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.3909e-05 2024-03-11T21:48:43.062 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:48:43.062 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:48:43.062 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "queued_for_pg", 2024-03-11T21:48:43.062 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:44:20.504849+0000", 2024-03-11T21:48:43.063 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 7.418e-06 2024-03-11T21:48:43.063 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:48:43.063 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:48:43.063 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-03-11T21:48:43.063 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:44:20.504907+0000", 2024-03-11T21:48:43.063 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 5.8601999999999999e-05 2024-03-11T21:48:43.063 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:48:43.063 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:48:43.063 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "waiting for peered", 2024-03-11T21:48:43.063 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:44:20.504916+0000", 2024-03-11T21:48:43.063 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 8.9819999999999997e-06 2024-03-11T21:48:43.063 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:48:43.063 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:48:43.063 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-03-11T21:48:43.063 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:44:21.444683+0000", 2024-03-11T21:48:43.063 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.93976688200000003 2024-03-11T21:48:43.064 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:48:43.064 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:48:43.064 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "waiting for peered", 2024-03-11T21:48:43.064 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:44:21.444698+0000", 2024-03-11T21:48:43.064 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.5019e-05 2024-03-11T21:48:43.064 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:48:43.064 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:48:43.064 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-03-11T21:48:43.064 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:44:22.589489+0000", 2024-03-11T21:48:43.064 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.144791109 2024-03-11T21:48:43.064 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:48:43.064 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:48:43.064 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "started", 2024-03-11T21:48:43.064 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:44:22.589627+0000", 2024-03-11T21:48:43.064 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00013739899999999999 2024-03-11T21:48:43.064 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:48:43.065 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:48:43.065 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "waiting for subops from 0,3,7", 2024-03-11T21:48:43.065 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:44:22.590016+0000", 2024-03-11T21:48:43.065 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.000389742 2024-03-11T21:48:43.065 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:48:43.065 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:48:43.065 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "sub_op_commit_rec", 2024-03-11T21:48:43.065 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:44:22.599105+0000", 2024-03-11T21:48:43.065 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.0090885830000000008 2024-03-11T21:48:43.065 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:48:43.065 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:48:43.065 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "op_commit", 2024-03-11T21:48:43.065 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:44:22.599369+0000", 2024-03-11T21:48:43.065 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00026393400000000001 2024-03-11T21:48:43.065 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:48:43.065 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:48:43.066 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "sub_op_commit_rec", 2024-03-11T21:48:43.066 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:44:22.600949+0000", 2024-03-11T21:48:43.066 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.0015797020000000001 2024-03-11T21:48:43.066 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:48:43.066 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:48:43.066 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "sub_op_commit_rec", 2024-03-11T21:48:43.066 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:44:22.672213+0000", 2024-03-11T21:48:43.066 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.071264211999999993 2024-03-11T21:48:43.066 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:48:43.066 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:48:43.066 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "commit_sent", 2024-03-11T21:48:43.066 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:44:22.672272+0000", 2024-03-11T21:48:43.066 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 5.9653e-05 2024-03-11T21:48:43.066 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:48:43.067 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:48:43.067 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "done", 2024-03-11T21:48:43.067 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:44:22.672389+0000", 2024-03-11T21:48:43.067 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.000117069 2024-03-11T21:48:43.067 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-03-11T21:48:43.067 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-03-11T21:48:43.067 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-03-11T21:48:43.067 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:48:43.067 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:48:43.067 INFO:teuthology.orchestra.run.smithi092.stdout: "description": "osd_op(client.15744.0:217044 10.13 10:c89bc29e:::benchmark_data_smithi092_161462_object107068:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e435)", 2024-03-11T21:48:43.067 INFO:teuthology.orchestra.run.smithi092.stdout: "initiated_at": "2024-03-11T21:44:24.458977+0000", 2024-03-11T21:48:43.067 INFO:teuthology.orchestra.run.smithi092.stdout: "age": 258.51345611199997, 2024-03-11T21:48:43.067 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.0342619019999999, 2024-03-11T21:48:43.067 INFO:teuthology.orchestra.run.smithi092.stdout: "type_data": { 2024-03-11T21:48:43.068 INFO:teuthology.orchestra.run.smithi092.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-11T21:48:43.068 INFO:teuthology.orchestra.run.smithi092.stdout: "client_info": { 2024-03-11T21:48:43.068 INFO:teuthology.orchestra.run.smithi092.stdout: "client": "client.15744", 2024-03-11T21:48:43.068 INFO:teuthology.orchestra.run.smithi092.stdout: "client_addr": "v1:172.21.15.92:0/3334000812", 2024-03-11T21:48:43.068 INFO:teuthology.orchestra.run.smithi092.stdout: "tid": 217044 2024-03-11T21:48:43.068 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:48:43.068 INFO:teuthology.orchestra.run.smithi092.stdout: "events": [ 2024-03-11T21:48:43.068 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:48:43.068 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "initiated", 2024-03-11T21:48:43.068 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:44:24.458977+0000", 2024-03-11T21:48:43.068 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-03-11T21:48:43.068 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:48:43.068 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:48:43.068 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "header_read", 2024-03-11T21:48:43.068 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:44:24.458977+0000", 2024-03-11T21:48:43.069 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-03-11T21:48:43.069 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:48:43.069 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:48:43.069 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "throttled", 2024-03-11T21:48:43.069 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:44:24.458979+0000", 2024-03-11T21:48:43.069 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 2.5000000000000002e-06 2024-03-11T21:48:43.069 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:48:43.069 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:48:43.069 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "all_read", 2024-03-11T21:48:43.069 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:44:24.458985+0000", 2024-03-11T21:48:43.069 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 5.84e-06 2024-03-11T21:48:43.069 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:48:43.069 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:48:43.069 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "dispatched", 2024-03-11T21:48:43.069 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:44:24.458995+0000", 2024-03-11T21:48:43.069 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.0033e-05 2024-03-11T21:48:43.070 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:48:43.070 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:48:43.070 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "queued_for_pg", 2024-03-11T21:48:43.070 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:44:24.459008+0000", 2024-03-11T21:48:43.070 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.3328e-05 2024-03-11T21:48:43.070 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:48:43.070 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:48:43.070 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-03-11T21:48:43.070 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:44:24.463538+0000", 2024-03-11T21:48:43.070 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.0045298059999999999 2024-03-11T21:48:43.070 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:48:43.070 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:48:43.071 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "waiting for peered", 2024-03-11T21:48:43.071 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:44:24.463548+0000", 2024-03-11T21:48:43.071 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 9.995e-06 2024-03-11T21:48:43.071 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:48:43.071 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:48:43.071 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-03-11T21:48:43.071 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:44:25.477996+0000", 2024-03-11T21:48:43.071 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.0144477240000001 2024-03-11T21:48:43.071 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:48:43.071 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:48:43.071 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "started", 2024-03-11T21:48:43.071 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:44:25.478128+0000", 2024-03-11T21:48:43.072 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.000131781 2024-03-11T21:48:43.072 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:48:43.072 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:48:43.072 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "waiting for subops from 4,5", 2024-03-11T21:48:43.072 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:44:25.478254+0000", 2024-03-11T21:48:43.072 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.000126532 2024-03-11T21:48:43.072 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:48:43.072 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:48:43.072 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "sub_op_commit_rec", 2024-03-11T21:48:43.073 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:44:25.481042+0000", 2024-03-11T21:48:43.073 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.0027881249999999998 2024-03-11T21:48:43.073 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:48:43.073 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:48:43.073 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "sub_op_commit_rec", 2024-03-11T21:48:43.073 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:44:25.481067+0000", 2024-03-11T21:48:43.073 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 2.5058e-05 2024-03-11T21:48:43.073 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:48:43.073 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:48:43.074 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "op_commit", 2024-03-11T21:48:43.074 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:44:25.493152+0000", 2024-03-11T21:48:43.074 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.012084978999999999 2024-03-11T21:48:43.074 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:48:43.074 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:48:43.074 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "commit_sent", 2024-03-11T21:48:43.074 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:44:25.493199+0000", 2024-03-11T21:48:43.074 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 4.6221999999999999e-05 2024-03-11T21:48:43.074 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:48:43.074 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:48:43.074 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "done", 2024-03-11T21:48:43.074 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:44:25.493239+0000", 2024-03-11T21:48:43.074 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 3.9978999999999998e-05 2024-03-11T21:48:43.074 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-03-11T21:48:43.075 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-03-11T21:48:43.075 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-03-11T21:48:43.075 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:48:43.075 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:48:43.075 INFO:teuthology.orchestra.run.smithi092.stdout: "description": "osd_op(client.15744.0:217046 10.13 10:cad0bdf2:::benchmark_data_smithi092_161462_object107070:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e435)", 2024-03-11T21:48:43.075 INFO:teuthology.orchestra.run.smithi092.stdout: "initiated_at": "2024-03-11T21:44:24.463433+0000", 2024-03-11T21:48:43.075 INFO:teuthology.orchestra.run.smithi092.stdout: "age": 258.50899955900002, 2024-03-11T21:48:43.075 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.0300054569999999, 2024-03-11T21:48:43.075 INFO:teuthology.orchestra.run.smithi092.stdout: "type_data": { 2024-03-11T21:48:43.075 INFO:teuthology.orchestra.run.smithi092.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-11T21:48:43.075 INFO:teuthology.orchestra.run.smithi092.stdout: "client_info": { 2024-03-11T21:48:43.075 INFO:teuthology.orchestra.run.smithi092.stdout: "client": "client.15744", 2024-03-11T21:48:43.075 INFO:teuthology.orchestra.run.smithi092.stdout: "client_addr": "v1:172.21.15.92:0/3334000812", 2024-03-11T21:48:43.075 INFO:teuthology.orchestra.run.smithi092.stdout: "tid": 217046 2024-03-11T21:48:43.076 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:48:43.076 INFO:teuthology.orchestra.run.smithi092.stdout: "events": [ 2024-03-11T21:48:43.076 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:48:43.076 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "initiated", 2024-03-11T21:48:43.076 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:44:24.463433+0000", 2024-03-11T21:48:43.076 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-03-11T21:48:43.076 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:48:43.076 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:48:43.076 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "header_read", 2024-03-11T21:48:43.077 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:44:24.463433+0000", 2024-03-11T21:48:43.077 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-03-11T21:48:43.077 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:48:43.077 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:48:43.078 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "throttled", 2024-03-11T21:48:43.078 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:44:24.463435+0000", 2024-03-11T21:48:43.078 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.7269999999999999e-06 2024-03-11T21:48:43.078 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:48:43.078 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:48:43.078 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "all_read", 2024-03-11T21:48:43.078 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:44:24.463443+0000", 2024-03-11T21:48:43.078 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 8.4030000000000007e-06 2024-03-11T21:48:43.078 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:48:43.078 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:48:43.078 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "dispatched", 2024-03-11T21:48:43.078 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:44:24.463468+0000", 2024-03-11T21:48:43.079 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 2.4519999999999999e-05 2024-03-11T21:48:43.079 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:48:43.079 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:48:43.079 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "queued_for_pg", 2024-03-11T21:48:43.079 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:44:24.463477+0000", 2024-03-11T21:48:43.079 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 9.0280000000000006e-06 2024-03-11T21:48:43.079 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:48:43.079 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:48:43.079 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-03-11T21:48:43.079 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:44:24.463586+0000", 2024-03-11T21:48:43.079 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.000108683 2024-03-11T21:48:43.079 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:48:43.079 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:48:43.079 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "waiting for peered", 2024-03-11T21:48:43.079 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:44:24.463593+0000", 2024-03-11T21:48:43.080 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 7.0659999999999998e-06 2024-03-11T21:48:43.080 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:48:43.080 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:48:43.080 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-03-11T21:48:43.080 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:44:25.481084+0000", 2024-03-11T21:48:43.080 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.0174910429999999 2024-03-11T21:48:43.080 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:48:43.080 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:48:43.080 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "started", 2024-03-11T21:48:43.080 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:44:25.481185+0000", 2024-03-11T21:48:43.080 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00010090399999999999 2024-03-11T21:48:43.080 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:48:43.080 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:48:43.081 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "waiting for subops from 4,5", 2024-03-11T21:48:43.081 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:44:25.481299+0000", 2024-03-11T21:48:43.081 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.000114403 2024-03-11T21:48:43.081 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:48:43.081 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:48:43.081 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "sub_op_commit_rec", 2024-03-11T21:48:43.081 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:44:25.487709+0000", 2024-03-11T21:48:43.081 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.0064103520000000002 2024-03-11T21:48:43.081 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:48:43.081 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:48:43.081 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "sub_op_commit_rec", 2024-03-11T21:48:43.081 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:44:25.487825+0000", 2024-03-11T21:48:43.081 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00011552099999999999 2024-03-11T21:48:43.081 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:48:43.081 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:48:43.082 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "op_commit", 2024-03-11T21:48:43.082 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:44:25.493352+0000", 2024-03-11T21:48:43.082 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.0055268030000000003 2024-03-11T21:48:43.082 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:48:43.082 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:48:43.082 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "commit_sent", 2024-03-11T21:48:43.082 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:44:25.493402+0000", 2024-03-11T21:48:43.082 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 5.0528000000000001e-05 2024-03-11T21:48:43.082 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:48:43.082 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:48:43.083 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "done", 2024-03-11T21:48:43.083 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:44:25.493439+0000", 2024-03-11T21:48:43.083 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 3.6476000000000001e-05 2024-03-11T21:48:43.083 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-03-11T21:48:43.083 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-03-11T21:48:43.083 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-03-11T21:48:43.083 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:48:43.083 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:48:43.083 INFO:teuthology.orchestra.run.smithi092.stdout: "description": "osd_op(client.15744.0:217062 10.15 10:afc0015c:::benchmark_data_smithi092_161462_object107086:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e435)", 2024-03-11T21:48:43.083 INFO:teuthology.orchestra.run.smithi092.stdout: "initiated_at": "2024-03-11T21:44:24.471893+0000", 2024-03-11T21:48:43.083 INFO:teuthology.orchestra.run.smithi092.stdout: "age": 258.50054011899999, 2024-03-11T21:48:43.083 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.019489799, 2024-03-11T21:48:43.083 INFO:teuthology.orchestra.run.smithi092.stdout: "type_data": { 2024-03-11T21:48:43.083 INFO:teuthology.orchestra.run.smithi092.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-11T21:48:43.083 INFO:teuthology.orchestra.run.smithi092.stdout: "client_info": { 2024-03-11T21:48:43.083 INFO:teuthology.orchestra.run.smithi092.stdout: "client": "client.15744", 2024-03-11T21:48:43.084 INFO:teuthology.orchestra.run.smithi092.stdout: "client_addr": "v1:172.21.15.92:0/3334000812", 2024-03-11T21:48:43.084 INFO:teuthology.orchestra.run.smithi092.stdout: "tid": 217062 2024-03-11T21:48:43.084 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:48:43.084 INFO:teuthology.orchestra.run.smithi092.stdout: "events": [ 2024-03-11T21:48:43.084 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:48:43.084 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "initiated", 2024-03-11T21:48:43.084 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:44:24.471893+0000", 2024-03-11T21:48:43.084 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-03-11T21:48:43.084 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:48:43.084 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:48:43.084 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "header_read", 2024-03-11T21:48:43.084 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:44:24.471893+0000", 2024-03-11T21:48:43.084 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-03-11T21:48:43.084 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:48:43.084 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:48:43.084 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "throttled", 2024-03-11T21:48:43.085 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:44:24.471895+0000", 2024-03-11T21:48:43.085 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 2.3e-06 2024-03-11T21:48:43.085 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:48:43.085 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:48:43.085 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "all_read", 2024-03-11T21:48:43.085 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:44:24.471902+0000", 2024-03-11T21:48:43.085 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 6.5719999999999999e-06 2024-03-11T21:48:43.085 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:48:43.085 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:48:43.085 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "dispatched", 2024-03-11T21:48:43.085 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:44:24.471919+0000", 2024-03-11T21:48:43.085 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.7578e-05 2024-03-11T21:48:43.085 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:48:43.085 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:48:43.085 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "queued_for_pg", 2024-03-11T21:48:43.085 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:44:24.471928+0000", 2024-03-11T21:48:43.086 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 9.1190000000000003e-06 2024-03-11T21:48:43.086 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:48:43.086 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:48:43.086 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-03-11T21:48:43.086 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:44:24.471993+0000", 2024-03-11T21:48:43.086 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 6.4778999999999998e-05 2024-03-11T21:48:43.086 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:48:43.086 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:48:43.086 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "waiting for peered", 2024-03-11T21:48:43.086 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:44:24.472008+0000", 2024-03-11T21:48:43.086 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.4905e-05 2024-03-11T21:48:43.086 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:48:43.086 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:48:43.087 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-03-11T21:48:43.087 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:44:25.480032+0000", 2024-03-11T21:48:43.087 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.0080243520000001 2024-03-11T21:48:43.087 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:48:43.087 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:48:43.087 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "started", 2024-03-11T21:48:43.087 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:44:25.480165+0000", 2024-03-11T21:48:43.087 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.000132583 2024-03-11T21:48:43.087 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:48:43.087 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:48:43.087 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "waiting for subops from 0,7", 2024-03-11T21:48:43.087 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:44:25.480300+0000", 2024-03-11T21:48:43.087 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00013515000000000001 2024-03-11T21:48:43.087 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:48:43.087 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:48:43.088 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "sub_op_commit_rec", 2024-03-11T21:48:43.088 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:44:25.482875+0000", 2024-03-11T21:48:43.088 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.0025746129999999999 2024-03-11T21:48:43.088 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:48:43.088 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:48:43.088 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "op_commit", 2024-03-11T21:48:43.088 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:44:25.488868+0000", 2024-03-11T21:48:43.088 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.0059934610000000003 2024-03-11T21:48:43.088 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:48:43.088 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:48:43.088 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "sub_op_commit_rec", 2024-03-11T21:48:43.088 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:44:25.491239+0000", 2024-03-11T21:48:43.088 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.002370891 2024-03-11T21:48:43.088 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:48:43.088 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:48:43.089 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "commit_sent", 2024-03-11T21:48:43.089 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:44:25.491332+0000", 2024-03-11T21:48:43.089 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 9.3543999999999997e-05 2024-03-11T21:48:43.089 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:48:43.089 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:48:43.089 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "done", 2024-03-11T21:48:43.089 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:44:25.491382+0000", 2024-03-11T21:48:43.089 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 4.9951999999999999e-05 2024-03-11T21:48:43.089 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-03-11T21:48:43.089 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-03-11T21:48:43.089 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-03-11T21:48:43.089 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:48:43.089 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:48:43.089 INFO:teuthology.orchestra.run.smithi092.stdout: "description": "osd_op(client.15744.0:217073 10.13 10:cebef65c:::benchmark_data_smithi092_161462_object107097:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e435)", 2024-03-11T21:48:43.089 INFO:teuthology.orchestra.run.smithi092.stdout: "initiated_at": "2024-03-11T21:44:24.481881+0000", 2024-03-11T21:48:43.090 INFO:teuthology.orchestra.run.smithi092.stdout: "age": 258.49055172300001, 2024-03-11T21:48:43.090 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.011640874, 2024-03-11T21:48:43.090 INFO:teuthology.orchestra.run.smithi092.stdout: "type_data": { 2024-03-11T21:48:43.090 INFO:teuthology.orchestra.run.smithi092.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-11T21:48:43.090 INFO:teuthology.orchestra.run.smithi092.stdout: "client_info": { 2024-03-11T21:48:43.090 INFO:teuthology.orchestra.run.smithi092.stdout: "client": "client.15744", 2024-03-11T21:48:43.090 INFO:teuthology.orchestra.run.smithi092.stdout: "client_addr": "v1:172.21.15.92:0/3334000812", 2024-03-11T21:48:43.090 INFO:teuthology.orchestra.run.smithi092.stdout: "tid": 217073 2024-03-11T21:48:43.090 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:48:43.090 INFO:teuthology.orchestra.run.smithi092.stdout: "events": [ 2024-03-11T21:48:43.090 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:48:43.090 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "initiated", 2024-03-11T21:48:43.091 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:44:24.481881+0000", 2024-03-11T21:48:43.091 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-03-11T21:48:43.091 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:48:43.091 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:48:43.091 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "header_read", 2024-03-11T21:48:43.091 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:44:24.481881+0000", 2024-03-11T21:48:43.091 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-03-11T21:48:43.091 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:48:43.091 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:48:43.091 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "throttled", 2024-03-11T21:48:43.091 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:44:24.481883+0000", 2024-03-11T21:48:43.091 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.5319999999999999e-06 2024-03-11T21:48:43.091 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:48:43.091 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:48:43.092 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "all_read", 2024-03-11T21:48:43.092 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:44:24.481887+0000", 2024-03-11T21:48:43.092 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 4.8609999999999997e-06 2024-03-11T21:48:43.092 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:48:43.092 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:48:43.092 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "dispatched", 2024-03-11T21:48:43.092 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:44:24.481900+0000", 2024-03-11T21:48:43.092 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.2503e-05 2024-03-11T21:48:43.092 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:48:43.092 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:48:43.092 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "queued_for_pg", 2024-03-11T21:48:43.092 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:44:24.481907+0000", 2024-03-11T21:48:43.093 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 7.2250000000000003e-06 2024-03-11T21:48:43.093 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:48:43.093 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:48:43.093 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-03-11T21:48:43.093 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:44:24.481959+0000", 2024-03-11T21:48:43.093 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 5.1468000000000003e-05 2024-03-11T21:48:43.093 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:48:43.093 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:48:43.093 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "waiting for peered", 2024-03-11T21:48:43.093 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:44:24.481966+0000", 2024-03-11T21:48:43.093 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 7.8679999999999999e-06 2024-03-11T21:48:43.093 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:48:43.093 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:48:43.093 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-03-11T21:48:43.094 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:44:25.484248+0000", 2024-03-11T21:48:43.094 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.002281534 2024-03-11T21:48:43.094 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:48:43.094 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:48:43.094 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "started", 2024-03-11T21:48:43.094 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:44:25.484351+0000", 2024-03-11T21:48:43.094 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.000102773 2024-03-11T21:48:43.094 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:48:43.094 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:48:43.094 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "waiting for subops from 4,5", 2024-03-11T21:48:43.094 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:44:25.484459+0000", 2024-03-11T21:48:43.094 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00010796100000000001 2024-03-11T21:48:43.094 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:48:43.095 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:48:43.095 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "sub_op_commit_rec", 2024-03-11T21:48:43.095 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:44:25.487755+0000", 2024-03-11T21:48:43.095 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.0032957809999999998 2024-03-11T21:48:43.095 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:48:43.095 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:48:43.095 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "sub_op_commit_rec", 2024-03-11T21:48:43.095 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:44:25.493056+0000", 2024-03-11T21:48:43.095 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.0053012099999999998 2024-03-11T21:48:43.095 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:48:43.095 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:48:43.095 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "op_commit", 2024-03-11T21:48:43.095 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:44:25.493447+0000", 2024-03-11T21:48:43.095 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00039108900000000001 2024-03-11T21:48:43.095 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:48:43.096 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:48:43.096 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "commit_sent", 2024-03-11T21:48:43.096 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:44:25.493487+0000", 2024-03-11T21:48:43.096 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 4.0357e-05 2024-03-11T21:48:43.096 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:48:43.096 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:48:43.096 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "done", 2024-03-11T21:48:43.096 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:44:25.493522+0000", 2024-03-11T21:48:43.096 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 3.4712000000000002e-05 2024-03-11T21:48:43.096 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-03-11T21:48:43.096 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-03-11T21:48:43.096 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-03-11T21:48:43.096 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:48:43.096 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:48:43.097 INFO:teuthology.orchestra.run.smithi092.stdout: "description": "osd_op(client.15744.0:217076 10.15 10:a84f7b05:::benchmark_data_smithi092_161462_object107100:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e435)", 2024-03-11T21:48:43.097 INFO:teuthology.orchestra.run.smithi092.stdout: "initiated_at": "2024-03-11T21:44:24.482146+0000", 2024-03-11T21:48:43.097 INFO:teuthology.orchestra.run.smithi092.stdout: "age": 258.490286389, 2024-03-11T21:48:43.097 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.0110986070000001, 2024-03-11T21:48:43.097 INFO:teuthology.orchestra.run.smithi092.stdout: "type_data": { 2024-03-11T21:48:43.097 INFO:teuthology.orchestra.run.smithi092.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-11T21:48:43.097 INFO:teuthology.orchestra.run.smithi092.stdout: "client_info": { 2024-03-11T21:48:43.097 INFO:teuthology.orchestra.run.smithi092.stdout: "client": "client.15744", 2024-03-11T21:48:43.097 INFO:teuthology.orchestra.run.smithi092.stdout: "client_addr": "v1:172.21.15.92:0/3334000812", 2024-03-11T21:48:43.097 INFO:teuthology.orchestra.run.smithi092.stdout: "tid": 217076 2024-03-11T21:48:43.097 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:48:43.097 INFO:teuthology.orchestra.run.smithi092.stdout: "events": [ 2024-03-11T21:48:43.097 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:48:43.097 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "initiated", 2024-03-11T21:48:43.098 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:44:24.482146+0000", 2024-03-11T21:48:43.098 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-03-11T21:48:43.098 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:48:43.098 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:48:43.098 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "header_read", 2024-03-11T21:48:43.098 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:44:24.482146+0000", 2024-03-11T21:48:43.098 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-03-11T21:48:43.098 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:48:43.098 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:48:43.098 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "throttled", 2024-03-11T21:48:43.098 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:44:24.482147+0000", 2024-03-11T21:48:43.098 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.0550000000000001e-06 2024-03-11T21:48:43.098 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:48:43.098 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:48:43.099 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "all_read", 2024-03-11T21:48:43.099 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:44:24.482152+0000", 2024-03-11T21:48:43.099 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 4.3359999999999997e-06 2024-03-11T21:48:43.099 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:48:43.099 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:48:43.099 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "dispatched", 2024-03-11T21:48:43.099 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:44:24.482160+0000", 2024-03-11T21:48:43.099 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 8.4069999999999993e-06 2024-03-11T21:48:43.099 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:48:43.099 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:48:43.099 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "queued_for_pg", 2024-03-11T21:48:43.099 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:44:24.482167+0000", 2024-03-11T21:48:43.099 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 6.703e-06 2024-03-11T21:48:43.099 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:48:43.099 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:48:43.100 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-03-11T21:48:43.100 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:44:24.482221+0000", 2024-03-11T21:48:43.100 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 5.4289000000000001e-05 2024-03-11T21:48:43.100 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:48:43.100 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:48:43.100 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "waiting for peered", 2024-03-11T21:48:43.100 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:44:24.482231+0000", 2024-03-11T21:48:43.100 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 9.9369999999999999e-06 2024-03-11T21:48:43.100 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:48:43.100 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:48:43.100 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-03-11T21:48:43.100 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:44:25.482903+0000", 2024-03-11T21:48:43.100 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.0006717510000001 2024-03-11T21:48:43.100 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:48:43.101 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:48:43.101 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "started", 2024-03-11T21:48:43.101 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:44:25.483015+0000", 2024-03-11T21:48:43.101 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.000111734 2024-03-11T21:48:43.101 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:48:43.101 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:48:43.101 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "waiting for subops from 0,7", 2024-03-11T21:48:43.101 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:44:25.483133+0000", 2024-03-11T21:48:43.101 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00011796800000000001 2024-03-11T21:48:43.101 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:48:43.101 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:48:43.101 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "sub_op_commit_rec", 2024-03-11T21:48:43.101 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:44:25.485205+0000", 2024-03-11T21:48:43.101 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.0020728650000000001 2024-03-11T21:48:43.101 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:48:43.101 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:48:43.102 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "op_commit", 2024-03-11T21:48:43.102 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:44:25.488968+0000", 2024-03-11T21:48:43.102 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.003762109 2024-03-11T21:48:43.102 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:48:43.102 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:48:43.102 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "sub_op_commit_rec", 2024-03-11T21:48:43.102 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:44:25.493082+0000", 2024-03-11T21:48:43.102 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.0041149079999999996 2024-03-11T21:48:43.102 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:48:43.102 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:48:43.102 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "commit_sent", 2024-03-11T21:48:43.102 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:44:25.493184+0000", 2024-03-11T21:48:43.102 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.000101709 2024-03-11T21:48:43.103 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:48:43.103 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:48:43.103 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "done", 2024-03-11T21:48:43.103 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:44:25.493245+0000", 2024-03-11T21:48:43.103 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 6.0835999999999998e-05 2024-03-11T21:48:43.103 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-03-11T21:48:43.103 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-03-11T21:48:43.103 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-03-11T21:48:43.103 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:48:43.103 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:48:43.103 INFO:teuthology.orchestra.run.smithi092.stdout: "description": "osd_op(client.15744.0:217089 10.13 10:cdae85f8:::benchmark_data_smithi092_161462_object107113:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e435)", 2024-03-11T21:48:43.104 INFO:teuthology.orchestra.run.smithi092.stdout: "initiated_at": "2024-03-11T21:44:24.495120+0000", 2024-03-11T21:48:43.104 INFO:teuthology.orchestra.run.smithi092.stdout: "age": 258.477313047, 2024-03-11T21:48:43.104 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.99924555500000001, 2024-03-11T21:48:43.104 INFO:teuthology.orchestra.run.smithi092.stdout: "type_data": { 2024-03-11T21:48:43.104 INFO:teuthology.orchestra.run.smithi092.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-11T21:48:43.104 INFO:teuthology.orchestra.run.smithi092.stdout: "client_info": { 2024-03-11T21:48:43.104 INFO:teuthology.orchestra.run.smithi092.stdout: "client": "client.15744", 2024-03-11T21:48:43.104 INFO:teuthology.orchestra.run.smithi092.stdout: "client_addr": "v1:172.21.15.92:0/3334000812", 2024-03-11T21:48:43.104 INFO:teuthology.orchestra.run.smithi092.stdout: "tid": 217089 2024-03-11T21:48:43.104 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:48:43.104 INFO:teuthology.orchestra.run.smithi092.stdout: "events": [ 2024-03-11T21:48:43.104 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:48:43.104 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "initiated", 2024-03-11T21:48:43.104 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:44:24.495120+0000", 2024-03-11T21:48:43.104 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-03-11T21:48:43.104 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:48:43.105 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:48:43.105 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "header_read", 2024-03-11T21:48:43.105 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:44:24.495120+0000", 2024-03-11T21:48:43.105 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-03-11T21:48:43.105 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:48:43.105 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:48:43.105 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "throttled", 2024-03-11T21:48:43.105 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:44:24.495122+0000", 2024-03-11T21:48:43.105 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 2.2340000000000001e-06 2024-03-11T21:48:43.105 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:48:43.105 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:48:43.105 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "all_read", 2024-03-11T21:48:43.105 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:44:24.495128+0000", 2024-03-11T21:48:43.105 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 6.4359999999999999e-06 2024-03-11T21:48:43.105 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:48:43.106 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:48:43.106 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "dispatched", 2024-03-11T21:48:43.106 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:44:24.495144+0000", 2024-03-11T21:48:43.106 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.5804e-05 2024-03-11T21:48:43.106 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:48:43.106 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:48:43.106 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "queued_for_pg", 2024-03-11T21:48:43.106 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:44:24.495153+0000", 2024-03-11T21:48:43.106 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 9.1220000000000001e-06 2024-03-11T21:48:43.106 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:48:43.106 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:48:43.106 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-03-11T21:48:43.107 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:44:24.495210+0000", 2024-03-11T21:48:43.107 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 5.6728999999999998e-05 2024-03-11T21:48:43.107 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:48:43.107 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:48:43.107 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "waiting for peered", 2024-03-11T21:48:43.107 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:44:24.495220+0000", 2024-03-11T21:48:43.107 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 9.7329999999999998e-06 2024-03-11T21:48:43.107 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:48:43.107 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:48:43.107 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-03-11T21:48:43.107 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:44:25.490125+0000", 2024-03-11T21:48:43.107 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.99490510899999995 2024-03-11T21:48:43.107 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:48:43.107 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:48:43.107 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "started", 2024-03-11T21:48:43.108 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:44:25.490229+0000", 2024-03-11T21:48:43.108 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.000104109 2024-03-11T21:48:43.108 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:48:43.108 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:48:43.108 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "waiting for subops from 4,5", 2024-03-11T21:48:43.108 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:44:25.490335+0000", 2024-03-11T21:48:43.108 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.000106259 2024-03-11T21:48:43.108 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:48:43.108 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:48:43.108 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "sub_op_commit_rec", 2024-03-11T21:48:43.108 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:44:25.493134+0000", 2024-03-11T21:48:43.108 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.0027984199999999998 2024-03-11T21:48:43.108 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:48:43.108 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:48:43.108 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "op_commit", 2024-03-11T21:48:43.109 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:44:25.493535+0000", 2024-03-11T21:48:43.109 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00040097199999999998 2024-03-11T21:48:43.109 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:48:43.109 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:48:43.109 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "sub_op_commit_rec", 2024-03-11T21:48:43.109 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:44:25.494282+0000", 2024-03-11T21:48:43.109 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00074706100000000004 2024-03-11T21:48:43.109 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:48:43.109 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:48:43.109 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "commit_sent", 2024-03-11T21:48:43.109 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:44:25.494328+0000", 2024-03-11T21:48:43.109 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 4.5986999999999999e-05 2024-03-11T21:48:43.109 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:48:43.109 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:48:43.110 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "done", 2024-03-11T21:48:43.110 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:44:25.494365+0000", 2024-03-11T21:48:43.110 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 3.7580000000000003e-05 2024-03-11T21:48:43.110 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-03-11T21:48:43.110 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-03-11T21:48:43.110 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-03-11T21:48:43.110 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:48:43.110 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:48:43.110 INFO:teuthology.orchestra.run.smithi092.stdout: "description": "osd_op(client.34352.0:4416 4.4 4:22d26bf9:::data_loggenerations_metadata:head [watch reconnect cookie 93924550620672 gen 4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e478)", 2024-03-11T21:48:43.110 INFO:teuthology.orchestra.run.smithi092.stdout: "initiated_at": "2024-03-11T21:47:19.268281+0000", 2024-03-11T21:48:43.110 INFO:teuthology.orchestra.run.smithi092.stdout: "age": 83.704151944000003, 2024-03-11T21:48:43.110 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.0130998769999999, 2024-03-11T21:48:43.110 INFO:teuthology.orchestra.run.smithi092.stdout: "type_data": { 2024-03-11T21:48:43.111 INFO:teuthology.orchestra.run.smithi092.stdout: "flag_point": "started", 2024-03-11T21:48:43.111 INFO:teuthology.orchestra.run.smithi092.stdout: "client_info": { 2024-03-11T21:48:43.111 INFO:teuthology.orchestra.run.smithi092.stdout: "client": "client.34352", 2024-03-11T21:48:43.111 INFO:teuthology.orchestra.run.smithi092.stdout: "client_addr": "v1:172.21.15.92:0/372197479", 2024-03-11T21:48:43.111 INFO:teuthology.orchestra.run.smithi092.stdout: "tid": 4416 2024-03-11T21:48:43.111 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:48:43.111 INFO:teuthology.orchestra.run.smithi092.stdout: "events": [ 2024-03-11T21:48:43.111 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:48:43.111 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "initiated", 2024-03-11T21:48:43.111 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:47:19.268281+0000", 2024-03-11T21:48:43.111 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-03-11T21:48:43.111 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:48:43.111 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:48:43.111 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "header_read", 2024-03-11T21:48:43.111 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:47:19.268281+0000", 2024-03-11T21:48:43.111 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-03-11T21:48:43.111 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:48:43.112 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:48:43.112 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "throttled", 2024-03-11T21:48:43.112 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:47:19.268282+0000", 2024-03-11T21:48:43.112 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 8.2900000000000002e-07 2024-03-11T21:48:43.112 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:48:43.112 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:48:43.112 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "all_read", 2024-03-11T21:48:43.112 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:47:19.268285+0000", 2024-03-11T21:48:43.112 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 3.76e-06 2024-03-11T21:48:43.112 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:48:43.112 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:48:43.112 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "dispatched", 2024-03-11T21:48:43.112 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:47:19.268292+0000", 2024-03-11T21:48:43.112 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 6.5780000000000004e-06 2024-03-11T21:48:43.112 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:48:43.112 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:48:43.113 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "queued_for_pg", 2024-03-11T21:48:43.113 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:47:19.268297+0000", 2024-03-11T21:48:43.113 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 5.4639999999999997e-06 2024-03-11T21:48:43.113 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:48:43.113 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:48:43.113 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-03-11T21:48:43.113 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:47:19.269313+0000", 2024-03-11T21:48:43.113 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.001015981 2024-03-11T21:48:43.113 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:48:43.113 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:48:43.113 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "op must wait for map", 2024-03-11T21:48:43.113 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:47:19.269320+0000", 2024-03-11T21:48:43.114 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 6.9380000000000003e-06 2024-03-11T21:48:43.114 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:48:43.114 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:48:43.114 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-03-11T21:48:43.114 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:47:19.301461+0000", 2024-03-11T21:48:43.114 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.032140780000000001 2024-03-11T21:48:43.114 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:48:43.114 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:48:43.114 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "waiting for peered", 2024-03-11T21:48:43.114 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:47:19.301472+0000", 2024-03-11T21:48:43.114 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.0752e-05 2024-03-11T21:48:43.114 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:48:43.114 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:48:43.115 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-03-11T21:48:43.115 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:47:20.278512+0000", 2024-03-11T21:48:43.115 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.97703962700000002 2024-03-11T21:48:43.115 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:48:43.115 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:48:43.115 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "started", 2024-03-11T21:48:43.115 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:47:20.278874+0000", 2024-03-11T21:48:43.115 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00036258500000000001 2024-03-11T21:48:43.115 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:48:43.115 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:48:43.115 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "done", 2024-03-11T21:48:43.115 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:47:20.281381+0000", 2024-03-11T21:48:43.115 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.0025065830000000002 2024-03-11T21:48:43.115 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-03-11T21:48:43.115 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-03-11T21:48:43.116 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-03-11T21:48:43.116 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:48:43.116 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:48:43.116 INFO:teuthology.orchestra.run.smithi092.stdout: "description": "osd_op(client.34352.0:4539 5.d 5:b4812045:::notify.4:head [watch reconnect cookie 93924550618368 gen 2] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e499)", 2024-03-11T21:48:43.116 INFO:teuthology.orchestra.run.smithi092.stdout: "initiated_at": "2024-03-11T21:48:23.825128+0000", 2024-03-11T21:48:43.116 INFO:teuthology.orchestra.run.smithi092.stdout: "age": 19.147304462000001, 2024-03-11T21:48:43.116 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.011857445, 2024-03-11T21:48:43.116 INFO:teuthology.orchestra.run.smithi092.stdout: "type_data": { 2024-03-11T21:48:43.116 INFO:teuthology.orchestra.run.smithi092.stdout: "flag_point": "started", 2024-03-11T21:48:43.116 INFO:teuthology.orchestra.run.smithi092.stdout: "client_info": { 2024-03-11T21:48:43.116 INFO:teuthology.orchestra.run.smithi092.stdout: "client": "client.34352", 2024-03-11T21:48:43.116 INFO:teuthology.orchestra.run.smithi092.stdout: "client_addr": "v1:172.21.15.92:0/372197479", 2024-03-11T21:48:43.116 INFO:teuthology.orchestra.run.smithi092.stdout: "tid": 4539 2024-03-11T21:48:43.116 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:48:43.116 INFO:teuthology.orchestra.run.smithi092.stdout: "events": [ 2024-03-11T21:48:43.116 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:48:43.117 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "initiated", 2024-03-11T21:48:43.117 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:48:23.825128+0000", 2024-03-11T21:48:43.117 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-03-11T21:48:43.117 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:48:43.117 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:48:43.117 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "header_read", 2024-03-11T21:48:43.117 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:48:23.825128+0000", 2024-03-11T21:48:43.117 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-03-11T21:48:43.117 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:48:43.117 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:48:43.117 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "throttled", 2024-03-11T21:48:43.117 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:48:23.825130+0000", 2024-03-11T21:48:43.117 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.945e-06 2024-03-11T21:48:43.117 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:48:43.117 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:48:43.118 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "all_read", 2024-03-11T21:48:43.118 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:48:23.825136+0000", 2024-03-11T21:48:43.118 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 5.5829999999999997e-06 2024-03-11T21:48:43.118 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:48:43.118 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:48:43.118 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "dispatched", 2024-03-11T21:48:43.118 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:48:23.825147+0000", 2024-03-11T21:48:43.118 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.1129e-05 2024-03-11T21:48:43.118 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:48:43.118 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:48:43.118 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "queued_for_pg", 2024-03-11T21:48:43.118 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:48:23.825156+0000", 2024-03-11T21:48:43.118 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 9.397e-06 2024-03-11T21:48:43.119 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:48:43.119 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:48:43.119 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-03-11T21:48:43.119 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:48:23.825946+0000", 2024-03-11T21:48:43.119 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00078945899999999995 2024-03-11T21:48:43.119 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:48:43.119 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:48:43.119 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "op must wait for map", 2024-03-11T21:48:43.119 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:48:23.825952+0000", 2024-03-11T21:48:43.119 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 6.4130000000000003e-06 2024-03-11T21:48:43.119 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:48:43.119 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:48:43.119 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-03-11T21:48:43.119 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:48:23.835026+0000", 2024-03-11T21:48:43.119 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.0090736629999999992 2024-03-11T21:48:43.120 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:48:43.120 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:48:43.120 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "waiting for peered", 2024-03-11T21:48:43.120 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:48:23.835036+0000", 2024-03-11T21:48:43.120 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.0370999999999999e-05 2024-03-11T21:48:43.120 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:48:43.120 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:48:43.120 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-03-11T21:48:43.120 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:48:24.833881+0000", 2024-03-11T21:48:43.120 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.99884509099999996 2024-03-11T21:48:43.120 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:48:43.120 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:48:43.120 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "waiting for degraded object", 2024-03-11T21:48:43.120 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:48:24.834217+0000", 2024-03-11T21:48:43.120 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.000335395 2024-03-11T21:48:43.120 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:48:43.121 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:48:43.121 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-03-11T21:48:43.121 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:48:24.835658+0000", 2024-03-11T21:48:43.121 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.001441339 2024-03-11T21:48:43.121 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:48:43.121 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:48:43.121 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "started", 2024-03-11T21:48:43.121 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:48:24.835711+0000", 2024-03-11T21:48:43.121 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 5.2626000000000001e-05 2024-03-11T21:48:43.121 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:48:43.121 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:48:43.121 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "done", 2024-03-11T21:48:43.121 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:48:24.836986+0000", 2024-03-11T21:48:43.121 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.001275034 2024-03-11T21:48:43.121 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-03-11T21:48:43.121 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-03-11T21:48:43.122 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-03-11T21:48:43.122 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-03-11T21:48:43.122 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-03-11T21:48:43.122 INFO:teuthology.orchestra.run.smithi092.stdout:} 2024-03-11T21:48:43.321 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:48:42 smithi097 ceph-mon[33317]: pgmap v1093: 132 pgs: 132 active+clean; 582 KiB data, 5.3 GiB used, 531 GiB / 536 GiB avail; 953 B/s rd, 0 op/s 2024-03-11T21:48:43.321 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:48:42 smithi097 ceph-mon[33317]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-03-11T21:48:43.321 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:48:42 smithi097 ceph-mon[33317]: from='client.? v1:172.21.15.92:0/1688281807' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-03-11T21:48:43.321 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:48:42 smithi097 ceph-mon[33317]: osdmap e508: 8 total, 8 up, 6 in 2024-03-11T21:48:43.321 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:48:42 smithi097 ceph-mon[33317]: 5.1d scrub starts 2024-03-11T21:48:43.355 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:48:42 smithi092 ceph-mon[37563]: pgmap v1093: 132 pgs: 132 active+clean; 582 KiB data, 5.3 GiB used, 531 GiB / 536 GiB avail; 953 B/s rd, 0 op/s 2024-03-11T21:48:43.355 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:48:42 smithi092 ceph-mon[37563]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-03-11T21:48:43.355 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:48:42 smithi092 ceph-mon[37563]: from='client.? v1:172.21.15.92:0/1688281807' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-03-11T21:48:43.355 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:48:42 smithi092 ceph-mon[37563]: osdmap e508: 8 total, 8 up, 6 in 2024-03-11T21:48:43.355 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:48:42 smithi092 ceph-mon[37563]: 5.1d scrub starts 2024-03-11T21:48:43.355 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:48:42 smithi092 ceph-mon[29572]: pgmap v1093: 132 pgs: 132 active+clean; 582 KiB data, 5.3 GiB used, 531 GiB / 536 GiB avail; 953 B/s rd, 0 op/s 2024-03-11T21:48:43.356 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:48:42 smithi092 ceph-mon[29572]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-03-11T21:48:43.356 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:48:42 smithi092 ceph-mon[29572]: from='client.? v1:172.21.15.92:0/1688281807' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-03-11T21:48:43.356 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:48:42 smithi092 ceph-mon[29572]: osdmap e508: 8 total, 8 up, 6 in 2024-03-11T21:48:43.356 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:48:42 smithi092 ceph-mon[29572]: 5.1d scrub starts 2024-03-11T21:48:43.585 INFO:journalctl@ceph.iscsi.iscsi.a.smithi097.stdout:Mar 11 21:48:43 smithi097 ceph-00db0016-dfed-11ee-95c8-87774f69a715-iscsi-iscsi-a[60943]: debug there is no tcmu-runner data available 2024-03-11T21:48:44.079 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:48:43 smithi092 ceph-mon[29572]: from='client.24469 v1:172.21.15.97:0/404208310' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:48:44.080 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:48:43 smithi092 ceph-mon[37563]: from='client.24469 v1:172.21.15.97:0/404208310' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:48:44.335 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:48:43 smithi097 ceph-mon[33317]: from='client.24469 v1:172.21.15.97:0/404208310' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:48:44.703 DEBUG:teuthology.orchestra.run.smithi097:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid 00db0016-dfed-11ee-95c8-87774f69a715 -- ceph daemon osd.4 dump_ops_in_flight 2024-03-11T21:48:44.927 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:48:44 smithi092 ceph-mon[29572]: pgmap v1095: 132 pgs: 132 active+clean; 582 KiB data, 5.3 GiB used, 531 GiB / 536 GiB avail; 511 B/s rd, 0 op/s 2024-03-11T21:48:44.950 INFO:teuthology.orchestra.run.smithi097.stderr:Inferring config /var/lib/ceph/00db0016-dfed-11ee-95c8-87774f69a715/mon.b/config 2024-03-11T21:48:45.072 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:48:44 smithi097 ceph-mon[33317]: pgmap v1095: 132 pgs: 132 active+clean; 582 KiB data, 5.3 GiB used, 531 GiB / 536 GiB avail; 511 B/s rd, 0 op/s 2024-03-11T21:48:45.369 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:48:44 smithi092 ceph-mon[37563]: pgmap v1095: 132 pgs: 132 active+clean; 582 KiB data, 5.3 GiB used, 531 GiB / 536 GiB avail; 511 B/s rd, 0 op/s 2024-03-11T21:48:45.573 DEBUG:teuthology.orchestra.run.smithi092:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid 00db0016-dfed-11ee-95c8-87774f69a715 -- ceph osd unset noscrub 2024-03-11T21:48:45.817 INFO:teuthology.orchestra.run.smithi092.stderr:Inferring config /var/lib/ceph/00db0016-dfed-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T21:48:45.936 INFO:teuthology.orchestra.run.smithi092.stderr:reweighted osd.0 to 1 (10000) 2024-03-11T21:48:46.195 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:48:45 smithi097 ceph-mon[33317]: from='client.? v1:172.21.15.92:0/185856718' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]: dispatch 2024-03-11T21:48:46.195 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:48:45 smithi097 ceph-mon[33317]: from='client.36044 ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]: dispatch 2024-03-11T21:48:46.369 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:48:45 smithi092 ceph-mon[29572]: from='client.? v1:172.21.15.92:0/185856718' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]: dispatch 2024-03-11T21:48:46.369 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:48:45 smithi092 ceph-mon[29572]: from='client.36044 ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]: dispatch 2024-03-11T21:48:46.369 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:48:45 smithi092 ceph-mon[37563]: from='client.? v1:172.21.15.92:0/185856718' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]: dispatch 2024-03-11T21:48:46.369 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:48:45 smithi092 ceph-mon[37563]: from='client.36044 ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]: dispatch 2024-03-11T21:48:46.839 INFO:teuthology.orchestra.run.smithi097.stdout:{ 2024-03-11T21:48:46.839 INFO:teuthology.orchestra.run.smithi097.stdout: "ops": [], 2024-03-11T21:48:46.839 INFO:teuthology.orchestra.run.smithi097.stdout: "num_ops": 0 2024-03-11T21:48:46.839 INFO:teuthology.orchestra.run.smithi097.stdout:} 2024-03-11T21:48:46.950 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:48:46 smithi092 ceph-mon[29572]: pgmap v1096: 132 pgs: 1 active+clean+scrubbing, 131 active+clean; 582 KiB data, 5.3 GiB used, 531 GiB / 536 GiB avail; 1.0 KiB/s rd, 1 op/s 2024-03-11T21:48:46.951 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:48:46 smithi092 ceph-mon[29572]: from='client.36044 ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]': finished 2024-03-11T21:48:46.951 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:48:46 smithi092 ceph-mon[29572]: osdmap e509: 8 total, 8 up, 6 in 2024-03-11T21:48:46.951 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:48:46 smithi092 ceph-mon[37563]: pgmap v1096: 132 pgs: 1 active+clean+scrubbing, 131 active+clean; 582 KiB data, 5.3 GiB used, 531 GiB / 536 GiB avail; 1.0 KiB/s rd, 1 op/s 2024-03-11T21:48:46.951 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:48:46 smithi092 ceph-mon[37563]: from='client.36044 ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]': finished 2024-03-11T21:48:46.951 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:48:46 smithi092 ceph-mon[37563]: osdmap e509: 8 total, 8 up, 6 in 2024-03-11T21:48:46.982 DEBUG:teuthology.orchestra.run.smithi092:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid 00db0016-dfed-11ee-95c8-87774f69a715 -- ceph osd reweight 1 1 2024-03-11T21:48:47.138 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:48:46 smithi097 ceph-mon[33317]: pgmap v1096: 132 pgs: 1 active+clean+scrubbing, 131 active+clean; 582 KiB data, 5.3 GiB used, 531 GiB / 536 GiB avail; 1.0 KiB/s rd, 1 op/s 2024-03-11T21:48:47.138 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:48:46 smithi097 ceph-mon[33317]: from='client.36044 ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]': finished 2024-03-11T21:48:47.138 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:48:46 smithi097 ceph-mon[33317]: osdmap e509: 8 total, 8 up, 6 in 2024-03-11T21:48:47.517 DEBUG:teuthology.orchestra.run.smithi097:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid 00db0016-dfed-11ee-95c8-87774f69a715 -- ceph daemon osd.4 dump_blocked_ops 2024-03-11T21:48:47.762 INFO:teuthology.orchestra.run.smithi097.stderr:Inferring config /var/lib/ceph/00db0016-dfed-11ee-95c8-87774f69a715/mon.b/config 2024-03-11T21:48:48.074 INFO:journalctl@ceph.mgr.y.smithi092.stdout:Mar 11 21:48:47 smithi092 ceph-00db0016-dfed-11ee-95c8-87774f69a715-mgr-y[29801]: ::ffff:172.21.15.97 - - [11/Mar/2024:21:48:47] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-11T21:48:48.626 INFO:teuthology.orchestra.run.smithi092.stderr:Inferring config /var/lib/ceph/00db0016-dfed-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T21:48:49.022 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:48:48 smithi092 ceph-mon[37563]: pgmap v1098: 132 pgs: 1 active+clean+scrubbing, 131 active+clean; 582 KiB data, 5.3 GiB used, 531 GiB / 536 GiB avail; 639 B/s rd, 0 op/s 2024-03-11T21:48:49.022 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:48:48 smithi092 ceph-mon[37563]: from='client.? v1:172.21.15.92:0/3203353986' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-11T21:48:49.022 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:48:48 smithi092 ceph-mon[37563]: from='client.36050 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-11T21:48:49.022 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:48:48 smithi092 ceph-mon[29572]: pgmap v1098: 132 pgs: 1 active+clean+scrubbing, 131 active+clean; 582 KiB data, 5.3 GiB used, 531 GiB / 536 GiB avail; 639 B/s rd, 0 op/s 2024-03-11T21:48:49.022 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:48:48 smithi092 ceph-mon[29572]: from='client.? v1:172.21.15.92:0/3203353986' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-11T21:48:49.023 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:48:48 smithi092 ceph-mon[29572]: from='client.36050 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-11T21:48:49.296 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:48:48 smithi097 ceph-mon[33317]: pgmap v1098: 132 pgs: 1 active+clean+scrubbing, 131 active+clean; 582 KiB data, 5.3 GiB used, 531 GiB / 536 GiB avail; 639 B/s rd, 0 op/s 2024-03-11T21:48:49.296 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:48:48 smithi097 ceph-mon[33317]: from='client.? v1:172.21.15.92:0/3203353986' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-11T21:48:49.296 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:48:48 smithi097 ceph-mon[33317]: from='client.36050 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-03-11T21:48:49.596 INFO:teuthology.orchestra.run.smithi092.stderr:noscrub is unset 2024-03-11T21:48:49.907 INFO:teuthology.orchestra.run.smithi097.stdout:{ 2024-03-11T21:48:49.907 INFO:teuthology.orchestra.run.smithi097.stdout: "ops": [], 2024-03-11T21:48:49.907 INFO:teuthology.orchestra.run.smithi097.stdout: "complaint_time": 30, 2024-03-11T21:48:49.908 INFO:teuthology.orchestra.run.smithi097.stdout: "num_blocked_ops": 0 2024-03-11T21:48:49.908 INFO:teuthology.orchestra.run.smithi097.stdout:} 2024-03-11T21:48:50.194 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:48:49 smithi097 ceph-mon[33317]: Health check cleared: OSDMAP_FLAGS (was: noscrub flag(s) set) 2024-03-11T21:48:50.195 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:48:49 smithi097 ceph-mon[33317]: Cluster is now healthy 2024-03-11T21:48:50.195 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:48:49 smithi097 ceph-mon[33317]: from='client.36050 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-03-11T21:48:50.195 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:48:49 smithi097 ceph-mon[33317]: osdmap e510: 8 total, 8 up, 6 in 2024-03-11T21:48:50.206 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:48:49 smithi092 ceph-mon[29572]: Health check cleared: OSDMAP_FLAGS (was: noscrub flag(s) set) 2024-03-11T21:48:50.206 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:48:49 smithi092 ceph-mon[29572]: Cluster is now healthy 2024-03-11T21:48:50.206 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:48:49 smithi092 ceph-mon[29572]: from='client.36050 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-03-11T21:48:50.206 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:48:49 smithi092 ceph-mon[29572]: osdmap e510: 8 total, 8 up, 6 in 2024-03-11T21:48:50.206 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:48:49 smithi092 ceph-mon[37563]: Health check cleared: OSDMAP_FLAGS (was: noscrub flag(s) set) 2024-03-11T21:48:50.206 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:48:49 smithi092 ceph-mon[37563]: Cluster is now healthy 2024-03-11T21:48:50.206 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:48:49 smithi092 ceph-mon[37563]: from='client.36050 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-03-11T21:48:50.206 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:48:49 smithi092 ceph-mon[37563]: osdmap e510: 8 total, 8 up, 6 in 2024-03-11T21:48:50.641 DEBUG:teuthology.orchestra.run.smithi097:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid 00db0016-dfed-11ee-95c8-87774f69a715 -- ceph daemon osd.4 dump_historic_ops 2024-03-11T21:48:50.884 INFO:teuthology.orchestra.run.smithi097.stderr:Inferring config /var/lib/ceph/00db0016-dfed-11ee-95c8-87774f69a715/mon.b/config 2024-03-11T21:48:51.068 DEBUG:teuthology.orchestra.run.smithi092:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid 00db0016-dfed-11ee-95c8-87774f69a715 -- ceph osd unset nodeep-scrub 2024-03-11T21:48:51.229 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:48:50 smithi097 ceph-mon[33317]: pgmap v1099: 132 pgs: 132 active+clean; 582 KiB data, 5.3 GiB used, 531 GiB / 536 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T21:48:51.229 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:48:50 smithi097 ceph-mon[33317]: from='client.? v1:172.21.15.92:0/1405605108' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]: dispatch 2024-03-11T21:48:51.318 INFO:teuthology.orchestra.run.smithi092.stderr:reweighted osd.1 to 1 (10000) 2024-03-11T21:48:51.319 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:48:50 smithi092 ceph-mon[29572]: pgmap v1099: 132 pgs: 132 active+clean; 582 KiB data, 5.3 GiB used, 531 GiB / 536 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T21:48:51.319 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:48:50 smithi092 ceph-mon[29572]: from='client.? v1:172.21.15.92:0/1405605108' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]: dispatch 2024-03-11T21:48:51.319 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:48:50 smithi092 ceph-mon[37563]: pgmap v1099: 132 pgs: 132 active+clean; 582 KiB data, 5.3 GiB used, 531 GiB / 536 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T21:48:51.319 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:48:50 smithi092 ceph-mon[37563]: from='client.? v1:172.21.15.92:0/1405605108' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]: dispatch 2024-03-11T21:48:51.998 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:48:51 smithi092 ceph-mon[29572]: 4.19 scrub starts 2024-03-11T21:48:51.998 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:48:51 smithi092 ceph-mon[29572]: 4.19 scrub ok 2024-03-11T21:48:51.998 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:48:51 smithi092 ceph-mon[29572]: from='client.? v1:172.21.15.92:0/1405605108' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]': finished 2024-03-11T21:48:51.998 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:48:51 smithi092 ceph-mon[29572]: osdmap e511: 8 total, 8 up, 6 in 2024-03-11T21:48:51.998 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:48:51 smithi092 ceph-mon[29572]: purged_snaps scrub starts 2024-03-11T21:48:51.998 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:48:51 smithi092 ceph-mon[29572]: purged_snaps scrub ok 2024-03-11T21:48:51.998 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:48:51 smithi092 ceph-mon[37563]: 4.19 scrub starts 2024-03-11T21:48:51.998 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:48:51 smithi092 ceph-mon[37563]: 4.19 scrub ok 2024-03-11T21:48:51.999 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:48:51 smithi092 ceph-mon[37563]: from='client.? v1:172.21.15.92:0/1405605108' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]': finished 2024-03-11T21:48:51.999 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:48:51 smithi092 ceph-mon[37563]: osdmap e511: 8 total, 8 up, 6 in 2024-03-11T21:48:51.999 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:48:51 smithi092 ceph-mon[37563]: purged_snaps scrub starts 2024-03-11T21:48:51.999 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:48:51 smithi092 ceph-mon[37563]: purged_snaps scrub ok 2024-03-11T21:48:52.024 DEBUG:teuthology.orchestra.run.smithi092:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid 00db0016-dfed-11ee-95c8-87774f69a715 -- ceph osd reweight 2 1 2024-03-11T21:48:52.074 INFO:teuthology.orchestra.run.smithi092.stderr:Inferring config /var/lib/ceph/00db0016-dfed-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T21:48:52.085 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:48:51 smithi097 ceph-mon[33317]: 4.19 scrub starts 2024-03-11T21:48:52.085 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:48:51 smithi097 ceph-mon[33317]: 4.19 scrub ok 2024-03-11T21:48:52.085 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:48:51 smithi097 ceph-mon[33317]: from='client.? v1:172.21.15.92:0/1405605108' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]': finished 2024-03-11T21:48:52.085 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:48:51 smithi097 ceph-mon[33317]: osdmap e511: 8 total, 8 up, 6 in 2024-03-11T21:48:52.085 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:48:51 smithi097 ceph-mon[33317]: purged_snaps scrub starts 2024-03-11T21:48:52.085 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:48:51 smithi097 ceph-mon[33317]: purged_snaps scrub ok 2024-03-11T21:48:52.745 INFO:teuthology.orchestra.run.smithi097.stdout:{ 2024-03-11T21:48:52.746 INFO:teuthology.orchestra.run.smithi097.stdout: "size": 20, 2024-03-11T21:48:52.746 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 600, 2024-03-11T21:48:52.746 INFO:teuthology.orchestra.run.smithi097.stdout: "ops": [ 2024-03-11T21:48:52.746 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:48:52.746 INFO:teuthology.orchestra.run.smithi097.stdout: "description": "osd_repop(client.15744.0:69290 10.6 e371/356 10:64e7935d:::benchmark_data_smithi092_161462_object69289:head v 371'4305, mlcod=371'4303)", 2024-03-11T21:48:52.746 INFO:teuthology.orchestra.run.smithi097.stdout: "initiated_at": "2024-03-11T21:41:27.813843+0000", 2024-03-11T21:48:52.746 INFO:teuthology.orchestra.run.smithi097.stdout: "age": 444.93837225999999, 2024-03-11T21:48:52.746 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.151117429, 2024-03-11T21:48:52.746 INFO:teuthology.orchestra.run.smithi097.stdout: "type_data": { 2024-03-11T21:48:52.746 INFO:teuthology.orchestra.run.smithi097.stdout: "flag_point": "started", 2024-03-11T21:48:52.747 INFO:teuthology.orchestra.run.smithi097.stdout: "events": [ 2024-03-11T21:48:52.747 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:48:52.747 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "initiated", 2024-03-11T21:48:52.747 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:41:27.813843+0000", 2024-03-11T21:48:52.747 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-03-11T21:48:52.747 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:48:52.747 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:48:52.747 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "header_read", 2024-03-11T21:48:52.747 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:41:27.813843+0000", 2024-03-11T21:48:52.747 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-03-11T21:48:52.747 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:48:52.747 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:48:52.747 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "throttled", 2024-03-11T21:48:52.748 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:41:27.813844+0000", 2024-03-11T21:48:52.748 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 4.15e-07 2024-03-11T21:48:52.748 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:48:52.748 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:48:52.748 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "all_read", 2024-03-11T21:48:52.748 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:41:27.813871+0000", 2024-03-11T21:48:52.748 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 2.7078999999999999e-05 2024-03-11T21:48:52.748 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:48:52.748 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:48:52.748 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "dispatched", 2024-03-11T21:48:52.748 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:41:27.813883+0000", 2024-03-11T21:48:52.748 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.1977000000000001e-05 2024-03-11T21:48:52.748 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:48:52.749 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:48:52.749 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "queued_for_pg", 2024-03-11T21:48:52.749 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:41:27.813892+0000", 2024-03-11T21:48:52.749 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 9.0999999999999993e-06 2024-03-11T21:48:52.749 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:48:52.749 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:48:52.749 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "reached_pg", 2024-03-11T21:48:52.749 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:41:27.814328+0000", 2024-03-11T21:48:52.749 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.00043647500000000002 2024-03-11T21:48:52.749 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:48:52.749 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:48:52.749 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "started", 2024-03-11T21:48:52.749 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:41:27.814350+0000", 2024-03-11T21:48:52.750 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 2.1223e-05 2024-03-11T21:48:52.750 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:48:52.750 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:48:52.750 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "done", 2024-03-11T21:48:52.750 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:41:27.964961+0000", 2024-03-11T21:48:52.750 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.15061115999999999 2024-03-11T21:48:52.750 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-03-11T21:48:52.750 INFO:teuthology.orchestra.run.smithi097.stdout: ] 2024-03-11T21:48:52.750 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-03-11T21:48:52.750 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:48:52.750 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:48:52.750 INFO:teuthology.orchestra.run.smithi097.stdout: "description": "osd_op(client.15744.0:69295 10.1 10.ecc5e7a1 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e371)", 2024-03-11T21:48:52.751 INFO:teuthology.orchestra.run.smithi097.stdout: "initiated_at": "2024-03-11T21:41:27.826949+0000", 2024-03-11T21:48:52.751 INFO:teuthology.orchestra.run.smithi097.stdout: "age": 444.92526615499997, 2024-03-11T21:48:52.751 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.14217101400000001, 2024-03-11T21:48:52.751 INFO:teuthology.orchestra.run.smithi097.stdout: "type_data": { 2024-03-11T21:48:52.751 INFO:teuthology.orchestra.run.smithi097.stdout: "flag_point": "reached pg", 2024-03-11T21:48:52.751 INFO:teuthology.orchestra.run.smithi097.stdout: "client_info": { 2024-03-11T21:48:52.751 INFO:teuthology.orchestra.run.smithi097.stdout: "client": "client.15744", 2024-03-11T21:48:52.751 INFO:teuthology.orchestra.run.smithi097.stdout: "client_addr": "v1:172.21.15.92:0/3334000812", 2024-03-11T21:48:52.751 INFO:teuthology.orchestra.run.smithi097.stdout: "tid": 69295 2024-03-11T21:48:52.751 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:48:52.751 INFO:teuthology.orchestra.run.smithi097.stdout: "events": [ 2024-03-11T21:48:52.751 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:48:52.751 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "initiated", 2024-03-11T21:48:52.752 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:41:27.826949+0000", 2024-03-11T21:48:52.752 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-03-11T21:48:52.752 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:48:52.752 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:48:52.752 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "header_read", 2024-03-11T21:48:52.752 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:41:27.826949+0000", 2024-03-11T21:48:52.752 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-03-11T21:48:52.752 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:48:52.752 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:48:52.752 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "throttled", 2024-03-11T21:48:52.752 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:41:27.826952+0000", 2024-03-11T21:48:52.752 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 2.751e-06 2024-03-11T21:48:52.752 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:48:52.753 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:48:52.753 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "all_read", 2024-03-11T21:48:52.753 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:41:27.826991+0000", 2024-03-11T21:48:52.753 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 3.9298000000000002e-05 2024-03-11T21:48:52.753 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:48:52.753 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:48:52.753 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "dispatched", 2024-03-11T21:48:52.753 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:41:27.827007+0000", 2024-03-11T21:48:52.753 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.5141e-05 2024-03-11T21:48:52.753 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:48:52.753 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:48:52.753 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "queued_for_pg", 2024-03-11T21:48:52.753 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:41:27.827016+0000", 2024-03-11T21:48:52.753 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 9.8460000000000003e-06 2024-03-11T21:48:52.753 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:48:52.754 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:48:52.754 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "reached_pg", 2024-03-11T21:48:52.754 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:41:27.969022+0000", 2024-03-11T21:48:52.754 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.14200602800000001 2024-03-11T21:48:52.754 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:48:52.754 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:48:52.754 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "done", 2024-03-11T21:48:52.754 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:41:27.969120+0000", 2024-03-11T21:48:52.754 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 9.7949999999999996e-05 2024-03-11T21:48:52.754 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-03-11T21:48:52.754 INFO:teuthology.orchestra.run.smithi097.stdout: ] 2024-03-11T21:48:52.754 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-03-11T21:48:52.754 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:48:52.754 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:48:52.755 INFO:teuthology.orchestra.run.smithi097.stdout: "description": "osd_op(client.15744.0:69295 10.1 10:85e7a337:::benchmark_data_smithi092_161462_object69294:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e372)", 2024-03-11T21:48:52.755 INFO:teuthology.orchestra.run.smithi097.stdout: "initiated_at": "2024-03-11T21:41:27.827037+0000", 2024-03-11T21:48:52.755 INFO:teuthology.orchestra.run.smithi097.stdout: "age": 444.92517899699999, 2024-03-11T21:48:52.755 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.0170514900000001, 2024-03-11T21:48:52.755 INFO:teuthology.orchestra.run.smithi097.stdout: "type_data": { 2024-03-11T21:48:52.755 INFO:teuthology.orchestra.run.smithi097.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-11T21:48:52.755 INFO:teuthology.orchestra.run.smithi097.stdout: "client_info": { 2024-03-11T21:48:52.755 INFO:teuthology.orchestra.run.smithi097.stdout: "client": "client.15744", 2024-03-11T21:48:52.755 INFO:teuthology.orchestra.run.smithi097.stdout: "client_addr": "v1:172.21.15.92:0/3334000812", 2024-03-11T21:48:52.755 INFO:teuthology.orchestra.run.smithi097.stdout: "tid": 69295 2024-03-11T21:48:52.755 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:48:52.755 INFO:teuthology.orchestra.run.smithi097.stdout: "events": [ 2024-03-11T21:48:52.755 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:48:52.755 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "initiated", 2024-03-11T21:48:52.756 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:41:27.827037+0000", 2024-03-11T21:48:52.756 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-03-11T21:48:52.756 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:48:52.756 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:48:52.756 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "header_read", 2024-03-11T21:48:52.756 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:41:27.827037+0000", 2024-03-11T21:48:52.756 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-03-11T21:48:52.756 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:48:52.756 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:48:52.756 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "throttled", 2024-03-11T21:48:52.756 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:41:27.827037+0000", 2024-03-11T21:48:52.756 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 7.0999999999999998e-07 2024-03-11T21:48:52.757 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:48:52.757 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:48:52.757 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "all_read", 2024-03-11T21:48:52.757 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:41:27.827067+0000", 2024-03-11T21:48:52.757 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 2.9376000000000001e-05 2024-03-11T21:48:52.757 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:48:52.757 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:48:52.757 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "dispatched", 2024-03-11T21:48:52.757 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:41:27.827074+0000", 2024-03-11T21:48:52.757 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 6.8539999999999999e-06 2024-03-11T21:48:52.757 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:48:52.757 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:48:52.757 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "queued_for_pg", 2024-03-11T21:48:52.757 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:41:27.827079+0000", 2024-03-11T21:48:52.757 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 5.6459999999999998e-06 2024-03-11T21:48:52.758 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:48:52.758 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:48:52.758 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "reached_pg", 2024-03-11T21:48:52.758 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:41:27.969273+0000", 2024-03-11T21:48:52.758 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.142194131 2024-03-11T21:48:52.758 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:48:52.758 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:48:52.758 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "waiting for peered", 2024-03-11T21:48:52.758 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:41:27.969284+0000", 2024-03-11T21:48:52.758 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.0491e-05 2024-03-11T21:48:52.758 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:48:52.758 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:48:52.758 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "reached_pg", 2024-03-11T21:48:52.758 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:41:28.830462+0000", 2024-03-11T21:48:52.758 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.861178208 2024-03-11T21:48:52.758 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:48:52.759 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:48:52.759 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "started", 2024-03-11T21:48:52.759 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:41:28.830604+0000", 2024-03-11T21:48:52.759 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.00014221599999999999 2024-03-11T21:48:52.759 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:48:52.759 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:48:52.759 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "waiting for subops from 0,7", 2024-03-11T21:48:52.759 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:41:28.830767+0000", 2024-03-11T21:48:52.759 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.00016259400000000001 2024-03-11T21:48:52.759 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:48:52.759 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:48:52.759 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "op_commit", 2024-03-11T21:48:52.759 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:41:28.836048+0000", 2024-03-11T21:48:52.759 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.0052813030000000002 2024-03-11T21:48:52.759 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:48:52.760 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:48:52.760 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "sub_op_commit_rec", 2024-03-11T21:48:52.760 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:41:28.843753+0000", 2024-03-11T21:48:52.760 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.0077048439999999998 2024-03-11T21:48:52.760 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:48:52.760 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:48:52.760 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "sub_op_commit_rec", 2024-03-11T21:48:52.760 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:41:28.843975+0000", 2024-03-11T21:48:52.760 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.000221788 2024-03-11T21:48:52.760 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:48:52.760 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:48:52.760 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "commit_sent", 2024-03-11T21:48:52.760 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:41:28.844039+0000", 2024-03-11T21:48:52.760 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 6.4274000000000002e-05 2024-03-11T21:48:52.760 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:48:52.761 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:48:52.761 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "done", 2024-03-11T21:48:52.761 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:41:28.844088+0000", 2024-03-11T21:48:52.761 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 4.9054999999999997e-05 2024-03-11T21:48:52.761 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-03-11T21:48:52.761 INFO:teuthology.orchestra.run.smithi097.stdout: ] 2024-03-11T21:48:52.761 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-03-11T21:48:52.761 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:48:52.761 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:48:52.761 INFO:teuthology.orchestra.run.smithi097.stdout: "description": "osd_op(client.15744.0:120195 10.1 10:873498a8:::benchmark_data_smithi092_161462_object10219:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e400)", 2024-03-11T21:48:52.761 INFO:teuthology.orchestra.run.smithi097.stdout: "initiated_at": "2024-03-11T21:42:36.641796+0000", 2024-03-11T21:48:52.761 INFO:teuthology.orchestra.run.smithi097.stdout: "age": 376.11041931800003, 2024-03-11T21:48:52.761 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.12297045199999999, 2024-03-11T21:48:52.761 INFO:teuthology.orchestra.run.smithi097.stdout: "type_data": { 2024-03-11T21:48:52.761 INFO:teuthology.orchestra.run.smithi097.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-11T21:48:52.762 INFO:teuthology.orchestra.run.smithi097.stdout: "client_info": { 2024-03-11T21:48:52.762 INFO:teuthology.orchestra.run.smithi097.stdout: "client": "client.15744", 2024-03-11T21:48:52.762 INFO:teuthology.orchestra.run.smithi097.stdout: "client_addr": "v1:172.21.15.92:0/3334000812", 2024-03-11T21:48:52.762 INFO:teuthology.orchestra.run.smithi097.stdout: "tid": 120195 2024-03-11T21:48:52.762 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:48:52.762 INFO:teuthology.orchestra.run.smithi097.stdout: "events": [ 2024-03-11T21:48:52.762 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:48:52.762 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "initiated", 2024-03-11T21:48:52.762 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:42:36.641796+0000", 2024-03-11T21:48:52.762 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-03-11T21:48:52.762 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:48:52.762 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:48:52.763 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "header_read", 2024-03-11T21:48:52.763 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:42:36.641796+0000", 2024-03-11T21:48:52.763 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-03-11T21:48:52.763 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:48:52.763 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:48:52.763 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "throttled", 2024-03-11T21:48:52.763 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:42:36.641798+0000", 2024-03-11T21:48:52.763 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.9589999999999998e-06 2024-03-11T21:48:52.763 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:48:52.763 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:48:52.763 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "all_read", 2024-03-11T21:48:52.763 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:42:36.641805+0000", 2024-03-11T21:48:52.764 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 7.0090000000000001e-06 2024-03-11T21:48:52.764 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:48:52.764 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:48:52.764 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "dispatched", 2024-03-11T21:48:52.764 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:42:36.641818+0000", 2024-03-11T21:48:52.764 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.3067e-05 2024-03-11T21:48:52.764 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:48:52.764 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:48:52.764 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "queued_for_pg", 2024-03-11T21:48:52.764 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:42:36.641827+0000", 2024-03-11T21:48:52.764 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 8.8859999999999993e-06 2024-03-11T21:48:52.764 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:48:52.764 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:48:52.765 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "reached_pg", 2024-03-11T21:48:52.765 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:42:36.641892+0000", 2024-03-11T21:48:52.765 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 6.4879e-05 2024-03-11T21:48:52.765 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:48:52.765 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:48:52.765 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "started", 2024-03-11T21:48:52.765 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:42:36.642022+0000", 2024-03-11T21:48:52.765 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.00013038899999999999 2024-03-11T21:48:52.765 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:48:52.765 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:48:52.765 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "waiting for subops from 0,7", 2024-03-11T21:48:52.765 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:42:36.642363+0000", 2024-03-11T21:48:52.765 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.00034059699999999999 2024-03-11T21:48:52.766 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:48:52.766 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:48:52.766 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "op_commit", 2024-03-11T21:48:52.766 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:42:36.645218+0000", 2024-03-11T21:48:52.766 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.0028546399999999999 2024-03-11T21:48:52.766 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:48:52.766 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:48:52.766 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "sub_op_commit_rec", 2024-03-11T21:48:52.766 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:42:36.648232+0000", 2024-03-11T21:48:52.766 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.003014255 2024-03-11T21:48:52.766 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:48:52.766 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:48:52.766 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "sub_op_commit_rec", 2024-03-11T21:48:52.767 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:42:36.764601+0000", 2024-03-11T21:48:52.767 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.116369163 2024-03-11T21:48:52.767 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:48:52.767 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:48:52.767 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "commit_sent", 2024-03-11T21:48:52.767 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:42:36.764707+0000", 2024-03-11T21:48:52.767 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.000105682 2024-03-11T21:48:52.767 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:48:52.767 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:48:52.767 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "done", 2024-03-11T21:48:52.767 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:42:36.764767+0000", 2024-03-11T21:48:52.767 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 5.9926000000000002e-05 2024-03-11T21:48:52.767 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-03-11T21:48:52.767 INFO:teuthology.orchestra.run.smithi097.stdout: ] 2024-03-11T21:48:52.768 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-03-11T21:48:52.768 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:48:52.768 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:48:52.768 INFO:teuthology.orchestra.run.smithi097.stdout: "description": "osd_op(client.15744.0:166095 10.10 10:0dd81a90:::benchmark_data_smithi092_161462_object56119:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e412)", 2024-03-11T21:48:52.768 INFO:teuthology.orchestra.run.smithi097.stdout: "initiated_at": "2024-03-11T21:43:28.028828+0000", 2024-03-11T21:48:52.768 INFO:teuthology.orchestra.run.smithi097.stdout: "age": 324.72338788500002, 2024-03-11T21:48:52.768 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.116539198, 2024-03-11T21:48:52.768 INFO:teuthology.orchestra.run.smithi097.stdout: "type_data": { 2024-03-11T21:48:52.768 INFO:teuthology.orchestra.run.smithi097.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-11T21:48:52.768 INFO:teuthology.orchestra.run.smithi097.stdout: "client_info": { 2024-03-11T21:48:52.768 INFO:teuthology.orchestra.run.smithi097.stdout: "client": "client.15744", 2024-03-11T21:48:52.768 INFO:teuthology.orchestra.run.smithi097.stdout: "client_addr": "v1:172.21.15.92:0/3334000812", 2024-03-11T21:48:52.769 INFO:teuthology.orchestra.run.smithi097.stdout: "tid": 166095 2024-03-11T21:48:52.769 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:48:52.769 INFO:teuthology.orchestra.run.smithi097.stdout: "events": [ 2024-03-11T21:48:52.769 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:48:52.769 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "initiated", 2024-03-11T21:48:52.769 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:43:28.028828+0000", 2024-03-11T21:48:52.769 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-03-11T21:48:52.769 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:48:52.769 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:48:52.769 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "header_read", 2024-03-11T21:48:52.769 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:43:28.028828+0000", 2024-03-11T21:48:52.769 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-03-11T21:48:52.769 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:48:52.770 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:48:52.770 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "throttled", 2024-03-11T21:48:52.770 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:43:28.028829+0000", 2024-03-11T21:48:52.770 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.5659999999999999e-06 2024-03-11T21:48:52.770 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:48:52.770 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:48:52.770 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "all_read", 2024-03-11T21:48:52.770 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:43:28.028835+0000", 2024-03-11T21:48:52.770 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 5.7400000000000001e-06 2024-03-11T21:48:52.770 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:48:52.770 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:48:52.770 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "dispatched", 2024-03-11T21:48:52.770 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:43:28.028845+0000", 2024-03-11T21:48:52.771 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.0200999999999999e-05 2024-03-11T21:48:52.771 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:48:52.771 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:48:52.771 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "queued_for_pg", 2024-03-11T21:48:52.771 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:43:28.028853+0000", 2024-03-11T21:48:52.771 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 7.5619999999999998e-06 2024-03-11T21:48:52.771 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:48:52.771 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:48:52.771 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "reached_pg", 2024-03-11T21:48:52.771 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:43:28.105350+0000", 2024-03-11T21:48:52.771 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.076497239999999994 2024-03-11T21:48:52.771 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:48:52.771 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:48:52.771 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "started", 2024-03-11T21:48:52.772 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:43:28.105465+0000", 2024-03-11T21:48:52.772 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.000115223 2024-03-11T21:48:52.772 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:48:52.772 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:48:52.772 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "waiting for subops from 6,7", 2024-03-11T21:48:52.772 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:43:28.105823+0000", 2024-03-11T21:48:52.772 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.00035800900000000001 2024-03-11T21:48:52.772 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:48:52.772 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:48:52.772 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "op_commit", 2024-03-11T21:48:52.772 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:43:28.107963+0000", 2024-03-11T21:48:52.772 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.002140229 2024-03-11T21:48:52.772 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:48:52.772 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:48:52.773 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "sub_op_commit_rec", 2024-03-11T21:48:52.773 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:43:28.115957+0000", 2024-03-11T21:48:52.773 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.0079932709999999997 2024-03-11T21:48:52.773 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:48:52.773 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:48:52.773 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "sub_op_commit_rec", 2024-03-11T21:48:52.773 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:43:28.145210+0000", 2024-03-11T21:48:52.773 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.029253126000000001 2024-03-11T21:48:52.773 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:48:52.773 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:48:52.773 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "commit_sent", 2024-03-11T21:48:52.773 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:43:28.145299+0000", 2024-03-11T21:48:52.773 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 8.9135000000000006e-05 2024-03-11T21:48:52.773 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:48:52.774 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:48:52.774 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "done", 2024-03-11T21:48:52.774 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:43:28.145367+0000", 2024-03-11T21:48:52.774 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 6.7896000000000002e-05 2024-03-11T21:48:52.774 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-03-11T21:48:52.774 INFO:teuthology.orchestra.run.smithi097.stdout: ] 2024-03-11T21:48:52.774 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-03-11T21:48:52.774 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:48:52.774 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:48:52.775 INFO:teuthology.orchestra.run.smithi097.stdout: "description": "osd_op(client.15744.0:166470 10.10 10:0a859604:::benchmark_data_smithi092_161462_object56494:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e412)", 2024-03-11T21:48:52.775 INFO:teuthology.orchestra.run.smithi097.stdout: "initiated_at": "2024-03-11T21:43:28.545582+0000", 2024-03-11T21:48:52.775 INFO:teuthology.orchestra.run.smithi097.stdout: "age": 324.20663332100003, 2024-03-11T21:48:52.775 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.117852763, 2024-03-11T21:48:52.776 INFO:teuthology.orchestra.run.smithi097.stdout: "type_data": { 2024-03-11T21:48:52.776 INFO:teuthology.orchestra.run.smithi097.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-11T21:48:52.776 INFO:teuthology.orchestra.run.smithi097.stdout: "client_info": { 2024-03-11T21:48:52.776 INFO:teuthology.orchestra.run.smithi097.stdout: "client": "client.15744", 2024-03-11T21:48:52.776 INFO:teuthology.orchestra.run.smithi097.stdout: "client_addr": "v1:172.21.15.92:0/3334000812", 2024-03-11T21:48:52.776 INFO:teuthology.orchestra.run.smithi097.stdout: "tid": 166470 2024-03-11T21:48:52.776 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:48:52.776 INFO:teuthology.orchestra.run.smithi097.stdout: "events": [ 2024-03-11T21:48:52.776 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:48:52.776 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "initiated", 2024-03-11T21:48:52.776 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:43:28.545582+0000", 2024-03-11T21:48:52.776 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-03-11T21:48:52.777 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:48:52.777 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:48:52.777 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "header_read", 2024-03-11T21:48:52.777 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:43:28.545582+0000", 2024-03-11T21:48:52.777 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-03-11T21:48:52.777 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:48:52.777 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:48:52.777 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "throttled", 2024-03-11T21:48:52.777 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:43:28.545584+0000", 2024-03-11T21:48:52.777 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.7099999999999999e-06 2024-03-11T21:48:52.777 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:48:52.777 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:48:52.777 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "all_read", 2024-03-11T21:48:52.777 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:43:28.545589+0000", 2024-03-11T21:48:52.778 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 5.2830000000000001e-06 2024-03-11T21:48:52.778 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:48:52.778 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:48:52.778 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "dispatched", 2024-03-11T21:48:52.778 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:43:28.545599+0000", 2024-03-11T21:48:52.778 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 9.9969999999999993e-06 2024-03-11T21:48:52.778 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:48:52.778 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:48:52.778 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "queued_for_pg", 2024-03-11T21:48:52.778 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:43:28.545606+0000", 2024-03-11T21:48:52.778 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 7.1790000000000002e-06 2024-03-11T21:48:52.778 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:48:52.779 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:48:52.779 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "reached_pg", 2024-03-11T21:48:52.779 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:43:28.611373+0000", 2024-03-11T21:48:52.779 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.065766609000000004 2024-03-11T21:48:52.779 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:48:52.779 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:48:52.779 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "started", 2024-03-11T21:48:52.779 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:43:28.611479+0000", 2024-03-11T21:48:52.779 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.000106086 2024-03-11T21:48:52.779 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:48:52.779 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:48:52.779 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "waiting for subops from 6,7", 2024-03-11T21:48:52.779 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:43:28.611683+0000", 2024-03-11T21:48:52.779 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.000203857 2024-03-11T21:48:52.779 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:48:52.780 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:48:52.780 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "op_commit", 2024-03-11T21:48:52.780 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:43:28.624789+0000", 2024-03-11T21:48:52.780 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.013105818999999999 2024-03-11T21:48:52.780 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:48:52.780 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:48:52.780 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "sub_op_commit_rec", 2024-03-11T21:48:52.780 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:43:28.628029+0000", 2024-03-11T21:48:52.780 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.0032404140000000001 2024-03-11T21:48:52.780 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:48:52.780 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:48:52.780 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "sub_op_commit_rec", 2024-03-11T21:48:52.780 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:43:28.663284+0000", 2024-03-11T21:48:52.780 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.035255226000000001 2024-03-11T21:48:52.781 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:48:52.781 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:48:52.781 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "commit_sent", 2024-03-11T21:48:52.781 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:43:28.663351+0000", 2024-03-11T21:48:52.781 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 6.7024000000000001e-05 2024-03-11T21:48:52.781 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:48:52.781 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:48:52.781 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "done", 2024-03-11T21:48:52.781 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:43:28.663435+0000", 2024-03-11T21:48:52.781 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 8.3559000000000004e-05 2024-03-11T21:48:52.781 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-03-11T21:48:52.781 INFO:teuthology.orchestra.run.smithi097.stdout: ] 2024-03-11T21:48:52.781 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-03-11T21:48:52.781 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:48:52.781 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:48:52.782 INFO:teuthology.orchestra.run.smithi097.stdout: "description": "osd_op(client.15744.0:166477 10.10 10:0fe276f5:::benchmark_data_smithi092_161462_object56501:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e412)", 2024-03-11T21:48:52.782 INFO:teuthology.orchestra.run.smithi097.stdout: "initiated_at": "2024-03-11T21:43:28.546600+0000", 2024-03-11T21:48:52.782 INFO:teuthology.orchestra.run.smithi097.stdout: "age": 324.20561572999998, 2024-03-11T21:48:52.782 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.11697613900000001, 2024-03-11T21:48:52.782 INFO:teuthology.orchestra.run.smithi097.stdout: "type_data": { 2024-03-11T21:48:52.782 INFO:teuthology.orchestra.run.smithi097.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-11T21:48:52.782 INFO:teuthology.orchestra.run.smithi097.stdout: "client_info": { 2024-03-11T21:48:52.782 INFO:teuthology.orchestra.run.smithi097.stdout: "client": "client.15744", 2024-03-11T21:48:52.782 INFO:teuthology.orchestra.run.smithi097.stdout: "client_addr": "v1:172.21.15.92:0/3334000812", 2024-03-11T21:48:52.782 INFO:teuthology.orchestra.run.smithi097.stdout: "tid": 166477 2024-03-11T21:48:52.782 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:48:52.782 INFO:teuthology.orchestra.run.smithi097.stdout: "events": [ 2024-03-11T21:48:52.783 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:48:52.783 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "initiated", 2024-03-11T21:48:52.783 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:43:28.546600+0000", 2024-03-11T21:48:52.783 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-03-11T21:48:52.783 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:48:52.783 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:48:52.783 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "header_read", 2024-03-11T21:48:52.783 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:43:28.546600+0000", 2024-03-11T21:48:52.783 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-03-11T21:48:52.783 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:48:52.783 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:48:52.783 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "throttled", 2024-03-11T21:48:52.783 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:43:28.546601+0000", 2024-03-11T21:48:52.783 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.429e-06 2024-03-11T21:48:52.783 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:48:52.784 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:48:52.784 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "all_read", 2024-03-11T21:48:52.784 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:43:28.546611+0000", 2024-03-11T21:48:52.784 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 9.5060000000000001e-06 2024-03-11T21:48:52.784 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:48:52.784 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:48:52.784 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "dispatched", 2024-03-11T21:48:52.784 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:43:28.546621+0000", 2024-03-11T21:48:52.784 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.0669000000000001e-05 2024-03-11T21:48:52.784 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:48:52.784 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:48:52.784 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "queued_for_pg", 2024-03-11T21:48:52.784 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:43:28.546629+0000", 2024-03-11T21:48:52.784 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 7.6059999999999997e-06 2024-03-11T21:48:52.784 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:48:52.784 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:48:52.785 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "reached_pg", 2024-03-11T21:48:52.785 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:43:28.617805+0000", 2024-03-11T21:48:52.785 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.071175955999999999 2024-03-11T21:48:52.785 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:48:52.785 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:48:52.785 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "started", 2024-03-11T21:48:52.785 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:43:28.617927+0000", 2024-03-11T21:48:52.785 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.00012186200000000001 2024-03-11T21:48:52.785 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:48:52.785 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:48:52.785 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "waiting for subops from 6,7", 2024-03-11T21:48:52.785 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:43:28.618403+0000", 2024-03-11T21:48:52.785 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.000475802 2024-03-11T21:48:52.785 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:48:52.786 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:48:52.786 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "op_commit", 2024-03-11T21:48:52.786 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:43:28.624902+0000", 2024-03-11T21:48:52.786 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.0064997500000000003 2024-03-11T21:48:52.786 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:48:52.786 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:48:52.786 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "sub_op_commit_rec", 2024-03-11T21:48:52.786 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:43:28.628066+0000", 2024-03-11T21:48:52.786 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.003163796 2024-03-11T21:48:52.786 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:48:52.786 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:48:52.786 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "sub_op_commit_rec", 2024-03-11T21:48:52.786 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:43:28.663476+0000", 2024-03-11T21:48:52.787 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.035410204000000001 2024-03-11T21:48:52.787 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:48:52.787 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:48:52.787 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "commit_sent", 2024-03-11T21:48:52.787 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:43:28.663531+0000", 2024-03-11T21:48:52.787 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 5.4675000000000001e-05 2024-03-11T21:48:52.787 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:48:52.787 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:48:52.787 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "done", 2024-03-11T21:48:52.787 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:43:28.663576+0000", 2024-03-11T21:48:52.787 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 4.4883999999999999e-05 2024-03-11T21:48:52.787 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-03-11T21:48:52.787 INFO:teuthology.orchestra.run.smithi097.stdout: ] 2024-03-11T21:48:52.787 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-03-11T21:48:52.787 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:48:52.788 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:48:52.788 INFO:teuthology.orchestra.run.smithi097.stdout: "description": "osd_op(client.15744.0:167508 10.10 10:0ab0c736:::benchmark_data_smithi092_161462_object57532:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e412)", 2024-03-11T21:48:52.788 INFO:teuthology.orchestra.run.smithi097.stdout: "initiated_at": "2024-03-11T21:43:29.740748+0000", 2024-03-11T21:48:52.788 INFO:teuthology.orchestra.run.smithi097.stdout: "age": 323.011467563, 2024-03-11T21:48:52.788 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.114395159, 2024-03-11T21:48:52.788 INFO:teuthology.orchestra.run.smithi097.stdout: "type_data": { 2024-03-11T21:48:52.788 INFO:teuthology.orchestra.run.smithi097.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-11T21:48:52.788 INFO:teuthology.orchestra.run.smithi097.stdout: "client_info": { 2024-03-11T21:48:52.788 INFO:teuthology.orchestra.run.smithi097.stdout: "client": "client.15744", 2024-03-11T21:48:52.788 INFO:teuthology.orchestra.run.smithi097.stdout: "client_addr": "v1:172.21.15.92:0/3334000812", 2024-03-11T21:48:52.788 INFO:teuthology.orchestra.run.smithi097.stdout: "tid": 167508 2024-03-11T21:48:52.788 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:48:52.788 INFO:teuthology.orchestra.run.smithi097.stdout: "events": [ 2024-03-11T21:48:52.788 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:48:52.788 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "initiated", 2024-03-11T21:48:52.788 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:43:29.740748+0000", 2024-03-11T21:48:52.789 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-03-11T21:48:52.789 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:48:52.789 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:48:52.789 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "header_read", 2024-03-11T21:48:52.789 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:43:29.740748+0000", 2024-03-11T21:48:52.789 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-03-11T21:48:52.789 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:48:52.789 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:48:52.789 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "throttled", 2024-03-11T21:48:52.789 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:43:29.740749+0000", 2024-03-11T21:48:52.789 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.001e-06 2024-03-11T21:48:52.789 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:48:52.789 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:48:52.789 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "all_read", 2024-03-11T21:48:52.790 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:43:29.740753+0000", 2024-03-11T21:48:52.790 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 4.1409999999999998e-06 2024-03-11T21:48:52.790 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:48:52.790 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:48:52.790 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "dispatched", 2024-03-11T21:48:52.790 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:43:29.740770+0000", 2024-03-11T21:48:52.790 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.6799e-05 2024-03-11T21:48:52.790 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:48:52.790 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:48:52.790 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "queued_for_pg", 2024-03-11T21:48:52.790 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:43:29.740778+0000", 2024-03-11T21:48:52.790 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 7.7640000000000005e-06 2024-03-11T21:48:52.790 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:48:52.790 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:48:52.791 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "reached_pg", 2024-03-11T21:48:52.791 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:43:29.815270+0000", 2024-03-11T21:48:52.791 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.074492244999999999 2024-03-11T21:48:52.791 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:48:52.791 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:48:52.791 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "started", 2024-03-11T21:48:52.791 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:43:29.815380+0000", 2024-03-11T21:48:52.791 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.000109696 2024-03-11T21:48:52.791 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:48:52.791 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:48:52.791 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "waiting for subops from 6,7", 2024-03-11T21:48:52.791 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:43:29.815735+0000", 2024-03-11T21:48:52.791 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.00035573499999999998 2024-03-11T21:48:52.791 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:48:52.791 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:48:52.792 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "op_commit", 2024-03-11T21:48:52.792 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:43:29.817863+0000", 2024-03-11T21:48:52.792 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.0021276369999999999 2024-03-11T21:48:52.792 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:48:52.792 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:48:52.792 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "sub_op_commit_rec", 2024-03-11T21:48:52.792 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:43:29.827837+0000", 2024-03-11T21:48:52.792 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.0099740179999999994 2024-03-11T21:48:52.792 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:48:52.792 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:48:52.792 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "sub_op_commit_rec", 2024-03-11T21:48:52.792 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:43:29.854796+0000", 2024-03-11T21:48:52.792 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.026959278 2024-03-11T21:48:52.792 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:48:52.792 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:48:52.792 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "commit_sent", 2024-03-11T21:48:52.793 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:43:29.854982+0000", 2024-03-11T21:48:52.793 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.00018553800000000001 2024-03-11T21:48:52.793 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:48:52.793 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:48:52.793 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "done", 2024-03-11T21:48:52.793 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:43:29.855143+0000", 2024-03-11T21:48:52.793 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.00016130699999999999 2024-03-11T21:48:52.793 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-03-11T21:48:52.793 INFO:teuthology.orchestra.run.smithi097.stdout: ] 2024-03-11T21:48:52.793 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-03-11T21:48:52.793 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:48:52.793 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:48:52.793 INFO:teuthology.orchestra.run.smithi097.stdout: "description": "osd_op(client.15744.0:167609 10.10 10:0a38f69c:::benchmark_data_smithi092_161462_object57633:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e412)", 2024-03-11T21:48:52.793 INFO:teuthology.orchestra.run.smithi097.stdout: "initiated_at": "2024-03-11T21:43:29.856636+0000", 2024-03-11T21:48:52.793 INFO:teuthology.orchestra.run.smithi097.stdout: "age": 322.89557933899999, 2024-03-11T21:48:52.793 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.112643593, 2024-03-11T21:48:52.794 INFO:teuthology.orchestra.run.smithi097.stdout: "type_data": { 2024-03-11T21:48:52.794 INFO:teuthology.orchestra.run.smithi097.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-11T21:48:52.794 INFO:teuthology.orchestra.run.smithi097.stdout: "client_info": { 2024-03-11T21:48:52.794 INFO:teuthology.orchestra.run.smithi097.stdout: "client": "client.15744", 2024-03-11T21:48:52.794 INFO:teuthology.orchestra.run.smithi097.stdout: "client_addr": "v1:172.21.15.92:0/3334000812", 2024-03-11T21:48:52.794 INFO:teuthology.orchestra.run.smithi097.stdout: "tid": 167609 2024-03-11T21:48:52.794 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:48:52.794 INFO:teuthology.orchestra.run.smithi097.stdout: "events": [ 2024-03-11T21:48:52.794 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:48:52.794 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "initiated", 2024-03-11T21:48:52.794 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:43:29.856636+0000", 2024-03-11T21:48:52.794 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-03-11T21:48:52.794 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:48:52.795 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:48:52.795 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "header_read", 2024-03-11T21:48:52.795 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:43:29.856636+0000", 2024-03-11T21:48:52.795 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-03-11T21:48:52.795 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:48:52.795 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:48:52.795 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "throttled", 2024-03-11T21:48:52.795 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:43:29.856637+0000", 2024-03-11T21:48:52.795 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.26e-06 2024-03-11T21:48:52.795 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:48:52.795 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:48:52.795 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "all_read", 2024-03-11T21:48:52.795 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:43:29.856642+0000", 2024-03-11T21:48:52.795 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 4.6110000000000004e-06 2024-03-11T21:48:52.796 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:48:52.796 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:48:52.796 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "dispatched", 2024-03-11T21:48:52.796 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:43:29.856657+0000", 2024-03-11T21:48:52.796 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.5282999999999999e-05 2024-03-11T21:48:52.796 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:48:52.796 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:48:52.796 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "queued_for_pg", 2024-03-11T21:48:52.796 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:43:29.856664+0000", 2024-03-11T21:48:52.796 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 6.9240000000000001e-06 2024-03-11T21:48:52.796 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:48:52.796 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:48:52.796 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "reached_pg", 2024-03-11T21:48:52.796 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:43:29.863965+0000", 2024-03-11T21:48:52.797 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.0073006900000000003 2024-03-11T21:48:52.797 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:48:52.797 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:48:52.797 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "started", 2024-03-11T21:48:52.797 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:43:29.864807+0000", 2024-03-11T21:48:52.797 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.00084223100000000003 2024-03-11T21:48:52.797 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:48:52.797 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:48:52.797 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "waiting for subops from 6,7", 2024-03-11T21:48:52.797 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:43:29.865470+0000", 2024-03-11T21:48:52.797 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.00066290499999999998 2024-03-11T21:48:52.797 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:48:52.797 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:48:52.798 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "sub_op_commit_rec", 2024-03-11T21:48:52.798 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:43:29.871990+0000", 2024-03-11T21:48:52.798 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.0065194529999999997 2024-03-11T21:48:52.798 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:48:52.798 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:48:52.798 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "op_commit", 2024-03-11T21:48:52.798 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:43:29.900561+0000", 2024-03-11T21:48:52.798 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.028571602000000001 2024-03-11T21:48:52.798 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:48:52.798 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:48:52.798 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "sub_op_commit_rec", 2024-03-11T21:48:52.798 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:43:29.969130+0000", 2024-03-11T21:48:52.798 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.068568585000000001 2024-03-11T21:48:52.799 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:48:52.799 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:48:52.799 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "commit_sent", 2024-03-11T21:48:52.799 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:43:29.969217+0000", 2024-03-11T21:48:52.799 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 8.7251999999999997e-05 2024-03-11T21:48:52.799 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:48:52.799 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:48:52.799 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "done", 2024-03-11T21:48:52.799 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:43:29.969280+0000", 2024-03-11T21:48:52.799 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 6.2797000000000002e-05 2024-03-11T21:48:52.799 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-03-11T21:48:52.799 INFO:teuthology.orchestra.run.smithi097.stdout: ] 2024-03-11T21:48:52.799 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-03-11T21:48:52.799 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:48:52.800 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:48:52.800 INFO:teuthology.orchestra.run.smithi097.stdout: "description": "osd_op(client.15744.0:168249 10.10 10:08f9f7a3:::benchmark_data_smithi092_161462_object58273:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e412)", 2024-03-11T21:48:52.800 INFO:teuthology.orchestra.run.smithi097.stdout: "initiated_at": "2024-03-11T21:43:30.578200+0000", 2024-03-11T21:48:52.800 INFO:teuthology.orchestra.run.smithi097.stdout: "age": 322.17401520300001, 2024-03-11T21:48:52.800 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.116707932, 2024-03-11T21:48:52.800 INFO:teuthology.orchestra.run.smithi097.stdout: "type_data": { 2024-03-11T21:48:52.800 INFO:teuthology.orchestra.run.smithi097.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-11T21:48:52.800 INFO:teuthology.orchestra.run.smithi097.stdout: "client_info": { 2024-03-11T21:48:52.800 INFO:teuthology.orchestra.run.smithi097.stdout: "client": "client.15744", 2024-03-11T21:48:52.800 INFO:teuthology.orchestra.run.smithi097.stdout: "client_addr": "v1:172.21.15.92:0/3334000812", 2024-03-11T21:48:52.800 INFO:teuthology.orchestra.run.smithi097.stdout: "tid": 168249 2024-03-11T21:48:52.800 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:48:52.800 INFO:teuthology.orchestra.run.smithi097.stdout: "events": [ 2024-03-11T21:48:52.800 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:48:52.801 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "initiated", 2024-03-11T21:48:52.801 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:43:30.578200+0000", 2024-03-11T21:48:52.801 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-03-11T21:48:52.801 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:48:52.801 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:48:52.801 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "header_read", 2024-03-11T21:48:52.801 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:43:30.578200+0000", 2024-03-11T21:48:52.801 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-03-11T21:48:52.801 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:48:52.801 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:48:52.801 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "throttled", 2024-03-11T21:48:52.801 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:43:30.578202+0000", 2024-03-11T21:48:52.801 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.7209999999999999e-06 2024-03-11T21:48:52.801 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:48:52.801 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:48:52.802 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "all_read", 2024-03-11T21:48:52.802 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:43:30.578206+0000", 2024-03-11T21:48:52.802 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 4.3900000000000003e-06 2024-03-11T21:48:52.802 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:48:52.802 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:48:52.802 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "dispatched", 2024-03-11T21:48:52.802 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:43:30.578218+0000", 2024-03-11T21:48:52.802 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.1303e-05 2024-03-11T21:48:52.802 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:48:52.802 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:48:52.802 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "queued_for_pg", 2024-03-11T21:48:52.802 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:43:30.578225+0000", 2024-03-11T21:48:52.803 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 7.4699999999999996e-06 2024-03-11T21:48:52.803 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:48:52.803 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:48:52.803 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "reached_pg", 2024-03-11T21:48:52.803 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:43:30.659659+0000", 2024-03-11T21:48:52.803 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.081433503000000004 2024-03-11T21:48:52.803 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:48:52.803 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:48:52.803 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "started", 2024-03-11T21:48:52.803 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:43:30.659782+0000", 2024-03-11T21:48:52.803 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.00012370099999999999 2024-03-11T21:48:52.803 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:48:52.803 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:48:52.804 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "waiting for subops from 6,7", 2024-03-11T21:48:52.804 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:43:30.660169+0000", 2024-03-11T21:48:52.804 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.000386268 2024-03-11T21:48:52.804 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:48:52.804 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:48:52.804 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "sub_op_commit_rec", 2024-03-11T21:48:52.804 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:43:30.668159+0000", 2024-03-11T21:48:52.804 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.0079898429999999999 2024-03-11T21:48:52.804 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:48:52.804 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:48:52.804 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "sub_op_commit_rec", 2024-03-11T21:48:52.804 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:43:30.671302+0000", 2024-03-11T21:48:52.804 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.0031431929999999999 2024-03-11T21:48:52.804 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:48:52.804 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:48:52.805 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "op_commit", 2024-03-11T21:48:52.805 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:43:30.694776+0000", 2024-03-11T21:48:52.805 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.023474343000000002 2024-03-11T21:48:52.805 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:48:52.805 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:48:52.805 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "commit_sent", 2024-03-11T21:48:52.805 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:43:30.694835+0000", 2024-03-11T21:48:52.805 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 5.8502000000000003e-05 2024-03-11T21:48:52.805 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:48:52.805 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:48:52.805 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "done", 2024-03-11T21:48:52.805 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:43:30.694908+0000", 2024-03-11T21:48:52.805 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 7.3695000000000005e-05 2024-03-11T21:48:52.805 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-03-11T21:48:52.805 INFO:teuthology.orchestra.run.smithi097.stdout: ] 2024-03-11T21:48:52.806 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-03-11T21:48:52.806 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:48:52.806 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:48:52.806 INFO:teuthology.orchestra.run.smithi097.stdout: "description": "osd_op(client.15744.0:168455 10.10 10:0ac7b8d6:::benchmark_data_smithi092_161462_object58479:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e412)", 2024-03-11T21:48:52.806 INFO:teuthology.orchestra.run.smithi097.stdout: "initiated_at": "2024-03-11T21:43:30.814472+0000", 2024-03-11T21:48:52.806 INFO:teuthology.orchestra.run.smithi097.stdout: "age": 321.93774374499998, 2024-03-11T21:48:52.806 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.16906712900000001, 2024-03-11T21:48:52.806 INFO:teuthology.orchestra.run.smithi097.stdout: "type_data": { 2024-03-11T21:48:52.806 INFO:teuthology.orchestra.run.smithi097.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-11T21:48:52.806 INFO:teuthology.orchestra.run.smithi097.stdout: "client_info": { 2024-03-11T21:48:52.806 INFO:teuthology.orchestra.run.smithi097.stdout: "client": "client.15744", 2024-03-11T21:48:52.806 INFO:teuthology.orchestra.run.smithi097.stdout: "client_addr": "v1:172.21.15.92:0/3334000812", 2024-03-11T21:48:52.806 INFO:teuthology.orchestra.run.smithi097.stdout: "tid": 168455 2024-03-11T21:48:52.807 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:48:52.807 INFO:teuthology.orchestra.run.smithi097.stdout: "events": [ 2024-03-11T21:48:52.807 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:48:52.807 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "initiated", 2024-03-11T21:48:52.807 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:43:30.814472+0000", 2024-03-11T21:48:52.807 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-03-11T21:48:52.807 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:48:52.807 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:48:52.807 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "header_read", 2024-03-11T21:48:52.807 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:43:30.814472+0000", 2024-03-11T21:48:52.807 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-03-11T21:48:52.807 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:48:52.807 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:48:52.808 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "throttled", 2024-03-11T21:48:52.808 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:43:30.814474+0000", 2024-03-11T21:48:52.808 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.9180000000000001e-06 2024-03-11T21:48:52.808 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:48:52.808 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:48:52.808 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "all_read", 2024-03-11T21:48:52.808 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:43:30.814479+0000", 2024-03-11T21:48:52.808 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 5.2650000000000004e-06 2024-03-11T21:48:52.808 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:48:52.808 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:48:52.808 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "dispatched", 2024-03-11T21:48:52.808 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:43:30.814492+0000", 2024-03-11T21:48:52.808 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.2996999999999999e-05 2024-03-11T21:48:52.808 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:48:52.808 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:48:52.809 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "queued_for_pg", 2024-03-11T21:48:52.809 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:43:30.814500+0000", 2024-03-11T21:48:52.809 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 8.1419999999999993e-06 2024-03-11T21:48:52.809 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:48:52.809 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:48:52.809 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "reached_pg", 2024-03-11T21:48:52.809 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:43:30.872217+0000", 2024-03-11T21:48:52.809 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.057716942 2024-03-11T21:48:52.809 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:48:52.809 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:48:52.809 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "started", 2024-03-11T21:48:52.809 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:43:30.872335+0000", 2024-03-11T21:48:52.809 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.000117736 2024-03-11T21:48:52.809 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:48:52.809 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:48:52.809 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "waiting for subops from 6,7", 2024-03-11T21:48:52.810 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:43:30.872712+0000", 2024-03-11T21:48:52.810 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.00037696799999999999 2024-03-11T21:48:52.810 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:48:52.810 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:48:52.810 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "op_commit", 2024-03-11T21:48:52.810 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:43:30.888330+0000", 2024-03-11T21:48:52.810 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.015618019 2024-03-11T21:48:52.810 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:48:52.810 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:48:52.810 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "sub_op_commit_rec", 2024-03-11T21:48:52.810 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:43:30.888366+0000", 2024-03-11T21:48:52.810 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 3.6591000000000003e-05 2024-03-11T21:48:52.810 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:48:52.811 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:48:52.811 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "sub_op_commit_rec", 2024-03-11T21:48:52.811 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:43:30.983417+0000", 2024-03-11T21:48:52.811 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.095050468999999999 2024-03-11T21:48:52.811 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:48:52.811 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:48:52.811 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "commit_sent", 2024-03-11T21:48:52.811 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:43:30.983472+0000", 2024-03-11T21:48:52.811 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 5.5461999999999999e-05 2024-03-11T21:48:52.811 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:48:52.811 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:48:52.811 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "done", 2024-03-11T21:48:52.811 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:43:30.983539+0000", 2024-03-11T21:48:52.811 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 6.6619999999999996e-05 2024-03-11T21:48:52.811 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-03-11T21:48:52.812 INFO:teuthology.orchestra.run.smithi097.stdout: ] 2024-03-11T21:48:52.812 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-03-11T21:48:52.812 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:48:52.812 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:48:52.812 INFO:teuthology.orchestra.run.smithi097.stdout: "description": "osd_op(client.15744.0:168457 10.10 10:08bf6365:::benchmark_data_smithi092_161462_object58481:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e412)", 2024-03-11T21:48:52.812 INFO:teuthology.orchestra.run.smithi097.stdout: "initiated_at": "2024-03-11T21:43:30.817874+0000", 2024-03-11T21:48:52.812 INFO:teuthology.orchestra.run.smithi097.stdout: "age": 321.93434184300003, 2024-03-11T21:48:52.812 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.165812404, 2024-03-11T21:48:52.812 INFO:teuthology.orchestra.run.smithi097.stdout: "type_data": { 2024-03-11T21:48:52.812 INFO:teuthology.orchestra.run.smithi097.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-11T21:48:52.812 INFO:teuthology.orchestra.run.smithi097.stdout: "client_info": { 2024-03-11T21:48:52.812 INFO:teuthology.orchestra.run.smithi097.stdout: "client": "client.15744", 2024-03-11T21:48:52.812 INFO:teuthology.orchestra.run.smithi097.stdout: "client_addr": "v1:172.21.15.92:0/3334000812", 2024-03-11T21:48:52.812 INFO:teuthology.orchestra.run.smithi097.stdout: "tid": 168457 2024-03-11T21:48:52.812 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:48:52.813 INFO:teuthology.orchestra.run.smithi097.stdout: "events": [ 2024-03-11T21:48:52.813 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:48:52.813 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "initiated", 2024-03-11T21:48:52.813 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:43:30.817874+0000", 2024-03-11T21:48:52.813 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-03-11T21:48:52.813 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:48:52.813 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:48:52.813 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "header_read", 2024-03-11T21:48:52.813 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:43:30.817874+0000", 2024-03-11T21:48:52.813 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-03-11T21:48:52.813 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:48:52.813 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:48:52.813 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "throttled", 2024-03-11T21:48:52.813 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:43:30.817875+0000", 2024-03-11T21:48:52.814 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.674e-06 2024-03-11T21:48:52.814 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:48:52.814 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:48:52.814 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "all_read", 2024-03-11T21:48:52.814 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:43:30.817881+0000", 2024-03-11T21:48:52.814 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 5.3750000000000002e-06 2024-03-11T21:48:52.814 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:48:52.814 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:48:52.814 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "dispatched", 2024-03-11T21:48:52.814 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:43:30.817894+0000", 2024-03-11T21:48:52.814 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.3239e-05 2024-03-11T21:48:52.814 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:48:52.814 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:48:52.815 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "queued_for_pg", 2024-03-11T21:48:52.815 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:43:30.817902+0000", 2024-03-11T21:48:52.815 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 7.6559999999999992e-06 2024-03-11T21:48:52.815 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:48:52.815 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:48:52.815 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "reached_pg", 2024-03-11T21:48:52.815 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:43:30.878126+0000", 2024-03-11T21:48:52.815 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.060223994000000003 2024-03-11T21:48:52.815 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:48:52.815 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:48:52.815 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "started", 2024-03-11T21:48:52.815 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:43:30.878232+0000", 2024-03-11T21:48:52.815 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.000106672 2024-03-11T21:48:52.815 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:48:52.815 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:48:52.816 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "waiting for subops from 6,7", 2024-03-11T21:48:52.816 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:43:30.878581+0000", 2024-03-11T21:48:52.816 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.000349002 2024-03-11T21:48:52.816 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:48:52.816 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:48:52.816 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "op_commit", 2024-03-11T21:48:52.816 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:43:30.888378+0000", 2024-03-11T21:48:52.816 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.0097962039999999993 2024-03-11T21:48:52.816 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:48:52.816 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:48:52.816 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "sub_op_commit_rec", 2024-03-11T21:48:52.816 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:43:30.890979+0000", 2024-03-11T21:48:52.816 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.0026017200000000001 2024-03-11T21:48:52.817 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:48:52.817 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:48:52.817 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "sub_op_commit_rec", 2024-03-11T21:48:52.817 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:43:30.983583+0000", 2024-03-11T21:48:52.817 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.092604231999999995 2024-03-11T21:48:52.817 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:48:52.817 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:48:52.817 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "commit_sent", 2024-03-11T21:48:52.817 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:43:30.983631+0000", 2024-03-11T21:48:52.817 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 4.7299999999999998e-05 2024-03-11T21:48:52.817 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:48:52.817 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:48:52.817 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "done", 2024-03-11T21:48:52.817 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:43:30.983686+0000", 2024-03-11T21:48:52.818 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 5.5336e-05 2024-03-11T21:48:52.818 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-03-11T21:48:52.818 INFO:teuthology.orchestra.run.smithi097.stdout: ] 2024-03-11T21:48:52.818 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-03-11T21:48:52.818 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:48:52.818 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:48:52.818 INFO:teuthology.orchestra.run.smithi097.stdout: "description": "osd_op(client.15744.0:168466 10.10 10:0e4995f9:::benchmark_data_smithi092_161462_object58490:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e412)", 2024-03-11T21:48:52.818 INFO:teuthology.orchestra.run.smithi097.stdout: "initiated_at": "2024-03-11T21:43:30.833613+0000", 2024-03-11T21:48:52.818 INFO:teuthology.orchestra.run.smithi097.stdout: "age": 321.91860292899997, 2024-03-11T21:48:52.818 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.15028402399999999, 2024-03-11T21:48:52.818 INFO:teuthology.orchestra.run.smithi097.stdout: "type_data": { 2024-03-11T21:48:52.818 INFO:teuthology.orchestra.run.smithi097.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-11T21:48:52.819 INFO:teuthology.orchestra.run.smithi097.stdout: "client_info": { 2024-03-11T21:48:52.819 INFO:teuthology.orchestra.run.smithi097.stdout: "client": "client.15744", 2024-03-11T21:48:52.819 INFO:teuthology.orchestra.run.smithi097.stdout: "client_addr": "v1:172.21.15.92:0/3334000812", 2024-03-11T21:48:52.819 INFO:teuthology.orchestra.run.smithi097.stdout: "tid": 168466 2024-03-11T21:48:52.819 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:48:52.819 INFO:teuthology.orchestra.run.smithi097.stdout: "events": [ 2024-03-11T21:48:52.819 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:48:52.819 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "initiated", 2024-03-11T21:48:52.819 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:43:30.833613+0000", 2024-03-11T21:48:52.819 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-03-11T21:48:52.819 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:48:52.819 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:48:52.819 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "header_read", 2024-03-11T21:48:52.819 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:43:30.833613+0000", 2024-03-11T21:48:52.820 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-03-11T21:48:52.820 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:48:52.820 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:48:52.820 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "throttled", 2024-03-11T21:48:52.820 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:43:30.833615+0000", 2024-03-11T21:48:52.820 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.9250000000000002e-06 2024-03-11T21:48:52.820 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:48:52.820 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:48:52.820 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "all_read", 2024-03-11T21:48:52.820 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:43:30.833624+0000", 2024-03-11T21:48:52.820 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 8.9949999999999994e-06 2024-03-11T21:48:52.820 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:48:52.820 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:48:52.820 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "dispatched", 2024-03-11T21:48:52.820 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:43:30.833637+0000", 2024-03-11T21:48:52.821 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.3185e-05 2024-03-11T21:48:52.821 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:48:52.821 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:48:52.821 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "queued_for_pg", 2024-03-11T21:48:52.821 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:43:30.833645+0000", 2024-03-11T21:48:52.821 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 7.9699999999999999e-06 2024-03-11T21:48:52.821 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:48:52.821 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:48:52.821 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "reached_pg", 2024-03-11T21:48:52.821 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:43:30.880617+0000", 2024-03-11T21:48:52.821 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.046971952999999997 2024-03-11T21:48:52.821 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:48:52.821 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:48:52.822 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "started", 2024-03-11T21:48:52.822 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:43:30.880726+0000", 2024-03-11T21:48:52.822 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.000109648 2024-03-11T21:48:52.822 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:48:52.822 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:48:52.822 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "waiting for subops from 6,7", 2024-03-11T21:48:52.822 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:43:30.880993+0000", 2024-03-11T21:48:52.822 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.00026681799999999999 2024-03-11T21:48:52.822 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:48:52.822 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:48:52.822 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "op_commit", 2024-03-11T21:48:52.822 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:43:30.888383+0000", 2024-03-11T21:48:52.822 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.0073894379999999999 2024-03-11T21:48:52.823 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:48:52.823 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:48:52.823 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "sub_op_commit_rec", 2024-03-11T21:48:52.823 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:43:30.891958+0000", 2024-03-11T21:48:52.823 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.003575801 2024-03-11T21:48:52.823 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:48:52.823 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:48:52.823 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "sub_op_commit_rec", 2024-03-11T21:48:52.823 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:43:30.983800+0000", 2024-03-11T21:48:52.823 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.091841171999999999 2024-03-11T21:48:52.823 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:48:52.823 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:48:52.823 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "commit_sent", 2024-03-11T21:48:52.823 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:43:30.983846+0000", 2024-03-11T21:48:52.823 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 4.6025000000000001e-05 2024-03-11T21:48:52.824 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:48:52.824 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:48:52.824 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "done", 2024-03-11T21:48:52.824 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:43:30.983897+0000", 2024-03-11T21:48:52.824 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 5.1094000000000002e-05 2024-03-11T21:48:52.824 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-03-11T21:48:52.824 INFO:teuthology.orchestra.run.smithi097.stdout: ] 2024-03-11T21:48:52.824 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-03-11T21:48:52.824 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:48:52.824 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:48:52.824 INFO:teuthology.orchestra.run.smithi097.stdout: "description": "osd_op(client.15744.0:168485 10.10 10:0e091f3c:::benchmark_data_smithi092_161462_object58509:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e412)", 2024-03-11T21:48:52.824 INFO:teuthology.orchestra.run.smithi097.stdout: "initiated_at": "2024-03-11T21:43:30.854429+0000", 2024-03-11T21:48:52.824 INFO:teuthology.orchestra.run.smithi097.stdout: "age": 321.89778672199998, 2024-03-11T21:48:52.824 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.12984559300000001, 2024-03-11T21:48:52.824 INFO:teuthology.orchestra.run.smithi097.stdout: "type_data": { 2024-03-11T21:48:52.825 INFO:teuthology.orchestra.run.smithi097.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-11T21:48:52.825 INFO:teuthology.orchestra.run.smithi097.stdout: "client_info": { 2024-03-11T21:48:52.825 INFO:teuthology.orchestra.run.smithi097.stdout: "client": "client.15744", 2024-03-11T21:48:52.825 INFO:teuthology.orchestra.run.smithi097.stdout: "client_addr": "v1:172.21.15.92:0/3334000812", 2024-03-11T21:48:52.825 INFO:teuthology.orchestra.run.smithi097.stdout: "tid": 168485 2024-03-11T21:48:52.825 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:48:52.825 INFO:teuthology.orchestra.run.smithi097.stdout: "events": [ 2024-03-11T21:48:52.825 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:48:52.825 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "initiated", 2024-03-11T21:48:52.825 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:43:30.854429+0000", 2024-03-11T21:48:52.825 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-03-11T21:48:52.825 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:48:52.825 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:48:52.826 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "header_read", 2024-03-11T21:48:52.826 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:43:30.854429+0000", 2024-03-11T21:48:52.826 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-03-11T21:48:52.826 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:48:52.826 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:48:52.826 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "throttled", 2024-03-11T21:48:52.826 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:43:30.854430+0000", 2024-03-11T21:48:52.826 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.395e-06 2024-03-11T21:48:52.826 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:48:52.826 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:48:52.826 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "all_read", 2024-03-11T21:48:52.826 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:43:30.854435+0000", 2024-03-11T21:48:52.826 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 4.9409999999999998e-06 2024-03-11T21:48:52.826 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:48:52.827 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:48:52.827 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "dispatched", 2024-03-11T21:48:52.827 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:43:30.854445+0000", 2024-03-11T21:48:52.827 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.0045e-05 2024-03-11T21:48:52.827 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:48:52.827 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:48:52.827 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "queued_for_pg", 2024-03-11T21:48:52.827 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:43:30.854453+0000", 2024-03-11T21:48:52.827 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 7.6190000000000002e-06 2024-03-11T21:48:52.827 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:48:52.827 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:48:52.827 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "reached_pg", 2024-03-11T21:48:52.827 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:43:30.883145+0000", 2024-03-11T21:48:52.827 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.028691952 2024-03-11T21:48:52.828 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:48:52.828 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:48:52.828 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "started", 2024-03-11T21:48:52.828 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:43:30.883253+0000", 2024-03-11T21:48:52.828 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.00010772099999999999 2024-03-11T21:48:52.828 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:48:52.828 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:48:52.828 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "waiting for subops from 6,7", 2024-03-11T21:48:52.828 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:43:30.883637+0000", 2024-03-11T21:48:52.828 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.00038442000000000001 2024-03-11T21:48:52.828 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:48:52.828 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:48:52.828 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "op_commit", 2024-03-11T21:48:52.828 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:43:30.888388+0000", 2024-03-11T21:48:52.829 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.0047506570000000001 2024-03-11T21:48:52.829 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:48:52.829 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:48:52.829 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "sub_op_commit_rec", 2024-03-11T21:48:52.829 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:43:30.891993+0000", 2024-03-11T21:48:52.829 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.0036058040000000002 2024-03-11T21:48:52.829 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:48:52.829 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:48:52.829 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "sub_op_commit_rec", 2024-03-11T21:48:52.829 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:43:30.984183+0000", 2024-03-11T21:48:52.829 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.092189560000000004 2024-03-11T21:48:52.829 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:48:52.829 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:48:52.829 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "commit_sent", 2024-03-11T21:48:52.829 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:43:30.984227+0000", 2024-03-11T21:48:52.830 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 4.3968000000000003e-05 2024-03-11T21:48:52.830 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:48:52.830 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:48:52.830 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "done", 2024-03-11T21:48:52.830 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:43:30.984274+0000", 2024-03-11T21:48:52.830 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 4.7511e-05 2024-03-11T21:48:52.830 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-03-11T21:48:52.830 INFO:teuthology.orchestra.run.smithi097.stdout: ] 2024-03-11T21:48:52.830 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-03-11T21:48:52.830 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:48:52.830 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:48:52.830 INFO:teuthology.orchestra.run.smithi097.stdout: "description": "osd_op(client.15744.0:168775 10.10 10:0f0b9868:::benchmark_data_smithi092_161462_object58799:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e413)", 2024-03-11T21:48:52.830 INFO:teuthology.orchestra.run.smithi097.stdout: "initiated_at": "2024-03-11T21:43:31.190070+0000", 2024-03-11T21:48:52.830 INFO:teuthology.orchestra.run.smithi097.stdout: "age": 321.56214587300002, 2024-03-11T21:48:52.831 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.127788502, 2024-03-11T21:48:52.831 INFO:teuthology.orchestra.run.smithi097.stdout: "type_data": { 2024-03-11T21:48:52.831 INFO:teuthology.orchestra.run.smithi097.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-11T21:48:52.831 INFO:teuthology.orchestra.run.smithi097.stdout: "client_info": { 2024-03-11T21:48:52.831 INFO:teuthology.orchestra.run.smithi097.stdout: "client": "client.15744", 2024-03-11T21:48:52.831 INFO:teuthology.orchestra.run.smithi097.stdout: "client_addr": "v1:172.21.15.92:0/3334000812", 2024-03-11T21:48:52.831 INFO:teuthology.orchestra.run.smithi097.stdout: "tid": 168775 2024-03-11T21:48:52.831 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:48:52.831 INFO:teuthology.orchestra.run.smithi097.stdout: "events": [ 2024-03-11T21:48:52.831 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:48:52.831 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "initiated", 2024-03-11T21:48:52.831 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:43:31.190070+0000", 2024-03-11T21:48:52.831 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-03-11T21:48:52.831 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:48:52.831 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:48:52.832 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "header_read", 2024-03-11T21:48:52.832 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:43:31.190070+0000", 2024-03-11T21:48:52.832 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-03-11T21:48:52.832 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:48:52.832 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:48:52.832 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "throttled", 2024-03-11T21:48:52.832 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:43:31.190071+0000", 2024-03-11T21:48:52.832 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.7659999999999999e-06 2024-03-11T21:48:52.832 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:48:52.832 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:48:52.832 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "all_read", 2024-03-11T21:48:52.832 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:43:31.190076+0000", 2024-03-11T21:48:52.832 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 4.4769999999999997e-06 2024-03-11T21:48:52.832 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:48:52.833 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:48:52.833 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "dispatched", 2024-03-11T21:48:52.833 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:43:31.190086+0000", 2024-03-11T21:48:52.833 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.0241000000000001e-05 2024-03-11T21:48:52.833 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:48:52.833 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:48:52.833 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "queued_for_pg", 2024-03-11T21:48:52.833 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:43:31.190095+0000", 2024-03-11T21:48:52.833 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 8.5189999999999992e-06 2024-03-11T21:48:52.833 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:48:52.833 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:48:52.833 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "reached_pg", 2024-03-11T21:48:52.833 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:43:31.266873+0000", 2024-03-11T21:48:52.834 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.076778585999999996 2024-03-11T21:48:52.834 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:48:52.834 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:48:52.834 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "started", 2024-03-11T21:48:52.834 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:43:31.267022+0000", 2024-03-11T21:48:52.834 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.000148858 2024-03-11T21:48:52.834 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:48:52.834 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:48:52.834 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "waiting for subops from 6,7", 2024-03-11T21:48:52.834 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:43:31.267292+0000", 2024-03-11T21:48:52.834 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.00027015900000000001 2024-03-11T21:48:52.834 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:48:52.834 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:48:52.835 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "op_commit", 2024-03-11T21:48:52.835 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:43:31.282627+0000", 2024-03-11T21:48:52.835 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.015334831 2024-03-11T21:48:52.835 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:48:52.835 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:48:52.835 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "sub_op_commit_rec", 2024-03-11T21:48:52.835 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:43:31.286267+0000", 2024-03-11T21:48:52.835 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.003639626 2024-03-11T21:48:52.835 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:48:52.835 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:48:52.835 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "sub_op_commit_rec", 2024-03-11T21:48:52.835 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:43:31.317752+0000", 2024-03-11T21:48:52.835 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.031485338000000002 2024-03-11T21:48:52.835 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:48:52.836 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:48:52.836 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "commit_sent", 2024-03-11T21:48:52.836 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:43:31.317800+0000", 2024-03-11T21:48:52.836 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 4.7783e-05 2024-03-11T21:48:52.836 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:48:52.836 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:48:52.836 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "done", 2024-03-11T21:48:52.836 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:43:31.317858+0000", 2024-03-11T21:48:52.836 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 5.8318e-05 2024-03-11T21:48:52.836 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-03-11T21:48:52.836 INFO:teuthology.orchestra.run.smithi097.stdout: ] 2024-03-11T21:48:52.836 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-03-11T21:48:52.836 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:48:52.836 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:48:52.836 INFO:teuthology.orchestra.run.smithi097.stdout: "description": "osd_op(client.15744.0:170193 10.10 10:0de9636d:::benchmark_data_smithi092_161462_object60217:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e413)", 2024-03-11T21:48:52.837 INFO:teuthology.orchestra.run.smithi097.stdout: "initiated_at": "2024-03-11T21:43:32.812912+0000", 2024-03-11T21:48:52.837 INFO:teuthology.orchestra.run.smithi097.stdout: "age": 319.939303725, 2024-03-11T21:48:52.837 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.11707511399999999, 2024-03-11T21:48:52.837 INFO:teuthology.orchestra.run.smithi097.stdout: "type_data": { 2024-03-11T21:48:52.837 INFO:teuthology.orchestra.run.smithi097.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-11T21:48:52.837 INFO:teuthology.orchestra.run.smithi097.stdout: "client_info": { 2024-03-11T21:48:52.837 INFO:teuthology.orchestra.run.smithi097.stdout: "client": "client.15744", 2024-03-11T21:48:52.837 INFO:teuthology.orchestra.run.smithi097.stdout: "client_addr": "v1:172.21.15.92:0/3334000812", 2024-03-11T21:48:52.837 INFO:teuthology.orchestra.run.smithi097.stdout: "tid": 170193 2024-03-11T21:48:52.837 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:48:52.837 INFO:teuthology.orchestra.run.smithi097.stdout: "events": [ 2024-03-11T21:48:52.837 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:48:52.837 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "initiated", 2024-03-11T21:48:52.837 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:43:32.812912+0000", 2024-03-11T21:48:52.838 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-03-11T21:48:52.838 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:48:52.838 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:48:52.838 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "header_read", 2024-03-11T21:48:52.838 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:43:32.812912+0000", 2024-03-11T21:48:52.838 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-03-11T21:48:52.838 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:48:52.838 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:48:52.838 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "throttled", 2024-03-11T21:48:52.838 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:43:32.812914+0000", 2024-03-11T21:48:52.838 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.7740000000000001e-06 2024-03-11T21:48:52.838 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:48:52.838 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:48:52.838 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "all_read", 2024-03-11T21:48:52.839 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:43:32.812919+0000", 2024-03-11T21:48:52.839 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 5.0019999999999997e-06 2024-03-11T21:48:52.839 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:48:52.839 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:48:52.839 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "dispatched", 2024-03-11T21:48:52.839 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:43:32.812928+0000", 2024-03-11T21:48:52.839 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 8.9050000000000003e-06 2024-03-11T21:48:52.839 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:48:52.839 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:48:52.839 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "queued_for_pg", 2024-03-11T21:48:52.839 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:43:32.812935+0000", 2024-03-11T21:48:52.839 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 7.092e-06 2024-03-11T21:48:52.839 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:48:52.839 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:48:52.840 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "reached_pg", 2024-03-11T21:48:52.840 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:43:32.828122+0000", 2024-03-11T21:48:52.840 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.015187256 2024-03-11T21:48:52.840 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:48:52.840 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:48:52.840 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "started", 2024-03-11T21:48:52.840 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:43:32.828238+0000", 2024-03-11T21:48:52.840 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.000115807 2024-03-11T21:48:52.840 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:48:52.840 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:48:52.840 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "waiting for subops from 6,7", 2024-03-11T21:48:52.840 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:43:32.828632+0000", 2024-03-11T21:48:52.840 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.00039436499999999998 2024-03-11T21:48:52.840 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:48:52.841 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:48:52.841 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "op_commit", 2024-03-11T21:48:52.841 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:43:32.832176+0000", 2024-03-11T21:48:52.841 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.0035439740000000001 2024-03-11T21:48:52.841 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:48:52.841 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:48:52.841 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "sub_op_commit_rec", 2024-03-11T21:48:52.841 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:43:32.929465+0000", 2024-03-11T21:48:52.841 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.097288702000000005 2024-03-11T21:48:52.841 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:48:52.841 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:48:52.841 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "sub_op_commit_rec", 2024-03-11T21:48:52.841 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:43:32.929633+0000", 2024-03-11T21:48:52.841 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.00016832300000000001 2024-03-11T21:48:52.842 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:48:52.842 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:48:52.842 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "commit_sent", 2024-03-11T21:48:52.842 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:43:32.929832+0000", 2024-03-11T21:48:52.842 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.00019918099999999999 2024-03-11T21:48:52.842 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:48:52.842 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:48:52.842 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "done", 2024-03-11T21:48:52.842 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:43:32.929987+0000", 2024-03-11T21:48:52.842 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.000154733 2024-03-11T21:48:52.842 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-03-11T21:48:52.842 INFO:teuthology.orchestra.run.smithi097.stdout: ] 2024-03-11T21:48:52.842 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-03-11T21:48:52.842 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:48:52.843 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:48:52.843 INFO:teuthology.orchestra.run.smithi097.stdout: "description": "osd_op(client.15744.0:170432 10.10 10:09914d52:::benchmark_data_smithi092_161462_object60456:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e413)", 2024-03-11T21:48:52.843 INFO:teuthology.orchestra.run.smithi097.stdout: "initiated_at": "2024-03-11T21:43:33.063255+0000", 2024-03-11T21:48:52.843 INFO:teuthology.orchestra.run.smithi097.stdout: "age": 319.68896075700002, 2024-03-11T21:48:52.844 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.110706422, 2024-03-11T21:48:52.844 INFO:teuthology.orchestra.run.smithi097.stdout: "type_data": { 2024-03-11T21:48:52.844 INFO:teuthology.orchestra.run.smithi097.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-11T21:48:52.844 INFO:teuthology.orchestra.run.smithi097.stdout: "client_info": { 2024-03-11T21:48:52.844 INFO:teuthology.orchestra.run.smithi097.stdout: "client": "client.15744", 2024-03-11T21:48:52.844 INFO:teuthology.orchestra.run.smithi097.stdout: "client_addr": "v1:172.21.15.92:0/3334000812", 2024-03-11T21:48:52.844 INFO:teuthology.orchestra.run.smithi097.stdout: "tid": 170432 2024-03-11T21:48:52.844 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:48:52.844 INFO:teuthology.orchestra.run.smithi097.stdout: "events": [ 2024-03-11T21:48:52.844 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:48:52.844 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "initiated", 2024-03-11T21:48:52.844 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:43:33.063255+0000", 2024-03-11T21:48:52.844 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-03-11T21:48:52.844 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:48:52.844 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:48:52.845 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "header_read", 2024-03-11T21:48:52.845 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:43:33.063255+0000", 2024-03-11T21:48:52.845 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-03-11T21:48:52.845 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:48:52.845 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:48:52.845 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "throttled", 2024-03-11T21:48:52.845 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:43:33.063257+0000", 2024-03-11T21:48:52.845 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.7260000000000001e-06 2024-03-11T21:48:52.845 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:48:52.845 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:48:52.845 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "all_read", 2024-03-11T21:48:52.845 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:43:33.063261+0000", 2024-03-11T21:48:52.845 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 4.6600000000000003e-06 2024-03-11T21:48:52.845 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:48:52.846 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:48:52.846 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "dispatched", 2024-03-11T21:48:52.846 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:43:33.063269+0000", 2024-03-11T21:48:52.846 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 7.9880000000000004e-06 2024-03-11T21:48:52.846 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:48:52.846 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:48:52.846 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "queued_for_pg", 2024-03-11T21:48:52.846 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:43:33.063276+0000", 2024-03-11T21:48:52.846 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 6.4579999999999998e-06 2024-03-11T21:48:52.846 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:48:52.846 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:48:52.846 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "reached_pg", 2024-03-11T21:48:52.846 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:43:33.064043+0000", 2024-03-11T21:48:52.847 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.00076738100000000003 2024-03-11T21:48:52.847 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:48:52.847 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:48:52.847 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "started", 2024-03-11T21:48:52.847 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:43:33.064417+0000", 2024-03-11T21:48:52.847 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.00037403499999999999 2024-03-11T21:48:52.847 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:48:52.847 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:48:52.847 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "waiting for subops from 6,7", 2024-03-11T21:48:52.847 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:43:33.065074+0000", 2024-03-11T21:48:52.847 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.00065659799999999999 2024-03-11T21:48:52.847 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:48:52.847 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:48:52.847 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "sub_op_commit_rec", 2024-03-11T21:48:52.847 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:43:33.083017+0000", 2024-03-11T21:48:52.848 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.017943187999999999 2024-03-11T21:48:52.848 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:48:52.848 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:48:52.848 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "op_commit", 2024-03-11T21:48:52.848 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:43:33.086802+0000", 2024-03-11T21:48:52.848 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.003784843 2024-03-11T21:48:52.848 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:48:52.848 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:48:52.848 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "sub_op_commit_rec", 2024-03-11T21:48:52.848 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:43:33.173859+0000", 2024-03-11T21:48:52.848 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.087057128999999997 2024-03-11T21:48:52.848 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:48:52.848 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:48:52.848 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "commit_sent", 2024-03-11T21:48:52.849 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:43:33.173905+0000", 2024-03-11T21:48:52.849 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 4.6523000000000002e-05 2024-03-11T21:48:52.849 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:48:52.849 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:48:52.849 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "done", 2024-03-11T21:48:52.849 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:43:33.173961+0000", 2024-03-11T21:48:52.849 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 5.5893000000000002e-05 2024-03-11T21:48:52.849 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-03-11T21:48:52.849 INFO:teuthology.orchestra.run.smithi097.stdout: ] 2024-03-11T21:48:52.849 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-03-11T21:48:52.849 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:48:52.849 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:48:52.849 INFO:teuthology.orchestra.run.smithi097.stdout: "description": "osd_op(client.15744.0:170597 10.10 10:0c0c80f7:::benchmark_data_smithi092_161462_object60621:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e413)", 2024-03-11T21:48:52.849 INFO:teuthology.orchestra.run.smithi097.stdout: "initiated_at": "2024-03-11T21:43:33.250794+0000", 2024-03-11T21:48:52.850 INFO:teuthology.orchestra.run.smithi097.stdout: "age": 319.50142112999998, 2024-03-11T21:48:52.850 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.11162490799999999, 2024-03-11T21:48:52.850 INFO:teuthology.orchestra.run.smithi097.stdout: "type_data": { 2024-03-11T21:48:52.850 INFO:teuthology.orchestra.run.smithi097.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-11T21:48:52.850 INFO:teuthology.orchestra.run.smithi097.stdout: "client_info": { 2024-03-11T21:48:52.850 INFO:teuthology.orchestra.run.smithi097.stdout: "client": "client.15744", 2024-03-11T21:48:52.850 INFO:teuthology.orchestra.run.smithi097.stdout: "client_addr": "v1:172.21.15.92:0/3334000812", 2024-03-11T21:48:52.850 INFO:teuthology.orchestra.run.smithi097.stdout: "tid": 170597 2024-03-11T21:48:52.850 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:48:52.850 INFO:teuthology.orchestra.run.smithi097.stdout: "events": [ 2024-03-11T21:48:52.850 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:48:52.850 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "initiated", 2024-03-11T21:48:52.850 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:43:33.250794+0000", 2024-03-11T21:48:52.851 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-03-11T21:48:52.851 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:48:52.851 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:48:52.851 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "header_read", 2024-03-11T21:48:52.851 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:43:33.250794+0000", 2024-03-11T21:48:52.851 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-03-11T21:48:52.851 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:48:52.851 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:48:52.851 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "throttled", 2024-03-11T21:48:52.851 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:43:33.250796+0000", 2024-03-11T21:48:52.851 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.7460000000000001e-06 2024-03-11T21:48:52.851 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:48:52.851 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:48:52.851 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "all_read", 2024-03-11T21:48:52.851 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:43:33.250801+0000", 2024-03-11T21:48:52.852 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 4.9289999999999997e-06 2024-03-11T21:48:52.852 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:48:52.852 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:48:52.852 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "dispatched", 2024-03-11T21:48:52.852 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:43:33.250814+0000", 2024-03-11T21:48:52.852 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.2812e-05 2024-03-11T21:48:52.852 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:48:52.852 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:48:52.852 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "queued_for_pg", 2024-03-11T21:48:52.852 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:43:33.250822+0000", 2024-03-11T21:48:52.852 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 7.7030000000000006e-06 2024-03-11T21:48:52.852 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:48:52.852 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:48:52.852 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "reached_pg", 2024-03-11T21:48:52.853 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:43:33.333823+0000", 2024-03-11T21:48:52.853 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.083001349000000002 2024-03-11T21:48:52.853 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:48:52.853 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:48:52.853 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "started", 2024-03-11T21:48:52.853 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:43:33.333965+0000", 2024-03-11T21:48:52.853 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.00014242399999999999 2024-03-11T21:48:52.853 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:48:52.853 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:48:52.853 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "waiting for subops from 6,7", 2024-03-11T21:48:52.853 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:43:33.334349+0000", 2024-03-11T21:48:52.853 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.00038314700000000002 2024-03-11T21:48:52.853 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:48:52.853 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:48:52.854 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "sub_op_commit_rec", 2024-03-11T21:48:52.854 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:43:33.340134+0000", 2024-03-11T21:48:52.854 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.00578504 2024-03-11T21:48:52.854 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:48:52.854 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:48:52.854 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "op_commit", 2024-03-11T21:48:52.854 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:43:33.344266+0000", 2024-03-11T21:48:52.854 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.0041321250000000004 2024-03-11T21:48:52.854 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:48:52.854 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:48:52.854 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "sub_op_commit_rec", 2024-03-11T21:48:52.854 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:43:33.362285+0000", 2024-03-11T21:48:52.854 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.018019215000000002 2024-03-11T21:48:52.855 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:48:52.855 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:48:52.855 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "commit_sent", 2024-03-11T21:48:52.855 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:43:33.362349+0000", 2024-03-11T21:48:52.855 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 6.4364999999999998e-05 2024-03-11T21:48:52.855 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:48:52.855 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:48:52.855 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "done", 2024-03-11T21:48:52.855 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:43:33.362419+0000", 2024-03-11T21:48:52.855 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 7.0053000000000002e-05 2024-03-11T21:48:52.855 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-03-11T21:48:52.855 INFO:teuthology.orchestra.run.smithi097.stdout: ] 2024-03-11T21:48:52.855 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-03-11T21:48:52.855 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:48:52.856 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:48:52.856 INFO:teuthology.orchestra.run.smithi097.stdout: "description": "osd_op(client.15744.0:171737 10.10 10:098cc278:::benchmark_data_smithi092_161462_object61761:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e413)", 2024-03-11T21:48:52.856 INFO:teuthology.orchestra.run.smithi097.stdout: "initiated_at": "2024-03-11T21:43:34.576591+0000", 2024-03-11T21:48:52.856 INFO:teuthology.orchestra.run.smithi097.stdout: "age": 318.175624185, 2024-03-11T21:48:52.856 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.121563038, 2024-03-11T21:48:52.856 INFO:teuthology.orchestra.run.smithi097.stdout: "type_data": { 2024-03-11T21:48:52.856 INFO:teuthology.orchestra.run.smithi097.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-11T21:48:52.856 INFO:teuthology.orchestra.run.smithi097.stdout: "client_info": { 2024-03-11T21:48:52.856 INFO:teuthology.orchestra.run.smithi097.stdout: "client": "client.15744", 2024-03-11T21:48:52.856 INFO:teuthology.orchestra.run.smithi097.stdout: "client_addr": "v1:172.21.15.92:0/3334000812", 2024-03-11T21:48:52.856 INFO:teuthology.orchestra.run.smithi097.stdout: "tid": 171737 2024-03-11T21:48:52.856 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:48:52.856 INFO:teuthology.orchestra.run.smithi097.stdout: "events": [ 2024-03-11T21:48:52.856 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:48:52.856 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "initiated", 2024-03-11T21:48:52.856 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:43:34.576591+0000", 2024-03-11T21:48:52.857 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-03-11T21:48:52.857 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:48:52.857 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:48:52.857 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "header_read", 2024-03-11T21:48:52.857 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:43:34.576591+0000", 2024-03-11T21:48:52.857 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-03-11T21:48:52.857 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:48:52.857 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:48:52.857 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "throttled", 2024-03-11T21:48:52.857 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:43:34.576593+0000", 2024-03-11T21:48:52.857 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.658e-06 2024-03-11T21:48:52.857 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:48:52.857 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:48:52.857 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "all_read", 2024-03-11T21:48:52.858 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:43:34.576598+0000", 2024-03-11T21:48:52.858 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 4.7859999999999996e-06 2024-03-11T21:48:52.858 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:48:52.858 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:48:52.858 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "dispatched", 2024-03-11T21:48:52.858 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:43:34.576607+0000", 2024-03-11T21:48:52.858 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 9.0850000000000003e-06 2024-03-11T21:48:52.858 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:48:52.858 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:48:52.858 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "queued_for_pg", 2024-03-11T21:48:52.858 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:43:34.576614+0000", 2024-03-11T21:48:52.858 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 7.0740000000000004e-06 2024-03-11T21:48:52.858 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:48:52.858 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:48:52.858 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "reached_pg", 2024-03-11T21:48:52.859 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:43:34.596536+0000", 2024-03-11T21:48:52.859 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.019921642999999999 2024-03-11T21:48:52.859 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:48:52.859 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:48:52.859 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "started", 2024-03-11T21:48:52.859 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:43:34.596612+0000", 2024-03-11T21:48:52.859 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 7.6410999999999997e-05 2024-03-11T21:48:52.859 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:48:52.859 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:48:52.859 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "waiting for subops from 6,7", 2024-03-11T21:48:52.859 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:43:34.596687+0000", 2024-03-11T21:48:52.859 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 7.4896999999999998e-05 2024-03-11T21:48:52.859 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:48:52.860 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:48:52.860 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "op_commit", 2024-03-11T21:48:52.860 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:43:34.602042+0000", 2024-03-11T21:48:52.860 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.0053553250000000002 2024-03-11T21:48:52.860 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:48:52.860 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:48:52.860 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "sub_op_commit_rec", 2024-03-11T21:48:52.860 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:43:34.616639+0000", 2024-03-11T21:48:52.860 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.014596444 2024-03-11T21:48:52.860 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:48:52.860 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:48:52.860 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "sub_op_commit_rec", 2024-03-11T21:48:52.860 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:43:34.698024+0000", 2024-03-11T21:48:52.860 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.081385682000000001 2024-03-11T21:48:52.860 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:48:52.860 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:48:52.861 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "commit_sent", 2024-03-11T21:48:52.861 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:43:34.698081+0000", 2024-03-11T21:48:52.861 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 5.6987999999999997e-05 2024-03-11T21:48:52.861 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:48:52.861 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:48:52.861 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "done", 2024-03-11T21:48:52.861 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:43:34.698154+0000", 2024-03-11T21:48:52.861 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 7.3045000000000003e-05 2024-03-11T21:48:52.861 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-03-11T21:48:52.861 INFO:teuthology.orchestra.run.smithi097.stdout: ] 2024-03-11T21:48:52.861 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-03-11T21:48:52.861 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:48:52.861 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:48:52.861 INFO:teuthology.orchestra.run.smithi097.stdout: "description": "osd_op(client.15744.0:171740 10.10 10:08235f9d:::benchmark_data_smithi092_161462_object61764:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e413)", 2024-03-11T21:48:52.861 INFO:teuthology.orchestra.run.smithi097.stdout: "initiated_at": "2024-03-11T21:43:34.579569+0000", 2024-03-11T21:48:52.861 INFO:teuthology.orchestra.run.smithi097.stdout: "age": 318.17264684100002, 2024-03-11T21:48:52.862 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.118767341, 2024-03-11T21:48:52.862 INFO:teuthology.orchestra.run.smithi097.stdout: "type_data": { 2024-03-11T21:48:52.862 INFO:teuthology.orchestra.run.smithi097.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-11T21:48:52.862 INFO:teuthology.orchestra.run.smithi097.stdout: "client_info": { 2024-03-11T21:48:52.862 INFO:teuthology.orchestra.run.smithi097.stdout: "client": "client.15744", 2024-03-11T21:48:52.862 INFO:teuthology.orchestra.run.smithi097.stdout: "client_addr": "v1:172.21.15.92:0/3334000812", 2024-03-11T21:48:52.862 INFO:teuthology.orchestra.run.smithi097.stdout: "tid": 171740 2024-03-11T21:48:52.862 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:48:52.862 INFO:teuthology.orchestra.run.smithi097.stdout: "events": [ 2024-03-11T21:48:52.862 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:48:52.862 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "initiated", 2024-03-11T21:48:52.863 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:43:34.579569+0000", 2024-03-11T21:48:52.863 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-03-11T21:48:52.863 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:48:52.863 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:48:52.863 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "header_read", 2024-03-11T21:48:52.863 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:43:34.579569+0000", 2024-03-11T21:48:52.863 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-03-11T21:48:52.863 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:48:52.863 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:48:52.863 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "throttled", 2024-03-11T21:48:52.863 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:43:34.579570+0000", 2024-03-11T21:48:52.863 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.7370000000000001e-06 2024-03-11T21:48:52.864 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:48:52.864 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:48:52.864 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "all_read", 2024-03-11T21:48:52.864 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:43:34.579576+0000", 2024-03-11T21:48:52.864 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 6.032e-06 2024-03-11T21:48:52.864 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:48:52.864 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:48:52.864 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "dispatched", 2024-03-11T21:48:52.864 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:43:34.579593+0000", 2024-03-11T21:48:52.864 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.6038000000000001e-05 2024-03-11T21:48:52.864 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:48:52.864 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:48:52.864 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "queued_for_pg", 2024-03-11T21:48:52.864 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:43:34.579600+0000", 2024-03-11T21:48:52.864 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 7.8399999999999995e-06 2024-03-11T21:48:52.865 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:48:52.865 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:48:52.865 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "reached_pg", 2024-03-11T21:48:52.865 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:43:34.598974+0000", 2024-03-11T21:48:52.865 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.019373221999999999 2024-03-11T21:48:52.865 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:48:52.865 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:48:52.865 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "started", 2024-03-11T21:48:52.865 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:43:34.599078+0000", 2024-03-11T21:48:52.865 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.000104237 2024-03-11T21:48:52.865 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:48:52.865 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:48:52.865 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "waiting for subops from 6,7", 2024-03-11T21:48:52.865 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:43:34.599217+0000", 2024-03-11T21:48:52.865 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.00013922500000000001 2024-03-11T21:48:52.865 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:48:52.866 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:48:52.866 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "op_commit", 2024-03-11T21:48:52.866 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:43:34.602507+0000", 2024-03-11T21:48:52.866 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.0032898139999999998 2024-03-11T21:48:52.866 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:48:52.866 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:48:52.866 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "sub_op_commit_rec", 2024-03-11T21:48:52.866 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:43:34.697913+0000", 2024-03-11T21:48:52.866 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.095406216000000002 2024-03-11T21:48:52.866 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:48:52.866 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:48:52.866 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "sub_op_commit_rec", 2024-03-11T21:48:52.866 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:43:34.698238+0000", 2024-03-11T21:48:52.867 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.00032525300000000001 2024-03-11T21:48:52.867 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:48:52.867 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:48:52.867 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "commit_sent", 2024-03-11T21:48:52.867 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:43:34.698290+0000", 2024-03-11T21:48:52.867 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 5.1465000000000003e-05 2024-03-11T21:48:52.867 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:48:52.867 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:48:52.867 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "done", 2024-03-11T21:48:52.867 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:43:34.698336+0000", 2024-03-11T21:48:52.867 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 4.6261999999999999e-05 2024-03-11T21:48:52.867 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-03-11T21:48:52.867 INFO:teuthology.orchestra.run.smithi097.stdout: ] 2024-03-11T21:48:52.867 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-03-11T21:48:52.867 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-03-11T21:48:52.868 INFO:teuthology.orchestra.run.smithi097.stdout: ] 2024-03-11T21:48:52.868 INFO:teuthology.orchestra.run.smithi097.stdout:} 2024-03-11T21:48:53.085 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:48:52 smithi097 ceph-mon[33317]: pgmap v1102: 132 pgs: 132 active+clean; 582 KiB data, 5.3 GiB used, 531 GiB / 536 GiB avail; 853 B/s rd, 0 op/s 2024-03-11T21:48:53.085 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:48:52 smithi097 ceph-mon[33317]: 4.1c scrub starts 2024-03-11T21:48:53.085 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:48:52 smithi097 ceph-mon[33317]: 4.1c scrub ok 2024-03-11T21:48:53.085 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:48:52 smithi097 ceph-mon[33317]: osdmap e512: 8 total, 8 up, 6 in 2024-03-11T21:48:53.188 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:48:52 smithi092 ceph-mon[29572]: pgmap v1102: 132 pgs: 132 active+clean; 582 KiB data, 5.3 GiB used, 531 GiB / 536 GiB avail; 853 B/s rd, 0 op/s 2024-03-11T21:48:53.189 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:48:52 smithi092 ceph-mon[29572]: 4.1c scrub starts 2024-03-11T21:48:53.189 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:48:52 smithi092 ceph-mon[29572]: 4.1c scrub ok 2024-03-11T21:48:53.189 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:48:52 smithi092 ceph-mon[29572]: osdmap e512: 8 total, 8 up, 6 in 2024-03-11T21:48:53.189 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:48:52 smithi092 ceph-mon[37563]: pgmap v1102: 132 pgs: 132 active+clean; 582 KiB data, 5.3 GiB used, 531 GiB / 536 GiB avail; 853 B/s rd, 0 op/s 2024-03-11T21:48:53.189 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:48:52 smithi092 ceph-mon[37563]: 4.1c scrub starts 2024-03-11T21:48:53.189 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:48:52 smithi092 ceph-mon[37563]: 4.1c scrub ok 2024-03-11T21:48:53.189 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:48:52 smithi092 ceph-mon[37563]: osdmap e512: 8 total, 8 up, 6 in 2024-03-11T21:48:53.236 INFO:teuthology.orchestra.run.smithi092.stderr:Inferring config /var/lib/ceph/00db0016-dfed-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T21:48:53.585 INFO:journalctl@ceph.iscsi.iscsi.a.smithi097.stdout:Mar 11 21:48:53 smithi097 ceph-00db0016-dfed-11ee-95c8-87774f69a715-iscsi-iscsi-a[60943]: debug there is no tcmu-runner data available 2024-03-11T21:48:53.769 DEBUG:teuthology.orchestra.run.smithi097:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid 00db0016-dfed-11ee-95c8-87774f69a715 -- ceph daemon osd.5 dump_ops_in_flight 2024-03-11T21:48:54.021 INFO:teuthology.orchestra.run.smithi097.stderr:Inferring config /var/lib/ceph/00db0016-dfed-11ee-95c8-87774f69a715/mon.b/config 2024-03-11T21:48:54.085 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:48:53 smithi097 ceph-mon[33317]: from='client.24469 v1:172.21.15.97:0/404208310' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:48:54.085 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:48:53 smithi097 ceph-mon[33317]: from='mgr.14607 v1:172.21.15.92:0/1702717597' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T21:48:54.085 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:48:53 smithi097 ceph-mon[33317]: osdmap e513: 8 total, 8 up, 6 in 2024-03-11T21:48:54.324 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:48:53 smithi092 ceph-mon[37563]: from='client.24469 v1:172.21.15.97:0/404208310' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:48:54.324 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:48:53 smithi092 ceph-mon[37563]: from='mgr.14607 v1:172.21.15.92:0/1702717597' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T21:48:54.324 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:48:53 smithi092 ceph-mon[37563]: osdmap e513: 8 total, 8 up, 6 in 2024-03-11T21:48:54.324 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:48:53 smithi092 ceph-mon[29572]: from='client.24469 v1:172.21.15.97:0/404208310' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:48:54.324 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:48:53 smithi092 ceph-mon[29572]: from='mgr.14607 v1:172.21.15.92:0/1702717597' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T21:48:54.324 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:48:53 smithi092 ceph-mon[29572]: osdmap e513: 8 total, 8 up, 6 in 2024-03-11T21:48:54.960 INFO:teuthology.orchestra.run.smithi092.stderr:nodeep-scrub is unset 2024-03-11T21:48:55.119 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:48:54 smithi092 ceph-mon[29572]: pgmap v1104: 132 pgs: 132 active+clean; 582 KiB data, 5.3 GiB used, 531 GiB / 536 GiB avail; 853 B/s rd, 0 op/s 2024-03-11T21:48:55.120 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:48:54 smithi092 ceph-mon[29572]: 3.0 scrub starts 2024-03-11T21:48:55.120 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:48:54 smithi092 ceph-mon[29572]: 3.0 scrub ok 2024-03-11T21:48:55.120 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:48:54 smithi092 ceph-mon[29572]: 2.5 scrub starts 2024-03-11T21:48:55.120 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:48:54 smithi092 ceph-mon[29572]: 2.5 scrub ok 2024-03-11T21:48:55.120 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:48:54 smithi092 ceph-mon[29572]: purged_snaps scrub starts 2024-03-11T21:48:55.120 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:48:54 smithi092 ceph-mon[29572]: purged_snaps scrub ok 2024-03-11T21:48:55.120 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:48:54 smithi092 ceph-mon[29572]: from='client.? v1:172.21.15.92:0/1263420072' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-11T21:48:55.120 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:48:54 smithi092 ceph-mon[37563]: pgmap v1104: 132 pgs: 132 active+clean; 582 KiB data, 5.3 GiB used, 531 GiB / 536 GiB avail; 853 B/s rd, 0 op/s 2024-03-11T21:48:55.120 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:48:54 smithi092 ceph-mon[37563]: 3.0 scrub starts 2024-03-11T21:48:55.120 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:48:54 smithi092 ceph-mon[37563]: 3.0 scrub ok 2024-03-11T21:48:55.120 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:48:54 smithi092 ceph-mon[37563]: 2.5 scrub starts 2024-03-11T21:48:55.120 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:48:54 smithi092 ceph-mon[37563]: 2.5 scrub ok 2024-03-11T21:48:55.120 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:48:54 smithi092 ceph-mon[37563]: purged_snaps scrub starts 2024-03-11T21:48:55.121 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:48:54 smithi092 ceph-mon[37563]: purged_snaps scrub ok 2024-03-11T21:48:55.121 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:48:54 smithi092 ceph-mon[37563]: from='client.? v1:172.21.15.92:0/1263420072' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-11T21:48:55.258 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:48:54 smithi097 ceph-mon[33317]: pgmap v1104: 132 pgs: 132 active+clean; 582 KiB data, 5.3 GiB used, 531 GiB / 536 GiB avail; 853 B/s rd, 0 op/s 2024-03-11T21:48:55.258 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:48:54 smithi097 ceph-mon[33317]: 3.0 scrub starts 2024-03-11T21:48:55.259 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:48:54 smithi097 ceph-mon[33317]: 3.0 scrub ok 2024-03-11T21:48:55.259 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:48:54 smithi097 ceph-mon[33317]: 2.5 scrub starts 2024-03-11T21:48:55.259 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:48:54 smithi097 ceph-mon[33317]: 2.5 scrub ok 2024-03-11T21:48:55.259 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:48:54 smithi097 ceph-mon[33317]: purged_snaps scrub starts 2024-03-11T21:48:55.259 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:48:54 smithi097 ceph-mon[33317]: purged_snaps scrub ok 2024-03-11T21:48:55.259 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:48:54 smithi097 ceph-mon[33317]: from='client.? v1:172.21.15.92:0/1263420072' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-03-11T21:48:55.833 INFO:teuthology.orchestra.run.smithi097.stdout:{ 2024-03-11T21:48:55.833 INFO:teuthology.orchestra.run.smithi097.stdout: "ops": [], 2024-03-11T21:48:55.833 INFO:teuthology.orchestra.run.smithi097.stdout: "num_ops": 0 2024-03-11T21:48:55.833 INFO:teuthology.orchestra.run.smithi097.stdout:} 2024-03-11T21:48:55.975 INFO:teuthology.orchestra.run.smithi092.stderr:reweighted osd.2 to 1 (10000) 2024-03-11T21:48:56.258 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:48:55 smithi097 ceph-mon[33317]: from='client.? v1:172.21.15.92:0/1263420072' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-03-11T21:48:56.258 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:48:55 smithi097 ceph-mon[33317]: osdmap e514: 8 total, 8 up, 6 in 2024-03-11T21:48:56.258 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:48:55 smithi097 ceph-mon[33317]: 5.8 scrub starts 2024-03-11T21:48:56.258 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:48:55 smithi097 ceph-mon[33317]: 5.8 scrub ok 2024-03-11T21:48:56.259 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:48:55 smithi097 ceph-mon[33317]: from='client.? v1:172.21.15.92:0/3420676520' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]: dispatch 2024-03-11T21:48:56.259 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:48:55 smithi097 ceph-mon[33317]: from='client.36068 ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]: dispatch 2024-03-11T21:48:56.356 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:48:55 smithi092 ceph-mon[29572]: from='client.? v1:172.21.15.92:0/1263420072' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-03-11T21:48:56.356 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:48:55 smithi092 ceph-mon[29572]: osdmap e514: 8 total, 8 up, 6 in 2024-03-11T21:48:56.356 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:48:55 smithi092 ceph-mon[29572]: 5.8 scrub starts 2024-03-11T21:48:56.357 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:48:55 smithi092 ceph-mon[29572]: 5.8 scrub ok 2024-03-11T21:48:56.357 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:48:55 smithi092 ceph-mon[29572]: from='client.? v1:172.21.15.92:0/3420676520' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]: dispatch 2024-03-11T21:48:56.357 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:48:55 smithi092 ceph-mon[29572]: from='client.36068 ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]: dispatch 2024-03-11T21:48:56.357 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:48:55 smithi092 ceph-mon[37563]: from='client.? v1:172.21.15.92:0/1263420072' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-03-11T21:48:56.357 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:48:55 smithi092 ceph-mon[37563]: osdmap e514: 8 total, 8 up, 6 in 2024-03-11T21:48:56.357 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:48:55 smithi092 ceph-mon[37563]: 5.8 scrub starts 2024-03-11T21:48:56.357 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:48:55 smithi092 ceph-mon[37563]: 5.8 scrub ok 2024-03-11T21:48:56.357 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:48:55 smithi092 ceph-mon[37563]: from='client.? v1:172.21.15.92:0/3420676520' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]: dispatch 2024-03-11T21:48:56.357 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:48:55 smithi092 ceph-mon[37563]: from='client.36068 ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]: dispatch 2024-03-11T21:48:56.523 DEBUG:teuthology.orchestra.run.smithi097:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid 00db0016-dfed-11ee-95c8-87774f69a715 -- ceph daemon osd.5 dump_blocked_ops 2024-03-11T21:48:56.732 DEBUG:teuthology.orchestra.run.smithi092:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid 00db0016-dfed-11ee-95c8-87774f69a715 -- ceph osd reweight 6 1 2024-03-11T21:48:56.761 INFO:teuthology.orchestra.run.smithi097.stderr:Inferring config /var/lib/ceph/00db0016-dfed-11ee-95c8-87774f69a715/mon.b/config 2024-03-11T21:48:56.975 INFO:teuthology.orchestra.run.smithi092.stderr:Inferring config /var/lib/ceph/00db0016-dfed-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T21:48:57.135 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:48:56 smithi097 ceph-mon[33317]: pgmap v1107: 132 pgs: 132 active+clean; 582 KiB data, 5.3 GiB used, 531 GiB / 536 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-03-11T21:48:57.135 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:48:56 smithi097 ceph-mon[33317]: 6.5 scrub starts 2024-03-11T21:48:57.135 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:48:56 smithi097 ceph-mon[33317]: 6.5 scrub ok 2024-03-11T21:48:57.135 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:48:56 smithi097 ceph-mon[33317]: from='client.36068 ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]': finished 2024-03-11T21:48:57.135 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:48:56 smithi097 ceph-mon[33317]: osdmap e515: 8 total, 8 up, 6 in 2024-03-11T21:48:57.135 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:48:56 smithi097 ceph-mon[33317]: 5.3 scrub starts 2024-03-11T21:48:57.135 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:48:56 smithi097 ceph-mon[33317]: 5.3 scrub ok 2024-03-11T21:48:57.135 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:48:56 smithi097 ceph-mon[33317]: 5.6 scrub starts 2024-03-11T21:48:57.135 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:48:56 smithi097 ceph-mon[33317]: 5.6 scrub ok 2024-03-11T21:48:57.260 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:48:56 smithi092 ceph-mon[29572]: pgmap v1107: 132 pgs: 132 active+clean; 582 KiB data, 5.3 GiB used, 531 GiB / 536 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-03-11T21:48:57.260 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:48:56 smithi092 ceph-mon[29572]: 6.5 scrub starts 2024-03-11T21:48:57.260 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:48:56 smithi092 ceph-mon[29572]: 6.5 scrub ok 2024-03-11T21:48:57.260 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:48:56 smithi092 ceph-mon[29572]: from='client.36068 ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]': finished 2024-03-11T21:48:57.260 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:48:56 smithi092 ceph-mon[29572]: osdmap e515: 8 total, 8 up, 6 in 2024-03-11T21:48:57.260 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:48:56 smithi092 ceph-mon[29572]: 5.3 scrub starts 2024-03-11T21:48:57.260 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:48:56 smithi092 ceph-mon[29572]: 5.3 scrub ok 2024-03-11T21:48:57.260 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:48:56 smithi092 ceph-mon[29572]: 5.6 scrub starts 2024-03-11T21:48:57.260 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:48:56 smithi092 ceph-mon[29572]: 5.6 scrub ok 2024-03-11T21:48:57.261 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:48:56 smithi092 ceph-mon[37563]: pgmap v1107: 132 pgs: 132 active+clean; 582 KiB data, 5.3 GiB used, 531 GiB / 536 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-03-11T21:48:57.261 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:48:56 smithi092 ceph-mon[37563]: 6.5 scrub starts 2024-03-11T21:48:57.261 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:48:56 smithi092 ceph-mon[37563]: 6.5 scrub ok 2024-03-11T21:48:57.261 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:48:56 smithi092 ceph-mon[37563]: from='client.36068 ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]': finished 2024-03-11T21:48:57.261 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:48:56 smithi092 ceph-mon[37563]: osdmap e515: 8 total, 8 up, 6 in 2024-03-11T21:48:57.261 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:48:56 smithi092 ceph-mon[37563]: 5.3 scrub starts 2024-03-11T21:48:57.261 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:48:56 smithi092 ceph-mon[37563]: 5.3 scrub ok 2024-03-11T21:48:57.261 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:48:56 smithi092 ceph-mon[37563]: 5.6 scrub starts 2024-03-11T21:48:57.261 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:48:56 smithi092 ceph-mon[37563]: 5.6 scrub ok 2024-03-11T21:48:58.169 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:48:57 smithi097 ceph-mon[33317]: 5.1d scrub starts 2024-03-11T21:48:58.169 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:48:57 smithi097 ceph-mon[33317]: 5.1d scrub ok 2024-03-11T21:48:58.169 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:48:57 smithi097 ceph-mon[33317]: 5.f scrub starts 2024-03-11T21:48:58.169 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:48:57 smithi097 ceph-mon[33317]: 5.f scrub ok 2024-03-11T21:48:58.169 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:48:57 smithi097 ceph-mon[33317]: 2.1a scrub starts 2024-03-11T21:48:58.169 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:48:57 smithi097 ceph-mon[33317]: 2.1a scrub ok 2024-03-11T21:48:58.331 INFO:journalctl@ceph.mgr.y.smithi092.stdout:Mar 11 21:48:57 smithi092 ceph-00db0016-dfed-11ee-95c8-87774f69a715-mgr-y[29801]: ::ffff:172.21.15.97 - - [11/Mar/2024:21:48:57] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-11T21:48:58.331 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:48:57 smithi092 ceph-mon[29572]: 5.1d scrub starts 2024-03-11T21:48:58.331 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:48:57 smithi092 ceph-mon[29572]: 5.1d scrub ok 2024-03-11T21:48:58.331 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:48:57 smithi092 ceph-mon[29572]: 5.f scrub starts 2024-03-11T21:48:58.331 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:48:57 smithi092 ceph-mon[29572]: 5.f scrub ok 2024-03-11T21:48:58.331 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:48:57 smithi092 ceph-mon[29572]: 2.1a scrub starts 2024-03-11T21:48:58.331 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:48:57 smithi092 ceph-mon[29572]: 2.1a scrub ok 2024-03-11T21:48:58.332 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:48:57 smithi092 ceph-mon[37563]: 5.1d scrub starts 2024-03-11T21:48:58.332 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:48:57 smithi092 ceph-mon[37563]: 5.1d scrub ok 2024-03-11T21:48:58.332 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:48:57 smithi092 ceph-mon[37563]: 5.f scrub starts 2024-03-11T21:48:58.332 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:48:57 smithi092 ceph-mon[37563]: 5.f scrub ok 2024-03-11T21:48:58.332 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:48:57 smithi092 ceph-mon[37563]: 2.1a scrub starts 2024-03-11T21:48:58.332 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:48:57 smithi092 ceph-mon[37563]: 2.1a scrub ok 2024-03-11T21:48:58.410 INFO:teuthology.orchestra.run.smithi097.stdout:{ 2024-03-11T21:48:58.411 INFO:teuthology.orchestra.run.smithi097.stdout: "ops": [], 2024-03-11T21:48:58.411 INFO:teuthology.orchestra.run.smithi097.stdout: "complaint_time": 30, 2024-03-11T21:48:58.411 INFO:teuthology.orchestra.run.smithi097.stdout: "num_blocked_ops": 0 2024-03-11T21:48:58.411 INFO:teuthology.orchestra.run.smithi097.stdout:} 2024-03-11T21:48:59.175 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:48:58 smithi097 ceph-mon[33317]: pgmap v1109: 132 pgs: 132 active+clean; 582 KiB data, 5.3 GiB used, 531 GiB / 536 GiB avail; 919 B/s rd, 0 op/s 2024-03-11T21:48:59.175 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:48:58 smithi097 ceph-mon[33317]: purged_snaps scrub starts 2024-03-11T21:48:59.176 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:48:58 smithi097 ceph-mon[33317]: purged_snaps scrub ok 2024-03-11T21:48:59.176 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:48:58 smithi097 ceph-mon[33317]: 2.1c scrub starts 2024-03-11T21:48:59.176 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:48:58 smithi097 ceph-mon[33317]: 2.1c scrub ok 2024-03-11T21:48:59.176 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:48:58 smithi097 ceph-mon[33317]: osdmap e516: 8 total, 8 up, 6 in 2024-03-11T21:48:59.176 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:48:58 smithi097 ceph-mon[33317]: 5.16 scrub starts 2024-03-11T21:48:59.176 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:48:58 smithi097 ceph-mon[33317]: 5.16 scrub ok 2024-03-11T21:48:59.176 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:48:58 smithi097 ceph-mon[33317]: 2.4 deep-scrub starts 2024-03-11T21:48:59.176 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:48:58 smithi097 ceph-mon[33317]: 2.4 deep-scrub ok 2024-03-11T21:48:59.176 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:48:58 smithi097 ceph-mon[33317]: osdmap e517: 8 total, 8 up, 6 in 2024-03-11T21:48:59.176 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:48:58 smithi097 ceph-mon[33317]: 6.14 scrub starts 2024-03-11T21:48:59.176 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:48:58 smithi097 ceph-mon[33317]: 6.14 scrub ok 2024-03-11T21:48:59.176 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:48:58 smithi097 ceph-mon[33317]: from='client.? v1:172.21.15.92:0/1994941636' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]: dispatch 2024-03-11T21:48:59.204 DEBUG:teuthology.orchestra.run.smithi097:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid 00db0016-dfed-11ee-95c8-87774f69a715 -- ceph daemon osd.5 dump_historic_ops 2024-03-11T21:48:59.369 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:48:58 smithi092 ceph-mon[29572]: pgmap v1109: 132 pgs: 132 active+clean; 582 KiB data, 5.3 GiB used, 531 GiB / 536 GiB avail; 919 B/s rd, 0 op/s 2024-03-11T21:48:59.370 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:48:58 smithi092 ceph-mon[29572]: purged_snaps scrub starts 2024-03-11T21:48:59.370 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:48:58 smithi092 ceph-mon[29572]: purged_snaps scrub ok 2024-03-11T21:48:59.370 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:48:58 smithi092 ceph-mon[29572]: 2.1c scrub starts 2024-03-11T21:48:59.370 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:48:58 smithi092 ceph-mon[29572]: 2.1c scrub ok 2024-03-11T21:48:59.370 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:48:58 smithi092 ceph-mon[29572]: osdmap e516: 8 total, 8 up, 6 in 2024-03-11T21:48:59.370 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:48:58 smithi092 ceph-mon[29572]: 5.16 scrub starts 2024-03-11T21:48:59.370 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:48:58 smithi092 ceph-mon[29572]: 5.16 scrub ok 2024-03-11T21:48:59.370 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:48:58 smithi092 ceph-mon[29572]: 2.4 deep-scrub starts 2024-03-11T21:48:59.370 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:48:58 smithi092 ceph-mon[29572]: 2.4 deep-scrub ok 2024-03-11T21:48:59.370 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:48:58 smithi092 ceph-mon[29572]: osdmap e517: 8 total, 8 up, 6 in 2024-03-11T21:48:59.370 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:48:58 smithi092 ceph-mon[29572]: 6.14 scrub starts 2024-03-11T21:48:59.370 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:48:58 smithi092 ceph-mon[29572]: 6.14 scrub ok 2024-03-11T21:48:59.371 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:48:58 smithi092 ceph-mon[29572]: from='client.? v1:172.21.15.92:0/1994941636' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]: dispatch 2024-03-11T21:48:59.371 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:48:58 smithi092 ceph-mon[37563]: pgmap v1109: 132 pgs: 132 active+clean; 582 KiB data, 5.3 GiB used, 531 GiB / 536 GiB avail; 919 B/s rd, 0 op/s 2024-03-11T21:48:59.371 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:48:58 smithi092 ceph-mon[37563]: purged_snaps scrub starts 2024-03-11T21:48:59.371 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:48:58 smithi092 ceph-mon[37563]: purged_snaps scrub ok 2024-03-11T21:48:59.371 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:48:58 smithi092 ceph-mon[37563]: 2.1c scrub starts 2024-03-11T21:48:59.371 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:48:58 smithi092 ceph-mon[37563]: 2.1c scrub ok 2024-03-11T21:48:59.371 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:48:58 smithi092 ceph-mon[37563]: osdmap e516: 8 total, 8 up, 6 in 2024-03-11T21:48:59.371 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:48:58 smithi092 ceph-mon[37563]: 5.16 scrub starts 2024-03-11T21:48:59.371 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:48:58 smithi092 ceph-mon[37563]: 5.16 scrub ok 2024-03-11T21:48:59.371 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:48:58 smithi092 ceph-mon[37563]: 2.4 deep-scrub starts 2024-03-11T21:48:59.371 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:48:58 smithi092 ceph-mon[37563]: 2.4 deep-scrub ok 2024-03-11T21:48:59.371 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:48:58 smithi092 ceph-mon[37563]: osdmap e517: 8 total, 8 up, 6 in 2024-03-11T21:48:59.372 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:48:58 smithi092 ceph-mon[37563]: 6.14 scrub starts 2024-03-11T21:48:59.372 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:48:58 smithi092 ceph-mon[37563]: 6.14 scrub ok 2024-03-11T21:48:59.372 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:48:58 smithi092 ceph-mon[37563]: from='client.? v1:172.21.15.92:0/1994941636' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]: dispatch 2024-03-11T21:48:59.444 INFO:teuthology.orchestra.run.smithi097.stderr:Inferring config /var/lib/ceph/00db0016-dfed-11ee-95c8-87774f69a715/mon.b/config 2024-03-11T21:48:59.599 INFO:teuthology.orchestra.run.smithi092.stderr:reweighted osd.6 to 1 (10000) 2024-03-11T21:49:00.362 DEBUG:teuthology.orchestra.run.smithi092:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid 00db0016-dfed-11ee-95c8-87774f69a715 -- ceph osd reweight 7 1 2024-03-11T21:49:00.613 INFO:teuthology.orchestra.run.smithi092.stderr:Inferring config /var/lib/ceph/00db0016-dfed-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T21:49:00.870 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:49:00 smithi092 ceph-mon[29572]: 2.d scrub starts 2024-03-11T21:49:00.870 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:49:00 smithi092 ceph-mon[29572]: 2.d scrub ok 2024-03-11T21:49:00.870 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:49:00 smithi092 ceph-mon[29572]: 4.c scrub starts 2024-03-11T21:49:00.870 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:49:00 smithi092 ceph-mon[29572]: 4.c scrub ok 2024-03-11T21:49:00.870 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:49:00 smithi092 ceph-mon[29572]: pgmap v1112: 132 pgs: 132 active+clean; 582 KiB data, 5.3 GiB used, 531 GiB / 536 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-03-11T21:49:00.870 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:49:00 smithi092 ceph-mon[29572]: from='client.? v1:172.21.15.92:0/1994941636' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]': finished 2024-03-11T21:49:00.870 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:49:00 smithi092 ceph-mon[29572]: osdmap e518: 8 total, 8 up, 6 in 2024-03-11T21:49:00.870 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:49:00 smithi092 ceph-mon[29572]: purged_snaps scrub starts 2024-03-11T21:49:00.870 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:49:00 smithi092 ceph-mon[29572]: purged_snaps scrub ok 2024-03-11T21:49:00.871 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:49:00 smithi092 ceph-mon[37563]: 2.d scrub starts 2024-03-11T21:49:00.871 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:49:00 smithi092 ceph-mon[37563]: 2.d scrub ok 2024-03-11T21:49:00.871 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:49:00 smithi092 ceph-mon[37563]: 4.c scrub starts 2024-03-11T21:49:00.871 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:49:00 smithi092 ceph-mon[37563]: 4.c scrub ok 2024-03-11T21:49:00.871 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:49:00 smithi092 ceph-mon[37563]: pgmap v1112: 132 pgs: 132 active+clean; 582 KiB data, 5.3 GiB used, 531 GiB / 536 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-03-11T21:49:00.871 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:49:00 smithi092 ceph-mon[37563]: from='client.? v1:172.21.15.92:0/1994941636' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]': finished 2024-03-11T21:49:00.871 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:49:00 smithi092 ceph-mon[37563]: osdmap e518: 8 total, 8 up, 6 in 2024-03-11T21:49:00.871 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:49:00 smithi092 ceph-mon[37563]: purged_snaps scrub starts 2024-03-11T21:49:00.871 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:49:00 smithi092 ceph-mon[37563]: purged_snaps scrub ok 2024-03-11T21:49:00.943 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:49:00 smithi097 ceph-mon[33317]: 2.d scrub starts 2024-03-11T21:49:00.943 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:49:00 smithi097 ceph-mon[33317]: 2.d scrub ok 2024-03-11T21:49:00.943 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:49:00 smithi097 ceph-mon[33317]: 4.c scrub starts 2024-03-11T21:49:00.943 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:49:00 smithi097 ceph-mon[33317]: 4.c scrub ok 2024-03-11T21:49:00.943 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:49:00 smithi097 ceph-mon[33317]: pgmap v1112: 132 pgs: 132 active+clean; 582 KiB data, 5.3 GiB used, 531 GiB / 536 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-03-11T21:49:00.943 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:49:00 smithi097 ceph-mon[33317]: from='client.? v1:172.21.15.92:0/1994941636' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]': finished 2024-03-11T21:49:00.943 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:49:00 smithi097 ceph-mon[33317]: osdmap e518: 8 total, 8 up, 6 in 2024-03-11T21:49:00.944 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:49:00 smithi097 ceph-mon[33317]: purged_snaps scrub starts 2024-03-11T21:49:00.944 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:49:00 smithi097 ceph-mon[33317]: purged_snaps scrub ok 2024-03-11T21:49:01.188 INFO:teuthology.orchestra.run.smithi097.stdout:{ 2024-03-11T21:49:01.188 INFO:teuthology.orchestra.run.smithi097.stdout: "size": 20, 2024-03-11T21:49:01.188 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 600, 2024-03-11T21:49:01.188 INFO:teuthology.orchestra.run.smithi097.stdout: "ops": [ 2024-03-11T21:49:01.189 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:49:01.189 INFO:teuthology.orchestra.run.smithi097.stdout: "description": "osd_op(client.15744.0:69277 10.9 10:92e95475:::benchmark_data_smithi092_161462_object69276:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e372)", 2024-03-11T21:49:01.189 INFO:teuthology.orchestra.run.smithi097.stdout: "initiated_at": "2024-03-11T21:41:27.829175+0000", 2024-03-11T21:49:01.189 INFO:teuthology.orchestra.run.smithi097.stdout: "age": 453.36623150700001, 2024-03-11T21:49:01.189 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.99767732799999997, 2024-03-11T21:49:01.189 INFO:teuthology.orchestra.run.smithi097.stdout: "type_data": { 2024-03-11T21:49:01.189 INFO:teuthology.orchestra.run.smithi097.stdout: "flag_point": "reached pg", 2024-03-11T21:49:01.189 INFO:teuthology.orchestra.run.smithi097.stdout: "client_info": { 2024-03-11T21:49:01.189 INFO:teuthology.orchestra.run.smithi097.stdout: "client": "client.15744", 2024-03-11T21:49:01.189 INFO:teuthology.orchestra.run.smithi097.stdout: "client_addr": "v1:172.21.15.92:0/3334000812", 2024-03-11T21:49:01.189 INFO:teuthology.orchestra.run.smithi097.stdout: "tid": 69277 2024-03-11T21:49:01.189 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:49:01.189 INFO:teuthology.orchestra.run.smithi097.stdout: "events": [ 2024-03-11T21:49:01.189 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:49:01.189 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "initiated", 2024-03-11T21:49:01.190 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:41:27.829175+0000", 2024-03-11T21:49:01.190 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-03-11T21:49:01.190 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:49:01.190 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:49:01.190 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "header_read", 2024-03-11T21:49:01.190 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:41:27.829175+0000", 2024-03-11T21:49:01.190 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-03-11T21:49:01.190 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:49:01.190 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:49:01.190 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "throttled", 2024-03-11T21:49:01.190 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:41:27.829179+0000", 2024-03-11T21:49:01.191 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 4.3080000000000001e-06 2024-03-11T21:49:01.191 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:49:01.191 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:49:01.191 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "all_read", 2024-03-11T21:49:01.191 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:41:27.829232+0000", 2024-03-11T21:49:01.191 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 5.3551999999999998e-05 2024-03-11T21:49:01.191 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:49:01.191 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:49:01.191 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "dispatched", 2024-03-11T21:49:01.191 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:41:27.829294+0000", 2024-03-11T21:49:01.191 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 6.2012000000000002e-05 2024-03-11T21:49:01.191 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:49:01.191 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:49:01.192 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "queued_for_pg", 2024-03-11T21:49:01.192 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:41:27.829309+0000", 2024-03-11T21:49:01.192 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.4506e-05 2024-03-11T21:49:01.192 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:49:01.192 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:49:01.192 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "reached_pg", 2024-03-11T21:49:01.192 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:41:27.977027+0000", 2024-03-11T21:49:01.192 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.147718087 2024-03-11T21:49:01.192 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:49:01.192 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:49:01.192 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "waiting for peered", 2024-03-11T21:49:01.192 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:41:27.977039+0000", 2024-03-11T21:49:01.192 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.2295e-05 2024-03-11T21:49:01.192 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:49:01.193 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:49:01.193 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "reached_pg", 2024-03-11T21:49:01.193 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:41:28.826770+0000", 2024-03-11T21:49:01.193 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.84973065199999998 2024-03-11T21:49:01.193 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:49:01.193 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:49:01.193 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "done", 2024-03-11T21:49:01.193 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:41:28.826852+0000", 2024-03-11T21:49:01.193 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 8.1916000000000006e-05 2024-03-11T21:49:01.193 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-03-11T21:49:01.193 INFO:teuthology.orchestra.run.smithi097.stdout: ] 2024-03-11T21:49:01.193 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-03-11T21:49:01.193 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:49:01.193 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:49:01.194 INFO:teuthology.orchestra.run.smithi097.stdout: "description": "osd_op(client.15744.0:69287 10.9 10:93b721b6:::benchmark_data_smithi092_161462_object69286:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e372)", 2024-03-11T21:49:01.194 INFO:teuthology.orchestra.run.smithi097.stdout: "initiated_at": "2024-03-11T21:41:27.829334+0000", 2024-03-11T21:49:01.194 INFO:teuthology.orchestra.run.smithi097.stdout: "age": 453.36607250499998, 2024-03-11T21:49:01.194 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.99759708400000002, 2024-03-11T21:49:01.194 INFO:teuthology.orchestra.run.smithi097.stdout: "type_data": { 2024-03-11T21:49:01.194 INFO:teuthology.orchestra.run.smithi097.stdout: "flag_point": "reached pg", 2024-03-11T21:49:01.194 INFO:teuthology.orchestra.run.smithi097.stdout: "client_info": { 2024-03-11T21:49:01.194 INFO:teuthology.orchestra.run.smithi097.stdout: "client": "client.15744", 2024-03-11T21:49:01.194 INFO:teuthology.orchestra.run.smithi097.stdout: "client_addr": "v1:172.21.15.92:0/3334000812", 2024-03-11T21:49:01.194 INFO:teuthology.orchestra.run.smithi097.stdout: "tid": 69287 2024-03-11T21:49:01.194 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:49:01.194 INFO:teuthology.orchestra.run.smithi097.stdout: "events": [ 2024-03-11T21:49:01.195 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:49:01.195 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "initiated", 2024-03-11T21:49:01.195 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:41:27.829334+0000", 2024-03-11T21:49:01.195 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-03-11T21:49:01.195 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:49:01.195 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:49:01.195 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "header_read", 2024-03-11T21:49:01.195 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:41:27.829334+0000", 2024-03-11T21:49:01.195 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-03-11T21:49:01.195 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:49:01.195 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:49:01.195 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "throttled", 2024-03-11T21:49:01.195 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:41:27.829334+0000", 2024-03-11T21:49:01.196 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 9.4200000000000004e-07 2024-03-11T21:49:01.196 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:49:01.196 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:49:01.196 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "all_read", 2024-03-11T21:49:01.196 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:41:27.829385+0000", 2024-03-11T21:49:01.196 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 5.0713e-05 2024-03-11T21:49:01.196 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:49:01.196 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:49:01.196 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "dispatched", 2024-03-11T21:49:01.196 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:41:27.829393+0000", 2024-03-11T21:49:01.196 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 7.7279999999999995e-06 2024-03-11T21:49:01.196 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:49:01.196 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:49:01.196 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "queued_for_pg", 2024-03-11T21:49:01.197 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:41:27.829400+0000", 2024-03-11T21:49:01.197 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 6.6980000000000001e-06 2024-03-11T21:49:01.197 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:49:01.197 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:49:01.197 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "reached_pg", 2024-03-11T21:49:01.197 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:41:27.977066+0000", 2024-03-11T21:49:01.197 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.147666511 2024-03-11T21:49:01.197 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:49:01.197 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:49:01.197 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "waiting for peered", 2024-03-11T21:49:01.197 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:41:27.977073+0000", 2024-03-11T21:49:01.197 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 7.3100000000000003e-06 2024-03-11T21:49:01.197 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:49:01.197 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:49:01.198 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "reached_pg", 2024-03-11T21:49:01.198 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:41:28.826877+0000", 2024-03-11T21:49:01.198 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.84980390699999997 2024-03-11T21:49:01.198 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:49:01.198 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:49:01.198 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "done", 2024-03-11T21:49:01.198 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:41:28.826931+0000", 2024-03-11T21:49:01.198 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 5.3275000000000001e-05 2024-03-11T21:49:01.198 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-03-11T21:49:01.198 INFO:teuthology.orchestra.run.smithi097.stdout: ] 2024-03-11T21:49:01.198 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-03-11T21:49:01.198 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:49:01.198 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:49:01.198 INFO:teuthology.orchestra.run.smithi097.stdout: "description": "osd_op(client.15744.0:69288 10.9 10:92935038:::benchmark_data_smithi092_161462_object69287:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e372)", 2024-03-11T21:49:01.199 INFO:teuthology.orchestra.run.smithi097.stdout: "initiated_at": "2024-03-11T21:41:27.829408+0000", 2024-03-11T21:49:01.199 INFO:teuthology.orchestra.run.smithi097.stdout: "age": 453.36599788900003, 2024-03-11T21:49:01.199 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.015902442, 2024-03-11T21:49:01.199 INFO:teuthology.orchestra.run.smithi097.stdout: "type_data": { 2024-03-11T21:49:01.199 INFO:teuthology.orchestra.run.smithi097.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-11T21:49:01.199 INFO:teuthology.orchestra.run.smithi097.stdout: "client_info": { 2024-03-11T21:49:01.199 INFO:teuthology.orchestra.run.smithi097.stdout: "client": "client.15744", 2024-03-11T21:49:01.199 INFO:teuthology.orchestra.run.smithi097.stdout: "client_addr": "v1:172.21.15.92:0/3334000812", 2024-03-11T21:49:01.199 INFO:teuthology.orchestra.run.smithi097.stdout: "tid": 69288 2024-03-11T21:49:01.199 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:49:01.199 INFO:teuthology.orchestra.run.smithi097.stdout: "events": [ 2024-03-11T21:49:01.199 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:49:01.199 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "initiated", 2024-03-11T21:49:01.199 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:41:27.829408+0000", 2024-03-11T21:49:01.199 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-03-11T21:49:01.200 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:49:01.200 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:49:01.200 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "header_read", 2024-03-11T21:49:01.200 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:41:27.829408+0000", 2024-03-11T21:49:01.200 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-03-11T21:49:01.200 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:49:01.200 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:49:01.200 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "throttled", 2024-03-11T21:49:01.200 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:41:27.829409+0000", 2024-03-11T21:49:01.200 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 7.8800000000000002e-07 2024-03-11T21:49:01.200 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:49:01.200 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:49:01.200 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "all_read", 2024-03-11T21:49:01.200 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:41:27.829459+0000", 2024-03-11T21:49:01.200 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 4.9675999999999997e-05 2024-03-11T21:49:01.201 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:49:01.201 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:49:01.201 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "dispatched", 2024-03-11T21:49:01.201 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:41:27.829465+0000", 2024-03-11T21:49:01.201 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 6.8750000000000002e-06 2024-03-11T21:49:01.201 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:49:01.201 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:49:01.201 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "queued_for_pg", 2024-03-11T21:49:01.201 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:41:27.829472+0000", 2024-03-11T21:49:01.201 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 6.2489999999999998e-06 2024-03-11T21:49:01.201 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:49:01.201 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:49:01.201 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "reached_pg", 2024-03-11T21:49:01.202 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:41:27.977097+0000", 2024-03-11T21:49:01.202 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.147624901 2024-03-11T21:49:01.202 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:49:01.202 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:49:01.202 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "waiting for peered", 2024-03-11T21:49:01.202 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:41:27.977104+0000", 2024-03-11T21:49:01.203 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 7.165e-06 2024-03-11T21:49:01.203 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:49:01.203 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:49:01.204 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "reached_pg", 2024-03-11T21:49:01.204 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:41:28.826951+0000", 2024-03-11T21:49:01.204 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.849847354 2024-03-11T21:49:01.204 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:49:01.204 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:49:01.204 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "started", 2024-03-11T21:49:01.204 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:41:28.827075+0000", 2024-03-11T21:49:01.204 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.00012410899999999999 2024-03-11T21:49:01.204 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:49:01.204 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:49:01.204 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "waiting for subops from 2,7", 2024-03-11T21:49:01.204 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:41:28.827231+0000", 2024-03-11T21:49:01.204 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.00015579 2024-03-11T21:49:01.204 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:49:01.205 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:49:01.205 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "op_commit", 2024-03-11T21:49:01.205 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:41:28.830775+0000", 2024-03-11T21:49:01.205 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.0035436370000000001 2024-03-11T21:49:01.205 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:49:01.205 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:49:01.205 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "sub_op_commit_rec", 2024-03-11T21:49:01.205 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:41:28.837674+0000", 2024-03-11T21:49:01.205 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.0068991429999999999 2024-03-11T21:49:01.205 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:49:01.205 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:49:01.205 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "sub_op_commit_rec", 2024-03-11T21:49:01.205 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:41:28.845212+0000", 2024-03-11T21:49:01.205 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.0075384919999999999 2024-03-11T21:49:01.205 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:49:01.206 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:49:01.206 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "commit_sent", 2024-03-11T21:49:01.206 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:41:28.845269+0000", 2024-03-11T21:49:01.206 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 5.7074000000000003e-05 2024-03-11T21:49:01.206 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:49:01.206 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:49:01.206 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "done", 2024-03-11T21:49:01.206 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:41:28.845311+0000", 2024-03-11T21:49:01.206 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 4.1189000000000002e-05 2024-03-11T21:49:01.206 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-03-11T21:49:01.207 INFO:teuthology.orchestra.run.smithi097.stdout: ] 2024-03-11T21:49:01.207 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-03-11T21:49:01.207 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:49:01.207 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:49:01.207 INFO:teuthology.orchestra.run.smithi097.stdout: "description": "osd_op(client.15744.0:69290 10.6 10:64e7935d:::benchmark_data_smithi092_161462_object69289:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e372)", 2024-03-11T21:49:01.207 INFO:teuthology.orchestra.run.smithi097.stdout: "initiated_at": "2024-03-11T21:41:27.829479+0000", 2024-03-11T21:49:01.207 INFO:teuthology.orchestra.run.smithi097.stdout: "age": 453.36592659600001, 2024-03-11T21:49:01.207 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.0144195920000001, 2024-03-11T21:49:01.207 INFO:teuthology.orchestra.run.smithi097.stdout: "type_data": { 2024-03-11T21:49:01.207 INFO:teuthology.orchestra.run.smithi097.stdout: "flag_point": "reached pg", 2024-03-11T21:49:01.207 INFO:teuthology.orchestra.run.smithi097.stdout: "client_info": { 2024-03-11T21:49:01.207 INFO:teuthology.orchestra.run.smithi097.stdout: "client": "client.15744", 2024-03-11T21:49:01.207 INFO:teuthology.orchestra.run.smithi097.stdout: "client_addr": "v1:172.21.15.92:0/3334000812", 2024-03-11T21:49:01.208 INFO:teuthology.orchestra.run.smithi097.stdout: "tid": 69290 2024-03-11T21:49:01.208 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:49:01.208 INFO:teuthology.orchestra.run.smithi097.stdout: "events": [ 2024-03-11T21:49:01.208 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:49:01.208 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "initiated", 2024-03-11T21:49:01.208 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:41:27.829479+0000", 2024-03-11T21:49:01.208 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-03-11T21:49:01.208 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:49:01.208 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:49:01.208 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "header_read", 2024-03-11T21:49:01.208 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:41:27.829479+0000", 2024-03-11T21:49:01.208 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-03-11T21:49:01.208 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:49:01.208 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:49:01.209 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "throttled", 2024-03-11T21:49:01.209 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:41:27.829480+0000", 2024-03-11T21:49:01.209 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 6.5499999999999998e-07 2024-03-11T21:49:01.209 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:49:01.209 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:49:01.209 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "all_read", 2024-03-11T21:49:01.209 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:41:27.829510+0000", 2024-03-11T21:49:01.209 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 2.9456999999999999e-05 2024-03-11T21:49:01.209 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:49:01.209 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:49:01.209 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "dispatched", 2024-03-11T21:49:01.209 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:41:27.829516+0000", 2024-03-11T21:49:01.209 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 6.4459999999999998e-06 2024-03-11T21:49:01.209 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:49:01.210 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:49:01.210 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "queued_for_pg", 2024-03-11T21:49:01.210 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:41:27.829522+0000", 2024-03-11T21:49:01.210 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 5.6529999999999999e-06 2024-03-11T21:49:01.210 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:49:01.210 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:49:01.210 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "reached_pg", 2024-03-11T21:49:01.210 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:41:28.000484+0000", 2024-03-11T21:49:01.210 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.170962688 2024-03-11T21:49:01.210 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:49:01.210 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:49:01.210 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "waiting for peered", 2024-03-11T21:49:01.210 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:41:28.000496+0000", 2024-03-11T21:49:01.211 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.2018000000000001e-05 2024-03-11T21:49:01.211 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:49:01.211 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:49:01.211 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "reached_pg", 2024-03-11T21:49:01.211 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:41:28.843770+0000", 2024-03-11T21:49:01.211 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.84327316600000002 2024-03-11T21:49:01.211 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:49:01.211 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:49:01.211 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "done", 2024-03-11T21:49:01.211 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:41:28.843899+0000", 2024-03-11T21:49:01.211 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.00012950900000000001 2024-03-11T21:49:01.211 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-03-11T21:49:01.211 INFO:teuthology.orchestra.run.smithi097.stdout: ] 2024-03-11T21:49:01.212 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-03-11T21:49:01.212 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:49:01.212 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:49:01.212 INFO:teuthology.orchestra.run.smithi097.stdout: "description": "osd_op(client.15744.0:70472 10.11 10.4ff1d3d1 (undecoded) ondisk+retry+write+known_if_redirected+supports_pool_eio e374)", 2024-03-11T21:49:01.212 INFO:teuthology.orchestra.run.smithi097.stdout: "initiated_at": "2024-03-11T21:41:29.844450+0000", 2024-03-11T21:49:01.212 INFO:teuthology.orchestra.run.smithi097.stdout: "age": 451.350956403, 2024-03-11T21:49:01.212 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.0282935, 2024-03-11T21:49:01.212 INFO:teuthology.orchestra.run.smithi097.stdout: "type_data": { 2024-03-11T21:49:01.212 INFO:teuthology.orchestra.run.smithi097.stdout: "flag_point": "reached pg", 2024-03-11T21:49:01.212 INFO:teuthology.orchestra.run.smithi097.stdout: "client_info": { 2024-03-11T21:49:01.212 INFO:teuthology.orchestra.run.smithi097.stdout: "client": "client.15744", 2024-03-11T21:49:01.212 INFO:teuthology.orchestra.run.smithi097.stdout: "client_addr": "v1:172.21.15.92:0/3334000812", 2024-03-11T21:49:01.212 INFO:teuthology.orchestra.run.smithi097.stdout: "tid": 70472 2024-03-11T21:49:01.212 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:49:01.213 INFO:teuthology.orchestra.run.smithi097.stdout: "events": [ 2024-03-11T21:49:01.213 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:49:01.213 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "initiated", 2024-03-11T21:49:01.213 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:41:29.844450+0000", 2024-03-11T21:49:01.213 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-03-11T21:49:01.213 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:49:01.213 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:49:01.213 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "header_read", 2024-03-11T21:49:01.213 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:41:29.844450+0000", 2024-03-11T21:49:01.213 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-03-11T21:49:01.213 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:49:01.213 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:49:01.213 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "throttled", 2024-03-11T21:49:01.214 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:41:29.844452+0000", 2024-03-11T21:49:01.214 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 2.0930000000000001e-06 2024-03-11T21:49:01.214 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:49:01.214 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:49:01.214 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "all_read", 2024-03-11T21:49:01.214 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:41:29.844516+0000", 2024-03-11T21:49:01.214 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 6.3874999999999994e-05 2024-03-11T21:49:01.214 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:49:01.214 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:49:01.214 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "dispatched", 2024-03-11T21:49:01.214 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:41:29.844533+0000", 2024-03-11T21:49:01.214 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.7371e-05 2024-03-11T21:49:01.214 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:49:01.215 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:49:01.215 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "queued_for_pg", 2024-03-11T21:49:01.215 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:41:29.844540+0000", 2024-03-11T21:49:01.215 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 7.1339999999999998e-06 2024-03-11T21:49:01.215 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:49:01.215 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:49:01.215 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "reached_pg", 2024-03-11T21:49:01.215 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:41:29.844595+0000", 2024-03-11T21:49:01.215 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 5.5065000000000002e-05 2024-03-11T21:49:01.215 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:49:01.215 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:49:01.215 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "waiting for peered", 2024-03-11T21:49:01.215 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:41:29.844604+0000", 2024-03-11T21:49:01.215 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 8.524e-06 2024-03-11T21:49:01.215 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:49:01.216 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:49:01.216 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "reached_pg", 2024-03-11T21:49:01.216 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:41:30.872734+0000", 2024-03-11T21:49:01.216 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.028130035 2024-03-11T21:49:01.216 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:49:01.216 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:49:01.216 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "done", 2024-03-11T21:49:01.216 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:41:30.872743+0000", 2024-03-11T21:49:01.216 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 9.4029999999999996e-06 2024-03-11T21:49:01.216 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-03-11T21:49:01.216 INFO:teuthology.orchestra.run.smithi097.stdout: ] 2024-03-11T21:49:01.216 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-03-11T21:49:01.216 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:49:01.216 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:49:01.216 INFO:teuthology.orchestra.run.smithi097.stdout: "description": "osd_op(client.15744.0:70484 10.11 10.b04c4bd1 (undecoded) ondisk+retry+write+known_if_redirected+supports_pool_eio e374)", 2024-03-11T21:49:01.217 INFO:teuthology.orchestra.run.smithi097.stdout: "initiated_at": "2024-03-11T21:41:29.844556+0000", 2024-03-11T21:49:01.217 INFO:teuthology.orchestra.run.smithi097.stdout: "age": 451.35085049399999, 2024-03-11T21:49:01.217 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.028216705, 2024-03-11T21:49:01.217 INFO:teuthology.orchestra.run.smithi097.stdout: "type_data": { 2024-03-11T21:49:01.217 INFO:teuthology.orchestra.run.smithi097.stdout: "flag_point": "reached pg", 2024-03-11T21:49:01.217 INFO:teuthology.orchestra.run.smithi097.stdout: "client_info": { 2024-03-11T21:49:01.217 INFO:teuthology.orchestra.run.smithi097.stdout: "client": "client.15744", 2024-03-11T21:49:01.217 INFO:teuthology.orchestra.run.smithi097.stdout: "client_addr": "v1:172.21.15.92:0/3334000812", 2024-03-11T21:49:01.217 INFO:teuthology.orchestra.run.smithi097.stdout: "tid": 70484 2024-03-11T21:49:01.217 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:49:01.217 INFO:teuthology.orchestra.run.smithi097.stdout: "events": [ 2024-03-11T21:49:01.217 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:49:01.217 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "initiated", 2024-03-11T21:49:01.217 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:41:29.844556+0000", 2024-03-11T21:49:01.218 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-03-11T21:49:01.218 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:49:01.218 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:49:01.218 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "header_read", 2024-03-11T21:49:01.218 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:41:29.844556+0000", 2024-03-11T21:49:01.218 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-03-11T21:49:01.218 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:49:01.218 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:49:01.218 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "throttled", 2024-03-11T21:49:01.218 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:41:29.844556+0000", 2024-03-11T21:49:01.218 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 8.0800000000000004e-07 2024-03-11T21:49:01.218 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:49:01.218 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:49:01.219 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "all_read", 2024-03-11T21:49:01.219 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:41:29.844612+0000", 2024-03-11T21:49:01.219 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 5.5507999999999998e-05 2024-03-11T21:49:01.219 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:49:01.219 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:49:01.219 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "dispatched", 2024-03-11T21:49:01.219 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:41:29.844618+0000", 2024-03-11T21:49:01.219 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 6.5699999999999998e-06 2024-03-11T21:49:01.219 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:49:01.219 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:49:01.219 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "queued_for_pg", 2024-03-11T21:49:01.219 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:41:29.844623+0000", 2024-03-11T21:49:01.219 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 4.6140000000000002e-06 2024-03-11T21:49:01.219 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:49:01.220 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:49:01.220 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "reached_pg", 2024-03-11T21:49:01.220 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:41:29.844659+0000", 2024-03-11T21:49:01.220 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 3.6102000000000001e-05 2024-03-11T21:49:01.220 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:49:01.220 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:49:01.220 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "waiting for peered", 2024-03-11T21:49:01.220 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:41:29.844665+0000", 2024-03-11T21:49:01.220 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 5.84e-06 2024-03-11T21:49:01.220 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:49:01.220 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:49:01.220 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "reached_pg", 2024-03-11T21:49:01.220 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:41:30.872766+0000", 2024-03-11T21:49:01.220 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.028101368 2024-03-11T21:49:01.220 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:49:01.221 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:49:01.221 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "done", 2024-03-11T21:49:01.221 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:41:30.872772+0000", 2024-03-11T21:49:01.221 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 5.8950000000000003e-06 2024-03-11T21:49:01.221 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-03-11T21:49:01.221 INFO:teuthology.orchestra.run.smithi097.stdout: ] 2024-03-11T21:49:01.221 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-03-11T21:49:01.221 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:49:01.221 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:49:01.221 INFO:teuthology.orchestra.run.smithi097.stdout: "description": "osd_op(client.15744.0:70492 10.11 10.25e828f1 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e374)", 2024-03-11T21:49:01.221 INFO:teuthology.orchestra.run.smithi097.stdout: "initiated_at": "2024-03-11T21:41:29.854518+0000", 2024-03-11T21:49:01.221 INFO:teuthology.orchestra.run.smithi097.stdout: "age": 451.34088781499997, 2024-03-11T21:49:01.221 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.018270996, 2024-03-11T21:49:01.221 INFO:teuthology.orchestra.run.smithi097.stdout: "type_data": { 2024-03-11T21:49:01.222 INFO:teuthology.orchestra.run.smithi097.stdout: "flag_point": "reached pg", 2024-03-11T21:49:01.222 INFO:teuthology.orchestra.run.smithi097.stdout: "client_info": { 2024-03-11T21:49:01.222 INFO:teuthology.orchestra.run.smithi097.stdout: "client": "client.15744", 2024-03-11T21:49:01.222 INFO:teuthology.orchestra.run.smithi097.stdout: "client_addr": "v1:172.21.15.92:0/3334000812", 2024-03-11T21:49:01.222 INFO:teuthology.orchestra.run.smithi097.stdout: "tid": 70492 2024-03-11T21:49:01.222 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:49:01.222 INFO:teuthology.orchestra.run.smithi097.stdout: "events": [ 2024-03-11T21:49:01.222 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:49:01.222 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "initiated", 2024-03-11T21:49:01.222 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:41:29.854518+0000", 2024-03-11T21:49:01.222 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-03-11T21:49:01.222 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:49:01.222 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:49:01.222 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "header_read", 2024-03-11T21:49:01.223 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:41:29.854518+0000", 2024-03-11T21:49:01.223 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-03-11T21:49:01.223 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:49:01.223 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:49:01.223 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "throttled", 2024-03-11T21:49:01.223 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:41:29.854519+0000", 2024-03-11T21:49:01.223 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 7.5300000000000003e-07 2024-03-11T21:49:01.223 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:49:01.223 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:49:01.223 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "all_read", 2024-03-11T21:49:01.223 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:41:29.854571+0000", 2024-03-11T21:49:01.223 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 5.2125e-05 2024-03-11T21:49:01.223 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:49:01.224 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:49:01.224 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "dispatched", 2024-03-11T21:49:01.224 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:41:29.854578+0000", 2024-03-11T21:49:01.224 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 7.0029999999999997e-06 2024-03-11T21:49:01.224 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:49:01.224 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:49:01.224 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "queued_for_pg", 2024-03-11T21:49:01.224 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:41:29.854584+0000", 2024-03-11T21:49:01.224 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 5.8680000000000004e-06 2024-03-11T21:49:01.224 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:49:01.224 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:49:01.224 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "reached_pg", 2024-03-11T21:49:01.224 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:41:29.865070+0000", 2024-03-11T21:49:01.224 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.010485789000000001 2024-03-11T21:49:01.225 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:49:01.225 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:49:01.225 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "waiting for peered", 2024-03-11T21:49:01.225 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:41:29.865078+0000", 2024-03-11T21:49:01.225 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 7.8490000000000006e-06 2024-03-11T21:49:01.225 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:49:01.225 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:49:01.225 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "reached_pg", 2024-03-11T21:49:01.225 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:41:30.872785+0000", 2024-03-11T21:49:01.225 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.00770771 2024-03-11T21:49:01.225 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:49:01.225 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:49:01.225 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "done", 2024-03-11T21:49:01.225 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:41:30.872789+0000", 2024-03-11T21:49:01.226 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 3.8990000000000003e-06 2024-03-11T21:49:01.226 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-03-11T21:49:01.226 INFO:teuthology.orchestra.run.smithi097.stdout: ] 2024-03-11T21:49:01.226 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-03-11T21:49:01.226 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:49:01.226 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:49:01.226 INFO:teuthology.orchestra.run.smithi097.stdout: "description": "osd_op(client.15744.0:70505 10.11 10.fdec1d51 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e374)", 2024-03-11T21:49:01.226 INFO:teuthology.orchestra.run.smithi097.stdout: "initiated_at": "2024-03-11T21:41:29.866294+0000", 2024-03-11T21:49:01.226 INFO:teuthology.orchestra.run.smithi097.stdout: "age": 451.32911205200003, 2024-03-11T21:49:01.226 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.006510808, 2024-03-11T21:49:01.226 INFO:teuthology.orchestra.run.smithi097.stdout: "type_data": { 2024-03-11T21:49:01.226 INFO:teuthology.orchestra.run.smithi097.stdout: "flag_point": "reached pg", 2024-03-11T21:49:01.226 INFO:teuthology.orchestra.run.smithi097.stdout: "client_info": { 2024-03-11T21:49:01.226 INFO:teuthology.orchestra.run.smithi097.stdout: "client": "client.15744", 2024-03-11T21:49:01.227 INFO:teuthology.orchestra.run.smithi097.stdout: "client_addr": "v1:172.21.15.92:0/3334000812", 2024-03-11T21:49:01.227 INFO:teuthology.orchestra.run.smithi097.stdout: "tid": 70505 2024-03-11T21:49:01.227 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:49:01.227 INFO:teuthology.orchestra.run.smithi097.stdout: "events": [ 2024-03-11T21:49:01.227 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:49:01.227 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "initiated", 2024-03-11T21:49:01.227 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:41:29.866294+0000", 2024-03-11T21:49:01.227 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-03-11T21:49:01.227 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:49:01.227 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:49:01.227 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "header_read", 2024-03-11T21:49:01.227 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:41:29.866294+0000", 2024-03-11T21:49:01.227 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-03-11T21:49:01.227 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:49:01.227 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:49:01.228 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "throttled", 2024-03-11T21:49:01.228 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:41:29.866296+0000", 2024-03-11T21:49:01.228 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 2.3659999999999999e-06 2024-03-11T21:49:01.228 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:49:01.228 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:49:01.228 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "all_read", 2024-03-11T21:49:01.228 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:41:29.866365+0000", 2024-03-11T21:49:01.228 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 6.8263999999999996e-05 2024-03-11T21:49:01.228 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:49:01.228 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:49:01.228 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "dispatched", 2024-03-11T21:49:01.228 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:41:29.866381+0000", 2024-03-11T21:49:01.228 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.6605000000000001e-05 2024-03-11T21:49:01.228 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:49:01.228 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:49:01.229 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "queued_for_pg", 2024-03-11T21:49:01.229 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:41:29.866398+0000", 2024-03-11T21:49:01.229 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.6764000000000001e-05 2024-03-11T21:49:01.229 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:49:01.229 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:49:01.229 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "reached_pg", 2024-03-11T21:49:01.229 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:41:29.866623+0000", 2024-03-11T21:49:01.229 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.00022518899999999999 2024-03-11T21:49:01.229 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:49:01.229 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:49:01.229 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "waiting for peered", 2024-03-11T21:49:01.229 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:41:29.866631+0000", 2024-03-11T21:49:01.230 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 7.4050000000000003e-06 2024-03-11T21:49:01.230 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:49:01.230 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:49:01.230 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "reached_pg", 2024-03-11T21:49:01.230 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:41:30.872801+0000", 2024-03-11T21:49:01.230 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.0061703289999999 2024-03-11T21:49:01.230 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:49:01.230 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:49:01.230 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "done", 2024-03-11T21:49:01.230 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:41:30.872805+0000", 2024-03-11T21:49:01.230 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 3.8859999999999997e-06 2024-03-11T21:49:01.230 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-03-11T21:49:01.231 INFO:teuthology.orchestra.run.smithi097.stdout: ] 2024-03-11T21:49:01.231 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-03-11T21:49:01.231 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:49:01.231 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:49:01.231 INFO:teuthology.orchestra.run.smithi097.stdout: "description": "osd_op(client.35813.0:30491 11.7 11:e3f707f0:::benchmark_data_smithi092_186520_object30490:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e452)", 2024-03-11T21:49:01.231 INFO:teuthology.orchestra.run.smithi097.stdout: "initiated_at": "2024-03-11T21:45:18.648367+0000", 2024-03-11T21:49:01.231 INFO:teuthology.orchestra.run.smithi097.stdout: "age": 222.547039453, 2024-03-11T21:49:01.231 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.17078143800000001, 2024-03-11T21:49:01.231 INFO:teuthology.orchestra.run.smithi097.stdout: "type_data": { 2024-03-11T21:49:01.231 INFO:teuthology.orchestra.run.smithi097.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-11T21:49:01.231 INFO:teuthology.orchestra.run.smithi097.stdout: "client_info": { 2024-03-11T21:49:01.231 INFO:teuthology.orchestra.run.smithi097.stdout: "client": "client.35813", 2024-03-11T21:49:01.231 INFO:teuthology.orchestra.run.smithi097.stdout: "client_addr": "v1:172.21.15.92:0/4293352030", 2024-03-11T21:49:01.231 INFO:teuthology.orchestra.run.smithi097.stdout: "tid": 30491 2024-03-11T21:49:01.231 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:49:01.232 INFO:teuthology.orchestra.run.smithi097.stdout: "events": [ 2024-03-11T21:49:01.232 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:49:01.232 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "initiated", 2024-03-11T21:49:01.232 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:45:18.648367+0000", 2024-03-11T21:49:01.232 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-03-11T21:49:01.232 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:49:01.232 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:49:01.232 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "header_read", 2024-03-11T21:49:01.232 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:45:18.648367+0000", 2024-03-11T21:49:01.232 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-03-11T21:49:01.232 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:49:01.232 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:49:01.232 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "throttled", 2024-03-11T21:49:01.232 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:45:18.648368+0000", 2024-03-11T21:49:01.232 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 9.5600000000000004e-07 2024-03-11T21:49:01.233 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:49:01.233 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:49:01.233 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "all_read", 2024-03-11T21:49:01.233 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:45:18.648440+0000", 2024-03-11T21:49:01.233 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 7.2179000000000001e-05 2024-03-11T21:49:01.233 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:49:01.233 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:49:01.233 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "dispatched", 2024-03-11T21:49:01.233 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:45:18.648448+0000", 2024-03-11T21:49:01.233 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 8.5730000000000008e-06 2024-03-11T21:49:01.233 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:49:01.233 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:49:01.233 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "queued_for_pg", 2024-03-11T21:49:01.233 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:45:18.648455+0000", 2024-03-11T21:49:01.234 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 6.6340000000000003e-06 2024-03-11T21:49:01.234 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:49:01.234 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:49:01.234 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "reached_pg", 2024-03-11T21:49:01.234 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:45:18.650190+0000", 2024-03-11T21:49:01.234 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.001735491 2024-03-11T21:49:01.234 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:49:01.234 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:49:01.234 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "started", 2024-03-11T21:49:01.234 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:45:18.650324+0000", 2024-03-11T21:49:01.234 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.000133731 2024-03-11T21:49:01.234 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:49:01.234 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:49:01.235 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "waiting for subops from 0,1", 2024-03-11T21:49:01.235 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:45:18.650464+0000", 2024-03-11T21:49:01.235 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.00014002900000000001 2024-03-11T21:49:01.235 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:49:01.235 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:49:01.235 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "op_commit", 2024-03-11T21:49:01.235 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:45:18.653601+0000", 2024-03-11T21:49:01.235 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.0031366200000000001 2024-03-11T21:49:01.235 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:49:01.235 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:49:01.235 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "sub_op_commit_rec", 2024-03-11T21:49:01.235 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:45:18.656230+0000", 2024-03-11T21:49:01.235 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.0026295580000000002 2024-03-11T21:49:01.235 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:49:01.235 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:49:01.236 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "sub_op_commit_rec", 2024-03-11T21:49:01.236 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:45:18.819021+0000", 2024-03-11T21:49:01.236 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.16279080300000001 2024-03-11T21:49:01.236 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:49:01.236 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:49:01.236 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "commit_sent", 2024-03-11T21:49:01.236 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:45:18.819081+0000", 2024-03-11T21:49:01.236 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 5.9490000000000001e-05 2024-03-11T21:49:01.236 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:49:01.236 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:49:01.236 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "done", 2024-03-11T21:49:01.236 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:45:18.819148+0000", 2024-03-11T21:49:01.236 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 6.7373999999999996e-05 2024-03-11T21:49:01.236 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-03-11T21:49:01.237 INFO:teuthology.orchestra.run.smithi097.stdout: ] 2024-03-11T21:49:01.237 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-03-11T21:49:01.237 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:49:01.237 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:49:01.237 INFO:teuthology.orchestra.run.smithi097.stdout: "description": "osd_op(client.35813.0:30495 11.7 11:ed4a895b:::benchmark_data_smithi092_186520_object30494:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e452)", 2024-03-11T21:49:01.237 INFO:teuthology.orchestra.run.smithi097.stdout: "initiated_at": "2024-03-11T21:45:18.649868+0000", 2024-03-11T21:49:01.237 INFO:teuthology.orchestra.run.smithi097.stdout: "age": 222.54553765599999, 2024-03-11T21:49:01.237 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.17149969100000001, 2024-03-11T21:49:01.237 INFO:teuthology.orchestra.run.smithi097.stdout: "type_data": { 2024-03-11T21:49:01.237 INFO:teuthology.orchestra.run.smithi097.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-11T21:49:01.237 INFO:teuthology.orchestra.run.smithi097.stdout: "client_info": { 2024-03-11T21:49:01.237 INFO:teuthology.orchestra.run.smithi097.stdout: "client": "client.35813", 2024-03-11T21:49:01.237 INFO:teuthology.orchestra.run.smithi097.stdout: "client_addr": "v1:172.21.15.92:0/4293352030", 2024-03-11T21:49:01.237 INFO:teuthology.orchestra.run.smithi097.stdout: "tid": 30495 2024-03-11T21:49:01.237 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:49:01.238 INFO:teuthology.orchestra.run.smithi097.stdout: "events": [ 2024-03-11T21:49:01.238 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:49:01.238 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "initiated", 2024-03-11T21:49:01.238 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:45:18.649868+0000", 2024-03-11T21:49:01.238 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-03-11T21:49:01.238 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:49:01.238 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:49:01.238 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "header_read", 2024-03-11T21:49:01.238 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:45:18.649868+0000", 2024-03-11T21:49:01.238 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-03-11T21:49:01.239 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:49:01.239 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:49:01.239 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "throttled", 2024-03-11T21:49:01.239 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:45:18.649884+0000", 2024-03-11T21:49:01.239 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.523e-05 2024-03-11T21:49:01.239 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:49:01.239 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:49:01.239 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "all_read", 2024-03-11T21:49:01.239 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:45:18.649928+0000", 2024-03-11T21:49:01.239 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 4.4755000000000001e-05 2024-03-11T21:49:01.239 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:49:01.239 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:49:01.239 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "dispatched", 2024-03-11T21:49:01.239 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:45:18.649938+0000", 2024-03-11T21:49:01.239 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 9.5640000000000002e-06 2024-03-11T21:49:01.240 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:49:01.240 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:49:01.240 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "queued_for_pg", 2024-03-11T21:49:01.240 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:45:18.649945+0000", 2024-03-11T21:49:01.240 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 7.5190000000000003e-06 2024-03-11T21:49:01.240 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:49:01.240 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:49:01.240 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "reached_pg", 2024-03-11T21:49:01.240 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:45:18.651883+0000", 2024-03-11T21:49:01.240 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.001938005 2024-03-11T21:49:01.240 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:49:01.240 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:49:01.240 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "started", 2024-03-11T21:49:01.240 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:45:18.651997+0000", 2024-03-11T21:49:01.240 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.00011343 2024-03-11T21:49:01.241 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:49:01.241 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:49:01.241 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "waiting for subops from 0,1", 2024-03-11T21:49:01.241 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:45:18.652138+0000", 2024-03-11T21:49:01.241 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.00014142399999999999 2024-03-11T21:49:01.241 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:49:01.241 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:49:01.241 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "op_commit", 2024-03-11T21:49:01.241 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:45:18.653716+0000", 2024-03-11T21:49:01.241 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.0015779940000000001 2024-03-11T21:49:01.241 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:49:01.241 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:49:01.241 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "sub_op_commit_rec", 2024-03-11T21:49:01.241 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:45:18.656265+0000", 2024-03-11T21:49:01.242 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.0025489620000000001 2024-03-11T21:49:01.242 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:49:01.242 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:49:01.242 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "sub_op_commit_rec", 2024-03-11T21:49:01.242 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:45:18.821237+0000", 2024-03-11T21:49:01.242 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.16497202399999999 2024-03-11T21:49:01.242 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:49:01.242 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:49:01.242 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "commit_sent", 2024-03-11T21:49:01.242 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:45:18.821295+0000", 2024-03-11T21:49:01.242 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 5.7741000000000002e-05 2024-03-11T21:49:01.243 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:49:01.243 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:49:01.243 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "done", 2024-03-11T21:49:01.243 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:45:18.821368+0000", 2024-03-11T21:49:01.243 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 7.3042999999999998e-05 2024-03-11T21:49:01.243 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-03-11T21:49:01.243 INFO:teuthology.orchestra.run.smithi097.stdout: ] 2024-03-11T21:49:01.243 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-03-11T21:49:01.243 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:49:01.244 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:49:01.244 INFO:teuthology.orchestra.run.smithi097.stdout: "description": "osd_op(client.35813.0:34850 11.7 11:ed823033:::benchmark_data_smithi092_186520_object34849:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e452)", 2024-03-11T21:49:01.244 INFO:teuthology.orchestra.run.smithi097.stdout: "initiated_at": "2024-03-11T21:45:21.055078+0000", 2024-03-11T21:49:01.244 INFO:teuthology.orchestra.run.smithi097.stdout: "age": 220.14032803800001, 2024-03-11T21:49:01.244 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.22242576, 2024-03-11T21:49:01.244 INFO:teuthology.orchestra.run.smithi097.stdout: "type_data": { 2024-03-11T21:49:01.244 INFO:teuthology.orchestra.run.smithi097.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-11T21:49:01.245 INFO:teuthology.orchestra.run.smithi097.stdout: "client_info": { 2024-03-11T21:49:01.245 INFO:teuthology.orchestra.run.smithi097.stdout: "client": "client.35813", 2024-03-11T21:49:01.245 INFO:teuthology.orchestra.run.smithi097.stdout: "client_addr": "v1:172.21.15.92:0/4293352030", 2024-03-11T21:49:01.245 INFO:teuthology.orchestra.run.smithi097.stdout: "tid": 34850 2024-03-11T21:49:01.245 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:49:01.245 INFO:teuthology.orchestra.run.smithi097.stdout: "events": [ 2024-03-11T21:49:01.246 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:49:01.246 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "initiated", 2024-03-11T21:49:01.246 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:45:21.055078+0000", 2024-03-11T21:49:01.246 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-03-11T21:49:01.246 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:49:01.246 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:49:01.246 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "header_read", 2024-03-11T21:49:01.247 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:45:21.055078+0000", 2024-03-11T21:49:01.247 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-03-11T21:49:01.247 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:49:01.247 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:49:01.247 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "throttled", 2024-03-11T21:49:01.247 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:45:21.055080+0000", 2024-03-11T21:49:01.249 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.9069999999999999e-06 2024-03-11T21:49:01.249 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:49:01.249 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:49:01.249 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "all_read", 2024-03-11T21:49:01.249 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:45:21.055137+0000", 2024-03-11T21:49:01.249 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 5.6962000000000001e-05 2024-03-11T21:49:01.249 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:49:01.249 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:49:01.250 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "dispatched", 2024-03-11T21:49:01.250 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:45:21.055151+0000", 2024-03-11T21:49:01.250 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.4234e-05 2024-03-11T21:49:01.250 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:49:01.250 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:49:01.250 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "queued_for_pg", 2024-03-11T21:49:01.250 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:45:21.055159+0000", 2024-03-11T21:49:01.250 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 7.4909999999999999e-06 2024-03-11T21:49:01.251 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:49:01.251 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:49:01.251 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "reached_pg", 2024-03-11T21:49:01.251 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:45:21.055225+0000", 2024-03-11T21:49:01.251 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 6.6041e-05 2024-03-11T21:49:01.251 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:49:01.251 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:49:01.251 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "started", 2024-03-11T21:49:01.252 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:45:21.055377+0000", 2024-03-11T21:49:01.252 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.00015268899999999999 2024-03-11T21:49:01.252 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:49:01.252 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:49:01.252 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "waiting for subops from 0,1", 2024-03-11T21:49:01.252 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:45:21.055521+0000", 2024-03-11T21:49:01.252 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.000144077 2024-03-11T21:49:01.252 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:49:01.253 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:49:01.253 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "op_commit", 2024-03-11T21:49:01.253 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:45:21.056955+0000", 2024-03-11T21:49:01.253 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.0014334840000000001 2024-03-11T21:49:01.253 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:49:01.253 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:49:01.253 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "sub_op_commit_rec", 2024-03-11T21:49:01.253 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:45:21.057573+0000", 2024-03-11T21:49:01.253 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.00061841800000000003 2024-03-11T21:49:01.254 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:49:01.254 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:49:01.254 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "sub_op_commit_rec", 2024-03-11T21:49:01.254 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:45:21.277395+0000", 2024-03-11T21:49:01.254 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.21982129 2024-03-11T21:49:01.254 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:49:01.254 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:49:01.254 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "commit_sent", 2024-03-11T21:49:01.254 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:45:21.277451+0000", 2024-03-11T21:49:01.254 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 5.6045999999999998e-05 2024-03-11T21:49:01.254 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:49:01.254 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:49:01.254 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "done", 2024-03-11T21:49:01.255 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:45:21.277504+0000", 2024-03-11T21:49:01.255 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 5.3121000000000002e-05 2024-03-11T21:49:01.255 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-03-11T21:49:01.255 INFO:teuthology.orchestra.run.smithi097.stdout: ] 2024-03-11T21:49:01.255 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-03-11T21:49:01.255 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:49:01.255 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:49:01.255 INFO:teuthology.orchestra.run.smithi097.stdout: "description": "osd_op(client.35813.0:34852 11.7 11:e42ce1d6:::benchmark_data_smithi092_186520_object34851:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e452)", 2024-03-11T21:49:01.255 INFO:teuthology.orchestra.run.smithi097.stdout: "initiated_at": "2024-03-11T21:45:21.057492+0000", 2024-03-11T21:49:01.255 INFO:teuthology.orchestra.run.smithi097.stdout: "age": 220.137914137, 2024-03-11T21:49:01.255 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.22018278699999999, 2024-03-11T21:49:01.255 INFO:teuthology.orchestra.run.smithi097.stdout: "type_data": { 2024-03-11T21:49:01.255 INFO:teuthology.orchestra.run.smithi097.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-11T21:49:01.255 INFO:teuthology.orchestra.run.smithi097.stdout: "client_info": { 2024-03-11T21:49:01.256 INFO:teuthology.orchestra.run.smithi097.stdout: "client": "client.35813", 2024-03-11T21:49:01.256 INFO:teuthology.orchestra.run.smithi097.stdout: "client_addr": "v1:172.21.15.92:0/4293352030", 2024-03-11T21:49:01.256 INFO:teuthology.orchestra.run.smithi097.stdout: "tid": 34852 2024-03-11T21:49:01.256 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:49:01.256 INFO:teuthology.orchestra.run.smithi097.stdout: "events": [ 2024-03-11T21:49:01.256 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:49:01.256 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "initiated", 2024-03-11T21:49:01.256 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:45:21.057492+0000", 2024-03-11T21:49:01.256 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-03-11T21:49:01.256 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:49:01.256 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:49:01.256 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "header_read", 2024-03-11T21:49:01.257 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:45:21.057492+0000", 2024-03-11T21:49:01.257 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-03-11T21:49:01.257 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:49:01.257 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:49:01.257 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "throttled", 2024-03-11T21:49:01.257 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:45:21.057493+0000", 2024-03-11T21:49:01.257 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.3260000000000001e-06 2024-03-11T21:49:01.257 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:49:01.257 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:49:01.257 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "all_read", 2024-03-11T21:49:01.257 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:45:21.057586+0000", 2024-03-11T21:49:01.257 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 9.2324000000000006e-05 2024-03-11T21:49:01.257 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:49:01.257 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:49:01.257 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "dispatched", 2024-03-11T21:49:01.258 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:45:21.057596+0000", 2024-03-11T21:49:01.258 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.0079e-05 2024-03-11T21:49:01.258 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:49:01.258 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:49:01.258 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "queued_for_pg", 2024-03-11T21:49:01.258 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:45:21.057603+0000", 2024-03-11T21:49:01.258 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 7.4909999999999999e-06 2024-03-11T21:49:01.258 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:49:01.258 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:49:01.258 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "reached_pg", 2024-03-11T21:49:01.258 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:45:21.057645+0000", 2024-03-11T21:49:01.258 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 4.1766e-05 2024-03-11T21:49:01.258 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:49:01.259 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:49:01.259 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "started", 2024-03-11T21:49:01.259 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:45:21.057778+0000", 2024-03-11T21:49:01.259 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.000133077 2024-03-11T21:49:01.259 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:49:01.259 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:49:01.259 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "waiting for subops from 0,1", 2024-03-11T21:49:01.259 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:45:21.057927+0000", 2024-03-11T21:49:01.260 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.00014905800000000001 2024-03-11T21:49:01.260 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:49:01.260 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:49:01.260 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "op_commit", 2024-03-11T21:49:01.260 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:45:21.059303+0000", 2024-03-11T21:49:01.260 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.0013764879999999999 2024-03-11T21:49:01.260 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:49:01.261 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:49:01.261 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "sub_op_commit_rec", 2024-03-11T21:49:01.261 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:45:21.061007+0000", 2024-03-11T21:49:01.261 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.0017039480000000001 2024-03-11T21:49:01.261 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:49:01.261 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:49:01.261 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "sub_op_commit_rec", 2024-03-11T21:49:01.261 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:45:21.277612+0000", 2024-03-11T21:49:01.261 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.21660477 2024-03-11T21:49:01.261 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:49:01.261 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:49:01.261 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "commit_sent", 2024-03-11T21:49:01.261 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:45:21.277648+0000", 2024-03-11T21:49:01.261 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 3.6028999999999997e-05 2024-03-11T21:49:01.262 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:49:01.262 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:49:01.262 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "done", 2024-03-11T21:49:01.262 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:45:21.277675+0000", 2024-03-11T21:49:01.262 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 2.6431000000000002e-05 2024-03-11T21:49:01.262 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-03-11T21:49:01.262 INFO:teuthology.orchestra.run.smithi097.stdout: ] 2024-03-11T21:49:01.262 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-03-11T21:49:01.262 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:49:01.262 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:49:01.262 INFO:teuthology.orchestra.run.smithi097.stdout: "description": "osd_op(client.35813.0:34856 11.7 11:e0299c5b:::benchmark_data_smithi092_186520_object34855:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e452)", 2024-03-11T21:49:01.262 INFO:teuthology.orchestra.run.smithi097.stdout: "initiated_at": "2024-03-11T21:45:21.059553+0000", 2024-03-11T21:49:01.262 INFO:teuthology.orchestra.run.smithi097.stdout: "age": 220.135853032, 2024-03-11T21:49:01.262 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.22068811999999999, 2024-03-11T21:49:01.263 INFO:teuthology.orchestra.run.smithi097.stdout: "type_data": { 2024-03-11T21:49:01.263 INFO:teuthology.orchestra.run.smithi097.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-11T21:49:01.263 INFO:teuthology.orchestra.run.smithi097.stdout: "client_info": { 2024-03-11T21:49:01.263 INFO:teuthology.orchestra.run.smithi097.stdout: "client": "client.35813", 2024-03-11T21:49:01.263 INFO:teuthology.orchestra.run.smithi097.stdout: "client_addr": "v1:172.21.15.92:0/4293352030", 2024-03-11T21:49:01.263 INFO:teuthology.orchestra.run.smithi097.stdout: "tid": 34856 2024-03-11T21:49:01.263 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:49:01.263 INFO:teuthology.orchestra.run.smithi097.stdout: "events": [ 2024-03-11T21:49:01.263 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:49:01.263 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "initiated", 2024-03-11T21:49:01.263 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:45:21.059553+0000", 2024-03-11T21:49:01.263 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-03-11T21:49:01.263 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:49:01.263 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:49:01.263 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "header_read", 2024-03-11T21:49:01.264 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:45:21.059553+0000", 2024-03-11T21:49:01.264 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-03-11T21:49:01.264 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:49:01.264 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:49:01.264 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "throttled", 2024-03-11T21:49:01.264 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:45:21.059555+0000", 2024-03-11T21:49:01.264 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.7630000000000001e-06 2024-03-11T21:49:01.264 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:49:01.264 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:49:01.264 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "all_read", 2024-03-11T21:49:01.264 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:45:21.059585+0000", 2024-03-11T21:49:01.264 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 3.0681e-05 2024-03-11T21:49:01.264 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:49:01.264 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:49:01.264 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "dispatched", 2024-03-11T21:49:01.265 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:45:21.059600+0000", 2024-03-11T21:49:01.265 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.4276e-05 2024-03-11T21:49:01.265 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:49:01.265 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:49:01.265 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "queued_for_pg", 2024-03-11T21:49:01.265 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:45:21.059612+0000", 2024-03-11T21:49:01.265 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.2252000000000001e-05 2024-03-11T21:49:01.265 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:49:01.265 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:49:01.265 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "reached_pg", 2024-03-11T21:49:01.265 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:45:21.059664+0000", 2024-03-11T21:49:01.265 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 5.2281999999999999e-05 2024-03-11T21:49:01.265 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:49:01.265 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:49:01.266 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "started", 2024-03-11T21:49:01.266 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:45:21.059778+0000", 2024-03-11T21:49:01.266 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.000113337 2024-03-11T21:49:01.266 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:49:01.266 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:49:01.266 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "waiting for subops from 0,1", 2024-03-11T21:49:01.266 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:45:21.059865+0000", 2024-03-11T21:49:01.266 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 8.7064000000000005e-05 2024-03-11T21:49:01.266 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:49:01.266 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:49:01.266 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "op_commit", 2024-03-11T21:49:01.266 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:45:21.061092+0000", 2024-03-11T21:49:01.266 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.001227227 2024-03-11T21:49:01.266 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:49:01.267 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:49:01.267 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "sub_op_commit_rec", 2024-03-11T21:49:01.267 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:45:21.061719+0000", 2024-03-11T21:49:01.267 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.00062713899999999997 2024-03-11T21:49:01.267 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:49:01.267 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:49:01.267 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "sub_op_commit_rec", 2024-03-11T21:49:01.267 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:45:21.280143+0000", 2024-03-11T21:49:01.267 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.218424275 2024-03-11T21:49:01.267 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:49:01.267 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:49:01.267 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "commit_sent", 2024-03-11T21:49:01.267 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:45:21.280200+0000", 2024-03-11T21:49:01.267 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 5.626e-05 2024-03-11T21:49:01.268 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:49:01.268 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:49:01.268 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "done", 2024-03-11T21:49:01.268 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:45:21.280241+0000", 2024-03-11T21:49:01.268 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 4.1563999999999997e-05 2024-03-11T21:49:01.268 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-03-11T21:49:01.268 INFO:teuthology.orchestra.run.smithi097.stdout: ] 2024-03-11T21:49:01.268 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-03-11T21:49:01.268 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:49:01.268 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:49:01.268 INFO:teuthology.orchestra.run.smithi097.stdout: "description": "osd_op(client.35813.0:141979 11.7 11:e5780949:::benchmark_data_smithi092_186520_object3631:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e469)", 2024-03-11T21:49:01.268 INFO:teuthology.orchestra.run.smithi097.stdout: "initiated_at": "2024-03-11T21:46:39.739692+0000", 2024-03-11T21:49:01.268 INFO:teuthology.orchestra.run.smithi097.stdout: "age": 141.45571367299999, 2024-03-11T21:49:01.268 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.053606041, 2024-03-11T21:49:01.269 INFO:teuthology.orchestra.run.smithi097.stdout: "type_data": { 2024-03-11T21:49:01.269 INFO:teuthology.orchestra.run.smithi097.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-11T21:49:01.269 INFO:teuthology.orchestra.run.smithi097.stdout: "client_info": { 2024-03-11T21:49:01.269 INFO:teuthology.orchestra.run.smithi097.stdout: "client": "client.35813", 2024-03-11T21:49:01.269 INFO:teuthology.orchestra.run.smithi097.stdout: "client_addr": "v1:172.21.15.92:0/4293352030", 2024-03-11T21:49:01.269 INFO:teuthology.orchestra.run.smithi097.stdout: "tid": 141979 2024-03-11T21:49:01.269 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:49:01.269 INFO:teuthology.orchestra.run.smithi097.stdout: "events": [ 2024-03-11T21:49:01.269 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:49:01.269 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "initiated", 2024-03-11T21:49:01.269 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:46:39.739692+0000", 2024-03-11T21:49:01.269 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-03-11T21:49:01.269 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:49:01.269 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:49:01.270 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "header_read", 2024-03-11T21:49:01.270 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:46:39.739692+0000", 2024-03-11T21:49:01.270 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-03-11T21:49:01.270 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:49:01.270 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:49:01.270 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "throttled", 2024-03-11T21:49:01.270 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:46:39.739695+0000", 2024-03-11T21:49:01.270 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 2.2029999999999999e-06 2024-03-11T21:49:01.270 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:49:01.270 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:49:01.270 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "all_read", 2024-03-11T21:49:01.270 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:46:39.739701+0000", 2024-03-11T21:49:01.270 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 6.0700000000000003e-06 2024-03-11T21:49:01.270 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:49:01.271 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:49:01.271 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "dispatched", 2024-03-11T21:49:01.271 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:46:39.739716+0000", 2024-03-11T21:49:01.271 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.5006e-05 2024-03-11T21:49:01.271 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:49:01.271 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:49:01.271 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "queued_for_pg", 2024-03-11T21:49:01.271 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:46:39.739724+0000", 2024-03-11T21:49:01.271 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 7.9389999999999997e-06 2024-03-11T21:49:01.271 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:49:01.271 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:49:01.271 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "reached_pg", 2024-03-11T21:49:01.271 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:46:39.739784+0000", 2024-03-11T21:49:01.271 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 6.0136000000000002e-05 2024-03-11T21:49:01.272 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:49:01.272 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:49:01.272 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "started", 2024-03-11T21:49:01.272 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:46:39.739891+0000", 2024-03-11T21:49:01.272 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.00010730600000000001 2024-03-11T21:49:01.272 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:49:01.272 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:49:01.272 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "waiting for subops from 0,1", 2024-03-11T21:49:01.272 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:46:39.739992+0000", 2024-03-11T21:49:01.272 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.000101084 2024-03-11T21:49:01.272 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:49:01.273 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:49:01.273 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "op_commit", 2024-03-11T21:49:01.273 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:46:39.742059+0000", 2024-03-11T21:49:01.273 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.00206653 2024-03-11T21:49:01.273 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:49:01.273 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:49:01.273 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "sub_op_commit_rec", 2024-03-11T21:49:01.273 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:46:39.742449+0000", 2024-03-11T21:49:01.273 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.00039009800000000002 2024-03-11T21:49:01.273 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:49:01.273 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:49:01.273 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "sub_op_commit_rec", 2024-03-11T21:49:01.273 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:46:39.793200+0000", 2024-03-11T21:49:01.273 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.050751034 2024-03-11T21:49:01.274 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:49:01.274 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:49:01.274 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "commit_sent", 2024-03-11T21:49:01.274 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:46:39.793261+0000", 2024-03-11T21:49:01.274 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 6.0758999999999999e-05 2024-03-11T21:49:01.274 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:49:01.274 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:49:01.274 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "done", 2024-03-11T21:49:01.274 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:46:39.793298+0000", 2024-03-11T21:49:01.275 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 3.7876000000000001e-05 2024-03-11T21:49:01.275 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-03-11T21:49:01.275 INFO:teuthology.orchestra.run.smithi097.stdout: ] 2024-03-11T21:49:01.275 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-03-11T21:49:01.275 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:49:01.275 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:49:01.275 INFO:teuthology.orchestra.run.smithi097.stdout: "description": "osd_op(client.35813.0:142225 11.7 11:e5f12e73:::benchmark_data_smithi092_186520_object3877:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e469)", 2024-03-11T21:49:01.275 INFO:teuthology.orchestra.run.smithi097.stdout: "initiated_at": "2024-03-11T21:46:39.934890+0000", 2024-03-11T21:49:01.275 INFO:teuthology.orchestra.run.smithi097.stdout: "age": 141.26051579899999, 2024-03-11T21:49:01.275 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.097977395999999994, 2024-03-11T21:49:01.276 INFO:teuthology.orchestra.run.smithi097.stdout: "type_data": { 2024-03-11T21:49:01.276 INFO:teuthology.orchestra.run.smithi097.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-11T21:49:01.276 INFO:teuthology.orchestra.run.smithi097.stdout: "client_info": { 2024-03-11T21:49:01.276 INFO:teuthology.orchestra.run.smithi097.stdout: "client": "client.35813", 2024-03-11T21:49:01.276 INFO:teuthology.orchestra.run.smithi097.stdout: "client_addr": "v1:172.21.15.92:0/4293352030", 2024-03-11T21:49:01.276 INFO:teuthology.orchestra.run.smithi097.stdout: "tid": 142225 2024-03-11T21:49:01.276 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:49:01.276 INFO:teuthology.orchestra.run.smithi097.stdout: "events": [ 2024-03-11T21:49:01.276 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:49:01.276 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "initiated", 2024-03-11T21:49:01.276 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:46:39.934890+0000", 2024-03-11T21:49:01.276 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-03-11T21:49:01.276 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:49:01.276 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:49:01.276 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "header_read", 2024-03-11T21:49:01.277 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:46:39.934890+0000", 2024-03-11T21:49:01.277 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-03-11T21:49:01.277 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:49:01.277 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:49:01.277 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "throttled", 2024-03-11T21:49:01.277 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:46:39.934891+0000", 2024-03-11T21:49:01.277 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.184e-06 2024-03-11T21:49:01.277 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:49:01.277 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:49:01.277 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "all_read", 2024-03-11T21:49:01.277 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:46:39.934895+0000", 2024-03-11T21:49:01.278 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 3.6650000000000001e-06 2024-03-11T21:49:01.278 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:49:01.278 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:49:01.278 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "dispatched", 2024-03-11T21:49:01.278 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:46:39.934905+0000", 2024-03-11T21:49:01.278 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 9.7340000000000003e-06 2024-03-11T21:49:01.278 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:49:01.278 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:49:01.278 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "queued_for_pg", 2024-03-11T21:49:01.279 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:46:39.934911+0000", 2024-03-11T21:49:01.279 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 6.3099999999999997e-06 2024-03-11T21:49:01.279 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:49:01.279 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:49:01.279 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "reached_pg", 2024-03-11T21:49:01.279 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:46:39.934974+0000", 2024-03-11T21:49:01.279 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 6.3069000000000002e-05 2024-03-11T21:49:01.279 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:49:01.279 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:49:01.279 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "started", 2024-03-11T21:49:01.279 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:46:39.935115+0000", 2024-03-11T21:49:01.279 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.000141008 2024-03-11T21:49:01.279 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:49:01.279 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:49:01.279 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "waiting for subops from 0,1", 2024-03-11T21:49:01.280 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:46:39.935230+0000", 2024-03-11T21:49:01.280 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.000115028 2024-03-11T21:49:01.280 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:49:01.280 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:49:01.280 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "op_commit", 2024-03-11T21:49:01.280 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:46:39.937403+0000", 2024-03-11T21:49:01.280 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.002172588 2024-03-11T21:49:01.280 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:49:01.280 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:49:01.280 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "sub_op_commit_rec", 2024-03-11T21:49:01.280 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:46:39.937711+0000", 2024-03-11T21:49:01.280 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.00030824299999999998 2024-03-11T21:49:01.280 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:49:01.280 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:49:01.281 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "sub_op_commit_rec", 2024-03-11T21:49:01.281 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:46:40.032789+0000", 2024-03-11T21:49:01.281 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.095078411000000002 2024-03-11T21:49:01.281 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:49:01.281 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:49:01.281 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "commit_sent", 2024-03-11T21:49:01.281 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:46:40.032829+0000", 2024-03-11T21:49:01.281 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 3.9959999999999997e-05 2024-03-11T21:49:01.281 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:49:01.281 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:49:01.281 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "done", 2024-03-11T21:49:01.281 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:46:40.032868+0000", 2024-03-11T21:49:01.281 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 3.8195999999999998e-05 2024-03-11T21:49:01.281 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-03-11T21:49:01.281 INFO:teuthology.orchestra.run.smithi097.stdout: ] 2024-03-11T21:49:01.282 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-03-11T21:49:01.282 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:49:01.282 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:49:01.282 INFO:teuthology.orchestra.run.smithi097.stdout: "description": "osd_op(client.35813.0:144080 11.7 11:e8d0d563:::benchmark_data_smithi092_186520_object5732:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e469)", 2024-03-11T21:49:01.282 INFO:teuthology.orchestra.run.smithi097.stdout: "initiated_at": "2024-03-11T21:46:41.244160+0000", 2024-03-11T21:49:01.282 INFO:teuthology.orchestra.run.smithi097.stdout: "age": 139.951246177, 2024-03-11T21:49:01.282 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.062668014999999994, 2024-03-11T21:49:01.282 INFO:teuthology.orchestra.run.smithi097.stdout: "type_data": { 2024-03-11T21:49:01.282 INFO:teuthology.orchestra.run.smithi097.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-11T21:49:01.282 INFO:teuthology.orchestra.run.smithi097.stdout: "client_info": { 2024-03-11T21:49:01.282 INFO:teuthology.orchestra.run.smithi097.stdout: "client": "client.35813", 2024-03-11T21:49:01.282 INFO:teuthology.orchestra.run.smithi097.stdout: "client_addr": "v1:172.21.15.92:0/4293352030", 2024-03-11T21:49:01.282 INFO:teuthology.orchestra.run.smithi097.stdout: "tid": 144080 2024-03-11T21:49:01.283 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:49:01.283 INFO:teuthology.orchestra.run.smithi097.stdout: "events": [ 2024-03-11T21:49:01.283 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:49:01.283 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "initiated", 2024-03-11T21:49:01.283 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:46:41.244160+0000", 2024-03-11T21:49:01.283 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-03-11T21:49:01.283 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:49:01.283 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:49:01.283 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "header_read", 2024-03-11T21:49:01.283 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:46:41.244160+0000", 2024-03-11T21:49:01.283 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-03-11T21:49:01.283 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:49:01.283 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:49:01.283 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "throttled", 2024-03-11T21:49:01.284 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:46:41.244162+0000", 2024-03-11T21:49:01.284 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.9889999999999999e-06 2024-03-11T21:49:01.284 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:49:01.284 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:49:01.284 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "all_read", 2024-03-11T21:49:01.284 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:46:41.244168+0000", 2024-03-11T21:49:01.284 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 6.0739999999999998e-06 2024-03-11T21:49:01.284 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:49:01.284 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:49:01.284 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "dispatched", 2024-03-11T21:49:01.284 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:46:41.244184+0000", 2024-03-11T21:49:01.284 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.5934999999999999e-05 2024-03-11T21:49:01.284 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:49:01.284 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:49:01.284 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "queued_for_pg", 2024-03-11T21:49:01.284 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:46:41.244192+0000", 2024-03-11T21:49:01.285 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 7.9820000000000008e-06 2024-03-11T21:49:01.285 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:49:01.285 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:49:01.285 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "reached_pg", 2024-03-11T21:49:01.285 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:46:41.244261+0000", 2024-03-11T21:49:01.285 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 6.9630999999999998e-05 2024-03-11T21:49:01.285 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:49:01.285 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:49:01.285 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "started", 2024-03-11T21:49:01.285 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:46:41.244409+0000", 2024-03-11T21:49:01.285 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.00014721299999999999 2024-03-11T21:49:01.285 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:49:01.285 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:49:01.285 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "waiting for subops from 0,1", 2024-03-11T21:49:01.286 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:46:41.244551+0000", 2024-03-11T21:49:01.286 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.00014257000000000001 2024-03-11T21:49:01.286 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:49:01.286 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:49:01.286 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "op_commit", 2024-03-11T21:49:01.286 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:46:41.246984+0000", 2024-03-11T21:49:01.286 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.002432544 2024-03-11T21:49:01.286 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:49:01.286 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:49:01.286 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "sub_op_commit_rec", 2024-03-11T21:49:01.286 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:46:41.247293+0000", 2024-03-11T21:49:01.286 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.000308881 2024-03-11T21:49:01.286 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:49:01.287 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:49:01.287 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "sub_op_commit_rec", 2024-03-11T21:49:01.287 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:46:41.306756+0000", 2024-03-11T21:49:01.287 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.059463325999999997 2024-03-11T21:49:01.287 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:49:01.287 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:49:01.287 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "commit_sent", 2024-03-11T21:49:01.287 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:46:41.306796+0000", 2024-03-11T21:49:01.287 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 3.9674e-05 2024-03-11T21:49:01.287 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:49:01.287 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:49:01.287 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "done", 2024-03-11T21:49:01.287 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:46:41.306828+0000", 2024-03-11T21:49:01.287 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 3.2196000000000001e-05 2024-03-11T21:49:01.288 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-03-11T21:49:01.288 INFO:teuthology.orchestra.run.smithi097.stdout: ] 2024-03-11T21:49:01.288 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-03-11T21:49:01.288 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:49:01.288 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:49:01.288 INFO:teuthology.orchestra.run.smithi097.stdout: "description": "osd_op(client.35813.0:144472 11.7 11:e97414da:::benchmark_data_smithi092_186520_object6124:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e469)", 2024-03-11T21:49:01.288 INFO:teuthology.orchestra.run.smithi097.stdout: "initiated_at": "2024-03-11T21:46:41.581866+0000", 2024-03-11T21:49:01.288 INFO:teuthology.orchestra.run.smithi097.stdout: "age": 139.613540476, 2024-03-11T21:49:01.288 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.047265349999999998, 2024-03-11T21:49:01.288 INFO:teuthology.orchestra.run.smithi097.stdout: "type_data": { 2024-03-11T21:49:01.288 INFO:teuthology.orchestra.run.smithi097.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-11T21:49:01.288 INFO:teuthology.orchestra.run.smithi097.stdout: "client_info": { 2024-03-11T21:49:01.288 INFO:teuthology.orchestra.run.smithi097.stdout: "client": "client.35813", 2024-03-11T21:49:01.288 INFO:teuthology.orchestra.run.smithi097.stdout: "client_addr": "v1:172.21.15.92:0/4293352030", 2024-03-11T21:49:01.288 INFO:teuthology.orchestra.run.smithi097.stdout: "tid": 144472 2024-03-11T21:49:01.289 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:49:01.289 INFO:teuthology.orchestra.run.smithi097.stdout: "events": [ 2024-03-11T21:49:01.289 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:49:01.289 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "initiated", 2024-03-11T21:49:01.289 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:46:41.581866+0000", 2024-03-11T21:49:01.289 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-03-11T21:49:01.289 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:49:01.289 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:49:01.289 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "header_read", 2024-03-11T21:49:01.289 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:46:41.581866+0000", 2024-03-11T21:49:01.289 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-03-11T21:49:01.289 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:49:01.289 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:49:01.289 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "throttled", 2024-03-11T21:49:01.289 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:46:41.581868+0000", 2024-03-11T21:49:01.290 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 2.0020000000000001e-06 2024-03-11T21:49:01.290 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:49:01.290 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:49:01.290 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "all_read", 2024-03-11T21:49:01.290 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:46:41.581874+0000", 2024-03-11T21:49:01.290 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 6.2820000000000002e-06 2024-03-11T21:49:01.290 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:49:01.290 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:49:01.290 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "dispatched", 2024-03-11T21:49:01.290 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:46:41.581886+0000", 2024-03-11T21:49:01.290 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.1989e-05 2024-03-11T21:49:01.290 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:49:01.290 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:49:01.291 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "queued_for_pg", 2024-03-11T21:49:01.291 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:46:41.581894+0000", 2024-03-11T21:49:01.291 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 7.9719999999999992e-06 2024-03-11T21:49:01.291 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:49:01.291 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:49:01.291 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "reached_pg", 2024-03-11T21:49:01.291 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:46:41.581987+0000", 2024-03-11T21:49:01.291 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 9.3163000000000002e-05 2024-03-11T21:49:01.291 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:49:01.291 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:49:01.291 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "started", 2024-03-11T21:49:01.291 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:46:41.582112+0000", 2024-03-11T21:49:01.291 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.000125 2024-03-11T21:49:01.292 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:49:01.292 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:49:01.292 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "waiting for subops from 0,1", 2024-03-11T21:49:01.292 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:46:41.582231+0000", 2024-03-11T21:49:01.292 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.00011875999999999999 2024-03-11T21:49:01.292 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:49:01.292 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:49:01.292 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "op_commit", 2024-03-11T21:49:01.292 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:46:41.584860+0000", 2024-03-11T21:49:01.292 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.0026288309999999999 2024-03-11T21:49:01.292 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:49:01.292 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:49:01.292 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "sub_op_commit_rec", 2024-03-11T21:49:01.292 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:46:41.585975+0000", 2024-03-11T21:49:01.292 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.0011159169999999999 2024-03-11T21:49:01.293 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:49:01.293 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:49:01.293 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "sub_op_commit_rec", 2024-03-11T21:49:01.293 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:46:41.629028+0000", 2024-03-11T21:49:01.293 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.043052356 2024-03-11T21:49:01.293 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:49:01.293 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:49:01.293 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "commit_sent", 2024-03-11T21:49:01.293 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:46:41.629081+0000", 2024-03-11T21:49:01.293 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 5.2743e-05 2024-03-11T21:49:01.293 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:49:01.293 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:49:01.293 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "done", 2024-03-11T21:49:01.293 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:46:41.629131+0000", 2024-03-11T21:49:01.294 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 5.0334999999999998e-05 2024-03-11T21:49:01.294 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-03-11T21:49:01.294 INFO:teuthology.orchestra.run.smithi097.stdout: ] 2024-03-11T21:49:01.294 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-03-11T21:49:01.294 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:49:01.294 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:49:01.294 INFO:teuthology.orchestra.run.smithi097.stdout: "description": "osd_op(client.35813.0:148627 11.7 11:e7907a99:::benchmark_data_smithi092_186520_object10279:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e471)", 2024-03-11T21:49:01.294 INFO:teuthology.orchestra.run.smithi097.stdout: "initiated_at": "2024-03-11T21:46:44.442693+0000", 2024-03-11T21:49:01.294 INFO:teuthology.orchestra.run.smithi097.stdout: "age": 136.75271319300001, 2024-03-11T21:49:01.294 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.039768930000000001, 2024-03-11T21:49:01.294 INFO:teuthology.orchestra.run.smithi097.stdout: "type_data": { 2024-03-11T21:49:01.294 INFO:teuthology.orchestra.run.smithi097.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-11T21:49:01.294 INFO:teuthology.orchestra.run.smithi097.stdout: "client_info": { 2024-03-11T21:49:01.295 INFO:teuthology.orchestra.run.smithi097.stdout: "client": "client.35813", 2024-03-11T21:49:01.295 INFO:teuthology.orchestra.run.smithi097.stdout: "client_addr": "v1:172.21.15.92:0/4293352030", 2024-03-11T21:49:01.295 INFO:teuthology.orchestra.run.smithi097.stdout: "tid": 148627 2024-03-11T21:49:01.295 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:49:01.295 INFO:teuthology.orchestra.run.smithi097.stdout: "events": [ 2024-03-11T21:49:01.295 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:49:01.295 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "initiated", 2024-03-11T21:49:01.295 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:46:44.442693+0000", 2024-03-11T21:49:01.295 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-03-11T21:49:01.295 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:49:01.295 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:49:01.295 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "header_read", 2024-03-11T21:49:01.295 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:46:44.442693+0000", 2024-03-11T21:49:01.295 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-03-11T21:49:01.296 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:49:01.296 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:49:01.296 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "throttled", 2024-03-11T21:49:01.296 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:46:44.442695+0000", 2024-03-11T21:49:01.296 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.885e-06 2024-03-11T21:49:01.296 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:49:01.296 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:49:01.296 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "all_read", 2024-03-11T21:49:01.296 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:46:44.442700+0000", 2024-03-11T21:49:01.296 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 5.3029999999999999e-06 2024-03-11T21:49:01.296 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:49:01.296 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:49:01.296 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "dispatched", 2024-03-11T21:49:01.296 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:46:44.442716+0000", 2024-03-11T21:49:01.297 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.5685e-05 2024-03-11T21:49:01.297 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:49:01.297 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:49:01.297 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "queued_for_pg", 2024-03-11T21:49:01.297 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:46:44.442724+0000", 2024-03-11T21:49:01.297 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 8.0350000000000001e-06 2024-03-11T21:49:01.297 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:49:01.297 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:49:01.297 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "reached_pg", 2024-03-11T21:49:01.297 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:46:44.442785+0000", 2024-03-11T21:49:01.297 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 6.1717999999999994e-05 2024-03-11T21:49:01.297 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:49:01.297 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:49:01.297 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "started", 2024-03-11T21:49:01.297 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:46:44.442896+0000", 2024-03-11T21:49:01.297 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.00011088500000000001 2024-03-11T21:49:01.298 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:49:01.298 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:49:01.298 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "waiting for subops from 0,1", 2024-03-11T21:49:01.298 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:46:44.442999+0000", 2024-03-11T21:49:01.298 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.000103139 2024-03-11T21:49:01.298 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:49:01.298 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:49:01.298 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "op_commit", 2024-03-11T21:49:01.298 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:46:44.445476+0000", 2024-03-11T21:49:01.298 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.002476083 2024-03-11T21:49:01.298 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:49:01.298 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:49:01.299 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "sub_op_commit_rec", 2024-03-11T21:49:01.299 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:46:44.445514+0000", 2024-03-11T21:49:01.299 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 3.8381999999999999e-05 2024-03-11T21:49:01.299 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:49:01.299 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:49:01.299 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "sub_op_commit_rec", 2024-03-11T21:49:01.299 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:46:44.482298+0000", 2024-03-11T21:49:01.299 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.036783918999999998 2024-03-11T21:49:01.299 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:49:01.299 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:49:01.299 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "commit_sent", 2024-03-11T21:49:01.299 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:46:44.482405+0000", 2024-03-11T21:49:01.299 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.000107032 2024-03-11T21:49:01.299 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:49:01.299 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:49:01.300 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "done", 2024-03-11T21:49:01.300 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:46:44.482462+0000", 2024-03-11T21:49:01.300 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 5.6864000000000003e-05 2024-03-11T21:49:01.300 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-03-11T21:49:01.300 INFO:teuthology.orchestra.run.smithi097.stdout: ] 2024-03-11T21:49:01.300 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-03-11T21:49:01.300 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:49:01.300 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:49:01.300 INFO:teuthology.orchestra.run.smithi097.stdout: "description": "osd_op(client.35813.0:149708 11.7 11:e5a3c731:::benchmark_data_smithi092_186520_object11360:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e471)", 2024-03-11T21:49:01.300 INFO:teuthology.orchestra.run.smithi097.stdout: "initiated_at": "2024-03-11T21:46:45.160385+0000", 2024-03-11T21:49:01.300 INFO:teuthology.orchestra.run.smithi097.stdout: "age": 136.035021073, 2024-03-11T21:49:01.300 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.043707652, 2024-03-11T21:49:01.300 INFO:teuthology.orchestra.run.smithi097.stdout: "type_data": { 2024-03-11T21:49:01.300 INFO:teuthology.orchestra.run.smithi097.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-11T21:49:01.300 INFO:teuthology.orchestra.run.smithi097.stdout: "client_info": { 2024-03-11T21:49:01.301 INFO:teuthology.orchestra.run.smithi097.stdout: "client": "client.35813", 2024-03-11T21:49:01.301 INFO:teuthology.orchestra.run.smithi097.stdout: "client_addr": "v1:172.21.15.92:0/4293352030", 2024-03-11T21:49:01.301 INFO:teuthology.orchestra.run.smithi097.stdout: "tid": 149708 2024-03-11T21:49:01.301 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:49:01.301 INFO:teuthology.orchestra.run.smithi097.stdout: "events": [ 2024-03-11T21:49:01.301 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:49:01.301 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "initiated", 2024-03-11T21:49:01.301 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:46:45.160385+0000", 2024-03-11T21:49:01.301 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-03-11T21:49:01.301 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:49:01.301 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:49:01.301 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "header_read", 2024-03-11T21:49:01.301 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:46:45.160385+0000", 2024-03-11T21:49:01.301 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-03-11T21:49:01.301 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:49:01.302 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:49:01.302 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "throttled", 2024-03-11T21:49:01.302 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:46:45.160387+0000", 2024-03-11T21:49:01.302 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 2.3690000000000001e-06 2024-03-11T21:49:01.302 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:49:01.302 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:49:01.302 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "all_read", 2024-03-11T21:49:01.302 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:46:45.160394+0000", 2024-03-11T21:49:01.302 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 6.2910000000000004e-06 2024-03-11T21:49:01.302 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:49:01.302 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:49:01.302 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "dispatched", 2024-03-11T21:49:01.302 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:46:45.160410+0000", 2024-03-11T21:49:01.302 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.6294999999999999e-05 2024-03-11T21:49:01.303 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:49:01.303 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:49:01.303 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "queued_for_pg", 2024-03-11T21:49:01.303 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:46:45.160418+0000", 2024-03-11T21:49:01.303 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 8.0520000000000002e-06 2024-03-11T21:49:01.303 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:49:01.303 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:49:01.303 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "reached_pg", 2024-03-11T21:49:01.303 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:46:45.160495+0000", 2024-03-11T21:49:01.303 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 7.6692000000000004e-05 2024-03-11T21:49:01.303 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:49:01.303 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:49:01.304 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "started", 2024-03-11T21:49:01.304 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:46:45.160626+0000", 2024-03-11T21:49:01.304 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.00013177100000000001 2024-03-11T21:49:01.304 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:49:01.304 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:49:01.304 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "waiting for subops from 0,1", 2024-03-11T21:49:01.304 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:46:45.161447+0000", 2024-03-11T21:49:01.304 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.000820139 2024-03-11T21:49:01.304 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:49:01.305 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:49:01.305 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "op_commit", 2024-03-11T21:49:01.305 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:46:45.165376+0000", 2024-03-11T21:49:01.305 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.0039295299999999997 2024-03-11T21:49:01.305 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:49:01.305 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:49:01.305 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "sub_op_commit_rec", 2024-03-11T21:49:01.305 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:46:45.165412+0000", 2024-03-11T21:49:01.305 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 3.6152999999999998e-05 2024-03-11T21:49:01.305 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:49:01.305 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:49:01.305 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "sub_op_commit_rec", 2024-03-11T21:49:01.305 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:46:45.204023+0000", 2024-03-11T21:49:01.305 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.038610658999999999 2024-03-11T21:49:01.306 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:49:01.306 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:49:01.306 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "commit_sent", 2024-03-11T21:49:01.306 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:46:45.204060+0000", 2024-03-11T21:49:01.306 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 3.7134000000000001e-05 2024-03-11T21:49:01.306 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:49:01.306 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:49:01.306 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "done", 2024-03-11T21:49:01.306 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:46:45.204093+0000", 2024-03-11T21:49:01.306 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 3.2567000000000002e-05 2024-03-11T21:49:01.306 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-03-11T21:49:01.306 INFO:teuthology.orchestra.run.smithi097.stdout: ] 2024-03-11T21:49:01.306 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-03-11T21:49:01.307 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:49:01.307 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:49:01.307 INFO:teuthology.orchestra.run.smithi097.stdout: "description": "osd_op(client.35813.0:149713 11.7 11:eefd3b26:::benchmark_data_smithi092_186520_object11365:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e471)", 2024-03-11T21:49:01.307 INFO:teuthology.orchestra.run.smithi097.stdout: "initiated_at": "2024-03-11T21:46:45.163342+0000", 2024-03-11T21:49:01.307 INFO:teuthology.orchestra.run.smithi097.stdout: "age": 136.03206438399999, 2024-03-11T21:49:01.307 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.040853245000000003, 2024-03-11T21:49:01.307 INFO:teuthology.orchestra.run.smithi097.stdout: "type_data": { 2024-03-11T21:49:01.307 INFO:teuthology.orchestra.run.smithi097.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-11T21:49:01.307 INFO:teuthology.orchestra.run.smithi097.stdout: "client_info": { 2024-03-11T21:49:01.307 INFO:teuthology.orchestra.run.smithi097.stdout: "client": "client.35813", 2024-03-11T21:49:01.307 INFO:teuthology.orchestra.run.smithi097.stdout: "client_addr": "v1:172.21.15.92:0/4293352030", 2024-03-11T21:49:01.308 INFO:teuthology.orchestra.run.smithi097.stdout: "tid": 149713 2024-03-11T21:49:01.308 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:49:01.308 INFO:teuthology.orchestra.run.smithi097.stdout: "events": [ 2024-03-11T21:49:01.308 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:49:01.308 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "initiated", 2024-03-11T21:49:01.308 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:46:45.163342+0000", 2024-03-11T21:49:01.308 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-03-11T21:49:01.308 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:49:01.308 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:49:01.308 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "header_read", 2024-03-11T21:49:01.308 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:46:45.163342+0000", 2024-03-11T21:49:01.308 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-03-11T21:49:01.308 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:49:01.308 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:49:01.308 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "throttled", 2024-03-11T21:49:01.309 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:46:45.163343+0000", 2024-03-11T21:49:01.309 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.6419999999999999e-06 2024-03-11T21:49:01.309 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:49:01.309 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:49:01.309 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "all_read", 2024-03-11T21:49:01.309 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:46:45.163350+0000", 2024-03-11T21:49:01.309 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 7.0509999999999999e-06 2024-03-11T21:49:01.309 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:49:01.309 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:49:01.309 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "dispatched", 2024-03-11T21:49:01.309 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:46:45.163360+0000", 2024-03-11T21:49:01.309 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 9.7890000000000006e-06 2024-03-11T21:49:01.309 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:49:01.309 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:49:01.310 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "queued_for_pg", 2024-03-11T21:49:01.310 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:46:45.163368+0000", 2024-03-11T21:49:01.310 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 7.5689999999999999e-06 2024-03-11T21:49:01.310 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:49:01.310 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:49:01.310 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "reached_pg", 2024-03-11T21:49:01.310 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:46:45.163601+0000", 2024-03-11T21:49:01.310 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.00023350899999999999 2024-03-11T21:49:01.310 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:49:01.310 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:49:01.310 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "started", 2024-03-11T21:49:01.310 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:46:45.163720+0000", 2024-03-11T21:49:01.310 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.00011864 2024-03-11T21:49:01.311 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:49:01.311 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:49:01.311 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "waiting for subops from 0,1", 2024-03-11T21:49:01.311 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:46:45.163865+0000", 2024-03-11T21:49:01.311 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.00014522100000000001 2024-03-11T21:49:01.311 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:49:01.311 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:49:01.311 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "op_commit", 2024-03-11T21:49:01.311 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:46:45.165472+0000", 2024-03-11T21:49:01.311 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.001606777 2024-03-11T21:49:01.311 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:49:01.311 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:49:01.311 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "sub_op_commit_rec", 2024-03-11T21:49:01.311 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:46:45.166843+0000", 2024-03-11T21:49:01.312 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.001371473 2024-03-11T21:49:01.312 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:49:01.312 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:49:01.312 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "sub_op_commit_rec", 2024-03-11T21:49:01.312 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:46:45.204135+0000", 2024-03-11T21:49:01.312 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.037291210999999998 2024-03-11T21:49:01.312 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:49:01.312 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:49:01.312 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "commit_sent", 2024-03-11T21:49:01.312 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:46:45.204163+0000", 2024-03-11T21:49:01.312 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 2.8418000000000001e-05 2024-03-11T21:49:01.312 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:49:01.313 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:49:01.313 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "done", 2024-03-11T21:49:01.313 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:46:45.204195+0000", 2024-03-11T21:49:01.313 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 3.1945e-05 2024-03-11T21:49:01.313 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-03-11T21:49:01.313 INFO:teuthology.orchestra.run.smithi097.stdout: ] 2024-03-11T21:49:01.313 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-03-11T21:49:01.313 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-03-11T21:49:01.313 INFO:teuthology.orchestra.run.smithi097.stdout: ] 2024-03-11T21:49:01.313 INFO:teuthology.orchestra.run.smithi097.stdout:} 2024-03-11T21:49:01.831 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:49:01 smithi092 ceph-mon[29572]: osdmap e519: 8 total, 8 up, 6 in 2024-03-11T21:49:01.831 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:49:01 smithi092 ceph-mon[37563]: osdmap e519: 8 total, 8 up, 6 in 2024-03-11T21:49:02.085 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:49:01 smithi097 ceph-mon[33317]: osdmap e519: 8 total, 8 up, 6 in 2024-03-11T21:49:02.254 DEBUG:teuthology.orchestra.run.smithi097:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid 00db0016-dfed-11ee-95c8-87774f69a715 -- ceph daemon osd.6 dump_ops_in_flight 2024-03-11T21:49:02.493 INFO:teuthology.orchestra.run.smithi097.stderr:Inferring config /var/lib/ceph/00db0016-dfed-11ee-95c8-87774f69a715/mon.b/config 2024-03-11T21:49:02.611 INFO:teuthology.orchestra.run.smithi092.stderr:reweighted osd.7 to 1 (10000) 2024-03-11T21:49:02.869 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:49:02 smithi092 ceph-mon[29572]: 5.c scrub starts 2024-03-11T21:49:02.869 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:49:02 smithi092 ceph-mon[29572]: 5.c scrub ok 2024-03-11T21:49:02.869 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:49:02 smithi092 ceph-mon[29572]: pgmap v1115: 132 pgs: 132 active+clean; 582 KiB data, 5.3 GiB used, 531 GiB / 536 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T21:49:02.869 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:49:02 smithi092 ceph-mon[29572]: 6.1a deep-scrub starts 2024-03-11T21:49:02.870 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:49:02 smithi092 ceph-mon[29572]: 6.1a deep-scrub ok 2024-03-11T21:49:02.870 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:49:02 smithi092 ceph-mon[29572]: from='client.? v1:172.21.15.92:0/3081767199' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]: dispatch 2024-03-11T21:49:02.870 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:49:02 smithi092 ceph-mon[37563]: 5.c scrub starts 2024-03-11T21:49:02.870 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:49:02 smithi092 ceph-mon[37563]: 5.c scrub ok 2024-03-11T21:49:02.870 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:49:02 smithi092 ceph-mon[37563]: pgmap v1115: 132 pgs: 132 active+clean; 582 KiB data, 5.3 GiB used, 531 GiB / 536 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T21:49:02.870 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:49:02 smithi092 ceph-mon[37563]: 6.1a deep-scrub starts 2024-03-11T21:49:02.870 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:49:02 smithi092 ceph-mon[37563]: 6.1a deep-scrub ok 2024-03-11T21:49:02.870 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:49:02 smithi092 ceph-mon[37563]: from='client.? v1:172.21.15.92:0/3081767199' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]: dispatch 2024-03-11T21:49:02.871 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:49:02 smithi097 ceph-mon[33317]: 5.c scrub starts 2024-03-11T21:49:02.872 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:49:02 smithi097 ceph-mon[33317]: 5.c scrub ok 2024-03-11T21:49:02.872 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:49:02 smithi097 ceph-mon[33317]: pgmap v1115: 132 pgs: 132 active+clean; 582 KiB data, 5.3 GiB used, 531 GiB / 536 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T21:49:02.872 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:49:02 smithi097 ceph-mon[33317]: 6.1a deep-scrub starts 2024-03-11T21:49:02.872 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:49:02 smithi097 ceph-mon[33317]: 6.1a deep-scrub ok 2024-03-11T21:49:02.872 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:49:02 smithi097 ceph-mon[33317]: from='client.? v1:172.21.15.92:0/3081767199' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]: dispatch 2024-03-11T21:49:03.354 DEBUG:teuthology.orchestra.run.smithi092:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid 00db0016-dfed-11ee-95c8-87774f69a715 -- ceph osd reweight 5 1 2024-03-11T21:49:03.601 INFO:teuthology.orchestra.run.smithi092.stderr:Inferring config /var/lib/ceph/00db0016-dfed-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T21:49:03.614 INFO:journalctl@ceph.iscsi.iscsi.a.smithi097.stdout:Mar 11 21:49:03 smithi097 ceph-00db0016-dfed-11ee-95c8-87774f69a715-iscsi-iscsi-a[60943]: debug there is no tcmu-runner data available 2024-03-11T21:49:03.865 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:49:03 smithi092 ceph-mon[29572]: 6.13 scrub starts 2024-03-11T21:49:03.865 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:49:03 smithi092 ceph-mon[29572]: 6.13 scrub ok 2024-03-11T21:49:03.865 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:49:03 smithi092 ceph-mon[29572]: from='client.? v1:172.21.15.92:0/3081767199' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]': finished 2024-03-11T21:49:03.865 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:49:03 smithi092 ceph-mon[29572]: osdmap e520: 8 total, 8 up, 6 in 2024-03-11T21:49:03.865 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:49:03 smithi092 ceph-mon[29572]: 5.b scrub starts 2024-03-11T21:49:03.865 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:49:03 smithi092 ceph-mon[29572]: 5.b scrub ok 2024-03-11T21:49:03.865 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:49:03 smithi092 ceph-mon[29572]: from='client.24469 v1:172.21.15.97:0/404208310' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:49:03.866 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:49:03 smithi092 ceph-mon[37563]: 6.13 scrub starts 2024-03-11T21:49:03.866 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:49:03 smithi092 ceph-mon[37563]: 6.13 scrub ok 2024-03-11T21:49:03.866 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:49:03 smithi092 ceph-mon[37563]: from='client.? v1:172.21.15.92:0/3081767199' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]': finished 2024-03-11T21:49:03.866 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:49:03 smithi092 ceph-mon[37563]: osdmap e520: 8 total, 8 up, 6 in 2024-03-11T21:49:03.866 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:49:03 smithi092 ceph-mon[37563]: 5.b scrub starts 2024-03-11T21:49:03.866 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:49:03 smithi092 ceph-mon[37563]: 5.b scrub ok 2024-03-11T21:49:03.866 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:49:03 smithi092 ceph-mon[37563]: from='client.24469 v1:172.21.15.97:0/404208310' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:49:03.968 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:49:03 smithi097 ceph-mon[33317]: 6.13 scrub starts 2024-03-11T21:49:03.968 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:49:03 smithi097 ceph-mon[33317]: 6.13 scrub ok 2024-03-11T21:49:03.968 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:49:03 smithi097 ceph-mon[33317]: from='client.? v1:172.21.15.92:0/3081767199' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]': finished 2024-03-11T21:49:03.968 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:49:03 smithi097 ceph-mon[33317]: osdmap e520: 8 total, 8 up, 6 in 2024-03-11T21:49:03.968 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:49:03 smithi097 ceph-mon[33317]: 5.b scrub starts 2024-03-11T21:49:03.968 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:49:03 smithi097 ceph-mon[33317]: 5.b scrub ok 2024-03-11T21:49:03.969 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:49:03 smithi097 ceph-mon[33317]: from='client.24469 v1:172.21.15.97:0/404208310' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:49:04.202 INFO:teuthology.orchestra.run.smithi097.stdout:{ 2024-03-11T21:49:04.202 INFO:teuthology.orchestra.run.smithi097.stdout: "ops": [], 2024-03-11T21:49:04.202 INFO:teuthology.orchestra.run.smithi097.stdout: "num_ops": 0 2024-03-11T21:49:04.202 INFO:teuthology.orchestra.run.smithi097.stdout:} 2024-03-11T21:49:04.825 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:49:04 smithi097 ceph-mon[33317]: purged_snaps scrub starts 2024-03-11T21:49:04.826 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:49:04 smithi097 ceph-mon[33317]: purged_snaps scrub ok 2024-03-11T21:49:04.826 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:49:04 smithi097 ceph-mon[33317]: pgmap v1117: 132 pgs: 132 active+clean; 582 KiB data, 5.3 GiB used, 531 GiB / 536 GiB avail 2024-03-11T21:49:04.826 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:49:04 smithi097 ceph-mon[33317]: 5.d scrub starts 2024-03-11T21:49:04.826 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:49:04 smithi097 ceph-mon[33317]: 5.d scrub ok 2024-03-11T21:49:04.826 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:49:04 smithi097 ceph-mon[33317]: osdmap e521: 8 total, 8 up, 6 in 2024-03-11T21:49:04.850 DEBUG:teuthology.orchestra.run.smithi097:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid 00db0016-dfed-11ee-95c8-87774f69a715 -- ceph daemon osd.6 dump_blocked_ops 2024-03-11T21:49:04.869 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:49:04 smithi092 ceph-mon[37563]: purged_snaps scrub starts 2024-03-11T21:49:04.869 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:49:04 smithi092 ceph-mon[37563]: purged_snaps scrub ok 2024-03-11T21:49:04.869 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:49:04 smithi092 ceph-mon[37563]: pgmap v1117: 132 pgs: 132 active+clean; 582 KiB data, 5.3 GiB used, 531 GiB / 536 GiB avail 2024-03-11T21:49:04.869 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:49:04 smithi092 ceph-mon[37563]: 5.d scrub starts 2024-03-11T21:49:04.869 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:49:04 smithi092 ceph-mon[37563]: 5.d scrub ok 2024-03-11T21:49:04.869 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:49:04 smithi092 ceph-mon[37563]: osdmap e521: 8 total, 8 up, 6 in 2024-03-11T21:49:04.870 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:49:04 smithi092 ceph-mon[29572]: purged_snaps scrub starts 2024-03-11T21:49:04.870 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:49:04 smithi092 ceph-mon[29572]: purged_snaps scrub ok 2024-03-11T21:49:04.870 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:49:04 smithi092 ceph-mon[29572]: pgmap v1117: 132 pgs: 132 active+clean; 582 KiB data, 5.3 GiB used, 531 GiB / 536 GiB avail 2024-03-11T21:49:04.870 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:49:04 smithi092 ceph-mon[29572]: 5.d scrub starts 2024-03-11T21:49:04.870 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:49:04 smithi092 ceph-mon[29572]: 5.d scrub ok 2024-03-11T21:49:04.870 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:49:04 smithi092 ceph-mon[29572]: osdmap e521: 8 total, 8 up, 6 in 2024-03-11T21:49:05.097 INFO:teuthology.orchestra.run.smithi097.stderr:Inferring config /var/lib/ceph/00db0016-dfed-11ee-95c8-87774f69a715/mon.b/config 2024-03-11T21:49:05.835 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:49:05 smithi097 ceph-mon[33317]: 5.9 scrub starts 2024-03-11T21:49:05.835 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:49:05 smithi097 ceph-mon[33317]: 5.9 scrub ok 2024-03-11T21:49:05.835 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:49:05 smithi097 ceph-mon[33317]: 6.f scrub starts 2024-03-11T21:49:05.835 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:49:05 smithi097 ceph-mon[33317]: purged_snaps scrub starts 2024-03-11T21:49:05.836 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:49:05 smithi097 ceph-mon[33317]: purged_snaps scrub ok 2024-03-11T21:49:06.120 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:49:05 smithi092 ceph-mon[29572]: 5.9 scrub starts 2024-03-11T21:49:06.120 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:49:05 smithi092 ceph-mon[29572]: 5.9 scrub ok 2024-03-11T21:49:06.120 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:49:05 smithi092 ceph-mon[29572]: 6.f scrub starts 2024-03-11T21:49:06.120 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:49:05 smithi092 ceph-mon[29572]: purged_snaps scrub starts 2024-03-11T21:49:06.120 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:49:05 smithi092 ceph-mon[29572]: purged_snaps scrub ok 2024-03-11T21:49:06.120 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:49:05 smithi092 ceph-mon[37563]: 5.9 scrub starts 2024-03-11T21:49:06.120 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:49:05 smithi092 ceph-mon[37563]: 5.9 scrub ok 2024-03-11T21:49:06.120 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:49:05 smithi092 ceph-mon[37563]: 6.f scrub starts 2024-03-11T21:49:06.120 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:49:05 smithi092 ceph-mon[37563]: purged_snaps scrub starts 2024-03-11T21:49:06.121 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:49:05 smithi092 ceph-mon[37563]: purged_snaps scrub ok 2024-03-11T21:49:06.631 INFO:teuthology.orchestra.run.smithi092.stderr:reweighted osd.5 to 1 (10000) 2024-03-11T21:49:06.817 INFO:teuthology.orchestra.run.smithi097.stdout:{ 2024-03-11T21:49:06.817 INFO:teuthology.orchestra.run.smithi097.stdout: "ops": [], 2024-03-11T21:49:06.817 INFO:teuthology.orchestra.run.smithi097.stdout: "complaint_time": 30, 2024-03-11T21:49:06.817 INFO:teuthology.orchestra.run.smithi097.stdout: "num_blocked_ops": 0 2024-03-11T21:49:06.817 INFO:teuthology.orchestra.run.smithi097.stdout:} 2024-03-11T21:49:06.817 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:49:06 smithi097 ceph-mon[33317]: 6.f scrub ok 2024-03-11T21:49:06.818 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:49:06 smithi097 ceph-mon[33317]: 4.1b scrub starts 2024-03-11T21:49:06.818 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:49:06 smithi097 ceph-mon[33317]: 4.1b scrub ok 2024-03-11T21:49:06.818 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:49:06 smithi097 ceph-mon[33317]: pgmap v1119: 132 pgs: 132 active+clean; 582 KiB data, 5.3 GiB used, 531 GiB / 536 GiB avail; 864 B/s rd, 0 op/s 2024-03-11T21:49:06.818 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:49:06 smithi097 ceph-mon[33317]: 4.d scrub starts 2024-03-11T21:49:06.818 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:49:06 smithi097 ceph-mon[33317]: 4.d scrub ok 2024-03-11T21:49:06.818 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:49:06 smithi097 ceph-mon[33317]: osdmap e522: 8 total, 8 up, 6 in 2024-03-11T21:49:06.818 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:49:06 smithi097 ceph-mon[33317]: from='client.? v1:172.21.15.92:0/1238901318' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]: dispatch 2024-03-11T21:49:06.930 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:49:06 smithi092 ceph-mon[29572]: 6.f scrub ok 2024-03-11T21:49:06.930 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:49:06 smithi092 ceph-mon[29572]: 4.1b scrub starts 2024-03-11T21:49:06.930 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:49:06 smithi092 ceph-mon[29572]: 4.1b scrub ok 2024-03-11T21:49:06.930 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:49:06 smithi092 ceph-mon[29572]: pgmap v1119: 132 pgs: 132 active+clean; 582 KiB data, 5.3 GiB used, 531 GiB / 536 GiB avail; 864 B/s rd, 0 op/s 2024-03-11T21:49:06.930 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:49:06 smithi092 ceph-mon[29572]: 4.d scrub starts 2024-03-11T21:49:06.930 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:49:06 smithi092 ceph-mon[29572]: 4.d scrub ok 2024-03-11T21:49:06.930 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:49:06 smithi092 ceph-mon[29572]: osdmap e522: 8 total, 8 up, 6 in 2024-03-11T21:49:06.931 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:49:06 smithi092 ceph-mon[29572]: from='client.? v1:172.21.15.92:0/1238901318' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]: dispatch 2024-03-11T21:49:06.931 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:49:06 smithi092 ceph-mon[37563]: 6.f scrub ok 2024-03-11T21:49:06.931 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:49:06 smithi092 ceph-mon[37563]: 4.1b scrub starts 2024-03-11T21:49:06.931 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:49:06 smithi092 ceph-mon[37563]: 4.1b scrub ok 2024-03-11T21:49:06.931 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:49:06 smithi092 ceph-mon[37563]: pgmap v1119: 132 pgs: 132 active+clean; 582 KiB data, 5.3 GiB used, 531 GiB / 536 GiB avail; 864 B/s rd, 0 op/s 2024-03-11T21:49:06.931 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:49:06 smithi092 ceph-mon[37563]: 4.d scrub starts 2024-03-11T21:49:06.931 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:49:06 smithi092 ceph-mon[37563]: 4.d scrub ok 2024-03-11T21:49:06.931 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:49:06 smithi092 ceph-mon[37563]: osdmap e522: 8 total, 8 up, 6 in 2024-03-11T21:49:06.931 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:49:06 smithi092 ceph-mon[37563]: from='client.? v1:172.21.15.92:0/1238901318' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]: dispatch 2024-03-11T21:49:07.254 INFO:tasks.cephadm.ceph_manager.ceph:waiting for recovery to complete 2024-03-11T21:49:07.255 DEBUG:teuthology.orchestra.run.smithi092:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid 00db0016-dfed-11ee-95c8-87774f69a715 -- ceph pg dump --format=json 2024-03-11T21:49:07.417 DEBUG:teuthology.orchestra.run.smithi097:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid 00db0016-dfed-11ee-95c8-87774f69a715 -- ceph daemon osd.6 dump_historic_ops 2024-03-11T21:49:07.499 INFO:teuthology.orchestra.run.smithi092.stderr:Inferring config /var/lib/ceph/00db0016-dfed-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T21:49:07.658 INFO:teuthology.orchestra.run.smithi097.stderr:Inferring config /var/lib/ceph/00db0016-dfed-11ee-95c8-87774f69a715/mon.b/config 2024-03-11T21:49:07.780 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:49:07 smithi097 ceph-mon[33317]: 2.1b scrub starts 2024-03-11T21:49:07.780 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:49:07 smithi097 ceph-mon[33317]: 2.1b scrub ok 2024-03-11T21:49:07.780 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:49:07 smithi097 ceph-mon[33317]: from='client.? v1:172.21.15.92:0/1238901318' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]': finished 2024-03-11T21:49:07.780 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:49:07 smithi097 ceph-mon[33317]: osdmap e523: 8 total, 8 up, 6 in 2024-03-11T21:49:07.901 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:49:07 smithi092 ceph-mon[29572]: 2.1b scrub starts 2024-03-11T21:49:07.901 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:49:07 smithi092 ceph-mon[29572]: 2.1b scrub ok 2024-03-11T21:49:07.901 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:49:07 smithi092 ceph-mon[29572]: from='client.? v1:172.21.15.92:0/1238901318' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]': finished 2024-03-11T21:49:07.901 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:49:07 smithi092 ceph-mon[29572]: osdmap e523: 8 total, 8 up, 6 in 2024-03-11T21:49:07.902 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:49:07 smithi092 ceph-mon[37563]: 2.1b scrub starts 2024-03-11T21:49:07.902 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:49:07 smithi092 ceph-mon[37563]: 2.1b scrub ok 2024-03-11T21:49:07.902 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:49:07 smithi092 ceph-mon[37563]: from='client.? v1:172.21.15.92:0/1238901318' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]': finished 2024-03-11T21:49:07.902 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:49:07 smithi092 ceph-mon[37563]: osdmap e523: 8 total, 8 up, 6 in 2024-03-11T21:49:08.358 INFO:journalctl@ceph.mgr.y.smithi092.stdout:Mar 11 21:49:07 smithi092 ceph-00db0016-dfed-11ee-95c8-87774f69a715-mgr-y[29801]: ::ffff:172.21.15.97 - - [11/Mar/2024:21:49:07] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-11T21:49:08.898 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:49:08 smithi097 ceph-mon[33317]: 4.16 scrub starts 2024-03-11T21:49:08.898 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:49:08 smithi097 ceph-mon[33317]: 4.16 scrub ok 2024-03-11T21:49:08.898 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:49:08 smithi097 ceph-mon[33317]: pgmap v1122: 132 pgs: 132 active+clean; 582 KiB data, 5.3 GiB used, 531 GiB / 536 GiB avail; 1.0 KiB/s rd, 1 op/s 2024-03-11T21:49:08.898 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:49:08 smithi097 ceph-mon[33317]: from='mgr.14607 v1:172.21.15.92:0/1702717597' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T21:49:08.899 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:49:08 smithi097 ceph-mon[33317]: from='mgr.14607 v1:172.21.15.92:0/1702717597' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T21:49:08.899 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:49:08 smithi097 ceph-mon[33317]: osdmap e524: 8 total, 8 up, 6 in 2024-03-11T21:49:08.979 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:49:08 smithi092 ceph-mon[29572]: 4.16 scrub starts 2024-03-11T21:49:08.979 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:49:08 smithi092 ceph-mon[29572]: 4.16 scrub ok 2024-03-11T21:49:08.979 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:49:08 smithi092 ceph-mon[29572]: pgmap v1122: 132 pgs: 132 active+clean; 582 KiB data, 5.3 GiB used, 531 GiB / 536 GiB avail; 1.0 KiB/s rd, 1 op/s 2024-03-11T21:49:08.979 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:49:08 smithi092 ceph-mon[29572]: from='mgr.14607 v1:172.21.15.92:0/1702717597' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T21:49:08.979 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:49:08 smithi092 ceph-mon[29572]: from='mgr.14607 v1:172.21.15.92:0/1702717597' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T21:49:08.979 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:49:08 smithi092 ceph-mon[29572]: osdmap e524: 8 total, 8 up, 6 in 2024-03-11T21:49:08.980 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:49:08 smithi092 ceph-mon[37563]: 4.16 scrub starts 2024-03-11T21:49:08.980 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:49:08 smithi092 ceph-mon[37563]: 4.16 scrub ok 2024-03-11T21:49:08.980 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:49:08 smithi092 ceph-mon[37563]: pgmap v1122: 132 pgs: 132 active+clean; 582 KiB data, 5.3 GiB used, 531 GiB / 536 GiB avail; 1.0 KiB/s rd, 1 op/s 2024-03-11T21:49:08.980 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:49:08 smithi092 ceph-mon[37563]: from='mgr.14607 v1:172.21.15.92:0/1702717597' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T21:49:08.980 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:49:08 smithi092 ceph-mon[37563]: from='mgr.14607 v1:172.21.15.92:0/1702717597' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T21:49:08.980 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:49:08 smithi092 ceph-mon[37563]: osdmap e524: 8 total, 8 up, 6 in 2024-03-11T21:49:09.464 INFO:teuthology.orchestra.run.smithi092.stdout: 2024-03-11T21:49:09.467 INFO:teuthology.orchestra.run.smithi092.stderr:dumped all 2024-03-11T21:49:09.486 INFO:teuthology.orchestra.run.smithi097.stdout:{ 2024-03-11T21:49:09.486 INFO:teuthology.orchestra.run.smithi097.stdout: "size": 20, 2024-03-11T21:49:09.486 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 600, 2024-03-11T21:49:09.486 INFO:teuthology.orchestra.run.smithi097.stdout: "ops": [ 2024-03-11T21:49:09.486 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:49:09.487 INFO:teuthology.orchestra.run.smithi097.stdout: "description": "osd_op(client.15744.0:69291 10.17 10:eb3f93ab:::benchmark_data_smithi092_161462_object69290:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e372)", 2024-03-11T21:49:09.487 INFO:teuthology.orchestra.run.smithi097.stdout: "initiated_at": "2024-03-11T21:41:27.826508+0000", 2024-03-11T21:49:09.487 INFO:teuthology.orchestra.run.smithi097.stdout: "age": 461.666795982, 2024-03-11T21:49:09.487 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.022082605, 2024-03-11T21:49:09.487 INFO:teuthology.orchestra.run.smithi097.stdout: "type_data": { 2024-03-11T21:49:09.487 INFO:teuthology.orchestra.run.smithi097.stdout: "flag_point": "reached pg", 2024-03-11T21:49:09.487 INFO:teuthology.orchestra.run.smithi097.stdout: "client_info": { 2024-03-11T21:49:09.487 INFO:teuthology.orchestra.run.smithi097.stdout: "client": "client.15744", 2024-03-11T21:49:09.487 INFO:teuthology.orchestra.run.smithi097.stdout: "client_addr": "v1:172.21.15.92:0/3334000812", 2024-03-11T21:49:09.487 INFO:teuthology.orchestra.run.smithi097.stdout: "tid": 69291 2024-03-11T21:49:09.487 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:49:09.487 INFO:teuthology.orchestra.run.smithi097.stdout: "events": [ 2024-03-11T21:49:09.487 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:49:09.487 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "initiated", 2024-03-11T21:49:09.488 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:41:27.826508+0000", 2024-03-11T21:49:09.488 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-03-11T21:49:09.488 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:49:09.488 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:49:09.488 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "header_read", 2024-03-11T21:49:09.488 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:41:27.826508+0000", 2024-03-11T21:49:09.488 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-03-11T21:49:09.488 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:49:09.488 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:49:09.488 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "throttled", 2024-03-11T21:49:09.488 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:41:27.826510+0000", 2024-03-11T21:49:09.488 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 2.1799999999999999e-06 2024-03-11T21:49:09.488 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:49:09.488 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:49:09.489 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "all_read", 2024-03-11T21:49:09.489 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:41:27.826614+0000", 2024-03-11T21:49:09.489 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.00010393199999999999 2024-03-11T21:49:09.489 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:49:09.489 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:49:09.489 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "dispatched", 2024-03-11T21:49:09.489 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:41:27.826625+0000", 2024-03-11T21:49:09.489 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.0386e-05 2024-03-11T21:49:09.489 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:49:09.489 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:49:09.489 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "queued_for_pg", 2024-03-11T21:49:09.489 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:41:27.826633+0000", 2024-03-11T21:49:09.489 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 8.1280000000000008e-06 2024-03-11T21:49:09.489 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:49:09.490 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:49:09.490 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "reached_pg", 2024-03-11T21:49:09.490 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:41:27.952571+0000", 2024-03-11T21:49:09.490 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.125938363 2024-03-11T21:49:09.490 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:49:09.490 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:49:09.490 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "waiting for peered", 2024-03-11T21:49:09.490 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:41:27.952582+0000", 2024-03-11T21:49:09.490 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.0701e-05 2024-03-11T21:49:09.490 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:49:09.490 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:49:09.490 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "reached_pg", 2024-03-11T21:49:09.490 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:41:28.848505+0000", 2024-03-11T21:49:09.491 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.89592317700000002 2024-03-11T21:49:09.491 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:49:09.491 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:49:09.491 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "done", 2024-03-11T21:49:09.491 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:41:28.848591+0000", 2024-03-11T21:49:09.491 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 8.5737999999999998e-05 2024-03-11T21:49:09.491 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-03-11T21:49:09.491 INFO:teuthology.orchestra.run.smithi097.stdout: ] 2024-03-11T21:49:09.491 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-03-11T21:49:09.491 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:49:09.491 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:49:09.491 INFO:teuthology.orchestra.run.smithi097.stdout: "description": "osd_op(client.15744.0:70476 10.12 10.8a949fd2 (undecoded) ondisk+retry+write+known_if_redirected+supports_pool_eio e374)", 2024-03-11T21:49:09.491 INFO:teuthology.orchestra.run.smithi097.stdout: "initiated_at": "2024-03-11T21:41:29.844672+0000", 2024-03-11T21:49:09.491 INFO:teuthology.orchestra.run.smithi097.stdout: "age": 459.64863177900003, 2024-03-11T21:49:09.492 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.013244195, 2024-03-11T21:49:09.492 INFO:teuthology.orchestra.run.smithi097.stdout: "type_data": { 2024-03-11T21:49:09.492 INFO:teuthology.orchestra.run.smithi097.stdout: "flag_point": "reached pg", 2024-03-11T21:49:09.492 INFO:teuthology.orchestra.run.smithi097.stdout: "client_info": { 2024-03-11T21:49:09.492 INFO:teuthology.orchestra.run.smithi097.stdout: "client": "client.15744", 2024-03-11T21:49:09.492 INFO:teuthology.orchestra.run.smithi097.stdout: "client_addr": "v1:172.21.15.92:0/3334000812", 2024-03-11T21:49:09.492 INFO:teuthology.orchestra.run.smithi097.stdout: "tid": 70476 2024-03-11T21:49:09.492 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:49:09.492 INFO:teuthology.orchestra.run.smithi097.stdout: "events": [ 2024-03-11T21:49:09.492 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:49:09.492 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "initiated", 2024-03-11T21:49:09.492 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:41:29.844672+0000", 2024-03-11T21:49:09.492 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-03-11T21:49:09.492 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:49:09.492 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:49:09.493 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "header_read", 2024-03-11T21:49:09.493 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:41:29.844672+0000", 2024-03-11T21:49:09.493 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-03-11T21:49:09.493 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:49:09.493 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:49:09.493 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "throttled", 2024-03-11T21:49:09.493 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:41:29.844674+0000", 2024-03-11T21:49:09.493 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 2.0590000000000001e-06 2024-03-11T21:49:09.493 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:49:09.493 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:49:09.493 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "all_read", 2024-03-11T21:49:09.493 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:41:29.844707+0000", 2024-03-11T21:49:09.493 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 3.2085000000000002e-05 2024-03-11T21:49:09.493 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:49:09.494 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:49:09.494 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "dispatched", 2024-03-11T21:49:09.494 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:41:29.844723+0000", 2024-03-11T21:49:09.494 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.6390999999999999e-05 2024-03-11T21:49:09.494 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:49:09.494 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:49:09.494 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "queued_for_pg", 2024-03-11T21:49:09.494 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:41:29.844732+0000", 2024-03-11T21:49:09.494 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 8.7630000000000006e-06 2024-03-11T21:49:09.494 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:49:09.494 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:49:09.494 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "reached_pg", 2024-03-11T21:49:09.494 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:41:29.844792+0000", 2024-03-11T21:49:09.494 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 6.0325000000000003e-05 2024-03-11T21:49:09.495 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:49:09.495 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:49:09.495 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "waiting for peered", 2024-03-11T21:49:09.495 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:41:29.844800+0000", 2024-03-11T21:49:09.495 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 8.2190000000000004e-06 2024-03-11T21:49:09.495 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:49:09.495 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:49:09.495 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "reached_pg", 2024-03-11T21:49:09.495 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:41:30.857903+0000", 2024-03-11T21:49:09.495 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.0131028959999999 2024-03-11T21:49:09.495 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:49:09.495 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:49:09.495 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "done", 2024-03-11T21:49:09.495 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:41:30.857917+0000", 2024-03-11T21:49:09.496 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.3457e-05 2024-03-11T21:49:09.496 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-03-11T21:49:09.496 INFO:teuthology.orchestra.run.smithi097.stdout: ] 2024-03-11T21:49:09.496 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-03-11T21:49:09.496 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:49:09.496 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:49:09.496 INFO:teuthology.orchestra.run.smithi097.stdout: "description": "osd_op(client.15744.0:70481 10.12 10.4b903cf2 (undecoded) ondisk+retry+write+known_if_redirected+supports_pool_eio e374)", 2024-03-11T21:49:09.496 INFO:teuthology.orchestra.run.smithi097.stdout: "initiated_at": "2024-03-11T21:41:29.844814+0000", 2024-03-11T21:49:09.496 INFO:teuthology.orchestra.run.smithi097.stdout: "age": 459.648490471, 2024-03-11T21:49:09.496 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.013178921, 2024-03-11T21:49:09.496 INFO:teuthology.orchestra.run.smithi097.stdout: "type_data": { 2024-03-11T21:49:09.496 INFO:teuthology.orchestra.run.smithi097.stdout: "flag_point": "reached pg", 2024-03-11T21:49:09.496 INFO:teuthology.orchestra.run.smithi097.stdout: "client_info": { 2024-03-11T21:49:09.496 INFO:teuthology.orchestra.run.smithi097.stdout: "client": "client.15744", 2024-03-11T21:49:09.496 INFO:teuthology.orchestra.run.smithi097.stdout: "client_addr": "v1:172.21.15.92:0/3334000812", 2024-03-11T21:49:09.497 INFO:teuthology.orchestra.run.smithi097.stdout: "tid": 70481 2024-03-11T21:49:09.497 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:49:09.497 INFO:teuthology.orchestra.run.smithi097.stdout: "events": [ 2024-03-11T21:49:09.497 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:49:09.497 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "initiated", 2024-03-11T21:49:09.497 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:41:29.844814+0000", 2024-03-11T21:49:09.497 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-03-11T21:49:09.497 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:49:09.497 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:49:09.497 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "header_read", 2024-03-11T21:49:09.497 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:41:29.844814+0000", 2024-03-11T21:49:09.497 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-03-11T21:49:09.497 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:49:09.498 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:49:09.498 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "throttled", 2024-03-11T21:49:09.498 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:41:29.844815+0000", 2024-03-11T21:49:09.498 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 9.3600000000000002e-07 2024-03-11T21:49:09.498 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:49:09.498 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:49:09.498 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "all_read", 2024-03-11T21:49:09.498 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:41:29.844844+0000", 2024-03-11T21:49:09.498 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 2.9799999999999999e-05 2024-03-11T21:49:09.498 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:49:09.498 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:49:09.498 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "dispatched", 2024-03-11T21:49:09.498 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:41:29.844852+0000", 2024-03-11T21:49:09.499 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 7.8590000000000005e-06 2024-03-11T21:49:09.499 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:49:09.499 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:49:09.499 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "queued_for_pg", 2024-03-11T21:49:09.499 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:41:29.844859+0000", 2024-03-11T21:49:09.499 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 6.6540000000000002e-06 2024-03-11T21:49:09.499 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:49:09.499 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:49:09.499 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "reached_pg", 2024-03-11T21:49:09.499 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:41:29.844888+0000", 2024-03-11T21:49:09.499 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 2.8994999999999999e-05 2024-03-11T21:49:09.499 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:49:09.499 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:49:09.499 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "waiting for peered", 2024-03-11T21:49:09.500 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:41:29.844892+0000", 2024-03-11T21:49:09.500 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 3.6239999999999999e-06 2024-03-11T21:49:09.500 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:49:09.500 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:49:09.500 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "reached_pg", 2024-03-11T21:49:09.500 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:41:30.857985+0000", 2024-03-11T21:49:09.500 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.013093762 2024-03-11T21:49:09.500 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:49:09.501 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:49:09.501 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "done", 2024-03-11T21:49:09.501 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:41:30.857993+0000", 2024-03-11T21:49:09.501 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 7.2910000000000002e-06 2024-03-11T21:49:09.501 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-03-11T21:49:09.501 INFO:teuthology.orchestra.run.smithi097.stdout: ] 2024-03-11T21:49:09.501 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-03-11T21:49:09.501 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:49:09.501 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:49:09.501 INFO:teuthology.orchestra.run.smithi097.stdout: "description": "osd_op(client.15744.0:70510 10.12 10.8c0d3692 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e374)", 2024-03-11T21:49:09.501 INFO:teuthology.orchestra.run.smithi097.stdout: "initiated_at": "2024-03-11T21:41:29.870022+0000", 2024-03-11T21:49:09.502 INFO:teuthology.orchestra.run.smithi097.stdout: "age": 459.62328171299998, 2024-03-11T21:49:09.502 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.98800167299999997, 2024-03-11T21:49:09.502 INFO:teuthology.orchestra.run.smithi097.stdout: "type_data": { 2024-03-11T21:49:09.502 INFO:teuthology.orchestra.run.smithi097.stdout: "flag_point": "reached pg", 2024-03-11T21:49:09.502 INFO:teuthology.orchestra.run.smithi097.stdout: "client_info": { 2024-03-11T21:49:09.502 INFO:teuthology.orchestra.run.smithi097.stdout: "client": "client.15744", 2024-03-11T21:49:09.502 INFO:teuthology.orchestra.run.smithi097.stdout: "client_addr": "v1:172.21.15.92:0/3334000812", 2024-03-11T21:49:09.502 INFO:teuthology.orchestra.run.smithi097.stdout: "tid": 70510 2024-03-11T21:49:09.502 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:49:09.502 INFO:teuthology.orchestra.run.smithi097.stdout: "events": [ 2024-03-11T21:49:09.502 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:49:09.502 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "initiated", 2024-03-11T21:49:09.502 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:41:29.870022+0000", 2024-03-11T21:49:09.502 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-03-11T21:49:09.503 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:49:09.503 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:49:09.503 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "header_read", 2024-03-11T21:49:09.503 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:41:29.870022+0000", 2024-03-11T21:49:09.503 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-03-11T21:49:09.503 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:49:09.503 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:49:09.503 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "throttled", 2024-03-11T21:49:09.503 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:41:29.870024+0000", 2024-03-11T21:49:09.503 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.1400000000000001e-06 2024-03-11T21:49:09.503 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:49:09.503 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:49:09.503 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "all_read", 2024-03-11T21:49:09.503 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:41:29.870084+0000", 2024-03-11T21:49:09.503 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 6.0297999999999998e-05 2024-03-11T21:49:09.504 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:49:09.504 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:49:09.504 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "dispatched", 2024-03-11T21:49:09.504 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:41:29.870093+0000", 2024-03-11T21:49:09.504 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 8.7579999999999998e-06 2024-03-11T21:49:09.504 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:49:09.504 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:49:09.504 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "queued_for_pg", 2024-03-11T21:49:09.504 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:41:29.870100+0000", 2024-03-11T21:49:09.504 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 7.0570000000000004e-06 2024-03-11T21:49:09.504 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:49:09.504 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:49:09.504 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "reached_pg", 2024-03-11T21:49:09.504 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:41:29.870165+0000", 2024-03-11T21:49:09.505 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 6.5172000000000006e-05 2024-03-11T21:49:09.505 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:49:09.505 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:49:09.505 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "waiting for peered", 2024-03-11T21:49:09.505 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:41:29.870172+0000", 2024-03-11T21:49:09.505 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 7.3139999999999998e-06 2024-03-11T21:49:09.505 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:49:09.505 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:49:09.505 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "reached_pg", 2024-03-11T21:49:09.505 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:41:30.858016+0000", 2024-03-11T21:49:09.505 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.98784347900000002 2024-03-11T21:49:09.505 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:49:09.505 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:49:09.505 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "done", 2024-03-11T21:49:09.506 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:41:30.858024+0000", 2024-03-11T21:49:09.506 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 8.4549999999999995e-06 2024-03-11T21:49:09.506 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-03-11T21:49:09.506 INFO:teuthology.orchestra.run.smithi097.stdout: ] 2024-03-11T21:49:09.506 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-03-11T21:49:09.506 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:49:09.506 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:49:09.506 INFO:teuthology.orchestra.run.smithi097.stdout: "description": "osd_op(client.15744.0:70517 10.12 10.2dc54812 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e374)", 2024-03-11T21:49:09.506 INFO:teuthology.orchestra.run.smithi097.stdout: "initiated_at": "2024-03-11T21:41:29.882950+0000", 2024-03-11T21:49:09.506 INFO:teuthology.orchestra.run.smithi097.stdout: "age": 459.61035448799998, 2024-03-11T21:49:09.506 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.97510392800000001, 2024-03-11T21:49:09.506 INFO:teuthology.orchestra.run.smithi097.stdout: "type_data": { 2024-03-11T21:49:09.506 INFO:teuthology.orchestra.run.smithi097.stdout: "flag_point": "reached pg", 2024-03-11T21:49:09.506 INFO:teuthology.orchestra.run.smithi097.stdout: "client_info": { 2024-03-11T21:49:09.507 INFO:teuthology.orchestra.run.smithi097.stdout: "client": "client.15744", 2024-03-11T21:49:09.507 INFO:teuthology.orchestra.run.smithi097.stdout: "client_addr": "v1:172.21.15.92:0/3334000812", 2024-03-11T21:49:09.507 INFO:teuthology.orchestra.run.smithi097.stdout: "tid": 70517 2024-03-11T21:49:09.507 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:49:09.507 INFO:teuthology.orchestra.run.smithi097.stdout: "events": [ 2024-03-11T21:49:09.507 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:49:09.507 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "initiated", 2024-03-11T21:49:09.507 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:41:29.882950+0000", 2024-03-11T21:49:09.507 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-03-11T21:49:09.507 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:49:09.507 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:49:09.507 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "header_read", 2024-03-11T21:49:09.507 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:41:29.882950+0000", 2024-03-11T21:49:09.508 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-03-11T21:49:09.508 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:49:09.508 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:49:09.508 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "throttled", 2024-03-11T21:49:09.508 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:41:29.882956+0000", 2024-03-11T21:49:09.508 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 6.0569999999999998e-06 2024-03-11T21:49:09.508 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:49:09.508 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:49:09.508 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "all_read", 2024-03-11T21:49:09.508 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:41:29.883003+0000", 2024-03-11T21:49:09.508 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 4.7611999999999997e-05 2024-03-11T21:49:09.508 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:49:09.508 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:49:09.508 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "dispatched", 2024-03-11T21:49:09.508 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:41:29.883023+0000", 2024-03-11T21:49:09.509 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.9514e-05 2024-03-11T21:49:09.509 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:49:09.509 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:49:09.509 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "queued_for_pg", 2024-03-11T21:49:09.509 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:41:29.883034+0000", 2024-03-11T21:49:09.509 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.0859000000000001e-05 2024-03-11T21:49:09.509 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:49:09.509 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:49:09.509 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "reached_pg", 2024-03-11T21:49:09.509 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:41:29.883099+0000", 2024-03-11T21:49:09.509 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 6.5210999999999996e-05 2024-03-11T21:49:09.509 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:49:09.509 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:49:09.509 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "waiting for peered", 2024-03-11T21:49:09.510 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:41:29.883119+0000", 2024-03-11T21:49:09.510 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.9930000000000001e-05 2024-03-11T21:49:09.510 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:49:09.510 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:49:09.510 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "reached_pg", 2024-03-11T21:49:09.510 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:41:30.858046+0000", 2024-03-11T21:49:09.510 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.97492744600000003 2024-03-11T21:49:09.510 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:49:09.510 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:49:09.510 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "done", 2024-03-11T21:49:09.510 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:41:30.858054+0000", 2024-03-11T21:49:09.510 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 7.2989999999999999e-06 2024-03-11T21:49:09.510 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-03-11T21:49:09.510 INFO:teuthology.orchestra.run.smithi097.stdout: ] 2024-03-11T21:49:09.510 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-03-11T21:49:09.511 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:49:09.511 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:49:09.511 INFO:teuthology.orchestra.run.smithi097.stdout: "description": "osd_op(client.15744.0:110532 10.16 10:6aef0a33:::benchmark_data_smithi092_161462_object556:head [delete] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e393)", 2024-03-11T21:49:09.511 INFO:teuthology.orchestra.run.smithi097.stdout: "initiated_at": "2024-03-11T21:42:24.508416+0000", 2024-03-11T21:49:09.511 INFO:teuthology.orchestra.run.smithi097.stdout: "age": 404.98488805400001, 2024-03-11T21:49:09.511 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.011266966, 2024-03-11T21:49:09.511 INFO:teuthology.orchestra.run.smithi097.stdout: "type_data": { 2024-03-11T21:49:09.511 INFO:teuthology.orchestra.run.smithi097.stdout: "flag_point": "reached pg", 2024-03-11T21:49:09.511 INFO:teuthology.orchestra.run.smithi097.stdout: "client_info": { 2024-03-11T21:49:09.511 INFO:teuthology.orchestra.run.smithi097.stdout: "client": "client.15744", 2024-03-11T21:49:09.511 INFO:teuthology.orchestra.run.smithi097.stdout: "client_addr": "v1:172.21.15.92:0/3334000812", 2024-03-11T21:49:09.511 INFO:teuthology.orchestra.run.smithi097.stdout: "tid": 110532 2024-03-11T21:49:09.511 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:49:09.511 INFO:teuthology.orchestra.run.smithi097.stdout: "events": [ 2024-03-11T21:49:09.512 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:49:09.512 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "initiated", 2024-03-11T21:49:09.512 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:42:24.508416+0000", 2024-03-11T21:49:09.512 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-03-11T21:49:09.512 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:49:09.512 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:49:09.512 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "header_read", 2024-03-11T21:49:09.512 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:42:24.508416+0000", 2024-03-11T21:49:09.512 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-03-11T21:49:09.512 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:49:09.512 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:49:09.512 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "throttled", 2024-03-11T21:49:09.512 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:42:24.508418+0000", 2024-03-11T21:49:09.512 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.6500000000000001e-06 2024-03-11T21:49:09.512 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:49:09.513 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:49:09.513 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "all_read", 2024-03-11T21:49:09.513 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:42:24.508423+0000", 2024-03-11T21:49:09.513 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 5.045e-06 2024-03-11T21:49:09.513 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:49:09.513 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:49:09.513 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "dispatched", 2024-03-11T21:49:09.513 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:42:24.508432+0000", 2024-03-11T21:49:09.513 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 8.8100000000000004e-06 2024-03-11T21:49:09.513 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:49:09.513 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:49:09.513 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "queued_for_pg", 2024-03-11T21:49:09.513 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:42:24.508437+0000", 2024-03-11T21:49:09.513 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 5.648e-06 2024-03-11T21:49:09.513 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:49:09.514 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:49:09.514 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "reached_pg", 2024-03-11T21:49:09.514 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:42:24.508484+0000", 2024-03-11T21:49:09.514 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 4.6956999999999998e-05 2024-03-11T21:49:09.514 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:49:09.514 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:49:09.514 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "waiting for peered", 2024-03-11T21:49:09.514 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:42:24.508493+0000", 2024-03-11T21:49:09.514 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 8.7819999999999999e-06 2024-03-11T21:49:09.514 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:49:09.514 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:49:09.514 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "reached_pg", 2024-03-11T21:49:09.514 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:42:25.519624+0000", 2024-03-11T21:49:09.515 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.0111309209999999 2024-03-11T21:49:09.515 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:49:09.515 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:49:09.515 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "done", 2024-03-11T21:49:09.515 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:42:25.519683+0000", 2024-03-11T21:49:09.515 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 5.9153000000000001e-05 2024-03-11T21:49:09.515 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-03-11T21:49:09.515 INFO:teuthology.orchestra.run.smithi097.stdout: ] 2024-03-11T21:49:09.515 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-03-11T21:49:09.515 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:49:09.515 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:49:09.515 INFO:teuthology.orchestra.run.smithi097.stdout: "description": "osd_op(client.15744.0:110564 10.16 10:6dc90f41:::benchmark_data_smithi092_161462_object588:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e393)", 2024-03-11T21:49:09.515 INFO:teuthology.orchestra.run.smithi097.stdout: "initiated_at": "2024-03-11T21:42:24.555481+0000", 2024-03-11T21:49:09.515 INFO:teuthology.orchestra.run.smithi097.stdout: "age": 404.93782307999999, 2024-03-11T21:49:09.515 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.97476978800000003, 2024-03-11T21:49:09.516 INFO:teuthology.orchestra.run.smithi097.stdout: "type_data": { 2024-03-11T21:49:09.516 INFO:teuthology.orchestra.run.smithi097.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-11T21:49:09.516 INFO:teuthology.orchestra.run.smithi097.stdout: "client_info": { 2024-03-11T21:49:09.516 INFO:teuthology.orchestra.run.smithi097.stdout: "client": "client.15744", 2024-03-11T21:49:09.516 INFO:teuthology.orchestra.run.smithi097.stdout: "client_addr": "v1:172.21.15.92:0/3334000812", 2024-03-11T21:49:09.516 INFO:teuthology.orchestra.run.smithi097.stdout: "tid": 110564 2024-03-11T21:49:09.516 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:49:09.516 INFO:teuthology.orchestra.run.smithi097.stdout: "events": [ 2024-03-11T21:49:09.516 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:49:09.516 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "initiated", 2024-03-11T21:49:09.516 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:42:24.555481+0000", 2024-03-11T21:49:09.516 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-03-11T21:49:09.516 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:49:09.516 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:49:09.516 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "header_read", 2024-03-11T21:49:09.517 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:42:24.555481+0000", 2024-03-11T21:49:09.517 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-03-11T21:49:09.517 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:49:09.517 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:49:09.517 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "throttled", 2024-03-11T21:49:09.517 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:42:24.555483+0000", 2024-03-11T21:49:09.517 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 2.4360000000000001e-06 2024-03-11T21:49:09.517 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:49:09.517 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:49:09.517 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "all_read", 2024-03-11T21:49:09.517 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:42:24.555491+0000", 2024-03-11T21:49:09.517 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 7.0720000000000002e-06 2024-03-11T21:49:09.517 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:49:09.517 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:49:09.517 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "dispatched", 2024-03-11T21:49:09.518 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:42:24.555504+0000", 2024-03-11T21:49:09.518 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.3489000000000001e-05 2024-03-11T21:49:09.518 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:49:09.518 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:49:09.518 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "queued_for_pg", 2024-03-11T21:49:09.518 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:42:24.555512+0000", 2024-03-11T21:49:09.518 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 7.8949999999999998e-06 2024-03-11T21:49:09.518 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:49:09.518 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:49:09.518 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "reached_pg", 2024-03-11T21:49:09.518 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:42:24.555582+0000", 2024-03-11T21:49:09.518 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 6.9824000000000001e-05 2024-03-11T21:49:09.518 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:49:09.519 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:49:09.519 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "waiting for peered", 2024-03-11T21:49:09.519 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:42:24.555594+0000", 2024-03-11T21:49:09.519 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.2238e-05 2024-03-11T21:49:09.519 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:49:09.519 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:49:09.519 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "reached_pg", 2024-03-11T21:49:09.519 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:42:25.520028+0000", 2024-03-11T21:49:09.519 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.964434232 2024-03-11T21:49:09.519 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:49:09.519 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:49:09.519 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "started", 2024-03-11T21:49:09.519 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:42:25.520131+0000", 2024-03-11T21:49:09.519 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.00010289399999999999 2024-03-11T21:49:09.519 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:49:09.520 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:49:09.520 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "waiting for subops from 1,5", 2024-03-11T21:49:09.520 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:42:25.520219+0000", 2024-03-11T21:49:09.520 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 8.8162000000000001e-05 2024-03-11T21:49:09.520 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:49:09.520 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:49:09.520 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "sub_op_commit_rec", 2024-03-11T21:49:09.520 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:42:25.525201+0000", 2024-03-11T21:49:09.520 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.0049820029999999996 2024-03-11T21:49:09.520 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:49:09.520 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:49:09.520 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "op_commit", 2024-03-11T21:49:09.520 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:42:25.525211+0000", 2024-03-11T21:49:09.520 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 9.2779999999999999e-06 2024-03-11T21:49:09.520 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:49:09.521 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:49:09.521 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "sub_op_commit_rec", 2024-03-11T21:49:09.521 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:42:25.530153+0000", 2024-03-11T21:49:09.521 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.0049424559999999996 2024-03-11T21:49:09.521 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:49:09.521 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:49:09.521 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "commit_sent", 2024-03-11T21:49:09.521 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:42:25.530206+0000", 2024-03-11T21:49:09.521 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 5.3285000000000002e-05 2024-03-11T21:49:09.521 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:49:09.521 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:49:09.521 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "done", 2024-03-11T21:49:09.521 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:42:25.530251+0000", 2024-03-11T21:49:09.521 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 4.4524000000000003e-05 2024-03-11T21:49:09.522 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-03-11T21:49:09.522 INFO:teuthology.orchestra.run.smithi097.stdout: ] 2024-03-11T21:49:09.522 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-03-11T21:49:09.522 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:49:09.522 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:49:09.522 INFO:teuthology.orchestra.run.smithi097.stdout: "description": "osd_op(client.15744.0:214899 10.17 10:ed6d6cd1:::benchmark_data_smithi092_161462_object104923:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e431)", 2024-03-11T21:49:09.522 INFO:teuthology.orchestra.run.smithi097.stdout: "initiated_at": "2024-03-11T21:44:20.436254+0000", 2024-03-11T21:49:09.522 INFO:teuthology.orchestra.run.smithi097.stdout: "age": 289.057049957, 2024-03-11T21:49:09.522 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 2.175189123, 2024-03-11T21:49:09.522 INFO:teuthology.orchestra.run.smithi097.stdout: "type_data": { 2024-03-11T21:49:09.522 INFO:teuthology.orchestra.run.smithi097.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-11T21:49:09.522 INFO:teuthology.orchestra.run.smithi097.stdout: "client_info": { 2024-03-11T21:49:09.522 INFO:teuthology.orchestra.run.smithi097.stdout: "client": "client.15744", 2024-03-11T21:49:09.522 INFO:teuthology.orchestra.run.smithi097.stdout: "client_addr": "v1:172.21.15.92:0/3334000812", 2024-03-11T21:49:09.522 INFO:teuthology.orchestra.run.smithi097.stdout: "tid": 214899 2024-03-11T21:49:09.523 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:49:09.523 INFO:teuthology.orchestra.run.smithi097.stdout: "events": [ 2024-03-11T21:49:09.523 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:49:09.523 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "initiated", 2024-03-11T21:49:09.523 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:44:20.436254+0000", 2024-03-11T21:49:09.523 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-03-11T21:49:09.523 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:49:09.523 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:49:09.523 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "header_read", 2024-03-11T21:49:09.523 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:44:20.436254+0000", 2024-03-11T21:49:09.523 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-03-11T21:49:09.523 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:49:09.523 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:49:09.523 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "throttled", 2024-03-11T21:49:09.524 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:44:20.436256+0000", 2024-03-11T21:49:09.524 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.8199999999999999e-06 2024-03-11T21:49:09.524 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:49:09.524 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:49:09.524 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "all_read", 2024-03-11T21:49:09.524 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:44:20.436261+0000", 2024-03-11T21:49:09.524 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 5.0699999999999997e-06 2024-03-11T21:49:09.524 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:49:09.524 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:49:09.524 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "dispatched", 2024-03-11T21:49:09.524 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:44:20.436271+0000", 2024-03-11T21:49:09.524 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.0003000000000001e-05 2024-03-11T21:49:09.524 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:49:09.524 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:49:09.525 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "queued_for_pg", 2024-03-11T21:49:09.525 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:44:20.436279+0000", 2024-03-11T21:49:09.525 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 7.7279999999999995e-06 2024-03-11T21:49:09.525 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:49:09.525 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:49:09.525 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "reached_pg", 2024-03-11T21:49:09.525 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:44:20.438162+0000", 2024-03-11T21:49:09.525 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.0018829999999999999 2024-03-11T21:49:09.525 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:49:09.525 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:49:09.525 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "waiting for peered", 2024-03-11T21:49:09.525 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:44:20.438174+0000", 2024-03-11T21:49:09.525 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.2132e-05 2024-03-11T21:49:09.525 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:49:09.525 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:49:09.525 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "reached_pg", 2024-03-11T21:49:09.526 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:44:21.432513+0000", 2024-03-11T21:49:09.526 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.99433888599999998 2024-03-11T21:49:09.526 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:49:09.526 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:49:09.526 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "waiting for peered", 2024-03-11T21:49:09.526 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:44:21.432523+0000", 2024-03-11T21:49:09.526 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 9.8949999999999993e-06 2024-03-11T21:49:09.526 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:49:09.526 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:49:09.526 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "reached_pg", 2024-03-11T21:49:09.526 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:44:22.595683+0000", 2024-03-11T21:49:09.526 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.163159931 2024-03-11T21:49:09.526 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:49:09.527 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:49:09.527 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "started", 2024-03-11T21:49:09.527 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:44:22.595829+0000", 2024-03-11T21:49:09.527 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.00014639899999999999 2024-03-11T21:49:09.527 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:49:09.527 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:49:09.527 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "waiting for subops from 2,3,5", 2024-03-11T21:49:09.527 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:44:22.596265+0000", 2024-03-11T21:49:09.527 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.00043560300000000001 2024-03-11T21:49:09.527 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:49:09.527 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:49:09.527 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "sub_op_commit_rec", 2024-03-11T21:49:09.527 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:44:22.605366+0000", 2024-03-11T21:49:09.527 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.0091015169999999999 2024-03-11T21:49:09.527 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:49:09.528 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:49:09.528 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "op_commit", 2024-03-11T21:49:09.528 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:44:22.605558+0000", 2024-03-11T21:49:09.528 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.00019154900000000001 2024-03-11T21:49:09.528 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:49:09.528 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:49:09.528 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "sub_op_commit_rec", 2024-03-11T21:49:09.528 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:44:22.611191+0000", 2024-03-11T21:49:09.528 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.0056330740000000001 2024-03-11T21:49:09.528 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:49:09.528 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:49:09.528 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "sub_op_commit_rec", 2024-03-11T21:49:09.528 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:44:22.611251+0000", 2024-03-11T21:49:09.528 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 5.9935000000000001e-05 2024-03-11T21:49:09.529 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:49:09.529 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:49:09.529 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "commit_sent", 2024-03-11T21:49:09.529 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:44:22.611309+0000", 2024-03-11T21:49:09.529 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 5.8301000000000003e-05 2024-03-11T21:49:09.529 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:49:09.529 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:49:09.529 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "done", 2024-03-11T21:49:09.529 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:44:22.611443+0000", 2024-03-11T21:49:09.529 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.00013428 2024-03-11T21:49:09.529 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-03-11T21:49:09.529 INFO:teuthology.orchestra.run.smithi097.stdout: ] 2024-03-11T21:49:09.529 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-03-11T21:49:09.530 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:49:09.530 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:49:09.530 INFO:teuthology.orchestra.run.smithi097.stdout: "description": "osd_op(client.15744.0:214911 10.17 10:ee14355d:::benchmark_data_smithi092_161462_object104935:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e432)", 2024-03-11T21:49:09.530 INFO:teuthology.orchestra.run.smithi097.stdout: "initiated_at": "2024-03-11T21:44:20.472141+0000", 2024-03-11T21:49:09.530 INFO:teuthology.orchestra.run.smithi097.stdout: "age": 289.02116267299999, 2024-03-11T21:49:09.530 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 2.1396579419999999, 2024-03-11T21:49:09.530 INFO:teuthology.orchestra.run.smithi097.stdout: "type_data": { 2024-03-11T21:49:09.530 INFO:teuthology.orchestra.run.smithi097.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-11T21:49:09.530 INFO:teuthology.orchestra.run.smithi097.stdout: "client_info": { 2024-03-11T21:49:09.530 INFO:teuthology.orchestra.run.smithi097.stdout: "client": "client.15744", 2024-03-11T21:49:09.530 INFO:teuthology.orchestra.run.smithi097.stdout: "client_addr": "v1:172.21.15.92:0/3334000812", 2024-03-11T21:49:09.530 INFO:teuthology.orchestra.run.smithi097.stdout: "tid": 214911 2024-03-11T21:49:09.530 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:49:09.530 INFO:teuthology.orchestra.run.smithi097.stdout: "events": [ 2024-03-11T21:49:09.531 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:49:09.531 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "initiated", 2024-03-11T21:49:09.531 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:44:20.472141+0000", 2024-03-11T21:49:09.531 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-03-11T21:49:09.531 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:49:09.531 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:49:09.531 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "header_read", 2024-03-11T21:49:09.531 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:44:20.472141+0000", 2024-03-11T21:49:09.531 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-03-11T21:49:09.531 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:49:09.531 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:49:09.531 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "throttled", 2024-03-11T21:49:09.531 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:44:20.472143+0000", 2024-03-11T21:49:09.531 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.8220000000000001e-06 2024-03-11T21:49:09.531 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:49:09.532 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:49:09.532 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "all_read", 2024-03-11T21:49:09.532 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:44:20.472147+0000", 2024-03-11T21:49:09.532 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 4.1849999999999997e-06 2024-03-11T21:49:09.532 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:49:09.532 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:49:09.532 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "dispatched", 2024-03-11T21:49:09.532 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:44:20.472159+0000", 2024-03-11T21:49:09.532 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.148e-05 2024-03-11T21:49:09.532 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:49:09.532 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:49:09.532 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "queued_for_pg", 2024-03-11T21:49:09.532 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:44:20.472165+0000", 2024-03-11T21:49:09.532 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 6.1489999999999999e-06 2024-03-11T21:49:09.532 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:49:09.533 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:49:09.533 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "reached_pg", 2024-03-11T21:49:09.533 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:44:20.472233+0000", 2024-03-11T21:49:09.533 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 6.8001000000000002e-05 2024-03-11T21:49:09.533 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:49:09.533 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:49:09.533 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "waiting for peered", 2024-03-11T21:49:09.533 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:44:20.472246+0000", 2024-03-11T21:49:09.533 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.2438e-05 2024-03-11T21:49:09.533 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:49:09.533 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:49:09.533 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "reached_pg", 2024-03-11T21:49:09.533 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:44:21.432557+0000", 2024-03-11T21:49:09.533 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.96031168600000005 2024-03-11T21:49:09.534 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:49:09.534 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:49:09.534 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "waiting for peered", 2024-03-11T21:49:09.534 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:44:21.432568+0000", 2024-03-11T21:49:09.534 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.0380999999999999e-05 2024-03-11T21:49:09.534 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:49:09.534 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:49:09.534 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "reached_pg", 2024-03-11T21:49:09.534 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:44:22.602674+0000", 2024-03-11T21:49:09.534 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.170106815 2024-03-11T21:49:09.534 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:49:09.534 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:49:09.534 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "started", 2024-03-11T21:49:09.534 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:44:22.602829+0000", 2024-03-11T21:49:09.535 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.00015442200000000001 2024-03-11T21:49:09.535 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:49:09.535 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:49:09.535 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "waiting for subops from 2,3,5", 2024-03-11T21:49:09.535 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:44:22.602982+0000", 2024-03-11T21:49:09.535 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.000153313 2024-03-11T21:49:09.535 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:49:09.535 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:49:09.535 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "sub_op_commit_rec", 2024-03-11T21:49:09.535 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:44:22.605591+0000", 2024-03-11T21:49:09.535 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.0026091450000000002 2024-03-11T21:49:09.535 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:49:09.535 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:49:09.535 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "op_commit", 2024-03-11T21:49:09.535 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:44:22.605603+0000", 2024-03-11T21:49:09.536 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.173e-05 2024-03-11T21:49:09.536 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:49:09.536 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:49:09.536 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "sub_op_commit_rec", 2024-03-11T21:49:09.536 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:44:22.611570+0000", 2024-03-11T21:49:09.536 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.0059670499999999998 2024-03-11T21:49:09.536 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:49:09.536 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:49:09.536 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "sub_op_commit_rec", 2024-03-11T21:49:09.536 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:44:22.611624+0000", 2024-03-11T21:49:09.536 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 5.3640999999999997e-05 2024-03-11T21:49:09.536 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:49:09.536 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:49:09.536 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "commit_sent", 2024-03-11T21:49:09.536 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:44:22.611678+0000", 2024-03-11T21:49:09.537 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 5.4713000000000003e-05 2024-03-11T21:49:09.537 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:49:09.537 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:49:09.537 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "done", 2024-03-11T21:49:09.537 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:44:22.611799+0000", 2024-03-11T21:49:09.537 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.000120971 2024-03-11T21:49:09.537 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-03-11T21:49:09.537 INFO:teuthology.orchestra.run.smithi097.stdout: ] 2024-03-11T21:49:09.537 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-03-11T21:49:09.537 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:49:09.537 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:49:09.537 INFO:teuthology.orchestra.run.smithi097.stdout: "description": "osd_op(client.15744.0:214914 10.17 10:e9987e1e:::benchmark_data_smithi092_161462_object104938:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e432)", 2024-03-11T21:49:09.537 INFO:teuthology.orchestra.run.smithi097.stdout: "initiated_at": "2024-03-11T21:44:20.480821+0000", 2024-03-11T21:49:09.537 INFO:teuthology.orchestra.run.smithi097.stdout: "age": 289.012483336, 2024-03-11T21:49:09.537 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 2.1312782029999999, 2024-03-11T21:49:09.538 INFO:teuthology.orchestra.run.smithi097.stdout: "type_data": { 2024-03-11T21:49:09.538 INFO:teuthology.orchestra.run.smithi097.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-11T21:49:09.538 INFO:teuthology.orchestra.run.smithi097.stdout: "client_info": { 2024-03-11T21:49:09.538 INFO:teuthology.orchestra.run.smithi097.stdout: "client": "client.15744", 2024-03-11T21:49:09.538 INFO:teuthology.orchestra.run.smithi097.stdout: "client_addr": "v1:172.21.15.92:0/3334000812", 2024-03-11T21:49:09.538 INFO:teuthology.orchestra.run.smithi097.stdout: "tid": 214914 2024-03-11T21:49:09.538 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:49:09.538 INFO:teuthology.orchestra.run.smithi097.stdout: "events": [ 2024-03-11T21:49:09.538 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:49:09.538 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "initiated", 2024-03-11T21:49:09.539 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:44:20.480821+0000", 2024-03-11T21:49:09.539 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-03-11T21:49:09.539 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:49:09.539 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:49:09.539 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "header_read", 2024-03-11T21:49:09.539 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:44:20.480821+0000", 2024-03-11T21:49:09.539 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-03-11T21:49:09.539 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:49:09.539 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:49:09.539 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "throttled", 2024-03-11T21:49:09.539 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:44:20.480823+0000", 2024-03-11T21:49:09.539 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.871e-06 2024-03-11T21:49:09.539 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:49:09.539 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:49:09.539 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "all_read", 2024-03-11T21:49:09.540 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:44:20.480828+0000", 2024-03-11T21:49:09.540 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 5.3580000000000002e-06 2024-03-11T21:49:09.540 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:49:09.540 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:49:09.540 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "dispatched", 2024-03-11T21:49:09.540 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:44:20.480840+0000", 2024-03-11T21:49:09.540 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.2170999999999999e-05 2024-03-11T21:49:09.540 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:49:09.540 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:49:09.540 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "queued_for_pg", 2024-03-11T21:49:09.540 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:44:20.480849+0000", 2024-03-11T21:49:09.540 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 8.4800000000000001e-06 2024-03-11T21:49:09.540 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:49:09.540 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:49:09.540 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "reached_pg", 2024-03-11T21:49:09.541 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:44:20.480889+0000", 2024-03-11T21:49:09.541 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 4.0513999999999999e-05 2024-03-11T21:49:09.541 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:49:09.541 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:49:09.542 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "waiting for peered", 2024-03-11T21:49:09.542 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:44:20.480897+0000", 2024-03-11T21:49:09.542 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 7.8229999999999995e-06 2024-03-11T21:49:09.542 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:49:09.542 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:49:09.542 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "reached_pg", 2024-03-11T21:49:09.542 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:44:21.432594+0000", 2024-03-11T21:49:09.542 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.95169700800000001 2024-03-11T21:49:09.542 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:49:09.542 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:49:09.543 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "waiting for peered", 2024-03-11T21:49:09.543 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:44:21.432603+0000", 2024-03-11T21:49:09.543 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 8.8070000000000006e-06 2024-03-11T21:49:09.543 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:49:09.543 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:49:09.543 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "reached_pg", 2024-03-11T21:49:09.543 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:44:22.605639+0000", 2024-03-11T21:49:09.543 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.1730364369999999 2024-03-11T21:49:09.543 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:49:09.543 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:49:09.543 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "started", 2024-03-11T21:49:09.543 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:44:22.605767+0000", 2024-03-11T21:49:09.543 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.000127391 2024-03-11T21:49:09.543 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:49:09.544 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:49:09.544 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "waiting for subops from 2,3,5", 2024-03-11T21:49:09.544 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:44:22.605931+0000", 2024-03-11T21:49:09.544 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.000164167 2024-03-11T21:49:09.544 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:49:09.544 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:49:09.544 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "op_commit", 2024-03-11T21:49:09.544 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:44:22.611458+0000", 2024-03-11T21:49:09.544 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.0055274230000000001 2024-03-11T21:49:09.544 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:49:09.544 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:49:09.544 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "sub_op_commit_rec", 2024-03-11T21:49:09.544 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:44:22.611517+0000", 2024-03-11T21:49:09.544 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 5.8532000000000001e-05 2024-03-11T21:49:09.544 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:49:09.545 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:49:09.545 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "sub_op_commit_rec", 2024-03-11T21:49:09.545 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:44:22.611879+0000", 2024-03-11T21:49:09.545 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.00036212199999999999 2024-03-11T21:49:09.545 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:49:09.545 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:49:09.545 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "sub_op_commit_rec", 2024-03-11T21:49:09.545 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:44:22.611927+0000", 2024-03-11T21:49:09.545 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 4.8219e-05 2024-03-11T21:49:09.545 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:49:09.545 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:49:09.545 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "commit_sent", 2024-03-11T21:49:09.545 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:44:22.611981+0000", 2024-03-11T21:49:09.546 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 5.3461000000000002e-05 2024-03-11T21:49:09.546 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:49:09.546 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:49:09.546 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "done", 2024-03-11T21:49:09.546 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:44:22.612099+0000", 2024-03-11T21:49:09.546 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.00011841900000000001 2024-03-11T21:49:09.546 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-03-11T21:49:09.546 INFO:teuthology.orchestra.run.smithi097.stdout: ] 2024-03-11T21:49:09.546 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-03-11T21:49:09.546 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:49:09.546 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:49:09.546 INFO:teuthology.orchestra.run.smithi097.stdout: "description": "osd_op(client.15744.0:214915 10.f 10:f1996a13:::benchmark_data_smithi092_161462_object104939:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e432)", 2024-03-11T21:49:09.547 INFO:teuthology.orchestra.run.smithi097.stdout: "initiated_at": "2024-03-11T21:44:20.480979+0000", 2024-03-11T21:49:09.547 INFO:teuthology.orchestra.run.smithi097.stdout: "age": 289.012324933, 2024-03-11T21:49:09.547 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 2.1245642980000001, 2024-03-11T21:49:09.547 INFO:teuthology.orchestra.run.smithi097.stdout: "type_data": { 2024-03-11T21:49:09.547 INFO:teuthology.orchestra.run.smithi097.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-11T21:49:09.547 INFO:teuthology.orchestra.run.smithi097.stdout: "client_info": { 2024-03-11T21:49:09.547 INFO:teuthology.orchestra.run.smithi097.stdout: "client": "client.15744", 2024-03-11T21:49:09.547 INFO:teuthology.orchestra.run.smithi097.stdout: "client_addr": "v1:172.21.15.92:0/3334000812", 2024-03-11T21:49:09.547 INFO:teuthology.orchestra.run.smithi097.stdout: "tid": 214915 2024-03-11T21:49:09.547 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:49:09.547 INFO:teuthology.orchestra.run.smithi097.stdout: "events": [ 2024-03-11T21:49:09.547 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:49:09.547 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "initiated", 2024-03-11T21:49:09.547 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:44:20.480979+0000", 2024-03-11T21:49:09.547 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-03-11T21:49:09.548 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:49:09.548 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:49:09.548 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "header_read", 2024-03-11T21:49:09.548 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:44:20.480979+0000", 2024-03-11T21:49:09.548 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-03-11T21:49:09.548 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:49:09.548 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:49:09.548 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "throttled", 2024-03-11T21:49:09.548 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:44:20.480980+0000", 2024-03-11T21:49:09.548 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.1349999999999999e-06 2024-03-11T21:49:09.548 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:49:09.548 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:49:09.548 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "all_read", 2024-03-11T21:49:09.548 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:44:20.480985+0000", 2024-03-11T21:49:09.548 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 4.2050000000000004e-06 2024-03-11T21:49:09.549 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:49:09.549 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:49:09.549 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "dispatched", 2024-03-11T21:49:09.549 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:44:20.480995+0000", 2024-03-11T21:49:09.549 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.0129000000000001e-05 2024-03-11T21:49:09.549 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:49:09.549 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:49:09.549 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "queued_for_pg", 2024-03-11T21:49:09.549 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:44:20.481003+0000", 2024-03-11T21:49:09.549 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 8.3809999999999999e-06 2024-03-11T21:49:09.549 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:49:09.549 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:49:09.549 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "reached_pg", 2024-03-11T21:49:09.549 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:44:20.481042+0000", 2024-03-11T21:49:09.549 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 3.8992000000000002e-05 2024-03-11T21:49:09.550 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:49:09.550 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:49:09.550 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "waiting for peered", 2024-03-11T21:49:09.550 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:44:20.481050+0000", 2024-03-11T21:49:09.550 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 7.4800000000000004e-06 2024-03-11T21:49:09.550 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:49:09.550 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:49:09.550 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "reached_pg", 2024-03-11T21:49:09.550 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:44:21.432468+0000", 2024-03-11T21:49:09.550 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.95141802399999997 2024-03-11T21:49:09.550 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:49:09.550 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:49:09.550 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "waiting for peered", 2024-03-11T21:49:09.550 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:44:21.432481+0000", 2024-03-11T21:49:09.551 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.393e-05 2024-03-11T21:49:09.551 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:49:09.551 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:49:09.551 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "reached_pg", 2024-03-11T21:49:09.551 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:44:22.582869+0000", 2024-03-11T21:49:09.551 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.1503876740000001 2024-03-11T21:49:09.551 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:49:09.551 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:49:09.551 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "started", 2024-03-11T21:49:09.551 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:44:22.583001+0000", 2024-03-11T21:49:09.551 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.00013150099999999999 2024-03-11T21:49:09.551 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:49:09.551 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:49:09.551 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "waiting for subops from 2,3,4", 2024-03-11T21:49:09.551 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:44:22.583407+0000", 2024-03-11T21:49:09.551 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.00040668799999999999 2024-03-11T21:49:09.552 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:49:09.552 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:49:09.552 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "sub_op_commit_rec", 2024-03-11T21:49:09.552 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:44:22.595606+0000", 2024-03-11T21:49:09.552 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.0121986 2024-03-11T21:49:09.552 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:49:09.552 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:49:09.552 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "op_commit", 2024-03-11T21:49:09.552 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:44:22.595663+0000", 2024-03-11T21:49:09.552 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 5.7444000000000001e-05 2024-03-11T21:49:09.552 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:49:09.552 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:49:09.552 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "sub_op_commit_rec", 2024-03-11T21:49:09.552 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:44:22.602662+0000", 2024-03-11T21:49:09.552 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.0069990119999999998 2024-03-11T21:49:09.553 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:49:09.553 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:49:09.553 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "sub_op_commit_rec", 2024-03-11T21:49:09.553 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:44:22.605365+0000", 2024-03-11T21:49:09.553 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.0027022470000000001 2024-03-11T21:49:09.553 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:49:09.553 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:49:09.553 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "commit_sent", 2024-03-11T21:49:09.553 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:44:22.605419+0000", 2024-03-11T21:49:09.553 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 5.452e-05 2024-03-11T21:49:09.553 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:49:09.553 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:49:09.553 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "done", 2024-03-11T21:49:09.553 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:44:22.605544+0000", 2024-03-11T21:49:09.553 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.00012433600000000001 2024-03-11T21:49:09.553 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-03-11T21:49:09.554 INFO:teuthology.orchestra.run.smithi097.stdout: ] 2024-03-11T21:49:09.554 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-03-11T21:49:09.554 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:49:09.554 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:49:09.554 INFO:teuthology.orchestra.run.smithi097.stdout: "description": "osd_op(client.15744.0:214916 10.17 10:ec630630:::benchmark_data_smithi092_161462_object104940:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e432)", 2024-03-11T21:49:09.554 INFO:teuthology.orchestra.run.smithi097.stdout: "initiated_at": "2024-03-11T21:44:20.481252+0000", 2024-03-11T21:49:09.554 INFO:teuthology.orchestra.run.smithi097.stdout: "age": 289.01205250599997, 2024-03-11T21:49:09.554 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 2.1345388320000001, 2024-03-11T21:49:09.554 INFO:teuthology.orchestra.run.smithi097.stdout: "type_data": { 2024-03-11T21:49:09.554 INFO:teuthology.orchestra.run.smithi097.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-11T21:49:09.554 INFO:teuthology.orchestra.run.smithi097.stdout: "client_info": { 2024-03-11T21:49:09.554 INFO:teuthology.orchestra.run.smithi097.stdout: "client": "client.15744", 2024-03-11T21:49:09.554 INFO:teuthology.orchestra.run.smithi097.stdout: "client_addr": "v1:172.21.15.92:0/3334000812", 2024-03-11T21:49:09.555 INFO:teuthology.orchestra.run.smithi097.stdout: "tid": 214916 2024-03-11T21:49:09.555 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:49:09.555 INFO:teuthology.orchestra.run.smithi097.stdout: "events": [ 2024-03-11T21:49:09.555 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:49:09.555 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "initiated", 2024-03-11T21:49:09.555 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:44:20.481252+0000", 2024-03-11T21:49:09.555 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-03-11T21:49:09.555 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:49:09.555 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:49:09.555 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "header_read", 2024-03-11T21:49:09.555 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:44:20.481252+0000", 2024-03-11T21:49:09.555 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-03-11T21:49:09.555 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:49:09.555 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:49:09.556 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "throttled", 2024-03-11T21:49:09.556 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:44:20.481253+0000", 2024-03-11T21:49:09.556 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.1000000000000001e-06 2024-03-11T21:49:09.556 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:49:09.556 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:49:09.556 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "all_read", 2024-03-11T21:49:09.556 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:44:20.481257+0000", 2024-03-11T21:49:09.556 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 4.2590000000000002e-06 2024-03-11T21:49:09.556 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:49:09.556 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:49:09.556 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "dispatched", 2024-03-11T21:49:09.556 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:44:20.481267+0000", 2024-03-11T21:49:09.556 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.0465000000000001e-05 2024-03-11T21:49:09.556 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:49:09.556 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:49:09.557 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "queued_for_pg", 2024-03-11T21:49:09.557 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:44:20.481275+0000", 2024-03-11T21:49:09.557 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 7.289e-06 2024-03-11T21:49:09.557 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:49:09.557 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:49:09.557 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "reached_pg", 2024-03-11T21:49:09.557 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:44:20.481310+0000", 2024-03-11T21:49:09.557 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 3.4955e-05 2024-03-11T21:49:09.557 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:49:09.557 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:49:09.557 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "waiting for peered", 2024-03-11T21:49:09.557 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:44:20.481316+0000", 2024-03-11T21:49:09.557 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 6.0430000000000004e-06 2024-03-11T21:49:09.557 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:49:09.557 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:49:09.558 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "reached_pg", 2024-03-11T21:49:09.558 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:44:21.432627+0000", 2024-03-11T21:49:09.558 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.95131141299999999 2024-03-11T21:49:09.558 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:49:09.558 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:49:09.558 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "waiting for peered", 2024-03-11T21:49:09.558 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:44:21.432636+0000", 2024-03-11T21:49:09.558 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 8.8170000000000005e-06 2024-03-11T21:49:09.558 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:49:09.558 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:49:09.558 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "reached_pg", 2024-03-11T21:49:09.558 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:44:22.608545+0000", 2024-03-11T21:49:09.558 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.175909144 2024-03-11T21:49:09.558 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:49:09.558 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:49:09.559 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "started", 2024-03-11T21:49:09.559 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:44:22.608666+0000", 2024-03-11T21:49:09.559 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.000121222 2024-03-11T21:49:09.559 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:49:09.559 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:49:09.559 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "waiting for subops from 2,3,5", 2024-03-11T21:49:09.559 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:44:22.608804+0000", 2024-03-11T21:49:09.559 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.000137365 2024-03-11T21:49:09.559 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:49:09.559 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:49:09.559 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "op_commit", 2024-03-11T21:49:09.559 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:44:22.611587+0000", 2024-03-11T21:49:09.559 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.002783593 2024-03-11T21:49:09.559 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:49:09.559 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:49:09.560 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "sub_op_commit_rec", 2024-03-11T21:49:09.560 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:44:22.611841+0000", 2024-03-11T21:49:09.560 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.00025328699999999999 2024-03-11T21:49:09.560 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:49:09.560 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:49:09.560 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "sub_op_commit_rec", 2024-03-11T21:49:09.560 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:44:22.614866+0000", 2024-03-11T21:49:09.560 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.003025823 2024-03-11T21:49:09.560 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:49:09.560 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:49:09.560 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "sub_op_commit_rec", 2024-03-11T21:49:09.560 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:44:22.614911+0000", 2024-03-11T21:49:09.560 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 4.4569e-05 2024-03-11T21:49:09.560 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:49:09.561 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:49:09.561 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "commit_sent", 2024-03-11T21:49:09.561 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:44:22.614973+0000", 2024-03-11T21:49:09.561 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 6.2244999999999998e-05 2024-03-11T21:49:09.561 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:49:09.561 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:49:09.561 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "done", 2024-03-11T21:49:09.561 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:44:22.615790+0000", 2024-03-11T21:49:09.561 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.00081724300000000005 2024-03-11T21:49:09.561 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-03-11T21:49:09.561 INFO:teuthology.orchestra.run.smithi097.stdout: ] 2024-03-11T21:49:09.561 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-03-11T21:49:09.561 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:49:09.561 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:49:09.561 INFO:teuthology.orchestra.run.smithi097.stdout: "description": "osd_op(client.15744.0:217010 10.17 10:ed2db962:::benchmark_data_smithi092_161462_object107034:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e434)", 2024-03-11T21:49:09.562 INFO:teuthology.orchestra.run.smithi097.stdout: "initiated_at": "2024-03-11T21:44:24.424632+0000", 2024-03-11T21:49:09.562 INFO:teuthology.orchestra.run.smithi097.stdout: "age": 285.06867190700001, 2024-03-11T21:49:09.562 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.0482623639999999, 2024-03-11T21:49:09.562 INFO:teuthology.orchestra.run.smithi097.stdout: "type_data": { 2024-03-11T21:49:09.562 INFO:teuthology.orchestra.run.smithi097.stdout: "flag_point": "reached pg", 2024-03-11T21:49:09.562 INFO:teuthology.orchestra.run.smithi097.stdout: "client_info": { 2024-03-11T21:49:09.562 INFO:teuthology.orchestra.run.smithi097.stdout: "client": "client.15744", 2024-03-11T21:49:09.562 INFO:teuthology.orchestra.run.smithi097.stdout: "client_addr": "v1:172.21.15.92:0/3334000812", 2024-03-11T21:49:09.562 INFO:teuthology.orchestra.run.smithi097.stdout: "tid": 217010 2024-03-11T21:49:09.562 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:49:09.562 INFO:teuthology.orchestra.run.smithi097.stdout: "events": [ 2024-03-11T21:49:09.562 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:49:09.562 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "initiated", 2024-03-11T21:49:09.562 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:44:24.424632+0000", 2024-03-11T21:49:09.562 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-03-11T21:49:09.563 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:49:09.563 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:49:09.563 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "header_read", 2024-03-11T21:49:09.563 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:44:24.424632+0000", 2024-03-11T21:49:09.563 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-03-11T21:49:09.563 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:49:09.563 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:49:09.563 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "throttled", 2024-03-11T21:49:09.563 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:44:24.424634+0000", 2024-03-11T21:49:09.563 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.66e-06 2024-03-11T21:49:09.563 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:49:09.563 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:49:09.563 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "all_read", 2024-03-11T21:49:09.563 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:44:24.424639+0000", 2024-03-11T21:49:09.563 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 5.0889999999999999e-06 2024-03-11T21:49:09.564 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:49:09.564 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:49:09.564 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "dispatched", 2024-03-11T21:49:09.564 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:44:24.424651+0000", 2024-03-11T21:49:09.564 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.1799e-05 2024-03-11T21:49:09.564 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:49:09.564 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:49:09.564 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "queued_for_pg", 2024-03-11T21:49:09.564 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:44:24.424658+0000", 2024-03-11T21:49:09.564 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 7.6359999999999994e-06 2024-03-11T21:49:09.564 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:49:09.564 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:49:09.564 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "reached_pg", 2024-03-11T21:49:09.564 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:44:24.424735+0000", 2024-03-11T21:49:09.564 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 7.7013000000000003e-05 2024-03-11T21:49:09.565 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:49:09.565 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:49:09.565 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "started", 2024-03-11T21:49:09.565 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:44:24.424803+0000", 2024-03-11T21:49:09.565 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 6.7836999999999994e-05 2024-03-11T21:49:09.565 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:49:09.565 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:49:09.565 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "waiting for subops from 2,3,5", 2024-03-11T21:49:09.565 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:44:24.424906+0000", 2024-03-11T21:49:09.565 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.00010268999999999999 2024-03-11T21:49:09.565 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:49:09.565 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:49:09.565 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "sub_op_commit_rec", 2024-03-11T21:49:09.565 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:44:24.432605+0000", 2024-03-11T21:49:09.565 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.0076993189999999996 2024-03-11T21:49:09.566 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:49:09.566 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:49:09.566 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "op_commit", 2024-03-11T21:49:09.566 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:44:24.432879+0000", 2024-03-11T21:49:09.566 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.00027418899999999999 2024-03-11T21:49:09.566 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:49:09.566 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:49:09.566 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "sub_op_commit_rec", 2024-03-11T21:49:09.566 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:44:24.441374+0000", 2024-03-11T21:49:09.566 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.0084943150000000005 2024-03-11T21:49:09.566 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:49:09.566 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:49:09.566 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "reached_pg", 2024-03-11T21:49:09.566 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:44:24.452526+0000", 2024-03-11T21:49:09.567 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.011152485 2024-03-11T21:49:09.567 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:49:09.567 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:49:09.567 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "waiting for peered", 2024-03-11T21:49:09.567 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:44:24.452538+0000", 2024-03-11T21:49:09.567 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.1905999999999999e-05 2024-03-11T21:49:09.567 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:49:09.567 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:49:09.567 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "reached_pg", 2024-03-11T21:49:09.567 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:44:25.472810+0000", 2024-03-11T21:49:09.567 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.0202715959999999 2024-03-11T21:49:09.567 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:49:09.567 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:49:09.567 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "done", 2024-03-11T21:49:09.567 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:44:25.472895+0000", 2024-03-11T21:49:09.568 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 8.4829999999999999e-05 2024-03-11T21:49:09.568 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-03-11T21:49:09.568 INFO:teuthology.orchestra.run.smithi097.stdout: ] 2024-03-11T21:49:09.568 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-03-11T21:49:09.568 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:49:09.568 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:49:09.568 INFO:teuthology.orchestra.run.smithi097.stdout: "description": "osd_op(client.15744.0:217014 10.17 10:efb597aa:::benchmark_data_smithi092_161462_object107038:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e434)", 2024-03-11T21:49:09.568 INFO:teuthology.orchestra.run.smithi097.stdout: "initiated_at": "2024-03-11T21:44:24.426189+0000", 2024-03-11T21:49:09.568 INFO:teuthology.orchestra.run.smithi097.stdout: "age": 285.06711488299999, 2024-03-11T21:49:09.568 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.0472537319999999, 2024-03-11T21:49:09.568 INFO:teuthology.orchestra.run.smithi097.stdout: "type_data": { 2024-03-11T21:49:09.568 INFO:teuthology.orchestra.run.smithi097.stdout: "flag_point": "reached pg", 2024-03-11T21:49:09.568 INFO:teuthology.orchestra.run.smithi097.stdout: "client_info": { 2024-03-11T21:49:09.568 INFO:teuthology.orchestra.run.smithi097.stdout: "client": "client.15744", 2024-03-11T21:49:09.569 INFO:teuthology.orchestra.run.smithi097.stdout: "client_addr": "v1:172.21.15.92:0/3334000812", 2024-03-11T21:49:09.569 INFO:teuthology.orchestra.run.smithi097.stdout: "tid": 217014 2024-03-11T21:49:09.569 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:49:09.569 INFO:teuthology.orchestra.run.smithi097.stdout: "events": [ 2024-03-11T21:49:09.569 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:49:09.569 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "initiated", 2024-03-11T21:49:09.569 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:44:24.426189+0000", 2024-03-11T21:49:09.569 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-03-11T21:49:09.569 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:49:09.569 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:49:09.569 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "header_read", 2024-03-11T21:49:09.569 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:44:24.426189+0000", 2024-03-11T21:49:09.569 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-03-11T21:49:09.569 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:49:09.569 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:49:09.570 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "throttled", 2024-03-11T21:49:09.570 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:44:24.426190+0000", 2024-03-11T21:49:09.570 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 9.8899999999999998e-07 2024-03-11T21:49:09.570 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:49:09.570 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:49:09.570 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "all_read", 2024-03-11T21:49:09.570 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:44:24.426194+0000", 2024-03-11T21:49:09.570 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 3.3689999999999998e-06 2024-03-11T21:49:09.570 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:49:09.570 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:49:09.570 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "dispatched", 2024-03-11T21:49:09.570 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:44:24.426200+0000", 2024-03-11T21:49:09.570 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 6.8539999999999999e-06 2024-03-11T21:49:09.571 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:49:09.571 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:49:09.571 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "queued_for_pg", 2024-03-11T21:49:09.571 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:44:24.426206+0000", 2024-03-11T21:49:09.571 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 5.9150000000000001e-06 2024-03-11T21:49:09.571 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:49:09.571 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:49:09.571 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "reached_pg", 2024-03-11T21:49:09.571 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:44:24.430013+0000", 2024-03-11T21:49:09.571 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.0038069359999999999 2024-03-11T21:49:09.571 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:49:09.571 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:49:09.571 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "started", 2024-03-11T21:49:09.571 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:44:24.430124+0000", 2024-03-11T21:49:09.571 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.000111045 2024-03-11T21:49:09.572 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:49:09.572 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:49:09.572 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "waiting for subops from 2,3,5", 2024-03-11T21:49:09.572 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:44:24.430248+0000", 2024-03-11T21:49:09.572 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.000123771 2024-03-11T21:49:09.572 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:49:09.572 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:49:09.572 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "sub_op_commit_rec", 2024-03-11T21:49:09.572 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:44:24.432654+0000", 2024-03-11T21:49:09.572 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.0024057829999999999 2024-03-11T21:49:09.572 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:49:09.572 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:49:09.572 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "op_commit", 2024-03-11T21:49:09.572 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:44:24.435213+0000", 2024-03-11T21:49:09.572 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.0025588239999999999 2024-03-11T21:49:09.573 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:49:09.573 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:49:09.573 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "sub_op_commit_rec", 2024-03-11T21:49:09.573 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:44:24.441459+0000", 2024-03-11T21:49:09.573 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.0062460429999999997 2024-03-11T21:49:09.573 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:49:09.573 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:49:09.573 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "reached_pg", 2024-03-11T21:49:09.573 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:44:24.452570+0000", 2024-03-11T21:49:09.573 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.011110764 2024-03-11T21:49:09.573 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:49:09.573 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:49:09.573 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "waiting for peered", 2024-03-11T21:49:09.573 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:44:24.452582+0000", 2024-03-11T21:49:09.573 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.2588e-05 2024-03-11T21:49:09.573 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:49:09.574 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:49:09.574 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "reached_pg", 2024-03-11T21:49:09.574 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:44:25.473391+0000", 2024-03-11T21:49:09.574 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.020808691 2024-03-11T21:49:09.574 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:49:09.574 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:49:09.574 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "done", 2024-03-11T21:49:09.574 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:44:25.473443+0000", 2024-03-11T21:49:09.574 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 5.2160000000000002e-05 2024-03-11T21:49:09.574 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-03-11T21:49:09.574 INFO:teuthology.orchestra.run.smithi097.stdout: ] 2024-03-11T21:49:09.574 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-03-11T21:49:09.574 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:49:09.575 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:49:09.575 INFO:teuthology.orchestra.run.smithi097.stdout: "description": "osd_op(client.15744.0:217043 10.17 10:e8bb19d6:::benchmark_data_smithi092_161462_object107067:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e435)", 2024-03-11T21:49:09.575 INFO:teuthology.orchestra.run.smithi097.stdout: "initiated_at": "2024-03-11T21:44:24.458969+0000", 2024-03-11T21:49:09.575 INFO:teuthology.orchestra.run.smithi097.stdout: "age": 285.03433523899997, 2024-03-11T21:49:09.575 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.0208789760000001, 2024-03-11T21:49:09.575 INFO:teuthology.orchestra.run.smithi097.stdout: "type_data": { 2024-03-11T21:49:09.575 INFO:teuthology.orchestra.run.smithi097.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-11T21:49:09.575 INFO:teuthology.orchestra.run.smithi097.stdout: "client_info": { 2024-03-11T21:49:09.575 INFO:teuthology.orchestra.run.smithi097.stdout: "client": "client.15744", 2024-03-11T21:49:09.575 INFO:teuthology.orchestra.run.smithi097.stdout: "client_addr": "v1:172.21.15.92:0/3334000812", 2024-03-11T21:49:09.575 INFO:teuthology.orchestra.run.smithi097.stdout: "tid": 217043 2024-03-11T21:49:09.575 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:49:09.575 INFO:teuthology.orchestra.run.smithi097.stdout: "events": [ 2024-03-11T21:49:09.575 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:49:09.575 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "initiated", 2024-03-11T21:49:09.576 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:44:24.458969+0000", 2024-03-11T21:49:09.576 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-03-11T21:49:09.576 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:49:09.576 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:49:09.576 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "header_read", 2024-03-11T21:49:09.576 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:44:24.458969+0000", 2024-03-11T21:49:09.576 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-03-11T21:49:09.576 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:49:09.576 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:49:09.576 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "throttled", 2024-03-11T21:49:09.576 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:44:24.458970+0000", 2024-03-11T21:49:09.576 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.4589999999999999e-06 2024-03-11T21:49:09.576 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:49:09.576 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:49:09.576 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "all_read", 2024-03-11T21:49:09.577 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:44:24.458975+0000", 2024-03-11T21:49:09.577 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 4.8590000000000004e-06 2024-03-11T21:49:09.577 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:49:09.577 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:49:09.577 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "dispatched", 2024-03-11T21:49:09.577 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:44:24.458986+0000", 2024-03-11T21:49:09.577 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.0626e-05 2024-03-11T21:49:09.577 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:49:09.577 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:49:09.577 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "queued_for_pg", 2024-03-11T21:49:09.577 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:44:24.458994+0000", 2024-03-11T21:49:09.577 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 7.8660000000000006e-06 2024-03-11T21:49:09.577 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:49:09.577 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:49:09.578 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "reached_pg", 2024-03-11T21:49:09.578 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:44:24.459205+0000", 2024-03-11T21:49:09.578 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.00021135899999999999 2024-03-11T21:49:09.578 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:49:09.578 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:49:09.578 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "waiting for peered", 2024-03-11T21:49:09.578 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:44:24.459213+0000", 2024-03-11T21:49:09.578 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 8.1519999999999992e-06 2024-03-11T21:49:09.578 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:49:09.578 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:49:09.578 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "reached_pg", 2024-03-11T21:49:09.578 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:44:25.473478+0000", 2024-03-11T21:49:09.578 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.014264906 2024-03-11T21:49:09.578 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:49:09.579 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:49:09.579 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "started", 2024-03-11T21:49:09.579 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:44:25.473561+0000", 2024-03-11T21:49:09.579 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 8.2705000000000002e-05 2024-03-11T21:49:09.579 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:49:09.579 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:49:09.579 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "waiting for subops from 2,5", 2024-03-11T21:49:09.579 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:44:25.473634+0000", 2024-03-11T21:49:09.579 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 7.2727999999999999e-05 2024-03-11T21:49:09.579 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:49:09.579 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:49:09.579 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "sub_op_commit_rec", 2024-03-11T21:49:09.579 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:44:25.477382+0000", 2024-03-11T21:49:09.579 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.0037484390000000001 2024-03-11T21:49:09.579 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:49:09.580 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:49:09.580 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "op_commit", 2024-03-11T21:49:09.580 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:44:25.477527+0000", 2024-03-11T21:49:09.580 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.00014497300000000001 2024-03-11T21:49:09.580 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:49:09.580 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:49:09.580 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "sub_op_commit_rec", 2024-03-11T21:49:09.580 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:44:25.479764+0000", 2024-03-11T21:49:09.580 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.0022373660000000002 2024-03-11T21:49:09.580 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:49:09.580 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:49:09.580 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "commit_sent", 2024-03-11T21:49:09.580 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:44:25.479801+0000", 2024-03-11T21:49:09.580 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 3.6600000000000002e-05 2024-03-11T21:49:09.580 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:49:09.581 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:49:09.581 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "done", 2024-03-11T21:49:09.581 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:44:25.479848+0000", 2024-03-11T21:49:09.581 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 4.6937999999999997e-05 2024-03-11T21:49:09.581 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-03-11T21:49:09.581 INFO:teuthology.orchestra.run.smithi097.stdout: ] 2024-03-11T21:49:09.581 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-03-11T21:49:09.581 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:49:09.581 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:49:09.581 INFO:teuthology.orchestra.run.smithi097.stdout: "description": "osd_op(client.15744.0:217128 10.17 10:edd91f94:::benchmark_data_smithi092_161462_object107152:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e435)", 2024-03-11T21:49:09.581 INFO:teuthology.orchestra.run.smithi097.stdout: "initiated_at": "2024-03-11T21:44:24.548360+0000", 2024-03-11T21:49:09.581 INFO:teuthology.orchestra.run.smithi097.stdout: "age": 284.94494452200001, 2024-03-11T21:49:09.581 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.93895680699999995, 2024-03-11T21:49:09.581 INFO:teuthology.orchestra.run.smithi097.stdout: "type_data": { 2024-03-11T21:49:09.582 INFO:teuthology.orchestra.run.smithi097.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-11T21:49:09.582 INFO:teuthology.orchestra.run.smithi097.stdout: "client_info": { 2024-03-11T21:49:09.582 INFO:teuthology.orchestra.run.smithi097.stdout: "client": "client.15744", 2024-03-11T21:49:09.582 INFO:teuthology.orchestra.run.smithi097.stdout: "client_addr": "v1:172.21.15.92:0/3334000812", 2024-03-11T21:49:09.582 INFO:teuthology.orchestra.run.smithi097.stdout: "tid": 217128 2024-03-11T21:49:09.582 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:49:09.582 INFO:teuthology.orchestra.run.smithi097.stdout: "events": [ 2024-03-11T21:49:09.582 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:49:09.582 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "initiated", 2024-03-11T21:49:09.582 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:44:24.548360+0000", 2024-03-11T21:49:09.582 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-03-11T21:49:09.582 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:49:09.582 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:49:09.582 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "header_read", 2024-03-11T21:49:09.583 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:44:24.548360+0000", 2024-03-11T21:49:09.583 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-03-11T21:49:09.583 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:49:09.583 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:49:09.583 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "throttled", 2024-03-11T21:49:09.583 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:44:24.548361+0000", 2024-03-11T21:49:09.583 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.454e-06 2024-03-11T21:49:09.583 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:49:09.583 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:49:09.583 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "all_read", 2024-03-11T21:49:09.583 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:44:24.548366+0000", 2024-03-11T21:49:09.583 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 4.9520000000000002e-06 2024-03-11T21:49:09.583 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:49:09.583 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:49:09.583 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "dispatched", 2024-03-11T21:49:09.584 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:44:24.548380+0000", 2024-03-11T21:49:09.584 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.3643e-05 2024-03-11T21:49:09.584 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:49:09.584 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:49:09.584 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "queued_for_pg", 2024-03-11T21:49:09.584 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:44:24.548388+0000", 2024-03-11T21:49:09.584 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 8.7530000000000007e-06 2024-03-11T21:49:09.584 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:49:09.584 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:49:09.584 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "reached_pg", 2024-03-11T21:49:09.584 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:44:24.548431+0000", 2024-03-11T21:49:09.584 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 4.2936999999999999e-05 2024-03-11T21:49:09.584 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:49:09.584 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:49:09.584 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "waiting for peered", 2024-03-11T21:49:09.585 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:44:24.548437+0000", 2024-03-11T21:49:09.585 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 5.6389999999999997e-06 2024-03-11T21:49:09.585 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:49:09.585 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:49:09.585 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "reached_pg", 2024-03-11T21:49:09.585 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:44:25.475472+0000", 2024-03-11T21:49:09.585 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.92703524400000004 2024-03-11T21:49:09.585 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:49:09.585 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:49:09.585 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "started", 2024-03-11T21:49:09.585 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:44:25.475549+0000", 2024-03-11T21:49:09.585 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 7.6334999999999993e-05 2024-03-11T21:49:09.585 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:49:09.585 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:49:09.586 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "waiting for subops from 2,5", 2024-03-11T21:49:09.586 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:44:25.475612+0000", 2024-03-11T21:49:09.586 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 6.3127999999999996e-05 2024-03-11T21:49:09.586 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:49:09.586 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:49:09.586 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "sub_op_commit_rec", 2024-03-11T21:49:09.586 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:44:25.477425+0000", 2024-03-11T21:49:09.586 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.0018130830000000001 2024-03-11T21:49:09.586 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:49:09.586 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:49:09.586 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "op_commit", 2024-03-11T21:49:09.586 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:44:25.477535+0000", 2024-03-11T21:49:09.586 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.00011052600000000001 2024-03-11T21:49:09.586 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:49:09.586 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:49:09.587 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "sub_op_commit_rec", 2024-03-11T21:49:09.587 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:44:25.487232+0000", 2024-03-11T21:49:09.587 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.0096962590000000005 2024-03-11T21:49:09.587 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:49:09.587 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:49:09.587 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "commit_sent", 2024-03-11T21:49:09.587 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:44:25.487267+0000", 2024-03-11T21:49:09.587 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 3.5373999999999997e-05 2024-03-11T21:49:09.587 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:49:09.587 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:49:09.587 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "done", 2024-03-11T21:49:09.587 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:44:25.487316+0000", 2024-03-11T21:49:09.587 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 4.9480000000000001e-05 2024-03-11T21:49:09.587 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-03-11T21:49:09.587 INFO:teuthology.orchestra.run.smithi097.stdout: ] 2024-03-11T21:49:09.588 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-03-11T21:49:09.588 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:49:09.588 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:49:09.588 INFO:teuthology.orchestra.run.smithi097.stdout: "description": "osd_op(client.35813.0:34853 11.1 11:802551d2:::benchmark_data_smithi092_186520_object34852:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e452)", 2024-03-11T21:49:09.588 INFO:teuthology.orchestra.run.smithi097.stdout: "initiated_at": "2024-03-11T21:45:21.057909+0000", 2024-03-11T21:49:09.588 INFO:teuthology.orchestra.run.smithi097.stdout: "age": 228.43539551699999, 2024-03-11T21:49:09.588 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.21724974899999999, 2024-03-11T21:49:09.588 INFO:teuthology.orchestra.run.smithi097.stdout: "type_data": { 2024-03-11T21:49:09.588 INFO:teuthology.orchestra.run.smithi097.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-11T21:49:09.588 INFO:teuthology.orchestra.run.smithi097.stdout: "client_info": { 2024-03-11T21:49:09.588 INFO:teuthology.orchestra.run.smithi097.stdout: "client": "client.35813", 2024-03-11T21:49:09.588 INFO:teuthology.orchestra.run.smithi097.stdout: "client_addr": "v1:172.21.15.92:0/4293352030", 2024-03-11T21:49:09.588 INFO:teuthology.orchestra.run.smithi097.stdout: "tid": 34853 2024-03-11T21:49:09.588 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:49:09.589 INFO:teuthology.orchestra.run.smithi097.stdout: "events": [ 2024-03-11T21:49:09.589 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:49:09.589 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "initiated", 2024-03-11T21:49:09.589 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:45:21.057909+0000", 2024-03-11T21:49:09.589 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-03-11T21:49:09.589 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:49:09.589 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:49:09.589 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "header_read", 2024-03-11T21:49:09.589 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:45:21.057909+0000", 2024-03-11T21:49:09.589 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-03-11T21:49:09.589 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:49:09.589 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:49:09.589 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "throttled", 2024-03-11T21:49:09.589 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:45:21.057911+0000", 2024-03-11T21:49:09.589 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.945e-06 2024-03-11T21:49:09.590 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:49:09.590 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:49:09.590 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "all_read", 2024-03-11T21:49:09.590 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:45:21.057949+0000", 2024-03-11T21:49:09.590 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 3.7991000000000003e-05 2024-03-11T21:49:09.590 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:49:09.590 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:49:09.590 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "dispatched", 2024-03-11T21:49:09.590 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:45:21.057959+0000", 2024-03-11T21:49:09.590 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.0441e-05 2024-03-11T21:49:09.590 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:49:09.590 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:49:09.590 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "queued_for_pg", 2024-03-11T21:49:09.590 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:45:21.057966+0000", 2024-03-11T21:49:09.591 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 7.3669999999999999e-06 2024-03-11T21:49:09.591 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:49:09.591 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:49:09.591 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "reached_pg", 2024-03-11T21:49:09.591 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:45:21.058064+0000", 2024-03-11T21:49:09.591 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 9.7529999999999996e-05 2024-03-11T21:49:09.591 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:49:09.591 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:49:09.591 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "started", 2024-03-11T21:49:09.591 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:45:21.058248+0000", 2024-03-11T21:49:09.591 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.000183745 2024-03-11T21:49:09.591 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:49:09.591 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:49:09.591 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "waiting for subops from 0,2", 2024-03-11T21:49:09.591 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:45:21.058521+0000", 2024-03-11T21:49:09.592 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.00027350900000000001 2024-03-11T21:49:09.592 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:49:09.592 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:49:09.592 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "sub_op_commit_rec", 2024-03-11T21:49:09.592 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:45:21.061444+0000", 2024-03-11T21:49:09.592 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.0029226819999999998 2024-03-11T21:49:09.592 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:49:09.592 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:49:09.592 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "op_commit", 2024-03-11T21:49:09.592 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:45:21.061462+0000", 2024-03-11T21:49:09.592 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.7799999999999999e-05 2024-03-11T21:49:09.592 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:49:09.592 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:49:09.592 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "sub_op_commit_rec", 2024-03-11T21:49:09.592 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:45:21.275079+0000", 2024-03-11T21:49:09.593 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.21361718599999999 2024-03-11T21:49:09.593 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:49:09.593 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:49:09.593 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "commit_sent", 2024-03-11T21:49:09.593 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:45:21.275125+0000", 2024-03-11T21:49:09.593 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 4.6538000000000001e-05 2024-03-11T21:49:09.593 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:49:09.593 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:49:09.593 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "done", 2024-03-11T21:49:09.593 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:45:21.275158+0000", 2024-03-11T21:49:09.593 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 3.3015000000000002e-05 2024-03-11T21:49:09.593 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-03-11T21:49:09.593 INFO:teuthology.orchestra.run.smithi097.stdout: ] 2024-03-11T21:49:09.593 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-03-11T21:49:09.593 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:49:09.594 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:49:09.594 INFO:teuthology.orchestra.run.smithi097.stdout: "description": "osd_op(client.35813.0:34854 11.1 11:8a25fd00:::benchmark_data_smithi092_186520_object34853:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e452)", 2024-03-11T21:49:09.594 INFO:teuthology.orchestra.run.smithi097.stdout: "initiated_at": "2024-03-11T21:45:21.058030+0000", 2024-03-11T21:49:09.594 INFO:teuthology.orchestra.run.smithi097.stdout: "age": 228.435273911, 2024-03-11T21:49:09.594 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.21721339200000001, 2024-03-11T21:49:09.594 INFO:teuthology.orchestra.run.smithi097.stdout: "type_data": { 2024-03-11T21:49:09.594 INFO:teuthology.orchestra.run.smithi097.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-11T21:49:09.594 INFO:teuthology.orchestra.run.smithi097.stdout: "client_info": { 2024-03-11T21:49:09.594 INFO:teuthology.orchestra.run.smithi097.stdout: "client": "client.35813", 2024-03-11T21:49:09.594 INFO:teuthology.orchestra.run.smithi097.stdout: "client_addr": "v1:172.21.15.92:0/4293352030", 2024-03-11T21:49:09.594 INFO:teuthology.orchestra.run.smithi097.stdout: "tid": 34854 2024-03-11T21:49:09.594 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:49:09.594 INFO:teuthology.orchestra.run.smithi097.stdout: "events": [ 2024-03-11T21:49:09.595 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:49:09.595 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "initiated", 2024-03-11T21:49:09.595 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:45:21.058030+0000", 2024-03-11T21:49:09.595 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-03-11T21:49:09.595 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:49:09.595 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:49:09.595 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "header_read", 2024-03-11T21:49:09.595 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:45:21.058030+0000", 2024-03-11T21:49:09.595 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-03-11T21:49:09.595 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:49:09.595 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:49:09.595 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "throttled", 2024-03-11T21:49:09.595 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:45:21.058031+0000", 2024-03-11T21:49:09.595 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 7.1800000000000005e-07 2024-03-11T21:49:09.595 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:49:09.596 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:49:09.596 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "all_read", 2024-03-11T21:49:09.596 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:45:21.058099+0000", 2024-03-11T21:49:09.596 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 6.8262000000000005e-05 2024-03-11T21:49:09.596 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:49:09.596 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:49:09.596 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "dispatched", 2024-03-11T21:49:09.596 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:45:21.058113+0000", 2024-03-11T21:49:09.596 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.3317e-05 2024-03-11T21:49:09.596 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:49:09.596 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:49:09.596 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "queued_for_pg", 2024-03-11T21:49:09.596 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:45:21.058120+0000", 2024-03-11T21:49:09.596 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 7.4379999999999998e-06 2024-03-11T21:49:09.597 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:49:09.597 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:49:09.597 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "reached_pg", 2024-03-11T21:49:09.597 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:45:21.059950+0000", 2024-03-11T21:49:09.597 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.001829841 2024-03-11T21:49:09.597 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:49:09.597 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:49:09.597 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "started", 2024-03-11T21:49:09.597 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:45:21.060118+0000", 2024-03-11T21:49:09.597 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.00016834099999999999 2024-03-11T21:49:09.597 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:49:09.597 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:49:09.597 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "waiting for subops from 0,2", 2024-03-11T21:49:09.597 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:45:21.060268+0000", 2024-03-11T21:49:09.597 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.000150328 2024-03-11T21:49:09.598 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:49:09.598 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:49:09.598 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "op_commit", 2024-03-11T21:49:09.598 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:45:21.061619+0000", 2024-03-11T21:49:09.598 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.0013500890000000001 2024-03-11T21:49:09.598 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:49:09.598 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:49:09.598 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "sub_op_commit_rec", 2024-03-11T21:49:09.598 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:45:21.061953+0000", 2024-03-11T21:49:09.598 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.00033418599999999999 2024-03-11T21:49:09.598 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:49:09.598 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:49:09.598 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "sub_op_commit_rec", 2024-03-11T21:49:09.598 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:45:21.275187+0000", 2024-03-11T21:49:09.599 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.21323399800000001 2024-03-11T21:49:09.599 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:49:09.599 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:49:09.599 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "commit_sent", 2024-03-11T21:49:09.599 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:45:21.275221+0000", 2024-03-11T21:49:09.599 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 3.4153999999999999e-05 2024-03-11T21:49:09.599 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:49:09.599 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:49:09.599 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "done", 2024-03-11T21:49:09.599 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:45:21.275244+0000", 2024-03-11T21:49:09.599 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 2.2719999999999999e-05 2024-03-11T21:49:09.599 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-03-11T21:49:09.599 INFO:teuthology.orchestra.run.smithi097.stdout: ] 2024-03-11T21:49:09.599 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-03-11T21:49:09.600 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:49:09.600 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:49:09.600 INFO:teuthology.orchestra.run.smithi097.stdout: "description": "osd_op(client.35813.0:189886 11.3 11:c4478f1e:::benchmark_data_smithi092_186520_object51538:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e477)", 2024-03-11T21:49:09.600 INFO:teuthology.orchestra.run.smithi097.stdout: "initiated_at": "2024-03-11T21:47:15.961459+0000", 2024-03-11T21:49:09.600 INFO:teuthology.orchestra.run.smithi097.stdout: "age": 113.531845057, 2024-03-11T21:49:09.600 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.24263093299999999, 2024-03-11T21:49:09.600 INFO:teuthology.orchestra.run.smithi097.stdout: "type_data": { 2024-03-11T21:49:09.600 INFO:teuthology.orchestra.run.smithi097.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-11T21:49:09.600 INFO:teuthology.orchestra.run.smithi097.stdout: "client_info": { 2024-03-11T21:49:09.600 INFO:teuthology.orchestra.run.smithi097.stdout: "client": "client.35813", 2024-03-11T21:49:09.600 INFO:teuthology.orchestra.run.smithi097.stdout: "client_addr": "v1:172.21.15.92:0/4293352030", 2024-03-11T21:49:09.600 INFO:teuthology.orchestra.run.smithi097.stdout: "tid": 189886 2024-03-11T21:49:09.600 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:49:09.600 INFO:teuthology.orchestra.run.smithi097.stdout: "events": [ 2024-03-11T21:49:09.600 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:49:09.601 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "initiated", 2024-03-11T21:49:09.601 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:47:15.961459+0000", 2024-03-11T21:49:09.601 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-03-11T21:49:09.601 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:49:09.601 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:49:09.601 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "header_read", 2024-03-11T21:49:09.601 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:47:15.961459+0000", 2024-03-11T21:49:09.601 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-03-11T21:49:09.601 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:49:09.601 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:49:09.601 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "throttled", 2024-03-11T21:49:09.601 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:47:15.961462+0000", 2024-03-11T21:49:09.601 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 2.7310000000000002e-06 2024-03-11T21:49:09.601 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:49:09.601 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:49:09.602 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "all_read", 2024-03-11T21:49:09.602 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:47:15.961468+0000", 2024-03-11T21:49:09.602 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 5.6960000000000002e-06 2024-03-11T21:49:09.602 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:49:09.602 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:49:09.602 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "dispatched", 2024-03-11T21:49:09.602 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:47:15.961484+0000", 2024-03-11T21:49:09.602 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.6844e-05 2024-03-11T21:49:09.602 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:49:09.602 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:49:09.602 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "queued_for_pg", 2024-03-11T21:49:09.602 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:47:15.961492+0000", 2024-03-11T21:49:09.602 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 8.028e-06 2024-03-11T21:49:09.602 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:49:09.603 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:49:09.603 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "reached_pg", 2024-03-11T21:49:09.603 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:47:15.961546+0000", 2024-03-11T21:49:09.603 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 5.3338999999999999e-05 2024-03-11T21:49:09.603 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:49:09.603 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:49:09.603 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "started", 2024-03-11T21:49:09.603 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:47:15.961648+0000", 2024-03-11T21:49:09.603 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.000102599 2024-03-11T21:49:09.603 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:49:09.603 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:49:09.603 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "waiting for subops from 1,7", 2024-03-11T21:49:09.603 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:47:15.961713+0000", 2024-03-11T21:49:09.603 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 6.5000999999999996e-05 2024-03-11T21:49:09.603 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:49:09.604 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:49:09.604 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "op_commit", 2024-03-11T21:49:09.604 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:47:15.963446+0000", 2024-03-11T21:49:09.604 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.0017322590000000001 2024-03-11T21:49:09.604 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:49:09.604 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:49:09.604 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "sub_op_commit_rec", 2024-03-11T21:49:09.604 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:47:15.963862+0000", 2024-03-11T21:49:09.604 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.00041665499999999999 2024-03-11T21:49:09.604 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:49:09.604 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:49:09.604 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "sub_op_commit_rec", 2024-03-11T21:49:09.604 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:47:16.204026+0000", 2024-03-11T21:49:09.604 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.24016363800000001 2024-03-11T21:49:09.604 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:49:09.605 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:49:09.605 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "commit_sent", 2024-03-11T21:49:09.605 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:47:16.204060+0000", 2024-03-11T21:49:09.605 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 3.4107999999999999e-05 2024-03-11T21:49:09.605 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:49:09.605 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:49:09.605 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "done", 2024-03-11T21:49:09.605 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:47:16.204090+0000", 2024-03-11T21:49:09.605 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 3.0035e-05 2024-03-11T21:49:09.605 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-03-11T21:49:09.605 INFO:teuthology.orchestra.run.smithi097.stdout: ] 2024-03-11T21:49:09.605 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-03-11T21:49:09.605 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:49:09.605 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:49:09.605 INFO:teuthology.orchestra.run.smithi097.stdout: "description": "osd_op(client.35813.0:189901 11.3 11:cbb05894:::benchmark_data_smithi092_186520_object51553:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e477)", 2024-03-11T21:49:09.606 INFO:teuthology.orchestra.run.smithi097.stdout: "initiated_at": "2024-03-11T21:47:15.973467+0000", 2024-03-11T21:49:09.606 INFO:teuthology.orchestra.run.smithi097.stdout: "age": 113.519837612, 2024-03-11T21:49:09.606 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.23448206799999999, 2024-03-11T21:49:09.606 INFO:teuthology.orchestra.run.smithi097.stdout: "type_data": { 2024-03-11T21:49:09.606 INFO:teuthology.orchestra.run.smithi097.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-11T21:49:09.606 INFO:teuthology.orchestra.run.smithi097.stdout: "client_info": { 2024-03-11T21:49:09.606 INFO:teuthology.orchestra.run.smithi097.stdout: "client": "client.35813", 2024-03-11T21:49:09.606 INFO:teuthology.orchestra.run.smithi097.stdout: "client_addr": "v1:172.21.15.92:0/4293352030", 2024-03-11T21:49:09.606 INFO:teuthology.orchestra.run.smithi097.stdout: "tid": 189901 2024-03-11T21:49:09.606 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:49:09.606 INFO:teuthology.orchestra.run.smithi097.stdout: "events": [ 2024-03-11T21:49:09.606 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:49:09.606 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "initiated", 2024-03-11T21:49:09.606 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:47:15.973467+0000", 2024-03-11T21:49:09.607 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-03-11T21:49:09.607 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:49:09.607 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:49:09.607 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "header_read", 2024-03-11T21:49:09.607 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:47:15.973467+0000", 2024-03-11T21:49:09.607 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-03-11T21:49:09.607 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:49:09.608 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:49:09.608 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "throttled", 2024-03-11T21:49:09.608 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:47:15.973467+0000", 2024-03-11T21:49:09.608 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 9.6700000000000002e-07 2024-03-11T21:49:09.608 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:49:09.608 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:49:09.608 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "all_read", 2024-03-11T21:49:09.608 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:47:15.973470+0000", 2024-03-11T21:49:09.608 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 2.9189999999999999e-06 2024-03-11T21:49:09.608 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:49:09.608 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:49:09.608 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "dispatched", 2024-03-11T21:49:09.608 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:47:15.973478+0000", 2024-03-11T21:49:09.608 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 7.1960000000000002e-06 2024-03-11T21:49:09.609 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:49:09.609 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:49:09.609 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "queued_for_pg", 2024-03-11T21:49:09.609 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:47:15.973482+0000", 2024-03-11T21:49:09.609 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 4.4159999999999997e-06 2024-03-11T21:49:09.609 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:49:09.609 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:49:09.609 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "reached_pg", 2024-03-11T21:49:09.609 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:47:15.973515+0000", 2024-03-11T21:49:09.609 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 3.2532e-05 2024-03-11T21:49:09.609 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:49:09.609 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:49:09.609 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "started", 2024-03-11T21:49:09.609 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:47:15.973594+0000", 2024-03-11T21:49:09.609 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 7.9758000000000004e-05 2024-03-11T21:49:09.610 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:49:09.610 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:49:09.610 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "waiting for subops from 1,7", 2024-03-11T21:49:09.610 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:47:15.973667+0000", 2024-03-11T21:49:09.610 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 7.3120000000000005e-05 2024-03-11T21:49:09.610 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:49:09.610 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:49:09.610 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "op_commit", 2024-03-11T21:49:09.610 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:47:15.975340+0000", 2024-03-11T21:49:09.610 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.0016726709999999999 2024-03-11T21:49:09.610 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:49:09.610 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:49:09.610 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "sub_op_commit_rec", 2024-03-11T21:49:09.610 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:47:15.975496+0000", 2024-03-11T21:49:09.611 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.00015597000000000001 2024-03-11T21:49:09.611 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:49:09.611 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:49:09.611 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "sub_op_commit_rec", 2024-03-11T21:49:09.611 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:47:16.207847+0000", 2024-03-11T21:49:09.611 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.23235088300000001 2024-03-11T21:49:09.611 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:49:09.611 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:49:09.611 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "commit_sent", 2024-03-11T21:49:09.611 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:47:16.207905+0000", 2024-03-11T21:49:09.611 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 5.8533999999999999e-05 2024-03-11T21:49:09.611 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:49:09.611 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:49:09.611 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "done", 2024-03-11T21:49:09.611 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:47:16.207949+0000", 2024-03-11T21:49:09.612 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 4.3102000000000002e-05 2024-03-11T21:49:09.612 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-03-11T21:49:09.612 INFO:teuthology.orchestra.run.smithi097.stdout: ] 2024-03-11T21:49:09.612 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-03-11T21:49:09.612 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-03-11T21:49:09.612 INFO:teuthology.orchestra.run.smithi097.stdout: ] 2024-03-11T21:49:09.612 INFO:teuthology.orchestra.run.smithi097.stdout:} 2024-03-11T21:49:09.835 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:49:09 smithi097 ceph-mon[33317]: 2.b scrub starts 2024-03-11T21:49:09.835 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:49:09 smithi097 ceph-mon[33317]: 2.b scrub ok 2024-03-11T21:49:09.835 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:49:09 smithi097 ceph-mon[33317]: 6.1 scrub starts 2024-03-11T21:49:09.835 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:49:09 smithi097 ceph-mon[33317]: 6.1 scrub ok 2024-03-11T21:49:09.835 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:49:09 smithi097 ceph-mon[33317]: from='mgr.14607 v1:172.21.15.92:0/1702717597' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T21:49:09.836 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:49:09 smithi097 ceph-mon[33317]: from='mgr.14607 v1:172.21.15.92:0/1702717597' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T21:49:09.836 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:49:09 smithi097 ceph-mon[33317]: from='mgr.14607 ' entity='mgr.y' 2024-03-11T21:49:10.119 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:49:09 smithi092 ceph-mon[37563]: 2.b scrub starts 2024-03-11T21:49:10.119 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:49:09 smithi092 ceph-mon[37563]: 2.b scrub ok 2024-03-11T21:49:10.120 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:49:09 smithi092 ceph-mon[37563]: 6.1 scrub starts 2024-03-11T21:49:10.120 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:49:09 smithi092 ceph-mon[37563]: 6.1 scrub ok 2024-03-11T21:49:10.120 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:49:09 smithi092 ceph-mon[37563]: from='mgr.14607 v1:172.21.15.92:0/1702717597' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T21:49:10.120 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:49:09 smithi092 ceph-mon[37563]: from='mgr.14607 v1:172.21.15.92:0/1702717597' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T21:49:10.120 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:49:09 smithi092 ceph-mon[37563]: from='mgr.14607 ' entity='mgr.y' 2024-03-11T21:49:10.120 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:49:09 smithi092 ceph-mon[29572]: 2.b scrub starts 2024-03-11T21:49:10.120 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:49:09 smithi092 ceph-mon[29572]: 2.b scrub ok 2024-03-11T21:49:10.120 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:49:09 smithi092 ceph-mon[29572]: 6.1 scrub starts 2024-03-11T21:49:10.120 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:49:09 smithi092 ceph-mon[29572]: 6.1 scrub ok 2024-03-11T21:49:10.120 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:49:09 smithi092 ceph-mon[29572]: from='mgr.14607 v1:172.21.15.92:0/1702717597' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T21:49:10.120 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:49:09 smithi092 ceph-mon[29572]: from='mgr.14607 v1:172.21.15.92:0/1702717597' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T21:49:10.121 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:49:09 smithi092 ceph-mon[29572]: from='mgr.14607 ' entity='mgr.y' 2024-03-11T21:49:10.168 INFO:teuthology.orchestra.run.smithi092.stdout:{"pg_ready":true,"pg_map":{"version":1123,"stamp":"2024-03-11T21:49:08.606152+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":596392,"num_objects":198,"num_object_clones":0,"num_object_copies":594,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":198,"num_whiteouts":0,"num_read":2239,"num_read_kb":2087,"num_write":1187,"num_write_kb":2461,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":497,"num_bytes_recovered":2964363,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1310,"ondisk_log_size":1310,"up":396,"acting":396,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":396,"num_osds":6,"num_per_pool_osds":6,"num_per_pool_omap_osds":6,"kb":562470912,"kb_used":5519164,"kb_used_data":7908,"kb_used_omap":9,"kb_used_meta":5511158,"kb_avail":556951748,"statfs":{"total":575970213888,"available":570318589952,"internally_reserved":0,"allocated":8097792,"data_stored":4997670,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":9564,"internal_metadata":5643426468},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":10,"num_read_kb":10,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"5.078678"},"pg_stats":[{"pgid":"6.1b","version":"0'0","reported_seq":611,"reported_epoch":523,"state":"active+clean","last_fresh":"2024-03-11T21:49:07.169911+0000","last_change":"2024-03-11T21:48:32.687207+0000","last_active":"2024-03-11T21:49:07.169911+0000","last_peered":"2024-03-11T21:49:07.169911+0000","last_clean":"2024-03-11T21:49:07.169911+0000","last_became_active":"2024-03-11T21:44:21.459293+0000","last_became_peered":"2024-03-11T21:44:21.459293+0000","last_unstale":"2024-03-11T21:49:07.169911+0000","last_undegraded":"2024-03-11T21:49:07.169911+0000","last_fullsized":"2024-03-11T21:49:07.169911+0000","mapping_epoch":432,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":433,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:48:32.687100+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:43.559452+0000","last_clean_scrub_stamp":"2024-03-11T21:48:32.687100+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:49:42.615228+0000","scrub_duration":110,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,1],"acting":[7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":197,"reported_epoch":522,"state":"active+clean","last_fresh":"2024-03-11T21:49:05.634948+0000","last_change":"2024-03-11T21:48:40.419493+0000","last_active":"2024-03-11T21:49:05.634948+0000","last_peered":"2024-03-11T21:49:05.634948+0000","last_clean":"2024-03-11T21:49:05.634948+0000","last_became_active":"2024-03-11T21:44:35.624802+0000","last_became_peered":"2024-03-11T21:44:35.624802+0000","last_unstale":"2024-03-11T21:49:05.634948+0000","last_undegraded":"2024-03-11T21:49:05.634948+0000","last_fullsized":"2024-03-11T21:49:05.634948+0000","mapping_epoch":437,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":438,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:48:40.419361+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:41.544550+0000","last_clean_scrub_stamp":"2024-03-11T21:48:40.419361+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:49:50.714093+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,2],"acting":[6,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.1f","version":"0'0","reported_seq":1012,"reported_epoch":521,"state":"active+clean","last_fresh":"2024-03-11T21:49:03.632459+0000","last_change":"2024-03-11T21:45:34.145990+0000","last_active":"2024-03-11T21:49:03.632459+0000","last_peered":"2024-03-11T21:49:03.632459+0000","last_clean":"2024-03-11T21:49:03.632459+0000","last_became_active":"2024-03-11T21:44:35.644794+0000","last_became_peered":"2024-03-11T21:44:35.644794+0000","last_unstale":"2024-03-11T21:49:03.632459+0000","last_undegraded":"2024-03-11T21:49:03.632459+0000","last_fullsized":"2024-03-11T21:49:03.632459+0000","mapping_epoch":437,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":438,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:45:34.145740+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:37.512018+0000","last_clean_scrub_stamp":"2024-03-11T21:45:34.145740+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,6],"acting":[0,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.19","version":"456'39","reported_seq":403,"reported_epoch":523,"state":"active+clean","last_fresh":"2024-03-11T21:49:06.638638+0000","last_change":"2024-03-11T21:48:50.702062+0000","last_active":"2024-03-11T21:49:06.638638+0000","last_peered":"2024-03-11T21:49:06.638638+0000","last_clean":"2024-03-11T21:49:06.638638+0000","last_became_active":"2024-03-11T21:48:28.627471+0000","last_became_peered":"2024-03-11T21:48:28.627471+0000","last_unstale":"2024-03-11T21:49:06.638638+0000","last_undegraded":"2024-03-11T21:49:06.638638+0000","last_fullsized":"2024-03-11T21:49:06.638638+0000","mapping_epoch":502,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":503,"parent":"0.0","parent_split_bits":0,"last_scrub":"456'39","last_scrub_stamp":"2024-03-11T21:48:50.701931+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:39.525554+0000","last_clean_scrub_stamp":"2024-03-11T21:48:50.701931+0000","objects_scrubbed":7,"log_size":39,"log_dups_size":0,"ondisk_log_size":39,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:50:20.674160+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":61,"num_read_kb":52,"num_write":40,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":21,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,5],"acting":[2,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.1a","version":"0'0","reported_seq":59,"reported_epoch":523,"state":"active+clean","last_fresh":"2024-03-11T21:49:06.863652+0000","last_change":"2024-03-11T21:49:02.060501+0000","last_active":"2024-03-11T21:49:06.863652+0000","last_peered":"2024-03-11T21:49:06.863652+0000","last_clean":"2024-03-11T21:49:06.863652+0000","last_became_active":"2024-03-11T21:48:28.636130+0000","last_became_peered":"2024-03-11T21:48:28.636130+0000","last_unstale":"2024-03-11T21:49:06.863652+0000","last_undegraded":"2024-03-11T21:49:06.863652+0000","last_fullsized":"2024-03-11T21:49:06.863652+0000","mapping_epoch":502,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":503,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:49:02.060422+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:49:02.060422+0000","last_clean_scrub_stamp":"2024-03-11T21:49:02.060422+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:50:31.070186+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,7],"acting":[5,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":51,"reported_epoch":523,"state":"active+clean","last_fresh":"2024-03-11T21:49:06.863592+0000","last_change":"2024-03-11T21:48:28.635845+0000","last_active":"2024-03-11T21:49:06.863592+0000","last_peered":"2024-03-11T21:49:06.863592+0000","last_clean":"2024-03-11T21:49:06.863592+0000","last_became_active":"2024-03-11T21:48:28.635475+0000","last_became_peered":"2024-03-11T21:48:28.635475+0000","last_unstale":"2024-03-11T21:49:06.863592+0000","last_undegraded":"2024-03-11T21:49:06.863592+0000","last_fullsized":"2024-03-11T21:49:06.863592+0000","mapping_epoch":502,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":503,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:46:29.023645+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:41.544550+0000","last_clean_scrub_stamp":"2024-03-11T21:46:29.023645+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,7],"acting":[5,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1e","version":"0'0","reported_seq":627,"reported_epoch":521,"state":"active+clean","last_fresh":"2024-03-11T21:49:03.632202+0000","last_change":"2024-03-11T21:48:28.630141+0000","last_active":"2024-03-11T21:49:03.632202+0000","last_peered":"2024-03-11T21:49:03.632202+0000","last_clean":"2024-03-11T21:49:03.632202+0000","last_became_active":"2024-03-11T21:48:28.629741+0000","last_became_peered":"2024-03-11T21:48:28.629741+0000","last_unstale":"2024-03-11T21:49:03.632202+0000","last_undegraded":"2024-03-11T21:49:03.632202+0000","last_fullsized":"2024-03-11T21:49:03.632202+0000","mapping_epoch":502,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":503,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:48:21.902034+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:37.512018+0000","last_clean_scrub_stamp":"2024-03-11T21:48:21.902034+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:49:25.942096+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.18","version":"456'21","reported_seq":668,"reported_epoch":522,"state":"active+clean","last_fresh":"2024-03-11T21:49:05.635077+0000","last_change":"2024-03-11T21:45:24.708559+0000","last_active":"2024-03-11T21:49:05.635077+0000","last_peered":"2024-03-11T21:49:05.635077+0000","last_clean":"2024-03-11T21:49:05.635077+0000","last_became_active":"2024-03-11T21:44:35.621442+0000","last_became_peered":"2024-03-11T21:44:35.621442+0000","last_unstale":"2024-03-11T21:49:05.635077+0000","last_undegraded":"2024-03-11T21:49:05.635077+0000","last_fullsized":"2024-03-11T21:49:05.635077+0000","mapping_epoch":437,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":438,"parent":"0.0","parent_split_bits":0,"last_scrub":"260'15","last_scrub_stamp":"2024-03-11T21:45:24.708228+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:39.525554+0000","last_clean_scrub_stamp":"2024-03-11T21:45:24.708228+0000","objects_scrubbed":4,"log_size":21,"log_dups_size":0,"ondisk_log_size":21,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":34,"num_read_kb":28,"num_write":22,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,2],"acting":[6,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":51,"reported_epoch":523,"state":"active+clean","last_fresh":"2024-03-11T21:49:06.863632+0000","last_change":"2024-03-11T21:48:28.637297+0000","last_active":"2024-03-11T21:49:06.863632+0000","last_peered":"2024-03-11T21:49:06.863632+0000","last_clean":"2024-03-11T21:49:06.863632+0000","last_became_active":"2024-03-11T21:48:28.636959+0000","last_became_peered":"2024-03-11T21:48:28.636959+0000","last_unstale":"2024-03-11T21:49:06.863632+0000","last_undegraded":"2024-03-11T21:49:06.863632+0000","last_fullsized":"2024-03-11T21:49:06.863632+0000","mapping_epoch":502,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":503,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:47:05.896361+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:32:12.654573+0000","last_clean_scrub_stamp":"2024-03-11T21:47:05.896361+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":83,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,2],"acting":[5,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":645,"reported_epoch":523,"state":"active+clean","last_fresh":"2024-03-11T21:49:07.170096+0000","last_change":"2024-03-11T21:48:22.571238+0000","last_active":"2024-03-11T21:49:07.170096+0000","last_peered":"2024-03-11T21:49:07.170096+0000","last_clean":"2024-03-11T21:49:07.170096+0000","last_became_active":"2024-03-11T21:44:35.626456+0000","last_became_peered":"2024-03-11T21:44:35.626456+0000","last_unstale":"2024-03-11T21:49:07.170096+0000","last_undegraded":"2024-03-11T21:49:07.170096+0000","last_fullsized":"2024-03-11T21:49:07.170096+0000","mapping_epoch":437,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":438,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:48:22.571126+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:41.544550+0000","last_clean_scrub_stamp":"2024-03-11T21:48:22.571126+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:49:47.345185+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,0],"acting":[7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.1d","version":"0'0","reported_seq":1002,"reported_epoch":523,"state":"active+clean","last_fresh":"2024-03-11T21:49:07.170495+0000","last_change":"2024-03-11T21:46:28.543345+0000","last_active":"2024-03-11T21:49:07.170495+0000","last_peered":"2024-03-11T21:49:07.170495+0000","last_clean":"2024-03-11T21:49:07.170495+0000","last_became_active":"2024-03-11T21:25:38.542628+0000","last_became_peered":"2024-03-11T21:25:38.542628+0000","last_unstale":"2024-03-11T21:49:07.170495+0000","last_undegraded":"2024-03-11T21:49:07.170495+0000","last_fullsized":"2024-03-11T21:49:07.170495+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:46:28.543232+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:37.512018+0000","last_clean_scrub_stamp":"2024-03-11T21:46:28.543232+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,0],"acting":[7,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.1b","version":"456'9","reported_seq":639,"reported_epoch":521,"state":"active+clean","last_fresh":"2024-03-11T21:49:03.824315+0000","last_change":"2024-03-11T21:46:08.932984+0000","last_active":"2024-03-11T21:49:03.824315+0000","last_peered":"2024-03-11T21:49:03.824315+0000","last_clean":"2024-03-11T21:49:03.824315+0000","last_became_active":"2024-03-11T21:44:35.639372+0000","last_became_peered":"2024-03-11T21:44:35.639372+0000","last_unstale":"2024-03-11T21:49:03.824315+0000","last_undegraded":"2024-03-11T21:49:03.824315+0000","last_fullsized":"2024-03-11T21:49:03.824315+0000","mapping_epoch":437,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":438,"parent":"0.0","parent_split_bits":0,"last_scrub":"456'9","last_scrub_stamp":"2024-03-11T21:46:08.932675+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:39.525554+0000","last_clean_scrub_stamp":"2024-03-11T21:46:08.932675+0000","objects_scrubbed":2,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":16,"num_read_kb":12,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,7],"acting":[1,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":972,"reported_epoch":521,"state":"active+clean","last_fresh":"2024-03-11T21:49:03.631567+0000","last_change":"2024-03-11T21:45:32.208394+0000","last_active":"2024-03-11T21:49:03.631567+0000","last_peered":"2024-03-11T21:49:03.631567+0000","last_clean":"2024-03-11T21:49:03.631567+0000","last_became_active":"2024-03-11T21:25:44.589786+0000","last_became_peered":"2024-03-11T21:25:44.589786+0000","last_unstale":"2024-03-11T21:49:03.631567+0000","last_undegraded":"2024-03-11T21:49:03.631567+0000","last_fullsized":"2024-03-11T21:49:03.631567+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:45:32.208161+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:43.559452+0000","last_clean_scrub_stamp":"2024-03-11T21:45:32.208161+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,7],"acting":[0,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":51,"reported_epoch":523,"state":"active+clean","last_fresh":"2024-03-11T21:49:06.864012+0000","last_change":"2024-03-11T21:48:28.644627+0000","last_active":"2024-03-11T21:49:06.864012+0000","last_peered":"2024-03-11T21:49:06.864012+0000","last_clean":"2024-03-11T21:49:06.864012+0000","last_became_active":"2024-03-11T21:48:28.644231+0000","last_became_peered":"2024-03-11T21:48:28.644231+0000","last_unstale":"2024-03-11T21:49:06.864012+0000","last_undegraded":"2024-03-11T21:49:06.864012+0000","last_fullsized":"2024-03-11T21:49:06.864012+0000","mapping_epoch":502,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":503,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:46:13.862444+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:43:35.536164+0000","last_clean_scrub_stamp":"2024-03-11T21:46:13.862444+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,7],"acting":[5,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1c","version":"0'0","reported_seq":1014,"reported_epoch":523,"state":"active+clean","last_fresh":"2024-03-11T21:49:07.170451+0000","last_change":"2024-03-11T21:48:57.842300+0000","last_active":"2024-03-11T21:49:07.170451+0000","last_peered":"2024-03-11T21:49:07.170451+0000","last_clean":"2024-03-11T21:49:07.170451+0000","last_became_active":"2024-03-11T21:48:28.628707+0000","last_became_peered":"2024-03-11T21:48:28.628707+0000","last_unstale":"2024-03-11T21:49:07.170451+0000","last_undegraded":"2024-03-11T21:49:07.170451+0000","last_fullsized":"2024-03-11T21:49:07.170451+0000","mapping_epoch":502,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":503,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:48:57.842218+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:28:32.566437+0000","last_clean_scrub_stamp":"2024-03-11T21:48:57.842218+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:49:58.846988+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.1a","version":"456'25","reported_seq":870,"reported_epoch":521,"state":"active+clean","last_fresh":"2024-03-11T21:49:03.631648+0000","last_change":"2024-03-11T21:46:06.451173+0000","last_active":"2024-03-11T21:49:03.631648+0000","last_peered":"2024-03-11T21:49:03.631648+0000","last_clean":"2024-03-11T21:49:03.631648+0000","last_became_active":"2024-03-11T21:44:35.631905+0000","last_became_peered":"2024-03-11T21:44:35.631905+0000","last_unstale":"2024-03-11T21:49:03.631648+0000","last_undegraded":"2024-03-11T21:49:03.631648+0000","last_fullsized":"2024-03-11T21:49:03.631648+0000","mapping_epoch":437,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":438,"parent":"0.0","parent_split_bits":0,"last_scrub":"456'25","last_scrub_stamp":"2024-03-11T21:46:06.451051+0000","last_deep_scrub":"350'19","last_deep_scrub_stamp":"2024-03-11T21:43:25.898708+0000","last_clean_scrub_stamp":"2024-03-11T21:46:06.451051+0000","objects_scrubbed":4,"log_size":25,"log_dups_size":0,"ondisk_log_size":25,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":35,"num_read_kb":29,"num_write":22,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,6],"acting":[0,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.1f","version":"0'0","reported_seq":67,"reported_epoch":521,"state":"active+clean","last_fresh":"2024-03-11T21:49:03.631854+0000","last_change":"2024-03-11T21:48:28.642776+0000","last_active":"2024-03-11T21:49:03.631854+0000","last_peered":"2024-03-11T21:49:03.631854+0000","last_clean":"2024-03-11T21:49:03.631854+0000","last_became_active":"2024-03-11T21:48:28.641925+0000","last_became_peered":"2024-03-11T21:48:28.641925+0000","last_unstale":"2024-03-11T21:49:03.631854+0000","last_undegraded":"2024-03-11T21:49:03.631854+0000","last_fullsized":"2024-03-11T21:49:03.631854+0000","mapping_epoch":502,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":503,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:48:25.151073+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:44:31.472397+0000","last_clean_scrub_stamp":"2024-03-11T21:48:25.151073+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:49:36.645328+0000","scrub_duration":193,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,1],"acting":[0,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.1c","version":"0'0","reported_seq":995,"reported_epoch":523,"state":"active+clean","last_fresh":"2024-03-11T21:49:06.639481+0000","last_change":"2024-03-11T21:48:36.643506+0000","last_active":"2024-03-11T21:49:06.639481+0000","last_peered":"2024-03-11T21:49:06.639481+0000","last_clean":"2024-03-11T21:49:06.639481+0000","last_became_active":"2024-03-11T21:44:35.624752+0000","last_became_peered":"2024-03-11T21:44:35.624752+0000","last_unstale":"2024-03-11T21:49:06.639481+0000","last_undegraded":"2024-03-11T21:49:06.639481+0000","last_fullsized":"2024-03-11T21:49:06.639481+0000","mapping_epoch":437,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":438,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:48:36.643362+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:43:32.021344+0000","last_clean_scrub_stamp":"2024-03-11T21:48:36.643362+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:49:41.697242+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,1],"acting":[2,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1b","version":"0'0","reported_seq":648,"reported_epoch":523,"state":"active+clean","last_fresh":"2024-03-11T21:49:07.169964+0000","last_change":"2024-03-11T21:49:05.759673+0000","last_active":"2024-03-11T21:49:07.169964+0000","last_peered":"2024-03-11T21:49:07.169964+0000","last_clean":"2024-03-11T21:49:07.169964+0000","last_became_active":"2024-03-11T21:48:28.626684+0000","last_became_peered":"2024-03-11T21:48:28.626684+0000","last_unstale":"2024-03-11T21:49:07.169964+0000","last_undegraded":"2024-03-11T21:49:07.169964+0000","last_fullsized":"2024-03-11T21:49:07.169964+0000","mapping_epoch":502,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":503,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:49:05.759541+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:34:27.998724+0000","last_clean_scrub_stamp":"2024-03-11T21:49:05.759541+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:50:12.023817+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,5],"acting":[7,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.1d","version":"456'24","reported_seq":385,"reported_epoch":521,"state":"active+clean","last_fresh":"2024-03-11T21:49:03.825092+0000","last_change":"2024-03-11T21:48:40.889498+0000","last_active":"2024-03-11T21:49:03.825092+0000","last_peered":"2024-03-11T21:49:03.825092+0000","last_clean":"2024-03-11T21:49:03.825092+0000","last_became_active":"2024-03-11T21:48:28.621370+0000","last_became_peered":"2024-03-11T21:48:28.621370+0000","last_unstale":"2024-03-11T21:49:03.825092+0000","last_undegraded":"2024-03-11T21:49:03.825092+0000","last_fullsized":"2024-03-11T21:49:03.825092+0000","mapping_epoch":502,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":503,"parent":"0.0","parent_split_bits":0,"last_scrub":"456'24","last_scrub_stamp":"2024-03-11T21:48:40.889386+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:39.525554+0000","last_clean_scrub_stamp":"2024-03-11T21:48:40.889386+0000","objects_scrubbed":5,"log_size":24,"log_dups_size":0,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:50:04.807403+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":42,"num_read_kb":33,"num_write":26,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":880,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,5],"acting":[1,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1e","version":"0'0","reported_seq":210,"reported_epoch":522,"state":"active+clean","last_fresh":"2024-03-11T21:49:05.635934+0000","last_change":"2024-03-11T21:48:28.642136+0000","last_active":"2024-03-11T21:49:05.635934+0000","last_peered":"2024-03-11T21:49:05.635934+0000","last_clean":"2024-03-11T21:49:05.635934+0000","last_became_active":"2024-03-11T21:48:28.641687+0000","last_became_peered":"2024-03-11T21:48:28.641687+0000","last_unstale":"2024-03-11T21:49:05.635934+0000","last_undegraded":"2024-03-11T21:49:05.635934+0000","last_fullsized":"2024-03-11T21:49:05.635934+0000","mapping_epoch":502,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":503,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:48:23.394601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:43.559452+0000","last_clean_scrub_stamp":"2024-03-11T21:48:23.394601+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:49:52.770231+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,0],"acting":[6,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.1d","version":"0'0","reported_seq":1005,"reported_epoch":521,"state":"active+clean","last_fresh":"2024-03-11T21:49:03.825115+0000","last_change":"2024-03-11T21:48:56.927970+0000","last_active":"2024-03-11T21:49:03.825115+0000","last_peered":"2024-03-11T21:49:03.825115+0000","last_clean":"2024-03-11T21:49:03.825115+0000","last_became_active":"2024-03-11T21:48:28.621503+0000","last_became_peered":"2024-03-11T21:48:28.621503+0000","last_unstale":"2024-03-11T21:49:03.825115+0000","last_undegraded":"2024-03-11T21:49:03.825115+0000","last_fullsized":"2024-03-11T21:49:03.825115+0000","mapping_epoch":502,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":503,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:48:56.927890+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:30:18.678531+0000","last_clean_scrub_stamp":"2024-03-11T21:48:56.927890+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:50:22.010957+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,5],"acting":[1,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1a","version":"0'0","reported_seq":1028,"reported_epoch":522,"state":"active+clean","last_fresh":"2024-03-11T21:49:05.635210+0000","last_change":"2024-03-11T21:48:57.428298+0000","last_active":"2024-03-11T21:49:05.635210+0000","last_peered":"2024-03-11T21:49:05.635210+0000","last_clean":"2024-03-11T21:49:05.635210+0000","last_became_active":"2024-03-11T21:44:35.630761+0000","last_became_peered":"2024-03-11T21:44:35.630761+0000","last_unstale":"2024-03-11T21:49:05.635210+0000","last_undegraded":"2024-03-11T21:49:05.635210+0000","last_fullsized":"2024-03-11T21:49:05.635210+0000","mapping_epoch":437,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":438,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:48:57.428222+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:37.512018+0000","last_clean_scrub_stamp":"2024-03-11T21:48:57.428222+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:50:02.779782+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,2],"acting":[6,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.1c","version":"456'39","reported_seq":1097,"reported_epoch":523,"state":"active+clean","last_fresh":"2024-03-11T21:49:06.639648+0000","last_change":"2024-03-11T21:48:51.712325+0000","last_active":"2024-03-11T21:49:06.639648+0000","last_peered":"2024-03-11T21:49:06.639648+0000","last_clean":"2024-03-11T21:49:06.639648+0000","last_became_active":"2024-03-11T21:44:21.465719+0000","last_became_peered":"2024-03-11T21:44:21.465719+0000","last_unstale":"2024-03-11T21:49:06.639648+0000","last_undegraded":"2024-03-11T21:49:06.639648+0000","last_fullsized":"2024-03-11T21:49:06.639648+0000","mapping_epoch":432,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":433,"parent":"0.0","parent_split_bits":0,"last_scrub":"456'39","last_scrub_stamp":"2024-03-11T21:48:51.712213+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:39.525554+0000","last_clean_scrub_stamp":"2024-03-11T21:48:51.712213+0000","objects_scrubbed":7,"log_size":39,"log_dups_size":0,"ondisk_log_size":39,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:50:09.639358+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":62,"num_read_kb":53,"num_write":40,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,0],"acting":[2,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.1d","version":"0'0","reported_seq":51,"reported_epoch":523,"state":"active+clean","last_fresh":"2024-03-11T21:49:06.864180+0000","last_change":"2024-03-11T21:48:28.637584+0000","last_active":"2024-03-11T21:49:06.864180+0000","last_peered":"2024-03-11T21:49:06.864180+0000","last_clean":"2024-03-11T21:49:06.864180+0000","last_became_active":"2024-03-11T21:48:28.636077+0000","last_became_peered":"2024-03-11T21:48:28.636077+0000","last_unstale":"2024-03-11T21:49:06.864180+0000","last_undegraded":"2024-03-11T21:49:06.864180+0000","last_fullsized":"2024-03-11T21:49:06.864180+0000","mapping_epoch":502,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":503,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:48:23.751807+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:43.559452+0000","last_clean_scrub_stamp":"2024-03-11T21:48:23.751807+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:49:30.913722+0000","scrub_duration":106,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,2],"acting":[5,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.1e","version":"0'0","reported_seq":986,"reported_epoch":521,"state":"active+clean","last_fresh":"2024-03-11T21:49:03.632301+0000","last_change":"2024-03-11T21:46:15.490825+0000","last_active":"2024-03-11T21:49:03.632301+0000","last_peered":"2024-03-11T21:49:03.632301+0000","last_clean":"2024-03-11T21:49:03.632301+0000","last_became_active":"2024-03-11T21:25:42.582959+0000","last_became_peered":"2024-03-11T21:25:42.582959+0000","last_unstale":"2024-03-11T21:49:03.632301+0000","last_undegraded":"2024-03-11T21:49:03.632301+0000","last_fullsized":"2024-03-11T21:49:03.632301+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:46:15.490533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:44:28.222102+0000","last_clean_scrub_stamp":"2024-03-11T21:46:15.490533+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.19","version":"67'2","reported_seq":277,"reported_epoch":522,"state":"active+clean","last_fresh":"2024-03-11T21:49:05.634978+0000","last_change":"2024-03-11T21:48:12.501989+0000","last_active":"2024-03-11T21:49:05.634978+0000","last_peered":"2024-03-11T21:49:05.634978+0000","last_clean":"2024-03-11T21:49:05.634978+0000","last_became_active":"2024-03-11T21:44:21.454547+0000","last_became_peered":"2024-03-11T21:44:21.454547+0000","last_unstale":"2024-03-11T21:49:05.634978+0000","last_undegraded":"2024-03-11T21:49:05.634978+0000","last_fullsized":"2024-03-11T21:49:05.634978+0000","mapping_epoch":432,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":433,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'2","last_scrub_stamp":"2024-03-11T21:48:12.501849+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:37.512018+0000","last_clean_scrub_stamp":"2024-03-11T21:48:12.501849+0000","objects_scrubbed":2,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:49:16.110129+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":957,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":957,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,2],"acting":[6,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.1f","version":"456'32","reported_seq":1110,"reported_epoch":522,"state":"active+clean","last_fresh":"2024-03-11T21:49:05.635822+0000","last_change":"2024-03-11T21:48:28.679601+0000","last_active":"2024-03-11T21:49:05.635822+0000","last_peered":"2024-03-11T21:49:05.635822+0000","last_clean":"2024-03-11T21:49:05.635822+0000","last_became_active":"2024-03-11T21:48:28.643699+0000","last_became_peered":"2024-03-11T21:48:28.643699+0000","last_unstale":"2024-03-11T21:49:05.635822+0000","last_undegraded":"2024-03-11T21:49:05.635822+0000","last_fullsized":"2024-03-11T21:49:05.635822+0000","mapping_epoch":502,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":503,"parent":"0.0","parent_split_bits":0,"last_scrub":"456'32","last_scrub_stamp":"2024-03-11T21:46:16.737252+0000","last_deep_scrub":"124'16","last_deep_scrub_stamp":"2024-03-11T21:35:38.173841+0000","last_clean_scrub_stamp":"2024-03-11T21:46:16.737252+0000","objects_scrubbed":5,"log_size":32,"log_dups_size":0,"ondisk_log_size":32,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":44,"num_read_kb":37,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,0],"acting":[6,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.1c","version":"467'2","reported_seq":656,"reported_epoch":523,"state":"active+clean","last_fresh":"2024-03-11T21:49:07.170428+0000","last_change":"2024-03-11T21:48:28.651514+0000","last_active":"2024-03-11T21:49:07.170428+0000","last_peered":"2024-03-11T21:49:07.170428+0000","last_clean":"2024-03-11T21:49:07.170428+0000","last_became_active":"2024-03-11T21:48:28.628768+0000","last_became_peered":"2024-03-11T21:48:28.628768+0000","last_unstale":"2024-03-11T21:49:07.170428+0000","last_undegraded":"2024-03-11T21:49:07.170428+0000","last_fullsized":"2024-03-11T21:49:07.170428+0000","mapping_epoch":502,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":503,"parent":"0.0","parent_split_bits":0,"last_scrub":"467'2","last_scrub_stamp":"2024-03-11T21:46:22.517102+0000","last_deep_scrub":"467'2","last_deep_scrub_stamp":"2024-03-11T21:46:22.517102+0000","last_clean_scrub_stamp":"2024-03-11T21:46:22.517102+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":370,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":1110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.1f","version":"438'11","reported_seq":1336,"reported_epoch":522,"state":"active+clean","last_fresh":"2024-03-11T21:49:05.635774+0000","last_change":"2024-03-11T21:48:15.506962+0000","last_active":"2024-03-11T21:49:05.635774+0000","last_peered":"2024-03-11T21:49:05.635774+0000","last_clean":"2024-03-11T21:49:05.635774+0000","last_became_active":"2024-03-11T21:44:35.647069+0000","last_became_peered":"2024-03-11T21:44:35.647069+0000","last_unstale":"2024-03-11T21:49:05.635774+0000","last_undegraded":"2024-03-11T21:49:05.635774+0000","last_fullsized":"2024-03-11T21:49:05.635774+0000","mapping_epoch":437,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":438,"parent":"0.0","parent_split_bits":0,"last_scrub":"438'11","last_scrub_stamp":"2024-03-11T21:48:15.506841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:41.544550+0000","last_clean_scrub_stamp":"2024-03-11T21:48:15.506841+0000","objects_scrubbed":1,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:49:32.316775+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,2],"acting":[6,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.18","version":"67'1","reported_seq":63,"reported_epoch":523,"state":"active+clean","last_fresh":"2024-03-11T21:49:06.863438+0000","last_change":"2024-03-11T21:48:37.974397+0000","last_active":"2024-03-11T21:49:06.863438+0000","last_peered":"2024-03-11T21:49:06.863438+0000","last_clean":"2024-03-11T21:49:06.863438+0000","last_became_active":"2024-03-11T21:48:28.640756+0000","last_became_peered":"2024-03-11T21:48:28.640756+0000","last_unstale":"2024-03-11T21:49:06.863438+0000","last_undegraded":"2024-03-11T21:49:06.863438+0000","last_fullsized":"2024-03-11T21:49:06.863438+0000","mapping_epoch":502,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":503,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'1","last_scrub_stamp":"2024-03-11T21:48:37.974310+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:37.512018+0000","last_clean_scrub_stamp":"2024-03-11T21:48:37.974310+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:49:58.741849+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":436,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":7,"num_read_kb":7,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":1308,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,6],"acting":[5,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1e","version":"456'30","reported_seq":1111,"reported_epoch":521,"state":"active+clean","last_fresh":"2024-03-11T21:49:03.632300+0000","last_change":"2024-03-11T21:48:40.055060+0000","last_active":"2024-03-11T21:49:03.632300+0000","last_peered":"2024-03-11T21:49:03.632300+0000","last_clean":"2024-03-11T21:49:03.632300+0000","last_became_active":"2024-03-11T21:44:21.467126+0000","last_became_peered":"2024-03-11T21:44:21.467126+0000","last_unstale":"2024-03-11T21:49:03.632300+0000","last_undegraded":"2024-03-11T21:49:03.632300+0000","last_fullsized":"2024-03-11T21:49:03.632300+0000","mapping_epoch":432,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":433,"parent":"0.0","parent_split_bits":0,"last_scrub":"456'30","last_scrub_stamp":"2024-03-11T21:48:40.054882+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:39.525554+0000","last_clean_scrub_stamp":"2024-03-11T21:48:40.054882+0000","objects_scrubbed":5,"log_size":30,"log_dups_size":0,"ondisk_log_size":30,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:50:01.724130+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":45,"num_read_kb":40,"num_write":30,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,1],"acting":[0,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.13","version":"0'0","reported_seq":546,"reported_epoch":521,"state":"active+clean","last_fresh":"2024-03-11T21:49:03.631793+0000","last_change":"2024-03-11T21:49:02.319830+0000","last_active":"2024-03-11T21:49:03.631793+0000","last_peered":"2024-03-11T21:49:03.631793+0000","last_clean":"2024-03-11T21:49:03.631793+0000","last_became_active":"2024-03-11T21:44:21.465436+0000","last_became_peered":"2024-03-11T21:44:21.465436+0000","last_unstale":"2024-03-11T21:49:03.631793+0000","last_undegraded":"2024-03-11T21:49:03.631793+0000","last_fullsized":"2024-03-11T21:49:03.631793+0000","mapping_epoch":432,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":433,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:49:02.319723+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:43.559452+0000","last_clean_scrub_stamp":"2024-03-11T21:49:02.319723+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:50:15.607012+0000","scrub_duration":149,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,2],"acting":[0,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":646,"reported_epoch":523,"state":"active+clean","last_fresh":"2024-03-11T21:49:07.169820+0000","last_change":"2024-03-11T21:46:24.563883+0000","last_active":"2024-03-11T21:49:07.169820+0000","last_peered":"2024-03-11T21:49:07.169820+0000","last_clean":"2024-03-11T21:49:07.169820+0000","last_became_active":"2024-03-11T21:44:35.627685+0000","last_became_peered":"2024-03-11T21:44:35.627685+0000","last_unstale":"2024-03-11T21:49:07.169820+0000","last_undegraded":"2024-03-11T21:49:07.169820+0000","last_fullsized":"2024-03-11T21:49:07.169820+0000","mapping_epoch":437,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":438,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:46:24.563752+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:43:26.665719+0000","last_clean_scrub_stamp":"2024-03-11T21:46:24.563752+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,0],"acting":[7,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.17","version":"0'0","reported_seq":1014,"reported_epoch":522,"state":"active+clean","last_fresh":"2024-03-11T21:49:05.635643+0000","last_change":"2024-03-11T21:48:28.624441+0000","last_active":"2024-03-11T21:49:05.635643+0000","last_peered":"2024-03-11T21:49:05.635643+0000","last_clean":"2024-03-11T21:49:05.635643+0000","last_became_active":"2024-03-11T21:48:28.623533+0000","last_became_peered":"2024-03-11T21:48:28.623533+0000","last_unstale":"2024-03-11T21:49:05.635643+0000","last_undegraded":"2024-03-11T21:49:05.635643+0000","last_fullsized":"2024-03-11T21:49:05.635643+0000","mapping_epoch":502,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":503,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:45:30.740865+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:45:30.740865+0000","last_clean_scrub_stamp":"2024-03-11T21:45:30.740865+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,2],"acting":[6,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.11","version":"456'27","reported_seq":714,"reported_epoch":523,"state":"active+clean","last_fresh":"2024-03-11T21:49:07.169853+0000","last_change":"2024-03-11T21:48:28.652986+0000","last_active":"2024-03-11T21:49:07.169853+0000","last_peered":"2024-03-11T21:49:07.169853+0000","last_clean":"2024-03-11T21:49:07.169853+0000","last_became_active":"2024-03-11T21:48:28.626324+0000","last_became_peered":"2024-03-11T21:48:28.626324+0000","last_unstale":"2024-03-11T21:49:07.169853+0000","last_undegraded":"2024-03-11T21:49:07.169853+0000","last_fullsized":"2024-03-11T21:49:07.169853+0000","mapping_epoch":502,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":503,"parent":"0.0","parent_split_bits":0,"last_scrub":"456'27","last_scrub_stamp":"2024-03-11T21:46:18.485911+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:39.525554+0000","last_clean_scrub_stamp":"2024-03-11T21:46:18.485911+0000","objects_scrubbed":5,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":44,"num_read_kb":37,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":440,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,5],"acting":[7,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":646,"reported_epoch":523,"state":"active+clean","last_fresh":"2024-03-11T21:49:07.169978+0000","last_change":"2024-03-11T21:48:36.557715+0000","last_active":"2024-03-11T21:49:07.169978+0000","last_peered":"2024-03-11T21:49:07.169978+0000","last_clean":"2024-03-11T21:49:07.169978+0000","last_became_active":"2024-03-11T21:44:35.634181+0000","last_became_peered":"2024-03-11T21:44:35.634181+0000","last_unstale":"2024-03-11T21:49:07.169978+0000","last_undegraded":"2024-03-11T21:49:07.169978+0000","last_fullsized":"2024-03-11T21:49:07.169978+0000","mapping_epoch":437,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":438,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:48:36.557613+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:43.559452+0000","last_clean_scrub_stamp":"2024-03-11T21:48:36.557613+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:49:54.779088+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,1],"acting":[7,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":1021,"reported_epoch":522,"state":"active+clean","last_fresh":"2024-03-11T21:49:05.634988+0000","last_change":"2024-03-11T21:45:39.613711+0000","last_active":"2024-03-11T21:49:05.634988+0000","last_peered":"2024-03-11T21:49:05.634988+0000","last_clean":"2024-03-11T21:49:05.634988+0000","last_became_active":"2024-03-11T21:44:35.630991+0000","last_became_peered":"2024-03-11T21:44:35.630991+0000","last_unstale":"2024-03-11T21:49:05.634988+0000","last_undegraded":"2024-03-11T21:49:05.634988+0000","last_fullsized":"2024-03-11T21:49:05.634988+0000","mapping_epoch":437,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":438,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:45:39.613600+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:43:29.425414+0000","last_clean_scrub_stamp":"2024-03-11T21:45:39.613600+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,0],"acting":[6,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.16","version":"0'0","reported_seq":59,"reported_epoch":523,"state":"active+clean","last_fresh":"2024-03-11T21:49:06.864328+0000","last_change":"2024-03-11T21:48:33.041585+0000","last_active":"2024-03-11T21:49:06.864328+0000","last_peered":"2024-03-11T21:49:06.864328+0000","last_clean":"2024-03-11T21:49:06.864328+0000","last_became_active":"2024-03-11T21:48:28.636823+0000","last_became_peered":"2024-03-11T21:48:28.636823+0000","last_unstale":"2024-03-11T21:49:06.864328+0000","last_undegraded":"2024-03-11T21:49:06.864328+0000","last_fullsized":"2024-03-11T21:49:06.864328+0000","mapping_epoch":502,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":503,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:48:33.041481+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:27:11.938909+0000","last_clean_scrub_stamp":"2024-03-11T21:48:33.041481+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:49:56.250770+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,2],"acting":[5,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.10","version":"456'14","reported_seq":328,"reported_epoch":522,"state":"active+clean","last_fresh":"2024-03-11T21:49:05.635162+0000","last_change":"2024-03-11T21:48:28.675345+0000","last_active":"2024-03-11T21:49:05.635162+0000","last_peered":"2024-03-11T21:49:05.635162+0000","last_clean":"2024-03-11T21:49:05.635162+0000","last_became_active":"2024-03-11T21:48:28.641869+0000","last_became_peered":"2024-03-11T21:48:28.641869+0000","last_unstale":"2024-03-11T21:49:05.635162+0000","last_undegraded":"2024-03-11T21:49:05.635162+0000","last_fullsized":"2024-03-11T21:49:05.635162+0000","mapping_epoch":502,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":503,"parent":"0.0","parent_split_bits":0,"last_scrub":"456'14","last_scrub_stamp":"2024-03-11T21:47:18.601648+0000","last_deep_scrub":"260'10","last_deep_scrub_stamp":"2024-03-11T21:44:32.059058+0000","last_clean_scrub_stamp":"2024-03-11T21:47:18.601648+0000","objects_scrubbed":2,"log_size":14,"log_dups_size":0,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":17,"num_read_kb":15,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,5],"acting":[6,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":603,"reported_epoch":521,"state":"active+clean","last_fresh":"2024-03-11T21:49:03.632471+0000","last_change":"2024-03-11T21:48:28.624430+0000","last_active":"2024-03-11T21:49:03.632471+0000","last_peered":"2024-03-11T21:49:03.632471+0000","last_clean":"2024-03-11T21:49:03.632471+0000","last_became_active":"2024-03-11T21:48:28.623984+0000","last_became_peered":"2024-03-11T21:48:28.623984+0000","last_unstale":"2024-03-11T21:49:03.632471+0000","last_undegraded":"2024-03-11T21:49:03.632471+0000","last_fullsized":"2024-03-11T21:49:03.632471+0000","mapping_epoch":502,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":503,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:45:36.298635+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:43.559452+0000","last_clean_scrub_stamp":"2024-03-11T21:45:36.298635+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":110,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":1004,"reported_epoch":521,"state":"active+clean","last_fresh":"2024-03-11T21:49:03.825066+0000","last_change":"2024-03-11T21:48:28.621244+0000","last_active":"2024-03-11T21:49:03.825066+0000","last_peered":"2024-03-11T21:49:03.825066+0000","last_clean":"2024-03-11T21:49:03.825066+0000","last_became_active":"2024-03-11T21:48:28.620928+0000","last_became_peered":"2024-03-11T21:48:28.620928+0000","last_unstale":"2024-03-11T21:49:03.825066+0000","last_undegraded":"2024-03-11T21:49:03.825066+0000","last_fullsized":"2024-03-11T21:49:03.825066+0000","mapping_epoch":502,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":503,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:48:20.738562+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:27:21.207303+0000","last_clean_scrub_stamp":"2024-03-11T21:48:20.738562+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:49:45.603977+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,7],"acting":[1,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.15","version":"0'0","reported_seq":1004,"reported_epoch":521,"state":"active+clean","last_fresh":"2024-03-11T21:49:03.824763+0000","last_change":"2024-03-11T21:48:36.783179+0000","last_active":"2024-03-11T21:49:03.824763+0000","last_peered":"2024-03-11T21:49:03.824763+0000","last_clean":"2024-03-11T21:49:03.824763+0000","last_became_active":"2024-03-11T21:48:28.620494+0000","last_became_peered":"2024-03-11T21:48:28.620494+0000","last_unstale":"2024-03-11T21:49:03.824763+0000","last_undegraded":"2024-03-11T21:49:03.824763+0000","last_fullsized":"2024-03-11T21:49:03.824763+0000","mapping_epoch":502,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":503,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:48:36.783099+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:26:44.860499+0000","last_clean_scrub_stamp":"2024-03-11T21:48:36.783099+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:49:39.385224+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.13","version":"456'32","reported_seq":738,"reported_epoch":522,"state":"active+clean","last_fresh":"2024-03-11T21:49:05.635376+0000","last_change":"2024-03-11T21:48:28.677318+0000","last_active":"2024-03-11T21:49:05.635376+0000","last_peered":"2024-03-11T21:49:05.635376+0000","last_clean":"2024-03-11T21:49:05.635376+0000","last_became_active":"2024-03-11T21:48:28.642093+0000","last_became_peered":"2024-03-11T21:48:28.642093+0000","last_unstale":"2024-03-11T21:49:05.635376+0000","last_undegraded":"2024-03-11T21:49:05.635376+0000","last_fullsized":"2024-03-11T21:49:05.635376+0000","mapping_epoch":502,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":503,"parent":"0.0","parent_split_bits":0,"last_scrub":"456'32","last_scrub_stamp":"2024-03-11T21:46:13.717488+0000","last_deep_scrub":"260'24","last_deep_scrub_stamp":"2024-03-11T21:40:25.595220+0000","last_clean_scrub_stamp":"2024-03-11T21:46:13.717488+0000","objects_scrubbed":5,"log_size":32,"log_dups_size":0,"ondisk_log_size":32,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":43,"num_read_kb":36,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":440,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,5],"acting":[6,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.10","version":"0'0","reported_seq":984,"reported_epoch":521,"state":"active+clean","last_fresh":"2024-03-11T21:49:03.631225+0000","last_change":"2024-03-11T21:48:28.642867+0000","last_active":"2024-03-11T21:49:03.631225+0000","last_peered":"2024-03-11T21:49:03.631225+0000","last_clean":"2024-03-11T21:49:03.631225+0000","last_became_active":"2024-03-11T21:48:28.641880+0000","last_became_peered":"2024-03-11T21:48:28.641880+0000","last_unstale":"2024-03-11T21:49:03.631225+0000","last_undegraded":"2024-03-11T21:49:03.631225+0000","last_fullsized":"2024-03-11T21:49:03.631225+0000","mapping_epoch":502,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":503,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:45:35.179951+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:27:25.417450+0000","last_clean_scrub_stamp":"2024-03-11T21:45:35.179951+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,1],"acting":[0,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":601,"reported_epoch":521,"state":"active+clean","last_fresh":"2024-03-11T21:49:03.631710+0000","last_change":"2024-03-11T21:48:28.632486+0000","last_active":"2024-03-11T21:49:03.631710+0000","last_peered":"2024-03-11T21:49:03.631710+0000","last_clean":"2024-03-11T21:49:03.631710+0000","last_became_active":"2024-03-11T21:48:28.630899+0000","last_became_peered":"2024-03-11T21:48:28.630899+0000","last_unstale":"2024-03-11T21:49:03.631710+0000","last_undegraded":"2024-03-11T21:49:03.631710+0000","last_fullsized":"2024-03-11T21:49:03.631710+0000","mapping_epoch":502,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":503,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:46:24.352079+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:31:15.367898+0000","last_clean_scrub_stamp":"2024-03-11T21:46:24.352079+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,5],"acting":[0,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.14","version":"0'0","reported_seq":1046,"reported_epoch":522,"state":"active+clean","last_fresh":"2024-03-11T21:49:05.635432+0000","last_change":"2024-03-11T21:48:28.644295+0000","last_active":"2024-03-11T21:49:05.635432+0000","last_peered":"2024-03-11T21:49:05.635432+0000","last_clean":"2024-03-11T21:49:05.635432+0000","last_became_active":"2024-03-11T21:48:28.641764+0000","last_became_peered":"2024-03-11T21:48:28.641764+0000","last_unstale":"2024-03-11T21:49:05.635432+0000","last_undegraded":"2024-03-11T21:49:05.635432+0000","last_fullsized":"2024-03-11T21:49:05.635432+0000","mapping_epoch":502,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":503,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:48:10.491899+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:37.512018+0000","last_clean_scrub_stamp":"2024-03-11T21:48:10.491899+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:49:17.952624+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,7],"acting":[6,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.12","version":"456'21","reported_seq":1023,"reported_epoch":521,"state":"active+clean","last_fresh":"2024-03-11T21:49:03.824967+0000","last_change":"2024-03-11T21:48:27.749527+0000","last_active":"2024-03-11T21:49:03.824967+0000","last_peered":"2024-03-11T21:49:03.824967+0000","last_clean":"2024-03-11T21:49:03.824967+0000","last_became_active":"2024-03-11T21:25:40.547082+0000","last_became_peered":"2024-03-11T21:25:40.547082+0000","last_unstale":"2024-03-11T21:49:03.824967+0000","last_undegraded":"2024-03-11T21:49:03.824967+0000","last_fullsized":"2024-03-11T21:49:03.824967+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"456'21","last_scrub_stamp":"2024-03-11T21:48:27.749435+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:39.525554+0000","last_clean_scrub_stamp":"2024-03-11T21:48:27.749435+0000","objects_scrubbed":4,"log_size":21,"log_dups_size":0,"ondisk_log_size":21,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:49:55.648470+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":35,"num_read_kb":29,"num_write":22,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":211,"reported_epoch":523,"state":"active+clean","last_fresh":"2024-03-11T21:49:06.639534+0000","last_change":"2024-03-11T21:48:28.627694+0000","last_active":"2024-03-11T21:49:06.639534+0000","last_peered":"2024-03-11T21:49:06.639534+0000","last_clean":"2024-03-11T21:49:06.639534+0000","last_became_active":"2024-03-11T21:48:28.627346+0000","last_became_peered":"2024-03-11T21:48:28.627346+0000","last_unstale":"2024-03-11T21:49:06.639534+0000","last_undegraded":"2024-03-11T21:49:06.639534+0000","last_fullsized":"2024-03-11T21:49:06.639534+0000","mapping_epoch":502,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":503,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:48:10.902895+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:27:09.801138+0000","last_clean_scrub_stamp":"2024-03-11T21:48:10.902895+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:49:27.252018+0000","scrub_duration":265,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,0],"acting":[2,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.14","version":"434'13","reported_seq":811,"reported_epoch":523,"state":"active+clean","last_fresh":"2024-03-11T21:49:07.170266+0000","last_change":"2024-03-11T21:48:11.632156+0000","last_active":"2024-03-11T21:49:07.170266+0000","last_peered":"2024-03-11T21:49:07.170266+0000","last_clean":"2024-03-11T21:49:07.170266+0000","last_became_active":"2024-03-11T21:44:21.465719+0000","last_became_peered":"2024-03-11T21:44:21.465719+0000","last_unstale":"2024-03-11T21:49:07.170266+0000","last_undegraded":"2024-03-11T21:49:07.170266+0000","last_fullsized":"2024-03-11T21:49:07.170266+0000","mapping_epoch":432,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":433,"parent":"0.0","parent_split_bits":0,"last_scrub":"434'13","last_scrub_stamp":"2024-03-11T21:48:11.632032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:41.544550+0000","last_clean_scrub_stamp":"2024-03-11T21:48:11.632032+0000","objects_scrubbed":1,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:49:34.840151+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,1],"acting":[7,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.13","version":"0'0","reported_seq":1018,"reported_epoch":521,"state":"active+clean","last_fresh":"2024-03-11T21:49:03.631924+0000","last_change":"2024-03-11T21:48:28.624700+0000","last_active":"2024-03-11T21:49:03.631924+0000","last_peered":"2024-03-11T21:49:03.631924+0000","last_clean":"2024-03-11T21:49:03.631924+0000","last_became_active":"2024-03-11T21:48:28.623947+0000","last_became_peered":"2024-03-11T21:48:28.623947+0000","last_unstale":"2024-03-11T21:49:03.631924+0000","last_undegraded":"2024-03-11T21:49:03.631924+0000","last_fullsized":"2024-03-11T21:49:03.631924+0000","mapping_epoch":502,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":503,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:46:05.444630+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:37.512018+0000","last_clean_scrub_stamp":"2024-03-11T21:46:05.444630+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,5],"acting":[0,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.15","version":"456'25","reported_seq":69,"reported_epoch":523,"state":"active+clean","last_fresh":"2024-03-11T21:49:06.864095+0000","last_change":"2024-03-11T21:48:32.001180+0000","last_active":"2024-03-11T21:49:06.864095+0000","last_peered":"2024-03-11T21:49:06.864095+0000","last_clean":"2024-03-11T21:49:06.864095+0000","last_became_active":"2024-03-11T21:48:28.640657+0000","last_became_peered":"2024-03-11T21:48:28.640657+0000","last_unstale":"2024-03-11T21:49:06.864095+0000","last_undegraded":"2024-03-11T21:49:06.864095+0000","last_fullsized":"2024-03-11T21:49:06.864095+0000","mapping_epoch":502,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":503,"parent":"0.0","parent_split_bits":0,"last_scrub":"456'25","last_scrub_stamp":"2024-03-11T21:48:32.001057+0000","last_deep_scrub":"94'13","last_deep_scrub_stamp":"2024-03-11T21:29:41.853158+0000","last_clean_scrub_stamp":"2024-03-11T21:48:32.001057+0000","objects_scrubbed":4,"log_size":25,"log_dups_size":0,"ondisk_log_size":25,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:49:59.598719+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":34,"num_read_kb":28,"num_write":22,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19,"num_bytes_recovered":440,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.16","version":"0'0","reported_seq":1025,"reported_epoch":521,"state":"active+clean","last_fresh":"2024-03-11T21:49:03.632025+0000","last_change":"2024-03-11T21:48:28.624609+0000","last_active":"2024-03-11T21:49:03.632025+0000","last_peered":"2024-03-11T21:49:03.632025+0000","last_clean":"2024-03-11T21:49:03.632025+0000","last_became_active":"2024-03-11T21:48:28.623242+0000","last_became_peered":"2024-03-11T21:48:28.623242+0000","last_unstale":"2024-03-11T21:49:03.632025+0000","last_undegraded":"2024-03-11T21:49:03.632025+0000","last_fullsized":"2024-03-11T21:49:03.632025+0000","mapping_epoch":502,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":503,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:46:18.525893+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:40:30.249679+0000","last_clean_scrub_stamp":"2024-03-11T21:46:18.525893+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":105,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,5],"acting":[0,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.15","version":"503'12","reported_seq":63,"reported_epoch":523,"state":"active+clean","last_fresh":"2024-03-11T21:49:06.863995+0000","last_change":"2024-03-11T21:48:28.669188+0000","last_active":"2024-03-11T21:49:06.863995+0000","last_peered":"2024-03-11T21:49:06.863995+0000","last_clean":"2024-03-11T21:49:06.863995+0000","last_became_active":"2024-03-11T21:48:28.646407+0000","last_became_peered":"2024-03-11T21:48:28.646407+0000","last_unstale":"2024-03-11T21:49:06.863995+0000","last_undegraded":"2024-03-11T21:49:06.863995+0000","last_fullsized":"2024-03-11T21:49:06.863995+0000","mapping_epoch":502,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":503,"parent":"0.0","parent_split_bits":0,"last_scrub":"98'9","last_scrub_stamp":"2024-03-11T21:46:08.945626+0000","last_deep_scrub":"98'9","last_deep_scrub_stamp":"2024-03-11T21:31:20.624923+0000","last_clean_scrub_stamp":"2024-03-11T21:46:08.945626+0000","objects_scrubbed":1,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.12","version":"0'0","reported_seq":59,"reported_epoch":523,"state":"active+clean","last_fresh":"2024-03-11T21:49:06.863800+0000","last_change":"2024-03-11T21:48:34.025883+0000","last_active":"2024-03-11T21:49:06.863800+0000","last_peered":"2024-03-11T21:49:06.863800+0000","last_clean":"2024-03-11T21:49:06.863800+0000","last_became_active":"2024-03-11T21:48:28.645153+0000","last_became_peered":"2024-03-11T21:48:28.645153+0000","last_unstale":"2024-03-11T21:49:06.863800+0000","last_undegraded":"2024-03-11T21:49:06.863800+0000","last_fullsized":"2024-03-11T21:49:06.863800+0000","mapping_epoch":502,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":503,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:48:34.025798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:48:34.025798+0000","last_clean_scrub_stamp":"2024-03-11T21:48:34.025798+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:49:41.357258+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,0],"acting":[5,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.14","version":"456'30","reported_seq":293,"reported_epoch":523,"state":"active+clean","last_fresh":"2024-03-11T21:49:07.170268+0000","last_change":"2024-03-11T21:47:20.278549+0000","last_active":"2024-03-11T21:49:07.170268+0000","last_peered":"2024-03-11T21:49:07.170268+0000","last_clean":"2024-03-11T21:49:07.170268+0000","last_became_active":"2024-03-11T21:47:20.278192+0000","last_became_peered":"2024-03-11T21:47:20.278192+0000","last_unstale":"2024-03-11T21:49:07.170268+0000","last_undegraded":"2024-03-11T21:49:07.170268+0000","last_fullsized":"2024-03-11T21:49:07.170268+0000","mapping_epoch":478,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":479,"parent":"0.0","parent_split_bits":0,"last_scrub":"456'30","last_scrub_stamp":"2024-03-11T21:47:17.566980+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:39.525554+0000","last_clean_scrub_stamp":"2024-03-11T21:47:17.566980+0000","objects_scrubbed":5,"log_size":30,"log_dups_size":0,"ondisk_log_size":30,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":43,"num_read_kb":38,"num_write":30,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,2],"acting":[7,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.15","version":"0'0","reported_seq":649,"reported_epoch":523,"state":"active+clean","last_fresh":"2024-03-11T21:49:07.170434+0000","last_change":"2024-03-11T21:48:28.628344+0000","last_active":"2024-03-11T21:49:07.170434+0000","last_peered":"2024-03-11T21:49:07.170434+0000","last_clean":"2024-03-11T21:49:07.170434+0000","last_became_active":"2024-03-11T21:48:28.628004+0000","last_became_peered":"2024-03-11T21:48:28.628004+0000","last_unstale":"2024-03-11T21:49:07.170434+0000","last_undegraded":"2024-03-11T21:49:07.170434+0000","last_fullsized":"2024-03-11T21:49:07.170434+0000","mapping_epoch":502,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":503,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:46:07.457807+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:43.559452+0000","last_clean_scrub_stamp":"2024-03-11T21:46:07.457807+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":30,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,5],"acting":[7,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":59,"reported_epoch":523,"state":"active+clean","last_fresh":"2024-03-11T21:49:06.864186+0000","last_change":"2024-03-11T21:48:57.999410+0000","last_active":"2024-03-11T21:49:06.864186+0000","last_peered":"2024-03-11T21:49:06.864186+0000","last_clean":"2024-03-11T21:49:06.864186+0000","last_became_active":"2024-03-11T21:48:28.638080+0000","last_became_peered":"2024-03-11T21:48:28.638080+0000","last_unstale":"2024-03-11T21:49:06.864186+0000","last_undegraded":"2024-03-11T21:49:06.864186+0000","last_fullsized":"2024-03-11T21:49:06.864186+0000","mapping_epoch":502,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":503,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:48:57.999278+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:43:36.585632+0000","last_clean_scrub_stamp":"2024-03-11T21:48:57.999278+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:50:13.474101+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.11","version":"0'0","reported_seq":1021,"reported_epoch":522,"state":"active+clean","last_fresh":"2024-03-11T21:49:05.635100+0000","last_change":"2024-03-11T21:48:28.641925+0000","last_active":"2024-03-11T21:49:05.635100+0000","last_peered":"2024-03-11T21:49:05.635100+0000","last_clean":"2024-03-11T21:49:05.635100+0000","last_became_active":"2024-03-11T21:48:28.641503+0000","last_became_peered":"2024-03-11T21:48:28.641503+0000","last_unstale":"2024-03-11T21:49:05.635100+0000","last_undegraded":"2024-03-11T21:49:05.635100+0000","last_fullsized":"2024-03-11T21:49:05.635100+0000","mapping_epoch":502,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":503,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:46:06.705444+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:37.512018+0000","last_clean_scrub_stamp":"2024-03-11T21:46:06.705444+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,5],"acting":[6,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.17","version":"456'21","reported_seq":692,"reported_epoch":523,"state":"active+clean","last_fresh":"2024-03-11T21:49:07.170612+0000","last_change":"2024-03-11T21:48:17.609503+0000","last_active":"2024-03-11T21:49:07.170612+0000","last_peered":"2024-03-11T21:49:07.170612+0000","last_clean":"2024-03-11T21:49:07.170612+0000","last_became_active":"2024-03-11T21:46:53.270498+0000","last_became_peered":"2024-03-11T21:46:53.270498+0000","last_unstale":"2024-03-11T21:49:07.170612+0000","last_undegraded":"2024-03-11T21:49:07.170612+0000","last_fullsized":"2024-03-11T21:49:07.170612+0000","mapping_epoch":473,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":474,"parent":"0.0","parent_split_bits":0,"last_scrub":"456'21","last_scrub_stamp":"2024-03-11T21:48:17.609367+0000","last_deep_scrub":"97'9","last_deep_scrub_stamp":"2024-03-11T21:27:25.803132+0000","last_clean_scrub_stamp":"2024-03-11T21:48:17.609367+0000","objects_scrubbed":3,"log_size":21,"log_dups_size":0,"ondisk_log_size":21,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:49:30.030597+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":27,"num_read_kb":24,"num_write":18,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,2],"acting":[7,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.14","version":"0'0","reported_seq":1008,"reported_epoch":523,"state":"active+clean","last_fresh":"2024-03-11T21:49:06.639250+0000","last_change":"2024-03-11T21:48:58.629494+0000","last_active":"2024-03-11T21:49:06.639250+0000","last_peered":"2024-03-11T21:49:06.639250+0000","last_clean":"2024-03-11T21:49:06.639250+0000","last_became_active":"2024-03-11T21:44:35.626384+0000","last_became_peered":"2024-03-11T21:44:35.626384+0000","last_unstale":"2024-03-11T21:49:06.639250+0000","last_undegraded":"2024-03-11T21:49:06.639250+0000","last_fullsized":"2024-03-11T21:49:06.639250+0000","mapping_epoch":437,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":438,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:48:58.629390+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:43.559452+0000","last_clean_scrub_stamp":"2024-03-11T21:48:58.629390+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:50:06.090017+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,0],"acting":[2,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":619,"reported_epoch":521,"state":"active+clean","last_fresh":"2024-03-11T21:49:03.825087+0000","last_change":"2024-03-11T21:48:35.744834+0000","last_active":"2024-03-11T21:49:03.825087+0000","last_peered":"2024-03-11T21:49:03.825087+0000","last_clean":"2024-03-11T21:49:03.825087+0000","last_became_active":"2024-03-11T21:44:21.458758+0000","last_became_peered":"2024-03-11T21:44:21.458758+0000","last_unstale":"2024-03-11T21:49:03.825087+0000","last_undegraded":"2024-03-11T21:49:03.825087+0000","last_fullsized":"2024-03-11T21:49:03.825087+0000","mapping_epoch":432,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":433,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:48:35.744749+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:48:35.744749+0000","last_clean_scrub_stamp":"2024-03-11T21:48:35.744749+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:49:46.514481+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,0],"acting":[1,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.10","version":"0'0","reported_seq":1003,"reported_epoch":523,"state":"active+clean","last_fresh":"2024-03-11T21:49:06.636658+0000","last_change":"2024-03-11T21:48:32.585436+0000","last_active":"2024-03-11T21:49:06.636658+0000","last_peered":"2024-03-11T21:49:06.636658+0000","last_clean":"2024-03-11T21:49:06.636658+0000","last_became_active":"2024-03-11T21:25:38.533138+0000","last_became_peered":"2024-03-11T21:25:38.533138+0000","last_unstale":"2024-03-11T21:49:06.636658+0000","last_undegraded":"2024-03-11T21:49:06.636658+0000","last_fullsized":"2024-03-11T21:49:06.636658+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:48:32.585314+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:37.512018+0000","last_clean_scrub_stamp":"2024-03-11T21:48:32.585314+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:49:41.783553+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,0],"acting":[2,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.16","version":"456'25","reported_seq":1084,"reported_epoch":521,"state":"active+clean","last_fresh":"2024-03-11T21:49:03.632127+0000","last_change":"2024-03-11T21:45:25.093159+0000","last_active":"2024-03-11T21:49:03.632127+0000","last_peered":"2024-03-11T21:49:03.632127+0000","last_clean":"2024-03-11T21:49:03.632127+0000","last_became_active":"2024-03-11T21:44:21.463330+0000","last_became_peered":"2024-03-11T21:44:21.463330+0000","last_unstale":"2024-03-11T21:49:03.632127+0000","last_undegraded":"2024-03-11T21:49:03.632127+0000","last_fullsized":"2024-03-11T21:49:03.632127+0000","mapping_epoch":432,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":433,"parent":"0.0","parent_split_bits":0,"last_scrub":"369'19","last_scrub_stamp":"2024-03-11T21:45:25.093077+0000","last_deep_scrub":"369'19","last_deep_scrub_stamp":"2024-03-11T21:41:22.115634+0000","last_clean_scrub_stamp":"2024-03-11T21:45:25.093077+0000","objects_scrubbed":4,"log_size":25,"log_dups_size":0,"ondisk_log_size":25,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":35,"num_read_kb":29,"num_write":22,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.b","version":"0'0","reported_seq":611,"reported_epoch":523,"state":"active+clean","last_fresh":"2024-03-11T21:49:07.170062+0000","last_change":"2024-03-11T21:48:21.597003+0000","last_active":"2024-03-11T21:49:07.170062+0000","last_peered":"2024-03-11T21:49:07.170062+0000","last_clean":"2024-03-11T21:49:07.170062+0000","last_became_active":"2024-03-11T21:44:21.449857+0000","last_became_peered":"2024-03-11T21:44:21.449857+0000","last_unstale":"2024-03-11T21:49:07.170062+0000","last_undegraded":"2024-03-11T21:49:07.170062+0000","last_fullsized":"2024-03-11T21:49:07.170062+0000","mapping_epoch":432,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":433,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:48:21.596882+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:44:35.622488+0000","last_clean_scrub_stamp":"2024-03-11T21:48:21.596882+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:49:49.619849+0000","scrub_duration":54,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,6],"acting":[7,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.f","version":"67'2","reported_seq":227,"reported_epoch":521,"state":"active+clean","last_fresh":"2024-03-11T21:49:03.632407+0000","last_change":"2024-03-11T21:48:28.670177+0000","last_active":"2024-03-11T21:49:03.632407+0000","last_peered":"2024-03-11T21:49:03.632407+0000","last_clean":"2024-03-11T21:49:03.632407+0000","last_became_active":"2024-03-11T21:48:28.641365+0000","last_became_peered":"2024-03-11T21:48:28.641365+0000","last_unstale":"2024-03-11T21:49:03.632407+0000","last_undegraded":"2024-03-11T21:49:03.632407+0000","last_fullsized":"2024-03-11T21:49:03.632407+0000","mapping_epoch":502,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":503,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'2","last_scrub_stamp":"2024-03-11T21:46:20.404697+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:37.512018+0000","last_clean_scrub_stamp":"2024-03-11T21:46:20.404697+0000","objects_scrubbed":2,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":92,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":276,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,5],"acting":[0,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.9","version":"456'29","reported_seq":676,"reported_epoch":521,"state":"active+clean","last_fresh":"2024-03-11T21:49:03.824314+0000","last_change":"2024-03-11T21:46:28.006582+0000","last_active":"2024-03-11T21:49:03.824314+0000","last_peered":"2024-03-11T21:49:03.824314+0000","last_clean":"2024-03-11T21:49:03.824314+0000","last_became_active":"2024-03-11T21:44:35.618968+0000","last_became_peered":"2024-03-11T21:44:35.618968+0000","last_unstale":"2024-03-11T21:49:03.824314+0000","last_undegraded":"2024-03-11T21:49:03.824314+0000","last_fullsized":"2024-03-11T21:49:03.824314+0000","mapping_epoch":437,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":438,"parent":"0.0","parent_split_bits":0,"last_scrub":"456'29","last_scrub_stamp":"2024-03-11T21:46:28.006437+0000","last_deep_scrub":"456'29","last_deep_scrub_stamp":"2024-03-11T21:46:28.006437+0000","last_clean_scrub_stamp":"2024-03-11T21:46:28.006437+0000","objects_scrubbed":5,"log_size":29,"log_dups_size":0,"ondisk_log_size":29,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":42,"num_read_kb":33,"num_write":26,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,0],"acting":[1,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":995,"reported_epoch":523,"state":"active+clean","last_fresh":"2024-03-11T21:49:06.636753+0000","last_change":"2024-03-11T21:48:55.615666+0000","last_active":"2024-03-11T21:49:06.636753+0000","last_peered":"2024-03-11T21:49:06.636753+0000","last_clean":"2024-03-11T21:49:06.636753+0000","last_became_active":"2024-03-11T21:25:42.573987+0000","last_became_peered":"2024-03-11T21:25:42.573987+0000","last_unstale":"2024-03-11T21:49:06.636753+0000","last_undegraded":"2024-03-11T21:49:06.636753+0000","last_fullsized":"2024-03-11T21:49:06.636753+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:48:55.615577+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:41.544550+0000","last_clean_scrub_stamp":"2024-03-11T21:48:55.615577+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:49:58.882101+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.0","version":"0'0","reported_seq":1001,"reported_epoch":521,"state":"active+clean","last_fresh":"2024-03-11T21:49:03.631293+0000","last_change":"2024-03-11T21:48:28.641760+0000","last_active":"2024-03-11T21:49:03.631293+0000","last_peered":"2024-03-11T21:49:03.631293+0000","last_clean":"2024-03-11T21:49:03.631293+0000","last_became_active":"2024-03-11T21:48:28.641428+0000","last_became_peered":"2024-03-11T21:48:28.641428+0000","last_unstale":"2024-03-11T21:49:03.631293+0000","last_undegraded":"2024-03-11T21:49:03.631293+0000","last_fullsized":"2024-03-11T21:49:03.631293+0000","mapping_epoch":502,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":503,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:46:13.467417+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:29:39.498784+0000","last_clean_scrub_stamp":"2024-03-11T21:46:13.467417+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,5],"acting":[0,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":474,"reported_epoch":521,"state":"active+clean","last_fresh":"2024-03-11T21:49:03.631803+0000","last_change":"2024-03-11T21:48:58.177402+0000","last_active":"2024-03-11T21:49:03.631803+0000","last_peered":"2024-03-11T21:49:03.631803+0000","last_clean":"2024-03-11T21:49:03.631803+0000","last_became_active":"2024-03-11T21:47:20.274227+0000","last_became_peered":"2024-03-11T21:47:20.274227+0000","last_unstale":"2024-03-11T21:49:03.631803+0000","last_undegraded":"2024-03-11T21:49:03.631803+0000","last_fullsized":"2024-03-11T21:49:03.631803+0000","mapping_epoch":478,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":479,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:48:58.177316+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:48:58.177316+0000","last_clean_scrub_stamp":"2024-03-11T21:48:58.177316+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:50:24.838411+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,7],"acting":[0,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.2","version":"456'35","reported_seq":1094,"reported_epoch":521,"state":"active+clean","last_fresh":"2024-03-11T21:49:03.824933+0000","last_change":"2024-03-11T21:48:28.660028+0000","last_active":"2024-03-11T21:49:03.824933+0000","last_peered":"2024-03-11T21:49:03.824933+0000","last_clean":"2024-03-11T21:49:03.824933+0000","last_became_active":"2024-03-11T21:48:28.630917+0000","last_became_peered":"2024-03-11T21:48:28.630917+0000","last_unstale":"2024-03-11T21:49:03.824933+0000","last_undegraded":"2024-03-11T21:49:03.824933+0000","last_fullsized":"2024-03-11T21:49:03.824933+0000","mapping_epoch":502,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":503,"parent":"0.0","parent_split_bits":0,"last_scrub":"456'35","last_scrub_stamp":"2024-03-11T21:47:11.668986+0000","last_deep_scrub":"94'15","last_deep_scrub_stamp":"2024-03-11T21:26:49.993060+0000","last_clean_scrub_stamp":"2024-03-11T21:47:11.668986+0000","objects_scrubbed":5,"log_size":35,"log_dups_size":0,"ondisk_log_size":35,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":57,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":45,"num_read_kb":40,"num_write":30,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.3","version":"503'10","reported_seq":1296,"reported_epoch":521,"state":"active+clean","last_fresh":"2024-03-11T21:49:03.632037+0000","last_change":"2024-03-11T21:48:56.097716+0000","last_active":"2024-03-11T21:49:03.632037+0000","last_peered":"2024-03-11T21:49:03.632037+0000","last_clean":"2024-03-11T21:49:03.632037+0000","last_became_active":"2024-03-11T21:48:28.641906+0000","last_became_peered":"2024-03-11T21:48:28.641906+0000","last_unstale":"2024-03-11T21:49:03.632037+0000","last_undegraded":"2024-03-11T21:49:03.632037+0000","last_fullsized":"2024-03-11T21:49:03.632037+0000","mapping_epoch":502,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":503,"parent":"0.0","parent_split_bits":0,"last_scrub":"503'10","last_scrub_stamp":"2024-03-11T21:48:56.097591+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:41.544550+0000","last_clean_scrub_stamp":"2024-03-11T21:48:56.097591+0000","objects_scrubbed":1,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:50:21.404866+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,5],"acting":[0,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":608,"reported_epoch":523,"state":"active+clean","last_fresh":"2024-03-11T21:49:07.170379+0000","last_change":"2024-03-11T21:48:33.604086+0000","last_active":"2024-03-11T21:49:07.170379+0000","last_peered":"2024-03-11T21:49:07.170379+0000","last_clean":"2024-03-11T21:49:07.170379+0000","last_became_active":"2024-03-11T21:48:28.626239+0000","last_became_peered":"2024-03-11T21:48:28.626239+0000","last_unstale":"2024-03-11T21:49:07.170379+0000","last_undegraded":"2024-03-11T21:49:07.170379+0000","last_fullsized":"2024-03-11T21:49:07.170379+0000","mapping_epoch":502,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":503,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:48:33.604004+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:28:30.563712+0000","last_clean_scrub_stamp":"2024-03-11T21:48:33.604004+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:49:34.815358+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,5],"acting":[7,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":51,"reported_epoch":523,"state":"active+clean","last_fresh":"2024-03-11T21:49:06.863875+0000","last_change":"2024-03-11T21:48:28.640601+0000","last_active":"2024-03-11T21:49:06.863875+0000","last_peered":"2024-03-11T21:49:06.863875+0000","last_clean":"2024-03-11T21:49:06.863875+0000","last_became_active":"2024-03-11T21:48:28.640205+0000","last_became_peered":"2024-03-11T21:48:28.640205+0000","last_unstale":"2024-03-11T21:49:06.863875+0000","last_undegraded":"2024-03-11T21:49:06.863875+0000","last_fullsized":"2024-03-11T21:49:06.863875+0000","mapping_epoch":502,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":503,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:48:26.741911+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:37.512018+0000","last_clean_scrub_stamp":"2024-03-11T21:48:26.741911+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:49:53.962330+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.4","version":"479'56","reported_seq":1352,"reported_epoch":523,"state":"active+clean","last_fresh":"2024-03-11T21:49:06.639846+0000","last_change":"2024-03-11T21:48:25.673157+0000","last_active":"2024-03-11T21:49:06.639846+0000","last_peered":"2024-03-11T21:49:06.639846+0000","last_clean":"2024-03-11T21:49:06.639846+0000","last_became_active":"2024-03-11T21:47:20.277794+0000","last_became_peered":"2024-03-11T21:47:20.277794+0000","last_unstale":"2024-03-11T21:49:06.639846+0000","last_undegraded":"2024-03-11T21:49:06.639846+0000","last_fullsized":"2024-03-11T21:49:06.639846+0000","mapping_epoch":478,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":479,"parent":"0.0","parent_split_bits":0,"last_scrub":"479'56","last_scrub_stamp":"2024-03-11T21:48:25.673057+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:39.525554+0000","last_clean_scrub_stamp":"2024-03-11T21:48:25.673057+0000","objects_scrubbed":10,"log_size":56,"log_dups_size":0,"ondisk_log_size":56,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:49:47.157799+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":358,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":83,"num_read_kb":68,"num_write":50,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":358,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,6],"acting":[2,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":984,"reported_epoch":521,"state":"active+clean","last_fresh":"2024-03-11T21:49:03.632228+0000","last_change":"2024-03-11T21:45:37.189812+0000","last_active":"2024-03-11T21:49:03.632228+0000","last_peered":"2024-03-11T21:49:03.632228+0000","last_clean":"2024-03-11T21:49:03.632228+0000","last_became_active":"2024-03-11T21:44:35.620226+0000","last_became_peered":"2024-03-11T21:44:35.620226+0000","last_unstale":"2024-03-11T21:49:03.632228+0000","last_undegraded":"2024-03-11T21:49:03.632228+0000","last_fullsized":"2024-03-11T21:49:03.632228+0000","mapping_epoch":437,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":438,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:45:37.189703+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:41.544550+0000","last_clean_scrub_stamp":"2024-03-11T21:45:37.189703+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,6],"acting":[0,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.5","version":"0'0","reported_seq":655,"reported_epoch":523,"state":"active+clean","last_fresh":"2024-03-11T21:49:07.170320+0000","last_change":"2024-03-11T21:48:55.863838+0000","last_active":"2024-03-11T21:49:07.170320+0000","last_peered":"2024-03-11T21:49:07.170320+0000","last_clean":"2024-03-11T21:49:07.170320+0000","last_became_active":"2024-03-11T21:44:21.454156+0000","last_became_peered":"2024-03-11T21:44:21.454156+0000","last_unstale":"2024-03-11T21:49:07.170320+0000","last_undegraded":"2024-03-11T21:49:07.170320+0000","last_fullsized":"2024-03-11T21:49:07.170320+0000","mapping_epoch":432,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":433,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:48:55.863712+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:43.559452+0000","last_clean_scrub_stamp":"2024-03-11T21:48:55.863712+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:50:13.185580+0000","scrub_duration":32,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,2],"acting":[7,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.0","version":"164'6","reported_seq":855,"reported_epoch":523,"state":"active+clean","last_fresh":"2024-03-11T21:49:07.529146+0000","last_change":"2024-03-11T21:48:53.654907+0000","last_active":"2024-03-11T21:49:07.529146+0000","last_peered":"2024-03-11T21:49:07.529146+0000","last_clean":"2024-03-11T21:49:07.529146+0000","last_became_active":"2024-03-11T21:47:20.277766+0000","last_became_peered":"2024-03-11T21:47:20.277766+0000","last_unstale":"2024-03-11T21:49:07.529146+0000","last_undegraded":"2024-03-11T21:49:07.529146+0000","last_fullsized":"2024-03-11T21:49:07.529146+0000","mapping_epoch":478,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":479,"parent":"0.0","parent_split_bits":0,"last_scrub":"164'6","last_scrub_stamp":"2024-03-11T21:48:53.654817+0000","last_deep_scrub":"164'6","last_deep_scrub_stamp":"2024-03-11T21:32:17.597666+0000","last_clean_scrub_stamp":"2024-03-11T21:48:53.654817+0000","objects_scrubbed":1,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:50:13.153353+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0.000753559,"stat_sum":{"num_bytes":389,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":495,"num_read_kb":490,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,6],"acting":[2,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":1019,"reported_epoch":523,"state":"active+clean","last_fresh":"2024-03-11T21:49:06.638979+0000","last_change":"2024-03-11T21:48:26.692563+0000","last_active":"2024-03-11T21:49:06.638979+0000","last_peered":"2024-03-11T21:49:06.638979+0000","last_clean":"2024-03-11T21:49:06.638979+0000","last_became_active":"2024-03-11T21:44:21.462115+0000","last_became_peered":"2024-03-11T21:44:21.462115+0000","last_unstale":"2024-03-11T21:49:06.638979+0000","last_undegraded":"2024-03-11T21:49:06.638979+0000","last_fullsized":"2024-03-11T21:49:06.638979+0000","mapping_epoch":432,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":433,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:48:26.692463+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:43:28.183315+0000","last_clean_scrub_stamp":"2024-03-11T21:48:26.692463+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:49:44.101622+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.7","version":"456'26","reported_seq":1072,"reported_epoch":521,"state":"active+clean","last_fresh":"2024-03-11T21:49:03.824673+0000","last_change":"2024-03-11T21:48:31.684836+0000","last_active":"2024-03-11T21:49:03.824673+0000","last_peered":"2024-03-11T21:49:03.824673+0000","last_clean":"2024-03-11T21:49:03.824673+0000","last_became_active":"2024-03-11T21:48:28.632227+0000","last_became_peered":"2024-03-11T21:48:28.632227+0000","last_unstale":"2024-03-11T21:49:03.824673+0000","last_undegraded":"2024-03-11T21:49:03.824673+0000","last_fullsized":"2024-03-11T21:49:03.824673+0000","mapping_epoch":502,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":503,"parent":"0.0","parent_split_bits":0,"last_scrub":"456'26","last_scrub_stamp":"2024-03-11T21:48:31.684721+0000","last_deep_scrub":"260'22","last_deep_scrub_stamp":"2024-03-11T21:41:27.232403+0000","last_clean_scrub_stamp":"2024-03-11T21:48:31.684721+0000","objects_scrubbed":5,"log_size":26,"log_dups_size":0,"ondisk_log_size":26,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:49:56.707197+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":42,"num_read_kb":31,"num_write":24,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":660,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":1019,"reported_epoch":523,"state":"active+clean","last_fresh":"2024-03-11T21:49:06.639558+0000","last_change":"2024-03-11T21:48:56.586124+0000","last_active":"2024-03-11T21:49:06.639558+0000","last_peered":"2024-03-11T21:49:06.639558+0000","last_clean":"2024-03-11T21:49:06.639558+0000","last_became_active":"2024-03-11T21:48:28.636457+0000","last_became_peered":"2024-03-11T21:48:28.636457+0000","last_unstale":"2024-03-11T21:49:06.639558+0000","last_undegraded":"2024-03-11T21:49:06.639558+0000","last_fullsized":"2024-03-11T21:49:06.639558+0000","mapping_epoch":502,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":503,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:48:56.586051+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:41.544550+0000","last_clean_scrub_stamp":"2024-03-11T21:48:56.586051+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:50:12.930267+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7],"acting":[2,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.7","version":"124'2","reported_seq":55,"reported_epoch":523,"state":"active+clean","last_fresh":"2024-03-11T21:49:06.864450+0000","last_change":"2024-03-11T21:48:28.652129+0000","last_active":"2024-03-11T21:49:06.864450+0000","last_peered":"2024-03-11T21:49:06.864450+0000","last_clean":"2024-03-11T21:49:06.864450+0000","last_became_active":"2024-03-11T21:48:28.635267+0000","last_became_peered":"2024-03-11T21:48:28.635267+0000","last_unstale":"2024-03-11T21:49:06.864450+0000","last_undegraded":"2024-03-11T21:49:06.864450+0000","last_fullsized":"2024-03-11T21:49:06.864450+0000","mapping_epoch":502,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":503,"parent":"0.0","parent_split_bits":0,"last_scrub":"124'2","last_scrub_stamp":"2024-03-11T21:46:27.025988+0000","last_deep_scrub":"124'2","last_deep_scrub_stamp":"2024-03-11T21:29:34.956908+0000","last_clean_scrub_stamp":"2024-03-11T21:46:27.025988+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":111,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":26,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,7],"acting":[5,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2","version":"71'2","reported_seq":63,"reported_epoch":523,"state":"active+clean","last_fresh":"2024-03-11T21:49:06.863655+0000","last_change":"2024-03-11T21:48:29.012420+0000","last_active":"2024-03-11T21:49:06.863655+0000","last_peered":"2024-03-11T21:49:06.863655+0000","last_clean":"2024-03-11T21:49:06.863655+0000","last_became_active":"2024-03-11T21:48:28.638169+0000","last_became_peered":"2024-03-11T21:48:28.638169+0000","last_unstale":"2024-03-11T21:49:06.863655+0000","last_undegraded":"2024-03-11T21:49:06.863655+0000","last_fullsized":"2024-03-11T21:49:06.863655+0000","mapping_epoch":502,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":503,"parent":"0.0","parent_split_bits":0,"last_scrub":"71'2","last_scrub_stamp":"2024-03-11T21:48:29.012327+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:37.512018+0000","last_clean_scrub_stamp":"2024-03-11T21:48:29.012327+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:49:37.866859+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.0014206449999999999,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":38,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,1],"acting":[5,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":51,"reported_epoch":523,"state":"active+clean","last_fresh":"2024-03-11T21:49:06.863911+0000","last_change":"2024-03-11T21:48:28.637004+0000","last_active":"2024-03-11T21:49:06.863911+0000","last_peered":"2024-03-11T21:49:06.863911+0000","last_clean":"2024-03-11T21:49:06.863911+0000","last_became_active":"2024-03-11T21:48:28.636579+0000","last_became_peered":"2024-03-11T21:48:28.636579+0000","last_unstale":"2024-03-11T21:49:06.863911+0000","last_undegraded":"2024-03-11T21:49:06.863911+0000","last_fullsized":"2024-03-11T21:49:06.863911+0000","mapping_epoch":502,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":503,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:46:17.861505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:37.512018+0000","last_clean_scrub_stamp":"2024-03-11T21:46:17.861505+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"87'128","reported_seq":1112,"reported_epoch":523,"state":"active+clean","last_fresh":"2024-03-11T21:49:07.169654+0000","last_change":"2024-03-11T21:48:15.542785+0000","last_active":"2024-03-11T21:49:07.169654+0000","last_peered":"2024-03-11T21:49:07.169654+0000","last_clean":"2024-03-11T21:49:07.169654+0000","last_became_active":"2024-03-11T21:34:44.164569+0000","last_became_peered":"2024-03-11T21:34:44.164569+0000","last_unstale":"2024-03-11T21:49:07.169654+0000","last_undegraded":"2024-03-11T21:49:07.169654+0000","last_fullsized":"2024-03-11T21:49:07.169654+0000","mapping_epoch":229,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":230,"parent":"0.0","parent_split_bits":0,"last_scrub":"87'128","last_scrub_stamp":"2024-03-11T21:48:15.542640+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:24.261974+0000","last_clean_scrub_stamp":"2024-03-11T21:48:15.542640+0000","objects_scrubbed":2,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:49:44.547543+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":198,"num_read_kb":294,"num_write":179,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":2951840,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.5","version":"456'41","reported_seq":1373,"reported_epoch":522,"state":"active+clean","last_fresh":"2024-03-11T21:49:05.635738+0000","last_change":"2024-03-11T21:48:11.506342+0000","last_active":"2024-03-11T21:49:05.635738+0000","last_peered":"2024-03-11T21:49:05.635738+0000","last_clean":"2024-03-11T21:49:05.635738+0000","last_became_active":"2024-03-11T21:44:35.634664+0000","last_became_peered":"2024-03-11T21:44:35.634664+0000","last_unstale":"2024-03-11T21:49:05.635738+0000","last_undegraded":"2024-03-11T21:49:05.635738+0000","last_fullsized":"2024-03-11T21:49:05.635738+0000","mapping_epoch":437,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":438,"parent":"0.0","parent_split_bits":0,"last_scrub":"456'41","last_scrub_stamp":"2024-03-11T21:48:11.506188+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:39.525554+0000","last_clean_scrub_stamp":"2024-03-11T21:48:11.506188+0000","objects_scrubbed":8,"log_size":41,"log_dups_size":0,"ondisk_log_size":41,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:49:37.999828+0000","scrub_duration":18,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":154,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":61,"num_read_kb":51,"num_write":37,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":154,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,0],"acting":[6,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":652,"reported_epoch":523,"state":"active+clean","last_fresh":"2024-03-11T21:49:07.170348+0000","last_change":"2024-03-11T21:48:28.629217+0000","last_active":"2024-03-11T21:49:07.170348+0000","last_peered":"2024-03-11T21:49:07.170348+0000","last_clean":"2024-03-11T21:49:07.170348+0000","last_became_active":"2024-03-11T21:48:28.627651+0000","last_became_peered":"2024-03-11T21:48:28.627651+0000","last_unstale":"2024-03-11T21:49:07.170348+0000","last_undegraded":"2024-03-11T21:49:07.170348+0000","last_fullsized":"2024-03-11T21:49:07.170348+0000","mapping_epoch":502,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":503,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:46:27.586495+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:32:21.266672+0000","last_clean_scrub_stamp":"2024-03-11T21:46:27.586495+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,5],"acting":[7,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.4","version":"0'0","reported_seq":1001,"reported_epoch":521,"state":"active+clean","last_fresh":"2024-03-11T21:49:03.824696+0000","last_change":"2024-03-11T21:48:28.620977+0000","last_active":"2024-03-11T21:49:03.824696+0000","last_peered":"2024-03-11T21:49:03.824696+0000","last_clean":"2024-03-11T21:49:03.824696+0000","last_became_active":"2024-03-11T21:48:28.620607+0000","last_became_peered":"2024-03-11T21:48:28.620607+0000","last_unstale":"2024-03-11T21:49:03.824696+0000","last_undegraded":"2024-03-11T21:49:03.824696+0000","last_fullsized":"2024-03-11T21:49:03.824696+0000","mapping_epoch":502,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":503,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:47:10.662269+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:43.559452+0000","last_clean_scrub_stamp":"2024-03-11T21:47:10.662269+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,6],"acting":[1,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":1034,"reported_epoch":521,"state":"active+clean","last_fresh":"2024-03-11T21:49:03.632299+0000","last_change":"2024-03-11T21:48:37.960673+0000","last_active":"2024-03-11T21:49:03.632299+0000","last_peered":"2024-03-11T21:49:03.632299+0000","last_clean":"2024-03-11T21:49:03.632299+0000","last_became_active":"2024-03-11T21:48:28.623854+0000","last_became_peered":"2024-03-11T21:48:28.623854+0000","last_unstale":"2024-03-11T21:49:03.632299+0000","last_undegraded":"2024-03-11T21:49:03.632299+0000","last_fullsized":"2024-03-11T21:49:03.632299+0000","mapping_epoch":502,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":503,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:48:37.960594+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:37.512018+0000","last_clean_scrub_stamp":"2024-03-11T21:48:37.960594+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:50:05.148862+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.00089342499999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":633,"reported_epoch":523,"state":"active+clean","last_fresh":"2024-03-11T21:49:07.169608+0000","last_change":"2024-03-11T21:46:19.530489+0000","last_active":"2024-03-11T21:49:07.169608+0000","last_peered":"2024-03-11T21:49:07.169608+0000","last_clean":"2024-03-11T21:49:07.169608+0000","last_became_active":"2024-03-11T21:34:41.149219+0000","last_became_peered":"2024-03-11T21:34:41.149219+0000","last_unstale":"2024-03-11T21:49:07.169608+0000","last_undegraded":"2024-03-11T21:49:07.169608+0000","last_fullsized":"2024-03-11T21:49:07.169608+0000","mapping_epoch":226,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":227,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:46:19.530414+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:43:30.734024+0000","last_clean_scrub_stamp":"2024-03-11T21:46:19.530414+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,0],"acting":[7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.6","version":"456'42","reported_seq":1089,"reported_epoch":521,"state":"active+clean","last_fresh":"2024-03-11T21:49:03.632102+0000","last_change":"2024-03-11T21:46:53.286336+0000","last_active":"2024-03-11T21:49:03.632102+0000","last_peered":"2024-03-11T21:49:03.632102+0000","last_clean":"2024-03-11T21:49:03.632102+0000","last_became_active":"2024-03-11T21:46:53.263790+0000","last_became_peered":"2024-03-11T21:46:53.263790+0000","last_unstale":"2024-03-11T21:49:03.632102+0000","last_undegraded":"2024-03-11T21:49:03.632102+0000","last_fullsized":"2024-03-11T21:49:03.632102+0000","mapping_epoch":473,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":474,"parent":"0.0","parent_split_bits":0,"last_scrub":"456'42","last_scrub_stamp":"2024-03-11T21:46:10.509062+0000","last_deep_scrub":"352'30","last_deep_scrub_stamp":"2024-03-11T21:40:31.175323+0000","last_clean_scrub_stamp":"2024-03-11T21:46:10.509062+0000","objects_scrubbed":6,"log_size":42,"log_dups_size":0,"ondisk_log_size":42,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":54,"num_read_kb":48,"num_write":36,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":59,"reported_epoch":523,"state":"active+clean","last_fresh":"2024-03-11T21:49:06.864336+0000","last_change":"2024-03-11T21:48:40.968279+0000","last_active":"2024-03-11T21:49:06.864336+0000","last_peered":"2024-03-11T21:49:06.864336+0000","last_clean":"2024-03-11T21:49:06.864336+0000","last_became_active":"2024-03-11T21:48:28.644298+0000","last_became_peered":"2024-03-11T21:48:28.644298+0000","last_unstale":"2024-03-11T21:49:06.864336+0000","last_undegraded":"2024-03-11T21:49:06.864336+0000","last_fullsized":"2024-03-11T21:49:06.864336+0000","mapping_epoch":502,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":503,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:48:40.968182+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:41.544550+0000","last_clean_scrub_stamp":"2024-03-11T21:48:40.968182+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:49:43.376979+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":982,"reported_epoch":521,"state":"active+clean","last_fresh":"2024-03-11T21:49:03.824898+0000","last_change":"2024-03-11T21:46:18.006161+0000","last_active":"2024-03-11T21:49:03.824898+0000","last_peered":"2024-03-11T21:49:03.824898+0000","last_clean":"2024-03-11T21:49:03.824898+0000","last_became_active":"2024-03-11T21:25:45.301416+0000","last_became_peered":"2024-03-11T21:25:45.301416+0000","last_unstale":"2024-03-11T21:49:03.824898+0000","last_undegraded":"2024-03-11T21:49:03.824898+0000","last_fullsized":"2024-03-11T21:49:03.824898+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:46:18.006081+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:43.559452+0000","last_clean_scrub_stamp":"2024-03-11T21:46:18.006081+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":29,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":646,"reported_epoch":523,"state":"active+clean","last_fresh":"2024-03-11T21:49:07.170378+0000","last_change":"2024-03-11T21:48:53.772518+0000","last_active":"2024-03-11T21:49:07.170378+0000","last_peered":"2024-03-11T21:49:07.170378+0000","last_clean":"2024-03-11T21:49:07.170378+0000","last_became_active":"2024-03-11T21:44:35.624442+0000","last_became_peered":"2024-03-11T21:44:35.624442+0000","last_unstale":"2024-03-11T21:49:07.170378+0000","last_undegraded":"2024-03-11T21:49:07.170378+0000","last_fullsized":"2024-03-11T21:49:07.170378+0000","mapping_epoch":437,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":438,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:48:53.772368+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:26:57.333669+0000","last_clean_scrub_stamp":"2024-03-11T21:48:53.772368+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:50:06.313649+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,1],"acting":[7,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.3","version":"456'47","reported_seq":1160,"reported_epoch":521,"state":"active+clean","last_fresh":"2024-03-11T21:49:03.631846+0000","last_change":"2024-03-11T21:48:28.662544+0000","last_active":"2024-03-11T21:49:03.631846+0000","last_peered":"2024-03-11T21:49:03.631846+0000","last_clean":"2024-03-11T21:49:03.631846+0000","last_became_active":"2024-03-11T21:48:28.623917+0000","last_became_peered":"2024-03-11T21:48:28.623917+0000","last_unstale":"2024-03-11T21:49:03.631846+0000","last_undegraded":"2024-03-11T21:49:03.631846+0000","last_fullsized":"2024-03-11T21:49:03.631846+0000","mapping_epoch":502,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":503,"parent":"0.0","parent_split_bits":0,"last_scrub":"456'47","last_scrub_stamp":"2024-03-11T21:46:27.467220+0000","last_deep_scrub":"165'27","last_deep_scrub_stamp":"2024-03-11T21:32:21.576732+0000","last_clean_scrub_stamp":"2024-03-11T21:46:27.467220+0000","objects_scrubbed":8,"log_size":47,"log_dups_size":0,"ondisk_log_size":47,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":69,"num_read_kb":55,"num_write":42,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":32,"num_bytes_recovered":1320,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":1024,"reported_epoch":522,"state":"active+clean","last_fresh":"2024-03-11T21:49:05.635300+0000","last_change":"2024-03-11T21:48:29.376975+0000","last_active":"2024-03-11T21:49:05.635300+0000","last_peered":"2024-03-11T21:49:05.635300+0000","last_clean":"2024-03-11T21:49:05.635300+0000","last_became_active":"2024-03-11T21:48:28.642354+0000","last_became_peered":"2024-03-11T21:48:28.642354+0000","last_unstale":"2024-03-11T21:49:05.635300+0000","last_undegraded":"2024-03-11T21:49:05.635300+0000","last_fullsized":"2024-03-11T21:49:05.635300+0000","mapping_epoch":502,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":503,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:48:29.376888+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:41:21.857205+0000","last_clean_scrub_stamp":"2024-03-11T21:48:29.376888+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:49:59.106992+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":998,"reported_epoch":523,"state":"active+clean","last_fresh":"2024-03-11T21:49:06.637652+0000","last_change":"2024-03-11T21:48:34.576513+0000","last_active":"2024-03-11T21:49:06.637652+0000","last_peered":"2024-03-11T21:49:06.637652+0000","last_clean":"2024-03-11T21:49:06.637652+0000","last_became_active":"2024-03-11T21:44:35.628373+0000","last_became_peered":"2024-03-11T21:44:35.628373+0000","last_unstale":"2024-03-11T21:49:06.637652+0000","last_undegraded":"2024-03-11T21:49:06.637652+0000","last_fullsized":"2024-03-11T21:49:06.637652+0000","mapping_epoch":437,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":438,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:48:34.576398+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:43.559452+0000","last_clean_scrub_stamp":"2024-03-11T21:48:34.576398+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:49:43.761917+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6,7],"acting":[2,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.6","version":"67'1","reported_seq":1004,"reported_epoch":521,"state":"active+clean","last_fresh":"2024-03-11T21:49:03.824462+0000","last_change":"2024-03-11T21:48:19.797460+0000","last_active":"2024-03-11T21:49:03.824462+0000","last_peered":"2024-03-11T21:49:03.824462+0000","last_clean":"2024-03-11T21:49:03.824462+0000","last_became_active":"2024-03-11T21:44:35.618830+0000","last_became_peered":"2024-03-11T21:44:35.618830+0000","last_unstale":"2024-03-11T21:49:03.824462+0000","last_undegraded":"2024-03-11T21:49:03.824462+0000","last_fullsized":"2024-03-11T21:49:03.824462+0000","mapping_epoch":437,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":438,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'1","last_scrub_stamp":"2024-03-11T21:48:19.796812+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:37.512018+0000","last_clean_scrub_stamp":"2024-03-11T21:48:19.796812+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:49:47.201709+0000","scrub_duration":21,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":46,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,0],"acting":[1,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.0","version":"456'50","reported_seq":732,"reported_epoch":523,"state":"active+clean","last_fresh":"2024-03-11T21:49:07.169753+0000","last_change":"2024-03-11T21:45:39.242650+0000","last_active":"2024-03-11T21:49:07.169753+0000","last_peered":"2024-03-11T21:49:07.169753+0000","last_clean":"2024-03-11T21:49:07.169753+0000","last_became_active":"2024-03-11T21:44:21.449580+0000","last_became_peered":"2024-03-11T21:44:21.449580+0000","last_unstale":"2024-03-11T21:49:07.169753+0000","last_undegraded":"2024-03-11T21:49:07.169753+0000","last_fullsized":"2024-03-11T21:49:07.169753+0000","mapping_epoch":432,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":433,"parent":"0.0","parent_split_bits":0,"last_scrub":"260'38","last_scrub_stamp":"2024-03-11T21:45:39.241846+0000","last_deep_scrub":"260'38","last_deep_scrub_stamp":"2024-03-11T21:45:39.241846+0000","last_clean_scrub_stamp":"2024-03-11T21:45:39.241846+0000","objects_scrubbed":8,"log_size":50,"log_dups_size":0,"ondisk_log_size":50,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":70,"num_read_kb":58,"num_write":44,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":440,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,1],"acting":[7,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":1020,"reported_epoch":523,"state":"active+clean","last_fresh":"2024-03-11T21:49:07.169974+0000","last_change":"2024-03-11T21:48:28.630225+0000","last_active":"2024-03-11T21:49:07.169974+0000","last_peered":"2024-03-11T21:49:07.169974+0000","last_clean":"2024-03-11T21:49:07.169974+0000","last_became_active":"2024-03-11T21:48:28.629828+0000","last_became_peered":"2024-03-11T21:48:28.629828+0000","last_unstale":"2024-03-11T21:49:07.169974+0000","last_undegraded":"2024-03-11T21:49:07.169974+0000","last_fullsized":"2024-03-11T21:49:07.169974+0000","mapping_epoch":502,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":503,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:46:09.504070+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:40:21.518840+0000","last_clean_scrub_stamp":"2024-03-11T21:46:09.504070+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,5],"acting":[7,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.3","version":"0'0","reported_seq":632,"reported_epoch":523,"state":"active+clean","last_fresh":"2024-03-11T21:49:07.170184+0000","last_change":"2024-03-11T21:46:25.645839+0000","last_active":"2024-03-11T21:49:07.170184+0000","last_peered":"2024-03-11T21:49:07.170184+0000","last_clean":"2024-03-11T21:49:07.170184+0000","last_became_active":"2024-03-11T21:34:41.142683+0000","last_became_peered":"2024-03-11T21:34:41.142683+0000","last_unstale":"2024-03-11T21:49:07.170184+0000","last_undegraded":"2024-03-11T21:49:07.170184+0000","last_fullsized":"2024-03-11T21:49:07.170184+0000","mapping_epoch":226,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":227,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:46:25.645722+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:46:25.645722+0000","last_clean_scrub_stamp":"2024-03-11T21:46:25.645722+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":44,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,2],"acting":[7,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":1031,"reported_epoch":522,"state":"active+clean","last_fresh":"2024-03-11T21:49:05.635649+0000","last_change":"2024-03-11T21:48:39.453675+0000","last_active":"2024-03-11T21:49:05.635649+0000","last_peered":"2024-03-11T21:49:05.635649+0000","last_clean":"2024-03-11T21:49:05.635649+0000","last_became_active":"2024-03-11T21:34:41.145628+0000","last_became_peered":"2024-03-11T21:34:41.145628+0000","last_unstale":"2024-03-11T21:49:05.635649+0000","last_undegraded":"2024-03-11T21:49:05.635649+0000","last_fullsized":"2024-03-11T21:49:05.635649+0000","mapping_epoch":226,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":227,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:48:39.453573+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:37.512018+0000","last_clean_scrub_stamp":"2024-03-11T21:48:39.453573+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:49:59.564982+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,2],"acting":[6,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.1","version":"507'49","reported_seq":82,"reported_epoch":523,"state":"active+clean","last_fresh":"2024-03-11T21:49:06.863700+0000","last_change":"2024-03-11T21:48:36.104153+0000","last_active":"2024-03-11T21:49:06.863700+0000","last_peered":"2024-03-11T21:49:06.863700+0000","last_clean":"2024-03-11T21:49:06.863700+0000","last_became_active":"2024-03-11T21:48:28.646753+0000","last_became_peered":"2024-03-11T21:48:28.646753+0000","last_unstale":"2024-03-11T21:49:06.863700+0000","last_undegraded":"2024-03-11T21:49:06.863700+0000","last_fullsized":"2024-03-11T21:49:06.863700+0000","mapping_epoch":502,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":503,"parent":"0.0","parent_split_bits":0,"last_scrub":"507'49","last_scrub_stamp":"2024-03-11T21:48:36.103940+0000","last_deep_scrub":"507'49","last_deep_scrub_stamp":"2024-03-11T21:48:36.103940+0000","last_clean_scrub_stamp":"2024-03-11T21:48:36.103940+0000","objects_scrubbed":7,"log_size":49,"log_dups_size":0,"ondisk_log_size":49,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:49:43.760149+0000","scrub_duration":162,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":61,"num_read_kb":54,"num_write":42,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,0],"acting":[5,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.0","version":"438'13","reported_seq":437,"reported_epoch":521,"state":"active+clean","last_fresh":"2024-03-11T21:49:03.824202+0000","last_change":"2024-03-11T21:47:12.660954+0000","last_active":"2024-03-11T21:49:03.824202+0000","last_peered":"2024-03-11T21:49:03.824202+0000","last_clean":"2024-03-11T21:49:03.824202+0000","last_became_active":"2024-03-11T21:44:35.621524+0000","last_became_peered":"2024-03-11T21:44:35.621524+0000","last_unstale":"2024-03-11T21:49:03.824202+0000","last_undegraded":"2024-03-11T21:49:03.824202+0000","last_fullsized":"2024-03-11T21:49:03.824202+0000","mapping_epoch":437,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":438,"parent":"0.0","parent_split_bits":0,"last_scrub":"438'13","last_scrub_stamp":"2024-03-11T21:47:12.660495+0000","last_deep_scrub":"98'9","last_deep_scrub_stamp":"2024-03-11T21:27:29.745711+0000","last_clean_scrub_stamp":"2024-03-11T21:47:12.660495+0000","objects_scrubbed":1,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.c","version":"0'0","reported_seq":609,"reported_epoch":522,"state":"active+clean","last_fresh":"2024-03-11T21:49:05.635499+0000","last_change":"2024-03-11T21:48:28.645298+0000","last_active":"2024-03-11T21:49:05.635499+0000","last_peered":"2024-03-11T21:49:05.635499+0000","last_clean":"2024-03-11T21:49:05.635499+0000","last_became_active":"2024-03-11T21:48:28.644535+0000","last_became_peered":"2024-03-11T21:48:28.644535+0000","last_unstale":"2024-03-11T21:49:05.635499+0000","last_undegraded":"2024-03-11T21:49:05.635499+0000","last_fullsized":"2024-03-11T21:49:05.635499+0000","mapping_epoch":502,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":503,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:48:20.437875+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:43.559452+0000","last_clean_scrub_stamp":"2024-03-11T21:48:20.437875+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:49:34.584798+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,7],"acting":[6,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.f","version":"0'0","reported_seq":59,"reported_epoch":523,"state":"active+clean","last_fresh":"2024-03-11T21:49:06.864419+0000","last_change":"2024-03-11T21:48:57.001226+0000","last_active":"2024-03-11T21:49:06.864419+0000","last_peered":"2024-03-11T21:49:06.864419+0000","last_clean":"2024-03-11T21:49:06.864419+0000","last_became_active":"2024-03-11T21:48:28.640404+0000","last_became_peered":"2024-03-11T21:48:28.640404+0000","last_unstale":"2024-03-11T21:49:06.864419+0000","last_undegraded":"2024-03-11T21:49:06.864419+0000","last_fullsized":"2024-03-11T21:49:06.864419+0000","mapping_epoch":502,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":503,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:48:57.001131+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:43:24.002346+0000","last_clean_scrub_stamp":"2024-03-11T21:48:57.001131+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:50:16.636770+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,2],"acting":[5,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.8","version":"0'0","reported_seq":632,"reported_epoch":523,"state":"active+clean","last_fresh":"2024-03-11T21:49:07.170081+0000","last_change":"2024-03-11T21:45:25.159896+0000","last_active":"2024-03-11T21:49:07.170081+0000","last_peered":"2024-03-11T21:49:07.170081+0000","last_clean":"2024-03-11T21:49:07.170081+0000","last_became_active":"2024-03-11T21:34:41.144681+0000","last_became_peered":"2024-03-11T21:34:41.144681+0000","last_unstale":"2024-03-11T21:49:07.170081+0000","last_undegraded":"2024-03-11T21:49:07.170081+0000","last_fullsized":"2024-03-11T21:49:07.170081+0000","mapping_epoch":226,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":227,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:45:25.159772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:37.512018+0000","last_clean_scrub_stamp":"2024-03-11T21:45:25.159772+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,1],"acting":[7,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.e","version":"456'32","reported_seq":721,"reported_epoch":522,"state":"active+clean","last_fresh":"2024-03-11T21:49:05.635943+0000","last_change":"2024-03-11T21:48:28.684357+0000","last_active":"2024-03-11T21:49:05.635943+0000","last_peered":"2024-03-11T21:49:05.635943+0000","last_clean":"2024-03-11T21:49:05.635943+0000","last_became_active":"2024-03-11T21:48:28.643691+0000","last_became_peered":"2024-03-11T21:48:28.643691+0000","last_unstale":"2024-03-11T21:49:05.635943+0000","last_undegraded":"2024-03-11T21:49:05.635943+0000","last_fullsized":"2024-03-11T21:49:05.635943+0000","mapping_epoch":502,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":503,"parent":"0.0","parent_split_bits":0,"last_scrub":"456'32","last_scrub_stamp":"2024-03-11T21:46:07.696550+0000","last_deep_scrub":"109'16","last_deep_scrub_stamp":"2024-03-11T21:28:24.739395+0000","last_clean_scrub_stamp":"2024-03-11T21:46:07.696550+0000","objects_scrubbed":5,"log_size":32,"log_dups_size":0,"ondisk_log_size":32,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":43,"num_read_kb":36,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,5],"acting":[6,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.d","version":"0'0","reported_seq":51,"reported_epoch":523,"state":"active+clean","last_fresh":"2024-03-11T21:49:06.864016+0000","last_change":"2024-03-11T21:48:28.645808+0000","last_active":"2024-03-11T21:49:06.864016+0000","last_peered":"2024-03-11T21:49:06.864016+0000","last_clean":"2024-03-11T21:49:06.864016+0000","last_became_active":"2024-03-11T21:48:28.645468+0000","last_became_peered":"2024-03-11T21:48:28.645468+0000","last_unstale":"2024-03-11T21:49:06.864016+0000","last_undegraded":"2024-03-11T21:49:06.864016+0000","last_fullsized":"2024-03-11T21:49:06.864016+0000","mapping_epoch":502,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":503,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:48:11.883442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:35:37.182850+0000","last_clean_scrub_stamp":"2024-03-11T21:48:11.883442+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:49:39.147524+0000","scrub_duration":182,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.e","version":"503'11","reported_seq":63,"reported_epoch":523,"state":"active+clean","last_fresh":"2024-03-11T21:49:06.864260+0000","last_change":"2024-03-11T21:48:28.668806+0000","last_active":"2024-03-11T21:49:06.864260+0000","last_peered":"2024-03-11T21:49:06.864260+0000","last_clean":"2024-03-11T21:49:06.864260+0000","last_became_active":"2024-03-11T21:48:28.646363+0000","last_became_peered":"2024-03-11T21:48:28.646363+0000","last_unstale":"2024-03-11T21:49:06.864260+0000","last_undegraded":"2024-03-11T21:49:06.864260+0000","last_fullsized":"2024-03-11T21:49:06.864260+0000","mapping_epoch":502,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":503,"parent":"0.0","parent_split_bits":0,"last_scrub":"474'10","last_scrub_stamp":"2024-03-11T21:48:14.792612+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:41.544550+0000","last_clean_scrub_stamp":"2024-03-11T21:48:14.792612+0000","objects_scrubbed":1,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:49:16.388849+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,2],"acting":[5,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.9","version":"0'0","reported_seq":1012,"reported_epoch":521,"state":"active+clean","last_fresh":"2024-03-11T21:49:03.824580+0000","last_change":"2024-03-11T21:47:21.614548+0000","last_active":"2024-03-11T21:49:03.824580+0000","last_peered":"2024-03-11T21:49:03.824580+0000","last_clean":"2024-03-11T21:49:03.824580+0000","last_became_active":"2024-03-11T21:44:21.454167+0000","last_became_peered":"2024-03-11T21:44:21.454167+0000","last_unstale":"2024-03-11T21:49:03.824580+0000","last_undegraded":"2024-03-11T21:49:03.824580+0000","last_fullsized":"2024-03-11T21:49:03.824580+0000","mapping_epoch":432,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":433,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:47:21.614435+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:26:50.991712+0000","last_clean_scrub_stamp":"2024-03-11T21:47:21.614435+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,2],"acting":[1,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.f","version":"456'39","reported_seq":1045,"reported_epoch":523,"state":"active+clean","last_fresh":"2024-03-11T21:49:06.639652+0000","last_change":"2024-03-11T21:48:30.632818+0000","last_active":"2024-03-11T21:49:06.639652+0000","last_peered":"2024-03-11T21:49:06.639652+0000","last_clean":"2024-03-11T21:49:06.639652+0000","last_became_active":"2024-03-11T21:48:28.636546+0000","last_became_peered":"2024-03-11T21:48:28.636546+0000","last_unstale":"2024-03-11T21:49:06.639652+0000","last_undegraded":"2024-03-11T21:49:06.639652+0000","last_fullsized":"2024-03-11T21:49:06.639652+0000","mapping_epoch":502,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":503,"parent":"0.0","parent_split_bits":0,"last_scrub":"456'39","last_scrub_stamp":"2024-03-11T21:48:30.632717+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:39.525554+0000","last_clean_scrub_stamp":"2024-03-11T21:48:30.632717+0000","objects_scrubbed":7,"log_size":39,"log_dups_size":0,"ondisk_log_size":39,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:49:35.422397+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":62,"num_read_kb":53,"num_write":40,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":34,"num_bytes_recovered":440,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,5],"acting":[2,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.e","version":"0'0","reported_seq":207,"reported_epoch":521,"state":"active+clean","last_fresh":"2024-03-11T21:49:03.824831+0000","last_change":"2024-03-11T21:48:28.621197+0000","last_active":"2024-03-11T21:49:03.824831+0000","last_peered":"2024-03-11T21:49:03.824831+0000","last_clean":"2024-03-11T21:49:03.824831+0000","last_became_active":"2024-03-11T21:48:28.620827+0000","last_became_peered":"2024-03-11T21:48:28.620827+0000","last_unstale":"2024-03-11T21:49:03.824831+0000","last_undegraded":"2024-03-11T21:49:03.824831+0000","last_fullsized":"2024-03-11T21:49:03.824831+0000","mapping_epoch":502,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":503,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:48:24.755205+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:43.559452+0000","last_clean_scrub_stamp":"2024-03-11T21:48:24.755205+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:49:28.240606+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,5],"acting":[1,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.d","version":"503'12","reported_seq":1317,"reported_epoch":523,"state":"active+clean","last_fresh":"2024-03-11T21:49:06.639053+0000","last_change":"2024-03-11T21:49:03.619155+0000","last_active":"2024-03-11T21:49:06.639053+0000","last_peered":"2024-03-11T21:49:06.639053+0000","last_clean":"2024-03-11T21:49:06.639053+0000","last_became_active":"2024-03-11T21:48:28.627926+0000","last_became_peered":"2024-03-11T21:48:28.627926+0000","last_unstale":"2024-03-11T21:49:06.639053+0000","last_undegraded":"2024-03-11T21:49:06.639053+0000","last_fullsized":"2024-03-11T21:49:06.639053+0000","mapping_epoch":502,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":503,"parent":"0.0","parent_split_bits":0,"last_scrub":"503'12","last_scrub_stamp":"2024-03-11T21:49:03.619053+0000","last_deep_scrub":"109'9","last_deep_scrub_stamp":"2024-03-11T21:28:27.082975+0000","last_clean_scrub_stamp":"2024-03-11T21:49:03.619053+0000","objects_scrubbed":1,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:50:24.912555+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6,5],"acting":[2,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.a","version":"0'0","reported_seq":1027,"reported_epoch":523,"state":"active+clean","last_fresh":"2024-03-11T21:49:07.169839+0000","last_change":"2024-03-11T21:48:28.637305+0000","last_active":"2024-03-11T21:49:07.169839+0000","last_peered":"2024-03-11T21:49:07.169839+0000","last_clean":"2024-03-11T21:49:07.169839+0000","last_became_active":"2024-03-11T21:48:28.636997+0000","last_became_peered":"2024-03-11T21:48:28.636997+0000","last_unstale":"2024-03-11T21:49:07.169839+0000","last_undegraded":"2024-03-11T21:49:07.169839+0000","last_fullsized":"2024-03-11T21:49:07.169839+0000","mapping_epoch":502,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":503,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:46:24.954274+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:44:27.996522+0000","last_clean_scrub_stamp":"2024-03-11T21:46:24.954274+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,5],"acting":[7,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.c","version":"456'30","reported_seq":685,"reported_epoch":522,"state":"active+clean","last_fresh":"2024-03-11T21:49:05.635432+0000","last_change":"2024-03-11T21:48:59.479192+0000","last_active":"2024-03-11T21:49:05.635432+0000","last_peered":"2024-03-11T21:49:05.635432+0000","last_clean":"2024-03-11T21:49:05.635432+0000","last_became_active":"2024-03-11T21:44:35.630913+0000","last_became_peered":"2024-03-11T21:44:35.630913+0000","last_unstale":"2024-03-11T21:49:05.635432+0000","last_undegraded":"2024-03-11T21:49:05.635432+0000","last_fullsized":"2024-03-11T21:49:05.635432+0000","mapping_epoch":437,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":438,"parent":"0.0","parent_split_bits":0,"last_scrub":"456'30","last_scrub_stamp":"2024-03-11T21:48:59.479093+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:39.525554+0000","last_clean_scrub_stamp":"2024-03-11T21:48:59.479093+0000","objects_scrubbed":5,"log_size":30,"log_dups_size":0,"ondisk_log_size":30,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:50:01.540725+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":44,"num_read_kb":39,"num_write":30,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,7],"acting":[6,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.f","version":"0'0","reported_seq":1012,"reported_epoch":523,"state":"active+clean","last_fresh":"2024-03-11T21:49:06.639475+0000","last_change":"2024-03-11T21:49:04.810575+0000","last_active":"2024-03-11T21:49:06.639475+0000","last_peered":"2024-03-11T21:49:06.639475+0000","last_clean":"2024-03-11T21:49:06.639475+0000","last_became_active":"2024-03-11T21:44:35.645282+0000","last_became_peered":"2024-03-11T21:44:35.645282+0000","last_unstale":"2024-03-11T21:49:06.639475+0000","last_undegraded":"2024-03-11T21:49:06.639475+0000","last_fullsized":"2024-03-11T21:49:06.639475+0000","mapping_epoch":437,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":438,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:49:04.810478+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:27:03.010969+0000","last_clean_scrub_stamp":"2024-03-11T21:49:04.810478+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:50:15.703232+0000","scrub_duration":197,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6,1],"acting":[2,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":902,"reported_epoch":521,"state":"active+clean","last_fresh":"2024-03-11T21:49:03.632360+0000","last_change":"2024-03-11T21:49:01.121860+0000","last_active":"2024-03-11T21:49:03.632360+0000","last_peered":"2024-03-11T21:49:03.632360+0000","last_clean":"2024-03-11T21:49:03.632360+0000","last_became_active":"2024-03-11T21:47:20.274869+0000","last_became_peered":"2024-03-11T21:47:20.274869+0000","last_unstale":"2024-03-11T21:49:03.632360+0000","last_undegraded":"2024-03-11T21:49:03.632360+0000","last_fullsized":"2024-03-11T21:49:03.632360+0000","mapping_epoch":478,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":479,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:49:01.121783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:30:22.750053+0000","last_clean_scrub_stamp":"2024-03-11T21:49:01.121783+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:50:21.278653+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,6],"acting":[0,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.b","version":"0'0","reported_seq":650,"reported_epoch":523,"state":"active+clean","last_fresh":"2024-03-11T21:49:07.170220+0000","last_change":"2024-03-11T21:48:28.628161+0000","last_active":"2024-03-11T21:49:07.170220+0000","last_peered":"2024-03-11T21:49:07.170220+0000","last_clean":"2024-03-11T21:49:07.170220+0000","last_became_active":"2024-03-11T21:48:28.627574+0000","last_became_peered":"2024-03-11T21:48:28.627574+0000","last_unstale":"2024-03-11T21:49:07.170220+0000","last_undegraded":"2024-03-11T21:49:07.170220+0000","last_fullsized":"2024-03-11T21:49:07.170220+0000","mapping_epoch":502,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":503,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:45:30.101973+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:45:30.101973+0000","last_clean_scrub_stamp":"2024-03-11T21:45:30.101973+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,6],"acting":[7,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.d","version":"456'53","reported_seq":731,"reported_epoch":523,"state":"active+clean","last_fresh":"2024-03-11T21:49:06.639224+0000","last_change":"2024-03-11T21:49:05.623558+0000","last_active":"2024-03-11T21:49:06.639224+0000","last_peered":"2024-03-11T21:49:06.639224+0000","last_clean":"2024-03-11T21:49:06.639224+0000","last_became_active":"2024-03-11T21:44:35.634778+0000","last_became_peered":"2024-03-11T21:44:35.634778+0000","last_unstale":"2024-03-11T21:49:06.639224+0000","last_undegraded":"2024-03-11T21:49:06.639224+0000","last_fullsized":"2024-03-11T21:49:06.639224+0000","mapping_epoch":437,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":438,"parent":"0.0","parent_split_bits":0,"last_scrub":"456'53","last_scrub_stamp":"2024-03-11T21:49:05.623446+0000","last_deep_scrub":"165'25","last_deep_scrub_stamp":"2024-03-11T21:32:21.678328+0000","last_clean_scrub_stamp":"2024-03-11T21:49:05.623446+0000","objects_scrubbed":8,"log_size":53,"log_dups_size":0,"ondisk_log_size":53,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:50:15.360664+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":70,"num_read_kb":60,"num_write":46,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,6],"acting":[2,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":638,"reported_epoch":523,"state":"active+clean","last_fresh":"2024-03-11T21:49:07.169716+0000","last_change":"2024-03-11T21:46:05.407295+0000","last_active":"2024-03-11T21:49:07.169716+0000","last_peered":"2024-03-11T21:49:07.169716+0000","last_clean":"2024-03-11T21:49:07.169716+0000","last_became_active":"2024-03-11T21:44:21.461352+0000","last_became_peered":"2024-03-11T21:44:21.461352+0000","last_unstale":"2024-03-11T21:49:07.169716+0000","last_undegraded":"2024-03-11T21:49:07.169716+0000","last_fullsized":"2024-03-11T21:49:07.169716+0000","mapping_epoch":432,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":433,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:46:05.407166+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:28:37.561826+0000","last_clean_scrub_stamp":"2024-03-11T21:46:05.407166+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,6],"acting":[7,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":1007,"reported_epoch":523,"state":"active+clean","last_fresh":"2024-03-11T21:49:06.638049+0000","last_change":"2024-03-11T21:49:02.635836+0000","last_active":"2024-03-11T21:49:06.638049+0000","last_peered":"2024-03-11T21:49:06.638049+0000","last_clean":"2024-03-11T21:49:06.638049+0000","last_became_active":"2024-03-11T21:48:28.627492+0000","last_became_peered":"2024-03-11T21:48:28.627492+0000","last_unstale":"2024-03-11T21:49:06.638049+0000","last_undegraded":"2024-03-11T21:49:06.638049+0000","last_fullsized":"2024-03-11T21:49:06.638049+0000","mapping_epoch":502,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":503,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:49:02.635765+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:41.544550+0000","last_clean_scrub_stamp":"2024-03-11T21:49:02.635765+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:50:24.699944+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,5],"acting":[2,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.c","version":"0'0","reported_seq":1007,"reported_epoch":523,"state":"active+clean","last_fresh":"2024-03-11T21:49:06.639639+0000","last_change":"2024-03-11T21:48:28.637324+0000","last_active":"2024-03-11T21:49:06.639639+0000","last_peered":"2024-03-11T21:49:06.639639+0000","last_clean":"2024-03-11T21:49:06.639639+0000","last_became_active":"2024-03-11T21:48:28.636860+0000","last_became_peered":"2024-03-11T21:48:28.636860+0000","last_unstale":"2024-03-11T21:49:06.639639+0000","last_undegraded":"2024-03-11T21:49:06.639639+0000","last_fullsized":"2024-03-11T21:49:06.639639+0000","mapping_epoch":502,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":503,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:48:21.649477+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:29:30.279910+0000","last_clean_scrub_stamp":"2024-03-11T21:48:21.649477+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:49:45.765783+0000","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,0],"acting":[2,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.a","version":"456'60","reported_seq":1147,"reported_epoch":522,"state":"active+clean","last_fresh":"2024-03-11T21:49:05.635190+0000","last_change":"2024-03-11T21:45:32.673815+0000","last_active":"2024-03-11T21:49:05.635190+0000","last_peered":"2024-03-11T21:49:05.635190+0000","last_clean":"2024-03-11T21:49:05.635190+0000","last_became_active":"2024-03-11T21:34:41.148863+0000","last_became_peered":"2024-03-11T21:34:41.148863+0000","last_unstale":"2024-03-11T21:49:05.635190+0000","last_undegraded":"2024-03-11T21:49:05.635190+0000","last_fullsized":"2024-03-11T21:49:05.635190+0000","mapping_epoch":226,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":227,"parent":"0.0","parent_split_bits":0,"last_scrub":"260'44","last_scrub_stamp":"2024-03-11T21:45:32.673622+0000","last_deep_scrub":"260'44","last_deep_scrub_stamp":"2024-03-11T21:45:32.673622+0000","last_clean_scrub_stamp":"2024-03-11T21:45:32.673622+0000","objects_scrubbed":9,"log_size":60,"log_dups_size":0,"ondisk_log_size":60,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":80,"num_read_kb":69,"num_write":52,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,7],"acting":[6,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":993,"reported_epoch":521,"state":"active+clean","last_fresh":"2024-03-11T21:49:03.632619+0000","last_change":"2024-03-11T21:48:15.917128+0000","last_active":"2024-03-11T21:49:03.632619+0000","last_peered":"2024-03-11T21:49:03.632619+0000","last_clean":"2024-03-11T21:49:03.632619+0000","last_became_active":"2024-03-11T21:34:41.154507+0000","last_became_peered":"2024-03-11T21:34:41.154507+0000","last_unstale":"2024-03-11T21:49:03.632619+0000","last_undegraded":"2024-03-11T21:49:03.632619+0000","last_fullsized":"2024-03-11T21:49:03.632619+0000","mapping_epoch":226,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":227,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:48:15.916918+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:27:11.361913+0000","last_clean_scrub_stamp":"2024-03-11T21:48:15.916918+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:49:21.099751+0000","scrub_duration":104,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":1008,"reported_epoch":523,"state":"active+clean","last_fresh":"2024-03-11T21:49:06.637467+0000","last_change":"2024-03-11T21:48:31.600504+0000","last_active":"2024-03-11T21:49:06.637467+0000","last_peered":"2024-03-11T21:49:06.637467+0000","last_clean":"2024-03-11T21:49:06.637467+0000","last_became_active":"2024-03-11T21:44:35.631810+0000","last_became_peered":"2024-03-11T21:44:35.631810+0000","last_unstale":"2024-03-11T21:49:06.637467+0000","last_undegraded":"2024-03-11T21:49:06.637467+0000","last_fullsized":"2024-03-11T21:49:06.637467+0000","mapping_epoch":437,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":438,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:48:31.600423+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:43:32.115556+0000","last_clean_scrub_stamp":"2024-03-11T21:48:31.600423+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:49:40.843188+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.d","version":"0'0","reported_seq":1014,"reported_epoch":521,"state":"active+clean","last_fresh":"2024-03-11T21:49:03.824938+0000","last_change":"2024-03-11T21:48:58.963224+0000","last_active":"2024-03-11T21:49:03.824938+0000","last_peered":"2024-03-11T21:49:03.824938+0000","last_clean":"2024-03-11T21:49:03.824938+0000","last_became_active":"2024-03-11T21:44:35.621273+0000","last_became_peered":"2024-03-11T21:44:35.621273+0000","last_unstale":"2024-03-11T21:49:03.824938+0000","last_undegraded":"2024-03-11T21:49:03.824938+0000","last_fullsized":"2024-03-11T21:49:03.824938+0000","mapping_epoch":437,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":438,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:48:58.963145+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:37.512018+0000","last_clean_scrub_stamp":"2024-03-11T21:48:58.963145+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:50:03.979428+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,6],"acting":[1,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.b","version":"456'21","reported_seq":1079,"reported_epoch":521,"state":"active+clean","last_fresh":"2024-03-11T21:49:03.631944+0000","last_change":"2024-03-11T21:48:28.672342+0000","last_active":"2024-03-11T21:49:03.631944+0000","last_peered":"2024-03-11T21:49:03.631944+0000","last_clean":"2024-03-11T21:49:03.631944+0000","last_became_active":"2024-03-11T21:48:28.641509+0000","last_became_peered":"2024-03-11T21:48:28.641509+0000","last_unstale":"2024-03-11T21:49:03.631944+0000","last_undegraded":"2024-03-11T21:49:03.631944+0000","last_fullsized":"2024-03-11T21:49:03.631944+0000","mapping_epoch":502,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":503,"parent":"0.0","parent_split_bits":0,"last_scrub":"260'15","last_scrub_stamp":"2024-03-11T21:45:30.172740+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:39.525554+0000","last_clean_scrub_stamp":"2024-03-11T21:45:30.172740+0000","objects_scrubbed":4,"log_size":21,"log_dups_size":0,"ondisk_log_size":21,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":35,"num_read_kb":29,"num_write":22,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":440,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,5],"acting":[0,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":59,"reported_epoch":523,"state":"active+clean","last_fresh":"2024-03-11T21:49:06.863777+0000","last_change":"2024-03-11T21:48:39.986016+0000","last_active":"2024-03-11T21:49:06.863777+0000","last_peered":"2024-03-11T21:49:06.863777+0000","last_clean":"2024-03-11T21:49:06.863777+0000","last_became_active":"2024-03-11T21:48:28.643247+0000","last_became_peered":"2024-03-11T21:48:28.643247+0000","last_unstale":"2024-03-11T21:49:06.863777+0000","last_undegraded":"2024-03-11T21:49:06.863777+0000","last_fullsized":"2024-03-11T21:49:06.863777+0000","mapping_epoch":502,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":503,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:48:39.985901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:43.559452+0000","last_clean_scrub_stamp":"2024-03-11T21:48:39.985901+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:49:51.884417+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,0],"acting":[5,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.e","version":"0'0","reported_seq":1030,"reported_epoch":523,"state":"active+clean","last_fresh":"2024-03-11T21:49:06.639457+0000","last_change":"2024-03-11T21:48:12.611346+0000","last_active":"2024-03-11T21:49:06.639457+0000","last_peered":"2024-03-11T21:49:06.639457+0000","last_clean":"2024-03-11T21:49:06.639457+0000","last_became_active":"2024-03-11T21:44:21.457534+0000","last_became_peered":"2024-03-11T21:44:21.457534+0000","last_unstale":"2024-03-11T21:49:06.639457+0000","last_undegraded":"2024-03-11T21:49:06.639457+0000","last_fullsized":"2024-03-11T21:49:06.639457+0000","mapping_epoch":432,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":433,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:48:12.611106+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:37.512018+0000","last_clean_scrub_stamp":"2024-03-11T21:48:12.611106+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:49:13.446921+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,6],"acting":[2,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.8","version":"456'46","reported_seq":67,"reported_epoch":523,"state":"active+clean","last_fresh":"2024-03-11T21:49:06.863455+0000","last_change":"2024-03-11T21:48:28.664020+0000","last_active":"2024-03-11T21:49:06.863455+0000","last_peered":"2024-03-11T21:49:06.863455+0000","last_clean":"2024-03-11T21:49:06.863455+0000","last_became_active":"2024-03-11T21:48:28.642553+0000","last_became_peered":"2024-03-11T21:48:28.642553+0000","last_unstale":"2024-03-11T21:49:06.863455+0000","last_undegraded":"2024-03-11T21:49:06.863455+0000","last_fullsized":"2024-03-11T21:49:06.863455+0000","mapping_epoch":502,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":503,"parent":"0.0","parent_split_bits":0,"last_scrub":"456'46","last_scrub_stamp":"2024-03-11T21:46:18.893747+0000","last_deep_scrub":"109'22","last_deep_scrub_stamp":"2024-03-11T21:32:07.573197+0000","last_clean_scrub_stamp":"2024-03-11T21:46:18.893747+0000","objects_scrubbed":7,"log_size":46,"log_dups_size":0,"ondisk_log_size":46,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":61,"num_read_kb":52,"num_write":40,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":28,"num_bytes_recovered":440,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.9","version":"438'12","reported_seq":828,"reported_epoch":523,"state":"active+clean","last_fresh":"2024-03-11T21:49:07.169968+0000","last_change":"2024-03-11T21:49:03.823881+0000","last_active":"2024-03-11T21:49:07.169968+0000","last_peered":"2024-03-11T21:49:07.169968+0000","last_clean":"2024-03-11T21:49:07.169968+0000","last_became_active":"2024-03-11T21:44:35.638711+0000","last_became_peered":"2024-03-11T21:44:35.638711+0000","last_unstale":"2024-03-11T21:49:07.169968+0000","last_undegraded":"2024-03-11T21:49:07.169968+0000","last_fullsized":"2024-03-11T21:49:07.169968+0000","mapping_epoch":437,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":438,"parent":"0.0","parent_split_bits":0,"last_scrub":"438'12","last_scrub_stamp":"2024-03-11T21:49:03.823758+0000","last_deep_scrub":"94'9","last_deep_scrub_stamp":"2024-03-11T21:26:53.422552+0000","last_clean_scrub_stamp":"2024-03-11T21:49:03.823758+0000","objects_scrubbed":1,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:50:16.324839+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,1],"acting":[7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":1531,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":7,"num_read_kb":7,"num_write":10,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":2587,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":98304,"data_stored":6124,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":6,"ondisk_log_size":6,"up":96,"acting":96,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":198,"num_read_kb":294,"num_write":179,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":2951840,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":2375680,"data_stored":2361472,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":128,"ondisk_log_size":128,"up":3,"acting":3,"num_store_stats":4},{"poolid":3,"num_pg":3,"stat_sum":{"num_bytes":408,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":495,"num_read_kb":490,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":38,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":1262,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":8,"ondisk_log_size":8,"up":9,"acting":9,"num_store_stats":8},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":3702,"num_objects":178,"num_object_clones":0,"num_object_copies":534,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":178,"num_whiteouts":0,"num_read":1539,"num_read_kb":1296,"num_write":989,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":451,"num_bytes_recovered":8762,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":544768,"data_stored":14478,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1070,"ondisk_log_size":1070,"up":96,"acting":96,"num_store_stats":8},{"poolid":5,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":17,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":94,"ondisk_log_size":94,"up":96,"acting":96,"num_store_stats":8},{"poolid":6,"num_pg":32,"stat_sum":{"num_bytes":383,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":1136,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":1175,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":4,"ondisk_log_size":4,"up":96,"acting":96,"num_store_stats":8}],"osd_stats":[{"osd":7,"up_from":60,"seq":257698038045,"num_pgs":67,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":862976,"kb_used_data":1588,"kb_used_omap":1,"kb_used_meta":861374,"kb_avail":92882176,"statfs":{"total":95995035648,"available":95111348224,"internally_reserved":0,"allocated":1626112,"data_stored":1127080,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":882047434},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":50,"seq":214748365091,"num_pgs":64,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":871332,"kb_used_data":1624,"kb_used_omap":1,"kb_used_meta":869694,"kb_avail":92873820,"statfs":{"total":95995035648,"available":95102791680,"internally_reserved":0,"allocated":1662976,"data_stored":1129140,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":890567110},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834575164,"num_pgs":67,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1020688,"kb_used_data":1024,"kb_used_omap":1,"kb_used_meta":1019646,"kb_avail":92724464,"statfs":{"total":95995035648,"available":94949851136,"internally_reserved":0,"allocated":1048576,"data_stored":530658,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":1044117957},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738689,"num_pgs":66,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1011532,"kb_used_data":1596,"kb_used_omap":1,"kb_used_meta":1009918,"kb_avail":92733620,"statfs":{"total":95995035648,"available":94959226880,"internally_reserved":0,"allocated":1634304,"data_stored":1121450,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":1034156485},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411639,"num_pgs":62,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":876312,"kb_used_data":1032,"kb_used_omap":1,"kb_used_meta":875262,"kb_avail":92868840,"statfs":{"total":95995035648,"available":95097692160,"internally_reserved":0,"allocated":1056768,"data_stored":545131,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":896268741},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":0,"seq":1189705941143,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":0,"seq":137438953772,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":41,"seq":176093659431,"num_pgs":70,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":876324,"kb_used_data":1044,"kb_used_omap":1,"kb_used_meta":875262,"kb_avail":92868828,"statfs":{"total":95995035648,"available":95097679872,"internally_reserved":0,"allocated":1069056,"data_stored":544211,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":896268741},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":1095,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":46,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1393,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":957,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":574,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":528,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1439,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":92,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":69632,"data_stored":1914,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":81920,"data_stored":2118,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":61440,"data_stored":1568,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":1898,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":1474,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":69632,"data_stored":1870,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":81920,"data_stored":2162,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":1474,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":370,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":383,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":383,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-11T21:49:10.174 DEBUG:teuthology.orchestra.run.smithi092:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid 00db0016-dfed-11ee-95c8-87774f69a715 -- ceph pg dump --format=json 2024-03-11T21:49:10.393 DEBUG:teuthology.orchestra.run.smithi097:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid 00db0016-dfed-11ee-95c8-87774f69a715 -- ceph daemon osd.7 dump_ops_in_flight 2024-03-11T21:49:10.416 INFO:teuthology.orchestra.run.smithi092.stderr:Inferring config /var/lib/ceph/00db0016-dfed-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T21:49:10.634 INFO:teuthology.orchestra.run.smithi097.stderr:Inferring config /var/lib/ceph/00db0016-dfed-11ee-95c8-87774f69a715/mon.b/config 2024-03-11T21:49:11.120 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:49:11 smithi092 ceph-mon[29572]: from='client.16248 v1:172.21.15.92:0/2051250191' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-11T21:49:11.120 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:49:11 smithi092 ceph-mon[29572]: pgmap v1124: 132 pgs: 132 active+clean; 582 KiB data, 5.3 GiB used, 531 GiB / 536 GiB avail; 867 B/s rd, 0 op/s 2024-03-11T21:49:11.120 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:49:11 smithi092 ceph-mon[29572]: 4.18 scrub starts 2024-03-11T21:49:11.120 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:49:11 smithi092 ceph-mon[29572]: 4.18 scrub ok 2024-03-11T21:49:11.120 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:49:11 smithi092 ceph-mon[29572]: 4.9 scrub starts 2024-03-11T21:49:11.120 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:49:11 smithi092 ceph-mon[29572]: 4.9 scrub ok 2024-03-11T21:49:11.120 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:49:11 smithi092 ceph-mon[29572]: purged_snaps scrub starts 2024-03-11T21:49:11.120 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:49:11 smithi092 ceph-mon[29572]: purged_snaps scrub ok 2024-03-11T21:49:11.120 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:49:11 smithi092 ceph-mon[37563]: from='client.16248 v1:172.21.15.92:0/2051250191' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-11T21:49:11.121 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:49:11 smithi092 ceph-mon[37563]: pgmap v1124: 132 pgs: 132 active+clean; 582 KiB data, 5.3 GiB used, 531 GiB / 536 GiB avail; 867 B/s rd, 0 op/s 2024-03-11T21:49:11.121 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:49:11 smithi092 ceph-mon[37563]: 4.18 scrub starts 2024-03-11T21:49:11.121 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:49:11 smithi092 ceph-mon[37563]: 4.18 scrub ok 2024-03-11T21:49:11.121 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:49:11 smithi092 ceph-mon[37563]: 4.9 scrub starts 2024-03-11T21:49:11.121 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:49:11 smithi092 ceph-mon[37563]: 4.9 scrub ok 2024-03-11T21:49:11.121 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:49:11 smithi092 ceph-mon[37563]: purged_snaps scrub starts 2024-03-11T21:49:11.121 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:49:11 smithi092 ceph-mon[37563]: purged_snaps scrub ok 2024-03-11T21:49:11.335 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:49:11 smithi097 ceph-mon[33317]: from='client.16248 v1:172.21.15.92:0/2051250191' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-11T21:49:11.335 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:49:11 smithi097 ceph-mon[33317]: pgmap v1124: 132 pgs: 132 active+clean; 582 KiB data, 5.3 GiB used, 531 GiB / 536 GiB avail; 867 B/s rd, 0 op/s 2024-03-11T21:49:11.335 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:49:11 smithi097 ceph-mon[33317]: 4.18 scrub starts 2024-03-11T21:49:11.335 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:49:11 smithi097 ceph-mon[33317]: 4.18 scrub ok 2024-03-11T21:49:11.336 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:49:11 smithi097 ceph-mon[33317]: 4.9 scrub starts 2024-03-11T21:49:11.336 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:49:11 smithi097 ceph-mon[33317]: 4.9 scrub ok 2024-03-11T21:49:11.336 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:49:11 smithi097 ceph-mon[33317]: purged_snaps scrub starts 2024-03-11T21:49:11.336 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:49:11 smithi097 ceph-mon[33317]: purged_snaps scrub ok 2024-03-11T21:49:12.259 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:49:12 smithi092 ceph-mon[37563]: 5.11 deep-scrub starts 2024-03-11T21:49:12.260 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:49:12 smithi092 ceph-mon[37563]: 5.11 deep-scrub ok 2024-03-11T21:49:12.260 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:49:12 smithi092 ceph-mon[37563]: 2.8 scrub starts 2024-03-11T21:49:12.260 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:49:12 smithi092 ceph-mon[37563]: 2.8 scrub ok 2024-03-11T21:49:12.260 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:49:12 smithi092 ceph-mon[37563]: 4.2 scrub starts 2024-03-11T21:49:12.260 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:49:12 smithi092 ceph-mon[37563]: 4.2 scrub ok 2024-03-11T21:49:12.260 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:49:12 smithi092 ceph-mon[37563]: osdmap e525: 8 total, 8 up, 6 in 2024-03-11T21:49:12.260 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:49:12 smithi092 ceph-mon[29572]: 5.11 deep-scrub starts 2024-03-11T21:49:12.260 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:49:12 smithi092 ceph-mon[29572]: 5.11 deep-scrub ok 2024-03-11T21:49:12.260 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:49:12 smithi092 ceph-mon[29572]: 2.8 scrub starts 2024-03-11T21:49:12.260 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:49:12 smithi092 ceph-mon[29572]: 2.8 scrub ok 2024-03-11T21:49:12.260 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:49:12 smithi092 ceph-mon[29572]: 4.2 scrub starts 2024-03-11T21:49:12.260 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:49:12 smithi092 ceph-mon[29572]: 4.2 scrub ok 2024-03-11T21:49:12.260 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:49:12 smithi092 ceph-mon[29572]: osdmap e525: 8 total, 8 up, 6 in 2024-03-11T21:49:12.293 INFO:teuthology.orchestra.run.smithi097.stdout:{ 2024-03-11T21:49:12.293 INFO:teuthology.orchestra.run.smithi097.stdout: "ops": [], 2024-03-11T21:49:12.293 INFO:teuthology.orchestra.run.smithi097.stdout: "num_ops": 0 2024-03-11T21:49:12.293 INFO:teuthology.orchestra.run.smithi097.stdout:} 2024-03-11T21:49:12.303 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:49:12 smithi097 ceph-mon[33317]: 5.11 deep-scrub starts 2024-03-11T21:49:12.303 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:49:12 smithi097 ceph-mon[33317]: 5.11 deep-scrub ok 2024-03-11T21:49:12.303 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:49:12 smithi097 ceph-mon[33317]: 2.8 scrub starts 2024-03-11T21:49:12.303 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:49:12 smithi097 ceph-mon[33317]: 2.8 scrub ok 2024-03-11T21:49:12.303 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:49:12 smithi097 ceph-mon[33317]: 4.2 scrub starts 2024-03-11T21:49:12.303 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:49:12 smithi097 ceph-mon[33317]: 4.2 scrub ok 2024-03-11T21:49:12.303 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:49:12 smithi097 ceph-mon[33317]: osdmap e525: 8 total, 8 up, 6 in 2024-03-11T21:49:12.573 INFO:teuthology.orchestra.run.smithi092.stdout: 2024-03-11T21:49:12.576 INFO:teuthology.orchestra.run.smithi092.stderr:dumped all 2024-03-11T21:49:12.865 DEBUG:teuthology.orchestra.run.smithi097:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid 00db0016-dfed-11ee-95c8-87774f69a715 -- ceph daemon osd.7 dump_blocked_ops 2024-03-11T21:49:13.107 INFO:teuthology.orchestra.run.smithi097.stderr:Inferring config /var/lib/ceph/00db0016-dfed-11ee-95c8-87774f69a715/mon.b/config 2024-03-11T21:49:13.198 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:49:13 smithi092 ceph-mon[29572]: pgmap v1126: 132 pgs: 132 active+clean; 582 KiB data, 5.3 GiB used, 531 GiB / 536 GiB avail; 868 B/s rd, 0 op/s 2024-03-11T21:49:13.198 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:49:13 smithi092 ceph-mon[29572]: 4.0 scrub starts 2024-03-11T21:49:13.198 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:49:13 smithi092 ceph-mon[29572]: 4.0 scrub ok 2024-03-11T21:49:13.198 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:49:13 smithi092 ceph-mon[29572]: 6.11 scrub starts 2024-03-11T21:49:13.198 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:49:13 smithi092 ceph-mon[29572]: 6.11 scrub ok 2024-03-11T21:49:13.198 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:49:13 smithi092 ceph-mon[37563]: pgmap v1126: 132 pgs: 132 active+clean; 582 KiB data, 5.3 GiB used, 531 GiB / 536 GiB avail; 868 B/s rd, 0 op/s 2024-03-11T21:49:13.198 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:49:13 smithi092 ceph-mon[37563]: 4.0 scrub starts 2024-03-11T21:49:13.198 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:49:13 smithi092 ceph-mon[37563]: 4.0 scrub ok 2024-03-11T21:49:13.198 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:49:13 smithi092 ceph-mon[37563]: 6.11 scrub starts 2024-03-11T21:49:13.198 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:49:13 smithi092 ceph-mon[37563]: 6.11 scrub ok 2024-03-11T21:49:13.227 INFO:teuthology.orchestra.run.smithi092.stdout:{"pg_ready":true,"pg_map":{"version":1126,"stamp":"2024-03-11T21:49:11.529492+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":596392,"num_objects":198,"num_object_clones":0,"num_object_copies":594,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":198,"num_whiteouts":0,"num_read":2239,"num_read_kb":2087,"num_write":1187,"num_write_kb":2461,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":497,"num_bytes_recovered":2964363,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1310,"ondisk_log_size":1310,"up":396,"acting":396,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":396,"num_osds":6,"num_per_pool_osds":6,"num_per_pool_omap_osds":6,"kb":562470912,"kb_used":5519228,"kb_used_data":7980,"kb_used_omap":9,"kb_used_meta":5511158,"kb_avail":556951684,"statfs":{"total":575970213888,"available":570318524416,"internally_reserved":0,"allocated":8171520,"data_stored":5037626,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":9564,"internal_metadata":5643426468},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":5,"num_read_kb":5,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"5.892558"},"pg_stats":[{"pgid":"6.1b","version":"0'0","reported_seq":611,"reported_epoch":523,"state":"active+clean","last_fresh":"2024-03-11T21:49:07.169911+0000","last_change":"2024-03-11T21:48:32.687207+0000","last_active":"2024-03-11T21:49:07.169911+0000","last_peered":"2024-03-11T21:49:07.169911+0000","last_clean":"2024-03-11T21:49:07.169911+0000","last_became_active":"2024-03-11T21:44:21.459293+0000","last_became_peered":"2024-03-11T21:44:21.459293+0000","last_unstale":"2024-03-11T21:49:07.169911+0000","last_undegraded":"2024-03-11T21:49:07.169911+0000","last_fullsized":"2024-03-11T21:49:07.169911+0000","mapping_epoch":432,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":433,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:48:32.687100+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:43.559452+0000","last_clean_scrub_stamp":"2024-03-11T21:48:32.687100+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:49:42.615228+0000","scrub_duration":110,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,1],"acting":[7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":197,"reported_epoch":522,"state":"active+clean","last_fresh":"2024-03-11T21:49:05.634948+0000","last_change":"2024-03-11T21:48:40.419493+0000","last_active":"2024-03-11T21:49:05.634948+0000","last_peered":"2024-03-11T21:49:05.634948+0000","last_clean":"2024-03-11T21:49:05.634948+0000","last_became_active":"2024-03-11T21:44:35.624802+0000","last_became_peered":"2024-03-11T21:44:35.624802+0000","last_unstale":"2024-03-11T21:49:05.634948+0000","last_undegraded":"2024-03-11T21:49:05.634948+0000","last_fullsized":"2024-03-11T21:49:05.634948+0000","mapping_epoch":437,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":438,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:48:40.419361+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:41.544550+0000","last_clean_scrub_stamp":"2024-03-11T21:48:40.419361+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:49:50.714093+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,2],"acting":[6,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.1f","version":"0'0","reported_seq":1018,"reported_epoch":524,"state":"active+clean","last_fresh":"2024-03-11T21:49:08.605543+0000","last_change":"2024-03-11T21:45:34.145990+0000","last_active":"2024-03-11T21:49:08.605543+0000","last_peered":"2024-03-11T21:49:08.605543+0000","last_clean":"2024-03-11T21:49:08.605543+0000","last_became_active":"2024-03-11T21:44:35.644794+0000","last_became_peered":"2024-03-11T21:44:35.644794+0000","last_unstale":"2024-03-11T21:49:08.605543+0000","last_undegraded":"2024-03-11T21:49:08.605543+0000","last_fullsized":"2024-03-11T21:49:08.605543+0000","mapping_epoch":437,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":438,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:45:34.145740+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:37.512018+0000","last_clean_scrub_stamp":"2024-03-11T21:45:34.145740+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,6],"acting":[0,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.19","version":"456'39","reported_seq":403,"reported_epoch":523,"state":"active+clean","last_fresh":"2024-03-11T21:49:06.638638+0000","last_change":"2024-03-11T21:48:50.702062+0000","last_active":"2024-03-11T21:49:06.638638+0000","last_peered":"2024-03-11T21:49:06.638638+0000","last_clean":"2024-03-11T21:49:06.638638+0000","last_became_active":"2024-03-11T21:48:28.627471+0000","last_became_peered":"2024-03-11T21:48:28.627471+0000","last_unstale":"2024-03-11T21:49:06.638638+0000","last_undegraded":"2024-03-11T21:49:06.638638+0000","last_fullsized":"2024-03-11T21:49:06.638638+0000","mapping_epoch":502,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":503,"parent":"0.0","parent_split_bits":0,"last_scrub":"456'39","last_scrub_stamp":"2024-03-11T21:48:50.701931+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:39.525554+0000","last_clean_scrub_stamp":"2024-03-11T21:48:50.701931+0000","objects_scrubbed":7,"log_size":39,"log_dups_size":0,"ondisk_log_size":39,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:50:20.674160+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":61,"num_read_kb":52,"num_write":40,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":21,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,5],"acting":[2,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.1a","version":"0'0","reported_seq":59,"reported_epoch":523,"state":"active+clean","last_fresh":"2024-03-11T21:49:06.863652+0000","last_change":"2024-03-11T21:49:02.060501+0000","last_active":"2024-03-11T21:49:06.863652+0000","last_peered":"2024-03-11T21:49:06.863652+0000","last_clean":"2024-03-11T21:49:06.863652+0000","last_became_active":"2024-03-11T21:48:28.636130+0000","last_became_peered":"2024-03-11T21:48:28.636130+0000","last_unstale":"2024-03-11T21:49:06.863652+0000","last_undegraded":"2024-03-11T21:49:06.863652+0000","last_fullsized":"2024-03-11T21:49:06.863652+0000","mapping_epoch":502,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":503,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:49:02.060422+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:49:02.060422+0000","last_clean_scrub_stamp":"2024-03-11T21:49:02.060422+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:50:31.070186+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,7],"acting":[5,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":51,"reported_epoch":523,"state":"active+clean","last_fresh":"2024-03-11T21:49:06.863592+0000","last_change":"2024-03-11T21:48:28.635845+0000","last_active":"2024-03-11T21:49:06.863592+0000","last_peered":"2024-03-11T21:49:06.863592+0000","last_clean":"2024-03-11T21:49:06.863592+0000","last_became_active":"2024-03-11T21:48:28.635475+0000","last_became_peered":"2024-03-11T21:48:28.635475+0000","last_unstale":"2024-03-11T21:49:06.863592+0000","last_undegraded":"2024-03-11T21:49:06.863592+0000","last_fullsized":"2024-03-11T21:49:06.863592+0000","mapping_epoch":502,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":503,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:46:29.023645+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:41.544550+0000","last_clean_scrub_stamp":"2024-03-11T21:46:29.023645+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,7],"acting":[5,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1e","version":"0'0","reported_seq":633,"reported_epoch":524,"state":"active+clean","last_fresh":"2024-03-11T21:49:08.605700+0000","last_change":"2024-03-11T21:48:28.630141+0000","last_active":"2024-03-11T21:49:08.605700+0000","last_peered":"2024-03-11T21:49:08.605700+0000","last_clean":"2024-03-11T21:49:08.605700+0000","last_became_active":"2024-03-11T21:48:28.629741+0000","last_became_peered":"2024-03-11T21:48:28.629741+0000","last_unstale":"2024-03-11T21:49:08.605700+0000","last_undegraded":"2024-03-11T21:49:08.605700+0000","last_fullsized":"2024-03-11T21:49:08.605700+0000","mapping_epoch":502,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":503,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:48:21.902034+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:37.512018+0000","last_clean_scrub_stamp":"2024-03-11T21:48:21.902034+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:49:25.942096+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.18","version":"456'21","reported_seq":668,"reported_epoch":522,"state":"active+clean","last_fresh":"2024-03-11T21:49:05.635077+0000","last_change":"2024-03-11T21:45:24.708559+0000","last_active":"2024-03-11T21:49:05.635077+0000","last_peered":"2024-03-11T21:49:05.635077+0000","last_clean":"2024-03-11T21:49:05.635077+0000","last_became_active":"2024-03-11T21:44:35.621442+0000","last_became_peered":"2024-03-11T21:44:35.621442+0000","last_unstale":"2024-03-11T21:49:05.635077+0000","last_undegraded":"2024-03-11T21:49:05.635077+0000","last_fullsized":"2024-03-11T21:49:05.635077+0000","mapping_epoch":437,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":438,"parent":"0.0","parent_split_bits":0,"last_scrub":"260'15","last_scrub_stamp":"2024-03-11T21:45:24.708228+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:39.525554+0000","last_clean_scrub_stamp":"2024-03-11T21:45:24.708228+0000","objects_scrubbed":4,"log_size":21,"log_dups_size":0,"ondisk_log_size":21,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":34,"num_read_kb":28,"num_write":22,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,2],"acting":[6,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":51,"reported_epoch":523,"state":"active+clean","last_fresh":"2024-03-11T21:49:06.863632+0000","last_change":"2024-03-11T21:48:28.637297+0000","last_active":"2024-03-11T21:49:06.863632+0000","last_peered":"2024-03-11T21:49:06.863632+0000","last_clean":"2024-03-11T21:49:06.863632+0000","last_became_active":"2024-03-11T21:48:28.636959+0000","last_became_peered":"2024-03-11T21:48:28.636959+0000","last_unstale":"2024-03-11T21:49:06.863632+0000","last_undegraded":"2024-03-11T21:49:06.863632+0000","last_fullsized":"2024-03-11T21:49:06.863632+0000","mapping_epoch":502,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":503,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:47:05.896361+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:32:12.654573+0000","last_clean_scrub_stamp":"2024-03-11T21:47:05.896361+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":83,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,2],"acting":[5,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":645,"reported_epoch":523,"state":"active+clean","last_fresh":"2024-03-11T21:49:07.170096+0000","last_change":"2024-03-11T21:48:22.571238+0000","last_active":"2024-03-11T21:49:07.170096+0000","last_peered":"2024-03-11T21:49:07.170096+0000","last_clean":"2024-03-11T21:49:07.170096+0000","last_became_active":"2024-03-11T21:44:35.626456+0000","last_became_peered":"2024-03-11T21:44:35.626456+0000","last_unstale":"2024-03-11T21:49:07.170096+0000","last_undegraded":"2024-03-11T21:49:07.170096+0000","last_fullsized":"2024-03-11T21:49:07.170096+0000","mapping_epoch":437,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":438,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:48:22.571126+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:41.544550+0000","last_clean_scrub_stamp":"2024-03-11T21:48:22.571126+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:49:47.345185+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,0],"acting":[7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.1d","version":"0'0","reported_seq":1002,"reported_epoch":523,"state":"active+clean","last_fresh":"2024-03-11T21:49:07.170495+0000","last_change":"2024-03-11T21:46:28.543345+0000","last_active":"2024-03-11T21:49:07.170495+0000","last_peered":"2024-03-11T21:49:07.170495+0000","last_clean":"2024-03-11T21:49:07.170495+0000","last_became_active":"2024-03-11T21:25:38.542628+0000","last_became_peered":"2024-03-11T21:25:38.542628+0000","last_unstale":"2024-03-11T21:49:07.170495+0000","last_undegraded":"2024-03-11T21:49:07.170495+0000","last_fullsized":"2024-03-11T21:49:07.170495+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:46:28.543232+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:37.512018+0000","last_clean_scrub_stamp":"2024-03-11T21:46:28.543232+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,0],"acting":[7,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.1b","version":"456'9","reported_seq":653,"reported_epoch":524,"state":"active+clean","last_fresh":"2024-03-11T21:49:08.659682+0000","last_change":"2024-03-11T21:49:04.997757+0000","last_active":"2024-03-11T21:49:08.659682+0000","last_peered":"2024-03-11T21:49:08.659682+0000","last_clean":"2024-03-11T21:49:08.659682+0000","last_became_active":"2024-03-11T21:44:35.639372+0000","last_became_peered":"2024-03-11T21:44:35.639372+0000","last_unstale":"2024-03-11T21:49:08.659682+0000","last_undegraded":"2024-03-11T21:49:08.659682+0000","last_fullsized":"2024-03-11T21:49:08.659682+0000","mapping_epoch":437,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":438,"parent":"0.0","parent_split_bits":0,"last_scrub":"456'9","last_scrub_stamp":"2024-03-11T21:49:04.997665+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:39.525554+0000","last_clean_scrub_stamp":"2024-03-11T21:49:04.997665+0000","objects_scrubbed":2,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:50:30.481692+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":16,"num_read_kb":12,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,7],"acting":[1,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":978,"reported_epoch":524,"state":"active+clean","last_fresh":"2024-03-11T21:49:08.605281+0000","last_change":"2024-03-11T21:45:32.208394+0000","last_active":"2024-03-11T21:49:08.605281+0000","last_peered":"2024-03-11T21:49:08.605281+0000","last_clean":"2024-03-11T21:49:08.605281+0000","last_became_active":"2024-03-11T21:25:44.589786+0000","last_became_peered":"2024-03-11T21:25:44.589786+0000","last_unstale":"2024-03-11T21:49:08.605281+0000","last_undegraded":"2024-03-11T21:49:08.605281+0000","last_fullsized":"2024-03-11T21:49:08.605281+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:45:32.208161+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:43.559452+0000","last_clean_scrub_stamp":"2024-03-11T21:45:32.208161+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,7],"acting":[0,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":51,"reported_epoch":523,"state":"active+clean","last_fresh":"2024-03-11T21:49:06.864012+0000","last_change":"2024-03-11T21:48:28.644627+0000","last_active":"2024-03-11T21:49:06.864012+0000","last_peered":"2024-03-11T21:49:06.864012+0000","last_clean":"2024-03-11T21:49:06.864012+0000","last_became_active":"2024-03-11T21:48:28.644231+0000","last_became_peered":"2024-03-11T21:48:28.644231+0000","last_unstale":"2024-03-11T21:49:06.864012+0000","last_undegraded":"2024-03-11T21:49:06.864012+0000","last_fullsized":"2024-03-11T21:49:06.864012+0000","mapping_epoch":502,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":503,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:46:13.862444+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:43:35.536164+0000","last_clean_scrub_stamp":"2024-03-11T21:46:13.862444+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,7],"acting":[5,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1c","version":"0'0","reported_seq":1014,"reported_epoch":523,"state":"active+clean","last_fresh":"2024-03-11T21:49:07.170451+0000","last_change":"2024-03-11T21:48:57.842300+0000","last_active":"2024-03-11T21:49:07.170451+0000","last_peered":"2024-03-11T21:49:07.170451+0000","last_clean":"2024-03-11T21:49:07.170451+0000","last_became_active":"2024-03-11T21:48:28.628707+0000","last_became_peered":"2024-03-11T21:48:28.628707+0000","last_unstale":"2024-03-11T21:49:07.170451+0000","last_undegraded":"2024-03-11T21:49:07.170451+0000","last_fullsized":"2024-03-11T21:49:07.170451+0000","mapping_epoch":502,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":503,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:48:57.842218+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:28:32.566437+0000","last_clean_scrub_stamp":"2024-03-11T21:48:57.842218+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:49:58.846988+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.1a","version":"456'25","reported_seq":876,"reported_epoch":524,"state":"active+clean","last_fresh":"2024-03-11T21:49:08.604959+0000","last_change":"2024-03-11T21:46:06.451173+0000","last_active":"2024-03-11T21:49:08.604959+0000","last_peered":"2024-03-11T21:49:08.604959+0000","last_clean":"2024-03-11T21:49:08.604959+0000","last_became_active":"2024-03-11T21:44:35.631905+0000","last_became_peered":"2024-03-11T21:44:35.631905+0000","last_unstale":"2024-03-11T21:49:08.604959+0000","last_undegraded":"2024-03-11T21:49:08.604959+0000","last_fullsized":"2024-03-11T21:49:08.604959+0000","mapping_epoch":437,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":438,"parent":"0.0","parent_split_bits":0,"last_scrub":"456'25","last_scrub_stamp":"2024-03-11T21:46:06.451051+0000","last_deep_scrub":"350'19","last_deep_scrub_stamp":"2024-03-11T21:43:25.898708+0000","last_clean_scrub_stamp":"2024-03-11T21:46:06.451051+0000","objects_scrubbed":4,"log_size":25,"log_dups_size":0,"ondisk_log_size":25,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":35,"num_read_kb":29,"num_write":22,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,6],"acting":[0,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.1f","version":"0'0","reported_seq":73,"reported_epoch":524,"state":"active+clean","last_fresh":"2024-03-11T21:49:08.605154+0000","last_change":"2024-03-11T21:48:28.642776+0000","last_active":"2024-03-11T21:49:08.605154+0000","last_peered":"2024-03-11T21:49:08.605154+0000","last_clean":"2024-03-11T21:49:08.605154+0000","last_became_active":"2024-03-11T21:48:28.641925+0000","last_became_peered":"2024-03-11T21:48:28.641925+0000","last_unstale":"2024-03-11T21:49:08.605154+0000","last_undegraded":"2024-03-11T21:49:08.605154+0000","last_fullsized":"2024-03-11T21:49:08.605154+0000","mapping_epoch":502,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":503,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:48:25.151073+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:44:31.472397+0000","last_clean_scrub_stamp":"2024-03-11T21:48:25.151073+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:49:36.645328+0000","scrub_duration":193,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,1],"acting":[0,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.1c","version":"0'0","reported_seq":995,"reported_epoch":523,"state":"active+clean","last_fresh":"2024-03-11T21:49:06.639481+0000","last_change":"2024-03-11T21:48:36.643506+0000","last_active":"2024-03-11T21:49:06.639481+0000","last_peered":"2024-03-11T21:49:06.639481+0000","last_clean":"2024-03-11T21:49:06.639481+0000","last_became_active":"2024-03-11T21:44:35.624752+0000","last_became_peered":"2024-03-11T21:44:35.624752+0000","last_unstale":"2024-03-11T21:49:06.639481+0000","last_undegraded":"2024-03-11T21:49:06.639481+0000","last_fullsized":"2024-03-11T21:49:06.639481+0000","mapping_epoch":437,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":438,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:48:36.643362+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:43:32.021344+0000","last_clean_scrub_stamp":"2024-03-11T21:48:36.643362+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:49:41.697242+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,1],"acting":[2,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1b","version":"0'0","reported_seq":648,"reported_epoch":523,"state":"active+clean","last_fresh":"2024-03-11T21:49:07.169964+0000","last_change":"2024-03-11T21:49:05.759673+0000","last_active":"2024-03-11T21:49:07.169964+0000","last_peered":"2024-03-11T21:49:07.169964+0000","last_clean":"2024-03-11T21:49:07.169964+0000","last_became_active":"2024-03-11T21:48:28.626684+0000","last_became_peered":"2024-03-11T21:48:28.626684+0000","last_unstale":"2024-03-11T21:49:07.169964+0000","last_undegraded":"2024-03-11T21:49:07.169964+0000","last_fullsized":"2024-03-11T21:49:07.169964+0000","mapping_epoch":502,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":503,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:49:05.759541+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:34:27.998724+0000","last_clean_scrub_stamp":"2024-03-11T21:49:05.759541+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:50:12.023817+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,5],"acting":[7,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.1d","version":"456'24","reported_seq":391,"reported_epoch":524,"state":"active+clean","last_fresh":"2024-03-11T21:49:08.659989+0000","last_change":"2024-03-11T21:48:40.889498+0000","last_active":"2024-03-11T21:49:08.659989+0000","last_peered":"2024-03-11T21:49:08.659989+0000","last_clean":"2024-03-11T21:49:08.659989+0000","last_became_active":"2024-03-11T21:48:28.621370+0000","last_became_peered":"2024-03-11T21:48:28.621370+0000","last_unstale":"2024-03-11T21:49:08.659989+0000","last_undegraded":"2024-03-11T21:49:08.659989+0000","last_fullsized":"2024-03-11T21:49:08.659989+0000","mapping_epoch":502,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":503,"parent":"0.0","parent_split_bits":0,"last_scrub":"456'24","last_scrub_stamp":"2024-03-11T21:48:40.889386+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:39.525554+0000","last_clean_scrub_stamp":"2024-03-11T21:48:40.889386+0000","objects_scrubbed":5,"log_size":24,"log_dups_size":0,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:50:04.807403+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":42,"num_read_kb":33,"num_write":26,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":880,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,5],"acting":[1,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1e","version":"0'0","reported_seq":210,"reported_epoch":522,"state":"active+clean","last_fresh":"2024-03-11T21:49:05.635934+0000","last_change":"2024-03-11T21:48:28.642136+0000","last_active":"2024-03-11T21:49:05.635934+0000","last_peered":"2024-03-11T21:49:05.635934+0000","last_clean":"2024-03-11T21:49:05.635934+0000","last_became_active":"2024-03-11T21:48:28.641687+0000","last_became_peered":"2024-03-11T21:48:28.641687+0000","last_unstale":"2024-03-11T21:49:05.635934+0000","last_undegraded":"2024-03-11T21:49:05.635934+0000","last_fullsized":"2024-03-11T21:49:05.635934+0000","mapping_epoch":502,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":503,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:48:23.394601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:43.559452+0000","last_clean_scrub_stamp":"2024-03-11T21:48:23.394601+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:49:52.770231+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,0],"acting":[6,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.1d","version":"0'0","reported_seq":1011,"reported_epoch":524,"state":"active+clean","last_fresh":"2024-03-11T21:49:08.658907+0000","last_change":"2024-03-11T21:48:56.927970+0000","last_active":"2024-03-11T21:49:08.658907+0000","last_peered":"2024-03-11T21:49:08.658907+0000","last_clean":"2024-03-11T21:49:08.658907+0000","last_became_active":"2024-03-11T21:48:28.621503+0000","last_became_peered":"2024-03-11T21:48:28.621503+0000","last_unstale":"2024-03-11T21:49:08.658907+0000","last_undegraded":"2024-03-11T21:49:08.658907+0000","last_fullsized":"2024-03-11T21:49:08.658907+0000","mapping_epoch":502,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":503,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:48:56.927890+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:30:18.678531+0000","last_clean_scrub_stamp":"2024-03-11T21:48:56.927890+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:50:22.010957+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,5],"acting":[1,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1a","version":"0'0","reported_seq":1028,"reported_epoch":522,"state":"active+clean","last_fresh":"2024-03-11T21:49:05.635210+0000","last_change":"2024-03-11T21:48:57.428298+0000","last_active":"2024-03-11T21:49:05.635210+0000","last_peered":"2024-03-11T21:49:05.635210+0000","last_clean":"2024-03-11T21:49:05.635210+0000","last_became_active":"2024-03-11T21:44:35.630761+0000","last_became_peered":"2024-03-11T21:44:35.630761+0000","last_unstale":"2024-03-11T21:49:05.635210+0000","last_undegraded":"2024-03-11T21:49:05.635210+0000","last_fullsized":"2024-03-11T21:49:05.635210+0000","mapping_epoch":437,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":438,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:48:57.428222+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:37.512018+0000","last_clean_scrub_stamp":"2024-03-11T21:48:57.428222+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:50:02.779782+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,2],"acting":[6,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.1c","version":"456'39","reported_seq":1097,"reported_epoch":523,"state":"active+clean","last_fresh":"2024-03-11T21:49:06.639648+0000","last_change":"2024-03-11T21:48:51.712325+0000","last_active":"2024-03-11T21:49:06.639648+0000","last_peered":"2024-03-11T21:49:06.639648+0000","last_clean":"2024-03-11T21:49:06.639648+0000","last_became_active":"2024-03-11T21:44:21.465719+0000","last_became_peered":"2024-03-11T21:44:21.465719+0000","last_unstale":"2024-03-11T21:49:06.639648+0000","last_undegraded":"2024-03-11T21:49:06.639648+0000","last_fullsized":"2024-03-11T21:49:06.639648+0000","mapping_epoch":432,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":433,"parent":"0.0","parent_split_bits":0,"last_scrub":"456'39","last_scrub_stamp":"2024-03-11T21:48:51.712213+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:39.525554+0000","last_clean_scrub_stamp":"2024-03-11T21:48:51.712213+0000","objects_scrubbed":7,"log_size":39,"log_dups_size":0,"ondisk_log_size":39,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:50:09.639358+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":62,"num_read_kb":53,"num_write":40,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,0],"acting":[2,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.1d","version":"0'0","reported_seq":51,"reported_epoch":523,"state":"active+clean","last_fresh":"2024-03-11T21:49:06.864180+0000","last_change":"2024-03-11T21:48:28.637584+0000","last_active":"2024-03-11T21:49:06.864180+0000","last_peered":"2024-03-11T21:49:06.864180+0000","last_clean":"2024-03-11T21:49:06.864180+0000","last_became_active":"2024-03-11T21:48:28.636077+0000","last_became_peered":"2024-03-11T21:48:28.636077+0000","last_unstale":"2024-03-11T21:49:06.864180+0000","last_undegraded":"2024-03-11T21:49:06.864180+0000","last_fullsized":"2024-03-11T21:49:06.864180+0000","mapping_epoch":502,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":503,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:48:23.751807+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:43.559452+0000","last_clean_scrub_stamp":"2024-03-11T21:48:23.751807+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:49:30.913722+0000","scrub_duration":106,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,2],"acting":[5,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.1e","version":"0'0","reported_seq":992,"reported_epoch":524,"state":"active+clean","last_fresh":"2024-03-11T21:49:08.605791+0000","last_change":"2024-03-11T21:46:15.490825+0000","last_active":"2024-03-11T21:49:08.605791+0000","last_peered":"2024-03-11T21:49:08.605791+0000","last_clean":"2024-03-11T21:49:08.605791+0000","last_became_active":"2024-03-11T21:25:42.582959+0000","last_became_peered":"2024-03-11T21:25:42.582959+0000","last_unstale":"2024-03-11T21:49:08.605791+0000","last_undegraded":"2024-03-11T21:49:08.605791+0000","last_fullsized":"2024-03-11T21:49:08.605791+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:46:15.490533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:44:28.222102+0000","last_clean_scrub_stamp":"2024-03-11T21:46:15.490533+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.19","version":"67'2","reported_seq":277,"reported_epoch":522,"state":"active+clean","last_fresh":"2024-03-11T21:49:05.634978+0000","last_change":"2024-03-11T21:48:12.501989+0000","last_active":"2024-03-11T21:49:05.634978+0000","last_peered":"2024-03-11T21:49:05.634978+0000","last_clean":"2024-03-11T21:49:05.634978+0000","last_became_active":"2024-03-11T21:44:21.454547+0000","last_became_peered":"2024-03-11T21:44:21.454547+0000","last_unstale":"2024-03-11T21:49:05.634978+0000","last_undegraded":"2024-03-11T21:49:05.634978+0000","last_fullsized":"2024-03-11T21:49:05.634978+0000","mapping_epoch":432,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":433,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'2","last_scrub_stamp":"2024-03-11T21:48:12.501849+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:37.512018+0000","last_clean_scrub_stamp":"2024-03-11T21:48:12.501849+0000","objects_scrubbed":2,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:49:16.110129+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":957,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":957,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,2],"acting":[6,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.1f","version":"456'32","reported_seq":1110,"reported_epoch":522,"state":"active+clean","last_fresh":"2024-03-11T21:49:05.635822+0000","last_change":"2024-03-11T21:48:28.679601+0000","last_active":"2024-03-11T21:49:05.635822+0000","last_peered":"2024-03-11T21:49:05.635822+0000","last_clean":"2024-03-11T21:49:05.635822+0000","last_became_active":"2024-03-11T21:48:28.643699+0000","last_became_peered":"2024-03-11T21:48:28.643699+0000","last_unstale":"2024-03-11T21:49:05.635822+0000","last_undegraded":"2024-03-11T21:49:05.635822+0000","last_fullsized":"2024-03-11T21:49:05.635822+0000","mapping_epoch":502,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":503,"parent":"0.0","parent_split_bits":0,"last_scrub":"456'32","last_scrub_stamp":"2024-03-11T21:46:16.737252+0000","last_deep_scrub":"124'16","last_deep_scrub_stamp":"2024-03-11T21:35:38.173841+0000","last_clean_scrub_stamp":"2024-03-11T21:46:16.737252+0000","objects_scrubbed":5,"log_size":32,"log_dups_size":0,"ondisk_log_size":32,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":44,"num_read_kb":37,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,0],"acting":[6,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.1c","version":"467'2","reported_seq":656,"reported_epoch":523,"state":"active+clean","last_fresh":"2024-03-11T21:49:07.170428+0000","last_change":"2024-03-11T21:48:28.651514+0000","last_active":"2024-03-11T21:49:07.170428+0000","last_peered":"2024-03-11T21:49:07.170428+0000","last_clean":"2024-03-11T21:49:07.170428+0000","last_became_active":"2024-03-11T21:48:28.628768+0000","last_became_peered":"2024-03-11T21:48:28.628768+0000","last_unstale":"2024-03-11T21:49:07.170428+0000","last_undegraded":"2024-03-11T21:49:07.170428+0000","last_fullsized":"2024-03-11T21:49:07.170428+0000","mapping_epoch":502,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":503,"parent":"0.0","parent_split_bits":0,"last_scrub":"467'2","last_scrub_stamp":"2024-03-11T21:46:22.517102+0000","last_deep_scrub":"467'2","last_deep_scrub_stamp":"2024-03-11T21:46:22.517102+0000","last_clean_scrub_stamp":"2024-03-11T21:46:22.517102+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":370,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":1110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.1f","version":"438'11","reported_seq":1336,"reported_epoch":522,"state":"active+clean","last_fresh":"2024-03-11T21:49:05.635774+0000","last_change":"2024-03-11T21:48:15.506962+0000","last_active":"2024-03-11T21:49:05.635774+0000","last_peered":"2024-03-11T21:49:05.635774+0000","last_clean":"2024-03-11T21:49:05.635774+0000","last_became_active":"2024-03-11T21:44:35.647069+0000","last_became_peered":"2024-03-11T21:44:35.647069+0000","last_unstale":"2024-03-11T21:49:05.635774+0000","last_undegraded":"2024-03-11T21:49:05.635774+0000","last_fullsized":"2024-03-11T21:49:05.635774+0000","mapping_epoch":437,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":438,"parent":"0.0","parent_split_bits":0,"last_scrub":"438'11","last_scrub_stamp":"2024-03-11T21:48:15.506841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:41.544550+0000","last_clean_scrub_stamp":"2024-03-11T21:48:15.506841+0000","objects_scrubbed":1,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:49:32.316775+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,2],"acting":[6,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.18","version":"67'1","reported_seq":63,"reported_epoch":523,"state":"active+clean","last_fresh":"2024-03-11T21:49:06.863438+0000","last_change":"2024-03-11T21:48:37.974397+0000","last_active":"2024-03-11T21:49:06.863438+0000","last_peered":"2024-03-11T21:49:06.863438+0000","last_clean":"2024-03-11T21:49:06.863438+0000","last_became_active":"2024-03-11T21:48:28.640756+0000","last_became_peered":"2024-03-11T21:48:28.640756+0000","last_unstale":"2024-03-11T21:49:06.863438+0000","last_undegraded":"2024-03-11T21:49:06.863438+0000","last_fullsized":"2024-03-11T21:49:06.863438+0000","mapping_epoch":502,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":503,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'1","last_scrub_stamp":"2024-03-11T21:48:37.974310+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:37.512018+0000","last_clean_scrub_stamp":"2024-03-11T21:48:37.974310+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:49:58.741849+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":436,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":7,"num_read_kb":7,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":1308,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,6],"acting":[5,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1e","version":"456'30","reported_seq":1117,"reported_epoch":524,"state":"active+clean","last_fresh":"2024-03-11T21:49:08.605790+0000","last_change":"2024-03-11T21:48:40.055060+0000","last_active":"2024-03-11T21:49:08.605790+0000","last_peered":"2024-03-11T21:49:08.605790+0000","last_clean":"2024-03-11T21:49:08.605790+0000","last_became_active":"2024-03-11T21:44:21.467126+0000","last_became_peered":"2024-03-11T21:44:21.467126+0000","last_unstale":"2024-03-11T21:49:08.605790+0000","last_undegraded":"2024-03-11T21:49:08.605790+0000","last_fullsized":"2024-03-11T21:49:08.605790+0000","mapping_epoch":432,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":433,"parent":"0.0","parent_split_bits":0,"last_scrub":"456'30","last_scrub_stamp":"2024-03-11T21:48:40.054882+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:39.525554+0000","last_clean_scrub_stamp":"2024-03-11T21:48:40.054882+0000","objects_scrubbed":5,"log_size":30,"log_dups_size":0,"ondisk_log_size":30,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:50:01.724130+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":45,"num_read_kb":40,"num_write":30,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,1],"acting":[0,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.13","version":"0'0","reported_seq":552,"reported_epoch":524,"state":"active+clean","last_fresh":"2024-03-11T21:49:08.604701+0000","last_change":"2024-03-11T21:49:02.319830+0000","last_active":"2024-03-11T21:49:08.604701+0000","last_peered":"2024-03-11T21:49:08.604701+0000","last_clean":"2024-03-11T21:49:08.604701+0000","last_became_active":"2024-03-11T21:44:21.465436+0000","last_became_peered":"2024-03-11T21:44:21.465436+0000","last_unstale":"2024-03-11T21:49:08.604701+0000","last_undegraded":"2024-03-11T21:49:08.604701+0000","last_fullsized":"2024-03-11T21:49:08.604701+0000","mapping_epoch":432,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":433,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:49:02.319723+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:43.559452+0000","last_clean_scrub_stamp":"2024-03-11T21:49:02.319723+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:50:15.607012+0000","scrub_duration":149,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,2],"acting":[0,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":646,"reported_epoch":523,"state":"active+clean","last_fresh":"2024-03-11T21:49:07.169820+0000","last_change":"2024-03-11T21:46:24.563883+0000","last_active":"2024-03-11T21:49:07.169820+0000","last_peered":"2024-03-11T21:49:07.169820+0000","last_clean":"2024-03-11T21:49:07.169820+0000","last_became_active":"2024-03-11T21:44:35.627685+0000","last_became_peered":"2024-03-11T21:44:35.627685+0000","last_unstale":"2024-03-11T21:49:07.169820+0000","last_undegraded":"2024-03-11T21:49:07.169820+0000","last_fullsized":"2024-03-11T21:49:07.169820+0000","mapping_epoch":437,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":438,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:46:24.563752+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:43:26.665719+0000","last_clean_scrub_stamp":"2024-03-11T21:46:24.563752+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,0],"acting":[7,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.17","version":"0'0","reported_seq":1014,"reported_epoch":522,"state":"active+clean","last_fresh":"2024-03-11T21:49:05.635643+0000","last_change":"2024-03-11T21:48:28.624441+0000","last_active":"2024-03-11T21:49:05.635643+0000","last_peered":"2024-03-11T21:49:05.635643+0000","last_clean":"2024-03-11T21:49:05.635643+0000","last_became_active":"2024-03-11T21:48:28.623533+0000","last_became_peered":"2024-03-11T21:48:28.623533+0000","last_unstale":"2024-03-11T21:49:05.635643+0000","last_undegraded":"2024-03-11T21:49:05.635643+0000","last_fullsized":"2024-03-11T21:49:05.635643+0000","mapping_epoch":502,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":503,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:45:30.740865+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:45:30.740865+0000","last_clean_scrub_stamp":"2024-03-11T21:45:30.740865+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,2],"acting":[6,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.11","version":"456'27","reported_seq":714,"reported_epoch":523,"state":"active+clean","last_fresh":"2024-03-11T21:49:07.169853+0000","last_change":"2024-03-11T21:48:28.652986+0000","last_active":"2024-03-11T21:49:07.169853+0000","last_peered":"2024-03-11T21:49:07.169853+0000","last_clean":"2024-03-11T21:49:07.169853+0000","last_became_active":"2024-03-11T21:48:28.626324+0000","last_became_peered":"2024-03-11T21:48:28.626324+0000","last_unstale":"2024-03-11T21:49:07.169853+0000","last_undegraded":"2024-03-11T21:49:07.169853+0000","last_fullsized":"2024-03-11T21:49:07.169853+0000","mapping_epoch":502,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":503,"parent":"0.0","parent_split_bits":0,"last_scrub":"456'27","last_scrub_stamp":"2024-03-11T21:46:18.485911+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:39.525554+0000","last_clean_scrub_stamp":"2024-03-11T21:46:18.485911+0000","objects_scrubbed":5,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":44,"num_read_kb":37,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":440,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,5],"acting":[7,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":646,"reported_epoch":523,"state":"active+clean","last_fresh":"2024-03-11T21:49:07.169978+0000","last_change":"2024-03-11T21:48:36.557715+0000","last_active":"2024-03-11T21:49:07.169978+0000","last_peered":"2024-03-11T21:49:07.169978+0000","last_clean":"2024-03-11T21:49:07.169978+0000","last_became_active":"2024-03-11T21:44:35.634181+0000","last_became_peered":"2024-03-11T21:44:35.634181+0000","last_unstale":"2024-03-11T21:49:07.169978+0000","last_undegraded":"2024-03-11T21:49:07.169978+0000","last_fullsized":"2024-03-11T21:49:07.169978+0000","mapping_epoch":437,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":438,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:48:36.557613+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:43.559452+0000","last_clean_scrub_stamp":"2024-03-11T21:48:36.557613+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:49:54.779088+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,1],"acting":[7,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":1021,"reported_epoch":522,"state":"active+clean","last_fresh":"2024-03-11T21:49:05.634988+0000","last_change":"2024-03-11T21:45:39.613711+0000","last_active":"2024-03-11T21:49:05.634988+0000","last_peered":"2024-03-11T21:49:05.634988+0000","last_clean":"2024-03-11T21:49:05.634988+0000","last_became_active":"2024-03-11T21:44:35.630991+0000","last_became_peered":"2024-03-11T21:44:35.630991+0000","last_unstale":"2024-03-11T21:49:05.634988+0000","last_undegraded":"2024-03-11T21:49:05.634988+0000","last_fullsized":"2024-03-11T21:49:05.634988+0000","mapping_epoch":437,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":438,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:45:39.613600+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:43:29.425414+0000","last_clean_scrub_stamp":"2024-03-11T21:45:39.613600+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,0],"acting":[6,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.16","version":"0'0","reported_seq":59,"reported_epoch":523,"state":"active+clean","last_fresh":"2024-03-11T21:49:06.864328+0000","last_change":"2024-03-11T21:48:33.041585+0000","last_active":"2024-03-11T21:49:06.864328+0000","last_peered":"2024-03-11T21:49:06.864328+0000","last_clean":"2024-03-11T21:49:06.864328+0000","last_became_active":"2024-03-11T21:48:28.636823+0000","last_became_peered":"2024-03-11T21:48:28.636823+0000","last_unstale":"2024-03-11T21:49:06.864328+0000","last_undegraded":"2024-03-11T21:49:06.864328+0000","last_fullsized":"2024-03-11T21:49:06.864328+0000","mapping_epoch":502,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":503,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:48:33.041481+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:27:11.938909+0000","last_clean_scrub_stamp":"2024-03-11T21:48:33.041481+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:49:56.250770+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,2],"acting":[5,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.10","version":"456'14","reported_seq":328,"reported_epoch":522,"state":"active+clean","last_fresh":"2024-03-11T21:49:05.635162+0000","last_change":"2024-03-11T21:48:28.675345+0000","last_active":"2024-03-11T21:49:05.635162+0000","last_peered":"2024-03-11T21:49:05.635162+0000","last_clean":"2024-03-11T21:49:05.635162+0000","last_became_active":"2024-03-11T21:48:28.641869+0000","last_became_peered":"2024-03-11T21:48:28.641869+0000","last_unstale":"2024-03-11T21:49:05.635162+0000","last_undegraded":"2024-03-11T21:49:05.635162+0000","last_fullsized":"2024-03-11T21:49:05.635162+0000","mapping_epoch":502,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":503,"parent":"0.0","parent_split_bits":0,"last_scrub":"456'14","last_scrub_stamp":"2024-03-11T21:47:18.601648+0000","last_deep_scrub":"260'10","last_deep_scrub_stamp":"2024-03-11T21:44:32.059058+0000","last_clean_scrub_stamp":"2024-03-11T21:47:18.601648+0000","objects_scrubbed":2,"log_size":14,"log_dups_size":0,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":17,"num_read_kb":15,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,5],"acting":[6,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":609,"reported_epoch":524,"state":"active+clean","last_fresh":"2024-03-11T21:49:08.604485+0000","last_change":"2024-03-11T21:48:28.624430+0000","last_active":"2024-03-11T21:49:08.604485+0000","last_peered":"2024-03-11T21:49:08.604485+0000","last_clean":"2024-03-11T21:49:08.604485+0000","last_became_active":"2024-03-11T21:48:28.623984+0000","last_became_peered":"2024-03-11T21:48:28.623984+0000","last_unstale":"2024-03-11T21:49:08.604485+0000","last_undegraded":"2024-03-11T21:49:08.604485+0000","last_fullsized":"2024-03-11T21:49:08.604485+0000","mapping_epoch":502,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":503,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:45:36.298635+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:43.559452+0000","last_clean_scrub_stamp":"2024-03-11T21:45:36.298635+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":110,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":1010,"reported_epoch":524,"state":"active+clean","last_fresh":"2024-03-11T21:49:08.660057+0000","last_change":"2024-03-11T21:48:28.621244+0000","last_active":"2024-03-11T21:49:08.660057+0000","last_peered":"2024-03-11T21:49:08.660057+0000","last_clean":"2024-03-11T21:49:08.660057+0000","last_became_active":"2024-03-11T21:48:28.620928+0000","last_became_peered":"2024-03-11T21:48:28.620928+0000","last_unstale":"2024-03-11T21:49:08.660057+0000","last_undegraded":"2024-03-11T21:49:08.660057+0000","last_fullsized":"2024-03-11T21:49:08.660057+0000","mapping_epoch":502,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":503,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:48:20.738562+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:27:21.207303+0000","last_clean_scrub_stamp":"2024-03-11T21:48:20.738562+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:49:45.603977+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,7],"acting":[1,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.15","version":"0'0","reported_seq":1010,"reported_epoch":524,"state":"active+clean","last_fresh":"2024-03-11T21:49:08.658968+0000","last_change":"2024-03-11T21:48:36.783179+0000","last_active":"2024-03-11T21:49:08.658968+0000","last_peered":"2024-03-11T21:49:08.658968+0000","last_clean":"2024-03-11T21:49:08.658968+0000","last_became_active":"2024-03-11T21:48:28.620494+0000","last_became_peered":"2024-03-11T21:48:28.620494+0000","last_unstale":"2024-03-11T21:49:08.658968+0000","last_undegraded":"2024-03-11T21:49:08.658968+0000","last_fullsized":"2024-03-11T21:49:08.658968+0000","mapping_epoch":502,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":503,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:48:36.783099+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:26:44.860499+0000","last_clean_scrub_stamp":"2024-03-11T21:48:36.783099+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:49:39.385224+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.13","version":"456'32","reported_seq":738,"reported_epoch":522,"state":"active+clean","last_fresh":"2024-03-11T21:49:05.635376+0000","last_change":"2024-03-11T21:48:28.677318+0000","last_active":"2024-03-11T21:49:05.635376+0000","last_peered":"2024-03-11T21:49:05.635376+0000","last_clean":"2024-03-11T21:49:05.635376+0000","last_became_active":"2024-03-11T21:48:28.642093+0000","last_became_peered":"2024-03-11T21:48:28.642093+0000","last_unstale":"2024-03-11T21:49:05.635376+0000","last_undegraded":"2024-03-11T21:49:05.635376+0000","last_fullsized":"2024-03-11T21:49:05.635376+0000","mapping_epoch":502,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":503,"parent":"0.0","parent_split_bits":0,"last_scrub":"456'32","last_scrub_stamp":"2024-03-11T21:46:13.717488+0000","last_deep_scrub":"260'24","last_deep_scrub_stamp":"2024-03-11T21:40:25.595220+0000","last_clean_scrub_stamp":"2024-03-11T21:46:13.717488+0000","objects_scrubbed":5,"log_size":32,"log_dups_size":0,"ondisk_log_size":32,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":43,"num_read_kb":36,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":440,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,5],"acting":[6,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.10","version":"0'0","reported_seq":991,"reported_epoch":524,"state":"active+clean","last_fresh":"2024-03-11T21:49:08.604332+0000","last_change":"2024-03-11T21:48:28.642867+0000","last_active":"2024-03-11T21:49:08.604332+0000","last_peered":"2024-03-11T21:49:08.604332+0000","last_clean":"2024-03-11T21:49:08.604332+0000","last_became_active":"2024-03-11T21:48:28.641880+0000","last_became_peered":"2024-03-11T21:48:28.641880+0000","last_unstale":"2024-03-11T21:49:08.604332+0000","last_undegraded":"2024-03-11T21:49:08.604332+0000","last_fullsized":"2024-03-11T21:49:08.604332+0000","mapping_epoch":502,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":503,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:45:35.179951+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:27:25.417450+0000","last_clean_scrub_stamp":"2024-03-11T21:45:35.179951+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,1],"acting":[0,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":607,"reported_epoch":524,"state":"active+clean","last_fresh":"2024-03-11T21:49:08.604802+0000","last_change":"2024-03-11T21:48:28.632486+0000","last_active":"2024-03-11T21:49:08.604802+0000","last_peered":"2024-03-11T21:49:08.604802+0000","last_clean":"2024-03-11T21:49:08.604802+0000","last_became_active":"2024-03-11T21:48:28.630899+0000","last_became_peered":"2024-03-11T21:48:28.630899+0000","last_unstale":"2024-03-11T21:49:08.604802+0000","last_undegraded":"2024-03-11T21:49:08.604802+0000","last_fullsized":"2024-03-11T21:49:08.604802+0000","mapping_epoch":502,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":503,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:46:24.352079+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:31:15.367898+0000","last_clean_scrub_stamp":"2024-03-11T21:46:24.352079+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,5],"acting":[0,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.14","version":"0'0","reported_seq":1046,"reported_epoch":522,"state":"active+clean","last_fresh":"2024-03-11T21:49:05.635432+0000","last_change":"2024-03-11T21:48:28.644295+0000","last_active":"2024-03-11T21:49:05.635432+0000","last_peered":"2024-03-11T21:49:05.635432+0000","last_clean":"2024-03-11T21:49:05.635432+0000","last_became_active":"2024-03-11T21:48:28.641764+0000","last_became_peered":"2024-03-11T21:48:28.641764+0000","last_unstale":"2024-03-11T21:49:05.635432+0000","last_undegraded":"2024-03-11T21:49:05.635432+0000","last_fullsized":"2024-03-11T21:49:05.635432+0000","mapping_epoch":502,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":503,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:48:10.491899+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:37.512018+0000","last_clean_scrub_stamp":"2024-03-11T21:48:10.491899+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:49:17.952624+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,7],"acting":[6,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.12","version":"456'21","reported_seq":1029,"reported_epoch":524,"state":"active+clean","last_fresh":"2024-03-11T21:49:08.659906+0000","last_change":"2024-03-11T21:48:27.749527+0000","last_active":"2024-03-11T21:49:08.659906+0000","last_peered":"2024-03-11T21:49:08.659906+0000","last_clean":"2024-03-11T21:49:08.659906+0000","last_became_active":"2024-03-11T21:25:40.547082+0000","last_became_peered":"2024-03-11T21:25:40.547082+0000","last_unstale":"2024-03-11T21:49:08.659906+0000","last_undegraded":"2024-03-11T21:49:08.659906+0000","last_fullsized":"2024-03-11T21:49:08.659906+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"456'21","last_scrub_stamp":"2024-03-11T21:48:27.749435+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:39.525554+0000","last_clean_scrub_stamp":"2024-03-11T21:48:27.749435+0000","objects_scrubbed":4,"log_size":21,"log_dups_size":0,"ondisk_log_size":21,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:49:55.648470+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":35,"num_read_kb":29,"num_write":22,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":211,"reported_epoch":523,"state":"active+clean","last_fresh":"2024-03-11T21:49:06.639534+0000","last_change":"2024-03-11T21:48:28.627694+0000","last_active":"2024-03-11T21:49:06.639534+0000","last_peered":"2024-03-11T21:49:06.639534+0000","last_clean":"2024-03-11T21:49:06.639534+0000","last_became_active":"2024-03-11T21:48:28.627346+0000","last_became_peered":"2024-03-11T21:48:28.627346+0000","last_unstale":"2024-03-11T21:49:06.639534+0000","last_undegraded":"2024-03-11T21:49:06.639534+0000","last_fullsized":"2024-03-11T21:49:06.639534+0000","mapping_epoch":502,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":503,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:48:10.902895+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:27:09.801138+0000","last_clean_scrub_stamp":"2024-03-11T21:48:10.902895+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:49:27.252018+0000","scrub_duration":265,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,0],"acting":[2,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.14","version":"434'13","reported_seq":811,"reported_epoch":523,"state":"active+clean","last_fresh":"2024-03-11T21:49:07.170266+0000","last_change":"2024-03-11T21:48:11.632156+0000","last_active":"2024-03-11T21:49:07.170266+0000","last_peered":"2024-03-11T21:49:07.170266+0000","last_clean":"2024-03-11T21:49:07.170266+0000","last_became_active":"2024-03-11T21:44:21.465719+0000","last_became_peered":"2024-03-11T21:44:21.465719+0000","last_unstale":"2024-03-11T21:49:07.170266+0000","last_undegraded":"2024-03-11T21:49:07.170266+0000","last_fullsized":"2024-03-11T21:49:07.170266+0000","mapping_epoch":432,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":433,"parent":"0.0","parent_split_bits":0,"last_scrub":"434'13","last_scrub_stamp":"2024-03-11T21:48:11.632032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:41.544550+0000","last_clean_scrub_stamp":"2024-03-11T21:48:11.632032+0000","objects_scrubbed":1,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:49:34.840151+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,1],"acting":[7,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.13","version":"0'0","reported_seq":1024,"reported_epoch":524,"state":"active+clean","last_fresh":"2024-03-11T21:49:08.605151+0000","last_change":"2024-03-11T21:48:28.624700+0000","last_active":"2024-03-11T21:49:08.605151+0000","last_peered":"2024-03-11T21:49:08.605151+0000","last_clean":"2024-03-11T21:49:08.605151+0000","last_became_active":"2024-03-11T21:48:28.623947+0000","last_became_peered":"2024-03-11T21:48:28.623947+0000","last_unstale":"2024-03-11T21:49:08.605151+0000","last_undegraded":"2024-03-11T21:49:08.605151+0000","last_fullsized":"2024-03-11T21:49:08.605151+0000","mapping_epoch":502,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":503,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:46:05.444630+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:37.512018+0000","last_clean_scrub_stamp":"2024-03-11T21:46:05.444630+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,5],"acting":[0,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.15","version":"456'25","reported_seq":69,"reported_epoch":523,"state":"active+clean","last_fresh":"2024-03-11T21:49:06.864095+0000","last_change":"2024-03-11T21:48:32.001180+0000","last_active":"2024-03-11T21:49:06.864095+0000","last_peered":"2024-03-11T21:49:06.864095+0000","last_clean":"2024-03-11T21:49:06.864095+0000","last_became_active":"2024-03-11T21:48:28.640657+0000","last_became_peered":"2024-03-11T21:48:28.640657+0000","last_unstale":"2024-03-11T21:49:06.864095+0000","last_undegraded":"2024-03-11T21:49:06.864095+0000","last_fullsized":"2024-03-11T21:49:06.864095+0000","mapping_epoch":502,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":503,"parent":"0.0","parent_split_bits":0,"last_scrub":"456'25","last_scrub_stamp":"2024-03-11T21:48:32.001057+0000","last_deep_scrub":"94'13","last_deep_scrub_stamp":"2024-03-11T21:29:41.853158+0000","last_clean_scrub_stamp":"2024-03-11T21:48:32.001057+0000","objects_scrubbed":4,"log_size":25,"log_dups_size":0,"ondisk_log_size":25,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:49:59.598719+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":34,"num_read_kb":28,"num_write":22,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19,"num_bytes_recovered":440,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.16","version":"0'0","reported_seq":1031,"reported_epoch":524,"state":"active+clean","last_fresh":"2024-03-11T21:49:08.605550+0000","last_change":"2024-03-11T21:48:28.624609+0000","last_active":"2024-03-11T21:49:08.605550+0000","last_peered":"2024-03-11T21:49:08.605550+0000","last_clean":"2024-03-11T21:49:08.605550+0000","last_became_active":"2024-03-11T21:48:28.623242+0000","last_became_peered":"2024-03-11T21:48:28.623242+0000","last_unstale":"2024-03-11T21:49:08.605550+0000","last_undegraded":"2024-03-11T21:49:08.605550+0000","last_fullsized":"2024-03-11T21:49:08.605550+0000","mapping_epoch":502,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":503,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:46:18.525893+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:40:30.249679+0000","last_clean_scrub_stamp":"2024-03-11T21:46:18.525893+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":105,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,5],"acting":[0,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.15","version":"503'12","reported_seq":63,"reported_epoch":523,"state":"active+clean","last_fresh":"2024-03-11T21:49:06.863995+0000","last_change":"2024-03-11T21:48:28.669188+0000","last_active":"2024-03-11T21:49:06.863995+0000","last_peered":"2024-03-11T21:49:06.863995+0000","last_clean":"2024-03-11T21:49:06.863995+0000","last_became_active":"2024-03-11T21:48:28.646407+0000","last_became_peered":"2024-03-11T21:48:28.646407+0000","last_unstale":"2024-03-11T21:49:06.863995+0000","last_undegraded":"2024-03-11T21:49:06.863995+0000","last_fullsized":"2024-03-11T21:49:06.863995+0000","mapping_epoch":502,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":503,"parent":"0.0","parent_split_bits":0,"last_scrub":"98'9","last_scrub_stamp":"2024-03-11T21:46:08.945626+0000","last_deep_scrub":"98'9","last_deep_scrub_stamp":"2024-03-11T21:31:20.624923+0000","last_clean_scrub_stamp":"2024-03-11T21:46:08.945626+0000","objects_scrubbed":1,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.12","version":"0'0","reported_seq":59,"reported_epoch":523,"state":"active+clean","last_fresh":"2024-03-11T21:49:06.863800+0000","last_change":"2024-03-11T21:48:34.025883+0000","last_active":"2024-03-11T21:49:06.863800+0000","last_peered":"2024-03-11T21:49:06.863800+0000","last_clean":"2024-03-11T21:49:06.863800+0000","last_became_active":"2024-03-11T21:48:28.645153+0000","last_became_peered":"2024-03-11T21:48:28.645153+0000","last_unstale":"2024-03-11T21:49:06.863800+0000","last_undegraded":"2024-03-11T21:49:06.863800+0000","last_fullsized":"2024-03-11T21:49:06.863800+0000","mapping_epoch":502,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":503,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:48:34.025798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:48:34.025798+0000","last_clean_scrub_stamp":"2024-03-11T21:48:34.025798+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:49:41.357258+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,0],"acting":[5,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.14","version":"456'30","reported_seq":293,"reported_epoch":523,"state":"active+clean","last_fresh":"2024-03-11T21:49:07.170268+0000","last_change":"2024-03-11T21:47:20.278549+0000","last_active":"2024-03-11T21:49:07.170268+0000","last_peered":"2024-03-11T21:49:07.170268+0000","last_clean":"2024-03-11T21:49:07.170268+0000","last_became_active":"2024-03-11T21:47:20.278192+0000","last_became_peered":"2024-03-11T21:47:20.278192+0000","last_unstale":"2024-03-11T21:49:07.170268+0000","last_undegraded":"2024-03-11T21:49:07.170268+0000","last_fullsized":"2024-03-11T21:49:07.170268+0000","mapping_epoch":478,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":479,"parent":"0.0","parent_split_bits":0,"last_scrub":"456'30","last_scrub_stamp":"2024-03-11T21:47:17.566980+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:39.525554+0000","last_clean_scrub_stamp":"2024-03-11T21:47:17.566980+0000","objects_scrubbed":5,"log_size":30,"log_dups_size":0,"ondisk_log_size":30,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":43,"num_read_kb":38,"num_write":30,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,2],"acting":[7,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.15","version":"0'0","reported_seq":649,"reported_epoch":523,"state":"active+clean","last_fresh":"2024-03-11T21:49:07.170434+0000","last_change":"2024-03-11T21:48:28.628344+0000","last_active":"2024-03-11T21:49:07.170434+0000","last_peered":"2024-03-11T21:49:07.170434+0000","last_clean":"2024-03-11T21:49:07.170434+0000","last_became_active":"2024-03-11T21:48:28.628004+0000","last_became_peered":"2024-03-11T21:48:28.628004+0000","last_unstale":"2024-03-11T21:49:07.170434+0000","last_undegraded":"2024-03-11T21:49:07.170434+0000","last_fullsized":"2024-03-11T21:49:07.170434+0000","mapping_epoch":502,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":503,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:46:07.457807+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:43.559452+0000","last_clean_scrub_stamp":"2024-03-11T21:46:07.457807+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":30,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,5],"acting":[7,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":59,"reported_epoch":523,"state":"active+clean","last_fresh":"2024-03-11T21:49:06.864186+0000","last_change":"2024-03-11T21:48:57.999410+0000","last_active":"2024-03-11T21:49:06.864186+0000","last_peered":"2024-03-11T21:49:06.864186+0000","last_clean":"2024-03-11T21:49:06.864186+0000","last_became_active":"2024-03-11T21:48:28.638080+0000","last_became_peered":"2024-03-11T21:48:28.638080+0000","last_unstale":"2024-03-11T21:49:06.864186+0000","last_undegraded":"2024-03-11T21:49:06.864186+0000","last_fullsized":"2024-03-11T21:49:06.864186+0000","mapping_epoch":502,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":503,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:48:57.999278+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:43:36.585632+0000","last_clean_scrub_stamp":"2024-03-11T21:48:57.999278+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:50:13.474101+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.11","version":"0'0","reported_seq":1021,"reported_epoch":522,"state":"active+clean","last_fresh":"2024-03-11T21:49:05.635100+0000","last_change":"2024-03-11T21:48:28.641925+0000","last_active":"2024-03-11T21:49:05.635100+0000","last_peered":"2024-03-11T21:49:05.635100+0000","last_clean":"2024-03-11T21:49:05.635100+0000","last_became_active":"2024-03-11T21:48:28.641503+0000","last_became_peered":"2024-03-11T21:48:28.641503+0000","last_unstale":"2024-03-11T21:49:05.635100+0000","last_undegraded":"2024-03-11T21:49:05.635100+0000","last_fullsized":"2024-03-11T21:49:05.635100+0000","mapping_epoch":502,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":503,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:46:06.705444+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:37.512018+0000","last_clean_scrub_stamp":"2024-03-11T21:46:06.705444+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,5],"acting":[6,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.17","version":"456'21","reported_seq":692,"reported_epoch":523,"state":"active+clean","last_fresh":"2024-03-11T21:49:07.170612+0000","last_change":"2024-03-11T21:48:17.609503+0000","last_active":"2024-03-11T21:49:07.170612+0000","last_peered":"2024-03-11T21:49:07.170612+0000","last_clean":"2024-03-11T21:49:07.170612+0000","last_became_active":"2024-03-11T21:46:53.270498+0000","last_became_peered":"2024-03-11T21:46:53.270498+0000","last_unstale":"2024-03-11T21:49:07.170612+0000","last_undegraded":"2024-03-11T21:49:07.170612+0000","last_fullsized":"2024-03-11T21:49:07.170612+0000","mapping_epoch":473,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":474,"parent":"0.0","parent_split_bits":0,"last_scrub":"456'21","last_scrub_stamp":"2024-03-11T21:48:17.609367+0000","last_deep_scrub":"97'9","last_deep_scrub_stamp":"2024-03-11T21:27:25.803132+0000","last_clean_scrub_stamp":"2024-03-11T21:48:17.609367+0000","objects_scrubbed":3,"log_size":21,"log_dups_size":0,"ondisk_log_size":21,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:49:30.030597+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":27,"num_read_kb":24,"num_write":18,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,2],"acting":[7,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.14","version":"0'0","reported_seq":1008,"reported_epoch":523,"state":"active+clean","last_fresh":"2024-03-11T21:49:06.639250+0000","last_change":"2024-03-11T21:48:58.629494+0000","last_active":"2024-03-11T21:49:06.639250+0000","last_peered":"2024-03-11T21:49:06.639250+0000","last_clean":"2024-03-11T21:49:06.639250+0000","last_became_active":"2024-03-11T21:44:35.626384+0000","last_became_peered":"2024-03-11T21:44:35.626384+0000","last_unstale":"2024-03-11T21:49:06.639250+0000","last_undegraded":"2024-03-11T21:49:06.639250+0000","last_fullsized":"2024-03-11T21:49:06.639250+0000","mapping_epoch":437,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":438,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:48:58.629390+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:43.559452+0000","last_clean_scrub_stamp":"2024-03-11T21:48:58.629390+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:50:06.090017+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,0],"acting":[2,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":625,"reported_epoch":524,"state":"active+clean","last_fresh":"2024-03-11T21:49:08.660214+0000","last_change":"2024-03-11T21:48:35.744834+0000","last_active":"2024-03-11T21:49:08.660214+0000","last_peered":"2024-03-11T21:49:08.660214+0000","last_clean":"2024-03-11T21:49:08.660214+0000","last_became_active":"2024-03-11T21:44:21.458758+0000","last_became_peered":"2024-03-11T21:44:21.458758+0000","last_unstale":"2024-03-11T21:49:08.660214+0000","last_undegraded":"2024-03-11T21:49:08.660214+0000","last_fullsized":"2024-03-11T21:49:08.660214+0000","mapping_epoch":432,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":433,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:48:35.744749+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:48:35.744749+0000","last_clean_scrub_stamp":"2024-03-11T21:48:35.744749+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:49:46.514481+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,0],"acting":[1,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.10","version":"0'0","reported_seq":1003,"reported_epoch":523,"state":"active+clean","last_fresh":"2024-03-11T21:49:06.636658+0000","last_change":"2024-03-11T21:48:32.585436+0000","last_active":"2024-03-11T21:49:06.636658+0000","last_peered":"2024-03-11T21:49:06.636658+0000","last_clean":"2024-03-11T21:49:06.636658+0000","last_became_active":"2024-03-11T21:25:38.533138+0000","last_became_peered":"2024-03-11T21:25:38.533138+0000","last_unstale":"2024-03-11T21:49:06.636658+0000","last_undegraded":"2024-03-11T21:49:06.636658+0000","last_fullsized":"2024-03-11T21:49:06.636658+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:48:32.585314+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:37.512018+0000","last_clean_scrub_stamp":"2024-03-11T21:48:32.585314+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:49:41.783553+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,0],"acting":[2,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.16","version":"456'25","reported_seq":1098,"reported_epoch":524,"state":"active+clean","last_fresh":"2024-03-11T21:49:08.605643+0000","last_change":"2024-03-11T21:49:07.173179+0000","last_active":"2024-03-11T21:49:08.605643+0000","last_peered":"2024-03-11T21:49:08.605643+0000","last_clean":"2024-03-11T21:49:08.605643+0000","last_became_active":"2024-03-11T21:44:21.463330+0000","last_became_peered":"2024-03-11T21:44:21.463330+0000","last_unstale":"2024-03-11T21:49:08.605643+0000","last_undegraded":"2024-03-11T21:49:08.605643+0000","last_fullsized":"2024-03-11T21:49:08.605643+0000","mapping_epoch":432,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":433,"parent":"0.0","parent_split_bits":0,"last_scrub":"456'25","last_scrub_stamp":"2024-03-11T21:49:07.173047+0000","last_deep_scrub":"369'19","last_deep_scrub_stamp":"2024-03-11T21:41:22.115634+0000","last_clean_scrub_stamp":"2024-03-11T21:49:07.173047+0000","objects_scrubbed":4,"log_size":25,"log_dups_size":0,"ondisk_log_size":25,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:50:29.581232+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":35,"num_read_kb":29,"num_write":22,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.b","version":"0'0","reported_seq":611,"reported_epoch":523,"state":"active+clean","last_fresh":"2024-03-11T21:49:07.170062+0000","last_change":"2024-03-11T21:48:21.597003+0000","last_active":"2024-03-11T21:49:07.170062+0000","last_peered":"2024-03-11T21:49:07.170062+0000","last_clean":"2024-03-11T21:49:07.170062+0000","last_became_active":"2024-03-11T21:44:21.449857+0000","last_became_peered":"2024-03-11T21:44:21.449857+0000","last_unstale":"2024-03-11T21:49:07.170062+0000","last_undegraded":"2024-03-11T21:49:07.170062+0000","last_fullsized":"2024-03-11T21:49:07.170062+0000","mapping_epoch":432,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":433,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:48:21.596882+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:44:35.622488+0000","last_clean_scrub_stamp":"2024-03-11T21:48:21.596882+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:49:49.619849+0000","scrub_duration":54,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,6],"acting":[7,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.f","version":"67'2","reported_seq":233,"reported_epoch":524,"state":"active+clean","last_fresh":"2024-03-11T21:49:08.605367+0000","last_change":"2024-03-11T21:48:28.670177+0000","last_active":"2024-03-11T21:49:08.605367+0000","last_peered":"2024-03-11T21:49:08.605367+0000","last_clean":"2024-03-11T21:49:08.605367+0000","last_became_active":"2024-03-11T21:48:28.641365+0000","last_became_peered":"2024-03-11T21:48:28.641365+0000","last_unstale":"2024-03-11T21:49:08.605367+0000","last_undegraded":"2024-03-11T21:49:08.605367+0000","last_fullsized":"2024-03-11T21:49:08.605367+0000","mapping_epoch":502,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":503,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'2","last_scrub_stamp":"2024-03-11T21:46:20.404697+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:37.512018+0000","last_clean_scrub_stamp":"2024-03-11T21:46:20.404697+0000","objects_scrubbed":2,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":92,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":276,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,5],"acting":[0,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.9","version":"456'29","reported_seq":682,"reported_epoch":524,"state":"active+clean","last_fresh":"2024-03-11T21:49:08.659453+0000","last_change":"2024-03-11T21:46:28.006582+0000","last_active":"2024-03-11T21:49:08.659453+0000","last_peered":"2024-03-11T21:49:08.659453+0000","last_clean":"2024-03-11T21:49:08.659453+0000","last_became_active":"2024-03-11T21:44:35.618968+0000","last_became_peered":"2024-03-11T21:44:35.618968+0000","last_unstale":"2024-03-11T21:49:08.659453+0000","last_undegraded":"2024-03-11T21:49:08.659453+0000","last_fullsized":"2024-03-11T21:49:08.659453+0000","mapping_epoch":437,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":438,"parent":"0.0","parent_split_bits":0,"last_scrub":"456'29","last_scrub_stamp":"2024-03-11T21:46:28.006437+0000","last_deep_scrub":"456'29","last_deep_scrub_stamp":"2024-03-11T21:46:28.006437+0000","last_clean_scrub_stamp":"2024-03-11T21:46:28.006437+0000","objects_scrubbed":5,"log_size":29,"log_dups_size":0,"ondisk_log_size":29,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":42,"num_read_kb":33,"num_write":26,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,0],"acting":[1,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":995,"reported_epoch":523,"state":"active+clean","last_fresh":"2024-03-11T21:49:06.636753+0000","last_change":"2024-03-11T21:48:55.615666+0000","last_active":"2024-03-11T21:49:06.636753+0000","last_peered":"2024-03-11T21:49:06.636753+0000","last_clean":"2024-03-11T21:49:06.636753+0000","last_became_active":"2024-03-11T21:25:42.573987+0000","last_became_peered":"2024-03-11T21:25:42.573987+0000","last_unstale":"2024-03-11T21:49:06.636753+0000","last_undegraded":"2024-03-11T21:49:06.636753+0000","last_fullsized":"2024-03-11T21:49:06.636753+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:48:55.615577+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:41.544550+0000","last_clean_scrub_stamp":"2024-03-11T21:48:55.615577+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:49:58.882101+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.0","version":"0'0","reported_seq":1007,"reported_epoch":524,"state":"active+clean","last_fresh":"2024-03-11T21:49:08.604379+0000","last_change":"2024-03-11T21:48:28.641760+0000","last_active":"2024-03-11T21:49:08.604379+0000","last_peered":"2024-03-11T21:49:08.604379+0000","last_clean":"2024-03-11T21:49:08.604379+0000","last_became_active":"2024-03-11T21:48:28.641428+0000","last_became_peered":"2024-03-11T21:48:28.641428+0000","last_unstale":"2024-03-11T21:49:08.604379+0000","last_undegraded":"2024-03-11T21:49:08.604379+0000","last_fullsized":"2024-03-11T21:49:08.604379+0000","mapping_epoch":502,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":503,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:46:13.467417+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:29:39.498784+0000","last_clean_scrub_stamp":"2024-03-11T21:46:13.467417+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,5],"acting":[0,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":480,"reported_epoch":524,"state":"active+clean","last_fresh":"2024-03-11T21:49:08.604704+0000","last_change":"2024-03-11T21:48:58.177402+0000","last_active":"2024-03-11T21:49:08.604704+0000","last_peered":"2024-03-11T21:49:08.604704+0000","last_clean":"2024-03-11T21:49:08.604704+0000","last_became_active":"2024-03-11T21:47:20.274227+0000","last_became_peered":"2024-03-11T21:47:20.274227+0000","last_unstale":"2024-03-11T21:49:08.604704+0000","last_undegraded":"2024-03-11T21:49:08.604704+0000","last_fullsized":"2024-03-11T21:49:08.604704+0000","mapping_epoch":478,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":479,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:48:58.177316+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:48:58.177316+0000","last_clean_scrub_stamp":"2024-03-11T21:48:58.177316+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:50:24.838411+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,7],"acting":[0,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.2","version":"456'35","reported_seq":1100,"reported_epoch":524,"state":"active+clean","last_fresh":"2024-03-11T21:49:08.658877+0000","last_change":"2024-03-11T21:48:28.660028+0000","last_active":"2024-03-11T21:49:08.658877+0000","last_peered":"2024-03-11T21:49:08.658877+0000","last_clean":"2024-03-11T21:49:08.658877+0000","last_became_active":"2024-03-11T21:48:28.630917+0000","last_became_peered":"2024-03-11T21:48:28.630917+0000","last_unstale":"2024-03-11T21:49:08.658877+0000","last_undegraded":"2024-03-11T21:49:08.658877+0000","last_fullsized":"2024-03-11T21:49:08.658877+0000","mapping_epoch":502,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":503,"parent":"0.0","parent_split_bits":0,"last_scrub":"456'35","last_scrub_stamp":"2024-03-11T21:47:11.668986+0000","last_deep_scrub":"94'15","last_deep_scrub_stamp":"2024-03-11T21:26:49.993060+0000","last_clean_scrub_stamp":"2024-03-11T21:47:11.668986+0000","objects_scrubbed":5,"log_size":35,"log_dups_size":0,"ondisk_log_size":35,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":57,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":45,"num_read_kb":40,"num_write":30,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.3","version":"503'10","reported_seq":1303,"reported_epoch":524,"state":"active+clean","last_fresh":"2024-03-11T21:49:08.605425+0000","last_change":"2024-03-11T21:48:56.097716+0000","last_active":"2024-03-11T21:49:08.605425+0000","last_peered":"2024-03-11T21:49:08.605425+0000","last_clean":"2024-03-11T21:49:08.605425+0000","last_became_active":"2024-03-11T21:48:28.641906+0000","last_became_peered":"2024-03-11T21:48:28.641906+0000","last_unstale":"2024-03-11T21:49:08.605425+0000","last_undegraded":"2024-03-11T21:49:08.605425+0000","last_fullsized":"2024-03-11T21:49:08.605425+0000","mapping_epoch":502,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":503,"parent":"0.0","parent_split_bits":0,"last_scrub":"503'10","last_scrub_stamp":"2024-03-11T21:48:56.097591+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:41.544550+0000","last_clean_scrub_stamp":"2024-03-11T21:48:56.097591+0000","objects_scrubbed":1,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:50:21.404866+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,5],"acting":[0,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":608,"reported_epoch":523,"state":"active+clean","last_fresh":"2024-03-11T21:49:07.170379+0000","last_change":"2024-03-11T21:48:33.604086+0000","last_active":"2024-03-11T21:49:07.170379+0000","last_peered":"2024-03-11T21:49:07.170379+0000","last_clean":"2024-03-11T21:49:07.170379+0000","last_became_active":"2024-03-11T21:48:28.626239+0000","last_became_peered":"2024-03-11T21:48:28.626239+0000","last_unstale":"2024-03-11T21:49:07.170379+0000","last_undegraded":"2024-03-11T21:49:07.170379+0000","last_fullsized":"2024-03-11T21:49:07.170379+0000","mapping_epoch":502,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":503,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:48:33.604004+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:28:30.563712+0000","last_clean_scrub_stamp":"2024-03-11T21:48:33.604004+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:49:34.815358+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,5],"acting":[7,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":51,"reported_epoch":523,"state":"active+clean","last_fresh":"2024-03-11T21:49:06.863875+0000","last_change":"2024-03-11T21:48:28.640601+0000","last_active":"2024-03-11T21:49:06.863875+0000","last_peered":"2024-03-11T21:49:06.863875+0000","last_clean":"2024-03-11T21:49:06.863875+0000","last_became_active":"2024-03-11T21:48:28.640205+0000","last_became_peered":"2024-03-11T21:48:28.640205+0000","last_unstale":"2024-03-11T21:49:06.863875+0000","last_undegraded":"2024-03-11T21:49:06.863875+0000","last_fullsized":"2024-03-11T21:49:06.863875+0000","mapping_epoch":502,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":503,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:48:26.741911+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:37.512018+0000","last_clean_scrub_stamp":"2024-03-11T21:48:26.741911+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:49:53.962330+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.4","version":"479'56","reported_seq":1352,"reported_epoch":523,"state":"active+clean","last_fresh":"2024-03-11T21:49:06.639846+0000","last_change":"2024-03-11T21:48:25.673157+0000","last_active":"2024-03-11T21:49:06.639846+0000","last_peered":"2024-03-11T21:49:06.639846+0000","last_clean":"2024-03-11T21:49:06.639846+0000","last_became_active":"2024-03-11T21:47:20.277794+0000","last_became_peered":"2024-03-11T21:47:20.277794+0000","last_unstale":"2024-03-11T21:49:06.639846+0000","last_undegraded":"2024-03-11T21:49:06.639846+0000","last_fullsized":"2024-03-11T21:49:06.639846+0000","mapping_epoch":478,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":479,"parent":"0.0","parent_split_bits":0,"last_scrub":"479'56","last_scrub_stamp":"2024-03-11T21:48:25.673057+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:39.525554+0000","last_clean_scrub_stamp":"2024-03-11T21:48:25.673057+0000","objects_scrubbed":10,"log_size":56,"log_dups_size":0,"ondisk_log_size":56,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:49:47.157799+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":358,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":83,"num_read_kb":68,"num_write":50,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":358,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,6],"acting":[2,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":990,"reported_epoch":524,"state":"active+clean","last_fresh":"2024-03-11T21:49:08.605175+0000","last_change":"2024-03-11T21:45:37.189812+0000","last_active":"2024-03-11T21:49:08.605175+0000","last_peered":"2024-03-11T21:49:08.605175+0000","last_clean":"2024-03-11T21:49:08.605175+0000","last_became_active":"2024-03-11T21:44:35.620226+0000","last_became_peered":"2024-03-11T21:44:35.620226+0000","last_unstale":"2024-03-11T21:49:08.605175+0000","last_undegraded":"2024-03-11T21:49:08.605175+0000","last_fullsized":"2024-03-11T21:49:08.605175+0000","mapping_epoch":437,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":438,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:45:37.189703+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:41.544550+0000","last_clean_scrub_stamp":"2024-03-11T21:45:37.189703+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,6],"acting":[0,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.5","version":"0'0","reported_seq":655,"reported_epoch":523,"state":"active+clean","last_fresh":"2024-03-11T21:49:07.170320+0000","last_change":"2024-03-11T21:48:55.863838+0000","last_active":"2024-03-11T21:49:07.170320+0000","last_peered":"2024-03-11T21:49:07.170320+0000","last_clean":"2024-03-11T21:49:07.170320+0000","last_became_active":"2024-03-11T21:44:21.454156+0000","last_became_peered":"2024-03-11T21:44:21.454156+0000","last_unstale":"2024-03-11T21:49:07.170320+0000","last_undegraded":"2024-03-11T21:49:07.170320+0000","last_fullsized":"2024-03-11T21:49:07.170320+0000","mapping_epoch":432,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":433,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:48:55.863712+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:43.559452+0000","last_clean_scrub_stamp":"2024-03-11T21:48:55.863712+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:50:13.185580+0000","scrub_duration":32,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,2],"acting":[7,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.0","version":"164'6","reported_seq":855,"reported_epoch":523,"state":"active+clean","last_fresh":"2024-03-11T21:49:07.529146+0000","last_change":"2024-03-11T21:48:53.654907+0000","last_active":"2024-03-11T21:49:07.529146+0000","last_peered":"2024-03-11T21:49:07.529146+0000","last_clean":"2024-03-11T21:49:07.529146+0000","last_became_active":"2024-03-11T21:47:20.277766+0000","last_became_peered":"2024-03-11T21:47:20.277766+0000","last_unstale":"2024-03-11T21:49:07.529146+0000","last_undegraded":"2024-03-11T21:49:07.529146+0000","last_fullsized":"2024-03-11T21:49:07.529146+0000","mapping_epoch":478,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":479,"parent":"0.0","parent_split_bits":0,"last_scrub":"164'6","last_scrub_stamp":"2024-03-11T21:48:53.654817+0000","last_deep_scrub":"164'6","last_deep_scrub_stamp":"2024-03-11T21:32:17.597666+0000","last_clean_scrub_stamp":"2024-03-11T21:48:53.654817+0000","objects_scrubbed":1,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:50:13.153353+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0.000753559,"stat_sum":{"num_bytes":389,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":495,"num_read_kb":490,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,6],"acting":[2,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":1019,"reported_epoch":523,"state":"active+clean","last_fresh":"2024-03-11T21:49:06.638979+0000","last_change":"2024-03-11T21:48:26.692563+0000","last_active":"2024-03-11T21:49:06.638979+0000","last_peered":"2024-03-11T21:49:06.638979+0000","last_clean":"2024-03-11T21:49:06.638979+0000","last_became_active":"2024-03-11T21:44:21.462115+0000","last_became_peered":"2024-03-11T21:44:21.462115+0000","last_unstale":"2024-03-11T21:49:06.638979+0000","last_undegraded":"2024-03-11T21:49:06.638979+0000","last_fullsized":"2024-03-11T21:49:06.638979+0000","mapping_epoch":432,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":433,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:48:26.692463+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:43:28.183315+0000","last_clean_scrub_stamp":"2024-03-11T21:48:26.692463+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:49:44.101622+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.7","version":"456'26","reported_seq":1078,"reported_epoch":524,"state":"active+clean","last_fresh":"2024-03-11T21:49:08.659045+0000","last_change":"2024-03-11T21:48:31.684836+0000","last_active":"2024-03-11T21:49:08.659045+0000","last_peered":"2024-03-11T21:49:08.659045+0000","last_clean":"2024-03-11T21:49:08.659045+0000","last_became_active":"2024-03-11T21:48:28.632227+0000","last_became_peered":"2024-03-11T21:48:28.632227+0000","last_unstale":"2024-03-11T21:49:08.659045+0000","last_undegraded":"2024-03-11T21:49:08.659045+0000","last_fullsized":"2024-03-11T21:49:08.659045+0000","mapping_epoch":502,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":503,"parent":"0.0","parent_split_bits":0,"last_scrub":"456'26","last_scrub_stamp":"2024-03-11T21:48:31.684721+0000","last_deep_scrub":"260'22","last_deep_scrub_stamp":"2024-03-11T21:41:27.232403+0000","last_clean_scrub_stamp":"2024-03-11T21:48:31.684721+0000","objects_scrubbed":5,"log_size":26,"log_dups_size":0,"ondisk_log_size":26,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:49:56.707197+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":42,"num_read_kb":31,"num_write":24,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":660,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":1019,"reported_epoch":523,"state":"active+clean","last_fresh":"2024-03-11T21:49:06.639558+0000","last_change":"2024-03-11T21:48:56.586124+0000","last_active":"2024-03-11T21:49:06.639558+0000","last_peered":"2024-03-11T21:49:06.639558+0000","last_clean":"2024-03-11T21:49:06.639558+0000","last_became_active":"2024-03-11T21:48:28.636457+0000","last_became_peered":"2024-03-11T21:48:28.636457+0000","last_unstale":"2024-03-11T21:49:06.639558+0000","last_undegraded":"2024-03-11T21:49:06.639558+0000","last_fullsized":"2024-03-11T21:49:06.639558+0000","mapping_epoch":502,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":503,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:48:56.586051+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:41.544550+0000","last_clean_scrub_stamp":"2024-03-11T21:48:56.586051+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:50:12.930267+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7],"acting":[2,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.7","version":"124'2","reported_seq":55,"reported_epoch":523,"state":"active+clean","last_fresh":"2024-03-11T21:49:06.864450+0000","last_change":"2024-03-11T21:48:28.652129+0000","last_active":"2024-03-11T21:49:06.864450+0000","last_peered":"2024-03-11T21:49:06.864450+0000","last_clean":"2024-03-11T21:49:06.864450+0000","last_became_active":"2024-03-11T21:48:28.635267+0000","last_became_peered":"2024-03-11T21:48:28.635267+0000","last_unstale":"2024-03-11T21:49:06.864450+0000","last_undegraded":"2024-03-11T21:49:06.864450+0000","last_fullsized":"2024-03-11T21:49:06.864450+0000","mapping_epoch":502,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":503,"parent":"0.0","parent_split_bits":0,"last_scrub":"124'2","last_scrub_stamp":"2024-03-11T21:46:27.025988+0000","last_deep_scrub":"124'2","last_deep_scrub_stamp":"2024-03-11T21:29:34.956908+0000","last_clean_scrub_stamp":"2024-03-11T21:46:27.025988+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":111,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":26,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,7],"acting":[5,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2","version":"71'2","reported_seq":63,"reported_epoch":523,"state":"active+clean","last_fresh":"2024-03-11T21:49:06.863655+0000","last_change":"2024-03-11T21:48:29.012420+0000","last_active":"2024-03-11T21:49:06.863655+0000","last_peered":"2024-03-11T21:49:06.863655+0000","last_clean":"2024-03-11T21:49:06.863655+0000","last_became_active":"2024-03-11T21:48:28.638169+0000","last_became_peered":"2024-03-11T21:48:28.638169+0000","last_unstale":"2024-03-11T21:49:06.863655+0000","last_undegraded":"2024-03-11T21:49:06.863655+0000","last_fullsized":"2024-03-11T21:49:06.863655+0000","mapping_epoch":502,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":503,"parent":"0.0","parent_split_bits":0,"last_scrub":"71'2","last_scrub_stamp":"2024-03-11T21:48:29.012327+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:37.512018+0000","last_clean_scrub_stamp":"2024-03-11T21:48:29.012327+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:49:37.866859+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.0014206449999999999,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":38,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,1],"acting":[5,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":51,"reported_epoch":523,"state":"active+clean","last_fresh":"2024-03-11T21:49:06.863911+0000","last_change":"2024-03-11T21:48:28.637004+0000","last_active":"2024-03-11T21:49:06.863911+0000","last_peered":"2024-03-11T21:49:06.863911+0000","last_clean":"2024-03-11T21:49:06.863911+0000","last_became_active":"2024-03-11T21:48:28.636579+0000","last_became_peered":"2024-03-11T21:48:28.636579+0000","last_unstale":"2024-03-11T21:49:06.863911+0000","last_undegraded":"2024-03-11T21:49:06.863911+0000","last_fullsized":"2024-03-11T21:49:06.863911+0000","mapping_epoch":502,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":503,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:46:17.861505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:37.512018+0000","last_clean_scrub_stamp":"2024-03-11T21:46:17.861505+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"87'128","reported_seq":1112,"reported_epoch":523,"state":"active+clean","last_fresh":"2024-03-11T21:49:07.169654+0000","last_change":"2024-03-11T21:48:15.542785+0000","last_active":"2024-03-11T21:49:07.169654+0000","last_peered":"2024-03-11T21:49:07.169654+0000","last_clean":"2024-03-11T21:49:07.169654+0000","last_became_active":"2024-03-11T21:34:44.164569+0000","last_became_peered":"2024-03-11T21:34:44.164569+0000","last_unstale":"2024-03-11T21:49:07.169654+0000","last_undegraded":"2024-03-11T21:49:07.169654+0000","last_fullsized":"2024-03-11T21:49:07.169654+0000","mapping_epoch":229,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":230,"parent":"0.0","parent_split_bits":0,"last_scrub":"87'128","last_scrub_stamp":"2024-03-11T21:48:15.542640+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:24.261974+0000","last_clean_scrub_stamp":"2024-03-11T21:48:15.542640+0000","objects_scrubbed":2,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:49:44.547543+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":198,"num_read_kb":294,"num_write":179,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":2951840,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.5","version":"456'41","reported_seq":1373,"reported_epoch":522,"state":"active+clean","last_fresh":"2024-03-11T21:49:05.635738+0000","last_change":"2024-03-11T21:48:11.506342+0000","last_active":"2024-03-11T21:49:05.635738+0000","last_peered":"2024-03-11T21:49:05.635738+0000","last_clean":"2024-03-11T21:49:05.635738+0000","last_became_active":"2024-03-11T21:44:35.634664+0000","last_became_peered":"2024-03-11T21:44:35.634664+0000","last_unstale":"2024-03-11T21:49:05.635738+0000","last_undegraded":"2024-03-11T21:49:05.635738+0000","last_fullsized":"2024-03-11T21:49:05.635738+0000","mapping_epoch":437,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":438,"parent":"0.0","parent_split_bits":0,"last_scrub":"456'41","last_scrub_stamp":"2024-03-11T21:48:11.506188+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:39.525554+0000","last_clean_scrub_stamp":"2024-03-11T21:48:11.506188+0000","objects_scrubbed":8,"log_size":41,"log_dups_size":0,"ondisk_log_size":41,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:49:37.999828+0000","scrub_duration":18,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":154,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":61,"num_read_kb":51,"num_write":37,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":154,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,0],"acting":[6,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":652,"reported_epoch":523,"state":"active+clean","last_fresh":"2024-03-11T21:49:07.170348+0000","last_change":"2024-03-11T21:48:28.629217+0000","last_active":"2024-03-11T21:49:07.170348+0000","last_peered":"2024-03-11T21:49:07.170348+0000","last_clean":"2024-03-11T21:49:07.170348+0000","last_became_active":"2024-03-11T21:48:28.627651+0000","last_became_peered":"2024-03-11T21:48:28.627651+0000","last_unstale":"2024-03-11T21:49:07.170348+0000","last_undegraded":"2024-03-11T21:49:07.170348+0000","last_fullsized":"2024-03-11T21:49:07.170348+0000","mapping_epoch":502,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":503,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:46:27.586495+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:32:21.266672+0000","last_clean_scrub_stamp":"2024-03-11T21:46:27.586495+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,5],"acting":[7,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.4","version":"0'0","reported_seq":1007,"reported_epoch":524,"state":"active+clean","last_fresh":"2024-03-11T21:49:08.660032+0000","last_change":"2024-03-11T21:48:28.620977+0000","last_active":"2024-03-11T21:49:08.660032+0000","last_peered":"2024-03-11T21:49:08.660032+0000","last_clean":"2024-03-11T21:49:08.660032+0000","last_became_active":"2024-03-11T21:48:28.620607+0000","last_became_peered":"2024-03-11T21:48:28.620607+0000","last_unstale":"2024-03-11T21:49:08.660032+0000","last_undegraded":"2024-03-11T21:49:08.660032+0000","last_fullsized":"2024-03-11T21:49:08.660032+0000","mapping_epoch":502,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":503,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:47:10.662269+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:43.559452+0000","last_clean_scrub_stamp":"2024-03-11T21:47:10.662269+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,6],"acting":[1,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":1040,"reported_epoch":524,"state":"active+clean","last_fresh":"2024-03-11T21:49:08.604365+0000","last_change":"2024-03-11T21:48:37.960673+0000","last_active":"2024-03-11T21:49:08.604365+0000","last_peered":"2024-03-11T21:49:08.604365+0000","last_clean":"2024-03-11T21:49:08.604365+0000","last_became_active":"2024-03-11T21:48:28.623854+0000","last_became_peered":"2024-03-11T21:48:28.623854+0000","last_unstale":"2024-03-11T21:49:08.604365+0000","last_undegraded":"2024-03-11T21:49:08.604365+0000","last_fullsized":"2024-03-11T21:49:08.604365+0000","mapping_epoch":502,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":503,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:48:37.960594+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:37.512018+0000","last_clean_scrub_stamp":"2024-03-11T21:48:37.960594+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:50:05.148862+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.00089342499999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":633,"reported_epoch":523,"state":"active+clean","last_fresh":"2024-03-11T21:49:07.169608+0000","last_change":"2024-03-11T21:46:19.530489+0000","last_active":"2024-03-11T21:49:07.169608+0000","last_peered":"2024-03-11T21:49:07.169608+0000","last_clean":"2024-03-11T21:49:07.169608+0000","last_became_active":"2024-03-11T21:34:41.149219+0000","last_became_peered":"2024-03-11T21:34:41.149219+0000","last_unstale":"2024-03-11T21:49:07.169608+0000","last_undegraded":"2024-03-11T21:49:07.169608+0000","last_fullsized":"2024-03-11T21:49:07.169608+0000","mapping_epoch":226,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":227,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:46:19.530414+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:43:30.734024+0000","last_clean_scrub_stamp":"2024-03-11T21:46:19.530414+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,0],"acting":[7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.6","version":"456'42","reported_seq":1095,"reported_epoch":524,"state":"active+clean","last_fresh":"2024-03-11T21:49:08.605608+0000","last_change":"2024-03-11T21:46:53.286336+0000","last_active":"2024-03-11T21:49:08.605608+0000","last_peered":"2024-03-11T21:49:08.605608+0000","last_clean":"2024-03-11T21:49:08.605608+0000","last_became_active":"2024-03-11T21:46:53.263790+0000","last_became_peered":"2024-03-11T21:46:53.263790+0000","last_unstale":"2024-03-11T21:49:08.605608+0000","last_undegraded":"2024-03-11T21:49:08.605608+0000","last_fullsized":"2024-03-11T21:49:08.605608+0000","mapping_epoch":473,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":474,"parent":"0.0","parent_split_bits":0,"last_scrub":"456'42","last_scrub_stamp":"2024-03-11T21:46:10.509062+0000","last_deep_scrub":"352'30","last_deep_scrub_stamp":"2024-03-11T21:40:31.175323+0000","last_clean_scrub_stamp":"2024-03-11T21:46:10.509062+0000","objects_scrubbed":6,"log_size":42,"log_dups_size":0,"ondisk_log_size":42,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":54,"num_read_kb":48,"num_write":36,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":59,"reported_epoch":523,"state":"active+clean","last_fresh":"2024-03-11T21:49:06.864336+0000","last_change":"2024-03-11T21:48:40.968279+0000","last_active":"2024-03-11T21:49:06.864336+0000","last_peered":"2024-03-11T21:49:06.864336+0000","last_clean":"2024-03-11T21:49:06.864336+0000","last_became_active":"2024-03-11T21:48:28.644298+0000","last_became_peered":"2024-03-11T21:48:28.644298+0000","last_unstale":"2024-03-11T21:49:06.864336+0000","last_undegraded":"2024-03-11T21:49:06.864336+0000","last_fullsized":"2024-03-11T21:49:06.864336+0000","mapping_epoch":502,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":503,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:48:40.968182+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:41.544550+0000","last_clean_scrub_stamp":"2024-03-11T21:48:40.968182+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:49:43.376979+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":996,"reported_epoch":524,"state":"active+clean","last_fresh":"2024-03-11T21:49:08.659634+0000","last_change":"2024-03-11T21:49:08.012276+0000","last_active":"2024-03-11T21:49:08.659634+0000","last_peered":"2024-03-11T21:49:08.659634+0000","last_clean":"2024-03-11T21:49:08.659634+0000","last_became_active":"2024-03-11T21:25:45.301416+0000","last_became_peered":"2024-03-11T21:25:45.301416+0000","last_unstale":"2024-03-11T21:49:08.659634+0000","last_undegraded":"2024-03-11T21:49:08.659634+0000","last_fullsized":"2024-03-11T21:49:08.659634+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:49:08.012181+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:43.559452+0000","last_clean_scrub_stamp":"2024-03-11T21:49:08.012181+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:50:33.936623+0000","scrub_duration":30,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":646,"reported_epoch":523,"state":"active+clean","last_fresh":"2024-03-11T21:49:07.170378+0000","last_change":"2024-03-11T21:48:53.772518+0000","last_active":"2024-03-11T21:49:07.170378+0000","last_peered":"2024-03-11T21:49:07.170378+0000","last_clean":"2024-03-11T21:49:07.170378+0000","last_became_active":"2024-03-11T21:44:35.624442+0000","last_became_peered":"2024-03-11T21:44:35.624442+0000","last_unstale":"2024-03-11T21:49:07.170378+0000","last_undegraded":"2024-03-11T21:49:07.170378+0000","last_fullsized":"2024-03-11T21:49:07.170378+0000","mapping_epoch":437,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":438,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:48:53.772368+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:26:57.333669+0000","last_clean_scrub_stamp":"2024-03-11T21:48:53.772368+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:50:06.313649+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,1],"acting":[7,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.3","version":"456'47","reported_seq":1166,"reported_epoch":524,"state":"active+clean","last_fresh":"2024-03-11T21:49:08.605044+0000","last_change":"2024-03-11T21:48:28.662544+0000","last_active":"2024-03-11T21:49:08.605044+0000","last_peered":"2024-03-11T21:49:08.605044+0000","last_clean":"2024-03-11T21:49:08.605044+0000","last_became_active":"2024-03-11T21:48:28.623917+0000","last_became_peered":"2024-03-11T21:48:28.623917+0000","last_unstale":"2024-03-11T21:49:08.605044+0000","last_undegraded":"2024-03-11T21:49:08.605044+0000","last_fullsized":"2024-03-11T21:49:08.605044+0000","mapping_epoch":502,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":503,"parent":"0.0","parent_split_bits":0,"last_scrub":"456'47","last_scrub_stamp":"2024-03-11T21:46:27.467220+0000","last_deep_scrub":"165'27","last_deep_scrub_stamp":"2024-03-11T21:32:21.576732+0000","last_clean_scrub_stamp":"2024-03-11T21:46:27.467220+0000","objects_scrubbed":8,"log_size":47,"log_dups_size":0,"ondisk_log_size":47,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":69,"num_read_kb":55,"num_write":42,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":32,"num_bytes_recovered":1320,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":1024,"reported_epoch":522,"state":"active+clean","last_fresh":"2024-03-11T21:49:05.635300+0000","last_change":"2024-03-11T21:48:29.376975+0000","last_active":"2024-03-11T21:49:05.635300+0000","last_peered":"2024-03-11T21:49:05.635300+0000","last_clean":"2024-03-11T21:49:05.635300+0000","last_became_active":"2024-03-11T21:48:28.642354+0000","last_became_peered":"2024-03-11T21:48:28.642354+0000","last_unstale":"2024-03-11T21:49:05.635300+0000","last_undegraded":"2024-03-11T21:49:05.635300+0000","last_fullsized":"2024-03-11T21:49:05.635300+0000","mapping_epoch":502,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":503,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:48:29.376888+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:41:21.857205+0000","last_clean_scrub_stamp":"2024-03-11T21:48:29.376888+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:49:59.106992+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":998,"reported_epoch":523,"state":"active+clean","last_fresh":"2024-03-11T21:49:06.637652+0000","last_change":"2024-03-11T21:48:34.576513+0000","last_active":"2024-03-11T21:49:06.637652+0000","last_peered":"2024-03-11T21:49:06.637652+0000","last_clean":"2024-03-11T21:49:06.637652+0000","last_became_active":"2024-03-11T21:44:35.628373+0000","last_became_peered":"2024-03-11T21:44:35.628373+0000","last_unstale":"2024-03-11T21:49:06.637652+0000","last_undegraded":"2024-03-11T21:49:06.637652+0000","last_fullsized":"2024-03-11T21:49:06.637652+0000","mapping_epoch":437,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":438,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:48:34.576398+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:43.559452+0000","last_clean_scrub_stamp":"2024-03-11T21:48:34.576398+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:49:43.761917+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6,7],"acting":[2,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.6","version":"67'1","reported_seq":1010,"reported_epoch":524,"state":"active+clean","last_fresh":"2024-03-11T21:49:08.659766+0000","last_change":"2024-03-11T21:48:19.797460+0000","last_active":"2024-03-11T21:49:08.659766+0000","last_peered":"2024-03-11T21:49:08.659766+0000","last_clean":"2024-03-11T21:49:08.659766+0000","last_became_active":"2024-03-11T21:44:35.618830+0000","last_became_peered":"2024-03-11T21:44:35.618830+0000","last_unstale":"2024-03-11T21:49:08.659766+0000","last_undegraded":"2024-03-11T21:49:08.659766+0000","last_fullsized":"2024-03-11T21:49:08.659766+0000","mapping_epoch":437,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":438,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'1","last_scrub_stamp":"2024-03-11T21:48:19.796812+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:37.512018+0000","last_clean_scrub_stamp":"2024-03-11T21:48:19.796812+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:49:47.201709+0000","scrub_duration":21,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":46,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,0],"acting":[1,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.0","version":"456'50","reported_seq":732,"reported_epoch":523,"state":"active+clean","last_fresh":"2024-03-11T21:49:07.169753+0000","last_change":"2024-03-11T21:45:39.242650+0000","last_active":"2024-03-11T21:49:07.169753+0000","last_peered":"2024-03-11T21:49:07.169753+0000","last_clean":"2024-03-11T21:49:07.169753+0000","last_became_active":"2024-03-11T21:44:21.449580+0000","last_became_peered":"2024-03-11T21:44:21.449580+0000","last_unstale":"2024-03-11T21:49:07.169753+0000","last_undegraded":"2024-03-11T21:49:07.169753+0000","last_fullsized":"2024-03-11T21:49:07.169753+0000","mapping_epoch":432,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":433,"parent":"0.0","parent_split_bits":0,"last_scrub":"260'38","last_scrub_stamp":"2024-03-11T21:45:39.241846+0000","last_deep_scrub":"260'38","last_deep_scrub_stamp":"2024-03-11T21:45:39.241846+0000","last_clean_scrub_stamp":"2024-03-11T21:45:39.241846+0000","objects_scrubbed":8,"log_size":50,"log_dups_size":0,"ondisk_log_size":50,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":70,"num_read_kb":58,"num_write":44,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":440,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,1],"acting":[7,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":1020,"reported_epoch":523,"state":"active+clean","last_fresh":"2024-03-11T21:49:07.169974+0000","last_change":"2024-03-11T21:48:28.630225+0000","last_active":"2024-03-11T21:49:07.169974+0000","last_peered":"2024-03-11T21:49:07.169974+0000","last_clean":"2024-03-11T21:49:07.169974+0000","last_became_active":"2024-03-11T21:48:28.629828+0000","last_became_peered":"2024-03-11T21:48:28.629828+0000","last_unstale":"2024-03-11T21:49:07.169974+0000","last_undegraded":"2024-03-11T21:49:07.169974+0000","last_fullsized":"2024-03-11T21:49:07.169974+0000","mapping_epoch":502,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":503,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:46:09.504070+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:40:21.518840+0000","last_clean_scrub_stamp":"2024-03-11T21:46:09.504070+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,5],"acting":[7,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.3","version":"0'0","reported_seq":632,"reported_epoch":523,"state":"active+clean","last_fresh":"2024-03-11T21:49:07.170184+0000","last_change":"2024-03-11T21:46:25.645839+0000","last_active":"2024-03-11T21:49:07.170184+0000","last_peered":"2024-03-11T21:49:07.170184+0000","last_clean":"2024-03-11T21:49:07.170184+0000","last_became_active":"2024-03-11T21:34:41.142683+0000","last_became_peered":"2024-03-11T21:34:41.142683+0000","last_unstale":"2024-03-11T21:49:07.170184+0000","last_undegraded":"2024-03-11T21:49:07.170184+0000","last_fullsized":"2024-03-11T21:49:07.170184+0000","mapping_epoch":226,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":227,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:46:25.645722+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:46:25.645722+0000","last_clean_scrub_stamp":"2024-03-11T21:46:25.645722+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":44,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,2],"acting":[7,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":1031,"reported_epoch":522,"state":"active+clean","last_fresh":"2024-03-11T21:49:05.635649+0000","last_change":"2024-03-11T21:48:39.453675+0000","last_active":"2024-03-11T21:49:05.635649+0000","last_peered":"2024-03-11T21:49:05.635649+0000","last_clean":"2024-03-11T21:49:05.635649+0000","last_became_active":"2024-03-11T21:34:41.145628+0000","last_became_peered":"2024-03-11T21:34:41.145628+0000","last_unstale":"2024-03-11T21:49:05.635649+0000","last_undegraded":"2024-03-11T21:49:05.635649+0000","last_fullsized":"2024-03-11T21:49:05.635649+0000","mapping_epoch":226,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":227,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:48:39.453573+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:37.512018+0000","last_clean_scrub_stamp":"2024-03-11T21:48:39.453573+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:49:59.564982+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,2],"acting":[6,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.1","version":"507'49","reported_seq":82,"reported_epoch":523,"state":"active+clean","last_fresh":"2024-03-11T21:49:06.863700+0000","last_change":"2024-03-11T21:48:36.104153+0000","last_active":"2024-03-11T21:49:06.863700+0000","last_peered":"2024-03-11T21:49:06.863700+0000","last_clean":"2024-03-11T21:49:06.863700+0000","last_became_active":"2024-03-11T21:48:28.646753+0000","last_became_peered":"2024-03-11T21:48:28.646753+0000","last_unstale":"2024-03-11T21:49:06.863700+0000","last_undegraded":"2024-03-11T21:49:06.863700+0000","last_fullsized":"2024-03-11T21:49:06.863700+0000","mapping_epoch":502,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":503,"parent":"0.0","parent_split_bits":0,"last_scrub":"507'49","last_scrub_stamp":"2024-03-11T21:48:36.103940+0000","last_deep_scrub":"507'49","last_deep_scrub_stamp":"2024-03-11T21:48:36.103940+0000","last_clean_scrub_stamp":"2024-03-11T21:48:36.103940+0000","objects_scrubbed":7,"log_size":49,"log_dups_size":0,"ondisk_log_size":49,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:49:43.760149+0000","scrub_duration":162,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":61,"num_read_kb":54,"num_write":42,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,0],"acting":[5,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.0","version":"438'13","reported_seq":444,"reported_epoch":524,"state":"active+clean","last_fresh":"2024-03-11T21:49:08.659281+0000","last_change":"2024-03-11T21:47:12.660954+0000","last_active":"2024-03-11T21:49:08.659281+0000","last_peered":"2024-03-11T21:49:08.659281+0000","last_clean":"2024-03-11T21:49:08.659281+0000","last_became_active":"2024-03-11T21:44:35.621524+0000","last_became_peered":"2024-03-11T21:44:35.621524+0000","last_unstale":"2024-03-11T21:49:08.659281+0000","last_undegraded":"2024-03-11T21:49:08.659281+0000","last_fullsized":"2024-03-11T21:49:08.659281+0000","mapping_epoch":437,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":438,"parent":"0.0","parent_split_bits":0,"last_scrub":"438'13","last_scrub_stamp":"2024-03-11T21:47:12.660495+0000","last_deep_scrub":"98'9","last_deep_scrub_stamp":"2024-03-11T21:27:29.745711+0000","last_clean_scrub_stamp":"2024-03-11T21:47:12.660495+0000","objects_scrubbed":1,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.c","version":"0'0","reported_seq":609,"reported_epoch":522,"state":"active+clean","last_fresh":"2024-03-11T21:49:05.635499+0000","last_change":"2024-03-11T21:48:28.645298+0000","last_active":"2024-03-11T21:49:05.635499+0000","last_peered":"2024-03-11T21:49:05.635499+0000","last_clean":"2024-03-11T21:49:05.635499+0000","last_became_active":"2024-03-11T21:48:28.644535+0000","last_became_peered":"2024-03-11T21:48:28.644535+0000","last_unstale":"2024-03-11T21:49:05.635499+0000","last_undegraded":"2024-03-11T21:49:05.635499+0000","last_fullsized":"2024-03-11T21:49:05.635499+0000","mapping_epoch":502,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":503,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:48:20.437875+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:43.559452+0000","last_clean_scrub_stamp":"2024-03-11T21:48:20.437875+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:49:34.584798+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,7],"acting":[6,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.f","version":"0'0","reported_seq":59,"reported_epoch":523,"state":"active+clean","last_fresh":"2024-03-11T21:49:06.864419+0000","last_change":"2024-03-11T21:48:57.001226+0000","last_active":"2024-03-11T21:49:06.864419+0000","last_peered":"2024-03-11T21:49:06.864419+0000","last_clean":"2024-03-11T21:49:06.864419+0000","last_became_active":"2024-03-11T21:48:28.640404+0000","last_became_peered":"2024-03-11T21:48:28.640404+0000","last_unstale":"2024-03-11T21:49:06.864419+0000","last_undegraded":"2024-03-11T21:49:06.864419+0000","last_fullsized":"2024-03-11T21:49:06.864419+0000","mapping_epoch":502,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":503,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:48:57.001131+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:43:24.002346+0000","last_clean_scrub_stamp":"2024-03-11T21:48:57.001131+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:50:16.636770+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,2],"acting":[5,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.8","version":"0'0","reported_seq":632,"reported_epoch":523,"state":"active+clean","last_fresh":"2024-03-11T21:49:07.170081+0000","last_change":"2024-03-11T21:45:25.159896+0000","last_active":"2024-03-11T21:49:07.170081+0000","last_peered":"2024-03-11T21:49:07.170081+0000","last_clean":"2024-03-11T21:49:07.170081+0000","last_became_active":"2024-03-11T21:34:41.144681+0000","last_became_peered":"2024-03-11T21:34:41.144681+0000","last_unstale":"2024-03-11T21:49:07.170081+0000","last_undegraded":"2024-03-11T21:49:07.170081+0000","last_fullsized":"2024-03-11T21:49:07.170081+0000","mapping_epoch":226,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":227,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:45:25.159772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:37.512018+0000","last_clean_scrub_stamp":"2024-03-11T21:45:25.159772+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,1],"acting":[7,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.e","version":"456'32","reported_seq":721,"reported_epoch":522,"state":"active+clean","last_fresh":"2024-03-11T21:49:05.635943+0000","last_change":"2024-03-11T21:48:28.684357+0000","last_active":"2024-03-11T21:49:05.635943+0000","last_peered":"2024-03-11T21:49:05.635943+0000","last_clean":"2024-03-11T21:49:05.635943+0000","last_became_active":"2024-03-11T21:48:28.643691+0000","last_became_peered":"2024-03-11T21:48:28.643691+0000","last_unstale":"2024-03-11T21:49:05.635943+0000","last_undegraded":"2024-03-11T21:49:05.635943+0000","last_fullsized":"2024-03-11T21:49:05.635943+0000","mapping_epoch":502,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":503,"parent":"0.0","parent_split_bits":0,"last_scrub":"456'32","last_scrub_stamp":"2024-03-11T21:46:07.696550+0000","last_deep_scrub":"109'16","last_deep_scrub_stamp":"2024-03-11T21:28:24.739395+0000","last_clean_scrub_stamp":"2024-03-11T21:46:07.696550+0000","objects_scrubbed":5,"log_size":32,"log_dups_size":0,"ondisk_log_size":32,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":43,"num_read_kb":36,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,5],"acting":[6,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.d","version":"0'0","reported_seq":51,"reported_epoch":523,"state":"active+clean","last_fresh":"2024-03-11T21:49:06.864016+0000","last_change":"2024-03-11T21:48:28.645808+0000","last_active":"2024-03-11T21:49:06.864016+0000","last_peered":"2024-03-11T21:49:06.864016+0000","last_clean":"2024-03-11T21:49:06.864016+0000","last_became_active":"2024-03-11T21:48:28.645468+0000","last_became_peered":"2024-03-11T21:48:28.645468+0000","last_unstale":"2024-03-11T21:49:06.864016+0000","last_undegraded":"2024-03-11T21:49:06.864016+0000","last_fullsized":"2024-03-11T21:49:06.864016+0000","mapping_epoch":502,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":503,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:48:11.883442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:35:37.182850+0000","last_clean_scrub_stamp":"2024-03-11T21:48:11.883442+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:49:39.147524+0000","scrub_duration":182,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.e","version":"503'11","reported_seq":63,"reported_epoch":523,"state":"active+clean","last_fresh":"2024-03-11T21:49:06.864260+0000","last_change":"2024-03-11T21:48:28.668806+0000","last_active":"2024-03-11T21:49:06.864260+0000","last_peered":"2024-03-11T21:49:06.864260+0000","last_clean":"2024-03-11T21:49:06.864260+0000","last_became_active":"2024-03-11T21:48:28.646363+0000","last_became_peered":"2024-03-11T21:48:28.646363+0000","last_unstale":"2024-03-11T21:49:06.864260+0000","last_undegraded":"2024-03-11T21:49:06.864260+0000","last_fullsized":"2024-03-11T21:49:06.864260+0000","mapping_epoch":502,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":503,"parent":"0.0","parent_split_bits":0,"last_scrub":"474'10","last_scrub_stamp":"2024-03-11T21:48:14.792612+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:41.544550+0000","last_clean_scrub_stamp":"2024-03-11T21:48:14.792612+0000","objects_scrubbed":1,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:49:16.388849+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,2],"acting":[5,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.9","version":"0'0","reported_seq":1018,"reported_epoch":524,"state":"active+clean","last_fresh":"2024-03-11T21:49:08.659558+0000","last_change":"2024-03-11T21:47:21.614548+0000","last_active":"2024-03-11T21:49:08.659558+0000","last_peered":"2024-03-11T21:49:08.659558+0000","last_clean":"2024-03-11T21:49:08.659558+0000","last_became_active":"2024-03-11T21:44:21.454167+0000","last_became_peered":"2024-03-11T21:44:21.454167+0000","last_unstale":"2024-03-11T21:49:08.659558+0000","last_undegraded":"2024-03-11T21:49:08.659558+0000","last_fullsized":"2024-03-11T21:49:08.659558+0000","mapping_epoch":432,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":433,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:47:21.614435+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:26:50.991712+0000","last_clean_scrub_stamp":"2024-03-11T21:47:21.614435+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,2],"acting":[1,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.f","version":"456'39","reported_seq":1045,"reported_epoch":523,"state":"active+clean","last_fresh":"2024-03-11T21:49:06.639652+0000","last_change":"2024-03-11T21:48:30.632818+0000","last_active":"2024-03-11T21:49:06.639652+0000","last_peered":"2024-03-11T21:49:06.639652+0000","last_clean":"2024-03-11T21:49:06.639652+0000","last_became_active":"2024-03-11T21:48:28.636546+0000","last_became_peered":"2024-03-11T21:48:28.636546+0000","last_unstale":"2024-03-11T21:49:06.639652+0000","last_undegraded":"2024-03-11T21:49:06.639652+0000","last_fullsized":"2024-03-11T21:49:06.639652+0000","mapping_epoch":502,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":503,"parent":"0.0","parent_split_bits":0,"last_scrub":"456'39","last_scrub_stamp":"2024-03-11T21:48:30.632717+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:39.525554+0000","last_clean_scrub_stamp":"2024-03-11T21:48:30.632717+0000","objects_scrubbed":7,"log_size":39,"log_dups_size":0,"ondisk_log_size":39,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:49:35.422397+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":62,"num_read_kb":53,"num_write":40,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":34,"num_bytes_recovered":440,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,5],"acting":[2,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.e","version":"0'0","reported_seq":214,"reported_epoch":524,"state":"active+clean","last_fresh":"2024-03-11T21:49:08.658994+0000","last_change":"2024-03-11T21:48:28.621197+0000","last_active":"2024-03-11T21:49:08.658994+0000","last_peered":"2024-03-11T21:49:08.658994+0000","last_clean":"2024-03-11T21:49:08.658994+0000","last_became_active":"2024-03-11T21:48:28.620827+0000","last_became_peered":"2024-03-11T21:48:28.620827+0000","last_unstale":"2024-03-11T21:49:08.658994+0000","last_undegraded":"2024-03-11T21:49:08.658994+0000","last_fullsized":"2024-03-11T21:49:08.658994+0000","mapping_epoch":502,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":503,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:48:24.755205+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:43.559452+0000","last_clean_scrub_stamp":"2024-03-11T21:48:24.755205+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:49:28.240606+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,5],"acting":[1,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.d","version":"503'12","reported_seq":1317,"reported_epoch":523,"state":"active+clean","last_fresh":"2024-03-11T21:49:06.639053+0000","last_change":"2024-03-11T21:49:03.619155+0000","last_active":"2024-03-11T21:49:06.639053+0000","last_peered":"2024-03-11T21:49:06.639053+0000","last_clean":"2024-03-11T21:49:06.639053+0000","last_became_active":"2024-03-11T21:48:28.627926+0000","last_became_peered":"2024-03-11T21:48:28.627926+0000","last_unstale":"2024-03-11T21:49:06.639053+0000","last_undegraded":"2024-03-11T21:49:06.639053+0000","last_fullsized":"2024-03-11T21:49:06.639053+0000","mapping_epoch":502,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":503,"parent":"0.0","parent_split_bits":0,"last_scrub":"503'12","last_scrub_stamp":"2024-03-11T21:49:03.619053+0000","last_deep_scrub":"109'9","last_deep_scrub_stamp":"2024-03-11T21:28:27.082975+0000","last_clean_scrub_stamp":"2024-03-11T21:49:03.619053+0000","objects_scrubbed":1,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:50:24.912555+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6,5],"acting":[2,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.a","version":"0'0","reported_seq":1027,"reported_epoch":523,"state":"active+clean","last_fresh":"2024-03-11T21:49:07.169839+0000","last_change":"2024-03-11T21:48:28.637305+0000","last_active":"2024-03-11T21:49:07.169839+0000","last_peered":"2024-03-11T21:49:07.169839+0000","last_clean":"2024-03-11T21:49:07.169839+0000","last_became_active":"2024-03-11T21:48:28.636997+0000","last_became_peered":"2024-03-11T21:48:28.636997+0000","last_unstale":"2024-03-11T21:49:07.169839+0000","last_undegraded":"2024-03-11T21:49:07.169839+0000","last_fullsized":"2024-03-11T21:49:07.169839+0000","mapping_epoch":502,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":503,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:46:24.954274+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:44:27.996522+0000","last_clean_scrub_stamp":"2024-03-11T21:46:24.954274+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,5],"acting":[7,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.c","version":"456'30","reported_seq":685,"reported_epoch":522,"state":"active+clean","last_fresh":"2024-03-11T21:49:05.635432+0000","last_change":"2024-03-11T21:48:59.479192+0000","last_active":"2024-03-11T21:49:05.635432+0000","last_peered":"2024-03-11T21:49:05.635432+0000","last_clean":"2024-03-11T21:49:05.635432+0000","last_became_active":"2024-03-11T21:44:35.630913+0000","last_became_peered":"2024-03-11T21:44:35.630913+0000","last_unstale":"2024-03-11T21:49:05.635432+0000","last_undegraded":"2024-03-11T21:49:05.635432+0000","last_fullsized":"2024-03-11T21:49:05.635432+0000","mapping_epoch":437,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":438,"parent":"0.0","parent_split_bits":0,"last_scrub":"456'30","last_scrub_stamp":"2024-03-11T21:48:59.479093+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:39.525554+0000","last_clean_scrub_stamp":"2024-03-11T21:48:59.479093+0000","objects_scrubbed":5,"log_size":30,"log_dups_size":0,"ondisk_log_size":30,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:50:01.540725+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":44,"num_read_kb":39,"num_write":30,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,7],"acting":[6,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.f","version":"0'0","reported_seq":1012,"reported_epoch":523,"state":"active+clean","last_fresh":"2024-03-11T21:49:06.639475+0000","last_change":"2024-03-11T21:49:04.810575+0000","last_active":"2024-03-11T21:49:06.639475+0000","last_peered":"2024-03-11T21:49:06.639475+0000","last_clean":"2024-03-11T21:49:06.639475+0000","last_became_active":"2024-03-11T21:44:35.645282+0000","last_became_peered":"2024-03-11T21:44:35.645282+0000","last_unstale":"2024-03-11T21:49:06.639475+0000","last_undegraded":"2024-03-11T21:49:06.639475+0000","last_fullsized":"2024-03-11T21:49:06.639475+0000","mapping_epoch":437,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":438,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:49:04.810478+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:27:03.010969+0000","last_clean_scrub_stamp":"2024-03-11T21:49:04.810478+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:50:15.703232+0000","scrub_duration":197,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6,1],"acting":[2,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":908,"reported_epoch":524,"state":"active+clean","last_fresh":"2024-03-11T21:49:08.604843+0000","last_change":"2024-03-11T21:49:01.121860+0000","last_active":"2024-03-11T21:49:08.604843+0000","last_peered":"2024-03-11T21:49:08.604843+0000","last_clean":"2024-03-11T21:49:08.604843+0000","last_became_active":"2024-03-11T21:47:20.274869+0000","last_became_peered":"2024-03-11T21:47:20.274869+0000","last_unstale":"2024-03-11T21:49:08.604843+0000","last_undegraded":"2024-03-11T21:49:08.604843+0000","last_fullsized":"2024-03-11T21:49:08.604843+0000","mapping_epoch":478,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":479,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:49:01.121783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:30:22.750053+0000","last_clean_scrub_stamp":"2024-03-11T21:49:01.121783+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:50:21.278653+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,6],"acting":[0,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.b","version":"0'0","reported_seq":650,"reported_epoch":523,"state":"active+clean","last_fresh":"2024-03-11T21:49:07.170220+0000","last_change":"2024-03-11T21:48:28.628161+0000","last_active":"2024-03-11T21:49:07.170220+0000","last_peered":"2024-03-11T21:49:07.170220+0000","last_clean":"2024-03-11T21:49:07.170220+0000","last_became_active":"2024-03-11T21:48:28.627574+0000","last_became_peered":"2024-03-11T21:48:28.627574+0000","last_unstale":"2024-03-11T21:49:07.170220+0000","last_undegraded":"2024-03-11T21:49:07.170220+0000","last_fullsized":"2024-03-11T21:49:07.170220+0000","mapping_epoch":502,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":503,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:45:30.101973+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:45:30.101973+0000","last_clean_scrub_stamp":"2024-03-11T21:45:30.101973+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,6],"acting":[7,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.d","version":"456'53","reported_seq":731,"reported_epoch":523,"state":"active+clean","last_fresh":"2024-03-11T21:49:06.639224+0000","last_change":"2024-03-11T21:49:05.623558+0000","last_active":"2024-03-11T21:49:06.639224+0000","last_peered":"2024-03-11T21:49:06.639224+0000","last_clean":"2024-03-11T21:49:06.639224+0000","last_became_active":"2024-03-11T21:44:35.634778+0000","last_became_peered":"2024-03-11T21:44:35.634778+0000","last_unstale":"2024-03-11T21:49:06.639224+0000","last_undegraded":"2024-03-11T21:49:06.639224+0000","last_fullsized":"2024-03-11T21:49:06.639224+0000","mapping_epoch":437,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":438,"parent":"0.0","parent_split_bits":0,"last_scrub":"456'53","last_scrub_stamp":"2024-03-11T21:49:05.623446+0000","last_deep_scrub":"165'25","last_deep_scrub_stamp":"2024-03-11T21:32:21.678328+0000","last_clean_scrub_stamp":"2024-03-11T21:49:05.623446+0000","objects_scrubbed":8,"log_size":53,"log_dups_size":0,"ondisk_log_size":53,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:50:15.360664+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":70,"num_read_kb":60,"num_write":46,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,6],"acting":[2,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":638,"reported_epoch":523,"state":"active+clean","last_fresh":"2024-03-11T21:49:07.169716+0000","last_change":"2024-03-11T21:46:05.407295+0000","last_active":"2024-03-11T21:49:07.169716+0000","last_peered":"2024-03-11T21:49:07.169716+0000","last_clean":"2024-03-11T21:49:07.169716+0000","last_became_active":"2024-03-11T21:44:21.461352+0000","last_became_peered":"2024-03-11T21:44:21.461352+0000","last_unstale":"2024-03-11T21:49:07.169716+0000","last_undegraded":"2024-03-11T21:49:07.169716+0000","last_fullsized":"2024-03-11T21:49:07.169716+0000","mapping_epoch":432,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":433,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:46:05.407166+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:28:37.561826+0000","last_clean_scrub_stamp":"2024-03-11T21:46:05.407166+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,6],"acting":[7,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":1007,"reported_epoch":523,"state":"active+clean","last_fresh":"2024-03-11T21:49:06.638049+0000","last_change":"2024-03-11T21:49:02.635836+0000","last_active":"2024-03-11T21:49:06.638049+0000","last_peered":"2024-03-11T21:49:06.638049+0000","last_clean":"2024-03-11T21:49:06.638049+0000","last_became_active":"2024-03-11T21:48:28.627492+0000","last_became_peered":"2024-03-11T21:48:28.627492+0000","last_unstale":"2024-03-11T21:49:06.638049+0000","last_undegraded":"2024-03-11T21:49:06.638049+0000","last_fullsized":"2024-03-11T21:49:06.638049+0000","mapping_epoch":502,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":503,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:49:02.635765+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:41.544550+0000","last_clean_scrub_stamp":"2024-03-11T21:49:02.635765+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:50:24.699944+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,5],"acting":[2,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.c","version":"0'0","reported_seq":1007,"reported_epoch":523,"state":"active+clean","last_fresh":"2024-03-11T21:49:06.639639+0000","last_change":"2024-03-11T21:48:28.637324+0000","last_active":"2024-03-11T21:49:06.639639+0000","last_peered":"2024-03-11T21:49:06.639639+0000","last_clean":"2024-03-11T21:49:06.639639+0000","last_became_active":"2024-03-11T21:48:28.636860+0000","last_became_peered":"2024-03-11T21:48:28.636860+0000","last_unstale":"2024-03-11T21:49:06.639639+0000","last_undegraded":"2024-03-11T21:49:06.639639+0000","last_fullsized":"2024-03-11T21:49:06.639639+0000","mapping_epoch":502,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":503,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:48:21.649477+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:29:30.279910+0000","last_clean_scrub_stamp":"2024-03-11T21:48:21.649477+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:49:45.765783+0000","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,0],"acting":[2,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.a","version":"456'60","reported_seq":1147,"reported_epoch":522,"state":"active+clean","last_fresh":"2024-03-11T21:49:05.635190+0000","last_change":"2024-03-11T21:45:32.673815+0000","last_active":"2024-03-11T21:49:05.635190+0000","last_peered":"2024-03-11T21:49:05.635190+0000","last_clean":"2024-03-11T21:49:05.635190+0000","last_became_active":"2024-03-11T21:34:41.148863+0000","last_became_peered":"2024-03-11T21:34:41.148863+0000","last_unstale":"2024-03-11T21:49:05.635190+0000","last_undegraded":"2024-03-11T21:49:05.635190+0000","last_fullsized":"2024-03-11T21:49:05.635190+0000","mapping_epoch":226,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":227,"parent":"0.0","parent_split_bits":0,"last_scrub":"260'44","last_scrub_stamp":"2024-03-11T21:45:32.673622+0000","last_deep_scrub":"260'44","last_deep_scrub_stamp":"2024-03-11T21:45:32.673622+0000","last_clean_scrub_stamp":"2024-03-11T21:45:32.673622+0000","objects_scrubbed":9,"log_size":60,"log_dups_size":0,"ondisk_log_size":60,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":80,"num_read_kb":69,"num_write":52,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,7],"acting":[6,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":999,"reported_epoch":524,"state":"active+clean","last_fresh":"2024-03-11T21:49:08.604872+0000","last_change":"2024-03-11T21:48:15.917128+0000","last_active":"2024-03-11T21:49:08.604872+0000","last_peered":"2024-03-11T21:49:08.604872+0000","last_clean":"2024-03-11T21:49:08.604872+0000","last_became_active":"2024-03-11T21:34:41.154507+0000","last_became_peered":"2024-03-11T21:34:41.154507+0000","last_unstale":"2024-03-11T21:49:08.604872+0000","last_undegraded":"2024-03-11T21:49:08.604872+0000","last_fullsized":"2024-03-11T21:49:08.604872+0000","mapping_epoch":226,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":227,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:48:15.916918+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:27:11.361913+0000","last_clean_scrub_stamp":"2024-03-11T21:48:15.916918+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:49:21.099751+0000","scrub_duration":104,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":1008,"reported_epoch":523,"state":"active+clean","last_fresh":"2024-03-11T21:49:06.637467+0000","last_change":"2024-03-11T21:48:31.600504+0000","last_active":"2024-03-11T21:49:06.637467+0000","last_peered":"2024-03-11T21:49:06.637467+0000","last_clean":"2024-03-11T21:49:06.637467+0000","last_became_active":"2024-03-11T21:44:35.631810+0000","last_became_peered":"2024-03-11T21:44:35.631810+0000","last_unstale":"2024-03-11T21:49:06.637467+0000","last_undegraded":"2024-03-11T21:49:06.637467+0000","last_fullsized":"2024-03-11T21:49:06.637467+0000","mapping_epoch":437,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":438,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:48:31.600423+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:43:32.115556+0000","last_clean_scrub_stamp":"2024-03-11T21:48:31.600423+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:49:40.843188+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.d","version":"0'0","reported_seq":1020,"reported_epoch":524,"state":"active+clean","last_fresh":"2024-03-11T21:49:08.659889+0000","last_change":"2024-03-11T21:48:58.963224+0000","last_active":"2024-03-11T21:49:08.659889+0000","last_peered":"2024-03-11T21:49:08.659889+0000","last_clean":"2024-03-11T21:49:08.659889+0000","last_became_active":"2024-03-11T21:44:35.621273+0000","last_became_peered":"2024-03-11T21:44:35.621273+0000","last_unstale":"2024-03-11T21:49:08.659889+0000","last_undegraded":"2024-03-11T21:49:08.659889+0000","last_fullsized":"2024-03-11T21:49:08.659889+0000","mapping_epoch":437,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":438,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:48:58.963145+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:37.512018+0000","last_clean_scrub_stamp":"2024-03-11T21:48:58.963145+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:50:03.979428+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,6],"acting":[1,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.b","version":"456'21","reported_seq":1085,"reported_epoch":524,"state":"active+clean","last_fresh":"2024-03-11T21:49:08.605263+0000","last_change":"2024-03-11T21:48:28.672342+0000","last_active":"2024-03-11T21:49:08.605263+0000","last_peered":"2024-03-11T21:49:08.605263+0000","last_clean":"2024-03-11T21:49:08.605263+0000","last_became_active":"2024-03-11T21:48:28.641509+0000","last_became_peered":"2024-03-11T21:48:28.641509+0000","last_unstale":"2024-03-11T21:49:08.605263+0000","last_undegraded":"2024-03-11T21:49:08.605263+0000","last_fullsized":"2024-03-11T21:49:08.605263+0000","mapping_epoch":502,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":503,"parent":"0.0","parent_split_bits":0,"last_scrub":"260'15","last_scrub_stamp":"2024-03-11T21:45:30.172740+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:39.525554+0000","last_clean_scrub_stamp":"2024-03-11T21:45:30.172740+0000","objects_scrubbed":4,"log_size":21,"log_dups_size":0,"ondisk_log_size":21,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":35,"num_read_kb":29,"num_write":22,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":440,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,5],"acting":[0,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":59,"reported_epoch":523,"state":"active+clean","last_fresh":"2024-03-11T21:49:06.863777+0000","last_change":"2024-03-11T21:48:39.986016+0000","last_active":"2024-03-11T21:49:06.863777+0000","last_peered":"2024-03-11T21:49:06.863777+0000","last_clean":"2024-03-11T21:49:06.863777+0000","last_became_active":"2024-03-11T21:48:28.643247+0000","last_became_peered":"2024-03-11T21:48:28.643247+0000","last_unstale":"2024-03-11T21:49:06.863777+0000","last_undegraded":"2024-03-11T21:49:06.863777+0000","last_fullsized":"2024-03-11T21:49:06.863777+0000","mapping_epoch":502,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":503,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:48:39.985901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:43.559452+0000","last_clean_scrub_stamp":"2024-03-11T21:48:39.985901+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:49:51.884417+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,0],"acting":[5,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.e","version":"0'0","reported_seq":1030,"reported_epoch":523,"state":"active+clean","last_fresh":"2024-03-11T21:49:06.639457+0000","last_change":"2024-03-11T21:48:12.611346+0000","last_active":"2024-03-11T21:49:06.639457+0000","last_peered":"2024-03-11T21:49:06.639457+0000","last_clean":"2024-03-11T21:49:06.639457+0000","last_became_active":"2024-03-11T21:44:21.457534+0000","last_became_peered":"2024-03-11T21:44:21.457534+0000","last_unstale":"2024-03-11T21:49:06.639457+0000","last_undegraded":"2024-03-11T21:49:06.639457+0000","last_fullsized":"2024-03-11T21:49:06.639457+0000","mapping_epoch":432,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":433,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:48:12.611106+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:37.512018+0000","last_clean_scrub_stamp":"2024-03-11T21:48:12.611106+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:49:13.446921+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,6],"acting":[2,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.8","version":"456'46","reported_seq":67,"reported_epoch":523,"state":"active+clean","last_fresh":"2024-03-11T21:49:06.863455+0000","last_change":"2024-03-11T21:48:28.664020+0000","last_active":"2024-03-11T21:49:06.863455+0000","last_peered":"2024-03-11T21:49:06.863455+0000","last_clean":"2024-03-11T21:49:06.863455+0000","last_became_active":"2024-03-11T21:48:28.642553+0000","last_became_peered":"2024-03-11T21:48:28.642553+0000","last_unstale":"2024-03-11T21:49:06.863455+0000","last_undegraded":"2024-03-11T21:49:06.863455+0000","last_fullsized":"2024-03-11T21:49:06.863455+0000","mapping_epoch":502,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":503,"parent":"0.0","parent_split_bits":0,"last_scrub":"456'46","last_scrub_stamp":"2024-03-11T21:46:18.893747+0000","last_deep_scrub":"109'22","last_deep_scrub_stamp":"2024-03-11T21:32:07.573197+0000","last_clean_scrub_stamp":"2024-03-11T21:46:18.893747+0000","objects_scrubbed":7,"log_size":46,"log_dups_size":0,"ondisk_log_size":46,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":61,"num_read_kb":52,"num_write":40,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":28,"num_bytes_recovered":440,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.9","version":"438'12","reported_seq":828,"reported_epoch":523,"state":"active+clean","last_fresh":"2024-03-11T21:49:07.169968+0000","last_change":"2024-03-11T21:49:03.823881+0000","last_active":"2024-03-11T21:49:07.169968+0000","last_peered":"2024-03-11T21:49:07.169968+0000","last_clean":"2024-03-11T21:49:07.169968+0000","last_became_active":"2024-03-11T21:44:35.638711+0000","last_became_peered":"2024-03-11T21:44:35.638711+0000","last_unstale":"2024-03-11T21:49:07.169968+0000","last_undegraded":"2024-03-11T21:49:07.169968+0000","last_fullsized":"2024-03-11T21:49:07.169968+0000","mapping_epoch":437,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":438,"parent":"0.0","parent_split_bits":0,"last_scrub":"438'12","last_scrub_stamp":"2024-03-11T21:49:03.823758+0000","last_deep_scrub":"94'9","last_deep_scrub_stamp":"2024-03-11T21:26:53.422552+0000","last_clean_scrub_stamp":"2024-03-11T21:49:03.823758+0000","objects_scrubbed":1,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:50:16.324839+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,1],"acting":[7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":1531,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":7,"num_read_kb":7,"num_write":10,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":2587,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":98304,"data_stored":6124,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":6,"ondisk_log_size":6,"up":96,"acting":96,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":198,"num_read_kb":294,"num_write":179,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":2951840,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":2375680,"data_stored":2361472,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":128,"ondisk_log_size":128,"up":3,"acting":3,"num_store_stats":4},{"poolid":3,"num_pg":3,"stat_sum":{"num_bytes":408,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":495,"num_read_kb":490,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":38,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":1262,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":8,"ondisk_log_size":8,"up":9,"acting":9,"num_store_stats":8},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":3702,"num_objects":178,"num_object_clones":0,"num_object_copies":534,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":178,"num_whiteouts":0,"num_read":1539,"num_read_kb":1296,"num_write":989,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":451,"num_bytes_recovered":8762,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":544768,"data_stored":14478,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1070,"ondisk_log_size":1070,"up":96,"acting":96,"num_store_stats":8},{"poolid":5,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":17,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":94,"ondisk_log_size":94,"up":96,"acting":96,"num_store_stats":8},{"poolid":6,"num_pg":32,"stat_sum":{"num_bytes":383,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":1136,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":1175,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":4,"ondisk_log_size":4,"up":96,"acting":96,"num_store_stats":8}],"osd_stats":[{"osd":7,"up_from":60,"seq":257698038045,"num_pgs":67,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":862976,"kb_used_data":1588,"kb_used_omap":1,"kb_used_meta":861374,"kb_avail":92882176,"statfs":{"total":95995035648,"available":95111348224,"internally_reserved":0,"allocated":1626112,"data_stored":1127080,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":882047434},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":50,"seq":214748365091,"num_pgs":64,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":871332,"kb_used_data":1624,"kb_used_omap":1,"kb_used_meta":869694,"kb_avail":92873820,"statfs":{"total":95995035648,"available":95102791680,"internally_reserved":0,"allocated":1662976,"data_stored":1129140,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":890567110},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834575165,"num_pgs":67,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1020720,"kb_used_data":1060,"kb_used_omap":1,"kb_used_meta":1019646,"kb_avail":92724432,"statfs":{"total":95995035648,"available":94949818368,"internally_reserved":0,"allocated":1085440,"data_stored":550636,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":1044117957},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738690,"num_pgs":66,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1011564,"kb_used_data":1632,"kb_used_omap":1,"kb_used_meta":1009918,"kb_avail":92733588,"statfs":{"total":95995035648,"available":94959194112,"internally_reserved":0,"allocated":1671168,"data_stored":1141428,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":1034156485},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411639,"num_pgs":62,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":876312,"kb_used_data":1032,"kb_used_omap":1,"kb_used_meta":875262,"kb_avail":92868840,"statfs":{"total":95995035648,"available":95097692160,"internally_reserved":0,"allocated":1056768,"data_stored":545131,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":896268741},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":0,"seq":1189705941144,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":0,"seq":137438953772,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":41,"seq":176093659431,"num_pgs":70,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":876324,"kb_used_data":1044,"kb_used_omap":1,"kb_used_meta":875262,"kb_avail":92868828,"statfs":{"total":95995035648,"available":95097679872,"internally_reserved":0,"allocated":1069056,"data_stored":544211,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":896268741},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":1095,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":46,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1393,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":957,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":574,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":528,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1439,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":92,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":69632,"data_stored":1914,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":81920,"data_stored":2118,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":61440,"data_stored":1568,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":1898,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":1474,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":69632,"data_stored":1870,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":81920,"data_stored":2162,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":1474,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":370,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":383,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":383,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-11T21:49:13.232 INFO:tasks.cephadm.ceph_manager.ceph:recovered! 2024-03-11T21:49:13.232 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-03-11T21:49:13.232 INFO:tasks.thrashosds.thrasher:Adding osd 3 2024-03-11T21:49:13.233 DEBUG:teuthology.orchestra.run.smithi092:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid 00db0016-dfed-11ee-95c8-87774f69a715 -- ceph osd in 3 2024-03-11T21:49:13.233 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:49:13 smithi097 ceph-mon[33317]: pgmap v1126: 132 pgs: 132 active+clean; 582 KiB data, 5.3 GiB used, 531 GiB / 536 GiB avail; 868 B/s rd, 0 op/s 2024-03-11T21:49:13.233 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:49:13 smithi097 ceph-mon[33317]: 4.0 scrub starts 2024-03-11T21:49:13.233 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:49:13 smithi097 ceph-mon[33317]: 4.0 scrub ok 2024-03-11T21:49:13.233 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:49:13 smithi097 ceph-mon[33317]: 6.11 scrub starts 2024-03-11T21:49:13.234 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:49:13 smithi097 ceph-mon[33317]: 6.11 scrub ok 2024-03-11T21:49:13.477 INFO:teuthology.orchestra.run.smithi092.stderr:Inferring config /var/lib/ceph/00db0016-dfed-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T21:49:13.501 INFO:journalctl@ceph.iscsi.iscsi.a.smithi097.stdout:Mar 11 21:49:13 smithi097 ceph-00db0016-dfed-11ee-95c8-87774f69a715-iscsi-iscsi-a[60943]: debug there is no tcmu-runner data available 2024-03-11T21:49:14.335 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:49:14 smithi097 ceph-mon[33317]: from='client.16254 v1:172.21.15.92:0/1221680998' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-11T21:49:14.336 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:49:14 smithi097 ceph-mon[33317]: 6.15 scrub starts 2024-03-11T21:49:14.336 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:49:14 smithi097 ceph-mon[33317]: 6.15 scrub ok 2024-03-11T21:49:14.336 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:49:14 smithi097 ceph-mon[33317]: from='client.24469 v1:172.21.15.97:0/404208310' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:49:14.336 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:49:14 smithi097 ceph-mon[33317]: osdmap e526: 8 total, 8 up, 6 in 2024-03-11T21:49:14.336 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:49:14 smithi097 ceph-mon[33317]: 4.a scrub starts 2024-03-11T21:49:14.336 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:49:14 smithi097 ceph-mon[33317]: 2.e scrub starts 2024-03-11T21:49:14.336 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:49:14 smithi097 ceph-mon[33317]: 2.e scrub ok 2024-03-11T21:49:14.355 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:49:14 smithi092 ceph-mon[37563]: from='client.16254 v1:172.21.15.92:0/1221680998' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-11T21:49:14.355 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:49:14 smithi092 ceph-mon[37563]: 6.15 scrub starts 2024-03-11T21:49:14.355 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:49:14 smithi092 ceph-mon[37563]: 6.15 scrub ok 2024-03-11T21:49:14.355 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:49:14 smithi092 ceph-mon[37563]: from='client.24469 v1:172.21.15.97:0/404208310' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:49:14.355 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:49:14 smithi092 ceph-mon[37563]: osdmap e526: 8 total, 8 up, 6 in 2024-03-11T21:49:14.355 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:49:14 smithi092 ceph-mon[37563]: 4.a scrub starts 2024-03-11T21:49:14.356 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:49:14 smithi092 ceph-mon[37563]: 2.e scrub starts 2024-03-11T21:49:14.356 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:49:14 smithi092 ceph-mon[37563]: 2.e scrub ok 2024-03-11T21:49:14.356 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:49:14 smithi092 ceph-mon[29572]: from='client.16254 v1:172.21.15.92:0/1221680998' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-11T21:49:14.356 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:49:14 smithi092 ceph-mon[29572]: 6.15 scrub starts 2024-03-11T21:49:14.356 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:49:14 smithi092 ceph-mon[29572]: 6.15 scrub ok 2024-03-11T21:49:14.356 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:49:14 smithi092 ceph-mon[29572]: from='client.24469 v1:172.21.15.97:0/404208310' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:49:14.356 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:49:14 smithi092 ceph-mon[29572]: osdmap e526: 8 total, 8 up, 6 in 2024-03-11T21:49:14.356 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:49:14 smithi092 ceph-mon[29572]: 4.a scrub starts 2024-03-11T21:49:14.356 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:49:14 smithi092 ceph-mon[29572]: 2.e scrub starts 2024-03-11T21:49:14.356 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:49:14 smithi092 ceph-mon[29572]: 2.e scrub ok 2024-03-11T21:49:14.781 INFO:teuthology.orchestra.run.smithi097.stdout:{ 2024-03-11T21:49:14.782 INFO:teuthology.orchestra.run.smithi097.stdout: "ops": [], 2024-03-11T21:49:14.782 INFO:teuthology.orchestra.run.smithi097.stdout: "complaint_time": 30, 2024-03-11T21:49:14.782 INFO:teuthology.orchestra.run.smithi097.stdout: "num_blocked_ops": 0 2024-03-11T21:49:14.782 INFO:teuthology.orchestra.run.smithi097.stdout:} 2024-03-11T21:49:15.369 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:49:15 smithi092 ceph-mon[29572]: pgmap v1127: 132 pgs: 132 active+clean; 582 KiB data, 5.3 GiB used, 531 GiB / 536 GiB avail; 742 B/s rd, 0 op/s 2024-03-11T21:49:15.370 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:49:15 smithi092 ceph-mon[29572]: 4.a scrub ok 2024-03-11T21:49:15.370 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:49:15 smithi092 ceph-mon[29572]: 5.5 scrub starts 2024-03-11T21:49:15.370 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:49:15 smithi092 ceph-mon[29572]: 5.5 scrub ok 2024-03-11T21:49:15.370 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:49:15 smithi092 ceph-mon[29572]: purged_snaps scrub starts 2024-03-11T21:49:15.370 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:49:15 smithi092 ceph-mon[29572]: purged_snaps scrub ok 2024-03-11T21:49:15.370 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:49:15 smithi092 ceph-mon[37563]: pgmap v1127: 132 pgs: 132 active+clean; 582 KiB data, 5.3 GiB used, 531 GiB / 536 GiB avail; 742 B/s rd, 0 op/s 2024-03-11T21:49:15.370 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:49:15 smithi092 ceph-mon[37563]: 4.a scrub ok 2024-03-11T21:49:15.370 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:49:15 smithi092 ceph-mon[37563]: 5.5 scrub starts 2024-03-11T21:49:15.370 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:49:15 smithi092 ceph-mon[37563]: 5.5 scrub ok 2024-03-11T21:49:15.371 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:49:15 smithi092 ceph-mon[37563]: purged_snaps scrub starts 2024-03-11T21:49:15.371 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:49:15 smithi092 ceph-mon[37563]: purged_snaps scrub ok 2024-03-11T21:49:15.421 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:49:15 smithi097 ceph-mon[33317]: pgmap v1127: 132 pgs: 132 active+clean; 582 KiB data, 5.3 GiB used, 531 GiB / 536 GiB avail; 742 B/s rd, 0 op/s 2024-03-11T21:49:15.421 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:49:15 smithi097 ceph-mon[33317]: 4.a scrub ok 2024-03-11T21:49:15.421 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:49:15 smithi097 ceph-mon[33317]: 5.5 scrub starts 2024-03-11T21:49:15.421 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:49:15 smithi097 ceph-mon[33317]: 5.5 scrub ok 2024-03-11T21:49:15.421 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:49:15 smithi097 ceph-mon[33317]: purged_snaps scrub starts 2024-03-11T21:49:15.421 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:49:15 smithi097 ceph-mon[33317]: purged_snaps scrub ok 2024-03-11T21:49:15.447 DEBUG:teuthology.orchestra.run.smithi097:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid 00db0016-dfed-11ee-95c8-87774f69a715 -- ceph daemon osd.7 dump_historic_ops 2024-03-11T21:49:15.702 INFO:teuthology.orchestra.run.smithi097.stderr:Inferring config /var/lib/ceph/00db0016-dfed-11ee-95c8-87774f69a715/mon.b/config 2024-03-11T21:49:15.869 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:49:15 smithi092 ceph-00db0016-dfed-11ee-95c8-87774f69a715-mon-a[29568]: 2024-03-11T21:49:15.423+0000 7f87a4389700 -1 mon.a@0(leader).osd e527 definitely_dead 0 2024-03-11T21:49:16.114 INFO:teuthology.orchestra.run.smithi092.stderr:marked in osd.3. 2024-03-11T21:49:16.335 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:49:16 smithi097 ceph-mon[33317]: 6.8 deep-scrub starts 2024-03-11T21:49:16.335 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:49:16 smithi097 ceph-mon[33317]: 6.8 deep-scrub ok 2024-03-11T21:49:16.335 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:49:16 smithi097 ceph-mon[33317]: osdmap e527: 8 total, 8 up, 6 in 2024-03-11T21:49:16.335 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:49:16 smithi097 ceph-mon[33317]: 4.8 scrub starts 2024-03-11T21:49:16.335 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:49:16 smithi097 ceph-mon[33317]: 4.8 scrub ok 2024-03-11T21:49:16.336 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:49:16 smithi097 ceph-mon[33317]: 4.b scrub starts 2024-03-11T21:49:16.336 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:49:16 smithi097 ceph-mon[33317]: 4.b scrub ok 2024-03-11T21:49:16.336 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:49:16 smithi097 ceph-mon[33317]: from='client.? v1:172.21.15.92:0/1429351511' entity='client.admin' cmd=[{"prefix": "osd in", "ids": ["3"]}]: dispatch 2024-03-11T21:49:16.369 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:49:16 smithi092 ceph-mon[29572]: 6.8 deep-scrub starts 2024-03-11T21:49:16.370 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:49:16 smithi092 ceph-mon[29572]: 6.8 deep-scrub ok 2024-03-11T21:49:16.370 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:49:16 smithi092 ceph-mon[29572]: osdmap e527: 8 total, 8 up, 6 in 2024-03-11T21:49:16.370 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:49:16 smithi092 ceph-mon[29572]: 4.8 scrub starts 2024-03-11T21:49:16.370 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:49:16 smithi092 ceph-mon[29572]: 4.8 scrub ok 2024-03-11T21:49:16.370 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:49:16 smithi092 ceph-mon[29572]: 4.b scrub starts 2024-03-11T21:49:16.370 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:49:16 smithi092 ceph-mon[29572]: 4.b scrub ok 2024-03-11T21:49:16.370 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:49:16 smithi092 ceph-mon[29572]: from='client.? v1:172.21.15.92:0/1429351511' entity='client.admin' cmd=[{"prefix": "osd in", "ids": ["3"]}]: dispatch 2024-03-11T21:49:16.370 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:49:16 smithi092 ceph-mon[37563]: 6.8 deep-scrub starts 2024-03-11T21:49:16.370 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:49:16 smithi092 ceph-mon[37563]: 6.8 deep-scrub ok 2024-03-11T21:49:16.370 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:49:16 smithi092 ceph-mon[37563]: osdmap e527: 8 total, 8 up, 6 in 2024-03-11T21:49:16.371 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:49:16 smithi092 ceph-mon[37563]: 4.8 scrub starts 2024-03-11T21:49:16.371 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:49:16 smithi092 ceph-mon[37563]: 4.8 scrub ok 2024-03-11T21:49:16.371 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:49:16 smithi092 ceph-mon[37563]: 4.b scrub starts 2024-03-11T21:49:16.371 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:49:16 smithi092 ceph-mon[37563]: 4.b scrub ok 2024-03-11T21:49:16.371 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:49:16 smithi092 ceph-mon[37563]: from='client.? v1:172.21.15.92:0/1429351511' entity='client.admin' cmd=[{"prefix": "osd in", "ids": ["3"]}]: dispatch 2024-03-11T21:49:16.971 INFO:tasks.thrashosds.thrasher:Added osd 3 2024-03-11T21:49:17.369 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:49:17 smithi092 ceph-mon[29572]: pgmap v1130: 132 pgs: 132 active+clean; 582 KiB data, 5.3 GiB used, 531 GiB / 536 GiB avail; 682 B/s rd, 0 op/s 2024-03-11T21:49:17.369 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:49:17 smithi092 ceph-mon[29572]: from='client.? v1:172.21.15.92:0/1429351511' entity='client.admin' cmd='[{"prefix": "osd in", "ids": ["3"]}]': finished 2024-03-11T21:49:17.369 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:49:17 smithi092 ceph-mon[29572]: osdmap e528: 8 total, 8 up, 7 in 2024-03-11T21:49:17.370 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:49:17 smithi092 ceph-mon[29572]: 6.10 deep-scrub starts 2024-03-11T21:49:17.370 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:49:17 smithi092 ceph-mon[29572]: 6.10 deep-scrub ok 2024-03-11T21:49:17.370 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:49:17 smithi092 ceph-mon[37563]: pgmap v1130: 132 pgs: 132 active+clean; 582 KiB data, 5.3 GiB used, 531 GiB / 536 GiB avail; 682 B/s rd, 0 op/s 2024-03-11T21:49:17.370 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:49:17 smithi092 ceph-mon[37563]: from='client.? v1:172.21.15.92:0/1429351511' entity='client.admin' cmd='[{"prefix": "osd in", "ids": ["3"]}]': finished 2024-03-11T21:49:17.370 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:49:17 smithi092 ceph-mon[37563]: osdmap e528: 8 total, 8 up, 7 in 2024-03-11T21:49:17.370 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:49:17 smithi092 ceph-mon[37563]: 6.10 deep-scrub starts 2024-03-11T21:49:17.370 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:49:17 smithi092 ceph-mon[37563]: 6.10 deep-scrub ok 2024-03-11T21:49:17.373 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:49:17 smithi097 ceph-mon[33317]: pgmap v1130: 132 pgs: 132 active+clean; 582 KiB data, 5.3 GiB used, 531 GiB / 536 GiB avail; 682 B/s rd, 0 op/s 2024-03-11T21:49:17.374 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:49:17 smithi097 ceph-mon[33317]: from='client.? v1:172.21.15.92:0/1429351511' entity='client.admin' cmd='[{"prefix": "osd in", "ids": ["3"]}]': finished 2024-03-11T21:49:17.374 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:49:17 smithi097 ceph-mon[33317]: osdmap e528: 8 total, 8 up, 7 in 2024-03-11T21:49:17.374 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:49:17 smithi097 ceph-mon[33317]: 6.10 deep-scrub starts 2024-03-11T21:49:17.374 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:49:17 smithi097 ceph-mon[33317]: 6.10 deep-scrub ok 2024-03-11T21:49:17.399 INFO:teuthology.orchestra.run.smithi097.stdout:{ 2024-03-11T21:49:17.400 INFO:teuthology.orchestra.run.smithi097.stdout: "size": 20, 2024-03-11T21:49:17.400 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 600, 2024-03-11T21:49:17.400 INFO:teuthology.orchestra.run.smithi097.stdout: "ops": [ 2024-03-11T21:49:17.400 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:49:17.400 INFO:teuthology.orchestra.run.smithi097.stdout: "description": "osd_op(client.35813.0:30497 11.f 11:fbc30b4e:::benchmark_data_smithi092_186520_object30496:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e452)", 2024-03-11T21:49:17.400 INFO:teuthology.orchestra.run.smithi097.stdout: "initiated_at": "2024-03-11T21:45:18.650860+0000", 2024-03-11T21:49:17.400 INFO:teuthology.orchestra.run.smithi097.stdout: "age": 238.755873452, 2024-03-11T21:49:17.400 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.170501545, 2024-03-11T21:49:17.400 INFO:teuthology.orchestra.run.smithi097.stdout: "type_data": { 2024-03-11T21:49:17.400 INFO:teuthology.orchestra.run.smithi097.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-11T21:49:17.401 INFO:teuthology.orchestra.run.smithi097.stdout: "client_info": { 2024-03-11T21:49:17.401 INFO:teuthology.orchestra.run.smithi097.stdout: "client": "client.35813", 2024-03-11T21:49:17.401 INFO:teuthology.orchestra.run.smithi097.stdout: "client_addr": "v1:172.21.15.92:0/4293352030", 2024-03-11T21:49:17.401 INFO:teuthology.orchestra.run.smithi097.stdout: "tid": 30497 2024-03-11T21:49:17.401 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:49:17.401 INFO:teuthology.orchestra.run.smithi097.stdout: "events": [ 2024-03-11T21:49:17.401 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:49:17.401 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "initiated", 2024-03-11T21:49:17.401 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:45:18.650860+0000", 2024-03-11T21:49:17.401 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-03-11T21:49:17.401 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:49:17.401 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:49:17.401 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "header_read", 2024-03-11T21:49:17.401 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:45:18.650860+0000", 2024-03-11T21:49:17.402 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-03-11T21:49:17.402 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:49:17.402 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:49:17.402 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "throttled", 2024-03-11T21:49:17.402 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:45:18.650860+0000", 2024-03-11T21:49:17.402 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 7.0200000000000001e-07 2024-03-11T21:49:17.402 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:49:17.402 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:49:17.402 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "all_read", 2024-03-11T21:49:17.402 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:45:18.650912+0000", 2024-03-11T21:49:17.402 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 5.1434000000000003e-05 2024-03-11T21:49:17.402 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:49:17.402 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:49:17.403 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "dispatched", 2024-03-11T21:49:17.403 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:45:18.650922+0000", 2024-03-11T21:49:17.403 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.0275000000000001e-05 2024-03-11T21:49:17.403 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:49:17.403 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:49:17.403 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "queued_for_pg", 2024-03-11T21:49:17.403 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:45:18.650928+0000", 2024-03-11T21:49:17.403 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 6.127e-06 2024-03-11T21:49:17.403 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:49:17.403 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:49:17.403 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "reached_pg", 2024-03-11T21:49:17.403 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:45:18.652064+0000", 2024-03-11T21:49:17.403 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.001136113 2024-03-11T21:49:17.403 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:49:17.404 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:49:17.404 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "started", 2024-03-11T21:49:17.404 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:45:18.652182+0000", 2024-03-11T21:49:17.404 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.00011782600000000001 2024-03-11T21:49:17.404 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:49:17.404 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:49:17.404 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "waiting for subops from 1,2", 2024-03-11T21:49:17.404 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:45:18.652315+0000", 2024-03-11T21:49:17.404 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.000132769 2024-03-11T21:49:17.404 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:49:17.404 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:49:17.404 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "op_commit", 2024-03-11T21:49:17.404 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:45:18.654746+0000", 2024-03-11T21:49:17.404 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.0024310529999999999 2024-03-11T21:49:17.405 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:49:17.405 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:49:17.405 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "sub_op_commit_rec", 2024-03-11T21:49:17.405 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:45:18.656734+0000", 2024-03-11T21:49:17.405 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.001987553 2024-03-11T21:49:17.405 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:49:17.405 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:49:17.405 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "sub_op_commit_rec", 2024-03-11T21:49:17.405 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:45:18.821252+0000", 2024-03-11T21:49:17.405 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.16451839400000001 2024-03-11T21:49:17.405 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:49:17.405 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:49:17.405 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "commit_sent", 2024-03-11T21:49:17.405 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:45:18.821316+0000", 2024-03-11T21:49:17.406 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 6.3646999999999996e-05 2024-03-11T21:49:17.406 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:49:17.406 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:49:17.406 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "done", 2024-03-11T21:49:17.406 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:45:18.821361+0000", 2024-03-11T21:49:17.406 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 4.5652000000000003e-05 2024-03-11T21:49:17.406 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-03-11T21:49:17.406 INFO:teuthology.orchestra.run.smithi097.stdout: ] 2024-03-11T21:49:17.406 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-03-11T21:49:17.406 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:49:17.406 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:49:17.406 INFO:teuthology.orchestra.run.smithi097.stdout: "description": "osd_op(client.35813.0:30500 11.f 11:f1245661:::benchmark_data_smithi092_186520_object30499:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e452)", 2024-03-11T21:49:17.406 INFO:teuthology.orchestra.run.smithi097.stdout: "initiated_at": "2024-03-11T21:45:18.653260+0000", 2024-03-11T21:49:17.406 INFO:teuthology.orchestra.run.smithi097.stdout: "age": 238.753473163, 2024-03-11T21:49:17.407 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.168229831, 2024-03-11T21:49:17.407 INFO:teuthology.orchestra.run.smithi097.stdout: "type_data": { 2024-03-11T21:49:17.407 INFO:teuthology.orchestra.run.smithi097.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-11T21:49:17.407 INFO:teuthology.orchestra.run.smithi097.stdout: "client_info": { 2024-03-11T21:49:17.407 INFO:teuthology.orchestra.run.smithi097.stdout: "client": "client.35813", 2024-03-11T21:49:17.407 INFO:teuthology.orchestra.run.smithi097.stdout: "client_addr": "v1:172.21.15.92:0/4293352030", 2024-03-11T21:49:17.407 INFO:teuthology.orchestra.run.smithi097.stdout: "tid": 30500 2024-03-11T21:49:17.407 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:49:17.407 INFO:teuthology.orchestra.run.smithi097.stdout: "events": [ 2024-03-11T21:49:17.407 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:49:17.407 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "initiated", 2024-03-11T21:49:17.407 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:45:18.653260+0000", 2024-03-11T21:49:17.407 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-03-11T21:49:17.407 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:49:17.407 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:49:17.408 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "header_read", 2024-03-11T21:49:17.408 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:45:18.653260+0000", 2024-03-11T21:49:17.408 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-03-11T21:49:17.408 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:49:17.408 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:49:17.408 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "throttled", 2024-03-11T21:49:17.408 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:45:18.653261+0000", 2024-03-11T21:49:17.408 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.37e-06 2024-03-11T21:49:17.408 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:49:17.408 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:49:17.408 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "all_read", 2024-03-11T21:49:17.408 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:45:18.653320+0000", 2024-03-11T21:49:17.408 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 5.8211999999999998e-05 2024-03-11T21:49:17.408 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:49:17.409 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:49:17.409 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "dispatched", 2024-03-11T21:49:17.409 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:45:18.653327+0000", 2024-03-11T21:49:17.409 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 7.7379999999999994e-06 2024-03-11T21:49:17.409 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:49:17.409 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:49:17.409 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "queued_for_pg", 2024-03-11T21:49:17.409 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:45:18.653333+0000", 2024-03-11T21:49:17.409 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 5.8340000000000004e-06 2024-03-11T21:49:17.409 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:49:17.409 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:49:17.409 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "reached_pg", 2024-03-11T21:49:17.409 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:45:18.653468+0000", 2024-03-11T21:49:17.409 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.000135012 2024-03-11T21:49:17.409 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:49:17.409 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:49:17.410 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "started", 2024-03-11T21:49:17.410 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:45:18.653594+0000", 2024-03-11T21:49:17.410 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.00012546000000000001 2024-03-11T21:49:17.410 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:49:17.410 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:49:17.410 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "waiting for subops from 1,2", 2024-03-11T21:49:17.410 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:45:18.653719+0000", 2024-03-11T21:49:17.410 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.000125289 2024-03-11T21:49:17.410 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:49:17.410 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:49:17.410 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "op_commit", 2024-03-11T21:49:17.410 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:45:18.655122+0000", 2024-03-11T21:49:17.410 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.001403446 2024-03-11T21:49:17.411 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:49:17.411 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:49:17.411 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "sub_op_commit_rec", 2024-03-11T21:49:17.411 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:45:18.656771+0000", 2024-03-11T21:49:17.411 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.001648299 2024-03-11T21:49:17.411 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:49:17.411 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:49:17.411 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "sub_op_commit_rec", 2024-03-11T21:49:17.411 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:45:18.821397+0000", 2024-03-11T21:49:17.411 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.16462605599999999 2024-03-11T21:49:17.411 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:49:17.411 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:49:17.411 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "commit_sent", 2024-03-11T21:49:17.411 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:45:18.821440+0000", 2024-03-11T21:49:17.411 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 4.3739000000000003e-05 2024-03-11T21:49:17.412 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:49:17.412 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:49:17.412 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "done", 2024-03-11T21:49:17.412 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:45:18.821490+0000", 2024-03-11T21:49:17.412 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 4.9376000000000003e-05 2024-03-11T21:49:17.412 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-03-11T21:49:17.412 INFO:teuthology.orchestra.run.smithi097.stdout: ] 2024-03-11T21:49:17.412 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-03-11T21:49:17.412 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:49:17.412 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:49:17.412 INFO:teuthology.orchestra.run.smithi097.stdout: "description": "osd_op(client.35813.0:30508 11.f 11:f1ae1cb6:::benchmark_data_smithi092_186520_object30507:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e452)", 2024-03-11T21:49:17.412 INFO:teuthology.orchestra.run.smithi097.stdout: "initiated_at": "2024-03-11T21:45:18.655158+0000", 2024-03-11T21:49:17.412 INFO:teuthology.orchestra.run.smithi097.stdout: "age": 238.75157476199999, 2024-03-11T21:49:17.412 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.16955424499999999, 2024-03-11T21:49:17.412 INFO:teuthology.orchestra.run.smithi097.stdout: "type_data": { 2024-03-11T21:49:17.413 INFO:teuthology.orchestra.run.smithi097.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-11T21:49:17.413 INFO:teuthology.orchestra.run.smithi097.stdout: "client_info": { 2024-03-11T21:49:17.413 INFO:teuthology.orchestra.run.smithi097.stdout: "client": "client.35813", 2024-03-11T21:49:17.413 INFO:teuthology.orchestra.run.smithi097.stdout: "client_addr": "v1:172.21.15.92:0/4293352030", 2024-03-11T21:49:17.414 INFO:teuthology.orchestra.run.smithi097.stdout: "tid": 30508 2024-03-11T21:49:17.414 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:49:17.414 INFO:teuthology.orchestra.run.smithi097.stdout: "events": [ 2024-03-11T21:49:17.414 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:49:17.414 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "initiated", 2024-03-11T21:49:17.414 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:45:18.655158+0000", 2024-03-11T21:49:17.414 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-03-11T21:49:17.414 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:49:17.414 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:49:17.414 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "header_read", 2024-03-11T21:49:17.414 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:45:18.655158+0000", 2024-03-11T21:49:17.415 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-03-11T21:49:17.415 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:49:17.415 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:49:17.415 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "throttled", 2024-03-11T21:49:17.415 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:45:18.655160+0000", 2024-03-11T21:49:17.415 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.609e-06 2024-03-11T21:49:17.415 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:49:17.415 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:49:17.415 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "all_read", 2024-03-11T21:49:17.415 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:45:18.655229+0000", 2024-03-11T21:49:17.415 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 6.8614000000000004e-05 2024-03-11T21:49:17.415 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:49:17.415 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:49:17.415 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "dispatched", 2024-03-11T21:49:17.415 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:45:18.655242+0000", 2024-03-11T21:49:17.416 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.3545e-05 2024-03-11T21:49:17.416 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:49:17.416 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:49:17.416 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "queued_for_pg", 2024-03-11T21:49:17.416 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:45:18.655250+0000", 2024-03-11T21:49:17.416 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 7.6260000000000003e-06 2024-03-11T21:49:17.416 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:49:17.416 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:49:17.416 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "reached_pg", 2024-03-11T21:49:17.416 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:45:18.655317+0000", 2024-03-11T21:49:17.416 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 6.7652999999999998e-05 2024-03-11T21:49:17.416 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:49:17.416 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:49:17.416 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "started", 2024-03-11T21:49:17.417 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:45:18.655440+0000", 2024-03-11T21:49:17.417 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.00012273399999999999 2024-03-11T21:49:17.417 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:49:17.417 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:49:17.417 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "waiting for subops from 1,2", 2024-03-11T21:49:17.417 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:45:18.655531+0000", 2024-03-11T21:49:17.417 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 9.0933000000000005e-05 2024-03-11T21:49:17.417 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:49:17.417 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:49:17.417 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "op_commit", 2024-03-11T21:49:17.417 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:45:18.657297+0000", 2024-03-11T21:49:17.417 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.0017660149999999999 2024-03-11T21:49:17.417 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:49:17.417 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:49:17.417 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "sub_op_commit_rec", 2024-03-11T21:49:17.418 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:45:18.657582+0000", 2024-03-11T21:49:17.418 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.00028497599999999999 2024-03-11T21:49:17.418 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:49:17.418 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:49:17.418 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "sub_op_commit_rec", 2024-03-11T21:49:17.418 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:45:18.824624+0000", 2024-03-11T21:49:17.418 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.167042051 2024-03-11T21:49:17.418 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:49:17.418 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:49:17.418 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "commit_sent", 2024-03-11T21:49:17.418 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:45:18.824673+0000", 2024-03-11T21:49:17.418 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 4.8708999999999997e-05 2024-03-11T21:49:17.418 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:49:17.418 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:49:17.419 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "done", 2024-03-11T21:49:17.419 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:45:18.824713+0000", 2024-03-11T21:49:17.419 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 3.9780000000000002e-05 2024-03-11T21:49:17.419 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-03-11T21:49:17.419 INFO:teuthology.orchestra.run.smithi097.stdout: ] 2024-03-11T21:49:17.419 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-03-11T21:49:17.419 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:49:17.419 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:49:17.419 INFO:teuthology.orchestra.run.smithi097.stdout: "description": "osd_op(client.35813.0:109097 11.f 11:ffec0c33:::benchmark_data_smithi092_186520_object109096:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e463)", 2024-03-11T21:49:17.419 INFO:teuthology.orchestra.run.smithi097.stdout: "initiated_at": "2024-03-11T21:46:14.643074+0000", 2024-03-11T21:49:17.419 INFO:teuthology.orchestra.run.smithi097.stdout: "age": 182.763659461, 2024-03-11T21:49:17.419 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.12437609099999999, 2024-03-11T21:49:17.419 INFO:teuthology.orchestra.run.smithi097.stdout: "type_data": { 2024-03-11T21:49:17.420 INFO:teuthology.orchestra.run.smithi097.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-11T21:49:17.420 INFO:teuthology.orchestra.run.smithi097.stdout: "client_info": { 2024-03-11T21:49:17.420 INFO:teuthology.orchestra.run.smithi097.stdout: "client": "client.35813", 2024-03-11T21:49:17.420 INFO:teuthology.orchestra.run.smithi097.stdout: "client_addr": "v1:172.21.15.92:0/4293352030", 2024-03-11T21:49:17.420 INFO:teuthology.orchestra.run.smithi097.stdout: "tid": 109097 2024-03-11T21:49:17.420 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:49:17.420 INFO:teuthology.orchestra.run.smithi097.stdout: "events": [ 2024-03-11T21:49:17.420 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:49:17.420 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "initiated", 2024-03-11T21:49:17.420 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:46:14.643074+0000", 2024-03-11T21:49:17.420 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-03-11T21:49:17.420 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:49:17.420 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:49:17.420 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "header_read", 2024-03-11T21:49:17.420 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:46:14.643074+0000", 2024-03-11T21:49:17.421 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-03-11T21:49:17.421 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:49:17.421 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:49:17.421 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "throttled", 2024-03-11T21:49:17.421 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:46:14.643074+0000", 2024-03-11T21:49:17.421 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 6.5700000000000002e-07 2024-03-11T21:49:17.421 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:49:17.421 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:49:17.421 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "all_read", 2024-03-11T21:49:17.421 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:46:14.643117+0000", 2024-03-11T21:49:17.421 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 4.2772999999999999e-05 2024-03-11T21:49:17.421 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:49:17.421 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:49:17.421 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "dispatched", 2024-03-11T21:49:17.421 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:46:14.643123+0000", 2024-03-11T21:49:17.421 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 5.8690000000000001e-06 2024-03-11T21:49:17.422 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:49:17.422 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:49:17.422 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "queued_for_pg", 2024-03-11T21:49:17.422 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:46:14.643128+0000", 2024-03-11T21:49:17.422 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 5.0769999999999998e-06 2024-03-11T21:49:17.422 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:49:17.422 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:49:17.422 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "reached_pg", 2024-03-11T21:49:17.422 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:46:14.643184+0000", 2024-03-11T21:49:17.422 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 5.5467000000000003e-05 2024-03-11T21:49:17.422 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:49:17.422 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:49:17.422 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "started", 2024-03-11T21:49:17.422 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:46:14.643301+0000", 2024-03-11T21:49:17.423 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.000117419 2024-03-11T21:49:17.423 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:49:17.423 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:49:17.423 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "waiting for subops from 1,2", 2024-03-11T21:49:17.423 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:46:14.643442+0000", 2024-03-11T21:49:17.423 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.000140612 2024-03-11T21:49:17.423 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:49:17.423 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:49:17.423 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "op_commit", 2024-03-11T21:49:17.423 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:46:14.647799+0000", 2024-03-11T21:49:17.423 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.0043573010000000001 2024-03-11T21:49:17.423 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:49:17.423 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:49:17.423 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "sub_op_commit_rec", 2024-03-11T21:49:17.423 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:46:14.648092+0000", 2024-03-11T21:49:17.424 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.00029267300000000002 2024-03-11T21:49:17.424 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:49:17.424 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:49:17.424 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "sub_op_commit_rec", 2024-03-11T21:49:17.424 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:46:14.767397+0000", 2024-03-11T21:49:17.424 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.119305473 2024-03-11T21:49:17.424 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:49:17.424 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:49:17.424 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "commit_sent", 2024-03-11T21:49:17.424 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:46:14.767427+0000", 2024-03-11T21:49:17.424 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 2.9915000000000001e-05 2024-03-11T21:49:17.424 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:49:17.424 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:49:17.424 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "done", 2024-03-11T21:49:17.425 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:46:14.767450+0000", 2024-03-11T21:49:17.425 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 2.2855000000000001e-05 2024-03-11T21:49:17.425 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-03-11T21:49:17.425 INFO:teuthology.orchestra.run.smithi097.stdout: ] 2024-03-11T21:49:17.425 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-03-11T21:49:17.425 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:49:17.425 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:49:17.425 INFO:teuthology.orchestra.run.smithi097.stdout: "description": "osd_op(client.35813.0:109107 11.f 11:f420dd0b:::benchmark_data_smithi092_186520_object109106:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e463)", 2024-03-11T21:49:17.425 INFO:teuthology.orchestra.run.smithi097.stdout: "initiated_at": "2024-03-11T21:46:14.643200+0000", 2024-03-11T21:49:17.425 INFO:teuthology.orchestra.run.smithi097.stdout: "age": 182.76353286599999, 2024-03-11T21:49:17.425 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.124409646, 2024-03-11T21:49:17.425 INFO:teuthology.orchestra.run.smithi097.stdout: "type_data": { 2024-03-11T21:49:17.425 INFO:teuthology.orchestra.run.smithi097.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-11T21:49:17.425 INFO:teuthology.orchestra.run.smithi097.stdout: "client_info": { 2024-03-11T21:49:17.426 INFO:teuthology.orchestra.run.smithi097.stdout: "client": "client.35813", 2024-03-11T21:49:17.426 INFO:teuthology.orchestra.run.smithi097.stdout: "client_addr": "v1:172.21.15.92:0/4293352030", 2024-03-11T21:49:17.426 INFO:teuthology.orchestra.run.smithi097.stdout: "tid": 109107 2024-03-11T21:49:17.426 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:49:17.426 INFO:teuthology.orchestra.run.smithi097.stdout: "events": [ 2024-03-11T21:49:17.426 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:49:17.426 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "initiated", 2024-03-11T21:49:17.426 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:46:14.643200+0000", 2024-03-11T21:49:17.426 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-03-11T21:49:17.426 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:49:17.426 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:49:17.426 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "header_read", 2024-03-11T21:49:17.426 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:46:14.643200+0000", 2024-03-11T21:49:17.426 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-03-11T21:49:17.427 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:49:17.427 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:49:17.427 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "throttled", 2024-03-11T21:49:17.427 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:46:14.643201+0000", 2024-03-11T21:49:17.427 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 5.8400000000000004e-07 2024-03-11T21:49:17.427 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:49:17.427 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:49:17.427 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "all_read", 2024-03-11T21:49:17.427 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:46:14.643227+0000", 2024-03-11T21:49:17.427 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 2.6503e-05 2024-03-11T21:49:17.427 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:49:17.427 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:49:17.427 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "dispatched", 2024-03-11T21:49:17.427 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:46:14.643233+0000", 2024-03-11T21:49:17.428 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 5.4929999999999998e-06 2024-03-11T21:49:17.428 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:49:17.428 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:49:17.428 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "queued_for_pg", 2024-03-11T21:49:17.428 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:46:14.643238+0000", 2024-03-11T21:49:17.428 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 5.4399999999999996e-06 2024-03-11T21:49:17.428 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:49:17.428 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:49:17.428 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "reached_pg", 2024-03-11T21:49:17.428 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:46:14.645654+0000", 2024-03-11T21:49:17.428 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.0024158159999999999 2024-03-11T21:49:17.428 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:49:17.428 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:49:17.428 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "started", 2024-03-11T21:49:17.429 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:46:14.645763+0000", 2024-03-11T21:49:17.429 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.00010857399999999999 2024-03-11T21:49:17.429 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:49:17.429 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:49:17.429 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "waiting for subops from 1,2", 2024-03-11T21:49:17.429 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:46:14.645895+0000", 2024-03-11T21:49:17.429 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.00013246800000000001 2024-03-11T21:49:17.429 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:49:17.429 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:49:17.429 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "op_commit", 2024-03-11T21:49:17.429 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:46:14.648668+0000", 2024-03-11T21:49:17.429 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.0027729019999999998 2024-03-11T21:49:17.429 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:49:17.429 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:49:17.430 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "sub_op_commit_rec", 2024-03-11T21:49:17.430 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:46:14.651411+0000", 2024-03-11T21:49:17.430 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.0027425169999999999 2024-03-11T21:49:17.430 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:49:17.430 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:49:17.430 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "sub_op_commit_rec", 2024-03-11T21:49:17.430 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:46:14.767500+0000", 2024-03-11T21:49:17.430 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.116089493 2024-03-11T21:49:17.430 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:49:17.430 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:49:17.430 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "commit_sent", 2024-03-11T21:49:17.430 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:46:14.767566+0000", 2024-03-11T21:49:17.430 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 6.5955000000000001e-05 2024-03-11T21:49:17.430 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:49:17.431 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:49:17.431 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "done", 2024-03-11T21:49:17.431 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:46:14.767610+0000", 2024-03-11T21:49:17.431 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 4.3900999999999999e-05 2024-03-11T21:49:17.431 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-03-11T21:49:17.431 INFO:teuthology.orchestra.run.smithi097.stdout: ] 2024-03-11T21:49:17.431 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-03-11T21:49:17.431 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:49:17.431 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:49:17.431 INFO:teuthology.orchestra.run.smithi097.stdout: "description": "osd_op(client.35813.0:189867 11.f 11:fbbcc9ff:::benchmark_data_smithi092_186520_object51519:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e477)", 2024-03-11T21:49:17.431 INFO:teuthology.orchestra.run.smithi097.stdout: "initiated_at": "2024-03-11T21:47:15.866472+0000", 2024-03-11T21:49:17.431 INFO:teuthology.orchestra.run.smithi097.stdout: "age": 121.54026127900001, 2024-03-11T21:49:17.431 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.33759599699999998, 2024-03-11T21:49:17.431 INFO:teuthology.orchestra.run.smithi097.stdout: "type_data": { 2024-03-11T21:49:17.432 INFO:teuthology.orchestra.run.smithi097.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-11T21:49:17.432 INFO:teuthology.orchestra.run.smithi097.stdout: "client_info": { 2024-03-11T21:49:17.432 INFO:teuthology.orchestra.run.smithi097.stdout: "client": "client.35813", 2024-03-11T21:49:17.432 INFO:teuthology.orchestra.run.smithi097.stdout: "client_addr": "v1:172.21.15.92:0/4293352030", 2024-03-11T21:49:17.432 INFO:teuthology.orchestra.run.smithi097.stdout: "tid": 189867 2024-03-11T21:49:17.432 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:49:17.432 INFO:teuthology.orchestra.run.smithi097.stdout: "events": [ 2024-03-11T21:49:17.432 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:49:17.432 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "initiated", 2024-03-11T21:49:17.432 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:47:15.866472+0000", 2024-03-11T21:49:17.432 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-03-11T21:49:17.432 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:49:17.432 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:49:17.432 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "header_read", 2024-03-11T21:49:17.433 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:47:15.866472+0000", 2024-03-11T21:49:17.433 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-03-11T21:49:17.433 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:49:17.433 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:49:17.433 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "throttled", 2024-03-11T21:49:17.433 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:47:15.866473+0000", 2024-03-11T21:49:17.433 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.4839999999999999e-06 2024-03-11T21:49:17.433 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:49:17.433 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:49:17.433 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "all_read", 2024-03-11T21:49:17.433 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:47:15.866479+0000", 2024-03-11T21:49:17.433 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 5.1970000000000004e-06 2024-03-11T21:49:17.433 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:49:17.433 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:49:17.434 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "dispatched", 2024-03-11T21:49:17.434 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:47:15.866493+0000", 2024-03-11T21:49:17.434 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.4207e-05 2024-03-11T21:49:17.434 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:49:17.434 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:49:17.434 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "queued_for_pg", 2024-03-11T21:49:17.434 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:47:15.866501+0000", 2024-03-11T21:49:17.434 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 8.2430000000000005e-06 2024-03-11T21:49:17.434 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:49:17.434 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:49:17.434 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "reached_pg", 2024-03-11T21:49:17.434 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:47:15.867091+0000", 2024-03-11T21:49:17.434 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.00058980199999999995 2024-03-11T21:49:17.434 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:49:17.435 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:49:17.435 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "started", 2024-03-11T21:49:17.435 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:47:15.867212+0000", 2024-03-11T21:49:17.435 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.00012129999999999999 2024-03-11T21:49:17.435 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:49:17.435 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:49:17.435 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "waiting for subops from 1,2", 2024-03-11T21:49:17.435 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:47:15.867321+0000", 2024-03-11T21:49:17.435 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.000109262 2024-03-11T21:49:17.435 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:49:17.435 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:49:17.435 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "op_commit", 2024-03-11T21:49:17.435 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:47:15.869561+0000", 2024-03-11T21:49:17.435 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.002239497 2024-03-11T21:49:17.436 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:49:17.436 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:49:17.436 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "sub_op_commit_rec", 2024-03-11T21:49:17.436 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:47:15.957463+0000", 2024-03-11T21:49:17.436 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.087902471999999995 2024-03-11T21:49:17.436 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:49:17.436 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:49:17.436 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "sub_op_commit_rec", 2024-03-11T21:49:17.436 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:47:16.203969+0000", 2024-03-11T21:49:17.436 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.246505372 2024-03-11T21:49:17.436 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:49:17.436 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:49:17.436 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "commit_sent", 2024-03-11T21:49:17.436 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:47:16.204024+0000", 2024-03-11T21:49:17.436 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 5.5643999999999998e-05 2024-03-11T21:49:17.437 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:49:17.437 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:49:17.437 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "done", 2024-03-11T21:49:17.437 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:47:16.204068+0000", 2024-03-11T21:49:17.437 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 4.3516999999999997e-05 2024-03-11T21:49:17.437 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-03-11T21:49:17.437 INFO:teuthology.orchestra.run.smithi097.stdout: ] 2024-03-11T21:49:17.437 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-03-11T21:49:17.437 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:49:17.437 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:49:17.437 INFO:teuthology.orchestra.run.smithi097.stdout: "description": "osd_op(client.35813.0:189868 11.f 11:f10b9a69:::benchmark_data_smithi092_186520_object51520:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e477)", 2024-03-11T21:49:17.437 INFO:teuthology.orchestra.run.smithi097.stdout: "initiated_at": "2024-03-11T21:47:15.867270+0000", 2024-03-11T21:49:17.437 INFO:teuthology.orchestra.run.smithi097.stdout: "age": 121.539462908, 2024-03-11T21:49:17.437 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.33695803099999999, 2024-03-11T21:49:17.438 INFO:teuthology.orchestra.run.smithi097.stdout: "type_data": { 2024-03-11T21:49:17.438 INFO:teuthology.orchestra.run.smithi097.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-11T21:49:17.438 INFO:teuthology.orchestra.run.smithi097.stdout: "client_info": { 2024-03-11T21:49:17.438 INFO:teuthology.orchestra.run.smithi097.stdout: "client": "client.35813", 2024-03-11T21:49:17.438 INFO:teuthology.orchestra.run.smithi097.stdout: "client_addr": "v1:172.21.15.92:0/4293352030", 2024-03-11T21:49:17.438 INFO:teuthology.orchestra.run.smithi097.stdout: "tid": 189868 2024-03-11T21:49:17.438 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:49:17.438 INFO:teuthology.orchestra.run.smithi097.stdout: "events": [ 2024-03-11T21:49:17.438 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:49:17.438 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "initiated", 2024-03-11T21:49:17.438 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:47:15.867270+0000", 2024-03-11T21:49:17.438 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-03-11T21:49:17.438 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:49:17.438 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:49:17.439 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "header_read", 2024-03-11T21:49:17.439 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:47:15.867270+0000", 2024-03-11T21:49:17.439 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-03-11T21:49:17.439 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:49:17.439 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:49:17.439 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "throttled", 2024-03-11T21:49:17.439 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:47:15.867272+0000", 2024-03-11T21:49:17.439 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.328e-06 2024-03-11T21:49:17.439 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:49:17.439 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:49:17.439 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "all_read", 2024-03-11T21:49:17.439 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:47:15.867276+0000", 2024-03-11T21:49:17.439 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 4.814e-06 2024-03-11T21:49:17.439 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:49:17.439 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:49:17.440 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "dispatched", 2024-03-11T21:49:17.440 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:47:15.867289+0000", 2024-03-11T21:49:17.440 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.2383e-05 2024-03-11T21:49:17.440 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:49:17.440 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:49:17.440 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "queued_for_pg", 2024-03-11T21:49:17.440 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:47:15.867296+0000", 2024-03-11T21:49:17.440 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 7.5599999999999996e-06 2024-03-11T21:49:17.440 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:49:17.440 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:49:17.440 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "reached_pg", 2024-03-11T21:49:17.440 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:47:15.868519+0000", 2024-03-11T21:49:17.440 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.0012221910000000001 2024-03-11T21:49:17.440 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:49:17.440 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:49:17.440 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "started", 2024-03-11T21:49:17.441 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:47:15.868603+0000", 2024-03-11T21:49:17.441 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 8.4325000000000003e-05 2024-03-11T21:49:17.441 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:49:17.441 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:49:17.441 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "waiting for subops from 1,2", 2024-03-11T21:49:17.441 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:47:15.868671+0000", 2024-03-11T21:49:17.441 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 6.7747e-05 2024-03-11T21:49:17.441 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:49:17.441 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:49:17.441 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "op_commit", 2024-03-11T21:49:17.441 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:47:15.869672+0000", 2024-03-11T21:49:17.441 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.0010016199999999999 2024-03-11T21:49:17.441 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:49:17.441 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:49:17.442 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "sub_op_commit_rec", 2024-03-11T21:49:17.442 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:47:15.957500+0000", 2024-03-11T21:49:17.442 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.087828205000000006 2024-03-11T21:49:17.442 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:49:17.442 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:49:17.442 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "sub_op_commit_rec", 2024-03-11T21:49:17.442 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:47:16.204143+0000", 2024-03-11T21:49:17.442 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.24664250200000001 2024-03-11T21:49:17.442 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:49:17.442 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:49:17.442 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "commit_sent", 2024-03-11T21:49:17.442 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:47:16.204198+0000", 2024-03-11T21:49:17.442 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 5.5028999999999998e-05 2024-03-11T21:49:17.442 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:49:17.443 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:49:17.443 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "done", 2024-03-11T21:49:17.443 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:47:16.204228+0000", 2024-03-11T21:49:17.443 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 3.0326999999999999e-05 2024-03-11T21:49:17.443 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-03-11T21:49:17.443 INFO:teuthology.orchestra.run.smithi097.stdout: ] 2024-03-11T21:49:17.443 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-03-11T21:49:17.443 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:49:17.443 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:49:17.443 INFO:teuthology.orchestra.run.smithi097.stdout: "description": "osd_op(client.35813.0:189872 11.f 11:f4892482:::benchmark_data_smithi092_186520_object51524:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e477)", 2024-03-11T21:49:17.443 INFO:teuthology.orchestra.run.smithi097.stdout: "initiated_at": "2024-03-11T21:47:15.872718+0000", 2024-03-11T21:49:17.443 INFO:teuthology.orchestra.run.smithi097.stdout: "age": 121.534015057, 2024-03-11T21:49:17.443 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.33160263800000001, 2024-03-11T21:49:17.443 INFO:teuthology.orchestra.run.smithi097.stdout: "type_data": { 2024-03-11T21:49:17.444 INFO:teuthology.orchestra.run.smithi097.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-11T21:49:17.444 INFO:teuthology.orchestra.run.smithi097.stdout: "client_info": { 2024-03-11T21:49:17.444 INFO:teuthology.orchestra.run.smithi097.stdout: "client": "client.35813", 2024-03-11T21:49:17.444 INFO:teuthology.orchestra.run.smithi097.stdout: "client_addr": "v1:172.21.15.92:0/4293352030", 2024-03-11T21:49:17.444 INFO:teuthology.orchestra.run.smithi097.stdout: "tid": 189872 2024-03-11T21:49:17.444 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:49:17.444 INFO:teuthology.orchestra.run.smithi097.stdout: "events": [ 2024-03-11T21:49:17.444 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:49:17.444 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "initiated", 2024-03-11T21:49:17.444 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:47:15.872718+0000", 2024-03-11T21:49:17.444 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-03-11T21:49:17.444 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:49:17.444 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:49:17.444 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "header_read", 2024-03-11T21:49:17.445 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:47:15.872718+0000", 2024-03-11T21:49:17.445 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-03-11T21:49:17.445 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:49:17.445 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:49:17.445 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "throttled", 2024-03-11T21:49:17.445 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:47:15.872719+0000", 2024-03-11T21:49:17.445 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.173e-06 2024-03-11T21:49:17.445 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:49:17.445 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:49:17.445 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "all_read", 2024-03-11T21:49:17.445 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:47:15.872723+0000", 2024-03-11T21:49:17.445 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 4.2119999999999997e-06 2024-03-11T21:49:17.445 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:49:17.445 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:49:17.445 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "dispatched", 2024-03-11T21:49:17.446 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:47:15.872730+0000", 2024-03-11T21:49:17.446 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 6.849e-06 2024-03-11T21:49:17.446 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:49:17.446 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:49:17.446 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "queued_for_pg", 2024-03-11T21:49:17.446 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:47:15.872735+0000", 2024-03-11T21:49:17.446 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 4.4909999999999999e-06 2024-03-11T21:49:17.446 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:49:17.446 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:49:17.446 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "reached_pg", 2024-03-11T21:49:17.446 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:47:15.872785+0000", 2024-03-11T21:49:17.446 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 5.0334999999999998e-05 2024-03-11T21:49:17.446 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:49:17.446 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:49:17.447 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "started", 2024-03-11T21:49:17.447 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:47:15.872908+0000", 2024-03-11T21:49:17.447 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.00012238700000000001 2024-03-11T21:49:17.447 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:49:17.447 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:49:17.447 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "waiting for subops from 1,2", 2024-03-11T21:49:17.447 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:47:15.872981+0000", 2024-03-11T21:49:17.447 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 7.3104999999999999e-05 2024-03-11T21:49:17.447 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:49:17.447 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:49:17.447 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "op_commit", 2024-03-11T21:49:17.447 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:47:15.873986+0000", 2024-03-11T21:49:17.447 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.0010055280000000001 2024-03-11T21:49:17.447 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:49:17.447 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:49:17.448 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "sub_op_commit_rec", 2024-03-11T21:49:17.448 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:47:15.958459+0000", 2024-03-11T21:49:17.448 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.084472951000000004 2024-03-11T21:49:17.448 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:49:17.448 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:49:17.448 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "sub_op_commit_rec", 2024-03-11T21:49:17.448 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:47:16.204266+0000", 2024-03-11T21:49:17.448 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.245807267 2024-03-11T21:49:17.448 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:49:17.448 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:49:17.448 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "commit_sent", 2024-03-11T21:49:17.448 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:47:16.204297+0000", 2024-03-11T21:49:17.448 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 3.1009000000000001e-05 2024-03-11T21:49:17.448 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:49:17.448 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:49:17.449 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "done", 2024-03-11T21:49:17.449 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:47:16.204321+0000", 2024-03-11T21:49:17.449 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 2.3331000000000001e-05 2024-03-11T21:49:17.449 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-03-11T21:49:17.449 INFO:teuthology.orchestra.run.smithi097.stdout: ] 2024-03-11T21:49:17.449 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-03-11T21:49:17.449 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:49:17.449 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:49:17.449 INFO:teuthology.orchestra.run.smithi097.stdout: "description": "osd_op(client.35813.0:189880 11.f 11:f105bd44:::benchmark_data_smithi092_186520_object51532:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e477)", 2024-03-11T21:49:17.449 INFO:teuthology.orchestra.run.smithi097.stdout: "initiated_at": "2024-03-11T21:47:15.954073+0000", 2024-03-11T21:49:17.449 INFO:teuthology.orchestra.run.smithi097.stdout: "age": 121.452660227, 2024-03-11T21:49:17.449 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.25602427799999999, 2024-03-11T21:49:17.449 INFO:teuthology.orchestra.run.smithi097.stdout: "type_data": { 2024-03-11T21:49:17.449 INFO:teuthology.orchestra.run.smithi097.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-11T21:49:17.450 INFO:teuthology.orchestra.run.smithi097.stdout: "client_info": { 2024-03-11T21:49:17.450 INFO:teuthology.orchestra.run.smithi097.stdout: "client": "client.35813", 2024-03-11T21:49:17.450 INFO:teuthology.orchestra.run.smithi097.stdout: "client_addr": "v1:172.21.15.92:0/4293352030", 2024-03-11T21:49:17.450 INFO:teuthology.orchestra.run.smithi097.stdout: "tid": 189880 2024-03-11T21:49:17.450 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:49:17.450 INFO:teuthology.orchestra.run.smithi097.stdout: "events": [ 2024-03-11T21:49:17.450 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:49:17.450 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "initiated", 2024-03-11T21:49:17.450 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:47:15.954073+0000", 2024-03-11T21:49:17.450 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-03-11T21:49:17.450 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:49:17.450 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:49:17.450 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "header_read", 2024-03-11T21:49:17.450 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:47:15.954073+0000", 2024-03-11T21:49:17.451 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-03-11T21:49:17.451 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:49:17.451 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:49:17.451 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "throttled", 2024-03-11T21:49:17.451 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:47:15.954075+0000", 2024-03-11T21:49:17.451 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.595e-06 2024-03-11T21:49:17.451 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:49:17.451 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:49:17.451 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "all_read", 2024-03-11T21:49:17.451 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:47:15.954079+0000", 2024-03-11T21:49:17.451 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 4.6890000000000003e-06 2024-03-11T21:49:17.451 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:49:17.451 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:49:17.451 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "dispatched", 2024-03-11T21:49:17.452 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:47:15.954087+0000", 2024-03-11T21:49:17.452 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 7.9710000000000004e-06 2024-03-11T21:49:17.452 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:49:17.452 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:49:17.452 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "queued_for_pg", 2024-03-11T21:49:17.452 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:47:15.954094+0000", 2024-03-11T21:49:17.452 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 6.5220000000000004e-06 2024-03-11T21:49:17.452 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:49:17.452 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:49:17.452 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "reached_pg", 2024-03-11T21:49:17.452 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:47:15.954153+0000", 2024-03-11T21:49:17.452 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 5.9630000000000003e-05 2024-03-11T21:49:17.452 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:49:17.452 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:49:17.453 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "started", 2024-03-11T21:49:17.453 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:47:15.954265+0000", 2024-03-11T21:49:17.453 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.00011181 2024-03-11T21:49:17.453 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:49:17.453 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:49:17.453 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "waiting for subops from 1,2", 2024-03-11T21:49:17.453 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:47:15.954421+0000", 2024-03-11T21:49:17.453 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.000155833 2024-03-11T21:49:17.453 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:49:17.453 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:49:17.453 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "op_commit", 2024-03-11T21:49:17.453 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:47:15.957426+0000", 2024-03-11T21:49:17.453 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.003005424 2024-03-11T21:49:17.453 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:49:17.453 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:49:17.454 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "sub_op_commit_rec", 2024-03-11T21:49:17.454 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:47:15.959064+0000", 2024-03-11T21:49:17.454 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.0016377679999999999 2024-03-11T21:49:17.454 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:49:17.454 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:49:17.454 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "sub_op_commit_rec", 2024-03-11T21:49:17.454 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:47:16.210009+0000", 2024-03-11T21:49:17.454 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.250944477 2024-03-11T21:49:17.454 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:49:17.454 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:49:17.454 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "commit_sent", 2024-03-11T21:49:17.454 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:47:16.210057+0000", 2024-03-11T21:49:17.454 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 4.8408999999999997e-05 2024-03-11T21:49:17.455 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:49:17.455 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:49:17.455 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "done", 2024-03-11T21:49:17.455 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:47:16.210097+0000", 2024-03-11T21:49:17.455 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 4.015e-05 2024-03-11T21:49:17.455 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-03-11T21:49:17.455 INFO:teuthology.orchestra.run.smithi097.stdout: ] 2024-03-11T21:49:17.455 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-03-11T21:49:17.455 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:49:17.455 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:49:17.455 INFO:teuthology.orchestra.run.smithi097.stdout: "description": "osd_op(client.35813.0:189884 11.f 11:f0dac027:::benchmark_data_smithi092_186520_object51536:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e477)", 2024-03-11T21:49:17.455 INFO:teuthology.orchestra.run.smithi097.stdout: "initiated_at": "2024-03-11T21:47:15.959426+0000", 2024-03-11T21:49:17.455 INFO:teuthology.orchestra.run.smithi097.stdout: "age": 121.447306754, 2024-03-11T21:49:17.455 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.250788815, 2024-03-11T21:49:17.456 INFO:teuthology.orchestra.run.smithi097.stdout: "type_data": { 2024-03-11T21:49:17.456 INFO:teuthology.orchestra.run.smithi097.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-11T21:49:17.456 INFO:teuthology.orchestra.run.smithi097.stdout: "client_info": { 2024-03-11T21:49:17.456 INFO:teuthology.orchestra.run.smithi097.stdout: "client": "client.35813", 2024-03-11T21:49:17.456 INFO:teuthology.orchestra.run.smithi097.stdout: "client_addr": "v1:172.21.15.92:0/4293352030", 2024-03-11T21:49:17.456 INFO:teuthology.orchestra.run.smithi097.stdout: "tid": 189884 2024-03-11T21:49:17.456 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:49:17.456 INFO:teuthology.orchestra.run.smithi097.stdout: "events": [ 2024-03-11T21:49:17.456 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:49:17.456 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "initiated", 2024-03-11T21:49:17.456 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:47:15.959426+0000", 2024-03-11T21:49:17.456 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-03-11T21:49:17.456 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:49:17.456 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:49:17.457 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "header_read", 2024-03-11T21:49:17.457 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:47:15.959426+0000", 2024-03-11T21:49:17.457 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-03-11T21:49:17.457 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:49:17.457 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:49:17.457 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "throttled", 2024-03-11T21:49:17.457 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:47:15.959428+0000", 2024-03-11T21:49:17.457 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.567e-06 2024-03-11T21:49:17.457 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:49:17.457 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:49:17.457 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "all_read", 2024-03-11T21:49:17.457 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:47:15.959432+0000", 2024-03-11T21:49:17.457 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 4.4819999999999996e-06 2024-03-11T21:49:17.457 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:49:17.457 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:49:17.458 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "dispatched", 2024-03-11T21:49:17.458 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:47:15.959443+0000", 2024-03-11T21:49:17.458 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.0071000000000001e-05 2024-03-11T21:49:17.458 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:49:17.458 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:49:17.458 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "queued_for_pg", 2024-03-11T21:49:17.458 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:47:15.959450+0000", 2024-03-11T21:49:17.458 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 7.0509999999999999e-06 2024-03-11T21:49:17.458 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:49:17.458 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:49:17.458 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "reached_pg", 2024-03-11T21:49:17.458 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:47:15.959491+0000", 2024-03-11T21:49:17.458 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 4.1588000000000002e-05 2024-03-11T21:49:17.458 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:49:17.459 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:49:17.459 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "started", 2024-03-11T21:49:17.459 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:47:15.959594+0000", 2024-03-11T21:49:17.459 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.000102889 2024-03-11T21:49:17.459 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:49:17.459 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:49:17.459 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "waiting for subops from 1,2", 2024-03-11T21:49:17.459 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:47:15.959700+0000", 2024-03-11T21:49:17.459 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.000105572 2024-03-11T21:49:17.459 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:49:17.459 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:49:17.459 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "op_commit", 2024-03-11T21:49:17.459 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:47:15.961326+0000", 2024-03-11T21:49:17.459 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.0016263720000000001 2024-03-11T21:49:17.460 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:49:17.460 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:49:17.460 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "sub_op_commit_rec", 2024-03-11T21:49:17.460 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:47:15.965292+0000", 2024-03-11T21:49:17.460 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.0039658549999999999 2024-03-11T21:49:17.460 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:49:17.460 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:49:17.460 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "sub_op_commit_rec", 2024-03-11T21:49:17.460 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:47:16.210144+0000", 2024-03-11T21:49:17.460 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.24485191100000001 2024-03-11T21:49:17.460 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:49:17.460 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:49:17.460 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "commit_sent", 2024-03-11T21:49:17.460 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:47:16.210183+0000", 2024-03-11T21:49:17.460 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 3.9320000000000003e-05 2024-03-11T21:49:17.461 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:49:17.461 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:49:17.461 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "done", 2024-03-11T21:49:17.461 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:47:16.210215+0000", 2024-03-11T21:49:17.461 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 3.2137000000000001e-05 2024-03-11T21:49:17.461 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-03-11T21:49:17.461 INFO:teuthology.orchestra.run.smithi097.stdout: ] 2024-03-11T21:49:17.461 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-03-11T21:49:17.461 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:49:17.461 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:49:17.461 INFO:teuthology.orchestra.run.smithi097.stdout: "description": "osd_op(client.35813.0:194245 11.0 11:04f61b4a:::benchmark_data_smithi092_186520_object55897:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e478)", 2024-03-11T21:49:17.461 INFO:teuthology.orchestra.run.smithi097.stdout: "initiated_at": "2024-03-11T21:47:19.289963+0000", 2024-03-11T21:49:17.461 INFO:teuthology.orchestra.run.smithi097.stdout: "age": 118.116769891, 2024-03-11T21:49:17.461 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.006909402, 2024-03-11T21:49:17.461 INFO:teuthology.orchestra.run.smithi097.stdout: "type_data": { 2024-03-11T21:49:17.462 INFO:teuthology.orchestra.run.smithi097.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-11T21:49:17.462 INFO:teuthology.orchestra.run.smithi097.stdout: "client_info": { 2024-03-11T21:49:17.462 INFO:teuthology.orchestra.run.smithi097.stdout: "client": "client.35813", 2024-03-11T21:49:17.462 INFO:teuthology.orchestra.run.smithi097.stdout: "client_addr": "v1:172.21.15.92:0/4293352030", 2024-03-11T21:49:17.462 INFO:teuthology.orchestra.run.smithi097.stdout: "tid": 194245 2024-03-11T21:49:17.462 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:49:17.462 INFO:teuthology.orchestra.run.smithi097.stdout: "events": [ 2024-03-11T21:49:17.462 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:49:17.462 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "initiated", 2024-03-11T21:49:17.462 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:47:19.289963+0000", 2024-03-11T21:49:17.462 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-03-11T21:49:17.462 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:49:17.462 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:49:17.462 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "header_read", 2024-03-11T21:49:17.463 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:47:19.289963+0000", 2024-03-11T21:49:17.463 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-03-11T21:49:17.463 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:49:17.463 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:49:17.463 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "throttled", 2024-03-11T21:49:17.463 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:47:19.289965+0000", 2024-03-11T21:49:17.463 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.8360000000000001e-06 2024-03-11T21:49:17.463 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:49:17.463 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:49:17.463 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "all_read", 2024-03-11T21:49:17.463 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:47:19.289970+0000", 2024-03-11T21:49:17.463 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 5.1220000000000003e-06 2024-03-11T21:49:17.463 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:49:17.463 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:49:17.463 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "dispatched", 2024-03-11T21:49:17.464 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:47:19.289978+0000", 2024-03-11T21:49:17.464 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 8.2269999999999993e-06 2024-03-11T21:49:17.464 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:49:17.464 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:49:17.464 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "queued_for_pg", 2024-03-11T21:49:17.464 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:47:19.289986+0000", 2024-03-11T21:49:17.464 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 7.1999999999999997e-06 2024-03-11T21:49:17.464 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:49:17.464 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:49:17.464 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "reached_pg", 2024-03-11T21:49:17.464 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:47:19.290035+0000", 2024-03-11T21:49:17.464 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 4.9607999999999997e-05 2024-03-11T21:49:17.464 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:49:17.464 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:49:17.464 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "waiting for peered", 2024-03-11T21:49:17.465 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:47:19.290042+0000", 2024-03-11T21:49:17.465 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 6.5529999999999998e-06 2024-03-11T21:49:17.465 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:49:17.465 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:49:17.465 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "reached_pg", 2024-03-11T21:49:17.465 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:47:20.291416+0000", 2024-03-11T21:49:17.465 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.001373906 2024-03-11T21:49:17.465 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:49:17.465 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:49:17.465 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "started", 2024-03-11T21:49:17.465 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:47:20.292200+0000", 2024-03-11T21:49:17.465 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.00078473699999999998 2024-03-11T21:49:17.465 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:49:17.465 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:49:17.465 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "waiting for subops from 1,6", 2024-03-11T21:49:17.466 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:47:20.292275+0000", 2024-03-11T21:49:17.466 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 7.4338999999999994e-05 2024-03-11T21:49:17.466 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:49:17.466 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:49:17.466 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "sub_op_commit_rec", 2024-03-11T21:49:17.466 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:47:20.296711+0000", 2024-03-11T21:49:17.466 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.0044361610000000001 2024-03-11T21:49:17.466 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:49:17.466 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:49:17.466 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "op_commit", 2024-03-11T21:49:17.466 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:47:20.296727+0000", 2024-03-11T21:49:17.466 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.5996000000000001e-05 2024-03-11T21:49:17.466 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:49:17.466 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:49:17.466 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "sub_op_commit_rec", 2024-03-11T21:49:17.467 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:47:20.296767+0000", 2024-03-11T21:49:17.468 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 4.0034999999999999e-05 2024-03-11T21:49:17.468 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:49:17.468 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:49:17.468 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "commit_sent", 2024-03-11T21:49:17.468 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:47:20.296822+0000", 2024-03-11T21:49:17.468 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 5.4577999999999998e-05 2024-03-11T21:49:17.468 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:49:17.468 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:49:17.468 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "done", 2024-03-11T21:49:17.468 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:47:20.296873+0000", 2024-03-11T21:49:17.468 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 5.1103999999999997e-05 2024-03-11T21:49:17.468 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-03-11T21:49:17.469 INFO:teuthology.orchestra.run.smithi097.stdout: ] 2024-03-11T21:49:17.469 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-03-11T21:49:17.469 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:49:17.469 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:49:17.469 INFO:teuthology.orchestra.run.smithi097.stdout: "description": "osd_op(client.35813.0:194252 11.0 11:0023d5c4:::benchmark_data_smithi092_186520_object55904:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e478)", 2024-03-11T21:49:17.469 INFO:teuthology.orchestra.run.smithi097.stdout: "initiated_at": "2024-03-11T21:47:19.302390+0000", 2024-03-11T21:49:17.469 INFO:teuthology.orchestra.run.smithi097.stdout: "age": 118.104343075, 2024-03-11T21:49:17.469 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.99809470600000005, 2024-03-11T21:49:17.469 INFO:teuthology.orchestra.run.smithi097.stdout: "type_data": { 2024-03-11T21:49:17.469 INFO:teuthology.orchestra.run.smithi097.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-11T21:49:17.469 INFO:teuthology.orchestra.run.smithi097.stdout: "client_info": { 2024-03-11T21:49:17.469 INFO:teuthology.orchestra.run.smithi097.stdout: "client": "client.35813", 2024-03-11T21:49:17.469 INFO:teuthology.orchestra.run.smithi097.stdout: "client_addr": "v1:172.21.15.92:0/4293352030", 2024-03-11T21:49:17.469 INFO:teuthology.orchestra.run.smithi097.stdout: "tid": 194252 2024-03-11T21:49:17.469 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:49:17.470 INFO:teuthology.orchestra.run.smithi097.stdout: "events": [ 2024-03-11T21:49:17.470 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:49:17.470 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "initiated", 2024-03-11T21:49:17.470 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:47:19.302390+0000", 2024-03-11T21:49:17.470 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-03-11T21:49:17.470 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:49:17.470 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:49:17.470 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "header_read", 2024-03-11T21:49:17.470 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:47:19.302390+0000", 2024-03-11T21:49:17.470 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-03-11T21:49:17.470 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:49:17.470 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:49:17.470 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "throttled", 2024-03-11T21:49:17.470 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:47:19.302392+0000", 2024-03-11T21:49:17.470 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.4309999999999999e-06 2024-03-11T21:49:17.471 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:49:17.471 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:49:17.471 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "all_read", 2024-03-11T21:49:17.471 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:47:19.302396+0000", 2024-03-11T21:49:17.471 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 4.5369999999999999e-06 2024-03-11T21:49:17.471 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:49:17.471 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:49:17.471 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "dispatched", 2024-03-11T21:49:17.471 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:47:19.302405+0000", 2024-03-11T21:49:17.471 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 8.9940000000000006e-06 2024-03-11T21:49:17.471 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:49:17.471 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:49:17.471 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "queued_for_pg", 2024-03-11T21:49:17.471 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:47:19.302412+0000", 2024-03-11T21:49:17.472 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 7.1840000000000002e-06 2024-03-11T21:49:17.472 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:49:17.472 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:49:17.472 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "reached_pg", 2024-03-11T21:49:17.472 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:47:19.302477+0000", 2024-03-11T21:49:17.472 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 6.5192999999999997e-05 2024-03-11T21:49:17.472 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:49:17.472 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:49:17.472 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "waiting for peered", 2024-03-11T21:49:17.472 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:47:19.302487+0000", 2024-03-11T21:49:17.472 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.0030000000000001e-05 2024-03-11T21:49:17.472 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:49:17.472 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:49:17.472 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "reached_pg", 2024-03-11T21:49:17.472 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:47:20.294124+0000", 2024-03-11T21:49:17.473 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.99163700399999999 2024-03-11T21:49:17.473 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:49:17.473 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:49:17.473 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "started", 2024-03-11T21:49:17.473 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:47:20.294236+0000", 2024-03-11T21:49:17.473 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.000111409 2024-03-11T21:49:17.473 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:49:17.473 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:49:17.473 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "waiting for subops from 1,6", 2024-03-11T21:49:17.473 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:47:20.294313+0000", 2024-03-11T21:49:17.473 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 7.7424999999999998e-05 2024-03-11T21:49:17.473 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:49:17.473 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:49:17.473 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "op_commit", 2024-03-11T21:49:17.474 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:47:20.296912+0000", 2024-03-11T21:49:17.474 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.0025983080000000001 2024-03-11T21:49:17.474 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:49:17.474 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:49:17.474 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "sub_op_commit_rec", 2024-03-11T21:49:17.474 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:47:20.300306+0000", 2024-03-11T21:49:17.474 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.0033948260000000001 2024-03-11T21:49:17.474 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:49:17.474 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:49:17.474 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "sub_op_commit_rec", 2024-03-11T21:49:17.474 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:47:20.300369+0000", 2024-03-11T21:49:17.474 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 6.2294999999999999e-05 2024-03-11T21:49:17.474 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:49:17.474 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:49:17.475 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "commit_sent", 2024-03-11T21:49:17.475 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:47:20.300441+0000", 2024-03-11T21:49:17.475 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 7.2601000000000005e-05 2024-03-11T21:49:17.475 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:49:17.475 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:49:17.475 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "done", 2024-03-11T21:49:17.475 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:47:20.300485+0000", 2024-03-11T21:49:17.475 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 4.3469e-05 2024-03-11T21:49:17.475 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-03-11T21:49:17.475 INFO:teuthology.orchestra.run.smithi097.stdout: ] 2024-03-11T21:49:17.475 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-03-11T21:49:17.475 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:49:17.475 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:49:17.475 INFO:teuthology.orchestra.run.smithi097.stdout: "description": "osd_op(client.35813.0:194276 11.0 11:0252cdd6:::benchmark_data_smithi092_186520_object55928:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e478)", 2024-03-11T21:49:17.475 INFO:teuthology.orchestra.run.smithi097.stdout: "initiated_at": "2024-03-11T21:47:19.325320+0000", 2024-03-11T21:49:17.476 INFO:teuthology.orchestra.run.smithi097.stdout: "age": 118.08141326099999, 2024-03-11T21:49:17.476 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.975387267, 2024-03-11T21:49:17.476 INFO:teuthology.orchestra.run.smithi097.stdout: "type_data": { 2024-03-11T21:49:17.476 INFO:teuthology.orchestra.run.smithi097.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-11T21:49:17.476 INFO:teuthology.orchestra.run.smithi097.stdout: "client_info": { 2024-03-11T21:49:17.476 INFO:teuthology.orchestra.run.smithi097.stdout: "client": "client.35813", 2024-03-11T21:49:17.476 INFO:teuthology.orchestra.run.smithi097.stdout: "client_addr": "v1:172.21.15.92:0/4293352030", 2024-03-11T21:49:17.476 INFO:teuthology.orchestra.run.smithi097.stdout: "tid": 194276 2024-03-11T21:49:17.476 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:49:17.476 INFO:teuthology.orchestra.run.smithi097.stdout: "events": [ 2024-03-11T21:49:17.476 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:49:17.476 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "initiated", 2024-03-11T21:49:17.476 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:47:19.325320+0000", 2024-03-11T21:49:17.476 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-03-11T21:49:17.477 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:49:17.477 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:49:17.477 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "header_read", 2024-03-11T21:49:17.477 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:47:19.325320+0000", 2024-03-11T21:49:17.477 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-03-11T21:49:17.477 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:49:17.477 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:49:17.477 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "throttled", 2024-03-11T21:49:17.477 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:47:19.325321+0000", 2024-03-11T21:49:17.477 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 6.9299999999999997e-07 2024-03-11T21:49:17.477 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:49:17.477 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:49:17.477 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "all_read", 2024-03-11T21:49:17.477 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:47:19.325324+0000", 2024-03-11T21:49:17.477 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 2.9210000000000001e-06 2024-03-11T21:49:17.477 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:49:17.478 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:49:17.478 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "dispatched", 2024-03-11T21:49:17.478 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:47:19.325330+0000", 2024-03-11T21:49:17.478 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 6.1129999999999997e-06 2024-03-11T21:49:17.478 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:49:17.478 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:49:17.478 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "queued_for_pg", 2024-03-11T21:49:17.478 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:47:19.325335+0000", 2024-03-11T21:49:17.478 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 5.7699999999999998e-06 2024-03-11T21:49:17.478 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:49:17.478 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:49:17.478 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "reached_pg", 2024-03-11T21:49:17.478 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:47:19.325392+0000", 2024-03-11T21:49:17.478 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 5.6215999999999998e-05 2024-03-11T21:49:17.479 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:49:17.479 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:49:17.479 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "waiting for peered", 2024-03-11T21:49:17.479 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:47:19.325403+0000", 2024-03-11T21:49:17.479 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.093e-05 2024-03-11T21:49:17.479 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:49:17.479 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:49:17.479 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "reached_pg", 2024-03-11T21:49:17.479 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:47:20.295461+0000", 2024-03-11T21:49:17.479 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.970058486 2024-03-11T21:49:17.479 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:49:17.479 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:49:17.479 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "started", 2024-03-11T21:49:17.479 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:47:20.295586+0000", 2024-03-11T21:49:17.479 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.00012492300000000001 2024-03-11T21:49:17.480 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:49:17.480 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:49:17.480 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "waiting for subops from 1,6", 2024-03-11T21:49:17.480 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:47:20.295649+0000", 2024-03-11T21:49:17.480 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 6.3167e-05 2024-03-11T21:49:17.480 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:49:17.480 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:49:17.480 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "op_commit", 2024-03-11T21:49:17.480 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:47:20.300501+0000", 2024-03-11T21:49:17.480 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.0048520050000000004 2024-03-11T21:49:17.480 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:49:17.480 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:49:17.480 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "sub_op_commit_rec", 2024-03-11T21:49:17.480 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:47:20.300546+0000", 2024-03-11T21:49:17.480 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 4.4808999999999997e-05 2024-03-11T21:49:17.481 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:49:17.481 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:49:17.481 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "sub_op_commit_rec", 2024-03-11T21:49:17.481 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:47:20.300609+0000", 2024-03-11T21:49:17.481 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 6.2675999999999994e-05 2024-03-11T21:49:17.481 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:49:17.481 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:49:17.481 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "commit_sent", 2024-03-11T21:49:17.481 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:47:20.300662+0000", 2024-03-11T21:49:17.481 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 5.3205999999999998e-05 2024-03-11T21:49:17.481 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:49:17.481 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:49:17.481 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "done", 2024-03-11T21:49:17.481 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:47:20.300707+0000", 2024-03-11T21:49:17.482 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 4.5352000000000002e-05 2024-03-11T21:49:17.482 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-03-11T21:49:17.482 INFO:teuthology.orchestra.run.smithi097.stdout: ] 2024-03-11T21:49:17.482 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-03-11T21:49:17.482 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:49:17.482 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:49:17.482 INFO:teuthology.orchestra.run.smithi097.stdout: "description": "osd_op(client.35813.0:194329 11.0 11:008f282f:::benchmark_data_smithi092_186520_object55981:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e478)", 2024-03-11T21:49:17.482 INFO:teuthology.orchestra.run.smithi097.stdout: "initiated_at": "2024-03-11T21:47:19.371815+0000", 2024-03-11T21:49:17.482 INFO:teuthology.orchestra.run.smithi097.stdout: "age": 118.034918629, 2024-03-11T21:49:17.482 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.92909916400000003, 2024-03-11T21:49:17.482 INFO:teuthology.orchestra.run.smithi097.stdout: "type_data": { 2024-03-11T21:49:17.482 INFO:teuthology.orchestra.run.smithi097.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-11T21:49:17.482 INFO:teuthology.orchestra.run.smithi097.stdout: "client_info": { 2024-03-11T21:49:17.482 INFO:teuthology.orchestra.run.smithi097.stdout: "client": "client.35813", 2024-03-11T21:49:17.483 INFO:teuthology.orchestra.run.smithi097.stdout: "client_addr": "v1:172.21.15.92:0/4293352030", 2024-03-11T21:49:17.483 INFO:teuthology.orchestra.run.smithi097.stdout: "tid": 194329 2024-03-11T21:49:17.483 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:49:17.483 INFO:teuthology.orchestra.run.smithi097.stdout: "events": [ 2024-03-11T21:49:17.483 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:49:17.483 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "initiated", 2024-03-11T21:49:17.483 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:47:19.371815+0000", 2024-03-11T21:49:17.483 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-03-11T21:49:17.483 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:49:17.483 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:49:17.483 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "header_read", 2024-03-11T21:49:17.483 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:47:19.371815+0000", 2024-03-11T21:49:17.483 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-03-11T21:49:17.483 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:49:17.483 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:49:17.484 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "throttled", 2024-03-11T21:49:17.484 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:47:19.371816+0000", 2024-03-11T21:49:17.484 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.5880000000000001e-06 2024-03-11T21:49:17.484 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:49:17.484 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:49:17.484 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "all_read", 2024-03-11T21:49:17.484 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:47:19.371821+0000", 2024-03-11T21:49:17.484 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 4.8940000000000001e-06 2024-03-11T21:49:17.484 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:49:17.484 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:49:17.484 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "dispatched", 2024-03-11T21:49:17.484 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:47:19.371834+0000", 2024-03-11T21:49:17.484 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.3360000000000001e-05 2024-03-11T21:49:17.484 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:49:17.485 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:49:17.485 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "queued_for_pg", 2024-03-11T21:49:17.485 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:47:19.371842+0000", 2024-03-11T21:49:17.485 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 7.4970000000000004e-06 2024-03-11T21:49:17.485 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:49:17.485 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:49:17.485 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "reached_pg", 2024-03-11T21:49:17.485 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:47:19.371904+0000", 2024-03-11T21:49:17.485 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 6.1849999999999999e-05 2024-03-11T21:49:17.485 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:49:17.485 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:49:17.485 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "waiting for peered", 2024-03-11T21:49:17.485 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:47:19.371916+0000", 2024-03-11T21:49:17.485 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.2278e-05 2024-03-11T21:49:17.485 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:49:17.486 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:49:17.486 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "reached_pg", 2024-03-11T21:49:17.486 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:47:20.296956+0000", 2024-03-11T21:49:17.486 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.92503979599999997 2024-03-11T21:49:17.486 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:49:17.486 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:49:17.486 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "started", 2024-03-11T21:49:17.486 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:47:20.297095+0000", 2024-03-11T21:49:17.486 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.00013955799999999999 2024-03-11T21:49:17.486 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:49:17.486 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:49:17.486 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "waiting for subops from 1,6", 2024-03-11T21:49:17.486 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:47:20.297213+0000", 2024-03-11T21:49:17.486 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.000117618 2024-03-11T21:49:17.486 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:49:17.487 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:49:17.487 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "op_commit", 2024-03-11T21:49:17.487 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:47:20.300507+0000", 2024-03-11T21:49:17.487 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.0032938849999999999 2024-03-11T21:49:17.487 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:49:17.487 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:49:17.487 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "sub_op_commit_rec", 2024-03-11T21:49:17.487 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:47:20.300756+0000", 2024-03-11T21:49:17.487 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.00024932399999999999 2024-03-11T21:49:17.487 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:49:17.487 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:49:17.487 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "sub_op_commit_rec", 2024-03-11T21:49:17.487 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:47:20.300817+0000", 2024-03-11T21:49:17.487 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 6.0885999999999999e-05 2024-03-11T21:49:17.487 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:49:17.488 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:49:17.488 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "commit_sent", 2024-03-11T21:49:17.488 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:47:20.300868+0000", 2024-03-11T21:49:17.488 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 5.1323000000000003e-05 2024-03-11T21:49:17.488 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:49:17.488 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:49:17.488 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "done", 2024-03-11T21:49:17.488 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:47:20.300914+0000", 2024-03-11T21:49:17.488 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 4.5306999999999998e-05 2024-03-11T21:49:17.488 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-03-11T21:49:17.488 INFO:teuthology.orchestra.run.smithi097.stdout: ] 2024-03-11T21:49:17.488 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-03-11T21:49:17.488 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:49:17.488 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:49:17.489 INFO:teuthology.orchestra.run.smithi097.stdout: "description": "osd_op(client.35813.0:194345 11.0 11:01d1e468:::benchmark_data_smithi092_186520_object55997:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e478)", 2024-03-11T21:49:17.489 INFO:teuthology.orchestra.run.smithi097.stdout: "initiated_at": "2024-03-11T21:47:19.387225+0000", 2024-03-11T21:49:17.489 INFO:teuthology.orchestra.run.smithi097.stdout: "age": 118.019508106, 2024-03-11T21:49:17.489 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.91392446599999999, 2024-03-11T21:49:17.489 INFO:teuthology.orchestra.run.smithi097.stdout: "type_data": { 2024-03-11T21:49:17.489 INFO:teuthology.orchestra.run.smithi097.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-11T21:49:17.489 INFO:teuthology.orchestra.run.smithi097.stdout: "client_info": { 2024-03-11T21:49:17.489 INFO:teuthology.orchestra.run.smithi097.stdout: "client": "client.35813", 2024-03-11T21:49:17.489 INFO:teuthology.orchestra.run.smithi097.stdout: "client_addr": "v1:172.21.15.92:0/4293352030", 2024-03-11T21:49:17.489 INFO:teuthology.orchestra.run.smithi097.stdout: "tid": 194345 2024-03-11T21:49:17.489 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:49:17.489 INFO:teuthology.orchestra.run.smithi097.stdout: "events": [ 2024-03-11T21:49:17.489 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:49:17.489 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "initiated", 2024-03-11T21:49:17.489 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:47:19.387225+0000", 2024-03-11T21:49:17.490 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-03-11T21:49:17.490 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:49:17.490 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:49:17.490 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "header_read", 2024-03-11T21:49:17.490 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:47:19.387225+0000", 2024-03-11T21:49:17.490 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-03-11T21:49:17.490 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:49:17.490 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:49:17.490 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "throttled", 2024-03-11T21:49:17.490 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:47:19.387226+0000", 2024-03-11T21:49:17.490 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.3999999999999999e-06 2024-03-11T21:49:17.490 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:49:17.490 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:49:17.490 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "all_read", 2024-03-11T21:49:17.491 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:47:19.387231+0000", 2024-03-11T21:49:17.491 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 4.2420000000000002e-06 2024-03-11T21:49:17.491 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:49:17.491 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:49:17.491 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "dispatched", 2024-03-11T21:49:17.491 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:47:19.387239+0000", 2024-03-11T21:49:17.491 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 8.2840000000000006e-06 2024-03-11T21:49:17.491 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:49:17.491 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:49:17.491 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "queued_for_pg", 2024-03-11T21:49:17.491 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:47:19.387245+0000", 2024-03-11T21:49:17.491 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 5.7570000000000001e-06 2024-03-11T21:49:17.491 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:49:17.491 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:49:17.491 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "reached_pg", 2024-03-11T21:49:17.492 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:47:19.387303+0000", 2024-03-11T21:49:17.492 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 5.8461999999999997e-05 2024-03-11T21:49:17.492 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:49:17.492 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:49:17.492 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "waiting for peered", 2024-03-11T21:49:17.492 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:47:19.387311+0000", 2024-03-11T21:49:17.492 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 7.7020000000000001e-06 2024-03-11T21:49:17.492 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:49:17.492 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:49:17.492 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "reached_pg", 2024-03-11T21:49:17.492 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:47:20.298598+0000", 2024-03-11T21:49:17.492 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.91128662000000005 2024-03-11T21:49:17.492 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:49:17.492 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:49:17.492 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "started", 2024-03-11T21:49:17.492 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:47:20.298721+0000", 2024-03-11T21:49:17.493 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.00012347599999999999 2024-03-11T21:49:17.493 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:49:17.493 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:49:17.493 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "waiting for subops from 1,6", 2024-03-11T21:49:17.493 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:47:20.298835+0000", 2024-03-11T21:49:17.493 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.00011415 2024-03-11T21:49:17.493 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:49:17.493 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:49:17.493 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "op_commit", 2024-03-11T21:49:17.493 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:47:20.300559+0000", 2024-03-11T21:49:17.493 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.0017239849999999999 2024-03-11T21:49:17.493 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:49:17.493 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:49:17.493 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "sub_op_commit_rec", 2024-03-11T21:49:17.494 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:47:20.300971+0000", 2024-03-11T21:49:17.494 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.000412273 2024-03-11T21:49:17.494 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:49:17.494 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:49:17.494 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "sub_op_commit_rec", 2024-03-11T21:49:17.494 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:47:20.301042+0000", 2024-03-11T21:49:17.494 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 7.0205999999999998e-05 2024-03-11T21:49:17.494 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:49:17.494 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:49:17.494 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "commit_sent", 2024-03-11T21:49:17.494 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:47:20.301103+0000", 2024-03-11T21:49:17.494 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 6.1290000000000004e-05 2024-03-11T21:49:17.494 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:49:17.494 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:49:17.495 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "done", 2024-03-11T21:49:17.495 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:47:20.301150+0000", 2024-03-11T21:49:17.495 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 4.6619000000000002e-05 2024-03-11T21:49:17.495 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-03-11T21:49:17.495 INFO:teuthology.orchestra.run.smithi097.stdout: ] 2024-03-11T21:49:17.495 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-03-11T21:49:17.495 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:49:17.495 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:49:17.495 INFO:teuthology.orchestra.run.smithi097.stdout: "description": "osd_op(client.35813.0:194363 11.0 11:08737563:::benchmark_data_smithi092_186520_object56015:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e478)", 2024-03-11T21:49:17.495 INFO:teuthology.orchestra.run.smithi097.stdout: "initiated_at": "2024-03-11T21:47:19.427430+0000", 2024-03-11T21:49:17.495 INFO:teuthology.orchestra.run.smithi097.stdout: "age": 117.97930342799999, 2024-03-11T21:49:17.495 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.88042194799999995, 2024-03-11T21:49:17.495 INFO:teuthology.orchestra.run.smithi097.stdout: "type_data": { 2024-03-11T21:49:17.496 INFO:teuthology.orchestra.run.smithi097.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-11T21:49:17.496 INFO:teuthology.orchestra.run.smithi097.stdout: "client_info": { 2024-03-11T21:49:17.496 INFO:teuthology.orchestra.run.smithi097.stdout: "client": "client.35813", 2024-03-11T21:49:17.496 INFO:teuthology.orchestra.run.smithi097.stdout: "client_addr": "v1:172.21.15.92:0/4293352030", 2024-03-11T21:49:17.496 INFO:teuthology.orchestra.run.smithi097.stdout: "tid": 194363 2024-03-11T21:49:17.496 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:49:17.496 INFO:teuthology.orchestra.run.smithi097.stdout: "events": [ 2024-03-11T21:49:17.496 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:49:17.496 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "initiated", 2024-03-11T21:49:17.496 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:47:19.427430+0000", 2024-03-11T21:49:17.496 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-03-11T21:49:17.496 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:49:17.496 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:49:17.496 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "header_read", 2024-03-11T21:49:17.497 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:47:19.427430+0000", 2024-03-11T21:49:17.497 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-03-11T21:49:17.497 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:49:17.497 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:49:17.497 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "throttled", 2024-03-11T21:49:17.497 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:47:19.427431+0000", 2024-03-11T21:49:17.497 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.37e-06 2024-03-11T21:49:17.497 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:49:17.497 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:49:17.497 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "all_read", 2024-03-11T21:49:17.497 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:47:19.427435+0000", 2024-03-11T21:49:17.497 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 4.2010000000000001e-06 2024-03-11T21:49:17.497 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:49:17.497 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:49:17.498 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "dispatched", 2024-03-11T21:49:17.498 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:47:19.427446+0000", 2024-03-11T21:49:17.498 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.0287e-05 2024-03-11T21:49:17.498 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:49:17.498 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:49:17.498 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "queued_for_pg", 2024-03-11T21:49:17.498 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:47:19.427452+0000", 2024-03-11T21:49:17.498 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 6.2160000000000003e-06 2024-03-11T21:49:17.498 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:49:17.498 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:49:17.498 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "reached_pg", 2024-03-11T21:49:17.498 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:47:19.427502+0000", 2024-03-11T21:49:17.498 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 4.9756000000000003e-05 2024-03-11T21:49:17.498 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:49:17.499 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:49:17.499 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "waiting for peered", 2024-03-11T21:49:17.499 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:47:19.427508+0000", 2024-03-11T21:49:17.499 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 6.5869999999999998e-06 2024-03-11T21:49:17.499 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:49:17.499 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:49:17.499 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "reached_pg", 2024-03-11T21:49:17.499 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:47:20.301197+0000", 2024-03-11T21:49:17.499 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.87368836999999999 2024-03-11T21:49:17.499 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:49:17.499 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:49:17.499 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "started", 2024-03-11T21:49:17.499 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:47:20.301305+0000", 2024-03-11T21:49:17.499 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.000108485 2024-03-11T21:49:17.499 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:49:17.500 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:49:17.500 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "waiting for subops from 1,6", 2024-03-11T21:49:17.500 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:47:20.302867+0000", 2024-03-11T21:49:17.500 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.0015616720000000001 2024-03-11T21:49:17.500 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:49:17.500 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:49:17.500 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "op_commit", 2024-03-11T21:49:17.500 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:47:20.306714+0000", 2024-03-11T21:49:17.500 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.0038477799999999999 2024-03-11T21:49:17.500 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:49:17.500 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:49:17.500 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "sub_op_commit_rec", 2024-03-11T21:49:17.500 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:47:20.307197+0000", 2024-03-11T21:49:17.500 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.00048263199999999998 2024-03-11T21:49:17.500 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:49:17.501 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:49:17.501 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "sub_op_commit_rec", 2024-03-11T21:49:17.501 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:47:20.307748+0000", 2024-03-11T21:49:17.501 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.00055130200000000004 2024-03-11T21:49:17.501 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:49:17.501 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:49:17.501 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "commit_sent", 2024-03-11T21:49:17.501 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:47:20.307805+0000", 2024-03-11T21:49:17.501 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 5.6768999999999998e-05 2024-03-11T21:49:17.501 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:49:17.501 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:49:17.501 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "done", 2024-03-11T21:49:17.501 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:47:20.307852+0000", 2024-03-11T21:49:17.501 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 4.6520999999999997e-05 2024-03-11T21:49:17.501 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-03-11T21:49:17.501 INFO:teuthology.orchestra.run.smithi097.stdout: ] 2024-03-11T21:49:17.502 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-03-11T21:49:17.502 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:49:17.502 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:49:17.502 INFO:teuthology.orchestra.run.smithi097.stdout: "description": "osd_op(client.35813.0:269290 11.4 11:21ba6696:::benchmark_data_smithi092_186520_object130942:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e498)", 2024-03-11T21:49:17.502 INFO:teuthology.orchestra.run.smithi097.stdout: "initiated_at": "2024-03-11T21:48:19.273002+0000", 2024-03-11T21:49:17.502 INFO:teuthology.orchestra.run.smithi097.stdout: "age": 58.133731505999997, 2024-03-11T21:49:17.502 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.12625803199999999, 2024-03-11T21:49:17.502 INFO:teuthology.orchestra.run.smithi097.stdout: "type_data": { 2024-03-11T21:49:17.502 INFO:teuthology.orchestra.run.smithi097.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-11T21:49:17.502 INFO:teuthology.orchestra.run.smithi097.stdout: "client_info": { 2024-03-11T21:49:17.502 INFO:teuthology.orchestra.run.smithi097.stdout: "client": "client.35813", 2024-03-11T21:49:17.502 INFO:teuthology.orchestra.run.smithi097.stdout: "client_addr": "v1:172.21.15.92:0/4293352030", 2024-03-11T21:49:17.502 INFO:teuthology.orchestra.run.smithi097.stdout: "tid": 269290 2024-03-11T21:49:17.502 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:49:17.503 INFO:teuthology.orchestra.run.smithi097.stdout: "events": [ 2024-03-11T21:49:17.503 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:49:17.503 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "initiated", 2024-03-11T21:49:17.503 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:48:19.273002+0000", 2024-03-11T21:49:17.503 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-03-11T21:49:17.503 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:49:17.503 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:49:17.503 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "header_read", 2024-03-11T21:49:17.503 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:48:19.273002+0000", 2024-03-11T21:49:17.503 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-03-11T21:49:17.503 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:49:17.503 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:49:17.503 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "throttled", 2024-03-11T21:49:17.503 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:48:19.273002+0000", 2024-03-11T21:49:17.503 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 6.37e-07 2024-03-11T21:49:17.503 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:49:17.504 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:49:17.504 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "all_read", 2024-03-11T21:49:17.504 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:48:19.273005+0000", 2024-03-11T21:49:17.504 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 3.0920000000000002e-06 2024-03-11T21:49:17.504 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:49:17.504 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:49:17.504 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "dispatched", 2024-03-11T21:49:17.504 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:48:19.273011+0000", 2024-03-11T21:49:17.504 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 5.57e-06 2024-03-11T21:49:17.504 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:49:17.504 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:49:17.504 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "queued_for_pg", 2024-03-11T21:49:17.504 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:48:19.273017+0000", 2024-03-11T21:49:17.504 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 5.8340000000000004e-06 2024-03-11T21:49:17.505 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:49:17.505 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:49:17.505 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "reached_pg", 2024-03-11T21:49:17.505 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:48:19.273065+0000", 2024-03-11T21:49:17.505 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 4.8503e-05 2024-03-11T21:49:17.505 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:49:17.505 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:49:17.505 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "started", 2024-03-11T21:49:17.505 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:48:19.273190+0000", 2024-03-11T21:49:17.505 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.00012427200000000001 2024-03-11T21:49:17.505 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:49:17.505 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:49:17.505 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "waiting for subops from 2,6", 2024-03-11T21:49:17.505 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:48:19.273303+0000", 2024-03-11T21:49:17.505 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.000113276 2024-03-11T21:49:17.506 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:49:17.506 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:49:17.506 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "op_commit", 2024-03-11T21:49:17.506 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:48:19.275041+0000", 2024-03-11T21:49:17.506 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.001737965 2024-03-11T21:49:17.506 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:49:17.506 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:49:17.506 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "sub_op_commit_rec", 2024-03-11T21:49:17.506 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:48:19.275421+0000", 2024-03-11T21:49:17.506 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.000380171 2024-03-11T21:49:17.506 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:49:17.506 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:49:17.506 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "sub_op_commit_rec", 2024-03-11T21:49:17.506 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:48:19.399181+0000", 2024-03-11T21:49:17.506 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.123760233 2024-03-11T21:49:17.507 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:49:17.507 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:49:17.507 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "commit_sent", 2024-03-11T21:49:17.507 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:48:19.399227+0000", 2024-03-11T21:49:17.507 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 4.5389000000000002e-05 2024-03-11T21:49:17.507 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:49:17.507 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:49:17.507 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "done", 2024-03-11T21:49:17.507 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:48:19.399260+0000", 2024-03-11T21:49:17.507 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 3.3090000000000003e-05 2024-03-11T21:49:17.507 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-03-11T21:49:17.507 INFO:teuthology.orchestra.run.smithi097.stdout: ] 2024-03-11T21:49:17.507 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-03-11T21:49:17.507 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:49:17.508 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:49:17.508 INFO:teuthology.orchestra.run.smithi097.stdout: "description": "osd_op(client.35813.0:269300 11.0 11:0b450850:::benchmark_data_smithi092_186520_object130952:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e498)", 2024-03-11T21:49:17.508 INFO:teuthology.orchestra.run.smithi097.stdout: "initiated_at": "2024-03-11T21:48:19.277123+0000", 2024-03-11T21:49:17.508 INFO:teuthology.orchestra.run.smithi097.stdout: "age": 58.129610212000003, 2024-03-11T21:49:17.508 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.12958527, 2024-03-11T21:49:17.508 INFO:teuthology.orchestra.run.smithi097.stdout: "type_data": { 2024-03-11T21:49:17.508 INFO:teuthology.orchestra.run.smithi097.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-11T21:49:17.508 INFO:teuthology.orchestra.run.smithi097.stdout: "client_info": { 2024-03-11T21:49:17.508 INFO:teuthology.orchestra.run.smithi097.stdout: "client": "client.35813", 2024-03-11T21:49:17.508 INFO:teuthology.orchestra.run.smithi097.stdout: "client_addr": "v1:172.21.15.92:0/4293352030", 2024-03-11T21:49:17.508 INFO:teuthology.orchestra.run.smithi097.stdout: "tid": 269300 2024-03-11T21:49:17.508 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:49:17.508 INFO:teuthology.orchestra.run.smithi097.stdout: "events": [ 2024-03-11T21:49:17.508 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:49:17.509 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "initiated", 2024-03-11T21:49:17.509 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:48:19.277123+0000", 2024-03-11T21:49:17.509 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-03-11T21:49:17.509 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:49:17.509 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:49:17.509 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "header_read", 2024-03-11T21:49:17.509 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:48:19.277123+0000", 2024-03-11T21:49:17.509 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-03-11T21:49:17.509 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:49:17.509 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:49:17.509 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "throttled", 2024-03-11T21:49:17.509 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:48:19.277124+0000", 2024-03-11T21:49:17.509 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 1.26e-06 2024-03-11T21:49:17.509 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:49:17.509 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:49:17.510 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "all_read", 2024-03-11T21:49:17.510 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:48:19.277128+0000", 2024-03-11T21:49:17.510 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 4.143e-06 2024-03-11T21:49:17.510 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:49:17.510 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:49:17.510 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "dispatched", 2024-03-11T21:49:17.510 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:48:19.277137+0000", 2024-03-11T21:49:17.510 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 8.9290000000000004e-06 2024-03-11T21:49:17.510 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:49:17.510 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:49:17.510 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "queued_for_pg", 2024-03-11T21:49:17.510 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:48:19.277143+0000", 2024-03-11T21:49:17.510 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 5.5690000000000004e-06 2024-03-11T21:49:17.510 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:49:17.511 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:49:17.511 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "reached_pg", 2024-03-11T21:49:17.511 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:48:19.277181+0000", 2024-03-11T21:49:17.511 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 3.7957999999999998e-05 2024-03-11T21:49:17.511 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:49:17.511 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:49:17.511 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "started", 2024-03-11T21:49:17.511 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:48:19.277258+0000", 2024-03-11T21:49:17.511 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 7.7390000000000003e-05 2024-03-11T21:49:17.511 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:49:17.511 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:49:17.511 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "waiting for subops from 1,6", 2024-03-11T21:49:17.511 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:48:19.277325+0000", 2024-03-11T21:49:17.511 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 6.7100999999999993e-05 2024-03-11T21:49:17.511 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:49:17.512 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:49:17.512 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "op_commit", 2024-03-11T21:49:17.512 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:48:19.278815+0000", 2024-03-11T21:49:17.512 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.001489764 2024-03-11T21:49:17.512 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:49:17.512 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:49:17.512 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "sub_op_commit_rec", 2024-03-11T21:49:17.512 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:48:19.279036+0000", 2024-03-11T21:49:17.512 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.000220929 2024-03-11T21:49:17.512 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:49:17.512 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:49:17.512 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "sub_op_commit_rec", 2024-03-11T21:49:17.512 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:48:19.406619+0000", 2024-03-11T21:49:17.512 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.12758270799999999 2024-03-11T21:49:17.513 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:49:17.513 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:49:17.513 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "commit_sent", 2024-03-11T21:49:17.513 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:48:19.406671+0000", 2024-03-11T21:49:17.513 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 5.1934000000000001e-05 2024-03-11T21:49:17.513 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:49:17.513 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:49:17.513 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "done", 2024-03-11T21:49:17.513 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:48:19.406708+0000", 2024-03-11T21:49:17.513 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 3.7585e-05 2024-03-11T21:49:17.513 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-03-11T21:49:17.513 INFO:teuthology.orchestra.run.smithi097.stdout: ] 2024-03-11T21:49:17.513 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-03-11T21:49:17.513 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:49:17.513 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:49:17.514 INFO:teuthology.orchestra.run.smithi097.stdout: "description": "osd_op(client.35813.0:269568 11.0 11:0eecae29:::benchmark_data_smithi092_186520_object131220:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e498)", 2024-03-11T21:49:17.514 INFO:teuthology.orchestra.run.smithi097.stdout: "initiated_at": "2024-03-11T21:48:19.607460+0000", 2024-03-11T21:49:17.514 INFO:teuthology.orchestra.run.smithi097.stdout: "age": 57.799273409999998, 2024-03-11T21:49:17.514 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.127574666, 2024-03-11T21:49:17.514 INFO:teuthology.orchestra.run.smithi097.stdout: "type_data": { 2024-03-11T21:49:17.514 INFO:teuthology.orchestra.run.smithi097.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-11T21:49:17.514 INFO:teuthology.orchestra.run.smithi097.stdout: "client_info": { 2024-03-11T21:49:17.514 INFO:teuthology.orchestra.run.smithi097.stdout: "client": "client.35813", 2024-03-11T21:49:17.514 INFO:teuthology.orchestra.run.smithi097.stdout: "client_addr": "v1:172.21.15.92:0/4293352030", 2024-03-11T21:49:17.514 INFO:teuthology.orchestra.run.smithi097.stdout: "tid": 269568 2024-03-11T21:49:17.514 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:49:17.514 INFO:teuthology.orchestra.run.smithi097.stdout: "events": [ 2024-03-11T21:49:17.514 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:49:17.514 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "initiated", 2024-03-11T21:49:17.515 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:48:19.607460+0000", 2024-03-11T21:49:17.515 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-03-11T21:49:17.515 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:49:17.515 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:49:17.515 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "header_read", 2024-03-11T21:49:17.515 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:48:19.607460+0000", 2024-03-11T21:49:17.515 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-03-11T21:49:17.515 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:49:17.515 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:49:17.515 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "throttled", 2024-03-11T21:49:17.515 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:48:19.607461+0000", 2024-03-11T21:49:17.515 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 9.0800000000000003e-07 2024-03-11T21:49:17.515 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:49:17.515 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:49:17.515 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "all_read", 2024-03-11T21:49:17.516 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:48:19.607464+0000", 2024-03-11T21:49:17.516 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 3.1889999999999999e-06 2024-03-11T21:49:17.516 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:49:17.516 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:49:17.516 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "dispatched", 2024-03-11T21:49:17.516 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:48:19.607474+0000", 2024-03-11T21:49:17.516 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 9.7489999999999993e-06 2024-03-11T21:49:17.516 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:49:17.516 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:49:17.516 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "queued_for_pg", 2024-03-11T21:49:17.516 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:48:19.607481+0000", 2024-03-11T21:49:17.516 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 7.2919999999999998e-06 2024-03-11T21:49:17.516 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:49:17.516 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:49:17.516 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "reached_pg", 2024-03-11T21:49:17.517 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:48:19.607539+0000", 2024-03-11T21:49:17.517 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 5.7887000000000003e-05 2024-03-11T21:49:17.517 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:49:17.517 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:49:17.517 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "started", 2024-03-11T21:49:17.517 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:48:19.607660+0000", 2024-03-11T21:49:17.517 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.000121388 2024-03-11T21:49:17.517 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:49:17.517 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:49:17.517 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "waiting for subops from 1,6", 2024-03-11T21:49:17.517 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:48:19.607771+0000", 2024-03-11T21:49:17.517 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.00011103 2024-03-11T21:49:17.517 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:49:17.517 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:49:17.517 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "op_commit", 2024-03-11T21:49:17.517 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:48:19.610770+0000", 2024-03-11T21:49:17.518 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.0029985099999999998 2024-03-11T21:49:17.518 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:49:17.518 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:49:17.518 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "sub_op_commit_rec", 2024-03-11T21:49:17.518 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:48:19.610802+0000", 2024-03-11T21:49:17.518 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 3.2014000000000002e-05 2024-03-11T21:49:17.518 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:49:17.518 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:49:17.518 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "sub_op_commit_rec", 2024-03-11T21:49:17.518 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:48:19.734930+0000", 2024-03-11T21:49:17.518 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.124127879 2024-03-11T21:49:17.518 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:49:17.518 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:49:17.518 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "commit_sent", 2024-03-11T21:49:17.519 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:48:19.734983+0000", 2024-03-11T21:49:17.519 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 5.3220999999999997e-05 2024-03-11T21:49:17.519 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:49:17.519 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:49:17.519 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "done", 2024-03-11T21:49:17.519 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:48:19.735034+0000", 2024-03-11T21:49:17.519 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 5.1598999999999998e-05 2024-03-11T21:49:17.519 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-03-11T21:49:17.519 INFO:teuthology.orchestra.run.smithi097.stdout: ] 2024-03-11T21:49:17.519 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-03-11T21:49:17.519 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:49:17.519 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:49:17.519 INFO:teuthology.orchestra.run.smithi097.stdout: "description": "osd_op(client.35813.0:269569 11.0 11:0b798a8c:::benchmark_data_smithi092_186520_object131221:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e498)", 2024-03-11T21:49:17.519 INFO:teuthology.orchestra.run.smithi097.stdout: "initiated_at": "2024-03-11T21:48:19.607783+0000", 2024-03-11T21:49:17.519 INFO:teuthology.orchestra.run.smithi097.stdout: "age": 57.798950464999997, 2024-03-11T21:49:17.520 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.13074226899999999, 2024-03-11T21:49:17.520 INFO:teuthology.orchestra.run.smithi097.stdout: "type_data": { 2024-03-11T21:49:17.520 INFO:teuthology.orchestra.run.smithi097.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-11T21:49:17.520 INFO:teuthology.orchestra.run.smithi097.stdout: "client_info": { 2024-03-11T21:49:17.520 INFO:teuthology.orchestra.run.smithi097.stdout: "client": "client.35813", 2024-03-11T21:49:17.520 INFO:teuthology.orchestra.run.smithi097.stdout: "client_addr": "v1:172.21.15.92:0/4293352030", 2024-03-11T21:49:17.520 INFO:teuthology.orchestra.run.smithi097.stdout: "tid": 269569 2024-03-11T21:49:17.520 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:49:17.520 INFO:teuthology.orchestra.run.smithi097.stdout: "events": [ 2024-03-11T21:49:17.520 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:49:17.520 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "initiated", 2024-03-11T21:49:17.520 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:48:19.607783+0000", 2024-03-11T21:49:17.520 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-03-11T21:49:17.520 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:49:17.521 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:49:17.521 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "header_read", 2024-03-11T21:49:17.521 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:48:19.607783+0000", 2024-03-11T21:49:17.521 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0 2024-03-11T21:49:17.521 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:49:17.521 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:49:17.521 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "throttled", 2024-03-11T21:49:17.521 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:48:19.607783+0000", 2024-03-11T21:49:17.521 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 7.8800000000000002e-07 2024-03-11T21:49:17.521 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:49:17.521 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:49:17.521 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "all_read", 2024-03-11T21:49:17.521 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:48:19.607787+0000", 2024-03-11T21:49:17.521 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 3.1200000000000002e-06 2024-03-11T21:49:17.522 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:49:17.522 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:49:17.522 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "dispatched", 2024-03-11T21:49:17.522 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:48:19.607792+0000", 2024-03-11T21:49:17.522 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 5.7219999999999996e-06 2024-03-11T21:49:17.522 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:49:17.522 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:49:17.522 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "queued_for_pg", 2024-03-11T21:49:17.522 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:48:19.607797+0000", 2024-03-11T21:49:17.522 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 4.3499999999999999e-06 2024-03-11T21:49:17.522 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:49:17.522 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:49:17.522 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "reached_pg", 2024-03-11T21:49:17.522 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:48:19.609515+0000", 2024-03-11T21:49:17.523 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.001718488 2024-03-11T21:49:17.523 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:49:17.523 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:49:17.523 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "started", 2024-03-11T21:49:17.523 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:48:19.609644+0000", 2024-03-11T21:49:17.523 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.000129252 2024-03-11T21:49:17.523 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:49:17.523 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:49:17.523 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "waiting for subops from 1,6", 2024-03-11T21:49:17.523 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:48:19.609718+0000", 2024-03-11T21:49:17.523 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 7.3348000000000003e-05 2024-03-11T21:49:17.523 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:49:17.524 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:49:17.524 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "op_commit", 2024-03-11T21:49:17.524 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:48:19.610833+0000", 2024-03-11T21:49:17.524 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.0011153339999999999 2024-03-11T21:49:17.524 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:49:17.524 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:49:17.524 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "sub_op_commit_rec", 2024-03-11T21:49:17.524 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:48:19.611313+0000", 2024-03-11T21:49:17.524 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.00047941800000000001 2024-03-11T21:49:17.524 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:49:17.524 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:49:17.524 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "sub_op_commit_rec", 2024-03-11T21:49:17.524 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:48:19.738452+0000", 2024-03-11T21:49:17.525 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 0.127139797 2024-03-11T21:49:17.525 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:49:17.525 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:49:17.525 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "commit_sent", 2024-03-11T21:49:17.525 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:48:19.738486+0000", 2024-03-11T21:49:17.525 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 3.3226999999999999e-05 2024-03-11T21:49:17.525 INFO:teuthology.orchestra.run.smithi097.stdout: }, 2024-03-11T21:49:17.525 INFO:teuthology.orchestra.run.smithi097.stdout: { 2024-03-11T21:49:17.525 INFO:teuthology.orchestra.run.smithi097.stdout: "event": "done", 2024-03-11T21:49:17.525 INFO:teuthology.orchestra.run.smithi097.stdout: "time": "2024-03-11T21:48:19.738525+0000", 2024-03-11T21:49:17.525 INFO:teuthology.orchestra.run.smithi097.stdout: "duration": 3.9425000000000003e-05 2024-03-11T21:49:17.525 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-03-11T21:49:17.525 INFO:teuthology.orchestra.run.smithi097.stdout: ] 2024-03-11T21:49:17.525 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-03-11T21:49:17.525 INFO:teuthology.orchestra.run.smithi097.stdout: } 2024-03-11T21:49:17.525 INFO:teuthology.orchestra.run.smithi097.stdout: ] 2024-03-11T21:49:17.526 INFO:teuthology.orchestra.run.smithi097.stdout:} 2024-03-11T21:49:18.229 DEBUG:teuthology.orchestra.run.smithi092:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid 00db0016-dfed-11ee-95c8-87774f69a715 -- ceph daemon osd.3 dump_ops_in_flight 2024-03-11T21:49:18.243 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:49:18 smithi097 ceph-mon[33317]: osdmap e529: 8 total, 8 up, 7 in 2024-03-11T21:49:18.256 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:49:18 smithi092 ceph-mon[29572]: osdmap e529: 8 total, 8 up, 7 in 2024-03-11T21:49:18.257 INFO:journalctl@ceph.mgr.y.smithi092.stdout:Mar 11 21:49:17 smithi092 ceph-00db0016-dfed-11ee-95c8-87774f69a715-mgr-y[29801]: ::ffff:172.21.15.97 - - [11/Mar/2024:21:49:17] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-11T21:49:18.257 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:49:18 smithi092 ceph-mon[37563]: osdmap e529: 8 total, 8 up, 7 in 2024-03-11T21:49:18.475 INFO:teuthology.orchestra.run.smithi092.stderr:Inferring config /var/lib/ceph/00db0016-dfed-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T21:49:19.369 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:49:19 smithi092 ceph-mon[29572]: pgmap v1133: 132 pgs: 1 active+recovering, 131 active+clean; 582 KiB data, 5.3 GiB used, 531 GiB / 536 GiB avail; 1023 B/s rd, 0 op/s; 1/594 objects misplaced (0.168%); 30 B/s, 0 objects/s recovering 2024-03-11T21:49:19.369 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:49:19 smithi092 ceph-mon[29572]: 2.0 scrub starts 2024-03-11T21:49:19.369 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:49:19 smithi092 ceph-mon[29572]: 2.0 scrub ok 2024-03-11T21:49:19.369 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:49:19 smithi092 ceph-mon[29572]: 6.4 deep-scrub starts 2024-03-11T21:49:19.370 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:49:19 smithi092 ceph-mon[29572]: 6.4 deep-scrub ok 2024-03-11T21:49:19.370 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:49:19 smithi092 ceph-mon[29572]: 2.1f scrub starts 2024-03-11T21:49:19.370 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:49:19 smithi092 ceph-mon[29572]: 2.1f scrub ok 2024-03-11T21:49:19.370 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:49:19 smithi092 ceph-mon[29572]: osdmap e530: 8 total, 8 up, 7 in 2024-03-11T21:49:19.370 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:49:19 smithi092 ceph-mon[37563]: pgmap v1133: 132 pgs: 1 active+recovering, 131 active+clean; 582 KiB data, 5.3 GiB used, 531 GiB / 536 GiB avail; 1023 B/s rd, 0 op/s; 1/594 objects misplaced (0.168%); 30 B/s, 0 objects/s recovering 2024-03-11T21:49:19.370 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:49:19 smithi092 ceph-mon[37563]: 2.0 scrub starts 2024-03-11T21:49:19.370 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:49:19 smithi092 ceph-mon[37563]: 2.0 scrub ok 2024-03-11T21:49:19.370 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:49:19 smithi092 ceph-mon[37563]: 6.4 deep-scrub starts 2024-03-11T21:49:19.370 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:49:19 smithi092 ceph-mon[37563]: 6.4 deep-scrub ok 2024-03-11T21:49:19.370 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:49:19 smithi092 ceph-mon[37563]: 2.1f scrub starts 2024-03-11T21:49:19.370 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:49:19 smithi092 ceph-mon[37563]: 2.1f scrub ok 2024-03-11T21:49:19.371 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:49:19 smithi092 ceph-mon[37563]: osdmap e530: 8 total, 8 up, 7 in 2024-03-11T21:49:19.441 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:49:19 smithi097 ceph-mon[33317]: pgmap v1133: 132 pgs: 1 active+recovering, 131 active+clean; 582 KiB data, 5.3 GiB used, 531 GiB / 536 GiB avail; 1023 B/s rd, 0 op/s; 1/594 objects misplaced (0.168%); 30 B/s, 0 objects/s recovering 2024-03-11T21:49:19.441 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:49:19 smithi097 ceph-mon[33317]: 2.0 scrub starts 2024-03-11T21:49:19.441 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:49:19 smithi097 ceph-mon[33317]: 2.0 scrub ok 2024-03-11T21:49:19.441 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:49:19 smithi097 ceph-mon[33317]: 6.4 deep-scrub starts 2024-03-11T21:49:19.441 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:49:19 smithi097 ceph-mon[33317]: 6.4 deep-scrub ok 2024-03-11T21:49:19.441 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:49:19 smithi097 ceph-mon[33317]: 2.1f scrub starts 2024-03-11T21:49:19.441 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:49:19 smithi097 ceph-mon[33317]: 2.1f scrub ok 2024-03-11T21:49:19.441 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:49:19 smithi097 ceph-mon[33317]: osdmap e530: 8 total, 8 up, 7 in 2024-03-11T21:49:20.354 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:49:20 smithi092 ceph-mon[29572]: 5.15 scrub starts 2024-03-11T21:49:20.354 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:49:20 smithi092 ceph-mon[29572]: 5.15 scrub ok 2024-03-11T21:49:20.354 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:49:20 smithi092 ceph-mon[29572]: 4.1a scrub starts 2024-03-11T21:49:20.354 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:49:20 smithi092 ceph-mon[29572]: 4.1a scrub ok 2024-03-11T21:49:20.355 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:49:20 smithi092 ceph-mon[37563]: 5.15 scrub starts 2024-03-11T21:49:20.355 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:49:20 smithi092 ceph-mon[37563]: 5.15 scrub ok 2024-03-11T21:49:20.355 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:49:20 smithi092 ceph-mon[37563]: 4.1a scrub starts 2024-03-11T21:49:20.355 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:49:20 smithi092 ceph-mon[37563]: 4.1a scrub ok 2024-03-11T21:49:20.477 INFO:teuthology.orchestra.run.smithi092.stdout:{ 2024-03-11T21:49:20.477 INFO:teuthology.orchestra.run.smithi092.stdout: "ops": [], 2024-03-11T21:49:20.477 INFO:teuthology.orchestra.run.smithi092.stdout: "num_ops": 0 2024-03-11T21:49:20.477 INFO:teuthology.orchestra.run.smithi092.stdout:} 2024-03-11T21:49:20.585 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:49:20 smithi097 ceph-mon[33317]: 5.15 scrub starts 2024-03-11T21:49:20.586 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:49:20 smithi097 ceph-mon[33317]: 5.15 scrub ok 2024-03-11T21:49:20.586 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:49:20 smithi097 ceph-mon[33317]: 4.1a scrub starts 2024-03-11T21:49:20.586 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:49:20 smithi097 ceph-mon[33317]: 4.1a scrub ok 2024-03-11T21:49:21.227 DEBUG:teuthology.orchestra.run.smithi092:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid 00db0016-dfed-11ee-95c8-87774f69a715 -- ceph daemon osd.3 dump_blocked_ops 2024-03-11T21:49:21.509 INFO:teuthology.orchestra.run.smithi092.stderr:Inferring config /var/lib/ceph/00db0016-dfed-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T21:49:21.585 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:49:21 smithi097 ceph-mon[33317]: pgmap v1135: 132 pgs: 1 active+recovering, 131 active+clean; 582 KiB data, 5.3 GiB used, 531 GiB / 536 GiB avail; 1.1 KiB/s rd, 1 op/s; 1/594 objects misplaced (0.168%); 267 B/s, 9 objects/s recovering 2024-03-11T21:49:21.585 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:49:21 smithi097 ceph-mon[33317]: 2.9 scrub starts 2024-03-11T21:49:21.585 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:49:21 smithi097 ceph-mon[33317]: 2.9 scrub ok 2024-03-11T21:49:21.619 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:49:21 smithi092 ceph-mon[29572]: pgmap v1135: 132 pgs: 1 active+recovering, 131 active+clean; 582 KiB data, 5.3 GiB used, 531 GiB / 536 GiB avail; 1.1 KiB/s rd, 1 op/s; 1/594 objects misplaced (0.168%); 267 B/s, 9 objects/s recovering 2024-03-11T21:49:21.620 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:49:21 smithi092 ceph-mon[29572]: 2.9 scrub starts 2024-03-11T21:49:21.620 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:49:21 smithi092 ceph-mon[29572]: 2.9 scrub ok 2024-03-11T21:49:21.620 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:49:21 smithi092 ceph-mon[37563]: pgmap v1135: 132 pgs: 1 active+recovering, 131 active+clean; 582 KiB data, 5.3 GiB used, 531 GiB / 536 GiB avail; 1.1 KiB/s rd, 1 op/s; 1/594 objects misplaced (0.168%); 267 B/s, 9 objects/s recovering 2024-03-11T21:49:21.620 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:49:21 smithi092 ceph-mon[37563]: 2.9 scrub starts 2024-03-11T21:49:21.620 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:49:21 smithi092 ceph-mon[37563]: 2.9 scrub ok 2024-03-11T21:49:21.972 INFO:tasks.thrashosds.thrasher:inning osd 2024-03-11T21:49:21.972 INFO:tasks.thrashosds.thrasher:Adding osd 4 2024-03-11T21:49:21.972 DEBUG:teuthology.orchestra.run.smithi092:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid 00db0016-dfed-11ee-95c8-87774f69a715 -- ceph osd in 4 2024-03-11T21:49:22.682 INFO:teuthology.orchestra.run.smithi092.stderr:Inferring config /var/lib/ceph/00db0016-dfed-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T21:49:23.369 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:49:23 smithi092 ceph-mon[37563]: pgmap v1136: 132 pgs: 1 active+recovering, 131 active+clean; 582 KiB data, 5.7 GiB used, 620 GiB / 626 GiB avail; 682 B/s rd, 0 op/s; 1/594 objects misplaced (0.168%); 326 B/s, 13 objects/s recovering 2024-03-11T21:49:23.370 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:49:23 smithi092 ceph-mon[37563]: 6.18 scrub starts 2024-03-11T21:49:23.370 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:49:23 smithi092 ceph-mon[37563]: 6.18 scrub ok 2024-03-11T21:49:23.370 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:49:23 smithi092 ceph-mon[29572]: pgmap v1136: 132 pgs: 1 active+recovering, 131 active+clean; 582 KiB data, 5.7 GiB used, 620 GiB / 626 GiB avail; 682 B/s rd, 0 op/s; 1/594 objects misplaced (0.168%); 326 B/s, 13 objects/s recovering 2024-03-11T21:49:23.370 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:49:23 smithi092 ceph-mon[29572]: 6.18 scrub starts 2024-03-11T21:49:23.370 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:49:23 smithi092 ceph-mon[29572]: 6.18 scrub ok 2024-03-11T21:49:23.388 INFO:journalctl@ceph.iscsi.iscsi.a.smithi097.stdout:Mar 11 21:49:23 smithi097 ceph-00db0016-dfed-11ee-95c8-87774f69a715-iscsi-iscsi-a[60943]: debug there is no tcmu-runner data available 2024-03-11T21:49:23.388 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:49:23 smithi097 ceph-mon[33317]: pgmap v1136: 132 pgs: 1 active+recovering, 131 active+clean; 582 KiB data, 5.7 GiB used, 620 GiB / 626 GiB avail; 682 B/s rd, 0 op/s; 1/594 objects misplaced (0.168%); 326 B/s, 13 objects/s recovering 2024-03-11T21:49:23.388 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:49:23 smithi097 ceph-mon[33317]: 6.18 scrub starts 2024-03-11T21:49:23.388 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:49:23 smithi097 ceph-mon[33317]: 6.18 scrub ok 2024-03-11T21:49:23.873 INFO:teuthology.orchestra.run.smithi092.stdout:{ 2024-03-11T21:49:23.873 INFO:teuthology.orchestra.run.smithi092.stdout: "ops": [], 2024-03-11T21:49:23.873 INFO:teuthology.orchestra.run.smithi092.stdout: "complaint_time": 30, 2024-03-11T21:49:23.874 INFO:teuthology.orchestra.run.smithi092.stdout: "num_blocked_ops": 0 2024-03-11T21:49:23.874 INFO:teuthology.orchestra.run.smithi092.stdout:} 2024-03-11T21:49:24.414 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:49:24 smithi092 ceph-mon[29572]: 2.17 scrub starts 2024-03-11T21:49:24.414 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:49:24 smithi092 ceph-mon[29572]: 2.17 scrub ok 2024-03-11T21:49:24.415 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:49:24 smithi092 ceph-mon[29572]: from='client.24469 v1:172.21.15.97:0/404208310' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:49:24.415 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:49:24 smithi092 ceph-mon[29572]: from='mgr.14607 v1:172.21.15.92:0/1702717597' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T21:49:24.415 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:49:24 smithi092 ceph-mon[29572]: from='mgr.14607 v1:172.21.15.92:0/1702717597' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.11", "id": [3, 5]}]: dispatch 2024-03-11T21:49:24.415 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:49:24 smithi092 ceph-mon[29572]: from='mgr.14607 v1:172.21.15.92:0/1702717597' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.15", "id": [3, 2]}]: dispatch 2024-03-11T21:49:24.415 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:49:24 smithi092 ceph-mon[29572]: from='mgr.14607 ' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.11", "id": [3, 5]}]: dispatch 2024-03-11T21:49:24.415 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:49:24 smithi092 ceph-mon[29572]: from='mgr.14607 v1:172.21.15.92:0/1702717597' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "5.1a", "id": [0, 3]}]: dispatch 2024-03-11T21:49:24.415 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:49:24 smithi092 ceph-mon[29572]: from='mgr.14607 ' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.15", "id": [3, 2]}]: dispatch 2024-03-11T21:49:24.415 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:49:24 smithi092 ceph-mon[29572]: from='mgr.14607 ' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "5.1a", "id": [0, 3]}]: dispatch 2024-03-11T21:49:24.415 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:49:24 smithi092 ceph-mon[37563]: 2.17 scrub starts 2024-03-11T21:49:24.415 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:49:24 smithi092 ceph-mon[37563]: 2.17 scrub ok 2024-03-11T21:49:24.415 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:49:24 smithi092 ceph-mon[37563]: from='client.24469 v1:172.21.15.97:0/404208310' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:49:24.416 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:49:24 smithi092 ceph-mon[37563]: from='mgr.14607 v1:172.21.15.92:0/1702717597' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T21:49:24.416 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:49:24 smithi092 ceph-mon[37563]: from='mgr.14607 v1:172.21.15.92:0/1702717597' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.11", "id": [3, 5]}]: dispatch 2024-03-11T21:49:24.416 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:49:24 smithi092 ceph-mon[37563]: from='mgr.14607 v1:172.21.15.92:0/1702717597' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.15", "id": [3, 2]}]: dispatch 2024-03-11T21:49:24.416 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:49:24 smithi092 ceph-mon[37563]: from='mgr.14607 ' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.11", "id": [3, 5]}]: dispatch 2024-03-11T21:49:24.416 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:49:24 smithi092 ceph-mon[37563]: from='mgr.14607 v1:172.21.15.92:0/1702717597' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "5.1a", "id": [0, 3]}]: dispatch 2024-03-11T21:49:24.416 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:49:24 smithi092 ceph-mon[37563]: from='mgr.14607 ' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.15", "id": [3, 2]}]: dispatch 2024-03-11T21:49:24.416 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:49:24 smithi092 ceph-mon[37563]: from='mgr.14607 ' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "5.1a", "id": [0, 3]}]: dispatch 2024-03-11T21:49:24.586 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:49:24 smithi097 ceph-mon[33317]: 2.17 scrub starts 2024-03-11T21:49:24.586 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:49:24 smithi097 ceph-mon[33317]: 2.17 scrub ok 2024-03-11T21:49:24.586 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:49:24 smithi097 ceph-mon[33317]: from='client.24469 v1:172.21.15.97:0/404208310' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:49:24.586 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:49:24 smithi097 ceph-mon[33317]: from='mgr.14607 v1:172.21.15.92:0/1702717597' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T21:49:24.586 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:49:24 smithi097 ceph-mon[33317]: from='mgr.14607 v1:172.21.15.92:0/1702717597' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.11", "id": [3, 5]}]: dispatch 2024-03-11T21:49:24.586 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:49:24 smithi097 ceph-mon[33317]: from='mgr.14607 v1:172.21.15.92:0/1702717597' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.15", "id": [3, 2]}]: dispatch 2024-03-11T21:49:24.586 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:49:24 smithi097 ceph-mon[33317]: from='mgr.14607 ' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.11", "id": [3, 5]}]: dispatch 2024-03-11T21:49:24.586 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:49:24 smithi097 ceph-mon[33317]: from='mgr.14607 v1:172.21.15.92:0/1702717597' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "5.1a", "id": [0, 3]}]: dispatch 2024-03-11T21:49:24.587 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:49:24 smithi097 ceph-mon[33317]: from='mgr.14607 ' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.15", "id": [3, 2]}]: dispatch 2024-03-11T21:49:24.587 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:49:24 smithi097 ceph-mon[33317]: from='mgr.14607 ' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "5.1a", "id": [0, 3]}]: dispatch 2024-03-11T21:49:25.468 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:49:25 smithi092 ceph-mon[37563]: pgmap v1137: 132 pgs: 132 active+clean; 582 KiB data, 5.7 GiB used, 620 GiB / 626 GiB avail; 551 B/s rd, 0 op/s; 263 B/s, 10 objects/s recovering 2024-03-11T21:49:25.468 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:49:25 smithi092 ceph-mon[37563]: 2.6 scrub starts 2024-03-11T21:49:25.468 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:49:25 smithi092 ceph-mon[37563]: 2.6 scrub ok 2024-03-11T21:49:25.468 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:49:25 smithi092 ceph-mon[37563]: from='mgr.14607 ' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.11", "id": [3, 5]}]': finished 2024-03-11T21:49:25.468 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:49:25 smithi092 ceph-mon[37563]: from='mgr.14607 ' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.15", "id": [3, 2]}]': finished 2024-03-11T21:49:25.468 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:49:25 smithi092 ceph-mon[37563]: from='mgr.14607 ' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "5.1a", "id": [0, 3]}]': finished 2024-03-11T21:49:25.468 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:49:25 smithi092 ceph-mon[37563]: osdmap e531: 8 total, 8 up, 7 in 2024-03-11T21:49:25.468 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:49:25 smithi092 ceph-mon[29572]: pgmap v1137: 132 pgs: 132 active+clean; 582 KiB data, 5.7 GiB used, 620 GiB / 626 GiB avail; 551 B/s rd, 0 op/s; 263 B/s, 10 objects/s recovering 2024-03-11T21:49:25.468 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:49:25 smithi092 ceph-mon[29572]: 2.6 scrub starts 2024-03-11T21:49:25.468 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:49:25 smithi092 ceph-mon[29572]: 2.6 scrub ok 2024-03-11T21:49:25.469 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:49:25 smithi092 ceph-mon[29572]: from='mgr.14607 ' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.11", "id": [3, 5]}]': finished 2024-03-11T21:49:25.469 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:49:25 smithi092 ceph-mon[29572]: from='mgr.14607 ' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.15", "id": [3, 2]}]': finished 2024-03-11T21:49:25.469 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:49:25 smithi092 ceph-mon[29572]: from='mgr.14607 ' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "5.1a", "id": [0, 3]}]': finished 2024-03-11T21:49:25.469 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:49:25 smithi092 ceph-mon[29572]: osdmap e531: 8 total, 8 up, 7 in 2024-03-11T21:49:25.585 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:49:25 smithi097 ceph-mon[33317]: pgmap v1137: 132 pgs: 132 active+clean; 582 KiB data, 5.7 GiB used, 620 GiB / 626 GiB avail; 551 B/s rd, 0 op/s; 263 B/s, 10 objects/s recovering 2024-03-11T21:49:25.585 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:49:25 smithi097 ceph-mon[33317]: 2.6 scrub starts 2024-03-11T21:49:25.585 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:49:25 smithi097 ceph-mon[33317]: 2.6 scrub ok 2024-03-11T21:49:25.586 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:49:25 smithi097 ceph-mon[33317]: from='mgr.14607 ' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.11", "id": [3, 5]}]': finished 2024-03-11T21:49:25.586 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:49:25 smithi097 ceph-mon[33317]: from='mgr.14607 ' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.15", "id": [3, 2]}]': finished 2024-03-11T21:49:25.586 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:49:25 smithi097 ceph-mon[33317]: from='mgr.14607 ' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "5.1a", "id": [0, 3]}]': finished 2024-03-11T21:49:25.586 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:49:25 smithi097 ceph-mon[33317]: osdmap e531: 8 total, 8 up, 7 in 2024-03-11T21:49:25.615 INFO:teuthology.orchestra.run.smithi092.stderr:marked in osd.4. 2024-03-11T21:49:25.766 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:49:25 smithi092 ceph-00db0016-dfed-11ee-95c8-87774f69a715-mon-a[29568]: 2024-03-11T21:49:25.475+0000 7f87a4389700 -1 mon.a@0(leader).osd e531 definitely_dead 0 2024-03-11T21:49:25.831 DEBUG:teuthology.orchestra.run.smithi092:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid 00db0016-dfed-11ee-95c8-87774f69a715 -- ceph daemon osd.3 dump_historic_ops 2024-03-11T21:49:26.178 INFO:teuthology.orchestra.run.smithi092.stderr:Inferring config /var/lib/ceph/00db0016-dfed-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T21:49:26.585 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:49:26 smithi097 ceph-mon[33317]: 4.1f deep-scrub starts 2024-03-11T21:49:26.586 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:49:26 smithi097 ceph-mon[33317]: 4.1f deep-scrub ok 2024-03-11T21:49:26.586 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:49:26 smithi097 ceph-mon[33317]: 2.13 scrub starts 2024-03-11T21:49:26.586 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:49:26 smithi097 ceph-mon[33317]: 2.13 scrub ok 2024-03-11T21:49:26.586 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:49:26 smithi097 ceph-mon[33317]: from='client.? v1:172.21.15.92:0/2280932350' entity='client.admin' cmd=[{"prefix": "osd in", "ids": ["4"]}]: dispatch 2024-03-11T21:49:26.586 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:49:26 smithi097 ceph-mon[33317]: from='client.? v1:172.21.15.92:0/2280932350' entity='client.admin' cmd='[{"prefix": "osd in", "ids": ["4"]}]': finished 2024-03-11T21:49:26.586 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:49:26 smithi097 ceph-mon[33317]: osdmap e532: 8 total, 8 up, 8 in 2024-03-11T21:49:26.620 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:49:26 smithi092 ceph-mon[29572]: 4.1f deep-scrub starts 2024-03-11T21:49:26.620 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:49:26 smithi092 ceph-mon[29572]: 4.1f deep-scrub ok 2024-03-11T21:49:26.620 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:49:26 smithi092 ceph-mon[29572]: 2.13 scrub starts 2024-03-11T21:49:26.620 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:49:26 smithi092 ceph-mon[29572]: 2.13 scrub ok 2024-03-11T21:49:26.620 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:49:26 smithi092 ceph-mon[29572]: from='client.? v1:172.21.15.92:0/2280932350' entity='client.admin' cmd=[{"prefix": "osd in", "ids": ["4"]}]: dispatch 2024-03-11T21:49:26.620 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:49:26 smithi092 ceph-mon[29572]: from='client.? v1:172.21.15.92:0/2280932350' entity='client.admin' cmd='[{"prefix": "osd in", "ids": ["4"]}]': finished 2024-03-11T21:49:26.620 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:49:26 smithi092 ceph-mon[29572]: osdmap e532: 8 total, 8 up, 8 in 2024-03-11T21:49:26.620 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:49:26 smithi092 ceph-mon[37563]: 4.1f deep-scrub starts 2024-03-11T21:49:26.620 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:49:26 smithi092 ceph-mon[37563]: 4.1f deep-scrub ok 2024-03-11T21:49:26.620 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:49:26 smithi092 ceph-mon[37563]: 2.13 scrub starts 2024-03-11T21:49:26.621 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:49:26 smithi092 ceph-mon[37563]: 2.13 scrub ok 2024-03-11T21:49:26.621 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:49:26 smithi092 ceph-mon[37563]: from='client.? v1:172.21.15.92:0/2280932350' entity='client.admin' cmd=[{"prefix": "osd in", "ids": ["4"]}]: dispatch 2024-03-11T21:49:26.621 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:49:26 smithi092 ceph-mon[37563]: from='client.? v1:172.21.15.92:0/2280932350' entity='client.admin' cmd='[{"prefix": "osd in", "ids": ["4"]}]': finished 2024-03-11T21:49:26.621 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:49:26 smithi092 ceph-mon[37563]: osdmap e532: 8 total, 8 up, 8 in 2024-03-11T21:49:26.719 INFO:tasks.thrashosds.thrasher:Added osd 4 2024-03-11T21:49:26.719 DEBUG:teuthology.orchestra.run.smithi092:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid 00db0016-dfed-11ee-95c8-87774f69a715 -- ceph -- tell 'mon.*' injectargs --mon_osd_down_out_interval=600 2024-03-11T21:49:27.358 INFO:teuthology.orchestra.run.smithi092.stderr:Inferring config /var/lib/ceph/00db0016-dfed-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T21:49:27.369 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:49:27 smithi092 ceph-mon[29572]: pgmap v1139: 132 pgs: 132 active+clean; 582 KiB data, 5.7 GiB used, 620 GiB / 626 GiB avail; 1.1 KiB/s rd, 1 op/s; 229 B/s, 9 objects/s recovering 2024-03-11T21:49:27.369 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:49:27 smithi092 ceph-mon[29572]: 5.4 scrub starts 2024-03-11T21:49:27.369 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:49:27 smithi092 ceph-mon[29572]: 5.4 scrub ok 2024-03-11T21:49:27.369 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:49:27 smithi092 ceph-mon[29572]: osdmap e533: 8 total, 8 up, 8 in 2024-03-11T21:49:27.370 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:49:27 smithi092 ceph-mon[37563]: pgmap v1139: 132 pgs: 132 active+clean; 582 KiB data, 5.7 GiB used, 620 GiB / 626 GiB avail; 1.1 KiB/s rd, 1 op/s; 229 B/s, 9 objects/s recovering 2024-03-11T21:49:27.370 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:49:27 smithi092 ceph-mon[37563]: 5.4 scrub starts 2024-03-11T21:49:27.370 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:49:27 smithi092 ceph-mon[37563]: 5.4 scrub ok 2024-03-11T21:49:27.370 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:49:27 smithi092 ceph-mon[37563]: osdmap e533: 8 total, 8 up, 8 in 2024-03-11T21:49:27.585 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:49:27 smithi097 ceph-mon[33317]: pgmap v1139: 132 pgs: 132 active+clean; 582 KiB data, 5.7 GiB used, 620 GiB / 626 GiB avail; 1.1 KiB/s rd, 1 op/s; 229 B/s, 9 objects/s recovering 2024-03-11T21:49:27.585 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:49:27 smithi097 ceph-mon[33317]: 5.4 scrub starts 2024-03-11T21:49:27.585 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:49:27 smithi097 ceph-mon[33317]: 5.4 scrub ok 2024-03-11T21:49:27.585 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:49:27 smithi097 ceph-mon[33317]: osdmap e533: 8 total, 8 up, 8 in 2024-03-11T21:49:28.120 INFO:journalctl@ceph.mgr.y.smithi092.stdout:Mar 11 21:49:27 smithi092 ceph-00db0016-dfed-11ee-95c8-87774f69a715-mgr-y[29801]: ::ffff:172.21.15.97 - - [11/Mar/2024:21:49:27] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-11T21:49:28.585 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:49:28 smithi097 ceph-mon[33317]: 2.11 scrub starts 2024-03-11T21:49:28.586 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:49:28 smithi097 ceph-mon[33317]: 2.11 scrub ok 2024-03-11T21:49:28.586 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:49:28 smithi097 ceph-mon[33317]: 5.13 scrub starts 2024-03-11T21:49:28.586 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:49:28 smithi097 ceph-mon[33317]: 5.13 scrub ok 2024-03-11T21:49:28.619 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:49:28 smithi092 ceph-mon[29572]: 2.11 scrub starts 2024-03-11T21:49:28.619 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:49:28 smithi092 ceph-mon[29572]: 2.11 scrub ok 2024-03-11T21:49:28.620 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:49:28 smithi092 ceph-mon[29572]: 5.13 scrub starts 2024-03-11T21:49:28.620 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:49:28 smithi092 ceph-mon[29572]: 5.13 scrub ok 2024-03-11T21:49:28.620 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:49:28 smithi092 ceph-mon[37563]: 2.11 scrub starts 2024-03-11T21:49:28.620 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:49:28 smithi092 ceph-mon[37563]: 2.11 scrub ok 2024-03-11T21:49:28.620 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:49:28 smithi092 ceph-mon[37563]: 5.13 scrub starts 2024-03-11T21:49:28.620 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:49:28 smithi092 ceph-mon[37563]: 5.13 scrub ok 2024-03-11T21:49:29.318 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:49:29 smithi092 ceph-mon[29572]: pgmap v1142: 132 pgs: 132 active+clean; 582 KiB data, 6.3 GiB used, 709 GiB / 715 GiB avail; 170 B/s rd, 0 op/s; 126 B/s, 3 objects/s recovering 2024-03-11T21:49:29.318 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:49:29 smithi092 ceph-mon[37563]: pgmap v1142: 132 pgs: 132 active+clean; 582 KiB data, 6.3 GiB used, 709 GiB / 715 GiB avail; 170 B/s rd, 0 op/s; 126 B/s, 3 objects/s recovering 2024-03-11T21:49:29.555 INFO:teuthology.orchestra.run.smithi092.stdout:{ 2024-03-11T21:49:29.556 INFO:teuthology.orchestra.run.smithi092.stdout: "size": 20, 2024-03-11T21:49:29.556 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 600, 2024-03-11T21:49:29.556 INFO:teuthology.orchestra.run.smithi092.stdout: "ops": [ 2024-03-11T21:49:29.556 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:49:29.557 INFO:teuthology.orchestra.run.smithi092.stdout: "description": "osd_op(client.15744.0:108448 10.15 10:addedd6d:::benchmark_data_smithi092_161462_object108447:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e390)", 2024-03-11T21:49:29.557 INFO:teuthology.orchestra.run.smithi092.stdout: "initiated_at": "2024-03-11T21:42:20.463644+0000", 2024-03-11T21:49:29.557 INFO:teuthology.orchestra.run.smithi092.stdout: "age": 429.09857960099998, 2024-03-11T21:49:29.557 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.0513548589999999, 2024-03-11T21:49:29.557 INFO:teuthology.orchestra.run.smithi092.stdout: "type_data": { 2024-03-11T21:49:29.557 INFO:teuthology.orchestra.run.smithi092.stdout: "flag_point": "reached pg", 2024-03-11T21:49:29.558 INFO:teuthology.orchestra.run.smithi092.stdout: "client_info": { 2024-03-11T21:49:29.558 INFO:teuthology.orchestra.run.smithi092.stdout: "client": "client.15744", 2024-03-11T21:49:29.558 INFO:teuthology.orchestra.run.smithi092.stdout: "client_addr": "v1:172.21.15.92:0/3334000812", 2024-03-11T21:49:29.558 INFO:teuthology.orchestra.run.smithi092.stdout: "tid": 108448 2024-03-11T21:49:29.558 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:49:29.558 INFO:teuthology.orchestra.run.smithi092.stdout: "events": [ 2024-03-11T21:49:29.559 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:49:29.559 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "initiated", 2024-03-11T21:49:29.559 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:42:20.463644+0000", 2024-03-11T21:49:29.559 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-03-11T21:49:29.559 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:49:29.559 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:49:29.559 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "header_read", 2024-03-11T21:49:29.560 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:42:20.463644+0000", 2024-03-11T21:49:29.560 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-03-11T21:49:29.560 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:49:29.560 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:49:29.560 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "throttled", 2024-03-11T21:49:29.560 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:42:20.463646+0000", 2024-03-11T21:49:29.561 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 2.2189999999999998e-06 2024-03-11T21:49:29.561 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:49:29.561 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:49:29.561 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "all_read", 2024-03-11T21:49:29.561 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:42:20.463670+0000", 2024-03-11T21:49:29.561 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 2.4176e-05 2024-03-11T21:49:29.562 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:49:29.562 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:49:29.562 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "dispatched", 2024-03-11T21:49:29.562 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:42:20.465066+0000", 2024-03-11T21:49:29.562 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.001395285 2024-03-11T21:49:29.563 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:49:29.563 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:49:29.563 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "queued_for_pg", 2024-03-11T21:49:29.563 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:42:20.465082+0000", 2024-03-11T21:49:29.563 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.6682000000000001e-05 2024-03-11T21:49:29.563 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:49:29.564 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:49:29.564 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-03-11T21:49:29.564 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:42:20.465711+0000", 2024-03-11T21:49:29.564 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00062905199999999995 2024-03-11T21:49:29.564 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:49:29.564 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:49:29.564 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "started", 2024-03-11T21:49:29.565 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:42:20.465870+0000", 2024-03-11T21:49:29.565 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00015818000000000001 2024-03-11T21:49:29.565 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:49:29.565 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:49:29.565 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "waiting for subops from 1,5", 2024-03-11T21:49:29.565 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:42:20.467415+0000", 2024-03-11T21:49:29.565 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.001545034 2024-03-11T21:49:29.565 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:49:29.565 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:49:29.566 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "op_commit", 2024-03-11T21:49:29.566 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:42:20.470088+0000", 2024-03-11T21:49:29.566 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.0026732309999999999 2024-03-11T21:49:29.566 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:49:29.566 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:49:29.566 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-03-11T21:49:29.566 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:42:20.478868+0000", 2024-03-11T21:49:29.567 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.0087802330000000001 2024-03-11T21:49:29.567 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:49:29.567 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:49:29.567 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "waiting for peered", 2024-03-11T21:49:29.567 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:42:20.478928+0000", 2024-03-11T21:49:29.567 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 6.0056999999999998e-05 2024-03-11T21:49:29.567 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:49:29.567 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:49:29.567 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-03-11T21:49:29.568 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:42:21.514922+0000", 2024-03-11T21:49:29.568 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.035993709 2024-03-11T21:49:29.568 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:49:29.568 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:49:29.568 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "done", 2024-03-11T21:49:29.568 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:42:21.514999+0000", 2024-03-11T21:49:29.568 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 7.7001000000000004e-05 2024-03-11T21:49:29.568 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-03-11T21:49:29.569 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-03-11T21:49:29.569 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-03-11T21:49:29.569 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:49:29.569 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:49:29.569 INFO:teuthology.orchestra.run.smithi092.stdout: "description": "osd_op(client.15744.0:108450 10.15 10:aa1b6734:::benchmark_data_smithi092_161462_object108449:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e390)", 2024-03-11T21:49:29.569 INFO:teuthology.orchestra.run.smithi092.stdout: "initiated_at": "2024-03-11T21:42:20.467134+0000", 2024-03-11T21:49:29.569 INFO:teuthology.orchestra.run.smithi092.stdout: "age": 429.095089696, 2024-03-11T21:49:29.569 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.064492443, 2024-03-11T21:49:29.569 INFO:teuthology.orchestra.run.smithi092.stdout: "type_data": { 2024-03-11T21:49:29.570 INFO:teuthology.orchestra.run.smithi092.stdout: "flag_point": "reached pg", 2024-03-11T21:49:29.570 INFO:teuthology.orchestra.run.smithi092.stdout: "client_info": { 2024-03-11T21:49:29.570 INFO:teuthology.orchestra.run.smithi092.stdout: "client": "client.15744", 2024-03-11T21:49:29.570 INFO:teuthology.orchestra.run.smithi092.stdout: "client_addr": "v1:172.21.15.92:0/3334000812", 2024-03-11T21:49:29.570 INFO:teuthology.orchestra.run.smithi092.stdout: "tid": 108450 2024-03-11T21:49:29.570 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:49:29.570 INFO:teuthology.orchestra.run.smithi092.stdout: "events": [ 2024-03-11T21:49:29.571 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:49:29.571 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "initiated", 2024-03-11T21:49:29.571 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:42:20.467134+0000", 2024-03-11T21:49:29.571 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-03-11T21:49:29.571 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:49:29.571 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:49:29.571 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "header_read", 2024-03-11T21:49:29.571 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:42:20.467134+0000", 2024-03-11T21:49:29.572 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-03-11T21:49:29.572 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:49:29.572 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:49:29.572 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "throttled", 2024-03-11T21:49:29.572 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:42:20.467135+0000", 2024-03-11T21:49:29.572 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.356e-06 2024-03-11T21:49:29.572 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:49:29.572 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:49:29.572 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "all_read", 2024-03-11T21:49:29.572 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:42:20.467160+0000", 2024-03-11T21:49:29.572 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 2.5148999999999999e-05 2024-03-11T21:49:29.572 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:49:29.573 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:49:29.573 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "dispatched", 2024-03-11T21:49:29.573 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:42:20.467172+0000", 2024-03-11T21:49:29.573 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.1385e-05 2024-03-11T21:49:29.573 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:49:29.573 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:49:29.573 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "queued_for_pg", 2024-03-11T21:49:29.573 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:42:20.467181+0000", 2024-03-11T21:49:29.573 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 9.0119999999999994e-06 2024-03-11T21:49:29.573 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:49:29.573 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:49:29.573 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-03-11T21:49:29.573 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:42:20.470116+0000", 2024-03-11T21:49:29.573 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.0029347869999999999 2024-03-11T21:49:29.574 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:49:29.574 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:49:29.574 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "started", 2024-03-11T21:49:29.574 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:42:20.470348+0000", 2024-03-11T21:49:29.574 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00023264600000000001 2024-03-11T21:49:29.574 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:49:29.574 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:49:29.574 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "waiting for subops from 1,5", 2024-03-11T21:49:29.574 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:42:20.474234+0000", 2024-03-11T21:49:29.574 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.0038860959999999999 2024-03-11T21:49:29.574 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:49:29.574 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:49:29.574 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-03-11T21:49:29.575 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:42:20.480193+0000", 2024-03-11T21:49:29.575 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.0059590950000000002 2024-03-11T21:49:29.575 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:49:29.575 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:49:29.575 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "waiting for peered", 2024-03-11T21:49:29.575 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:42:20.480225+0000", 2024-03-11T21:49:29.575 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 3.1541000000000002e-05 2024-03-11T21:49:29.575 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:49:29.575 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:49:29.575 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-03-11T21:49:29.575 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:42:21.515329+0000", 2024-03-11T21:49:29.575 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.035103605 2024-03-11T21:49:29.576 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:49:29.576 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:49:29.576 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "waiting for degraded object", 2024-03-11T21:49:29.576 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:42:21.516041+0000", 2024-03-11T21:49:29.576 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00071204799999999996 2024-03-11T21:49:29.576 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:49:29.576 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:49:29.576 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-03-11T21:49:29.576 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:42:21.530916+0000", 2024-03-11T21:49:29.576 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.014875571000000001 2024-03-11T21:49:29.576 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:49:29.576 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:49:29.577 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "done", 2024-03-11T21:49:29.577 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:42:21.531626+0000", 2024-03-11T21:49:29.577 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00071015200000000003 2024-03-11T21:49:29.577 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-03-11T21:49:29.577 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-03-11T21:49:29.577 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-03-11T21:49:29.577 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:49:29.577 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:49:29.577 INFO:teuthology.orchestra.run.smithi092.stdout: "description": "osd_op(client.15744.0:108463 10.15 10:a9f3739c:::benchmark_data_smithi092_161462_object108462:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e391)", 2024-03-11T21:49:29.577 INFO:teuthology.orchestra.run.smithi092.stdout: "initiated_at": "2024-03-11T21:42:20.484576+0000", 2024-03-11T21:49:29.577 INFO:teuthology.orchestra.run.smithi092.stdout: "age": 429.07764809299999, 2024-03-11T21:49:29.577 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.051885159, 2024-03-11T21:49:29.577 INFO:teuthology.orchestra.run.smithi092.stdout: "type_data": { 2024-03-11T21:49:29.577 INFO:teuthology.orchestra.run.smithi092.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-11T21:49:29.578 INFO:teuthology.orchestra.run.smithi092.stdout: "client_info": { 2024-03-11T21:49:29.578 INFO:teuthology.orchestra.run.smithi092.stdout: "client": "client.15744", 2024-03-11T21:49:29.578 INFO:teuthology.orchestra.run.smithi092.stdout: "client_addr": "v1:172.21.15.92:0/3334000812", 2024-03-11T21:49:29.578 INFO:teuthology.orchestra.run.smithi092.stdout: "tid": 108463 2024-03-11T21:49:29.578 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:49:29.578 INFO:teuthology.orchestra.run.smithi092.stdout: "events": [ 2024-03-11T21:49:29.578 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:49:29.578 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "initiated", 2024-03-11T21:49:29.578 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:42:20.484576+0000", 2024-03-11T21:49:29.578 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-03-11T21:49:29.578 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:49:29.578 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:49:29.578 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "header_read", 2024-03-11T21:49:29.579 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:42:20.484576+0000", 2024-03-11T21:49:29.579 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-03-11T21:49:29.579 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:49:29.579 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:49:29.579 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "throttled", 2024-03-11T21:49:29.579 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:42:20.484578+0000", 2024-03-11T21:49:29.579 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 2.357e-06 2024-03-11T21:49:29.579 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:49:29.579 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:49:29.579 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "all_read", 2024-03-11T21:49:29.579 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:42:20.484606+0000", 2024-03-11T21:49:29.579 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 2.7960999999999999e-05 2024-03-11T21:49:29.580 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:49:29.580 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:49:29.580 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "dispatched", 2024-03-11T21:49:29.580 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:42:20.484619+0000", 2024-03-11T21:49:29.580 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.2731000000000001e-05 2024-03-11T21:49:29.580 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:49:29.580 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:49:29.580 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "queued_for_pg", 2024-03-11T21:49:29.581 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:42:20.484635+0000", 2024-03-11T21:49:29.581 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.6368999999999999e-05 2024-03-11T21:49:29.581 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:49:29.581 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:49:29.581 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-03-11T21:49:29.581 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:42:20.484681+0000", 2024-03-11T21:49:29.581 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 4.5920000000000001e-05 2024-03-11T21:49:29.581 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:49:29.582 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:49:29.582 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "waiting for peered", 2024-03-11T21:49:29.582 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:42:20.484692+0000", 2024-03-11T21:49:29.582 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.0628000000000001e-05 2024-03-11T21:49:29.582 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:49:29.582 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:49:29.582 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-03-11T21:49:29.582 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:42:21.516068+0000", 2024-03-11T21:49:29.582 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.0313760940000001 2024-03-11T21:49:29.582 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:49:29.582 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:49:29.583 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "started", 2024-03-11T21:49:29.583 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:42:21.516148+0000", 2024-03-11T21:49:29.583 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 8.03e-05 2024-03-11T21:49:29.583 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:49:29.583 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:49:29.583 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "waiting for subops from 1,5", 2024-03-11T21:49:29.583 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:42:21.516438+0000", 2024-03-11T21:49:29.583 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00029059099999999998 2024-03-11T21:49:29.583 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:49:29.583 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:49:29.583 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "op_commit", 2024-03-11T21:49:29.583 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:42:21.520062+0000", 2024-03-11T21:49:29.583 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.0036235429999999999 2024-03-11T21:49:29.583 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:49:29.583 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:49:29.584 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "sub_op_commit_rec", 2024-03-11T21:49:29.584 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:42:21.523073+0000", 2024-03-11T21:49:29.584 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.0030108090000000001 2024-03-11T21:49:29.584 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:49:29.584 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:49:29.584 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "sub_op_commit_rec", 2024-03-11T21:49:29.584 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:42:21.536154+0000", 2024-03-11T21:49:29.584 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.013081136 2024-03-11T21:49:29.584 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:49:29.584 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:49:29.584 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "commit_sent", 2024-03-11T21:49:29.584 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:42:21.536308+0000", 2024-03-11T21:49:29.584 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.000154065 2024-03-11T21:49:29.584 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:49:29.585 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:49:29.585 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "done", 2024-03-11T21:49:29.585 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:42:21.536461+0000", 2024-03-11T21:49:29.585 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00015265499999999999 2024-03-11T21:49:29.585 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-03-11T21:49:29.585 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-03-11T21:49:29.585 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-03-11T21:49:29.585 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:49:29.585 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:49:29.585 INFO:teuthology.orchestra.run.smithi092.stdout: "description": "osd_op(client.15744.0:108510 10.15 10:ac71a222:::benchmark_data_smithi092_161462_object108509:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e391)", 2024-03-11T21:49:29.585 INFO:teuthology.orchestra.run.smithi092.stdout: "initiated_at": "2024-03-11T21:42:20.582045+0000", 2024-03-11T21:49:29.585 INFO:teuthology.orchestra.run.smithi092.stdout: "age": 428.98017895200002, 2024-03-11T21:49:29.585 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.95462952599999995, 2024-03-11T21:49:29.585 INFO:teuthology.orchestra.run.smithi092.stdout: "type_data": { 2024-03-11T21:49:29.586 INFO:teuthology.orchestra.run.smithi092.stdout: "flag_point": "commit sent; apply or cleanup", 2024-03-11T21:49:29.586 INFO:teuthology.orchestra.run.smithi092.stdout: "client_info": { 2024-03-11T21:49:29.586 INFO:teuthology.orchestra.run.smithi092.stdout: "client": "client.15744", 2024-03-11T21:49:29.586 INFO:teuthology.orchestra.run.smithi092.stdout: "client_addr": "v1:172.21.15.92:0/3334000812", 2024-03-11T21:49:29.586 INFO:teuthology.orchestra.run.smithi092.stdout: "tid": 108510 2024-03-11T21:49:29.586 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:49:29.586 INFO:teuthology.orchestra.run.smithi092.stdout: "events": [ 2024-03-11T21:49:29.586 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:49:29.586 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "initiated", 2024-03-11T21:49:29.587 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:42:20.582045+0000", 2024-03-11T21:49:29.587 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-03-11T21:49:29.587 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:49:29.587 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:49:29.587 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "header_read", 2024-03-11T21:49:29.587 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:42:20.582045+0000", 2024-03-11T21:49:29.587 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-03-11T21:49:29.587 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:49:29.587 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:49:29.587 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "throttled", 2024-03-11T21:49:29.587 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:42:20.582047+0000", 2024-03-11T21:49:29.587 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 2.514e-06 2024-03-11T21:49:29.587 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:49:29.587 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:49:29.587 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "all_read", 2024-03-11T21:49:29.588 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:42:20.582081+0000", 2024-03-11T21:49:29.588 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 3.3383999999999997e-05 2024-03-11T21:49:29.588 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:49:29.588 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:49:29.588 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "dispatched", 2024-03-11T21:49:29.588 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:42:20.582120+0000", 2024-03-11T21:49:29.588 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 3.9728000000000003e-05 2024-03-11T21:49:29.588 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:49:29.588 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:49:29.588 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "queued_for_pg", 2024-03-11T21:49:29.588 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:42:20.582128+0000", 2024-03-11T21:49:29.588 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 8.1259999999999998e-06 2024-03-11T21:49:29.588 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:49:29.588 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:49:29.589 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-03-11T21:49:29.589 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:42:20.582319+0000", 2024-03-11T21:49:29.589 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.000190581 2024-03-11T21:49:29.589 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:49:29.589 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:49:29.589 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "waiting for peered", 2024-03-11T21:49:29.589 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:42:20.582344+0000", 2024-03-11T21:49:29.589 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 2.4797e-05 2024-03-11T21:49:29.589 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:49:29.589 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:49:29.589 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-03-11T21:49:29.589 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:42:21.518102+0000", 2024-03-11T21:49:29.589 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.93575861999999999 2024-03-11T21:49:29.589 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:49:29.590 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:49:29.590 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "started", 2024-03-11T21:49:29.590 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:42:21.518232+0000", 2024-03-11T21:49:29.590 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00012929 2024-03-11T21:49:29.590 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:49:29.590 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:49:29.590 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "waiting for subops from 1,5", 2024-03-11T21:49:29.590 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:42:21.518515+0000", 2024-03-11T21:49:29.590 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00028313600000000002 2024-03-11T21:49:29.590 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:49:29.590 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:49:29.590 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "op_commit", 2024-03-11T21:49:29.590 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:42:21.523031+0000", 2024-03-11T21:49:29.590 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.0045159780000000004 2024-03-11T21:49:29.591 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:49:29.591 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:49:29.591 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "sub_op_commit_rec", 2024-03-11T21:49:29.591 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:42:21.523124+0000", 2024-03-11T21:49:29.591 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 9.2898000000000004e-05 2024-03-11T21:49:29.591 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:49:29.591 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:49:29.591 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "sub_op_commit_rec", 2024-03-11T21:49:29.591 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:42:21.536526+0000", 2024-03-11T21:49:29.591 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.013401912 2024-03-11T21:49:29.591 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:49:29.591 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:49:29.591 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "commit_sent", 2024-03-11T21:49:29.591 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:42:21.536576+0000", 2024-03-11T21:49:29.591 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 5.0510000000000003e-05 2024-03-11T21:49:29.592 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:49:29.592 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:49:29.592 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "done", 2024-03-11T21:49:29.592 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:42:21.536674+0000", 2024-03-11T21:49:29.592 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 9.8052000000000002e-05 2024-03-11T21:49:29.592 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-03-11T21:49:29.592 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-03-11T21:49:29.592 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-03-11T21:49:29.592 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:49:29.592 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:49:29.593 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:49:29 smithi097 ceph-mon[33317]: pgmap v1142: 132 pgs: 132 active+clean; 582 KiB data, 6.3 GiB used, 709 GiB / 715 GiB avail; 170 B/s rd, 0 op/s; 126 B/s, 3 objects/s recovering 2024-03-11T21:49:29.594 INFO:teuthology.orchestra.run.smithi092.stdout: "description": "MOSDPGPush(4.10 529/528 [PushOp(4:0fd950d8:::obj_delete_at_hint.0000000015:head, version: 456'12, data_included: [], data_size: 0, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(4:0fd950d8:::obj_delete_at_hint.0000000015:head@456'12, size: 0, copy_subset: [], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 0, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-03-11T21:49:29.594 INFO:teuthology.orchestra.run.smithi092.stdout: "initiated_at": "2024-03-11T21:49:17.152211+0000", 2024-03-11T21:49:29.594 INFO:teuthology.orchestra.run.smithi092.stdout: "age": 12.410012168, 2024-03-11T21:49:29.594 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00081696999999999998, 2024-03-11T21:49:29.594 INFO:teuthology.orchestra.run.smithi092.stdout: "type_data": { 2024-03-11T21:49:29.594 INFO:teuthology.orchestra.run.smithi092.stdout: "flag_point": "started", 2024-03-11T21:49:29.594 INFO:teuthology.orchestra.run.smithi092.stdout: "events": [ 2024-03-11T21:49:29.594 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:49:29.594 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "initiated", 2024-03-11T21:49:29.594 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:49:17.152211+0000", 2024-03-11T21:49:29.595 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-03-11T21:49:29.595 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:49:29.595 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:49:29.595 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "header_read", 2024-03-11T21:49:29.595 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:49:17.152211+0000", 2024-03-11T21:49:29.595 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-03-11T21:49:29.595 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:49:29.595 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:49:29.595 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "throttled", 2024-03-11T21:49:29.595 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:49:17.152213+0000", 2024-03-11T21:49:29.595 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.0720000000000001e-06 2024-03-11T21:49:29.595 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:49:29.595 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:49:29.595 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "all_read", 2024-03-11T21:49:29.596 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:49:17.152224+0000", 2024-03-11T21:49:29.596 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.1768e-05 2024-03-11T21:49:29.596 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:49:29.596 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:49:29.596 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "dispatched", 2024-03-11T21:49:29.596 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:49:17.152239+0000", 2024-03-11T21:49:29.596 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.4494e-05 2024-03-11T21:49:29.596 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:49:29.596 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:49:29.596 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "queued_for_pg", 2024-03-11T21:49:29.596 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:49:17.152254+0000", 2024-03-11T21:49:29.596 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.5511000000000001e-05 2024-03-11T21:49:29.596 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:49:29.596 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:49:29.596 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-03-11T21:49:29.597 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:49:17.152325+0000", 2024-03-11T21:49:29.597 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 7.1136999999999993e-05 2024-03-11T21:49:29.597 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:49:29.597 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:49:29.597 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "started", 2024-03-11T21:49:29.597 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:49:17.152339+0000", 2024-03-11T21:49:29.597 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.3422e-05 2024-03-11T21:49:29.597 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:49:29.597 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:49:29.597 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "done", 2024-03-11T21:49:29.597 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:49:17.153028+0000", 2024-03-11T21:49:29.597 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00068956599999999996 2024-03-11T21:49:29.597 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-03-11T21:49:29.597 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-03-11T21:49:29.597 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-03-11T21:49:29.598 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:49:29.598 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:49:29.598 INFO:teuthology.orchestra.run.smithi092.stdout: "description": "MOSDPGPush(2.6 529/528 [PushOp(2:66436ebe:::default.zonegroup.:head, version: 67'1, data_included: [0~46], data_size: 46, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(2:66436ebe:::default.zonegroup.:head@67'1, size: 46, copy_subset: [(0, 46)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 46, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-03-11T21:49:29.598 INFO:teuthology.orchestra.run.smithi092.stdout: "initiated_at": "2024-03-11T21:49:17.155929+0000", 2024-03-11T21:49:29.598 INFO:teuthology.orchestra.run.smithi092.stdout: "age": 12.406294143, 2024-03-11T21:49:29.598 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00054024699999999999, 2024-03-11T21:49:29.598 INFO:teuthology.orchestra.run.smithi092.stdout: "type_data": { 2024-03-11T21:49:29.598 INFO:teuthology.orchestra.run.smithi092.stdout: "flag_point": "started", 2024-03-11T21:49:29.598 INFO:teuthology.orchestra.run.smithi092.stdout: "events": [ 2024-03-11T21:49:29.598 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:49:29.598 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "initiated", 2024-03-11T21:49:29.598 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:49:17.155929+0000", 2024-03-11T21:49:29.598 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-03-11T21:49:29.598 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:49:29.599 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:49:29.599 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "header_read", 2024-03-11T21:49:29.599 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:49:17.155929+0000", 2024-03-11T21:49:29.599 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-03-11T21:49:29.599 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:49:29.599 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:49:29.599 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "throttled", 2024-03-11T21:49:29.599 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:49:17.155930+0000", 2024-03-11T21:49:29.599 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 4.9100000000000004e-07 2024-03-11T21:49:29.599 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:49:29.599 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:49:29.599 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "all_read", 2024-03-11T21:49:29.599 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:49:17.155941+0000", 2024-03-11T21:49:29.599 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.0513000000000001e-05 2024-03-11T21:49:29.599 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:49:29.600 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:49:29.600 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "dispatched", 2024-03-11T21:49:29.600 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:49:17.155953+0000", 2024-03-11T21:49:29.600 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.2565000000000001e-05 2024-03-11T21:49:29.600 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:49:29.600 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:49:29.600 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "queued_for_pg", 2024-03-11T21:49:29.600 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:49:17.155967+0000", 2024-03-11T21:49:29.600 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.3953e-05 2024-03-11T21:49:29.600 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:49:29.600 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:49:29.600 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-03-11T21:49:29.600 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:49:17.156253+0000", 2024-03-11T21:49:29.600 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00028631799999999998 2024-03-11T21:49:29.600 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:49:29.601 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:49:29.601 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "started", 2024-03-11T21:49:29.601 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:49:17.156265+0000", 2024-03-11T21:49:29.601 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.2109999999999999e-05 2024-03-11T21:49:29.601 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:49:29.601 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:49:29.601 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "done", 2024-03-11T21:49:29.601 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:49:17.156470+0000", 2024-03-11T21:49:29.601 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00020429700000000001 2024-03-11T21:49:29.601 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-03-11T21:49:29.601 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-03-11T21:49:29.601 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-03-11T21:49:29.601 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:49:29.601 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:49:29.601 INFO:teuthology.orchestra.run.smithi092.stdout: "description": "MOSDPGPush(4.1b 529/528 [PushOp(4:db9d130d:gc::gc.24:head, version: 74'3, data_included: [0~110], data_size: 110, omap_header_size: 0, omap_entries_size: 0, attrset_size: 4, recovery_info: ObjectRecoveryInfo(4:db9d130d:gc::gc.24:head@74'3, size: 110, copy_subset: [(0, 110)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 110, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-03-11T21:49:29.602 INFO:teuthology.orchestra.run.smithi092.stdout: "initiated_at": "2024-03-11T21:49:17.164476+0000", 2024-03-11T21:49:29.602 INFO:teuthology.orchestra.run.smithi092.stdout: "age": 12.397748071000001, 2024-03-11T21:49:29.602 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00068807999999999999, 2024-03-11T21:49:29.602 INFO:teuthology.orchestra.run.smithi092.stdout: "type_data": { 2024-03-11T21:49:29.602 INFO:teuthology.orchestra.run.smithi092.stdout: "flag_point": "started", 2024-03-11T21:49:29.602 INFO:teuthology.orchestra.run.smithi092.stdout: "events": [ 2024-03-11T21:49:29.602 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:49:29.602 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "initiated", 2024-03-11T21:49:29.602 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:49:17.164476+0000", 2024-03-11T21:49:29.602 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-03-11T21:49:29.602 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:49:29.602 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:49:29.602 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "header_read", 2024-03-11T21:49:29.602 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:49:17.164476+0000", 2024-03-11T21:49:29.602 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-03-11T21:49:29.603 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:49:29.603 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:49:29.603 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "throttled", 2024-03-11T21:49:29.603 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:49:17.164477+0000", 2024-03-11T21:49:29.603 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 9.3699999999999999e-07 2024-03-11T21:49:29.603 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:49:29.603 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:49:29.603 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "all_read", 2024-03-11T21:49:29.603 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:49:17.164489+0000", 2024-03-11T21:49:29.603 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.2699e-05 2024-03-11T21:49:29.603 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:49:29.603 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:49:29.603 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "dispatched", 2024-03-11T21:49:29.603 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:49:17.164505+0000", 2024-03-11T21:49:29.604 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.5434000000000002e-05 2024-03-11T21:49:29.604 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:49:29.604 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:49:29.604 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "queued_for_pg", 2024-03-11T21:49:29.604 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:49:17.164519+0000", 2024-03-11T21:49:29.604 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.3944999999999999e-05 2024-03-11T21:49:29.604 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:49:29.604 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:49:29.604 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-03-11T21:49:29.604 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:49:17.164732+0000", 2024-03-11T21:49:29.604 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00021380999999999999 2024-03-11T21:49:29.604 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:49:29.604 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:49:29.604 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "started", 2024-03-11T21:49:29.604 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:49:17.164746+0000", 2024-03-11T21:49:29.605 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.3691e-05 2024-03-11T21:49:29.605 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:49:29.605 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:49:29.605 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "done", 2024-03-11T21:49:29.605 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:49:17.165164+0000", 2024-03-11T21:49:29.605 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00041756399999999999 2024-03-11T21:49:29.605 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-03-11T21:49:29.605 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-03-11T21:49:29.605 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-03-11T21:49:29.605 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:49:29.605 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:49:29.605 INFO:teuthology.orchestra.run.smithi092.stdout: "description": "MOSDPGPush(3.2 529/528 [PushOp(3:5cea7035:::rbd_info:head, version: 71'2, data_included: [0~19], data_size: 19, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(3:5cea7035:::rbd_info:head@71'2, size: 19, copy_subset: [(0, 19)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 19, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-03-11T21:49:29.605 INFO:teuthology.orchestra.run.smithi092.stdout: "initiated_at": "2024-03-11T21:49:17.165606+0000", 2024-03-11T21:49:29.605 INFO:teuthology.orchestra.run.smithi092.stdout: "age": 12.39661763, 2024-03-11T21:49:29.606 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00053092500000000002, 2024-03-11T21:49:29.606 INFO:teuthology.orchestra.run.smithi092.stdout: "type_data": { 2024-03-11T21:49:29.606 INFO:teuthology.orchestra.run.smithi092.stdout: "flag_point": "started", 2024-03-11T21:49:29.606 INFO:teuthology.orchestra.run.smithi092.stdout: "events": [ 2024-03-11T21:49:29.606 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:49:29.606 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "initiated", 2024-03-11T21:49:29.606 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:49:17.165606+0000", 2024-03-11T21:49:29.606 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-03-11T21:49:29.606 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:49:29.606 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:49:29.606 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "header_read", 2024-03-11T21:49:29.606 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:49:17.165606+0000", 2024-03-11T21:49:29.606 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-03-11T21:49:29.606 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:49:29.607 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:49:29.607 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "throttled", 2024-03-11T21:49:29.607 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:49:17.165606+0000", 2024-03-11T21:49:29.607 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 4.0200000000000003e-07 2024-03-11T21:49:29.607 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:49:29.607 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:49:29.607 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "all_read", 2024-03-11T21:49:29.607 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:49:17.165616+0000", 2024-03-11T21:49:29.607 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 9.645e-06 2024-03-11T21:49:29.607 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:49:29.607 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:49:29.607 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "dispatched", 2024-03-11T21:49:29.607 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:49:17.165626+0000", 2024-03-11T21:49:29.607 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 9.6399999999999992e-06 2024-03-11T21:49:29.607 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:49:29.608 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:49:29.608 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "queued_for_pg", 2024-03-11T21:49:29.608 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:49:17.165637+0000", 2024-03-11T21:49:29.608 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.1629e-05 2024-03-11T21:49:29.608 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:49:29.608 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:49:29.608 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-03-11T21:49:29.608 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:49:17.165795+0000", 2024-03-11T21:49:29.608 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00015808900000000001 2024-03-11T21:49:29.608 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:49:29.608 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:49:29.608 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "started", 2024-03-11T21:49:29.608 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:49:17.165808+0000", 2024-03-11T21:49:29.608 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.2639999999999999e-05 2024-03-11T21:49:29.608 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:49:29.609 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:49:29.609 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "done", 2024-03-11T21:49:29.609 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:49:17.166137+0000", 2024-03-11T21:49:29.609 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00032887999999999998 2024-03-11T21:49:29.609 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-03-11T21:49:29.609 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-03-11T21:49:29.609 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-03-11T21:49:29.609 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:49:29.609 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:49:29.609 INFO:teuthology.orchestra.run.smithi092.stdout: "description": "MOSDPGPush(4.f 529/528 [PushOp(4:f24cb4a8:gc::gc.31:head, version: 74'5, data_included: [0~110], data_size: 110, omap_header_size: 0, omap_entries_size: 0, attrset_size: 4, recovery_info: ObjectRecoveryInfo(4:f24cb4a8:gc::gc.31:head@74'5, size: 110, copy_subset: [(0, 110)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 110, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-03-11T21:49:29.609 INFO:teuthology.orchestra.run.smithi092.stdout: "initiated_at": "2024-03-11T21:49:17.166782+0000", 2024-03-11T21:49:29.609 INFO:teuthology.orchestra.run.smithi092.stdout: "age": 12.395441420999999, 2024-03-11T21:49:29.609 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.005890986, 2024-03-11T21:49:29.609 INFO:teuthology.orchestra.run.smithi092.stdout: "type_data": { 2024-03-11T21:49:29.609 INFO:teuthology.orchestra.run.smithi092.stdout: "flag_point": "started", 2024-03-11T21:49:29.610 INFO:teuthology.orchestra.run.smithi092.stdout: "events": [ 2024-03-11T21:49:29.610 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:49:29.610 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "initiated", 2024-03-11T21:49:29.610 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:49:17.166782+0000", 2024-03-11T21:49:29.610 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-03-11T21:49:29.610 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:49:29.610 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:49:29.610 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "header_read", 2024-03-11T21:49:29.610 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:49:17.166782+0000", 2024-03-11T21:49:29.610 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-03-11T21:49:29.610 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:49:29.610 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:49:29.610 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "throttled", 2024-03-11T21:49:29.610 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:49:17.166783+0000", 2024-03-11T21:49:29.611 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 9.4099999999999997e-07 2024-03-11T21:49:29.611 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:49:29.611 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:49:29.611 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "all_read", 2024-03-11T21:49:29.611 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:49:17.166797+0000", 2024-03-11T21:49:29.611 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.3726999999999999e-05 2024-03-11T21:49:29.611 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:49:29.611 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:49:29.611 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "dispatched", 2024-03-11T21:49:29.611 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:49:17.166812+0000", 2024-03-11T21:49:29.611 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.4966999999999999e-05 2024-03-11T21:49:29.611 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:49:29.611 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:49:29.611 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "queued_for_pg", 2024-03-11T21:49:29.611 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:49:17.166825+0000", 2024-03-11T21:49:29.612 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.2764e-05 2024-03-11T21:49:29.612 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:49:29.612 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:49:29.612 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-03-11T21:49:29.612 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:49:17.172452+0000", 2024-03-11T21:49:29.612 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.005627684 2024-03-11T21:49:29.612 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:49:29.612 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:49:29.612 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "started", 2024-03-11T21:49:29.612 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:49:17.172465+0000", 2024-03-11T21:49:29.612 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.2993000000000001e-05 2024-03-11T21:49:29.612 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:49:29.612 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:49:29.612 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "done", 2024-03-11T21:49:29.612 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:49:17.172673+0000", 2024-03-11T21:49:29.613 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00020791000000000001 2024-03-11T21:49:29.613 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-03-11T21:49:29.613 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-03-11T21:49:29.613 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-03-11T21:49:29.613 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:49:29.613 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:49:29.613 INFO:teuthology.orchestra.run.smithi092.stdout: "description": "MOSDPGPush(4.1b 529/528 [PushOp(4:dc490aaa:::obj_delete_at_hint.0000000067:head, version: 456'9, data_included: [], data_size: 0, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(4:dc490aaa:::obj_delete_at_hint.0000000067:head@456'9, size: 0, copy_subset: [], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 0, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-03-11T21:49:29.613 INFO:teuthology.orchestra.run.smithi092.stdout: "initiated_at": "2024-03-11T21:49:17.175674+0000", 2024-03-11T21:49:29.613 INFO:teuthology.orchestra.run.smithi092.stdout: "age": 12.386549156999999, 2024-03-11T21:49:29.613 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.000503277, 2024-03-11T21:49:29.613 INFO:teuthology.orchestra.run.smithi092.stdout: "type_data": { 2024-03-11T21:49:29.613 INFO:teuthology.orchestra.run.smithi092.stdout: "flag_point": "started", 2024-03-11T21:49:29.613 INFO:teuthology.orchestra.run.smithi092.stdout: "events": [ 2024-03-11T21:49:29.613 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:49:29.613 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "initiated", 2024-03-11T21:49:29.614 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:49:17.175674+0000", 2024-03-11T21:49:29.614 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-03-11T21:49:29.614 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:49:29.614 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:49:29.614 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "header_read", 2024-03-11T21:49:29.614 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:49:17.175674+0000", 2024-03-11T21:49:29.614 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-03-11T21:49:29.614 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:49:29.614 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:49:29.614 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "throttled", 2024-03-11T21:49:29.614 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:49:17.175676+0000", 2024-03-11T21:49:29.614 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.3549999999999999e-06 2024-03-11T21:49:29.614 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:49:29.615 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:49:29.615 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "all_read", 2024-03-11T21:49:29.615 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:49:17.175687+0000", 2024-03-11T21:49:29.615 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.0879999999999999e-05 2024-03-11T21:49:29.615 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:49:29.615 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:49:29.615 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "dispatched", 2024-03-11T21:49:29.615 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:49:17.175703+0000", 2024-03-11T21:49:29.615 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.6124999999999999e-05 2024-03-11T21:49:29.615 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:49:29.615 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:49:29.615 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "queued_for_pg", 2024-03-11T21:49:29.615 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:49:17.175714+0000", 2024-03-11T21:49:29.615 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.0740999999999999e-05 2024-03-11T21:49:29.615 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:49:29.616 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:49:29.616 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-03-11T21:49:29.616 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:49:17.175793+0000", 2024-03-11T21:49:29.616 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 7.9240999999999996e-05 2024-03-11T21:49:29.616 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:49:29.616 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:49:29.616 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "started", 2024-03-11T21:49:29.616 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:49:17.175805+0000", 2024-03-11T21:49:29.616 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.2605e-05 2024-03-11T21:49:29.616 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:49:29.616 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:49:29.616 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "done", 2024-03-11T21:49:29.616 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:49:17.176178+0000", 2024-03-11T21:49:29.616 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00037232999999999998 2024-03-11T21:49:29.617 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-03-11T21:49:29.617 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-03-11T21:49:29.617 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-03-11T21:49:29.617 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:49:29.617 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:49:29.617 INFO:teuthology.orchestra.run.smithi092.stdout: "description": "MOSDPGPush(4.f 529/528 [PushOp(4:f1983ba0:::obj_delete_at_hint.0000000001:head, version: 456'29, data_included: [], data_size: 0, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(4:f1983ba0:::obj_delete_at_hint.0000000001:head@456'29, size: 0, copy_subset: [], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 0, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-03-11T21:49:29.617 INFO:teuthology.orchestra.run.smithi092.stdout: "initiated_at": "2024-03-11T21:49:17.178072+0000", 2024-03-11T21:49:29.617 INFO:teuthology.orchestra.run.smithi092.stdout: "age": 12.384151579999999, 2024-03-11T21:49:29.617 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00057603199999999998, 2024-03-11T21:49:29.617 INFO:teuthology.orchestra.run.smithi092.stdout: "type_data": { 2024-03-11T21:49:29.617 INFO:teuthology.orchestra.run.smithi092.stdout: "flag_point": "started", 2024-03-11T21:49:29.617 INFO:teuthology.orchestra.run.smithi092.stdout: "events": [ 2024-03-11T21:49:29.617 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:49:29.618 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "initiated", 2024-03-11T21:49:29.618 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:49:17.178072+0000", 2024-03-11T21:49:29.618 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-03-11T21:49:29.618 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:49:29.618 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:49:29.618 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "header_read", 2024-03-11T21:49:29.618 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:49:17.178072+0000", 2024-03-11T21:49:29.618 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-03-11T21:49:29.618 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:49:29.618 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:49:29.618 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "throttled", 2024-03-11T21:49:29.618 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:49:17.178073+0000", 2024-03-11T21:49:29.618 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 9.1999999999999998e-07 2024-03-11T21:49:29.618 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:49:29.618 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:49:29.619 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "all_read", 2024-03-11T21:49:29.619 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:49:17.178271+0000", 2024-03-11T21:49:29.619 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.000198142 2024-03-11T21:49:29.619 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:49:29.619 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:49:29.619 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "dispatched", 2024-03-11T21:49:29.619 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:49:17.178382+0000", 2024-03-11T21:49:29.619 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.000110643 2024-03-11T21:49:29.619 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:49:29.619 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:49:29.619 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "queued_for_pg", 2024-03-11T21:49:29.619 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:49:17.178403+0000", 2024-03-11T21:49:29.619 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 2.1276000000000001e-05 2024-03-11T21:49:29.619 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:49:29.620 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:49:29.620 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-03-11T21:49:29.620 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:49:17.178487+0000", 2024-03-11T21:49:29.620 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 8.3726000000000004e-05 2024-03-11T21:49:29.620 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:49:29.620 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:49:29.620 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "started", 2024-03-11T21:49:29.620 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:49:17.178499+0000", 2024-03-11T21:49:29.620 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.2323999999999999e-05 2024-03-11T21:49:29.620 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:49:29.620 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:49:29.620 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "done", 2024-03-11T21:49:29.620 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:49:17.178648+0000", 2024-03-11T21:49:29.620 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00014900100000000001 2024-03-11T21:49:29.620 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-03-11T21:49:29.621 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-03-11T21:49:29.621 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-03-11T21:49:29.621 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:49:29.621 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:49:29.621 INFO:teuthology.orchestra.run.smithi092.stdout: "description": "MOSDPGPush(4.b 529/528 [PushOp(4:d242335b:gc::gc.2:head, version: 74'3, data_included: [0~110], data_size: 110, omap_header_size: 0, omap_entries_size: 0, attrset_size: 4, recovery_info: ObjectRecoveryInfo(4:d242335b:gc::gc.2:head@74'3, size: 110, copy_subset: [(0, 110)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 110, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-03-11T21:49:29.621 INFO:teuthology.orchestra.run.smithi092.stdout: "initiated_at": "2024-03-11T21:49:17.187833+0000", 2024-03-11T21:49:29.621 INFO:teuthology.orchestra.run.smithi092.stdout: "age": 12.374390424, 2024-03-11T21:49:29.621 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00063863199999999998, 2024-03-11T21:49:29.621 INFO:teuthology.orchestra.run.smithi092.stdout: "type_data": { 2024-03-11T21:49:29.621 INFO:teuthology.orchestra.run.smithi092.stdout: "flag_point": "started", 2024-03-11T21:49:29.621 INFO:teuthology.orchestra.run.smithi092.stdout: "events": [ 2024-03-11T21:49:29.621 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:49:29.621 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "initiated", 2024-03-11T21:49:29.621 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:49:17.187833+0000", 2024-03-11T21:49:29.621 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-03-11T21:49:29.622 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:49:29.622 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:49:29.622 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "header_read", 2024-03-11T21:49:29.622 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:49:17.187833+0000", 2024-03-11T21:49:29.622 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-03-11T21:49:29.622 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:49:29.622 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:49:29.622 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "throttled", 2024-03-11T21:49:29.622 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:49:17.187834+0000", 2024-03-11T21:49:29.622 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 9.1900000000000001e-07 2024-03-11T21:49:29.622 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:49:29.622 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:49:29.622 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "all_read", 2024-03-11T21:49:29.623 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:49:17.187847+0000", 2024-03-11T21:49:29.623 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.2571999999999999e-05 2024-03-11T21:49:29.623 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:49:29.623 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:49:29.623 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "dispatched", 2024-03-11T21:49:29.623 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:49:17.188181+0000", 2024-03-11T21:49:29.623 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00033419999999999999 2024-03-11T21:49:29.623 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:49:29.623 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:49:29.623 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "queued_for_pg", 2024-03-11T21:49:29.623 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:49:17.188197+0000", 2024-03-11T21:49:29.623 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.6512999999999999e-05 2024-03-11T21:49:29.623 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:49:29.623 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:49:29.623 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-03-11T21:49:29.624 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:49:17.188265+0000", 2024-03-11T21:49:29.624 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 6.7958000000000002e-05 2024-03-11T21:49:29.624 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:49:29.624 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:49:29.624 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "started", 2024-03-11T21:49:29.624 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:49:17.188276+0000", 2024-03-11T21:49:29.624 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.0505e-05 2024-03-11T21:49:29.624 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:49:29.624 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:49:29.624 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "done", 2024-03-11T21:49:29.624 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:49:17.188472+0000", 2024-03-11T21:49:29.624 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00019596500000000001 2024-03-11T21:49:29.624 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-03-11T21:49:29.624 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-03-11T21:49:29.625 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-03-11T21:49:29.625 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:49:29.625 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:49:29.625 INFO:teuthology.orchestra.run.smithi092.stdout: "description": "MOSDPGPush(4.11 529/528 [PushOp(4:8eb99c53:gc::gc.22:head, version: 74'3, data_included: [0~110], data_size: 110, omap_header_size: 0, omap_entries_size: 0, attrset_size: 4, recovery_info: ObjectRecoveryInfo(4:8eb99c53:gc::gc.22:head@74'3, size: 110, copy_subset: [(0, 110)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 110, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-03-11T21:49:29.625 INFO:teuthology.orchestra.run.smithi092.stdout: "initiated_at": "2024-03-11T21:49:17.195288+0000", 2024-03-11T21:49:29.625 INFO:teuthology.orchestra.run.smithi092.stdout: "age": 12.366935283, 2024-03-11T21:49:29.625 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00049557400000000004, 2024-03-11T21:49:29.625 INFO:teuthology.orchestra.run.smithi092.stdout: "type_data": { 2024-03-11T21:49:29.625 INFO:teuthology.orchestra.run.smithi092.stdout: "flag_point": "started", 2024-03-11T21:49:29.625 INFO:teuthology.orchestra.run.smithi092.stdout: "events": [ 2024-03-11T21:49:29.625 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:49:29.625 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "initiated", 2024-03-11T21:49:29.625 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:49:17.195288+0000", 2024-03-11T21:49:29.625 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-03-11T21:49:29.626 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:49:29.626 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:49:29.626 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "header_read", 2024-03-11T21:49:29.626 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:49:17.195288+0000", 2024-03-11T21:49:29.626 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-03-11T21:49:29.626 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:49:29.626 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:49:29.626 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "throttled", 2024-03-11T21:49:29.626 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:49:17.195289+0000", 2024-03-11T21:49:29.626 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 9.8100000000000001e-07 2024-03-11T21:49:29.626 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:49:29.626 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:49:29.626 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "all_read", 2024-03-11T21:49:29.626 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:49:17.195299+0000", 2024-03-11T21:49:29.626 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 9.8779999999999993e-06 2024-03-11T21:49:29.627 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:49:29.627 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:49:29.627 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "dispatched", 2024-03-11T21:49:29.627 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:49:17.195314+0000", 2024-03-11T21:49:29.627 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.4348000000000001e-05 2024-03-11T21:49:29.627 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:49:29.627 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:49:29.627 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "queued_for_pg", 2024-03-11T21:49:29.627 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:49:17.195359+0000", 2024-03-11T21:49:29.627 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 4.5589e-05 2024-03-11T21:49:29.627 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:49:29.627 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:49:29.627 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-03-11T21:49:29.627 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:49:17.195464+0000", 2024-03-11T21:49:29.628 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00010469899999999999 2024-03-11T21:49:29.628 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:49:29.628 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:49:29.628 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "started", 2024-03-11T21:49:29.628 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:49:17.195478+0000", 2024-03-11T21:49:29.628 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.3767999999999999e-05 2024-03-11T21:49:29.628 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:49:29.628 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:49:29.628 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "done", 2024-03-11T21:49:29.628 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:49:17.195784+0000", 2024-03-11T21:49:29.628 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00030631099999999997 2024-03-11T21:49:29.628 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-03-11T21:49:29.628 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-03-11T21:49:29.628 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-03-11T21:49:29.628 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:49:29.629 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:49:29.629 INFO:teuthology.orchestra.run.smithi092.stdout: "description": "MOSDPGPush(4.1d 529/528 [PushOp(4:b88e3271:gc::gc.28:head, version: 74'8, data_included: [0~110], data_size: 110, omap_header_size: 0, omap_entries_size: 0, attrset_size: 4, recovery_info: ObjectRecoveryInfo(4:b88e3271:gc::gc.28:head@74'8, size: 110, copy_subset: [(0, 110)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 110, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-03-11T21:49:29.629 INFO:teuthology.orchestra.run.smithi092.stdout: "initiated_at": "2024-03-11T21:49:17.214576+0000", 2024-03-11T21:49:29.629 INFO:teuthology.orchestra.run.smithi092.stdout: "age": 12.347647238, 2024-03-11T21:49:29.629 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.000586273, 2024-03-11T21:49:29.629 INFO:teuthology.orchestra.run.smithi092.stdout: "type_data": { 2024-03-11T21:49:29.629 INFO:teuthology.orchestra.run.smithi092.stdout: "flag_point": "started", 2024-03-11T21:49:29.629 INFO:teuthology.orchestra.run.smithi092.stdout: "events": [ 2024-03-11T21:49:29.629 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:49:29.629 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "initiated", 2024-03-11T21:49:29.629 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:49:17.214576+0000", 2024-03-11T21:49:29.629 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-03-11T21:49:29.629 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:49:29.629 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:49:29.630 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "header_read", 2024-03-11T21:49:29.630 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:49:17.214576+0000", 2024-03-11T21:49:29.630 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-03-11T21:49:29.630 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:49:29.630 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:49:29.630 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "throttled", 2024-03-11T21:49:29.630 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:49:17.214577+0000", 2024-03-11T21:49:29.630 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 7.3900000000000004e-07 2024-03-11T21:49:29.630 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:49:29.630 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:49:29.630 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "all_read", 2024-03-11T21:49:29.630 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:49:17.214588+0000", 2024-03-11T21:49:29.630 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.1285e-05 2024-03-11T21:49:29.630 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:49:29.630 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:49:29.631 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "dispatched", 2024-03-11T21:49:29.631 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:49:17.214603+0000", 2024-03-11T21:49:29.631 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.4921e-05 2024-03-11T21:49:29.631 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:49:29.631 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:49:29.631 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "queued_for_pg", 2024-03-11T21:49:29.631 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:49:17.214618+0000", 2024-03-11T21:49:29.631 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.4214e-05 2024-03-11T21:49:29.631 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:49:29.631 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:49:29.631 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-03-11T21:49:29.631 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:49:17.214722+0000", 2024-03-11T21:49:29.631 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.000104261 2024-03-11T21:49:29.631 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:49:29.632 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:49:29.632 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "started", 2024-03-11T21:49:29.632 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:49:17.214743+0000", 2024-03-11T21:49:29.632 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 2.1271999999999999e-05 2024-03-11T21:49:29.632 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:49:29.632 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:49:29.632 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "done", 2024-03-11T21:49:29.632 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:49:17.215163+0000", 2024-03-11T21:49:29.632 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00041958099999999998 2024-03-11T21:49:29.632 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-03-11T21:49:29.632 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-03-11T21:49:29.632 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-03-11T21:49:29.632 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:49:29.632 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:49:29.632 INFO:teuthology.orchestra.run.smithi092.stdout: "description": "MOSDPGPush(4.c 529/528 [PushOp(4:31288c1b:::obj_delete_at_hint.0000000096:head, version: 456'30, data_included: [], data_size: 0, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(4:31288c1b:::obj_delete_at_hint.0000000096:head@456'30, size: 0, copy_subset: [], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 0, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-03-11T21:49:29.633 INFO:teuthology.orchestra.run.smithi092.stdout: "initiated_at": "2024-03-11T21:49:17.252794+0000", 2024-03-11T21:49:29.633 INFO:teuthology.orchestra.run.smithi092.stdout: "age": 12.30942933, 2024-03-11T21:49:29.633 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00057160800000000003, 2024-03-11T21:49:29.633 INFO:teuthology.orchestra.run.smithi092.stdout: "type_data": { 2024-03-11T21:49:29.633 INFO:teuthology.orchestra.run.smithi092.stdout: "flag_point": "started", 2024-03-11T21:49:29.633 INFO:teuthology.orchestra.run.smithi092.stdout: "events": [ 2024-03-11T21:49:29.633 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:49:29.633 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "initiated", 2024-03-11T21:49:29.633 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:49:17.252794+0000", 2024-03-11T21:49:29.633 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-03-11T21:49:29.633 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:49:29.633 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:49:29.633 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "header_read", 2024-03-11T21:49:29.633 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:49:17.252794+0000", 2024-03-11T21:49:29.633 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-03-11T21:49:29.634 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:49:29.634 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:49:29.634 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "throttled", 2024-03-11T21:49:29.634 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:49:17.252795+0000", 2024-03-11T21:49:29.634 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 6.6300000000000005e-07 2024-03-11T21:49:29.634 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:49:29.634 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:49:29.634 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "all_read", 2024-03-11T21:49:29.634 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:49:17.252804+0000", 2024-03-11T21:49:29.634 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 9.329e-06 2024-03-11T21:49:29.634 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:49:29.634 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:49:29.634 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "dispatched", 2024-03-11T21:49:29.634 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:49:17.252817+0000", 2024-03-11T21:49:29.634 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.2539e-05 2024-03-11T21:49:29.635 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:49:29.635 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:49:29.635 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "queued_for_pg", 2024-03-11T21:49:29.635 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:49:17.252848+0000", 2024-03-11T21:49:29.635 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 3.1409999999999999e-05 2024-03-11T21:49:29.635 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:49:29.635 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:49:29.635 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-03-11T21:49:29.635 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:49:17.253089+0000", 2024-03-11T21:49:29.635 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00024039999999999999 2024-03-11T21:49:29.635 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:49:29.635 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:49:29.635 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "started", 2024-03-11T21:49:29.635 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:49:17.253097+0000", 2024-03-11T21:49:29.635 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 8.3259999999999996e-06 2024-03-11T21:49:29.636 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:49:29.636 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:49:29.636 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "done", 2024-03-11T21:49:29.636 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:49:17.253366+0000", 2024-03-11T21:49:29.636 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00026894100000000003 2024-03-11T21:49:29.636 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-03-11T21:49:29.636 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-03-11T21:49:29.636 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-03-11T21:49:29.636 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:49:29.636 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:49:29.636 INFO:teuthology.orchestra.run.smithi092.stdout: "description": "MOSDPGPush(4.0 529/528 [PushOp(4:013d6c33:gc::gc.30:head, version: 93'20, data_included: [0~110], data_size: 110, omap_header_size: 0, omap_entries_size: 0, attrset_size: 4, recovery_info: ObjectRecoveryInfo(4:013d6c33:gc::gc.30:head@93'20, size: 110, copy_subset: [(0, 110)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 110, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-03-11T21:49:29.636 INFO:teuthology.orchestra.run.smithi092.stdout: "initiated_at": "2024-03-11T21:49:17.255027+0000", 2024-03-11T21:49:29.636 INFO:teuthology.orchestra.run.smithi092.stdout: "age": 12.307196586, 2024-03-11T21:49:29.636 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00054473, 2024-03-11T21:49:29.636 INFO:teuthology.orchestra.run.smithi092.stdout: "type_data": { 2024-03-11T21:49:29.637 INFO:teuthology.orchestra.run.smithi092.stdout: "flag_point": "started", 2024-03-11T21:49:29.637 INFO:teuthology.orchestra.run.smithi092.stdout: "events": [ 2024-03-11T21:49:29.637 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:49:29.637 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "initiated", 2024-03-11T21:49:29.637 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:49:17.255027+0000", 2024-03-11T21:49:29.637 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-03-11T21:49:29.637 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:49:29.637 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:49:29.637 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "header_read", 2024-03-11T21:49:29.637 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:49:17.255027+0000", 2024-03-11T21:49:29.637 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-03-11T21:49:29.637 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:49:29.637 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:49:29.637 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "throttled", 2024-03-11T21:49:29.637 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:49:17.255028+0000", 2024-03-11T21:49:29.638 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.071e-06 2024-03-11T21:49:29.638 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:49:29.638 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:49:29.638 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "all_read", 2024-03-11T21:49:29.638 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:49:17.255040+0000", 2024-03-11T21:49:29.638 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.1637999999999999e-05 2024-03-11T21:49:29.638 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:49:29.638 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:49:29.638 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "dispatched", 2024-03-11T21:49:29.638 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:49:17.255058+0000", 2024-03-11T21:49:29.638 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.8040999999999999e-05 2024-03-11T21:49:29.638 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:49:29.639 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:49:29.639 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "queued_for_pg", 2024-03-11T21:49:29.639 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:49:17.255071+0000", 2024-03-11T21:49:29.639 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.3413e-05 2024-03-11T21:49:29.639 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:49:29.639 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:49:29.639 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-03-11T21:49:29.639 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:49:17.255163+0000", 2024-03-11T21:49:29.639 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 9.1360999999999995e-05 2024-03-11T21:49:29.639 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:49:29.639 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:49:29.639 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "started", 2024-03-11T21:49:29.639 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:49:17.255176+0000", 2024-03-11T21:49:29.639 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.3737000000000001e-05 2024-03-11T21:49:29.639 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:49:29.640 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:49:29.640 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "done", 2024-03-11T21:49:29.640 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:49:17.255572+0000", 2024-03-11T21:49:29.640 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00039546900000000002 2024-03-11T21:49:29.640 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-03-11T21:49:29.640 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-03-11T21:49:29.640 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-03-11T21:49:29.640 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:49:29.640 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:49:29.640 INFO:teuthology.orchestra.run.smithi092.stdout: "description": "MOSDPGPush(4.19 529/528 [PushOp(4:9a6dd8c0:gc::gc.0:head, version: 74'3, data_included: [0~110], data_size: 110, omap_header_size: 0, omap_entries_size: 0, attrset_size: 4, recovery_info: ObjectRecoveryInfo(4:9a6dd8c0:gc::gc.0:head@74'3, size: 110, copy_subset: [(0, 110)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 110, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-03-11T21:49:29.640 INFO:teuthology.orchestra.run.smithi092.stdout: "initiated_at": "2024-03-11T21:49:17.255740+0000", 2024-03-11T21:49:29.640 INFO:teuthology.orchestra.run.smithi092.stdout: "age": 12.306483918, 2024-03-11T21:49:29.640 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00051114599999999997, 2024-03-11T21:49:29.640 INFO:teuthology.orchestra.run.smithi092.stdout: "type_data": { 2024-03-11T21:49:29.641 INFO:teuthology.orchestra.run.smithi092.stdout: "flag_point": "started", 2024-03-11T21:49:29.641 INFO:teuthology.orchestra.run.smithi092.stdout: "events": [ 2024-03-11T21:49:29.641 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:49:29.641 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "initiated", 2024-03-11T21:49:29.641 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:49:17.255740+0000", 2024-03-11T21:49:29.641 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-03-11T21:49:29.641 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:49:29.641 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:49:29.641 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "header_read", 2024-03-11T21:49:29.641 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:49:17.255740+0000", 2024-03-11T21:49:29.641 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-03-11T21:49:29.641 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:49:29.641 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:49:29.641 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "throttled", 2024-03-11T21:49:29.641 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:49:17.255741+0000", 2024-03-11T21:49:29.642 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 9.5300000000000002e-07 2024-03-11T21:49:29.642 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:49:29.642 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:49:29.642 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "all_read", 2024-03-11T21:49:29.642 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:49:17.255753+0000", 2024-03-11T21:49:29.642 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.2296000000000001e-05 2024-03-11T21:49:29.642 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:49:29.642 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:49:29.642 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "dispatched", 2024-03-11T21:49:29.642 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:49:17.255768+0000", 2024-03-11T21:49:29.642 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.4628e-05 2024-03-11T21:49:29.642 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:49:29.642 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:49:29.642 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "queued_for_pg", 2024-03-11T21:49:29.643 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:49:17.255780+0000", 2024-03-11T21:49:29.643 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.2862e-05 2024-03-11T21:49:29.643 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:49:29.643 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:49:29.643 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-03-11T21:49:29.643 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:49:17.255848+0000", 2024-03-11T21:49:29.643 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 6.7496999999999994e-05 2024-03-11T21:49:29.643 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:49:29.643 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:49:29.643 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "started", 2024-03-11T21:49:29.643 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:49:17.255860+0000", 2024-03-11T21:49:29.643 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.2201000000000001e-05 2024-03-11T21:49:29.643 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:49:29.643 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:49:29.644 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "done", 2024-03-11T21:49:29.644 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:49:17.256251+0000", 2024-03-11T21:49:29.644 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00039070899999999999 2024-03-11T21:49:29.644 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-03-11T21:49:29.644 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-03-11T21:49:29.644 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-03-11T21:49:29.644 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:49:29.644 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:49:29.644 INFO:teuthology.orchestra.run.smithi092.stdout: "description": "osd_op(client.34352.0:4666 5.0 5:05bf5b68:::notify.1:head [watch reconnect cookie 93924550614912 gen 7] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e530)", 2024-03-11T21:49:29.644 INFO:teuthology.orchestra.run.smithi092.stdout: "initiated_at": "2024-03-11T21:49:18.876728+0000", 2024-03-11T21:49:29.644 INFO:teuthology.orchestra.run.smithi092.stdout: "age": 10.68549608, 2024-03-11T21:49:29.644 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.0028870670000000001, 2024-03-11T21:49:29.644 INFO:teuthology.orchestra.run.smithi092.stdout: "type_data": { 2024-03-11T21:49:29.644 INFO:teuthology.orchestra.run.smithi092.stdout: "flag_point": "started", 2024-03-11T21:49:29.644 INFO:teuthology.orchestra.run.smithi092.stdout: "client_info": { 2024-03-11T21:49:29.645 INFO:teuthology.orchestra.run.smithi092.stdout: "client": "client.34352", 2024-03-11T21:49:29.645 INFO:teuthology.orchestra.run.smithi092.stdout: "client_addr": "v1:172.21.15.92:0/372197479", 2024-03-11T21:49:29.645 INFO:teuthology.orchestra.run.smithi092.stdout: "tid": 4666 2024-03-11T21:49:29.645 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:49:29.645 INFO:teuthology.orchestra.run.smithi092.stdout: "events": [ 2024-03-11T21:49:29.645 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:49:29.645 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "initiated", 2024-03-11T21:49:29.645 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:49:18.876728+0000", 2024-03-11T21:49:29.645 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-03-11T21:49:29.645 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:49:29.645 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:49:29.645 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "header_read", 2024-03-11T21:49:29.645 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:49:18.876728+0000", 2024-03-11T21:49:29.645 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-03-11T21:49:29.645 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:49:29.646 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:49:29.646 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "throttled", 2024-03-11T21:49:29.646 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:49:18.876730+0000", 2024-03-11T21:49:29.646 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 2.3599999999999999e-06 2024-03-11T21:49:29.646 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:49:29.646 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:49:29.646 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "all_read", 2024-03-11T21:49:29.646 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:49:18.876738+0000", 2024-03-11T21:49:29.646 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 7.6070000000000002e-06 2024-03-11T21:49:29.646 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:49:29.646 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:49:29.646 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "dispatched", 2024-03-11T21:49:29.646 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:49:18.876756+0000", 2024-03-11T21:49:29.646 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.8646000000000001e-05 2024-03-11T21:49:29.647 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:49:29.647 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:49:29.647 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "queued_for_pg", 2024-03-11T21:49:29.647 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:49:18.876767+0000", 2024-03-11T21:49:29.647 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.1168999999999999e-05 2024-03-11T21:49:29.647 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:49:29.647 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:49:29.647 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-03-11T21:49:29.647 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:49:18.876853+0000", 2024-03-11T21:49:29.647 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 8.5257e-05 2024-03-11T21:49:29.647 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:49:29.647 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:49:29.647 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "op must wait for map", 2024-03-11T21:49:29.647 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:49:18.876860+0000", 2024-03-11T21:49:29.647 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 7.0929999999999997e-06 2024-03-11T21:49:29.648 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:49:29.648 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:49:29.648 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-03-11T21:49:29.648 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:49:18.878112+0000", 2024-03-11T21:49:29.648 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.0012518309999999999 2024-03-11T21:49:29.648 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:49:29.648 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:49:29.648 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "started", 2024-03-11T21:49:29.648 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:49:18.878206+0000", 2024-03-11T21:49:29.648 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 9.4871999999999996e-05 2024-03-11T21:49:29.648 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:49:29.648 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:49:29.648 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "done", 2024-03-11T21:49:29.648 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:49:18.879615+0000", 2024-03-11T21:49:29.648 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.0014082319999999999 2024-03-11T21:49:29.649 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-03-11T21:49:29.649 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-03-11T21:49:29.649 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-03-11T21:49:29.649 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:49:29.649 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:49:29.649 INFO:teuthology.orchestra.run.smithi092.stdout: "description": "osd_op(client.34352.0:4668 5.14 5:2b04a3e9:::notify.6:head [watch reconnect cookie 93924550609152 gen 8] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e530)", 2024-03-11T21:49:29.649 INFO:teuthology.orchestra.run.smithi092.stdout: "initiated_at": "2024-03-11T21:49:18.876803+0000", 2024-03-11T21:49:29.649 INFO:teuthology.orchestra.run.smithi092.stdout: "age": 10.685420819000001, 2024-03-11T21:49:29.649 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.0031826530000000001, 2024-03-11T21:49:29.649 INFO:teuthology.orchestra.run.smithi092.stdout: "type_data": { 2024-03-11T21:49:29.649 INFO:teuthology.orchestra.run.smithi092.stdout: "flag_point": "started", 2024-03-11T21:49:29.649 INFO:teuthology.orchestra.run.smithi092.stdout: "client_info": { 2024-03-11T21:49:29.649 INFO:teuthology.orchestra.run.smithi092.stdout: "client": "client.34352", 2024-03-11T21:49:29.649 INFO:teuthology.orchestra.run.smithi092.stdout: "client_addr": "v1:172.21.15.92:0/372197479", 2024-03-11T21:49:29.649 INFO:teuthology.orchestra.run.smithi092.stdout: "tid": 4668 2024-03-11T21:49:29.650 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:49:29.650 INFO:teuthology.orchestra.run.smithi092.stdout: "events": [ 2024-03-11T21:49:29.650 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:49:29.650 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "initiated", 2024-03-11T21:49:29.650 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:49:18.876803+0000", 2024-03-11T21:49:29.650 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-03-11T21:49:29.650 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:49:29.650 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:49:29.650 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "header_read", 2024-03-11T21:49:29.650 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:49:18.876803+0000", 2024-03-11T21:49:29.650 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-03-11T21:49:29.650 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:49:29.650 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:49:29.650 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "throttled", 2024-03-11T21:49:29.651 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:49:18.876804+0000", 2024-03-11T21:49:29.651 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 8.47e-07 2024-03-11T21:49:29.651 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:49:29.651 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:49:29.651 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "all_read", 2024-03-11T21:49:29.651 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:49:18.876808+0000", 2024-03-11T21:49:29.651 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 3.8709999999999999e-06 2024-03-11T21:49:29.651 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:49:29.651 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:49:29.651 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "dispatched", 2024-03-11T21:49:29.651 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:49:18.876823+0000", 2024-03-11T21:49:29.651 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.5089000000000001e-05 2024-03-11T21:49:29.651 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:49:29.651 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:49:29.652 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "queued_for_pg", 2024-03-11T21:49:29.652 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:49:18.876830+0000", 2024-03-11T21:49:29.652 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 7.1049999999999997e-06 2024-03-11T21:49:29.652 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:49:29.652 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:49:29.652 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-03-11T21:49:29.652 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:49:18.876896+0000", 2024-03-11T21:49:29.652 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 6.6624000000000005e-05 2024-03-11T21:49:29.652 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:49:29.652 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:49:29.652 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "op must wait for map", 2024-03-11T21:49:29.652 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:49:18.876901+0000", 2024-03-11T21:49:29.652 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 4.2690000000000001e-06 2024-03-11T21:49:29.652 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:49:29.652 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:49:29.653 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-03-11T21:49:29.653 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:49:18.878914+0000", 2024-03-11T21:49:29.653 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.002013028 2024-03-11T21:49:29.653 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:49:29.653 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:49:29.653 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "started", 2024-03-11T21:49:29.653 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:49:18.878990+0000", 2024-03-11T21:49:29.653 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 7.6469000000000003e-05 2024-03-11T21:49:29.653 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:49:29.653 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:49:29.653 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "done", 2024-03-11T21:49:29.653 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:49:18.879985+0000", 2024-03-11T21:49:29.653 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.000995351 2024-03-11T21:49:29.653 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-03-11T21:49:29.653 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-03-11T21:49:29.654 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-03-11T21:49:29.654 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:49:29.654 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:49:29.654 INFO:teuthology.orchestra.run.smithi092.stdout: "description": "osd_op(client.34352.0:4689 5.0 5:05bf5b68:::notify.1:head [watch reconnect cookie 93924550614912 gen 8] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e533)", 2024-03-11T21:49:29.654 INFO:teuthology.orchestra.run.smithi092.stdout: "initiated_at": "2024-03-11T21:49:28.878741+0000", 2024-03-11T21:49:29.654 INFO:teuthology.orchestra.run.smithi092.stdout: "age": 0.68348214699999998, 2024-03-11T21:49:29.654 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.0016401569999999999, 2024-03-11T21:49:29.654 INFO:teuthology.orchestra.run.smithi092.stdout: "type_data": { 2024-03-11T21:49:29.654 INFO:teuthology.orchestra.run.smithi092.stdout: "flag_point": "started", 2024-03-11T21:49:29.654 INFO:teuthology.orchestra.run.smithi092.stdout: "client_info": { 2024-03-11T21:49:29.654 INFO:teuthology.orchestra.run.smithi092.stdout: "client": "client.34352", 2024-03-11T21:49:29.654 INFO:teuthology.orchestra.run.smithi092.stdout: "client_addr": "v1:172.21.15.92:0/372197479", 2024-03-11T21:49:29.654 INFO:teuthology.orchestra.run.smithi092.stdout: "tid": 4689 2024-03-11T21:49:29.654 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:49:29.655 INFO:teuthology.orchestra.run.smithi092.stdout: "events": [ 2024-03-11T21:49:29.655 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:49:29.655 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "initiated", 2024-03-11T21:49:29.655 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:49:28.878741+0000", 2024-03-11T21:49:29.655 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-03-11T21:49:29.655 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:49:29.655 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:49:29.655 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "header_read", 2024-03-11T21:49:29.655 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:49:28.878741+0000", 2024-03-11T21:49:29.655 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0 2024-03-11T21:49:29.655 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:49:29.655 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:49:29.655 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "throttled", 2024-03-11T21:49:29.655 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:49:28.878743+0000", 2024-03-11T21:49:29.655 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 1.1930000000000001e-06 2024-03-11T21:49:29.656 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:49:29.656 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:49:29.656 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "all_read", 2024-03-11T21:49:29.656 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:49:28.878747+0000", 2024-03-11T21:49:29.656 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 4.0849999999999999e-06 2024-03-11T21:49:29.656 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:49:29.656 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:49:29.656 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "dispatched", 2024-03-11T21:49:29.656 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:49:28.878756+0000", 2024-03-11T21:49:29.656 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 9.2650000000000002e-06 2024-03-11T21:49:29.656 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:49:29.656 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:49:29.656 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "queued_for_pg", 2024-03-11T21:49:29.656 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:49:28.878763+0000", 2024-03-11T21:49:29.656 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 6.9990000000000002e-06 2024-03-11T21:49:29.657 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:49:29.657 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:49:29.657 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "reached_pg", 2024-03-11T21:49:29.657 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:49:28.878819+0000", 2024-03-11T21:49:29.657 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 5.6103999999999997e-05 2024-03-11T21:49:29.657 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:49:29.657 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:49:29.657 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "started", 2024-03-11T21:49:29.657 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:49:28.878886+0000", 2024-03-11T21:49:29.657 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 6.7291000000000003e-05 2024-03-11T21:49:29.657 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2024-03-11T21:49:29.657 INFO:teuthology.orchestra.run.smithi092.stdout: { 2024-03-11T21:49:29.657 INFO:teuthology.orchestra.run.smithi092.stdout: "event": "done", 2024-03-11T21:49:29.657 INFO:teuthology.orchestra.run.smithi092.stdout: "time": "2024-03-11T21:49:28.880382+0000", 2024-03-11T21:49:29.657 INFO:teuthology.orchestra.run.smithi092.stdout: "duration": 0.00149522 2024-03-11T21:49:29.658 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-03-11T21:49:29.658 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-03-11T21:49:29.658 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-03-11T21:49:29.658 INFO:teuthology.orchestra.run.smithi092.stdout: } 2024-03-11T21:49:29.658 INFO:teuthology.orchestra.run.smithi092.stdout: ] 2024-03-11T21:49:29.658 INFO:teuthology.orchestra.run.smithi092.stdout:} 2024-03-11T21:49:30.414 INFO:teuthology.orchestra.run.smithi092.stdout:mon.a: {} 2024-03-11T21:49:30.415 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:49:30 smithi092 ceph-mon[29572]: 5.1 deep-scrub starts 2024-03-11T21:49:30.415 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:49:30 smithi092 ceph-mon[29572]: 5.1 deep-scrub ok 2024-03-11T21:49:30.415 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:49:30 smithi092 ceph-mon[37563]: 5.1 deep-scrub starts 2024-03-11T21:49:30.415 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:49:30 smithi092 ceph-mon[37563]: 5.1 deep-scrub ok 2024-03-11T21:49:30.425 INFO:teuthology.orchestra.run.smithi092.stdout:mon.b: {} 2024-03-11T21:49:30.437 INFO:teuthology.orchestra.run.smithi092.stdout:mon.c: {} 2024-03-11T21:49:30.438 INFO:teuthology.orchestra.run.smithi092.stderr:mon.a: mon_osd_down_out_interval = '' (not observed, change may require restart) 2024-03-11T21:49:30.438 INFO:teuthology.orchestra.run.smithi092.stderr:mon.b: mon_osd_down_out_interval = '' (not observed, change may require restart) 2024-03-11T21:49:30.438 INFO:teuthology.orchestra.run.smithi092.stderr:mon.c: mon_osd_down_out_interval = '' (not observed, change may require restart) 2024-03-11T21:49:30.585 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:49:30 smithi097 ceph-mon[33317]: 5.1 deep-scrub starts 2024-03-11T21:49:30.585 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:49:30 smithi097 ceph-mon[33317]: 5.1 deep-scrub ok 2024-03-11T21:49:31.220 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:49:31 smithi092 ceph-mon[29572]: pgmap v1143: 132 pgs: 132 active+clean; 582 KiB data, 6.3 GiB used, 709 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s; 178 B/s, 6 objects/s recovering 2024-03-11T21:49:31.220 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:49:31 smithi092 ceph-mon[29572]: 6.16 scrub starts 2024-03-11T21:49:31.221 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:49:31 smithi092 ceph-mon[29572]: 6.16 scrub ok 2024-03-11T21:49:31.221 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:49:31 smithi092 ceph-mon[29572]: from='client.? v1:172.21.15.92:0/3651862094' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-03-11T21:49:31.221 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:49:31 smithi092 ceph-mon[37563]: pgmap v1143: 132 pgs: 132 active+clean; 582 KiB data, 6.3 GiB used, 709 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s; 178 B/s, 6 objects/s recovering 2024-03-11T21:49:31.221 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:49:31 smithi092 ceph-mon[37563]: 6.16 scrub starts 2024-03-11T21:49:31.221 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:49:31 smithi092 ceph-mon[37563]: 6.16 scrub ok 2024-03-11T21:49:31.221 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:49:31 smithi092 ceph-mon[37563]: from='client.? v1:172.21.15.92:0/3651862094' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-03-11T21:49:31.585 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:49:31 smithi097 ceph-mon[33317]: pgmap v1143: 132 pgs: 132 active+clean; 582 KiB data, 6.3 GiB used, 709 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s; 178 B/s, 6 objects/s recovering 2024-03-11T21:49:31.586 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:49:31 smithi097 ceph-mon[33317]: 6.16 scrub starts 2024-03-11T21:49:31.586 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:49:31 smithi097 ceph-mon[33317]: 6.16 scrub ok 2024-03-11T21:49:31.586 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:49:31 smithi097 ceph-mon[33317]: from='client.? v1:172.21.15.92:0/3651862094' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-03-11T21:49:31.768 DEBUG:teuthology.orchestra.run.smithi092:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid 00db0016-dfed-11ee-95c8-87774f69a715 -- ceph osd reweight 0 1 2024-03-11T21:49:32.014 INFO:teuthology.orchestra.run.smithi092.stderr:Inferring config /var/lib/ceph/00db0016-dfed-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T21:49:32.448 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:49:32 smithi092 ceph-mon[29572]: 4.6 deep-scrub starts 2024-03-11T21:49:32.448 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:49:32 smithi092 ceph-mon[29572]: 4.6 deep-scrub ok 2024-03-11T21:49:32.448 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:49:32 smithi092 ceph-mon[37563]: 4.6 deep-scrub starts 2024-03-11T21:49:32.448 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:49:32 smithi092 ceph-mon[37563]: 4.6 deep-scrub ok 2024-03-11T21:49:32.585 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:49:32 smithi097 ceph-mon[33317]: 4.6 deep-scrub starts 2024-03-11T21:49:32.585 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:49:32 smithi097 ceph-mon[33317]: 4.6 deep-scrub ok 2024-03-11T21:49:33.441 INFO:journalctl@ceph.iscsi.iscsi.a.smithi097.stdout:Mar 11 21:49:33 smithi097 ceph-00db0016-dfed-11ee-95c8-87774f69a715-iscsi-iscsi-a[60943]: debug there is no tcmu-runner data available 2024-03-11T21:49:33.441 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:49:33 smithi097 ceph-mon[33317]: pgmap v1144: 132 pgs: 132 active+clean; 582 KiB data, 6.3 GiB used, 709 GiB / 715 GiB avail; 147 B/s rd, 0 op/s; 202 B/s, 7 objects/s recovering 2024-03-11T21:49:33.441 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:49:33 smithi097 ceph-mon[33317]: 2.a scrub starts 2024-03-11T21:49:33.441 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:49:33 smithi097 ceph-mon[33317]: 2.a scrub ok 2024-03-11T21:49:33.487 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:49:33 smithi092 ceph-mon[29572]: pgmap v1144: 132 pgs: 132 active+clean; 582 KiB data, 6.3 GiB used, 709 GiB / 715 GiB avail; 147 B/s rd, 0 op/s; 202 B/s, 7 objects/s recovering 2024-03-11T21:49:33.487 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:49:33 smithi092 ceph-mon[29572]: 2.a scrub starts 2024-03-11T21:49:33.487 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:49:33 smithi092 ceph-mon[29572]: 2.a scrub ok 2024-03-11T21:49:33.487 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:49:33 smithi092 ceph-mon[37563]: pgmap v1144: 132 pgs: 132 active+clean; 582 KiB data, 6.3 GiB used, 709 GiB / 715 GiB avail; 147 B/s rd, 0 op/s; 202 B/s, 7 objects/s recovering 2024-03-11T21:49:33.487 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:49:33 smithi092 ceph-mon[37563]: 2.a scrub starts 2024-03-11T21:49:33.488 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:49:33 smithi092 ceph-mon[37563]: 2.a scrub ok 2024-03-11T21:49:34.585 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:49:34 smithi097 ceph-mon[33317]: 4.5 deep-scrub starts 2024-03-11T21:49:34.586 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:49:34 smithi097 ceph-mon[33317]: 4.5 deep-scrub ok 2024-03-11T21:49:34.586 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:49:34 smithi097 ceph-mon[33317]: 4.13 scrub starts 2024-03-11T21:49:34.586 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:49:34 smithi097 ceph-mon[33317]: 4.13 scrub ok 2024-03-11T21:49:34.586 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:49:34 smithi097 ceph-mon[33317]: from='client.24469 v1:172.21.15.97:0/404208310' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:49:34.586 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:49:34 smithi097 ceph-mon[33317]: 4.10 scrub starts 2024-03-11T21:49:34.586 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:49:34 smithi097 ceph-mon[33317]: 4.10 scrub ok 2024-03-11T21:49:34.586 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:49:34 smithi097 ceph-mon[33317]: from='client.? v1:172.21.15.92:0/422133225' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]: dispatch 2024-03-11T21:49:34.608 INFO:teuthology.orchestra.run.smithi092.stderr:reweighted osd.0 to 1 (10000) 2024-03-11T21:49:34.610 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:49:34 smithi092 ceph-mon[29572]: 4.5 deep-scrub starts 2024-03-11T21:49:34.610 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:49:34 smithi092 ceph-mon[29572]: 4.5 deep-scrub ok 2024-03-11T21:49:34.610 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:49:34 smithi092 ceph-mon[29572]: 4.13 scrub starts 2024-03-11T21:49:34.610 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:49:34 smithi092 ceph-mon[29572]: 4.13 scrub ok 2024-03-11T21:49:34.610 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:49:34 smithi092 ceph-mon[29572]: from='client.24469 v1:172.21.15.97:0/404208310' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:49:34.611 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:49:34 smithi092 ceph-mon[29572]: 4.10 scrub starts 2024-03-11T21:49:34.611 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:49:34 smithi092 ceph-mon[29572]: 4.10 scrub ok 2024-03-11T21:49:34.611 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:49:34 smithi092 ceph-mon[29572]: from='client.? v1:172.21.15.92:0/422133225' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]: dispatch 2024-03-11T21:49:34.611 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:49:34 smithi092 ceph-mon[37563]: 4.5 deep-scrub starts 2024-03-11T21:49:34.611 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:49:34 smithi092 ceph-mon[37563]: 4.5 deep-scrub ok 2024-03-11T21:49:34.612 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:49:34 smithi092 ceph-mon[37563]: 4.13 scrub starts 2024-03-11T21:49:34.612 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:49:34 smithi092 ceph-mon[37563]: 4.13 scrub ok 2024-03-11T21:49:34.612 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:49:34 smithi092 ceph-mon[37563]: from='client.24469 v1:172.21.15.97:0/404208310' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:49:34.612 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:49:34 smithi092 ceph-mon[37563]: 4.10 scrub starts 2024-03-11T21:49:34.612 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:49:34 smithi092 ceph-mon[37563]: 4.10 scrub ok 2024-03-11T21:49:34.612 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:49:34 smithi092 ceph-mon[37563]: from='client.? v1:172.21.15.92:0/422133225' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]: dispatch 2024-03-11T21:49:35.352 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:49:35 smithi092 ceph-mon[29572]: pgmap v1145: 132 pgs: 132 active+clean; 582 KiB data, 6.3 GiB used, 709 GiB / 715 GiB avail; 127 B/s rd, 0 op/s; 175 B/s, 6 objects/s recovering 2024-03-11T21:49:35.352 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:49:35 smithi092 ceph-mon[29572]: 4.e scrub starts 2024-03-11T21:49:35.352 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:49:35 smithi092 ceph-mon[29572]: 4.e scrub ok 2024-03-11T21:49:35.352 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:49:35 smithi092 ceph-mon[29572]: 4.3 scrub starts 2024-03-11T21:49:35.353 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:49:35 smithi092 ceph-mon[29572]: 4.3 scrub ok 2024-03-11T21:49:35.353 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:49:35 smithi092 ceph-mon[29572]: from='client.? v1:172.21.15.92:0/422133225' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]': finished 2024-03-11T21:49:35.353 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:49:35 smithi092 ceph-mon[29572]: osdmap e534: 8 total, 8 up, 8 in 2024-03-11T21:49:35.353 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:49:35 smithi092 ceph-mon[29572]: 5.10 deep-scrub starts 2024-03-11T21:49:35.353 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:49:35 smithi092 ceph-mon[29572]: 5.10 deep-scrub ok 2024-03-11T21:49:35.353 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:49:35 smithi092 ceph-mon[29572]: 5.0 scrub starts 2024-03-11T21:49:35.353 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:49:35 smithi092 ceph-mon[29572]: 5.0 scrub ok 2024-03-11T21:49:35.353 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:49:35 smithi092 ceph-mon[37563]: pgmap v1145: 132 pgs: 132 active+clean; 582 KiB data, 6.3 GiB used, 709 GiB / 715 GiB avail; 127 B/s rd, 0 op/s; 175 B/s, 6 objects/s recovering 2024-03-11T21:49:35.353 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:49:35 smithi092 ceph-mon[37563]: 4.e scrub starts 2024-03-11T21:49:35.353 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:49:35 smithi092 ceph-mon[37563]: 4.e scrub ok 2024-03-11T21:49:35.353 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:49:35 smithi092 ceph-mon[37563]: 4.3 scrub starts 2024-03-11T21:49:35.354 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:49:35 smithi092 ceph-mon[37563]: 4.3 scrub ok 2024-03-11T21:49:35.354 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:49:35 smithi092 ceph-mon[37563]: from='client.? v1:172.21.15.92:0/422133225' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]': finished 2024-03-11T21:49:35.354 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:49:35 smithi092 ceph-mon[37563]: osdmap e534: 8 total, 8 up, 8 in 2024-03-11T21:49:35.354 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:49:35 smithi092 ceph-mon[37563]: 5.10 deep-scrub starts 2024-03-11T21:49:35.354 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:49:35 smithi092 ceph-mon[37563]: 5.10 deep-scrub ok 2024-03-11T21:49:35.354 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:49:35 smithi092 ceph-mon[37563]: 5.0 scrub starts 2024-03-11T21:49:35.354 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:49:35 smithi092 ceph-mon[37563]: 5.0 scrub ok 2024-03-11T21:49:35.384 DEBUG:teuthology.orchestra.run.smithi092:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid 00db0016-dfed-11ee-95c8-87774f69a715 -- ceph osd primary-affinity 0 1 2024-03-11T21:49:35.585 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:49:35 smithi097 ceph-mon[33317]: pgmap v1145: 132 pgs: 132 active+clean; 582 KiB data, 6.3 GiB used, 709 GiB / 715 GiB avail; 127 B/s rd, 0 op/s; 175 B/s, 6 objects/s recovering 2024-03-11T21:49:35.586 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:49:35 smithi097 ceph-mon[33317]: 4.e scrub starts 2024-03-11T21:49:35.586 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:49:35 smithi097 ceph-mon[33317]: 4.e scrub ok 2024-03-11T21:49:35.586 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:49:35 smithi097 ceph-mon[33317]: 4.3 scrub starts 2024-03-11T21:49:35.586 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:49:35 smithi097 ceph-mon[33317]: 4.3 scrub ok 2024-03-11T21:49:35.586 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:49:35 smithi097 ceph-mon[33317]: from='client.? v1:172.21.15.92:0/422133225' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]': finished 2024-03-11T21:49:35.586 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:49:35 smithi097 ceph-mon[33317]: osdmap e534: 8 total, 8 up, 8 in 2024-03-11T21:49:35.586 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:49:35 smithi097 ceph-mon[33317]: 5.10 deep-scrub starts 2024-03-11T21:49:35.586 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:49:35 smithi097 ceph-mon[33317]: 5.10 deep-scrub ok 2024-03-11T21:49:35.586 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:49:35 smithi097 ceph-mon[33317]: 5.0 scrub starts 2024-03-11T21:49:35.586 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:49:35 smithi097 ceph-mon[33317]: 5.0 scrub ok 2024-03-11T21:49:35.628 INFO:teuthology.orchestra.run.smithi092.stderr:Inferring config /var/lib/ceph/00db0016-dfed-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T21:49:36.370 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:49:36 smithi092 ceph-mon[29572]: 5.1b scrub starts 2024-03-11T21:49:36.370 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:49:36 smithi092 ceph-mon[29572]: 5.1b scrub ok 2024-03-11T21:49:36.370 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:49:36 smithi092 ceph-mon[29572]: 6.0 scrub starts 2024-03-11T21:49:36.370 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:49:36 smithi092 ceph-mon[29572]: 6.0 scrub ok 2024-03-11T21:49:36.370 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:49:36 smithi092 ceph-mon[29572]: 2.1d scrub starts 2024-03-11T21:49:36.370 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:49:36 smithi092 ceph-mon[29572]: 2.1d scrub ok 2024-03-11T21:49:36.370 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:49:36 smithi092 ceph-mon[37563]: 5.1b scrub starts 2024-03-11T21:49:36.370 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:49:36 smithi092 ceph-mon[37563]: 5.1b scrub ok 2024-03-11T21:49:36.370 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:49:36 smithi092 ceph-mon[37563]: 6.0 scrub starts 2024-03-11T21:49:36.371 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:49:36 smithi092 ceph-mon[37563]: 6.0 scrub ok 2024-03-11T21:49:36.371 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:49:36 smithi092 ceph-mon[37563]: 2.1d scrub starts 2024-03-11T21:49:36.371 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:49:36 smithi092 ceph-mon[37563]: 2.1d scrub ok 2024-03-11T21:49:36.586 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:49:36 smithi097 ceph-mon[33317]: 5.1b scrub starts 2024-03-11T21:49:36.586 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:49:36 smithi097 ceph-mon[33317]: 5.1b scrub ok 2024-03-11T21:49:36.586 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:49:36 smithi097 ceph-mon[33317]: 6.0 scrub starts 2024-03-11T21:49:36.586 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:49:36 smithi097 ceph-mon[33317]: 6.0 scrub ok 2024-03-11T21:49:36.586 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:49:36 smithi097 ceph-mon[33317]: 2.1d scrub starts 2024-03-11T21:49:36.586 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:49:36 smithi097 ceph-mon[33317]: 2.1d scrub ok 2024-03-11T21:49:37.370 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:49:37 smithi092 ceph-mon[29572]: pgmap v1147: 132 pgs: 132 active+clean; 582 KiB data, 6.3 GiB used, 709 GiB / 715 GiB avail; 688 B/s rd, 0 op/s; 157 B/s, 5 objects/s recovering 2024-03-11T21:49:37.370 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:49:37 smithi092 ceph-mon[29572]: 2.14 scrub starts 2024-03-11T21:49:37.370 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:49:37 smithi092 ceph-mon[29572]: 2.14 scrub ok 2024-03-11T21:49:37.370 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:49:37 smithi092 ceph-mon[29572]: 6.1c scrub starts 2024-03-11T21:49:37.371 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:49:37 smithi092 ceph-mon[29572]: 6.1c scrub ok 2024-03-11T21:49:37.371 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:49:37 smithi092 ceph-mon[29572]: 4.14 deep-scrub starts 2024-03-11T21:49:37.371 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:49:37 smithi092 ceph-mon[29572]: 4.14 deep-scrub ok 2024-03-11T21:49:37.371 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:49:37 smithi092 ceph-mon[37563]: pgmap v1147: 132 pgs: 132 active+clean; 582 KiB data, 6.3 GiB used, 709 GiB / 715 GiB avail; 688 B/s rd, 0 op/s; 157 B/s, 5 objects/s recovering 2024-03-11T21:49:37.371 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:49:37 smithi092 ceph-mon[37563]: 2.14 scrub starts 2024-03-11T21:49:37.371 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:49:37 smithi092 ceph-mon[37563]: 2.14 scrub ok 2024-03-11T21:49:37.371 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:49:37 smithi092 ceph-mon[37563]: 6.1c scrub starts 2024-03-11T21:49:37.371 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:49:37 smithi092 ceph-mon[37563]: 6.1c scrub ok 2024-03-11T21:49:37.371 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:49:37 smithi092 ceph-mon[37563]: 4.14 deep-scrub starts 2024-03-11T21:49:37.371 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:49:37 smithi092 ceph-mon[37563]: 4.14 deep-scrub ok 2024-03-11T21:49:37.585 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:49:37 smithi097 ceph-mon[33317]: pgmap v1147: 132 pgs: 132 active+clean; 582 KiB data, 6.3 GiB used, 709 GiB / 715 GiB avail; 688 B/s rd, 0 op/s; 157 B/s, 5 objects/s recovering 2024-03-11T21:49:37.586 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:49:37 smithi097 ceph-mon[33317]: 2.14 scrub starts 2024-03-11T21:49:37.586 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:49:37 smithi097 ceph-mon[33317]: 2.14 scrub ok 2024-03-11T21:49:37.586 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:49:37 smithi097 ceph-mon[33317]: 6.1c scrub starts 2024-03-11T21:49:37.586 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:49:37 smithi097 ceph-mon[33317]: 6.1c scrub ok 2024-03-11T21:49:37.586 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:49:37 smithi097 ceph-mon[33317]: 4.14 deep-scrub starts 2024-03-11T21:49:37.586 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:49:37 smithi097 ceph-mon[33317]: 4.14 deep-scrub ok 2024-03-11T21:49:38.190 INFO:teuthology.orchestra.run.smithi092.stderr:set osd.0 primary-affinity to 1 (8655362) 2024-03-11T21:49:38.191 INFO:journalctl@ceph.mgr.y.smithi092.stdout:Mar 11 21:49:37 smithi092 ceph-00db0016-dfed-11ee-95c8-87774f69a715-mgr-y[29801]: ::ffff:172.21.15.97 - - [11/Mar/2024:21:49:37] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-11T21:49:38.509 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:49:38 smithi092 ceph-mon[29572]: 2.f scrub starts 2024-03-11T21:49:38.510 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:49:38 smithi092 ceph-mon[29572]: 2.f scrub ok 2024-03-11T21:49:38.510 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:49:38 smithi092 ceph-mon[29572]: 5.1e scrub starts 2024-03-11T21:49:38.510 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:49:38 smithi092 ceph-mon[29572]: 5.1e scrub ok 2024-03-11T21:49:38.510 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:49:38 smithi092 ceph-mon[29572]: from='client.? v1:172.21.15.92:0/2638556186' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 0, "weight": 1.0}]: dispatch 2024-03-11T21:49:38.510 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:49:38 smithi092 ceph-mon[37563]: 2.f scrub starts 2024-03-11T21:49:38.510 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:49:38 smithi092 ceph-mon[37563]: 2.f scrub ok 2024-03-11T21:49:38.510 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:49:38 smithi092 ceph-mon[37563]: 5.1e scrub starts 2024-03-11T21:49:38.510 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:49:38 smithi092 ceph-mon[37563]: 5.1e scrub ok 2024-03-11T21:49:38.510 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:49:38 smithi092 ceph-mon[37563]: from='client.? v1:172.21.15.92:0/2638556186' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 0, "weight": 1.0}]: dispatch 2024-03-11T21:49:38.585 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:49:38 smithi097 ceph-mon[33317]: 2.f scrub starts 2024-03-11T21:49:38.586 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:49:38 smithi097 ceph-mon[33317]: 2.f scrub ok 2024-03-11T21:49:38.586 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:49:38 smithi097 ceph-mon[33317]: 5.1e scrub starts 2024-03-11T21:49:38.586 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:49:38 smithi097 ceph-mon[33317]: 5.1e scrub ok 2024-03-11T21:49:38.586 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:49:38 smithi097 ceph-mon[33317]: from='client.? v1:172.21.15.92:0/2638556186' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 0, "weight": 1.0}]: dispatch 2024-03-11T21:49:38.912 DEBUG:teuthology.orchestra.run.smithi092:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid 00db0016-dfed-11ee-95c8-87774f69a715 -- ceph osd reweight 1 1 2024-03-11T21:49:39.153 INFO:teuthology.orchestra.run.smithi092.stderr:Inferring config /var/lib/ceph/00db0016-dfed-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T21:49:39.448 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:49:39 smithi092 ceph-mon[37563]: pgmap v1148: 132 pgs: 132 active+clean; 582 KiB data, 6.3 GiB used, 709 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s; 63 B/s, 3 objects/s recovering 2024-03-11T21:49:39.448 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:49:39 smithi092 ceph-mon[37563]: 4.11 scrub starts 2024-03-11T21:49:39.449 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:49:39 smithi092 ceph-mon[37563]: 4.11 scrub ok 2024-03-11T21:49:39.449 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:49:39 smithi092 ceph-mon[37563]: from='client.? v1:172.21.15.92:0/2638556186' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 0, "weight": 1.0}]': finished 2024-03-11T21:49:39.449 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:49:39 smithi092 ceph-mon[37563]: osdmap e535: 8 total, 8 up, 8 in 2024-03-11T21:49:39.449 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:49:39 smithi092 ceph-mon[37563]: from='mgr.14607 v1:172.21.15.92:0/1702717597' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T21:49:39.449 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:49:39 smithi092 ceph-mon[37563]: 2.19 scrub starts 2024-03-11T21:49:39.449 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:49:39 smithi092 ceph-mon[37563]: 2.19 scrub ok 2024-03-11T21:49:39.449 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:49:39 smithi092 ceph-mon[29572]: pgmap v1148: 132 pgs: 132 active+clean; 582 KiB data, 6.3 GiB used, 709 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s; 63 B/s, 3 objects/s recovering 2024-03-11T21:49:39.449 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:49:39 smithi092 ceph-mon[29572]: 4.11 scrub starts 2024-03-11T21:49:39.449 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:49:39 smithi092 ceph-mon[29572]: 4.11 scrub ok 2024-03-11T21:49:39.449 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:49:39 smithi092 ceph-mon[29572]: from='client.? v1:172.21.15.92:0/2638556186' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 0, "weight": 1.0}]': finished 2024-03-11T21:49:39.449 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:49:39 smithi092 ceph-mon[29572]: osdmap e535: 8 total, 8 up, 8 in 2024-03-11T21:49:39.449 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:49:39 smithi092 ceph-mon[29572]: from='mgr.14607 v1:172.21.15.92:0/1702717597' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T21:49:39.450 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:49:39 smithi092 ceph-mon[29572]: 2.19 scrub starts 2024-03-11T21:49:39.450 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:49:39 smithi092 ceph-mon[29572]: 2.19 scrub ok 2024-03-11T21:49:39.586 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:49:39 smithi097 ceph-mon[33317]: pgmap v1148: 132 pgs: 132 active+clean; 582 KiB data, 6.3 GiB used, 709 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s; 63 B/s, 3 objects/s recovering 2024-03-11T21:49:39.586 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:49:39 smithi097 ceph-mon[33317]: 4.11 scrub starts 2024-03-11T21:49:39.586 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:49:39 smithi097 ceph-mon[33317]: 4.11 scrub ok 2024-03-11T21:49:39.586 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:49:39 smithi097 ceph-mon[33317]: from='client.? v1:172.21.15.92:0/2638556186' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 0, "weight": 1.0}]': finished 2024-03-11T21:49:39.586 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:49:39 smithi097 ceph-mon[33317]: osdmap e535: 8 total, 8 up, 8 in 2024-03-11T21:49:39.586 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:49:39 smithi097 ceph-mon[33317]: from='mgr.14607 v1:172.21.15.92:0/1702717597' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T21:49:39.586 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:49:39 smithi097 ceph-mon[33317]: 2.19 scrub starts 2024-03-11T21:49:39.586 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:49:39 smithi097 ceph-mon[33317]: 2.19 scrub ok 2024-03-11T21:49:40.551 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:49:40 smithi092 ceph-mon[29572]: 6.3 scrub starts 2024-03-11T21:49:40.551 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:49:40 smithi092 ceph-mon[29572]: 6.3 scrub ok 2024-03-11T21:49:40.551 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:49:40 smithi092 ceph-mon[29572]: 2.3 scrub starts 2024-03-11T21:49:40.551 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:49:40 smithi092 ceph-mon[29572]: 2.3 scrub ok 2024-03-11T21:49:40.551 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:49:40 smithi092 ceph-mon[29572]: 6.9 scrub starts 2024-03-11T21:49:40.552 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:49:40 smithi092 ceph-mon[29572]: 6.9 scrub ok 2024-03-11T21:49:40.552 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:49:40 smithi092 ceph-mon[37563]: 6.3 scrub starts 2024-03-11T21:49:40.552 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:49:40 smithi092 ceph-mon[37563]: 6.3 scrub ok 2024-03-11T21:49:40.552 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:49:40 smithi092 ceph-mon[37563]: 2.3 scrub starts 2024-03-11T21:49:40.552 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:49:40 smithi092 ceph-mon[37563]: 2.3 scrub ok 2024-03-11T21:49:40.553 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:49:40 smithi092 ceph-mon[37563]: 6.9 scrub starts 2024-03-11T21:49:40.553 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:49:40 smithi092 ceph-mon[37563]: 6.9 scrub ok 2024-03-11T21:49:40.586 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:49:40 smithi097 ceph-mon[33317]: 6.3 scrub starts 2024-03-11T21:49:40.586 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:49:40 smithi097 ceph-mon[33317]: 6.3 scrub ok 2024-03-11T21:49:40.586 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:49:40 smithi097 ceph-mon[33317]: 2.3 scrub starts 2024-03-11T21:49:40.587 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:49:40 smithi097 ceph-mon[33317]: 2.3 scrub ok 2024-03-11T21:49:40.587 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:49:40 smithi097 ceph-mon[33317]: 6.9 scrub starts 2024-03-11T21:49:40.587 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:49:40 smithi097 ceph-mon[33317]: 6.9 scrub ok 2024-03-11T21:49:41.206 INFO:teuthology.orchestra.run.smithi092.stderr:reweighted osd.1 to 1 (10000) 2024-03-11T21:49:41.501 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:49:41 smithi092 ceph-mon[29572]: pgmap v1150: 132 pgs: 132 active+clean; 582 KiB data, 6.3 GiB used, 709 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T21:49:41.501 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:49:41 smithi092 ceph-mon[29572]: 5.1a scrub starts 2024-03-11T21:49:41.501 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:49:41 smithi092 ceph-mon[29572]: 5.1a scrub ok 2024-03-11T21:49:41.502 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:49:41 smithi092 ceph-mon[29572]: from='client.? v1:172.21.15.92:0/1599156319' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]: dispatch 2024-03-11T21:49:41.502 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:49:41 smithi092 ceph-mon[37563]: pgmap v1150: 132 pgs: 132 active+clean; 582 KiB data, 6.3 GiB used, 709 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T21:49:41.502 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:49:41 smithi092 ceph-mon[37563]: 5.1a scrub starts 2024-03-11T21:49:41.502 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:49:41 smithi092 ceph-mon[37563]: 5.1a scrub ok 2024-03-11T21:49:41.502 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:49:41 smithi092 ceph-mon[37563]: from='client.? v1:172.21.15.92:0/1599156319' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]: dispatch 2024-03-11T21:49:41.586 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:49:41 smithi097 ceph-mon[33317]: pgmap v1150: 132 pgs: 132 active+clean; 582 KiB data, 6.3 GiB used, 709 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T21:49:41.586 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:49:41 smithi097 ceph-mon[33317]: 5.1a scrub starts 2024-03-11T21:49:41.586 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:49:41 smithi097 ceph-mon[33317]: 5.1a scrub ok 2024-03-11T21:49:41.586 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:49:41 smithi097 ceph-mon[33317]: from='client.? v1:172.21.15.92:0/1599156319' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]: dispatch 2024-03-11T21:49:41.937 DEBUG:teuthology.orchestra.run.smithi092:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid 00db0016-dfed-11ee-95c8-87774f69a715 -- ceph osd primary-affinity 1 1 2024-03-11T21:49:42.180 INFO:teuthology.orchestra.run.smithi092.stderr:Inferring config /var/lib/ceph/00db0016-dfed-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T21:49:42.470 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:49:42 smithi092 ceph-mon[37563]: 5.19 deep-scrub starts 2024-03-11T21:49:42.470 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:49:42 smithi092 ceph-mon[37563]: 5.19 deep-scrub ok 2024-03-11T21:49:42.470 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:49:42 smithi092 ceph-mon[37563]: from='client.? v1:172.21.15.92:0/1599156319' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]': finished 2024-03-11T21:49:42.471 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:49:42 smithi092 ceph-mon[37563]: osdmap e536: 8 total, 8 up, 8 in 2024-03-11T21:49:42.471 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:49:42 smithi092 ceph-mon[29572]: 5.19 deep-scrub starts 2024-03-11T21:49:42.471 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:49:42 smithi092 ceph-mon[29572]: 5.19 deep-scrub ok 2024-03-11T21:49:42.471 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:49:42 smithi092 ceph-mon[29572]: from='client.? v1:172.21.15.92:0/1599156319' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]': finished 2024-03-11T21:49:42.471 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:49:42 smithi092 ceph-mon[29572]: osdmap e536: 8 total, 8 up, 8 in 2024-03-11T21:49:42.585 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:49:42 smithi097 ceph-mon[33317]: 5.19 deep-scrub starts 2024-03-11T21:49:42.586 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:49:42 smithi097 ceph-mon[33317]: 5.19 deep-scrub ok 2024-03-11T21:49:42.586 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:49:42 smithi097 ceph-mon[33317]: from='client.? v1:172.21.15.92:0/1599156319' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]': finished 2024-03-11T21:49:42.586 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:49:42 smithi097 ceph-mon[33317]: osdmap e536: 8 total, 8 up, 8 in 2024-03-11T21:49:43.473 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:49:43 smithi092 ceph-mon[29572]: pgmap v1152: 132 pgs: 132 active+clean; 582 KiB data, 6.3 GiB used, 709 GiB / 715 GiB avail; 739 B/s rd, 0 op/s 2024-03-11T21:49:43.474 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:49:43 smithi092 ceph-mon[29572]: 4.4 scrub starts 2024-03-11T21:49:43.474 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:49:43 smithi092 ceph-mon[29572]: 4.4 scrub ok 2024-03-11T21:49:43.474 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:49:43 smithi092 ceph-mon[29572]: 6.e scrub starts 2024-03-11T21:49:43.474 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:49:43 smithi092 ceph-mon[29572]: 6.e scrub ok 2024-03-11T21:49:43.474 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:49:43 smithi092 ceph-mon[29572]: 2.15 scrub starts 2024-03-11T21:49:43.474 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:49:43 smithi092 ceph-mon[29572]: 2.15 scrub ok 2024-03-11T21:49:43.474 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:49:43 smithi092 ceph-mon[37563]: pgmap v1152: 132 pgs: 132 active+clean; 582 KiB data, 6.3 GiB used, 709 GiB / 715 GiB avail; 739 B/s rd, 0 op/s 2024-03-11T21:49:43.474 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:49:43 smithi092 ceph-mon[37563]: 4.4 scrub starts 2024-03-11T21:49:43.475 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:49:43 smithi092 ceph-mon[37563]: 4.4 scrub ok 2024-03-11T21:49:43.475 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:49:43 smithi092 ceph-mon[37563]: 6.e scrub starts 2024-03-11T21:49:43.475 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:49:43 smithi092 ceph-mon[37563]: 6.e scrub ok 2024-03-11T21:49:43.475 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:49:43 smithi092 ceph-mon[37563]: 2.15 scrub starts 2024-03-11T21:49:43.475 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:49:43 smithi092 ceph-mon[37563]: 2.15 scrub ok 2024-03-11T21:49:43.585 INFO:journalctl@ceph.iscsi.iscsi.a.smithi097.stdout:Mar 11 21:49:43 smithi097 ceph-00db0016-dfed-11ee-95c8-87774f69a715-iscsi-iscsi-a[60943]: debug there is no tcmu-runner data available 2024-03-11T21:49:43.586 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:49:43 smithi097 ceph-mon[33317]: pgmap v1152: 132 pgs: 132 active+clean; 582 KiB data, 6.3 GiB used, 709 GiB / 715 GiB avail; 739 B/s rd, 0 op/s 2024-03-11T21:49:43.586 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:49:43 smithi097 ceph-mon[33317]: 4.4 scrub starts 2024-03-11T21:49:43.586 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:49:43 smithi097 ceph-mon[33317]: 4.4 scrub ok 2024-03-11T21:49:43.586 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:49:43 smithi097 ceph-mon[33317]: 6.e scrub starts 2024-03-11T21:49:43.586 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:49:43 smithi097 ceph-mon[33317]: 6.e scrub ok 2024-03-11T21:49:43.586 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:49:43 smithi097 ceph-mon[33317]: 2.15 scrub starts 2024-03-11T21:49:43.586 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:49:43 smithi097 ceph-mon[33317]: 2.15 scrub ok 2024-03-11T21:49:44.370 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:49:44 smithi092 ceph-mon[37563]: 6.7 scrub starts 2024-03-11T21:49:44.370 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:49:44 smithi092 ceph-mon[37563]: 6.7 scrub ok 2024-03-11T21:49:44.370 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:49:44 smithi092 ceph-mon[37563]: from='client.24469 v1:172.21.15.97:0/404208310' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:49:44.370 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:49:44 smithi092 ceph-mon[37563]: 6.b scrub starts 2024-03-11T21:49:44.370 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:49:44 smithi092 ceph-mon[37563]: 6.b scrub ok 2024-03-11T21:49:44.371 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:49:44 smithi092 ceph-mon[29572]: 6.7 scrub starts 2024-03-11T21:49:44.371 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:49:44 smithi092 ceph-mon[29572]: 6.7 scrub ok 2024-03-11T21:49:44.371 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:49:44 smithi092 ceph-mon[29572]: from='client.24469 v1:172.21.15.97:0/404208310' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:49:44.371 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:49:44 smithi092 ceph-mon[29572]: 6.b scrub starts 2024-03-11T21:49:44.371 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:49:44 smithi092 ceph-mon[29572]: 6.b scrub ok 2024-03-11T21:49:44.585 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:49:44 smithi097 ceph-mon[33317]: 6.7 scrub starts 2024-03-11T21:49:44.586 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:49:44 smithi097 ceph-mon[33317]: 6.7 scrub ok 2024-03-11T21:49:44.586 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:49:44 smithi097 ceph-mon[33317]: from='client.24469 v1:172.21.15.97:0/404208310' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:49:44.586 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:49:44 smithi097 ceph-mon[33317]: 6.b scrub starts 2024-03-11T21:49:44.586 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:49:44 smithi097 ceph-mon[33317]: 6.b scrub ok 2024-03-11T21:49:45.233 INFO:teuthology.orchestra.run.smithi092.stderr:set osd.1 primary-affinity to 1 (8655362) 2024-03-11T21:49:45.585 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:49:45 smithi097 ceph-mon[33317]: pgmap v1153: 132 pgs: 132 active+clean; 582 KiB data, 6.3 GiB used, 709 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T21:49:45.585 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:49:45 smithi097 ceph-mon[33317]: 4.1 deep-scrub starts 2024-03-11T21:49:45.586 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:49:45 smithi097 ceph-mon[33317]: 4.1 deep-scrub ok 2024-03-11T21:49:45.586 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:49:45 smithi097 ceph-mon[33317]: 6.19 deep-scrub starts 2024-03-11T21:49:45.586 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:49:45 smithi097 ceph-mon[33317]: 6.19 deep-scrub ok 2024-03-11T21:49:45.586 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:49:45 smithi097 ceph-mon[33317]: from='client.? v1:172.21.15.92:0/3844479720' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 1, "weight": 1.0}]: dispatch 2024-03-11T21:49:45.602 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:49:45 smithi092 ceph-mon[29572]: pgmap v1153: 132 pgs: 132 active+clean; 582 KiB data, 6.3 GiB used, 709 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T21:49:45.603 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:49:45 smithi092 ceph-mon[29572]: 4.1 deep-scrub starts 2024-03-11T21:49:45.603 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:49:45 smithi092 ceph-mon[29572]: 4.1 deep-scrub ok 2024-03-11T21:49:45.603 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:49:45 smithi092 ceph-mon[29572]: 6.19 deep-scrub starts 2024-03-11T21:49:45.603 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:49:45 smithi092 ceph-mon[29572]: 6.19 deep-scrub ok 2024-03-11T21:49:45.603 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:49:45 smithi092 ceph-mon[29572]: from='client.? v1:172.21.15.92:0/3844479720' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 1, "weight": 1.0}]: dispatch 2024-03-11T21:49:45.603 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:49:45 smithi092 ceph-mon[37563]: pgmap v1153: 132 pgs: 132 active+clean; 582 KiB data, 6.3 GiB used, 709 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T21:49:45.603 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:49:45 smithi092 ceph-mon[37563]: 4.1 deep-scrub starts 2024-03-11T21:49:45.603 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:49:45 smithi092 ceph-mon[37563]: 4.1 deep-scrub ok 2024-03-11T21:49:45.603 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:49:45 smithi092 ceph-mon[37563]: 6.19 deep-scrub starts 2024-03-11T21:49:45.603 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:49:45 smithi092 ceph-mon[37563]: 6.19 deep-scrub ok 2024-03-11T21:49:45.603 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:49:45 smithi092 ceph-mon[37563]: from='client.? v1:172.21.15.92:0/3844479720' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 1, "weight": 1.0}]: dispatch 2024-03-11T21:49:45.966 DEBUG:teuthology.orchestra.run.smithi092:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid 00db0016-dfed-11ee-95c8-87774f69a715 -- ceph osd reweight 2 1 2024-03-11T21:49:46.208 INFO:teuthology.orchestra.run.smithi092.stderr:Inferring config /var/lib/ceph/00db0016-dfed-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T21:49:46.533 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:49:46 smithi092 ceph-mon[29572]: from='client.? v1:172.21.15.92:0/3844479720' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 1, "weight": 1.0}]': finished 2024-03-11T21:49:46.533 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:49:46 smithi092 ceph-mon[29572]: osdmap e537: 8 total, 8 up, 8 in 2024-03-11T21:49:46.533 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:49:46 smithi092 ceph-mon[29572]: 5.14 deep-scrub starts 2024-03-11T21:49:46.533 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:49:46 smithi092 ceph-mon[29572]: 5.14 deep-scrub ok 2024-03-11T21:49:46.533 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:49:46 smithi092 ceph-mon[37563]: from='client.? v1:172.21.15.92:0/3844479720' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 1, "weight": 1.0}]': finished 2024-03-11T21:49:46.534 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:49:46 smithi092 ceph-mon[37563]: osdmap e537: 8 total, 8 up, 8 in 2024-03-11T21:49:46.534 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:49:46 smithi092 ceph-mon[37563]: 5.14 deep-scrub starts 2024-03-11T21:49:46.534 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:49:46 smithi092 ceph-mon[37563]: 5.14 deep-scrub ok 2024-03-11T21:49:46.585 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:49:46 smithi097 ceph-mon[33317]: from='client.? v1:172.21.15.92:0/3844479720' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 1, "weight": 1.0}]': finished 2024-03-11T21:49:46.586 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:49:46 smithi097 ceph-mon[33317]: osdmap e537: 8 total, 8 up, 8 in 2024-03-11T21:49:46.586 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:49:46 smithi097 ceph-mon[33317]: 5.14 deep-scrub starts 2024-03-11T21:49:46.586 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:49:46 smithi097 ceph-mon[33317]: 5.14 deep-scrub ok 2024-03-11T21:49:47.547 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:49:47 smithi092 ceph-mon[29572]: pgmap v1155: 132 pgs: 132 active+clean; 582 KiB data, 6.3 GiB used, 709 GiB / 715 GiB avail; 1.4 KiB/s rd, 1 op/s 2024-03-11T21:49:47.547 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:49:47 smithi092 ceph-mon[29572]: 2.10 scrub starts 2024-03-11T21:49:47.547 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:49:47 smithi092 ceph-mon[29572]: 2.10 scrub ok 2024-03-11T21:49:47.547 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:49:47 smithi092 ceph-mon[29572]: osdmap e538: 8 total, 8 up, 8 in 2024-03-11T21:49:47.547 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:49:47 smithi092 ceph-mon[37563]: pgmap v1155: 132 pgs: 132 active+clean; 582 KiB data, 6.3 GiB used, 709 GiB / 715 GiB avail; 1.4 KiB/s rd, 1 op/s 2024-03-11T21:49:47.548 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:49:47 smithi092 ceph-mon[37563]: 2.10 scrub starts 2024-03-11T21:49:47.548 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:49:47 smithi092 ceph-mon[37563]: 2.10 scrub ok 2024-03-11T21:49:47.548 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:49:47 smithi092 ceph-mon[37563]: osdmap e538: 8 total, 8 up, 8 in 2024-03-11T21:49:47.585 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:49:47 smithi097 ceph-mon[33317]: pgmap v1155: 132 pgs: 132 active+clean; 582 KiB data, 6.3 GiB used, 709 GiB / 715 GiB avail; 1.4 KiB/s rd, 1 op/s 2024-03-11T21:49:47.586 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:49:47 smithi097 ceph-mon[33317]: 2.10 scrub starts 2024-03-11T21:49:47.586 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:49:47 smithi097 ceph-mon[33317]: 2.10 scrub ok 2024-03-11T21:49:47.586 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:49:47 smithi097 ceph-mon[33317]: osdmap e538: 8 total, 8 up, 8 in 2024-03-11T21:49:48.247 INFO:journalctl@ceph.mgr.y.smithi092.stdout:Mar 11 21:49:47 smithi092 ceph-00db0016-dfed-11ee-95c8-87774f69a715-mgr-y[29801]: ::ffff:172.21.15.97 - - [11/Mar/2024:21:49:47] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-11T21:49:48.585 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:49:48 smithi097 ceph-mon[33317]: 1.0 scrub starts 2024-03-11T21:49:48.586 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:49:48 smithi097 ceph-mon[33317]: 1.0 scrub ok 2024-03-11T21:49:48.619 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:49:48 smithi092 ceph-mon[29572]: 1.0 scrub starts 2024-03-11T21:49:48.619 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:49:48 smithi092 ceph-mon[29572]: 1.0 scrub ok 2024-03-11T21:49:48.620 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:49:48 smithi092 ceph-mon[37563]: 1.0 scrub starts 2024-03-11T21:49:48.620 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:49:48 smithi092 ceph-mon[37563]: 1.0 scrub ok 2024-03-11T21:49:49.258 INFO:teuthology.orchestra.run.smithi092.stderr:reweighted osd.2 to 1 (10000) 2024-03-11T21:49:49.585 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:49:49 smithi097 ceph-mon[33317]: pgmap v1157: 132 pgs: 132 active+clean; 582 KiB data, 6.3 GiB used, 709 GiB / 715 GiB avail; 809 B/s rd, 0 op/s 2024-03-11T21:49:49.586 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:49:49 smithi097 ceph-mon[33317]: 5.1f scrub starts 2024-03-11T21:49:49.586 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:49:49 smithi097 ceph-mon[33317]: 5.1f scrub ok 2024-03-11T21:49:49.586 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:49:49 smithi097 ceph-mon[33317]: from='client.? v1:172.21.15.92:0/194203867' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]: dispatch 2024-03-11T21:49:49.586 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:49:49 smithi097 ceph-mon[33317]: from='client.36152 ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]: dispatch 2024-03-11T21:49:49.586 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:49:49 smithi097 ceph-mon[33317]: 3.2 scrub starts 2024-03-11T21:49:49.586 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:49:49 smithi097 ceph-mon[33317]: 3.2 scrub ok 2024-03-11T21:49:49.619 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:49:49 smithi092 ceph-mon[29572]: pgmap v1157: 132 pgs: 132 active+clean; 582 KiB data, 6.3 GiB used, 709 GiB / 715 GiB avail; 809 B/s rd, 0 op/s 2024-03-11T21:49:49.620 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:49:49 smithi092 ceph-mon[29572]: 5.1f scrub starts 2024-03-11T21:49:49.620 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:49:49 smithi092 ceph-mon[29572]: 5.1f scrub ok 2024-03-11T21:49:49.620 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:49:49 smithi092 ceph-mon[29572]: from='client.? v1:172.21.15.92:0/194203867' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]: dispatch 2024-03-11T21:49:49.620 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:49:49 smithi092 ceph-mon[29572]: from='client.36152 ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]: dispatch 2024-03-11T21:49:49.620 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:49:49 smithi092 ceph-mon[29572]: 3.2 scrub starts 2024-03-11T21:49:49.620 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:49:49 smithi092 ceph-mon[29572]: 3.2 scrub ok 2024-03-11T21:49:49.620 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:49:49 smithi092 ceph-mon[37563]: pgmap v1157: 132 pgs: 132 active+clean; 582 KiB data, 6.3 GiB used, 709 GiB / 715 GiB avail; 809 B/s rd, 0 op/s 2024-03-11T21:49:49.620 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:49:49 smithi092 ceph-mon[37563]: 5.1f scrub starts 2024-03-11T21:49:49.620 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:49:49 smithi092 ceph-mon[37563]: 5.1f scrub ok 2024-03-11T21:49:49.620 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:49:49 smithi092 ceph-mon[37563]: from='client.? v1:172.21.15.92:0/194203867' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]: dispatch 2024-03-11T21:49:49.620 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:49:49 smithi092 ceph-mon[37563]: from='client.36152 ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]: dispatch 2024-03-11T21:49:49.621 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:49:49 smithi092 ceph-mon[37563]: 3.2 scrub starts 2024-03-11T21:49:49.621 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:49:49 smithi092 ceph-mon[37563]: 3.2 scrub ok 2024-03-11T21:49:50.045 DEBUG:teuthology.orchestra.run.smithi092:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid 00db0016-dfed-11ee-95c8-87774f69a715 -- ceph osd primary-affinity 2 1 2024-03-11T21:49:50.294 INFO:teuthology.orchestra.run.smithi092.stderr:Inferring config /var/lib/ceph/00db0016-dfed-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T21:49:50.370 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:49:50 smithi092 ceph-mon[29572]: 5.a scrub starts 2024-03-11T21:49:50.370 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:49:50 smithi092 ceph-mon[29572]: 5.a scrub ok 2024-03-11T21:49:50.370 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:49:50 smithi092 ceph-mon[29572]: 6.17 scrub starts 2024-03-11T21:49:50.370 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:49:50 smithi092 ceph-mon[29572]: 4.15 scrub starts 2024-03-11T21:49:50.370 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:49:50 smithi092 ceph-mon[29572]: 4.15 scrub ok 2024-03-11T21:49:50.370 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:49:50 smithi092 ceph-mon[29572]: 6.17 scrub ok 2024-03-11T21:49:50.370 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:49:50 smithi092 ceph-mon[29572]: from='client.36152 ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]': finished 2024-03-11T21:49:50.370 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:49:50 smithi092 ceph-mon[29572]: osdmap e539: 8 total, 8 up, 8 in 2024-03-11T21:49:50.370 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:49:50 smithi092 ceph-mon[37563]: 5.a scrub starts 2024-03-11T21:49:50.371 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:49:50 smithi092 ceph-mon[37563]: 5.a scrub ok 2024-03-11T21:49:50.371 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:49:50 smithi092 ceph-mon[37563]: 6.17 scrub starts 2024-03-11T21:49:50.371 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:49:50 smithi092 ceph-mon[37563]: 4.15 scrub starts 2024-03-11T21:49:50.371 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:49:50 smithi092 ceph-mon[37563]: 4.15 scrub ok 2024-03-11T21:49:50.371 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:49:50 smithi092 ceph-mon[37563]: 6.17 scrub ok 2024-03-11T21:49:50.371 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:49:50 smithi092 ceph-mon[37563]: from='client.36152 ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]': finished 2024-03-11T21:49:50.371 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:49:50 smithi092 ceph-mon[37563]: osdmap e539: 8 total, 8 up, 8 in 2024-03-11T21:49:50.585 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:49:50 smithi097 ceph-mon[33317]: 5.a scrub starts 2024-03-11T21:49:50.586 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:49:50 smithi097 ceph-mon[33317]: 5.a scrub ok 2024-03-11T21:49:50.586 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:49:50 smithi097 ceph-mon[33317]: 6.17 scrub starts 2024-03-11T21:49:50.586 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:49:50 smithi097 ceph-mon[33317]: 4.15 scrub starts 2024-03-11T21:49:50.586 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:49:50 smithi097 ceph-mon[33317]: 4.15 scrub ok 2024-03-11T21:49:50.586 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:49:50 smithi097 ceph-mon[33317]: 6.17 scrub ok 2024-03-11T21:49:50.586 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:49:50 smithi097 ceph-mon[33317]: from='client.36152 ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]': finished 2024-03-11T21:49:50.586 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:49:50 smithi097 ceph-mon[33317]: osdmap e539: 8 total, 8 up, 8 in 2024-03-11T21:49:51.538 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:49:51 smithi092 ceph-mon[29572]: pgmap v1159: 132 pgs: 132 active+clean; 582 KiB data, 6.3 GiB used, 709 GiB / 715 GiB avail; 127 KiB/s rd, 126 op/s 2024-03-11T21:49:51.538 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:49:51 smithi092 ceph-mon[37563]: pgmap v1159: 132 pgs: 132 active+clean; 582 KiB data, 6.3 GiB used, 709 GiB / 715 GiB avail; 127 KiB/s rd, 126 op/s 2024-03-11T21:49:51.585 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:49:51 smithi097 ceph-mon[33317]: pgmap v1159: 132 pgs: 132 active+clean; 582 KiB data, 6.3 GiB used, 709 GiB / 715 GiB avail; 127 KiB/s rd, 126 op/s 2024-03-11T21:49:52.586 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:49:52 smithi097 ceph-mon[33317]: 6.1f scrub starts 2024-03-11T21:49:52.586 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:49:52 smithi097 ceph-mon[33317]: 6.1f scrub ok 2024-03-11T21:49:52.619 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:49:52 smithi092 ceph-mon[29572]: 6.1f scrub starts 2024-03-11T21:49:52.619 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:49:52 smithi092 ceph-mon[29572]: 6.1f scrub ok 2024-03-11T21:49:52.620 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:49:52 smithi092 ceph-mon[37563]: 6.1f scrub starts 2024-03-11T21:49:52.620 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:49:52 smithi092 ceph-mon[37563]: 6.1f scrub ok 2024-03-11T21:49:53.281 INFO:teuthology.orchestra.run.smithi092.stderr:set osd.2 primary-affinity to 1 (8655362) 2024-03-11T21:49:53.585 INFO:journalctl@ceph.iscsi.iscsi.a.smithi097.stdout:Mar 11 21:49:53 smithi097 ceph-00db0016-dfed-11ee-95c8-87774f69a715-iscsi-iscsi-a[60943]: debug there is no tcmu-runner data available 2024-03-11T21:49:53.586 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:49:53 smithi097 ceph-mon[33317]: pgmap v1160: 132 pgs: 132 active+clean; 582 KiB data, 6.3 GiB used, 709 GiB / 715 GiB avail; 121 KiB/s rd, 120 op/s 2024-03-11T21:49:53.586 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:49:53 smithi097 ceph-mon[33317]: from='client.? v1:172.21.15.92:0/2593341481' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 2, "weight": 1.0}]: dispatch 2024-03-11T21:49:53.619 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:49:53 smithi092 ceph-mon[29572]: pgmap v1160: 132 pgs: 132 active+clean; 582 KiB data, 6.3 GiB used, 709 GiB / 715 GiB avail; 121 KiB/s rd, 120 op/s 2024-03-11T21:49:53.619 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:49:53 smithi092 ceph-mon[29572]: from='client.? v1:172.21.15.92:0/2593341481' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 2, "weight": 1.0}]: dispatch 2024-03-11T21:49:53.620 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:49:53 smithi092 ceph-mon[37563]: pgmap v1160: 132 pgs: 132 active+clean; 582 KiB data, 6.3 GiB used, 709 GiB / 715 GiB avail; 121 KiB/s rd, 120 op/s 2024-03-11T21:49:53.620 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:49:53 smithi092 ceph-mon[37563]: from='client.? v1:172.21.15.92:0/2593341481' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 2, "weight": 1.0}]: dispatch 2024-03-11T21:49:54.071 DEBUG:teuthology.orchestra.run.smithi092:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid 00db0016-dfed-11ee-95c8-87774f69a715 -- ceph osd reweight 4 1 2024-03-11T21:49:54.334 INFO:teuthology.orchestra.run.smithi092.stderr:Inferring config /var/lib/ceph/00db0016-dfed-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T21:49:54.369 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:49:54 smithi092 ceph-mon[29572]: 2.1 scrub starts 2024-03-11T21:49:54.370 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:49:54 smithi092 ceph-mon[29572]: 2.1 scrub ok 2024-03-11T21:49:54.370 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:49:54 smithi092 ceph-mon[29572]: 5.12 scrub starts 2024-03-11T21:49:54.370 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:49:54 smithi092 ceph-mon[29572]: 5.12 scrub ok 2024-03-11T21:49:54.370 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:49:54 smithi092 ceph-mon[29572]: from='client.? v1:172.21.15.92:0/2593341481' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 2, "weight": 1.0}]': finished 2024-03-11T21:49:54.370 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:49:54 smithi092 ceph-mon[29572]: osdmap e540: 8 total, 8 up, 8 in 2024-03-11T21:49:54.370 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:49:54 smithi092 ceph-mon[29572]: from='mgr.14607 v1:172.21.15.92:0/1702717597' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T21:49:54.370 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:49:54 smithi092 ceph-mon[37563]: 2.1 scrub starts 2024-03-11T21:49:54.370 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:49:54 smithi092 ceph-mon[37563]: 2.1 scrub ok 2024-03-11T21:49:54.370 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:49:54 smithi092 ceph-mon[37563]: 5.12 scrub starts 2024-03-11T21:49:54.370 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:49:54 smithi092 ceph-mon[37563]: 5.12 scrub ok 2024-03-11T21:49:54.371 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:49:54 smithi092 ceph-mon[37563]: from='client.? v1:172.21.15.92:0/2593341481' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 2, "weight": 1.0}]': finished 2024-03-11T21:49:54.371 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:49:54 smithi092 ceph-mon[37563]: osdmap e540: 8 total, 8 up, 8 in 2024-03-11T21:49:54.371 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:49:54 smithi092 ceph-mon[37563]: from='mgr.14607 v1:172.21.15.92:0/1702717597' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T21:49:54.585 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:49:54 smithi097 ceph-mon[33317]: 2.1 scrub starts 2024-03-11T21:49:54.586 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:49:54 smithi097 ceph-mon[33317]: 2.1 scrub ok 2024-03-11T21:49:54.586 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:49:54 smithi097 ceph-mon[33317]: 5.12 scrub starts 2024-03-11T21:49:54.586 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:49:54 smithi097 ceph-mon[33317]: 5.12 scrub ok 2024-03-11T21:49:54.586 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:49:54 smithi097 ceph-mon[33317]: from='client.? v1:172.21.15.92:0/2593341481' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 2, "weight": 1.0}]': finished 2024-03-11T21:49:54.586 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:49:54 smithi097 ceph-mon[33317]: osdmap e540: 8 total, 8 up, 8 in 2024-03-11T21:49:54.586 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:49:54 smithi097 ceph-mon[33317]: from='mgr.14607 v1:172.21.15.92:0/1702717597' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T21:49:55.512 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:49:55 smithi092 ceph-mon[37563]: from='client.24469 v1:172.21.15.97:0/404208310' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:49:55.513 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:49:55 smithi092 ceph-mon[37563]: pgmap v1162: 132 pgs: 132 active+clean; 582 KiB data, 6.3 GiB used, 709 GiB / 715 GiB avail; 105 KiB/s rd, 104 op/s 2024-03-11T21:49:55.513 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:49:55 smithi092 ceph-mon[29572]: from='client.24469 v1:172.21.15.97:0/404208310' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:49:55.513 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:49:55 smithi092 ceph-mon[29572]: pgmap v1162: 132 pgs: 132 active+clean; 582 KiB data, 6.3 GiB used, 709 GiB / 715 GiB avail; 105 KiB/s rd, 104 op/s 2024-03-11T21:49:55.586 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:49:55 smithi097 ceph-mon[33317]: from='client.24469 v1:172.21.15.97:0/404208310' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:49:55.586 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:49:55 smithi097 ceph-mon[33317]: pgmap v1162: 132 pgs: 132 active+clean; 582 KiB data, 6.3 GiB used, 709 GiB / 715 GiB avail; 105 KiB/s rd, 104 op/s 2024-03-11T21:49:56.585 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:49:56 smithi097 ceph-mon[33317]: 2.c scrub starts 2024-03-11T21:49:56.586 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:49:56 smithi097 ceph-mon[33317]: 2.c scrub ok 2024-03-11T21:49:56.586 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:49:56 smithi097 ceph-mon[33317]: 2.1e scrub starts 2024-03-11T21:49:56.586 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:49:56 smithi097 ceph-mon[33317]: 2.1e scrub ok 2024-03-11T21:49:56.619 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:49:56 smithi092 ceph-mon[29572]: 2.c scrub starts 2024-03-11T21:49:56.619 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:49:56 smithi092 ceph-mon[29572]: 2.c scrub ok 2024-03-11T21:49:56.620 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:49:56 smithi092 ceph-mon[29572]: 2.1e scrub starts 2024-03-11T21:49:56.620 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:49:56 smithi092 ceph-mon[29572]: 2.1e scrub ok 2024-03-11T21:49:56.620 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:49:56 smithi092 ceph-mon[37563]: 2.c scrub starts 2024-03-11T21:49:56.620 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:49:56 smithi092 ceph-mon[37563]: 2.c scrub ok 2024-03-11T21:49:56.620 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:49:56 smithi092 ceph-mon[37563]: 2.1e scrub starts 2024-03-11T21:49:56.620 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:49:56 smithi092 ceph-mon[37563]: 2.1e scrub ok 2024-03-11T21:49:57.299 INFO:teuthology.orchestra.run.smithi092.stderr:reweighted osd.4 to 1 (10000) 2024-03-11T21:49:57.585 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:49:57 smithi097 ceph-mon[33317]: pgmap v1163: 132 pgs: 132 active+clean; 582 KiB data, 6.3 GiB used, 709 GiB / 715 GiB avail; 96 KiB/s rd, 95 op/s 2024-03-11T21:49:57.586 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:49:57 smithi097 ceph-mon[33317]: 5.e scrub starts 2024-03-11T21:49:57.586 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:49:57 smithi097 ceph-mon[33317]: 5.e scrub ok 2024-03-11T21:49:57.586 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:49:57 smithi097 ceph-mon[33317]: 6.12 scrub starts 2024-03-11T21:49:57.586 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:49:57 smithi097 ceph-mon[33317]: 6.12 scrub ok 2024-03-11T21:49:57.586 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:49:57 smithi097 ceph-mon[33317]: from='client.? v1:172.21.15.92:0/2816669325' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]: dispatch 2024-03-11T21:49:57.586 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:49:57 smithi097 ceph-mon[33317]: from='client.36164 ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]: dispatch 2024-03-11T21:49:57.586 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:49:57 smithi097 ceph-mon[33317]: 6.1b scrub starts 2024-03-11T21:49:57.586 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:49:57 smithi097 ceph-mon[33317]: 6.1b scrub ok 2024-03-11T21:49:57.619 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:49:57 smithi092 ceph-mon[29572]: pgmap v1163: 132 pgs: 132 active+clean; 582 KiB data, 6.3 GiB used, 709 GiB / 715 GiB avail; 96 KiB/s rd, 95 op/s 2024-03-11T21:49:57.620 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:49:57 smithi092 ceph-mon[29572]: 5.e scrub starts 2024-03-11T21:49:57.620 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:49:57 smithi092 ceph-mon[29572]: 5.e scrub ok 2024-03-11T21:49:57.620 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:49:57 smithi092 ceph-mon[29572]: 6.12 scrub starts 2024-03-11T21:49:57.620 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:49:57 smithi092 ceph-mon[29572]: 6.12 scrub ok 2024-03-11T21:49:57.620 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:49:57 smithi092 ceph-mon[29572]: from='client.? v1:172.21.15.92:0/2816669325' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]: dispatch 2024-03-11T21:49:57.620 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:49:57 smithi092 ceph-mon[29572]: from='client.36164 ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]: dispatch 2024-03-11T21:49:57.620 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:49:57 smithi092 ceph-mon[29572]: 6.1b scrub starts 2024-03-11T21:49:57.620 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:49:57 smithi092 ceph-mon[29572]: 6.1b scrub ok 2024-03-11T21:49:57.620 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:49:57 smithi092 ceph-mon[37563]: pgmap v1163: 132 pgs: 132 active+clean; 582 KiB data, 6.3 GiB used, 709 GiB / 715 GiB avail; 96 KiB/s rd, 95 op/s 2024-03-11T21:49:57.621 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:49:57 smithi092 ceph-mon[37563]: 5.e scrub starts 2024-03-11T21:49:57.621 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:49:57 smithi092 ceph-mon[37563]: 5.e scrub ok 2024-03-11T21:49:57.621 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:49:57 smithi092 ceph-mon[37563]: 6.12 scrub starts 2024-03-11T21:49:57.621 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:49:57 smithi092 ceph-mon[37563]: 6.12 scrub ok 2024-03-11T21:49:57.621 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:49:57 smithi092 ceph-mon[37563]: from='client.? v1:172.21.15.92:0/2816669325' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]: dispatch 2024-03-11T21:49:57.621 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:49:57 smithi092 ceph-mon[37563]: from='client.36164 ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]: dispatch 2024-03-11T21:49:57.621 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:49:57 smithi092 ceph-mon[37563]: 6.1b scrub starts 2024-03-11T21:49:57.621 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:49:57 smithi092 ceph-mon[37563]: 6.1b scrub ok 2024-03-11T21:49:58.060 DEBUG:teuthology.orchestra.run.smithi092:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid 00db0016-dfed-11ee-95c8-87774f69a715 -- ceph osd primary-affinity 4 1 2024-03-11T21:49:58.301 INFO:journalctl@ceph.mgr.y.smithi092.stdout:Mar 11 21:49:57 smithi092 ceph-00db0016-dfed-11ee-95c8-87774f69a715-mgr-y[29801]: ::ffff:172.21.15.97 - - [11/Mar/2024:21:49:57] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-11T21:49:58.334 INFO:teuthology.orchestra.run.smithi092.stderr:Inferring config /var/lib/ceph/00db0016-dfed-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T21:49:58.586 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:49:58 smithi097 ceph-mon[33317]: 6.d scrub starts 2024-03-11T21:49:58.586 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:49:58 smithi097 ceph-mon[33317]: 6.d scrub ok 2024-03-11T21:49:58.586 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:49:58 smithi097 ceph-mon[33317]: purged_snaps scrub starts 2024-03-11T21:49:58.586 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:49:58 smithi097 ceph-mon[33317]: purged_snaps scrub ok 2024-03-11T21:49:58.586 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:49:58 smithi097 ceph-mon[33317]: from='client.36164 ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]': finished 2024-03-11T21:49:58.586 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:49:58 smithi097 ceph-mon[33317]: osdmap e541: 8 total, 8 up, 8 in 2024-03-11T21:49:58.588 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:49:58 smithi092 ceph-mon[29572]: 6.d scrub starts 2024-03-11T21:49:58.588 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:49:58 smithi092 ceph-mon[29572]: 6.d scrub ok 2024-03-11T21:49:58.588 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:49:58 smithi092 ceph-mon[29572]: purged_snaps scrub starts 2024-03-11T21:49:58.588 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:49:58 smithi092 ceph-mon[29572]: purged_snaps scrub ok 2024-03-11T21:49:58.588 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:49:58 smithi092 ceph-mon[29572]: from='client.36164 ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]': finished 2024-03-11T21:49:58.588 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:49:58 smithi092 ceph-mon[29572]: osdmap e541: 8 total, 8 up, 8 in 2024-03-11T21:49:58.588 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:49:58 smithi092 ceph-mon[37563]: 6.d scrub starts 2024-03-11T21:49:58.589 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:49:58 smithi092 ceph-mon[37563]: 6.d scrub ok 2024-03-11T21:49:58.589 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:49:58 smithi092 ceph-mon[37563]: purged_snaps scrub starts 2024-03-11T21:49:58.589 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:49:58 smithi092 ceph-mon[37563]: purged_snaps scrub ok 2024-03-11T21:49:58.589 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:49:58 smithi092 ceph-mon[37563]: from='client.36164 ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]': finished 2024-03-11T21:49:58.589 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:49:58 smithi092 ceph-mon[37563]: osdmap e541: 8 total, 8 up, 8 in 2024-03-11T21:49:59.585 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:49:59 smithi097 ceph-mon[33317]: pgmap v1165: 132 pgs: 132 active+clean; 582 KiB data, 6.3 GiB used, 709 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-03-11T21:49:59.586 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:49:59 smithi097 ceph-mon[33317]: 5.7 scrub starts 2024-03-11T21:49:59.586 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:49:59 smithi097 ceph-mon[33317]: 5.7 scrub ok 2024-03-11T21:49:59.586 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:49:59 smithi097 ceph-mon[33317]: 6.1d scrub starts 2024-03-11T21:49:59.586 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:49:59 smithi097 ceph-mon[33317]: 6.1d scrub ok 2024-03-11T21:49:59.586 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:49:59 smithi097 ceph-mon[33317]: 4.17 scrub starts 2024-03-11T21:49:59.586 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:49:59 smithi097 ceph-mon[33317]: 4.17 scrub ok 2024-03-11T21:49:59.592 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:49:59 smithi092 ceph-mon[29572]: pgmap v1165: 132 pgs: 132 active+clean; 582 KiB data, 6.3 GiB used, 709 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-03-11T21:49:59.592 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:49:59 smithi092 ceph-mon[29572]: 5.7 scrub starts 2024-03-11T21:49:59.592 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:49:59 smithi092 ceph-mon[29572]: 5.7 scrub ok 2024-03-11T21:49:59.592 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:49:59 smithi092 ceph-mon[29572]: 6.1d scrub starts 2024-03-11T21:49:59.592 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:49:59 smithi092 ceph-mon[29572]: 6.1d scrub ok 2024-03-11T21:49:59.592 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:49:59 smithi092 ceph-mon[29572]: 4.17 scrub starts 2024-03-11T21:49:59.592 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:49:59 smithi092 ceph-mon[29572]: 4.17 scrub ok 2024-03-11T21:49:59.592 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:49:59 smithi092 ceph-mon[37563]: pgmap v1165: 132 pgs: 132 active+clean; 582 KiB data, 6.3 GiB used, 709 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-03-11T21:49:59.592 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:49:59 smithi092 ceph-mon[37563]: 5.7 scrub starts 2024-03-11T21:49:59.592 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:49:59 smithi092 ceph-mon[37563]: 5.7 scrub ok 2024-03-11T21:49:59.593 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:49:59 smithi092 ceph-mon[37563]: 6.1d scrub starts 2024-03-11T21:49:59.593 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:49:59 smithi092 ceph-mon[37563]: 6.1d scrub ok 2024-03-11T21:49:59.593 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:49:59 smithi092 ceph-mon[37563]: 4.17 scrub starts 2024-03-11T21:49:59.593 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:49:59 smithi092 ceph-mon[37563]: 4.17 scrub ok 2024-03-11T21:50:00.311 INFO:teuthology.orchestra.run.smithi092.stderr:set osd.4 primary-affinity to 1 (8655362) 2024-03-11T21:50:00.585 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:50:00 smithi097 ceph-mon[33317]: overall HEALTH_OK 2024-03-11T21:50:00.586 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:50:00 smithi097 ceph-mon[33317]: from='client.? v1:172.21.15.92:0/2376930300' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 4, "weight": 1.0}]: dispatch 2024-03-11T21:50:00.586 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:50:00 smithi097 ceph-mon[33317]: from='client.36170 ' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 4, "weight": 1.0}]: dispatch 2024-03-11T21:50:00.619 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:50:00 smithi092 ceph-mon[37563]: overall HEALTH_OK 2024-03-11T21:50:00.620 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:50:00 smithi092 ceph-mon[37563]: from='client.? v1:172.21.15.92:0/2376930300' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 4, "weight": 1.0}]: dispatch 2024-03-11T21:50:00.620 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:50:00 smithi092 ceph-mon[37563]: from='client.36170 ' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 4, "weight": 1.0}]: dispatch 2024-03-11T21:50:00.620 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:50:00 smithi092 ceph-mon[29572]: overall HEALTH_OK 2024-03-11T21:50:00.620 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:50:00 smithi092 ceph-mon[29572]: from='client.? v1:172.21.15.92:0/2376930300' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 4, "weight": 1.0}]: dispatch 2024-03-11T21:50:00.620 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:50:00 smithi092 ceph-mon[29572]: from='client.36170 ' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 4, "weight": 1.0}]: dispatch 2024-03-11T21:50:00.989 DEBUG:teuthology.orchestra.run.smithi092:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid 00db0016-dfed-11ee-95c8-87774f69a715 -- ceph osd reweight 5 1 2024-03-11T21:50:01.233 INFO:teuthology.orchestra.run.smithi092.stderr:Inferring config /var/lib/ceph/00db0016-dfed-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T21:50:01.585 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:50:01 smithi097 ceph-mon[33317]: pgmap v1166: 132 pgs: 132 active+clean; 582 KiB data, 6.3 GiB used, 709 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T21:50:01.586 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:50:01 smithi097 ceph-mon[33317]: 4.12 scrub starts 2024-03-11T21:50:01.586 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:50:01 smithi097 ceph-mon[33317]: 4.12 scrub ok 2024-03-11T21:50:01.586 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:50:01 smithi097 ceph-mon[33317]: from='client.36170 ' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 4, "weight": 1.0}]': finished 2024-03-11T21:50:01.586 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:50:01 smithi097 ceph-mon[33317]: osdmap e542: 8 total, 8 up, 8 in 2024-03-11T21:50:01.620 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:50:01 smithi092 ceph-mon[37563]: pgmap v1166: 132 pgs: 132 active+clean; 582 KiB data, 6.3 GiB used, 709 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T21:50:01.620 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:50:01 smithi092 ceph-mon[37563]: 4.12 scrub starts 2024-03-11T21:50:01.620 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:50:01 smithi092 ceph-mon[37563]: 4.12 scrub ok 2024-03-11T21:50:01.620 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:50:01 smithi092 ceph-mon[37563]: from='client.36170 ' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 4, "weight": 1.0}]': finished 2024-03-11T21:50:01.620 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:50:01 smithi092 ceph-mon[37563]: osdmap e542: 8 total, 8 up, 8 in 2024-03-11T21:50:01.620 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:50:01 smithi092 ceph-mon[29572]: pgmap v1166: 132 pgs: 132 active+clean; 582 KiB data, 6.3 GiB used, 709 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T21:50:01.620 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:50:01 smithi092 ceph-mon[29572]: 4.12 scrub starts 2024-03-11T21:50:01.620 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:50:01 smithi092 ceph-mon[29572]: 4.12 scrub ok 2024-03-11T21:50:01.620 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:50:01 smithi092 ceph-mon[29572]: from='client.36170 ' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 4, "weight": 1.0}]': finished 2024-03-11T21:50:01.621 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:50:01 smithi092 ceph-mon[29572]: osdmap e542: 8 total, 8 up, 8 in 2024-03-11T21:50:02.544 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:50:02 smithi092 ceph-mon[29572]: 4.1c scrub starts 2024-03-11T21:50:02.544 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:50:02 smithi092 ceph-mon[29572]: 4.1c scrub ok 2024-03-11T21:50:02.544 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:50:02 smithi092 ceph-mon[29572]: purged_snaps scrub starts 2024-03-11T21:50:02.544 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:50:02 smithi092 ceph-mon[29572]: purged_snaps scrub ok 2024-03-11T21:50:02.544 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:50:02 smithi092 ceph-mon[37563]: 4.1c scrub starts 2024-03-11T21:50:02.544 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:50:02 smithi092 ceph-mon[37563]: 4.1c scrub ok 2024-03-11T21:50:02.544 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:50:02 smithi092 ceph-mon[37563]: purged_snaps scrub starts 2024-03-11T21:50:02.544 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:50:02 smithi092 ceph-mon[37563]: purged_snaps scrub ok 2024-03-11T21:50:02.586 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:50:02 smithi097 ceph-mon[33317]: 4.1c scrub starts 2024-03-11T21:50:02.586 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:50:02 smithi097 ceph-mon[33317]: 4.1c scrub ok 2024-03-11T21:50:02.586 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:50:02 smithi097 ceph-mon[33317]: purged_snaps scrub starts 2024-03-11T21:50:02.586 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:50:02 smithi097 ceph-mon[33317]: purged_snaps scrub ok 2024-03-11T21:50:03.324 INFO:teuthology.orchestra.run.smithi092.stderr:reweighted osd.5 to 1 (10000) 2024-03-11T21:50:03.585 INFO:journalctl@ceph.iscsi.iscsi.a.smithi097.stdout:Mar 11 21:50:03 smithi097 ceph-00db0016-dfed-11ee-95c8-87774f69a715-iscsi-iscsi-a[60943]: debug there is no tcmu-runner data available 2024-03-11T21:50:03.586 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:50:03 smithi097 ceph-mon[33317]: pgmap v1168: 132 pgs: 132 active+clean; 582 KiB data, 6.3 GiB used, 709 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T21:50:03.586 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:50:03 smithi097 ceph-mon[33317]: 2.12 deep-scrub starts 2024-03-11T21:50:03.586 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:50:03 smithi097 ceph-mon[33317]: 2.12 deep-scrub ok 2024-03-11T21:50:03.586 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:50:03 smithi097 ceph-mon[33317]: 6.2 scrub starts 2024-03-11T21:50:03.586 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:50:03 smithi097 ceph-mon[33317]: 6.2 scrub ok 2024-03-11T21:50:03.586 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:50:03 smithi097 ceph-mon[33317]: osdmap e543: 8 total, 8 up, 8 in 2024-03-11T21:50:03.586 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:50:03 smithi097 ceph-mon[33317]: from='client.? v1:172.21.15.92:0/2077558219' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]: dispatch 2024-03-11T21:50:03.586 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:50:03 smithi097 ceph-mon[33317]: from='client.36176 ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]: dispatch 2024-03-11T21:50:03.619 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:50:03 smithi092 ceph-mon[29572]: pgmap v1168: 132 pgs: 132 active+clean; 582 KiB data, 6.3 GiB used, 709 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T21:50:03.619 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:50:03 smithi092 ceph-mon[29572]: 2.12 deep-scrub starts 2024-03-11T21:50:03.620 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:50:03 smithi092 ceph-mon[29572]: 2.12 deep-scrub ok 2024-03-11T21:50:03.620 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:50:03 smithi092 ceph-mon[29572]: 6.2 scrub starts 2024-03-11T21:50:03.620 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:50:03 smithi092 ceph-mon[29572]: 6.2 scrub ok 2024-03-11T21:50:03.620 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:50:03 smithi092 ceph-mon[29572]: osdmap e543: 8 total, 8 up, 8 in 2024-03-11T21:50:03.620 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:50:03 smithi092 ceph-mon[29572]: from='client.? v1:172.21.15.92:0/2077558219' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]: dispatch 2024-03-11T21:50:03.620 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:50:03 smithi092 ceph-mon[29572]: from='client.36176 ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]: dispatch 2024-03-11T21:50:03.620 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:50:03 smithi092 ceph-mon[37563]: pgmap v1168: 132 pgs: 132 active+clean; 582 KiB data, 6.3 GiB used, 709 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T21:50:03.620 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:50:03 smithi092 ceph-mon[37563]: 2.12 deep-scrub starts 2024-03-11T21:50:03.620 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:50:03 smithi092 ceph-mon[37563]: 2.12 deep-scrub ok 2024-03-11T21:50:03.620 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:50:03 smithi092 ceph-mon[37563]: 6.2 scrub starts 2024-03-11T21:50:03.620 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:50:03 smithi092 ceph-mon[37563]: 6.2 scrub ok 2024-03-11T21:50:03.620 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:50:03 smithi092 ceph-mon[37563]: osdmap e543: 8 total, 8 up, 8 in 2024-03-11T21:50:03.621 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:50:03 smithi092 ceph-mon[37563]: from='client.? v1:172.21.15.92:0/2077558219' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]: dispatch 2024-03-11T21:50:03.621 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:50:03 smithi092 ceph-mon[37563]: from='client.36176 ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]: dispatch 2024-03-11T21:50:04.224 DEBUG:teuthology.orchestra.run.smithi092:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid 00db0016-dfed-11ee-95c8-87774f69a715 -- ceph osd primary-affinity 5 1 2024-03-11T21:50:04.478 INFO:teuthology.orchestra.run.smithi092.stderr:Inferring config /var/lib/ceph/00db0016-dfed-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T21:50:04.586 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:50:04 smithi097 ceph-mon[33317]: 4.7 scrub starts 2024-03-11T21:50:04.586 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:50:04 smithi097 ceph-mon[33317]: 4.7 scrub ok 2024-03-11T21:50:04.586 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:50:04 smithi097 ceph-mon[33317]: 6.1e scrub starts 2024-03-11T21:50:04.586 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:50:04 smithi097 ceph-mon[33317]: 6.1e scrub ok 2024-03-11T21:50:04.586 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:50:04 smithi097 ceph-mon[33317]: 3.1 scrub starts 2024-03-11T21:50:04.586 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:50:04 smithi097 ceph-mon[33317]: 3.1 scrub ok 2024-03-11T21:50:04.586 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:50:04 smithi097 ceph-mon[33317]: from='client.36176 ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]': finished 2024-03-11T21:50:04.586 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:50:04 smithi097 ceph-mon[33317]: osdmap e544: 8 total, 8 up, 8 in 2024-03-11T21:50:04.586 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:50:04 smithi097 ceph-mon[33317]: 6.c scrub starts 2024-03-11T21:50:04.586 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:50:04 smithi097 ceph-mon[33317]: 6.c scrub ok 2024-03-11T21:50:04.587 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:50:04 smithi097 ceph-mon[33317]: 2.2 scrub starts 2024-03-11T21:50:04.587 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:50:04 smithi097 ceph-mon[33317]: 2.2 scrub ok 2024-03-11T21:50:04.600 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:50:04 smithi092 ceph-mon[37563]: 4.7 scrub starts 2024-03-11T21:50:04.600 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:50:04 smithi092 ceph-mon[37563]: 4.7 scrub ok 2024-03-11T21:50:04.601 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:50:04 smithi092 ceph-mon[37563]: 6.1e scrub starts 2024-03-11T21:50:04.601 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:50:04 smithi092 ceph-mon[37563]: 6.1e scrub ok 2024-03-11T21:50:04.601 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:50:04 smithi092 ceph-mon[37563]: 3.1 scrub starts 2024-03-11T21:50:04.601 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:50:04 smithi092 ceph-mon[37563]: 3.1 scrub ok 2024-03-11T21:50:04.601 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:50:04 smithi092 ceph-mon[37563]: from='client.36176 ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]': finished 2024-03-11T21:50:04.601 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:50:04 smithi092 ceph-mon[37563]: osdmap e544: 8 total, 8 up, 8 in 2024-03-11T21:50:04.601 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:50:04 smithi092 ceph-mon[37563]: 6.c scrub starts 2024-03-11T21:50:04.601 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:50:04 smithi092 ceph-mon[37563]: 6.c scrub ok 2024-03-11T21:50:04.601 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:50:04 smithi092 ceph-mon[37563]: 2.2 scrub starts 2024-03-11T21:50:04.601 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:50:04 smithi092 ceph-mon[37563]: 2.2 scrub ok 2024-03-11T21:50:04.601 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:50:04 smithi092 ceph-mon[29572]: 4.7 scrub starts 2024-03-11T21:50:04.602 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:50:04 smithi092 ceph-mon[29572]: 4.7 scrub ok 2024-03-11T21:50:04.602 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:50:04 smithi092 ceph-mon[29572]: 6.1e scrub starts 2024-03-11T21:50:04.602 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:50:04 smithi092 ceph-mon[29572]: 6.1e scrub ok 2024-03-11T21:50:04.602 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:50:04 smithi092 ceph-mon[29572]: 3.1 scrub starts 2024-03-11T21:50:04.602 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:50:04 smithi092 ceph-mon[29572]: 3.1 scrub ok 2024-03-11T21:50:04.602 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:50:04 smithi092 ceph-mon[29572]: from='client.36176 ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]': finished 2024-03-11T21:50:04.602 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:50:04 smithi092 ceph-mon[29572]: osdmap e544: 8 total, 8 up, 8 in 2024-03-11T21:50:04.602 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:50:04 smithi092 ceph-mon[29572]: 6.c scrub starts 2024-03-11T21:50:04.602 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:50:04 smithi092 ceph-mon[29572]: 6.c scrub ok 2024-03-11T21:50:04.602 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:50:04 smithi092 ceph-mon[29572]: 2.2 scrub starts 2024-03-11T21:50:04.602 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:50:04 smithi092 ceph-mon[29572]: 2.2 scrub ok 2024-03-11T21:50:05.586 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:50:05 smithi097 ceph-mon[33317]: from='client.24469 v1:172.21.15.97:0/404208310' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:50:05.586 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:50:05 smithi097 ceph-mon[33317]: pgmap v1171: 132 pgs: 132 active+clean; 582 KiB data, 6.3 GiB used, 709 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-11T21:50:05.586 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:50:05 smithi097 ceph-mon[33317]: 3.0 scrub starts 2024-03-11T21:50:05.586 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:50:05 smithi097 ceph-mon[33317]: 3.0 scrub ok 2024-03-11T21:50:05.620 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:50:05 smithi092 ceph-mon[37563]: from='client.24469 v1:172.21.15.97:0/404208310' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:50:05.620 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:50:05 smithi092 ceph-mon[37563]: pgmap v1171: 132 pgs: 132 active+clean; 582 KiB data, 6.3 GiB used, 709 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-11T21:50:05.620 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:50:05 smithi092 ceph-mon[37563]: 3.0 scrub starts 2024-03-11T21:50:05.620 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:50:05 smithi092 ceph-mon[37563]: 3.0 scrub ok 2024-03-11T21:50:05.620 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:50:05 smithi092 ceph-mon[29572]: from='client.24469 v1:172.21.15.97:0/404208310' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:50:05.620 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:50:05 smithi092 ceph-mon[29572]: pgmap v1171: 132 pgs: 132 active+clean; 582 KiB data, 6.3 GiB used, 709 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-11T21:50:05.620 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:50:05 smithi092 ceph-mon[29572]: 3.0 scrub starts 2024-03-11T21:50:05.620 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:50:05 smithi092 ceph-mon[29572]: 3.0 scrub ok 2024-03-11T21:50:07.343 INFO:teuthology.orchestra.run.smithi092.stderr:set osd.5 primary-affinity to 1 (8655362) 2024-03-11T21:50:07.620 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:50:07 smithi092 ceph-mon[29572]: pgmap v1172: 132 pgs: 132 active+clean; 582 KiB data, 6.3 GiB used, 709 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-11T21:50:07.620 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:50:07 smithi092 ceph-mon[29572]: 5.1c scrub starts 2024-03-11T21:50:07.620 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:50:07 smithi092 ceph-mon[29572]: 5.1c scrub ok 2024-03-11T21:50:07.620 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:50:07 smithi092 ceph-mon[29572]: 2.1c scrub starts 2024-03-11T21:50:07.620 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:50:07 smithi092 ceph-mon[29572]: 2.1c scrub ok 2024-03-11T21:50:07.620 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:50:07 smithi092 ceph-mon[29572]: from='client.? v1:172.21.15.92:0/72971101' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 5, "weight": 1.0}]: dispatch 2024-03-11T21:50:07.620 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:50:07 smithi092 ceph-mon[29572]: from='client.36182 ' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 5, "weight": 1.0}]: dispatch 2024-03-11T21:50:07.620 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:50:07 smithi092 ceph-mon[29572]: 5.17 deep-scrub starts 2024-03-11T21:50:07.620 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:50:07 smithi092 ceph-mon[29572]: 5.17 deep-scrub ok 2024-03-11T21:50:07.621 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:50:07 smithi092 ceph-mon[37563]: pgmap v1172: 132 pgs: 132 active+clean; 582 KiB data, 6.3 GiB used, 709 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-11T21:50:07.621 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:50:07 smithi092 ceph-mon[37563]: 5.1c scrub starts 2024-03-11T21:50:07.621 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:50:07 smithi092 ceph-mon[37563]: 5.1c scrub ok 2024-03-11T21:50:07.621 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:50:07 smithi092 ceph-mon[37563]: 2.1c scrub starts 2024-03-11T21:50:07.621 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:50:07 smithi092 ceph-mon[37563]: 2.1c scrub ok 2024-03-11T21:50:07.621 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:50:07 smithi092 ceph-mon[37563]: from='client.? v1:172.21.15.92:0/72971101' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 5, "weight": 1.0}]: dispatch 2024-03-11T21:50:07.621 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:50:07 smithi092 ceph-mon[37563]: from='client.36182 ' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 5, "weight": 1.0}]: dispatch 2024-03-11T21:50:07.621 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:50:07 smithi092 ceph-mon[37563]: 5.17 deep-scrub starts 2024-03-11T21:50:07.621 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:50:07 smithi092 ceph-mon[37563]: 5.17 deep-scrub ok 2024-03-11T21:50:07.836 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:50:07 smithi097 ceph-mon[33317]: pgmap v1172: 132 pgs: 132 active+clean; 582 KiB data, 6.3 GiB used, 709 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-11T21:50:07.836 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:50:07 smithi097 ceph-mon[33317]: 5.1c scrub starts 2024-03-11T21:50:07.836 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:50:07 smithi097 ceph-mon[33317]: 5.1c scrub ok 2024-03-11T21:50:07.836 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:50:07 smithi097 ceph-mon[33317]: 2.1c scrub starts 2024-03-11T21:50:07.836 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:50:07 smithi097 ceph-mon[33317]: 2.1c scrub ok 2024-03-11T21:50:07.836 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:50:07 smithi097 ceph-mon[33317]: from='client.? v1:172.21.15.92:0/72971101' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 5, "weight": 1.0}]: dispatch 2024-03-11T21:50:07.836 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:50:07 smithi097 ceph-mon[33317]: from='client.36182 ' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 5, "weight": 1.0}]: dispatch 2024-03-11T21:50:07.836 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:50:07 smithi097 ceph-mon[33317]: 5.17 deep-scrub starts 2024-03-11T21:50:07.836 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:50:07 smithi097 ceph-mon[33317]: 5.17 deep-scrub ok 2024-03-11T21:50:08.102 INFO:journalctl@ceph.mgr.y.smithi092.stdout:Mar 11 21:50:07 smithi092 ceph-00db0016-dfed-11ee-95c8-87774f69a715-mgr-y[29801]: ::ffff:172.21.15.97 - - [11/Mar/2024:21:50:07] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-11T21:50:08.133 DEBUG:teuthology.orchestra.run.smithi092:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid 00db0016-dfed-11ee-95c8-87774f69a715 -- ceph osd reweight 6 1 2024-03-11T21:50:08.369 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:50:08 smithi092 ceph-mon[29572]: 5.8 scrub starts 2024-03-11T21:50:08.370 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:50:08 smithi092 ceph-mon[29572]: 5.8 scrub ok 2024-03-11T21:50:08.370 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:50:08 smithi092 ceph-mon[29572]: 2.d scrub starts 2024-03-11T21:50:08.370 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:50:08 smithi092 ceph-mon[29572]: 2.d scrub ok 2024-03-11T21:50:08.370 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:50:08 smithi092 ceph-mon[29572]: 6.a scrub starts 2024-03-11T21:50:08.370 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:50:08 smithi092 ceph-mon[29572]: 6.a scrub ok 2024-03-11T21:50:08.370 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:50:08 smithi092 ceph-mon[29572]: 5.2 scrub starts 2024-03-11T21:50:08.370 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:50:08 smithi092 ceph-mon[29572]: 5.2 scrub ok 2024-03-11T21:50:08.370 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:50:08 smithi092 ceph-mon[29572]: from='client.36182 ' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 5, "weight": 1.0}]': finished 2024-03-11T21:50:08.370 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:50:08 smithi092 ceph-mon[29572]: osdmap e545: 8 total, 8 up, 8 in 2024-03-11T21:50:08.370 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:50:08 smithi092 ceph-mon[37563]: 5.8 scrub starts 2024-03-11T21:50:08.370 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:50:08 smithi092 ceph-mon[37563]: 5.8 scrub ok 2024-03-11T21:50:08.371 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:50:08 smithi092 ceph-mon[37563]: 2.d scrub starts 2024-03-11T21:50:08.371 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:50:08 smithi092 ceph-mon[37563]: 2.d scrub ok 2024-03-11T21:50:08.371 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:50:08 smithi092 ceph-mon[37563]: 6.a scrub starts 2024-03-11T21:50:08.371 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:50:08 smithi092 ceph-mon[37563]: 6.a scrub ok 2024-03-11T21:50:08.371 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:50:08 smithi092 ceph-mon[37563]: 5.2 scrub starts 2024-03-11T21:50:08.371 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:50:08 smithi092 ceph-mon[37563]: 5.2 scrub ok 2024-03-11T21:50:08.371 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:50:08 smithi092 ceph-mon[37563]: from='client.36182 ' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 5, "weight": 1.0}]': finished 2024-03-11T21:50:08.371 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:50:08 smithi092 ceph-mon[37563]: osdmap e545: 8 total, 8 up, 8 in 2024-03-11T21:50:08.382 INFO:teuthology.orchestra.run.smithi092.stderr:Inferring config /var/lib/ceph/00db0016-dfed-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T21:50:08.836 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:50:08 smithi097 ceph-mon[33317]: 5.8 scrub starts 2024-03-11T21:50:08.836 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:50:08 smithi097 ceph-mon[33317]: 5.8 scrub ok 2024-03-11T21:50:08.836 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:50:08 smithi097 ceph-mon[33317]: 2.d scrub starts 2024-03-11T21:50:08.836 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:50:08 smithi097 ceph-mon[33317]: 2.d scrub ok 2024-03-11T21:50:08.836 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:50:08 smithi097 ceph-mon[33317]: 6.a scrub starts 2024-03-11T21:50:08.836 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:50:08 smithi097 ceph-mon[33317]: 6.a scrub ok 2024-03-11T21:50:08.836 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:50:08 smithi097 ceph-mon[33317]: 5.2 scrub starts 2024-03-11T21:50:08.836 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:50:08 smithi097 ceph-mon[33317]: 5.2 scrub ok 2024-03-11T21:50:08.836 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:50:08 smithi097 ceph-mon[33317]: from='client.36182 ' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 5, "weight": 1.0}]': finished 2024-03-11T21:50:08.836 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:50:08 smithi097 ceph-mon[33317]: osdmap e545: 8 total, 8 up, 8 in 2024-03-11T21:50:09.620 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:50:09 smithi092 ceph-mon[29572]: pgmap v1174: 132 pgs: 132 active+clean; 582 KiB data, 6.3 GiB used, 709 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-11T21:50:09.620 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:50:09 smithi092 ceph-mon[29572]: 2.7 scrub starts 2024-03-11T21:50:09.620 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:50:09 smithi092 ceph-mon[29572]: 2.7 scrub ok 2024-03-11T21:50:09.620 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:50:09 smithi092 ceph-mon[29572]: from='mgr.14607 v1:172.21.15.92:0/1702717597' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T21:50:09.620 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:50:09 smithi092 ceph-mon[29572]: from='mgr.14607 v1:172.21.15.92:0/1702717597' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T21:50:09.620 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:50:09 smithi092 ceph-mon[37563]: pgmap v1174: 132 pgs: 132 active+clean; 582 KiB data, 6.3 GiB used, 709 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-11T21:50:09.620 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:50:09 smithi092 ceph-mon[37563]: 2.7 scrub starts 2024-03-11T21:50:09.620 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:50:09 smithi092 ceph-mon[37563]: 2.7 scrub ok 2024-03-11T21:50:09.620 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:50:09 smithi092 ceph-mon[37563]: from='mgr.14607 v1:172.21.15.92:0/1702717597' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T21:50:09.620 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:50:09 smithi092 ceph-mon[37563]: from='mgr.14607 v1:172.21.15.92:0/1702717597' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T21:50:09.836 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:50:09 smithi097 ceph-mon[33317]: pgmap v1174: 132 pgs: 132 active+clean; 582 KiB data, 6.3 GiB used, 709 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-11T21:50:09.836 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:50:09 smithi097 ceph-mon[33317]: 2.7 scrub starts 2024-03-11T21:50:09.836 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:50:09 smithi097 ceph-mon[33317]: 2.7 scrub ok 2024-03-11T21:50:09.836 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:50:09 smithi097 ceph-mon[33317]: from='mgr.14607 v1:172.21.15.92:0/1702717597' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T21:50:09.836 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:50:09 smithi097 ceph-mon[33317]: from='mgr.14607 v1:172.21.15.92:0/1702717597' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T21:50:10.620 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:50:10 smithi092 ceph-mon[29572]: 4.f scrub starts 2024-03-11T21:50:10.620 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:50:10 smithi092 ceph-mon[29572]: 4.f scrub ok 2024-03-11T21:50:10.620 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:50:10 smithi092 ceph-mon[29572]: 4.1e scrub starts 2024-03-11T21:50:10.620 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:50:10 smithi092 ceph-mon[29572]: 4.1e scrub ok 2024-03-11T21:50:10.620 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:50:10 smithi092 ceph-mon[29572]: from='mgr.14607 v1:172.21.15.92:0/1702717597' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T21:50:10.620 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:50:10 smithi092 ceph-mon[29572]: from='mgr.14607 v1:172.21.15.92:0/1702717597' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T21:50:10.620 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:50:10 smithi092 ceph-mon[29572]: from='mgr.14607 ' entity='mgr.y' 2024-03-11T21:50:10.620 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:50:10 smithi092 ceph-mon[37563]: 4.f scrub starts 2024-03-11T21:50:10.620 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:50:10 smithi092 ceph-mon[37563]: 4.f scrub ok 2024-03-11T21:50:10.621 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:50:10 smithi092 ceph-mon[37563]: 4.1e scrub starts 2024-03-11T21:50:10.621 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:50:10 smithi092 ceph-mon[37563]: 4.1e scrub ok 2024-03-11T21:50:10.621 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:50:10 smithi092 ceph-mon[37563]: from='mgr.14607 v1:172.21.15.92:0/1702717597' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T21:50:10.621 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:50:10 smithi092 ceph-mon[37563]: from='mgr.14607 v1:172.21.15.92:0/1702717597' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T21:50:10.621 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:50:10 smithi092 ceph-mon[37563]: from='mgr.14607 ' entity='mgr.y' 2024-03-11T21:50:10.836 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:50:10 smithi097 ceph-mon[33317]: 4.f scrub starts 2024-03-11T21:50:10.836 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:50:10 smithi097 ceph-mon[33317]: 4.f scrub ok 2024-03-11T21:50:10.836 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:50:10 smithi097 ceph-mon[33317]: 4.1e scrub starts 2024-03-11T21:50:10.836 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:50:10 smithi097 ceph-mon[33317]: 4.1e scrub ok 2024-03-11T21:50:10.836 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:50:10 smithi097 ceph-mon[33317]: from='mgr.14607 v1:172.21.15.92:0/1702717597' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T21:50:10.836 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:50:10 smithi097 ceph-mon[33317]: from='mgr.14607 v1:172.21.15.92:0/1702717597' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T21:50:10.836 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:50:10 smithi097 ceph-mon[33317]: from='mgr.14607 ' entity='mgr.y' 2024-03-11T21:50:11.362 INFO:teuthology.orchestra.run.smithi092.stderr:reweighted osd.6 to 1 (10000) 2024-03-11T21:50:11.620 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:50:11 smithi092 ceph-mon[29572]: pgmap v1175: 132 pgs: 132 active+clean; 582 KiB data, 6.3 GiB used, 709 GiB / 715 GiB avail; 1.4 KiB/s rd, 1 op/s 2024-03-11T21:50:11.620 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:50:11 smithi092 ceph-mon[29572]: from='client.? v1:172.21.15.92:0/680829602' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]: dispatch 2024-03-11T21:50:11.620 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:50:11 smithi092 ceph-mon[29572]: 4.19 scrub starts 2024-03-11T21:50:11.620 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:50:11 smithi092 ceph-mon[29572]: 4.19 scrub ok 2024-03-11T21:50:11.620 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:50:11 smithi092 ceph-mon[29572]: purged_snaps scrub starts 2024-03-11T21:50:11.620 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:50:11 smithi092 ceph-mon[29572]: purged_snaps scrub ok 2024-03-11T21:50:11.620 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:50:11 smithi092 ceph-mon[37563]: pgmap v1175: 132 pgs: 132 active+clean; 582 KiB data, 6.3 GiB used, 709 GiB / 715 GiB avail; 1.4 KiB/s rd, 1 op/s 2024-03-11T21:50:11.620 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:50:11 smithi092 ceph-mon[37563]: from='client.? v1:172.21.15.92:0/680829602' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]: dispatch 2024-03-11T21:50:11.621 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:50:11 smithi092 ceph-mon[37563]: 4.19 scrub starts 2024-03-11T21:50:11.621 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:50:11 smithi092 ceph-mon[37563]: 4.19 scrub ok 2024-03-11T21:50:11.621 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:50:11 smithi092 ceph-mon[37563]: purged_snaps scrub starts 2024-03-11T21:50:11.621 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:50:11 smithi092 ceph-mon[37563]: purged_snaps scrub ok 2024-03-11T21:50:11.836 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:50:11 smithi097 ceph-mon[33317]: pgmap v1175: 132 pgs: 132 active+clean; 582 KiB data, 6.3 GiB used, 709 GiB / 715 GiB avail; 1.4 KiB/s rd, 1 op/s 2024-03-11T21:50:11.836 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:50:11 smithi097 ceph-mon[33317]: from='client.? v1:172.21.15.92:0/680829602' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]: dispatch 2024-03-11T21:50:11.836 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:50:11 smithi097 ceph-mon[33317]: 4.19 scrub starts 2024-03-11T21:50:11.836 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:50:11 smithi097 ceph-mon[33317]: 4.19 scrub ok 2024-03-11T21:50:11.836 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:50:11 smithi097 ceph-mon[33317]: purged_snaps scrub starts 2024-03-11T21:50:11.836 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:50:11 smithi097 ceph-mon[33317]: purged_snaps scrub ok 2024-03-11T21:50:12.209 DEBUG:teuthology.orchestra.run.smithi092:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid 00db0016-dfed-11ee-95c8-87774f69a715 -- ceph osd primary-affinity 6 1 2024-03-11T21:50:12.458 INFO:teuthology.orchestra.run.smithi092.stderr:Inferring config /var/lib/ceph/00db0016-dfed-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T21:50:12.580 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:50:12 smithi092 ceph-mon[29572]: 6.14 scrub starts 2024-03-11T21:50:12.580 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:50:12 smithi092 ceph-mon[29572]: 6.14 scrub ok 2024-03-11T21:50:12.580 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:50:12 smithi092 ceph-mon[29572]: purged_snaps scrub starts 2024-03-11T21:50:12.580 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:50:12 smithi092 ceph-mon[29572]: purged_snaps scrub ok 2024-03-11T21:50:12.581 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:50:12 smithi092 ceph-mon[29572]: from='client.? v1:172.21.15.92:0/680829602' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]': finished 2024-03-11T21:50:12.581 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:50:12 smithi092 ceph-mon[29572]: osdmap e546: 8 total, 8 up, 8 in 2024-03-11T21:50:12.581 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:50:12 smithi092 ceph-mon[29572]: 2.16 scrub starts 2024-03-11T21:50:12.581 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:50:12 smithi092 ceph-mon[29572]: 2.16 scrub ok 2024-03-11T21:50:12.581 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:50:12 smithi092 ceph-mon[37563]: 6.14 scrub starts 2024-03-11T21:50:12.581 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:50:12 smithi092 ceph-mon[37563]: 6.14 scrub ok 2024-03-11T21:50:12.581 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:50:12 smithi092 ceph-mon[37563]: purged_snaps scrub starts 2024-03-11T21:50:12.581 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:50:12 smithi092 ceph-mon[37563]: purged_snaps scrub ok 2024-03-11T21:50:12.581 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:50:12 smithi092 ceph-mon[37563]: from='client.? v1:172.21.15.92:0/680829602' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]': finished 2024-03-11T21:50:12.581 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:50:12 smithi092 ceph-mon[37563]: osdmap e546: 8 total, 8 up, 8 in 2024-03-11T21:50:12.581 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:50:12 smithi092 ceph-mon[37563]: 2.16 scrub starts 2024-03-11T21:50:12.582 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:50:12 smithi092 ceph-mon[37563]: 2.16 scrub ok 2024-03-11T21:50:12.836 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:50:12 smithi097 ceph-mon[33317]: 6.14 scrub starts 2024-03-11T21:50:12.836 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:50:12 smithi097 ceph-mon[33317]: 6.14 scrub ok 2024-03-11T21:50:12.836 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:50:12 smithi097 ceph-mon[33317]: purged_snaps scrub starts 2024-03-11T21:50:12.836 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:50:12 smithi097 ceph-mon[33317]: purged_snaps scrub ok 2024-03-11T21:50:12.836 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:50:12 smithi097 ceph-mon[33317]: from='client.? v1:172.21.15.92:0/680829602' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]': finished 2024-03-11T21:50:12.836 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:50:12 smithi097 ceph-mon[33317]: osdmap e546: 8 total, 8 up, 8 in 2024-03-11T21:50:12.836 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:50:12 smithi097 ceph-mon[33317]: 2.16 scrub starts 2024-03-11T21:50:12.836 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:50:12 smithi097 ceph-mon[33317]: 2.16 scrub ok 2024-03-11T21:50:13.619 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:50:13 smithi092 ceph-mon[29572]: pgmap v1177: 132 pgs: 132 active+clean; 582 KiB data, 6.3 GiB used, 709 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T21:50:13.620 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:50:13 smithi092 ceph-mon[29572]: 4.1d scrub starts 2024-03-11T21:50:13.620 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:50:13 smithi092 ceph-mon[29572]: 4.1d scrub ok 2024-03-11T21:50:13.620 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:50:13 smithi092 ceph-mon[37563]: pgmap v1177: 132 pgs: 132 active+clean; 582 KiB data, 6.3 GiB used, 709 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T21:50:13.620 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:50:13 smithi092 ceph-mon[37563]: 4.1d scrub starts 2024-03-11T21:50:13.620 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:50:13 smithi092 ceph-mon[37563]: 4.1d scrub ok 2024-03-11T21:50:13.836 INFO:journalctl@ceph.iscsi.iscsi.a.smithi097.stdout:Mar 11 21:50:13 smithi097 ceph-00db0016-dfed-11ee-95c8-87774f69a715-iscsi-iscsi-a[60943]: debug there is no tcmu-runner data available 2024-03-11T21:50:13.836 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:50:13 smithi097 ceph-mon[33317]: pgmap v1177: 132 pgs: 132 active+clean; 582 KiB data, 6.3 GiB used, 709 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T21:50:13.836 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:50:13 smithi097 ceph-mon[33317]: 4.1d scrub starts 2024-03-11T21:50:13.836 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:50:13 smithi097 ceph-mon[33317]: 4.1d scrub ok 2024-03-11T21:50:14.619 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:50:14 smithi092 ceph-mon[29572]: 6.5 scrub starts 2024-03-11T21:50:14.620 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:50:14 smithi092 ceph-mon[29572]: 6.5 scrub ok 2024-03-11T21:50:14.620 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:50:14 smithi092 ceph-mon[37563]: 6.5 scrub starts 2024-03-11T21:50:14.620 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:50:14 smithi092 ceph-mon[37563]: 6.5 scrub ok 2024-03-11T21:50:14.835 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:50:14 smithi097 ceph-mon[33317]: 6.5 scrub starts 2024-03-11T21:50:14.836 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:50:14 smithi097 ceph-mon[33317]: 6.5 scrub ok 2024-03-11T21:50:15.390 INFO:teuthology.orchestra.run.smithi092.stderr:set osd.6 primary-affinity to 1 (8655362) 2024-03-11T21:50:15.736 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:50:15 smithi092 ceph-mon[29572]: from='client.24469 v1:172.21.15.97:0/404208310' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:50:15.736 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:50:15 smithi092 ceph-mon[29572]: pgmap v1178: 132 pgs: 132 active+clean; 582 KiB data, 6.3 GiB used, 709 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-03-11T21:50:15.736 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:50:15 smithi092 ceph-mon[29572]: 5.1d scrub starts 2024-03-11T21:50:15.736 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:50:15 smithi092 ceph-mon[29572]: 5.1d scrub ok 2024-03-11T21:50:15.736 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:50:15 smithi092 ceph-mon[29572]: 5.18 scrub starts 2024-03-11T21:50:15.736 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:50:15 smithi092 ceph-mon[29572]: 5.18 scrub ok 2024-03-11T21:50:15.736 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:50:15 smithi092 ceph-mon[29572]: 2.5 scrub starts 2024-03-11T21:50:15.737 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:50:15 smithi092 ceph-mon[29572]: 2.5 scrub ok 2024-03-11T21:50:15.737 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:50:15 smithi092 ceph-mon[29572]: 6.6 deep-scrub starts 2024-03-11T21:50:15.737 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:50:15 smithi092 ceph-mon[29572]: 6.6 deep-scrub ok 2024-03-11T21:50:15.737 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:50:15 smithi092 ceph-mon[29572]: from='client.? v1:172.21.15.92:0/2179595943' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 6, "weight": 1.0}]: dispatch 2024-03-11T21:50:15.737 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:50:15 smithi092 ceph-mon[29572]: purged_snaps scrub starts 2024-03-11T21:50:15.737 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:50:15 smithi092 ceph-mon[29572]: purged_snaps scrub ok 2024-03-11T21:50:15.737 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:50:15 smithi092 ceph-mon[29572]: 2.18 scrub starts 2024-03-11T21:50:15.737 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:50:15 smithi092 ceph-mon[29572]: 2.18 scrub ok 2024-03-11T21:50:15.737 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:50:15 smithi092 ceph-mon[37563]: from='client.24469 v1:172.21.15.97:0/404208310' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:50:15.737 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:50:15 smithi092 ceph-mon[37563]: pgmap v1178: 132 pgs: 132 active+clean; 582 KiB data, 6.3 GiB used, 709 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-03-11T21:50:15.737 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:50:15 smithi092 ceph-mon[37563]: 5.1d scrub starts 2024-03-11T21:50:15.738 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:50:15 smithi092 ceph-mon[37563]: 5.1d scrub ok 2024-03-11T21:50:15.738 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:50:15 smithi092 ceph-mon[37563]: 5.18 scrub starts 2024-03-11T21:50:15.738 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:50:15 smithi092 ceph-mon[37563]: 5.18 scrub ok 2024-03-11T21:50:15.738 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:50:15 smithi092 ceph-mon[37563]: 2.5 scrub starts 2024-03-11T21:50:15.738 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:50:15 smithi092 ceph-mon[37563]: 2.5 scrub ok 2024-03-11T21:50:15.738 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:50:15 smithi092 ceph-mon[37563]: 6.6 deep-scrub starts 2024-03-11T21:50:15.738 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:50:15 smithi092 ceph-mon[37563]: 6.6 deep-scrub ok 2024-03-11T21:50:15.738 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:50:15 smithi092 ceph-mon[37563]: from='client.? v1:172.21.15.92:0/2179595943' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 6, "weight": 1.0}]: dispatch 2024-03-11T21:50:15.738 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:50:15 smithi092 ceph-mon[37563]: purged_snaps scrub starts 2024-03-11T21:50:15.738 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:50:15 smithi092 ceph-mon[37563]: purged_snaps scrub ok 2024-03-11T21:50:15.738 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:50:15 smithi092 ceph-mon[37563]: 2.18 scrub starts 2024-03-11T21:50:15.738 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:50:15 smithi092 ceph-mon[37563]: 2.18 scrub ok 2024-03-11T21:50:15.836 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:50:15 smithi097 ceph-mon[33317]: from='client.24469 v1:172.21.15.97:0/404208310' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:50:15.836 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:50:15 smithi097 ceph-mon[33317]: pgmap v1178: 132 pgs: 132 active+clean; 582 KiB data, 6.3 GiB used, 709 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-03-11T21:50:15.836 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:50:15 smithi097 ceph-mon[33317]: 5.1d scrub starts 2024-03-11T21:50:15.836 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:50:15 smithi097 ceph-mon[33317]: 5.1d scrub ok 2024-03-11T21:50:15.836 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:50:15 smithi097 ceph-mon[33317]: 5.18 scrub starts 2024-03-11T21:50:15.836 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:50:15 smithi097 ceph-mon[33317]: 5.18 scrub ok 2024-03-11T21:50:15.836 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:50:15 smithi097 ceph-mon[33317]: 2.5 scrub starts 2024-03-11T21:50:15.836 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:50:15 smithi097 ceph-mon[33317]: 2.5 scrub ok 2024-03-11T21:50:15.836 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:50:15 smithi097 ceph-mon[33317]: 6.6 deep-scrub starts 2024-03-11T21:50:15.836 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:50:15 smithi097 ceph-mon[33317]: 6.6 deep-scrub ok 2024-03-11T21:50:15.836 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:50:15 smithi097 ceph-mon[33317]: from='client.? v1:172.21.15.92:0/2179595943' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 6, "weight": 1.0}]: dispatch 2024-03-11T21:50:15.837 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:50:15 smithi097 ceph-mon[33317]: purged_snaps scrub starts 2024-03-11T21:50:15.837 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:50:15 smithi097 ceph-mon[33317]: purged_snaps scrub ok 2024-03-11T21:50:15.837 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:50:15 smithi097 ceph-mon[33317]: 2.18 scrub starts 2024-03-11T21:50:15.837 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:50:15 smithi097 ceph-mon[33317]: 2.18 scrub ok 2024-03-11T21:50:16.035 DEBUG:teuthology.orchestra.run.smithi092:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid 00db0016-dfed-11ee-95c8-87774f69a715 -- ceph osd reweight 7 1 2024-03-11T21:50:16.282 INFO:teuthology.orchestra.run.smithi092.stderr:Inferring config /var/lib/ceph/00db0016-dfed-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T21:50:16.689 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:50:16 smithi092 ceph-mon[29572]: 5.6 scrub starts 2024-03-11T21:50:16.689 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:50:16 smithi092 ceph-mon[29572]: 5.6 scrub ok 2024-03-11T21:50:16.689 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:50:16 smithi092 ceph-mon[29572]: 6.15 scrub starts 2024-03-11T21:50:16.689 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:50:16 smithi092 ceph-mon[29572]: 6.15 scrub ok 2024-03-11T21:50:16.690 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:50:16 smithi092 ceph-mon[29572]: from='client.? v1:172.21.15.92:0/2179595943' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 6, "weight": 1.0}]': finished 2024-03-11T21:50:16.690 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:50:16 smithi092 ceph-mon[29572]: osdmap e547: 8 total, 8 up, 8 in 2024-03-11T21:50:16.690 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:50:16 smithi092 ceph-mon[29572]: pgmap v1180: 132 pgs: 132 active+clean; 582 KiB data, 6.3 GiB used, 709 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T21:50:16.690 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:50:16 smithi092 ceph-mon[29572]: 5.16 scrub starts 2024-03-11T21:50:16.690 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:50:16 smithi092 ceph-mon[29572]: 5.16 scrub ok 2024-03-11T21:50:16.690 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:50:16 smithi092 ceph-mon[37563]: 5.6 scrub starts 2024-03-11T21:50:16.690 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:50:16 smithi092 ceph-mon[37563]: 5.6 scrub ok 2024-03-11T21:50:16.690 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:50:16 smithi092 ceph-mon[37563]: 6.15 scrub starts 2024-03-11T21:50:16.690 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:50:16 smithi092 ceph-mon[37563]: 6.15 scrub ok 2024-03-11T21:50:16.690 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:50:16 smithi092 ceph-mon[37563]: from='client.? v1:172.21.15.92:0/2179595943' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 6, "weight": 1.0}]': finished 2024-03-11T21:50:16.690 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:50:16 smithi092 ceph-mon[37563]: osdmap e547: 8 total, 8 up, 8 in 2024-03-11T21:50:16.691 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:50:16 smithi092 ceph-mon[37563]: pgmap v1180: 132 pgs: 132 active+clean; 582 KiB data, 6.3 GiB used, 709 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T21:50:16.691 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:50:16 smithi092 ceph-mon[37563]: 5.16 scrub starts 2024-03-11T21:50:16.691 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:50:16 smithi092 ceph-mon[37563]: 5.16 scrub ok 2024-03-11T21:50:16.835 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:50:16 smithi097 ceph-mon[33317]: 5.6 scrub starts 2024-03-11T21:50:16.836 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:50:16 smithi097 ceph-mon[33317]: 5.6 scrub ok 2024-03-11T21:50:16.836 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:50:16 smithi097 ceph-mon[33317]: 6.15 scrub starts 2024-03-11T21:50:16.836 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:50:16 smithi097 ceph-mon[33317]: 6.15 scrub ok 2024-03-11T21:50:16.836 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:50:16 smithi097 ceph-mon[33317]: from='client.? v1:172.21.15.92:0/2179595943' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 6, "weight": 1.0}]': finished 2024-03-11T21:50:16.836 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:50:16 smithi097 ceph-mon[33317]: osdmap e547: 8 total, 8 up, 8 in 2024-03-11T21:50:16.836 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:50:16 smithi097 ceph-mon[33317]: pgmap v1180: 132 pgs: 132 active+clean; 582 KiB data, 6.3 GiB used, 709 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T21:50:16.836 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:50:16 smithi097 ceph-mon[33317]: 5.16 scrub starts 2024-03-11T21:50:16.836 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:50:16 smithi097 ceph-mon[33317]: 5.16 scrub ok 2024-03-11T21:50:17.568 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:50:17 smithi092 ceph-mon[29572]: 4.c deep-scrub starts 2024-03-11T21:50:17.569 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:50:17 smithi092 ceph-mon[29572]: 4.c deep-scrub ok 2024-03-11T21:50:17.569 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:50:17 smithi092 ceph-mon[29572]: 5.f deep-scrub starts 2024-03-11T21:50:17.569 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:50:17 smithi092 ceph-mon[29572]: 5.f deep-scrub ok 2024-03-11T21:50:17.569 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:50:17 smithi092 ceph-mon[37563]: 4.c deep-scrub starts 2024-03-11T21:50:17.569 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:50:17 smithi092 ceph-mon[37563]: 4.c deep-scrub ok 2024-03-11T21:50:17.569 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:50:17 smithi092 ceph-mon[37563]: 5.f deep-scrub starts 2024-03-11T21:50:17.569 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:50:17 smithi092 ceph-mon[37563]: 5.f deep-scrub ok 2024-03-11T21:50:17.836 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:50:17 smithi097 ceph-mon[33317]: 4.c deep-scrub starts 2024-03-11T21:50:17.836 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:50:17 smithi097 ceph-mon[33317]: 4.c deep-scrub ok 2024-03-11T21:50:17.836 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:50:17 smithi097 ceph-mon[33317]: 5.f deep-scrub starts 2024-03-11T21:50:17.836 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:50:17 smithi097 ceph-mon[33317]: 5.f deep-scrub ok 2024-03-11T21:50:18.369 INFO:journalctl@ceph.mgr.y.smithi092.stdout:Mar 11 21:50:17 smithi092 ceph-00db0016-dfed-11ee-95c8-87774f69a715-mgr-y[29801]: ::ffff:172.21.15.97 - - [11/Mar/2024:21:50:17] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-11T21:50:18.403 INFO:teuthology.orchestra.run.smithi092.stderr:reweighted osd.7 to 1 (10000) 2024-03-11T21:50:18.814 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:50:18 smithi092 ceph-mon[29572]: 4.2 scrub starts 2024-03-11T21:50:18.814 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:50:18 smithi092 ceph-mon[29572]: 4.2 scrub ok 2024-03-11T21:50:18.815 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:50:18 smithi092 ceph-mon[29572]: pgmap v1181: 132 pgs: 132 active+clean; 582 KiB data, 6.3 GiB used, 709 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-03-11T21:50:18.815 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:50:18 smithi092 ceph-mon[29572]: from='client.? v1:172.21.15.92:0/3916171637' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]: dispatch 2024-03-11T21:50:18.815 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:50:18 smithi092 ceph-mon[37563]: 4.2 scrub starts 2024-03-11T21:50:18.815 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:50:18 smithi092 ceph-mon[37563]: 4.2 scrub ok 2024-03-11T21:50:18.815 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:50:18 smithi092 ceph-mon[37563]: pgmap v1181: 132 pgs: 132 active+clean; 582 KiB data, 6.3 GiB used, 709 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-03-11T21:50:18.815 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:50:18 smithi092 ceph-mon[37563]: from='client.? v1:172.21.15.92:0/3916171637' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]: dispatch 2024-03-11T21:50:18.835 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:50:18 smithi097 ceph-mon[33317]: 4.2 scrub starts 2024-03-11T21:50:18.835 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:50:18 smithi097 ceph-mon[33317]: 4.2 scrub ok 2024-03-11T21:50:18.836 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:50:18 smithi097 ceph-mon[33317]: pgmap v1181: 132 pgs: 132 active+clean; 582 KiB data, 6.3 GiB used, 709 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-03-11T21:50:18.836 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:50:18 smithi097 ceph-mon[33317]: from='client.? v1:172.21.15.92:0/3916171637' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]: dispatch 2024-03-11T21:50:19.082 DEBUG:teuthology.orchestra.run.smithi092:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid 00db0016-dfed-11ee-95c8-87774f69a715 -- ceph osd primary-affinity 7 1 2024-03-11T21:50:19.330 INFO:teuthology.orchestra.run.smithi092.stderr:Inferring config /var/lib/ceph/00db0016-dfed-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T21:50:19.696 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:50:19 smithi092 ceph-mon[29572]: from='client.? v1:172.21.15.92:0/3916171637' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]': finished 2024-03-11T21:50:19.697 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:50:19 smithi092 ceph-mon[29572]: osdmap e548: 8 total, 8 up, 8 in 2024-03-11T21:50:19.697 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:50:19 smithi092 ceph-mon[37563]: from='client.? v1:172.21.15.92:0/3916171637' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]': finished 2024-03-11T21:50:19.697 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:50:19 smithi092 ceph-mon[37563]: osdmap e548: 8 total, 8 up, 8 in 2024-03-11T21:50:19.836 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:50:19 smithi097 ceph-mon[33317]: from='client.? v1:172.21.15.92:0/3916171637' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]': finished 2024-03-11T21:50:19.836 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:50:19 smithi097 ceph-mon[33317]: osdmap e548: 8 total, 8 up, 8 in 2024-03-11T21:50:20.535 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:50:20 smithi092 ceph-mon[29572]: 6.8 scrub starts 2024-03-11T21:50:20.535 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:50:20 smithi092 ceph-mon[29572]: 6.8 scrub ok 2024-03-11T21:50:20.535 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:50:20 smithi092 ceph-mon[29572]: 2.1b scrub starts 2024-03-11T21:50:20.535 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:50:20 smithi092 ceph-mon[29572]: 2.1b scrub ok 2024-03-11T21:50:20.535 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:50:20 smithi092 ceph-mon[29572]: pgmap v1183: 132 pgs: 132 active+clean; 582 KiB data, 6.3 GiB used, 709 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T21:50:20.535 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:50:20 smithi092 ceph-mon[37563]: 6.8 scrub starts 2024-03-11T21:50:20.536 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:50:20 smithi092 ceph-mon[37563]: 6.8 scrub ok 2024-03-11T21:50:20.536 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:50:20 smithi092 ceph-mon[37563]: 2.1b scrub starts 2024-03-11T21:50:20.536 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:50:20 smithi092 ceph-mon[37563]: 2.1b scrub ok 2024-03-11T21:50:20.536 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:50:20 smithi092 ceph-mon[37563]: pgmap v1183: 132 pgs: 132 active+clean; 582 KiB data, 6.3 GiB used, 709 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T21:50:20.836 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:50:20 smithi097 ceph-mon[33317]: 6.8 scrub starts 2024-03-11T21:50:20.836 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:50:20 smithi097 ceph-mon[33317]: 6.8 scrub ok 2024-03-11T21:50:20.836 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:50:20 smithi097 ceph-mon[33317]: 2.1b scrub starts 2024-03-11T21:50:20.836 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:50:20 smithi097 ceph-mon[33317]: 2.1b scrub ok 2024-03-11T21:50:20.836 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:50:20 smithi097 ceph-mon[33317]: pgmap v1183: 132 pgs: 132 active+clean; 582 KiB data, 6.3 GiB used, 709 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T21:50:21.420 INFO:teuthology.orchestra.run.smithi092.stderr:set osd.7 primary-affinity to 1 (8655362) 2024-03-11T21:50:21.741 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:50:21 smithi092 ceph-mon[29572]: 5.9 scrub starts 2024-03-11T21:50:21.742 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:50:21 smithi092 ceph-mon[29572]: 5.9 scrub ok 2024-03-11T21:50:21.742 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:50:21 smithi092 ceph-mon[29572]: from='client.? v1:172.21.15.92:0/3490028265' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 7, "weight": 1.0}]: dispatch 2024-03-11T21:50:21.742 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:50:21 smithi092 ceph-mon[37563]: 5.9 scrub starts 2024-03-11T21:50:21.742 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:50:21 smithi092 ceph-mon[37563]: 5.9 scrub ok 2024-03-11T21:50:21.742 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:50:21 smithi092 ceph-mon[37563]: from='client.? v1:172.21.15.92:0/3490028265' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 7, "weight": 1.0}]: dispatch 2024-03-11T21:50:21.836 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:50:21 smithi097 ceph-mon[33317]: 5.9 scrub starts 2024-03-11T21:50:21.836 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:50:21 smithi097 ceph-mon[33317]: 5.9 scrub ok 2024-03-11T21:50:21.836 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:50:21 smithi097 ceph-mon[33317]: from='client.? v1:172.21.15.92:0/3490028265' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 7, "weight": 1.0}]: dispatch 2024-03-11T21:50:22.073 DEBUG:teuthology.orchestra.run.smithi092:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid 00db0016-dfed-11ee-95c8-87774f69a715 -- ceph osd reweight 3 1 2024-03-11T21:50:22.315 INFO:teuthology.orchestra.run.smithi092.stderr:Inferring config /var/lib/ceph/00db0016-dfed-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T21:50:22.746 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:50:22 smithi092 ceph-mon[37563]: 5.c scrub starts 2024-03-11T21:50:22.746 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:50:22 smithi092 ceph-mon[37563]: 5.c scrub ok 2024-03-11T21:50:22.746 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:50:22 smithi092 ceph-mon[37563]: 2.b deep-scrub starts 2024-03-11T21:50:22.746 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:50:22 smithi092 ceph-mon[37563]: 2.b deep-scrub ok 2024-03-11T21:50:22.746 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:50:22 smithi092 ceph-mon[37563]: 6.13 scrub starts 2024-03-11T21:50:22.747 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:50:22 smithi092 ceph-mon[37563]: from='client.? v1:172.21.15.92:0/3490028265' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 7, "weight": 1.0}]': finished 2024-03-11T21:50:22.747 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:50:22 smithi092 ceph-mon[37563]: osdmap e549: 8 total, 8 up, 8 in 2024-03-11T21:50:22.747 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:50:22 smithi092 ceph-mon[37563]: 6.13 scrub ok 2024-03-11T21:50:22.747 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:50:22 smithi092 ceph-mon[37563]: pgmap v1185: 132 pgs: 132 active+clean; 582 KiB data, 6.3 GiB used, 709 GiB / 715 GiB avail; 832 B/s rd, 0 op/s 2024-03-11T21:50:22.747 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:50:22 smithi092 ceph-mon[29572]: 5.c scrub starts 2024-03-11T21:50:22.747 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:50:22 smithi092 ceph-mon[29572]: 5.c scrub ok 2024-03-11T21:50:22.747 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:50:22 smithi092 ceph-mon[29572]: 2.b deep-scrub starts 2024-03-11T21:50:22.747 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:50:22 smithi092 ceph-mon[29572]: 2.b deep-scrub ok 2024-03-11T21:50:22.747 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:50:22 smithi092 ceph-mon[29572]: 6.13 scrub starts 2024-03-11T21:50:22.747 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:50:22 smithi092 ceph-mon[29572]: from='client.? v1:172.21.15.92:0/3490028265' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 7, "weight": 1.0}]': finished 2024-03-11T21:50:22.747 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:50:22 smithi092 ceph-mon[29572]: osdmap e549: 8 total, 8 up, 8 in 2024-03-11T21:50:22.747 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:50:22 smithi092 ceph-mon[29572]: 6.13 scrub ok 2024-03-11T21:50:22.748 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:50:22 smithi092 ceph-mon[29572]: pgmap v1185: 132 pgs: 132 active+clean; 582 KiB data, 6.3 GiB used, 709 GiB / 715 GiB avail; 832 B/s rd, 0 op/s 2024-03-11T21:50:22.836 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:50:22 smithi097 ceph-mon[33317]: 5.c scrub starts 2024-03-11T21:50:22.836 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:50:22 smithi097 ceph-mon[33317]: 5.c scrub ok 2024-03-11T21:50:22.836 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:50:22 smithi097 ceph-mon[33317]: 2.b deep-scrub starts 2024-03-11T21:50:22.836 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:50:22 smithi097 ceph-mon[33317]: 2.b deep-scrub ok 2024-03-11T21:50:22.836 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:50:22 smithi097 ceph-mon[33317]: 6.13 scrub starts 2024-03-11T21:50:22.836 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:50:22 smithi097 ceph-mon[33317]: from='client.? v1:172.21.15.92:0/3490028265' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 7, "weight": 1.0}]': finished 2024-03-11T21:50:22.836 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:50:22 smithi097 ceph-mon[33317]: osdmap e549: 8 total, 8 up, 8 in 2024-03-11T21:50:22.836 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:50:22 smithi097 ceph-mon[33317]: 6.13 scrub ok 2024-03-11T21:50:22.836 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:50:22 smithi097 ceph-mon[33317]: pgmap v1185: 132 pgs: 132 active+clean; 582 KiB data, 6.3 GiB used, 709 GiB / 715 GiB avail; 832 B/s rd, 0 op/s 2024-03-11T21:50:23.732 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:50:23 smithi092 ceph-mon[37563]: 2.e deep-scrub starts 2024-03-11T21:50:23.732 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:50:23 smithi092 ceph-mon[37563]: 2.e deep-scrub ok 2024-03-11T21:50:23.732 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:50:23 smithi092 ceph-mon[37563]: 2.4 scrub starts 2024-03-11T21:50:23.732 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:50:23 smithi092 ceph-mon[37563]: 2.4 scrub ok 2024-03-11T21:50:23.732 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:50:23 smithi092 ceph-mon[37563]: 5.11 deep-scrub starts 2024-03-11T21:50:23.732 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:50:23 smithi092 ceph-mon[37563]: 5.11 deep-scrub ok 2024-03-11T21:50:23.732 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:50:23 smithi092 ceph-mon[37563]: from='mgr.14607 v1:172.21.15.92:0/1702717597' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T21:50:23.732 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:50:23 smithi092 ceph-mon[29572]: 2.e deep-scrub starts 2024-03-11T21:50:23.733 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:50:23 smithi092 ceph-mon[29572]: 2.e deep-scrub ok 2024-03-11T21:50:23.733 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:50:23 smithi092 ceph-mon[29572]: 2.4 scrub starts 2024-03-11T21:50:23.733 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:50:23 smithi092 ceph-mon[29572]: 2.4 scrub ok 2024-03-11T21:50:23.733 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:50:23 smithi092 ceph-mon[29572]: 5.11 deep-scrub starts 2024-03-11T21:50:23.733 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:50:23 smithi092 ceph-mon[29572]: 5.11 deep-scrub ok 2024-03-11T21:50:23.733 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:50:23 smithi092 ceph-mon[29572]: from='mgr.14607 v1:172.21.15.92:0/1702717597' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T21:50:23.737 INFO:journalctl@ceph.iscsi.iscsi.a.smithi097.stdout:Mar 11 21:50:23 smithi097 ceph-00db0016-dfed-11ee-95c8-87774f69a715-iscsi-iscsi-a[60943]: debug there is no tcmu-runner data available 2024-03-11T21:50:23.737 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:50:23 smithi097 ceph-mon[33317]: 2.e deep-scrub starts 2024-03-11T21:50:23.737 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:50:23 smithi097 ceph-mon[33317]: 2.e deep-scrub ok 2024-03-11T21:50:23.737 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:50:23 smithi097 ceph-mon[33317]: 2.4 scrub starts 2024-03-11T21:50:23.737 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:50:23 smithi097 ceph-mon[33317]: 2.4 scrub ok 2024-03-11T21:50:23.737 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:50:23 smithi097 ceph-mon[33317]: 5.11 deep-scrub starts 2024-03-11T21:50:23.737 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:50:23 smithi097 ceph-mon[33317]: 5.11 deep-scrub ok 2024-03-11T21:50:23.738 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:50:23 smithi097 ceph-mon[33317]: from='mgr.14607 v1:172.21.15.92:0/1702717597' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T21:50:24.446 INFO:teuthology.orchestra.run.smithi092.stderr:reweighted osd.3 to 1 (10000) 2024-03-11T21:50:24.741 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:50:24 smithi092 ceph-mon[29572]: 2.1a scrub starts 2024-03-11T21:50:24.742 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:50:24 smithi092 ceph-mon[29572]: 2.1a scrub ok 2024-03-11T21:50:24.742 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:50:24 smithi092 ceph-mon[29572]: from='client.24469 v1:172.21.15.97:0/404208310' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:50:24.742 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:50:24 smithi092 ceph-mon[29572]: pgmap v1186: 132 pgs: 132 active+clean; 582 KiB data, 6.3 GiB used, 709 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-03-11T21:50:24.742 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:50:24 smithi092 ceph-mon[29572]: from='client.? v1:172.21.15.92:0/2285787927' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]: dispatch 2024-03-11T21:50:24.742 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:50:24 smithi092 ceph-mon[29572]: from='client.36212 ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]: dispatch 2024-03-11T21:50:24.742 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:50:24 smithi092 ceph-mon[37563]: 2.1a scrub starts 2024-03-11T21:50:24.742 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:50:24 smithi092 ceph-mon[37563]: 2.1a scrub ok 2024-03-11T21:50:24.742 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:50:24 smithi092 ceph-mon[37563]: from='client.24469 v1:172.21.15.97:0/404208310' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:50:24.742 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:50:24 smithi092 ceph-mon[37563]: pgmap v1186: 132 pgs: 132 active+clean; 582 KiB data, 6.3 GiB used, 709 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-03-11T21:50:24.742 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:50:24 smithi092 ceph-mon[37563]: from='client.? v1:172.21.15.92:0/2285787927' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]: dispatch 2024-03-11T21:50:24.742 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:50:24 smithi092 ceph-mon[37563]: from='client.36212 ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]: dispatch 2024-03-11T21:50:24.836 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:50:24 smithi097 ceph-mon[33317]: 2.1a scrub starts 2024-03-11T21:50:24.836 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:50:24 smithi097 ceph-mon[33317]: 2.1a scrub ok 2024-03-11T21:50:24.836 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:50:24 smithi097 ceph-mon[33317]: from='client.24469 v1:172.21.15.97:0/404208310' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:50:24.836 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:50:24 smithi097 ceph-mon[33317]: pgmap v1186: 132 pgs: 132 active+clean; 582 KiB data, 6.3 GiB used, 709 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-03-11T21:50:24.836 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:50:24 smithi097 ceph-mon[33317]: from='client.? v1:172.21.15.92:0/2285787927' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]: dispatch 2024-03-11T21:50:24.836 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:50:24 smithi097 ceph-mon[33317]: from='client.36212 ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]: dispatch 2024-03-11T21:50:25.129 DEBUG:teuthology.orchestra.run.smithi092:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid 00db0016-dfed-11ee-95c8-87774f69a715 -- ceph osd primary-affinity 3 1 2024-03-11T21:50:25.373 INFO:teuthology.orchestra.run.smithi092.stderr:Inferring config /var/lib/ceph/00db0016-dfed-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T21:50:25.729 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:50:25 smithi092 ceph-mon[29572]: purged_snaps scrub starts 2024-03-11T21:50:25.729 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:50:25 smithi092 ceph-mon[29572]: purged_snaps scrub ok 2024-03-11T21:50:25.729 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:50:25 smithi092 ceph-mon[29572]: 4.9 scrub starts 2024-03-11T21:50:25.729 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:50:25 smithi092 ceph-mon[29572]: 4.9 scrub ok 2024-03-11T21:50:25.729 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:50:25 smithi092 ceph-mon[29572]: 5.3 scrub starts 2024-03-11T21:50:25.729 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:50:25 smithi092 ceph-mon[29572]: 5.3 scrub ok 2024-03-11T21:50:25.730 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:50:25 smithi092 ceph-mon[29572]: from='client.36212 ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]': finished 2024-03-11T21:50:25.730 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:50:25 smithi092 ceph-mon[29572]: osdmap e550: 8 total, 8 up, 8 in 2024-03-11T21:50:25.730 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:50:25 smithi092 ceph-mon[37563]: purged_snaps scrub starts 2024-03-11T21:50:25.730 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:50:25 smithi092 ceph-mon[37563]: purged_snaps scrub ok 2024-03-11T21:50:25.730 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:50:25 smithi092 ceph-mon[37563]: 4.9 scrub starts 2024-03-11T21:50:25.730 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:50:25 smithi092 ceph-mon[37563]: 4.9 scrub ok 2024-03-11T21:50:25.730 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:50:25 smithi092 ceph-mon[37563]: 5.3 scrub starts 2024-03-11T21:50:25.730 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:50:25 smithi092 ceph-mon[37563]: 5.3 scrub ok 2024-03-11T21:50:25.730 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:50:25 smithi092 ceph-mon[37563]: from='client.36212 ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]': finished 2024-03-11T21:50:25.730 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:50:25 smithi092 ceph-mon[37563]: osdmap e550: 8 total, 8 up, 8 in 2024-03-11T21:50:25.836 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:50:25 smithi097 ceph-mon[33317]: purged_snaps scrub starts 2024-03-11T21:50:25.836 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:50:25 smithi097 ceph-mon[33317]: purged_snaps scrub ok 2024-03-11T21:50:25.836 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:50:25 smithi097 ceph-mon[33317]: 4.9 scrub starts 2024-03-11T21:50:25.836 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:50:25 smithi097 ceph-mon[33317]: 4.9 scrub ok 2024-03-11T21:50:25.836 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:50:25 smithi097 ceph-mon[33317]: 5.3 scrub starts 2024-03-11T21:50:25.836 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:50:25 smithi097 ceph-mon[33317]: 5.3 scrub ok 2024-03-11T21:50:25.836 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:50:25 smithi097 ceph-mon[33317]: from='client.36212 ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]': finished 2024-03-11T21:50:25.836 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:50:25 smithi097 ceph-mon[33317]: osdmap e550: 8 total, 8 up, 8 in 2024-03-11T21:50:26.516 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:50:26 smithi092 ceph-mon[29572]: pgmap v1188: 132 pgs: 132 active+clean; 582 KiB data, 6.3 GiB used, 709 GiB / 715 GiB avail; 1.4 KiB/s rd, 1 op/s 2024-03-11T21:50:26.517 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:50:26 smithi092 ceph-mon[37563]: pgmap v1188: 132 pgs: 132 active+clean; 582 KiB data, 6.3 GiB used, 709 GiB / 715 GiB avail; 1.4 KiB/s rd, 1 op/s 2024-03-11T21:50:26.836 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:50:26 smithi097 ceph-mon[33317]: pgmap v1188: 132 pgs: 132 active+clean; 582 KiB data, 6.3 GiB used, 709 GiB / 715 GiB avail; 1.4 KiB/s rd, 1 op/s 2024-03-11T21:50:27.836 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:50:27 smithi097 ceph-mon[33317]: 2.8 scrub starts 2024-03-11T21:50:27.836 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:50:27 smithi097 ceph-mon[33317]: 2.1f scrub starts 2024-03-11T21:50:27.836 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:50:27 smithi097 ceph-mon[33317]: 2.8 scrub ok 2024-03-11T21:50:27.836 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:50:27 smithi097 ceph-mon[33317]: 2.1f scrub ok 2024-03-11T21:50:27.836 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:50:27 smithi097 ceph-mon[33317]: purged_snaps scrub starts 2024-03-11T21:50:27.836 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:50:27 smithi097 ceph-mon[33317]: purged_snaps scrub ok 2024-03-11T21:50:27.836 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:50:27 smithi097 ceph-mon[33317]: 5.15 scrub starts 2024-03-11T21:50:27.836 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:50:27 smithi097 ceph-mon[33317]: 5.15 scrub ok 2024-03-11T21:50:27.870 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:50:27 smithi092 ceph-mon[37563]: 2.8 scrub starts 2024-03-11T21:50:27.870 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:50:27 smithi092 ceph-mon[37563]: 2.1f scrub starts 2024-03-11T21:50:27.870 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:50:27 smithi092 ceph-mon[37563]: 2.8 scrub ok 2024-03-11T21:50:27.870 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:50:27 smithi092 ceph-mon[37563]: 2.1f scrub ok 2024-03-11T21:50:27.870 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:50:27 smithi092 ceph-mon[37563]: purged_snaps scrub starts 2024-03-11T21:50:27.870 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:50:27 smithi092 ceph-mon[37563]: purged_snaps scrub ok 2024-03-11T21:50:27.870 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:50:27 smithi092 ceph-mon[37563]: 5.15 scrub starts 2024-03-11T21:50:27.870 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:50:27 smithi092 ceph-mon[37563]: 5.15 scrub ok 2024-03-11T21:50:27.871 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:50:27 smithi092 ceph-mon[29572]: 2.8 scrub starts 2024-03-11T21:50:27.871 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:50:27 smithi092 ceph-mon[29572]: 2.1f scrub starts 2024-03-11T21:50:27.871 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:50:27 smithi092 ceph-mon[29572]: 2.8 scrub ok 2024-03-11T21:50:27.871 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:50:27 smithi092 ceph-mon[29572]: 2.1f scrub ok 2024-03-11T21:50:27.871 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:50:27 smithi092 ceph-mon[29572]: purged_snaps scrub starts 2024-03-11T21:50:27.871 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:50:27 smithi092 ceph-mon[29572]: purged_snaps scrub ok 2024-03-11T21:50:27.871 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:50:27 smithi092 ceph-mon[29572]: 5.15 scrub starts 2024-03-11T21:50:27.871 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:50:27 smithi092 ceph-mon[29572]: 5.15 scrub ok 2024-03-11T21:50:28.370 INFO:journalctl@ceph.mgr.y.smithi092.stdout:Mar 11 21:50:27 smithi092 ceph-00db0016-dfed-11ee-95c8-87774f69a715-mgr-y[29801]: ::ffff:172.21.15.97 - - [11/Mar/2024:21:50:27] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-11T21:50:28.486 INFO:teuthology.orchestra.run.smithi092.stderr:set osd.3 primary-affinity to 1 (8655362) 2024-03-11T21:50:28.836 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:50:28 smithi097 ceph-mon[33317]: 6.f scrub starts 2024-03-11T21:50:28.836 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:50:28 smithi097 ceph-mon[33317]: 6.f scrub ok 2024-03-11T21:50:28.836 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:50:28 smithi097 ceph-mon[33317]: osdmap e551: 8 total, 8 up, 8 in 2024-03-11T21:50:28.836 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:50:28 smithi097 ceph-mon[33317]: from='client.? v1:172.21.15.92:0/3141812881' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 3, "weight": 1.0}]: dispatch 2024-03-11T21:50:28.836 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:50:28 smithi097 ceph-mon[33317]: from='client.36218 ' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 3, "weight": 1.0}]: dispatch 2024-03-11T21:50:28.836 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:50:28 smithi097 ceph-mon[33317]: pgmap v1190: 132 pgs: 132 active+clean; 582 KiB data, 6.3 GiB used, 709 GiB / 715 GiB avail; 836 B/s rd, 0 op/s 2024-03-11T21:50:28.836 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:50:28 smithi097 ceph-mon[33317]: 5.b scrub starts 2024-03-11T21:50:28.836 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:50:28 smithi097 ceph-mon[33317]: 5.b scrub ok 2024-03-11T21:50:28.870 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:50:28 smithi092 ceph-mon[37563]: 6.f scrub starts 2024-03-11T21:50:28.870 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:50:28 smithi092 ceph-mon[37563]: 6.f scrub ok 2024-03-11T21:50:28.870 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:50:28 smithi092 ceph-mon[37563]: osdmap e551: 8 total, 8 up, 8 in 2024-03-11T21:50:28.870 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:50:28 smithi092 ceph-mon[37563]: from='client.? v1:172.21.15.92:0/3141812881' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 3, "weight": 1.0}]: dispatch 2024-03-11T21:50:28.870 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:50:28 smithi092 ceph-mon[37563]: from='client.36218 ' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 3, "weight": 1.0}]: dispatch 2024-03-11T21:50:28.870 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:50:28 smithi092 ceph-mon[37563]: pgmap v1190: 132 pgs: 132 active+clean; 582 KiB data, 6.3 GiB used, 709 GiB / 715 GiB avail; 836 B/s rd, 0 op/s 2024-03-11T21:50:28.870 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:50:28 smithi092 ceph-mon[37563]: 5.b scrub starts 2024-03-11T21:50:28.870 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:50:28 smithi092 ceph-mon[37563]: 5.b scrub ok 2024-03-11T21:50:28.870 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:50:28 smithi092 ceph-mon[29572]: 6.f scrub starts 2024-03-11T21:50:28.870 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:50:28 smithi092 ceph-mon[29572]: 6.f scrub ok 2024-03-11T21:50:28.871 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:50:28 smithi092 ceph-mon[29572]: osdmap e551: 8 total, 8 up, 8 in 2024-03-11T21:50:28.871 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:50:28 smithi092 ceph-mon[29572]: from='client.? v1:172.21.15.92:0/3141812881' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 3, "weight": 1.0}]: dispatch 2024-03-11T21:50:28.871 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:50:28 smithi092 ceph-mon[29572]: from='client.36218 ' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 3, "weight": 1.0}]: dispatch 2024-03-11T21:50:28.871 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:50:28 smithi092 ceph-mon[29572]: pgmap v1190: 132 pgs: 132 active+clean; 582 KiB data, 6.3 GiB used, 709 GiB / 715 GiB avail; 836 B/s rd, 0 op/s 2024-03-11T21:50:28.871 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:50:28 smithi092 ceph-mon[29572]: 5.b scrub starts 2024-03-11T21:50:28.871 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:50:28 smithi092 ceph-mon[29572]: 5.b scrub ok 2024-03-11T21:50:29.355 INFO:tasks.thrashosds.thrasher:joining the do_sighup greenlet 2024-03-11T21:50:29.355 INFO:tasks.thrashosds.thrasher:joining the do_optrack_toggle greenlet 2024-03-11T21:50:29.355 INFO:tasks.thrashosds.thrasher:joining the do_dump_ops greenlet 2024-03-11T21:50:29.356 INFO:tasks.thrashosds.thrasher:joining the do_noscrub_toggle greenlet 2024-03-11T21:50:29.356 INFO:tasks.cephadm.ceph_manager.ceph:waiting for all up 2024-03-11T21:50:29.356 DEBUG:teuthology.orchestra.run.smithi092:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid 00db0016-dfed-11ee-95c8-87774f69a715 -- ceph osd dump --format=json 2024-03-11T21:50:29.605 INFO:teuthology.orchestra.run.smithi092.stderr:Inferring config /var/lib/ceph/00db0016-dfed-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T21:50:29.619 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:50:29 smithi092 ceph-mon[37563]: from='client.36218 ' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 3, "weight": 1.0}]': finished 2024-03-11T21:50:29.620 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:50:29 smithi092 ceph-mon[37563]: osdmap e552: 8 total, 8 up, 8 in 2024-03-11T21:50:29.620 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:50:29 smithi092 ceph-mon[29572]: from='client.36218 ' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 3, "weight": 1.0}]': finished 2024-03-11T21:50:29.620 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:50:29 smithi092 ceph-mon[29572]: osdmap e552: 8 total, 8 up, 8 in 2024-03-11T21:50:29.836 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:50:29 smithi097 ceph-mon[33317]: from='client.36218 ' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 3, "weight": 1.0}]': finished 2024-03-11T21:50:29.836 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:50:29 smithi097 ceph-mon[33317]: osdmap e552: 8 total, 8 up, 8 in 2024-03-11T21:50:30.770 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:50:30 smithi092 ceph-mon[37563]: pgmap v1192: 132 pgs: 132 active+clean; 582 KiB data, 6.3 GiB used, 709 GiB / 715 GiB avail; 1.7 KiB/s rd, 1 op/s 2024-03-11T21:50:30.771 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:50:30 smithi092 ceph-mon[29572]: pgmap v1192: 132 pgs: 132 active+clean; 582 KiB data, 6.3 GiB used, 709 GiB / 715 GiB avail; 1.7 KiB/s rd, 1 op/s 2024-03-11T21:50:30.836 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:50:30 smithi097 ceph-mon[33317]: pgmap v1192: 132 pgs: 132 active+clean; 582 KiB data, 6.3 GiB used, 709 GiB / 715 GiB avail; 1.7 KiB/s rd, 1 op/s 2024-03-11T21:50:31.577 INFO:teuthology.orchestra.run.smithi092.stdout: 2024-03-11T21:50:31.577 INFO:teuthology.orchestra.run.smithi092.stdout:{"epoch":553,"fsid":"00db0016-dfed-11ee-95c8-87774f69a715","created":"2024-03-11T21:20:10.488059+0000","modified":"2024-03-11T21:50:31.487236+0000","last_up_change":"2024-03-11T21:36:34.026798+0000","last_in_change":"2024-03-11T21:49:25.608679+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":19,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":11,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-11T21:23:23.805029+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"121","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":7.8899998664855957,"score_stable":7.8899998664855957,"optimal_score":0.37999999523162842,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":".rgw.root","create_time":"2024-03-11T21:25:37.044504+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"68","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rgw":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"datapool","create_time":"2024-03-11T21:25:37.280820+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":3,"pg_placement_num":3,"pg_placement_num_target":3,"pg_num_target":3,"pg_num_pending":3,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"71","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":2,"snap_epoch":71,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_type":"Fair distribution","score_acting":2.6500000953674316,"score_stable":2.6500000953674316,"optimal_score":0.87999999523162842,"raw_score_acting":2.3299999237060547,"raw_score_stable":2.3299999237060547,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"default.rgw.log","create_time":"2024-03-11T21:25:38.850207+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"70","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rgw":{}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"default.rgw.control","create_time":"2024-03-11T21:25:40.794686+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"72","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rgw":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.25,"score_stable":1.25,"optimal_score":1,"raw_score_acting":1.25,"raw_score_stable":1.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":6,"pool_name":"default.rgw.meta","create_time":"2024-03-11T21:25:42.872077+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"74","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4},"application_metadata":{"rgw":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"79f15843-f0ec-44aa-931e-4e07a74d3768","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":532,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.92:6801","nonce":2844100201}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.92:6802","nonce":2844100201}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.92:6804","nonce":2844100201}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.92:6803","nonce":2844100201}]},"public_addr":"172.21.15.92:6801/2844100201","cluster_addr":"172.21.15.92:6802/2844100201","heartbeat_back_addr":"172.21.15.92:6804/2844100201","heartbeat_front_addr":"172.21.15.92:6803/2844100201","state":["exists","up"]},{"osd":1,"uuid":"3a5a0e78-1293-4ace-a93f-69156098571d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":537,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.92:6805","nonce":3577871157}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.92:6806","nonce":3577871157}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.92:6808","nonce":3577871157}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.92:6807","nonce":3577871157}]},"public_addr":"172.21.15.92:6805/3577871157","cluster_addr":"172.21.15.92:6806/3577871157","heartbeat_back_addr":"172.21.15.92:6808/3577871157","heartbeat_front_addr":"172.21.15.92:6807/3577871157","state":["exists","up"]},{"osd":2,"uuid":"f207b743-0df4-40ed-a735-950e272fea60","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":532,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.92:6809","nonce":760825152}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.92:6810","nonce":760825152}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.92:6812","nonce":760825152}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.92:6811","nonce":760825152}]},"public_addr":"172.21.15.92:6809/760825152","cluster_addr":"172.21.15.92:6810/760825152","heartbeat_back_addr":"172.21.15.92:6812/760825152","heartbeat_front_addr":"172.21.15.92:6811/760825152","state":["exists","up"]},{"osd":3,"uuid":"bbb6f69a-8f0f-47e6-937a-062e1902ae13","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":26,"last_clean_end":251,"up_from":277,"up_thru":532,"down_at":252,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.92:6813","nonce":3215041236}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.92:6814","nonce":3215041236}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.92:6816","nonce":3215041236}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.92:6815","nonce":3215041236}]},"public_addr":"172.21.15.92:6813/3215041236","cluster_addr":"172.21.15.92:6814/3215041236","heartbeat_back_addr":"172.21.15.92:6816/3215041236","heartbeat_front_addr":"172.21.15.92:6815/3215041236","state":["exists","up"]},{"osd":4,"uuid":"00266cbc-d2cf-4dee-8a96-3753ebc95ca8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":32,"up_thru":532,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.97:6800","nonce":420733620}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.97:6801","nonce":420733620}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.97:6803","nonce":420733620}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.97:6802","nonce":420733620}]},"public_addr":"172.21.15.97:6800/420733620","cluster_addr":"172.21.15.97:6801/420733620","heartbeat_back_addr":"172.21.15.97:6803/420733620","heartbeat_front_addr":"172.21.15.97:6802/420733620","state":["exists","up"]},{"osd":5,"uuid":"593a6f6b-8874-40ae-8848-d51c5723a5e4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":41,"up_thru":532,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.97:6804","nonce":312056041}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.97:6805","nonce":312056041}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.97:6807","nonce":312056041}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.97:6806","nonce":312056041}]},"public_addr":"172.21.15.97:6804/312056041","cluster_addr":"172.21.15.97:6805/312056041","heartbeat_back_addr":"172.21.15.97:6807/312056041","heartbeat_front_addr":"172.21.15.97:6806/312056041","state":["exists","up"]},{"osd":6,"uuid":"91035ecc-6f1a-4cec-bda6-090af9d53eff","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":50,"up_thru":532,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.97:6808","nonce":178994999}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.97:6809","nonce":178994999}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.97:6811","nonce":178994999}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.97:6810","nonce":178994999}]},"public_addr":"172.21.15.97:6808/178994999","cluster_addr":"172.21.15.97:6809/178994999","heartbeat_back_addr":"172.21.15.97:6811/178994999","heartbeat_front_addr":"172.21.15.97:6810/178994999","state":["exists","up"]},{"osd":7,"uuid":"3eb28c24-664a-4bd6-869d-6bf5db98f52f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":60,"up_thru":532,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.97:6812","nonce":3241560509}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.97:6813","nonce":3241560509}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.97:6815","nonce":3241560509}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.97:6814","nonce":3241560509}]},"public_addr":"172.21.15.97:6812/3241560509","cluster_addr":"172.21.15.97:6813/3241560509","heartbeat_back_addr":"172.21.15.97:6815/3241560509","heartbeat_front_addr":"172.21.15.97:6814/3241560509","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-11T21:50:14.615645+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-11T21:50:01.522744+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-11T21:49:57.149486+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-11T21:50:31.019949+0000","dead_epoch":251},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-11T21:50:10.926268+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-11T21:50:10.502682+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-11T21:50:23.927539+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-11T21:50:26.698543+0000","dead_epoch":0}],"pg_upmap":[{"pgid":"6.a","osds":[4,0,3]}],"pg_upmap_items":[{"pgid":"2.18","mappings":[{"from":7,"to":2}]},{"pgid":"4.b","mappings":[{"from":1,"to":2}]},{"pgid":"4.f","mappings":[{"from":1,"to":2}]},{"pgid":"4.11","mappings":[{"from":3,"to":5}]},{"pgid":"4.13","mappings":[{"from":1,"to":2}]},{"pgid":"4.15","mappings":[{"from":3,"to":2}]},{"pgid":"4.1f","mappings":[{"from":1,"to":0}]},{"pgid":"5.d","mappings":[{"from":7,"to":6}]},{"pgid":"6.1f","mappings":[{"from":6,"to":0}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.92:0/303695351":"2024-03-12T21:26:20.943013+0000","172.21.15.92:0/437911229":"2024-03-12T21:26:20.943013+0000","172.21.15.92:0/4246054050":"2024-03-12T21:26:20.943013+0000","172.21.15.92:0/2912406656":"2024-03-12T21:26:20.943013+0000","172.21.15.92:0/2706821241":"2024-03-12T21:26:20.943013+0000","172.21.15.92:0/3817549754":"2024-03-12T21:21:09.473916+0000","172.21.15.92:0/2211594569":"2024-03-12T21:21:09.473916+0000","172.21.15.92:0/3569900179":"2024-03-12T21:20:43.677016+0000","172.21.15.92:0/3380359534":"2024-03-12T21:20:43.677016+0000","172.21.15.92:0/2509373130":"2024-03-12T21:20:43.677016+0000","172.21.15.92:6800/2343383852":"2024-03-12T21:26:20.943013+0000","172.21.15.92:6800/1559007462":"2024-03-12T21:20:43.677016+0000","172.21.15.92:6800/1270902819":"2024-03-12T21:21:09.473916+0000","172.21.15.92:0/3562645553":"2024-03-12T21:21:09.473916+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-11T21:50:31.836 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:50:31 smithi097 ceph-mon[33317]: 5.4 scrub starts 2024-03-11T21:50:31.836 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:50:31 smithi097 ceph-mon[33317]: 5.4 scrub ok 2024-03-11T21:50:31.836 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:50:31 smithi097 ceph-mon[33317]: 4.8 scrub starts 2024-03-11T21:50:31.836 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:50:31 smithi097 ceph-mon[33317]: 4.8 scrub ok 2024-03-11T21:50:31.836 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:50:31 smithi097 ceph-mon[33317]: purged_snaps scrub starts 2024-03-11T21:50:31.836 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:50:31 smithi097 ceph-mon[33317]: purged_snaps scrub ok 2024-03-11T21:50:31.870 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:50:31 smithi092 ceph-mon[37563]: 5.4 scrub starts 2024-03-11T21:50:31.870 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:50:31 smithi092 ceph-mon[37563]: 5.4 scrub ok 2024-03-11T21:50:31.870 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:50:31 smithi092 ceph-mon[37563]: 4.8 scrub starts 2024-03-11T21:50:31.870 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:50:31 smithi092 ceph-mon[37563]: 4.8 scrub ok 2024-03-11T21:50:31.870 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:50:31 smithi092 ceph-mon[37563]: purged_snaps scrub starts 2024-03-11T21:50:31.870 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:50:31 smithi092 ceph-mon[37563]: purged_snaps scrub ok 2024-03-11T21:50:31.870 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:50:31 smithi092 ceph-mon[29572]: 5.4 scrub starts 2024-03-11T21:50:31.870 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:50:31 smithi092 ceph-mon[29572]: 5.4 scrub ok 2024-03-11T21:50:31.871 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:50:31 smithi092 ceph-mon[29572]: 4.8 scrub starts 2024-03-11T21:50:31.871 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:50:31 smithi092 ceph-mon[29572]: 4.8 scrub ok 2024-03-11T21:50:31.871 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:50:31 smithi092 ceph-mon[29572]: purged_snaps scrub starts 2024-03-11T21:50:31.871 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:50:31 smithi092 ceph-mon[29572]: purged_snaps scrub ok 2024-03-11T21:50:32.372 INFO:tasks.cephadm.ceph_manager.ceph:all up! 2024-03-11T21:50:32.372 DEBUG:teuthology.orchestra.run.smithi092:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid 00db0016-dfed-11ee-95c8-87774f69a715 -- ceph osd dump --format=json 2024-03-11T21:50:32.618 INFO:teuthology.orchestra.run.smithi092.stderr:Inferring config /var/lib/ceph/00db0016-dfed-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T21:50:32.620 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:50:32 smithi092 ceph-mon[29572]: 5.d scrub starts 2024-03-11T21:50:32.620 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:50:32 smithi092 ceph-mon[29572]: 5.d scrub ok 2024-03-11T21:50:32.620 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:50:32 smithi092 ceph-mon[29572]: 4.1b scrub starts 2024-03-11T21:50:32.620 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:50:32 smithi092 ceph-mon[29572]: 4.1b scrub ok 2024-03-11T21:50:32.620 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:50:32 smithi092 ceph-mon[29572]: 4.b scrub starts 2024-03-11T21:50:32.620 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:50:32 smithi092 ceph-mon[29572]: 4.b scrub ok 2024-03-11T21:50:32.620 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:50:32 smithi092 ceph-mon[29572]: osdmap e553: 8 total, 8 up, 8 in 2024-03-11T21:50:32.620 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:50:32 smithi092 ceph-mon[29572]: pgmap v1194: 132 pgs: 132 active+clean; 582 KiB data, 6.3 GiB used, 709 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-11T21:50:32.620 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:50:32 smithi092 ceph-mon[29572]: from='client.? v1:172.21.15.92:0/1952801296' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-11T21:50:32.620 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:50:32 smithi092 ceph-mon[29572]: 6.11 deep-scrub starts 2024-03-11T21:50:32.620 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:50:32 smithi092 ceph-mon[29572]: 6.11 deep-scrub ok 2024-03-11T21:50:32.621 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:50:32 smithi092 ceph-mon[37563]: 5.d scrub starts 2024-03-11T21:50:32.621 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:50:32 smithi092 ceph-mon[37563]: 5.d scrub ok 2024-03-11T21:50:32.621 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:50:32 smithi092 ceph-mon[37563]: 4.1b scrub starts 2024-03-11T21:50:32.621 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:50:32 smithi092 ceph-mon[37563]: 4.1b scrub ok 2024-03-11T21:50:32.621 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:50:32 smithi092 ceph-mon[37563]: 4.b scrub starts 2024-03-11T21:50:32.621 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:50:32 smithi092 ceph-mon[37563]: 4.b scrub ok 2024-03-11T21:50:32.621 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:50:32 smithi092 ceph-mon[37563]: osdmap e553: 8 total, 8 up, 8 in 2024-03-11T21:50:32.621 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:50:32 smithi092 ceph-mon[37563]: pgmap v1194: 132 pgs: 132 active+clean; 582 KiB data, 6.3 GiB used, 709 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-11T21:50:32.621 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:50:32 smithi092 ceph-mon[37563]: from='client.? v1:172.21.15.92:0/1952801296' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-11T21:50:32.621 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:50:32 smithi092 ceph-mon[37563]: 6.11 deep-scrub starts 2024-03-11T21:50:32.621 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:50:32 smithi092 ceph-mon[37563]: 6.11 deep-scrub ok 2024-03-11T21:50:32.836 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:50:32 smithi097 ceph-mon[33317]: 5.d scrub starts 2024-03-11T21:50:32.836 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:50:32 smithi097 ceph-mon[33317]: 5.d scrub ok 2024-03-11T21:50:32.836 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:50:32 smithi097 ceph-mon[33317]: 4.1b scrub starts 2024-03-11T21:50:32.836 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:50:32 smithi097 ceph-mon[33317]: 4.1b scrub ok 2024-03-11T21:50:32.836 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:50:32 smithi097 ceph-mon[33317]: 4.b scrub starts 2024-03-11T21:50:32.836 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:50:32 smithi097 ceph-mon[33317]: 4.b scrub ok 2024-03-11T21:50:32.836 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:50:32 smithi097 ceph-mon[33317]: osdmap e553: 8 total, 8 up, 8 in 2024-03-11T21:50:32.836 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:50:32 smithi097 ceph-mon[33317]: pgmap v1194: 132 pgs: 132 active+clean; 582 KiB data, 6.3 GiB used, 709 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-11T21:50:32.836 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:50:32 smithi097 ceph-mon[33317]: from='client.? v1:172.21.15.92:0/1952801296' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-11T21:50:32.836 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:50:32 smithi097 ceph-mon[33317]: 6.11 deep-scrub starts 2024-03-11T21:50:32.837 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:50:32 smithi097 ceph-mon[33317]: 6.11 deep-scrub ok 2024-03-11T21:50:33.620 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:50:33 smithi092 ceph-mon[37563]: 4.a scrub starts 2024-03-11T21:50:33.620 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:50:33 smithi092 ceph-mon[37563]: 4.a scrub ok 2024-03-11T21:50:33.620 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:50:33 smithi092 ceph-mon[29572]: 4.a scrub starts 2024-03-11T21:50:33.620 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:50:33 smithi092 ceph-mon[29572]: 4.a scrub ok 2024-03-11T21:50:33.836 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:50:33 smithi097 ceph-mon[33317]: 4.a scrub starts 2024-03-11T21:50:33.836 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:50:33 smithi097 ceph-mon[33317]: 4.a scrub ok 2024-03-11T21:50:33.836 INFO:journalctl@ceph.iscsi.iscsi.a.smithi097.stdout:Mar 11 21:50:33 smithi097 ceph-00db0016-dfed-11ee-95c8-87774f69a715-iscsi-iscsi-a[60943]: debug there is no tcmu-runner data available 2024-03-11T21:50:34.739 INFO:teuthology.orchestra.run.smithi092.stdout: 2024-03-11T21:50:34.740 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:50:34 smithi092 ceph-mon[29572]: 4.d scrub starts 2024-03-11T21:50:34.740 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:50:34 smithi092 ceph-mon[29572]: 4.d scrub ok 2024-03-11T21:50:34.740 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:50:34 smithi092 ceph-mon[29572]: 5.5 scrub starts 2024-03-11T21:50:34.740 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:50:34 smithi092 ceph-mon[29572]: 5.5 scrub ok 2024-03-11T21:50:34.740 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:50:34 smithi092 ceph-mon[29572]: from='client.24469 v1:172.21.15.97:0/404208310' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:50:34.740 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:50:34 smithi092 ceph-mon[29572]: pgmap v1195: 132 pgs: 132 active+clean; 582 KiB data, 6.3 GiB used, 709 GiB / 715 GiB avail; 841 B/s rd, 0 op/s 2024-03-11T21:50:34.740 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:50:34 smithi092 ceph-mon[29572]: 5.13 deep-scrub starts 2024-03-11T21:50:34.741 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:50:34 smithi092 ceph-mon[29572]: 5.13 deep-scrub ok 2024-03-11T21:50:34.741 INFO:teuthology.orchestra.run.smithi092.stdout:{"epoch":553,"fsid":"00db0016-dfed-11ee-95c8-87774f69a715","created":"2024-03-11T21:20:10.488059+0000","modified":"2024-03-11T21:50:31.487236+0000","last_up_change":"2024-03-11T21:36:34.026798+0000","last_in_change":"2024-03-11T21:49:25.608679+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":19,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":11,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-11T21:23:23.805029+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"121","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":7.8899998664855957,"score_stable":7.8899998664855957,"optimal_score":0.37999999523162842,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":".rgw.root","create_time":"2024-03-11T21:25:37.044504+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"68","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rgw":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"datapool","create_time":"2024-03-11T21:25:37.280820+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":3,"pg_placement_num":3,"pg_placement_num_target":3,"pg_num_target":3,"pg_num_pending":3,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"71","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":2,"snap_epoch":71,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_type":"Fair distribution","score_acting":2.6500000953674316,"score_stable":2.6500000953674316,"optimal_score":0.87999999523162842,"raw_score_acting":2.3299999237060547,"raw_score_stable":2.3299999237060547,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"default.rgw.log","create_time":"2024-03-11T21:25:38.850207+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"70","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rgw":{}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"default.rgw.control","create_time":"2024-03-11T21:25:40.794686+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"72","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rgw":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.25,"score_stable":1.25,"optimal_score":1,"raw_score_acting":1.25,"raw_score_stable":1.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":6,"pool_name":"default.rgw.meta","create_time":"2024-03-11T21:25:42.872077+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"74","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4},"application_metadata":{"rgw":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"79f15843-f0ec-44aa-931e-4e07a74d3768","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":532,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.92:6801","nonce":2844100201}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.92:6802","nonce":2844100201}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.92:6804","nonce":2844100201}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.92:6803","nonce":2844100201}]},"public_addr":"172.21.15.92:6801/2844100201","cluster_addr":"172.21.15.92:6802/2844100201","heartbeat_back_addr":"172.21.15.92:6804/2844100201","heartbeat_front_addr":"172.21.15.92:6803/2844100201","state":["exists","up"]},{"osd":1,"uuid":"3a5a0e78-1293-4ace-a93f-69156098571d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":537,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.92:6805","nonce":3577871157}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.92:6806","nonce":3577871157}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.92:6808","nonce":3577871157}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.92:6807","nonce":3577871157}]},"public_addr":"172.21.15.92:6805/3577871157","cluster_addr":"172.21.15.92:6806/3577871157","heartbeat_back_addr":"172.21.15.92:6808/3577871157","heartbeat_front_addr":"172.21.15.92:6807/3577871157","state":["exists","up"]},{"osd":2,"uuid":"f207b743-0df4-40ed-a735-950e272fea60","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":532,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.92:6809","nonce":760825152}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.92:6810","nonce":760825152}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.92:6812","nonce":760825152}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.92:6811","nonce":760825152}]},"public_addr":"172.21.15.92:6809/760825152","cluster_addr":"172.21.15.92:6810/760825152","heartbeat_back_addr":"172.21.15.92:6812/760825152","heartbeat_front_addr":"172.21.15.92:6811/760825152","state":["exists","up"]},{"osd":3,"uuid":"bbb6f69a-8f0f-47e6-937a-062e1902ae13","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":26,"last_clean_end":251,"up_from":277,"up_thru":532,"down_at":252,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.92:6813","nonce":3215041236}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.92:6814","nonce":3215041236}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.92:6816","nonce":3215041236}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.92:6815","nonce":3215041236}]},"public_addr":"172.21.15.92:6813/3215041236","cluster_addr":"172.21.15.92:6814/3215041236","heartbeat_back_addr":"172.21.15.92:6816/3215041236","heartbeat_front_addr":"172.21.15.92:6815/3215041236","state":["exists","up"]},{"osd":4,"uuid":"00266cbc-d2cf-4dee-8a96-3753ebc95ca8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":32,"up_thru":532,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.97:6800","nonce":420733620}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.97:6801","nonce":420733620}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.97:6803","nonce":420733620}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.97:6802","nonce":420733620}]},"public_addr":"172.21.15.97:6800/420733620","cluster_addr":"172.21.15.97:6801/420733620","heartbeat_back_addr":"172.21.15.97:6803/420733620","heartbeat_front_addr":"172.21.15.97:6802/420733620","state":["exists","up"]},{"osd":5,"uuid":"593a6f6b-8874-40ae-8848-d51c5723a5e4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":41,"up_thru":532,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.97:6804","nonce":312056041}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.97:6805","nonce":312056041}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.97:6807","nonce":312056041}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.97:6806","nonce":312056041}]},"public_addr":"172.21.15.97:6804/312056041","cluster_addr":"172.21.15.97:6805/312056041","heartbeat_back_addr":"172.21.15.97:6807/312056041","heartbeat_front_addr":"172.21.15.97:6806/312056041","state":["exists","up"]},{"osd":6,"uuid":"91035ecc-6f1a-4cec-bda6-090af9d53eff","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":50,"up_thru":532,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.97:6808","nonce":178994999}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.97:6809","nonce":178994999}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.97:6811","nonce":178994999}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.97:6810","nonce":178994999}]},"public_addr":"172.21.15.97:6808/178994999","cluster_addr":"172.21.15.97:6809/178994999","heartbeat_back_addr":"172.21.15.97:6811/178994999","heartbeat_front_addr":"172.21.15.97:6810/178994999","state":["exists","up"]},{"osd":7,"uuid":"3eb28c24-664a-4bd6-869d-6bf5db98f52f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":60,"up_thru":532,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.97:6812","nonce":3241560509}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.97:6813","nonce":3241560509}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.97:6815","nonce":3241560509}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.97:6814","nonce":3241560509}]},"public_addr":"172.21.15.97:6812/3241560509","cluster_addr":"172.21.15.97:6813/3241560509","heartbeat_back_addr":"172.21.15.97:6815/3241560509","heartbeat_front_addr":"172.21.15.97:6814/3241560509","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-11T21:50:14.615645+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-11T21:50:01.522744+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-11T21:49:57.149486+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-11T21:50:31.019949+0000","dead_epoch":251},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-11T21:50:10.926268+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-11T21:50:10.502682+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-11T21:50:23.927539+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-11T21:50:26.698543+0000","dead_epoch":0}],"pg_upmap":[{"pgid":"6.a","osds":[4,0,3]}],"pg_upmap_items":[{"pgid":"2.18","mappings":[{"from":7,"to":2}]},{"pgid":"4.b","mappings":[{"from":1,"to":2}]},{"pgid":"4.f","mappings":[{"from":1,"to":2}]},{"pgid":"4.11","mappings":[{"from":3,"to":5}]},{"pgid":"4.13","mappings":[{"from":1,"to":2}]},{"pgid":"4.15","mappings":[{"from":3,"to":2}]},{"pgid":"4.1f","mappings":[{"from":1,"to":0}]},{"pgid":"5.d","mappings":[{"from":7,"to":6}]},{"pgid":"6.1f","mappings":[{"from":6,"to":0}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.92:0/303695351":"2024-03-12T21:26:20.943013+0000","172.21.15.92:0/437911229":"2024-03-12T21:26:20.943013+0000","172.21.15.92:0/4246054050":"2024-03-12T21:26:20.943013+0000","172.21.15.92:0/2912406656":"2024-03-12T21:26:20.943013+0000","172.21.15.92:0/2706821241":"2024-03-12T21:26:20.943013+0000","172.21.15.92:0/3817549754":"2024-03-12T21:21:09.473916+0000","172.21.15.92:0/2211594569":"2024-03-12T21:21:09.473916+0000","172.21.15.92:0/3569900179":"2024-03-12T21:20:43.677016+0000","172.21.15.92:0/3380359534":"2024-03-12T21:20:43.677016+0000","172.21.15.92:0/2509373130":"2024-03-12T21:20:43.677016+0000","172.21.15.92:6800/2343383852":"2024-03-12T21:26:20.943013+0000","172.21.15.92:6800/1559007462":"2024-03-12T21:20:43.677016+0000","172.21.15.92:6800/1270902819":"2024-03-12T21:21:09.473916+0000","172.21.15.92:0/3562645553":"2024-03-12T21:21:09.473916+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-11T21:50:34.741 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:50:34 smithi092 ceph-mon[37563]: 4.d scrub starts 2024-03-11T21:50:34.741 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:50:34 smithi092 ceph-mon[37563]: 4.d scrub ok 2024-03-11T21:50:34.741 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:50:34 smithi092 ceph-mon[37563]: 5.5 scrub starts 2024-03-11T21:50:34.741 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:50:34 smithi092 ceph-mon[37563]: 5.5 scrub ok 2024-03-11T21:50:34.741 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:50:34 smithi092 ceph-mon[37563]: from='client.24469 v1:172.21.15.97:0/404208310' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:50:34.742 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:50:34 smithi092 ceph-mon[37563]: pgmap v1195: 132 pgs: 132 active+clean; 582 KiB data, 6.3 GiB used, 709 GiB / 715 GiB avail; 841 B/s rd, 0 op/s 2024-03-11T21:50:34.742 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:50:34 smithi092 ceph-mon[37563]: 5.13 deep-scrub starts 2024-03-11T21:50:34.742 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:50:34 smithi092 ceph-mon[37563]: 5.13 deep-scrub ok 2024-03-11T21:50:34.836 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:50:34 smithi097 ceph-mon[33317]: 4.d scrub starts 2024-03-11T21:50:34.836 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:50:34 smithi097 ceph-mon[33317]: 4.d scrub ok 2024-03-11T21:50:34.836 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:50:34 smithi097 ceph-mon[33317]: 5.5 scrub starts 2024-03-11T21:50:34.836 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:50:34 smithi097 ceph-mon[33317]: 5.5 scrub ok 2024-03-11T21:50:34.836 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:50:34 smithi097 ceph-mon[33317]: from='client.24469 v1:172.21.15.97:0/404208310' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:50:34.836 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:50:34 smithi097 ceph-mon[33317]: pgmap v1195: 132 pgs: 132 active+clean; 582 KiB data, 6.3 GiB used, 709 GiB / 715 GiB avail; 841 B/s rd, 0 op/s 2024-03-11T21:50:34.836 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:50:34 smithi097 ceph-mon[33317]: 5.13 deep-scrub starts 2024-03-11T21:50:34.836 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:50:34 smithi097 ceph-mon[33317]: 5.13 deep-scrub ok 2024-03-11T21:50:35.479 DEBUG:teuthology.orchestra.run.smithi092:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid 00db0016-dfed-11ee-95c8-87774f69a715 -- ceph tell osd.0 flush_pg_stats 2024-03-11T21:50:35.479 DEBUG:teuthology.orchestra.run.smithi092:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid 00db0016-dfed-11ee-95c8-87774f69a715 -- ceph tell osd.1 flush_pg_stats 2024-03-11T21:50:35.480 DEBUG:teuthology.orchestra.run.smithi092:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid 00db0016-dfed-11ee-95c8-87774f69a715 -- ceph tell osd.2 flush_pg_stats 2024-03-11T21:50:35.480 DEBUG:teuthology.orchestra.run.smithi092:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid 00db0016-dfed-11ee-95c8-87774f69a715 -- ceph tell osd.3 flush_pg_stats 2024-03-11T21:50:35.481 DEBUG:teuthology.orchestra.run.smithi092:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid 00db0016-dfed-11ee-95c8-87774f69a715 -- ceph tell osd.4 flush_pg_stats 2024-03-11T21:50:35.481 DEBUG:teuthology.orchestra.run.smithi092:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid 00db0016-dfed-11ee-95c8-87774f69a715 -- ceph tell osd.5 flush_pg_stats 2024-03-11T21:50:35.481 DEBUG:teuthology.orchestra.run.smithi092:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid 00db0016-dfed-11ee-95c8-87774f69a715 -- ceph tell osd.6 flush_pg_stats 2024-03-11T21:50:35.482 DEBUG:teuthology.orchestra.run.smithi092:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid 00db0016-dfed-11ee-95c8-87774f69a715 -- ceph tell osd.7 flush_pg_stats 2024-03-11T21:50:35.836 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:50:35 smithi097 ceph-mon[33317]: 2.6 scrub starts 2024-03-11T21:50:35.836 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:50:35 smithi097 ceph-mon[33317]: 2.6 scrub ok 2024-03-11T21:50:35.836 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:50:35 smithi097 ceph-mon[33317]: from='client.? v1:172.21.15.92:0/2488996765' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-11T21:50:35.869 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:50:35 smithi092 ceph-mon[37563]: 2.6 scrub starts 2024-03-11T21:50:35.870 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:50:35 smithi092 ceph-mon[37563]: 2.6 scrub ok 2024-03-11T21:50:35.870 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:50:35 smithi092 ceph-mon[37563]: from='client.? v1:172.21.15.92:0/2488996765' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-11T21:50:35.870 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:50:35 smithi092 ceph-mon[29572]: 2.6 scrub starts 2024-03-11T21:50:35.870 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:50:35 smithi092 ceph-mon[29572]: 2.6 scrub ok 2024-03-11T21:50:35.870 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:50:35 smithi092 ceph-mon[29572]: from='client.? v1:172.21.15.92:0/2488996765' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-11T21:50:35.967 INFO:teuthology.orchestra.run.smithi092.stderr:Inferring config /var/lib/ceph/00db0016-dfed-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T21:50:36.009 INFO:teuthology.orchestra.run.smithi092.stderr:Inferring config /var/lib/ceph/00db0016-dfed-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T21:50:36.010 INFO:teuthology.orchestra.run.smithi092.stderr:Inferring config /var/lib/ceph/00db0016-dfed-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T21:50:36.011 INFO:teuthology.orchestra.run.smithi092.stderr:Inferring config /var/lib/ceph/00db0016-dfed-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T21:50:36.012 INFO:teuthology.orchestra.run.smithi092.stderr:Inferring config /var/lib/ceph/00db0016-dfed-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T21:50:36.013 INFO:teuthology.orchestra.run.smithi092.stderr:Inferring config /var/lib/ceph/00db0016-dfed-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T21:50:36.019 INFO:teuthology.orchestra.run.smithi092.stderr:Inferring config /var/lib/ceph/00db0016-dfed-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T21:50:36.667 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:50:36 smithi092 ceph-mon[29572]: 6.1 scrub starts 2024-03-11T21:50:36.667 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:50:36 smithi092 ceph-mon[29572]: 6.1 scrub ok 2024-03-11T21:50:36.667 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:50:36 smithi092 ceph-mon[29572]: pgmap v1196: 132 pgs: 132 active+clean; 582 KiB data, 6.3 GiB used, 709 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T21:50:36.668 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:50:36 smithi092 ceph-mon[37563]: 6.1 scrub starts 2024-03-11T21:50:36.668 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:50:36 smithi092 ceph-mon[37563]: 6.1 scrub ok 2024-03-11T21:50:36.668 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:50:36 smithi092 ceph-mon[37563]: pgmap v1196: 132 pgs: 132 active+clean; 582 KiB data, 6.3 GiB used, 709 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T21:50:36.836 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:50:36 smithi097 ceph-mon[33317]: 6.1 scrub starts 2024-03-11T21:50:36.836 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:50:36 smithi097 ceph-mon[33317]: 6.1 scrub ok 2024-03-11T21:50:36.836 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:50:36 smithi097 ceph-mon[33317]: pgmap v1196: 132 pgs: 132 active+clean; 582 KiB data, 6.3 GiB used, 709 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T21:50:38.370 INFO:journalctl@ceph.mgr.y.smithi092.stdout:Mar 11 21:50:37 smithi092 ceph-00db0016-dfed-11ee-95c8-87774f69a715-mgr-y[29801]: ::ffff:172.21.15.97 - - [11/Mar/2024:21:50:37] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-11T21:50:38.713 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:50:38 smithi092 ceph-mon[29572]: pgmap v1197: 132 pgs: 132 active+clean; 582 KiB data, 6.3 GiB used, 709 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-03-11T21:50:38.713 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:50:38 smithi092 ceph-mon[29572]: 4.0 scrub starts 2024-03-11T21:50:38.713 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:50:38 smithi092 ceph-mon[29572]: 4.0 scrub ok 2024-03-11T21:50:38.713 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:50:38 smithi092 ceph-mon[29572]: from='mgr.14607 v1:172.21.15.92:0/1702717597' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T21:50:38.714 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:50:38 smithi092 ceph-mon[37563]: pgmap v1197: 132 pgs: 132 active+clean; 582 KiB data, 6.3 GiB used, 709 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-03-11T21:50:38.714 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:50:38 smithi092 ceph-mon[37563]: 4.0 scrub starts 2024-03-11T21:50:38.714 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:50:38 smithi092 ceph-mon[37563]: 4.0 scrub ok 2024-03-11T21:50:38.714 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:50:38 smithi092 ceph-mon[37563]: from='mgr.14607 v1:172.21.15.92:0/1702717597' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T21:50:39.086 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:50:38 smithi097 ceph-mon[33317]: pgmap v1197: 132 pgs: 132 active+clean; 582 KiB data, 6.3 GiB used, 709 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-03-11T21:50:39.086 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:50:38 smithi097 ceph-mon[33317]: 4.0 scrub starts 2024-03-11T21:50:39.086 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:50:38 smithi097 ceph-mon[33317]: 4.0 scrub ok 2024-03-11T21:50:39.086 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:50:38 smithi097 ceph-mon[33317]: from='mgr.14607 v1:172.21.15.92:0/1702717597' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T21:50:39.844 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:50:39 smithi092 ceph-mon[29572]: 4.16 scrub starts 2024-03-11T21:50:39.844 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:50:39 smithi092 ceph-mon[29572]: 4.16 scrub ok 2024-03-11T21:50:39.844 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:50:39 smithi092 ceph-mon[29572]: 6.1a scrub starts 2024-03-11T21:50:39.844 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:50:39 smithi092 ceph-mon[29572]: 6.1a scrub ok 2024-03-11T21:50:39.844 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:50:39 smithi092 ceph-mon[37563]: 4.16 scrub starts 2024-03-11T21:50:39.844 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:50:39 smithi092 ceph-mon[37563]: 4.16 scrub ok 2024-03-11T21:50:39.844 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:50:39 smithi092 ceph-mon[37563]: 6.1a scrub starts 2024-03-11T21:50:39.844 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:50:39 smithi092 ceph-mon[37563]: 6.1a scrub ok 2024-03-11T21:50:40.086 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:50:39 smithi097 ceph-mon[33317]: 4.16 scrub starts 2024-03-11T21:50:40.086 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:50:39 smithi097 ceph-mon[33317]: 4.16 scrub ok 2024-03-11T21:50:40.086 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:50:39 smithi097 ceph-mon[33317]: 6.1a scrub starts 2024-03-11T21:50:40.086 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:50:39 smithi097 ceph-mon[33317]: 6.1a scrub ok 2024-03-11T21:50:40.870 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:50:40 smithi092 ceph-mon[29572]: pgmap v1198: 132 pgs: 132 active+clean; 582 KiB data, 6.3 GiB used, 709 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-11T21:50:40.870 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:50:40 smithi092 ceph-mon[37563]: pgmap v1198: 132 pgs: 132 active+clean; 582 KiB data, 6.3 GiB used, 709 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-11T21:50:41.086 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:50:40 smithi097 ceph-mon[33317]: pgmap v1198: 132 pgs: 132 active+clean; 582 KiB data, 6.3 GiB used, 709 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-11T21:50:41.424 INFO:teuthology.orchestra.run.smithi092.stderr:Inferring config /var/lib/ceph/00db0016-dfed-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T21:50:41.757 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:50:41 smithi092 ceph-mon[29572]: 2.a scrub starts 2024-03-11T21:50:41.758 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:50:41 smithi092 ceph-mon[29572]: 2.a scrub ok 2024-03-11T21:50:41.758 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:50:41 smithi092 ceph-mon[29572]: 4.10 scrub starts 2024-03-11T21:50:41.758 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:50:41 smithi092 ceph-mon[29572]: 4.10 scrub ok 2024-03-11T21:50:41.758 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:50:41 smithi092 ceph-mon[37563]: 2.a scrub starts 2024-03-11T21:50:41.758 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:50:41 smithi092 ceph-mon[37563]: 2.a scrub ok 2024-03-11T21:50:41.758 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:50:41 smithi092 ceph-mon[37563]: 4.10 scrub starts 2024-03-11T21:50:41.758 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:50:41 smithi092 ceph-mon[37563]: 4.10 scrub ok 2024-03-11T21:50:42.086 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:50:41 smithi097 ceph-mon[33317]: 2.a scrub starts 2024-03-11T21:50:42.086 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:50:41 smithi097 ceph-mon[33317]: 2.a scrub ok 2024-03-11T21:50:42.086 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:50:41 smithi097 ceph-mon[33317]: 4.10 scrub starts 2024-03-11T21:50:42.086 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:50:41 smithi097 ceph-mon[33317]: 4.10 scrub ok 2024-03-11T21:50:42.811 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:50:42 smithi092 ceph-mon[29572]: pgmap v1199: 132 pgs: 132 active+clean; 582 KiB data, 6.3 GiB used, 709 GiB / 715 GiB avail; 1018 B/s rd, 0 op/s 2024-03-11T21:50:42.811 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:50:42 smithi092 ceph-mon[29572]: 4.18 scrub starts 2024-03-11T21:50:42.811 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:50:42 smithi092 ceph-mon[29572]: 4.18 scrub ok 2024-03-11T21:50:42.811 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:50:42 smithi092 ceph-mon[37563]: pgmap v1199: 132 pgs: 132 active+clean; 582 KiB data, 6.3 GiB used, 709 GiB / 715 GiB avail; 1018 B/s rd, 0 op/s 2024-03-11T21:50:42.811 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:50:42 smithi092 ceph-mon[37563]: 4.18 scrub starts 2024-03-11T21:50:42.812 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:50:42 smithi092 ceph-mon[37563]: 4.18 scrub ok 2024-03-11T21:50:43.086 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:50:42 smithi097 ceph-mon[33317]: pgmap v1199: 132 pgs: 132 active+clean; 582 KiB data, 6.3 GiB used, 709 GiB / 715 GiB avail; 1018 B/s rd, 0 op/s 2024-03-11T21:50:43.086 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:50:42 smithi097 ceph-mon[33317]: 4.18 scrub starts 2024-03-11T21:50:43.086 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:50:42 smithi097 ceph-mon[33317]: 4.18 scrub ok 2024-03-11T21:50:43.783 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:50:43 smithi092 ceph-mon[29572]: 6.4 scrub starts 2024-03-11T21:50:43.784 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:50:43 smithi092 ceph-mon[29572]: 6.4 scrub ok 2024-03-11T21:50:43.784 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:50:43 smithi092 ceph-mon[29572]: 2.17 deep-scrub starts 2024-03-11T21:50:43.784 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:50:43 smithi092 ceph-mon[29572]: 2.17 deep-scrub ok 2024-03-11T21:50:43.784 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:50:43 smithi092 ceph-mon[37563]: 6.4 scrub starts 2024-03-11T21:50:43.784 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:50:43 smithi092 ceph-mon[37563]: 6.4 scrub ok 2024-03-11T21:50:43.784 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:50:43 smithi092 ceph-mon[37563]: 2.17 deep-scrub starts 2024-03-11T21:50:43.784 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:50:43 smithi092 ceph-mon[37563]: 2.17 deep-scrub ok 2024-03-11T21:50:43.836 INFO:journalctl@ceph.iscsi.iscsi.a.smithi097.stdout:Mar 11 21:50:43 smithi097 ceph-00db0016-dfed-11ee-95c8-87774f69a715-iscsi-iscsi-a[60943]: debug there is no tcmu-runner data available 2024-03-11T21:50:43.836 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:50:43 smithi097 ceph-mon[33317]: 6.4 scrub starts 2024-03-11T21:50:43.836 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:50:43 smithi097 ceph-mon[33317]: 6.4 scrub ok 2024-03-11T21:50:43.836 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:50:43 smithi097 ceph-mon[33317]: 2.17 deep-scrub starts 2024-03-11T21:50:43.836 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:50:43 smithi097 ceph-mon[33317]: 2.17 deep-scrub ok 2024-03-11T21:50:44.726 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:50:44 smithi092 ceph-mon[29572]: 4.5 scrub starts 2024-03-11T21:50:44.726 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:50:44 smithi092 ceph-mon[29572]: 4.5 scrub ok 2024-03-11T21:50:44.726 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:50:44 smithi092 ceph-mon[29572]: from='client.24469 v1:172.21.15.97:0/404208310' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:50:44.726 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:50:44 smithi092 ceph-mon[29572]: pgmap v1200: 132 pgs: 132 active+clean; 582 KiB data, 6.3 GiB used, 709 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-11T21:50:44.726 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:50:44 smithi092 ceph-mon[29572]: 2.19 scrub starts 2024-03-11T21:50:44.726 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:50:44 smithi092 ceph-mon[29572]: 2.19 scrub ok 2024-03-11T21:50:44.727 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:50:44 smithi092 ceph-mon[37563]: 4.5 scrub starts 2024-03-11T21:50:44.727 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:50:44 smithi092 ceph-mon[37563]: 4.5 scrub ok 2024-03-11T21:50:44.727 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:50:44 smithi092 ceph-mon[37563]: from='client.24469 v1:172.21.15.97:0/404208310' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:50:44.727 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:50:44 smithi092 ceph-mon[37563]: pgmap v1200: 132 pgs: 132 active+clean; 582 KiB data, 6.3 GiB used, 709 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-11T21:50:44.727 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:50:44 smithi092 ceph-mon[37563]: 2.19 scrub starts 2024-03-11T21:50:44.727 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:50:44 smithi092 ceph-mon[37563]: 2.19 scrub ok 2024-03-11T21:50:45.086 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:50:44 smithi097 ceph-mon[33317]: 4.5 scrub starts 2024-03-11T21:50:45.086 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:50:44 smithi097 ceph-mon[33317]: 4.5 scrub ok 2024-03-11T21:50:45.086 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:50:44 smithi097 ceph-mon[33317]: from='client.24469 v1:172.21.15.97:0/404208310' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:50:45.086 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:50:44 smithi097 ceph-mon[33317]: pgmap v1200: 132 pgs: 132 active+clean; 582 KiB data, 6.3 GiB used, 709 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-11T21:50:45.086 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:50:44 smithi097 ceph-mon[33317]: 2.19 scrub starts 2024-03-11T21:50:45.086 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:50:44 smithi097 ceph-mon[33317]: 2.19 scrub ok 2024-03-11T21:50:45.851 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:50:45 smithi092 ceph-mon[29572]: 6.18 scrub starts 2024-03-11T21:50:45.851 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:50:45 smithi092 ceph-mon[29572]: 6.18 scrub ok 2024-03-11T21:50:45.852 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:50:45 smithi092 ceph-mon[29572]: 6.19 scrub starts 2024-03-11T21:50:45.852 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:50:45 smithi092 ceph-mon[29572]: 6.19 scrub ok 2024-03-11T21:50:45.852 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:50:45 smithi092 ceph-mon[29572]: 5.1 deep-scrub starts 2024-03-11T21:50:45.852 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:50:45 smithi092 ceph-mon[29572]: 5.1 deep-scrub ok 2024-03-11T21:50:45.852 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:50:45 smithi092 ceph-mon[37563]: 6.18 scrub starts 2024-03-11T21:50:45.852 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:50:45 smithi092 ceph-mon[37563]: 6.18 scrub ok 2024-03-11T21:50:45.852 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:50:45 smithi092 ceph-mon[37563]: 6.19 scrub starts 2024-03-11T21:50:45.852 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:50:45 smithi092 ceph-mon[37563]: 6.19 scrub ok 2024-03-11T21:50:45.852 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:50:45 smithi092 ceph-mon[37563]: 5.1 deep-scrub starts 2024-03-11T21:50:45.852 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:50:45 smithi092 ceph-mon[37563]: 5.1 deep-scrub ok 2024-03-11T21:50:46.086 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:50:45 smithi097 ceph-mon[33317]: 6.18 scrub starts 2024-03-11T21:50:46.086 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:50:45 smithi097 ceph-mon[33317]: 6.18 scrub ok 2024-03-11T21:50:46.086 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:50:45 smithi097 ceph-mon[33317]: 6.19 scrub starts 2024-03-11T21:50:46.086 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:50:45 smithi097 ceph-mon[33317]: 6.19 scrub ok 2024-03-11T21:50:46.086 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:50:45 smithi097 ceph-mon[33317]: 5.1 deep-scrub starts 2024-03-11T21:50:46.086 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:50:45 smithi097 ceph-mon[33317]: 5.1 deep-scrub ok 2024-03-11T21:50:46.583 INFO:teuthology.orchestra.run.smithi092.stdout:77309411658 2024-03-11T21:50:46.584 DEBUG:teuthology.orchestra.run.smithi092:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid 00db0016-dfed-11ee-95c8-87774f69a715 -- ceph osd last-stat-seq osd.2 2024-03-11T21:50:46.709 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:50:46 smithi092 ceph-mon[29572]: 2.0 scrub starts 2024-03-11T21:50:46.709 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:50:46 smithi092 ceph-mon[29572]: 2.0 scrub ok 2024-03-11T21:50:46.709 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:50:46 smithi092 ceph-mon[29572]: pgmap v1201: 132 pgs: 132 active+clean; 582 KiB data, 6.3 GiB used, 709 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T21:50:46.709 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:50:46 smithi092 ceph-mon[37563]: 2.0 scrub starts 2024-03-11T21:50:46.709 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:50:46 smithi092 ceph-mon[37563]: 2.0 scrub ok 2024-03-11T21:50:46.709 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:50:46 smithi092 ceph-mon[37563]: pgmap v1201: 132 pgs: 132 active+clean; 582 KiB data, 6.3 GiB used, 709 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T21:50:47.086 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:50:46 smithi097 ceph-mon[33317]: 2.0 scrub starts 2024-03-11T21:50:47.086 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:50:46 smithi097 ceph-mon[33317]: 2.0 scrub ok 2024-03-11T21:50:47.086 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:50:46 smithi097 ceph-mon[33317]: pgmap v1201: 132 pgs: 132 active+clean; 582 KiB data, 6.3 GiB used, 709 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T21:50:47.783 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:50:47 smithi092 ceph-mon[29572]: 6.10 scrub starts 2024-03-11T21:50:47.784 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:50:47 smithi092 ceph-mon[29572]: 6.10 scrub ok 2024-03-11T21:50:47.784 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:50:47 smithi092 ceph-mon[37563]: 6.10 scrub starts 2024-03-11T21:50:47.784 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:50:47 smithi092 ceph-mon[37563]: 6.10 scrub ok 2024-03-11T21:50:47.812 INFO:teuthology.orchestra.run.smithi092.stdout:257698038065 2024-03-11T21:50:47.813 DEBUG:teuthology.orchestra.run.smithi092:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid 00db0016-dfed-11ee-95c8-87774f69a715 -- ceph osd last-stat-seq osd.7 2024-03-11T21:50:48.086 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:50:47 smithi097 ceph-mon[33317]: 6.10 scrub starts 2024-03-11T21:50:48.086 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:50:47 smithi097 ceph-mon[33317]: 6.10 scrub ok 2024-03-11T21:50:48.102 INFO:journalctl@ceph.mgr.y.smithi092.stdout:Mar 11 21:50:47 smithi092 ceph-00db0016-dfed-11ee-95c8-87774f69a715-mgr-y[29801]: ::ffff:172.21.15.97 - - [11/Mar/2024:21:50:47] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-11T21:50:48.941 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:50:48 smithi092 ceph-mon[29572]: 2.9 scrub starts 2024-03-11T21:50:48.942 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:50:48 smithi092 ceph-mon[29572]: 2.9 scrub ok 2024-03-11T21:50:48.942 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:50:48 smithi092 ceph-mon[29572]: pgmap v1202: 132 pgs: 132 active+clean; 582 KiB data, 6.3 GiB used, 709 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-11T21:50:48.942 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:50:48 smithi092 ceph-mon[37563]: 2.9 scrub starts 2024-03-11T21:50:48.942 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:50:48 smithi092 ceph-mon[37563]: 2.9 scrub ok 2024-03-11T21:50:48.942 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:50:48 smithi092 ceph-mon[37563]: pgmap v1202: 132 pgs: 132 active+clean; 582 KiB data, 6.3 GiB used, 709 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-11T21:50:49.086 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:50:48 smithi097 ceph-mon[33317]: 2.9 scrub starts 2024-03-11T21:50:49.086 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:50:48 smithi097 ceph-mon[33317]: 2.9 scrub ok 2024-03-11T21:50:49.086 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:50:48 smithi097 ceph-mon[33317]: pgmap v1202: 132 pgs: 132 active+clean; 582 KiB data, 6.3 GiB used, 709 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-11T21:50:49.367 INFO:teuthology.orchestra.run.smithi092.stdout:214748365111 2024-03-11T21:50:49.367 DEBUG:teuthology.orchestra.run.smithi092:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid 00db0016-dfed-11ee-95c8-87774f69a715 -- ceph osd last-stat-seq osd.6 2024-03-11T21:50:49.859 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:50:49 smithi092 ceph-mon[29572]: 6.3 scrub starts 2024-03-11T21:50:49.859 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:50:49 smithi092 ceph-mon[29572]: 6.3 scrub ok 2024-03-11T21:50:49.859 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:50:49 smithi092 ceph-mon[37563]: 6.3 scrub starts 2024-03-11T21:50:49.860 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:50:49 smithi092 ceph-mon[37563]: 6.3 scrub ok 2024-03-11T21:50:50.086 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:50:49 smithi097 ceph-mon[33317]: 6.3 scrub starts 2024-03-11T21:50:50.086 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:50:49 smithi097 ceph-mon[33317]: 6.3 scrub ok 2024-03-11T21:50:50.683 INFO:teuthology.orchestra.run.smithi092.stdout:176093659451 2024-03-11T21:50:50.684 DEBUG:teuthology.orchestra.run.smithi092:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid 00db0016-dfed-11ee-95c8-87774f69a715 -- ceph osd last-stat-seq osd.5 2024-03-11T21:50:50.796 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:50:50 smithi092 ceph-mon[29572]: 6.9 deep-scrub starts 2024-03-11T21:50:50.796 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:50:50 smithi092 ceph-mon[29572]: 5.10 scrub starts 2024-03-11T21:50:50.796 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:50:50 smithi092 ceph-mon[29572]: 5.10 scrub ok 2024-03-11T21:50:50.796 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:50:50 smithi092 ceph-mon[29572]: 6.9 deep-scrub ok 2024-03-11T21:50:50.797 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:50:50 smithi092 ceph-mon[29572]: pgmap v1203: 132 pgs: 132 active+clean; 582 KiB data, 6.3 GiB used, 709 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T21:50:50.797 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:50:50 smithi092 ceph-mon[37563]: 6.9 deep-scrub starts 2024-03-11T21:50:50.797 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:50:50 smithi092 ceph-mon[37563]: 5.10 scrub starts 2024-03-11T21:50:50.797 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:50:50 smithi092 ceph-mon[37563]: 5.10 scrub ok 2024-03-11T21:50:50.797 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:50:50 smithi092 ceph-mon[37563]: 6.9 deep-scrub ok 2024-03-11T21:50:50.797 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:50:50 smithi092 ceph-mon[37563]: pgmap v1203: 132 pgs: 132 active+clean; 582 KiB data, 6.3 GiB used, 709 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T21:50:51.086 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:50:50 smithi097 ceph-mon[33317]: 6.9 deep-scrub starts 2024-03-11T21:50:51.086 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:50:50 smithi097 ceph-mon[33317]: 5.10 scrub starts 2024-03-11T21:50:51.086 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:50:50 smithi097 ceph-mon[33317]: 5.10 scrub ok 2024-03-11T21:50:51.086 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:50:50 smithi097 ceph-mon[33317]: 6.9 deep-scrub ok 2024-03-11T21:50:51.086 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:50:50 smithi097 ceph-mon[33317]: pgmap v1203: 132 pgs: 132 active+clean; 582 KiB data, 6.3 GiB used, 709 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T21:50:51.492 INFO:teuthology.orchestra.run.smithi092.stdout:34359738710 2024-03-11T21:50:51.493 DEBUG:teuthology.orchestra.run.smithi092:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid 00db0016-dfed-11ee-95c8-87774f69a715 -- ceph osd last-stat-seq osd.0 2024-03-11T21:50:51.870 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:50:51 smithi092 ceph-mon[29572]: 6.16 deep-scrub starts 2024-03-11T21:50:51.870 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:50:51 smithi092 ceph-mon[29572]: 6.16 deep-scrub ok 2024-03-11T21:50:51.870 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:50:51 smithi092 ceph-mon[29572]: 2.14 scrub starts 2024-03-11T21:50:51.870 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:50:51 smithi092 ceph-mon[29572]: 2.14 scrub ok 2024-03-11T21:50:51.870 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:50:51 smithi092 ceph-mon[37563]: 6.16 deep-scrub starts 2024-03-11T21:50:51.871 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:50:51 smithi092 ceph-mon[37563]: 6.16 deep-scrub ok 2024-03-11T21:50:51.871 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:50:51 smithi092 ceph-mon[37563]: 2.14 scrub starts 2024-03-11T21:50:51.871 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:50:51 smithi092 ceph-mon[37563]: 2.14 scrub ok 2024-03-11T21:50:52.086 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:50:51 smithi097 ceph-mon[33317]: 6.16 deep-scrub starts 2024-03-11T21:50:52.086 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:50:51 smithi097 ceph-mon[33317]: 6.16 deep-scrub ok 2024-03-11T21:50:52.086 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:50:51 smithi097 ceph-mon[33317]: 2.14 scrub starts 2024-03-11T21:50:52.086 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:50:51 smithi097 ceph-mon[33317]: 2.14 scrub ok 2024-03-11T21:50:52.355 INFO:teuthology.orchestra.run.smithi092.stdout:137438953793 2024-03-11T21:50:52.356 DEBUG:teuthology.orchestra.run.smithi092:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid 00db0016-dfed-11ee-95c8-87774f69a715 -- ceph osd last-stat-seq osd.4 2024-03-11T21:50:52.844 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:50:52 smithi092 ceph-mon[29572]: pgmap v1204: 132 pgs: 132 active+clean; 582 KiB data, 6.3 GiB used, 709 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-11T21:50:52.845 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:50:52 smithi092 ceph-mon[37563]: pgmap v1204: 132 pgs: 132 active+clean; 582 KiB data, 6.3 GiB used, 709 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-11T21:50:53.086 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:50:52 smithi097 ceph-mon[33317]: pgmap v1204: 132 pgs: 132 active+clean; 582 KiB data, 6.3 GiB used, 709 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-11T21:50:53.260 INFO:teuthology.orchestra.run.smithi092.stdout:1189705941165 2024-03-11T21:50:53.260 DEBUG:teuthology.orchestra.run.smithi092:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid 00db0016-dfed-11ee-95c8-87774f69a715 -- ceph osd last-stat-seq osd.3 2024-03-11T21:50:53.498 INFO:teuthology.orchestra.run.smithi092.stderr:Inferring config /var/lib/ceph/00db0016-dfed-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T21:50:53.498 INFO:teuthology.orchestra.run.smithi092.stderr:Inferring config /var/lib/ceph/00db0016-dfed-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T21:50:53.499 INFO:teuthology.orchestra.run.smithi092.stderr:Inferring config /var/lib/ceph/00db0016-dfed-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T21:50:53.836 INFO:journalctl@ceph.iscsi.iscsi.a.smithi097.stdout:Mar 11 21:50:53 smithi097 ceph-00db0016-dfed-11ee-95c8-87774f69a715-iscsi-iscsi-a[60943]: debug there is no tcmu-runner data available 2024-03-11T21:50:53.836 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:50:53 smithi097 ceph-mon[33317]: 4.11 scrub starts 2024-03-11T21:50:53.836 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:50:53 smithi097 ceph-mon[33317]: 4.11 scrub ok 2024-03-11T21:50:53.837 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:50:53 smithi097 ceph-mon[33317]: 4.1f scrub starts 2024-03-11T21:50:53.837 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:50:53 smithi097 ceph-mon[33317]: 4.1f scrub ok 2024-03-11T21:50:53.837 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:50:53 smithi097 ceph-mon[33317]: 4.13 scrub starts 2024-03-11T21:50:53.837 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:50:53 smithi097 ceph-mon[33317]: 4.13 scrub ok 2024-03-11T21:50:53.837 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:50:53 smithi097 ceph-mon[33317]: from='mgr.14607 v1:172.21.15.92:0/1702717597' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T21:50:53.859 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:50:53 smithi092 ceph-mon[29572]: 4.11 scrub starts 2024-03-11T21:50:53.859 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:50:53 smithi092 ceph-mon[29572]: 4.11 scrub ok 2024-03-11T21:50:53.860 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:50:53 smithi092 ceph-mon[29572]: 4.1f scrub starts 2024-03-11T21:50:53.860 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:50:53 smithi092 ceph-mon[29572]: 4.1f scrub ok 2024-03-11T21:50:53.860 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:50:53 smithi092 ceph-mon[29572]: 4.13 scrub starts 2024-03-11T21:50:53.860 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:50:53 smithi092 ceph-mon[29572]: 4.13 scrub ok 2024-03-11T21:50:53.860 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:50:53 smithi092 ceph-mon[29572]: from='mgr.14607 v1:172.21.15.92:0/1702717597' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T21:50:53.860 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:50:53 smithi092 ceph-mon[37563]: 4.11 scrub starts 2024-03-11T21:50:53.860 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:50:53 smithi092 ceph-mon[37563]: 4.11 scrub ok 2024-03-11T21:50:53.860 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:50:53 smithi092 ceph-mon[37563]: 4.1f scrub starts 2024-03-11T21:50:53.860 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:50:53 smithi092 ceph-mon[37563]: 4.1f scrub ok 2024-03-11T21:50:53.860 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:50:53 smithi092 ceph-mon[37563]: 4.13 scrub starts 2024-03-11T21:50:53.860 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:50:53 smithi092 ceph-mon[37563]: 4.13 scrub ok 2024-03-11T21:50:53.861 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:50:53 smithi092 ceph-mon[37563]: from='mgr.14607 v1:172.21.15.92:0/1702717597' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T21:50:53.899 INFO:teuthology.orchestra.run.smithi092.stdout:55834575186 2024-03-11T21:50:53.900 DEBUG:teuthology.orchestra.run.smithi092:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid 00db0016-dfed-11ee-95c8-87774f69a715 -- ceph osd last-stat-seq osd.1 2024-03-11T21:50:54.870 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:50:54 smithi092 ceph-mon[29572]: from='client.24469 v1:172.21.15.97:0/404208310' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:50:54.870 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:50:54 smithi092 ceph-mon[29572]: pgmap v1205: 132 pgs: 132 active+clean; 582 KiB data, 6.3 GiB used, 709 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T21:50:54.870 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:50:54 smithi092 ceph-mon[29572]: 2.11 scrub starts 2024-03-11T21:50:54.870 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:50:54 smithi092 ceph-mon[29572]: 2.11 scrub ok 2024-03-11T21:50:54.870 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:50:54 smithi092 ceph-mon[37563]: from='client.24469 v1:172.21.15.97:0/404208310' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:50:54.871 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:50:54 smithi092 ceph-mon[37563]: pgmap v1205: 132 pgs: 132 active+clean; 582 KiB data, 6.3 GiB used, 709 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T21:50:54.871 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:50:54 smithi092 ceph-mon[37563]: 2.11 scrub starts 2024-03-11T21:50:54.871 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:50:54 smithi092 ceph-mon[37563]: 2.11 scrub ok 2024-03-11T21:50:55.086 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:50:54 smithi097 ceph-mon[33317]: from='client.24469 v1:172.21.15.97:0/404208310' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:50:55.086 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:50:54 smithi097 ceph-mon[33317]: pgmap v1205: 132 pgs: 132 active+clean; 582 KiB data, 6.3 GiB used, 709 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T21:50:55.086 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:50:54 smithi097 ceph-mon[33317]: 2.11 scrub starts 2024-03-11T21:50:55.086 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:50:54 smithi097 ceph-mon[33317]: 2.11 scrub ok 2024-03-11T21:50:55.300 INFO:teuthology.orchestra.run.smithi092.stderr:Inferring config /var/lib/ceph/00db0016-dfed-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T21:50:55.301 INFO:teuthology.orchestra.run.smithi092.stderr:Inferring config /var/lib/ceph/00db0016-dfed-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T21:50:55.302 INFO:teuthology.orchestra.run.smithi092.stderr:Inferring config /var/lib/ceph/00db0016-dfed-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T21:50:55.302 INFO:teuthology.orchestra.run.smithi092.stderr:Inferring config /var/lib/ceph/00db0016-dfed-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T21:50:56.086 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:50:55 smithi097 ceph-mon[33317]: 6.0 scrub starts 2024-03-11T21:50:56.086 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:50:55 smithi097 ceph-mon[33317]: 6.0 scrub ok 2024-03-11T21:50:56.086 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:50:55 smithi097 ceph-mon[33317]: 5.a scrub starts 2024-03-11T21:50:56.086 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:50:55 smithi097 ceph-mon[33317]: 5.a scrub ok 2024-03-11T21:50:56.120 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:50:55 smithi092 ceph-mon[29572]: 6.0 scrub starts 2024-03-11T21:50:56.120 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:50:55 smithi092 ceph-mon[29572]: 6.0 scrub ok 2024-03-11T21:50:56.120 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:50:55 smithi092 ceph-mon[29572]: 5.a scrub starts 2024-03-11T21:50:56.120 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:50:55 smithi092 ceph-mon[29572]: 5.a scrub ok 2024-03-11T21:50:56.120 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:50:55 smithi092 ceph-mon[37563]: 6.0 scrub starts 2024-03-11T21:50:56.120 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:50:55 smithi092 ceph-mon[37563]: 6.0 scrub ok 2024-03-11T21:50:56.121 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:50:55 smithi092 ceph-mon[37563]: 5.a scrub starts 2024-03-11T21:50:56.121 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:50:55 smithi092 ceph-mon[37563]: 5.a scrub ok 2024-03-11T21:50:56.870 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:50:56 smithi092 ceph-mon[29572]: 2.13 scrub starts 2024-03-11T21:50:56.870 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:50:56 smithi092 ceph-mon[29572]: 2.13 scrub ok 2024-03-11T21:50:56.870 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:50:56 smithi092 ceph-mon[29572]: pgmap v1206: 132 pgs: 132 active+clean; 582 KiB data, 6.3 GiB used, 709 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T21:50:56.870 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:50:56 smithi092 ceph-mon[29572]: 6.7 scrub starts 2024-03-11T21:50:56.870 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:50:56 smithi092 ceph-mon[29572]: 6.7 scrub ok 2024-03-11T21:50:56.870 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:50:56 smithi092 ceph-mon[37563]: 2.13 scrub starts 2024-03-11T21:50:56.871 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:50:56 smithi092 ceph-mon[37563]: 2.13 scrub ok 2024-03-11T21:50:56.871 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:50:56 smithi092 ceph-mon[37563]: pgmap v1206: 132 pgs: 132 active+clean; 582 KiB data, 6.3 GiB used, 709 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T21:50:56.871 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:50:56 smithi092 ceph-mon[37563]: 6.7 scrub starts 2024-03-11T21:50:56.871 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:50:56 smithi092 ceph-mon[37563]: 6.7 scrub ok 2024-03-11T21:50:57.086 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:50:56 smithi097 ceph-mon[33317]: 2.13 scrub starts 2024-03-11T21:50:57.086 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:50:56 smithi097 ceph-mon[33317]: 2.13 scrub ok 2024-03-11T21:50:57.086 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:50:56 smithi097 ceph-mon[33317]: pgmap v1206: 132 pgs: 132 active+clean; 582 KiB data, 6.3 GiB used, 709 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T21:50:57.086 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:50:56 smithi097 ceph-mon[33317]: 6.7 scrub starts 2024-03-11T21:50:57.086 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:50:56 smithi097 ceph-mon[33317]: 6.7 scrub ok 2024-03-11T21:50:57.933 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:50:57 smithi092 ceph-mon[29572]: 4.3 scrub starts 2024-03-11T21:50:57.933 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:50:57 smithi092 ceph-mon[29572]: 4.3 scrub ok 2024-03-11T21:50:57.933 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:50:57 smithi092 ceph-mon[29572]: 4.14 scrub starts 2024-03-11T21:50:57.933 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:50:57 smithi092 ceph-mon[29572]: 4.14 scrub ok 2024-03-11T21:50:57.934 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:50:57 smithi092 ceph-mon[37563]: 4.3 scrub starts 2024-03-11T21:50:57.934 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:50:57 smithi092 ceph-mon[37563]: 4.3 scrub ok 2024-03-11T21:50:57.934 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:50:57 smithi092 ceph-mon[37563]: 4.14 scrub starts 2024-03-11T21:50:57.934 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:50:57 smithi092 ceph-mon[37563]: 4.14 scrub ok 2024-03-11T21:50:58.086 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:50:57 smithi097 ceph-mon[33317]: 4.3 scrub starts 2024-03-11T21:50:58.086 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:50:57 smithi097 ceph-mon[33317]: 4.3 scrub ok 2024-03-11T21:50:58.086 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:50:57 smithi097 ceph-mon[33317]: 4.14 scrub starts 2024-03-11T21:50:58.086 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:50:57 smithi097 ceph-mon[33317]: 4.14 scrub ok 2024-03-11T21:50:58.248 INFO:journalctl@ceph.mgr.y.smithi092.stdout:Mar 11 21:50:57 smithi092 ceph-00db0016-dfed-11ee-95c8-87774f69a715-mgr-y[29801]: ::ffff:172.21.15.97 - - [11/Mar/2024:21:50:57] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-11T21:50:58.837 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:50:58 smithi092 ceph-mon[29572]: 5.19 scrub starts 2024-03-11T21:50:58.837 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:50:58 smithi092 ceph-mon[29572]: 5.19 scrub ok 2024-03-11T21:50:58.837 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:50:58 smithi092 ceph-mon[29572]: 2.1d scrub starts 2024-03-11T21:50:58.837 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:50:58 smithi092 ceph-mon[29572]: 2.1d scrub ok 2024-03-11T21:50:58.837 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:50:58 smithi092 ceph-mon[29572]: pgmap v1207: 132 pgs: 132 active+clean; 582 KiB data, 6.3 GiB used, 709 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-11T21:50:58.837 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:50:58 smithi092 ceph-mon[29572]: from='client.? v1:172.21.15.92:0/1178681305' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 6}]: dispatch 2024-03-11T21:50:58.838 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:50:58 smithi092 ceph-mon[37563]: 5.19 scrub starts 2024-03-11T21:50:58.838 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:50:58 smithi092 ceph-mon[37563]: 5.19 scrub ok 2024-03-11T21:50:58.838 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:50:58 smithi092 ceph-mon[37563]: 2.1d scrub starts 2024-03-11T21:50:58.838 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:50:58 smithi092 ceph-mon[37563]: 2.1d scrub ok 2024-03-11T21:50:58.838 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:50:58 smithi092 ceph-mon[37563]: pgmap v1207: 132 pgs: 132 active+clean; 582 KiB data, 6.3 GiB used, 709 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-11T21:50:58.838 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:50:58 smithi092 ceph-mon[37563]: from='client.? v1:172.21.15.92:0/1178681305' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 6}]: dispatch 2024-03-11T21:50:59.086 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:50:58 smithi097 ceph-mon[33317]: 5.19 scrub starts 2024-03-11T21:50:59.086 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:50:58 smithi097 ceph-mon[33317]: 5.19 scrub ok 2024-03-11T21:50:59.086 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:50:58 smithi097 ceph-mon[33317]: 2.1d scrub starts 2024-03-11T21:50:59.086 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:50:58 smithi097 ceph-mon[33317]: 2.1d scrub ok 2024-03-11T21:50:59.086 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:50:58 smithi097 ceph-mon[33317]: pgmap v1207: 132 pgs: 132 active+clean; 582 KiB data, 6.3 GiB used, 709 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-11T21:50:59.087 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:50:58 smithi097 ceph-mon[33317]: from='client.? v1:172.21.15.92:0/1178681305' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 6}]: dispatch 2024-03-11T21:50:59.266 INFO:teuthology.orchestra.run.smithi092.stdout:214748365113 2024-03-11T21:50:59.954 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:50:59 smithi092 ceph-mon[29572]: 5.1e scrub starts 2024-03-11T21:50:59.955 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:50:59 smithi092 ceph-mon[29572]: 5.1e scrub ok 2024-03-11T21:50:59.955 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:50:59 smithi092 ceph-mon[29572]: 2.3 scrub starts 2024-03-11T21:50:59.955 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:50:59 smithi092 ceph-mon[29572]: 2.3 scrub ok 2024-03-11T21:50:59.955 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:50:59 smithi092 ceph-mon[29572]: 2.1e scrub starts 2024-03-11T21:50:59.955 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:50:59 smithi092 ceph-mon[29572]: 2.1e scrub ok 2024-03-11T21:50:59.955 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:50:59 smithi092 ceph-mon[29572]: 4.1a scrub starts 2024-03-11T21:50:59.955 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:50:59 smithi092 ceph-mon[29572]: 4.1a scrub ok 2024-03-11T21:50:59.955 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:50:59 smithi092 ceph-mon[37563]: 5.1e scrub starts 2024-03-11T21:50:59.955 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:50:59 smithi092 ceph-mon[37563]: 5.1e scrub ok 2024-03-11T21:50:59.956 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:50:59 smithi092 ceph-mon[37563]: 2.3 scrub starts 2024-03-11T21:50:59.956 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:50:59 smithi092 ceph-mon[37563]: 2.3 scrub ok 2024-03-11T21:50:59.956 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:50:59 smithi092 ceph-mon[37563]: 2.1e scrub starts 2024-03-11T21:50:59.956 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:50:59 smithi092 ceph-mon[37563]: 2.1e scrub ok 2024-03-11T21:50:59.956 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:50:59 smithi092 ceph-mon[37563]: 4.1a scrub starts 2024-03-11T21:50:59.956 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:50:59 smithi092 ceph-mon[37563]: 4.1a scrub ok 2024-03-11T21:50:59.989 INFO:tasks.cephadm.ceph_manager.ceph:need seq 214748365111 got 214748365113 for osd.6 2024-03-11T21:50:59.990 DEBUG:teuthology.parallel:result is None 2024-03-11T21:51:00.086 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:50:59 smithi097 ceph-mon[33317]: 5.1e scrub starts 2024-03-11T21:51:00.086 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:50:59 smithi097 ceph-mon[33317]: 5.1e scrub ok 2024-03-11T21:51:00.086 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:50:59 smithi097 ceph-mon[33317]: 2.3 scrub starts 2024-03-11T21:51:00.086 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:50:59 smithi097 ceph-mon[33317]: 2.3 scrub ok 2024-03-11T21:51:00.086 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:50:59 smithi097 ceph-mon[33317]: 2.1e scrub starts 2024-03-11T21:51:00.086 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:50:59 smithi097 ceph-mon[33317]: 2.1e scrub ok 2024-03-11T21:51:00.086 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:50:59 smithi097 ceph-mon[33317]: 4.1a scrub starts 2024-03-11T21:51:00.086 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:50:59 smithi097 ceph-mon[33317]: 4.1a scrub ok 2024-03-11T21:51:00.426 INFO:teuthology.orchestra.run.smithi092.stdout:77309411662 2024-03-11T21:51:00.844 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:51:00 smithi092 ceph-mon[37563]: 4.4 deep-scrub starts 2024-03-11T21:51:00.844 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:51:00 smithi092 ceph-mon[37563]: 4.4 deep-scrub ok 2024-03-11T21:51:00.844 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:51:00 smithi092 ceph-mon[37563]: 4.6 scrub starts 2024-03-11T21:51:00.844 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:51:00 smithi092 ceph-mon[37563]: 4.6 scrub ok 2024-03-11T21:51:00.844 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:51:00 smithi092 ceph-mon[37563]: pgmap v1208: 132 pgs: 132 active+clean; 582 KiB data, 6.3 GiB used, 709 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T21:51:00.844 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:51:00 smithi092 ceph-mon[37563]: 5.14 scrub starts 2024-03-11T21:51:00.844 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:51:00 smithi092 ceph-mon[37563]: 5.14 scrub ok 2024-03-11T21:51:00.844 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:51:00 smithi092 ceph-mon[37563]: from='client.? v1:172.21.15.92:0/1702570500' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 2}]: dispatch 2024-03-11T21:51:00.844 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:51:00 smithi092 ceph-mon[29572]: 4.4 deep-scrub starts 2024-03-11T21:51:00.845 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:51:00 smithi092 ceph-mon[29572]: 4.4 deep-scrub ok 2024-03-11T21:51:00.845 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:51:00 smithi092 ceph-mon[29572]: 4.6 scrub starts 2024-03-11T21:51:00.845 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:51:00 smithi092 ceph-mon[29572]: 4.6 scrub ok 2024-03-11T21:51:00.845 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:51:00 smithi092 ceph-mon[29572]: pgmap v1208: 132 pgs: 132 active+clean; 582 KiB data, 6.3 GiB used, 709 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T21:51:00.845 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:51:00 smithi092 ceph-mon[29572]: 5.14 scrub starts 2024-03-11T21:51:00.845 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:51:00 smithi092 ceph-mon[29572]: 5.14 scrub ok 2024-03-11T21:51:00.845 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:51:00 smithi092 ceph-mon[29572]: from='client.? v1:172.21.15.92:0/1702570500' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 2}]: dispatch 2024-03-11T21:51:01.086 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:51:00 smithi097 ceph-mon[33317]: 4.4 deep-scrub starts 2024-03-11T21:51:01.086 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:51:00 smithi097 ceph-mon[33317]: 4.4 deep-scrub ok 2024-03-11T21:51:01.086 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:51:00 smithi097 ceph-mon[33317]: 4.6 scrub starts 2024-03-11T21:51:01.086 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:51:00 smithi097 ceph-mon[33317]: 4.6 scrub ok 2024-03-11T21:51:01.087 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:51:00 smithi097 ceph-mon[33317]: pgmap v1208: 132 pgs: 132 active+clean; 582 KiB data, 6.3 GiB used, 709 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T21:51:01.087 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:51:00 smithi097 ceph-mon[33317]: 5.14 scrub starts 2024-03-11T21:51:01.087 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:51:00 smithi097 ceph-mon[33317]: 5.14 scrub ok 2024-03-11T21:51:01.087 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:51:00 smithi097 ceph-mon[33317]: from='client.? v1:172.21.15.92:0/1702570500' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 2}]: dispatch 2024-03-11T21:51:03.086 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:51:02 smithi097 ceph-mon[33317]: pgmap v1209: 132 pgs: 132 active+clean; 582 KiB data, 6.3 GiB used, 709 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-11T21:51:03.120 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:51:02 smithi092 ceph-mon[29572]: pgmap v1209: 132 pgs: 132 active+clean; 582 KiB data, 6.3 GiB used, 709 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-11T21:51:03.120 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:51:02 smithi092 ceph-mon[37563]: pgmap v1209: 132 pgs: 132 active+clean; 582 KiB data, 6.3 GiB used, 709 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-11T21:51:03.836 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:51:03 smithi097 ceph-mon[33317]: 5.1b deep-scrub starts 2024-03-11T21:51:03.836 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:51:03 smithi097 ceph-mon[33317]: 5.1b deep-scrub ok 2024-03-11T21:51:03.836 INFO:journalctl@ceph.iscsi.iscsi.a.smithi097.stdout:Mar 11 21:51:03 smithi097 ceph-00db0016-dfed-11ee-95c8-87774f69a715-iscsi-iscsi-a[60943]: debug there is no tcmu-runner data available 2024-03-11T21:51:03.842 INFO:teuthology.orchestra.run.smithi092.stdout:257698038069 2024-03-11T21:51:04.120 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:51:03 smithi092 ceph-mon[29572]: 5.1b deep-scrub starts 2024-03-11T21:51:04.120 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:51:03 smithi092 ceph-mon[29572]: 5.1b deep-scrub ok 2024-03-11T21:51:04.120 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:51:03 smithi092 ceph-mon[37563]: 5.1b deep-scrub starts 2024-03-11T21:51:04.120 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:51:03 smithi092 ceph-mon[37563]: 5.1b deep-scrub ok 2024-03-11T21:51:04.122 INFO:teuthology.orchestra.run.smithi092.stdout:137438953796 2024-03-11T21:51:04.676 INFO:tasks.cephadm.ceph_manager.ceph:need seq 257698038065 got 257698038069 for osd.7 2024-03-11T21:51:04.677 DEBUG:teuthology.parallel:result is None 2024-03-11T21:51:04.944 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:51:04 smithi092 ceph-mon[29572]: from='client.24469 v1:172.21.15.97:0/404208310' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:51:04.944 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:51:04 smithi092 ceph-mon[29572]: pgmap v1210: 132 pgs: 132 active+clean; 582 KiB data, 6.3 GiB used, 709 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-11T21:51:04.945 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:51:04 smithi092 ceph-mon[29572]: 6.d deep-scrub starts 2024-03-11T21:51:04.945 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:51:04 smithi092 ceph-mon[29572]: 6.d deep-scrub ok 2024-03-11T21:51:04.945 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:51:04 smithi092 ceph-mon[29572]: from='client.? v1:172.21.15.92:0/1330498458' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 7}]: dispatch 2024-03-11T21:51:04.945 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:51:04 smithi092 ceph-mon[29572]: 5.0 scrub starts 2024-03-11T21:51:04.945 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:51:04 smithi092 ceph-mon[29572]: 5.0 scrub ok 2024-03-11T21:51:04.945 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:51:04 smithi092 ceph-mon[29572]: 5.1f deep-scrub starts 2024-03-11T21:51:04.945 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:51:04 smithi092 ceph-mon[29572]: 5.1f deep-scrub ok 2024-03-11T21:51:04.945 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:51:04 smithi092 ceph-mon[29572]: from='client.? v1:172.21.15.92:0/3150342060' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 4}]: dispatch 2024-03-11T21:51:04.945 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:51:04 smithi092 ceph-mon[37563]: from='client.24469 v1:172.21.15.97:0/404208310' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:51:04.945 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:51:04 smithi092 ceph-mon[37563]: pgmap v1210: 132 pgs: 132 active+clean; 582 KiB data, 6.3 GiB used, 709 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-11T21:51:04.945 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:51:04 smithi092 ceph-mon[37563]: 6.d deep-scrub starts 2024-03-11T21:51:04.945 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:51:04 smithi092 ceph-mon[37563]: 6.d deep-scrub ok 2024-03-11T21:51:04.946 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:51:04 smithi092 ceph-mon[37563]: from='client.? v1:172.21.15.92:0/1330498458' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 7}]: dispatch 2024-03-11T21:51:04.946 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:51:04 smithi092 ceph-mon[37563]: 5.0 scrub starts 2024-03-11T21:51:04.946 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:51:04 smithi092 ceph-mon[37563]: 5.0 scrub ok 2024-03-11T21:51:04.946 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:51:04 smithi092 ceph-mon[37563]: 5.1f deep-scrub starts 2024-03-11T21:51:04.946 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:51:04 smithi092 ceph-mon[37563]: 5.1f deep-scrub ok 2024-03-11T21:51:04.946 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:51:04 smithi092 ceph-mon[37563]: from='client.? v1:172.21.15.92:0/3150342060' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 4}]: dispatch 2024-03-11T21:51:05.065 INFO:teuthology.orchestra.run.smithi092.stdout:34359738713 2024-03-11T21:51:05.086 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:51:04 smithi097 ceph-mon[33317]: from='client.24469 v1:172.21.15.97:0/404208310' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:51:05.086 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:51:04 smithi097 ceph-mon[33317]: pgmap v1210: 132 pgs: 132 active+clean; 582 KiB data, 6.3 GiB used, 709 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-03-11T21:51:05.086 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:51:04 smithi097 ceph-mon[33317]: 6.d deep-scrub starts 2024-03-11T21:51:05.086 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:51:04 smithi097 ceph-mon[33317]: 6.d deep-scrub ok 2024-03-11T21:51:05.086 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:51:04 smithi097 ceph-mon[33317]: from='client.? v1:172.21.15.92:0/1330498458' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 7}]: dispatch 2024-03-11T21:51:05.086 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:51:04 smithi097 ceph-mon[33317]: 5.0 scrub starts 2024-03-11T21:51:05.086 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:51:04 smithi097 ceph-mon[33317]: 5.0 scrub ok 2024-03-11T21:51:05.086 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:51:04 smithi097 ceph-mon[33317]: 5.1f deep-scrub starts 2024-03-11T21:51:05.087 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:51:04 smithi097 ceph-mon[33317]: 5.1f deep-scrub ok 2024-03-11T21:51:05.087 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:51:04 smithi097 ceph-mon[33317]: from='client.? v1:172.21.15.92:0/3150342060' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 4}]: dispatch 2024-03-11T21:51:05.322 INFO:teuthology.orchestra.run.smithi092.stderr:Inferring config /var/lib/ceph/00db0016-dfed-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T21:51:06.016 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:51:05 smithi092 ceph-mon[29572]: 2.15 scrub starts 2024-03-11T21:51:06.016 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:51:05 smithi092 ceph-mon[29572]: 2.15 scrub ok 2024-03-11T21:51:06.016 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:51:05 smithi092 ceph-mon[29572]: 5.1a deep-scrub starts 2024-03-11T21:51:06.016 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:51:05 smithi092 ceph-mon[29572]: 5.1a deep-scrub ok 2024-03-11T21:51:06.016 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:51:05 smithi092 ceph-mon[29572]: 2.2 deep-scrub starts 2024-03-11T21:51:06.016 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:51:05 smithi092 ceph-mon[29572]: 2.2 deep-scrub ok 2024-03-11T21:51:06.016 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:51:05 smithi092 ceph-mon[29572]: from='client.? v1:172.21.15.92:0/439547755' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 0}]: dispatch 2024-03-11T21:51:06.016 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:51:05 smithi092 ceph-mon[37563]: 2.15 scrub starts 2024-03-11T21:51:06.017 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:51:05 smithi092 ceph-mon[37563]: 2.15 scrub ok 2024-03-11T21:51:06.017 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:51:05 smithi092 ceph-mon[37563]: 5.1a deep-scrub starts 2024-03-11T21:51:06.017 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:51:05 smithi092 ceph-mon[37563]: 5.1a deep-scrub ok 2024-03-11T21:51:06.017 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:51:05 smithi092 ceph-mon[37563]: 2.2 deep-scrub starts 2024-03-11T21:51:06.017 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:51:05 smithi092 ceph-mon[37563]: 2.2 deep-scrub ok 2024-03-11T21:51:06.017 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:51:05 smithi092 ceph-mon[37563]: from='client.? v1:172.21.15.92:0/439547755' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 0}]: dispatch 2024-03-11T21:51:06.086 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:51:05 smithi097 ceph-mon[33317]: 2.15 scrub starts 2024-03-11T21:51:06.086 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:51:05 smithi097 ceph-mon[33317]: 2.15 scrub ok 2024-03-11T21:51:06.086 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:51:05 smithi097 ceph-mon[33317]: 5.1a deep-scrub starts 2024-03-11T21:51:06.086 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:51:05 smithi097 ceph-mon[33317]: 5.1a deep-scrub ok 2024-03-11T21:51:06.086 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:51:05 smithi097 ceph-mon[33317]: 2.2 deep-scrub starts 2024-03-11T21:51:06.087 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:51:05 smithi097 ceph-mon[33317]: 2.2 deep-scrub ok 2024-03-11T21:51:06.087 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:51:05 smithi097 ceph-mon[33317]: from='client.? v1:172.21.15.92:0/439547755' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 0}]: dispatch 2024-03-11T21:51:06.552 INFO:tasks.cephadm.ceph_manager.ceph:need seq 77309411658 got 77309411662 for osd.2 2024-03-11T21:51:06.552 DEBUG:teuthology.parallel:result is None 2024-03-11T21:51:06.829 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:51:06 smithi092 ceph-mon[29572]: 6.1c scrub starts 2024-03-11T21:51:06.829 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:51:06 smithi092 ceph-mon[29572]: 6.1c scrub ok 2024-03-11T21:51:06.829 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:51:06 smithi092 ceph-mon[29572]: pgmap v1211: 132 pgs: 132 active+clean; 582 KiB data, 6.3 GiB used, 709 GiB / 715 GiB avail; 938 B/s rd, 0 op/s 2024-03-11T21:51:06.829 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:51:06 smithi092 ceph-mon[29572]: 5.7 scrub starts 2024-03-11T21:51:06.829 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:51:06 smithi092 ceph-mon[29572]: 5.7 scrub ok 2024-03-11T21:51:06.829 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:51:06 smithi092 ceph-mon[29572]: 2.10 scrub starts 2024-03-11T21:51:06.829 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:51:06 smithi092 ceph-mon[29572]: 2.10 scrub ok 2024-03-11T21:51:06.829 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:51:06 smithi092 ceph-mon[29572]: purged_snaps scrub starts 2024-03-11T21:51:06.829 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:51:06 smithi092 ceph-mon[29572]: purged_snaps scrub ok 2024-03-11T21:51:06.830 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:51:06 smithi092 ceph-mon[37563]: 6.1c scrub starts 2024-03-11T21:51:06.830 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:51:06 smithi092 ceph-mon[37563]: 6.1c scrub ok 2024-03-11T21:51:06.830 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:51:06 smithi092 ceph-mon[37563]: pgmap v1211: 132 pgs: 132 active+clean; 582 KiB data, 6.3 GiB used, 709 GiB / 715 GiB avail; 938 B/s rd, 0 op/s 2024-03-11T21:51:06.830 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:51:06 smithi092 ceph-mon[37563]: 5.7 scrub starts 2024-03-11T21:51:06.830 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:51:06 smithi092 ceph-mon[37563]: 5.7 scrub ok 2024-03-11T21:51:06.830 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:51:06 smithi092 ceph-mon[37563]: 2.10 scrub starts 2024-03-11T21:51:06.830 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:51:06 smithi092 ceph-mon[37563]: 2.10 scrub ok 2024-03-11T21:51:06.830 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:51:06 smithi092 ceph-mon[37563]: purged_snaps scrub starts 2024-03-11T21:51:06.830 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:51:06 smithi092 ceph-mon[37563]: purged_snaps scrub ok 2024-03-11T21:51:07.086 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:51:06 smithi097 ceph-mon[33317]: 6.1c scrub starts 2024-03-11T21:51:07.086 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:51:06 smithi097 ceph-mon[33317]: 6.1c scrub ok 2024-03-11T21:51:07.086 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:51:06 smithi097 ceph-mon[33317]: pgmap v1211: 132 pgs: 132 active+clean; 582 KiB data, 6.3 GiB used, 709 GiB / 715 GiB avail; 938 B/s rd, 0 op/s 2024-03-11T21:51:07.086 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:51:06 smithi097 ceph-mon[33317]: 5.7 scrub starts 2024-03-11T21:51:07.087 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:51:06 smithi097 ceph-mon[33317]: 5.7 scrub ok 2024-03-11T21:51:07.087 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:51:06 smithi097 ceph-mon[33317]: 2.10 scrub starts 2024-03-11T21:51:07.087 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:51:06 smithi097 ceph-mon[33317]: 2.10 scrub ok 2024-03-11T21:51:07.087 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:51:06 smithi097 ceph-mon[33317]: purged_snaps scrub starts 2024-03-11T21:51:07.087 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:51:06 smithi097 ceph-mon[33317]: purged_snaps scrub ok 2024-03-11T21:51:07.379 INFO:tasks.cephadm.ceph_manager.ceph:need seq 137438953793 got 137438953796 for osd.4 2024-03-11T21:51:07.379 DEBUG:teuthology.parallel:result is None 2024-03-11T21:51:08.010 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:51:07 smithi092 ceph-mon[29572]: osdmap e554: 8 total, 8 up, 8 in 2024-03-11T21:51:08.010 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:51:07 smithi092 ceph-mon[37563]: osdmap e554: 8 total, 8 up, 8 in 2024-03-11T21:51:08.011 INFO:journalctl@ceph.mgr.y.smithi092.stdout:Mar 11 21:51:07 smithi092 ceph-00db0016-dfed-11ee-95c8-87774f69a715-mgr-y[29801]: ::ffff:172.21.15.97 - - [11/Mar/2024:21:51:07] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-11T21:51:08.086 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:51:07 smithi097 ceph-mon[33317]: osdmap e554: 8 total, 8 up, 8 in 2024-03-11T21:51:08.657 INFO:tasks.cephadm.ceph_manager.ceph:need seq 34359738710 got 34359738713 for osd.0 2024-03-11T21:51:08.657 DEBUG:teuthology.parallel:result is None 2024-03-11T21:51:08.981 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:51:08 smithi092 ceph-mon[29572]: pgmap v1213: 132 pgs: 132 active+clean; 582 KiB data, 6.3 GiB used, 709 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-11T21:51:08.982 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:51:08 smithi092 ceph-mon[29572]: from='mgr.14607 v1:172.21.15.92:0/1702717597' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T21:51:08.982 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:51:08 smithi092 ceph-mon[37563]: pgmap v1213: 132 pgs: 132 active+clean; 582 KiB data, 6.3 GiB used, 709 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-11T21:51:08.982 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:51:08 smithi092 ceph-mon[37563]: from='mgr.14607 v1:172.21.15.92:0/1702717597' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T21:51:09.063 INFO:teuthology.orchestra.run.smithi092.stdout:1189705941168 2024-03-11T21:51:09.086 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:51:08 smithi097 ceph-mon[33317]: pgmap v1213: 132 pgs: 132 active+clean; 582 KiB data, 6.3 GiB used, 709 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-11T21:51:09.086 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:51:08 smithi097 ceph-mon[33317]: from='mgr.14607 v1:172.21.15.92:0/1702717597' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-11T21:51:09.719 INFO:tasks.cephadm.ceph_manager.ceph:need seq 1189705941165 got 1189705941168 for osd.3 2024-03-11T21:51:09.719 DEBUG:teuthology.parallel:result is None 2024-03-11T21:51:10.086 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:51:09 smithi097 ceph-mon[33317]: 2.12 scrub starts 2024-03-11T21:51:10.086 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:51:09 smithi097 ceph-mon[33317]: 2.12 scrub ok 2024-03-11T21:51:10.086 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:51:09 smithi097 ceph-mon[33317]: from='client.? v1:172.21.15.92:0/3614464000' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 3}]: dispatch 2024-03-11T21:51:10.096 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:51:09 smithi092 ceph-mon[29572]: 2.12 scrub starts 2024-03-11T21:51:10.096 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:51:09 smithi092 ceph-mon[29572]: 2.12 scrub ok 2024-03-11T21:51:10.096 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:51:09 smithi092 ceph-mon[29572]: from='client.? v1:172.21.15.92:0/3614464000' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 3}]: dispatch 2024-03-11T21:51:10.096 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:51:09 smithi092 ceph-mon[37563]: 2.12 scrub starts 2024-03-11T21:51:10.096 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:51:09 smithi092 ceph-mon[37563]: 2.12 scrub ok 2024-03-11T21:51:10.096 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:51:09 smithi092 ceph-mon[37563]: from='client.? v1:172.21.15.92:0/3614464000' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 3}]: dispatch 2024-03-11T21:51:10.153 INFO:teuthology.orchestra.run.smithi092.stdout:176093659456 2024-03-11T21:51:10.870 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:51:10 smithi092 ceph-mon[37563]: 4.12 scrub starts 2024-03-11T21:51:10.870 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:51:10 smithi092 ceph-mon[37563]: 4.12 scrub ok 2024-03-11T21:51:10.870 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:51:10 smithi092 ceph-mon[37563]: 6.12 scrub starts 2024-03-11T21:51:10.870 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:51:10 smithi092 ceph-mon[37563]: 6.12 scrub ok 2024-03-11T21:51:10.870 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:51:10 smithi092 ceph-mon[37563]: pgmap v1214: 132 pgs: 132 active+clean; 582 KiB data, 6.3 GiB used, 709 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-11T21:51:10.870 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:51:10 smithi092 ceph-mon[37563]: from='mgr.14607 v1:172.21.15.92:0/1702717597' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T21:51:10.871 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:51:10 smithi092 ceph-mon[37563]: 6.1f scrub starts 2024-03-11T21:51:10.871 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:51:10 smithi092 ceph-mon[37563]: 2.1 scrub starts 2024-03-11T21:51:10.871 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:51:10 smithi092 ceph-mon[37563]: 2.1 scrub ok 2024-03-11T21:51:10.871 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:51:10 smithi092 ceph-mon[37563]: from='client.? v1:172.21.15.92:0/506656457' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 5}]: dispatch 2024-03-11T21:51:10.871 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:51:10 smithi092 ceph-mon[29572]: 4.12 scrub starts 2024-03-11T21:51:10.871 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:51:10 smithi092 ceph-mon[29572]: 4.12 scrub ok 2024-03-11T21:51:10.871 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:51:10 smithi092 ceph-mon[29572]: 6.12 scrub starts 2024-03-11T21:51:10.871 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:51:10 smithi092 ceph-mon[29572]: 6.12 scrub ok 2024-03-11T21:51:10.871 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:51:10 smithi092 ceph-mon[29572]: pgmap v1214: 132 pgs: 132 active+clean; 582 KiB data, 6.3 GiB used, 709 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-11T21:51:10.871 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:51:10 smithi092 ceph-mon[29572]: from='mgr.14607 v1:172.21.15.92:0/1702717597' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T21:51:10.871 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:51:10 smithi092 ceph-mon[29572]: 6.1f scrub starts 2024-03-11T21:51:10.871 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:51:10 smithi092 ceph-mon[29572]: 2.1 scrub starts 2024-03-11T21:51:10.872 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:51:10 smithi092 ceph-mon[29572]: 2.1 scrub ok 2024-03-11T21:51:10.872 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:51:10 smithi092 ceph-mon[29572]: from='client.? v1:172.21.15.92:0/506656457' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 5}]: dispatch 2024-03-11T21:51:11.017 INFO:tasks.cephadm.ceph_manager.ceph:need seq 176093659451 got 176093659456 for osd.5 2024-03-11T21:51:11.017 DEBUG:teuthology.parallel:result is None 2024-03-11T21:51:11.086 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:51:10 smithi097 ceph-mon[33317]: 4.12 scrub starts 2024-03-11T21:51:11.086 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:51:10 smithi097 ceph-mon[33317]: 4.12 scrub ok 2024-03-11T21:51:11.086 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:51:10 smithi097 ceph-mon[33317]: 6.12 scrub starts 2024-03-11T21:51:11.086 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:51:10 smithi097 ceph-mon[33317]: 6.12 scrub ok 2024-03-11T21:51:11.086 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:51:10 smithi097 ceph-mon[33317]: pgmap v1214: 132 pgs: 132 active+clean; 582 KiB data, 6.3 GiB used, 709 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-11T21:51:11.086 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:51:10 smithi097 ceph-mon[33317]: from='mgr.14607 v1:172.21.15.92:0/1702717597' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T21:51:11.086 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:51:10 smithi097 ceph-mon[33317]: 6.1f scrub starts 2024-03-11T21:51:11.086 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:51:10 smithi097 ceph-mon[33317]: 2.1 scrub starts 2024-03-11T21:51:11.087 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:51:10 smithi097 ceph-mon[33317]: 2.1 scrub ok 2024-03-11T21:51:11.087 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:51:10 smithi097 ceph-mon[33317]: from='client.? v1:172.21.15.92:0/506656457' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 5}]: dispatch 2024-03-11T21:51:11.845 INFO:teuthology.orchestra.run.smithi092.stdout:55834575190 2024-03-11T21:51:11.845 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:51:11 smithi092 ceph-mon[37563]: 6.1f scrub ok 2024-03-11T21:51:11.845 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:51:11 smithi092 ceph-mon[37563]: from='mgr.14607 v1:172.21.15.92:0/1702717597' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T21:51:11.845 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:51:11 smithi092 ceph-mon[37563]: from='mgr.14607 v1:172.21.15.92:0/1702717597' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T21:51:11.845 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:51:11 smithi092 ceph-mon[37563]: from='mgr.14607 ' entity='mgr.y' 2024-03-11T21:51:11.845 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:51:11 smithi092 ceph-mon[29572]: 6.1f scrub ok 2024-03-11T21:51:11.846 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:51:11 smithi092 ceph-mon[29572]: from='mgr.14607 v1:172.21.15.92:0/1702717597' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T21:51:11.846 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:51:11 smithi092 ceph-mon[29572]: from='mgr.14607 v1:172.21.15.92:0/1702717597' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T21:51:11.846 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:51:11 smithi092 ceph-mon[29572]: from='mgr.14607 ' entity='mgr.y' 2024-03-11T21:51:12.086 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:51:11 smithi097 ceph-mon[33317]: 6.1f scrub ok 2024-03-11T21:51:12.086 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:51:11 smithi097 ceph-mon[33317]: from='mgr.14607 v1:172.21.15.92:0/1702717597' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T21:51:12.086 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:51:11 smithi097 ceph-mon[33317]: from='mgr.14607 v1:172.21.15.92:0/1702717597' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T21:51:12.086 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:51:11 smithi097 ceph-mon[33317]: from='mgr.14607 ' entity='mgr.y' 2024-03-11T21:51:12.587 INFO:tasks.cephadm.ceph_manager.ceph:need seq 55834575186 got 55834575190 for osd.1 2024-03-11T21:51:12.588 DEBUG:teuthology.parallel:result is None 2024-03-11T21:51:12.588 INFO:tasks.cephadm.ceph_manager.ceph:waiting for recovery to complete 2024-03-11T21:51:12.588 DEBUG:teuthology.orchestra.run.smithi092:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid 00db0016-dfed-11ee-95c8-87774f69a715 -- ceph pg dump --format=json 2024-03-11T21:51:12.836 INFO:teuthology.orchestra.run.smithi092.stderr:Inferring config /var/lib/ceph/00db0016-dfed-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T21:51:12.870 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:51:12 smithi092 ceph-mon[37563]: pgmap v1215: 132 pgs: 132 active+clean; 582 KiB data, 6.3 GiB used, 709 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-11T21:51:12.870 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:51:12 smithi092 ceph-mon[37563]: from='client.? v1:172.21.15.92:0/761818' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 1}]: dispatch 2024-03-11T21:51:12.870 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:51:12 smithi092 ceph-mon[37563]: 4.f scrub starts 2024-03-11T21:51:12.870 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:51:12 smithi092 ceph-mon[37563]: 4.f scrub ok 2024-03-11T21:51:12.870 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:51:12 smithi092 ceph-mon[29572]: pgmap v1215: 132 pgs: 132 active+clean; 582 KiB data, 6.3 GiB used, 709 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-11T21:51:12.870 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:51:12 smithi092 ceph-mon[29572]: from='client.? v1:172.21.15.92:0/761818' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 1}]: dispatch 2024-03-11T21:51:12.870 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:51:12 smithi092 ceph-mon[29572]: 4.f scrub starts 2024-03-11T21:51:12.870 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:51:12 smithi092 ceph-mon[29572]: 4.f scrub ok 2024-03-11T21:51:13.086 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:51:12 smithi097 ceph-mon[33317]: pgmap v1215: 132 pgs: 132 active+clean; 582 KiB data, 6.3 GiB used, 709 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-11T21:51:13.086 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:51:12 smithi097 ceph-mon[33317]: from='client.? v1:172.21.15.92:0/761818' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 1}]: dispatch 2024-03-11T21:51:13.086 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:51:12 smithi097 ceph-mon[33317]: 4.f scrub starts 2024-03-11T21:51:13.086 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:51:12 smithi097 ceph-mon[33317]: 4.f scrub ok 2024-03-11T21:51:13.836 INFO:journalctl@ceph.iscsi.iscsi.a.smithi097.stdout:Mar 11 21:51:13 smithi097 ceph-00db0016-dfed-11ee-95c8-87774f69a715-iscsi-iscsi-a[60943]: debug there is no tcmu-runner data available 2024-03-11T21:51:14.838 INFO:teuthology.orchestra.run.smithi092.stdout: 2024-03-11T21:51:14.840 INFO:teuthology.orchestra.run.smithi092.stderr:dumped all 2024-03-11T21:51:14.907 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:51:14 smithi092 ceph-mon[29572]: 1.0 deep-scrub starts 2024-03-11T21:51:14.907 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:51:14 smithi092 ceph-mon[29572]: 1.0 deep-scrub ok 2024-03-11T21:51:14.908 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:51:14 smithi092 ceph-mon[29572]: from='client.24469 v1:172.21.15.97:0/404208310' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:51:14.908 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:51:14 smithi092 ceph-mon[29572]: pgmap v1216: 132 pgs: 132 active+clean; 582 KiB data, 6.3 GiB used, 709 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-11T21:51:14.908 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:51:14 smithi092 ceph-mon[37563]: 1.0 deep-scrub starts 2024-03-11T21:51:14.908 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:51:14 smithi092 ceph-mon[37563]: 1.0 deep-scrub ok 2024-03-11T21:51:14.908 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:51:14 smithi092 ceph-mon[37563]: from='client.24469 v1:172.21.15.97:0/404208310' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:51:14.908 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:51:14 smithi092 ceph-mon[37563]: pgmap v1216: 132 pgs: 132 active+clean; 582 KiB data, 6.3 GiB used, 709 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-11T21:51:15.086 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:51:14 smithi097 ceph-mon[33317]: 1.0 deep-scrub starts 2024-03-11T21:51:15.086 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:51:14 smithi097 ceph-mon[33317]: 1.0 deep-scrub ok 2024-03-11T21:51:15.086 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:51:14 smithi097 ceph-mon[33317]: from='client.24469 v1:172.21.15.97:0/404208310' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-03-11T21:51:15.087 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:51:14 smithi097 ceph-mon[33317]: pgmap v1216: 132 pgs: 132 active+clean; 582 KiB data, 6.3 GiB used, 709 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-11T21:51:15.577 INFO:teuthology.orchestra.run.smithi092.stdout:{"pg_ready":true,"pg_map":{"version":1216,"stamp":"2024-03-11T21:51:13.561322+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":596392,"num_objects":198,"num_object_clones":0,"num_object_copies":594,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":198,"num_whiteouts":0,"num_read":3110,"num_read_kb":2958,"num_write":1187,"num_write_kb":2461,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":618,"num_bytes_recovered":2967363,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1349,"ondisk_log_size":1349,"up":396,"acting":396,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":396,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":749961216,"kb_used":6651244,"kb_used_data":7580,"kb_used_omap":15,"kb_used_meta":6643568,"kb_avail":743309972,"statfs":{"total":767960285184,"available":761149411328,"internally_reserved":0,"allocated":7761920,"data_stored":4700360,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15601,"internal_metadata":6803014415},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":10,"num_read_kb":10,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"10.002701"},"pg_stats":[{"pgid":"6.1b","version":"0'0","reported_seq":69,"reported_epoch":554,"state":"active+clean","last_fresh":"2024-03-11T21:51:07.586557+0000","last_change":"2024-03-11T21:49:56.669997+0000","last_active":"2024-03-11T21:51:07.586557+0000","last_peered":"2024-03-11T21:51:07.586557+0000","last_clean":"2024-03-11T21:51:07.586557+0000","last_became_active":"2024-03-11T21:49:17.142926+0000","last_became_peered":"2024-03-11T21:49:17.142926+0000","last_unstale":"2024-03-11T21:51:07.586557+0000","last_undegraded":"2024-03-11T21:51:07.586557+0000","last_fullsized":"2024-03-11T21:51:07.586557+0000","mapping_epoch":528,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":529,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:49:56.669893+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:43.559452+0000","last_clean_scrub_stamp":"2024-03-11T21:49:56.669893+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:51:14.065489+0000","scrub_duration":110,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":61,"reported_epoch":554,"state":"active+clean","last_fresh":"2024-03-11T21:51:06.722019+0000","last_change":"2024-03-11T21:50:14.043813+0000","last_active":"2024-03-11T21:51:06.722019+0000","last_peered":"2024-03-11T21:51:06.722019+0000","last_clean":"2024-03-11T21:51:06.722019+0000","last_became_active":"2024-03-11T21:49:26.642290+0000","last_became_peered":"2024-03-11T21:49:26.642290+0000","last_unstale":"2024-03-11T21:51:06.722019+0000","last_undegraded":"2024-03-11T21:51:06.722019+0000","last_fullsized":"2024-03-11T21:51:06.722019+0000","mapping_epoch":532,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":533,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:50:14.043668+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:41.544550+0000","last_clean_scrub_stamp":"2024-03-11T21:50:14.043668+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:51:21.827740+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1f","version":"0'0","reported_seq":1101,"reported_epoch":554,"state":"active+clean","last_fresh":"2024-03-11T21:51:06.722055+0000","last_change":"2024-03-11T21:50:26.304441+0000","last_active":"2024-03-11T21:51:06.722055+0000","last_peered":"2024-03-11T21:51:06.722055+0000","last_clean":"2024-03-11T21:51:06.722055+0000","last_became_active":"2024-03-11T21:49:26.643480+0000","last_became_peered":"2024-03-11T21:49:26.643480+0000","last_unstale":"2024-03-11T21:51:06.722055+0000","last_undegraded":"2024-03-11T21:51:06.722055+0000","last_fullsized":"2024-03-11T21:51:06.722055+0000","mapping_epoch":532,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":533,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:50:26.304337+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:37.512018+0000","last_clean_scrub_stamp":"2024-03-11T21:50:26.304337+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:51:40.664486+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,4],"acting":[0,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.19","version":"456'39","reported_seq":85,"reported_epoch":554,"state":"active+clean","last_fresh":"2024-03-11T21:51:07.586290+0000","last_change":"2024-03-11T21:50:10.491803+0000","last_active":"2024-03-11T21:51:07.586290+0000","last_peered":"2024-03-11T21:51:07.586290+0000","last_clean":"2024-03-11T21:51:07.586290+0000","last_became_active":"2024-03-11T21:49:17.158133+0000","last_became_peered":"2024-03-11T21:49:17.158133+0000","last_unstale":"2024-03-11T21:51:07.586290+0000","last_undegraded":"2024-03-11T21:51:07.586290+0000","last_fullsized":"2024-03-11T21:51:07.586290+0000","mapping_epoch":528,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":529,"parent":"0.0","parent_split_bits":0,"last_scrub":"456'39","last_scrub_stamp":"2024-03-11T21:50:10.491705+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:39.525554+0000","last_clean_scrub_stamp":"2024-03-11T21:50:10.491705+0000","objects_scrubbed":7,"log_size":39,"log_dups_size":0,"ondisk_log_size":39,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:51:20.845568+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":60,"num_read_kb":51,"num_write":40,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.1a","version":"0'0","reported_seq":61,"reported_epoch":554,"state":"active+clean","last_fresh":"2024-03-11T21:51:06.722495+0000","last_change":"2024-03-11T21:50:39.258564+0000","last_active":"2024-03-11T21:51:06.722495+0000","last_peered":"2024-03-11T21:51:06.722495+0000","last_clean":"2024-03-11T21:51:06.722495+0000","last_became_active":"2024-03-11T21:49:26.646919+0000","last_became_peered":"2024-03-11T21:49:26.646919+0000","last_unstale":"2024-03-11T21:51:06.722495+0000","last_undegraded":"2024-03-11T21:51:06.722495+0000","last_fullsized":"2024-03-11T21:51:06.722495+0000","mapping_epoch":532,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":533,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:50:39.258444+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:49:02.060422+0000","last_clean_scrub_stamp":"2024-03-11T21:50:39.258444+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:51:39.565559+0000","scrub_duration":42,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,1],"acting":[4,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":956,"reported_epoch":554,"state":"active+clean","last_fresh":"2024-03-11T21:51:06.961417+0000","last_change":"2024-03-11T21:50:57.005135+0000","last_active":"2024-03-11T21:51:06.961417+0000","last_peered":"2024-03-11T21:51:06.961417+0000","last_clean":"2024-03-11T21:51:06.961417+0000","last_became_active":"2024-03-11T21:49:46.949177+0000","last_became_peered":"2024-03-11T21:49:46.949177+0000","last_unstale":"2024-03-11T21:51:06.961417+0000","last_undegraded":"2024-03-11T21:51:06.961417+0000","last_fullsized":"2024-03-11T21:51:06.961417+0000","mapping_epoch":537,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":538,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:50:57.005051+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:49:40.979721+0000","last_clean_scrub_stamp":"2024-03-11T21:50:57.005051+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:52:16.657490+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,7],"acting":[1,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1e","version":"0'0","reported_seq":77,"reported_epoch":554,"state":"active+clean","last_fresh":"2024-03-11T21:51:07.586910+0000","last_change":"2024-03-11T21:50:58.822295+0000","last_active":"2024-03-11T21:51:07.586910+0000","last_peered":"2024-03-11T21:51:07.586910+0000","last_clean":"2024-03-11T21:51:07.586910+0000","last_became_active":"2024-03-11T21:49:17.157153+0000","last_became_peered":"2024-03-11T21:49:17.157153+0000","last_unstale":"2024-03-11T21:51:07.586910+0000","last_undegraded":"2024-03-11T21:51:07.586910+0000","last_fullsized":"2024-03-11T21:51:07.586910+0000","mapping_epoch":528,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":529,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:50:58.822186+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:37.512018+0000","last_clean_scrub_stamp":"2024-03-11T21:50:58.822186+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:52:21.020159+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.18","version":"456'21","reported_seq":71,"reported_epoch":554,"state":"active+clean","last_fresh":"2024-03-11T21:51:06.721956+0000","last_change":"2024-03-11T21:50:42.147723+0000","last_active":"2024-03-11T21:51:06.721956+0000","last_peered":"2024-03-11T21:51:06.721956+0000","last_clean":"2024-03-11T21:51:06.721956+0000","last_became_active":"2024-03-11T21:49:26.657847+0000","last_became_peered":"2024-03-11T21:49:26.657847+0000","last_unstale":"2024-03-11T21:51:06.721956+0000","last_undegraded":"2024-03-11T21:51:06.721956+0000","last_fullsized":"2024-03-11T21:51:06.721956+0000","mapping_epoch":532,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":533,"parent":"0.0","parent_split_bits":0,"last_scrub":"456'21","last_scrub_stamp":"2024-03-11T21:50:42.147616+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:39.525554+0000","last_clean_scrub_stamp":"2024-03-11T21:50:42.147616+0000","objects_scrubbed":4,"log_size":21,"log_dups_size":0,"ondisk_log_size":21,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:51:52.391955+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":33,"num_read_kb":27,"num_write":22,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":136,"reported_epoch":554,"state":"active+clean","last_fresh":"2024-03-11T21:51:07.481565+0000","last_change":"2024-03-11T21:50:44.904094+0000","last_active":"2024-03-11T21:51:07.481565+0000","last_peered":"2024-03-11T21:51:07.481565+0000","last_clean":"2024-03-11T21:51:07.481565+0000","last_became_active":"2024-03-11T21:49:17.146172+0000","last_became_peered":"2024-03-11T21:49:17.146172+0000","last_unstale":"2024-03-11T21:51:07.481565+0000","last_undegraded":"2024-03-11T21:51:07.481565+0000","last_fullsized":"2024-03-11T21:51:07.481565+0000","mapping_epoch":528,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":529,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:50:44.903949+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:49:44.002008+0000","last_clean_scrub_stamp":"2024-03-11T21:50:44.903949+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:51:45.592855+0000","scrub_duration":80,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3],"acting":[5,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":729,"reported_epoch":554,"state":"active+clean","last_fresh":"2024-03-11T21:51:07.587143+0000","last_change":"2024-03-11T21:51:04.451193+0000","last_active":"2024-03-11T21:51:07.587143+0000","last_peered":"2024-03-11T21:51:07.587143+0000","last_clean":"2024-03-11T21:51:07.587143+0000","last_became_active":"2024-03-11T21:49:26.644138+0000","last_became_peered":"2024-03-11T21:49:26.644138+0000","last_unstale":"2024-03-11T21:51:07.587143+0000","last_undegraded":"2024-03-11T21:51:07.587143+0000","last_fullsized":"2024-03-11T21:51:07.587143+0000","mapping_epoch":532,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":533,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:51:04.451016+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:51:04.451016+0000","last_clean_scrub_stamp":"2024-03-11T21:51:04.451016+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:52:26.827639+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.1d","version":"0'0","reported_seq":1081,"reported_epoch":554,"state":"active+clean","last_fresh":"2024-03-11T21:51:07.587778+0000","last_change":"2024-03-11T21:50:57.457890+0000","last_active":"2024-03-11T21:51:07.587778+0000","last_peered":"2024-03-11T21:51:07.587778+0000","last_clean":"2024-03-11T21:51:07.587778+0000","last_became_active":"2024-03-11T21:25:38.542628+0000","last_became_peered":"2024-03-11T21:25:38.542628+0000","last_unstale":"2024-03-11T21:51:07.587778+0000","last_undegraded":"2024-03-11T21:51:07.587778+0000","last_fullsized":"2024-03-11T21:51:07.587778+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:50:57.457794+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:37.512018+0000","last_clean_scrub_stamp":"2024-03-11T21:50:57.457794+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:52:21.140653+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,0],"acting":[7,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.1b","version":"456'9","reported_seq":67,"reported_epoch":554,"state":"active+clean","last_fresh":"2024-03-11T21:51:06.722321+0000","last_change":"2024-03-11T21:50:31.115835+0000","last_active":"2024-03-11T21:51:06.722321+0000","last_peered":"2024-03-11T21:51:06.722321+0000","last_clean":"2024-03-11T21:51:06.722321+0000","last_became_active":"2024-03-11T21:49:26.639069+0000","last_became_peered":"2024-03-11T21:49:26.639069+0000","last_unstale":"2024-03-11T21:51:06.722321+0000","last_undegraded":"2024-03-11T21:51:06.722321+0000","last_fullsized":"2024-03-11T21:51:06.722321+0000","mapping_epoch":532,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":533,"parent":"0.0","parent_split_bits":0,"last_scrub":"456'9","last_scrub_stamp":"2024-03-11T21:50:31.115727+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:39.525554+0000","last_clean_scrub_stamp":"2024-03-11T21:50:31.115727+0000","objects_scrubbed":2,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:51:33.406175+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":16,"num_read_kb":12,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,1],"acting":[4,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":1055,"reported_epoch":554,"state":"active+clean","last_fresh":"2024-03-11T21:51:06.720888+0000","last_change":"2024-03-11T21:50:44.360542+0000","last_active":"2024-03-11T21:51:06.720888+0000","last_peered":"2024-03-11T21:51:06.720888+0000","last_clean":"2024-03-11T21:51:06.720888+0000","last_became_active":"2024-03-11T21:25:44.589786+0000","last_became_peered":"2024-03-11T21:25:44.589786+0000","last_unstale":"2024-03-11T21:51:06.720888+0000","last_undegraded":"2024-03-11T21:51:06.720888+0000","last_fullsized":"2024-03-11T21:51:06.720888+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:50:44.360459+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:43.559452+0000","last_clean_scrub_stamp":"2024-03-11T21:50:44.360459+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:51:52.384583+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,7],"acting":[0,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":130,"reported_epoch":554,"state":"active+clean","last_fresh":"2024-03-11T21:51:07.481812+0000","last_change":"2024-03-11T21:51:02.647015+0000","last_active":"2024-03-11T21:51:07.481812+0000","last_peered":"2024-03-11T21:51:07.481812+0000","last_clean":"2024-03-11T21:51:07.481812+0000","last_became_active":"2024-03-11T21:48:28.644231+0000","last_became_peered":"2024-03-11T21:48:28.644231+0000","last_unstale":"2024-03-11T21:51:07.481812+0000","last_undegraded":"2024-03-11T21:51:07.481812+0000","last_fullsized":"2024-03-11T21:51:07.481812+0000","mapping_epoch":502,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":503,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:51:02.646863+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:51:02.646863+0000","last_clean_scrub_stamp":"2024-03-11T21:51:02.646863+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:52:20.449867+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,7],"acting":[5,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1c","version":"0'0","reported_seq":1085,"reported_epoch":554,"state":"active+clean","last_fresh":"2024-03-11T21:51:07.587566+0000","last_change":"2024-03-11T21:50:06.133038+0000","last_active":"2024-03-11T21:51:07.587566+0000","last_peered":"2024-03-11T21:51:07.587566+0000","last_clean":"2024-03-11T21:51:07.587566+0000","last_became_active":"2024-03-11T21:48:28.628707+0000","last_became_peered":"2024-03-11T21:48:28.628707+0000","last_unstale":"2024-03-11T21:51:07.587566+0000","last_undegraded":"2024-03-11T21:51:07.587566+0000","last_fullsized":"2024-03-11T21:51:07.587566+0000","mapping_epoch":502,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":503,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:50:06.132933+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:28:32.566437+0000","last_clean_scrub_stamp":"2024-03-11T21:50:06.132933+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:51:14.269913+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.1a","version":"456'25","reported_seq":71,"reported_epoch":554,"state":"active+clean","last_fresh":"2024-03-11T21:51:06.722430+0000","last_change":"2024-03-11T21:50:59.258725+0000","last_active":"2024-03-11T21:51:06.722430+0000","last_peered":"2024-03-11T21:51:06.722430+0000","last_clean":"2024-03-11T21:51:06.722430+0000","last_became_active":"2024-03-11T21:49:26.635996+0000","last_became_peered":"2024-03-11T21:49:26.635996+0000","last_unstale":"2024-03-11T21:51:06.722430+0000","last_undegraded":"2024-03-11T21:51:06.722430+0000","last_fullsized":"2024-03-11T21:51:06.722430+0000","mapping_epoch":532,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":533,"parent":"0.0","parent_split_bits":0,"last_scrub":"456'25","last_scrub_stamp":"2024-03-11T21:50:59.258609+0000","last_deep_scrub":"350'19","last_deep_scrub_stamp":"2024-03-11T21:43:25.898708+0000","last_clean_scrub_stamp":"2024-03-11T21:50:59.258609+0000","objects_scrubbed":4,"log_size":25,"log_dups_size":0,"ondisk_log_size":25,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:52:15.787098+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":35,"num_read_kb":29,"num_write":22,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":11,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0],"acting":[4,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1f","version":"0'0","reported_seq":69,"reported_epoch":554,"state":"active+clean","last_fresh":"2024-03-11T21:51:07.587023+0000","last_change":"2024-03-11T21:49:51.750151+0000","last_active":"2024-03-11T21:51:07.587023+0000","last_peered":"2024-03-11T21:51:07.587023+0000","last_clean":"2024-03-11T21:51:07.587023+0000","last_became_active":"2024-03-11T21:49:17.153410+0000","last_became_peered":"2024-03-11T21:49:17.153410+0000","last_unstale":"2024-03-11T21:51:07.587023+0000","last_undegraded":"2024-03-11T21:51:07.587023+0000","last_fullsized":"2024-03-11T21:51:07.587023+0000","mapping_epoch":528,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":529,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:49:51.750046+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:44:31.472397+0000","last_clean_scrub_stamp":"2024-03-11T21:49:51.750046+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:51:08.458556+0000","scrub_duration":136,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1c","version":"0'0","reported_seq":61,"reported_epoch":554,"state":"active+clean","last_fresh":"2024-03-11T21:51:06.722509+0000","last_change":"2024-03-11T21:50:06.044885+0000","last_active":"2024-03-11T21:51:06.722509+0000","last_peered":"2024-03-11T21:51:06.722509+0000","last_clean":"2024-03-11T21:51:06.722509+0000","last_became_active":"2024-03-11T21:49:26.636081+0000","last_became_peered":"2024-03-11T21:49:26.636081+0000","last_unstale":"2024-03-11T21:51:06.722509+0000","last_undegraded":"2024-03-11T21:51:06.722509+0000","last_fullsized":"2024-03-11T21:51:06.722509+0000","mapping_epoch":532,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":533,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:50:06.044783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:43:32.021344+0000","last_clean_scrub_stamp":"2024-03-11T21:50:06.044783+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:51:34.182002+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1b","version":"0'0","reported_seq":69,"reported_epoch":554,"state":"active+clean","last_fresh":"2024-03-11T21:51:07.586466+0000","last_change":"2024-03-11T21:50:19.422317+0000","last_active":"2024-03-11T21:51:07.586466+0000","last_peered":"2024-03-11T21:51:07.586466+0000","last_clean":"2024-03-11T21:51:07.586466+0000","last_became_active":"2024-03-11T21:49:17.146126+0000","last_became_peered":"2024-03-11T21:49:17.146126+0000","last_unstale":"2024-03-11T21:51:07.586466+0000","last_undegraded":"2024-03-11T21:51:07.586466+0000","last_fullsized":"2024-03-11T21:51:07.586466+0000","mapping_epoch":528,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":529,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:50:19.422221+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:34:27.998724+0000","last_clean_scrub_stamp":"2024-03-11T21:50:19.422221+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:51:44.003421+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1d","version":"456'24","reported_seq":99,"reported_epoch":554,"state":"active+clean","last_fresh":"2024-03-11T21:51:07.586668+0000","last_change":"2024-03-11T21:50:12.450286+0000","last_active":"2024-03-11T21:51:07.586668+0000","last_peered":"2024-03-11T21:51:07.586668+0000","last_clean":"2024-03-11T21:51:07.586668+0000","last_became_active":"2024-03-11T21:49:26.650024+0000","last_became_peered":"2024-03-11T21:49:26.650024+0000","last_unstale":"2024-03-11T21:51:07.586668+0000","last_undegraded":"2024-03-11T21:51:07.586668+0000","last_fullsized":"2024-03-11T21:51:07.586668+0000","mapping_epoch":532,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":533,"parent":"0.0","parent_split_bits":0,"last_scrub":"456'24","last_scrub_stamp":"2024-03-11T21:50:12.450186+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:39.525554+0000","last_clean_scrub_stamp":"2024-03-11T21:50:12.450186+0000","objects_scrubbed":5,"log_size":24,"log_dups_size":0,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:51:24.175921+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":42,"num_read_kb":33,"num_write":26,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":30,"num_bytes_recovered":1320,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.1e","version":"0'0","reported_seq":61,"reported_epoch":554,"state":"active+clean","last_fresh":"2024-03-11T21:51:06.723134+0000","last_change":"2024-03-11T21:50:03.044169+0000","last_active":"2024-03-11T21:51:06.723134+0000","last_peered":"2024-03-11T21:51:06.723134+0000","last_clean":"2024-03-11T21:51:06.723134+0000","last_became_active":"2024-03-11T21:49:26.653347+0000","last_became_peered":"2024-03-11T21:49:26.653347+0000","last_unstale":"2024-03-11T21:51:06.723134+0000","last_undegraded":"2024-03-11T21:51:06.723134+0000","last_fullsized":"2024-03-11T21:51:06.723134+0000","mapping_epoch":532,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":533,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:50:03.044014+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:43.559452+0000","last_clean_scrub_stamp":"2024-03-11T21:50:03.044014+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:51:12.981238+0000","scrub_duration":35,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5],"acting":[4,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1d","version":"0'0","reported_seq":1086,"reported_epoch":554,"state":"active+clean","last_fresh":"2024-03-11T21:51:06.961966+0000","last_change":"2024-03-11T21:50:13.997689+0000","last_active":"2024-03-11T21:51:06.961966+0000","last_peered":"2024-03-11T21:51:06.961966+0000","last_clean":"2024-03-11T21:51:06.961966+0000","last_became_active":"2024-03-11T21:49:26.644697+0000","last_became_peered":"2024-03-11T21:49:26.644697+0000","last_unstale":"2024-03-11T21:51:06.961966+0000","last_undegraded":"2024-03-11T21:51:06.961966+0000","last_fullsized":"2024-03-11T21:51:06.961966+0000","mapping_epoch":532,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":533,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:50:13.997608+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:30:18.678531+0000","last_clean_scrub_stamp":"2024-03-11T21:50:13.997608+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:51:43.853121+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1a","version":"0'0","reported_seq":1107,"reported_epoch":554,"state":"active+clean","last_fresh":"2024-03-11T21:51:07.586478+0000","last_change":"2024-03-11T21:50:23.130342+0000","last_active":"2024-03-11T21:51:07.586478+0000","last_peered":"2024-03-11T21:51:07.586478+0000","last_clean":"2024-03-11T21:51:07.586478+0000","last_became_active":"2024-03-11T21:49:26.637141+0000","last_became_peered":"2024-03-11T21:49:26.637141+0000","last_unstale":"2024-03-11T21:51:07.586478+0000","last_undegraded":"2024-03-11T21:51:07.586478+0000","last_fullsized":"2024-03-11T21:51:07.586478+0000","mapping_epoch":532,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":533,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:50:23.130244+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:37.512018+0000","last_clean_scrub_stamp":"2024-03-11T21:50:23.130244+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:51:49.179992+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.1c","version":"456'39","reported_seq":1190,"reported_epoch":554,"state":"active+clean","last_fresh":"2024-03-11T21:51:06.721412+0000","last_change":"2024-03-11T21:50:00.832261+0000","last_active":"2024-03-11T21:51:06.721412+0000","last_peered":"2024-03-11T21:51:06.721412+0000","last_clean":"2024-03-11T21:51:06.721412+0000","last_became_active":"2024-03-11T21:49:17.153704+0000","last_became_peered":"2024-03-11T21:49:17.153704+0000","last_unstale":"2024-03-11T21:51:06.721412+0000","last_undegraded":"2024-03-11T21:51:06.721412+0000","last_fullsized":"2024-03-11T21:51:06.721412+0000","mapping_epoch":528,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":529,"parent":"0.0","parent_split_bits":0,"last_scrub":"456'39","last_scrub_stamp":"2024-03-11T21:50:00.832150+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:39.525554+0000","last_clean_scrub_stamp":"2024-03-11T21:50:00.832150+0000","objects_scrubbed":7,"log_size":39,"log_dups_size":0,"ondisk_log_size":39,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:51:29.109545+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":62,"num_read_kb":53,"num_write":40,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.1d","version":"0'0","reported_seq":938,"reported_epoch":554,"state":"active+clean","last_fresh":"2024-03-11T21:51:06.961143+0000","last_change":"2024-03-11T21:49:58.218020+0000","last_active":"2024-03-11T21:51:06.961143+0000","last_peered":"2024-03-11T21:51:06.961143+0000","last_clean":"2024-03-11T21:51:06.961143+0000","last_became_active":"2024-03-11T21:49:46.945621+0000","last_became_peered":"2024-03-11T21:49:46.945621+0000","last_unstale":"2024-03-11T21:51:06.961143+0000","last_undegraded":"2024-03-11T21:51:06.961143+0000","last_fullsized":"2024-03-11T21:51:06.961143+0000","mapping_epoch":537,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":538,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:49:58.217912+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:43.559452+0000","last_clean_scrub_stamp":"2024-03-11T21:49:58.217912+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:51:26.736914+0000","scrub_duration":229,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.1e","version":"0'0","reported_seq":1069,"reported_epoch":554,"state":"active+clean","last_fresh":"2024-03-11T21:51:06.722017+0000","last_change":"2024-03-11T21:50:58.397266+0000","last_active":"2024-03-11T21:51:06.722017+0000","last_peered":"2024-03-11T21:51:06.722017+0000","last_clean":"2024-03-11T21:51:06.722017+0000","last_became_active":"2024-03-11T21:25:42.582959+0000","last_became_peered":"2024-03-11T21:25:42.582959+0000","last_unstale":"2024-03-11T21:51:06.722017+0000","last_undegraded":"2024-03-11T21:51:06.722017+0000","last_fullsized":"2024-03-11T21:51:06.722017+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:50:58.397183+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:44:28.222102+0000","last_clean_scrub_stamp":"2024-03-11T21:50:58.397183+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:52:16.575984+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.19","version":"67'2","reported_seq":83,"reported_epoch":554,"state":"active+clean","last_fresh":"2024-03-11T21:51:07.586189+0000","last_change":"2024-03-11T21:50:43.666057+0000","last_active":"2024-03-11T21:51:07.586189+0000","last_peered":"2024-03-11T21:51:07.586189+0000","last_clean":"2024-03-11T21:51:07.586189+0000","last_became_active":"2024-03-11T21:49:17.157291+0000","last_became_peered":"2024-03-11T21:49:17.157291+0000","last_unstale":"2024-03-11T21:51:07.586189+0000","last_undegraded":"2024-03-11T21:51:07.586189+0000","last_fullsized":"2024-03-11T21:51:07.586189+0000","mapping_epoch":528,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":529,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'2","last_scrub_stamp":"2024-03-11T21:50:43.665964+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:37.512018+0000","last_clean_scrub_stamp":"2024-03-11T21:50:43.665964+0000","objects_scrubbed":2,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:52:10.221545+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":957,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":957,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,0],"acting":[3,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1f","version":"456'32","reported_seq":1191,"reported_epoch":554,"state":"active+clean","last_fresh":"2024-03-11T21:51:07.587704+0000","last_change":"2024-03-11T21:50:53.122596+0000","last_active":"2024-03-11T21:51:07.587704+0000","last_peered":"2024-03-11T21:51:07.587704+0000","last_clean":"2024-03-11T21:51:07.587704+0000","last_became_active":"2024-03-11T21:48:28.643699+0000","last_became_peered":"2024-03-11T21:48:28.643699+0000","last_unstale":"2024-03-11T21:51:07.587704+0000","last_undegraded":"2024-03-11T21:51:07.587704+0000","last_fullsized":"2024-03-11T21:51:07.587704+0000","mapping_epoch":502,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":503,"parent":"0.0","parent_split_bits":0,"last_scrub":"456'32","last_scrub_stamp":"2024-03-11T21:50:53.122483+0000","last_deep_scrub":"456'32","last_deep_scrub_stamp":"2024-03-11T21:49:24.792724+0000","last_clean_scrub_stamp":"2024-03-11T21:50:53.122483+0000","objects_scrubbed":5,"log_size":32,"log_dups_size":0,"ondisk_log_size":32,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:51:53.163767+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":44,"num_read_kb":37,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,0],"acting":[6,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.1c","version":"467'2","reported_seq":735,"reported_epoch":554,"state":"active+clean","last_fresh":"2024-03-11T21:51:07.587420+0000","last_change":"2024-03-11T21:51:05.424581+0000","last_active":"2024-03-11T21:51:07.587420+0000","last_peered":"2024-03-11T21:51:07.587420+0000","last_clean":"2024-03-11T21:51:07.587420+0000","last_became_active":"2024-03-11T21:48:28.628768+0000","last_became_peered":"2024-03-11T21:48:28.628768+0000","last_unstale":"2024-03-11T21:51:07.587420+0000","last_undegraded":"2024-03-11T21:51:07.587420+0000","last_fullsized":"2024-03-11T21:51:07.587420+0000","mapping_epoch":502,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":503,"parent":"0.0","parent_split_bits":0,"last_scrub":"467'2","last_scrub_stamp":"2024-03-11T21:51:05.424440+0000","last_deep_scrub":"467'2","last_deep_scrub_stamp":"2024-03-11T21:46:22.517102+0000","last_clean_scrub_stamp":"2024-03-11T21:51:05.424440+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:52:13.086992+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":370,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":1110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.1f","version":"553'13","reported_seq":1453,"reported_epoch":554,"state":"active+clean","last_fresh":"2024-03-11T21:51:07.587703+0000","last_change":"2024-03-11T21:51:03.961974+0000","last_active":"2024-03-11T21:51:07.587703+0000","last_peered":"2024-03-11T21:51:07.587703+0000","last_clean":"2024-03-11T21:51:07.587703+0000","last_became_active":"2024-03-11T21:49:26.637213+0000","last_became_peered":"2024-03-11T21:49:26.637213+0000","last_unstale":"2024-03-11T21:51:07.587703+0000","last_undegraded":"2024-03-11T21:51:07.587703+0000","last_fullsized":"2024-03-11T21:51:07.587703+0000","mapping_epoch":532,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":533,"parent":"0.0","parent_split_bits":0,"last_scrub":"553'13","last_scrub_stamp":"2024-03-11T21:51:03.961840+0000","last_deep_scrub":"553'13","last_deep_scrub_stamp":"2024-03-11T21:51:03.961840+0000","last_clean_scrub_stamp":"2024-03-11T21:51:03.961840+0000","objects_scrubbed":1,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:52:06.143607+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.18","version":"67'1","reported_seq":144,"reported_epoch":554,"state":"active+clean","last_fresh":"2024-03-11T21:51:07.481405+0000","last_change":"2024-03-11T21:50:14.934032+0000","last_active":"2024-03-11T21:51:07.481405+0000","last_peered":"2024-03-11T21:51:07.481405+0000","last_clean":"2024-03-11T21:51:07.481405+0000","last_became_active":"2024-03-11T21:49:26.650243+0000","last_became_peered":"2024-03-11T21:49:26.650243+0000","last_unstale":"2024-03-11T21:51:07.481405+0000","last_undegraded":"2024-03-11T21:51:07.481405+0000","last_fullsized":"2024-03-11T21:51:07.481405+0000","mapping_epoch":532,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":533,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'1","last_scrub_stamp":"2024-03-11T21:50:14.933934+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:37.512018+0000","last_clean_scrub_stamp":"2024-03-11T21:50:14.933934+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:51:39.849515+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":436,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":7,"num_read_kb":7,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1744,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,2],"acting":[5,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1e","version":"456'30","reported_seq":1204,"reported_epoch":554,"state":"active+clean","last_fresh":"2024-03-11T21:51:06.722015+0000","last_change":"2024-03-11T21:50:09.119727+0000","last_active":"2024-03-11T21:51:06.722015+0000","last_peered":"2024-03-11T21:51:06.722015+0000","last_clean":"2024-03-11T21:51:06.722015+0000","last_became_active":"2024-03-11T21:49:17.148991+0000","last_became_peered":"2024-03-11T21:49:17.148991+0000","last_unstale":"2024-03-11T21:51:06.722015+0000","last_undegraded":"2024-03-11T21:51:06.722015+0000","last_fullsized":"2024-03-11T21:51:06.722015+0000","mapping_epoch":528,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":529,"parent":"0.0","parent_split_bits":0,"last_scrub":"456'30","last_scrub_stamp":"2024-03-11T21:50:09.119582+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:39.525554+0000","last_clean_scrub_stamp":"2024-03-11T21:50:09.119582+0000","objects_scrubbed":5,"log_size":30,"log_dups_size":0,"ondisk_log_size":30,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:51:25.770663+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":45,"num_read_kb":40,"num_write":30,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.13","version":"0'0","reported_seq":69,"reported_epoch":554,"state":"active+clean","last_fresh":"2024-03-11T21:51:07.586676+0000","last_change":"2024-03-11T21:50:21.523564+0000","last_active":"2024-03-11T21:51:07.586676+0000","last_peered":"2024-03-11T21:51:07.586676+0000","last_clean":"2024-03-11T21:51:07.586676+0000","last_became_active":"2024-03-11T21:49:17.154298+0000","last_became_peered":"2024-03-11T21:49:17.154298+0000","last_unstale":"2024-03-11T21:51:07.586676+0000","last_undegraded":"2024-03-11T21:51:07.586676+0000","last_fullsized":"2024-03-11T21:51:07.586676+0000","mapping_epoch":528,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":529,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:50:21.523443+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:43.559452+0000","last_clean_scrub_stamp":"2024-03-11T21:50:21.523443+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:51:39.157450+0000","scrub_duration":119,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":731,"reported_epoch":554,"state":"active+clean","last_fresh":"2024-03-11T21:51:07.586651+0000","last_change":"2024-03-11T21:50:49.413889+0000","last_active":"2024-03-11T21:51:07.586651+0000","last_peered":"2024-03-11T21:51:07.586651+0000","last_clean":"2024-03-11T21:51:07.586651+0000","last_became_active":"2024-03-11T21:49:26.643590+0000","last_became_peered":"2024-03-11T21:49:26.643590+0000","last_unstale":"2024-03-11T21:51:07.586651+0000","last_undegraded":"2024-03-11T21:51:07.586651+0000","last_fullsized":"2024-03-11T21:51:07.586651+0000","mapping_epoch":532,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":533,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:50:49.413780+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:49:34.678322+0000","last_clean_scrub_stamp":"2024-03-11T21:50:49.413780+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:52:18.899709+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.17","version":"0'0","reported_seq":1095,"reported_epoch":554,"state":"active+clean","last_fresh":"2024-03-11T21:51:07.587538+0000","last_change":"2024-03-11T21:50:42.135239+0000","last_active":"2024-03-11T21:51:07.587538+0000","last_peered":"2024-03-11T21:51:07.587538+0000","last_clean":"2024-03-11T21:51:07.587538+0000","last_became_active":"2024-03-11T21:48:28.623533+0000","last_became_peered":"2024-03-11T21:48:28.623533+0000","last_unstale":"2024-03-11T21:51:07.587538+0000","last_undegraded":"2024-03-11T21:51:07.587538+0000","last_fullsized":"2024-03-11T21:51:07.587538+0000","mapping_epoch":502,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":503,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:50:42.135139+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:50:42.135139+0000","last_clean_scrub_stamp":"2024-03-11T21:50:42.135139+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:51:45.993396+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,2],"acting":[6,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.11","version":"456'27","reported_seq":83,"reported_epoch":554,"state":"active+clean","last_fresh":"2024-03-11T21:51:07.481405+0000","last_change":"2024-03-11T21:50:52.714482+0000","last_active":"2024-03-11T21:51:07.481405+0000","last_peered":"2024-03-11T21:51:07.481405+0000","last_clean":"2024-03-11T21:51:07.481405+0000","last_became_active":"2024-03-11T21:49:25.632411+0000","last_became_peered":"2024-03-11T21:49:25.632411+0000","last_unstale":"2024-03-11T21:51:07.481405+0000","last_undegraded":"2024-03-11T21:51:07.481405+0000","last_fullsized":"2024-03-11T21:51:07.481405+0000","mapping_epoch":531,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":532,"parent":"0.0","parent_split_bits":0,"last_scrub":"456'27","last_scrub_stamp":"2024-03-11T21:50:52.714351+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:39.525554+0000","last_clean_scrub_stamp":"2024-03-11T21:50:52.714351+0000","objects_scrubbed":5,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:52:13.665274+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":44,"num_read_kb":37,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25,"num_bytes_recovered":550,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":737,"reported_epoch":554,"state":"active+clean","last_fresh":"2024-03-11T21:51:09.430134+0000","last_change":"2024-03-11T21:51:09.430185+0000","last_active":"2024-03-11T21:51:09.430134+0000","last_peered":"2024-03-11T21:51:09.430134+0000","last_clean":"2024-03-11T21:51:09.430134+0000","last_became_active":"2024-03-11T21:49:26.644022+0000","last_became_peered":"2024-03-11T21:49:26.644022+0000","last_unstale":"2024-03-11T21:51:09.430134+0000","last_undegraded":"2024-03-11T21:51:09.430134+0000","last_fullsized":"2024-03-11T21:51:09.430134+0000","mapping_epoch":532,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":533,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:51:09.430039+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:43.559452+0000","last_clean_scrub_stamp":"2024-03-11T21:51:09.430039+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:52:22.974534+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,4],"acting":[7,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":1107,"reported_epoch":554,"state":"active+clean","last_fresh":"2024-03-11T21:51:07.586434+0000","last_change":"2024-03-11T21:50:22.136299+0000","last_active":"2024-03-11T21:51:07.586434+0000","last_peered":"2024-03-11T21:51:07.586434+0000","last_clean":"2024-03-11T21:51:07.586434+0000","last_became_active":"2024-03-11T21:49:26.638240+0000","last_became_peered":"2024-03-11T21:49:26.638240+0000","last_unstale":"2024-03-11T21:51:07.586434+0000","last_undegraded":"2024-03-11T21:51:07.586434+0000","last_fullsized":"2024-03-11T21:51:07.586434+0000","mapping_epoch":532,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":533,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:50:22.136209+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:50:22.136209+0000","last_clean_scrub_stamp":"2024-03-11T21:50:22.136209+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:51:25.274309+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.16","version":"0'0","reported_seq":130,"reported_epoch":554,"state":"active+clean","last_fresh":"2024-03-11T21:51:07.482209+0000","last_change":"2024-03-11T21:50:11.905644+0000","last_active":"2024-03-11T21:51:07.482209+0000","last_peered":"2024-03-11T21:51:07.482209+0000","last_clean":"2024-03-11T21:51:07.482209+0000","last_became_active":"2024-03-11T21:48:28.636823+0000","last_became_peered":"2024-03-11T21:48:28.636823+0000","last_unstale":"2024-03-11T21:51:07.482209+0000","last_undegraded":"2024-03-11T21:51:07.482209+0000","last_fullsized":"2024-03-11T21:51:07.482209+0000","mapping_epoch":502,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":503,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:50:11.905527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:27:11.938909+0000","last_clean_scrub_stamp":"2024-03-11T21:50:11.905527+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:51:20.181506+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,2],"acting":[5,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.10","version":"456'14","reported_seq":83,"reported_epoch":554,"state":"active+clean","last_fresh":"2024-03-11T21:51:07.586053+0000","last_change":"2024-03-11T21:50:40.647648+0000","last_active":"2024-03-11T21:51:07.586053+0000","last_peered":"2024-03-11T21:51:07.586053+0000","last_clean":"2024-03-11T21:51:07.586053+0000","last_became_active":"2024-03-11T21:49:17.132671+0000","last_became_peered":"2024-03-11T21:49:17.132671+0000","last_unstale":"2024-03-11T21:51:07.586053+0000","last_undegraded":"2024-03-11T21:51:07.586053+0000","last_fullsized":"2024-03-11T21:51:07.586053+0000","mapping_epoch":528,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":529,"parent":"0.0","parent_split_bits":0,"last_scrub":"456'14","last_scrub_stamp":"2024-03-11T21:50:40.647495+0000","last_deep_scrub":"260'10","last_deep_scrub_stamp":"2024-03-11T21:44:32.059058+0000","last_clean_scrub_stamp":"2024-03-11T21:50:40.647495+0000","objects_scrubbed":2,"log_size":14,"log_dups_size":0,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:51:45.955329+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":18,"num_read_kb":16,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6],"acting":[3,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":69,"reported_epoch":554,"state":"active+clean","last_fresh":"2024-03-11T21:51:07.586298+0000","last_change":"2024-03-11T21:50:31.635995+0000","last_active":"2024-03-11T21:51:07.586298+0000","last_peered":"2024-03-11T21:51:07.586298+0000","last_clean":"2024-03-11T21:51:07.586298+0000","last_became_active":"2024-03-11T21:49:17.159872+0000","last_became_peered":"2024-03-11T21:49:17.159872+0000","last_unstale":"2024-03-11T21:51:07.586298+0000","last_undegraded":"2024-03-11T21:51:07.586298+0000","last_fullsized":"2024-03-11T21:51:07.586298+0000","mapping_epoch":528,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":529,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:50:31.635877+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:50:31.635877+0000","last_clean_scrub_stamp":"2024-03-11T21:50:31.635877+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:51:58.858039+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":1084,"reported_epoch":554,"state":"active+clean","last_fresh":"2024-03-11T21:51:06.961729+0000","last_change":"2024-03-11T21:49:53.032361+0000","last_active":"2024-03-11T21:51:06.961729+0000","last_peered":"2024-03-11T21:51:06.961729+0000","last_clean":"2024-03-11T21:51:06.961729+0000","last_became_active":"2024-03-11T21:49:17.138352+0000","last_became_peered":"2024-03-11T21:49:17.138352+0000","last_unstale":"2024-03-11T21:51:06.961729+0000","last_undegraded":"2024-03-11T21:51:06.961729+0000","last_fullsized":"2024-03-11T21:51:06.961729+0000","mapping_epoch":528,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":529,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:49:53.032269+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:27:21.207303+0000","last_clean_scrub_stamp":"2024-03-11T21:49:53.032269+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:51:18.975741+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.15","version":"0'0","reported_seq":1087,"reported_epoch":554,"state":"active+clean","last_fresh":"2024-03-11T21:51:06.961781+0000","last_change":"2024-03-11T21:51:03.973908+0000","last_active":"2024-03-11T21:51:06.961781+0000","last_peered":"2024-03-11T21:51:06.961781+0000","last_clean":"2024-03-11T21:51:06.961781+0000","last_became_active":"2024-03-11T21:48:28.620494+0000","last_became_peered":"2024-03-11T21:48:28.620494+0000","last_unstale":"2024-03-11T21:51:06.961781+0000","last_undegraded":"2024-03-11T21:51:06.961781+0000","last_fullsized":"2024-03-11T21:51:06.961781+0000","mapping_epoch":502,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":503,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:51:03.973831+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:26:44.860499+0000","last_clean_scrub_stamp":"2024-03-11T21:51:03.973831+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:52:11.763534+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.13","version":"456'32","reported_seq":81,"reported_epoch":554,"state":"active+clean","last_fresh":"2024-03-11T21:51:06.722453+0000","last_change":"2024-03-11T21:50:53.237986+0000","last_active":"2024-03-11T21:51:06.722453+0000","last_peered":"2024-03-11T21:51:06.722453+0000","last_clean":"2024-03-11T21:51:06.722453+0000","last_became_active":"2024-03-11T21:49:26.637845+0000","last_became_peered":"2024-03-11T21:49:26.637845+0000","last_unstale":"2024-03-11T21:51:06.722453+0000","last_undegraded":"2024-03-11T21:51:06.722453+0000","last_fullsized":"2024-03-11T21:51:06.722453+0000","mapping_epoch":532,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":533,"parent":"0.0","parent_split_bits":0,"last_scrub":"456'32","last_scrub_stamp":"2024-03-11T21:50:53.237819+0000","last_deep_scrub":"260'24","last_deep_scrub_stamp":"2024-03-11T21:40:25.595220+0000","last_clean_scrub_stamp":"2024-03-11T21:50:53.237819+0000","objects_scrubbed":5,"log_size":32,"log_dups_size":0,"ondisk_log_size":32,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:52:01.982237+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":42,"num_read_kb":35,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,2],"acting":[4,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.10","version":"0'0","reported_seq":1067,"reported_epoch":554,"state":"active+clean","last_fresh":"2024-03-11T21:51:06.720392+0000","last_change":"2024-03-11T21:50:46.410870+0000","last_active":"2024-03-11T21:51:06.720392+0000","last_peered":"2024-03-11T21:51:06.720392+0000","last_clean":"2024-03-11T21:51:06.720392+0000","last_became_active":"2024-03-11T21:48:28.641880+0000","last_became_peered":"2024-03-11T21:48:28.641880+0000","last_unstale":"2024-03-11T21:51:06.720392+0000","last_undegraded":"2024-03-11T21:51:06.720392+0000","last_fullsized":"2024-03-11T21:51:06.720392+0000","mapping_epoch":502,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":503,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:50:46.410778+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:49:16.199994+0000","last_clean_scrub_stamp":"2024-03-11T21:50:46.410778+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:51:56.329722+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,1],"acting":[0,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":77,"reported_epoch":554,"state":"active+clean","last_fresh":"2024-03-11T21:51:07.586670+0000","last_change":"2024-03-11T21:50:33.651519+0000","last_active":"2024-03-11T21:51:07.586670+0000","last_peered":"2024-03-11T21:51:07.586670+0000","last_clean":"2024-03-11T21:51:07.586670+0000","last_became_active":"2024-03-11T21:49:17.157943+0000","last_became_peered":"2024-03-11T21:49:17.157943+0000","last_unstale":"2024-03-11T21:51:07.586670+0000","last_undegraded":"2024-03-11T21:51:07.586670+0000","last_fullsized":"2024-03-11T21:51:07.586670+0000","mapping_epoch":528,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":529,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:50:33.651412+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:50:33.651412+0000","last_clean_scrub_stamp":"2024-03-11T21:50:33.651412+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:51:33.993223+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.14","version":"0'0","reported_seq":1132,"reported_epoch":554,"state":"active+clean","last_fresh":"2024-03-11T21:51:07.586935+0000","last_change":"2024-03-11T21:50:51.099964+0000","last_active":"2024-03-11T21:51:07.586935+0000","last_peered":"2024-03-11T21:51:07.586935+0000","last_clean":"2024-03-11T21:51:07.586935+0000","last_became_active":"2024-03-11T21:49:17.133506+0000","last_became_peered":"2024-03-11T21:49:17.133506+0000","last_unstale":"2024-03-11T21:51:07.586935+0000","last_undegraded":"2024-03-11T21:51:07.586935+0000","last_fullsized":"2024-03-11T21:51:07.586935+0000","mapping_epoch":528,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":529,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:50:51.099855+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:37.512018+0000","last_clean_scrub_stamp":"2024-03-11T21:50:51.099855+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:52:09.556377+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,5],"acting":[6,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.12","version":"456'21","reported_seq":1105,"reported_epoch":554,"state":"active+clean","last_fresh":"2024-03-11T21:51:08.907773+0000","last_change":"2024-03-11T21:51:08.907831+0000","last_active":"2024-03-11T21:51:08.907773+0000","last_peered":"2024-03-11T21:51:08.907773+0000","last_clean":"2024-03-11T21:51:08.907773+0000","last_became_active":"2024-03-11T21:25:40.547082+0000","last_became_peered":"2024-03-11T21:25:40.547082+0000","last_unstale":"2024-03-11T21:51:08.907773+0000","last_undegraded":"2024-03-11T21:51:08.907773+0000","last_fullsized":"2024-03-11T21:51:08.907773+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"456'21","last_scrub_stamp":"2024-03-11T21:51:08.907678+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:39.525554+0000","last_clean_scrub_stamp":"2024-03-11T21:51:08.907678+0000","objects_scrubbed":4,"log_size":21,"log_dups_size":0,"ondisk_log_size":21,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:52:30.134024+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":35,"num_read_kb":29,"num_write":22,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":61,"reported_epoch":554,"state":"active+clean","last_fresh":"2024-03-11T21:51:06.723097+0000","last_change":"2024-03-11T21:49:48.995871+0000","last_active":"2024-03-11T21:51:06.723097+0000","last_peered":"2024-03-11T21:51:06.723097+0000","last_clean":"2024-03-11T21:51:06.723097+0000","last_became_active":"2024-03-11T21:49:26.647985+0000","last_became_peered":"2024-03-11T21:49:26.647985+0000","last_unstale":"2024-03-11T21:51:06.723097+0000","last_undegraded":"2024-03-11T21:51:06.723097+0000","last_fullsized":"2024-03-11T21:51:06.723097+0000","mapping_epoch":532,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":533,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:49:48.995734+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:27:09.801138+0000","last_clean_scrub_stamp":"2024-03-11T21:49:48.995734+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":159,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,5],"acting":[4,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.14","version":"537'15","reported_seq":105,"reported_epoch":554,"state":"active+clean","last_fresh":"2024-03-11T21:51:07.586538+0000","last_change":"2024-03-11T21:50:59.847065+0000","last_active":"2024-03-11T21:51:07.586538+0000","last_peered":"2024-03-11T21:51:07.586538+0000","last_clean":"2024-03-11T21:51:07.586538+0000","last_became_active":"2024-03-11T21:49:17.151384+0000","last_became_peered":"2024-03-11T21:49:17.151384+0000","last_unstale":"2024-03-11T21:51:07.586538+0000","last_undegraded":"2024-03-11T21:51:07.586538+0000","last_fullsized":"2024-03-11T21:51:07.586538+0000","mapping_epoch":528,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":529,"parent":"0.0","parent_split_bits":0,"last_scrub":"537'15","last_scrub_stamp":"2024-03-11T21:50:59.846905+0000","last_deep_scrub":"537'15","last_deep_scrub_stamp":"2024-03-11T21:49:45.799879+0000","last_clean_scrub_stamp":"2024-03-11T21:50:59.846905+0000","objects_scrubbed":1,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:52:10.761086+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,2],"acting":[3,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.13","version":"0'0","reported_seq":1112,"reported_epoch":554,"state":"active+clean","last_fresh":"2024-03-11T21:51:06.721148+0000","last_change":"2024-03-11T21:50:55.345249+0000","last_active":"2024-03-11T21:51:06.721148+0000","last_peered":"2024-03-11T21:51:06.721148+0000","last_clean":"2024-03-11T21:51:06.721148+0000","last_became_active":"2024-03-11T21:49:26.643686+0000","last_became_peered":"2024-03-11T21:49:26.643686+0000","last_unstale":"2024-03-11T21:51:06.721148+0000","last_undegraded":"2024-03-11T21:51:06.721148+0000","last_fullsized":"2024-03-11T21:51:06.721148+0000","mapping_epoch":532,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":533,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:50:55.345155+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:37.512018+0000","last_clean_scrub_stamp":"2024-03-11T21:50:55.345155+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:52:13.680891+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.15","version":"456'25","reported_seq":172,"reported_epoch":554,"state":"active+clean","last_fresh":"2024-03-11T21:51:07.482175+0000","last_change":"2024-03-11T21:49:48.905906+0000","last_active":"2024-03-11T21:51:07.482175+0000","last_peered":"2024-03-11T21:51:07.482175+0000","last_clean":"2024-03-11T21:51:07.482175+0000","last_became_active":"2024-03-11T21:49:25.629022+0000","last_became_peered":"2024-03-11T21:49:25.629022+0000","last_unstale":"2024-03-11T21:51:07.482175+0000","last_undegraded":"2024-03-11T21:51:07.482175+0000","last_fullsized":"2024-03-11T21:51:07.482175+0000","mapping_epoch":531,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":532,"parent":"0.0","parent_split_bits":0,"last_scrub":"456'25","last_scrub_stamp":"2024-03-11T21:49:48.905801+0000","last_deep_scrub":"94'13","last_deep_scrub_stamp":"2024-03-11T21:29:41.853158+0000","last_clean_scrub_stamp":"2024-03-11T21:49:48.905801+0000","objects_scrubbed":4,"log_size":25,"log_dups_size":0,"ondisk_log_size":25,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:51:18.172354+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":34,"num_read_kb":28,"num_write":22,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":27,"num_bytes_recovered":660,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,2],"acting":[5,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.16","version":"0'0","reported_seq":1114,"reported_epoch":554,"state":"active+clean","last_fresh":"2024-03-11T21:51:06.721598+0000","last_change":"2024-03-11T21:50:50.407683+0000","last_active":"2024-03-11T21:51:06.721598+0000","last_peered":"2024-03-11T21:51:06.721598+0000","last_clean":"2024-03-11T21:51:06.721598+0000","last_became_active":"2024-03-11T21:49:17.147073+0000","last_became_peered":"2024-03-11T21:49:17.147073+0000","last_unstale":"2024-03-11T21:51:06.721598+0000","last_undegraded":"2024-03-11T21:51:06.721598+0000","last_fullsized":"2024-03-11T21:51:06.721598+0000","mapping_epoch":528,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":529,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:50:50.407607+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:50:50.407607+0000","last_clean_scrub_stamp":"2024-03-11T21:50:50.407607+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:52:19.224933+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.15","version":"503'12","reported_seq":167,"reported_epoch":554,"state":"active+clean","last_fresh":"2024-03-11T21:51:07.481964+0000","last_change":"2024-03-11T21:50:27.013987+0000","last_active":"2024-03-11T21:51:07.481964+0000","last_peered":"2024-03-11T21:51:07.481964+0000","last_clean":"2024-03-11T21:51:07.481964+0000","last_became_active":"2024-03-11T21:48:28.646407+0000","last_became_peered":"2024-03-11T21:48:28.646407+0000","last_unstale":"2024-03-11T21:51:07.481964+0000","last_undegraded":"2024-03-11T21:51:07.481964+0000","last_fullsized":"2024-03-11T21:51:07.481964+0000","mapping_epoch":502,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":503,"parent":"0.0","parent_split_bits":0,"last_scrub":"503'12","last_scrub_stamp":"2024-03-11T21:50:27.013892+0000","last_deep_scrub":"98'9","last_deep_scrub_stamp":"2024-03-11T21:31:20.624923+0000","last_clean_scrub_stamp":"2024-03-11T21:50:27.013892+0000","objects_scrubbed":1,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:51:50.165683+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.12","version":"0'0","reported_seq":144,"reported_epoch":554,"state":"active+clean","last_fresh":"2024-03-11T21:51:08.761997+0000","last_change":"2024-03-11T21:51:08.762020+0000","last_active":"2024-03-11T21:51:08.761997+0000","last_peered":"2024-03-11T21:51:08.761997+0000","last_clean":"2024-03-11T21:51:08.761997+0000","last_became_active":"2024-03-11T21:49:17.146154+0000","last_became_peered":"2024-03-11T21:49:17.146154+0000","last_unstale":"2024-03-11T21:51:08.761997+0000","last_undegraded":"2024-03-11T21:51:08.761997+0000","last_fullsized":"2024-03-11T21:51:08.761997+0000","mapping_epoch":528,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":529,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:51:08.761930+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:50:01.868835+0000","last_clean_scrub_stamp":"2024-03-11T21:51:08.761930+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:52:31.245167+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.14","version":"534'35","reported_seq":94,"reported_epoch":554,"state":"active+clean","last_fresh":"2024-03-11T21:51:07.586548+0000","last_change":"2024-03-11T21:50:56.839722+0000","last_active":"2024-03-11T21:51:07.586548+0000","last_peered":"2024-03-11T21:51:07.586548+0000","last_clean":"2024-03-11T21:51:07.586548+0000","last_became_active":"2024-03-11T21:49:17.151223+0000","last_became_peered":"2024-03-11T21:49:17.151223+0000","last_unstale":"2024-03-11T21:51:07.586548+0000","last_undegraded":"2024-03-11T21:51:07.586548+0000","last_fullsized":"2024-03-11T21:51:07.586548+0000","mapping_epoch":528,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":529,"parent":"0.0","parent_split_bits":0,"last_scrub":"534'35","last_scrub_stamp":"2024-03-11T21:50:56.839627+0000","last_deep_scrub":"534'35","last_deep_scrub_stamp":"2024-03-11T21:49:36.847051+0000","last_clean_scrub_stamp":"2024-03-11T21:50:56.839627+0000","objects_scrubbed":5,"log_size":35,"log_dups_size":0,"ondisk_log_size":35,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:52:20.633021+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":44,"num_read_kb":39,"num_write":30,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.15","version":"0'0","reported_seq":734,"reported_epoch":554,"state":"active+clean","last_fresh":"2024-03-11T21:51:07.587655+0000","last_change":"2024-03-11T21:50:15.289444+0000","last_active":"2024-03-11T21:51:07.587655+0000","last_peered":"2024-03-11T21:51:07.587655+0000","last_clean":"2024-03-11T21:51:07.587655+0000","last_became_active":"2024-03-11T21:49:26.644817+0000","last_became_peered":"2024-03-11T21:49:26.644817+0000","last_unstale":"2024-03-11T21:51:07.587655+0000","last_undegraded":"2024-03-11T21:51:07.587655+0000","last_fullsized":"2024-03-11T21:51:07.587655+0000","mapping_epoch":532,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":533,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:50:15.289313+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:43.559452+0000","last_clean_scrub_stamp":"2024-03-11T21:50:15.289313+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:51:29.118124+0000","scrub_duration":95,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":136,"reported_epoch":554,"state":"active+clean","last_fresh":"2024-03-11T21:51:07.481972+0000","last_change":"2024-03-11T21:50:15.931089+0000","last_active":"2024-03-11T21:51:07.481972+0000","last_peered":"2024-03-11T21:51:07.481972+0000","last_clean":"2024-03-11T21:51:07.481972+0000","last_became_active":"2024-03-11T21:49:17.147086+0000","last_became_peered":"2024-03-11T21:49:17.147086+0000","last_unstale":"2024-03-11T21:51:07.481972+0000","last_undegraded":"2024-03-11T21:51:07.481972+0000","last_fullsized":"2024-03-11T21:51:07.481972+0000","mapping_epoch":528,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":529,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:50:15.930961+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:43:36.585632+0000","last_clean_scrub_stamp":"2024-03-11T21:50:15.930961+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:51:34.383497+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,1],"acting":[5,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.11","version":"0'0","reported_seq":1107,"reported_epoch":554,"state":"active+clean","last_fresh":"2024-03-11T21:51:07.586551+0000","last_change":"2024-03-11T21:50:54.080401+0000","last_active":"2024-03-11T21:51:07.586551+0000","last_peered":"2024-03-11T21:51:07.586551+0000","last_clean":"2024-03-11T21:51:07.586551+0000","last_became_active":"2024-03-11T21:49:26.637633+0000","last_became_peered":"2024-03-11T21:49:26.637633+0000","last_unstale":"2024-03-11T21:51:07.586551+0000","last_undegraded":"2024-03-11T21:51:07.586551+0000","last_fullsized":"2024-03-11T21:51:07.586551+0000","mapping_epoch":532,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":533,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:50:54.080300+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:37.512018+0000","last_clean_scrub_stamp":"2024-03-11T21:50:54.080300+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:52:19.672449+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.17","version":"456'21","reported_seq":77,"reported_epoch":554,"state":"active+clean","last_fresh":"2024-03-11T21:51:07.586907+0000","last_change":"2024-03-11T21:49:58.581189+0000","last_active":"2024-03-11T21:51:07.586907+0000","last_peered":"2024-03-11T21:51:07.586907+0000","last_clean":"2024-03-11T21:51:07.586907+0000","last_became_active":"2024-03-11T21:49:17.141874+0000","last_became_peered":"2024-03-11T21:49:17.141874+0000","last_unstale":"2024-03-11T21:51:07.586907+0000","last_undegraded":"2024-03-11T21:51:07.586907+0000","last_fullsized":"2024-03-11T21:51:07.586907+0000","mapping_epoch":528,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":529,"parent":"0.0","parent_split_bits":0,"last_scrub":"456'21","last_scrub_stamp":"2024-03-11T21:49:58.581084+0000","last_deep_scrub":"97'9","last_deep_scrub_stamp":"2024-03-11T21:27:25.803132+0000","last_clean_scrub_stamp":"2024-03-11T21:49:58.581084+0000","objects_scrubbed":3,"log_size":21,"log_dups_size":0,"ondisk_log_size":21,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:51:21.644085+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":27,"num_read_kb":24,"num_write":18,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.14","version":"0'0","reported_seq":1085,"reported_epoch":554,"state":"active+clean","last_fresh":"2024-03-11T21:51:06.721211+0000","last_change":"2024-03-11T21:50:10.894796+0000","last_active":"2024-03-11T21:51:06.721211+0000","last_peered":"2024-03-11T21:51:06.721211+0000","last_clean":"2024-03-11T21:51:06.721211+0000","last_became_active":"2024-03-11T21:49:26.637667+0000","last_became_peered":"2024-03-11T21:49:26.637667+0000","last_unstale":"2024-03-11T21:51:06.721211+0000","last_undegraded":"2024-03-11T21:51:06.721211+0000","last_fullsized":"2024-03-11T21:51:06.721211+0000","mapping_epoch":532,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":533,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:50:10.894715+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:43.559452+0000","last_clean_scrub_stamp":"2024-03-11T21:50:10.894715+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:51:11.919095+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,7],"acting":[2,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":69,"reported_epoch":554,"state":"active+clean","last_fresh":"2024-03-11T21:51:07.587020+0000","last_change":"2024-03-11T21:50:06.554164+0000","last_active":"2024-03-11T21:51:07.587020+0000","last_peered":"2024-03-11T21:51:07.587020+0000","last_clean":"2024-03-11T21:51:07.587020+0000","last_became_active":"2024-03-11T21:49:17.143360+0000","last_became_peered":"2024-03-11T21:49:17.143360+0000","last_unstale":"2024-03-11T21:51:07.587020+0000","last_undegraded":"2024-03-11T21:51:07.587020+0000","last_fullsized":"2024-03-11T21:51:07.587020+0000","mapping_epoch":528,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":529,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:50:06.554083+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:50:06.554083+0000","last_clean_scrub_stamp":"2024-03-11T21:50:06.554083+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:51:20.135608+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.10","version":"0'0","reported_seq":1081,"reported_epoch":554,"state":"active+clean","last_fresh":"2024-03-11T21:51:06.720450+0000","last_change":"2024-03-11T21:51:05.983502+0000","last_active":"2024-03-11T21:51:06.720450+0000","last_peered":"2024-03-11T21:51:06.720450+0000","last_clean":"2024-03-11T21:51:06.720450+0000","last_became_active":"2024-03-11T21:25:38.533138+0000","last_became_peered":"2024-03-11T21:25:38.533138+0000","last_unstale":"2024-03-11T21:51:06.720450+0000","last_undegraded":"2024-03-11T21:51:06.720450+0000","last_fullsized":"2024-03-11T21:51:06.720450+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:51:05.983372+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:37.512018+0000","last_clean_scrub_stamp":"2024-03-11T21:51:05.983372+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:52:19.603119+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,0],"acting":[2,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.16","version":"456'25","reported_seq":1183,"reported_epoch":554,"state":"active+clean","last_fresh":"2024-03-11T21:51:06.721793+0000","last_change":"2024-03-11T21:50:38.397638+0000","last_active":"2024-03-11T21:51:06.721793+0000","last_peered":"2024-03-11T21:51:06.721793+0000","last_clean":"2024-03-11T21:51:06.721793+0000","last_became_active":"2024-03-11T21:49:17.147316+0000","last_became_peered":"2024-03-11T21:49:17.147316+0000","last_unstale":"2024-03-11T21:51:06.721793+0000","last_undegraded":"2024-03-11T21:51:06.721793+0000","last_fullsized":"2024-03-11T21:51:06.721793+0000","mapping_epoch":528,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":529,"parent":"0.0","parent_split_bits":0,"last_scrub":"456'25","last_scrub_stamp":"2024-03-11T21:50:38.397443+0000","last_deep_scrub":"369'19","last_deep_scrub_stamp":"2024-03-11T21:41:22.115634+0000","last_clean_scrub_stamp":"2024-03-11T21:50:38.397443+0000","objects_scrubbed":4,"log_size":25,"log_dups_size":0,"ondisk_log_size":25,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:52:00.906018+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":35,"num_read_kb":29,"num_write":22,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19,"num_bytes_recovered":440,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.b","version":"0'0","reported_seq":69,"reported_epoch":554,"state":"active+clean","last_fresh":"2024-03-11T21:51:07.586360+0000","last_change":"2024-03-11T21:49:43.923280+0000","last_active":"2024-03-11T21:51:07.586360+0000","last_peered":"2024-03-11T21:51:07.586360+0000","last_clean":"2024-03-11T21:51:07.586360+0000","last_became_active":"2024-03-11T21:49:17.146927+0000","last_became_peered":"2024-03-11T21:49:17.146927+0000","last_unstale":"2024-03-11T21:51:07.586360+0000","last_undegraded":"2024-03-11T21:51:07.586360+0000","last_fullsized":"2024-03-11T21:51:07.586360+0000","mapping_epoch":528,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":529,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:49:43.923140+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:44:35.622488+0000","last_clean_scrub_stamp":"2024-03-11T21:49:43.923140+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:51:12.639907+0000","scrub_duration":119,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.f","version":"67'2","reported_seq":67,"reported_epoch":554,"state":"active+clean","last_fresh":"2024-03-11T21:51:06.723121+0000","last_change":"2024-03-11T21:49:36.908598+0000","last_active":"2024-03-11T21:51:06.723121+0000","last_peered":"2024-03-11T21:51:06.723121+0000","last_clean":"2024-03-11T21:51:06.723121+0000","last_became_active":"2024-03-11T21:49:26.656928+0000","last_became_peered":"2024-03-11T21:49:26.656928+0000","last_unstale":"2024-03-11T21:51:06.723121+0000","last_undegraded":"2024-03-11T21:51:06.723121+0000","last_fullsized":"2024-03-11T21:51:06.723121+0000","mapping_epoch":532,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":533,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'2","last_scrub_stamp":"2024-03-11T21:49:36.908500+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:37.512018+0000","last_clean_scrub_stamp":"2024-03-11T21:49:36.908500+0000","objects_scrubbed":2,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":92,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":368,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,7],"acting":[4,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.9","version":"456'29","reported_seq":73,"reported_epoch":554,"state":"active+clean","last_fresh":"2024-03-11T21:51:06.722137+0000","last_change":"2024-03-11T21:50:24.143508+0000","last_active":"2024-03-11T21:51:06.722137+0000","last_peered":"2024-03-11T21:51:06.722137+0000","last_clean":"2024-03-11T21:51:06.722137+0000","last_became_active":"2024-03-11T21:49:26.657961+0000","last_became_peered":"2024-03-11T21:49:26.657961+0000","last_unstale":"2024-03-11T21:51:06.722137+0000","last_undegraded":"2024-03-11T21:51:06.722137+0000","last_fullsized":"2024-03-11T21:51:06.722137+0000","mapping_epoch":532,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":533,"parent":"0.0","parent_split_bits":0,"last_scrub":"456'29","last_scrub_stamp":"2024-03-11T21:50:24.143389+0000","last_deep_scrub":"456'29","last_deep_scrub_stamp":"2024-03-11T21:46:28.006437+0000","last_clean_scrub_stamp":"2024-03-11T21:50:24.143389+0000","objects_scrubbed":5,"log_size":29,"log_dups_size":0,"ondisk_log_size":29,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:51:32.175677+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":42,"num_read_kb":33,"num_write":26,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":660,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,3],"acting":[4,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":1066,"reported_epoch":554,"state":"active+clean","last_fresh":"2024-03-11T21:51:06.720541+0000","last_change":"2024-03-11T21:50:06.892558+0000","last_active":"2024-03-11T21:51:06.720541+0000","last_peered":"2024-03-11T21:51:06.720541+0000","last_clean":"2024-03-11T21:51:06.720541+0000","last_became_active":"2024-03-11T21:25:42.573987+0000","last_became_peered":"2024-03-11T21:25:42.573987+0000","last_unstale":"2024-03-11T21:51:06.720541+0000","last_undegraded":"2024-03-11T21:51:06.720541+0000","last_fullsized":"2024-03-11T21:51:06.720541+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:50:06.892477+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:41.544550+0000","last_clean_scrub_stamp":"2024-03-11T21:50:06.892477+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:51:35.689863+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.0","version":"0'0","reported_seq":1090,"reported_epoch":554,"state":"active+clean","last_fresh":"2024-03-11T21:51:06.720507+0000","last_change":"2024-03-11T21:50:54.324503+0000","last_active":"2024-03-11T21:51:06.720507+0000","last_peered":"2024-03-11T21:51:06.720507+0000","last_clean":"2024-03-11T21:51:06.720507+0000","last_became_active":"2024-03-11T21:49:17.146239+0000","last_became_peered":"2024-03-11T21:49:17.146239+0000","last_unstale":"2024-03-11T21:51:06.720507+0000","last_undegraded":"2024-03-11T21:51:06.720507+0000","last_fullsized":"2024-03-11T21:51:06.720507+0000","mapping_epoch":528,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":529,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:50:54.324421+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:29:39.498784+0000","last_clean_scrub_stamp":"2024-03-11T21:50:54.324421+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:51:54.677449+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":960,"reported_epoch":554,"state":"active+clean","last_fresh":"2024-03-11T21:51:06.961130+0000","last_change":"2024-03-11T21:50:21.949955+0000","last_active":"2024-03-11T21:51:06.961130+0000","last_peered":"2024-03-11T21:51:06.961130+0000","last_clean":"2024-03-11T21:51:06.961130+0000","last_became_active":"2024-03-11T21:49:46.949173+0000","last_became_peered":"2024-03-11T21:49:46.949173+0000","last_unstale":"2024-03-11T21:51:06.961130+0000","last_undegraded":"2024-03-11T21:51:06.961130+0000","last_fullsized":"2024-03-11T21:51:06.961130+0000","mapping_epoch":537,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":538,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:50:21.949868+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:48:58.177316+0000","last_clean_scrub_stamp":"2024-03-11T21:50:21.949868+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:51:22.766616+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,7],"acting":[1,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.2","version":"456'35","reported_seq":1195,"reported_epoch":554,"state":"active+clean","last_fresh":"2024-03-11T21:51:06.961672+0000","last_change":"2024-03-11T21:50:16.982670+0000","last_active":"2024-03-11T21:51:06.961672+0000","last_peered":"2024-03-11T21:51:06.961672+0000","last_clean":"2024-03-11T21:51:06.961672+0000","last_became_active":"2024-03-11T21:49:26.644586+0000","last_became_peered":"2024-03-11T21:49:26.644586+0000","last_unstale":"2024-03-11T21:51:06.961672+0000","last_undegraded":"2024-03-11T21:51:06.961672+0000","last_fullsized":"2024-03-11T21:51:06.961672+0000","mapping_epoch":532,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":533,"parent":"0.0","parent_split_bits":0,"last_scrub":"456'35","last_scrub_stamp":"2024-03-11T21:50:16.982580+0000","last_deep_scrub":"94'15","last_deep_scrub_stamp":"2024-03-11T21:26:49.993060+0000","last_clean_scrub_stamp":"2024-03-11T21:50:16.982580+0000","objects_scrubbed":5,"log_size":35,"log_dups_size":0,"ondisk_log_size":35,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:51:23.909581+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":45,"num_read_kb":40,"num_write":30,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.3","version":"503'10","reported_seq":1396,"reported_epoch":554,"state":"active+clean","last_fresh":"2024-03-11T21:51:06.721357+0000","last_change":"2024-03-11T21:50:24.313886+0000","last_active":"2024-03-11T21:51:06.721357+0000","last_peered":"2024-03-11T21:51:06.721357+0000","last_clean":"2024-03-11T21:51:06.721357+0000","last_became_active":"2024-03-11T21:48:28.641906+0000","last_became_peered":"2024-03-11T21:48:28.641906+0000","last_unstale":"2024-03-11T21:51:06.721357+0000","last_undegraded":"2024-03-11T21:51:06.721357+0000","last_fullsized":"2024-03-11T21:51:06.721357+0000","mapping_epoch":502,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":503,"parent":"0.0","parent_split_bits":0,"last_scrub":"503'10","last_scrub_stamp":"2024-03-11T21:50:24.313803+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:41.544550+0000","last_clean_scrub_stamp":"2024-03-11T21:50:24.313803+0000","objects_scrubbed":1,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:51:26.111087+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,5],"acting":[0,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":75,"reported_epoch":554,"state":"active+clean","last_fresh":"2024-03-11T21:51:07.586906+0000","last_change":"2024-03-11T21:50:14.457423+0000","last_active":"2024-03-11T21:51:07.586906+0000","last_peered":"2024-03-11T21:51:07.586906+0000","last_clean":"2024-03-11T21:51:07.586906+0000","last_became_active":"2024-03-11T21:49:26.638739+0000","last_became_peered":"2024-03-11T21:49:26.638739+0000","last_unstale":"2024-03-11T21:51:07.586906+0000","last_undegraded":"2024-03-11T21:51:07.586906+0000","last_fullsized":"2024-03-11T21:51:07.586906+0000","mapping_epoch":532,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":533,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:50:14.457308+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:50:14.457308+0000","last_clean_scrub_stamp":"2024-03-11T21:50:14.457308+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:51:17.178186+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,7],"acting":[3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":130,"reported_epoch":554,"state":"active+clean","last_fresh":"2024-03-11T21:51:07.481792+0000","last_change":"2024-03-11T21:51:04.701911+0000","last_active":"2024-03-11T21:51:07.481792+0000","last_peered":"2024-03-11T21:51:07.481792+0000","last_clean":"2024-03-11T21:51:07.481792+0000","last_became_active":"2024-03-11T21:48:28.640205+0000","last_became_peered":"2024-03-11T21:48:28.640205+0000","last_unstale":"2024-03-11T21:51:07.481792+0000","last_undegraded":"2024-03-11T21:51:07.481792+0000","last_fullsized":"2024-03-11T21:51:07.481792+0000","mapping_epoch":502,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":503,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:51:04.701778+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:51:04.701778+0000","last_clean_scrub_stamp":"2024-03-11T21:51:04.701778+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:52:26.493598+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.4","version":"553'68","reported_seq":1364,"reported_epoch":554,"state":"active+clean","last_fresh":"2024-03-11T21:51:06.961664+0000","last_change":"2024-03-11T21:50:59.004099+0000","last_active":"2024-03-11T21:51:06.961664+0000","last_peered":"2024-03-11T21:51:06.961664+0000","last_clean":"2024-03-11T21:51:06.961664+0000","last_became_active":"2024-03-11T21:49:46.258448+0000","last_became_peered":"2024-03-11T21:49:46.258448+0000","last_unstale":"2024-03-11T21:51:06.961664+0000","last_undegraded":"2024-03-11T21:51:06.961664+0000","last_fullsized":"2024-03-11T21:51:06.961664+0000","mapping_epoch":537,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":538,"parent":"0.0","parent_split_bits":0,"last_scrub":"553'68","last_scrub_stamp":"2024-03-11T21:50:59.003898+0000","last_deep_scrub":"553'68","last_deep_scrub_stamp":"2024-03-11T21:50:59.003898+0000","last_clean_scrub_stamp":"2024-03-11T21:50:59.003898+0000","objects_scrubbed":10,"log_size":68,"log_dups_size":0,"ondisk_log_size":68,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:52:26.135652+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":358,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":84,"num_read_kb":69,"num_write":50,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":358,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3],"acting":[1,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":1073,"reported_epoch":554,"state":"active+clean","last_fresh":"2024-03-11T21:51:06.721730+0000","last_change":"2024-03-11T21:50:33.294601+0000","last_active":"2024-03-11T21:51:06.721730+0000","last_peered":"2024-03-11T21:51:06.721730+0000","last_clean":"2024-03-11T21:51:06.721730+0000","last_became_active":"2024-03-11T21:49:26.643886+0000","last_became_peered":"2024-03-11T21:49:26.643886+0000","last_unstale":"2024-03-11T21:51:06.721730+0000","last_undegraded":"2024-03-11T21:51:06.721730+0000","last_fullsized":"2024-03-11T21:51:06.721730+0000","mapping_epoch":532,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":533,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:50:33.294521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:41.544550+0000","last_clean_scrub_stamp":"2024-03-11T21:50:33.294521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:51:55.612537+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.5","version":"0'0","reported_seq":732,"reported_epoch":554,"state":"active+clean","last_fresh":"2024-03-11T21:51:07.587510+0000","last_change":"2024-03-11T21:50:13.315051+0000","last_active":"2024-03-11T21:51:07.587510+0000","last_peered":"2024-03-11T21:51:07.587510+0000","last_clean":"2024-03-11T21:51:07.587510+0000","last_became_active":"2024-03-11T21:49:17.149583+0000","last_became_peered":"2024-03-11T21:49:17.149583+0000","last_unstale":"2024-03-11T21:51:07.587510+0000","last_undegraded":"2024-03-11T21:51:07.587510+0000","last_fullsized":"2024-03-11T21:51:07.587510+0000","mapping_epoch":528,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":529,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:50:13.314944+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:43.559452+0000","last_clean_scrub_stamp":"2024-03-11T21:50:13.314944+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:51:29.726449+0000","scrub_duration":96,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,3],"acting":[7,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.0","version":"164'6","reported_seq":2334,"reported_epoch":554,"state":"active+clean","last_fresh":"2024-03-11T21:51:07.400971+0000","last_change":"2024-03-11T21:50:03.940744+0000","last_active":"2024-03-11T21:51:07.400971+0000","last_peered":"2024-03-11T21:51:07.400971+0000","last_clean":"2024-03-11T21:51:07.400971+0000","last_became_active":"2024-03-11T21:49:46.257628+0000","last_became_peered":"2024-03-11T21:49:46.257628+0000","last_unstale":"2024-03-11T21:51:07.400971+0000","last_undegraded":"2024-03-11T21:51:07.400971+0000","last_fullsized":"2024-03-11T21:51:07.400971+0000","mapping_epoch":537,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":538,"parent":"0.0","parent_split_bits":0,"last_scrub":"164'6","last_scrub_stamp":"2024-03-11T21:50:03.940628+0000","last_deep_scrub":"164'6","last_deep_scrub_stamp":"2024-03-11T21:32:17.597666+0000","last_clean_scrub_stamp":"2024-03-11T21:50:03.940628+0000","objects_scrubbed":1,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.000753559,"stat_sum":{"num_bytes":389,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1370,"num_read_kb":1365,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,6],"acting":[1,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":1104,"reported_epoch":554,"state":"active+clean","last_fresh":"2024-03-11T21:51:09.976429+0000","last_change":"2024-03-11T21:51:09.976448+0000","last_active":"2024-03-11T21:51:09.976429+0000","last_peered":"2024-03-11T21:51:09.976429+0000","last_clean":"2024-03-11T21:51:09.976429+0000","last_became_active":"2024-03-11T21:49:17.138476+0000","last_became_peered":"2024-03-11T21:49:17.138476+0000","last_unstale":"2024-03-11T21:51:09.976429+0000","last_undegraded":"2024-03-11T21:51:09.976429+0000","last_fullsized":"2024-03-11T21:51:09.976429+0000","mapping_epoch":528,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":529,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:51:09.976370+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:43:28.183315+0000","last_clean_scrub_stamp":"2024-03-11T21:51:09.976370+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:52:11.007881+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.7","version":"456'26","reported_seq":1147,"reported_epoch":554,"state":"active+clean","last_fresh":"2024-03-11T21:51:06.961914+0000","last_change":"2024-03-11T21:50:02.957549+0000","last_active":"2024-03-11T21:51:06.961914+0000","last_peered":"2024-03-11T21:51:06.961914+0000","last_clean":"2024-03-11T21:51:06.961914+0000","last_became_active":"2024-03-11T21:48:28.632227+0000","last_became_peered":"2024-03-11T21:48:28.632227+0000","last_unstale":"2024-03-11T21:51:06.961914+0000","last_undegraded":"2024-03-11T21:51:06.961914+0000","last_fullsized":"2024-03-11T21:51:06.961914+0000","mapping_epoch":502,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":503,"parent":"0.0","parent_split_bits":0,"last_scrub":"456'26","last_scrub_stamp":"2024-03-11T21:50:02.957446+0000","last_deep_scrub":"260'22","last_deep_scrub_stamp":"2024-03-11T21:41:27.232403+0000","last_clean_scrub_stamp":"2024-03-11T21:50:02.957446+0000","objects_scrubbed":5,"log_size":26,"log_dups_size":0,"ondisk_log_size":26,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:51:16.879974+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":42,"num_read_kb":31,"num_write":24,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":660,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":1090,"reported_epoch":554,"state":"active+clean","last_fresh":"2024-03-11T21:51:06.722329+0000","last_change":"2024-03-11T21:50:14.768174+0000","last_active":"2024-03-11T21:51:06.722329+0000","last_peered":"2024-03-11T21:51:06.722329+0000","last_clean":"2024-03-11T21:51:06.722329+0000","last_became_active":"2024-03-11T21:48:28.636457+0000","last_became_peered":"2024-03-11T21:48:28.636457+0000","last_unstale":"2024-03-11T21:51:06.722329+0000","last_undegraded":"2024-03-11T21:51:06.722329+0000","last_fullsized":"2024-03-11T21:51:06.722329+0000","mapping_epoch":502,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":503,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:50:14.768096+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:41.544550+0000","last_clean_scrub_stamp":"2024-03-11T21:50:14.768096+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:51:19.117559+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7],"acting":[2,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.7","version":"124'2","reported_seq":154,"reported_epoch":554,"state":"active+clean","last_fresh":"2024-03-11T21:51:07.482379+0000","last_change":"2024-03-11T21:50:55.867991+0000","last_active":"2024-03-11T21:51:07.482379+0000","last_peered":"2024-03-11T21:51:07.482379+0000","last_clean":"2024-03-11T21:51:07.482379+0000","last_became_active":"2024-03-11T21:49:26.651720+0000","last_became_peered":"2024-03-11T21:49:26.651720+0000","last_unstale":"2024-03-11T21:51:07.482379+0000","last_undegraded":"2024-03-11T21:51:07.482379+0000","last_fullsized":"2024-03-11T21:51:07.482379+0000","mapping_epoch":532,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":533,"parent":"0.0","parent_split_bits":0,"last_scrub":"124'2","last_scrub_stamp":"2024-03-11T21:50:55.867884+0000","last_deep_scrub":"124'2","last_deep_scrub_stamp":"2024-03-11T21:29:34.956908+0000","last_clean_scrub_stamp":"2024-03-11T21:50:55.867884+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:52:01.413473+0000","scrub_duration":129,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":52,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,4],"acting":[5,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2","version":"71'2","reported_seq":73,"reported_epoch":554,"state":"active+clean","last_fresh":"2024-03-11T21:51:07.586272+0000","last_change":"2024-03-11T21:49:48.708764+0000","last_active":"2024-03-11T21:51:07.586272+0000","last_peered":"2024-03-11T21:51:07.586272+0000","last_clean":"2024-03-11T21:51:07.586272+0000","last_became_active":"2024-03-11T21:49:17.136844+0000","last_became_peered":"2024-03-11T21:49:17.136844+0000","last_unstale":"2024-03-11T21:51:07.586272+0000","last_undegraded":"2024-03-11T21:51:07.586272+0000","last_fullsized":"2024-03-11T21:51:07.586272+0000","mapping_epoch":528,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":529,"parent":"0.0","parent_split_bits":0,"last_scrub":"71'2","last_scrub_stamp":"2024-03-11T21:49:48.708634+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:37.512018+0000","last_clean_scrub_stamp":"2024-03-11T21:49:48.708634+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:51:09.002959+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.0014206449999999999,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":38,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":130,"reported_epoch":554,"state":"active+clean","last_fresh":"2024-03-11T21:51:07.481765+0000","last_change":"2024-03-11T21:50:58.641704+0000","last_active":"2024-03-11T21:51:07.481765+0000","last_peered":"2024-03-11T21:51:07.481765+0000","last_clean":"2024-03-11T21:51:07.481765+0000","last_became_active":"2024-03-11T21:48:28.636579+0000","last_became_peered":"2024-03-11T21:48:28.636579+0000","last_unstale":"2024-03-11T21:51:07.481765+0000","last_undegraded":"2024-03-11T21:51:07.481765+0000","last_fullsized":"2024-03-11T21:51:07.481765+0000","mapping_epoch":502,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":503,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:50:58.641573+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:37.512018+0000","last_clean_scrub_stamp":"2024-03-11T21:50:58.641573+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:52:23.519875+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"87'128","reported_seq":1183,"reported_epoch":554,"state":"active+clean","last_fresh":"2024-03-11T21:51:07.586330+0000","last_change":"2024-03-11T21:49:46.978593+0000","last_active":"2024-03-11T21:51:07.586330+0000","last_peered":"2024-03-11T21:51:07.586330+0000","last_clean":"2024-03-11T21:51:07.586330+0000","last_became_active":"2024-03-11T21:34:44.164569+0000","last_became_peered":"2024-03-11T21:34:44.164569+0000","last_unstale":"2024-03-11T21:51:07.586330+0000","last_undegraded":"2024-03-11T21:51:07.586330+0000","last_fullsized":"2024-03-11T21:51:07.586330+0000","mapping_epoch":229,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":230,"parent":"0.0","parent_split_bits":0,"last_scrub":"87'128","last_scrub_stamp":"2024-03-11T21:49:46.978479+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:24.261974+0000","last_clean_scrub_stamp":"2024-03-11T21:49:46.978479+0000","objects_scrubbed":2,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:51:11.989712+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":198,"num_read_kb":294,"num_write":179,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":2951840,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.5","version":"533'50","reported_seq":1511,"reported_epoch":554,"state":"active+clean","last_fresh":"2024-03-11T21:51:07.587530+0000","last_change":"2024-03-11T21:50:43.124806+0000","last_active":"2024-03-11T21:51:07.587530+0000","last_peered":"2024-03-11T21:51:07.587530+0000","last_clean":"2024-03-11T21:51:07.587530+0000","last_became_active":"2024-03-11T21:49:26.637870+0000","last_became_peered":"2024-03-11T21:49:26.637870+0000","last_unstale":"2024-03-11T21:51:07.587530+0000","last_undegraded":"2024-03-11T21:51:07.587530+0000","last_fullsized":"2024-03-11T21:51:07.587530+0000","mapping_epoch":532,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":533,"parent":"0.0","parent_split_bits":0,"last_scrub":"533'50","last_scrub_stamp":"2024-03-11T21:50:43.124673+0000","last_deep_scrub":"533'50","last_deep_scrub_stamp":"2024-03-11T21:49:32.782974+0000","last_clean_scrub_stamp":"2024-03-11T21:50:43.124673+0000","objects_scrubbed":8,"log_size":50,"log_dups_size":0,"ondisk_log_size":50,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:51:47.509858+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":154,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":61,"num_read_kb":51,"num_write":37,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":308,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":731,"reported_epoch":554,"state":"active+clean","last_fresh":"2024-03-11T21:51:07.587415+0000","last_change":"2024-03-11T21:50:30.303207+0000","last_active":"2024-03-11T21:51:07.587415+0000","last_peered":"2024-03-11T21:51:07.587415+0000","last_clean":"2024-03-11T21:51:07.587415+0000","last_became_active":"2024-03-11T21:48:28.627651+0000","last_became_peered":"2024-03-11T21:48:28.627651+0000","last_unstale":"2024-03-11T21:51:07.587415+0000","last_undegraded":"2024-03-11T21:51:07.587415+0000","last_fullsized":"2024-03-11T21:51:07.587415+0000","mapping_epoch":502,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":503,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:50:30.303099+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:32:21.266672+0000","last_clean_scrub_stamp":"2024-03-11T21:50:30.303099+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:51:48.779449+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,5],"acting":[7,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.4","version":"0'0","reported_seq":1090,"reported_epoch":554,"state":"active+clean","last_fresh":"2024-03-11T21:51:06.961734+0000","last_change":"2024-03-11T21:50:41.927835+0000","last_active":"2024-03-11T21:51:06.961734+0000","last_peered":"2024-03-11T21:51:06.961734+0000","last_clean":"2024-03-11T21:51:06.961734+0000","last_became_active":"2024-03-11T21:49:17.138104+0000","last_became_peered":"2024-03-11T21:49:17.138104+0000","last_unstale":"2024-03-11T21:51:06.961734+0000","last_undegraded":"2024-03-11T21:51:06.961734+0000","last_fullsized":"2024-03-11T21:51:06.961734+0000","mapping_epoch":528,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":529,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:50:41.927752+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:49:18.000732+0000","last_clean_scrub_stamp":"2024-03-11T21:50:41.927752+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:51:55.023362+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":1121,"reported_epoch":554,"state":"active+clean","last_fresh":"2024-03-11T21:51:06.721118+0000","last_change":"2024-03-11T21:50:03.138901+0000","last_active":"2024-03-11T21:51:06.721118+0000","last_peered":"2024-03-11T21:51:06.721118+0000","last_clean":"2024-03-11T21:51:06.721118+0000","last_became_active":"2024-03-11T21:49:26.643353+0000","last_became_peered":"2024-03-11T21:49:26.643353+0000","last_unstale":"2024-03-11T21:51:06.721118+0000","last_undegraded":"2024-03-11T21:51:06.721118+0000","last_fullsized":"2024-03-11T21:51:06.721118+0000","mapping_epoch":532,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":533,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:50:03.138820+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:37.512018+0000","last_clean_scrub_stamp":"2024-03-11T21:50:03.138820+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:51:29.637981+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.00089342499999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":712,"reported_epoch":554,"state":"active+clean","last_fresh":"2024-03-11T21:51:07.586280+0000","last_change":"2024-03-11T21:50:45.460269+0000","last_active":"2024-03-11T21:51:07.586280+0000","last_peered":"2024-03-11T21:51:07.586280+0000","last_clean":"2024-03-11T21:51:07.586280+0000","last_became_active":"2024-03-11T21:34:41.149219+0000","last_became_peered":"2024-03-11T21:34:41.149219+0000","last_unstale":"2024-03-11T21:51:07.586280+0000","last_undegraded":"2024-03-11T21:51:07.586280+0000","last_fullsized":"2024-03-11T21:51:07.586280+0000","mapping_epoch":226,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":227,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:50:45.460178+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:43:30.734024+0000","last_clean_scrub_stamp":"2024-03-11T21:50:45.460178+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:52:00.950582+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,0],"acting":[7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.6","version":"456'42","reported_seq":1172,"reported_epoch":554,"state":"active+clean","last_fresh":"2024-03-11T21:51:06.721697+0000","last_change":"2024-03-11T21:50:59.395853+0000","last_active":"2024-03-11T21:51:06.721697+0000","last_peered":"2024-03-11T21:51:06.721697+0000","last_clean":"2024-03-11T21:51:06.721697+0000","last_became_active":"2024-03-11T21:46:53.263790+0000","last_became_peered":"2024-03-11T21:46:53.263790+0000","last_unstale":"2024-03-11T21:51:06.721697+0000","last_undegraded":"2024-03-11T21:51:06.721697+0000","last_fullsized":"2024-03-11T21:51:06.721697+0000","mapping_epoch":473,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":474,"parent":"0.0","parent_split_bits":0,"last_scrub":"456'42","last_scrub_stamp":"2024-03-11T21:50:59.395753+0000","last_deep_scrub":"456'42","last_deep_scrub_stamp":"2024-03-11T21:49:31.238617+0000","last_clean_scrub_stamp":"2024-03-11T21:50:59.395753+0000","objects_scrubbed":6,"log_size":42,"log_dups_size":0,"ondisk_log_size":42,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:52:23.359650+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":54,"num_read_kb":48,"num_write":36,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":138,"reported_epoch":554,"state":"active+clean","last_fresh":"2024-03-11T21:51:07.482247+0000","last_change":"2024-03-11T21:51:05.710542+0000","last_active":"2024-03-11T21:51:07.482247+0000","last_peered":"2024-03-11T21:51:07.482247+0000","last_clean":"2024-03-11T21:51:07.482247+0000","last_became_active":"2024-03-11T21:48:28.644298+0000","last_became_peered":"2024-03-11T21:48:28.644298+0000","last_unstale":"2024-03-11T21:51:07.482247+0000","last_undegraded":"2024-03-11T21:51:07.482247+0000","last_fullsized":"2024-03-11T21:51:07.482247+0000","mapping_epoch":502,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":503,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:51:05.710446+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:41.544550+0000","last_clean_scrub_stamp":"2024-03-11T21:51:05.710446+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:52:23.406037+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":1065,"reported_epoch":554,"state":"active+clean","last_fresh":"2024-03-11T21:51:06.961489+0000","last_change":"2024-03-11T21:50:35.034430+0000","last_active":"2024-03-11T21:51:06.961489+0000","last_peered":"2024-03-11T21:51:06.961489+0000","last_clean":"2024-03-11T21:51:06.961489+0000","last_became_active":"2024-03-11T21:25:45.301416+0000","last_became_peered":"2024-03-11T21:25:45.301416+0000","last_unstale":"2024-03-11T21:51:06.961489+0000","last_undegraded":"2024-03-11T21:51:06.961489+0000","last_fullsized":"2024-03-11T21:51:06.961489+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:50:35.034308+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:43.559452+0000","last_clean_scrub_stamp":"2024-03-11T21:50:35.034308+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:51:49.148312+0000","scrub_duration":104,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":723,"reported_epoch":554,"state":"active+clean","last_fresh":"2024-03-11T21:51:07.587958+0000","last_change":"2024-03-11T21:50:14.240816+0000","last_active":"2024-03-11T21:51:07.587958+0000","last_peered":"2024-03-11T21:51:07.587958+0000","last_clean":"2024-03-11T21:51:07.587958+0000","last_became_active":"2024-03-11T21:49:26.644277+0000","last_became_peered":"2024-03-11T21:49:26.644277+0000","last_unstale":"2024-03-11T21:51:07.587958+0000","last_undegraded":"2024-03-11T21:51:07.587958+0000","last_fullsized":"2024-03-11T21:51:07.587958+0000","mapping_epoch":532,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":533,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:50:14.240723+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:26:57.333669+0000","last_clean_scrub_stamp":"2024-03-11T21:50:14.240723+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:51:39.046762+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.3","version":"456'47","reported_seq":1243,"reported_epoch":554,"state":"active+clean","last_fresh":"2024-03-11T21:51:06.721058+0000","last_change":"2024-03-11T21:50:56.393665+0000","last_active":"2024-03-11T21:51:06.721058+0000","last_peered":"2024-03-11T21:51:06.721058+0000","last_clean":"2024-03-11T21:51:06.721058+0000","last_became_active":"2024-03-11T21:48:28.623917+0000","last_became_peered":"2024-03-11T21:48:28.623917+0000","last_unstale":"2024-03-11T21:51:06.721058+0000","last_undegraded":"2024-03-11T21:51:06.721058+0000","last_fullsized":"2024-03-11T21:51:06.721058+0000","mapping_epoch":502,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":503,"parent":"0.0","parent_split_bits":0,"last_scrub":"456'47","last_scrub_stamp":"2024-03-11T21:50:56.393540+0000","last_deep_scrub":"165'27","last_deep_scrub_stamp":"2024-03-11T21:32:21.576732+0000","last_clean_scrub_stamp":"2024-03-11T21:50:56.393540+0000","objects_scrubbed":8,"log_size":47,"log_dups_size":0,"ondisk_log_size":47,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:52:15.226687+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":69,"num_read_kb":55,"num_write":42,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":32,"num_bytes_recovered":1320,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":1097,"reported_epoch":554,"state":"active+clean","last_fresh":"2024-03-11T21:51:07.586533+0000","last_change":"2024-03-11T21:50:07.070790+0000","last_active":"2024-03-11T21:51:07.586533+0000","last_peered":"2024-03-11T21:51:07.586533+0000","last_clean":"2024-03-11T21:51:07.586533+0000","last_became_active":"2024-03-11T21:48:28.642354+0000","last_became_peered":"2024-03-11T21:48:28.642354+0000","last_unstale":"2024-03-11T21:51:07.586533+0000","last_undegraded":"2024-03-11T21:51:07.586533+0000","last_fullsized":"2024-03-11T21:51:07.586533+0000","mapping_epoch":502,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":503,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:50:07.070658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:41:21.857205+0000","last_clean_scrub_stamp":"2024-03-11T21:50:07.070658+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:51:17.405424+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":61,"reported_epoch":554,"state":"active+clean","last_fresh":"2024-03-11T21:51:06.722292+0000","last_change":"2024-03-11T21:50:02.026756+0000","last_active":"2024-03-11T21:51:06.722292+0000","last_peered":"2024-03-11T21:51:06.722292+0000","last_clean":"2024-03-11T21:51:06.722292+0000","last_became_active":"2024-03-11T21:49:26.642045+0000","last_became_peered":"2024-03-11T21:49:26.642045+0000","last_unstale":"2024-03-11T21:51:06.722292+0000","last_undegraded":"2024-03-11T21:51:06.722292+0000","last_fullsized":"2024-03-11T21:51:06.722292+0000","mapping_epoch":532,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":533,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:50:02.026631+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:43.559452+0000","last_clean_scrub_stamp":"2024-03-11T21:50:02.026631+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:51:10.962724+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.6","version":"67'1","reported_seq":1107,"reported_epoch":554,"state":"active+clean","last_fresh":"2024-03-11T21:51:06.961833+0000","last_change":"2024-03-11T21:50:33.961085+0000","last_active":"2024-03-11T21:51:06.961833+0000","last_peered":"2024-03-11T21:51:06.961833+0000","last_clean":"2024-03-11T21:51:06.961833+0000","last_became_active":"2024-03-11T21:49:26.644153+0000","last_became_peered":"2024-03-11T21:49:26.644153+0000","last_unstale":"2024-03-11T21:51:06.961833+0000","last_undegraded":"2024-03-11T21:51:06.961833+0000","last_fullsized":"2024-03-11T21:51:06.961833+0000","mapping_epoch":532,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":533,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'1","last_scrub_stamp":"2024-03-11T21:50:33.960986+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:37.512018+0000","last_clean_scrub_stamp":"2024-03-11T21:50:33.960986+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:51:44.085831+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":138,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,4],"acting":[1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.0","version":"456'50","reported_seq":87,"reported_epoch":554,"state":"active+clean","last_fresh":"2024-03-11T21:51:07.585980+0000","last_change":"2024-03-11T21:50:37.619125+0000","last_active":"2024-03-11T21:51:07.585980+0000","last_peered":"2024-03-11T21:51:07.585980+0000","last_clean":"2024-03-11T21:51:07.585980+0000","last_became_active":"2024-03-11T21:49:17.155148+0000","last_became_peered":"2024-03-11T21:49:17.155148+0000","last_unstale":"2024-03-11T21:51:07.585980+0000","last_undegraded":"2024-03-11T21:51:07.585980+0000","last_fullsized":"2024-03-11T21:51:07.585980+0000","mapping_epoch":528,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":529,"parent":"0.0","parent_split_bits":0,"last_scrub":"456'50","last_scrub_stamp":"2024-03-11T21:50:37.619018+0000","last_deep_scrub":"260'38","last_deep_scrub_stamp":"2024-03-11T21:45:39.241846+0000","last_clean_scrub_stamp":"2024-03-11T21:50:37.619018+0000","objects_scrubbed":8,"log_size":50,"log_dups_size":0,"ondisk_log_size":50,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:52:00.939522+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":69,"num_read_kb":57,"num_write":44,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":24,"num_bytes_recovered":660,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":69,"reported_epoch":554,"state":"active+clean","last_fresh":"2024-03-11T21:51:06.722170+0000","last_change":"2024-03-11T21:50:45.179659+0000","last_active":"2024-03-11T21:51:06.722170+0000","last_peered":"2024-03-11T21:51:06.722170+0000","last_clean":"2024-03-11T21:51:06.722170+0000","last_became_active":"2024-03-11T21:49:26.645160+0000","last_became_peered":"2024-03-11T21:49:26.645160+0000","last_unstale":"2024-03-11T21:51:06.722170+0000","last_undegraded":"2024-03-11T21:51:06.722170+0000","last_fullsized":"2024-03-11T21:51:06.722170+0000","mapping_epoch":532,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":533,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:50:45.179558+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:50:45.179558+0000","last_clean_scrub_stamp":"2024-03-11T21:50:45.179558+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:51:47.048378+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,7],"acting":[4,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.3","version":"0'0","reported_seq":711,"reported_epoch":554,"state":"active+clean","last_fresh":"2024-03-11T21:51:07.587260+0000","last_change":"2024-03-11T21:50:48.516514+0000","last_active":"2024-03-11T21:51:07.587260+0000","last_peered":"2024-03-11T21:51:07.587260+0000","last_clean":"2024-03-11T21:51:07.587260+0000","last_became_active":"2024-03-11T21:34:41.142683+0000","last_became_peered":"2024-03-11T21:34:41.142683+0000","last_unstale":"2024-03-11T21:51:07.587260+0000","last_undegraded":"2024-03-11T21:51:07.587260+0000","last_fullsized":"2024-03-11T21:51:07.587260+0000","mapping_epoch":226,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":227,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:50:48.516411+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:46:25.645722+0000","last_clean_scrub_stamp":"2024-03-11T21:50:48.516411+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:52:04.201716+0000","scrub_duration":127,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,2],"acting":[7,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":1104,"reported_epoch":554,"state":"active+clean","last_fresh":"2024-03-11T21:51:07.587553+0000","last_change":"2024-03-11T21:50:08.119410+0000","last_active":"2024-03-11T21:51:07.587553+0000","last_peered":"2024-03-11T21:51:07.587553+0000","last_clean":"2024-03-11T21:51:07.587553+0000","last_became_active":"2024-03-11T21:34:41.145628+0000","last_became_peered":"2024-03-11T21:34:41.145628+0000","last_unstale":"2024-03-11T21:51:07.587553+0000","last_undegraded":"2024-03-11T21:51:07.587553+0000","last_fullsized":"2024-03-11T21:51:07.587553+0000","mapping_epoch":226,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":227,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:50:08.119254+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:37.512018+0000","last_clean_scrub_stamp":"2024-03-11T21:50:08.119254+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:51:29.262758+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,2],"acting":[6,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.1","version":"507'49","reported_seq":77,"reported_epoch":554,"state":"active+clean","last_fresh":"2024-03-11T21:51:06.722297+0000","last_change":"2024-03-11T21:49:43.852514+0000","last_active":"2024-03-11T21:51:06.722297+0000","last_peered":"2024-03-11T21:51:06.722297+0000","last_clean":"2024-03-11T21:51:06.722297+0000","last_became_active":"2024-03-11T21:49:26.647366+0000","last_became_peered":"2024-03-11T21:49:26.647366+0000","last_unstale":"2024-03-11T21:51:06.722297+0000","last_undegraded":"2024-03-11T21:51:06.722297+0000","last_fullsized":"2024-03-11T21:51:06.722297+0000","mapping_epoch":532,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":533,"parent":"0.0","parent_split_bits":0,"last_scrub":"507'49","last_scrub_stamp":"2024-03-11T21:49:43.852390+0000","last_deep_scrub":"507'49","last_deep_scrub_stamp":"2024-03-11T21:49:43.852390+0000","last_clean_scrub_stamp":"2024-03-11T21:49:43.852390+0000","objects_scrubbed":7,"log_size":49,"log_dups_size":0,"ondisk_log_size":49,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":61,"num_read_kb":54,"num_write":42,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":21,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6],"acting":[4,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.0","version":"533'15","reported_seq":114,"reported_epoch":554,"state":"active+clean","last_fresh":"2024-03-11T21:51:07.586183+0000","last_change":"2024-03-11T21:51:03.864178+0000","last_active":"2024-03-11T21:51:07.586183+0000","last_peered":"2024-03-11T21:51:07.586183+0000","last_clean":"2024-03-11T21:51:07.586183+0000","last_became_active":"2024-03-11T21:49:26.638749+0000","last_became_peered":"2024-03-11T21:49:26.638749+0000","last_unstale":"2024-03-11T21:51:07.586183+0000","last_undegraded":"2024-03-11T21:51:07.586183+0000","last_fullsized":"2024-03-11T21:51:07.586183+0000","mapping_epoch":532,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":533,"parent":"0.0","parent_split_bits":0,"last_scrub":"533'15","last_scrub_stamp":"2024-03-11T21:51:03.864033+0000","last_deep_scrub":"98'9","last_deep_scrub_stamp":"2024-03-11T21:27:29.745711+0000","last_clean_scrub_stamp":"2024-03-11T21:51:03.864033+0000","objects_scrubbed":1,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:52:09.359413+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.c","version":"0'0","reported_seq":69,"reported_epoch":554,"state":"active+clean","last_fresh":"2024-03-11T21:51:07.586652+0000","last_change":"2024-03-11T21:50:03.603744+0000","last_active":"2024-03-11T21:51:07.586652+0000","last_peered":"2024-03-11T21:51:07.586652+0000","last_clean":"2024-03-11T21:51:07.586652+0000","last_became_active":"2024-03-11T21:49:17.141485+0000","last_became_peered":"2024-03-11T21:49:17.141485+0000","last_unstale":"2024-03-11T21:51:07.586652+0000","last_undegraded":"2024-03-11T21:51:07.586652+0000","last_fullsized":"2024-03-11T21:51:07.586652+0000","mapping_epoch":528,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":529,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:50:03.603643+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:43.559452+0000","last_clean_scrub_stamp":"2024-03-11T21:50:03.603643+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:51:24.851391+0000","scrub_duration":39,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.f","version":"0'0","reported_seq":136,"reported_epoch":554,"state":"active+clean","last_fresh":"2024-03-11T21:51:07.482373+0000","last_change":"2024-03-11T21:50:16.964646+0000","last_active":"2024-03-11T21:51:07.482373+0000","last_peered":"2024-03-11T21:51:07.482373+0000","last_clean":"2024-03-11T21:51:07.482373+0000","last_became_active":"2024-03-11T21:49:26.649455+0000","last_became_peered":"2024-03-11T21:49:26.649455+0000","last_unstale":"2024-03-11T21:51:07.482373+0000","last_undegraded":"2024-03-11T21:51:07.482373+0000","last_fullsized":"2024-03-11T21:51:07.482373+0000","mapping_epoch":532,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":533,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:50:16.964541+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:50:16.964541+0000","last_clean_scrub_stamp":"2024-03-11T21:50:16.964541+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:51:21.063024+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.8","version":"0'0","reported_seq":711,"reported_epoch":554,"state":"active+clean","last_fresh":"2024-03-11T21:51:07.586769+0000","last_change":"2024-03-11T21:50:26.303889+0000","last_active":"2024-03-11T21:51:07.586769+0000","last_peered":"2024-03-11T21:51:07.586769+0000","last_clean":"2024-03-11T21:51:07.586769+0000","last_became_active":"2024-03-11T21:34:41.144681+0000","last_became_peered":"2024-03-11T21:34:41.144681+0000","last_unstale":"2024-03-11T21:51:07.586769+0000","last_undegraded":"2024-03-11T21:51:07.586769+0000","last_fullsized":"2024-03-11T21:51:07.586769+0000","mapping_epoch":226,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":227,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:50:26.303787+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:37.512018+0000","last_clean_scrub_stamp":"2024-03-11T21:50:26.303787+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:51:31.143084+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,1],"acting":[7,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.e","version":"456'32","reported_seq":73,"reported_epoch":554,"state":"active+clean","last_fresh":"2024-03-11T21:51:06.722837+0000","last_change":"2024-03-11T21:49:33.992325+0000","last_active":"2024-03-11T21:51:06.722837+0000","last_peered":"2024-03-11T21:51:06.722837+0000","last_clean":"2024-03-11T21:51:06.722837+0000","last_became_active":"2024-03-11T21:49:26.658056+0000","last_became_peered":"2024-03-11T21:49:26.658056+0000","last_unstale":"2024-03-11T21:51:06.722837+0000","last_undegraded":"2024-03-11T21:51:06.722837+0000","last_fullsized":"2024-03-11T21:51:06.722837+0000","mapping_epoch":532,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":533,"parent":"0.0","parent_split_bits":0,"last_scrub":"456'32","last_scrub_stamp":"2024-03-11T21:49:33.992205+0000","last_deep_scrub":"109'16","last_deep_scrub_stamp":"2024-03-11T21:28:24.739395+0000","last_clean_scrub_stamp":"2024-03-11T21:49:33.992205+0000","objects_scrubbed":5,"log_size":32,"log_dups_size":0,"ondisk_log_size":32,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":42,"num_read_kb":35,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.d","version":"0'0","reported_seq":130,"reported_epoch":554,"state":"active+clean","last_fresh":"2024-03-11T21:51:07.482050+0000","last_change":"2024-03-11T21:51:03.801395+0000","last_active":"2024-03-11T21:51:07.482050+0000","last_peered":"2024-03-11T21:51:07.482050+0000","last_clean":"2024-03-11T21:51:07.482050+0000","last_became_active":"2024-03-11T21:48:28.645468+0000","last_became_peered":"2024-03-11T21:48:28.645468+0000","last_unstale":"2024-03-11T21:51:07.482050+0000","last_undegraded":"2024-03-11T21:51:07.482050+0000","last_fullsized":"2024-03-11T21:51:07.482050+0000","mapping_epoch":502,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":503,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:51:03.801263+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:51:03.801263+0000","last_clean_scrub_stamp":"2024-03-11T21:51:03.801263+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:52:07.125936+0000","scrub_duration":110,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.e","version":"533'12","reported_seq":86,"reported_epoch":554,"state":"active+clean","last_fresh":"2024-03-11T21:51:06.722843+0000","last_change":"2024-03-11T21:49:55.917066+0000","last_active":"2024-03-11T21:51:06.722843+0000","last_peered":"2024-03-11T21:51:06.722843+0000","last_clean":"2024-03-11T21:51:06.722843+0000","last_became_active":"2024-03-11T21:49:26.647248+0000","last_became_peered":"2024-03-11T21:49:26.647248+0000","last_unstale":"2024-03-11T21:51:06.722843+0000","last_undegraded":"2024-03-11T21:51:06.722843+0000","last_fullsized":"2024-03-11T21:51:06.722843+0000","mapping_epoch":532,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":533,"parent":"0.0","parent_split_bits":0,"last_scrub":"533'12","last_scrub_stamp":"2024-03-11T21:49:55.916961+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:41.544550+0000","last_clean_scrub_stamp":"2024-03-11T21:49:55.916961+0000","objects_scrubbed":1,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,0],"acting":[4,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.9","version":"0'0","reported_seq":1101,"reported_epoch":554,"state":"active+clean","last_fresh":"2024-03-11T21:51:06.961368+0000","last_change":"2024-03-11T21:50:46.940418+0000","last_active":"2024-03-11T21:51:06.961368+0000","last_peered":"2024-03-11T21:51:06.961368+0000","last_clean":"2024-03-11T21:51:06.961368+0000","last_became_active":"2024-03-11T21:49:17.138195+0000","last_became_peered":"2024-03-11T21:49:17.138195+0000","last_unstale":"2024-03-11T21:51:06.961368+0000","last_undegraded":"2024-03-11T21:51:06.961368+0000","last_fullsized":"2024-03-11T21:51:06.961368+0000","mapping_epoch":528,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":529,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:50:46.940341+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:26:50.991712+0000","last_clean_scrub_stamp":"2024-03-11T21:50:46.940341+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:52:02.972249+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,3],"acting":[1,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.f","version":"456'39","reported_seq":1168,"reported_epoch":554,"state":"active+clean","last_fresh":"2024-03-11T21:51:11.993162+0000","last_change":"2024-03-11T21:51:11.993195+0000","last_active":"2024-03-11T21:51:11.993162+0000","last_peered":"2024-03-11T21:51:11.993162+0000","last_clean":"2024-03-11T21:51:11.993162+0000","last_became_active":"2024-03-11T21:49:26.637780+0000","last_became_peered":"2024-03-11T21:49:26.637780+0000","last_unstale":"2024-03-11T21:51:11.993162+0000","last_undegraded":"2024-03-11T21:51:11.993162+0000","last_fullsized":"2024-03-11T21:51:11.993162+0000","mapping_epoch":532,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":533,"parent":"0.0","parent_split_bits":0,"last_scrub":"456'39","last_scrub_stamp":"2024-03-11T21:51:11.993102+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:39.525554+0000","last_clean_scrub_stamp":"2024-03-11T21:51:11.993102+0000","objects_scrubbed":7,"log_size":39,"log_dups_size":0,"ondisk_log_size":39,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:52:19.152297+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":62,"num_read_kb":53,"num_write":40,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":48,"num_bytes_recovered":660,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,4],"acting":[2,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.e","version":"0'0","reported_seq":61,"reported_epoch":554,"state":"active+clean","last_fresh":"2024-03-11T21:51:06.722928+0000","last_change":"2024-03-11T21:49:41.835396+0000","last_active":"2024-03-11T21:51:06.722928+0000","last_peered":"2024-03-11T21:51:06.722928+0000","last_clean":"2024-03-11T21:51:06.722928+0000","last_became_active":"2024-03-11T21:49:26.639018+0000","last_became_peered":"2024-03-11T21:49:26.639018+0000","last_unstale":"2024-03-11T21:51:06.722928+0000","last_undegraded":"2024-03-11T21:51:06.722928+0000","last_fullsized":"2024-03-11T21:51:06.722928+0000","mapping_epoch":532,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":533,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:49:41.835252+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:43.559452+0000","last_clean_scrub_stamp":"2024-03-11T21:49:41.835252+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":37,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.d","version":"503'12","reported_seq":1413,"reported_epoch":554,"state":"active+clean","last_fresh":"2024-03-11T21:51:06.722225+0000","last_change":"2024-03-11T21:50:30.688221+0000","last_active":"2024-03-11T21:51:06.722225+0000","last_peered":"2024-03-11T21:51:06.722225+0000","last_clean":"2024-03-11T21:51:06.722225+0000","last_became_active":"2024-03-11T21:48:28.627926+0000","last_became_peered":"2024-03-11T21:48:28.627926+0000","last_unstale":"2024-03-11T21:51:06.722225+0000","last_undegraded":"2024-03-11T21:51:06.722225+0000","last_fullsized":"2024-03-11T21:51:06.722225+0000","mapping_epoch":502,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":503,"parent":"0.0","parent_split_bits":0,"last_scrub":"503'12","last_scrub_stamp":"2024-03-11T21:50:30.688111+0000","last_deep_scrub":"109'9","last_deep_scrub_stamp":"2024-03-11T21:28:27.082975+0000","last_clean_scrub_stamp":"2024-03-11T21:50:30.688111+0000","objects_scrubbed":1,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:51:51.903565+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6,5],"acting":[2,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.a","version":"0'0","reported_seq":1030,"reported_epoch":554,"state":"active+clean","last_fresh":"2024-03-11T21:51:06.961573+0000","last_change":"2024-03-11T21:50:39.942550+0000","last_active":"2024-03-11T21:51:06.961573+0000","last_peered":"2024-03-11T21:51:06.961573+0000","last_clean":"2024-03-11T21:51:06.961573+0000","last_became_active":"2024-03-11T21:49:46.949130+0000","last_became_peered":"2024-03-11T21:49:46.949130+0000","last_unstale":"2024-03-11T21:51:06.961573+0000","last_undegraded":"2024-03-11T21:51:06.961573+0000","last_fullsized":"2024-03-11T21:51:06.961573+0000","mapping_epoch":537,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":538,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:50:39.942447+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:44:27.996522+0000","last_clean_scrub_stamp":"2024-03-11T21:50:39.942447+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:51:53.447270+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,7],"acting":[1,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.c","version":"547'35","reported_seq":78,"reported_epoch":554,"state":"active+clean","last_fresh":"2024-03-11T21:51:06.722507+0000","last_change":"2024-03-11T21:50:16.011651+0000","last_active":"2024-03-11T21:51:06.722507+0000","last_peered":"2024-03-11T21:51:06.722507+0000","last_clean":"2024-03-11T21:51:06.722507+0000","last_became_active":"2024-03-11T21:49:26.652747+0000","last_became_peered":"2024-03-11T21:49:26.652747+0000","last_unstale":"2024-03-11T21:51:06.722507+0000","last_undegraded":"2024-03-11T21:51:06.722507+0000","last_fullsized":"2024-03-11T21:51:06.722507+0000","mapping_epoch":532,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":533,"parent":"0.0","parent_split_bits":0,"last_scrub":"547'35","last_scrub_stamp":"2024-03-11T21:50:16.011524+0000","last_deep_scrub":"547'35","last_deep_scrub_stamp":"2024-03-11T21:50:16.011524+0000","last_clean_scrub_stamp":"2024-03-11T21:50:16.011524+0000","objects_scrubbed":5,"log_size":35,"log_dups_size":0,"ondisk_log_size":35,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:51:45.395920+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":43,"num_read_kb":38,"num_write":30,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.f","version":"0'0","reported_seq":1095,"reported_epoch":554,"state":"active+clean","last_fresh":"2024-03-11T21:51:06.722362+0000","last_change":"2024-03-11T21:50:26.857208+0000","last_active":"2024-03-11T21:51:06.722362+0000","last_peered":"2024-03-11T21:51:06.722362+0000","last_clean":"2024-03-11T21:51:06.722362+0000","last_became_active":"2024-03-11T21:49:26.637689+0000","last_became_peered":"2024-03-11T21:49:26.637689+0000","last_unstale":"2024-03-11T21:51:06.722362+0000","last_undegraded":"2024-03-11T21:51:06.722362+0000","last_fullsized":"2024-03-11T21:51:06.722362+0000","mapping_epoch":532,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":533,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:50:26.857094+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:27:03.010969+0000","last_clean_scrub_stamp":"2024-03-11T21:50:26.857094+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:51:35.841803+0000","scrub_duration":178,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,4],"acting":[2,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":945,"reported_epoch":554,"state":"active+clean","last_fresh":"2024-03-11T21:51:06.961704+0000","last_change":"2024-03-11T21:50:20.931523+0000","last_active":"2024-03-11T21:51:06.961704+0000","last_peered":"2024-03-11T21:51:06.961704+0000","last_clean":"2024-03-11T21:51:06.961704+0000","last_became_active":"2024-03-11T21:49:46.255714+0000","last_became_peered":"2024-03-11T21:49:46.255714+0000","last_unstale":"2024-03-11T21:51:06.961704+0000","last_undegraded":"2024-03-11T21:51:06.961704+0000","last_fullsized":"2024-03-11T21:51:06.961704+0000","mapping_epoch":537,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":538,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:50:20.931438+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:30:22.750053+0000","last_clean_scrub_stamp":"2024-03-11T21:50:20.931438+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:51:29.155091+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.b","version":"0'0","reported_seq":741,"reported_epoch":554,"state":"active+clean","last_fresh":"2024-03-11T21:51:07.587317+0000","last_change":"2024-03-11T21:50:21.184234+0000","last_active":"2024-03-11T21:51:07.587317+0000","last_peered":"2024-03-11T21:51:07.587317+0000","last_clean":"2024-03-11T21:51:07.587317+0000","last_became_active":"2024-03-11T21:49:26.645565+0000","last_became_peered":"2024-03-11T21:49:26.645565+0000","last_unstale":"2024-03-11T21:51:07.587317+0000","last_undegraded":"2024-03-11T21:51:07.587317+0000","last_fullsized":"2024-03-11T21:51:07.587317+0000","mapping_epoch":532,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":533,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:50:21.184074+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:50:21.184074+0000","last_clean_scrub_stamp":"2024-03-11T21:50:21.184074+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:51:28.521325+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,5],"acting":[7,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.d","version":"456'53","reported_seq":79,"reported_epoch":554,"state":"active+clean","last_fresh":"2024-03-11T21:51:06.722981+0000","last_change":"2024-03-11T21:50:33.152694+0000","last_active":"2024-03-11T21:51:06.722981+0000","last_peered":"2024-03-11T21:51:06.722981+0000","last_clean":"2024-03-11T21:51:06.722981+0000","last_became_active":"2024-03-11T21:49:26.645458+0000","last_became_peered":"2024-03-11T21:49:26.645458+0000","last_unstale":"2024-03-11T21:51:06.722981+0000","last_undegraded":"2024-03-11T21:51:06.722981+0000","last_fullsized":"2024-03-11T21:51:06.722981+0000","mapping_epoch":532,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":533,"parent":"0.0","parent_split_bits":0,"last_scrub":"456'53","last_scrub_stamp":"2024-03-11T21:50:33.152576+0000","last_deep_scrub":"165'25","last_deep_scrub_stamp":"2024-03-11T21:32:21.678328+0000","last_clean_scrub_stamp":"2024-03-11T21:50:33.152576+0000","objects_scrubbed":8,"log_size":53,"log_dups_size":0,"ondisk_log_size":53,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:51:35.751007+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":69,"num_read_kb":59,"num_write":46,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,1],"acting":[4,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":723,"reported_epoch":554,"state":"active+clean","last_fresh":"2024-03-11T21:51:07.586464+0000","last_change":"2024-03-11T21:50:19.218149+0000","last_active":"2024-03-11T21:51:07.586464+0000","last_peered":"2024-03-11T21:51:07.586464+0000","last_clean":"2024-03-11T21:51:07.586464+0000","last_became_active":"2024-03-11T21:49:17.149515+0000","last_became_peered":"2024-03-11T21:49:17.149515+0000","last_unstale":"2024-03-11T21:51:07.586464+0000","last_undegraded":"2024-03-11T21:51:07.586464+0000","last_fullsized":"2024-03-11T21:51:07.586464+0000","mapping_epoch":528,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":529,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:50:19.218038+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:49:14.834307+0000","last_clean_scrub_stamp":"2024-03-11T21:50:19.218038+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:51:33.744094+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,3],"acting":[7,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":1078,"reported_epoch":554,"state":"active+clean","last_fresh":"2024-03-11T21:51:06.720890+0000","last_change":"2024-03-11T21:50:27.632429+0000","last_active":"2024-03-11T21:51:06.720890+0000","last_peered":"2024-03-11T21:51:06.720890+0000","last_clean":"2024-03-11T21:51:06.720890+0000","last_became_active":"2024-03-11T21:48:28.627492+0000","last_became_peered":"2024-03-11T21:48:28.627492+0000","last_unstale":"2024-03-11T21:51:06.720890+0000","last_undegraded":"2024-03-11T21:51:06.720890+0000","last_fullsized":"2024-03-11T21:51:06.720890+0000","mapping_epoch":502,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":503,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:50:27.632354+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:41.544550+0000","last_clean_scrub_stamp":"2024-03-11T21:50:27.632354+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:51:35.529300+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,5],"acting":[2,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.c","version":"0'0","reported_seq":1078,"reported_epoch":554,"state":"active+clean","last_fresh":"2024-03-11T21:51:06.721352+0000","last_change":"2024-03-11T21:49:54.834640+0000","last_active":"2024-03-11T21:51:06.721352+0000","last_peered":"2024-03-11T21:51:06.721352+0000","last_clean":"2024-03-11T21:51:06.721352+0000","last_became_active":"2024-03-11T21:48:28.636860+0000","last_became_peered":"2024-03-11T21:48:28.636860+0000","last_unstale":"2024-03-11T21:51:06.721352+0000","last_undegraded":"2024-03-11T21:51:06.721352+0000","last_fullsized":"2024-03-11T21:51:06.721352+0000","mapping_epoch":502,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":503,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:49:54.834561+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:29:30.279910+0000","last_clean_scrub_stamp":"2024-03-11T21:49:54.834561+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:51:12.239609+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,0],"acting":[2,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.a","version":"456'60","reported_seq":1228,"reported_epoch":554,"state":"active+clean","last_fresh":"2024-03-11T21:51:07.586355+0000","last_change":"2024-03-11T21:50:32.207942+0000","last_active":"2024-03-11T21:51:07.586355+0000","last_peered":"2024-03-11T21:51:07.586355+0000","last_clean":"2024-03-11T21:51:07.586355+0000","last_became_active":"2024-03-11T21:34:41.148863+0000","last_became_peered":"2024-03-11T21:34:41.148863+0000","last_unstale":"2024-03-11T21:51:07.586355+0000","last_undegraded":"2024-03-11T21:51:07.586355+0000","last_fullsized":"2024-03-11T21:51:07.586355+0000","mapping_epoch":226,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":227,"parent":"0.0","parent_split_bits":0,"last_scrub":"456'60","last_scrub_stamp":"2024-03-11T21:50:32.207776+0000","last_deep_scrub":"260'44","last_deep_scrub_stamp":"2024-03-11T21:45:32.673622+0000","last_clean_scrub_stamp":"2024-03-11T21:50:32.207776+0000","objects_scrubbed":9,"log_size":60,"log_dups_size":0,"ondisk_log_size":60,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:51:54.831082+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":80,"num_read_kb":69,"num_write":52,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,7],"acting":[6,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":1076,"reported_epoch":554,"state":"active+clean","last_fresh":"2024-03-11T21:51:06.721435+0000","last_change":"2024-03-11T21:50:49.456275+0000","last_active":"2024-03-11T21:51:06.721435+0000","last_peered":"2024-03-11T21:51:06.721435+0000","last_clean":"2024-03-11T21:51:06.721435+0000","last_became_active":"2024-03-11T21:34:41.154507+0000","last_became_peered":"2024-03-11T21:34:41.154507+0000","last_unstale":"2024-03-11T21:51:06.721435+0000","last_undegraded":"2024-03-11T21:51:06.721435+0000","last_fullsized":"2024-03-11T21:51:06.721435+0000","mapping_epoch":226,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":227,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:50:49.456170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:50:49.456170+0000","last_clean_scrub_stamp":"2024-03-11T21:50:49.456170+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:52:15.226895+0000","scrub_duration":83,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":1099,"reported_epoch":554,"state":"active+clean","last_fresh":"2024-03-11T21:51:06.720537+0000","last_change":"2024-03-11T21:50:54.928949+0000","last_active":"2024-03-11T21:51:06.720537+0000","last_peered":"2024-03-11T21:51:06.720537+0000","last_clean":"2024-03-11T21:51:06.720537+0000","last_became_active":"2024-03-11T21:49:26.637683+0000","last_became_peered":"2024-03-11T21:49:26.637683+0000","last_unstale":"2024-03-11T21:51:06.720537+0000","last_undegraded":"2024-03-11T21:51:06.720537+0000","last_fullsized":"2024-03-11T21:51:06.720537+0000","mapping_epoch":532,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":533,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:50:54.928873+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:43:32.115556+0000","last_clean_scrub_stamp":"2024-03-11T21:50:54.928873+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:52:16.926036+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,3],"acting":[2,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.d","version":"0'0","reported_seq":1101,"reported_epoch":554,"state":"active+clean","last_fresh":"2024-03-11T21:51:06.961897+0000","last_change":"2024-03-11T21:50:06.956815+0000","last_active":"2024-03-11T21:51:06.961897+0000","last_peered":"2024-03-11T21:51:06.961897+0000","last_clean":"2024-03-11T21:51:06.961897+0000","last_became_active":"2024-03-11T21:49:26.645233+0000","last_became_peered":"2024-03-11T21:49:26.645233+0000","last_unstale":"2024-03-11T21:51:06.961897+0000","last_undegraded":"2024-03-11T21:51:06.961897+0000","last_fullsized":"2024-03-11T21:51:06.961897+0000","mapping_epoch":532,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":533,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:50:06.956743+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:37.512018+0000","last_clean_scrub_stamp":"2024-03-11T21:50:06.956743+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:51:31.187236+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,3],"acting":[1,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.b","version":"456'21","reported_seq":1194,"reported_epoch":554,"state":"active+clean","last_fresh":"2024-03-11T21:51:06.721227+0000","last_change":"2024-03-11T21:50:31.295800+0000","last_active":"2024-03-11T21:51:06.721227+0000","last_peered":"2024-03-11T21:51:06.721227+0000","last_clean":"2024-03-11T21:51:06.721227+0000","last_became_active":"2024-03-11T21:49:26.643714+0000","last_became_peered":"2024-03-11T21:49:26.643714+0000","last_unstale":"2024-03-11T21:51:06.721227+0000","last_undegraded":"2024-03-11T21:51:06.721227+0000","last_fullsized":"2024-03-11T21:51:06.721227+0000","mapping_epoch":532,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":533,"parent":"0.0","parent_split_bits":0,"last_scrub":"456'21","last_scrub_stamp":"2024-03-11T21:50:31.295707+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:39.525554+0000","last_clean_scrub_stamp":"2024-03-11T21:50:31.295707+0000","objects_scrubbed":4,"log_size":21,"log_dups_size":0,"ondisk_log_size":21,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:51:45.942912+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":35,"num_read_kb":29,"num_write":22,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":24,"num_bytes_recovered":660,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,4],"acting":[0,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":61,"reported_epoch":554,"state":"active+clean","last_fresh":"2024-03-11T21:51:06.722382+0000","last_change":"2024-03-11T21:50:07.044282+0000","last_active":"2024-03-11T21:51:06.722382+0000","last_peered":"2024-03-11T21:51:06.722382+0000","last_clean":"2024-03-11T21:51:06.722382+0000","last_became_active":"2024-03-11T21:49:26.636259+0000","last_became_peered":"2024-03-11T21:49:26.636259+0000","last_unstale":"2024-03-11T21:51:06.722382+0000","last_undegraded":"2024-03-11T21:51:06.722382+0000","last_fullsized":"2024-03-11T21:51:06.722382+0000","mapping_epoch":532,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":533,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:50:07.044182+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:43.559452+0000","last_clean_scrub_stamp":"2024-03-11T21:50:07.044182+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:51:30.522244+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,3],"acting":[4,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.e","version":"0'0","reported_seq":1115,"reported_epoch":554,"state":"active+clean","last_fresh":"2024-03-11T21:51:06.722239+0000","last_change":"2024-03-11T21:50:21.703018+0000","last_active":"2024-03-11T21:51:06.722239+0000","last_peered":"2024-03-11T21:51:06.722239+0000","last_clean":"2024-03-11T21:51:06.722239+0000","last_became_active":"2024-03-11T21:49:17.144426+0000","last_became_peered":"2024-03-11T21:49:17.144426+0000","last_unstale":"2024-03-11T21:51:06.722239+0000","last_undegraded":"2024-03-11T21:51:06.722239+0000","last_fullsized":"2024-03-11T21:51:06.722239+0000","mapping_epoch":528,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":529,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:50:21.702930+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:50:21.702930+0000","last_clean_scrub_stamp":"2024-03-11T21:50:21.702930+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:51:39.114694+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,7],"acting":[2,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.8","version":"456'46","reported_seq":146,"reported_epoch":554,"state":"active+clean","last_fresh":"2024-03-11T21:51:07.481507+0000","last_change":"2024-03-11T21:50:30.920888+0000","last_active":"2024-03-11T21:51:07.481507+0000","last_peered":"2024-03-11T21:51:07.481507+0000","last_clean":"2024-03-11T21:51:07.481507+0000","last_became_active":"2024-03-11T21:48:28.642553+0000","last_became_peered":"2024-03-11T21:48:28.642553+0000","last_unstale":"2024-03-11T21:51:07.481507+0000","last_undegraded":"2024-03-11T21:51:07.481507+0000","last_fullsized":"2024-03-11T21:51:07.481507+0000","mapping_epoch":502,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":503,"parent":"0.0","parent_split_bits":0,"last_scrub":"456'46","last_scrub_stamp":"2024-03-11T21:50:30.920781+0000","last_deep_scrub":"109'22","last_deep_scrub_stamp":"2024-03-11T21:32:07.573197+0000","last_clean_scrub_stamp":"2024-03-11T21:50:30.920781+0000","objects_scrubbed":7,"log_size":46,"log_dups_size":0,"ondisk_log_size":46,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:51:35.367621+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":61,"num_read_kb":52,"num_write":40,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":28,"num_bytes_recovered":440,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.9","version":"533'13","reported_seq":934,"reported_epoch":554,"state":"active+clean","last_fresh":"2024-03-11T21:51:07.586852+0000","last_change":"2024-03-11T21:50:20.194234+0000","last_active":"2024-03-11T21:51:07.586852+0000","last_peered":"2024-03-11T21:51:07.586852+0000","last_clean":"2024-03-11T21:51:07.586852+0000","last_became_active":"2024-03-11T21:49:26.643974+0000","last_became_peered":"2024-03-11T21:49:26.643974+0000","last_unstale":"2024-03-11T21:51:07.586852+0000","last_undegraded":"2024-03-11T21:51:07.586852+0000","last_fullsized":"2024-03-11T21:51:07.586852+0000","mapping_epoch":532,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":533,"parent":"0.0","parent_split_bits":0,"last_scrub":"533'13","last_scrub_stamp":"2024-03-11T21:50:20.194127+0000","last_deep_scrub":"94'9","last_deep_scrub_stamp":"2024-03-11T21:26:53.422552+0000","last_clean_scrub_stamp":"2024-03-11T21:50:20.194127+0000","objects_scrubbed":1,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:51:36.985515+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":1531,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":7,"num_read_kb":7,"num_write":10,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":17,"num_bytes_recovered":3207,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":4593,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":6,"ondisk_log_size":6,"up":96,"acting":96,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":198,"num_read_kb":294,"num_write":179,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":2951840,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":2375680,"data_stored":2361472,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":128,"ondisk_log_size":128,"up":3,"acting":3,"num_store_stats":4},{"poolid":3,"num_pg":3,"stat_sum":{"num_bytes":408,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1370,"num_read_kb":1365,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":38,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1224,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":8,"ondisk_log_size":8,"up":9,"acting":9,"num_store_stats":8},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":3702,"num_objects":178,"num_object_clones":0,"num_object_copies":534,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":178,"num_whiteouts":0,"num_read":1535,"num_read_kb":1292,"num_write":989,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":563,"num_bytes_recovered":11116,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":417792,"data_stored":11106,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1101,"ondisk_log_size":1101,"up":96,"acting":96,"num_store_stats":8},{"poolid":5,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":102,"ondisk_log_size":102,"up":96,"acting":96,"num_store_stats":8},{"poolid":6,"num_pg":32,"stat_sum":{"num_bytes":383,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":1162,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1149,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":4,"ondisk_log_size":4,"up":96,"acting":96,"num_store_stats":8}],"osd_stats":[{"osd":7,"up_from":60,"seq":257698038071,"num_pgs":55,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":866844,"kb_used_data":1300,"kb_used_omap":1,"kb_used_meta":865534,"kb_avail":92878308,"statfs":{"total":95995035648,"available":95107387392,"internally_reserved":0,"allocated":1331200,"data_stored":955982,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":886307274},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":50,"seq":214748365117,"num_pgs":43,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":889056,"kb_used_data":1300,"kb_used_omap":1,"kb_used_meta":887742,"kb_avail":92856096,"statfs":{"total":95995035648,"available":95084642304,"internally_reserved":0,"allocated":1331200,"data_stored":956711,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":909048262},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834575190,"num_pgs":52,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1024564,"kb_used_data":744,"kb_used_omap":1,"kb_used_meta":1023806,"kb_avail":92720588,"statfs":{"total":95995035648,"available":94945882112,"internally_reserved":0,"allocated":761856,"data_stored":366231,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":1048377797},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738715,"num_pgs":49,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1011244,"kb_used_data":1312,"kb_used_omap":1,"kb_used_meta":1009918,"kb_avail":92733908,"statfs":{"total":95995035648,"available":94959521792,"internally_reserved":0,"allocated":1343488,"data_stored":956745,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":1034156485},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411665,"num_pgs":42,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":880156,"kb_used_data":724,"kb_used_omap":1,"kb_used_meta":879422,"kb_avail":92864996,"statfs":{"total":95995035648,"available":95093755904,"internally_reserved":0,"allocated":741376,"data_stored":366331,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":900528581},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":277,"seq":1189705941169,"num_pgs":54,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":439796,"kb_used_data":744,"kb_used_omap":4,"kb_used_meta":439035,"kb_avail":93305356,"statfs":{"total":95995035648,"available":95544684544,"internally_reserved":0,"allocated":761856,"data_stored":366565,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4442,"internal_metadata":449572518},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":32,"seq":137438953798,"num_pgs":53,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":663592,"kb_used_data":736,"kb_used_omap":1,"kb_used_meta":662846,"kb_avail":93081560,"statfs":{"total":95995035648,"available":95315517440,"internally_reserved":0,"allocated":753664,"data_stored":365959,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":678754757},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":41,"seq":176093659457,"num_pgs":48,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":875992,"kb_used_data":720,"kb_used_omap":1,"kb_used_meta":875262,"kb_avail":92869160,"statfs":{"total":95995035648,"available":95098019840,"internally_reserved":0,"allocated":737280,"data_stored":365836,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":896268741},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1049,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":46,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":436,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":957,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":574,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":436,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1003,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":92,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":1430,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":1898,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":1238,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":1678,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":1474,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":1100,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":1034,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":1254,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":370,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":383,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":370,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-11T21:51:15.583 DEBUG:teuthology.orchestra.run.smithi092:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid 00db0016-dfed-11ee-95c8-87774f69a715 -- ceph pg dump --format=json 2024-03-11T21:51:15.828 INFO:teuthology.orchestra.run.smithi092.stderr:Inferring config /var/lib/ceph/00db0016-dfed-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T21:51:15.870 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:51:15 smithi092 ceph-mon[29572]: from='client.36332 v1:172.21.15.92:0/1132183736' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-11T21:51:15.870 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:51:15 smithi092 ceph-mon[29572]: 6.14 scrub starts 2024-03-11T21:51:15.870 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:51:15 smithi092 ceph-mon[29572]: 6.14 scrub ok 2024-03-11T21:51:15.870 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:51:15 smithi092 ceph-mon[37563]: from='client.36332 v1:172.21.15.92:0/1132183736' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-11T21:51:15.871 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:51:15 smithi092 ceph-mon[37563]: 6.14 scrub starts 2024-03-11T21:51:15.871 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:51:15 smithi092 ceph-mon[37563]: 6.14 scrub ok 2024-03-11T21:51:16.086 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:51:15 smithi097 ceph-mon[33317]: from='client.36332 v1:172.21.15.92:0/1132183736' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-11T21:51:16.086 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:51:15 smithi097 ceph-mon[33317]: 6.14 scrub starts 2024-03-11T21:51:16.086 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:51:15 smithi097 ceph-mon[33317]: 6.14 scrub ok 2024-03-11T21:51:16.973 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:51:16 smithi092 ceph-mon[29572]: pgmap v1217: 132 pgs: 132 active+clean; 582 KiB data, 6.3 GiB used, 709 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-11T21:51:16.973 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:51:16 smithi092 ceph-mon[29572]: 4.e scrub starts 2024-03-11T21:51:16.973 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:51:16 smithi092 ceph-mon[29572]: 4.e scrub ok 2024-03-11T21:51:16.973 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:51:16 smithi092 ceph-mon[37563]: pgmap v1217: 132 pgs: 132 active+clean; 582 KiB data, 6.3 GiB used, 709 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-11T21:51:16.974 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:51:16 smithi092 ceph-mon[37563]: 4.e scrub starts 2024-03-11T21:51:16.974 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:51:16 smithi092 ceph-mon[37563]: 4.e scrub ok 2024-03-11T21:51:17.086 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:51:16 smithi097 ceph-mon[33317]: pgmap v1217: 132 pgs: 132 active+clean; 582 KiB data, 6.3 GiB used, 709 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-03-11T21:51:17.086 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:51:16 smithi097 ceph-mon[33317]: 4.e scrub starts 2024-03-11T21:51:17.086 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:51:16 smithi097 ceph-mon[33317]: 4.e scrub ok 2024-03-11T21:51:17.733 INFO:teuthology.orchestra.run.smithi092.stdout: 2024-03-11T21:51:17.738 INFO:teuthology.orchestra.run.smithi092.stderr:dumped all 2024-03-11T21:51:18.086 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:51:17 smithi097 ceph-mon[33317]: 2.c scrub starts 2024-03-11T21:51:18.086 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:51:17 smithi097 ceph-mon[33317]: 2.c scrub ok 2024-03-11T21:51:18.086 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:51:17 smithi097 ceph-mon[33317]: 6.e scrub starts 2024-03-11T21:51:18.086 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:51:17 smithi097 ceph-mon[33317]: 6.e scrub ok 2024-03-11T21:51:18.113 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:51:17 smithi092 ceph-mon[29572]: 2.c scrub starts 2024-03-11T21:51:18.114 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:51:17 smithi092 ceph-mon[29572]: 2.c scrub ok 2024-03-11T21:51:18.114 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:51:17 smithi092 ceph-mon[29572]: 6.e scrub starts 2024-03-11T21:51:18.114 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:51:17 smithi092 ceph-mon[29572]: 6.e scrub ok 2024-03-11T21:51:18.114 INFO:journalctl@ceph.mgr.y.smithi092.stdout:Mar 11 21:51:17 smithi092 ceph-00db0016-dfed-11ee-95c8-87774f69a715-mgr-y[29801]: ::ffff:172.21.15.97 - - [11/Mar/2024:21:51:17] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-03-11T21:51:18.114 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:51:17 smithi092 ceph-mon[37563]: 2.c scrub starts 2024-03-11T21:51:18.114 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:51:17 smithi092 ceph-mon[37563]: 2.c scrub ok 2024-03-11T21:51:18.114 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:51:17 smithi092 ceph-mon[37563]: 6.e scrub starts 2024-03-11T21:51:18.115 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:51:17 smithi092 ceph-mon[37563]: 6.e scrub ok 2024-03-11T21:51:18.569 INFO:teuthology.orchestra.run.smithi092.stdout:{"pg_ready":true,"pg_map":{"version":1218,"stamp":"2024-03-11T21:51:17.562341+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":596392,"num_objects":198,"num_object_clones":0,"num_object_copies":594,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":198,"num_whiteouts":0,"num_read":3115,"num_read_kb":2963,"num_write":1187,"num_write_kb":2461,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":618,"num_bytes_recovered":2967363,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1349,"ondisk_log_size":1349,"up":396,"acting":396,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":396,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":749961216,"kb_used":6651232,"kb_used_data":7580,"kb_used_omap":15,"kb_used_meta":6643568,"kb_avail":743309984,"statfs":{"total":767960285184,"available":761149423616,"internally_reserved":0,"allocated":7761920,"data_stored":4700360,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":15601,"internal_metadata":6803014415},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":10,"num_read_kb":10,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"10.840689"},"pg_stats":[{"pgid":"6.1b","version":"0'0","reported_seq":69,"reported_epoch":554,"state":"active+clean","last_fresh":"2024-03-11T21:51:07.586557+0000","last_change":"2024-03-11T21:49:56.669997+0000","last_active":"2024-03-11T21:51:07.586557+0000","last_peered":"2024-03-11T21:51:07.586557+0000","last_clean":"2024-03-11T21:51:07.586557+0000","last_became_active":"2024-03-11T21:49:17.142926+0000","last_became_peered":"2024-03-11T21:49:17.142926+0000","last_unstale":"2024-03-11T21:51:07.586557+0000","last_undegraded":"2024-03-11T21:51:07.586557+0000","last_fullsized":"2024-03-11T21:51:07.586557+0000","mapping_epoch":528,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":529,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:49:56.669893+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:43.559452+0000","last_clean_scrub_stamp":"2024-03-11T21:49:56.669893+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:51:14.065489+0000","scrub_duration":110,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":61,"reported_epoch":554,"state":"active+clean","last_fresh":"2024-03-11T21:51:06.722019+0000","last_change":"2024-03-11T21:50:14.043813+0000","last_active":"2024-03-11T21:51:06.722019+0000","last_peered":"2024-03-11T21:51:06.722019+0000","last_clean":"2024-03-11T21:51:06.722019+0000","last_became_active":"2024-03-11T21:49:26.642290+0000","last_became_peered":"2024-03-11T21:49:26.642290+0000","last_unstale":"2024-03-11T21:51:06.722019+0000","last_undegraded":"2024-03-11T21:51:06.722019+0000","last_fullsized":"2024-03-11T21:51:06.722019+0000","mapping_epoch":532,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":533,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:50:14.043668+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:41.544550+0000","last_clean_scrub_stamp":"2024-03-11T21:50:14.043668+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:51:21.827740+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1f","version":"0'0","reported_seq":1101,"reported_epoch":554,"state":"active+clean","last_fresh":"2024-03-11T21:51:06.722055+0000","last_change":"2024-03-11T21:50:26.304441+0000","last_active":"2024-03-11T21:51:06.722055+0000","last_peered":"2024-03-11T21:51:06.722055+0000","last_clean":"2024-03-11T21:51:06.722055+0000","last_became_active":"2024-03-11T21:49:26.643480+0000","last_became_peered":"2024-03-11T21:49:26.643480+0000","last_unstale":"2024-03-11T21:51:06.722055+0000","last_undegraded":"2024-03-11T21:51:06.722055+0000","last_fullsized":"2024-03-11T21:51:06.722055+0000","mapping_epoch":532,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":533,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:50:26.304337+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:37.512018+0000","last_clean_scrub_stamp":"2024-03-11T21:50:26.304337+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:51:40.664486+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,4],"acting":[0,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.19","version":"456'39","reported_seq":85,"reported_epoch":554,"state":"active+clean","last_fresh":"2024-03-11T21:51:07.586290+0000","last_change":"2024-03-11T21:50:10.491803+0000","last_active":"2024-03-11T21:51:07.586290+0000","last_peered":"2024-03-11T21:51:07.586290+0000","last_clean":"2024-03-11T21:51:07.586290+0000","last_became_active":"2024-03-11T21:49:17.158133+0000","last_became_peered":"2024-03-11T21:49:17.158133+0000","last_unstale":"2024-03-11T21:51:07.586290+0000","last_undegraded":"2024-03-11T21:51:07.586290+0000","last_fullsized":"2024-03-11T21:51:07.586290+0000","mapping_epoch":528,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":529,"parent":"0.0","parent_split_bits":0,"last_scrub":"456'39","last_scrub_stamp":"2024-03-11T21:50:10.491705+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:39.525554+0000","last_clean_scrub_stamp":"2024-03-11T21:50:10.491705+0000","objects_scrubbed":7,"log_size":39,"log_dups_size":0,"ondisk_log_size":39,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:51:20.845568+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":60,"num_read_kb":51,"num_write":40,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.1a","version":"0'0","reported_seq":61,"reported_epoch":554,"state":"active+clean","last_fresh":"2024-03-11T21:51:06.722495+0000","last_change":"2024-03-11T21:50:39.258564+0000","last_active":"2024-03-11T21:51:06.722495+0000","last_peered":"2024-03-11T21:51:06.722495+0000","last_clean":"2024-03-11T21:51:06.722495+0000","last_became_active":"2024-03-11T21:49:26.646919+0000","last_became_peered":"2024-03-11T21:49:26.646919+0000","last_unstale":"2024-03-11T21:51:06.722495+0000","last_undegraded":"2024-03-11T21:51:06.722495+0000","last_fullsized":"2024-03-11T21:51:06.722495+0000","mapping_epoch":532,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":533,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:50:39.258444+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:49:02.060422+0000","last_clean_scrub_stamp":"2024-03-11T21:50:39.258444+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:51:39.565559+0000","scrub_duration":42,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,1],"acting":[4,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":956,"reported_epoch":554,"state":"active+clean","last_fresh":"2024-03-11T21:51:06.961417+0000","last_change":"2024-03-11T21:50:57.005135+0000","last_active":"2024-03-11T21:51:06.961417+0000","last_peered":"2024-03-11T21:51:06.961417+0000","last_clean":"2024-03-11T21:51:06.961417+0000","last_became_active":"2024-03-11T21:49:46.949177+0000","last_became_peered":"2024-03-11T21:49:46.949177+0000","last_unstale":"2024-03-11T21:51:06.961417+0000","last_undegraded":"2024-03-11T21:51:06.961417+0000","last_fullsized":"2024-03-11T21:51:06.961417+0000","mapping_epoch":537,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":538,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:50:57.005051+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:49:40.979721+0000","last_clean_scrub_stamp":"2024-03-11T21:50:57.005051+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:52:16.657490+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,7],"acting":[1,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1e","version":"0'0","reported_seq":77,"reported_epoch":554,"state":"active+clean","last_fresh":"2024-03-11T21:51:07.586910+0000","last_change":"2024-03-11T21:50:58.822295+0000","last_active":"2024-03-11T21:51:07.586910+0000","last_peered":"2024-03-11T21:51:07.586910+0000","last_clean":"2024-03-11T21:51:07.586910+0000","last_became_active":"2024-03-11T21:49:17.157153+0000","last_became_peered":"2024-03-11T21:49:17.157153+0000","last_unstale":"2024-03-11T21:51:07.586910+0000","last_undegraded":"2024-03-11T21:51:07.586910+0000","last_fullsized":"2024-03-11T21:51:07.586910+0000","mapping_epoch":528,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":529,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:50:58.822186+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:37.512018+0000","last_clean_scrub_stamp":"2024-03-11T21:50:58.822186+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:52:21.020159+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.18","version":"456'21","reported_seq":71,"reported_epoch":554,"state":"active+clean","last_fresh":"2024-03-11T21:51:06.721956+0000","last_change":"2024-03-11T21:50:42.147723+0000","last_active":"2024-03-11T21:51:06.721956+0000","last_peered":"2024-03-11T21:51:06.721956+0000","last_clean":"2024-03-11T21:51:06.721956+0000","last_became_active":"2024-03-11T21:49:26.657847+0000","last_became_peered":"2024-03-11T21:49:26.657847+0000","last_unstale":"2024-03-11T21:51:06.721956+0000","last_undegraded":"2024-03-11T21:51:06.721956+0000","last_fullsized":"2024-03-11T21:51:06.721956+0000","mapping_epoch":532,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":533,"parent":"0.0","parent_split_bits":0,"last_scrub":"456'21","last_scrub_stamp":"2024-03-11T21:50:42.147616+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:39.525554+0000","last_clean_scrub_stamp":"2024-03-11T21:50:42.147616+0000","objects_scrubbed":4,"log_size":21,"log_dups_size":0,"ondisk_log_size":21,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:51:52.391955+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":33,"num_read_kb":27,"num_write":22,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":136,"reported_epoch":554,"state":"active+clean","last_fresh":"2024-03-11T21:51:07.481565+0000","last_change":"2024-03-11T21:50:44.904094+0000","last_active":"2024-03-11T21:51:07.481565+0000","last_peered":"2024-03-11T21:51:07.481565+0000","last_clean":"2024-03-11T21:51:07.481565+0000","last_became_active":"2024-03-11T21:49:17.146172+0000","last_became_peered":"2024-03-11T21:49:17.146172+0000","last_unstale":"2024-03-11T21:51:07.481565+0000","last_undegraded":"2024-03-11T21:51:07.481565+0000","last_fullsized":"2024-03-11T21:51:07.481565+0000","mapping_epoch":528,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":529,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:50:44.903949+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:49:44.002008+0000","last_clean_scrub_stamp":"2024-03-11T21:50:44.903949+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:51:45.592855+0000","scrub_duration":80,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3],"acting":[5,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":729,"reported_epoch":554,"state":"active+clean","last_fresh":"2024-03-11T21:51:07.587143+0000","last_change":"2024-03-11T21:51:04.451193+0000","last_active":"2024-03-11T21:51:07.587143+0000","last_peered":"2024-03-11T21:51:07.587143+0000","last_clean":"2024-03-11T21:51:07.587143+0000","last_became_active":"2024-03-11T21:49:26.644138+0000","last_became_peered":"2024-03-11T21:49:26.644138+0000","last_unstale":"2024-03-11T21:51:07.587143+0000","last_undegraded":"2024-03-11T21:51:07.587143+0000","last_fullsized":"2024-03-11T21:51:07.587143+0000","mapping_epoch":532,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":533,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:51:04.451016+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:51:04.451016+0000","last_clean_scrub_stamp":"2024-03-11T21:51:04.451016+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:52:26.827639+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.1d","version":"0'0","reported_seq":1081,"reported_epoch":554,"state":"active+clean","last_fresh":"2024-03-11T21:51:07.587778+0000","last_change":"2024-03-11T21:50:57.457890+0000","last_active":"2024-03-11T21:51:07.587778+0000","last_peered":"2024-03-11T21:51:07.587778+0000","last_clean":"2024-03-11T21:51:07.587778+0000","last_became_active":"2024-03-11T21:25:38.542628+0000","last_became_peered":"2024-03-11T21:25:38.542628+0000","last_unstale":"2024-03-11T21:51:07.587778+0000","last_undegraded":"2024-03-11T21:51:07.587778+0000","last_fullsized":"2024-03-11T21:51:07.587778+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:50:57.457794+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:37.512018+0000","last_clean_scrub_stamp":"2024-03-11T21:50:57.457794+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:52:21.140653+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,0],"acting":[7,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.1b","version":"456'9","reported_seq":67,"reported_epoch":554,"state":"active+clean","last_fresh":"2024-03-11T21:51:06.722321+0000","last_change":"2024-03-11T21:50:31.115835+0000","last_active":"2024-03-11T21:51:06.722321+0000","last_peered":"2024-03-11T21:51:06.722321+0000","last_clean":"2024-03-11T21:51:06.722321+0000","last_became_active":"2024-03-11T21:49:26.639069+0000","last_became_peered":"2024-03-11T21:49:26.639069+0000","last_unstale":"2024-03-11T21:51:06.722321+0000","last_undegraded":"2024-03-11T21:51:06.722321+0000","last_fullsized":"2024-03-11T21:51:06.722321+0000","mapping_epoch":532,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":533,"parent":"0.0","parent_split_bits":0,"last_scrub":"456'9","last_scrub_stamp":"2024-03-11T21:50:31.115727+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:39.525554+0000","last_clean_scrub_stamp":"2024-03-11T21:50:31.115727+0000","objects_scrubbed":2,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:51:33.406175+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":16,"num_read_kb":12,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,1],"acting":[4,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":1055,"reported_epoch":554,"state":"active+clean","last_fresh":"2024-03-11T21:51:06.720888+0000","last_change":"2024-03-11T21:50:44.360542+0000","last_active":"2024-03-11T21:51:06.720888+0000","last_peered":"2024-03-11T21:51:06.720888+0000","last_clean":"2024-03-11T21:51:06.720888+0000","last_became_active":"2024-03-11T21:25:44.589786+0000","last_became_peered":"2024-03-11T21:25:44.589786+0000","last_unstale":"2024-03-11T21:51:06.720888+0000","last_undegraded":"2024-03-11T21:51:06.720888+0000","last_fullsized":"2024-03-11T21:51:06.720888+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:50:44.360459+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:43.559452+0000","last_clean_scrub_stamp":"2024-03-11T21:50:44.360459+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:51:52.384583+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,7],"acting":[0,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":130,"reported_epoch":554,"state":"active+clean","last_fresh":"2024-03-11T21:51:07.481812+0000","last_change":"2024-03-11T21:51:02.647015+0000","last_active":"2024-03-11T21:51:07.481812+0000","last_peered":"2024-03-11T21:51:07.481812+0000","last_clean":"2024-03-11T21:51:07.481812+0000","last_became_active":"2024-03-11T21:48:28.644231+0000","last_became_peered":"2024-03-11T21:48:28.644231+0000","last_unstale":"2024-03-11T21:51:07.481812+0000","last_undegraded":"2024-03-11T21:51:07.481812+0000","last_fullsized":"2024-03-11T21:51:07.481812+0000","mapping_epoch":502,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":503,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:51:02.646863+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:51:02.646863+0000","last_clean_scrub_stamp":"2024-03-11T21:51:02.646863+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:52:20.449867+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,7],"acting":[5,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1c","version":"0'0","reported_seq":1085,"reported_epoch":554,"state":"active+clean","last_fresh":"2024-03-11T21:51:07.587566+0000","last_change":"2024-03-11T21:50:06.133038+0000","last_active":"2024-03-11T21:51:07.587566+0000","last_peered":"2024-03-11T21:51:07.587566+0000","last_clean":"2024-03-11T21:51:07.587566+0000","last_became_active":"2024-03-11T21:48:28.628707+0000","last_became_peered":"2024-03-11T21:48:28.628707+0000","last_unstale":"2024-03-11T21:51:07.587566+0000","last_undegraded":"2024-03-11T21:51:07.587566+0000","last_fullsized":"2024-03-11T21:51:07.587566+0000","mapping_epoch":502,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":503,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:50:06.132933+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:28:32.566437+0000","last_clean_scrub_stamp":"2024-03-11T21:50:06.132933+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:51:14.269913+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.1a","version":"456'25","reported_seq":71,"reported_epoch":554,"state":"active+clean","last_fresh":"2024-03-11T21:51:06.722430+0000","last_change":"2024-03-11T21:50:59.258725+0000","last_active":"2024-03-11T21:51:06.722430+0000","last_peered":"2024-03-11T21:51:06.722430+0000","last_clean":"2024-03-11T21:51:06.722430+0000","last_became_active":"2024-03-11T21:49:26.635996+0000","last_became_peered":"2024-03-11T21:49:26.635996+0000","last_unstale":"2024-03-11T21:51:06.722430+0000","last_undegraded":"2024-03-11T21:51:06.722430+0000","last_fullsized":"2024-03-11T21:51:06.722430+0000","mapping_epoch":532,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":533,"parent":"0.0","parent_split_bits":0,"last_scrub":"456'25","last_scrub_stamp":"2024-03-11T21:50:59.258609+0000","last_deep_scrub":"350'19","last_deep_scrub_stamp":"2024-03-11T21:43:25.898708+0000","last_clean_scrub_stamp":"2024-03-11T21:50:59.258609+0000","objects_scrubbed":4,"log_size":25,"log_dups_size":0,"ondisk_log_size":25,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:52:15.787098+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":35,"num_read_kb":29,"num_write":22,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":11,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0],"acting":[4,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1f","version":"0'0","reported_seq":77,"reported_epoch":554,"state":"active+clean","last_fresh":"2024-03-11T21:51:10.121143+0000","last_change":"2024-03-11T21:51:10.121167+0000","last_active":"2024-03-11T21:51:10.121143+0000","last_peered":"2024-03-11T21:51:10.121143+0000","last_clean":"2024-03-11T21:51:10.121143+0000","last_became_active":"2024-03-11T21:49:17.153410+0000","last_became_peered":"2024-03-11T21:49:17.153410+0000","last_unstale":"2024-03-11T21:51:10.121143+0000","last_undegraded":"2024-03-11T21:51:10.121143+0000","last_fullsized":"2024-03-11T21:51:10.121143+0000","mapping_epoch":528,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":529,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:51:10.121057+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:44:31.472397+0000","last_clean_scrub_stamp":"2024-03-11T21:51:10.121057+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:52:33.250445+0000","scrub_duration":150,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1c","version":"0'0","reported_seq":61,"reported_epoch":554,"state":"active+clean","last_fresh":"2024-03-11T21:51:06.722509+0000","last_change":"2024-03-11T21:50:06.044885+0000","last_active":"2024-03-11T21:51:06.722509+0000","last_peered":"2024-03-11T21:51:06.722509+0000","last_clean":"2024-03-11T21:51:06.722509+0000","last_became_active":"2024-03-11T21:49:26.636081+0000","last_became_peered":"2024-03-11T21:49:26.636081+0000","last_unstale":"2024-03-11T21:51:06.722509+0000","last_undegraded":"2024-03-11T21:51:06.722509+0000","last_fullsized":"2024-03-11T21:51:06.722509+0000","mapping_epoch":532,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":533,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:50:06.044783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:43:32.021344+0000","last_clean_scrub_stamp":"2024-03-11T21:50:06.044783+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:51:34.182002+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1b","version":"0'0","reported_seq":69,"reported_epoch":554,"state":"active+clean","last_fresh":"2024-03-11T21:51:07.586466+0000","last_change":"2024-03-11T21:50:19.422317+0000","last_active":"2024-03-11T21:51:07.586466+0000","last_peered":"2024-03-11T21:51:07.586466+0000","last_clean":"2024-03-11T21:51:07.586466+0000","last_became_active":"2024-03-11T21:49:17.146126+0000","last_became_peered":"2024-03-11T21:49:17.146126+0000","last_unstale":"2024-03-11T21:51:07.586466+0000","last_undegraded":"2024-03-11T21:51:07.586466+0000","last_fullsized":"2024-03-11T21:51:07.586466+0000","mapping_epoch":528,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":529,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:50:19.422221+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:34:27.998724+0000","last_clean_scrub_stamp":"2024-03-11T21:50:19.422221+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:51:44.003421+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1d","version":"456'24","reported_seq":99,"reported_epoch":554,"state":"active+clean","last_fresh":"2024-03-11T21:51:07.586668+0000","last_change":"2024-03-11T21:50:12.450286+0000","last_active":"2024-03-11T21:51:07.586668+0000","last_peered":"2024-03-11T21:51:07.586668+0000","last_clean":"2024-03-11T21:51:07.586668+0000","last_became_active":"2024-03-11T21:49:26.650024+0000","last_became_peered":"2024-03-11T21:49:26.650024+0000","last_unstale":"2024-03-11T21:51:07.586668+0000","last_undegraded":"2024-03-11T21:51:07.586668+0000","last_fullsized":"2024-03-11T21:51:07.586668+0000","mapping_epoch":532,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":533,"parent":"0.0","parent_split_bits":0,"last_scrub":"456'24","last_scrub_stamp":"2024-03-11T21:50:12.450186+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:39.525554+0000","last_clean_scrub_stamp":"2024-03-11T21:50:12.450186+0000","objects_scrubbed":5,"log_size":24,"log_dups_size":0,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:51:24.175921+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":42,"num_read_kb":33,"num_write":26,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":30,"num_bytes_recovered":1320,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.1e","version":"0'0","reported_seq":61,"reported_epoch":554,"state":"active+clean","last_fresh":"2024-03-11T21:51:06.723134+0000","last_change":"2024-03-11T21:50:03.044169+0000","last_active":"2024-03-11T21:51:06.723134+0000","last_peered":"2024-03-11T21:51:06.723134+0000","last_clean":"2024-03-11T21:51:06.723134+0000","last_became_active":"2024-03-11T21:49:26.653347+0000","last_became_peered":"2024-03-11T21:49:26.653347+0000","last_unstale":"2024-03-11T21:51:06.723134+0000","last_undegraded":"2024-03-11T21:51:06.723134+0000","last_fullsized":"2024-03-11T21:51:06.723134+0000","mapping_epoch":532,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":533,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:50:03.044014+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:43.559452+0000","last_clean_scrub_stamp":"2024-03-11T21:50:03.044014+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:51:12.981238+0000","scrub_duration":35,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5],"acting":[4,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1d","version":"0'0","reported_seq":1086,"reported_epoch":554,"state":"active+clean","last_fresh":"2024-03-11T21:51:06.961966+0000","last_change":"2024-03-11T21:50:13.997689+0000","last_active":"2024-03-11T21:51:06.961966+0000","last_peered":"2024-03-11T21:51:06.961966+0000","last_clean":"2024-03-11T21:51:06.961966+0000","last_became_active":"2024-03-11T21:49:26.644697+0000","last_became_peered":"2024-03-11T21:49:26.644697+0000","last_unstale":"2024-03-11T21:51:06.961966+0000","last_undegraded":"2024-03-11T21:51:06.961966+0000","last_fullsized":"2024-03-11T21:51:06.961966+0000","mapping_epoch":532,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":533,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:50:13.997608+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:30:18.678531+0000","last_clean_scrub_stamp":"2024-03-11T21:50:13.997608+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:51:43.853121+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1a","version":"0'0","reported_seq":1107,"reported_epoch":554,"state":"active+clean","last_fresh":"2024-03-11T21:51:07.586478+0000","last_change":"2024-03-11T21:50:23.130342+0000","last_active":"2024-03-11T21:51:07.586478+0000","last_peered":"2024-03-11T21:51:07.586478+0000","last_clean":"2024-03-11T21:51:07.586478+0000","last_became_active":"2024-03-11T21:49:26.637141+0000","last_became_peered":"2024-03-11T21:49:26.637141+0000","last_unstale":"2024-03-11T21:51:07.586478+0000","last_undegraded":"2024-03-11T21:51:07.586478+0000","last_fullsized":"2024-03-11T21:51:07.586478+0000","mapping_epoch":532,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":533,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:50:23.130244+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:37.512018+0000","last_clean_scrub_stamp":"2024-03-11T21:50:23.130244+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:51:49.179992+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.1c","version":"456'39","reported_seq":1190,"reported_epoch":554,"state":"active+clean","last_fresh":"2024-03-11T21:51:06.721412+0000","last_change":"2024-03-11T21:50:00.832261+0000","last_active":"2024-03-11T21:51:06.721412+0000","last_peered":"2024-03-11T21:51:06.721412+0000","last_clean":"2024-03-11T21:51:06.721412+0000","last_became_active":"2024-03-11T21:49:17.153704+0000","last_became_peered":"2024-03-11T21:49:17.153704+0000","last_unstale":"2024-03-11T21:51:06.721412+0000","last_undegraded":"2024-03-11T21:51:06.721412+0000","last_fullsized":"2024-03-11T21:51:06.721412+0000","mapping_epoch":528,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":529,"parent":"0.0","parent_split_bits":0,"last_scrub":"456'39","last_scrub_stamp":"2024-03-11T21:50:00.832150+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:39.525554+0000","last_clean_scrub_stamp":"2024-03-11T21:50:00.832150+0000","objects_scrubbed":7,"log_size":39,"log_dups_size":0,"ondisk_log_size":39,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:51:29.109545+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":62,"num_read_kb":53,"num_write":40,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.1d","version":"0'0","reported_seq":938,"reported_epoch":554,"state":"active+clean","last_fresh":"2024-03-11T21:51:06.961143+0000","last_change":"2024-03-11T21:49:58.218020+0000","last_active":"2024-03-11T21:51:06.961143+0000","last_peered":"2024-03-11T21:51:06.961143+0000","last_clean":"2024-03-11T21:51:06.961143+0000","last_became_active":"2024-03-11T21:49:46.945621+0000","last_became_peered":"2024-03-11T21:49:46.945621+0000","last_unstale":"2024-03-11T21:51:06.961143+0000","last_undegraded":"2024-03-11T21:51:06.961143+0000","last_fullsized":"2024-03-11T21:51:06.961143+0000","mapping_epoch":537,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":538,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:49:58.217912+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:43.559452+0000","last_clean_scrub_stamp":"2024-03-11T21:49:58.217912+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:51:26.736914+0000","scrub_duration":229,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.1e","version":"0'0","reported_seq":1069,"reported_epoch":554,"state":"active+clean","last_fresh":"2024-03-11T21:51:06.722017+0000","last_change":"2024-03-11T21:50:58.397266+0000","last_active":"2024-03-11T21:51:06.722017+0000","last_peered":"2024-03-11T21:51:06.722017+0000","last_clean":"2024-03-11T21:51:06.722017+0000","last_became_active":"2024-03-11T21:25:42.582959+0000","last_became_peered":"2024-03-11T21:25:42.582959+0000","last_unstale":"2024-03-11T21:51:06.722017+0000","last_undegraded":"2024-03-11T21:51:06.722017+0000","last_fullsized":"2024-03-11T21:51:06.722017+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:50:58.397183+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:44:28.222102+0000","last_clean_scrub_stamp":"2024-03-11T21:50:58.397183+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:52:16.575984+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.19","version":"67'2","reported_seq":83,"reported_epoch":554,"state":"active+clean","last_fresh":"2024-03-11T21:51:07.586189+0000","last_change":"2024-03-11T21:50:43.666057+0000","last_active":"2024-03-11T21:51:07.586189+0000","last_peered":"2024-03-11T21:51:07.586189+0000","last_clean":"2024-03-11T21:51:07.586189+0000","last_became_active":"2024-03-11T21:49:17.157291+0000","last_became_peered":"2024-03-11T21:49:17.157291+0000","last_unstale":"2024-03-11T21:51:07.586189+0000","last_undegraded":"2024-03-11T21:51:07.586189+0000","last_fullsized":"2024-03-11T21:51:07.586189+0000","mapping_epoch":528,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":529,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'2","last_scrub_stamp":"2024-03-11T21:50:43.665964+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:37.512018+0000","last_clean_scrub_stamp":"2024-03-11T21:50:43.665964+0000","objects_scrubbed":2,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:52:10.221545+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":957,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":957,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,0],"acting":[3,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1f","version":"456'32","reported_seq":1191,"reported_epoch":554,"state":"active+clean","last_fresh":"2024-03-11T21:51:07.587704+0000","last_change":"2024-03-11T21:50:53.122596+0000","last_active":"2024-03-11T21:51:07.587704+0000","last_peered":"2024-03-11T21:51:07.587704+0000","last_clean":"2024-03-11T21:51:07.587704+0000","last_became_active":"2024-03-11T21:48:28.643699+0000","last_became_peered":"2024-03-11T21:48:28.643699+0000","last_unstale":"2024-03-11T21:51:07.587704+0000","last_undegraded":"2024-03-11T21:51:07.587704+0000","last_fullsized":"2024-03-11T21:51:07.587704+0000","mapping_epoch":502,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":503,"parent":"0.0","parent_split_bits":0,"last_scrub":"456'32","last_scrub_stamp":"2024-03-11T21:50:53.122483+0000","last_deep_scrub":"456'32","last_deep_scrub_stamp":"2024-03-11T21:49:24.792724+0000","last_clean_scrub_stamp":"2024-03-11T21:50:53.122483+0000","objects_scrubbed":5,"log_size":32,"log_dups_size":0,"ondisk_log_size":32,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:51:53.163767+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":44,"num_read_kb":37,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,0],"acting":[6,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.1c","version":"467'2","reported_seq":735,"reported_epoch":554,"state":"active+clean","last_fresh":"2024-03-11T21:51:07.587420+0000","last_change":"2024-03-11T21:51:05.424581+0000","last_active":"2024-03-11T21:51:07.587420+0000","last_peered":"2024-03-11T21:51:07.587420+0000","last_clean":"2024-03-11T21:51:07.587420+0000","last_became_active":"2024-03-11T21:48:28.628768+0000","last_became_peered":"2024-03-11T21:48:28.628768+0000","last_unstale":"2024-03-11T21:51:07.587420+0000","last_undegraded":"2024-03-11T21:51:07.587420+0000","last_fullsized":"2024-03-11T21:51:07.587420+0000","mapping_epoch":502,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":503,"parent":"0.0","parent_split_bits":0,"last_scrub":"467'2","last_scrub_stamp":"2024-03-11T21:51:05.424440+0000","last_deep_scrub":"467'2","last_deep_scrub_stamp":"2024-03-11T21:46:22.517102+0000","last_clean_scrub_stamp":"2024-03-11T21:51:05.424440+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:52:13.086992+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":370,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":1110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.1f","version":"553'13","reported_seq":1453,"reported_epoch":554,"state":"active+clean","last_fresh":"2024-03-11T21:51:07.587703+0000","last_change":"2024-03-11T21:51:03.961974+0000","last_active":"2024-03-11T21:51:07.587703+0000","last_peered":"2024-03-11T21:51:07.587703+0000","last_clean":"2024-03-11T21:51:07.587703+0000","last_became_active":"2024-03-11T21:49:26.637213+0000","last_became_peered":"2024-03-11T21:49:26.637213+0000","last_unstale":"2024-03-11T21:51:07.587703+0000","last_undegraded":"2024-03-11T21:51:07.587703+0000","last_fullsized":"2024-03-11T21:51:07.587703+0000","mapping_epoch":532,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":533,"parent":"0.0","parent_split_bits":0,"last_scrub":"553'13","last_scrub_stamp":"2024-03-11T21:51:03.961840+0000","last_deep_scrub":"553'13","last_deep_scrub_stamp":"2024-03-11T21:51:03.961840+0000","last_clean_scrub_stamp":"2024-03-11T21:51:03.961840+0000","objects_scrubbed":1,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:52:06.143607+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.18","version":"67'1","reported_seq":144,"reported_epoch":554,"state":"active+clean","last_fresh":"2024-03-11T21:51:07.481405+0000","last_change":"2024-03-11T21:50:14.934032+0000","last_active":"2024-03-11T21:51:07.481405+0000","last_peered":"2024-03-11T21:51:07.481405+0000","last_clean":"2024-03-11T21:51:07.481405+0000","last_became_active":"2024-03-11T21:49:26.650243+0000","last_became_peered":"2024-03-11T21:49:26.650243+0000","last_unstale":"2024-03-11T21:51:07.481405+0000","last_undegraded":"2024-03-11T21:51:07.481405+0000","last_fullsized":"2024-03-11T21:51:07.481405+0000","mapping_epoch":532,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":533,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'1","last_scrub_stamp":"2024-03-11T21:50:14.933934+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:37.512018+0000","last_clean_scrub_stamp":"2024-03-11T21:50:14.933934+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:51:39.849515+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":436,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":7,"num_read_kb":7,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1744,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,2],"acting":[5,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1e","version":"456'30","reported_seq":1204,"reported_epoch":554,"state":"active+clean","last_fresh":"2024-03-11T21:51:06.722015+0000","last_change":"2024-03-11T21:50:09.119727+0000","last_active":"2024-03-11T21:51:06.722015+0000","last_peered":"2024-03-11T21:51:06.722015+0000","last_clean":"2024-03-11T21:51:06.722015+0000","last_became_active":"2024-03-11T21:49:17.148991+0000","last_became_peered":"2024-03-11T21:49:17.148991+0000","last_unstale":"2024-03-11T21:51:06.722015+0000","last_undegraded":"2024-03-11T21:51:06.722015+0000","last_fullsized":"2024-03-11T21:51:06.722015+0000","mapping_epoch":528,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":529,"parent":"0.0","parent_split_bits":0,"last_scrub":"456'30","last_scrub_stamp":"2024-03-11T21:50:09.119582+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:39.525554+0000","last_clean_scrub_stamp":"2024-03-11T21:50:09.119582+0000","objects_scrubbed":5,"log_size":30,"log_dups_size":0,"ondisk_log_size":30,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:51:25.770663+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":45,"num_read_kb":40,"num_write":30,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.13","version":"0'0","reported_seq":69,"reported_epoch":554,"state":"active+clean","last_fresh":"2024-03-11T21:51:07.586676+0000","last_change":"2024-03-11T21:50:21.523564+0000","last_active":"2024-03-11T21:51:07.586676+0000","last_peered":"2024-03-11T21:51:07.586676+0000","last_clean":"2024-03-11T21:51:07.586676+0000","last_became_active":"2024-03-11T21:49:17.154298+0000","last_became_peered":"2024-03-11T21:49:17.154298+0000","last_unstale":"2024-03-11T21:51:07.586676+0000","last_undegraded":"2024-03-11T21:51:07.586676+0000","last_fullsized":"2024-03-11T21:51:07.586676+0000","mapping_epoch":528,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":529,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:50:21.523443+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:43.559452+0000","last_clean_scrub_stamp":"2024-03-11T21:50:21.523443+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:51:39.157450+0000","scrub_duration":119,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":731,"reported_epoch":554,"state":"active+clean","last_fresh":"2024-03-11T21:51:07.586651+0000","last_change":"2024-03-11T21:50:49.413889+0000","last_active":"2024-03-11T21:51:07.586651+0000","last_peered":"2024-03-11T21:51:07.586651+0000","last_clean":"2024-03-11T21:51:07.586651+0000","last_became_active":"2024-03-11T21:49:26.643590+0000","last_became_peered":"2024-03-11T21:49:26.643590+0000","last_unstale":"2024-03-11T21:51:07.586651+0000","last_undegraded":"2024-03-11T21:51:07.586651+0000","last_fullsized":"2024-03-11T21:51:07.586651+0000","mapping_epoch":532,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":533,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:50:49.413780+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:49:34.678322+0000","last_clean_scrub_stamp":"2024-03-11T21:50:49.413780+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:52:18.899709+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.17","version":"0'0","reported_seq":1095,"reported_epoch":554,"state":"active+clean","last_fresh":"2024-03-11T21:51:07.587538+0000","last_change":"2024-03-11T21:50:42.135239+0000","last_active":"2024-03-11T21:51:07.587538+0000","last_peered":"2024-03-11T21:51:07.587538+0000","last_clean":"2024-03-11T21:51:07.587538+0000","last_became_active":"2024-03-11T21:48:28.623533+0000","last_became_peered":"2024-03-11T21:48:28.623533+0000","last_unstale":"2024-03-11T21:51:07.587538+0000","last_undegraded":"2024-03-11T21:51:07.587538+0000","last_fullsized":"2024-03-11T21:51:07.587538+0000","mapping_epoch":502,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":503,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:50:42.135139+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:50:42.135139+0000","last_clean_scrub_stamp":"2024-03-11T21:50:42.135139+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:51:45.993396+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,2],"acting":[6,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.11","version":"456'27","reported_seq":83,"reported_epoch":554,"state":"active+clean","last_fresh":"2024-03-11T21:51:07.481405+0000","last_change":"2024-03-11T21:50:52.714482+0000","last_active":"2024-03-11T21:51:07.481405+0000","last_peered":"2024-03-11T21:51:07.481405+0000","last_clean":"2024-03-11T21:51:07.481405+0000","last_became_active":"2024-03-11T21:49:25.632411+0000","last_became_peered":"2024-03-11T21:49:25.632411+0000","last_unstale":"2024-03-11T21:51:07.481405+0000","last_undegraded":"2024-03-11T21:51:07.481405+0000","last_fullsized":"2024-03-11T21:51:07.481405+0000","mapping_epoch":531,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":532,"parent":"0.0","parent_split_bits":0,"last_scrub":"456'27","last_scrub_stamp":"2024-03-11T21:50:52.714351+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:39.525554+0000","last_clean_scrub_stamp":"2024-03-11T21:50:52.714351+0000","objects_scrubbed":5,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:52:13.665274+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":44,"num_read_kb":37,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25,"num_bytes_recovered":550,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":737,"reported_epoch":554,"state":"active+clean","last_fresh":"2024-03-11T21:51:09.430134+0000","last_change":"2024-03-11T21:51:09.430185+0000","last_active":"2024-03-11T21:51:09.430134+0000","last_peered":"2024-03-11T21:51:09.430134+0000","last_clean":"2024-03-11T21:51:09.430134+0000","last_became_active":"2024-03-11T21:49:26.644022+0000","last_became_peered":"2024-03-11T21:49:26.644022+0000","last_unstale":"2024-03-11T21:51:09.430134+0000","last_undegraded":"2024-03-11T21:51:09.430134+0000","last_fullsized":"2024-03-11T21:51:09.430134+0000","mapping_epoch":532,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":533,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:51:09.430039+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:43.559452+0000","last_clean_scrub_stamp":"2024-03-11T21:51:09.430039+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:52:22.974534+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,4],"acting":[7,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":1107,"reported_epoch":554,"state":"active+clean","last_fresh":"2024-03-11T21:51:07.586434+0000","last_change":"2024-03-11T21:50:22.136299+0000","last_active":"2024-03-11T21:51:07.586434+0000","last_peered":"2024-03-11T21:51:07.586434+0000","last_clean":"2024-03-11T21:51:07.586434+0000","last_became_active":"2024-03-11T21:49:26.638240+0000","last_became_peered":"2024-03-11T21:49:26.638240+0000","last_unstale":"2024-03-11T21:51:07.586434+0000","last_undegraded":"2024-03-11T21:51:07.586434+0000","last_fullsized":"2024-03-11T21:51:07.586434+0000","mapping_epoch":532,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":533,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:50:22.136209+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:50:22.136209+0000","last_clean_scrub_stamp":"2024-03-11T21:50:22.136209+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:51:25.274309+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.16","version":"0'0","reported_seq":130,"reported_epoch":554,"state":"active+clean","last_fresh":"2024-03-11T21:51:07.482209+0000","last_change":"2024-03-11T21:50:11.905644+0000","last_active":"2024-03-11T21:51:07.482209+0000","last_peered":"2024-03-11T21:51:07.482209+0000","last_clean":"2024-03-11T21:51:07.482209+0000","last_became_active":"2024-03-11T21:48:28.636823+0000","last_became_peered":"2024-03-11T21:48:28.636823+0000","last_unstale":"2024-03-11T21:51:07.482209+0000","last_undegraded":"2024-03-11T21:51:07.482209+0000","last_fullsized":"2024-03-11T21:51:07.482209+0000","mapping_epoch":502,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":503,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:50:11.905527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:27:11.938909+0000","last_clean_scrub_stamp":"2024-03-11T21:50:11.905527+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:51:20.181506+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,2],"acting":[5,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.10","version":"456'14","reported_seq":83,"reported_epoch":554,"state":"active+clean","last_fresh":"2024-03-11T21:51:07.586053+0000","last_change":"2024-03-11T21:50:40.647648+0000","last_active":"2024-03-11T21:51:07.586053+0000","last_peered":"2024-03-11T21:51:07.586053+0000","last_clean":"2024-03-11T21:51:07.586053+0000","last_became_active":"2024-03-11T21:49:17.132671+0000","last_became_peered":"2024-03-11T21:49:17.132671+0000","last_unstale":"2024-03-11T21:51:07.586053+0000","last_undegraded":"2024-03-11T21:51:07.586053+0000","last_fullsized":"2024-03-11T21:51:07.586053+0000","mapping_epoch":528,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":529,"parent":"0.0","parent_split_bits":0,"last_scrub":"456'14","last_scrub_stamp":"2024-03-11T21:50:40.647495+0000","last_deep_scrub":"260'10","last_deep_scrub_stamp":"2024-03-11T21:44:32.059058+0000","last_clean_scrub_stamp":"2024-03-11T21:50:40.647495+0000","objects_scrubbed":2,"log_size":14,"log_dups_size":0,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:51:45.955329+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":18,"num_read_kb":16,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6],"acting":[3,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":69,"reported_epoch":554,"state":"active+clean","last_fresh":"2024-03-11T21:51:07.586298+0000","last_change":"2024-03-11T21:50:31.635995+0000","last_active":"2024-03-11T21:51:07.586298+0000","last_peered":"2024-03-11T21:51:07.586298+0000","last_clean":"2024-03-11T21:51:07.586298+0000","last_became_active":"2024-03-11T21:49:17.159872+0000","last_became_peered":"2024-03-11T21:49:17.159872+0000","last_unstale":"2024-03-11T21:51:07.586298+0000","last_undegraded":"2024-03-11T21:51:07.586298+0000","last_fullsized":"2024-03-11T21:51:07.586298+0000","mapping_epoch":528,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":529,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:50:31.635877+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:50:31.635877+0000","last_clean_scrub_stamp":"2024-03-11T21:50:31.635877+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:51:58.858039+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":1084,"reported_epoch":554,"state":"active+clean","last_fresh":"2024-03-11T21:51:06.961729+0000","last_change":"2024-03-11T21:49:53.032361+0000","last_active":"2024-03-11T21:51:06.961729+0000","last_peered":"2024-03-11T21:51:06.961729+0000","last_clean":"2024-03-11T21:51:06.961729+0000","last_became_active":"2024-03-11T21:49:17.138352+0000","last_became_peered":"2024-03-11T21:49:17.138352+0000","last_unstale":"2024-03-11T21:51:06.961729+0000","last_undegraded":"2024-03-11T21:51:06.961729+0000","last_fullsized":"2024-03-11T21:51:06.961729+0000","mapping_epoch":528,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":529,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:49:53.032269+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:27:21.207303+0000","last_clean_scrub_stamp":"2024-03-11T21:49:53.032269+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:51:18.975741+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.15","version":"0'0","reported_seq":1087,"reported_epoch":554,"state":"active+clean","last_fresh":"2024-03-11T21:51:06.961781+0000","last_change":"2024-03-11T21:51:03.973908+0000","last_active":"2024-03-11T21:51:06.961781+0000","last_peered":"2024-03-11T21:51:06.961781+0000","last_clean":"2024-03-11T21:51:06.961781+0000","last_became_active":"2024-03-11T21:48:28.620494+0000","last_became_peered":"2024-03-11T21:48:28.620494+0000","last_unstale":"2024-03-11T21:51:06.961781+0000","last_undegraded":"2024-03-11T21:51:06.961781+0000","last_fullsized":"2024-03-11T21:51:06.961781+0000","mapping_epoch":502,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":503,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:51:03.973831+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:26:44.860499+0000","last_clean_scrub_stamp":"2024-03-11T21:51:03.973831+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:52:11.763534+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.13","version":"456'32","reported_seq":81,"reported_epoch":554,"state":"active+clean","last_fresh":"2024-03-11T21:51:06.722453+0000","last_change":"2024-03-11T21:50:53.237986+0000","last_active":"2024-03-11T21:51:06.722453+0000","last_peered":"2024-03-11T21:51:06.722453+0000","last_clean":"2024-03-11T21:51:06.722453+0000","last_became_active":"2024-03-11T21:49:26.637845+0000","last_became_peered":"2024-03-11T21:49:26.637845+0000","last_unstale":"2024-03-11T21:51:06.722453+0000","last_undegraded":"2024-03-11T21:51:06.722453+0000","last_fullsized":"2024-03-11T21:51:06.722453+0000","mapping_epoch":532,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":533,"parent":"0.0","parent_split_bits":0,"last_scrub":"456'32","last_scrub_stamp":"2024-03-11T21:50:53.237819+0000","last_deep_scrub":"260'24","last_deep_scrub_stamp":"2024-03-11T21:40:25.595220+0000","last_clean_scrub_stamp":"2024-03-11T21:50:53.237819+0000","objects_scrubbed":5,"log_size":32,"log_dups_size":0,"ondisk_log_size":32,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:52:01.982237+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":42,"num_read_kb":35,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,2],"acting":[4,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.10","version":"0'0","reported_seq":1067,"reported_epoch":554,"state":"active+clean","last_fresh":"2024-03-11T21:51:06.720392+0000","last_change":"2024-03-11T21:50:46.410870+0000","last_active":"2024-03-11T21:51:06.720392+0000","last_peered":"2024-03-11T21:51:06.720392+0000","last_clean":"2024-03-11T21:51:06.720392+0000","last_became_active":"2024-03-11T21:48:28.641880+0000","last_became_peered":"2024-03-11T21:48:28.641880+0000","last_unstale":"2024-03-11T21:51:06.720392+0000","last_undegraded":"2024-03-11T21:51:06.720392+0000","last_fullsized":"2024-03-11T21:51:06.720392+0000","mapping_epoch":502,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":503,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:50:46.410778+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:49:16.199994+0000","last_clean_scrub_stamp":"2024-03-11T21:50:46.410778+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:51:56.329722+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,1],"acting":[0,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":77,"reported_epoch":554,"state":"active+clean","last_fresh":"2024-03-11T21:51:07.586670+0000","last_change":"2024-03-11T21:50:33.651519+0000","last_active":"2024-03-11T21:51:07.586670+0000","last_peered":"2024-03-11T21:51:07.586670+0000","last_clean":"2024-03-11T21:51:07.586670+0000","last_became_active":"2024-03-11T21:49:17.157943+0000","last_became_peered":"2024-03-11T21:49:17.157943+0000","last_unstale":"2024-03-11T21:51:07.586670+0000","last_undegraded":"2024-03-11T21:51:07.586670+0000","last_fullsized":"2024-03-11T21:51:07.586670+0000","mapping_epoch":528,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":529,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:50:33.651412+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:50:33.651412+0000","last_clean_scrub_stamp":"2024-03-11T21:50:33.651412+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:51:33.993223+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.14","version":"0'0","reported_seq":1132,"reported_epoch":554,"state":"active+clean","last_fresh":"2024-03-11T21:51:07.586935+0000","last_change":"2024-03-11T21:50:51.099964+0000","last_active":"2024-03-11T21:51:07.586935+0000","last_peered":"2024-03-11T21:51:07.586935+0000","last_clean":"2024-03-11T21:51:07.586935+0000","last_became_active":"2024-03-11T21:49:17.133506+0000","last_became_peered":"2024-03-11T21:49:17.133506+0000","last_unstale":"2024-03-11T21:51:07.586935+0000","last_undegraded":"2024-03-11T21:51:07.586935+0000","last_fullsized":"2024-03-11T21:51:07.586935+0000","mapping_epoch":528,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":529,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:50:51.099855+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:37.512018+0000","last_clean_scrub_stamp":"2024-03-11T21:50:51.099855+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:52:09.556377+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,5],"acting":[6,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.12","version":"456'21","reported_seq":1105,"reported_epoch":554,"state":"active+clean","last_fresh":"2024-03-11T21:51:08.907773+0000","last_change":"2024-03-11T21:51:08.907831+0000","last_active":"2024-03-11T21:51:08.907773+0000","last_peered":"2024-03-11T21:51:08.907773+0000","last_clean":"2024-03-11T21:51:08.907773+0000","last_became_active":"2024-03-11T21:25:40.547082+0000","last_became_peered":"2024-03-11T21:25:40.547082+0000","last_unstale":"2024-03-11T21:51:08.907773+0000","last_undegraded":"2024-03-11T21:51:08.907773+0000","last_fullsized":"2024-03-11T21:51:08.907773+0000","mapping_epoch":68,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":69,"parent":"0.0","parent_split_bits":0,"last_scrub":"456'21","last_scrub_stamp":"2024-03-11T21:51:08.907678+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:39.525554+0000","last_clean_scrub_stamp":"2024-03-11T21:51:08.907678+0000","objects_scrubbed":4,"log_size":21,"log_dups_size":0,"ondisk_log_size":21,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:52:30.134024+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":35,"num_read_kb":29,"num_write":22,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":61,"reported_epoch":554,"state":"active+clean","last_fresh":"2024-03-11T21:51:06.723097+0000","last_change":"2024-03-11T21:49:48.995871+0000","last_active":"2024-03-11T21:51:06.723097+0000","last_peered":"2024-03-11T21:51:06.723097+0000","last_clean":"2024-03-11T21:51:06.723097+0000","last_became_active":"2024-03-11T21:49:26.647985+0000","last_became_peered":"2024-03-11T21:49:26.647985+0000","last_unstale":"2024-03-11T21:51:06.723097+0000","last_undegraded":"2024-03-11T21:51:06.723097+0000","last_fullsized":"2024-03-11T21:51:06.723097+0000","mapping_epoch":532,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":533,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:49:48.995734+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:27:09.801138+0000","last_clean_scrub_stamp":"2024-03-11T21:49:48.995734+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":159,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,5],"acting":[4,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.14","version":"537'15","reported_seq":106,"reported_epoch":554,"state":"active+clean","last_fresh":"2024-03-11T21:51:08.895482+0000","last_change":"2024-03-11T21:50:59.847065+0000","last_active":"2024-03-11T21:51:08.895482+0000","last_peered":"2024-03-11T21:51:08.895482+0000","last_clean":"2024-03-11T21:51:08.895482+0000","last_became_active":"2024-03-11T21:49:17.151384+0000","last_became_peered":"2024-03-11T21:49:17.151384+0000","last_unstale":"2024-03-11T21:51:08.895482+0000","last_undegraded":"2024-03-11T21:51:08.895482+0000","last_fullsized":"2024-03-11T21:51:08.895482+0000","mapping_epoch":528,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":529,"parent":"0.0","parent_split_bits":0,"last_scrub":"537'15","last_scrub_stamp":"2024-03-11T21:50:59.846905+0000","last_deep_scrub":"537'15","last_deep_scrub_stamp":"2024-03-11T21:49:45.799879+0000","last_clean_scrub_stamp":"2024-03-11T21:50:59.846905+0000","objects_scrubbed":1,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:52:10.761086+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,2],"acting":[3,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.13","version":"0'0","reported_seq":1112,"reported_epoch":554,"state":"active+clean","last_fresh":"2024-03-11T21:51:06.721148+0000","last_change":"2024-03-11T21:50:55.345249+0000","last_active":"2024-03-11T21:51:06.721148+0000","last_peered":"2024-03-11T21:51:06.721148+0000","last_clean":"2024-03-11T21:51:06.721148+0000","last_became_active":"2024-03-11T21:49:26.643686+0000","last_became_peered":"2024-03-11T21:49:26.643686+0000","last_unstale":"2024-03-11T21:51:06.721148+0000","last_undegraded":"2024-03-11T21:51:06.721148+0000","last_fullsized":"2024-03-11T21:51:06.721148+0000","mapping_epoch":532,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":533,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:50:55.345155+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:37.512018+0000","last_clean_scrub_stamp":"2024-03-11T21:50:55.345155+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:52:13.680891+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.15","version":"456'25","reported_seq":172,"reported_epoch":554,"state":"active+clean","last_fresh":"2024-03-11T21:51:07.482175+0000","last_change":"2024-03-11T21:49:48.905906+0000","last_active":"2024-03-11T21:51:07.482175+0000","last_peered":"2024-03-11T21:51:07.482175+0000","last_clean":"2024-03-11T21:51:07.482175+0000","last_became_active":"2024-03-11T21:49:25.629022+0000","last_became_peered":"2024-03-11T21:49:25.629022+0000","last_unstale":"2024-03-11T21:51:07.482175+0000","last_undegraded":"2024-03-11T21:51:07.482175+0000","last_fullsized":"2024-03-11T21:51:07.482175+0000","mapping_epoch":531,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":532,"parent":"0.0","parent_split_bits":0,"last_scrub":"456'25","last_scrub_stamp":"2024-03-11T21:49:48.905801+0000","last_deep_scrub":"94'13","last_deep_scrub_stamp":"2024-03-11T21:29:41.853158+0000","last_clean_scrub_stamp":"2024-03-11T21:49:48.905801+0000","objects_scrubbed":4,"log_size":25,"log_dups_size":0,"ondisk_log_size":25,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:51:18.172354+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":34,"num_read_kb":28,"num_write":22,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":27,"num_bytes_recovered":660,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,2],"acting":[5,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.16","version":"0'0","reported_seq":1114,"reported_epoch":554,"state":"active+clean","last_fresh":"2024-03-11T21:51:06.721598+0000","last_change":"2024-03-11T21:50:50.407683+0000","last_active":"2024-03-11T21:51:06.721598+0000","last_peered":"2024-03-11T21:51:06.721598+0000","last_clean":"2024-03-11T21:51:06.721598+0000","last_became_active":"2024-03-11T21:49:17.147073+0000","last_became_peered":"2024-03-11T21:49:17.147073+0000","last_unstale":"2024-03-11T21:51:06.721598+0000","last_undegraded":"2024-03-11T21:51:06.721598+0000","last_fullsized":"2024-03-11T21:51:06.721598+0000","mapping_epoch":528,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":529,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:50:50.407607+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:50:50.407607+0000","last_clean_scrub_stamp":"2024-03-11T21:50:50.407607+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:52:19.224933+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.15","version":"503'12","reported_seq":168,"reported_epoch":554,"state":"active+clean","last_fresh":"2024-03-11T21:51:08.895494+0000","last_change":"2024-03-11T21:50:27.013987+0000","last_active":"2024-03-11T21:51:08.895494+0000","last_peered":"2024-03-11T21:51:08.895494+0000","last_clean":"2024-03-11T21:51:08.895494+0000","last_became_active":"2024-03-11T21:48:28.646407+0000","last_became_peered":"2024-03-11T21:48:28.646407+0000","last_unstale":"2024-03-11T21:51:08.895494+0000","last_undegraded":"2024-03-11T21:51:08.895494+0000","last_fullsized":"2024-03-11T21:51:08.895494+0000","mapping_epoch":502,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":503,"parent":"0.0","parent_split_bits":0,"last_scrub":"503'12","last_scrub_stamp":"2024-03-11T21:50:27.013892+0000","last_deep_scrub":"98'9","last_deep_scrub_stamp":"2024-03-11T21:31:20.624923+0000","last_clean_scrub_stamp":"2024-03-11T21:50:27.013892+0000","objects_scrubbed":1,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:51:50.165683+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.12","version":"0'0","reported_seq":144,"reported_epoch":554,"state":"active+clean","last_fresh":"2024-03-11T21:51:08.761997+0000","last_change":"2024-03-11T21:51:08.762020+0000","last_active":"2024-03-11T21:51:08.761997+0000","last_peered":"2024-03-11T21:51:08.761997+0000","last_clean":"2024-03-11T21:51:08.761997+0000","last_became_active":"2024-03-11T21:49:17.146154+0000","last_became_peered":"2024-03-11T21:49:17.146154+0000","last_unstale":"2024-03-11T21:51:08.761997+0000","last_undegraded":"2024-03-11T21:51:08.761997+0000","last_fullsized":"2024-03-11T21:51:08.761997+0000","mapping_epoch":528,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":529,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:51:08.761930+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:50:01.868835+0000","last_clean_scrub_stamp":"2024-03-11T21:51:08.761930+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:52:31.245167+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.14","version":"534'35","reported_seq":94,"reported_epoch":554,"state":"active+clean","last_fresh":"2024-03-11T21:51:07.586548+0000","last_change":"2024-03-11T21:50:56.839722+0000","last_active":"2024-03-11T21:51:07.586548+0000","last_peered":"2024-03-11T21:51:07.586548+0000","last_clean":"2024-03-11T21:51:07.586548+0000","last_became_active":"2024-03-11T21:49:17.151223+0000","last_became_peered":"2024-03-11T21:49:17.151223+0000","last_unstale":"2024-03-11T21:51:07.586548+0000","last_undegraded":"2024-03-11T21:51:07.586548+0000","last_fullsized":"2024-03-11T21:51:07.586548+0000","mapping_epoch":528,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":529,"parent":"0.0","parent_split_bits":0,"last_scrub":"534'35","last_scrub_stamp":"2024-03-11T21:50:56.839627+0000","last_deep_scrub":"534'35","last_deep_scrub_stamp":"2024-03-11T21:49:36.847051+0000","last_clean_scrub_stamp":"2024-03-11T21:50:56.839627+0000","objects_scrubbed":5,"log_size":35,"log_dups_size":0,"ondisk_log_size":35,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:52:20.633021+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":44,"num_read_kb":39,"num_write":30,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.15","version":"0'0","reported_seq":734,"reported_epoch":554,"state":"active+clean","last_fresh":"2024-03-11T21:51:07.587655+0000","last_change":"2024-03-11T21:50:15.289444+0000","last_active":"2024-03-11T21:51:07.587655+0000","last_peered":"2024-03-11T21:51:07.587655+0000","last_clean":"2024-03-11T21:51:07.587655+0000","last_became_active":"2024-03-11T21:49:26.644817+0000","last_became_peered":"2024-03-11T21:49:26.644817+0000","last_unstale":"2024-03-11T21:51:07.587655+0000","last_undegraded":"2024-03-11T21:51:07.587655+0000","last_fullsized":"2024-03-11T21:51:07.587655+0000","mapping_epoch":532,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":533,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:50:15.289313+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:43.559452+0000","last_clean_scrub_stamp":"2024-03-11T21:50:15.289313+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:51:29.118124+0000","scrub_duration":95,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":136,"reported_epoch":554,"state":"active+clean","last_fresh":"2024-03-11T21:51:07.481972+0000","last_change":"2024-03-11T21:50:15.931089+0000","last_active":"2024-03-11T21:51:07.481972+0000","last_peered":"2024-03-11T21:51:07.481972+0000","last_clean":"2024-03-11T21:51:07.481972+0000","last_became_active":"2024-03-11T21:49:17.147086+0000","last_became_peered":"2024-03-11T21:49:17.147086+0000","last_unstale":"2024-03-11T21:51:07.481972+0000","last_undegraded":"2024-03-11T21:51:07.481972+0000","last_fullsized":"2024-03-11T21:51:07.481972+0000","mapping_epoch":528,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":529,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:50:15.930961+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:43:36.585632+0000","last_clean_scrub_stamp":"2024-03-11T21:50:15.930961+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:51:34.383497+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,1],"acting":[5,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.11","version":"0'0","reported_seq":1107,"reported_epoch":554,"state":"active+clean","last_fresh":"2024-03-11T21:51:07.586551+0000","last_change":"2024-03-11T21:50:54.080401+0000","last_active":"2024-03-11T21:51:07.586551+0000","last_peered":"2024-03-11T21:51:07.586551+0000","last_clean":"2024-03-11T21:51:07.586551+0000","last_became_active":"2024-03-11T21:49:26.637633+0000","last_became_peered":"2024-03-11T21:49:26.637633+0000","last_unstale":"2024-03-11T21:51:07.586551+0000","last_undegraded":"2024-03-11T21:51:07.586551+0000","last_fullsized":"2024-03-11T21:51:07.586551+0000","mapping_epoch":532,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":533,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:50:54.080300+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:37.512018+0000","last_clean_scrub_stamp":"2024-03-11T21:50:54.080300+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:52:19.672449+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.17","version":"456'21","reported_seq":77,"reported_epoch":554,"state":"active+clean","last_fresh":"2024-03-11T21:51:07.586907+0000","last_change":"2024-03-11T21:49:58.581189+0000","last_active":"2024-03-11T21:51:07.586907+0000","last_peered":"2024-03-11T21:51:07.586907+0000","last_clean":"2024-03-11T21:51:07.586907+0000","last_became_active":"2024-03-11T21:49:17.141874+0000","last_became_peered":"2024-03-11T21:49:17.141874+0000","last_unstale":"2024-03-11T21:51:07.586907+0000","last_undegraded":"2024-03-11T21:51:07.586907+0000","last_fullsized":"2024-03-11T21:51:07.586907+0000","mapping_epoch":528,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":529,"parent":"0.0","parent_split_bits":0,"last_scrub":"456'21","last_scrub_stamp":"2024-03-11T21:49:58.581084+0000","last_deep_scrub":"97'9","last_deep_scrub_stamp":"2024-03-11T21:27:25.803132+0000","last_clean_scrub_stamp":"2024-03-11T21:49:58.581084+0000","objects_scrubbed":3,"log_size":21,"log_dups_size":0,"ondisk_log_size":21,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:51:21.644085+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":27,"num_read_kb":24,"num_write":18,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.14","version":"0'0","reported_seq":1085,"reported_epoch":554,"state":"active+clean","last_fresh":"2024-03-11T21:51:06.721211+0000","last_change":"2024-03-11T21:50:10.894796+0000","last_active":"2024-03-11T21:51:06.721211+0000","last_peered":"2024-03-11T21:51:06.721211+0000","last_clean":"2024-03-11T21:51:06.721211+0000","last_became_active":"2024-03-11T21:49:26.637667+0000","last_became_peered":"2024-03-11T21:49:26.637667+0000","last_unstale":"2024-03-11T21:51:06.721211+0000","last_undegraded":"2024-03-11T21:51:06.721211+0000","last_fullsized":"2024-03-11T21:51:06.721211+0000","mapping_epoch":532,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":533,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:50:10.894715+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:43.559452+0000","last_clean_scrub_stamp":"2024-03-11T21:50:10.894715+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:51:11.919095+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,7],"acting":[2,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":69,"reported_epoch":554,"state":"active+clean","last_fresh":"2024-03-11T21:51:07.587020+0000","last_change":"2024-03-11T21:50:06.554164+0000","last_active":"2024-03-11T21:51:07.587020+0000","last_peered":"2024-03-11T21:51:07.587020+0000","last_clean":"2024-03-11T21:51:07.587020+0000","last_became_active":"2024-03-11T21:49:17.143360+0000","last_became_peered":"2024-03-11T21:49:17.143360+0000","last_unstale":"2024-03-11T21:51:07.587020+0000","last_undegraded":"2024-03-11T21:51:07.587020+0000","last_fullsized":"2024-03-11T21:51:07.587020+0000","mapping_epoch":528,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":529,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:50:06.554083+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:50:06.554083+0000","last_clean_scrub_stamp":"2024-03-11T21:50:06.554083+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:51:20.135608+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.10","version":"0'0","reported_seq":1081,"reported_epoch":554,"state":"active+clean","last_fresh":"2024-03-11T21:51:06.720450+0000","last_change":"2024-03-11T21:51:05.983502+0000","last_active":"2024-03-11T21:51:06.720450+0000","last_peered":"2024-03-11T21:51:06.720450+0000","last_clean":"2024-03-11T21:51:06.720450+0000","last_became_active":"2024-03-11T21:25:38.533138+0000","last_became_peered":"2024-03-11T21:25:38.533138+0000","last_unstale":"2024-03-11T21:51:06.720450+0000","last_undegraded":"2024-03-11T21:51:06.720450+0000","last_fullsized":"2024-03-11T21:51:06.720450+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:51:05.983372+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:37.512018+0000","last_clean_scrub_stamp":"2024-03-11T21:51:05.983372+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:52:19.603119+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,0],"acting":[2,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.16","version":"456'25","reported_seq":1183,"reported_epoch":554,"state":"active+clean","last_fresh":"2024-03-11T21:51:06.721793+0000","last_change":"2024-03-11T21:50:38.397638+0000","last_active":"2024-03-11T21:51:06.721793+0000","last_peered":"2024-03-11T21:51:06.721793+0000","last_clean":"2024-03-11T21:51:06.721793+0000","last_became_active":"2024-03-11T21:49:17.147316+0000","last_became_peered":"2024-03-11T21:49:17.147316+0000","last_unstale":"2024-03-11T21:51:06.721793+0000","last_undegraded":"2024-03-11T21:51:06.721793+0000","last_fullsized":"2024-03-11T21:51:06.721793+0000","mapping_epoch":528,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":529,"parent":"0.0","parent_split_bits":0,"last_scrub":"456'25","last_scrub_stamp":"2024-03-11T21:50:38.397443+0000","last_deep_scrub":"369'19","last_deep_scrub_stamp":"2024-03-11T21:41:22.115634+0000","last_clean_scrub_stamp":"2024-03-11T21:50:38.397443+0000","objects_scrubbed":4,"log_size":25,"log_dups_size":0,"ondisk_log_size":25,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:52:00.906018+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":35,"num_read_kb":29,"num_write":22,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19,"num_bytes_recovered":440,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.b","version":"0'0","reported_seq":69,"reported_epoch":554,"state":"active+clean","last_fresh":"2024-03-11T21:51:07.586360+0000","last_change":"2024-03-11T21:49:43.923280+0000","last_active":"2024-03-11T21:51:07.586360+0000","last_peered":"2024-03-11T21:51:07.586360+0000","last_clean":"2024-03-11T21:51:07.586360+0000","last_became_active":"2024-03-11T21:49:17.146927+0000","last_became_peered":"2024-03-11T21:49:17.146927+0000","last_unstale":"2024-03-11T21:51:07.586360+0000","last_undegraded":"2024-03-11T21:51:07.586360+0000","last_fullsized":"2024-03-11T21:51:07.586360+0000","mapping_epoch":528,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":529,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:49:43.923140+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:44:35.622488+0000","last_clean_scrub_stamp":"2024-03-11T21:49:43.923140+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:51:12.639907+0000","scrub_duration":119,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.f","version":"67'2","reported_seq":67,"reported_epoch":554,"state":"active+clean","last_fresh":"2024-03-11T21:51:06.723121+0000","last_change":"2024-03-11T21:49:36.908598+0000","last_active":"2024-03-11T21:51:06.723121+0000","last_peered":"2024-03-11T21:51:06.723121+0000","last_clean":"2024-03-11T21:51:06.723121+0000","last_became_active":"2024-03-11T21:49:26.656928+0000","last_became_peered":"2024-03-11T21:49:26.656928+0000","last_unstale":"2024-03-11T21:51:06.723121+0000","last_undegraded":"2024-03-11T21:51:06.723121+0000","last_fullsized":"2024-03-11T21:51:06.723121+0000","mapping_epoch":532,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":533,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'2","last_scrub_stamp":"2024-03-11T21:49:36.908500+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:37.512018+0000","last_clean_scrub_stamp":"2024-03-11T21:49:36.908500+0000","objects_scrubbed":2,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":92,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":368,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,7],"acting":[4,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.9","version":"456'29","reported_seq":73,"reported_epoch":554,"state":"active+clean","last_fresh":"2024-03-11T21:51:06.722137+0000","last_change":"2024-03-11T21:50:24.143508+0000","last_active":"2024-03-11T21:51:06.722137+0000","last_peered":"2024-03-11T21:51:06.722137+0000","last_clean":"2024-03-11T21:51:06.722137+0000","last_became_active":"2024-03-11T21:49:26.657961+0000","last_became_peered":"2024-03-11T21:49:26.657961+0000","last_unstale":"2024-03-11T21:51:06.722137+0000","last_undegraded":"2024-03-11T21:51:06.722137+0000","last_fullsized":"2024-03-11T21:51:06.722137+0000","mapping_epoch":532,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":533,"parent":"0.0","parent_split_bits":0,"last_scrub":"456'29","last_scrub_stamp":"2024-03-11T21:50:24.143389+0000","last_deep_scrub":"456'29","last_deep_scrub_stamp":"2024-03-11T21:46:28.006437+0000","last_clean_scrub_stamp":"2024-03-11T21:50:24.143389+0000","objects_scrubbed":5,"log_size":29,"log_dups_size":0,"ondisk_log_size":29,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:51:32.175677+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":42,"num_read_kb":33,"num_write":26,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":660,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,3],"acting":[4,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":1066,"reported_epoch":554,"state":"active+clean","last_fresh":"2024-03-11T21:51:06.720541+0000","last_change":"2024-03-11T21:50:06.892558+0000","last_active":"2024-03-11T21:51:06.720541+0000","last_peered":"2024-03-11T21:51:06.720541+0000","last_clean":"2024-03-11T21:51:06.720541+0000","last_became_active":"2024-03-11T21:25:42.573987+0000","last_became_peered":"2024-03-11T21:25:42.573987+0000","last_unstale":"2024-03-11T21:51:06.720541+0000","last_undegraded":"2024-03-11T21:51:06.720541+0000","last_fullsized":"2024-03-11T21:51:06.720541+0000","mapping_epoch":70,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":71,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:50:06.892477+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:41.544550+0000","last_clean_scrub_stamp":"2024-03-11T21:50:06.892477+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:51:35.689863+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.0","version":"0'0","reported_seq":1090,"reported_epoch":554,"state":"active+clean","last_fresh":"2024-03-11T21:51:06.720507+0000","last_change":"2024-03-11T21:50:54.324503+0000","last_active":"2024-03-11T21:51:06.720507+0000","last_peered":"2024-03-11T21:51:06.720507+0000","last_clean":"2024-03-11T21:51:06.720507+0000","last_became_active":"2024-03-11T21:49:17.146239+0000","last_became_peered":"2024-03-11T21:49:17.146239+0000","last_unstale":"2024-03-11T21:51:06.720507+0000","last_undegraded":"2024-03-11T21:51:06.720507+0000","last_fullsized":"2024-03-11T21:51:06.720507+0000","mapping_epoch":528,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":529,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:50:54.324421+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:29:39.498784+0000","last_clean_scrub_stamp":"2024-03-11T21:50:54.324421+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:51:54.677449+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":960,"reported_epoch":554,"state":"active+clean","last_fresh":"2024-03-11T21:51:06.961130+0000","last_change":"2024-03-11T21:50:21.949955+0000","last_active":"2024-03-11T21:51:06.961130+0000","last_peered":"2024-03-11T21:51:06.961130+0000","last_clean":"2024-03-11T21:51:06.961130+0000","last_became_active":"2024-03-11T21:49:46.949173+0000","last_became_peered":"2024-03-11T21:49:46.949173+0000","last_unstale":"2024-03-11T21:51:06.961130+0000","last_undegraded":"2024-03-11T21:51:06.961130+0000","last_fullsized":"2024-03-11T21:51:06.961130+0000","mapping_epoch":537,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":538,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:50:21.949868+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:48:58.177316+0000","last_clean_scrub_stamp":"2024-03-11T21:50:21.949868+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:51:22.766616+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,7],"acting":[1,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.2","version":"456'35","reported_seq":1195,"reported_epoch":554,"state":"active+clean","last_fresh":"2024-03-11T21:51:06.961672+0000","last_change":"2024-03-11T21:50:16.982670+0000","last_active":"2024-03-11T21:51:06.961672+0000","last_peered":"2024-03-11T21:51:06.961672+0000","last_clean":"2024-03-11T21:51:06.961672+0000","last_became_active":"2024-03-11T21:49:26.644586+0000","last_became_peered":"2024-03-11T21:49:26.644586+0000","last_unstale":"2024-03-11T21:51:06.961672+0000","last_undegraded":"2024-03-11T21:51:06.961672+0000","last_fullsized":"2024-03-11T21:51:06.961672+0000","mapping_epoch":532,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":533,"parent":"0.0","parent_split_bits":0,"last_scrub":"456'35","last_scrub_stamp":"2024-03-11T21:50:16.982580+0000","last_deep_scrub":"94'15","last_deep_scrub_stamp":"2024-03-11T21:26:49.993060+0000","last_clean_scrub_stamp":"2024-03-11T21:50:16.982580+0000","objects_scrubbed":5,"log_size":35,"log_dups_size":0,"ondisk_log_size":35,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:51:23.909581+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":45,"num_read_kb":40,"num_write":30,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.3","version":"503'10","reported_seq":1397,"reported_epoch":554,"state":"active+clean","last_fresh":"2024-03-11T21:51:08.895402+0000","last_change":"2024-03-11T21:50:24.313886+0000","last_active":"2024-03-11T21:51:08.895402+0000","last_peered":"2024-03-11T21:51:08.895402+0000","last_clean":"2024-03-11T21:51:08.895402+0000","last_became_active":"2024-03-11T21:48:28.641906+0000","last_became_peered":"2024-03-11T21:48:28.641906+0000","last_unstale":"2024-03-11T21:51:08.895402+0000","last_undegraded":"2024-03-11T21:51:08.895402+0000","last_fullsized":"2024-03-11T21:51:08.895402+0000","mapping_epoch":502,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":503,"parent":"0.0","parent_split_bits":0,"last_scrub":"503'10","last_scrub_stamp":"2024-03-11T21:50:24.313803+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:41.544550+0000","last_clean_scrub_stamp":"2024-03-11T21:50:24.313803+0000","objects_scrubbed":1,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:51:26.111087+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,5],"acting":[0,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":75,"reported_epoch":554,"state":"active+clean","last_fresh":"2024-03-11T21:51:07.586906+0000","last_change":"2024-03-11T21:50:14.457423+0000","last_active":"2024-03-11T21:51:07.586906+0000","last_peered":"2024-03-11T21:51:07.586906+0000","last_clean":"2024-03-11T21:51:07.586906+0000","last_became_active":"2024-03-11T21:49:26.638739+0000","last_became_peered":"2024-03-11T21:49:26.638739+0000","last_unstale":"2024-03-11T21:51:07.586906+0000","last_undegraded":"2024-03-11T21:51:07.586906+0000","last_fullsized":"2024-03-11T21:51:07.586906+0000","mapping_epoch":532,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":533,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:50:14.457308+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:50:14.457308+0000","last_clean_scrub_stamp":"2024-03-11T21:50:14.457308+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:51:17.178186+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,7],"acting":[3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":130,"reported_epoch":554,"state":"active+clean","last_fresh":"2024-03-11T21:51:07.481792+0000","last_change":"2024-03-11T21:51:04.701911+0000","last_active":"2024-03-11T21:51:07.481792+0000","last_peered":"2024-03-11T21:51:07.481792+0000","last_clean":"2024-03-11T21:51:07.481792+0000","last_became_active":"2024-03-11T21:48:28.640205+0000","last_became_peered":"2024-03-11T21:48:28.640205+0000","last_unstale":"2024-03-11T21:51:07.481792+0000","last_undegraded":"2024-03-11T21:51:07.481792+0000","last_fullsized":"2024-03-11T21:51:07.481792+0000","mapping_epoch":502,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":503,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:51:04.701778+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:51:04.701778+0000","last_clean_scrub_stamp":"2024-03-11T21:51:04.701778+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:52:26.493598+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.4","version":"553'68","reported_seq":1365,"reported_epoch":554,"state":"active+clean","last_fresh":"2024-03-11T21:51:08.895446+0000","last_change":"2024-03-11T21:50:59.004099+0000","last_active":"2024-03-11T21:51:08.895446+0000","last_peered":"2024-03-11T21:51:08.895446+0000","last_clean":"2024-03-11T21:51:08.895446+0000","last_became_active":"2024-03-11T21:49:46.258448+0000","last_became_peered":"2024-03-11T21:49:46.258448+0000","last_unstale":"2024-03-11T21:51:08.895446+0000","last_undegraded":"2024-03-11T21:51:08.895446+0000","last_fullsized":"2024-03-11T21:51:08.895446+0000","mapping_epoch":537,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":538,"parent":"0.0","parent_split_bits":0,"last_scrub":"553'68","last_scrub_stamp":"2024-03-11T21:50:59.003898+0000","last_deep_scrub":"553'68","last_deep_scrub_stamp":"2024-03-11T21:50:59.003898+0000","last_clean_scrub_stamp":"2024-03-11T21:50:59.003898+0000","objects_scrubbed":10,"log_size":68,"log_dups_size":0,"ondisk_log_size":68,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:52:26.135652+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":358,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":84,"num_read_kb":69,"num_write":50,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":358,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3],"acting":[1,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":1073,"reported_epoch":554,"state":"active+clean","last_fresh":"2024-03-11T21:51:06.721730+0000","last_change":"2024-03-11T21:50:33.294601+0000","last_active":"2024-03-11T21:51:06.721730+0000","last_peered":"2024-03-11T21:51:06.721730+0000","last_clean":"2024-03-11T21:51:06.721730+0000","last_became_active":"2024-03-11T21:49:26.643886+0000","last_became_peered":"2024-03-11T21:49:26.643886+0000","last_unstale":"2024-03-11T21:51:06.721730+0000","last_undegraded":"2024-03-11T21:51:06.721730+0000","last_fullsized":"2024-03-11T21:51:06.721730+0000","mapping_epoch":532,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":533,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:50:33.294521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:41.544550+0000","last_clean_scrub_stamp":"2024-03-11T21:50:33.294521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:51:55.612537+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.5","version":"0'0","reported_seq":732,"reported_epoch":554,"state":"active+clean","last_fresh":"2024-03-11T21:51:07.587510+0000","last_change":"2024-03-11T21:50:13.315051+0000","last_active":"2024-03-11T21:51:07.587510+0000","last_peered":"2024-03-11T21:51:07.587510+0000","last_clean":"2024-03-11T21:51:07.587510+0000","last_became_active":"2024-03-11T21:49:17.149583+0000","last_became_peered":"2024-03-11T21:49:17.149583+0000","last_unstale":"2024-03-11T21:51:07.587510+0000","last_undegraded":"2024-03-11T21:51:07.587510+0000","last_fullsized":"2024-03-11T21:51:07.587510+0000","mapping_epoch":528,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":529,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:50:13.314944+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:43.559452+0000","last_clean_scrub_stamp":"2024-03-11T21:50:13.314944+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:51:29.726449+0000","scrub_duration":96,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,3],"acting":[7,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.0","version":"164'6","reported_seq":2339,"reported_epoch":554,"state":"active+clean","last_fresh":"2024-03-11T21:51:12.410463+0000","last_change":"2024-03-11T21:50:03.940744+0000","last_active":"2024-03-11T21:51:12.410463+0000","last_peered":"2024-03-11T21:51:12.410463+0000","last_clean":"2024-03-11T21:51:12.410463+0000","last_became_active":"2024-03-11T21:49:46.257628+0000","last_became_peered":"2024-03-11T21:49:46.257628+0000","last_unstale":"2024-03-11T21:51:12.410463+0000","last_undegraded":"2024-03-11T21:51:12.410463+0000","last_fullsized":"2024-03-11T21:51:12.410463+0000","mapping_epoch":537,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":538,"parent":"0.0","parent_split_bits":0,"last_scrub":"164'6","last_scrub_stamp":"2024-03-11T21:50:03.940628+0000","last_deep_scrub":"164'6","last_deep_scrub_stamp":"2024-03-11T21:32:17.597666+0000","last_clean_scrub_stamp":"2024-03-11T21:50:03.940628+0000","objects_scrubbed":1,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.000753559,"stat_sum":{"num_bytes":389,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1375,"num_read_kb":1370,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,6],"acting":[1,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":1104,"reported_epoch":554,"state":"active+clean","last_fresh":"2024-03-11T21:51:09.976429+0000","last_change":"2024-03-11T21:51:09.976448+0000","last_active":"2024-03-11T21:51:09.976429+0000","last_peered":"2024-03-11T21:51:09.976429+0000","last_clean":"2024-03-11T21:51:09.976429+0000","last_became_active":"2024-03-11T21:49:17.138476+0000","last_became_peered":"2024-03-11T21:49:17.138476+0000","last_unstale":"2024-03-11T21:51:09.976429+0000","last_undegraded":"2024-03-11T21:51:09.976429+0000","last_fullsized":"2024-03-11T21:51:09.976429+0000","mapping_epoch":528,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":529,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:51:09.976370+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:43:28.183315+0000","last_clean_scrub_stamp":"2024-03-11T21:51:09.976370+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:52:11.007881+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.7","version":"456'26","reported_seq":1147,"reported_epoch":554,"state":"active+clean","last_fresh":"2024-03-11T21:51:06.961914+0000","last_change":"2024-03-11T21:50:02.957549+0000","last_active":"2024-03-11T21:51:06.961914+0000","last_peered":"2024-03-11T21:51:06.961914+0000","last_clean":"2024-03-11T21:51:06.961914+0000","last_became_active":"2024-03-11T21:48:28.632227+0000","last_became_peered":"2024-03-11T21:48:28.632227+0000","last_unstale":"2024-03-11T21:51:06.961914+0000","last_undegraded":"2024-03-11T21:51:06.961914+0000","last_fullsized":"2024-03-11T21:51:06.961914+0000","mapping_epoch":502,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":503,"parent":"0.0","parent_split_bits":0,"last_scrub":"456'26","last_scrub_stamp":"2024-03-11T21:50:02.957446+0000","last_deep_scrub":"260'22","last_deep_scrub_stamp":"2024-03-11T21:41:27.232403+0000","last_clean_scrub_stamp":"2024-03-11T21:50:02.957446+0000","objects_scrubbed":5,"log_size":26,"log_dups_size":0,"ondisk_log_size":26,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:51:16.879974+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":42,"num_read_kb":31,"num_write":24,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":660,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":1090,"reported_epoch":554,"state":"active+clean","last_fresh":"2024-03-11T21:51:06.722329+0000","last_change":"2024-03-11T21:50:14.768174+0000","last_active":"2024-03-11T21:51:06.722329+0000","last_peered":"2024-03-11T21:51:06.722329+0000","last_clean":"2024-03-11T21:51:06.722329+0000","last_became_active":"2024-03-11T21:48:28.636457+0000","last_became_peered":"2024-03-11T21:48:28.636457+0000","last_unstale":"2024-03-11T21:51:06.722329+0000","last_undegraded":"2024-03-11T21:51:06.722329+0000","last_fullsized":"2024-03-11T21:51:06.722329+0000","mapping_epoch":502,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":503,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:50:14.768096+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:41.544550+0000","last_clean_scrub_stamp":"2024-03-11T21:50:14.768096+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:51:19.117559+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7],"acting":[2,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.7","version":"124'2","reported_seq":154,"reported_epoch":554,"state":"active+clean","last_fresh":"2024-03-11T21:51:07.482379+0000","last_change":"2024-03-11T21:50:55.867991+0000","last_active":"2024-03-11T21:51:07.482379+0000","last_peered":"2024-03-11T21:51:07.482379+0000","last_clean":"2024-03-11T21:51:07.482379+0000","last_became_active":"2024-03-11T21:49:26.651720+0000","last_became_peered":"2024-03-11T21:49:26.651720+0000","last_unstale":"2024-03-11T21:51:07.482379+0000","last_undegraded":"2024-03-11T21:51:07.482379+0000","last_fullsized":"2024-03-11T21:51:07.482379+0000","mapping_epoch":532,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":533,"parent":"0.0","parent_split_bits":0,"last_scrub":"124'2","last_scrub_stamp":"2024-03-11T21:50:55.867884+0000","last_deep_scrub":"124'2","last_deep_scrub_stamp":"2024-03-11T21:29:34.956908+0000","last_clean_scrub_stamp":"2024-03-11T21:50:55.867884+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:52:01.413473+0000","scrub_duration":129,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":52,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,4],"acting":[5,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2","version":"71'2","reported_seq":73,"reported_epoch":554,"state":"active+clean","last_fresh":"2024-03-11T21:51:07.586272+0000","last_change":"2024-03-11T21:49:48.708764+0000","last_active":"2024-03-11T21:51:07.586272+0000","last_peered":"2024-03-11T21:51:07.586272+0000","last_clean":"2024-03-11T21:51:07.586272+0000","last_became_active":"2024-03-11T21:49:17.136844+0000","last_became_peered":"2024-03-11T21:49:17.136844+0000","last_unstale":"2024-03-11T21:51:07.586272+0000","last_undegraded":"2024-03-11T21:51:07.586272+0000","last_fullsized":"2024-03-11T21:51:07.586272+0000","mapping_epoch":528,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":529,"parent":"0.0","parent_split_bits":0,"last_scrub":"71'2","last_scrub_stamp":"2024-03-11T21:49:48.708634+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:37.512018+0000","last_clean_scrub_stamp":"2024-03-11T21:49:48.708634+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:51:09.002959+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.0014206449999999999,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":38,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":130,"reported_epoch":554,"state":"active+clean","last_fresh":"2024-03-11T21:51:07.481765+0000","last_change":"2024-03-11T21:50:58.641704+0000","last_active":"2024-03-11T21:51:07.481765+0000","last_peered":"2024-03-11T21:51:07.481765+0000","last_clean":"2024-03-11T21:51:07.481765+0000","last_became_active":"2024-03-11T21:48:28.636579+0000","last_became_peered":"2024-03-11T21:48:28.636579+0000","last_unstale":"2024-03-11T21:51:07.481765+0000","last_undegraded":"2024-03-11T21:51:07.481765+0000","last_fullsized":"2024-03-11T21:51:07.481765+0000","mapping_epoch":502,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":503,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:50:58.641573+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:37.512018+0000","last_clean_scrub_stamp":"2024-03-11T21:50:58.641573+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:52:23.519875+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"87'128","reported_seq":1183,"reported_epoch":554,"state":"active+clean","last_fresh":"2024-03-11T21:51:07.586330+0000","last_change":"2024-03-11T21:49:46.978593+0000","last_active":"2024-03-11T21:51:07.586330+0000","last_peered":"2024-03-11T21:51:07.586330+0000","last_clean":"2024-03-11T21:51:07.586330+0000","last_became_active":"2024-03-11T21:34:44.164569+0000","last_became_peered":"2024-03-11T21:34:44.164569+0000","last_unstale":"2024-03-11T21:51:07.586330+0000","last_undegraded":"2024-03-11T21:51:07.586330+0000","last_fullsized":"2024-03-11T21:51:07.586330+0000","mapping_epoch":229,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":230,"parent":"0.0","parent_split_bits":0,"last_scrub":"87'128","last_scrub_stamp":"2024-03-11T21:49:46.978479+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:23:24.261974+0000","last_clean_scrub_stamp":"2024-03-11T21:49:46.978479+0000","objects_scrubbed":2,"log_size":128,"log_dups_size":0,"ondisk_log_size":128,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:51:11.989712+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":198,"num_read_kb":294,"num_write":179,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":2951840,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.5","version":"533'50","reported_seq":1511,"reported_epoch":554,"state":"active+clean","last_fresh":"2024-03-11T21:51:07.587530+0000","last_change":"2024-03-11T21:50:43.124806+0000","last_active":"2024-03-11T21:51:07.587530+0000","last_peered":"2024-03-11T21:51:07.587530+0000","last_clean":"2024-03-11T21:51:07.587530+0000","last_became_active":"2024-03-11T21:49:26.637870+0000","last_became_peered":"2024-03-11T21:49:26.637870+0000","last_unstale":"2024-03-11T21:51:07.587530+0000","last_undegraded":"2024-03-11T21:51:07.587530+0000","last_fullsized":"2024-03-11T21:51:07.587530+0000","mapping_epoch":532,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":533,"parent":"0.0","parent_split_bits":0,"last_scrub":"533'50","last_scrub_stamp":"2024-03-11T21:50:43.124673+0000","last_deep_scrub":"533'50","last_deep_scrub_stamp":"2024-03-11T21:49:32.782974+0000","last_clean_scrub_stamp":"2024-03-11T21:50:43.124673+0000","objects_scrubbed":8,"log_size":50,"log_dups_size":0,"ondisk_log_size":50,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:51:47.509858+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":154,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":61,"num_read_kb":51,"num_write":37,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":308,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":731,"reported_epoch":554,"state":"active+clean","last_fresh":"2024-03-11T21:51:07.587415+0000","last_change":"2024-03-11T21:50:30.303207+0000","last_active":"2024-03-11T21:51:07.587415+0000","last_peered":"2024-03-11T21:51:07.587415+0000","last_clean":"2024-03-11T21:51:07.587415+0000","last_became_active":"2024-03-11T21:48:28.627651+0000","last_became_peered":"2024-03-11T21:48:28.627651+0000","last_unstale":"2024-03-11T21:51:07.587415+0000","last_undegraded":"2024-03-11T21:51:07.587415+0000","last_fullsized":"2024-03-11T21:51:07.587415+0000","mapping_epoch":502,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":503,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:50:30.303099+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:32:21.266672+0000","last_clean_scrub_stamp":"2024-03-11T21:50:30.303099+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:51:48.779449+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,5],"acting":[7,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.4","version":"0'0","reported_seq":1090,"reported_epoch":554,"state":"active+clean","last_fresh":"2024-03-11T21:51:06.961734+0000","last_change":"2024-03-11T21:50:41.927835+0000","last_active":"2024-03-11T21:51:06.961734+0000","last_peered":"2024-03-11T21:51:06.961734+0000","last_clean":"2024-03-11T21:51:06.961734+0000","last_became_active":"2024-03-11T21:49:17.138104+0000","last_became_peered":"2024-03-11T21:49:17.138104+0000","last_unstale":"2024-03-11T21:51:06.961734+0000","last_undegraded":"2024-03-11T21:51:06.961734+0000","last_fullsized":"2024-03-11T21:51:06.961734+0000","mapping_epoch":528,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":529,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:50:41.927752+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:49:18.000732+0000","last_clean_scrub_stamp":"2024-03-11T21:50:41.927752+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:51:55.023362+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":1121,"reported_epoch":554,"state":"active+clean","last_fresh":"2024-03-11T21:51:06.721118+0000","last_change":"2024-03-11T21:50:03.138901+0000","last_active":"2024-03-11T21:51:06.721118+0000","last_peered":"2024-03-11T21:51:06.721118+0000","last_clean":"2024-03-11T21:51:06.721118+0000","last_became_active":"2024-03-11T21:49:26.643353+0000","last_became_peered":"2024-03-11T21:49:26.643353+0000","last_unstale":"2024-03-11T21:51:06.721118+0000","last_undegraded":"2024-03-11T21:51:06.721118+0000","last_fullsized":"2024-03-11T21:51:06.721118+0000","mapping_epoch":532,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":533,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:50:03.138820+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:37.512018+0000","last_clean_scrub_stamp":"2024-03-11T21:50:03.138820+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:51:29.637981+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.00089342499999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":712,"reported_epoch":554,"state":"active+clean","last_fresh":"2024-03-11T21:51:07.586280+0000","last_change":"2024-03-11T21:50:45.460269+0000","last_active":"2024-03-11T21:51:07.586280+0000","last_peered":"2024-03-11T21:51:07.586280+0000","last_clean":"2024-03-11T21:51:07.586280+0000","last_became_active":"2024-03-11T21:34:41.149219+0000","last_became_peered":"2024-03-11T21:34:41.149219+0000","last_unstale":"2024-03-11T21:51:07.586280+0000","last_undegraded":"2024-03-11T21:51:07.586280+0000","last_fullsized":"2024-03-11T21:51:07.586280+0000","mapping_epoch":226,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":227,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:50:45.460178+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:43:30.734024+0000","last_clean_scrub_stamp":"2024-03-11T21:50:45.460178+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:52:00.950582+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,0],"acting":[7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.6","version":"456'42","reported_seq":1172,"reported_epoch":554,"state":"active+clean","last_fresh":"2024-03-11T21:51:06.721697+0000","last_change":"2024-03-11T21:50:59.395853+0000","last_active":"2024-03-11T21:51:06.721697+0000","last_peered":"2024-03-11T21:51:06.721697+0000","last_clean":"2024-03-11T21:51:06.721697+0000","last_became_active":"2024-03-11T21:46:53.263790+0000","last_became_peered":"2024-03-11T21:46:53.263790+0000","last_unstale":"2024-03-11T21:51:06.721697+0000","last_undegraded":"2024-03-11T21:51:06.721697+0000","last_fullsized":"2024-03-11T21:51:06.721697+0000","mapping_epoch":473,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":474,"parent":"0.0","parent_split_bits":0,"last_scrub":"456'42","last_scrub_stamp":"2024-03-11T21:50:59.395753+0000","last_deep_scrub":"456'42","last_deep_scrub_stamp":"2024-03-11T21:49:31.238617+0000","last_clean_scrub_stamp":"2024-03-11T21:50:59.395753+0000","objects_scrubbed":6,"log_size":42,"log_dups_size":0,"ondisk_log_size":42,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:52:23.359650+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":54,"num_read_kb":48,"num_write":36,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":138,"reported_epoch":554,"state":"active+clean","last_fresh":"2024-03-11T21:51:07.482247+0000","last_change":"2024-03-11T21:51:05.710542+0000","last_active":"2024-03-11T21:51:07.482247+0000","last_peered":"2024-03-11T21:51:07.482247+0000","last_clean":"2024-03-11T21:51:07.482247+0000","last_became_active":"2024-03-11T21:48:28.644298+0000","last_became_peered":"2024-03-11T21:48:28.644298+0000","last_unstale":"2024-03-11T21:51:07.482247+0000","last_undegraded":"2024-03-11T21:51:07.482247+0000","last_fullsized":"2024-03-11T21:51:07.482247+0000","mapping_epoch":502,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":503,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:51:05.710446+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:41.544550+0000","last_clean_scrub_stamp":"2024-03-11T21:51:05.710446+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:52:23.406037+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":1065,"reported_epoch":554,"state":"active+clean","last_fresh":"2024-03-11T21:51:06.961489+0000","last_change":"2024-03-11T21:50:35.034430+0000","last_active":"2024-03-11T21:51:06.961489+0000","last_peered":"2024-03-11T21:51:06.961489+0000","last_clean":"2024-03-11T21:51:06.961489+0000","last_became_active":"2024-03-11T21:25:45.301416+0000","last_became_peered":"2024-03-11T21:25:45.301416+0000","last_unstale":"2024-03-11T21:51:06.961489+0000","last_undegraded":"2024-03-11T21:51:06.961489+0000","last_fullsized":"2024-03-11T21:51:06.961489+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:50:35.034308+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:43.559452+0000","last_clean_scrub_stamp":"2024-03-11T21:50:35.034308+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:51:49.148312+0000","scrub_duration":104,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":723,"reported_epoch":554,"state":"active+clean","last_fresh":"2024-03-11T21:51:07.587958+0000","last_change":"2024-03-11T21:50:14.240816+0000","last_active":"2024-03-11T21:51:07.587958+0000","last_peered":"2024-03-11T21:51:07.587958+0000","last_clean":"2024-03-11T21:51:07.587958+0000","last_became_active":"2024-03-11T21:49:26.644277+0000","last_became_peered":"2024-03-11T21:49:26.644277+0000","last_unstale":"2024-03-11T21:51:07.587958+0000","last_undegraded":"2024-03-11T21:51:07.587958+0000","last_fullsized":"2024-03-11T21:51:07.587958+0000","mapping_epoch":532,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":533,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:50:14.240723+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:26:57.333669+0000","last_clean_scrub_stamp":"2024-03-11T21:50:14.240723+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:51:39.046762+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.3","version":"456'47","reported_seq":1243,"reported_epoch":554,"state":"active+clean","last_fresh":"2024-03-11T21:51:06.721058+0000","last_change":"2024-03-11T21:50:56.393665+0000","last_active":"2024-03-11T21:51:06.721058+0000","last_peered":"2024-03-11T21:51:06.721058+0000","last_clean":"2024-03-11T21:51:06.721058+0000","last_became_active":"2024-03-11T21:48:28.623917+0000","last_became_peered":"2024-03-11T21:48:28.623917+0000","last_unstale":"2024-03-11T21:51:06.721058+0000","last_undegraded":"2024-03-11T21:51:06.721058+0000","last_fullsized":"2024-03-11T21:51:06.721058+0000","mapping_epoch":502,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":503,"parent":"0.0","parent_split_bits":0,"last_scrub":"456'47","last_scrub_stamp":"2024-03-11T21:50:56.393540+0000","last_deep_scrub":"165'27","last_deep_scrub_stamp":"2024-03-11T21:32:21.576732+0000","last_clean_scrub_stamp":"2024-03-11T21:50:56.393540+0000","objects_scrubbed":8,"log_size":47,"log_dups_size":0,"ondisk_log_size":47,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:52:15.226687+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":69,"num_read_kb":55,"num_write":42,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":32,"num_bytes_recovered":1320,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":1097,"reported_epoch":554,"state":"active+clean","last_fresh":"2024-03-11T21:51:07.586533+0000","last_change":"2024-03-11T21:50:07.070790+0000","last_active":"2024-03-11T21:51:07.586533+0000","last_peered":"2024-03-11T21:51:07.586533+0000","last_clean":"2024-03-11T21:51:07.586533+0000","last_became_active":"2024-03-11T21:48:28.642354+0000","last_became_peered":"2024-03-11T21:48:28.642354+0000","last_unstale":"2024-03-11T21:51:07.586533+0000","last_undegraded":"2024-03-11T21:51:07.586533+0000","last_fullsized":"2024-03-11T21:51:07.586533+0000","mapping_epoch":502,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":503,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:50:07.070658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:41:21.857205+0000","last_clean_scrub_stamp":"2024-03-11T21:50:07.070658+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:51:17.405424+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":61,"reported_epoch":554,"state":"active+clean","last_fresh":"2024-03-11T21:51:06.722292+0000","last_change":"2024-03-11T21:50:02.026756+0000","last_active":"2024-03-11T21:51:06.722292+0000","last_peered":"2024-03-11T21:51:06.722292+0000","last_clean":"2024-03-11T21:51:06.722292+0000","last_became_active":"2024-03-11T21:49:26.642045+0000","last_became_peered":"2024-03-11T21:49:26.642045+0000","last_unstale":"2024-03-11T21:51:06.722292+0000","last_undegraded":"2024-03-11T21:51:06.722292+0000","last_fullsized":"2024-03-11T21:51:06.722292+0000","mapping_epoch":532,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":533,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:50:02.026631+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:43.559452+0000","last_clean_scrub_stamp":"2024-03-11T21:50:02.026631+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:51:10.962724+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.6","version":"67'1","reported_seq":1107,"reported_epoch":554,"state":"active+clean","last_fresh":"2024-03-11T21:51:06.961833+0000","last_change":"2024-03-11T21:50:33.961085+0000","last_active":"2024-03-11T21:51:06.961833+0000","last_peered":"2024-03-11T21:51:06.961833+0000","last_clean":"2024-03-11T21:51:06.961833+0000","last_became_active":"2024-03-11T21:49:26.644153+0000","last_became_peered":"2024-03-11T21:49:26.644153+0000","last_unstale":"2024-03-11T21:51:06.961833+0000","last_undegraded":"2024-03-11T21:51:06.961833+0000","last_fullsized":"2024-03-11T21:51:06.961833+0000","mapping_epoch":532,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":533,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'1","last_scrub_stamp":"2024-03-11T21:50:33.960986+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:37.512018+0000","last_clean_scrub_stamp":"2024-03-11T21:50:33.960986+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:51:44.085831+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":138,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,4],"acting":[1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.0","version":"456'50","reported_seq":87,"reported_epoch":554,"state":"active+clean","last_fresh":"2024-03-11T21:51:07.585980+0000","last_change":"2024-03-11T21:50:37.619125+0000","last_active":"2024-03-11T21:51:07.585980+0000","last_peered":"2024-03-11T21:51:07.585980+0000","last_clean":"2024-03-11T21:51:07.585980+0000","last_became_active":"2024-03-11T21:49:17.155148+0000","last_became_peered":"2024-03-11T21:49:17.155148+0000","last_unstale":"2024-03-11T21:51:07.585980+0000","last_undegraded":"2024-03-11T21:51:07.585980+0000","last_fullsized":"2024-03-11T21:51:07.585980+0000","mapping_epoch":528,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":529,"parent":"0.0","parent_split_bits":0,"last_scrub":"456'50","last_scrub_stamp":"2024-03-11T21:50:37.619018+0000","last_deep_scrub":"260'38","last_deep_scrub_stamp":"2024-03-11T21:45:39.241846+0000","last_clean_scrub_stamp":"2024-03-11T21:50:37.619018+0000","objects_scrubbed":8,"log_size":50,"log_dups_size":0,"ondisk_log_size":50,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:52:00.939522+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":69,"num_read_kb":57,"num_write":44,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":24,"num_bytes_recovered":660,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":69,"reported_epoch":554,"state":"active+clean","last_fresh":"2024-03-11T21:51:06.722170+0000","last_change":"2024-03-11T21:50:45.179659+0000","last_active":"2024-03-11T21:51:06.722170+0000","last_peered":"2024-03-11T21:51:06.722170+0000","last_clean":"2024-03-11T21:51:06.722170+0000","last_became_active":"2024-03-11T21:49:26.645160+0000","last_became_peered":"2024-03-11T21:49:26.645160+0000","last_unstale":"2024-03-11T21:51:06.722170+0000","last_undegraded":"2024-03-11T21:51:06.722170+0000","last_fullsized":"2024-03-11T21:51:06.722170+0000","mapping_epoch":532,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":533,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:50:45.179558+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:50:45.179558+0000","last_clean_scrub_stamp":"2024-03-11T21:50:45.179558+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:51:47.048378+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,7],"acting":[4,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.3","version":"0'0","reported_seq":711,"reported_epoch":554,"state":"active+clean","last_fresh":"2024-03-11T21:51:07.587260+0000","last_change":"2024-03-11T21:50:48.516514+0000","last_active":"2024-03-11T21:51:07.587260+0000","last_peered":"2024-03-11T21:51:07.587260+0000","last_clean":"2024-03-11T21:51:07.587260+0000","last_became_active":"2024-03-11T21:34:41.142683+0000","last_became_peered":"2024-03-11T21:34:41.142683+0000","last_unstale":"2024-03-11T21:51:07.587260+0000","last_undegraded":"2024-03-11T21:51:07.587260+0000","last_fullsized":"2024-03-11T21:51:07.587260+0000","mapping_epoch":226,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":227,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:50:48.516411+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:46:25.645722+0000","last_clean_scrub_stamp":"2024-03-11T21:50:48.516411+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:52:04.201716+0000","scrub_duration":127,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,2],"acting":[7,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":1104,"reported_epoch":554,"state":"active+clean","last_fresh":"2024-03-11T21:51:07.587553+0000","last_change":"2024-03-11T21:50:08.119410+0000","last_active":"2024-03-11T21:51:07.587553+0000","last_peered":"2024-03-11T21:51:07.587553+0000","last_clean":"2024-03-11T21:51:07.587553+0000","last_became_active":"2024-03-11T21:34:41.145628+0000","last_became_peered":"2024-03-11T21:34:41.145628+0000","last_unstale":"2024-03-11T21:51:07.587553+0000","last_undegraded":"2024-03-11T21:51:07.587553+0000","last_fullsized":"2024-03-11T21:51:07.587553+0000","mapping_epoch":226,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":227,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:50:08.119254+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:37.512018+0000","last_clean_scrub_stamp":"2024-03-11T21:50:08.119254+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:51:29.262758+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,2],"acting":[6,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.1","version":"507'49","reported_seq":77,"reported_epoch":554,"state":"active+clean","last_fresh":"2024-03-11T21:51:06.722297+0000","last_change":"2024-03-11T21:49:43.852514+0000","last_active":"2024-03-11T21:51:06.722297+0000","last_peered":"2024-03-11T21:51:06.722297+0000","last_clean":"2024-03-11T21:51:06.722297+0000","last_became_active":"2024-03-11T21:49:26.647366+0000","last_became_peered":"2024-03-11T21:49:26.647366+0000","last_unstale":"2024-03-11T21:51:06.722297+0000","last_undegraded":"2024-03-11T21:51:06.722297+0000","last_fullsized":"2024-03-11T21:51:06.722297+0000","mapping_epoch":532,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":533,"parent":"0.0","parent_split_bits":0,"last_scrub":"507'49","last_scrub_stamp":"2024-03-11T21:49:43.852390+0000","last_deep_scrub":"507'49","last_deep_scrub_stamp":"2024-03-11T21:49:43.852390+0000","last_clean_scrub_stamp":"2024-03-11T21:49:43.852390+0000","objects_scrubbed":7,"log_size":49,"log_dups_size":0,"ondisk_log_size":49,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":61,"num_read_kb":54,"num_write":42,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":21,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6],"acting":[4,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.0","version":"533'15","reported_seq":115,"reported_epoch":554,"state":"active+clean","last_fresh":"2024-03-11T21:51:08.895425+0000","last_change":"2024-03-11T21:51:03.864178+0000","last_active":"2024-03-11T21:51:08.895425+0000","last_peered":"2024-03-11T21:51:08.895425+0000","last_clean":"2024-03-11T21:51:08.895425+0000","last_became_active":"2024-03-11T21:49:26.638749+0000","last_became_peered":"2024-03-11T21:49:26.638749+0000","last_unstale":"2024-03-11T21:51:08.895425+0000","last_undegraded":"2024-03-11T21:51:08.895425+0000","last_fullsized":"2024-03-11T21:51:08.895425+0000","mapping_epoch":532,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":533,"parent":"0.0","parent_split_bits":0,"last_scrub":"533'15","last_scrub_stamp":"2024-03-11T21:51:03.864033+0000","last_deep_scrub":"98'9","last_deep_scrub_stamp":"2024-03-11T21:27:29.745711+0000","last_clean_scrub_stamp":"2024-03-11T21:51:03.864033+0000","objects_scrubbed":1,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:52:09.359413+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.c","version":"0'0","reported_seq":69,"reported_epoch":554,"state":"active+clean","last_fresh":"2024-03-11T21:51:07.586652+0000","last_change":"2024-03-11T21:50:03.603744+0000","last_active":"2024-03-11T21:51:07.586652+0000","last_peered":"2024-03-11T21:51:07.586652+0000","last_clean":"2024-03-11T21:51:07.586652+0000","last_became_active":"2024-03-11T21:49:17.141485+0000","last_became_peered":"2024-03-11T21:49:17.141485+0000","last_unstale":"2024-03-11T21:51:07.586652+0000","last_undegraded":"2024-03-11T21:51:07.586652+0000","last_fullsized":"2024-03-11T21:51:07.586652+0000","mapping_epoch":528,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":529,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:50:03.603643+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:43.559452+0000","last_clean_scrub_stamp":"2024-03-11T21:50:03.603643+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:51:24.851391+0000","scrub_duration":39,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.f","version":"0'0","reported_seq":136,"reported_epoch":554,"state":"active+clean","last_fresh":"2024-03-11T21:51:07.482373+0000","last_change":"2024-03-11T21:50:16.964646+0000","last_active":"2024-03-11T21:51:07.482373+0000","last_peered":"2024-03-11T21:51:07.482373+0000","last_clean":"2024-03-11T21:51:07.482373+0000","last_became_active":"2024-03-11T21:49:26.649455+0000","last_became_peered":"2024-03-11T21:49:26.649455+0000","last_unstale":"2024-03-11T21:51:07.482373+0000","last_undegraded":"2024-03-11T21:51:07.482373+0000","last_fullsized":"2024-03-11T21:51:07.482373+0000","mapping_epoch":532,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":533,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:50:16.964541+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:50:16.964541+0000","last_clean_scrub_stamp":"2024-03-11T21:50:16.964541+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:51:21.063024+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.8","version":"0'0","reported_seq":711,"reported_epoch":554,"state":"active+clean","last_fresh":"2024-03-11T21:51:07.586769+0000","last_change":"2024-03-11T21:50:26.303889+0000","last_active":"2024-03-11T21:51:07.586769+0000","last_peered":"2024-03-11T21:51:07.586769+0000","last_clean":"2024-03-11T21:51:07.586769+0000","last_became_active":"2024-03-11T21:34:41.144681+0000","last_became_peered":"2024-03-11T21:34:41.144681+0000","last_unstale":"2024-03-11T21:51:07.586769+0000","last_undegraded":"2024-03-11T21:51:07.586769+0000","last_fullsized":"2024-03-11T21:51:07.586769+0000","mapping_epoch":226,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":227,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:50:26.303787+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:37.512018+0000","last_clean_scrub_stamp":"2024-03-11T21:50:26.303787+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:51:31.143084+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,1],"acting":[7,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.e","version":"456'32","reported_seq":81,"reported_epoch":554,"state":"active+clean","last_fresh":"2024-03-11T21:51:16.198097+0000","last_change":"2024-03-11T21:51:16.198161+0000","last_active":"2024-03-11T21:51:16.198097+0000","last_peered":"2024-03-11T21:51:16.198097+0000","last_clean":"2024-03-11T21:51:16.198097+0000","last_became_active":"2024-03-11T21:49:26.658056+0000","last_became_peered":"2024-03-11T21:49:26.658056+0000","last_unstale":"2024-03-11T21:51:16.198097+0000","last_undegraded":"2024-03-11T21:51:16.198097+0000","last_fullsized":"2024-03-11T21:51:16.198097+0000","mapping_epoch":532,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":533,"parent":"0.0","parent_split_bits":0,"last_scrub":"456'32","last_scrub_stamp":"2024-03-11T21:51:16.198009+0000","last_deep_scrub":"109'16","last_deep_scrub_stamp":"2024-03-11T21:28:24.739395+0000","last_clean_scrub_stamp":"2024-03-11T21:51:16.198009+0000","objects_scrubbed":5,"log_size":32,"log_dups_size":0,"ondisk_log_size":32,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:52:43.336629+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":42,"num_read_kb":35,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.d","version":"0'0","reported_seq":130,"reported_epoch":554,"state":"active+clean","last_fresh":"2024-03-11T21:51:07.482050+0000","last_change":"2024-03-11T21:51:03.801395+0000","last_active":"2024-03-11T21:51:07.482050+0000","last_peered":"2024-03-11T21:51:07.482050+0000","last_clean":"2024-03-11T21:51:07.482050+0000","last_became_active":"2024-03-11T21:48:28.645468+0000","last_became_peered":"2024-03-11T21:48:28.645468+0000","last_unstale":"2024-03-11T21:51:07.482050+0000","last_undegraded":"2024-03-11T21:51:07.482050+0000","last_fullsized":"2024-03-11T21:51:07.482050+0000","mapping_epoch":502,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":503,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:51:03.801263+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:51:03.801263+0000","last_clean_scrub_stamp":"2024-03-11T21:51:03.801263+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:52:07.125936+0000","scrub_duration":110,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.e","version":"533'12","reported_seq":87,"reported_epoch":554,"state":"active+clean","last_fresh":"2024-03-11T21:51:08.895499+0000","last_change":"2024-03-11T21:49:55.917066+0000","last_active":"2024-03-11T21:51:08.895499+0000","last_peered":"2024-03-11T21:51:08.895499+0000","last_clean":"2024-03-11T21:51:08.895499+0000","last_became_active":"2024-03-11T21:49:26.647248+0000","last_became_peered":"2024-03-11T21:49:26.647248+0000","last_unstale":"2024-03-11T21:51:08.895499+0000","last_undegraded":"2024-03-11T21:51:08.895499+0000","last_fullsized":"2024-03-11T21:51:08.895499+0000","mapping_epoch":532,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":533,"parent":"0.0","parent_split_bits":0,"last_scrub":"533'12","last_scrub_stamp":"2024-03-11T21:49:55.916961+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:41.544550+0000","last_clean_scrub_stamp":"2024-03-11T21:49:55.916961+0000","objects_scrubbed":1,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,0],"acting":[4,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.9","version":"0'0","reported_seq":1101,"reported_epoch":554,"state":"active+clean","last_fresh":"2024-03-11T21:51:06.961368+0000","last_change":"2024-03-11T21:50:46.940418+0000","last_active":"2024-03-11T21:51:06.961368+0000","last_peered":"2024-03-11T21:51:06.961368+0000","last_clean":"2024-03-11T21:51:06.961368+0000","last_became_active":"2024-03-11T21:49:17.138195+0000","last_became_peered":"2024-03-11T21:49:17.138195+0000","last_unstale":"2024-03-11T21:51:06.961368+0000","last_undegraded":"2024-03-11T21:51:06.961368+0000","last_fullsized":"2024-03-11T21:51:06.961368+0000","mapping_epoch":528,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":529,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:50:46.940341+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:26:50.991712+0000","last_clean_scrub_stamp":"2024-03-11T21:50:46.940341+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:52:02.972249+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,3],"acting":[1,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.f","version":"456'39","reported_seq":1168,"reported_epoch":554,"state":"active+clean","last_fresh":"2024-03-11T21:51:11.993162+0000","last_change":"2024-03-11T21:51:11.993195+0000","last_active":"2024-03-11T21:51:11.993162+0000","last_peered":"2024-03-11T21:51:11.993162+0000","last_clean":"2024-03-11T21:51:11.993162+0000","last_became_active":"2024-03-11T21:49:26.637780+0000","last_became_peered":"2024-03-11T21:49:26.637780+0000","last_unstale":"2024-03-11T21:51:11.993162+0000","last_undegraded":"2024-03-11T21:51:11.993162+0000","last_fullsized":"2024-03-11T21:51:11.993162+0000","mapping_epoch":532,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":533,"parent":"0.0","parent_split_bits":0,"last_scrub":"456'39","last_scrub_stamp":"2024-03-11T21:51:11.993102+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:39.525554+0000","last_clean_scrub_stamp":"2024-03-11T21:51:11.993102+0000","objects_scrubbed":7,"log_size":39,"log_dups_size":0,"ondisk_log_size":39,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:52:19.152297+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":62,"num_read_kb":53,"num_write":40,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":48,"num_bytes_recovered":660,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,4],"acting":[2,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.e","version":"0'0","reported_seq":69,"reported_epoch":554,"state":"active+clean","last_fresh":"2024-03-11T21:51:17.219588+0000","last_change":"2024-03-11T21:51:17.219612+0000","last_active":"2024-03-11T21:51:17.219588+0000","last_peered":"2024-03-11T21:51:17.219588+0000","last_clean":"2024-03-11T21:51:17.219588+0000","last_became_active":"2024-03-11T21:49:26.639018+0000","last_became_peered":"2024-03-11T21:49:26.639018+0000","last_unstale":"2024-03-11T21:51:17.219588+0000","last_undegraded":"2024-03-11T21:51:17.219588+0000","last_fullsized":"2024-03-11T21:51:17.219588+0000","mapping_epoch":532,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":533,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:51:17.219515+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:43.559452+0000","last_clean_scrub_stamp":"2024-03-11T21:51:17.219515+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:52:44.383916+0000","scrub_duration":39,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.d","version":"503'12","reported_seq":1413,"reported_epoch":554,"state":"active+clean","last_fresh":"2024-03-11T21:51:06.722225+0000","last_change":"2024-03-11T21:50:30.688221+0000","last_active":"2024-03-11T21:51:06.722225+0000","last_peered":"2024-03-11T21:51:06.722225+0000","last_clean":"2024-03-11T21:51:06.722225+0000","last_became_active":"2024-03-11T21:48:28.627926+0000","last_became_peered":"2024-03-11T21:48:28.627926+0000","last_unstale":"2024-03-11T21:51:06.722225+0000","last_undegraded":"2024-03-11T21:51:06.722225+0000","last_fullsized":"2024-03-11T21:51:06.722225+0000","mapping_epoch":502,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":503,"parent":"0.0","parent_split_bits":0,"last_scrub":"503'12","last_scrub_stamp":"2024-03-11T21:50:30.688111+0000","last_deep_scrub":"109'9","last_deep_scrub_stamp":"2024-03-11T21:28:27.082975+0000","last_clean_scrub_stamp":"2024-03-11T21:50:30.688111+0000","objects_scrubbed":1,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:51:51.903565+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6,5],"acting":[2,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.a","version":"0'0","reported_seq":1030,"reported_epoch":554,"state":"active+clean","last_fresh":"2024-03-11T21:51:06.961573+0000","last_change":"2024-03-11T21:50:39.942550+0000","last_active":"2024-03-11T21:51:06.961573+0000","last_peered":"2024-03-11T21:51:06.961573+0000","last_clean":"2024-03-11T21:51:06.961573+0000","last_became_active":"2024-03-11T21:49:46.949130+0000","last_became_peered":"2024-03-11T21:49:46.949130+0000","last_unstale":"2024-03-11T21:51:06.961573+0000","last_undegraded":"2024-03-11T21:51:06.961573+0000","last_fullsized":"2024-03-11T21:51:06.961573+0000","mapping_epoch":537,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":538,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:50:39.942447+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:44:27.996522+0000","last_clean_scrub_stamp":"2024-03-11T21:50:39.942447+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:51:53.447270+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,7],"acting":[1,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.c","version":"547'35","reported_seq":78,"reported_epoch":554,"state":"active+clean","last_fresh":"2024-03-11T21:51:06.722507+0000","last_change":"2024-03-11T21:50:16.011651+0000","last_active":"2024-03-11T21:51:06.722507+0000","last_peered":"2024-03-11T21:51:06.722507+0000","last_clean":"2024-03-11T21:51:06.722507+0000","last_became_active":"2024-03-11T21:49:26.652747+0000","last_became_peered":"2024-03-11T21:49:26.652747+0000","last_unstale":"2024-03-11T21:51:06.722507+0000","last_undegraded":"2024-03-11T21:51:06.722507+0000","last_fullsized":"2024-03-11T21:51:06.722507+0000","mapping_epoch":532,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":533,"parent":"0.0","parent_split_bits":0,"last_scrub":"547'35","last_scrub_stamp":"2024-03-11T21:50:16.011524+0000","last_deep_scrub":"547'35","last_deep_scrub_stamp":"2024-03-11T21:50:16.011524+0000","last_clean_scrub_stamp":"2024-03-11T21:50:16.011524+0000","objects_scrubbed":5,"log_size":35,"log_dups_size":0,"ondisk_log_size":35,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:51:45.395920+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":43,"num_read_kb":38,"num_write":30,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.f","version":"0'0","reported_seq":1095,"reported_epoch":554,"state":"active+clean","last_fresh":"2024-03-11T21:51:06.722362+0000","last_change":"2024-03-11T21:50:26.857208+0000","last_active":"2024-03-11T21:51:06.722362+0000","last_peered":"2024-03-11T21:51:06.722362+0000","last_clean":"2024-03-11T21:51:06.722362+0000","last_became_active":"2024-03-11T21:49:26.637689+0000","last_became_peered":"2024-03-11T21:49:26.637689+0000","last_unstale":"2024-03-11T21:51:06.722362+0000","last_undegraded":"2024-03-11T21:51:06.722362+0000","last_fullsized":"2024-03-11T21:51:06.722362+0000","mapping_epoch":532,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":533,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:50:26.857094+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:27:03.010969+0000","last_clean_scrub_stamp":"2024-03-11T21:50:26.857094+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:51:35.841803+0000","scrub_duration":178,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,4],"acting":[2,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":945,"reported_epoch":554,"state":"active+clean","last_fresh":"2024-03-11T21:51:06.961704+0000","last_change":"2024-03-11T21:50:20.931523+0000","last_active":"2024-03-11T21:51:06.961704+0000","last_peered":"2024-03-11T21:51:06.961704+0000","last_clean":"2024-03-11T21:51:06.961704+0000","last_became_active":"2024-03-11T21:49:46.255714+0000","last_became_peered":"2024-03-11T21:49:46.255714+0000","last_unstale":"2024-03-11T21:51:06.961704+0000","last_undegraded":"2024-03-11T21:51:06.961704+0000","last_fullsized":"2024-03-11T21:51:06.961704+0000","mapping_epoch":537,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":538,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:50:20.931438+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:30:22.750053+0000","last_clean_scrub_stamp":"2024-03-11T21:50:20.931438+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:51:29.155091+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.b","version":"0'0","reported_seq":741,"reported_epoch":554,"state":"active+clean","last_fresh":"2024-03-11T21:51:07.587317+0000","last_change":"2024-03-11T21:50:21.184234+0000","last_active":"2024-03-11T21:51:07.587317+0000","last_peered":"2024-03-11T21:51:07.587317+0000","last_clean":"2024-03-11T21:51:07.587317+0000","last_became_active":"2024-03-11T21:49:26.645565+0000","last_became_peered":"2024-03-11T21:49:26.645565+0000","last_unstale":"2024-03-11T21:51:07.587317+0000","last_undegraded":"2024-03-11T21:51:07.587317+0000","last_fullsized":"2024-03-11T21:51:07.587317+0000","mapping_epoch":532,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":533,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:50:21.184074+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:50:21.184074+0000","last_clean_scrub_stamp":"2024-03-11T21:50:21.184074+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:51:28.521325+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,5],"acting":[7,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.d","version":"456'53","reported_seq":79,"reported_epoch":554,"state":"active+clean","last_fresh":"2024-03-11T21:51:06.722981+0000","last_change":"2024-03-11T21:50:33.152694+0000","last_active":"2024-03-11T21:51:06.722981+0000","last_peered":"2024-03-11T21:51:06.722981+0000","last_clean":"2024-03-11T21:51:06.722981+0000","last_became_active":"2024-03-11T21:49:26.645458+0000","last_became_peered":"2024-03-11T21:49:26.645458+0000","last_unstale":"2024-03-11T21:51:06.722981+0000","last_undegraded":"2024-03-11T21:51:06.722981+0000","last_fullsized":"2024-03-11T21:51:06.722981+0000","mapping_epoch":532,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":533,"parent":"0.0","parent_split_bits":0,"last_scrub":"456'53","last_scrub_stamp":"2024-03-11T21:50:33.152576+0000","last_deep_scrub":"165'25","last_deep_scrub_stamp":"2024-03-11T21:32:21.678328+0000","last_clean_scrub_stamp":"2024-03-11T21:50:33.152576+0000","objects_scrubbed":8,"log_size":53,"log_dups_size":0,"ondisk_log_size":53,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:51:35.751007+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":69,"num_read_kb":59,"num_write":46,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,1],"acting":[4,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":723,"reported_epoch":554,"state":"active+clean","last_fresh":"2024-03-11T21:51:07.586464+0000","last_change":"2024-03-11T21:50:19.218149+0000","last_active":"2024-03-11T21:51:07.586464+0000","last_peered":"2024-03-11T21:51:07.586464+0000","last_clean":"2024-03-11T21:51:07.586464+0000","last_became_active":"2024-03-11T21:49:17.149515+0000","last_became_peered":"2024-03-11T21:49:17.149515+0000","last_unstale":"2024-03-11T21:51:07.586464+0000","last_undegraded":"2024-03-11T21:51:07.586464+0000","last_fullsized":"2024-03-11T21:51:07.586464+0000","mapping_epoch":528,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":529,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:50:19.218038+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:49:14.834307+0000","last_clean_scrub_stamp":"2024-03-11T21:50:19.218038+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:51:33.744094+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,3],"acting":[7,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":1078,"reported_epoch":554,"state":"active+clean","last_fresh":"2024-03-11T21:51:06.720890+0000","last_change":"2024-03-11T21:50:27.632429+0000","last_active":"2024-03-11T21:51:06.720890+0000","last_peered":"2024-03-11T21:51:06.720890+0000","last_clean":"2024-03-11T21:51:06.720890+0000","last_became_active":"2024-03-11T21:48:28.627492+0000","last_became_peered":"2024-03-11T21:48:28.627492+0000","last_unstale":"2024-03-11T21:51:06.720890+0000","last_undegraded":"2024-03-11T21:51:06.720890+0000","last_fullsized":"2024-03-11T21:51:06.720890+0000","mapping_epoch":502,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":503,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:50:27.632354+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:41.544550+0000","last_clean_scrub_stamp":"2024-03-11T21:50:27.632354+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:51:35.529300+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,5],"acting":[2,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.c","version":"0'0","reported_seq":1078,"reported_epoch":554,"state":"active+clean","last_fresh":"2024-03-11T21:51:06.721352+0000","last_change":"2024-03-11T21:49:54.834640+0000","last_active":"2024-03-11T21:51:06.721352+0000","last_peered":"2024-03-11T21:51:06.721352+0000","last_clean":"2024-03-11T21:51:06.721352+0000","last_became_active":"2024-03-11T21:48:28.636860+0000","last_became_peered":"2024-03-11T21:48:28.636860+0000","last_unstale":"2024-03-11T21:51:06.721352+0000","last_undegraded":"2024-03-11T21:51:06.721352+0000","last_fullsized":"2024-03-11T21:51:06.721352+0000","mapping_epoch":502,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":503,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:49:54.834561+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:29:30.279910+0000","last_clean_scrub_stamp":"2024-03-11T21:49:54.834561+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:51:12.239609+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,0],"acting":[2,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.a","version":"456'60","reported_seq":1228,"reported_epoch":554,"state":"active+clean","last_fresh":"2024-03-11T21:51:07.586355+0000","last_change":"2024-03-11T21:50:32.207942+0000","last_active":"2024-03-11T21:51:07.586355+0000","last_peered":"2024-03-11T21:51:07.586355+0000","last_clean":"2024-03-11T21:51:07.586355+0000","last_became_active":"2024-03-11T21:34:41.148863+0000","last_became_peered":"2024-03-11T21:34:41.148863+0000","last_unstale":"2024-03-11T21:51:07.586355+0000","last_undegraded":"2024-03-11T21:51:07.586355+0000","last_fullsized":"2024-03-11T21:51:07.586355+0000","mapping_epoch":226,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":227,"parent":"0.0","parent_split_bits":0,"last_scrub":"456'60","last_scrub_stamp":"2024-03-11T21:50:32.207776+0000","last_deep_scrub":"260'44","last_deep_scrub_stamp":"2024-03-11T21:45:32.673622+0000","last_clean_scrub_stamp":"2024-03-11T21:50:32.207776+0000","objects_scrubbed":9,"log_size":60,"log_dups_size":0,"ondisk_log_size":60,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:51:54.831082+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":80,"num_read_kb":69,"num_write":52,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,7],"acting":[6,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":1076,"reported_epoch":554,"state":"active+clean","last_fresh":"2024-03-11T21:51:06.721435+0000","last_change":"2024-03-11T21:50:49.456275+0000","last_active":"2024-03-11T21:51:06.721435+0000","last_peered":"2024-03-11T21:51:06.721435+0000","last_clean":"2024-03-11T21:51:06.721435+0000","last_became_active":"2024-03-11T21:34:41.154507+0000","last_became_peered":"2024-03-11T21:34:41.154507+0000","last_unstale":"2024-03-11T21:51:06.721435+0000","last_undegraded":"2024-03-11T21:51:06.721435+0000","last_fullsized":"2024-03-11T21:51:06.721435+0000","mapping_epoch":226,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":227,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:50:49.456170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:50:49.456170+0000","last_clean_scrub_stamp":"2024-03-11T21:50:49.456170+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:52:15.226895+0000","scrub_duration":83,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":1099,"reported_epoch":554,"state":"active+clean","last_fresh":"2024-03-11T21:51:06.720537+0000","last_change":"2024-03-11T21:50:54.928949+0000","last_active":"2024-03-11T21:51:06.720537+0000","last_peered":"2024-03-11T21:51:06.720537+0000","last_clean":"2024-03-11T21:51:06.720537+0000","last_became_active":"2024-03-11T21:49:26.637683+0000","last_became_peered":"2024-03-11T21:49:26.637683+0000","last_unstale":"2024-03-11T21:51:06.720537+0000","last_undegraded":"2024-03-11T21:51:06.720537+0000","last_fullsized":"2024-03-11T21:51:06.720537+0000","mapping_epoch":532,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":533,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:50:54.928873+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:43:32.115556+0000","last_clean_scrub_stamp":"2024-03-11T21:50:54.928873+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:52:16.926036+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,3],"acting":[2,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.d","version":"0'0","reported_seq":1101,"reported_epoch":554,"state":"active+clean","last_fresh":"2024-03-11T21:51:06.961897+0000","last_change":"2024-03-11T21:50:06.956815+0000","last_active":"2024-03-11T21:51:06.961897+0000","last_peered":"2024-03-11T21:51:06.961897+0000","last_clean":"2024-03-11T21:51:06.961897+0000","last_became_active":"2024-03-11T21:49:26.645233+0000","last_became_peered":"2024-03-11T21:49:26.645233+0000","last_unstale":"2024-03-11T21:51:06.961897+0000","last_undegraded":"2024-03-11T21:51:06.961897+0000","last_fullsized":"2024-03-11T21:51:06.961897+0000","mapping_epoch":532,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":533,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:50:06.956743+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:37.512018+0000","last_clean_scrub_stamp":"2024-03-11T21:50:06.956743+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:51:31.187236+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,3],"acting":[1,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.b","version":"456'21","reported_seq":1194,"reported_epoch":554,"state":"active+clean","last_fresh":"2024-03-11T21:51:06.721227+0000","last_change":"2024-03-11T21:50:31.295800+0000","last_active":"2024-03-11T21:51:06.721227+0000","last_peered":"2024-03-11T21:51:06.721227+0000","last_clean":"2024-03-11T21:51:06.721227+0000","last_became_active":"2024-03-11T21:49:26.643714+0000","last_became_peered":"2024-03-11T21:49:26.643714+0000","last_unstale":"2024-03-11T21:51:06.721227+0000","last_undegraded":"2024-03-11T21:51:06.721227+0000","last_fullsized":"2024-03-11T21:51:06.721227+0000","mapping_epoch":532,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":533,"parent":"0.0","parent_split_bits":0,"last_scrub":"456'21","last_scrub_stamp":"2024-03-11T21:50:31.295707+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:39.525554+0000","last_clean_scrub_stamp":"2024-03-11T21:50:31.295707+0000","objects_scrubbed":4,"log_size":21,"log_dups_size":0,"ondisk_log_size":21,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:51:45.942912+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":35,"num_read_kb":29,"num_write":22,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":24,"num_bytes_recovered":660,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,4],"acting":[0,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":61,"reported_epoch":554,"state":"active+clean","last_fresh":"2024-03-11T21:51:06.722382+0000","last_change":"2024-03-11T21:50:07.044282+0000","last_active":"2024-03-11T21:51:06.722382+0000","last_peered":"2024-03-11T21:51:06.722382+0000","last_clean":"2024-03-11T21:51:06.722382+0000","last_became_active":"2024-03-11T21:49:26.636259+0000","last_became_peered":"2024-03-11T21:49:26.636259+0000","last_unstale":"2024-03-11T21:51:06.722382+0000","last_undegraded":"2024-03-11T21:51:06.722382+0000","last_fullsized":"2024-03-11T21:51:06.722382+0000","mapping_epoch":532,"log_start":"0'0","ondisk_log_start":"0'0","created":72,"last_epoch_clean":533,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:50:07.044182+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:25:43.559452+0000","last_clean_scrub_stamp":"2024-03-11T21:50:07.044182+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:51:30.522244+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,3],"acting":[4,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.e","version":"0'0","reported_seq":1115,"reported_epoch":554,"state":"active+clean","last_fresh":"2024-03-11T21:51:06.722239+0000","last_change":"2024-03-11T21:50:21.703018+0000","last_active":"2024-03-11T21:51:06.722239+0000","last_peered":"2024-03-11T21:51:06.722239+0000","last_clean":"2024-03-11T21:51:06.722239+0000","last_became_active":"2024-03-11T21:49:17.144426+0000","last_became_peered":"2024-03-11T21:49:17.144426+0000","last_unstale":"2024-03-11T21:51:06.722239+0000","last_undegraded":"2024-03-11T21:51:06.722239+0000","last_fullsized":"2024-03-11T21:51:06.722239+0000","mapping_epoch":528,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":529,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-11T21:50:21.702930+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-11T21:50:21.702930+0000","last_clean_scrub_stamp":"2024-03-11T21:50:21.702930+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:51:39.114694+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,7],"acting":[2,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.8","version":"456'46","reported_seq":146,"reported_epoch":554,"state":"active+clean","last_fresh":"2024-03-11T21:51:07.481507+0000","last_change":"2024-03-11T21:50:30.920888+0000","last_active":"2024-03-11T21:51:07.481507+0000","last_peered":"2024-03-11T21:51:07.481507+0000","last_clean":"2024-03-11T21:51:07.481507+0000","last_became_active":"2024-03-11T21:48:28.642553+0000","last_became_peered":"2024-03-11T21:48:28.642553+0000","last_unstale":"2024-03-11T21:51:07.481507+0000","last_undegraded":"2024-03-11T21:51:07.481507+0000","last_fullsized":"2024-03-11T21:51:07.481507+0000","mapping_epoch":502,"log_start":"0'0","ondisk_log_start":"0'0","created":68,"last_epoch_clean":503,"parent":"0.0","parent_split_bits":0,"last_scrub":"456'46","last_scrub_stamp":"2024-03-11T21:50:30.920781+0000","last_deep_scrub":"109'22","last_deep_scrub_stamp":"2024-03-11T21:32:07.573197+0000","last_clean_scrub_stamp":"2024-03-11T21:50:30.920781+0000","objects_scrubbed":7,"log_size":46,"log_dups_size":0,"ondisk_log_size":46,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:51:35.367621+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":61,"num_read_kb":52,"num_write":40,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":28,"num_bytes_recovered":440,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.9","version":"533'13","reported_seq":934,"reported_epoch":554,"state":"active+clean","last_fresh":"2024-03-11T21:51:07.586852+0000","last_change":"2024-03-11T21:50:20.194234+0000","last_active":"2024-03-11T21:51:07.586852+0000","last_peered":"2024-03-11T21:51:07.586852+0000","last_clean":"2024-03-11T21:51:07.586852+0000","last_became_active":"2024-03-11T21:49:26.643974+0000","last_became_peered":"2024-03-11T21:49:26.643974+0000","last_unstale":"2024-03-11T21:51:07.586852+0000","last_undegraded":"2024-03-11T21:51:07.586852+0000","last_fullsized":"2024-03-11T21:51:07.586852+0000","mapping_epoch":532,"log_start":"0'0","ondisk_log_start":"0'0","created":70,"last_epoch_clean":533,"parent":"0.0","parent_split_bits":0,"last_scrub":"533'13","last_scrub_stamp":"2024-03-11T21:50:20.194127+0000","last_deep_scrub":"94'9","last_deep_scrub_stamp":"2024-03-11T21:26:53.422552+0000","last_clean_scrub_stamp":"2024-03-11T21:50:20.194127+0000","objects_scrubbed":1,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-11T21:51:36.985515+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":1531,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":7,"num_read_kb":7,"num_write":10,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":17,"num_bytes_recovered":3207,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":4593,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":6,"ondisk_log_size":6,"up":96,"acting":96,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":198,"num_read_kb":294,"num_write":179,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":2951840,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":2375680,"data_stored":2361472,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":128,"ondisk_log_size":128,"up":3,"acting":3,"num_store_stats":4},{"poolid":3,"num_pg":3,"stat_sum":{"num_bytes":408,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1375,"num_read_kb":1370,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":38,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1224,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":8,"ondisk_log_size":8,"up":9,"acting":9,"num_store_stats":8},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":3702,"num_objects":178,"num_object_clones":0,"num_object_copies":534,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":178,"num_whiteouts":0,"num_read":1535,"num_read_kb":1292,"num_write":989,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":563,"num_bytes_recovered":11116,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":417792,"data_stored":11106,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1101,"ondisk_log_size":1101,"up":96,"acting":96,"num_store_stats":8},{"poolid":5,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":102,"ondisk_log_size":102,"up":96,"acting":96,"num_store_stats":8},{"poolid":6,"num_pg":32,"stat_sum":{"num_bytes":383,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":1162,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1149,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":4,"ondisk_log_size":4,"up":96,"acting":96,"num_store_stats":8}],"osd_stats":[{"osd":7,"up_from":60,"seq":257698038071,"num_pgs":55,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":866844,"kb_used_data":1300,"kb_used_omap":1,"kb_used_meta":865534,"kb_avail":92878308,"statfs":{"total":95995035648,"available":95107387392,"internally_reserved":0,"allocated":1331200,"data_stored":955982,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":886307274},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":50,"seq":214748365117,"num_pgs":43,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":889056,"kb_used_data":1300,"kb_used_omap":1,"kb_used_meta":887742,"kb_avail":92856096,"statfs":{"total":95995035648,"available":95084642304,"internally_reserved":0,"allocated":1331200,"data_stored":956711,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":909048262},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834575191,"num_pgs":52,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1024560,"kb_used_data":744,"kb_used_omap":1,"kb_used_meta":1023806,"kb_avail":92720592,"statfs":{"total":95995035648,"available":94945886208,"internally_reserved":0,"allocated":761856,"data_stored":366231,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":1048377797},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738716,"num_pgs":49,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1011240,"kb_used_data":1312,"kb_used_omap":1,"kb_used_meta":1009918,"kb_avail":92733912,"statfs":{"total":95995035648,"available":94959525888,"internally_reserved":0,"allocated":1343488,"data_stored":956745,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":1034156485},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411665,"num_pgs":42,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":880156,"kb_used_data":724,"kb_used_omap":1,"kb_used_meta":879422,"kb_avail":92864996,"statfs":{"total":95995035648,"available":95093755904,"internally_reserved":0,"allocated":741376,"data_stored":366331,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":900528581},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":277,"seq":1189705941170,"num_pgs":54,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":439792,"kb_used_data":744,"kb_used_omap":4,"kb_used_meta":439035,"kb_avail":93305360,"statfs":{"total":95995035648,"available":95544688640,"internally_reserved":0,"allocated":761856,"data_stored":366565,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4442,"internal_metadata":449572518},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":32,"seq":137438953799,"num_pgs":53,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":663592,"kb_used_data":736,"kb_used_omap":1,"kb_used_meta":662846,"kb_avail":93081560,"statfs":{"total":95995035648,"available":95315517440,"internally_reserved":0,"allocated":753664,"data_stored":365959,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":678754757},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":41,"seq":176093659458,"num_pgs":48,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":875992,"kb_used_data":720,"kb_used_omap":1,"kb_used_meta":875262,"kb_avail":92869160,"statfs":{"total":95995035648,"available":95098019840,"internally_reserved":0,"allocated":737280,"data_stored":365836,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":896268741},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1049,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":46,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":436,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":957,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":574,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":436,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1003,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":92,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":1430,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":1898,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":1238,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":1678,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":1474,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":1100,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":1034,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":1254,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":370,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":383,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":370,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-11T21:51:18.575 INFO:tasks.cephadm.ceph_manager.ceph:recovered! 2024-03-11T21:51:18.575 DEBUG:teuthology.orchestra.run.smithi092:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b shell --fsid 00db0016-dfed-11ee-95c8-87774f69a715 -- ceph config rm mgr mgr_debug_aggressive_pg_num_changes 2024-03-11T21:51:18.820 INFO:teuthology.orchestra.run.smithi092.stderr:Inferring config /var/lib/ceph/00db0016-dfed-11ee-95c8-87774f69a715/mon.a/config 2024-03-11T21:51:18.870 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:51:18 smithi092 ceph-mon[29572]: pgmap v1218: 132 pgs: 132 active+clean; 582 KiB data, 6.3 GiB used, 709 GiB / 715 GiB avail; 944 B/s rd, 0 op/s 2024-03-11T21:51:18.871 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:51:18 smithi092 ceph-mon[29572]: from='client.36338 v1:172.21.15.92:0/1659125042' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-11T21:51:18.871 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:51:18 smithi092 ceph-mon[29572]: 2.f scrub starts 2024-03-11T21:51:18.871 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:51:18 smithi092 ceph-mon[29572]: 2.f scrub ok 2024-03-11T21:51:18.871 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:51:18 smithi092 ceph-mon[37563]: pgmap v1218: 132 pgs: 132 active+clean; 582 KiB data, 6.3 GiB used, 709 GiB / 715 GiB avail; 944 B/s rd, 0 op/s 2024-03-11T21:51:18.871 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:51:18 smithi092 ceph-mon[37563]: from='client.36338 v1:172.21.15.92:0/1659125042' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-11T21:51:18.871 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:51:18 smithi092 ceph-mon[37563]: 2.f scrub starts 2024-03-11T21:51:18.871 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:51:18 smithi092 ceph-mon[37563]: 2.f scrub ok 2024-03-11T21:51:19.086 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:51:18 smithi097 ceph-mon[33317]: pgmap v1218: 132 pgs: 132 active+clean; 582 KiB data, 6.3 GiB used, 709 GiB / 715 GiB avail; 944 B/s rd, 0 op/s 2024-03-11T21:51:19.086 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:51:18 smithi097 ceph-mon[33317]: from='client.36338 v1:172.21.15.92:0/1659125042' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-11T21:51:19.086 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:51:18 smithi097 ceph-mon[33317]: 2.f scrub starts 2024-03-11T21:51:19.086 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:51:18 smithi097 ceph-mon[33317]: 2.f scrub ok 2024-03-11T21:51:20.086 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:51:19 smithi097 ceph-mon[33317]: 3.2 scrub starts 2024-03-11T21:51:20.086 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:51:19 smithi097 ceph-mon[33317]: 3.2 scrub ok 2024-03-11T21:51:20.086 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:51:19 smithi097 ceph-mon[33317]: 2.1c deep-scrub starts 2024-03-11T21:51:20.087 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:51:19 smithi097 ceph-mon[33317]: 2.1c deep-scrub ok 2024-03-11T21:51:20.120 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:51:19 smithi092 ceph-mon[29572]: 3.2 scrub starts 2024-03-11T21:51:20.120 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:51:19 smithi092 ceph-mon[29572]: 3.2 scrub ok 2024-03-11T21:51:20.120 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:51:19 smithi092 ceph-mon[29572]: 2.1c deep-scrub starts 2024-03-11T21:51:20.120 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:51:19 smithi092 ceph-mon[29572]: 2.1c deep-scrub ok 2024-03-11T21:51:20.121 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:51:19 smithi092 ceph-mon[37563]: 3.2 scrub starts 2024-03-11T21:51:20.121 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:51:19 smithi092 ceph-mon[37563]: 3.2 scrub ok 2024-03-11T21:51:20.121 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:51:19 smithi092 ceph-mon[37563]: 2.1c deep-scrub starts 2024-03-11T21:51:20.121 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:51:19 smithi092 ceph-mon[37563]: 2.1c deep-scrub ok 2024-03-11T21:51:20.840 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:51:20 smithi092 ceph-mon[29572]: pgmap v1219: 132 pgs: 132 active+clean; 582 KiB data, 6.3 GiB used, 709 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T21:51:20.840 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:51:20 smithi092 ceph-mon[29572]: 4.15 scrub starts 2024-03-11T21:51:20.840 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:51:20 smithi092 ceph-mon[29572]: 4.15 scrub ok 2024-03-11T21:51:20.840 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:51:20 smithi092 ceph-mon[29572]: 5.2 deep-scrub starts 2024-03-11T21:51:20.840 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:51:20 smithi092 ceph-mon[29572]: 5.2 deep-scrub ok 2024-03-11T21:51:20.840 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:51:20 smithi092 ceph-mon[29572]: 4.1 scrub starts 2024-03-11T21:51:20.840 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:51:20 smithi092 ceph-mon[29572]: 4.1 scrub ok 2024-03-11T21:51:20.841 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:51:20 smithi092 ceph-mon[37563]: pgmap v1219: 132 pgs: 132 active+clean; 582 KiB data, 6.3 GiB used, 709 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T21:51:20.841 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:51:20 smithi092 ceph-mon[37563]: 4.15 scrub starts 2024-03-11T21:51:20.841 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:51:20 smithi092 ceph-mon[37563]: 4.15 scrub ok 2024-03-11T21:51:20.841 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:51:20 smithi092 ceph-mon[37563]: 5.2 deep-scrub starts 2024-03-11T21:51:20.841 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:51:20 smithi092 ceph-mon[37563]: 5.2 deep-scrub ok 2024-03-11T21:51:20.841 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:51:20 smithi092 ceph-mon[37563]: 4.1 scrub starts 2024-03-11T21:51:20.841 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:51:20 smithi092 ceph-mon[37563]: 4.1 scrub ok 2024-03-11T21:51:21.086 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:51:20 smithi097 ceph-mon[33317]: pgmap v1219: 132 pgs: 132 active+clean; 582 KiB data, 6.3 GiB used, 709 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-03-11T21:51:21.086 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:51:20 smithi097 ceph-mon[33317]: 4.15 scrub starts 2024-03-11T21:51:21.086 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:51:20 smithi097 ceph-mon[33317]: 4.15 scrub ok 2024-03-11T21:51:21.086 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:51:20 smithi097 ceph-mon[33317]: 5.2 deep-scrub starts 2024-03-11T21:51:21.086 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:51:20 smithi097 ceph-mon[33317]: 5.2 deep-scrub ok 2024-03-11T21:51:21.086 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:51:20 smithi097 ceph-mon[33317]: 4.1 scrub starts 2024-03-11T21:51:21.087 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:51:20 smithi097 ceph-mon[33317]: 4.1 scrub ok 2024-03-11T21:51:21.120 INFO:journalctl@ceph.mgr.y.smithi092.stdout:Mar 11 21:51:20 smithi092 ceph-00db0016-dfed-11ee-95c8-87774f69a715-mgr-y[29801]: [11/Mar/2024:21:51:20] ENGINE Bus STOPPING 2024-03-11T21:51:21.120 INFO:journalctl@ceph.mgr.y.smithi092.stdout:Mar 11 21:51:20 smithi092 ceph-00db0016-dfed-11ee-95c8-87774f69a715-mgr-y[29801]: [11/Mar/2024:21:51:20] ENGINE HTTP Server cherrypy._cpwsgi_server.CPWSGIServer(('::', 9283)) shut down 2024-03-11T21:51:21.120 INFO:journalctl@ceph.mgr.y.smithi092.stdout:Mar 11 21:51:20 smithi092 ceph-00db0016-dfed-11ee-95c8-87774f69a715-mgr-y[29801]: [11/Mar/2024:21:51:20] ENGINE Bus STOPPED 2024-03-11T21:51:21.121 INFO:journalctl@ceph.mgr.y.smithi092.stdout:Mar 11 21:51:20 smithi092 ceph-00db0016-dfed-11ee-95c8-87774f69a715-mgr-y[29801]: [11/Mar/2024:21:51:20] ENGINE Bus STARTING 2024-03-11T21:51:21.121 INFO:journalctl@ceph.mgr.y.smithi092.stdout:Mar 11 21:51:20 smithi092 ceph-00db0016-dfed-11ee-95c8-87774f69a715-mgr-y[29801]: [11/Mar/2024:21:51:20] ENGINE Serving on http://:::9283 2024-03-11T21:51:21.121 INFO:journalctl@ceph.mgr.y.smithi092.stdout:Mar 11 21:51:20 smithi092 ceph-00db0016-dfed-11ee-95c8-87774f69a715-mgr-y[29801]: [11/Mar/2024:21:51:20] ENGINE Bus STARTED 2024-03-11T21:51:21.567 DEBUG:teuthology.run_tasks:Unwinding manager cephadm 2024-03-11T21:51:21.577 INFO:tasks.cephadm:Teardown begin 2024-03-11T21:51:21.577 DEBUG:teuthology.orchestra.run.smithi092:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-03-11T21:51:21.622 DEBUG:teuthology.orchestra.run.smithi097:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-03-11T21:51:21.696 INFO:tasks.cephadm:Cleaning up testdir ceph.* files... 2024-03-11T21:51:21.697 DEBUG:teuthology.orchestra.run.smithi092:> rm -f /home/ubuntu/cephtest/seed.ceph.conf /home/ubuntu/cephtest/ceph.pub 2024-03-11T21:51:21.739 DEBUG:teuthology.orchestra.run.smithi097:> rm -f /home/ubuntu/cephtest/seed.ceph.conf /home/ubuntu/cephtest/ceph.pub 2024-03-11T21:51:21.757 INFO:tasks.cephadm:Stopping all daemons... 2024-03-11T21:51:21.758 INFO:tasks.cephadm.mon.a:Stopping mon.a... 2024-03-11T21:51:21.758 DEBUG:teuthology.orchestra.run.smithi092:> sudo systemctl stop ceph-00db0016-dfed-11ee-95c8-87774f69a715@mon.a 2024-03-11T21:51:21.804 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:51:21 smithi092 ceph-mon[37563]: from='client.? v1:172.21.15.92:0/2433365262' entity='client.admin' cmd=[{"prefix": "config rm", "who": "mgr", "name": "mgr_debug_aggressive_pg_num_changes"}]: dispatch 2024-03-11T21:51:21.804 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:51:21 smithi092 ceph-mon[37563]: from='client.36344 ' entity='client.admin' cmd=[{"prefix": "config rm", "who": "mgr", "name": "mgr_debug_aggressive_pg_num_changes"}]: dispatch 2024-03-11T21:51:21.804 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:51:21 smithi092 ceph-mon[37563]: from='client.36344 ' entity='client.admin' cmd='[{"prefix": "config rm", "who": "mgr", "name": "mgr_debug_aggressive_pg_num_changes"}]': finished 2024-03-11T21:51:21.804 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:51:21 smithi092 ceph-mon[37563]: from='mgr.14607 v1:172.21.15.92:0/1702717597' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T21:51:21.804 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:51:21 smithi092 ceph-mon[37563]: from='mgr.14607 v1:172.21.15.92:0/1702717597' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T21:51:21.805 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:51:21 smithi092 ceph-mon[37563]: from='mgr.14607 v1:172.21.15.92:0/1702717597' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T21:51:21.805 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:51:21 smithi092 ceph-mon[37563]: from='mgr.14607 ' entity='mgr.y' 2024-03-11T21:51:21.805 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:51:21 smithi092 ceph-mon[37563]: 2.16 scrub starts 2024-03-11T21:51:21.805 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:51:21 smithi092 ceph-mon[37563]: 2.16 scrub ok 2024-03-11T21:51:21.805 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:51:21 smithi092 ceph-mon[37563]: 6.17 scrub starts 2024-03-11T21:51:21.805 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:51:21 smithi092 ceph-mon[37563]: 6.17 scrub ok 2024-03-11T21:51:21.805 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:51:21 smithi092 ceph-mon[29572]: from='client.? v1:172.21.15.92:0/2433365262' entity='client.admin' cmd=[{"prefix": "config rm", "who": "mgr", "name": "mgr_debug_aggressive_pg_num_changes"}]: dispatch 2024-03-11T21:51:21.805 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:51:21 smithi092 ceph-mon[29572]: from='client.36344 ' entity='client.admin' cmd=[{"prefix": "config rm", "who": "mgr", "name": "mgr_debug_aggressive_pg_num_changes"}]: dispatch 2024-03-11T21:51:21.805 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:51:21 smithi092 ceph-mon[29572]: from='client.36344 ' entity='client.admin' cmd='[{"prefix": "config rm", "who": "mgr", "name": "mgr_debug_aggressive_pg_num_changes"}]': finished 2024-03-11T21:51:21.805 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:51:21 smithi092 ceph-mon[29572]: from='mgr.14607 v1:172.21.15.92:0/1702717597' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T21:51:21.805 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:51:21 smithi092 ceph-mon[29572]: from='mgr.14607 v1:172.21.15.92:0/1702717597' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T21:51:21.806 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:51:21 smithi092 ceph-mon[29572]: from='mgr.14607 v1:172.21.15.92:0/1702717597' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T21:51:21.806 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:51:21 smithi092 ceph-mon[29572]: from='mgr.14607 ' entity='mgr.y' 2024-03-11T21:51:21.806 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:51:21 smithi092 ceph-mon[29572]: 2.16 scrub starts 2024-03-11T21:51:21.806 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:51:21 smithi092 ceph-mon[29572]: 2.16 scrub ok 2024-03-11T21:51:21.806 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:51:21 smithi092 ceph-mon[29572]: 6.17 scrub starts 2024-03-11T21:51:21.806 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:51:21 smithi092 ceph-mon[29572]: 6.17 scrub ok 2024-03-11T21:51:22.086 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:51:21 smithi097 ceph-mon[33317]: from='client.? v1:172.21.15.92:0/2433365262' entity='client.admin' cmd=[{"prefix": "config rm", "who": "mgr", "name": "mgr_debug_aggressive_pg_num_changes"}]: dispatch 2024-03-11T21:51:22.086 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:51:21 smithi097 ceph-mon[33317]: from='client.36344 ' entity='client.admin' cmd=[{"prefix": "config rm", "who": "mgr", "name": "mgr_debug_aggressive_pg_num_changes"}]: dispatch 2024-03-11T21:51:22.087 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:51:21 smithi097 ceph-mon[33317]: from='client.36344 ' entity='client.admin' cmd='[{"prefix": "config rm", "who": "mgr", "name": "mgr_debug_aggressive_pg_num_changes"}]': finished 2024-03-11T21:51:22.087 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:51:21 smithi097 ceph-mon[33317]: from='mgr.14607 v1:172.21.15.92:0/1702717597' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-11T21:51:22.087 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:51:21 smithi097 ceph-mon[33317]: from='mgr.14607 v1:172.21.15.92:0/1702717597' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-11T21:51:22.087 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:51:21 smithi097 ceph-mon[33317]: from='mgr.14607 v1:172.21.15.92:0/1702717597' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-11T21:51:22.087 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:51:21 smithi097 ceph-mon[33317]: from='mgr.14607 ' entity='mgr.y' 2024-03-11T21:51:22.087 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:51:21 smithi097 ceph-mon[33317]: 2.16 scrub starts 2024-03-11T21:51:22.088 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:51:21 smithi097 ceph-mon[33317]: 2.16 scrub ok 2024-03-11T21:51:22.088 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:51:21 smithi097 ceph-mon[33317]: 6.17 scrub starts 2024-03-11T21:51:22.088 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:51:21 smithi097 ceph-mon[33317]: 6.17 scrub ok 2024-03-11T21:51:22.102 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:51:21 smithi092 systemd[1]: Stopping Ceph mon.a for 00db0016-dfed-11ee-95c8-87774f69a715... 2024-03-11T21:51:22.370 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:51:22 smithi092 ceph-00db0016-dfed-11ee-95c8-87774f69a715-mon-a[29568]: 2024-03-11T21:51:22.109+0000 7f87a9b94700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.a -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2024-03-11T21:51:22.370 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:51:22 smithi092 ceph-00db0016-dfed-11ee-95c8-87774f69a715-mon-a[29568]: 2024-03-11T21:51:22.110+0000 7f87a9b94700 -1 mon.a@0(leader) e3 *** Got Signal Terminated *** 2024-03-11T21:51:22.370 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:51:22 smithi092 podman[216485]: 2024-03-11 21:51:22.283767823 +0000 UTC m=+0.289732853 container died 4cb262f5a41b5058d15402d9dda414ecb4d6e1da08df8a3f7b60b57c80e321a5 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b, name=ceph-00db0016-dfed-11ee-95c8-87774f69a715-mon-a, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240307, RELEASE=main-2b0101e, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, ceph=True, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-03-11T21:51:22.870 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:51:22 smithi092 podman[216485]: 2024-03-11 21:51:22.67728601 +0000 UTC m=+0.683251039 container cleanup 4cb262f5a41b5058d15402d9dda414ecb4d6e1da08df8a3f7b60b57c80e321a5 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b, name=ceph-00db0016-dfed-11ee-95c8-87774f69a715-mon-a, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, RELEASE=main-2b0101e, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240307, ceph=True, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, GIT_CLEAN=True) 2024-03-11T21:51:22.874 INFO:journalctl@ceph.mon.a.smithi092.stdout:Mar 11 21:51:22 smithi092 bash[216485]: ceph-00db0016-dfed-11ee-95c8-87774f69a715-mon-a 2024-03-11T21:51:23.078 DEBUG:teuthology.orchestra.run.smithi092:> sudo pkill -f 'journalctl -f -n 0 -u ceph-00db0016-dfed-11ee-95c8-87774f69a715@mon.a.service' 2024-03-11T21:51:23.174 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-11T21:51:23.174 INFO:tasks.cephadm.mon.a:Stopped mon.a 2024-03-11T21:51:23.175 INFO:tasks.cephadm.mon.b:Stopping mon.c... 2024-03-11T21:51:23.175 DEBUG:teuthology.orchestra.run.smithi092:> sudo systemctl stop ceph-00db0016-dfed-11ee-95c8-87774f69a715@mon.c 2024-03-11T21:51:23.620 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:51:23 smithi092 systemd[1]: Stopping Ceph mon.c for 00db0016-dfed-11ee-95c8-87774f69a715... 2024-03-11T21:51:23.620 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:51:23 smithi092 ceph-00db0016-dfed-11ee-95c8-87774f69a715-mon-c[37559]: 2024-03-11T21:51:23.464+0000 7f120b3bf700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.c -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2024-03-11T21:51:23.621 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:51:23 smithi092 ceph-00db0016-dfed-11ee-95c8-87774f69a715-mon-c[37559]: 2024-03-11T21:51:23.464+0000 7f120b3bf700 -1 mon.c@2(peon) e3 *** Got Signal Terminated *** 2024-03-11T21:51:23.836 INFO:journalctl@ceph.iscsi.iscsi.a.smithi097.stdout:Mar 11 21:51:23 smithi097 ceph-00db0016-dfed-11ee-95c8-87774f69a715-iscsi-iscsi-a[60943]: debug there is no tcmu-runner data available 2024-03-11T21:51:23.932 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:51:23 smithi092 podman[216608]: 2024-03-11 21:51:23.6488578 +0000 UTC m=+0.285425959 container died f3c58c2e4c882f276ebfab3f9f184559400cdd4a5ebca9bfeb2cdaee68940465 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:8fe9635f3beb0247496d7e321f3068bd10c333c433ef4824d7e1c1a7d3d78379, name=ceph-00db0016-dfed-11ee-95c8-87774f69a715-mon-c, org.label-schema.build-date=20240307, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, ceph=True, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_BRANCH=HEAD, RELEASE=main-2b0101e, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0) 2024-03-11T21:51:24.370 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:51:24 smithi092 podman[216608]: 2024-03-11 21:51:24.165531027 +0000 UTC m=+0.802099231 container cleanup f3c58c2e4c882f276ebfab3f9f184559400cdd4a5ebca9bfeb2cdaee68940465 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:8fe9635f3beb0247496d7e321f3068bd10c333c433ef4824d7e1c1a7d3d78379, name=ceph-00db0016-dfed-11ee-95c8-87774f69a715-mon-c, org.label-schema.schema-version=1.0, ceph=True, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , RELEASE=main-2b0101e, GIT_CLEAN=True, org.label-schema.build-date=20240307, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30) 2024-03-11T21:51:24.370 INFO:journalctl@ceph.mon.c.smithi092.stdout:Mar 11 21:51:24 smithi092 bash[216608]: ceph-00db0016-dfed-11ee-95c8-87774f69a715-mon-c 2024-03-11T21:51:24.498 DEBUG:teuthology.orchestra.run.smithi092:> sudo pkill -f 'journalctl -f -n 0 -u ceph-00db0016-dfed-11ee-95c8-87774f69a715@mon.c.service' 2024-03-11T21:51:24.534 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-11T21:51:24.535 INFO:tasks.cephadm.mon.b:Stopped mon.c 2024-03-11T21:51:24.535 INFO:tasks.cephadm.mon.b:Stopping mon.b... 2024-03-11T21:51:24.535 DEBUG:teuthology.orchestra.run.smithi097:> sudo systemctl stop ceph-00db0016-dfed-11ee-95c8-87774f69a715@mon.b 2024-03-11T21:51:24.836 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:51:24 smithi097 systemd[1]: Stopping Ceph mon.b for 00db0016-dfed-11ee-95c8-87774f69a715... 2024-03-11T21:51:25.272 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:51:24 smithi097 ceph-00db0016-dfed-11ee-95c8-87774f69a715-mon-b[33301]: 2024-03-11T21:51:24.887+0000 7f1d3cd5c700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.b -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2024-03-11T21:51:25.272 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:51:24 smithi097 ceph-00db0016-dfed-11ee-95c8-87774f69a715-mon-b[33301]: 2024-03-11T21:51:24.887+0000 7f1d3cd5c700 -1 mon.b@1(peon) e3 *** Got Signal Terminated *** 2024-03-11T21:51:25.272 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:51:25 smithi097 podman[160350]: 2024-03-11 21:51:25.010580982 +0000 UTC m=+0.231281482 container died e9a6e13a4c8cd04522ab77f3f97498b354b835e680e338c138c81de58300af28 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:8fe9635f3beb0247496d7e321f3068bd10c333c433ef4824d7e1c1a7d3d78379, name=ceph-00db0016-dfed-11ee-95c8-87774f69a715-mon-b, GIT_CLEAN=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, org.label-schema.build-date=20240307, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, ceph=True, RELEASE=main-2b0101e) 2024-03-11T21:51:25.586 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:51:25 smithi097 podman[160350]: 2024-03-11 21:51:25.472274884 +0000 UTC m=+0.692975435 container cleanup e9a6e13a4c8cd04522ab77f3f97498b354b835e680e338c138c81de58300af28 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:8fe9635f3beb0247496d7e321f3068bd10c333c433ef4824d7e1c1a7d3d78379, name=ceph-00db0016-dfed-11ee-95c8-87774f69a715-mon-b, RELEASE=main-2b0101e, GIT_CLEAN=True, ceph=True, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240307, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-03-11T21:51:25.586 INFO:journalctl@ceph.mon.b.smithi097.stdout:Mar 11 21:51:25 smithi097 bash[160350]: ceph-00db0016-dfed-11ee-95c8-87774f69a715-mon-b 2024-03-11T21:51:25.890 DEBUG:teuthology.orchestra.run.smithi097:> sudo pkill -f 'journalctl -f -n 0 -u ceph-00db0016-dfed-11ee-95c8-87774f69a715@mon.b.service' 2024-03-11T21:51:25.948 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-11T21:51:25.948 INFO:tasks.cephadm.mon.b:Stopped mon.b 2024-03-11T21:51:25.948 INFO:tasks.cephadm.mgr.y:Stopping mgr.y... 2024-03-11T21:51:25.949 DEBUG:teuthology.orchestra.run.smithi092:> sudo systemctl stop ceph-00db0016-dfed-11ee-95c8-87774f69a715@mgr.y 2024-03-11T21:51:26.253 INFO:journalctl@ceph.mgr.y.smithi092.stdout:Mar 11 21:51:26 smithi092 systemd[1]: Stopping Ceph mgr.y for 00db0016-dfed-11ee-95c8-87774f69a715... 2024-03-11T21:51:26.529 INFO:journalctl@ceph.mgr.y.smithi092.stdout:Mar 11 21:51:26 smithi092 ceph-00db0016-dfed-11ee-95c8-87774f69a715-mgr-y[29801]: 2024-03-11T21:51:26.260+0000 7f2bcabce700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mgr -n mgr.y -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-11T21:51:26.530 INFO:journalctl@ceph.mgr.y.smithi092.stdout:Mar 11 21:51:26 smithi092 podman[216729]: 2024-03-11 21:51:26.293171038 +0000 UTC m=+0.141595275 container died abde081611ff778737555a9529dd6cbe4f8c3095cb952172ed284c6a45e17a69 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b, name=ceph-00db0016-dfed-11ee-95c8-87774f69a715-mgr-y, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.schema-version=1.0, ceph=True, org.label-schema.license=GPLv2, RELEASE=main-2b0101e, org.label-schema.build-date=20240307, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, GIT_CLEAN=True, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-03-11T21:51:26.870 INFO:journalctl@ceph.mgr.y.smithi092.stdout:Mar 11 21:51:26 smithi092 podman[216729]: 2024-03-11 21:51:26.761957278 +0000 UTC m=+0.610381512 container cleanup abde081611ff778737555a9529dd6cbe4f8c3095cb952172ed284c6a45e17a69 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:2b0101eb74d60610d544cb8a473e23a35ac1cc0b, name=ceph-00db0016-dfed-11ee-95c8-87774f69a715-mgr-y, ceph=True, GIT_CLEAN=True, RELEASE=main-2b0101e, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240307) 2024-03-11T21:51:26.870 INFO:journalctl@ceph.mgr.y.smithi092.stdout:Mar 11 21:51:26 smithi092 bash[216729]: ceph-00db0016-dfed-11ee-95c8-87774f69a715-mgr-y 2024-03-11T21:51:27.187 DEBUG:teuthology.orchestra.run.smithi092:> sudo pkill -f 'journalctl -f -n 0 -u ceph-00db0016-dfed-11ee-95c8-87774f69a715@mgr.y.service' 2024-03-11T21:51:27.222 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-11T21:51:27.223 INFO:tasks.cephadm.mgr.y:Stopped mgr.y 2024-03-11T21:51:27.223 INFO:tasks.cephadm.mgr.x:Stopping mgr.x... 2024-03-11T21:51:27.223 DEBUG:teuthology.orchestra.run.smithi097:> sudo systemctl stop ceph-00db0016-dfed-11ee-95c8-87774f69a715@mgr.x 2024-03-11T21:51:27.565 INFO:journalctl@ceph.mgr.x.smithi097.stdout:Mar 11 21:51:27 smithi097 systemd[1]: Stopping Ceph mgr.x for 00db0016-dfed-11ee-95c8-87774f69a715... 2024-03-11T21:51:27.836 INFO:journalctl@ceph.mgr.x.smithi097.stdout:Mar 11 21:51:27 smithi097 podman[160472]: 2024-03-11 21:51:27.574113991 +0000 UTC m=+0.147933338 container died b3c990ab09b5f1119217e21ef3929d7b4a26c4a3930ba40c7eb2a64b60cc69f2 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:8fe9635f3beb0247496d7e321f3068bd10c333c433ef4824d7e1c1a7d3d78379, name=ceph-00db0016-dfed-11ee-95c8-87774f69a715-mgr-x, org.label-schema.license=GPLv2, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, RELEASE=main-2b0101e, maintainer=Guillaume Abrioux , GIT_CLEAN=True, org.label-schema.build-date=20240307, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD) 2024-03-11T21:51:28.336 INFO:journalctl@ceph.mgr.x.smithi097.stdout:Mar 11 21:51:28 smithi097 podman[160472]: 2024-03-11 21:51:28.051804315 +0000 UTC m=+0.625623669 container cleanup b3c990ab09b5f1119217e21ef3929d7b4a26c4a3930ba40c7eb2a64b60cc69f2 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:8fe9635f3beb0247496d7e321f3068bd10c333c433ef4824d7e1c1a7d3d78379, name=ceph-00db0016-dfed-11ee-95c8-87774f69a715-mgr-x, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, RELEASE=main-2b0101e, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240307, maintainer=Guillaume Abrioux , ceph=True, org.label-schema.license=GPLv2) 2024-03-11T21:51:28.337 INFO:journalctl@ceph.mgr.x.smithi097.stdout:Mar 11 21:51:28 smithi097 bash[160472]: ceph-00db0016-dfed-11ee-95c8-87774f69a715-mgr-x 2024-03-11T21:51:28.554 DEBUG:teuthology.orchestra.run.smithi097:> sudo pkill -f 'journalctl -f -n 0 -u ceph-00db0016-dfed-11ee-95c8-87774f69a715@mgr.x.service' 2024-03-11T21:51:28.594 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-11T21:51:28.594 INFO:tasks.cephadm.mgr.x:Stopped mgr.x 2024-03-11T21:51:28.595 INFO:tasks.cephadm.osd.0:Stopping osd.0... 2024-03-11T21:51:28.595 DEBUG:teuthology.orchestra.run.smithi092:> sudo systemctl stop ceph-00db0016-dfed-11ee-95c8-87774f69a715@osd.0 2024-03-11T21:51:28.925 INFO:journalctl@ceph.osd.0.smithi092.stdout:Mar 11 21:51:28 smithi092 systemd[1]: Stopping Ceph osd.0 for 00db0016-dfed-11ee-95c8-87774f69a715... 2024-03-11T21:51:29.370 INFO:journalctl@ceph.osd.0.smithi092.stdout:Mar 11 21:51:28 smithi092 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd-0[41435]: 2024-03-11T21:51:28.932+0000 7fa415f4d700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.0 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-11T21:51:29.370 INFO:journalctl@ceph.osd.0.smithi092.stdout:Mar 11 21:51:28 smithi092 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd-0[41435]: 2024-03-11T21:51:28.932+0000 7fa415f4d700 -1 osd.0 554 *** Got signal Terminated *** 2024-03-11T21:51:29.370 INFO:journalctl@ceph.osd.0.smithi092.stdout:Mar 11 21:51:28 smithi092 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd-0[41435]: 2024-03-11T21:51:28.932+0000 7fa415f4d700 -1 osd.0 554 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-03-11T21:51:34.833 INFO:journalctl@ceph.osd.0.smithi092.stdout:Mar 11 21:51:34 smithi092 podman[216851]: 2024-03-11 21:51:34.565225771 +0000 UTC m=+5.780363405 container died 5402a35e95dc32d2df56d2af097b3dbfc298b8e90dffd6e208a29ce0e123ea37 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:8fe9635f3beb0247496d7e321f3068bd10c333c433ef4824d7e1c1a7d3d78379, name=ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd-0, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, GIT_CLEAN=True, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=main-2b0101e, org.label-schema.license=GPLv2, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240307, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux ) 2024-03-11T21:51:35.120 INFO:journalctl@ceph.osd.0.smithi092.stdout:Mar 11 21:51:35 smithi092 podman[216851]: 2024-03-11 21:51:35.016407199 +0000 UTC m=+6.231544830 container cleanup 5402a35e95dc32d2df56d2af097b3dbfc298b8e90dffd6e208a29ce0e123ea37 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:8fe9635f3beb0247496d7e321f3068bd10c333c433ef4824d7e1c1a7d3d78379, name=ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd-0, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, GIT_CLEAN=True, io.buildah.version=1.33.5, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=main-2b0101e, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240307, GIT_BRANCH=HEAD, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0) 2024-03-11T21:51:35.120 INFO:journalctl@ceph.osd.0.smithi092.stdout:Mar 11 21:51:35 smithi092 bash[216851]: ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd-0 2024-03-11T21:51:35.776 INFO:journalctl@ceph.osd.0.smithi092.stdout:Mar 11 21:51:35 smithi092 podman[216868]: 2024-03-11 21:51:35.409894814 +0000 UTC m=+0.849080852 container remove 5402a35e95dc32d2df56d2af097b3dbfc298b8e90dffd6e208a29ce0e123ea37 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:8fe9635f3beb0247496d7e321f3068bd10c333c433ef4824d7e1c1a7d3d78379, name=ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd-0, RELEASE=main-2b0101e, maintainer=Guillaume Abrioux , ceph=True, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.build-date=20240307, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, GIT_CLEAN=True, io.buildah.version=1.33.5) 2024-03-11T21:51:36.055 INFO:journalctl@ceph.osd.0.smithi092.stdout:Mar 11 21:51:35 smithi092 podman[216956]: 2024-03-11 21:51:35.686327254 +0000 UTC m=+0.017804817 image pull 4133f8bc41bb201212a28168427a2edb1f6f3795e694bf61031ea9e2f3ff5e87 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:8fe9635f3beb0247496d7e321f3068bd10c333c433ef4824d7e1c1a7d3d78379 2024-03-11T21:51:36.056 INFO:journalctl@ceph.osd.0.smithi092.stdout:Mar 11 21:51:35 smithi092 podman[216956]: 2024-03-11 21:51:35.793861975 +0000 UTC m=+0.125339522 container create d757aa566a2c8f24bb827e7fa43a589b918f33bc73bef79b0bf9bd1a4241b498 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:8fe9635f3beb0247496d7e321f3068bd10c333c433ef4824d7e1c1a7d3d78379, name=ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd-0-deactivate, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, org.label-schema.build-date=20240307, RELEASE=main-2b0101e, GIT_BRANCH=HEAD) 2024-03-11T21:51:36.370 INFO:journalctl@ceph.osd.0.smithi092.stdout:Mar 11 21:51:36 smithi092 podman[216956]: 2024-03-11 21:51:36.063083294 +0000 UTC m=+0.394560848 container init d757aa566a2c8f24bb827e7fa43a589b918f33bc73bef79b0bf9bd1a4241b498 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:8fe9635f3beb0247496d7e321f3068bd10c333c433ef4824d7e1c1a7d3d78379, name=ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd-0-deactivate, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, RELEASE=main-2b0101e, org.label-schema.build-date=20240307, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.license=GPLv2, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30) 2024-03-11T21:51:36.370 INFO:journalctl@ceph.osd.0.smithi092.stdout:Mar 11 21:51:36 smithi092 podman[216956]: 2024-03-11 21:51:36.067818893 +0000 UTC m=+0.399296441 container start d757aa566a2c8f24bb827e7fa43a589b918f33bc73bef79b0bf9bd1a4241b498 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:8fe9635f3beb0247496d7e321f3068bd10c333c433ef4824d7e1c1a7d3d78379, name=ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd-0-deactivate, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240307, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, ceph=True, GIT_BRANCH=HEAD, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , RELEASE=main-2b0101e) 2024-03-11T21:51:36.370 INFO:journalctl@ceph.osd.0.smithi092.stdout:Mar 11 21:51:36 smithi092 podman[216956]: 2024-03-11 21:51:36.196538821 +0000 UTC m=+0.528016452 container attach d757aa566a2c8f24bb827e7fa43a589b918f33bc73bef79b0bf9bd1a4241b498 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:8fe9635f3beb0247496d7e321f3068bd10c333c433ef4824d7e1c1a7d3d78379, name=ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd-0-deactivate, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, GIT_CLEAN=True, org.label-schema.build-date=20240307, RELEASE=main-2b0101e, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, io.buildah.version=1.33.5, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image) 2024-03-11T21:51:37.750 INFO:journalctl@ceph.osd.0.smithi092.stdout:Mar 11 21:51:37 smithi092 podman[216956]: 2024-03-11 21:51:37.429108137 +0000 UTC m=+1.760585684 container died d757aa566a2c8f24bb827e7fa43a589b918f33bc73bef79b0bf9bd1a4241b498 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:8fe9635f3beb0247496d7e321f3068bd10c333c433ef4824d7e1c1a7d3d78379, name=ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd-0-deactivate, RELEASE=main-2b0101e, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, org.label-schema.build-date=20240307, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-03-11T21:51:38.080 DEBUG:teuthology.orchestra.run.smithi092:> sudo pkill -f 'journalctl -f -n 0 -u ceph-00db0016-dfed-11ee-95c8-87774f69a715@osd.0.service' 2024-03-11T21:51:38.123 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-11T21:51:38.124 INFO:tasks.cephadm.osd.0:Stopped osd.0 2024-03-11T21:51:38.124 INFO:tasks.cephadm.osd.1:Stopping osd.1... 2024-03-11T21:51:38.124 DEBUG:teuthology.orchestra.run.smithi092:> sudo systemctl stop ceph-00db0016-dfed-11ee-95c8-87774f69a715@osd.1 2024-03-11T21:51:38.370 INFO:journalctl@ceph.osd.1.smithi092.stdout:Mar 11 21:51:38 smithi092 systemd[1]: Stopping Ceph osd.1 for 00db0016-dfed-11ee-95c8-87774f69a715... 2024-03-11T21:51:38.870 INFO:journalctl@ceph.osd.1.smithi092.stdout:Mar 11 21:51:38 smithi092 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd-1[47398]: 2024-03-11T21:51:38.478+0000 7f704d951700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.1 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-11T21:51:38.870 INFO:journalctl@ceph.osd.1.smithi092.stdout:Mar 11 21:51:38 smithi092 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd-1[47398]: 2024-03-11T21:51:38.478+0000 7f704d951700 -1 osd.1 554 *** Got signal Terminated *** 2024-03-11T21:51:38.870 INFO:journalctl@ceph.osd.1.smithi092.stdout:Mar 11 21:51:38 smithi092 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd-1[47398]: 2024-03-11T21:51:38.478+0000 7f704d951700 -1 osd.1 554 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-03-11T21:51:44.370 INFO:journalctl@ceph.osd.1.smithi092.stdout:Mar 11 21:51:44 smithi092 podman[217077]: 2024-03-11 21:51:44.127717181 +0000 UTC m=+5.766748626 container died 251cced99e8076853a1c819cf2c08ae5f3b8e7442b338953aa123608afbe05c4 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:8fe9635f3beb0247496d7e321f3068bd10c333c433ef4824d7e1c1a7d3d78379, name=ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd-1, org.label-schema.build-date=20240307, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=main-2b0101e, ceph=True, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, GIT_CLEAN=True, CEPH_POINT_RELEASE=) 2024-03-11T21:51:44.795 INFO:journalctl@ceph.osd.1.smithi092.stdout:Mar 11 21:51:44 smithi092 podman[217077]: 2024-03-11 21:51:44.569284098 +0000 UTC m=+6.208315543 container cleanup 251cced99e8076853a1c819cf2c08ae5f3b8e7442b338953aa123608afbe05c4 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:8fe9635f3beb0247496d7e321f3068bd10c333c433ef4824d7e1c1a7d3d78379, name=ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd-1, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240307, GIT_CLEAN=True, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, RELEASE=main-2b0101e, ceph=True, GIT_BRANCH=HEAD) 2024-03-11T21:51:44.796 INFO:journalctl@ceph.osd.1.smithi092.stdout:Mar 11 21:51:44 smithi092 bash[217077]: ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd-1 2024-03-11T21:51:45.120 INFO:journalctl@ceph.osd.1.smithi092.stdout:Mar 11 21:51:44 smithi092 podman[217094]: 2024-03-11 21:51:44.804038932 +0000 UTC m=+0.669174552 container remove 251cced99e8076853a1c819cf2c08ae5f3b8e7442b338953aa123608afbe05c4 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:8fe9635f3beb0247496d7e321f3068bd10c333c433ef4824d7e1c1a7d3d78379, name=ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd-1, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, org.label-schema.build-date=20240307, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, RELEASE=main-2b0101e, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD) 2024-03-11T21:51:45.491 INFO:journalctl@ceph.osd.1.smithi092.stdout:Mar 11 21:51:45 smithi092 podman[217185]: 2024-03-11 21:51:45.106239474 +0000 UTC m=+0.017417583 image pull 4133f8bc41bb201212a28168427a2edb1f6f3795e694bf61031ea9e2f3ff5e87 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:8fe9635f3beb0247496d7e321f3068bd10c333c433ef4824d7e1c1a7d3d78379 2024-03-11T21:51:45.491 INFO:journalctl@ceph.osd.1.smithi092.stdout:Mar 11 21:51:45 smithi092 podman[217185]: 2024-03-11 21:51:45.213148643 +0000 UTC m=+0.124326732 container create 05bf34016278bbd6e347b9bcb6d6206eae114f71b34371816c954a04a75aa60f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:8fe9635f3beb0247496d7e321f3068bd10c333c433ef4824d7e1c1a7d3d78379, name=ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd-1-deactivate, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240307, org.label-schema.license=GPLv2, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=main-2b0101e, ceph=True, io.buildah.version=1.33.5) 2024-03-11T21:51:45.492 INFO:journalctl@ceph.osd.1.smithi092.stdout:Mar 11 21:51:45 smithi092 podman[217185]: 2024-03-11 21:51:45.42409429 +0000 UTC m=+0.335272382 container init 05bf34016278bbd6e347b9bcb6d6206eae114f71b34371816c954a04a75aa60f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:8fe9635f3beb0247496d7e321f3068bd10c333c433ef4824d7e1c1a7d3d78379, name=ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd-1-deactivate, GIT_CLEAN=True, ceph=True, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240307, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=main-2b0101e) 2024-03-11T21:51:45.492 INFO:journalctl@ceph.osd.1.smithi092.stdout:Mar 11 21:51:45 smithi092 podman[217185]: 2024-03-11 21:51:45.427038715 +0000 UTC m=+0.338216804 container start 05bf34016278bbd6e347b9bcb6d6206eae114f71b34371816c954a04a75aa60f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:8fe9635f3beb0247496d7e321f3068bd10c333c433ef4824d7e1c1a7d3d78379, name=ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd-1-deactivate, ceph=True, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240307, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=main-2b0101e) 2024-03-11T21:51:45.870 INFO:journalctl@ceph.osd.1.smithi092.stdout:Mar 11 21:51:45 smithi092 podman[217185]: 2024-03-11 21:51:45.498995274 +0000 UTC m=+0.410173392 container attach 05bf34016278bbd6e347b9bcb6d6206eae114f71b34371816c954a04a75aa60f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:8fe9635f3beb0247496d7e321f3068bd10c333c433ef4824d7e1c1a7d3d78379, name=ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd-1-deactivate, org.label-schema.build-date=20240307, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, RELEASE=main-2b0101e, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, ceph=True, io.buildah.version=1.33.5, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True) 2024-03-11T21:51:45.870 INFO:journalctl@ceph.osd.1.smithi092.stdout:Mar 11 21:51:45 smithi092 podman[217185]: 2024-03-11 21:51:45.714673159 +0000 UTC m=+0.625851251 container died 05bf34016278bbd6e347b9bcb6d6206eae114f71b34371816c954a04a75aa60f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:8fe9635f3beb0247496d7e321f3068bd10c333c433ef4824d7e1c1a7d3d78379, name=ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd-1-deactivate, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, RELEASE=main-2b0101e, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240307, GIT_CLEAN=True, ceph=True, io.buildah.version=1.33.5) 2024-03-11T21:51:46.417 DEBUG:teuthology.orchestra.run.smithi092:> sudo pkill -f 'journalctl -f -n 0 -u ceph-00db0016-dfed-11ee-95c8-87774f69a715@osd.1.service' 2024-03-11T21:51:46.467 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-11T21:51:46.468 INFO:tasks.cephadm.osd.1:Stopped osd.1 2024-03-11T21:51:46.468 INFO:tasks.cephadm.osd.2:Stopping osd.2... 2024-03-11T21:51:46.468 DEBUG:teuthology.orchestra.run.smithi092:> sudo systemctl stop ceph-00db0016-dfed-11ee-95c8-87774f69a715@osd.2 2024-03-11T21:51:46.772 INFO:journalctl@ceph.osd.2.smithi092.stdout:Mar 11 21:51:46 smithi092 systemd[1]: Stopping Ceph osd.2 for 00db0016-dfed-11ee-95c8-87774f69a715... 2024-03-11T21:51:47.112 INFO:journalctl@ceph.osd.2.smithi092.stdout:Mar 11 21:51:46 smithi092 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd-2[53445]: 2024-03-11T21:51:46.779+0000 7f80cb4ad700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.2 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-11T21:51:47.113 INFO:journalctl@ceph.osd.2.smithi092.stdout:Mar 11 21:51:46 smithi092 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd-2[53445]: 2024-03-11T21:51:46.779+0000 7f80cb4ad700 -1 osd.2 554 *** Got signal Terminated *** 2024-03-11T21:51:47.113 INFO:journalctl@ceph.osd.2.smithi092.stdout:Mar 11 21:51:46 smithi092 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd-2[53445]: 2024-03-11T21:51:46.779+0000 7f80cb4ad700 -1 osd.2 554 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-03-11T21:51:52.556 INFO:journalctl@ceph.osd.2.smithi092.stdout:Mar 11 21:51:52 smithi092 podman[217291]: 2024-03-11 21:51:52.286626789 +0000 UTC m=+5.614849713 container died 8a38dbae5dea710b01ea704f4c7f8ba2780054af6ed1de82d1eb51f6ddf6d4b4 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:8fe9635f3beb0247496d7e321f3068bd10c333c433ef4824d7e1c1a7d3d78379, name=ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd-2, GIT_CLEAN=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, RELEASE=main-2b0101e, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, org.label-schema.build-date=20240307, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image) 2024-03-11T21:51:52.870 INFO:journalctl@ceph.osd.2.smithi092.stdout:Mar 11 21:51:52 smithi092 podman[217291]: 2024-03-11 21:51:52.713255548 +0000 UTC m=+6.041478517 container cleanup 8a38dbae5dea710b01ea704f4c7f8ba2780054af6ed1de82d1eb51f6ddf6d4b4 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:8fe9635f3beb0247496d7e321f3068bd10c333c433ef4824d7e1c1a7d3d78379, name=ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd-2, GIT_CLEAN=True, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, org.label-schema.build-date=20240307, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, RELEASE=main-2b0101e, CEPH_POINT_RELEASE=, ceph=True, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux ) 2024-03-11T21:51:52.870 INFO:journalctl@ceph.osd.2.smithi092.stdout:Mar 11 21:51:52 smithi092 bash[217291]: ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd-2 2024-03-11T21:51:53.313 INFO:journalctl@ceph.osd.2.smithi092.stdout:Mar 11 21:51:52 smithi092 podman[217310]: 2024-03-11 21:51:52.901608582 +0000 UTC m=+0.612557033 container remove 8a38dbae5dea710b01ea704f4c7f8ba2780054af6ed1de82d1eb51f6ddf6d4b4 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:8fe9635f3beb0247496d7e321f3068bd10c333c433ef4824d7e1c1a7d3d78379, name=ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd-2, GIT_BRANCH=HEAD, GIT_CLEAN=True, RELEASE=main-2b0101e, ceph=True, org.label-schema.build-date=20240307, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30) 2024-03-11T21:51:53.568 INFO:journalctl@ceph.osd.2.smithi092.stdout:Mar 11 21:51:53 smithi092 podman[217400]: 2024-03-11 21:51:53.225773321 +0000 UTC m=+0.019902843 image pull 4133f8bc41bb201212a28168427a2edb1f6f3795e694bf61031ea9e2f3ff5e87 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:8fe9635f3beb0247496d7e321f3068bd10c333c433ef4824d7e1c1a7d3d78379 2024-03-11T21:51:53.568 INFO:journalctl@ceph.osd.2.smithi092.stdout:Mar 11 21:51:53 smithi092 podman[217400]: 2024-03-11 21:51:53.348644466 +0000 UTC m=+0.142773988 container create 41a650f7ab97bc46abe4c0045280db9b0bf800aa7c874e2963406e33f44b9621 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:8fe9635f3beb0247496d7e321f3068bd10c333c433ef4824d7e1c1a7d3d78379, name=ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd-2-deactivate, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, org.label-schema.build-date=20240307, GIT_CLEAN=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, RELEASE=main-2b0101e, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5) 2024-03-11T21:51:53.568 INFO:journalctl@ceph.osd.2.smithi092.stdout:Mar 11 21:51:53 smithi092 podman[217400]: 2024-03-11 21:51:53.576155283 +0000 UTC m=+0.370284805 container init 41a650f7ab97bc46abe4c0045280db9b0bf800aa7c874e2963406e33f44b9621 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:8fe9635f3beb0247496d7e321f3068bd10c333c433ef4824d7e1c1a7d3d78379, name=ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd-2-deactivate, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.build-date=20240307, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, GIT_CLEAN=True, RELEASE=main-2b0101e) 2024-03-11T21:51:53.865 INFO:journalctl@ceph.osd.2.smithi092.stdout:Mar 11 21:51:53 smithi092 podman[217400]: 2024-03-11 21:51:53.579209196 +0000 UTC m=+0.373338725 container start 41a650f7ab97bc46abe4c0045280db9b0bf800aa7c874e2963406e33f44b9621 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:8fe9635f3beb0247496d7e321f3068bd10c333c433ef4824d7e1c1a7d3d78379, name=ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd-2-deactivate, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , GIT_CLEAN=True, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240307, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, RELEASE=main-2b0101e) 2024-03-11T21:51:53.865 INFO:journalctl@ceph.osd.2.smithi092.stdout:Mar 11 21:51:53 smithi092 podman[217400]: 2024-03-11 21:51:53.684179245 +0000 UTC m=+0.478308768 container attach 41a650f7ab97bc46abe4c0045280db9b0bf800aa7c874e2963406e33f44b9621 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:8fe9635f3beb0247496d7e321f3068bd10c333c433ef4824d7e1c1a7d3d78379, name=ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd-2-deactivate, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240307, RELEASE=main-2b0101e, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, ceph=True, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD) 2024-03-11T21:51:54.120 INFO:journalctl@ceph.osd.2.smithi092.stdout:Mar 11 21:51:53 smithi092 podman[217400]: 2024-03-11 21:51:53.873253813 +0000 UTC m=+0.667383334 container died 41a650f7ab97bc46abe4c0045280db9b0bf800aa7c874e2963406e33f44b9621 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:8fe9635f3beb0247496d7e321f3068bd10c333c433ef4824d7e1c1a7d3d78379, name=ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd-2-deactivate, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240307, ceph=True, GIT_BRANCH=HEAD, RELEASE=main-2b0101e, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS) 2024-03-11T21:51:54.549 DEBUG:teuthology.orchestra.run.smithi092:> sudo pkill -f 'journalctl -f -n 0 -u ceph-00db0016-dfed-11ee-95c8-87774f69a715@osd.2.service' 2024-03-11T21:51:54.593 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-11T21:51:54.593 INFO:tasks.cephadm.osd.2:Stopped osd.2 2024-03-11T21:51:54.593 INFO:tasks.cephadm.osd.3:Stopping osd.3... 2024-03-11T21:51:54.593 DEBUG:teuthology.orchestra.run.smithi092:> sudo systemctl stop ceph-00db0016-dfed-11ee-95c8-87774f69a715@osd.3 2024-03-11T21:51:54.870 INFO:journalctl@ceph.osd.3.smithi092.stdout:Mar 11 21:51:54 smithi092 systemd[1]: Stopping Ceph osd.3 for 00db0016-dfed-11ee-95c8-87774f69a715... 2024-03-11T21:51:55.336 INFO:journalctl@ceph.osd.4.smithi097.stdout:Mar 11 21:51:55 smithi097 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd-4[38221]: 2024-03-11T21:51:55.031+0000 7f9fa976a700 -1 osd.4 554 heartbeat_check: no reply from 172.21.15.92:6803 osd.0 since back 2024-03-11T21:51:31.363947+0000 front 2024-03-11T21:51:31.364021+0000 (oldest deadline 2024-03-11T21:51:54.863875+0000) 2024-03-11T21:51:55.370 INFO:journalctl@ceph.osd.3.smithi092.stdout:Mar 11 21:51:54 smithi092 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd-3[134074]: 2024-03-11T21:51:54.939+0000 7f07e3944700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.3 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-11T21:51:55.370 INFO:journalctl@ceph.osd.3.smithi092.stdout:Mar 11 21:51:54 smithi092 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd-3[134074]: 2024-03-11T21:51:54.939+0000 7f07e3944700 -1 osd.3 554 *** Got signal Terminated *** 2024-03-11T21:51:55.370 INFO:journalctl@ceph.osd.3.smithi092.stdout:Mar 11 21:51:54 smithi092 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd-3[134074]: 2024-03-11T21:51:54.939+0000 7f07e3944700 -1 osd.3 554 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-03-11T21:51:56.336 INFO:journalctl@ceph.osd.4.smithi097.stdout:Mar 11 21:51:56 smithi097 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd-4[38221]: 2024-03-11T21:51:56.044+0000 7f9fa976a700 -1 osd.4 554 heartbeat_check: no reply from 172.21.15.92:6803 osd.0 since back 2024-03-11T21:51:31.363947+0000 front 2024-03-11T21:51:31.364021+0000 (oldest deadline 2024-03-11T21:51:54.863875+0000) 2024-03-11T21:51:56.336 INFO:journalctl@ceph.osd.5.smithi097.stdout:Mar 11 21:51:56 smithi097 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd-5[44269]: 2024-03-11T21:51:56.070+0000 7f6c4e1e6700 -1 osd.5 554 heartbeat_check: no reply from 172.21.15.92:6803 osd.0 since back 2024-03-11T21:51:31.785021+0000 front 2024-03-11T21:51:31.785129+0000 (oldest deadline 2024-03-11T21:51:55.885103+0000) 2024-03-11T21:51:57.336 INFO:journalctl@ceph.osd.5.smithi097.stdout:Mar 11 21:51:57 smithi097 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd-5[44269]: 2024-03-11T21:51:57.092+0000 7f6c4e1e6700 -1 osd.5 554 heartbeat_check: no reply from 172.21.15.92:6803 osd.0 since back 2024-03-11T21:51:31.785021+0000 front 2024-03-11T21:51:31.785129+0000 (oldest deadline 2024-03-11T21:51:55.885103+0000) 2024-03-11T21:51:57.336 INFO:journalctl@ceph.osd.4.smithi097.stdout:Mar 11 21:51:57 smithi097 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd-4[38221]: 2024-03-11T21:51:57.045+0000 7f9fa976a700 -1 osd.4 554 heartbeat_check: no reply from 172.21.15.92:6803 osd.0 since back 2024-03-11T21:51:31.363947+0000 front 2024-03-11T21:51:31.364021+0000 (oldest deadline 2024-03-11T21:51:54.863875+0000) 2024-03-11T21:51:58.337 INFO:journalctl@ceph.osd.5.smithi097.stdout:Mar 11 21:51:58 smithi097 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd-5[44269]: 2024-03-11T21:51:58.106+0000 7f6c4e1e6700 -1 osd.5 554 heartbeat_check: no reply from 172.21.15.92:6803 osd.0 since back 2024-03-11T21:51:31.785021+0000 front 2024-03-11T21:51:31.785129+0000 (oldest deadline 2024-03-11T21:51:55.885103+0000) 2024-03-11T21:51:58.337 INFO:journalctl@ceph.osd.4.smithi097.stdout:Mar 11 21:51:58 smithi097 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd-4[38221]: 2024-03-11T21:51:58.070+0000 7f9fa976a700 -1 osd.4 554 heartbeat_check: no reply from 172.21.15.92:6803 osd.0 since back 2024-03-11T21:51:31.363947+0000 front 2024-03-11T21:51:31.364021+0000 (oldest deadline 2024-03-11T21:51:54.863875+0000) 2024-03-11T21:51:58.337 INFO:journalctl@ceph.osd.6.smithi097.stdout:Mar 11 21:51:58 smithi097 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd-6[50273]: 2024-03-11T21:51:58.014+0000 7f51a9328700 -1 osd.6 554 heartbeat_check: no reply from 172.21.15.92:6803 osd.0 since back 2024-03-11T21:51:31.564540+0000 front 2024-03-11T21:51:31.564552+0000 (oldest deadline 2024-03-11T21:51:57.464467+0000) 2024-03-11T21:51:58.370 INFO:journalctl@ceph.osd.3.smithi092.stdout:Mar 11 21:51:58 smithi092 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd-3[134074]: 2024-03-11T21:51:58.005+0000 7f07dbf30700 -1 osd.3 554 heartbeat_check: no reply from 172.21.15.92:6803 osd.0 since back 2024-03-11T21:51:32.152594+0000 front 2024-03-11T21:51:32.152526+0000 (oldest deadline 2024-03-11T21:51:57.452659+0000) 2024-03-11T21:51:58.836 INFO:journalctl@ceph.osd.7.smithi097.stdout:Mar 11 21:51:58 smithi097 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd-7[56319]: 2024-03-11T21:51:58.460+0000 7ffbb45e2700 -1 osd.7 554 heartbeat_check: no reply from 172.21.15.92:6803 osd.0 since back 2024-03-11T21:51:32.154875+0000 front 2024-03-11T21:51:32.154922+0000 (oldest deadline 2024-03-11T21:51:57.455053+0000) 2024-03-11T21:51:59.336 INFO:journalctl@ceph.osd.5.smithi097.stdout:Mar 11 21:51:59 smithi097 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd-5[44269]: 2024-03-11T21:51:59.128+0000 7f6c4e1e6700 -1 osd.5 554 heartbeat_check: no reply from 172.21.15.92:6803 osd.0 since back 2024-03-11T21:51:31.785021+0000 front 2024-03-11T21:51:31.785129+0000 (oldest deadline 2024-03-11T21:51:55.885103+0000) 2024-03-11T21:51:59.337 INFO:journalctl@ceph.osd.6.smithi097.stdout:Mar 11 21:51:58 smithi097 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd-6[50273]: 2024-03-11T21:51:58.983+0000 7f51a9328700 -1 osd.6 554 heartbeat_check: no reply from 172.21.15.92:6803 osd.0 since back 2024-03-11T21:51:31.564540+0000 front 2024-03-11T21:51:31.564552+0000 (oldest deadline 2024-03-11T21:51:57.464467+0000) 2024-03-11T21:51:59.337 INFO:journalctl@ceph.osd.4.smithi097.stdout:Mar 11 21:51:59 smithi097 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd-4[38221]: 2024-03-11T21:51:59.044+0000 7f9fa976a700 -1 osd.4 554 heartbeat_check: no reply from 172.21.15.92:6803 osd.0 since back 2024-03-11T21:51:31.363947+0000 front 2024-03-11T21:51:31.364021+0000 (oldest deadline 2024-03-11T21:51:54.863875+0000) 2024-03-11T21:51:59.370 INFO:journalctl@ceph.osd.3.smithi092.stdout:Mar 11 21:51:58 smithi092 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd-3[134074]: 2024-03-11T21:51:58.956+0000 7f07dbf30700 -1 osd.3 554 heartbeat_check: no reply from 172.21.15.92:6803 osd.0 since back 2024-03-11T21:51:32.152594+0000 front 2024-03-11T21:51:32.152526+0000 (oldest deadline 2024-03-11T21:51:57.452659+0000) 2024-03-11T21:51:59.836 INFO:journalctl@ceph.osd.7.smithi097.stdout:Mar 11 21:51:59 smithi097 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd-7[56319]: 2024-03-11T21:51:59.455+0000 7ffbb45e2700 -1 osd.7 554 heartbeat_check: no reply from 172.21.15.92:6803 osd.0 since back 2024-03-11T21:51:32.154875+0000 front 2024-03-11T21:51:32.154922+0000 (oldest deadline 2024-03-11T21:51:57.455053+0000) 2024-03-11T21:52:00.336 INFO:journalctl@ceph.osd.4.smithi097.stdout:Mar 11 21:52:00 smithi097 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd-4[38221]: 2024-03-11T21:52:00.063+0000 7f9fa976a700 -1 osd.4 554 heartbeat_check: no reply from 172.21.15.92:6803 osd.0 since back 2024-03-11T21:51:31.363947+0000 front 2024-03-11T21:51:31.364021+0000 (oldest deadline 2024-03-11T21:51:54.863875+0000) 2024-03-11T21:52:00.337 INFO:journalctl@ceph.osd.5.smithi097.stdout:Mar 11 21:52:00 smithi097 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd-5[44269]: 2024-03-11T21:52:00.167+0000 7f6c4e1e6700 -1 osd.5 554 heartbeat_check: no reply from 172.21.15.92:6803 osd.0 since back 2024-03-11T21:51:31.785021+0000 front 2024-03-11T21:51:31.785129+0000 (oldest deadline 2024-03-11T21:51:55.885103+0000) 2024-03-11T21:52:00.337 INFO:journalctl@ceph.osd.6.smithi097.stdout:Mar 11 21:52:00 smithi097 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd-6[50273]: 2024-03-11T21:52:00.016+0000 7f51a9328700 -1 osd.6 554 heartbeat_check: no reply from 172.21.15.92:6803 osd.0 since back 2024-03-11T21:51:31.564540+0000 front 2024-03-11T21:51:31.564552+0000 (oldest deadline 2024-03-11T21:51:57.464467+0000) 2024-03-11T21:52:00.821 INFO:journalctl@ceph.osd.3.smithi092.stdout:Mar 11 21:52:00 smithi092 podman[217505]: 2024-03-11 21:52:00.568267096 +0000 UTC m=+5.744152691 container died ea196e969b3c874af0c911ce9d141e04fc33650a47e5e97222467872860add15 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:8fe9635f3beb0247496d7e321f3068bd10c333c433ef4824d7e1c1a7d3d78379, name=ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd-3, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240307, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, RELEASE=main-2b0101e) 2024-03-11T21:52:00.836 INFO:journalctl@ceph.osd.7.smithi097.stdout:Mar 11 21:52:00 smithi097 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd-7[56319]: 2024-03-11T21:52:00.450+0000 7ffbb45e2700 -1 osd.7 554 heartbeat_check: no reply from 172.21.15.92:6803 osd.0 since back 2024-03-11T21:51:32.154875+0000 front 2024-03-11T21:51:32.154922+0000 (oldest deadline 2024-03-11T21:51:57.455053+0000) 2024-03-11T21:52:01.120 INFO:journalctl@ceph.osd.3.smithi092.stdout:Mar 11 21:52:00 smithi092 podman[217505]: 2024-03-11 21:52:00.981342732 +0000 UTC m=+6.157228324 container cleanup ea196e969b3c874af0c911ce9d141e04fc33650a47e5e97222467872860add15 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:8fe9635f3beb0247496d7e321f3068bd10c333c433ef4824d7e1c1a7d3d78379, name=ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd-3, org.label-schema.build-date=20240307, org.label-schema.vendor=CentOS, GIT_CLEAN=True, GIT_BRANCH=HEAD, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, RELEASE=main-2b0101e, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , ceph=True) 2024-03-11T21:52:01.120 INFO:journalctl@ceph.osd.3.smithi092.stdout:Mar 11 21:52:00 smithi092 bash[217505]: ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd-3 2024-03-11T21:52:01.336 INFO:journalctl@ceph.osd.4.smithi097.stdout:Mar 11 21:52:01 smithi097 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd-4[38221]: 2024-03-11T21:52:01.068+0000 7f9fa976a700 -1 osd.4 554 heartbeat_check: no reply from 172.21.15.92:6803 osd.0 since back 2024-03-11T21:51:31.363947+0000 front 2024-03-11T21:51:31.364021+0000 (oldest deadline 2024-03-11T21:51:54.863875+0000) 2024-03-11T21:52:01.336 INFO:journalctl@ceph.osd.6.smithi097.stdout:Mar 11 21:52:01 smithi097 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd-6[50273]: 2024-03-11T21:52:01.005+0000 7f51a9328700 -1 osd.6 554 heartbeat_check: no reply from 172.21.15.92:6803 osd.0 since back 2024-03-11T21:51:31.564540+0000 front 2024-03-11T21:51:31.564552+0000 (oldest deadline 2024-03-11T21:51:57.464467+0000) 2024-03-11T21:52:01.337 INFO:journalctl@ceph.osd.5.smithi097.stdout:Mar 11 21:52:01 smithi097 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd-5[44269]: 2024-03-11T21:52:01.118+0000 7f6c4e1e6700 -1 osd.5 554 heartbeat_check: no reply from 172.21.15.92:6803 osd.0 since back 2024-03-11T21:51:31.785021+0000 front 2024-03-11T21:51:31.785129+0000 (oldest deadline 2024-03-11T21:51:55.885103+0000) 2024-03-11T21:52:01.566 INFO:journalctl@ceph.osd.3.smithi092.stdout:Mar 11 21:52:01 smithi092 podman[217524]: 2024-03-11 21:52:01.166088118 +0000 UTC m=+0.587539427 container remove ea196e969b3c874af0c911ce9d141e04fc33650a47e5e97222467872860add15 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:8fe9635f3beb0247496d7e321f3068bd10c333c433ef4824d7e1c1a7d3d78379, name=ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd-3, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240307, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, GIT_CLEAN=True, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, ceph=True, RELEASE=main-2b0101e) 2024-03-11T21:52:01.836 INFO:journalctl@ceph.osd.7.smithi097.stdout:Mar 11 21:52:01 smithi097 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd-7[56319]: 2024-03-11T21:52:01.426+0000 7ffbb45e2700 -1 osd.7 554 heartbeat_check: no reply from 172.21.15.92:6803 osd.0 since back 2024-03-11T21:51:32.154875+0000 front 2024-03-11T21:51:32.154922+0000 (oldest deadline 2024-03-11T21:51:57.455053+0000) 2024-03-11T21:52:01.853 INFO:journalctl@ceph.osd.3.smithi092.stdout:Mar 11 21:52:01 smithi092 podman[217615]: 2024-03-11 21:52:01.575040606 +0000 UTC m=+0.102958777 container create b41a3aefa54899db9f0d76f5de83413d3f1b8ac0dd34bce62e59f0ea4781d057 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:8fe9635f3beb0247496d7e321f3068bd10c333c433ef4824d7e1c1a7d3d78379, name=ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd-3-deactivate, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, RELEASE=main-2b0101e, org.label-schema.schema-version=1.0, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240307, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS) 2024-03-11T21:52:01.853 INFO:journalctl@ceph.osd.3.smithi092.stdout:Mar 11 21:52:01 smithi092 podman[217615]: 2024-03-11 21:52:01.490245784 +0000 UTC m=+0.018163974 image pull 4133f8bc41bb201212a28168427a2edb1f6f3795e694bf61031ea9e2f3ff5e87 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:8fe9635f3beb0247496d7e321f3068bd10c333c433ef4824d7e1c1a7d3d78379 2024-03-11T21:52:01.853 INFO:journalctl@ceph.osd.3.smithi092.stdout:Mar 11 21:52:01 smithi092 podman[217615]: 2024-03-11 21:52:01.777279753 +0000 UTC m=+0.305197938 container init b41a3aefa54899db9f0d76f5de83413d3f1b8ac0dd34bce62e59f0ea4781d057 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:8fe9635f3beb0247496d7e321f3068bd10c333c433ef4824d7e1c1a7d3d78379, name=ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd-3-deactivate, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240307, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, ceph=True, RELEASE=main-2b0101e) 2024-03-11T21:52:01.853 INFO:journalctl@ceph.osd.3.smithi092.stdout:Mar 11 21:52:01 smithi092 podman[217615]: 2024-03-11 21:52:01.780412953 +0000 UTC m=+0.308331128 container start b41a3aefa54899db9f0d76f5de83413d3f1b8ac0dd34bce62e59f0ea4781d057 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:8fe9635f3beb0247496d7e321f3068bd10c333c433ef4824d7e1c1a7d3d78379, name=ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd-3-deactivate, RELEASE=main-2b0101e, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, ceph=True, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, GIT_CLEAN=True, org.label-schema.build-date=20240307, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image) 2024-03-11T21:52:01.853 INFO:journalctl@ceph.osd.3.smithi092.stdout:Mar 11 21:52:01 smithi092 podman[217615]: 2024-03-11 21:52:01.860807959 +0000 UTC m=+0.388726140 container attach b41a3aefa54899db9f0d76f5de83413d3f1b8ac0dd34bce62e59f0ea4781d057 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:8fe9635f3beb0247496d7e321f3068bd10c333c433ef4824d7e1c1a7d3d78379, name=ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd-3-deactivate, CEPH_POINT_RELEASE=, RELEASE=main-2b0101e, GIT_CLEAN=True, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, org.label-schema.build-date=20240307, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.vendor=CentOS, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2) 2024-03-11T21:52:02.121 INFO:journalctl@ceph.osd.3.smithi092.stdout:Mar 11 21:52:02 smithi092 podman[217615]: 2024-03-11 21:52:02.070593895 +0000 UTC m=+0.598512058 container died b41a3aefa54899db9f0d76f5de83413d3f1b8ac0dd34bce62e59f0ea4781d057 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:8fe9635f3beb0247496d7e321f3068bd10c333c433ef4824d7e1c1a7d3d78379, name=ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd-3-deactivate, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240307, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=main-2b0101e, GIT_CLEAN=True, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-03-11T21:52:02.336 INFO:journalctl@ceph.osd.4.smithi097.stdout:Mar 11 21:52:02 smithi097 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd-4[38221]: 2024-03-11T21:52:02.090+0000 7f9fa976a700 -1 osd.4 554 heartbeat_check: no reply from 172.21.15.92:6803 osd.0 since back 2024-03-11T21:51:31.363947+0000 front 2024-03-11T21:51:31.364021+0000 (oldest deadline 2024-03-11T21:51:54.863875+0000) 2024-03-11T21:52:02.336 INFO:journalctl@ceph.osd.5.smithi097.stdout:Mar 11 21:52:02 smithi097 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd-5[44269]: 2024-03-11T21:52:02.084+0000 7f6c4e1e6700 -1 osd.5 554 heartbeat_check: no reply from 172.21.15.92:6803 osd.0 since back 2024-03-11T21:51:31.785021+0000 front 2024-03-11T21:51:31.785129+0000 (oldest deadline 2024-03-11T21:51:55.885103+0000) 2024-03-11T21:52:02.337 INFO:journalctl@ceph.osd.6.smithi097.stdout:Mar 11 21:52:02 smithi097 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd-6[50273]: 2024-03-11T21:52:02.006+0000 7f51a9328700 -1 osd.6 554 heartbeat_check: no reply from 172.21.15.92:6803 osd.0 since back 2024-03-11T21:51:31.564540+0000 front 2024-03-11T21:51:31.564552+0000 (oldest deadline 2024-03-11T21:51:57.464467+0000) 2024-03-11T21:52:02.700 DEBUG:teuthology.orchestra.run.smithi092:> sudo pkill -f 'journalctl -f -n 0 -u ceph-00db0016-dfed-11ee-95c8-87774f69a715@osd.3.service' 2024-03-11T21:52:02.742 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-11T21:52:02.742 INFO:tasks.cephadm.osd.3:Stopped osd.3 2024-03-11T21:52:02.742 INFO:tasks.cephadm.osd.4:Stopping osd.4... 2024-03-11T21:52:02.742 DEBUG:teuthology.orchestra.run.smithi097:> sudo systemctl stop ceph-00db0016-dfed-11ee-95c8-87774f69a715@osd.4 2024-03-11T21:52:02.774 INFO:journalctl@ceph.osd.7.smithi097.stdout:Mar 11 21:52:02 smithi097 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd-7[56319]: 2024-03-11T21:52:02.465+0000 7ffbb45e2700 -1 osd.7 554 heartbeat_check: no reply from 172.21.15.92:6803 osd.0 since back 2024-03-11T21:51:32.154875+0000 front 2024-03-11T21:51:32.154922+0000 (oldest deadline 2024-03-11T21:51:57.455053+0000) 2024-03-11T21:52:03.038 INFO:journalctl@ceph.osd.4.smithi097.stdout:Mar 11 21:52:02 smithi097 systemd[1]: Stopping Ceph osd.4 for 00db0016-dfed-11ee-95c8-87774f69a715... 2024-03-11T21:52:03.038 INFO:journalctl@ceph.osd.6.smithi097.stdout:Mar 11 21:52:02 smithi097 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd-6[50273]: 2024-03-11T21:52:02.984+0000 7f51a9328700 -1 osd.6 554 heartbeat_check: no reply from 172.21.15.92:6803 osd.0 since back 2024-03-11T21:51:31.564540+0000 front 2024-03-11T21:51:31.564552+0000 (oldest deadline 2024-03-11T21:51:57.464467+0000) 2024-03-11T21:52:03.336 INFO:journalctl@ceph.osd.5.smithi097.stdout:Mar 11 21:52:03 smithi097 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd-5[44269]: 2024-03-11T21:52:03.045+0000 7f6c4e1e6700 -1 osd.5 554 heartbeat_check: no reply from 172.21.15.92:6803 osd.0 since back 2024-03-11T21:51:31.785021+0000 front 2024-03-11T21:51:31.785129+0000 (oldest deadline 2024-03-11T21:51:55.885103+0000) 2024-03-11T21:52:03.336 INFO:journalctl@ceph.osd.4.smithi097.stdout:Mar 11 21:52:03 smithi097 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd-4[38221]: 2024-03-11T21:52:03.098+0000 7f9fa976a700 -1 osd.4 554 heartbeat_check: no reply from 172.21.15.92:6803 osd.0 since back 2024-03-11T21:51:31.363947+0000 front 2024-03-11T21:51:31.364021+0000 (oldest deadline 2024-03-11T21:51:54.863875+0000) 2024-03-11T21:52:03.337 INFO:journalctl@ceph.osd.4.smithi097.stdout:Mar 11 21:52:03 smithi097 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd-4[38221]: 2024-03-11T21:52:03.254+0000 7f9fad176700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.4 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-11T21:52:03.337 INFO:journalctl@ceph.osd.4.smithi097.stdout:Mar 11 21:52:03 smithi097 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd-4[38221]: 2024-03-11T21:52:03.254+0000 7f9fad176700 -1 osd.4 554 *** Got signal Terminated *** 2024-03-11T21:52:03.337 INFO:journalctl@ceph.osd.4.smithi097.stdout:Mar 11 21:52:03 smithi097 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd-4[38221]: 2024-03-11T21:52:03.254+0000 7f9fad176700 -1 osd.4 554 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-03-11T21:52:03.836 INFO:journalctl@ceph.osd.7.smithi097.stdout:Mar 11 21:52:03 smithi097 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd-7[56319]: 2024-03-11T21:52:03.464+0000 7ffbb45e2700 -1 osd.7 554 heartbeat_check: no reply from 172.21.15.92:6803 osd.0 since back 2024-03-11T21:51:32.154875+0000 front 2024-03-11T21:51:32.154922+0000 (oldest deadline 2024-03-11T21:51:57.455053+0000) 2024-03-11T21:52:04.336 INFO:journalctl@ceph.osd.4.smithi097.stdout:Mar 11 21:52:04 smithi097 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd-4[38221]: 2024-03-11T21:52:04.049+0000 7f9fa976a700 -1 osd.4 554 heartbeat_check: no reply from 172.21.15.92:6803 osd.0 since back 2024-03-11T21:51:31.363947+0000 front 2024-03-11T21:51:31.364021+0000 (oldest deadline 2024-03-11T21:51:54.863875+0000) 2024-03-11T21:52:04.337 INFO:journalctl@ceph.osd.5.smithi097.stdout:Mar 11 21:52:04 smithi097 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd-5[44269]: 2024-03-11T21:52:04.087+0000 7f6c4e1e6700 -1 osd.5 554 heartbeat_check: no reply from 172.21.15.92:6803 osd.0 since back 2024-03-11T21:51:31.785021+0000 front 2024-03-11T21:51:31.785129+0000 (oldest deadline 2024-03-11T21:51:55.885103+0000) 2024-03-11T21:52:04.337 INFO:journalctl@ceph.osd.6.smithi097.stdout:Mar 11 21:52:03 smithi097 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd-6[50273]: 2024-03-11T21:52:03.976+0000 7f51a9328700 -1 osd.6 554 heartbeat_check: no reply from 172.21.15.92:6803 osd.0 since back 2024-03-11T21:51:31.564540+0000 front 2024-03-11T21:51:31.564552+0000 (oldest deadline 2024-03-11T21:51:57.464467+0000) 2024-03-11T21:52:04.836 INFO:journalctl@ceph.osd.7.smithi097.stdout:Mar 11 21:52:04 smithi097 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd-7[56319]: 2024-03-11T21:52:04.432+0000 7ffbb45e2700 -1 osd.7 554 heartbeat_check: no reply from 172.21.15.92:6803 osd.0 since back 2024-03-11T21:51:32.154875+0000 front 2024-03-11T21:51:32.154922+0000 (oldest deadline 2024-03-11T21:51:57.455053+0000) 2024-03-11T21:52:05.336 INFO:journalctl@ceph.osd.4.smithi097.stdout:Mar 11 21:52:05 smithi097 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd-4[38221]: 2024-03-11T21:52:05.050+0000 7f9fa976a700 -1 osd.4 554 heartbeat_check: no reply from 172.21.15.92:6803 osd.0 since back 2024-03-11T21:51:31.363947+0000 front 2024-03-11T21:51:31.364021+0000 (oldest deadline 2024-03-11T21:51:54.863875+0000) 2024-03-11T21:52:05.336 INFO:journalctl@ceph.osd.5.smithi097.stdout:Mar 11 21:52:05 smithi097 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd-5[44269]: 2024-03-11T21:52:05.048+0000 7f6c4e1e6700 -1 osd.5 554 heartbeat_check: no reply from 172.21.15.92:6803 osd.0 since back 2024-03-11T21:51:31.785021+0000 front 2024-03-11T21:51:31.785129+0000 (oldest deadline 2024-03-11T21:51:55.885103+0000) 2024-03-11T21:52:05.337 INFO:journalctl@ceph.osd.5.smithi097.stdout:Mar 11 21:52:05 smithi097 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd-5[44269]: 2024-03-11T21:52:05.048+0000 7f6c4e1e6700 -1 osd.5 554 heartbeat_check: no reply from 172.21.15.92:6807 osd.1 since back 2024-03-11T21:51:39.886656+0000 front 2024-03-11T21:51:39.886672+0000 (oldest deadline 2024-03-11T21:52:04.586633+0000) 2024-03-11T21:52:05.337 INFO:journalctl@ceph.osd.6.smithi097.stdout:Mar 11 21:52:04 smithi097 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd-6[50273]: 2024-03-11T21:52:04.952+0000 7f51a9328700 -1 osd.6 554 heartbeat_check: no reply from 172.21.15.92:6803 osd.0 since back 2024-03-11T21:51:31.564540+0000 front 2024-03-11T21:51:31.564552+0000 (oldest deadline 2024-03-11T21:51:57.464467+0000) 2024-03-11T21:52:05.836 INFO:journalctl@ceph.osd.7.smithi097.stdout:Mar 11 21:52:05 smithi097 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd-7[56319]: 2024-03-11T21:52:05.459+0000 7ffbb45e2700 -1 osd.7 554 heartbeat_check: no reply from 172.21.15.92:6803 osd.0 since back 2024-03-11T21:51:32.154875+0000 front 2024-03-11T21:51:32.154922+0000 (oldest deadline 2024-03-11T21:51:57.455053+0000) 2024-03-11T21:52:06.336 INFO:journalctl@ceph.osd.6.smithi097.stdout:Mar 11 21:52:05 smithi097 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd-6[50273]: 2024-03-11T21:52:05.949+0000 7f51a9328700 -1 osd.6 554 heartbeat_check: no reply from 172.21.15.92:6803 osd.0 since back 2024-03-11T21:51:31.564540+0000 front 2024-03-11T21:51:31.564552+0000 (oldest deadline 2024-03-11T21:51:57.464467+0000) 2024-03-11T21:52:06.336 INFO:journalctl@ceph.osd.5.smithi097.stdout:Mar 11 21:52:06 smithi097 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd-5[44269]: 2024-03-11T21:52:06.021+0000 7f6c4e1e6700 -1 osd.5 554 heartbeat_check: no reply from 172.21.15.92:6803 osd.0 since back 2024-03-11T21:51:31.785021+0000 front 2024-03-11T21:51:31.785129+0000 (oldest deadline 2024-03-11T21:51:55.885103+0000) 2024-03-11T21:52:06.337 INFO:journalctl@ceph.osd.5.smithi097.stdout:Mar 11 21:52:06 smithi097 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd-5[44269]: 2024-03-11T21:52:06.021+0000 7f6c4e1e6700 -1 osd.5 554 heartbeat_check: no reply from 172.21.15.92:6807 osd.1 since back 2024-03-11T21:51:39.886656+0000 front 2024-03-11T21:51:39.886672+0000 (oldest deadline 2024-03-11T21:52:04.586633+0000) 2024-03-11T21:52:06.337 INFO:journalctl@ceph.osd.4.smithi097.stdout:Mar 11 21:52:06 smithi097 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd-4[38221]: 2024-03-11T21:52:06.012+0000 7f9fa976a700 -1 osd.4 554 heartbeat_check: no reply from 172.21.15.92:6803 osd.0 since back 2024-03-11T21:51:31.363947+0000 front 2024-03-11T21:51:31.364021+0000 (oldest deadline 2024-03-11T21:51:54.863875+0000) 2024-03-11T21:52:06.337 INFO:journalctl@ceph.osd.4.smithi097.stdout:Mar 11 21:52:06 smithi097 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd-4[38221]: 2024-03-11T21:52:06.012+0000 7f9fa976a700 -1 osd.4 554 heartbeat_check: no reply from 172.21.15.92:6807 osd.1 since back 2024-03-11T21:51:41.265288+0000 front 2024-03-11T21:51:41.265306+0000 (oldest deadline 2024-03-11T21:52:05.365505+0000) 2024-03-11T21:52:06.836 INFO:journalctl@ceph.osd.7.smithi097.stdout:Mar 11 21:52:06 smithi097 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd-7[56319]: 2024-03-11T21:52:06.494+0000 7ffbb45e2700 -1 osd.7 554 heartbeat_check: no reply from 172.21.15.92:6803 osd.0 since back 2024-03-11T21:51:32.154875+0000 front 2024-03-11T21:51:32.154922+0000 (oldest deadline 2024-03-11T21:51:57.455053+0000) 2024-03-11T21:52:06.836 INFO:journalctl@ceph.osd.7.smithi097.stdout:Mar 11 21:52:06 smithi097 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd-7[56319]: 2024-03-11T21:52:06.494+0000 7ffbb45e2700 -1 osd.7 554 heartbeat_check: no reply from 172.21.15.92:6807 osd.1 since back 2024-03-11T21:51:39.656754+0000 front 2024-03-11T21:51:39.656727+0000 (oldest deadline 2024-03-11T21:52:05.556480+0000) 2024-03-11T21:52:07.336 INFO:journalctl@ceph.osd.6.smithi097.stdout:Mar 11 21:52:06 smithi097 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd-6[50273]: 2024-03-11T21:52:06.983+0000 7f51a9328700 -1 osd.6 554 heartbeat_check: no reply from 172.21.15.92:6803 osd.0 since back 2024-03-11T21:51:31.564540+0000 front 2024-03-11T21:51:31.564552+0000 (oldest deadline 2024-03-11T21:51:57.464467+0000) 2024-03-11T21:52:07.337 INFO:journalctl@ceph.osd.5.smithi097.stdout:Mar 11 21:52:07 smithi097 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd-5[44269]: 2024-03-11T21:52:07.010+0000 7f6c4e1e6700 -1 osd.5 554 heartbeat_check: no reply from 172.21.15.92:6803 osd.0 since back 2024-03-11T21:51:31.785021+0000 front 2024-03-11T21:51:31.785129+0000 (oldest deadline 2024-03-11T21:51:55.885103+0000) 2024-03-11T21:52:07.337 INFO:journalctl@ceph.osd.5.smithi097.stdout:Mar 11 21:52:07 smithi097 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd-5[44269]: 2024-03-11T21:52:07.010+0000 7f6c4e1e6700 -1 osd.5 554 heartbeat_check: no reply from 172.21.15.92:6807 osd.1 since back 2024-03-11T21:51:39.886656+0000 front 2024-03-11T21:51:39.886672+0000 (oldest deadline 2024-03-11T21:52:04.586633+0000) 2024-03-11T21:52:07.337 INFO:journalctl@ceph.osd.4.smithi097.stdout:Mar 11 21:52:06 smithi097 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd-4[38221]: 2024-03-11T21:52:06.994+0000 7f9fa976a700 -1 osd.4 554 heartbeat_check: no reply from 172.21.15.92:6803 osd.0 since back 2024-03-11T21:51:31.363947+0000 front 2024-03-11T21:51:31.364021+0000 (oldest deadline 2024-03-11T21:51:54.863875+0000) 2024-03-11T21:52:07.337 INFO:journalctl@ceph.osd.4.smithi097.stdout:Mar 11 21:52:06 smithi097 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd-4[38221]: 2024-03-11T21:52:06.994+0000 7f9fa976a700 -1 osd.4 554 heartbeat_check: no reply from 172.21.15.92:6807 osd.1 since back 2024-03-11T21:51:41.265288+0000 front 2024-03-11T21:51:41.265306+0000 (oldest deadline 2024-03-11T21:52:05.365505+0000) 2024-03-11T21:52:07.836 INFO:journalctl@ceph.osd.7.smithi097.stdout:Mar 11 21:52:07 smithi097 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd-7[56319]: 2024-03-11T21:52:07.480+0000 7ffbb45e2700 -1 osd.7 554 heartbeat_check: no reply from 172.21.15.92:6803 osd.0 since back 2024-03-11T21:51:32.154875+0000 front 2024-03-11T21:51:32.154922+0000 (oldest deadline 2024-03-11T21:51:57.455053+0000) 2024-03-11T21:52:07.837 INFO:journalctl@ceph.osd.7.smithi097.stdout:Mar 11 21:52:07 smithi097 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd-7[56319]: 2024-03-11T21:52:07.480+0000 7ffbb45e2700 -1 osd.7 554 heartbeat_check: no reply from 172.21.15.92:6807 osd.1 since back 2024-03-11T21:51:39.656754+0000 front 2024-03-11T21:51:39.656727+0000 (oldest deadline 2024-03-11T21:52:05.556480+0000) 2024-03-11T21:52:08.336 INFO:journalctl@ceph.osd.6.smithi097.stdout:Mar 11 21:52:08 smithi097 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd-6[50273]: 2024-03-11T21:52:08.013+0000 7f51a9328700 -1 osd.6 554 heartbeat_check: no reply from 172.21.15.92:6803 osd.0 since back 2024-03-11T21:51:31.564540+0000 front 2024-03-11T21:51:31.564552+0000 (oldest deadline 2024-03-11T21:51:57.464467+0000) 2024-03-11T21:52:08.337 INFO:journalctl@ceph.osd.5.smithi097.stdout:Mar 11 21:52:07 smithi097 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd-5[44269]: 2024-03-11T21:52:07.991+0000 7f6c4e1e6700 -1 osd.5 554 heartbeat_check: no reply from 172.21.15.92:6803 osd.0 since back 2024-03-11T21:51:31.785021+0000 front 2024-03-11T21:51:31.785129+0000 (oldest deadline 2024-03-11T21:51:55.885103+0000) 2024-03-11T21:52:08.337 INFO:journalctl@ceph.osd.5.smithi097.stdout:Mar 11 21:52:07 smithi097 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd-5[44269]: 2024-03-11T21:52:07.991+0000 7f6c4e1e6700 -1 osd.5 554 heartbeat_check: no reply from 172.21.15.92:6807 osd.1 since back 2024-03-11T21:51:39.886656+0000 front 2024-03-11T21:51:39.886672+0000 (oldest deadline 2024-03-11T21:52:04.586633+0000) 2024-03-11T21:52:08.337 INFO:journalctl@ceph.osd.4.smithi097.stdout:Mar 11 21:52:07 smithi097 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd-4[38221]: 2024-03-11T21:52:07.966+0000 7f9fa976a700 -1 osd.4 554 heartbeat_check: no reply from 172.21.15.92:6803 osd.0 since back 2024-03-11T21:51:31.363947+0000 front 2024-03-11T21:51:31.364021+0000 (oldest deadline 2024-03-11T21:51:54.863875+0000) 2024-03-11T21:52:08.337 INFO:journalctl@ceph.osd.4.smithi097.stdout:Mar 11 21:52:07 smithi097 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd-4[38221]: 2024-03-11T21:52:07.966+0000 7f9fa976a700 -1 osd.4 554 heartbeat_check: no reply from 172.21.15.92:6807 osd.1 since back 2024-03-11T21:51:41.265288+0000 front 2024-03-11T21:51:41.265306+0000 (oldest deadline 2024-03-11T21:52:05.365505+0000) 2024-03-11T21:52:08.724 INFO:journalctl@ceph.osd.7.smithi097.stdout:Mar 11 21:52:08 smithi097 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd-7[56319]: 2024-03-11T21:52:08.449+0000 7ffbb45e2700 -1 osd.7 554 heartbeat_check: no reply from 172.21.15.92:6803 osd.0 since back 2024-03-11T21:51:32.154875+0000 front 2024-03-11T21:51:32.154922+0000 (oldest deadline 2024-03-11T21:51:57.455053+0000) 2024-03-11T21:52:08.724 INFO:journalctl@ceph.osd.7.smithi097.stdout:Mar 11 21:52:08 smithi097 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd-7[56319]: 2024-03-11T21:52:08.449+0000 7ffbb45e2700 -1 osd.7 554 heartbeat_check: no reply from 172.21.15.92:6807 osd.1 since back 2024-03-11T21:51:39.656754+0000 front 2024-03-11T21:51:39.656727+0000 (oldest deadline 2024-03-11T21:52:05.556480+0000) 2024-03-11T21:52:08.979 INFO:journalctl@ceph.osd.4.smithi097.stdout:Mar 11 21:52:08 smithi097 podman[160609]: 2024-03-11 21:52:08.732285815 +0000 UTC m=+5.775131311 container died 16db23c380a898487b0f854fd74663b3587f4de9b8208e1cc83d1055b03fac9b (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:8fe9635f3beb0247496d7e321f3068bd10c333c433ef4824d7e1c1a7d3d78379, name=ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd-4, org.label-schema.build-date=20240307, org.label-schema.vendor=CentOS, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.license=GPLv2, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, RELEASE=main-2b0101e) 2024-03-11T21:52:09.336 INFO:journalctl@ceph.osd.4.smithi097.stdout:Mar 11 21:52:09 smithi097 podman[160609]: 2024-03-11 21:52:09.197168761 +0000 UTC m=+6.240014311 container cleanup 16db23c380a898487b0f854fd74663b3587f4de9b8208e1cc83d1055b03fac9b (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:8fe9635f3beb0247496d7e321f3068bd10c333c433ef4824d7e1c1a7d3d78379, name=ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd-4, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240307, GIT_CLEAN=True, ceph=True, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=main-2b0101e, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30) 2024-03-11T21:52:09.336 INFO:journalctl@ceph.osd.4.smithi097.stdout:Mar 11 21:52:09 smithi097 bash[160609]: ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd-4 2024-03-11T21:52:09.337 INFO:journalctl@ceph.osd.5.smithi097.stdout:Mar 11 21:52:08 smithi097 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd-5[44269]: 2024-03-11T21:52:08.985+0000 7f6c4e1e6700 -1 osd.5 554 heartbeat_check: no reply from 172.21.15.92:6803 osd.0 since back 2024-03-11T21:51:31.785021+0000 front 2024-03-11T21:51:31.785129+0000 (oldest deadline 2024-03-11T21:51:55.885103+0000) 2024-03-11T21:52:09.337 INFO:journalctl@ceph.osd.5.smithi097.stdout:Mar 11 21:52:08 smithi097 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd-5[44269]: 2024-03-11T21:52:08.985+0000 7f6c4e1e6700 -1 osd.5 554 heartbeat_check: no reply from 172.21.15.92:6807 osd.1 since back 2024-03-11T21:51:39.886656+0000 front 2024-03-11T21:51:39.886672+0000 (oldest deadline 2024-03-11T21:52:04.586633+0000) 2024-03-11T21:52:09.337 INFO:journalctl@ceph.osd.6.smithi097.stdout:Mar 11 21:52:09 smithi097 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd-6[50273]: 2024-03-11T21:52:09.044+0000 7f51a9328700 -1 osd.6 554 heartbeat_check: no reply from 172.21.15.92:6803 osd.0 since back 2024-03-11T21:51:31.564540+0000 front 2024-03-11T21:51:31.564552+0000 (oldest deadline 2024-03-11T21:51:57.464467+0000) 2024-03-11T21:52:09.337 INFO:journalctl@ceph.osd.6.smithi097.stdout:Mar 11 21:52:09 smithi097 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd-6[50273]: 2024-03-11T21:52:09.044+0000 7f51a9328700 -1 osd.6 554 heartbeat_check: no reply from 172.21.15.92:6807 osd.1 since back 2024-03-11T21:51:42.666096+0000 front 2024-03-11T21:51:42.666115+0000 (oldest deadline 2024-03-11T21:52:08.566007+0000) 2024-03-11T21:52:09.836 INFO:journalctl@ceph.osd.4.smithi097.stdout:Mar 11 21:52:09 smithi097 podman[160628]: 2024-03-11 21:52:09.559446816 +0000 UTC m=+0.825467847 container remove 16db23c380a898487b0f854fd74663b3587f4de9b8208e1cc83d1055b03fac9b (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:8fe9635f3beb0247496d7e321f3068bd10c333c433ef4824d7e1c1a7d3d78379, name=ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd-4, org.label-schema.build-date=20240307, io.buildah.version=1.33.5, ceph=True, CEPH_POINT_RELEASE=, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, maintainer=Guillaume Abrioux , GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, RELEASE=main-2b0101e, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image) 2024-03-11T21:52:09.836 INFO:journalctl@ceph.osd.7.smithi097.stdout:Mar 11 21:52:09 smithi097 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd-7[56319]: 2024-03-11T21:52:09.451+0000 7ffbb45e2700 -1 osd.7 554 heartbeat_check: no reply from 172.21.15.92:6803 osd.0 since back 2024-03-11T21:51:32.154875+0000 front 2024-03-11T21:51:32.154922+0000 (oldest deadline 2024-03-11T21:51:57.455053+0000) 2024-03-11T21:52:09.837 INFO:journalctl@ceph.osd.7.smithi097.stdout:Mar 11 21:52:09 smithi097 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd-7[56319]: 2024-03-11T21:52:09.451+0000 7ffbb45e2700 -1 osd.7 554 heartbeat_check: no reply from 172.21.15.92:6807 osd.1 since back 2024-03-11T21:51:39.656754+0000 front 2024-03-11T21:51:39.656727+0000 (oldest deadline 2024-03-11T21:52:05.556480+0000) 2024-03-11T21:52:10.214 INFO:journalctl@ceph.osd.5.smithi097.stdout:Mar 11 21:52:09 smithi097 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd-5[44269]: 2024-03-11T21:52:09.974+0000 7f6c4e1e6700 -1 osd.5 554 heartbeat_check: no reply from 172.21.15.92:6803 osd.0 since back 2024-03-11T21:51:31.785021+0000 front 2024-03-11T21:51:31.785129+0000 (oldest deadline 2024-03-11T21:51:55.885103+0000) 2024-03-11T21:52:10.214 INFO:journalctl@ceph.osd.5.smithi097.stdout:Mar 11 21:52:09 smithi097 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd-5[44269]: 2024-03-11T21:52:09.974+0000 7f6c4e1e6700 -1 osd.5 554 heartbeat_check: no reply from 172.21.15.92:6807 osd.1 since back 2024-03-11T21:51:39.886656+0000 front 2024-03-11T21:51:39.886672+0000 (oldest deadline 2024-03-11T21:52:04.586633+0000) 2024-03-11T21:52:10.215 INFO:journalctl@ceph.osd.6.smithi097.stdout:Mar 11 21:52:10 smithi097 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd-6[50273]: 2024-03-11T21:52:10.091+0000 7f51a9328700 -1 osd.6 554 heartbeat_check: no reply from 172.21.15.92:6803 osd.0 since back 2024-03-11T21:51:31.564540+0000 front 2024-03-11T21:51:31.564552+0000 (oldest deadline 2024-03-11T21:51:57.464467+0000) 2024-03-11T21:52:10.215 INFO:journalctl@ceph.osd.6.smithi097.stdout:Mar 11 21:52:10 smithi097 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd-6[50273]: 2024-03-11T21:52:10.091+0000 7f51a9328700 -1 osd.6 554 heartbeat_check: no reply from 172.21.15.92:6807 osd.1 since back 2024-03-11T21:51:42.666096+0000 front 2024-03-11T21:51:42.666115+0000 (oldest deadline 2024-03-11T21:52:08.566007+0000) 2024-03-11T21:52:10.215 INFO:journalctl@ceph.osd.4.smithi097.stdout:Mar 11 21:52:09 smithi097 podman[160719]: 2024-03-11 21:52:09.874474677 +0000 UTC m=+0.017771971 image pull 4133f8bc41bb201212a28168427a2edb1f6f3795e694bf61031ea9e2f3ff5e87 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:8fe9635f3beb0247496d7e321f3068bd10c333c433ef4824d7e1c1a7d3d78379 2024-03-11T21:52:10.215 INFO:journalctl@ceph.osd.4.smithi097.stdout:Mar 11 21:52:09 smithi097 podman[160719]: 2024-03-11 21:52:09.983339125 +0000 UTC m=+0.126636462 container create 8c7002ce2fae79f19d5563390e0caf95ca681fe2f40fc29f06ec3c885c3a9ea0 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:8fe9635f3beb0247496d7e321f3068bd10c333c433ef4824d7e1c1a7d3d78379, name=ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd-4-deactivate, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240307, org.label-schema.vendor=CentOS, GIT_CLEAN=True, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=main-2b0101e) 2024-03-11T21:52:10.215 INFO:journalctl@ceph.osd.4.smithi097.stdout:Mar 11 21:52:10 smithi097 podman[160719]: 2024-03-11 21:52:10.219086037 +0000 UTC m=+0.362383332 container init 8c7002ce2fae79f19d5563390e0caf95ca681fe2f40fc29f06ec3c885c3a9ea0 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:8fe9635f3beb0247496d7e321f3068bd10c333c433ef4824d7e1c1a7d3d78379, name=ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd-4-deactivate, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, org.label-schema.build-date=20240307, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, GIT_CLEAN=True, org.label-schema.license=GPLv2, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=main-2b0101e) 2024-03-11T21:52:10.215 INFO:journalctl@ceph.osd.4.smithi097.stdout:Mar 11 21:52:10 smithi097 podman[160719]: 2024-03-11 21:52:10.222640844 +0000 UTC m=+0.365938139 container start 8c7002ce2fae79f19d5563390e0caf95ca681fe2f40fc29f06ec3c885c3a9ea0 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:8fe9635f3beb0247496d7e321f3068bd10c333c433ef4824d7e1c1a7d3d78379, name=ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd-4-deactivate, ceph=True, RELEASE=main-2b0101e, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, org.label-schema.build-date=20240307, GIT_CLEAN=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5) 2024-03-11T21:52:10.586 INFO:journalctl@ceph.osd.7.smithi097.stdout:Mar 11 21:52:10 smithi097 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd-7[56319]: 2024-03-11T21:52:10.438+0000 7ffbb45e2700 -1 osd.7 554 heartbeat_check: no reply from 172.21.15.92:6803 osd.0 since back 2024-03-11T21:51:32.154875+0000 front 2024-03-11T21:51:32.154922+0000 (oldest deadline 2024-03-11T21:51:57.455053+0000) 2024-03-11T21:52:10.587 INFO:journalctl@ceph.osd.7.smithi097.stdout:Mar 11 21:52:10 smithi097 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd-7[56319]: 2024-03-11T21:52:10.438+0000 7ffbb45e2700 -1 osd.7 554 heartbeat_check: no reply from 172.21.15.92:6807 osd.1 since back 2024-03-11T21:51:39.656754+0000 front 2024-03-11T21:51:39.656727+0000 (oldest deadline 2024-03-11T21:52:05.556480+0000) 2024-03-11T21:52:10.587 INFO:journalctl@ceph.osd.4.smithi097.stdout:Mar 11 21:52:10 smithi097 podman[160719]: 2024-03-11 21:52:10.377699508 +0000 UTC m=+0.520996804 container attach 8c7002ce2fae79f19d5563390e0caf95ca681fe2f40fc29f06ec3c885c3a9ea0 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:8fe9635f3beb0247496d7e321f3068bd10c333c433ef4824d7e1c1a7d3d78379, name=ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd-4-deactivate, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, io.buildah.version=1.33.5, org.label-schema.build-date=20240307, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, ceph=True, org.label-schema.vendor=CentOS, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=main-2b0101e, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD) 2024-03-11T21:52:11.135 INFO:journalctl@ceph.osd.5.smithi097.stdout:Mar 11 21:52:10 smithi097 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd-5[44269]: 2024-03-11T21:52:10.961+0000 7f6c4e1e6700 -1 osd.5 554 heartbeat_check: no reply from 172.21.15.92:6803 osd.0 since back 2024-03-11T21:51:31.785021+0000 front 2024-03-11T21:51:31.785129+0000 (oldest deadline 2024-03-11T21:51:55.885103+0000) 2024-03-11T21:52:11.135 INFO:journalctl@ceph.osd.5.smithi097.stdout:Mar 11 21:52:10 smithi097 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd-5[44269]: 2024-03-11T21:52:10.961+0000 7f6c4e1e6700 -1 osd.5 554 heartbeat_check: no reply from 172.21.15.92:6807 osd.1 since back 2024-03-11T21:51:39.886656+0000 front 2024-03-11T21:51:39.886672+0000 (oldest deadline 2024-03-11T21:52:04.586633+0000) 2024-03-11T21:52:11.136 INFO:journalctl@ceph.osd.6.smithi097.stdout:Mar 11 21:52:11 smithi097 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd-6[50273]: 2024-03-11T21:52:11.106+0000 7f51a9328700 -1 osd.6 554 heartbeat_check: no reply from 172.21.15.92:6803 osd.0 since back 2024-03-11T21:51:31.564540+0000 front 2024-03-11T21:51:31.564552+0000 (oldest deadline 2024-03-11T21:51:57.464467+0000) 2024-03-11T21:52:11.136 INFO:journalctl@ceph.osd.6.smithi097.stdout:Mar 11 21:52:11 smithi097 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd-6[50273]: 2024-03-11T21:52:11.106+0000 7f51a9328700 -1 osd.6 554 heartbeat_check: no reply from 172.21.15.92:6807 osd.1 since back 2024-03-11T21:51:42.666096+0000 front 2024-03-11T21:51:42.666115+0000 (oldest deadline 2024-03-11T21:52:08.566007+0000) 2024-03-11T21:52:11.136 INFO:journalctl@ceph.osd.4.smithi097.stdout:Mar 11 21:52:10 smithi097 podman[160719]: 2024-03-11 21:52:10.876415935 +0000 UTC m=+1.019713229 container died 8c7002ce2fae79f19d5563390e0caf95ca681fe2f40fc29f06ec3c885c3a9ea0 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:8fe9635f3beb0247496d7e321f3068bd10c333c433ef4824d7e1c1a7d3d78379, name=ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd-4-deactivate, org.label-schema.build-date=20240307, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, RELEASE=main-2b0101e, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, ceph=True) 2024-03-11T21:52:11.501 DEBUG:teuthology.orchestra.run.smithi097:> sudo pkill -f 'journalctl -f -n 0 -u ceph-00db0016-dfed-11ee-95c8-87774f69a715@osd.4.service' 2024-03-11T21:52:11.583 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-11T21:52:11.584 INFO:tasks.cephadm.osd.4:Stopped osd.4 2024-03-11T21:52:11.584 INFO:tasks.cephadm.osd.5:Stopping osd.5... 2024-03-11T21:52:11.584 DEBUG:teuthology.orchestra.run.smithi097:> sudo systemctl stop ceph-00db0016-dfed-11ee-95c8-87774f69a715@osd.5 2024-03-11T21:52:11.673 INFO:journalctl@ceph.osd.7.smithi097.stdout:Mar 11 21:52:11 smithi097 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd-7[56319]: 2024-03-11T21:52:11.421+0000 7ffbb45e2700 -1 osd.7 554 heartbeat_check: no reply from 172.21.15.92:6803 osd.0 since back 2024-03-11T21:51:32.154875+0000 front 2024-03-11T21:51:32.154922+0000 (oldest deadline 2024-03-11T21:51:57.455053+0000) 2024-03-11T21:52:11.673 INFO:journalctl@ceph.osd.7.smithi097.stdout:Mar 11 21:52:11 smithi097 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd-7[56319]: 2024-03-11T21:52:11.421+0000 7ffbb45e2700 -1 osd.7 554 heartbeat_check: no reply from 172.21.15.92:6807 osd.1 since back 2024-03-11T21:51:39.656754+0000 front 2024-03-11T21:51:39.656727+0000 (oldest deadline 2024-03-11T21:52:05.556480+0000) 2024-03-11T21:52:11.950 INFO:journalctl@ceph.osd.5.smithi097.stdout:Mar 11 21:52:11 smithi097 systemd[1]: Stopping Ceph osd.5 for 00db0016-dfed-11ee-95c8-87774f69a715... 2024-03-11T21:52:11.950 INFO:journalctl@ceph.osd.5.smithi097.stdout:Mar 11 21:52:11 smithi097 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd-5[44269]: 2024-03-11T21:52:11.926+0000 7f6c55bfa700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.5 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-11T21:52:11.950 INFO:journalctl@ceph.osd.5.smithi097.stdout:Mar 11 21:52:11 smithi097 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd-5[44269]: 2024-03-11T21:52:11.926+0000 7f6c55bfa700 -1 osd.5 554 *** Got signal Terminated *** 2024-03-11T21:52:11.950 INFO:journalctl@ceph.osd.5.smithi097.stdout:Mar 11 21:52:11 smithi097 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd-5[44269]: 2024-03-11T21:52:11.926+0000 7f6c55bfa700 -1 osd.5 554 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-03-11T21:52:11.950 INFO:journalctl@ceph.osd.5.smithi097.stdout:Mar 11 21:52:11 smithi097 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd-5[44269]: 2024-03-11T21:52:11.956+0000 7f6c4e1e6700 -1 osd.5 554 heartbeat_check: no reply from 172.21.15.92:6803 osd.0 since back 2024-03-11T21:51:31.785021+0000 front 2024-03-11T21:51:31.785129+0000 (oldest deadline 2024-03-11T21:51:55.885103+0000) 2024-03-11T21:52:11.950 INFO:journalctl@ceph.osd.5.smithi097.stdout:Mar 11 21:52:11 smithi097 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd-5[44269]: 2024-03-11T21:52:11.956+0000 7f6c4e1e6700 -1 osd.5 554 heartbeat_check: no reply from 172.21.15.92:6807 osd.1 since back 2024-03-11T21:51:39.886656+0000 front 2024-03-11T21:51:39.886672+0000 (oldest deadline 2024-03-11T21:52:04.586633+0000) 2024-03-11T21:52:12.336 INFO:journalctl@ceph.osd.6.smithi097.stdout:Mar 11 21:52:12 smithi097 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd-6[50273]: 2024-03-11T21:52:12.066+0000 7f51a9328700 -1 osd.6 554 heartbeat_check: no reply from 172.21.15.92:6803 osd.0 since back 2024-03-11T21:51:31.564540+0000 front 2024-03-11T21:51:31.564552+0000 (oldest deadline 2024-03-11T21:51:57.464467+0000) 2024-03-11T21:52:12.336 INFO:journalctl@ceph.osd.6.smithi097.stdout:Mar 11 21:52:12 smithi097 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd-6[50273]: 2024-03-11T21:52:12.066+0000 7f51a9328700 -1 osd.6 554 heartbeat_check: no reply from 172.21.15.92:6807 osd.1 since back 2024-03-11T21:51:42.666096+0000 front 2024-03-11T21:51:42.666115+0000 (oldest deadline 2024-03-11T21:52:08.566007+0000) 2024-03-11T21:52:12.836 INFO:journalctl@ceph.osd.7.smithi097.stdout:Mar 11 21:52:12 smithi097 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd-7[56319]: 2024-03-11T21:52:12.375+0000 7ffbb45e2700 -1 osd.7 554 heartbeat_check: no reply from 172.21.15.92:6803 osd.0 since back 2024-03-11T21:51:32.154875+0000 front 2024-03-11T21:51:32.154922+0000 (oldest deadline 2024-03-11T21:51:57.455053+0000) 2024-03-11T21:52:12.836 INFO:journalctl@ceph.osd.7.smithi097.stdout:Mar 11 21:52:12 smithi097 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd-7[56319]: 2024-03-11T21:52:12.375+0000 7ffbb45e2700 -1 osd.7 554 heartbeat_check: no reply from 172.21.15.92:6807 osd.1 since back 2024-03-11T21:51:39.656754+0000 front 2024-03-11T21:51:39.656727+0000 (oldest deadline 2024-03-11T21:52:05.556480+0000) 2024-03-11T21:52:13.336 INFO:journalctl@ceph.osd.5.smithi097.stdout:Mar 11 21:52:12 smithi097 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd-5[44269]: 2024-03-11T21:52:12.991+0000 7f6c4e1e6700 -1 osd.5 554 heartbeat_check: no reply from 172.21.15.92:6803 osd.0 since back 2024-03-11T21:51:31.785021+0000 front 2024-03-11T21:51:31.785129+0000 (oldest deadline 2024-03-11T21:51:55.885103+0000) 2024-03-11T21:52:13.336 INFO:journalctl@ceph.osd.5.smithi097.stdout:Mar 11 21:52:12 smithi097 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd-5[44269]: 2024-03-11T21:52:12.991+0000 7f6c4e1e6700 -1 osd.5 554 heartbeat_check: no reply from 172.21.15.92:6807 osd.1 since back 2024-03-11T21:51:39.886656+0000 front 2024-03-11T21:51:39.886672+0000 (oldest deadline 2024-03-11T21:52:04.586633+0000) 2024-03-11T21:52:13.336 INFO:journalctl@ceph.osd.6.smithi097.stdout:Mar 11 21:52:13 smithi097 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd-6[50273]: 2024-03-11T21:52:13.070+0000 7f51a9328700 -1 osd.6 554 heartbeat_check: no reply from 172.21.15.92:6803 osd.0 since back 2024-03-11T21:51:31.564540+0000 front 2024-03-11T21:51:31.564552+0000 (oldest deadline 2024-03-11T21:51:57.464467+0000) 2024-03-11T21:52:13.337 INFO:journalctl@ceph.osd.6.smithi097.stdout:Mar 11 21:52:13 smithi097 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd-6[50273]: 2024-03-11T21:52:13.070+0000 7f51a9328700 -1 osd.6 554 heartbeat_check: no reply from 172.21.15.92:6807 osd.1 since back 2024-03-11T21:51:42.666096+0000 front 2024-03-11T21:51:42.666115+0000 (oldest deadline 2024-03-11T21:52:08.566007+0000) 2024-03-11T21:52:13.836 INFO:journalctl@ceph.osd.7.smithi097.stdout:Mar 11 21:52:13 smithi097 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd-7[56319]: 2024-03-11T21:52:13.372+0000 7ffbb45e2700 -1 osd.7 554 heartbeat_check: no reply from 172.21.15.92:6803 osd.0 since back 2024-03-11T21:51:32.154875+0000 front 2024-03-11T21:51:32.154922+0000 (oldest deadline 2024-03-11T21:51:57.455053+0000) 2024-03-11T21:52:13.836 INFO:journalctl@ceph.osd.7.smithi097.stdout:Mar 11 21:52:13 smithi097 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd-7[56319]: 2024-03-11T21:52:13.372+0000 7ffbb45e2700 -1 osd.7 554 heartbeat_check: no reply from 172.21.15.92:6807 osd.1 since back 2024-03-11T21:51:39.656754+0000 front 2024-03-11T21:51:39.656727+0000 (oldest deadline 2024-03-11T21:52:05.556480+0000) 2024-03-11T21:52:14.336 INFO:journalctl@ceph.osd.5.smithi097.stdout:Mar 11 21:52:13 smithi097 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd-5[44269]: 2024-03-11T21:52:13.953+0000 7f6c4e1e6700 -1 osd.5 554 heartbeat_check: no reply from 172.21.15.92:6803 osd.0 since back 2024-03-11T21:51:31.785021+0000 front 2024-03-11T21:51:31.785129+0000 (oldest deadline 2024-03-11T21:51:55.885103+0000) 2024-03-11T21:52:14.336 INFO:journalctl@ceph.osd.5.smithi097.stdout:Mar 11 21:52:13 smithi097 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd-5[44269]: 2024-03-11T21:52:13.953+0000 7f6c4e1e6700 -1 osd.5 554 heartbeat_check: no reply from 172.21.15.92:6807 osd.1 since back 2024-03-11T21:51:39.886656+0000 front 2024-03-11T21:51:39.886672+0000 (oldest deadline 2024-03-11T21:52:04.586633+0000) 2024-03-11T21:52:14.337 INFO:journalctl@ceph.osd.6.smithi097.stdout:Mar 11 21:52:14 smithi097 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd-6[50273]: 2024-03-11T21:52:14.023+0000 7f51a9328700 -1 osd.6 554 heartbeat_check: no reply from 172.21.15.92:6803 osd.0 since back 2024-03-11T21:51:31.564540+0000 front 2024-03-11T21:51:31.564552+0000 (oldest deadline 2024-03-11T21:51:57.464467+0000) 2024-03-11T21:52:14.337 INFO:journalctl@ceph.osd.6.smithi097.stdout:Mar 11 21:52:14 smithi097 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd-6[50273]: 2024-03-11T21:52:14.023+0000 7f51a9328700 -1 osd.6 554 heartbeat_check: no reply from 172.21.15.92:6807 osd.1 since back 2024-03-11T21:51:42.666096+0000 front 2024-03-11T21:51:42.666115+0000 (oldest deadline 2024-03-11T21:52:08.566007+0000) 2024-03-11T21:52:14.836 INFO:journalctl@ceph.osd.7.smithi097.stdout:Mar 11 21:52:14 smithi097 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd-7[56319]: 2024-03-11T21:52:14.400+0000 7ffbb45e2700 -1 osd.7 554 heartbeat_check: no reply from 172.21.15.92:6803 osd.0 since back 2024-03-11T21:51:32.154875+0000 front 2024-03-11T21:51:32.154922+0000 (oldest deadline 2024-03-11T21:51:57.455053+0000) 2024-03-11T21:52:14.836 INFO:journalctl@ceph.osd.7.smithi097.stdout:Mar 11 21:52:14 smithi097 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd-7[56319]: 2024-03-11T21:52:14.400+0000 7ffbb45e2700 -1 osd.7 554 heartbeat_check: no reply from 172.21.15.92:6807 osd.1 since back 2024-03-11T21:51:39.656754+0000 front 2024-03-11T21:51:39.656727+0000 (oldest deadline 2024-03-11T21:52:05.556480+0000) 2024-03-11T21:52:15.336 INFO:journalctl@ceph.osd.5.smithi097.stdout:Mar 11 21:52:14 smithi097 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd-5[44269]: 2024-03-11T21:52:14.926+0000 7f6c4e1e6700 -1 osd.5 554 heartbeat_check: no reply from 172.21.15.92:6803 osd.0 since back 2024-03-11T21:51:31.785021+0000 front 2024-03-11T21:51:31.785129+0000 (oldest deadline 2024-03-11T21:51:55.885103+0000) 2024-03-11T21:52:15.337 INFO:journalctl@ceph.osd.5.smithi097.stdout:Mar 11 21:52:14 smithi097 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd-5[44269]: 2024-03-11T21:52:14.926+0000 7f6c4e1e6700 -1 osd.5 554 heartbeat_check: no reply from 172.21.15.92:6807 osd.1 since back 2024-03-11T21:51:39.886656+0000 front 2024-03-11T21:51:39.886672+0000 (oldest deadline 2024-03-11T21:52:04.586633+0000) 2024-03-11T21:52:15.337 INFO:journalctl@ceph.osd.6.smithi097.stdout:Mar 11 21:52:15 smithi097 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd-6[50273]: 2024-03-11T21:52:15.055+0000 7f51a9328700 -1 osd.6 554 heartbeat_check: no reply from 172.21.15.92:6803 osd.0 since back 2024-03-11T21:51:31.564540+0000 front 2024-03-11T21:51:31.564552+0000 (oldest deadline 2024-03-11T21:51:57.464467+0000) 2024-03-11T21:52:15.337 INFO:journalctl@ceph.osd.6.smithi097.stdout:Mar 11 21:52:15 smithi097 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd-6[50273]: 2024-03-11T21:52:15.055+0000 7f51a9328700 -1 osd.6 554 heartbeat_check: no reply from 172.21.15.92:6807 osd.1 since back 2024-03-11T21:51:42.666096+0000 front 2024-03-11T21:51:42.666115+0000 (oldest deadline 2024-03-11T21:52:08.566007+0000) 2024-03-11T21:52:15.337 INFO:journalctl@ceph.osd.6.smithi097.stdout:Mar 11 21:52:15 smithi097 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd-6[50273]: 2024-03-11T21:52:15.055+0000 7f51a9328700 -1 osd.6 554 heartbeat_check: no reply from 172.21.15.92:6811 osd.2 since back 2024-03-11T21:51:50.866983+0000 front 2024-03-11T21:51:50.867022+0000 (oldest deadline 2024-03-11T21:52:14.967047+0000) 2024-03-11T21:52:15.836 INFO:journalctl@ceph.osd.7.smithi097.stdout:Mar 11 21:52:15 smithi097 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd-7[56319]: 2024-03-11T21:52:15.352+0000 7ffbb45e2700 -1 osd.7 554 heartbeat_check: no reply from 172.21.15.92:6803 osd.0 since back 2024-03-11T21:51:32.154875+0000 front 2024-03-11T21:51:32.154922+0000 (oldest deadline 2024-03-11T21:51:57.455053+0000) 2024-03-11T21:52:15.836 INFO:journalctl@ceph.osd.7.smithi097.stdout:Mar 11 21:52:15 smithi097 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd-7[56319]: 2024-03-11T21:52:15.352+0000 7ffbb45e2700 -1 osd.7 554 heartbeat_check: no reply from 172.21.15.92:6807 osd.1 since back 2024-03-11T21:51:39.656754+0000 front 2024-03-11T21:51:39.656727+0000 (oldest deadline 2024-03-11T21:52:05.556480+0000) 2024-03-11T21:52:16.336 INFO:journalctl@ceph.osd.6.smithi097.stdout:Mar 11 21:52:16 smithi097 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd-6[50273]: 2024-03-11T21:52:16.017+0000 7f51a9328700 -1 osd.6 554 heartbeat_check: no reply from 172.21.15.92:6803 osd.0 since back 2024-03-11T21:51:31.564540+0000 front 2024-03-11T21:51:31.564552+0000 (oldest deadline 2024-03-11T21:51:57.464467+0000) 2024-03-11T21:52:16.336 INFO:journalctl@ceph.osd.6.smithi097.stdout:Mar 11 21:52:16 smithi097 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd-6[50273]: 2024-03-11T21:52:16.017+0000 7f51a9328700 -1 osd.6 554 heartbeat_check: no reply from 172.21.15.92:6807 osd.1 since back 2024-03-11T21:51:42.666096+0000 front 2024-03-11T21:51:42.666115+0000 (oldest deadline 2024-03-11T21:52:08.566007+0000) 2024-03-11T21:52:16.336 INFO:journalctl@ceph.osd.6.smithi097.stdout:Mar 11 21:52:16 smithi097 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd-6[50273]: 2024-03-11T21:52:16.017+0000 7f51a9328700 -1 osd.6 554 heartbeat_check: no reply from 172.21.15.92:6811 osd.2 since back 2024-03-11T21:51:50.866983+0000 front 2024-03-11T21:51:50.867022+0000 (oldest deadline 2024-03-11T21:52:14.967047+0000) 2024-03-11T21:52:16.337 INFO:journalctl@ceph.osd.5.smithi097.stdout:Mar 11 21:52:15 smithi097 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd-5[44269]: 2024-03-11T21:52:15.939+0000 7f6c4e1e6700 -1 osd.5 554 heartbeat_check: no reply from 172.21.15.92:6803 osd.0 since back 2024-03-11T21:51:31.785021+0000 front 2024-03-11T21:51:31.785129+0000 (oldest deadline 2024-03-11T21:51:55.885103+0000) 2024-03-11T21:52:16.337 INFO:journalctl@ceph.osd.5.smithi097.stdout:Mar 11 21:52:15 smithi097 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd-5[44269]: 2024-03-11T21:52:15.939+0000 7f6c4e1e6700 -1 osd.5 554 heartbeat_check: no reply from 172.21.15.92:6807 osd.1 since back 2024-03-11T21:51:39.886656+0000 front 2024-03-11T21:51:39.886672+0000 (oldest deadline 2024-03-11T21:52:04.586633+0000) 2024-03-11T21:52:16.337 INFO:journalctl@ceph.osd.5.smithi097.stdout:Mar 11 21:52:15 smithi097 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd-5[44269]: 2024-03-11T21:52:15.939+0000 7f6c4e1e6700 -1 osd.5 554 heartbeat_check: no reply from 172.21.15.92:6811 osd.2 since back 2024-03-11T21:51:49.788003+0000 front 2024-03-11T21:51:49.788066+0000 (oldest deadline 2024-03-11T21:52:15.088007+0000) 2024-03-11T21:52:16.836 INFO:journalctl@ceph.osd.7.smithi097.stdout:Mar 11 21:52:16 smithi097 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd-7[56319]: 2024-03-11T21:52:16.381+0000 7ffbb45e2700 -1 osd.7 554 heartbeat_check: no reply from 172.21.15.92:6803 osd.0 since back 2024-03-11T21:51:32.154875+0000 front 2024-03-11T21:51:32.154922+0000 (oldest deadline 2024-03-11T21:51:57.455053+0000) 2024-03-11T21:52:16.836 INFO:journalctl@ceph.osd.7.smithi097.stdout:Mar 11 21:52:16 smithi097 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd-7[56319]: 2024-03-11T21:52:16.381+0000 7ffbb45e2700 -1 osd.7 554 heartbeat_check: no reply from 172.21.15.92:6807 osd.1 since back 2024-03-11T21:51:39.656754+0000 front 2024-03-11T21:51:39.656727+0000 (oldest deadline 2024-03-11T21:52:05.556480+0000) 2024-03-11T21:52:17.336 INFO:journalctl@ceph.osd.6.smithi097.stdout:Mar 11 21:52:17 smithi097 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd-6[50273]: 2024-03-11T21:52:17.010+0000 7f51a9328700 -1 osd.6 554 heartbeat_check: no reply from 172.21.15.92:6803 osd.0 since back 2024-03-11T21:51:31.564540+0000 front 2024-03-11T21:51:31.564552+0000 (oldest deadline 2024-03-11T21:51:57.464467+0000) 2024-03-11T21:52:17.336 INFO:journalctl@ceph.osd.6.smithi097.stdout:Mar 11 21:52:17 smithi097 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd-6[50273]: 2024-03-11T21:52:17.010+0000 7f51a9328700 -1 osd.6 554 heartbeat_check: no reply from 172.21.15.92:6807 osd.1 since back 2024-03-11T21:51:42.666096+0000 front 2024-03-11T21:51:42.666115+0000 (oldest deadline 2024-03-11T21:52:08.566007+0000) 2024-03-11T21:52:17.336 INFO:journalctl@ceph.osd.6.smithi097.stdout:Mar 11 21:52:17 smithi097 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd-6[50273]: 2024-03-11T21:52:17.010+0000 7f51a9328700 -1 osd.6 554 heartbeat_check: no reply from 172.21.15.92:6811 osd.2 since back 2024-03-11T21:51:50.866983+0000 front 2024-03-11T21:51:50.867022+0000 (oldest deadline 2024-03-11T21:52:14.967047+0000) 2024-03-11T21:52:17.753 INFO:journalctl@ceph.osd.5.smithi097.stdout:Mar 11 21:52:17 smithi097 podman[160834]: 2024-03-11 21:52:17.498154745 +0000 UTC m=+5.675530078 container died b8092e9e4bdff4def127fd748a89e349b9e135fced1886851ebdb7c81ff6c30f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:8fe9635f3beb0247496d7e321f3068bd10c333c433ef4824d7e1c1a7d3d78379, name=ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd-5, GIT_CLEAN=True, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=main-2b0101e, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240307) 2024-03-11T21:52:17.753 INFO:journalctl@ceph.osd.7.smithi097.stdout:Mar 11 21:52:17 smithi097 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd-7[56319]: 2024-03-11T21:52:17.406+0000 7ffbb45e2700 -1 osd.7 554 heartbeat_check: no reply from 172.21.15.92:6803 osd.0 since back 2024-03-11T21:51:32.154875+0000 front 2024-03-11T21:51:32.154922+0000 (oldest deadline 2024-03-11T21:51:57.455053+0000) 2024-03-11T21:52:17.753 INFO:journalctl@ceph.osd.7.smithi097.stdout:Mar 11 21:52:17 smithi097 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd-7[56319]: 2024-03-11T21:52:17.406+0000 7ffbb45e2700 -1 osd.7 554 heartbeat_check: no reply from 172.21.15.92:6807 osd.1 since back 2024-03-11T21:51:39.656754+0000 front 2024-03-11T21:51:39.656727+0000 (oldest deadline 2024-03-11T21:52:05.556480+0000) 2024-03-11T21:52:18.022 INFO:journalctl@ceph.osd.5.smithi097.stdout:Mar 11 21:52:17 smithi097 podman[160834]: 2024-03-11 21:52:17.860338496 +0000 UTC m=+6.037713829 container cleanup b8092e9e4bdff4def127fd748a89e349b9e135fced1886851ebdb7c81ff6c30f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:8fe9635f3beb0247496d7e321f3068bd10c333c433ef4824d7e1c1a7d3d78379, name=ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd-5, org.label-schema.build-date=20240307, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, GIT_CLEAN=True, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, RELEASE=main-2b0101e, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30) 2024-03-11T21:52:18.022 INFO:journalctl@ceph.osd.5.smithi097.stdout:Mar 11 21:52:17 smithi097 bash[160834]: ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd-5 2024-03-11T21:52:18.336 INFO:journalctl@ceph.osd.5.smithi097.stdout:Mar 11 21:52:18 smithi097 podman[160852]: 2024-03-11 21:52:18.161588815 +0000 UTC m=+0.661474305 container remove b8092e9e4bdff4def127fd748a89e349b9e135fced1886851ebdb7c81ff6c30f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:8fe9635f3beb0247496d7e321f3068bd10c333c433ef4824d7e1c1a7d3d78379, name=ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd-5, org.label-schema.build-date=20240307, ceph=True, GIT_CLEAN=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=main-2b0101e, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD) 2024-03-11T21:52:18.336 INFO:journalctl@ceph.osd.6.smithi097.stdout:Mar 11 21:52:18 smithi097 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd-6[50273]: 2024-03-11T21:52:18.028+0000 7f51a9328700 -1 osd.6 554 heartbeat_check: no reply from 172.21.15.92:6803 osd.0 since back 2024-03-11T21:51:31.564540+0000 front 2024-03-11T21:51:31.564552+0000 (oldest deadline 2024-03-11T21:51:57.464467+0000) 2024-03-11T21:52:18.337 INFO:journalctl@ceph.osd.6.smithi097.stdout:Mar 11 21:52:18 smithi097 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd-6[50273]: 2024-03-11T21:52:18.028+0000 7f51a9328700 -1 osd.6 554 heartbeat_check: no reply from 172.21.15.92:6807 osd.1 since back 2024-03-11T21:51:42.666096+0000 front 2024-03-11T21:51:42.666115+0000 (oldest deadline 2024-03-11T21:52:08.566007+0000) 2024-03-11T21:52:18.337 INFO:journalctl@ceph.osd.6.smithi097.stdout:Mar 11 21:52:18 smithi097 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd-6[50273]: 2024-03-11T21:52:18.028+0000 7f51a9328700 -1 osd.6 554 heartbeat_check: no reply from 172.21.15.92:6811 osd.2 since back 2024-03-11T21:51:50.866983+0000 front 2024-03-11T21:51:50.867022+0000 (oldest deadline 2024-03-11T21:52:14.967047+0000) 2024-03-11T21:52:18.707 INFO:journalctl@ceph.osd.5.smithi097.stdout:Mar 11 21:52:18 smithi097 podman[160942]: 2024-03-11 21:52:18.417712816 +0000 UTC m=+0.017664103 image pull 4133f8bc41bb201212a28168427a2edb1f6f3795e694bf61031ea9e2f3ff5e87 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:8fe9635f3beb0247496d7e321f3068bd10c333c433ef4824d7e1c1a7d3d78379 2024-03-11T21:52:18.707 INFO:journalctl@ceph.osd.5.smithi097.stdout:Mar 11 21:52:18 smithi097 podman[160942]: 2024-03-11 21:52:18.537457941 +0000 UTC m=+0.137409215 container create 3a43729941991bc1167dc589761ef617e8f9f610340d7d7d20312f6a46fa05b1 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:8fe9635f3beb0247496d7e321f3068bd10c333c433ef4824d7e1c1a7d3d78379, name=ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd-5-deactivate, io.buildah.version=1.33.5, RELEASE=main-2b0101e, maintainer=Guillaume Abrioux , GIT_CLEAN=True, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240307, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.license=GPLv2) 2024-03-11T21:52:18.707 INFO:journalctl@ceph.osd.7.smithi097.stdout:Mar 11 21:52:18 smithi097 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd-7[56319]: 2024-03-11T21:52:18.451+0000 7ffbb45e2700 -1 osd.7 554 heartbeat_check: no reply from 172.21.15.92:6803 osd.0 since back 2024-03-11T21:51:32.154875+0000 front 2024-03-11T21:51:32.154922+0000 (oldest deadline 2024-03-11T21:51:57.455053+0000) 2024-03-11T21:52:18.707 INFO:journalctl@ceph.osd.7.smithi097.stdout:Mar 11 21:52:18 smithi097 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd-7[56319]: 2024-03-11T21:52:18.451+0000 7ffbb45e2700 -1 osd.7 554 heartbeat_check: no reply from 172.21.15.92:6807 osd.1 since back 2024-03-11T21:51:39.656754+0000 front 2024-03-11T21:51:39.656727+0000 (oldest deadline 2024-03-11T21:52:05.556480+0000) 2024-03-11T21:52:18.707 INFO:journalctl@ceph.osd.7.smithi097.stdout:Mar 11 21:52:18 smithi097 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd-7[56319]: 2024-03-11T21:52:18.451+0000 7ffbb45e2700 -1 osd.7 554 heartbeat_check: no reply from 172.21.15.92:6811 osd.2 since back 2024-03-11T21:51:49.057616+0000 front 2024-03-11T21:51:49.057558+0000 (oldest deadline 2024-03-11T21:52:17.857957+0000) 2024-03-11T21:52:18.997 INFO:journalctl@ceph.osd.5.smithi097.stdout:Mar 11 21:52:18 smithi097 podman[160942]: 2024-03-11 21:52:18.7150024 +0000 UTC m=+0.314953683 container init 3a43729941991bc1167dc589761ef617e8f9f610340d7d7d20312f6a46fa05b1 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:8fe9635f3beb0247496d7e321f3068bd10c333c433ef4824d7e1c1a7d3d78379, name=ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd-5-deactivate, org.label-schema.build-date=20240307, ceph=True, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, RELEASE=main-2b0101e, org.label-schema.vendor=CentOS) 2024-03-11T21:52:18.997 INFO:journalctl@ceph.osd.5.smithi097.stdout:Mar 11 21:52:18 smithi097 podman[160942]: 2024-03-11 21:52:18.71891363 +0000 UTC m=+0.318864904 container start 3a43729941991bc1167dc589761ef617e8f9f610340d7d7d20312f6a46fa05b1 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:8fe9635f3beb0247496d7e321f3068bd10c333c433ef4824d7e1c1a7d3d78379, name=ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd-5-deactivate, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, ceph=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.build-date=20240307, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, RELEASE=main-2b0101e, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2) 2024-03-11T21:52:18.997 INFO:journalctl@ceph.osd.5.smithi097.stdout:Mar 11 21:52:18 smithi097 podman[160942]: 2024-03-11 21:52:18.806842017 +0000 UTC m=+0.406793305 container attach 3a43729941991bc1167dc589761ef617e8f9f610340d7d7d20312f6a46fa05b1 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:8fe9635f3beb0247496d7e321f3068bd10c333c433ef4824d7e1c1a7d3d78379, name=ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd-5-deactivate, ceph=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, GIT_CLEAN=True, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240307, GIT_BRANCH=HEAD, RELEASE=main-2b0101e, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-03-11T21:52:19.285 INFO:journalctl@ceph.osd.5.smithi097.stdout:Mar 11 21:52:19 smithi097 podman[160942]: 2024-03-11 21:52:19.004775294 +0000 UTC m=+0.604726570 container died 3a43729941991bc1167dc589761ef617e8f9f610340d7d7d20312f6a46fa05b1 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:8fe9635f3beb0247496d7e321f3068bd10c333c433ef4824d7e1c1a7d3d78379, name=ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd-5-deactivate, ceph=True, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, RELEASE=main-2b0101e, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.build-date=20240307, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux ) 2024-03-11T21:52:19.286 INFO:journalctl@ceph.osd.6.smithi097.stdout:Mar 11 21:52:19 smithi097 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd-6[50273]: 2024-03-11T21:52:19.065+0000 7f51a9328700 -1 osd.6 554 heartbeat_check: no reply from 172.21.15.92:6803 osd.0 since back 2024-03-11T21:51:31.564540+0000 front 2024-03-11T21:51:31.564552+0000 (oldest deadline 2024-03-11T21:51:57.464467+0000) 2024-03-11T21:52:19.286 INFO:journalctl@ceph.osd.6.smithi097.stdout:Mar 11 21:52:19 smithi097 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd-6[50273]: 2024-03-11T21:52:19.065+0000 7f51a9328700 -1 osd.6 554 heartbeat_check: no reply from 172.21.15.92:6807 osd.1 since back 2024-03-11T21:51:42.666096+0000 front 2024-03-11T21:51:42.666115+0000 (oldest deadline 2024-03-11T21:52:08.566007+0000) 2024-03-11T21:52:19.286 INFO:journalctl@ceph.osd.6.smithi097.stdout:Mar 11 21:52:19 smithi097 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd-6[50273]: 2024-03-11T21:52:19.065+0000 7f51a9328700 -1 osd.6 554 heartbeat_check: no reply from 172.21.15.92:6811 osd.2 since back 2024-03-11T21:51:50.866983+0000 front 2024-03-11T21:51:50.867022+0000 (oldest deadline 2024-03-11T21:52:14.967047+0000) 2024-03-11T21:52:19.577 INFO:journalctl@ceph.osd.7.smithi097.stdout:Mar 11 21:52:19 smithi097 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd-7[56319]: 2024-03-11T21:52:19.409+0000 7ffbb45e2700 -1 osd.7 554 heartbeat_check: no reply from 172.21.15.92:6803 osd.0 since back 2024-03-11T21:51:32.154875+0000 front 2024-03-11T21:51:32.154922+0000 (oldest deadline 2024-03-11T21:51:57.455053+0000) 2024-03-11T21:52:19.577 INFO:journalctl@ceph.osd.7.smithi097.stdout:Mar 11 21:52:19 smithi097 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd-7[56319]: 2024-03-11T21:52:19.409+0000 7ffbb45e2700 -1 osd.7 554 heartbeat_check: no reply from 172.21.15.92:6807 osd.1 since back 2024-03-11T21:51:39.656754+0000 front 2024-03-11T21:51:39.656727+0000 (oldest deadline 2024-03-11T21:52:05.556480+0000) 2024-03-11T21:52:19.577 INFO:journalctl@ceph.osd.7.smithi097.stdout:Mar 11 21:52:19 smithi097 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd-7[56319]: 2024-03-11T21:52:19.409+0000 7ffbb45e2700 -1 osd.7 554 heartbeat_check: no reply from 172.21.15.92:6811 osd.2 since back 2024-03-11T21:51:49.057616+0000 front 2024-03-11T21:51:49.057558+0000 (oldest deadline 2024-03-11T21:52:17.857957+0000) 2024-03-11T21:52:19.611 DEBUG:teuthology.orchestra.run.smithi097:> sudo pkill -f 'journalctl -f -n 0 -u ceph-00db0016-dfed-11ee-95c8-87774f69a715@osd.5.service' 2024-03-11T21:52:19.653 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-11T21:52:19.653 INFO:tasks.cephadm.osd.5:Stopped osd.5 2024-03-11T21:52:19.653 INFO:tasks.cephadm.osd.6:Stopping osd.6... 2024-03-11T21:52:19.653 DEBUG:teuthology.orchestra.run.smithi097:> sudo systemctl stop ceph-00db0016-dfed-11ee-95c8-87774f69a715@osd.6 2024-03-11T21:52:19.836 INFO:journalctl@ceph.osd.6.smithi097.stdout:Mar 11 21:52:19 smithi097 systemd[1]: Stopping Ceph osd.6 for 00db0016-dfed-11ee-95c8-87774f69a715... 2024-03-11T21:52:20.336 INFO:journalctl@ceph.osd.6.smithi097.stdout:Mar 11 21:52:19 smithi097 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd-6[50273]: 2024-03-11T21:52:19.995+0000 7f51acd34700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.6 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-11T21:52:20.337 INFO:journalctl@ceph.osd.6.smithi097.stdout:Mar 11 21:52:19 smithi097 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd-6[50273]: 2024-03-11T21:52:19.995+0000 7f51acd34700 -1 osd.6 554 *** Got signal Terminated *** 2024-03-11T21:52:20.337 INFO:journalctl@ceph.osd.6.smithi097.stdout:Mar 11 21:52:19 smithi097 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd-6[50273]: 2024-03-11T21:52:19.995+0000 7f51acd34700 -1 osd.6 554 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-03-11T21:52:20.337 INFO:journalctl@ceph.osd.6.smithi097.stdout:Mar 11 21:52:20 smithi097 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd-6[50273]: 2024-03-11T21:52:20.048+0000 7f51a9328700 -1 osd.6 554 heartbeat_check: no reply from 172.21.15.92:6803 osd.0 since back 2024-03-11T21:51:31.564540+0000 front 2024-03-11T21:51:31.564552+0000 (oldest deadline 2024-03-11T21:51:57.464467+0000) 2024-03-11T21:52:20.337 INFO:journalctl@ceph.osd.6.smithi097.stdout:Mar 11 21:52:20 smithi097 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd-6[50273]: 2024-03-11T21:52:20.048+0000 7f51a9328700 -1 osd.6 554 heartbeat_check: no reply from 172.21.15.92:6807 osd.1 since back 2024-03-11T21:51:42.666096+0000 front 2024-03-11T21:51:42.666115+0000 (oldest deadline 2024-03-11T21:52:08.566007+0000) 2024-03-11T21:52:20.337 INFO:journalctl@ceph.osd.6.smithi097.stdout:Mar 11 21:52:20 smithi097 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd-6[50273]: 2024-03-11T21:52:20.048+0000 7f51a9328700 -1 osd.6 554 heartbeat_check: no reply from 172.21.15.92:6811 osd.2 since back 2024-03-11T21:51:50.866983+0000 front 2024-03-11T21:51:50.867022+0000 (oldest deadline 2024-03-11T21:52:14.967047+0000) 2024-03-11T21:52:20.836 INFO:journalctl@ceph.osd.7.smithi097.stdout:Mar 11 21:52:20 smithi097 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd-7[56319]: 2024-03-11T21:52:20.366+0000 7ffbb45e2700 -1 osd.7 554 heartbeat_check: no reply from 172.21.15.92:6803 osd.0 since back 2024-03-11T21:51:32.154875+0000 front 2024-03-11T21:51:32.154922+0000 (oldest deadline 2024-03-11T21:51:57.455053+0000) 2024-03-11T21:52:20.836 INFO:journalctl@ceph.osd.7.smithi097.stdout:Mar 11 21:52:20 smithi097 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd-7[56319]: 2024-03-11T21:52:20.366+0000 7ffbb45e2700 -1 osd.7 554 heartbeat_check: no reply from 172.21.15.92:6807 osd.1 since back 2024-03-11T21:51:39.656754+0000 front 2024-03-11T21:51:39.656727+0000 (oldest deadline 2024-03-11T21:52:05.556480+0000) 2024-03-11T21:52:20.836 INFO:journalctl@ceph.osd.7.smithi097.stdout:Mar 11 21:52:20 smithi097 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd-7[56319]: 2024-03-11T21:52:20.366+0000 7ffbb45e2700 -1 osd.7 554 heartbeat_check: no reply from 172.21.15.92:6811 osd.2 since back 2024-03-11T21:51:49.057616+0000 front 2024-03-11T21:51:49.057558+0000 (oldest deadline 2024-03-11T21:52:17.857957+0000) 2024-03-11T21:52:21.336 INFO:journalctl@ceph.osd.6.smithi097.stdout:Mar 11 21:52:21 smithi097 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd-6[50273]: 2024-03-11T21:52:21.042+0000 7f51a9328700 -1 osd.6 554 heartbeat_check: no reply from 172.21.15.92:6803 osd.0 since back 2024-03-11T21:51:31.564540+0000 front 2024-03-11T21:51:31.564552+0000 (oldest deadline 2024-03-11T21:51:57.464467+0000) 2024-03-11T21:52:21.336 INFO:journalctl@ceph.osd.6.smithi097.stdout:Mar 11 21:52:21 smithi097 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd-6[50273]: 2024-03-11T21:52:21.042+0000 7f51a9328700 -1 osd.6 554 heartbeat_check: no reply from 172.21.15.92:6807 osd.1 since back 2024-03-11T21:51:42.666096+0000 front 2024-03-11T21:51:42.666115+0000 (oldest deadline 2024-03-11T21:52:08.566007+0000) 2024-03-11T21:52:21.336 INFO:journalctl@ceph.osd.6.smithi097.stdout:Mar 11 21:52:21 smithi097 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd-6[50273]: 2024-03-11T21:52:21.042+0000 7f51a9328700 -1 osd.6 554 heartbeat_check: no reply from 172.21.15.92:6811 osd.2 since back 2024-03-11T21:51:50.866983+0000 front 2024-03-11T21:51:50.867022+0000 (oldest deadline 2024-03-11T21:52:14.967047+0000) 2024-03-11T21:52:21.836 INFO:journalctl@ceph.osd.7.smithi097.stdout:Mar 11 21:52:21 smithi097 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd-7[56319]: 2024-03-11T21:52:21.355+0000 7ffbb45e2700 -1 osd.7 554 heartbeat_check: no reply from 172.21.15.92:6803 osd.0 since back 2024-03-11T21:51:32.154875+0000 front 2024-03-11T21:51:32.154922+0000 (oldest deadline 2024-03-11T21:51:57.455053+0000) 2024-03-11T21:52:21.836 INFO:journalctl@ceph.osd.7.smithi097.stdout:Mar 11 21:52:21 smithi097 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd-7[56319]: 2024-03-11T21:52:21.355+0000 7ffbb45e2700 -1 osd.7 554 heartbeat_check: no reply from 172.21.15.92:6807 osd.1 since back 2024-03-11T21:51:39.656754+0000 front 2024-03-11T21:51:39.656727+0000 (oldest deadline 2024-03-11T21:52:05.556480+0000) 2024-03-11T21:52:21.837 INFO:journalctl@ceph.osd.7.smithi097.stdout:Mar 11 21:52:21 smithi097 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd-7[56319]: 2024-03-11T21:52:21.355+0000 7ffbb45e2700 -1 osd.7 554 heartbeat_check: no reply from 172.21.15.92:6811 osd.2 since back 2024-03-11T21:51:49.057616+0000 front 2024-03-11T21:51:49.057558+0000 (oldest deadline 2024-03-11T21:52:17.857957+0000) 2024-03-11T21:52:22.336 INFO:journalctl@ceph.osd.6.smithi097.stdout:Mar 11 21:52:21 smithi097 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd-6[50273]: 2024-03-11T21:52:21.995+0000 7f51a9328700 -1 osd.6 554 heartbeat_check: no reply from 172.21.15.92:6803 osd.0 since back 2024-03-11T21:51:31.564540+0000 front 2024-03-11T21:51:31.564552+0000 (oldest deadline 2024-03-11T21:51:57.464467+0000) 2024-03-11T21:52:22.336 INFO:journalctl@ceph.osd.6.smithi097.stdout:Mar 11 21:52:21 smithi097 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd-6[50273]: 2024-03-11T21:52:21.995+0000 7f51a9328700 -1 osd.6 554 heartbeat_check: no reply from 172.21.15.92:6807 osd.1 since back 2024-03-11T21:51:42.666096+0000 front 2024-03-11T21:51:42.666115+0000 (oldest deadline 2024-03-11T21:52:08.566007+0000) 2024-03-11T21:52:22.337 INFO:journalctl@ceph.osd.6.smithi097.stdout:Mar 11 21:52:21 smithi097 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd-6[50273]: 2024-03-11T21:52:21.995+0000 7f51a9328700 -1 osd.6 554 heartbeat_check: no reply from 172.21.15.92:6811 osd.2 since back 2024-03-11T21:51:50.866983+0000 front 2024-03-11T21:51:50.867022+0000 (oldest deadline 2024-03-11T21:52:14.967047+0000) 2024-03-11T21:52:22.836 INFO:journalctl@ceph.osd.7.smithi097.stdout:Mar 11 21:52:22 smithi097 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd-7[56319]: 2024-03-11T21:52:22.372+0000 7ffbb45e2700 -1 osd.7 554 heartbeat_check: no reply from 172.21.15.92:6803 osd.0 since back 2024-03-11T21:51:32.154875+0000 front 2024-03-11T21:51:32.154922+0000 (oldest deadline 2024-03-11T21:51:57.455053+0000) 2024-03-11T21:52:22.836 INFO:journalctl@ceph.osd.7.smithi097.stdout:Mar 11 21:52:22 smithi097 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd-7[56319]: 2024-03-11T21:52:22.372+0000 7ffbb45e2700 -1 osd.7 554 heartbeat_check: no reply from 172.21.15.92:6807 osd.1 since back 2024-03-11T21:51:39.656754+0000 front 2024-03-11T21:51:39.656727+0000 (oldest deadline 2024-03-11T21:52:05.556480+0000) 2024-03-11T21:52:22.836 INFO:journalctl@ceph.osd.7.smithi097.stdout:Mar 11 21:52:22 smithi097 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd-7[56319]: 2024-03-11T21:52:22.372+0000 7ffbb45e2700 -1 osd.7 554 heartbeat_check: no reply from 172.21.15.92:6811 osd.2 since back 2024-03-11T21:51:49.057616+0000 front 2024-03-11T21:51:49.057558+0000 (oldest deadline 2024-03-11T21:52:17.857957+0000) 2024-03-11T21:52:23.336 INFO:journalctl@ceph.osd.6.smithi097.stdout:Mar 11 21:52:22 smithi097 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd-6[50273]: 2024-03-11T21:52:22.955+0000 7f51a9328700 -1 osd.6 554 heartbeat_check: no reply from 172.21.15.92:6803 osd.0 since back 2024-03-11T21:51:31.564540+0000 front 2024-03-11T21:51:31.564552+0000 (oldest deadline 2024-03-11T21:51:57.464467+0000) 2024-03-11T21:52:23.337 INFO:journalctl@ceph.osd.6.smithi097.stdout:Mar 11 21:52:22 smithi097 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd-6[50273]: 2024-03-11T21:52:22.955+0000 7f51a9328700 -1 osd.6 554 heartbeat_check: no reply from 172.21.15.92:6807 osd.1 since back 2024-03-11T21:51:42.666096+0000 front 2024-03-11T21:51:42.666115+0000 (oldest deadline 2024-03-11T21:52:08.566007+0000) 2024-03-11T21:52:23.337 INFO:journalctl@ceph.osd.6.smithi097.stdout:Mar 11 21:52:22 smithi097 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd-6[50273]: 2024-03-11T21:52:22.955+0000 7f51a9328700 -1 osd.6 554 heartbeat_check: no reply from 172.21.15.92:6811 osd.2 since back 2024-03-11T21:51:50.866983+0000 front 2024-03-11T21:51:50.867022+0000 (oldest deadline 2024-03-11T21:52:14.967047+0000) 2024-03-11T21:52:23.836 INFO:journalctl@ceph.osd.7.smithi097.stdout:Mar 11 21:52:23 smithi097 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd-7[56319]: 2024-03-11T21:52:23.390+0000 7ffbb45e2700 -1 osd.7 554 heartbeat_check: no reply from 172.21.15.92:6803 osd.0 since back 2024-03-11T21:51:32.154875+0000 front 2024-03-11T21:51:32.154922+0000 (oldest deadline 2024-03-11T21:51:57.455053+0000) 2024-03-11T21:52:23.837 INFO:journalctl@ceph.osd.7.smithi097.stdout:Mar 11 21:52:23 smithi097 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd-7[56319]: 2024-03-11T21:52:23.390+0000 7ffbb45e2700 -1 osd.7 554 heartbeat_check: no reply from 172.21.15.92:6807 osd.1 since back 2024-03-11T21:51:39.656754+0000 front 2024-03-11T21:51:39.656727+0000 (oldest deadline 2024-03-11T21:52:05.556480+0000) 2024-03-11T21:52:23.837 INFO:journalctl@ceph.osd.7.smithi097.stdout:Mar 11 21:52:23 smithi097 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd-7[56319]: 2024-03-11T21:52:23.390+0000 7ffbb45e2700 -1 osd.7 554 heartbeat_check: no reply from 172.21.15.92:6811 osd.2 since back 2024-03-11T21:51:49.057616+0000 front 2024-03-11T21:51:49.057558+0000 (oldest deadline 2024-03-11T21:52:17.857957+0000) 2024-03-11T21:52:23.837 INFO:journalctl@ceph.osd.7.smithi097.stdout:Mar 11 21:52:23 smithi097 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd-7[56319]: 2024-03-11T21:52:23.390+0000 7ffbb45e2700 -1 osd.7 554 heartbeat_check: no reply from 172.21.15.92:6815 osd.3 since back 2024-03-11T21:51:57.858544+0000 front 2024-03-11T21:51:57.858438+0000 (oldest deadline 2024-03-11T21:52:23.158422+0000) 2024-03-11T21:52:24.336 INFO:journalctl@ceph.osd.6.smithi097.stdout:Mar 11 21:52:23 smithi097 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd-6[50273]: 2024-03-11T21:52:23.987+0000 7f51a9328700 -1 osd.6 554 heartbeat_check: no reply from 172.21.15.92:6803 osd.0 since back 2024-03-11T21:51:31.564540+0000 front 2024-03-11T21:51:31.564552+0000 (oldest deadline 2024-03-11T21:51:57.464467+0000) 2024-03-11T21:52:24.337 INFO:journalctl@ceph.osd.6.smithi097.stdout:Mar 11 21:52:23 smithi097 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd-6[50273]: 2024-03-11T21:52:23.987+0000 7f51a9328700 -1 osd.6 554 heartbeat_check: no reply from 172.21.15.92:6807 osd.1 since back 2024-03-11T21:51:42.666096+0000 front 2024-03-11T21:51:42.666115+0000 (oldest deadline 2024-03-11T21:52:08.566007+0000) 2024-03-11T21:52:24.337 INFO:journalctl@ceph.osd.6.smithi097.stdout:Mar 11 21:52:23 smithi097 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd-6[50273]: 2024-03-11T21:52:23.987+0000 7f51a9328700 -1 osd.6 554 heartbeat_check: no reply from 172.21.15.92:6811 osd.2 since back 2024-03-11T21:51:50.866983+0000 front 2024-03-11T21:51:50.867022+0000 (oldest deadline 2024-03-11T21:52:14.967047+0000) 2024-03-11T21:52:24.836 INFO:journalctl@ceph.osd.7.smithi097.stdout:Mar 11 21:52:24 smithi097 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd-7[56319]: 2024-03-11T21:52:24.344+0000 7ffbb45e2700 -1 osd.7 554 heartbeat_check: no reply from 172.21.15.92:6803 osd.0 since back 2024-03-11T21:51:32.154875+0000 front 2024-03-11T21:51:32.154922+0000 (oldest deadline 2024-03-11T21:51:57.455053+0000) 2024-03-11T21:52:24.837 INFO:journalctl@ceph.osd.7.smithi097.stdout:Mar 11 21:52:24 smithi097 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd-7[56319]: 2024-03-11T21:52:24.344+0000 7ffbb45e2700 -1 osd.7 554 heartbeat_check: no reply from 172.21.15.92:6807 osd.1 since back 2024-03-11T21:51:39.656754+0000 front 2024-03-11T21:51:39.656727+0000 (oldest deadline 2024-03-11T21:52:05.556480+0000) 2024-03-11T21:52:24.837 INFO:journalctl@ceph.osd.7.smithi097.stdout:Mar 11 21:52:24 smithi097 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd-7[56319]: 2024-03-11T21:52:24.344+0000 7ffbb45e2700 -1 osd.7 554 heartbeat_check: no reply from 172.21.15.92:6811 osd.2 since back 2024-03-11T21:51:49.057616+0000 front 2024-03-11T21:51:49.057558+0000 (oldest deadline 2024-03-11T21:52:17.857957+0000) 2024-03-11T21:52:24.837 INFO:journalctl@ceph.osd.7.smithi097.stdout:Mar 11 21:52:24 smithi097 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd-7[56319]: 2024-03-11T21:52:24.344+0000 7ffbb45e2700 -1 osd.7 554 heartbeat_check: no reply from 172.21.15.92:6815 osd.3 since back 2024-03-11T21:51:57.858544+0000 front 2024-03-11T21:51:57.858438+0000 (oldest deadline 2024-03-11T21:52:23.158422+0000) 2024-03-11T21:52:25.586 INFO:journalctl@ceph.osd.6.smithi097.stdout:Mar 11 21:52:25 smithi097 podman[161049]: 2024-03-11 21:52:25.491332669 +0000 UTC m=+5.604544094 container died 9b4690056a8bd8f410b3f14d3cc4a7a721fc2181efa8270d8db39ec775636d7c (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:8fe9635f3beb0247496d7e321f3068bd10c333c433ef4824d7e1c1a7d3d78379, name=ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd-6, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=main-2b0101e, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240307, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, ceph=True, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30) 2024-03-11T21:52:25.587 INFO:journalctl@ceph.osd.7.smithi097.stdout:Mar 11 21:52:25 smithi097 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd-7[56319]: 2024-03-11T21:52:25.335+0000 7ffbb45e2700 -1 osd.7 554 heartbeat_check: no reply from 172.21.15.92:6803 osd.0 since back 2024-03-11T21:51:32.154875+0000 front 2024-03-11T21:51:32.154922+0000 (oldest deadline 2024-03-11T21:51:57.455053+0000) 2024-03-11T21:52:25.587 INFO:journalctl@ceph.osd.7.smithi097.stdout:Mar 11 21:52:25 smithi097 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd-7[56319]: 2024-03-11T21:52:25.335+0000 7ffbb45e2700 -1 osd.7 554 heartbeat_check: no reply from 172.21.15.92:6807 osd.1 since back 2024-03-11T21:51:39.656754+0000 front 2024-03-11T21:51:39.656727+0000 (oldest deadline 2024-03-11T21:52:05.556480+0000) 2024-03-11T21:52:25.587 INFO:journalctl@ceph.osd.7.smithi097.stdout:Mar 11 21:52:25 smithi097 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd-7[56319]: 2024-03-11T21:52:25.335+0000 7ffbb45e2700 -1 osd.7 554 heartbeat_check: no reply from 172.21.15.92:6811 osd.2 since back 2024-03-11T21:51:49.057616+0000 front 2024-03-11T21:51:49.057558+0000 (oldest deadline 2024-03-11T21:52:17.857957+0000) 2024-03-11T21:52:25.587 INFO:journalctl@ceph.osd.7.smithi097.stdout:Mar 11 21:52:25 smithi097 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd-7[56319]: 2024-03-11T21:52:25.335+0000 7ffbb45e2700 -1 osd.7 554 heartbeat_check: no reply from 172.21.15.92:6815 osd.3 since back 2024-03-11T21:51:57.858544+0000 front 2024-03-11T21:51:57.858438+0000 (oldest deadline 2024-03-11T21:52:23.158422+0000) 2024-03-11T21:52:25.587 INFO:journalctl@ceph.prometheus.a.smithi097.stdout:Mar 11 21:52:25 smithi097 ceph-00db0016-dfed-11ee-95c8-87774f69a715-prometheus-a[73147]: ts=2024-03-11T21:52:25.315Z caller=refresh.go:99 level=error component="discovery manager scrape" discovery=http config=ceph-exporter msg="Unable to refresh target groups" err="Get \"http://172.21.15.92:8765/sd/prometheus/sd-config?service=ceph-exporter\": dial tcp 172.21.15.92:8765: connect: connection refused" 2024-03-11T21:52:25.587 INFO:journalctl@ceph.prometheus.a.smithi097.stdout:Mar 11 21:52:25 smithi097 ceph-00db0016-dfed-11ee-95c8-87774f69a715-prometheus-a[73147]: ts=2024-03-11T21:52:25.317Z caller=refresh.go:99 level=error component="discovery manager notify" discovery=http config=config-0 msg="Unable to refresh target groups" err="Get \"http://172.21.15.92:8765/sd/prometheus/sd-config?service=alertmanager\": dial tcp 172.21.15.92:8765: connect: connection refused" 2024-03-11T21:52:25.587 INFO:journalctl@ceph.prometheus.a.smithi097.stdout:Mar 11 21:52:25 smithi097 ceph-00db0016-dfed-11ee-95c8-87774f69a715-prometheus-a[73147]: ts=2024-03-11T21:52:25.318Z caller=refresh.go:99 level=error component="discovery manager scrape" discovery=http config=node msg="Unable to refresh target groups" err="Get \"http://172.21.15.92:8765/sd/prometheus/sd-config?service=node-exporter\": dial tcp 172.21.15.92:8765: connect: connection refused" 2024-03-11T21:52:25.587 INFO:journalctl@ceph.prometheus.a.smithi097.stdout:Mar 11 21:52:25 smithi097 ceph-00db0016-dfed-11ee-95c8-87774f69a715-prometheus-a[73147]: ts=2024-03-11T21:52:25.320Z caller=refresh.go:99 level=error component="discovery manager scrape" discovery=http config=ceph msg="Unable to refresh target groups" err="Get \"http://172.21.15.92:8765/sd/prometheus/sd-config?service=mgr-prometheus\": dial tcp 172.21.15.92:8765: connect: connection refused" 2024-03-11T21:52:25.587 INFO:journalctl@ceph.prometheus.a.smithi097.stdout:Mar 11 21:52:25 smithi097 ceph-00db0016-dfed-11ee-95c8-87774f69a715-prometheus-a[73147]: ts=2024-03-11T21:52:25.320Z caller=refresh.go:99 level=error component="discovery manager scrape" discovery=http config=nvmeof msg="Unable to refresh target groups" err="Get \"http://172.21.15.92:8765/sd/prometheus/sd-config?service=nvmeof\": dial tcp 172.21.15.92:8765: connect: connection refused" 2024-03-11T21:52:25.846 INFO:journalctl@ceph.osd.6.smithi097.stdout:Mar 11 21:52:25 smithi097 podman[161049]: 2024-03-11 21:52:25.838058837 +0000 UTC m=+5.951270268 container cleanup 9b4690056a8bd8f410b3f14d3cc4a7a721fc2181efa8270d8db39ec775636d7c (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:8fe9635f3beb0247496d7e321f3068bd10c333c433ef4824d7e1c1a7d3d78379, name=ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd-6, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, RELEASE=main-2b0101e, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240307, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.schema-version=1.0) 2024-03-11T21:52:25.846 INFO:journalctl@ceph.osd.6.smithi097.stdout:Mar 11 21:52:25 smithi097 bash[161049]: ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd-6 2024-03-11T21:52:26.510 INFO:journalctl@ceph.osd.6.smithi097.stdout:Mar 11 21:52:26 smithi097 podman[161067]: 2024-03-11 21:52:26.189818398 +0000 UTC m=+0.696731584 container remove 9b4690056a8bd8f410b3f14d3cc4a7a721fc2181efa8270d8db39ec775636d7c (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:8fe9635f3beb0247496d7e321f3068bd10c333c433ef4824d7e1c1a7d3d78379, name=ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd-6, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, maintainer=Guillaume Abrioux , ceph=True, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, RELEASE=main-2b0101e, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240307, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, io.buildah.version=1.33.5) 2024-03-11T21:52:26.510 INFO:journalctl@ceph.osd.7.smithi097.stdout:Mar 11 21:52:26 smithi097 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd-7[56319]: 2024-03-11T21:52:26.289+0000 7ffbb45e2700 -1 osd.7 554 heartbeat_check: no reply from 172.21.15.92:6803 osd.0 since back 2024-03-11T21:51:32.154875+0000 front 2024-03-11T21:51:32.154922+0000 (oldest deadline 2024-03-11T21:51:57.455053+0000) 2024-03-11T21:52:26.510 INFO:journalctl@ceph.osd.7.smithi097.stdout:Mar 11 21:52:26 smithi097 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd-7[56319]: 2024-03-11T21:52:26.289+0000 7ffbb45e2700 -1 osd.7 554 heartbeat_check: no reply from 172.21.15.92:6807 osd.1 since back 2024-03-11T21:51:39.656754+0000 front 2024-03-11T21:51:39.656727+0000 (oldest deadline 2024-03-11T21:52:05.556480+0000) 2024-03-11T21:52:26.510 INFO:journalctl@ceph.osd.7.smithi097.stdout:Mar 11 21:52:26 smithi097 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd-7[56319]: 2024-03-11T21:52:26.289+0000 7ffbb45e2700 -1 osd.7 554 heartbeat_check: no reply from 172.21.15.92:6811 osd.2 since back 2024-03-11T21:51:49.057616+0000 front 2024-03-11T21:51:49.057558+0000 (oldest deadline 2024-03-11T21:52:17.857957+0000) 2024-03-11T21:52:26.510 INFO:journalctl@ceph.osd.7.smithi097.stdout:Mar 11 21:52:26 smithi097 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd-7[56319]: 2024-03-11T21:52:26.289+0000 7ffbb45e2700 -1 osd.7 554 heartbeat_check: no reply from 172.21.15.92:6815 osd.3 since back 2024-03-11T21:51:57.858544+0000 front 2024-03-11T21:51:57.858438+0000 (oldest deadline 2024-03-11T21:52:23.158422+0000) 2024-03-11T21:52:26.808 INFO:journalctl@ceph.osd.6.smithi097.stdout:Mar 11 21:52:26 smithi097 podman[161167]: 2024-03-11 21:52:26.421113051 +0000 UTC m=+0.018723733 image pull 4133f8bc41bb201212a28168427a2edb1f6f3795e694bf61031ea9e2f3ff5e87 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:8fe9635f3beb0247496d7e321f3068bd10c333c433ef4824d7e1c1a7d3d78379 2024-03-11T21:52:26.808 INFO:journalctl@ceph.osd.6.smithi097.stdout:Mar 11 21:52:26 smithi097 podman[161167]: 2024-03-11 21:52:26.546334696 +0000 UTC m=+0.143945371 container create 0e15dfd257e329c253c9db6447f2e114a143463a815903f27559daad2a4d784f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:8fe9635f3beb0247496d7e321f3068bd10c333c433ef4824d7e1c1a7d3d78379, name=ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd-6-deactivate, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, ceph=True, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, RELEASE=main-2b0101e, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.build-date=20240307) 2024-03-11T21:52:26.808 INFO:journalctl@ceph.osd.6.smithi097.stdout:Mar 11 21:52:26 smithi097 podman[161167]: 2024-03-11 21:52:26.731926398 +0000 UTC m=+0.329537080 container init 0e15dfd257e329c253c9db6447f2e114a143463a815903f27559daad2a4d784f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:8fe9635f3beb0247496d7e321f3068bd10c333c433ef4824d7e1c1a7d3d78379, name=ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd-6-deactivate, RELEASE=main-2b0101e, ceph=True, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240307, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0) 2024-03-11T21:52:26.808 INFO:journalctl@ceph.osd.6.smithi097.stdout:Mar 11 21:52:26 smithi097 podman[161167]: 2024-03-11 21:52:26.734964186 +0000 UTC m=+0.332574869 container start 0e15dfd257e329c253c9db6447f2e114a143463a815903f27559daad2a4d784f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:8fe9635f3beb0247496d7e321f3068bd10c333c433ef4824d7e1c1a7d3d78379, name=ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd-6-deactivate, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, org.label-schema.build-date=20240307, io.buildah.version=1.33.5, ceph=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, RELEASE=main-2b0101e, GIT_CLEAN=True) 2024-03-11T21:52:27.087 INFO:journalctl@ceph.osd.6.smithi097.stdout:Mar 11 21:52:26 smithi097 podman[161167]: 2024-03-11 21:52:26.815294462 +0000 UTC m=+0.412905139 container attach 0e15dfd257e329c253c9db6447f2e114a143463a815903f27559daad2a4d784f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:8fe9635f3beb0247496d7e321f3068bd10c333c433ef4824d7e1c1a7d3d78379, name=ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd-6-deactivate, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, org.label-schema.build-date=20240307, org.label-schema.license=GPLv2, GIT_CLEAN=True, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, RELEASE=main-2b0101e) 2024-03-11T21:52:27.087 INFO:journalctl@ceph.osd.6.smithi097.stdout:Mar 11 21:52:27 smithi097 podman[161167]: 2024-03-11 21:52:27.036740242 +0000 UTC m=+0.634350917 container died 0e15dfd257e329c253c9db6447f2e114a143463a815903f27559daad2a4d784f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:8fe9635f3beb0247496d7e321f3068bd10c333c433ef4824d7e1c1a7d3d78379, name=ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd-6-deactivate, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, ceph=True, org.label-schema.build-date=20240307, RELEASE=main-2b0101e, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, GIT_CLEAN=True) 2024-03-11T21:52:27.586 INFO:journalctl@ceph.osd.7.smithi097.stdout:Mar 11 21:52:27 smithi097 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd-7[56319]: 2024-03-11T21:52:27.247+0000 7ffbb45e2700 -1 osd.7 554 heartbeat_check: no reply from 172.21.15.92:6803 osd.0 since back 2024-03-11T21:51:32.154875+0000 front 2024-03-11T21:51:32.154922+0000 (oldest deadline 2024-03-11T21:51:57.455053+0000) 2024-03-11T21:52:27.587 INFO:journalctl@ceph.osd.7.smithi097.stdout:Mar 11 21:52:27 smithi097 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd-7[56319]: 2024-03-11T21:52:27.247+0000 7ffbb45e2700 -1 osd.7 554 heartbeat_check: no reply from 172.21.15.92:6807 osd.1 since back 2024-03-11T21:51:39.656754+0000 front 2024-03-11T21:51:39.656727+0000 (oldest deadline 2024-03-11T21:52:05.556480+0000) 2024-03-11T21:52:27.587 INFO:journalctl@ceph.osd.7.smithi097.stdout:Mar 11 21:52:27 smithi097 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd-7[56319]: 2024-03-11T21:52:27.247+0000 7ffbb45e2700 -1 osd.7 554 heartbeat_check: no reply from 172.21.15.92:6811 osd.2 since back 2024-03-11T21:51:49.057616+0000 front 2024-03-11T21:51:49.057558+0000 (oldest deadline 2024-03-11T21:52:17.857957+0000) 2024-03-11T21:52:27.587 INFO:journalctl@ceph.osd.7.smithi097.stdout:Mar 11 21:52:27 smithi097 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd-7[56319]: 2024-03-11T21:52:27.247+0000 7ffbb45e2700 -1 osd.7 554 heartbeat_check: no reply from 172.21.15.92:6815 osd.3 since back 2024-03-11T21:51:57.858544+0000 front 2024-03-11T21:51:57.858438+0000 (oldest deadline 2024-03-11T21:52:23.158422+0000) 2024-03-11T21:52:27.644 DEBUG:teuthology.orchestra.run.smithi097:> sudo pkill -f 'journalctl -f -n 0 -u ceph-00db0016-dfed-11ee-95c8-87774f69a715@osd.6.service' 2024-03-11T21:52:27.687 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-11T21:52:27.688 INFO:tasks.cephadm.osd.6:Stopped osd.6 2024-03-11T21:52:27.688 INFO:tasks.cephadm.osd.7:Stopping osd.7... 2024-03-11T21:52:27.688 DEBUG:teuthology.orchestra.run.smithi097:> sudo systemctl stop ceph-00db0016-dfed-11ee-95c8-87774f69a715@osd.7 2024-03-11T21:52:28.029 INFO:journalctl@ceph.osd.7.smithi097.stdout:Mar 11 21:52:27 smithi097 systemd[1]: Stopping Ceph osd.7 for 00db0016-dfed-11ee-95c8-87774f69a715... 2024-03-11T21:52:28.336 INFO:journalctl@ceph.osd.7.smithi097.stdout:Mar 11 21:52:28 smithi097 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd-7[56319]: 2024-03-11T21:52:28.036+0000 7ffbb7fee700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.7 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-11T21:52:28.337 INFO:journalctl@ceph.osd.7.smithi097.stdout:Mar 11 21:52:28 smithi097 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd-7[56319]: 2024-03-11T21:52:28.036+0000 7ffbb7fee700 -1 osd.7 554 *** Got signal Terminated *** 2024-03-11T21:52:28.337 INFO:journalctl@ceph.osd.7.smithi097.stdout:Mar 11 21:52:28 smithi097 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd-7[56319]: 2024-03-11T21:52:28.036+0000 7ffbb7fee700 -1 osd.7 554 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-03-11T21:52:28.337 INFO:journalctl@ceph.osd.7.smithi097.stdout:Mar 11 21:52:28 smithi097 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd-7[56319]: 2024-03-11T21:52:28.234+0000 7ffbb45e2700 -1 osd.7 554 heartbeat_check: no reply from 172.21.15.92:6803 osd.0 since back 2024-03-11T21:51:32.154875+0000 front 2024-03-11T21:51:32.154922+0000 (oldest deadline 2024-03-11T21:51:57.455053+0000) 2024-03-11T21:52:28.337 INFO:journalctl@ceph.osd.7.smithi097.stdout:Mar 11 21:52:28 smithi097 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd-7[56319]: 2024-03-11T21:52:28.234+0000 7ffbb45e2700 -1 osd.7 554 heartbeat_check: no reply from 172.21.15.92:6807 osd.1 since back 2024-03-11T21:51:39.656754+0000 front 2024-03-11T21:51:39.656727+0000 (oldest deadline 2024-03-11T21:52:05.556480+0000) 2024-03-11T21:52:28.337 INFO:journalctl@ceph.osd.7.smithi097.stdout:Mar 11 21:52:28 smithi097 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd-7[56319]: 2024-03-11T21:52:28.234+0000 7ffbb45e2700 -1 osd.7 554 heartbeat_check: no reply from 172.21.15.92:6811 osd.2 since back 2024-03-11T21:51:49.057616+0000 front 2024-03-11T21:51:49.057558+0000 (oldest deadline 2024-03-11T21:52:17.857957+0000) 2024-03-11T21:52:28.337 INFO:journalctl@ceph.osd.7.smithi097.stdout:Mar 11 21:52:28 smithi097 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd-7[56319]: 2024-03-11T21:52:28.234+0000 7ffbb45e2700 -1 osd.7 554 heartbeat_check: no reply from 172.21.15.92:6815 osd.3 since back 2024-03-11T21:51:57.858544+0000 front 2024-03-11T21:51:57.858438+0000 (oldest deadline 2024-03-11T21:52:23.158422+0000) 2024-03-11T21:52:29.586 INFO:journalctl@ceph.osd.7.smithi097.stdout:Mar 11 21:52:29 smithi097 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd-7[56319]: 2024-03-11T21:52:29.239+0000 7ffbb45e2700 -1 osd.7 554 heartbeat_check: no reply from 172.21.15.92:6803 osd.0 since back 2024-03-11T21:51:32.154875+0000 front 2024-03-11T21:51:32.154922+0000 (oldest deadline 2024-03-11T21:51:57.455053+0000) 2024-03-11T21:52:29.587 INFO:journalctl@ceph.osd.7.smithi097.stdout:Mar 11 21:52:29 smithi097 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd-7[56319]: 2024-03-11T21:52:29.239+0000 7ffbb45e2700 -1 osd.7 554 heartbeat_check: no reply from 172.21.15.92:6807 osd.1 since back 2024-03-11T21:51:39.656754+0000 front 2024-03-11T21:51:39.656727+0000 (oldest deadline 2024-03-11T21:52:05.556480+0000) 2024-03-11T21:52:29.587 INFO:journalctl@ceph.osd.7.smithi097.stdout:Mar 11 21:52:29 smithi097 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd-7[56319]: 2024-03-11T21:52:29.239+0000 7ffbb45e2700 -1 osd.7 554 heartbeat_check: no reply from 172.21.15.92:6811 osd.2 since back 2024-03-11T21:51:49.057616+0000 front 2024-03-11T21:51:49.057558+0000 (oldest deadline 2024-03-11T21:52:17.857957+0000) 2024-03-11T21:52:29.587 INFO:journalctl@ceph.osd.7.smithi097.stdout:Mar 11 21:52:29 smithi097 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd-7[56319]: 2024-03-11T21:52:29.239+0000 7ffbb45e2700 -1 osd.7 554 heartbeat_check: no reply from 172.21.15.92:6815 osd.3 since back 2024-03-11T21:51:57.858544+0000 front 2024-03-11T21:51:57.858438+0000 (oldest deadline 2024-03-11T21:52:23.158422+0000) 2024-03-11T21:52:29.587 INFO:journalctl@ceph.osd.7.smithi097.stdout:Mar 11 21:52:29 smithi097 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd-7[56319]: 2024-03-11T21:52:29.239+0000 7ffbb45e2700 -1 osd.7 554 heartbeat_check: no reply from 172.21.15.97:6802 osd.4 since back 2024-03-11T21:52:05.459530+0000 front 2024-03-11T21:52:05.459396+0000 (oldest deadline 2024-03-11T21:52:28.959461+0000) 2024-03-11T21:52:30.587 INFO:journalctl@ceph.osd.7.smithi097.stdout:Mar 11 21:52:30 smithi097 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd-7[56319]: 2024-03-11T21:52:30.263+0000 7ffbb45e2700 -1 osd.7 554 heartbeat_check: no reply from 172.21.15.92:6803 osd.0 since back 2024-03-11T21:51:32.154875+0000 front 2024-03-11T21:51:32.154922+0000 (oldest deadline 2024-03-11T21:51:57.455053+0000) 2024-03-11T21:52:30.587 INFO:journalctl@ceph.osd.7.smithi097.stdout:Mar 11 21:52:30 smithi097 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd-7[56319]: 2024-03-11T21:52:30.263+0000 7ffbb45e2700 -1 osd.7 554 heartbeat_check: no reply from 172.21.15.92:6807 osd.1 since back 2024-03-11T21:51:39.656754+0000 front 2024-03-11T21:51:39.656727+0000 (oldest deadline 2024-03-11T21:52:05.556480+0000) 2024-03-11T21:52:30.587 INFO:journalctl@ceph.osd.7.smithi097.stdout:Mar 11 21:52:30 smithi097 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd-7[56319]: 2024-03-11T21:52:30.263+0000 7ffbb45e2700 -1 osd.7 554 heartbeat_check: no reply from 172.21.15.92:6811 osd.2 since back 2024-03-11T21:51:49.057616+0000 front 2024-03-11T21:51:49.057558+0000 (oldest deadline 2024-03-11T21:52:17.857957+0000) 2024-03-11T21:52:30.587 INFO:journalctl@ceph.osd.7.smithi097.stdout:Mar 11 21:52:30 smithi097 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd-7[56319]: 2024-03-11T21:52:30.263+0000 7ffbb45e2700 -1 osd.7 554 heartbeat_check: no reply from 172.21.15.92:6815 osd.3 since back 2024-03-11T21:51:57.858544+0000 front 2024-03-11T21:51:57.858438+0000 (oldest deadline 2024-03-11T21:52:23.158422+0000) 2024-03-11T21:52:30.587 INFO:journalctl@ceph.osd.7.smithi097.stdout:Mar 11 21:52:30 smithi097 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd-7[56319]: 2024-03-11T21:52:30.263+0000 7ffbb45e2700 -1 osd.7 554 heartbeat_check: no reply from 172.21.15.97:6802 osd.4 since back 2024-03-11T21:52:05.459530+0000 front 2024-03-11T21:52:05.459396+0000 (oldest deadline 2024-03-11T21:52:28.959461+0000) 2024-03-11T21:52:31.586 INFO:journalctl@ceph.osd.7.smithi097.stdout:Mar 11 21:52:31 smithi097 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd-7[56319]: 2024-03-11T21:52:31.262+0000 7ffbb45e2700 -1 osd.7 554 heartbeat_check: no reply from 172.21.15.92:6803 osd.0 since back 2024-03-11T21:51:32.154875+0000 front 2024-03-11T21:51:32.154922+0000 (oldest deadline 2024-03-11T21:51:57.455053+0000) 2024-03-11T21:52:31.587 INFO:journalctl@ceph.osd.7.smithi097.stdout:Mar 11 21:52:31 smithi097 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd-7[56319]: 2024-03-11T21:52:31.262+0000 7ffbb45e2700 -1 osd.7 554 heartbeat_check: no reply from 172.21.15.92:6807 osd.1 since back 2024-03-11T21:51:39.656754+0000 front 2024-03-11T21:51:39.656727+0000 (oldest deadline 2024-03-11T21:52:05.556480+0000) 2024-03-11T21:52:31.587 INFO:journalctl@ceph.osd.7.smithi097.stdout:Mar 11 21:52:31 smithi097 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd-7[56319]: 2024-03-11T21:52:31.262+0000 7ffbb45e2700 -1 osd.7 554 heartbeat_check: no reply from 172.21.15.92:6811 osd.2 since back 2024-03-11T21:51:49.057616+0000 front 2024-03-11T21:51:49.057558+0000 (oldest deadline 2024-03-11T21:52:17.857957+0000) 2024-03-11T21:52:31.587 INFO:journalctl@ceph.osd.7.smithi097.stdout:Mar 11 21:52:31 smithi097 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd-7[56319]: 2024-03-11T21:52:31.262+0000 7ffbb45e2700 -1 osd.7 554 heartbeat_check: no reply from 172.21.15.92:6815 osd.3 since back 2024-03-11T21:51:57.858544+0000 front 2024-03-11T21:51:57.858438+0000 (oldest deadline 2024-03-11T21:52:23.158422+0000) 2024-03-11T21:52:31.587 INFO:journalctl@ceph.osd.7.smithi097.stdout:Mar 11 21:52:31 smithi097 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd-7[56319]: 2024-03-11T21:52:31.262+0000 7ffbb45e2700 -1 osd.7 554 heartbeat_check: no reply from 172.21.15.97:6802 osd.4 since back 2024-03-11T21:52:05.459530+0000 front 2024-03-11T21:52:05.459396+0000 (oldest deadline 2024-03-11T21:52:28.959461+0000) 2024-03-11T21:52:32.586 INFO:journalctl@ceph.osd.7.smithi097.stdout:Mar 11 21:52:32 smithi097 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd-7[56319]: 2024-03-11T21:52:32.297+0000 7ffbb45e2700 -1 osd.7 554 heartbeat_check: no reply from 172.21.15.92:6803 osd.0 since back 2024-03-11T21:51:32.154875+0000 front 2024-03-11T21:51:32.154922+0000 (oldest deadline 2024-03-11T21:51:57.455053+0000) 2024-03-11T21:52:32.587 INFO:journalctl@ceph.osd.7.smithi097.stdout:Mar 11 21:52:32 smithi097 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd-7[56319]: 2024-03-11T21:52:32.297+0000 7ffbb45e2700 -1 osd.7 554 heartbeat_check: no reply from 172.21.15.92:6807 osd.1 since back 2024-03-11T21:51:39.656754+0000 front 2024-03-11T21:51:39.656727+0000 (oldest deadline 2024-03-11T21:52:05.556480+0000) 2024-03-11T21:52:32.587 INFO:journalctl@ceph.osd.7.smithi097.stdout:Mar 11 21:52:32 smithi097 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd-7[56319]: 2024-03-11T21:52:32.297+0000 7ffbb45e2700 -1 osd.7 554 heartbeat_check: no reply from 172.21.15.92:6811 osd.2 since back 2024-03-11T21:51:49.057616+0000 front 2024-03-11T21:51:49.057558+0000 (oldest deadline 2024-03-11T21:52:17.857957+0000) 2024-03-11T21:52:32.587 INFO:journalctl@ceph.osd.7.smithi097.stdout:Mar 11 21:52:32 smithi097 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd-7[56319]: 2024-03-11T21:52:32.297+0000 7ffbb45e2700 -1 osd.7 554 heartbeat_check: no reply from 172.21.15.92:6815 osd.3 since back 2024-03-11T21:51:57.858544+0000 front 2024-03-11T21:51:57.858438+0000 (oldest deadline 2024-03-11T21:52:23.158422+0000) 2024-03-11T21:52:32.587 INFO:journalctl@ceph.osd.7.smithi097.stdout:Mar 11 21:52:32 smithi097 ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd-7[56319]: 2024-03-11T21:52:32.297+0000 7ffbb45e2700 -1 osd.7 554 heartbeat_check: no reply from 172.21.15.97:6802 osd.4 since back 2024-03-11T21:52:05.459530+0000 front 2024-03-11T21:52:05.459396+0000 (oldest deadline 2024-03-11T21:52:28.959461+0000) 2024-03-11T21:52:33.863 INFO:journalctl@ceph.osd.7.smithi097.stdout:Mar 11 21:52:33 smithi097 podman[161273]: 2024-03-11 21:52:33.614558951 +0000 UTC m=+5.695185206 container died 8a50dafbf1558ff9c51dc48f1145aefd22b31abdfaf4c161f2165898edcaf284 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:8fe9635f3beb0247496d7e321f3068bd10c333c433ef4824d7e1c1a7d3d78379, name=ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd-7, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, ceph=True, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, GIT_CLEAN=True, io.buildah.version=1.33.5, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=main-2b0101e, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, org.label-schema.build-date=20240307) 2024-03-11T21:52:34.280 INFO:journalctl@ceph.osd.7.smithi097.stdout:Mar 11 21:52:33 smithi097 podman[161273]: 2024-03-11 21:52:33.970314242 +0000 UTC m=+6.050940497 container cleanup 8a50dafbf1558ff9c51dc48f1145aefd22b31abdfaf4c161f2165898edcaf284 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:8fe9635f3beb0247496d7e321f3068bd10c333c433ef4824d7e1c1a7d3d78379, name=ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd-7, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=main-2b0101e, io.buildah.version=1.33.5, org.label-schema.build-date=20240307, GIT_BRANCH=HEAD, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.vendor=CentOS, ceph=True, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-03-11T21:52:34.280 INFO:journalctl@ceph.osd.7.smithi097.stdout:Mar 11 21:52:33 smithi097 bash[161273]: ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd-7 2024-03-11T21:52:34.573 INFO:journalctl@ceph.osd.7.smithi097.stdout:Mar 11 21:52:34 smithi097 podman[161291]: 2024-03-11 21:52:34.288364261 +0000 UTC m=+0.667893176 container remove 8a50dafbf1558ff9c51dc48f1145aefd22b31abdfaf4c161f2165898edcaf284 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:8fe9635f3beb0247496d7e321f3068bd10c333c433ef4824d7e1c1a7d3d78379, name=ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd-7, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240307, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, ceph=True, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, GIT_CLEAN=True, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , RELEASE=main-2b0101e) 2024-03-11T21:52:34.836 INFO:journalctl@ceph.osd.7.smithi097.stdout:Mar 11 21:52:34 smithi097 podman[161386]: 2024-03-11 21:52:34.580858709 +0000 UTC m=+0.111574780 container create 44be7a2f4925bbf9fbb763feef6526b79d2914f484d25c6feadb3ea141e2c644 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:8fe9635f3beb0247496d7e321f3068bd10c333c433ef4824d7e1c1a7d3d78379, name=ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd-7-deactivate, io.buildah.version=1.33.5, RELEASE=main-2b0101e, ceph=True, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, GIT_CLEAN=True, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240307, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux ) 2024-03-11T21:52:34.837 INFO:journalctl@ceph.osd.7.smithi097.stdout:Mar 11 21:52:34 smithi097 podman[161386]: 2024-03-11 21:52:34.486591595 +0000 UTC m=+0.017307671 image pull 4133f8bc41bb201212a28168427a2edb1f6f3795e694bf61031ea9e2f3ff5e87 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:8fe9635f3beb0247496d7e321f3068bd10c333c433ef4824d7e1c1a7d3d78379 2024-03-11T21:52:34.837 INFO:journalctl@ceph.osd.7.smithi097.stdout:Mar 11 21:52:34 smithi097 podman[161386]: 2024-03-11 21:52:34.800021565 +0000 UTC m=+0.330737647 container init 44be7a2f4925bbf9fbb763feef6526b79d2914f484d25c6feadb3ea141e2c644 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:8fe9635f3beb0247496d7e321f3068bd10c333c433ef4824d7e1c1a7d3d78379, name=ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd-7-deactivate, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240307, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, GIT_CLEAN=True, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , RELEASE=main-2b0101e, org.label-schema.license=GPLv2, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, io.buildah.version=1.33.5) 2024-03-11T21:52:34.837 INFO:journalctl@ceph.osd.7.smithi097.stdout:Mar 11 21:52:34 smithi097 podman[161386]: 2024-03-11 21:52:34.803275703 +0000 UTC m=+0.333991774 container start 44be7a2f4925bbf9fbb763feef6526b79d2914f484d25c6feadb3ea141e2c644 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:8fe9635f3beb0247496d7e321f3068bd10c333c433ef4824d7e1c1a7d3d78379, name=ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd-7-deactivate, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, RELEASE=main-2b0101e, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.build-date=20240307, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux ) 2024-03-11T21:52:35.337 INFO:journalctl@ceph.osd.7.smithi097.stdout:Mar 11 21:52:34 smithi097 podman[161386]: 2024-03-11 21:52:34.88319526 +0000 UTC m=+0.413911348 container attach 44be7a2f4925bbf9fbb763feef6526b79d2914f484d25c6feadb3ea141e2c644 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:8fe9635f3beb0247496d7e321f3068bd10c333c433ef4824d7e1c1a7d3d78379, name=ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd-7-deactivate, GIT_CLEAN=True, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, org.label-schema.build-date=20240307, RELEASE=main-2b0101e, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, maintainer=Guillaume Abrioux ) 2024-03-11T21:52:35.337 INFO:journalctl@ceph.osd.7.smithi097.stdout:Mar 11 21:52:35 smithi097 podman[161386]: 2024-03-11 21:52:35.087838046 +0000 UTC m=+0.618554127 container died 44be7a2f4925bbf9fbb763feef6526b79d2914f484d25c6feadb3ea141e2c644 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:8fe9635f3beb0247496d7e321f3068bd10c333c433ef4824d7e1c1a7d3d78379, name=ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd-7-deactivate, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, RELEASE=main-2b0101e, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240307, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30) 2024-03-11T21:52:35.630 INFO:journalctl@ceph.osd.7.smithi097.stdout:Mar 11 21:52:35 smithi097 podman[161412]: 2024-03-11 21:52:35.63631861 +0000 UTC m=+0.543711119 container remove 44be7a2f4925bbf9fbb763feef6526b79d2914f484d25c6feadb3ea141e2c644 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:8fe9635f3beb0247496d7e321f3068bd10c333c433ef4824d7e1c1a7d3d78379, name=ceph-00db0016-dfed-11ee-95c8-87774f69a715-osd-7-deactivate, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, GIT_CLEAN=True, RELEASE=main-2b0101e, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240307, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS) 2024-03-11T21:52:35.656 DEBUG:teuthology.orchestra.run.smithi097:> sudo pkill -f 'journalctl -f -n 0 -u ceph-00db0016-dfed-11ee-95c8-87774f69a715@osd.7.service' 2024-03-11T21:52:35.698 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-11T21:52:35.698 INFO:tasks.cephadm.osd.7:Stopped osd.7 2024-03-11T21:52:35.698 INFO:tasks.cephadm.ceph.rgw.foo.a:Stopping rgw.foo.a... 2024-03-11T21:52:35.698 DEBUG:teuthology.orchestra.run.smithi092:> sudo systemctl stop ceph-00db0016-dfed-11ee-95c8-87774f69a715@rgw.foo.a 2024-03-11T21:52:36.011 INFO:journalctl@ceph.rgw.foo.a.smithi092.stdout:Mar 11 21:52:35 smithi092 systemd[1]: Stopping Ceph rgw.foo.a for 00db0016-dfed-11ee-95c8-87774f69a715... 2024-03-11T21:52:36.268 INFO:journalctl@ceph.rgw.foo.a.smithi092.stdout:Mar 11 21:52:36 smithi092 ceph-00db0016-dfed-11ee-95c8-87774f69a715-rgw-foo-a[64262]: 2024-03-11T21:52:36.018+0000 7fcd658eb700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/radosgw -n client.rgw.foo.a -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-11T21:52:36.268 INFO:journalctl@ceph.rgw.foo.a.smithi092.stdout:Mar 11 21:52:36 smithi092 ceph-00db0016-dfed-11ee-95c8-87774f69a715-rgw-foo-a[64262]: 2024-03-11T21:52:36.018+0000 7fcd74163ac0 -1 shutting down 2024-03-11T21:52:46.306 INFO:journalctl@ceph.rgw.foo.a.smithi092.stdout:Mar 11 21:52:46 smithi092 bash[217721]: time="2024-03-11T21:52:46Z" level=warning msg="StopSignal SIGTERM failed to stop container ceph-00db0016-dfed-11ee-95c8-87774f69a715-rgw-foo-a in 10 seconds, resorting to SIGKILL" 2024-03-11T21:52:46.307 INFO:journalctl@ceph.rgw.foo.a.smithi092.stdout:Mar 11 21:52:46 smithi092 podman[217721]: 2024-03-11 21:52:46.035763116 +0000 UTC m=+10.124661591 container died 81ad59000ca20f444683eed7ed2e88d14839dfff4afe044b028e922cdbe68b1f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:8fe9635f3beb0247496d7e321f3068bd10c333c433ef4824d7e1c1a7d3d78379, name=ceph-00db0016-dfed-11ee-95c8-87774f69a715-rgw-foo-a, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, GIT_CLEAN=True, io.buildah.version=1.33.5, org.label-schema.build-date=20240307, ceph=True, GIT_BRANCH=HEAD, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=main-2b0101e) 2024-03-11T21:52:46.620 INFO:journalctl@ceph.rgw.foo.a.smithi092.stdout:Mar 11 21:52:46 smithi092 podman[217721]: 2024-03-11 21:52:46.492225989 +0000 UTC m=+10.581124492 container cleanup 81ad59000ca20f444683eed7ed2e88d14839dfff4afe044b028e922cdbe68b1f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:8fe9635f3beb0247496d7e321f3068bd10c333c433ef4824d7e1c1a7d3d78379, name=ceph-00db0016-dfed-11ee-95c8-87774f69a715-rgw-foo-a, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=main-2b0101e, org.label-schema.build-date=20240307, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, GIT_CLEAN=True, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=) 2024-03-11T21:52:46.621 INFO:journalctl@ceph.rgw.foo.a.smithi092.stdout:Mar 11 21:52:46 smithi092 bash[217721]: ceph-00db0016-dfed-11ee-95c8-87774f69a715-rgw-foo-a 2024-03-11T21:52:47.040 INFO:journalctl@ceph.rgw.foo.a.smithi092.stdout:Mar 11 21:52:46 smithi092 podman[217748]: 2024-03-11 21:52:46.785229569 +0000 UTC m=+0.731746658 container remove 81ad59000ca20f444683eed7ed2e88d14839dfff4afe044b028e922cdbe68b1f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:8fe9635f3beb0247496d7e321f3068bd10c333c433ef4824d7e1c1a7d3d78379, name=ceph-00db0016-dfed-11ee-95c8-87774f69a715-rgw-foo-a, org.label-schema.license=GPLv2, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.build-date=20240307, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, GIT_CLEAN=True, RELEASE=main-2b0101e, ceph=True, io.buildah.version=1.33.5) 2024-03-11T21:52:47.040 INFO:journalctl@ceph.rgw.foo.a.smithi092.stdout:Mar 11 21:52:46 smithi092 systemd[1]: ceph-00db0016-dfed-11ee-95c8-87774f69a715@rgw.foo.a.service: Main process exited, code=exited, status=137/n/a 2024-03-11T21:52:47.067 DEBUG:teuthology.orchestra.run.smithi092:> sudo pkill -f 'journalctl -f -n 0 -u ceph-00db0016-dfed-11ee-95c8-87774f69a715@rgw.foo.a.service' 2024-03-11T21:52:47.104 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-11T21:52:47.105 INFO:tasks.cephadm.ceph.rgw.foo.a:Stopped rgw.foo.a 2024-03-11T21:52:47.105 INFO:tasks.cephadm.prometheus.a:Stopping prometheus.a... 2024-03-11T21:52:47.105 DEBUG:teuthology.orchestra.run.smithi097:> sudo systemctl stop ceph-00db0016-dfed-11ee-95c8-87774f69a715@prometheus.a 2024-03-11T21:52:47.413 INFO:journalctl@ceph.prometheus.a.smithi097.stdout:Mar 11 21:52:47 smithi097 systemd[1]: Stopping Ceph prometheus.a for 00db0016-dfed-11ee-95c8-87774f69a715... 2024-03-11T21:52:47.709 INFO:journalctl@ceph.prometheus.a.smithi097.stdout:Mar 11 21:52:47 smithi097 ceph-00db0016-dfed-11ee-95c8-87774f69a715-prometheus-a[73147]: ts=2024-03-11T21:52:47.421Z caller=main.go:840 level=warn msg="Received SIGTERM, exiting gracefully..." 2024-03-11T21:52:47.710 INFO:journalctl@ceph.prometheus.a.smithi097.stdout:Mar 11 21:52:47 smithi097 ceph-00db0016-dfed-11ee-95c8-87774f69a715-prometheus-a[73147]: ts=2024-03-11T21:52:47.421Z caller=main.go:864 level=info msg="Stopping scrape discovery manager..." 2024-03-11T21:52:47.710 INFO:journalctl@ceph.prometheus.a.smithi097.stdout:Mar 11 21:52:47 smithi097 ceph-00db0016-dfed-11ee-95c8-87774f69a715-prometheus-a[73147]: ts=2024-03-11T21:52:47.421Z caller=main.go:878 level=info msg="Stopping notify discovery manager..." 2024-03-11T21:52:47.710 INFO:journalctl@ceph.prometheus.a.smithi097.stdout:Mar 11 21:52:47 smithi097 ceph-00db0016-dfed-11ee-95c8-87774f69a715-prometheus-a[73147]: ts=2024-03-11T21:52:47.421Z caller=manager.go:988 level=info component="rule manager" msg="Stopping rule manager..." 2024-03-11T21:52:47.710 INFO:journalctl@ceph.prometheus.a.smithi097.stdout:Mar 11 21:52:47 smithi097 ceph-00db0016-dfed-11ee-95c8-87774f69a715-prometheus-a[73147]: ts=2024-03-11T21:52:47.421Z caller=main.go:860 level=info msg="Scrape discovery manager stopped" 2024-03-11T21:52:47.710 INFO:journalctl@ceph.prometheus.a.smithi097.stdout:Mar 11 21:52:47 smithi097 ceph-00db0016-dfed-11ee-95c8-87774f69a715-prometheus-a[73147]: ts=2024-03-11T21:52:47.421Z caller=main.go:874 level=info msg="Notify discovery manager stopped" 2024-03-11T21:52:47.710 INFO:journalctl@ceph.prometheus.a.smithi097.stdout:Mar 11 21:52:47 smithi097 ceph-00db0016-dfed-11ee-95c8-87774f69a715-prometheus-a[73147]: ts=2024-03-11T21:52:47.421Z caller=manager.go:998 level=info component="rule manager" msg="Rule manager stopped" 2024-03-11T21:52:47.710 INFO:journalctl@ceph.prometheus.a.smithi097.stdout:Mar 11 21:52:47 smithi097 ceph-00db0016-dfed-11ee-95c8-87774f69a715-prometheus-a[73147]: ts=2024-03-11T21:52:47.421Z caller=main.go:915 level=info msg="Stopping scrape manager..." 2024-03-11T21:52:47.710 INFO:journalctl@ceph.prometheus.a.smithi097.stdout:Mar 11 21:52:47 smithi097 ceph-00db0016-dfed-11ee-95c8-87774f69a715-prometheus-a[73147]: ts=2024-03-11T21:52:47.431Z caller=main.go:907 level=info msg="Scrape manager stopped" 2024-03-11T21:52:47.710 INFO:journalctl@ceph.prometheus.a.smithi097.stdout:Mar 11 21:52:47 smithi097 ceph-00db0016-dfed-11ee-95c8-87774f69a715-prometheus-a[73147]: ts=2024-03-11T21:52:47.431Z caller=notifier.go:606 level=info component=notifier msg="Stopping notification manager..." 2024-03-11T21:52:47.710 INFO:journalctl@ceph.prometheus.a.smithi097.stdout:Mar 11 21:52:47 smithi097 ceph-00db0016-dfed-11ee-95c8-87774f69a715-prometheus-a[73147]: ts=2024-03-11T21:52:47.431Z caller=main.go:1135 level=info msg="Notifier manager stopped" 2024-03-11T21:52:47.710 INFO:journalctl@ceph.prometheus.a.smithi097.stdout:Mar 11 21:52:47 smithi097 ceph-00db0016-dfed-11ee-95c8-87774f69a715-prometheus-a[73147]: ts=2024-03-11T21:52:47.431Z caller=main.go:1147 level=info msg="See you next time!" 2024-03-11T21:52:47.710 INFO:journalctl@ceph.prometheus.a.smithi097.stdout:Mar 11 21:52:47 smithi097 podman[161492]: 2024-03-11 21:52:47.442385124 +0000 UTC m=+0.128770670 container died cc9a759a8c93574c1f7b2325c32c3dabb61c319a00fdfd60c1a7307137141f14 (image=quay.io/prometheus/prometheus:v2.43.0, name=ceph-00db0016-dfed-11ee-95c8-87774f69a715-prometheus-a, maintainer=The Prometheus Authors ) 2024-03-11T21:52:48.086 INFO:journalctl@ceph.prometheus.a.smithi097.stdout:Mar 11 21:52:47 smithi097 podman[161492]: 2024-03-11 21:52:47.830092 +0000 UTC m=+0.516477546 container cleanup cc9a759a8c93574c1f7b2325c32c3dabb61c319a00fdfd60c1a7307137141f14 (image=quay.io/prometheus/prometheus:v2.43.0, name=ceph-00db0016-dfed-11ee-95c8-87774f69a715-prometheus-a, maintainer=The Prometheus Authors ) 2024-03-11T21:52:48.086 INFO:journalctl@ceph.prometheus.a.smithi097.stdout:Mar 11 21:52:47 smithi097 bash[161492]: ceph-00db0016-dfed-11ee-95c8-87774f69a715-prometheus-a 2024-03-11T21:52:48.213 DEBUG:teuthology.orchestra.run.smithi097:> sudo pkill -f 'journalctl -f -n 0 -u ceph-00db0016-dfed-11ee-95c8-87774f69a715@prometheus.a.service' 2024-03-11T21:52:48.250 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-11T21:52:48.250 INFO:tasks.cephadm.prometheus.a:Stopped prometheus.a 2024-03-11T21:52:48.250 DEBUG:teuthology.orchestra.run.smithi092:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid 00db0016-dfed-11ee-95c8-87774f69a715 --force --keep-logs 2024-03-11T21:52:48.439 INFO:teuthology.orchestra.run.smithi092.stdout:Deleting cluster with fsid: 00db0016-dfed-11ee-95c8-87774f69a715 2024-03-11T21:52:50.871 INFO:journalctl@ceph.alertmanager.a.smithi092.stdout:Mar 11 21:52:50 smithi092 systemd[1]: Stopping Ceph alertmanager.a for 00db0016-dfed-11ee-95c8-87774f69a715... 2024-03-11T21:52:51.303 INFO:journalctl@ceph.alertmanager.a.smithi092.stdout:Mar 11 21:52:50 smithi092 ceph-00db0016-dfed-11ee-95c8-87774f69a715-alertmanager-a[85324]: ts=2024-03-11T21:52:50.980Z caller=main.go:583 level=info msg="Received SIGTERM, exiting gracefully..." 2024-03-11T21:52:51.303 INFO:journalctl@ceph.alertmanager.a.smithi092.stdout:Mar 11 21:52:51 smithi092 podman[218229]: 2024-03-11 21:52:51.021368018 +0000 UTC m=+0.164350036 container died bddae98c3565f589db50ce3c14882c9a14da74bb75ef29d00e1b4c80145f9275 (image=quay.io/prometheus/alertmanager:v0.25.0, name=ceph-00db0016-dfed-11ee-95c8-87774f69a715-alertmanager-a, maintainer=The Prometheus Authors ) 2024-03-11T21:52:51.620 INFO:journalctl@ceph.alertmanager.a.smithi092.stdout:Mar 11 21:52:51 smithi092 podman[218229]: 2024-03-11 21:52:51.397054615 +0000 UTC m=+0.540036638 container cleanup bddae98c3565f589db50ce3c14882c9a14da74bb75ef29d00e1b4c80145f9275 (image=quay.io/prometheus/alertmanager:v0.25.0, name=ceph-00db0016-dfed-11ee-95c8-87774f69a715-alertmanager-a, maintainer=The Prometheus Authors ) 2024-03-11T21:52:51.620 INFO:journalctl@ceph.alertmanager.a.smithi092.stdout:Mar 11 21:52:51 smithi092 bash[218229]: ceph-00db0016-dfed-11ee-95c8-87774f69a715-alertmanager-a 2024-03-11T21:52:52.045 INFO:journalctl@ceph.alertmanager.a.smithi092.stdout:Mar 11 21:52:51 smithi092 podman[218243]: 2024-03-11 21:52:51.662254127 +0000 UTC m=+0.631146999 container remove bddae98c3565f589db50ce3c14882c9a14da74bb75ef29d00e1b4c80145f9275 (image=quay.io/prometheus/alertmanager:v0.25.0, name=ceph-00db0016-dfed-11ee-95c8-87774f69a715-alertmanager-a, maintainer=The Prometheus Authors ) 2024-03-11T21:52:52.045 INFO:journalctl@ceph.alertmanager.a.smithi092.stdout:Mar 11 21:52:51 smithi092 podman[218243]: 2024-03-11 21:52:51.749699897 +0000 UTC m=+0.718592777 volume remove 631a33a17d119ca9f206939521b4e19f0a6f733b1a9f317862f3bdc9a1e37143 2024-03-11T21:52:52.045 INFO:journalctl@ceph.alertmanager.a.smithi092.stdout:Mar 11 21:52:51 smithi092 systemd[1]: ceph-00db0016-dfed-11ee-95c8-87774f69a715@alertmanager.a.service: Deactivated successfully. 2024-03-11T21:52:52.045 INFO:journalctl@ceph.alertmanager.a.smithi092.stdout:Mar 11 21:52:51 smithi092 systemd[1]: Stopped Ceph alertmanager.a for 00db0016-dfed-11ee-95c8-87774f69a715. 2024-03-11T21:52:52.045 INFO:journalctl@ceph.alertmanager.a.smithi092.stdout:Mar 11 21:52:51 smithi092 systemd[1]: ceph-00db0016-dfed-11ee-95c8-87774f69a715@alertmanager.a.service: Consumed 1.685s CPU time. 2024-03-11T21:52:52.370 INFO:journalctl@ceph.node-exporter.a.smithi092.stdout:Mar 11 21:52:52 smithi092 systemd[1]: Stopping Ceph node-exporter.a for 00db0016-dfed-11ee-95c8-87774f69a715... 2024-03-11T21:52:52.751 INFO:journalctl@ceph.node-exporter.a.smithi092.stdout:Mar 11 21:52:52 smithi092 podman[218355]: 2024-03-11 21:52:52.444976771 +0000 UTC m=+0.253293624 container died b3a2398e1da535b9ba3b93fe29546cc745fbf1b243495aa59563654260e010ed (image=quay.io/prometheus/node-exporter:v1.5.0, name=ceph-00db0016-dfed-11ee-95c8-87774f69a715-node-exporter-a, maintainer=The Prometheus Authors ) 2024-03-11T21:52:52.751 INFO:journalctl@ceph.node-exporter.a.smithi092.stdout:Mar 11 21:52:52 smithi092 podman[218355]: 2024-03-11 21:52:52.75939603 +0000 UTC m=+0.567712893 container cleanup b3a2398e1da535b9ba3b93fe29546cc745fbf1b243495aa59563654260e010ed (image=quay.io/prometheus/node-exporter:v1.5.0, name=ceph-00db0016-dfed-11ee-95c8-87774f69a715-node-exporter-a, maintainer=The Prometheus Authors ) 2024-03-11T21:52:53.120 INFO:journalctl@ceph.node-exporter.a.smithi092.stdout:Mar 11 21:52:52 smithi092 bash[218355]: ceph-00db0016-dfed-11ee-95c8-87774f69a715-node-exporter-a 2024-03-11T21:52:53.120 INFO:journalctl@ceph.node-exporter.a.smithi092.stdout:Mar 11 21:52:53 smithi092 podman[218369]: 2024-03-11 21:52:53.002220144 +0000 UTC m=+0.558123022 container remove b3a2398e1da535b9ba3b93fe29546cc745fbf1b243495aa59563654260e010ed (image=quay.io/prometheus/node-exporter:v1.5.0, name=ceph-00db0016-dfed-11ee-95c8-87774f69a715-node-exporter-a, maintainer=The Prometheus Authors ) 2024-03-11T21:52:53.120 INFO:journalctl@ceph.node-exporter.a.smithi092.stdout:Mar 11 21:52:53 smithi092 systemd[1]: ceph-00db0016-dfed-11ee-95c8-87774f69a715@node-exporter.a.service: Main process exited, code=exited, status=143/n/a 2024-03-11T21:52:53.465 INFO:journalctl@ceph.node-exporter.a.smithi092.stdout:Mar 11 21:52:53 smithi092 systemd[1]: ceph-00db0016-dfed-11ee-95c8-87774f69a715@node-exporter.a.service: Failed with result 'exit-code'. 2024-03-11T21:52:53.465 INFO:journalctl@ceph.node-exporter.a.smithi092.stdout:Mar 11 21:52:53 smithi092 systemd[1]: Stopped Ceph node-exporter.a for 00db0016-dfed-11ee-95c8-87774f69a715. 2024-03-11T21:52:53.465 INFO:journalctl@ceph.node-exporter.a.smithi092.stdout:Mar 11 21:52:53 smithi092 systemd[1]: ceph-00db0016-dfed-11ee-95c8-87774f69a715@node-exporter.a.service: Consumed 4.973s CPU time. 2024-03-11T21:52:53.885 DEBUG:teuthology.orchestra.run.smithi097:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid 00db0016-dfed-11ee-95c8-87774f69a715 --force --keep-logs 2024-03-11T21:52:54.077 INFO:teuthology.orchestra.run.smithi097.stdout:Deleting cluster with fsid: 00db0016-dfed-11ee-95c8-87774f69a715 2024-03-11T21:52:56.086 INFO:journalctl@ceph.iscsi.iscsi.a.smithi097.stdout:Mar 11 21:52:55 smithi097 systemd[1]: Stopping Ceph iscsi.iscsi.a for 00db0016-dfed-11ee-95c8-87774f69a715... 2024-03-11T21:52:56.586 INFO:journalctl@ceph.iscsi.iscsi.a.smithi097.stdout:Mar 11 21:52:56 smithi097 ceph-00db0016-dfed-11ee-95c8-87774f69a715-iscsi-iscsi-a[60943]: debug Shutdown received 2024-03-11T21:53:06.511 INFO:journalctl@ceph.iscsi.iscsi.a.smithi097.stdout:Mar 11 21:53:06 smithi097 bash[161902]: time="2024-03-11T21:53:06Z" level=warning msg="StopSignal SIGTERM failed to stop container ceph-00db0016-dfed-11ee-95c8-87774f69a715-iscsi-iscsi-a in 10 seconds, resorting to SIGKILL" 2024-03-11T21:53:06.512 INFO:journalctl@ceph.iscsi.iscsi.a.smithi097.stdout:Mar 11 21:53:06 smithi097 podman[161902]: 2024-03-11 21:53:06.211623942 +0000 UTC m=+10.135907883 container died 8bd740122036d3a69d3780a0f0a02517a39b42ae3bde406eefdaf62d1aec85be (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:8fe9635f3beb0247496d7e321f3068bd10c333c433ef4824d7e1c1a7d3d78379, name=ceph-00db0016-dfed-11ee-95c8-87774f69a715-iscsi-iscsi-a, org.label-schema.schema-version=1.0, ceph=True, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240307, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, GIT_CLEAN=True, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , RELEASE=main-2b0101e) 2024-03-11T21:53:06.836 INFO:journalctl@ceph.iscsi.iscsi.a.smithi097.stdout:Mar 11 21:53:06 smithi097 podman[161902]: 2024-03-11 21:53:06.609005819 +0000 UTC m=+10.533289760 container cleanup 8bd740122036d3a69d3780a0f0a02517a39b42ae3bde406eefdaf62d1aec85be (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:8fe9635f3beb0247496d7e321f3068bd10c333c433ef4824d7e1c1a7d3d78379, name=ceph-00db0016-dfed-11ee-95c8-87774f69a715-iscsi-iscsi-a, org.label-schema.schema-version=1.0, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=main-2b0101e, io.buildah.version=1.33.5, ceph=True, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.build-date=20240307, GIT_BRANCH=HEAD) 2024-03-11T21:53:06.837 INFO:journalctl@ceph.iscsi.iscsi.a.smithi097.stdout:Mar 11 21:53:06 smithi097 bash[161902]: ceph-00db0016-dfed-11ee-95c8-87774f69a715-iscsi-iscsi-a 2024-03-11T21:53:07.248 INFO:journalctl@ceph.iscsi.iscsi.a.smithi097.stdout:Mar 11 21:53:06 smithi097 podman[161929]: 2024-03-11 21:53:06.868490736 +0000 UTC m=+0.653657894 container remove 8bd740122036d3a69d3780a0f0a02517a39b42ae3bde406eefdaf62d1aec85be (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:8fe9635f3beb0247496d7e321f3068bd10c333c433ef4824d7e1c1a7d3d78379, name=ceph-00db0016-dfed-11ee-95c8-87774f69a715-iscsi-iscsi-a, ceph=True, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_CLEAN=True, GIT_BRANCH=HEAD, org.label-schema.build-date=20240307, RELEASE=main-2b0101e, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-03-11T21:53:07.248 INFO:journalctl@ceph.iscsi.iscsi.a.smithi097.stdout:Mar 11 21:53:06 smithi097 systemd[1]: ceph-00db0016-dfed-11ee-95c8-87774f69a715@iscsi.iscsi.a.service: Main process exited, code=exited, status=137/n/a 2024-03-11T21:53:07.248 INFO:journalctl@ceph.iscsi.iscsi.a.smithi097.stdout:Mar 11 21:53:07 smithi097 systemd[1]: ceph-00db0016-dfed-11ee-95c8-87774f69a715@iscsi.iscsi.a.service: Failed with result 'exit-code'. 2024-03-11T21:53:07.248 INFO:journalctl@ceph.iscsi.iscsi.a.smithi097.stdout:Mar 11 21:53:07 smithi097 systemd[1]: Stopped Ceph iscsi.iscsi.a for 00db0016-dfed-11ee-95c8-87774f69a715. 2024-03-11T21:53:07.248 INFO:journalctl@ceph.iscsi.iscsi.a.smithi097.stdout:Mar 11 21:53:07 smithi097 systemd[1]: ceph-00db0016-dfed-11ee-95c8-87774f69a715@iscsi.iscsi.a.service: Consumed 3.552s CPU time. 2024-03-11T21:53:08.786 INFO:journalctl@ceph.grafana.a.smithi097.stdout:Mar 11 21:53:08 smithi097 systemd[1]: Stopping Ceph grafana.a for 00db0016-dfed-11ee-95c8-87774f69a715... 2024-03-11T21:53:09.076 INFO:journalctl@ceph.grafana.a.smithi097.stdout:Mar 11 21:53:08 smithi097 podman[162189]: 2024-03-11 21:53:08.794295603 +0000 UTC m=+0.108055324 container died 2bc78bd29e96e10effd38c8e2bda8c99c1d139e375736df4ef16defdbd141aeb (image=quay.io/ceph/ceph-grafana:9.4.12, name=ceph-00db0016-dfed-11ee-95c8-87774f69a715-grafana-a, io.k8s.display-name=Red Hat Universal Base Image 8, com.redhat.component=ubi8-container, io.buildah.version=1.32.0, maintainer=Paul Cuzner , io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., distribution-scope=public, io.openshift.expose-services=, architecture=x86_64, version=8.5, summary=Grafana Container configured for Ceph mgr/dashboard integration, com.redhat.license_terms=https://www.redhat.com/en/about/red-hat-end-user-license-agreements#UBI, io.openshift.tags=base rhel8, name=ubi8, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.5-239.1651231664, vcs-ref=3aadd00326f3dd6cfe65ee31017ab98915fddb56, vendor=Red Hat, Inc., com.redhat.build-host=cpt-1007.osbs.prod.upshift.rdu2.redhat.com, vcs-type=git, build-date=2022-04-29T11:48:11.055897, description=Ceph Grafana Container, release=239.1651231664) 2024-03-11T21:53:09.336 INFO:journalctl@ceph.grafana.a.smithi097.stdout:Mar 11 21:53:09 smithi097 podman[162189]: 2024-03-11 21:53:09.192069163 +0000 UTC m=+0.505828932 container cleanup 2bc78bd29e96e10effd38c8e2bda8c99c1d139e375736df4ef16defdbd141aeb (image=quay.io/ceph/ceph-grafana:9.4.12, name=ceph-00db0016-dfed-11ee-95c8-87774f69a715-grafana-a, io.k8s.display-name=Red Hat Universal Base Image 8, name=ubi8, release=239.1651231664, vcs-ref=3aadd00326f3dd6cfe65ee31017ab98915fddb56, io.openshift.tags=base rhel8, architecture=x86_64, maintainer=Paul Cuzner , com.redhat.license_terms=https://www.redhat.com/en/about/red-hat-end-user-license-agreements#UBI, vendor=Red Hat, Inc., io.buildah.version=1.32.0, summary=Grafana Container configured for Ceph mgr/dashboard integration, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., com.redhat.build-host=cpt-1007.osbs.prod.upshift.rdu2.redhat.com, build-date=2022-04-29T11:48:11.055897, com.redhat.component=ubi8-container, io.openshift.expose-services=, vcs-type=git, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.5-239.1651231664, description=Ceph Grafana Container, distribution-scope=public, version=8.5) 2024-03-11T21:53:09.337 INFO:journalctl@ceph.grafana.a.smithi097.stdout:Mar 11 21:53:09 smithi097 bash[162189]: ceph-00db0016-dfed-11ee-95c8-87774f69a715-grafana-a 2024-03-11T21:53:09.837 INFO:journalctl@ceph.grafana.a.smithi097.stdout:Mar 11 21:53:09 smithi097 podman[162203]: 2024-03-11 21:53:09.451516896 +0000 UTC m=+0.659677176 container remove 2bc78bd29e96e10effd38c8e2bda8c99c1d139e375736df4ef16defdbd141aeb (image=quay.io/ceph/ceph-grafana:9.4.12, name=ceph-00db0016-dfed-11ee-95c8-87774f69a715-grafana-a, vcs-type=git, com.redhat.license_terms=https://www.redhat.com/en/about/red-hat-end-user-license-agreements#UBI, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.5-239.1651231664, vcs-ref=3aadd00326f3dd6cfe65ee31017ab98915fddb56, io.k8s.display-name=Red Hat Universal Base Image 8, build-date=2022-04-29T11:48:11.055897, description=Ceph Grafana Container, name=ubi8, io.openshift.tags=base rhel8, distribution-scope=public, com.redhat.component=ubi8-container, io.openshift.expose-services=, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., release=239.1651231664, io.buildah.version=1.32.0, com.redhat.build-host=cpt-1007.osbs.prod.upshift.rdu2.redhat.com, summary=Grafana Container configured for Ceph mgr/dashboard integration, vendor=Red Hat, Inc., architecture=x86_64, version=8.5, maintainer=Paul Cuzner ) 2024-03-11T21:53:09.837 INFO:journalctl@ceph.grafana.a.smithi097.stdout:Mar 11 21:53:09 smithi097 systemd[1]: ceph-00db0016-dfed-11ee-95c8-87774f69a715@grafana.a.service: Main process exited, code=exited, status=143/n/a 2024-03-11T21:53:09.837 INFO:journalctl@ceph.grafana.a.smithi097.stdout:Mar 11 21:53:09 smithi097 systemd[1]: ceph-00db0016-dfed-11ee-95c8-87774f69a715@grafana.a.service: Failed with result 'exit-code'. 2024-03-11T21:53:09.837 INFO:journalctl@ceph.grafana.a.smithi097.stdout:Mar 11 21:53:09 smithi097 systemd[1]: Stopped Ceph grafana.a for 00db0016-dfed-11ee-95c8-87774f69a715. 2024-03-11T21:53:09.837 INFO:journalctl@ceph.grafana.a.smithi097.stdout:Mar 11 21:53:09 smithi097 systemd[1]: ceph-00db0016-dfed-11ee-95c8-87774f69a715@grafana.a.service: Consumed 13.391s CPU time. 2024-03-11T21:53:10.336 INFO:journalctl@ceph.node-exporter.b.smithi097.stdout:Mar 11 21:53:09 smithi097 systemd[1]: Stopping Ceph node-exporter.b for 00db0016-dfed-11ee-95c8-87774f69a715... 2024-03-11T21:53:10.337 INFO:journalctl@ceph.node-exporter.b.smithi097.stdout:Mar 11 21:53:10 smithi097 podman[162314]: 2024-03-11 21:53:10.098267367 +0000 UTC m=+0.104038249 container died 02c16113a38a9503d3369f699573e38bf8895d2509a21b25536a2ac3ac5623a3 (image=quay.io/prometheus/node-exporter:v1.5.0, name=ceph-00db0016-dfed-11ee-95c8-87774f69a715-node-exporter-b, maintainer=The Prometheus Authors ) 2024-03-11T21:53:10.614 INFO:journalctl@ceph.node-exporter.b.smithi097.stdout:Mar 11 21:53:10 smithi097 podman[162314]: 2024-03-11 21:53:10.429235454 +0000 UTC m=+0.435006328 container cleanup 02c16113a38a9503d3369f699573e38bf8895d2509a21b25536a2ac3ac5623a3 (image=quay.io/prometheus/node-exporter:v1.5.0, name=ceph-00db0016-dfed-11ee-95c8-87774f69a715-node-exporter-b, maintainer=The Prometheus Authors ) 2024-03-11T21:53:10.614 INFO:journalctl@ceph.node-exporter.b.smithi097.stdout:Mar 11 21:53:10 smithi097 bash[162314]: ceph-00db0016-dfed-11ee-95c8-87774f69a715-node-exporter-b 2024-03-11T21:53:10.864 INFO:journalctl@ceph.node-exporter.b.smithi097.stdout:Mar 11 21:53:10 smithi097 podman[162328]: 2024-03-11 21:53:10.621908923 +0000 UTC m=+0.528166076 container remove 02c16113a38a9503d3369f699573e38bf8895d2509a21b25536a2ac3ac5623a3 (image=quay.io/prometheus/node-exporter:v1.5.0, name=ceph-00db0016-dfed-11ee-95c8-87774f69a715-node-exporter-b, maintainer=The Prometheus Authors ) 2024-03-11T21:53:10.864 INFO:journalctl@ceph.node-exporter.b.smithi097.stdout:Mar 11 21:53:10 smithi097 systemd[1]: ceph-00db0016-dfed-11ee-95c8-87774f69a715@node-exporter.b.service: Main process exited, code=exited, status=143/n/a 2024-03-11T21:53:10.864 INFO:journalctl@ceph.node-exporter.b.smithi097.stdout:Mar 11 21:53:10 smithi097 systemd[1]: ceph-00db0016-dfed-11ee-95c8-87774f69a715@node-exporter.b.service: Failed with result 'exit-code'. 2024-03-11T21:53:10.865 INFO:journalctl@ceph.node-exporter.b.smithi097.stdout:Mar 11 21:53:10 smithi097 systemd[1]: Stopped Ceph node-exporter.b for 00db0016-dfed-11ee-95c8-87774f69a715. 2024-03-11T21:53:10.865 INFO:journalctl@ceph.node-exporter.b.smithi097.stdout:Mar 11 21:53:10 smithi097 systemd[1]: ceph-00db0016-dfed-11ee-95c8-87774f69a715@node-exporter.b.service: Consumed 4.642s CPU time. 2024-03-11T21:53:11.511 DEBUG:teuthology.orchestra.run.smithi092:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-03-11T21:53:11.545 DEBUG:teuthology.orchestra.run.smithi097:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-03-11T21:53:11.574 INFO:tasks.cephadm:Archiving crash dumps... 2024-03-11T21:53:11.577 DEBUG:teuthology.misc:Transferring archived files from smithi092:/var/lib/ceph/00db0016-dfed-11ee-95c8-87774f69a715/crash to /home/teuthworker/archive/teuthology-2024-03-11_20:08:13-orch-main-distro-default-smithi/7591804/remote/smithi092/crash 2024-03-11T21:53:11.577 DEBUG:teuthology.orchestra.run.smithi092:> sudo tar c -f - -C /var/lib/ceph/00db0016-dfed-11ee-95c8-87774f69a715/crash -- . 2024-03-11T21:53:11.634 INFO:teuthology.orchestra.run.smithi092.stderr:tar: /var/lib/ceph/00db0016-dfed-11ee-95c8-87774f69a715/crash: Cannot open: No such file or directory 2024-03-11T21:53:11.635 INFO:teuthology.orchestra.run.smithi092.stderr:tar: Error is not recoverable: exiting now 2024-03-11T21:53:11.637 DEBUG:teuthology.misc:Transferring archived files from smithi097:/var/lib/ceph/00db0016-dfed-11ee-95c8-87774f69a715/crash to /home/teuthworker/archive/teuthology-2024-03-11_20:08:13-orch-main-distro-default-smithi/7591804/remote/smithi097/crash 2024-03-11T21:53:11.637 DEBUG:teuthology.orchestra.run.smithi097:> sudo tar c -f - -C /var/lib/ceph/00db0016-dfed-11ee-95c8-87774f69a715/crash -- . 2024-03-11T21:53:11.679 INFO:teuthology.orchestra.run.smithi097.stderr:tar: /var/lib/ceph/00db0016-dfed-11ee-95c8-87774f69a715/crash: Cannot open: No such file or directory 2024-03-11T21:53:11.679 INFO:teuthology.orchestra.run.smithi097.stderr:tar: Error is not recoverable: exiting now 2024-03-11T21:53:11.680 INFO:tasks.cephadm:Checking cluster log for badness... 2024-03-11T21:53:11.681 DEBUG:teuthology.orchestra.run.smithi092:> sudo egrep '\[ERR\]|\[WRN\]|\[SEC\]' /var/log/ceph/00db0016-dfed-11ee-95c8-87774f69a715/ceph.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v 'but it is still running' | egrep -v 'objects unfound and apparently lost' | egrep -v '\(POOL_APP_NOT_ENABLED\)' | head -n 1 2024-03-11T21:53:11.805 INFO:teuthology.orchestra.run.smithi092.stdout:2024-03-11T21:30:00.002162+0000 mon.a (mon.0) 921 : cluster 3 [WRN] OSDMAP_FLAGS: noscrub,nodeep-scrub flag(s) set 2024-03-11T21:53:11.806 WARNING:tasks.cephadm:Found errors (ERR|WRN|SEC) in cluster log 2024-03-11T21:53:11.806 DEBUG:teuthology.orchestra.run.smithi092:> sudo egrep '\[SEC\]' /var/log/ceph/00db0016-dfed-11ee-95c8-87774f69a715/ceph.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v 'but it is still running' | egrep -v 'objects unfound and apparently lost' | egrep -v '\(POOL_APP_NOT_ENABLED\)' | head -n 1 2024-03-11T21:53:11.837 DEBUG:teuthology.orchestra.run.smithi092:> sudo egrep '\[ERR\]' /var/log/ceph/00db0016-dfed-11ee-95c8-87774f69a715/ceph.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v 'but it is still running' | egrep -v 'objects unfound and apparently lost' | egrep -v '\(POOL_APP_NOT_ENABLED\)' | head -n 1 2024-03-11T21:53:11.908 DEBUG:teuthology.orchestra.run.smithi092:> sudo egrep '\[WRN\]' /var/log/ceph/00db0016-dfed-11ee-95c8-87774f69a715/ceph.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v 'but it is still running' | egrep -v 'objects unfound and apparently lost' | egrep -v '\(POOL_APP_NOT_ENABLED\)' | head -n 1 2024-03-11T21:53:11.977 INFO:teuthology.orchestra.run.smithi092.stdout:2024-03-11T21:30:00.002162+0000 mon.a (mon.0) 921 : cluster 3 [WRN] OSDMAP_FLAGS: noscrub,nodeep-scrub flag(s) set 2024-03-11T21:53:11.977 INFO:tasks.cephadm:Compressing logs... 2024-03-11T21:53:11.977 DEBUG:teuthology.orchestra.run.smithi092:> time sudo find /var/log/ceph /var/log/rbd-target-api -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-03-11T21:53:12.020 DEBUG:teuthology.orchestra.run.smithi097:> time sudo find /var/log/ceph /var/log/rbd-target-api -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-03-11T21:53:12.075 INFO:teuthology.orchestra.run.smithi097.stderr:find: ‘/var/log/rbd-target-api’: No such file or directory 2024-03-11T21:53:12.075 INFO:teuthology.orchestra.run.smithi097.stderr:gzip -5 --verbose -- /var/log/ceph/cephadm.log 2024-03-11T21:53:12.085 INFO:teuthology.orchestra.run.smithi097.stderr:gzip -5 --verbose -- /var/log/ceph/00db0016-dfed-11ee-95c8-87774f69a715/ceph-osd.6.log 2024-03-11T21:53:12.086 INFO:teuthology.orchestra.run.smithi097.stderr:/var/log/ceph/cephadm.log: gzip -5 --verbose -- /var/log/ceph/00db0016-dfed-11ee-95c8-87774f69a715/ceph.audit.log 2024-03-11T21:53:12.086 INFO:teuthology.orchestra.run.smithi097.stderr:/var/log/ceph/00db0016-dfed-11ee-95c8-87774f69a715/ceph-osd.6.log: gzip -5 --verbose -- /var/log/ceph/00db0016-dfed-11ee-95c8-87774f69a715/ceph-osd.4.log 2024-03-11T21:53:12.086 INFO:teuthology.orchestra.run.smithi097.stderr:/var/log/ceph/00db0016-dfed-11ee-95c8-87774f69a715/ceph.audit.log: gzip -5 --verbose -- /var/log/ceph/00db0016-dfed-11ee-95c8-87774f69a715/ceph-osd.7.log 2024-03-11T21:53:12.086 INFO:teuthology.orchestra.run.smithi097.stderr:/var/log/ceph/00db0016-dfed-11ee-95c8-87774f69a715/ceph-osd.4.log: gzip -5 --verbose -- /var/log/ceph/00db0016-dfed-11ee-95c8-87774f69a715/ceph-osd.5.log 2024-03-11T21:53:12.086 INFO:teuthology.orchestra.run.smithi097.stderr:/var/log/ceph/00db0016-dfed-11ee-95c8-87774f69a715/ceph-osd.7.log: gzip -5 --verbose -- /var/log/ceph/00db0016-dfed-11ee-95c8-87774f69a715/ceph-mgr.x.log 2024-03-11T21:53:12.086 INFO:teuthology.orchestra.run.smithi097.stderr:/var/log/ceph/00db0016-dfed-11ee-95c8-87774f69a715/ceph-osd.5.log: gzip -5 --verbose -- /var/log/ceph/00db0016-dfed-11ee-95c8-87774f69a715/tcmu-runner.log 2024-03-11T21:53:12.086 INFO:teuthology.orchestra.run.smithi097.stderr:/var/log/ceph/00db0016-dfed-11ee-95c8-87774f69a715/ceph-mgr.x.log: gzip -5 --verbose -- /var/log/ceph/00db0016-dfed-11ee-95c8-87774f69a715/ceph.cephadm.log 2024-03-11T21:53:12.086 INFO:teuthology.orchestra.run.smithi097.stderr:/var/log/ceph/00db0016-dfed-11ee-95c8-87774f69a715/tcmu-runner.log: gzip -5 --verbose -- /var/log/ceph/00db0016-dfed-11ee-95c8-87774f69a715/ceph.log 2024-03-11T21:53:12.086 INFO:teuthology.orchestra.run.smithi097.stderr:/var/log/ceph/00db0016-dfed-11ee-95c8-87774f69a715/ceph.cephadm.log: gzip -5 --verbose -- /var/log/ceph/00db0016-dfed-11ee-95c8-87774f69a715/ceph-volume.log 2024-03-11T21:53:12.086 INFO:teuthology.orchestra.run.smithi097.stderr:/var/log/ceph/00db0016-dfed-11ee-95c8-87774f69a715/ceph.log: gzip -5 --verbose -- /var/log/ceph/00db0016-dfed-11ee-95c8-87774f69a715/ceph-mon.b.log 2024-03-11T21:53:12.110 INFO:teuthology.orchestra.run.smithi092.stderr:find: ‘/var/log/rbd-target-api’: No such file or directory 2024-03-11T21:53:12.110 INFO:teuthology.orchestra.run.smithi092.stderr:gzip -5 --verbose -- /var/log/ceph/cephadm.log 2024-03-11T21:53:12.116 INFO:teuthology.orchestra.run.smithi092.stderr:gzip -5 --verbose -- /var/log/ceph/00db0016-dfed-11ee-95c8-87774f69a715/ceph-mon.a.log 2024-03-11T21:53:12.116 INFO:teuthology.orchestra.run.smithi092.stderr:gzip -5 --verbose -- /var/log/ceph/00db0016-dfed-11ee-95c8-87774f69a715/ceph.audit.log 2024-03-11T21:53:12.116 INFO:teuthology.orchestra.run.smithi092.stderr:/var/log/ceph/cephadm.log: /var/log/ceph/00db0016-dfed-11ee-95c8-87774f69a715/ceph-mon.a.log: gzip -5 --verbose -- /var/log/ceph/00db0016-dfed-11ee-95c8-87774f69a715/ceph-client.rgw.foo.a.log 2024-03-11T21:53:12.116 INFO:teuthology.orchestra.run.smithi092.stderr:/var/log/ceph/00db0016-dfed-11ee-95c8-87774f69a715/ceph.audit.log: gzip -5 --verbose -- /var/log/ceph/00db0016-dfed-11ee-95c8-87774f69a715/ceph-osd.1.log 2024-03-11T21:53:12.116 INFO:teuthology.orchestra.run.smithi092.stderr:/var/log/ceph/00db0016-dfed-11ee-95c8-87774f69a715/ceph-client.rgw.foo.a.log: gzip -5 --verbose -- /var/log/ceph/00db0016-dfed-11ee-95c8-87774f69a715/ceph-osd.3.log 2024-03-11T21:53:12.117 INFO:teuthology.orchestra.run.smithi092.stderr: 57.6% -- replaced with /var/log/ceph/00db0016-dfed-11ee-95c8-87774f69a715/ceph-client.rgw.foo.a.log.gz 2024-03-11T21:53:12.117 INFO:teuthology.orchestra.run.smithi092.stderr:/var/log/ceph/00db0016-dfed-11ee-95c8-87774f69a715/ceph-osd.1.log: gzip -5 --verbose -- /var/log/ceph/00db0016-dfed-11ee-95c8-87774f69a715/ceph-mgr.y.log 2024-03-11T21:53:12.117 INFO:teuthology.orchestra.run.smithi092.stderr:/var/log/ceph/00db0016-dfed-11ee-95c8-87774f69a715/ceph-osd.3.log: gzip -5 --verbose -- /var/log/ceph/00db0016-dfed-11ee-95c8-87774f69a715/ceph-osd.0.log 2024-03-11T21:53:12.117 INFO:teuthology.orchestra.run.smithi092.stderr:/var/log/ceph/00db0016-dfed-11ee-95c8-87774f69a715/ceph-mgr.y.log: gzip -5 --verbose -- /var/log/ceph/00db0016-dfed-11ee-95c8-87774f69a715/ceph.cephadm.log 2024-03-11T21:53:12.117 INFO:teuthology.orchestra.run.smithi092.stderr:/var/log/ceph/00db0016-dfed-11ee-95c8-87774f69a715/ceph-osd.0.log: gzip -5 --verbose -- /var/log/ceph/00db0016-dfed-11ee-95c8-87774f69a715/ceph.log 2024-03-11T21:53:12.117 INFO:teuthology.orchestra.run.smithi092.stderr:/var/log/ceph/00db0016-dfed-11ee-95c8-87774f69a715/ceph.cephadm.log: gzip -5 --verbose -- /var/log/ceph/00db0016-dfed-11ee-95c8-87774f69a715/ceph-osd.2.log 2024-03-11T21:53:12.117 INFO:teuthology.orchestra.run.smithi092.stderr:/var/log/ceph/00db0016-dfed-11ee-95c8-87774f69a715/ceph.log: gzip -5 --verbose -- /var/log/ceph/00db0016-dfed-11ee-95c8-87774f69a715/ceph-volume.log 2024-03-11T21:53:12.117 INFO:teuthology.orchestra.run.smithi092.stderr:/var/log/ceph/00db0016-dfed-11ee-95c8-87774f69a715/ceph-osd.2.log: gzip -5 --verbose -- /var/log/ceph/00db0016-dfed-11ee-95c8-87774f69a715/objectstore_tool.2.log 2024-03-11T21:53:12.117 INFO:teuthology.orchestra.run.smithi092.stderr:/var/log/ceph/00db0016-dfed-11ee-95c8-87774f69a715/ceph-volume.log: gzip -5 --verbose -- /var/log/ceph/00db0016-dfed-11ee-95c8-87774f69a715/ceph-mon.c.log 2024-03-11T21:53:12.158 INFO:teuthology.orchestra.run.smithi097.stderr:/var/log/ceph/00db0016-dfed-11ee-95c8-87774f69a715/ceph-volume.log: /var/log/ceph/00db0016-dfed-11ee-95c8-87774f69a715/ceph-mon.b.log: 92.0% -- replaced with /var/log/ceph/00db0016-dfed-11ee-95c8-87774f69a715/ceph-mgr.x.log.gz 2024-03-11T21:53:12.183 INFO:teuthology.orchestra.run.smithi092.stderr:/var/log/ceph/00db0016-dfed-11ee-95c8-87774f69a715/objectstore_tool.2.log: /var/log/ceph/00db0016-dfed-11ee-95c8-87774f69a715/ceph-mon.c.log: 91.6% -- replaced with /var/log/ceph/00db0016-dfed-11ee-95c8-87774f69a715/ceph.log.gz 2024-03-11T21:53:12.188 INFO:teuthology.orchestra.run.smithi097.stderr: 89.4% -- replaced with /var/log/ceph/cephadm.log.gz 2024-03-11T21:53:12.191 INFO:teuthology.orchestra.run.smithi097.stderr: 89.9% -- replaced with /var/log/ceph/00db0016-dfed-11ee-95c8-87774f69a715/ceph.audit.log.gz 2024-03-11T21:53:12.229 INFO:teuthology.orchestra.run.smithi097.stderr: 67.6% -- replaced with /var/log/ceph/00db0016-dfed-11ee-95c8-87774f69a715/tcmu-runner.log.gz 2024-03-11T21:53:12.268 INFO:teuthology.orchestra.run.smithi092.stderr: 91.1% -- replaced with /var/log/ceph/00db0016-dfed-11ee-95c8-87774f69a715/ceph-volume.log.gz 2024-03-11T21:53:12.310 INFO:teuthology.orchestra.run.smithi097.stderr: 91.2% -- replaced with /var/log/ceph/00db0016-dfed-11ee-95c8-87774f69a715/ceph-volume.log.gz 2024-03-11T21:53:12.439 INFO:teuthology.orchestra.run.smithi097.stderr: 81.7% -- replaced with /var/log/ceph/00db0016-dfed-11ee-95c8-87774f69a715/ceph.cephadm.log.gz 2024-03-11T21:53:12.522 INFO:teuthology.orchestra.run.smithi097.stderr: 85.1% -- replaced with /var/log/ceph/00db0016-dfed-11ee-95c8-87774f69a715/ceph.log.gz 2024-03-11T21:53:12.617 INFO:teuthology.orchestra.run.smithi092.stderr: 94.3% -- replaced with /var/log/ceph/00db0016-dfed-11ee-95c8-87774f69a715/objectstore_tool.2.log.gz 2024-03-11T21:53:12.631 INFO:teuthology.orchestra.run.smithi092.stderr: 93.9% -- replaced with /var/log/ceph/00db0016-dfed-11ee-95c8-87774f69a715/ceph.audit.log.gz 2024-03-11T21:53:12.665 INFO:teuthology.orchestra.run.smithi092.stderr: 89.1% -- replaced with /var/log/ceph/00db0016-dfed-11ee-95c8-87774f69a715/ceph.cephadm.log.gz 2024-03-11T21:53:12.702 INFO:teuthology.orchestra.run.smithi092.stderr: 95.3% -- replaced with /var/log/ceph/cephadm.log.gz 2024-03-11T21:53:13.034 INFO:teuthology.orchestra.run.smithi092.stderr: 89.1% -- replaced with /var/log/ceph/00db0016-dfed-11ee-95c8-87774f69a715/ceph-mgr.y.log.gz 2024-03-11T21:53:17.810 INFO:teuthology.orchestra.run.smithi097.stderr: 90.6% -- replaced with /var/log/ceph/00db0016-dfed-11ee-95c8-87774f69a715/ceph-mon.b.log.gz 2024-03-11T21:53:21.995 INFO:teuthology.orchestra.run.smithi092.stderr: 90.7% -- replaced with /var/log/ceph/00db0016-dfed-11ee-95c8-87774f69a715/ceph-mon.c.log.gz 2024-03-11T21:53:39.447 INFO:teuthology.orchestra.run.smithi092.stderr: 90.1% -- replaced with /var/log/ceph/00db0016-dfed-11ee-95c8-87774f69a715/ceph-mon.a.log.gz 2024-03-11T21:54:45.452 INFO:teuthology.orchestra.run.smithi092.stderr: 92.8% -- replaced with /var/log/ceph/00db0016-dfed-11ee-95c8-87774f69a715/ceph-osd.2.log.gz 2024-03-11T21:58:09.158 INFO:teuthology.orchestra.run.smithi092.stderr: 93.1% -- replaced with /var/log/ceph/00db0016-dfed-11ee-95c8-87774f69a715/ceph-osd.3.log.gz 2024-03-11T21:58:40.546 INFO:teuthology.orchestra.run.smithi097.stderr: 92.6% -- replaced with /var/log/ceph/00db0016-dfed-11ee-95c8-87774f69a715/ceph-osd.4.log.gz 2024-03-11T22:01:35.915 INFO:teuthology.orchestra.run.smithi097.stderr: 93.0% -- replaced with /var/log/ceph/00db0016-dfed-11ee-95c8-87774f69a715/ceph-osd.7.log.gz 2024-03-11T22:01:50.185 INFO:teuthology.orchestra.run.smithi092.stderr: 92.4% -- replaced with /var/log/ceph/00db0016-dfed-11ee-95c8-87774f69a715/ceph-osd.1.log.gz 2024-03-11T22:02:31.807 INFO:teuthology.orchestra.run.smithi092.stderr: 92.3% -- replaced with /var/log/ceph/00db0016-dfed-11ee-95c8-87774f69a715/ceph-osd.0.log.gz 2024-03-11T22:02:31.809 INFO:teuthology.orchestra.run.smithi092.stderr: 2024-03-11T22:02:31.809 INFO:teuthology.orchestra.run.smithi092.stderr:real 9m19.775s 2024-03-11T22:02:31.809 INFO:teuthology.orchestra.run.smithi092.stderr:user 7m24.572s 2024-03-11T22:02:31.809 INFO:teuthology.orchestra.run.smithi092.stderr:sys 0m19.240s 2024-03-11T22:03:44.406 INFO:teuthology.orchestra.run.smithi097.stderr: 92.5% -- replaced with /var/log/ceph/00db0016-dfed-11ee-95c8-87774f69a715/ceph-osd.5.log.gz 2024-03-11T22:03:50.354 INFO:teuthology.orchestra.run.smithi097.stderr: 93.0% -- replaced with /var/log/ceph/00db0016-dfed-11ee-95c8-87774f69a715/ceph-osd.6.log.gz 2024-03-11T22:03:50.356 INFO:teuthology.orchestra.run.smithi097.stderr: 2024-03-11T22:03:50.356 INFO:teuthology.orchestra.run.smithi097.stderr:real 10m38.320s 2024-03-11T22:03:50.356 INFO:teuthology.orchestra.run.smithi097.stderr:user 6m16.904s 2024-03-11T22:03:50.357 INFO:teuthology.orchestra.run.smithi097.stderr:sys 0m16.993s 2024-03-11T22:03:50.357 INFO:tasks.cephadm:Archiving logs... 2024-03-11T22:03:50.357 DEBUG:teuthology.misc:Transferring archived files from smithi092:/var/log/ceph to /home/teuthworker/archive/teuthology-2024-03-11_20:08:13-orch-main-distro-default-smithi/7591804/remote/smithi092/log 2024-03-11T22:03:50.358 DEBUG:teuthology.orchestra.run.smithi092:> sudo tar c -f - -C /var/log/ceph -- . 2024-03-11T22:05:03.052 DEBUG:teuthology.misc:Transferring archived files from smithi097:/var/log/ceph to /home/teuthworker/archive/teuthology-2024-03-11_20:08:13-orch-main-distro-default-smithi/7591804/remote/smithi097/log 2024-03-11T22:05:03.054 DEBUG:teuthology.orchestra.run.smithi097:> sudo tar c -f - -C /var/log/ceph -- . 2024-03-11T22:06:03.005 INFO:tasks.cephadm:Removing cluster... 2024-03-11T22:06:03.006 DEBUG:teuthology.orchestra.run.smithi092:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid 00db0016-dfed-11ee-95c8-87774f69a715 --force 2024-03-11T22:06:03.250 INFO:teuthology.orchestra.run.smithi092.stdout:Deleting cluster with fsid: 00db0016-dfed-11ee-95c8-87774f69a715 2024-03-11T22:06:04.136 DEBUG:teuthology.orchestra.run.smithi097:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid 00db0016-dfed-11ee-95c8-87774f69a715 --force 2024-03-11T22:06:04.433 INFO:teuthology.orchestra.run.smithi097.stdout:Deleting cluster with fsid: 00db0016-dfed-11ee-95c8-87774f69a715 2024-03-11T22:06:05.184 INFO:tasks.cephadm:Removing cephadm ... 2024-03-11T22:06:05.184 DEBUG:teuthology.orchestra.run.smithi092:> rm -rf /home/ubuntu/cephtest/cephadm 2024-03-11T22:06:05.216 DEBUG:teuthology.orchestra.run.smithi097:> rm -rf /home/ubuntu/cephtest/cephadm 2024-03-11T22:06:05.236 INFO:tasks.cephadm:Teardown complete 2024-03-11T22:06:05.237 DEBUG:teuthology.run_tasks:Unwinding manager install 2024-03-11T22:06:05.247 INFO:teuthology.task.install.util:Removing shipped files: /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer... 2024-03-11T22:06:05.247 DEBUG:teuthology.orchestra.run.smithi092:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-03-11T22:06:05.259 DEBUG:teuthology.orchestra.run.smithi097:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-03-11T22:06:05.389 DEBUG:teuthology.run_tasks:Unwinding manager clock 2024-03-11T22:06:05.397 INFO:teuthology.task.clock:Checking final clock skew... 2024-03-11T22:06:05.398 DEBUG:teuthology.orchestra.run.smithi092:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-03-11T22:06:05.400 DEBUG:teuthology.orchestra.run.smithi097:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-03-11T22:06:05.419 INFO:teuthology.orchestra.run.smithi097.stderr:bash: line 1: ntpq: command not found 2024-03-11T22:06:05.423 INFO:teuthology.orchestra.run.smithi092.stderr:bash: line 1: ntpq: command not found 2024-03-11T22:06:05.450 INFO:teuthology.orchestra.run.smithi097.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-03-11T22:06:05.450 INFO:teuthology.orchestra.run.smithi097.stdout:=============================================================================== 2024-03-11T22:06:05.450 INFO:teuthology.orchestra.run.smithi097.stdout:^+ hv01.front.sepia.ceph.com 2 6 377 2 +4890us[+4890us] +/- 111ms 2024-03-11T22:06:05.450 INFO:teuthology.orchestra.run.smithi097.stdout:^* hv02.front.sepia.ceph.com 3 6 377 8 -2152us[-2148us] +/- 59ms 2024-03-11T22:06:05.450 INFO:teuthology.orchestra.run.smithi097.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 7 -678us[ -678us] +/- 87ms 2024-03-11T22:06:05.450 INFO:teuthology.orchestra.run.smithi097.stdout:^? hv04.front.sepia.ceph.com 0 9 0 - +0ns[ +0ns] +/- 0ns 2024-03-11T22:06:05.452 INFO:teuthology.orchestra.run.smithi092.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-03-11T22:06:05.452 INFO:teuthology.orchestra.run.smithi092.stdout:=============================================================================== 2024-03-11T22:06:05.452 INFO:teuthology.orchestra.run.smithi092.stdout:^+ hv01.front.sepia.ceph.com 2 6 377 10 +4928us[+4928us] +/- 111ms 2024-03-11T22:06:05.452 INFO:teuthology.orchestra.run.smithi092.stdout:^* hv02.front.sepia.ceph.com 3 7 377 73 -2078us[-2094us] +/- 58ms 2024-03-11T22:06:05.452 INFO:teuthology.orchestra.run.smithi092.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 9 -611us[ -611us] +/- 87ms 2024-03-11T22:06:05.452 INFO:teuthology.orchestra.run.smithi092.stdout:^? hv04.front.sepia.ceph.com 0 9 0 - +0ns[ +0ns] +/- 0ns 2024-03-11T22:06:05.453 DEBUG:teuthology.run_tasks:Unwinding manager ansible.cephlab 2024-03-11T22:06:05.462 INFO:teuthology.task.ansible:Skipping ansible cleanup... 2024-03-11T22:06:05.463 DEBUG:teuthology.run_tasks:Unwinding manager selinux 2024-03-11T22:06:05.471 DEBUG:teuthology.orchestra.run.smithi092:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2024-03-11T22:06:05.599 DEBUG:teuthology.orchestra.run.smithi097:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2024-03-11T22:06:05.674 DEBUG:teuthology.orchestra.run.smithi092:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e scontext=system_u:system_r:logrotate_t:s0 2024-03-11T22:06:05.707 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:06:05.708 DEBUG:teuthology.orchestra.run.smithi097:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e scontext=system_u:system_r:logrotate_t:s0 2024-03-11T22:06:05.735 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:06:05.736 DEBUG:teuthology.run_tasks:Unwinding manager pcp 2024-03-11T22:06:05.744 DEBUG:teuthology.run_tasks:Unwinding manager internal.timer 2024-03-11T22:06:05.752 INFO:teuthology.task.internal:Duration was 3370.875646 seconds 2024-03-11T22:06:05.753 DEBUG:teuthology.run_tasks:Unwinding manager internal.syslog 2024-03-11T22:06:05.761 INFO:teuthology.task.internal.syslog:Shutting down syslog monitoring... 2024-03-11T22:06:05.761 DEBUG:teuthology.orchestra.run.smithi092:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-03-11T22:06:05.764 DEBUG:teuthology.orchestra.run.smithi097:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-03-11T22:06:05.825 INFO:teuthology.orchestra.run.smithi097.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-03-11T22:06:05.851 INFO:teuthology.orchestra.run.smithi092.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-03-11T22:06:06.203 INFO:teuthology.task.internal.syslog:Checking logs for errors... 2024-03-11T22:06:06.203 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi092.front.sepia.ceph.com 2024-03-11T22:06:06.203 DEBUG:teuthology.orchestra.run.smithi092:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-03-11T22:06:06.227 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi097.front.sepia.ceph.com 2024-03-11T22:06:06.227 DEBUG:teuthology.orchestra.run.smithi097:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-03-11T22:06:06.259 INFO:teuthology.task.internal.syslog:Compressing syslogs... 2024-03-11T22:06:06.259 DEBUG:teuthology.orchestra.run.smithi092:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-03-11T22:06:06.269 DEBUG:teuthology.orchestra.run.smithi097:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-03-11T22:06:06.328 INFO:teuthology.task.internal.syslog:Gathering journactl -b0... 2024-03-11T22:06:06.328 DEBUG:teuthology.orchestra.run.smithi092:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-03-11T22:06:06.342 DEBUG:teuthology.orchestra.run.smithi097:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-03-11T22:06:07.807 DEBUG:teuthology.run_tasks:Unwinding manager internal.sudo 2024-03-11T22:06:07.817 INFO:teuthology.task.internal:Restoring /etc/sudoers... 2024-03-11T22:06:07.818 DEBUG:teuthology.orchestra.run.smithi092:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-03-11T22:06:07.845 DEBUG:teuthology.orchestra.run.smithi097:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-03-11T22:06:07.877 DEBUG:teuthology.run_tasks:Unwinding manager internal.coredump 2024-03-11T22:06:07.886 DEBUG:teuthology.orchestra.run.smithi092:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-03-11T22:06:07.888 DEBUG:teuthology.orchestra.run.smithi097:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-03-11T22:06:07.940 INFO:teuthology.orchestra.run.smithi092.stdout:kernel.core_pattern = core 2024-03-11T22:06:07.955 INFO:teuthology.orchestra.run.smithi097.stdout:kernel.core_pattern = core 2024-03-11T22:06:07.984 DEBUG:teuthology.orchestra.run.smithi092:> test -e /home/ubuntu/cephtest/archive/coredump 2024-03-11T22:06:07.997 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:06:07.998 DEBUG:teuthology.orchestra.run.smithi097:> test -e /home/ubuntu/cephtest/archive/coredump 2024-03-11T22:06:08.031 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-11T22:06:08.032 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive 2024-03-11T22:06:08.042 INFO:teuthology.task.internal:Transferring archived files... 2024-03-11T22:06:08.042 DEBUG:teuthology.misc:Transferring archived files from smithi092:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/teuthology-2024-03-11_20:08:13-orch-main-distro-default-smithi/7591804/remote/smithi092 2024-03-11T22:06:08.043 DEBUG:teuthology.orchestra.run.smithi092:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-03-11T22:06:08.113 DEBUG:teuthology.misc:Transferring archived files from smithi097:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/teuthology-2024-03-11_20:08:13-orch-main-distro-default-smithi/7591804/remote/smithi097 2024-03-11T22:06:08.114 DEBUG:teuthology.orchestra.run.smithi097:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-03-11T22:06:08.155 INFO:teuthology.task.internal:Removing archive directory... 2024-03-11T22:06:08.155 DEBUG:teuthology.orchestra.run.smithi092:> rm -rf -- /home/ubuntu/cephtest/archive 2024-03-11T22:06:08.157 DEBUG:teuthology.orchestra.run.smithi097:> rm -rf -- /home/ubuntu/cephtest/archive 2024-03-11T22:06:08.204 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive_upload 2024-03-11T22:06:08.216 INFO:teuthology.task.internal:Not uploading archives. 2024-03-11T22:06:08.216 DEBUG:teuthology.run_tasks:Unwinding manager internal.base 2024-03-11T22:06:08.225 INFO:teuthology.task.internal:Tidying up after the test... 2024-03-11T22:06:08.225 DEBUG:teuthology.orchestra.run.smithi092:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-03-11T22:06:08.228 DEBUG:teuthology.orchestra.run.smithi097:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-03-11T22:06:08.240 INFO:teuthology.orchestra.run.smithi092.stdout: 265570 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 11 22:06 /home/ubuntu/cephtest 2024-03-11T22:06:08.260 INFO:teuthology.orchestra.run.smithi097.stdout: 265300 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 11 22:06 /home/ubuntu/cephtest 2024-03-11T22:06:08.261 DEBUG:teuthology.run_tasks:Unwinding manager kernel 2024-03-11T22:06:08.270 DEBUG:teuthology.run_tasks:Unwinding manager console_log 2024-03-11T22:06:08.290 INFO:teuthology.nuke:Checking targets against current locks 2024-03-11T22:06:08.317 DEBUG:teuthology.nuke:shortname: smithi092 2024-03-11T22:06:08.317 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-03-11T22:06:08.340 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi092.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/teuthology-2024-03-11_20:08:13-orch-main-distro-default-smithi/7591804', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-03-11 20:59:37.405444', 'locked_by': 'scheduled_teuthology@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBO2eOnecSN6CH/yt9an69HTrIAcybKj1w6rVksVunYh88HFogz8kG7ra0xdBHXWEF9Ga340/EqBYclg4h7diz1w='} 2024-03-11T22:06:08.343 DEBUG:teuthology.nuke:shortname: smithi097 2024-03-11T22:06:08.343 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-03-11T22:06:08.364 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi097.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/teuthology-2024-03-11_20:08:13-orch-main-distro-default-smithi/7591804', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-03-11 20:59:37.406977', 'locked_by': 'scheduled_teuthology@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBGoKhCCw7+ZfNtOLLnmD5QxlgNJ6jjDm1kgY07XdEwuq15OTBdIwGlcAaxlKl8z7Gx0OncR9d446T8YZC3lFVMw='} 2024-03-11T22:06:08.389 INFO:teuthology.orchestra.console.smithi092:Power off 2024-03-11T22:06:08.390 DEBUG:teuthology.orchestra.console.smithi092:pexpect command: ipmitool -H smithi092.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2024-03-11T22:06:08.413 DEBUG:teuthology.orchestra.console.smithi092:power off output: Chassis Power Control: Down/Off 2024-03-11T22:06:08.413 DEBUG:teuthology.orchestra.console.smithi092:pexpect command: ipmitool -H smithi092.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-11T22:06:08.430 INFO:teuthology.orchestra.console.smithi097:Power off 2024-03-11T22:06:08.430 DEBUG:teuthology.orchestra.console.smithi097:pexpect command: ipmitool -H smithi097.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2024-03-11T22:06:08.439 DEBUG:teuthology.orchestra.console.smithi092:check power output: Chassis Power is on 2024-03-11T22:06:08.452 DEBUG:teuthology.orchestra.console.smithi097:power off output: Chassis Power Control: Down/Off 2024-03-11T22:06:08.452 DEBUG:teuthology.orchestra.console.smithi097:pexpect command: ipmitool -H smithi097.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-11T22:06:08.474 DEBUG:teuthology.orchestra.console.smithi097:check power output: Chassis Power is on 2024-03-11T22:06:12.440 DEBUG:teuthology.orchestra.console.smithi092:pexpect command: ipmitool -H smithi092.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-11T22:06:12.475 DEBUG:teuthology.orchestra.console.smithi097:pexpect command: ipmitool -H smithi097.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-11T22:06:12.551 DEBUG:teuthology.orchestra.console.smithi092:check power output: Chassis Power is on 2024-03-11T22:06:12.584 DEBUG:teuthology.orchestra.console.smithi097:check power output: Chassis Power is on 2024-03-11T22:06:16.555 DEBUG:teuthology.orchestra.console.smithi092:pexpect command: ipmitool -H smithi092.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-11T22:06:16.585 DEBUG:teuthology.orchestra.console.smithi097:pexpect command: ipmitool -H smithi097.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-11T22:06:16.665 DEBUG:teuthology.orchestra.console.smithi092:check power output: Chassis Power is off 2024-03-11T22:06:16.694 DEBUG:teuthology.orchestra.console.smithi097:check power output: Chassis Power is on 2024-03-11T22:06:16.766 INFO:teuthology.orchestra.console.smithi092:Power off completed 2024-03-11T22:06:20.698 DEBUG:teuthology.orchestra.console.smithi097:pexpect command: ipmitool -H smithi097.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-11T22:06:20.808 DEBUG:teuthology.orchestra.console.smithi097:check power output: Chassis Power is off 2024-03-11T22:06:20.909 INFO:teuthology.orchestra.console.smithi097:Power off completed 2024-03-11T22:06:21.012 INFO:teuthology.run:Summary data: description: orch/cephadm/thrash/{0-distro/centos_9.stream 1-start 2-thrash 3-tasks/radosbench fixed-2 msgr/async-v1only root} duration: 3370.875646352768 failure_reason: '"2024-03-11T21:30:00.002162+0000 mon.a (mon.0) 921 : cluster 3 [WRN] OSDMAP_FLAGS: noscrub,nodeep-scrub flag(s) set" in cluster log' flavor: default owner: scheduled_teuthology@teuthology success: false 2024-03-11T22:06:21.012 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-03-11T22:06:21.092 INFO:teuthology.run:FAIL